Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
480756d832 | ||
|
|
135fb34e00 | ||
|
|
9efa3bbc8a | ||
|
|
fad2bd3353 | ||
|
|
930e42b0b0 | ||
|
|
16b4923bd7 | ||
|
|
1cfd062835 | ||
|
|
21233bd578 | ||
|
|
990cc739df | ||
|
|
3500bd4624 | ||
|
|
374575ec01 | ||
|
|
f57b430052 | ||
|
|
8a9d60a5a7 | ||
|
|
b963311b43 | ||
|
|
50bdb8232b | ||
|
|
ee8e24ffc6 | ||
|
|
44b94caacf | ||
|
|
cbb4b467ad | ||
|
|
e65cf0d958 | ||
|
|
4a57c91e29 | ||
|
|
2289107911 | ||
|
|
782097420d | ||
|
|
82382d90f2 | ||
|
|
5a16a45922 | ||
|
|
6a6f6cedbc | ||
|
|
9639a03bfe | ||
|
|
48c10b7436 | ||
|
|
4db6438245 | ||
|
|
fb108ec5ab | ||
|
|
a1ada06a76 | ||
|
|
b5ed47902c | ||
|
|
8a19c2a3f4 | ||
|
|
1c0d78b552 |
@@ -1,3 +0,0 @@
|
||||
{
|
||||
"extends": ["@commitlint/config-conventional"]
|
||||
}
|
||||
@@ -0,0 +1,408 @@
|
||||
---
|
||||
name: Технический аудит EvoBGP
|
||||
overview: "Полный технический аудит EvoBGP для production-сценария (10+ клиентов, нестабильная сеть). Архитектура — hybrid control plane; сильные стороны: stale fallback, CDN/RIPEstat resilience, Ed25519 bundles. Критичные риски: in-process jobs, DoH без retry, misconfiguration demo-seed, отсутствие HA API."
|
||||
todos:
|
||||
- id: quick-ops-checklist
|
||||
content: Применить production-checklist (SEED_DEMO=0, BUNDLE_SEED_HEX, DB/JOB/CONCURRENCY tuning, TLS)
|
||||
status: pending
|
||||
- id: fix-doh-retry
|
||||
content: Добавить DoWithRetry для DoH в internal/pipeline/refresh.go
|
||||
status: pending
|
||||
- id: fix-job-meta-race
|
||||
content: Исправить чтение j.Meta в worker.go через Snapshot() или locked accessor
|
||||
status: pending
|
||||
- id: cdn-preview-resilience
|
||||
content: Перевести CDN preview на upstreamHTTPDo в routes_crud.go
|
||||
status: pending
|
||||
- id: partial-cdn-failure
|
||||
content: "Partial CDN source failure: skip/degrade вместо fail всего модуля"
|
||||
status: pending
|
||||
- id: graceful-shutdown
|
||||
content: Cancel/drain jobs при SIGTERM в cmd/evobgp-api и evobgp-all
|
||||
status: pending
|
||||
- id: ha-job-queue
|
||||
content: "Roadmap: распределённая очередь jobs (PG claim или NATS) для HA API"
|
||||
status: pending
|
||||
isProject: false
|
||||
---
|
||||
|
||||
# Технический аудит EvoBGP
|
||||
|
||||
## Executive summary
|
||||
|
||||
EvoBGP — **hybrid control plane**: один процесс [`evobgp-all`](cmd/evobgp-all/main.go) (monolith) или **reference Compose** с разделёнными воркерами ([`docs/architecture.md`](docs/architecture.md)). Data plane (BIRD + agent) отделён от control plane (API + PostgreSQL + jobs).
|
||||
|
||||
**Сильные стороны для нестабильной сети:**
|
||||
- Stale snapshot fallback по умолчанию (`EVOBGP_STALE_ON_UPSTREAM_ERROR=1`) — [`internal/pipeline/collect_stale.go`](internal/pipeline/collect_stale.go)
|
||||
- CDN/RIPEstat: retry (3×) + per-host circuit breaker — [`internal/httpclient/httpclient.go`](internal/httpclient/httpclient.go), [`circuit.go`](internal/httpclient/circuit.go)
|
||||
- ETag conditional GET, ASN TTL-кэш, parallel collect с cap
|
||||
- Подписанные бандлы Ed25519, verify перед apply
|
||||
|
||||
**Главные риски для 10+ клиентов:**
|
||||
1. `jobs.Registry` — **in-memory, только в процессе API** (ARCH-04)
|
||||
2. DoH — **без retry/breaker** (критично при блокировках провайдеров)
|
||||
3. Один failed CDN source **без stale cache валит весь модуль**
|
||||
4. Production misconfiguration: `Bearer dev`, HTTP API, ephemeral bundle key
|
||||
5. Data race на `Job.Meta` и alias pointers в `store.Memory`
|
||||
|
||||
---
|
||||
|
||||
## 1. Архитектура
|
||||
|
||||
### Стиль
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph hybrid [Hybrid deployment]
|
||||
All[evobgp_all monolith]
|
||||
Split[evobgp_api + workers]
|
||||
end
|
||||
subgraph cp [Control plane]
|
||||
API[HTTP API]
|
||||
Jobs[jobs.Registry in-process]
|
||||
PG[(PostgreSQL)]
|
||||
end
|
||||
subgraph dp [Data plane per speaker]
|
||||
Agent[evobgp_agent]
|
||||
BIRD[BIRD2]
|
||||
NodeCLI[evobgp_node]
|
||||
end
|
||||
All --> API
|
||||
Split --> API
|
||||
API --> Jobs
|
||||
API --> PG
|
||||
NodeCLI --> API
|
||||
Agent --> API
|
||||
Agent --> BIRD
|
||||
```
|
||||
|
||||
| Профиль | Стиль | Когда |
|
||||
|---------|-------|-------|
|
||||
| `microvps` / `evobgp-all` | Monolith | 1 VPS, shared Registry |
|
||||
| reference Compose | Microservices-lite | API + scheduler/ingest/render/deploy |
|
||||
| Remote speakers | Edge agents | Panel→Node dispatch |
|
||||
|
||||
### Узкие места (bottlenecks)
|
||||
|
||||
| # | Bottleneck | Где | Impact |
|
||||
|---|------------|-----|--------|
|
||||
| B1 | **In-process job queue** | [`internal/jobs/job.go:175-177`](internal/jobs/job.go) | HA API невозможен без потери/дублирования jobs; scheduler без `EVOBGP_CONTROL_PLANE_URL` создаёт **отдельный Registry** — [`cmd/evobgp-scheduler/main.go:58-60`](cmd/evobgp-scheduler/main.go) |
|
||||
| B2 | **Module refresh = sync upstream fan-out** | [`internal/pipeline/collect_parallel.go`](internal/pipeline/collect_parallel.go) | До `EVOBGP_COLLECT_CONCURRENCY` (8 default, max 32) параллельных HTTP; worst case ~45s × retries на источник |
|
||||
| B3 | **Default job concurrency = 8** | [`internal/jobs/job.go:264-268`](internal/jobs/job.go) | При burst refresh 10+ tenants — очередь растёт, goroutine блокируются на sem |
|
||||
| B4 | **PostgreSQL pool default** | [`internal/db/open.go:28-38`](internal/db/open.go) | pgx default ~4 conns; при `JOB_MAX=16` + HTTP — contention без `EVOBGP_DB_MAX_CONNS=25` |
|
||||
| B5 | **Live endpoints fan-out** | [`internal/httpapi/peers_live.go`](internal/httpapi/peers_live.go) | N goroutines × N speakers, 12s timeout каждый |
|
||||
| B6 | **Broker — заглушка** | [`internal/broker`](internal/broker) | NATS URL логируется, очередь не распределена |
|
||||
|
||||
### Масштабируемость
|
||||
|
||||
- **Вертикальная:** хорошо до ~10–20 tenants при `evobgp-all` + tuning ([`docs/production-checklist.md`](docs/production-checklist.md))
|
||||
- **Горизонтальная API:** **не поддерживается** — два `evobgp-api` = два независимых Registry; `job_audit` в PG — audit only, не очередь исполнения
|
||||
- **Workers (ingest/render/deploy):** координируются через **общую БД**, не через jobs — OK для prefetch/drift
|
||||
|
||||
### Отказоустойчивость
|
||||
|
||||
| Сценарий | Поведение | Оценка |
|
||||
|----------|-----------|--------|
|
||||
| CDN/RIPEstat недоступен | Stale snapshot + circuit breaker | **Хорошо** (если был prior snapshot) |
|
||||
| DoH недоступен | Fail модуля или stale domain snapshot | **Средне** (нет HTTP retry) |
|
||||
| API restart mid-job | Job теряется из Registry; audit может быть inconsistent | **Плохо** |
|
||||
| PG недоступен | API `/v1/ready` → 503 | **OK** |
|
||||
| Agent unreachable | Deploy job succeed, drift в `evobgp-deploy` | **Частичный fail** (by design) |
|
||||
|
||||
**Рекомендация:** для 10+ клиентов — **`evobgp-all` на каждом CP** или один CP + tuning; HA API требует **распределённой очереди** (NATS/Redis + worker pool) — задокументировано как future work.
|
||||
|
||||
---
|
||||
|
||||
## 2. Анализ кода
|
||||
|
||||
### Антипаттерны
|
||||
|
||||
| ID | Проблема | Файл | Критичность |
|
||||
|----|----------|------|-------------|
|
||||
| A1 | **Concurrent map read/write** — worker читает `j.Meta` без lock, handler пишет через `mergeMeta`/`Snapshot` | [`worker.go:108,263,379`](internal/jobs/worker.go), [`job.go:103-111`](internal/jobs/job.go) | **high** |
|
||||
| A2 | **Escape internal pointers** из Memory store | [`store/memory.go:416-475`](internal/store/memory.go) | **high** (tests/dev); **low** (prod PG) |
|
||||
| A3 | **Fire-and-forget goroutine** на каждый auth | [`auth.go:79-81`](internal/httpapi/auth.go) | **medium** |
|
||||
| A4 | **Silent error swallow** в prefetch | [`internal/ingest/run.go`](internal/ingest/run.go), `prefetch.go` | **medium** |
|
||||
| A5 | **Bypass resilience layer** — CDN preview прямой `Do` | [`routes_crud.go:267`](internal/httpapi/routes_crud.go) | **medium** |
|
||||
| A6 | **`EVOBGP_DEV_INSECURE` — dead code** | compose + [`server.go`](internal/httpapi/server.go) | **low** (misleading ops) |
|
||||
| A7 | **Unused Registry** в ingest/render/deploy binaries | [`cmd/evobgp-ingest/main.go`](cmd/evobgp-ingest/main.go) | **low** (resource waste) |
|
||||
|
||||
### Maintainability
|
||||
|
||||
**Плюсы:** чёткое разделение слоёв (ARCH-01..10), `store.Backend`, OpenAPI как контракт, engineering rules, table-driven tests в birdfmt/pipeline.
|
||||
|
||||
**Минусы:**
|
||||
- Дублирование retry-логики (httpclient vs nodedispatch inline loop)
|
||||
- Env-tuning разбросан (`EVOBGP_*` в 15+ местах без central config struct для pipeline)
|
||||
- `Job` comment «персистенция в БД пока не подключена» устарел — hooks есть в [`bootstrap.go:67-92`](internal/httpapi/bootstrap.go)
|
||||
|
||||
### Потенциальные баги и race conditions
|
||||
|
||||
1. **`j.Meta` data race** — `-race` на `TestParallelModuleRefresh_*` + concurrent `GET /v1/jobs/{id}` polling
|
||||
2. **Memory store alias** — `deploy.Run` читает `LastAppliedRevisionID` пока worker пишет
|
||||
3. **peerLiveCache** возвращает slice без копии — [`peers_live.go:82-84`](internal/httpapi/peers_live.go)
|
||||
4. **TOCTOU idempotency** — terminal job удаляется из `byIdempo`, повторный POST создаст новый job (by design, но клиент должен знать)
|
||||
|
||||
### Error handling
|
||||
|
||||
**Хорошо:**
|
||||
- Префиксы ошибок (`httpclient:`, `birdfmt:`)
|
||||
- HTTP 5xx через `writeProblem`, без raw `err.Error()` (ERR-01)
|
||||
- `context.Context` в pipeline workers
|
||||
|
||||
**Пробелы:**
|
||||
- `runRollback` без `workContext` — не отменяется — [`worker.go:500+`](internal/jobs/worker.go)
|
||||
- Prefetch/ingest: ошибки не логируются
|
||||
- `mergeBirdPostApplyMeta` — `context.Background()` 8s, игнорирует job cancel
|
||||
|
||||
---
|
||||
|
||||
## 3. Производительность
|
||||
|
||||
### Блокирующие операции
|
||||
|
||||
| Участок | Блокировка | Риск |
|
||||
|---------|------------|------|
|
||||
| `POST .../cdn-sources/preview` | Sync CDN fetch до 45s в HTTP handler | UI timeout, worker starvation |
|
||||
| `GET /v1/peers/live` | N × agent HTTP, wg.Wait | Slow при многих speakers |
|
||||
| Module refresh job | Sequential: ingest → render revision → optional deploy | Long job chain |
|
||||
| `bird -p` / `birdc configure` | Subprocess в deploy | Disk I/O на ноде |
|
||||
|
||||
### Неэффективные алгоритмы / лишние запросы
|
||||
|
||||
- **Tenant refresh:** `aggregateTenantPrefixRowsAll` — parallel по модулям, но каждый модуль может refetch все CDN/ASN/DoH — [`aggregate.go:28+`](internal/pipeline/aggregate.go). Snapshot skip есть через `module_hash` — проверять hit rate в meta.
|
||||
- **ASN resolve:** `PolitePause()` 150ms между AS — [`asnresolve/ripestat.go`](internal/asnresolve/ripestat.go) — при 50 AS = +7.5s minimum.
|
||||
- **GetModulePrefixSnapshot** вызывается многократно в одном refresh (cdn_snapshot, collect_parallel) — potential duplicate DB reads.
|
||||
- **Auth TouchAPIKeyLastUsed:** UPDATE на каждый request (async) — load на PG при high RPS.
|
||||
|
||||
### Кэширование
|
||||
|
||||
| Кэш | TTL | Gap |
|
||||
|-----|-----|-----|
|
||||
| ASN prefix cache | 1800s (`EVOBGP_ASN_CACHE_TTL_SEC`) | OK |
|
||||
| CDN ETag in DB | Until 304/change | OK |
|
||||
| Module prefix snapshot | Content-hash based skip | OK |
|
||||
| peerLiveCache | In-memory, per-process | Не shared между API replicas; нет defensive copy |
|
||||
| Circuit breaker state | Per-process | Не shared |
|
||||
|
||||
### Конкретные улучшения
|
||||
|
||||
```go
|
||||
// 1. CDN preview — использовать upstreamHTTPDo вместо прямого Do
|
||||
resp, err := pipeline.UpstreamHTTPDo(r.Context(), s.cdnHTTP, req) // extract upstreamHTTPDo
|
||||
|
||||
// 2. Job.Meta — читать под lock или через Snapshot()
|
||||
st := j.Snapshot()
|
||||
mid, _ := st["meta"].(map[string]any)["module_id"].(string)
|
||||
|
||||
// 3. Memory store — возвращать копии (как Postgres)
|
||||
modCopy := *mod
|
||||
return &modCopy, nil
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Сетевое взаимодействие (критично)
|
||||
|
||||
### Текущее состояние
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph resilient [Resilient path]
|
||||
CDN[CDN fetch]
|
||||
RIPE[RIPEstat]
|
||||
CDN --> Breaker[Circuit breaker]
|
||||
RIPE --> Breaker
|
||||
Breaker --> Retry[DoWithRetry 3x linear 2s]
|
||||
end
|
||||
subgraph fragile [Fragile path]
|
||||
DoH[DoH resolve]
|
||||
Preview[CDN preview API]
|
||||
AgentHealth[Agent health/bird]
|
||||
DoH --> SingleDo[Single hc.Do]
|
||||
Preview --> SingleDo
|
||||
AgentHealth --> SingleDo
|
||||
end
|
||||
subgraph fallback [App-level fallback]
|
||||
Stale[Stale snapshot]
|
||||
SysDNS[System DNS]
|
||||
DoH --> SysDNS
|
||||
CDN --> Stale
|
||||
RIPE --> Stale
|
||||
end
|
||||
```
|
||||
|
||||
| Upstream | Timeout | Retry | Breaker | Stale fallback |
|
||||
|----------|---------|-------|---------|----------------|
|
||||
| CDN ingest | 45s | 3× linear | per-host | yes |
|
||||
| RIPEstat | 45s | 3× | per-host | yes + cache |
|
||||
| DoH | 10s/profile | **no** | **no** | domain snapshot |
|
||||
| CDN preview | 45s | **no** | **no** | N/A |
|
||||
| Scheduler→API | 45s | 3× | no | N/A |
|
||||
| Node dispatch | 30s | inline 3× | no | N/A |
|
||||
|
||||
### Пробелы для блокировок провайдеров
|
||||
|
||||
1. **DoH без retry** — transient timeout = fail; failover между profiles есть, но каждый profile — single shot
|
||||
2. **429/408 не ретраятся** — только `>= 500`
|
||||
3. **Нет jitter** — thundering herd при mass tenant refresh
|
||||
4. **DNS rebinding TOCTOU** — SSRF check до fetch, HTTP dial без pinned IP — [`cdn_url.go:75-115`](internal/pipeline/cdn_url.go)
|
||||
5. **Circuit breaker без half-open** — после 30s cooldown сразу full traffic — [`circuit.go:29-33`](internal/httpclient/circuit.go)
|
||||
6. **Breaker per-process** — ingest container ≠ API container
|
||||
|
||||
### Рекомендации для нестабильной сети
|
||||
|
||||
| # | Изменение | Effort | Effect |
|
||||
|---|-----------|--------|--------|
|
||||
| N1 | DoH через `DoWithRetry` + optional breaker | Low | **High** для DOMAINS modules |
|
||||
| N2 | Retry 429/503 с `Retry-After` + exponential backoff + jitter | Medium | **High** при rate limits |
|
||||
| N3 | **Partial CDN failure** — continue с stale per-source, не fail whole module | Medium | **High** |
|
||||
| N4 | Multiple DoH profiles + `failover` policy (already exists) — **документировать ops playbook** | Low | **High** (config, not code) |
|
||||
| N5 | Pinned dialer / custom `Transport.DialContext` после SSRF resolve | Medium | **Medium** (SSRF hardening) |
|
||||
| N6 | Proxy support (`HTTP_PROXY` / `EVOBGP_HTTP_PROXY`) для CDN/DoH | Medium | **High** в censored networks |
|
||||
| N7 | Unify CDN preview на `upstreamHTTPDo` | Low | **Medium** |
|
||||
|
||||
---
|
||||
|
||||
## 5. Устойчивость и надёжность
|
||||
|
||||
### Graceful degradation
|
||||
|
||||
**Работает:**
|
||||
- `EVOBGP_STALE_ON_UPSTREAM_ERROR=1` — ASN/CDN/domain stale — [`collect_stale.go`](internal/pipeline/collect_stale.go)
|
||||
- CDN 304 без local cache → forced full GET — [`cdn_snapshot.go:141-159`](internal/pipeline/cdn_snapshot.go)
|
||||
- DoH → system DNS fallback — [`doh_resolve.go:75-93`](internal/pipeline/doh_resolve.go)
|
||||
- Deploy: job succeed even if agent wake fails (drift detection)
|
||||
|
||||
**Не работает / частично:**
|
||||
- Один CDN source fail без cache → **весь module_refresh failed** — [`collect_parallel.go:221-223`](internal/pipeline/collect_parallel.go)
|
||||
- Circuit open → immediate error, stale only if prior data exists
|
||||
- API shutdown: HTTP drain 15s, **jobs не cancel/drain** — [`cmd/evobgp-api/main.go:67-72`](cmd/evobgp-api/main.go)
|
||||
|
||||
### Сценарии отказов
|
||||
|
||||
| Событие | Что произойдёт |
|
||||
|---------|----------------|
|
||||
| **Потеря CP↔PG** | Ready=false; running jobs fail; no new jobs persist audit reliably |
|
||||
| **Потеря CP↔CDN** | Stale prefixes если были; иначе job fail; breaker opens 30s |
|
||||
| **Потеря CP↔agent** | Deploy meta `dispatch_failed`; BIRD на старой ревизии; drift logs |
|
||||
| **RIPEstat rate limit** | 429 → no retry → stale or fail |
|
||||
| **Рост нагрузки** | Job queue; goroutine pile-up; PG pool exhaustion; `/metrics` shows queue depth |
|
||||
| **API restart** | In-flight jobs lost; clients poll 404 or stale terminal state |
|
||||
|
||||
---
|
||||
|
||||
## 6. Безопасность
|
||||
|
||||
| ID | Finding | Severity | Fix |
|
||||
|----|---------|----------|-----|
|
||||
| S1 | `Bearer dev` → operator при demo-seed | **high** (misconfig) | `EVOBGP_SEED_DEMO=0` — [`auth.go:66-92`](internal/httpapi/auth.go) |
|
||||
| S2 | API plain HTTP | **high** (ops) | TLS на edge (Traefik/nginx) |
|
||||
| S3 | Ephemeral bundle key без `EVOBGP_BUNDLE_SEED_HEX` | **high** (ops) | Stable seed + pubkey на нодах |
|
||||
| S4 | Compose defaults: weak PG password, `sslmode=disable` | **high** (ops) | Secrets manager, `sslmode=require` |
|
||||
| S5 | `/metrics` без auth | **medium** | Network policy / mTLS |
|
||||
| S6 | No rate limiting on auth | **medium** | Middleware limiter (e.g. per-IP) |
|
||||
| S7 | CDN SSRF DNS rebinding | **medium** | Pinned dialer after resolve |
|
||||
| S8 | `EVOBGP_CDN_ALLOW_PRIVATE=1` | **medium** | Never in prod |
|
||||
| S9 | `EVOBGP_NODE_DISPATCH_INSECURE_TLS=1` | **medium** | Valid TLS to agent |
|
||||
| S10 | Plaintext `EVOBGP_API_KEYS` in env | **medium** | DB keys via API |
|
||||
| S11 | `editor` can cancel jobs | **low** | Restrict to operator |
|
||||
| S12 | agent_secret `==` compare | **low** | `subtle.ConstantTimeCompare` |
|
||||
|
||||
**SQL injection:** не обнаружено — параметризованные запросы в [`repository/`](internal/repository/).
|
||||
|
||||
**Bundle crypto:** Ed25519 корректно; path traversal blocked в tar extract.
|
||||
|
||||
---
|
||||
|
||||
## 7. Конкретные рекомендации (prioritized backlog)
|
||||
|
||||
### High
|
||||
|
||||
| # | Описание | Как исправить |
|
||||
|---|----------|---------------|
|
||||
| H1 | DoH без retry | Обернуть `hc.Do` в `DoWithRetry(ctx, hc, req, 3)` в [`refresh.go:288,360`](internal/pipeline/refresh.go) |
|
||||
| H2 | Data race `Job.Meta` | Читать через `Snapshot()` или добавить `MetaLocked()` accessor |
|
||||
| H3 | CDN source partial failure | В `collectCDNPrefixRows`: при err без stale — log warning + skip source вместо `return nil, r.err` (config flag `EVOBGP_CDN_PARTIAL_OK=1`) |
|
||||
| H4 | Production checklist enforcement | CI/deploy validation: reject `SEED_DEMO=1`, require `BUNDLE_SEED_HEX` |
|
||||
| H5 | Job queue HA roadmap | Persist queued jobs in PG + worker claim (`SELECT FOR UPDATE SKIP LOCKED`) или NATS — ARCH-04 |
|
||||
|
||||
### Medium
|
||||
|
||||
| # | Описание | Как исправить |
|
||||
|---|----------|---------------|
|
||||
| M1 | CDN preview bypass | [`routes_crud.go:267`](internal/httpapi/routes_crud.go) → `upstreamHTTPDo` |
|
||||
| M2 | Retry 429/503 | Extend `DoWithRetry` status check + parse `Retry-After` |
|
||||
| M3 | Graceful shutdown | On SIGTERM: `Registry.RequestCancelAll()` + wait workers with timeout |
|
||||
| M4 | Auth goroutine storm | Worker pool или sync touch with debounce |
|
||||
| M5 | HTTP proxy support | Custom Transport reading `EVOBGP_HTTP_PROXY` |
|
||||
| M6 | Memory store copies | Defensive copy in Get/List (dev/test safety) |
|
||||
| M7 | Rate limiting | `golang.org/x/time/rate` on auth middleware |
|
||||
|
||||
### Low
|
||||
|
||||
| # | Описание | Как исправить |
|
||||
|---|----------|---------------|
|
||||
| L1 | Jitter in backoff | `wait + rand.Intn(wait/2)` in DoWithRetry |
|
||||
| L2 | Half-open breaker | Single probe request after cooldown |
|
||||
| L3 | Remove dead `EVOBGP_DEV_INSECURE` from compose | Docs + compose cleanup |
|
||||
| L4 | Prefetch error logging | `log.Printf` or structured log in prefetch |
|
||||
| L5 | peerLiveCache defensive copy | `append([]T(nil), views...)` on store |
|
||||
|
||||
---
|
||||
|
||||
## 8. Quick wins (максимальный эффект / минимум усилий)
|
||||
|
||||
1. **Ops (0 code):** [`docs/production-checklist.md`](docs/production-checklist.md) — `SEED_DEMO=0`, `BUNDLE_SEED_HEX`, `DB_MAX_CONNS=25`, `JOB_MAX=16`, `COLLECT_CONCURRENCY=16`, TLS edge, restrict metrics
|
||||
2. **DoH retry** — 5–10 строк в `refresh.go`, reuse existing `DoWithRetry`
|
||||
3. **CDN preview → upstreamHTTPDo** — 1 line change in handler
|
||||
4. **Job.Meta read fix** — replace 4 reads in `worker.go` with `Snapshot()` parsing
|
||||
5. **Log prefetch failures** — visibility без изменения behavior
|
||||
6. **Document DoH failover playbook** — multiple profiles (Cloudflare, Google, Quad9) + `failover` policy for censored regions
|
||||
7. **Run `go test -race ./internal/jobs/...`** in CI — catch Meta race
|
||||
8. **Prefer `evobgp-all`** over split reference for <20 tenants — eliminates Registry split bug
|
||||
|
||||
---
|
||||
|
||||
## Диаграмма: refresh под сетевым stress
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Op as Operator
|
||||
participant API as evobgp_api
|
||||
participant Job as module_refresh
|
||||
participant CDN as CDN_upstream
|
||||
participant PG as PostgreSQL
|
||||
|
||||
Op->>API: POST /modules/id/refresh
|
||||
API->>Job: Enqueue
|
||||
Job->>CDN: GET with ETag
|
||||
alt CDN timeout or 5xx
|
||||
CDN-->>Job: error after 3 retries
|
||||
Job->>PG: load prior snapshot
|
||||
alt stale exists
|
||||
Job->>PG: CreateRenderRevision stale
|
||||
Job-->>API: succeeded degraded
|
||||
else no stale
|
||||
Job-->>API: failed
|
||||
end
|
||||
else CDN 200
|
||||
CDN-->>Job: new prefixes
|
||||
Job->>PG: CreateRenderRevision
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Итоговая оценка зрелости
|
||||
|
||||
| Область | Оценка | Комментарий |
|
||||
|---------|--------|-------------|
|
||||
| Архитектура | 7/10 | Чистые слои; HA/API scaling — слабое место |
|
||||
| Сеть/resilience | 6/10 | CDN/ASN хорошо; DoH/preview — пробелы |
|
||||
| Concurrency | 6/10 | Registry продуман; Meta race, shutdown |
|
||||
| Performance | 7/10 | Parallel collect, caching; tuning needed at scale |
|
||||
| Security | 6/10 | Crypto OK; ops/config risks dominate |
|
||||
| Maintainability | 8/10 | Docs, rules, OpenAPI, tests |
|
||||
|
||||
**Вердикт:** проект **готов для 10+ клиентов в single-CP deployment** (`evobgp-all` + PostgreSQL + production checklist) при условии ops discipline. Для **multi-CP HA** и **агрессивных сетевых блокировок** — приоритет: DoH retry, partial CDN failure, distributed job queue, HTTP proxy.
|
||||
@@ -142,6 +142,8 @@ feat(web): add module create dialog on /modules
|
||||
| `.cursor/` | `chore` |
|
||||
| прочее в корне | `chore` |
|
||||
|
||||
**Запрещено:** несколько scope через запятую (`refactor(web, httpapi): …`) — semantic-release не распознает `type`, релиз не будет (см. [docs/releasing.md](../../docs/releasing.md)).
|
||||
|
||||
`type` определять по **содержимому diff**, не только по пути.
|
||||
|
||||
## Multi-change
|
||||
|
||||
@@ -303,6 +303,8 @@ jobs:
|
||||
cache-dependency-path: package-lock.json
|
||||
- name: Install release tooling
|
||||
run: npm ci
|
||||
- name: Verify releasable commit messages
|
||||
run: node scripts/commit/verify-release-commits.mjs
|
||||
- name: Semantic release
|
||||
run: npx semantic-release
|
||||
env:
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
|
||||
"evobgp/internal/birdfmt"
|
||||
"evobgp/internal/config"
|
||||
"evobgp/internal/dbcli"
|
||||
"evobgp/internal/deploy"
|
||||
"evobgp/internal/httpapi"
|
||||
"evobgp/internal/ingest"
|
||||
@@ -24,6 +25,9 @@ import (
|
||||
|
||||
// microVPS entrypoint: один процесс — HTTP API и фоновые воркеры scheduler, ingest, render, deploy (общий store и jobs.Registry).
|
||||
func main() {
|
||||
if len(os.Args) > 1 && os.Args[1] == "db" {
|
||||
os.Exit(dbcli.Run(os.Args[2:]))
|
||||
}
|
||||
cfg := config.Load()
|
||||
opts := httpapi.Options{
|
||||
APIKeys: os.Getenv("EVOBGP_API_KEYS"),
|
||||
@@ -52,6 +56,7 @@ func main() {
|
||||
go render.Run(ctx, renderDeps)
|
||||
go deploy.Run(ctx, deployDeps)
|
||||
|
||||
srv.StartBackground(ctx)
|
||||
startBirdMetricsPoller(ctx)
|
||||
|
||||
httpSrv := &http.Server{
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
|
||||
"evobgp/internal/birdfmt"
|
||||
"evobgp/internal/config"
|
||||
"evobgp/internal/dbcli"
|
||||
"evobgp/internal/httpapi"
|
||||
"evobgp/internal/observability"
|
||||
"evobgp/internal/platform"
|
||||
@@ -19,6 +20,9 @@ import (
|
||||
)
|
||||
|
||||
func main() {
|
||||
if len(os.Args) > 1 && os.Args[1] == "db" {
|
||||
os.Exit(dbcli.Run(os.Args[2:]))
|
||||
}
|
||||
cfg := config.Load()
|
||||
seedDemo := os.Getenv("EVOBGP_SEED_DEMO") != "0"
|
||||
opts := httpapi.Options{
|
||||
@@ -39,6 +43,7 @@ func main() {
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
srv.StartBackground(ctx)
|
||||
startBirdMetricsPoller(ctx)
|
||||
|
||||
httpSrv := &http.Server{
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
/** @type {import('@commitlint/types').UserConfig} */
|
||||
module.exports = {
|
||||
extends: ['@commitlint/config-conventional'],
|
||||
plugins: [
|
||||
{
|
||||
rules: {
|
||||
'scope-no-commas': ({ scope }) => {
|
||||
if (scope && scope.includes(',')) {
|
||||
return [
|
||||
false,
|
||||
'scope must not contain commas (semantic-release will not parse the commit type)'
|
||||
];
|
||||
}
|
||||
return [true];
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
rules: {
|
||||
'scope-no-commas': [2, 'always']
|
||||
}
|
||||
};
|
||||
@@ -27,9 +27,7 @@ services:
|
||||
network_mode: host
|
||||
cap_add:
|
||||
- NET_ADMIN
|
||||
sysctls:
|
||||
net.ipv4.ip_forward: "1"
|
||||
net.ipv6.conf.all.forwarding: "1"
|
||||
# sysctls нельзя с network_mode: host — включите ip_forward на VPS (см. docs/remote-speakers.md)
|
||||
volumes:
|
||||
- bird_etc:/etc/bird
|
||||
- bird_run:/run/bird
|
||||
|
||||
@@ -52,6 +52,18 @@ opkey|01ARZ3NDEKTSV4RRFFQ69G5FAV|operator,nodekey|01ARZ3NDEKTSV4RRFFQ69G5FAV|nod
|
||||
|
||||
**Запрещено** в продакшене: не оставляйте demo-seed с известным токеном `dev` на боевых данных. Переменная `EVOBGP_DEV_INSECURE` в текущей версии **не влияет** на аутентификацию (оставлена в compose для совместимости; не включайте в production — см. SEC-02 в инженерных правилах).
|
||||
|
||||
### PostgreSQL monitoring и maintenance (control plane)
|
||||
|
||||
При `EVOBGP_DATABASE_URL` (не memory backend):
|
||||
|
||||
| Операция | Минимальная роль |
|
||||
|----------|------------------|
|
||||
| `GET /v1/monitoring/postgres/*`, `GET /v1/monitoring/correlation` | viewer |
|
||||
| `POST /v1/postgres/vacuum`, `vacuum-analyze`, `analyze`, `reindex`, `cleanup` | **operator** (async job, rate limit 60s на kind) |
|
||||
| `GET /v1/postgres/maintenance/logs` | viewer |
|
||||
|
||||
Метрики **instance-level** (не per-tenant). CLI: `evobgp-api db …` / `evobgp-all db …`.
|
||||
|
||||
### Синхронные «тяжёлые» GET (control plane)
|
||||
|
||||
- `POST /v1/modules/{module_id}/cdn-sources/preview` — загрузка CDN в том же HTTP-запросе (лимит тела ~8 MiB, см. OpenAPI).
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
# Диагностика схемы PostgreSQL (EvoBGP)
|
||||
|
||||
Runbook для оценки объёма БД и узких мест **перед** и **после** миграций оптимизации схемы. Выполнять на staging или production read-only сессией.
|
||||
|
||||
## Подключение
|
||||
|
||||
```bash
|
||||
psql "$EVOBGP_DATABASE_URL"
|
||||
```
|
||||
|
||||
## HTTP API (панель / мониторинг)
|
||||
|
||||
При подключённом PostgreSQL control plane отдаёт instance-level метрики (роль **viewer+**):
|
||||
|
||||
- `GET /v1/monitoring/postgres/overview` — подключения, TPS, cache hit, размер БД
|
||||
- `GET /v1/monitoring/postgres/queries` — top queries (`pg_stat_statements`, если extension включён)
|
||||
- `GET /v1/monitoring/postgres/locks`, `/tables`, `/recommendations`
|
||||
- `GET /v1/monitoring/correlation?window=60` — корреляция refresh jobs и cache hit
|
||||
|
||||
Обслуживание (**operator**, async `202` + `job_id`): `POST /v1/postgres/vacuum`, `vacuum-analyze`, `analyze`, `reindex`, `cleanup`; журнал `GET /v1/postgres/maintenance/logs`.
|
||||
|
||||
CLI на CP: `evobgp-api db report|vacuum|analyze|cleanup` (см. `internal/dbcli`).
|
||||
|
||||
Миграция `000023` создаёт `pg_stat_statements`; для сбора статистики **обязательно** preload и перезапуск Postgres:
|
||||
|
||||
```text
|
||||
# postgresql.conf или command в compose
|
||||
shared_preload_libraries = 'pg_stat_statements'
|
||||
```
|
||||
|
||||
После изменения — restart контейнера/сервиса Postgres. Без этого API `/v1/monitoring/postgres/queries` вернёт пустой список (`statements_available: false`), без 5xx.
|
||||
|
||||
## 1. Размеры таблиц и индексов
|
||||
|
||||
```sql
|
||||
SELECT relname,
|
||||
pg_size_pretty(pg_total_relation_size(relid)) AS total,
|
||||
pg_size_pretty(pg_relation_size(relid)) AS heap,
|
||||
pg_size_pretty(pg_indexes_size(relid)) AS indexes
|
||||
FROM pg_catalog.pg_statio_user_tables
|
||||
ORDER BY pg_total_relation_size(relid) DESC;
|
||||
```
|
||||
|
||||
**Ожидание:** лидеры — `revision_materialized_prefix`, `config_revision` (TOAST от preview), JSONB-кэши.
|
||||
|
||||
## 2. Seq scan (горячие таблицы)
|
||||
|
||||
```sql
|
||||
SELECT schemaname, relname, seq_scan, seq_tup_read, idx_scan
|
||||
FROM pg_stat_user_tables
|
||||
WHERE schemaname = 'public'
|
||||
ORDER BY seq_tup_read DESC;
|
||||
```
|
||||
|
||||
Сброс статистики после деплоя: `SELECT pg_stat_reset();` (только осознанно, теряется baseline).
|
||||
|
||||
## 3. Неиспользуемые индексы
|
||||
|
||||
```sql
|
||||
SELECT indexrelname, idx_scan, pg_size_pretty(pg_relation_size(indexrelid)) AS size
|
||||
FROM pg_stat_user_indexes
|
||||
WHERE schemaname = 'public' AND idx_scan = 0
|
||||
ORDER BY pg_relation_size(indexrelid) DESC;
|
||||
```
|
||||
|
||||
## 4. Дубликаты в materialized prefixes
|
||||
|
||||
Перед UNIQUE `(revision_id, prefix, community_id, source)`:
|
||||
|
||||
```sql
|
||||
SELECT revision_id, prefix, community_id, source, COUNT(*) AS n
|
||||
FROM revision_materialized_prefix
|
||||
GROUP BY 1, 2, 3, 4
|
||||
HAVING COUNT(*) > 1
|
||||
LIMIT 20;
|
||||
```
|
||||
|
||||
## 5. Шаблон отчёта staging
|
||||
|
||||
| Метрика | До | После | Дата |
|
||||
|---------|-----|-------|------|
|
||||
| `revision_materialized_prefix` total | | | |
|
||||
| `config_revision` total | | | |
|
||||
| `module_prefix_snapshot` total | | | |
|
||||
| `asn_prefix_cache` total | | | |
|
||||
| Top seq_scan table | | | |
|
||||
| Unused indexes (count) | | | |
|
||||
|
||||
## 6. EXPLAIN для типовых запросов
|
||||
|
||||
```sql
|
||||
-- Список префиксов ревизии (keyset)
|
||||
EXPLAIN (ANALYZE, BUFFERS)
|
||||
SELECT prefix::text, community_id::text, source
|
||||
FROM revision_materialized_prefix
|
||||
WHERE revision_id = '<revision-uuid>'::uuid
|
||||
ORDER BY id
|
||||
LIMIT 51;
|
||||
|
||||
-- Diff added (anti-join)
|
||||
EXPLAIN (ANALYZE, BUFFERS)
|
||||
SELECT b.prefix::text
|
||||
FROM revision_materialized_prefix b
|
||||
LEFT JOIN revision_materialized_prefix a
|
||||
ON a.revision_id = '<rev-a>'::uuid AND a.prefix = b.prefix
|
||||
WHERE b.revision_id = '<rev-b>'::uuid
|
||||
AND a.prefix IS NULL
|
||||
ORDER BY b.prefix
|
||||
LIMIT 5001;
|
||||
```
|
||||
|
||||
Цель: Index Scan / Bitmap Index Scan по `(revision_id, …)`, без Seq Scan на больших таблицах.
|
||||
@@ -49,6 +49,8 @@ tags:
|
||||
description: Управление API-ключами tenant (operator). Секрет возвращается только при создании и ротации.
|
||||
- name: Auth
|
||||
description: Сессия текущего API-ключа (tenant и роль).
|
||||
- name: Monitoring
|
||||
description: Наблюдаемость PostgreSQL и корреляция (instance-level, viewer+). Maintenance — operator.
|
||||
|
||||
security:
|
||||
- bearerAuth: []
|
||||
@@ -755,6 +757,42 @@ components:
|
||||
bgp_speaker_id:
|
||||
type: ["string", "null"]
|
||||
description: "`null` - политика для всех спикеров."
|
||||
connected_speaker_id:
|
||||
type: ["string", "null"]
|
||||
description: >
|
||||
Live (GET /v1/peers?live=1): спикер, на котором сессия Established; опрос CP birdc + agent /v1/agent/bird/protocols.
|
||||
connected_speaker_label:
|
||||
type: string
|
||||
description: Человекочитаемая метка ноды из live-опроса.
|
||||
session_on_speakers:
|
||||
type: array
|
||||
description: Состояние протокола пира на каждой опрошенной ноде.
|
||||
items:
|
||||
type: object
|
||||
properties:
|
||||
speaker_id:
|
||||
type: string
|
||||
label:
|
||||
type: string
|
||||
state:
|
||||
type: string
|
||||
established_on_speakers:
|
||||
type: array
|
||||
description: Ноды, где сессия в состоянии Established (один пир может быть на нескольких).
|
||||
items:
|
||||
type: object
|
||||
properties:
|
||||
speaker_id:
|
||||
type: string
|
||||
label:
|
||||
type: string
|
||||
state:
|
||||
type: string
|
||||
session_mismatch:
|
||||
type: boolean
|
||||
description: >
|
||||
true если bgp_speaker_id задан, но на этой ноде нет Established
|
||||
(сессия может быть на других нодах — это не ошибка для tenant-wide пиров).
|
||||
policies_json:
|
||||
type: string
|
||||
description: >
|
||||
@@ -809,6 +847,147 @@ components:
|
||||
description: >
|
||||
Расширяемый объект. Ключи agent_domain, agent_secret (только при создании),
|
||||
agent_port, node_ipv4, bird_bgp_source_ipv4, bird_bgp_source_ipv6.
|
||||
live:
|
||||
$ref: "#/components/schemas/SpeakerLiveStatus"
|
||||
description: >
|
||||
При GET /v1/speakers?live=1 — runtime-статус agent и BGP-опроса на ноде.
|
||||
additionalProperties: true
|
||||
|
||||
SpeakerLiveStatus:
|
||||
type: object
|
||||
description: Live runtime snapshot for one speaker (GET /v1/speakers?live=1).
|
||||
properties:
|
||||
label:
|
||||
type: string
|
||||
description: Человекочитаемая метка ноды (agent domain или CP master).
|
||||
agent_ok:
|
||||
type: boolean
|
||||
description: true если agent /v1/agent/health успешен (master — local birdc poll).
|
||||
agent_error:
|
||||
type: string
|
||||
agent_last_sync_at:
|
||||
type: string
|
||||
format: date-time
|
||||
agent_last_applied_revision_id:
|
||||
type: string
|
||||
bgp_poll_ok:
|
||||
type: boolean
|
||||
description: true если birdc (CP) или GET /v1/agent/bird/protocols (replica) успешен.
|
||||
bgp_poll_error:
|
||||
type: string
|
||||
bgp_sessions_total:
|
||||
type: integer
|
||||
bgp_established:
|
||||
type: integer
|
||||
sessions:
|
||||
type: array
|
||||
items:
|
||||
$ref: "#/components/schemas/BgpSessionLive"
|
||||
additionalProperties: true
|
||||
|
||||
BgpSessionLive:
|
||||
type: object
|
||||
properties:
|
||||
name:
|
||||
type: string
|
||||
neighbor:
|
||||
type: string
|
||||
state:
|
||||
type: string
|
||||
additionalProperties: true
|
||||
|
||||
LiveSpeakerPoll:
|
||||
type: object
|
||||
description: Метаданные опроса одной ноды в GET /v1/peers?live=1.
|
||||
properties:
|
||||
speaker_id:
|
||||
type: string
|
||||
label:
|
||||
type: string
|
||||
ok:
|
||||
type: boolean
|
||||
session_count:
|
||||
type: integer
|
||||
poll_error:
|
||||
type: string
|
||||
additionalProperties: true
|
||||
|
||||
PostgresOverview:
|
||||
type: object
|
||||
description: Instance-level PostgreSQL snapshot (GET /v1/monitoring/postgres/overview).
|
||||
additionalProperties: true
|
||||
|
||||
PostgresQueriesResponse:
|
||||
type: object
|
||||
properties:
|
||||
collected_at:
|
||||
type: string
|
||||
format: date-time
|
||||
source:
|
||||
type: string
|
||||
enum: [live, snapshot]
|
||||
items:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
additionalProperties: true
|
||||
|
||||
PostgresRecommendations:
|
||||
type: object
|
||||
properties:
|
||||
collected_at:
|
||||
type: string
|
||||
format: date-time
|
||||
items:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
properties:
|
||||
severity:
|
||||
type: string
|
||||
code:
|
||||
type: string
|
||||
title:
|
||||
type: string
|
||||
detail:
|
||||
type: string
|
||||
refs:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
|
||||
PostgresMaintenanceBody:
|
||||
type: object
|
||||
properties:
|
||||
table:
|
||||
type: string
|
||||
dry_run:
|
||||
type: boolean
|
||||
default: false
|
||||
policy:
|
||||
type: string
|
||||
description: job_audit_retention | asn_cache_retention
|
||||
limit:
|
||||
type: integer
|
||||
|
||||
BirdLocalStatus:
|
||||
type: object
|
||||
description: Статус локального BIRD на хосте API (GET /v1/bird/status).
|
||||
properties:
|
||||
birdc_configured:
|
||||
type: boolean
|
||||
message:
|
||||
type: string
|
||||
error:
|
||||
type: string
|
||||
protocols_excerpt:
|
||||
type: string
|
||||
bgp_sessions_total:
|
||||
type: integer
|
||||
bgp_established:
|
||||
type: integer
|
||||
healthy:
|
||||
type: ["boolean", "null"]
|
||||
additionalProperties: true
|
||||
|
||||
BundleSigningPublicKey:
|
||||
@@ -2156,6 +2335,13 @@ paths:
|
||||
- $ref: "#/components/parameters/Cursor"
|
||||
- $ref: "#/components/parameters/Limit"
|
||||
- $ref: "#/components/parameters/SpeakerFilter"
|
||||
- name: live
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
enum: ["1"]
|
||||
description: >
|
||||
Опрос birdc на CP и GET /v1/agent/bird/protocols на репликах; обогащает session_state и connected_speaker_*.
|
||||
responses:
|
||||
"200":
|
||||
description: Успешно.
|
||||
@@ -2173,6 +2359,12 @@ paths:
|
||||
type: ["string", "null"]
|
||||
has_more:
|
||||
type: boolean
|
||||
live_speaker_poll:
|
||||
type: array
|
||||
description: >
|
||||
При live=1 — результат опроса каждой ноды (CP birdc + agent protocols).
|
||||
items:
|
||||
$ref: "#/components/schemas/LiveSpeakerPoll"
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
post:
|
||||
@@ -2293,6 +2485,14 @@ paths:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
- $ref: "#/components/parameters/Cursor"
|
||||
- $ref: "#/components/parameters/Limit"
|
||||
- name: live
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
enum: ["1"]
|
||||
description: >
|
||||
Live-опрос agent /v1/agent/health и BGP protocols на репликах; CP — local birdc.
|
||||
Обогащает каждый item полем `live`.
|
||||
responses:
|
||||
"200":
|
||||
description: Успешно.
|
||||
@@ -2627,6 +2827,26 @@ paths:
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/bird/status:
|
||||
get:
|
||||
tags: [Deploy]
|
||||
summary: Статус локального BIRD на хосте API
|
||||
description: >
|
||||
Опрос birdc через EVOBGP_BIRDC_SOCKET на процессе API (обычно CP master).
|
||||
На репликах без birdc на CP — birdc_configured=false.
|
||||
operationId: getBirdStatus
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
responses:
|
||||
"200":
|
||||
description: Успешно.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/BirdLocalStatus"
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/bird/reload:
|
||||
post:
|
||||
tags: [Deploy]
|
||||
@@ -2975,6 +3195,278 @@ paths:
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/monitoring/postgres/overview:
|
||||
get:
|
||||
tags: [Monitoring]
|
||||
summary: PostgreSQL overview (instance-level)
|
||||
operationId: getPostgresOverview
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
responses:
|
||||
"200":
|
||||
description: Успешно.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/PostgresOverview"
|
||||
"503":
|
||||
description: PostgreSQL backend не подключён.
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/monitoring/postgres/queries:
|
||||
get:
|
||||
tags: [Monitoring]
|
||||
summary: Top queries (pg_stat_statements or snapshot)
|
||||
operationId: getPostgresQueries
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
- $ref: "#/components/parameters/Limit"
|
||||
responses:
|
||||
"200":
|
||||
description: Успешно.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/PostgresQueriesResponse"
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/monitoring/postgres/locks:
|
||||
get:
|
||||
tags: [Monitoring]
|
||||
summary: Active locks
|
||||
operationId: getPostgresLocks
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
responses:
|
||||
"200":
|
||||
description: Успешно.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
properties:
|
||||
items:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
additionalProperties: true
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/monitoring/postgres/tables:
|
||||
get:
|
||||
tags: [Monitoring]
|
||||
summary: Table sizes and scan stats
|
||||
operationId: getPostgresTables
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
- $ref: "#/components/parameters/Limit"
|
||||
responses:
|
||||
"200":
|
||||
description: Успешно.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
properties:
|
||||
items:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
additionalProperties: true
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/monitoring/postgres/recommendations:
|
||||
get:
|
||||
tags: [Monitoring]
|
||||
summary: Heuristic optimization recommendations
|
||||
operationId: getPostgresRecommendations
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
responses:
|
||||
"200":
|
||||
description: Успешно.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/PostgresRecommendations"
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/monitoring/correlation:
|
||||
get:
|
||||
tags: [Monitoring]
|
||||
summary: Timeline correlation (jobs vs cache hit)
|
||||
operationId: getMonitoringCorrelation
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
- name: window
|
||||
in: query
|
||||
schema:
|
||||
type: integer
|
||||
default: 60
|
||||
description: Window in minutes (max 1440).
|
||||
responses:
|
||||
"200":
|
||||
description: Успешно.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
additionalProperties: true
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/postgres/vacuum:
|
||||
post:
|
||||
tags: [Monitoring]
|
||||
summary: VACUUM (async job, operator)
|
||||
operationId: postPostgresVacuum
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
- $ref: "#/components/parameters/IdempotencyKey"
|
||||
requestBody:
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/PostgresMaintenanceBody"
|
||||
responses:
|
||||
"202":
|
||||
description: Задача поставлена.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/AsyncJobAccepted"
|
||||
"403":
|
||||
$ref: "#/components/responses/Forbidden"
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/postgres/vacuum-analyze:
|
||||
post:
|
||||
tags: [Monitoring]
|
||||
summary: VACUUM ANALYZE (async job, operator)
|
||||
operationId: postPostgresVacuumAnalyze
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
- $ref: "#/components/parameters/IdempotencyKey"
|
||||
requestBody:
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/PostgresMaintenanceBody"
|
||||
responses:
|
||||
"202":
|
||||
description: Задача поставлена.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/AsyncJobAccepted"
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/postgres/analyze:
|
||||
post:
|
||||
tags: [Monitoring]
|
||||
summary: ANALYZE (async job, operator)
|
||||
operationId: postPostgresAnalyze
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
- $ref: "#/components/parameters/IdempotencyKey"
|
||||
requestBody:
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/PostgresMaintenanceBody"
|
||||
responses:
|
||||
"202":
|
||||
description: Задача поставлена.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/AsyncJobAccepted"
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/postgres/reindex:
|
||||
post:
|
||||
tags: [Monitoring]
|
||||
summary: REINDEX TABLE (async job, operator)
|
||||
operationId: postPostgresReindex
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
- $ref: "#/components/parameters/IdempotencyKey"
|
||||
requestBody:
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/PostgresMaintenanceBody"
|
||||
responses:
|
||||
"202":
|
||||
description: Задача поставлена.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/AsyncJobAccepted"
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/postgres/cleanup:
|
||||
post:
|
||||
tags: [Monitoring]
|
||||
summary: Retention cleanup (async job, operator)
|
||||
operationId: postPostgresCleanup
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
- $ref: "#/components/parameters/IdempotencyKey"
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/PostgresMaintenanceBody"
|
||||
responses:
|
||||
"202":
|
||||
description: Задача поставлена.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/AsyncJobAccepted"
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/postgres/maintenance/logs:
|
||||
get:
|
||||
tags: [Monitoring]
|
||||
summary: Maintenance audit log
|
||||
operationId: listPostgresMaintenanceLogs
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/TenantId"
|
||||
- $ref: "#/components/parameters/Cursor"
|
||||
- $ref: "#/components/parameters/Limit"
|
||||
responses:
|
||||
"200":
|
||||
description: Успешно.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
properties:
|
||||
items:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
additionalProperties: true
|
||||
next_cursor:
|
||||
type: string
|
||||
has_more:
|
||||
type: boolean
|
||||
default:
|
||||
$ref: "#/components/responses/DefaultProblem"
|
||||
|
||||
/v1/settings:
|
||||
get:
|
||||
tags: [Settings]
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
# Production checklist
|
||||
|
||||
Краткий чеклист перед выводом EvoBGP в production (10+ клиентов, нестабильная сеть).
|
||||
|
||||
## Обязательно
|
||||
|
||||
- `EVOBGP_SEED_DEMO=0` — отключить demo-tenant и токен `Bearer dev`.
|
||||
- `EVOBGP_DEV_INSECURE` не задавать или `0` — не использовать lab-флаги в prod.
|
||||
- `EVOBGP_BUNDLE_SEED_HEX` — задать стабильный hex-ключ подписи бандлов; сохранить pubkey для нод.
|
||||
- PostgreSQL с TLS (`sslmode` не `disable`) при доступе вне private network.
|
||||
- `EVOBGP_CORS_ORIGINS` — явный whitelist origin веб-панели.
|
||||
- `EVOBGP_STALE_ON_UPSTREAM_ERROR=1` (по умолчанию) — stale snapshot при сбоях CDN/ASN/DoH.
|
||||
|
||||
## Рекомендуется
|
||||
|
||||
- `EVOBGP_JOB_MAX_CONCURRENT=16`, `EVOBGP_DB_MAX_CONNS=25`, `EVOBGP_COLLECT_CONCURRENCY=16` при росте tenants.
|
||||
- `EVOBGP_NODE_DISPATCH_INSECURE_TLS=0` — только валидный TLS к agent.
|
||||
- Ограничить `/metrics` сетевой политикой или reverse proxy.
|
||||
- Профиль `evobgp-all` или HA API + персистентная `job_audit` (PostgreSQL).
|
||||
- Мониторинг drift: `evobgp-deploy`, `last_applied_revision_id` vs published.
|
||||
|
||||
## Не использовать в prod
|
||||
|
||||
- `EVOBGP_CDN_ALLOW_PRIVATE=1` — только тесты/lab.
|
||||
- Plaintext `EVOBGP_API_KEYS` без ротации (break-glass — временно).
|
||||
- Ручное редактирование `evobgp_*.conf` на нодах без ревизии.
|
||||
@@ -11,6 +11,8 @@ EvoBGP использует [Conventional Commits](https://www.conventionalcommi
|
||||
| `feat!`, `fix!` или `BREAKING CHANGE:` в теле | major (1.0.0 → 2.0.0) |
|
||||
| `docs`, `chore`, `test` | без релиза |
|
||||
|
||||
**Scope:** один идентификатор **без запятых** (`web`, `httpapi`, `api`). Заголовок `refactor(a, b): …` **не парсится** semantic-release → релиз не создаётся (commitlint на PR это тоже отклонит). Подробнее — раздел «Scope и semantic-release» ниже.
|
||||
|
||||
`refactor` — patch без новых функций: перестройка кода/UI при том же поведении для пользователя. По semver на одном уровне с `fix`, но семантически «мельче» `feat` (не minor).
|
||||
|
||||
Отдельного суффикса `1.x.y.fix` в semver нет: «fix» в Conventional Commits означает **patch** (третья цифра). Для починки пайплайна без смены продукта — `fix(ci):` или `ci:` (оба дают patch после настройки `.releaserc.json`).
|
||||
@@ -62,6 +64,21 @@ API: `GET /version`, `GET /v1/version` — поля `version`, `git_sha`, `build
|
||||
|
||||
Web UI показывает версию из API (footer sidebar, страница «Мониторинг»).
|
||||
|
||||
## Scope и semantic-release
|
||||
|
||||
Парсер [conventional-commits-parser](https://github.com/conventional-changelog/conventional-changelog/tree/master/packages/conventional-commits-parser) (его использует semantic-release) **не понимает запятые в scope**:
|
||||
|
||||
| Заголовок | Парсится | Релиз |
|
||||
|-----------|----------|-------|
|
||||
| `refactor(web): fix layout` | да, `refactor` | patch |
|
||||
| `refactor(NetworkOverviewTab, NetworkSpeakersCard): fix layout` | **нет**, `type: null` | **нет** |
|
||||
|
||||
Правило: **один scope** из таблицы в [.cursor/rules/conventional-commits.mdc](../.cursor/rules/conventional-commits.mdc) (`web`, `httpapi`, `api`, …).
|
||||
|
||||
На push в `main` job **release** запускает `scripts/commit/verify-release-commits.mjs` — в логе будут предупреждения о непарсящихся коммитах.
|
||||
|
||||
Если релиз «не создался», а CI зелёный: смотрите лог release — часто `No releasable commits`. Исправление: новый коммит с корректным заголовком (например `refactor(web): …`).
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
Release notes — в Gitea Release; файл `CHANGELOG.md` генерируется в CI и прикрепляется как asset, **не** попадает в git history.
|
||||
|
||||
+15
-1
@@ -47,10 +47,22 @@ Runbook для реплик **bird2 + evobgp-agent** на отдельных VPS
|
||||
|
||||
| Порт | Кто | Зачем |
|
||||
|------|-----|-------|
|
||||
| **443** | IP CP (`PANEL_IP_WHITELIST`) | HTTPS dispatch + health |
|
||||
| **443** | IP CP (`PANEL_IP_WHITELIST`) | HTTPS dispatch, health, **`GET /v1/agent/bird/protocols`** (live peer sessions) |
|
||||
| **179** | BGP peers | Data plane |
|
||||
| **80** | ACME | Traefik → 443 |
|
||||
|
||||
## Подготовка VPS (перед `docker compose up`)
|
||||
|
||||
`bird2` — **`network_mode: host`**. Docker **не может** задать `net.ipv4.ip_forward` в таком контейнере; включите на **хосте**:
|
||||
|
||||
```bash
|
||||
sysctl -w net.ipv4.ip_forward=1
|
||||
sysctl -w net.ipv6.conf.all.forwarding=1
|
||||
echo 'net.ipv4.ip_forward=1' | tee /etc/sysctl.d/99-evobgp-bird.conf
|
||||
echo 'net.ipv6.conf.all.forwarding=1' >> /etc/sysctl.d/99-evobgp-bird.conf
|
||||
sysctl --system
|
||||
```
|
||||
|
||||
## Безопасность (три участка)
|
||||
|
||||
1. **CP → реплика:** HTTPS (LE) + Traefik ipallowlist + `agent_secret`.
|
||||
@@ -82,6 +94,8 @@ Tenant `/v1/settings` (`bird_bgp_source_ipv4`) — fallback для master / ес
|
||||
|
||||
| Симптом | Проверка |
|
||||
|---------|----------|
|
||||
| `sysctl net.ipv4.ip_forward not allowed in host network` | Уберите sysctls из compose (уже так в main); включите ip_forward на VPS (см. выше) |
|
||||
| `no service selected` | `--profile production` или `COMPOSE_PROFILES=production` |
|
||||
| Offline в UI | `GET https://AGENT_DOMAIN/v1/agent/health` с CP; LE cert; whitelist |
|
||||
| dispatch error | CP logs job meta; firewall 443; `agent_secret` |
|
||||
| verify-bundle fail | pubkey совпадает с CP seed; пересоберите pubkey после смены seed |
|
||||
|
||||
@@ -10,9 +10,12 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"evobgp/internal/birdfmt"
|
||||
"evobgp/internal/nodecli"
|
||||
)
|
||||
|
||||
const upstreamErrorDetail = "upstream request failed"
|
||||
|
||||
// Config holds evobgp-agent serve settings.
|
||||
type Config struct {
|
||||
Listen string
|
||||
@@ -41,6 +44,7 @@ type Server struct {
|
||||
func New(cfg Config) *Server {
|
||||
s := &Server{cfg: cfg, mux: http.NewServeMux()}
|
||||
s.mux.HandleFunc("GET /v1/agent/health", s.handleHealth)
|
||||
s.mux.HandleFunc("GET /v1/agent/bird/protocols", s.handleBirdProtocols)
|
||||
s.mux.HandleFunc("POST /v1/agent/sync", s.handleSync)
|
||||
return s
|
||||
}
|
||||
@@ -68,6 +72,30 @@ func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusOK, body)
|
||||
}
|
||||
|
||||
func (s *Server) handleBirdProtocols(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.authorize(r) {
|
||||
writeProblem(w, http.StatusUnauthorized, "missing or invalid Authorization")
|
||||
return
|
||||
}
|
||||
sock := strings.TrimSpace(s.cfg.Socket)
|
||||
if sock == "" {
|
||||
writeProblem(w, http.StatusServiceUnavailable, "EVOBGP_BIRDC_SOCKET not configured")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second)
|
||||
defer cancel()
|
||||
out, err := birdfmt.ShowProtocols(ctx, sock, strings.TrimSpace(s.cfg.BirdcBin))
|
||||
if err != nil {
|
||||
log.Printf("agentserver: bird protocols: %v", err)
|
||||
writeProblem(w, http.StatusBadGateway, upstreamErrorDetail)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{
|
||||
"ok": true,
|
||||
"sessions": birdfmt.ParseBGPSessions(out),
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) handleSync(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.authorize(r) {
|
||||
writeProblem(w, http.StatusUnauthorized, "missing or invalid Authorization")
|
||||
@@ -100,7 +128,7 @@ func (s *Server) handleSync(w http.ResponseWriter, r *http.Request) {
|
||||
})
|
||||
if err != nil {
|
||||
log.Printf("agentserver: sync: %v", err)
|
||||
writeProblem(w, http.StatusBadGateway, err.Error())
|
||||
writeProblem(w, http.StatusBadGateway, upstreamErrorDetail)
|
||||
return
|
||||
}
|
||||
if s.cfg.OnSyncSuccess != nil {
|
||||
|
||||
@@ -13,6 +13,8 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"evobgp/internal/httpclient"
|
||||
)
|
||||
|
||||
// DefaultRIPEStatURL is the RIPEstat announced-prefixes data call (no API key).
|
||||
@@ -24,7 +26,7 @@ const DefaultASOverviewURL = "https://stat.ripe.net/data/as-overview/data.json"
|
||||
// AnnouncedPrefixes returns currently announced IPv4/IPv6 prefixes for the ASN (best-effort via RIPEstat).
|
||||
func AnnouncedPrefixes(ctx context.Context, hc *http.Client, asn int64) ([]netip.Prefix, error) {
|
||||
if hc == nil {
|
||||
hc = http.DefaultClient
|
||||
hc = httpclient.New(httpclient.DefaultTimeout)
|
||||
}
|
||||
base := strings.TrimSpace(os.Getenv("EVOBGP_RIPESTAT_ANNOUNCED_PREFIXES_URL"))
|
||||
if base == "" {
|
||||
@@ -38,7 +40,7 @@ func AnnouncedPrefixes(ctx context.Context, hc *http.Client, asn int64) ([]netip
|
||||
req.Header.Set("Accept", "application/json")
|
||||
req.Header.Set("User-Agent", "evobgp-asnresolve/1.0")
|
||||
|
||||
resp, err := hc.Do(req)
|
||||
resp, err := httpclient.DoWithBreaker(ctx, hc, req, 3)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("ripestat fetch AS%d: %w", asn, err)
|
||||
}
|
||||
@@ -86,7 +88,7 @@ func AnnouncedPrefixes(ctx context.Context, hc *http.Client, asn int64) ([]netip
|
||||
// ASHolderName returns the holder / organization label for the ASN from RIPEstat as-overview (best-effort).
|
||||
func ASHolderName(ctx context.Context, hc *http.Client, asn int64) (string, error) {
|
||||
if hc == nil {
|
||||
hc = http.DefaultClient
|
||||
hc = httpclient.New(httpclient.DefaultTimeout)
|
||||
}
|
||||
base := strings.TrimSpace(os.Getenv("EVOBGP_RIPESTAT_AS_OVERVIEW_URL"))
|
||||
if base == "" {
|
||||
@@ -100,7 +102,7 @@ func ASHolderName(ctx context.Context, hc *http.Client, asn int64) (string, erro
|
||||
req.Header.Set("Accept", "application/json")
|
||||
req.Header.Set("User-Agent", "evobgp-asnresolve/1.0")
|
||||
|
||||
resp, err := hc.Do(req)
|
||||
resp, err := httpclient.DoWithBreaker(ctx, hc, req, 3)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("ripestat as-overview AS%d: %w", asn, err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
package birdfmt
|
||||
|
||||
import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
// BGPSession is one BGP protocol block from `birdc show protocols all`.
|
||||
type BGPSession struct {
|
||||
Name string `json:"name"`
|
||||
Neighbor string `json:"neighbor,omitempty"`
|
||||
State string `json:"state"`
|
||||
}
|
||||
|
||||
// ParseBGPSessions extracts BGP protocol name, state, and neighbor (if present) from birdc output.
|
||||
func ParseBGPSessions(output string) []BGPSession {
|
||||
var out []BGPSession
|
||||
var cur *BGPSession
|
||||
for _, raw := range strings.Split(output, "\n") {
|
||||
line := strings.TrimRight(raw, "\r")
|
||||
trim := strings.TrimSpace(line)
|
||||
if trim == "" {
|
||||
cur = nil
|
||||
continue
|
||||
}
|
||||
low := strings.ToLower(trim)
|
||||
if strings.HasPrefix(low, "bird ") || strings.HasPrefix(low, "name ") || strings.HasPrefix(low, "table ") {
|
||||
continue
|
||||
}
|
||||
if !strings.HasPrefix(line, " ") && !strings.HasPrefix(line, "\t") {
|
||||
if isBGPProtocolSummaryRow(trim) {
|
||||
fields := strings.Fields(trim)
|
||||
state := extractBGPSessionStateLine(trim)
|
||||
if state == "" && len(fields) >= 4 {
|
||||
state = fields[3]
|
||||
}
|
||||
out = append(out, BGPSession{Name: fields[0], State: state})
|
||||
cur = &out[len(out)-1]
|
||||
} else {
|
||||
cur = nil
|
||||
}
|
||||
continue
|
||||
}
|
||||
if cur == nil {
|
||||
continue
|
||||
}
|
||||
for _, prefix := range []string{"Neighbor address:", "Neighbor Address:", "Neighbor:"} {
|
||||
if idx := strings.Index(trim, prefix); idx >= 0 {
|
||||
cur.Neighbor = strings.TrimSpace(trim[idx+len(prefix):])
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
package birdfmt
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestParseBGPSessions_neighborAndState(t *testing.T) {
|
||||
sample := `
|
||||
BIRD 2.14 ready.
|
||||
Name Proto Table State Since Info
|
||||
device1 Device --- up 10:00:00
|
||||
evobgp_p_abc123 BGP master4 up 10:00:05 Established
|
||||
Neighbor address: 198.51.100.2
|
||||
Neighbor AS: 65001
|
||||
evobgp_p_def456 BGP master4 up 10:00:06 Active
|
||||
Neighbor address: 2001:db8::2
|
||||
`
|
||||
sessions := ParseBGPSessions(sample)
|
||||
if len(sessions) != 2 {
|
||||
t.Fatalf("got %d sessions want 2", len(sessions))
|
||||
}
|
||||
if sessions[0].Name != "evobgp_p_abc123" || sessions[0].State != "Established" || sessions[0].Neighbor != "198.51.100.2" {
|
||||
t.Fatalf("session0: %+v", sessions[0])
|
||||
}
|
||||
if sessions[1].Neighbor != "2001:db8::2" || sessions[1].State != "Active" {
|
||||
t.Fatalf("session1: %+v", sessions[1])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package birdfmt
|
||||
|
||||
import "strings"
|
||||
|
||||
// PeerProtocolName returns the BIRD protocol name for a control-plane peer UUID.
|
||||
// Must stay in sync with pipeline peer rendering.
|
||||
func PeerProtocolName(peerID string) string {
|
||||
s := strings.ReplaceAll(strings.TrimSpace(peerID), "-", "")
|
||||
if len(s) > 16 {
|
||||
s = s[:16]
|
||||
}
|
||||
if s == "" {
|
||||
s = "x"
|
||||
}
|
||||
return "evobgp_p_" + s
|
||||
}
|
||||
@@ -0,0 +1,209 @@
|
||||
// Package dbcli implements control-plane PostgreSQL maintenance CLI (HTTP or local DSN).
|
||||
package dbcli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"evobgp/internal/db"
|
||||
"evobgp/internal/httpclient"
|
||||
"evobgp/internal/pgmonitor"
|
||||
)
|
||||
|
||||
// Run executes db subcommands; args exclude program name and "db".
|
||||
func Run(args []string) int {
|
||||
if len(args) == 0 {
|
||||
printUsage()
|
||||
return 2
|
||||
}
|
||||
switch args[0] {
|
||||
case "report":
|
||||
return cmdReport(args[1:])
|
||||
case "vacuum":
|
||||
return cmdMaint(args[1:], "vacuum", "/v1/postgres/vacuum")
|
||||
case "analyze":
|
||||
return cmdMaint(args[1:], "analyze", "/v1/postgres/analyze")
|
||||
case "cleanup":
|
||||
return cmdCleanup(args[1:])
|
||||
default:
|
||||
fmt.Fprintf(os.Stderr, "dbcli: unknown command %q\n", args[0])
|
||||
printUsage()
|
||||
return 2
|
||||
}
|
||||
}
|
||||
|
||||
func printUsage() {
|
||||
fmt.Fprintln(os.Stderr, `usage:
|
||||
evobgp-api db report [--api-url URL] [--token TOKEN] [--format json]
|
||||
evobgp-api db vacuum [--table NAME] [--dry-run] [--api-url URL] [--token TOKEN]
|
||||
evobgp-api db analyze [--table NAME] [--dry-run] [--api-url URL] [--token TOKEN]
|
||||
evobgp-api db cleanup --policy NAME [--dry-run] [--limit N] [--api-url URL] [--token TOKEN]
|
||||
Local break-glass: set EVOBGP_DATABASE_URL (report only uses direct SQL).`)
|
||||
}
|
||||
|
||||
func cmdReport(args []string) int {
|
||||
fs := flag.NewFlagSet("report", flag.ExitOnError)
|
||||
apiURL := fs.String("api-url", "", "control plane base URL")
|
||||
token := fs.String("token", "", "Bearer token (operator)")
|
||||
format := fs.String("format", "json", "output format (json)")
|
||||
_ = fs.Parse(args)
|
||||
|
||||
if dsn := strings.TrimSpace(os.Getenv("EVOBGP_DATABASE_URL")); dsn != "" && *apiURL == "" {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
pool, err := db.OpenPostgresPool(ctx, dsn)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
return 1
|
||||
}
|
||||
defer pool.Close()
|
||||
svc := pgmonitor.NewService(pool)
|
||||
ov, err := svc.Overview(ctx)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
return 1
|
||||
}
|
||||
return writeJSONStdout(ov, *format)
|
||||
}
|
||||
if *apiURL == "" || *token == "" {
|
||||
fmt.Fprintln(os.Stderr, "report: --api-url and --token required without EVOBGP_DATABASE_URL")
|
||||
return 2
|
||||
}
|
||||
body, err := apiGET(*apiURL, *token, "/v1/monitoring/postgres/overview")
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
return 1
|
||||
}
|
||||
var pretty any
|
||||
if err := json.Unmarshal(body, &pretty); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
return 1
|
||||
}
|
||||
return writeJSONStdout(pretty, *format)
|
||||
}
|
||||
|
||||
func cmdMaint(args []string, _ string, path string) int {
|
||||
fs := flag.NewFlagSet("maint", flag.ExitOnError)
|
||||
table := fs.String("table", "", "table name")
|
||||
dryRun := fs.Bool("dry-run", false, "dry run only")
|
||||
apiURL := fs.String("api-url", "", "control plane base URL")
|
||||
token := fs.String("token", "", "Bearer token (operator)")
|
||||
_ = fs.Parse(args)
|
||||
if *apiURL == "" || *token == "" {
|
||||
fmt.Fprintln(os.Stderr, "maintenance: --api-url and --token are required")
|
||||
return 2
|
||||
}
|
||||
payload := map[string]any{"dry_run": *dryRun}
|
||||
if *table != "" {
|
||||
payload["table"] = *table
|
||||
}
|
||||
body, err := apiPOST(*apiURL, *token, path, payload)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
return 1
|
||||
}
|
||||
return writeRawJSON(body)
|
||||
}
|
||||
|
||||
func cmdCleanup(args []string) int {
|
||||
fs := flag.NewFlagSet("cleanup", flag.ExitOnError)
|
||||
policy := fs.String("policy", "", "cleanup policy name")
|
||||
dryRun := fs.Bool("dry-run", true, "dry run")
|
||||
limit := fs.Int("limit", 10000, "max rows")
|
||||
apiURL := fs.String("api-url", "", "control plane base URL")
|
||||
token := fs.String("token", "", "Bearer token (operator)")
|
||||
_ = fs.Parse(args)
|
||||
if *policy == "" {
|
||||
fmt.Fprintln(os.Stderr, "cleanup: --policy is required")
|
||||
return 2
|
||||
}
|
||||
if *apiURL == "" || *token == "" {
|
||||
fmt.Fprintln(os.Stderr, "cleanup: --api-url and --token are required")
|
||||
return 2
|
||||
}
|
||||
payload := map[string]any{"policy": *policy, "dry_run": *dryRun, "limit": *limit}
|
||||
body, err := apiPOST(*apiURL, *token, "/v1/postgres/cleanup", payload)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
return 1
|
||||
}
|
||||
return writeRawJSON(body)
|
||||
}
|
||||
|
||||
func apiGET(base, token, path string) ([]byte, error) {
|
||||
u := strings.TrimRight(base, "/") + path
|
||||
req, err := http.NewRequest(http.MethodGet, u, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
resp, err := httpclient.DoWithRetry(ctx, httpclient.New(60*time.Second), req, 3)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode >= 300 {
|
||||
return nil, fmt.Errorf("dbcli: GET %s: %s: %s", path, resp.Status, strings.TrimSpace(string(b)))
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
|
||||
func apiPOST(base, token, path string, payload map[string]any) ([]byte, error) {
|
||||
b, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
u := strings.TrimRight(base, "/") + path
|
||||
req, err := http.NewRequest(http.MethodPost, u, bytes.NewReader(b))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
resp, err := httpclient.DoWithRetry(ctx, httpclient.New(60*time.Second), req, 3)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
out, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != http.StatusAccepted && resp.StatusCode != http.StatusOK {
|
||||
return nil, fmt.Errorf("dbcli: POST %s: %s: %s", path, resp.Status, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func writeJSONStdout(v any, format string) int {
|
||||
if format != "json" {
|
||||
fmt.Fprintln(os.Stderr, "only json format supported")
|
||||
return 2
|
||||
}
|
||||
enc := json.NewEncoder(os.Stdout)
|
||||
enc.SetIndent("", " ")
|
||||
if err := enc.Encode(v); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func writeRawJSON(b []byte) int {
|
||||
var v any
|
||||
if err := json.Unmarshal(b, &v); err != nil {
|
||||
_, _ = os.Stdout.Write(b)
|
||||
return 0
|
||||
}
|
||||
return writeJSONStdout(v, "json")
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"time"
|
||||
|
||||
"evobgp/internal/db"
|
||||
"evobgp/internal/httpclient"
|
||||
"evobgp/internal/jobs"
|
||||
"evobgp/internal/observability"
|
||||
"evobgp/internal/repository"
|
||||
@@ -17,7 +18,7 @@ import (
|
||||
|
||||
// NewCDNHTTPClient returns the shared HTTP client for CDN and preview fetches (PERF-02 / ERR-03).
|
||||
func NewCDNHTTPClient() *http.Client {
|
||||
return &http.Client{Timeout: 45 * time.Second}
|
||||
return httpclient.New(httpclient.DefaultTimeout)
|
||||
}
|
||||
|
||||
// BootstrapWorkers opens the same store.Backend and jobs.Registry as New (without HTTP or bundle keys).
|
||||
@@ -50,23 +51,45 @@ func BootstrapWorkers(ctx context.Context, opts Options) (store.Backend, *jobs.R
|
||||
}
|
||||
|
||||
cdnHTTP := NewCDNHTTPClient()
|
||||
wk := &jobs.Worker{Store: backend, HTTPClient: cdnHTTP}
|
||||
wk := &jobs.Worker{Store: backend, PgPool: pool, HTTPClient: cdnHTTP}
|
||||
reg := jobs.NewRegistry(wk.Process)
|
||||
wk.Registry = reg
|
||||
if pool != nil {
|
||||
audit := repository.NewJobAuditWriter(pool)
|
||||
reg.SetTerminalHook(func(j *jobs.Job) {
|
||||
jobMeta := func(j *jobs.Job) map[string]any {
|
||||
if j == nil {
|
||||
return
|
||||
return nil
|
||||
}
|
||||
st := j.Snapshot()
|
||||
status, _ := st["status"].(string)
|
||||
var errMsg *string
|
||||
if e, ok := st["error"].(string); ok && e != "" {
|
||||
errMsg = &e
|
||||
}
|
||||
audit.MarkTerminal(context.Background(), j.TenantID, j.ID, status, errMsg, time.Now().UTC())
|
||||
})
|
||||
meta, _ := st["meta"].(map[string]any)
|
||||
return meta
|
||||
}
|
||||
reg.SetPersistHooks(
|
||||
func(j *jobs.Job) {
|
||||
if j == nil {
|
||||
return
|
||||
}
|
||||
audit.UpsertQueued(context.Background(), j.TenantID, j.ID, j.Kind, j.IdempotencyKey, j.ModuleID, jobMeta(j))
|
||||
},
|
||||
func(j *jobs.Job) {
|
||||
if j == nil {
|
||||
return
|
||||
}
|
||||
audit.UpsertRunning(context.Background(), j.TenantID, j.ID, j.Kind, j.IdempotencyKey, jobMeta(j))
|
||||
},
|
||||
func(j *jobs.Job) {
|
||||
if j == nil {
|
||||
return
|
||||
}
|
||||
st := j.Snapshot()
|
||||
status, _ := st["status"].(string)
|
||||
var errMsg *string
|
||||
if e, ok := st["error"].(string); ok && e != "" {
|
||||
errMsg = &e
|
||||
}
|
||||
audit.MarkTerminal(context.Background(), j.TenantID, j.ID, status, errMsg, time.Now().UTC())
|
||||
},
|
||||
)
|
||||
}
|
||||
observability.RegisterStoreBackend(backend)
|
||||
return backend, reg, pool, nil
|
||||
|
||||
@@ -0,0 +1,295 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/netip"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"evobgp/internal/birdfmt"
|
||||
"evobgp/internal/nodedispatch"
|
||||
"evobgp/internal/store"
|
||||
)
|
||||
|
||||
const peerLiveCacheTTL = 15 * time.Second
|
||||
|
||||
type speakerBGPLive struct {
|
||||
SpeakerID string
|
||||
Label string
|
||||
Sessions []birdfmt.BGPSession
|
||||
Error string
|
||||
}
|
||||
|
||||
type peerLiveCacheEntry struct {
|
||||
at time.Time
|
||||
views []speakerBGPLive
|
||||
}
|
||||
|
||||
var peerLiveCache sync.Map // tenantID -> peerLiveCacheEntry
|
||||
|
||||
type peerSessionOnSpeaker struct {
|
||||
SpeakerID string `json:"speaker_id"`
|
||||
Label string `json:"label"`
|
||||
State string `json:"state"`
|
||||
PollError string `json:"poll_error,omitempty"`
|
||||
}
|
||||
|
||||
type liveSpeakerPoll struct {
|
||||
SpeakerID string `json:"speaker_id"`
|
||||
Label string `json:"label"`
|
||||
OK bool `json:"ok"`
|
||||
SessionCount int `json:"session_count"`
|
||||
PollError string `json:"poll_error,omitempty"`
|
||||
}
|
||||
|
||||
func speakerDisplayLabel(sp *store.Speaker) string {
|
||||
if sp == nil {
|
||||
return ""
|
||||
}
|
||||
meta := store.ParseSpeakerMeta(sp.MetaJSON)
|
||||
host := strings.TrimSpace(meta.AgentDomain)
|
||||
if host == "" {
|
||||
host = strings.TrimSpace(sp.Endpoint)
|
||||
}
|
||||
if strings.EqualFold(strings.TrimSpace(sp.Role), "master") {
|
||||
if host != "" {
|
||||
return "CP · " + host
|
||||
}
|
||||
return "CP (master)"
|
||||
}
|
||||
if host != "" {
|
||||
return host
|
||||
}
|
||||
return sp.ID
|
||||
}
|
||||
|
||||
func masterSpeakerID(speakers []*store.Speaker) string {
|
||||
for _, sp := range speakers {
|
||||
if sp != nil && strings.EqualFold(strings.TrimSpace(sp.Role), "master") {
|
||||
return sp.ID
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (s *Server) collectSpeakerBGPLive(ctx context.Context, tenantID string, fresh bool) []speakerBGPLive {
|
||||
if !fresh {
|
||||
if v, ok := peerLiveCache.Load(tenantID); ok {
|
||||
ent := v.(peerLiveCacheEntry)
|
||||
if time.Since(ent.at) < peerLiveCacheTTL {
|
||||
return ent.views
|
||||
}
|
||||
}
|
||||
}
|
||||
speakers := s.store.ListSpeakersForTenant(tenantID)
|
||||
views := make([]speakerBGPLive, 0, len(speakers)+1)
|
||||
|
||||
if sock := strings.TrimSpace(os.Getenv("EVOBGP_BIRDC_SOCKET")); sock != "" {
|
||||
v := speakerBGPLive{Label: "CP (local BIRD)"}
|
||||
if mid := masterSpeakerID(speakers); mid != "" {
|
||||
v.SpeakerID = mid
|
||||
for _, sp := range speakers {
|
||||
if sp != nil && sp.ID == mid {
|
||||
v.Label = speakerDisplayLabel(sp)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
out, err := birdfmt.ShowProtocols(ctx, sock, strings.TrimSpace(os.Getenv("EVOBGP_BIRDC_BIN")))
|
||||
if err != nil {
|
||||
v.Error = err.Error()
|
||||
} else {
|
||||
v.Sessions = birdfmt.ParseBGPSessions(out)
|
||||
}
|
||||
views = append(views, v)
|
||||
}
|
||||
|
||||
opts := nodedispatch.Options{Timeout: 8 * time.Second}
|
||||
type resWrap struct {
|
||||
sp *store.Speaker
|
||||
res nodedispatch.BirdProtocolsResult
|
||||
}
|
||||
ch := make(chan resWrap, len(speakers))
|
||||
var wg sync.WaitGroup
|
||||
for _, sp := range speakers {
|
||||
if sp == nil {
|
||||
continue
|
||||
}
|
||||
meta := store.ParseSpeakerMeta(sp.MetaJSON)
|
||||
if !store.SpeakerNeedsRemoteDispatch(sp.Role, meta) {
|
||||
continue
|
||||
}
|
||||
wg.Add(1)
|
||||
go func(speaker *store.Speaker) {
|
||||
defer wg.Done()
|
||||
ch <- resWrap{
|
||||
sp: speaker,
|
||||
res: nodedispatch.FetchBirdProtocols(ctx, speaker, opts),
|
||||
}
|
||||
}(sp)
|
||||
}
|
||||
wg.Wait()
|
||||
close(ch)
|
||||
for rw := range ch {
|
||||
views = append(views, speakerBGPLive{
|
||||
SpeakerID: rw.sp.ID,
|
||||
Label: speakerDisplayLabel(rw.sp),
|
||||
Sessions: rw.res.Sessions,
|
||||
Error: rw.res.Error,
|
||||
})
|
||||
}
|
||||
|
||||
peerLiveCache.Store(tenantID, peerLiveCacheEntry{at: time.Now(), views: views})
|
||||
return views
|
||||
}
|
||||
|
||||
func liveSpeakerPollJSON(views []speakerBGPLive) []liveSpeakerPoll {
|
||||
out := make([]liveSpeakerPoll, 0, len(views))
|
||||
for _, v := range views {
|
||||
out = append(out, liveSpeakerPoll{
|
||||
SpeakerID: v.SpeakerID,
|
||||
Label: v.Label,
|
||||
OK: v.Error == "",
|
||||
SessionCount: len(v.Sessions),
|
||||
PollError: v.Error,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func findPeerSession(sessions []birdfmt.BGPSession, protoName string, neighbor netip.Addr, hasNeighbor bool) *birdfmt.BGPSession {
|
||||
for i := range sessions {
|
||||
if peerSessionMatches(sessions[i], protoName, neighbor, hasNeighbor) {
|
||||
return &sessions[i]
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func matchPeerOnSpeakers(peer *store.BGPPeer, views []speakerBGPLive) (
|
||||
bestState string,
|
||||
connectedID string,
|
||||
connectedLabel string,
|
||||
establishedOn []peerSessionOnSpeaker,
|
||||
on []peerSessionOnSpeaker,
|
||||
mismatch bool,
|
||||
) {
|
||||
if peer == nil {
|
||||
return "", "", "", nil, nil, false
|
||||
}
|
||||
neighbor, hasNeighbor := store.ParsePeerNeighbor(peer.Neighbor)
|
||||
protoName := birdfmt.PeerProtocolName(peer.ID)
|
||||
|
||||
for _, v := range views {
|
||||
if v.Error != "" && len(v.Sessions) == 0 {
|
||||
on = append(on, peerSessionOnSpeaker{
|
||||
SpeakerID: v.SpeakerID,
|
||||
Label: v.Label,
|
||||
PollError: v.Error,
|
||||
})
|
||||
continue
|
||||
}
|
||||
sess := findPeerSession(v.Sessions, protoName, neighbor, hasNeighbor)
|
||||
if sess == nil {
|
||||
on = append(on, peerSessionOnSpeaker{
|
||||
SpeakerID: v.SpeakerID,
|
||||
Label: v.Label,
|
||||
State: "absent",
|
||||
})
|
||||
continue
|
||||
}
|
||||
hit := peerSessionOnSpeaker{
|
||||
SpeakerID: v.SpeakerID,
|
||||
Label: v.Label,
|
||||
State: sess.State,
|
||||
}
|
||||
on = append(on, hit)
|
||||
if strings.EqualFold(strings.TrimSpace(sess.State), "Established") {
|
||||
establishedOn = append(establishedOn, hit)
|
||||
}
|
||||
if bestState == "" || sessionStateRank(sess.State) > sessionStateRank(bestState) {
|
||||
bestState = sess.State
|
||||
}
|
||||
}
|
||||
|
||||
if len(establishedOn) > 0 {
|
||||
bestState = "Established"
|
||||
labels := make([]string, 0, len(establishedOn))
|
||||
for _, e := range establishedOn {
|
||||
labels = append(labels, e.Label)
|
||||
}
|
||||
connectedLabel = strings.Join(labels, ", ")
|
||||
if len(establishedOn) == 1 {
|
||||
connectedID = establishedOn[0].SpeakerID
|
||||
}
|
||||
} else if len(on) == 1 && on[0].PollError == "" && on[0].State != "" {
|
||||
connectedID = on[0].SpeakerID
|
||||
connectedLabel = on[0].Label
|
||||
}
|
||||
|
||||
if peer.SpeakerID != nil && strings.TrimSpace(*peer.SpeakerID) != "" && len(establishedOn) > 0 {
|
||||
want := strings.TrimSpace(*peer.SpeakerID)
|
||||
found := false
|
||||
for _, e := range establishedOn {
|
||||
if strings.EqualFold(strings.TrimSpace(e.SpeakerID), want) {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
mismatch = !found
|
||||
}
|
||||
return bestState, connectedID, connectedLabel, establishedOn, on, mismatch
|
||||
}
|
||||
|
||||
func peerSessionMatches(sess birdfmt.BGPSession, protoName string, neighbor netip.Addr, hasNeighbor bool) bool {
|
||||
if strings.EqualFold(strings.TrimSpace(sess.Name), protoName) {
|
||||
return true
|
||||
}
|
||||
if !hasNeighbor || strings.TrimSpace(sess.Neighbor) == "" {
|
||||
return false
|
||||
}
|
||||
peerAddr, ok := store.ParsePeerNeighbor(sess.Neighbor)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
return peerAddr == neighbor
|
||||
}
|
||||
|
||||
func sessionStateRank(state string) int {
|
||||
switch strings.ToLower(strings.TrimSpace(state)) {
|
||||
case "established":
|
||||
return 100
|
||||
case "openconfirm", "opensent":
|
||||
return 80
|
||||
case "active", "connect":
|
||||
return 60
|
||||
case "idle":
|
||||
return 20
|
||||
default:
|
||||
return 10
|
||||
}
|
||||
}
|
||||
|
||||
func applyPeerLiveFields(row map[string]any, peer *store.BGPPeer, views []speakerBGPLive) {
|
||||
state, connID, connLabel, establishedOn, on, mismatch := matchPeerOnSpeakers(peer, views)
|
||||
row["session_on_speakers"] = on
|
||||
row["established_on_speakers"] = establishedOn
|
||||
row["session_conflict"] = false
|
||||
row["session_mismatch"] = mismatch
|
||||
if state != "" {
|
||||
row["session_state"] = state
|
||||
}
|
||||
if connLabel != "" {
|
||||
row["connected_speaker_label"] = connLabel
|
||||
}
|
||||
row["connected_speaker_id"] = peerLiveSpeakerIDOrNull(connID)
|
||||
}
|
||||
|
||||
func peerLiveSpeakerIDOrNull(id string) any {
|
||||
if strings.TrimSpace(id) == "" {
|
||||
return nil
|
||||
}
|
||||
return id
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"evobgp/internal/birdfmt"
|
||||
"evobgp/internal/store"
|
||||
)
|
||||
|
||||
func TestMatchPeerOnSpeakers_establishedOnReplica(t *testing.T) {
|
||||
peer := &store.BGPPeer{
|
||||
ID: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
|
||||
Neighbor: "198.51.100.2",
|
||||
}
|
||||
views := []speakerBGPLive{
|
||||
{
|
||||
SpeakerID: "master-id",
|
||||
Label: "CP · bgp.shz.su",
|
||||
Sessions: []birdfmt.BGPSession{
|
||||
{Name: birdfmt.PeerProtocolName(peer.ID), Neighbor: "198.51.100.2", State: "Established"},
|
||||
},
|
||||
},
|
||||
{
|
||||
SpeakerID: "replica-id",
|
||||
Label: "bgp2.shz.su",
|
||||
Sessions: []birdfmt.BGPSession{
|
||||
{Name: birdfmt.PeerProtocolName(peer.ID), Neighbor: "198.51.100.2", State: "Established"},
|
||||
},
|
||||
},
|
||||
}
|
||||
state, connID, connLabel, established, on, mismatch := matchPeerOnSpeakers(peer, views)
|
||||
if state != "Established" || connID != "" || connLabel != "CP · bgp.shz.su, bgp2.shz.su" {
|
||||
t.Fatalf("got state=%q conn=%q label=%q", state, connID, connLabel)
|
||||
}
|
||||
if mismatch || len(on) != 2 || len(established) != 2 {
|
||||
t.Fatalf("on=%+v established=%+v mismatch=%v", on, established, mismatch)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchPeerOnSpeakers_multipleEstablished(t *testing.T) {
|
||||
peer := &store.BGPPeer{ID: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", Neighbor: "198.51.100.2/32"}
|
||||
views := []speakerBGPLive{
|
||||
{SpeakerID: "a", Label: "n1", Sessions: []birdfmt.BGPSession{{Name: birdfmt.PeerProtocolName(peer.ID), State: "Established"}}},
|
||||
{SpeakerID: "b", Label: "n2", Sessions: []birdfmt.BGPSession{{Name: birdfmt.PeerProtocolName(peer.ID), State: "Established"}}},
|
||||
}
|
||||
_, connID, label, established, on, mismatch := matchPeerOnSpeakers(peer, views)
|
||||
if mismatch || connID != "" || label != "n1, n2" || len(established) != 2 || len(on) != 2 {
|
||||
t.Fatalf("connID=%q label=%q established=%+v on=%+v mismatch=%v", connID, label, established, on, mismatch)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchPeerOnSpeakers_mismatchConfiguredSpeaker(t *testing.T) {
|
||||
replica := "replica-id"
|
||||
peer := &store.BGPPeer{
|
||||
ID: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
|
||||
Neighbor: "198.51.100.2",
|
||||
SpeakerID: &replica,
|
||||
}
|
||||
views := []speakerBGPLive{
|
||||
{SpeakerID: "master-id", Label: "CP", Sessions: []birdfmt.BGPSession{{Name: birdfmt.PeerProtocolName(peer.ID), State: "Established"}}},
|
||||
{SpeakerID: replica, Label: "bgp2", Sessions: []birdfmt.BGPSession{{Name: birdfmt.PeerProtocolName(peer.ID), State: "Idle"}}},
|
||||
}
|
||||
_, _, _, _, _, mismatch := matchPeerOnSpeakers(peer, views)
|
||||
if !mismatch {
|
||||
t.Fatal("expected mismatch when configured replica has no Established")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchPeerOnSpeakers_pollError(t *testing.T) {
|
||||
peer := &store.BGPPeer{ID: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", Neighbor: "198.51.100.2"}
|
||||
views := []speakerBGPLive{
|
||||
{Label: "CP (local BIRD)", Sessions: []birdfmt.BGPSession{{Name: birdfmt.PeerProtocolName(peer.ID), State: "Established"}}},
|
||||
{SpeakerID: "replica-id", Label: "bgp2.shz.su", Error: "HTTP 404: Not Found"},
|
||||
}
|
||||
_, _, _, established, on, _ := matchPeerOnSpeakers(peer, views)
|
||||
if len(established) != 1 || len(on) != 2 || on[1].PollError == "" {
|
||||
t.Fatalf("on=%+v established=%+v", on, established)
|
||||
}
|
||||
}
|
||||
+28
-16
@@ -76,6 +76,8 @@ func (s *Server) registerV1(m *http.ServeMux) {
|
||||
m.HandleFunc("GET /speakers/{speaker_id}/bundle/{revision_id}", s.handleNodeBundle)
|
||||
m.HandleFunc("POST /nodes/enroll", s.handleNodeEnroll)
|
||||
s.registerCRUDRoutes(m)
|
||||
s.registerPostgresMonitoringRoutes(m)
|
||||
s.registerPostgresMaintenanceRoutes(m)
|
||||
}
|
||||
|
||||
func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -283,17 +285,21 @@ func (s *Server) handleListPeers(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
allPeers := s.store.ListPeers(a.TenantID)
|
||||
page, next, more := store.PaginateOffset(allPeers, r.URL.Query().Get("cursor"), parseListLimit(r))
|
||||
liveStates := s.liveBGPProtocolStates(r)
|
||||
fresh := r != nil && strings.EqualFold(strings.TrimSpace(r.URL.Query().Get("live")), "1")
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 12*time.Second)
|
||||
defer cancel()
|
||||
liveViews := s.collectSpeakerBGPLive(ctx, a.TenantID, fresh)
|
||||
items := make([]map[string]any, 0, len(page))
|
||||
for _, p := range page {
|
||||
row := peerJSON(p)
|
||||
if st, ok := liveStates[peerProtocolNameForID(p.ID)]; ok && strings.TrimSpace(st) != "" {
|
||||
row["session_state"] = strings.TrimSpace(st)
|
||||
}
|
||||
applyPeerLiveFields(row, p, liveViews)
|
||||
items = append(items, row)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{
|
||||
"items": items, "next_cursor": strPtrOrNull(next), "has_more": more,
|
||||
"items": items,
|
||||
"next_cursor": strPtrOrNull(next),
|
||||
"has_more": more,
|
||||
"live_speaker_poll": liveSpeakerPollJSON(liveViews),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -366,16 +372,9 @@ func extractBGPSessionState(line string) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
// peerProtocolNameForID must stay in sync with pipeline peer protocol naming.
|
||||
// peerProtocolNameForID forwards to birdfmt for tests and legacy callers.
|
||||
func peerProtocolNameForID(peerID string) string {
|
||||
s := strings.ReplaceAll(strings.TrimSpace(peerID), "-", "")
|
||||
if len(s) > 16 {
|
||||
s = s[:16]
|
||||
}
|
||||
if s == "" {
|
||||
s = "x"
|
||||
}
|
||||
return "evobgp_p_" + s
|
||||
return birdfmt.PeerProtocolName(peerID)
|
||||
}
|
||||
|
||||
func (s *Server) handleListSpeakers(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -388,9 +387,22 @@ func (s *Server) handleListSpeakers(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
speakers := s.store.ListSpeakersForTenant(a.TenantID)
|
||||
fresh := r != nil && strings.EqualFold(strings.TrimSpace(r.URL.Query().Get("live")), "1")
|
||||
var liveByID map[string]map[string]any
|
||||
if fresh {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 12*time.Second)
|
||||
defer cancel()
|
||||
liveByID = s.collectSpeakerLiveStatus(ctx, a.TenantID, true, speakers)
|
||||
}
|
||||
items := make([]map[string]any, 0, len(speakers))
|
||||
for _, sp := range speakers {
|
||||
items = append(items, speakerJSONFromStore(s.store, sp))
|
||||
row := speakerJSONFromStore(s.store, sp)
|
||||
if liveByID != nil {
|
||||
if live, ok := liveByID[sp.ID]; ok {
|
||||
row["live"] = live
|
||||
}
|
||||
}
|
||||
items = append(items, row)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{
|
||||
"items": items, "next_cursor": nil, "has_more": false,
|
||||
@@ -569,7 +581,7 @@ func (s *Server) handleGetRevision(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.requireAtLeast(w, a, "viewer") {
|
||||
return
|
||||
}
|
||||
rev, err := s.store.GetRevision(a.TenantID, r.PathValue("revision_id"))
|
||||
rev, err := s.store.GetRevisionSummary(a.TenantID, r.PathValue("revision_id"))
|
||||
if err != nil {
|
||||
writeProblem(w, http.StatusNotFound, "Not Found", "revision not found")
|
||||
return
|
||||
|
||||
@@ -242,6 +242,14 @@ func (s *Server) handlePreviewCDNSource(w http.ResponseWriter, r *http.Request)
|
||||
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", "url is required")
|
||||
return
|
||||
}
|
||||
if _, err := pipeline.ValidateCDNURL(u); err != nil {
|
||||
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", invalidInputDetail)
|
||||
return
|
||||
}
|
||||
if err := pipeline.ResolveCDNURLHost(r.Context(), u); err != nil {
|
||||
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", invalidInputDetail)
|
||||
return
|
||||
}
|
||||
mod, err := s.store.GetModule(a.TenantID, r.PathValue("module_id"))
|
||||
if err != nil {
|
||||
writeStoreErr(w, err)
|
||||
@@ -304,6 +312,16 @@ func (s *Server) handlePostCDNSource(w http.ResponseWriter, r *http.Request) {
|
||||
writeProblem(w, http.StatusBadRequest, "Bad Request", "invalid json")
|
||||
return
|
||||
}
|
||||
if body.URL != "" {
|
||||
if _, err := pipeline.ValidateCDNURL(body.URL); err != nil {
|
||||
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", invalidInputDetail)
|
||||
return
|
||||
}
|
||||
if err := pipeline.ResolveCDNURLHost(r.Context(), body.URL); err != nil {
|
||||
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", invalidInputDetail)
|
||||
return
|
||||
}
|
||||
}
|
||||
mid := r.PathValue("module_id")
|
||||
x, err := s.store.CreateCDNSource(a.TenantID, mid, &body)
|
||||
if err != nil {
|
||||
@@ -324,6 +342,16 @@ func (s *Server) handlePatchCDNSource(w http.ResponseWriter, r *http.Request) {
|
||||
writeProblem(w, http.StatusBadRequest, "Bad Request", "invalid json")
|
||||
return
|
||||
}
|
||||
if body.URL != nil && strings.TrimSpace(*body.URL) != "" {
|
||||
if _, err := pipeline.ValidateCDNURL(*body.URL); err != nil {
|
||||
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", invalidInputDetail)
|
||||
return
|
||||
}
|
||||
if err := pipeline.ResolveCDNURLHost(r.Context(), *body.URL); err != nil {
|
||||
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", invalidInputDetail)
|
||||
return
|
||||
}
|
||||
}
|
||||
mid := r.PathValue("module_id")
|
||||
x, err := s.store.UpdateCDNSource(a.TenantID, mid, r.PathValue("source_id"), &body)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"evobgp/internal/jobs"
|
||||
"evobgp/internal/pgmonitor"
|
||||
)
|
||||
|
||||
var (
|
||||
pgMaintRateMu sync.Mutex
|
||||
pgMaintLastByTK = map[string]time.Time{}
|
||||
)
|
||||
|
||||
func (s *Server) registerPostgresMaintenanceRoutes(m *http.ServeMux) {
|
||||
m.HandleFunc("POST /postgres/vacuum", s.handlePostgresVacuum)
|
||||
m.HandleFunc("POST /postgres/vacuum-analyze", s.handlePostgresVacuumAnalyze)
|
||||
m.HandleFunc("POST /postgres/analyze", s.handlePostgresAnalyze)
|
||||
m.HandleFunc("POST /postgres/reindex", s.handlePostgresReindex)
|
||||
m.HandleFunc("POST /postgres/cleanup", s.handlePostgresCleanup)
|
||||
m.HandleFunc("GET /postgres/maintenance/logs", s.handlePostgresMaintenanceLogs)
|
||||
}
|
||||
|
||||
func (s *Server) requireOperatorStrict(w http.ResponseWriter, a Auth) bool {
|
||||
if strings.ToLower(a.Role) != "operator" {
|
||||
writeProblem(w, http.StatusForbidden, "Forbidden", "operator role required")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *Server) checkPgMaintRateLimit(tenantID, kind string) bool {
|
||||
key := tenantID + ":" + kind
|
||||
pgMaintRateMu.Lock()
|
||||
defer pgMaintRateMu.Unlock()
|
||||
if t, ok := pgMaintLastByTK[key]; ok && time.Since(t) < 60*time.Second {
|
||||
return false
|
||||
}
|
||||
pgMaintLastByTK[key] = time.Now().UTC()
|
||||
return true
|
||||
}
|
||||
|
||||
type pgMaintBody struct {
|
||||
Table string `json:"table"`
|
||||
DryRun bool `json:"dry_run"`
|
||||
Index string `json:"index"`
|
||||
Policy string `json:"policy"`
|
||||
Limit int `json:"limit"`
|
||||
}
|
||||
|
||||
func (s *Server) decodePgMaintBody(r *http.Request) (pgMaintBody, bool) {
|
||||
var body pgMaintBody
|
||||
if r.Body == nil || r.ContentLength == 0 {
|
||||
return body, true
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&body); err != nil && err != io.EOF {
|
||||
return body, false
|
||||
}
|
||||
return body, true
|
||||
}
|
||||
|
||||
func (s *Server) enqueuePostgresMaint(w http.ResponseWriter, r *http.Request, a Auth, kind string, meta map[string]any) {
|
||||
if !s.requirePostgres(w) || !s.requireOperatorStrict(w, a) {
|
||||
return
|
||||
}
|
||||
if !s.checkPgMaintRateLimit(a.TenantID, kind) {
|
||||
writeProblem(w, http.StatusTooManyRequests, "Too Many Requests", "wait before repeating this maintenance operation")
|
||||
return
|
||||
}
|
||||
idem := strings.TrimSpace(r.Header.Get("Idempotency-Key"))
|
||||
var idemPtr *string
|
||||
if idem != "" {
|
||||
idemPtr = &idem
|
||||
}
|
||||
meta["actor_prefix"] = actorPrefix(a)
|
||||
j, _, err := s.jobs.Enqueue(a.TenantID, kind, idemPtr, nil, meta)
|
||||
if err != nil {
|
||||
writeInternalError(w, "postgres_maint_enqueue", err)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Location", "/v1/jobs/"+j.ID)
|
||||
snap := j.Snapshot()
|
||||
writeJSON(w, http.StatusAccepted, map[string]any{"job_id": snap["job_id"], "status": snap["status"]})
|
||||
}
|
||||
|
||||
func (s *Server) handlePostgresVacuum(w http.ResponseWriter, r *http.Request) {
|
||||
a, ok := authFromContext(r.Context())
|
||||
if !ok {
|
||||
writeProblem(w, http.StatusUnauthorized, "Unauthorized", "missing auth")
|
||||
return
|
||||
}
|
||||
body, ok2 := s.decodePgMaintBody(r)
|
||||
if !ok2 {
|
||||
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
|
||||
return
|
||||
}
|
||||
s.enqueuePostgresMaint(w, r, a, jobs.KindPostgresVacuum, map[string]any{
|
||||
"table": body.Table, "dry_run": body.DryRun, "job_title": "PostgreSQL VACUUM",
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) handlePostgresVacuumAnalyze(w http.ResponseWriter, r *http.Request) {
|
||||
a, ok := authFromContext(r.Context())
|
||||
if !ok {
|
||||
writeProblem(w, http.StatusUnauthorized, "Unauthorized", "missing auth")
|
||||
return
|
||||
}
|
||||
body, ok2 := s.decodePgMaintBody(r)
|
||||
if !ok2 {
|
||||
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
|
||||
return
|
||||
}
|
||||
s.enqueuePostgresMaint(w, r, a, jobs.KindPostgresVacuumAnalyze, map[string]any{
|
||||
"table": body.Table, "dry_run": body.DryRun, "job_title": "PostgreSQL VACUUM ANALYZE",
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) handlePostgresAnalyze(w http.ResponseWriter, r *http.Request) {
|
||||
a, ok := authFromContext(r.Context())
|
||||
if !ok {
|
||||
writeProblem(w, http.StatusUnauthorized, "Unauthorized", "missing auth")
|
||||
return
|
||||
}
|
||||
body, ok2 := s.decodePgMaintBody(r)
|
||||
if !ok2 {
|
||||
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
|
||||
return
|
||||
}
|
||||
s.enqueuePostgresMaint(w, r, a, jobs.KindPostgresAnalyze, map[string]any{
|
||||
"table": body.Table, "dry_run": body.DryRun, "job_title": "PostgreSQL ANALYZE",
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) handlePostgresReindex(w http.ResponseWriter, r *http.Request) {
|
||||
a, ok := authFromContext(r.Context())
|
||||
if !ok {
|
||||
writeProblem(w, http.StatusUnauthorized, "Unauthorized", "missing auth")
|
||||
return
|
||||
}
|
||||
body, ok2 := s.decodePgMaintBody(r)
|
||||
if !ok2 {
|
||||
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
|
||||
return
|
||||
}
|
||||
table := body.Table
|
||||
if table == "" {
|
||||
table = body.Index
|
||||
}
|
||||
s.enqueuePostgresMaint(w, r, a, jobs.KindPostgresReindex, map[string]any{
|
||||
"table": table, "dry_run": body.DryRun, "job_title": "PostgreSQL REINDEX",
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) handlePostgresCleanup(w http.ResponseWriter, r *http.Request) {
|
||||
a, ok := authFromContext(r.Context())
|
||||
if !ok {
|
||||
writeProblem(w, http.StatusUnauthorized, "Unauthorized", "missing auth")
|
||||
return
|
||||
}
|
||||
body, ok2 := s.decodePgMaintBody(r)
|
||||
if !ok2 {
|
||||
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(body.Policy) == "" {
|
||||
writeProblem(w, http.StatusBadRequest, "Bad Request", "policy is required")
|
||||
return
|
||||
}
|
||||
s.enqueuePostgresMaint(w, r, a, jobs.KindPostgresCleanup, map[string]any{
|
||||
"policy": body.Policy, "dry_run": body.DryRun, "limit": body.Limit,
|
||||
"job_title": "PostgreSQL cleanup",
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) handlePostgresMaintenanceLogs(w http.ResponseWriter, r *http.Request) {
|
||||
a, ok := authFromContext(r.Context())
|
||||
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
|
||||
return
|
||||
}
|
||||
cursor := r.URL.Query().Get("cursor")
|
||||
limit := parseLimitQuery(r, 20, 100)
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
|
||||
defer cancel()
|
||||
items, next, hasMore, err := pgmonitor.ListMaintenanceLogs(ctx, s.pgMonitor.Pool(), cursor, limit)
|
||||
if err != nil {
|
||||
writeInternalError(w, "postgres_maint_logs", err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"items": items, "next_cursor": next, "has_more": hasMore})
|
||||
}
|
||||
|
||||
func actorPrefix(a Auth) string {
|
||||
if len(a.Token) >= 8 {
|
||||
return a.Token[:8]
|
||||
}
|
||||
return a.Role
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
func (s *Server) registerPostgresMonitoringRoutes(m *http.ServeMux) {
|
||||
m.HandleFunc("GET /monitoring/postgres/overview", s.handlePostgresOverview)
|
||||
m.HandleFunc("GET /monitoring/postgres/queries", s.handlePostgresQueries)
|
||||
m.HandleFunc("GET /monitoring/postgres/locks", s.handlePostgresLocks)
|
||||
m.HandleFunc("GET /monitoring/postgres/tables", s.handlePostgresTables)
|
||||
m.HandleFunc("GET /monitoring/postgres/recommendations", s.handlePostgresRecommendations)
|
||||
m.HandleFunc("GET /monitoring/correlation", s.handleMonitoringCorrelation)
|
||||
}
|
||||
|
||||
func (s *Server) requirePostgres(w http.ResponseWriter) bool {
|
||||
if s.pgMonitor == nil {
|
||||
writeProblem(w, http.StatusServiceUnavailable, "Unavailable", "postgresql backend required")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func parseLimitQuery(r *http.Request, def, max int) int {
|
||||
if v := r.URL.Query().Get("limit"); v != "" {
|
||||
if n, err := strconv.Atoi(v); err == nil {
|
||||
return n
|
||||
}
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func (s *Server) handlePostgresOverview(w http.ResponseWriter, r *http.Request) {
|
||||
a, ok := authFromContext(r.Context())
|
||||
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
|
||||
defer cancel()
|
||||
out, err := s.pgMonitor.Overview(ctx)
|
||||
if err != nil {
|
||||
writeInternalError(w, "postgres_overview", err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (s *Server) handlePostgresQueries(w http.ResponseWriter, r *http.Request) {
|
||||
a, ok := authFromContext(r.Context())
|
||||
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
|
||||
defer cancel()
|
||||
out, err := s.pgMonitor.TopQueries(ctx, parseLimitQuery(r, 20, 100))
|
||||
if err != nil {
|
||||
writeInternalError(w, "postgres_queries", err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (s *Server) handlePostgresLocks(w http.ResponseWriter, r *http.Request) {
|
||||
a, ok := authFromContext(r.Context())
|
||||
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
|
||||
defer cancel()
|
||||
out, err := s.pgMonitor.Locks(ctx)
|
||||
if err != nil {
|
||||
writeInternalError(w, "postgres_locks", err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"items": out})
|
||||
}
|
||||
|
||||
func (s *Server) handlePostgresTables(w http.ResponseWriter, r *http.Request) {
|
||||
a, ok := authFromContext(r.Context())
|
||||
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
|
||||
defer cancel()
|
||||
out, err := s.pgMonitor.Tables(ctx, parseLimitQuery(r, 20, 100))
|
||||
if err != nil {
|
||||
writeInternalError(w, "postgres_tables", err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"items": out})
|
||||
}
|
||||
|
||||
func (s *Server) handlePostgresRecommendations(w http.ResponseWriter, r *http.Request) {
|
||||
a, ok := authFromContext(r.Context())
|
||||
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
|
||||
defer cancel()
|
||||
out, err := s.pgMonitor.Recommendations(ctx)
|
||||
if err != nil {
|
||||
writeInternalError(w, "postgres_recommendations", err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (s *Server) handleMonitoringCorrelation(w http.ResponseWriter, r *http.Request) {
|
||||
a, ok := authFromContext(r.Context())
|
||||
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
|
||||
return
|
||||
}
|
||||
window := 60
|
||||
if v := r.URL.Query().Get("window"); v != "" {
|
||||
if n, err := strconv.Atoi(v); err == nil {
|
||||
window = n
|
||||
}
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
|
||||
defer cancel()
|
||||
out, err := s.pgMonitor.Correlation(ctx, window)
|
||||
if err != nil {
|
||||
writeInternalError(w, "monitoring_correlation", err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestPostgresOverviewMemoryBackend503(t *testing.T) {
|
||||
srv, err := New(Options{SeedDemo: true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/monitoring/postgres/overview", nil)
|
||||
req.Header.Set("Authorization", "Bearer dev")
|
||||
rec := httptest.NewRecorder()
|
||||
srv.Handler().ServeHTTP(rec, req)
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("status=%d body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"strings"
|
||||
|
||||
"evobgp/internal/jobs"
|
||||
"evobgp/internal/pgmonitor"
|
||||
"evobgp/internal/store"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
@@ -19,6 +20,7 @@ import (
|
||||
type Server struct {
|
||||
store store.Backend
|
||||
pgPool *pgxpool.Pool
|
||||
pgMonitor *pgmonitor.Service
|
||||
jobs *jobs.Registry
|
||||
bundlePriv ed25519.PrivateKey
|
||||
keyResolver *apiKeyResolver
|
||||
@@ -63,9 +65,14 @@ func New(opts Options) (*Server, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var pgMon *pgmonitor.Service
|
||||
if pool != nil {
|
||||
pgMon = pgmonitor.NewService(pool)
|
||||
}
|
||||
s := &Server{
|
||||
store: backend,
|
||||
pgPool: pool,
|
||||
pgMonitor: pgMon,
|
||||
jobs: reg,
|
||||
bundlePriv: priv,
|
||||
keyResolver: resolver,
|
||||
@@ -89,3 +96,10 @@ func (s *Server) Store() store.Backend { return s.store }
|
||||
|
||||
// Jobs exposes the in-process async job registry (for scheduler / evobgp-all).
|
||||
func (s *Server) Jobs() *jobs.Registry { return s.jobs }
|
||||
|
||||
// StartBackground starts PostgreSQL monitoring scheduler until ctx is cancelled.
|
||||
func (s *Server) StartBackground(ctx context.Context) {
|
||||
if s != nil && s.pgPool != nil {
|
||||
pgmonitor.StartScheduler(ctx, s.pgPool)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -39,7 +39,10 @@ func speakerJSONFromStore(st store.Backend, sp *store.Speaker) map[string]any {
|
||||
if strings.TrimSpace(sp.MetaJSON) != "" && sp.MetaJSON != "{}" {
|
||||
var raw map[string]any
|
||||
if json.Unmarshal([]byte(sp.MetaJSON), &raw) == nil {
|
||||
m["meta_json"] = raw
|
||||
delete(raw, "agent_secret")
|
||||
if len(raw) > 0 {
|
||||
m["meta_json"] = raw
|
||||
}
|
||||
}
|
||||
}
|
||||
if meta.AgentDomain != "" {
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"evobgp/internal/birdfmt"
|
||||
"evobgp/internal/nodedispatch"
|
||||
"evobgp/internal/store"
|
||||
)
|
||||
|
||||
func countBGPSessions(sessions []birdfmt.BGPSession) (total, established int) {
|
||||
total = len(sessions)
|
||||
for _, s := range sessions {
|
||||
if strings.EqualFold(strings.TrimSpace(s.State), "Established") {
|
||||
established++
|
||||
}
|
||||
}
|
||||
return total, established
|
||||
}
|
||||
|
||||
func speakerLiveStatusJSON(sp *store.Speaker, view speakerBGPLive, health *nodedispatch.AgentHealthResult) map[string]any {
|
||||
total, established := countBGPSessions(view.Sessions)
|
||||
m := map[string]any{
|
||||
"label": view.Label,
|
||||
"bgp_poll_ok": view.Error == "",
|
||||
"bgp_sessions_total": total,
|
||||
"bgp_established": established,
|
||||
}
|
||||
if view.Error != "" {
|
||||
m["bgp_poll_error"] = view.Error
|
||||
}
|
||||
if health != nil {
|
||||
m["agent_ok"] = health.OK
|
||||
if health.Error != "" {
|
||||
m["agent_error"] = health.Error
|
||||
}
|
||||
if health.LastSyncAt != "" {
|
||||
m["agent_last_sync_at"] = health.LastSyncAt
|
||||
}
|
||||
if health.LastAppliedRevisionID != "" {
|
||||
m["agent_last_applied_revision_id"] = health.LastAppliedRevisionID
|
||||
}
|
||||
} else if sp != nil && strings.EqualFold(strings.TrimSpace(sp.Role), "master") {
|
||||
m["agent_ok"] = view.Error == ""
|
||||
if view.Error != "" {
|
||||
m["agent_error"] = view.Error
|
||||
}
|
||||
} else if sp != nil && store.SpeakerNeedsRemoteDispatch(sp.Role, store.ParseSpeakerMeta(sp.MetaJSON)) {
|
||||
m["agent_ok"] = false
|
||||
m["agent_error"] = "agent health not polled"
|
||||
}
|
||||
if len(view.Sessions) > 0 {
|
||||
sess := make([]map[string]any, 0, len(view.Sessions))
|
||||
for _, s := range view.Sessions {
|
||||
row := map[string]any{
|
||||
"name": s.Name,
|
||||
"state": s.State,
|
||||
}
|
||||
if strings.TrimSpace(s.Neighbor) != "" {
|
||||
row["neighbor"] = s.Neighbor
|
||||
}
|
||||
sess = append(sess, row)
|
||||
}
|
||||
m["sessions"] = sess
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func (s *Server) collectSpeakerLiveStatus(ctx context.Context, tenantID string, fresh bool, speakers []*store.Speaker) map[string]map[string]any {
|
||||
views := s.collectSpeakerBGPLive(ctx, tenantID, fresh)
|
||||
viewByID := make(map[string]speakerBGPLive, len(views))
|
||||
for _, v := range views {
|
||||
if v.SpeakerID != "" {
|
||||
viewByID[v.SpeakerID] = v
|
||||
}
|
||||
}
|
||||
|
||||
opts := nodedispatch.Options{Timeout: 8 * time.Second}
|
||||
type healthWrap struct {
|
||||
id string
|
||||
h nodedispatch.AgentHealthResult
|
||||
}
|
||||
healthCh := make(chan healthWrap, len(speakers))
|
||||
var wg sync.WaitGroup
|
||||
for _, sp := range speakers {
|
||||
if sp == nil {
|
||||
continue
|
||||
}
|
||||
meta := store.ParseSpeakerMeta(sp.MetaJSON)
|
||||
if !store.SpeakerNeedsRemoteDispatch(sp.Role, meta) {
|
||||
continue
|
||||
}
|
||||
wg.Add(1)
|
||||
go func(speaker *store.Speaker) {
|
||||
defer wg.Done()
|
||||
healthCh <- healthWrap{
|
||||
id: speaker.ID,
|
||||
h: nodedispatch.FetchAgentHealth(ctx, speaker, opts),
|
||||
}
|
||||
}(sp)
|
||||
}
|
||||
wg.Wait()
|
||||
close(healthCh)
|
||||
healthByID := make(map[string]nodedispatch.AgentHealthResult, len(speakers))
|
||||
for hw := range healthCh {
|
||||
healthByID[hw.id] = hw.h
|
||||
}
|
||||
|
||||
out := make(map[string]map[string]any, len(speakers))
|
||||
for _, sp := range speakers {
|
||||
if sp == nil {
|
||||
continue
|
||||
}
|
||||
view, ok := viewByID[sp.ID]
|
||||
if !ok {
|
||||
view = speakerBGPLive{SpeakerID: sp.ID, Label: speakerDisplayLabel(sp)}
|
||||
}
|
||||
var hp *nodedispatch.AgentHealthResult
|
||||
if h, ok := healthByID[sp.ID]; ok {
|
||||
hCopy := h
|
||||
hp = &hCopy
|
||||
}
|
||||
out[sp.ID] = speakerLiveStatusJSON(sp, view, hp)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"evobgp/internal/birdfmt"
|
||||
"evobgp/internal/nodedispatch"
|
||||
"evobgp/internal/store"
|
||||
)
|
||||
|
||||
func TestCountBGPSessions(t *testing.T) {
|
||||
total, est := countBGPSessions([]birdfmt.BGPSession{
|
||||
{Name: "p1", State: "Established"},
|
||||
{Name: "p2", State: "Idle"},
|
||||
{Name: "p3", State: "established"},
|
||||
})
|
||||
if total != 3 || est != 2 {
|
||||
t.Fatalf("total=%d established=%d", total, est)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSpeakerLiveStatusJSON_masterUsesBirdPoll(t *testing.T) {
|
||||
sp := &store.Speaker{ID: "m1", Role: "master", Endpoint: "https://cp.example"}
|
||||
view := speakerBGPLive{
|
||||
SpeakerID: "m1",
|
||||
Label: "CP · cp.example",
|
||||
Sessions: []birdfmt.BGPSession{
|
||||
{Name: "evobgp_peer_x", State: "Established"},
|
||||
},
|
||||
}
|
||||
m := speakerLiveStatusJSON(sp, view, nil)
|
||||
if m["agent_ok"] != true || m["bgp_established"] != 1 || m["bgp_sessions_total"] != 1 {
|
||||
t.Fatalf("got %#v", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSpeakerLiveStatusJSON_replicaWithHealth(t *testing.T) {
|
||||
sp := &store.Speaker{
|
||||
ID: "r1",
|
||||
Role: "replica",
|
||||
Endpoint: "https://node.example",
|
||||
MetaJSON: `{"agent_domain":"node.example","agent_secret":"s"}`,
|
||||
}
|
||||
view := speakerBGPLive{
|
||||
SpeakerID: "r1",
|
||||
Label: "node.example",
|
||||
Sessions: []birdfmt.BGPSession{{Name: "p", State: "Idle"}},
|
||||
}
|
||||
health := &nodedispatch.AgentHealthResult{
|
||||
OK: true,
|
||||
LastSyncAt: "2026-05-21T12:00:00Z",
|
||||
LastAppliedRevisionID: "rev-1",
|
||||
}
|
||||
m := speakerLiveStatusJSON(sp, view, health)
|
||||
if m["agent_ok"] != true || m["agent_last_sync_at"] != "2026-05-21T12:00:00Z" {
|
||||
t.Fatalf("got %#v", m)
|
||||
}
|
||||
if m["bgp_established"] != 0 || m["bgp_poll_ok"] != true {
|
||||
t.Fatalf("bgp fields: %#v", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSpeakerLiveStatusJSON_pollError(t *testing.T) {
|
||||
sp := &store.Speaker{ID: "r1", Role: "replica", MetaJSON: `{"agent_domain":"x.example"}`}
|
||||
view := speakerBGPLive{SpeakerID: "r1", Label: "x.example", Error: "HTTP 503"}
|
||||
health := &nodedispatch.AgentHealthResult{OK: false, Error: "timeout"}
|
||||
m := speakerLiveStatusJSON(sp, view, health)
|
||||
if m["bgp_poll_ok"] != false || m["bgp_poll_error"] != "HTTP 503" {
|
||||
t.Fatalf("got %#v", m)
|
||||
}
|
||||
if m["agent_ok"] != false {
|
||||
t.Fatalf("agent_ok: %#v", m)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGetSpeaker_redactsAgentSecret(t *testing.T) {
|
||||
srv, err := New(Options{InsecureDev: true, SeedDemo: true, BundleSeedHex: testBundleSeed})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer srv.Close()
|
||||
tenant, _, _, _, demoSpk := srv.Store().DemoIDs()
|
||||
mustSetTestAPIKeys(t, srv, "vwkey|"+tenant+"|viewer")
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/speakers/"+demoSpk, nil)
|
||||
req.Header.Set("Authorization", "Bearer vwkey")
|
||||
rec := httptest.NewRecorder()
|
||||
srv.Handler().ServeHTTP(rec, req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d body %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var out map[string]any
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if out["agent_secret"] != nil {
|
||||
t.Fatalf("agent_secret must not appear at top level: %#v", out["agent_secret"])
|
||||
}
|
||||
meta, _ := out["meta_json"].(map[string]any)
|
||||
if meta != nil {
|
||||
if v, ok := meta["agent_secret"]; ok && v != nil && v != "" {
|
||||
t.Fatalf("agent_secret must be redacted from meta_json: %#v", v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestListSpeakers_redactsAgentSecret(t *testing.T) {
|
||||
srv, err := New(Options{InsecureDev: true, SeedDemo: true, BundleSeedHex: testBundleSeed})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer srv.Close()
|
||||
tenant, _, _, _, _ := srv.Store().DemoIDs()
|
||||
mustSetTestAPIKeys(t, srv, "vwkey|"+tenant+"|viewer")
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/speakers", nil)
|
||||
req.Header.Set("Authorization", "Bearer vwkey")
|
||||
rec := httptest.NewRecorder()
|
||||
srv.Handler().ServeHTTP(rec, req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d body %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
body := rec.Body.String()
|
||||
if strings.Contains(body, "agent_secret") {
|
||||
t.Fatalf("list response must not contain agent_secret: %s", body)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
package httpclient
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultBreakerThreshold = 5
|
||||
defaultBreakerCooldown = 30 * time.Second
|
||||
)
|
||||
|
||||
type hostBreaker struct {
|
||||
mu sync.Mutex
|
||||
failures int
|
||||
openUntil time.Time
|
||||
}
|
||||
|
||||
var hostBreakers sync.Map // string -> *hostBreaker
|
||||
|
||||
func breakerForHost(host string) *hostBreaker {
|
||||
if host == "" {
|
||||
host = "_"
|
||||
}
|
||||
v, _ := hostBreakers.LoadOrStore(host, &hostBreaker{})
|
||||
return v.(*hostBreaker)
|
||||
}
|
||||
|
||||
func (b *hostBreaker) allow() bool {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
return time.Now().After(b.openUntil)
|
||||
}
|
||||
|
||||
func (b *hostBreaker) recordSuccess() {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
b.failures = 0
|
||||
b.openUntil = time.Time{}
|
||||
}
|
||||
|
||||
func (b *hostBreaker) recordFailure() {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
b.failures++
|
||||
if b.failures >= defaultBreakerThreshold {
|
||||
b.openUntil = time.Now().Add(defaultBreakerCooldown)
|
||||
b.failures = 0
|
||||
}
|
||||
}
|
||||
|
||||
// ResetHostBreakers clears all circuit breakers (tests only).
|
||||
func ResetHostBreakers() {
|
||||
hostBreakers = sync.Map{}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package httpclient
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestDoWithBreaker_opensAfterFailures(t *testing.T) {
|
||||
ResetHostBreakers()
|
||||
var calls atomic.Int32
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
calls.Add(1)
|
||||
http.Error(w, "fail", http.StatusBadGateway)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
hc := New(5 * time.Second)
|
||||
for i := 0; i < defaultBreakerThreshold*3; i++ {
|
||||
req, _ := http.NewRequest(http.MethodGet, srv.URL, nil)
|
||||
_, _ = DoWithBreaker(context.Background(), hc, req, 1)
|
||||
}
|
||||
req, _ := http.NewRequest(http.MethodGet, srv.URL, nil)
|
||||
_, err := DoWithBreaker(context.Background(), hc, req, 1)
|
||||
if err == nil || err.Error() == "" {
|
||||
t.Fatal("expected circuit open error")
|
||||
}
|
||||
if got := calls.Load(); got == 0 {
|
||||
t.Fatal("expected at least one upstream call")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
// Package httpclient provides shared HTTP clients and retry helpers for outbound calls.
|
||||
package httpclient
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
|
||||
const DefaultTimeout = 45 * time.Second
|
||||
|
||||
// New returns an HTTP client with timeout and tuned idle connection pooling.
|
||||
func New(timeout time.Duration) *http.Client {
|
||||
if timeout <= 0 {
|
||||
timeout = DefaultTimeout
|
||||
}
|
||||
tr := http.DefaultTransport.(*http.Transport).Clone()
|
||||
tr.MaxIdleConns = 100
|
||||
tr.MaxIdleConnsPerHost = 10
|
||||
return &http.Client{Timeout: timeout, Transport: tr}
|
||||
}
|
||||
|
||||
// DoWithRetry executes hc.Do(req) up to maxAttempts times with linear backoff.
|
||||
func DoWithRetry(ctx context.Context, hc *http.Client, req *http.Request, maxAttempts int) (*http.Response, error) {
|
||||
if maxAttempts <= 0 {
|
||||
maxAttempts = 3
|
||||
}
|
||||
var lastErr error
|
||||
for attempt := 0; attempt < maxAttempts; attempt++ {
|
||||
if attempt > 0 {
|
||||
wait := time.Duration(attempt) * 2 * time.Second
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
case <-time.After(wait):
|
||||
}
|
||||
if req.GetBody != nil {
|
||||
body, err := req.GetBody()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Body = body
|
||||
}
|
||||
}
|
||||
reqClone := req.Clone(ctx)
|
||||
resp, err := hc.Do(reqClone)
|
||||
if err != nil {
|
||||
lastErr = err
|
||||
continue
|
||||
}
|
||||
if resp.StatusCode >= 500 {
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
_ = resp.Body.Close()
|
||||
lastErr = fmt.Errorf("httpclient: upstream %s", resp.Status)
|
||||
continue
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
if lastErr != nil {
|
||||
return nil, lastErr
|
||||
}
|
||||
return nil, fmt.Errorf("httpclient: request failed after %d attempts", maxAttempts)
|
||||
}
|
||||
|
||||
// DoWithBreaker applies per-host circuit breaking then retries transient failures.
|
||||
func DoWithBreaker(ctx context.Context, hc *http.Client, req *http.Request, maxAttempts int) (*http.Response, error) {
|
||||
if req == nil || req.URL == nil {
|
||||
return nil, fmt.Errorf("httpclient: nil request")
|
||||
}
|
||||
br := breakerForHost(req.URL.Hostname())
|
||||
if !br.allow() {
|
||||
return nil, fmt.Errorf("httpclient: circuit open for %s", req.URL.Hostname())
|
||||
}
|
||||
resp, err := DoWithRetry(ctx, hc, req, maxAttempts)
|
||||
if err != nil {
|
||||
br.recordFailure()
|
||||
return nil, err
|
||||
}
|
||||
if resp.StatusCode >= 500 {
|
||||
br.recordFailure()
|
||||
return resp, nil
|
||||
}
|
||||
br.recordSuccess()
|
||||
return resp, nil
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package httpclient
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestDoWithRetry_retriesOn500(t *testing.T) {
|
||||
var calls int
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
calls++
|
||||
if calls < 3 {
|
||||
http.Error(w, "fail", http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write([]byte("ok"))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
req, err := http.NewRequest(http.MethodGet, srv.URL, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp, err := DoWithRetry(context.Background(), New(5*time.Second), req, 3)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("status %d", resp.StatusCode)
|
||||
}
|
||||
if calls != 3 {
|
||||
t.Fatalf("want 3 calls, got %d", calls)
|
||||
}
|
||||
}
|
||||
+12
-3
@@ -3,11 +3,11 @@ package ingest
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"evobgp/internal/broker"
|
||||
"evobgp/internal/config"
|
||||
"evobgp/internal/httpclient"
|
||||
"evobgp/internal/pipeline"
|
||||
"evobgp/internal/store"
|
||||
)
|
||||
@@ -17,6 +17,8 @@ type Deps struct {
|
||||
Store store.Backend
|
||||
}
|
||||
|
||||
var lastMaintenance time.Time
|
||||
|
||||
// Run blocks until ctx is cancelled.
|
||||
func Run(ctx context.Context, deps *Deps) {
|
||||
cfg := config.Load()
|
||||
@@ -24,7 +26,7 @@ func Run(ctx context.Context, deps *Deps) {
|
||||
if deps == nil || deps.Store == nil {
|
||||
log.Fatalf("evobgp-ingest: missing store (pass ingest.Deps from BootstrapWorkers or evobgp-all)")
|
||||
}
|
||||
hc := &http.Client{Timeout: 45 * time.Second}
|
||||
hc := httpclient.New(httpclient.DefaultTimeout)
|
||||
t := time.NewTicker(60 * time.Second)
|
||||
defer t.Stop()
|
||||
log.Printf("evobgp-ingest: active (CDN conditional GET / ETag prefetch)")
|
||||
@@ -34,7 +36,14 @@ func Run(ctx context.Context, deps *Deps) {
|
||||
log.Printf("evobgp-ingest: stopped")
|
||||
return
|
||||
case <-t.C:
|
||||
if err := pipeline.PrefetchCDNSourceETags(context.Background(), deps.Store, hc); err != nil {
|
||||
if deps.Store != nil && time.Since(lastMaintenance) > time.Hour {
|
||||
deps.Store.RunPeriodicMaintenance(ctx)
|
||||
lastMaintenance = time.Now()
|
||||
}
|
||||
prefetchCtx, cancel := context.WithTimeout(ctx, 50*time.Second)
|
||||
err := pipeline.PrefetchCDNSourceETags(prefetchCtx, deps.Store, hc)
|
||||
cancel()
|
||||
if err != nil {
|
||||
log.Printf("evobgp-ingest: prefetch: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
+49
-4
@@ -182,6 +182,8 @@ type Registry struct {
|
||||
workerStart func(j *Job)
|
||||
workerSem chan struct{}
|
||||
onTerminal func(j *Job)
|
||||
onEnqueued func(j *Job)
|
||||
onRunning func(j *Job)
|
||||
}
|
||||
|
||||
type idempoKey struct {
|
||||
@@ -209,6 +211,44 @@ func (r *Registry) SetTerminalHook(fn func(j *Job)) {
|
||||
r.onTerminal = fn
|
||||
}
|
||||
|
||||
// SetPersistHooks registers best-effort callbacks for job lifecycle persistence.
|
||||
func (r *Registry) SetPersistHooks(onEnqueued, onRunning, onTerminal func(j *Job)) {
|
||||
if r == nil {
|
||||
return
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.onEnqueued = onEnqueued
|
||||
r.onRunning = onRunning
|
||||
if onTerminal != nil {
|
||||
r.onTerminal = onTerminal
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Registry) fireEnqueued(j *Job) {
|
||||
if r == nil || j == nil {
|
||||
return
|
||||
}
|
||||
r.mu.RLock()
|
||||
fn := r.onEnqueued
|
||||
r.mu.RUnlock()
|
||||
if fn != nil {
|
||||
fn(j)
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Registry) fireRunning(j *Job) {
|
||||
if r == nil || j == nil {
|
||||
return
|
||||
}
|
||||
r.mu.RLock()
|
||||
fn := r.onRunning
|
||||
r.mu.RUnlock()
|
||||
if fn != nil {
|
||||
fn(j)
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Registry) fireTerminal(j *Job) {
|
||||
if r == nil || j == nil {
|
||||
return
|
||||
@@ -271,8 +311,6 @@ func (r *Registry) pruneTerminalIfOver(maxJobs int) {
|
||||
// Enqueue creates a job or returns an existing one for the same idempotency key.
|
||||
func (r *Registry) Enqueue(tenantID, kind string, idempotencyKey *string, moduleID *string, meta map[string]any) (*Job, bool, error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
|
||||
maxJobs := registryMaxJobsFromEnv()
|
||||
r.pruneTerminalIfOver(maxJobs)
|
||||
|
||||
@@ -281,6 +319,7 @@ func (r *Registry) Enqueue(tenantID, kind string, idempotencyKey *string, module
|
||||
if existing, ok := r.byIdempo[k]; ok {
|
||||
st := existing.statusLocked()
|
||||
if st == StatusQueued || st == StatusRunning {
|
||||
r.mu.Unlock()
|
||||
return existing, false, nil
|
||||
}
|
||||
delete(r.byIdempo, k)
|
||||
@@ -302,8 +341,14 @@ func (r *Registry) Enqueue(tenantID, kind string, idempotencyKey *string, module
|
||||
}
|
||||
r.byID[j.ID] = j
|
||||
r.pruneTerminalIfOver(maxJobs)
|
||||
enqueuedHook := r.onEnqueued
|
||||
workerStart := r.workerStart
|
||||
r.mu.Unlock()
|
||||
|
||||
if r.workerStart != nil {
|
||||
if enqueuedHook != nil {
|
||||
enqueuedHook(j)
|
||||
}
|
||||
if workerStart != nil {
|
||||
go func() {
|
||||
r.workerSem <- struct{}{}
|
||||
active := len(r.workerSem)
|
||||
@@ -313,7 +358,7 @@ func (r *Registry) Enqueue(tenantID, kind string, idempotencyKey *string, module
|
||||
<-r.workerSem
|
||||
observability.RecordJobQueueDepth(len(r.workerSem), capacity)
|
||||
}()
|
||||
r.workerStart(j)
|
||||
workerStart(j)
|
||||
}()
|
||||
}
|
||||
return j, true, nil
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
package jobs
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"evobgp/internal/pgmonitor"
|
||||
)
|
||||
|
||||
func (w *Worker) pgService() *pgmonitor.Service {
|
||||
if w == nil || w.PgPool == nil {
|
||||
return nil
|
||||
}
|
||||
return pgmonitor.NewService(w.PgPool)
|
||||
}
|
||||
|
||||
func (w *Worker) runPostgresMetricsRefresh(j *Job) {
|
||||
s := w.pgService()
|
||||
if s == nil {
|
||||
j.Fail("postgresql not configured")
|
||||
return
|
||||
}
|
||||
ctx, cancel := j.workContext()
|
||||
defer cancel()
|
||||
if err := s.RefreshMetricsSnapshot(ctx); err != nil {
|
||||
j.Fail(err.Error())
|
||||
return
|
||||
}
|
||||
j.Succeed()
|
||||
}
|
||||
|
||||
func (w *Worker) runPostgresSlowQueryAgg(j *Job) {
|
||||
s := w.pgService()
|
||||
if s == nil {
|
||||
j.Fail("postgresql not configured")
|
||||
return
|
||||
}
|
||||
ctx, cancel := j.workContext()
|
||||
defer cancel()
|
||||
if err := s.AggregateSlowQueries(ctx, 30); err != nil {
|
||||
j.Fail(err.Error())
|
||||
return
|
||||
}
|
||||
j.Succeed()
|
||||
}
|
||||
|
||||
func (w *Worker) runPostgresTableBloat(j *Job) {
|
||||
s := w.pgService()
|
||||
if s == nil {
|
||||
j.Fail("postgresql not configured")
|
||||
return
|
||||
}
|
||||
ctx, cancel := j.workContext()
|
||||
defer cancel()
|
||||
if err := s.EstimateTableBloat(ctx); err != nil {
|
||||
j.Fail(err.Error())
|
||||
return
|
||||
}
|
||||
j.Succeed()
|
||||
}
|
||||
|
||||
func (w *Worker) runPostgresIndexUsage(j *Job) {
|
||||
s := w.pgService()
|
||||
if s == nil {
|
||||
j.Fail("postgresql not configured")
|
||||
return
|
||||
}
|
||||
ctx, cancel := j.workContext()
|
||||
defer cancel()
|
||||
if err := s.AnalyzeIndexUsage(ctx); err != nil {
|
||||
j.Fail(err.Error())
|
||||
return
|
||||
}
|
||||
j.Succeed()
|
||||
}
|
||||
|
||||
func (w *Worker) runPostgresAutovacuumLag(j *Job) {
|
||||
s := w.pgService()
|
||||
if s == nil {
|
||||
j.Fail("postgresql not configured")
|
||||
return
|
||||
}
|
||||
ctx, cancel := j.workContext()
|
||||
defer cancel()
|
||||
if err := s.DetectAutovacuumLag(ctx); err != nil {
|
||||
j.Fail(err.Error())
|
||||
return
|
||||
}
|
||||
j.Succeed()
|
||||
}
|
||||
|
||||
func (w *Worker) runPostgresMaint(j *Job, kind string) {
|
||||
if w == nil || w.PgPool == nil {
|
||||
j.Fail("postgresql not configured")
|
||||
return
|
||||
}
|
||||
table, _ := j.Meta["table"].(string)
|
||||
dryRun, _ := j.Meta["dry_run"].(bool)
|
||||
actor, _ := j.Meta["actor_prefix"].(string)
|
||||
ctx, cancel := j.workContext()
|
||||
defer cancel()
|
||||
auditID, _ := pgmonitor.InsertMaintenanceAudit(ctx, w.PgPool, j.TenantID, actor, kind, table, dryRun)
|
||||
detail, err := pgmonitor.ExecMaintenance(ctx, w.PgPool, kind, table, dryRun)
|
||||
var errMsg *string
|
||||
status := StatusSucceeded
|
||||
if err != nil {
|
||||
s := err.Error()
|
||||
errMsg = &s
|
||||
status = StatusFailed
|
||||
j.Fail(s)
|
||||
} else {
|
||||
j.mergeMeta(map[string]any{"maintenance": detail, "audit_id": auditID})
|
||||
j.Succeed()
|
||||
}
|
||||
if auditID != "" {
|
||||
_ = pgmonitor.FinishMaintenanceAudit(ctx, w.PgPool, auditID, status, detail, errMsg)
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Worker) runPostgresCleanup(j *Job) {
|
||||
if w == nil || w.PgPool == nil {
|
||||
j.Fail("postgresql not configured")
|
||||
return
|
||||
}
|
||||
policy, _ := j.Meta["policy"].(string)
|
||||
dryRun, _ := j.Meta["dry_run"].(bool)
|
||||
limit := 0
|
||||
if v, ok := j.Meta["limit"].(float64); ok {
|
||||
limit = int(v)
|
||||
}
|
||||
actor, _ := j.Meta["actor_prefix"].(string)
|
||||
ctx, cancel := j.workContext()
|
||||
defer cancel()
|
||||
auditID, _ := pgmonitor.InsertMaintenanceAudit(ctx, w.PgPool, j.TenantID, actor, "cleanup", policy, dryRun)
|
||||
detail, err := pgmonitor.RunCleanup(ctx, w.PgPool, strings.TrimSpace(policy), dryRun, limit)
|
||||
var errMsg *string
|
||||
status := StatusSucceeded
|
||||
if err != nil {
|
||||
s := err.Error()
|
||||
errMsg = &s
|
||||
status = StatusFailed
|
||||
j.Fail(s)
|
||||
} else {
|
||||
j.mergeMeta(map[string]any{"cleanup": detail, "audit_id": auditID})
|
||||
j.Succeed()
|
||||
}
|
||||
if auditID != "" {
|
||||
_ = pgmonitor.FinishMaintenanceAudit(ctx, w.PgPool, auditID, status, detail, errMsg)
|
||||
}
|
||||
}
|
||||
|
||||
// EnqueuePostgresAnalyzerJobs enqueues periodic analyzer jobs (global tenant id).
|
||||
func EnqueuePostgresAnalyzerJobs(reg *Registry, tenantID string) {
|
||||
if reg == nil || tenantID == "" {
|
||||
return
|
||||
}
|
||||
kinds := []string{
|
||||
KindPostgresMetricsRefresh,
|
||||
KindPostgresSlowQueryAgg,
|
||||
KindPostgresTableBloat,
|
||||
KindPostgresIndexUsage,
|
||||
KindPostgresAutovacuumLag,
|
||||
}
|
||||
for _, k := range kinds {
|
||||
key := fmt.Sprintf("pgmon-%s-%s", k, tenantID)
|
||||
idem := key
|
||||
_, _, _ = reg.Enqueue(tenantID, k, &idem, nil, map[string]any{"trigger": "scheduler"})
|
||||
}
|
||||
}
|
||||
+44
-7
@@ -13,10 +13,13 @@ import (
|
||||
|
||||
"evobgp/internal/birddeploy"
|
||||
"evobgp/internal/birdfmt"
|
||||
"evobgp/internal/httpclient"
|
||||
"evobgp/internal/nodedispatch"
|
||||
"evobgp/internal/observability"
|
||||
"evobgp/internal/pipeline"
|
||||
"evobgp/internal/store"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// mergeBirdPostApplyMeta attaches a birdc snapshot after deploy/reload (best-effort).
|
||||
@@ -42,17 +45,28 @@ func mergeBirdPostApplyMeta(j *Job) {
|
||||
}
|
||||
|
||||
const (
|
||||
KindModuleRefresh = "module_refresh"
|
||||
KindTenantRefresh = "tenant_refresh"
|
||||
KindPeerReconcile = "peer_reconcile"
|
||||
KindDeployApply = "deploy_apply"
|
||||
KindRevisionRollback = "revision_rollback"
|
||||
KindBirdReload = "bird_reload"
|
||||
KindModuleRefresh = "module_refresh"
|
||||
KindTenantRefresh = "tenant_refresh"
|
||||
KindPeerReconcile = "peer_reconcile"
|
||||
KindDeployApply = "deploy_apply"
|
||||
KindRevisionRollback = "revision_rollback"
|
||||
KindBirdReload = "bird_reload"
|
||||
KindPostgresMetricsRefresh = "postgres_metrics_refresh"
|
||||
KindPostgresSlowQueryAgg = "postgres_slow_query_aggregate"
|
||||
KindPostgresTableBloat = "postgres_table_bloat_estimate"
|
||||
KindPostgresIndexUsage = "postgres_index_usage_analyze"
|
||||
KindPostgresAutovacuumLag = "postgres_autovacuum_lag_detect"
|
||||
KindPostgresVacuum = "postgres_vacuum"
|
||||
KindPostgresVacuumAnalyze = "postgres_vacuum_analyze"
|
||||
KindPostgresAnalyze = "postgres_analyze"
|
||||
KindPostgresReindex = "postgres_reindex"
|
||||
KindPostgresCleanup = "postgres_cleanup"
|
||||
)
|
||||
|
||||
// Worker executes queued jobs against store.Backend (memory or SQL).
|
||||
type Worker struct {
|
||||
Store store.Backend
|
||||
PgPool *pgxpool.Pool
|
||||
HTTPClient *http.Client // optional; CDN refresh uses this (default 45s timeout).
|
||||
// Registry is set after BootstrapWorkers creates the job queue; used to chain deploy_apply after refresh/rollback.
|
||||
Registry *Registry
|
||||
@@ -70,7 +84,7 @@ type revisionLogEntry struct {
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
var defaultWorkerHTTP = &http.Client{Timeout: 45 * time.Second}
|
||||
var defaultWorkerHTTP = httpclient.New(httpclient.DefaultTimeout)
|
||||
|
||||
func (w *Worker) httpClient() *http.Client {
|
||||
if w != nil && w.HTTPClient != nil {
|
||||
@@ -94,6 +108,9 @@ func (w *Worker) Process(j *Job) {
|
||||
return
|
||||
}
|
||||
j.MarkRunning()
|
||||
if w != nil && w.Registry != nil {
|
||||
w.Registry.fireRunning(j)
|
||||
}
|
||||
if j.IsCancelRequested() {
|
||||
j.MarkCancelled()
|
||||
return
|
||||
@@ -151,6 +168,26 @@ func (w *Worker) Process(j *Job) {
|
||||
}
|
||||
mergeBirdPostApplyMeta(j)
|
||||
j.Succeed()
|
||||
case KindPostgresMetricsRefresh:
|
||||
w.runPostgresMetricsRefresh(j)
|
||||
case KindPostgresSlowQueryAgg:
|
||||
w.runPostgresSlowQueryAgg(j)
|
||||
case KindPostgresTableBloat:
|
||||
w.runPostgresTableBloat(j)
|
||||
case KindPostgresIndexUsage:
|
||||
w.runPostgresIndexUsage(j)
|
||||
case KindPostgresAutovacuumLag:
|
||||
w.runPostgresAutovacuumLag(j)
|
||||
case KindPostgresVacuum:
|
||||
w.runPostgresMaint(j, "vacuum")
|
||||
case KindPostgresVacuumAnalyze:
|
||||
w.runPostgresMaint(j, "vacuum_analyze")
|
||||
case KindPostgresAnalyze:
|
||||
w.runPostgresMaint(j, "analyze")
|
||||
case KindPostgresReindex:
|
||||
w.runPostgresMaint(j, "reindex")
|
||||
case KindPostgresCleanup:
|
||||
w.runPostgresCleanup(j)
|
||||
default:
|
||||
j.Fail("unknown job kind")
|
||||
}
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
|
||||
"evobgp/internal/birdfmt"
|
||||
"evobgp/internal/bundle"
|
||||
"evobgp/internal/httpclient"
|
||||
"evobgp/internal/signing"
|
||||
)
|
||||
|
||||
@@ -55,6 +56,10 @@ func CmdPullBundle(args []string) int {
|
||||
return 0
|
||||
}
|
||||
|
||||
func nodeHTTPClient() *http.Client {
|
||||
return httpclient.New(60 * time.Second)
|
||||
}
|
||||
|
||||
func fetchLatestRevision(base, token, speaker string) (string, error) {
|
||||
u := strings.TrimRight(base, "/") + "/v1/speakers/" + speaker + "/revisions/latest"
|
||||
req, err := http.NewRequest(http.MethodGet, u, nil)
|
||||
@@ -62,7 +67,9 @@ func fetchLatestRevision(base, token, speaker string) (string, error) {
|
||||
return "", err
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
resp, err := httpclient.DoWithRetry(ctx, nodeHTTPClient(), req, 3)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
@@ -90,7 +97,9 @@ func fetchBundle(base, token, speaker, revision string) ([]byte, error) {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second)
|
||||
defer cancel()
|
||||
resp, err := httpclient.DoWithRetry(ctx, nodeHTTPClient(), req, 3)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
package nodedispatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"evobgp/internal/birdfmt"
|
||||
"evobgp/internal/store"
|
||||
)
|
||||
|
||||
// BirdProtocolsResult is agent birdc scrape outcome.
|
||||
type BirdProtocolsResult struct {
|
||||
SpeakerID string `json:"speaker_id,omitempty"`
|
||||
Sessions []birdfmt.BGPSession `json:"sessions"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// FetchBirdProtocols GETs /v1/agent/bird/protocols on a replica agent.
|
||||
func FetchBirdProtocols(ctx context.Context, sp *store.Speaker, opts Options) BirdProtocolsResult {
|
||||
res := BirdProtocolsResult{}
|
||||
if sp != nil {
|
||||
res.SpeakerID = sp.ID
|
||||
}
|
||||
if sp == nil {
|
||||
res.Error = "nil speaker"
|
||||
return res
|
||||
}
|
||||
meta := store.ParseSpeakerMeta(sp.MetaJSON)
|
||||
url := store.AgentBirdProtocolsURL(meta)
|
||||
if url == "" {
|
||||
res.Error = "agent_domain not configured"
|
||||
return res
|
||||
}
|
||||
secret := strings.TrimSpace(meta.AgentSecret)
|
||||
if secret == "" {
|
||||
res.Error = "agent_secret missing"
|
||||
return res
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
res.Error = err.Error()
|
||||
return res
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+secret)
|
||||
resp, err := opts.client().Do(req)
|
||||
if err != nil {
|
||||
res.Error = err.Error()
|
||||
return res
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
res.Error = fmt.Sprintf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(b)))
|
||||
return res
|
||||
}
|
||||
var out struct {
|
||||
Sessions []birdfmt.BGPSession `json:"sessions"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &out); err != nil {
|
||||
res.Error = err.Error()
|
||||
return res
|
||||
}
|
||||
res.Sessions = out.Sessions
|
||||
return res
|
||||
}
|
||||
@@ -156,33 +156,14 @@ func WakeReplicas(ctx context.Context, st store.Backend, tenantID, revisionID st
|
||||
return out
|
||||
}
|
||||
|
||||
// CheckHealth GETs /v1/agent/health for UI Connected/Offline status.
|
||||
// CheckHealth is deprecated; use FetchAgentHealth.
|
||||
func CheckHealth(ctx context.Context, sp *store.Speaker, opts Options) (ok bool, detail string) {
|
||||
if sp == nil {
|
||||
return false, "nil speaker"
|
||||
}
|
||||
meta := store.ParseSpeakerMeta(sp.MetaJSON)
|
||||
url := store.AgentHealthURL(meta)
|
||||
if url == "" {
|
||||
return false, "agent_domain not configured"
|
||||
}
|
||||
secret := strings.TrimSpace(meta.AgentSecret)
|
||||
if secret == "" {
|
||||
return false, "agent_secret missing"
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return false, err.Error()
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+secret)
|
||||
resp, err := opts.client().Do(req)
|
||||
if err != nil {
|
||||
return false, err.Error()
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
|
||||
res := FetchAgentHealth(ctx, sp, opts)
|
||||
if res.OK {
|
||||
return true, "connected"
|
||||
}
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
return false, fmt.Sprintf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(b)))
|
||||
if res.Error != "" {
|
||||
return false, res.Error
|
||||
}
|
||||
return false, "agent unhealthy"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
package nodedispatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"evobgp/internal/store"
|
||||
)
|
||||
|
||||
// AgentHealthResult is the parsed outcome of GET /v1/agent/health on a replica.
|
||||
type AgentHealthResult struct {
|
||||
OK bool
|
||||
Error string
|
||||
LastAppliedRevisionID string
|
||||
LastSyncAt string
|
||||
}
|
||||
|
||||
// FetchAgentHealth GETs /v1/agent/health for UI Connected/Offline status.
|
||||
func FetchAgentHealth(ctx context.Context, sp *store.Speaker, opts Options) AgentHealthResult {
|
||||
if sp == nil {
|
||||
return AgentHealthResult{Error: "nil speaker"}
|
||||
}
|
||||
meta := store.ParseSpeakerMeta(sp.MetaJSON)
|
||||
url := store.AgentHealthURL(meta)
|
||||
if url == "" {
|
||||
return AgentHealthResult{Error: "agent_domain not configured"}
|
||||
}
|
||||
secret := strings.TrimSpace(meta.AgentSecret)
|
||||
if secret == "" {
|
||||
return AgentHealthResult{Error: "agent_secret missing"}
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return AgentHealthResult{Error: err.Error()}
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+secret)
|
||||
resp, err := opts.client().Do(req)
|
||||
if err != nil {
|
||||
return AgentHealthResult{Error: err.Error()}
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return AgentHealthResult{
|
||||
Error: fmt.Sprintf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(b))),
|
||||
}
|
||||
}
|
||||
var out struct {
|
||||
OK bool `json:"ok"`
|
||||
LastAppliedRevisionID string `json:"last_applied_revision_id"`
|
||||
LastSyncAt string `json:"last_sync_at"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &out); err != nil {
|
||||
return AgentHealthResult{Error: err.Error()}
|
||||
}
|
||||
res := AgentHealthResult{
|
||||
OK: out.OK,
|
||||
LastAppliedRevisionID: strings.TrimSpace(out.LastAppliedRevisionID),
|
||||
LastSyncAt: strings.TrimSpace(out.LastSyncAt),
|
||||
}
|
||||
if !res.OK {
|
||||
res.Error = "agent reported ok=false"
|
||||
}
|
||||
return res
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
package pgmonitor
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
type cacheEntry struct {
|
||||
at time.Time
|
||||
data any
|
||||
}
|
||||
|
||||
type ttlCache struct {
|
||||
mu sync.RWMutex
|
||||
ttl time.Duration
|
||||
items map[string]cacheEntry
|
||||
}
|
||||
|
||||
func newTTLCache(ttl time.Duration) *ttlCache {
|
||||
return &ttlCache{ttl: ttl, items: make(map[string]cacheEntry)}
|
||||
}
|
||||
|
||||
func (c *ttlCache) get(key string) (any, bool) {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
e, ok := c.items[key]
|
||||
if !ok || time.Since(e.at) > c.ttl {
|
||||
return nil, false
|
||||
}
|
||||
return e.data, true
|
||||
}
|
||||
|
||||
func (c *ttlCache) set(key string, data any) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.items[key] = cacheEntry{at: time.Now().UTC(), data: data}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package pgmonitor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// Correlation builds aligned timeline points from job_audit and overview cache.
|
||||
func (s *Service) Correlation(ctx context.Context, windowMinutes int) (CorrelationResponse, error) {
|
||||
if s == nil || s.pool == nil {
|
||||
return CorrelationResponse{}, fmt.Errorf("pgmonitor: postgres not configured")
|
||||
}
|
||||
if windowMinutes <= 0 {
|
||||
windowMinutes = 60
|
||||
}
|
||||
if windowMinutes > 1440 {
|
||||
windowMinutes = 1440
|
||||
}
|
||||
since := time.Now().UTC().Add(-time.Duration(windowMinutes) * time.Minute)
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT date_trunc('minute', finished_at) AS bucket,
|
||||
percentile_cont(0.99) WITHIN GROUP (ORDER BY
|
||||
EXTRACT(EPOCH FROM (finished_at - started_at)) * 1000)
|
||||
FROM job_audit
|
||||
WHERE finished_at >= $1 AND kind IN ('module_refresh', 'tenant_refresh')
|
||||
AND status = 'succeeded' AND started_at IS NOT NULL
|
||||
GROUP BY 1
|
||||
ORDER BY 1`, since)
|
||||
if err != nil {
|
||||
return CorrelationResponse{}, fmt.Errorf("pgmonitor: correlation jobs: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
points := make(map[time.Time]*CorrelationPoint)
|
||||
for rows.Next() {
|
||||
var bucket time.Time
|
||||
var p99 *float64
|
||||
if err := rows.Scan(&bucket, &p99); err != nil {
|
||||
return CorrelationResponse{}, err
|
||||
}
|
||||
bucket = bucket.UTC()
|
||||
pt := points[bucket]
|
||||
if pt == nil {
|
||||
pt = &CorrelationPoint{Timestamp: bucket}
|
||||
points[bucket] = pt
|
||||
}
|
||||
if p99 != nil {
|
||||
pt.PipelineRefreshP99Ms = *p99
|
||||
}
|
||||
}
|
||||
|
||||
ov, err := s.Overview(ctx)
|
||||
if err == nil && ov.Database.CacheHitPct > 0 {
|
||||
now := time.Now().UTC().Truncate(time.Minute)
|
||||
pt := points[now]
|
||||
if pt == nil {
|
||||
pt = &CorrelationPoint{Timestamp: now}
|
||||
points[now] = pt
|
||||
}
|
||||
pt.CacheHitPct = ov.Database.CacheHitPct
|
||||
}
|
||||
|
||||
out := make([]CorrelationPoint, 0, len(points))
|
||||
for _, p := range points {
|
||||
out = append(out, *p)
|
||||
}
|
||||
// simple sort by time
|
||||
for i := 0; i < len(out); i++ {
|
||||
for j := i + 1; j < len(out); j++ {
|
||||
if out[j].Timestamp.Before(out[i].Timestamp) {
|
||||
out[i], out[j] = out[j], out[i]
|
||||
}
|
||||
}
|
||||
}
|
||||
return CorrelationResponse{WindowMinutes: windowMinutes, Points: out}, nil
|
||||
}
|
||||
|
||||
// RecordCorrelationSnapshot is a hook for future Prometheus samples (no-op placeholder).
|
||||
func RecordCorrelationSnapshot(_ *pgxpool.Pool) {}
|
||||
@@ -0,0 +1,154 @@
|
||||
package pgmonitor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// CleanupPolicy names safe retention policies.
|
||||
type CleanupPolicy string
|
||||
|
||||
const (
|
||||
PolicyJobAuditRetention CleanupPolicy = "job_audit_retention"
|
||||
PolicyASNCacheRetention CleanupPolicy = "asn_cache_retention"
|
||||
)
|
||||
|
||||
// CleanupRequest for POST /postgres/cleanup.
|
||||
type CleanupRequest struct {
|
||||
Policy string `json:"policy"`
|
||||
DryRun bool `json:"dry_run"`
|
||||
Limit int `json:"limit"`
|
||||
}
|
||||
|
||||
// RunCleanup executes a named retention policy.
|
||||
func RunCleanup(ctx context.Context, pool *pgxpool.Pool, policy string, dryRun bool, limit int) (map[string]any, error) {
|
||||
if pool == nil {
|
||||
return nil, fmt.Errorf("pgmonitor: postgres not configured")
|
||||
}
|
||||
if limit <= 0 {
|
||||
limit = 10000
|
||||
}
|
||||
if limit > 100000 {
|
||||
limit = 100000
|
||||
}
|
||||
detail := map[string]any{"policy": policy, "dry_run": dryRun, "limit": limit}
|
||||
switch CleanupPolicy(policy) {
|
||||
case PolicyJobAuditRetention:
|
||||
cutoff := time.Now().UTC().Add(-90 * 24 * time.Hour)
|
||||
if dryRun {
|
||||
var n int64
|
||||
err := pool.QueryRow(ctx, `
|
||||
SELECT count(*) FROM job_audit
|
||||
WHERE created_at < $1 AND status IN ('succeeded', 'failed', 'cancelled')`, cutoff).Scan(&n)
|
||||
detail["would_delete"] = n
|
||||
return detail, err
|
||||
}
|
||||
tag, err := pool.Exec(ctx, `
|
||||
DELETE FROM job_audit
|
||||
WHERE id IN (
|
||||
SELECT id FROM job_audit
|
||||
WHERE created_at < $1 AND status IN ('succeeded', 'failed', 'cancelled')
|
||||
LIMIT $2
|
||||
)`, cutoff, limit)
|
||||
if err != nil {
|
||||
return detail, err
|
||||
}
|
||||
detail["deleted"] = tag.RowsAffected()
|
||||
return detail, nil
|
||||
case PolicyASNCacheRetention:
|
||||
cutoff := time.Now().UTC().Add(-7 * 24 * time.Hour)
|
||||
if dryRun {
|
||||
var n int64
|
||||
err := pool.QueryRow(ctx, `SELECT count(*) FROM asn_prefix_cache WHERE fetched_at < $1`, cutoff).Scan(&n)
|
||||
detail["would_delete"] = n
|
||||
return detail, err
|
||||
}
|
||||
tag, err := pool.Exec(ctx, `
|
||||
DELETE FROM asn_prefix_cache WHERE fetched_at < $1`, cutoff)
|
||||
if err != nil {
|
||||
return detail, err
|
||||
}
|
||||
detail["deleted"] = tag.RowsAffected()
|
||||
return detail, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("pgmonitor: unknown cleanup policy %q", policy)
|
||||
}
|
||||
}
|
||||
|
||||
// InsertMaintenanceAudit records an audit row at job start.
|
||||
func InsertMaintenanceAudit(ctx context.Context, pool *pgxpool.Pool, tenantID, actorPrefix, kind, table string, dryRun bool) (string, error) {
|
||||
id := uuid.New().String()
|
||||
_, err := pool.Exec(ctx, `
|
||||
INSERT INTO postgres_maintenance_audit
|
||||
(id, tenant_id, actor_prefix, kind, target_table, dry_run, status, created_at)
|
||||
VALUES ($1, NULLIF($2,''), NULLIF($3,''), $4, NULLIF($5,''), $6, 'running', now())`,
|
||||
id, tenantID, actorPrefix, kind, table, dryRun)
|
||||
return id, err
|
||||
}
|
||||
|
||||
// FinishMaintenanceAudit updates terminal state.
|
||||
func FinishMaintenanceAudit(ctx context.Context, pool *pgxpool.Pool, id, status string, detail map[string]any, errMsg *string) error {
|
||||
var detailJSON []byte
|
||||
if detail != nil {
|
||||
detailJSON, _ = json.Marshal(detail)
|
||||
}
|
||||
_, err := pool.Exec(ctx, `
|
||||
UPDATE postgres_maintenance_audit
|
||||
SET status = $2, detail_json = $3::jsonb, error_message = $4,
|
||||
finished_at = now(), started_at = COALESCE(started_at, now())
|
||||
WHERE id = $1`,
|
||||
id, status, string(detailJSON), errMsg)
|
||||
return err
|
||||
}
|
||||
|
||||
// ListMaintenanceLogs returns paginated audit rows.
|
||||
func ListMaintenanceLogs(ctx context.Context, pool *pgxpool.Pool, cursor string, limit int) ([]MaintenanceLogRow, string, bool, error) {
|
||||
limit = clampLimit(limit, 20, 100)
|
||||
args := []any{limit + 1}
|
||||
q := `
|
||||
SELECT id, COALESCE(tenant_id,''), COALESCE(actor_prefix,''), kind,
|
||||
COALESCE(target_table,''), dry_run, status,
|
||||
detail_json, COALESCE(error_message,''), created_at, started_at, finished_at
|
||||
FROM postgres_maintenance_audit`
|
||||
if cursor != "" {
|
||||
q += ` WHERE created_at < (SELECT created_at FROM postgres_maintenance_audit WHERE id = $2)`
|
||||
args = append(args, cursor)
|
||||
}
|
||||
q += ` ORDER BY created_at DESC LIMIT $1`
|
||||
|
||||
rows, err := pool.Query(ctx, q, args...)
|
||||
if err != nil {
|
||||
return nil, "", false, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []MaintenanceLogRow
|
||||
for rows.Next() {
|
||||
var r MaintenanceLogRow
|
||||
var detailRaw []byte
|
||||
var started, finished *time.Time
|
||||
if err := rows.Scan(&r.ID, &r.TenantID, &r.ActorPrefix, &r.Kind, &r.TargetTable,
|
||||
&r.DryRun, &r.Status, &detailRaw, &r.Error, &r.CreatedAt, &started, &finished); err != nil {
|
||||
return nil, "", false, err
|
||||
}
|
||||
r.StartedAt = started
|
||||
r.FinishedAt = finished
|
||||
if len(detailRaw) > 0 {
|
||||
_ = json.Unmarshal(detailRaw, &r.Detail)
|
||||
}
|
||||
out = append(out, r)
|
||||
}
|
||||
hasMore := len(out) > limit
|
||||
if hasMore {
|
||||
out = out[:limit]
|
||||
}
|
||||
next := ""
|
||||
if hasMore && len(out) > 0 {
|
||||
next = out[len(out)-1].ID
|
||||
}
|
||||
return out, next, hasMore, rows.Err()
|
||||
}
|
||||
@@ -0,0 +1,350 @@
|
||||
package pgmonitor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
func clampLimit(limit, def, max int) int {
|
||||
if limit <= 0 {
|
||||
return def
|
||||
}
|
||||
if limit > max {
|
||||
return max
|
||||
}
|
||||
return limit
|
||||
}
|
||||
|
||||
func (s *Service) fetchOverview(ctx context.Context) (Overview, error) {
|
||||
now := time.Now().UTC()
|
||||
out := Overview{CollectedAt: now}
|
||||
|
||||
var active, idle, total, maxConn int
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
SELECT
|
||||
count(*) FILTER (WHERE state = 'active'),
|
||||
count(*) FILTER (WHERE state = 'idle'),
|
||||
count(*),
|
||||
(SELECT setting::int FROM pg_settings WHERE name = 'max_connections')
|
||||
FROM pg_stat_activity
|
||||
WHERE datname = current_database()`).Scan(&active, &idle, &total, &maxConn)
|
||||
if err != nil {
|
||||
return out, fmt.Errorf("pgmonitor: connections: %w", err)
|
||||
}
|
||||
out.Connections = Connections{Active: active, Idle: idle, Total: total, MaxConnections: maxConn}
|
||||
|
||||
var cachePct *float64
|
||||
err = s.pool.QueryRow(ctx, `
|
||||
SELECT numbackends, xact_commit, xact_rollback, deadlocks, blks_hit, blks_read,
|
||||
CASE WHEN blks_hit + blks_read > 0
|
||||
THEN round(100.0 * blks_hit::numeric / (blks_hit + blks_read), 2) END
|
||||
FROM pg_stat_database WHERE datname = current_database()`).Scan(
|
||||
&out.Database.Backends,
|
||||
&out.Database.XactCommit,
|
||||
&out.Database.XactRollback,
|
||||
&out.Database.Deadlocks,
|
||||
&out.Database.BlksHit,
|
||||
&out.Database.BlksRead,
|
||||
&cachePct,
|
||||
)
|
||||
if err != nil {
|
||||
return out, fmt.Errorf("pgmonitor: database stats: %w", err)
|
||||
}
|
||||
if cachePct != nil {
|
||||
out.Database.CacheHitPct = *cachePct
|
||||
}
|
||||
|
||||
_ = s.pool.QueryRow(ctx, `
|
||||
SELECT checkpoints_timed, checkpoints_req, buffers_checkpoint, buffers_clean,
|
||||
maxwritten_clean, buffers_backend, buffers_alloc
|
||||
FROM pg_stat_bgwriter`).Scan(
|
||||
&out.Bgwriter.CheckpointsTimed,
|
||||
&out.Bgwriter.CheckpointsReq,
|
||||
&out.Bgwriter.BuffersCheckpoint,
|
||||
&out.Bgwriter.BuffersClean,
|
||||
&out.Bgwriter.MaxWrittenClean,
|
||||
&out.Bgwriter.BuffersBackend,
|
||||
&out.Bgwriter.BuffersAlloc,
|
||||
)
|
||||
|
||||
_ = s.pool.QueryRow(ctx, `SELECT pg_database_size(current_database())`).Scan(&out.SizeBytes)
|
||||
|
||||
_ = s.pool.QueryRow(ctx, `
|
||||
SELECT
|
||||
(SELECT setting FROM pg_settings WHERE name = 'shared_buffers'),
|
||||
(SELECT setting FROM pg_settings WHERE name = 'work_mem'),
|
||||
(SELECT setting FROM pg_settings WHERE name = 'effective_cache_size')`).Scan(
|
||||
&out.MemorySettings.SharedBuffers,
|
||||
&out.MemorySettings.WorkMem,
|
||||
&out.MemorySettings.EffectiveCacheSize,
|
||||
)
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT client_addr::text, state, sync_state,
|
||||
EXTRACT(EPOCH FROM COALESCE(write_lag, flush_lag, replay_lag)) * 1000
|
||||
FROM pg_stat_replication`)
|
||||
if err == nil {
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var peer ReplicationPeer
|
||||
var lagMs *float64
|
||||
if err := rows.Scan(&peer.ClientAddr, &peer.State, &peer.SyncState, &lagMs); err != nil {
|
||||
continue
|
||||
}
|
||||
if lagMs != nil {
|
||||
v := int64(*lagMs)
|
||||
peer.LagMs = &v
|
||||
}
|
||||
out.Replication = append(out.Replication, peer)
|
||||
}
|
||||
}
|
||||
|
||||
out.StatementsEnabled = s.statementsQueryable(ctx)
|
||||
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func queryLocks(ctx context.Context, pool *pgxpool.Pool) ([]LockRow, error) {
|
||||
rows, err := pool.Query(ctx, `
|
||||
SELECT l.locktype, l.mode, l.granted, a.pid, COALESCE(a.usename, ''),
|
||||
COALESCE(a.state, ''), COALESCE(left(a.query, 300), ''),
|
||||
NOT l.granted AS blocked
|
||||
FROM pg_locks l
|
||||
JOIN pg_stat_activity a ON a.pid = l.pid
|
||||
WHERE a.datname = current_database()
|
||||
AND (NOT l.granted OR l.mode LIKE '%Exclusive%')
|
||||
ORDER BY l.granted ASC, a.query_start NULLS LAST
|
||||
LIMIT 200`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pgmonitor: locks: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []LockRow
|
||||
for rows.Next() {
|
||||
var r LockRow
|
||||
if err := rows.Scan(&r.Locktype, &r.Mode, &r.Granted, &r.PID, &r.User, &r.State, &r.Query, &r.Blocked); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, r)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
func queryTables(ctx context.Context, pool *pgxpool.Pool, limit int) ([]TableStat, error) {
|
||||
limit = clampLimit(limit, 20, 100)
|
||||
rows, err := pool.Query(ctx, `
|
||||
SELECT t.relname,
|
||||
pg_total_relation_size(t.relid),
|
||||
s.heap_blks_read, s.heap_blks_hit,
|
||||
t.idx_scan, t.seq_scan, t.n_dead_tup, t.last_autovacuum,
|
||||
CASE WHEN t.n_live_tup + t.n_dead_tup > 0
|
||||
THEN round(t.n_dead_tup::numeric / (t.n_live_tup + t.n_dead_tup), 4)
|
||||
ELSE 0 END
|
||||
FROM pg_statio_user_tables s
|
||||
JOIN pg_stat_user_tables t ON t.relid = s.relid
|
||||
WHERE t.schemaname = 'public'
|
||||
ORDER BY pg_total_relation_size(t.relid) DESC
|
||||
LIMIT $1`, limit)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pgmonitor: tables: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []TableStat
|
||||
for rows.Next() {
|
||||
var r TableStat
|
||||
var last *time.Time
|
||||
if err := rows.Scan(&r.Relname, &r.TotalBytes, &r.HeapBlksRead, &r.HeapBlksHit,
|
||||
&r.IdxScan, &r.SeqScan, &r.DeadTuples, &last, &r.BloatRatio); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
r.LastAutovacuum = last
|
||||
out = append(out, r)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// TopQueries loads from pg_stat_statements when available.
|
||||
func (s *Service) TopQueries(ctx context.Context, limit int) (QueriesResponse, error) {
|
||||
if s == nil || s.pool == nil {
|
||||
return QueriesResponse{}, errors.New("pgmonitor: postgres not configured")
|
||||
}
|
||||
limit = clampLimit(limit, 20, 100)
|
||||
now := time.Now().UTC()
|
||||
|
||||
if snap, ok, err := s.loadSnapshot(ctx, "slow_queries", 15*time.Minute); err == nil && ok {
|
||||
var items []QueryStat
|
||||
if err := decodePayload(snap.Payload, &items); err == nil {
|
||||
return QueriesResponse{
|
||||
CollectedAt: snap.CollectedAt,
|
||||
Source: "snapshot",
|
||||
Items: items,
|
||||
StatementsAvailable: true,
|
||||
}, nil
|
||||
}
|
||||
}
|
||||
|
||||
if !s.statementsQueryable(ctx) {
|
||||
return queriesUnavailable(now), nil
|
||||
}
|
||||
items, err := queryTopStatements(ctx, s.pool, limit)
|
||||
if err != nil {
|
||||
if isPgStatStatementsUnavailable(err) {
|
||||
s.markStatementsUnavailable()
|
||||
return queriesUnavailable(now), nil
|
||||
}
|
||||
return QueriesResponse{}, err
|
||||
}
|
||||
return QueriesResponse{
|
||||
CollectedAt: now,
|
||||
Source: "live",
|
||||
Items: items,
|
||||
StatementsAvailable: true,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func queriesUnavailable(at time.Time) QueriesResponse {
|
||||
return QueriesResponse{
|
||||
CollectedAt: at,
|
||||
Source: "unavailable",
|
||||
Items: nil,
|
||||
StatementsAvailable: false,
|
||||
StatementsHint: statementsUnavailableHint,
|
||||
}
|
||||
}
|
||||
|
||||
const statementsUnavailableHint = "pg_stat_statements requires shared_preload_libraries and PostgreSQL restart (see docs/db-diagnostics.md)"
|
||||
|
||||
// statementsQueryable returns true only when pg_stat_statements can be queried (not merely installed).
|
||||
func (s *Service) statementsQueryable(ctx context.Context) bool {
|
||||
if s == nil || s.pool == nil {
|
||||
return false
|
||||
}
|
||||
if v, ok := s.cache.get("stmt_queryable"); ok {
|
||||
if b, ok := v.(bool); ok {
|
||||
return b
|
||||
}
|
||||
}
|
||||
ok := probePgStatStatements(ctx, s.pool)
|
||||
s.cache.set("stmt_queryable", ok)
|
||||
return ok
|
||||
}
|
||||
|
||||
func (s *Service) markStatementsUnavailable() {
|
||||
s.cache.set("stmt_queryable", false)
|
||||
}
|
||||
|
||||
func probePgStatStatements(ctx context.Context, pool *pgxpool.Pool) bool {
|
||||
var dummy int64
|
||||
err := pool.QueryRow(ctx, `
|
||||
SELECT COALESCE(SUM(calls), 0)::bigint FROM pg_stat_statements LIMIT 1`).Scan(&dummy)
|
||||
if err == nil {
|
||||
return true
|
||||
}
|
||||
return !isPgStatStatementsUnavailable(err)
|
||||
}
|
||||
|
||||
func queryTopStatements(ctx context.Context, pool *pgxpool.Pool, limit int) ([]QueryStat, error) {
|
||||
rows, err := pool.Query(ctx, `
|
||||
SELECT queryid, left(query, 500), calls, total_exec_time, mean_exec_time, rows
|
||||
FROM pg_stat_statements
|
||||
WHERE dbid = (SELECT oid FROM pg_database WHERE datname = current_database())
|
||||
ORDER BY mean_exec_time DESC
|
||||
LIMIT $1`, limit)
|
||||
if err != nil {
|
||||
if isPgStatStatementsUnavailable(err) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, fmt.Errorf("pgmonitor: pg_stat_statements: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []QueryStat
|
||||
for rows.Next() {
|
||||
var r QueryStat
|
||||
if err := rows.Scan(&r.QueryID, &r.Query, &r.Calls, &r.TotalExecMs, &r.MeanExecMs, &r.Rows); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, r)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// isPgStatStatementsUnavailable reports extension missing or not loaded via shared_preload_libraries.
|
||||
func isPgStatStatementsUnavailable(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
var pgErr *pgconn.PgError
|
||||
if errors.As(err, &pgErr) {
|
||||
switch pgErr.Code {
|
||||
case "42P01", "42704", "55000":
|
||||
return true
|
||||
}
|
||||
msg := strings.ToLower(pgErr.Message)
|
||||
if strings.Contains(msg, "shared_preload_libraries") || strings.Contains(msg, "pg_stat_statements") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
low := strings.ToLower(err.Error())
|
||||
return strings.Contains(low, "shared_preload_libraries") || strings.Contains(low, "pg_stat_statements")
|
||||
}
|
||||
|
||||
func isSafeIdent(name string) bool {
|
||||
if name == "" {
|
||||
return true
|
||||
}
|
||||
for _, r := range name {
|
||||
if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '_' {
|
||||
continue
|
||||
}
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// ExecMaintenance runs VACUUM/ANALYZE/REINDEX with optional dry-run (returns SQL executed or planned).
|
||||
func ExecMaintenance(ctx context.Context, pool *pgxpool.Pool, kind, table string, dryRun bool) (detail map[string]any, err error) {
|
||||
if pool == nil {
|
||||
return nil, errors.New("pgmonitor: postgres not configured")
|
||||
}
|
||||
table = strings.TrimSpace(table)
|
||||
if table != "" && !isSafeIdent(table) {
|
||||
return nil, errors.New("pgmonitor: invalid table name")
|
||||
}
|
||||
qual := ""
|
||||
if table != "" {
|
||||
qual = " " + pgx.Identifier{table}.Sanitize()
|
||||
}
|
||||
var sql string
|
||||
switch kind {
|
||||
case "vacuum":
|
||||
sql = "VACUUM" + qual
|
||||
case "vacuum_analyze":
|
||||
sql = "VACUUM ANALYZE" + qual
|
||||
case "analyze":
|
||||
sql = "ANALYZE" + qual
|
||||
case "reindex":
|
||||
if table == "" {
|
||||
return nil, errors.New("pgmonitor: reindex requires table")
|
||||
}
|
||||
sql = "REINDEX TABLE" + qual
|
||||
default:
|
||||
return nil, fmt.Errorf("pgmonitor: unknown maintenance kind %q", kind)
|
||||
}
|
||||
detail = map[string]any{"sql": sql, "dry_run": dryRun}
|
||||
if dryRun {
|
||||
return detail, nil
|
||||
}
|
||||
_, err = pool.Exec(ctx, sql)
|
||||
if err != nil {
|
||||
return detail, fmt.Errorf("pgmonitor: %s: %w", kind, err)
|
||||
}
|
||||
detail["executed"] = true
|
||||
return detail, nil
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
package pgmonitor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Recommendations builds heuristic items from live stats and snapshots.
|
||||
func (s *Service) Recommendations(ctx context.Context) (RecommendationsResponse, error) {
|
||||
now := time.Now().UTC()
|
||||
var items []RecommendationItem
|
||||
|
||||
ov, err := s.Overview(ctx)
|
||||
if err == nil {
|
||||
if ov.Database.CacheHitPct > 0 && ov.Database.CacheHitPct < 90 {
|
||||
items = append(items, RecommendationItem{
|
||||
Severity: "warn",
|
||||
Code: "low_cache_hit",
|
||||
Title: "Низкий cache hit ratio",
|
||||
Detail: "Buffer cache hit ниже 90%; проверьте shared_buffers и горячие seq scan.",
|
||||
})
|
||||
}
|
||||
if ov.Database.Deadlocks > 0 {
|
||||
items = append(items, RecommendationItem{
|
||||
Severity: "warn",
|
||||
Code: "deadlocks",
|
||||
Title: "Зафиксированы deadlocks",
|
||||
Detail: "Проверьте конкурирующие транзакции и порядок блокировок.",
|
||||
})
|
||||
}
|
||||
if ov.Connections.MaxConnections > 0 &&
|
||||
float64(ov.Connections.Total)/float64(ov.Connections.MaxConnections) > 0.8 {
|
||||
items = append(items, RecommendationItem{
|
||||
Severity: "critical",
|
||||
Code: "connections_high",
|
||||
Title: "Много подключений к PostgreSQL",
|
||||
Detail: "Использование max_connections выше 80%; увеличьте pool tuning или лимит.",
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
tables, err := s.Tables(ctx, 30)
|
||||
if err == nil {
|
||||
for _, t := range tables {
|
||||
if t.SeqScan > 1000 && t.IdxScan < t.SeqScan/10 {
|
||||
items = append(items, RecommendationItem{
|
||||
Severity: "warn",
|
||||
Code: "missing_index",
|
||||
Title: "Высокий seq_scan",
|
||||
Detail: "Таблица часто сканируется последовательно; рассмотрите индекс.",
|
||||
Refs: []string{t.Relname},
|
||||
})
|
||||
}
|
||||
if t.BloatRatio > 0.2 && t.DeadTuples > 5000 {
|
||||
items = append(items, RecommendationItem{
|
||||
Severity: "info",
|
||||
Code: "autovacuum_lag",
|
||||
Title: "Возможный bloat / мёртвые строки",
|
||||
Detail: "Высокая доля n_dead_tup; запланируйте VACUUM.",
|
||||
Refs: []string{t.Relname},
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if snap, ok, _ := s.loadSnapshot(ctx, "unused_indexes", 30*time.Minute); ok {
|
||||
type unused struct {
|
||||
Index string `json:"index"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
}
|
||||
var list []unused
|
||||
if decodePayload(snap.Payload, &list) == nil {
|
||||
for _, u := range list {
|
||||
if u.SizeBytes < 1024*1024 {
|
||||
continue
|
||||
}
|
||||
items = append(items, RecommendationItem{
|
||||
Severity: "info",
|
||||
Code: "unused_index",
|
||||
Title: "Неиспользуемый индекс",
|
||||
Detail: "idx_scan=0; проверьте перед удалением.",
|
||||
Refs: []string{u.Index},
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
q, err := s.TopQueries(ctx, 5)
|
||||
if err == nil {
|
||||
for _, qs := range q.Items {
|
||||
if qs.MeanExecMs > 500 {
|
||||
items = append(items, RecommendationItem{
|
||||
Severity: "warn",
|
||||
Code: "slow_query",
|
||||
Title: "Медленный запрос",
|
||||
Detail: "Среднее время выполнения выше 500ms.",
|
||||
Refs: []string{qs.Query},
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return RecommendationsResponse{CollectedAt: now, Items: items}, nil
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package pgmonitor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// StartScheduler runs periodic PostgreSQL analyzer snapshots until ctx is cancelled.
|
||||
func StartScheduler(ctx context.Context, pool *pgxpool.Pool) {
|
||||
if pool == nil {
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
t5 := time.NewTicker(5 * time.Minute)
|
||||
t15 := time.NewTicker(15 * time.Minute)
|
||||
defer t5.Stop()
|
||||
defer t15.Stop()
|
||||
s := NewService(pool)
|
||||
runLight := func() {
|
||||
c, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
defer cancel()
|
||||
if err := s.RefreshMetricsSnapshot(c); err != nil {
|
||||
log.Printf("pgmonitor: metrics refresh: %v", err)
|
||||
}
|
||||
if err := s.DetectAutovacuumLag(c); err != nil {
|
||||
log.Printf("pgmonitor: autovacuum lag: %v", err)
|
||||
}
|
||||
}
|
||||
runHeavy := func() {
|
||||
c, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
|
||||
defer cancel()
|
||||
if err := s.AggregateSlowQueries(c, 30); err != nil {
|
||||
log.Printf("pgmonitor: slow queries snapshot: %v", err)
|
||||
}
|
||||
if err := s.EstimateTableBloat(c); err != nil {
|
||||
log.Printf("pgmonitor: bloat: %v", err)
|
||||
}
|
||||
if err := s.AnalyzeIndexUsage(c); err != nil {
|
||||
log.Printf("pgmonitor: index usage: %v", err)
|
||||
}
|
||||
}
|
||||
runLight()
|
||||
runHeavy()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t5.C:
|
||||
runLight()
|
||||
case <-t15.C:
|
||||
runHeavy()
|
||||
}
|
||||
}
|
||||
}()
|
||||
log.Printf("pgmonitor: scheduler started (5m light / 15m heavy)")
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package pgmonitor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// Service provides PostgreSQL observability and maintenance helpers (control plane instance scope).
|
||||
type Service struct {
|
||||
pool *pgxpool.Pool
|
||||
cache *ttlCache
|
||||
}
|
||||
|
||||
// NewService constructs a metrics service for the API PostgreSQL pool.
|
||||
func NewService(pool *pgxpool.Pool) *Service {
|
||||
if pool == nil {
|
||||
return nil
|
||||
}
|
||||
return &Service{
|
||||
pool: pool,
|
||||
cache: newTTLCache(10 * time.Second),
|
||||
}
|
||||
}
|
||||
|
||||
// Pool exposes the underlying pool for job workers.
|
||||
func (s *Service) Pool() *pgxpool.Pool {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
return s.pool
|
||||
}
|
||||
|
||||
// Overview returns cached instance-level stats.
|
||||
func (s *Service) Overview(ctx context.Context) (Overview, error) {
|
||||
if s == nil || s.pool == nil {
|
||||
return Overview{}, errors.New("pgmonitor: postgres not configured")
|
||||
}
|
||||
if v, ok := s.cache.get("overview"); ok {
|
||||
if o, ok := v.(Overview); ok {
|
||||
return o, nil
|
||||
}
|
||||
}
|
||||
o, err := s.fetchOverview(ctx)
|
||||
if err != nil {
|
||||
return Overview{}, err
|
||||
}
|
||||
s.cache.set("overview", o)
|
||||
return o, nil
|
||||
}
|
||||
|
||||
// Locks returns active / blocking locks.
|
||||
func (s *Service) Locks(ctx context.Context) ([]LockRow, error) {
|
||||
if s == nil || s.pool == nil {
|
||||
return nil, errors.New("pgmonitor: postgres not configured")
|
||||
}
|
||||
if v, ok := s.cache.get("locks"); ok {
|
||||
if rows, ok := v.([]LockRow); ok {
|
||||
return rows, nil
|
||||
}
|
||||
}
|
||||
rows, err := queryLocks(ctx, s.pool)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
s.cache.set("locks", rows)
|
||||
return rows, nil
|
||||
}
|
||||
|
||||
// Tables returns top tables by size with I/O stats.
|
||||
func (s *Service) Tables(ctx context.Context, limit int) ([]TableStat, error) {
|
||||
if s == nil || s.pool == nil {
|
||||
return nil, errors.New("pgmonitor: postgres not configured")
|
||||
}
|
||||
key := fmt.Sprintf("tables:%d", limit)
|
||||
if v, ok := s.cache.get(key); ok {
|
||||
if rows, ok := v.([]TableStat); ok {
|
||||
return rows, nil
|
||||
}
|
||||
}
|
||||
rows, err := queryTables(ctx, s.pool, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
s.cache.set(key, rows)
|
||||
return rows, nil
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
package pgmonitor
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
)
|
||||
|
||||
func TestClampLimit(t *testing.T) {
|
||||
if clampLimit(0, 20, 100) != 20 {
|
||||
t.Fatal("default")
|
||||
}
|
||||
if clampLimit(200, 20, 100) != 100 {
|
||||
t.Fatal("max")
|
||||
}
|
||||
if clampLimit(5, 20, 100) != 5 {
|
||||
t.Fatal("value")
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsSafeIdent(t *testing.T) {
|
||||
if !isSafeIdent("revision_materialized_prefix") {
|
||||
t.Fatal("valid")
|
||||
}
|
||||
if isSafeIdent("bad-name") {
|
||||
t.Fatal("invalid")
|
||||
}
|
||||
if !isSafeIdent("") {
|
||||
t.Fatal("empty ok")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewServiceNilPool(t *testing.T) {
|
||||
if NewService(nil) != nil {
|
||||
t.Fatal("expected nil service")
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsPgStatStatementsUnavailable(t *testing.T) {
|
||||
err := &pgconn.PgError{Code: "55000", Message: "pg_stat_statements must be loaded via shared_preload_libraries"}
|
||||
if !isPgStatStatementsUnavailable(err) {
|
||||
t.Fatal("55000")
|
||||
}
|
||||
if isPgStatStatementsUnavailable(errors.New("other")) {
|
||||
t.Fatal("unrelated")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
package pgmonitor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
type snapshotRow struct {
|
||||
ID string
|
||||
CollectedAt time.Time
|
||||
Payload json.RawMessage
|
||||
}
|
||||
|
||||
func (s *Service) loadSnapshot(ctx context.Context, id string, maxAge time.Duration) (snapshotRow, bool, error) {
|
||||
var row snapshotRow
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
SELECT id, collected_at, payload_json
|
||||
FROM postgres_monitor_snapshot
|
||||
WHERE id = $1 AND collected_at >= $2`,
|
||||
id, time.Now().UTC().Add(-maxAge)).Scan(&row.ID, &row.CollectedAt, &row.Payload)
|
||||
if err != nil {
|
||||
return snapshotRow{}, false, nil
|
||||
}
|
||||
return row, true, nil
|
||||
}
|
||||
|
||||
func (s *Service) UpsertSnapshot(ctx context.Context, id string, payload any) error {
|
||||
if s == nil || s.pool == nil {
|
||||
return fmt.Errorf("pgmonitor: postgres not configured")
|
||||
}
|
||||
b, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = s.pool.Exec(ctx, `
|
||||
INSERT INTO postgres_monitor_snapshot (id, collected_at, payload_json)
|
||||
VALUES ($1, now(), $2::jsonb)
|
||||
ON CONFLICT (id) DO UPDATE SET collected_at = EXCLUDED.collected_at, payload_json = EXCLUDED.payload_json`,
|
||||
id, string(b))
|
||||
return err
|
||||
}
|
||||
|
||||
func decodePayload(raw json.RawMessage, dest any) error {
|
||||
return json.Unmarshal(raw, dest)
|
||||
}
|
||||
|
||||
// RefreshMetricsSnapshot stores overview and tables for heavy reads.
|
||||
func (s *Service) RefreshMetricsSnapshot(ctx context.Context) error {
|
||||
ov, err := s.fetchOverview(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.UpsertSnapshot(ctx, "overview", ov); err != nil {
|
||||
return err
|
||||
}
|
||||
tables, err := queryTables(ctx, s.pool, 50)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.UpsertSnapshot(ctx, "tables", tables)
|
||||
}
|
||||
|
||||
// AggregateSlowQueries stores top statements snapshot.
|
||||
func (s *Service) AggregateSlowQueries(ctx context.Context, limit int) error {
|
||||
if !s.statementsQueryable(ctx) {
|
||||
return s.UpsertSnapshot(ctx, "slow_queries", []QueryStat{})
|
||||
}
|
||||
items, err := queryTopStatements(ctx, s.pool, clampLimit(limit, 20, 100))
|
||||
if err != nil {
|
||||
if isPgStatStatementsUnavailable(err) {
|
||||
s.markStatementsUnavailable()
|
||||
return s.UpsertSnapshot(ctx, "slow_queries", []QueryStat{})
|
||||
}
|
||||
return err
|
||||
}
|
||||
return s.UpsertSnapshot(ctx, "slow_queries", items)
|
||||
}
|
||||
|
||||
// EstimateTableBloat refreshes bloat heuristics on tables snapshot.
|
||||
func (s *Service) EstimateTableBloat(ctx context.Context) error {
|
||||
tables, err := queryTables(ctx, s.pool, 100)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.UpsertSnapshot(ctx, "table_bloat", tables)
|
||||
}
|
||||
|
||||
// AnalyzeIndexUsage stores unused indexes.
|
||||
func (s *Service) AnalyzeIndexUsage(ctx context.Context) error {
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT indexrelname, idx_scan, pg_relation_size(indexrelid)
|
||||
FROM pg_stat_user_indexes
|
||||
WHERE schemaname = 'public' AND idx_scan = 0
|
||||
ORDER BY pg_relation_size(indexrelid) DESC
|
||||
LIMIT 50`)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pgmonitor: index usage: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
type unused struct {
|
||||
Index string `json:"index"`
|
||||
IdxScan int64 `json:"idx_scan"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
}
|
||||
var items []unused
|
||||
for rows.Next() {
|
||||
var u unused
|
||||
if err := rows.Scan(&u.Index, &u.IdxScan, &u.SizeBytes); err != nil {
|
||||
return err
|
||||
}
|
||||
items = append(items, u)
|
||||
}
|
||||
return s.UpsertSnapshot(ctx, "unused_indexes", items)
|
||||
}
|
||||
|
||||
// DetectAutovacuumLag stores tables with high dead tuple ratio.
|
||||
func (s *Service) DetectAutovacuumLag(ctx context.Context) error {
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT relname, n_dead_tup, last_autovacuum,
|
||||
CASE WHEN n_live_tup + n_dead_tup > 0
|
||||
THEN round(n_dead_tup::numeric / (n_live_tup + n_dead_tup), 4) ELSE 0 END
|
||||
FROM pg_stat_user_tables
|
||||
WHERE schemaname = 'public' AND n_dead_tup > 1000
|
||||
ORDER BY n_dead_tup DESC
|
||||
LIMIT 30`)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pgmonitor: autovacuum lag: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
type lagRow struct {
|
||||
Relname string `json:"relname"`
|
||||
DeadTuples int64 `json:"n_dead_tup"`
|
||||
LastAutovacuum *time.Time `json:"last_autovacuum,omitempty"`
|
||||
Ratio float64 `json:"ratio"`
|
||||
}
|
||||
var items []lagRow
|
||||
for rows.Next() {
|
||||
var r lagRow
|
||||
if err := rows.Scan(&r.Relname, &r.DeadTuples, &r.LastAutovacuum, &r.Ratio); err != nil {
|
||||
return err
|
||||
}
|
||||
items = append(items, r)
|
||||
}
|
||||
return s.UpsertSnapshot(ctx, "autovacuum_lag", items)
|
||||
}
|
||||
|
||||
// RunPeriodicAnalyzerJobs runs all snapshot analyzers (for scheduler).
|
||||
func RunPeriodicAnalyzerJobs(ctx context.Context, pool *pgxpool.Pool) {
|
||||
s := NewService(pool)
|
||||
if s == nil {
|
||||
return
|
||||
}
|
||||
_ = s.RefreshMetricsSnapshot(ctx)
|
||||
_ = s.AggregateSlowQueries(ctx, 30)
|
||||
_ = s.EstimateTableBloat(ctx)
|
||||
_ = s.AnalyzeIndexUsage(ctx)
|
||||
_ = s.DetectAutovacuumLag(ctx)
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
package pgmonitor
|
||||
|
||||
import "time"
|
||||
|
||||
// Overview is instance-level PostgreSQL health snapshot.
|
||||
type Overview struct {
|
||||
CollectedAt time.Time `json:"collected_at"`
|
||||
Connections Connections `json:"connections"`
|
||||
Database DatabaseStats `json:"database"`
|
||||
Bgwriter BgwriterStats `json:"bgwriter"`
|
||||
SizeBytes int64 `json:"database_size_bytes"`
|
||||
MemorySettings MemorySettings `json:"memory_settings"`
|
||||
Replication []ReplicationPeer `json:"replication"`
|
||||
StatementsEnabled bool `json:"pg_stat_statements_enabled"`
|
||||
}
|
||||
|
||||
// Connections summarizes pg_stat_activity for current database.
|
||||
type Connections struct {
|
||||
Active int `json:"active"`
|
||||
Idle int `json:"idle"`
|
||||
Total int `json:"total"`
|
||||
MaxConnections int `json:"max_connections"`
|
||||
}
|
||||
|
||||
// DatabaseStats from pg_stat_database.
|
||||
type DatabaseStats struct {
|
||||
Backends int `json:"backends"`
|
||||
XactCommit int64 `json:"xact_commit"`
|
||||
XactRollback int64 `json:"xact_rollback"`
|
||||
Deadlocks int64 `json:"deadlocks"`
|
||||
BlksHit int64 `json:"blks_hit"`
|
||||
BlksRead int64 `json:"blks_read"`
|
||||
CacheHitPct float64 `json:"cache_hit_pct"`
|
||||
}
|
||||
|
||||
// BgwriterStats from pg_stat_bgwriter.
|
||||
type BgwriterStats struct {
|
||||
CheckpointsTimed int64 `json:"checkpoints_timed"`
|
||||
CheckpointsReq int64 `json:"checkpoints_req"`
|
||||
BuffersCheckpoint int64 `json:"buffers_checkpoint"`
|
||||
BuffersClean int64 `json:"buffers_clean"`
|
||||
MaxWrittenClean int64 `json:"maxwritten_clean"`
|
||||
BuffersBackend int64 `json:"buffers_backend"`
|
||||
BuffersAlloc int64 `json:"buffers_alloc"`
|
||||
}
|
||||
|
||||
// MemorySettings is best-effort from pg_settings (not RSS).
|
||||
type MemorySettings struct {
|
||||
SharedBuffers string `json:"shared_buffers"`
|
||||
WorkMem string `json:"work_mem"`
|
||||
EffectiveCacheSize string `json:"effective_cache_size"`
|
||||
}
|
||||
|
||||
// ReplicationPeer from pg_stat_replication.
|
||||
type ReplicationPeer struct {
|
||||
ClientAddr string `json:"client_addr,omitempty"`
|
||||
State string `json:"state"`
|
||||
SyncState string `json:"sync_state,omitempty"`
|
||||
LagMs *int64 `json:"lag_ms,omitempty"`
|
||||
}
|
||||
|
||||
// QueryStat is a row from pg_stat_statements or snapshot.
|
||||
type QueryStat struct {
|
||||
QueryID int64 `json:"queryid,omitempty"`
|
||||
Query string `json:"query"`
|
||||
Calls int64 `json:"calls"`
|
||||
TotalExecMs float64 `json:"total_exec_ms"`
|
||||
MeanExecMs float64 `json:"mean_exec_ms"`
|
||||
Rows int64 `json:"rows"`
|
||||
}
|
||||
|
||||
// QueriesResponse for GET /monitoring/postgres/queries.
|
||||
type QueriesResponse struct {
|
||||
CollectedAt time.Time `json:"collected_at"`
|
||||
Source string `json:"source"` // live | snapshot | unavailable
|
||||
Items []QueryStat `json:"items"`
|
||||
StatementsAvailable bool `json:"statements_available"`
|
||||
StatementsHint string `json:"statements_hint,omitempty"`
|
||||
}
|
||||
|
||||
// LockRow describes a lock / blocking session.
|
||||
type LockRow struct {
|
||||
Locktype string `json:"locktype"`
|
||||
Mode string `json:"mode"`
|
||||
Granted bool `json:"granted"`
|
||||
PID int32 `json:"pid"`
|
||||
User string `json:"usename,omitempty"`
|
||||
State string `json:"state,omitempty"`
|
||||
Query string `json:"query,omitempty"`
|
||||
Blocked bool `json:"blocked"`
|
||||
}
|
||||
|
||||
// TableStat combines size and scan stats for a user table.
|
||||
type TableStat struct {
|
||||
Relname string `json:"relname"`
|
||||
TotalBytes int64 `json:"total_bytes"`
|
||||
HeapBlksRead int64 `json:"heap_blks_read"`
|
||||
HeapBlksHit int64 `json:"heap_blks_hit"`
|
||||
IdxScan int64 `json:"idx_scan"`
|
||||
SeqScan int64 `json:"seq_scan"`
|
||||
DeadTuples int64 `json:"n_dead_tup"`
|
||||
LastAutovacuum *time.Time `json:"last_autovacuum,omitempty"`
|
||||
BloatRatio float64 `json:"bloat_ratio,omitempty"`
|
||||
}
|
||||
|
||||
// RecommendationItem is a heuristic ops hint.
|
||||
type RecommendationItem struct {
|
||||
Severity string `json:"severity"` // info | warn | critical
|
||||
Code string `json:"code"`
|
||||
Title string `json:"title"`
|
||||
Detail string `json:"detail"`
|
||||
Refs []string `json:"refs,omitempty"`
|
||||
}
|
||||
|
||||
// RecommendationsResponse for GET recommendations.
|
||||
type RecommendationsResponse struct {
|
||||
CollectedAt time.Time `json:"collected_at"`
|
||||
Items []RecommendationItem `json:"items"`
|
||||
}
|
||||
|
||||
// CorrelationPoint is one aligned sample for overlay charts.
|
||||
type CorrelationPoint struct {
|
||||
Timestamp time.Time `json:"timestamp"`
|
||||
PipelineRefreshP99Ms float64 `json:"pipeline_refresh_p99_ms,omitempty"`
|
||||
BirdScrapeOK *float64 `json:"bird_scrape_ok,omitempty"`
|
||||
HTTPRequestRate float64 `json:"http_request_rate,omitempty"`
|
||||
CacheHitPct float64 `json:"cache_hit_pct,omitempty"`
|
||||
}
|
||||
|
||||
// CorrelationResponse for GET /monitoring/correlation.
|
||||
type CorrelationResponse struct {
|
||||
WindowMinutes int `json:"window_minutes"`
|
||||
Points []CorrelationPoint `json:"points"`
|
||||
}
|
||||
|
||||
// MaintenanceLogRow is an audit entry.
|
||||
type MaintenanceLogRow struct {
|
||||
ID string `json:"id"`
|
||||
TenantID string `json:"tenant_id,omitempty"`
|
||||
ActorPrefix string `json:"actor_prefix,omitempty"`
|
||||
Kind string `json:"kind"`
|
||||
TargetTable string `json:"target_table,omitempty"`
|
||||
DryRun bool `json:"dry_run"`
|
||||
Status string `json:"status"`
|
||||
Detail map[string]any `json:"detail,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||
}
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
)
|
||||
|
||||
func TestCollectModulePrefixRows_CDNSendsIfNoneMatch(t *testing.T) {
|
||||
t.Setenv("EVOBGP_CDN_ALLOW_PRIVATE", "1")
|
||||
m := store.NewMemory()
|
||||
m.SeedDemo()
|
||||
tenant, _, _, _, _ := m.DemoIDs()
|
||||
@@ -25,7 +26,7 @@ func TestCollectModulePrefixRows_CDNSendsIfNoneMatch(t *testing.T) {
|
||||
}
|
||||
|
||||
var gotIfNoneMatch string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
gotIfNoneMatch = strings.TrimSpace(r.Header.Get("If-None-Match"))
|
||||
w.Header().Set("ETag", "etag-new")
|
||||
_, _ = w.Write([]byte("198.51.100.0/24\n"))
|
||||
@@ -54,6 +55,7 @@ func TestCollectModulePrefixRows_CDNSendsIfNoneMatch(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestCollectModulePrefixRows_CDN304UsesSnapshot(t *testing.T) {
|
||||
t.Setenv("EVOBGP_CDN_ALLOW_PRIVATE", "1")
|
||||
m := store.NewMemory()
|
||||
m.SeedDemo()
|
||||
tenant, _, _, _, _ := m.DemoIDs()
|
||||
@@ -67,7 +69,7 @@ func TestCollectModulePrefixRows_CDN304UsesSnapshot(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusNotModified)
|
||||
}))
|
||||
defer srv.Close()
|
||||
@@ -99,6 +101,7 @@ func TestCollectModulePrefixRows_CDN304UsesSnapshot(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRefreshModuleIngest_CDN304UsesStoredSnapshot(t *testing.T) {
|
||||
t.Setenv("EVOBGP_CDN_ALLOW_PRIVATE", "1")
|
||||
m := store.NewMemory()
|
||||
m.SeedDemo()
|
||||
tenant, _, _, _, _ := m.DemoIDs()
|
||||
@@ -112,7 +115,7 @@ func TestRefreshModuleIngest_CDN304UsesStoredSnapshot(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusNotModified)
|
||||
}))
|
||||
defer srv.Close()
|
||||
@@ -140,6 +143,7 @@ func TestRefreshModuleIngest_CDN304UsesStoredSnapshot(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestCollectModulePrefixRows_CDN304RetriesWithoutETag(t *testing.T) {
|
||||
t.Setenv("EVOBGP_CDN_ALLOW_PRIVATE", "1")
|
||||
m := store.NewMemory()
|
||||
m.SeedDemo()
|
||||
tenant, _, _, _, _ := m.DemoIDs()
|
||||
@@ -154,7 +158,7 @@ func TestCollectModulePrefixRows_CDN304RetriesWithoutETag(t *testing.T) {
|
||||
}
|
||||
|
||||
var calls int
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
calls++
|
||||
if calls == 1 {
|
||||
if got := strings.TrimSpace(r.Header.Get("If-None-Match")); got != "etag-stable" {
|
||||
|
||||
@@ -119,6 +119,12 @@ func applyCDNSourceHTTPResult(ctx context.Context, st store.Backend, hc *http.Cl
|
||||
if u == "" {
|
||||
return nil, nil
|
||||
}
|
||||
if _, err := ValidateCDNURL(u); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := ResolveCDNURLHost(ctx, u); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
sourceKey := cdnSourceKey(src.ID)
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
|
||||
if err != nil {
|
||||
@@ -127,7 +133,7 @@ func applyCDNSourceHTTPResult(ctx context.Context, st store.Backend, hc *http.Cl
|
||||
if etag := strings.TrimSpace(src.Etag); etag != "" {
|
||||
req.Header.Set("If-None-Match", etag)
|
||||
}
|
||||
resp, err := hc.Do(req)
|
||||
resp, err := upstreamHTTPDo(ctx, hc, req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cdn fetch %s: %w", u, err)
|
||||
}
|
||||
@@ -143,7 +149,7 @@ func applyCDNSourceHTTPResult(ctx context.Context, st store.Backend, hc *http.Cl
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resp, err = hc.Do(req2)
|
||||
resp, err = upstreamHTTPDo(ctx, hc, req2)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cdn fetch %s: %w", u, err)
|
||||
}
|
||||
@@ -189,6 +195,12 @@ func fetchCDNSourceRows(ctx context.Context, st store.Backend, hc *http.Client,
|
||||
if u == "" {
|
||||
return nil, nil
|
||||
}
|
||||
if _, err := ValidateCDNURL(u); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := ResolveCDNURLHost(ctx, u); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
sourceKey := cdnSourceKey(src.ID)
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
|
||||
if err != nil {
|
||||
@@ -197,7 +209,7 @@ func fetchCDNSourceRows(ctx context.Context, st store.Backend, hc *http.Client,
|
||||
if etag := strings.TrimSpace(src.Etag); etag != "" {
|
||||
req.Header.Set("If-None-Match", etag)
|
||||
}
|
||||
resp, err := hc.Do(req)
|
||||
resp, err := upstreamHTTPDo(ctx, hc, req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cdn fetch %s: %w", u, err)
|
||||
}
|
||||
@@ -212,7 +224,7 @@ func fetchCDNSourceRows(ctx context.Context, st store.Backend, hc *http.Client,
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resp, err = hc.Do(req2)
|
||||
resp, err = upstreamHTTPDo(ctx, hc, req2)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cdn fetch %s: %w", u, err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
func isBlockedCDNIP(ip netip.Addr) bool {
|
||||
if allowPrivateCDNURLs() {
|
||||
return false
|
||||
}
|
||||
if !ip.IsValid() {
|
||||
return true
|
||||
}
|
||||
return ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsMulticast() ||
|
||||
ip.IsUnspecified() || ip == netip.MustParseAddr("169.254.169.254")
|
||||
}
|
||||
|
||||
func allowPrivateCDNURLs() bool {
|
||||
v := strings.TrimSpace(os.Getenv("EVOBGP_CDN_ALLOW_PRIVATE"))
|
||||
return v == "1" || strings.EqualFold(v, "true")
|
||||
}
|
||||
|
||||
func isBlockedCDNHostname(host string) bool {
|
||||
if allowPrivateCDNURLs() {
|
||||
return false
|
||||
}
|
||||
h := strings.ToLower(strings.TrimSpace(host))
|
||||
if h == "" || h == "localhost" {
|
||||
return true
|
||||
}
|
||||
if strings.HasSuffix(h, ".local") || strings.HasSuffix(h, ".internal") || strings.HasSuffix(h, ".localhost") {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// ValidateCDNURL checks CDN source URLs for SSRF-safe HTTPS endpoints (hostname only; no DNS resolve).
|
||||
func ValidateCDNURL(raw string) (string, error) {
|
||||
raw = strings.TrimSpace(raw)
|
||||
if raw == "" {
|
||||
return "", fmt.Errorf("pipeline: cdn url is required")
|
||||
}
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("pipeline: cdn url invalid: %w", err)
|
||||
}
|
||||
if u.Scheme != "https" {
|
||||
return "", fmt.Errorf("pipeline: cdn url must use https")
|
||||
}
|
||||
if u.User != nil {
|
||||
return "", fmt.Errorf("pipeline: cdn url must not include credentials")
|
||||
}
|
||||
host := strings.TrimSpace(u.Hostname())
|
||||
if host == "" {
|
||||
return "", fmt.Errorf("pipeline: cdn url missing host")
|
||||
}
|
||||
if isBlockedCDNHostname(host) {
|
||||
return "", fmt.Errorf("pipeline: cdn url blocked host")
|
||||
}
|
||||
if ip, err := netip.ParseAddr(host); err == nil {
|
||||
if isBlockedCDNIP(ip) {
|
||||
return "", fmt.Errorf("pipeline: cdn url blocked host")
|
||||
}
|
||||
}
|
||||
return u.String(), nil
|
||||
}
|
||||
|
||||
// ResolveCDNURLHost resolves a CDN hostname and rejects private/link-local targets (SSRF at fetch time).
|
||||
func ResolveCDNURLHost(ctx context.Context, raw string) error {
|
||||
u, err := url.Parse(strings.TrimSpace(raw))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
host := strings.TrimSpace(u.Hostname())
|
||||
if host == "" {
|
||||
return fmt.Errorf("pipeline: cdn url missing host")
|
||||
}
|
||||
if ip, err := netip.ParseAddr(host); err == nil {
|
||||
if isBlockedCDNIP(ip) {
|
||||
return fmt.Errorf("pipeline: cdn url blocked host")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if isBlockedCDNHostname(host) {
|
||||
return fmt.Errorf("pipeline: cdn url blocked host")
|
||||
}
|
||||
if ctx == nil {
|
||||
ctx = context.Background()
|
||||
}
|
||||
resolveCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
ips, err := net.DefaultResolver.LookupIP(resolveCtx, "ip", host)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pipeline: cdn url dns lookup: %w", err)
|
||||
}
|
||||
if len(ips) == 0 {
|
||||
return fmt.Errorf("pipeline: cdn url dns lookup: no addresses")
|
||||
}
|
||||
for _, ip := range ips {
|
||||
addr, ok := netip.AddrFromSlice(ip)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if isBlockedCDNIP(addr) {
|
||||
return fmt.Errorf("pipeline: cdn url resolves to blocked address")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package pipeline
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestValidateCDNURL(t *testing.T) {
|
||||
tests := []struct {
|
||||
raw string
|
||||
ok bool
|
||||
want string
|
||||
}{
|
||||
{"https://cdn.example.com/prefixes.txt", true, "https://cdn.example.com/prefixes.txt"},
|
||||
{"http://cdn.example.com/x", false, ""},
|
||||
{"https://127.0.0.1/x", false, ""},
|
||||
{"https://10.0.0.1/x", false, ""},
|
||||
{"https://169.254.169.254/latest/meta-data", false, ""},
|
||||
{"https://localhost/x", false, ""},
|
||||
{"file:///etc/passwd", false, ""},
|
||||
{"https://user:pass@cdn.example.com/x", false, ""},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
got, err := ValidateCDNURL(tc.raw)
|
||||
if tc.ok && err != nil {
|
||||
t.Errorf("%q: unexpected err %v", tc.raw, err)
|
||||
continue
|
||||
}
|
||||
if !tc.ok && err == nil {
|
||||
t.Errorf("%q: expected error", tc.raw)
|
||||
continue
|
||||
}
|
||||
if tc.ok && got != tc.want {
|
||||
t.Errorf("%q: got %q want %q", tc.raw, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -25,7 +25,7 @@ func prefixRowsForSource(rows []store.PrefixRow, sourceKey string) []store.Prefi
|
||||
return out
|
||||
}
|
||||
|
||||
func collectASPrefixRows(ctx context.Context, st store.Backend, hc *http.Client, tenantID string, mod *store.Module, list []*store.ASEntry) ([]store.PrefixRow, error) {
|
||||
func collectASPrefixRows(ctx context.Context, st store.Backend, hc *http.Client, tenantID string, mod *store.Module, list []*store.ASEntry, priorSnapshot []store.PrefixRow) ([]store.PrefixRow, error) {
|
||||
moduleID := mod.ID
|
||||
legacy := strings.TrimSpace(os.Getenv("EVOBGP_ASN_RESOLVE")) == "0"
|
||||
if legacy {
|
||||
@@ -76,6 +76,41 @@ func collectASPrefixRows(ctx context.Context, st store.Backend, hc *http.Client,
|
||||
}
|
||||
pfxs, holder, err := resolveASNForEntry(ctx, st, hc, entry.ASN)
|
||||
if err != nil {
|
||||
if staleOnUpstreamError() {
|
||||
if staleRows, staleHolder, ok := staleASNPrefixes(st, priorSnapshot, entry.ASN); ok {
|
||||
logStaleUpstream("asn", fmt.Sprintf("AS%d: %v", entry.ASN, err))
|
||||
src := fmt.Sprintf("as:%d", entry.ASN)
|
||||
rows := append([]store.PrefixRow(nil), staleRows...)
|
||||
for i := range rows {
|
||||
rows[i].CommunityID = comm
|
||||
rows[i].Source = src
|
||||
}
|
||||
results[idx] = entryResult{
|
||||
rows: rows,
|
||||
metaID: entry.ID,
|
||||
asn: entry.ASN,
|
||||
holder: staleHolder,
|
||||
count: int64(len(rows)),
|
||||
}
|
||||
return
|
||||
}
|
||||
if pfxs2, holder2, ok := asnCacheExpired(st, entry.ASN); ok {
|
||||
logStaleUpstream("asn", fmt.Sprintf("AS%d expired cache: %v", entry.ASN, err))
|
||||
src := fmt.Sprintf("as:%d", entry.ASN)
|
||||
var rows []store.PrefixRow
|
||||
for _, pfx := range pfxs2 {
|
||||
rows = append(rows, store.PrefixRow{Prefix: pfx.String(), CommunityID: comm, Source: src})
|
||||
}
|
||||
results[idx] = entryResult{
|
||||
rows: rows,
|
||||
metaID: entry.ID,
|
||||
asn: entry.ASN,
|
||||
holder: holder2,
|
||||
count: int64(len(pfxs2)),
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
results[idx] = entryResult{err: fmt.Errorf("resolve AS%d: %w", entry.ASN, err)}
|
||||
return
|
||||
}
|
||||
@@ -167,6 +202,13 @@ func collectCDNPrefixRows(ctx context.Context, st store.Backend, hc *http.Client
|
||||
}
|
||||
rows, err := fetchCDNSourceRows(ctx, st, hc, tenantID, moduleID, mod, src, priorSnapshot, now)
|
||||
if err != nil {
|
||||
if staleOnUpstreamError() {
|
||||
if cached, ok := staleCDNPrefixes(st, tenantID, moduleID, priorSnapshot, src.ID); ok {
|
||||
logStaleUpstream("cdn", fmt.Sprintf("source %s: %v", src.ID, err))
|
||||
results[idx] = srcResult{rows: cached}
|
||||
return
|
||||
}
|
||||
}
|
||||
results[idx] = srcResult{err: err}
|
||||
return
|
||||
}
|
||||
@@ -190,7 +232,7 @@ func collectCDNPrefixRows(ctx context.Context, st store.Backend, hc *http.Client
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func collectDomainPrefixRows(ctx context.Context, hc *http.Client, mod *store.Module, profiles []*store.DohProfile, policy string, entries []*store.DomainEntry) ([]store.PrefixRow, error) {
|
||||
func collectDomainPrefixRows(ctx context.Context, hc *http.Client, mod *store.Module, profiles []*store.DohProfile, policy string, entries []*store.DomainEntry, priorSnapshot []store.PrefixRow) ([]store.PrefixRow, error) {
|
||||
var validDom []*store.DomainEntry
|
||||
for _, e := range entries {
|
||||
if e != nil {
|
||||
@@ -219,6 +261,19 @@ func collectDomainPrefixRows(ctx context.Context, hc *http.Client, mod *store.Mo
|
||||
}
|
||||
addrs, err := resolveDomainIPsWithPolicy(ctx, hc, profiles, policy, entry.FQDN)
|
||||
if err != nil {
|
||||
if staleOnUpstreamError() {
|
||||
if cached, ok := staleDomainPrefixes(priorSnapshot, entry.FQDN); ok {
|
||||
logStaleUpstream("domain", fmt.Sprintf("%q: %v", entry.FQDN, err))
|
||||
rows := append([]store.PrefixRow(nil), cached...)
|
||||
for i := range rows {
|
||||
if rows[i].CommunityID == nil {
|
||||
rows[i].CommunityID = comm
|
||||
}
|
||||
}
|
||||
results[idx] = domResult{rows: rows}
|
||||
return
|
||||
}
|
||||
}
|
||||
results[idx] = domResult{err: fmt.Errorf("resolve domain %q: %w", entry.FQDN, err)}
|
||||
return
|
||||
}
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"net/netip"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"evobgp/internal/store"
|
||||
)
|
||||
|
||||
// staleOnUpstreamError reports whether ingest should keep last-known prefixes when an upstream fetch fails.
|
||||
// Enabled by default; set EVOBGP_STALE_ON_UPSTREAM_ERROR=0 to restore fail-fast behavior.
|
||||
func staleOnUpstreamError() bool {
|
||||
v := strings.TrimSpace(os.Getenv("EVOBGP_STALE_ON_UPSTREAM_ERROR"))
|
||||
if v == "" || v == "1" || strings.EqualFold(v, "true") {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func logStaleUpstream(kind, detail string) {
|
||||
log.Printf("pipeline: stale upstream fallback (%s): %s", kind, detail)
|
||||
}
|
||||
|
||||
func staleASNPrefixes(st store.Backend, priorSnapshot []store.PrefixRow, asn int64) ([]store.PrefixRow, string, bool) {
|
||||
sourceKey := fmt.Sprintf("as:%d", asn)
|
||||
if cached := prefixRowsForSource(priorSnapshot, sourceKey); len(cached) > 0 {
|
||||
return cached, "", true
|
||||
}
|
||||
if st == nil {
|
||||
return nil, "", false
|
||||
}
|
||||
ent, ok, err := st.GetASNPrefixCache(asn)
|
||||
if err != nil || !ok || ent == nil || len(ent.Prefixes) == 0 {
|
||||
return nil, "", false
|
||||
}
|
||||
var rows []store.PrefixRow
|
||||
for _, p := range ent.Prefixes {
|
||||
pfx, perr := netip.ParsePrefix(strings.TrimSpace(p))
|
||||
if perr != nil {
|
||||
continue
|
||||
}
|
||||
rows = append(rows, store.PrefixRow{Prefix: pfx.Masked().String(), Source: sourceKey})
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
return nil, "", false
|
||||
}
|
||||
return rows, ent.Holder, true
|
||||
}
|
||||
|
||||
func staleDomainPrefixes(priorSnapshot []store.PrefixRow, fqdn string) ([]store.PrefixRow, bool) {
|
||||
sourceKey := "domain:" + strings.TrimSpace(fqdn)
|
||||
cached := prefixRowsForSource(priorSnapshot, sourceKey)
|
||||
return cached, len(cached) > 0
|
||||
}
|
||||
|
||||
func staleCDNPrefixes(st store.Backend, tenantID, moduleID string, priorSnapshot []store.PrefixRow, sourceID string) ([]store.PrefixRow, bool) {
|
||||
sourceKey := cdnSourceKey(sourceID)
|
||||
cached := cachedCDNPrefixRows(st, tenantID, moduleID, priorSnapshot, sourceKey)
|
||||
return cached, len(cached) > 0
|
||||
}
|
||||
|
||||
// asnCacheExpired returns cached ASN prefixes even past TTL (for stale fallback only).
|
||||
func asnCacheExpired(st store.Backend, asn int64) ([]netip.Prefix, string, bool) {
|
||||
if st == nil {
|
||||
return nil, "", false
|
||||
}
|
||||
ent, ok, err := st.GetASNPrefixCache(asn)
|
||||
if err != nil || !ok || ent == nil || len(ent.Prefixes) == 0 {
|
||||
return nil, "", false
|
||||
}
|
||||
out := make([]netip.Prefix, 0, len(ent.Prefixes))
|
||||
for _, p := range ent.Prefixes {
|
||||
pfx, perr := netip.ParsePrefix(strings.TrimSpace(p))
|
||||
if perr != nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, pfx.Masked())
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, "", false
|
||||
}
|
||||
return out, ent.Holder, true
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"evobgp/internal/store"
|
||||
)
|
||||
|
||||
func TestCollectCDNPrefixRows_StaleOnFetchError(t *testing.T) {
|
||||
t.Setenv("EVOBGP_STALE_ON_UPSTREAM_ERROR", "1")
|
||||
t.Setenv("EVOBGP_CDN_ALLOW_PRIVATE", "1")
|
||||
|
||||
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "upstream down", http.StatusServiceUnavailable)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
st := store.NewMemory()
|
||||
st.SeedDemo()
|
||||
tenant, _, _, _, _ := st.DemoIDs()
|
||||
mod, err := st.CreateModule(tenant, &store.Module{Type: "CDN_CIDRS", Name: "cdn", Enabled: true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := st.CreateCDNSource(tenant, mod.ID, &store.CDNSource{
|
||||
ID: "s1", URL: srv.URL, SourceKind: "plain",
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
prior := []store.PrefixRow{
|
||||
{Prefix: "203.0.113.0/24", Source: "cdn:s1"},
|
||||
}
|
||||
rows, err := collectCDNPrefixRows(context.Background(), st, srv.Client(), tenant, mod, []*store.CDNSource{{ID: "s1", URL: srv.URL, SourceKind: "plain"}}, prior)
|
||||
if err != nil {
|
||||
t.Fatalf("expected stale fallback, got err: %v", err)
|
||||
}
|
||||
if len(rows) != 1 || rows[0].Prefix != "203.0.113.0/24" {
|
||||
t.Fatalf("unexpected rows: %+v", rows)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectCDNPrefixRows_FailFastWhenNoStale(t *testing.T) {
|
||||
t.Setenv("EVOBGP_STALE_ON_UPSTREAM_ERROR", "0")
|
||||
t.Setenv("EVOBGP_CDN_ALLOW_PRIVATE", "1")
|
||||
|
||||
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "upstream down", http.StatusServiceUnavailable)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
st := store.NewMemory()
|
||||
st.SeedDemo()
|
||||
tenant, _, _, _, _ := st.DemoIDs()
|
||||
mod, err := st.CreateModule(tenant, &store.Module{Type: "CDN_CIDRS", Name: "cdn", Enabled: true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := st.CreateCDNSource(tenant, mod.ID, &store.CDNSource{
|
||||
ID: "s1", URL: srv.URL, SourceKind: "plain",
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
_, err = collectCDNPrefixRows(context.Background(), st, srv.Client(), tenant, mod, []*store.CDNSource{{ID: "s1", URL: srv.URL, SourceKind: "plain"}}, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected error when stale disabled and no cache")
|
||||
}
|
||||
}
|
||||
@@ -5,86 +5,116 @@ import (
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"evobgp/internal/httpclient"
|
||||
"evobgp/internal/store"
|
||||
)
|
||||
|
||||
type prefetchTask struct {
|
||||
tenantID string
|
||||
mod *store.Module
|
||||
src *store.CDNSource
|
||||
}
|
||||
|
||||
// PrefetchCDNSourceETags performs conditional GETs for CDN sources; on 200 parses CIDRs into module_prefix_snapshot.
|
||||
func PrefetchCDNSourceETags(ctx context.Context, st store.Backend, hc *http.Client) error {
|
||||
if hc == nil {
|
||||
hc = http.DefaultClient
|
||||
hc = httpclient.New(httpclient.DefaultTimeout)
|
||||
}
|
||||
if ctx == nil {
|
||||
ctx = context.Background()
|
||||
}
|
||||
tenants, err := st.ListTenantIDs()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
var tasks []prefetchTask
|
||||
for _, tid := range tenants {
|
||||
for _, mod := range st.ListModules(tid) {
|
||||
if mod == nil || !mod.Enabled || mod.Type != "CDN_CIDRS" {
|
||||
continue
|
||||
}
|
||||
omod, err := st.GetModule(tid, mod.ID)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
sources, err := st.ListCDNSources(tid, mod.ID)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var prior []store.PrefixRow
|
||||
if snap, ok, _ := st.GetModulePrefixSnapshot(tid, mod.ID); ok && snap != nil {
|
||||
prior = snap.Prefixes
|
||||
}
|
||||
for _, src := range sources {
|
||||
if src == nil || strings.TrimSpace(src.URL) == "" {
|
||||
continue
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimSpace(src.URL), nil)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if etag := strings.TrimSpace(src.Etag); etag != "" {
|
||||
req.Header.Set("If-None-Match", etag)
|
||||
}
|
||||
resp, err := hc.Do(req)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if resp.StatusCode == http.StatusNotModified {
|
||||
_ = resp.Body.Close()
|
||||
continue
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
_ = resp.Body.Close()
|
||||
continue
|
||||
}
|
||||
body, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
|
||||
_ = resp.Body.Close()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
prefixStrs, err := parseCDNBody(string(body), src)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
newEtag := strings.TrimSpace(resp.Header.Get("ETag"))
|
||||
patch := &store.CDNSourcePatch{LastRefreshedAt: &now}
|
||||
if newEtag != "" && newEtag != strings.TrimSpace(src.Etag) {
|
||||
e := newEtag
|
||||
patch.Etag = &e
|
||||
}
|
||||
_, _ = st.UpdateCDNSource(tid, mod.ID, src.ID, patch)
|
||||
rows := cdnRowsFromParsed(omod, src, prefixStrs)
|
||||
_ = mergeCDNSourceIntoModuleSnapshot(st, tid, omod, src.ID, rows)
|
||||
_ = prior // prior may be stale after merge; refresh for next source in loop
|
||||
if snap, ok, _ := st.GetModulePrefixSnapshot(tid, mod.ID); ok && snap != nil {
|
||||
prior = snap.Prefixes
|
||||
if src != nil && strings.TrimSpace(src.URL) != "" {
|
||||
tasks = append(tasks, prefetchTask{tenantID: tid, mod: mod, src: src})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(tasks) == 0 {
|
||||
return nil
|
||||
}
|
||||
sem := make(chan struct{}, collectConcurrency())
|
||||
var wg sync.WaitGroup
|
||||
for _, task := range tasks {
|
||||
wg.Add(1)
|
||||
go func(t prefetchTask) {
|
||||
defer wg.Done()
|
||||
sem <- struct{}{}
|
||||
defer func() { <-sem }()
|
||||
prefetchOneCDNSource(ctx, st, hc, t)
|
||||
}(task)
|
||||
}
|
||||
wg.Wait()
|
||||
return nil
|
||||
}
|
||||
|
||||
func prefetchOneCDNSource(ctx context.Context, st store.Backend, hc *http.Client, t prefetchTask) {
|
||||
now := time.Now().UTC()
|
||||
tid, mod, src := t.tenantID, t.mod, t.src
|
||||
u := strings.TrimSpace(src.URL)
|
||||
if _, err := ValidateCDNURL(u); err != nil {
|
||||
return
|
||||
}
|
||||
if err := ResolveCDNURLHost(ctx, u); err != nil {
|
||||
return
|
||||
}
|
||||
omod, err := st.GetModule(tid, mod.ID)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if etag := strings.TrimSpace(src.Etag); etag != "" {
|
||||
req.Header.Set("If-None-Match", etag)
|
||||
}
|
||||
resp, err := upstreamHTTPDo(ctx, hc, req)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if resp.StatusCode == http.StatusNotModified {
|
||||
_ = resp.Body.Close()
|
||||
return
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
_ = resp.Body.Close()
|
||||
return
|
||||
}
|
||||
body, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
|
||||
_ = resp.Body.Close()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
prefixStrs, err := parseCDNBody(string(body), src)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
newEtag := strings.TrimSpace(resp.Header.Get("ETag"))
|
||||
patch := &store.CDNSourcePatch{LastRefreshedAt: &now}
|
||||
if newEtag != "" && newEtag != strings.TrimSpace(src.Etag) {
|
||||
e := newEtag
|
||||
patch.Etag = &e
|
||||
}
|
||||
_, _ = st.UpdateCDNSource(tid, mod.ID, src.ID, patch)
|
||||
rows := cdnRowsFromParsed(omod, src, prefixStrs)
|
||||
_ = mergeCDNSourceIntoModuleSnapshot(st, tid, omod, src.ID, rows)
|
||||
}
|
||||
|
||||
@@ -19,6 +19,7 @@ import (
|
||||
"time"
|
||||
|
||||
"evobgp/internal/birdfmt"
|
||||
"evobgp/internal/httpclient"
|
||||
"evobgp/internal/observability"
|
||||
"evobgp/internal/store"
|
||||
|
||||
@@ -50,7 +51,7 @@ func MaterializedASPrefixKey(asn int64) string {
|
||||
// It does not create a new config revision.
|
||||
func RefreshModuleIngest(ctx context.Context, st store.Backend, hc *http.Client, tenantID, moduleID string) error {
|
||||
if hc == nil {
|
||||
hc = http.DefaultClient
|
||||
hc = httpclient.New(httpclient.DefaultTimeout)
|
||||
}
|
||||
start := time.Now()
|
||||
mod, err := st.GetModule(tenantID, moduleID)
|
||||
@@ -84,7 +85,7 @@ func RefreshModuleIngest(ctx context.Context, st store.Backend, hc *http.Client,
|
||||
// If materialized prefixes are unchanged, returns latest revision id without creating a duplicate.
|
||||
func RenderTenantRevision(ctx context.Context, st store.Backend, hc *http.Client, tenantID, triggerModuleID string) (revisionID string, err error) {
|
||||
if hc == nil {
|
||||
hc = http.DefaultClient
|
||||
hc = httpclient.New(httpclient.DefaultTimeout)
|
||||
}
|
||||
agg, err := aggregateTenantPrefixRowsAll(ctx, st, hc, tenantID)
|
||||
if err != nil {
|
||||
@@ -117,7 +118,7 @@ func RenderTenantRevision(ctx context.Context, st store.Backend, hc *http.Client
|
||||
func RenderTenantRevisionFromPrefixes(ctx context.Context, st store.Backend, hc *http.Client, tenantID, triggerModuleID string, rows []store.PrefixRow) (revisionID string, err error) {
|
||||
_ = ctx
|
||||
if hc == nil {
|
||||
hc = http.DefaultClient
|
||||
hc = httpclient.New(httpclient.DefaultTimeout)
|
||||
}
|
||||
agg := append([]store.PrefixRow(nil), rows...)
|
||||
rawCount := len(agg)
|
||||
@@ -176,7 +177,7 @@ func collectModulePrefixRows(ctx context.Context, st store.Backend, hc *http.Cli
|
||||
return nil, err
|
||||
}
|
||||
sort.Slice(list, func(i, j int) bool { return list[i].ASN < list[j].ASN })
|
||||
return collectASPrefixRows(ctx, st, hc, tenantID, mod, list)
|
||||
return collectASPrefixRows(ctx, st, hc, tenantID, mod, list, priorSnapshot)
|
||||
case "CDN_CIDRS":
|
||||
sources, err := st.ListCDNSources(tenantID, moduleID)
|
||||
if err != nil {
|
||||
@@ -192,7 +193,7 @@ func collectModulePrefixRows(ctx context.Context, st store.Backend, hc *http.Cli
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return collectDomainPrefixRows(ctx, hc, mod, profiles, policy, entries)
|
||||
return collectDomainPrefixRows(ctx, hc, mod, profiles, policy, entries, priorSnapshot)
|
||||
default:
|
||||
return nil, fmt.Errorf("pipeline: unknown module type %q", mod.Type)
|
||||
}
|
||||
@@ -1029,7 +1030,7 @@ func renderPeersBirdFragment(st store.Backend, tenantID string, loc birdLocals)
|
||||
}
|
||||
pol := parsePeerPolicies(p.PoliciesJSON)
|
||||
lv4, lv6, asn := effectivePeerLocals(loc, pol)
|
||||
proto := peerProtocolName(p.ID)
|
||||
proto := birdfmt.PeerProtocolName(p.ID)
|
||||
ra := uint32(p.RemoteASN)
|
||||
if addr.Is4() {
|
||||
opts := birdfmt.BGPPeerFromTemplateOptions{
|
||||
@@ -1083,17 +1084,6 @@ func parsePeerPolicies(raw string) peerPolicyJSON {
|
||||
return pol
|
||||
}
|
||||
|
||||
func peerProtocolName(peerID string) string {
|
||||
s := strings.ReplaceAll(strings.TrimSpace(peerID), "-", "")
|
||||
if len(s) > 16 {
|
||||
s = s[:16]
|
||||
}
|
||||
if s == "" {
|
||||
s = "x"
|
||||
}
|
||||
return "evobgp_p_" + s
|
||||
}
|
||||
|
||||
// buildExpandedBirdText concatenates bird.conf and the contents of each standard include (for UI / preview).
|
||||
func buildExpandedBirdText(main string, frags map[string]string) string {
|
||||
var b strings.Builder
|
||||
|
||||
@@ -7,13 +7,14 @@ import (
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"evobgp/internal/httpclient"
|
||||
"evobgp/internal/store"
|
||||
)
|
||||
|
||||
// RefreshTenantModules ingests all listed modules in parallel and updates per-module snapshots.
|
||||
func RefreshTenantModules(ctx context.Context, st store.Backend, hc *http.Client, tenantID string, moduleIDs []string) error {
|
||||
if hc == nil {
|
||||
hc = http.DefaultClient
|
||||
hc = httpclient.New(httpclient.DefaultTimeout)
|
||||
}
|
||||
var ids []string
|
||||
seen := make(map[string]struct{})
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
|
||||
"evobgp/internal/httpclient"
|
||||
)
|
||||
|
||||
func upstreamHTTPDo(ctx context.Context, hc *http.Client, req *http.Request) (*http.Response, error) {
|
||||
if hc == nil {
|
||||
hc = httpclient.New(httpclient.DefaultTimeout)
|
||||
}
|
||||
resp, err := httpclient.DoWithBreaker(ctx, hc, req, 3)
|
||||
if err != nil {
|
||||
if req.URL != nil {
|
||||
return nil, fmt.Errorf("cdn fetch %s: %w", req.URL.String(), err)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
@@ -11,14 +11,22 @@ import (
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
func asnCacheRowTableExists(ctx context.Context, q queryRower) bool {
|
||||
var n int
|
||||
err := q.QueryRow(ctx, `
|
||||
SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = 'public' AND table_name = 'asn_prefix_cache_row'
|
||||
LIMIT 1`).Scan(&n)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func (p *Postgres) GetASNPrefixCache(asn int64) (*store.ASNPrefixCacheEntry, bool, error) {
|
||||
ctx := context.Background()
|
||||
var holder string
|
||||
var fetchedAt time.Time
|
||||
var raw []byte
|
||||
err := p.pool.QueryRow(ctx, `
|
||||
SELECT holder, fetched_at, prefixes_json FROM asn_prefix_cache WHERE asn = $1`, asn).
|
||||
Scan(&holder, &fetchedAt, &raw)
|
||||
SELECT holder, fetched_at FROM asn_prefix_cache WHERE asn = $1`, asn).
|
||||
Scan(&holder, &fetchedAt)
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, false, nil
|
||||
@@ -26,8 +34,25 @@ func (p *Postgres) GetASNPrefixCache(asn int64) (*store.ASNPrefixCacheEntry, boo
|
||||
return nil, false, err
|
||||
}
|
||||
var prefixes []string
|
||||
if len(raw) > 0 {
|
||||
_ = json.Unmarshal(raw, &prefixes)
|
||||
if asnCacheRowTableExists(ctx, p.pool) {
|
||||
rows, qerr := p.pool.Query(ctx, `
|
||||
SELECT prefix::text FROM asn_prefix_cache_row WHERE asn = $1 ORDER BY prefix`, asn)
|
||||
if qerr != nil {
|
||||
return nil, false, qerr
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var s string
|
||||
if err := rows.Scan(&s); err != nil {
|
||||
continue
|
||||
}
|
||||
prefixes = append(prefixes, s)
|
||||
}
|
||||
} else {
|
||||
var raw []byte
|
||||
if err := p.pool.QueryRow(ctx, `SELECT prefixes_json FROM asn_prefix_cache WHERE asn = $1`, asn).Scan(&raw); err == nil && len(raw) > 0 {
|
||||
_ = json.Unmarshal(raw, &prefixes)
|
||||
}
|
||||
}
|
||||
return &store.ASNPrefixCacheEntry{
|
||||
ASN: asn,
|
||||
@@ -38,18 +63,40 @@ func (p *Postgres) GetASNPrefixCache(asn int64) (*store.ASNPrefixCacheEntry, boo
|
||||
}
|
||||
|
||||
func (p *Postgres) SetASNPrefixCache(asn int64, holder string, prefixes []string) error {
|
||||
raw, err := json.Marshal(prefixes)
|
||||
ctx := context.Background()
|
||||
tx, err := p.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ctx := context.Background()
|
||||
_, err = p.pool.Exec(ctx, `
|
||||
INSERT INTO asn_prefix_cache (asn, holder, prefixes_json, fetched_at)
|
||||
VALUES ($1, $2, $3::jsonb, now())
|
||||
defer func() { _ = tx.Rollback(ctx) }()
|
||||
_, err = tx.Exec(ctx, `
|
||||
INSERT INTO asn_prefix_cache (asn, holder, fetched_at)
|
||||
VALUES ($1, $2, now())
|
||||
ON CONFLICT (asn) DO UPDATE SET
|
||||
holder = EXCLUDED.holder,
|
||||
prefixes_json = EXCLUDED.prefixes_json,
|
||||
fetched_at = EXCLUDED.fetched_at`,
|
||||
asn, holder, string(raw))
|
||||
return err
|
||||
fetched_at = EXCLUDED.fetched_at`, asn, holder)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if asnCacheRowTableExists(ctx, tx) {
|
||||
if _, err := tx.Exec(ctx, `DELETE FROM asn_prefix_cache_row WHERE asn = $1`, asn); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, pfx := range prefixes {
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO asn_prefix_cache_row (asn, prefix) VALUES ($1, $2::cidr)`, asn, pfx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
} else {
|
||||
raw, err := json.Marshal(prefixes)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE asn_prefix_cache SET prefixes_json = $2::jsonb WHERE asn = $1`, asn, string(raw)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
package repository
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
const defaultRepoTimeout = 60 * time.Second
|
||||
|
||||
// boundedRepoCtx returns a context with default repository I/O timeout.
|
||||
func boundedRepoCtx(parent context.Context) (context.Context, context.CancelFunc) {
|
||||
if parent == nil {
|
||||
parent = context.Background()
|
||||
}
|
||||
if _, ok := parent.Deadline(); ok {
|
||||
return parent, func() {}
|
||||
}
|
||||
return context.WithTimeout(parent, defaultRepoTimeout)
|
||||
}
|
||||
@@ -20,6 +20,28 @@ func NewJobAuditWriter(pool *pgxpool.Pool) *JobAuditWriter {
|
||||
return &JobAuditWriter{pool: pool}
|
||||
}
|
||||
|
||||
// UpsertQueued inserts a queued job row (best-effort).
|
||||
func (w *JobAuditWriter) UpsertQueued(ctx context.Context, tenantID, jobID, kind string, idempotencyKey *string, moduleID *string, meta map[string]any) {
|
||||
if w == nil || w.pool == nil {
|
||||
return
|
||||
}
|
||||
metaJSON, _ := json.Marshal(meta)
|
||||
var idem any
|
||||
if idempotencyKey != nil && *idempotencyKey != "" {
|
||||
idem = *idempotencyKey
|
||||
}
|
||||
var mod any
|
||||
if moduleID != nil && *moduleID != "" {
|
||||
mod = *moduleID
|
||||
}
|
||||
_, _ = w.pool.Exec(ctx, `
|
||||
INSERT INTO job_audit (id, tenant_id, kind, status, idempotency_key, module_id, meta_json, created_at)
|
||||
VALUES ($1::uuid, $2::uuid, $3, 'queued', $4, $5::uuid, $6::jsonb, now())
|
||||
ON CONFLICT (tenant_id, idempotency_key) WHERE idempotency_key IS NOT NULL
|
||||
DO UPDATE SET status='queued', meta_json=EXCLUDED.meta_json, module_id=EXCLUDED.module_id`,
|
||||
jobID, tenantID, kind, idem, mod, metaJSON)
|
||||
}
|
||||
|
||||
// UpsertRunning inserts or updates a running job row (best-effort).
|
||||
func (w *JobAuditWriter) UpsertRunning(ctx context.Context, tenantID, jobID, kind string, idempotencyKey *string, meta map[string]any) {
|
||||
if w == nil || w.pool == nil {
|
||||
@@ -33,8 +55,7 @@ func (w *JobAuditWriter) UpsertRunning(ctx context.Context, tenantID, jobID, kin
|
||||
_, _ = w.pool.Exec(ctx, `
|
||||
INSERT INTO job_audit (id, tenant_id, kind, status, idempotency_key, meta_json, created_at, started_at)
|
||||
VALUES ($1::uuid, $2::uuid, $3, 'running', $4, $5::jsonb, now(), now())
|
||||
ON CONFLICT (tenant_id, idempotency_key) WHERE idempotency_key IS NOT NULL
|
||||
DO UPDATE SET status='running', started_at=now(), meta_json=EXCLUDED.meta_json`,
|
||||
ON CONFLICT (id) DO UPDATE SET status='running', started_at=COALESCE(job_audit.started_at, now()), meta_json=EXCLUDED.meta_json`,
|
||||
jobID, tenantID, kind, idem, metaJSON)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
package repository
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
jobAuditRetentionDays = 90
|
||||
asnCacheRetentionDays = 7
|
||||
)
|
||||
|
||||
// RunPeriodicMaintenance prunes stale job_audit and asn_prefix_cache rows (PostgreSQL).
|
||||
func (p *Postgres) RunPeriodicMaintenance(ctx context.Context) {
|
||||
if p == nil || p.pool == nil {
|
||||
return
|
||||
}
|
||||
if ctx == nil {
|
||||
ctx = context.Background()
|
||||
}
|
||||
jobCutoff := time.Now().UTC().Add(-time.Duration(jobAuditRetentionDays) * 24 * time.Hour)
|
||||
_, _ = p.pool.Exec(ctx, `
|
||||
DELETE FROM job_audit
|
||||
WHERE created_at < $1
|
||||
AND status IN ('succeeded', 'failed', 'cancelled')`, jobCutoff)
|
||||
asnCutoff := time.Now().UTC().Add(-time.Duration(asnCacheRetentionDays) * 24 * time.Hour)
|
||||
_, _ = p.pool.Exec(ctx, `
|
||||
DELETE FROM asn_prefix_cache WHERE fetched_at < $1`, asnCutoff)
|
||||
}
|
||||
@@ -12,16 +12,24 @@ import (
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
func moduleSnapshotRowTableExists(ctx context.Context, q queryRower) bool {
|
||||
var n int
|
||||
err := q.QueryRow(ctx, `
|
||||
SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = 'public' AND table_name = 'module_prefix_snapshot_row'
|
||||
LIMIT 1`).Scan(&n)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func (p *Postgres) GetModulePrefixSnapshot(tenantID, moduleID string) (*store.ModulePrefixSnapshot, bool, error) {
|
||||
ctx := context.Background()
|
||||
var inputHash string
|
||||
var collectedAt time.Time
|
||||
var raw []byte
|
||||
err := p.pool.QueryRow(ctx, `
|
||||
SELECT input_hash, collected_at, prefixes_json
|
||||
SELECT input_hash, collected_at
|
||||
FROM module_prefix_snapshot
|
||||
WHERE tenant_id = $1 AND module_id = $2`,
|
||||
tenantID, moduleID).Scan(&inputHash, &collectedAt, &raw)
|
||||
tenantID, moduleID).Scan(&inputHash, &collectedAt)
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, false, nil
|
||||
@@ -29,9 +37,31 @@ func (p *Postgres) GetModulePrefixSnapshot(tenantID, moduleID string) (*store.Mo
|
||||
return nil, false, err
|
||||
}
|
||||
var prefixes []store.PrefixRow
|
||||
if len(raw) > 0 {
|
||||
if err := json.Unmarshal(raw, &prefixes); err != nil {
|
||||
return nil, false, err
|
||||
if moduleSnapshotRowTableExists(ctx, p.pool) {
|
||||
rows, qerr := p.pool.Query(ctx, `
|
||||
SELECT prefix, community_id::text, source
|
||||
FROM module_prefix_snapshot_row
|
||||
WHERE tenant_id = $1::uuid AND module_id = $2::uuid
|
||||
ORDER BY ord`, tenantID, moduleID)
|
||||
if qerr != nil {
|
||||
return nil, false, qerr
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var pr store.PrefixRow
|
||||
var comm *string
|
||||
if err := rows.Scan(&pr.Prefix, &comm, &pr.Source); err != nil {
|
||||
continue
|
||||
}
|
||||
pr.CommunityID = comm
|
||||
prefixes = append(prefixes, pr)
|
||||
}
|
||||
} else {
|
||||
var raw []byte
|
||||
if err := p.pool.QueryRow(ctx, `
|
||||
SELECT prefixes_json FROM module_prefix_snapshot
|
||||
WHERE tenant_id = $1 AND module_id = $2`, tenantID, moduleID).Scan(&raw); err == nil && len(raw) > 0 {
|
||||
_ = json.Unmarshal(raw, &prefixes)
|
||||
}
|
||||
}
|
||||
return &store.ModulePrefixSnapshot{
|
||||
@@ -45,20 +75,57 @@ func (p *Postgres) SetModulePrefixSnapshot(tenantID, moduleID, inputHash string,
|
||||
if strings.TrimSpace(tenantID) == "" || strings.TrimSpace(moduleID) == "" || strings.TrimSpace(inputHash) == "" {
|
||||
return store.ErrInvalidInput
|
||||
}
|
||||
raw, err := json.Marshal(prefixes)
|
||||
ctx := context.Background()
|
||||
tx, err := p.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ctx := context.Background()
|
||||
_, err = p.pool.Exec(ctx, `
|
||||
INSERT INTO module_prefix_snapshot (tenant_id, module_id, input_hash, collected_at, prefixes_json)
|
||||
VALUES ($1::uuid, $2::uuid, $3, now(), $4::jsonb)
|
||||
defer func() { _ = tx.Rollback(ctx) }()
|
||||
_, err = tx.Exec(ctx, `
|
||||
INSERT INTO module_prefix_snapshot (tenant_id, module_id, input_hash, collected_at)
|
||||
VALUES ($1::uuid, $2::uuid, $3, now())
|
||||
ON CONFLICT (tenant_id, module_id) DO UPDATE SET
|
||||
input_hash = EXCLUDED.input_hash,
|
||||
collected_at = EXCLUDED.collected_at,
|
||||
prefixes_json = EXCLUDED.prefixes_json`,
|
||||
tenantID, moduleID, inputHash, string(raw))
|
||||
return err
|
||||
collected_at = EXCLUDED.collected_at`,
|
||||
tenantID, moduleID, inputHash)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if moduleSnapshotRowTableExists(ctx, tx) {
|
||||
if _, err := tx.Exec(ctx, `
|
||||
DELETE FROM module_prefix_snapshot_row
|
||||
WHERE tenant_id = $1::uuid AND module_id = $2::uuid`, tenantID, moduleID); err != nil {
|
||||
return err
|
||||
}
|
||||
for i, pr := range prefixes {
|
||||
var comm any
|
||||
if pr.CommunityID != nil && strings.TrimSpace(*pr.CommunityID) != "" {
|
||||
comm = strings.TrimSpace(*pr.CommunityID)
|
||||
}
|
||||
src := pr.Source
|
||||
if strings.TrimSpace(src) == "" {
|
||||
src = "render"
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO module_prefix_snapshot_row (tenant_id, module_id, ord, prefix, community_id, source)
|
||||
VALUES ($1::uuid, $2::uuid, $3, $4, $5::uuid, $6)`,
|
||||
tenantID, moduleID, i, strings.TrimSpace(pr.Prefix), comm, src); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
} else {
|
||||
raw, err := json.Marshal(prefixes)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE module_prefix_snapshot SET prefixes_json = $3::jsonb
|
||||
WHERE tenant_id = $1::uuid AND module_id = $2::uuid`,
|
||||
tenantID, moduleID, string(raw)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
|
||||
func (p *Postgres) DeleteModulePrefixSnapshot(tenantID, moduleID string) error {
|
||||
|
||||
+106
-67
@@ -659,7 +659,8 @@ func (p *Postgres) DeleteSpeaker(tenantID, id string) error {
|
||||
}
|
||||
|
||||
func (p *Postgres) GetRevision(tenantID, revisionID string) (*store.Revision, error) {
|
||||
ctx := context.Background()
|
||||
ctx, cancel := boundedRepoCtx(context.Background())
|
||||
defer cancel()
|
||||
var r store.Revision
|
||||
var mod *string
|
||||
var parent *string
|
||||
@@ -686,11 +687,38 @@ func (p *Postgres) GetRevision(tenantID, revisionID string) (*store.Revision, er
|
||||
if mj.PreviewFragments == nil {
|
||||
mj.PreviewFragments = map[string]string{}
|
||||
}
|
||||
r.PreviewFragments = mj.PreviewFragments
|
||||
r.PreviewFragments = loadRevisionPreview(ctx, p.pool, revisionID, mj.PreviewFragments)
|
||||
r.MaterializedPrefixCount = mj.MaterializedPrefixCount
|
||||
return &r, nil
|
||||
}
|
||||
|
||||
func (p *Postgres) GetRevisionSummary(tenantID, revisionID string) (*store.Revision, error) {
|
||||
ctx, cancel := boundedRepoCtx(context.Background())
|
||||
defer cancel()
|
||||
var r store.Revision
|
||||
var mod *string
|
||||
var parent *string
|
||||
var prefixCount int
|
||||
err := p.pool.QueryRow(ctx, `
|
||||
SELECT id::text, tenant_id::text, module_id::text, content_hash, parent_revision_id::text,
|
||||
COALESCE((meta_json->>'materialized_prefix_count')::int, 0), created_at
|
||||
FROM config_revision WHERE id=$1 AND tenant_id=$2`, revisionID, tenantID).Scan(
|
||||
&r.ID, &r.TenantID, &mod, &r.ContentHash, &parent, &prefixCount, &r.CreatedAt)
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, store.ErrNotFound
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if mod != nil {
|
||||
r.ModuleID = *mod
|
||||
}
|
||||
r.ParentRevisionID = strOrNil(parent)
|
||||
r.MaterializedPrefixCount = prefixCount
|
||||
r.PreviewFragments = map[string]string{}
|
||||
return &r, nil
|
||||
}
|
||||
|
||||
func (p *Postgres) ListRevisions(tenantID, moduleID string, cursor string, limit int) ([]*store.Revision, string, bool) {
|
||||
if limit <= 0 {
|
||||
limit = 50
|
||||
@@ -761,12 +789,7 @@ func (p *Postgres) ListRevisionPrefixes(tenantID, revisionID string, cursor stri
|
||||
if limit <= 0 {
|
||||
limit = 50
|
||||
}
|
||||
off := 0
|
||||
if cursor != "" {
|
||||
if n, err := strconv.Atoi(cursor); err == nil && n >= 0 {
|
||||
off = n
|
||||
}
|
||||
}
|
||||
afterID, off, useOffset := store.ParsePrefixPageCursor(cursor)
|
||||
ctx := context.Background()
|
||||
var one int
|
||||
if err := p.pool.QueryRow(ctx, `
|
||||
@@ -777,34 +800,55 @@ func (p *Postgres) ListRevisionPrefixes(tenantID, revisionID string, cursor stri
|
||||
}
|
||||
return nil, "", false
|
||||
}
|
||||
rows, err := p.pool.Query(ctx, `
|
||||
SELECT prefix::text, community_id::text, source FROM revision_materialized_prefix
|
||||
WHERE revision_id=$1 ORDER BY id
|
||||
LIMIT $2 OFFSET $3`, revisionID, limit+1, off)
|
||||
if snapID, ok := p.revisionPrefixSnapshotID(ctx, revisionID); ok {
|
||||
return p.listSnapshotPrefixes(ctx, snapID, cursor, limit)
|
||||
}
|
||||
var rows pgx.Rows
|
||||
var err error
|
||||
if useOffset {
|
||||
rows, err = p.pool.Query(ctx, `
|
||||
SELECT id, prefix::text, community_id::text, source FROM revision_materialized_prefix
|
||||
WHERE revision_id=$1::uuid ORDER BY id
|
||||
LIMIT $2 OFFSET $3`, revisionID, limit+1, off)
|
||||
} else {
|
||||
var afterArg any
|
||||
if afterID != nil {
|
||||
afterArg = *afterID
|
||||
}
|
||||
rows, err = p.pool.Query(ctx, `
|
||||
SELECT id, prefix::text, community_id::text, source FROM revision_materialized_prefix
|
||||
WHERE revision_id=$1::uuid AND ($2::bigint IS NULL OR id > $2::bigint)
|
||||
ORDER BY id
|
||||
LIMIT $3`, revisionID, afterArg, limit+1)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, "", false
|
||||
}
|
||||
defer rows.Close()
|
||||
var all []store.PrefixRow
|
||||
var ids []int64
|
||||
for rows.Next() {
|
||||
var rowID int64
|
||||
var pr store.PrefixRow
|
||||
var comm *string
|
||||
if err := rows.Scan(&pr.Prefix, &comm, &pr.Source); err != nil {
|
||||
if err := rows.Scan(&rowID, &pr.Prefix, &comm, &pr.Source); err != nil {
|
||||
continue
|
||||
}
|
||||
pr.CommunityID = comm
|
||||
ids = append(ids, rowID)
|
||||
all = append(all, pr)
|
||||
}
|
||||
agentDebugNDJSON3214("B", "repository/postgres.go:ListRevisionPrefixes", "list_prefixes_fetched", map[string]any{
|
||||
"rows": len(all), "limit": limit, "offset": off,
|
||||
"rows": len(all), "limit": limit, "keyset": !useOffset,
|
||||
})
|
||||
more := len(all) > limit
|
||||
if more {
|
||||
all = all[:limit]
|
||||
ids = ids[:limit]
|
||||
}
|
||||
next := ""
|
||||
if more {
|
||||
next = fmt.Sprintf("%d", off+limit)
|
||||
if more && len(ids) > 0 {
|
||||
next = store.FormatPrefixPageCursor(ids[len(ids)-1])
|
||||
}
|
||||
if len(all) == 0 {
|
||||
return nil, "", false
|
||||
@@ -820,26 +864,35 @@ func (p *Postgres) CreateRollbackRevision(tenantID, sourceRevisionID string) (st
|
||||
ctx := context.Background()
|
||||
newID := uuid.NewString()
|
||||
parent := sourceRevisionID
|
||||
meta, _ := json.Marshal(map[string]any{
|
||||
"preview_fragments": src.PreviewFragments,
|
||||
"materialized_prefix_count": src.MaterializedPrefixCount,
|
||||
})
|
||||
meta, err := revisionMetaWithoutPreview(src.MaterializedPrefixCount)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var modArg any
|
||||
if strings.TrimSpace(src.ModuleID) != "" {
|
||||
modArg = src.ModuleID
|
||||
}
|
||||
_, err = p.pool.Exec(ctx, `
|
||||
INSERT INTO config_revision (id, tenant_id, module_id, content_hash, parent_revision_id, meta_json)
|
||||
VALUES ($1,$2,$3,$4,$5::uuid,$6::jsonb)`,
|
||||
newID, tenantID, modArg, src.ContentHash+":rollback", parent, string(meta))
|
||||
tx, err := p.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
// copy materialized prefixes
|
||||
_, _ = p.pool.Exec(ctx, `
|
||||
INSERT INTO revision_materialized_prefix (revision_id, prefix, community_id, source, meta_json)
|
||||
SELECT $1::uuid, prefix, community_id, source, meta_json FROM revision_materialized_prefix WHERE revision_id=$2::uuid`,
|
||||
newID, sourceRevisionID)
|
||||
defer func() { _ = tx.Rollback(ctx) }()
|
||||
_, err = tx.Exec(ctx, `
|
||||
INSERT INTO config_revision (id, tenant_id, module_id, content_hash, parent_revision_id, meta_json)
|
||||
VALUES ($1,$2,$3,$4,$5::uuid,$6::jsonb)`,
|
||||
newID, tenantID, modArg, src.ContentHash+":rollback", parent, meta)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := copyRevisionPreview(ctx, tx, newID, sourceRevisionID); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := p.copyRevisionPrefixSnapshotRef(ctx, tx, newID, sourceRevisionID); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return newID, nil
|
||||
}
|
||||
|
||||
@@ -856,20 +909,18 @@ func (p *Postgres) RevisionDiff(tenantID, aID, bID string) (map[string]any, erro
|
||||
var unchanged int
|
||||
err := p.pool.QueryRow(ctx, `
|
||||
SELECT COUNT(*)::int FROM (
|
||||
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$1::uuid
|
||||
INTERSECT
|
||||
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$2::uuid
|
||||
SELECT b.prefix FROM (`+sqlRevisionPrefixes("$2")+`) b
|
||||
INNER JOIN (`+sqlRevisionPrefixes("$1")+`) a ON a.prefix = b.prefix
|
||||
) t`, aID, bID).Scan(&unchanged)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// added: в B, нет в A; removed: в A, нет в B — без загрузки полных снапшотов в память.
|
||||
rowsAdded, err := p.pool.Query(ctx, `
|
||||
SELECT prefix::text FROM (
|
||||
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$1::uuid
|
||||
EXCEPT
|
||||
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$2::uuid
|
||||
) s ORDER BY 1 LIMIT $3`, bID, aID, maxRevisionDiffRows+1)
|
||||
SELECT b.prefix::text FROM (`+sqlRevisionPrefixes("$2")+`) b
|
||||
LEFT JOIN (`+sqlRevisionPrefixes("$1")+`) a ON a.prefix = b.prefix
|
||||
WHERE a.prefix IS NULL
|
||||
ORDER BY b.prefix
|
||||
LIMIT $3`, aID, bID, maxRevisionDiffRows+1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -888,11 +939,11 @@ func (p *Postgres) RevisionDiff(tenantID, aID, bID string) (map[string]any, erro
|
||||
}
|
||||
addedTruncated := len(added) >= maxRevisionDiffRows
|
||||
rowsRem, err := p.pool.Query(ctx, `
|
||||
SELECT prefix::text FROM (
|
||||
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$1::uuid
|
||||
EXCEPT
|
||||
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$2::uuid
|
||||
) s ORDER BY 1 LIMIT $3`, aID, bID, maxRevisionDiffRows+1)
|
||||
SELECT a.prefix::text FROM (`+sqlRevisionPrefixes("$1")+`) a
|
||||
LEFT JOIN (`+sqlRevisionPrefixes("$2")+`) b ON b.prefix = a.prefix
|
||||
WHERE b.prefix IS NULL
|
||||
ORDER BY a.prefix
|
||||
LIMIT $3`, aID, bID, maxRevisionDiffRows+1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -1028,10 +1079,7 @@ func (p *Postgres) CreateRenderRevision(revisionID, tenantID, moduleID string, p
|
||||
if previewFragments == nil {
|
||||
previewFragments = map[string]string{}
|
||||
}
|
||||
meta, err := json.Marshal(map[string]any{
|
||||
"preview_fragments": previewFragments,
|
||||
"materialized_prefix_count": len(prefixes),
|
||||
})
|
||||
meta, err := revisionMetaWithoutPreview(len(prefixes))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -1045,32 +1093,23 @@ func (p *Postgres) CreateRenderRevision(revisionID, tenantID, moduleID string, p
|
||||
if parentRevisionID != nil && strings.TrimSpace(*parentRevisionID) != "" {
|
||||
parent = strings.TrimSpace(*parentRevisionID)
|
||||
}
|
||||
revID := strings.TrimSpace(revisionID)
|
||||
_, err = tx.Exec(ctx, `
|
||||
INSERT INTO config_revision (id, tenant_id, module_id, content_hash, parent_revision_id, meta_json)
|
||||
VALUES ($1::uuid, $2::uuid, $3::uuid, $4, $5::uuid, $6::jsonb)`,
|
||||
strings.TrimSpace(revisionID), tenantID, moduleID, strings.TrimSpace(contentHash), parent, string(meta))
|
||||
revID, tenantID, moduleID, strings.TrimSpace(contentHash), parent, meta)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(prefixes) > 0 {
|
||||
_, err = tx.CopyFrom(ctx,
|
||||
pgx.Identifier{"revision_materialized_prefix"},
|
||||
[]string{"revision_id", "prefix", "community_id", "source"},
|
||||
pgx.CopyFromSlice(len(prefixes), func(i int) ([]any, error) {
|
||||
pr := prefixes[i]
|
||||
var comm any
|
||||
if pr.CommunityID != nil && strings.TrimSpace(*pr.CommunityID) != "" {
|
||||
comm = strings.TrimSpace(*pr.CommunityID)
|
||||
}
|
||||
src := pr.Source
|
||||
if strings.TrimSpace(src) == "" {
|
||||
src = "render"
|
||||
}
|
||||
return []any{strings.TrimSpace(revisionID), strings.TrimSpace(pr.Prefix), comm, src}, nil
|
||||
}))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := insertRevisionPreview(ctx, tx, revID, previewFragments); err != nil {
|
||||
return err
|
||||
}
|
||||
snapID, err := p.ensurePrefixSnapshot(ctx, tx, contentHash, prefixes)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := p.linkRevisionPrefixSnapshot(ctx, tx, revID, snapID); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return err
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
package repository
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
|
||||
"evobgp/internal/store"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
func prefixSnapshotTableExists(ctx context.Context, q queryRower) bool {
|
||||
var n int
|
||||
err := q.QueryRow(ctx, `
|
||||
SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = 'public' AND table_name = 'prefix_snapshot'
|
||||
LIMIT 1`).Scan(&n)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func normalizeSnapshotHash(contentHash string) string {
|
||||
h := strings.TrimSpace(contentHash)
|
||||
if strings.HasPrefix(h, "sha256:") {
|
||||
h = strings.TrimPrefix(h, "sha256:")
|
||||
}
|
||||
if len(h) > 64 {
|
||||
h = h[:64]
|
||||
}
|
||||
if len(h) < 64 {
|
||||
h = h + strings.Repeat("0", 64-len(h))
|
||||
}
|
||||
return h
|
||||
}
|
||||
|
||||
func (p *Postgres) revisionPrefixSnapshotID(ctx context.Context, revisionID string) (string, bool) {
|
||||
if !prefixSnapshotTableExists(ctx, p.pool) {
|
||||
return "", false
|
||||
}
|
||||
var snap *string
|
||||
err := p.pool.QueryRow(ctx, `
|
||||
SELECT prefix_snapshot_id::text FROM config_revision
|
||||
WHERE id = $1::uuid AND prefix_snapshot_id IS NOT NULL`, revisionID).Scan(&snap)
|
||||
if err != nil || snap == nil || strings.TrimSpace(*snap) == "" {
|
||||
return "", false
|
||||
}
|
||||
return *snap, true
|
||||
}
|
||||
|
||||
func (p *Postgres) ensurePrefixSnapshot(ctx context.Context, db execQuerier, contentHash string, prefixes []store.PrefixRow) (string, error) {
|
||||
if !prefixSnapshotTableExists(ctx, db) {
|
||||
return "", nil
|
||||
}
|
||||
hash := normalizeSnapshotHash(contentHash)
|
||||
var existing string
|
||||
err := db.QueryRow(ctx, `SELECT id::text FROM prefix_snapshot WHERE content_hash = $1`, hash).Scan(&existing)
|
||||
if err == nil && existing != "" {
|
||||
return existing, nil
|
||||
}
|
||||
if err != nil && !errors.Is(err, pgx.ErrNoRows) {
|
||||
return "", err
|
||||
}
|
||||
snapID := uuid.NewString()
|
||||
if _, err := db.Exec(ctx, `
|
||||
INSERT INTO prefix_snapshot (id, content_hash) VALUES ($1::uuid, $2)
|
||||
ON CONFLICT (content_hash) DO NOTHING`, snapID, hash); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := db.QueryRow(ctx, `SELECT id::text FROM prefix_snapshot WHERE content_hash = $1`, hash).Scan(&snapID); err != nil {
|
||||
return "", err
|
||||
}
|
||||
var rowCount int
|
||||
_ = db.QueryRow(ctx, `SELECT COUNT(*)::int FROM prefix_snapshot_row WHERE snapshot_id = $1::uuid`, snapID).Scan(&rowCount)
|
||||
if rowCount > 0 {
|
||||
return snapID, nil
|
||||
}
|
||||
for i, pr := range prefixes {
|
||||
var comm any
|
||||
if pr.CommunityID != nil && strings.TrimSpace(*pr.CommunityID) != "" {
|
||||
comm = strings.TrimSpace(*pr.CommunityID)
|
||||
}
|
||||
src := pr.Source
|
||||
if strings.TrimSpace(src) == "" {
|
||||
src = "render"
|
||||
}
|
||||
if _, err := db.Exec(ctx, `
|
||||
INSERT INTO prefix_snapshot_row (snapshot_id, ord, prefix, community_id, source)
|
||||
VALUES ($1::uuid, $2, $3, $4::uuid, $5)`,
|
||||
snapID, i, strings.TrimSpace(pr.Prefix), comm, src); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
return snapID, nil
|
||||
}
|
||||
|
||||
func (p *Postgres) listSnapshotPrefixes(ctx context.Context, snapshotID, cursor string, limit int) ([]store.PrefixRow, string, bool) {
|
||||
afterOrd, off, useOffset := store.ParsePrefixPageCursor(cursor)
|
||||
var rows pgx.Rows
|
||||
var err error
|
||||
if useOffset {
|
||||
rows, err = p.pool.Query(ctx, `
|
||||
SELECT ord, prefix::text, community_id::text, source
|
||||
FROM prefix_snapshot_row
|
||||
WHERE snapshot_id = $1::uuid
|
||||
ORDER BY ord
|
||||
LIMIT $2 OFFSET $3`, snapshotID, limit+1, off)
|
||||
} else {
|
||||
var afterArg any
|
||||
if afterOrd != nil {
|
||||
afterArg = int(*afterOrd)
|
||||
}
|
||||
rows, err = p.pool.Query(ctx, `
|
||||
SELECT ord, prefix::text, community_id::text, source
|
||||
FROM prefix_snapshot_row
|
||||
WHERE snapshot_id = $1::uuid AND ($2::int IS NULL OR ord > $2::int)
|
||||
ORDER BY ord
|
||||
LIMIT $3`, snapshotID, afterArg, limit+1)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, "", false
|
||||
}
|
||||
defer rows.Close()
|
||||
var all []store.PrefixRow
|
||||
var ords []int64
|
||||
for rows.Next() {
|
||||
var ord int
|
||||
var pr store.PrefixRow
|
||||
var comm *string
|
||||
if err := rows.Scan(&ord, &pr.Prefix, &comm, &pr.Source); err != nil {
|
||||
continue
|
||||
}
|
||||
pr.CommunityID = comm
|
||||
ords = append(ords, int64(ord))
|
||||
all = append(all, pr)
|
||||
}
|
||||
more := len(all) > limit
|
||||
if more {
|
||||
all = all[:limit]
|
||||
ords = ords[:limit]
|
||||
}
|
||||
next := ""
|
||||
if more && len(ords) > 0 {
|
||||
next = store.FormatPrefixPageCursor(ords[len(ords)-1])
|
||||
}
|
||||
if len(all) == 0 {
|
||||
return nil, "", false
|
||||
}
|
||||
return all, next, more
|
||||
}
|
||||
|
||||
func (p *Postgres) linkRevisionPrefixSnapshot(ctx context.Context, db execQuerier, revisionID, snapshotID string) error {
|
||||
if snapshotID == "" || !prefixSnapshotTableExists(ctx, db) {
|
||||
return nil
|
||||
}
|
||||
_, err := db.Exec(ctx, `
|
||||
UPDATE config_revision SET prefix_snapshot_id = $2::uuid WHERE id = $1::uuid`,
|
||||
revisionID, snapshotID)
|
||||
return err
|
||||
}
|
||||
|
||||
func (p *Postgres) copyRevisionPrefixSnapshotRef(ctx context.Context, db execQuerier, dstRevisionID, srcRevisionID string) error {
|
||||
if !prefixSnapshotTableExists(ctx, db) {
|
||||
return nil
|
||||
}
|
||||
_, err := db.Exec(ctx, `
|
||||
UPDATE config_revision dst
|
||||
SET prefix_snapshot_id = src.prefix_snapshot_id
|
||||
FROM config_revision src
|
||||
WHERE dst.id = $1::uuid AND src.id = $2::uuid AND src.prefix_snapshot_id IS NOT NULL`,
|
||||
dstRevisionID, srcRevisionID)
|
||||
return err
|
||||
}
|
||||
|
||||
func sqlRevisionPrefixes(revParam string) string {
|
||||
return `SELECT psr.prefix FROM config_revision cr
|
||||
JOIN prefix_snapshot_row psr ON psr.snapshot_id = cr.prefix_snapshot_id
|
||||
WHERE cr.id = ` + revParam + `::uuid AND cr.prefix_snapshot_id IS NOT NULL
|
||||
UNION ALL
|
||||
SELECT rmp.prefix FROM revision_materialized_prefix rmp
|
||||
WHERE rmp.revision_id = ` + revParam + `::uuid
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM config_revision cr2
|
||||
WHERE cr2.id = ` + revParam + `::uuid AND cr2.prefix_snapshot_id IS NOT NULL
|
||||
)`
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package repository
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
)
|
||||
|
||||
func revisionPreviewTableExists(ctx context.Context, q queryRower) bool {
|
||||
var n int
|
||||
err := q.QueryRow(ctx, `
|
||||
SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = 'public' AND table_name = 'config_revision_preview'
|
||||
LIMIT 1`).Scan(&n)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
type queryRower interface {
|
||||
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
|
||||
}
|
||||
|
||||
func loadRevisionPreview(ctx context.Context, q queryRower, revisionID string, metaPreview map[string]string) map[string]string {
|
||||
if revisionPreviewTableExists(ctx, q) {
|
||||
var raw []byte
|
||||
err := q.QueryRow(ctx, `
|
||||
SELECT fragments FROM config_revision_preview WHERE revision_id = $1::uuid`,
|
||||
revisionID).Scan(&raw)
|
||||
if err == nil {
|
||||
out := map[string]string{}
|
||||
_ = json.Unmarshal(raw, &out)
|
||||
if out == nil {
|
||||
out = map[string]string{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
if !errors.Is(err, pgx.ErrNoRows) {
|
||||
return metaPreview
|
||||
}
|
||||
}
|
||||
if metaPreview == nil {
|
||||
return map[string]string{}
|
||||
}
|
||||
return metaPreview
|
||||
}
|
||||
|
||||
type execQuerier interface {
|
||||
queryRower
|
||||
Exec(ctx context.Context, sql string, arguments ...any) (pgconn.CommandTag, error)
|
||||
}
|
||||
|
||||
func insertRevisionPreview(ctx context.Context, db execQuerier, revisionID string, preview map[string]string) error {
|
||||
if !revisionPreviewTableExists(ctx, db) {
|
||||
return nil
|
||||
}
|
||||
if preview == nil {
|
||||
preview = map[string]string{}
|
||||
}
|
||||
raw, err := json.Marshal(preview)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = db.Exec(ctx, `
|
||||
INSERT INTO config_revision_preview (revision_id, fragments)
|
||||
VALUES ($1::uuid, $2::jsonb)
|
||||
ON CONFLICT (revision_id) DO UPDATE SET fragments = EXCLUDED.fragments`,
|
||||
revisionID, string(raw))
|
||||
return err
|
||||
}
|
||||
|
||||
func copyRevisionPreview(ctx context.Context, db execQuerier, dstRevisionID, srcRevisionID string) error {
|
||||
if !revisionPreviewTableExists(ctx, db) {
|
||||
return nil
|
||||
}
|
||||
_, err := db.Exec(ctx, `
|
||||
INSERT INTO config_revision_preview (revision_id, fragments)
|
||||
SELECT $1::uuid, fragments FROM config_revision_preview WHERE revision_id = $2::uuid
|
||||
ON CONFLICT (revision_id) DO UPDATE SET fragments = EXCLUDED.fragments`,
|
||||
dstRevisionID, srcRevisionID)
|
||||
return err
|
||||
}
|
||||
|
||||
func revisionMetaWithoutPreview(materializedPrefixCount int) (string, error) {
|
||||
raw, err := json.Marshal(map[string]any{
|
||||
"materialized_prefix_count": materializedPrefixCount,
|
||||
})
|
||||
return string(raw), err
|
||||
}
|
||||
@@ -26,9 +26,8 @@ func (p *Postgres) seedDemo(ctx context.Context) error {
|
||||
p1 := uuid.NewString()
|
||||
p2 := uuid.NewString()
|
||||
|
||||
preview := map[string]any{
|
||||
"preview_fragments": map[string]string{
|
||||
"bird.conf": `# EvoBGP demo bundle
|
||||
previewFrags := map[string]string{
|
||||
"bird.conf": `# EvoBGP demo bundle
|
||||
router id 192.0.2.1;
|
||||
|
||||
protocol device {
|
||||
@@ -39,15 +38,12 @@ protocol direct {
|
||||
ipv6;
|
||||
}
|
||||
`,
|
||||
"bird.d/evobgp_demo.conf": "# static demo fragment\n",
|
||||
},
|
||||
"materialized_prefix_count": 128,
|
||||
"bird.d/evobgp_demo.conf": "# static demo fragment\n",
|
||||
}
|
||||
previewB, _ := json.Marshal(preview)
|
||||
parentMeta, _ := json.Marshal(map[string]any{
|
||||
"preview_fragments": map[string]string{"bird.conf": "# parent revision\n"},
|
||||
"materialized_prefix_count": 0,
|
||||
})
|
||||
previewMeta, _ := json.Marshal(map[string]any{"materialized_prefix_count": 128})
|
||||
parentMeta, _ := json.Marshal(map[string]any{"materialized_prefix_count": 0})
|
||||
parentPreview, _ := json.Marshal(map[string]string{"bird.conf": "# parent revision\n"})
|
||||
previewFragsB, _ := json.Marshal(previewFrags)
|
||||
|
||||
tx, err := p.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
@@ -77,13 +73,32 @@ protocol direct {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO config_revision (id, tenant_id, content_hash, parent_revision_id, meta_json)
|
||||
VALUES ($1,$2,'sha256:demo-rev-1',$3::uuid,$4::jsonb)`, rid, tid, parent, string(previewB)); err != nil {
|
||||
INSERT INTO config_revision_preview (revision_id, fragments) VALUES ($1::uuid, $2::jsonb)`, parent, string(parentPreview)); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO revision_materialized_prefix (revision_id, prefix, community_id, source)
|
||||
VALUES ($1::uuid,'203.0.113.0/24',$2::uuid,'demo'), ($1::uuid,'2001:db8::/32',$2::uuid,'demo')`, rid, cid); err != nil {
|
||||
INSERT INTO config_revision (id, tenant_id, content_hash, parent_revision_id, meta_json)
|
||||
VALUES ($1,$2,'sha256:demo-rev-1',$3::uuid,$4::jsonb)`, rid, tid, parent, string(previewMeta)); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO config_revision_preview (revision_id, fragments) VALUES ($1::uuid, $2::jsonb)`, rid, string(previewFragsB)); err != nil {
|
||||
return err
|
||||
}
|
||||
snapID := uuid.NewString()
|
||||
demoHash := normalizeSnapshotHash("sha256:demo-rev-1")
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO prefix_snapshot (id, content_hash) VALUES ($1::uuid, $2)`, snapID, demoHash); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE config_revision SET prefix_snapshot_id = $2::uuid WHERE id = $1::uuid`, rid, snapID); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO prefix_snapshot_row (snapshot_id, ord, prefix, community_id, source)
|
||||
VALUES ($1::uuid, 0, '203.0.113.0/24', $2::uuid, 'demo'),
|
||||
($1::uuid, 1, '2001:db8::/32', $2::uuid, 'demo')`, snapID, cid); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
|
||||
"evobgp/internal/broker"
|
||||
"evobgp/internal/config"
|
||||
"evobgp/internal/httpclient"
|
||||
"evobgp/internal/jobs"
|
||||
"evobgp/internal/pipeline"
|
||||
"evobgp/internal/store"
|
||||
@@ -108,6 +109,9 @@ func postTenantRefresh(ctx context.Context, deps *Deps, moduleIDs []string, idem
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.GetBody = func() (io.ReadCloser, error) {
|
||||
return io.NopCloser(bytes.NewReader(body)), nil
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+strings.TrimSpace(deps.APIToken))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
if idempotencyKey != "" {
|
||||
@@ -115,9 +119,9 @@ func postTenantRefresh(ctx context.Context, deps *Deps, moduleIDs []string, idem
|
||||
}
|
||||
hc := deps.HTTP
|
||||
if hc == nil {
|
||||
hc = http.DefaultClient
|
||||
hc = httpclient.New(httpclient.DefaultTimeout)
|
||||
}
|
||||
resp, err := hc.Do(req)
|
||||
resp, err := httpclient.DoWithRetry(ctx, hc, req, 3)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -76,6 +76,8 @@ type Backend interface {
|
||||
DeleteSpeaker(tenantID, id string) error
|
||||
|
||||
GetRevision(tenantID, revisionID string) (*Revision, error)
|
||||
// GetRevisionSummary returns revision metadata without preview_fragments payloads.
|
||||
GetRevisionSummary(tenantID, revisionID string) (*Revision, error)
|
||||
ListRevisions(tenantID, moduleID string, cursor string, limit int) (items []*Revision, nextCursor string, hasMore bool)
|
||||
ListRevisionPrefixes(tenantID, revisionID string, cursor string, limit int) (prefixes []PrefixRow, next string, more bool)
|
||||
CreateRollbackRevision(tenantID, sourceRevisionID string) (newID string, err error)
|
||||
@@ -110,6 +112,9 @@ type Backend interface {
|
||||
|
||||
// Ping verifies backend connectivity (no-op for in-memory).
|
||||
Ping(ctx context.Context) error
|
||||
|
||||
// RunPeriodicMaintenance prunes stale DB rows (no-op for in-memory).
|
||||
RunPeriodicMaintenance(ctx context.Context)
|
||||
}
|
||||
|
||||
// ASNPrefixCacheEntry is a cached RIPEstat response for one ASN.
|
||||
@@ -309,7 +314,7 @@ type SpeakerPatch struct {
|
||||
|
||||
// PrefixRow is one materialized prefix for GET /revisions/.../prefixes.
|
||||
type PrefixRow struct {
|
||||
Prefix string
|
||||
CommunityID *string
|
||||
Source string
|
||||
Prefix string `json:"prefix"`
|
||||
CommunityID *string `json:"community_id,omitempty"`
|
||||
Source string `json:"source,omitempty"`
|
||||
}
|
||||
|
||||
@@ -305,6 +305,11 @@ func (m *Memory) Ping(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// RunPeriodicMaintenance is a no-op for the in-memory backend.
|
||||
func (m *Memory) RunPeriodicMaintenance(ctx context.Context) {
|
||||
_ = ctx
|
||||
}
|
||||
|
||||
// ListTenantIDs returns tenant ids sorted lexicographically.
|
||||
func (m *Memory) ListTenantIDs() ([]string, error) {
|
||||
m.mu.RLock()
|
||||
@@ -424,14 +429,19 @@ func (m *Memory) GetModule(tenantID, moduleID string) (*Module, error) {
|
||||
func (m *Memory) GetRevision(tenantID, revisionID string) (*Revision, error) {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
rev, ok := m.revisions[revisionID]
|
||||
if !ok {
|
||||
return nil, ErrNotFound
|
||||
return m.getRevisionLocked(tenantID, revisionID)
|
||||
}
|
||||
|
||||
func (m *Memory) GetRevisionSummary(tenantID, revisionID string) (*Revision, error) {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
rev, err := m.getRevisionLocked(tenantID, revisionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if rev.TenantID != tenantID {
|
||||
return nil, ErrTenantScope
|
||||
}
|
||||
return rev, nil
|
||||
cp := *rev
|
||||
cp.PreviewFragments = nil
|
||||
return &cp, nil
|
||||
}
|
||||
|
||||
func (m *Memory) GetSpeaker(tenantID, speakerID string) (*Speaker, error) {
|
||||
|
||||
@@ -1,8 +1,6 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -795,31 +793,37 @@ func (m *Memory) ListRevisionPrefixes(tenantID, revisionID string, cursor string
|
||||
if limit <= 0 {
|
||||
limit = 50
|
||||
}
|
||||
afterID, off, useOffset := ParsePrefixPageCursor(cursor)
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
if _, err := m.getRevisionLocked(tenantID, revisionID); err != nil {
|
||||
return nil, "", false
|
||||
}
|
||||
all := m.revPrefixes[revisionID]
|
||||
off := 0
|
||||
if cursor != "" {
|
||||
if n, err := strconv.Atoi(cursor); err == nil && n >= 0 {
|
||||
off = n
|
||||
}
|
||||
allRows := m.revPrefixes[revisionID]
|
||||
start := 0
|
||||
if useOffset {
|
||||
start = off
|
||||
} else if afterID != nil {
|
||||
start = int(*afterID) + 1
|
||||
}
|
||||
end := off + limit
|
||||
next := ""
|
||||
more := false
|
||||
if end > len(all) {
|
||||
end = len(all)
|
||||
} else {
|
||||
more = true
|
||||
next = fmt.Sprintf("%d", end)
|
||||
}
|
||||
if off >= len(all) {
|
||||
if start > len(allRows) {
|
||||
return nil, "", false
|
||||
}
|
||||
return all[off:end], next, more
|
||||
end := start + limit
|
||||
next := ""
|
||||
more := false
|
||||
if end > len(allRows) {
|
||||
end = len(allRows)
|
||||
} else {
|
||||
more = true
|
||||
next = FormatPrefixPageCursor(int64(end - 1))
|
||||
}
|
||||
if start >= end {
|
||||
return nil, "", false
|
||||
}
|
||||
out := make([]PrefixRow, end-start)
|
||||
copy(out, allRows[start:end])
|
||||
return out, next, more
|
||||
}
|
||||
|
||||
func (m *Memory) ListGlobalSettings(tenantID string) (map[string]any, error) {
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ParsePrefixPageCursor decodes opaque cursors for revision prefix pagination.
|
||||
func ParsePrefixPageCursor(cursor string) (afterID *int64, offset int, useOffset bool) {
|
||||
cursor = strings.TrimSpace(cursor)
|
||||
if cursor == "" {
|
||||
return nil, 0, false
|
||||
}
|
||||
if strings.HasPrefix(cursor, "o:") {
|
||||
n, err := strconv.Atoi(strings.TrimPrefix(cursor, "o:"))
|
||||
if err != nil || n < 0 {
|
||||
return nil, 0, false
|
||||
}
|
||||
return nil, n, true
|
||||
}
|
||||
if n, err := strconv.ParseInt(cursor, 10, 64); err == nil && n >= 0 {
|
||||
return &n, 0, false
|
||||
}
|
||||
return nil, 0, false
|
||||
}
|
||||
|
||||
// FormatPrefixPageCursor encodes the keyset cursor (last row id or slice index).
|
||||
func FormatPrefixPageCursor(lastID int64) string {
|
||||
return strconv.FormatInt(lastID, 10)
|
||||
}
|
||||
@@ -72,7 +72,9 @@ func MergeSpeakerMetaJSON(existing string, patch SpeakerMeta) string {
|
||||
if patch.LastDispatchAt != "" {
|
||||
cur.LastDispatchAt = patch.LastDispatchAt
|
||||
}
|
||||
if patch.LastDispatchError != "" {
|
||||
if patch.LastDispatchStatus == "ok" {
|
||||
cur.LastDispatchError = ""
|
||||
} else if patch.LastDispatchError != "" {
|
||||
cur.LastDispatchError = patch.LastDispatchError
|
||||
}
|
||||
if patch.LastDispatchStatus != "" {
|
||||
@@ -124,11 +126,20 @@ func AgentSyncURL(meta SpeakerMeta) string {
|
||||
|
||||
// AgentHealthURL returns HTTPS health URL for agent_domain.
|
||||
func AgentHealthURL(meta SpeakerMeta) string {
|
||||
return agentHTTPSURL(meta, "/v1/agent/health")
|
||||
}
|
||||
|
||||
// AgentBirdProtocolsURL returns HTTPS bird protocols URL for agent_domain.
|
||||
func AgentBirdProtocolsURL(meta SpeakerMeta) string {
|
||||
return agentHTTPSURL(meta, "/v1/agent/bird/protocols")
|
||||
}
|
||||
|
||||
func agentHTTPSURL(meta SpeakerMeta, path string) string {
|
||||
domain := strings.TrimSpace(meta.AgentDomain)
|
||||
if domain == "" {
|
||||
return ""
|
||||
}
|
||||
return "https://" + strings.TrimSuffix(domain, "/") + "/v1/agent/health"
|
||||
return "https://" + strings.TrimSuffix(domain, "/") + path
|
||||
}
|
||||
|
||||
// SpeakerNeedsRemoteDispatch reports whether deploy_apply should wake this speaker via agent HTTP.
|
||||
|
||||
@@ -34,3 +34,24 @@ func TestAgentSyncURL(t *testing.T) {
|
||||
t.Fatalf("got %q", u)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeSpeakerMetaJSON_clearsDispatchErrorOnOk(t *testing.T) {
|
||||
t.Parallel()
|
||||
existing := store.SpeakerMetaJSON(store.SpeakerMeta{
|
||||
LastDispatchError: "HTTP 502: bundle 403",
|
||||
LastDispatchStatus: "error",
|
||||
SyncStatus: "error",
|
||||
})
|
||||
merged := store.MergeSpeakerMetaJSON(existing, store.SpeakerMeta{
|
||||
LastDispatchStatus: "ok",
|
||||
SyncStatus: "synced",
|
||||
LastDispatchAt: "2026-05-21T15:06:43Z",
|
||||
})
|
||||
m := store.ParseSpeakerMeta(merged)
|
||||
if m.LastDispatchError != "" {
|
||||
t.Fatalf("LastDispatchError should clear on ok dispatch, got %q", m.LastDispatchError)
|
||||
}
|
||||
if m.LastDispatchStatus != "ok" || m.SyncStatus != "synced" {
|
||||
t.Fatalf("status: dispatch=%q sync=%q", m.LastDispatchStatus, m.SyncStatus)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
DROP INDEX IF EXISTS idx_rev_mat_prefix_uniq;
|
||||
DROP INDEX IF EXISTS idx_rev_mat_prefix_rev_cover;
|
||||
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_rev_id
|
||||
ON revision_materialized_prefix (revision_id, id);
|
||||
DROP INDEX IF EXISTS idx_module_tenant_active_sort;
|
||||
DROP INDEX IF EXISTS idx_rev_mat_prefix_rev_prefix;
|
||||
@@ -0,0 +1,17 @@
|
||||
-- Query indexes: revision diff/list and module sort (H4 + M4).
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_rev_prefix
|
||||
ON revision_materialized_prefix (revision_id, prefix);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_module_tenant_active_sort
|
||||
ON module (tenant_id, priority, name)
|
||||
WHERE deleted_at IS NULL;
|
||||
|
||||
-- Covering index for prefix list pagination (PostgreSQL 11+ INCLUDE).
|
||||
DROP INDEX IF EXISTS idx_rev_mat_prefix_rev_id;
|
||||
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_rev_cover
|
||||
ON revision_materialized_prefix (revision_id, id)
|
||||
INCLUDE (prefix, community_id, source);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_rev_mat_prefix_uniq
|
||||
ON revision_materialized_prefix (revision_id, prefix, community_id, source);
|
||||
@@ -0,0 +1,6 @@
|
||||
UPDATE config_revision cr
|
||||
SET meta_json = cr.meta_json || jsonb_build_object('preview_fragments', COALESCE(p.fragments, '{}'::jsonb))
|
||||
FROM config_revision_preview p
|
||||
WHERE p.revision_id = cr.id;
|
||||
|
||||
DROP TABLE IF EXISTS config_revision_preview;
|
||||
@@ -0,0 +1,15 @@
|
||||
-- Split BIRD preview fragments out of config_revision.meta_json (H1).
|
||||
|
||||
CREATE TABLE config_revision_preview (
|
||||
revision_id UUID PRIMARY KEY REFERENCES config_revision (id) ON DELETE CASCADE,
|
||||
fragments JSONB NOT NULL DEFAULT '{}'
|
||||
);
|
||||
|
||||
INSERT INTO config_revision_preview (revision_id, fragments)
|
||||
SELECT id, COALESCE(meta_json->'preview_fragments', '{}'::jsonb)
|
||||
FROM config_revision
|
||||
WHERE meta_json ? 'preview_fragments';
|
||||
|
||||
UPDATE config_revision
|
||||
SET meta_json = meta_json - 'preview_fragments'
|
||||
WHERE meta_json ? 'preview_fragments';
|
||||
@@ -0,0 +1,3 @@
|
||||
ALTER TABLE config_revision DROP COLUMN IF EXISTS prefix_snapshot_id;
|
||||
DROP TABLE IF EXISTS prefix_snapshot_row;
|
||||
DROP TABLE IF EXISTS prefix_snapshot;
|
||||
@@ -0,0 +1,22 @@
|
||||
-- Content-addressed prefix snapshots (H2 expand).
|
||||
|
||||
CREATE TABLE prefix_snapshot (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
content_hash CHAR(64) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CONSTRAINT prefix_snapshot_hash_uniq UNIQUE (content_hash)
|
||||
);
|
||||
|
||||
CREATE TABLE prefix_snapshot_row (
|
||||
snapshot_id UUID NOT NULL REFERENCES prefix_snapshot (id) ON DELETE CASCADE,
|
||||
ord INTEGER NOT NULL,
|
||||
prefix TEXT NOT NULL,
|
||||
community_id UUID REFERENCES bgp_community (id) ON DELETE SET NULL,
|
||||
source TEXT NOT NULL DEFAULT '',
|
||||
PRIMARY KEY (snapshot_id, ord)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_prefix_snapshot_row_snapshot ON prefix_snapshot_row (snapshot_id);
|
||||
|
||||
ALTER TABLE config_revision
|
||||
ADD COLUMN prefix_snapshot_id UUID REFERENCES prefix_snapshot (id) ON DELETE RESTRICT;
|
||||
@@ -0,0 +1,3 @@
|
||||
DELETE FROM prefix_snapshot_row;
|
||||
UPDATE config_revision SET prefix_snapshot_id = NULL WHERE prefix_snapshot_id IS NOT NULL;
|
||||
DELETE FROM prefix_snapshot;
|
||||
@@ -0,0 +1,35 @@
|
||||
-- Backfill prefix snapshots from revision_materialized_prefix.
|
||||
-- prefix_snapshot_row.prefix is TEXT (revision_materialized_prefix.prefix since 000003).
|
||||
|
||||
ALTER TABLE prefix_snapshot_row
|
||||
ALTER COLUMN prefix TYPE TEXT USING prefix::text;
|
||||
|
||||
WITH new_snaps AS (
|
||||
INSERT INTO prefix_snapshot (id, content_hash)
|
||||
SELECT gen_random_uuid(),
|
||||
substr(replace(cr.id::text, '-', '') || replace(cr.id::text, '-', ''), 1, 64)
|
||||
FROM config_revision cr
|
||||
WHERE cr.prefix_snapshot_id IS NULL
|
||||
AND EXISTS (
|
||||
SELECT 1 FROM revision_materialized_prefix rmp WHERE rmp.revision_id = cr.id
|
||||
)
|
||||
RETURNING id, content_hash
|
||||
)
|
||||
UPDATE config_revision cr
|
||||
SET prefix_snapshot_id = ns.id
|
||||
FROM new_snaps ns
|
||||
WHERE cr.prefix_snapshot_id IS NULL
|
||||
AND ns.content_hash = substr(replace(cr.id::text, '-', '') || replace(cr.id::text, '-', ''), 1, 64);
|
||||
|
||||
INSERT INTO prefix_snapshot_row (snapshot_id, ord, prefix, community_id, source)
|
||||
SELECT cr.prefix_snapshot_id,
|
||||
(row_number() OVER (PARTITION BY cr.id ORDER BY rmp.id) - 1)::int,
|
||||
rmp.prefix,
|
||||
rmp.community_id,
|
||||
rmp.source
|
||||
FROM config_revision cr
|
||||
JOIN revision_materialized_prefix rmp ON rmp.revision_id = cr.id
|
||||
WHERE cr.prefix_snapshot_id IS NOT NULL
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM prefix_snapshot_row psr WHERE psr.snapshot_id = cr.prefix_snapshot_id
|
||||
);
|
||||
@@ -0,0 +1 @@
|
||||
COMMENT ON COLUMN config_revision.prefix_snapshot_id IS NULL;
|
||||
@@ -0,0 +1,3 @@
|
||||
-- H2 contract marker: new revisions use prefix_snapshot_id only (enforced in application code).
|
||||
|
||||
COMMENT ON COLUMN config_revision.prefix_snapshot_id IS 'Materialized prefixes; revision_materialized_prefix deprecated for new rows';
|
||||
@@ -0,0 +1,16 @@
|
||||
ALTER TABLE module_prefix_snapshot ADD COLUMN prefixes_json JSONB NOT NULL DEFAULT '[]';
|
||||
|
||||
UPDATE module_prefix_snapshot mps
|
||||
SET prefixes_json = COALESCE((
|
||||
SELECT jsonb_agg(
|
||||
jsonb_build_object(
|
||||
'prefix', psr.prefix::text,
|
||||
'community_id', psr.community_id,
|
||||
'source', psr.source
|
||||
) ORDER BY psr.ord
|
||||
)
|
||||
FROM module_prefix_snapshot_row psr
|
||||
WHERE psr.tenant_id = mps.tenant_id AND psr.module_id = mps.module_id
|
||||
), '[]'::jsonb);
|
||||
|
||||
DROP TABLE IF EXISTS module_prefix_snapshot_row;
|
||||
@@ -0,0 +1,46 @@
|
||||
CREATE TABLE module_prefix_snapshot_row (
|
||||
tenant_id UUID NOT NULL,
|
||||
module_id UUID NOT NULL,
|
||||
ord INTEGER NOT NULL,
|
||||
prefix TEXT NOT NULL,
|
||||
community_id UUID,
|
||||
source TEXT NOT NULL DEFAULT '',
|
||||
PRIMARY KEY (tenant_id, module_id, ord),
|
||||
FOREIGN KEY (tenant_id, module_id)
|
||||
REFERENCES module_prefix_snapshot (tenant_id, module_id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- prefixes_json from Go json.Marshal(PrefixRow) used "Prefix"/"CommunityID"/"Source" before json tags.
|
||||
INSERT INTO module_prefix_snapshot_row (tenant_id, module_id, ord, prefix, community_id, source)
|
||||
SELECT tenant_id,
|
||||
module_id,
|
||||
(row_number() OVER (PARTITION BY tenant_id, module_id ORDER BY ordinality) - 1)::int,
|
||||
prefix,
|
||||
NULLIF(community_id, '')::uuid,
|
||||
COALESCE(source, '')
|
||||
FROM (
|
||||
SELECT mps.tenant_id,
|
||||
mps.module_id,
|
||||
t.ordinality,
|
||||
COALESCE(
|
||||
NULLIF(trim(t.elem->>'prefix'), ''),
|
||||
NULLIF(trim(t.elem->>'Prefix'), '')
|
||||
) AS prefix,
|
||||
COALESCE(
|
||||
NULLIF(trim(t.elem->>'community_id'), ''),
|
||||
NULLIF(trim(t.elem->>'CommunityID'), '')
|
||||
) AS community_id,
|
||||
COALESCE(
|
||||
NULLIF(trim(t.elem->>'source'), ''),
|
||||
NULLIF(trim(t.elem->>'Source'), ''),
|
||||
''
|
||||
) AS source
|
||||
FROM module_prefix_snapshot mps
|
||||
CROSS JOIN LATERAL jsonb_array_elements(mps.prefixes_json) WITH ORDINALITY AS t(elem, ordinality)
|
||||
WHERE jsonb_typeof(mps.prefixes_json) = 'array'
|
||||
AND jsonb_array_length(mps.prefixes_json) > 0
|
||||
) parsed
|
||||
WHERE parsed.prefix IS NOT NULL
|
||||
AND trim(parsed.prefix) <> '';
|
||||
|
||||
ALTER TABLE module_prefix_snapshot DROP COLUMN prefixes_json;
|
||||
@@ -0,0 +1,10 @@
|
||||
ALTER TABLE asn_prefix_cache ADD COLUMN prefixes_json JSONB NOT NULL DEFAULT '[]';
|
||||
|
||||
UPDATE asn_prefix_cache apc
|
||||
SET prefixes_json = COALESCE((
|
||||
SELECT jsonb_agg(apcr.prefix::text ORDER BY apcr.prefix::text)
|
||||
FROM asn_prefix_cache_row apcr
|
||||
WHERE apcr.asn = apc.asn
|
||||
), '[]'::jsonb);
|
||||
|
||||
DROP TABLE IF EXISTS asn_prefix_cache_row;
|
||||
@@ -0,0 +1,14 @@
|
||||
CREATE TABLE asn_prefix_cache_row (
|
||||
asn BIGINT NOT NULL REFERENCES asn_prefix_cache (asn) ON DELETE CASCADE,
|
||||
prefix CIDR NOT NULL,
|
||||
PRIMARY KEY (asn, prefix)
|
||||
);
|
||||
|
||||
INSERT INTO asn_prefix_cache_row (asn, prefix)
|
||||
SELECT apc.asn, t.elem::cidr
|
||||
FROM asn_prefix_cache apc
|
||||
CROSS JOIN LATERAL jsonb_array_elements_text(apc.prefixes_json) AS t(elem)
|
||||
WHERE jsonb_typeof(apc.prefixes_json) = 'array'
|
||||
AND jsonb_array_length(apc.prefixes_json) > 0;
|
||||
|
||||
ALTER TABLE asn_prefix_cache DROP COLUMN prefixes_json;
|
||||
@@ -0,0 +1 @@
|
||||
DROP INDEX IF EXISTS idx_job_audit_created_brin;
|
||||
@@ -0,0 +1,2 @@
|
||||
CREATE INDEX IF NOT EXISTS idx_job_audit_created_brin
|
||||
ON job_audit USING BRIN (created_at);
|
||||
@@ -0,0 +1,18 @@
|
||||
CREATE TABLE IF NOT EXISTS module_cdn_fetch_log (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
module_id UUID NOT NULL REFERENCES module (id) ON DELETE CASCADE,
|
||||
source_id UUID REFERENCES module_cdn_source (id) ON DELETE SET NULL,
|
||||
http_status INTEGER,
|
||||
bytes BIGINT,
|
||||
error TEXT,
|
||||
fetched_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_module_cdn_fetch_log_module ON module_cdn_fetch_log (module_id, fetched_at DESC);
|
||||
|
||||
ALTER TABLE revision_materialized_prefix ADD COLUMN IF NOT EXISTS meta_json JSONB NOT NULL DEFAULT '{}';
|
||||
ALTER TABLE module_domain_entry ADD COLUMN IF NOT EXISTS resolve_meta JSONB NOT NULL DEFAULT '{}';
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_revision ON revision_materialized_prefix (revision_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_module_tenant ON module (tenant_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_module_prefix_snapshot_collected ON module_prefix_snapshot (collected_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_value ON revision_materialized_prefix (prefix);
|
||||
@@ -0,0 +1,8 @@
|
||||
ALTER TABLE revision_materialized_prefix DROP COLUMN IF EXISTS meta_json;
|
||||
ALTER TABLE module_domain_entry DROP COLUMN IF EXISTS resolve_meta;
|
||||
DROP TABLE IF EXISTS module_cdn_fetch_log;
|
||||
|
||||
DROP INDEX IF EXISTS idx_rev_mat_prefix_revision;
|
||||
DROP INDEX IF EXISTS idx_module_tenant;
|
||||
DROP INDEX IF EXISTS idx_module_prefix_snapshot_collected;
|
||||
DROP INDEX IF EXISTS idx_rev_mat_prefix_value;
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user