Compare commits

..
26 Commits
Author SHA1 Message Date
Denozordec aa4e3d0180 docs(agents): update engineering rules and add Context7 documentation references
CI / changes (push) Successful in 9s
CI / commitlint (push) Has been skipped
CI / openapi (push) Successful in 25s
CI / web (push) Successful in 31s
CI / go (push) Successful in 2m14s
CI / bird2 (push) Successful in 15s
CI / release (push) Successful in 3m53s
Added Context7 documentation links for stack IDs and skills to the agents guide, enhancing clarity on library usage and integration.
2026-06-12 13:48:37 +07:00
DenozordecandCursor 1ccffc85da test(maintenance): add policy executor and handler tests
Табличные тесты PolicyExecutor, ConfigProvider reload, memory CRUD политик и 503 для /v1/maintenance/* на memory-бэкенде без PostgreSQL.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-12 13:32:01 +07:00
DenozordecandCursor d38ee68c4e feat(web): add maintenance policies UI
Вкладка политик обслуживания PostgreSQL: CRUD через /v1/maintenance/policies, run/dry-run, форма с Zod. Hardcoded кнопки vacuum/cleanup в MonitoringPostgresTab заменены на MaintenancePoliciesTab.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-12 13:30:43 +07:00
DenozordecandCursor aaef47c7a7 feat(observability): add maintenance policy metrics
Prometheus: runs, duration, rows_deleted, config_changes; инкремент при CRUD и Execute.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-12 13:28:20 +07:00
DenozordecandCursor cbf345b25f refactor(maintenance): remove hardcoded retention and wire scheduler
RunPeriodicMaintenance и RunCleanup удалены; scheduler политик в StartBackground; deprecated /postgres/cleanup принимает policy_id.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-12 13:27:49 +07:00
DenozordecandCursor f548d0671f feat(api): add /v1/maintenance policies and run endpoints
OpenAPI, httpapi CRUD/run/dry-run, job maintenance_policy_run и audit с policy_id.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-12 13:23:10 +07:00
DenozordecandCursor 6510a9ca22 feat(maintenance): add policy executor and config provider
ConfigProvider, PolicyExecutor, DBStatsProvider, scheduler и safety; зависимость robfig/cron/v3.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-12 13:21:32 +07:00
DenozordecandCursor 07c3de4939 feat(store): add MaintenancePolicy CRUD backend
Типы maintenance_policy, методы store.Backend и реализации для PostgreSQL и in-memory.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-12 13:20:27 +07:00
DenozordecandCursor 948dac34fd feat(db): add maintenance_policy migration 000025
Добавлены таблицы maintenance_policy и maintenance_policy_config_audit для postgres и sqlite; в postgres_maintenance_audit — колонка policy_id.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-12 13:19:27 +07:00
Denozordec 480756d832 feat(settings): enhance revision retention minutes validation
CI / changes (push) Successful in 8s
CI / commitlint (push) Has been skipped
CI / openapi (push) Has been skipped
CI / web (push) Successful in 29s
CI / go (push) Has been skipped
CI / bird2 (push) Has been skipped
CI / release (push) Successful in 3m14s
Introduced a preprocessing function to normalize input for the revision retention minutes field, ensuring it handles various input types correctly. Updated the schema to utilize this new validation method, improving data integrity and user experience.
2026-06-01 14:35:51 +07:00
Denozordec 135fb34e00 fix(pgmonitor): standardize field alignment in QueriesResponse struct
CI / changes (push) Successful in 7s
CI / commitlint (push) Has been skipped
CI / openapi (push) Has been skipped
CI / web (push) Has been skipped
CI / go (push) Successful in 54s
CI / bird2 (push) Successful in 15s
CI / release (push) Successful in 3m46s
2026-06-01 14:18:03 +07:00
Denozordec 9efa3bbc8a feat(db): enhance PostgreSQL statistics monitoring and error handling
CI / changes (push) Successful in 8s
CI / commitlint (push) Has been skipped
CI / openapi (push) Has been skipped
CI / web (push) Successful in 30s
CI / go (push) Failing after 24s
CI / bird2 (push) Has been skipped
CI / release (push) Has been skipped
Updated the PostgreSQL monitoring service to improve handling of `pg_stat_statements` availability. Introduced a new method to check if the extension is queryable and updated the response structure to include availability status and hints. Enhanced the documentation to clarify the requirements for enabling `pg_stat_statements`. Adjusted related components to reflect these changes, ensuring better user feedback in the monitoring interface.
2026-06-01 14:15:38 +07:00
Denozordec fad2bd3353 feat(db): implement PostgreSQL monitoring and maintenance features
CI / changes (push) Successful in 9s
CI / commitlint (push) Has been skipped
CI / openapi (push) Successful in 26s
CI / web (push) Successful in 33s
CI / go (push) Successful in 2m11s
CI / bird2 (push) Successful in 16s
CI / release (push) Successful in 3m27s
Added PostgreSQL monitoring and maintenance capabilities to the API, including new endpoints for instance-level metrics, maintenance operations, and job scheduling. Updated the HTTP API to support PostgreSQL monitoring routes and integrated a background scheduler for metrics collection. Enhanced the CLI with database commands for maintenance tasks. Updated documentation to reflect these changes.
2026-06-01 13:43:33 +07:00
Denozordec 930e42b0b0 fix(db): update prefix handling in module_prefix_snapshot_row
CI / changes (push) Successful in 8s
CI / commitlint (push) Has been skipped
CI / openapi (push) Has been skipped
CI / web (push) Has been skipped
CI / go (push) Successful in 55s
CI / bird2 (push) Successful in 14s
CI / release (push) Successful in 3m21s
Modified the prefix column type in the module_prefix_snapshot_row table to TEXT, allowing for more flexible input. Adjusted related SQL queries and Go struct tags to ensure compatibility with JSON serialization. Cleaned up migration logic to handle prefix and community_id fields more robustly.
2026-05-25 11:16:03 +07:00
Denozordec 16b4923bd7 fix(db): change prefix column type to TEXT in prefix_snapshot_row
CI / changes (push) Successful in 8s
CI / openapi (push) Has been skipped
CI / commitlint (push) Has been skipped
CI / web (push) Has been skipped
CI / go (push) Successful in 54s
CI / bird2 (push) Successful in 14s
CI / release (push) Successful in 3m44s
Updated the prefix column in the prefix_snapshot_row table from CIDR to TEXT to accommodate broader input formats. Adjusted related SQL insert statements accordingly.
2026-05-25 11:08:06 +07:00
DenozordecandCursor 1cfd062835 chore(db): drop unused schema and redundant indexes
CI / changes (push) Successful in 8s
CI / commitlint (push) Has been skipped
CI / openapi (push) Has been skipped
CI / web (push) Has been skipped
CI / go (push) Successful in 54s
CI / bird2 (push) Successful in 14s
CI / release (push) Successful in 3m44s
Удалены мёртвые колонки, fetch_log и избыточные индексы.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:57 +07:00
DenozordecandCursor 21233bd578 chore(db): add job_audit brin index
BRIN-индекс по created_at для retention-удалений.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:57 +07:00
DenozordecandCursor 990cc739df refactor(db): normalize asn prefix cache and ttl cleanup
Строки asn_prefix_cache_row; периодический prune через evobgp-ingest.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:57 +07:00
DenozordecandCursor 3500bd4624 refactor(db): normalize module prefix snapshot rows
Строки префиксов в module_prefix_snapshot_row вместо JSONB blobs.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:56 +07:00
DenozordecandCursor 374575ec01 refactor(db): mark prefix snapshot contract migration
Маркер миграции contract-фазы H2.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:01 +07:00
DenozordecandCursor f57b430052 feat(db): backfill prefix snapshots and switch read path
Backfill миграция; чтение префиксов через snapshot; запись без дублирования rows.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:01 +07:00
DenozordecandCursor 8a9d60a5a7 feat(db): add prefix snapshot tables (expand)
Таблицы prefix_snapshot и prefix_snapshot_row; колонка prefix_snapshot_id.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:01 +07:00
DenozordecandCursor b963311b43 perf(db): keyset pagination for revision prefixes
Keyset по id/ord вместо OFFSET; anti-join для RevisionDiff.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:56:46 +07:00
DenozordecandCursor 50bdb8232b refactor(db): split revision preview from meta_json
Preview BIRD-фрагменты в config_revision_preview; meta_json только счётчик префиксов.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:55:52 +07:00
DenozordecandCursor ee8e24ffc6 perf(db): add revision and module query indexes
Индексы для diff/list префиксов и сортировки модулей; covering index для pagination.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:55:10 +07:00
DenozordecandCursor 44b94caacf docs(db): add postgres schema diagnostics runbook
Runbook для оценки размеров таблиц, seq scan и EXPLAIN перед/после миграций схемы.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:55:02 +07:00
122 changed files with 7857 additions and 485 deletions
+14
View File
@@ -0,0 +1,14 @@
{
"permissions": {
"allow": [
"mcp__codegraph__codegraph_explore",
"mcp__codegraph__codegraph_search",
"mcp__codegraph__codegraph_node",
"mcp__codegraph__codegraph_callers",
"mcp__codegraph__codegraph_callees",
"mcp__codegraph__codegraph_impact",
"mcp__codegraph__codegraph_files",
"mcp__codegraph__codegraph_status"
]
}
}
+16
View File
@@ -0,0 +1,16 @@
# CodeGraph data files
# These are local to each machine and should not be committed
# Database
*.db
*.db-wal
*.db-shm
# Cache
cache/
# Logs
*.log
# Hook markers
.dirty
+6
View File
@@ -0,0 +1,6 @@
{
"pid": 44608,
"version": "0.9.9",
"socketPath": "\\\\.\\pipe\\codegraph-97b92efdcc5351da",
"startedAt": 1781240018712
}
+14
View File
@@ -0,0 +1,14 @@
{
"mcpServers": {
"codegraph": {
"type": "stdio",
"command": "codegraph",
"args": [
"serve",
"--mcp",
"--path",
"C:\\Users\\shats\\Dev\\EvoBGP"
]
}
}
}
@@ -0,0 +1,408 @@
---
name: Технический аудит EvoBGP
overview: "Полный технический аудит EvoBGP для production-сценария (10+ клиентов, нестабильная сеть). Архитектура — hybrid control plane; сильные стороны: stale fallback, CDN/RIPEstat resilience, Ed25519 bundles. Критичные риски: in-process jobs, DoH без retry, misconfiguration demo-seed, отсутствие HA API."
todos:
- id: quick-ops-checklist
content: Применить production-checklist (SEED_DEMO=0, BUNDLE_SEED_HEX, DB/JOB/CONCURRENCY tuning, TLS)
status: pending
- id: fix-doh-retry
content: Добавить DoWithRetry для DoH в internal/pipeline/refresh.go
status: pending
- id: fix-job-meta-race
content: Исправить чтение j.Meta в worker.go через Snapshot() или locked accessor
status: pending
- id: cdn-preview-resilience
content: Перевести CDN preview на upstreamHTTPDo в routes_crud.go
status: pending
- id: partial-cdn-failure
content: "Partial CDN source failure: skip/degrade вместо fail всего модуля"
status: pending
- id: graceful-shutdown
content: Cancel/drain jobs при SIGTERM в cmd/evobgp-api и evobgp-all
status: pending
- id: ha-job-queue
content: "Roadmap: распределённая очередь jobs (PG claim или NATS) для HA API"
status: pending
isProject: false
---
# Технический аудит EvoBGP
## Executive summary
EvoBGP — **hybrid control plane**: один процесс [`evobgp-all`](cmd/evobgp-all/main.go) (monolith) или **reference Compose** с разделёнными воркерами ([`docs/architecture.md`](docs/architecture.md)). Data plane (BIRD + agent) отделён от control plane (API + PostgreSQL + jobs).
**Сильные стороны для нестабильной сети:**
- Stale snapshot fallback по умолчанию (`EVOBGP_STALE_ON_UPSTREAM_ERROR=1`) — [`internal/pipeline/collect_stale.go`](internal/pipeline/collect_stale.go)
- CDN/RIPEstat: retry (3×) + per-host circuit breaker — [`internal/httpclient/httpclient.go`](internal/httpclient/httpclient.go), [`circuit.go`](internal/httpclient/circuit.go)
- ETag conditional GET, ASN TTL-кэш, parallel collect с cap
- Подписанные бандлы Ed25519, verify перед apply
**Главные риски для 10+ клиентов:**
1. `jobs.Registry`**in-memory, только в процессе API** (ARCH-04)
2. DoH — **без retry/breaker** (критично при блокировках провайдеров)
3. Один failed CDN source **без stale cache валит весь модуль**
4. Production misconfiguration: `Bearer dev`, HTTP API, ephemeral bundle key
5. Data race на `Job.Meta` и alias pointers в `store.Memory`
---
## 1. Архитектура
### Стиль
```mermaid
flowchart TB
subgraph hybrid [Hybrid deployment]
All[evobgp_all monolith]
Split[evobgp_api + workers]
end
subgraph cp [Control plane]
API[HTTP API]
Jobs[jobs.Registry in-process]
PG[(PostgreSQL)]
end
subgraph dp [Data plane per speaker]
Agent[evobgp_agent]
BIRD[BIRD2]
NodeCLI[evobgp_node]
end
All --> API
Split --> API
API --> Jobs
API --> PG
NodeCLI --> API
Agent --> API
Agent --> BIRD
```
| Профиль | Стиль | Когда |
|---------|-------|-------|
| `microvps` / `evobgp-all` | Monolith | 1 VPS, shared Registry |
| reference Compose | Microservices-lite | API + scheduler/ingest/render/deploy |
| Remote speakers | Edge agents | Panel→Node dispatch |
### Узкие места (bottlenecks)
| # | Bottleneck | Где | Impact |
|---|------------|-----|--------|
| B1 | **In-process job queue** | [`internal/jobs/job.go:175-177`](internal/jobs/job.go) | HA API невозможен без потери/дублирования jobs; scheduler без `EVOBGP_CONTROL_PLANE_URL` создаёт **отдельный Registry** — [`cmd/evobgp-scheduler/main.go:58-60`](cmd/evobgp-scheduler/main.go) |
| B2 | **Module refresh = sync upstream fan-out** | [`internal/pipeline/collect_parallel.go`](internal/pipeline/collect_parallel.go) | До `EVOBGP_COLLECT_CONCURRENCY` (8 default, max 32) параллельных HTTP; worst case ~45s × retries на источник |
| B3 | **Default job concurrency = 8** | [`internal/jobs/job.go:264-268`](internal/jobs/job.go) | При burst refresh 10+ tenants — очередь растёт, goroutine блокируются на sem |
| B4 | **PostgreSQL pool default** | [`internal/db/open.go:28-38`](internal/db/open.go) | pgx default ~4 conns; при `JOB_MAX=16` + HTTP — contention без `EVOBGP_DB_MAX_CONNS=25` |
| B5 | **Live endpoints fan-out** | [`internal/httpapi/peers_live.go`](internal/httpapi/peers_live.go) | N goroutines × N speakers, 12s timeout каждый |
| B6 | **Broker — заглушка** | [`internal/broker`](internal/broker) | NATS URL логируется, очередь не распределена |
### Масштабируемость
- **Вертикальная:** хорошо до ~10–20 tenants при `evobgp-all` + tuning ([`docs/production-checklist.md`](docs/production-checklist.md))
- **Горизонтальная API:** **не поддерживается** — два `evobgp-api` = два независимых Registry; `job_audit` в PG — audit only, не очередь исполнения
- **Workers (ingest/render/deploy):** координируются через **общую БД**, не через jobs — OK для prefetch/drift
### Отказоустойчивость
| Сценарий | Поведение | Оценка |
|----------|-----------|--------|
| CDN/RIPEstat недоступен | Stale snapshot + circuit breaker | **Хорошо** (если был prior snapshot) |
| DoH недоступен | Fail модуля или stale domain snapshot | **Средне** (нет HTTP retry) |
| API restart mid-job | Job теряется из Registry; audit может быть inconsistent | **Плохо** |
| PG недоступен | API `/v1/ready` → 503 | **OK** |
| Agent unreachable | Deploy job succeed, drift в `evobgp-deploy` | **Частичный fail** (by design) |
**Рекомендация:** для 10+ клиентов — **`evobgp-all` на каждом CP** или один CP + tuning; HA API требует **распределённой очереди** (NATS/Redis + worker pool) — задокументировано как future work.
---
## 2. Анализ кода
### Антипаттерны
| ID | Проблема | Файл | Критичность |
|----|----------|------|-------------|
| A1 | **Concurrent map read/write** — worker читает `j.Meta` без lock, handler пишет через `mergeMeta`/`Snapshot` | [`worker.go:108,263,379`](internal/jobs/worker.go), [`job.go:103-111`](internal/jobs/job.go) | **high** |
| A2 | **Escape internal pointers** из Memory store | [`store/memory.go:416-475`](internal/store/memory.go) | **high** (tests/dev); **low** (prod PG) |
| A3 | **Fire-and-forget goroutine** на каждый auth | [`auth.go:79-81`](internal/httpapi/auth.go) | **medium** |
| A4 | **Silent error swallow** в prefetch | [`internal/ingest/run.go`](internal/ingest/run.go), `prefetch.go` | **medium** |
| A5 | **Bypass resilience layer** — CDN preview прямой `Do` | [`routes_crud.go:267`](internal/httpapi/routes_crud.go) | **medium** |
| A6 | **`EVOBGP_DEV_INSECURE` — dead code** | compose + [`server.go`](internal/httpapi/server.go) | **low** (misleading ops) |
| A7 | **Unused Registry** в ingest/render/deploy binaries | [`cmd/evobgp-ingest/main.go`](cmd/evobgp-ingest/main.go) | **low** (resource waste) |
### Maintainability
**Плюсы:** чёткое разделение слоёв (ARCH-01..10), `store.Backend`, OpenAPI как контракт, engineering rules, table-driven tests в birdfmt/pipeline.
**Минусы:**
- Дублирование retry-логики (httpclient vs nodedispatch inline loop)
- Env-tuning разбросан (`EVOBGP_*` в 15+ местах без central config struct для pipeline)
- `Job` comment «персистенция в БД пока не подключена» устарел — hooks есть в [`bootstrap.go:67-92`](internal/httpapi/bootstrap.go)
### Потенциальные баги и race conditions
1. **`j.Meta` data race** — `-race` на `TestParallelModuleRefresh_*` + concurrent `GET /v1/jobs/{id}` polling
2. **Memory store alias**`deploy.Run` читает `LastAppliedRevisionID` пока worker пишет
3. **peerLiveCache** возвращает slice без копии — [`peers_live.go:82-84`](internal/httpapi/peers_live.go)
4. **TOCTOU idempotency** — terminal job удаляется из `byIdempo`, повторный POST создаст новый job (by design, но клиент должен знать)
### Error handling
**Хорошо:**
- Префиксы ошибок (`httpclient:`, `birdfmt:`)
- HTTP 5xx через `writeProblem`, без raw `err.Error()` (ERR-01)
- `context.Context` в pipeline workers
**Пробелы:**
- `runRollback` без `workContext` — не отменяется — [`worker.go:500+`](internal/jobs/worker.go)
- Prefetch/ingest: ошибки не логируются
- `mergeBirdPostApplyMeta``context.Background()` 8s, игнорирует job cancel
---
## 3. Производительность
### Блокирующие операции
| Участок | Блокировка | Риск |
|---------|------------|------|
| `POST .../cdn-sources/preview` | Sync CDN fetch до 45s в HTTP handler | UI timeout, worker starvation |
| `GET /v1/peers/live` | N × agent HTTP, wg.Wait | Slow при многих speakers |
| Module refresh job | Sequential: ingest → render revision → optional deploy | Long job chain |
| `bird -p` / `birdc configure` | Subprocess в deploy | Disk I/O на ноде |
### Неэффективные алгоритмы / лишние запросы
- **Tenant refresh:** `aggregateTenantPrefixRowsAll` — parallel по модулям, но каждый модуль может refetch все CDN/ASN/DoH — [`aggregate.go:28+`](internal/pipeline/aggregate.go). Snapshot skip есть через `module_hash` — проверять hit rate в meta.
- **ASN resolve:** `PolitePause()` 150ms между AS — [`asnresolve/ripestat.go`](internal/asnresolve/ripestat.go) — при 50 AS = +7.5s minimum.
- **GetModulePrefixSnapshot** вызывается многократно в одном refresh (cdn_snapshot, collect_parallel) — potential duplicate DB reads.
- **Auth TouchAPIKeyLastUsed:** UPDATE на каждый request (async) — load на PG при high RPS.
### Кэширование
| Кэш | TTL | Gap |
|-----|-----|-----|
| ASN prefix cache | 1800s (`EVOBGP_ASN_CACHE_TTL_SEC`) | OK |
| CDN ETag in DB | Until 304/change | OK |
| Module prefix snapshot | Content-hash based skip | OK |
| peerLiveCache | In-memory, per-process | Не shared между API replicas; нет defensive copy |
| Circuit breaker state | Per-process | Не shared |
### Конкретные улучшения
```go
// 1. CDN preview — использовать upstreamHTTPDo вместо прямого Do
resp, err := pipeline.UpstreamHTTPDo(r.Context(), s.cdnHTTP, req) // extract upstreamHTTPDo
// 2. Job.Meta — читать под lock или через Snapshot()
st := j.Snapshot()
mid, _ := st["meta"].(map[string]any)["module_id"].(string)
// 3. Memory store — возвращать копии (как Postgres)
modCopy := *mod
return &modCopy, nil
```
---
## 4. Сетевое взаимодействие (критично)
### Текущее состояние
```mermaid
flowchart LR
subgraph resilient [Resilient path]
CDN[CDN fetch]
RIPE[RIPEstat]
CDN --> Breaker[Circuit breaker]
RIPE --> Breaker
Breaker --> Retry[DoWithRetry 3x linear 2s]
end
subgraph fragile [Fragile path]
DoH[DoH resolve]
Preview[CDN preview API]
AgentHealth[Agent health/bird]
DoH --> SingleDo[Single hc.Do]
Preview --> SingleDo
AgentHealth --> SingleDo
end
subgraph fallback [App-level fallback]
Stale[Stale snapshot]
SysDNS[System DNS]
DoH --> SysDNS
CDN --> Stale
RIPE --> Stale
end
```
| Upstream | Timeout | Retry | Breaker | Stale fallback |
|----------|---------|-------|---------|----------------|
| CDN ingest | 45s | 3× linear | per-host | yes |
| RIPEstat | 45s | 3× | per-host | yes + cache |
| DoH | 10s/profile | **no** | **no** | domain snapshot |
| CDN preview | 45s | **no** | **no** | N/A |
| Scheduler→API | 45s | 3× | no | N/A |
| Node dispatch | 30s | inline 3× | no | N/A |
### Пробелы для блокировок провайдеров
1. **DoH без retry** — transient timeout = fail; failover между profiles есть, но каждый profile — single shot
2. **429/408 не ретраятся** — только `>= 500`
3. **Нет jitter** — thundering herd при mass tenant refresh
4. **DNS rebinding TOCTOU** — SSRF check до fetch, HTTP dial без pinned IP — [`cdn_url.go:75-115`](internal/pipeline/cdn_url.go)
5. **Circuit breaker без half-open** — после 30s cooldown сразу full traffic — [`circuit.go:29-33`](internal/httpclient/circuit.go)
6. **Breaker per-process** — ingest container ≠ API container
### Рекомендации для нестабильной сети
| # | Изменение | Effort | Effect |
|---|-----------|--------|--------|
| N1 | DoH через `DoWithRetry` + optional breaker | Low | **High** для DOMAINS modules |
| N2 | Retry 429/503 с `Retry-After` + exponential backoff + jitter | Medium | **High** при rate limits |
| N3 | **Partial CDN failure** — continue с stale per-source, не fail whole module | Medium | **High** |
| N4 | Multiple DoH profiles + `failover` policy (already exists) — **документировать ops playbook** | Low | **High** (config, not code) |
| N5 | Pinned dialer / custom `Transport.DialContext` после SSRF resolve | Medium | **Medium** (SSRF hardening) |
| N6 | Proxy support (`HTTP_PROXY` / `EVOBGP_HTTP_PROXY`) для CDN/DoH | Medium | **High** в censored networks |
| N7 | Unify CDN preview на `upstreamHTTPDo` | Low | **Medium** |
---
## 5. Устойчивость и надёжность
### Graceful degradation
**Работает:**
- `EVOBGP_STALE_ON_UPSTREAM_ERROR=1` — ASN/CDN/domain stale — [`collect_stale.go`](internal/pipeline/collect_stale.go)
- CDN 304 без local cache → forced full GET — [`cdn_snapshot.go:141-159`](internal/pipeline/cdn_snapshot.go)
- DoH → system DNS fallback — [`doh_resolve.go:75-93`](internal/pipeline/doh_resolve.go)
- Deploy: job succeed even if agent wake fails (drift detection)
**Не работает / частично:**
- Один CDN source fail без cache → **весь module_refresh failed** — [`collect_parallel.go:221-223`](internal/pipeline/collect_parallel.go)
- Circuit open → immediate error, stale only if prior data exists
- API shutdown: HTTP drain 15s, **jobs не cancel/drain** — [`cmd/evobgp-api/main.go:67-72`](cmd/evobgp-api/main.go)
### Сценарии отказов
| Событие | Что произойдёт |
|---------|----------------|
| **Потеря CP↔PG** | Ready=false; running jobs fail; no new jobs persist audit reliably |
| **Потеря CP↔CDN** | Stale prefixes если были; иначе job fail; breaker opens 30s |
| **Потеря CP↔agent** | Deploy meta `dispatch_failed`; BIRD на старой ревизии; drift logs |
| **RIPEstat rate limit** | 429 → no retry → stale or fail |
| **Рост нагрузки** | Job queue; goroutine pile-up; PG pool exhaustion; `/metrics` shows queue depth |
| **API restart** | In-flight jobs lost; clients poll 404 or stale terminal state |
---
## 6. Безопасность
| ID | Finding | Severity | Fix |
|----|---------|----------|-----|
| S1 | `Bearer dev` → operator при demo-seed | **high** (misconfig) | `EVOBGP_SEED_DEMO=0` — [`auth.go:66-92`](internal/httpapi/auth.go) |
| S2 | API plain HTTP | **high** (ops) | TLS на edge (Traefik/nginx) |
| S3 | Ephemeral bundle key без `EVOBGP_BUNDLE_SEED_HEX` | **high** (ops) | Stable seed + pubkey на нодах |
| S4 | Compose defaults: weak PG password, `sslmode=disable` | **high** (ops) | Secrets manager, `sslmode=require` |
| S5 | `/metrics` без auth | **medium** | Network policy / mTLS |
| S6 | No rate limiting on auth | **medium** | Middleware limiter (e.g. per-IP) |
| S7 | CDN SSRF DNS rebinding | **medium** | Pinned dialer after resolve |
| S8 | `EVOBGP_CDN_ALLOW_PRIVATE=1` | **medium** | Never in prod |
| S9 | `EVOBGP_NODE_DISPATCH_INSECURE_TLS=1` | **medium** | Valid TLS to agent |
| S10 | Plaintext `EVOBGP_API_KEYS` in env | **medium** | DB keys via API |
| S11 | `editor` can cancel jobs | **low** | Restrict to operator |
| S12 | agent_secret `==` compare | **low** | `subtle.ConstantTimeCompare` |
**SQL injection:** не обнаружено — параметризованные запросы в [`repository/`](internal/repository/).
**Bundle crypto:** Ed25519 корректно; path traversal blocked в tar extract.
---
## 7. Конкретные рекомендации (prioritized backlog)
### High
| # | Описание | Как исправить |
|---|----------|---------------|
| H1 | DoH без retry | Обернуть `hc.Do` в `DoWithRetry(ctx, hc, req, 3)` в [`refresh.go:288,360`](internal/pipeline/refresh.go) |
| H2 | Data race `Job.Meta` | Читать через `Snapshot()` или добавить `MetaLocked()` accessor |
| H3 | CDN source partial failure | В `collectCDNPrefixRows`: при err без stale — log warning + skip source вместо `return nil, r.err` (config flag `EVOBGP_CDN_PARTIAL_OK=1`) |
| H4 | Production checklist enforcement | CI/deploy validation: reject `SEED_DEMO=1`, require `BUNDLE_SEED_HEX` |
| H5 | Job queue HA roadmap | Persist queued jobs in PG + worker claim (`SELECT FOR UPDATE SKIP LOCKED`) или NATS — ARCH-04 |
### Medium
| # | Описание | Как исправить |
|---|----------|---------------|
| M1 | CDN preview bypass | [`routes_crud.go:267`](internal/httpapi/routes_crud.go) → `upstreamHTTPDo` |
| M2 | Retry 429/503 | Extend `DoWithRetry` status check + parse `Retry-After` |
| M3 | Graceful shutdown | On SIGTERM: `Registry.RequestCancelAll()` + wait workers with timeout |
| M4 | Auth goroutine storm | Worker pool или sync touch with debounce |
| M5 | HTTP proxy support | Custom Transport reading `EVOBGP_HTTP_PROXY` |
| M6 | Memory store copies | Defensive copy in Get/List (dev/test safety) |
| M7 | Rate limiting | `golang.org/x/time/rate` on auth middleware |
### Low
| # | Описание | Как исправить |
|---|----------|---------------|
| L1 | Jitter in backoff | `wait + rand.Intn(wait/2)` in DoWithRetry |
| L2 | Half-open breaker | Single probe request after cooldown |
| L3 | Remove dead `EVOBGP_DEV_INSECURE` from compose | Docs + compose cleanup |
| L4 | Prefetch error logging | `log.Printf` or structured log in prefetch |
| L5 | peerLiveCache defensive copy | `append([]T(nil), views...)` on store |
---
## 8. Quick wins (максимальный эффект / минимум усилий)
1. **Ops (0 code):** [`docs/production-checklist.md`](docs/production-checklist.md) — `SEED_DEMO=0`, `BUNDLE_SEED_HEX`, `DB_MAX_CONNS=25`, `JOB_MAX=16`, `COLLECT_CONCURRENCY=16`, TLS edge, restrict metrics
2. **DoH retry** — 510 строк в `refresh.go`, reuse existing `DoWithRetry`
3. **CDN preview → upstreamHTTPDo** — 1 line change in handler
4. **Job.Meta read fix** — replace 4 reads in `worker.go` with `Snapshot()` parsing
5. **Log prefetch failures** — visibility без изменения behavior
6. **Document DoH failover playbook** — multiple profiles (Cloudflare, Google, Quad9) + `failover` policy for censored regions
7. **Run `go test -race ./internal/jobs/...`** in CI — catch Meta race
8. **Prefer `evobgp-all`** over split reference for &lt;20 tenants — eliminates Registry split bug
---
## Диаграмма: refresh под сетевым stress
```mermaid
sequenceDiagram
participant Op as Operator
participant API as evobgp_api
participant Job as module_refresh
participant CDN as CDN_upstream
participant PG as PostgreSQL
Op->>API: POST /modules/id/refresh
API->>Job: Enqueue
Job->>CDN: GET with ETag
alt CDN timeout or 5xx
CDN-->>Job: error after 3 retries
Job->>PG: load prior snapshot
alt stale exists
Job->>PG: CreateRenderRevision stale
Job-->>API: succeeded degraded
else no stale
Job-->>API: failed
end
else CDN 200
CDN-->>Job: new prefixes
Job->>PG: CreateRenderRevision
end
```
---
## Итоговая оценка зрелости
| Область | Оценка | Комментарий |
|---------|--------|-------------|
| Архитектура | 7/10 | Чистые слои; HA/API scaling — слабое место |
| Сеть/resilience | 6/10 | CDN/ASN хорошо; DoH/preview — пробелы |
| Concurrency | 6/10 | Registry продуман; Meta race, shutdown |
| Performance | 7/10 | Parallel collect, caching; tuning needed at scale |
| Security | 6/10 | Crypto OK; ops/config risks dominate |
| Maintainability | 8/10 | Docs, rules, OpenAPI, tests |
**Вердикт:** проект **готов для 10+ клиентов в single-CP deployment** (`evobgp-all` + PostgreSQL + production checklist) при условии ops discipline. Для **multi-CP HA** и **агрессивных сетевых блокировок** — приоритет: DoH retry, partial CDN failure, distributed job queue, HTTP proxy.
+93
View File
@@ -0,0 +1,93 @@
---
description: Context7 — закреплённые ID библиотек и документации стека EvoBGP
alwaysApply: true
---
# Context7 — стек EvoBGP
При вопросах об API, синтаксисе, конфигурации и миграциях библиотек **сначала** `query-docs` с ID из таблицы ниже. Шаг `resolve-library-id` **пропускать**, если библиотека уже перечислена (кроме неоднозначного случая).
Локальные версии: `go.mod`, `web/package.json`. При расхождении с ID — предпочитать версию из репозитория.
---
## Backend (Go)
| Библиотека | Context7 ID | Версия в проекте | Когда |
|------------|-------------|------------------|-------|
| Go stdlib | `/golang/go/go1_24_6` | Go 1.24 | `net/http`, `context`, тесты, concurrency |
| pgx | `/websites/pkg_go_dev_github_com_jackc_pgx_v5` | v5.7.2 | PostgreSQL, pool, транзакции, типы |
| Prometheus Go client | `/prometheus/client_golang` | v1.20.5 | метрики, `/metrics`, middleware |
| modernc SQLite | `/websites/pkg_go_dev_modernc_org_sqlite` | v1.34.5 | SQLite-бэкенд, миграции sqlite |
| miekg/dns | `/miekg/dns` | v1.1.72 | DNS-запросы, DoH, pipeline |
---
## HTTP-контракт и спецификации
| Библиотека | Context7 ID | Версия в проекте | Когда |
|------------|-------------|------------------|-------|
| OpenAPI | `/oai/openapi-specification` | 3.x в `docs/openapi.yaml` | схемы, operationId, problem+json |
| Redocly CLI | `/redocly/redocly-cli` | CI `@redocly/cli` | lint OpenAPI, `npx @redocly/cli lint` |
---
## Web UI (`web/`)
| Библиотека | Context7 ID | Версия в проекте | Когда |
|------------|-------------|------------------|-------|
| Svelte | `/websites/svelte_dev` | ^5.54 | runes, компоненты, реактивность |
| SvelteKit | `/sveltejs/kit` | ^2.50 | routing, `load`, adapters, SSR |
| Vite | `/vitejs/vite/v7.3.1` | ^7.3.1 | dev server, build, plugins |
| TypeScript | `/microsoft/typescript/v5.9.3` | ^5.9.3 | типы, strict, tsconfig |
| Tailwind CSS | `/tailwindlabs/tailwindcss.com` | ^4.1 | v4, `@tailwindcss/vite`, утилиты |
| shadcn-svelte | `/websites/shadcn-svelte` | CLI | примитивы `ui/core`, theming |
| Bits UI | `/llmstxt/bits-ui_llms_txt` | ^2.17 | headless-примитивы под shadcn |
| sveltekit-superforms | `/ciscoheat/sveltekit-superforms` | ^2.30 | формы, server actions |
| Formsnap | `/svecosystem/formsnap` | ^2.0 | доступные поля форм |
| Zod | `/websites/zod_dev_v4` | ^4.4 | схемы валидации |
| TanStack Table | `/websites/tanstack_table` | table-core ^8.21 | `AppDataTable`, колонки, сортировка |
UI-правила репозитория: `.cursor/rules/web-shadcn.mdc` (shadcn-svelte docs — первичный источник для компонентов).
---
## Data plane / BGP
| Библиотека | Context7 ID | Версия в проекте | Когда |
|------------|-------------|------------------|-------|
| BIRD 2 | `/llmstxt/bird_xmsl_dev_llms_txt` | BIRD2 в compose | `birdfmt`, фильтры, протоколы |
| BIRD (исходники) | `/cz-nic/bird` | — | низкоуровневый синтаксис daemon |
Сетевые правила: `.cursor/rules/networking-bird.mdc`.
---
## DevOps
| Библиотека | Context7 ID | Версия в проекте | Когда |
|------------|-------------|------------------|-------|
| Docker Compose | `/docker/compose` | `deploy/compose/` | сервисы, profiles, volumes |
| Docker | `/docker/docs` | — | образы, bake, networking |
---
## Приоритет источников
1. **Контракт HTTP** — `docs/openapi.yaml` (не Context7).
2. **Context7** — синтаксис и API библиотек из таблицы.
3. **Локальные docs** — `docs/`, `web/README.md`, `AGENTS.md`.
4. **Официальный сайт** — BIRD: https://bird.network.cz/?get_doc (если Context7 не покрыл кейс).
## Примеры запросов
```
/docs /websites/svelte_dev runes $state $derived
/docs /golang/go/go1_24_6 net/http ServeMux pattern matching
/docs /websites/pkg_go_dev_github_com_jackc_pgx_v5 pool acquire rows
/docs /llmstxt/bird_xmsl_dev_llms_txt filter bgp import
```
## Не через Context7
Рефакторинг `internal/*`, бизнес-логика EvoBGP, code review — код репозитория и `codegraph`. Context7 — только внешние библиотеки и инструменты.
+3
View File
@@ -2,6 +2,9 @@
"plugins": {
"svelte": {
"enabled": true
},
"claude-plugins-official/gopls-lsp": {
"enabled": true
}
}
}
+31
View File
@@ -0,0 +1,31 @@
---
name: context7-evobgp
description: Context7 lookup для стека EvoBGP — использовать закреплённые library ID из .cursor/rules/context7-stack.mdc вместо resolve-library-id.
---
# Context7 — EvoBGP stack
Перед `query-docs` открой `.cursor/rules/context7-stack.mdc` и выбери ID из таблицы по области задачи.
## Workflow
1. Определи область: `internal/` (Go), `web/` (Svelte), `docs/openapi.yaml`, `birdfmt`/`pipeline` (BIRD), `deploy/compose` (Docker).
2. Найди строку в таблице `context7-stack.mdc`.
3. Вызови `query-docs` с `libraryId` из таблицы и полным вопросом пользователя.
4. `resolve-library-id` — только если библиотеки нет в таблице или нужна другая major-версия.
## Быстрые ID (частые)
| Задача | libraryId |
|--------|-----------|
| Svelte 5 runes | `/websites/svelte_dev` |
| SvelteKit load/forms | `/sveltejs/kit` |
| shadcn-svelte компонент | `/websites/shadcn-svelte` |
| pgx pool/query | `/websites/pkg_go_dev_github_com_jackc_pgx_v5` |
| Go net/http | `/golang/go/go1_24_6` |
| OpenAPI lint | `/redocly/redocly-cli` |
| BIRD config | `/llmstxt/bird_xmsl_dev_llms_txt` |
| Tailwind v4 | `/tailwindlabs/tailwindcss.com` |
| Zod 4 schema | `/websites/zod_dev_v4` |
Полный список и версии — в `context7-stack.mdc`.
+12
View File
@@ -0,0 +1,12 @@
{
"mcpServers": {
"codegraph": {
"type": "stdio",
"command": "codegraph",
"args": [
"serve",
"--mcp"
]
}
}
}
+12
View File
@@ -0,0 +1,12 @@
{
"mcpServers": {
"codegraph": {
"type": "stdio",
"command": "codegraph",
"args": [
"serve",
"--mcp"
]
}
}
}
+1 -1
View File
@@ -4,7 +4,7 @@
## С чего начать (минимум чтения)
0. **Инженерные правила** — при изменении кода следовать [.cursor/rules/engineering.mdc](.cursor/rules/engineering.mdc); для `web/` — [.cursor/rules/web-shadcn.mdc](.cursor/rules/web-shadcn.mdc); для `birdfmt` / `pipeline` / BIRD — [.cursor/rules/networking-bird.mdc](.cursor/rules/networking-bird.mdc).
0. **Инженерные правила** — при изменении кода следовать [.cursor/rules/engineering.mdc](.cursor/rules/engineering.mdc); для `web/` — [.cursor/rules/web-shadcn.mdc](.cursor/rules/web-shadcn.mdc); для `birdfmt` / `pipeline` / BIRD — [.cursor/rules/networking-bird.mdc](.cursor/rules/networking-bird.mdc). **Context7 (документация библиотек)** — закреплённые ID стека: [.cursor/rules/context7-stack.mdc](.cursor/rules/context7-stack.mdc); скилл [.cursor/skills/context7-evobgp/SKILL.md](.cursor/skills/context7-evobgp/SKILL.md).
1. **[docs/README.md](docs/README.md)** — оглавление и роли читателя.
2. **[docs/architecture.md](docs/architecture.md)** — компоненты `cmd/`, карта `internal/`, потоки данных (одного этого файла обычно достаточно для ориентации).
3. Задача-специфично: [docs/api.md](docs/api.md), [docs/access.md](docs/access.md), [web/README.md](web/README.md) — только если меняете API, доступ или фронт.
+5
View File
@@ -12,6 +12,7 @@ import (
"evobgp/internal/birdfmt"
"evobgp/internal/config"
"evobgp/internal/dbcli"
"evobgp/internal/deploy"
"evobgp/internal/httpapi"
"evobgp/internal/ingest"
@@ -24,6 +25,9 @@ import (
// microVPS entrypoint: один процесс — HTTP API и фоновые воркеры scheduler, ingest, render, deploy (общий store и jobs.Registry).
func main() {
if len(os.Args) > 1 && os.Args[1] == "db" {
os.Exit(dbcli.Run(os.Args[2:]))
}
cfg := config.Load()
opts := httpapi.Options{
APIKeys: os.Getenv("EVOBGP_API_KEYS"),
@@ -52,6 +56,7 @@ func main() {
go render.Run(ctx, renderDeps)
go deploy.Run(ctx, deployDeps)
srv.StartBackground(ctx)
startBirdMetricsPoller(ctx)
httpSrv := &http.Server{
+5
View File
@@ -12,6 +12,7 @@ import (
"evobgp/internal/birdfmt"
"evobgp/internal/config"
"evobgp/internal/dbcli"
"evobgp/internal/httpapi"
"evobgp/internal/observability"
"evobgp/internal/platform"
@@ -19,6 +20,9 @@ import (
)
func main() {
if len(os.Args) > 1 && os.Args[1] == "db" {
os.Exit(dbcli.Run(os.Args[2:]))
}
cfg := config.Load()
seedDemo := os.Getenv("EVOBGP_SEED_DEMO") != "0"
opts := httpapi.Options{
@@ -39,6 +43,7 @@ func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
srv.StartBackground(ctx)
startBirdMetricsPoller(ctx)
httpSrv := &http.Server{
+12
View File
@@ -52,6 +52,18 @@ opkey|01ARZ3NDEKTSV4RRFFQ69G5FAV|operator,nodekey|01ARZ3NDEKTSV4RRFFQ69G5FAV|nod
**Запрещено** в продакшене: не оставляйте demo-seed с известным токеном `dev` на боевых данных. Переменная `EVOBGP_DEV_INSECURE` в текущей версии **не влияет** на аутентификацию (оставлена в compose для совместимости; не включайте в production — см. SEC-02 в инженерных правилах).
### PostgreSQL monitoring и maintenance (control plane)
При `EVOBGP_DATABASE_URL` (не memory backend):
| Операция | Минимальная роль |
|----------|------------------|
| `GET /v1/monitoring/postgres/*`, `GET /v1/monitoring/correlation` | viewer |
| `POST /v1/postgres/vacuum`, `vacuum-analyze`, `analyze`, `reindex`, `cleanup` | **operator** (async job, rate limit 60s на kind) |
| `GET /v1/postgres/maintenance/logs` | viewer |
Метрики **instance-level** (не per-tenant). CLI: `evobgp-api db …` / `evobgp-all db …`.
### Синхронные «тяжёлые» GET (control plane)
- `POST /v1/modules/{module_id}/cdn-sources/preview` — загрузка CDN в том же HTTP-запросе (лимит тела ~8 MiB, см. OpenAPI).
+112
View File
@@ -0,0 +1,112 @@
# Диагностика схемы PostgreSQL (EvoBGP)
Runbook для оценки объёма БД и узких мест **перед** и **после** миграций оптимизации схемы. Выполнять на staging или production read-only сессией.
## Подключение
```bash
psql "$EVOBGP_DATABASE_URL"
```
## HTTP API (панель / мониторинг)
При подключённом PostgreSQL control plane отдаёт instance-level метрики (роль **viewer+**):
- `GET /v1/monitoring/postgres/overview` — подключения, TPS, cache hit, размер БД
- `GET /v1/monitoring/postgres/queries` — top queries (`pg_stat_statements`, если extension включён)
- `GET /v1/monitoring/postgres/locks`, `/tables`, `/recommendations`
- `GET /v1/monitoring/correlation?window=60` — корреляция refresh jobs и cache hit
Обслуживание (**operator**, async `202` + `job_id`): `POST /v1/postgres/vacuum`, `vacuum-analyze`, `analyze`, `reindex`, `cleanup`; журнал `GET /v1/postgres/maintenance/logs`.
CLI на CP: `evobgp-api db report|vacuum|analyze|cleanup` (см. `internal/dbcli`).
Миграция `000023` создаёт `pg_stat_statements`; для сбора статистики **обязательно** preload и перезапуск Postgres:
```text
# postgresql.conf или command в compose
shared_preload_libraries = 'pg_stat_statements'
```
После изменения — restart контейнера/сервиса Postgres. Без этого API `/v1/monitoring/postgres/queries` вернёт пустой список (`statements_available: false`), без 5xx.
## 1. Размеры таблиц и индексов
```sql
SELECT relname,
pg_size_pretty(pg_total_relation_size(relid)) AS total,
pg_size_pretty(pg_relation_size(relid)) AS heap,
pg_size_pretty(pg_indexes_size(relid)) AS indexes
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC;
```
**Ожидание:** лидеры — `revision_materialized_prefix`, `config_revision` (TOAST от preview), JSONB-кэши.
## 2. Seq scan (горячие таблицы)
```sql
SELECT schemaname, relname, seq_scan, seq_tup_read, idx_scan
FROM pg_stat_user_tables
WHERE schemaname = 'public'
ORDER BY seq_tup_read DESC;
```
Сброс статистики после деплоя: `SELECT pg_stat_reset();` (только осознанно, теряется baseline).
## 3. Неиспользуемые индексы
```sql
SELECT indexrelname, idx_scan, pg_size_pretty(pg_relation_size(indexrelid)) AS size
FROM pg_stat_user_indexes
WHERE schemaname = 'public' AND idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC;
```
## 4. Дубликаты в materialized prefixes
Перед UNIQUE `(revision_id, prefix, community_id, source)`:
```sql
SELECT revision_id, prefix, community_id, source, COUNT(*) AS n
FROM revision_materialized_prefix
GROUP BY 1, 2, 3, 4
HAVING COUNT(*) > 1
LIMIT 20;
```
## 5. Шаблон отчёта staging
| Метрика | До | После | Дата |
|---------|-----|-------|------|
| `revision_materialized_prefix` total | | | |
| `config_revision` total | | | |
| `module_prefix_snapshot` total | | | |
| `asn_prefix_cache` total | | | |
| Top seq_scan table | | | |
| Unused indexes (count) | | | |
## 6. EXPLAIN для типовых запросов
```sql
-- Список префиксов ревизии (keyset)
EXPLAIN (ANALYZE, BUFFERS)
SELECT prefix::text, community_id::text, source
FROM revision_materialized_prefix
WHERE revision_id = '<revision-uuid>'::uuid
ORDER BY id
LIMIT 51;
-- Diff added (anti-join)
EXPLAIN (ANALYZE, BUFFERS)
SELECT b.prefix::text
FROM revision_materialized_prefix b
LEFT JOIN revision_materialized_prefix a
ON a.revision_id = '<rev-a>'::uuid AND a.prefix = b.prefix
WHERE b.revision_id = '<rev-b>'::uuid
AND a.prefix IS NULL
ORDER BY b.prefix
LIMIT 5001;
```
Цель: Index Scan / Bitmap Index Scan по `(revision_id, …)`, без Seq Scan на больших таблицах.
+626
View File
@@ -49,6 +49,10 @@ tags:
description: Управление API-ключами tenant (operator). Секрет возвращается только при создании и ротации.
- name: Auth
description: Сессия текущего API-ключа (tenant и роль).
- name: Monitoring
description: Наблюдаемость PostgreSQL и корреляция (instance-level, viewer+). Maintenance — operator.
- name: Maintenance
description: Политики обслуживания PostgreSQL (instance-scoped). CRUD и запуск — operator.
security:
- bearerAuth: []
@@ -910,6 +914,152 @@ components:
type: string
additionalProperties: true
PostgresOverview:
type: object
description: Instance-level PostgreSQL snapshot (GET /v1/monitoring/postgres/overview).
additionalProperties: true
PostgresQueriesResponse:
type: object
properties:
collected_at:
type: string
format: date-time
source:
type: string
enum: [live, snapshot]
items:
type: array
items:
type: object
additionalProperties: true
PostgresRecommendations:
type: object
properties:
collected_at:
type: string
format: date-time
items:
type: array
items:
type: object
properties:
severity:
type: string
code:
type: string
title:
type: string
detail:
type: string
refs:
type: array
items:
type: string
PostgresMaintenanceBody:
type: object
properties:
table:
type: string
dry_run:
type: boolean
default: false
policy:
type: string
description: Deprecated; use maintenance policies API.
limit:
type: integer
MaintenancePolicy:
type: object
required: [name, table_name, schedule, vacuum_strategy]
properties:
id:
$ref: "#/components/schemas/ResourceId"
name:
type: string
table_name:
type: string
condition:
type: string
default: "true"
retention_period_sec:
type: integer
minimum: 1
max_rows:
type: integer
minimum: 1
maximum: 100000
vacuum_strategy:
type: string
enum: [none, vacuum, analyze, vacuum_analyze, reindex]
schedule:
type: string
description: Cron expression (5-field, UTC).
enabled:
type: boolean
default: true
dry_run_enabled:
type: boolean
default: false
last_run_at:
type: string
format: date-time
last_status:
type: string
last_error:
type: string
created_at:
type: string
format: date-time
updated_at:
type: string
format: date-time
MaintenancePolicyPatch:
type: object
properties:
name:
type: string
table_name:
type: string
condition:
type: string
retention_period_sec:
type: integer
max_rows:
type: integer
vacuum_strategy:
type: string
enum: [none, vacuum, analyze, vacuum_analyze, reindex]
schedule:
type: string
enabled:
type: boolean
dry_run_enabled:
type: boolean
MaintenanceRunBody:
type: object
required: [policy_id]
properties:
policy_id:
$ref: "#/components/schemas/ResourceId"
MaintenancePolicyList:
type: object
properties:
items:
type: array
items:
$ref: "#/components/schemas/MaintenancePolicy"
next_cursor:
type: string
has_more:
type: boolean
BirdLocalStatus:
type: object
description: Статус локального BIRD на хосте API (GET /v1/bird/status).
@@ -3135,6 +3285,482 @@ paths:
default:
$ref: "#/components/responses/DefaultProblem"
/v1/monitoring/postgres/overview:
get:
tags: [Monitoring]
summary: PostgreSQL overview (instance-level)
operationId: getPostgresOverview
parameters:
- $ref: "#/components/parameters/TenantId"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
$ref: "#/components/schemas/PostgresOverview"
"503":
description: PostgreSQL backend не подключён.
default:
$ref: "#/components/responses/DefaultProblem"
/v1/monitoring/postgres/queries:
get:
tags: [Monitoring]
summary: Top queries (pg_stat_statements or snapshot)
operationId: getPostgresQueries
parameters:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/Limit"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
$ref: "#/components/schemas/PostgresQueriesResponse"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/monitoring/postgres/locks:
get:
tags: [Monitoring]
summary: Active locks
operationId: getPostgresLocks
parameters:
- $ref: "#/components/parameters/TenantId"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
type: object
properties:
items:
type: array
items:
type: object
additionalProperties: true
default:
$ref: "#/components/responses/DefaultProblem"
/v1/monitoring/postgres/tables:
get:
tags: [Monitoring]
summary: Table sizes and scan stats
operationId: getPostgresTables
parameters:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/Limit"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
type: object
properties:
items:
type: array
items:
type: object
additionalProperties: true
default:
$ref: "#/components/responses/DefaultProblem"
/v1/monitoring/postgres/recommendations:
get:
tags: [Monitoring]
summary: Heuristic optimization recommendations
operationId: getPostgresRecommendations
parameters:
- $ref: "#/components/parameters/TenantId"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
$ref: "#/components/schemas/PostgresRecommendations"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/monitoring/correlation:
get:
tags: [Monitoring]
summary: Timeline correlation (jobs vs cache hit)
operationId: getMonitoringCorrelation
parameters:
- $ref: "#/components/parameters/TenantId"
- name: window
in: query
schema:
type: integer
default: 60
description: Window in minutes (max 1440).
responses:
"200":
description: Успешно.
content:
application/json:
schema:
type: object
additionalProperties: true
default:
$ref: "#/components/responses/DefaultProblem"
/v1/postgres/vacuum:
post:
tags: [Monitoring]
summary: VACUUM (async job, operator)
operationId: postPostgresVacuum
parameters:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/IdempotencyKey"
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/PostgresMaintenanceBody"
responses:
"202":
description: Задача поставлена.
content:
application/json:
schema:
$ref: "#/components/schemas/AsyncJobAccepted"
"403":
$ref: "#/components/responses/Forbidden"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/postgres/vacuum-analyze:
post:
tags: [Monitoring]
summary: VACUUM ANALYZE (async job, operator)
operationId: postPostgresVacuumAnalyze
parameters:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/IdempotencyKey"
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/PostgresMaintenanceBody"
responses:
"202":
description: Задача поставлена.
content:
application/json:
schema:
$ref: "#/components/schemas/AsyncJobAccepted"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/postgres/analyze:
post:
tags: [Monitoring]
summary: ANALYZE (async job, operator)
operationId: postPostgresAnalyze
parameters:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/IdempotencyKey"
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/PostgresMaintenanceBody"
responses:
"202":
description: Задача поставлена.
content:
application/json:
schema:
$ref: "#/components/schemas/AsyncJobAccepted"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/postgres/reindex:
post:
tags: [Monitoring]
summary: REINDEX TABLE (async job, operator)
operationId: postPostgresReindex
parameters:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/IdempotencyKey"
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/PostgresMaintenanceBody"
responses:
"202":
description: Задача поставлена.
content:
application/json:
schema:
$ref: "#/components/schemas/AsyncJobAccepted"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/postgres/cleanup:
post:
tags: [Monitoring]
summary: Retention cleanup (async job, operator)
operationId: postPostgresCleanup
parameters:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/IdempotencyKey"
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/PostgresMaintenanceBody"
responses:
"202":
description: Задача поставлена.
content:
application/json:
schema:
$ref: "#/components/schemas/AsyncJobAccepted"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/postgres/maintenance/logs:
get:
tags: [Monitoring]
summary: Maintenance audit log
operationId: listPostgresMaintenanceLogs
parameters:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/Cursor"
- $ref: "#/components/parameters/Limit"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
type: object
properties:
items:
type: array
items:
type: object
additionalProperties: true
next_cursor:
type: string
has_more:
type: boolean
default:
$ref: "#/components/responses/DefaultProblem"
/v1/maintenance/policies:
get:
tags: [Maintenance]
summary: List maintenance policies
operationId: listMaintenancePolicies
parameters:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/Cursor"
- $ref: "#/components/parameters/Limit"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
$ref: "#/components/schemas/MaintenancePolicyList"
default:
$ref: "#/components/responses/DefaultProblem"
post:
tags: [Maintenance]
summary: Create maintenance policy
operationId: createMaintenancePolicy
parameters:
- $ref: "#/components/parameters/TenantId"
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/MaintenancePolicy"
responses:
"201":
description: Создано.
content:
application/json:
schema:
$ref: "#/components/schemas/MaintenancePolicy"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/maintenance/policies/{id}:
get:
tags: [Maintenance]
summary: Get maintenance policy
operationId: getMaintenancePolicy
parameters:
- $ref: "#/components/parameters/TenantId"
- name: id
in: path
required: true
schema:
$ref: "#/components/schemas/ResourceId"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
$ref: "#/components/schemas/MaintenancePolicy"
default:
$ref: "#/components/responses/DefaultProblem"
patch:
tags: [Maintenance]
summary: Update maintenance policy
operationId: patchMaintenancePolicy
parameters:
- $ref: "#/components/parameters/TenantId"
- name: id
in: path
required: true
schema:
$ref: "#/components/schemas/ResourceId"
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/MaintenancePolicyPatch"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
$ref: "#/components/schemas/MaintenancePolicy"
default:
$ref: "#/components/responses/DefaultProblem"
delete:
tags: [Maintenance]
summary: Delete maintenance policy
operationId: deleteMaintenancePolicy
parameters:
- $ref: "#/components/parameters/TenantId"
- name: id
in: path
required: true
schema:
$ref: "#/components/schemas/ResourceId"
responses:
"204":
description: Удалено.
default:
$ref: "#/components/responses/DefaultProblem"
/v1/maintenance/policies/{id}/hints:
get:
tags: [Maintenance]
summary: PostgreSQL hints for policy table
operationId: getMaintenancePolicyHints
parameters:
- $ref: "#/components/parameters/TenantId"
- name: id
in: path
required: true
schema:
$ref: "#/components/schemas/ResourceId"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
type: object
additionalProperties: true
default:
$ref: "#/components/responses/DefaultProblem"
/v1/maintenance/config-audit:
get:
tags: [Maintenance]
summary: Maintenance policy configuration audit log
operationId: listMaintenanceConfigAudit
parameters:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/Cursor"
- $ref: "#/components/parameters/Limit"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
type: object
properties:
items:
type: array
items:
type: object
additionalProperties: true
next_cursor:
type: string
has_more:
type: boolean
default:
$ref: "#/components/responses/DefaultProblem"
/v1/maintenance/run:
post:
tags: [Maintenance]
summary: Run maintenance policy (async job)
operationId: postMaintenanceRun
parameters:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/IdempotencyKey"
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/MaintenanceRunBody"
responses:
"202":
description: Задача поставлена.
content:
application/json:
schema:
$ref: "#/components/schemas/AsyncJobAccepted"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/maintenance/dry-run:
post:
tags: [Maintenance]
summary: Dry-run maintenance policy (async job)
operationId: postMaintenanceDryRun
parameters:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/IdempotencyKey"
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/MaintenanceRunBody"
responses:
"202":
description: Задача поставлена.
content:
application/json:
schema:
$ref: "#/components/schemas/AsyncJobAccepted"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/settings:
get:
tags: [Settings]
+1
View File
@@ -25,6 +25,7 @@ require (
github.com/prometheus/common v0.55.0 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/robfig/cron/v3 v3.0.1 // indirect
golang.org/x/crypto v0.46.0 // indirect
golang.org/x/mod v0.31.0 // indirect
golang.org/x/net v0.48.0 // indirect
+2
View File
@@ -46,6 +46,8 @@ github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0leargg
github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+212
View File
@@ -0,0 +1,212 @@
// Package dbcli implements control-plane PostgreSQL maintenance CLI (HTTP or local DSN).
package dbcli
import (
"bytes"
"context"
"encoding/json"
"flag"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
"evobgp/internal/db"
"evobgp/internal/httpclient"
"evobgp/internal/pgmonitor"
)
// Run executes db subcommands; args exclude program name and "db".
func Run(args []string) int {
if len(args) == 0 {
printUsage()
return 2
}
switch args[0] {
case "report":
return cmdReport(args[1:])
case "vacuum":
return cmdMaint(args[1:], "vacuum", "/v1/postgres/vacuum")
case "analyze":
return cmdMaint(args[1:], "analyze", "/v1/postgres/analyze")
case "cleanup":
return cmdCleanup(args[1:])
default:
fmt.Fprintf(os.Stderr, "dbcli: unknown command %q\n", args[0])
printUsage()
return 2
}
}
func printUsage() {
fmt.Fprintln(os.Stderr, `usage:
evobgp-api db report [--api-url URL] [--token TOKEN] [--format json]
evobgp-api db vacuum [--table NAME] [--dry-run] [--api-url URL] [--token TOKEN]
evobgp-api db analyze [--table NAME] [--dry-run] [--api-url URL] [--token TOKEN]
evobgp-api db cleanup --policy NAME [--dry-run] [--limit N] [--api-url URL] [--token TOKEN]
Local break-glass: set EVOBGP_DATABASE_URL (report only uses direct SQL).`)
}
func cmdReport(args []string) int {
fs := flag.NewFlagSet("report", flag.ExitOnError)
apiURL := fs.String("api-url", "", "control plane base URL")
token := fs.String("token", "", "Bearer token (operator)")
format := fs.String("format", "json", "output format (json)")
_ = fs.Parse(args)
if dsn := strings.TrimSpace(os.Getenv("EVOBGP_DATABASE_URL")); dsn != "" && *apiURL == "" {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
pool, err := db.OpenPostgresPool(ctx, dsn)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
defer pool.Close()
svc := pgmonitor.NewService(pool)
ov, err := svc.Overview(ctx)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
return writeJSONStdout(ov, *format)
}
if *apiURL == "" || *token == "" {
fmt.Fprintln(os.Stderr, "report: --api-url and --token required without EVOBGP_DATABASE_URL")
return 2
}
body, err := apiGET(*apiURL, *token, "/v1/monitoring/postgres/overview")
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
var pretty any
if err := json.Unmarshal(body, &pretty); err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
return writeJSONStdout(pretty, *format)
}
func cmdMaint(args []string, _ string, path string) int {
fs := flag.NewFlagSet("maint", flag.ExitOnError)
table := fs.String("table", "", "table name")
dryRun := fs.Bool("dry-run", false, "dry run only")
apiURL := fs.String("api-url", "", "control plane base URL")
token := fs.String("token", "", "Bearer token (operator)")
_ = fs.Parse(args)
if *apiURL == "" || *token == "" {
fmt.Fprintln(os.Stderr, "maintenance: --api-url and --token are required")
return 2
}
payload := map[string]any{"dry_run": *dryRun}
if *table != "" {
payload["table"] = *table
}
body, err := apiPOST(*apiURL, *token, path, payload)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
return writeRawJSON(body)
}
func cmdCleanup(args []string) int {
fs := flag.NewFlagSet("cleanup", flag.ExitOnError)
policyID := fs.String("policy-id", "", "maintenance policy UUID")
dryRun := fs.Bool("dry-run", true, "dry run")
apiURL := fs.String("api-url", "", "control plane base URL")
token := fs.String("token", "", "Bearer token (operator)")
_ = fs.Parse(args)
if *policyID == "" {
fmt.Fprintln(os.Stderr, "cleanup: --policy-id is required")
return 2
}
if *apiURL == "" || *token == "" {
fmt.Fprintln(os.Stderr, "cleanup: --api-url and --token are required")
return 2
}
path := "/v1/maintenance/run"
if *dryRun {
path = "/v1/maintenance/dry-run"
}
payload := map[string]any{"policy_id": *policyID}
body, err := apiPOST(*apiURL, *token, path, payload)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
return writeRawJSON(body)
}
func apiGET(base, token, path string) ([]byte, error) {
u := strings.TrimRight(base, "/") + path
req, err := http.NewRequest(http.MethodGet, u, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+token)
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
resp, err := httpclient.DoWithRetry(ctx, httpclient.New(60*time.Second), req, 3)
if err != nil {
return nil, err
}
defer func() { _ = resp.Body.Close() }()
b, _ := io.ReadAll(resp.Body)
if resp.StatusCode >= 300 {
return nil, fmt.Errorf("dbcli: GET %s: %s: %s", path, resp.Status, strings.TrimSpace(string(b)))
}
return b, nil
}
func apiPOST(base, token, path string, payload map[string]any) ([]byte, error) {
b, err := json.Marshal(payload)
if err != nil {
return nil, err
}
u := strings.TrimRight(base, "/") + path
req, err := http.NewRequest(http.MethodPost, u, bytes.NewReader(b))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+token)
req.Header.Set("Content-Type", "application/json")
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
resp, err := httpclient.DoWithRetry(ctx, httpclient.New(60*time.Second), req, 3)
if err != nil {
return nil, err
}
defer func() { _ = resp.Body.Close() }()
out, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusAccepted && resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("dbcli: POST %s: %s: %s", path, resp.Status, strings.TrimSpace(string(out)))
}
return out, nil
}
func writeJSONStdout(v any, format string) int {
if format != "json" {
fmt.Fprintln(os.Stderr, "only json format supported")
return 2
}
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
if err := enc.Encode(v); err != nil {
fmt.Fprintln(os.Stderr, err)
return 1
}
return 0
}
func writeRawJSON(b []byte) int {
var v any
if err := json.Unmarshal(b, &v); err != nil {
_, _ = os.Stdout.Write(b)
return 0
}
return writeJSONStdout(v, "json")
}
+1 -1
View File
@@ -51,7 +51,7 @@ func BootstrapWorkers(ctx context.Context, opts Options) (store.Backend, *jobs.R
}
cdnHTTP := NewCDNHTTPClient()
wk := &jobs.Worker{Store: backend, HTTPClient: cdnHTTP}
wk := &jobs.Worker{Store: backend, PgPool: pool, HTTPClient: cdnHTTP}
reg := jobs.NewRegistry(wk.Process)
wk.Registry = reg
if pool != nil {
+3
View File
@@ -76,6 +76,9 @@ func (s *Server) registerV1(m *http.ServeMux) {
m.HandleFunc("GET /speakers/{speaker_id}/bundle/{revision_id}", s.handleNodeBundle)
m.HandleFunc("POST /nodes/enroll", s.handleNodeEnroll)
s.registerCRUDRoutes(m)
s.registerPostgresMonitoringRoutes(m)
s.registerPostgresMaintenanceRoutes(m)
s.registerMaintenanceRoutes(m)
}
func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
+274
View File
@@ -0,0 +1,274 @@
package httpapi
import (
"encoding/json"
"net/http"
"strings"
"evobgp/internal/jobs"
"evobgp/internal/observability"
"evobgp/internal/store"
)
func (s *Server) registerMaintenanceRoutes(m *http.ServeMux) {
m.HandleFunc("GET /maintenance/policies", s.handleListMaintenancePolicies)
m.HandleFunc("POST /maintenance/policies", s.handleCreateMaintenancePolicy)
m.HandleFunc("GET /maintenance/policies/{id}", s.handleGetMaintenancePolicy)
m.HandleFunc("PATCH /maintenance/policies/{id}", s.handlePatchMaintenancePolicy)
m.HandleFunc("DELETE /maintenance/policies/{id}", s.handleDeleteMaintenancePolicy)
m.HandleFunc("GET /maintenance/policies/{id}/hints", s.handleMaintenancePolicyHints)
m.HandleFunc("GET /maintenance/config-audit", s.handleListMaintenanceConfigAudit)
m.HandleFunc("POST /maintenance/run", s.handleMaintenanceRun)
m.HandleFunc("POST /maintenance/dry-run", s.handleMaintenanceDryRun)
}
func maintenancePolicyJSON(p *store.MaintenancePolicy) map[string]any {
if p == nil {
return map[string]any{}
}
out := map[string]any{
"id": p.ID,
"name": p.Name,
"table_name": p.TableName,
"condition": p.Condition,
"vacuum_strategy": p.VacuumStrategy,
"schedule": p.Schedule,
"enabled": p.Enabled,
"dry_run_enabled": p.DryRunEnabled,
}
if p.RetentionPeriodSec != nil {
out["retention_period_sec"] = *p.RetentionPeriodSec
}
if p.MaxRows != nil {
out["max_rows"] = *p.MaxRows
}
if p.LastRunAt != nil {
out["last_run_at"] = p.LastRunAt.UTC().Format("2006-01-02T15:04:05Z")
}
if p.LastStatus != "" {
out["last_status"] = p.LastStatus
}
if p.LastError != "" {
out["last_error"] = p.LastError
}
if !p.CreatedAt.IsZero() {
out["created_at"] = p.CreatedAt.UTC().Format("2006-01-02T15:04:05Z")
}
if !p.UpdatedAt.IsZero() {
out["updated_at"] = p.UpdatedAt.UTC().Format("2006-01-02T15:04:05Z")
}
return out
}
func (s *Server) handleListMaintenancePolicies(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
return
}
cursor := r.URL.Query().Get("cursor")
limit := parseLimitQuery(r, 20, 100)
items, next, hasMore, err := s.store.ListMaintenancePolicies(cursor, limit)
if err != nil {
writeInternalError(w, "maintenance_policies_list", err)
return
}
out := make([]map[string]any, 0, len(items))
for _, p := range items {
out = append(out, maintenancePolicyJSON(p))
}
writeJSON(w, http.StatusOK, map[string]any{"items": out, "next_cursor": next, "has_more": hasMore})
}
func (s *Server) handleGetMaintenancePolicy(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
return
}
p, err := s.store.GetMaintenancePolicy(r.PathValue("id"))
if err != nil {
writeStoreErr(w, err)
return
}
writeJSON(w, http.StatusOK, maintenancePolicyJSON(p))
}
func (s *Server) handleCreateMaintenancePolicy(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requirePostgres(w) || !s.requireOperatorStrict(w, a) {
return
}
var body store.MaintenancePolicy
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
return
}
p, err := s.store.CreateMaintenancePolicy(&body)
if err != nil {
writeStoreErr(w, err)
return
}
_ = s.store.AppendMaintenancePolicyConfigAudit(actorPrefix(a), p.ID, "create", nil, maintenancePolicyJSON(p))
observability.IncMaintenanceConfigChange("create")
s.reloadMaintenanceConfig(r)
writeJSON(w, http.StatusCreated, maintenancePolicyJSON(p))
}
func (s *Server) handlePatchMaintenancePolicy(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requirePostgres(w) || !s.requireOperatorStrict(w, a) {
return
}
id := r.PathValue("id")
before, err := s.store.GetMaintenancePolicy(id)
if err != nil {
writeStoreErr(w, err)
return
}
var patch store.MaintenancePolicyPatch
if err := json.NewDecoder(r.Body).Decode(&patch); err != nil {
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
return
}
updated, err := s.store.UpdateMaintenancePolicy(id, &patch)
if err != nil {
writeStoreErr(w, err)
return
}
_ = s.store.AppendMaintenancePolicyConfigAudit(actorPrefix(a), id, "update", maintenancePolicyJSON(before), maintenancePolicyJSON(updated))
observability.IncMaintenanceConfigChange("update")
s.reloadMaintenanceConfig(r)
writeJSON(w, http.StatusOK, maintenancePolicyJSON(updated))
}
func (s *Server) handleDeleteMaintenancePolicy(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requirePostgres(w) || !s.requireOperatorStrict(w, a) {
return
}
id := r.PathValue("id")
before, err := s.store.GetMaintenancePolicy(id)
if err != nil {
writeStoreErr(w, err)
return
}
if err := s.store.DeleteMaintenancePolicy(id); err != nil {
writeStoreErr(w, err)
return
}
_ = s.store.AppendMaintenancePolicyConfigAudit(actorPrefix(a), id, "delete", maintenancePolicyJSON(before), nil)
observability.IncMaintenanceConfigChange("delete")
s.reloadMaintenanceConfig(r)
w.WriteHeader(http.StatusNoContent)
}
func (s *Server) handleMaintenancePolicyHints(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
return
}
if s.maintStats == nil {
writeProblem(w, http.StatusServiceUnavailable, "Unavailable", "postgresql backend required")
return
}
p, err := s.store.GetMaintenancePolicy(r.PathValue("id"))
if err != nil {
writeStoreErr(w, err)
return
}
hints, err := s.maintStats.Hints(r.Context(), p.TableName)
if err != nil {
writeInternalError(w, "maintenance_policy_hints", err)
return
}
writeJSON(w, http.StatusOK, hints)
}
func (s *Server) handleListMaintenanceConfigAudit(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
return
}
cursor := r.URL.Query().Get("cursor")
limit := parseLimitQuery(r, 20, 100)
items, next, hasMore, err := s.store.ListMaintenancePolicyConfigAudit(cursor, limit)
if err != nil {
writeInternalError(w, "maintenance_config_audit", err)
return
}
out := make([]map[string]any, 0, len(items))
for _, row := range items {
out = append(out, map[string]any{
"id": row.ID,
"policy_id": row.PolicyID,
"actor_prefix": row.ActorPrefix,
"action": row.Action,
"before": row.Before,
"after": row.After,
"created_at": row.CreatedAt.UTC().Format("2006-01-02T15:04:05Z"),
})
}
writeJSON(w, http.StatusOK, map[string]any{"items": out, "next_cursor": next, "has_more": hasMore})
}
type maintenanceRunBody struct {
PolicyID string `json:"policy_id"`
}
func (s *Server) handleMaintenanceRun(w http.ResponseWriter, r *http.Request) {
s.enqueueMaintenancePolicy(w, r, false)
}
func (s *Server) handleMaintenanceDryRun(w http.ResponseWriter, r *http.Request) {
s.enqueueMaintenancePolicy(w, r, true)
}
func (s *Server) enqueueMaintenancePolicy(w http.ResponseWriter, r *http.Request, dryRun bool) {
a, ok := authFromContext(r.Context())
if !ok || !s.requirePostgres(w) || !s.requireOperatorStrict(w, a) {
return
}
var body maintenanceRunBody
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
return
}
policyID := strings.TrimSpace(body.PolicyID)
if policyID == "" {
writeProblem(w, http.StatusBadRequest, "Bad Request", "policy_id is required")
return
}
if _, err := s.store.GetMaintenancePolicy(policyID); err != nil {
writeStoreErr(w, err)
return
}
kind := "maintenance_policy_run"
if !s.checkPgMaintRateLimit(a.TenantID, kind+":"+policyID) {
writeProblem(w, http.StatusTooManyRequests, "Too Many Requests", "wait before repeating this maintenance operation")
return
}
idem := strings.TrimSpace(r.Header.Get("Idempotency-Key"))
var idemPtr *string
if idem != "" {
idemPtr = &idem
}
title := "Maintenance policy run"
if dryRun {
title = "Maintenance policy dry-run"
}
j, _, err := s.jobs.Enqueue(a.TenantID, jobs.KindMaintenancePolicyRun, idemPtr, nil, map[string]any{
"policy_id": policyID, "dry_run": dryRun, "actor_prefix": actorPrefix(a), "job_title": title,
})
if err != nil {
writeInternalError(w, "maintenance_policy_enqueue", err)
return
}
w.Header().Set("Location", "/v1/jobs/"+j.ID)
snap := j.Snapshot()
writeJSON(w, http.StatusAccepted, map[string]any{"job_id": snap["job_id"], "status": snap["status"]})
}
func (s *Server) reloadMaintenanceConfig(r *http.Request) {
if s.maintConfig != nil {
_ = s.maintConfig.Reload(r.Context())
}
}
@@ -0,0 +1,46 @@
package httpapi
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestMaintenancePoliciesMemoryBackend503(t *testing.T) {
srv, err := New(Options{SeedDemo: true, InsecureDev: true})
if err != nil {
t.Fatal(err)
}
defer srv.Close()
handler := srv.Handler()
tests := []struct {
method string
path string
body string
}{
{http.MethodGet, "/v1/maintenance/policies", ""},
{http.MethodPost, "/v1/maintenance/policies", `{"name":"x","table_name":"job_audit","schedule":"0 3 * * *"}`},
{http.MethodPost, "/v1/maintenance/run", `{"policy_id":"00000000-0000-0000-0000-000000000001"}`},
{http.MethodGet, "/v1/maintenance/config-audit", ""},
}
for _, tc := range tests {
t.Run(tc.method+" "+tc.path, func(t *testing.T) {
var req *http.Request
if tc.body != "" {
req = httptest.NewRequest(tc.method, tc.path, strings.NewReader(tc.body))
req.Header.Set("Content-Type", "application/json")
} else {
req = httptest.NewRequest(tc.method, tc.path, nil)
}
req.Header.Set("Authorization", "Bearer dev")
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, req)
if rec.Code != http.StatusServiceUnavailable {
t.Fatalf("status=%d body=%s", rec.Code, rec.Body.String())
}
})
}
}
@@ -0,0 +1,224 @@
package httpapi
import (
"context"
"encoding/json"
"io"
"net/http"
"strings"
"sync"
"time"
"evobgp/internal/jobs"
"evobgp/internal/pgmonitor"
)
var (
pgMaintRateMu sync.Mutex
pgMaintLastByTK = map[string]time.Time{}
)
func (s *Server) registerPostgresMaintenanceRoutes(m *http.ServeMux) {
m.HandleFunc("POST /postgres/vacuum", s.handlePostgresVacuum)
m.HandleFunc("POST /postgres/vacuum-analyze", s.handlePostgresVacuumAnalyze)
m.HandleFunc("POST /postgres/analyze", s.handlePostgresAnalyze)
m.HandleFunc("POST /postgres/reindex", s.handlePostgresReindex)
m.HandleFunc("POST /postgres/cleanup", s.handlePostgresCleanup)
m.HandleFunc("GET /postgres/maintenance/logs", s.handlePostgresMaintenanceLogs)
}
func (s *Server) requireOperatorStrict(w http.ResponseWriter, a Auth) bool {
if strings.ToLower(a.Role) != "operator" {
writeProblem(w, http.StatusForbidden, "Forbidden", "operator role required")
return false
}
return true
}
func (s *Server) checkPgMaintRateLimit(tenantID, kind string) bool {
key := tenantID + ":" + kind
pgMaintRateMu.Lock()
defer pgMaintRateMu.Unlock()
if t, ok := pgMaintLastByTK[key]; ok && time.Since(t) < 60*time.Second {
return false
}
pgMaintLastByTK[key] = time.Now().UTC()
return true
}
type pgMaintBody struct {
Table string `json:"table"`
DryRun bool `json:"dry_run"`
Index string `json:"index"`
Policy string `json:"policy"`
PolicyID string `json:"policy_id"`
Limit int `json:"limit"`
}
func (s *Server) decodePgMaintBody(r *http.Request) (pgMaintBody, bool) {
var body pgMaintBody
if r.Body == nil || r.ContentLength == 0 {
return body, true
}
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&body); err != nil && err != io.EOF {
return body, false
}
return body, true
}
func (s *Server) enqueuePostgresMaint(w http.ResponseWriter, r *http.Request, a Auth, kind string, meta map[string]any) {
if !s.requirePostgres(w) || !s.requireOperatorStrict(w, a) {
return
}
if !s.checkPgMaintRateLimit(a.TenantID, kind) {
writeProblem(w, http.StatusTooManyRequests, "Too Many Requests", "wait before repeating this maintenance operation")
return
}
idem := strings.TrimSpace(r.Header.Get("Idempotency-Key"))
var idemPtr *string
if idem != "" {
idemPtr = &idem
}
meta["actor_prefix"] = actorPrefix(a)
j, _, err := s.jobs.Enqueue(a.TenantID, kind, idemPtr, nil, meta)
if err != nil {
writeInternalError(w, "postgres_maint_enqueue", err)
return
}
w.Header().Set("Location", "/v1/jobs/"+j.ID)
snap := j.Snapshot()
writeJSON(w, http.StatusAccepted, map[string]any{"job_id": snap["job_id"], "status": snap["status"]})
}
func (s *Server) handlePostgresVacuum(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok {
writeProblem(w, http.StatusUnauthorized, "Unauthorized", "missing auth")
return
}
body, ok2 := s.decodePgMaintBody(r)
if !ok2 {
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
return
}
s.enqueuePostgresMaint(w, r, a, jobs.KindPostgresVacuum, map[string]any{
"table": body.Table, "dry_run": body.DryRun, "job_title": "PostgreSQL VACUUM",
})
}
func (s *Server) handlePostgresVacuumAnalyze(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok {
writeProblem(w, http.StatusUnauthorized, "Unauthorized", "missing auth")
return
}
body, ok2 := s.decodePgMaintBody(r)
if !ok2 {
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
return
}
s.enqueuePostgresMaint(w, r, a, jobs.KindPostgresVacuumAnalyze, map[string]any{
"table": body.Table, "dry_run": body.DryRun, "job_title": "PostgreSQL VACUUM ANALYZE",
})
}
func (s *Server) handlePostgresAnalyze(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok {
writeProblem(w, http.StatusUnauthorized, "Unauthorized", "missing auth")
return
}
body, ok2 := s.decodePgMaintBody(r)
if !ok2 {
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
return
}
s.enqueuePostgresMaint(w, r, a, jobs.KindPostgresAnalyze, map[string]any{
"table": body.Table, "dry_run": body.DryRun, "job_title": "PostgreSQL ANALYZE",
})
}
func (s *Server) handlePostgresReindex(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok {
writeProblem(w, http.StatusUnauthorized, "Unauthorized", "missing auth")
return
}
body, ok2 := s.decodePgMaintBody(r)
if !ok2 {
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
return
}
table := body.Table
if table == "" {
table = body.Index
}
s.enqueuePostgresMaint(w, r, a, jobs.KindPostgresReindex, map[string]any{
"table": table, "dry_run": body.DryRun, "job_title": "PostgreSQL REINDEX",
})
}
func (s *Server) handlePostgresCleanup(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok {
writeProblem(w, http.StatusUnauthorized, "Unauthorized", "missing auth")
return
}
body, ok2 := s.decodePgMaintBody(r)
if !ok2 {
writeProblem(w, http.StatusBadRequest, "Bad Request", invalidInputDetail)
return
}
policyID := strings.TrimSpace(body.PolicyID)
if policyID == "" {
policyID = strings.TrimSpace(body.Policy)
}
if policyID == "" {
writeProblem(w, http.StatusBadRequest, "Bad Request", "policy_id is required")
return
}
if _, err := s.store.GetMaintenancePolicy(policyID); err != nil {
writeStoreErr(w, err)
return
}
idem := strings.TrimSpace(r.Header.Get("Idempotency-Key"))
var idemPtr *string
if idem != "" {
idemPtr = &idem
}
j, _, err := s.jobs.Enqueue(a.TenantID, jobs.KindMaintenancePolicyRun, idemPtr, nil, map[string]any{
"policy_id": policyID, "dry_run": body.DryRun, "actor_prefix": actorPrefix(a),
"job_title": "PostgreSQL cleanup (deprecated path)",
})
if err != nil {
writeInternalError(w, "postgres_maint_enqueue", err)
return
}
w.Header().Set("Location", "/v1/jobs/"+j.ID)
snap := j.Snapshot()
writeJSON(w, http.StatusAccepted, map[string]any{"job_id": snap["job_id"], "status": snap["status"]})
}
func (s *Server) handlePostgresMaintenanceLogs(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
return
}
cursor := r.URL.Query().Get("cursor")
limit := parseLimitQuery(r, 20, 100)
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
defer cancel()
items, next, hasMore, err := pgmonitor.ListMaintenanceLogs(ctx, s.pgMonitor.Pool(), cursor, limit)
if err != nil {
writeInternalError(w, "postgres_maint_logs", err)
return
}
writeJSON(w, http.StatusOK, map[string]any{"items": items, "next_cursor": next, "has_more": hasMore})
}
func actorPrefix(a Auth) string {
if len(a.Token) >= 8 {
return a.Token[:8]
}
return a.Role
}
@@ -0,0 +1,130 @@
package httpapi
import (
"context"
"net/http"
"strconv"
"time"
)
func (s *Server) registerPostgresMonitoringRoutes(m *http.ServeMux) {
m.HandleFunc("GET /monitoring/postgres/overview", s.handlePostgresOverview)
m.HandleFunc("GET /monitoring/postgres/queries", s.handlePostgresQueries)
m.HandleFunc("GET /monitoring/postgres/locks", s.handlePostgresLocks)
m.HandleFunc("GET /monitoring/postgres/tables", s.handlePostgresTables)
m.HandleFunc("GET /monitoring/postgres/recommendations", s.handlePostgresRecommendations)
m.HandleFunc("GET /monitoring/correlation", s.handleMonitoringCorrelation)
}
func (s *Server) requirePostgres(w http.ResponseWriter) bool {
if s.pgMonitor == nil {
writeProblem(w, http.StatusServiceUnavailable, "Unavailable", "postgresql backend required")
return false
}
return true
}
func parseLimitQuery(r *http.Request, def, max int) int {
if v := r.URL.Query().Get("limit"); v != "" {
if n, err := strconv.Atoi(v); err == nil {
return n
}
}
return def
}
func (s *Server) handlePostgresOverview(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
return
}
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
defer cancel()
out, err := s.pgMonitor.Overview(ctx)
if err != nil {
writeInternalError(w, "postgres_overview", err)
return
}
writeJSON(w, http.StatusOK, out)
}
func (s *Server) handlePostgresQueries(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
return
}
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
defer cancel()
out, err := s.pgMonitor.TopQueries(ctx, parseLimitQuery(r, 20, 100))
if err != nil {
writeInternalError(w, "postgres_queries", err)
return
}
writeJSON(w, http.StatusOK, out)
}
func (s *Server) handlePostgresLocks(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
return
}
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
defer cancel()
out, err := s.pgMonitor.Locks(ctx)
if err != nil {
writeInternalError(w, "postgres_locks", err)
return
}
writeJSON(w, http.StatusOK, map[string]any{"items": out})
}
func (s *Server) handlePostgresTables(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
return
}
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
defer cancel()
out, err := s.pgMonitor.Tables(ctx, parseLimitQuery(r, 20, 100))
if err != nil {
writeInternalError(w, "postgres_tables", err)
return
}
writeJSON(w, http.StatusOK, map[string]any{"items": out})
}
func (s *Server) handlePostgresRecommendations(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
return
}
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
defer cancel()
out, err := s.pgMonitor.Recommendations(ctx)
if err != nil {
writeInternalError(w, "postgres_recommendations", err)
return
}
writeJSON(w, http.StatusOK, out)
}
func (s *Server) handleMonitoringCorrelation(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") || !s.requirePostgres(w) {
return
}
window := 60
if v := r.URL.Query().Get("window"); v != "" {
if n, err := strconv.Atoi(v); err == nil {
window = n
}
}
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
defer cancel()
out, err := s.pgMonitor.Correlation(ctx, window)
if err != nil {
writeInternalError(w, "monitoring_correlation", err)
return
}
writeJSON(w, http.StatusOK, out)
}
+21
View File
@@ -0,0 +1,21 @@
package httpapi
import (
"net/http"
"net/http/httptest"
"testing"
)
func TestPostgresOverviewMemoryBackend503(t *testing.T) {
srv, err := New(Options{SeedDemo: true})
if err != nil {
t.Fatal(err)
}
req := httptest.NewRequest(http.MethodGet, "/v1/monitoring/postgres/overview", nil)
req.Header.Set("Authorization", "Bearer dev")
rec := httptest.NewRecorder()
srv.Handler().ServeHTTP(rec, req)
if rec.Code != http.StatusServiceUnavailable {
t.Fatalf("status=%d body=%s", rec.Code, rec.Body.String())
}
}
+35
View File
@@ -8,8 +8,11 @@ import (
"errors"
"net/http"
"strings"
"time"
"evobgp/internal/jobs"
"evobgp/internal/maintenance"
"evobgp/internal/pgmonitor"
"evobgp/internal/store"
"github.com/jackc/pgx/v5/pgxpool"
@@ -19,6 +22,9 @@ import (
type Server struct {
store store.Backend
pgPool *pgxpool.Pool
pgMonitor *pgmonitor.Service
maintConfig *maintenance.ConfigProvider
maintStats *maintenance.DBStatsProvider
jobs *jobs.Registry
bundlePriv ed25519.PrivateKey
keyResolver *apiKeyResolver
@@ -63,9 +69,21 @@ func New(opts Options) (*Server, error) {
if err != nil {
return nil, err
}
var pgMon *pgmonitor.Service
var maintCfg *maintenance.ConfigProvider
var maintStats *maintenance.DBStatsProvider
if pool != nil {
pgMon = pgmonitor.NewService(pool)
maintCfg = maintenance.NewConfigProvider(backend)
_ = maintCfg.Reload(context.Background())
maintStats = maintenance.NewDBStatsProvider(pgMon)
}
s := &Server{
store: backend,
pgPool: pool,
pgMonitor: pgMon,
maintConfig: maintCfg,
maintStats: maintStats,
jobs: reg,
bundlePriv: priv,
keyResolver: resolver,
@@ -89,3 +107,20 @@ func (s *Server) Store() store.Backend { return s.store }
// Jobs exposes the in-process async job registry (for scheduler / evobgp-all).
func (s *Server) Jobs() *jobs.Registry { return s.jobs }
// StartBackground starts PostgreSQL monitoring and maintenance schedulers until ctx is cancelled.
func (s *Server) StartBackground(ctx context.Context) {
if s != nil && s.pgPool != nil {
pgmonitor.StartScheduler(ctx, s.pgPool)
}
if s != nil && s.maintConfig != nil && s.jobs != nil {
maintenance.StartScheduler(ctx, s.maintConfig, func(policyID string, dryRun bool, idem string) {
key := idem
_, _, _ = s.jobs.Enqueue("", jobs.KindMaintenancePolicyRun, &key, nil, map[string]any{
"policy_id": policyID,
"dry_run": dryRun,
"trigger": "scheduler",
})
}, 30*time.Second)
}
}
+63
View File
@@ -0,0 +1,63 @@
package jobs
import (
"errors"
"strings"
"evobgp/internal/maintenance"
"evobgp/internal/pgmonitor"
"evobgp/internal/store"
)
func (w *Worker) maintenanceExecutor() *maintenance.PolicyExecutor {
if w == nil {
return nil
}
return &maintenance.PolicyExecutor{Store: w.Store, Pool: w.PgPool}
}
func (w *Worker) runMaintenancePolicy(j *Job) {
if w == nil || w.PgPool == nil {
j.Fail("postgresql not configured")
return
}
policyID, _ := j.Meta["policy_id"].(string)
policyID = strings.TrimSpace(policyID)
if policyID == "" {
j.Fail("missing policy_id in job meta")
return
}
dryRun, _ := j.Meta["dry_run"].(bool)
actor, _ := j.Meta["actor_prefix"].(string)
ctx, cancel := j.workContext()
defer cancel()
pol, err := w.Store.GetMaintenancePolicy(policyID)
if err != nil {
if errors.Is(err, store.ErrNotFound) {
j.Fail("maintenance policy not found")
return
}
j.Fail(err.Error())
return
}
auditID, _ := pgmonitor.InsertMaintenanceAuditWithPolicy(ctx, w.PgPool, j.TenantID, actor, "maintenance_policy_run", pol.TableName, policyID, dryRun)
exec := w.maintenanceExecutor()
detail, err := exec.Execute(ctx, pol, dryRun)
var errMsg *string
status := StatusSucceeded
if err != nil {
s := err.Error()
errMsg = &s
status = StatusFailed
_ = w.Store.TouchMaintenancePolicyRun(policyID, status, s)
j.Fail(s)
} else {
j.mergeMeta(map[string]any{"maintenance": detail, "audit_id": auditID, "policy_id": policyID})
j.Succeed()
}
if auditID != "" {
_ = pgmonitor.FinishMaintenanceAudit(ctx, w.PgPool, auditID, status, detail, errMsg)
}
}
+140
View File
@@ -0,0 +1,140 @@
package jobs
import (
"fmt"
"evobgp/internal/pgmonitor"
)
func (w *Worker) pgService() *pgmonitor.Service {
if w == nil || w.PgPool == nil {
return nil
}
return pgmonitor.NewService(w.PgPool)
}
func (w *Worker) runPostgresMetricsRefresh(j *Job) {
s := w.pgService()
if s == nil {
j.Fail("postgresql not configured")
return
}
ctx, cancel := j.workContext()
defer cancel()
if err := s.RefreshMetricsSnapshot(ctx); err != nil {
j.Fail(err.Error())
return
}
j.Succeed()
}
func (w *Worker) runPostgresSlowQueryAgg(j *Job) {
s := w.pgService()
if s == nil {
j.Fail("postgresql not configured")
return
}
ctx, cancel := j.workContext()
defer cancel()
if err := s.AggregateSlowQueries(ctx, 30); err != nil {
j.Fail(err.Error())
return
}
j.Succeed()
}
func (w *Worker) runPostgresTableBloat(j *Job) {
s := w.pgService()
if s == nil {
j.Fail("postgresql not configured")
return
}
ctx, cancel := j.workContext()
defer cancel()
if err := s.EstimateTableBloat(ctx); err != nil {
j.Fail(err.Error())
return
}
j.Succeed()
}
func (w *Worker) runPostgresIndexUsage(j *Job) {
s := w.pgService()
if s == nil {
j.Fail("postgresql not configured")
return
}
ctx, cancel := j.workContext()
defer cancel()
if err := s.AnalyzeIndexUsage(ctx); err != nil {
j.Fail(err.Error())
return
}
j.Succeed()
}
func (w *Worker) runPostgresAutovacuumLag(j *Job) {
s := w.pgService()
if s == nil {
j.Fail("postgresql not configured")
return
}
ctx, cancel := j.workContext()
defer cancel()
if err := s.DetectAutovacuumLag(ctx); err != nil {
j.Fail(err.Error())
return
}
j.Succeed()
}
func (w *Worker) runPostgresMaint(j *Job, kind string) {
if w == nil || w.PgPool == nil {
j.Fail("postgresql not configured")
return
}
table, _ := j.Meta["table"].(string)
dryRun, _ := j.Meta["dry_run"].(bool)
actor, _ := j.Meta["actor_prefix"].(string)
ctx, cancel := j.workContext()
defer cancel()
auditID, _ := pgmonitor.InsertMaintenanceAudit(ctx, w.PgPool, j.TenantID, actor, kind, table, dryRun)
detail, err := pgmonitor.ExecMaintenance(ctx, w.PgPool, kind, table, dryRun)
var errMsg *string
status := StatusSucceeded
if err != nil {
s := err.Error()
errMsg = &s
status = StatusFailed
j.Fail(s)
} else {
j.mergeMeta(map[string]any{"maintenance": detail, "audit_id": auditID})
j.Succeed()
}
if auditID != "" {
_ = pgmonitor.FinishMaintenanceAudit(ctx, w.PgPool, auditID, status, detail, errMsg)
}
}
func (w *Worker) runPostgresCleanup(j *Job) {
j.Fail("postgres_cleanup deprecated: configure maintenance_policy in UI and use maintenance_policy_run")
}
// EnqueuePostgresAnalyzerJobs enqueues periodic analyzer jobs (global tenant id).
func EnqueuePostgresAnalyzerJobs(reg *Registry, tenantID string) {
if reg == nil || tenantID == "" {
return
}
kinds := []string{
KindPostgresMetricsRefresh,
KindPostgresSlowQueryAgg,
KindPostgresTableBloat,
KindPostgresIndexUsage,
KindPostgresAutovacuumLag,
}
for _, k := range kinds {
key := fmt.Sprintf("pgmon-%s-%s", k, tenantID)
idem := key
_, _, _ = reg.Enqueue(tenantID, k, &idem, nil, map[string]any{"trigger": "scheduler"})
}
}
+42 -6
View File
@@ -18,6 +18,8 @@ import (
"evobgp/internal/observability"
"evobgp/internal/pipeline"
"evobgp/internal/store"
"github.com/jackc/pgx/v5/pgxpool"
)
// mergeBirdPostApplyMeta attaches a birdc snapshot after deploy/reload (best-effort).
@@ -43,17 +45,29 @@ func mergeBirdPostApplyMeta(j *Job) {
}
const (
KindModuleRefresh = "module_refresh"
KindTenantRefresh = "tenant_refresh"
KindPeerReconcile = "peer_reconcile"
KindDeployApply = "deploy_apply"
KindRevisionRollback = "revision_rollback"
KindBirdReload = "bird_reload"
KindModuleRefresh = "module_refresh"
KindTenantRefresh = "tenant_refresh"
KindPeerReconcile = "peer_reconcile"
KindDeployApply = "deploy_apply"
KindRevisionRollback = "revision_rollback"
KindBirdReload = "bird_reload"
KindPostgresMetricsRefresh = "postgres_metrics_refresh"
KindPostgresSlowQueryAgg = "postgres_slow_query_aggregate"
KindPostgresTableBloat = "postgres_table_bloat_estimate"
KindPostgresIndexUsage = "postgres_index_usage_analyze"
KindPostgresAutovacuumLag = "postgres_autovacuum_lag_detect"
KindPostgresVacuum = "postgres_vacuum"
KindPostgresVacuumAnalyze = "postgres_vacuum_analyze"
KindPostgresAnalyze = "postgres_analyze"
KindPostgresReindex = "postgres_reindex"
KindPostgresCleanup = "postgres_cleanup"
KindMaintenancePolicyRun = "maintenance_policy_run"
)
// Worker executes queued jobs against store.Backend (memory or SQL).
type Worker struct {
Store store.Backend
PgPool *pgxpool.Pool
HTTPClient *http.Client // optional; CDN refresh uses this (default 45s timeout).
// Registry is set after BootstrapWorkers creates the job queue; used to chain deploy_apply after refresh/rollback.
Registry *Registry
@@ -155,6 +169,28 @@ func (w *Worker) Process(j *Job) {
}
mergeBirdPostApplyMeta(j)
j.Succeed()
case KindPostgresMetricsRefresh:
w.runPostgresMetricsRefresh(j)
case KindPostgresSlowQueryAgg:
w.runPostgresSlowQueryAgg(j)
case KindPostgresTableBloat:
w.runPostgresTableBloat(j)
case KindPostgresIndexUsage:
w.runPostgresIndexUsage(j)
case KindPostgresAutovacuumLag:
w.runPostgresAutovacuumLag(j)
case KindPostgresVacuum:
w.runPostgresMaint(j, "vacuum")
case KindPostgresVacuumAnalyze:
w.runPostgresMaint(j, "vacuum_analyze")
case KindPostgresAnalyze:
w.runPostgresMaint(j, "analyze")
case KindPostgresReindex:
w.runPostgresMaint(j, "reindex")
case KindPostgresCleanup:
w.runPostgresCleanup(j)
case KindMaintenancePolicyRun:
w.runMaintenancePolicy(j)
default:
j.Fail("unknown job kind")
}
+67
View File
@@ -0,0 +1,67 @@
package maintenance
import (
"context"
"sync"
"evobgp/internal/store"
)
// ConfigProvider caches maintenance policies from store.Backend with hot reload.
type ConfigProvider struct {
store store.Backend
mu sync.RWMutex
items []*store.MaintenancePolicy
}
// NewConfigProvider constructs a provider; call Reload before use.
func NewConfigProvider(st store.Backend) *ConfigProvider {
return &ConfigProvider{store: st}
}
// Reload loads all policies from the database into memory.
func (c *ConfigProvider) Reload(ctx context.Context) error {
if c == nil || c.store == nil {
return nil
}
_ = ctx
items, _, _, err := c.store.ListMaintenancePolicies("", 1000)
if err != nil {
return err
}
cp := make([]*store.MaintenancePolicy, len(items))
copy(cp, items)
c.mu.Lock()
c.items = cp
c.mu.Unlock()
return nil
}
// Snapshot returns a copy of cached policies.
func (c *ConfigProvider) Snapshot() []*store.MaintenancePolicy {
if c == nil {
return nil
}
c.mu.RLock()
defer c.mu.RUnlock()
out := make([]*store.MaintenancePolicy, len(c.items))
copy(out, c.items)
return out
}
// Get returns one policy by id from cache or store.
func (c *ConfigProvider) Get(ctx context.Context, id string) (*store.MaintenancePolicy, error) {
if c == nil || c.store == nil {
return nil, store.ErrNotFound
}
c.mu.RLock()
for _, p := range c.items {
if p.ID == id {
cp := *p
c.mu.RUnlock()
return &cp, nil
}
}
c.mu.RUnlock()
return c.store.GetMaintenancePolicy(id)
}
@@ -0,0 +1,43 @@
package maintenance
import (
"context"
"testing"
"evobgp/internal/store"
)
func TestConfigProviderReloadAndSnapshot(t *testing.T) {
mem := store.NewMemory()
ret := 3600
if _, err := mem.CreateMaintenancePolicy(&store.MaintenancePolicy{
Name: "p1",
TableName: "job_audit",
Schedule: "0 3 * * *",
RetentionPeriodSec: &ret,
Enabled: true,
}); err != nil {
t.Fatal(err)
}
cp := NewConfigProvider(mem)
if err := cp.Reload(context.Background()); err != nil {
t.Fatal(err)
}
snap := cp.Snapshot()
if len(snap) != 1 || snap[0].Name != "p1" {
t.Fatalf("snapshot: %+v", snap)
}
newName := "p1-updated"
if _, err := mem.UpdateMaintenancePolicy(snap[0].ID, &store.MaintenancePolicyPatch{Name: &newName}); err != nil {
t.Fatal(err)
}
if err := cp.Reload(context.Background()); err != nil {
t.Fatal(err)
}
snap2 := cp.Snapshot()
if len(snap2) != 1 || snap2[0].Name != newName {
t.Fatalf("after reload: %+v", snap2)
}
}
+62
View File
@@ -0,0 +1,62 @@
package maintenance
import (
"context"
"fmt"
"evobgp/internal/pgmonitor"
)
// TableHints are PostgreSQL statistics hints for UI recommendations.
type TableHints struct {
TableName string `json:"table_name"`
DeadTuples int64 `json:"n_dead_tup"`
BloatRatio float64 `json:"bloat_ratio,omitempty"`
LastAutovacuum string `json:"last_autovacuum,omitempty"`
RecommendVacuum bool `json:"recommend_vacuum"`
Detail string `json:"detail,omitempty"`
Refs []string `json:"refs,omitempty"`
}
// DBStatsProvider wraps pgmonitor for maintenance policy hints.
type DBStatsProvider struct {
pg *pgmonitor.Service
}
// NewDBStatsProvider constructs a stats provider.
func NewDBStatsProvider(pg *pgmonitor.Service) *DBStatsProvider {
return &DBStatsProvider{pg: pg}
}
// Hints returns table-level vacuum/bloat hints.
func (d *DBStatsProvider) Hints(ctx context.Context, tableName string) (TableHints, error) {
out := TableHints{TableName: tableName}
if d == nil || d.pg == nil {
return out, fmt.Errorf("maintenance: postgres monitoring not configured")
}
if err := ValidateTableName(tableName); err != nil {
return out, err
}
tables, err := d.pg.Tables(ctx, 100)
if err != nil {
return out, err
}
for _, t := range tables {
if t.Relname != tableName {
continue
}
out.DeadTuples = t.DeadTuples
out.BloatRatio = t.BloatRatio
if t.LastAutovacuum != nil {
out.LastAutovacuum = t.LastAutovacuum.UTC().Format("2006-01-02T15:04:05Z")
}
if t.BloatRatio > 0.2 && t.DeadTuples > 5000 {
out.RecommendVacuum = true
out.Detail = "Высокая доля n_dead_tup; рекомендуется VACUUM."
out.Refs = []string{t.Relname}
}
return out, nil
}
out.Detail = "Таблица не найдена в pg_stat_user_tables (top by size)."
return out, nil
}
+2
View File
@@ -0,0 +1,2 @@
// Package maintenance implements PostgreSQL maintenance policies loaded from the database.
package maintenance
+195
View File
@@ -0,0 +1,195 @@
package maintenance
import (
"context"
"fmt"
"hash/fnv"
"strings"
"time"
"evobgp/internal/observability"
"evobgp/internal/pgmonitor"
"evobgp/internal/store"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
// PolicyExecutor runs maintenance policies against PostgreSQL.
type PolicyExecutor struct {
Store store.Backend
Pool *pgxpool.Pool
}
// Execute runs cleanup and/or vacuum steps for a policy.
func (e *PolicyExecutor) Execute(ctx context.Context, policy *store.MaintenancePolicy, dryRun bool) (map[string]any, error) {
start := time.Now()
if policy == nil {
return nil, store.ErrInvalidInput
}
action := policyAction(policy)
record := func(status string, detail map[string]any) {
observability.RecordMaintenancePolicyRun(policy.ID, action, status, dryRun, time.Since(start), rowsDeletedFromDetail(detail))
}
if e == nil || e.Pool == nil {
record("failed", nil)
return nil, fmt.Errorf("maintenance: postgres not configured")
}
if err := ValidateTableName(policy.TableName); err != nil {
record("failed", nil)
return nil, err
}
if err := ValidateCondition(policy.Condition); err != nil {
record("failed", nil)
return nil, err
}
if !store.ValidVacuumStrategy(policy.VacuumStrategy) {
record("failed", nil)
return nil, store.ErrInvalidInput
}
detail := map[string]any{
"policy_id": policy.ID,
"table": policy.TableName,
"dry_run": dryRun,
}
if policy.RetentionPeriodSec != nil || policy.MaxRows != nil {
cleanupDetail, err := e.runCleanup(ctx, policy, dryRun)
for k, v := range cleanupDetail {
detail[k] = v
}
if err != nil {
record("failed", detail)
return detail, err
}
}
if policy.VacuumStrategy != store.VacuumStrategyNone {
kind := vacuumKind(policy.VacuumStrategy)
vacDetail, err := pgmonitor.ExecMaintenance(ctx, e.Pool, kind, policy.TableName, dryRun)
if vacDetail != nil {
detail["vacuum"] = vacDetail
}
if err != nil {
record("failed", detail)
return detail, err
}
}
if !dryRun && e.Store != nil {
_ = e.Store.TouchMaintenancePolicyRun(policy.ID, "succeeded", "")
}
record("succeeded", detail)
return detail, nil
}
func policyAction(p *store.MaintenancePolicy) string {
if p == nil {
return "run"
}
if p.RetentionPeriodSec != nil || p.MaxRows != nil {
if p.VacuumStrategy != store.VacuumStrategyNone {
return "cleanup_vacuum"
}
return "cleanup"
}
if p.VacuumStrategy != store.VacuumStrategyNone {
return p.VacuumStrategy
}
return "run"
}
func rowsDeletedFromDetail(detail map[string]any) int64 {
if detail == nil {
return 0
}
switch v := detail["deleted"].(type) {
case int64:
return v
case int:
return int64(v)
case float64:
return int64(v)
default:
return 0
}
}
func (e *PolicyExecutor) runCleanup(ctx context.Context, policy *store.MaintenancePolicy, dryRun bool) (map[string]any, error) {
detail := map[string]any{"cleanup": true}
limit := NormalizeBatchLimit(policy.MaxRows)
qualTable := pgx.Identifier{policy.TableName}.Sanitize()
cond := store.NormalizeMaintenancePolicyCondition(policy.Condition)
tx, err := e.Pool.Begin(ctx)
if err != nil {
return detail, fmt.Errorf("maintenance: begin tx: %w", err)
}
defer func() { _ = tx.Rollback(ctx) }()
lockKey := advisoryKey(policy.ID)
if _, err := tx.Exec(ctx, `SELECT pg_advisory_xact_lock($1)`, lockKey); err != nil {
return detail, fmt.Errorf("maintenance: advisory lock: %w", err)
}
if _, err := tx.Exec(ctx, fmt.Sprintf(`SET LOCAL statement_timeout = '%ds'`, DefaultStatementTimeoutSec)); err != nil {
return detail, fmt.Errorf("maintenance: statement_timeout: %w", err)
}
var args []any
where := cond
argN := 1
if policy.RetentionPeriodSec != nil && *policy.RetentionPeriodSec > 0 {
cutoff := time.Now().UTC().Add(-time.Duration(*policy.RetentionPeriodSec) * time.Second)
where = fmt.Sprintf("(%s) AND created_at < $%d", cond, argN)
args = append(args, cutoff)
argN++
}
countSQL := fmt.Sprintf(`SELECT count(*) FROM %s WHERE %s`, qualTable, where)
var wouldDelete int64
if err := tx.QueryRow(ctx, countSQL, args...).Scan(&wouldDelete); err != nil {
return detail, fmt.Errorf("maintenance: count: %w", err)
}
detail["would_delete"] = wouldDelete
if dryRun {
return detail, nil
}
deleteSQL := fmt.Sprintf(`
DELETE FROM %s WHERE ctid IN (
SELECT ctid FROM %s WHERE %s LIMIT $%d
)`, qualTable, qualTable, where, argN)
args = append(args, limit)
tag, err := tx.Exec(ctx, deleteSQL, args...)
if err != nil {
return detail, fmt.Errorf("maintenance: delete: %w", err)
}
if err := tx.Commit(ctx); err != nil {
return detail, fmt.Errorf("maintenance: commit: %w", err)
}
detail["deleted"] = tag.RowsAffected()
return detail, nil
}
func vacuumKind(strategy string) string {
switch strings.TrimSpace(strategy) {
case store.VacuumStrategyVacuum:
return "vacuum"
case store.VacuumStrategyAnalyze:
return "analyze"
case store.VacuumStrategyVacuumAnalyze:
return "vacuum_analyze"
case store.VacuumStrategyReindex:
return "reindex"
default:
return "vacuum"
}
}
func advisoryKey(policyID string) int64 {
h := fnv.New64a()
_, _ = h.Write([]byte("maint:" + policyID))
return int64(h.Sum64())
}
@@ -0,0 +1,161 @@
package maintenance
import (
"context"
"errors"
"strings"
"testing"
"evobgp/internal/store"
)
func validPolicy() *store.MaintenancePolicy {
ret := 86400
return &store.MaintenancePolicy{
ID: "11111111-1111-1111-1111-111111111111",
Name: "job audit",
TableName: "job_audit",
Condition: "true",
RetentionPeriodSec: &ret,
VacuumStrategy: store.VacuumStrategyNone,
Schedule: "0 3 * * *",
Enabled: true,
}
}
func TestPolicyExecutorExecuteValidation(t *testing.T) {
ctx := context.Background()
mem := store.NewMemory()
base := validPolicy()
tests := []struct {
name string
exec *PolicyExecutor
policy *store.MaintenancePolicy
wantErr error
contains string
}{
{
name: "nil policy",
exec: &PolicyExecutor{Store: mem},
policy: nil,
wantErr: store.ErrInvalidInput,
},
{
name: "nil pool",
exec: &PolicyExecutor{Store: mem, Pool: nil},
policy: base,
contains: "postgres not configured",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
_, err := tc.exec.Execute(ctx, tc.policy, false)
if tc.wantErr != nil {
if !errors.Is(err, tc.wantErr) {
t.Fatalf("Execute() err=%v want %v", err, tc.wantErr)
}
return
}
if err == nil {
t.Fatal("Execute() expected error")
}
if tc.contains != "" && !strings.Contains(err.Error(), tc.contains) {
t.Fatalf("Execute() err=%q want substring %q", err, tc.contains)
}
})
}
}
func TestPolicyExecutorPreExecuteValidation(t *testing.T) {
p := validPolicy()
p.TableName = "tenant"
if err := ValidateTableName(p.TableName); err == nil {
t.Fatal("expected blocked table error")
}
p = validPolicy()
p.Condition = "1=1; DROP TABLE job_audit"
if err := ValidateCondition(p.Condition); err == nil {
t.Fatal("expected unsafe condition error")
}
p = validPolicy()
p.VacuumStrategy = "invalid"
if !store.ValidVacuumStrategy(p.VacuumStrategy) {
return
}
t.Fatal("expected invalid vacuum strategy")
}
func TestPolicyAction(t *testing.T) {
ret := 3600
tests := []struct {
name string
p *store.MaintenancePolicy
want string
}{
{"nil", nil, "run"},
{"cleanup only", &store.MaintenancePolicy{RetentionPeriodSec: &ret, VacuumStrategy: store.VacuumStrategyNone}, "cleanup"},
{"vacuum only", &store.MaintenancePolicy{VacuumStrategy: store.VacuumStrategyVacuum}, "vacuum"},
{"cleanup+vacuum", &store.MaintenancePolicy{MaxRows: &ret, VacuumStrategy: store.VacuumStrategyAnalyze}, "cleanup_vacuum"},
{"noop run", &store.MaintenancePolicy{VacuumStrategy: store.VacuumStrategyNone}, "run"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if got := policyAction(tc.p); got != tc.want {
t.Fatalf("policyAction()=%q want %q", got, tc.want)
}
})
}
}
func TestVacuumKind(t *testing.T) {
tests := []struct {
strategy string
want string
}{
{store.VacuumStrategyVacuum, "vacuum"},
{store.VacuumStrategyAnalyze, "analyze"},
{store.VacuumStrategyVacuumAnalyze, "vacuum_analyze"},
{store.VacuumStrategyReindex, "reindex"},
{"unknown", "vacuum"},
}
for _, tc := range tests {
if got := vacuumKind(tc.strategy); got != tc.want {
t.Fatalf("vacuumKind(%q)=%q want %q", tc.strategy, got, tc.want)
}
}
}
func TestRowsDeletedFromDetail(t *testing.T) {
tests := []struct {
name string
detail map[string]any
want int64
}{
{"nil", nil, 0},
{"int64", map[string]any{"deleted": int64(42)}, 42},
{"int", map[string]any{"deleted": 7}, 7},
{"float64", map[string]any{"deleted": float64(3)}, 3},
{"missing", map[string]any{"other": 1}, 0},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if got := rowsDeletedFromDetail(tc.detail); got != tc.want {
t.Fatalf("rowsDeletedFromDetail()=%d want %d", got, tc.want)
}
})
}
}
func TestAdvisoryKeyStable(t *testing.T) {
a := advisoryKey("policy-a")
b := advisoryKey("policy-a")
c := advisoryKey("policy-b")
if a != b {
t.Fatal("advisory key not stable for same id")
}
if a == c {
t.Fatal("advisory key collision for different ids")
}
}
+72
View File
@@ -0,0 +1,72 @@
package maintenance
import (
"fmt"
"regexp"
"strings"
)
const (
DefaultBatchRows = 10000
MaxBatchRows = 100000
DefaultStatementTimeoutSec = 30
)
var (
blockedTableNames = map[string]struct{}{
"schema_migrations": {},
"tenant": {},
"maintenance_policy": {},
"maintenance_policy_config_audit": {},
}
sqlForbidden = regexp.MustCompile(`(?i)(;|--|/\*|\b(drop|truncate|insert|update|alter|create|grant|revoke|copy)\b)`)
)
// ValidateTableName ensures table is a safe identifier and not blocked.
func ValidateTableName(name string) error {
name = strings.TrimSpace(name)
if name == "" || !isSafeIdent(name) {
return fmt.Errorf("maintenance: invalid table name")
}
if _, blocked := blockedTableNames[strings.ToLower(name)]; blocked {
return fmt.Errorf("maintenance: table %q is not allowed", name)
}
return nil
}
// ValidateCondition ensures the WHERE fragment is safe for parameterized cleanup.
func ValidateCondition(condition string) error {
c := strings.TrimSpace(condition)
if c == "" {
return nil
}
if sqlForbidden.MatchString(c) {
return fmt.Errorf("maintenance: unsafe condition")
}
return nil
}
// NormalizeBatchLimit clamps delete batch size.
func NormalizeBatchLimit(maxRows *int) int {
if maxRows == nil || *maxRows <= 0 {
return DefaultBatchRows
}
if *maxRows > MaxBatchRows {
return MaxBatchRows
}
return *maxRows
}
func isSafeIdent(name string) bool {
if name == "" {
return false
}
for _, r := range name {
if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '_' {
continue
}
return false
}
return true
}
+46
View File
@@ -0,0 +1,46 @@
package maintenance
import "testing"
func TestValidateCondition(t *testing.T) {
tests := []struct {
cond string
ok bool
}{
{"true", true},
{"status IN ('succeeded', 'failed')", true},
{"1=1; DROP TABLE tenant", false},
{"x -- comment", false},
}
for _, tc := range tests {
err := ValidateCondition(tc.cond)
if tc.ok && err != nil {
t.Fatalf("cond %q: want ok, got %v", tc.cond, err)
}
if !tc.ok && err == nil {
t.Fatalf("cond %q: want error", tc.cond)
}
}
}
func TestValidateTableName(t *testing.T) {
if err := ValidateTableName("job_audit"); err != nil {
t.Fatal(err)
}
if err := ValidateTableName("tenant"); err == nil {
t.Fatal("expected blocked table")
}
if err := ValidateTableName("bad-name"); err == nil {
t.Fatal("expected invalid ident")
}
}
func TestNormalizeBatchLimit(t *testing.T) {
if got := NormalizeBatchLimit(nil); got != DefaultBatchRows {
t.Fatalf("default=%d got=%d", DefaultBatchRows, got)
}
max := 200000
if got := NormalizeBatchLimit(&max); got != MaxBatchRows {
t.Fatalf("max=%d got=%d", MaxBatchRows, got)
}
}
+85
View File
@@ -0,0 +1,85 @@
package maintenance
import (
"context"
"fmt"
"log"
"strings"
"sync"
"time"
"github.com/robfig/cron/v3"
)
// StartScheduler enqueues maintenance_policy_run jobs when cron schedules match.
func StartScheduler(ctx context.Context, provider *ConfigProvider, enqueue func(policyID string, dryRun bool, idempotencyKey string), tick time.Duration) {
if provider == nil || enqueue == nil {
return
}
if tick <= 0 {
tick = 30 * time.Second
}
go func() {
parser := cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow)
var mu sync.Mutex
schedules := map[string]cron.Schedule{}
lastFired := map[string]time.Time{}
rebuild := func() {
mu.Lock()
defer mu.Unlock()
schedules = map[string]cron.Schedule{}
for _, p := range provider.Snapshot() {
if p == nil || !p.Enabled || strings.TrimSpace(p.Schedule) == "" {
continue
}
sched, err := parser.Parse(p.Schedule)
if err != nil {
log.Printf("maintenance: invalid cron for policy %s: %v", p.ID, err)
continue
}
schedules[p.ID] = sched
}
}
rebuild()
t := time.NewTicker(tick)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
rebuild()
now := time.Now().UTC()
mu.Lock()
for _, p := range provider.Snapshot() {
if p == nil || !p.Enabled {
continue
}
sched, ok := schedules[p.ID]
if !ok {
continue
}
prev := lastFired[p.ID]
if prev.IsZero() {
prev = now.Add(-time.Minute)
}
next := sched.Next(prev)
if next.After(now) {
continue
}
slot := next.Unix() / 60
if lf, ok := lastFired[p.ID]; ok && lf.Unix()/60 == slot {
continue
}
lastFired[p.ID] = next
idem := fmt.Sprintf("maint-%s-%d", p.ID, slot)
enqueue(p.ID, p.DryRunEnabled, idem)
}
mu.Unlock()
}
}
}()
log.Printf("maintenance: policy scheduler started (tick=%s)", tick)
}
@@ -0,0 +1,72 @@
package observability
import (
"strconv"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var (
maintenancePolicyRuns = promauto.NewCounterVec(
prometheus.CounterOpts{
Namespace: namespace,
Name: "maintenance_policy_runs_total",
Help: "Maintenance policy executions by outcome.",
},
[]string{"policy_id", "action", "status", "dry_run"},
)
maintenancePolicyDuration = promauto.NewHistogramVec(
prometheus.HistogramOpts{
Namespace: namespace,
Name: "maintenance_policy_duration_seconds",
Help: "Maintenance policy execution duration.",
Buckets: prometheus.ExponentialBuckets(0.05, 2, 12),
},
[]string{"policy_id", "action"},
)
maintenanceRowsDeleted = promauto.NewCounterVec(
prometheus.CounterOpts{
Namespace: namespace,
Name: "maintenance_policy_rows_deleted_total",
Help: "Rows deleted by maintenance cleanup policies.",
},
[]string{"policy_id"},
)
maintenanceConfigChanges = promauto.NewCounterVec(
prometheus.CounterOpts{
Namespace: namespace,
Name: "maintenance_config_changes_total",
Help: "Maintenance policy configuration changes from UI/API.",
},
[]string{"action"},
)
)
// RecordMaintenancePolicyRun updates run counters and histograms.
func RecordMaintenancePolicyRun(policyID, action, status string, dryRun bool, duration time.Duration, rowsDeleted int64) {
if policyID == "" {
policyID = "unknown"
}
if action == "" {
action = "run"
}
dry := strconv.FormatBool(dryRun)
maintenancePolicyRuns.WithLabelValues(policyID, action, status, dry).Inc()
maintenancePolicyDuration.WithLabelValues(policyID, action).Observe(duration.Seconds())
if rowsDeleted > 0 && !dryRun {
maintenanceRowsDeleted.WithLabelValues(policyID).Add(float64(rowsDeleted))
}
}
// IncMaintenanceConfigChange increments config audit metric.
func IncMaintenanceConfigChange(action string) {
if action == "" {
action = "unknown"
}
maintenanceConfigChanges.WithLabelValues(action).Inc()
}
+37
View File
@@ -0,0 +1,37 @@
package pgmonitor
import (
"sync"
"time"
)
type cacheEntry struct {
at time.Time
data any
}
type ttlCache struct {
mu sync.RWMutex
ttl time.Duration
items map[string]cacheEntry
}
func newTTLCache(ttl time.Duration) *ttlCache {
return &ttlCache{ttl: ttl, items: make(map[string]cacheEntry)}
}
func (c *ttlCache) get(key string) (any, bool) {
c.mu.RLock()
defer c.mu.RUnlock()
e, ok := c.items[key]
if !ok || time.Since(e.at) > c.ttl {
return nil, false
}
return e.data, true
}
func (c *ttlCache) set(key string, data any) {
c.mu.Lock()
defer c.mu.Unlock()
c.items[key] = cacheEntry{at: time.Now().UTC(), data: data}
}
+83
View File
@@ -0,0 +1,83 @@
package pgmonitor
import (
"context"
"fmt"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
// Correlation builds aligned timeline points from job_audit and overview cache.
func (s *Service) Correlation(ctx context.Context, windowMinutes int) (CorrelationResponse, error) {
if s == nil || s.pool == nil {
return CorrelationResponse{}, fmt.Errorf("pgmonitor: postgres not configured")
}
if windowMinutes <= 0 {
windowMinutes = 60
}
if windowMinutes > 1440 {
windowMinutes = 1440
}
since := time.Now().UTC().Add(-time.Duration(windowMinutes) * time.Minute)
rows, err := s.pool.Query(ctx, `
SELECT date_trunc('minute', finished_at) AS bucket,
percentile_cont(0.99) WITHIN GROUP (ORDER BY
EXTRACT(EPOCH FROM (finished_at - started_at)) * 1000)
FROM job_audit
WHERE finished_at >= $1 AND kind IN ('module_refresh', 'tenant_refresh')
AND status = 'succeeded' AND started_at IS NOT NULL
GROUP BY 1
ORDER BY 1`, since)
if err != nil {
return CorrelationResponse{}, fmt.Errorf("pgmonitor: correlation jobs: %w", err)
}
defer rows.Close()
points := make(map[time.Time]*CorrelationPoint)
for rows.Next() {
var bucket time.Time
var p99 *float64
if err := rows.Scan(&bucket, &p99); err != nil {
return CorrelationResponse{}, err
}
bucket = bucket.UTC()
pt := points[bucket]
if pt == nil {
pt = &CorrelationPoint{Timestamp: bucket}
points[bucket] = pt
}
if p99 != nil {
pt.PipelineRefreshP99Ms = *p99
}
}
ov, err := s.Overview(ctx)
if err == nil && ov.Database.CacheHitPct > 0 {
now := time.Now().UTC().Truncate(time.Minute)
pt := points[now]
if pt == nil {
pt = &CorrelationPoint{Timestamp: now}
points[now] = pt
}
pt.CacheHitPct = ov.Database.CacheHitPct
}
out := make([]CorrelationPoint, 0, len(points))
for _, p := range points {
out = append(out, *p)
}
// simple sort by time
for i := 0; i < len(out); i++ {
for j := i + 1; j < len(out); j++ {
if out[j].Timestamp.Before(out[i].Timestamp) {
out[i], out[j] = out[j], out[i]
}
}
}
return CorrelationResponse{WindowMinutes: windowMinutes, Points: out}, nil
}
// RecordCorrelationSnapshot is a hook for future Prometheus samples (no-op placeholder).
func RecordCorrelationSnapshot(_ *pgxpool.Pool) {}
+96
View File
@@ -0,0 +1,96 @@
package pgmonitor
import (
"context"
"encoding/json"
"time"
"github.com/google/uuid"
"github.com/jackc/pgx/v5/pgxpool"
)
// CleanupRequest for deprecated POST /postgres/cleanup (use /v1/maintenance/run).
type CleanupRequest struct {
PolicyID string `json:"policy_id"`
Policy string `json:"policy"`
DryRun bool `json:"dry_run"`
Limit int `json:"limit"`
}
// InsertMaintenanceAudit records an audit row at job start.
func InsertMaintenanceAudit(ctx context.Context, pool *pgxpool.Pool, tenantID, actorPrefix, kind, table string, dryRun bool) (string, error) {
return InsertMaintenanceAuditWithPolicy(ctx, pool, tenantID, actorPrefix, kind, table, "", dryRun)
}
// InsertMaintenanceAuditWithPolicy records an audit row linked to maintenance_policy.
func InsertMaintenanceAuditWithPolicy(ctx context.Context, pool *pgxpool.Pool, tenantID, actorPrefix, kind, table, policyID string, dryRun bool) (string, error) {
id := uuid.New().String()
_, err := pool.Exec(ctx, `
INSERT INTO postgres_maintenance_audit
(id, tenant_id, actor_prefix, kind, target_table, policy_id, dry_run, status, created_at)
VALUES ($1, NULLIF($2,''), NULLIF($3,''), $4, NULLIF($5,''), NULLIF($6,''), $7, 'running', now())`,
id, tenantID, actorPrefix, kind, table, policyID, dryRun)
return id, err
}
// FinishMaintenanceAudit updates terminal state.
func FinishMaintenanceAudit(ctx context.Context, pool *pgxpool.Pool, id, status string, detail map[string]any, errMsg *string) error {
var detailJSON []byte
if detail != nil {
detailJSON, _ = json.Marshal(detail)
}
_, err := pool.Exec(ctx, `
UPDATE postgres_maintenance_audit
SET status = $2, detail_json = $3::jsonb, error_message = $4,
finished_at = now(), started_at = COALESCE(started_at, now())
WHERE id = $1`,
id, status, string(detailJSON), errMsg)
return err
}
// ListMaintenanceLogs returns paginated audit rows.
func ListMaintenanceLogs(ctx context.Context, pool *pgxpool.Pool, cursor string, limit int) ([]MaintenanceLogRow, string, bool, error) {
limit = clampLimit(limit, 20, 100)
args := []any{limit + 1}
q := `
SELECT id, COALESCE(tenant_id,''), COALESCE(actor_prefix,''), kind,
COALESCE(target_table,''), dry_run, status,
detail_json, COALESCE(error_message,''), created_at, started_at, finished_at
FROM postgres_maintenance_audit`
if cursor != "" {
q += ` WHERE created_at < (SELECT created_at FROM postgres_maintenance_audit WHERE id = $2)`
args = append(args, cursor)
}
q += ` ORDER BY created_at DESC LIMIT $1`
rows, err := pool.Query(ctx, q, args...)
if err != nil {
return nil, "", false, err
}
defer rows.Close()
var out []MaintenanceLogRow
for rows.Next() {
var r MaintenanceLogRow
var detailRaw []byte
var started, finished *time.Time
if err := rows.Scan(&r.ID, &r.TenantID, &r.ActorPrefix, &r.Kind, &r.TargetTable,
&r.DryRun, &r.Status, &detailRaw, &r.Error, &r.CreatedAt, &started, &finished); err != nil {
return nil, "", false, err
}
r.StartedAt = started
r.FinishedAt = finished
if len(detailRaw) > 0 {
_ = json.Unmarshal(detailRaw, &r.Detail)
}
out = append(out, r)
}
hasMore := len(out) > limit
if hasMore {
out = out[:limit]
}
next := ""
if hasMore && len(out) > 0 {
next = out[len(out)-1].ID
}
return out, next, hasMore, rows.Err()
}
+350
View File
@@ -0,0 +1,350 @@
package pgmonitor
import (
"context"
"errors"
"fmt"
"strings"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
"github.com/jackc/pgx/v5/pgxpool"
)
func clampLimit(limit, def, max int) int {
if limit <= 0 {
return def
}
if limit > max {
return max
}
return limit
}
func (s *Service) fetchOverview(ctx context.Context) (Overview, error) {
now := time.Now().UTC()
out := Overview{CollectedAt: now}
var active, idle, total, maxConn int
err := s.pool.QueryRow(ctx, `
SELECT
count(*) FILTER (WHERE state = 'active'),
count(*) FILTER (WHERE state = 'idle'),
count(*),
(SELECT setting::int FROM pg_settings WHERE name = 'max_connections')
FROM pg_stat_activity
WHERE datname = current_database()`).Scan(&active, &idle, &total, &maxConn)
if err != nil {
return out, fmt.Errorf("pgmonitor: connections: %w", err)
}
out.Connections = Connections{Active: active, Idle: idle, Total: total, MaxConnections: maxConn}
var cachePct *float64
err = s.pool.QueryRow(ctx, `
SELECT numbackends, xact_commit, xact_rollback, deadlocks, blks_hit, blks_read,
CASE WHEN blks_hit + blks_read > 0
THEN round(100.0 * blks_hit::numeric / (blks_hit + blks_read), 2) END
FROM pg_stat_database WHERE datname = current_database()`).Scan(
&out.Database.Backends,
&out.Database.XactCommit,
&out.Database.XactRollback,
&out.Database.Deadlocks,
&out.Database.BlksHit,
&out.Database.BlksRead,
&cachePct,
)
if err != nil {
return out, fmt.Errorf("pgmonitor: database stats: %w", err)
}
if cachePct != nil {
out.Database.CacheHitPct = *cachePct
}
_ = s.pool.QueryRow(ctx, `
SELECT checkpoints_timed, checkpoints_req, buffers_checkpoint, buffers_clean,
maxwritten_clean, buffers_backend, buffers_alloc
FROM pg_stat_bgwriter`).Scan(
&out.Bgwriter.CheckpointsTimed,
&out.Bgwriter.CheckpointsReq,
&out.Bgwriter.BuffersCheckpoint,
&out.Bgwriter.BuffersClean,
&out.Bgwriter.MaxWrittenClean,
&out.Bgwriter.BuffersBackend,
&out.Bgwriter.BuffersAlloc,
)
_ = s.pool.QueryRow(ctx, `SELECT pg_database_size(current_database())`).Scan(&out.SizeBytes)
_ = s.pool.QueryRow(ctx, `
SELECT
(SELECT setting FROM pg_settings WHERE name = 'shared_buffers'),
(SELECT setting FROM pg_settings WHERE name = 'work_mem'),
(SELECT setting FROM pg_settings WHERE name = 'effective_cache_size')`).Scan(
&out.MemorySettings.SharedBuffers,
&out.MemorySettings.WorkMem,
&out.MemorySettings.EffectiveCacheSize,
)
rows, err := s.pool.Query(ctx, `
SELECT client_addr::text, state, sync_state,
EXTRACT(EPOCH FROM COALESCE(write_lag, flush_lag, replay_lag)) * 1000
FROM pg_stat_replication`)
if err == nil {
defer rows.Close()
for rows.Next() {
var peer ReplicationPeer
var lagMs *float64
if err := rows.Scan(&peer.ClientAddr, &peer.State, &peer.SyncState, &lagMs); err != nil {
continue
}
if lagMs != nil {
v := int64(*lagMs)
peer.LagMs = &v
}
out.Replication = append(out.Replication, peer)
}
}
out.StatementsEnabled = s.statementsQueryable(ctx)
return out, nil
}
func queryLocks(ctx context.Context, pool *pgxpool.Pool) ([]LockRow, error) {
rows, err := pool.Query(ctx, `
SELECT l.locktype, l.mode, l.granted, a.pid, COALESCE(a.usename, ''),
COALESCE(a.state, ''), COALESCE(left(a.query, 300), ''),
NOT l.granted AS blocked
FROM pg_locks l
JOIN pg_stat_activity a ON a.pid = l.pid
WHERE a.datname = current_database()
AND (NOT l.granted OR l.mode LIKE '%Exclusive%')
ORDER BY l.granted ASC, a.query_start NULLS LAST
LIMIT 200`)
if err != nil {
return nil, fmt.Errorf("pgmonitor: locks: %w", err)
}
defer rows.Close()
var out []LockRow
for rows.Next() {
var r LockRow
if err := rows.Scan(&r.Locktype, &r.Mode, &r.Granted, &r.PID, &r.User, &r.State, &r.Query, &r.Blocked); err != nil {
return nil, err
}
out = append(out, r)
}
return out, rows.Err()
}
func queryTables(ctx context.Context, pool *pgxpool.Pool, limit int) ([]TableStat, error) {
limit = clampLimit(limit, 20, 100)
rows, err := pool.Query(ctx, `
SELECT t.relname,
pg_total_relation_size(t.relid),
s.heap_blks_read, s.heap_blks_hit,
t.idx_scan, t.seq_scan, t.n_dead_tup, t.last_autovacuum,
CASE WHEN t.n_live_tup + t.n_dead_tup > 0
THEN round(t.n_dead_tup::numeric / (t.n_live_tup + t.n_dead_tup), 4)
ELSE 0 END
FROM pg_statio_user_tables s
JOIN pg_stat_user_tables t ON t.relid = s.relid
WHERE t.schemaname = 'public'
ORDER BY pg_total_relation_size(t.relid) DESC
LIMIT $1`, limit)
if err != nil {
return nil, fmt.Errorf("pgmonitor: tables: %w", err)
}
defer rows.Close()
var out []TableStat
for rows.Next() {
var r TableStat
var last *time.Time
if err := rows.Scan(&r.Relname, &r.TotalBytes, &r.HeapBlksRead, &r.HeapBlksHit,
&r.IdxScan, &r.SeqScan, &r.DeadTuples, &last, &r.BloatRatio); err != nil {
return nil, err
}
r.LastAutovacuum = last
out = append(out, r)
}
return out, rows.Err()
}
// TopQueries loads from pg_stat_statements when available.
func (s *Service) TopQueries(ctx context.Context, limit int) (QueriesResponse, error) {
if s == nil || s.pool == nil {
return QueriesResponse{}, errors.New("pgmonitor: postgres not configured")
}
limit = clampLimit(limit, 20, 100)
now := time.Now().UTC()
if snap, ok, err := s.loadSnapshot(ctx, "slow_queries", 15*time.Minute); err == nil && ok {
var items []QueryStat
if err := decodePayload(snap.Payload, &items); err == nil {
return QueriesResponse{
CollectedAt: snap.CollectedAt,
Source: "snapshot",
Items: items,
StatementsAvailable: true,
}, nil
}
}
if !s.statementsQueryable(ctx) {
return queriesUnavailable(now), nil
}
items, err := queryTopStatements(ctx, s.pool, limit)
if err != nil {
if isPgStatStatementsUnavailable(err) {
s.markStatementsUnavailable()
return queriesUnavailable(now), nil
}
return QueriesResponse{}, err
}
return QueriesResponse{
CollectedAt: now,
Source: "live",
Items: items,
StatementsAvailable: true,
}, nil
}
func queriesUnavailable(at time.Time) QueriesResponse {
return QueriesResponse{
CollectedAt: at,
Source: "unavailable",
Items: nil,
StatementsAvailable: false,
StatementsHint: statementsUnavailableHint,
}
}
const statementsUnavailableHint = "pg_stat_statements requires shared_preload_libraries and PostgreSQL restart (see docs/db-diagnostics.md)"
// statementsQueryable returns true only when pg_stat_statements can be queried (not merely installed).
func (s *Service) statementsQueryable(ctx context.Context) bool {
if s == nil || s.pool == nil {
return false
}
if v, ok := s.cache.get("stmt_queryable"); ok {
if b, ok := v.(bool); ok {
return b
}
}
ok := probePgStatStatements(ctx, s.pool)
s.cache.set("stmt_queryable", ok)
return ok
}
func (s *Service) markStatementsUnavailable() {
s.cache.set("stmt_queryable", false)
}
func probePgStatStatements(ctx context.Context, pool *pgxpool.Pool) bool {
var dummy int64
err := pool.QueryRow(ctx, `
SELECT COALESCE(SUM(calls), 0)::bigint FROM pg_stat_statements LIMIT 1`).Scan(&dummy)
if err == nil {
return true
}
return !isPgStatStatementsUnavailable(err)
}
func queryTopStatements(ctx context.Context, pool *pgxpool.Pool, limit int) ([]QueryStat, error) {
rows, err := pool.Query(ctx, `
SELECT queryid, left(query, 500), calls, total_exec_time, mean_exec_time, rows
FROM pg_stat_statements
WHERE dbid = (SELECT oid FROM pg_database WHERE datname = current_database())
ORDER BY mean_exec_time DESC
LIMIT $1`, limit)
if err != nil {
if isPgStatStatementsUnavailable(err) {
return nil, nil
}
return nil, fmt.Errorf("pgmonitor: pg_stat_statements: %w", err)
}
defer rows.Close()
var out []QueryStat
for rows.Next() {
var r QueryStat
if err := rows.Scan(&r.QueryID, &r.Query, &r.Calls, &r.TotalExecMs, &r.MeanExecMs, &r.Rows); err != nil {
return nil, err
}
out = append(out, r)
}
return out, rows.Err()
}
// isPgStatStatementsUnavailable reports extension missing or not loaded via shared_preload_libraries.
func isPgStatStatementsUnavailable(err error) bool {
if err == nil {
return false
}
var pgErr *pgconn.PgError
if errors.As(err, &pgErr) {
switch pgErr.Code {
case "42P01", "42704", "55000":
return true
}
msg := strings.ToLower(pgErr.Message)
if strings.Contains(msg, "shared_preload_libraries") || strings.Contains(msg, "pg_stat_statements") {
return true
}
}
low := strings.ToLower(err.Error())
return strings.Contains(low, "shared_preload_libraries") || strings.Contains(low, "pg_stat_statements")
}
func isSafeIdent(name string) bool {
if name == "" {
return true
}
for _, r := range name {
if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '_' {
continue
}
return false
}
return true
}
// ExecMaintenance runs VACUUM/ANALYZE/REINDEX with optional dry-run (returns SQL executed or planned).
func ExecMaintenance(ctx context.Context, pool *pgxpool.Pool, kind, table string, dryRun bool) (detail map[string]any, err error) {
if pool == nil {
return nil, errors.New("pgmonitor: postgres not configured")
}
table = strings.TrimSpace(table)
if table != "" && !isSafeIdent(table) {
return nil, errors.New("pgmonitor: invalid table name")
}
qual := ""
if table != "" {
qual = " " + pgx.Identifier{table}.Sanitize()
}
var sql string
switch kind {
case "vacuum":
sql = "VACUUM" + qual
case "vacuum_analyze":
sql = "VACUUM ANALYZE" + qual
case "analyze":
sql = "ANALYZE" + qual
case "reindex":
if table == "" {
return nil, errors.New("pgmonitor: reindex requires table")
}
sql = "REINDEX TABLE" + qual
default:
return nil, fmt.Errorf("pgmonitor: unknown maintenance kind %q", kind)
}
detail = map[string]any{"sql": sql, "dry_run": dryRun}
if dryRun {
return detail, nil
}
_, err = pool.Exec(ctx, sql)
if err != nil {
return detail, fmt.Errorf("pgmonitor: %s: %w", kind, err)
}
detail["executed"] = true
return detail, nil
}
+104
View File
@@ -0,0 +1,104 @@
package pgmonitor
import (
"context"
"time"
)
// Recommendations builds heuristic items from live stats and snapshots.
func (s *Service) Recommendations(ctx context.Context) (RecommendationsResponse, error) {
now := time.Now().UTC()
var items []RecommendationItem
ov, err := s.Overview(ctx)
if err == nil {
if ov.Database.CacheHitPct > 0 && ov.Database.CacheHitPct < 90 {
items = append(items, RecommendationItem{
Severity: "warn",
Code: "low_cache_hit",
Title: "Низкий cache hit ratio",
Detail: "Buffer cache hit ниже 90%; проверьте shared_buffers и горячие seq scan.",
})
}
if ov.Database.Deadlocks > 0 {
items = append(items, RecommendationItem{
Severity: "warn",
Code: "deadlocks",
Title: "Зафиксированы deadlocks",
Detail: "Проверьте конкурирующие транзакции и порядок блокировок.",
})
}
if ov.Connections.MaxConnections > 0 &&
float64(ov.Connections.Total)/float64(ov.Connections.MaxConnections) > 0.8 {
items = append(items, RecommendationItem{
Severity: "critical",
Code: "connections_high",
Title: "Много подключений к PostgreSQL",
Detail: "Использование max_connections выше 80%; увеличьте pool tuning или лимит.",
})
}
}
tables, err := s.Tables(ctx, 30)
if err == nil {
for _, t := range tables {
if t.SeqScan > 1000 && t.IdxScan < t.SeqScan/10 {
items = append(items, RecommendationItem{
Severity: "warn",
Code: "missing_index",
Title: "Высокий seq_scan",
Detail: "Таблица часто сканируется последовательно; рассмотрите индекс.",
Refs: []string{t.Relname},
})
}
if t.BloatRatio > 0.2 && t.DeadTuples > 5000 {
items = append(items, RecommendationItem{
Severity: "info",
Code: "autovacuum_lag",
Title: "Возможный bloat / мёртвые строки",
Detail: "Высокая доля n_dead_tup; запланируйте VACUUM.",
Refs: []string{t.Relname},
})
}
}
}
if snap, ok, _ := s.loadSnapshot(ctx, "unused_indexes", 30*time.Minute); ok {
type unused struct {
Index string `json:"index"`
SizeBytes int64 `json:"size_bytes"`
}
var list []unused
if decodePayload(snap.Payload, &list) == nil {
for _, u := range list {
if u.SizeBytes < 1024*1024 {
continue
}
items = append(items, RecommendationItem{
Severity: "info",
Code: "unused_index",
Title: "Неиспользуемый индекс",
Detail: "idx_scan=0; проверьте перед удалением.",
Refs: []string{u.Index},
})
}
}
}
q, err := s.TopQueries(ctx, 5)
if err == nil {
for _, qs := range q.Items {
if qs.MeanExecMs > 500 {
items = append(items, RecommendationItem{
Severity: "warn",
Code: "slow_query",
Title: "Медленный запрос",
Detail: "Среднее время выполнения выше 500ms.",
Refs: []string{qs.Query},
})
}
}
}
return RecommendationsResponse{CollectedAt: now, Items: items}, nil
}
+59
View File
@@ -0,0 +1,59 @@
package pgmonitor
import (
"context"
"log"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
// StartScheduler runs periodic PostgreSQL analyzer snapshots until ctx is cancelled.
func StartScheduler(ctx context.Context, pool *pgxpool.Pool) {
if pool == nil {
return
}
go func() {
t5 := time.NewTicker(5 * time.Minute)
t15 := time.NewTicker(15 * time.Minute)
defer t5.Stop()
defer t15.Stop()
s := NewService(pool)
runLight := func() {
c, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
if err := s.RefreshMetricsSnapshot(c); err != nil {
log.Printf("pgmonitor: metrics refresh: %v", err)
}
if err := s.DetectAutovacuumLag(c); err != nil {
log.Printf("pgmonitor: autovacuum lag: %v", err)
}
}
runHeavy := func() {
c, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
defer cancel()
if err := s.AggregateSlowQueries(c, 30); err != nil {
log.Printf("pgmonitor: slow queries snapshot: %v", err)
}
if err := s.EstimateTableBloat(c); err != nil {
log.Printf("pgmonitor: bloat: %v", err)
}
if err := s.AnalyzeIndexUsage(c); err != nil {
log.Printf("pgmonitor: index usage: %v", err)
}
}
runLight()
runHeavy()
for {
select {
case <-ctx.Done():
return
case <-t5.C:
runLight()
case <-t15.C:
runHeavy()
}
}
}()
log.Printf("pgmonitor: scheduler started (5m light / 15m heavy)")
}
+90
View File
@@ -0,0 +1,90 @@
package pgmonitor
import (
"context"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
// Service provides PostgreSQL observability and maintenance helpers (control plane instance scope).
type Service struct {
pool *pgxpool.Pool
cache *ttlCache
}
// NewService constructs a metrics service for the API PostgreSQL pool.
func NewService(pool *pgxpool.Pool) *Service {
if pool == nil {
return nil
}
return &Service{
pool: pool,
cache: newTTLCache(10 * time.Second),
}
}
// Pool exposes the underlying pool for job workers.
func (s *Service) Pool() *pgxpool.Pool {
if s == nil {
return nil
}
return s.pool
}
// Overview returns cached instance-level stats.
func (s *Service) Overview(ctx context.Context) (Overview, error) {
if s == nil || s.pool == nil {
return Overview{}, errors.New("pgmonitor: postgres not configured")
}
if v, ok := s.cache.get("overview"); ok {
if o, ok := v.(Overview); ok {
return o, nil
}
}
o, err := s.fetchOverview(ctx)
if err != nil {
return Overview{}, err
}
s.cache.set("overview", o)
return o, nil
}
// Locks returns active / blocking locks.
func (s *Service) Locks(ctx context.Context) ([]LockRow, error) {
if s == nil || s.pool == nil {
return nil, errors.New("pgmonitor: postgres not configured")
}
if v, ok := s.cache.get("locks"); ok {
if rows, ok := v.([]LockRow); ok {
return rows, nil
}
}
rows, err := queryLocks(ctx, s.pool)
if err != nil {
return nil, err
}
s.cache.set("locks", rows)
return rows, nil
}
// Tables returns top tables by size with I/O stats.
func (s *Service) Tables(ctx context.Context, limit int) ([]TableStat, error) {
if s == nil || s.pool == nil {
return nil, errors.New("pgmonitor: postgres not configured")
}
key := fmt.Sprintf("tables:%d", limit)
if v, ok := s.cache.get(key); ok {
if rows, ok := v.([]TableStat); ok {
return rows, nil
}
}
rows, err := queryTables(ctx, s.pool, limit)
if err != nil {
return nil, err
}
s.cache.set(key, rows)
return rows, nil
}
+48
View File
@@ -0,0 +1,48 @@
package pgmonitor
import (
"errors"
"testing"
"github.com/jackc/pgx/v5/pgconn"
)
func TestClampLimit(t *testing.T) {
if clampLimit(0, 20, 100) != 20 {
t.Fatal("default")
}
if clampLimit(200, 20, 100) != 100 {
t.Fatal("max")
}
if clampLimit(5, 20, 100) != 5 {
t.Fatal("value")
}
}
func TestIsSafeIdent(t *testing.T) {
if !isSafeIdent("revision_materialized_prefix") {
t.Fatal("valid")
}
if isSafeIdent("bad-name") {
t.Fatal("invalid")
}
if !isSafeIdent("") {
t.Fatal("empty ok")
}
}
func TestNewServiceNilPool(t *testing.T) {
if NewService(nil) != nil {
t.Fatal("expected nil service")
}
}
func TestIsPgStatStatementsUnavailable(t *testing.T) {
err := &pgconn.PgError{Code: "55000", Message: "pg_stat_statements must be loaded via shared_preload_libraries"}
if !isPgStatStatementsUnavailable(err) {
t.Fatal("55000")
}
if isPgStatStatementsUnavailable(errors.New("other")) {
t.Fatal("unrelated")
}
}
+162
View File
@@ -0,0 +1,162 @@
package pgmonitor
import (
"context"
"encoding/json"
"fmt"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
type snapshotRow struct {
ID string
CollectedAt time.Time
Payload json.RawMessage
}
func (s *Service) loadSnapshot(ctx context.Context, id string, maxAge time.Duration) (snapshotRow, bool, error) {
var row snapshotRow
err := s.pool.QueryRow(ctx, `
SELECT id, collected_at, payload_json
FROM postgres_monitor_snapshot
WHERE id = $1 AND collected_at >= $2`,
id, time.Now().UTC().Add(-maxAge)).Scan(&row.ID, &row.CollectedAt, &row.Payload)
if err != nil {
return snapshotRow{}, false, nil
}
return row, true, nil
}
func (s *Service) UpsertSnapshot(ctx context.Context, id string, payload any) error {
if s == nil || s.pool == nil {
return fmt.Errorf("pgmonitor: postgres not configured")
}
b, err := json.Marshal(payload)
if err != nil {
return err
}
_, err = s.pool.Exec(ctx, `
INSERT INTO postgres_monitor_snapshot (id, collected_at, payload_json)
VALUES ($1, now(), $2::jsonb)
ON CONFLICT (id) DO UPDATE SET collected_at = EXCLUDED.collected_at, payload_json = EXCLUDED.payload_json`,
id, string(b))
return err
}
func decodePayload(raw json.RawMessage, dest any) error {
return json.Unmarshal(raw, dest)
}
// RefreshMetricsSnapshot stores overview and tables for heavy reads.
func (s *Service) RefreshMetricsSnapshot(ctx context.Context) error {
ov, err := s.fetchOverview(ctx)
if err != nil {
return err
}
if err := s.UpsertSnapshot(ctx, "overview", ov); err != nil {
return err
}
tables, err := queryTables(ctx, s.pool, 50)
if err != nil {
return err
}
return s.UpsertSnapshot(ctx, "tables", tables)
}
// AggregateSlowQueries stores top statements snapshot.
func (s *Service) AggregateSlowQueries(ctx context.Context, limit int) error {
if !s.statementsQueryable(ctx) {
return s.UpsertSnapshot(ctx, "slow_queries", []QueryStat{})
}
items, err := queryTopStatements(ctx, s.pool, clampLimit(limit, 20, 100))
if err != nil {
if isPgStatStatementsUnavailable(err) {
s.markStatementsUnavailable()
return s.UpsertSnapshot(ctx, "slow_queries", []QueryStat{})
}
return err
}
return s.UpsertSnapshot(ctx, "slow_queries", items)
}
// EstimateTableBloat refreshes bloat heuristics on tables snapshot.
func (s *Service) EstimateTableBloat(ctx context.Context) error {
tables, err := queryTables(ctx, s.pool, 100)
if err != nil {
return err
}
return s.UpsertSnapshot(ctx, "table_bloat", tables)
}
// AnalyzeIndexUsage stores unused indexes.
func (s *Service) AnalyzeIndexUsage(ctx context.Context) error {
rows, err := s.pool.Query(ctx, `
SELECT indexrelname, idx_scan, pg_relation_size(indexrelid)
FROM pg_stat_user_indexes
WHERE schemaname = 'public' AND idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC
LIMIT 50`)
if err != nil {
return fmt.Errorf("pgmonitor: index usage: %w", err)
}
defer rows.Close()
type unused struct {
Index string `json:"index"`
IdxScan int64 `json:"idx_scan"`
SizeBytes int64 `json:"size_bytes"`
}
var items []unused
for rows.Next() {
var u unused
if err := rows.Scan(&u.Index, &u.IdxScan, &u.SizeBytes); err != nil {
return err
}
items = append(items, u)
}
return s.UpsertSnapshot(ctx, "unused_indexes", items)
}
// DetectAutovacuumLag stores tables with high dead tuple ratio.
func (s *Service) DetectAutovacuumLag(ctx context.Context) error {
rows, err := s.pool.Query(ctx, `
SELECT relname, n_dead_tup, last_autovacuum,
CASE WHEN n_live_tup + n_dead_tup > 0
THEN round(n_dead_tup::numeric / (n_live_tup + n_dead_tup), 4) ELSE 0 END
FROM pg_stat_user_tables
WHERE schemaname = 'public' AND n_dead_tup > 1000
ORDER BY n_dead_tup DESC
LIMIT 30`)
if err != nil {
return fmt.Errorf("pgmonitor: autovacuum lag: %w", err)
}
defer rows.Close()
type lagRow struct {
Relname string `json:"relname"`
DeadTuples int64 `json:"n_dead_tup"`
LastAutovacuum *time.Time `json:"last_autovacuum,omitempty"`
Ratio float64 `json:"ratio"`
}
var items []lagRow
for rows.Next() {
var r lagRow
if err := rows.Scan(&r.Relname, &r.DeadTuples, &r.LastAutovacuum, &r.Ratio); err != nil {
return err
}
items = append(items, r)
}
return s.UpsertSnapshot(ctx, "autovacuum_lag", items)
}
// RunPeriodicAnalyzerJobs runs all snapshot analyzers (for scheduler).
func RunPeriodicAnalyzerJobs(ctx context.Context, pool *pgxpool.Pool) {
s := NewService(pool)
if s == nil {
return
}
_ = s.RefreshMetricsSnapshot(ctx)
_ = s.AggregateSlowQueries(ctx, 30)
_ = s.EstimateTableBloat(ctx)
_ = s.AnalyzeIndexUsage(ctx)
_ = s.DetectAutovacuumLag(ctx)
}
+150
View File
@@ -0,0 +1,150 @@
package pgmonitor
import "time"
// Overview is instance-level PostgreSQL health snapshot.
type Overview struct {
CollectedAt time.Time `json:"collected_at"`
Connections Connections `json:"connections"`
Database DatabaseStats `json:"database"`
Bgwriter BgwriterStats `json:"bgwriter"`
SizeBytes int64 `json:"database_size_bytes"`
MemorySettings MemorySettings `json:"memory_settings"`
Replication []ReplicationPeer `json:"replication"`
StatementsEnabled bool `json:"pg_stat_statements_enabled"`
}
// Connections summarizes pg_stat_activity for current database.
type Connections struct {
Active int `json:"active"`
Idle int `json:"idle"`
Total int `json:"total"`
MaxConnections int `json:"max_connections"`
}
// DatabaseStats from pg_stat_database.
type DatabaseStats struct {
Backends int `json:"backends"`
XactCommit int64 `json:"xact_commit"`
XactRollback int64 `json:"xact_rollback"`
Deadlocks int64 `json:"deadlocks"`
BlksHit int64 `json:"blks_hit"`
BlksRead int64 `json:"blks_read"`
CacheHitPct float64 `json:"cache_hit_pct"`
}
// BgwriterStats from pg_stat_bgwriter.
type BgwriterStats struct {
CheckpointsTimed int64 `json:"checkpoints_timed"`
CheckpointsReq int64 `json:"checkpoints_req"`
BuffersCheckpoint int64 `json:"buffers_checkpoint"`
BuffersClean int64 `json:"buffers_clean"`
MaxWrittenClean int64 `json:"maxwritten_clean"`
BuffersBackend int64 `json:"buffers_backend"`
BuffersAlloc int64 `json:"buffers_alloc"`
}
// MemorySettings is best-effort from pg_settings (not RSS).
type MemorySettings struct {
SharedBuffers string `json:"shared_buffers"`
WorkMem string `json:"work_mem"`
EffectiveCacheSize string `json:"effective_cache_size"`
}
// ReplicationPeer from pg_stat_replication.
type ReplicationPeer struct {
ClientAddr string `json:"client_addr,omitempty"`
State string `json:"state"`
SyncState string `json:"sync_state,omitempty"`
LagMs *int64 `json:"lag_ms,omitempty"`
}
// QueryStat is a row from pg_stat_statements or snapshot.
type QueryStat struct {
QueryID int64 `json:"queryid,omitempty"`
Query string `json:"query"`
Calls int64 `json:"calls"`
TotalExecMs float64 `json:"total_exec_ms"`
MeanExecMs float64 `json:"mean_exec_ms"`
Rows int64 `json:"rows"`
}
// QueriesResponse for GET /monitoring/postgres/queries.
type QueriesResponse struct {
CollectedAt time.Time `json:"collected_at"`
Source string `json:"source"` // live | snapshot | unavailable
Items []QueryStat `json:"items"`
StatementsAvailable bool `json:"statements_available"`
StatementsHint string `json:"statements_hint,omitempty"`
}
// LockRow describes a lock / blocking session.
type LockRow struct {
Locktype string `json:"locktype"`
Mode string `json:"mode"`
Granted bool `json:"granted"`
PID int32 `json:"pid"`
User string `json:"usename,omitempty"`
State string `json:"state,omitempty"`
Query string `json:"query,omitempty"`
Blocked bool `json:"blocked"`
}
// TableStat combines size and scan stats for a user table.
type TableStat struct {
Relname string `json:"relname"`
TotalBytes int64 `json:"total_bytes"`
HeapBlksRead int64 `json:"heap_blks_read"`
HeapBlksHit int64 `json:"heap_blks_hit"`
IdxScan int64 `json:"idx_scan"`
SeqScan int64 `json:"seq_scan"`
DeadTuples int64 `json:"n_dead_tup"`
LastAutovacuum *time.Time `json:"last_autovacuum,omitempty"`
BloatRatio float64 `json:"bloat_ratio,omitempty"`
}
// RecommendationItem is a heuristic ops hint.
type RecommendationItem struct {
Severity string `json:"severity"` // info | warn | critical
Code string `json:"code"`
Title string `json:"title"`
Detail string `json:"detail"`
Refs []string `json:"refs,omitempty"`
}
// RecommendationsResponse for GET recommendations.
type RecommendationsResponse struct {
CollectedAt time.Time `json:"collected_at"`
Items []RecommendationItem `json:"items"`
}
// CorrelationPoint is one aligned sample for overlay charts.
type CorrelationPoint struct {
Timestamp time.Time `json:"timestamp"`
PipelineRefreshP99Ms float64 `json:"pipeline_refresh_p99_ms,omitempty"`
BirdScrapeOK *float64 `json:"bird_scrape_ok,omitempty"`
HTTPRequestRate float64 `json:"http_request_rate,omitempty"`
CacheHitPct float64 `json:"cache_hit_pct,omitempty"`
}
// CorrelationResponse for GET /monitoring/correlation.
type CorrelationResponse struct {
WindowMinutes int `json:"window_minutes"`
Points []CorrelationPoint `json:"points"`
}
// MaintenanceLogRow is an audit entry.
type MaintenanceLogRow struct {
ID string `json:"id"`
TenantID string `json:"tenant_id,omitempty"`
ActorPrefix string `json:"actor_prefix,omitempty"`
Kind string `json:"kind"`
TargetTable string `json:"target_table,omitempty"`
DryRun bool `json:"dry_run"`
Status string `json:"status"`
Detail map[string]any `json:"detail,omitempty"`
Error string `json:"error,omitempty"`
CreatedAt time.Time `json:"created_at"`
StartedAt *time.Time `json:"started_at,omitempty"`
FinishedAt *time.Time `json:"finished_at,omitempty"`
}
+61 -14
View File
@@ -11,14 +11,22 @@ import (
"github.com/jackc/pgx/v5"
)
func asnCacheRowTableExists(ctx context.Context, q queryRower) bool {
var n int
err := q.QueryRow(ctx, `
SELECT 1 FROM information_schema.tables
WHERE table_schema = 'public' AND table_name = 'asn_prefix_cache_row'
LIMIT 1`).Scan(&n)
return err == nil
}
func (p *Postgres) GetASNPrefixCache(asn int64) (*store.ASNPrefixCacheEntry, bool, error) {
ctx := context.Background()
var holder string
var fetchedAt time.Time
var raw []byte
err := p.pool.QueryRow(ctx, `
SELECT holder, fetched_at, prefixes_json FROM asn_prefix_cache WHERE asn = $1`, asn).
Scan(&holder, &fetchedAt, &raw)
SELECT holder, fetched_at FROM asn_prefix_cache WHERE asn = $1`, asn).
Scan(&holder, &fetchedAt)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return nil, false, nil
@@ -26,8 +34,25 @@ func (p *Postgres) GetASNPrefixCache(asn int64) (*store.ASNPrefixCacheEntry, boo
return nil, false, err
}
var prefixes []string
if len(raw) > 0 {
_ = json.Unmarshal(raw, &prefixes)
if asnCacheRowTableExists(ctx, p.pool) {
rows, qerr := p.pool.Query(ctx, `
SELECT prefix::text FROM asn_prefix_cache_row WHERE asn = $1 ORDER BY prefix`, asn)
if qerr != nil {
return nil, false, qerr
}
defer rows.Close()
for rows.Next() {
var s string
if err := rows.Scan(&s); err != nil {
continue
}
prefixes = append(prefixes, s)
}
} else {
var raw []byte
if err := p.pool.QueryRow(ctx, `SELECT prefixes_json FROM asn_prefix_cache WHERE asn = $1`, asn).Scan(&raw); err == nil && len(raw) > 0 {
_ = json.Unmarshal(raw, &prefixes)
}
}
return &store.ASNPrefixCacheEntry{
ASN: asn,
@@ -38,18 +63,40 @@ func (p *Postgres) GetASNPrefixCache(asn int64) (*store.ASNPrefixCacheEntry, boo
}
func (p *Postgres) SetASNPrefixCache(asn int64, holder string, prefixes []string) error {
raw, err := json.Marshal(prefixes)
ctx := context.Background()
tx, err := p.pool.Begin(ctx)
if err != nil {
return err
}
ctx := context.Background()
_, err = p.pool.Exec(ctx, `
INSERT INTO asn_prefix_cache (asn, holder, prefixes_json, fetched_at)
VALUES ($1, $2, $3::jsonb, now())
defer func() { _ = tx.Rollback(ctx) }()
_, err = tx.Exec(ctx, `
INSERT INTO asn_prefix_cache (asn, holder, fetched_at)
VALUES ($1, $2, now())
ON CONFLICT (asn) DO UPDATE SET
holder = EXCLUDED.holder,
prefixes_json = EXCLUDED.prefixes_json,
fetched_at = EXCLUDED.fetched_at`,
asn, holder, string(raw))
return err
fetched_at = EXCLUDED.fetched_at`, asn, holder)
if err != nil {
return err
}
if asnCacheRowTableExists(ctx, tx) {
if _, err := tx.Exec(ctx, `DELETE FROM asn_prefix_cache_row WHERE asn = $1`, asn); err != nil {
return err
}
for _, pfx := range prefixes {
if _, err := tx.Exec(ctx, `
INSERT INTO asn_prefix_cache_row (asn, prefix) VALUES ($1, $2::cidr)`, asn, pfx); err != nil {
return err
}
}
} else {
raw, err := json.Marshal(prefixes)
if err != nil {
return err
}
if _, err := tx.Exec(ctx, `
UPDATE asn_prefix_cache SET prefixes_json = $2::jsonb WHERE asn = $1`, asn, string(raw)); err != nil {
return err
}
}
return tx.Commit(ctx)
}
+8
View File
@@ -0,0 +1,8 @@
package repository
import "context"
// RunPeriodicMaintenance is a no-op; retention is driven by maintenance_policy rows (UI-configured).
func (p *Postgres) RunPeriodicMaintenance(ctx context.Context) {
_ = ctx
}
+82 -15
View File
@@ -12,16 +12,24 @@ import (
"github.com/jackc/pgx/v5"
)
func moduleSnapshotRowTableExists(ctx context.Context, q queryRower) bool {
var n int
err := q.QueryRow(ctx, `
SELECT 1 FROM information_schema.tables
WHERE table_schema = 'public' AND table_name = 'module_prefix_snapshot_row'
LIMIT 1`).Scan(&n)
return err == nil
}
func (p *Postgres) GetModulePrefixSnapshot(tenantID, moduleID string) (*store.ModulePrefixSnapshot, bool, error) {
ctx := context.Background()
var inputHash string
var collectedAt time.Time
var raw []byte
err := p.pool.QueryRow(ctx, `
SELECT input_hash, collected_at, prefixes_json
SELECT input_hash, collected_at
FROM module_prefix_snapshot
WHERE tenant_id = $1 AND module_id = $2`,
tenantID, moduleID).Scan(&inputHash, &collectedAt, &raw)
tenantID, moduleID).Scan(&inputHash, &collectedAt)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return nil, false, nil
@@ -29,9 +37,31 @@ func (p *Postgres) GetModulePrefixSnapshot(tenantID, moduleID string) (*store.Mo
return nil, false, err
}
var prefixes []store.PrefixRow
if len(raw) > 0 {
if err := json.Unmarshal(raw, &prefixes); err != nil {
return nil, false, err
if moduleSnapshotRowTableExists(ctx, p.pool) {
rows, qerr := p.pool.Query(ctx, `
SELECT prefix, community_id::text, source
FROM module_prefix_snapshot_row
WHERE tenant_id = $1::uuid AND module_id = $2::uuid
ORDER BY ord`, tenantID, moduleID)
if qerr != nil {
return nil, false, qerr
}
defer rows.Close()
for rows.Next() {
var pr store.PrefixRow
var comm *string
if err := rows.Scan(&pr.Prefix, &comm, &pr.Source); err != nil {
continue
}
pr.CommunityID = comm
prefixes = append(prefixes, pr)
}
} else {
var raw []byte
if err := p.pool.QueryRow(ctx, `
SELECT prefixes_json FROM module_prefix_snapshot
WHERE tenant_id = $1 AND module_id = $2`, tenantID, moduleID).Scan(&raw); err == nil && len(raw) > 0 {
_ = json.Unmarshal(raw, &prefixes)
}
}
return &store.ModulePrefixSnapshot{
@@ -45,20 +75,57 @@ func (p *Postgres) SetModulePrefixSnapshot(tenantID, moduleID, inputHash string,
if strings.TrimSpace(tenantID) == "" || strings.TrimSpace(moduleID) == "" || strings.TrimSpace(inputHash) == "" {
return store.ErrInvalidInput
}
raw, err := json.Marshal(prefixes)
ctx := context.Background()
tx, err := p.pool.Begin(ctx)
if err != nil {
return err
}
ctx := context.Background()
_, err = p.pool.Exec(ctx, `
INSERT INTO module_prefix_snapshot (tenant_id, module_id, input_hash, collected_at, prefixes_json)
VALUES ($1::uuid, $2::uuid, $3, now(), $4::jsonb)
defer func() { _ = tx.Rollback(ctx) }()
_, err = tx.Exec(ctx, `
INSERT INTO module_prefix_snapshot (tenant_id, module_id, input_hash, collected_at)
VALUES ($1::uuid, $2::uuid, $3, now())
ON CONFLICT (tenant_id, module_id) DO UPDATE SET
input_hash = EXCLUDED.input_hash,
collected_at = EXCLUDED.collected_at,
prefixes_json = EXCLUDED.prefixes_json`,
tenantID, moduleID, inputHash, string(raw))
return err
collected_at = EXCLUDED.collected_at`,
tenantID, moduleID, inputHash)
if err != nil {
return err
}
if moduleSnapshotRowTableExists(ctx, tx) {
if _, err := tx.Exec(ctx, `
DELETE FROM module_prefix_snapshot_row
WHERE tenant_id = $1::uuid AND module_id = $2::uuid`, tenantID, moduleID); err != nil {
return err
}
for i, pr := range prefixes {
var comm any
if pr.CommunityID != nil && strings.TrimSpace(*pr.CommunityID) != "" {
comm = strings.TrimSpace(*pr.CommunityID)
}
src := pr.Source
if strings.TrimSpace(src) == "" {
src = "render"
}
if _, err := tx.Exec(ctx, `
INSERT INTO module_prefix_snapshot_row (tenant_id, module_id, ord, prefix, community_id, source)
VALUES ($1::uuid, $2::uuid, $3, $4, $5::uuid, $6)`,
tenantID, moduleID, i, strings.TrimSpace(pr.Prefix), comm, src); err != nil {
return err
}
}
} else {
raw, err := json.Marshal(prefixes)
if err != nil {
return err
}
if _, err := tx.Exec(ctx, `
UPDATE module_prefix_snapshot SET prefixes_json = $3::jsonb
WHERE tenant_id = $1::uuid AND module_id = $2::uuid`,
tenantID, moduleID, string(raw)); err != nil {
return err
}
}
return tx.Commit(ctx)
}
func (p *Postgres) DeleteModulePrefixSnapshot(tenantID, moduleID string) error {
+77 -66
View File
@@ -687,7 +687,7 @@ func (p *Postgres) GetRevision(tenantID, revisionID string) (*store.Revision, er
if mj.PreviewFragments == nil {
mj.PreviewFragments = map[string]string{}
}
r.PreviewFragments = mj.PreviewFragments
r.PreviewFragments = loadRevisionPreview(ctx, p.pool, revisionID, mj.PreviewFragments)
r.MaterializedPrefixCount = mj.MaterializedPrefixCount
return &r, nil
}
@@ -789,12 +789,7 @@ func (p *Postgres) ListRevisionPrefixes(tenantID, revisionID string, cursor stri
if limit <= 0 {
limit = 50
}
off := 0
if cursor != "" {
if n, err := strconv.Atoi(cursor); err == nil && n >= 0 {
off = n
}
}
afterID, off, useOffset := store.ParsePrefixPageCursor(cursor)
ctx := context.Background()
var one int
if err := p.pool.QueryRow(ctx, `
@@ -805,34 +800,55 @@ func (p *Postgres) ListRevisionPrefixes(tenantID, revisionID string, cursor stri
}
return nil, "", false
}
rows, err := p.pool.Query(ctx, `
SELECT prefix::text, community_id::text, source FROM revision_materialized_prefix
WHERE revision_id=$1 ORDER BY id
LIMIT $2 OFFSET $3`, revisionID, limit+1, off)
if snapID, ok := p.revisionPrefixSnapshotID(ctx, revisionID); ok {
return p.listSnapshotPrefixes(ctx, snapID, cursor, limit)
}
var rows pgx.Rows
var err error
if useOffset {
rows, err = p.pool.Query(ctx, `
SELECT id, prefix::text, community_id::text, source FROM revision_materialized_prefix
WHERE revision_id=$1::uuid ORDER BY id
LIMIT $2 OFFSET $3`, revisionID, limit+1, off)
} else {
var afterArg any
if afterID != nil {
afterArg = *afterID
}
rows, err = p.pool.Query(ctx, `
SELECT id, prefix::text, community_id::text, source FROM revision_materialized_prefix
WHERE revision_id=$1::uuid AND ($2::bigint IS NULL OR id > $2::bigint)
ORDER BY id
LIMIT $3`, revisionID, afterArg, limit+1)
}
if err != nil {
return nil, "", false
}
defer rows.Close()
var all []store.PrefixRow
var ids []int64
for rows.Next() {
var rowID int64
var pr store.PrefixRow
var comm *string
if err := rows.Scan(&pr.Prefix, &comm, &pr.Source); err != nil {
if err := rows.Scan(&rowID, &pr.Prefix, &comm, &pr.Source); err != nil {
continue
}
pr.CommunityID = comm
ids = append(ids, rowID)
all = append(all, pr)
}
agentDebugNDJSON3214("B", "repository/postgres.go:ListRevisionPrefixes", "list_prefixes_fetched", map[string]any{
"rows": len(all), "limit": limit, "offset": off,
"rows": len(all), "limit": limit, "keyset": !useOffset,
})
more := len(all) > limit
if more {
all = all[:limit]
ids = ids[:limit]
}
next := ""
if more {
next = fmt.Sprintf("%d", off+limit)
if more && len(ids) > 0 {
next = store.FormatPrefixPageCursor(ids[len(ids)-1])
}
if len(all) == 0 {
return nil, "", false
@@ -848,26 +864,35 @@ func (p *Postgres) CreateRollbackRevision(tenantID, sourceRevisionID string) (st
ctx := context.Background()
newID := uuid.NewString()
parent := sourceRevisionID
meta, _ := json.Marshal(map[string]any{
"preview_fragments": src.PreviewFragments,
"materialized_prefix_count": src.MaterializedPrefixCount,
})
meta, err := revisionMetaWithoutPreview(src.MaterializedPrefixCount)
if err != nil {
return "", err
}
var modArg any
if strings.TrimSpace(src.ModuleID) != "" {
modArg = src.ModuleID
}
_, err = p.pool.Exec(ctx, `
INSERT INTO config_revision (id, tenant_id, module_id, content_hash, parent_revision_id, meta_json)
VALUES ($1,$2,$3,$4,$5::uuid,$6::jsonb)`,
newID, tenantID, modArg, src.ContentHash+":rollback", parent, string(meta))
tx, err := p.pool.Begin(ctx)
if err != nil {
return "", err
}
// copy materialized prefixes
_, _ = p.pool.Exec(ctx, `
INSERT INTO revision_materialized_prefix (revision_id, prefix, community_id, source, meta_json)
SELECT $1::uuid, prefix, community_id, source, meta_json FROM revision_materialized_prefix WHERE revision_id=$2::uuid`,
newID, sourceRevisionID)
defer func() { _ = tx.Rollback(ctx) }()
_, err = tx.Exec(ctx, `
INSERT INTO config_revision (id, tenant_id, module_id, content_hash, parent_revision_id, meta_json)
VALUES ($1,$2,$3,$4,$5::uuid,$6::jsonb)`,
newID, tenantID, modArg, src.ContentHash+":rollback", parent, meta)
if err != nil {
return "", err
}
if err := copyRevisionPreview(ctx, tx, newID, sourceRevisionID); err != nil {
return "", err
}
if err := p.copyRevisionPrefixSnapshotRef(ctx, tx, newID, sourceRevisionID); err != nil {
return "", err
}
if err := tx.Commit(ctx); err != nil {
return "", err
}
return newID, nil
}
@@ -884,20 +909,18 @@ func (p *Postgres) RevisionDiff(tenantID, aID, bID string) (map[string]any, erro
var unchanged int
err := p.pool.QueryRow(ctx, `
SELECT COUNT(*)::int FROM (
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$1::uuid
INTERSECT
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$2::uuid
SELECT b.prefix FROM (`+sqlRevisionPrefixes("$2")+`) b
INNER JOIN (`+sqlRevisionPrefixes("$1")+`) a ON a.prefix = b.prefix
) t`, aID, bID).Scan(&unchanged)
if err != nil {
return nil, err
}
// added: в B, нет в A; removed: в A, нет в B — без загрузки полных снапшотов в память.
rowsAdded, err := p.pool.Query(ctx, `
SELECT prefix::text FROM (
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$1::uuid
EXCEPT
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$2::uuid
) s ORDER BY 1 LIMIT $3`, bID, aID, maxRevisionDiffRows+1)
SELECT b.prefix::text FROM (`+sqlRevisionPrefixes("$2")+`) b
LEFT JOIN (`+sqlRevisionPrefixes("$1")+`) a ON a.prefix = b.prefix
WHERE a.prefix IS NULL
ORDER BY b.prefix
LIMIT $3`, aID, bID, maxRevisionDiffRows+1)
if err != nil {
return nil, err
}
@@ -916,11 +939,11 @@ func (p *Postgres) RevisionDiff(tenantID, aID, bID string) (map[string]any, erro
}
addedTruncated := len(added) >= maxRevisionDiffRows
rowsRem, err := p.pool.Query(ctx, `
SELECT prefix::text FROM (
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$1::uuid
EXCEPT
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$2::uuid
) s ORDER BY 1 LIMIT $3`, aID, bID, maxRevisionDiffRows+1)
SELECT a.prefix::text FROM (`+sqlRevisionPrefixes("$1")+`) a
LEFT JOIN (`+sqlRevisionPrefixes("$2")+`) b ON b.prefix = a.prefix
WHERE b.prefix IS NULL
ORDER BY a.prefix
LIMIT $3`, aID, bID, maxRevisionDiffRows+1)
if err != nil {
return nil, err
}
@@ -1056,10 +1079,7 @@ func (p *Postgres) CreateRenderRevision(revisionID, tenantID, moduleID string, p
if previewFragments == nil {
previewFragments = map[string]string{}
}
meta, err := json.Marshal(map[string]any{
"preview_fragments": previewFragments,
"materialized_prefix_count": len(prefixes),
})
meta, err := revisionMetaWithoutPreview(len(prefixes))
if err != nil {
return err
}
@@ -1073,32 +1093,23 @@ func (p *Postgres) CreateRenderRevision(revisionID, tenantID, moduleID string, p
if parentRevisionID != nil && strings.TrimSpace(*parentRevisionID) != "" {
parent = strings.TrimSpace(*parentRevisionID)
}
revID := strings.TrimSpace(revisionID)
_, err = tx.Exec(ctx, `
INSERT INTO config_revision (id, tenant_id, module_id, content_hash, parent_revision_id, meta_json)
VALUES ($1::uuid, $2::uuid, $3::uuid, $4, $5::uuid, $6::jsonb)`,
strings.TrimSpace(revisionID), tenantID, moduleID, strings.TrimSpace(contentHash), parent, string(meta))
revID, tenantID, moduleID, strings.TrimSpace(contentHash), parent, meta)
if err != nil {
return err
}
if len(prefixes) > 0 {
_, err = tx.CopyFrom(ctx,
pgx.Identifier{"revision_materialized_prefix"},
[]string{"revision_id", "prefix", "community_id", "source"},
pgx.CopyFromSlice(len(prefixes), func(i int) ([]any, error) {
pr := prefixes[i]
var comm any
if pr.CommunityID != nil && strings.TrimSpace(*pr.CommunityID) != "" {
comm = strings.TrimSpace(*pr.CommunityID)
}
src := pr.Source
if strings.TrimSpace(src) == "" {
src = "render"
}
return []any{strings.TrimSpace(revisionID), strings.TrimSpace(pr.Prefix), comm, src}, nil
}))
if err != nil {
return err
}
if err := insertRevisionPreview(ctx, tx, revID, previewFragments); err != nil {
return err
}
snapID, err := p.ensurePrefixSnapshot(ctx, tx, contentHash, prefixes)
if err != nil {
return err
}
if err := p.linkRevisionPrefixSnapshot(ctx, tx, revID, snapID); err != nil {
return err
}
if err := tx.Commit(ctx); err != nil {
return err
@@ -0,0 +1,304 @@
package repository
import (
"context"
"encoding/json"
"errors"
"strconv"
"strings"
"time"
"evobgp/internal/store"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
const maintenancePolicySelect = `
SELECT id, name, table_name, condition_sql, retention_period_sec, max_rows,
vacuum_strategy, schedule_cron, enabled, dry_run_enabled,
last_run_at, COALESCE(last_status, ''), COALESCE(last_error, ''),
created_at, updated_at
FROM maintenance_policy`
func scanMaintenancePolicy(row pgx.Row) (*store.MaintenancePolicy, error) {
var p store.MaintenancePolicy
var retention, maxRows *int32
var lastRun *time.Time
err := row.Scan(
&p.ID, &p.Name, &p.TableName, &p.Condition, &retention, &maxRows,
&p.VacuumStrategy, &p.Schedule, &p.Enabled, &p.DryRunEnabled,
&lastRun, &p.LastStatus, &p.LastError, &p.CreatedAt, &p.UpdatedAt,
)
if err != nil {
return nil, err
}
if retention != nil {
v := int(*retention)
p.RetentionPeriodSec = &v
}
if maxRows != nil {
v := int(*maxRows)
p.MaxRows = &v
}
if lastRun != nil {
t := lastRun.UTC()
p.LastRunAt = &t
}
return &p, nil
}
func (p *Postgres) ListMaintenancePolicies(cursor string, limit int) ([]*store.MaintenancePolicy, string, bool, error) {
if limit <= 0 {
limit = 50
}
off := 0
if cursor != "" {
if n, err := strconv.Atoi(cursor); err == nil && n >= 0 {
off = n
}
}
ctx := context.Background()
rows, err := p.pool.Query(ctx, maintenancePolicySelect+`
ORDER BY created_at DESC, id DESC
LIMIT $1 OFFSET $2`, limit+1, off)
if err != nil {
return nil, "", false, err
}
defer rows.Close()
var out []*store.MaintenancePolicy
for rows.Next() {
pol, err := scanMaintenancePolicy(rows)
if err != nil {
continue
}
out = append(out, pol)
}
more := len(out) > limit
if more {
out = out[:limit]
}
next := ""
if more {
next = strconv.Itoa(off + limit)
}
return out, next, more, rows.Err()
}
func (p *Postgres) GetMaintenancePolicy(id string) (*store.MaintenancePolicy, error) {
ctx := context.Background()
row := p.pool.QueryRow(ctx, maintenancePolicySelect+` WHERE id=$1`, id)
pol, err := scanMaintenancePolicy(row)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return nil, store.ErrNotFound
}
return nil, err
}
return pol, nil
}
func (p *Postgres) CreateMaintenancePolicy(in *store.MaintenancePolicy) (*store.MaintenancePolicy, error) {
if in == nil {
return nil, store.ErrInvalidInput
}
vacuum := in.VacuumStrategy
if vacuum == "" {
vacuum = store.VacuumStrategyNone
}
if err := store.ValidateMaintenancePolicyInput(in.Name, in.TableName, vacuum, in.Schedule); err != nil {
return nil, err
}
ctx := context.Background()
id := uuid.NewString()
now := time.Now().UTC()
condition := store.NormalizeMaintenancePolicyCondition(in.Condition)
var retention, maxRows *int32
if in.RetentionPeriodSec != nil {
v := int32(*in.RetentionPeriodSec)
retention = &v
}
if in.MaxRows != nil {
v := int32(*in.MaxRows)
maxRows = &v
}
_, err := p.pool.Exec(ctx, `
INSERT INTO maintenance_policy (
id, name, table_name, condition_sql, retention_period_sec, max_rows,
vacuum_strategy, schedule_cron, enabled, dry_run_enabled, created_at, updated_at
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$11)`,
id, strings.TrimSpace(in.Name), strings.TrimSpace(in.TableName), condition,
retention, maxRows, vacuum, strings.TrimSpace(in.Schedule),
in.Enabled, in.DryRunEnabled, now)
if err != nil {
return nil, err
}
return p.GetMaintenancePolicy(id)
}
func (p *Postgres) UpdateMaintenancePolicy(id string, patch *store.MaintenancePolicyPatch) (*store.MaintenancePolicy, error) {
if patch == nil {
return nil, store.ErrInvalidInput
}
cur, err := p.GetMaintenancePolicy(id)
if err != nil {
return nil, err
}
if patch.Name != nil {
cur.Name = strings.TrimSpace(*patch.Name)
}
if patch.TableName != nil {
cur.TableName = strings.TrimSpace(*patch.TableName)
}
if patch.Condition != nil {
cur.Condition = store.NormalizeMaintenancePolicyCondition(*patch.Condition)
}
if patch.RetentionPeriodSec != nil {
cur.RetentionPeriodSec = patch.RetentionPeriodSec
}
if patch.MaxRows != nil {
cur.MaxRows = patch.MaxRows
}
if patch.VacuumStrategy != nil {
if !store.ValidVacuumStrategy(*patch.VacuumStrategy) {
return nil, store.ErrInvalidInput
}
cur.VacuumStrategy = strings.TrimSpace(*patch.VacuumStrategy)
}
if patch.Schedule != nil {
cur.Schedule = strings.TrimSpace(*patch.Schedule)
}
if patch.Enabled != nil {
cur.Enabled = *patch.Enabled
}
if patch.DryRunEnabled != nil {
cur.DryRunEnabled = *patch.DryRunEnabled
}
if err := store.ValidateMaintenancePolicyInput(cur.Name, cur.TableName, cur.VacuumStrategy, cur.Schedule); err != nil {
return nil, err
}
var retention, maxRows *int32
if cur.RetentionPeriodSec != nil {
v := int32(*cur.RetentionPeriodSec)
retention = &v
}
if cur.MaxRows != nil {
v := int32(*cur.MaxRows)
maxRows = &v
}
ctx := context.Background()
tag, err := p.pool.Exec(ctx, `
UPDATE maintenance_policy SET
name=$2, table_name=$3, condition_sql=$4, retention_period_sec=$5, max_rows=$6,
vacuum_strategy=$7, schedule_cron=$8, enabled=$9, dry_run_enabled=$10, updated_at=now()
WHERE id=$1`,
id, cur.Name, cur.TableName, cur.Condition, retention, maxRows,
cur.VacuumStrategy, cur.Schedule, cur.Enabled, cur.DryRunEnabled)
if err != nil {
return nil, err
}
if tag.RowsAffected() == 0 {
return nil, store.ErrNotFound
}
return p.GetMaintenancePolicy(id)
}
func (p *Postgres) DeleteMaintenancePolicy(id string) error {
ctx := context.Background()
tag, err := p.pool.Exec(ctx, `DELETE FROM maintenance_policy WHERE id=$1`, id)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return store.ErrNotFound
}
return nil
}
func (p *Postgres) TouchMaintenancePolicyRun(id, status, errMsg string) error {
ctx := context.Background()
tag, err := p.pool.Exec(ctx, `
UPDATE maintenance_policy SET
last_run_at=now(), last_status=$2, last_error=NULLIF($3,''), updated_at=now()
WHERE id=$1`, id, status, errMsg)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return store.ErrNotFound
}
return nil
}
func (p *Postgres) AppendMaintenancePolicyConfigAudit(actor, policyID, action string, before, after map[string]any) error {
ctx := context.Background()
id := uuid.NewString()
var beforeJSON, afterJSON []byte
if before != nil {
beforeJSON, _ = json.Marshal(before)
}
if after != nil {
afterJSON, _ = json.Marshal(after)
}
_, err := p.pool.Exec(ctx, `
INSERT INTO maintenance_policy_config_audit
(id, policy_id, actor_prefix, action, before_json, after_json, created_at)
VALUES ($1, NULLIF($2,''), $3, $4, $5::jsonb, $6::jsonb, now())`,
id, policyID, strings.TrimSpace(actor), action,
nullJSONBytes(beforeJSON), nullJSONBytes(afterJSON))
return err
}
func nullJSONBytes(b []byte) any {
if len(b) == 0 {
return nil
}
return string(b)
}
func (p *Postgres) ListMaintenancePolicyConfigAudit(cursor string, limit int) ([]*store.MaintenancePolicyConfigAudit, string, bool, error) {
if limit <= 0 {
limit = 50
}
off := 0
if cursor != "" {
if n, err := strconv.Atoi(cursor); err == nil && n >= 0 {
off = n
}
}
ctx := context.Background()
rows, err := p.pool.Query(ctx, `
SELECT id, COALESCE(policy_id,''), actor_prefix, action,
before_json, after_json, created_at
FROM maintenance_policy_config_audit
ORDER BY created_at DESC, id DESC
LIMIT $1 OFFSET $2`, limit+1, off)
if err != nil {
return nil, "", false, err
}
defer rows.Close()
var out []*store.MaintenancePolicyConfigAudit
for rows.Next() {
var r store.MaintenancePolicyConfigAudit
var beforeRaw, afterRaw []byte
if err := rows.Scan(&r.ID, &r.PolicyID, &r.ActorPrefix, &r.Action, &beforeRaw, &afterRaw, &r.CreatedAt); err != nil {
continue
}
if len(beforeRaw) > 0 {
_ = json.Unmarshal(beforeRaw, &r.Before)
}
if len(afterRaw) > 0 {
_ = json.Unmarshal(afterRaw, &r.After)
}
out = append(out, &r)
}
more := len(out) > limit
if more {
out = out[:limit]
}
next := ""
if more {
next = strconv.Itoa(off + limit)
}
return out, next, more, rows.Err()
}
@@ -0,0 +1,186 @@
package repository
import (
"context"
"errors"
"strings"
"evobgp/internal/store"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
func prefixSnapshotTableExists(ctx context.Context, q queryRower) bool {
var n int
err := q.QueryRow(ctx, `
SELECT 1 FROM information_schema.tables
WHERE table_schema = 'public' AND table_name = 'prefix_snapshot'
LIMIT 1`).Scan(&n)
return err == nil
}
func normalizeSnapshotHash(contentHash string) string {
h := strings.TrimSpace(contentHash)
if strings.HasPrefix(h, "sha256:") {
h = strings.TrimPrefix(h, "sha256:")
}
if len(h) > 64 {
h = h[:64]
}
if len(h) < 64 {
h = h + strings.Repeat("0", 64-len(h))
}
return h
}
func (p *Postgres) revisionPrefixSnapshotID(ctx context.Context, revisionID string) (string, bool) {
if !prefixSnapshotTableExists(ctx, p.pool) {
return "", false
}
var snap *string
err := p.pool.QueryRow(ctx, `
SELECT prefix_snapshot_id::text FROM config_revision
WHERE id = $1::uuid AND prefix_snapshot_id IS NOT NULL`, revisionID).Scan(&snap)
if err != nil || snap == nil || strings.TrimSpace(*snap) == "" {
return "", false
}
return *snap, true
}
func (p *Postgres) ensurePrefixSnapshot(ctx context.Context, db execQuerier, contentHash string, prefixes []store.PrefixRow) (string, error) {
if !prefixSnapshotTableExists(ctx, db) {
return "", nil
}
hash := normalizeSnapshotHash(contentHash)
var existing string
err := db.QueryRow(ctx, `SELECT id::text FROM prefix_snapshot WHERE content_hash = $1`, hash).Scan(&existing)
if err == nil && existing != "" {
return existing, nil
}
if err != nil && !errors.Is(err, pgx.ErrNoRows) {
return "", err
}
snapID := uuid.NewString()
if _, err := db.Exec(ctx, `
INSERT INTO prefix_snapshot (id, content_hash) VALUES ($1::uuid, $2)
ON CONFLICT (content_hash) DO NOTHING`, snapID, hash); err != nil {
return "", err
}
if err := db.QueryRow(ctx, `SELECT id::text FROM prefix_snapshot WHERE content_hash = $1`, hash).Scan(&snapID); err != nil {
return "", err
}
var rowCount int
_ = db.QueryRow(ctx, `SELECT COUNT(*)::int FROM prefix_snapshot_row WHERE snapshot_id = $1::uuid`, snapID).Scan(&rowCount)
if rowCount > 0 {
return snapID, nil
}
for i, pr := range prefixes {
var comm any
if pr.CommunityID != nil && strings.TrimSpace(*pr.CommunityID) != "" {
comm = strings.TrimSpace(*pr.CommunityID)
}
src := pr.Source
if strings.TrimSpace(src) == "" {
src = "render"
}
if _, err := db.Exec(ctx, `
INSERT INTO prefix_snapshot_row (snapshot_id, ord, prefix, community_id, source)
VALUES ($1::uuid, $2, $3, $4::uuid, $5)`,
snapID, i, strings.TrimSpace(pr.Prefix), comm, src); err != nil {
return "", err
}
}
return snapID, nil
}
func (p *Postgres) listSnapshotPrefixes(ctx context.Context, snapshotID, cursor string, limit int) ([]store.PrefixRow, string, bool) {
afterOrd, off, useOffset := store.ParsePrefixPageCursor(cursor)
var rows pgx.Rows
var err error
if useOffset {
rows, err = p.pool.Query(ctx, `
SELECT ord, prefix::text, community_id::text, source
FROM prefix_snapshot_row
WHERE snapshot_id = $1::uuid
ORDER BY ord
LIMIT $2 OFFSET $3`, snapshotID, limit+1, off)
} else {
var afterArg any
if afterOrd != nil {
afterArg = int(*afterOrd)
}
rows, err = p.pool.Query(ctx, `
SELECT ord, prefix::text, community_id::text, source
FROM prefix_snapshot_row
WHERE snapshot_id = $1::uuid AND ($2::int IS NULL OR ord > $2::int)
ORDER BY ord
LIMIT $3`, snapshotID, afterArg, limit+1)
}
if err != nil {
return nil, "", false
}
defer rows.Close()
var all []store.PrefixRow
var ords []int64
for rows.Next() {
var ord int
var pr store.PrefixRow
var comm *string
if err := rows.Scan(&ord, &pr.Prefix, &comm, &pr.Source); err != nil {
continue
}
pr.CommunityID = comm
ords = append(ords, int64(ord))
all = append(all, pr)
}
more := len(all) > limit
if more {
all = all[:limit]
ords = ords[:limit]
}
next := ""
if more && len(ords) > 0 {
next = store.FormatPrefixPageCursor(ords[len(ords)-1])
}
if len(all) == 0 {
return nil, "", false
}
return all, next, more
}
func (p *Postgres) linkRevisionPrefixSnapshot(ctx context.Context, db execQuerier, revisionID, snapshotID string) error {
if snapshotID == "" || !prefixSnapshotTableExists(ctx, db) {
return nil
}
_, err := db.Exec(ctx, `
UPDATE config_revision SET prefix_snapshot_id = $2::uuid WHERE id = $1::uuid`,
revisionID, snapshotID)
return err
}
func (p *Postgres) copyRevisionPrefixSnapshotRef(ctx context.Context, db execQuerier, dstRevisionID, srcRevisionID string) error {
if !prefixSnapshotTableExists(ctx, db) {
return nil
}
_, err := db.Exec(ctx, `
UPDATE config_revision dst
SET prefix_snapshot_id = src.prefix_snapshot_id
FROM config_revision src
WHERE dst.id = $1::uuid AND src.id = $2::uuid AND src.prefix_snapshot_id IS NOT NULL`,
dstRevisionID, srcRevisionID)
return err
}
func sqlRevisionPrefixes(revParam string) string {
return `SELECT psr.prefix FROM config_revision cr
JOIN prefix_snapshot_row psr ON psr.snapshot_id = cr.prefix_snapshot_id
WHERE cr.id = ` + revParam + `::uuid AND cr.prefix_snapshot_id IS NOT NULL
UNION ALL
SELECT rmp.prefix FROM revision_materialized_prefix rmp
WHERE rmp.revision_id = ` + revParam + `::uuid
AND NOT EXISTS (
SELECT 1 FROM config_revision cr2
WHERE cr2.id = ` + revParam + `::uuid AND cr2.prefix_snapshot_id IS NOT NULL
)`
}
@@ -0,0 +1,90 @@
package repository
import (
"context"
"encoding/json"
"errors"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
)
func revisionPreviewTableExists(ctx context.Context, q queryRower) bool {
var n int
err := q.QueryRow(ctx, `
SELECT 1 FROM information_schema.tables
WHERE table_schema = 'public' AND table_name = 'config_revision_preview'
LIMIT 1`).Scan(&n)
return err == nil
}
type queryRower interface {
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
}
func loadRevisionPreview(ctx context.Context, q queryRower, revisionID string, metaPreview map[string]string) map[string]string {
if revisionPreviewTableExists(ctx, q) {
var raw []byte
err := q.QueryRow(ctx, `
SELECT fragments FROM config_revision_preview WHERE revision_id = $1::uuid`,
revisionID).Scan(&raw)
if err == nil {
out := map[string]string{}
_ = json.Unmarshal(raw, &out)
if out == nil {
out = map[string]string{}
}
return out
}
if !errors.Is(err, pgx.ErrNoRows) {
return metaPreview
}
}
if metaPreview == nil {
return map[string]string{}
}
return metaPreview
}
type execQuerier interface {
queryRower
Exec(ctx context.Context, sql string, arguments ...any) (pgconn.CommandTag, error)
}
func insertRevisionPreview(ctx context.Context, db execQuerier, revisionID string, preview map[string]string) error {
if !revisionPreviewTableExists(ctx, db) {
return nil
}
if preview == nil {
preview = map[string]string{}
}
raw, err := json.Marshal(preview)
if err != nil {
return err
}
_, err = db.Exec(ctx, `
INSERT INTO config_revision_preview (revision_id, fragments)
VALUES ($1::uuid, $2::jsonb)
ON CONFLICT (revision_id) DO UPDATE SET fragments = EXCLUDED.fragments`,
revisionID, string(raw))
return err
}
func copyRevisionPreview(ctx context.Context, db execQuerier, dstRevisionID, srcRevisionID string) error {
if !revisionPreviewTableExists(ctx, db) {
return nil
}
_, err := db.Exec(ctx, `
INSERT INTO config_revision_preview (revision_id, fragments)
SELECT $1::uuid, fragments FROM config_revision_preview WHERE revision_id = $2::uuid
ON CONFLICT (revision_id) DO UPDATE SET fragments = EXCLUDED.fragments`,
dstRevisionID, srcRevisionID)
return err
}
func revisionMetaWithoutPreview(materializedPrefixCount int) (string, error) {
raw, err := json.Marshal(map[string]any{
"materialized_prefix_count": materializedPrefixCount,
})
return string(raw), err
}
+30 -15
View File
@@ -26,9 +26,8 @@ func (p *Postgres) seedDemo(ctx context.Context) error {
p1 := uuid.NewString()
p2 := uuid.NewString()
preview := map[string]any{
"preview_fragments": map[string]string{
"bird.conf": `# EvoBGP demo bundle
previewFrags := map[string]string{
"bird.conf": `# EvoBGP demo bundle
router id 192.0.2.1;
protocol device {
@@ -39,15 +38,12 @@ protocol direct {
ipv6;
}
`,
"bird.d/evobgp_demo.conf": "# static demo fragment\n",
},
"materialized_prefix_count": 128,
"bird.d/evobgp_demo.conf": "# static demo fragment\n",
}
previewB, _ := json.Marshal(preview)
parentMeta, _ := json.Marshal(map[string]any{
"preview_fragments": map[string]string{"bird.conf": "# parent revision\n"},
"materialized_prefix_count": 0,
})
previewMeta, _ := json.Marshal(map[string]any{"materialized_prefix_count": 128})
parentMeta, _ := json.Marshal(map[string]any{"materialized_prefix_count": 0})
parentPreview, _ := json.Marshal(map[string]string{"bird.conf": "# parent revision\n"})
previewFragsB, _ := json.Marshal(previewFrags)
tx, err := p.pool.Begin(ctx)
if err != nil {
@@ -77,13 +73,32 @@ protocol direct {
return err
}
if _, err := tx.Exec(ctx, `
INSERT INTO config_revision (id, tenant_id, content_hash, parent_revision_id, meta_json)
VALUES ($1,$2,'sha256:demo-rev-1',$3::uuid,$4::jsonb)`, rid, tid, parent, string(previewB)); err != nil {
INSERT INTO config_revision_preview (revision_id, fragments) VALUES ($1::uuid, $2::jsonb)`, parent, string(parentPreview)); err != nil {
return err
}
if _, err := tx.Exec(ctx, `
INSERT INTO revision_materialized_prefix (revision_id, prefix, community_id, source)
VALUES ($1::uuid,'203.0.113.0/24',$2::uuid,'demo'), ($1::uuid,'2001:db8::/32',$2::uuid,'demo')`, rid, cid); err != nil {
INSERT INTO config_revision (id, tenant_id, content_hash, parent_revision_id, meta_json)
VALUES ($1,$2,'sha256:demo-rev-1',$3::uuid,$4::jsonb)`, rid, tid, parent, string(previewMeta)); err != nil {
return err
}
if _, err := tx.Exec(ctx, `
INSERT INTO config_revision_preview (revision_id, fragments) VALUES ($1::uuid, $2::jsonb)`, rid, string(previewFragsB)); err != nil {
return err
}
snapID := uuid.NewString()
demoHash := normalizeSnapshotHash("sha256:demo-rev-1")
if _, err := tx.Exec(ctx, `
INSERT INTO prefix_snapshot (id, content_hash) VALUES ($1::uuid, $2)`, snapID, demoHash); err != nil {
return err
}
if _, err := tx.Exec(ctx, `
UPDATE config_revision SET prefix_snapshot_id = $2::uuid WHERE id = $1::uuid`, rid, snapID); err != nil {
return err
}
if _, err := tx.Exec(ctx, `
INSERT INTO prefix_snapshot_row (snapshot_id, ord, prefix, community_id, source)
VALUES ($1::uuid, 0, '203.0.113.0/24', $2::uuid, 'demo'),
($1::uuid, 1, '2001:db8::/32', $2::uuid, 'demo')`, snapID, cid); err != nil {
return err
}
if _, err := tx.Exec(ctx, `
+16 -3
View File
@@ -112,6 +112,19 @@ type Backend interface {
// Ping verifies backend connectivity (no-op for in-memory).
Ping(ctx context.Context) error
// RunPeriodicMaintenance prunes stale DB rows (no-op for in-memory).
RunPeriodicMaintenance(ctx context.Context)
// Maintenance policies (instance-scoped PostgreSQL maintenance configuration).
ListMaintenancePolicies(cursor string, limit int) ([]*MaintenancePolicy, string, bool, error)
GetMaintenancePolicy(id string) (*MaintenancePolicy, error)
CreateMaintenancePolicy(in *MaintenancePolicy) (*MaintenancePolicy, error)
UpdateMaintenancePolicy(id string, patch *MaintenancePolicyPatch) (*MaintenancePolicy, error)
DeleteMaintenancePolicy(id string) error
TouchMaintenancePolicyRun(id, status, errMsg string) error
AppendMaintenancePolicyConfigAudit(actor, policyID, action string, before, after map[string]any) error
ListMaintenancePolicyConfigAudit(cursor string, limit int) ([]*MaintenancePolicyConfigAudit, string, bool, error)
}
// ASNPrefixCacheEntry is a cached RIPEstat response for one ASN.
@@ -311,7 +324,7 @@ type SpeakerPatch struct {
// PrefixRow is one materialized prefix for GET /revisions/.../prefixes.
type PrefixRow struct {
Prefix string
CommunityID *string
Source string
Prefix string `json:"prefix"`
CommunityID *string `json:"community_id,omitempty"`
Source string `json:"source,omitempty"`
}
+89
View File
@@ -0,0 +1,89 @@
package store
import (
"strings"
"time"
)
// Vacuum strategy values for maintenance_policy.vacuum_strategy.
const (
VacuumStrategyNone = "none"
VacuumStrategyVacuum = "vacuum"
VacuumStrategyAnalyze = "analyze"
VacuumStrategyVacuumAnalyze = "vacuum_analyze"
VacuumStrategyReindex = "reindex"
)
// MaintenancePolicy is an instance-scoped PostgreSQL maintenance policy (control plane DB).
type MaintenancePolicy struct {
ID string `json:"id,omitempty"`
Name string `json:"name"`
TableName string `json:"table_name"`
Condition string `json:"condition"`
RetentionPeriodSec *int `json:"retention_period_sec,omitempty"`
MaxRows *int `json:"max_rows,omitempty"`
VacuumStrategy string `json:"vacuum_strategy"`
Schedule string `json:"schedule"`
Enabled bool `json:"enabled"`
DryRunEnabled bool `json:"dry_run_enabled"`
LastRunAt *time.Time `json:"last_run_at,omitempty"`
LastStatus string `json:"last_status,omitempty"`
LastError string `json:"last_error,omitempty"`
CreatedAt time.Time `json:"created_at,omitempty"`
UpdatedAt time.Time `json:"updated_at,omitempty"`
}
// MaintenancePolicyPatch is a partial update for maintenance_policy.
type MaintenancePolicyPatch struct {
Name *string `json:"name,omitempty"`
TableName *string `json:"table_name,omitempty"`
Condition *string `json:"condition,omitempty"`
RetentionPeriodSec *int `json:"retention_period_sec,omitempty"`
MaxRows *int `json:"max_rows,omitempty"`
VacuumStrategy *string `json:"vacuum_strategy,omitempty"`
Schedule *string `json:"schedule,omitempty"`
Enabled *bool `json:"enabled,omitempty"`
DryRunEnabled *bool `json:"dry_run_enabled,omitempty"`
}
// MaintenancePolicyConfigAudit is a configuration change log entry.
type MaintenancePolicyConfigAudit struct {
ID string `json:"id"`
PolicyID string `json:"policy_id,omitempty"`
ActorPrefix string `json:"actor_prefix"`
Action string `json:"action"`
Before map[string]any `json:"before,omitempty"`
After map[string]any `json:"after,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
// ValidVacuumStrategy reports whether s is an allowed vacuum_strategy value.
func ValidVacuumStrategy(s string) bool {
switch strings.TrimSpace(s) {
case VacuumStrategyNone, VacuumStrategyVacuum, VacuumStrategyAnalyze,
VacuumStrategyVacuumAnalyze, VacuumStrategyReindex:
return true
default:
return false
}
}
// NormalizeMaintenancePolicyCondition returns a safe default WHERE fragment.
func NormalizeMaintenancePolicyCondition(condition string) string {
c := strings.TrimSpace(condition)
if c == "" {
return "true"
}
return c
}
// ValidateMaintenancePolicyInput checks required fields for create/update payloads.
func ValidateMaintenancePolicyInput(name, tableName, vacuumStrategy, schedule string) error {
if strings.TrimSpace(name) == "" || strings.TrimSpace(tableName) == "" || strings.TrimSpace(schedule) == "" {
return ErrInvalidInput
}
if !ValidVacuumStrategy(vacuumStrategy) {
return ErrInvalidInput
}
return nil
}
+37 -28
View File
@@ -33,17 +33,19 @@ type Memory struct {
peers map[string]*BGPPeer
dohProfiles map[string]*DohProfile
communities map[string]*Community
cdnSources map[string]*CDNSource
asEntries map[string]*ASEntry
domainEnt map[string]*DomainEntry
ipRanges map[string]*IPRangeEntry
settings map[string]map[string]any // tenantID -> key -> JSON-compatible value
revPrefixes map[string][]PrefixRow
moduleSnapshots map[string]*moduleSnapshotRec
asnPrefixCache map[int64]*ASNPrefixCacheEntry
apiKeys map[string]*apiKeyRec
dohProfiles map[string]*DohProfile
communities map[string]*Community
cdnSources map[string]*CDNSource
asEntries map[string]*ASEntry
domainEnt map[string]*DomainEntry
ipRanges map[string]*IPRangeEntry
settings map[string]map[string]any // tenantID -> key -> JSON-compatible value
revPrefixes map[string][]PrefixRow
moduleSnapshots map[string]*moduleSnapshotRec
asnPrefixCache map[int64]*ASNPrefixCacheEntry
apiKeys map[string]*apiKeyRec
maintenancePolicies map[string]*MaintenancePolicy
maintConfigAudit []*MaintenancePolicyConfigAudit
// DemoIDs valid after SeedDemo()
demoTenantID string
@@ -123,23 +125,25 @@ type Speaker struct {
func NewMemory() *Memory {
return &Memory{
tenants: make(map[string]*Tenant),
modules: make(map[string]*Module),
revisions: make(map[string]*Revision),
speakers: make(map[string]*Speaker),
publishedRevision: make(map[string]publishedInfo),
peers: make(map[string]*BGPPeer),
dohProfiles: make(map[string]*DohProfile),
communities: make(map[string]*Community),
cdnSources: make(map[string]*CDNSource),
asEntries: make(map[string]*ASEntry),
domainEnt: make(map[string]*DomainEntry),
ipRanges: make(map[string]*IPRangeEntry),
settings: make(map[string]map[string]any),
revPrefixes: make(map[string][]PrefixRow),
moduleSnapshots: make(map[string]*moduleSnapshotRec),
asnPrefixCache: make(map[int64]*ASNPrefixCacheEntry),
apiKeys: make(map[string]*apiKeyRec),
tenants: make(map[string]*Tenant),
modules: make(map[string]*Module),
revisions: make(map[string]*Revision),
speakers: make(map[string]*Speaker),
publishedRevision: make(map[string]publishedInfo),
peers: make(map[string]*BGPPeer),
dohProfiles: make(map[string]*DohProfile),
communities: make(map[string]*Community),
cdnSources: make(map[string]*CDNSource),
asEntries: make(map[string]*ASEntry),
domainEnt: make(map[string]*DomainEntry),
ipRanges: make(map[string]*IPRangeEntry),
settings: make(map[string]map[string]any),
revPrefixes: make(map[string][]PrefixRow),
moduleSnapshots: make(map[string]*moduleSnapshotRec),
asnPrefixCache: make(map[int64]*ASNPrefixCacheEntry),
apiKeys: make(map[string]*apiKeyRec),
maintenancePolicies: make(map[string]*MaintenancePolicy),
maintConfigAudit: nil,
}
}
@@ -305,6 +309,11 @@ func (m *Memory) Ping(ctx context.Context) error {
return nil
}
// RunPeriodicMaintenance is a no-op for the in-memory backend.
func (m *Memory) RunPeriodicMaintenance(ctx context.Context) {
_ = ctx
}
// ListTenantIDs returns tenant ids sorted lexicographically.
func (m *Memory) ListTenantIDs() ([]string, error) {
m.mu.RLock()
+23 -19
View File
@@ -1,8 +1,6 @@
package store
import (
"fmt"
"strconv"
"strings"
"time"
@@ -795,31 +793,37 @@ func (m *Memory) ListRevisionPrefixes(tenantID, revisionID string, cursor string
if limit <= 0 {
limit = 50
}
afterID, off, useOffset := ParsePrefixPageCursor(cursor)
m.mu.RLock()
defer m.mu.RUnlock()
if _, err := m.getRevisionLocked(tenantID, revisionID); err != nil {
return nil, "", false
}
all := m.revPrefixes[revisionID]
off := 0
if cursor != "" {
if n, err := strconv.Atoi(cursor); err == nil && n >= 0 {
off = n
}
allRows := m.revPrefixes[revisionID]
start := 0
if useOffset {
start = off
} else if afterID != nil {
start = int(*afterID) + 1
}
end := off + limit
next := ""
more := false
if end > len(all) {
end = len(all)
} else {
more = true
next = fmt.Sprintf("%d", end)
}
if off >= len(all) {
if start > len(allRows) {
return nil, "", false
}
return all[off:end], next, more
end := start + limit
next := ""
more := false
if end > len(allRows) {
end = len(allRows)
} else {
more = true
next = FormatPrefixPageCursor(int64(end - 1))
}
if start >= end {
return nil, "", false
}
out := make([]PrefixRow, end-start)
copy(out, allRows[start:end])
return out, next, more
}
func (m *Memory) ListGlobalSettings(tenantID string) (map[string]any, error) {
+244
View File
@@ -0,0 +1,244 @@
package store
import (
"sort"
"strconv"
"strings"
"time"
"github.com/google/uuid"
)
func (m *Memory) ListMaintenancePolicies(cursor string, limit int) ([]*MaintenancePolicy, string, bool, error) {
if limit <= 0 {
limit = 50
}
m.mu.RLock()
defer m.mu.RUnlock()
all := make([]*MaintenancePolicy, 0, len(m.maintenancePolicies))
for _, p := range m.maintenancePolicies {
all = append(all, p)
}
sort.Slice(all, func(i, j int) bool {
if all[i].CreatedAt.Equal(all[j].CreatedAt) {
return all[i].ID > all[j].ID
}
return all[i].CreatedAt.After(all[j].CreatedAt)
})
off := parseMaintCursor(cursor)
end := off + limit
next := ""
hasMore := false
if end > len(all) {
end = len(all)
} else if end < len(all) {
hasMore = true
next = formatMaintCursor(end)
}
if off >= len(all) {
return nil, "", false, nil
}
out := make([]*MaintenancePolicy, end-off)
copy(out, all[off:end])
return out, next, hasMore, nil
}
func (m *Memory) GetMaintenancePolicy(id string) (*MaintenancePolicy, error) {
m.mu.RLock()
defer m.mu.RUnlock()
p, ok := m.maintenancePolicies[id]
if !ok {
return nil, ErrNotFound
}
return cloneMaintenancePolicy(p), nil
}
func (m *Memory) CreateMaintenancePolicy(in *MaintenancePolicy) (*MaintenancePolicy, error) {
if in == nil {
return nil, ErrInvalidInput
}
vacuum := in.VacuumStrategy
if vacuum == "" {
vacuum = VacuumStrategyNone
}
if err := ValidateMaintenancePolicyInput(in.Name, in.TableName, vacuum, in.Schedule); err != nil {
return nil, err
}
m.mu.Lock()
defer m.mu.Unlock()
now := time.Now().UTC()
id := uuid.NewString()
p := &MaintenancePolicy{
ID: id,
Name: strings.TrimSpace(in.Name),
TableName: strings.TrimSpace(in.TableName),
Condition: NormalizeMaintenancePolicyCondition(in.Condition),
RetentionPeriodSec: in.RetentionPeriodSec,
MaxRows: in.MaxRows,
VacuumStrategy: vacuum,
Schedule: strings.TrimSpace(in.Schedule),
Enabled: in.Enabled,
DryRunEnabled: in.DryRunEnabled,
CreatedAt: now,
UpdatedAt: now,
}
m.maintenancePolicies[id] = p
return cloneMaintenancePolicy(p), nil
}
func (m *Memory) UpdateMaintenancePolicy(id string, patch *MaintenancePolicyPatch) (*MaintenancePolicy, error) {
if patch == nil {
return nil, ErrInvalidInput
}
m.mu.Lock()
defer m.mu.Unlock()
p, ok := m.maintenancePolicies[id]
if !ok {
return nil, ErrNotFound
}
if patch.Name != nil {
p.Name = strings.TrimSpace(*patch.Name)
}
if patch.TableName != nil {
p.TableName = strings.TrimSpace(*patch.TableName)
}
if patch.Condition != nil {
p.Condition = NormalizeMaintenancePolicyCondition(*patch.Condition)
}
if patch.RetentionPeriodSec != nil {
p.RetentionPeriodSec = patch.RetentionPeriodSec
}
if patch.MaxRows != nil {
p.MaxRows = patch.MaxRows
}
if patch.VacuumStrategy != nil {
if !ValidVacuumStrategy(*patch.VacuumStrategy) {
return nil, ErrInvalidInput
}
p.VacuumStrategy = strings.TrimSpace(*patch.VacuumStrategy)
}
if patch.Schedule != nil {
p.Schedule = strings.TrimSpace(*patch.Schedule)
}
if patch.Enabled != nil {
p.Enabled = *patch.Enabled
}
if patch.DryRunEnabled != nil {
p.DryRunEnabled = *patch.DryRunEnabled
}
if err := ValidateMaintenancePolicyInput(p.Name, p.TableName, p.VacuumStrategy, p.Schedule); err != nil {
return nil, err
}
p.UpdatedAt = time.Now().UTC()
return cloneMaintenancePolicy(p), nil
}
func (m *Memory) DeleteMaintenancePolicy(id string) error {
m.mu.Lock()
defer m.mu.Unlock()
if _, ok := m.maintenancePolicies[id]; !ok {
return ErrNotFound
}
delete(m.maintenancePolicies, id)
return nil
}
func (m *Memory) TouchMaintenancePolicyRun(id, status, errMsg string) error {
m.mu.Lock()
defer m.mu.Unlock()
p, ok := m.maintenancePolicies[id]
if !ok {
return ErrNotFound
}
now := time.Now().UTC()
p.LastRunAt = &now
p.LastStatus = status
p.LastError = errMsg
p.UpdatedAt = now
return nil
}
func (m *Memory) AppendMaintenancePolicyConfigAudit(actor, policyID, action string, before, after map[string]any) error {
m.mu.Lock()
defer m.mu.Unlock()
row := &MaintenancePolicyConfigAudit{
ID: uuid.NewString(),
PolicyID: policyID,
ActorPrefix: strings.TrimSpace(actor),
Action: action,
Before: before,
After: after,
CreatedAt: time.Now().UTC(),
}
m.maintConfigAudit = append(m.maintConfigAudit, row)
return nil
}
func (m *Memory) ListMaintenancePolicyConfigAudit(cursor string, limit int) ([]*MaintenancePolicyConfigAudit, string, bool, error) {
if limit <= 0 {
limit = 50
}
m.mu.RLock()
defer m.mu.RUnlock()
all := append([]*MaintenancePolicyConfigAudit(nil), m.maintConfigAudit...)
sort.Slice(all, func(i, j int) bool {
if all[i].CreatedAt.Equal(all[j].CreatedAt) {
return all[i].ID > all[j].ID
}
return all[i].CreatedAt.After(all[j].CreatedAt)
})
off := parseMaintCursor(cursor)
end := off + limit
next := ""
hasMore := false
if end > len(all) {
end = len(all)
} else if end < len(all) {
hasMore = true
next = formatMaintCursor(end)
}
if off >= len(all) {
return nil, "", false, nil
}
out := make([]*MaintenancePolicyConfigAudit, end-off)
copy(out, all[off:end])
return out, next, hasMore, nil
}
func cloneMaintenancePolicy(p *MaintenancePolicy) *MaintenancePolicy {
if p == nil {
return nil
}
cp := *p
if p.RetentionPeriodSec != nil {
v := *p.RetentionPeriodSec
cp.RetentionPeriodSec = &v
}
if p.MaxRows != nil {
v := *p.MaxRows
cp.MaxRows = &v
}
if p.LastRunAt != nil {
t := *p.LastRunAt
cp.LastRunAt = &t
}
return &cp
}
func parseMaintCursor(cursor string) int {
if cursor == "" {
return 0
}
var off int
for _, r := range cursor {
if r < '0' || r > '9' {
return 0
}
off = off*10 + int(r-'0')
}
return off
}
func formatMaintCursor(off int) string {
return strconv.Itoa(off)
}
@@ -0,0 +1,96 @@
package store
import "testing"
func TestMemoryMaintenancePolicyCRUD(t *testing.T) {
m := NewMemory()
ret := int(86400)
max := 5000
created, err := m.CreateMaintenancePolicy(&MaintenancePolicy{
Name: "audit cleanup",
TableName: "job_audit",
Condition: "status = 'succeeded'",
RetentionPeriodSec: &ret,
MaxRows: &max,
VacuumStrategy: VacuumStrategyVacuumAnalyze,
Schedule: "0 4 * * *",
Enabled: true,
DryRunEnabled: true,
})
if err != nil {
t.Fatal(err)
}
if created.ID == "" {
t.Fatal("missing id")
}
items, _, hasMore, err := m.ListMaintenancePolicies("", 10)
if err != nil {
t.Fatal(err)
}
if len(items) != 1 || hasMore {
t.Fatalf("list: len=%d hasMore=%v", len(items), hasMore)
}
got, err := m.GetMaintenancePolicy(created.ID)
if err != nil {
t.Fatal(err)
}
if got.Name != "audit cleanup" || got.VacuumStrategy != VacuumStrategyVacuumAnalyze {
t.Fatalf("get: %+v", got)
}
newName := "renamed"
disabled := false
updated, err := m.UpdateMaintenancePolicy(created.ID, &MaintenancePolicyPatch{
Name: &newName,
Enabled: &disabled,
})
if err != nil {
t.Fatal(err)
}
if updated.Name != newName || updated.Enabled {
t.Fatalf("update: %+v", updated)
}
if err := m.TouchMaintenancePolicyRun(created.ID, "succeeded", ""); err != nil {
t.Fatal(err)
}
afterTouch, err := m.GetMaintenancePolicy(created.ID)
if err != nil {
t.Fatal(err)
}
if afterTouch.LastStatus != "succeeded" || afterTouch.LastRunAt == nil {
t.Fatalf("touch: %+v", afterTouch)
}
if err := m.AppendMaintenancePolicyConfigAudit("op:test", created.ID, "update", map[string]any{"name": "old"}, map[string]any{"name": newName}); err != nil {
t.Fatal(err)
}
audit, next, hasMore, err := m.ListMaintenancePolicyConfigAudit("", 10)
if err != nil {
t.Fatal(err)
}
if len(audit) != 1 || audit[0].Action != "update" || next != "" || hasMore {
t.Fatalf("audit: %+v next=%q hasMore=%v", audit, next, hasMore)
}
if err := m.DeleteMaintenancePolicy(created.ID); err != nil {
t.Fatal(err)
}
if _, err := m.GetMaintenancePolicy(created.ID); err != ErrNotFound {
t.Fatalf("after delete: %v", err)
}
}
func TestCreateMaintenancePolicyInvalid(t *testing.T) {
m := NewMemory()
_, err := m.CreateMaintenancePolicy(&MaintenancePolicy{Name: "", TableName: "job_audit", Schedule: "0 3 * * *"})
if err != ErrInvalidInput {
t.Fatalf("want ErrInvalidInput got %v", err)
}
_, err = m.CreateMaintenancePolicy(&MaintenancePolicy{Name: "x", TableName: "job_audit", Schedule: "0 3 * * *", VacuumStrategy: "bad"})
if err != ErrInvalidInput {
t.Fatalf("want ErrInvalidInput got %v", err)
}
}
+30
View File
@@ -0,0 +1,30 @@
package store
import (
"strconv"
"strings"
)
// ParsePrefixPageCursor decodes opaque cursors for revision prefix pagination.
func ParsePrefixPageCursor(cursor string) (afterID *int64, offset int, useOffset bool) {
cursor = strings.TrimSpace(cursor)
if cursor == "" {
return nil, 0, false
}
if strings.HasPrefix(cursor, "o:") {
n, err := strconv.Atoi(strings.TrimPrefix(cursor, "o:"))
if err != nil || n < 0 {
return nil, 0, false
}
return nil, n, true
}
if n, err := strconv.ParseInt(cursor, 10, 64); err == nil && n >= 0 {
return &n, 0, false
}
return nil, 0, false
}
// FormatPrefixPageCursor encodes the keyset cursor (last row id or slice index).
func FormatPrefixPageCursor(lastID int64) string {
return strconv.FormatInt(lastID, 10)
}
@@ -0,0 +1,6 @@
DROP INDEX IF EXISTS idx_rev_mat_prefix_uniq;
DROP INDEX IF EXISTS idx_rev_mat_prefix_rev_cover;
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_rev_id
ON revision_materialized_prefix (revision_id, id);
DROP INDEX IF EXISTS idx_module_tenant_active_sort;
DROP INDEX IF EXISTS idx_rev_mat_prefix_rev_prefix;
@@ -0,0 +1,17 @@
-- Query indexes: revision diff/list and module sort (H4 + M4).
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_rev_prefix
ON revision_materialized_prefix (revision_id, prefix);
CREATE INDEX IF NOT EXISTS idx_module_tenant_active_sort
ON module (tenant_id, priority, name)
WHERE deleted_at IS NULL;
-- Covering index for prefix list pagination (PostgreSQL 11+ INCLUDE).
DROP INDEX IF EXISTS idx_rev_mat_prefix_rev_id;
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_rev_cover
ON revision_materialized_prefix (revision_id, id)
INCLUDE (prefix, community_id, source);
CREATE UNIQUE INDEX IF NOT EXISTS idx_rev_mat_prefix_uniq
ON revision_materialized_prefix (revision_id, prefix, community_id, source);
@@ -0,0 +1,6 @@
UPDATE config_revision cr
SET meta_json = cr.meta_json || jsonb_build_object('preview_fragments', COALESCE(p.fragments, '{}'::jsonb))
FROM config_revision_preview p
WHERE p.revision_id = cr.id;
DROP TABLE IF EXISTS config_revision_preview;
@@ -0,0 +1,15 @@
-- Split BIRD preview fragments out of config_revision.meta_json (H1).
CREATE TABLE config_revision_preview (
revision_id UUID PRIMARY KEY REFERENCES config_revision (id) ON DELETE CASCADE,
fragments JSONB NOT NULL DEFAULT '{}'
);
INSERT INTO config_revision_preview (revision_id, fragments)
SELECT id, COALESCE(meta_json->'preview_fragments', '{}'::jsonb)
FROM config_revision
WHERE meta_json ? 'preview_fragments';
UPDATE config_revision
SET meta_json = meta_json - 'preview_fragments'
WHERE meta_json ? 'preview_fragments';
@@ -0,0 +1,3 @@
ALTER TABLE config_revision DROP COLUMN IF EXISTS prefix_snapshot_id;
DROP TABLE IF EXISTS prefix_snapshot_row;
DROP TABLE IF EXISTS prefix_snapshot;
@@ -0,0 +1,22 @@
-- Content-addressed prefix snapshots (H2 expand).
CREATE TABLE prefix_snapshot (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
content_hash CHAR(64) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT prefix_snapshot_hash_uniq UNIQUE (content_hash)
);
CREATE TABLE prefix_snapshot_row (
snapshot_id UUID NOT NULL REFERENCES prefix_snapshot (id) ON DELETE CASCADE,
ord INTEGER NOT NULL,
prefix TEXT NOT NULL,
community_id UUID REFERENCES bgp_community (id) ON DELETE SET NULL,
source TEXT NOT NULL DEFAULT '',
PRIMARY KEY (snapshot_id, ord)
);
CREATE INDEX idx_prefix_snapshot_row_snapshot ON prefix_snapshot_row (snapshot_id);
ALTER TABLE config_revision
ADD COLUMN prefix_snapshot_id UUID REFERENCES prefix_snapshot (id) ON DELETE RESTRICT;
@@ -0,0 +1,3 @@
DELETE FROM prefix_snapshot_row;
UPDATE config_revision SET prefix_snapshot_id = NULL WHERE prefix_snapshot_id IS NOT NULL;
DELETE FROM prefix_snapshot;
@@ -0,0 +1,35 @@
-- Backfill prefix snapshots from revision_materialized_prefix.
-- prefix_snapshot_row.prefix is TEXT (revision_materialized_prefix.prefix since 000003).
ALTER TABLE prefix_snapshot_row
ALTER COLUMN prefix TYPE TEXT USING prefix::text;
WITH new_snaps AS (
INSERT INTO prefix_snapshot (id, content_hash)
SELECT gen_random_uuid(),
substr(replace(cr.id::text, '-', '') || replace(cr.id::text, '-', ''), 1, 64)
FROM config_revision cr
WHERE cr.prefix_snapshot_id IS NULL
AND EXISTS (
SELECT 1 FROM revision_materialized_prefix rmp WHERE rmp.revision_id = cr.id
)
RETURNING id, content_hash
)
UPDATE config_revision cr
SET prefix_snapshot_id = ns.id
FROM new_snaps ns
WHERE cr.prefix_snapshot_id IS NULL
AND ns.content_hash = substr(replace(cr.id::text, '-', '') || replace(cr.id::text, '-', ''), 1, 64);
INSERT INTO prefix_snapshot_row (snapshot_id, ord, prefix, community_id, source)
SELECT cr.prefix_snapshot_id,
(row_number() OVER (PARTITION BY cr.id ORDER BY rmp.id) - 1)::int,
rmp.prefix,
rmp.community_id,
rmp.source
FROM config_revision cr
JOIN revision_materialized_prefix rmp ON rmp.revision_id = cr.id
WHERE cr.prefix_snapshot_id IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM prefix_snapshot_row psr WHERE psr.snapshot_id = cr.prefix_snapshot_id
);
@@ -0,0 +1 @@
COMMENT ON COLUMN config_revision.prefix_snapshot_id IS NULL;
@@ -0,0 +1,3 @@
-- H2 contract marker: new revisions use prefix_snapshot_id only (enforced in application code).
COMMENT ON COLUMN config_revision.prefix_snapshot_id IS 'Materialized prefixes; revision_materialized_prefix deprecated for new rows';
@@ -0,0 +1,16 @@
ALTER TABLE module_prefix_snapshot ADD COLUMN prefixes_json JSONB NOT NULL DEFAULT '[]';
UPDATE module_prefix_snapshot mps
SET prefixes_json = COALESCE((
SELECT jsonb_agg(
jsonb_build_object(
'prefix', psr.prefix::text,
'community_id', psr.community_id,
'source', psr.source
) ORDER BY psr.ord
)
FROM module_prefix_snapshot_row psr
WHERE psr.tenant_id = mps.tenant_id AND psr.module_id = mps.module_id
), '[]'::jsonb);
DROP TABLE IF EXISTS module_prefix_snapshot_row;
@@ -0,0 +1,46 @@
CREATE TABLE module_prefix_snapshot_row (
tenant_id UUID NOT NULL,
module_id UUID NOT NULL,
ord INTEGER NOT NULL,
prefix TEXT NOT NULL,
community_id UUID,
source TEXT NOT NULL DEFAULT '',
PRIMARY KEY (tenant_id, module_id, ord),
FOREIGN KEY (tenant_id, module_id)
REFERENCES module_prefix_snapshot (tenant_id, module_id) ON DELETE CASCADE
);
-- prefixes_json from Go json.Marshal(PrefixRow) used "Prefix"/"CommunityID"/"Source" before json tags.
INSERT INTO module_prefix_snapshot_row (tenant_id, module_id, ord, prefix, community_id, source)
SELECT tenant_id,
module_id,
(row_number() OVER (PARTITION BY tenant_id, module_id ORDER BY ordinality) - 1)::int,
prefix,
NULLIF(community_id, '')::uuid,
COALESCE(source, '')
FROM (
SELECT mps.tenant_id,
mps.module_id,
t.ordinality,
COALESCE(
NULLIF(trim(t.elem->>'prefix'), ''),
NULLIF(trim(t.elem->>'Prefix'), '')
) AS prefix,
COALESCE(
NULLIF(trim(t.elem->>'community_id'), ''),
NULLIF(trim(t.elem->>'CommunityID'), '')
) AS community_id,
COALESCE(
NULLIF(trim(t.elem->>'source'), ''),
NULLIF(trim(t.elem->>'Source'), ''),
''
) AS source
FROM module_prefix_snapshot mps
CROSS JOIN LATERAL jsonb_array_elements(mps.prefixes_json) WITH ORDINALITY AS t(elem, ordinality)
WHERE jsonb_typeof(mps.prefixes_json) = 'array'
AND jsonb_array_length(mps.prefixes_json) > 0
) parsed
WHERE parsed.prefix IS NOT NULL
AND trim(parsed.prefix) <> '';
ALTER TABLE module_prefix_snapshot DROP COLUMN prefixes_json;
@@ -0,0 +1,10 @@
ALTER TABLE asn_prefix_cache ADD COLUMN prefixes_json JSONB NOT NULL DEFAULT '[]';
UPDATE asn_prefix_cache apc
SET prefixes_json = COALESCE((
SELECT jsonb_agg(apcr.prefix::text ORDER BY apcr.prefix::text)
FROM asn_prefix_cache_row apcr
WHERE apcr.asn = apc.asn
), '[]'::jsonb);
DROP TABLE IF EXISTS asn_prefix_cache_row;
@@ -0,0 +1,14 @@
CREATE TABLE asn_prefix_cache_row (
asn BIGINT NOT NULL REFERENCES asn_prefix_cache (asn) ON DELETE CASCADE,
prefix CIDR NOT NULL,
PRIMARY KEY (asn, prefix)
);
INSERT INTO asn_prefix_cache_row (asn, prefix)
SELECT apc.asn, t.elem::cidr
FROM asn_prefix_cache apc
CROSS JOIN LATERAL jsonb_array_elements_text(apc.prefixes_json) AS t(elem)
WHERE jsonb_typeof(apc.prefixes_json) = 'array'
AND jsonb_array_length(apc.prefixes_json) > 0;
ALTER TABLE asn_prefix_cache DROP COLUMN prefixes_json;
@@ -0,0 +1 @@
DROP INDEX IF EXISTS idx_job_audit_created_brin;
@@ -0,0 +1,2 @@
CREATE INDEX IF NOT EXISTS idx_job_audit_created_brin
ON job_audit USING BRIN (created_at);
@@ -0,0 +1,18 @@
CREATE TABLE IF NOT EXISTS module_cdn_fetch_log (
id BIGSERIAL PRIMARY KEY,
module_id UUID NOT NULL REFERENCES module (id) ON DELETE CASCADE,
source_id UUID REFERENCES module_cdn_source (id) ON DELETE SET NULL,
http_status INTEGER,
bytes BIGINT,
error TEXT,
fetched_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_module_cdn_fetch_log_module ON module_cdn_fetch_log (module_id, fetched_at DESC);
ALTER TABLE revision_materialized_prefix ADD COLUMN IF NOT EXISTS meta_json JSONB NOT NULL DEFAULT '{}';
ALTER TABLE module_domain_entry ADD COLUMN IF NOT EXISTS resolve_meta JSONB NOT NULL DEFAULT '{}';
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_revision ON revision_materialized_prefix (revision_id);
CREATE INDEX IF NOT EXISTS idx_module_tenant ON module (tenant_id);
CREATE INDEX IF NOT EXISTS idx_module_prefix_snapshot_collected ON module_prefix_snapshot (collected_at);
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_value ON revision_materialized_prefix (prefix);
@@ -0,0 +1,8 @@
ALTER TABLE revision_materialized_prefix DROP COLUMN IF EXISTS meta_json;
ALTER TABLE module_domain_entry DROP COLUMN IF EXISTS resolve_meta;
DROP TABLE IF EXISTS module_cdn_fetch_log;
DROP INDEX IF EXISTS idx_rev_mat_prefix_revision;
DROP INDEX IF EXISTS idx_module_tenant;
DROP INDEX IF EXISTS idx_module_prefix_snapshot_collected;
DROP INDEX IF EXISTS idx_rev_mat_prefix_value;
@@ -0,0 +1 @@
DROP EXTENSION IF EXISTS pg_stat_statements;
@@ -0,0 +1,2 @@
-- pg_stat_statements requires shared_preload_libraries on the server; extension may fail on dev without restart.
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
@@ -0,0 +1,2 @@
DROP TABLE IF EXISTS postgres_maintenance_audit;
DROP TABLE IF EXISTS postgres_monitor_snapshot;
@@ -0,0 +1,29 @@
CREATE TABLE IF NOT EXISTS postgres_monitor_snapshot (
id TEXT PRIMARY KEY,
collected_at TIMESTAMPTZ NOT NULL,
payload_json JSONB NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_postgres_monitor_snapshot_collected
ON postgres_monitor_snapshot (collected_at DESC);
CREATE TABLE IF NOT EXISTS postgres_maintenance_audit (
id TEXT PRIMARY KEY,
tenant_id TEXT,
actor_prefix TEXT,
kind TEXT NOT NULL,
target_table TEXT,
dry_run BOOLEAN NOT NULL DEFAULT false,
status TEXT NOT NULL,
detail_json JSONB,
error_message TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
started_at TIMESTAMPTZ,
finished_at TIMESTAMPTZ,
CONSTRAINT postgres_maintenance_audit_status_chk CHECK (
status IN ('queued', 'running', 'succeeded', 'failed', 'cancelled')
)
);
CREATE INDEX IF NOT EXISTS idx_postgres_maintenance_audit_created
ON postgres_maintenance_audit (created_at DESC);
@@ -0,0 +1,7 @@
DROP INDEX IF EXISTS idx_postgres_maintenance_audit_policy;
ALTER TABLE postgres_maintenance_audit DROP COLUMN IF EXISTS policy_id;
DROP TABLE IF EXISTS maintenance_policy_config_audit;
DROP TABLE IF EXISTS maintenance_policy;
@@ -0,0 +1,47 @@
CREATE TABLE IF NOT EXISTS maintenance_policy (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
table_name TEXT NOT NULL,
condition_sql TEXT NOT NULL DEFAULT 'true',
retention_period_sec INTEGER,
max_rows INTEGER,
vacuum_strategy TEXT NOT NULL DEFAULT 'none',
schedule_cron TEXT NOT NULL,
enabled BOOLEAN NOT NULL DEFAULT true,
dry_run_enabled BOOLEAN NOT NULL DEFAULT false,
last_run_at TIMESTAMPTZ,
last_status TEXT,
last_error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT maintenance_policy_name_chk CHECK (length(trim(name)) > 0),
CONSTRAINT maintenance_policy_table_name_chk CHECK (length(trim(table_name)) > 0),
CONSTRAINT maintenance_policy_vacuum_strategy_chk CHECK (
vacuum_strategy IN ('none', 'vacuum', 'analyze', 'vacuum_analyze', 'reindex')
)
);
CREATE INDEX IF NOT EXISTS idx_maintenance_policy_enabled
ON maintenance_policy (enabled);
CREATE TABLE IF NOT EXISTS maintenance_policy_config_audit (
id TEXT PRIMARY KEY,
policy_id TEXT,
actor_prefix TEXT NOT NULL,
action TEXT NOT NULL,
before_json JSONB,
after_json JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT maintenance_policy_config_audit_action_chk CHECK (
action IN ('create', 'update', 'delete')
)
);
CREATE INDEX IF NOT EXISTS idx_maintenance_policy_config_audit_created
ON maintenance_policy_config_audit (created_at DESC);
ALTER TABLE postgres_maintenance_audit
ADD COLUMN IF NOT EXISTS policy_id TEXT REFERENCES maintenance_policy (id) ON DELETE SET NULL;
CREATE INDEX IF NOT EXISTS idx_postgres_maintenance_audit_policy
ON postgres_maintenance_audit (policy_id);
@@ -0,0 +1,3 @@
DROP INDEX IF EXISTS idx_rev_mat_prefix_uniq;
DROP INDEX IF EXISTS idx_module_tenant_active_sort;
DROP INDEX IF EXISTS idx_rev_mat_prefix_rev_prefix;
@@ -0,0 +1,9 @@
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_rev_prefix
ON revision_materialized_prefix (revision_id, prefix);
CREATE INDEX IF NOT EXISTS idx_module_tenant_active_sort
ON module (tenant_id, priority, name)
WHERE deleted_at IS NULL;
CREATE UNIQUE INDEX IF NOT EXISTS idx_rev_mat_prefix_uniq
ON revision_materialized_prefix (revision_id, prefix, community_id, source);
@@ -0,0 +1,9 @@
UPDATE config_revision
SET meta_json = json_set(
meta_json,
'$.preview_fragments',
json(COALESCE((SELECT fragments FROM config_revision_preview p WHERE p.revision_id = config_revision.id), '{}'))
)
WHERE id IN (SELECT revision_id FROM config_revision_preview);
DROP TABLE IF EXISTS config_revision_preview;
@@ -0,0 +1,13 @@
CREATE TABLE config_revision_preview (
revision_id TEXT NOT NULL PRIMARY KEY REFERENCES config_revision (id) ON DELETE CASCADE,
fragments TEXT NOT NULL DEFAULT '{}'
);
INSERT INTO config_revision_preview (revision_id, fragments)
SELECT id, json(COALESCE(json_extract(meta_json, '$.preview_fragments'), '{}'))
FROM config_revision
WHERE json_extract(meta_json, '$.preview_fragments') IS NOT NULL;
UPDATE config_revision
SET meta_json = json_remove(meta_json, '$.preview_fragments')
WHERE json_extract(meta_json, '$.preview_fragments') IS NOT NULL;
@@ -0,0 +1,3 @@
ALTER TABLE config_revision DROP COLUMN prefix_snapshot_id;
DROP TABLE IF EXISTS prefix_snapshot_row;
DROP TABLE IF EXISTS prefix_snapshot;
@@ -0,0 +1,18 @@
CREATE TABLE prefix_snapshot (
id TEXT NOT NULL PRIMARY KEY,
content_hash TEXT NOT NULL UNIQUE,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
);
CREATE TABLE prefix_snapshot_row (
snapshot_id TEXT NOT NULL REFERENCES prefix_snapshot (id) ON DELETE CASCADE,
ord INTEGER NOT NULL,
prefix TEXT NOT NULL,
community_id TEXT,
source TEXT NOT NULL DEFAULT '',
PRIMARY KEY (snapshot_id, ord)
);
CREATE INDEX idx_prefix_snapshot_row_snapshot ON prefix_snapshot_row (snapshot_id);
ALTER TABLE config_revision ADD COLUMN prefix_snapshot_id TEXT REFERENCES prefix_snapshot (id) ON DELETE RESTRICT;
@@ -0,0 +1,3 @@
DELETE FROM prefix_snapshot_row;
UPDATE config_revision SET prefix_snapshot_id = NULL;
DELETE FROM prefix_snapshot;
@@ -0,0 +1,30 @@
-- SQLite backfill: one snapshot per revision with materialized prefixes.
INSERT INTO prefix_snapshot (id, content_hash)
SELECT lower(hex(randomblob(16))),
substr(replace(cr.id, '-', '') || replace(cr.id, '-', ''), 1, 64)
FROM config_revision cr
WHERE cr.prefix_snapshot_id IS NULL
AND EXISTS (SELECT 1 FROM revision_materialized_prefix rmp WHERE rmp.revision_id = cr.id);
UPDATE config_revision
SET prefix_snapshot_id = (
SELECT ps.id FROM prefix_snapshot ps
WHERE ps.content_hash = substr(replace(config_revision.id, '-', '') || replace(config_revision.id, '-', ''), 1, 64)
)
WHERE prefix_snapshot_id IS NULL
AND EXISTS (SELECT 1 FROM revision_materialized_prefix rmp WHERE rmp.revision_id = config_revision.id);
INSERT INTO prefix_snapshot_row (snapshot_id, ord, prefix, community_id, source)
SELECT cr.prefix_snapshot_id,
(SELECT COUNT(*) FROM revision_materialized_prefix r2
WHERE r2.revision_id = cr.id AND r2.id <= rmp.id) - 1,
rmp.prefix,
rmp.community_id,
rmp.source
FROM config_revision cr
JOIN revision_materialized_prefix rmp ON rmp.revision_id = cr.id
WHERE cr.prefix_snapshot_id IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM prefix_snapshot_row psr WHERE psr.snapshot_id = cr.prefix_snapshot_id
);
@@ -0,0 +1 @@
-- H2 contract marker (application stops writing revision_materialized_prefix for new revisions).
@@ -0,0 +1,2 @@
ALTER TABLE module_prefix_snapshot ADD COLUMN prefixes_json TEXT NOT NULL DEFAULT '[]';
DROP TABLE IF EXISTS module_prefix_snapshot_row;

Some files were not shown because too many files have changed in this diff Show More