Compare commits

...
29 Commits
Author SHA1 Message Date
DenozordecandCursor 1cfd062835 chore(db): drop unused schema and redundant indexes
CI / changes (push) Successful in 8s
CI / commitlint (push) Has been skipped
CI / openapi (push) Has been skipped
CI / web (push) Has been skipped
CI / go (push) Successful in 54s
CI / bird2 (push) Successful in 14s
CI / release (push) Successful in 3m44s
Удалены мёртвые колонки, fetch_log и избыточные индексы.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:57 +07:00
DenozordecandCursor 21233bd578 chore(db): add job_audit brin index
BRIN-индекс по created_at для retention-удалений.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:57 +07:00
DenozordecandCursor 990cc739df refactor(db): normalize asn prefix cache and ttl cleanup
Строки asn_prefix_cache_row; периодический prune через evobgp-ingest.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:57 +07:00
DenozordecandCursor 3500bd4624 refactor(db): normalize module prefix snapshot rows
Строки префиксов в module_prefix_snapshot_row вместо JSONB blobs.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:56 +07:00
DenozordecandCursor 374575ec01 refactor(db): mark prefix snapshot contract migration
Маркер миграции contract-фазы H2.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:01 +07:00
DenozordecandCursor f57b430052 feat(db): backfill prefix snapshots and switch read path
Backfill миграция; чтение префиксов через snapshot; запись без дублирования rows.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:01 +07:00
DenozordecandCursor 8a9d60a5a7 feat(db): add prefix snapshot tables (expand)
Таблицы prefix_snapshot и prefix_snapshot_row; колонка prefix_snapshot_id.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:58:01 +07:00
DenozordecandCursor b963311b43 perf(db): keyset pagination for revision prefixes
Keyset по id/ord вместо OFFSET; anti-join для RevisionDiff.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:56:46 +07:00
DenozordecandCursor 50bdb8232b refactor(db): split revision preview from meta_json
Preview BIRD-фрагменты в config_revision_preview; meta_json только счётчик префиксов.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:55:52 +07:00
DenozordecandCursor ee8e24ffc6 perf(db): add revision and module query indexes
Индексы для diff/list префиксов и сортировки модулей; covering index для pagination.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:55:10 +07:00
DenozordecandCursor 44b94caacf docs(db): add postgres schema diagnostics runbook
Runbook для оценки размеров таблиц, seq scan и EXPLAIN перед/после миграций схемы.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:55:02 +07:00
DenozordecandCursor cbb4b467ad fix(agentserver): hide upstream errors from clients
CI / changes (push) Successful in 8s
CI / commitlint (push) Has been skipped
CI / openapi (push) Has been skipped
CI / web (push) Has been skipped
CI / go (push) Successful in 58s
CI / bird2 (push) Successful in 14s
CI / release (push) Successful in 3m23s
Generic 502 detail как в httpapi; добавлен docs/production-checklist.md.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:34:03 +07:00
DenozordecandCursor e65cf0d958 feat(jobs): persist job lifecycle to PostgreSQL job_audit
UpsertQueued/Running/MarkTerminal через SetPersistHooks; исправлен deadlock
fireEnqueued под Registry mutex.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:33:57 +07:00
DenozordecandCursor 4a57c91e29 perf(pipeline): parallel prefetch and lean revision GET
GetRevisionSummary без preview_fragments; boundedRepoCtx для GetRevision;
parallel CDN prefetch с ctx из ingest.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:33:53 +07:00
DenozordecandCursor 2289107911 feat(httpclient): add circuit breaker for CDN and RIPEstat
Per-host circuit breaker с retry для CDN fetch и RIPEstat; порог 5 ошибок,
cooldown 30s.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:15:32 +07:00
DenozordecandCursor 782097420d fix(httpclient): replace DefaultClient with timed clients and retry
Пакет httpclient: timeout 45s, idle pool, DoWithRetry. Scheduler и nodecli
используют retry; pipeline/asnresolve/jobs — httpclient.New вместо DefaultClient.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:14:53 +07:00
DenozordecandCursor 82382d90f2 fix(pipeline): validate CDN URLs against SSRF
HTTPS-only CDN URLs; блокировка private/loopback/metadata IP и DNS-resolve
на fetch; проверка в httpapi при create/preview/patch CDN sources.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:14:10 +07:00
DenozordecandCursor 5a16a45922 fix(httpapi): redact agent_secret from speaker list responses
Секрет agent больше не отдаётся viewer/editor в GET/list speakers;
остаётся только при создании спикера.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:13:28 +07:00
DenozordecandCursor 6a6f6cedbc feat(pipeline): use stale snapshot when upstream fetch fails
При ошибке CDN/ASN/DoH ingest использует последний снимок префиксов
(или просроченный ASN-кэш), если EVOBGP_STALE_ON_UPSTREAM_ERROR не равен 0.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:13:07 +07:00
Denozordec 9639a03bfe feat(store): add test for clearing dispatch error on successful merge
CI / changes (push) Successful in 9s
CI / commitlint (push) Has been skipped
CI / openapi (push) Has been skipped
CI / web (push) Successful in 39s
CI / go (push) Successful in 58s
CI / bird2 (push) Successful in 16s
CI / release (push) Successful in 4m18s
- Introduced a new test case in speaker_meta_test.go to verify that the LastDispatchError is cleared when a successful dispatch status is merged.
- Updated MergeSpeakerMetaJSON function in speaker_meta.go to clear LastDispatchError if LastDispatchStatus is "ok".
- Enhanced speakerDispatchError function in network-metrics.ts to handle stale dispatch errors after successful agent sync.
2026-05-21 18:10:51 +07:00
DenozordecandCursor 48c10b7436 refactor(web): trigger release for network dashboard layout fixes
CI / changes (push) Successful in 7s
CI / commitlint (push) Has been skipped
CI / openapi (push) Has been skipped
CI / web (push) Has been skipped
CI / go (push) Successful in 40s
CI / bird2 (push) Successful in 16s
CI / release (push) Successful in 3m37s
Follow-up для semantic-release: правки overview/sheet и dispatch-ошибок уже в fb108ec, заголовок с запятой в scope не парсился. Уточнена формулировка verify в releasing.md.

Semver: patch.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-21 17:56:03 +07:00
Denozordec 4db6438245 remove commitlint configuration and update release documentation to enforce single scope in commit messages
CI / openapi (push) Has been cancelled
CI / web (push) Has been cancelled
CI / changes (push) Has been cancelled
CI / go (push) Has been cancelled
CI / bird2 (push) Has been cancelled
CI / commitlint (push) Has been cancelled
CI / release (push) Has been cancelled
2026-05-21 17:54:41 +07:00
Denozordec fb108ec5ab refactor(NetworkOverviewTab, NetworkSpeakerDetailSheet, NetworkSpeakerStatusCard): improve layout and error handling
CI / changes (push) Successful in 9s
CI / commitlint (push) Has been skipped
CI / openapi (push) Has been skipped
CI / web (push) Successful in 39s
CI / go (push) Has been skipped
CI / bird2 (push) Has been skipped
CI / release (push) Successful in 19s
- Refactored the layout of NetworkOverviewTab and NetworkSpeakerDetailSheet for better structure and readability.
- Enhanced error handling in NetworkSpeakerDetailSheet by introducing new error types for dispatch and agent errors.
- Updated NetworkSpeakerStatusCard to improve the display of speaker information and status.
- Adjusted styles in various components to ensure consistent spacing and alignment.
- Modified network-metrics.ts to include new functions for formatting speaker errors, improving user feedback on dispatch issues.
2026-05-21 17:50:27 +07:00
Denozordec a1ada06a76 feat(api): add live status tracking for speakers and BGP sessions
CI / changes (push) Successful in 9s
CI / commitlint (push) Has been skipped
CI / openapi (push) Successful in 24s
CI / web (push) Successful in 29s
CI / go (push) Successful in 43s
CI / bird2 (push) Successful in 15s
CI / release (push) Successful in 3m29s
- Introduced new schemas for `SpeakerLiveStatus`, `BgpSessionLive`, and `LiveSpeakerPoll` in OpenAPI documentation to support live status queries.
- Enhanced the `/v1/speakers` endpoint to include a `live` query parameter, allowing retrieval of real-time speaker and BGP status.
- Updated the HTTP API to collect and return live status data for speakers, improving monitoring capabilities.
- Modified frontend components to display live status information, enhancing user visibility into speaker health and BGP session states.
- Added a new endpoint `/v1/bird/status` for retrieving the local BIRD status, further enriching the network monitoring features.
2026-05-21 17:36:50 +07:00
Denozordec b5ed47902c feat(api): enhance peer session tracking and error handling
CI / changes (push) Successful in 8s
CI / commitlint (push) Has been skipped
CI / openapi (push) Has been skipped
CI / web (push) Successful in 35s
CI / go (push) Successful in 52s
CI / bird2 (push) Successful in 16s
CI / release (push) Successful in 3m58s
- Added `PollError` field to `peerSessionOnSpeaker` and `liveSpeakerPoll` types to capture polling errors for speakers.
- Updated `matchPeerOnSpeakers` function to handle polling errors and adjust session state reporting.
- Modified frontend components to display polling error messages alongside session states, improving user visibility into peer connection statuses.
- Enhanced API response structure to include live speaker polling information, facilitating better monitoring of speaker health.
2026-05-21 15:40:44 +07:00
Denozordec 8a19c2a3f4 refactor(NetworkPeersCard): improve sessionBadge function formatting
CI / changes (push) Successful in 8s
CI / commitlint (push) Has been skipped
CI / openapi (push) Has been skipped
CI / web (push) Successful in 36s
CI / go (push) Has been skipped
CI / bird2 (push) Has been skipped
CI / release (push) Successful in 3m21s
- Reformatted the sessionBadge function for better readability by adjusting the parameter layout.
- No functional changes were made; this is purely a code style improvement.
2026-05-21 15:22:10 +07:00
Denozordec 1c0d78b552 feat(api): enhance peer session management and documentation
CI / changes (push) Successful in 8s
CI / commitlint (push) Has been skipped
CI / openapi (push) Successful in 27s
CI / web (push) Failing after 35s
CI / go (push) Successful in 55s
CI / bird2 (push) Successful in 15s
CI / release (push) Has been skipped
- Added new fields to the API for tracking connected speakers and session states across multiple nodes, including `connected_speaker_id`, `connected_speaker_label`, `session_on_speakers`, `established_on_speakers`, and `session_mismatch`.
- Implemented a new endpoint for retrieving bird protocol sessions, enhancing the agent server functionality.
- Updated the OpenAPI documentation to reflect the new fields and query parameters, improving clarity for API consumers.
- Modified the frontend to display connected speaker information and session states, providing better visibility into peer connections.
- Updated deployment documentation to clarify the configuration requirements for enabling IP forwarding on VPS.
2026-05-21 15:18:21 +07:00
Denozordec e0a912a693 fix(worker): ensure job success is called before enqueueing deploy speakers
CI / changes (push) Successful in 7s
CI / openapi (push) Has been skipped
CI / commitlint (push) Has been skipped
CI / web (push) Has been skipped
CI / go (push) Successful in 54s
CI / bird2 (push) Successful in 15s
CI / release (push) Successful in 3m47s
- Moved the call to j.Succeed() to occur before enqueueing deploy speakers in the runPeerReconcile and finishModuleRefreshSuccess methods, ensuring proper job success handling.
2026-05-21 13:21:33 +07:00
Denozordec 9740a34fdc feat(api): add delete speaker functionality and corresponding tests
CI / changes (push) Successful in 8s
CI / commitlint (push) Has been skipped
CI / openapi (push) Successful in 26s
CI / web (push) Successful in 34s
CI / go (push) Failing after 36s
CI / bird2 (push) Has been skipped
CI / release (push) Has been skipped
- Implemented a DELETE endpoint for removing speakers, including necessary authorization checks and response handling.
- Added a handler for the delete operation in the HTTP API.
- Created unit tests to verify the delete functionality, ensuring proper deletion and response codes.
- Updated OpenAPI documentation to reflect the new delete speaker endpoint.
2026-05-21 13:12:50 +07:00
113 changed files with 4646 additions and 422 deletions
-3
View File
@@ -1,3 +0,0 @@
{
"extends": ["@commitlint/config-conventional"]
}
+2
View File
@@ -142,6 +142,8 @@ feat(web): add module create dialog on /modules
| `.cursor/` | `chore` |
| прочее в корне | `chore` |
**Запрещено:** несколько scope через запятую (`refactor(web, httpapi): …`) — semantic-release не распознает `type`, релиз не будет (см. [docs/releasing.md](../../docs/releasing.md)).
`type` определять по **содержимому diff**, не только по пути.
## Multi-change
+2
View File
@@ -303,6 +303,8 @@ jobs:
cache-dependency-path: package-lock.json
- name: Install release tooling
run: npm ci
- name: Verify releasable commit messages
run: node scripts/commit/verify-release-commits.mjs
- name: Semantic release
run: npx semantic-release
env:
+22
View File
@@ -0,0 +1,22 @@
/** @type {import('@commitlint/types').UserConfig} */
module.exports = {
extends: ['@commitlint/config-conventional'],
plugins: [
{
rules: {
'scope-no-commas': ({ scope }) => {
if (scope && scope.includes(',')) {
return [
false,
'scope must not contain commas (semantic-release will not parse the commit type)'
];
}
return [true];
}
}
}
],
rules: {
'scope-no-commas': [2, 'always']
}
};
@@ -27,9 +27,7 @@ services:
network_mode: host
cap_add:
- NET_ADMIN
sysctls:
net.ipv4.ip_forward: "1"
net.ipv6.conf.all.forwarding: "1"
# sysctls нельзя с network_mode: host — включите ip_forward на VPS (см. docs/remote-speakers.md)
volumes:
- bird_etc:/etc/bird
- bird_run:/run/bird
+90
View File
@@ -0,0 +1,90 @@
# Диагностика схемы PostgreSQL (EvoBGP)
Runbook для оценки объёма БД и узких мест **перед** и **после** миграций оптимизации схемы. Выполнять на staging или production read-only сессией.
## Подключение
```bash
psql "$EVOBGP_DATABASE_URL"
```
## 1. Размеры таблиц и индексов
```sql
SELECT relname,
pg_size_pretty(pg_total_relation_size(relid)) AS total,
pg_size_pretty(pg_relation_size(relid)) AS heap,
pg_size_pretty(pg_indexes_size(relid)) AS indexes
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC;
```
**Ожидание:** лидеры — `revision_materialized_prefix`, `config_revision` (TOAST от preview), JSONB-кэши.
## 2. Seq scan (горячие таблицы)
```sql
SELECT schemaname, relname, seq_scan, seq_tup_read, idx_scan
FROM pg_stat_user_tables
WHERE schemaname = 'public'
ORDER BY seq_tup_read DESC;
```
Сброс статистики после деплоя: `SELECT pg_stat_reset();` (только осознанно, теряется baseline).
## 3. Неиспользуемые индексы
```sql
SELECT indexrelname, idx_scan, pg_size_pretty(pg_relation_size(indexrelid)) AS size
FROM pg_stat_user_indexes
WHERE schemaname = 'public' AND idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC;
```
## 4. Дубликаты в materialized prefixes
Перед UNIQUE `(revision_id, prefix, community_id, source)`:
```sql
SELECT revision_id, prefix, community_id, source, COUNT(*) AS n
FROM revision_materialized_prefix
GROUP BY 1, 2, 3, 4
HAVING COUNT(*) > 1
LIMIT 20;
```
## 5. Шаблон отчёта staging
| Метрика | До | После | Дата |
|---------|-----|-------|------|
| `revision_materialized_prefix` total | | | |
| `config_revision` total | | | |
| `module_prefix_snapshot` total | | | |
| `asn_prefix_cache` total | | | |
| Top seq_scan table | | | |
| Unused indexes (count) | | | |
## 6. EXPLAIN для типовых запросов
```sql
-- Список префиксов ревизии (keyset)
EXPLAIN (ANALYZE, BUFFERS)
SELECT prefix::text, community_id::text, source
FROM revision_materialized_prefix
WHERE revision_id = '<revision-uuid>'::uuid
ORDER BY id
LIMIT 51;
-- Diff added (anti-join)
EXPLAIN (ANALYZE, BUFFERS)
SELECT b.prefix::text
FROM revision_materialized_prefix b
LEFT JOIN revision_materialized_prefix a
ON a.revision_id = '<rev-a>'::uuid AND a.prefix = b.prefix
WHERE b.revision_id = '<rev-b>'::uuid
AND a.prefix IS NULL
ORDER BY b.prefix
LIMIT 5001;
```
Цель: Index Scan / Bitmap Index Scan по `(revision_id, …)`, без Seq Scan на больших таблицах.
+175
View File
@@ -755,6 +755,42 @@ components:
bgp_speaker_id:
type: ["string", "null"]
description: "`null` - политика для всех спикеров."
connected_speaker_id:
type: ["string", "null"]
description: >
Live (GET /v1/peers?live=1): спикер, на котором сессия Established; опрос CP birdc + agent /v1/agent/bird/protocols.
connected_speaker_label:
type: string
description: Человекочитаемая метка ноды из live-опроса.
session_on_speakers:
type: array
description: Состояние протокола пира на каждой опрошенной ноде.
items:
type: object
properties:
speaker_id:
type: string
label:
type: string
state:
type: string
established_on_speakers:
type: array
description: Ноды, где сессия в состоянии Established (один пир может быть на нескольких).
items:
type: object
properties:
speaker_id:
type: string
label:
type: string
state:
type: string
session_mismatch:
type: boolean
description: >
true если bgp_speaker_id задан, но на этой ноде нет Established
(сессия может быть на других нодах — это не ошибка для tenant-wide пиров).
policies_json:
type: string
description: >
@@ -809,6 +845,89 @@ components:
description: >
Расширяемый объект. Ключи agent_domain, agent_secret (только при создании),
agent_port, node_ipv4, bird_bgp_source_ipv4, bird_bgp_source_ipv6.
live:
$ref: "#/components/schemas/SpeakerLiveStatus"
description: >
При GET /v1/speakers?live=1 — runtime-статус agent и BGP-опроса на ноде.
additionalProperties: true
SpeakerLiveStatus:
type: object
description: Live runtime snapshot for one speaker (GET /v1/speakers?live=1).
properties:
label:
type: string
description: Человекочитаемая метка ноды (agent domain или CP master).
agent_ok:
type: boolean
description: true если agent /v1/agent/health успешен (master — local birdc poll).
agent_error:
type: string
agent_last_sync_at:
type: string
format: date-time
agent_last_applied_revision_id:
type: string
bgp_poll_ok:
type: boolean
description: true если birdc (CP) или GET /v1/agent/bird/protocols (replica) успешен.
bgp_poll_error:
type: string
bgp_sessions_total:
type: integer
bgp_established:
type: integer
sessions:
type: array
items:
$ref: "#/components/schemas/BgpSessionLive"
additionalProperties: true
BgpSessionLive:
type: object
properties:
name:
type: string
neighbor:
type: string
state:
type: string
additionalProperties: true
LiveSpeakerPoll:
type: object
description: Метаданные опроса одной ноды в GET /v1/peers?live=1.
properties:
speaker_id:
type: string
label:
type: string
ok:
type: boolean
session_count:
type: integer
poll_error:
type: string
additionalProperties: true
BirdLocalStatus:
type: object
description: Статус локального BIRD на хосте API (GET /v1/bird/status).
properties:
birdc_configured:
type: boolean
message:
type: string
error:
type: string
protocols_excerpt:
type: string
bgp_sessions_total:
type: integer
bgp_established:
type: integer
healthy:
type: ["boolean", "null"]
additionalProperties: true
BundleSigningPublicKey:
@@ -2156,6 +2275,13 @@ paths:
- $ref: "#/components/parameters/Cursor"
- $ref: "#/components/parameters/Limit"
- $ref: "#/components/parameters/SpeakerFilter"
- name: live
in: query
schema:
type: string
enum: ["1"]
description: >
Опрос birdc на CP и GET /v1/agent/bird/protocols на репликах; обогащает session_state и connected_speaker_*.
responses:
"200":
description: Успешно.
@@ -2173,6 +2299,12 @@ paths:
type: ["string", "null"]
has_more:
type: boolean
live_speaker_poll:
type: array
description: >
При live=1 — результат опроса каждой ноды (CP birdc + agent protocols).
items:
$ref: "#/components/schemas/LiveSpeakerPoll"
default:
$ref: "#/components/responses/DefaultProblem"
post:
@@ -2293,6 +2425,14 @@ paths:
- $ref: "#/components/parameters/TenantId"
- $ref: "#/components/parameters/Cursor"
- $ref: "#/components/parameters/Limit"
- name: live
in: query
schema:
type: string
enum: ["1"]
description: >
Live-опрос agent /v1/agent/health и BGP protocols на репликах; CP — local birdc.
Обогащает каждый item полем `live`.
responses:
"200":
description: Успешно.
@@ -2379,6 +2519,21 @@ paths:
$ref: "#/components/responses/NotFound"
default:
$ref: "#/components/responses/DefaultProblem"
delete:
tags: [Speakers]
summary: Удалить спикер
description: >
Удаляет BGP-спикер. Пиры с `bgp_speaker_id` этого спикера остаются, привязка сбрасывается (ON DELETE SET NULL).
operationId: deleteSpeaker
parameters:
- $ref: "#/components/parameters/IdempotencyKey"
responses:
"204":
description: Удалено.
"404":
$ref: "#/components/responses/NotFound"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/revisions:
get:
@@ -2612,6 +2767,26 @@ paths:
default:
$ref: "#/components/responses/DefaultProblem"
/v1/bird/status:
get:
tags: [Deploy]
summary: Статус локального BIRD на хосте API
description: >
Опрос birdc через EVOBGP_BIRDC_SOCKET на процессе API (обычно CP master).
На репликах без birdc на CP — birdc_configured=false.
operationId: getBirdStatus
parameters:
- $ref: "#/components/parameters/TenantId"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
$ref: "#/components/schemas/BirdLocalStatus"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/bird/reload:
post:
tags: [Deploy]
+26
View File
@@ -0,0 +1,26 @@
# Production checklist
Краткий чеклист перед выводом EvoBGP в production (10+ клиентов, нестабильная сеть).
## Обязательно
- `EVOBGP_SEED_DEMO=0` — отключить demo-tenant и токен `Bearer dev`.
- `EVOBGP_DEV_INSECURE` не задавать или `0` — не использовать lab-флаги в prod.
- `EVOBGP_BUNDLE_SEED_HEX` — задать стабильный hex-ключ подписи бандлов; сохранить pubkey для нод.
- PostgreSQL с TLS (`sslmode` не `disable`) при доступе вне private network.
- `EVOBGP_CORS_ORIGINS` — явный whitelist origin веб-панели.
- `EVOBGP_STALE_ON_UPSTREAM_ERROR=1` (по умолчанию) — stale snapshot при сбоях CDN/ASN/DoH.
## Рекомендуется
- `EVOBGP_JOB_MAX_CONCURRENT=16`, `EVOBGP_DB_MAX_CONNS=25`, `EVOBGP_COLLECT_CONCURRENCY=16` при росте tenants.
- `EVOBGP_NODE_DISPATCH_INSECURE_TLS=0` — только валидный TLS к agent.
- Ограничить `/metrics` сетевой политикой или reverse proxy.
- Профиль `evobgp-all` или HA API + персистентная `job_audit` (PostgreSQL).
- Мониторинг drift: `evobgp-deploy`, `last_applied_revision_id` vs published.
## Не использовать в prod
- `EVOBGP_CDN_ALLOW_PRIVATE=1` — только тесты/lab.
- Plaintext `EVOBGP_API_KEYS` без ротации (break-glass — временно).
- Ручное редактирование `evobgp_*.conf` на нодах без ревизии.
+17
View File
@@ -11,6 +11,8 @@ EvoBGP использует [Conventional Commits](https://www.conventionalcommi
| `feat!`, `fix!` или `BREAKING CHANGE:` в теле | major (1.0.0 → 2.0.0) |
| `docs`, `chore`, `test` | без релиза |
**Scope:** один идентификатор **без запятых** (`web`, `httpapi`, `api`). Заголовок `refactor(a, b): …` **не парсится** semantic-release → релиз не создаётся (commitlint на PR это тоже отклонит). Подробнее — раздел «Scope и semantic-release» ниже.
`refactor` — patch без новых функций: перестройка кода/UI при том же поведении для пользователя. По semver на одном уровне с `fix`, но семантически «мельче» `feat` (не minor).
Отдельного суффикса `1.x.y.fix` в semver нет: «fix» в Conventional Commits означает **patch** (третья цифра). Для починки пайплайна без смены продукта — `fix(ci):` или `ci:` (оба дают patch после настройки `.releaserc.json`).
@@ -62,6 +64,21 @@ API: `GET /version`, `GET /v1/version` — поля `version`, `git_sha`, `build
Web UI показывает версию из API (footer sidebar, страница «Мониторинг»).
## Scope и semantic-release
Парсер [conventional-commits-parser](https://github.com/conventional-changelog/conventional-changelog/tree/master/packages/conventional-commits-parser) (его использует semantic-release) **не понимает запятые в scope**:
| Заголовок | Парсится | Релиз |
|-----------|----------|-------|
| `refactor(web): fix layout` | да, `refactor` | patch |
| `refactor(NetworkOverviewTab, NetworkSpeakersCard): fix layout` | **нет**, `type: null` | **нет** |
Правило: **один scope** из таблицы в [.cursor/rules/conventional-commits.mdc](../.cursor/rules/conventional-commits.mdc) (`web`, `httpapi`, `api`, …).
На push в `main` job **release** запускает `scripts/commit/verify-release-commits.mjs` — в логе будут предупреждения о непарсящихся коммитах.
Если релиз «не создался», а CI зелёный: смотрите лог release — часто `No releasable commits`. Исправление: новый коммит с корректным заголовком (например `refactor(web): …`).
## CHANGELOG
Release notes — в Gitea Release; файл `CHANGELOG.md` генерируется в CI и прикрепляется как asset, **не** попадает в git history.
+15 -1
View File
@@ -47,10 +47,22 @@ Runbook для реплик **bird2 + evobgp-agent** на отдельных VPS
| Порт | Кто | Зачем |
|------|-----|-------|
| **443** | IP CP (`PANEL_IP_WHITELIST`) | HTTPS dispatch + health |
| **443** | IP CP (`PANEL_IP_WHITELIST`) | HTTPS dispatch, health, **`GET /v1/agent/bird/protocols`** (live peer sessions) |
| **179** | BGP peers | Data plane |
| **80** | ACME | Traefik → 443 |
## Подготовка VPS (перед `docker compose up`)
`bird2` — **`network_mode: host`**. Docker **не может** задать `net.ipv4.ip_forward` в таком контейнере; включите на **хосте**:
```bash
sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv6.conf.all.forwarding=1
echo 'net.ipv4.ip_forward=1' | tee /etc/sysctl.d/99-evobgp-bird.conf
echo 'net.ipv6.conf.all.forwarding=1' >> /etc/sysctl.d/99-evobgp-bird.conf
sysctl --system
```
## Безопасность (три участка)
1. **CP → реплика:** HTTPS (LE) + Traefik ipallowlist + `agent_secret`.
@@ -82,6 +94,8 @@ Tenant `/v1/settings` (`bird_bgp_source_ipv4`) — fallback для master / ес
| Симптом | Проверка |
|---------|----------|
| `sysctl net.ipv4.ip_forward not allowed in host network` | Уберите sysctls из compose (уже так в main); включите ip_forward на VPS (см. выше) |
| `no service selected` | `--profile production` или `COMPOSE_PROFILES=production` |
| Offline в UI | `GET https://AGENT_DOMAIN/v1/agent/health` с CP; LE cert; whitelist |
| dispatch error | CP logs job meta; firewall 443; `agent_secret` |
| verify-bundle fail | pubkey совпадает с CP seed; пересоберите pubkey после смены seed |
+29 -1
View File
@@ -10,9 +10,12 @@ import (
"strings"
"time"
"evobgp/internal/birdfmt"
"evobgp/internal/nodecli"
)
const upstreamErrorDetail = "upstream request failed"
// Config holds evobgp-agent serve settings.
type Config struct {
Listen string
@@ -41,6 +44,7 @@ type Server struct {
func New(cfg Config) *Server {
s := &Server{cfg: cfg, mux: http.NewServeMux()}
s.mux.HandleFunc("GET /v1/agent/health", s.handleHealth)
s.mux.HandleFunc("GET /v1/agent/bird/protocols", s.handleBirdProtocols)
s.mux.HandleFunc("POST /v1/agent/sync", s.handleSync)
return s
}
@@ -68,6 +72,30 @@ func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, body)
}
func (s *Server) handleBirdProtocols(w http.ResponseWriter, r *http.Request) {
if !s.authorize(r) {
writeProblem(w, http.StatusUnauthorized, "missing or invalid Authorization")
return
}
sock := strings.TrimSpace(s.cfg.Socket)
if sock == "" {
writeProblem(w, http.StatusServiceUnavailable, "EVOBGP_BIRDC_SOCKET not configured")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second)
defer cancel()
out, err := birdfmt.ShowProtocols(ctx, sock, strings.TrimSpace(s.cfg.BirdcBin))
if err != nil {
log.Printf("agentserver: bird protocols: %v", err)
writeProblem(w, http.StatusBadGateway, upstreamErrorDetail)
return
}
writeJSON(w, http.StatusOK, map[string]any{
"ok": true,
"sessions": birdfmt.ParseBGPSessions(out),
})
}
func (s *Server) handleSync(w http.ResponseWriter, r *http.Request) {
if !s.authorize(r) {
writeProblem(w, http.StatusUnauthorized, "missing or invalid Authorization")
@@ -100,7 +128,7 @@ func (s *Server) handleSync(w http.ResponseWriter, r *http.Request) {
})
if err != nil {
log.Printf("agentserver: sync: %v", err)
writeProblem(w, http.StatusBadGateway, err.Error())
writeProblem(w, http.StatusBadGateway, upstreamErrorDetail)
return
}
if s.cfg.OnSyncSuccess != nil {
+6 -4
View File
@@ -13,6 +13,8 @@ import (
"strconv"
"strings"
"time"
"evobgp/internal/httpclient"
)
// DefaultRIPEStatURL is the RIPEstat announced-prefixes data call (no API key).
@@ -24,7 +26,7 @@ const DefaultASOverviewURL = "https://stat.ripe.net/data/as-overview/data.json"
// AnnouncedPrefixes returns currently announced IPv4/IPv6 prefixes for the ASN (best-effort via RIPEstat).
func AnnouncedPrefixes(ctx context.Context, hc *http.Client, asn int64) ([]netip.Prefix, error) {
if hc == nil {
hc = http.DefaultClient
hc = httpclient.New(httpclient.DefaultTimeout)
}
base := strings.TrimSpace(os.Getenv("EVOBGP_RIPESTAT_ANNOUNCED_PREFIXES_URL"))
if base == "" {
@@ -38,7 +40,7 @@ func AnnouncedPrefixes(ctx context.Context, hc *http.Client, asn int64) ([]netip
req.Header.Set("Accept", "application/json")
req.Header.Set("User-Agent", "evobgp-asnresolve/1.0")
resp, err := hc.Do(req)
resp, err := httpclient.DoWithBreaker(ctx, hc, req, 3)
if err != nil {
return nil, fmt.Errorf("ripestat fetch AS%d: %w", asn, err)
}
@@ -86,7 +88,7 @@ func AnnouncedPrefixes(ctx context.Context, hc *http.Client, asn int64) ([]netip
// ASHolderName returns the holder / organization label for the ASN from RIPEstat as-overview (best-effort).
func ASHolderName(ctx context.Context, hc *http.Client, asn int64) (string, error) {
if hc == nil {
hc = http.DefaultClient
hc = httpclient.New(httpclient.DefaultTimeout)
}
base := strings.TrimSpace(os.Getenv("EVOBGP_RIPESTAT_AS_OVERVIEW_URL"))
if base == "" {
@@ -100,7 +102,7 @@ func ASHolderName(ctx context.Context, hc *http.Client, asn int64) (string, erro
req.Header.Set("Accept", "application/json")
req.Header.Set("User-Agent", "evobgp-asnresolve/1.0")
resp, err := hc.Do(req)
resp, err := httpclient.DoWithBreaker(ctx, hc, req, 3)
if err != nil {
return "", fmt.Errorf("ripestat as-overview AS%d: %w", asn, err)
}
+54
View File
@@ -0,0 +1,54 @@
package birdfmt
import (
"strings"
)
// BGPSession is one BGP protocol block from `birdc show protocols all`.
type BGPSession struct {
Name string `json:"name"`
Neighbor string `json:"neighbor,omitempty"`
State string `json:"state"`
}
// ParseBGPSessions extracts BGP protocol name, state, and neighbor (if present) from birdc output.
func ParseBGPSessions(output string) []BGPSession {
var out []BGPSession
var cur *BGPSession
for _, raw := range strings.Split(output, "\n") {
line := strings.TrimRight(raw, "\r")
trim := strings.TrimSpace(line)
if trim == "" {
cur = nil
continue
}
low := strings.ToLower(trim)
if strings.HasPrefix(low, "bird ") || strings.HasPrefix(low, "name ") || strings.HasPrefix(low, "table ") {
continue
}
if !strings.HasPrefix(line, " ") && !strings.HasPrefix(line, "\t") {
if isBGPProtocolSummaryRow(trim) {
fields := strings.Fields(trim)
state := extractBGPSessionStateLine(trim)
if state == "" && len(fields) >= 4 {
state = fields[3]
}
out = append(out, BGPSession{Name: fields[0], State: state})
cur = &out[len(out)-1]
} else {
cur = nil
}
continue
}
if cur == nil {
continue
}
for _, prefix := range []string{"Neighbor address:", "Neighbor Address:", "Neighbor:"} {
if idx := strings.Index(trim, prefix); idx >= 0 {
cur.Neighbor = strings.TrimSpace(trim[idx+len(prefix):])
break
}
}
}
return out
}
+26
View File
@@ -0,0 +1,26 @@
package birdfmt
import "testing"
func TestParseBGPSessions_neighborAndState(t *testing.T) {
sample := `
BIRD 2.14 ready.
Name Proto Table State Since Info
device1 Device --- up 10:00:00
evobgp_p_abc123 BGP master4 up 10:00:05 Established
Neighbor address: 198.51.100.2
Neighbor AS: 65001
evobgp_p_def456 BGP master4 up 10:00:06 Active
Neighbor address: 2001:db8::2
`
sessions := ParseBGPSessions(sample)
if len(sessions) != 2 {
t.Fatalf("got %d sessions want 2", len(sessions))
}
if sessions[0].Name != "evobgp_p_abc123" || sessions[0].State != "Established" || sessions[0].Neighbor != "198.51.100.2" {
t.Fatalf("session0: %+v", sessions[0])
}
if sessions[1].Neighbor != "2001:db8::2" || sessions[1].State != "Active" {
t.Fatalf("session1: %+v", sessions[1])
}
}
+16
View File
@@ -0,0 +1,16 @@
package birdfmt
import "strings"
// PeerProtocolName returns the BIRD protocol name for a control-plane peer UUID.
// Must stay in sync with pipeline peer rendering.
func PeerProtocolName(peerID string) string {
s := strings.ReplaceAll(strings.TrimSpace(peerID), "-", "")
if len(s) > 16 {
s = s[:16]
}
if s == "" {
s = "x"
}
return "evobgp_p_" + s
}
+33 -10
View File
@@ -7,6 +7,7 @@ import (
"time"
"evobgp/internal/db"
"evobgp/internal/httpclient"
"evobgp/internal/jobs"
"evobgp/internal/observability"
"evobgp/internal/repository"
@@ -17,7 +18,7 @@ import (
// NewCDNHTTPClient returns the shared HTTP client for CDN and preview fetches (PERF-02 / ERR-03).
func NewCDNHTTPClient() *http.Client {
return &http.Client{Timeout: 45 * time.Second}
return httpclient.New(httpclient.DefaultTimeout)
}
// BootstrapWorkers opens the same store.Backend and jobs.Registry as New (without HTTP or bundle keys).
@@ -55,18 +56,40 @@ func BootstrapWorkers(ctx context.Context, opts Options) (store.Backend, *jobs.R
wk.Registry = reg
if pool != nil {
audit := repository.NewJobAuditWriter(pool)
reg.SetTerminalHook(func(j *jobs.Job) {
jobMeta := func(j *jobs.Job) map[string]any {
if j == nil {
return
return nil
}
st := j.Snapshot()
status, _ := st["status"].(string)
var errMsg *string
if e, ok := st["error"].(string); ok && e != "" {
errMsg = &e
}
audit.MarkTerminal(context.Background(), j.TenantID, j.ID, status, errMsg, time.Now().UTC())
})
meta, _ := st["meta"].(map[string]any)
return meta
}
reg.SetPersistHooks(
func(j *jobs.Job) {
if j == nil {
return
}
audit.UpsertQueued(context.Background(), j.TenantID, j.ID, j.Kind, j.IdempotencyKey, j.ModuleID, jobMeta(j))
},
func(j *jobs.Job) {
if j == nil {
return
}
audit.UpsertRunning(context.Background(), j.TenantID, j.ID, j.Kind, j.IdempotencyKey, jobMeta(j))
},
func(j *jobs.Job) {
if j == nil {
return
}
st := j.Snapshot()
status, _ := st["status"].(string)
var errMsg *string
if e, ok := st["error"].(string); ok && e != "" {
errMsg = &e
}
audit.MarkTerminal(context.Background(), j.TenantID, j.ID, status, errMsg, time.Now().UTC())
},
)
}
observability.RegisterStoreBackend(backend)
return backend, reg, pool, nil
+295
View File
@@ -0,0 +1,295 @@
package httpapi
import (
"context"
"net/netip"
"os"
"strings"
"sync"
"time"
"evobgp/internal/birdfmt"
"evobgp/internal/nodedispatch"
"evobgp/internal/store"
)
const peerLiveCacheTTL = 15 * time.Second
type speakerBGPLive struct {
SpeakerID string
Label string
Sessions []birdfmt.BGPSession
Error string
}
type peerLiveCacheEntry struct {
at time.Time
views []speakerBGPLive
}
var peerLiveCache sync.Map // tenantID -> peerLiveCacheEntry
type peerSessionOnSpeaker struct {
SpeakerID string `json:"speaker_id"`
Label string `json:"label"`
State string `json:"state"`
PollError string `json:"poll_error,omitempty"`
}
type liveSpeakerPoll struct {
SpeakerID string `json:"speaker_id"`
Label string `json:"label"`
OK bool `json:"ok"`
SessionCount int `json:"session_count"`
PollError string `json:"poll_error,omitempty"`
}
func speakerDisplayLabel(sp *store.Speaker) string {
if sp == nil {
return ""
}
meta := store.ParseSpeakerMeta(sp.MetaJSON)
host := strings.TrimSpace(meta.AgentDomain)
if host == "" {
host = strings.TrimSpace(sp.Endpoint)
}
if strings.EqualFold(strings.TrimSpace(sp.Role), "master") {
if host != "" {
return "CP · " + host
}
return "CP (master)"
}
if host != "" {
return host
}
return sp.ID
}
func masterSpeakerID(speakers []*store.Speaker) string {
for _, sp := range speakers {
if sp != nil && strings.EqualFold(strings.TrimSpace(sp.Role), "master") {
return sp.ID
}
}
return ""
}
func (s *Server) collectSpeakerBGPLive(ctx context.Context, tenantID string, fresh bool) []speakerBGPLive {
if !fresh {
if v, ok := peerLiveCache.Load(tenantID); ok {
ent := v.(peerLiveCacheEntry)
if time.Since(ent.at) < peerLiveCacheTTL {
return ent.views
}
}
}
speakers := s.store.ListSpeakersForTenant(tenantID)
views := make([]speakerBGPLive, 0, len(speakers)+1)
if sock := strings.TrimSpace(os.Getenv("EVOBGP_BIRDC_SOCKET")); sock != "" {
v := speakerBGPLive{Label: "CP (local BIRD)"}
if mid := masterSpeakerID(speakers); mid != "" {
v.SpeakerID = mid
for _, sp := range speakers {
if sp != nil && sp.ID == mid {
v.Label = speakerDisplayLabel(sp)
break
}
}
}
out, err := birdfmt.ShowProtocols(ctx, sock, strings.TrimSpace(os.Getenv("EVOBGP_BIRDC_BIN")))
if err != nil {
v.Error = err.Error()
} else {
v.Sessions = birdfmt.ParseBGPSessions(out)
}
views = append(views, v)
}
opts := nodedispatch.Options{Timeout: 8 * time.Second}
type resWrap struct {
sp *store.Speaker
res nodedispatch.BirdProtocolsResult
}
ch := make(chan resWrap, len(speakers))
var wg sync.WaitGroup
for _, sp := range speakers {
if sp == nil {
continue
}
meta := store.ParseSpeakerMeta(sp.MetaJSON)
if !store.SpeakerNeedsRemoteDispatch(sp.Role, meta) {
continue
}
wg.Add(1)
go func(speaker *store.Speaker) {
defer wg.Done()
ch <- resWrap{
sp: speaker,
res: nodedispatch.FetchBirdProtocols(ctx, speaker, opts),
}
}(sp)
}
wg.Wait()
close(ch)
for rw := range ch {
views = append(views, speakerBGPLive{
SpeakerID: rw.sp.ID,
Label: speakerDisplayLabel(rw.sp),
Sessions: rw.res.Sessions,
Error: rw.res.Error,
})
}
peerLiveCache.Store(tenantID, peerLiveCacheEntry{at: time.Now(), views: views})
return views
}
func liveSpeakerPollJSON(views []speakerBGPLive) []liveSpeakerPoll {
out := make([]liveSpeakerPoll, 0, len(views))
for _, v := range views {
out = append(out, liveSpeakerPoll{
SpeakerID: v.SpeakerID,
Label: v.Label,
OK: v.Error == "",
SessionCount: len(v.Sessions),
PollError: v.Error,
})
}
return out
}
func findPeerSession(sessions []birdfmt.BGPSession, protoName string, neighbor netip.Addr, hasNeighbor bool) *birdfmt.BGPSession {
for i := range sessions {
if peerSessionMatches(sessions[i], protoName, neighbor, hasNeighbor) {
return &sessions[i]
}
}
return nil
}
func matchPeerOnSpeakers(peer *store.BGPPeer, views []speakerBGPLive) (
bestState string,
connectedID string,
connectedLabel string,
establishedOn []peerSessionOnSpeaker,
on []peerSessionOnSpeaker,
mismatch bool,
) {
if peer == nil {
return "", "", "", nil, nil, false
}
neighbor, hasNeighbor := store.ParsePeerNeighbor(peer.Neighbor)
protoName := birdfmt.PeerProtocolName(peer.ID)
for _, v := range views {
if v.Error != "" && len(v.Sessions) == 0 {
on = append(on, peerSessionOnSpeaker{
SpeakerID: v.SpeakerID,
Label: v.Label,
PollError: v.Error,
})
continue
}
sess := findPeerSession(v.Sessions, protoName, neighbor, hasNeighbor)
if sess == nil {
on = append(on, peerSessionOnSpeaker{
SpeakerID: v.SpeakerID,
Label: v.Label,
State: "absent",
})
continue
}
hit := peerSessionOnSpeaker{
SpeakerID: v.SpeakerID,
Label: v.Label,
State: sess.State,
}
on = append(on, hit)
if strings.EqualFold(strings.TrimSpace(sess.State), "Established") {
establishedOn = append(establishedOn, hit)
}
if bestState == "" || sessionStateRank(sess.State) > sessionStateRank(bestState) {
bestState = sess.State
}
}
if len(establishedOn) > 0 {
bestState = "Established"
labels := make([]string, 0, len(establishedOn))
for _, e := range establishedOn {
labels = append(labels, e.Label)
}
connectedLabel = strings.Join(labels, ", ")
if len(establishedOn) == 1 {
connectedID = establishedOn[0].SpeakerID
}
} else if len(on) == 1 && on[0].PollError == "" && on[0].State != "" {
connectedID = on[0].SpeakerID
connectedLabel = on[0].Label
}
if peer.SpeakerID != nil && strings.TrimSpace(*peer.SpeakerID) != "" && len(establishedOn) > 0 {
want := strings.TrimSpace(*peer.SpeakerID)
found := false
for _, e := range establishedOn {
if strings.EqualFold(strings.TrimSpace(e.SpeakerID), want) {
found = true
break
}
}
mismatch = !found
}
return bestState, connectedID, connectedLabel, establishedOn, on, mismatch
}
func peerSessionMatches(sess birdfmt.BGPSession, protoName string, neighbor netip.Addr, hasNeighbor bool) bool {
if strings.EqualFold(strings.TrimSpace(sess.Name), protoName) {
return true
}
if !hasNeighbor || strings.TrimSpace(sess.Neighbor) == "" {
return false
}
peerAddr, ok := store.ParsePeerNeighbor(sess.Neighbor)
if !ok {
return false
}
return peerAddr == neighbor
}
func sessionStateRank(state string) int {
switch strings.ToLower(strings.TrimSpace(state)) {
case "established":
return 100
case "openconfirm", "opensent":
return 80
case "active", "connect":
return 60
case "idle":
return 20
default:
return 10
}
}
func applyPeerLiveFields(row map[string]any, peer *store.BGPPeer, views []speakerBGPLive) {
state, connID, connLabel, establishedOn, on, mismatch := matchPeerOnSpeakers(peer, views)
row["session_on_speakers"] = on
row["established_on_speakers"] = establishedOn
row["session_conflict"] = false
row["session_mismatch"] = mismatch
if state != "" {
row["session_state"] = state
}
if connLabel != "" {
row["connected_speaker_label"] = connLabel
}
row["connected_speaker_id"] = peerLiveSpeakerIDOrNull(connID)
}
func peerLiveSpeakerIDOrNull(id string) any {
if strings.TrimSpace(id) == "" {
return nil
}
return id
}
+79
View File
@@ -0,0 +1,79 @@
package httpapi
import (
"testing"
"evobgp/internal/birdfmt"
"evobgp/internal/store"
)
func TestMatchPeerOnSpeakers_establishedOnReplica(t *testing.T) {
peer := &store.BGPPeer{
ID: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
Neighbor: "198.51.100.2",
}
views := []speakerBGPLive{
{
SpeakerID: "master-id",
Label: "CP · bgp.shz.su",
Sessions: []birdfmt.BGPSession{
{Name: birdfmt.PeerProtocolName(peer.ID), Neighbor: "198.51.100.2", State: "Established"},
},
},
{
SpeakerID: "replica-id",
Label: "bgp2.shz.su",
Sessions: []birdfmt.BGPSession{
{Name: birdfmt.PeerProtocolName(peer.ID), Neighbor: "198.51.100.2", State: "Established"},
},
},
}
state, connID, connLabel, established, on, mismatch := matchPeerOnSpeakers(peer, views)
if state != "Established" || connID != "" || connLabel != "CP · bgp.shz.su, bgp2.shz.su" {
t.Fatalf("got state=%q conn=%q label=%q", state, connID, connLabel)
}
if mismatch || len(on) != 2 || len(established) != 2 {
t.Fatalf("on=%+v established=%+v mismatch=%v", on, established, mismatch)
}
}
func TestMatchPeerOnSpeakers_multipleEstablished(t *testing.T) {
peer := &store.BGPPeer{ID: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", Neighbor: "198.51.100.2/32"}
views := []speakerBGPLive{
{SpeakerID: "a", Label: "n1", Sessions: []birdfmt.BGPSession{{Name: birdfmt.PeerProtocolName(peer.ID), State: "Established"}}},
{SpeakerID: "b", Label: "n2", Sessions: []birdfmt.BGPSession{{Name: birdfmt.PeerProtocolName(peer.ID), State: "Established"}}},
}
_, connID, label, established, on, mismatch := matchPeerOnSpeakers(peer, views)
if mismatch || connID != "" || label != "n1, n2" || len(established) != 2 || len(on) != 2 {
t.Fatalf("connID=%q label=%q established=%+v on=%+v mismatch=%v", connID, label, established, on, mismatch)
}
}
func TestMatchPeerOnSpeakers_mismatchConfiguredSpeaker(t *testing.T) {
replica := "replica-id"
peer := &store.BGPPeer{
ID: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
Neighbor: "198.51.100.2",
SpeakerID: &replica,
}
views := []speakerBGPLive{
{SpeakerID: "master-id", Label: "CP", Sessions: []birdfmt.BGPSession{{Name: birdfmt.PeerProtocolName(peer.ID), State: "Established"}}},
{SpeakerID: replica, Label: "bgp2", Sessions: []birdfmt.BGPSession{{Name: birdfmt.PeerProtocolName(peer.ID), State: "Idle"}}},
}
_, _, _, _, _, mismatch := matchPeerOnSpeakers(peer, views)
if !mismatch {
t.Fatal("expected mismatch when configured replica has no Established")
}
}
func TestMatchPeerOnSpeakers_pollError(t *testing.T) {
peer := &store.BGPPeer{ID: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", Neighbor: "198.51.100.2"}
views := []speakerBGPLive{
{Label: "CP (local BIRD)", Sessions: []birdfmt.BGPSession{{Name: birdfmt.PeerProtocolName(peer.ID), State: "Established"}}},
{SpeakerID: "replica-id", Label: "bgp2.shz.su", Error: "HTTP 404: Not Found"},
}
_, _, _, established, on, _ := matchPeerOnSpeakers(peer, views)
if len(established) != 1 || len(on) != 2 || on[1].PollError == "" {
t.Fatalf("on=%+v established=%+v", on, established)
}
}
+26 -16
View File
@@ -283,17 +283,21 @@ func (s *Server) handleListPeers(w http.ResponseWriter, r *http.Request) {
}
allPeers := s.store.ListPeers(a.TenantID)
page, next, more := store.PaginateOffset(allPeers, r.URL.Query().Get("cursor"), parseListLimit(r))
liveStates := s.liveBGPProtocolStates(r)
fresh := r != nil && strings.EqualFold(strings.TrimSpace(r.URL.Query().Get("live")), "1")
ctx, cancel := context.WithTimeout(r.Context(), 12*time.Second)
defer cancel()
liveViews := s.collectSpeakerBGPLive(ctx, a.TenantID, fresh)
items := make([]map[string]any, 0, len(page))
for _, p := range page {
row := peerJSON(p)
if st, ok := liveStates[peerProtocolNameForID(p.ID)]; ok && strings.TrimSpace(st) != "" {
row["session_state"] = strings.TrimSpace(st)
}
applyPeerLiveFields(row, p, liveViews)
items = append(items, row)
}
writeJSON(w, http.StatusOK, map[string]any{
"items": items, "next_cursor": strPtrOrNull(next), "has_more": more,
"items": items,
"next_cursor": strPtrOrNull(next),
"has_more": more,
"live_speaker_poll": liveSpeakerPollJSON(liveViews),
})
}
@@ -366,16 +370,9 @@ func extractBGPSessionState(line string) string {
return ""
}
// peerProtocolNameForID must stay in sync with pipeline peer protocol naming.
// peerProtocolNameForID forwards to birdfmt for tests and legacy callers.
func peerProtocolNameForID(peerID string) string {
s := strings.ReplaceAll(strings.TrimSpace(peerID), "-", "")
if len(s) > 16 {
s = s[:16]
}
if s == "" {
s = "x"
}
return "evobgp_p_" + s
return birdfmt.PeerProtocolName(peerID)
}
func (s *Server) handleListSpeakers(w http.ResponseWriter, r *http.Request) {
@@ -388,9 +385,22 @@ func (s *Server) handleListSpeakers(w http.ResponseWriter, r *http.Request) {
return
}
speakers := s.store.ListSpeakersForTenant(a.TenantID)
fresh := r != nil && strings.EqualFold(strings.TrimSpace(r.URL.Query().Get("live")), "1")
var liveByID map[string]map[string]any
if fresh {
ctx, cancel := context.WithTimeout(r.Context(), 12*time.Second)
defer cancel()
liveByID = s.collectSpeakerLiveStatus(ctx, a.TenantID, true, speakers)
}
items := make([]map[string]any, 0, len(speakers))
for _, sp := range speakers {
items = append(items, speakerJSONFromStore(s.store, sp))
row := speakerJSONFromStore(s.store, sp)
if liveByID != nil {
if live, ok := liveByID[sp.ID]; ok {
row["live"] = live
}
}
items = append(items, row)
}
writeJSON(w, http.StatusOK, map[string]any{
"items": items, "next_cursor": nil, "has_more": false,
@@ -569,7 +579,7 @@ func (s *Server) handleGetRevision(w http.ResponseWriter, r *http.Request) {
if !s.requireAtLeast(w, a, "viewer") {
return
}
rev, err := s.store.GetRevision(a.TenantID, r.PathValue("revision_id"))
rev, err := s.store.GetRevisionSummary(a.TenantID, r.PathValue("revision_id"))
if err != nil {
writeProblem(w, http.StatusNotFound, "Not Found", "revision not found")
return
+41
View File
@@ -65,6 +65,7 @@ func (s *Server) registerCRUDRoutes(m *http.ServeMux) {
m.HandleFunc("POST /speakers", s.handlePostSpeaker)
m.HandleFunc("GET /speakers/{speaker_id}", s.handleGetSpeakerByID)
m.HandleFunc("PATCH /speakers/{speaker_id}", s.handlePatchSpeaker)
m.HandleFunc("DELETE /speakers/{speaker_id}", s.handleDeleteSpeaker)
m.HandleFunc("GET /revisions/{revision_id}/prefixes", s.handleRevisionPrefixes)
@@ -241,6 +242,14 @@ func (s *Server) handlePreviewCDNSource(w http.ResponseWriter, r *http.Request)
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", "url is required")
return
}
if _, err := pipeline.ValidateCDNURL(u); err != nil {
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", invalidInputDetail)
return
}
if err := pipeline.ResolveCDNURLHost(r.Context(), u); err != nil {
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", invalidInputDetail)
return
}
mod, err := s.store.GetModule(a.TenantID, r.PathValue("module_id"))
if err != nil {
writeStoreErr(w, err)
@@ -303,6 +312,16 @@ func (s *Server) handlePostCDNSource(w http.ResponseWriter, r *http.Request) {
writeProblem(w, http.StatusBadRequest, "Bad Request", "invalid json")
return
}
if body.URL != "" {
if _, err := pipeline.ValidateCDNURL(body.URL); err != nil {
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", invalidInputDetail)
return
}
if err := pipeline.ResolveCDNURLHost(r.Context(), body.URL); err != nil {
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", invalidInputDetail)
return
}
}
mid := r.PathValue("module_id")
x, err := s.store.CreateCDNSource(a.TenantID, mid, &body)
if err != nil {
@@ -323,6 +342,16 @@ func (s *Server) handlePatchCDNSource(w http.ResponseWriter, r *http.Request) {
writeProblem(w, http.StatusBadRequest, "Bad Request", "invalid json")
return
}
if body.URL != nil && strings.TrimSpace(*body.URL) != "" {
if _, err := pipeline.ValidateCDNURL(*body.URL); err != nil {
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", invalidInputDetail)
return
}
if err := pipeline.ResolveCDNURLHost(r.Context(), *body.URL); err != nil {
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", invalidInputDetail)
return
}
}
mid := r.PathValue("module_id")
x, err := s.store.UpdateCDNSource(a.TenantID, mid, r.PathValue("source_id"), &body)
if err != nil {
@@ -1021,6 +1050,18 @@ func (s *Server) handlePatchSpeaker(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, speakerJSONFromStore(s.store, x))
}
func (s *Server) handleDeleteSpeaker(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "editor") {
return
}
if err := s.store.DeleteSpeaker(a.TenantID, r.PathValue("speaker_id")); err != nil {
writeStoreErr(w, err)
return
}
w.WriteHeader(http.StatusNoContent)
}
func (s *Server) handleRevisionPrefixes(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") {
+4 -1
View File
@@ -39,7 +39,10 @@ func speakerJSONFromStore(st store.Backend, sp *store.Speaker) map[string]any {
if strings.TrimSpace(sp.MetaJSON) != "" && sp.MetaJSON != "{}" {
var raw map[string]any
if json.Unmarshal([]byte(sp.MetaJSON), &raw) == nil {
m["meta_json"] = raw
delete(raw, "agent_secret")
if len(raw) > 0 {
m["meta_json"] = raw
}
}
}
if meta.AgentDomain != "" {
+129
View File
@@ -0,0 +1,129 @@
package httpapi
import (
"context"
"strings"
"sync"
"time"
"evobgp/internal/birdfmt"
"evobgp/internal/nodedispatch"
"evobgp/internal/store"
)
func countBGPSessions(sessions []birdfmt.BGPSession) (total, established int) {
total = len(sessions)
for _, s := range sessions {
if strings.EqualFold(strings.TrimSpace(s.State), "Established") {
established++
}
}
return total, established
}
func speakerLiveStatusJSON(sp *store.Speaker, view speakerBGPLive, health *nodedispatch.AgentHealthResult) map[string]any {
total, established := countBGPSessions(view.Sessions)
m := map[string]any{
"label": view.Label,
"bgp_poll_ok": view.Error == "",
"bgp_sessions_total": total,
"bgp_established": established,
}
if view.Error != "" {
m["bgp_poll_error"] = view.Error
}
if health != nil {
m["agent_ok"] = health.OK
if health.Error != "" {
m["agent_error"] = health.Error
}
if health.LastSyncAt != "" {
m["agent_last_sync_at"] = health.LastSyncAt
}
if health.LastAppliedRevisionID != "" {
m["agent_last_applied_revision_id"] = health.LastAppliedRevisionID
}
} else if sp != nil && strings.EqualFold(strings.TrimSpace(sp.Role), "master") {
m["agent_ok"] = view.Error == ""
if view.Error != "" {
m["agent_error"] = view.Error
}
} else if sp != nil && store.SpeakerNeedsRemoteDispatch(sp.Role, store.ParseSpeakerMeta(sp.MetaJSON)) {
m["agent_ok"] = false
m["agent_error"] = "agent health not polled"
}
if len(view.Sessions) > 0 {
sess := make([]map[string]any, 0, len(view.Sessions))
for _, s := range view.Sessions {
row := map[string]any{
"name": s.Name,
"state": s.State,
}
if strings.TrimSpace(s.Neighbor) != "" {
row["neighbor"] = s.Neighbor
}
sess = append(sess, row)
}
m["sessions"] = sess
}
return m
}
func (s *Server) collectSpeakerLiveStatus(ctx context.Context, tenantID string, fresh bool, speakers []*store.Speaker) map[string]map[string]any {
views := s.collectSpeakerBGPLive(ctx, tenantID, fresh)
viewByID := make(map[string]speakerBGPLive, len(views))
for _, v := range views {
if v.SpeakerID != "" {
viewByID[v.SpeakerID] = v
}
}
opts := nodedispatch.Options{Timeout: 8 * time.Second}
type healthWrap struct {
id string
h nodedispatch.AgentHealthResult
}
healthCh := make(chan healthWrap, len(speakers))
var wg sync.WaitGroup
for _, sp := range speakers {
if sp == nil {
continue
}
meta := store.ParseSpeakerMeta(sp.MetaJSON)
if !store.SpeakerNeedsRemoteDispatch(sp.Role, meta) {
continue
}
wg.Add(1)
go func(speaker *store.Speaker) {
defer wg.Done()
healthCh <- healthWrap{
id: speaker.ID,
h: nodedispatch.FetchAgentHealth(ctx, speaker, opts),
}
}(sp)
}
wg.Wait()
close(healthCh)
healthByID := make(map[string]nodedispatch.AgentHealthResult, len(speakers))
for hw := range healthCh {
healthByID[hw.id] = hw.h
}
out := make(map[string]map[string]any, len(speakers))
for _, sp := range speakers {
if sp == nil {
continue
}
view, ok := viewByID[sp.ID]
if !ok {
view = speakerBGPLive{SpeakerID: sp.ID, Label: speakerDisplayLabel(sp)}
}
var hp *nodedispatch.AgentHealthResult
if h, ok := healthByID[sp.ID]; ok {
hCopy := h
hp = &hCopy
}
out[sp.ID] = speakerLiveStatusJSON(sp, view, hp)
}
return out
}
+74
View File
@@ -0,0 +1,74 @@
package httpapi
import (
"testing"
"evobgp/internal/birdfmt"
"evobgp/internal/nodedispatch"
"evobgp/internal/store"
)
func TestCountBGPSessions(t *testing.T) {
total, est := countBGPSessions([]birdfmt.BGPSession{
{Name: "p1", State: "Established"},
{Name: "p2", State: "Idle"},
{Name: "p3", State: "established"},
})
if total != 3 || est != 2 {
t.Fatalf("total=%d established=%d", total, est)
}
}
func TestSpeakerLiveStatusJSON_masterUsesBirdPoll(t *testing.T) {
sp := &store.Speaker{ID: "m1", Role: "master", Endpoint: "https://cp.example"}
view := speakerBGPLive{
SpeakerID: "m1",
Label: "CP · cp.example",
Sessions: []birdfmt.BGPSession{
{Name: "evobgp_peer_x", State: "Established"},
},
}
m := speakerLiveStatusJSON(sp, view, nil)
if m["agent_ok"] != true || m["bgp_established"] != 1 || m["bgp_sessions_total"] != 1 {
t.Fatalf("got %#v", m)
}
}
func TestSpeakerLiveStatusJSON_replicaWithHealth(t *testing.T) {
sp := &store.Speaker{
ID: "r1",
Role: "replica",
Endpoint: "https://node.example",
MetaJSON: `{"agent_domain":"node.example","agent_secret":"s"}`,
}
view := speakerBGPLive{
SpeakerID: "r1",
Label: "node.example",
Sessions: []birdfmt.BGPSession{{Name: "p", State: "Idle"}},
}
health := &nodedispatch.AgentHealthResult{
OK: true,
LastSyncAt: "2026-05-21T12:00:00Z",
LastAppliedRevisionID: "rev-1",
}
m := speakerLiveStatusJSON(sp, view, health)
if m["agent_ok"] != true || m["agent_last_sync_at"] != "2026-05-21T12:00:00Z" {
t.Fatalf("got %#v", m)
}
if m["bgp_established"] != 0 || m["bgp_poll_ok"] != true {
t.Fatalf("bgp fields: %#v", m)
}
}
func TestSpeakerLiveStatusJSON_pollError(t *testing.T) {
sp := &store.Speaker{ID: "r1", Role: "replica", MetaJSON: `{"agent_domain":"x.example"}`}
view := speakerBGPLive{SpeakerID: "r1", Label: "x.example", Error: "HTTP 503"}
health := &nodedispatch.AgentHealthResult{OK: false, Error: "timeout"}
m := speakerLiveStatusJSON(sp, view, health)
if m["bgp_poll_ok"] != false || m["bgp_poll_error"] != "HTTP 503" {
t.Fatalf("got %#v", m)
}
if m["agent_ok"] != false {
t.Fatalf("agent_ok: %#v", m)
}
}
+62
View File
@@ -0,0 +1,62 @@
package httpapi
import (
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestGetSpeaker_redactsAgentSecret(t *testing.T) {
srv, err := New(Options{InsecureDev: true, SeedDemo: true, BundleSeedHex: testBundleSeed})
if err != nil {
t.Fatal(err)
}
defer srv.Close()
tenant, _, _, _, demoSpk := srv.Store().DemoIDs()
mustSetTestAPIKeys(t, srv, "vwkey|"+tenant+"|viewer")
req := httptest.NewRequest(http.MethodGet, "/v1/speakers/"+demoSpk, nil)
req.Header.Set("Authorization", "Bearer vwkey")
rec := httptest.NewRecorder()
srv.Handler().ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status %d body %s", rec.Code, rec.Body.String())
}
var out map[string]any
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatal(err)
}
if out["agent_secret"] != nil {
t.Fatalf("agent_secret must not appear at top level: %#v", out["agent_secret"])
}
meta, _ := out["meta_json"].(map[string]any)
if meta != nil {
if v, ok := meta["agent_secret"]; ok && v != nil && v != "" {
t.Fatalf("agent_secret must be redacted from meta_json: %#v", v)
}
}
}
func TestListSpeakers_redactsAgentSecret(t *testing.T) {
srv, err := New(Options{InsecureDev: true, SeedDemo: true, BundleSeedHex: testBundleSeed})
if err != nil {
t.Fatal(err)
}
defer srv.Close()
tenant, _, _, _, _ := srv.Store().DemoIDs()
mustSetTestAPIKeys(t, srv, "vwkey|"+tenant+"|viewer")
req := httptest.NewRequest(http.MethodGet, "/v1/speakers", nil)
req.Header.Set("Authorization", "Bearer vwkey")
rec := httptest.NewRecorder()
srv.Handler().ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status %d body %s", rec.Code, rec.Body.String())
}
body := rec.Body.String()
if strings.Contains(body, "agent_secret") {
t.Fatalf("list response must not contain agent_secret: %s", body)
}
}
+21
View File
@@ -41,6 +41,27 @@ func TestPostSpeaker_defaultsFromEndpointIP(t *testing.T) {
}
}
func TestDeleteSpeaker(t *testing.T) {
srv, err := New(Options{InsecureDev: true, SeedDemo: true, BundleSeedHex: testBundleSeed})
if err != nil {
t.Fatal(err)
}
defer srv.Close()
tenant, _, _, _, demoSpk := srv.Store().DemoIDs()
mustSetTestAPIKeys(t, srv, "edkey|"+tenant+"|editor")
req := httptest.NewRequest(http.MethodDelete, "/v1/speakers/"+demoSpk, nil)
req.Header.Set("Authorization", "Bearer edkey")
rec := httptest.NewRecorder()
srv.Handler().ServeHTTP(rec, req)
if rec.Code != http.StatusNoContent {
t.Fatalf("status %d body %s", rec.Code, rec.Body.String())
}
if _, err := srv.Store().GetSpeaker(tenant, demoSpk); err == nil {
t.Fatal("speaker should be deleted")
}
}
func TestGetBundleSigningPublicKey(t *testing.T) {
srv, err := New(Options{InsecureDev: true, SeedDemo: true, BundleSeedHex: testBundleSeed})
if err != nil {
+55
View File
@@ -0,0 +1,55 @@
package httpclient
import (
"sync"
"time"
)
const (
defaultBreakerThreshold = 5
defaultBreakerCooldown = 30 * time.Second
)
type hostBreaker struct {
mu sync.Mutex
failures int
openUntil time.Time
}
var hostBreakers sync.Map // string -> *hostBreaker
func breakerForHost(host string) *hostBreaker {
if host == "" {
host = "_"
}
v, _ := hostBreakers.LoadOrStore(host, &hostBreaker{})
return v.(*hostBreaker)
}
func (b *hostBreaker) allow() bool {
b.mu.Lock()
defer b.mu.Unlock()
return time.Now().After(b.openUntil)
}
func (b *hostBreaker) recordSuccess() {
b.mu.Lock()
defer b.mu.Unlock()
b.failures = 0
b.openUntil = time.Time{}
}
func (b *hostBreaker) recordFailure() {
b.mu.Lock()
defer b.mu.Unlock()
b.failures++
if b.failures >= defaultBreakerThreshold {
b.openUntil = time.Now().Add(defaultBreakerCooldown)
b.failures = 0
}
}
// ResetHostBreakers clears all circuit breakers (tests only).
func ResetHostBreakers() {
hostBreakers = sync.Map{}
}
+34
View File
@@ -0,0 +1,34 @@
package httpclient
import (
"context"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"time"
)
func TestDoWithBreaker_opensAfterFailures(t *testing.T) {
ResetHostBreakers()
var calls atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls.Add(1)
http.Error(w, "fail", http.StatusBadGateway)
}))
defer srv.Close()
hc := New(5 * time.Second)
for i := 0; i < defaultBreakerThreshold*3; i++ {
req, _ := http.NewRequest(http.MethodGet, srv.URL, nil)
_, _ = DoWithBreaker(context.Background(), hc, req, 1)
}
req, _ := http.NewRequest(http.MethodGet, srv.URL, nil)
_, err := DoWithBreaker(context.Background(), hc, req, 1)
if err == nil || err.Error() == "" {
t.Fatal("expected circuit open error")
}
if got := calls.Load(); got == 0 {
t.Fatal("expected at least one upstream call")
}
}
+87
View File
@@ -0,0 +1,87 @@
// Package httpclient provides shared HTTP clients and retry helpers for outbound calls.
package httpclient
import (
"context"
"fmt"
"io"
"net/http"
"time"
)
const DefaultTimeout = 45 * time.Second
// New returns an HTTP client with timeout and tuned idle connection pooling.
func New(timeout time.Duration) *http.Client {
if timeout <= 0 {
timeout = DefaultTimeout
}
tr := http.DefaultTransport.(*http.Transport).Clone()
tr.MaxIdleConns = 100
tr.MaxIdleConnsPerHost = 10
return &http.Client{Timeout: timeout, Transport: tr}
}
// DoWithRetry executes hc.Do(req) up to maxAttempts times with linear backoff.
func DoWithRetry(ctx context.Context, hc *http.Client, req *http.Request, maxAttempts int) (*http.Response, error) {
if maxAttempts <= 0 {
maxAttempts = 3
}
var lastErr error
for attempt := 0; attempt < maxAttempts; attempt++ {
if attempt > 0 {
wait := time.Duration(attempt) * 2 * time.Second
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(wait):
}
if req.GetBody != nil {
body, err := req.GetBody()
if err != nil {
return nil, err
}
req.Body = body
}
}
reqClone := req.Clone(ctx)
resp, err := hc.Do(reqClone)
if err != nil {
lastErr = err
continue
}
if resp.StatusCode >= 500 {
_, _ = io.Copy(io.Discard, resp.Body)
_ = resp.Body.Close()
lastErr = fmt.Errorf("httpclient: upstream %s", resp.Status)
continue
}
return resp, nil
}
if lastErr != nil {
return nil, lastErr
}
return nil, fmt.Errorf("httpclient: request failed after %d attempts", maxAttempts)
}
// DoWithBreaker applies per-host circuit breaking then retries transient failures.
func DoWithBreaker(ctx context.Context, hc *http.Client, req *http.Request, maxAttempts int) (*http.Response, error) {
if req == nil || req.URL == nil {
return nil, fmt.Errorf("httpclient: nil request")
}
br := breakerForHost(req.URL.Hostname())
if !br.allow() {
return nil, fmt.Errorf("httpclient: circuit open for %s", req.URL.Hostname())
}
resp, err := DoWithRetry(ctx, hc, req, maxAttempts)
if err != nil {
br.recordFailure()
return nil, err
}
if resp.StatusCode >= 500 {
br.recordFailure()
return resp, nil
}
br.recordSuccess()
return resp, nil
}
+39
View File
@@ -0,0 +1,39 @@
package httpclient
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"time"
)
func TestDoWithRetry_retriesOn500(t *testing.T) {
var calls int
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
if calls < 3 {
http.Error(w, "fail", http.StatusBadGateway)
return
}
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ok"))
}))
defer srv.Close()
req, err := http.NewRequest(http.MethodGet, srv.URL, nil)
if err != nil {
t.Fatal(err)
}
resp, err := DoWithRetry(context.Background(), New(5*time.Second), req, 3)
if err != nil {
t.Fatal(err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
t.Fatalf("status %d", resp.StatusCode)
}
if calls != 3 {
t.Fatalf("want 3 calls, got %d", calls)
}
}
+12 -3
View File
@@ -3,11 +3,11 @@ package ingest
import (
"context"
"log"
"net/http"
"time"
"evobgp/internal/broker"
"evobgp/internal/config"
"evobgp/internal/httpclient"
"evobgp/internal/pipeline"
"evobgp/internal/store"
)
@@ -17,6 +17,8 @@ type Deps struct {
Store store.Backend
}
var lastMaintenance time.Time
// Run blocks until ctx is cancelled.
func Run(ctx context.Context, deps *Deps) {
cfg := config.Load()
@@ -24,7 +26,7 @@ func Run(ctx context.Context, deps *Deps) {
if deps == nil || deps.Store == nil {
log.Fatalf("evobgp-ingest: missing store (pass ingest.Deps from BootstrapWorkers or evobgp-all)")
}
hc := &http.Client{Timeout: 45 * time.Second}
hc := httpclient.New(httpclient.DefaultTimeout)
t := time.NewTicker(60 * time.Second)
defer t.Stop()
log.Printf("evobgp-ingest: active (CDN conditional GET / ETag prefetch)")
@@ -34,7 +36,14 @@ func Run(ctx context.Context, deps *Deps) {
log.Printf("evobgp-ingest: stopped")
return
case <-t.C:
if err := pipeline.PrefetchCDNSourceETags(context.Background(), deps.Store, hc); err != nil {
if deps.Store != nil && time.Since(lastMaintenance) > time.Hour {
deps.Store.RunPeriodicMaintenance(ctx)
lastMaintenance = time.Now()
}
prefetchCtx, cancel := context.WithTimeout(ctx, 50*time.Second)
err := pipeline.PrefetchCDNSourceETags(prefetchCtx, deps.Store, hc)
cancel()
if err != nil {
log.Printf("evobgp-ingest: prefetch: %v", err)
}
}
+49 -4
View File
@@ -182,6 +182,8 @@ type Registry struct {
workerStart func(j *Job)
workerSem chan struct{}
onTerminal func(j *Job)
onEnqueued func(j *Job)
onRunning func(j *Job)
}
type idempoKey struct {
@@ -209,6 +211,44 @@ func (r *Registry) SetTerminalHook(fn func(j *Job)) {
r.onTerminal = fn
}
// SetPersistHooks registers best-effort callbacks for job lifecycle persistence.
func (r *Registry) SetPersistHooks(onEnqueued, onRunning, onTerminal func(j *Job)) {
if r == nil {
return
}
r.mu.Lock()
defer r.mu.Unlock()
r.onEnqueued = onEnqueued
r.onRunning = onRunning
if onTerminal != nil {
r.onTerminal = onTerminal
}
}
func (r *Registry) fireEnqueued(j *Job) {
if r == nil || j == nil {
return
}
r.mu.RLock()
fn := r.onEnqueued
r.mu.RUnlock()
if fn != nil {
fn(j)
}
}
func (r *Registry) fireRunning(j *Job) {
if r == nil || j == nil {
return
}
r.mu.RLock()
fn := r.onRunning
r.mu.RUnlock()
if fn != nil {
fn(j)
}
}
func (r *Registry) fireTerminal(j *Job) {
if r == nil || j == nil {
return
@@ -271,8 +311,6 @@ func (r *Registry) pruneTerminalIfOver(maxJobs int) {
// Enqueue creates a job or returns an existing one for the same idempotency key.
func (r *Registry) Enqueue(tenantID, kind string, idempotencyKey *string, moduleID *string, meta map[string]any) (*Job, bool, error) {
r.mu.Lock()
defer r.mu.Unlock()
maxJobs := registryMaxJobsFromEnv()
r.pruneTerminalIfOver(maxJobs)
@@ -281,6 +319,7 @@ func (r *Registry) Enqueue(tenantID, kind string, idempotencyKey *string, module
if existing, ok := r.byIdempo[k]; ok {
st := existing.statusLocked()
if st == StatusQueued || st == StatusRunning {
r.mu.Unlock()
return existing, false, nil
}
delete(r.byIdempo, k)
@@ -302,8 +341,14 @@ func (r *Registry) Enqueue(tenantID, kind string, idempotencyKey *string, module
}
r.byID[j.ID] = j
r.pruneTerminalIfOver(maxJobs)
enqueuedHook := r.onEnqueued
workerStart := r.workerStart
r.mu.Unlock()
if r.workerStart != nil {
if enqueuedHook != nil {
enqueuedHook(j)
}
if workerStart != nil {
go func() {
r.workerSem <- struct{}{}
active := len(r.workerSem)
@@ -313,7 +358,7 @@ func (r *Registry) Enqueue(tenantID, kind string, idempotencyKey *string, module
<-r.workerSem
observability.RecordJobQueueDepth(len(r.workerSem), capacity)
}()
r.workerStart(j)
workerStart(j)
}()
}
return j, true, nil
+7 -3
View File
@@ -13,6 +13,7 @@ import (
"evobgp/internal/birddeploy"
"evobgp/internal/birdfmt"
"evobgp/internal/httpclient"
"evobgp/internal/nodedispatch"
"evobgp/internal/observability"
"evobgp/internal/pipeline"
@@ -70,7 +71,7 @@ type revisionLogEntry struct {
Message string `json:"message"`
}
var defaultWorkerHTTP = &http.Client{Timeout: 45 * time.Second}
var defaultWorkerHTTP = httpclient.New(httpclient.DefaultTimeout)
func (w *Worker) httpClient() *http.Client {
if w != nil && w.HTTPClient != nil {
@@ -94,6 +95,9 @@ func (w *Worker) Process(j *Job) {
return
}
j.MarkRunning()
if w != nil && w.Registry != nil {
w.Registry.fireRunning(j)
}
if j.IsCancelRequested() {
j.MarkCancelled()
return
@@ -219,8 +223,8 @@ func (w *Worker) runPeerReconcile(j *Job) {
} else {
j.mergeMeta(map[string]any{"log_build_error": err.Error()})
}
j.Succeed()
w.enqueueDeployAllSpeakers(j, j.TenantID, revID)
j.Succeed()
}
func (w *Worker) peerTriggerModuleID(tenantID string, latest []*store.Revision) (string, error) {
@@ -346,8 +350,8 @@ func (w *Worker) finishModuleRefreshSuccess(j *Job, triggerModuleID string) {
} else {
j.mergeMeta(map[string]any{"log_build_error": err.Error()})
}
j.Succeed()
w.enqueueDeployAllSpeakers(j, j.TenantID, rev)
j.Succeed()
}
// enqueueDeployAllSpeakers queues the same work as POST /v1/apply (all speakers, no speaker_id).
+11 -2
View File
@@ -16,6 +16,7 @@ import (
"evobgp/internal/birdfmt"
"evobgp/internal/bundle"
"evobgp/internal/httpclient"
"evobgp/internal/signing"
)
@@ -55,6 +56,10 @@ func CmdPullBundle(args []string) int {
return 0
}
func nodeHTTPClient() *http.Client {
return httpclient.New(60 * time.Second)
}
func fetchLatestRevision(base, token, speaker string) (string, error) {
u := strings.TrimRight(base, "/") + "/v1/speakers/" + speaker + "/revisions/latest"
req, err := http.NewRequest(http.MethodGet, u, nil)
@@ -62,7 +67,9 @@ func fetchLatestRevision(base, token, speaker string) (string, error) {
return "", err
}
req.Header.Set("Authorization", "Bearer "+token)
resp, err := http.DefaultClient.Do(req)
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
resp, err := httpclient.DoWithRetry(ctx, nodeHTTPClient(), req, 3)
if err != nil {
return "", err
}
@@ -90,7 +97,9 @@ func fetchBundle(base, token, speaker, revision string) ([]byte, error) {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+token)
resp, err := http.DefaultClient.Do(req)
ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second)
defer cancel()
resp, err := httpclient.DoWithRetry(ctx, nodeHTTPClient(), req, 3)
if err != nil {
return nil, err
}
+69
View File
@@ -0,0 +1,69 @@
package nodedispatch
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"evobgp/internal/birdfmt"
"evobgp/internal/store"
)
// BirdProtocolsResult is agent birdc scrape outcome.
type BirdProtocolsResult struct {
SpeakerID string `json:"speaker_id,omitempty"`
Sessions []birdfmt.BGPSession `json:"sessions"`
Error string `json:"error,omitempty"`
}
// FetchBirdProtocols GETs /v1/agent/bird/protocols on a replica agent.
func FetchBirdProtocols(ctx context.Context, sp *store.Speaker, opts Options) BirdProtocolsResult {
res := BirdProtocolsResult{}
if sp != nil {
res.SpeakerID = sp.ID
}
if sp == nil {
res.Error = "nil speaker"
return res
}
meta := store.ParseSpeakerMeta(sp.MetaJSON)
url := store.AgentBirdProtocolsURL(meta)
if url == "" {
res.Error = "agent_domain not configured"
return res
}
secret := strings.TrimSpace(meta.AgentSecret)
if secret == "" {
res.Error = "agent_secret missing"
return res
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
res.Error = err.Error()
return res
}
req.Header.Set("Authorization", "Bearer "+secret)
resp, err := opts.client().Do(req)
if err != nil {
res.Error = err.Error()
return res
}
defer func() { _ = resp.Body.Close() }()
b, _ := io.ReadAll(resp.Body)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
res.Error = fmt.Sprintf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(b)))
return res
}
var out struct {
Sessions []birdfmt.BGPSession `json:"sessions"`
}
if err := json.Unmarshal(b, &out); err != nil {
res.Error = err.Error()
return res
}
res.Sessions = out.Sessions
return res
}
+7 -26
View File
@@ -156,33 +156,14 @@ func WakeReplicas(ctx context.Context, st store.Backend, tenantID, revisionID st
return out
}
// CheckHealth GETs /v1/agent/health for UI Connected/Offline status.
// CheckHealth is deprecated; use FetchAgentHealth.
func CheckHealth(ctx context.Context, sp *store.Speaker, opts Options) (ok bool, detail string) {
if sp == nil {
return false, "nil speaker"
}
meta := store.ParseSpeakerMeta(sp.MetaJSON)
url := store.AgentHealthURL(meta)
if url == "" {
return false, "agent_domain not configured"
}
secret := strings.TrimSpace(meta.AgentSecret)
if secret == "" {
return false, "agent_secret missing"
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return false, err.Error()
}
req.Header.Set("Authorization", "Bearer "+secret)
resp, err := opts.client().Do(req)
if err != nil {
return false, err.Error()
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
res := FetchAgentHealth(ctx, sp, opts)
if res.OK {
return true, "connected"
}
b, _ := io.ReadAll(resp.Body)
return false, fmt.Sprintf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(b)))
if res.Error != "" {
return false, res.Error
}
return false, "agent unhealthy"
}
+69
View File
@@ -0,0 +1,69 @@
package nodedispatch
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"evobgp/internal/store"
)
// AgentHealthResult is the parsed outcome of GET /v1/agent/health on a replica.
type AgentHealthResult struct {
OK bool
Error string
LastAppliedRevisionID string
LastSyncAt string
}
// FetchAgentHealth GETs /v1/agent/health for UI Connected/Offline status.
func FetchAgentHealth(ctx context.Context, sp *store.Speaker, opts Options) AgentHealthResult {
if sp == nil {
return AgentHealthResult{Error: "nil speaker"}
}
meta := store.ParseSpeakerMeta(sp.MetaJSON)
url := store.AgentHealthURL(meta)
if url == "" {
return AgentHealthResult{Error: "agent_domain not configured"}
}
secret := strings.TrimSpace(meta.AgentSecret)
if secret == "" {
return AgentHealthResult{Error: "agent_secret missing"}
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return AgentHealthResult{Error: err.Error()}
}
req.Header.Set("Authorization", "Bearer "+secret)
resp, err := opts.client().Do(req)
if err != nil {
return AgentHealthResult{Error: err.Error()}
}
defer func() { _ = resp.Body.Close() }()
b, _ := io.ReadAll(resp.Body)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return AgentHealthResult{
Error: fmt.Sprintf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(b))),
}
}
var out struct {
OK bool `json:"ok"`
LastAppliedRevisionID string `json:"last_applied_revision_id"`
LastSyncAt string `json:"last_sync_at"`
}
if err := json.Unmarshal(b, &out); err != nil {
return AgentHealthResult{Error: err.Error()}
}
res := AgentHealthResult{
OK: out.OK,
LastAppliedRevisionID: strings.TrimSpace(out.LastAppliedRevisionID),
LastSyncAt: strings.TrimSpace(out.LastSyncAt),
}
if !res.OK {
res.Error = "agent reported ok=false"
}
return res
}
+8 -4
View File
@@ -11,6 +11,7 @@ import (
)
func TestCollectModulePrefixRows_CDNSendsIfNoneMatch(t *testing.T) {
t.Setenv("EVOBGP_CDN_ALLOW_PRIVATE", "1")
m := store.NewMemory()
m.SeedDemo()
tenant, _, _, _, _ := m.DemoIDs()
@@ -25,7 +26,7 @@ func TestCollectModulePrefixRows_CDNSendsIfNoneMatch(t *testing.T) {
}
var gotIfNoneMatch string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotIfNoneMatch = strings.TrimSpace(r.Header.Get("If-None-Match"))
w.Header().Set("ETag", "etag-new")
_, _ = w.Write([]byte("198.51.100.0/24\n"))
@@ -54,6 +55,7 @@ func TestCollectModulePrefixRows_CDNSendsIfNoneMatch(t *testing.T) {
}
func TestCollectModulePrefixRows_CDN304UsesSnapshot(t *testing.T) {
t.Setenv("EVOBGP_CDN_ALLOW_PRIVATE", "1")
m := store.NewMemory()
m.SeedDemo()
tenant, _, _, _, _ := m.DemoIDs()
@@ -67,7 +69,7 @@ func TestCollectModulePrefixRows_CDN304UsesSnapshot(t *testing.T) {
t.Fatal(err)
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotModified)
}))
defer srv.Close()
@@ -99,6 +101,7 @@ func TestCollectModulePrefixRows_CDN304UsesSnapshot(t *testing.T) {
}
func TestRefreshModuleIngest_CDN304UsesStoredSnapshot(t *testing.T) {
t.Setenv("EVOBGP_CDN_ALLOW_PRIVATE", "1")
m := store.NewMemory()
m.SeedDemo()
tenant, _, _, _, _ := m.DemoIDs()
@@ -112,7 +115,7 @@ func TestRefreshModuleIngest_CDN304UsesStoredSnapshot(t *testing.T) {
t.Fatal(err)
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotModified)
}))
defer srv.Close()
@@ -140,6 +143,7 @@ func TestRefreshModuleIngest_CDN304UsesStoredSnapshot(t *testing.T) {
}
func TestCollectModulePrefixRows_CDN304RetriesWithoutETag(t *testing.T) {
t.Setenv("EVOBGP_CDN_ALLOW_PRIVATE", "1")
m := store.NewMemory()
m.SeedDemo()
tenant, _, _, _, _ := m.DemoIDs()
@@ -154,7 +158,7 @@ func TestCollectModulePrefixRows_CDN304RetriesWithoutETag(t *testing.T) {
}
var calls int
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
if calls == 1 {
if got := strings.TrimSpace(r.Header.Get("If-None-Match")); got != "etag-stable" {
+16 -4
View File
@@ -119,6 +119,12 @@ func applyCDNSourceHTTPResult(ctx context.Context, st store.Backend, hc *http.Cl
if u == "" {
return nil, nil
}
if _, err := ValidateCDNURL(u); err != nil {
return nil, err
}
if err := ResolveCDNURLHost(ctx, u); err != nil {
return nil, err
}
sourceKey := cdnSourceKey(src.ID)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
if err != nil {
@@ -127,7 +133,7 @@ func applyCDNSourceHTTPResult(ctx context.Context, st store.Backend, hc *http.Cl
if etag := strings.TrimSpace(src.Etag); etag != "" {
req.Header.Set("If-None-Match", etag)
}
resp, err := hc.Do(req)
resp, err := upstreamHTTPDo(ctx, hc, req)
if err != nil {
return nil, fmt.Errorf("cdn fetch %s: %w", u, err)
}
@@ -143,7 +149,7 @@ func applyCDNSourceHTTPResult(ctx context.Context, st store.Backend, hc *http.Cl
if err != nil {
return nil, err
}
resp, err = hc.Do(req2)
resp, err = upstreamHTTPDo(ctx, hc, req2)
if err != nil {
return nil, fmt.Errorf("cdn fetch %s: %w", u, err)
}
@@ -189,6 +195,12 @@ func fetchCDNSourceRows(ctx context.Context, st store.Backend, hc *http.Client,
if u == "" {
return nil, nil
}
if _, err := ValidateCDNURL(u); err != nil {
return nil, err
}
if err := ResolveCDNURLHost(ctx, u); err != nil {
return nil, err
}
sourceKey := cdnSourceKey(src.ID)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
if err != nil {
@@ -197,7 +209,7 @@ func fetchCDNSourceRows(ctx context.Context, st store.Backend, hc *http.Client,
if etag := strings.TrimSpace(src.Etag); etag != "" {
req.Header.Set("If-None-Match", etag)
}
resp, err := hc.Do(req)
resp, err := upstreamHTTPDo(ctx, hc, req)
if err != nil {
return nil, fmt.Errorf("cdn fetch %s: %w", u, err)
}
@@ -212,7 +224,7 @@ func fetchCDNSourceRows(ctx context.Context, st store.Backend, hc *http.Client,
if err != nil {
return nil, err
}
resp, err = hc.Do(req2)
resp, err = upstreamHTTPDo(ctx, hc, req2)
if err != nil {
return nil, fmt.Errorf("cdn fetch %s: %w", u, err)
}
+116
View File
@@ -0,0 +1,116 @@
package pipeline
import (
"context"
"fmt"
"net"
"net/netip"
"net/url"
"os"
"strings"
"time"
)
func isBlockedCDNIP(ip netip.Addr) bool {
if allowPrivateCDNURLs() {
return false
}
if !ip.IsValid() {
return true
}
return ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsMulticast() ||
ip.IsUnspecified() || ip == netip.MustParseAddr("169.254.169.254")
}
func allowPrivateCDNURLs() bool {
v := strings.TrimSpace(os.Getenv("EVOBGP_CDN_ALLOW_PRIVATE"))
return v == "1" || strings.EqualFold(v, "true")
}
func isBlockedCDNHostname(host string) bool {
if allowPrivateCDNURLs() {
return false
}
h := strings.ToLower(strings.TrimSpace(host))
if h == "" || h == "localhost" {
return true
}
if strings.HasSuffix(h, ".local") || strings.HasSuffix(h, ".internal") || strings.HasSuffix(h, ".localhost") {
return true
}
return false
}
// ValidateCDNURL checks CDN source URLs for SSRF-safe HTTPS endpoints (hostname only; no DNS resolve).
func ValidateCDNURL(raw string) (string, error) {
raw = strings.TrimSpace(raw)
if raw == "" {
return "", fmt.Errorf("pipeline: cdn url is required")
}
u, err := url.Parse(raw)
if err != nil {
return "", fmt.Errorf("pipeline: cdn url invalid: %w", err)
}
if u.Scheme != "https" {
return "", fmt.Errorf("pipeline: cdn url must use https")
}
if u.User != nil {
return "", fmt.Errorf("pipeline: cdn url must not include credentials")
}
host := strings.TrimSpace(u.Hostname())
if host == "" {
return "", fmt.Errorf("pipeline: cdn url missing host")
}
if isBlockedCDNHostname(host) {
return "", fmt.Errorf("pipeline: cdn url blocked host")
}
if ip, err := netip.ParseAddr(host); err == nil {
if isBlockedCDNIP(ip) {
return "", fmt.Errorf("pipeline: cdn url blocked host")
}
}
return u.String(), nil
}
// ResolveCDNURLHost resolves a CDN hostname and rejects private/link-local targets (SSRF at fetch time).
func ResolveCDNURLHost(ctx context.Context, raw string) error {
u, err := url.Parse(strings.TrimSpace(raw))
if err != nil {
return err
}
host := strings.TrimSpace(u.Hostname())
if host == "" {
return fmt.Errorf("pipeline: cdn url missing host")
}
if ip, err := netip.ParseAddr(host); err == nil {
if isBlockedCDNIP(ip) {
return fmt.Errorf("pipeline: cdn url blocked host")
}
return nil
}
if isBlockedCDNHostname(host) {
return fmt.Errorf("pipeline: cdn url blocked host")
}
if ctx == nil {
ctx = context.Background()
}
resolveCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
ips, err := net.DefaultResolver.LookupIP(resolveCtx, "ip", host)
if err != nil {
return fmt.Errorf("pipeline: cdn url dns lookup: %w", err)
}
if len(ips) == 0 {
return fmt.Errorf("pipeline: cdn url dns lookup: no addresses")
}
for _, ip := range ips {
addr, ok := netip.AddrFromSlice(ip)
if !ok {
continue
}
if isBlockedCDNIP(addr) {
return fmt.Errorf("pipeline: cdn url resolves to blocked address")
}
}
return nil
}
+34
View File
@@ -0,0 +1,34 @@
package pipeline
import "testing"
func TestValidateCDNURL(t *testing.T) {
tests := []struct {
raw string
ok bool
want string
}{
{"https://cdn.example.com/prefixes.txt", true, "https://cdn.example.com/prefixes.txt"},
{"http://cdn.example.com/x", false, ""},
{"https://127.0.0.1/x", false, ""},
{"https://10.0.0.1/x", false, ""},
{"https://169.254.169.254/latest/meta-data", false, ""},
{"https://localhost/x", false, ""},
{"file:///etc/passwd", false, ""},
{"https://user:pass@cdn.example.com/x", false, ""},
}
for _, tc := range tests {
got, err := ValidateCDNURL(tc.raw)
if tc.ok && err != nil {
t.Errorf("%q: unexpected err %v", tc.raw, err)
continue
}
if !tc.ok && err == nil {
t.Errorf("%q: expected error", tc.raw)
continue
}
if tc.ok && got != tc.want {
t.Errorf("%q: got %q want %q", tc.raw, got, tc.want)
}
}
}
+57 -2
View File
@@ -25,7 +25,7 @@ func prefixRowsForSource(rows []store.PrefixRow, sourceKey string) []store.Prefi
return out
}
func collectASPrefixRows(ctx context.Context, st store.Backend, hc *http.Client, tenantID string, mod *store.Module, list []*store.ASEntry) ([]store.PrefixRow, error) {
func collectASPrefixRows(ctx context.Context, st store.Backend, hc *http.Client, tenantID string, mod *store.Module, list []*store.ASEntry, priorSnapshot []store.PrefixRow) ([]store.PrefixRow, error) {
moduleID := mod.ID
legacy := strings.TrimSpace(os.Getenv("EVOBGP_ASN_RESOLVE")) == "0"
if legacy {
@@ -76,6 +76,41 @@ func collectASPrefixRows(ctx context.Context, st store.Backend, hc *http.Client,
}
pfxs, holder, err := resolveASNForEntry(ctx, st, hc, entry.ASN)
if err != nil {
if staleOnUpstreamError() {
if staleRows, staleHolder, ok := staleASNPrefixes(st, priorSnapshot, entry.ASN); ok {
logStaleUpstream("asn", fmt.Sprintf("AS%d: %v", entry.ASN, err))
src := fmt.Sprintf("as:%d", entry.ASN)
rows := append([]store.PrefixRow(nil), staleRows...)
for i := range rows {
rows[i].CommunityID = comm
rows[i].Source = src
}
results[idx] = entryResult{
rows: rows,
metaID: entry.ID,
asn: entry.ASN,
holder: staleHolder,
count: int64(len(rows)),
}
return
}
if pfxs2, holder2, ok := asnCacheExpired(st, entry.ASN); ok {
logStaleUpstream("asn", fmt.Sprintf("AS%d expired cache: %v", entry.ASN, err))
src := fmt.Sprintf("as:%d", entry.ASN)
var rows []store.PrefixRow
for _, pfx := range pfxs2 {
rows = append(rows, store.PrefixRow{Prefix: pfx.String(), CommunityID: comm, Source: src})
}
results[idx] = entryResult{
rows: rows,
metaID: entry.ID,
asn: entry.ASN,
holder: holder2,
count: int64(len(pfxs2)),
}
return
}
}
results[idx] = entryResult{err: fmt.Errorf("resolve AS%d: %w", entry.ASN, err)}
return
}
@@ -167,6 +202,13 @@ func collectCDNPrefixRows(ctx context.Context, st store.Backend, hc *http.Client
}
rows, err := fetchCDNSourceRows(ctx, st, hc, tenantID, moduleID, mod, src, priorSnapshot, now)
if err != nil {
if staleOnUpstreamError() {
if cached, ok := staleCDNPrefixes(st, tenantID, moduleID, priorSnapshot, src.ID); ok {
logStaleUpstream("cdn", fmt.Sprintf("source %s: %v", src.ID, err))
results[idx] = srcResult{rows: cached}
return
}
}
results[idx] = srcResult{err: err}
return
}
@@ -190,7 +232,7 @@ func collectCDNPrefixRows(ctx context.Context, st store.Backend, hc *http.Client
return out, nil
}
func collectDomainPrefixRows(ctx context.Context, hc *http.Client, mod *store.Module, profiles []*store.DohProfile, policy string, entries []*store.DomainEntry) ([]store.PrefixRow, error) {
func collectDomainPrefixRows(ctx context.Context, hc *http.Client, mod *store.Module, profiles []*store.DohProfile, policy string, entries []*store.DomainEntry, priorSnapshot []store.PrefixRow) ([]store.PrefixRow, error) {
var validDom []*store.DomainEntry
for _, e := range entries {
if e != nil {
@@ -219,6 +261,19 @@ func collectDomainPrefixRows(ctx context.Context, hc *http.Client, mod *store.Mo
}
addrs, err := resolveDomainIPsWithPolicy(ctx, hc, profiles, policy, entry.FQDN)
if err != nil {
if staleOnUpstreamError() {
if cached, ok := staleDomainPrefixes(priorSnapshot, entry.FQDN); ok {
logStaleUpstream("domain", fmt.Sprintf("%q: %v", entry.FQDN, err))
rows := append([]store.PrefixRow(nil), cached...)
for i := range rows {
if rows[i].CommunityID == nil {
rows[i].CommunityID = comm
}
}
results[idx] = domResult{rows: rows}
return
}
}
results[idx] = domResult{err: fmt.Errorf("resolve domain %q: %w", entry.FQDN, err)}
return
}
+86
View File
@@ -0,0 +1,86 @@
package pipeline
import (
"fmt"
"log"
"net/netip"
"os"
"strings"
"evobgp/internal/store"
)
// staleOnUpstreamError reports whether ingest should keep last-known prefixes when an upstream fetch fails.
// Enabled by default; set EVOBGP_STALE_ON_UPSTREAM_ERROR=0 to restore fail-fast behavior.
func staleOnUpstreamError() bool {
v := strings.TrimSpace(os.Getenv("EVOBGP_STALE_ON_UPSTREAM_ERROR"))
if v == "" || v == "1" || strings.EqualFold(v, "true") {
return true
}
return false
}
func logStaleUpstream(kind, detail string) {
log.Printf("pipeline: stale upstream fallback (%s): %s", kind, detail)
}
func staleASNPrefixes(st store.Backend, priorSnapshot []store.PrefixRow, asn int64) ([]store.PrefixRow, string, bool) {
sourceKey := fmt.Sprintf("as:%d", asn)
if cached := prefixRowsForSource(priorSnapshot, sourceKey); len(cached) > 0 {
return cached, "", true
}
if st == nil {
return nil, "", false
}
ent, ok, err := st.GetASNPrefixCache(asn)
if err != nil || !ok || ent == nil || len(ent.Prefixes) == 0 {
return nil, "", false
}
var rows []store.PrefixRow
for _, p := range ent.Prefixes {
pfx, perr := netip.ParsePrefix(strings.TrimSpace(p))
if perr != nil {
continue
}
rows = append(rows, store.PrefixRow{Prefix: pfx.Masked().String(), Source: sourceKey})
}
if len(rows) == 0 {
return nil, "", false
}
return rows, ent.Holder, true
}
func staleDomainPrefixes(priorSnapshot []store.PrefixRow, fqdn string) ([]store.PrefixRow, bool) {
sourceKey := "domain:" + strings.TrimSpace(fqdn)
cached := prefixRowsForSource(priorSnapshot, sourceKey)
return cached, len(cached) > 0
}
func staleCDNPrefixes(st store.Backend, tenantID, moduleID string, priorSnapshot []store.PrefixRow, sourceID string) ([]store.PrefixRow, bool) {
sourceKey := cdnSourceKey(sourceID)
cached := cachedCDNPrefixRows(st, tenantID, moduleID, priorSnapshot, sourceKey)
return cached, len(cached) > 0
}
// asnCacheExpired returns cached ASN prefixes even past TTL (for stale fallback only).
func asnCacheExpired(st store.Backend, asn int64) ([]netip.Prefix, string, bool) {
if st == nil {
return nil, "", false
}
ent, ok, err := st.GetASNPrefixCache(asn)
if err != nil || !ok || ent == nil || len(ent.Prefixes) == 0 {
return nil, "", false
}
out := make([]netip.Prefix, 0, len(ent.Prefixes))
for _, p := range ent.Prefixes {
pfx, perr := netip.ParsePrefix(strings.TrimSpace(p))
if perr != nil {
continue
}
out = append(out, pfx.Masked())
}
if len(out) == 0 {
return nil, "", false
}
return out, ent.Holder, true
}
+72
View File
@@ -0,0 +1,72 @@
package pipeline
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"evobgp/internal/store"
)
func TestCollectCDNPrefixRows_StaleOnFetchError(t *testing.T) {
t.Setenv("EVOBGP_STALE_ON_UPSTREAM_ERROR", "1")
t.Setenv("EVOBGP_CDN_ALLOW_PRIVATE", "1")
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "upstream down", http.StatusServiceUnavailable)
}))
defer srv.Close()
st := store.NewMemory()
st.SeedDemo()
tenant, _, _, _, _ := st.DemoIDs()
mod, err := st.CreateModule(tenant, &store.Module{Type: "CDN_CIDRS", Name: "cdn", Enabled: true})
if err != nil {
t.Fatal(err)
}
if _, err := st.CreateCDNSource(tenant, mod.ID, &store.CDNSource{
ID: "s1", URL: srv.URL, SourceKind: "plain",
}); err != nil {
t.Fatal(err)
}
prior := []store.PrefixRow{
{Prefix: "203.0.113.0/24", Source: "cdn:s1"},
}
rows, err := collectCDNPrefixRows(context.Background(), st, srv.Client(), tenant, mod, []*store.CDNSource{{ID: "s1", URL: srv.URL, SourceKind: "plain"}}, prior)
if err != nil {
t.Fatalf("expected stale fallback, got err: %v", err)
}
if len(rows) != 1 || rows[0].Prefix != "203.0.113.0/24" {
t.Fatalf("unexpected rows: %+v", rows)
}
}
func TestCollectCDNPrefixRows_FailFastWhenNoStale(t *testing.T) {
t.Setenv("EVOBGP_STALE_ON_UPSTREAM_ERROR", "0")
t.Setenv("EVOBGP_CDN_ALLOW_PRIVATE", "1")
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "upstream down", http.StatusServiceUnavailable)
}))
defer srv.Close()
st := store.NewMemory()
st.SeedDemo()
tenant, _, _, _, _ := st.DemoIDs()
mod, err := st.CreateModule(tenant, &store.Module{Type: "CDN_CIDRS", Name: "cdn", Enabled: true})
if err != nil {
t.Fatal(err)
}
if _, err := st.CreateCDNSource(tenant, mod.ID, &store.CDNSource{
ID: "s1", URL: srv.URL, SourceKind: "plain",
}); err != nil {
t.Fatal(err)
}
_, err = collectCDNPrefixRows(context.Background(), st, srv.Client(), tenant, mod, []*store.CDNSource{{ID: "s1", URL: srv.URL, SourceKind: "plain"}}, nil)
if err == nil {
t.Fatal("expected error when stale disabled and no cache")
}
}
+84 -54
View File
@@ -5,86 +5,116 @@ import (
"io"
"net/http"
"strings"
"sync"
"time"
"evobgp/internal/httpclient"
"evobgp/internal/store"
)
type prefetchTask struct {
tenantID string
mod *store.Module
src *store.CDNSource
}
// PrefetchCDNSourceETags performs conditional GETs for CDN sources; on 200 parses CIDRs into module_prefix_snapshot.
func PrefetchCDNSourceETags(ctx context.Context, st store.Backend, hc *http.Client) error {
if hc == nil {
hc = http.DefaultClient
hc = httpclient.New(httpclient.DefaultTimeout)
}
if ctx == nil {
ctx = context.Background()
}
tenants, err := st.ListTenantIDs()
if err != nil {
return err
}
now := time.Now().UTC()
var tasks []prefetchTask
for _, tid := range tenants {
for _, mod := range st.ListModules(tid) {
if mod == nil || !mod.Enabled || mod.Type != "CDN_CIDRS" {
continue
}
omod, err := st.GetModule(tid, mod.ID)
if err != nil {
continue
}
sources, err := st.ListCDNSources(tid, mod.ID)
if err != nil {
continue
}
var prior []store.PrefixRow
if snap, ok, _ := st.GetModulePrefixSnapshot(tid, mod.ID); ok && snap != nil {
prior = snap.Prefixes
}
for _, src := range sources {
if src == nil || strings.TrimSpace(src.URL) == "" {
continue
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimSpace(src.URL), nil)
if err != nil {
continue
}
if etag := strings.TrimSpace(src.Etag); etag != "" {
req.Header.Set("If-None-Match", etag)
}
resp, err := hc.Do(req)
if err != nil {
continue
}
if resp.StatusCode == http.StatusNotModified {
_ = resp.Body.Close()
continue
}
if resp.StatusCode != http.StatusOK {
_, _ = io.Copy(io.Discard, resp.Body)
_ = resp.Body.Close()
continue
}
body, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
_ = resp.Body.Close()
if err != nil {
continue
}
prefixStrs, err := parseCDNBody(string(body), src)
if err != nil {
continue
}
newEtag := strings.TrimSpace(resp.Header.Get("ETag"))
patch := &store.CDNSourcePatch{LastRefreshedAt: &now}
if newEtag != "" && newEtag != strings.TrimSpace(src.Etag) {
e := newEtag
patch.Etag = &e
}
_, _ = st.UpdateCDNSource(tid, mod.ID, src.ID, patch)
rows := cdnRowsFromParsed(omod, src, prefixStrs)
_ = mergeCDNSourceIntoModuleSnapshot(st, tid, omod, src.ID, rows)
_ = prior // prior may be stale after merge; refresh for next source in loop
if snap, ok, _ := st.GetModulePrefixSnapshot(tid, mod.ID); ok && snap != nil {
prior = snap.Prefixes
if src != nil && strings.TrimSpace(src.URL) != "" {
tasks = append(tasks, prefetchTask{tenantID: tid, mod: mod, src: src})
}
}
}
}
if len(tasks) == 0 {
return nil
}
sem := make(chan struct{}, collectConcurrency())
var wg sync.WaitGroup
for _, task := range tasks {
wg.Add(1)
go func(t prefetchTask) {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
prefetchOneCDNSource(ctx, st, hc, t)
}(task)
}
wg.Wait()
return nil
}
func prefetchOneCDNSource(ctx context.Context, st store.Backend, hc *http.Client, t prefetchTask) {
now := time.Now().UTC()
tid, mod, src := t.tenantID, t.mod, t.src
u := strings.TrimSpace(src.URL)
if _, err := ValidateCDNURL(u); err != nil {
return
}
if err := ResolveCDNURLHost(ctx, u); err != nil {
return
}
omod, err := st.GetModule(tid, mod.ID)
if err != nil {
return
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
if err != nil {
return
}
if etag := strings.TrimSpace(src.Etag); etag != "" {
req.Header.Set("If-None-Match", etag)
}
resp, err := upstreamHTTPDo(ctx, hc, req)
if err != nil {
return
}
if resp.StatusCode == http.StatusNotModified {
_ = resp.Body.Close()
return
}
if resp.StatusCode != http.StatusOK {
_, _ = io.Copy(io.Discard, resp.Body)
_ = resp.Body.Close()
return
}
body, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
_ = resp.Body.Close()
if err != nil {
return
}
prefixStrs, err := parseCDNBody(string(body), src)
if err != nil {
return
}
newEtag := strings.TrimSpace(resp.Header.Get("ETag"))
patch := &store.CDNSourcePatch{LastRefreshedAt: &now}
if newEtag != "" && newEtag != strings.TrimSpace(src.Etag) {
e := newEtag
patch.Etag = &e
}
_, _ = st.UpdateCDNSource(tid, mod.ID, src.ID, patch)
rows := cdnRowsFromParsed(omod, src, prefixStrs)
_ = mergeCDNSourceIntoModuleSnapshot(st, tid, omod, src.ID, rows)
}
+7 -17
View File
@@ -19,6 +19,7 @@ import (
"time"
"evobgp/internal/birdfmt"
"evobgp/internal/httpclient"
"evobgp/internal/observability"
"evobgp/internal/store"
@@ -50,7 +51,7 @@ func MaterializedASPrefixKey(asn int64) string {
// It does not create a new config revision.
func RefreshModuleIngest(ctx context.Context, st store.Backend, hc *http.Client, tenantID, moduleID string) error {
if hc == nil {
hc = http.DefaultClient
hc = httpclient.New(httpclient.DefaultTimeout)
}
start := time.Now()
mod, err := st.GetModule(tenantID, moduleID)
@@ -84,7 +85,7 @@ func RefreshModuleIngest(ctx context.Context, st store.Backend, hc *http.Client,
// If materialized prefixes are unchanged, returns latest revision id without creating a duplicate.
func RenderTenantRevision(ctx context.Context, st store.Backend, hc *http.Client, tenantID, triggerModuleID string) (revisionID string, err error) {
if hc == nil {
hc = http.DefaultClient
hc = httpclient.New(httpclient.DefaultTimeout)
}
agg, err := aggregateTenantPrefixRowsAll(ctx, st, hc, tenantID)
if err != nil {
@@ -117,7 +118,7 @@ func RenderTenantRevision(ctx context.Context, st store.Backend, hc *http.Client
func RenderTenantRevisionFromPrefixes(ctx context.Context, st store.Backend, hc *http.Client, tenantID, triggerModuleID string, rows []store.PrefixRow) (revisionID string, err error) {
_ = ctx
if hc == nil {
hc = http.DefaultClient
hc = httpclient.New(httpclient.DefaultTimeout)
}
agg := append([]store.PrefixRow(nil), rows...)
rawCount := len(agg)
@@ -176,7 +177,7 @@ func collectModulePrefixRows(ctx context.Context, st store.Backend, hc *http.Cli
return nil, err
}
sort.Slice(list, func(i, j int) bool { return list[i].ASN < list[j].ASN })
return collectASPrefixRows(ctx, st, hc, tenantID, mod, list)
return collectASPrefixRows(ctx, st, hc, tenantID, mod, list, priorSnapshot)
case "CDN_CIDRS":
sources, err := st.ListCDNSources(tenantID, moduleID)
if err != nil {
@@ -192,7 +193,7 @@ func collectModulePrefixRows(ctx context.Context, st store.Backend, hc *http.Cli
if err != nil {
return nil, err
}
return collectDomainPrefixRows(ctx, hc, mod, profiles, policy, entries)
return collectDomainPrefixRows(ctx, hc, mod, profiles, policy, entries, priorSnapshot)
default:
return nil, fmt.Errorf("pipeline: unknown module type %q", mod.Type)
}
@@ -1029,7 +1030,7 @@ func renderPeersBirdFragment(st store.Backend, tenantID string, loc birdLocals)
}
pol := parsePeerPolicies(p.PoliciesJSON)
lv4, lv6, asn := effectivePeerLocals(loc, pol)
proto := peerProtocolName(p.ID)
proto := birdfmt.PeerProtocolName(p.ID)
ra := uint32(p.RemoteASN)
if addr.Is4() {
opts := birdfmt.BGPPeerFromTemplateOptions{
@@ -1083,17 +1084,6 @@ func parsePeerPolicies(raw string) peerPolicyJSON {
return pol
}
func peerProtocolName(peerID string) string {
s := strings.ReplaceAll(strings.TrimSpace(peerID), "-", "")
if len(s) > 16 {
s = s[:16]
}
if s == "" {
s = "x"
}
return "evobgp_p_" + s
}
// buildExpandedBirdText concatenates bird.conf and the contents of each standard include (for UI / preview).
func buildExpandedBirdText(main string, frags map[string]string) string {
var b strings.Builder
+2 -1
View File
@@ -7,13 +7,14 @@ import (
"strings"
"sync"
"evobgp/internal/httpclient"
"evobgp/internal/store"
)
// RefreshTenantModules ingests all listed modules in parallel and updates per-module snapshots.
func RefreshTenantModules(ctx context.Context, st store.Backend, hc *http.Client, tenantID string, moduleIDs []string) error {
if hc == nil {
hc = http.DefaultClient
hc = httpclient.New(httpclient.DefaultTimeout)
}
var ids []string
seen := make(map[string]struct{})
+23
View File
@@ -0,0 +1,23 @@
package pipeline
import (
"context"
"fmt"
"net/http"
"evobgp/internal/httpclient"
)
func upstreamHTTPDo(ctx context.Context, hc *http.Client, req *http.Request) (*http.Response, error) {
if hc == nil {
hc = httpclient.New(httpclient.DefaultTimeout)
}
resp, err := httpclient.DoWithBreaker(ctx, hc, req, 3)
if err != nil {
if req.URL != nil {
return nil, fmt.Errorf("cdn fetch %s: %w", req.URL.String(), err)
}
return nil, err
}
return resp, nil
}
+61 -14
View File
@@ -11,14 +11,22 @@ import (
"github.com/jackc/pgx/v5"
)
func asnCacheRowTableExists(ctx context.Context, q queryRower) bool {
var n int
err := q.QueryRow(ctx, `
SELECT 1 FROM information_schema.tables
WHERE table_schema = 'public' AND table_name = 'asn_prefix_cache_row'
LIMIT 1`).Scan(&n)
return err == nil
}
func (p *Postgres) GetASNPrefixCache(asn int64) (*store.ASNPrefixCacheEntry, bool, error) {
ctx := context.Background()
var holder string
var fetchedAt time.Time
var raw []byte
err := p.pool.QueryRow(ctx, `
SELECT holder, fetched_at, prefixes_json FROM asn_prefix_cache WHERE asn = $1`, asn).
Scan(&holder, &fetchedAt, &raw)
SELECT holder, fetched_at FROM asn_prefix_cache WHERE asn = $1`, asn).
Scan(&holder, &fetchedAt)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return nil, false, nil
@@ -26,8 +34,25 @@ func (p *Postgres) GetASNPrefixCache(asn int64) (*store.ASNPrefixCacheEntry, boo
return nil, false, err
}
var prefixes []string
if len(raw) > 0 {
_ = json.Unmarshal(raw, &prefixes)
if asnCacheRowTableExists(ctx, p.pool) {
rows, qerr := p.pool.Query(ctx, `
SELECT prefix::text FROM asn_prefix_cache_row WHERE asn = $1 ORDER BY prefix`, asn)
if qerr != nil {
return nil, false, qerr
}
defer rows.Close()
for rows.Next() {
var s string
if err := rows.Scan(&s); err != nil {
continue
}
prefixes = append(prefixes, s)
}
} else {
var raw []byte
if err := p.pool.QueryRow(ctx, `SELECT prefixes_json FROM asn_prefix_cache WHERE asn = $1`, asn).Scan(&raw); err == nil && len(raw) > 0 {
_ = json.Unmarshal(raw, &prefixes)
}
}
return &store.ASNPrefixCacheEntry{
ASN: asn,
@@ -38,18 +63,40 @@ func (p *Postgres) GetASNPrefixCache(asn int64) (*store.ASNPrefixCacheEntry, boo
}
func (p *Postgres) SetASNPrefixCache(asn int64, holder string, prefixes []string) error {
raw, err := json.Marshal(prefixes)
ctx := context.Background()
tx, err := p.pool.Begin(ctx)
if err != nil {
return err
}
ctx := context.Background()
_, err = p.pool.Exec(ctx, `
INSERT INTO asn_prefix_cache (asn, holder, prefixes_json, fetched_at)
VALUES ($1, $2, $3::jsonb, now())
defer func() { _ = tx.Rollback(ctx) }()
_, err = tx.Exec(ctx, `
INSERT INTO asn_prefix_cache (asn, holder, fetched_at)
VALUES ($1, $2, now())
ON CONFLICT (asn) DO UPDATE SET
holder = EXCLUDED.holder,
prefixes_json = EXCLUDED.prefixes_json,
fetched_at = EXCLUDED.fetched_at`,
asn, holder, string(raw))
return err
fetched_at = EXCLUDED.fetched_at`, asn, holder)
if err != nil {
return err
}
if asnCacheRowTableExists(ctx, tx) {
if _, err := tx.Exec(ctx, `DELETE FROM asn_prefix_cache_row WHERE asn = $1`, asn); err != nil {
return err
}
for _, pfx := range prefixes {
if _, err := tx.Exec(ctx, `
INSERT INTO asn_prefix_cache_row (asn, prefix) VALUES ($1, $2::cidr)`, asn, pfx); err != nil {
return err
}
}
} else {
raw, err := json.Marshal(prefixes)
if err != nil {
return err
}
if _, err := tx.Exec(ctx, `
UPDATE asn_prefix_cache SET prefixes_json = $2::jsonb WHERE asn = $1`, asn, string(raw)); err != nil {
return err
}
}
return tx.Commit(ctx)
}
+19
View File
@@ -0,0 +1,19 @@
package repository
import (
"context"
"time"
)
const defaultRepoTimeout = 60 * time.Second
// boundedRepoCtx returns a context with default repository I/O timeout.
func boundedRepoCtx(parent context.Context) (context.Context, context.CancelFunc) {
if parent == nil {
parent = context.Background()
}
if _, ok := parent.Deadline(); ok {
return parent, func() {}
}
return context.WithTimeout(parent, defaultRepoTimeout)
}
+23 -2
View File
@@ -20,6 +20,28 @@ func NewJobAuditWriter(pool *pgxpool.Pool) *JobAuditWriter {
return &JobAuditWriter{pool: pool}
}
// UpsertQueued inserts a queued job row (best-effort).
func (w *JobAuditWriter) UpsertQueued(ctx context.Context, tenantID, jobID, kind string, idempotencyKey *string, moduleID *string, meta map[string]any) {
if w == nil || w.pool == nil {
return
}
metaJSON, _ := json.Marshal(meta)
var idem any
if idempotencyKey != nil && *idempotencyKey != "" {
idem = *idempotencyKey
}
var mod any
if moduleID != nil && *moduleID != "" {
mod = *moduleID
}
_, _ = w.pool.Exec(ctx, `
INSERT INTO job_audit (id, tenant_id, kind, status, idempotency_key, module_id, meta_json, created_at)
VALUES ($1::uuid, $2::uuid, $3, 'queued', $4, $5::uuid, $6::jsonb, now())
ON CONFLICT (tenant_id, idempotency_key) WHERE idempotency_key IS NOT NULL
DO UPDATE SET status='queued', meta_json=EXCLUDED.meta_json, module_id=EXCLUDED.module_id`,
jobID, tenantID, kind, idem, mod, metaJSON)
}
// UpsertRunning inserts or updates a running job row (best-effort).
func (w *JobAuditWriter) UpsertRunning(ctx context.Context, tenantID, jobID, kind string, idempotencyKey *string, meta map[string]any) {
if w == nil || w.pool == nil {
@@ -33,8 +55,7 @@ func (w *JobAuditWriter) UpsertRunning(ctx context.Context, tenantID, jobID, kin
_, _ = w.pool.Exec(ctx, `
INSERT INTO job_audit (id, tenant_id, kind, status, idempotency_key, meta_json, created_at, started_at)
VALUES ($1::uuid, $2::uuid, $3, 'running', $4, $5::jsonb, now(), now())
ON CONFLICT (tenant_id, idempotency_key) WHERE idempotency_key IS NOT NULL
DO UPDATE SET status='running', started_at=now(), meta_json=EXCLUDED.meta_json`,
ON CONFLICT (id) DO UPDATE SET status='running', started_at=COALESCE(job_audit.started_at, now()), meta_json=EXCLUDED.meta_json`,
jobID, tenantID, kind, idem, metaJSON)
}
+29
View File
@@ -0,0 +1,29 @@
package repository
import (
"context"
"time"
)
const (
jobAuditRetentionDays = 90
asnCacheRetentionDays = 7
)
// RunPeriodicMaintenance prunes stale job_audit and asn_prefix_cache rows (PostgreSQL).
func (p *Postgres) RunPeriodicMaintenance(ctx context.Context) {
if p == nil || p.pool == nil {
return
}
if ctx == nil {
ctx = context.Background()
}
jobCutoff := time.Now().UTC().Add(-time.Duration(jobAuditRetentionDays) * 24 * time.Hour)
_, _ = p.pool.Exec(ctx, `
DELETE FROM job_audit
WHERE created_at < $1
AND status IN ('succeeded', 'failed', 'cancelled')`, jobCutoff)
asnCutoff := time.Now().UTC().Add(-time.Duration(asnCacheRetentionDays) * 24 * time.Hour)
_, _ = p.pool.Exec(ctx, `
DELETE FROM asn_prefix_cache WHERE fetched_at < $1`, asnCutoff)
}
+82 -15
View File
@@ -12,16 +12,24 @@ import (
"github.com/jackc/pgx/v5"
)
func moduleSnapshotRowTableExists(ctx context.Context, q queryRower) bool {
var n int
err := q.QueryRow(ctx, `
SELECT 1 FROM information_schema.tables
WHERE table_schema = 'public' AND table_name = 'module_prefix_snapshot_row'
LIMIT 1`).Scan(&n)
return err == nil
}
func (p *Postgres) GetModulePrefixSnapshot(tenantID, moduleID string) (*store.ModulePrefixSnapshot, bool, error) {
ctx := context.Background()
var inputHash string
var collectedAt time.Time
var raw []byte
err := p.pool.QueryRow(ctx, `
SELECT input_hash, collected_at, prefixes_json
SELECT input_hash, collected_at
FROM module_prefix_snapshot
WHERE tenant_id = $1 AND module_id = $2`,
tenantID, moduleID).Scan(&inputHash, &collectedAt, &raw)
tenantID, moduleID).Scan(&inputHash, &collectedAt)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return nil, false, nil
@@ -29,9 +37,31 @@ func (p *Postgres) GetModulePrefixSnapshot(tenantID, moduleID string) (*store.Mo
return nil, false, err
}
var prefixes []store.PrefixRow
if len(raw) > 0 {
if err := json.Unmarshal(raw, &prefixes); err != nil {
return nil, false, err
if moduleSnapshotRowTableExists(ctx, p.pool) {
rows, qerr := p.pool.Query(ctx, `
SELECT prefix::text, community_id::text, source
FROM module_prefix_snapshot_row
WHERE tenant_id = $1::uuid AND module_id = $2::uuid
ORDER BY ord`, tenantID, moduleID)
if qerr != nil {
return nil, false, qerr
}
defer rows.Close()
for rows.Next() {
var pr store.PrefixRow
var comm *string
if err := rows.Scan(&pr.Prefix, &comm, &pr.Source); err != nil {
continue
}
pr.CommunityID = comm
prefixes = append(prefixes, pr)
}
} else {
var raw []byte
if err := p.pool.QueryRow(ctx, `
SELECT prefixes_json FROM module_prefix_snapshot
WHERE tenant_id = $1 AND module_id = $2`, tenantID, moduleID).Scan(&raw); err == nil && len(raw) > 0 {
_ = json.Unmarshal(raw, &prefixes)
}
}
return &store.ModulePrefixSnapshot{
@@ -45,20 +75,57 @@ func (p *Postgres) SetModulePrefixSnapshot(tenantID, moduleID, inputHash string,
if strings.TrimSpace(tenantID) == "" || strings.TrimSpace(moduleID) == "" || strings.TrimSpace(inputHash) == "" {
return store.ErrInvalidInput
}
raw, err := json.Marshal(prefixes)
ctx := context.Background()
tx, err := p.pool.Begin(ctx)
if err != nil {
return err
}
ctx := context.Background()
_, err = p.pool.Exec(ctx, `
INSERT INTO module_prefix_snapshot (tenant_id, module_id, input_hash, collected_at, prefixes_json)
VALUES ($1::uuid, $2::uuid, $3, now(), $4::jsonb)
defer func() { _ = tx.Rollback(ctx) }()
_, err = tx.Exec(ctx, `
INSERT INTO module_prefix_snapshot (tenant_id, module_id, input_hash, collected_at)
VALUES ($1::uuid, $2::uuid, $3, now())
ON CONFLICT (tenant_id, module_id) DO UPDATE SET
input_hash = EXCLUDED.input_hash,
collected_at = EXCLUDED.collected_at,
prefixes_json = EXCLUDED.prefixes_json`,
tenantID, moduleID, inputHash, string(raw))
return err
collected_at = EXCLUDED.collected_at`,
tenantID, moduleID, inputHash)
if err != nil {
return err
}
if moduleSnapshotRowTableExists(ctx, tx) {
if _, err := tx.Exec(ctx, `
DELETE FROM module_prefix_snapshot_row
WHERE tenant_id = $1::uuid AND module_id = $2::uuid`, tenantID, moduleID); err != nil {
return err
}
for i, pr := range prefixes {
var comm any
if pr.CommunityID != nil && strings.TrimSpace(*pr.CommunityID) != "" {
comm = strings.TrimSpace(*pr.CommunityID)
}
src := pr.Source
if strings.TrimSpace(src) == "" {
src = "render"
}
if _, err := tx.Exec(ctx, `
INSERT INTO module_prefix_snapshot_row (tenant_id, module_id, ord, prefix, community_id, source)
VALUES ($1::uuid, $2::uuid, $3, $4::cidr, $5::uuid, $6)`,
tenantID, moduleID, i, strings.TrimSpace(pr.Prefix), comm, src); err != nil {
return err
}
}
} else {
raw, err := json.Marshal(prefixes)
if err != nil {
return err
}
if _, err := tx.Exec(ctx, `
UPDATE module_prefix_snapshot SET prefixes_json = $3::jsonb
WHERE tenant_id = $1::uuid AND module_id = $2::uuid`,
tenantID, moduleID, string(raw)); err != nil {
return err
}
}
return tx.Commit(ctx)
}
func (p *Postgres) DeleteModulePrefixSnapshot(tenantID, moduleID string) error {
+118 -67
View File
@@ -646,8 +646,21 @@ func (p *Postgres) UpdateSpeaker(tenantID, id string, patch *store.SpeakerPatch)
return p.GetSpeaker(tenantID, id)
}
func (p *Postgres) GetRevision(tenantID, revisionID string) (*store.Revision, error) {
func (p *Postgres) DeleteSpeaker(tenantID, id string) error {
ctx := context.Background()
tag, err := p.pool.Exec(ctx, `DELETE FROM bgp_speaker WHERE id=$1 AND tenant_id=$2`, id, tenantID)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return store.ErrNotFound
}
return nil
}
func (p *Postgres) GetRevision(tenantID, revisionID string) (*store.Revision, error) {
ctx, cancel := boundedRepoCtx(context.Background())
defer cancel()
var r store.Revision
var mod *string
var parent *string
@@ -674,11 +687,38 @@ func (p *Postgres) GetRevision(tenantID, revisionID string) (*store.Revision, er
if mj.PreviewFragments == nil {
mj.PreviewFragments = map[string]string{}
}
r.PreviewFragments = mj.PreviewFragments
r.PreviewFragments = loadRevisionPreview(ctx, p.pool, revisionID, mj.PreviewFragments)
r.MaterializedPrefixCount = mj.MaterializedPrefixCount
return &r, nil
}
func (p *Postgres) GetRevisionSummary(tenantID, revisionID string) (*store.Revision, error) {
ctx, cancel := boundedRepoCtx(context.Background())
defer cancel()
var r store.Revision
var mod *string
var parent *string
var prefixCount int
err := p.pool.QueryRow(ctx, `
SELECT id::text, tenant_id::text, module_id::text, content_hash, parent_revision_id::text,
COALESCE((meta_json->>'materialized_prefix_count')::int, 0), created_at
FROM config_revision WHERE id=$1 AND tenant_id=$2`, revisionID, tenantID).Scan(
&r.ID, &r.TenantID, &mod, &r.ContentHash, &parent, &prefixCount, &r.CreatedAt)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return nil, store.ErrNotFound
}
return nil, err
}
if mod != nil {
r.ModuleID = *mod
}
r.ParentRevisionID = strOrNil(parent)
r.MaterializedPrefixCount = prefixCount
r.PreviewFragments = map[string]string{}
return &r, nil
}
func (p *Postgres) ListRevisions(tenantID, moduleID string, cursor string, limit int) ([]*store.Revision, string, bool) {
if limit <= 0 {
limit = 50
@@ -749,12 +789,7 @@ func (p *Postgres) ListRevisionPrefixes(tenantID, revisionID string, cursor stri
if limit <= 0 {
limit = 50
}
off := 0
if cursor != "" {
if n, err := strconv.Atoi(cursor); err == nil && n >= 0 {
off = n
}
}
afterID, off, useOffset := store.ParsePrefixPageCursor(cursor)
ctx := context.Background()
var one int
if err := p.pool.QueryRow(ctx, `
@@ -765,34 +800,55 @@ func (p *Postgres) ListRevisionPrefixes(tenantID, revisionID string, cursor stri
}
return nil, "", false
}
rows, err := p.pool.Query(ctx, `
SELECT prefix::text, community_id::text, source FROM revision_materialized_prefix
WHERE revision_id=$1 ORDER BY id
LIMIT $2 OFFSET $3`, revisionID, limit+1, off)
if snapID, ok := p.revisionPrefixSnapshotID(ctx, revisionID); ok {
return p.listSnapshotPrefixes(ctx, snapID, cursor, limit)
}
var rows pgx.Rows
var err error
if useOffset {
rows, err = p.pool.Query(ctx, `
SELECT id, prefix::text, community_id::text, source FROM revision_materialized_prefix
WHERE revision_id=$1::uuid ORDER BY id
LIMIT $2 OFFSET $3`, revisionID, limit+1, off)
} else {
var afterArg any
if afterID != nil {
afterArg = *afterID
}
rows, err = p.pool.Query(ctx, `
SELECT id, prefix::text, community_id::text, source FROM revision_materialized_prefix
WHERE revision_id=$1::uuid AND ($2::bigint IS NULL OR id > $2::bigint)
ORDER BY id
LIMIT $3`, revisionID, afterArg, limit+1)
}
if err != nil {
return nil, "", false
}
defer rows.Close()
var all []store.PrefixRow
var ids []int64
for rows.Next() {
var rowID int64
var pr store.PrefixRow
var comm *string
if err := rows.Scan(&pr.Prefix, &comm, &pr.Source); err != nil {
if err := rows.Scan(&rowID, &pr.Prefix, &comm, &pr.Source); err != nil {
continue
}
pr.CommunityID = comm
ids = append(ids, rowID)
all = append(all, pr)
}
agentDebugNDJSON3214("B", "repository/postgres.go:ListRevisionPrefixes", "list_prefixes_fetched", map[string]any{
"rows": len(all), "limit": limit, "offset": off,
"rows": len(all), "limit": limit, "keyset": !useOffset,
})
more := len(all) > limit
if more {
all = all[:limit]
ids = ids[:limit]
}
next := ""
if more {
next = fmt.Sprintf("%d", off+limit)
if more && len(ids) > 0 {
next = store.FormatPrefixPageCursor(ids[len(ids)-1])
}
if len(all) == 0 {
return nil, "", false
@@ -808,26 +864,35 @@ func (p *Postgres) CreateRollbackRevision(tenantID, sourceRevisionID string) (st
ctx := context.Background()
newID := uuid.NewString()
parent := sourceRevisionID
meta, _ := json.Marshal(map[string]any{
"preview_fragments": src.PreviewFragments,
"materialized_prefix_count": src.MaterializedPrefixCount,
})
meta, err := revisionMetaWithoutPreview(src.MaterializedPrefixCount)
if err != nil {
return "", err
}
var modArg any
if strings.TrimSpace(src.ModuleID) != "" {
modArg = src.ModuleID
}
_, err = p.pool.Exec(ctx, `
INSERT INTO config_revision (id, tenant_id, module_id, content_hash, parent_revision_id, meta_json)
VALUES ($1,$2,$3,$4,$5::uuid,$6::jsonb)`,
newID, tenantID, modArg, src.ContentHash+":rollback", parent, string(meta))
tx, err := p.pool.Begin(ctx)
if err != nil {
return "", err
}
// copy materialized prefixes
_, _ = p.pool.Exec(ctx, `
INSERT INTO revision_materialized_prefix (revision_id, prefix, community_id, source, meta_json)
SELECT $1::uuid, prefix, community_id, source, meta_json FROM revision_materialized_prefix WHERE revision_id=$2::uuid`,
newID, sourceRevisionID)
defer func() { _ = tx.Rollback(ctx) }()
_, err = tx.Exec(ctx, `
INSERT INTO config_revision (id, tenant_id, module_id, content_hash, parent_revision_id, meta_json)
VALUES ($1,$2,$3,$4,$5::uuid,$6::jsonb)`,
newID, tenantID, modArg, src.ContentHash+":rollback", parent, meta)
if err != nil {
return "", err
}
if err := copyRevisionPreview(ctx, tx, newID, sourceRevisionID); err != nil {
return "", err
}
if err := p.copyRevisionPrefixSnapshotRef(ctx, tx, newID, sourceRevisionID); err != nil {
return "", err
}
if err := tx.Commit(ctx); err != nil {
return "", err
}
return newID, nil
}
@@ -844,20 +909,18 @@ func (p *Postgres) RevisionDiff(tenantID, aID, bID string) (map[string]any, erro
var unchanged int
err := p.pool.QueryRow(ctx, `
SELECT COUNT(*)::int FROM (
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$1::uuid
INTERSECT
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$2::uuid
SELECT b.prefix FROM (`+sqlRevisionPrefixes("$2")+`) b
INNER JOIN (`+sqlRevisionPrefixes("$1")+`) a ON a.prefix = b.prefix
) t`, aID, bID).Scan(&unchanged)
if err != nil {
return nil, err
}
// added: в B, нет в A; removed: в A, нет в B — без загрузки полных снапшотов в память.
rowsAdded, err := p.pool.Query(ctx, `
SELECT prefix::text FROM (
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$1::uuid
EXCEPT
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$2::uuid
) s ORDER BY 1 LIMIT $3`, bID, aID, maxRevisionDiffRows+1)
SELECT b.prefix::text FROM (`+sqlRevisionPrefixes("$2")+`) b
LEFT JOIN (`+sqlRevisionPrefixes("$1")+`) a ON a.prefix = b.prefix
WHERE a.prefix IS NULL
ORDER BY b.prefix
LIMIT $3`, aID, bID, maxRevisionDiffRows+1)
if err != nil {
return nil, err
}
@@ -876,11 +939,11 @@ func (p *Postgres) RevisionDiff(tenantID, aID, bID string) (map[string]any, erro
}
addedTruncated := len(added) >= maxRevisionDiffRows
rowsRem, err := p.pool.Query(ctx, `
SELECT prefix::text FROM (
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$1::uuid
EXCEPT
SELECT prefix FROM revision_materialized_prefix WHERE revision_id=$2::uuid
) s ORDER BY 1 LIMIT $3`, aID, bID, maxRevisionDiffRows+1)
SELECT a.prefix::text FROM (`+sqlRevisionPrefixes("$1")+`) a
LEFT JOIN (`+sqlRevisionPrefixes("$2")+`) b ON b.prefix = a.prefix
WHERE b.prefix IS NULL
ORDER BY a.prefix
LIMIT $3`, aID, bID, maxRevisionDiffRows+1)
if err != nil {
return nil, err
}
@@ -1016,10 +1079,7 @@ func (p *Postgres) CreateRenderRevision(revisionID, tenantID, moduleID string, p
if previewFragments == nil {
previewFragments = map[string]string{}
}
meta, err := json.Marshal(map[string]any{
"preview_fragments": previewFragments,
"materialized_prefix_count": len(prefixes),
})
meta, err := revisionMetaWithoutPreview(len(prefixes))
if err != nil {
return err
}
@@ -1033,32 +1093,23 @@ func (p *Postgres) CreateRenderRevision(revisionID, tenantID, moduleID string, p
if parentRevisionID != nil && strings.TrimSpace(*parentRevisionID) != "" {
parent = strings.TrimSpace(*parentRevisionID)
}
revID := strings.TrimSpace(revisionID)
_, err = tx.Exec(ctx, `
INSERT INTO config_revision (id, tenant_id, module_id, content_hash, parent_revision_id, meta_json)
VALUES ($1::uuid, $2::uuid, $3::uuid, $4, $5::uuid, $6::jsonb)`,
strings.TrimSpace(revisionID), tenantID, moduleID, strings.TrimSpace(contentHash), parent, string(meta))
revID, tenantID, moduleID, strings.TrimSpace(contentHash), parent, meta)
if err != nil {
return err
}
if len(prefixes) > 0 {
_, err = tx.CopyFrom(ctx,
pgx.Identifier{"revision_materialized_prefix"},
[]string{"revision_id", "prefix", "community_id", "source"},
pgx.CopyFromSlice(len(prefixes), func(i int) ([]any, error) {
pr := prefixes[i]
var comm any
if pr.CommunityID != nil && strings.TrimSpace(*pr.CommunityID) != "" {
comm = strings.TrimSpace(*pr.CommunityID)
}
src := pr.Source
if strings.TrimSpace(src) == "" {
src = "render"
}
return []any{strings.TrimSpace(revisionID), strings.TrimSpace(pr.Prefix), comm, src}, nil
}))
if err != nil {
return err
}
if err := insertRevisionPreview(ctx, tx, revID, previewFragments); err != nil {
return err
}
snapID, err := p.ensurePrefixSnapshot(ctx, tx, contentHash, prefixes)
if err != nil {
return err
}
if err := p.linkRevisionPrefixSnapshot(ctx, tx, revID, snapID); err != nil {
return err
}
if err := tx.Commit(ctx); err != nil {
return err
@@ -0,0 +1,186 @@
package repository
import (
"context"
"errors"
"strings"
"evobgp/internal/store"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
func prefixSnapshotTableExists(ctx context.Context, q queryRower) bool {
var n int
err := q.QueryRow(ctx, `
SELECT 1 FROM information_schema.tables
WHERE table_schema = 'public' AND table_name = 'prefix_snapshot'
LIMIT 1`).Scan(&n)
return err == nil
}
func normalizeSnapshotHash(contentHash string) string {
h := strings.TrimSpace(contentHash)
if strings.HasPrefix(h, "sha256:") {
h = strings.TrimPrefix(h, "sha256:")
}
if len(h) > 64 {
h = h[:64]
}
if len(h) < 64 {
h = h + strings.Repeat("0", 64-len(h))
}
return h
}
func (p *Postgres) revisionPrefixSnapshotID(ctx context.Context, revisionID string) (string, bool) {
if !prefixSnapshotTableExists(ctx, p.pool) {
return "", false
}
var snap *string
err := p.pool.QueryRow(ctx, `
SELECT prefix_snapshot_id::text FROM config_revision
WHERE id = $1::uuid AND prefix_snapshot_id IS NOT NULL`, revisionID).Scan(&snap)
if err != nil || snap == nil || strings.TrimSpace(*snap) == "" {
return "", false
}
return *snap, true
}
func (p *Postgres) ensurePrefixSnapshot(ctx context.Context, db execQuerier, contentHash string, prefixes []store.PrefixRow) (string, error) {
if !prefixSnapshotTableExists(ctx, db) {
return "", nil
}
hash := normalizeSnapshotHash(contentHash)
var existing string
err := db.QueryRow(ctx, `SELECT id::text FROM prefix_snapshot WHERE content_hash = $1`, hash).Scan(&existing)
if err == nil && existing != "" {
return existing, nil
}
if err != nil && !errors.Is(err, pgx.ErrNoRows) {
return "", err
}
snapID := uuid.NewString()
if _, err := db.Exec(ctx, `
INSERT INTO prefix_snapshot (id, content_hash) VALUES ($1::uuid, $2)
ON CONFLICT (content_hash) DO NOTHING`, snapID, hash); err != nil {
return "", err
}
if err := db.QueryRow(ctx, `SELECT id::text FROM prefix_snapshot WHERE content_hash = $1`, hash).Scan(&snapID); err != nil {
return "", err
}
var rowCount int
_ = db.QueryRow(ctx, `SELECT COUNT(*)::int FROM prefix_snapshot_row WHERE snapshot_id = $1::uuid`, snapID).Scan(&rowCount)
if rowCount > 0 {
return snapID, nil
}
for i, pr := range prefixes {
var comm any
if pr.CommunityID != nil && strings.TrimSpace(*pr.CommunityID) != "" {
comm = strings.TrimSpace(*pr.CommunityID)
}
src := pr.Source
if strings.TrimSpace(src) == "" {
src = "render"
}
if _, err := db.Exec(ctx, `
INSERT INTO prefix_snapshot_row (snapshot_id, ord, prefix, community_id, source)
VALUES ($1::uuid, $2, $3::cidr, $4::uuid, $5)`,
snapID, i, strings.TrimSpace(pr.Prefix), comm, src); err != nil {
return "", err
}
}
return snapID, nil
}
func (p *Postgres) listSnapshotPrefixes(ctx context.Context, snapshotID, cursor string, limit int) ([]store.PrefixRow, string, bool) {
afterOrd, off, useOffset := store.ParsePrefixPageCursor(cursor)
var rows pgx.Rows
var err error
if useOffset {
rows, err = p.pool.Query(ctx, `
SELECT ord, prefix::text, community_id::text, source
FROM prefix_snapshot_row
WHERE snapshot_id = $1::uuid
ORDER BY ord
LIMIT $2 OFFSET $3`, snapshotID, limit+1, off)
} else {
var afterArg any
if afterOrd != nil {
afterArg = int(*afterOrd)
}
rows, err = p.pool.Query(ctx, `
SELECT ord, prefix::text, community_id::text, source
FROM prefix_snapshot_row
WHERE snapshot_id = $1::uuid AND ($2::int IS NULL OR ord > $2::int)
ORDER BY ord
LIMIT $3`, snapshotID, afterArg, limit+1)
}
if err != nil {
return nil, "", false
}
defer rows.Close()
var all []store.PrefixRow
var ords []int64
for rows.Next() {
var ord int
var pr store.PrefixRow
var comm *string
if err := rows.Scan(&ord, &pr.Prefix, &comm, &pr.Source); err != nil {
continue
}
pr.CommunityID = comm
ords = append(ords, int64(ord))
all = append(all, pr)
}
more := len(all) > limit
if more {
all = all[:limit]
ords = ords[:limit]
}
next := ""
if more && len(ords) > 0 {
next = store.FormatPrefixPageCursor(ords[len(ords)-1])
}
if len(all) == 0 {
return nil, "", false
}
return all, next, more
}
func (p *Postgres) linkRevisionPrefixSnapshot(ctx context.Context, db execQuerier, revisionID, snapshotID string) error {
if snapshotID == "" || !prefixSnapshotTableExists(ctx, db) {
return nil
}
_, err := db.Exec(ctx, `
UPDATE config_revision SET prefix_snapshot_id = $2::uuid WHERE id = $1::uuid`,
revisionID, snapshotID)
return err
}
func (p *Postgres) copyRevisionPrefixSnapshotRef(ctx context.Context, db execQuerier, dstRevisionID, srcRevisionID string) error {
if !prefixSnapshotTableExists(ctx, db) {
return nil
}
_, err := db.Exec(ctx, `
UPDATE config_revision dst
SET prefix_snapshot_id = src.prefix_snapshot_id
FROM config_revision src
WHERE dst.id = $1::uuid AND src.id = $2::uuid AND src.prefix_snapshot_id IS NOT NULL`,
dstRevisionID, srcRevisionID)
return err
}
func sqlRevisionPrefixes(revParam string) string {
return `SELECT psr.prefix FROM config_revision cr
JOIN prefix_snapshot_row psr ON psr.snapshot_id = cr.prefix_snapshot_id
WHERE cr.id = ` + revParam + `::uuid AND cr.prefix_snapshot_id IS NOT NULL
UNION ALL
SELECT rmp.prefix FROM revision_materialized_prefix rmp
WHERE rmp.revision_id = ` + revParam + `::uuid
AND NOT EXISTS (
SELECT 1 FROM config_revision cr2
WHERE cr2.id = ` + revParam + `::uuid AND cr2.prefix_snapshot_id IS NOT NULL
)`
}
@@ -0,0 +1,90 @@
package repository
import (
"context"
"encoding/json"
"errors"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
)
func revisionPreviewTableExists(ctx context.Context, q queryRower) bool {
var n int
err := q.QueryRow(ctx, `
SELECT 1 FROM information_schema.tables
WHERE table_schema = 'public' AND table_name = 'config_revision_preview'
LIMIT 1`).Scan(&n)
return err == nil
}
type queryRower interface {
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
}
func loadRevisionPreview(ctx context.Context, q queryRower, revisionID string, metaPreview map[string]string) map[string]string {
if revisionPreviewTableExists(ctx, q) {
var raw []byte
err := q.QueryRow(ctx, `
SELECT fragments FROM config_revision_preview WHERE revision_id = $1::uuid`,
revisionID).Scan(&raw)
if err == nil {
out := map[string]string{}
_ = json.Unmarshal(raw, &out)
if out == nil {
out = map[string]string{}
}
return out
}
if !errors.Is(err, pgx.ErrNoRows) {
return metaPreview
}
}
if metaPreview == nil {
return map[string]string{}
}
return metaPreview
}
type execQuerier interface {
queryRower
Exec(ctx context.Context, sql string, arguments ...any) (pgconn.CommandTag, error)
}
func insertRevisionPreview(ctx context.Context, db execQuerier, revisionID string, preview map[string]string) error {
if !revisionPreviewTableExists(ctx, db) {
return nil
}
if preview == nil {
preview = map[string]string{}
}
raw, err := json.Marshal(preview)
if err != nil {
return err
}
_, err = db.Exec(ctx, `
INSERT INTO config_revision_preview (revision_id, fragments)
VALUES ($1::uuid, $2::jsonb)
ON CONFLICT (revision_id) DO UPDATE SET fragments = EXCLUDED.fragments`,
revisionID, string(raw))
return err
}
func copyRevisionPreview(ctx context.Context, db execQuerier, dstRevisionID, srcRevisionID string) error {
if !revisionPreviewTableExists(ctx, db) {
return nil
}
_, err := db.Exec(ctx, `
INSERT INTO config_revision_preview (revision_id, fragments)
SELECT $1::uuid, fragments FROM config_revision_preview WHERE revision_id = $2::uuid
ON CONFLICT (revision_id) DO UPDATE SET fragments = EXCLUDED.fragments`,
dstRevisionID, srcRevisionID)
return err
}
func revisionMetaWithoutPreview(materializedPrefixCount int) (string, error) {
raw, err := json.Marshal(map[string]any{
"materialized_prefix_count": materializedPrefixCount,
})
return string(raw), err
}
+30 -15
View File
@@ -26,9 +26,8 @@ func (p *Postgres) seedDemo(ctx context.Context) error {
p1 := uuid.NewString()
p2 := uuid.NewString()
preview := map[string]any{
"preview_fragments": map[string]string{
"bird.conf": `# EvoBGP demo bundle
previewFrags := map[string]string{
"bird.conf": `# EvoBGP demo bundle
router id 192.0.2.1;
protocol device {
@@ -39,15 +38,12 @@ protocol direct {
ipv6;
}
`,
"bird.d/evobgp_demo.conf": "# static demo fragment\n",
},
"materialized_prefix_count": 128,
"bird.d/evobgp_demo.conf": "# static demo fragment\n",
}
previewB, _ := json.Marshal(preview)
parentMeta, _ := json.Marshal(map[string]any{
"preview_fragments": map[string]string{"bird.conf": "# parent revision\n"},
"materialized_prefix_count": 0,
})
previewMeta, _ := json.Marshal(map[string]any{"materialized_prefix_count": 128})
parentMeta, _ := json.Marshal(map[string]any{"materialized_prefix_count": 0})
parentPreview, _ := json.Marshal(map[string]string{"bird.conf": "# parent revision\n"})
previewFragsB, _ := json.Marshal(previewFrags)
tx, err := p.pool.Begin(ctx)
if err != nil {
@@ -77,13 +73,32 @@ protocol direct {
return err
}
if _, err := tx.Exec(ctx, `
INSERT INTO config_revision (id, tenant_id, content_hash, parent_revision_id, meta_json)
VALUES ($1,$2,'sha256:demo-rev-1',$3::uuid,$4::jsonb)`, rid, tid, parent, string(previewB)); err != nil {
INSERT INTO config_revision_preview (revision_id, fragments) VALUES ($1::uuid, $2::jsonb)`, parent, string(parentPreview)); err != nil {
return err
}
if _, err := tx.Exec(ctx, `
INSERT INTO revision_materialized_prefix (revision_id, prefix, community_id, source)
VALUES ($1::uuid,'203.0.113.0/24',$2::uuid,'demo'), ($1::uuid,'2001:db8::/32',$2::uuid,'demo')`, rid, cid); err != nil {
INSERT INTO config_revision (id, tenant_id, content_hash, parent_revision_id, meta_json)
VALUES ($1,$2,'sha256:demo-rev-1',$3::uuid,$4::jsonb)`, rid, tid, parent, string(previewMeta)); err != nil {
return err
}
if _, err := tx.Exec(ctx, `
INSERT INTO config_revision_preview (revision_id, fragments) VALUES ($1::uuid, $2::jsonb)`, rid, string(previewFragsB)); err != nil {
return err
}
snapID := uuid.NewString()
demoHash := normalizeSnapshotHash("sha256:demo-rev-1")
if _, err := tx.Exec(ctx, `
INSERT INTO prefix_snapshot (id, content_hash) VALUES ($1::uuid, $2)`, snapID, demoHash); err != nil {
return err
}
if _, err := tx.Exec(ctx, `
UPDATE config_revision SET prefix_snapshot_id = $2::uuid WHERE id = $1::uuid`, rid, snapID); err != nil {
return err
}
if _, err := tx.Exec(ctx, `
INSERT INTO prefix_snapshot_row (snapshot_id, ord, prefix, community_id, source)
VALUES ($1::uuid, 0, '203.0.113.0/24'::cidr, $2::uuid, 'demo'),
($1::uuid, 1, '2001:db8::/32'::cidr, $2::uuid, 'demo')`, snapID, cid); err != nil {
return err
}
if _, err := tx.Exec(ctx, `
+6 -2
View File
@@ -14,6 +14,7 @@ import (
"evobgp/internal/broker"
"evobgp/internal/config"
"evobgp/internal/httpclient"
"evobgp/internal/jobs"
"evobgp/internal/pipeline"
"evobgp/internal/store"
@@ -108,6 +109,9 @@ func postTenantRefresh(ctx context.Context, deps *Deps, moduleIDs []string, idem
if err != nil {
return err
}
req.GetBody = func() (io.ReadCloser, error) {
return io.NopCloser(bytes.NewReader(body)), nil
}
req.Header.Set("Authorization", "Bearer "+strings.TrimSpace(deps.APIToken))
req.Header.Set("Content-Type", "application/json")
if idempotencyKey != "" {
@@ -115,9 +119,9 @@ func postTenantRefresh(ctx context.Context, deps *Deps, moduleIDs []string, idem
}
hc := deps.HTTP
if hc == nil {
hc = http.DefaultClient
hc = httpclient.New(httpclient.DefaultTimeout)
}
resp, err := hc.Do(req)
resp, err := httpclient.DoWithRetry(ctx, hc, req, 3)
if err != nil {
return err
}
+6
View File
@@ -73,8 +73,11 @@ type Backend interface {
GetSpeakerAnyTenant(speakerID string) (*Speaker, error)
CreateSpeaker(tenantID string, in *Speaker) (*Speaker, error)
UpdateSpeaker(tenantID, id string, patch *SpeakerPatch) (*Speaker, error)
DeleteSpeaker(tenantID, id string) error
GetRevision(tenantID, revisionID string) (*Revision, error)
// GetRevisionSummary returns revision metadata without preview_fragments payloads.
GetRevisionSummary(tenantID, revisionID string) (*Revision, error)
ListRevisions(tenantID, moduleID string, cursor string, limit int) (items []*Revision, nextCursor string, hasMore bool)
ListRevisionPrefixes(tenantID, revisionID string, cursor string, limit int) (prefixes []PrefixRow, next string, more bool)
CreateRollbackRevision(tenantID, sourceRevisionID string) (newID string, err error)
@@ -109,6 +112,9 @@ type Backend interface {
// Ping verifies backend connectivity (no-op for in-memory).
Ping(ctx context.Context) error
// RunPeriodicMaintenance prunes stale DB rows (no-op for in-memory).
RunPeriodicMaintenance(ctx context.Context)
}
// ASNPrefixCacheEntry is a cached RIPEstat response for one ASN.
+17 -7
View File
@@ -305,6 +305,11 @@ func (m *Memory) Ping(ctx context.Context) error {
return nil
}
// RunPeriodicMaintenance is a no-op for the in-memory backend.
func (m *Memory) RunPeriodicMaintenance(ctx context.Context) {
_ = ctx
}
// ListTenantIDs returns tenant ids sorted lexicographically.
func (m *Memory) ListTenantIDs() ([]string, error) {
m.mu.RLock()
@@ -424,14 +429,19 @@ func (m *Memory) GetModule(tenantID, moduleID string) (*Module, error) {
func (m *Memory) GetRevision(tenantID, revisionID string) (*Revision, error) {
m.mu.RLock()
defer m.mu.RUnlock()
rev, ok := m.revisions[revisionID]
if !ok {
return nil, ErrNotFound
return m.getRevisionLocked(tenantID, revisionID)
}
func (m *Memory) GetRevisionSummary(tenantID, revisionID string) (*Revision, error) {
m.mu.RLock()
defer m.mu.RUnlock()
rev, err := m.getRevisionLocked(tenantID, revisionID)
if err != nil {
return nil, err
}
if rev.TenantID != tenantID {
return nil, ErrTenantScope
}
return rev, nil
cp := *rev
cp.PreviewFragments = nil
return &cp, nil
}
func (m *Memory) GetSpeaker(tenantID, speakerID string) (*Speaker, error) {
+35 -19
View File
@@ -1,8 +1,6 @@
package store
import (
"fmt"
"strconv"
"strings"
"time"
@@ -779,35 +777,53 @@ func (m *Memory) UpdateSpeaker(tenantID, id string, patch *SpeakerPatch) (*Speak
return sp, nil
}
func (m *Memory) DeleteSpeaker(tenantID, id string) error {
m.mu.Lock()
defer m.mu.Unlock()
sp, ok := m.speakers[id]
if !ok || sp.TenantID != tenantID {
return ErrNotFound
}
delete(m.speakers, id)
delete(m.publishedRevision, id)
return nil
}
func (m *Memory) ListRevisionPrefixes(tenantID, revisionID string, cursor string, limit int) ([]PrefixRow, string, bool) {
if limit <= 0 {
limit = 50
}
afterID, off, useOffset := ParsePrefixPageCursor(cursor)
m.mu.RLock()
defer m.mu.RUnlock()
if _, err := m.getRevisionLocked(tenantID, revisionID); err != nil {
return nil, "", false
}
all := m.revPrefixes[revisionID]
off := 0
if cursor != "" {
if n, err := strconv.Atoi(cursor); err == nil && n >= 0 {
off = n
}
allRows := m.revPrefixes[revisionID]
start := 0
if useOffset {
start = off
} else if afterID != nil {
start = int(*afterID) + 1
}
end := off + limit
next := ""
more := false
if end > len(all) {
end = len(all)
} else {
more = true
next = fmt.Sprintf("%d", end)
}
if off >= len(all) {
if start > len(allRows) {
return nil, "", false
}
return all[off:end], next, more
end := start + limit
next := ""
more := false
if end > len(allRows) {
end = len(allRows)
} else {
more = true
next = FormatPrefixPageCursor(int64(end - 1))
}
if start >= end {
return nil, "", false
}
out := make([]PrefixRow, end-start)
copy(out, allRows[start:end])
return out, next, more
}
func (m *Memory) ListGlobalSettings(tenantID string) (map[string]any, error) {
+30
View File
@@ -0,0 +1,30 @@
package store
import (
"strconv"
"strings"
)
// ParsePrefixPageCursor decodes opaque cursors for revision prefix pagination.
func ParsePrefixPageCursor(cursor string) (afterID *int64, offset int, useOffset bool) {
cursor = strings.TrimSpace(cursor)
if cursor == "" {
return nil, 0, false
}
if strings.HasPrefix(cursor, "o:") {
n, err := strconv.Atoi(strings.TrimPrefix(cursor, "o:"))
if err != nil || n < 0 {
return nil, 0, false
}
return nil, n, true
}
if n, err := strconv.ParseInt(cursor, 10, 64); err == nil && n >= 0 {
return &n, 0, false
}
return nil, 0, false
}
// FormatPrefixPageCursor encodes the keyset cursor (last row id or slice index).
func FormatPrefixPageCursor(lastID int64) string {
return strconv.FormatInt(lastID, 10)
}
+13 -2
View File
@@ -72,7 +72,9 @@ func MergeSpeakerMetaJSON(existing string, patch SpeakerMeta) string {
if patch.LastDispatchAt != "" {
cur.LastDispatchAt = patch.LastDispatchAt
}
if patch.LastDispatchError != "" {
if patch.LastDispatchStatus == "ok" {
cur.LastDispatchError = ""
} else if patch.LastDispatchError != "" {
cur.LastDispatchError = patch.LastDispatchError
}
if patch.LastDispatchStatus != "" {
@@ -124,11 +126,20 @@ func AgentSyncURL(meta SpeakerMeta) string {
// AgentHealthURL returns HTTPS health URL for agent_domain.
func AgentHealthURL(meta SpeakerMeta) string {
return agentHTTPSURL(meta, "/v1/agent/health")
}
// AgentBirdProtocolsURL returns HTTPS bird protocols URL for agent_domain.
func AgentBirdProtocolsURL(meta SpeakerMeta) string {
return agentHTTPSURL(meta, "/v1/agent/bird/protocols")
}
func agentHTTPSURL(meta SpeakerMeta, path string) string {
domain := strings.TrimSpace(meta.AgentDomain)
if domain == "" {
return ""
}
return "https://" + strings.TrimSuffix(domain, "/") + "/v1/agent/health"
return "https://" + strings.TrimSuffix(domain, "/") + path
}
// SpeakerNeedsRemoteDispatch reports whether deploy_apply should wake this speaker via agent HTTP.
+21
View File
@@ -34,3 +34,24 @@ func TestAgentSyncURL(t *testing.T) {
t.Fatalf("got %q", u)
}
}
func TestMergeSpeakerMetaJSON_clearsDispatchErrorOnOk(t *testing.T) {
t.Parallel()
existing := store.SpeakerMetaJSON(store.SpeakerMeta{
LastDispatchError: "HTTP 502: bundle 403",
LastDispatchStatus: "error",
SyncStatus: "error",
})
merged := store.MergeSpeakerMetaJSON(existing, store.SpeakerMeta{
LastDispatchStatus: "ok",
SyncStatus: "synced",
LastDispatchAt: "2026-05-21T15:06:43Z",
})
m := store.ParseSpeakerMeta(merged)
if m.LastDispatchError != "" {
t.Fatalf("LastDispatchError should clear on ok dispatch, got %q", m.LastDispatchError)
}
if m.LastDispatchStatus != "ok" || m.SyncStatus != "synced" {
t.Fatalf("status: dispatch=%q sync=%q", m.LastDispatchStatus, m.SyncStatus)
}
}
@@ -0,0 +1,6 @@
DROP INDEX IF EXISTS idx_rev_mat_prefix_uniq;
DROP INDEX IF EXISTS idx_rev_mat_prefix_rev_cover;
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_rev_id
ON revision_materialized_prefix (revision_id, id);
DROP INDEX IF EXISTS idx_module_tenant_active_sort;
DROP INDEX IF EXISTS idx_rev_mat_prefix_rev_prefix;
@@ -0,0 +1,17 @@
-- Query indexes: revision diff/list and module sort (H4 + M4).
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_rev_prefix
ON revision_materialized_prefix (revision_id, prefix);
CREATE INDEX IF NOT EXISTS idx_module_tenant_active_sort
ON module (tenant_id, priority, name)
WHERE deleted_at IS NULL;
-- Covering index for prefix list pagination (PostgreSQL 11+ INCLUDE).
DROP INDEX IF EXISTS idx_rev_mat_prefix_rev_id;
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_rev_cover
ON revision_materialized_prefix (revision_id, id)
INCLUDE (prefix, community_id, source);
CREATE UNIQUE INDEX IF NOT EXISTS idx_rev_mat_prefix_uniq
ON revision_materialized_prefix (revision_id, prefix, community_id, source);
@@ -0,0 +1,6 @@
UPDATE config_revision cr
SET meta_json = cr.meta_json || jsonb_build_object('preview_fragments', COALESCE(p.fragments, '{}'::jsonb))
FROM config_revision_preview p
WHERE p.revision_id = cr.id;
DROP TABLE IF EXISTS config_revision_preview;
@@ -0,0 +1,15 @@
-- Split BIRD preview fragments out of config_revision.meta_json (H1).
CREATE TABLE config_revision_preview (
revision_id UUID PRIMARY KEY REFERENCES config_revision (id) ON DELETE CASCADE,
fragments JSONB NOT NULL DEFAULT '{}'
);
INSERT INTO config_revision_preview (revision_id, fragments)
SELECT id, COALESCE(meta_json->'preview_fragments', '{}'::jsonb)
FROM config_revision
WHERE meta_json ? 'preview_fragments';
UPDATE config_revision
SET meta_json = meta_json - 'preview_fragments'
WHERE meta_json ? 'preview_fragments';
@@ -0,0 +1,3 @@
ALTER TABLE config_revision DROP COLUMN IF EXISTS prefix_snapshot_id;
DROP TABLE IF EXISTS prefix_snapshot_row;
DROP TABLE IF EXISTS prefix_snapshot;
@@ -0,0 +1,22 @@
-- Content-addressed prefix snapshots (H2 expand).
CREATE TABLE prefix_snapshot (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
content_hash CHAR(64) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT prefix_snapshot_hash_uniq UNIQUE (content_hash)
);
CREATE TABLE prefix_snapshot_row (
snapshot_id UUID NOT NULL REFERENCES prefix_snapshot (id) ON DELETE CASCADE,
ord INTEGER NOT NULL,
prefix CIDR NOT NULL,
community_id UUID REFERENCES bgp_community (id) ON DELETE SET NULL,
source TEXT NOT NULL DEFAULT '',
PRIMARY KEY (snapshot_id, ord)
);
CREATE INDEX idx_prefix_snapshot_row_snapshot ON prefix_snapshot_row (snapshot_id);
ALTER TABLE config_revision
ADD COLUMN prefix_snapshot_id UUID REFERENCES prefix_snapshot (id) ON DELETE RESTRICT;
@@ -0,0 +1,3 @@
DELETE FROM prefix_snapshot_row;
UPDATE config_revision SET prefix_snapshot_id = NULL WHERE prefix_snapshot_id IS NOT NULL;
DELETE FROM prefix_snapshot;
@@ -0,0 +1,31 @@
-- Backfill prefix snapshots from revision_materialized_prefix.
WITH new_snaps AS (
INSERT INTO prefix_snapshot (id, content_hash)
SELECT gen_random_uuid(),
substr(replace(cr.id::text, '-', '') || replace(cr.id::text, '-', ''), 1, 64)
FROM config_revision cr
WHERE cr.prefix_snapshot_id IS NULL
AND EXISTS (
SELECT 1 FROM revision_materialized_prefix rmp WHERE rmp.revision_id = cr.id
)
RETURNING id, content_hash
)
UPDATE config_revision cr
SET prefix_snapshot_id = ns.id
FROM new_snaps ns
WHERE cr.prefix_snapshot_id IS NULL
AND ns.content_hash = substr(replace(cr.id::text, '-', '') || replace(cr.id::text, '-', ''), 1, 64);
INSERT INTO prefix_snapshot_row (snapshot_id, ord, prefix, community_id, source)
SELECT cr.prefix_snapshot_id,
(row_number() OVER (PARTITION BY cr.id ORDER BY rmp.id) - 1)::int,
rmp.prefix,
rmp.community_id,
rmp.source
FROM config_revision cr
JOIN revision_materialized_prefix rmp ON rmp.revision_id = cr.id
WHERE cr.prefix_snapshot_id IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM prefix_snapshot_row psr WHERE psr.snapshot_id = cr.prefix_snapshot_id
);
@@ -0,0 +1 @@
COMMENT ON COLUMN config_revision.prefix_snapshot_id IS NULL;
@@ -0,0 +1,3 @@
-- H2 contract marker: new revisions use prefix_snapshot_id only (enforced in application code).
COMMENT ON COLUMN config_revision.prefix_snapshot_id IS 'Materialized prefixes; revision_materialized_prefix deprecated for new rows';
@@ -0,0 +1,16 @@
ALTER TABLE module_prefix_snapshot ADD COLUMN prefixes_json JSONB NOT NULL DEFAULT '[]';
UPDATE module_prefix_snapshot mps
SET prefixes_json = COALESCE((
SELECT jsonb_agg(
jsonb_build_object(
'prefix', psr.prefix::text,
'community_id', psr.community_id,
'source', psr.source
) ORDER BY psr.ord
)
FROM module_prefix_snapshot_row psr
WHERE psr.tenant_id = mps.tenant_id AND psr.module_id = mps.module_id
), '[]'::jsonb);
DROP TABLE IF EXISTS module_prefix_snapshot_row;
@@ -0,0 +1,25 @@
CREATE TABLE module_prefix_snapshot_row (
tenant_id UUID NOT NULL,
module_id UUID NOT NULL,
ord INTEGER NOT NULL,
prefix CIDR NOT NULL,
community_id UUID,
source TEXT NOT NULL DEFAULT '',
PRIMARY KEY (tenant_id, module_id, ord),
FOREIGN KEY (tenant_id, module_id)
REFERENCES module_prefix_snapshot (tenant_id, module_id) ON DELETE CASCADE
);
INSERT INTO module_prefix_snapshot_row (tenant_id, module_id, ord, prefix, community_id, source)
SELECT mps.tenant_id,
mps.module_id,
(t.ordinality - 1)::int,
(t.elem->>'prefix')::cidr,
NULLIF(t.elem->>'community_id', '')::uuid,
COALESCE(NULLIF(t.elem->>'source', ''), '')
FROM module_prefix_snapshot mps
CROSS JOIN LATERAL jsonb_array_elements(mps.prefixes_json) WITH ORDINALITY AS t(elem, ordinality)
WHERE jsonb_typeof(mps.prefixes_json) = 'array'
AND jsonb_array_length(mps.prefixes_json) > 0;
ALTER TABLE module_prefix_snapshot DROP COLUMN prefixes_json;
@@ -0,0 +1,10 @@
ALTER TABLE asn_prefix_cache ADD COLUMN prefixes_json JSONB NOT NULL DEFAULT '[]';
UPDATE asn_prefix_cache apc
SET prefixes_json = COALESCE((
SELECT jsonb_agg(apcr.prefix::text ORDER BY apcr.prefix::text)
FROM asn_prefix_cache_row apcr
WHERE apcr.asn = apc.asn
), '[]'::jsonb);
DROP TABLE IF EXISTS asn_prefix_cache_row;
@@ -0,0 +1,14 @@
CREATE TABLE asn_prefix_cache_row (
asn BIGINT NOT NULL REFERENCES asn_prefix_cache (asn) ON DELETE CASCADE,
prefix CIDR NOT NULL,
PRIMARY KEY (asn, prefix)
);
INSERT INTO asn_prefix_cache_row (asn, prefix)
SELECT apc.asn, t.elem::cidr
FROM asn_prefix_cache apc
CROSS JOIN LATERAL jsonb_array_elements_text(apc.prefixes_json) AS t(elem)
WHERE jsonb_typeof(apc.prefixes_json) = 'array'
AND jsonb_array_length(apc.prefixes_json) > 0;
ALTER TABLE asn_prefix_cache DROP COLUMN prefixes_json;
@@ -0,0 +1 @@
DROP INDEX IF EXISTS idx_job_audit_created_brin;
@@ -0,0 +1,2 @@
CREATE INDEX IF NOT EXISTS idx_job_audit_created_brin
ON job_audit USING BRIN (created_at);
@@ -0,0 +1,18 @@
CREATE TABLE IF NOT EXISTS module_cdn_fetch_log (
id BIGSERIAL PRIMARY KEY,
module_id UUID NOT NULL REFERENCES module (id) ON DELETE CASCADE,
source_id UUID REFERENCES module_cdn_source (id) ON DELETE SET NULL,
http_status INTEGER,
bytes BIGINT,
error TEXT,
fetched_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_module_cdn_fetch_log_module ON module_cdn_fetch_log (module_id, fetched_at DESC);
ALTER TABLE revision_materialized_prefix ADD COLUMN IF NOT EXISTS meta_json JSONB NOT NULL DEFAULT '{}';
ALTER TABLE module_domain_entry ADD COLUMN IF NOT EXISTS resolve_meta JSONB NOT NULL DEFAULT '{}';
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_revision ON revision_materialized_prefix (revision_id);
CREATE INDEX IF NOT EXISTS idx_module_tenant ON module (tenant_id);
CREATE INDEX IF NOT EXISTS idx_module_prefix_snapshot_collected ON module_prefix_snapshot (collected_at);
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_value ON revision_materialized_prefix (prefix);
@@ -0,0 +1,8 @@
ALTER TABLE revision_materialized_prefix DROP COLUMN IF EXISTS meta_json;
ALTER TABLE module_domain_entry DROP COLUMN IF EXISTS resolve_meta;
DROP TABLE IF EXISTS module_cdn_fetch_log;
DROP INDEX IF EXISTS idx_rev_mat_prefix_revision;
DROP INDEX IF EXISTS idx_module_tenant;
DROP INDEX IF EXISTS idx_module_prefix_snapshot_collected;
DROP INDEX IF EXISTS idx_rev_mat_prefix_value;
@@ -0,0 +1,3 @@
DROP INDEX IF EXISTS idx_rev_mat_prefix_uniq;
DROP INDEX IF EXISTS idx_module_tenant_active_sort;
DROP INDEX IF EXISTS idx_rev_mat_prefix_rev_prefix;
@@ -0,0 +1,9 @@
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_rev_prefix
ON revision_materialized_prefix (revision_id, prefix);
CREATE INDEX IF NOT EXISTS idx_module_tenant_active_sort
ON module (tenant_id, priority, name)
WHERE deleted_at IS NULL;
CREATE UNIQUE INDEX IF NOT EXISTS idx_rev_mat_prefix_uniq
ON revision_materialized_prefix (revision_id, prefix, community_id, source);
@@ -0,0 +1,9 @@
UPDATE config_revision
SET meta_json = json_set(
meta_json,
'$.preview_fragments',
json(COALESCE((SELECT fragments FROM config_revision_preview p WHERE p.revision_id = config_revision.id), '{}'))
)
WHERE id IN (SELECT revision_id FROM config_revision_preview);
DROP TABLE IF EXISTS config_revision_preview;
@@ -0,0 +1,13 @@
CREATE TABLE config_revision_preview (
revision_id TEXT NOT NULL PRIMARY KEY REFERENCES config_revision (id) ON DELETE CASCADE,
fragments TEXT NOT NULL DEFAULT '{}'
);
INSERT INTO config_revision_preview (revision_id, fragments)
SELECT id, json(COALESCE(json_extract(meta_json, '$.preview_fragments'), '{}'))
FROM config_revision
WHERE json_extract(meta_json, '$.preview_fragments') IS NOT NULL;
UPDATE config_revision
SET meta_json = json_remove(meta_json, '$.preview_fragments')
WHERE json_extract(meta_json, '$.preview_fragments') IS NOT NULL;
@@ -0,0 +1,3 @@
ALTER TABLE config_revision DROP COLUMN prefix_snapshot_id;
DROP TABLE IF EXISTS prefix_snapshot_row;
DROP TABLE IF EXISTS prefix_snapshot;
@@ -0,0 +1,18 @@
CREATE TABLE prefix_snapshot (
id TEXT NOT NULL PRIMARY KEY,
content_hash TEXT NOT NULL UNIQUE,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
);
CREATE TABLE prefix_snapshot_row (
snapshot_id TEXT NOT NULL REFERENCES prefix_snapshot (id) ON DELETE CASCADE,
ord INTEGER NOT NULL,
prefix TEXT NOT NULL,
community_id TEXT,
source TEXT NOT NULL DEFAULT '',
PRIMARY KEY (snapshot_id, ord)
);
CREATE INDEX idx_prefix_snapshot_row_snapshot ON prefix_snapshot_row (snapshot_id);
ALTER TABLE config_revision ADD COLUMN prefix_snapshot_id TEXT REFERENCES prefix_snapshot (id) ON DELETE RESTRICT;
@@ -0,0 +1,3 @@
DELETE FROM prefix_snapshot_row;
UPDATE config_revision SET prefix_snapshot_id = NULL;
DELETE FROM prefix_snapshot;
@@ -0,0 +1,30 @@
-- SQLite backfill: one snapshot per revision with materialized prefixes.
INSERT INTO prefix_snapshot (id, content_hash)
SELECT lower(hex(randomblob(16))),
substr(replace(cr.id, '-', '') || replace(cr.id, '-', ''), 1, 64)
FROM config_revision cr
WHERE cr.prefix_snapshot_id IS NULL
AND EXISTS (SELECT 1 FROM revision_materialized_prefix rmp WHERE rmp.revision_id = cr.id);
UPDATE config_revision
SET prefix_snapshot_id = (
SELECT ps.id FROM prefix_snapshot ps
WHERE ps.content_hash = substr(replace(config_revision.id, '-', '') || replace(config_revision.id, '-', ''), 1, 64)
)
WHERE prefix_snapshot_id IS NULL
AND EXISTS (SELECT 1 FROM revision_materialized_prefix rmp WHERE rmp.revision_id = config_revision.id);
INSERT INTO prefix_snapshot_row (snapshot_id, ord, prefix, community_id, source)
SELECT cr.prefix_snapshot_id,
(SELECT COUNT(*) FROM revision_materialized_prefix r2
WHERE r2.revision_id = cr.id AND r2.id <= rmp.id) - 1,
rmp.prefix,
rmp.community_id,
rmp.source
FROM config_revision cr
JOIN revision_materialized_prefix rmp ON rmp.revision_id = cr.id
WHERE cr.prefix_snapshot_id IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM prefix_snapshot_row psr WHERE psr.snapshot_id = cr.prefix_snapshot_id
);
@@ -0,0 +1 @@
-- H2 contract marker (application stops writing revision_materialized_prefix for new revisions).
@@ -0,0 +1,2 @@
ALTER TABLE module_prefix_snapshot ADD COLUMN prefixes_json TEXT NOT NULL DEFAULT '[]';
DROP TABLE IF EXISTS module_prefix_snapshot_row;
@@ -0,0 +1,13 @@
CREATE TABLE module_prefix_snapshot_row (
tenant_id TEXT NOT NULL,
module_id TEXT NOT NULL,
ord INTEGER NOT NULL,
prefix TEXT NOT NULL,
community_id TEXT,
source TEXT NOT NULL DEFAULT '',
PRIMARY KEY (tenant_id, module_id, ord),
FOREIGN KEY (tenant_id, module_id)
REFERENCES module_prefix_snapshot (tenant_id, module_id) ON DELETE CASCADE
);
ALTER TABLE module_prefix_snapshot DROP COLUMN prefixes_json;
@@ -0,0 +1,2 @@
ALTER TABLE asn_prefix_cache ADD COLUMN prefixes_json TEXT NOT NULL DEFAULT '[]';
DROP TABLE IF EXISTS asn_prefix_cache_row;
@@ -0,0 +1,7 @@
CREATE TABLE asn_prefix_cache_row (
asn INTEGER NOT NULL REFERENCES asn_prefix_cache (asn) ON DELETE CASCADE,
prefix TEXT NOT NULL,
PRIMARY KEY (asn, prefix)
);
ALTER TABLE asn_prefix_cache DROP COLUMN prefixes_json;
@@ -0,0 +1 @@
DROP INDEX IF EXISTS idx_job_audit_created_brin;
@@ -0,0 +1 @@
CREATE INDEX IF NOT EXISTS idx_job_audit_created_brin ON job_audit (created_at);
@@ -0,0 +1,18 @@
ALTER TABLE revision_materialized_prefix ADD COLUMN meta_json TEXT NOT NULL DEFAULT '{}';
ALTER TABLE module_domain_entry ADD COLUMN resolve_meta TEXT NOT NULL DEFAULT '{}';
CREATE TABLE IF NOT EXISTS module_cdn_fetch_log (
id INTEGER PRIMARY KEY AUTOINCREMENT,
module_id TEXT NOT NULL REFERENCES module (id) ON DELETE CASCADE,
source_id TEXT REFERENCES module_cdn_source (id) ON DELETE SET NULL,
http_status INTEGER,
bytes INTEGER,
error TEXT,
fetched_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
);
CREATE INDEX IF NOT EXISTS idx_module_cdn_fetch_log_module ON module_cdn_fetch_log (module_id, fetched_at DESC);
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_revision ON revision_materialized_prefix (revision_id);
CREATE INDEX IF NOT EXISTS idx_module_tenant ON module (tenant_id);
CREATE INDEX IF NOT EXISTS idx_module_prefix_snapshot_collected ON module_prefix_snapshot (collected_at);
CREATE INDEX IF NOT EXISTS idx_rev_mat_prefix_value ON revision_materialized_prefix (prefix);
@@ -0,0 +1,8 @@
ALTER TABLE revision_materialized_prefix DROP COLUMN meta_json;
ALTER TABLE module_domain_entry DROP COLUMN resolve_meta;
DROP TABLE IF EXISTS module_cdn_fetch_log;
DROP INDEX IF EXISTS idx_rev_mat_prefix_revision;
DROP INDEX IF EXISTS idx_module_tenant;
DROP INDEX IF EXISTS idx_module_prefix_snapshot_collected;
DROP INDEX IF EXISTS idx_rev_mat_prefix_value;
+64
View File
@@ -0,0 +1,64 @@
#!/usr/bin/env node
/**
* Warns about commits since the last tag that semantic-release cannot parse.
* Exit 0 always semantic-release still decides release/no-op.
*/
import { execSync } from 'node:child_process';
import parser from 'conventional-commits-parser';
const RELEASABLE = new Set(['feat', 'fix', 'perf', 'ci', 'refactor']);
function lastTag() {
try {
return execSync('git describe --tags --abbrev=0', { encoding: 'utf8' }).trim();
} catch {
return '';
}
}
function commitsSince(ref) {
const range = ref ? `${ref}..HEAD` : 'HEAD';
const out = execSync(`git log ${range} --format=%H%x09%s`, { encoding: 'utf8' }).trim();
if (!out) return [];
return out.split('\n').map((line) => {
const [hash, subject] = line.split('\t');
return { hash: hash.trim(), subject: subject.trim() };
});
}
const tag = lastTag();
const commits = commitsSince(tag);
const unparseable = [];
const releasable = [];
for (const { hash, subject } of commits) {
const parsed = parser.sync(subject);
if (!parsed.type) {
unparseable.push({ hash: hash.slice(0, 7), subject });
continue;
}
if (RELEASABLE.has(parsed.type)) {
releasable.push({ hash: hash.slice(0, 7), subject, type: parsed.type });
}
}
if (commits.length === 0) {
console.log(`No new commits since ${tag || 'initial'}.`);
process.exit(0);
}
if (unparseable.length > 0) {
console.warn('::warning:: Commits not parseable by semantic-release (no version bump):');
for (const b of unparseable) {
console.warn(` ${b.hash} ${b.subject}`);
}
console.warn('Fix: single scope without commas, e.g. refactor(web): summary');
}
if (releasable.length > 0) {
console.log(`Releasable since ${tag}: ${releasable.length} commit(s).`);
} else {
console.warn('::warning:: No releasable commits since last tag — release job will no-op.');
}
process.exit(0);

Some files were not shown because too many files have changed in this diff Show More