Compare commits

...
2 Commits
Author SHA1 Message Date
Denozordec db75126bea feat(runtime-logs): enhance auto-cleanup features and documentation
CI / changes (push) Successful in 9s
CI / commitlint (push) Has been skipped
CI / openapi (push) Successful in 26s
CI / web (push) Successful in 33s
CI / go (push) Successful in 56s
CI / bird2 (push) Successful in 14s
CI / release (push) Successful in 20s
Added new endpoints for estimating and executing runtime log auto-cleanup based on tenant settings. Introduced configuration options for auto-cleanup policies, including scheduling and file size limits. Updated the API documentation and UI components to reflect these changes, improving user interaction with runtime log management. Enhanced error handling and added new UI elements for better visibility of audit logs and cleanup actions.
2026-06-12 22:44:39 +07:00
Denozordec f39df7c4bf feat(revisions): add pruning estimate and cleanup endpoints
CI / changes (push) Successful in 8s
CI / commitlint (push) Has been skipped
CI / openapi (push) Successful in 25s
CI / web (push) Successful in 32s
CI / go (push) Successful in 57s
CI / bird2 (push) Successful in 15s
CI / release (push) Successful in 3m18s
Implemented new endpoints for estimating and pruning revisions, including detailed schemas for requests and responses. The `RevisionPruneEstimate` and `RevisionPruneResult` components were added to the OpenAPI documentation, enhancing the API's functionality for managing revision retention. Updated the backend to support these operations and integrated them into the tenant settings UI for improved user interaction.
2026-06-12 21:56:36 +07:00
41 changed files with 2662 additions and 210 deletions
+7 -6
View File
@@ -30,12 +30,13 @@ func main() {
}
cfg := config.Load()
opts := httpapi.Options{
APIKeys: os.Getenv("EVOBGP_API_KEYS"),
DatabaseURL: cfg.DatabaseURL,
InsecureDev: os.Getenv("EVOBGP_DEV_INSECURE") == "1",
SeedDemo: os.Getenv("EVOBGP_SEED_DEMO") != "0",
BundleSeedHex: strings.TrimSpace(os.Getenv("EVOBGP_BUNDLE_SEED_HEX")),
CORSAllowedOrigins: strings.TrimSpace(os.Getenv("EVOBGP_CORS_ORIGINS")),
APIKeys: os.Getenv("EVOBGP_API_KEYS"),
DatabaseURL: cfg.DatabaseURL,
InsecureDev: os.Getenv("EVOBGP_DEV_INSECURE") == "1",
SeedDemo: os.Getenv("EVOBGP_SEED_DEMO") != "0",
BundleSeedHex: strings.TrimSpace(os.Getenv("EVOBGP_BUNDLE_SEED_HEX")),
CORSAllowedOrigins: strings.TrimSpace(os.Getenv("EVOBGP_CORS_ORIGINS")),
RuntimeLogsPolicyTenant: cfg.RuntimeLogsPolicyTenant,
}
srv, err := httpapi.New(opts)
if err != nil {
+5 -1
View File
@@ -110,7 +110,11 @@ EVOBGP_SERVICE=evobgp-all
EVOBGP_RUNTIME_LOGS_DIR=/opt/evobgp/runtime-logs
```
В dev-профиле compose каталог на хосте обычно `./runtime-logs`, в контейнере — mount на `/opt/evobgp/runtime-logs`. Если каталог не задан или роль процесса не `evobgp-all`, эндпоинты `/v1/runtime-logs/*` отвечают **503** (`runtime_logs_unavailable`). Очистка файлов — роль **operator+**; операции пишутся в таблицу `runtime_log_cleanup_audit`.
В dev-профиле compose каталог на хосте обычно `./runtime-logs`, в контейнере — mount на `/opt/evobgp/runtime-logs`. Если каталог не задан или роль процесса не `evobgp-all`, FS-эндпоинты (`/files`, `/auto-*`) отвечают **503** (`runtime_logs_unavailable`). `GET /v1/runtime-logs/cleanup-audit` доступен без volume.
Очистка файлов — роль **operator+**; операции пишутся в `runtime_log_cleanup_audit`. Автоочистка настраивается в tenant settings (`runtime_logs_auto_enabled`, `runtime_logs_max_file_mb`, `runtime_logs_auto_schedule`, `runtime_logs_auto_mode`); scheduler — только в `evobgp-all`. Опционально: `EVOBGP_RUNTIME_LOGS_POLICY_TENANT` — tenant, чьи settings читает scheduler (иначе первый tenant с включённой автоочисткой).
Retention строк audit: пресет maintenance policy `runtime_log_cleanup_audit` (90d) в Monitoring → PostgreSQL → Политики.
## CORS для веб-интерфейса
+4 -2
View File
@@ -90,7 +90,7 @@
### Settings
- `GET /v1/settings`, `PATCH /v1/settings` — tenant KV (`global_settings`): BIRD, `revision_retention_minutes`, произвольные ключи. Чтение — viewer+; `PATCH` — operator+.
- `GET /v1/settings`, `PATCH /v1/settings` — tenant KV (`global_settings`): BIRD, `revision_retention_minutes`, `runtime_logs_*` (автоочистка FS), произвольные ключи. Чтение — viewer+; `PATCH` — operator+.
### RuntimeLogs
@@ -101,7 +101,9 @@
| `GET` | `/v1/runtime-logs/files` | viewer+ | Список `*.log` (имя, размер, mtime) |
| `GET` | `/v1/runtime-logs/files/{filename}` | viewer+ | Хвост файла (`?lines=`, `?bytes=`, `?grep=`) |
| `DELETE` | `/v1/runtime-logs/files/{filename}` | operator+ | Синхронная очистка (`?mode=truncate\|delete`, default truncate); max 512 MiB |
| `GET` | `/v1/runtime-logs/cleanup-audit` | viewer+ | Пагинированный audit очистки (`cursor`, `limit`) |
| `GET` | `/v1/runtime-logs/cleanup-audit` | viewer+ | Пагинированный audit очистки (`cursor`, `limit`); **без FS volume** |
| `GET` | `/v1/runtime-logs/auto-estimate` | operator+ | Файлы выше порога из tenant settings |
| `POST` | `/v1/runtime-logs/auto-run` | operator+ | Немедленный прогон (`?dry_run=true`); audit с `actor_prefix=auto:scheduler` |
`{filename}` — только basename, паттерн `^[a-z0-9][a-z0-9_.-]*\.log$`. Очистка пишет строку в таблицу `runtime_log_cleanup_audit` (миграция `000026`).
+4 -3
View File
@@ -112,15 +112,16 @@ EvoBGP управляет генерацией и применением BGP-к
| Маршрут | Назначение |
|---------|------------|
| `/settings` | Только браузер: API-токен, тема (localStorage). Tenant KV здесь **не** редактируются. |
| `/tenant-settings` | Все tenant-параметры из `/v1/settings`: вкладки **BIRD**, **Ревизии** (`revision_retention_minutes`), **Дополнительно** (custom KV). Пункт nav **«Параметры»**. |
| `/tenant-settings` | Все tenant-параметры из `/v1/settings`: вкладки **BIRD**, **Ревизии**, **Файловые логи** (автоочистка FS), **Дополнительно** (custom KV). Пункт nav **«Параметры»**. |
| `/network` → Control plane | Краткая сводка BIRD + ссылка на `/tenant-settings?tab=bird`. |
| `/operations` | Ревизии, diff, jobs; вкладка «Система» перенесена в `/tenant-settings`. |
### Web UI: файловые runtime-логи
- **Мониторинг** → вкладка **«Файловые логи»** (`/monitoring?tab=runtime-logs`).
- Подвкладки: **Файлы** (список, preview хвоста, очистка operator) и **Audit очистки**.
- При **503** на списке файлов: FS API недоступен (не `evobgp-all` или нет volume); audit из БД может отображаться отдельно.
- Подвкладки: **Файлы** (список, preview хвоста, очистка operator) и **Audit очистки** (история из БД, в т.ч. `auto:scheduler`).
- Автоочистка: **Параметры****Файловые логи** — порог MiB, UTC cron, режим truncate/delete; scheduler в `evobgp-all`.
- При **503** на списке файлов: FS API недоступен (не `evobgp-all` или нет volume); `GET /v1/runtime-logs/cleanup-audit` работает без volume.
- Deploy: `EVOBGP_RUNTIME_LOGS_DIR`, bind-mount на `evobgp-all`, sidecar `stack-runtime-logs` — [quickstart.md](quickstart.md#файловые-runtime-логи-api-v1runtime-logs), [access.md](access.md).
## 7. Эксплуатация и runbook
+246
View File
@@ -1168,6 +1168,54 @@ components:
has_more:
type: boolean
RuntimeLogAutoPolicy:
type: object
properties:
enabled:
type: boolean
max_file_bytes:
type: integer
format: int64
schedule:
type: string
description: UTC cron (minute hour dom month dow).
mode:
$ref: "#/components/schemas/RuntimeLogCleanupMode"
RuntimeLogAutoEstimateItem:
type: object
required: [filename, size_bytes, would_cleanup]
properties:
filename:
type: string
size_bytes:
type: integer
format: int64
would_cleanup:
type: boolean
skip_reason:
type: string
description: under_threshold, too_large, или текст ошибки.
RuntimeLogAutoEstimate:
type: object
properties:
policy:
$ref: "#/components/schemas/RuntimeLogAutoPolicy"
items:
type: array
items:
$ref: "#/components/schemas/RuntimeLogAutoEstimateItem"
would_count:
type: integer
minimum: 0
RuntimeLogAutoRunResult:
type: object
additionalProperties: true
description: |
dry_run, trigger, policy, cleaned[], skipped[], cleaned_count, skipped_count.
BirdLocalStatus:
type: object
description: Статус локального BIRD на хосте API (GET /v1/bird/status).
@@ -1212,6 +1260,70 @@ components:
format: date-time
additionalProperties: true
RevisionPruneEstimate:
type: object
required:
- retention_minutes
- cutoff_at
- revision_count
- prefix_row_count
- orphan_snapshot_count
- bytes_estimate
properties:
retention_minutes:
type: integer
minimum: 15
maximum: 43200
cutoff_at:
type: string
format: date-time
revision_count:
type: integer
minimum: 0
prefix_row_count:
type: integer
minimum: 0
description: Строки prefix_snapshot_row в освобождаемых снимках.
orphan_snapshot_count:
type: integer
minimum: 0
bytes_estimate:
type: integer
format: int64
minimum: 0
description: Ориентировочный логический объём данных (байты).
RevisionPruneResult:
type: object
required:
- deleted_revisions
- deleted_prefix_snapshots
- deleted_prefix_rows
- bytes_estimate
properties:
deleted_revisions:
type: integer
minimum: 0
deleted_prefix_snapshots:
type: integer
minimum: 0
deleted_prefix_rows:
type: integer
minimum: 0
bytes_estimate:
type: integer
format: int64
minimum: 0
RevisionPruneRequest:
type: object
properties:
retention_minutes:
type: integer
minimum: 15
maximum: 43200
description: TTL в минутах; если не задан — из revision_retention_minutes tenant settings.
PrefixSnapshotItem:
type: object
description: >
@@ -1324,6 +1436,23 @@ components:
bird_bgp_source_ipv6:
type: string
description: Зарезервировано; в текущей генерации BGP не используется.
revision_retention_minutes:
type: integer
minimum: 15
maximum: 43200
runtime_logs_auto_enabled:
type: boolean
description: Автоочистка *.log на evobgp-all по расписанию (только при FS volume).
runtime_logs_max_file_mb:
type: integer
minimum: 1
maximum: 512
description: Truncate/delete файлов строго больше порога (MiB).
runtime_logs_auto_schedule:
type: string
description: UTC cron для автоочистки (по умолчанию `0 */6 * * *`).
runtime_logs_auto_mode:
$ref: "#/components/schemas/RuntimeLogCleanupMode"
additionalProperties: true
RevisionDiff:
@@ -2823,6 +2952,66 @@ paths:
default:
$ref: "#/components/responses/DefaultProblem"
/v1/revisions/prune-estimate:
get:
tags: [Revisions]
summary: Оценка очистки ревизий по retention
description: >
Считает ревизии и ориентировочный объём данных, которые будут удалены при prune
(те же правила, что applyRevisionRetention: последняя ревизия tenant и раскатанные на спикерах сохраняются).
operationId: getRevisionPruneEstimate
parameters:
- $ref: "#/components/parameters/TenantId"
- name: retention_minutes
in: query
required: false
schema:
type: integer
minimum: 15
maximum: 43200
description: TTL в минутах; если не задан — из tenant settings (default 30d).
responses:
"200":
description: Оценка.
content:
application/json:
schema:
$ref: "#/components/schemas/RevisionPruneEstimate"
"422":
$ref: "#/components/responses/UnprocessableEntity"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/revisions/prune:
post:
tags: [Revisions]
summary: Очистить старые ревизии (синхронно)
description: >
Удаляет ревизии старше cutoff по retention и GC неиспользуемых prefix_snapshot.
Operator-only.
operationId: pruneRevisions
parameters:
- $ref: "#/components/parameters/TenantId"
requestBody:
required: false
content:
application/json:
schema:
$ref: "#/components/schemas/RevisionPruneRequest"
responses:
"200":
description: Результат очистки.
content:
application/json:
schema:
$ref: "#/components/schemas/RevisionPruneResult"
"403":
$ref: "#/components/responses/Forbidden"
"422":
$ref: "#/components/responses/UnprocessableEntity"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/revisions/{revision_id}:
parameters:
- $ref: "#/components/parameters/TenantId"
@@ -3992,6 +4181,63 @@ paths:
default:
$ref: "#/components/responses/DefaultProblem"
/v1/runtime-logs/auto-estimate:
get:
tags: [RuntimeLogs]
summary: Оценка автоочистки runtime log-файлов
description: |
Список файлов, которые будут затронуты текущей политикой tenant settings.
Требует evobgp-all с примонтированным каталогом runtime-logs.
operationId: estimateRuntimeLogAutoCleanup
parameters:
- $ref: "#/components/parameters/TenantId"
responses:
"200":
description: Успешно.
content:
application/json:
schema:
$ref: "#/components/schemas/RuntimeLogAutoEstimate"
"503":
description: FS API недоступен (не evobgp-all или нет volume).
content:
application/problem+json:
schema:
$ref: "#/components/schemas/Problem"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/runtime-logs/auto-run:
post:
tags: [RuntimeLogs]
summary: Запустить автоочистку runtime log-файлов
description: |
Немедленный прогон политики из tenant settings. `dry_run=true` — только оценка без FS-изменений.
Записи audit с `actor_prefix=auto:scheduler`.
operationId: runRuntimeLogAutoCleanup
parameters:
- $ref: "#/components/parameters/TenantId"
- name: dry_run
in: query
schema:
type: boolean
default: false
responses:
"200":
description: Успешно.
content:
application/json:
schema:
$ref: "#/components/schemas/RuntimeLogAutoRunResult"
"503":
description: FS API недоступен.
content:
application/problem+json:
schema:
$ref: "#/components/schemas/Problem"
default:
$ref: "#/components/responses/DefaultProblem"
/v1/runtime-logs/cleanup-audit:
get:
tags: [RuntimeLogs]
+3
View File
@@ -18,6 +18,8 @@ type Env struct {
BrokerURL string
// RuntimeLogsDir is the absolute host path to Docker runtime log files (evobgp-all only).
RuntimeLogsDir string
// RuntimeLogsPolicyTenant overrides which tenant global_settings drive auto-cleanup (optional).
RuntimeLogsPolicyTenant string
}
// Load reads EVOBGP_* environment variables with safe defaults.
@@ -34,6 +36,7 @@ func Load() Env {
e.GitSHA = strings.TrimSpace(os.Getenv("EVOBGP_GIT_SHA"))
e.DatabaseURL = strings.TrimSpace(os.Getenv("EVOBGP_DATABASE_URL"))
e.BrokerURL = strings.TrimSpace(os.Getenv("EVOBGP_BROKER_URL"))
e.RuntimeLogsPolicyTenant = strings.TrimSpace(os.Getenv("EVOBGP_RUNTIME_LOGS_POLICY_TENANT"))
if v := strings.TrimSpace(os.Getenv("EVOBGP_RUNTIME_LOGS_DIR")); v != "" {
if abs, err := filepath.Abs(v); err == nil {
e.RuntimeLogsDir = abs
+2
View File
@@ -59,6 +59,8 @@ func (s *Server) registerV1(m *http.ServeMux) {
m.HandleFunc("POST /modules/{module_id}/refresh", s.handleModuleRefresh)
m.HandleFunc("POST /tenant/refresh", s.handleTenantRefresh)
m.HandleFunc("GET /revisions", s.handleListRevisions)
m.HandleFunc("GET /revisions/prune-estimate", s.handleRevisionPruneEstimate)
m.HandleFunc("POST /revisions/prune", s.handleRevisionPrune)
m.HandleFunc("GET /revisions/{revision_id}", s.handleGetRevision)
m.HandleFunc("GET /revisions/{revision_id}/preview", s.handleRevisionPreview)
m.HandleFunc("GET /revisions/{revision_id}/diagnostic-log", s.handleRevisionDiagnosticLog)
+5
View File
@@ -14,6 +14,7 @@ import (
"evobgp/internal/importer"
"evobgp/internal/pipeline"
"evobgp/internal/runtimelogs"
"evobgp/internal/store"
)
@@ -1117,6 +1118,10 @@ func (s *Server) handlePatchSettings(w http.ResponseWriter, r *http.Request) {
}
body["revision_retention_minutes"] = v
}
if !runtimelogs.ValidateRuntimeLogsSettingsPatch(body) {
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", "invalid runtime_logs_* settings")
return
}
if err := s.store.PatchGlobalSettings(a.TenantID, body); err != nil {
writeStoreErr(w, err)
return
+101
View File
@@ -0,0 +1,101 @@
package httpapi
import (
"encoding/json"
"net/http"
"strconv"
"time"
"evobgp/internal/pipeline"
)
func (s *Server) resolveRevisionRetentionMinutesQuery(r *http.Request, tenantID string) (minutes int, ok bool) {
if raw := r.URL.Query().Get("retention_minutes"); raw != "" {
n, err := strconv.Atoi(raw)
if err != nil {
return 0, false
}
if n < pipeline.RevisionMinTTLMin || n > pipeline.RevisionMaxTTLMin {
return 0, false
}
return n, true
}
return s.defaultRevisionRetentionMinutes(tenantID)
}
func (s *Server) defaultRevisionRetentionMinutes(tenantID string) (int, bool) {
settings, err := s.store.ListGlobalSettings(tenantID)
if err != nil {
return pipeline.ClampRevisionRetentionMinutes(0), true
}
return pipeline.ClampRevisionRetentionMinutes(pipeline.RevisionRetentionMinutesFromSettings(settings)), true
}
func (s *Server) resolveRevisionRetentionMinutesBody(r *http.Request, tenantID string) (minutes int, ok bool) {
var body struct {
RetentionMinutes *int `json:"retention_minutes"`
}
if r.Body != nil && r.ContentLength != 0 {
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
return 0, false
}
if body.RetentionMinutes != nil {
m := *body.RetentionMinutes
if m < pipeline.RevisionMinTTLMin || m > pipeline.RevisionMaxTTLMin {
return 0, false
}
return m, true
}
}
return s.defaultRevisionRetentionMinutes(tenantID)
}
func (s *Server) handleRevisionPruneEstimate(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") {
return
}
minutes, valid := s.resolveRevisionRetentionMinutesQuery(r, a.TenantID)
if !valid {
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", "retention_minutes must be an integer in range 15..43200")
return
}
cutoff := pipeline.RevisionCutoffFromMinutes(minutes)
est, err := s.store.EstimateRevisionPrune(a.TenantID, cutoff, minutes)
if err != nil {
writeInternalError(w, "revision prune estimate", err)
return
}
writeJSON(w, http.StatusOK, map[string]any{
"retention_minutes": est.RetentionMinutes,
"cutoff_at": est.CutoffAt.UTC().Format(time.RFC3339Nano),
"revision_count": est.RevisionCount,
"prefix_row_count": est.PrefixRowCount,
"orphan_snapshot_count": est.OrphanSnapshotCount,
"bytes_estimate": est.BytesEstimate,
})
}
func (s *Server) handleRevisionPrune(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "operator") {
return
}
minutes, valid := s.resolveRevisionRetentionMinutesBody(r, a.TenantID)
if !valid {
writeProblem(w, http.StatusUnprocessableEntity, "Unprocessable Entity", "retention_minutes must be an integer in range 15..43200")
return
}
cutoff := pipeline.RevisionCutoffFromMinutes(minutes)
res, err := s.store.PruneRevisionsWithStats(a.TenantID, cutoff)
if err != nil {
writeInternalError(w, "revision prune", err)
return
}
writeJSON(w, http.StatusOK, map[string]any{
"deleted_revisions": res.DeletedRevisions,
"deleted_prefix_snapshots": res.DeletedPrefixSnapshots,
"deleted_prefix_rows": res.DeletedPrefixRows,
"bytes_estimate": res.BytesEstimate,
})
}
@@ -0,0 +1,94 @@
package httpapi
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestRevisionPruneEstimateViewerOK(t *testing.T) {
srv, err := New(Options{InsecureDev: true, SeedDemo: true})
if err != nil {
t.Fatal(err)
}
defer srv.Close()
tenant, _, _, _, _ := srv.Store().DemoIDs()
mustSetTestAPIKeys(t, srv, "vwkey|"+tenant+"|viewer")
ts := httptest.NewServer(srv.Handler())
defer ts.Close()
req, _ := http.NewRequest(http.MethodGet, ts.URL+"/v1/revisions/prune-estimate?retention_minutes=15", nil)
req.Header.Set("Authorization", "Bearer vwkey")
resp, err := ts.Client().Do(req)
if err != nil {
t.Fatal(err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
b, _ := io.ReadAll(resp.Body)
t.Fatalf("status %d: %s", resp.StatusCode, b)
}
var body map[string]any
if err := json.NewDecoder(resp.Body).Decode(&body); err != nil {
t.Fatal(err)
}
if _, ok := body["revision_count"]; !ok {
t.Fatalf("missing revision_count: %v", body)
}
}
func TestRevisionPruneViewerForbidden(t *testing.T) {
srv, err := New(Options{InsecureDev: true, SeedDemo: true})
if err != nil {
t.Fatal(err)
}
defer srv.Close()
tenant, _, _, _, _ := srv.Store().DemoIDs()
mustSetTestAPIKeys(t, srv, "vwkey|"+tenant+"|viewer")
ts := httptest.NewServer(srv.Handler())
defer ts.Close()
req, _ := http.NewRequest(http.MethodPost, ts.URL+"/v1/revisions/prune", strings.NewReader(`{"retention_minutes":43200}`))
req.Header.Set("Authorization", "Bearer vwkey")
req.Header.Set("Content-Type", "application/json")
resp, err := ts.Client().Do(req)
if err != nil {
t.Fatal(err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusForbidden {
b, _ := io.ReadAll(resp.Body)
t.Fatalf("status %d: %s", resp.StatusCode, b)
}
}
func TestRevisionPruneOperatorOK(t *testing.T) {
srv, err := New(Options{InsecureDev: true, SeedDemo: true})
if err != nil {
t.Fatal(err)
}
defer srv.Close()
tenant, _, _, _, _ := srv.Store().DemoIDs()
mustSetTestAPIKeys(t, srv, "opkey|"+tenant+"|operator")
ts := httptest.NewServer(srv.Handler())
defer ts.Close()
req, _ := http.NewRequest(http.MethodPost, ts.URL+"/v1/revisions/prune", strings.NewReader(`{"retention_minutes":15}`))
req.Header.Set("Authorization", "Bearer opkey")
req.Header.Set("Content-Type", "application/json")
resp, err := ts.Client().Do(req)
if err != nil {
t.Fatal(err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
b, _ := io.ReadAll(resp.Body)
t.Fatalf("status %d: %s", resp.StatusCode, b)
}
}
+75
View File
@@ -14,6 +14,8 @@ func (s *Server) registerRuntimeLogsRoutes(m *http.ServeMux) {
m.HandleFunc("GET /runtime-logs/files/{filename}", s.handleGetRuntimeLogTail)
m.HandleFunc("DELETE /runtime-logs/files/{filename}", s.handleDeleteRuntimeLogFile)
m.HandleFunc("GET /runtime-logs/cleanup-audit", s.handleListRuntimeLogCleanupAudit)
m.HandleFunc("GET /runtime-logs/auto-estimate", s.handleRuntimeLogAutoEstimate)
m.HandleFunc("POST /runtime-logs/auto-run", s.handleRuntimeLogAutoRun)
}
func (s *Server) requireRuntimeLogs(w http.ResponseWriter) bool {
@@ -144,6 +146,79 @@ func (s *Server) handleDeleteRuntimeLogFile(w http.ResponseWriter, r *http.Reque
writeJSON(w, http.StatusOK, out)
}
func (s *Server) runtimeLogAutoPolicy(w http.ResponseWriter, r *http.Request, tenantID string) (runtimelogs.AutoPolicy, bool) {
settings, err := s.store.ListGlobalSettings(tenantID)
if err != nil {
writeInternalError(w, "runtime_logs_policy", err)
return runtimelogs.AutoPolicy{}, false
}
return runtimelogs.PolicyFromSettings(settings), true
}
func (s *Server) handleRuntimeLogAutoEstimate(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "operator") || !s.requireRuntimeLogs(w) {
return
}
policy, ok := s.runtimeLogAutoPolicy(w, r, a.TenantID)
if !ok {
return
}
items, err := runtimelogs.EstimateAutoCleanup(s.runtimeLogs, policy)
if err != nil {
writeRuntimeLogsErr(w, "runtime_logs_auto_estimate", err)
return
}
out := make([]map[string]any, 0, len(items))
var wouldCount int
for _, it := range items {
if it.WouldCleanup {
wouldCount++
}
row := map[string]any{
"filename": it.Filename,
"size_bytes": it.SizeBytes,
"would_cleanup": it.WouldCleanup,
}
if it.SkipReason != "" {
row["skip_reason"] = it.SkipReason
}
out = append(out, row)
}
writeJSON(w, http.StatusOK, map[string]any{
"policy": map[string]any{
"enabled": policy.Enabled,
"max_file_bytes": policy.MaxFileBytes,
"schedule": policy.Schedule,
"mode": policy.Mode,
},
"items": out,
"would_count": wouldCount,
})
}
func (s *Server) handleRuntimeLogAutoRun(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "operator") || !s.requireRuntimeLogs(w) {
return
}
policy, ok := s.runtimeLogAutoPolicy(w, r, a.TenantID)
if !ok {
return
}
dryRun := r.URL.Query().Get("dry_run") == "true"
result, err := runtimelogs.RunAutoCleanup(r.Context(), runtimelogs.AutoCleanupDeps{
Service: s.runtimeLogs,
Store: s.store,
TenantID: a.TenantID,
}, policy, dryRun, "manual")
if err != nil {
writeRuntimeLogsErr(w, "runtime_logs_auto_run", err)
return
}
writeJSON(w, http.StatusOK, result)
}
func (s *Server) handleListRuntimeLogCleanupAudit(w http.ResponseWriter, r *http.Request) {
a, ok := authFromContext(r.Context())
if !ok || !s.requireAtLeast(w, a, "viewer") {
+43 -1
View File
@@ -77,7 +77,7 @@ func TestRuntimeLogsHappyPath(t *testing.T) {
ServiceName: runtimelogs.ServiceNameAll,
})
handler := srv.Handler()
tenant := "00000000-0000-0000-0000-000000000001"
tenant, _, _, _, _ := srv.Store().DemoIDs()
mustSetTestAPIKeys(t, srv, "vwkey|"+tenant+"|viewer,opkey|"+tenant+"|operator")
t.Run("list", func(t *testing.T) {
@@ -146,4 +146,46 @@ func TestRuntimeLogsHappyPath(t *testing.T) {
t.Fatalf("expected audit entry, body=%s", auditRec.Body.String())
}
})
t.Run("auto estimate and run", func(t *testing.T) {
bigPath := filepath.Join(dir, "big.log")
if err := os.WriteFile(bigPath, make([]byte, 2*1024*1024), 0o644); err != nil {
t.Fatal(err)
}
if err := srv.store.PatchGlobalSettings(tenant, map[string]any{
runtimelogs.KeyMaxFileMB: 1,
runtimelogs.KeyAutoMode: "truncate",
}); err != nil {
t.Fatal(err)
}
estReq := httptest.NewRequest(http.MethodGet, "/v1/runtime-logs/auto-estimate", nil)
estReq.Header.Set("Authorization", "Bearer opkey")
estRec := httptest.NewRecorder()
handler.ServeHTTP(estRec, estReq)
if estRec.Code != http.StatusOK {
t.Fatalf("estimate status=%d body=%s", estRec.Code, estRec.Body.String())
}
if !strings.Contains(estRec.Body.String(), `"would_count":1`) {
t.Fatalf("expected would_count=1, body=%s", estRec.Body.String())
}
runReq := httptest.NewRequest(http.MethodPost, "/v1/runtime-logs/auto-run", nil)
runReq.Header.Set("Authorization", "Bearer opkey")
runRec := httptest.NewRecorder()
handler.ServeHTTP(runRec, runReq)
if runRec.Code != http.StatusOK {
t.Fatalf("auto-run status=%d body=%s", runRec.Code, runRec.Body.String())
}
if !strings.Contains(runRec.Body.String(), "auto:scheduler") && !strings.Contains(runRec.Body.String(), `"cleaned_count":1`) {
t.Fatalf("unexpected auto-run body=%s", runRec.Body.String())
}
st, err := os.Stat(bigPath)
if err != nil {
t.Fatal(err)
}
if st.Size() != 0 {
t.Fatalf("expected truncated big.log, size=%d", st.Size())
}
})
}
+34 -23
View File
@@ -21,18 +21,19 @@ import (
// Server implements EvoBGP control-plane HTTP API.
type Server struct {
store store.Backend
pgPool *pgxpool.Pool
pgMonitor *pgmonitor.Service
maintConfig *maintenance.ConfigProvider
maintStats *maintenance.DBStatsProvider
jobs *jobs.Registry
bundlePriv ed25519.PrivateKey
keyResolver *apiKeyResolver
corsOrigins []string
cdnHTTP *http.Client
runtimeLogs *runtimelogs.Service
mux *http.ServeMux
store store.Backend
pgPool *pgxpool.Pool
pgMonitor *pgmonitor.Service
maintConfig *maintenance.ConfigProvider
maintStats *maintenance.DBStatsProvider
jobs *jobs.Registry
bundlePriv ed25519.PrivateKey
keyResolver *apiKeyResolver
corsOrigins []string
cdnHTTP *http.Client
runtimeLogs *runtimelogs.Service
runtimeLogsPolicyTenant string
mux *http.ServeMux
}
// Options configures the API server.
@@ -44,6 +45,8 @@ type Options struct {
SeedDemo bool
BundleSeedHex string
CORSAllowedOrigins string
// RuntimeLogsPolicyTenant overrides tenant for auto-cleanup scheduler settings (optional).
RuntimeLogsPolicyTenant string
}
// New constructs Server and wiring for async jobs.
@@ -81,17 +84,18 @@ func New(opts Options) (*Server, error) {
maintStats = maintenance.NewDBStatsProvider(pgMon)
}
s := &Server{
store: backend,
pgPool: pool,
pgMonitor: pgMon,
maintConfig: maintCfg,
maintStats: maintStats,
jobs: reg,
bundlePriv: priv,
keyResolver: resolver,
corsOrigins: parseCORSOrigins(opts.CORSAllowedOrigins),
cdnHTTP: NewCDNHTTPClient(),
runtimeLogs: runtimelogs.NewService(runtimelogs.ConfigFromEnv()),
store: backend,
pgPool: pool,
pgMonitor: pgMon,
maintConfig: maintCfg,
maintStats: maintStats,
jobs: reg,
bundlePriv: priv,
keyResolver: resolver,
corsOrigins: parseCORSOrigins(opts.CORSAllowedOrigins),
cdnHTTP: NewCDNHTTPClient(),
runtimeLogs: runtimelogs.NewService(runtimelogs.ConfigFromEnv()),
runtimeLogsPolicyTenant: strings.TrimSpace(opts.RuntimeLogsPolicyTenant),
}
s.mux = http.NewServeMux()
s.registerRoutes()
@@ -126,4 +130,11 @@ func (s *Server) StartBackground(ctx context.Context) {
})
}, 30*time.Second)
}
if s != nil && s.runtimeLogs != nil && s.store != nil {
runtimelogs.StartAutoCleanupScheduler(ctx, runtimelogs.SchedulerDeps{
Service: s.runtimeLogs,
Store: s.store,
PolicyTenant: s.runtimeLogsPolicyTenant,
}, 30*time.Second)
}
}
+2 -15
View File
@@ -31,10 +31,7 @@ const (
birdFilterNameV4 = "evobgp_export_v4"
birdFilterNameV6 = "evobgp_export_v6"
auxBirdFullExpanded = "_bird_full_expanded.conf"
revisionTTLKey = "revision_retention_minutes"
revisionMinTTLMin = 15
revisionMaxTTLMin = 30 * 24 * 60
revisionDefaultTTL = 30 * 24 * time.Hour
revisionTTLKey = RevisionRetentionKey
)
// AuxBirdFullExpandedKey returns the preview map key for the expanded BIRD config (generated on demand).
@@ -969,17 +966,7 @@ func applyRevisionRetention(st store.Backend, tenantID string) {
if err != nil {
return
}
ttl := revisionDefaultTTL
if minutes := intFromSettingsMap(settings, revisionTTLKey); minutes > 0 {
if minutes < revisionMinTTLMin {
minutes = revisionMinTTLMin
}
if minutes > revisionMaxTTLMin {
minutes = revisionMaxTTLMin
}
ttl = time.Duration(minutes) * time.Minute
}
cutoff := time.Now().UTC().Add(-ttl)
cutoff := RevisionRetentionCutoff(settings)
_, _ = st.PruneRevisionsBefore(tenantID, cutoff)
}
+75
View File
@@ -0,0 +1,75 @@
package pipeline
import (
"strconv"
"strings"
"time"
)
const (
// RevisionRetentionKey is the global_settings KV for revision TTL (minutes).
RevisionRetentionKey = "revision_retention_minutes"
RevisionMinTTLMin = 15
RevisionMaxTTLMin = 30 * 24 * 60
RevisionDefaultTTL = 30 * 24 * time.Hour
)
// RevisionCutoffFromMinutes returns created_at cutoff for revision prune (UTC now minus clamped TTL).
func RevisionCutoffFromMinutes(minutes int) time.Time {
ttl := RevisionDefaultTTL
if minutes > 0 {
m := minutes
if m < RevisionMinTTLMin {
m = RevisionMinTTLMin
}
if m > RevisionMaxTTLMin {
m = RevisionMaxTTLMin
}
ttl = time.Duration(m) * time.Minute
}
return time.Now().UTC().Add(-ttl)
}
// RevisionRetentionMinutesFromSettings reads revision_retention_minutes from tenant KV (0 if unset).
func RevisionRetentionMinutesFromSettings(settings map[string]any) int {
if settings == nil {
return 0
}
v, ok := settings[RevisionRetentionKey]
if !ok || v == nil {
return 0
}
switch x := v.(type) {
case float64:
return int(x)
case int:
return x
case int64:
return int(x)
case string:
n, err := strconv.Atoi(strings.TrimSpace(x))
if err == nil {
return n
}
}
return 0
}
// RevisionRetentionCutoff resolves tenant revision_retention_minutes from settings (default 30d).
func RevisionRetentionCutoff(settings map[string]any) time.Time {
return RevisionCutoffFromMinutes(RevisionRetentionMinutesFromSettings(settings))
}
// ClampRevisionRetentionMinutes normalizes user input to the allowed range.
func ClampRevisionRetentionMinutes(minutes int) int {
if minutes <= 0 {
return int(RevisionDefaultTTL / time.Minute)
}
if minutes < RevisionMinTTLMin {
return RevisionMinTTLMin
}
if minutes > RevisionMaxTTLMin {
return RevisionMaxTTLMin
}
return minutes
}
@@ -0,0 +1,30 @@
package pipeline
import (
"testing"
"time"
)
func TestClampRevisionRetentionMinutes(t *testing.T) {
if got := ClampRevisionRetentionMinutes(0); got != int(RevisionDefaultTTL/time.Minute) {
t.Fatalf("default: got %d", got)
}
if got := ClampRevisionRetentionMinutes(5); got != RevisionMinTTLMin {
t.Fatalf("min clamp: got %d", got)
}
if got := ClampRevisionRetentionMinutes(999999); got != RevisionMaxTTLMin {
t.Fatalf("max clamp: got %d", got)
}
if got := ClampRevisionRetentionMinutes(120); got != 120 {
t.Fatalf("unchanged: got %d", got)
}
}
func TestRevisionCutoffFromMinutes(t *testing.T) {
before := time.Now().UTC()
cutoff := RevisionCutoffFromMinutes(60)
after := time.Now().UTC().Add(-59 * time.Minute)
if cutoff.After(before.Add(-59*time.Minute)) || cutoff.Before(after.Add(-2*time.Minute)) {
t.Fatalf("cutoff out of range: %v", cutoff)
}
}
-37
View File
@@ -970,43 +970,6 @@ func (p *Postgres) RevisionDiff(tenantID, aID, bID string) (map[string]any, erro
}, nil
}
func (p *Postgres) PruneRevisionsBefore(tenantID string, cutoff time.Time) (int, error) {
ctx := context.Background()
total := 0
const batchSize = 50
for {
cmd, err := p.pool.Exec(ctx, `
DELETE FROM config_revision AS cr
WHERE cr.id IN (
SELECT id FROM config_revision
WHERE tenant_id = $1
AND created_at < $2
AND id <> (
SELECT id FROM config_revision
WHERE tenant_id = $1
ORDER BY created_at DESC
LIMIT 1
)
AND NOT EXISTS (
SELECT 1 FROM bgp_speaker AS sp
WHERE sp.tenant_id = $1
AND (sp.last_applied_revision_id = config_revision.id OR sp.published_revision_id = config_revision.id)
)
ORDER BY created_at ASC
LIMIT $3
)`, tenantID, cutoff.UTC(), batchSize)
if err != nil {
return total, err
}
n := int(cmd.RowsAffected())
total += n
if n < batchSize {
break
}
}
return total, nil
}
func (p *Postgres) SetLastAppliedRevision(tenantID, speakerID, revisionID string) error {
ctx := context.Background()
tag, err := p.pool.Exec(ctx, `
+23 -15
View File
@@ -53,21 +53,8 @@ func (p *Postgres) ensurePrefixSnapshot(ctx context.Context, db execQuerier, con
return "", nil
}
hash := normalizeSnapshotHash(contentHash)
var existing string
err := db.QueryRow(ctx, `SELECT id::text FROM prefix_snapshot WHERE content_hash = $1`, hash).Scan(&existing)
if err == nil && existing != "" {
return existing, nil
}
if err != nil && !errors.Is(err, pgx.ErrNoRows) {
return "", err
}
snapID := uuid.NewString()
if _, err := db.Exec(ctx, `
INSERT INTO prefix_snapshot (id, content_hash) VALUES ($1::uuid, $2)
ON CONFLICT (content_hash) DO NOTHING`, snapID, hash); err != nil {
return "", err
}
if err := db.QueryRow(ctx, `SELECT id::text FROM prefix_snapshot WHERE content_hash = $1`, hash).Scan(&snapID); err != nil {
snapID, err := p.resolvePrefixSnapshotID(ctx, db, hash)
if err != nil {
return "", err
}
var rowCount int
@@ -94,6 +81,27 @@ func (p *Postgres) ensurePrefixSnapshot(ctx context.Context, db execQuerier, con
return snapID, nil
}
func (p *Postgres) resolvePrefixSnapshotID(ctx context.Context, db execQuerier, hash string) (string, error) {
var existing string
err := db.QueryRow(ctx, `SELECT id::text FROM prefix_snapshot WHERE content_hash = $1`, hash).Scan(&existing)
if err == nil && existing != "" {
return existing, nil
}
if err != nil && !errors.Is(err, pgx.ErrNoRows) {
return "", err
}
snapID := uuid.NewString()
if _, err := db.Exec(ctx, `
INSERT INTO prefix_snapshot (id, content_hash) VALUES ($1::uuid, $2)
ON CONFLICT (content_hash) DO NOTHING`, snapID, hash); err != nil {
return "", err
}
if err := db.QueryRow(ctx, `SELECT id::text FROM prefix_snapshot WHERE content_hash = $1`, hash).Scan(&snapID); err != nil {
return "", err
}
return snapID, nil
}
func (p *Postgres) listSnapshotPrefixes(ctx context.Context, snapshotID, cursor string, limit int) ([]store.PrefixRow, string, bool) {
afterOrd, off, useOffset := store.ParsePrefixPageCursor(cursor)
var rows pgx.Rows
@@ -0,0 +1,119 @@
package repository
import (
"context"
"os"
"testing"
"evobgp/internal/db"
"evobgp/internal/store"
"github.com/google/uuid"
)
func TestEnsurePrefixSnapshotFillsEmptyExistingIntegration(t *testing.T) {
dsn := os.Getenv("EVOBGP_TEST_DATABASE_URL")
if dsn == "" {
t.Skip("EVOBGP_TEST_DATABASE_URL not set")
}
ctx := context.Background()
pool, err := db.OpenPostgresPool(ctx, dsn)
if err != nil {
t.Fatal(err)
}
defer pool.Close()
pg, err := NewPostgres(ctx, pool, false)
if err != nil {
t.Fatal(err)
}
hash := normalizeSnapshotHash("sha256:empty-fill-" + uuid.NewString())
snapID := uuid.NewString()
if _, err := pool.Exec(ctx, `INSERT INTO prefix_snapshot (id, content_hash) VALUES ($1::uuid, $2)`, snapID, hash); err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_, _ = pool.Exec(ctx, `DELETE FROM prefix_snapshot WHERE id = $1::uuid`, snapID)
})
tx, err := pool.Begin(ctx)
if err != nil {
t.Fatal(err)
}
defer func() { _ = tx.Rollback(ctx) }()
got, err := pg.ensurePrefixSnapshot(ctx, tx, "sha256:"+hash, []store.PrefixRow{
{Prefix: "203.0.113.1/32", Source: "test"},
})
if err != nil {
t.Fatal(err)
}
if got != snapID {
t.Fatalf("snap id: got %q want %q", got, snapID)
}
var rowCount int
if err := tx.QueryRow(ctx, `SELECT COUNT(*)::int FROM prefix_snapshot_row WHERE snapshot_id = $1::uuid`, snapID).Scan(&rowCount); err != nil {
t.Fatal(err)
}
if rowCount != 1 {
t.Fatalf("row count: got %d", rowCount)
}
if err := tx.Commit(ctx); err != nil {
t.Fatal(err)
}
}
func TestEnsurePrefixSnapshotIdempotentIntegration(t *testing.T) {
dsn := os.Getenv("EVOBGP_TEST_DATABASE_URL")
if dsn == "" {
t.Skip("EVOBGP_TEST_DATABASE_URL not set")
}
ctx := context.Background()
pool, err := db.OpenPostgresPool(ctx, dsn)
if err != nil {
t.Fatal(err)
}
defer pool.Close()
pg, err := NewPostgres(ctx, pool, false)
if err != nil {
t.Fatal(err)
}
contentHash := "sha256:idempotent-" + uuid.NewString()
prefixes := []store.PrefixRow{{Prefix: "198.51.100.0/24", Source: "test"}}
tx1, err := pool.Begin(ctx)
if err != nil {
t.Fatal(err)
}
id1, err := pg.ensurePrefixSnapshot(ctx, tx1, contentHash, prefixes)
if err != nil {
t.Fatal(err)
}
if err := tx1.Commit(ctx); err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_, _ = pool.Exec(ctx, `DELETE FROM prefix_snapshot WHERE id = $1::uuid`, id1)
})
tx2, err := pool.Begin(ctx)
if err != nil {
t.Fatal(err)
}
defer func() { _ = tx2.Rollback(ctx) }()
id2, err := pg.ensurePrefixSnapshot(ctx, tx2, contentHash, prefixes)
if err != nil {
t.Fatal(err)
}
if id1 != id2 {
t.Fatalf("ids differ: %q vs %q", id1, id2)
}
var rowCount int
if err := tx2.QueryRow(ctx, `SELECT COUNT(*)::int FROM prefix_snapshot_row WHERE snapshot_id = $1::uuid`, id1).Scan(&rowCount); err != nil {
t.Fatal(err)
}
if rowCount != 1 {
t.Fatalf("expected 1 row, got %d", rowCount)
}
}
@@ -0,0 +1,176 @@
package repository
import (
"context"
"time"
"evobgp/internal/store"
)
const revisionPruneBatchSize = 50
// sqlPrunableRevisionsWhere appends prunable revision predicates (tenant + cutoff params).
func sqlPrunableRevisionsWhere(tenantParam, cutoffParam string) string {
return `tenant_id = ` + tenantParam + `
AND created_at < ` + cutoffParam + `
AND id <> (
SELECT id FROM config_revision
WHERE tenant_id = ` + tenantParam + `
ORDER BY created_at DESC
LIMIT 1
)
AND NOT EXISTS (
SELECT 1 FROM bgp_speaker AS sp
WHERE sp.tenant_id = ` + tenantParam + `
AND (sp.last_applied_revision_id = config_revision.id OR sp.published_revision_id = config_revision.id)
)`
}
func (p *Postgres) EstimateRevisionPrune(tenantID string, cutoff time.Time, retentionMinutes int) (store.RevisionPruneEstimate, error) {
ctx := context.Background()
out := store.RevisionPruneEstimate{
RetentionMinutes: retentionMinutes,
CutoffAt: cutoff.UTC(),
}
where := sqlPrunableRevisionsWhere("$1", "$2")
var revBytes, previewBytes, rmpBytes, snapRowBytes int64
err := p.pool.QueryRow(ctx, `
WITH prunable AS (
SELECT id, prefix_snapshot_id, content_hash, meta_json
FROM config_revision
WHERE `+where+`
),
freed_snaps AS (
SELECT DISTINCT pr.prefix_snapshot_id AS snap_id
FROM prunable pr
WHERE pr.prefix_snapshot_id IS NOT NULL
EXCEPT
SELECT DISTINCT cr.prefix_snapshot_id
FROM config_revision cr
WHERE cr.prefix_snapshot_id IS NOT NULL
AND cr.id NOT IN (SELECT id FROM prunable)
)
SELECT
(SELECT COUNT(*)::int FROM prunable),
COALESCE((SELECT SUM(octet_length(content_hash) + octet_length(meta_json::text))::bigint FROM prunable), 0),
COALESCE((
SELECT SUM(octet_length(value))::bigint
FROM config_revision_preview p
JOIN prunable pr ON pr.id = p.revision_id
CROSS JOIN LATERAL jsonb_each_text(COALESCE(p.fragments, '{}'::jsonb))
), 0),
COALESCE((
SELECT SUM(octet_length(rmp.prefix) + octet_length(COALESCE(rmp.source, '')))::bigint
FROM revision_materialized_prefix rmp
WHERE rmp.revision_id IN (SELECT id FROM prunable)
), 0),
(SELECT COUNT(*)::int FROM freed_snaps),
COALESCE((SELECT COUNT(*)::int FROM prefix_snapshot_row psr WHERE psr.snapshot_id IN (SELECT snap_id FROM freed_snaps)), 0),
COALESCE((
SELECT SUM(octet_length(psr.prefix) + octet_length(COALESCE(psr.source, '')))::bigint
FROM prefix_snapshot_row psr
WHERE psr.snapshot_id IN (SELECT snap_id FROM freed_snaps)
), 0)`,
tenantID, cutoff.UTC()).Scan(
&out.RevisionCount,
&revBytes,
&previewBytes,
&rmpBytes,
&out.OrphanSnapshotCount,
&out.PrefixRowCount,
&snapRowBytes,
)
if err != nil {
return out, err
}
out.BytesEstimate = revBytes + previewBytes + rmpBytes + snapRowBytes
return out, nil
}
func (p *Postgres) PruneRevisionsWithStats(tenantID string, cutoff time.Time) (store.RevisionPruneResult, error) {
ctx := context.Background()
est, err := p.EstimateRevisionPrune(tenantID, cutoff, 0)
if err != nil {
return store.RevisionPruneResult{}, err
}
out := store.RevisionPruneResult{BytesEstimate: est.BytesEstimate}
where := sqlPrunableRevisionsWhere("$1", "$2")
for {
cmd, err := p.pool.Exec(ctx, `
DELETE FROM config_revision AS cr
WHERE cr.id IN (
SELECT id FROM config_revision
WHERE `+where+`
ORDER BY created_at ASC
LIMIT $3
)`, tenantID, cutoff.UTC(), revisionPruneBatchSize)
if err != nil {
return out, err
}
n := int(cmd.RowsAffected())
out.DeletedRevisions += n
if n < revisionPruneBatchSize {
break
}
}
for {
snaps, rows, err := p.pruneUnreferencedPrefixSnapshots(ctx, revisionPruneBatchSize)
if err != nil {
return out, err
}
out.DeletedPrefixSnapshots += snaps
out.DeletedPrefixRows += rows
if snaps < revisionPruneBatchSize {
break
}
}
return out, nil
}
func (p *Postgres) PruneRevisionsBefore(tenantID string, cutoff time.Time) (int, error) {
res, err := p.PruneRevisionsWithStats(tenantID, cutoff)
if err != nil {
return 0, err
}
return res.DeletedRevisions, nil
}
func (p *Postgres) pruneUnreferencedPrefixSnapshots(ctx context.Context, batchSize int) (deletedSnapshots int, deletedRows int, err error) {
if !prefixSnapshotTableExists(ctx, p.pool) {
return 0, 0, nil
}
var snapCount, rowCount int
err = p.pool.QueryRow(ctx, `
WITH doomed AS (
SELECT ps.id FROM prefix_snapshot ps
WHERE NOT EXISTS (
SELECT 1 FROM config_revision cr WHERE cr.prefix_snapshot_id = ps.id
)
LIMIT $1
)
SELECT
(SELECT COUNT(*)::int FROM doomed),
(SELECT COUNT(*)::int FROM prefix_snapshot_row psr WHERE psr.snapshot_id IN (SELECT id FROM doomed))`,
batchSize).Scan(&snapCount, &rowCount)
if err != nil {
return 0, 0, err
}
if snapCount == 0 {
return 0, 0, nil
}
_, err = p.pool.Exec(ctx, `
DELETE FROM prefix_snapshot
WHERE id IN (
SELECT ps.id FROM prefix_snapshot ps
WHERE NOT EXISTS (
SELECT 1 FROM config_revision cr WHERE cr.prefix_snapshot_id = ps.id
)
LIMIT $1
)`, batchSize)
if err != nil {
return 0, 0, err
}
return snapCount, rowCount, nil
}
@@ -0,0 +1,83 @@
package repository
import (
"context"
"os"
"testing"
"evobgp/internal/db"
"evobgp/internal/pipeline"
"github.com/google/uuid"
)
func TestPostgresPruneUnreferencedPrefixSnapshotsIntegration(t *testing.T) {
dsn := os.Getenv("EVOBGP_TEST_DATABASE_URL")
if dsn == "" {
t.Skip("EVOBGP_TEST_DATABASE_URL not set")
}
ctx := context.Background()
pool, err := db.OpenPostgresPool(ctx, dsn)
if err != nil {
t.Fatal(err)
}
defer pool.Close()
pg, err := NewPostgres(ctx, pool, false)
if err != nil {
t.Fatal(err)
}
snapID := uuid.NewString()
hash := normalizeSnapshotHash("sha256:test-orphan-" + snapID)
if _, err := pool.Exec(ctx, `INSERT INTO prefix_snapshot (id, content_hash) VALUES ($1::uuid, $2)`, snapID, hash); err != nil {
t.Fatal(err)
}
if _, err := pool.Exec(ctx, `
INSERT INTO prefix_snapshot_row (snapshot_id, ord, prefix, source)
VALUES ($1::uuid, 0, '203.0.113.0/24', 'test')`, snapID); err != nil {
t.Fatal(err)
}
snaps, rows, err := pg.pruneUnreferencedPrefixSnapshots(ctx, 50)
if err != nil {
t.Fatal(err)
}
if snaps < 1 || rows < 1 {
t.Fatalf("expected orphan cleanup, got snaps=%d rows=%d", snaps, rows)
}
var n int
if err := pool.QueryRow(ctx, `SELECT COUNT(*)::int FROM prefix_snapshot WHERE id = $1::uuid`, snapID).Scan(&n); err != nil {
t.Fatal(err)
}
if n != 0 {
t.Fatalf("snapshot still exists")
}
}
func TestPostgresEstimateRevisionPruneIntegration(t *testing.T) {
dsn := os.Getenv("EVOBGP_TEST_DATABASE_URL")
if dsn == "" {
t.Skip("EVOBGP_TEST_DATABASE_URL not set")
}
ctx := context.Background()
pool, err := db.OpenPostgresPool(ctx, dsn)
if err != nil {
t.Fatal(err)
}
defer pool.Close()
pg, err := NewPostgres(ctx, pool, false)
if err != nil {
t.Fatal(err)
}
tenant := uuid.NewString()
cutoff := pipeline.RevisionCutoffFromMinutes(15)
est, err := pg.EstimateRevisionPrune(tenant, cutoff, 15)
if err != nil {
t.Fatal(err)
}
if est.RevisionCount != 0 {
t.Fatalf("expected 0 revisions for empty tenant, got %d", est.RevisionCount)
}
}
+1 -1
View File
@@ -67,7 +67,7 @@ func (p *Postgres) ListRuntimeLogCleanupAudit(tenantID, cursor string, limit int
var sizeAfter *int64
if err := rows.Scan(&r.ID, &r.TenantID, &r.ActorPrefix, &r.Filename, &r.Action,
&r.SizeBefore, &sizeAfter, &detailRaw, &r.CreatedAt); err != nil {
continue
return nil, "", false, err
}
r.SizeAfter = sizeAfter
if len(detailRaw) > 0 {
+147
View File
@@ -0,0 +1,147 @@
package runtimelogs
import (
"context"
"fmt"
"evobgp/internal/store"
)
const autoSchedulerActor = "auto:scheduler"
// FileEstimate describes whether a log file would be cleaned by auto policy.
type FileEstimate struct {
Filename string `json:"filename"`
SizeBytes int64 `json:"size_bytes"`
WouldCleanup bool `json:"would_cleanup"`
SkipReason string `json:"skip_reason,omitempty"`
}
// AutoCleanupDeps wires FS cleanup with audit persistence.
type AutoCleanupDeps struct {
Service *Service
Store store.Backend
TenantID string
}
// EstimateAutoCleanup lists files that exceed the configured size threshold.
func EstimateAutoCleanup(svc *Service, policy AutoPolicy) ([]FileEstimate, error) {
if svc == nil || !svc.Available() {
return nil, ErrUnavailable
}
files, err := svc.ListFiles()
if err != nil {
return nil, err
}
out := make([]FileEstimate, 0, len(files))
for _, f := range files {
est := FileEstimate{
Filename: f.Name,
SizeBytes: f.SizeBytes,
}
if f.SizeBytes <= policy.MaxFileBytes {
est.SkipReason = "under_threshold"
} else if f.SizeBytes > MaxCleanupBytes {
est.SkipReason = "too_large"
} else {
est.WouldCleanup = true
}
out = append(out, est)
}
return out, nil
}
// RunAutoCleanup applies auto policy to eligible files and writes audit rows.
func RunAutoCleanup(ctx context.Context, deps AutoCleanupDeps, policy AutoPolicy, dryRun bool, trigger string) (map[string]any, error) {
if deps.Service == nil || !deps.Service.Available() {
return nil, ErrUnavailable
}
if deps.Store == nil || deps.TenantID == "" {
return nil, fmt.Errorf("runtimelogs: store tenant required for auto cleanup")
}
_ = ctx
estimates, err := EstimateAutoCleanup(deps.Service, policy)
if err != nil {
return nil, err
}
detailBase := map[string]any{
"trigger": trigger,
"dry_run": dryRun,
"max_file_bytes": policy.MaxFileBytes,
"mode": policy.Mode,
"files_processed": 0,
}
cleaned := make([]map[string]any, 0)
skipped := make([]map[string]any, 0)
for _, est := range estimates {
if !est.WouldCleanup {
if est.SkipReason != "" {
skipped = append(skipped, map[string]any{
"filename": est.Filename,
"reason": est.SkipReason,
"size": est.SizeBytes,
})
}
continue
}
if dryRun {
cleaned = append(cleaned, map[string]any{
"filename": est.Filename,
"size": est.SizeBytes,
"dry_run": true,
})
continue
}
sizeBefore, sizeAfter, err := deps.Service.Cleanup(est.Filename, policy.Mode)
if err != nil {
skipped = append(skipped, map[string]any{
"filename": est.Filename,
"reason": err.Error(),
"size": est.SizeBytes,
})
continue
}
auditDetail := map[string]any{
"trigger": trigger,
"mode": policy.Mode,
}
auditID, err := deps.Store.AppendRuntimeLogCleanupAudit(
deps.TenantID, autoSchedulerActor, est.Filename, policy.Mode, sizeBefore, sizeAfter, auditDetail)
if err != nil {
return nil, fmt.Errorf("runtimelogs: audit: %w", err)
}
entry := map[string]any{
"filename": est.Filename,
"audit_id": auditID,
"size_before": sizeBefore,
"action": policy.Mode,
}
if sizeAfter != nil {
entry["size_after"] = *sizeAfter
}
cleaned = append(cleaned, entry)
}
detailBase["files_processed"] = len(cleaned)
return map[string]any{
"dry_run": dryRun,
"trigger": trigger,
"policy": policySnapshot(policy),
"cleaned": cleaned,
"skipped": skipped,
"cleaned_count": len(cleaned),
"skipped_count": len(skipped),
}, nil
}
func policySnapshot(p AutoPolicy) map[string]any {
return map[string]any{
"enabled": p.Enabled,
"max_file_bytes": p.MaxFileBytes,
"schedule": p.Schedule,
"mode": p.Mode,
}
}
+81
View File
@@ -0,0 +1,81 @@
package runtimelogs
import (
"context"
"os"
"path/filepath"
"testing"
"evobgp/internal/store"
)
func TestEstimateAndRunAutoCleanup(t *testing.T) {
dir := t.TempDir()
small := filepath.Join(dir, "small.log")
large := filepath.Join(dir, "large.log")
if err := os.WriteFile(small, []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
payload := make([]byte, 2*1024*1024)
if err := os.WriteFile(large, payload, 0o644); err != nil {
t.Fatal(err)
}
svc := NewService(Config{RootDir: dir, ServiceName: ServiceNameAll})
policy := AutoPolicy{
Enabled: true,
MaxFileBytes: 1024 * 1024,
Mode: store.RuntimeLogCleanupTruncate,
}
est, err := EstimateAutoCleanup(svc, policy)
if err != nil {
t.Fatal(err)
}
if len(est) != 2 {
t.Fatalf("estimates=%d", len(est))
}
var would int
for _, e := range est {
if e.WouldCleanup {
would++
if e.Filename != "large.log" {
t.Fatalf("unexpected cleanup target %q", e.Filename)
}
}
}
if would != 1 {
t.Fatalf("would=%d", would)
}
mem := store.NewMemory()
tenant := "tenant-a"
result, err := RunAutoCleanup(context.Background(), AutoCleanupDeps{
Service: svc,
Store: mem,
TenantID: tenant,
}, policy, false, "test")
if err != nil {
t.Fatal(err)
}
if result["cleaned_count"] != 1 {
t.Fatalf("cleaned=%v", result["cleaned_count"])
}
st, err := os.Stat(large)
if err != nil {
t.Fatal(err)
}
if st.Size() != 0 {
t.Fatalf("expected truncated large.log, size=%d", st.Size())
}
items, _, _, err := mem.ListRuntimeLogCleanupAudit(tenant, "", 10)
if err != nil {
t.Fatal(err)
}
if len(items) != 1 {
t.Fatalf("audit rows=%d", len(items))
}
if items[0].ActorPrefix != autoSchedulerActor {
t.Fatalf("actor=%q", items[0].ActorPrefix)
}
}
+225
View File
@@ -0,0 +1,225 @@
package runtimelogs
import (
"fmt"
"strconv"
"strings"
"evobgp/internal/store"
"github.com/robfig/cron/v3"
)
// global_settings keys for runtime log auto-cleanup.
const (
KeyAutoEnabled = "runtime_logs_auto_enabled"
KeyMaxFileMB = "runtime_logs_max_file_mb"
KeyAutoSchedule = "runtime_logs_auto_schedule"
KeyAutoMode = "runtime_logs_auto_mode"
DefaultMaxFileMB = 128
DefaultSchedule = "0 */6 * * *"
)
// AutoPolicy is tenant KV configuration for scheduled FS log cleanup.
type AutoPolicy struct {
Enabled bool
MaxFileBytes int64
Schedule string
Mode string
}
// DefaultAutoPolicy returns policy defaults when settings are unset.
func DefaultAutoPolicy() AutoPolicy {
return AutoPolicy{
Enabled: false,
MaxFileBytes: int64(DefaultMaxFileMB) * 1024 * 1024,
Schedule: DefaultSchedule,
Mode: store.RuntimeLogCleanupTruncate,
}
}
// PolicyFromSettings reads auto-cleanup policy from tenant global_settings KV.
func PolicyFromSettings(settings map[string]any) AutoPolicy {
p := DefaultAutoPolicy()
if settings == nil {
return p
}
if v, ok := settings[KeyAutoEnabled]; ok {
p.Enabled = parseBoolSetting(v)
}
if mb := parseIntSetting(settings[KeyMaxFileMB]); mb > 0 {
p.MaxFileBytes = int64(ClampMaxFileMB(mb)) * 1024 * 1024
}
if s := parseStringSetting(settings[KeyAutoSchedule]); s != "" {
p.Schedule = s
}
if m := parseStringSetting(settings[KeyAutoMode]); store.ValidRuntimeLogCleanupAction(m) {
p.Mode = m
}
return p
}
// ClampMaxFileMB normalizes max file size to 1..512 MiB.
func ClampMaxFileMB(mb int) int {
if mb <= 0 {
return DefaultMaxFileMB
}
if mb < 1 {
return 1
}
if mb > 512 {
return 512
}
return mb
}
// ValidAutoSchedule reports whether schedule is a valid 5-field UTC cron expression.
func ValidAutoSchedule(schedule string) bool {
schedule = strings.TrimSpace(schedule)
if schedule == "" {
return false
}
parser := cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow)
_, err := parser.Parse(schedule)
return err == nil
}
// ValidateRuntimeLogsSettingsPatch normalizes and validates runtime log settings in a PATCH body.
// Returns false if any present key is invalid.
func ValidateRuntimeLogsSettingsPatch(body map[string]any) bool {
if raw, ok := body[KeyAutoEnabled]; ok && raw != nil {
v, ok := normalizeBoolSetting(raw)
if !ok {
return false
}
body[KeyAutoEnabled] = v
}
if raw, ok := body[KeyMaxFileMB]; ok && raw != nil {
mb, ok := parsePatchInt(raw)
if !ok {
return false
}
mb = ClampMaxFileMB(mb)
body[KeyMaxFileMB] = mb
}
if raw, ok := body[KeyAutoSchedule]; ok && raw != nil {
s := parseStringSetting(raw)
if !ValidAutoSchedule(s) {
return false
}
body[KeyAutoSchedule] = s
}
if raw, ok := body[KeyAutoMode]; ok && raw != nil {
m := parseStringSetting(raw)
if !store.ValidRuntimeLogCleanupAction(m) {
return false
}
body[KeyAutoMode] = m
}
return true
}
func parseBoolSetting(v any) bool {
switch x := v.(type) {
case bool:
return x
case string:
s := strings.TrimSpace(strings.ToLower(x))
return s == "1" || s == "true" || s == "yes"
case float64:
return x != 0
case int:
return x != 0
case int64:
return x != 0
default:
return false
}
}
func normalizeBoolSetting(v any) (bool, bool) {
switch x := v.(type) {
case bool:
return x, true
case string:
s := strings.TrimSpace(strings.ToLower(x))
switch s {
case "1", "true", "yes":
return true, true
case "0", "false", "no":
return false, true
default:
return false, false
}
case float64:
if x != 0 && x != 1 {
return false, false
}
return x != 0, true
case int:
if x != 0 && x != 1 {
return false, false
}
return x != 0, true
case int64:
if x != 0 && x != 1 {
return false, false
}
return x != 0, true
default:
return false, false
}
}
func parseIntSetting(v any) int {
switch x := v.(type) {
case float64:
return int(x)
case int:
return x
case int64:
return int(x)
case string:
n, err := strconv.Atoi(strings.TrimSpace(x))
if err == nil {
return n
}
}
return 0
}
func parsePatchInt(v any) (int, bool) {
switch x := v.(type) {
case float64:
if x != float64(int(x)) {
return 0, false
}
n := int(x)
return n, n >= 1 && n <= 512
case int:
return x, x >= 1 && x <= 512
case int64:
n := int(x)
return n, n >= 1 && n <= 512
case string:
n, err := strconv.Atoi(strings.TrimSpace(x))
if err != nil {
return 0, false
}
return n, n >= 1 && n <= 512
default:
return 0, false
}
}
func parseStringSetting(v any) string {
switch x := v.(type) {
case string:
return strings.TrimSpace(x)
default:
if v == nil {
return ""
}
return strings.TrimSpace(fmt.Sprint(v))
}
}
+61
View File
@@ -0,0 +1,61 @@
package runtimelogs
import (
"testing"
"evobgp/internal/store"
)
func TestPolicyFromSettingsDefaults(t *testing.T) {
p := PolicyFromSettings(nil)
if p.Enabled {
t.Fatal("expected disabled by default")
}
if p.MaxFileBytes != int64(DefaultMaxFileMB)*1024*1024 {
t.Fatalf("max bytes=%d", p.MaxFileBytes)
}
if p.Schedule != DefaultSchedule {
t.Fatalf("schedule=%q", p.Schedule)
}
if p.Mode != store.RuntimeLogCleanupTruncate {
t.Fatalf("mode=%q", p.Mode)
}
}
func TestClampMaxFileMB(t *testing.T) {
if ClampMaxFileMB(0) != DefaultMaxFileMB {
t.Fatal("zero should default")
}
if ClampMaxFileMB(999) != 512 {
t.Fatal("cap 512")
}
}
func TestValidateRuntimeLogsSettingsPatch(t *testing.T) {
body := map[string]any{
KeyAutoEnabled: true,
KeyMaxFileMB: 64,
KeyAutoSchedule: "0 3 * * *",
KeyAutoMode: "truncate",
}
if !ValidateRuntimeLogsSettingsPatch(body) {
t.Fatal("expected valid patch")
}
if body[KeyMaxFileMB] != 64 {
t.Fatalf("max mb=%v", body[KeyMaxFileMB])
}
bad := map[string]any{KeyAutoSchedule: "not a cron"}
if ValidateRuntimeLogsSettingsPatch(bad) {
t.Fatal("expected invalid cron")
}
}
func TestValidAutoSchedule(t *testing.T) {
if !ValidAutoSchedule("0 */6 * * *") {
t.Fatal("expected valid schedule")
}
if ValidAutoSchedule("invalid") {
t.Fatal("expected invalid schedule")
}
}
+126
View File
@@ -0,0 +1,126 @@
package runtimelogs
import (
"context"
"fmt"
"log"
"strings"
"sync"
"time"
"evobgp/internal/store"
"github.com/robfig/cron/v3"
)
// SchedulerDeps configures the runtime log auto-cleanup background scheduler.
type SchedulerDeps struct {
Service *Service
Store store.Backend
PolicyTenant string
}
// ResolvePolicyTenant picks the tenant whose settings drive auto-cleanup.
func ResolvePolicyTenant(st store.Backend, explicit string) (string, error) {
explicit = strings.TrimSpace(explicit)
if explicit != "" {
return explicit, nil
}
if st == nil {
return "", fmt.Errorf("runtimelogs: store required")
}
ids, err := st.ListTenantIDs()
if err != nil {
return "", err
}
for _, id := range ids {
settings, err := st.ListGlobalSettings(id)
if err != nil {
continue
}
if PolicyFromSettings(settings).Enabled {
return id, nil
}
}
if len(ids) > 0 {
return ids[0], nil
}
return "", fmt.Errorf("runtimelogs: no tenants configured")
}
// StartAutoCleanupScheduler runs periodic auto-cleanup on evobgp-all when FS is available.
func StartAutoCleanupScheduler(ctx context.Context, deps SchedulerDeps, tick time.Duration) {
if deps.Service == nil || !deps.Service.Available() {
log.Printf("runtimelogs: auto-cleanup scheduler disabled (FS unavailable)")
return
}
if deps.Store == nil {
log.Printf("runtimelogs: auto-cleanup scheduler disabled (no store)")
return
}
if tick <= 0 {
tick = 30 * time.Second
}
go func() {
parser := cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow)
var mu sync.Mutex
lastFired := map[string]time.Time{}
t := time.NewTicker(tick)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
tenantID, err := ResolvePolicyTenant(deps.Store, deps.PolicyTenant)
if err != nil {
continue
}
settings, err := deps.Store.ListGlobalSettings(tenantID)
if err != nil {
continue
}
policy := PolicyFromSettings(settings)
if !policy.Enabled {
continue
}
sched, err := parser.Parse(policy.Schedule)
if err != nil {
log.Printf("runtimelogs: invalid cron %q: %v", policy.Schedule, err)
continue
}
now := time.Now().UTC()
mu.Lock()
prev := lastFired[policy.Schedule]
if prev.IsZero() {
prev = now.Add(-time.Minute)
}
next := sched.Next(prev)
if next.After(now) {
mu.Unlock()
continue
}
slot := next.Unix() / 60
if lf, ok := lastFired[policy.Schedule]; ok && lf.Unix()/60 == slot {
mu.Unlock()
continue
}
lastFired[policy.Schedule] = next
mu.Unlock()
_, err = RunAutoCleanup(ctx, AutoCleanupDeps{
Service: deps.Service,
Store: deps.Store,
TenantID: tenantID,
}, policy, false, "scheduler")
if err != nil {
log.Printf("runtimelogs: auto-cleanup run failed: %v", err)
}
}
}
}()
log.Printf("runtimelogs: auto-cleanup scheduler started (tick=%s)", tick)
}
+2
View File
@@ -84,6 +84,8 @@ type Backend interface {
// CreateRenderRevision inserts a new config_revision (revID must be unique) with materialized prefixes and preview fragments.
CreateRenderRevision(revisionID, tenantID, moduleID string, parentRevisionID *string, contentHash string, previewFragments map[string]string, prefixes []PrefixRow) error
RevisionDiff(tenantID, aID, bID string) (map[string]any, error)
EstimateRevisionPrune(tenantID string, cutoff time.Time, retentionMinutes int) (RevisionPruneEstimate, error)
PruneRevisionsWithStats(tenantID string, cutoff time.Time) (RevisionPruneResult, error)
PruneRevisionsBefore(tenantID string, cutoff time.Time) (deleted int, err error)
SetLastAppliedRevision(tenantID, speakerID, revisionID string) error
PublishRevisionForSpeaker(speakerID, revisionID string) error
-56
View File
@@ -616,62 +616,6 @@ func (m *Memory) RevisionDiff(tenantID, aID, bID string) (map[string]any, error)
}, nil
}
func (m *Memory) PruneRevisionsBefore(tenantID string, cutoff time.Time) (int, error) {
m.mu.Lock()
defer m.mu.Unlock()
var newest *Revision
for _, rev := range m.revisions {
if rev.TenantID != tenantID {
continue
}
if newest == nil || rev.CreatedAt.After(newest.CreatedAt) {
newest = rev
}
}
if newest == nil {
return 0, nil
}
protected := map[string]struct{}{
newest.ID: {},
}
for _, sp := range m.speakers {
if sp == nil || sp.TenantID != tenantID {
continue
}
if sp.LastAppliedRevisionID != nil && strings.TrimSpace(*sp.LastAppliedRevisionID) != "" {
protected[strings.TrimSpace(*sp.LastAppliedRevisionID)] = struct{}{}
}
}
for speakerID, info := range m.publishedRevision {
sp, ok := m.speakers[speakerID]
if !ok || sp == nil || sp.TenantID != tenantID {
continue
}
if strings.TrimSpace(info.RevisionID) != "" {
protected[strings.TrimSpace(info.RevisionID)] = struct{}{}
}
}
deleted := 0
for id, rev := range m.revisions {
if rev == nil || rev.TenantID != tenantID {
continue
}
if !rev.CreatedAt.Before(cutoff) {
continue
}
if _, keep := protected[id]; keep {
continue
}
delete(m.revisions, id)
delete(m.revPrefixes, id)
deleted++
}
return deleted, nil
}
func (m *Memory) getRevisionLocked(tenantID, revisionID string) (*Revision, error) {
rev, ok := m.revisions[revisionID]
if !ok {
+118
View File
@@ -0,0 +1,118 @@
package store
import (
"strings"
"time"
)
func (m *Memory) prunableRevisionIDsLocked(tenantID string, cutoff time.Time) []string {
var newest *Revision
for _, rev := range m.revisions {
if rev == nil || rev.TenantID != tenantID {
continue
}
if newest == nil || rev.CreatedAt.After(newest.CreatedAt) {
newest = rev
}
}
if newest == nil {
return nil
}
protected := map[string]struct{}{newest.ID: {}}
for _, sp := range m.speakers {
if sp == nil || sp.TenantID != tenantID {
continue
}
if sp.LastAppliedRevisionID != nil && strings.TrimSpace(*sp.LastAppliedRevisionID) != "" {
protected[strings.TrimSpace(*sp.LastAppliedRevisionID)] = struct{}{}
}
}
for speakerID, info := range m.publishedRevision {
sp, ok := m.speakers[speakerID]
if !ok || sp == nil || sp.TenantID != tenantID {
continue
}
if strings.TrimSpace(info.RevisionID) != "" {
protected[strings.TrimSpace(info.RevisionID)] = struct{}{}
}
}
var ids []string
for id, rev := range m.revisions {
if rev == nil || rev.TenantID != tenantID {
continue
}
if !rev.CreatedAt.Before(cutoff) {
continue
}
if _, keep := protected[id]; keep {
continue
}
ids = append(ids, id)
}
return ids
}
func revisionBytesEstimate(rev *Revision, prefixes []PrefixRow) int64 {
if rev == nil {
return 0
}
var n int64
n += int64(len(rev.ContentHash))
for _, v := range rev.PreviewFragments {
n += int64(len(v))
}
for _, pr := range prefixes {
n += int64(len(pr.Prefix) + len(pr.Source))
}
return n
}
func (m *Memory) EstimateRevisionPrune(tenantID string, cutoff time.Time, retentionMinutes int) (RevisionPruneEstimate, error) {
m.mu.Lock()
defer m.mu.Unlock()
ids := m.prunableRevisionIDsLocked(tenantID, cutoff)
var bytes int64
for _, id := range ids {
bytes += revisionBytesEstimate(m.revisions[id], m.revPrefixes[id])
}
prefixRows := 0
for _, id := range ids {
prefixRows += len(m.revPrefixes[id])
}
return RevisionPruneEstimate{
RetentionMinutes: retentionMinutes,
CutoffAt: cutoff.UTC(),
RevisionCount: len(ids),
PrefixRowCount: prefixRows,
OrphanSnapshotCount: 0,
BytesEstimate: bytes,
}, nil
}
func (m *Memory) PruneRevisionsWithStats(tenantID string, cutoff time.Time) (RevisionPruneResult, error) {
est, err := m.EstimateRevisionPrune(tenantID, cutoff, 0)
if err != nil {
return RevisionPruneResult{}, err
}
m.mu.Lock()
defer m.mu.Unlock()
ids := m.prunableRevisionIDsLocked(tenantID, cutoff)
for _, id := range ids {
delete(m.revisions, id)
delete(m.revPrefixes, id)
}
return RevisionPruneResult{
DeletedRevisions: len(ids),
DeletedPrefixSnapshots: 0,
DeletedPrefixRows: est.PrefixRowCount,
BytesEstimate: est.BytesEstimate,
}, nil
}
func (m *Memory) PruneRevisionsBefore(tenantID string, cutoff time.Time) (int, error) {
res, err := m.PruneRevisionsWithStats(tenantID, cutoff)
if err != nil {
return 0, err
}
return res.DeletedRevisions, nil
}
+21
View File
@@ -0,0 +1,21 @@
package store
import "time"
// RevisionPruneEstimate describes revisions and storage that would be removed by prune.
type RevisionPruneEstimate struct {
RetentionMinutes int `json:"retention_minutes"`
CutoffAt time.Time `json:"cutoff_at"`
RevisionCount int `json:"revision_count"`
PrefixRowCount int `json:"prefix_row_count"`
OrphanSnapshotCount int `json:"orphan_snapshot_count"`
BytesEstimate int64 `json:"bytes_estimate"`
}
// RevisionPruneResult is the outcome of a revision prune run.
type RevisionPruneResult struct {
DeletedRevisions int `json:"deleted_revisions"`
DeletedPrefixSnapshots int `json:"deleted_prefix_snapshots"`
DeletedPrefixRows int `json:"deleted_prefix_rows"`
BytesEstimate int64 `json:"bytes_estimate"`
}
@@ -66,6 +66,7 @@
let files = $state<RuntimeLogFile[]>([]);
let auditLoading = $state(false);
let auditError = $state<string | null>(null);
let auditItems = $state<RuntimeLogCleanupAudit[]>([]);
let auditCursor = $state<string | undefined>(undefined);
let auditHasMore = $state(false);
@@ -107,6 +108,7 @@
sortable: true,
sortValue: (r) => r.created_at
},
{ id: 'source', label: 'Источник' },
{ id: 'actor', label: 'Actor' },
{ id: 'filename', label: 'Файл', sortable: true, sortValue: (r) => r.filename },
{ id: 'action', label: 'Действие' },
@@ -140,6 +142,7 @@
async function loadAudit(reset = true) {
auditLoading = true;
if (reset) auditError = null;
try {
const page = await listRuntimeLogCleanupAudit({
limit: 20,
@@ -148,13 +151,22 @@
auditItems = reset ? (page.items ?? []) : [...auditItems, ...(page.items ?? [])];
auditCursor = page.next_cursor;
auditHasMore = Boolean(page.has_more && page.next_cursor);
auditError = null;
} catch (e) {
auditError = e instanceof Error ? e.message : 'Не удалось загрузить audit';
if (reset) auditItems = [];
notifyApiError(e, 'Audit очистки логов');
} finally {
auditLoading = false;
}
}
function auditSourceLabel(actor: string): string {
if (actor.startsWith('auto:')) return 'Авто';
if (actor.startsWith('op:')) return 'Оператор';
return '—';
}
async function refreshAll() {
await Promise.all([loadFiles(), loadAudit(true)]);
}
@@ -210,12 +222,6 @@
await loadAudit(true);
})();
});
$effect(() => {
if (subTab === 'audit' && auditItems.length === 0 && !auditLoading) {
void loadAudit(true);
}
});
</script>
<div class="flex flex-col gap-4">
@@ -359,33 +365,47 @@
</CardDescription>
</CardHeader>
<CardContent class="min-w-0 space-y-3 p-4 pt-0">
<AppDataTable
columns={auditColumns}
rows={auditItems}
rowKey={(r) => r.id}
loading={auditLoading && auditItems.length === 0}
emptyTitle="Записей пока нет"
emptyDescription="Очистка log-файлов появится здесь после operator DELETE."
>
{#snippet cell({ row, column })}
{#if column.id === 'created'}
<span class="text-sm">{formatDateTime(row.created_at)}</span>
{:else if column.id === 'actor'}
<span class="font-mono text-xs">{row.actor_prefix}</span>
{:else if column.id === 'filename'}
<span class="font-mono text-xs">{row.filename}</span>
{:else if column.id === 'action'}
<Badge variant={actionBadgeVariant(row.action)}>{row.action}</Badge>
{:else if column.id === 'sizes'}
<span class="text-xs tabular-nums">
{formatBytes(row.size_before)}
{#if row.size_after != null}
{formatBytes(row.size_after)}
{/if}
</span>
{/if}
{/snippet}
</AppDataTable>
{#if auditError && auditItems.length === 0}
<EmptyState icon={FileText} title="Не удалось загрузить audit" description={auditError}>
{#snippet action()}
<Button variant="outline" size="sm" onclick={() => loadAudit(true)}>
Повторить
</Button>
{/snippet}
</EmptyState>
{:else}
<AppDataTable
columns={auditColumns}
rows={auditItems}
rowKey={(r) => r.id}
loading={auditLoading && auditItems.length === 0}
emptyTitle="Записей пока нет"
emptyDescription="Очистка появится после ручного DELETE или автоочистки. Настройки — Параметры → Файловые логи."
>
{#snippet cell({ row, column })}
{#if column.id === 'created'}
<span class="text-sm">{formatDateTime(row.created_at)}</span>
{:else if column.id === 'source'}
<Badge variant={row.actor_prefix.startsWith('auto:') ? 'secondary' : 'outline'}>
{auditSourceLabel(row.actor_prefix)}
</Badge>
{:else if column.id === 'actor'}
<span class="font-mono text-xs">{row.actor_prefix}</span>
{:else if column.id === 'filename'}
<span class="font-mono text-xs">{row.filename}</span>
{:else if column.id === 'action'}
<Badge variant={actionBadgeVariant(row.action)}>{row.action}</Badge>
{:else if column.id === 'sizes'}
<span class="text-xs tabular-nums">
{formatBytes(row.size_before)}
{#if row.size_after != null}
{formatBytes(row.size_after)}
{/if}
</span>
{/if}
{/snippet}
</AppDataTable>
{/if}
{#if auditHasMore}
<Button
variant="outline"
@@ -2,10 +2,17 @@
import { onMount } from 'svelte';
import { defaults, superForm } from 'sveltekit-superforms';
import { zod4 } from 'sveltekit-superforms/adapters';
import type { AuthSession } from '$lib/api/types.js';
import { apiJSON } from '$lib/api/client.js';
import {
emptyRevisionSettingsForm,
revisionSettingsSchema
} from '$lib/settings/revision-settings.schema.js';
import {
fetchRevisionPruneEstimate,
pruneRevisionsNow,
type RevisionPruneEstimate
} from '$lib/settings/revision-prune-api.js';
import {
buildPayloadFromFormFields,
loadSettings,
@@ -13,6 +20,7 @@
patchSettings
} from '$lib/settings/settings-api.js';
import { REVISION_SETTING_KEYS } from '$lib/settings/settings-known-keys.js';
import { formatBytes } from '$lib/monitoring/postgres.js';
import { Button } from '$lib/ui/core/button/index.js';
import {
Card,
@@ -23,12 +31,18 @@
} from '$lib/ui/core/card/index.js';
import { Input } from '$lib/ui/core/input/index.js';
import FormField from '$lib/ui/patterns/form/form-field.svelte';
import { confirm } from '$lib/ui/patterns/confirm/confirm-state.svelte.js';
import { notify, notifyApiError } from '$lib/ui/app/toast.js';
import Save from '@lucide/svelte/icons/save';
import Trash2 from '@lucide/svelte/icons/trash-2';
let loading = $state(false);
let saving = $state(false);
let pruning = $state(false);
let loaded = $state(false);
let session = $state<AuthSession | null>(null);
let estimateLoading = $state(false);
let estimate = $state<RevisionPruneEstimate | null>(null);
const { form, errors, reset, validateForm } = superForm(
defaults(emptyRevisionSettingsForm(), zod4(revisionSettingsSchema)),
@@ -39,13 +53,60 @@
}
);
let hasValidationErrors = $derived(Boolean($errors.revision_retention_minutes?.length));
const isOperator = $derived(session?.role === 'operator');
const hasValidationErrors = $derived(Boolean($errors.revision_retention_minutes?.length));
let canSave = $derived.by(() => {
const parsedRetentionMinutes = $derived.by(() => {
const s = String($form.revision_retention_minutes ?? '').trim();
if (s === '' || !/^\d+$/.test(s)) return null;
const n = Number(s);
if (!Number.isInteger(n) || n < 15 || n > 43200) return null;
return n;
});
const canSave = $derived.by(() => {
if (loading || saving || hasValidationErrors || !loaded) return false;
return String($form.revision_retention_minutes ?? '').trim() !== '';
});
const canPruneNow = $derived.by(() => {
if (!isOperator || !loaded || pruning || saving || parsedRetentionMinutes === null)
return false;
return (estimate?.revision_count ?? 0) > 0;
});
async function loadSession() {
try {
session = await apiJSON<AuthSession>('/v1/auth/session');
} catch {
session = null;
}
}
async function refreshEstimate(minutes: number) {
estimateLoading = true;
try {
estimate = await fetchRevisionPruneEstimate(minutes);
} catch (e) {
estimate = null;
notifyApiError(e, 'Не удалось рассчитать оценку очистки');
} finally {
estimateLoading = false;
}
}
$effect(() => {
const minutes = parsedRetentionMinutes;
if (!loaded || minutes === null) {
estimate = null;
return;
}
const handle = setTimeout(() => {
void refreshEstimate(minutes);
}, 400);
return () => clearTimeout(handle);
});
async function load() {
loading = true;
try {
@@ -89,7 +150,46 @@
}
}
async function requestPruneNow() {
const minutes = parsedRetentionMinutes;
if (minutes === null) return;
let est: RevisionPruneEstimate;
try {
est = await fetchRevisionPruneEstimate(minutes);
} catch (e) {
notifyApiError(e);
return;
}
estimate = est;
if (est.revision_count === 0) {
notify.info('Нет ревизий для удаления по выбранному retention');
return;
}
void confirm({
title: 'Очистить старые ревизии?',
description: `Будет удалено ${est.revision_count} ревизий. Ориентировочно освободится ~${formatBytes(est.bytes_estimate)}. Действие необратимо.`,
confirmLabel: 'Очистить',
destructive: true,
onConfirm: async () => {
pruning = true;
try {
const res = await pruneRevisionsNow(minutes);
notify.success(
`Удалено ревизий: ${res.deleted_revisions}, освобождено ~${formatBytes(res.bytes_estimate)}`
);
await refreshEstimate(minutes);
} catch (e) {
notifyApiError(e);
throw e;
} finally {
pruning = false;
}
}
});
}
onMount(() => {
void loadSession();
void load();
});
</script>
@@ -98,7 +198,8 @@
<CardHeader>
<CardTitle>Хранение ревизий</CardTitle>
<CardDescription>
Автоматическая очистка старых ревизий. Последняя раскатанная ревизия не удаляется.
Автоматическая очистка старых ревизий. Последняя ревизия и раскатанные на спикерах не
удаляются.
</CardDescription>
</CardHeader>
<CardContent class="space-y-4">
@@ -123,16 +224,54 @@
/>
</FormField>
{#if parsedRetentionMinutes !== null}
<div class="space-y-1 rounded-md border bg-muted/30 p-4 text-sm">
<p class="font-medium">Оценка очистки по введённому retention</p>
{#if estimateLoading}
<p class="text-muted-foreground">Расчёт…</p>
{:else if estimate}
<p>
Будет удалено ревизий: <strong>{estimate.revision_count}</strong>
</p>
<p>
Освободится ориентировочно: <strong>~{formatBytes(estimate.bytes_estimate)}</strong>
</p>
{#if estimate.prefix_row_count > 0}
<p class="text-xs text-muted-foreground">
Строк префиксов в снимках: {estimate.prefix_row_count}
{#if estimate.orphan_snapshot_count > 0}
· снимков: {estimate.orphan_snapshot_count}
{/if}
</p>
{/if}
{:else}
<p class="text-muted-foreground">Оценка недоступна</p>
{/if}
<p class="pt-1 text-xs text-muted-foreground">
Учитываются те же правила, что при автоочистке: последняя ревизия и раскатанные на
спикерах не удаляются.
</p>
</div>
{/if}
{#if hasValidationErrors}
<p class="text-sm text-destructive">
Есть ошибки в полях. Исправьте их, чтобы сохранить изменения.
</p>
{/if}
<Button onclick={save} disabled={!canSave}>
<Save />
{saving ? 'Сохранение…' : 'Применить параметры ревизий'}
</Button>
<div class="flex flex-wrap gap-2">
<Button onclick={save} disabled={!canSave}>
<Save />
{saving ? 'Сохранение…' : 'Применить параметры ревизий'}
</Button>
{#if isOperator}
<Button variant="destructive" disabled={!canPruneNow} onclick={requestPruneNow}>
<Trash2 />
{pruning ? 'Очистка…' : 'Очистить сейчас'}
</Button>
{/if}
</div>
{/if}
</CardContent>
</Card>
@@ -0,0 +1,340 @@
<script lang="ts">
import { onMount } from 'svelte';
import { defaults, superForm } from 'sveltekit-superforms';
import { zod4 } from 'sveltekit-superforms/adapters';
import type { AuthSession } from '$lib/api/types.js';
import { apiJSON } from '$lib/api/client.js';
import {
emptyRuntimeLogsSettingsForm,
runtimeLogsSettingsSchema
} from '$lib/settings/runtime-logs-settings.schema.js';
import {
buildPayloadFromFormFields,
loadSettings,
partitionSettings,
patchSettings
} from '$lib/settings/settings-api.js';
import { RUNTIME_LOGS_SETTING_KEYS } from '$lib/settings/settings-known-keys.js';
import {
fetchRuntimeLogAutoEstimate,
runRuntimeLogAutoCleanup,
type RuntimeLogAutoEstimate
} from '$lib/runtime-logs/runtime-logs-auto-api.js';
import { isRuntimeLogsUnavailable } from '$lib/runtime-logs/runtime-logs-api.js';
import { formatBytes } from '$lib/monitoring/postgres.js';
import { Button } from '$lib/ui/core/button/index.js';
import {
Card,
CardContent,
CardDescription,
CardHeader,
CardTitle
} from '$lib/ui/core/card/index.js';
import { Input } from '$lib/ui/core/input/index.js';
import { Switch } from '$lib/ui/core/switch/index.js';
import { Select, SelectContent, SelectItem, SelectTrigger } from '$lib/ui/core/select/index.js';
import FormField from '$lib/ui/patterns/form/form-field.svelte';
import { confirm } from '$lib/ui/patterns/confirm/confirm-state.svelte.js';
import { notify, notifyApiError } from '$lib/ui/app/toast.js';
import Save from '@lucide/svelte/icons/save';
import Play from '@lucide/svelte/icons/play';
let loading = $state(false);
let saving = $state(false);
let running = $state(false);
let loaded = $state(false);
let session = $state<AuthSession | null>(null);
let estimateLoading = $state(false);
let estimate = $state<RuntimeLogAutoEstimate | null>(null);
let fsUnavailable = $state(false);
const { form, errors, reset, validateForm } = superForm(
defaults(emptyRuntimeLogsSettingsForm(), zod4(runtimeLogsSettingsSchema)),
{
validators: zod4(runtimeLogsSettingsSchema),
SPA: true,
dataType: 'json'
}
);
const isOperator = $derived(session?.role === 'operator');
const autoEnabled = $derived($form.runtime_logs_auto_enabled === 'true');
const hasValidationErrors = $derived(
Boolean(
$errors.runtime_logs_max_file_mb?.length ||
$errors.runtime_logs_auto_schedule?.length ||
$errors.runtime_logs_auto_mode?.length
)
);
const parsedMaxMb = $derived.by(() => {
const s = String($form.runtime_logs_max_file_mb ?? '').trim();
if (s === '' || !/^\d+$/.test(s)) return null;
const n = Number(s);
if (!Number.isInteger(n) || n < 1 || n > 512) return null;
return n;
});
const canSave = $derived.by(() => {
if (loading || saving || hasValidationErrors || !loaded || !isOperator) return false;
return true;
});
const canRunNow = $derived.by(() => {
if (!isOperator || !loaded || running || saving || parsedMaxMb === null || fsUnavailable)
return false;
return (estimate?.would_count ?? 0) > 0;
});
async function loadSession() {
try {
session = await apiJSON<AuthSession>('/v1/auth/session');
} catch {
session = null;
}
}
async function refreshEstimate() {
if (fsUnavailable || parsedMaxMb === null) {
estimate = null;
return;
}
estimateLoading = true;
try {
estimate = await fetchRuntimeLogAutoEstimate();
fsUnavailable = false;
} catch (e) {
estimate = null;
if (isRuntimeLogsUnavailable(e)) {
fsUnavailable = true;
return;
}
notifyApiError(e, 'Оценка автоочистки');
} finally {
estimateLoading = false;
}
}
$effect(() => {
if (!loaded || parsedMaxMb === null) {
estimate = null;
return;
}
const handle = setTimeout(() => {
void refreshEstimate();
}, 400);
return () => clearTimeout(handle);
});
async function load() {
loading = true;
try {
const settings = await loadSettings();
const { partitioned } = partitionSettings(settings);
reset({ data: partitioned.runtimeLogs });
loaded = true;
} catch (e) {
notifyApiError(e);
} finally {
loading = false;
}
}
async function save() {
const validation = await validateForm({ update: true });
if (!validation.valid) {
notify.error('Исправьте ошибки в полях настроек');
return;
}
const payload = buildPayloadFromFormFields(
RUNTIME_LOGS_SETTING_KEYS,
$form as Record<string, string>,
$errors as Partial<Record<string, string[]>>
);
if ($form.runtime_logs_auto_mode === 'truncate' || $form.runtime_logs_auto_mode === 'delete') {
payload.runtime_logs_auto_mode = $form.runtime_logs_auto_mode;
}
saving = true;
try {
await patchSettings(payload);
notify.success('Параметры автоочистки логов сохранены');
await load();
} catch (e) {
notifyApiError(e);
} finally {
saving = false;
}
}
async function requestRunNow() {
if (parsedMaxMb === null) return;
let est: RuntimeLogAutoEstimate;
try {
est = await fetchRuntimeLogAutoEstimate();
} catch (e) {
notifyApiError(e);
return;
}
estimate = est;
const count = est.would_count ?? 0;
if (count === 0) {
notify.info('Нет файлов выше порога для очистки');
return;
}
const mode = $form.runtime_logs_auto_mode === 'delete' ? 'удаление' : 'truncate';
void confirm({
title: 'Запустить автоочистку сейчас?',
description: `Будет затронуто файлов: ${count}. Режим: ${mode}. Записи появятся в Monitoring → Файловые логи → Audit.`,
confirmLabel: 'Запустить',
destructive: true,
onConfirm: async () => {
running = true;
try {
const res = await runRuntimeLogAutoCleanup(false);
notify.success(`Очищено файлов: ${res.cleaned_count ?? 0}`);
await refreshEstimate();
} catch (e) {
notifyApiError(e);
throw e;
} finally {
running = false;
}
}
});
}
onMount(() => {
void loadSession();
void load();
});
</script>
<Card>
<CardHeader>
<CardTitle>Файловые логи (runtime-logs)</CardTitle>
<CardDescription>
Автоочистка <code class="text-xs">*.log</code> на диске evobgp-all (sidecar
stack-runtime-logs). Требуется volume <code class="text-xs">EVOBGP_RUNTIME_LOGS_DIR</code>.
</CardDescription>
</CardHeader>
<CardContent class="space-y-4">
{#if loading && !loaded}
<p class="text-sm text-muted-foreground">Загрузка…</p>
{:else if !loaded}
<Button variant="outline" onclick={load}>Загрузить настройки</Button>
{:else}
<FormField
id="runtime-logs-auto-enabled"
label="Автоочистка по расписанию"
description="Scheduler в evobgp-all (UTC cron ниже)."
>
<Switch
id="runtime-logs-auto-enabled"
checked={autoEnabled}
disabled={!isOperator}
onCheckedChange={(v) => {
$form.runtime_logs_auto_enabled = v ? 'true' : 'false';
}}
/>
</FormField>
<FormField
id="runtime-logs-max-mb"
label="Порог размера файла, MiB"
error={$errors.runtime_logs_max_file_mb?.[0]}
description="Очищать файлы строго больше порога (1–512 MiB)."
>
<Input
id="runtime-logs-max-mb"
type="number"
min="1"
max="512"
bind:value={$form.runtime_logs_max_file_mb}
disabled={!isOperator}
/>
</FormField>
<FormField
id="runtime-logs-schedule"
label="Расписание (UTC cron)"
error={$errors.runtime_logs_auto_schedule?.[0]}
description="5 полей: минута час день месяц день_недели. По умолчанию каждые 6 часов."
>
<Input
id="runtime-logs-schedule"
bind:value={$form.runtime_logs_auto_schedule}
placeholder="0 */6 * * *"
disabled={!isOperator}
/>
</FormField>
<FormField id="runtime-logs-mode" label="Режим очистки">
<Select
type="single"
value={$form.runtime_logs_auto_mode || 'truncate'}
disabled={!isOperator}
onValueChange={(v) => {
if (v === 'truncate' || v === 'delete') $form.runtime_logs_auto_mode = v;
}}
>
<SelectTrigger id="runtime-logs-mode" class="w-full max-w-xs">
{$form.runtime_logs_auto_mode === 'delete'
? 'delete — удалить файл'
: 'truncate — обнулить'}
</SelectTrigger>
<SelectContent>
<SelectItem value="truncate">truncate — обнулить</SelectItem>
<SelectItem value="delete">delete — удалить файл</SelectItem>
</SelectContent>
</Select>
</FormField>
{#if fsUnavailable}
<p class="text-sm text-muted-foreground">
FS API недоступен (не evobgp-all или нет volume). Оценка и «Запустить сейчас» недоступны;
настройки сохраняются для будущего прогона scheduler.
</p>
{:else if parsedMaxMb !== null}
<div class="space-y-1 rounded-md border bg-muted/30 p-4 text-sm">
<p class="font-medium">Оценка по текущему порогу</p>
{#if estimateLoading}
<p class="text-muted-foreground">Расчёт…</p>
{:else if estimate}
<p>
Файлов к очистке: <strong>{estimate.would_count ?? 0}</strong>
</p>
{#if estimate.items?.length}
<ul class="mt-2 space-y-1 text-xs text-muted-foreground">
{#each estimate.items.filter((i) => i.would_cleanup) as item (item.filename)}
<li class="font-mono">
{item.filename} · {formatBytes(item.size_bytes)}
</li>
{/each}
</ul>
{/if}
{:else}
<p class="text-muted-foreground">Оценка недоступна</p>
{/if}
</div>
{/if}
{#if hasValidationErrors}
<p class="text-sm text-destructive">Исправьте ошибки в полях перед сохранением.</p>
{/if}
<div class="flex flex-wrap gap-2">
{#if isOperator}
<Button onclick={save} disabled={!canSave}>
<Save />
{saving ? 'Сохранение…' : 'Сохранить'}
</Button>
<Button variant="destructive" disabled={!canRunNow} onclick={requestRunNow}>
<Play />
{running ? 'Запуск…' : 'Запустить сейчас'}
</Button>
{/if}
</div>
{/if}
</CardContent>
</Card>
@@ -8,13 +8,14 @@
import TenantBirdSettingsCard from '$lib/components/tenant-settings/TenantBirdSettingsCard.svelte';
import TenantRevisionSettingsCard from '$lib/components/tenant-settings/TenantRevisionSettingsCard.svelte';
import TenantAdditionalSettingsCard from '$lib/components/tenant-settings/TenantAdditionalSettingsCard.svelte';
import TenantRuntimeLogsSettingsCard from '$lib/components/tenant-settings/TenantRuntimeLogsSettingsCard.svelte';
import SlidersHorizontal from '@lucide/svelte/icons/sliders-horizontal';
import Info from '@lucide/svelte/icons/info';
type TenantSettingsTab = 'bird' | 'revision' | 'additional';
type TenantSettingsTab = 'bird' | 'revision' | 'runtime-logs' | 'additional';
function parseTenantSettingsTab(value: string | null): TenantSettingsTab {
if (value === 'revision' || value === 'additional') return value;
if (value === 'revision' || value === 'runtime-logs' || value === 'additional') return value;
return 'bird';
}
@@ -65,6 +66,7 @@
<TabsList class="inline-flex min-w-max">
<TabsTrigger value="bird">BIRD</TabsTrigger>
<TabsTrigger value="revision">Ревизии</TabsTrigger>
<TabsTrigger value="runtime-logs">Файловые логи</TabsTrigger>
<TabsTrigger value="additional">Дополнительно</TabsTrigger>
</TabsList>
</div>
@@ -77,6 +79,10 @@
<TenantRevisionSettingsCard />
</TabsContent>
<TabsContent value="runtime-logs" class="mt-4">
<TenantRuntimeLogsSettingsCard />
</TabsContent>
<TabsContent value="additional" class="mt-4">
<TenantAdditionalSettingsCard />
</TabsContent>
+18
View File
@@ -33,6 +33,24 @@ export const maintenancePolicyPresets: MaintenancePolicyPreset[] = [
dry_run_enabled: true
}
},
{
id: 'runtime_log_cleanup_audit_retention',
label: 'Runtime log cleanup audit — 90d',
description:
'Удаляет записи runtime_log_cleanup_audit старше 90 дней (ручная и автоочистка FS).',
tableHint: 'runtime_log_cleanup_audit',
form: {
name: 'Runtime log cleanup audit (90d)',
table_name: 'runtime_log_cleanup_audit',
condition: 'true',
retention_period_sec: String(90 * DAY_SEC),
max_rows: '5000',
vacuum_strategy: 'none',
schedule: '0 4 * * *',
enabled: true,
dry_run_enabled: true
}
},
{
id: 'postgres_maintenance_audit_retention',
label: 'Maintenance audit — 30d',
@@ -0,0 +1,37 @@
import { apiJSON, apiMutate } from '$lib/api/client.js';
export type RuntimeLogAutoEstimateItem = {
filename: string;
size_bytes: number;
would_cleanup: boolean;
skip_reason?: string;
};
export type RuntimeLogAutoEstimate = {
policy?: {
enabled?: boolean;
max_file_bytes?: number;
schedule?: string;
mode?: 'truncate' | 'delete';
};
items?: RuntimeLogAutoEstimateItem[];
would_count?: number;
};
export type RuntimeLogAutoRunResult = {
dry_run?: boolean;
trigger?: string;
cleaned_count?: number;
skipped_count?: number;
cleaned?: unknown[];
skipped?: unknown[];
};
export async function fetchRuntimeLogAutoEstimate(): Promise<RuntimeLogAutoEstimate> {
return apiJSON<RuntimeLogAutoEstimate>('/v1/runtime-logs/auto-estimate');
}
export async function runRuntimeLogAutoCleanup(dryRun = false): Promise<RuntimeLogAutoRunResult> {
const q = dryRun ? '?dry_run=true' : '';
return apiMutate<RuntimeLogAutoRunResult>(`/v1/runtime-logs/auto-run${q}`, 'POST');
}
@@ -0,0 +1,30 @@
import { apiJSON, apiMutate } from '$lib/api/client.js';
export type RevisionPruneEstimate = {
retention_minutes: number;
cutoff_at: string;
revision_count: number;
prefix_row_count: number;
orphan_snapshot_count: number;
bytes_estimate: number;
};
export type RevisionPruneResult = {
deleted_revisions: number;
deleted_prefix_snapshots: number;
deleted_prefix_rows: number;
bytes_estimate: number;
};
export async function fetchRevisionPruneEstimate(
retentionMinutes: number
): Promise<RevisionPruneEstimate> {
const q = new URLSearchParams({ retention_minutes: String(retentionMinutes) });
return apiJSON<RevisionPruneEstimate>(`/v1/revisions/prune-estimate?${q}`);
}
export async function pruneRevisionsNow(retentionMinutes: number): Promise<RevisionPruneResult> {
return apiMutate<RevisionPruneResult>('/v1/revisions/prune', 'POST', {
retention_minutes: retentionMinutes
});
}
@@ -0,0 +1,66 @@
import { z } from 'zod';
function boolInput(val: unknown): string {
if (val === undefined || val === null) return 'false';
if (typeof val === 'boolean') return val ? 'true' : 'false';
if (typeof val === 'number') return val !== 0 ? 'true' : 'false';
return String(val).trim().toLowerCase() === 'true' || String(val).trim() === '1'
? 'true'
: 'false';
}
function stringInput(val: unknown): string {
if (val === undefined || val === null) return '';
return String(val);
}
const runtimeLogsAutoEnabled = z.preprocess(boolInput, z.enum(['true', 'false']));
const runtimeLogsMaxFileMb = z.preprocess(
stringInput,
z.string().refine(
(v) => {
const s = v.trim();
if (s === '') return true;
const n = Number(s);
return /^\d+$/.test(s) && Number.isInteger(n) && n >= 1 && n <= 512;
},
{ message: 'Порог должен быть целым числом от 1 до 512 MiB' }
)
);
const runtimeLogsAutoSchedule = z.preprocess(
stringInput,
z.string().refine(
(v) => {
const s = v.trim();
if (s === '') return true;
const parts = s.split(/\s+/);
return parts.length === 5;
},
{ message: 'Cron: 5 полей (минута час день месяц день_недели), UTC' }
)
);
const runtimeLogsAutoMode = z.preprocess(
stringInput,
z.enum(['truncate', 'delete', '']).refine((v) => v === '' || v === 'truncate' || v === 'delete', {
message: 'Режим: truncate или delete'
})
);
export const runtimeLogsSettingsSchema = z.object({
runtime_logs_auto_enabled: runtimeLogsAutoEnabled,
runtime_logs_max_file_mb: runtimeLogsMaxFileMb,
runtime_logs_auto_schedule: runtimeLogsAutoSchedule,
runtime_logs_auto_mode: runtimeLogsAutoMode
});
export type RuntimeLogsSettingsForm = z.infer<typeof runtimeLogsSettingsSchema>;
export const emptyRuntimeLogsSettingsForm = (): RuntimeLogsSettingsForm => ({
runtime_logs_auto_enabled: 'false',
runtime_logs_max_file_mb: '128',
runtime_logs_auto_schedule: '0 */6 * * *',
runtime_logs_auto_mode: 'truncate'
});
+34 -6
View File
@@ -5,13 +5,20 @@ import {
emptyRevisionSettingsForm,
type RevisionSettingsForm
} from './revision-settings.schema.js';
import {
emptyRuntimeLogsSettingsForm,
type RuntimeLogsSettingsForm
} from './runtime-logs-settings.schema.js';
import {
BIRD_SETTING_KEYS,
BOOLEAN_SETTING_KEYS,
KNOWN_SETTING_KEYS,
NUMERIC_SETTING_KEYS,
RUNTIME_LOGS_SETTING_KEYS,
type BirdSettingKey,
type KnownSettingKey,
type RevisionSettingKey
type RevisionSettingKey,
type RuntimeLogsSettingKey
} from './settings-known-keys.js';
export type AdditionalSettingEntry = { id: number; key: string; value: string };
@@ -19,6 +26,7 @@ export type AdditionalSettingEntry = { id: number; key: string; value: string };
export type PartitionedSettings = {
bird: BirdSettingsForm;
revision: RevisionSettingsForm;
runtimeLogs: RuntimeLogsSettingsForm;
additional: AdditionalSettingEntry[];
};
@@ -38,6 +46,7 @@ export function partitionSettings(
): { partitioned: PartitionedSettings; nextId: number } {
const bird = emptyBirdSettingsForm();
const revision = emptyRevisionSettingsForm();
const runtimeLogs = emptyRuntimeLogsSettingsForm();
const additional: AdditionalSettingEntry[] = [];
let idCounter = nextId;
@@ -46,6 +55,18 @@ export function partitionSettings(
bird[key as BirdSettingKey] = parseKnownValue(key as KnownSettingKey, value);
} else if (key === 'revision_retention_minutes') {
revision.revision_retention_minutes = parseKnownValue(key as RevisionSettingKey, value);
} else if ((RUNTIME_LOGS_SETTING_KEYS as readonly string[]).includes(key)) {
const rk = key as RuntimeLogsSettingKey;
if (rk === 'runtime_logs_auto_enabled') {
runtimeLogs.runtime_logs_auto_enabled =
value === true || value === 1 || value === 'true' || value === '1' ? 'true' : 'false';
} else if (rk === 'runtime_logs_auto_mode') {
const m = String(value ?? '').trim();
runtimeLogs.runtime_logs_auto_mode =
m === 'delete' ? 'delete' : m === 'truncate' ? 'truncate' : '';
} else {
runtimeLogs[rk] = parseKnownValue(key as KnownSettingKey, value);
}
} else {
additional.push({
id: idCounter++,
@@ -56,7 +77,7 @@ export function partitionSettings(
}
return {
partitioned: { bird, revision, additional },
partitioned: { bird, revision, runtimeLogs, additional },
nextId: idCounter
};
}
@@ -65,7 +86,9 @@ export async function loadSettings(): Promise<AppSettings> {
return apiJSON<AppSettings>('/v1/settings');
}
export async function patchSettings(payload: Record<string, string | number>): Promise<void> {
export async function patchSettings(
payload: Record<string, string | number | boolean>
): Promise<void> {
await apiMutate('/v1/settings', 'PATCH', payload);
}
@@ -73,11 +96,16 @@ export function buildPayloadFromFormFields(
keys: readonly KnownSettingKey[],
form: Record<string, string>,
errors: Partial<Record<string, string[]>>
): Record<string, string | number> {
const payload: Record<string, string | number> = {};
): Record<string, string | number | boolean> {
const payload: Record<string, string | number | boolean> = {};
for (const key of keys) {
const value = String(form[key] ?? '').trim();
if (!value || errors[key]?.length) continue;
if (errors[key]?.length) continue;
if (BOOLEAN_SETTING_KEYS.has(key)) {
payload[key] = value === 'true';
continue;
}
if (!value) continue;
if (NUMERIC_SETTING_KEYS.has(key)) payload[key] = Number(value);
else payload[key] = value;
}
+17 -2
View File
@@ -9,13 +9,28 @@ export const BIRD_SETTING_KEYS = [
export const REVISION_SETTING_KEYS = ['revision_retention_minutes'] as const;
export const KNOWN_SETTING_KEYS = [...BIRD_SETTING_KEYS, ...REVISION_SETTING_KEYS] as const;
export const RUNTIME_LOGS_SETTING_KEYS = [
'runtime_logs_auto_enabled',
'runtime_logs_max_file_mb',
'runtime_logs_auto_schedule',
'runtime_logs_auto_mode'
] as const;
export const KNOWN_SETTING_KEYS = [
...BIRD_SETTING_KEYS,
...REVISION_SETTING_KEYS,
...RUNTIME_LOGS_SETTING_KEYS
] as const;
export type BirdSettingKey = (typeof BIRD_SETTING_KEYS)[number];
export type RevisionSettingKey = (typeof REVISION_SETTING_KEYS)[number];
export type RuntimeLogsSettingKey = (typeof RUNTIME_LOGS_SETTING_KEYS)[number];
export type KnownSettingKey = (typeof KNOWN_SETTING_KEYS)[number];
export const NUMERIC_SETTING_KEYS = new Set<KnownSettingKey>([
'bird_local_asn',
'revision_retention_minutes'
'revision_retention_minutes',
'runtime_logs_max_file_mb'
]);
export const BOOLEAN_SETTING_KEYS = new Set<KnownSettingKey>(['runtime_logs_auto_enabled']);