From 8c8ccce6ee715de13f3eb996890b46e2cbf51e3c Mon Sep 17 00:00:00 2001 From: Denozordec Date: Mon, 30 Mar 2026 19:17:29 +0700 Subject: [PATCH] Enhance API and UI for incident management and live updates - Added a new endpoint `/api/agg/incidents` to provide a normalized snapshot of incidents for fleet triage, including severity and recommended actions. - Implemented live event streaming via `/api/live/events` for real-time updates on fleet status and incidents, enhancing observability. - Updated the Web UI to include dedicated sections for incidents and live updates, improving user navigation and access to critical information. - Enhanced API documentation to reflect new endpoints and their functionalities, ensuring clarity for developers and users. --- README.md | 12 +- docs/AGGREGATE.md | 17 ++ docs/AGGREGATE_OPENAPI.yaml | 59 ++++ docs/INCIDENT_ROLLOUT.md | 73 +++++ docs/OPERATIONS_BASELINE.md | 49 +++ internal/aggregate/handlers.go | 19 ++ internal/aggregate/incidents.go | 154 ++++++++++ internal/aggregate/types.go | 46 +++ internal/server/gateway.go | 83 ++++- internal/server/metrics.go | 12 +- web/README.md | 19 ++ web/src/lib/api/client.ts | 48 +++ web/src/lib/components/app-sidebar.svelte | 22 ++ web/src/routes/+page.svelte | 124 +++++++- web/src/routes/incidents/+page.svelte | 351 ++++++++++++++++++++++ web/src/routes/ips/+page.svelte | 122 +++++++- web/src/routes/live/+page.svelte | 348 +++++++++++++++++++++ web/src/routes/users/+page.svelte | 125 +++++++- 18 files changed, 1655 insertions(+), 28 deletions(-) create mode 100644 docs/INCIDENT_ROLLOUT.md create mode 100644 docs/OPERATIONS_BASELINE.md create mode 100644 internal/aggregate/incidents.go create mode 100644 web/src/routes/incidents/+page.svelte create mode 100644 web/src/routes/live/+page.svelte diff --git a/README.md b/README.md index 901355d..afcba9c 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # telemt-api -HTTP‑шлюз на Go для [Telemt Control API](docs/API.md): один порт, **белый список IP (CIDR)**, маршруты вида `/api/{alias}/…` → `{base_url}/v1/…`, агрегация нескольких инстансов — [`/api/agg/…`](docs/AGGREGATE.md), метрики Prometheus на `/metrics`. **Web UI** (SvelteKit) встроен в тот же процесс/образ: статика на `/`, API на `/api/…` и `/health`. +HTTP‑шлюз на Go для [Telemt Control API](docs/API.md): один порт, **белый список IP (CIDR)**, маршруты вида `/api/{alias}/…` → `{base_url}/v1/…`, агрегация нескольких инстансов — [`/api/agg/…`](docs/AGGREGATE.md), live SSE поток — `/api/live/events`, метрики Prometheus на `/metrics`. **Web UI** (SvelteKit) встроен в тот же процесс/образ: статика на `/`, API на `/api/…` и `/health`. ## Быстрый старт (Linux) @@ -81,6 +81,8 @@ cors_allowed_origins: | **[docs/API.md](docs/API.md)** | Контракт Telemt Control API (`/v1/…`) | | **[docs/AGGREGATE.md](docs/AGGREGATE.md)** | Агрегирующие эндпоинты шлюза (`/api/agg/…`), CORS, кэш | | **[docs/AGGREGATE_OPENAPI.yaml](docs/AGGREGATE_OPENAPI.yaml)** | OpenAPI 3 черновик для `/api/agg/*` (генерация типов для UI) | +| **[docs/OPERATIONS_BASELINE.md](docs/OPERATIONS_BASELINE.md)** | Baseline UX/SLO для панели быстрого реагирования (MTTD/MTTR и критерии успеха) | +| **[docs/INCIDENT_ROLLOUT.md](docs/INCIDENT_ROLLOUT.md)** | Пошаговый rollout incidents/live функций и настройка alert policy | | **[web/README.md](web/README.md)** | Web UI (SvelteKit): разработка с Vite, `PUBLIC_TELEMT_GATEWAY_URL`, встраивание в образ шлюза | | **[docs/GEOIP.md](docs/GEOIP.md)** | GeoLite2 City (страна/город) и опционально ASN (номер AS, организация) для IP в `unique-ips` | @@ -90,6 +92,14 @@ cors_allowed_origins: go mod tidy && go test ./... ``` +## Observability (gateway) + +Prometheus метрики доступны на `/metrics`, включая: + +- `telemt_gateway_http_in_flight` +- `telemt_gateway_http_requests_total{code,method,alias,endpoint}` +- `telemt_gateway_http_request_duration_seconds{method,alias,endpoint}` + ## CI/CD В репозитории: [.gitea/workflows/docker.yaml](.gitea/workflows/docker.yaml) — тесты Go, сборка и публикация образа в Container Registry Gitea (см. раздел «Обновление и CI/CD» в [docs/GATEWAY_RUN.md](docs/GATEWAY_RUN.md)). diff --git a/docs/AGGREGATE.md b/docs/AGGREGATE.md index 02d9877..61437b7 100644 --- a/docs/AGGREGATE.md +++ b/docs/AGGREGATE.md @@ -33,6 +33,7 @@ | GET | `/api/agg/users` | Объединённый список пользователей с `by_server`, суммарным `total_megabytes` и **смерженными лимитами** (см. ниже). | | GET | `/api/agg/user/{username}` | Один пользователь в том же формате, что элементы `/api/agg/users` (без списка всех). Имя в пути: `[A-Za-z0-9_.-]+`. Ответ **`404`**, если пользователь не найден ни на одном успешном upstream. | | GET | `/api/agg/fleet-status` | По каждому алиасу: параллельно health + system/info; в `data.servers[]` — статусы подзапросов и тела `health` / `system_info` при успехе. См. [AGGREGATE_OPENAPI.yaml](AGGREGATE_OPENAPI.yaml). | +| GET | `/api/agg/incidents` | Нормализованный snapshot инцидентов для triage-панели: `critical/warning/info`, `affected_aliases`, рекомендуемые `actions` (runbook/deep links), счётчики по severity. | Все методы — **GET**; действует тот же whitelist, что и для остального API шлюза. @@ -57,6 +58,18 @@ | `include_links` | `users`, `user/…` | `true` — добавить сгенерированные `tg://proxy` ссылки (берётся первая успешная запись по пользователю). | | `min_total_megabytes` | `users` | Порог суммарного трафика пользователя в MiB (строго больше 0). | | `min_total_octets` | `users` | Устаревший вариант порога в октетах (если задан `min_total_megabytes`, он приоритетнее). | +| `aliases` | `incidents` | Список алиасов через запятую; позволяет строить incidents snapshot по выбранной группе нод. | + +## Live stream (SSE) + +Для оперативного режима NOC доступен поток событий: + +- **`GET /api/live/events`** (`text/event-stream`) +- query: `aliases` (опционально, как в `/api/agg/*`) +- событие: `event: snapshot` +- payload: JSON со статусом флота (`healthy/degraded/critical`), `partial`, и массивом `incidents` + +Поток рассчитан на UI-клиент с авто-reconnect (на фронте используется экспоненциальный backoff). ## Конфигурация (опционально) @@ -99,4 +112,8 @@ curl -sS "http://127.0.0.1:8080/api/agg/unique-ips" curl -sS "http://127.0.0.1:8080/api/agg/users?include_links=false&min_total_megabytes=1" curl -sS "http://127.0.0.1:8080/api/agg/fleet-status" curl -sS "http://127.0.0.1:8080/api/agg/user/myuser?aliases=gt1" +curl -sS "http://127.0.0.1:8080/api/agg/incidents?aliases=gt1,gt2" + +# SSE поток snapshot-событий (пример с curl) +curl -N "http://127.0.0.1:8080/api/live/events?aliases=gt1,gt2" ``` diff --git a/docs/AGGREGATE_OPENAPI.yaml b/docs/AGGREGATE_OPENAPI.yaml index cf1ad02..0a89e87 100644 --- a/docs/AGGREGATE_OPENAPI.yaml +++ b/docs/AGGREGATE_OPENAPI.yaml @@ -106,6 +106,18 @@ paths: application/json: schema: { $ref: '#/components/schemas/AggEnvelopeFleetStatus' } + /api/agg/incidents: + get: + summary: Нормализованный snapshot инцидентов по флоту + parameters: + - $ref: '#/components/parameters/aliases' + responses: + '200': + description: OK + content: + application/json: + schema: { $ref: '#/components/schemas/AggEnvelopeIncidents' } + components: parameters: aliases: @@ -173,6 +185,13 @@ components: properties: data: { $ref: '#/components/schemas/FleetStatusData' } + AggEnvelopeIncidents: + allOf: + - $ref: '#/components/schemas/AggSuccessBase' + - type: object + properties: + data: { $ref: '#/components/schemas/IncidentsData' } + TrafficRow: type: object properties: @@ -297,3 +316,43 @@ components: servers_total: { type: integer } servers_all_ok: { type: integer } servers_failed: { type: integer } + + IncidentAction: + type: object + properties: + label: { type: string } + href: { type: string } + + IncidentItem: + type: object + properties: + id: { type: string } + kind: { type: string } + severity: + type: string + enum: [info, warning, critical] + status: + type: string + enum: [firing] + title: { type: string } + summary: { type: string } + affected_aliases: + type: array + items: { type: string } + metric_name: { type: string } + metric_value: { type: number, format: float } + metric_threshold: { type: number, format: float } + actions: + type: array + items: { $ref: '#/components/schemas/IncidentAction' } + + IncidentsData: + type: object + properties: + items: + type: array + items: { $ref: '#/components/schemas/IncidentItem' } + total: { type: integer } + critical_total: { type: integer } + warning_total: { type: integer } + info_total: { type: integer } diff --git a/docs/INCIDENT_ROLLOUT.md b/docs/INCIDENT_ROLLOUT.md new file mode 100644 index 0000000..02f0ba8 --- /dev/null +++ b/docs/INCIDENT_ROLLOUT.md @@ -0,0 +1,73 @@ +# Incident Rollout Playbook + +## Scope + +This playbook describes staged rollout for: + +- `/api/agg/incidents` +- `/api/live/events` (SSE snapshots) +- UI pages `/incidents` and `/live` +- Live polling controls and stale indicators on key pages + +## Stage 0 - Baseline Capture (2-3 days) + +- Record current MTTD and MTTR from on-call logs. +- Record manual refresh usage on main pages. +- Save top recurring failure patterns (degraded nodes, read-only modes, bad connections spikes). + +Outputs: + +- baseline MTTD / MTTR +- top 5 incident categories by frequency + +## Stage 1 - Shadow Mode (3-5 days) + +- Enable incidents and live pages for operators. +- Do not change paging/escalation yet. +- Compare incident feed against existing monitoring and mark false positives. + +Targets: + +- false positive ratio < 20% +- no increase in upstream load beyond acceptable budget + +## Stage 2 - Assisted Triage (1 week) + +- Use `/incidents` as primary triage board. +- Require owner + ack for active critical incidents. +- Use runbook links from incident items. + +Targets: + +- ack coverage for critical incidents >= 90% +- owner coverage for critical incidents >= 90% + +## Stage 3 - Policy Tuning (ongoing) + +- Adjust thresholds: + - `bad_connections_warn` (default 1000) + - `bad_connections_high` (default 10000) +- Review alert fatigue weekly. +- Promote stable thresholds into documented policy. + +## KPI Tracking + +- MTTD (minutes): incident first observed -> first ack +- MTTR (minutes): incident first observed -> resolved +- Stale time share: percentage of time live views are stale +- Manual refresh share: manual refresh / total data update actions + +## Fast Rollback + +If noise or load is excessive: + +1. disable auto-refresh by setting `refresh=0` in shared ops links +2. switch operators back to dashboard summary only +3. keep `/api/agg/incidents` for diagnostics while disabling SSE consumers + +## Weekly Review Template + +- KPI delta (MTTD, MTTR) vs baseline +- top noisy rules +- incidents with missing owner/ack +- policy changes applied this week diff --git a/docs/OPERATIONS_BASELINE.md b/docs/OPERATIONS_BASELINE.md new file mode 100644 index 0000000..e79f6c1 --- /dev/null +++ b/docs/OPERATIONS_BASELINE.md @@ -0,0 +1,49 @@ +# Operations Baseline (Telemt Panel) + +## Purpose + +This document defines the baseline operating model and target SLO metrics for the Telemt response panel. +It is used to measure impact of live updates, incidents workflow, and observability improvements. + +## Current Baseline (Before Rollout) + +- Dashboard refresh mode: mostly manual (`Refresh` buttons on key pages). +- Unified incident queue: not present. +- Alert ownership and acknowledgement flow: not present. +- MTTR tracking: not formalized in product UI. +- MTTD tracking: not formalized in product UI. +- Cross-page filtering consistency: partial, per-page controls only. + +## Baseline Risks + +- Slow detection when operators do not refresh at the right time. +- High context switching cost between pages during incidents. +- No single place to triage partial/degraded node states. +- Weak evidence trail for post-incident analysis. + +## SLO Targets (Phase Goals) + +- MTTD reduction: at least 30% versus baseline. +- MTTR reduction: at least 25% versus baseline. +- Manual refresh share during incident windows: under 10%. +- Incident cards with `owner + ack + runbook`: over 90%. + +## Measurement Inputs + +- Gateway API: + - `/api/agg/fleet-status` + - `/api/agg/summary` + - `/api/agg/incidents` (planned/implemented in this rollout) +- Gateway metrics endpoint: + - `/metrics` +- Frontend telemetry (local panel interaction metrics): + - refresh actions + - ack/resolve actions + - stale/live state durations + +## Rollout Validation Checklist + +- [ ] Baseline values captured before enabling auto-refresh. +- [ ] Incident flow tested with simulated degraded upstream. +- [ ] Alert noise review completed after first week. +- [ ] MTTD/MTTR comparison published for phase review. diff --git a/internal/aggregate/handlers.go b/internal/aggregate/handlers.go index 231e8a6..531960f 100644 --- a/internal/aggregate/handlers.go +++ b/internal/aggregate/handlers.go @@ -35,6 +35,11 @@ type Handler struct { cache map[string]cacheEntry } +// ResolveAliasesForLive resolves aliases for external endpoints (SSE/live). +func (h *Handler) ResolveAliasesForLive(r *http.Request) ([]string, error) { + return h.resolveAliases(r) +} + // NewHandler builds an aggregate handler; client must use a non-nil Transport (e.g. gateway shared transport). // Geo may be nil (no GeoLite2 lookups). cacheTTL 0 disables response caching. func NewHandler(p *config.Parsed, client *http.Client, geo *geoip.Service, cacheTTL time.Duration) *Handler { @@ -108,6 +113,8 @@ func (h *Handler) dispatch(w http.ResponseWriter, r *http.Request, sub string) { h.handleUsers(w, r) case sub == "fleet-status": h.handleFleetStatus(w, r) + case sub == "incidents": + h.handleIncidents(w, r) case strings.HasPrefix(sub, "user/"): username := strings.TrimPrefix(sub, "user/") if username == "" { @@ -304,6 +311,18 @@ func (h *Handler) handleUserOne(w http.ResponseWriter, r *http.Request, username writeAggOK(w, partial, row) } +func (h *Handler) handleIncidents(w http.ResponseWriter, r *http.Request) { + aliases, err := h.resolveAliases(r) + if err != nil { + writeBadRequest(w, err) + return + } + ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second) + defer cancel() + data, partial := BuildIncidents(ctx, h, aliases) + writeAggOK(w, partial, data) +} + func writeBadRequest(w http.ResponseWriter, err error) { w.Header().Set("Content-Type", "application/json; charset=utf-8") w.WriteHeader(http.StatusBadRequest) diff --git a/internal/aggregate/incidents.go b/internal/aggregate/incidents.go new file mode 100644 index 0000000..d1b8623 --- /dev/null +++ b/internal/aggregate/incidents.go @@ -0,0 +1,154 @@ +package aggregate + +import ( + "context" + "fmt" + "sort" + "strings" + "time" +) + +type statsSummaryLite struct { + ConnectionsBadTotal uint64 `json:"connections_bad_total"` + ConnectionsTotal uint64 `json:"connections_total"` +} + +func BuildIncidents(ctx context.Context, h *Handler, aliases []string) (IncidentsData, bool) { + out := IncidentsData{Items: make([]IncidentItem, 0)} + partial := false + + fleet := FetchFleetStatus(ctx, h.Client, h.Parsed, aliases) + if fleet.ServersFailed > 0 { + partial = true + } + + for _, s := range fleet.Servers { + if !s.OK { + severity := SeverityWarning + if !s.HealthOK && !s.SystemInfoOK { + severity = SeverityCritical + } + out.Items = append(out.Items, IncidentItem{ + ID: fmt.Sprintf("node_degraded:%s", s.Alias), + Kind: "node_degraded", + Severity: severity, + Status: IncidentStatusFiring, + Title: fmt.Sprintf("Нода %s degraded", s.Alias), + Summary: "health/system_info не прошли полностью", + AffectedAliases: []string{s.Alias}, + Actions: []IncidentAction{ + {Label: "Открыть ноду", Href: "/servers/" + s.Alias}, + {Label: "Runtime", Href: "/servers/" + s.Alias + "/runtime"}, + }, + }) + } + if s.Health != nil && s.Health.ReadOnly { + out.Items = append(out.Items, IncidentItem{ + ID: fmt.Sprintf("node_read_only:%s", s.Alias), + Kind: "node_read_only", + Severity: SeverityWarning, + Status: IncidentStatusFiring, + Title: fmt.Sprintf("Нода %s в read_only", s.Alias), + Summary: "API ноды не принимает mutating операции", + AffectedAliases: []string{s.Alias}, + Actions: []IncidentAction{ + {Label: "Users", Href: "/servers/" + s.Alias + "/users"}, + {Label: "Security", Href: "/servers/" + s.Alias + "/security"}, + }, + }) + } + } + + for _, alias := range aliases { + data, meta := FetchTelemtGET[statsSummaryLite](ctx, h.Client, h.Parsed, alias, "stats/summary") + if !meta.OK { + partial = true + continue + } + if data.ConnectionsBadTotal >= 10000 { + out.Items = append(out.Items, IncidentItem{ + ID: fmt.Sprintf("bad_connections_high:%s", alias), + Kind: "bad_connections_high", + Severity: SeverityCritical, + Status: IncidentStatusFiring, + Title: fmt.Sprintf("Высокий bad connections на %s", alias), + Summary: "Резкий рост ошибок клиентских соединений", + AffectedAliases: []string{alias}, + MetricName: "connections_bad_total", + MetricValue: float64(data.ConnectionsBadTotal), + MetricThreshold: 10000, + Actions: []IncidentAction{ + {Label: "Node dashboard", Href: "/servers/" + alias}, + }, + }) + } else if data.ConnectionsBadTotal >= 1000 { + out.Items = append(out.Items, IncidentItem{ + ID: fmt.Sprintf("bad_connections_warn:%s", alias), + Kind: "bad_connections_warn", + Severity: SeverityWarning, + Status: IncidentStatusFiring, + Title: fmt.Sprintf("Рост bad connections на %s", alias), + Summary: "Наблюдается рост ошибок клиентских соединений", + AffectedAliases: []string{alias}, + MetricName: "connections_bad_total", + MetricValue: float64(data.ConnectionsBadTotal), + MetricThreshold: 1000, + Actions: []IncidentAction{ + {Label: "Node dashboard", Href: "/servers/" + alias}, + }, + }) + } + } + + sort.Slice(out.Items, func(i, j int) bool { + if severityWeight(out.Items[i].Severity) != severityWeight(out.Items[j].Severity) { + return severityWeight(out.Items[i].Severity) > severityWeight(out.Items[j].Severity) + } + return strings.Compare(out.Items[i].ID, out.Items[j].ID) < 0 + }) + + out.Total = len(out.Items) + for _, it := range out.Items { + switch it.Severity { + case SeverityCritical: + out.CriticalTotal++ + case SeverityWarning: + out.WarningTotal++ + default: + out.InfoTotal++ + } + } + return out, partial +} + +func severityWeight(s IncidentSeverity) int { + switch s { + case SeverityCritical: + return 3 + case SeverityWarning: + return 2 + case SeverityInfo: + return 1 + default: + return 0 + } +} + +func BuildLiveEnvelope(ctx context.Context, h *Handler, aliases []string) map[string]any { + inc, partial := BuildIncidents(ctx, h, aliases) + status := "healthy" + if inc.CriticalTotal > 0 { + status = "critical" + } else if inc.WarningTotal > 0 { + status = "degraded" + } + return map[string]any{ + "type": "live_snapshot", + "timestamp": time.Now().UTC().Format(time.RFC3339Nano), + "status": status, + "partial": partial, + "incidents": inc.Items, + "counts": map[string]int{"total": inc.Total, "critical": inc.CriticalTotal, "warning": inc.WarningTotal, "info": inc.InfoTotal}, + "aliases_used": aliases, + } +} diff --git a/internal/aggregate/types.go b/internal/aggregate/types.go index 70fe85b..1be7b23 100644 --- a/internal/aggregate/types.go +++ b/internal/aggregate/types.go @@ -149,6 +149,52 @@ type SummaryData struct { TopUsersByUniqueIPs []TopUserByUniqueIPs `json:"top_users_by_unique_ips"` } +// IncidentSeverity is normalized severity used by aggregate incidents. +type IncidentSeverity string + +const ( + SeverityInfo IncidentSeverity = "info" + SeverityWarning IncidentSeverity = "warning" + SeverityCritical IncidentSeverity = "critical" +) + +// IncidentStatus is current calculated incident state. +type IncidentStatus string + +const ( + IncidentStatusFiring IncidentStatus = "firing" +) + +// IncidentAction points to a UI route with details/runbook context. +type IncidentAction struct { + Label string `json:"label"` + Href string `json:"href"` +} + +// IncidentItem is one normalized fleet incident. +type IncidentItem struct { + ID string `json:"id"` + Kind string `json:"kind"` + Severity IncidentSeverity `json:"severity"` + Status IncidentStatus `json:"status"` + Title string `json:"title"` + Summary string `json:"summary"` + AffectedAliases []string `json:"affected_aliases,omitempty"` + MetricName string `json:"metric_name,omitempty"` + MetricValue float64 `json:"metric_value,omitempty"` + MetricThreshold float64 `json:"metric_threshold,omitempty"` + Actions []IncidentAction `json:"actions,omitempty"` +} + +// IncidentsData is aggregate response payload for /api/agg/incidents. +type IncidentsData struct { + Items []IncidentItem `json:"items"` + Total int `json:"total"` + CriticalTotal int `json:"critical_total"` + WarningTotal int `json:"warning_total"` + InfoTotal int `json:"info_total"` +} + // TopUser by summed traffic across servers for one username (мегабайты). type TopUser struct { Username string `json:"username"` diff --git a/internal/server/gateway.go b/internal/server/gateway.go index 48baf09..4576558 100644 --- a/internal/server/gateway.go +++ b/internal/server/gateway.go @@ -3,6 +3,7 @@ package server import ( "context" "encoding/json" + "fmt" "log/slog" "net" "net/http" @@ -181,8 +182,9 @@ func (g *Gateway) withMetrics(next http.Handler) http.Handler { httpInFlight.Inc() start := time.Now() alias := routeAlias(r.URL.Path) + endpoint := routeEndpoint(r.URL.Path) lw := &statusWriter{ResponseWriter: w, status: http.StatusOK} - defer observeRequest(r.Method, alias, lw.status, start) + defer observeRequest(r.Method, alias, endpoint, lw.status, start) next.ServeHTTP(lw, r) }) } @@ -206,6 +208,33 @@ func routeAlias(path string) string { return rest[:i] } +func routeEndpoint(path string) string { + if path == "" || path == "/" { + return "ui_root" + } + if path == "/health" || path == "/metrics" { + return strings.TrimPrefix(path, "/") + } + if strings.HasPrefix(path, "/api/agg/") { + return strings.TrimPrefix(path, "/api/agg/") + } + if path == "/api/agg" { + return "agg" + } + if strings.HasPrefix(path, "/api/live/events") { + return "live_events" + } + if strings.HasPrefix(path, "/api/") { + rest := strings.TrimPrefix(path, "/api/") + i := strings.IndexByte(rest, '/') + if i < 0 { + return "proxy_root" + } + return "proxy_" + rest[i+1:] + } + return "ui" +} + func (g *Gateway) serve(w http.ResponseWriter, r *http.Request) { switch r.URL.Path { case "/health": @@ -233,6 +262,10 @@ func (g *Gateway) serve(w http.ResponseWriter, r *http.Request) { g.agg.ServeHTTP(w, r) return } + if r.URL.Path == "/api/live/events" { + g.serveLiveEvents(w, r) + return + } if !strings.HasPrefix(r.URL.Path, prefix) { g.webUI.ServeHTTP(w, r) return @@ -265,6 +298,54 @@ func (g *Gateway) serve(w http.ResponseWriter, r *http.Request) { rp.ServeHTTP(w, r) } +func (g *Gateway) serveLiveEvents(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + return + } + flusher, ok := w.(http.Flusher) + if !ok { + http.Error(w, "stream unsupported", http.StatusInternalServerError) + return + } + aliases, err := g.agg.ResolveAliasesForLive(r) + if err != nil { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + w.WriteHeader(http.StatusBadRequest) + _ = json.NewEncoder(w).Encode(map[string]any{ + "ok": false, + "error": map[string]string{"code": "bad_request", "message": err.Error()}, + }) + return + } + w.Header().Set("Content-Type", "text/event-stream") + w.Header().Set("Cache-Control", "no-cache") + w.Header().Set("Connection", "keep-alive") + w.WriteHeader(http.StatusOK) + + ticker := time.NewTicker(5 * time.Second) + defer ticker.Stop() + + push := func() { + ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second) + defer cancel() + payload := aggregate.BuildLiveEnvelope(ctx, g.agg, aliases) + b, _ := json.Marshal(payload) + _, _ = fmt.Fprintf(w, "event: snapshot\n") + _, _ = fmt.Fprintf(w, "data: %s\n\n", string(b)) + flusher.Flush() + } + push() + for { + select { + case <-r.Context().Done(): + return + case <-ticker.C: + push() + } + } +} + type statusWriter struct { http.ResponseWriter status int diff --git a/internal/server/metrics.go b/internal/server/metrics.go index 0db6300..9b5115e 100644 --- a/internal/server/metrics.go +++ b/internal/server/metrics.go @@ -15,17 +15,17 @@ var ( }) httpRequests = promauto.NewCounterVec(prometheus.CounterOpts{ Name: "telemt_gateway_http_requests_total", - Help: "HTTP requests by status, method, alias.", - }, []string{"code", "method", "alias"}) + Help: "HTTP requests by status, method, alias, endpoint.", + }, []string{"code", "method", "alias", "endpoint"}) httpDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{ Name: "telemt_gateway_http_request_duration_seconds", Help: "Request duration in seconds.", Buckets: prometheus.DefBuckets, - }, []string{"method", "alias"}) + }, []string{"method", "alias", "endpoint"}) ) -func observeRequest(method, alias string, status int, started time.Time) { +func observeRequest(method, alias, endpoint string, status int, started time.Time) { httpInFlight.Dec() - httpRequests.WithLabelValues(strconv.Itoa(status), method, alias).Inc() - httpDuration.WithLabelValues(method, alias).Observe(time.Since(started).Seconds()) + httpRequests.WithLabelValues(strconv.Itoa(status), method, alias, endpoint).Inc() + httpDuration.WithLabelValues(method, alias, endpoint).Observe(time.Since(started).Seconds()) } diff --git a/web/README.md b/web/README.md index 5eaca71..e437c2e 100644 --- a/web/README.md +++ b/web/README.md @@ -2,6 +2,14 @@ SvelteKit + shadcn-svelte. В **production** статика собирается и **встраивается в образ шлюза** ([Dockerfile](../Dockerfile) в корне репозитория): панель и API на **одном порту** (например `http://127.0.0.1:8080/` — UI, `/api/…` — шлюз). +## Основные разделы панели + +- `/` — обзор флота (KPI, активные IP, сводка по нодам) +- `/users`, `/users/[username]` — пользователи и детали +- `/ips` — unique IP + GeoIP карта +- `/incidents` — triage-интерфейс инцидентов (`ack/resolved/owner/note` в localStorage) +- `/live` — live snapshot (SSE, авто-reconnect) + ## Переменная `PUBLIC_TELEMT_GATEWAY_URL` | Значение | Когда | @@ -42,6 +50,17 @@ npm run gen:api Источник: [../docs/AGGREGATE_OPENAPI.yaml](../docs/AGGREGATE_OPENAPI.yaml). +## URL-driven controls (операторский режим) + +На ключевых страницах используются query-параметры: + +- `aliases=node-a,node-b` — фильтр по нодам +- `refresh=0|10..300` — auto-refresh (0 выключает polling) +- `include_links=0|1` — для `/users` +- `geo=0|1` — для `/ips` + +Live-страница `/live` работает через SSE endpoint шлюза: `/api/live/events`. + ## Ограничения - Секреты upstream к Telemt задаются на шлюзе (`authorization_env`), не в браузере. diff --git a/web/src/lib/api/client.ts b/web/src/lib/api/client.ts index 8c4d4b4..6298670 100644 --- a/web/src/lib/api/client.ts +++ b/web/src/lib/api/client.ts @@ -20,6 +20,37 @@ export type AggEnvelope = { data: T; }; +export type IncidentSeverity = 'info' | 'warning' | 'critical'; + +export type IncidentStatus = 'firing'; + +export type IncidentAction = { + label: string; + href: string; +}; + +export type IncidentItem = { + id: string; + kind: string; + severity: IncidentSeverity; + status: IncidentStatus; + title: string; + summary: string; + affected_aliases?: string[]; + metric_name?: string; + metric_value?: number; + metric_threshold?: number; + actions?: IncidentAction[]; +}; + +export type IncidentsData = { + items: IncidentItem[]; + total: number; + critical_total: number; + warning_total: number; + info_total: number; +}; + export class ApiError extends Error { constructor( message: string, @@ -120,6 +151,23 @@ export async function fetchAggUser( return body as AggEnvelope; } +export async function fetchAggIncidents(params?: { aliases?: string }): Promise> { + const q = new URLSearchParams(); + if (params?.aliases) q.set('aliases', params.aliases); + const url = `${gatewayBase()}/api/agg/incidents${q.toString() ? `?${q}` : ''}`; + const res = await fetch(url); + const body = (await parseJson(res)) as Record | null; + if (!res.ok) throw new ApiError(`incidents HTTP ${res.status}`, res.status, body); + if (!body || body.ok !== true) throw new ApiError('incidents: ok !== true', res.status, body); + return body as AggEnvelope; +} + +export function liveEventsUrl(params?: { aliases?: string }): string { + const q = new URLSearchParams(); + if (params?.aliases) q.set('aliases', params.aliases); + return `${gatewayBase()}/api/live/events${q.toString() ? `?${q}` : ''}`; +} + /** Путь к upstream без префикса /v1 — шлюз сам добавляет path_prefix. */ function apiUrl(alias: string, path: string): string { const p = path.replace(/^\/+/, ''); diff --git a/web/src/lib/components/app-sidebar.svelte b/web/src/lib/components/app-sidebar.svelte index bc0f767..6dd63aa 100644 --- a/web/src/lib/components/app-sidebar.svelte +++ b/web/src/lib/components/app-sidebar.svelte @@ -9,6 +9,8 @@ import NetworkIcon from '@lucide/svelte/icons/network'; import SettingsIcon from '@lucide/svelte/icons/settings'; import RefreshCwIcon from '@lucide/svelte/icons/refresh-cw'; + import SirenIcon from '@lucide/svelte/icons/siren'; + import RadioIcon from '@lucide/svelte/icons/radio'; import * as Sidebar from '$lib/components/ui/sidebar/index.js'; let { @@ -76,6 +78,26 @@ {/snippet} + + + {#snippet child({ props })} + + + Инциденты + + {/snippet} + + + + + {#snippet child({ props })} + + + Live + + {/snippet} + + {#if serverAliases.length > 0} diff --git a/web/src/routes/+page.svelte b/web/src/routes/+page.svelte index 120c738..919e2ee 100644 --- a/web/src/routes/+page.svelte +++ b/web/src/routes/+page.svelte @@ -1,5 +1,7 @@ + +
+
+

Инциденты

+

Сводка и triage по всему флоту.

+ {#if generatedAt} +

Снимок: {new Date(generatedAt).toLocaleString()}

+ {/if} +
+
+ + {isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if} + + +
+
+ +
+ + + +
+ +{#if partial} + + Частичные данные + Не все upstream ответили успешно. + +{/if} + +{#if err} + + Ошибка + {err} + +{:else} +
+ + + Critical + {data?.critical_total ?? 0} + + + + + Warning + {data?.warning_total ?? 0} + + + + + Info + {data?.info_total ?? 0} + + +
+ + + + + + + Severity + Инцидент + Серверы + Runbook + Triage + + + + {#if (data?.items.length ?? 0) === 0} + + + Инцидентов нет + + + {:else} + {#each data?.items ?? [] as item (item.id)} + + + {item.severity} + + +
{item.title}
+
{item.summary}
+ {#if item.metric_name} +
+ {item.metric_name}: {item.metric_value} / {item.metric_threshold} +
+ {/if} + + +
+ {#each item.affected_aliases ?? [] as alias (alias)} + {alias} + {/each} +
+ + +
+ {#each item.actions ?? [] as action (action.label + action.href)} + + {action.label} + + {/each} +
+ + + {@const triage = triageFor(item.id)} +
+ + +
+
+ + patchTriage(item.id, { + owner: (e.currentTarget as HTMLInputElement).value + })} + /> +
+
+ + patchTriage(item.id, { + note: (e.currentTarget as HTMLInputElement).value + })} + /> +
+ + + {/each} + {/if} + + + + +{/if} diff --git a/web/src/routes/ips/+page.svelte b/web/src/routes/ips/+page.svelte index e06da38..0e055fa 100644 --- a/web/src/routes/ips/+page.svelte +++ b/web/src/routes/ips/+page.svelte @@ -1,5 +1,7 @@
@@ -301,17 +378,50 @@

Снимок active/recent с шлюза; GeoIP из конфига шлюза.

+ + {isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if} + -
+
+ + + +
+ {#if partial} Частичные данные diff --git a/web/src/routes/live/+page.svelte b/web/src/routes/live/+page.svelte new file mode 100644 index 0000000..64bb1f5 --- /dev/null +++ b/web/src/routes/live/+page.svelte @@ -0,0 +1,348 @@ + + +
+
+

Live

+

SSE поток инцидентов и статуса флота.

+
+
+ + {connected ? 'connected' : 'disconnected'} + + + {isStale ? 'stale' : 'live'}{#if staleSeconds != null} · {staleSeconds}s{/if} + + +
+
+ +
+ + +
+ +{#if err} + + Live stream + + {err} + {#if reconnectInSec != null} + Переподключение через {reconnectInSec}s. + {/if} + + +{/if} + +
+ + + Общий статус + + + {snapshot?.status ?? 'healthy'} + + + + + + + Partial + {snapshot?.partial ? 'yes' : 'no'} + + + + + Timestamp + + {snapshot?.timestamp ? new Date(snapshot.timestamp).toLocaleString() : '—'} + + + +
+ + + + Последние incidents из snapshot + Всего: {snapshot?.counts?.total ?? snapshot?.incidents?.length ?? 0} + + + + + + Severity + Title + Summary + Aliases + + + + {#if (snapshot?.incidents?.length ?? 0) === 0} + + + Инцидентов нет + + + {:else} + {#each snapshot?.incidents ?? [] as item (item.id)} + + + {item.severity} + + {item.title} + {item.summary} + +
+ {#each item.affected_aliases ?? [] as alias (alias)} + {alias} + {/each} +
+ + + {/each} + {/if} + + + + diff --git a/web/src/routes/users/+page.svelte b/web/src/routes/users/+page.svelte index 01bb779..7b3b717 100644 --- a/web/src/routes/users/+page.svelte +++ b/web/src/routes/users/+page.svelte @@ -1,9 +1,11 @@