Denozordec
9fa6269174
Implement reliable agent apply, telemetry v2, analytics retention, guarded rollouts, enrollment, labels and webhooks. Improve ReUI frontend data safety and bounded list loading, and add Docker smoke, recovery and resource checks. Document local verification and pending native acceptance.
5.5 KiB
Source refresh checkpoint
JSON URL sources and EvoBGP now share a bounded Node HTTP(S) transport. It resolves DNS inside the connection's lookup callback and returns the vetted address/addresses directly to the socket. It does not perform a preflight lookup followed by an independent connection lookup. Every resolved address must be public; mixed public/private DNS answers fail closed. IP literals receive the same policy check before connecting. Loopback, private/link-local, reserved, mapped and IPv6 transition destinations require an explicit operator allowlist. HTTP(S) URLs with embedded credentials and other protocols are rejected.
Private source origins are permitted only through the server environment, for example:
environment:
EVOFW_SOURCE_ALLOWED_ORIGINS: "https://bgp.internal.example:8443,http://10.20.0.5:8080"
Entries are comma-separated exact origins (scheme, hostname/IP and port). No wildcard, path, query, suffix or list-config bypass is supported. https://bgp.internal.example:8443 does not authorize port 443, another hostname or its redirects. Keep this operator configuration narrow; an allowed internal service is intentionally trusted to resolve to private addresses. Existing internal EvoBGP deployments must add their origin before enabling these changes.
Each response has an 8MiB body limit, 16KiB header limit and a 45s wall-clock deadline covering DNS, headers and body. Both declared and streamed body sizes are checked. Requests ask for identity encoding and reject compressed responses, preventing an unbounded decompression path. JSON URL data is limited to 100,000 input elements, including duplicates. Redirects are processed manually with a maximum of three hops and a shared deadline; each destination is revalidated. HTTPS-to-HTTP redirects fail. Secret headers remain available only for same-origin redirects; after an origin change they are removed permanently, including a subsequent redirect back to the original origin. TLS certificate checks and the original hostname/SNI remain enabled.
EvoBGP pagination accepts at most 1,000 pages, 100,000 elements and 32MiB total, with a 45s deadline across the complete collection. Missing/repeated/invalid cursors, invalid continuation metadata and malformed page items fail the snapshot. Pages are collected completely before publishing any prefix. Community-label healing may first collect the communities endpoint and then the prefixes endpoint; each collection has its own bounded deadline. The autocomplete proxy uses the same transport and pagination and rejects malformed community records with an upstream error.
Refreshes are coalesced per database/list and limited to two concurrently across the process. A refresh captures the source list revision before network access and checks it before cascade staging; the existing transaction guard rechecks the staged dependency snapshot before commit. Results from a changed list cannot overwrite its newer configuration. EvoBGP integration settings are checked again before publishing. Parent DNS/materialization failures retain the whole previous cascade.
An empty upstream snapshot is quarantined unless the list configuration explicitly contains "allow_empty": true. Failed/partial/oversized/empty/stale results do not replace the last successful prefix data. Identical CIDR sets preserve entry IDs and agent generations. Diagnostics use existing fields: last_error records the bounded error, entry_count records the published volume, and refreshed_at now represents the last successful refresh. Failure no longer changes that timestamp. Stale requests do not write an error onto a newly edited list. No secrets or complete request URLs are included in transport errors.
Hostname materialization queries A and AAAA concurrently using one resolver and a shared five-second deadline; timeout cancels both queries. Only ENODATA permits publishing the successful other family, because it indicates authoritative absence for that record type. ENOTFOUND/NXDOMAIN concerns the whole hostname; contradictory partial answers and transient failures such as SERVFAIL/TIMEOUT reject the complete snapshot. An all-empty DNS result also preserves the previous cache. List cascade generation updates now include agents whose port ACL references the list directly, and deduplicate agent IDs across all changed lists/parents/sets in that publication. A policy affected through multiple dependencies receives one generation increment, while unchanged snapshots receive none.
Verification commands from apps/api:
node node_modules/typescript/bin/tsc --noEmit
node node_modules/vitest/vitest.mjs run src/services/source-hardening.test.ts src/services/source-dns.test.ts src/services/backend-hardening.test.ts
Tests cover actual local HTTP sockets (explicitly allowed origins), DNS-address handoff, mixed/private DNS rejection, malformed allowlist handling, redirects and credential removal, streamed/declared limits, headers, timeouts, format/encoding rejection, pagination budgets, stale edits, empty quarantine, refresh coalescing/concurrency and previous cascade regressions. Windows sandbox blocked localhost connections; the same focused tests passed with an approved sandbox escalation. Linux container execution, real EvoBGP integration and load/resource measurements remain pending. Policy ETags/caching, event-loop telemetry and the expanded NAT-agent report/retry benchmark are separate pending work; the current ops benchmark only measures policy/heartbeat sweeps and one import.