Denozordec
9fa6269174
quality / changes (push) Successful in 7s
quality / commitlint (push) Skipped
quality / openapi (push) Successful in 30s
quality / web (push) Successful in 1m3s
quality / api (push) Successful in 1m15s
quality / linux-agent-smoke (push) Failing after 27s
quality / docker-check (push) Failing after 2m28s
CD / quality (push) Failing after 5m52s
CD / publish (push) Skipped
Implement reliable agent apply, telemetry v2, analytics retention, guarded rollouts, enrollment, labels and webhooks. Improve ReUI frontend data safety and bounded list loading, and add Docker smoke, recovery and resource checks. Document local verification and pending native acceptance.
32 lines
5.5 KiB
Markdown
32 lines
5.5 KiB
Markdown
# Source refresh checkpoint
|
|
|
|
JSON URL sources and EvoBGP now share a bounded Node HTTP(S) transport. It resolves DNS inside the connection's `lookup` callback and returns the vetted address/addresses directly to the socket. It does not perform a preflight lookup followed by an independent connection lookup. Every resolved address must be public; mixed public/private DNS answers fail closed. IP literals receive the same policy check before connecting. Loopback, private/link-local, reserved, mapped and IPv6 transition destinations require an explicit operator allowlist. HTTP(S) URLs with embedded credentials and other protocols are rejected.
|
|
|
|
Private source origins are permitted only through the server environment, for example:
|
|
|
|
```yaml
|
|
environment:
|
|
EVOFW_SOURCE_ALLOWED_ORIGINS: "https://bgp.internal.example:8443,http://10.20.0.5:8080"
|
|
```
|
|
|
|
Entries are comma-separated **exact origins** (scheme, hostname/IP and port). No wildcard, path, query, suffix or list-config bypass is supported. `https://bgp.internal.example:8443` does not authorize port 443, another hostname or its redirects. Keep this operator configuration narrow; an allowed internal service is intentionally trusted to resolve to private addresses. Existing internal EvoBGP deployments must add their origin before enabling these changes.
|
|
|
|
Each response has an 8MiB body limit, 16KiB header limit and a 45s wall-clock deadline covering DNS, headers and body. Both declared and streamed body sizes are checked. Requests ask for identity encoding and reject compressed responses, preventing an unbounded decompression path. JSON URL data is limited to 100,000 input elements, including duplicates. Redirects are processed manually with a maximum of three hops and a shared deadline; each destination is revalidated. HTTPS-to-HTTP redirects fail. Secret headers remain available only for same-origin redirects; after an origin change they are removed permanently, including a subsequent redirect back to the original origin. TLS certificate checks and the original hostname/SNI remain enabled.
|
|
|
|
EvoBGP pagination accepts at most 1,000 pages, 100,000 elements and 32MiB total, with a 45s deadline across the complete collection. Missing/repeated/invalid cursors, invalid continuation metadata and malformed page items fail the snapshot. Pages are collected completely before publishing any prefix. Community-label healing may first collect the communities endpoint and then the prefixes endpoint; each collection has its own bounded deadline. The autocomplete proxy uses the same transport and pagination and rejects malformed community records with an upstream error.
|
|
|
|
Refreshes are coalesced per database/list and limited to two concurrently across the process. A refresh captures the source list revision before network access and checks it before cascade staging; the existing transaction guard rechecks the staged dependency snapshot before commit. Results from a changed list cannot overwrite its newer configuration. EvoBGP integration settings are checked again before publishing. Parent DNS/materialization failures retain the whole previous cascade.
|
|
|
|
An empty upstream snapshot is quarantined unless the list configuration explicitly contains `"allow_empty": true`. Failed/partial/oversized/empty/stale results do not replace the last successful prefix data. Identical CIDR sets preserve entry IDs and agent generations. Diagnostics use existing fields: `last_error` records the bounded error, `entry_count` records the published volume, and `refreshed_at` now represents the last **successful** refresh. Failure no longer changes that timestamp. Stale requests do not write an error onto a newly edited list. No secrets or complete request URLs are included in transport errors.
|
|
|
|
Hostname materialization queries A and AAAA concurrently using one resolver and a shared five-second deadline; timeout cancels both queries. Only `ENODATA` permits publishing the successful other family, because it indicates authoritative absence for that record type. `ENOTFOUND`/NXDOMAIN concerns the whole hostname; contradictory partial answers and transient failures such as SERVFAIL/TIMEOUT reject the complete snapshot. An all-empty DNS result also preserves the previous cache. List cascade generation updates now include agents whose port ACL references the list directly, and deduplicate agent IDs across all changed lists/parents/sets in that publication. A policy affected through multiple dependencies receives one generation increment, while unchanged snapshots receive none.
|
|
|
|
Verification commands from `apps/api`:
|
|
|
|
```sh
|
|
node node_modules/typescript/bin/tsc --noEmit
|
|
node node_modules/vitest/vitest.mjs run src/services/source-hardening.test.ts src/services/source-dns.test.ts src/services/backend-hardening.test.ts
|
|
```
|
|
|
|
Tests cover actual local HTTP sockets (explicitly allowed origins), DNS-address handoff, mixed/private DNS rejection, malformed allowlist handling, redirects and credential removal, streamed/declared limits, headers, timeouts, format/encoding rejection, pagination budgets, stale edits, empty quarantine, refresh coalescing/concurrency and previous cascade regressions. Windows sandbox blocked localhost connections; the same focused tests passed with an approved sandbox escalation. Linux container execution, real EvoBGP integration and load/resource measurements remain pending. Policy ETags/caching, event-loop telemetry and the expanded NAT-agent report/retry benchmark are separate pending work; the current ops benchmark only measures policy/heartbeat sweeps and one import.
|