Hub stage 4: config history, rollback, drift check, baseline export and import

Every saved config, and every earlier baseline, is listed per app. You
can view it, diff it against a server's current config, and roll it back
through preview and apply. Earlier baselines can be restored.

A drift check (HUB_DRIFT_MINUTES, default 15) compares each server with
its effective config. Drift shows in the header and the matrix, with
Accept as baseline and Revert to baseline. All baselines export to one
file. Importing that file shows a dry run before it writes.

Rolling back preselects, and diffs against, the server the version came
from. The line-diff op "del" is renamed to "remove" for the language lint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hub/README.md

@@ -22,6 +22,7 @@ apps Query Logs reads from) and Settings (the blocking state).
 | `<ID>_TOKEN` | required | API token of each server; the id is uppercased with non-alphanumerics as `_` (`dns` gives `DNS_TOKEN`, `my-dns` gives `MY_DNS_TOKEN`). |
 | `HUB_USER`, `HUB_PASSWORD` | see Users | Bootstrap: one `admin`, used only while `${HUB_DATA}/users.json` does not exist. Set both or neither. |
 | `HUB_DATA` | `/data` | Folder of `users.json` and `audit.log`. Mount a volume on it. |
+| `HUB_DRIFT_MINUTES` | `15` | How often the hub checks that every server still has the config it should (see Drift). `0` turns the check off. |
 | `HUB_TRUST_PROXY` | off | `1` believes the first `X-Forwarded-For` entry as the client address (set it behind Traefik). Leave it off when clients reach the hub directly, or anyone can fake their address. |
 | `<ID>_ADMIN_TOKEN` | optional | A write-capable token per server (`MDNS_ADMIN_TOKEN`). Only app writes use it, and only it: a server without one is read-only for the hub (the matrix and the session list the writable servers). Never logged or returned. |
 | `HUB_PORT` | `8080` | Listening port. |
@@ -128,6 +129,38 @@ matrix hash ignores them too.
 | `hub/apps/config/preview` | POST `{name, targets}` | `apps.write` | Per target: current and effective text, the diff, warnings (a missing local value, a version older than the newest installed, no admin token) and `currentHash`. |
 | `hub/apps/config/apply` | POST `{name, targets, expect?}` | `apps.write` | Per target: re-reads the config; refuses if its hash differs from `expect[server]`; saves the old config to the history; `apps/config/set` with the admin token; reads it back and checks it. A server that rejects the config keeps its old one. |
 
+## History, rollback, drift, export
+
+**History.** Everything the hub replaces is kept in `${HUB_DATA}/apps/<app>/history/` (50 files per app):
+`<time>__<server>__before-apply.txt`, `before-set` (a single-server config save), `before-uninstall`, and
+`<time>__baseline__before-save.txt` for the baseline's earlier text on every baseline save, import or restore.
+
+| Endpoint | Method | Needs | Does |
+|---|---|---|---|
+| `hub/apps/history/list?name=` | GET | read | Newest first: `{ id, at, server, kind, size, hash }`, never the text. |
+| `hub/apps/history/get?name=&id=` | GET | read | The text. `id` has to be a name the listing returns, so nothing outside the folder can be read. |
+| `hub/apps/history/diff?name=&id=&server=` | GET | read | A version against the server's config now (`changes`, `currentHash`, `unchanged`). |
+| `hub/apps/history/rollback` | POST `{name, id, targets, expect?}` | `apps.write` | Puts that version on the targets through the same machinery as apply (old config to the history, `apps/config/set` with the admin token, read back and verify, a result per target, audit). Not for baseline entries. |
+| `hub/apps/baseline/restore` | POST `{name, id}` | `apps.write` | An old baseline entry becomes the baseline again (the current text goes to the history). |
+
+**Drift.** Every `HUB_DRIFT_MINUTES` (and once shortly after start) the hub reads each server's config of every app that has a
+baseline, with the read token, and compares it by meaning with the server's effective config (baseline plus its local values). The
+result is kept in memory and in `${HUB_DATA}/drift.json`: `{ checkedAt, apps: [{ app, server, since, kind }] }`, where `kind` is
+`differs`, `unreadable` (the server or the read failed; listed, not an alarm) or `not-installed`, and `since` is the first check that
+saw it. Two checks never run at once.
+
+- `GET hub/drift/get` (read): the state. `POST hub/drift/check` (`apps.write`): check now.
+- `user/session/get` carries `info.hubDrift`, the number of `differs` entries; the console shows it in the header.
+- `POST hub/apps/drift/accept {name, server}` (`apps.write`): "Accept as baseline". The server's config becomes the baseline; if the
+  app has local fields, they keep the baseline's old values and the server's own values go to the overrides instead.
+- "Revert to baseline" is the normal apply on that server.
+
+**Export and import.** `GET hub/apps/export` (`apps.write`, because configs can hold secrets) downloads
+`{ hub: 'technitium-hub', version: 1, exportedAt, apps: [{ app, baseline, overrides }] }`. `POST hub/apps/import {data, apply}`
+(`apps.write`, up to 5 MB, signed-in callers only) validates the file and answers per app `new`, `changed` or `same`; with
+`apply=true` it writes the baselines and overrides (the replaced baselines go to the history). Local values for servers this hub
+does not have are ignored and reported. It never writes to a server.
+
 Reading configs needs View on Apps for the `hub-reader` user (and `*_ADMIN_TOKEN` is used as a fallback if a server refuses the
 read token). The audit log records who did what, with the size and a short hash of a config, never the text.
 

hub/apps-config.ts

@@ -10,6 +10,7 @@ but never written to the audit log.
 import type { HubConfig, HubServer } from './config.ts'
 import {
   MAX_CONFIG,
+  acceptServerConfig,
   cleanValues,
   diffConfigs,
   effectiveConfig,
@@ -22,7 +23,19 @@ import {
   type Diff,
 } from './apps-sync.ts'
 import { Upstream, UpstreamError } from './upstream.ts'
-import { readBaseline, readOverrides, writeBaseline, writeHistory, writeOverrides, type Baseline, type Overrides } from './store.ts'
+import {
+  listBaselineApps,
+  listHistory,
+  readBaseline,
+  readHistory,
+  readOverrides,
+  writeBaseline,
+  writeHistory,
+  writeOverrides,
+  type Baseline,
+  type HistoryEntry,
+  type Overrides,
+} from './store.ts'
 
 export class UserError extends Error {
   constructor(message: string) {
@@ -90,6 +103,19 @@ export function compareVersions(a: string, b: string): number {
   return 0
 }
 
+/* The read token first, the admin token when the server refuses it. */
+export async function readConfigText(upstream: Upstream, server: HubServer, name: string): Promise<string | null> {
+  let response: unknown
+  try {
+    response = await upstream.call(server, 'apps/config/get', { name })
+  } catch (e) {
+    if (!(e instanceof UpstreamError) || e.kind !== 'rejected' || server.adminToken === undefined) throw e
+    response = await upstream.call(server, 'apps/config/get', { name }, server.adminToken)
+  }
+  const config = (response as { config?: unknown } | null)?.config
+  return typeof config === 'string' ? config : null
+}
+
 export class AppsConfig {
   #config: HubConfig
   #upstream: Upstream
@@ -113,18 +139,9 @@ export class AppsConfig {
     return unique.map((id) => this.#server(id))
   }
 
-  /* The read token first, the admin token when the server refuses it. `fresh` skips the hub's cache. */
   async #readConfig(server: HubServer, name: string, fresh: boolean): Promise<string | null> {
     if (fresh) this.#upstream.invalidate(server.id)
-    let response: unknown
-    try {
-      response = await this.#upstream.call(server, 'apps/config/get', { name })
-    } catch (e) {
-      if (!(e instanceof UpstreamError) || e.kind !== 'rejected' || server.adminToken === undefined) throw e
-      response = await this.#upstream.call(server, 'apps/config/get', { name }, server.adminToken)
-    }
-    const config = (response as { config?: unknown } | null)?.config
-    return typeof config === 'string' ? config : null
+    return readConfigText(this.#upstream, server, name)
   }
 
   async #live(server: HubServer, name: string, fresh: boolean): Promise<Live> {
@@ -206,24 +223,36 @@ export class AppsConfig {
     }
     if (text === null || text.trim() === '') throw new UserError(`${serverId} has no config for "${name}" to import.`)
     if (text.length > MAX_CONFIG) throw new UserError('The config is larger than 1 MB.')
-    writeBaseline(this.#config.dataDir, { app: name, text, importedFrom: serverId, updatedAt: new Date(this.#now()).toISOString(), updatedBy: user })
+    this.#setBaseline(name, text, serverId, user)
     return { detail: `from ${serverId}, ${text.length} bytes, ${textHash(text)}` }
   }
 
+  /* Writes the baseline; the text it replaces goes to the history first (when it differs). */
+  #setBaseline(name: string, text: string, importedFrom: string | undefined, user: string): void {
+    const old = this.getBaseline(name)
+    if (old !== null && old.text !== text) {
+      try {
+        writeHistory(this.#config.dataDir, name, 'baseline', old.text, 'before-save', this.#now())
+      } catch (e) {
+        throw new UserError(`The old baseline could not be saved first (${reason(e)}); nothing was changed.`)
+      }
+    }
+    writeBaseline(this.#config.dataDir, {
+      app: name,
+      text,
+      ...(importedFrom === undefined ? {} : { importedFrom }),
+      updatedAt: new Date(this.#now()).toISOString(),
+      updatedBy: user,
+    })
+  }
+
   saveBaseline(name: string, text: string, user: string): { detail: string } {
     if (text.length > MAX_CONFIG) throw new UserError('The baseline is larger than 1 MB.')
     if (looksLikeJson(text)) {
       const parsed = parseJson(text)
       if (!parsed.ok) throw new UserError(`The baseline is not valid JSON: ${parsed.error}`)
     }
-    const old = this.getBaseline(name)
-    writeBaseline(this.#config.dataDir, {
-      app: name,
-      text,
-      ...(old?.importedFrom === undefined ? {} : { importedFrom: old.importedFrom }),
-      updatedAt: new Date(this.#now()).toISOString(),
-      updatedBy: user,
-    })
+    this.#setBaseline(name, text, this.getBaseline(name)?.importedFrom, user)
     return { detail: `${text.length} bytes, ${textHash(text)}` }
   }
 
@@ -292,47 +321,192 @@ export class AppsConfig {
     }
   }
 
-  async apply(name: string, ids: string[], expect: Record<string, string>): Promise<{ results: TargetResult[]; details: Record<string, string> }> {
+  /*
+  Brings one server to `desired`: re-read, refuse if it changed since `expectHash`, save the old
+  config to the history, write with the admin token, read back and check.
+  */
+  async #applyText(s: HubServer, name: string, desired: (current: string) => string, expectHash: string | undefined): Promise<{ result: TargetResult; detail?: string }> {
+    if (s.adminToken === undefined) return { result: { server: s.id, ok: false, message: 'no admin token configured' } }
+    try {
+      const l = await this.#live(s, name, true)
+      if (!l.ok) return { result: { server: s.id, ok: false, message: l.error ?? 'did not answer' } }
+      if (!l.installed) return { result: { server: s.id, ok: false, message: 'the app is not installed on this server' } }
+      const current = l.text ?? ''
+      if (expectHash !== undefined && expectHash !== textHash(current)) {
+        return { result: { server: s.id, ok: false, message: 'the config changed on the server since the preview; run the preview again' } }
+      }
+      const text = desired(current)
+      if (sameConfig(current, text)) return { result: { server: s.id, ok: true, message: 'Already in sync; nothing was written.' } }
+      if (text.length > MAX_CONFIG) return { result: { server: s.id, ok: false, message: 'the config is larger than 1 MB' } }
+      try {
+        writeHistory(this.#config.dataDir, name, s.id, current, 'before-apply', this.#now())
+      } catch (e) {
+        return { result: { server: s.id, ok: false, message: `the old config could not be saved first (${reason(e)}); nothing was changed` } }
+      }
+      await this.#upstream.write(s, 'apps/config/set', {}, new URLSearchParams({ name, config: text }))
+      const after = await this.#readConfig(s, name, true)
+      const detail = `${textHash(current)} to ${textHash(text)}, ${text.length} bytes`
+      if (after === null || !sameConfig(after, text)) {
+        return { result: { server: s.id, ok: false, message: 'the server accepted the config but reads back a different one' }, detail }
+      }
+      return { result: { server: s.id, ok: true, message: 'Applied and verified.' }, detail }
+    } catch (e) {
+      return { result: { server: s.id, ok: false, message: reason(e) } }
+    }
+  }
+
+  async #applyMany(
+    name: string,
+    ids: string[],
+    desiredFor: (s: HubServer) => (current: string) => string,
+    expect: Record<string, string>,
+  ): Promise<{ results: TargetResult[]; details: Record<string, string> }> {
     const targets = this.#targets(ids)
+    const done = await Promise.all(targets.map((s) => this.#applyText(s, name, desiredFor(s), expect[s.id])))
+    const details: Record<string, string> = {}
+    for (const d of done) if (d.detail !== undefined) details[d.result.server] = d.detail
+    return { results: done.map((d) => d.result), details }
+  }
+
+  async apply(name: string, ids: string[], expect: Record<string, string>): Promise<{ results: TargetResult[]; details: Record<string, string> }> {
     const baseline = this.getBaseline(name)
     if (baseline === null) throw new UserError('This app has no baseline yet. Import one from a server first.')
     const overrides = this.getOverrides(name)
-    const details: Record<string, string> = {}
+    return this.#applyMany(name, ids, (s) => () => effectiveConfig(baseline.text, overrides, s.id).text, expect)
+  }
 
-    const results = await Promise.all(
-      targets.map(async (s): Promise<TargetResult> => {
-        if (s.adminToken === undefined) return { server: s.id, ok: false, message: 'no admin token configured' }
-        try {
-          const l = await this.#live(s, name, true)
-          if (!l.ok) return { server: s.id, ok: false, message: l.error ?? 'did not answer' }
-          if (!l.installed) return { server: s.id, ok: false, message: 'the app is not installed on this server' }
-          const current = l.text ?? ''
-          if (expect[s.id] !== undefined && expect[s.id] !== textHash(current)) {
-            return { server: s.id, ok: false, message: 'the config changed on the server since the preview; run the preview again' }
-          }
-          const effective = effectiveConfig(baseline.text, overrides, s.id)
-          if (sameConfig(current, effective.text)) return { server: s.id, ok: true, message: 'Already in sync; nothing was written.' }
-          if (effective.text.length > MAX_CONFIG) return { server: s.id, ok: false, message: 'the config is larger than 1 MB' }
+  /* ---- History ---- */
 
-          try {
-            writeHistory(this.#config.dataDir, name, s.id, current, 'before-apply', this.#now())
-          } catch (e) {
-            return { server: s.id, ok: false, message: `the old config could not be saved first (${reason(e)}); nothing was changed` }
-          }
-          await this.#upstream.write(s, 'apps/config/set', {}, new URLSearchParams({ name, config: effective.text }))
+  historyList(name: string): HistoryEntry[] {
+    return listHistory(this.#config.dataDir, name)
+  }
 
-          const after = await this.#readConfig(s, name, true)
-          details[s.id] = `${textHash(current)} to ${textHash(effective.text)}, ${effective.text.length} bytes`
-          if (after === null || !sameConfig(after, effective.text)) {
-            return { server: s.id, ok: false, message: 'the server accepted the config but reads back a different one' }
-          }
-          return { server: s.id, ok: true, message: 'Applied and verified.' }
-        } catch (e) {
-          return { server: s.id, ok: false, message: reason(e) }
-        }
-      }),
-    )
-    return { results, details }
+  #entry(name: string, id: string): { entry: HistoryEntry; text: string } {
+    const entry = this.historyList(name).find((e) => e.id === id)
+    const text = entry === undefined ? null : readHistory(this.#config.dataDir, name, id)
+    if (entry === undefined || text === null) throw new UserError('There is no such history entry.')
+    return { entry, text }
+  }
+
+  historyGet(name: string, id: string): { entry: HistoryEntry; text: string } {
+    return this.#entry(name, id)
+  }
+
+  /* A saved version against a server's config now: what rolling back would change. */
+  async historyDiff(name: string, id: string, serverId: string): Promise<unknown> {
+    const { entry, text } = this.#entry(name, id)
+    const server = this.#server(serverId)
+    const l = await this.#live(server, name, true)
+    if (!l.ok || !l.installed) throw new UserError(l.ok ? `The app is not installed on ${serverId}.` : `${serverId}: ${l.error ?? 'did not answer'}`)
+    const current = l.text ?? ''
+    return {
+      entry,
+      server: serverId,
+      writable: server.adminToken !== undefined,
+      current,
+      currentHash: textHash(current),
+      version: text,
+      changes: diffConfigs(current, text, []),
+      unchanged: sameConfig(current, text),
+    }
+  }
+
+  async rollback(name: string, id: string, ids: string[], expect: Record<string, string>): Promise<{ results: TargetResult[]; details: Record<string, string> }> {
+    const { entry, text } = this.#entry(name, id)
+    if (entry.kind === 'before-save') throw new UserError('That entry is an old baseline. Restore it as the baseline instead.')
+    return this.#applyMany(name, ids, () => () => text, expect)
+  }
+
+  restoreBaseline(name: string, id: string, user: string): { detail: string } {
+    const { entry, text } = this.#entry(name, id)
+    if (entry.kind !== 'before-save') throw new UserError('Only an old baseline can be restored as the baseline.')
+    this.#setBaseline(name, text, this.getBaseline(name)?.importedFrom, user)
+    return { detail: `entry ${entry.hash}, ${text.length} bytes` }
+  }
+
+  /*
+  "Accept as baseline" for a drifted server: its config becomes the baseline, except that local
+  fields keep the baseline's values and the server's own values go into the overrides.
+  */
+  async acceptDrift(name: string, serverId: string, user: string): Promise<{ detail: string }> {
+    const server = this.#server(serverId)
+    let text: string | null
+    try {
+      text = await this.#readConfig(server, name, true)
+    } catch (e) {
+      throw new UserError(`${serverId}: ${reason(e)}`)
+    }
+    if (text === null || text.trim() === '') throw new UserError(`${serverId} has no config for "${name}".`)
+    const overrides = this.getOverrides(name)
+    let next
+    try {
+      next = acceptServerConfig(text, this.getBaseline(name)?.text ?? null, overrides, serverId)
+    } catch (e) {
+      throw new UserError(reason(e))
+    }
+    this.#setBaseline(name, next.baselineText, serverId, user)
+    if (overrides.paths.length > 0) writeOverrides(this.#config.dataDir, name, next.overrides)
+    return { detail: `from ${serverId}, ${next.baselineText.length} bytes, ${textHash(next.baselineText)}` }
+  }
+
+  /* ---- Export and import of all baselines ---- */
+
+  exportAll(): { hub: string; version: number; exportedAt: string; apps: { app: string; baseline: Baseline; overrides: Overrides }[] } {
+    const apps = listBaselineApps(this.#config.dataDir).flatMap((app) => {
+      const baseline = this.getBaseline(app)
+      return baseline === null ? [] : [{ app, baseline, overrides: this.getOverrides(app) }]
+    })
+    return { hub: 'technitium-hub', version: 1, exportedAt: new Date(this.#now()).toISOString(), apps }
+  }
+
+  /* Validates a file and says what it would change; with `apply` also writes baselines and overrides (never to a server). */
+  importAll(raw: string, apply: boolean, user: string): { apps: { app: string; status: 'new' | 'changed' | 'same'; ignoredServers: string[] }[]; applied: boolean } {
+    const parsed = parseJson(raw)
+    if (!parsed.ok) throw new UserError('The file is not valid JSON.')
+    const file = parsed.value as { hub?: unknown; version?: unknown; apps?: unknown }
+    if (file === null || typeof file !== 'object' || file.hub !== 'technitium-hub') throw new UserError('This is not a baselines file of the hub.')
+    if (file.version !== 1) throw new UserError('This file has a version the hub does not know.')
+    if (!Array.isArray(file.apps) || file.apps.length > 500) throw new UserError('The file has no list of apps, or has more than 500.')
+    const known = this.#config.servers.map((s) => s.id)
+    const plan: { app: string; text: string; overrides: Overrides; status: 'new' | 'changed' | 'same'; ignoredServers: string[] }[] = []
+    const seen = new Set<string>()
+    for (const item of file.apps as Record<string, unknown>[]) {
+      const app = item?.app
+      const baseline = item?.baseline as { text?: unknown } | null | undefined
+      if (typeof app !== 'string' || app === '' || app.length > 200 || seen.has(app)) throw new UserError('An app in the file has no valid, unique name.')
+      seen.add(app)
+      if (typeof baseline?.text !== 'string') throw new UserError(`"${app}" has no baseline text.`)
+      if (baseline.text.length > MAX_CONFIG) throw new UserError(`The baseline of "${app}" is larger than 1 MB.`)
+      if (looksLikeJson(baseline.text) && !parseJson(baseline.text).ok) throw new UserError(`The baseline of "${app}" is not valid JSON.`)
+      const rawOverrides = (item.overrides ?? { paths: [], values: {} }) as { paths?: unknown; values?: unknown }
+      let paths: string[]
+      try {
+        paths = validatePaths(rawOverrides.paths ?? [])
+      } catch (e) {
+        throw new UserError(`"${app}": ${reason(e)}`)
+      }
+      if (paths.length > 0 && !looksLikeJson(baseline.text)) throw new UserError(`"${app}": a plain text baseline cannot have local fields.`)
+      const values = rawOverrides.values !== null && typeof rawOverrides.values === 'object' && !Array.isArray(rawOverrides.values) ? (rawOverrides.values as Record<string, unknown>) : {}
+      const ignoredServers = Object.keys(values).filter((id) => !known.includes(id))
+      let cleaned: Overrides['values']
+      try {
+        cleaned = cleanValues(Object.fromEntries(Object.entries(values).filter(([id]) => known.includes(id))), paths, known)
+      } catch (e) {
+        throw new UserError(`"${app}": ${reason(e)}`)
+      }
+      const overrides: Overrides = { paths, values: cleaned }
+      const old = this.getBaseline(app)
+      const oldOverrides = this.getOverrides(app)
+      const status = old === null ? 'new' : old.text === baseline.text && JSON.stringify(oldOverrides) === JSON.stringify(overrides) ? 'same' : 'changed'
+      plan.push({ app, text: baseline.text, overrides, status, ignoredServers })
+    }
+    if (apply) {
+      for (const p of plan) {
+        if (p.status === 'same') continue
+        this.#setBaseline(p.app, p.text, this.getBaseline(p.app)?.importedFrom, user)
+        writeOverrides(this.#config.dataDir, p.app, p.overrides)
+      }
+    }
+    return { apps: plan.map(({ app, status, ignoredServers }) => ({ app, status, ignoredServers })), applied: apply }
   }
 }
-

hub/apps-history.test.ts

@@ -0,0 +1,439 @@
+// @vitest-environment node
+import http from 'node:http'
+import type { AddressInfo } from 'node:net'
+import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import path from 'node:path'
+import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'
+import { parseConfig } from './config.ts'
+import { createHub } from './server.ts'
+import { acceptServerConfig } from './apps-sync.ts'
+import { addUser, writeUsersFile } from './users.ts'
+
+const json = (o: unknown): string => JSON.stringify(o, null, 2)
+
+interface Fake {
+  server: http.Server
+  url: string
+  apps: Record<string, { version: string; config: string | null }>
+  calls: { path: string; token: string; method: string }[]
+  rejectSet: string | null
+}
+
+function fake(): Promise<Fake> {
+  return new Promise((resolve) => {
+    const f: Fake = { server: http.createServer(), url: '', apps: {}, calls: [], rejectSet: null }
+    f.server.on('request', (req, res) => {
+      const chunks: Buffer[] = []
+      req.on('data', (c: Buffer) => chunks.push(c))
+      req.on('end', () => {
+        const url = new URL(req.url ?? '/', 'http://fake')
+        const apiPath = url.pathname.replace(/^\/api\//, '')
+        const token = url.searchParams.get('token') ?? ''
+        const form = new URLSearchParams(Buffer.concat(chunks).toString('utf8'))
+        f.calls.push({ path: apiPath, token, method: req.method ?? '' })
+        const name = url.searchParams.get('name') ?? form.get('name') ?? ''
+        res.setHeader('content-type', 'application/json')
+        const ok = (response: unknown = {}) => res.end(JSON.stringify({ status: 'ok', response }))
+        const error = (m: string) => res.end(JSON.stringify({ status: 'error', errorMessage: m }))
+        if (apiPath === 'apps/list') return ok({ apps: Object.entries(f.apps).map(([n, a]) => ({ name: n, version: a.version })) })
+        if (apiPath === 'apps/config/get') return f.apps[name] ? ok({ config: f.apps[name].config }) : error('not found')
+        if (apiPath === 'apps/config/set') {
+          if (!token.startsWith('admin-')) return error('Access was denied.')
+          if (f.rejectSet !== null) return error(f.rejectSet)
+          f.apps[name].config = form.get('config')
+          return ok()
+        }
+        ok()
+      })
+    })
+    f.server.listen(0, '127.0.0.1', () => {
+      f.url = `http://127.0.0.1:${(f.server.address() as AddressInfo).port}`
+      resolve(f)
+    })
+  })
+}
+
+let a: Fake
+let b: Fake
+let dataDir: string
+let staticDir: string
+let hub: http.Server | undefined
+let base: string
+const tokens: Record<string, string> = {}
+
+const V1 = json({ ttl: 30, db: '/base.db' })
+const V2 = json({ ttl: 45, db: '/base.db' })
+
+function reset(): void {
+  for (const f of [a, b]) {
+    f.calls = []
+    f.rejectSet = null
+  }
+  a.apps = { Blocker: { version: '1.0', config: V1 } }
+  b.apps = { Blocker: { version: '1.0', config: V1 } }
+}
+
+async function start(dir: string = dataDir): Promise<void> {
+  hub?.close()
+  hub = createHub(
+    parseConfig({
+      HUB_SERVERS: `a=${a.url},b=${b.url}`,
+      A_TOKEN: 'read-a',
+      B_TOKEN: 'read-b',
+      A_ADMIN_TOKEN: 'admin-a',
+      B_ADMIN_TOKEN: 'admin-b',
+      HUB_DATA: dir,
+      HUB_STATIC: staticDir,
+      HUB_CACHE_SECONDS: '0',
+      HUB_TIMEOUT_MS: '2000',
+      HUB_DRIFT_MINUTES: '0',
+    }),
+  )
+  await new Promise<void>((resolve) => hub!.listen(0, '127.0.0.1', resolve))
+  base = `http://127.0.0.1:${(hub.address() as AddressInfo).port}`
+  for (const role of ['viewer', 'apps-admin', 'admin']) {
+    const res = await fetch(`${base}/api/user/login`, {
+      method: 'POST',
+      headers: { 'content-type': 'application/x-www-form-urlencoded' },
+      body: `user=${role}&pass=password-${role}`,
+    })
+    tokens[role] = ((await res.json()) as { token: string }).token
+  }
+}
+
+async function raw(role: string, apiPath: string, init: { params?: Record<string, string>; json?: unknown; form?: Record<string, string> } = {}) {
+  const headers: Record<string, string> = { authorization: `Bearer ${tokens[role]}` }
+  let body: string | undefined
+  if (init.json !== undefined) {
+    headers['content-type'] = 'application/json'
+    body = JSON.stringify(init.json)
+  } else if (init.form !== undefined) {
+    headers['content-type'] = 'application/x-www-form-urlencoded'
+    body = new URLSearchParams(init.form).toString()
+  }
+  return fetch(`${base}/api/${apiPath}?${new URLSearchParams(init.params ?? {})}`, { method: body === undefined ? 'GET' : 'POST', headers, body })
+}
+
+async function api(role: string, apiPath: string, init: Parameters<typeof raw>[2] = {}) {
+  const res = await raw(role, apiPath, init)
+  return { code: res.status, body: (await res.json()) as Record<string, any> }
+}
+
+const dirOf = (d: string = dataDir) => path.join(d, 'apps', 'Blocker')
+const historyDir = (d: string = dataDir) => path.join(dirOf(d), 'history')
+const writes = (f: Fake) => f.calls.filter((c) => c.path === 'apps/config/set')
+const list = async (role = 'viewer') => (await api(role, 'hub/apps/history/list', { params: { name: 'Blocker' } })).body.response.entries as Record<string, any>[]
+
+beforeAll(async () => {
+  a = await fake()
+  b = await fake()
+  dataDir = mkdtempSync(path.join(tmpdir(), 'hub-hist-data-'))
+  staticDir = mkdtempSync(path.join(tmpdir(), 'hub-hist-static-'))
+  let users = addUser([], 'viewer', 'viewer', 'password-viewer')
+  users = addUser(users, 'apps-admin', 'apps-admin', 'password-apps-admin')
+  users = addUser(users, 'admin', 'admin', 'password-admin')
+  writeUsersFile(dataDir, users)
+})
+
+afterAll(() => {
+  hub?.close()
+  a.server.close()
+  b.server.close()
+  rmSync(dataDir, { recursive: true, force: true })
+  rmSync(staticDir, { recursive: true, force: true })
+})
+
+beforeEach(async () => {
+  reset()
+  rmSync(path.join(dataDir, 'apps'), { recursive: true, force: true })
+  rmSync(path.join(dataDir, 'audit.log'), { force: true })
+  rmSync(path.join(dataDir, 'drift.json'), { force: true })
+  await start()
+})
+
+afterEach(() => {
+  hub?.close()
+})
+
+const post = (role: string, p: string, body: Record<string, unknown>) => api(role, p, { json: body })
+const importFrom = (server: string) => post('apps-admin', 'hub/apps/baseline/import', { name: 'Blocker', server })
+const saveBaseline = (text: string) => post('apps-admin', 'hub/apps/baseline/save', { name: 'Blocker', text })
+
+describe('history', () => {
+  it('keeps the old baseline text on every save and import, and lists it without the text', async () => {
+    await importFrom('a')
+    expect(await list()).toEqual([])
+    await saveBaseline(V2)
+    await importFrom('b')
+    const entries = await list()
+    expect(entries).toHaveLength(2)
+    expect(entries[0]).toMatchObject({ server: 'baseline', kind: 'before-save', size: V2.length })
+    expect(entries[0].at).toMatch(/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\.\d{3}Z$/)
+    expect(entries[0].hash).toMatch(/^[0-9a-f]{16}$/)
+    expect(entries[0].text).toBeUndefined()
+    expect(entries[1].size).toBe(V1.length)
+    expect(entries[0].id > entries[1].id).toBe(true)
+  })
+
+  it('does not add an entry when the baseline text did not change', async () => {
+    await importFrom('a')
+    await saveBaseline(V1)
+    expect(await list()).toEqual([])
+  })
+
+  it('reads one entry by the id the listing gives and refuses anything else', async () => {
+    await importFrom('a')
+    await saveBaseline(V2)
+    const [entry] = await list()
+    const got = await api('viewer', 'hub/apps/history/get', { params: { name: 'Blocker', id: entry.id } })
+    expect(got.body.response.text).toBe(V1)
+    for (const id of ['../../../users.json', '..\\..\\users.json', entry.id.replace('2', '3'), entry.id + '/', '/etc/passwd', '', `../history/${entry.id}`, 'baseline.json']) {
+      const r = await api('viewer', 'hub/apps/history/get', { params: { name: 'Blocker', id } })
+      expect([id, r.code]).toEqual([id, 400])
+    }
+    // The app name cannot walk out either.
+    const r = await api('viewer', 'hub/apps/history/get', { params: { name: '../..', id: 'users.json' } })
+    expect(r.code).toBe(400)
+  })
+
+  it('ignores files in the folder that are not history entries', async () => {
+    await importFrom('a')
+    mkdirSync(historyDir(), { recursive: true })
+    writeFileSync(path.join(historyDir(), 'notes.txt'), 'x')
+    expect(await list()).toEqual([])
+  })
+
+  it('shows what a saved version would change on a server', async () => {
+    await importFrom('a')
+    await saveBaseline(V2)
+    const [entry] = await list()
+    const r = await api('viewer', 'hub/apps/history/diff', { params: { name: 'Blocker', id: entry.id, server: 'b' } })
+    expect(r.body.response).toMatchObject({ server: 'b', writable: true, unchanged: true, current: V1 })
+    a.apps.Blocker.config = V2
+    const d = await api('viewer', 'hub/apps/history/diff', { params: { name: 'Blocker', id: entry.id, server: 'a' } })
+    expect(d.body.response.changes.changes.map((c: any) => [c.path, c.a, c.b])).toEqual([['ttl', 45, 30]])
+    expect(d.body.response.unchanged).toBe(false)
+  })
+
+  it('keeps at most 50 entries per app', async () => {
+    await importFrom('a')
+    for (let i = 0; i < 55; i++) await saveBaseline(json({ n: i }))
+    expect((await list()).length).toBe(50)
+  })
+
+  it('also records the config a single-server set replaced', async () => {
+    await api('apps-admin', 'apps/config/set', { params: { node: 'a' }, json: { name: 'Blocker', config: '{}' } })
+    const entries = await list()
+    expect(entries[0]).toMatchObject({ server: 'a', kind: 'before-set' })
+  })
+})
+
+describe('rollback and restore', () => {
+  async function twoVersions() {
+    // Both servers were changed by the hub once: their old config (V1) is in the history.
+    await importFrom('a')
+    await saveBaseline(V2)
+    await post('apps-admin', 'hub/apps/config/apply', { name: 'Blocker', targets: ['a', 'b'] })
+    return (await list()).filter((e) => e.kind === 'before-apply')
+  }
+
+  it('rolls a version back to the chosen servers, one result each, history first, verified', async () => {
+    const applied = await twoVersions()
+    expect(applied.map((e) => e.server).sort()).toEqual(['a', 'b'])
+    expect(a.apps.Blocker.config).toBe(V2)
+    const entry = applied.find((e) => e.server === 'a') as Record<string, any>
+    b.rejectSet = 'Disk full.'
+    const r = await post('apps-admin', 'hub/apps/history/rollback', { name: 'Blocker', id: entry.id, targets: ['a', 'b'] })
+    expect(r.body.response.results).toEqual([
+      { server: 'a', ok: true, message: 'Applied and verified.' },
+      { server: 'b', ok: false, message: 'Disk full.' },
+    ])
+    expect(a.apps.Blocker.config).toBe(V1)
+    expect(b.apps.Blocker.config).toBe(V2)
+    // The config rolled away from is kept too.
+    const files = readdirSync(historyDir()).filter((f) => f.endsWith('__a__before-apply.txt'))
+    expect(files).toHaveLength(2)
+    expect(writes(a).at(-1)?.token).toBe('admin-a')
+  })
+
+  it('refuses a rollback when the config changed since the preview, and for an old baseline entry', async () => {
+    const applied = await twoVersions()
+    const entry = applied[0]
+    const stale = await post('apps-admin', 'hub/apps/history/rollback', { name: 'Blocker', id: entry.id, targets: ['a'], expect: { a: 'deadbeefdeadbeef' } })
+    expect(stale.body.response.results[0].message).toMatch(/changed on the server since the preview/)
+    const baselineEntry = (await list()).find((e) => e.kind === 'before-save') as Record<string, any>
+    const refused = await post('apps-admin', 'hub/apps/history/rollback', { name: 'Blocker', id: baselineEntry.id, targets: ['a'] })
+    expect(refused.code).toBe(400)
+    expect(refused.body.errorMessage).toMatch(/Restore it as the baseline/)
+  })
+
+  it('restores an old baseline as the baseline, keeping the current one in the history', async () => {
+    await importFrom('a')
+    await saveBaseline(V2)
+    const old = (await list())[0]
+    const r = await post('apps-admin', 'hub/apps/baseline/restore', { name: 'Blocker', id: old.id })
+    expect(r.body.response.baseline.text).toBe(V1)
+    const entries = await list()
+    expect(entries[0]).toMatchObject({ kind: 'before-save', size: V2.length })
+    expect((await post('apps-admin', 'hub/apps/baseline/restore', { name: 'Blocker', id: 'nope' })).code).toBe(400)
+    // A server entry is not a baseline.
+    b.apps.Blocker.config = V2
+    await post('apps-admin', 'hub/apps/config/apply', { name: 'Blocker', targets: ['b'] })
+    const serverEntry = (await list()).find((e) => e.kind === 'before-apply') as Record<string, any>
+    expect((await post('apps-admin', 'hub/apps/baseline/restore', { name: 'Blocker', id: serverEntry.id })).code).toBe(400)
+  })
+
+  it('lets a viewer look but not roll back or restore', async () => {
+    await twoVersions()
+    const [entry] = await list()
+    expect((await post('viewer', 'hub/apps/history/rollback', { name: 'Blocker', id: entry.id, targets: ['a'] })).code).toBe(403)
+    expect((await post('viewer', 'hub/apps/baseline/restore', { name: 'Blocker', id: entry.id })).code).toBe(403)
+    expect((await api('viewer', 'hub/apps/history/get', { params: { name: 'Blocker', id: entry.id } })).code).toBe(200)
+  })
+
+  it('audits rollbacks and restores without any config text', async () => {
+    const applied = await twoVersions()
+    await post('apps-admin', 'hub/apps/history/rollback', { name: 'Blocker', id: applied[0].id, targets: ['a'] })
+    const text = readFileSync(path.join(dataDir, 'audit.log'), 'utf8')
+    expect(text).toContain('apps.history-rollback')
+    for (const secret of ['/base.db', 'admin-a', 'admin-b']) expect(text).not.toContain(secret)
+  })
+})
+
+describe('drift over the API', () => {
+  it('counts a drifted server in the session, shows it in drift/get and clears it when fixed', async () => {
+    await importFrom('a')
+    expect((await api('viewer', 'user/session/get')).body.info.hubDrift).toBe(0)
+    b.apps.Blocker.config = json({ ttl: 99, db: '/base.db' })
+
+    expect((await post('viewer', 'hub/drift/check', {})).code).toBe(403)
+    const checked = await post('apps-admin', 'hub/drift/check', {})
+    expect(checked.body.response.apps).toMatchObject([{ app: 'Blocker', server: 'b', kind: 'differs' }])
+    expect((await api('viewer', 'user/session/get')).body.info.hubDrift).toBe(1)
+    const got = await api('viewer', 'hub/drift/get')
+    expect(got.body.response.apps).toHaveLength(1)
+    expect(got.body.response.checking).toBe(false)
+
+    // Revert to the baseline is the sync apply on that server.
+    await post('apps-admin', 'hub/apps/config/apply', { name: 'Blocker', targets: ['b'] })
+    await post('apps-admin', 'hub/drift/check', {})
+    expect((await api('viewer', 'user/session/get')).body.info.hubDrift).toBe(0)
+    expect((await api('viewer', 'hub/drift/get')).body.response.apps).toEqual([])
+    expect(readFileSync(path.join(dataDir, 'drift.json'), 'utf8')).toContain('"apps": []')
+  })
+
+  it('accepts a drifted config as the baseline', async () => {
+    await importFrom('a')
+    b.apps.Blocker.config = json({ ttl: 99, db: '/base.db' })
+    const r = await post('apps-admin', 'hub/apps/drift/accept', { name: 'Blocker', server: 'b' })
+    expect(JSON.parse(r.body.response.baseline.text)).toEqual({ ttl: 99, db: '/base.db' })
+    expect(r.body.response.baseline.importedFrom).toBe('b')
+    expect((await list())[0]).toMatchObject({ server: 'baseline', kind: 'before-save' })
+    expect((await api('viewer', 'user/session/get')).body.info.hubDrift).toBe(1)
+    expect((await post('viewer', 'hub/apps/drift/accept', { name: 'Blocker', server: 'b' })).code).toBe(403)
+  })
+
+  it('on accept keeps local fields out of the baseline and moves the server values into the overrides', () => {
+    const out = acceptServerConfig(
+      json({ ttl: 99, db: '/b.db' }),
+      json({ ttl: 30, db: '/base.db' }),
+      { paths: ['db'], values: { a: { db: '/a.db' } } },
+      'b',
+    )
+    expect(JSON.parse(out.baselineText)).toEqual({ ttl: 99, db: '/base.db' })
+    expect(out.overrides.values).toEqual({ a: { db: '/a.db' }, b: { db: '/b.db' } })
+    expect(() => acceptServerConfig('plain', null, { paths: ['db'], values: {} }, 'b')).toThrow(/plain text/)
+  })
+})
+
+describe('export and import of baselines', () => {
+  async function setup() {
+    await importFrom('a')
+    await post('apps-admin', 'hub/apps/overrides/save', { name: 'Blocker', paths: ['db'], values: JSON.stringify({ a: { db: '/a.db' }, b: { db: '/b.db' } }) })
+  }
+
+  it('exports one file with every baseline and its local fields, as a download', async () => {
+    await setup()
+    const res = await raw('apps-admin', 'hub/apps/export')
+    expect(res.headers.get('content-disposition')).toMatch(/^attachment; filename="hub-baselines-\d{4}-\d\d-\d\d\.json"$/)
+    const file = (await res.json()) as Record<string, any>
+    expect(file).toMatchObject({ hub: 'technitium-hub', version: 1 })
+    expect(file.apps).toHaveLength(1)
+    expect(file.apps[0]).toMatchObject({ app: 'Blocker', baseline: { text: V1 }, overrides: { paths: ['db'], values: { a: { db: '/a.db' } } } })
+    expect((await api('viewer', 'hub/apps/export')).code).toBe(403)
+    expect(readFileSync(path.join(dataDir, 'audit.log'), 'utf8')).toContain('apps.export')
+  })
+
+  it('imports into another hub: a dry run says what is new, applying writes it, and no server is touched', async () => {
+    await setup()
+    const file = await (await raw('apps-admin', 'hub/apps/export')).text()
+    const other = mkdtempSync(path.join(tmpdir(), 'hub-hist-other-'))
+    writeUsersFile(other, [...(await import('./users.ts')).readUsersFile(dataDir)!])
+    try {
+      await start(other)
+      const before = a.calls.length + b.calls.length
+      const dry = await post('apps-admin', 'hub/apps/import', { data: file })
+      expect(dry.body.response).toEqual({ apps: [{ app: 'Blocker', status: 'new', ignoredServers: [] }], applied: false })
+      expect(readdirSync(other)).not.toContain('apps')
+      const done = await post('apps-admin', 'hub/apps/import', { data: file, apply: true })
+      expect(done.body.response.applied).toBe(true)
+      expect(JSON.parse(readFileSync(path.join(dirOf(other), 'baseline.json'), 'utf8')).text).toBe(V1)
+      expect(JSON.parse(readFileSync(path.join(dirOf(other), 'overrides.json'), 'utf8')).values.b).toEqual({ db: '/b.db' })
+      expect(a.calls.length + b.calls.length).toBe(before)
+      // The same file again changes nothing.
+      expect((await post('apps-admin', 'hub/apps/import', { data: file })).body.response.apps[0].status).toBe('same')
+    } finally {
+      rmSync(other, { recursive: true, force: true })
+    }
+  })
+
+  it('says changed, and keeps the replaced baseline in the history when applying', async () => {
+    await setup()
+    const file = JSON.parse(await (await raw('apps-admin', 'hub/apps/export')).text())
+    file.apps[0].baseline.text = V2
+    await saveBaseline('{"x":1}')
+    const dry = await post('apps-admin', 'hub/apps/import', { data: JSON.stringify(file) })
+    expect(dry.body.response.apps[0].status).toBe('changed')
+    await post('apps-admin', 'hub/apps/import', { data: JSON.stringify(file), apply: true })
+    expect(JSON.parse(readFileSync(path.join(dirOf(), 'baseline.json'), 'utf8')).text).toBe(V2)
+    expect((await list()).some((e) => e.kind === 'before-save' && e.size === '{"x":1}'.length)).toBe(true)
+  })
+
+  it('ignores local values of servers this hub does not have, and says so', async () => {
+    await setup()
+    const file = JSON.parse(await (await raw('apps-admin', 'hub/apps/export')).text())
+    file.apps[0].overrides.values.zzz = { db: '/z.db' }
+    const r = await post('apps-admin', 'hub/apps/import', { data: JSON.stringify(file) })
+    expect(r.body.response.apps[0].ignoredServers).toEqual(['zzz'])
+  })
+
+  it('refuses a file that is not valid, writing nothing', async () => {
+    await setup()
+    const good = JSON.parse(await (await raw('apps-admin', 'hub/apps/export')).text())
+    const cases: [string, unknown][] = [
+      ['not json', 'this is not json'],
+      ['another file', { hello: 1 }],
+      ['wrong version', { ...good, version: 2 }],
+      ['no apps', { ...good, apps: 'x' }],
+      ['bad json baseline', { ...good, apps: [{ app: 'X', baseline: { text: '{ nope' }, overrides: { paths: [], values: {} } }] }],
+      ['no text', { ...good, apps: [{ app: 'X', baseline: {}, overrides: {} }] }],
+      ['bad path', { ...good, apps: [{ app: 'X', baseline: { text: '{}' }, overrides: { paths: ['a b'], values: {} } }] }],
+      ['duplicate app', { ...good, apps: [good.apps[0], good.apps[0]] }],
+      ['text with local fields', { ...good, apps: [{ app: 'X', baseline: { text: 'plain' }, overrides: { paths: ['a'], values: {} } }] }],
+    ]
+    for (const [label, content] of cases) {
+      const r = await post('apps-admin', 'hub/apps/import', { data: typeof content === 'string' ? content : JSON.stringify(content), apply: true })
+      expect([label, r.code]).toEqual([label, 400])
+    }
+    expect(readdirSync(path.join(dataDir, 'apps'))).toEqual(['Blocker'])
+  })
+
+  it('is for writers only, and accepts a file up to 5 MB from a signed-in caller', async () => {
+    const big = JSON.stringify({ hub: 'technitium-hub', version: 1, apps: [{ app: 'Big', baseline: { text: JSON.stringify({ pad: 'x'.repeat(900_000) }) }, overrides: { paths: [], values: {} } }, ...[1, 2, 3].map((n) => ({ app: `Big${n}`, baseline: { text: JSON.stringify({ pad: 'y'.repeat(900_000) }) }, overrides: { paths: [], values: {} } }))] })
+    expect(big.length).toBeGreaterThan(2 * 1024 * 1024)
+    expect((await post('viewer', 'hub/apps/import', { data: big })).code).toBe(403)
+    const r = await post('apps-admin', 'hub/apps/import', { data: big })
+    expect(r.body.response.apps).toHaveLength(4)
+  })
+})

hub/apps-sync.test.ts

@@ -109,7 +109,7 @@ describe('comparisons that ignore local fields', () => {
     expect(d).toMatchObject({ changed: 3, truncated: false })
     expect((d as { lines: { op: string; text: string }[] }).lines).toEqual([
       { op: 'same', text: 'one' },
-      { op: 'del', text: 'two' },
+      { op: 'remove', text: 'two' },
       { op: 'add', text: '2' },
       { op: 'same', text: 'three' },
       { op: 'add', text: 'four' },

hub/apps-sync.ts

@@ -181,7 +181,7 @@ export interface FieldChange {
 }
 
 export interface LineOp {
-  op: 'same' | 'add' | 'del'
+  op: 'same' | 'add' | 'remove'
   text: string
 }
 
@@ -227,7 +227,7 @@ export function lineDiff(a: string, b: string): { lines: LineOp[]; truncated: bo
       j++
     } else if (j < y.length && (i === x.length || table[i * w + j + 1] > table[(i + 1) * w + j])) {
       lines.push({ op: 'add', text: y[j++] })
-    } else lines.push({ op: 'del', text: x[i++] })
+    } else lines.push({ op: 'remove', text: x[i++] })
   }
   return { lines, truncated: false, changed: lines.filter((l) => l.op !== 'same').length }
 }
@@ -257,6 +257,37 @@ export function importLocalValues(configs: Record<string, string | null>, paths:
   return out
 }
 
+/*
+"Accept as baseline": the baseline becomes this server's config, except that the local fields keep
+the baseline's old values, and this server's own values for them move into the overrides.
+*/
+export function acceptServerConfig(
+  serverText: string,
+  oldBaselineText: string | null,
+  overrides: Overrides,
+  server: string,
+): { baselineText: string; overrides: Overrides } {
+  if (overrides.paths.length === 0) return { baselineText: serverText, overrides }
+  const parsed = parseJson(serverText)
+  if (!parsed.ok || !looksLikeJson(serverText)) throw new Error('The config is plain text, so its local fields cannot be separated out.')
+  const value = clone(parsed.value)
+  const old = oldBaselineText === null ? null : parseJson(oldBaselineText)
+  const mine: Record<string, Json> = {}
+  for (const path of overrides.paths) {
+    const segments = parsePath(path)
+    if (segments === null) continue
+    const here = getAt(value, segments)
+    if (here.found) mine[path] = clone(here.value)
+    const before = old !== null && old.ok ? getAt(old.value, segments) : { found: false }
+    if (before.found) setAt(value, segments, clone(before.value))
+    else removeAt(value, segments)
+  }
+  return {
+    baselineText: JSON.stringify(value, null, 2),
+    overrides: { paths: overrides.paths, values: { ...overrides.values, [server]: { ...(overrides.values[server] ?? {}), ...mine } } },
+  }
+}
+
 /* Local values for paths that are listed, from servers that exist; anything else is dropped. */
 export function cleanValues(values: unknown, paths: string[], servers: string[]): Overrides['values'] {
   if (values === null || typeof values !== 'object' || Array.isArray(values)) throw new Error('values must be an object of servers.')

hub/apps-write.ts

@@ -77,7 +77,7 @@ export class AppsWriter {
   }
 
   /* Saves the app's current config next to the hub's data; throws if it cannot. */
-  async saveHistory(server: HubServer, name: string, kind: 'before-uninstall' | 'before-apply' = 'before-uninstall'): Promise<string> {
+  async saveHistory(server: HubServer, name: string, kind: 'before-uninstall' | 'before-set' = 'before-uninstall'): Promise<string> {
     if (server.adminToken === undefined) throw new Error('no admin token configured')
     this.#upstream.invalidate(server.id)
     const response = (await this.#upstream.call(server, 'apps/config/get', { name }, server.adminToken)) as { config?: unknown } | null
@@ -105,7 +105,7 @@ export class AppsWriter {
       case 'config-set': {
         if (config === undefined || config.length > MAX_CONFIG) throw new Error('the config is missing or larger than 1 MB')
         try {
-          await this.saveHistory(server, name, 'before-apply')
+          await this.saveHistory(server, name, 'before-set')
         } catch (e) {
           throw new Error(`the old config could not be saved first (${reason(e)}); nothing was changed`)
         }

hub/config.ts

@@ -29,6 +29,8 @@ export interface HubConfig {
   trustProxy: boolean
   port: number
   cacheSeconds: number
+  /** How often the drift check runs (HUB_DRIFT_MINUTES); 0 turns it off. */
+  driftMinutes: number
   timeoutMs: number
   /** Shown as `dnsServerDomain` (and `clusterDomain`) in the session. */
   domain: string
@@ -90,6 +92,7 @@ export function parseConfig(env: Env): HubConfig {
     trustProxy: ['1', 'true'].includes((env.HUB_TRUST_PROXY ?? '').trim().toLowerCase()),
     port: number(env, 'HUB_PORT', 8080, 0),
     cacheSeconds: number(env, 'HUB_CACHE_SECONDS', 15, 0),
+    driftMinutes: number(env, 'HUB_DRIFT_MINUTES', 15, 0),
     timeoutMs: number(env, 'HUB_TIMEOUT_MS', 8000, 1),
     domain: env.HUB_DOMAIN?.trim() || 'hub',
     staticDir: env.HUB_STATIC?.trim() || './dist',

hub/drift.test.ts

@@ -0,0 +1,196 @@
+// @vitest-environment node
+import { mkdtempSync, readFileSync, rmSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import path from 'node:path'
+import { afterEach, beforeEach, describe, expect, it } from 'vitest'
+import { parseConfig } from './config.ts'
+import { Drift } from './drift.ts'
+import { writeBaseline, writeOverrides } from './store.ts'
+import { Upstream } from './upstream.ts'
+
+const json = (o: unknown): string => JSON.stringify(o, null, 2)
+
+interface FakeServer {
+  apps: Record<string, string>
+  down: boolean
+  denyConfig: boolean
+}
+
+let dir: string
+let servers: Record<string, FakeServer>
+let calls: string[]
+let gate: Promise<void> | null
+let clock: number
+
+/* A fetch that answers from memory: no sockets, no timers. */
+const fakeFetch = (async (input: string | URL | Request) => {
+  const url = new URL(String(input))
+  const id = url.port === '1' ? 'a' : 'b'
+  const server = servers[id]
+  const apiPath = url.pathname.replace(/^\/api\//, '')
+  calls.push(`${id}:${apiPath}`)
+  if (gate !== null) await gate
+  if (server.down) throw new TypeError('fetch failed')
+  const name = url.searchParams.get('name') ?? ''
+  const reply = (response: unknown) => Response.json({ status: 'ok', response })
+  if (apiPath === 'apps/list') return reply({ apps: Object.keys(server.apps).map((n) => ({ name: n, version: '1.0' })) })
+  if (apiPath === 'apps/config/get') {
+    if (server.denyConfig) return Response.json({ status: 'error', errorMessage: 'Access was denied.' })
+    return reply({ config: server.apps[name] ?? null })
+  }
+  return reply({})
+}) as typeof fetch
+
+function make() {
+  const config = parseConfig({
+    HUB_SERVERS: 'a=http://127.0.0.1:1,b=http://127.0.0.1:2',
+    A_TOKEN: 'ra',
+    B_TOKEN: 'rb',
+    HUB_DATA: dir,
+    HUB_CACHE_SECONDS: '0',
+  })
+  const drift = new Drift(config, new Upstream(config, fakeFetch, () => clock), () => clock)
+  return { config, drift }
+}
+
+const baseline = (app: string, text: string) =>
+  writeBaseline(dir, { app, text, updatedAt: '2026-01-01T00:00:00Z', updatedBy: 'test' })
+
+beforeEach(() => {
+  dir = mkdtempSync(path.join(tmpdir(), 'hub-drift-'))
+  servers = { a: { apps: {}, down: false, denyConfig: false }, b: { apps: {}, down: false, denyConfig: false } }
+  calls = []
+  gate = null
+  clock = Date.parse('2026-10-31T10:00:00Z')
+})
+
+afterEach(() => rmSync(dir, { recursive: true, force: true }))
+
+describe('drift check', () => {
+  it('finds nothing when no app has a baseline, and calls no server', async () => {
+    servers.a.apps.X = json({ a: 1 })
+    const { drift } = make()
+    const state = await drift.check()
+    expect(state.apps).toEqual([])
+    expect(state.checkedAt).toBe('2026-10-31T10:00:00.000Z')
+    expect(calls).toEqual([])
+    expect(drift.count()).toBe(0)
+  })
+
+  it('reports a server that differs from the baseline and not one that matches', async () => {
+    baseline('Blocker', json({ ttl: 30 }))
+    servers.a.apps.Blocker = json({ ttl: 30 })
+    servers.b.apps.Blocker = json({ ttl: 99 })
+    const { drift } = make()
+    const state = await drift.check()
+    expect(state.apps).toEqual([{ app: 'Blocker', server: 'b', since: '2026-10-31T10:00:00.000Z', kind: 'differs' }])
+    expect(drift.count()).toBe(1)
+  })
+
+  it('compares by meaning, not by spacing or key order', async () => {
+    baseline('Blocker', json({ a: 1, b: 2 }))
+    servers.a.apps.Blocker = '{"b":2,"a":1}'
+    servers.b.apps.Blocker = json({ a: 1, b: 2 })
+    expect((await make().drift.check()).apps).toEqual([])
+  })
+
+  it('uses each server own local values, so a local difference is not drift', async () => {
+    baseline('Blocker', json({ ttl: 30, db: '/base' }))
+    writeOverrides(dir, 'Blocker', { paths: ['db'], values: { a: { db: '/a.db' }, b: { db: '/b.db' } } })
+    servers.a.apps.Blocker = json({ ttl: 30, db: '/a.db' })
+    servers.b.apps.Blocker = json({ ttl: 30, db: '/b.db' })
+    const { drift } = make()
+    expect((await drift.check()).apps).toEqual([])
+    // A wrong local value is drift.
+    servers.b.apps.Blocker = json({ ttl: 30, db: '/elsewhere.db' })
+    expect((await drift.check()).apps.map((e) => [e.server, e.kind])).toEqual([['b', 'differs']])
+  })
+
+  it('lists a server that does not answer as unreadable, and does not count it', async () => {
+    baseline('Blocker', json({ ttl: 30 }))
+    servers.a.apps.Blocker = json({ ttl: 30 })
+    servers.b.down = true
+    const { drift } = make()
+    const state = await drift.check()
+    expect(state.apps).toEqual([{ app: 'Blocker', server: 'b', since: '2026-10-31T10:00:00.000Z', kind: 'unreadable' }])
+    expect(drift.count()).toBe(0)
+  })
+
+  it('lists a refused config read as unreadable and an app that is missing as not-installed', async () => {
+    baseline('Blocker', json({ ttl: 30 }))
+    servers.a.apps.Blocker = json({ ttl: 30 })
+    servers.a.denyConfig = true
+    const { drift } = make()
+    const state = await drift.check()
+    expect(state.apps.map((e) => [e.server, e.kind])).toEqual([['a', 'unreadable'], ['b', 'not-installed']])
+  })
+
+  it('keeps `since` across checks and drops the entry once the problem is fixed', async () => {
+    baseline('Blocker', json({ ttl: 30 }))
+    servers.a.apps.Blocker = json({ ttl: 30 })
+    servers.b.apps.Blocker = json({ ttl: 99 })
+    const { drift } = make()
+    await drift.check()
+    clock += 15 * 60_000
+    const second = await drift.check()
+    expect(second.apps[0].since).toBe('2026-10-31T10:00:00.000Z')
+    expect(second.checkedAt).toBe('2026-10-31T10:15:00.000Z')
+
+    servers.b.apps.Blocker = json({ ttl: 30 })
+    clock += 15 * 60_000
+    expect((await drift.check()).apps).toEqual([])
+    expect(drift.count()).toBe(0)
+
+    // It drifts again: a new `since`.
+    servers.b.apps.Blocker = json({ ttl: 5 })
+    clock += 15 * 60_000
+    expect((await drift.check()).apps[0].since).toBe('2026-10-31T10:45:00.000Z')
+  })
+
+  it('changes the kind (and `since`) when the problem changes', async () => {
+    baseline('Blocker', json({ ttl: 30 }))
+    servers.a.apps.Blocker = json({ ttl: 30 })
+    servers.b.apps.Blocker = json({ ttl: 99 })
+    const { drift } = make()
+    await drift.check()
+    servers.b.down = true
+    clock += 60_000
+    const state = await drift.check()
+    expect(state.apps[0]).toMatchObject({ kind: 'unreadable', since: '2026-10-31T10:01:00.000Z' })
+  })
+
+  it('never runs two checks at once', async () => {
+    baseline('Blocker', json({ ttl: 30 }))
+    servers.a.apps.Blocker = json({ ttl: 30 })
+    servers.b.apps.Blocker = json({ ttl: 30 })
+    let open!: () => void
+    gate = new Promise<void>((resolve) => (open = resolve))
+    const { drift } = make()
+    const first = drift.check()
+    const second = drift.check()
+    expect(drift.get().checking).toBe(true)
+    open()
+    expect(await second).toBe(await first)
+    // Two servers, two app lists and two config reads: one check's worth of calls.
+    expect(calls).toHaveLength(4)
+    expect(drift.get().checking).toBe(false)
+    gate = null
+    await drift.check()
+    expect(calls).toHaveLength(8)
+  })
+
+  it('keeps the last result in drift.json and loads it again', async () => {
+    baseline('Blocker', json({ ttl: 30 }))
+    servers.a.apps.Blocker = json({ ttl: 30 })
+    servers.b.apps.Blocker = json({ ttl: 99 })
+    await make().drift.check()
+    const file = JSON.parse(readFileSync(path.join(dir, 'drift.json'), 'utf8'))
+    expect(file).toEqual({
+      checkedAt: '2026-10-31T10:00:00.000Z',
+      apps: [{ app: 'Blocker', server: 'b', since: '2026-10-31T10:00:00.000Z', kind: 'differs' }],
+    })
+    const again = make().drift
+    expect(again.get()).toMatchObject({ checkedAt: '2026-10-31T10:00:00.000Z', checking: false })
+    expect(again.count()).toBe(1)
+  })
+})

hub/drift.ts

@@ -0,0 +1,134 @@
+/*
+Drift: a server whose config is no longer what the hub says it should be (the baseline plus the
+server's own local values), usually because someone changed it in that server's own console.
+
+`check` reads every server's config for every app that has a baseline, with the read token, and
+compares it with the effective config as the sync does. The result is kept in memory and in
+`${HUB_DATA}/drift.json`; `since` is the first check that saw the same problem, and it is dropped
+once the problem is gone. Two checks never run at once. The timer lives in server.ts, so the
+tests drive `check` themselves.
+*/
+import { existsSync, readFileSync } from 'node:fs'
+import path from 'node:path'
+import type { HubConfig } from './config.ts'
+import { effectiveConfig, sameConfig } from './apps-sync.ts'
+import { readConfigText } from './apps-config.ts'
+import { listBaselineApps, readBaseline, readOverrides, writeAtomic } from './store.ts'
+import type { Upstream } from './upstream.ts'
+
+export type DriftKind = 'differs' | 'unreadable' | 'not-installed'
+
+export interface DriftEntry {
+  app: string
+  server: string
+  since: string
+  kind: DriftKind
+}
+
+export interface DriftState {
+  checkedAt: string | null
+  apps: DriftEntry[]
+}
+
+const KINDS: DriftKind[] = ['differs', 'unreadable', 'not-installed']
+
+export class Drift {
+  #config: HubConfig
+  #upstream: Upstream
+  #now: () => number
+  #state: DriftState
+  #running: Promise<DriftState> | null = null
+
+  constructor(config: HubConfig, upstream: Upstream, now: () => number = Date.now) {
+    this.#config = config
+    this.#upstream = upstream
+    this.#now = now
+    this.#state = this.#load()
+  }
+
+  #file(): string {
+    return path.join(this.#config.dataDir, 'drift.json')
+  }
+
+  #load(): DriftState {
+    try {
+      if (!existsSync(this.#file())) return { checkedAt: null, apps: [] }
+      const data = JSON.parse(readFileSync(this.#file(), 'utf8')) as Partial<DriftState>
+      const apps = (Array.isArray(data.apps) ? data.apps : []).filter(
+        (e): e is DriftEntry => typeof e?.app === 'string' && typeof e.server === 'string' && typeof e.since === 'string' && KINDS.includes(e.kind),
+      )
+      return { checkedAt: typeof data.checkedAt === 'string' ? data.checkedAt : null, apps }
+    } catch {
+      return { checkedAt: null, apps: [] }
+    }
+  }
+
+  get(): DriftState & { checking: boolean } {
+    return { ...this.#state, checking: this.#running !== null }
+  }
+
+  /** How many servers differ from their baseline (the other kinds are listed, not counted). */
+  count(): number {
+    return this.#state.apps.filter((e) => e.kind === 'differs').length
+  }
+
+  /* Runs a check, or joins the one that is already running. */
+  check(): Promise<DriftState> {
+    if (this.#running !== null) return this.#running
+    const run = this.#run().finally(() => {
+      this.#running = null
+    })
+    this.#running = run
+    return run
+  }
+
+  async #run(): Promise<DriftState> {
+    const apps = listBaselineApps(this.#config.dataDir).map((app) => ({
+      app,
+      baseline: readBaseline(this.#config.dataDir, app),
+      overrides: readOverrides(this.#config.dataDir, app),
+    }))
+    const found: { app: string; server: string; kind: DriftKind }[] = []
+
+    await Promise.all(
+      this.#config.servers.map(async (server) => {
+        if (apps.length === 0) return
+        this.#upstream.invalidate(server.id)
+        let installed: Set<string>
+        try {
+          const list = (await this.#upstream.call(server, 'apps/list')) as { apps?: { name: string }[] } | null
+          installed = new Set((list?.apps ?? []).map((a) => a.name))
+        } catch {
+          for (const a of apps) found.push({ app: a.app, server: server.id, kind: 'unreadable' })
+          return
+        }
+        await Promise.all(
+          apps.map(async (a) => {
+            if (a.baseline === null) return
+            if (!installed.has(a.app)) return void found.push({ app: a.app, server: server.id, kind: 'not-installed' })
+            try {
+              const text = (await readConfigText(this.#upstream, server, a.app)) ?? ''
+              const effective = effectiveConfig(a.baseline.text, a.overrides, server.id).text
+              if (!sameConfig(text, effective)) found.push({ app: a.app, server: server.id, kind: 'differs' })
+            } catch {
+              found.push({ app: a.app, server: server.id, kind: 'unreadable' })
+            }
+          }),
+        )
+      }),
+    )
+
+    const at = new Date(this.#now()).toISOString()
+    const before = new Map(this.#state.apps.map((e) => [`${e.app}\n${e.server}\n${e.kind}`, e.since]))
+    const entries = found
+      .map((f): DriftEntry => ({ ...f, since: before.get(`${f.app}\n${f.server}\n${f.kind}`) ?? at }))
+      .sort((x, y) => x.app.localeCompare(y.app) || x.server.localeCompare(y.server))
+    this.#state = { checkedAt: at, apps: entries }
+    try {
+      writeAtomic(this.#file(), JSON.stringify(this.#state, null, 2) + '\n')
+    } catch (e) {
+      console.error(`hub: drift.json could not be written (${e instanceof Error ? e.message : String(e)}).`)
+    }
+    return this.#state
+  }
+}

hub/routes.ts

@@ -49,6 +49,18 @@ export const ROUTES: Record<string, Route> = {
   'hub/apps/baseline/save': writeAll('apps.write'),
   'hub/apps/overrides/save': writeAll('apps.write'),
   'hub/apps/config/apply': writeAll('apps.write'),
+  // History, drift, and moving baselines between hubs (apps-config.ts, drift.ts).
+  'hub/apps/history/list': read(),
+  'hub/apps/history/get': read(),
+  'hub/apps/history/diff': read(),
+  'hub/apps/history/rollback': writeAll('apps.write'),
+  'hub/apps/baseline/restore': writeAll('apps.write'),
+  'hub/apps/drift/accept': writeAll('apps.write'),
+  'hub/drift/get': read(),
+  'hub/drift/check': writeAll('apps.write'),
+  // Configs can hold secrets, so the file of all baselines needs apps.write even to read.
+  'hub/apps/export': read('apps.write'),
+  'hub/apps/import': writeAll('apps.write'),
   'hub/apps/run': writeAll('apps.write'),
   'hub/apps/upload': writeAll('apps.write'),
 }

hub/server.ts

@@ -25,6 +25,7 @@ import { buildMatrix } from './apps-matrix.ts'
 import { AppsWriter, actionForPath, type FileAction, type TargetResult } from './apps-write.ts'
 import { TooLarge, parseMultipart, readLimited, type UploadedFile } from './multipart.ts'
 import { AppsConfig, UserError } from './apps-config.ts'
+import { Drift } from './drift.ts'
 import {
   mergeQueryLogs,
   mergeSettled,
@@ -41,6 +42,8 @@ type Json = Record<string, unknown>
 const MAX_BODY = 64 * 1024
 /* The config endpoints carry config texts (up to 1 MB). */
 const MAX_CONFIG_BODY = 2 * 1024 * 1024
+/* A file of all baselines. */
+const MAX_IMPORT_BODY = 5 * 1024 * 1024
 
 const TYPES: Record<string, string> = {
   '.html': 'text/html; charset=utf-8',
@@ -94,7 +97,7 @@ async function readForm(req: http.IncomingMessage, limit: number = MAX_BODY): Pr
     try {
       const data = JSON.parse(text) as Record<string, unknown>
       for (const [k, v] of Object.entries(data)) {
-        if (Array.isArray(v)) out.set(k, v.map(String).join(','))
+        if (Array.isArray(v)) out.set(k, v.some((x) => x !== null && typeof x === 'object') ? JSON.stringify(v) : v.map(String).join(','))
         else if (typeof v === 'string' || typeof v === 'number' || typeof v === 'boolean') out.set(k, String(v))
         else if (v !== null && typeof v === 'object') out.set(k, JSON.stringify(v))
       }
@@ -131,7 +134,8 @@ function clientIp(req: http.IncomingMessage, trustProxy: boolean): string {
 
 export function createHub(config: HubConfig, users: Users = Users.load(config.dataDir, config.bootstrap)): http.Server {
   const upstream = new Upstream(config)
-  const sessions = new Sessions(config, users)
+  const drift = new Drift(config, upstream)
+  const sessions = new Sessions(config, users, Date.now, () => drift.count())
   const limiter = new LoginLimiter()
   const audit = new Audit(config.dataDir)
   const writer = new AppsWriter(config, upstream)
@@ -267,12 +271,31 @@ export function createHub(config: HubConfig, users: Users = Users.load(config.da
   async function configRead(res: http.ServerResponse, apiPath: string, params: Params, who: Who, ip: string): Promise<void> {
     const name = params.name ?? ''
     try {
+      if (apiPath === 'hub/apps/export') {
+        const file = appsConfig.exportAll()
+        audit.log({ user: who.user, role: who.role, ip, action: 'apps.export', outcome: 'ok', detail: `${file.apps.length} app(s)` })
+        const text = JSON.stringify(file, null, 2) + '\n'
+        res.writeHead(200, {
+          'content-type': 'application/json; charset=utf-8',
+          'content-disposition': `attachment; filename="hub-baselines-${file.exportedAt.slice(0, 10)}.json"`,
+          'cache-control': 'no-store',
+          'content-length': Buffer.byteLength(text),
+        })
+        res.end(text)
+        return
+      }
       if (name === '') throw new UserError('An app name is required.')
       if (apiPath === 'hub/apps/config/compare') send(res, 200, ok(await appsConfig.compare(name)))
       else if (apiPath === 'hub/apps/config/preview') {
         const preview = await appsConfig.preview(name, csv(params.targets))
         audit.log({ user: who.user, role: who.role, ip, action: 'apps.config-preview', target: name, outcome: 'ok', detail: csv(params.targets).join(',') })
         send(res, 200, ok(preview))
+      } else if (apiPath === 'hub/apps/history/list') {
+        send(res, 200, ok({ entries: appsConfig.historyList(name) }))
+      } else if (apiPath === 'hub/apps/history/get') {
+        send(res, 200, ok(appsConfig.historyGet(name, params.id ?? '')))
+      } else if (apiPath === 'hub/apps/history/diff') {
+        send(res, 200, ok(await appsConfig.historyDiff(name, params.id ?? '', params.server ?? '')))
       } else if (apiPath === 'hub/apps/overrides/current') {
         send(res, 200, ok({ values: await appsConfig.currentLocalValues(name, csv(params.paths)) }))
       } else send(res, 403, fail('The hub is read-only.'))
@@ -284,9 +307,16 @@ export function createHub(config: HubConfig, users: Users = Users.load(config.da
   /* baseline import and save, local fields save, apply. */
   async function configWrite(res: http.ServerResponse, apiPath: string, params: Params, who: Who, ip: string): Promise<void> {
     const name = params.name ?? ''
-    const base = { user: who.user, role: who.role, ip, target: name }
+    const base = { user: who.user, role: who.role, ip, target: name === '' ? undefined : name }
     const action = apiPath.replace('hub/apps/', 'apps.').replace(/\//g, '-')
     try {
+      if (apiPath === 'hub/apps/import') {
+        const done = appsConfig.importAll(params.data ?? '', params.apply === 'true', who.user)
+        const count = (status: string) => done.apps.filter((a) => a.status === status).length
+        audit.log({ ...base, action: 'apps.import', outcome: 'ok', detail: `${done.applied ? 'applied' : 'dry run'}: ${count('new')} new, ${count('changed')} changed, ${count('same')} same` })
+        send(res, 200, ok(done))
+        return
+      }
       if (name === '' || name.length > 200) throw new UserError('An app name is required.')
       if (apiPath === 'hub/apps/baseline/import') {
         const done = await appsConfig.importBaseline(name, params.server ?? '', who.user)
@@ -296,6 +326,22 @@ export function createHub(config: HubConfig, users: Users = Users.load(config.da
         const done = appsConfig.saveBaseline(name, params.text ?? '', who.user)
         audit.log({ ...base, action, outcome: 'ok', detail: done.detail })
         send(res, 200, ok({ baseline: appsConfig.getBaseline(name) }))
+      } else if (apiPath === 'hub/apps/baseline/restore') {
+        const done = appsConfig.restoreBaseline(name, params.id ?? '', who.user)
+        audit.log({ ...base, action, outcome: 'ok', detail: done.detail })
+        send(res, 200, ok({ baseline: appsConfig.getBaseline(name) }))
+      } else if (apiPath === 'hub/apps/drift/accept') {
+        const done = await appsConfig.acceptDrift(name, params.server ?? '', who.user)
+        audit.log({ ...base, action: 'apps.drift-accept', server: params.server, outcome: 'ok', detail: done.detail })
+        await drift.check()
+        send(res, 200, ok({ baseline: appsConfig.getBaseline(name), overrides: appsConfig.getOverrides(name) }))
+      } else if (apiPath === 'hub/apps/history/rollback') {
+        const expect = jsonParam(params.expect, {}) as Record<string, string>
+        const done = await appsConfig.rollback(name, params.id ?? '', csv(params.targets), expect)
+        for (const r of done.results) {
+          audit.log({ ...base, action: 'apps.history-rollback', server: r.server, outcome: r.ok ? 'ok' : 'failed', detail: [r.message, done.details[r.server]].filter(Boolean).join(' | ') })
+        }
+        send(res, 200, ok({ results: done.results }))
       } else if (apiPath === 'hub/apps/overrides/save') {
         const done = appsConfig.saveOverrides(name, csv(params.paths), jsonParam(params.values, {}))
         audit.log({ ...base, action, outcome: 'ok', detail: done.detail })
@@ -323,7 +369,13 @@ export function createHub(config: HubConfig, users: Users = Users.load(config.da
         send(res, 405, fail('Use POST.'))
         return
       }
-      if (apiPath.startsWith('hub/apps/baseline/') || apiPath.startsWith('hub/apps/overrides/') || apiPath === 'hub/apps/config/apply') {
+      if (apiPath === 'hub/drift/check') {
+        const state = await drift.check()
+        audit.log({ user: who.user, role: who.role, ip, action: 'drift.check', outcome: 'ok', detail: `${state.apps.length} finding(s), ${drift.count()} differ` })
+        send(res, 200, ok(drift.get()))
+        return
+      }
+      if (apiPath.startsWith('hub/apps/baseline/') || apiPath.startsWith('hub/apps/overrides/') || apiPath.startsWith('hub/apps/history/') || apiPath.startsWith('hub/apps/drift/') || apiPath === 'hub/apps/import' || apiPath === 'hub/apps/config/apply') {
         await configWrite(res, apiPath, params, who, ip)
         return
       }
@@ -415,7 +467,7 @@ export function createHub(config: HubConfig, users: Users = Users.load(config.da
     // A config body may be large, but only for a signed-in caller (token in the header or the query).
     const large = (apiPath.startsWith('hub/apps/') || apiPath === 'apps/config/set')
       && sessions.touch(bearer(req) ?? url.searchParams.get('token')) !== null
-    const form = isMultipart(req) ? new URLSearchParams() : await readForm(req, large ? MAX_CONFIG_BODY : MAX_BODY)
+    const form = isMultipart(req) ? new URLSearchParams() : await readForm(req, !large ? MAX_BODY : apiPath === 'hub/apps/import' ? MAX_IMPORT_BODY : MAX_CONFIG_BODY)
     const params: Params = {}
     for (const [k, v] of url.searchParams) params[k] = v
     for (const [k, v] of form) params[k] = v
@@ -489,10 +541,14 @@ export function createHub(config: HubConfig, users: Users = Users.load(config.da
       await write(req, res, apiPath, route.kind, params, who, ip)
       return
     }
-    if (apiPath.startsWith('hub/apps/config/') || apiPath === 'hub/apps/overrides/current') {
+    if (apiPath.startsWith('hub/apps/config/') || apiPath.startsWith('hub/apps/history/') || apiPath === 'hub/apps/overrides/current' || apiPath === 'hub/apps/export') {
       await configRead(res, apiPath, params, who, ip)
       return
     }
+    if (apiPath === 'hub/drift/get') {
+      send(res, 200, ok(drift.get()))
+      return
+    }
     if (apiPath === 'apps/config/get' && !byId.has(params.node ?? '')) {
       send(res, 400, fail('Choose one server: a config is read from a single server.'))
       return
@@ -616,7 +672,17 @@ export function createHub(config: HubConfig, users: Users = Users.load(config.da
       }
     })()
   })
-  server.on('close', () => clearInterval(sweeper))
+  // The drift check: once shortly after start, then every HUB_DRIFT_MINUTES. Off when that is 0.
+  const timers: NodeJS.Timeout[] = []
+  if (config.driftMinutes > 0) {
+    const run = () => void drift.check().catch((e: unknown) => console.error(`hub: drift check failed (${e instanceof Error ? e.message : String(e)}).`))
+    timers.push(setTimeout(run, 5000), setInterval(run, config.driftMinutes * 60_000))
+    for (const t of timers) t.unref()
+  }
+  server.on('close', () => {
+    clearInterval(sweeper)
+    for (const t of timers) clearTimeout(t)
+  })
   return server
 }
 

hub/session.ts

@@ -38,7 +38,10 @@ export class Sessions {
   #started: string
   #seen = new Map<string, Session>()
 
-  constructor(config: HubConfig, users: Users, now: () => number = Date.now) {
+  #drift: () => number
+
+  constructor(config: HubConfig, users: Users, now: () => number = Date.now, drift: () => number = () => 0) {
+    this.#drift = drift
     this.#config = config
     this.#users = users
     this.#now = now
@@ -102,6 +105,8 @@ export class Sessions {
       hub: true,
       hubRole: role,
       /* The servers the hub holds an admin token for: the only ones a write can reach. */
+      /* How many servers differ from their baseline now: the console marks the header with it. */
+      hubDrift: this.#drift(),
       hubWritable: servers.filter((s) => s.adminToken !== undefined).map((s) => s.id),
     }
   }

hub/store.ts

@@ -8,7 +8,7 @@ What the hub keeps per app under `${HUB_DATA}/apps/<app>/`:
 Every write is a temp file and a rename, mode 0600: these files hold configs.
 */
 import { createHash } from 'node:crypto'
-import { existsSync, mkdirSync, readFileSync, readdirSync, renameSync, rmSync, writeFileSync } from 'node:fs'
+import { existsSync, mkdirSync, readFileSync, readdirSync, renameSync, rmSync, statSync, writeFileSync } from 'node:fs'
 import path from 'node:path'
 
 export const KEEP_HISTORY = 50
@@ -87,6 +87,71 @@ export function writeOverrides(dataDir: string, app: string, overrides: Override
   writeAtomic(path.join(appDir(dataDir, app), 'overrides.json'), JSON.stringify(overrides, null, 2) + '\n')
 }
 
+export type HistoryKind = 'before-uninstall' | 'before-apply' | 'before-set' | 'before-save'
+
+export interface HistoryEntry {
+  /** The file name; the only handle `readHistory` accepts. */
+  id: string
+  at: string
+  /** A server id, or `baseline` for an earlier text of the baseline. */
+  server: string
+  kind: string
+  size: number
+  hash: string
+}
+
+const HISTORY_FILE = /^(\d{4}-\d{2}-\d{2})T(\d{2})-(\d{2})-(\d{2}\.\d{3}Z)__(.+)__(before-[a-z-]+)\.txt$/
+
+const historyDir = (dataDir: string, app: string): string => path.join(appDir(dataDir, app), 'history')
+
+/* Newest first. Never the text. */
+export function listHistory(dataDir: string, app: string): HistoryEntry[] {
+  const dir = historyDir(dataDir, app)
+  if (!existsSync(dir)) return []
+  const entries: HistoryEntry[] = []
+  for (const name of readdirSync(dir)) {
+    const m = HISTORY_FILE.exec(name)
+    if (m === null) continue
+    try {
+      const text = readFileSync(path.join(dir, name), 'utf8')
+      entries.push({
+        id: name,
+        at: `${m[1]}T${m[2]}:${m[3]}:${m[4]}`,
+        server: m[5],
+        kind: m[6],
+        size: Buffer.byteLength(text),
+        hash: createHash('sha256').update(text).digest('hex').slice(0, 16),
+      })
+    } catch {
+      // A file that vanished while listing is skipped.
+    }
+  }
+  return entries.sort((a, b) => (a.id < b.id ? 1 : a.id > b.id ? -1 : 0))
+}
+
+/* The text of one entry. `id` has to be a name the listing returns, so nothing outside the folder can be read. */
+export function readHistory(dataDir: string, app: string, id: string): string | null {
+  const known = listHistory(dataDir, app).find((e) => e.id === id)
+  return known === undefined ? null : readFileSync(path.join(historyDir(dataDir, app), known.id), 'utf8')
+}
+
+/* The apps that have a baseline, by the name stored in the file. */
+export function listBaselineApps(dataDir: string): string[] {
+  const root = path.join(dataDir, 'apps')
+  if (!existsSync(root)) return []
+  const apps: string[] = []
+  for (const dir of readdirSync(root)) {
+    try {
+      if (!statSync(path.join(root, dir)).isDirectory()) continue
+      const data = readJson(path.join(root, dir, 'baseline.json')) as Partial<Baseline> | null
+      if (data !== null && typeof data.app === 'string' && typeof data.text === 'string') apps.push(data.app)
+    } catch {
+      // Not an app folder.
+    }
+  }
+  return apps.sort((a, b) => a.localeCompare(b))
+}
+
 /*
 Saves a server's config as it was before the hub changed it. The name starts with the time, so
 sorting the names sorts the files by age; the oldest beyond KEEP_HISTORY are deleted.
@@ -96,7 +161,7 @@ export function writeHistory(
   app: string,
   server: string,
   text: string,
-  kind: 'before-uninstall' | 'before-apply',
+  kind: HistoryKind,
   now: number,
 ): string {
   const dir = path.join(appDir(dataDir, app), 'history')