# Proofwarden — implementation spec v0.1 (testing & CI authority web)

**Sibling to** mgf-fed-web. Reuses its DNA: the `fed-resources/` federation directory, the `SourceProvider` seam (disk now, PyPI packages later), **generated-not-asserted + freshness-gated** data, the `live/` NDJSON pattern, committed **run-records**, read-mostly web, **honest-absence**, **refuse-don't-guess**. New here: a **run authority** (the warden broker + lease registry) and a new contract surface, `testctl`.

**Naming (proposed):** web = **Proofwarden**; per-project authority daemon = **warden** (the "test supervisor"); CLI = **testctl**. Confirm or override (`CONFIRM D-NAME`).

**Reference prototype:** `proofwarden_console_wireframe.html` — open it; behaviour is real (simulated warden, leases, streaming logs, the refused double-run made visible, history, CI watch-link), data is mock.

---

## 1. The run authority (the heart — single-flight, end-to-end)

The one rule that defines this product: **a given test can never be executing twice at once — regardless of how it was launched** (CLI, web, AI agent, CI). This is enforced **by construction**, not by convention.

### 1.1 The warden is the *sole executor*
Per project, a resident **warden** broker is the **only** thing that ever spawns a test process. The website, the CLI, the AI agent, and the CI pipeline **do not run tests** — they submit *run requests* to the warden, which decides what happens. (Same shape as mgf-fed-web's rule that the web never writes into a governed repo: here the web never executes a test directly.) If the warden is down, **no test runs** — the CLI refuses rather than executing on its own (**fail-closed**), because executing outside the warden would defeat the single entry point.

### 1.2 The lease registry is the single source of truth for "what's running"
The warden owns an atomic **lease registry**. A lease is the right to run, held for the duration of a run:
```
lease = { run_id, key, holder:{origin, who, pid|session}, acquired_at, heartbeat_at, ttl_s, test_id, config_digest }
```
- **`key`** is what's being single-flighted. Default **`key = project`** (one run at a time per project — see §1.4); a test declared `independent:true` uses **`key = test_id`** so it may run alongside others.
- The web and CLI **read** the registry (via `testctl status`) to show run-state; they **never** decide it. The registry — not a guessed UI state — is the truth.

### 1.3 Acquire decision (this is the enforcement)
When a run request arrives, the warden does **one atomic** check-and-act:
- **free** → grant lease, spawn the test, start streaming.
- **held for the *same* key by the *same* test** → **coalesce**: do **not** start a second run; attach the requester to the in-flight run as a watcher and return its `run_id`. (Directly kills the "agent kicks it while it's already running" case — and it's better than a flat reject.)
- **held for the same key by a *different* test** (serialized project) → **queue** the request; it starts when the lease frees.
- Every outcome — acquire / coalesce / queue / release / **reclaim** / **refused** — is an event in the run's log and run-record (§4), so *"did the authority actually prevent a collision"* is **checkable after the fact**. A correctly-refused second launch becomes visible proof the lock works.

### 1.4 Serialize-by-default (eliminates the flaky-collision class)
Default invariant: **one test run at a time per project; the rest queue.** This removes the whole class of two-runs-touching-the-same-resource (a board, a serial port, the filesystem, a fixed network port) that makes test suites flaky. A test the author *proves* independent sets `independent:true` in its catalog entry to opt into parallel (its own `key`). Parallelism is **opt-in, off by default** (`CONFIRM D-RUN-1` — default chosen: serialize).

### 1.5 Lease, not lock — crash safety
A bare lock file leaks on crash: the test dies, the lock never releases, everything blocks forever. A lease carries a **heartbeat + TTL**. The warden heartbeats while the test child is alive. If a lease's heartbeat ages past TTL (holder died), the warden **reclaims** it with an audited `lease_reclaimed` event — **never a silent steal**, always recorded. This is the answer to "other cases that may cause issues."

### 1.6 CI participates in the same lease
For single-flight to truly span **local + CI**, the Woodpecker pipeline acquires/releases **the same lease**: the pipeline's first step calls `testctl run … --from-ci`. A local run then sees a CI run of the same key and coalesces/queues, and vice-versa. This only *fully* enforces if the registry is reachable from CI:
- **Warden reachable from CI** (self-hosted on the same host/network — **default assumption**, `CONFIRM D-CI-1`) → real cross-environment enforcement.
- **CI remote / can't call back** → CI single-flight is **advisory**; the web still surfaces the conflict **loudly** (a CI run and a local run of the same key shown as a flagged collision) rather than pretending it's safe.

### 1.7 Authority form
**Resident daemon broker** (the default): the test only ever runs as the warden's child; fails closed if the warden is down. The alternative — a **no-daemon atomic-lock CLI** (acquire via `O_EXCL`/`flock`) — works with nothing resident but loses heartbeat, coalesce, and clean reclaim; documented as the fallback, not the default (`CONFIRM D-RUN-2` — default chosen: daemon, fail-closed).

---

## 2. The project contract — the `testctl` CLI

Sibling to `fedres`. A `testctl` binary (or make-targets, `CONFIRM D-NAME`). All commands deterministic where they can be, JSON out, nonzero on failure. **None of these execute a test directly — they talk to the warden.**

- **`testctl catalog [--refresh] --format json`** — emit/refresh the **test catalog** (§4.1): the tree of category → group → test plus each test's config + schema. Generated from the project's real runners (`zig build test --list`, `pytest --collect-only`, …); freshness-gated like any `generated/` file.
- **`testctl status --format json`** — the live **lease registry + queue + recent refusals** (§4.2): what's running, who holds it, heartbeat age, what's queued. Non-mutating. Warden down → `{ "broker":"down", … }` (honest, not an error-that-looks-like-nothing).
- **`testctl run <test-id> [--config k=v …] [--watch] [--from-ci] --format json`** — submit a run **request** to the warden. Returns one of `started` (new lease + run_id), `coalesced` (attached to an in-flight run_id — did **not** start a second), or `queued` (position). **Never spawns a test itself.**
- **`testctl watch <run-id|test-id> [--format json]`** — attach to a live run's log stream (the warden multiplexes one stream to many watchers). Non-mutating.
- **`testctl cancel <run-id>`** — request cancellation; the warden owns the child, stops it, releases the lease, records it.
- **`testctl ci start <pipeline> [--config …] --format json`** — trigger a Woodpecker pipeline; returns its **run URL** (the watch-link) + status. The pipeline itself uses `testctl run --from-ci` so it joins the lease (§1.6).
- **`testctl serve`** — run the warden (the resident authority). Usually auto-started/supervised; listed for completeness.
- **Freshness gate (project CI):** the catalog must satisfy `regenerate → identical`, exactly like `fed-resources` — so the test list the web shows is provably the real one, never a stale hand-maintained list.

---

## 3. Proofwarden ↔ project — the seam

Two seams, both already in the family:
- **`SourceProvider`** (read): `DiskProvider` now reads `generated/` (catalog, run-records) and tails `live/` (run logs) from the sibling tree; `PackageProvider` later for published packages. Identical to mgf-fed-web.
- **`WardenClient`** (authority): how the web reaches the warden to read `status` and submit `run`/`cancel`/`watch`. Local: a local socket / loopback HTTP to the resident warden (or shelling `testctl`). The web is **read-mostly**: it may submit run **requests** (the user's intent), but it never executes and never edits committed state. If the warden is unreachable → fail-closed, shown plainly.

Live logs are append-only NDJSON in `live/runs/<run-id>.ndjson` with a `live/CURRENT` pointer; the lease registry mirror persists to `live/leases.json` for read-visibility + crash-recovery (the **authority** is the warden process, the file is its journal — the web reads the warden, not the file, when the warden is up).

---

## 4. Data schemas (canonical JSON, `schema_version: "0.1"`)

### 4.1 `test_catalog.json` — what tests exist, and their configs (generated, gated)
```json
{ "schema_version":"0.1","kind":"test_catalog","project":"soothfast",
  "source_commit":"7160a43","generated_at":"2026-06-28T07:10:00Z",
  "runners":[{ "id":"zig","cmd":"zig build test","enumerate":"zig build test --list" }],
  "categories":[
    { "id":"unit","label":"Unit","groups":[
      { "id":"verify","label":"verify() invariants","tests":[
        { "id":"P-F3","label":"emit atomicity suite","runner":"zig","independent":true,
          "tags":["emit","atomicity"],"last_status":"pass","last_run_id":"run_20260628T0641_a1",
          "config":{ "seed":0,"iterations":1000 },
          "config_schema":[
            { "name":"seed","type":"int","default":0,"desc":"PRNG seed for the fuzz" },
            { "name":"iterations","type":"int","default":1000,"min":1,"max":100000,"desc":"fuzz iterations" } ] },
        { "id":"C12-1","label":"typed-Address byte-exactness","runner":"zig","independent":false,
          "tags":["address"],"last_status":"pass","config":{},"config_schema":[] } ]}
    ]},
    { "id":"cross","label":"Cross-arch (G1)","groups":[
      { "id":"g1","label":"byte-identity","tests":[
        { "id":"G1-riscv64","label":"riscv64 build identity","runner":"zig","independent":false,
          "tags":["cross","g1"],"last_status":"pass",
          "config":{ "arch":"riscv64" },
          "config_schema":[{ "name":"arch","type":"enum","default":"riscv64","options":["x86_64","aarch64","riscv64","s390x"],"desc":"target arch" }] }]}
    ]}
  ] }
```
`last_status ∈ {pass, fail, error, skipped, unknown}`. `config` = committed defaults (read-only); `config_schema` drives the editable form (§5). `independent` gates parallel (§1.4). Tree depth is fixed: **category → group → test**.

### 4.2 `testctl status` — the lease registry (live authority state, read-only to the web)
```json
{ "schema_version":"0.1","kind":"test_status","project":"soothfast","broker":"up","serialize":true,
  "running":[
    { "run_id":"run_20260628T1850_x9","key":"soothfast","test_id":"P-F3",
      "holder":{ "origin":"web","who":"founder" },"acquired_at":"2026-06-28T18:50:02Z",
      "heartbeat_at":"2026-06-28T18:50:18Z","ttl_s":30,"progress":0.45,"watchers":2 } ],
  "queued":[
    { "run_id":"run_20260628T1850_z3","test_id":"C12-1","origin":"agent","since":"2026-06-28T18:50:09Z","position":1 } ],
  "recent":[
    { "ts":"2026-06-28T18:50:07Z","event":"coalesced","test_id":"P-F3","origin":"agent","attached_to":"run_20260628T1850_x9",
      "note":"single-flight: already running; attached as watcher, did not start a second run" } ] }
```
`broker ∈ {up, down}` (down ⇒ fail-closed, shown plainly). `recent[]` is the rolling audit of refusals/coalesces/reclaims — the **visible proof** the authority works.

### 4.3 Run log — `live/runs/<run-id>.ndjson` (ephemeral, gitignored, never gated)
One JSON object per line; `seq` monotonic; the file *is* the run record so a dropped poll loses nothing.
```json
{"v":"0.1","run_id":"run_…_x9","seq":1,"ts":"…","kind":"run_request","test_id":"P-F3","origin":"web","requested_config":{"seed":7,"iterations":2000}}
{"v":"0.1","run_id":"run_…_x9","seq":2,"ts":"…","kind":"lease_acquired","key":"soothfast","holder":{"origin":"web","who":"founder"},"ttl_s":30}
{"v":"0.1","run_id":"run_…_x9","seq":3,"ts":"…","kind":"test_start","effective_config":{"seed":7,"iterations":2000}}
{"v":"0.1","run_id":"run_…_x9","seq":4,"ts":"…","kind":"log","stream":"stdout","msg":"fuzzing emit atomicity… 2000 iters"}
{"v":"0.1","run_id":"run_…_x9","seq":5,"ts":"…","kind":"progress","progress":0.6}
{"v":"0.1","run_id":"run_…_x9","seq":6,"ts":"…","kind":"assert","name":"out/*.json byte-identical","result":"pass"}
{"v":"0.1","run_id":"run_…_x9","seq":7,"ts":"…","kind":"test_end","result":"pass","duration_s":3}
{"v":"0.1","run_id":"run_…_x9","seq":8,"ts":"…","kind":"lease_released","key":"soothfast"}
```
`kind ∈ {run_request, lease_acquired, test_start, step, log, progress, assert, artifact, test_end, lease_released, coalesced, queued, lease_reclaimed, cancelled, error}`. A refused double-run is a `coalesced` (or `queued`) event on the *requesting* run id, pointing at the in-flight one. These event kinds are also the **notification triggers** for a future mobile app (§8) — keep them stable.

### 4.4 `test_run_record.json` — durable history (`generated/runs/`, gated-but-exempt-from-reproduce)
```json
{ "schema_version":"0.1","kind":"test_run_record","run_id":"run_…_x9","project":"soothfast","test_id":"P-F3",
  "result":"pass","started_at":"2026-06-28T18:50:02Z","duration_s":3,"origin":"web","who":"founder",
  "effective_config":{ "seed":7,"iterations":2000 },"config_digest":"sha256:…",
  "lease":{ "acquired_at":"…","released_at":"…","heartbeats":3,"reclaimed":false },
  "audit":[ {"ts":"…","event":"lease_acquired"},{"ts":"…","event":"coalesced","other_run":"run_…_z1","origin":"agent"},{"ts":"…","event":"lease_released"} ] }
```
This is the **verifiable** "what happened": every result is traceable to the **effective config** that produced it and the **full lease audit** (including any refused double-launch it absorbed). Like mgf-fed-web's run-records, these are **integrity-hashed in the manifest but exempt from the regenerate→identical assertion** (history can't be regenerated) — `CONFIRM D-RUN-4`, default chosen: exempt.

### 4.5 Config overlay (per-run, never a mutation)
Committed `config` defaults stay read-only. A user's edits in the web ride with the run **request** as an overlay (`requested_config`); the warden computes the **effective config** = defaults ⊕ overlay and records it. Defaults on disk are never changed by a run. The web shows configs **collapsed by default**, expandable to view, editable to override (§5).

### 4.6 CI
`testctl ci start` returns `{ "run_url":"https://woodpecker.<host>/repos/…/pipeline/123","pipeline":"emit-suite","status":"running" }`. The web stores recent CI runs (url + status) and renders a **Watch** button that deep-links into the Woodpecker host. If the pipeline calls back via `testctl run --from-ci`, its lease appears in §4.2 and joins single-flight; otherwise it shows as an advisory CI run (§1.6).

---

## 5. The web — layout & interaction

Sibling chrome to mgf-fed-web (dark instrument bar + rail, light printable readout, monospace for all run/lease identity). The **hero is the run authority**, not sync.

- **Top bar:** product mark · a global **runner pill** (`1 running · 2 queued` / `idle` / **`warden down`** in red) · `Open CI` link · `Print / Export`.
- **Left rail:** project list, each row showing a running/idle/▮queued indicator + a `● live` dot when a run is streaming; a legend pinning the status language.
- **Per-project views (the lenses of this tool):**
  1. **Catalog** (default) — the test tree (category → group → test), collapsible. Each test row: `last_status` chip · runner · tags · a **Run** button · an **expandable Config panel** (collapsed by default; expands to a form built from `config_schema` — view, then edit to override for the next run). Running a test = submitting a request to the warden; the row reflects `started` / `coalesced` / `queued` honestly.
  2. **Running** — the **lease registry** made visible: what's executing now (test, **holder/origin**, heartbeat age, progress, watcher count), the **queue**, and the **live log stream** for the selected run (NDJSON rendered like a console, auto-scroll, Pause). A prominent **authority audit** strip shows `coalesced` / `queued` / `lease_reclaimed` events — so a refused double-run is *seen*. Cancel here.
  3. **History** — past runs from run-records: test · result · duration · **origin** · **effective config** (expand) · the **lease audit** (expand). Filter by result/test/origin. This is the "what ran before, with all associated info" surface, fed from local storage.
  4. **CI** — start a pipeline (with optional config), the **Watch** deep-link into Woodpecker, and recent CI runs with status; any CI run that joined the lease is flagged where it appears in Running.
- **Honest-absence (mandatory):** warden down → "test authority unreachable — **no runs can start** (fail-closed)"; catalog stale → STALE banner + "the visible test list may be out of date — refresh"; a run whose stream stalls → "stream idle — last event 45s ago"; CI remote/un-joinable → "CI single-flight is advisory; conflicts are surfaced, not prevented"; unknown runner → the test row shows `unknown` status, never a fabricated pass.
- **Never confuse layers:** live log lines wear a **telemetry** tag (ephemeral), distinct from the committed run-record and from the lease registry (authority state). If the log tail breaks, the authority view stays truthful.

---

## 6. Implementation path (milestones with acceptance)

- **M0 — schemas + validator.** Publish the `0.1` schemas (§4) + a validator. *Done when:* it passes the example payloads and fails closed on a malformed catalog/record.
- **M1 — the warden + single-flight (the core).** The resident broker, the atomic lease registry, `testctl run/status/cancel`, serialize-by-default, coalesce, queue, heartbeat + reclaim — for one project (dogfood on soothfast). *Done when:* two concurrent `testctl run P-F3` (from different origins) result in **one** execution + one `coalesced` record; a different test queues; killing a holder mid-run yields a `lease_reclaimed` event and the queue advances; with the warden stopped, `testctl run` **refuses** (nonzero) and never spawns a test.
- **M2 — catalog + the web, read-only.** `testctl catalog` + the Catalog view (tree, status chips, expandable configs) + the Running view reading `testctl status`. *Done when:* the tree renders from real `generated/test_catalog.json`, configs expand, and Running reflects true lease state (honest-absence when the warden is down).
- **M3 — run + live logs from the web.** Run/cancel/Pause through `WardenClient`; the live NDJSON stream rendered in Running; the run-record written on completion. *Done when:* a run started from the web streams live, a second launch of the same test from the CLI **coalesces** and is shown, and the finished run appears in History.
- **M4 — history + config overlays.** History from run-records (filters, effective config, lease audit); per-run config editing that records the effective config. *Done when:* a run with an overridden config shows that exact config in its record, and History reconstructs a past run fully from local storage.
- **M5 — CI.** `testctl ci start` + the Woodpecker Watch deep-link + recent CI runs; the `--from-ci` lease join (or the advisory-conflict surfacing if remote). *Done when:* starting CI returns a working Watch link, and a CI run of a key that's also running locally is shown as joined (self-hosted) or flagged (remote).
- **M6 — package mode (pre-release).** `PackageProvider` for read; live/authority degrade to "local-dev only." *Done when:* a published package renders catalog + history read-only with the live/authority limits stated, no change to M2–M5 shape.

---

## 7. Non-negotiables & decisions

**Non-negotiables:** single entry point **by construction** (warden = sole executor); single-flight enforced via the atomic lease registry, never UI politeness; **fail-closed** when the authority is down; lease (heartbeat+TTL+audited reclaim), never a bare lock; every acquire/coalesce/queue/reclaim/**refusal** recorded as **visible proof**; catalog **generated-not-asserted + gated** (the shown test list is provably real); configs are a **per-run overlay**, committed defaults read-only, **effective config recorded**; live logs ephemeral/gitignored/**never gated** and **never** styled as provenance; the web is **read-mostly** (submits run requests, never executes, never edits state); honest-absence and refuse-don't-guess throughout; everything printable.

**Resolved (your "go with defaults"):** **D-RUN-1** serialize-by-default, parallel opt-in for `independent` tests · **D-RUN-2** resident daemon broker, fail-closed (lockfile CLI is the documented fallback) · **D-RUN-4** run-records integrity-hashed but exempt from the reproduce-gate · **D-CI-1** assume warden reachable from a self-hosted Woodpecker (real cross-env enforcement); remote ⇒ advisory + loud surfacing · conflict policy: **coalesce** same-test, **queue** different-test, never silently double-run.

**Confirm before/while building (smaller, fine for round 1 with Claude Code):**
- **D-NAME** — `Proofwarden` / `warden` / `testctl`, or descriptive (`mgf-test-web`, `mgf-test`); binary vs make-targets.
- **D-CI-2** — Woodpecker trigger mechanism: API token, CLI (`woodpecker-cli`), or a webhook; and whether CI status is polled (Woodpecker API) or pushed via a pipeline-emitted run-record.
- **D-TTL** — lease TTL + heartbeat interval (default 30s / 10s) and whether long tests send progress heartbeats to avoid false reclaim.
- **D-RUN-3** — queue policy details: FIFO (default), max depth, and whether the agent may request "run-or-attach" vs "run-or-queue" explicitly.
- **D-CFG** — config types beyond `int/enum/bool/string` (e.g. ranges, file paths), and whether overrides may be persisted as named presets (vs per-run only — default per-run only).
- **D-RUNNERS** — the per-project runner adapters (enumerate + run + parse result): `zig build test` (kernel), `pytest` (forge GUI), shell, mixed — confirm the set and how each reports pass/fail/skip.

---

## 8. Future directions (post-v1, recorded so they shape v1)

- **Notifications / mobile** (later): the §4.3 event kinds + lease events double as triggers — test failed, run reclaimed, **a refused double-run absorbed**, CI finished, queue cleared. A small relay watching `live/` + run-records fans out; the future mobile companion (one-project-per-request, like mgf-fed-web's) subscribes. Keep the event kinds stable now.
- **Parallel execution** (later): flip `independent:true` tests to real concurrency behind per-test leases once a project has proven isolation; the lease model already supports it (per-test `key`).
- **Flake tracking** (later): the run-record history already carries effective-config + result, so a flake-rate / quarantine lens is additive — no schema change.
- **Live push** (later): swap poll/tail for SSE/`watch` from the warden; the log file stays the record, push is latency only.
