# FEEDBACK — mgf-cloud-provision

> Paper-tracking file for issues, friction, and requests against
> this sibling. Consumers and supervisors file `PAPER-NN` entries here.
>
> **Where to file what:**
>
> - **mgf-cloud-provision-specific** (provisioning recipes, `ops/` self-heal
>   scripts, backup recipes, the CI-host runbook) → file HERE.
> - **Federation-wide** (cross-sibling patterns, standards proposals) →
>   file in
>   [mgf-standard/FEEDBACK.md](https://codeberg.org/magogi-admin/mgf-standard/src/branch/main/FEEDBACK.md).
>
> Lifecycle: `OPEN → SHIPPED / DEFERRED / WONTFIX / MOVED` (FB-01..04).

### PAPER-01 — `ops/agent-reconciler.py`: mint the Woodpecker JWT per-run (static token rots silently)

| Field | Value |
|---|---|
| Status | SHIPPED (in-repo 2026-07-10; live deployment pending) |
| Submitter | mgf-standard (PAPER-111 investigation) on 2026-07-10 |
| Submitted | 2026-07-10 |
| Severity | 🟡 Sharp edge (proven 17-day silent outage of the CI safety net) |
| Federation paper | relayed by the supervisor from `mgf-standard/pending-feedback/mgf-cloud-provision.md` (FB-05) |

- **Severity:** 🟡 Sharp edge (proven 17-day silent outage of the CI safety net)
- **Filed by:** mgf-standard (PAPER-111 investigation) on 2026-07-10
- **What hurts:** the deployed reconciler reads a **static** JWT from
  `/opt/woodpecker/reconciler.token`. Woodpecker user JWTs are HS256-signed
  with the user's DB `hash` as the key — any hash rotation invalidates every
  previously-minted token. Observed: the token 401'd from 2026-06-23 to
  2026-07-10 (~1,950 `api error` runs), so the fleet's wedge-recovery net was
  down for ~2.5 weeks and the 2026-07-01 wedge (mgf-standard PAPER-111)
  outlived it. Repaired 2026-07-10 by re-minting on-host — but the next hash
  rotation re-creates the outage.
- **Why it matters:** the reconciler is the recovery path for the single most
  critical shared resource (CI capacity), and its failure mode is silent.
- **Evidence:** `/var/log/woodpecker-reconciler.log` (`api error` from
  2026-06-23T20:36Z; recoveries 2026-06-21/22 before it); fresh on-host mint →
  `GET /api/queue/info` 200; mint math = `mgf` workspace
  `tools/woodpecker_api.sh` (proven byte-identical to Woodpecker's own by
  `rotate_woodpecker_token.sh --check`).
- **Suggested shape:** the reconciler runs as root on the server host — have it
  mint per-run exactly like `woodpecker_api.sh` (read `id|hash` from the
  `woodpecker-postgres` container, HS256-sign the user JWT; ~10 lines, stdlib
  `hmac`/`hashlib`) and drop `reconciler.token` entirely. Fall back to the file
  only if the DB is unreachable.
- **Resolution:** (2026-07-10, queue sweep) Implemented exactly as suggested, in
  the canonical home. `ops/agent-reconciler.py` now mints per-run:
  `mint_user_jwt()` (pure, stdlib `hmac`/`hashlib`/`base64`/`json`;
  ops/agent-reconciler.py:76), `fetch_id_hash()` (`sudo docker exec … psql
  -tAc "select id||chr(124)||hash …"`; :96), `get_token()` (mint first,
  `reconciler.token` file **only** when the DB is unreachable; :112); every run
  logs which path it used (`ok token=db-mint` / `token=file-fallback (…)`).
  Mint math verified byte-identical to the proven `mgf`
  `tools/woodpecker_api.sh` reference on identical inputs (executed locally:
  "mint byte-identical to woodpecker_api.sh math: OK"). Unit-tested without
  docker (tests/unit/mgf/cloud_provision/test_ops_reconciler.py — mint shape +
  signature, hash-containing-`|` split, fallback chain, both-paths-fail);
  gates green: `ruff check` clean, `mypy src` clean, `pytest -q` 65 passed.
  **Deployment to woodpecker-prod is pending** (sweep scope is code-only; the
  supervisor deploys — copy `ops/agent-reconciler.py` over
  `/opt/woodpecker/agent-reconciler.py`; `reconciler.token` may stay as the
  documented DB-down fallback, runbook §4/§6.3 updated).
- **DEPLOYED 2026-07-11 (founder-authorized, VERIFIED_BY_EXECUTION):** installed on `woodpecker-prod` (`/opt/woodpecker/agent-reconciler.py`; prior file kept as `.bak-2026-07-11`). Unattended cron runs now log `ok token=db-mint pending=0` — the JWT is minted **per-run from the Woodpecker Postgres**, so a user-hash rotation can no longer strand it. **Zero `api error` lines today** (against 1,951 lifetime, all from the 06-23 → 07-10 silent outage). The agent container was NOT restarted by the run (uptime unchanged) — the detection bounds + 2-hit guard hold. The static token file survives ONLY as the DB-unreachable fallback, and every run logs which path it took.
- **Status:** SHIPPED + DEPLOYED (2026-07-11)

### PAPER-02 — self-heal crons POST success-heartbeats to OO; watcher pages on staleness (PAPER-111 signal)

| Field | Value |
|---|---|
| Status | SHIPPED (in-repo 2026-07-10; live deployment pending) |
| Submitter | mgf-standard (PAPER-111 investigation) on 2026-07-10 |
| Submitted | 2026-07-10 |
| Severity | 🟡 Sharp edge |
| Federation paper | relayed by the supervisor from `mgf-standard/pending-feedback/mgf-cloud-provision.md` (FB-05) |

- **Severity:** 🟡 Sharp edge
- **Filed by:** mgf-standard (PAPER-111 investigation) on 2026-07-10
- **What hurts:** healthy reconciler runs are **silent by design** (its log
  cannot distinguish "healthy" from "cron dead"), `self-heal.sh` logs only
  locally, and the estate watcher (agent2) probes uptime only — so a dead
  safety net pages nobody (proven: the 17-day 401 outage above). MGF_INFRA §5
  already names the mechanism ("the self-heal signals as they migrate" to the
  `backup_heartbeats` stream); it just isn't implemented.
- **Suggested shape:** (a) both crons POST a success-only heartbeat record
  (`job=agent-reconciler` / `job=self-heal`) to the OO `backup_heartbeats`
  stream via the existing `/etc/mgf/backup-heartbeat.{url,token}` root-600
  files; (b) `estate-watch.sh` gains the freshness check MGF_INFRA §5 already
  plans (stale > N → page), with N ≈ 30 min for the self-heal jobs vs 26 h for
  backups; (c) optionally the watcher also probes `GET /api/queue/info`
  directly once an admin credential exists on agent2 — until then the
  heartbeat-staleness route needs no new secret distribution.
- **Resolution:** (2026-07-10, queue sweep) (a) + (b) implemented in the
  canonical home; (c) deliberately not taken (the staleness route needs no new
  secret, as the paper itself prefers). Shipped:
  - **Emitters (a):** `ops/agent-reconciler.py` POSTs `job=agent-reconciler`
    at the end of every successful `main()` (fail-open `post_heartbeat()`,
    :145, called :237); `ops/self-heal.sh` POSTs `job=self-heal` as its last
    line (:28/:78); the agent2 cloud-init self-heal mirror gets the same
    (examples/woodpecker-agent-cloud-init.yml:65 + the byte-pinned packaged
    copy in src/mgf/cloud_provision/data/). All use the existing
    `/etc/mgf/backup-heartbeat.{url,token}` root-600 files (backups/*
    contract) and are fail-open — a heartbeat failure can never break a heal.
    Proven against a local OO stub: valid `[{"job":"self-heal","status":"ok",
    "host":…,"ts":…}]` record received; missing-cred and POST-failure paths
    log the warning and return success.
  - **Watcher (b):** `estate-watch.sh` is NOT templated in this repo (it lives
    on agent2, `/opt/watcher/estate-watch.sh` — MGF_INFRA §5), so the
    freshness check ships as the new `ops/estate-watch-heartbeats.sh`:
    queries OO `_search` (endpoint derived from the ingest URL — no new
    secret), windows `agent-reconciler:1800 self-heal:1800` (N≈30 min) and
    `{pg-dump,devpi-export,oo-snapshot}:93600` (26 h); exit 1 stale, exit 2
    check-couldn't-run (page on both). Deployment + wiring steps are in the
    script header. Exercised against a local OO stub: exit 0 all-fresh,
    exit 1 with `STALE: job=agent-reconciler …` when that job stops
    heartbeating, exit 2 on missing cred files.
  - Tests: tests/unit/mgf/cloud_provision/test_ops_reconciler.py (13 new;
    `pytest -q` 65 passed); runbook §4/§6.4 updated.
  **Live deployment pending** (sweep is code-only): push the two updated ops
  scripts to their hosts, push `/etc/mgf/backup-heartbeat.{url,token}` to
  agent2, and wire `estate-watch-heartbeats.sh` into the live watcher per the
  script header.
- **DEPLOYED 2026-07-11 (founder-authorized, VERIFIED_BY_EXECUTION) — write side AND read side:**
  - **Credential (the real blocker — it never existed):** `/etc/mgf/backup-heartbeat.{url,token}` were **absent on every host**; the OO heartbeat that `MGF_INFRA.md` §5 describes had never been wired. Minted a **least-privilege OpenObserve service account** (`heartbeat@magogi.org`, org `default`) — root creds deliberately NOT reused — and installed the endpoint + `Basic` header as **root-600** files on `woodpecker-prod` and `woodpecker-agent2` (values generated on-host and piped host-to-host; never printed).
  - **Write side:** `agent-reconciler.py` (prod) + `self-heal.sh` (BOTH hosts) POST success-only heartbeats. Confirmed **landed in OO** by querying the stream back: `job=agent-reconciler host=woodpecker-prod`, `job=self-heal host=woodpecker-prod`, `job=self-heal host=woodpecker-agent2`.
  - **Read side:** `ops/estate-watch-heartbeats.sh` installed on **agent2** (`/opt/watcher/`, 0750) and **wired into the live `*/10` `estate-watch.sh`** through its existing deduped Codeberg-issue pager. Proven **both ways**: a never-heartbeating job → `STALE` + exit 1 (would page); the real scoped set → `heartbeats OK` + exit 0 (silent). The live watcher run is green and filed no issue.
  - **Scope taken deliberately:** the pager's `CHECKS` covers the **self-heal layer only** (`agent-reconciler:1800 self-heal:1800`). The nightly **backup** jobs are excluded because they do **not** heartbeat at all — see **PAPER-03**; including them would page on day one for jobs that are in fact running.
  - **Deploy gotcha found + fixed:** agent2's `*/5` cron ran `self-heal.sh` with no `WOODPECKER_STACK_DIR`, and the shared script defaults to `/opt/woodpecker` — which does not exist on agent2. Deploying as-is would have `cd`-failed every run and **silently disabled agent2's self-heal**. The cron now sets `WOODPECKER_STACK_DIR=/opt/woodpecker-agent` (the script's own deploy note); verified by running it exactly as cron does (exit 0, agent untouched). Rollback kept: `*.bak-2026-07-11` + `/tmp/cron.bak-2026-07-11`.
- **Status:** SHIPPED + DEPLOYED (2026-07-11; self-heal layer — backup-job heartbeats tracked as PAPER-03)

### PAPER-03 — The nightly backups do NOT heartbeat: `MGF_INFRA.md` §5 describes a dead-man's-switch that was never wired

- **Severity:** 🔴 Blocker (the backup safety net is unobservable — a silently-dead backup looks identical to a healthy one)
- **Filed by:** mgf-cloud-provision (PAPER-01/02 deployment) on 2026-07-11
- **What hurts:** `MGF_INFRA.md` §5 states that "**every** backup script POSTs a heartbeat record only on success" to the OO `backup_heartbeats` stream, with endpoint + auth in `/etc/mgf/backup-heartbeat.{url,token}`. On the live hosts, **neither was true**: those credential files did not exist anywhere, and the deployed `pg-dump-nightly.sh` only appends `HEARTBEAT pg-backup OK` to a **local log file** — nothing is ever sent to OpenObserve. The same is expected of `devpi-export` and `oo-snapshot` (not yet inspected).
- **Why it matters:** the whole point of the dead-man's-switch is that a backup which stops running, or starts failing silently, **stops heartbeating** and pages. As deployed, a dead backup cron is indistinguishable from a healthy one — exactly the failure mode that let the reconciler rot for 17 days (PAPER-01). A local log nobody reads is not a safety net. This is also a documentation-vs-reality drift (AG-10): the infra map asserts a mechanism the running system does not have.
- **Evidence:** `/etc/mgf/` did not exist on `woodpecker-prod`, `woodpecker-agent2`, or `openobserve-prod` before 2026-07-11; `sudo grep -n heartbeat /opt/backups/pg-dump-nightly.sh` → only `echo "[$TS] HEARTBEAT pg-backup OK …" >> "$LOG"` (a local log line, no `curl`). The OO `backup_heartbeats` stream contained **zero** `pg-dump` / `devpi-export` / `oo-snapshot` records; the freshness checker reports all three `STALE` while the newly-wired self-heal jobs report fresh.
- **Suggested shape:** (a) add the same fail-open `post_heartbeat` helper (now proven in `ops/self-heal.sh`) to `backups/pg-dump-nightly.sh`, `backups/devpi-export.sh`, `backups/oo-snapshot.sh` in **this repo**, and re-deploy them (the credential now exists on the hosts — the OO service account minted 2026-07-11); (b) then extend the watcher's `CHECKS` to `pg-dump:93600 devpi-export:93600 oo-snapshot:93600` so a missed nightly backup pages; (c) correct `MGF_INFRA.md` §5 to describe what is actually deployed until (a) lands.
- **DEPLOYED 2026-07-11 (founder-authorized) — and the investigation found something far worse than a missing heartbeat (PAPER-04 🔴🔴).** Deliberate deviation from (a): the repo's `backups/*.sh` were **NOT** deployed wholesale — they diverge from the LIVE scripts in **BACKUP_DIR** (`/var/backups` vs the deployed `/opt/backups`) and dump method (`docker compose exec` vs `docker exec`), so swapping them in would have **silently relocated every backup**. Instead the live scripts took a **minimal, additive, fail-open `post_heartbeat`** patch on all three hosts (`.bak-2026-07-11` kept); the repo↔host divergence is filed as **PAPER-05**. **Verified by execution:** `pg-dump` run for real → fresh 25 MB dump + heartbeat **confirmed landed in OO**; `oo-snapshot` run for real → 38 MB snapshot, OO back up in 21 s, heartbeat landed. Credential distributed root-600 to the devpi + OO hosts. Watcher `CHECKS` extended to `pg-dump:93600 oo-snapshot:93600` (live, green) — `devpi-export` joins after its first heartbeat.
- **Status:** SHIPPED + DEPLOYED (2026-07-11 — pg-dump + oo-snapshot heartbeating and watched; devpi-export armed)

### PAPER-04 — 🔴🔴 devpi had NO backup schedule AT ALL: the package index has been unbacked-up since 2026-07-04

- **Severity:** 🔴🔴 Blocker (disaster-recovery hole on the store holding every published wheel)
- **Filed by:** Court/CTO (PAPER-03 deployment) on 2026-07-11
- **What hurts:** `devpi-prod` (`195.15.203.250`) — the index holding **every published `mgf-*` wheel and the fleet's entire release history** — had **no backup cron installed anywhere**: root crontab empty, every user crontab empty, nothing in `/etc/cron.d`, no systemd timer. The `devpi-export-nightly.sh` script was present and works — **nothing ever ran it**. The only export on disk was dated **2026-07-04**, taken by hand. `MGF_INFRA.md` §5 asserts this job runs **nightly at 02:17, 7-day retention**. That was never true.
- **Why it matters:** losing the devpi host would have cost every release artifact back to 2026-07-04. The stated objective is **RPO 24 h**; the real RPO was **7 days and growing**. And it was **silent** — nothing watched it, nothing paged, and the infra map asserted it was fine. This is exactly the failure shape the dead-man's-switch exists to catch, and precisely why PAPER-03 matters: with heartbeats, the watcher would have paged on day one.
- **Evidence:** `for u in $(cut -d: -f1 /etc/passwd); do sudo crontab -l -u "$u"; done` → empty for every user; `/etc/cron.d` → only `e2scrub_all` + `sysstat`; `systemctl list-timers` → only `dpkg-db-backup`. `sudo find /opt/backups -name 'devpi-export-*.tar.gz'` → exactly ONE file, `devpi-export-20260704-1242.tar.gz` (53 MB), 7 days old. (Contrast: the `pg-dump` and `oo-snapshot` crons WERE installed and running daily.)
- **Resolution:** **FIXED 2026-07-11** — cron installed (`17 2 * * * /opt/backups/devpi-export-nightly.sh`, matching §5's claimed schedule) and the script now heartbeats, so a future failure **pages** instead of hiding. A fresh manual export + first heartbeat follow immediately (deferred only to avoid stopping devpi while CI was installing from it).
- **Lesson (bigger than the fix):** **a documented schedule is not a running schedule.** Verify the crontab, not the doc.
- **Status:** SHIPPED (cron installed + heartbeat armed; fresh export pending)

### PAPER-05 — The committed `backups/*.sh` recipes are NOT what runs on the hosts (paths + dump method diverge)

- **Severity:** 🟡 Sharp edge (with a security question attached)
- **Filed by:** Court/CTO (PAPER-03 deployment) on 2026-07-11
- **What hurts:** this repo's `backups/*.sh` are billed as "the nightly recipes as code" — but they are **not what is deployed**. Repo `pg-dump-nightly.sh` writes `BACKUP_DIR=/var/backups/woodpecker-pg` and dumps via `docker compose exec -T woodpecker-postgres`; the **live** script writes `/opt/backups/` and dumps via `docker exec woodpecker-woodpecker-postgres-1`. Same class of divergence on the other two. Deploying the repo versions as-is would have **silently relocated every backup** and orphaned the existing dumps from the retention sweep — which is why the 2026-07-11 rollout patched the LIVE scripts additively instead.
- **Why it matters:** "recipes as code" that nobody deploys is fiction — it invites exactly the confident-but-wrong action this deployment nearly took, and it means the tested, byte-pinned artifacts in this repo are **not** the ones protecting our data. It also hides a **security question**: the deployed dumps land in `/opt/backups` while `MGF_INFRA.md` §5 says backups live in `/var/backups/…` **root-only** — and the Woodpecker pg dump contains **all CI state plus the org secrets**.
- **Evidence:** md5(deployed) ≠ md5(repo) for all three scripts; `sudo grep -c post_heartbeat <deployed>` was **0** on every host before this session, while the repo versions have carried the heartbeat since `f63c608`.
- **Suggested shape:** reconcile deliberately — **founder call on the storage location**, because it is a security decision, not a cosmetic one: (a) adopt the repo recipes on the hosts with `BACKUP_DIR` overridden to today's `/opt/backups` (behaviour preserved, code canonical); or (b) migrate to root-only `/var/backups` as §5 already claims (the better posture for dumps containing org secrets), updating the restore runbook; or (c) fold the deployed reality back into the repo. Whichever wins, add a `--check` comparing deployed vs committed so this cannot silently drift again.
- **Resolution (2026-07-11, founder-directed):** the founder picked a variant of (a)/(c) with a twist the investigation forced — **the deployed scripts turned out to be the *older, less-safe* versions** (no `set -euo pipefail`; count-based not age-based retention; devpi restarted via a `fail()` function, not a `trap`, so an unexpected exit mid-export could leave devpi **stopped**; the OO one hardcoded an `https://195…sslip.io` healthcheck the repo's own hygiene test forbids — folding them in verbatim would have imported the weaker scripts and broken 7 CI tests). So instead of blessing the deployed reality, we reconciled to a **best-of** (the repo's safety design — pipefail, trap-restart, age-based `-mtime` retention, no committed endpoints — plus the hosts' genuinely-good additions: `umask 077`, `gunzip -t` integrity, a localhost post-backup serving check, and the shared `/opt/backups/backup.log` trail) and **DEPLOYED it to all three hosts**. Security posture re-verified first-hand: `/opt/backups` is `700 root:root`, dumps `600 root:root`, none world-readable. **Each script manually run-verified on its host:** pg **27 MB** dump (root-600, gzip-valid) · oo **39 MB** snapshot (OO cold-stopped → **restarted + serving**) · devpi **63 MB** export (devpi stopped → **restarted, HTTP 200 serving**). Now **repo == deployed == fragment** (byte-identical): the three graduated from `verify-deployed.sh` `KNOWN_DIVERGED` into `TRACKED` (**IN SYNC**), the `examples/*-backup-cloud-init-fragment.yml` re-embeds are byte-pinned by `test_backup_fragments.py`, and `MGF_INFRA.md` §5's stale `/var/backups` claim was corrected to `/opt/backups`. The `--check` the paper asked for already exists and now covers these three. "Recipes as code" is true — and the hosts are safer than before.
- **Status:** SHIPPED (2026-07-11 — best-of reconciled + deployed + byte-tracked; hosts strictly safer)

### PAPER-08 — Restoring the DATA does not restore the SERVICE: every host's config + credentials are in no backup

- **Severity:** 🔴 Blocker (RTO blocker — the restore drill proved the data comes back; this is what still stops the service coming back)
- **Filed by:** Court/CTO (the 2026-07-11 restore drill) on 2026-07-11
- **What hurts:** the three backup jobs capture **data only** — the Postgres dump, the devpi export, the OO data volume. The **host-level configuration and credentials that a service needs in order to START** are captured by nothing:
  | File | Host | Without it, after a host loss… |
  |---|---|---|
  | `/opt/openobserve/CREDENTIALS` | openobserve-prod | you restore the volume but **cannot log in** — a restored OO carries its ORIGINAL user set, and `ZO_ROOT_USER_*` only seeds a *fresh* volume (proven in the drill) |
  | `/opt/woodpecker/.env` | woodpecker-prod | no `WOODPECKER_AGENT_SECRET` → **agents cannot connect**; no OAuth client secret → **nobody can log in**; no `DEVPI_URL` server var → every generated pipeline fails its guard |
  | `/opt/logship/collector.env` | openobserve-prod | log ingestion is dead until the token is re-minted and every consumer updated |
  | `/opt/watcher/.codeberg-token` | agent2 | **the pager goes silent** — failures stop reaching you |
  | `/opt/watcher/backup-pull-key` | agent2 | **the off-host mirror stops pulling** — i.e. losing agent2 also breaks the mechanism that protects everything else |
- **Why it matters:** "we can restore" was proven today for the data. But an RTO of 4 h assumes you can bring the *service* back — and today that requires hand-reconstructing an agent secret, a re-registered OAuth app, a re-minted collector token, and an OO root password that **no longer exists anywhere**. Worst of all, the mirror's own pull key is in the same boat: the one artifact protecting every store is protected by nothing.
- **Evidence:** the 2026-07-11 drill (runbook §5) — all three stores restored, but the OO scratch instance could not be authenticated against because the restored volume carries the original users and the root credential lives only in `/opt/openobserve/CREDENTIALS`, which no backup touches. Confirmed by inspection: `oo-snapshot` tars only `/var/lib/docker/volumes/openobserve_openobserve-data/_data`; `pg-dump` dumps only the DB; `devpi-export` exports only the server data. All five files above exist on their hosts and appear in **no** artifact.
- **Suggested shape:** add a **`config-snapshot`** step per host — a root-0600 tar of that host's `/opt/<svc>/{.env,CREDENTIALS,docker-compose.yml}` (+ agent2's watcher key/token) — dropped into `/opt/backups/` so the existing off-host mirror carries it automatically. Blast radius is **not** materially widened: the Postgres dump the mirror already holds contains **all 17 org secrets** (proven in the drill), so agent2 is already trusted with the crown jewels. **Founder call worth making explicitly:** keep an **offline** copy of the three bootstrap credentials that cannot be recovered from any restore (OO root password · `WOODPECKER_AGENT_SECRET` · the mirror's pull key) — a password manager, not a host.
- **Resolution:** **SHIPPED + DEPLOYED 2026-07-11.** `backups/config-snapshot-nightly.sh` runs on all four hosts (cron 02:05, ahead of the 03:00 mirror pull), producing a root-0600 tar (born under `umask 077`) of that host's config + credentials; the off-host mirror pulls all three `cfg-*` sources (per-source size floor — a config tar is legitimately ~1-2 KB, and the flat 10 KB floor red-flagged it, caught by *running* it). Verified: the off-host snapshot for woodpecker-prod contains `/opt/woodpecker/.env` (the agent secret) and for openobserve-prod contains `/opt/openobserve/CREDENTIALS` (the root password) — i.e. the two credentials whose absence blocked the drill's service recovery are now off-host. `config-snapshot` heartbeats and is watched.
- **Status:** SHIPPED + DEPLOYED — **one founder call remains:** keep an **OFFLINE** copy of the three bootstrap credentials that no restore can regenerate (OO root · `WOODPECKER_AGENT_SECRET` · the mirror's own pull key). A backup you can only read by restoring the thing it restores is a circular dependency; agent2 holding everything is a concentration, not a second copy.
### PAPER-09 — The self-heal cron KILLS HEALTHY PIPELINES: `created && !running` is exactly what a starting pipeline looks like

- **Severity:** 🔴 Blocker (the safety net was itself a leading cause of CI flakiness)
- **Filed by:** Court/CTO on 2026-07-11 — caught red-handed while diagnosing a "flaky" CI failure that I had in fact just caused
- **What hurts:** `self-heal.sh`'s wedge test was `created >= 1 && running == 0` — **with no notion of how long** a container had been in `Created`. That is **precisely the state of a healthy pipeline during its startup window**: the step containers exist in `Created` and none has flipped to `Running` yet. When the `*/5` cron landed in that window it did the worst possible thing — **`docker rm -f`'d the very containers that were about to run, then restarted the agent** — killing a perfectly good pipeline. The build comes back as **"Canceled" with every step failed at once**, which reads like an infrastructure gremlin and gets shrugged off as "CI is flaky."
- **Why it matters:** this is not a missed rescue, it is **damage done by the rescuer**. It fired **11 times** (log: `grep -c WEDGE`), including **twice five minutes apart** on 2026-07-11 (`00:00:02`, `00:05:01` — consecutive cron ticks). It is a strong candidate for a real share of the fleet's unexplained CI failures, and plausibly for the "exit 126 / same-commit contention" family already papered against the templates (mgf-standard PAPER-109/114) — those were treated as forge/agent quirks, but a cron that force-removes starting containers produces exactly that signature. Worse, it *teaches people to distrust red*, which is how a genuine break slips through.
- **Evidence:** I reproduced it by accident and then confirmed it: while diagnosing mgf-fed pipeline #255, my manual `self-heal` run on agent2 logged `21:57:52 WEDGE: 2 stuck Created, 0 running -> restarting agent` — and #255, which was **running at that moment**, came back `Canceled` with all 14 steps failed. The trigger is `/opt/woodpecker-agent/self-heal.sh:69` (`if [ "$created" -ge 1 ] && [ "$running" -eq 0 ]`), followed by `docker rm -f` of every `Created` `wp_*` container and `docker compose restart woodpecker-agent`. The script's own comment claimed *"a healthy pipeline's containers pass through Created only transiently, so Created alone is not a wedge"* — the intent was right; **the code never implemented it.**
- **Resolution:** **SHIPPED + DEPLOYED 2026-07-11.** The wedge test now requires the `Created` containers to be **STALE**: each candidate's age is read from `docker inspect .Created`, and only those older than `STUCK_AGE_S` (default **180 s**) count. A genuinely wedged container sits in `Created` indefinitely; a starting one flips to `Running` in seconds — so **wedge recovery is preserved and the race disappears**. Added `DRY_RUN=1` so the destructive path can be fire-tested without restarting an agent. **Fire-tested both ways on the live host:** a *young* `wp_*` container → `stuck=0` (correctly seen as a starting pipeline, NOT a wedge); the **same** container with `STUCK_AGE_S=0` → `stuck=1`, wedge detected (recovery still works — and the fact that the second test still *found* the container proves the first test did not simply reap it). Deployed to both hosts; old scripts kept as `.bak2-2026-07-11`.
- **Lesson:** an automated remediation is a **privileged, destructive actor**. It needs the same burden of proof we demand of a human before `rm -f` — "I observed a bad state" is not enough; it must be "I observed a bad state **that has persisted**." Every self-healing action should be judged on what it does to a *healthy* system, not only on whether it rescues a broken one.
- **Status:** SHIPPED + DEPLOYED

### PAPER-10 — The reconciler carried the SAME blind spot as the self-heal (a starting agent reads as idle)

- **Severity:** 🟡 Sharp edge (narrower than PAPER-09 — the 2-hit guard hid it — but the same destructive class)
- **Filed by:** Court/CTO on 2026-07-11 — found by applying PAPER-09's lesson to the *other* healer instead of declaring victory
- **What hurts:** `agent-reconciler.py`'s idle test used `docker ps --filter name=wp_` — **running containers only**. So an agent that is *starting* a pipeline (containers created, image still pulling) counts as **idle**. Combined with a job pending, that satisfies the wedge signature `schedulable + idle while N pending (orphaned slots)` — and a startup slower than the 2-hit guard (~3 min: a large image pull is easily that) would have got a **perfectly healthy agent force-recreated**, killing the very pipeline it was busy starting.
- **Why it matters:** it is PAPER-09's bug in the other healer. PAPER-09 was caught only because its window is wide (the `*/5` cron vs a seconds-long startup); this one needs a slow pull, so it would have stayed rare, unreproducible, and blamed on "flaky CI" indefinitely — the worst kind of bug. **The 2-hit guard made it rarer; it did not make it correct.**
- **Evidence (fire-tested on the live host):** with one `wp_*` container in `Created` and none Running — i.e. exactly a starting pipeline — the **old** test (`docker ps`) returns **0** → reads IDLE → would force-recreate the agent; the **new** test (`docker ps -a`) returns **1** → reads BUSY → agent left alone. A genuinely wedged agent has **no** `wp_*` containers at all (0 under either test), so the true wedge is still detected.
- **Resolution:** **SHIPPED + DEPLOYED 2026-07-11** — `build_containers()` now counts containers in **any state** (`docker ps -a`). A wedged agent has none; a starting one has them in `Created`. Counting every state is what actually tells those two apart. Old script kept as `.bak2-2026-07-11`.
- **Lesson (the generalisation of PAPER-09):** when a healer decides whether a system is *broken*, it must not confuse **"not finished yet"** with **"never going to finish."** Both healers made that error in different words — one on containers, one on workers. **After fixing a destructive automation, go and check its siblings for the same misreading; do not stop at the instance you happened to catch.**
- **Status:** SHIPPED + DEPLOYED
