# MGF estate — backup & restore runbook

Wave-2 P0 (survival). Before this, the estate could die quietly: devpi (every
install, every release), the Woodpecker Postgres (all CI state + the org
secrets) and OpenObserve had **no backup at all**, and nothing paged. This
runbook + `backups/` are the codified answer; `mgf-standard/MGF_INFRA.md`
§Backups & paging is the fleet-level index.

## 0. Objectives (stated, not aspirational)

| | |
|---|---|
| **RPO** | **24 h** — nightly backups; at most one day of state is lost. |
| **RTO** | **4 h** — any single store restored to service within four hours, per the drilled procedures below. |
| Scope | devpi state · Woodpecker Postgres · OpenObserve volume. VMs themselves are cattle (re-provisioned from this repo's specs). |

## 1. The three nightly jobs

Recipes as code in [`backups/`](../backups/) — parameterized via env (defaults
match the live estate), each ends by POSTing a heartbeat. Install per host via
the matching `examples/*-backup-cloud-init-fragment.yml` (fragments are
byte-pinned to the scripts by unit tests).

| Store | Host | Script | Schedule (UTC) | Retention | Heartbeat `job` |
|---|---|---|---|---|---|
| devpi (`/opt/devpi/server`) | `195.15.203.250` | `backups/devpi-export-nightly.sh` — stop → `devpi-export` → start → tar.gz → prune → optional Swift upload | 02:17 | 7 d local (+ object storage if `SWIFT_CONTAINER` set) | `devpi-export` |
| Woodpecker Postgres | `195.15.241.48` | `backups/pg-dump-nightly.sh` — `docker compose exec pg_dump \| gzip` → prune | 02:41 | 14 d | `pg-dump` |
| OpenObserve volume | `195.15.203.114` | `backups/oo-snapshot-nightly.sh` — tar the data volume (live; `STOP_DURING_SNAPSHOT=1` for cold) → prune | 03:03 | 7 d | `oo-snapshot` |

Backups land in `/var/backups/{devpi,woodpecker-pg,openobserve}/` (root-only).
Logs: `/var/log/mgf-backup.log` on each host.

### Heartbeats — the dead-man's-switch

- Each script POSTs one JSON record **only on success** to the endpoint named in
  `/etc/mgf/backup-heartbeat.url`, authenticated with the header value in
  `/etc/mgf/backup-heartbeat.token` (both root-600, pushed by the operator —
  **values are never committed**; the committed artifacts carry only the file
  *paths*). Target: the OpenObserve `backup_heartbeats` stream (`/api/<org>/backup_heartbeats/_json`,
  scoped service-account auth).
- Record shape: `{"job","status":"ok","host","artifact","bytes","ts"}`.
- **The watcher** (Court-deployed, runs on `woodpecker-agent2` — deliberately a
  *different* host from the stores it watches) checks heartbeat freshness +
  uptime probes (devpi `/+simple/`, Woodpecker `/healthz`, OO ingest) and
  **files a Codeberg issue** (the pager until real paging integrates) when a
  heartbeat is stale >26 h or a probe fails. A failed backup, a dead cron, a
  wedged host and an unconfigured heartbeat all look the same — a missed
  heartbeat — and all page.

### Deploy by hand (what the Court does today, exact paths)

```bash
# per host: script + cron (or merge the fragment into the host's cloud-init)
scp backups/devpi-export-nightly.sh   ubuntu@195.15.203.250:/tmp/ # → sudo install -m 0750 /tmp/… /usr/local/bin/
scp backups/pg-dump-nightly.sh        ubuntu@195.15.241.48:/tmp/
scp backups/oo-snapshot-nightly.sh    ubuntu@195.15.203.114:/tmp/
# cron.d lines: see the matching examples/*-backup-cloud-init-fragment.yml
# heartbeat config (values from my_stuff, never committed):
#   /etc/mgf/backup-heartbeat.url    — the OO ingest endpoint
#   /etc/mgf/backup-heartbeat.token  — the Authorization header value (scoped SA)
```

## 2. Restore — devpi (the fleet's sole package artery)

1. **VM** (if gone): `mgf-cloud-provision apply examples/devpi.toml --provider
   infomaniak --clouds-yaml …/PCP-HOESFJV-clouds.yaml --cloud PCP-HOESFJV-dc3-a
   --yes` — cloud-init converges devpi + Caddy (fresh, empty index).
2. Copy the newest `devpi-export-*.tar.gz` to the host, then:
   ```bash
   sudo systemctl stop devpi
   sudo rm -rf /opt/devpi/server            # the fresh init is discarded
   tar xzf devpi-export-<ts>.tar.gz
   sudo /opt/devpi/venv/bin/devpi-import --serverdir /opt/devpi/server ./export
   sudo chown -R devpi:devpi /opt/devpi/server
   sudo systemctl start devpi
   ```
3. **Password reconcile** — imported users carry the backed-up credentials. If
   the Woodpecker org secret `devpi_password` disagrees (rotated since the
   export, or a fresh init was kept):
   ```bash
   DEVPI_MAGOGI_PASSWORD='<the org-secret value>' sudo -E /usr/local/bin/devpi-password-reconcile.sh
   ```
   **Never mint a random password** — CI's secret must match (P0.4b; the script
   refuses to run without an operator-provided value).
4. Verify: `curl -s https://<ip>.sslip.io/magogi/prod/+simple/ | head`, then a
   real `pip install mgf-common --index-url …/+simple/`, then a CI re-run of any
   repo. RTO clock stops at a green install.

## 3. Restore — Woodpecker Postgres (CI state + org secrets)

1. **VM** (if gone): `mgf-cloud-provision ci-host --key-name woodpecker-key
   --provider infomaniak --clouds-yaml … --cloud PCP-HOESFJV-dc3-a --yes`, then
   push `.env` (rebuilt from `examples/woodpecker.env.example` + the secret
   store) and `netrc` to `/opt/woodpecker/`.
2. Start **only Postgres**, restore, then the rest:
   ```bash
   cd /opt/woodpecker
   sudo docker compose up -d woodpecker-postgres        # fresh volume = empty db
   gunzip -c woodpecker-pg-<ts>.sql.gz | sudo docker compose exec -T woodpecker-postgres \
       psql -U woodpecker -d woodpecker
   sudo docker compose up -d
   ```
3. **Org secrets**: the dump *contains* the org secrets table as of the backup.
   Any secret rotated after the dump must be re-entered (Woodpecker UI → org
   `magogi-admin` → Secrets): `devpi_password`, `codeberg_package_token`,
   `codeberg_package_user`, `codeberg_repo_read`, `clouds_yaml` (cost-watch).
   Sources: the CI runbook §4 + the git-ignored `my_stuff/` stores.
4. Agents re-register against the restored DB (ids persist in their
   agent-config volumes; if an agent volume was also lost, update the id map in
   `ops/agent-reconciler.py` — runbook §9).
5. Verify: `/healthz` 200, both agents in `/api/agents`, one repo pipeline
   green end-to-end.

## 4. Restore — OpenObserve volume

```bash
cd /opt/openobserve
sudo docker compose stop openobserve
MP=$(sudo docker volume inspect -f '{{.Mountpoint}}' openobserve_openobserve-data)
sudo find "$MP" -mindepth 1 -delete
sudo tar xzf oo-snapshot-<ts>.tar.gz -C "$MP"
sudo docker compose up -d
```

Logship's fail-open spool re-delivers what apps shipped during the outage.
Verify: UI reachable, `backup_heartbeats` stream shows the next nightly beats.

## 4b. Is the estate running the code we think it is? (`ops/verify-deployed.sh`)

Run this **before trusting any of the above**, and after every ops deploy:

```sh
ops/verify-deployed.sh        # exit 0 = in sync · exit 1 = drift
```

It sha256-compares each canonical script in this repo against the file actually on the
host. **Why it exists:** every serious failure found on 2026-07-11 lived in the gap
between *committed* and *deployed* — the backup scripts had carried a heartbeat in git
for weeks while the hosts ran older copies with none (PAPER-03); the off-host mirror,
the one thing standing between us and single-VM data loss, **was not in git at all**
(PAPER-06). *"Recipes as code" that nobody deploys is worse than no recipes: it invites
a confident, wrong deploy.*

Divergences we have **decided** on (currently the three nightly backup scripts —
PAPER-05, pending the founder's storage-location call) are listed explicitly and
reported separately. **A divergence we have decided is a different thing from one we
have not noticed** — this tool exists to keep that distinction honest.

**Honest limitation:** it is a supervisor-run tool (it needs SSH to every host and the
repo on disk), not an automated gate. It does not yet run on a cron.

## 5. Restore drill record

P0 acceptance: **a real restore of each store from last night's backup, on a
throwaway VM, documented with timings** (Court executes; fill this table — it
is the evidence the RTO above is real, not aspirational).

> **Executed 2026-07-11 (Court).** All three stores restored **from the OFF-HOST MIRROR**
> (`/opt/backup-mirror/*` on agent2 — the artifact a real disaster would actually reach for,
> not the copy sitting on the source host), each into a **throwaway, isolated container**.
> Production was never touched; no secret ever left its host.
> **Result: all three restore. The RPO/RTO claim is now evidence, not aspiration.**

| Date | Store | Backup used | Duration | Within RTO 4h? | Notes |
|---|---|---|---|---|---|
| 2026-07-11 | **woodpecker-pg** | `woodpecker-pg-20260711-0614.sql.gz` (25 MB, off-host mirror) | **11 s** | ✅ (RTO is bounded by VM re-provision, not the restore) | `psql` restore into a `--network none` `postgres:16-alpine`: rc=0, **zero** stderr. **19 tables · 3,433 pipelines · 54 repos · 2 agents.** **All 17 org secrets recovered with non-empty values** (incl. `devpi_password`, the Codeberg tokens, `infomaniak_ssh_key`) — i.e. a total loss of woodpecker-prod is survivable. Data current: newest pipeline `2026-07-11 06:11`, matching the 06:14 dump. |
| 2026-07-11 | **devpi** | `devpi-export-20260711-0630.tar.gz` (63 MB, off-host mirror) | **18 s** | ✅ | Real `devpi-import` into a throwaway container: **SUCCESS — 220 wheels, 28 `mgf-common` versions**, the `magogi` index present. Export holds 3 indexes (`root/pypi`, `magogi/prod`, `magogi/dev`), 282 distributions, 309 `mgf-*` artifacts. ⚠️ **GOTCHA (cost me a failed attempt):** do **NOT** run `devpi-init` before `devpi-import` — the import refuses a serverdir that already has data (`fatal: The path '/srv' already contains devpi-server data`). Import into an **empty** serverdir; it creates it. |
| 2026-07-11 | **openobserve** | `oo-data-20260711-0615.tar.gz` (39 MB, off-host mirror) | **10 s** | ✅ | Volume extracted (592 MB) and a scratch `openobserve:v0.91.0` **booted on the restored volume** → `/healthz` **HTTP 200**. ⚠️ **GOTCHA:** a restored OO carries its **original** user set — the `ZO_ROOT_USER_*` env only seeds a *fresh* volume, so you need the **original root credentials** to log in. Those live in `/opt/openobserve/CREDENTIALS` **on the host — which is in no backup** (see PAPER-08). |

### What the drill proved — and what it exposed

**Proved:** every store restores from the off-host copy, fast, with its data intact. The
`RPO 24 h / RTO 4 h` line above is now backed by evidence rather than intent.

**Exposed (PAPER-08, 🔴):** restoring the **data** does not restore the **service**. The
host-level config and credentials — OO's `CREDENTIALS`, Woodpecker's `.env`
(`WOODPECKER_AGENT_SECRET`, OAuth), the logship collector token, the pager's Codeberg
token, and even **the SSH key the mirror itself uses to pull backups** — exist only on
their hosts and are in **no** backup. Lose a host and you can restore its data but cannot
bring the service back up without reconstructing those by hand.
