# MGF Woodpecker CI — Operator Runbook

The federation's self-hosted CI: a [Woodpecker](https://woodpecker-ci.org) server + two
agents on Infomaniak Public Cloud (OpenStack), driving builds for the Codeberg-hosted
mgf-* repos. This runbook is the single source of truth for operating, debugging, and
rebuilding it. The infrastructure-as-code lives beside it in `examples/`.

> Provenance: built out 2026-06; Postgres migration + second agent + self-healing added
> 2026-06-21. Sanitized compose/cloud-init/reconciler are in `examples/`; secrets live
> only on the hosts (root-600) and are **never committed**.

---

## 1. Architecture at a glance

```
        Codeberg (Forgejo forge)                 devpi  (package index)
        codeberg.org  ── webhooks/OAuth ──┐      195.15.203.250.sslip.io
                                          │      /magogi/prod  (separate VM: devpi-prod)
                                          ▼
   ┌───────────────────── woodpecker-prod  (195.15.241.48, a4-ram8) ─────────────────────┐
   │  caddy:2  ──443──►  woodpecker-server v3.8.0  ──►  woodpecker-postgres:16            │
   │   ├─ https://195.15.241.48.sslip.io        (web UI / REST API → server :8000)        │
   │   └─ https://grpc.195.15.241.48.sslip.io   (h2c → server gRPC :9000, for agents)     │
   │  woodpecker-agent  (agent id 14, local, MAX_WORKFLOWS=2)                             │
   └──────────────────────────────────────────────────────────────────────────────────────┘
                                          ▲
                            gRPC over TLS (WOODPECKER_GRPC_SECURE=true)
                                          │
   ┌──────────────── woodpecker-agent2  (195.15.203.243, a4-ram8) ────────────────┐
   │  woodpecker-agent  (agent id 15, remote, MAX_WORKFLOWS=2)                     │
   └──────────────────────────────────────────────────────────────────────────────┘

   Fleet CI capacity = 2 agents × MAX_WORKFLOWS 2 = 4 concurrent workflows.
```

- **Forge:** Codeberg (Forgejo). OAuth app "MGF Woodpecker CI" — **confidential** client
  (public clients require PKCE and fail), Client ID `3c559ac2-07ff-422d-9799-4f93257147af`,
  redirect `https://195.15.241.48.sslip.io/authorize`. Webhooks deliver push/PR events.
- **Package index:** builds install from **devpi** (`https://195.15.203.250.sslip.io/magogi/prod/+simple/`),
  a single anonymous-read index serving mgf-* and mirroring PyPI (no Codeberg token needed at build time).
- **DB:** **Postgres 16** (migrated from SQLite 2026-06-21 — SQLite's single-writer lock
  hit "database is locked" under fleet-wide CI bursts).

## 2. Hosts & access

| Host | IP | Flavour | OpenStack | Role | Stack dir |
|---|---|---|---|---|---|
| `woodpecker-prod` | `195.15.241.48` | a4-ram8-disk20-perf1 (4 vCPU/8 GB) | PCP-HOESFJV / dc3-a | server + Postgres + Caddy + agent 14 | `/opt/woodpecker/` |
| `woodpecker-agent2` | `195.15.203.243` | a4-ram8-disk20-perf1 | PCP-HOESFJV / dc3-a (id `e0f27bb0-…`) | agent 15 only | `/opt/woodpecker-agent/` |
| `devpi-prod` | `195.15.203.250` | a2-ram4-disk50-perf1 | PCP-HOESFJV / dc3-a | package index (separate concern) | `/opt/devpi/` |

SSH (OpenStack keypair `woodpecker-key`): `ssh -i mgf-cloud/my_stuff/infomaniak/woodpecker_id_ed25519 ubuntu@<ip>`.
`ubuntu` has passwordless sudo. Compose ops need `sudo` (run as root).

## 3. Components & config

- **woodpecker-server** (`v3.8.0`) — `/opt/woodpecker/docker-compose.yml`. Env from
  `/opt/woodpecker/.env`. Postgres via `WOODPECKER_DATABASE_DRIVER=postgres` +
  `WOODPECKER_DATABASE_DATASOURCE=postgres://woodpecker:…@woodpecker-postgres:5432/woodpecker?sslmode=disable`.
- **woodpecker-postgres** (`postgres:16-alpine`) — volume `postgres-data`. Password in `.env`
  (`WOODPECKER_POSTGRES_PASSWORD`).
- **woodpecker-agent** (`v3.8.0`) — one per host. Key settings: `WOODPECKER_MAX_WORKFLOWS=2`,
  `stop_grace_period: 3600s` (graceful drain — see §6), agent-config volume (persists the
  agent id across restarts so it reuses one record). Remote agent adds
  `WOODPECKER_SERVER=grpc.195.15.241.48.sslip.io:443` + `WOODPECKER_GRPC_SECURE=true`.
- **caddy** (`caddy:2`) — `/opt/woodpecker/Caddyfile`, auto-TLS (Let's Encrypt). Two vhosts:
  `{$WOODPECKER_DOMAIN}` → `server:8000` (web/API); `grpc.{$WOODPECKER_DOMAIN}` →
  `h2c://server:9000` (agent gRPC).
- **Agent ids are stable** (persisted in each agent's `agent.conf` in its agent-config
  volume): **14 = local** (server host), **15 = remote** (agent2). Several automations
  hardcode this map — see §6/§8.

## 4. Secrets (all root-600 on the hosts, NEVER committed)

| Path | What |
|---|---|
| `/opt/woodpecker/.env` | server: HOST, DOMAIN, ADMIN, FORGEJO_CLIENT/SECRET, AGENT_SECRET, POSTGRES_PASSWORD |
| `/opt/woodpecker-agent/.env` | agent2: AGENT_SECRET (shared with server), GRPC_SERVER |
| `/opt/woodpecker[-agent]/netrc` | Codeberg `read:package` netrc (mounted into build steps) |
| `/opt/woodpecker/reconciler.token` | **FALLBACK ONLY** — the reconciler now mints its JWT per-run from the DB (PAPER-01; a static JWT dies on any user-hash rotation, proven 2026-06-23→07-10); the file is read only when the Postgres container is unreachable |
| `/opt/woodpecker/agent2_id` | the `woodpecker-key` private key, for the reconciler's SSH to agent2 |
| `/etc/mgf/backup-heartbeat.{url,token}` | OO `backup_heartbeats` ingest endpoint + auth (operator-pushed; shared by `backups/*` AND the self-heal layer's success heartbeats, PAPER-02) |

Woodpecker **org secrets** (in the DB, surfaced to pipelines): `codeberg_package_token`,
`codeberg_package_user`, `codeberg_repo_read`, `devpi_password`. The OAuth client secret is
mirrored in `mgf-standard/my_stuff/OAuth2/codberge` (gitignored).

## 5. Health checks (one-liners)

Use a Woodpecker admin token (`WPT`) and the REST API:

```bash
# fleet capacity: free=4 + run/pend=0 ⇒ fully idle & healthy. free<4 while idle ⇒ a stuck agent.
curl -sk -H "Authorization: Bearer $WPT" https://195.15.241.48.sslip.io/api/queue/info \
  | python3 -c 'import sys,json;s=json.load(sys.stdin)["stats"];print(s)'
# agents: both 14 & 15, no_schedule=false, recent last_contact
curl -sk -H "Authorization: Bearer $WPT" https://195.15.241.48.sslip.io/api/agents
```

On a host: `cd /opt/woodpecker[-agent] && sudo docker compose ps`. Build debris check:
`sudo docker ps -aq --filter name=wp_ | wc -l` (0 when idle).

`worker_count` ("free") = sum of free worker slots across schedulable agents. **free=2 while
both agents are idle ⇒ one agent is "connected but not scheduling"** (see §6.3).

## 6. Self-healing — three layers

Restarts/recovery are automated; understand each before intervening. The deployable
scripts live in **`ops/`** (`ops/self-heal.sh`, `ops/agent-reconciler.py`) — the
canonical home since Wave-2 P0 (the copies in `mgf-fed/deploy/woodpecker/` are
deprecated). Nightly **backups** are the fourth survival layer — `backups/` +
`docs/backup-restore-runbook.md`.

### 6.1 `stop_grace_period: 3600s` (on agents)
Restarting an agent **drains gracefully** — it finishes running pipelines, then exits clean.
Prevents the SIGKILL-mid-pipeline (default 10s grace) that orphans server-side worker slots.
Consequence: `docker compose restart/stop` of a **busy** agent **blocks until it drains**
(desired). For an emergency fast kill: `docker compose kill && docker compose up -d`.

### 6.2 `self-heal.sh` (cron `*/5`, on both hosts)
Reaps leaked `wp_*` Docker debris from **daemon-teardown timeouts** (under load the Docker
daemon times out on container/network teardown — `context deadline exceeded` — leaving
`wp_*` networks that make the next workflow fail with "network already exists"). Prunes
unused networks + dangling `wp_*` volumes + dead containers; restarts the agent only if it
holds stuck "created" debris with **nothing running** (never kills live work).

### 6.3 `agent-reconciler.py` (cron `*/3`, server host only)
Recovers two states that 6.1/6.2 and the `ping` healthcheck don't:
- **STUCK** — agent container UP + schedulable + idle while work pends (orphaned worker
  slots from a force-kill or a `no_schedule` toggle). `DELETE /api/agents/<id>` can't clear
  it (409 on a connected agent) — only a restart.
- **CRASH / DOWN** — agent container exited (`docker kill`, OOM, crash-loop). Detected
  directly via container state (a crashed agent also leaks orphaned `wp_*` containers, so a
  "0 build containers = idle" heuristic would misread it as busy).

Recovery (either case, after 2 consecutive detections to avoid transients): reap `wp_*`
debris + `docker network prune` + **`docker compose up -d --force-recreate`** (a plain
`restart` fails on a crashed container's stale network). Local agent via docker; agent2 over
SSH. Reads `agent2_id` from §4. Log: `/var/log/woodpecker-reconciler.log`.

**Auth (PAPER-01):** the API JWT is minted **per-run** from the Woodpecker DB (`id|hash`
via `docker exec … psql`, HS256 with the user hash as key — the same math as the `mgf`
workspace `tools/woodpecker_api.sh`), so a hash rotation can never strand it on a stale
credential again (the static `reconciler.token` 401'd silently 2026-06-23 → 2026-07-10).
The token file is a fallback for DB-down runs only; every run logs which path it used
(`ok token=db-mint …` / `token=file-fallback …`).

### 6.4 Heartbeats + the estate watcher (PAPER-02 — a dead safety net must page)

Every **successful** run of `agent-reconciler.py` and `self-heal.sh` POSTs a
success-only heartbeat (`job=agent-reconciler` / `job=self-heal`) to the OO
`backup_heartbeats` stream via `/etc/mgf/backup-heartbeat.{url,token}` (§4) — the same
dead-man's-switch contract as `backups/*`. Heartbeats are **fail-open**: a heartbeat
failure never breaks the heal; the watcher pages on the MISS.

The read side is `ops/estate-watch-heartbeats.sh` — deploy to agent2 and wire into the
live `/opt/watcher/estate-watch.sh` (deploy steps in the script header). Freshness
windows: **30 min** for the self-heal layer (crons run `*/3` and `*/5`), **26 h** for the
nightly backups. Exit 1 = stale job(s); exit 2 = the check itself couldn't run (page on
both — a blind watcher is a dead watcher).

## 7. Common operations

```bash
# --- restart the server (brief CI downtime) ---
ssh … woodpecker-prod 'cd /opt/woodpecker && sudo docker compose restart woodpecker-server'

# --- restart an agent (busy agent DRAINS first; idle is instant) ---
ssh … <host> 'cd /opt/woodpecker[-agent] && sudo docker compose restart woodpecker-agent'
#   emergency fast kill: sudo docker compose kill woodpecker-agent && sudo docker compose up -d

# --- add / re-enable a repo --- Woodpecker UI → Repositories → + (push + pull_request only)

# --- pin work to one agent (testing) --- PATCH no_schedule, then RESTART that agent after
#     re-enabling (v3.8.0 does NOT re-spawn workers on no_schedule false→true → §9):
curl -sk -X PATCH -H "Authorization: Bearer $WPT" -H 'Content-Type: application/json' \
  https://195.15.241.48.sslip.io/api/agents/14 -d '{"no_schedule":true}'

# --- trigger a pipeline (smoke test) ---
curl -sk -X POST -H "Authorization: Bearer $WPT" \
  https://195.15.241.48.sslip.io/api/repos/<repo_id>/pipelines -d '{"branch":"main"}'
```

## 8. Failure-mode catalogue

| Symptom | Cause | Recovery |
|---|---|---|
| `database is locked`, server API 500s, agent can't register | SQLite single-writer under burst | **Fixed by Postgres migration.** If reverted: re-apply Postgres (§10). |
| Agent `worker_count`/queue stuck at 0, `wp_*` networks leak, `context deadline exceeded` | Docker daemon teardown timeout under load | `self-heal.sh` (auto, §6.2); manual: prune `wp_*` + restart agent. |
| `free` < capacity while agents idle; work pends; agent "healthy" but not scheduling | Orphaned worker slots (force-kill mid-pipeline, or `no_schedule` toggle) | `agent-reconciler.py` (auto, §6.3); manual: `docker compose up -d --force-recreate` the agent. |
| Agent container exited and stays down | `docker kill`/`docker stop` don't trip `restart: unless-stopped`; or crash-loop | `agent-reconciler.py` (auto). A real process crash DOES auto-restart via `restart: unless-stopped`. |
| `docker compose restart` of a crashed agent fails: `network … not found` | Stale network on a container that died mid-pipeline | Use `up -d --force-recreate` (+ `network prune`). The reconciler does this. |
| All steps fail `exit 126` / `network … not found` right after clone | Daemon overload during concurrent teardown burst | Re-run the pipeline once load clears; self-heal mitigates. |
| Pipeline green but Codeberg commit-status stays pending → merge blocked (405) | Woodpecker→Codeberg status not posted (the required `ci/woodpecker/pr` context missing) | Re-trigger the PR pipeline (force-push an amend, or restart the WP pipeline); merge once the required context is `success`. |

## 9. Known gotchas

- **`no_schedule` toggle bug (v3.8.0):** setting an agent `no_schedule` true→false does NOT
  re-spawn its worker pollers — it stays connected but not scheduling. **Restart the agent
  after re-enabling.** (The reconciler also catches this.)
- **`docker kill`/`docker stop` ≠ crash:** they're user-initiated, so `restart: unless-stopped`
  won't bring the container back. A real process crash will. The reconciler covers both.
- **`stop_grace_period: 3600s`:** a `restart`/`stop` of a busy agent blocks until it drains.
  Use `kill` + `up -d` for an emergency.
- **Agent ids (14/15) are hardcoded** in `agent-reconciler.py`'s `AGENTS` map. They're stable
  (persisted in `agent.conf`), but if an agent-config volume is wiped the agent re-registers
  with a NEW id — update the map.
- **Version string `next-c97467ddcd`** is just v3.8.0's embedded build id, not a dev build.

## 10. Rebuild / provision from scratch

IaC in `examples/` (sanitized — inject secrets out-of-band after provisioning). The
server VM is one verb: **`mgf-cloud-provision ci-host`** (absorbed from the deprecated
`mgf-fed provision-ci-host`); its cloud-init renders the **current** Postgres compose +
two-vhost Caddyfile from a packaged bundle byte-pinned to `examples/`.

1. **Server host** —
   ```bash
   mgf-cloud-provision ci-host --key-name woodpecker-key --provider infomaniak \
       --clouds-yaml …/PCP-HOESFJV-clouds.yaml --cloud PCP-HOESFJV-dc3-a --yes
   ```
   Then fill a real `.env` from `examples/woodpecker.env.example`, push it + `netrc` to
   `/opt/woodpecker/`, and `docker compose up -d`. Postgres builds its schema on first
   boot; **restore last night's `pg_dump` + re-enter the org secrets** per
   `docs/backup-restore-runbook.md` (the org secrets live only in that DB). To migrate
   data from a legacy SQLite: stand up empty Postgres → start the server so Woodpecker
   builds the schema → stop → `pgloader WITH data only, truncate, disable triggers,
   reset sequences` from a `journal_mode=DELETE` copy of the sqlite (pgloader can't read
   a WAL-mode db read-only) → start. (A full pgloader migration FAILS — xorm loops
   dropping sqlite-named indexes.)
2. **gRPC for remote agents** — already in the rendered bundle: the Caddyfile ships both
   vhosts (`{$WOODPECKER_DOMAIN}` → server:8000, `grpc.{$WOODPECKER_DOMAIN}` → h2c
   server:9000; `examples/woodpecker-Caddyfile`).
3. **Second agent** — `mgf-cloud-provision ci-host --with-agent --yes` (same flags), or
   separately: `mgf-cloud-provision apply examples/woodpecker-agent.toml --provider
   infomaniak --clouds-yaml …/PCP-HOESFJV-clouds.yaml --cloud PCP-HOESFJV-dc3-a --yes`.
   Then SSH in and write `/opt/woodpecker-agent/.env` (shared AGENT_SECRET +
   GRPC_SERVER) + `netrc`, and `docker compose up -d`.
4. **Self-healing** — deploy `ops/self-heal.sh` (both hosts, cron `*/5`) and
   `ops/agent-reconciler.py` (server, cron `*/3`) + its token/key (§4).
5. **Backups** — deploy the nightly backup crons (`backups/`, see
   `docs/backup-restore-runbook.md`) — a rebuilt estate without its backup crons is the
   pre-P0 zero-backup state all over again.

Rollback (Postgres→SQLite): restore `/opt/woodpecker/docker-compose.yml.pre-pg.bak` (or
comment the 2 `WOODPECKER_DATABASE_*` lines) + restart — the legacy `woodpecker.sqlite`
(WAL) is retained in the server-data volume alongside a `.pre-pg.<ts>` backup.
