# MGF_INFRA — the Magogi Foundation infrastructure & access registry

> **The authoritative inventory of what the Foundation *runs and owns*, and how to
> reach it.** Sibling of [`MGF_LIBRARIES.md`](MGF_LIBRARIES.md): that registry is
> "what we publish"; this is "what we operate." Maintained by the mgf-common
> supervisor; keep it current as infra changes.
>
> **Why this exists (read before declaring anything "blocked").** Agents kept
> treating a transient failure against *our own* infrastructure — an expired
> Woodpecker token, a 401, a stopped container — as a terminal, human-gated wall.
> It never is. **If a dependency is in this registry, it is Magogi-owned: you have
> an access path and a recovery path. Reason from capabilities (what we control +
> how to reach it), not from a recorded "blocked" conclusion.** This is binding as
> **AG-20** (see [docs/standards/AGENTS.md](docs/standards/AGENTS.md)).
>
> **No secrets here.** This file is committed. It records *where* credentials live
> (always under a repo's git-ignored `my_stuff/`, AG-18) and *how* to obtain a
> token — never the values themselves.

## §1 What we own (inventory)

All VMs run on **Infomaniak Public Cloud** (OpenStack, project `PCP-HOESFJV`,
region `dc3-a`), reachable over `ext-net1` direct-public, fronted by Caddy + Let's
Encrypt. **One SSH key reaches every host:**
`mgf-standard/my_stuff/infomaniak/woodpecker_id_ed25519` (OpenStack keypair
`woodpecker-key`), user `ubuntu`.

| Service | Role | URL | Host (SSH `ubuntu@…`) | Self-hosted? |
|---|---|---|---|---|
| **Woodpecker CI** | the fleet's CI/CD (`woodpecker-prod`, v3.8.0; server+agent14+postgres+caddy) | https://195.15.241.48.sslip.io | `195.15.241.48` | ✅ ours |
| **Woodpecker agent2** | second CI agent VM (`woodpecker-agent2`, a4-ram8, agent id 15) — capacity: **2 agents × MAX_WORKFLOWS 2 = 4 slots** fleet-wide; rebuild per `mgf-cloud-provision/docs/woodpecker-ci-runbook.md` §10 | — (gRPC to the server) | `195.15.203.243` | ✅ ours |
| **devpi index** | the private PyPI the fleet publishes to + resolves from (`magogi/prod`) | https://195.15.203.250.sslip.io/magogi/prod/+simple/ | `195.15.203.250` | ✅ ours |
| **OpenObserve** | central logs (app→logship→collector→OO) | https://195.15.203.114.sslip.io | `195.15.203.114` | ✅ ours |
| **Codeberg** | the forge (org `magogi-admin`; repos, PRs, package webhooks) | https://codeberg.org/magogi-admin | — (SaaS, but **our org**) | external, owned-account |
| **Infomaniak** | the cloud provider (VMs, storage, networking) | https://manager.infomaniak.com · https://dashboard.pub1.infomaniak.cloud | — | external, owned-account |

Cost: the VMs run 24/7 (~€38/mo); tracked by `mgf-cost-watch` (Woodpecker cron,
repo id 25) — since Wave-2 P0 the cron **fails red when the monthly estimate
exceeds `COST_ALERT_MONTHLY_CHF`** (default 100; founder adjusts the Woodpecker
variable — gate P1). Tear-down/resize via `mgf-cloud vm …`.

## §2 Access & recovery paths (the "you are not blocked" details)

### Woodpecker CI (the one that bit us)
- **Codeberg commit-status contexts:** `ci/woodpecker/{pr,push,tag,manual}/woodpecker`. Branch protection requires `ci/woodpecker/pr/woodpecker`.
- **API token (the recurring gate):** the stored token (`my_stuff/woodpecker_token.txt`) is a JWT that **expires** (→ 401). **Do not treat that as blocked, and do not need a browser/OAuth.** Mint a fresh admin token from the host (it's ours):
  1. `ssh -i my_stuff/infomaniak/woodpecker_id_ed25519 ubuntu@195.15.241.48`
  2. A v3 user/CLI token is an **HS256 JWT, claims `{"type":"user","user-id":"<id>"}`, signed with the user's `hash`** (NOT the global `jwt-secret`). Get id+hash: `sudo docker exec woodpecker-woodpecker-postgres-1 psql -U woodpecker -d woodpecker -tAc "select id, hash from users where login='magogi-admin'"` (magogi-admin = id 1, admin). Hand-roll the JWT (hmac-sha256, base64url no padding). `GET /api/user` → 200. *(Verified against v3.8.0 `shared/token/token.go` + `server/api/user.go`.)*
- **Activate a new repo's CI:** `POST /api/repos?forge_remote_id=<codeberg-repo-id>` (get the id from `GET codeberg.org/api/v1/repos/<owner>/<repo>`).
- **Publish a release:** push a `v*` tag → the repo's pipeline runs the publish step using the **org-level** `devpi_password` secret (`events:["tag"]`, applies to all repos in org id 1). No manual twine needed.
- **Agent wedge (worker_count=0):** `cd /opt/woodpecker && sudo docker compose restart woodpecker-agent` (a `*/5` self-heal cron also exists). Server config: `/opt/woodpecker/.env`.
  - ⚠️ **Known gap (FEEDBACK.md PAPER-111):** the `*/5` self-heal did **not** clear an observed 15-min wedge (queue `worker_count:0, pending:13, running:4`) — it cleared only on the **manual** restart above. Likely the cron detects *container-down*, not the *`workers==0`-with-a-backlog* wedge shape (or it restarts-but-re-wedges). **Host-side fix to apply:** (1) verify the cron actually runs (`crontab -l`; inspect its log); (2) make its trigger the **wedge condition** — restart when `worker_count==0 && pending>0` (poll `GET /api/queue/info` with an admin token), not just container-down; (3) **emit a signal** (an OpenObserve record / push) whenever it fires *or* a wedge persists > N min, so a silent-failing self-heal is itself observable; (4) optionally a compose `healthcheck` that restarts the agent on a failed heartbeat. Until deployed, a wedge = the manual restart above.
    **Status (verified on-host, 2026-07-10 — the PAPER-111 investigation):**
    (1) ✅ **both crons run** — root crontab: `*/5 self-heal.sh` (healthy; logs `ok` every 5 min; 13 historical WEDGE recoveries) + `*/3 agent-reconciler.py`. They cover **different wedge shapes**: `self-heal.sh` = docker-debris (`Created`-stuck && nothing running — the observed wedge had `running:4`, so it correctly did not fire); `agent-reconciler.py` = the observed shape (agent DOWN, or schedulable+idle while work pends; 2-hit guard; recovers via force-recreate; covers agent2 over SSH).
    (2) ✅-after-repair **the reconciler had been silently dead since 2026-06-23** — every run `api error` (~1,950 log lines): its static `/opt/woodpecker/reconciler.token` JWT went **401** (Woodpecker JWTs are HS256-signed with the user's DB `hash`; a hash rotation kills every previously-minted static token). **That is why the 2026-07-01 wedge outlived the safety net.** Re-minted on-host 2026-07-10 (the `tools/woodpecker_api.sh` mint; old file kept as `.bak-2026-07-10`), verified: mint → `queue/info` 200, next cron ticks clean. **Durable fix (companion, `mgf-cloud-provision/ops/`):** the reconciler should mint per-run from the DB exactly like `woodpecker_api.sh` (it runs as root beside the DB) — no static token to rot.
    (3) ✅ **DELIVERED 2026-07-11 — the signal now exists, and the net is self-observing.** `agent-reconciler.py` **mints its JWT per-run from the Woodpecker Postgres** (no static token left to rot; unattended cron runs log `ok token=db-mint pending=…`, zero api-errors since), and both it and `self-heal.sh` (BOTH hosts) POST **success-only heartbeats** to the OO `backup_heartbeats` stream; the agent2 estate watcher (`*/10`) runs `estate-watch-heartbeats.sh` and **pages via its deduped Codeberg issue** when a heartbeat goes stale (>30 min for the self-heal layer). Proven both ways on deploy (stale job → exit 1 → page; fresh → exit 0 → silent). Credential: a **least-privilege OO service account** (`heartbeat@magogi.org`), root-600 at `/etc/mgf/backup-heartbeat.{url,token}` on both hosts — it did **not exist before** (§5's heartbeat had never actually been wired). Code + deploy record: `mgf-cloud-provision` (`ops/`, PAPER-01/02). ⚠️ **agent2 cron MUST set `WOODPECKER_STACK_DIR=/opt/woodpecker-agent`** — the shared script defaults to `/opt/woodpecker`, absent there (deploying without it silently disables agent2's self-heal). (4) compose `healthcheck`: still optional/unimplemented.

### devpi (publish / inspect)
- Releases land here via the Woodpecker tag pipeline (above). The upload account is `magogi`; the password is the Woodpecker org secret `devpi_password` (not stored in any repo). Anonymous read at the `+simple/` index. Host SSH for server-side ops.
- **Recovery posture (decided 2026-07-10, PAPER-112 §8):** `magogi/prod` is **strictly append-only** — a botched upload is recovered by **bumping to the next patch version**, never by removal (published versions are immutable, PKG-07). The volatile-toggle + `devpi remove` capability is an **emergency-only** admin op that requires a founder decision package (the destructive-op hard limit). Identity stays the single `magogi` account (accepted risk; revisit at the public flip).
- **`$DEVPI_URL` (CI indirection, PAPER-113):** generated pipelines never hardcode the index endpoint — they reference the Woodpecker **server-level** variable `DEVPI_URL` (injected into every step via `WOODPECKER_ENVIRONMENT` in `/opt/woodpecker/.env` + compose; set 2026-07-04; value = the §1 endpoint `https://195.15.203.250.sslip.io/magogi/prod/+simple/`), with a fail-fast guard line in each consuming step. Variable name is coordinated with `sooth-standard` (their conversion shipped first, live-green). A devpi host/IP move is therefore: update the server variable + the §1 row here + `mgf-fed`'s `_DEVPI_INDEX` constant (`mgf.fed._woodpecker` — the generator-side installs still carry the literal until that companion lands), then run the WF-13 re-render wave — never a hand-search across committed pipelines.

### Codeberg (forge)
- API token: `~/.config/tea/config.yml` (used by `tea` + the REST API). Org repos are **private**; branch-protect via `POST /repos/<owner>/<repo>/branch_protections`.

## §3 Discipline (AG-20)

Before reporting an action as **blocked / human-gated / impossible**:
1. **Check this registry** — is the dependency Magogi-owned?
2. **If owned, attempt the self-owned recovery path** above (SSH, mint/rotate a credential, restart a service). An auth/access failure on owned infra is a *solvable sub-problem*, not a terminal wall.
3. **If it genuinely can't proceed, state the blocking assumption explicitly** ("treating X as blocked because Y") so a human can catch a wrong assumption cheaply.
4. **Recorded "blocked"/"gated" notes** (memory, trackers) MUST carry their unblock-path + as-of date, and be **re-derived against current capabilities** — never inherited as settled fact.

## §4 Maintenance

Update when infra is added/removed/moved, a host IP changes, or an access/recovery
path changes — or when a backup job, heartbeat, or the watcher (§5) changes. Per-host
live details live in each consumer's git-ignored
`my_stuff/` (e.g. `mgf-standard/my_stuff/infomaniak/FACTS.md`); this file is the
fleet-level index + the access/recovery playbook. Related standards: **AG-20**
(this discipline), **AG-18** (`my_stuff/` git-ignored), **PKG-06** (devpi index),
**WF-13** (generated CI).

## §5 Backups & paging (Wave-2 P0 — the estate no longer dies quietly)

> Before P0 the estate had **zero backups** (devpi — every install and every release;
> the Woodpecker Postgres — all CI state + the org secrets; OpenObserve) and **nothing
> paged**. This section is the fleet-level truth; the deployable code + the full
> procedures live in **`mgf-cloud-provision`**: recipes `backups/`, self-heal `ops/`,
> restore procedures + drill record `docs/backup-restore-runbook.md`.

**Objectives: RPO 24 h · RTO 4 h** (stated in the restore runbook; proven by the P0
restore drill on a throwaway VM — the drill record table lives in that runbook).

### The three nightly backup crons

Recipes as code in `mgf-cloud-provision/backups/` (parameterized; installed per host via
the matching `examples/*-backup-cloud-init-fragment.yml`, byte-pinned by its tests):

| Job (`heartbeat job`) | Host | What | Schedule (UTC) | Retention |
|---|---|---|---|---|
| `devpi-export` | `195.15.203.250` | stop → `devpi-export` → start → tar.gz → prune → optional Swift upload | 02:17 | 7 d (+ object storage) |
| `pg-dump` | `195.15.241.48` | `docker exec pg_dump \| gzip` → integrity-check → prune | 02:41 | 14 d |
| `oo-snapshot` | `195.15.203.114` | OpenObserve data-volume tar → prune | 03:03 | 7 d |

Backups land in `/opt/backups/` (`700 root:root`, dumps `600`) on each host; job log `/opt/backups/backup.log`.

### The heartbeat pattern (dead-man's-switch)

The design: a job **POSTs a heartbeat record only on success** to the OpenObserve
`backup_heartbeats` stream; a failed job, a dead cron, a wedged host and an unconfigured
heartbeat then all look identical — a **missed heartbeat** — and all page.

**Credential (live since 2026-07-11):** endpoint + auth header in
`/etc/mgf/backup-heartbeat.url` + `…heartbeat.token` (**root-600**; values never
committed — the committed artifacts carry only the file paths). Auth is a
**least-privilege OpenObserve service account** (`heartbeat@magogi.org`, org `default`;
the OO root credential is deliberately NOT reused). Installed on `woodpecker-prod` +
`woodpecker-agent2`. *Before that date these files did not exist on any host — the
mechanism below was documented but never wired.*

**What actually heartbeats today — the honest table.** (The rest is the work list, not
the state; a heartbeat you have not seen land in OO is not a heartbeat.)

| Job | Heartbeats to OO? | Watched by the pager? |
|---|---|---|
| `agent-reconciler` (prod, `*/3`) | ✅ since 2026-07-11 | ✅ stale > 30 min → pages |
| `self-heal` (prod + agent2, `*/5`) | ✅ since 2026-07-11 | ✅ stale > 30 min → pages |
| `pg-dump` (woodpecker-prod, 02:20) | ✅ since 2026-07-11 | ✅ stale > 26 h → pages |
| `devpi-export` (devpi-prod, **02:17 — cron INSTALLED 2026-07-11; it had NEVER been scheduled**, see below) | ✅ since 2026-07-11 | ✅ stale > 26 h → pages |
| `oo-snapshot` (openobserve-prod, 02:30) | ✅ since 2026-07-11 | ✅ stale > 26 h → pages |

✅ **COMPLETE since 2026-07-11** — every self-heal AND every backup job now heartbeats,
and the watcher pages on staleness (`CHECKS="agent-reconciler:1800 self-heal:1800
pg-dump:93600 devpi-export:93600 oo-snapshot:93600"`). A dead cron can no longer look
healthy. (`mgf-cloud-provision` PAPER-03.)

> 🔴🔴 **What this uncovered — `devpi` had NO backup schedule AT ALL.** The index holding
> **every published wheel and the fleet's entire release history** had no cron on any user,
> nothing in `/etc/cron.d`, and no systemd timer — the `devpi-export-nightly.sh` script was
> present and worked, but **nothing ever ran it**. Its only export was **7 days old**, taken
> by hand, while this document asserted a nightly 02:17 job. Real RPO was **7 days**, against
> a stated objective of 24 h — and it was silent. **Fixed 2026-07-11:** cron installed, a
> fresh 61 MB export taken, and the job now heartbeats + is watched. Full record:
> `mgf-cloud-provision` FEEDBACK **PAPER-04**.
> **Lesson: a documented schedule is not a running schedule — verify the crontab, not the doc.**

> ✅ **The committed `backups/*.sh` ARE what runs on the hosts** (`mgf-cloud-provision` PAPER-05,
> resolved 2026-07-11). The reconciliation found the deployed scripts were the *older, less-safe*
> versions, so rather than bless them we deployed a **best-of** (repo safety design + the hosts'
> good additions) to all three hosts, each manually run-verified. `verify-deployed.sh` now byte-tracks
> them (IN SYNC). Dumps containing the org secrets live in `/opt/backups` (`700 root:root`, dumps `600`).

### The watcher (the pager, until real paging integrates)

A cron on **agent2** (`195.15.203.243` — deliberately a *different* host from every
store it watches): `/opt/watcher/estate-watch.sh`, `*/10`. **Status: ✅ LIVE
(verified on-host 2026-07-10** — root crontab entry + `watch.log` heartbeating
`estate-watch OK` same-day**)**, with a narrower scope than first planned:
- **Deployed today:** uptime probes (devpi `/+simple/`, Woodpecker `/healthz` with a
  `/` fallback, OO `/`; each double-probed 10 s apart) → **files a deduped Codeberg
  issue** on `magogi-admin/mgf-standard` (the founder's watched surface) + writes its
  own heartbeat line so a dead watcher is diagnosable.
- **Heartbeat freshness — ✅ LIVE since 2026-07-11:** `/opt/watcher/estate-watch-heartbeats.sh`
  (0750) runs inside the same `*/10` cron and routes a stale heartbeat through the
  existing deduped issue pager. `CHECKS` currently covers the **self-heal layer only**
  (`agent-reconciler:1800 self-heal:1800`) — the nightly backups are excluded **because
  they do not heartbeat at all** (the honest table above; `mgf-cloud-provision`
  PAPER-03). Verified both ways at deploy: a never-heartbeating job → `STALE` + exit 1 →
  pages; the real set → `heartbeats OK` + exit 0 → silent.
  This closes the §2 PAPER-111 signal gap: the self-heal net is now **self-observing** —
  if the reconciler dies (or silently 401s, as it did for 17 days) it stops heartbeating
  and the watcher pages within ~30 min.
- **Still not implemented:** a direct `GET /api/queue/info` wedge probe from agent2
  (would need an admin credential there; the heartbeat-staleness route above covers the
  same failure without distributing a new secret).
Cost alerting is the `mgf-cost-watch` red cron (§1, `COST_ALERT_MONTHLY_CHF`).

### Restore

Per-store procedures (devpi import + **password-reconcile to the org-secret value**,
Postgres restore + org-secret re-entry, OO volume restore):
**`mgf-cloud-provision/docs/backup-restore-runbook.md`**. Rebuild-from-scratch:
`mgf-cloud-provision ci-host` + its CI runbook §10 (the `mgf-fed provision-ci-host`
verb is a deprecation shim).
