mgf-cloud-provision

Shape. Magogi Foundation infrastructure library. Declarative, idempotent provisioning of the federation's own self-hosted services on Infomaniak Public Cloud. Status. Pre-0.1.0 skeleton (Phase 1a). The core API + the devpi service spec exist and run end-to-end against mgf-cloud's in-memory FakeProvider; the live provider + cloud-init converge land in Phase 1b.

mgf-cloud-provision sits above mgf-cloud (the OpenStack/Infomaniak adapter). Where mgf-cloud is "talk to OpenStack," this is "declare + converge our services." A ServiceSpec describes the desired state of one service-bearing VM (devpi, a CI agent, Vault, …) as a VM + cloud-init; plan / apply / destroy converge the cloud to it.

from mgf.cloud.providers import get_provider
from mgf.cloud_provision import devpi_service_spec, plan, apply

spec = devpi_service_spec(key_name="woodpecker-key")
print(plan(spec, get_provider("fake")).action)      # Action.CREATE
apply(spec, get_provider("fake"))                    # creates the VM (fake, in-memory)

CLI:

mgf-cloud-provision plan    examples/devpi.toml --provider fake
mgf-cloud-provision apply   examples/devpi.toml --provider fake --yes
mgf-cloud-provision destroy examples/devpi.toml --provider fake --yes
mgf-cloud-provision ci-host --dry-run                # the Woodpecker CI host (built-in spec)
mgf-cloud-provision ci-host --with-agent --yes       # … + woodpecker-agent2

The whole surface runs with no cloud and no credentials (the fake provider), so it is fully testable in CI. The real provider (--provider infomaniak) loads credentials via mgf.cloud.config.load.

First customer — devpi

The reason this library exists now is the self-hosted package index rollout (replacing the Codeberg PyPI registry — closes the dependency-confusion findings of the 2026-06-18 CI audit). devpi_service_spec() / examples/devpi.toml encode the Phase-0-resolved sizing: a2-ram4-disk50-perf1 on ext-net1, reachable at <ip>.sslip.io. See the rollout plan in ~/.claude/plans/plan-self-hosted-pypi-sequencing-2026-06-18.md.

CI host — Woodpecker

Full operator runbook: docs/woodpecker-ci-runbook.md (architecture, hosts/access, secrets, self-healing, operations, failure-mode catalogue, rebuild). Backups + restore: docs/backup-restore-runbook.md (nightly recipes in backups/, heartbeats, RPO/RTO, per-store restore procedures).

mgf-cloud-provision ci-host (absorbed from the deprecated mgf-fed provision-ci-host — Wave-2 P0.4, one provisioner) provisions the server VM with the current production bundle rendered into its cloud-init, and --with-agent converges woodpecker-agent2 in the same run. The packaged bundle is byte-pinned by the unit tests to examples/, so the verb can never again render a stale topology.

The federation's self-hosted Woodpecker CI runs from examples/woodpecker-compose.yml (server + Postgres + agent + Caddy auto-TLS) — the sanitized live /opt/woodpecker compose, with every secret a ${VAR} sourced from .env (template: examples/woodpecker.env.example). The server was migrated SQLite → Postgres on 2026-06-21 to remove the single-writer "database is locked" ceiling that surfaced under fleet-wide CI bursts. The two-vhost Caddyfile (examples/woodpecker-Caddyfile) serves the web UI/API and the grpc.<domain> h2c endpoint remote agents dial.

A second agent on its own VM (examples/woodpecker-agent.toml + examples/woodpecker-agent-cloud-init.yml) removes the single-agent / single-Docker-daemon ceiling — it connects to the server's gRPC over TLS (a Caddy h2c proxy at grpc.<server-domain>:443; WOODPECKER_GRPC_SECURE=true). The shared WOODPECKER_AGENT_SECRET and the netrc are injected post-provision over SSH, never committed. Fleet capacity is now 2 agents × MAX_WORKFLOWS=2 = 4 concurrent workflows.

Two self-healing crons keep the agents honest — canonical home ops/ (co-located here in Wave-2 P0; the mgf-fed copies are deprecated): ops/self-heal.sh (every 5 min, reaps leaked wp_* Docker debris from daemon-teardown timeouts) and ops/agent-reconciler.py (every 3 min on the server host) — the reconciler recovers the "connected but not scheduling" state (an agent whose server-side worker slots were orphaned, sitting idle while work pends) by restarting that agent (local docker compose, or agent2 over SSH). It reads its API token + agent2 SSH key from root-600 files on the host (/opt/woodpecker/reconciler.token, /opt/woodpecker/agent2_id), never committed; cron: */3 * * * * python3 /opt/woodpecker/agent-reconciler.py.

Four green gates

ruff check src tests
mypy src
pytest --cov --cov-fail-under=80
lint-imports

Roadmap

• ✅ Phase 1a (this skeleton) — ServiceSpec + plan/apply/destroy over a CloudProvider; TOML spec loader; devpi service spec; CLI; full test coverage against the fake provider.
• ✅ Phase 1b — cloud-init converge for devpi (devpi-server + Caddy/LE + index init); devpi-prod live on Infomaniak.
• ✅ Wave-2 P0 (survival) — ci-host verb absorbed from mgf-fed provision-ci-host (current Postgres bundle, byte-pinned); backup recipes as code (backups/) + restore runbook; devpi password-reconcile; self-heal scripts co-located (ops/).
• ⏳ Later — vault, registry-mirror service specs; plan-side drift report + idempotent re-apply across the fleet.

License

MIT.