# mgf-cloud — design & architecture

> The planning + design record for `mgf-cloud`. Companion: the build tracker
> [`../inprogress/mgf-cloud-progress.md`](../inprogress/mgf-cloud-progress.md).
> **Status: living** — updated as each pass lands.

## 1. Why mgf-cloud exists

The federation needs to **self-host Woodpecker CI** (Codeberg's shared CI is
approval-gated and denied `magogi-admin`), which requires an Infomaniak VM. More
broadly, the federation had **no programmatic way to manage cloud resources** —
provisioning was hand-run `openstack` CLI plus Terraform/Ansible living outside
the fleet. `mgf-cloud` is the first-class, conformant **interface to create and
manage cloud VMs and storage** (Infomaniak Public Cloud first; AWS-ready), with a
CLI and a local web GUI. `mgf-fed` is the sole consumer and the owner of the
resources; `mgf-cloud` is the mechanism. WF-11 (self-hosted Woodpecker) resumes
on top of it (Pass 6).

### 1.1 Ownership boundary — where cloud access lives (and where it must not)

This is a hard architectural boundary, not a convention:

- **`mgf-cloud` is the *only* library in the federation that accesses Infomaniak /
  OpenStack compute resources.** It is the sole importer of `openstacksdk` (via the
  `[infomaniak]` extra) and the sole reader of OpenStack credentials. Verified
  2026-06-15: `mgf-cloud` is the only repo whose `src/` references
  `openstack` / `infomaniak` / `clouds.yaml` / `OS_*`.
- **The credential never leaves `mgf-cloud`.** The live `clouds.yaml`/`openrc`/
  password + the consolidated `FACTS.md` live in **`mgf-cloud/my_stuff/infomaniak/`**
  (gitignored) — co-located with the code that owns them. No other repo holds a copy.
- **`mgf-fed` does *not* access Infomaniak.** It manages the *dev-infra* cloud
  (Codeberg git hosting + Woodpecker CI) — a different cloud, reached over its own
  Codeberg token, never via OpenStack. When `mgf-fed` needs a VM it calls `mgf-cloud`
  through the runtime `[cloud]` extra (§9); it never imports `openstacksdk` and never
  reads a `clouds.yaml`. The two clouds are kept strictly separate.
- **One door in.** All OpenStack auth flows through `config.py` (§5) → a typed
  `Credentials`; all OpenStack *calls* flow through `providers/infomaniak.py` behind
  the `CloudProvider` Protocol (§4). Nothing else opens a session. An
  `import-linter` contract keeps the core SDK-free (§3.2).

## 2. Design decisions

| Aspect | Decision | Rationale |
|---|---|---|
| **Scope** | IaaS primitives (VM / block / object / network) + cloud-init + **SSH remote-bootstrap** | Enough to stand a host fully up (install Docker, drop a compose bundle, bring it up). App-deploy (Postgres/Redis roles) is out of scope. |
| **Architecture** | Provider-agnostic **core** + per-provider **adapters**; Infomaniak first, AWS-ready | The explicit multi-provider intent; one interface, many backends. |
| **Dep layering** | Core light (mgf-common + pydantic + pyyaml); heavy SDKs in extras (`[infomaniak]`/`[s3]`/`[aws]`/`[remote]`/`[web]`) | `import mgf.cloud` pulls no SDK; provider choice is explicit; matches the fleet extras idiom. |
| **State** | **Stateless / live-query** + `mgf:owner` resource tagging | The cloud is the source of truth; no state file to drift or lose. |
| **mgf-fed link** | **Runtime import** via a `mgf-fed[cloud]` extra (`mgf-cloud[infomaniak]`) + graceful degrade | Tight typed integration without forcing openstacksdk onto every mgf-fed install. |
| **Web GUI** | **Local single-user** (`mgf-cloud web`, 127.0.0.1, no auth), FastAPI via mgf-fastapi | Fast + safe (no public surface managing live cloud); hosted/multi-user deferred. |
| **Object storage** | Swift primary (covered by app-creds); S3 optional (`[s3]`, EC2 creds) | Swift needs no extra credential. |
| **Auth** | OpenStack v3 via `clouds.yaml` / `OS_*` / `my_stuff`; `config.py` accepts **either** username/password (`v3password`) **or** Application Credential (`v3applicationcredential`) and auto-selects | Infomaniak = OpenStack. The live `mgf-fed` project is provisioned with **username/password** (what the dashboard's `clouds.yaml` ships); Application Credentials are the recommended hardening and are also accepted — no code change to switch. |
| **Guardrails** | create/delete require `--yes`/confirm; `--dry-run` everywhere; secrets never logged; web localhost-only; TLS verify on | It manages billable, live infrastructure. |

## 3. Architecture

### 3.1 Package layout (`src/mgf/cloud/`)

| Module | Responsibility | Pass |
|---|---|---|
| `__init__.py` | version + the thin public surface | 1 |
| `__main__.py` + `cli/` | the `mgf-cloud` argparse CLI (dispatch + subcommands) | 1 / 4 |
| `models.py` | typed resource models (Server, Flavour, Image, Keypair, SecurityGroup/Rule, FloatingIP, Network, Volume, VolumeSnapshot, ObjectContainer, ObjectInfo) + `CloudConfig`/`Credentials` | 2 |
| `config.py` | load credentials from `clouds.yaml` / `OS_*` env / `my_stuff` token (pydantic-settings) | 2 |
| `exceptions.py` | error types mapped onto `mgf-common`'s `AppError` hierarchy | 2 |
| `providers/base.py` | the `CloudProvider` Protocol (compute/block/object/network) + a `FakeProvider` for tests | 2 |
| `providers/__init__.py` | `get_provider(name)` factory — lazy import of the adapter | 2 |
| `providers/infomaniak.py` | the openstacksdk adapter (`[infomaniak]`) | 3 |
| `providers/aws.py` | AWS stub (`[aws]`) | 3 |
| `remote.py` | SSH exec / file-push + `bootstrap()` (`[remote]`) | 3 |
| `web/` | the local FastAPI GUI (`[web]`) | 5 |
| `defaults.py` | Infomaniak/Magogi defaults (§6) | 3 |

### 3.2 Layering — the core stays light

Importing `mgf.cloud` must never pull a provider SDK or the web framework. The
`providers/__init__.py` factory `get_provider("infomaniak")` does an
`importlib`-style lazy import of the adapter, so openstacksdk loads only when an
Infomaniak provider is actually constructed. An **import-linter** contract
(`.importlinter`) enforces the closed box; the per-module "core imports no SDK"
contract is added in Pass 2 once `models`/`config`/`providers.base` exist.

### 3.3 Dependency layering

- **Core (always):** `mgf-common`, `pydantic`, `pydantic-settings`, `pyyaml`.
- **`[infomaniak]`:** `openstacksdk` (compute/block/Swift/network).
- **`[s3]` / `[aws]`:** `boto3` (S3-compatible object storage / AWS).
- **`[remote]`:** `paramiko` (SSH bootstrap).
- **`[web]`:** `fastapi`, `uvicorn`, `jinja2`, `mgf-fastapi`.
- **`[standards]`:** `mgf-common[standards]` (corpus for the federation CLI).

## 4. Provider model

One `CloudProvider` Protocol defines the operations (compute, block storage,
object storage, networking). `get_provider(name, config)` returns the concrete
adapter. Adapters are **stateless**: every call queries the live cloud. Resources
the federation creates are tagged `mgf:owner=<value>` (OpenStack server metadata /
object metadata) so mgf-cloud can identify "its" resources without a local state
file. A `FakeProvider` (in-memory) backs unit + CLI tests so the suite needs no
live cloud.

## 5. Credentials & config

Infomaniak Public Cloud speaks OpenStack Keystone (Identity v3). `config.py` is the
**single** credential resolver — it accepts **either** of the two v3 methods and
picks the right `auth_type` itself (`Credentials.auth_type`: `v3applicationcredential`
when an app-cred id+secret is present, else `v3password`):

- **username/password** — what Infomaniak's dashboard `clouds.yaml`/`openrc` ships; the
  current `mgf-fed` project uses this.
- **Application Credential** (id + secret) — recommended hardening; create it in
  Horizon and drop it into the same `clouds.yaml`. No code change to switch.

Resolution order:

1. an explicit `clouds.yaml` path (the file Infomaniak's dashboard hands you);
2. `$MGF_CLOUD_CLOUDS_YAML` pointing at that file;
3. the standard OpenStack `OS_*` environment (so a sourced `openrc.sh` just works);
4. the standard `clouds.yaml` search paths (`~/.config/openstack/`, `/etc/openstack/`, `./`).

> **Cloud-name caveat (not an assumption — verified against the live file):**
> `from_clouds_yaml(path, cloud=...)` defaults `cloud="openstack"`, but Infomaniak's
> downloaded file names its entries **`PCP-<id>-<region>`** (e.g. `PCP-HOESFJV-dc3-a`,
> DR `PCP-HOESFJV-dc4-a`) — **not** `openstack`. Pass the cloud name explicitly, or
> rename the top key. The live account's exact identity (user, project UUID, region,
> endpoint, the two Manager-vs-OpenStack IDs) is recorded **once** in
> `my_stuff/infomaniak/FACTS.md` (gitignored) — the single source of truth; nothing
> is hard-coded or guessed elsewhere.

Object storage S3 (optional) uses separate EC2 credentials (`openstack ec2
credentials create`). Credentials are `SecretStr`, never logged and never serialised
into errors or the web GUI.

## 6. Infomaniak domain defaults (mined from the ops playbooks)

From `info_store/playbooks/tech/infomaniak_vm.md` (the Magogi VM playbook) +
`PROCESSING_LATER/django-infomaniak-deploy` (Terraform/Ansible). These seed
`defaults.py` and the adapter. **The auth_url, region, image, flavour, and network
rows below are confirmed against the live `mgf-fed` project** (2026-06-15 — a nano VM
was created → reached ACTIVE → destroyed via the adapter); the remaining rows stay as
defaults to confirm before relying on:

| Decision | Default |
|---|---|
| Keystone auth_url | `https://api.pub1.infomaniak.cloud/identity/v3` |
| Region | `dc3-a` (Geneva); DR `dc4-a` |
| OS image | `Ubuntu 24.04 LTS Noble Numbat` (live image name — carries the codename) |
| Flavour (small service) | `a1-ram4-disk20-perf1` (1 vCPU / 4 GB / 20 GB — the smallest live shape; there is no 2 GB flavour); scale `a2-ram4-disk20-perf1` |
| Network | `ext-net1` — **a direct-public network: VMs get a routable public IPv4 + IPv6 on boot** (confirmed live 2026-06-15, e.g. `195.15.x.x`, type `fixed`) |
| Floating IPv4 | **not needed on `ext-net1`** (the instance IP is already public). FIPs are for private-network topologies, allocated from the external pool `ext-floating1` (allocating from `ext-net1` is a 400 "not a valid external network") |
| IPv6 | assigned on boot (free) |
| Volume | persistent, `Delete-on-Terminate = No` |
| SSH user | `ubuntu` (image default), key-only |
| Hostname pattern | `<service>-<env>` (e.g. `pypi-prod`) |
| cloud-init | the org-hardening template (unattended-upgrades, ufw, fail2ban, SSH lockdown, sysctl, journald cap, reboot 03:30 Europe/Zurich) |
| Security group | mirror ufw; open service ports (80/443) + 22; **block egress :25** (no direct mail) |
| Object storage | Swift (native); S3 endpoint `https://s3.pub1.infomaniak.cloud/` |
| Approval | **two-person** for prod create/delete/floating-IP/resize/key-rotation |

The two-person-approval norm maps to mgf-cloud's `--yes`/confirm guardrail on
create/delete and to `--dry-run` plan-before-apply.

## 7. CLI surface (`mgf-cloud`, Pass 4)

`provider` · `vm {create,list,show,delete,reboot,resize,snapshot}` · `volume
{create,list,attach,detach,delete,snapshot}` · `object {container,put,get,ls,rm}`
· `net floating-ip {…}` · `secgroup {…}` · `keypair {…}` · `remote {exec,push}` ·
`bootstrap <host>` · `web`. Global flags: `--json`, `--dry-run`, `--yes`. Creating
or deleting requires `--yes` or an interactive confirmation; secrets are never
printed.

## 8. Web GUI (Pass 5)

`mgf-cloud web` launches a **localhost**, single-operator FastAPI app (via the
mgf-fastapi adapters — request-id, exception translation, lifespan) with Jinja
templates: a dashboard (VMs / volumes / floating-IPs / containers), VM
detail/create/delete/reboot, volume attach/detach, and an object browser. It
calls the same core API as the CLI. No auth and no public bind in v1 — it is an
operator's local console, not a hosted service.

## 9. mgf-fed integration (Pass 6)

`mgf-fed` gains a `[cloud]` extra (`mgf-cloud[infomaniak]`) and imports the
`mgf.cloud` API (graceful degradation when the extra is absent). A new
`mgf-fed provision-ci-host` command provisions the Woodpecker host end-to-end:
create the VM (with the Docker-installing cloud-init) → attach a data volume →
allocate + attach a floating IP → open security-group ports 80/443/22 → deploy
the `mgf-fed/deploy/woodpecker/` compose bundle (cloud-init or `remote`) → bring
it up. WF-11 then resumes: enable repos on the self-hosted Woodpecker,
branch-protection, pilot-verify, fleet.

## 10. Guardrails & safety

- `--dry-run` on every mutating command; `--yes`/confirm required to create or delete.
- Credentials are read only from `clouds.yaml`/`OS_*`/`my_stuff`; never logged, never in tracebacks or the GUI.
- openstacksdk TLS verification stays on; the web GUI binds `127.0.0.1` only.
- Adapters clean up partial failures and never leak billable resources (verified by the Pass 7 live teardown smoke).

## 11. The 7-pass build

Tracked in [`../inprogress/mgf-cloud-progress.md`](../inprogress/mgf-cloud-progress.md):
①scaffold+conformance → ②core abstraction → ③Infomaniak adapter → ④CLI →
⑤web GUI → ⑥mgf-fed integration (resume WF-11) → ⑦review pass + release.
