# Deployment

> **Conformance:** L1 / L2 / L3 (see `README.md`)
> **Audience:** Human engineers AND AI coding agents.
> **Propagation (DOC-18):** changing a rule here ripples — walk DOCUMENTATION.md §10.4 and update its dependents in the same commit (README *Index of MUSTs*, the `mgf-fed standards` / `agents` CLI, and any parallel copies / pins the matrix names).

> **Scope.** Cross-project deployment standard for **stateful
> web-service consumer applications** (Django, FastAPI, Flask,
> async workers, scheduled jobs). The rules describe how a
> web service goes from a git tag to a running production
> environment — infrastructure provisioning, VM configuration,
> release rollout, secret management, observability wiring,
> rollback discipline. Operability of the running service is
> covered separately by [`OPERABILITY.md`](OPERABILITY.md);
> the two standards are complementary (deployment is "how do
> you get bits onto servers"; operability is "how do you run
> them once they're there"). See **§9 Cross-references** for
> the explicit overlap map.
>
> **Out of scope.** Federation libraries (no runtime — see
> the federation-sibling shape in [`DOCUMENTATION.md`](DOCUMENTATION.md) §2.0).
> Desktop applications (distributed as binaries — see the
> consumer-application shape). Static websites — addressed
> as a thin sub-section in §8.2.
>
> **Cloud agnostic.** The rules describe WHAT must be true
> (infra is IaC, config is code, artefacts are immutable,
> deploys are reversible). The HOW is tool-agnostic:
> Terraform / Pulumi / OpenTofu / CDK for IaC; Ansible /
> Chef / Salt / cloud-init for config-as-code. The Magogi
> foundation's main reference platform is **Infomaniak
> Public Cloud** (§8.1) — but the rules apply identically
> on AWS, Azure, GCP, Fly.io, Render, etc.

---

## Rules box (read this first)

| ID | Rule | Applies to |
|---|---|---|
| <a id="DEP-01"></a>**DEP-01** | **Cloud infrastructure MUST be Infrastructure-as-Code.** Every cloud resource (VM, network, load balancer, bucket, security group, DNS record, certificate) MUST be declared in version-controlled IaC and applied via the same automation in every environment. Manual cloud-console clicks ("just one more security-group rule") are a defect — they bypass review, audit trail, and the rebuild path. Reference tools: Terraform, Pulumi, OpenTofu, AWS CDK. | service |
| <a id="DEP-02"></a>**DEP-02** | **VM / container state MUST be Configuration-as-Code.** Every package installed, service configured, file written on a production VM (or container image) MUST be expressed in idempotent automation. No `ssh prod-vm && apt install …` allowed; no "I'll fix it in the VM and document later." Reference tools: Ansible, Chef, Salt, cloud-init, Dockerfile + a build pipeline. | service |
| <a id="DEP-03"></a>**DEP-03** | **Release artefacts MUST be immutable.** Same git SHA → byte-identical deployable artefact. No `pip install --upgrade` at runtime in production; no resolving package versions from any package registry on each deploy — neither public PyPI for third-party deps nor the devpi index for `mgf-*` packages (per [`PACKAGE_DISTRIBUTION.md`](PACKAGE_DISTRIBUTION.md) **PKG-07** version-immutability). Dependencies MUST be fully locked at build time (`uv.lock` / `pip-tools` `*.lock` / container image digest), so a deploy installs only from the lock with no index lookup. The artefact's identity is the contract; mutability is the defect. | service |
| <a id="DEP-04"></a>**DEP-04** | **Production secrets MUST live outside the repo, encrypted at rest.** DB credentials, signing keys, API tokens, TLS private keys, OAuth secrets — none of these in committed files. Acceptable storage: Ansible Vault, sealed secrets, cloud KMS / Secrets Manager / Key Vault, HashiCorp Vault. Per **SC-01**. The decryption key (Ansible vault password, KMS access role) MUST itself live outside the repo. | service |
| <a id="DEP-05"></a>**DEP-05** | **Deploy commands MUST be environment-explicit.** Every deploy / backup / restore / rollback script MUST take an environment argument (`prod`, `staging`, `dev`) and MUST refuse to run with no environment. `./deploy.sh` with no arg is a defect — bare commands are how someone deploys staging to prod at 02:00. The scripts MUST be idempotent (re-running the same deploy command MUST converge, not break). | service |
| <a id="DEP-06"></a>**DEP-06** | **Deploys MUST be reversible with a documented, drill-measured, bounded rollback time stated in the runbook.** Every release MUST have a documented rollback path: blue/green, rolling-with-keep-previous, atomic-symlink-with-previous-version, container-image-tag-swap, snapshot-restore, or equivalent. The rollback procedure MUST be a numbered runbook (3–7 steps), exercised at least once on a non-production target, with the measured rollback time recorded as an explicit bound. Instant-cutover shapes (blue/green, symlink-swap, tag-swap) SHOULD meet ≤5 minutes from decision to restored state; a slower shape (snapshot-restore) MAY state a higher justified bound, as long as the bound is drill-measured and written down. Per **OP-10**. | service |
| <a id="DEP-07"></a>**DEP-07** | **The `deploy/` directory MUST follow the standard shape.** Every web-service consumer MUST organise deployment artefacts in a top-level `deploy/` directory with (at minimum): a `README.md` entry point, at least one IaC sub-directory (`terraform/` or equivalent), at least one config-management sub-directory (`ansible/` or equivalent — skip only if the platform is fully managed-service with no VM ownership), a `scripts/` sub-directory for operator tooling, a `monitoring/` sub-directory for dashboards + alert rules, and a `docs/` sub-directory for the phase-by-phase reference + runbook + SLO. See §3 for the canonical layout. | service |
| <a id="DEP-08"></a>**DEP-08** | **Production MUST be observable end-to-end.** Metrics, logs, traces MUST flow to a sink the operator can query without SSH-ing into a VM. The SLI/SLO contract MUST be a committed file (`deploy/docs/slo.md`); error budget + burn-rate alerts documented. Alerts MUST route to a human paging channel (ntfy, Slack, PagerDuty) and EACH alert MUST link a runbook entry per **OP-11**. | service |
| <a id="DEP-09"></a>**DEP-09** | **DNS + TLS MUST be managed infrastructure.** DNS records MUST live in IaC OR a clearly-documented external manager with the record list committed (e.g., Cloudflare DNS with `deploy/docs/dns.md` listing every record). TLS certificates MUST be auto-renewed (Let's Encrypt, cloud-managed, or equivalent) — manual cert rotation is a defect. Certificate-expiry monitoring is required. | service |
| <a id="DEP-10"></a>**DEP-10** | **A first-production-cut runbook MUST exist.** `deploy/docs/first-production-cut.md` (or equivalent) MUST document the green-field provisioning sequence end-to-end — every IaC apply, every config-management playbook, every manual step that cannot be automated — in execution order with prerequisites and verification steps per phase. The runbook MUST be exercised at least once before go-live. Updated at each major architectural change. | service |
| <a id="DEP-11"></a>**DEP-11** | **Multiple environments MUST be deployable from the same code.** Every web-service consumer MUST be deployable to at least two environments (production + at minimum one of staging / dev) from the same IaC + config-management code, parameterised by an environment file. Manual divergence between environments ("prod has nginx config X; staging has Y") is a defect. The second environment is the smoke-test surface for the deploy itself. | service |

---

## 1. Mental model — deployment is the contract between code and reality

Code that runs on a developer's laptop is **a working
demo**. Code that runs in production is **a service**.

The difference is everything that lives in `deploy/`:
the VM provisioning, the OS hardening, the DB tuning, the
TLS certificate, the load balancer health check, the
secret-rotation procedure, the rollback path, the
backup-restore drill, the on-call runbook. None of that
content is in the application source tree — it lives next
to it, in a clearly-named directory, and it ships with
the same version-control discipline as the code.

The rules in this document collapse to one principle:
**every piece of production state has a documented,
automated, version-controlled path from git to running.**
That includes the obvious (the application code) and the
non-obvious (which DNS records exist, what cron jobs run,
which security-group rules allow what traffic, what the
monitoring agent's configuration file looks like). If a
human had to make a decision at 02:00 to recreate it,
that decision is documentation — not in someone's head.

**The four orthogonal concerns** carry across the
reliability + security + performance + operability framing
from `OPERABILITY.md` §1, but at the deployment layer:

- **Reliability** — can the deploy itself fail without
  taking the service down? (Atomic releases, rollback,
  health checks.)
- **Security** — is every secret encrypted at rest?
  Every private network actually private? Every cert
  auto-renewed?
- **Performance** — does the deploy preserve warm caches /
  long-lived connections / database pool state? Is there
  a measurable "deploy → first request" budget?
- **Operability** — when an alert pages at 02:00, can the
  operator find the runbook, the dashboard, and the
  rollback command in <60 seconds?

The bar is mechanical: every cloud resource has a name in
IaC, every VM state-change has a playbook line, every
release has a previous-version tag to roll back to, every
alert has a runbook URL, every secret has a documented
rotation procedure.

---

## 2. The shape of a deployed service

### 2.1 What gets deployed where

A typical Magogi web-service consumer has these moving pieces:

| Layer | What lives here | Typical Infomaniak shape | Typical AWS shape |
|---|---|---|---|
| **Edge** | DNS, TLS termination, WAF, DDoS, CDN | Cloudflare (DNS + edge TLS + WAF) | Route 53 + CloudFront + WAF |
| **Load balancer** | HTTP/HTTPS termination, health checks, round-robin | Octavia LB | ALB |
| **Application VM(s)** | ASGI/WSGI server, background workers, scheduled tasks | OpenStack VM running Uvicorn + Celery via systemd | EC2 + ASG or ECS / EKS |
| **Database VM** | Postgres + connection pooler | OpenStack VM running PostgreSQL + PgBouncer | RDS or Aurora |
| **Cache / queue VM** | Redis (cache + Celery broker + WebSocket channel layer) | OpenStack VM running Redis 7 | ElastiCache |
| **Object storage** | Media uploads, static assets, DB backups | Infomaniak Object Storage (S3-compatible) | S3 |
| **Email** | Transactional SMTP | mail.infomaniak.com:587 | SES |
| **Observability** | Metrics, logs, traces, dashboards, alerts | Grafana Cloud + Promtail + Prometheus | CloudWatch + X-Ray + Managed Grafana |

The reference architecture for stateful Django/FastAPI apps
is **multi-VM with a private network** (App VM + DB VM +
Cache VM, all on a private subnet, with only the LB
publicly addressable). Single-VM is acceptable for L1 /
small consumers; managed Kubernetes is overkill until a
project crosses ~50 deploys/week or needs multi-region.

### 2.2 What this rules out (and why)

- **No "just SSH and edit a config file."** A change that
  isn't in IaC or config-as-code doesn't survive a VM
  rebuild. Rule **DEP-01** / **DEP-02**.
- **No public DB.** A PostgreSQL VM with port 5432 open to
  the internet is a defect even with strong passwords —
  bypassing the LB removes the only audited ingress point.
- **No `:latest` tag in production.** Same reason as
  **OP-10** — rollback requires a previous-version tag to
  roll back TO.
- **No long-lived SSH sessions during a deploy.** A deploy
  that requires "I left the SSH window open running tail
  -f" is not a deploy; it's a debugging session. Deploy
  scripts MUST run to completion + return an exit code.

---

## 3. The `deploy/` directory shape (rule DEP-07)

Every web-service consumer organises deployment artefacts
in a top-level `deploy/` directory:

```
deploy/
├── README.md                      ← entry point (quick-start + layout + conventions)
├── CLOUD_vs_LOCAL.md              ← component-by-component dev↔prod comparison (recommended)
├── APPENDIX.md                    ← misc operator references (optional)
├── docs/                          ← phase-by-phase reference
│   ├── README_DEPLOY.md           ←   start here for the full architecture + phases
│   ├── runbook.md                 ←   on-call incident triage (per OP-11)
│   ├── slo.md                     ←   SLO target + burn-rate alerts (per DEP-08)
│   ├── first-production-cut.md    ←   green-field provisioning sequence (per DEP-10)
│   ├── phase-1-infra.md           ←   IaC: VMs, networks, LBs, security groups
│   ├── phase-2-vm-config.md       ←   config-as-code: OS hardening, DB, app server
│   ├── phase-3-app-deploy.md      ←   DNS, TLS, first live deploy, smoke test
│   ├── phase-4-monitoring.md      ←   observability wiring (per DEP-08)
│   └── phase-5-operations.md      ←   deploy, rollback, backup, restore, scaling
├── terraform/                     ← IaC (or pulumi/, cdk/, etc.)
│   ├── README.md
│   ├── modules/                   ←   reusable: compute, lb, networking, security_groups, object_storage
│   └── environments/
│       ├── prod/
│       └── staging/               ← (or dev/) — per DEP-11
├── ansible/                       ← config-management (or chef/, salt/, etc.)
│   ├── README.md
│   ├── ansible.cfg
│   ├── inventory/
│   │   ├── hosts.ini              ←   per-environment inventory
│   │   └── group_vars/
│   │       └── all.vault.yml      ←   ansible-vault encrypted (per DEP-04)
│   ├── playbooks/                 ←   site.yml + per-role playbooks
│   └── roles/                     ←   one per service (postgresql, redis, app, nginx, …)
├── monitoring/                    ← observability assets (per DEP-08)
│   ├── README.md
│   ├── dashboards/                ←   Grafana JSON exports
│   └── alert-rules/               ←   Prometheus / Grafana alert rules
└── scripts/                       ← operator-side bash/python (per DEP-05)
    ├── README.md
    ├── deploy.sh                  ←   environment-explicit; refuses bare invocation
    ├── health-check.sh            ←   60-second triage probe per OP-02 / OP-06
    ├── backup-db.sh               ←   on-demand DB snapshot (per OP-12)
    ├── restore-db.sh              ←   restore from a snapshot (per OP-12)
    └── open-tunnel.sh             ←   SSH tunnel to private DB (last-resort access)
```

The reference implementation is **[`django-infomaniak-deploy`](https://codeberg.org/magogi-admin/django-infomaniak-deploy)** —
a reusable framework that produces this exact shape via a
`setup.sh` bootstrap step. A new web-service consumer
SHOULD start there.

**Project-shape mapping** (from [`DOCUMENTATION.md`](DOCUMENTATION.md) §2.0):

- `deploy/` lives at the **repo root** of a consumer
  application — it is NOT a `docs/` sub-directory. The
  audience separation is "operator looking at deploy" vs
  "contributor looking at code" (per the `CLOUD_vs_LOCAL.md`
  pattern).
- A federation sibling that grows its own deploy footprint
  (rare) MAY use the same shape, but the sibling's
  deployment is usually its own integration tests in CI —
  not a production service.
- A consumer that runs on a managed-service platform
  (Fly.io, Render, Heroku, Vercel) MAY have a thinner
  `deploy/` with platform-specific config files
  (`fly.toml`, `render.yaml`, `Procfile`) instead of
  `terraform/` + `ansible/` — DEP-07's "config-management
  sub-directory" is skippable when the platform owns the
  VM. The README + runbook + SLO + scripts subset still
  applies.

---

## 4. The rules in detail

### 4.1 DEP-01 — Infrastructure-as-Code

Every cloud resource MUST be declared in version-controlled
IaC. The list to cover:

- **Compute** — VMs, container clusters, serverless functions.
- **Networking** — VPCs / VLANs, subnets, route tables,
  internet gateways, NAT.
- **Load balancers** — listeners, target pools, health
  checks.
- **Security** — security groups / firewall rules, IAM
  roles, KMS keys (the key resources, not their values).
- **Storage** — buckets, lifecycle rules, block volumes.
- **DNS** — A / AAAA / CNAME / MX records.
- **Certificates** — Let's Encrypt automation config, or
  ACM / cloud-managed cert records.

**The audit trail.** Every change to production infra
flows: developer commit → PR → CI plan-output review →
merge → CI apply. Manual changes in the cloud console
MUST surface as `terraform plan` drift on the next CI
run; drift is an incident, not normal operation.

**Per-environment parameterisation.** The same module
code applied with different `*.tfvars` produces prod and
staging. Module-level branches (`if env == "prod"`) inside
the module are an anti-pattern — push the conditional to
the variables.

### 4.2 DEP-02 — Configuration-as-Code

Every package installed on a production VM, every systemd
unit configured, every config file rendered MUST be in
config-as-code. Ansible roles are the reference shape:

```
ansible/roles/<service>/
├── defaults/main.yml      ← role variables with defaults
├── tasks/main.yml         ← idempotent task list
├── templates/             ← Jinja2 templates for config files
├── handlers/main.yml      ← service-restart hooks
└── README.md              ← what this role does + cross-refs
```

**The idempotency contract.** Running the playbook twice
in a row MUST produce zero changes the second time. An
Ansible run that reports `changed=0` is the green signal;
`changed=N` after a no-op deploy is a smell — usually a
template that re-renders with a different timestamp, or a
file mode that doesn't actually match.

**Don't reach for shell.** `apt install foo` in a `tasks`
file is wrong; use the `apt` module. `cp src dst` is
wrong; use `copy` or `template`. Shell tasks are a last
resort and MUST carry `changed_when:` / `creates:` /
`removes:` to remain idempotent.

### 4.3 DEP-03 — Immutable release artefacts

A release is identified by:

- A git SHA (the canonical identifier).
- A semver tag (the human label).
- An artefact — a container image digest, a wheel set, or
  a source tree at the SHA.

The artefact MUST be byte-identical across rollouts of the
same release. Concretely:

- **Container-image deploys:** the image MUST be pulled
  by digest (`sha256:…`), not by tag. Tag-based pulls can
  silently change.
- **Source-tree deploys:** the deploy script MUST check
  out a specific SHA, NOT `main` or `latest`. Dependency
  resolution MUST happen against a lock file checked in
  to the repo (`uv.lock`, `requirements/*.lock`).
- **No package-registry lookups at deploy time.** The
  deploy step MUST install only from the lock — it MUST
  NOT resolve versions from *any* index, neither public
  PyPI (third-party deps) nor the devpi index
  (`mgf-*` packages). `pip install -r requirements.txt
  --no-deps` against a fully-locked file is fine. `pip
  install -r requirements.txt` against an unlocked file —
  or any `uv`/`pip` invocation that re-resolves against
  an index at deploy time — is a defect. The lock is
  produced once at build time (when an index lookup is
  legitimate); the deploy consumes it offline. Because
  `mgf-*` registry versions are immutable by policy (per
  [`PACKAGE_DISTRIBUTION.md`](PACKAGE_DISTRIBUTION.md)
  **PKG-07**), the pinned version in the lock is
  guaranteed byte-stable — so pre-resolving at build time
  and never touching the registry at deploy time is both
  safe and the rule.

**Why this matters.** Two deploys of the "same version"
that produce different running code is a class of bug
that costs days to debug. The lock-file + SHA discipline
makes that bug class structurally impossible.

### 4.4 DEP-04 — Secrets management

Production secrets MUST be encrypted at rest. The
acceptable storage backends:

| Backend | Use when | Per **SC-01** notes |
|---|---|---|
| **Ansible Vault** | Single-team operator workflow; secrets live in `inventory/group_vars/all.vault.yml`. Decryption password lives outside the repo (operator workstation `~/.vault_pass` + CI secret-store env var). | Workable for L1; the encryption is AES-256, vault password rotation is a deliberate procedure. |
| **Cloud KMS / Secrets Manager / Key Vault** | Multi-team / regulated / large-scale. Secrets stay in the cloud's encrypted store; app reads at startup via IAM-scoped role. | Required for L2 in regulated environments. The IAM role replaces the vault password as the trust anchor. |
| **HashiCorp Vault** | Self-managed equivalent of cloud KMS. | Valid; documented operator-rotation procedure required. |
| **Sealed Secrets (Kubernetes)** | k8s-shaped projects. Per-cluster public key encrypts; controller decrypts in-cluster. | Valid; covered in §8 for the k8s shape. |

**Forbidden:**
- Plain-text production secrets in any committed file.
- `.env` files committed to the repo with real production
  values (even encrypted with a key in the same repo).
- Secrets passed as command-line arguments to a process
  (visible in `ps aux`).
- Secrets baked into a container image.

**The rotation procedure** MUST be documented in
`deploy/docs/phase-5-operations.md` (or equivalent) with
the steps + verification — covered by **OP-11** runbook
discipline.

### 4.5 DEP-05 — Environment-explicit deploy commands

Every operator script MUST take an explicit environment
argument:

```bash
deploy/scripts/deploy.sh prod              # OK
deploy/scripts/deploy.sh staging           # OK
deploy/scripts/deploy.sh                   # MUST refuse — no default
deploy/scripts/restore-db.sh prod <snap>   # OK
deploy/scripts/restore-db.sh <snap>        # MUST refuse — destructive ops need env
```

The mechanical shape (bash):

```bash
#!/usr/bin/env bash
set -euo pipefail

if [[ $# -lt 1 ]]; then
    echo "usage: $0 <env> [args...]" >&2
    echo "  env: prod | staging | dev" >&2
    exit 2
fi

ENV="$1"; shift
case "$ENV" in
    prod|staging|dev) ;;
    *) echo "unknown env: $ENV (allowed: prod, staging, dev)" >&2; exit 2 ;;
esac

# ... environment-aware deploy logic ...
```

**Idempotency.** Re-running the same deploy command MUST
either:
- Detect that the target SHA is already deployed and exit
  with a clear "no-op" message, OR
- Re-run the deploy steps idempotently (Ansible's
  changed=0 contract).

A deploy script whose second run breaks something is a
defect.

### 4.6 DEP-06 — Reversible deploys

Every release MUST have a documented rollback path. The
canonical shapes:

- **Blue/green.** Two identical environments; the LB
  swaps traffic from blue to green; rollback is the
  reverse swap. Heaviest shape; cleanest rollback (≤30s
  if cutover is instantaneous).
- **Rolling with keep-previous.** N+1 instances; new
  version deployed to one at a time; old version kept
  until verification passes. Rollback redirects traffic
  to the kept instances.
- **Atomic-symlink-with-previous.** Single-VM shape;
  `current` symlink swaps between `releases/<sha>` dirs;
  rollback is `ln -snf releases/<previous-sha> current
  && systemctl restart`. Reference shape for L1
  single-VM Django deploys.
- **Container-image-tag-swap.** Container runtime;
  rollback is `docker pull old:digest && docker
  service update --image old:digest`.

**The rollback procedure** MUST be a numbered runbook,
3–7 steps, with the time bound documented:

```markdown
## Rollback procedure

Time bound: ≤5 minutes from decision to restored state.
Pre-tested: 2026-04-15 against staging.

1. Identify the previous version: `cat /opt/<app>/releases/PREVIOUS_SHA`
2. Swap the symlink: `ln -snf releases/<sha> /opt/<app>/current`
3. Restart services: `sudo systemctl restart <app>.service celery.service`
4. Verify HTTP 200: `curl -sf https://<host>/healthz`
5. Verify worker: `celery -A <app> inspect ping`
6. Announce in #ops: "rolled back to <sha> at <timestamp>"
7. File the incident: `deploy/docs/incidents/<date>-<slug>.md`
```

**Database migrations** are the load-bearing edge case.
A migration that drops a column makes the previous
version's code break — rollback then requires also
running the down-migration. The discipline:

- Migrations MUST be backward-compatible across the
  current + previous release where possible (add column
  with default; deploy code that writes it; remove old
  column in a later release).
- Migrations that cannot be backward-compatible MUST
  carry an explicit "non-rollback-safe" tag in the
  CHANGELOG and the cutover guide. The rollback path is
  then "restore from DB snapshot" — still DEP-06-conformant
  PROVIDED that snapshot-restore path is documented as a
  numbered runbook with its own drill-measured time bound
  (which will exceed the ≤5-minute instant-cutover target).
  An undocumented or undrilled snapshot-restore is the
  defect — not the longer time itself.

### 4.7 DEP-07 — The `deploy/` directory shape

Covered in detail in §3. The audit shape:

```bash
# A web-service consumer MUST have:
test -f deploy/README.md
test -d deploy/scripts
test -f deploy/scripts/deploy.sh
test -d deploy/docs
test -f deploy/docs/runbook.md
test -f deploy/docs/slo.md
# IaC sub-directory (one of):
[ -d deploy/terraform ] || [ -d deploy/pulumi ] || [ -d deploy/cdk ]
# config-management sub-directory (one of — skip if fully managed-service):
[ -d deploy/ansible ] || [ -d deploy/chef ] || [ -d deploy/salt ] || \
    test -f deploy/fly.toml || test -f deploy/render.yaml
```

### 4.8 DEP-08 — End-to-end observability

Production MUST export:

- **Metrics** — at minimum: request rate, error rate,
  p50/p95/p99 latency, CPU/memory/disk per VM.
- **Logs** — structured JSON, redacted per **LG-04**,
  forwarded to a queryable sink.
- **Traces** — OpenTelemetry from the app through to
  downstream calls, per **LG-03**.

The sink MUST be queryable without SSH access to the
VM. Acceptable sinks: Grafana Cloud, CloudWatch, Datadog,
Honeycomb, ELK, self-managed Loki + Prometheus +
Tempo.

**SLI/SLO contract.** A `deploy/docs/slo.md` MUST exist
and MUST define:

- One or more **SLIs** with their measurement procedure
  (the probe shape — what URL, what assertion, what
  cadence).
- A **monthly SLO target** (99.5% is a sensible starting
  point for single-VM deploys; 99.9% requires HA).
- An **error budget** = (1 − SLO) × window.
- **Burn-rate alerts** that page when the error budget
  burns faster than a documented threshold (1h ≫ budget
  burning at 14×; 6h ≫ 6×; standard SRE-shape).

**Runbook linkage.** Per **OP-11**, every alert rule
MUST carry an `annotations.runbook_url` pointing at a
`deploy/docs/runbook.md` section.

### 4.9 DEP-09 — DNS + TLS

**DNS records** MUST be one of:

- **In IaC** — Terraform `cloudflare_record` / `aws_route53_record`
  / equivalent. The DNS resources sit next to the LB +
  VM resources in the same plan.
- **External manager with a committed record list** —
  Cloudflare DNS UI is operator-friendly but bypasses
  IaC; the workaround is `deploy/docs/dns.md` enumerating
  every record with type / name / value / TTL /
  rationale. CI can lint this against a `cloudflare-cli`
  export.

**TLS certificates** MUST auto-renew. The reference
shapes:

- **Cloud-managed** — ACM (AWS), Cloud-managed cert
  (GCP), App-Service-managed (Azure), Cloudflare edge
  TLS (Universal SSL or Advanced Certificate Manager).
  Zero operator action; just verify cert expiry
  monitoring.
- **Let's Encrypt with cert-manager / acme.sh /
  certbot** — running on the LB or an app VM; renewal
  cron + monitoring.
- **Forbidden** — manually-uploaded long-lived certs
  (1-year purchased cert sitting on a VM with a calendar
  reminder to rotate). That's a defect — calendar
  reminders fail.

**Certificate-expiry monitoring** — a Prometheus rule
firing 14 days before expiry is the canonical shape.

### 4.10 DEP-10 — First-production-cut runbook

`deploy/docs/first-production-cut.md` documents the
green-field provisioning sequence. Mandatory sections:

1. **Prerequisites.** Cloud account credentials, IaC tool
   versions, vault password, DNS access, operator
   workstation tooling.
2. **Phase 1 — Infrastructure.** `terraform init` →
   `terraform plan` → `terraform apply` per environment.
3. **Phase 2 — VM configuration.** Ansible playbook
   run order (base → DB → cache → app → nginx).
4. **Phase 3 — First deploy.** First `deploy.sh prod`
   run, DNS cutover, TLS verification, smoke test.
5. **Phase 4 — Observability wiring.** Promtail/agent
   install, Grafana dashboard provisioning, alert routes
   tested.
6. **Phase 5 — Verification.** Synthetic-alert end-to-end
   test, backup-restore drill, rollback drill on staging.

The runbook MUST be exercised at least once before
go-live (typically against a "staging" environment that
is provisioned the same way and then torn down or
promoted to prod).

### 4.11 DEP-11 — Multiple environments from the same code

Every web-service consumer MUST be deployable to ≥2
environments — production + at least one of
staging / dev — from the same IaC + config-management
code, parameterised by an environment file.

```
deploy/terraform/environments/
├── prod/
│   ├── main.tf            ← module instantiation only; no logic
│   ├── variables.tf
│   ├── terraform.tfvars   ← env-specific values (gitignored if it has secrets)
│   └── backend.tf         ← prod state backend (separate from staging)
├── staging/
│   ├── main.tf            ← SAME module instantiation as prod
│   ├── variables.tf
│   ├── terraform.tfvars
│   └── backend.tf
```

The staging environment is the **deploy's own smoke
test** — every prod change ships first to staging, the
deploy is verified there, and only then promoted to
prod. Skipping staging is allowed for trivial
documentation-only changes but the discipline is
mechanical: a `prod-only` deploy is a defect-prone
shape.

**Cost-conscious staging.** Staging doesn't have to be
the same size as prod. A single-VM staging that mirrors
the structure (LB + app + DB) but uses smaller VMs is
sufficient for catching most deploy bugs. The
expensive-thing (multi-region replicas, very-large DB,
etc.) MAY be prod-only — but flag those as documented
"missing in staging" in the README so the operator
knows what isn't tested before prod cutover.

---

## 5. AS-IS — reference implementations

The Magogi foundation has two reference deployment
implementations in active use:

| Implementation | Role | Where it lives |
|---|---|---|
| **`django-infomaniak-deploy`** | Reusable framework — clone-as-starting-point for a new Django/FastAPI consumer. Produces the full §3 shape via `setup.sh`. | [`codeberg.org/magogi-admin/django-infomaniak-deploy`](https://codeberg.org/magogi-admin/django-infomaniak-deploy) |
| **`inthewords/deploy`** | First consumer of the framework — a Django web app deployed on Infomaniak Public Cloud. The §3 shape implemented end-to-end, plus the `CLOUD_vs_LOCAL.md` reference (component-by-component dev↔prod table). | `inthewords/deploy/` |

Status per rule:

| Rule | django-infomaniak-deploy | inthewords/deploy | Notes |
|---|---|---|---|
| DEP-01 IaC | ✅ Terraform (5 modules: compute, lb, networking, security_groups, object_storage) | ✅ inherited from framework | OpenStack provider for Infomaniak; AWS / Azure modules would be drop-in replacements at the same boundary. |
| DEP-02 Config-as-Code | ✅ Ansible (9 roles: app, base, deploy-user, monitoring-server, monitoring-node, nginx, pgbouncer, postgresql, redis) | ✅ inherited | Per-role README; idempotent. |
| DEP-03 Immutable artefacts | ✅ source-tree-at-SHA shape; pip-tools `requirements/*.lock` | ✅ pip-tools `requirements/{base,dev,prod}.lock` | Container-image shape is the recommended next step. |
| DEP-04 Secrets management | ✅ Ansible Vault (`inventory/group_vars/all.vault.yml`) | ✅ | Documented rotation procedure in `phase-5-operations.md`. |
| DEP-05 Env-explicit scripts | ✅ all scripts take `prod` / `staging` / `dev` arg | ✅ | `deploy.sh prod` shape is the canonical pattern. |
| DEP-06 Reversible deploys | ✅ atomic-symlink-with-previous-version (releases/`<sha>` dirs) | ✅ | 5-step rollback runbook in `runbook.md`. |
| DEP-07 `deploy/` shape | ✅ canonical (§3 above is extracted from this reference) | ✅ | |
| DEP-08 Observability | ✅ Grafana Cloud + Promtail + alert rules + SLO file | ✅ | 99.5% SLO; burn-rate alerts at 1h/6h windows. |
| DEP-09 DNS + TLS | ✅ Cloudflare DNS + edge TLS; auto-renewed | ✅ | DNS records listed in `phase-3-application-deploy.md`. |
| DEP-10 First-cut runbook | ✅ `phase-{1,2,3,4,5}-*.md` covers green-field | ✅ | |
| DEP-11 Multiple environments | ✅ `terraform/environments/{prod,dev}` | ✅ | Single staging VM, mirrors prod structure. |

---

## 6. Anti-patterns (forbidden)

| Anti-pattern | What's wrong | Right shape |
|---|---|---|
| "Just SSH and `apt install` real quick" | Change doesn't survive VM rebuild; bypasses review | Ansible task with explicit module (DEP-02) |
| `terraform apply` from a developer laptop with credentials | Untracked state; concurrent applies corrupt the state file | CI runs apply; state in remote backend (DEP-01) |
| `image: <project>:latest` in production manifest | No rollback target; pulls change silently | Tag with SHA or semver; pull by digest (DEP-03 + OP-10) |
| `.env.prod` committed with real values | Secret in version control | Ansible Vault / cloud KMS (DEP-04) |
| `./deploy.sh` with no arg defaults to prod | One-character typo deploys to prod | `./deploy.sh` refuses; explicit env required (DEP-05) |
| "Rollback procedure: redeploy the old git tag and pray" | Untested + no stated bound + risks DB-migration-induced corruption | Documented, drill-measured, bound-stated runbook; ≤5 min target for instant-cutover, a justified higher bound for snapshot-restore (DEP-06) |
| `deploy/` is just a single 2000-line `deploy.sh` script | No layering; no IaC; no config-as-code | Layered: terraform/ + ansible/ + scripts/ + docs/ + monitoring/ (DEP-07) |
| Logs go to `journalctl` only; operator SSHs to grep | Operator can't query; loses context after rotation | Forward to centralised sink (DEP-08) |
| TLS cert with a calendar reminder to rotate next year | Reminders fail | Auto-renew via Let's Encrypt or cloud-managed (DEP-09) |
| "We figured out the prod deploy by trial-and-error; nobody wrote it down" | Bus-factor 1; loss of any operator strands the project | first-production-cut.md walks the sequence (DEP-10) |
| Staging exists but is hand-tuned to match prod | Divergence accumulates; "works in staging" stops meaning anything | Same code + tfvars (DEP-11) |

---

## 7. Self-review checklist (every deploy-touching PR)

- [ ] **DEP-01** — every new cloud resource lives in IaC; no console-only changes.
- [ ] **DEP-02** — every new VM-side change lives in a role / playbook; idempotent.
- [ ] **DEP-03** — the release lock file (`uv.lock` / `requirements/*.lock`) is regenerated and committed; no `--upgrade` at deploy time.
- [ ] **DEP-04** — no new plaintext secrets; new secrets land in vault / KMS; rotation procedure updated.
- [ ] **DEP-05** — new scripts take an env arg; refuse bare invocation; pass `shellcheck`.
- [ ] **DEP-06** — every breaking-data-shape change ships with a non-rollback-safe tag OR a backward-compatible migration plan.
- [ ] **DEP-07** — `deploy/` shape still complete; new sub-buckets carry a README charter.
- [ ] **DEP-08** — new endpoints have an SLI probe; new background jobs have a metric; new alerts link a runbook.
- [ ] **DEP-09** — DNS changes documented; cert-expiry monitoring covers new hosts.
- [ ] **DEP-10** — `first-production-cut.md` covers any new phase step.
- [ ] **DEP-11** — change deployed to staging first; verified there before prod cutover.

---

## 8. Cloud-specific guidance

### 8.1 Infomaniak Public Cloud (Magogi foundation's main platform)

The Magogi foundation's primary cloud is **Infomaniak
Public Cloud** (OpenStack-based, Swiss-hosted). The
reference deployment patterns below are validated against
Infomaniak; they translate directly to other OpenStack
clouds and translate with module substitutions to
AWS / Azure / GCP.

**Service catalogue at Infomaniak** (March 2026 snapshot):

| Need | Available product | Notes |
|---|---|---|
| Compute | Public Cloud VMs (OpenStack Nova) | Per-second billing; A-series flavors from ~€3.28/mo |
| Load balancer | Public Cloud LB (Octavia) | HTTP/HTTPS/TCP listeners; TLS termination; WebSocket passthrough; ~€10/mo |
| Private network | Public Cloud VPC (Neutron) | VLANs, security groups, floating IPs (IPv4 billed, IPv6 free) |
| Object storage | S3-compatible | €0.01/GB/mo; free egress to 10 TiB/mo; endpoint `s3.pub1.infomaniak.cloud` |
| Block storage | Public Cloud volumes | PERF1 (500 IOPS), PERF2 (1k IOPS); attached to VMs |
| VM snapshots | OpenStack native | For DR |
| Backup | Swiss Backup (Acronis) | From €2.40/mo; S3 endpoint; filesystem/block-level |
| Email/SMTP | mail.infomaniak.com:587 | STARTTLS; included in mail subscription |

**Gaps to work around:**

| Need | Gap | Workaround |
|---|---|---|
| Managed PostgreSQL | Only MySQL is managed | Self-host on a dedicated VM (`postgresql` role in `django-infomaniak-deploy`) |
| Managed Redis | Not offered | Self-host on a dedicated VM (`redis` role) |
| CDN | No first-party | Cloudflare free tier in front of Object Storage |
| Transactional email API | No Mailgun/SendGrid equivalent | SMTP relay via mail.infomaniak.com (included) |
| Managed DB backups | No native pg_dump scheduling / WAL archiving | Cron `pg_dump | gzip` → Object Storage; lifecycle rule for 30-day retention |

**Reference architecture (single-tenant SaaS / Django shape):**

```
                     Cloudflare (DNS + edge TLS + WAF + CDN)
                                    │
                       Octavia LB (HTTP:80 + TCP:443)
                                    │ floating IP
                          App VM (10.0.1.10)
                          ├─ Uvicorn (4 workers, ASGI, port 8000)
                          ├─ Celery realtime worker (queue: realtime)
                          ├─ Celery email worker (queue: email)
                          └─ Celery-beat scheduler
                                  │       │
                       DB VM (10.0.1.20)  Redis VM (10.0.1.30)
                       ├─ PostgreSQL 16   └─ Redis 7
                       └─ PgBouncer (6432, transaction pooling)

                                  │
                       Object Storage (S3-compatible)
                       ├─ inthewords-media     (uploads + static; ACL: private + public-read for static/)
                       └─ inthewords-backups   (DB dumps; 30-day lifecycle)
```

**Auth note (OpenStack quirk).** Infomaniak issues
**application credentials** (not username/password) for
the OpenStack API. Object Storage uses a **separate access
key + secret** issued from the Object Storage UI. The two
credential types are NOT interchangeable; both end up in
the Ansible Vault.

**SSH user convention.** The reference framework
provisions a dedicated `<project>` deploy user (e.g.,
`inthewords`) — not `ubuntu` or `root`. The deploy user
owns `/opt/<project>/` and runs the systemd units. SSH
keys for operators are pushed via the `deploy-user`
Ansible role.

### 8.2 Static websites (lightweight shape)

Static-site projects (e.g., `magogi-website`) deploy
to **Object Storage + CDN** with no VM:

```
deploy/
├── README.md                ← entry point
├── terraform/               ← IaC for the bucket + CDN + DNS
│   ├── modules/static_site/ ←   bucket + CDN + DNS module
│   └── environments/prod/
├── scripts/
│   ├── publish.sh prod      ←   build site → upload to bucket → invalidate CDN
│   └── rollback.sh prod     ←   restore previous-version objects from versioned bucket
└── docs/
    ├── runbook.md           ←   "site is down" triage (DNS / CDN / bucket)
    └── slo.md               ←   simpler SLI: 99.9% availability of the bucket itself
```

Static-site DEP rules are a subset:

- **DEP-01, 04, 05, 09, 10, 11** apply unchanged.
- **DEP-02** does not apply (no VM).
- **DEP-03** applies — the artefact is the rendered
  static-site output, byte-identical per release.
- **DEP-06** is enabled by Object Storage versioning +
  the previous-objects rollback script.
- **DEP-07** has a simpler shape (no `ansible/`, no
  `monitoring/` beyond CDN dashboards).
- **DEP-08** has a thinner observability story — uptime
  ping + CDN edge-error rate is usually enough.

### 8.3 AWS / Azure / GCP (shape-translation pointers)

For consumers on the hyperscalers:

| Concept | AWS | Azure | GCP |
|---|---|---|---|
| VM | EC2 (or Fargate / ECS for containers) | VM Scale Set | Compute Engine |
| LB | ALB | Application Gateway | Cloud Load Balancing |
| Private network | VPC | Virtual Network | VPC |
| Managed DB | RDS (Postgres) | Azure Database for PostgreSQL | Cloud SQL |
| Managed cache | ElastiCache | Azure Cache for Redis | Memorystore |
| Object storage | S3 | Blob Storage | GCS |
| Secrets | Secrets Manager / KMS | Key Vault | Secret Manager |
| Observability | CloudWatch + X-Ray | App Insights + Monitor | Cloud Operations Suite |
| IaC | CDK / Terraform / CloudFormation | Bicep / Terraform | Terraform / Deployment Manager |
| Email | SES | Communication Services | n/a — use third-party |

The DEP-* rules apply identically. The §3 `deploy/`
shape applies identically. The hyperscaler-managed-PaaS
options (RDS, ElastiCache) let DEP-02 (config-as-code for
VM state) **shrink to "the DB and cache configuration
parameters" only** — the OS-side roles aren't needed
because there's no VM. That simplification is fine.

### 8.4 Managed-platform shapes (Fly.io / Render / Heroku / Vercel)

For consumers on managed platforms:

- The platform owns the VM lifecycle → DEP-02 is N/A.
- The platform owns the LB / TLS → DEP-09 is "verify the
  platform's auto-renew is enabled."
- The platform owns the runtime artefact build → DEP-03
  applies (lock the source) but the artefact-build step
  is platform-driven (not your CI).
- The platform owns the deploy command → DEP-05 still
  applies; the wrapper is `fly deploy --app prod` /
  `render deploys create prod` with the same
  "env-explicit, refuses bare" discipline.

The other rules (DEP-01, 04, 06, 07, 08, 10, 11) apply
unchanged.

---

## 9. Cross-references

- [`OPERABILITY.md`](OPERABILITY.md) — the runtime ops
  side of the same problem. OP-09 (production safety
  guards), OP-10 (versioned artefacts + rollback), OP-11
  (alerts page humans + runbooks), OP-12 (backup +
  restore drill against the real target) are the
  per-running-service contracts; DEP-* are the per-deploy
  contracts that put those services in place.
- [`SECURITY_STANDARD.md`](SECURITY_STANDARD.md) — SC-01
  (secrets in vault — covered by DEP-04), SC-04 (atomic
  + 0o600 file writes — applies to VM-side state files),
  SC-12 (vulnerability-reporting policy — every deploy
  ships a `SECURITY.md`).
- [`LOGGING.md`](LOGGING.md) — LG-03 (OpenTelemetry
  correlation) + LG-04 (sensitive-field redaction) are
  the observability primitives DEP-08 builds on.
- [`CONFIGURATION.md`](CONFIGURATION.md) — CF-02 (config
  precedence) + the environment-tier layering (§15) are
  the application-side contract that the deploy
  populates.
- [`RELEASING.md`](RELEASING.md) — REL-01..09 cover the
  per-release flow at the library boundary; DEP-* covers
  the deploy of an application that consumes those
  libraries.
- [`PACKAGE_DISTRIBUTION.md`](PACKAGE_DISTRIBUTION.md) —
  PKG-01..12 define where `mgf-*` packages are published
  and resolved from (the self-hosted devpi index
  `magogi/prod`, not pypi.org). PKG-07
  (version-immutability) is the contract DEP-03 leans on:
  because a published `mgf-*` version never changes, the
  deploy can pin it in the lock and skip every
  registry lookup at deploy time. Third-party deps still
  resolve from public PyPI at build time.
- [`DOCUMENTATION.md`](DOCUMENTATION.md) — §2.0 project
  shape taxonomy; DEP-* applies to "consumer
  application" shape projects.
- [`django-infomaniak-deploy`](https://codeberg.org/magogi-admin/django-infomaniak-deploy)
  — the reusable framework that realises this standard
  for Django/FastAPI on Infomaniak.
- [`inthewords/deploy/`](https://codeberg.org/magogi-admin/inthewords/src/branch/main/deploy)
  — first end-to-end consumer of the framework; the
  reference shape for §3.

---

*End of `DEPLOYMENT.md`.*
