# Operability

> **Conformance:** L1 / L2 / L3 (see `README.md`)
> **Audience:** Human engineers AND AI coding agents.
> **Propagation (DOC-18):** changing a rule here ripples — walk DOCUMENTATION.md §10.4 and update its dependents in the same commit (README *Index of MUSTs*, the `mgf-fed standards` / `agents` CLI, and any parallel copies / pins the matrix names).

> **Scope.** Cross-project operability standard. **"Operability"
> means: when something goes wrong, can a human (the user, the
> operator, the maintainer, a future contributor, an AI agent
> looking at the situation cold) actually run it?** Can they
> tell what's installed, ask the app for its state, find the
> logs, file a useful bug report, restore from a backup, roll
> back a bad deploy?
>
> **Two shapes covered:** *service* projects (web apps, daemons
> — multi-user, deployed, with operators distinct from users)
> and *user-facing* projects (desktops, CLIs, single-user tools
> — the user IS the operator). The shared rules are universal;
> the rules that only apply to one shape carry an explicit
> `applies to` column.
>
> Distilled from two real OPERABILITY trackers — one for a
> web/SaaS service (46 items across alerting / deploy / DR /
> docs / AI ops) and one for a desktop/CLI app (16 items across
> bug-report routing / self-diagnostic / SOPs / schema
> migration). The shared discipline is what landed here.

---

## Rules box (read this first)

| ID | Rule | Applies to |
|---|---|---|
| <a id="OP-01"></a>**OP-01** | **The app MUST report its rich version on demand.** A `version` query (CLI: `--version` rich output; service: `/version` endpoint or equivalent) returns: the package version, the git commit SHA built into the artefact, the active environment label, and the load-bearing runtime info for the project's shape (Python version, key dependency versions, OS / arch). "What revision is running?" MUST have a one-command answer. | all |
| <a id="OP-02"></a>**OP-02** | **Every project MUST ship a `doctor` self-diagnostic command** that probes the app's OWN state (config validity, persistent stores well-formed, key invariants hold), distinct from any "host-doctor" that scans the machine. Output is a pass/fail check list with `file:line` pointers on failures. The first triage question — "is the app's own state the problem?" — MUST have a one-command answer. | all |
| <a id="OP-03"></a>**OP-03** | **Project URLs MUST resolve.** `pyproject.toml`'s `Homepage` / `Documentation` / `Issues` URLs, and every URL printed in `--help` footers / CLI errors / GUI menu items, MUST point at the real repo / docs / issue tracker. CI MUST verify (an HTTP-HEAD lint, or at minimum a "no `github.com/<placeholder>`" grep). A 404 on a bug-report link routes every issue to nowhere. | all |
| <a id="OP-04"></a>**OP-04** | **Headline facts in `README.md` / `STARTHERE.md` / project front-matter MUST match reality.** Phase markers, test counts, source-file counts, dep version pins, ADR statuses are load-bearing for the mental model a newcomer / AI agent builds in their first 60 seconds. A stale phase marker or a wrong dep pin compounds — the reader builds the wrong model and wastes hours. CI SHOULD verify what's mechanically checkable (dep pins via `pip show`, test counts via `pytest --collect-only`). | all |
| <a id="OP-05"></a>**OP-05** | **The active configuration MUST be inspectable from a running app**, with secrets redacted per **LG-04**. CLI: `<app> explain` (or equivalent). Service: an admin-only `/config` endpoint OR a redacted runtime snapshot written to a known path. "What is the app configured with right now?" MUST have a one-command answer. | all |
| <a id="OP-06"></a>**OP-06** | **Recent logs MUST be accessible without source-tree knowledge.** CLI: a `<app> logs` command (tail the rotating file). GUI: a "View Logs" menu item under Help (per **LG-07**). Service: a documented aggregator URL + query syntax. Telling a user "logs are at `$XDG_STATE_HOME/<app>/logs/<app>.log`" is a defect — they shouldn't need to know the path. | all |
| <a id="OP-07"></a>**OP-07** | **Every project MUST ship a `bundle` / diagnostic-tarball command** that produces a single attachable artefact for bug reports: redacted config + last N log lines + recent crash reports + `<app> doctor` output + system info (Python, OS, key deps, env flags). Bug triage goes from "please run these five commands and paste the output" to "attach `<app>-bundle-<timestamp>.tar.gz`". | all |
| <a id="OP-08"></a>**OP-08** | **State directories MUST have bounded growth.** Crash reports, log rotations, ephemeral caches — each MUST have a documented retention policy (size cap, age cap, or generational rotation). A regression that crashes on every invocation MUST NOT fill the user's disk before they notice. Suggested default: keep the most recent 100 crash reports OR 7 days, whichever is larger; rotate logs at 10 MB × 5 generations. | all |
| <a id="OP-09"></a>**OP-09** | **Production safety guards MUST refuse dev-shaped values.** A service started with `environment=production` MUST fail-fast (during config validation, before listening) if any setting still carries a development-time value — mock backends enabled, debug ports open, default passwords, `dev_*` flags, sample data fixtures. A copy-pasted `.env` from staging MUST NOT silently boot a broken prod. | service |
| <a id="OP-10"></a>**OP-10** | **Every deployable artefact MUST carry a stable version tag.** `image: <project>:latest` in a production compose / k8s manifest is a defect — rollback requires a previous-version tag to roll back TO. Tag images with the git SHA AND the semver version; the rollback procedure (documented, exercised at least once) restores the previous tag. | service |
| <a id="OP-11"></a>**OP-11** | **Every alert MUST reach a human, and every alert MUST link to a runbook.** Alerts firing into a void are noise; alerts without runbooks are panic. The paging path (alertmanager → ntfy / Slack / PagerDuty / email) MUST be wired and tested with a synthetic alert before go-live. Each alert rule carries an `annotations.runbook_url` pointing at a `docs/runbooks/<alert>.md` with the "first 30 minutes" procedure. **No business-hours gates on production-critical alerts** — a 22:30 outage is still an outage. | service |
| <a id="OP-12"></a>**OP-12** | **Backup + restore MUST be exercised end-to-end against the real target.** A backup script that succeeds in CI against a minio mock proves nothing about its behaviour against the production S3 / R2 / GCS credentials, rate limits, eventual-consistency window, region. The restore drill MUST run against the real target at least once before go-live and SHOULD be repeated quarterly. RTO claims in ADRs are unmeasured fiction until the drill runs. | service |

---

## 1. Mental model — operability is *can-you-run-it*

Reliability is about whether the code holds up under stress.
Security is about whether it stays trustworthy under attack.
Performance is about whether it stays fast under load.

**Operability is the orthogonal fourth concern:** when the
other three fail anyway — and eventually one will — *can the
human running it actually run it*. Not "in theory, with a
debugger and a fresh checkout"; not "if they read 500 lines of
source first." In practice, at 02:00 on the wrong timezone,
with a worried user / customer / partner waiting.

The bar is mechanical: every recoverable failure mode MUST
have a documented path back to a working state, and every
piece of state the operator needs MUST be reachable by one
command. The cost of getting this wrong is paid the first
time something breaks — and by then, fixing it is panic-coding
under pressure.

**The two shapes:**

| Shape | Who is "the operator" | What "operability" emphasises |
|---|---|---|
| **Service** (web app, daemon, scheduled worker) | A maintainer or SRE, distinct from end users; multi-tenant; on-call rotation; deployed to remote infrastructure | Paging, runbooks, rollback, blue/green, multi-replica, observability correlation, backups, SLOs |
| **User-facing** (desktop GUI, CLI, single-user tool) | The user IS the operator; one machine; no on-call; usually no deployment | Bug-report routing, self-diagnostic, log access, version visibility, schema migrations, state hygiene |

Both share the **8 universal rules** (OP-01..OP-08); the
**4 service-only rules** (OP-09..OP-12) apply where a real
deployment + alerting story exists.

---

## 2. The rules in detail

### 2.1 OP-01 — Rich version visible

Every project MUST answer "which revision is running?" in one
command. The form differs by shape:

**CLI / desktop** (example shape):

```bash
$ <app> --version
<app> 0.33.0 (commit 7a1c3f9, built 2026-05-15)
  Python      3.12.3
  OS          Linux 6.17 x86_64
  mgf-common  0.33.0
  <key dep>   <version> (binding <version>)
  <APP>_USE_MOCK=0
```

**Service** (example shape — JSON over HTTP):

```bash
$ curl http://localhost:8080/version
{
  "version": "0.33.0",
  "commit": "7a1c3f9",
  "environment": "production",
  "started_at": "2026-05-15T03:14:00Z",
  "python": "3.12.3",
  "mgf_common": "0.33.0"
}
```

**Why it's load-bearing:** when a user files "the bug is back,"
the first triage question is "what version are you running?"
Without OP-01, the maintainer asks; the user is unsure; the
back-and-forth costs hours. With OP-01, it's one paste.

**Implementation:** bake `GIT_SHA` as a Docker build-arg / a
hatch hook / a `_version.py` regenerated at build. Expose via
the CLI / endpoint. Log the rich version line at INFO on
startup (unconditional — never filtered out, so it's always in
the bundle).

### 2.2 OP-02 — Self-diagnostic `doctor`

The `doctor` command probes the app's own state, not the
host's:

- Config validity (does `<app> validate` pass on the loaded
  config?).
- Persistent stores well-formed (schemas / YAML / SQLite — each
  loads without raising).
- Required state directories exist + correct permissions.
- Schema versions match what the code expects.
- Key invariants hold (no stale lock files, no orphan
  subprocesses for a single-user app, no half-committed
  migrations).

Output is **a check list with pass/fail per row** and
`file:line` pointers on failures so the user has somewhere to
look. Exit code is the count of failures (0 = clean).

`mgf-fed app-doctor` is the reference implementation; see §3.

### 2.3 OP-03 — Project URLs resolve

`pyproject.toml`'s `Homepage`, `Documentation`, `Issues` URLs
MUST point at the real project. Same for every URL printed in
`--help`, CLI error messages, GUI menu items, log warnings
("see `<url>` for details").

**Mechanical check:** a CI step that does `curl -fsSI` (head
request, fail on 4xx/5xx) against every URL grep'd from
`pyproject.toml` + `README.md`. Cheap, runs in <5 s.

**Failure mode this prevents:** PyPI metadata's "report issue"
link routes to `github.com/<placeholder>/<placeholder>` and
every issue lands in a void. Cheap mistake to make, expensive
mistake to notice: nobody who hits it can file a bug to tell
you the bug-report link is broken.

### 2.4 OP-04 — Headline facts true

The first 60 seconds a newcomer (human or AI) spends with the
project shapes the mental model they build. Stale facts in
`README.md` / `STARTHERE.md` poison that model:

- A `Phase 2` marker in a project that's at Phase 4 → the
  reader expects an immature surface and second-guesses the
  shipped code.
- A `mgf-common = "0.25,<0.26"` pin in README when reality is
  `0.33,<0.34` → the reader pins the wrong version, hits API
  mismatches, files an issue against the wrong contract.
- A test count claiming 1643 when reality is 3627 → the reader
  thinks the test-coverage story is thinner than it is, makes
  the wrong investment decision.

**Mechanical defenses:** CI lints what it can — dep version
pin (verify against `pip show`), test count (collect-only),
ADR-status keywords. Phase markers are review-level: the
release flow (REL-07) MUST refresh the README.

### 2.5 OP-05 — Config introspectable

Running app → "what am I configured with?" → one command.
Secrets redacted per **LG-04**. Sources visible (env var / CLI
flag / `.env` file / default — same precedence as **CF-02**).

The mgf-common reference: `mgf-fed explain` prints the
active `MgfSettings` with each field's source.

For services that can't easily ship a CLI inside the
container, an admin-only HTTP endpoint OR a periodic
runtime-snapshot write to a known path (per **SC-04** — 0o600
atomic) is the equivalent.

### 2.6 OP-06 — Logs accessible without path knowledge

The operator should NOT need to know `$XDG_STATE_HOME/<app>/logs/`
or `journalctl -u <app>.service` syntax to see what just
happened.

- **CLI:** `<app> logs` tails the rotating file (mirrors
  `journalctl -u <unit> -n 100`).
- **GUI:** Help → View Logs (per **LG-07**).
- **Service:** documented aggregator URL + query syntax in
  the runbook.

### 2.7 OP-07 — Diagnostic bundle

When a user files "the app crashed when I…" the maintainer
needs:

- The exact config (redacted).
- The last ~500 log lines around the crash.
- The crash-report JSON.
- `<app> doctor` output.
- System info: Python version, OS, key dep versions, env flags.

One command produces a `<app>-bundle-<timestamp>.tar.gz` the
user can attach. Without it, bug triage takes N follow-up
messages; with it, one.

Privacy: the bundle MUST go through the same Redactor pass
(rule **LG-04** + **SC-11**) as crash reports.

### 2.8 OP-08 — Bounded state growth

Crash reports, log rotations, ephemeral caches MUST have a
retention policy. Examples:

- Crash reports: keep most recent 100 OR 7 days (whichever is
  larger).
- Logs: rotate at 10 MB × 5 generations (the stdlib
  `RotatingFileHandler` default).
- Caches: invalidate on schema version bump.

**Failure mode this prevents:** a regression that crashes on
every invocation fills the user's disk over a weekend; the
next session won't start because there's no space for a new
crash report.

### 2.9 OP-09 — Production safety guards (service)

When `environment=production` is set, the config validator MUST
refuse:

- Mock backends (`use_mock=True`, dict-backed resolvers, any
  `MOCK_*` envvar wired in).
- Development-time secrets (`dev_mock_token`, default
  passwords, sample API keys).
- Debug flags (`debug=True`, `DEBUG_SQL_ECHO=1`).
- Sample data fixtures.

Fail-fast during `validate_production()` — before the listener
opens, before any traffic, before any side-effect. The error
message MUST name the offending field + how to set it
correctly (per **EH-09** actionable hints).

### 2.10 OP-10 — Versioned artefacts + rollback (service)

`image: <project>:latest` is forbidden in production manifests.
Every deployable artefact MUST carry a stable version tag —
either the semver version OR the git SHA. Rollback requires a
previous-version tag to roll back TO; `latest` overwrites that
target on every push.

The rollback procedure MUST be a numbered runbook (3–7 steps),
exercised at least once on a non-production target, with a
documented blast radius ("rolls back the API tier, not the
worker tier; if a migration ran since the prior version,
re-run the down migration first").

### 2.11 OP-11 — Alerts page humans + carry runbooks (service)

Two failure modes this rule kills:

- **Alerts that don't page.** Prometheus rules wired but
  `alertmanager` not configured → alerts fire into the
  dashboard, no one sees them, the outage runs until a user
  reports it.
- **Alerts that page but don't tell you what to do.** PagerDuty
  notification at 02:00 with a metric name and no playbook →
  the on-call diagnoses from scratch under stress.

**The rule's two halves:**

1. Wire the paging path. The cheapest option is genuinely
   cheap: `alertmanager` → `ntfy.sh` (free) or a Slack
   webhook. Validate end-to-end with a synthetic alert.
2. Every alert rule carries an `annotations.runbook_url` field
   pointing at a `docs/runbooks/<alert-name>.md` with at
   minimum: symptom, first-30-min diagnostics, escalation
   path, common-cause fixes.

**Carve-out:** `severity: info` alerts that don't page (just
log) MAY omit the runbook. Anything that pages MUST have one.

**No business-hours gates** (`expr: ... and hour() in [6, 22]`)
on production-critical alerts. A 22:30 outage is still an
outage.

### 2.12 OP-12 — Backup + restore drill against real target (service)

A backup script that works against a CI mock proves nothing
about prod. The S3 / R2 / GCS production target has:

- Real credentials (rotated, scoped, time-limited).
- Rate limits (large restores hit them).
- Eventual consistency windows (the "I just uploaded it, why
  isn't it there?" 30 seconds).
- Region-specific quirks (transit costs, retrieval tiers).

The restore drill MUST run against the real target at least
once before go-live. RTO claims in ADRs ("RTO ≤ 30 min") are
fiction until measured.

**Cadence:** quarterly drill, recorded in a `docs/drills/<date>.md`
log: target, duration, bytes restored, any surprises.

---

## 3. AS-IS in `mgf-common`

mgf-common ships the universal-rule (OP-01..OP-08) primitives;
consumers wire them. Service-only rules (OP-09..OP-12) live in
the consumer-side runbooks + CI flow.

| Rule | mgf-common surface | Status |
|---|---|---|
| OP-01 version visibility | `mgf.common.__version__`; consumers expose via their CLI / endpoint | ✅ shipped (primitive); consumer-side exposure required |
| OP-02 doctor | `mgf-fed app-doctor` (`--live` / `--redaction` / `--standards` modes); reusable via `mgf.common.cli._doctor` | ✅ shipped |
| OP-03 URLs resolve | `pyproject.toml` carries valid `Homepage` + `Issues` (Codeberg `magogi-admin/mgf-common`) | ✅ shipped; CI URL-resolve lint not yet wired (open) |
| OP-04 headline facts | `README.md` + `STARTHERE.md` refresh discipline (REL-07 close-out) | ✅ followed (verified by Pass A + Pass B audits) |
| OP-05 config introspect | `mgf-fed explain` prints active `MgfSettings` + per-field source | ✅ shipped |
| OP-06 log access | Operator reads the rotating-file sink path; `<app> logs` consumer-side wrapper recommended | ⏳ primitive shipped; the `<app> logs` wrapper is consumer-side |
| OP-07 bundle | `MGF_DOCTOR_RUNTIME_PATH` writes a 0o600 atomic runtime snapshot (per **SC-04**) redacted per **SC-09**; full bundle command is consumer-side | ⏳ partial — the building blocks (redacted snapshot, crash dir, doctor) ship; a single bundle CLI is consumer-shaped |
| OP-08 bounded state | Atomic-write helper (`mgf.common._io.atomic_write`) + crash-dir mode 0o600; retention policy is consumer-side | ✅ atomic primitives shipped; retention discipline is consumer-side |
| OP-09..12 | service-only — applied by each service consumer | n/a (mgf-common is a leaf library, not a service) |

The pattern: **mgf-common ships the lego pieces**; consumers
wire them to their own operator UX, with service-shape and
desktop-shape consumers diverging on OP-09..12. The standard
pins the contract; the implementations differ by shape.

---

## 4. Anti-patterns (forbidden)

| Anti-pattern | What's wrong | Right shape |
|---|---|---|
| `image: myproject:latest` in production manifest | No rollback target; every push overwrites | Tag images with semver + git SHA; document rollback path (OP-10) |
| `Homepage = "https://github.com/<placeholder>"` | PyPI metadata routes bug reports to 404 | Verified URLs; CI HEAD-check (OP-03) |
| `--version` returning just `"0.1.0dev0"` | First bug-report follow-up: "what Python, what OS, what deps" | Rich version: SHA + Python + key deps + env (OP-01) |
| "Logs are at `$XDG_STATE_HOME/<app>/logs/<app>.log`" in the docs | Operator shouldn't need path knowledge | `<app> logs` CLI or GUI menu (OP-06) |
| Alert rule with no `runbook_url` | On-call paged at 02:00, no playbook | Every alert links a runbook (OP-11) |
| `expr: rate(...) and hour() in [6, 22]` | 22:30 outage invisible until morning | Drop the gate; if the alert is too noisy off-hours, fix the threshold not the time-window (OP-11) |
| Crash dir grows forever | Disk fills before user notices | Bounded retention (OP-08) |
| `validate_production()` silently accepts `use_mock=True` | Copy-pasted dev env boots broken prod | Production guard refuses dev fields (OP-09) |
| "Phase 2" in README when project is at Phase 4 | Newcomer / AI mental model wrong | Headline-facts truth discipline (OP-04) |
| `disaster_recovery.md` references a `backup-cron` step that doesn't exist in `.woodpecker.yml` | Backup never runs; restore drill never tested | Backup wired in CI; quarterly real-target restore drill (OP-12) |

---

## 5. Self-review checklist (every release-track PR)

Service projects:

- [ ] **OP-01** — `/version` (or `--version`) returns rich
  identity (version + commit SHA + env + key deps).
- [ ] **OP-02** — `<app> doctor` passes against the current
  branch's defaults.
- [ ] **OP-03** — `pyproject.toml` URLs and `README.md` URLs
  resolve (CI green).
- [ ] **OP-04** — `README.md` phase marker / counts / dep
  pins match reality.
- [ ] **OP-05** — `<app> explain` shows the active config.
- [ ] **OP-06** — recent logs are reachable without source-tree
  knowledge.
- [ ] **OP-07** — `<app> bundle` produces a redacted support
  tarball.
- [ ] **OP-08** — state directories have documented retention.
- [ ] **OP-09** — `validate_production()` refuses every
  development-shaped field.
- [ ] **OP-10** — production image tagged (no `:latest`);
  rollback runbook exists.
- [ ] **OP-11** — every alert has a runbook link + reaches a
  human; synthetic-alert end-to-end test passed.
- [ ] **OP-12** — backup ran against the real target this
  quarter; restore drill recorded.

Desktop / CLI projects: items 1–8 only.

---

## 6. Mechanical enforcement matrix

| Rule | Mechanical check | Notes |
|---|---|---|
| OP-01 version | grep CLI / endpoint output for SHA pattern | CI step: `<app> --version 2>&1 \| grep -E '[a-f0-9]{7,}'` |
| OP-02 doctor | `<app> doctor` exits 0 on the green-main branch | CI step |
| OP-03 URLs resolve | `curl -fsSI` against each URL grep'd from pyproject + README | Implementable as a 30-line CI step |
| OP-04 headline facts | dep pin lint + test count lint | Doable; phase marker is review-level |
| OP-05 config introspect | `<app> explain` exits 0 | CI step |
| OP-06 log access | `<app> logs --help` shows usage (presence-check) | CI step |
| OP-07 bundle | `<app> bundle` produces a non-empty tar | CI step |
| OP-08 bounded state | review-level + integration test that asserts retention triggers | Implementable |
| OP-09 production guard | unit test: `Settings(environment="production", use_mock=True)` raises | Mechanical |
| OP-10 versioned artefacts | grep production compose files for `:latest` → must return zero hits | CI step |
| OP-11 alerts | every Prometheus rule MUST have an `annotations.runbook_url` field — `promtool` or a yaml-lint step | Mechanical |
| OP-12 restore drill | `docs/drills/` has at least one pre-go-live entry — lint by presence | Mechanical (presence grep); quarterly cadence is review-level |

Every MUST in this doc has a mechanical check OR a clear
review-level discipline. No rule is purely aspirational.

---

## 7. Cross-references

- [`LOGGING.md`](LOGGING.md) — LG-07 (GUI log viewer) +
  redaction across egress paths. OP-06 / OP-07 build on the
  logging stack.
- [`SECURITY_STANDARD.md`](SECURITY_STANDARD.md) — SC-04 atomic
  + 0o600 (covers runtime snapshots, crash reports, logs);
  SC-09 sink redaction; SC-11 crash/traceback redaction —
  together they protect what OP-05 / OP-07 expose.
- [`ERROR_HANDLING.md`](ERROR_HANDLING.md) — EH-09 actionable
  hints; OP-02 surfaces the same "what's broken / what to do"
  discipline for the whole app.
- [`CONFIGURATION.md`](CONFIGURATION.md) — CF-02 precedence;
  OP-05 introspects exactly that precedence chain.
- [`RELEASING.md`](RELEASING.md) — REL-06 release flow; REL-07
  FEEDBACK→SHIPPED closure is the lever for OP-04 headline-
  facts refresh.
- [`docs/standards/WORKFLOWS_PATTERNS.md`](WORKFLOWS_PATTERNS.md)
  — paste-ready CI recipes; OP-03 URL-resolve step would land
  here once the canonical shape is settled.
- [`DEPLOYMENT.md`](DEPLOYMENT.md) — the per-deploy contracts
  that PUT services in place (IaC, config-as-code, immutable
  artefacts, secrets). DEP-06 (reversible deploys) is the deploy
  side of OP-10; DEP-08 (end-to-end observability) is the
  per-deploy side of OP-11. The DEPLOYMENT standard ships the
  `deploy/docs/runbook.md` that OP-11's per-alert runbook
  entries link into.
- mgf-common's `mgf-fed app-doctor` + `mgf-fed explain` are
  the reference implementations consumers can lift / lean on.

---

*End of `OPERABILITY.md`.*
