# CI Workflows

> **Conformance:** L1 / L2 / L3 (see `README.md`)
> **Audience:** Human engineers AND AI coding agents responsible for project CI configuration.
> **Propagation (DOC-18):** changing a rule here ripples — walk DOCUMENTATION.md §10.4 and update its dependents in the same commit (README *Index of MUSTs*, the `mgf-fed standards` / `agents` CLI, and any parallel copies / pins the matrix names).

> **Distribution target.** For `mgf-*` libraries the publish/verify
> destination is the **self-hosted devpi index** (`magogi/prod` on
> `195.15.203.250`), not pypi.org or the Codeberg registry — see
> [`PACKAGE_DISTRIBUTION.md`](PACKAGE_DISTRIBUTION.md) (rules
> PKG-01..PKG-12) for the canonical endpoints, auth model, and
> verify protocol. WF-03/04/06/08 below operationalise that target
> in CI. Third-party deps still resolve from public PyPI, via devpi's
> `root/pypi` base (PKG-01 §1).

> **Scope.** Cross-project CI / continuous-integration standard for
> any project in the MGF family — libraries that publish an `mgf-*`
> package to the devpi index, services with deployment
> pipelines, internal tools with no publish step. The rules are
> written for [Woodpecker
> CI](https://woodpecker-ci.org/) on Codeberg / Forgejo (the MGF
> family's primary CI host) but apply to every CI platform: GitHub
> Actions, GitLab CI, Jenkins, Buildkite. The mechanics differ;
> the discipline doesn't.
>
> Filed in response to a maintainer-side incident on 2026-05-02:
> three releases were cut by hand from a developer machine,
> reusing wheel artifacts that drifted from the published registry
> versions and leaking project-wide publish tokens through chat.
> [`RELEASING.md`](RELEASING.md) is the *what* and *why* of a
> release; this document is the *how* — how CI enforces the
> release rules so a human (or agent) can't accidentally bypass
> them.

---

## Rules box (read this first)

| ID | Rule |
|---|---|
| <a id="WF-01"></a>**WF-01** | **Every project MUST have CI that runs on every push to a long-lived branch and every pull request.** No "ship from local laptop" paths. The CI run is the canonical proof that the code passes the project's gates; a developer machine's pytest run is not. |
| <a id="WF-02"></a>**WF-02** | **The four green gates (lint / typecheck / test / import-linter) MUST be CI-enforced** and MUST block merge on red. The gates are the contract; CI is the enforcement. Branch-protection rules on the platform side make the block automatic. |
| <a id="WF-03"></a>**WF-03** | **A library that publishes an `mgf-*` package MUST have a tag-driven publish workflow that targets the self-hosted devpi index** (per [`PACKAGE_DISTRIBUTION.md`](PACKAGE_DISTRIBUTION.md) **PKG-01/PKG-09**). Pushing a `v<X.Y.Z>` tag is the trigger; pushes without a tag MUST NOT publish. The tag is the maintainer's signed declaration "this is the release point" (rule **REL-01**); CI consumes that declaration and uploads to `https://195.15.203.250.sslip.io/magogi/prod/` (**PKG-02**). |
| <a id="WF-04"></a>**WF-04** | **Publish credentials MUST be stored in the CI platform's secret store**, scoped to tag-event runs where supported. The credential is the **devpi `magogi` account's upload password** held in the Woodpecker **org** secret named `devpi_password` (per [`PACKAGE_DISTRIBUTION.md`](PACKAGE_DISTRIBUTION.md) **PKG-03/PKG-09**); install/read is anonymous. Secrets in the workflow file, in environment variables passed at trigger time, or in chat are forbidden. There is **no OIDC / Trusted Publisher on devpi** — the password-in-CI-secret model is the standing answer, not a fallback. |
| <a id="WF-05"></a>**WF-05** | **The publish workflow MUST verify tag-version coherence before uploading.** The `v<X.Y.Z>` tag name MUST match `pyproject.toml::project.version` AND `__version__`. Any mismatch aborts the run with a loud error. This catches the "I tagged before I bumped" footgun and the "I bumped after I tagged" footgun. |
| <a id="WF-06"></a>**WF-06** | **The wheel uploaded to the devpi index MUST be the one built in the CI run that's about to publish.** Never re-use a wheel built earlier — in a prior CI run, on a developer machine, sitting in `dist/`. Per **REL-02** local `dist/` artifacts can drift from the version string they claim; the only safe wheel is one built fresh in the publish run (and the same artifact checked is the one uploaded — [`PACKAGE_DISTRIBUTION.md`](PACKAGE_DISTRIBUTION.md) **PKG-04**, no rebuild between check and upload). |
| <a id="WF-07"></a>**WF-07** | **Pre-publish smoke MUST install the freshly-built wheel into an ephemeral venv** and import the top-level package + `__version__`. A test suite passing against the source tree does not prove the wheel installs and imports cleanly — packaging-metadata bugs (missing `package_data`, broken `requires-python`, wrong extras) only surface against the wheel itself. Per **REL-08**. |
| <a id="WF-08"></a>**WF-08** | **Post-publish verification MUST confirm the artifact appeared in the devpi index** — poll `GET …/magogi/prod/+simple/<name>/` (anonymous) until `<version>` is listed within a polling window (per [`PACKAGE_DISTRIBUTION.md`](PACKAGE_DISTRIBUTION.md) **PKG-05**); NOT pypi.org or Codeberg. Failure to verify = treat as failed publish, not silent success. The upload step returning 0 is necessary but not sufficient — the index might 5xx or not yet surface the version. |
| <a id="WF-09"></a>**WF-09** | **Workflow files MUST pin every external dependency to a specific version.** Python interpreter image (`python:3.12-slim`, not `python:slim`); plugin actions (`@v1.2.3`, not `@v1` or `@latest`); base images by digest where the platform supports it. Pinning eliminates "the build worked yesterday but breaks today" surprises. |
| <a id="WF-10"></a>**WF-10** | **Independent CI steps MUST run in parallel.** Sequential ordering is reserved for genuine dependencies (build before publish; smoke before upload; tag-coherence-check before build). The matrix axes (Python versions, optional extras) MUST fan out, never serialise. |
| <a id="WF-11"></a>**WF-11** | **A CI run MUST be reproducible from its inputs alone.** No environment variables outside what the workflow declares; no host-machine state; no implicit network resources beyond what's pinned. Re-running the same workflow against the same commit MUST produce the same outcome (modulo external-flake retries, which MUST be explicit). |
| <a id="WF-12"></a>**WF-12** | **Every CI step MUST surface its exit code, the exact command that ran, and enough log context** (last 50+ lines of stdout/stderr) in the platform UI on failure. Silent failures — green ticks for skipped or no-op steps, hidden errors swallowed by `|| true` — are forbidden. A red CI run MUST tell the on-call exactly what to look at. |
| <a id="WF-13"></a>**WF-13** | **The `.woodpecker.yml` MUST be a generated artifact**, rendered from the canonical template (`mgf-standard/templates/woodpecker/`) by `mgf-fed scaffold` — never hand-edited. `mgf-fed drift-check` enforces generation-currency (on-disk == generated); legitimately-bespoke pipelines are listed in `policy.toml [exempt].woodpecker` and checked by the step-name fallback. Regenerate after a template change; commit the regenerated file. (Generation ≠ execution: activating remote Woodpecker so the pipeline actually RUNS is a separate operational step that makes WF-01/WF-02 real — see [`../recipes/activate-ci.md`](../recipes/activate-ci.md).) |

---

## 1. Mental model — CI as the contract enforcement layer

The standards in this folder declare what good engineering looks
like. CI is what makes the standards mechanical instead of
aspirational.

### 1.1 The asymmetry between human discipline and machine discipline

A human maintainer who reads the standards once a quarter and
"tries to follow them" will, on a long enough timeline, miss one.
Tired evening; "just this once" exception; a dependency upgrade
that surfaces a new violation no one was watching for. The
maintainer's intent doesn't fail; their attention does.

CI doesn't get tired. The four-gate baseline (rule WF-02) runs
the same way on every push, every PR, every release. A
maintainer who skips a gate locally still gets caught at PR time.
A maintainer who skips the publish smoke locally cannot even
trigger a publish — the workflow is the only path.

This is the load-bearing reason to push manual discipline INTO
the workflow file. Every rule that survives only if a human
remembers it is one bad day from being broken; every rule
encoded in CI survives until someone explicitly removes it.

### 1.2 The two populations of CI failures

Like API consumers (cf. [`API_DESIGN.md`](API_DESIGN.md) §1.1),
CI runs have two populations:

- **Visible failures** — the PR you're working on, the release
  you're cutting. You see the red tick and fix it.
- **Invisible failures** — the cron job that scans for new CVEs
  on your dependency tree, the matrix axis you don't routinely
  watch, the post-publish verification that flagged a CDN
  propagation issue at 3am. Easy to miss, easy to silently
  ignore, easy to disable.

The standards-grade discipline: every CI failure surfaces
loudly, and disabling a check requires a written justification
in the workflow file. Drift in the silent direction (more
exemptions, more `|| true`s, more suppressed gates) is how the
discipline rots.

---

## 2. The four-gate baseline (rule WF-02)

Every MGF project's CI runs four gates on every push and PR.
Red on any gate blocks merge. Order in the workflow file is
parallelism-first; the dependency graph is below.

### 2.1 The four gates

| Gate | Tool | What it catches |
|---|---|---|
| **Lint** | `ruff check src tests` | Style + correctness lints (unused imports, mutable defaults, broad `except`, etc.). Format-check via `ruff format --check src tests`. |
| **Typecheck** | `mypy --strict src/` | Type errors, missing annotations, unsafe `Any` propagation. The `--strict` flag is non-negotiable; consumers expect typed surfaces (rule **AP-09**). |
| **Test** | `pytest -q` (with coverage on the canonical Python version) | Unit + integration tests. Coverage gate (per project: 80% / 85% / 90%) blocks regression. |
| **Import-linter** | `lint-imports` | Layering invariants from `.importlinter` — closed-box rules, federation-leaf rules, no-cycle rules. The discipline that keeps the architecture from degrading. |

### 2.2 Parallelisation (rule WF-10)

The four gates have no dependencies on each other. They MUST
run in parallel — Woodpecker step graph, GitHub Actions matrix,
or equivalent. Sequential gates burn CI minutes for no benefit.

The Python-version matrix axis is independent of the gate axis:
on a 3-version matrix × 4 gates, you have 12 parallel jobs
(though typecheck + lint + import-linter only need to run on
the canonical version — they're version-agnostic). The
canonical pattern: gates × 1 version + test × N versions.

### 2.3 What "block merge on red" means concretely

Codeberg/Forgejo: branch-protection rule on `master` requires
the CI status check to be green before the merge button is
enabled. GitHub: same shape via "Require status checks to
pass". The rule is enforced by the platform, not by the
maintainer's good intentions.

A CI failure in a non-blocking branch (a feature branch) is the
maintainer's signal to fix before opening the PR. A CI failure
on the PR itself MUST be fixed before merge — no override
without recorded justification.

### 2.4 Extended gates — security + supply chain (rules SC-01, SC-10, SC-18)

The four-gate baseline is the **floor**. A mature CI adds three
more gates that protect against failure modes lint/typecheck/test
cannot catch:

| Gate | Tool | What it catches | Rule |
|---|---|---|---|
| **Vulnerability scan** | `pip-audit --strict --desc` against the installed environment | A pinned dependency has a known CVE since the last build | SC-10 |
| **Secret scan** | `gitleaks detect --source . --config .gitleaks.toml` against the working tree + commit history | An API token / private key / PEM block slipped into a commit (`my_stuff/`-style ignores are not enough — gitleaks scans history) | SC-01 |
| **SBOM emission** | `cyclonedx-py environment --pyproject pyproject.toml --of JSON --output-reproducible -o dist/<pkg>-<ver>.cdx.json` | The release flow produces no supply-chain manifest; consumers' SCA tooling has nothing to ingest | SC-18 |

**Shape (Woodpecker `.woodpecker.yml`):**

```yaml
- name: vuln-scan
  image: python:3.12-slim
  commands:
    - pip install --quiet pip-audit -e ".[dev,observability]"
    - pip-audit --strict --desc || (sleep 30 && pip-audit --strict --desc)

- name: secret-scan
  image: zricethezav/gitleaks:v8.30.1   # exact pin per WF-09
  commands:
    - gitleaks detect --source . --config .gitleaks.toml --no-banner --redact --verbose

- name: sbom
  image: python:3.12-slim
  depends_on: [build]
  commands:
    - pip install --quiet 'cyclonedx-bom>=7,<8'
    - pip install --quiet ".[observability,vault]"  # the optional surface — NOT [dev]
    - cyclonedx-py environment --pyproject pyproject.toml --mc-type library
        --of JSON --output-reproducible
        -o "dist/<pkg>-${VERSION}.cdx.json" "$(which python)"
```

**Configuration hygiene.**

- `pip-audit` reads the installed environment, so it covers
  whatever the preceding install step produced.
- `gitleaks` uses an in-repo `.gitleaks.toml` that allowlists
  files whose stated purpose is to embed secret-shaped strings
  (redactor test fixtures, pattern literals, pedagogical
  example placeholders). Without the allowlist, the bundled
  rule pack flags the redactor's own canonical examples.
- `cyclonedx-py environment` runs against the **optional**-extras
  surface a consumer might install (`.[observability,vault]`)
  — NOT against `.[dev]` (pytest / mypy / ruff are NOT runtime
  deps and would pollute the consumer-facing SBOM).
  `--output-reproducible` strips timestamps so a re-run on the
  same commit produces a byte-identical SBOM (supports REL-02).
- The `publish` step's `depends_on:` MUST include all three
  extended gates — a release tag cannot ship without secret-scan
  green AND a valid SBOM emitted.

**Library projects with multiple consumers** SHOULD additionally
add a **compat-matrix** step that installs the in-repo library +
each known consumer (or a representative `example_app`) and runs
that consumer's test suite against the in-repo library. This is
the second-consumer regression signal: a public-API break in the
library surfaces here BEFORE a release lands and a downstream
consumer files an issue. See mgf-common's `compat-matrix-example-app`
step for the canonical shape.

For the per-step shape and the cross-platform variations
(GitHub Actions / GitLab CI translations), see
[`WORKFLOWS_PATTERNS.md`](WORKFLOWS_PATTERNS.md).

---

## 3. Build + smoke before publish (rules WF-06, WF-07)

The bridge between "tests passed" and "a wheel that consumers
can `pip install`" is wider than it looks. A green test suite
proves the source tree works; the wheel might still be broken.

### 3.1 What can break in the wheel that tests don't catch

- **Missing package data.** A YAML / JSON / standards doc that
  the source-tree test reads via a relative path is missing
  from the built wheel because `pyproject.toml`'s
  `package_data` / `force-include` rules don't cover it.
- **`requires-python` mismatch.** The source tree runs fine on
  3.13; the wheel claims `>=3.11` but uses a 3.13-only stdlib
  feature. Source-tree tests on 3.13 pass; consumers on 3.11
  ImportError.
- **Optional-extras misconfiguration.** `pip install
  mypackage[fastapi]` doesn't pull in the right deps because
  `[project.optional-dependencies]` has a typo.
- **Console-script entry-point broken.** The `[project.scripts]`
  entry references a function that no longer exists; `pip
  install` succeeds but `mypackage --version` ImportErrors.
- **`py.typed` marker missing.** Wheel ships without the PEP 561
  marker; consumers' `mypy --strict` can't read your annotations.

### 3.2 The pre-publish smoke recipe

```bash
# Build the wheel (must be the only wheel in dist/ for this version)
uv build

# Create a clean, ephemeral venv and install the wheel ONLY (no editable, no source tree)
python -m venv /tmp/smoke-venv
/tmp/smoke-venv/bin/pip install --no-cache-dir dist/<package>-<version>-*.whl

# Smoke: import + version check
/tmp/smoke-venv/bin/python -c "
import <package>
assert <package>.__version__ == '<version>', f'expected <version>, got {<package>.__version__}'
print(f'OK: {<package>.__version__}')
"

# Cleanup
rm -rf /tmp/smoke-venv
```

If the smoke fails, the wheel is broken — abort the publish.
This catches every packaging-metadata bug short of functional
regression.

For projects with a console-script entry point, add an
invocation smoke too:

```bash
/tmp/smoke-venv/bin/<package> --version
```

### 3.3 What pre-publish smoke is NOT

It is NOT a substitute for the test suite. It is a fast
sanity check that the artifact installs and imports. The test
suite proves the code works; the smoke proves the wheel ships.

---

## 4. Tag-driven publish (rule WF-03)

The canonical publish trigger is a `v<X.Y.Z>` tag pushed to the
remote. Pushes to master without a tag MUST NOT publish.

### 4.1 Why tag-driven

The tag is the maintainer's signal "this commit is the release
point" (rule **REL-01**). It's intentional, durable, and
inspectable (`git show v0.X.Y` answers "what did 0.X.Y
contain?"). Push-to-master triggers are wrong: a merged PR is
not a release intent.

The trigger is the tag arriving on the server, not the commit
the tag points at. A maintainer can push commits freely; the
release happens only when they push a tag.

### 4.2 The full publish flow (operationalises REL-06 / PKG-04)

For `mgf-*` packages the upload + verify targets are the devpi
index, not pypi.org — see
[`PACKAGE_DISTRIBUTION.md`](PACKAGE_DISTRIBUTION.md) §5 (PKG-04) for
the canonical ordered ritual.

```
1. Tag-coherence check  — tag name v<X.Y.Z> matches
                          pyproject.toml::version AND __version__.
                          Mismatch aborts.
2. Build                — uv build (sdist + wheel, fresh in CI).
3. twine check          — metadata validates (the check is
                          index-agnostic; keep it).
4. Pre-publish smoke    — install wheel into /tmp venv, import + assert
                          __version__.
5. Upload               — twine upload to the devpi index
                          (--repository-url https://195.15.203.250.sslip.io/magogi/prod/),
                          TWINE_USERNAME = magogi, TWINE_PASSWORD = devpi_password
                          from the secret store.
6. Post-publish verify  — poll the devpi index (anonymous)
                          …/magogi/prod/+simple/<name>/
                          until <version> is listed (PKG-05); fail if the
                          polling window expires.
```

Each step's failure aborts the run. The published version becomes
resolvable only after step 5 succeeds; step 6 is the watchdog that
the upload actually landed. Like pypi.org (and unlike the old Codeberg
registry), the devpi prod stage is non-volatile — a version is immutable
once uploaded (PKG-07); a botched step 5 is recovered only by a
host-admin `devpi remove`, so get the build right before upload.

### 4.3 Tag-version coherence (rule WF-05)

The check is mechanical. In the workflow:

```bash
TAG_VERSION="${CI_COMMIT_TAG#v}"   # strip leading 'v'
PYPROJECT_VERSION="$(grep -E '^version = ' pyproject.toml | cut -d'"' -f2)"
PACKAGE_VERSION="$(python -c 'import <package>; print(<package>.__version__)')"

if [ "$TAG_VERSION" != "$PYPROJECT_VERSION" ] || [ "$TAG_VERSION" != "$PACKAGE_VERSION" ]; then
    echo "::error::Tag $CI_COMMIT_TAG does not match pyproject ($PYPROJECT_VERSION) or __version__ ($PACKAGE_VERSION)"
    exit 1
fi
```

This catches:
- Tagged before bumping the version files (tag = `v0.16.0`,
  pyproject still `0.15.0`).
- Bumped pyproject but not `__version__` (rule **AP-13** forbids
  this; the AP-13 sync test catches it locally; this catches
  it at CI time).
- Tagged the wrong commit (the maintainer ran `git tag v0.16.0
  HEAD~3` against a commit that didn't have the bump yet).

---

## 5. Token storage (rule WF-04)

Publish authentication is where shortcuts compound into breaches.
A leaked publish token authorises an attacker to ship a malicious
patch release that every consumer's `uv sync` / `pip install -U`
will pull. The authoritative model lives in
[`PACKAGE_DISTRIBUTION.md`](PACKAGE_DISTRIBUTION.md) §4 (PKG-03);
this section is its CI projection.

### 5.1 The answer: the devpi publish password in the CI secret store

`mgf-*` packages publish to the self-hosted devpi index, and
**devpi has no PyPI-style Trusted Publisher / OIDC** — so the
standing answer is a stored secret, not a fallback. Every
`pypa/gh-action-pypi-publish`, `id-token: write`, or
`docs.pypi.org/trusted-publishers/` reference is removed (PKG-03).

The credential is the **`magogi` devpi account's upload password**
(PKG-03), held in the Woodpecker **org** secret named
**`devpi_password`** and consumed as `TWINE_PASSWORD` (with
`TWINE_USERNAME` = the literal `magogi`). Install/read is anonymous —
no consumer token.

### 5.2 The discipline

- The token is held in the **CI platform's secret store**, never in
  the workflow file, never in environment variables passed at
  trigger time, never in chat.
- The secret is **restricted to tag events** where the platform
  supports it (Woodpecker: `events: [tag]` on the secret
  definition). A push to master cannot use the token even if
  the workflow tries (PKG-09).
- Scope: install/read needs **no credential** (anonymous), so only
  the publish password is sensitive — hold it as the tag-scoped org
  secret and nowhere else. (A dedicated CI-upload identity vs the
  admin identity that can `devpi remove` is a possible future
  hardening — PKG-03 / PACKAGE_DISTRIBUTION.md §4.)
- Token **rotation is part of the release-engineering rhythm**.
  Annual rotation minimum; immediate rotation on any leak signal
  (token appeared in a chat transcript, in a commit, in a
  screenshot — any of these counts as leaked).

### 5.3 Anti-patterns

| Anti-pattern | Why it's wrong | Fix |
|---|---|---|
| Token in workflow file | Anyone with read access to the repo can publish | Move to platform secret store |
| Token passed via env var at workflow trigger | Lands in CI logs; visible to anyone with run-history access | Use `from_secret` / equivalent |
| Baking the publish password into an install/CI step | Install needs no credential (anonymous); embedding the publish password where only reads happen widens the blast radius | Keep `devpi_password` to the tag-scoped publish step only; installs stay anonymous |
| Publish password shared via Slack / email / chat | Auditability gone; the leaked-password blast radius includes everyone in the channel | Keep it in the platform secret store; rotate on any leak signal |
| Token reused after a leak | The leaked token authenticates forever until revoked | Revoke immediately; issue fresh; never reuse |
| Reaching for OIDC / Trusted Publisher | devpi does not offer it; the reference is dead | Use the `devpi_password` CI secret (PKG-03) |

---

## 6. Matrix testing

Two matrix axes are in scope for most projects: **Python
versions** and **optional extras**.

### 6.1 Python-version matrix

A project that declares `requires-python = ">=3.11"` MUST test
on every supported version. A regression that only manifests on
3.11 (you used a 3.12-only stdlib feature) MUST be caught at CI
time, not by a downstream consumer's `pip install` failing.

The canonical version (the one that runs the coverage gate +
the gates that aren't version-sensitive — lint, typecheck) is
the project's lead version: typically the second-newest stable
Python at any moment.

### 6.2 Optional-extras matrix

A project that ships `[project.optional-dependencies]` MUST
test each extra in isolation AND in combination with the others
that consumers commonly request together. The base install
(no extras) MUST also be tested — a consumer who runs `pip
install <package>` and gets an `ImportError` because the
default code path imported an opt-in dep is the worst-case
break.

For `mgf-common` specifically: base + `[observability]` + each
service-shape extra (`[fastapi]`, `[db]`, `[alembic]`,
`[django]`, `[http]`) + `[dev]` for the test runners.

### 6.3 Matrix size and CI minutes

Matrix size is unbounded if you're not careful; 3 versions × 8
extra combinations × 4 gates = 96 jobs. Trim by:

- Running the typecheck / lint / import-linter gates on the
  canonical version only (they're version-agnostic).
- Running the optional-extras matrix on the canonical version
  only (they're extras-related, not version-related).
- Running the Python-version matrix with the *base* extras
  (not all combinations).

Result: lean matrix that catches the failure modes that matter
without burning a CI hour per push.

---

## 7. Reproducibility + pinning (rules WF-09, WF-11)

A CI run that produces different results on different days is
worse than no CI run — it teaches the team to ignore failures.

### 7.1 What MUST be pinned

| Component | How to pin |
|---|---|
| Python interpreter | `python:3.12.7-slim` (specific patch), not `python:3.12-slim` (rolling minor) when reproducibility-critical. Most projects accept rolling-minor for the security-update benefits. |
| Plugin actions / CI images | A specific tag, e.g. `zricethezav/gitleaks:v8.30.1` or an action at `@v1.2.3`, not `@v1` or `@latest`. (`mgf-*` publish does not use a Trusted-Publisher action — it uploads to the devpi index with the `magogi` account password; see §5.) |
| Base images (where supported) | By digest: `python:3.12-slim@sha256:abc123...`. Enforces byte-identical environments. |
| Dependencies | Lockfile (`uv.lock`, `poetry.lock`, `pip-compile` output) checked into the repo; CI installs from lock, not from `pyproject.toml` directly. |

### 7.2 What CANNOT be pinned

- **Time.** Tests that depend on `datetime.now()` MUST mock the
  clock or use frozen time (`freezegun`).
- **Network state.** Tests that hit external APIs MUST mock the
  network or use VCR-style cassettes.
- **CI-platform behavior.** The platform itself updates;
  workflow files survive across platform updates only if they
  use the documented stable interfaces.

### 7.3 The "works on my machine" anti-pattern

Every CI failure that "doesn't reproduce locally" is a signal
that the project's reproducibility is leaking. The right
response is not "rerun the failed job until green" — it's
"identify what's different and pin it". The fix is in the
workflow file, the lockfile, or the test code.

---

## 8. Failure modes + recovery

CI runs fail. The standard is about how loudly they fail and
how cleanly the team recovers.

### 8.1 The publish-step failure matrix

| Where it failed | What it means | Recovery |
|---|---|---|
| Tag-coherence check | Tag points at a commit that doesn't have the version bump (or vice versa) | Delete the tag, fix the version files, re-tag, re-push. The publish never happened — no registry side-effects. |
| Build | Packaging metadata broken (typed extras, py.typed missing, hatchling config) | Fix `pyproject.toml`; re-run. The publish never happened. |
| `twine check` | Metadata won't pass the validators | Fix the metadata; re-run. |
| Pre-publish smoke | The built wheel doesn't install or import cleanly | Fix the packaging; re-run. The publish never happened. |
| Upload (to devpi) | Network error, index 5xx, or — rare — the publish password rejected | Re-run. devpi is non-volatile: if the upload landed and the version already exists, the re-run fails with "version already exists" — treat as success (it's live), OR (per PKG-07) recover via a host-admin `devpi remove` if the artifact was botched and no consumer has resolved it. |
| Post-publish verify | Upload succeeded but the devpi index didn't surface the version in the polling window (PKG-05) | Wait + retry once. Persistent failure = a devpi incident; check the `…/magogi/prod/+simple/<name>/` index and the devpi host (`MGF_INFRA.md`). |

### 8.2 The cardinal recovery rule: a published version is immutable (non-volatile index)

The devpi `magogi/prod` stage is **non-volatile** — it rejects
re-uploading an existing `name+version` (like pypi.org, unlike the
old Codeberg registry). A botched upload is recovered **only** by a
host-admin `devpi remove` — for
botched-upload recovery within the same release session, before
any consumer could have resolved the version, and it MUST be
recorded (per [`PACKAGE_DISTRIBUTION.md`](PACKAGE_DISTRIBUTION.md)
**PKG-07**). Once a consumer may have resolved the version, treat
it as immutable: if the artifact is wrong (broken wheel, leaked
secret in the source distribution, wrong contents) the recovery is:

1. **Stop shipping** the bad version — for in-session botches,
   delete it (PKG-07) and re-upload the corrected artifact;
   for an already-resolvable version, leave it and move on.
2. **Bump** to the next patch / minor (`0.X.(Y+1)` or
   `0.(X+1).0`).
3. **Ship** the fixed version.

Never silently swap the bytes of a version others may hold. Per
**REL-05** / **PKG-07**.

### 8.3 The CHANGELOG-of-CI

When a CI step changes meaningfully (new gate added, gate
removed, threshold tightened), the workflow file MUST carry an
inline comment with the reason and the date. Future-you (or
your replacement) reading the workflow MUST be able to answer
"why is this step here?" without git-blame archaeology.

Example, from `mgf-common`'s `.woodpecker.yml`:

```yaml
# The 80% floor — lower than some consumer projects' 85% —
# reflects that mgf.common has a higher proportion of
# conditional sink code (Sentry, OTLP, syslog, journald, HTTP
# webhook) that can only be meaningfully exercised against real
# backends. ...
```

The comment carries the rationale; the value carries the rule.

### 8.4 The pipeline is a generated artifact (rule WF-13)

The `.woodpecker.yml` is not hand-authored per repo — it is
**rendered** from one canonical template
(`mgf-standard/templates/woodpecker/`: a `base.yml` plus opt-in
`frag-*.yml` fragments, and `base-zig.yml` for the Zig shape) by
`mgf-fed scaffold`. The renderer derives each repo's parameters
(import name, dist name, extras, coverage floor, console-script,
cornerstone/supervisor carve-outs) from its `pyproject.toml` +
on-disk layout, so one template change re-flows to the whole
fleet with a single regeneration sweep instead of 22 hand-edits
that inevitably drift apart.

The discipline this buys, and how it is enforced:

- **Never hand-edit `.woodpecker.yml`.** Change the template (or
  the repo's `[tool.mgf-fed]` carve-out inputs), then regenerate
  and commit the result. A hand-edit is overwritten on the next
  `scaffold` and flagged in the meantime.
- **`mgf-fed drift-check` enforces generation-currency**: the
  on-disk pipeline MUST byte-equal a fresh render. A divergence
  reports `[ci-generated] .woodpecker.yml differs from generated`
  with the fix (`run mgf-fed scaffold`). This is the WF-13
  mechanical check.
- **Legitimately-bespoke pipelines** that cannot be generated
  (e.g. a Zig build + multi-tier supervisor matrix, or a custom
  slow-suite tier) are listed in
  `policy.toml [exempt].woodpecker`. Those skip the
  generation-currency comparison and fall back to the
  **step-name list** (`woodpecker_required_steps` all present),
  so the standard gates are still enforced even though the file
  is owned by hand. `mgf-fed scaffold` leaves an exempt repo's
  pipeline untouched while still applying the other byte-exact
  shared files.

WF-13 governs that the pipeline *is generated and current*; it
says nothing about whether the pipeline actually *runs* — that
is the separate **operational activation step** (connecting the
repo to remote Woodpecker), which makes **WF-01** ("CI runs on
every push + PR") and **WF-02** ("the four gates block merge")
real. It is not a numbered rule of its own — see
[`../recipes/activate-ci.md`](../recipes/activate-ci.md). A repo
can be WF-13 green (pipeline present + current) while its remote
CI is still dormant (un-activated).

---

## 9. Mechanical-enforcement matrix

| Rule | Enforcement mechanism | Locus |
|---|---|---|
| WF-01 (CI on every push/PR) | `when: [event: push, event: pull_request]` in the workflow file; branch-protection rule on the platform side | `.woodpecker.yml` + Codeberg/Forgejo settings |
| WF-02 (four gates block merge) | Branch-protection rule requires the CI status check; workflow file enforces the four steps | Platform settings + `.woodpecker.yml` |
| WF-03 (tag-driven publish) | `when: event: tag` on the publish step; ref filter `refs/tags/v*` | `.woodpecker.yml` |
| WF-04 (secret in secret store) | `devpi_password` (the `magogi` upload password) never appears in workflow file; `from_secret` references; secret restricted to tag events; no OIDC/Trusted-Publisher (PKG-03/PKG-09) | Platform secret store + `.woodpecker.yml` |
| WF-05 (tag-version coherence) | Pre-build script in the workflow that compares tag to pyproject + `__version__`; aborts on mismatch | `.woodpecker.yml` |
| WF-06 (CI-built wheel only) | Wheel built in the publish run's own `build` step; the upload step references `dist/<pkg>-${CI_COMMIT_TAG#v}*` from this run only; same artifact checked + uploaded (PKG-04) | `.woodpecker.yml` |
| WF-07 (pre-publish smoke) | Smoke step between build and upload; ephemeral venv; aborts on import failure | `.woodpecker.yml` |
| WF-08 (post-publish verify) | Verify step after upload; polls the devpi `+simple/<name>/` index (anonymous) until `<version>` is listed (PKG-05); aborts on missing artifact | `.woodpecker.yml` |
| WF-09 (pinned versions) | Code review of workflow file; enforced by the absence of `latest` / `*` / unversioned image refs | Code review |
| WF-10 (parallel by default) | Workflow-graph design; sequential `depends_on` only where necessary | `.woodpecker.yml` |
| WF-11 (reproducibility) | Lockfile checked in; CI installs from lock; clock + network mocked in tests | Code review + test design |
| WF-12 (no silent failures) | Code review of workflow file; absence of `|| true`; explicit retry loops with logged context | Code review |
| WF-13 (generated pipeline) | `mgf-fed drift-check` compares the on-disk `.woodpecker.yml` to a fresh render of the canonical template (`mgf-fed scaffold` applies it); bespoke pipelines listed in `policy.toml [exempt].woodpecker` fall back to the step-name list | `mgf-fed` + `mgf-standard/templates/woodpecker/` + `policy.toml` |

---

## 10. Cross-references

- [`PACKAGE_DISTRIBUTION.md`](PACKAGE_DISTRIBUTION.md) — the
  canonical `mgf-*` distribution channel: the self-hosted devpi
  index, its endpoints, the publish-password auth model
  (**PKG-03**, anonymous read, no OIDC), and the `+simple/` verify protocol (**PKG-05**).
  WF-03/04/06/08 are the CI projection of **PKG-04/05/09**; this
  doc defers to it on every "where do we publish / how do we
  verify" question.
- [`RELEASING.md`](RELEASING.md) — the *what* and *why* of a
  release. This doc operationalises **REL-01** (tag discipline),
  **REL-04** (token storage), **REL-05** (release immutability),
  **REL-06** (release flow ordering), **REL-08** (pre-publish
  smoke) into mechanical CI steps.
- [`API_DESIGN.md`](API_DESIGN.md) — **AP-13** (`__version__` ↔
  `pyproject.toml::version` sync) is the static-side invariant
  this doc enforces at CI time via WF-05.
- [`SECURITY_STANDARD.md`](SECURITY_STANDARD.md) — **SC-10**
  (dependencies pinned + scanned in CI) maps directly onto a
  CI step. **SC-12** (vulnerability-reporting policy) names the
  contact path used when a CI run surfaces a CVE.
- [`TESTING.md`](TESTING.md) — defines the test suite that the
  WF-02 test gate runs. The test gate's coverage threshold is
  set per-project per the **TS-** rules.
- [`WORKFLOWS_PATTERNS.md`](WORKFLOWS_PATTERNS.md) — paste-ready
  Woodpecker snippets implementing every rule in this document.
  When starting a new project, copy from there; when designing
  a new pattern, file it back here.
- [`../internal/CI_STARTHERE.md`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/docs/release/ci.md) —
  developer-facing runbook: cut a release, set up CI for a new
  project, diagnose a red run. Action-oriented; this standard
  is the rationale, that doc is the recipe.

---

## 11. Self-review checklist

Before merging a workflow-file change, walk this list:

- [ ] **WF-01** — every long-lived branch (`master`, release
  branches) is in the trigger list.
- [ ] **WF-02** — the four gates are present and parallel.
- [ ] **WF-03** — if this project publishes, the publish step is
  tag-gated (`when: event: tag` AND `ref: refs/tags/v*`).
- [ ] **WF-04** — no tokens / passwords / API keys appear in the
  workflow file. The publish credential is the
  `devpi_password` org secret (the `magogi` upload password) consumed via
  `from_secret`; no OIDC / Trusted-Publisher reference (PKG-03).
- [ ] **WF-05** — the publish step runs the tag-coherence check
  before the build.
- [ ] **WF-06** — the wheel uploaded is the one built in this
  run. Build step is in the publish workflow, not a separate run.
- [ ] **WF-07** — pre-publish smoke installs the wheel into an
  ephemeral venv and asserts `__version__`.
- [ ] **WF-08** — post-publish verify polls the devpi `+simple/<name>/`
  index (anonymous) until `<version>` is listed, not pypi.org (PKG-05).
- [ ] **WF-09** — every external dep is pinned (image tag, action
  version, lockfile reference).
- [ ] **WF-10** — independent steps run in parallel; serial
  dependencies are explicit.
- [ ] **WF-11** — no host-machine state, no implicit network
  resources.
- [ ] **WF-12** — every step's failure mode is loud; no `|| true`
  without a comment justifying it.
- [ ] **WF-13** — the `.woodpecker.yml` was regenerated
  (`mgf-fed scaffold`), not hand-edited; `mgf-fed drift-check` is
  green (or the repo is an `[exempt].woodpecker` bespoke pipeline
  that still carries every required step).
