# Run CI — locally + remotely (`mgf-fed local-ci` / the generated pipeline)

> **Audience:** maintainer / agent who wants to run a repo's CI pipeline — the *full*
> gate set, not just the fast static gates — locally before pushing, and understands how
> the remote run is triggered.
> **Governs:** [`WORKFLOWS.md`](../standards/WORKFLOWS.md) **WF-13** (the pipeline is a
> generated artifact), **WF-11** (a CI run is reproducible / activating remote Woodpecker),
> **WF-02** (the four gates).
> **Tooling:** `mgf-fed local-ci` (wraps `woodpecker-cli exec`); `mgf-fed scaffold` /
> `drift-check` (generate + verify the pipeline); `mgf-fed release-local` /
> `release-remote` (the two-mode release).
> **History:** Codified 2026-06-14 alongside the CI-unification (generated `.woodpecker.yml`).

---

## The one-paragraph model

Every repo's `.woodpecker.yml` is a **generated artifact** (WF-13), rendered from the
canonical template by `mgf-fed scaffold`. You run it **locally** with
`mgf-fed local-ci <repo>` (which executes the real pipeline in containers via
`woodpecker-cli exec`), and **remotely** by pushing / opening a PR — once remote Woodpecker
is activated (WF-11; today the pipelines are present-but-dormant). A release is the same
pipeline plus the publish path: `mgf-fed release-local` → `mgf-fed release-remote`.

## 1. The pipeline is generated — don't hand-edit it (WF-13)

The `.woodpecker.yml` in each repo is **not** authored by hand. It is rendered from the
single canonical template (`mgf-standard/templates/woodpecker/`: `base.yml` + opt-in
`frag-*.yml` fragments; `base-zig.yml` for the Zig shape, which carries the same opt-in
`fedres-check` fragment marker — gated on the repo's `fed-resources/source/project.toml`) by:

```sh
mgf-fed scaffold <repo>          # (re)write the repo's .woodpecker.yml from the template
mgf-fed scaffold --dry-run <repo># preview without writing
```

`mgf-fed drift-check` enforces **generation-currency** — the on-disk file MUST byte-equal a
fresh render — and reports `[ci-generated] .woodpecker.yml differs from generated` when it
drifts. So:

- **Change the template (or the repo's `[tool.mgf-fed]` inputs), then regenerate** — never
  edit `.woodpecker.yml` directly (the edit is overwritten on the next `scaffold` and flagged
  by `drift-check` in the meantime).
- **Commit the regenerated file** in the same change.
- **Legitimately-bespoke pipelines** (e.g. `mgf-hrb`'s Zig + multi-tier supervisor matrix,
  `mgf-test-supervisor`'s slow-suite tier, `mgf-website-template`'s hand-authored
  cookiecutter pipeline) are listed in `policy.toml [exempt].woodpecker`: `scaffold` leaves
  them untouched, and `drift-check` holds them to the **step-name fallback**
  (every `woodpecker_required_steps` present) instead of a byte-identical render.

## 2. Run CI locally

```sh
mgf-fed local-ci mgf-common            # run one repo's real pipeline in containers
mgf-fed local-ci mgf-http mgf-auth     # several
```

This wraps `woodpecker-cli exec`, so it runs the **full** gate set each step installs its
own deps (lint / typecheck / the pytest matrix / vuln-scan / sbom / build / smoke) — the
dependency-heavy complement to the fast static `mgf-fed gates`.

**Requirements + caveats** (inherent to local exec):

- **Docker must be running** and `woodpecker-cli` (or `woodpecker`) on `PATH`
  (<https://woodpecker-ci.org/docs/cli>). `local-ci` exits `2` if either is missing.
- **Tag-gated steps are skipped** without a tag — `publish` / `verify-devpi` only fire on
  a `v<X.Y.Z>` tag, so a plain `exec` validates everything *up to* publish. Use the release
  recipes for the publish path.
- **Pulling private `mgf-*` deps** inside a step needs no credential — `mgf-*` resolve
  **anonymously** from the devpi index (`--index-url …/magogi/prod/+simple/`).
- **Server-level variables are NOT forwarded — steps that guard on `$DEVPI_URL` red
  deterministically.** `DEVPI_URL` is a Woodpecker **server-level** variable (PAPER-113;
  `MGF_INFRA.md` §2), injected by the remote server. `local-ci` shells
  `woodpecker-cli exec` with **no `--env` flags**, so the variable is absent inside the
  container even when it is exported in your shell — and every generated step that opens
  with `[ -n "$DEVPI_URL" ] || { …; exit 1; }` (typically `app-doctor`,
  `cross-layer-audit`, `fedres-check`, plus the tag-gated `publish` / `verify-devpi`)
  fails on its guard line. **A red confined to exactly those steps, on the guard message,
  is the expected local result — not a regression**, and it means a full-green local
  `exec` is not achievable today on a repo carrying them. Run them by hand with
  `DEVPI_URL` exported, and declare the gap in the repo's `## CI delta` section
  ([AG-21](../standards/AGENTS.md#AG-21)). **Tracked fix:** teach `local-ci` to forward
  the variable (`--env DEVPI_URL=…`); supervisor-owned per
  [AG-13](../standards/AGENTS.md#AG-13).
- **Shared-workspace caveat (parallel supervisor steps).** `woodpecker-cli exec` runs the
  steps against a **single shared workspace** on one host, rather than the isolated
  per-step volumes a remote multi-runner backend gives you. Pipelines whose parallel steps
  write overlapping paths — notably the **supervisor tiers** in `mgf-hrb` /
  `mgf-test-supervisor` (which fan out tier-`0-2` / tier-`0-3` in parallel) — can collide on
  that shared workspace under local `exec`. The generated template serialises the supervisor
  steps with `depends_on` to make local exec deterministic; if you add new parallel steps that
  share state, expect to serialise them too (or rely on the remote backend's isolation).

## 3. Run CI remotely

Remote CI is **trigger-by-push**: pushing to a long-lived branch or opening a PR runs the
pipeline on Woodpecker (WF-01). **This is gated on WF-11 — activating remote Woodpecker.**
Today the generated pipelines are **present-but-dormant**: the `.woodpecker.yml` is committed
and current in every repo, but the remote server isn't wired up yet, so the canonical proof
of a green run is still `mgf-fed local-ci` until WF-11 lands. Nothing else changes when remote
CI activates — the *same generated pipeline* simply starts running on push/PR.

### ~~Known transient: a clone-only red~~ — ROOT-CAUSED AND FIXED (2026-07-11)

> **This is no longer a transient.** The clone credential was Woodpecker's netrc, derived
> from a **one-hour Forgejo OAuth token**; a pipeline starting in the expiry/refresh window
> got a dead credential for the step's whole lifetime (`could not read Username`, exit 128).
> Worse, Woodpecker refreshes that token *by calling Codeberg* — which **rate-limits us**
> (PAPER-110) — so our own fleet automation was starving the refresh and breaking our own
> clones. The canonical templates now **pin the clone credential** to the long-lived
> `codeberg_clone_token` secret. Proven both ways (mgf-brand #10 fails on a wrong
> credential with the exact signature; #11 green with the pinned token).
> **If you still see a clone-only red after the re-render wave, it is a NEW bug — investigate it.**

<details><summary>Historical: the old "restart it" guidance (pre-fix)</summary>

### Known transient: a clone-only red

Occasionally a remote run reds on the **`clone` step alone**, with `fatal: could not
read Username for 'https://codeberg.org': No such device or address` (exit 128) — the
clone got no credentials, so every downstream step is skipped and the pipeline reds
through no fault of the change. This is a **known transient** (a credential-injection
flake, correlated with a Woodpecker agent restart window), and is **distinct from
WF-09's treeless-partial-clone 504** that `partial: false` already fixes. **A plain
restart of the pipeline clears it** — don't re-investigate from scratch. (Filed as
FEEDBACK.md **PAPER-109**; if it turns frequent, bounded-retry-wrap the base clone.)

### Known signature: merge-moment concurrency clobber (re-run solo before diagnosing)

Merging a PR spawns **two pipelines at once** on the same commit — the merged head's PR
pipeline and the new `main`'s push pipeline — which share a workspace on the runner. The push
run can execute in an **empty workspace after a green clone**, so several unrelated steps red
at once with plausible *file-not-found* errors (`No pyproject.toml found`, `open .gitleaks.toml:
no such file`, commit-trailers/lint failing the same way). **This is not a bad commit.**

- **Signature:** multiple, unrelated steps failing with "file not found" **immediately after a
  merge**, while a sibling pipeline for the same commit is green. (FEEDBACK **PAPER-124** / CIR-07.)
- **Interim remedy:** **re-run the reded pipeline solo** — it passes on a quiet workspace. Do not
  re-fix already-correct code.
- **Durable fix (tracked):** per-pipeline isolated workspace volumes on the agents
  (`mgf-cloud-provision`); until then this note + the agent2 watcher flagging concurrent
  same-repo pipelines.

## 4. Release (the pipeline + the publish path)

A release runs the same pipeline with the tag-gated publish steps active, in two modes:

```sh
mgf-fed release-local  <repo> …   # build once → validate against the local index → publish there
mgf-fed release-remote <repo> …   # promote the SAME wheel to the devpi index (tag → publish → verify-devpi)
```

Always `release-local` first (the QA / pre-promotion gate), then `release-remote` to promote
the build-once wheel (REL-02). See [`local-release.md`](local-release.md) and
[`remote-release.md`](remote-release.md) for the full per-stage breakdown.

## See also

- [`../standards/WORKFLOWS.md`](../standards/WORKFLOWS.md) — WF-01..WF-13 (the CI standard;
  WF-13 = generated pipeline, WF-11 = reproducible / remote activation).
- [`local-release.md`](local-release.md) / [`remote-release.md`](remote-release.md) — the
  two-mode release this pipeline feeds.
- [`../standards/WORKFLOWS_PATTERNS.md`](../standards/WORKFLOWS_PATTERNS.md) — the canonical
  pipeline snippets the template is built from.

</details>
