# Federation cohort-release wave recipe

> **Audience:** maintainer (human or AI agent) shipping a `mgf-common` minor bump that breaks any sibling's existing pin upper-bound.
> **Closes:** REL-10 / REL-11 / REL-12 in [`docs/standards/RELEASING.md`](../standards/RELEASING.md).
> **Distribution:** Phase B publishes to and verifies against the **self-hosted devpi index** (`magogi/prod` on `195.15.203.250`), per [`docs/standards/PACKAGE_DISTRIBUTION.md`](../standards/PACKAGE_DISTRIBUTION.md) (PKG-04 publish flow, PKG-05 `+simple/` verify). Neither `pypi.org` nor the Codeberg package registry is an `mgf-*` publish/resolution target.
> **History:** Codified 2026-05-26 after PAPER-84 (cross-sibling pin consistency) + PAPER-85 (sub-agent pytest permission gap) surfaced the gap; retargeted to the Codeberg registry 2026-06-03, then to the self-hosted devpi index 2026-07-01 (PAPER-112) — both per PACKAGE_DISTRIBUTION.md.

---

## When to run this recipe

You're shipping a federation-wide release wave when **any** of these is true:

- `mgf-common` is bumping a MINOR (e.g., 0.41 → 0.42) and at least one sibling library currently pins `mgf-common<X.Y` where `X.Y` is the old minor.
- A previously-broken-by-PAPER-84 cohort has been waiting for sibling-side pin walks.
- You're hitting `mgf-fed catch-up`'s cross-sibling-consistency check (PAPER-84) refuse a consumer's pin bump because a sibling's published pin excludes the target.

If **none** of the above applies, you don't need a cohort wave — just a normal single-package release.

---

## The two-phase split (REL-11)

```
Phase A — Edits (parallel, one sub-agent per sibling)
────────────────────────────────────────────────────
  pyproject.toml pin walks        (mgf-common + mgf-test-supervisor)
  __init__.py / _version.py bump  (sync against pyproject version)
  CHANGELOG.md entry              (consistent shape across siblings)
  uv sync --all-extras            (verify dep resolution)
                                  │
                                  ▼  hand back
Phase B — Ship (sequential, maintainer)
────────────────────────────────────────────────────
  pytest -q --tb=short            (verify cleanliness)
  git commit                      (federation-standard trailers)
  git tag -a v<X.Y.Z>             (annotated, per REL-01)
  git push + push tag             (Codeberg)
  rm -rf dist && uv build         (fresh wheel)
  uv publish --publish-url …      (devpi magogi/prod; magogi + devpi_password — PKG-04)
  verify via the devpi +simple/   (version listed = live — PKG-05)
```

**Why this split.** Phase A is mechanical work that scales linearly with sibling count — perfect for parallel agents. Phase B is the publish-safety boundary; verifying each sibling sequentially keeps a single failing pytest from leaking past the boundary into a published wheel. If you run Phase B in parallel, one sibling's test failure can race with another sibling's wheel upload + publish, and you've lost the invariant.

---

## Phase A — Parallel edit agents

For each sibling (mgf-http / mgf-fastapi / mgf-django / mgf-sqlalchemy / mgf-alembic / mgf-apiprobe — plus whatever else exists in the federation), spawn ONE sub-agent with the canonical prompt template below.

### Pre-wave ownership-boundary gate (REL-13's practical arm) — MANDATORY

Before anything else, run the fleet ownership-boundary gate from the
federation parent dir (all sibling checkouts present):

```bash
python3 mgf-standard/scripts/fleet_ownership_audit.py --check
```

Exit 0 is the bar. A RED gate means at least one repo's `SCOPE.md`
boundary block is stale against its live imports — releasing a wave on
top of stale boundary claims ships wrong ownership facts fleet-wide.
Fix first (`--inject` + commit in each stale repo, via the supervisor
per AG-13), then start the wave.

> **Why this lives here and not in CI (honest limitation):** the gate
> needs **every sibling checkout on one disk** to compute the fleet-wide
> import matrix. CI containers check out one repo at a time, so no
> `.woodpecker.yml` step can host it — which also means CI silence is
> NOT evidence the gate is green. The two places it is load-bearing are
> exactly the two places it must be run by hand: (1) here, before any
> cohort wave; (2) the standards-lock preconditions. If a future CI
> shape gets a full-fleet workspace, wire it there and drop this note.

### Pre-Phase-A audit (REL-10)

Before launching agents, verify no sibling needs source-level changes:

```bash
# Audit each sibling for usage of mgf-common's breaking-change names.
# Adjust the grep list to whatever's being deleted in the new minor.
DELETED='setup_logging|setup_otel|install_excepthook|build_parser'
for sib in mgf-http mgf-fastapi mgf-django mgf-sqlalchemy mgf-alembic mgf-apiprobe; do
  hits=$(grep -rn "$DELETED" /home/bassam/PycharmProjects/$sib/src/ 2>/dev/null)
  if [ -n "$hits" ]; then
    echo "--- $sib (AFFECTED) ---"
    echo "$hits"
  else
    echo "$sib: clean"
  fi
done
```

If any sibling is `AFFECTED`, that sibling needs **source migration** before the pin walk. Cohort-walk does NOT cover source-level migration; file that as its own paper.

### The canonical Phase-A agent prompt template (REL-12)

```markdown
You are releasing **<sibling-name> <new-version>** — a patch release that
walks the federation pins forward. <sibling-name>'s source has zero
usage of mgf-common's <new-mgf-common-minor> deletions (audit
verified).

**Repository:** `/home/bassam/PycharmProjects/<sibling-name>/`
**Current version:** <current-version>
**Target version:** <new-version>

**Three pyproject changes** in `<sibling>/pyproject.toml`:

1. **mgf-common pin walk:** `"mgf-common>=<old>,<<old-major>.<old-minor+1>"` →
   `"mgf-common>=<old>,<<new-major>.<new-minor+1>"`. The wider window
   accepts mgf-common <new>.x while keeping backward compatibility.
   Pre-v1.0 AP-03 discipline.

2. **mgf-test-supervisor swap (if still on git+ssh):** Replace
   `"mgf-test-supervisor @ git+ssh://..."` with
   `"mgf-test-supervisor>=<latest-devpi-ver>,<<next-minor>"`
   (resolve `<latest-devpi-ver>` from the devpi index, not
   pypi.org — PKG-11). Under devpi's single-index model the sibling
   resolves every `mgf-*` from the one `[[tool.uv.index]]`
   (`…/magogi/prod/+simple/`, `default = true`) — no per-package
   `[tool.uv.sources]` mapping is needed (PKG-06; see
   PACKAGE_DISTRIBUTION.md §6). Update the surrounding comment.

3. **Drop `[tool.hatch.metadata]` block** if present (no longer
   needed once the supervisor dep is a plain registry version pin
   resolved from the devpi index).

**Then:**

4. **Bump version** in pyproject.toml.
5. **Sync `src/<sibling>/__init__.py::__version__`** if present
   (test_version_in_sync.py pins this).
6. **CHANGELOG entry** — prepend before previous version's entry,
   matching the supplied block verbatim:

   ```markdown
   ## [<new-version>] — <date> — federation pin walks

   Federation cohort walk: <sibling> now accepts mgf-common
   <new-mgf-common-minor>.x. ...
   ```

7. **`cd <sibling> && env -u VIRTUAL_ENV uv sync --all-extras`.**
   Must resolve.

8. **Run pytest:** `env -u VIRTUAL_ENV uv run pytest -q --tb=short`.

   **PAPER-85 / REL-12 — graceful pytest-denial fallback:**
   If pytest invocation is denied (`Permission to use Bash has been
   denied`, sandbox restriction, etc.) on 3 successive attempts,
   STOP HERE. Report:
     - the file diff summary (each modified file + a one-line
       summary of changes).
     - "Phase A complete; pytest denied; ready for Phase B
       verification."
   Do NOT proceed to commit / tag / push / build / publish. The
   maintainer runs pytest in the parent session + drives Phase B
   from there.

9. **If pytest passes:** the maintainer is welcome to drive Phase
   B from here OR you may continue if explicitly authorised. Per
   REL-11, the default is to **stop after pytest succeeds** and
   hand back for Phase B serialization.

**Report back, under 350 words:** what changed in pyproject, the
new version + CHANGELOG snippet, pytest result (PASS / DENIED /
FAIL), and explicit handoff to Phase B.
```

### Spawn order

Spawn agents in parallel for siblings WITHOUT internal cross-deps. Hold one sibling back when it depends on another (e.g., mgf-alembic depends on mgf-sqlalchemy — but the dependency is unchanged by the cohort walk, so they can also run in parallel today). If in doubt, sequential is safe.

---

## Phase B — Sequential ship

For each sibling whose Phase A agent reported back "ready for verification" or "pytest PASS, awaiting handoff":

```bash
SIB=mgf-http  # one at a time
NEW_VER=0.4.3
SSH_KEY=/home/bassam/PycharmProjects/vm-vmanager-app/my_stuff/ssh_key/codeberg
# devpi publish (PKG-03): upload user `magogi` + the account's upload password.
# CI does this via the tag-scoped Woodpecker org secret `devpi_password`; a manual
# publish reads the password from a chmod-600 file, never pasted inline.
PW_FILE=/home/bassam/PycharmProjects/mgf-standard/my_stuff/tokens/devpi_password.txt
DEVPI_PASSWORD=$(tr -d '[:space:]' < "$PW_FILE")

cd /home/bassam/PycharmProjects/$SIB

# Step 1 — verify (skipped only if Phase A agent already ran pytest cleanly).
env -u VIRTUAL_ENV uv run pytest -q --tb=short

# Step 2 — commit.
git add pyproject.toml CHANGELOG.md uv.lock src/mgf/*/__init__.py 2>/dev/null
git commit -m "release: $NEW_VER — federation pin walks (...)

Federation cohort walk: $SIB now accepts mgf-common <new>.x.
...

Signed-off-by: Bassam Alsanie <bassam@magogi.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"

# Step 3 — push commit.
GIT_SSH_COMMAND="ssh -i $SSH_KEY -o IdentitiesOnly=yes" git push

# Step 4 — annotated tag (REL-01).
git tag -a v$NEW_VER -m "$SIB $NEW_VER — federation pin walks"
GIT_SSH_COMMAND="ssh -i $SSH_KEY -o IdentitiesOnly=yes" git push origin v$NEW_VER

# Step 5 — build (REL-08).
rm -rf dist/ build/
env -u VIRTUAL_ENV uv build

# Step 6 — wheel-metadata smoke (verify no direct-refs).
python3 -c "
import zipfile
with zipfile.ZipFile(f'dist/${SIB//-/_}-${NEW_VER}-py3-none-any.whl') as z:
    name = [n for n in z.namelist() if n.endswith('METADATA')][0]
    m = z.read(name).decode()
for line in m.split('\n'):
    if 'mgf-common' in line or 'mgf-test-supervisor' in line:
        print(line)
"

# Step 7 — publish to the devpi index (PKG-04; password never via chat — PKG-03).
# Upload to magogi/prod as user `magogi`; the upload password comes from the
# chmod-600 file ($DEVPI_PASSWORD set above), never pasted inline.
env -u VIRTUAL_ENV uv publish \
  --publish-url https://195.15.203.250.sslip.io/magogi/prod/ \
  -u magogi -p "$DEVPI_PASSWORD" \
  dist/${SIB//-/_}-${NEW_VER}-py3-none-any.whl dist/${SIB//-/_}-${NEW_VER}.tar.gz

# Step 8 — verify via the devpi +simple/ index (PKG-05; version listed = live).
# Anonymous read — no credential. No CDN: the version lists as soon as the upload commits.
until curl -fsS "https://195.15.203.250.sslip.io/magogi/prod/+simple/$SIB/" \
  | grep -q "${SIB//-/_}-${NEW_VER}"; do sleep 5; done
echo "$SIB $NEW_VER live on devpi."
```

Repeat for each sibling. **Do NOT parallelize Phase B** — the wall-clock cost is per-sibling × ~1-2 minutes (build + publish + devpi `+simple/` verify, no CDN wait); the safety gain is "one sibling's broken pytest doesn't race a parallel sibling's wheel upload."

---

## Failure modes + recovery

### Phase A agent reports "edits applied, pytest DENIED"

Expected per REL-12. Maintainer runs pytest in the parent session:

```bash
cd /home/bassam/PycharmProjects/<sibling>
env -u VIRTUAL_ENV uv run pytest -q --tb=short
```

If pytest passes: proceed to Phase B for that sibling.
If pytest fails: this is a real bug surfaced by the cohort walk. The pin widen exposes a latent incompatibility with the new mgf-common. STOP the cohort wave for this sibling, file a paper in the sibling's `FEEDBACK.md` (or `mgf-common`'s if cross-cutting), wait for the source-level fix, then restart Phase A for this sibling.

### Phase A agent reports "pytest FAIL — actual test failures, not denial"

Same response as above: real source-level incompatibility. File a paper.

### Publish rejected: wheel carries a `@ git+ssh://...` direct reference

Wheel metadata contains a `@ git+ssh://...` direct reference. Most likely the agent missed step 2 (the mgf-test-supervisor swap) or step 3 (the hatch metadata block). Inspect:

```bash
python3 -c "import zipfile; z=zipfile.ZipFile('dist/<wheel>'); print(z.read('<...>.dist-info/METADATA').decode())" | grep -i 'requires-dist\|provides-extra'
```

Fix the pyproject, rebuild, republish.

**Re-upload on devpi is blocked — the stage is non-volatile (PKG-07).** Like pypi.org (and **unlike** the old Codeberg registry), the `magogi/prod` stage **rejects** re-uploading an existing `name+version` ("409 / version already exists"). Botched-upload recovery is therefore a **host-admin op on the devpi server**, not a routine token/REST call — there is no API to DELETE a version:

```bash
# On the devpi host (admin), recover a not-yet-resolved botched upload:
#   devpi index magogi/prod volatile=True      # temporarily allow removal
#   devpi remove <sibling>==<version>          # drop the bad version
#   # re-run Phase B step 7 (uv publish --publish-url …)
#   devpi index magogi/prod volatile=False     # re-lock (append-only)
# — or, on a strictly append-only prod, just bump a fresh patch version instead.
```

This host-admin `devpi remove` is the PKG-07 escape hatch — botched-upload recovery only, recorded (CHANGELOG / release log), never a silent content swap of a version a consumer may already hold. If the broken version was already resolvable by anyone, bump a fresh patch instead. (PACKAGE_DISTRIBUTION.md §8 flags the exact recovery procedure — strictly append-only vs a volatile-toggle + `devpi remove` — as an open host-ops confirm.)

### devpi publish: no first-publish cooldown

There is **no 429 "too many new projects" cooldown on devpi.** A brand-new sibling package name publishes on the first try with no wait — devpi has no PyPI-style new-project rate limit. The only first-publish prerequisite is the `magogi` account's upload password (PKG-03); a 401/403 means the credential is missing/wrong, not rate-limited. Watch instead for the devpi host's package-volume (PKG-08) — retention prunes superseded versions before the host disk fills; there is no external cloud quota to raise.

### Verify-loop is fast — no CDN propagation lag

The devpi `+simple/<pkg>/` index lists the version as soon as the upload commits — there is no PyPI-style CDN where a `latest` view lags a versioned URL. The Phase B step 8 wait-loop normally returns on the first poll; a persistent miss means the upload did not land, not propagation lag.

---

## Coordination across the wave

For waves involving 5+ siblings, track state in a single file (e.g., `/tmp/cohort-wave-<date>.md`) with a one-line status per sibling:

```
mgf-http        — Phase A ✓ (pytest 103 passed), Phase B ✓, devpi 0.4.3 live
mgf-fastapi     — Phase A ✓ (pytest 153 passed), Phase B ✓, devpi 0.6.2 live
mgf-django      — Phase A ✓ (pytest DENIED → maintainer ran 55 passed), Phase B ✓, devpi 0.4.2 live
mgf-sqlalchemy  — Phase A ✓ (pytest 102 passed), Phase B ✓, devpi 0.4.2 live
mgf-alembic     — Phase A ✓ (pytest 42 passed), Phase B ✓, devpi 0.4.2 live
mgf-apiprobe    — Phase A ✓ (pytest 849 passed), Phase B ✓, devpi 0.5.2 live
```

The tracker is the single source of truth during the wave; close it out by archiving to `docs/inprogress/cohort-wave-<date>.md` if any sibling needed post-wave follow-up.

---

## Post-wave verification (REL-10's invariant)

After every sibling has published, run the PAPER-84 check across the federation:

```bash
# From mgf-common's source tree, verify the gate opens for all
# expected cohorts.
for cohort in 0.4 0.5 0.6; do
  echo "=== mgf-common >=$cohort,<<next> consumer simulation ==="
  # Create a tmpdir with a pyproject that pins one sibling at a
  # known version + mgf-common at the cohort window; run
  # catch-up and confirm no PAPER-84 conflict surfaces.
done
```

If the check still refuses bumps for any expected cohort, a sibling's pin walk was incomplete. Re-investigate that sibling's pyproject.

---

## Cross-references

- [`docs/standards/PACKAGE_DISTRIBUTION.md`](../standards/PACKAGE_DISTRIBUTION.md) — PKG-01..PKG-12; the self-hosted devpi index (`magogi/prod`) is the `mgf-*` publish/resolution contract (PKG-04 publish flow, PKG-05 `+simple/` verify, PKG-07 non-volatile immutability, PKG-08 host-disk retention) that Phase B follows.
- [`docs/standards/RELEASING.md`](../standards/RELEASING.md) — REL-01 through REL-12.
- [`FEEDBACK.md`](../../FEEDBACK.md) PAPER-84 — the cross-sibling consistency check that enforces REL-10.
- [`FEEDBACK.md`](../../FEEDBACK.md) PAPER-85 — the sub-agent pytest permission gap that codified REL-12.
- [`docs/cutover/v0.35-through-v0.41.md`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/docs/cutover/v0.35-through-v0.41.md) — the consolidated cutover guide that documents the migration consumers see when crossing a cohort-walk window.
- `mgf-fed catch-up --apply` — the consumer-side tool that PAPER-84's check gates.
