# FEEDBACK — mgf-http

> Paper-tracking file for issues, friction, and requests against
> this sibling. Consumer projects (PlasmaMapper, inthewords,
> vm-vmanager-app, figrecovery, and any future adopter) file PAPER-NN
> entries here.
>
> **Where to file what:**
>
> - **mgf-http-specific** (transport behaviour, retry policy,
>   redaction, streaming) → file HERE.
> - **Federation-wide** (cross-sibling patterns, standards
>   proposals) → file in
>   [mgf-common/FEEDBACK.md](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/FEEDBACK.md).
>
> **Numbering convention:** paper numbers (`PAPER-NN`) are
> local to this repo. When a paper closes elsewhere or
> cross-references a federation-wide paper, the federation
> number appears in the metadata table as a cross-reference row.
>
> **Discipline:** see
> [mgf-standard/docs/standards/DOCUMENTATION.md](https://codeberg.org/magogi-admin/mgf-standard/src/branch/main/docs/standards/DOCUMENTATION.md)
> §FEEDBACK (rules FB-01..FB-04, DOC-13).

---

## §1 Process

1. **File freely.** Any consumer hitting friction MUST file
   a PAPER-NN under §2. Don't wait for permission; the
   maintainer triages.
2. **Status flips** through `OPEN` → `SHIPPED` / `DEFERRED` /
   `WONTFIX` / `MOVED` as the paper resolves. A `SHIPPED` row
   carries a commit-SHA reference + release-version reference.
3. **Renumbering** never happens within this repo — papers
   keep their PAPER-NN forever (per FB-01).

---

## §2 Open feedback (active)

#### `PAPER-01` — Streaming + body cap absent from `AsyncHttpClient.request()`

| Field | Value |
|---|---|
| Federation paper | [PAPER-36 in mgf-common](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/FEEDBACK.md#paper-36) (original filing — sibling-scoped per DOC-13) |
| Status | ✅ SHIPPED |
| Severity | 🟡 High |
| Submitter | inthewords (commit [`36cd832`](https://codeberg.org/magogi-admin/inthewords/commit/36cd832)) |
| Submitted | 2026-05-18 |
| Decision | accept (both Approach A and Approach B, sharing one internal implementation) |
| Decided on | 2026-05-18 |
| Scheduled for | mgf-http v0.3.1 (next minor) |
| Shipped in | mgf-http commit [`4722bb3`](https://codeberg.org/magogi-admin/mgf-http/commit/4722bb3) |

**Concern.** `AsyncHttpClient.request()` returns a fully-buffered
`httpx.Response`. There is no `stream=True` flag, no
`max_response_size` constructor parameter, and no
`AsyncHttpClient.stream()` convenience method. A malicious or
misconfigured upstream that responds with N MB of body forces
httpx to buffer the full N MB into memory before the wrapper
hands the response back. The 5s `timeout` bounds wall-clock, not
bytes — on a fast link the practical cap is bandwidth × timeout
(gigabits/sec × seconds = potentially hundreds of MB).

**Why it matters.** Any consumer fetching user-submitted URLs
(link previews, OG-metadata scrapers, webhook URL verifiers,
citation enrichers) needs a body-size cap as a SSRF/DoS guard.
The library forces a binary choice today: use `.request()` for
retry + redaction + typed `HttpTransportError` translation OR
drop to the `.client` escape hatch for streaming and lose all
three. The `_client.py:246-247` docstring explicitly calls this
out as a downside.

**Concrete evidence.** inthewords' citation-metadata fetcher
caps body size at 1 MB (`CITATION_FETCH_MAX_BYTES = 1_000_000`).
Pre-adoption (urllib): `resp.read(N)` was a single line. Post-
adoption (`apps/messaging/tasks.py:_fetch_url_async`, commit
[`36cd832`](https://codeberg.org/magogi-admin/inthewords/commit/36cd832)):
the body fetch had to be split — `client.request()` for the
redirect-walk (kept for the retry / typed-error benefits), then
`client.client.stream(...)` (the escape hatch) for the actual
body read. The hop through the escape hatch silently bypasses
`RetryPolicy` and the `HttpTransportError` wrap, forcing a
separate `except httpx.HTTPError` catch downstream.

**Suggested shape.** Two non-exclusive proposals.

**Approach A — constructor-level cap.** Add
`AsyncHttpClient(max_response_size: int | None = None)`. When
set, the wrapper streams the response under the hood and raises
`HttpTransportError` with a new
`cause_summary="response-too-large"` if the cap is exceeded.
Default `None` preserves current behaviour. Lowest-effort
adoption: consumers add one kwarg and the wrapper does the
right thing.

**Approach B — streaming variant on the public surface.** Add
`AsyncHttpClient.stream(method, url, ...)` that yields the
response under an async context manager AND keeps the wrapper's
retry policy + typed-error translation. Body-cap becomes an
idiom the consumer drives (`break` out of the
`aiter_bytes` loop), but the wrapper still owns the protocol-
level posture.

**Recommendation.** Both. Approach A is the right default for
"I don't care, just cap it"; Approach B is the right shape for
consumers needing streaming-as-a-feature (large downloads,
chunked metadata extraction). They share an internal
implementation — both wrap `httpx.AsyncClient.stream()` with the
mgf-http envelope.

**Decision rationale.** Both approaches shipped as recommended.
The submitter's framing (Approach A is the simpler "just cap it"
shape; Approach B is the lower-level "I want streaming control"
shape) matched the implementation: `request()` with
`max_response_size` set routes through `stream()` internally —
single implementation of the streaming + retry loop, two public
surfaces. The existing `_send_with_retries` (the buffered path)
stays unchanged, preserving 71 existing tests; the new code path
is covered by 16 new tests across two classes
(`TestMaxResponseSizeConstruction` + `TestMaxResponseSize` +
`TestStream`).

**Retrospective.** Shipped exactly as suggested. The one
notable in-flight refinement: `_log_response` reads
`response.elapsed`, which httpx only sets after the response is
closed — moved the log call to AFTER the inner `async with
self._client.stream(...)` exits in the streaming path. The
shared cookie header-injection guard (SH-01 / RA-03) was
extracted into a new private helper `_assemble_request_headers`
to keep cookie validation in one place across both surfaces.
Total surface: +16 tests, +1 ctor kwarg, +1 public method, +1
private helper, no breaking changes.

---
#### `PAPER-02` — SSRF-safe redirect-walk recipe missing from `docs/recipes/http.md`

| Field | Value |
|---|---|
| Federation paper | [PAPER-37 in mgf-common](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/FEEDBACK.md#paper-37) (original filing — sibling-scoped per DOC-13) |
| Status | ✅ SHIPPED |
| Severity | 🟢 Nice-to-have |
| Submitter | inthewords (commit [`36cd832`](https://codeberg.org/magogi-admin/inthewords/commit/36cd832)) |
| Submitted | 2026-05-18 |
| Decision | accept (suggested shape verbatim) |
| Decided on | 2026-05-18 |
| Scheduled for | bundled with PAPER-36 in mgf-http v0.3.1 |
| Shipped in | mgf-http commit [`4722bb3`](https://codeberg.org/magogi-admin/mgf-http/commit/4722bb3) — `docs/recipes/http.md` |

**Concern.** mgf-http correctly leaves SSRF posture to the
consumer — TLS-on-by-default and a one-shot `verify_tls=False`
warning are the only SSRF-adjacent primitives. But the
consumer-side recipe for "https-only-on-every-hop redirect
walking" is non-trivial: set `follow_redirects=False`, walk
the `Location` header manually, re-check the scheme on each
hop, resolve relative URLs via `urljoin`, cap the redirect
depth, translate redirect loops to a typed error. The recipe
docs cover the happy path but not this shape.

**Why it matters.** Any web-shaped consumer fetching user-
submitted URLs reinvents this loop. inthewords just did; future
consumers will too unless the shape is documented in one place.

**Concrete evidence.** ~50 lines of `_fetch_url_async` in
`apps/messaging/tasks.py` (inthewords commit
[`36cd832`](https://codeberg.org/magogi-admin/inthewords/commit/36cd832))
implement the canonical pattern. The pattern is straightforward
once written but takes 30-60 minutes to derive from first
principles + httpx docs.

**Suggested shape.** Add a section to
`docs/recipes/http.md` titled "SSRF-safe redirect walking",
showing the canonical pattern:

```python
async with AsyncHttpClient(follow_redirects=False, ...) as client:
    current_url = url
    for _ in range(MAX_REDIRECTS + 1):
        if not current_url.startswith("https://"):
            return None  # consumer-chosen failure mode
        response = await client.request("GET", current_url)
        if response.status_code in (301, 302, 303, 307, 308):
            location = response.headers.get("location") or ""
            current_url = urljoin(current_url, location)
            continue
        # ... handle non-redirect response ...
        break
    else:
        return None  # too many redirects
```

A reference consumer (inthewords) is now available to cite from
the recipe; the citation fetcher is the worked example.

**Decision rationale.** Bundled with PAPER-36 since both touch
the same recipe doc and surface the same body-cap + redirect-walk
pattern. The recipe lands as a new "SSRF-safe redirect walking"
section in `docs/recipes/http.md` (mgf-http) alongside the new
"Bounded response sizes — `max_response_size`" and "Streaming
responses — `.stream()`" sections from PAPER-36.

**Retrospective.** Shipped verbatim per the suggested shape. The
recipe cites inthewords' `apps/messaging/tasks.py:_fetch_url_async`
as the worked production example, closing the "future consumers
will reinvent this loop" loop the filing called out.

---

#### `PAPER-03` — README quickstart doesn't surface that 4xx / 5xx return as Response (not raised); consumers bounce to PUBLIC_API.md to confirm

| Field | Value |
|---|---|
| Status | ✅ SHIPPED |
| Severity | 🟢 Nice-to-have |
| Submitter | inthewords (commit [`36cd832`](https://codeberg.org/magogi-admin/inthewords/commit/36cd832)) |
| Submitted | 2026-05-18 |
| Decision | accepted |
| Decided on | 2026-07-01 |
| Scheduled for | next mgf-http tag (≥ v0.4.12) |
| Shipped in | commit `251c163` (merged via PR #33, `0dc0d36`) — README quickstart blockquote; on `main` now, rides the next tag |

**Concern.** The README quickstart shows
`AsyncHttpClient(base_url=..., ...).get("/users")` — the `.get()`
convenience shortcut + `base_url` kwarg. That works perfectly,
but readers landing on the README also need to know two
non-obvious facts that aren't in the quickstart:

1. `.request(method, url, ...)` is the underlying generic verb
   (used for non-GET cases and any consumer that wants the
   `RetryPolicy` / typed-error envelope around an arbitrary
   method).
2. 4xx and 5xx responses are NOT raised automatically — they
   return as `httpx.Response`. The caller invokes
   `response.raise_for_status()` explicitly if they want a
   typed exception.

Both are documented in `PUBLIC_API.md` but not surfaced in the
README quickstart.

**Why it matters.** During the inthewords citation-fetcher
adoption (commit
[`36cd832`](https://codeberg.org/magogi-admin/inthewords/commit/36cd832)),
I bounced between README and `PUBLIC_API.md` three times to
confirm "does a 404 raise, or do I get a Response back?" The
answer is "Response back" but it took reading `PUBLIC_API.md` to
be sure. The library's actual behaviour is the right one (httpx
convention; gives the caller control over status-class handling)
— but a reader following the quickstart literally and writing
`response = await client.request(...)` then `response.json()`
without a status check is silently parsing the error-page HTML
as JSON.

**Concrete evidence.** README quickstart (today) ends at
`.get("/users")` with no mention of the non-2xx contract.
`PUBLIC_API.md` covers it explicitly but consumers don't read
`PUBLIC_API.md` cold — they read the README, write code, and
hit the surprise.

**Suggested shape.** Two-sentence addition to the README
quickstart, immediately after the example:

```markdown
> `AsyncHttpClient` returns 4xx / 5xx responses as
> `httpx.Response` — they are NOT raised. Call
> `response.raise_for_status()` if you want a typed exception,
> or branch on `response.status_code` to handle status classes
> explicitly. Transport-level failures (connection refused,
> timeout, DNS failure) DO raise `HttpTransportError`.
```

Cost: 4 lines of README prose. Saves every new consumer the
three-round-trip "wait, does this raise?" investigation.

Optional polish: link "the underlying `.request()` verb is
documented in [PUBLIC_API.md](docs/PUBLIC_API.md#request)" so
the reader knows where to go for the non-GET shapes.

**Decision rationale.** Accepted as filed (🟢, near-zero cost, high consumer value). Added the paper's suggested clarification to the README quickstart — a blockquote stating the non-2xx contract (4xx / 5xx return as `httpx.Response`, not raised; call `raise_for_status()` or branch on `status_code`), that transport failures raise `HttpTransportError`, and that `.request()` is the generic verb behind `.get()`. Wording verified against `src/mgf/http/_errors.py` + `_download.py`.

**Retrospective.** Shipped in commit `251c163` ("docs(readme): surface the non-2xx contract in the quickstart (PAPER-03)", merged via PR #33).

- **Resolution (2026-07-10 queue sweep — verified in repo):** the
  blockquote lives in `README.md` immediately after the quickstart
  example ("**Non-2xx responses return; they don't raise.** …
  Transport-level failures … **do** raise `HttpTransportError` …
  The generic verb behind `.get()` is `.request(method, url, ...)`").
  All three facts the paper asked for are surfaced. The change is on
  `main`; the repo is still at v0.4.11, so the "next tag ≥ v0.4.12"
  condition is satisfied automatically when the next release cuts —
  docs-only, nothing further to do. Status flipped to SHIPPED.

---

#### `PAPER-04` — Default `mgf-common` pin window is too tight (single minor) and blocks consumers from federation-contract bumps

| Field | Value |
|---|---|
| Status | ✅ SHIPPED (AP-16 codified in mgf-standard; mgf-http's own wide pin + compat-matrix landed) |
| Severity | 🟡 High |
| Submitter | figrecovery (commit [`50ac5b8`](https://codeberg.org/magogi-admin/figrecovery/commit/50ac5b8)) |
| Submitted | 2026-05-18 |
| Decision | accept Approach A, formalized fleet-wide as rule **AP-16** (federation pin-window): wide window `>=<floor>,<1.0` + a mandatory compatibility-matrix CI + a cornerstone-release ping (B-lite). Upper bound `<1.0` (fully CI-gated); NOT Approach B (couples release cadences), and the heavier Approach C (a `federation-windows.toml`) deferred as the aspirational end-state. |
| Decided on | 2026-07-01 |
| Scheduled for | AP-16 shipped in mgf-standard (PR #181). Enforcement + rollout — the compat-matrix CI template + the ping script + the fleet cohort wave — is the follow-up phase. |
| Shipped in | The discipline: mgf-standard **AP-16** / `API_DESIGN.md` §4.4 (PR #181). mgf-http's own wide pin + compat-matrix ship with the cohort rollout. |

**Concern.** `mgf-http` pins `mgf-common` to a single MINOR window — 0.4.0 / 0.4.1 both said `mgf-common>=0.36,<0.37`; 0.4.2 widened to `>=0.36,<0.39`. The tight pin is defensible per AP-03's "0.x can break per MINOR" discipline, but in practice mgf-http's import surface against mgf-common is narrow (AppError, OperationError, observability hooks, Settings types — none of which changed across 0.36 / 0.37 / 0.38). Pinning a single minor when nothing actually breaks creates federation-wide friction: every mgf-common MINOR bump that happens to coincide with a federation-contract change forces a new mgf-http release JUST to widen the pin, and downstream consumers can't catch up until that release lands.

**Why it matters.** Real federation cascade observed this session:

1. mgf-common 0.38.0 ships the v2.10 federation contract (FEDERATION.md / DOC-16 / DOC-17) — no breaking changes to the API surface that mgf-http uses.
2. figrecovery + mgf-usb both want to pin `mgf-common>=0.38,<0.39` to track the contract.
3. `uv sync` fails: `mgf-http==0.4.1 depends on mgf-common>=0.36,<0.37, we can conclude that mgf-http==0.4.1 cannot be used.` Two consumers blocked on a transitive dep.
4. Cut a new mgf-http release (0.4.2) widening the pin to `<0.39`. No source change. Pure pin-window maintenance.
5. Re-`uv sync` on the consumers; everything resolves.

Total cycle ~30 minutes of operator time for what was a docs-only mgf-common bump. The same cascade will recur on every mgf-common MINOR until mgf-http's pin-window discipline relaxes. Worth noting: every other sibling carries the same convention, so the cascade is shaped (not just a one-off).

**Concrete evidence.** mgf-http git history showing the rhythm:

- [`bd33322`](https://codeberg.org/magogi-admin/mgf-http/commit/bd33322) — v0.4.1 (2026-05-18, morning) — federation sync docs only; pin still `<0.37`.
- [`c0196b3`](https://codeberg.org/magogi-admin/mgf-http/commit/c0196b3) — v0.4.2 (2026-05-18, evening) — pin widened to `<0.39`. No source change. Commit message: "Pin: `mgf-common>=0.36,<0.37` → `mgf-common>=0.36,<0.39`. No source changes."

Two releases in the same day, both of which exist solely to widen the pin window once mgf-common moved. The shape says: today's discipline produces a maintenance-tax release per mgf-common MINOR.

**Suggested shape.** Approach A vs Approach B vs Approach C:

- **Approach A — widen the default pin window to N minors (e.g. 3).** Set `mgf-common>=X,<X+3` at every mgf-http release. Periodic compatibility-matrix CI verifies the wider window still works. When mgf-common ships a real breaking change in mgf-http's import surface, that's when mgf-http cuts a release (with a re-tightened window for the consumer's protection). Today's tight window pays the cost of every release whether or not anything broke; a wider window pays the cost ONLY when something actually breaks.
- **Approach B — automated sibling-release on mgf-common MINOR.** mgf-common's release process triggers a CI job in every sibling that bumps the upper-bound + cuts a patch release. Removes the operator-time burden but introduces a release-coordination dependency between mgf-common and every sibling.
- **Approach C — "federation-window" file at mgf-common root.** mgf-common publishes `federation-windows.toml` declaring, per MINOR release, which sibling versions are known-compatible. Siblings (or consumers) read this to derive their effective pin without re-cutting a release per mgf-common MINOR. Heavier upfront design; eliminates both per-MINOR releases AND the consumer-bumps-the-pin step.

**Recommendation:** Approach A as the immediate ship — least invasive, immediate relief. Approach C as the aspirational direction if/when there's appetite for a coordination layer. Approach B is theoretically clean but couples release cadences in a way that's likely to introduce more friction than it removes.

**Decision rationale.** Accepted Approach A, formalized fleet-wide as **AP-16** (mgf-standard, `API_DESIGN.md` §4.4) rather than a mgf-http-local pin change, because the tax is federation-wide (every sibling pins the cornerstone). Grounding survey (2026-07-01): the fleet had already drifted to a wide `>=<floor>,<0.50` pin ad-hoc — but tied to the cornerstone (all ~27 siblings re-walk at mgf-common 0.50), un-codified (§4.3's example prescribed the opposite), and un-verified. AP-16 makes it `<1.0` (fully CI-gated), codified, and safe via the mandatory compat-matrix (a real break reds CI → re-tighten + release) + the B-lite ping (cornerstone release → sibling matrix, so a break surfaces before a consumer pulls the untested combo). The accepted residual risk (the narrow window before the ping) is documented in §4.4; Approach C is the deferred end-state if that window ever bites.

**Retrospective.** The rollout the paper was pending on has landed
for both halves.

- **Resolution (2026-07-10 queue sweep — verified in repo +
  standard):**
  1. **The discipline** — mgf-standard `docs/standards/API_DESIGN.md`
     rule **AP-16** is live as a MUST ("Federation siblings MUST pin
     the `mgf-common` cornerstone with the wide window
     `>=<floor>,<1.0` … backed by a mandatory compatibility-matrix
     CI"), with the full §4.4 treatment. It explicitly cites this
     paper (PAPER-04) as the motivating evidence — the standard
     supersedes any local pin decision here.
  2. **mgf-http's own rollout** — `pyproject.toml` now carries
     `mgf-common>=0.41,<1.0` (commit `19e97db`, "chore(ap16): widen
     mgf-common pin to <1.0 + compat-matrix", merged via PR #35
     `8ada116`, with the clamped matrix floor in `651b490`), and
     `.woodpecker.yml` carries the AP-16 mgf-common compat-matrix
     lane. The AP-16 conformance-ledger row landed in `75bdd74`.
  The exact cascade the paper documented (a new mgf-http release per
  mgf-common MINOR) can no longer occur inside the `<1.0` window.
  Status flipped to SHIPPED.

---

## §3 Reviewer's meta-feedback

### From vm-vmanager-app (consumer of mgf-common, planned Phase 5 consumer of mgf-http) — 2026-05-18

Cross-cutting observation derived from vm-vmanager-app's `0.34 → 0.38`
catch-up to mgf-common ([commit `212808f`](https://codeberg.org/magogi-admin/vm-vmanager-app/commit/212808f),
2026-05-18). I am NOT a current mgf-http consumer — this is a
forward note for when mgf-http reaches the surface vm-vmanager-app
adopts.

**The observation.** mgf-common pushed four MINORs (0.35 / 0.36 /
0.37 / 0.38) to PyPI in a single day. The consumer-side experience:
the federation-sync message targeted `>=0.37,<0.38` while 0.38
(the federation-contract release) was already live, and PyPI
propagation lag meant `uv pip install --dry-run "mgf-common>=0.37"`
returned "unsatisfiable" for hours despite the wheel existing in
the sibling repo's `dist/`. Diagnosis required cross-checking
PyPI's JSON API directly + reading the sibling repo's CHANGELOG
to know which MINOR was the canonical target.

**Implication for mgf-http.** When mgf-http moves past v0.0 and
ships its first real release cycle:

- **Avoid same-day multi-MINOR PyPI bursts** when downstream
  consumers may be mid-adoption. Even three released-in-sequence
  MINORs over a few hours create a "which one is canonical?"
  question for a consumer reading the changelog hours later.
- **Document the per-MINOR consumer-side migration as part of
  the release-cut workflow.** A consumer reading the CHANGELOG
  should be able to size the catch-up at a glance: NEW public
  surface / removed public surface / unchanged interface / known
  consumer-side rework needed. mgf-common 0.35–0.38's "drop-in
  upgrade — no public API removals or renames" tagline worked
  well; the same convention scales.

**Volunteering as a reference consumer.** vm-vmanager-app's
[`docs/inprogress/MGF_ADOPTION.md`](https://codeberg.org/magogi-admin/vm-vmanager-app/src/branch/main/docs/inprogress/MGF_ADOPTION.md)
§3 lists "Phase 5 — Migrate VM-image download to `mgf-http`
(resumable + retry)" as a planned phase. When mgf-http ships the
resumable-download primitive, vm-vmanager-app is happy to be the
reference consumer — file paper here when ready, we'll integrate
and report friction.

---

## §4 How consumers submit feedback

1. **Open a Codeberg issue** at
   <https://codeberg.org/magogi-admin/mgf-http/issues> if you
   have a quick question. Maintainer triages.
2. **For a structured pain point**: file a PAPER-NN here as a PR
   that adds the entry to §2. Use the existing entries as the
   shape template (status table → concern → why-it-matters →
   concrete evidence → suggested shape → decision rationale on
   close).
3. **For AI-agent-driven consumer sessions**: see your project's
   `docs/CLAUDE.md` auto-block (managed by `mgf-common catch-up`)
   — the `feedback-discipline` section spells out where to file
   what.

---

## §5 Cross-references

- [mgf-common/FEEDBACK.md](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/FEEDBACK.md)
  — federation-wide PAPERs (cross-sibling concerns, standards
  proposals).
- [mgf-standard/docs/standards/DOCUMENTATION.md](https://codeberg.org/magogi-admin/mgf-standard/src/branch/main/docs/standards/DOCUMENTATION.md)
  — FEEDBACK discipline rules (FB-01..FB-04, DOC-13).
- [mgf-common/docs/recipes/feedback-md-template.md](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/docs/recipes/feedback-md-template.md)
  — the canonical shape this file was built from.
