# Recipe — Async HTTP client with `mgf.http`

> **Audience:** consumers integrating with REST APIs (third-party
> providers OR your own services). Fits both "make 3 HTTP calls in a
> CLI tool" and "5,000 calls/sec from a FastAPI service."
> **Optional extra:** `pip install 'mgf-http'`.

## At a glance

```python
from mgf.http import AsyncHttpClient, RetryPolicy

async with AsyncHttpClient(
    base_url="https://api.example.com",
    timeout=10.0,
    retry=RetryPolicy(max_attempts=3),
) as http:
    response = await http.get("/users", params={"limit": 50})
    response.raise_for_status()
    users = response.json()
```

That's a complete client with:
- User-Agent auto-set (`<your-app>/<your-ver> mgf-common/<lib-ver>`)
- Retry on transient failures (timeouts, 503s, network errors) with
  exponential backoff + jitter
- `Authorization` / `Cookie` / `X-Api-Key` headers redacted in logs
  (the value, not the name — the name in logs helps debugging)
- TLS verification on by default; loud `UserWarning` if you ever
  pass `verify_tls=False`
- `httpx.TimeoutException` etc. wrapped into `HttpTransportError`
  (subclass of `mgf.common.exceptions.OperationError`) so the
  standard CLI exit-code translator routes them correctly
- Best-effort OTel client span when `[observability]` extra is
  installed (no-op otherwise)

## What's in the box

### `AsyncHttpClient`

```python
AsyncHttpClient(
    *,
    base_url: str = "",
    timeout: float = 30.0,
    verify_tls: bool = True,
    user_agent: str | None = None,
    default_headers: Mapping[str, str] | None = None,
    retry: RetryPolicy | None = None,
)
```

- `base_url` — relative URLs in `request(url=...)` calls join here.
- `timeout` — per-request timeout in seconds. Per-call override via
  `request(timeout=...)`.
- `verify_tls` — TLS cert verification. **Always leave at default
  in production.**
- `user_agent` — override the auto-derived UA.
- `default_headers` — sent on every request; per-call headers
  merge over them.
- `retry` — see :class:`RetryPolicy`. Default: zero retries.

Convenience methods:
`get`, `post`, `put`, `patch`, `delete`, `head`. Each forwards to
`request(METHOD, url, ...)`.

Lifecycle: use as `async with AsyncHttpClient() as http:` OR call
`await http.aclose()` explicitly. Idempotent; safe to call twice.

Escape hatch: `http.client` returns the wrapped
`httpx.AsyncClient` for streaming downloads, multipart uploads, or
custom transports. Reaching through skips the wrapper's retry +
redaction + error translation.

### `RetryPolicy`

```python
RetryPolicy(
    max_attempts: int = 0,
    base_backoff_seconds: float = 0.5,
    max_backoff_seconds: float = 30.0,
    retryable_methods: frozenset[str] = {"GET", "HEAD", "OPTIONS", "PUT", "DELETE"},
    retryable_statuses: frozenset[int] = {429, 503, 504},
)
```

- `max_attempts` — number of RETRIES after the first attempt. Total =
  `max_attempts + 1`. Default 0 = single attempt = deterministic.
- Backoff is **exponential with full jitter**: attempt N waits a
  uniform random in `[0, base * 2^N]`, capped at `max_backoff`.
  Avoids the thundering-herd shape pure-exponential produces.
- `retryable_methods` defaults to RFC 7231 idempotent + DELETE
  (which is idempotent in practice). **POST is excluded** — repeating
  POST may double-submit. Override explicitly if your endpoint is
  safely retryable.
- `retryable_statuses` covers the canonical "service unavailable,
  try again" set.

Frozen dataclass; safe to share between requests.

### `HttpTransportError`

```python
class HttpTransportError(OperationError):
    url: str
    cause_summary: str
    attempts: int
```

Raised AFTER retries exhaust on transport-level failures. Original
`httpx` exception preserved via `__cause__`. 4xx / 5xx responses
**don't raise** here — they return as `httpx.Response`; use
`response.raise_for_status()` if you want a typed exception.

## Common patterns

### Provider auth via per-request header

```python
async with AsyncHttpClient(base_url="https://api.example.com") as http:
    response = await http.get(
        "/private",
        headers={"Authorization": f"Bearer {token}"},
    )
```

Logs the header NAME (so you can see auth was attempted) but
redact the VALUE (`<redacted>` in the log line).

### Vault-resolved auth

```python
from mgf.common.vault import EnvVault

vault = EnvVault()

async with AsyncHttpClient() as http:
    token = vault.get("STRIPE_SECRET_KEY")  # raises VaultError if absent
    response = await http.get(
        "https://api.stripe.com/v1/charges",
        headers={"Authorization": f"Bearer {token}"},
    )
```

### Per-request retry override

```python
# Use stricter retry for a critical idempotent call.
response = await http.get(
    "/health",
    retry=RetryPolicy(max_attempts=5, base_backoff_seconds=0.1),
)
```

### Reusing a client across the lifetime of a service

In FastAPI / a long-running service, construct once at lifespan
startup:

```python
@app.on_event("startup")
async def _startup() -> None:
    app.state.http = AsyncHttpClient(
        base_url=settings.upstream_api,
        retry=RetryPolicy(max_attempts=3),
    )

@app.on_event("shutdown")
async def _shutdown() -> None:
    await app.state.http.aclose()

# Inside handlers:
async def my_handler(request: Request) -> dict:
    response = await request.app.state.http.get("/users")
    ...
```

The connection pool persists across requests — far cheaper than a
fresh client per request.

(See `mgf.fastapi` (sibling package `mgf-fastapi`) for a Depends-based
helper that wraps this same pattern.)

### Testing with respx

```python
import httpx
import respx
import pytest
from mgf.http import AsyncHttpClient

@pytest.mark.asyncio
@respx.mock
async def test_my_call() -> None:
    respx.get("https://api.example.com/users").mock(
        return_value=httpx.Response(200, json=[{"id": 1}])
    )
    async with AsyncHttpClient(base_url="https://api.example.com") as http:
        response = await http.get("/users")
    assert response.json() == [{"id": 1}]
```

`respx` mocks at the transport layer; the wrapper is transparent.
Install via the `[http-test]` extra: `pip install 'mgf-http[test]'`.

## Migrating an existing HTTP transport

If your project already has a custom httpx wrapper (apiprobe's
`HttpxTransport` is the canonical example), this is the migration
shape that preserves your public surface while delegating the
mechanics to `AsyncHttpClient`.

### Step 1 — keep your transport class as the public adapter

Don't replace your transport with `AsyncHttpClient` directly —
keep your existing class so callers' imports don't change. Make
it a thin adapter:

```python
# myapp/transport/httpx_transport.py — was a hand-rolled httpx wrapper
from mgf.http import AsyncHttpClient, HttpTransportError

from myapp.transport.errors import TransportError # your own typed error

class HttpxTransport:
    """Project-internal transport. Adapts AsyncHttpClient to the
    project's own error type + auth conventions."""

    def __init__(self, *, base_url: str, api_token: str) -> None:
        self._client = AsyncHttpClient(
            base_url=base_url,
            default_headers={"Authorization": f"Bearer {api_token}"},
        )

    async def get(self, path: str, **kw) -> dict:
        try:
            response = await self._client.get(path, **kw)
            response.raise_for_status()
            return response.json()
        except HttpTransportError as e:
            raise TransportError(
                f"upstream call failed: {e}"
            ) from e

    async def aclose(self) -> None:
        await self._client.aclose()
```

The adapter pattern is the migration win: callers keep their
existing typed exception (`TransportError`); the wrapper-of-a-wrapper
gives them retry policy + redaction + identity headers + OTel for
free, AND consumers don't see the change.

### Step 2 — translate httpx errors to your errors at the seam

Your domain code shouldn't catch `HttpTransportError` or `httpx.HTTPError`
directly — that leaks the transport into the domain. The adapter
class is the seam where translation happens (the second `raise`
above). Your domain code catches `TransportError` only.

The original httpx exception is preserved via `__cause__` chain
(httpx → HttpTransportError → TransportError); for debugging, walk the
chain.

### Step 3 — let mgf-common own the retry policy

If your hand-rolled transport had its own retry loop, delete it.
Pass a `RetryPolicy` to `AsyncHttpClient` instead:

```python
from mgf.http import AsyncHttpClient, RetryPolicy

self._client = AsyncHttpClient(
    base_url=base_url,
    retry=RetryPolicy(max_attempts=3),  # default excludes POST/PATCH
)
```

The library's policy uses full-jitter exponential backoff, the
shape that survives every retry-storm postmortem in the literature.
Don't roll your own.

### Step 4 — drop your User-Agent override

`AsyncHttpClient` derives the User-Agent from `mgf.common`'s
identity automatically (`<app_name>/<app_version>` after `bootstrap()`).
Hand-rolled UA strings drift. Delete yours unless you have a
contractual reason to override.

### Step 5 — tighten the dep pin

```toml
# pyproject.toml — was
"mgf-common>=0.4,<0.6"
"httpx>=0.27"

# becomes
"mgf-http>=0.6.0,<0.8"
# httpx falls out — pulled by the [http] extra.
```

The `[http]` extra is mgf-common's promise to keep httpx in scope.
Don't dual-pin; let the extra own it.

### Step 6 — verify behavior in CI

The adapter shape is observably equivalent to the original wrapper.
Two checks:

1. Your existing transport tests (mocked at the httpx level via
   respx) MUST still pass — that's the regression gate.
2. Your TransportError chain is intact: assert `isinstance(e.__cause__,
   HttpTransportError)` and `isinstance(e.__cause__.__cause__,
   httpx.HTTPError)` for transport-level failures.

### Worked migration: `mgf-apiprobe`

`mgf-apiprobe`'s `HttpxTransport` migrated to this shape in
(apiprobe commit `ae03466`, against mgf-common ).
Net deletion: ~200 LOC of retry / redaction / UA / span boilerplate.
The public `Transport` Protocol stayed unchanged; consumer code
in apiprobe didn't move.

## Anti-patterns

- **Don't construct a fresh `AsyncHttpClient` per request.** The
  connection pool is the value. Hold one client per process or per
  upstream API.
- **Don't enable retries on every method.** POST + PATCH on
  non-idempotent endpoints can double-submit on retry. Default policy
  excludes them; override explicitly only when you know it's safe.
- **Don't catch `httpx.HTTPError` directly.** The wrapper translates
  them to `HttpTransportError` before they reach you. Catch the typed
  one + use `__cause__` if you need the original.
- **Don't pass `verify_tls=False` in production.** The runtime
  warning fires on every construction — silencing it is a code
  smell.
- **Don't mutate `client.headers` directly.** Use the `default_headers`
  constructor arg or per-call `headers=` kwarg. The wrapper assumes
  the underlying httpx client's headers are stable.

## Bounded response sizes — `max_response_size`

When a consumer fetches user-submitted URLs — link previews,
OpenGraph metadata scrapers, webhook URL verifiers, citation
enrichers — a malicious or misconfigured upstream can respond
with arbitrarily large body bytes. The default `request()` path
buffers the full body before returning, which puts the consumer
one bad URL away from RAM exhaustion. The `timeout` only bounds
wall-clock, not bytes-per-second × seconds.

The `max_response_size=` constructor kwarg sets a hard byte cap:

```python
from mgf.http import AsyncHttpClient, HttpTransportError

async with AsyncHttpClient(
    timeout=5.0,
    max_response_size=1_000_000,   # 1 MB hard cap
) as http:
    try:
        response = await http.get("https://unknown-host.example/")
        # response.content is up to 1 MB; over-cap responses
        # never get here.
    except HttpTransportError as exc:
        if exc.cause_summary == "response-too-large":
            # Recover gracefully — log + skip, never OOM.
            ...
        else:
            raise
```

When the cap fires, the wrapper streams the response under the
hood and aborts as soon as the accumulated body exceeds it.
Memory growth is bounded by `max_response_size + one chunk`,
regardless of what the upstream advertises or sends.

Use this for:

- **User-submitted URL fetching** (link previews, citation
  metadata, webhook source verification).
- **Untrusted third-party providers** where you don't control
  the response shape.
- **Defence in depth** alongside `timeout=` — both are needed.
  Timeout bounds time; `max_response_size` bounds bytes.

The cap is opt-in; the default `None` preserves the historical
buffer-fully behaviour, which is correct for trusted-provider
calls where response shapes are known and bounded by contract.

## SSRF-safe redirect walking

Fetching user-submitted URLs has a second concern beyond size:
**every redirect hop is another URL the consumer didn't choose**.
An `http://` redirect target slips past an `https://`-only guard
unless every hop is re-checked. The canonical pattern:

```python
from urllib.parse import urljoin
from mgf.http import AsyncHttpClient

MAX_REDIRECTS = 5

async def fetch_https_only(url: str) -> tuple[str, bytes] | None:
    """Fetch `url`, refusing http:// on the initial URL or any redirect hop."""
    async with AsyncHttpClient(
        timeout=5.0,
        follow_redirects=False,   # we walk redirects manually
        max_response_size=1_000_000,
    ) as http:
        current_url = url
        for _ in range(MAX_REDIRECTS + 1):
            if not current_url.startswith("https://"):
                return None
            response = await http.request("GET", current_url)
            if response.status_code in (301, 302, 303, 307, 308):
                location = response.headers.get("location") or ""
                if not location:
                    return None
                current_url = urljoin(current_url, location)
                continue
            if response.status_code >= 400:
                return None
            return current_url, response.content
        return None  # too many redirects
```

Why each piece matters:

- **`follow_redirects=False`** — httpx would otherwise auto-follow
  and we'd never see the intermediate `Location` headers.
- **`urljoin`** — handles relative redirect targets (`/path`,
  `../path`) by resolving against the current URL.
- **`https://` check on every hop** — including the initial URL,
  so a misconfigured caller passing `http://` gets caught early.
- **`max_response_size`** — bounds memory regardless of what the
  upstream sends.
- **`MAX_REDIRECTS + 1`** — the `+1` is because the first iteration
  is the initial fetch, not a redirect.

Reference consumer: inthewords'
`apps/messaging/tasks.py:_fetch_url_async` (citation metadata
fetcher) is the worked production example of this pattern.
Closes [PAPER-37](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/FEEDBACK.md).

## Streaming responses — `.stream()`

For large downloads, server-sent events, NDJSON, or any flow
where buffering wastes memory, use `.stream()` instead of
`.request()`. The same retry + identity + redaction + typed-error
posture applies — you just drive the body read yourself:

```python
async with AsyncHttpClient() as http:
    async with http.stream("GET", "https://api.example.com/big") as response:
        response.raise_for_status()
        async for chunk in response.aiter_bytes(chunk_size=64 * 1024):
            await write_to_disk(chunk)
```

Retry behaviour matches `request()` until the response is
yielded — transport failures and retryable statuses (429/503/504)
both trigger a fresh attempt. Once yielded, the response is
yours; if the body stream fails mid-read, the caller catches
`httpx.HTTPError` and decides what to do.

`max_response_size=` on the constructor is the simpler "I just
want a hard cap" shape — it routes through `.stream()` internally
to enforce the cap, transparent to the caller. Use `.stream()`
when you want the cap to depend on what you've already read, or
when you need true streaming-as-a-feature (not buffer-with-cap).

## See also

- `docs/recipes/cli.md` — building CLIs that call HTTP APIs.
- `docs/recipes/testing.md` — pytest fixtures for HTTP-using code.
- `docs/reference/vault.md` — secrets resolution patterns.
- `docs/reference/lifecycle.md` — `bootstrap()` + AppContext.
