# Error Handling

> **Conformance:** L1 / L2 / L3 (see `README.md`)
> **Audience:** Human engineers AND AI coding agents.
> **Propagation (DOC-18):** changing a rule here ripples — walk DOCUMENTATION.md §10.4 and update its dependents in the same commit (README *Index of MUSTs*, the `mgf-fed standards` / `agents` CLI, and any parallel copies / pins the matrix names).

> **📦 Note.** The base `AppError`, the seven category bases (`ConfigError`, `ProviderError`, `OperationError`, `GuestError`, `HostEnvironmentError`, `VaultError`, `SchedulerError`), and seven generic concretes (`SchemaValidationError`, `AppConfigError`, `CommandError`, `ResourceNotFoundError`, `ResourceAlreadyExistsError`, `GuestUnreachableError`, `GuestCommandError`) live in **`mgf.common.exceptions`**. The crash reporter, `install_excepthook`, and `install_threading_excepthook` live in **`mgf.common.observability`**. Consumers import from `mgf.common.*` directly; the gap-closure plans below remain as worked examples of how to bring a new project up to L1/L2.

---

## Rules box (read this first)

| ID | Rule |
|---|---|
| <a id="EH-01"></a>**EH-01** | Every exception raised by **application code** MUST be a subclass of the project's base error class (e.g., `AppError`). Never raise a bare `Exception` or a third-party exception across an internal API. |
| <a id="EH-02"></a>**EH-02** | Third-party exceptions MUST be **translated** to typed application exceptions at the boundary that owns the third-party call (provider, shell wrapper, HTTP client, DB driver). |
| <a id="EH-03"></a>**EH-03** | When re-raising a translated exception, the original cause MUST be chained: `raise NewError(...) from original`. |
| <a id="EH-04"></a>**EH-04** | Every entry point MUST install a top-level handler. CLI: `_run_and_translate_exceptions`. Process: `sys.excepthook`. Threads: `threading.excepthook`. asyncio: `loop.set_exception_handler`. Qt: `sys.excepthook` integration + `qInstallMessageHandler` for Qt-native messages. |
| <a id="EH-05"></a>**EH-05** | Unhandled exceptions MUST produce a **local crash report** (JSON file under the app's state dir) AND, if remote reporting is enabled, MUST be forwarded to the configured remote sink (Sentry / OTLP). |
| <a id="EH-06"></a>**EH-06** | GUI workers MUST catch **every** exception (typed AND untyped), report via `error(str)` signal, and never crash the worker thread silently. Untyped exceptions MUST be routed through the crash reporter before being surfaced as a string. |
| <a id="EH-07"></a>**EH-07** | `except Exception:` (without re-raise or typed translation) MUST NOT appear except in: (a) cleanup paths inside `finally:` / `contextlib.suppress(...)`, (b) "best-effort observer" loops that log + continue and document the invariant in a docstring. |
| <a id="EH-08"></a>**EH-08** | Every typed exception MUST carry **machine-readable fields** as constructor args (e.g., `CommandError.argv`, `SpecValidationError.errors`), not just a formatted message string. |
| <a id="EH-09"></a>**EH-09** | Error messages SHOWN to users MUST end in an actionable next step where possible ("rerun with `--host <ip>`", "stop the VM and retry", "install systemd or run with --dry-run"). |
| <a id="EH-10"></a>**EH-10** | "Best-effort observer" code (hook executors, scheduler tick loops, alert evaluators, libvirt event callbacks) MUST follow the **never-raises** invariant: catch broadly, log at WARNING, return a bool, continue to the next observer. |
| <a id="EH-11"></a>**EH-11** | **Retries MUST use exponential backoff with full jitter and a finite cap.** Linear backoff and immediate retry are forbidden. The retry budget MUST be configurable and observable (`audit=true` log per attempt). See §13.1. |
| <a id="EH-12"></a>**EH-12** | **Operations against external systems with multi-attempt cost (writes, sends) MUST be idempotent.** An operation that may run twice MUST produce the same result as running once. The idempotency key MUST be the natural key when one exists. See §13.5. |
| <a id="EH-13"></a>**EH-13** | **Upstream-amplifying load MUST be bounded for any external call that, when failing, can make an already-struggling upstream worse** (unbounded retries against an overloaded service deepen the outage). The **circuit breaker (open / half-open / closed states + thresholds + cooldown) is the canonical mechanism** — see §13.2. Where the retry budget (a finite cap per **EH-11**) AND the call rate already bound amplification to a documented ceiling, the agent MAY record that judgment (per [`non-adoption-is-valid`](principles/non-adoption-is-valid.md)) in place of a breaker; record WHAT bounds the load and WHY a breaker adds nothing. Binding the outcome, not the mechanism — but "no breaker AND no recorded bound" is a defect. |

---

## 1. Mental model

Application errors are *expected*. Unexpected errors are *bugs*. Both must be:

1. **Caught at a known boundary.**
2. **Translated into the application's typed hierarchy** so the rest of the code can reason about them.
3. **Surfaced to the user with an actionable message** (if user-facing) or a structured log + crash report (if internal).
4. **Reported through a single pipeline** so observability tools see them all.

If you ever find yourself writing `except Exception: pass` or printing a stack trace and continuing, stop and re-read this document.

---

## 2. The exception hierarchy

### 2.1 Shape

The hierarchy is intentionally **shallow** and **typed by category**. Deep hierarchies (5+ levels) become hard to remember; flat hierarchies (one base, many siblings) lose the ability to catch a category. The sweet spot is **2–3 levels** with 5–10 categories at level 2.

```
AppError ← project base, abstract in spirit
├── ConfigError ← config schema / validation problems
│ ├── SchemaValidationError ← user input failed validation; carries .errors list
│ ├── AppConfigError ← config file missing/malformed/wrong version
│ └── ... ← project-specific subclasses
├── ProviderError ← anything raised by a provider implementation
│ ├── ResourceNotFoundError ← carries .name
│ ├── ResourceAlreadyExistsError ← carries .name
│ └── ...
├── OperationError ← anything raised by a use case / orchestrator
│ ├── CommandError ← subprocess exit != 0; carries argv, returncode, stderr
│ ├── BackupError, MigrationError, ... ← per-operation subclasses
│ └── ...
├── GuestError ← anything raised when talking to a remote agent
│ ├── GuestUnreachableError
│ ├── GuestCommandError
│ └── HealthCheckError
├── SchedulerError ← scheduler / hooks / cron problems
├── VaultError ← credential storage problems
└── HostEnvironmentError ← host pre-flight / OS problems
```

### 2.2 What each category is for

| Category | When to raise | What carries it |
|---|---|---|
| **ConfigError** | A typed config field failed validation, the YAML is malformed, the schema version is unknown, a user spec has missing fields | Anything in L4 `config.py` / `creator/` / spec parsers |
| **ProviderError** | A `<X>Provider` method failed because the underlying SDK / daemon / API returned an error or unexpected state | `providers/*_provider.py` |
| **OperationError** | An L2 use case (clone, backup, migrate, install, deploy) failed during execution | `operations/`, `lifecycle/` |
| **GuestError** | An attempt to talk to a remote system (SSH, agent, health probe) failed | Anything that opens an SSH connection, a TCP socket, an HTTP probe |
| **SchedulerError** | A cron expression was invalid or a scheduled run failed | `scheduler/`, `hooks/` |
| **VaultError** | The credential vault was unavailable or returned an error | `vault/` |
| **HostEnvironmentError** | A host-level pre-flight check failed (CPU lacks vmx, kernel module missing, required binary not installed) | `environment/`, `installer/` |

### 2.3 Drop-in template — `<app>/exceptions.py`

```python
"""Typed exception hierarchy for <app>.

Every exception raised by <app> code is a subclass of :class:`AppError`.
The CLI top-level handler catches :class:`AppError` and maps subtypes
to documented exit codes; the GUI worker layer catches at the same
level and forwards via signal. Third-party exceptions MUST be
translated to a subclass of `AppError` at the boundary that owns the
third-party call (rule EH-02).

Design rules:
  - Hierarchy is shallow (2-3 levels). Deep is unmemorable; flat
    cannot be caught by category.
  - Every typed exception carries machine-readable fields (rule EH-08),
    not just a message string. The string is for humans; the fields are
    for the program.
  - Re-raises chain via `from e` (rule EH-03). The original cause is
    needed to debug.
"""

from __future__ import annotations

# ---------------------------------------------------------------------------
# Base
# ---------------------------------------------------------------------------

class AppError(Exception):
    """Base class for every exception raised by <app> code.

    Subclasses MUST set machine-readable attributes in ``__init__``
    BEFORE calling ``super().__init__(message)``, so callers can read
    ``.attr`` without parsing the message.
    """

# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------

class ConfigError(AppError):
    """Base for configuration / spec validation failures."""

class SchemaValidationError(ConfigError):
    """A typed spec failed validation.

    Always carries the *complete* list of errors so callers can show
    every problem at once (don't make the user re-run for each one).
    """

    def __init__(self, errors: list[str]) -> None:
        if not errors:
            raise ValueError("SchemaValidationError requires at least one error")
        self.errors: list[str] = list(errors)  # copy — defensive
        super().__init__(f"validation failed: {'; '.join(errors)}")

class AppConfigError(ConfigError):
    """The application config file is missing, malformed, or wrong version."""

# ---------------------------------------------------------------------------
# Provider
# ---------------------------------------------------------------------------

class ProviderError(AppError):
    """Base for failures inside a provider implementation."""

class ResourceNotFoundError(ProviderError):
    """A name or id did not resolve to any known resource."""

    def __init__(self, name: str) -> None:
        self.name: str = name
        super().__init__(f"no resource named {name!r}")

class ResourceAlreadyExistsError(ProviderError):
    """A create operation collided with an existing resource."""

    def __init__(self, name: str) -> None:
        self.name: str = name
        super().__init__(f"a resource named {name!r} already exists")

# ---------------------------------------------------------------------------
# Operations
# ---------------------------------------------------------------------------

class OperationError(AppError):
    """Base for runtime failures during a long-running operation."""

class CommandError(OperationError):
    """A subprocess invoked via the shell wrapper exited non-zero.

    Carries argv, returncode, and stderr so callers can surface a
    precise error without re-running the command.
    """

    def __init__(
        self,
        argv: list[str],
        returncode: int,
        stderr: str = "",
    ) -> None:
        self.argv: list[str] = list(argv)
        self.returncode: int = returncode
        self.stderr: str = stderr
        cmd = " ".join(argv)
        tail = f": {stderr.strip()}" if stderr.strip() else ""
        super().__init__(f"command failed (exit {returncode}): {cmd}{tail}")

# ---------------------------------------------------------------------------
# Guest / network
# ---------------------------------------------------------------------------

class GuestError(AppError):
    """Base for failures communicating with a remote agent."""

class GuestUnreachableError(GuestError):
    """Could not open a connection to a remote host."""

class GuestCommandError(GuestError):
    """A command ran on the remote host but returned non-zero."""

class HealthCheckError(GuestError):
    """A health check (TCP/SSH/HTTP) timed out or failed."""

# ---------------------------------------------------------------------------
# Vault / environment / scheduler — mirror the same shape
# ---------------------------------------------------------------------------

class VaultError(AppError):
    """Base for credential vault failures."""

class HostEnvironmentError(AppError):
    """Base for host pre-flight / OS-level failures.

    Named HostEnvironmentError (not EnvironmentError) on purpose: the Python builtin
    EnvironmentError is an alias of OSError, so a same-named subclass would
    shadow it and make `except EnvironmentError` silently miss real OSErrors.
    """

class SchedulerError(AppError):
    """Base for scheduler / hook subsystem failures."""

__all__ = [
    "AppError",
    "AppConfigError",
    "CommandError",
    "ConfigError",
    "HostEnvironmentError",
    "GuestCommandError",
    "GuestError",
    "GuestUnreachableError",
    "HealthCheckError",
    "OperationError",
    "ProviderError",
    "ResourceAlreadyExistsError",
    "ResourceNotFoundError",
    "SchedulerError",
    "SchemaValidationError",
    "VaultError",
]
```

**MUSTs:**
- Every project that adopts these standards MUST have a single `exceptions.py` (or `errors/` package for very large projects) at L4.
- Every subclass MUST be exported in `__all__`.
- Categories above (`ConfigError`, `ProviderError`, `OperationError`, `GuestError`, `VaultError`, `HostEnvironmentError`, `SchedulerError`) are the **canonical** category set. Add new categories only when none fit; never duplicate a category by another name ("DataError" instead of "ConfigError" because the project doesn't have config… no — call it ConfigError for category-level catchability). **These seven are the CLI / desktop / SDK baseline (an `applies_to` shape, per DESIGN_PRINCIPLES §14.2)** — a project whose domain has error classes none of them fit (e.g. a web service's HTTP `401/403/404/409/503`) SHOULD add its own domain category set, and if that shape generalises, propose it upstream (a FEEDBACK paper, per [`non-adoption-is-valid`](principles/non-adoption-is-valid.md)) so the next consumer of that shape inherits it instead of re-deriving. Adding a category for a real domain gap is conformant; contorting an error into the wrong category to avoid adding one is the defect.

**AS-IS in `mgf-common`.** `src/mgf/common/exceptions/_base.py`
defines `AppError` + the 7 category bases; `_well_known.py`
adds the 7 generic concretes (`SchemaValidationError`,
`CommandError`, `ResourceNotFoundError`, etc.). Every
exception with non-trivial state carries machine-readable
fields rather than relying on the message string.

---

## 3. Translation seams

A **translation seam** is the **single function / method boundary** where a third-party exception becomes a typed application exception. Identifying these seams up-front prevents the most common error-handling pathology: scattered `try/except` blocks all translating slightly differently.

### 3.1 The three canonical seams

#### Seam 1 — The provider boundary

Every method on a `Provider` ABC implementation MUST translate the underlying SDK's exceptions. Two equivalent shapes — pick one per project and stay consistent:

**Shape A — `_translate()` returns the typed exception; caller raises with `from e`:**

```python
# src/<app>/providers/<vendor>_provider.py — Shape A
def _translate(e: vendor_sdk.SDKError, *, name: str | None = None) -> ProviderError:
    """Map vendor errors to our typed hierarchy.

    Returned — not raised — so the caller can do
    ``raise _translate(e) from e`` to preserve the original cause.
    Mapping table lives in one place per provider.
    """
    code = getattr(e, "error_code", None)
    if code == vendor_sdk.ERR_NOT_FOUND:
        return ResourceNotFoundError(name or "unknown")
    if code == vendor_sdk.ERR_EXISTS:
        return ResourceAlreadyExistsError(name or "unknown")
    return ProviderError(f"vendor error {code}: {e}")

class VendorProvider(ResourceProvider):
    def get(self, name: str) -> EntityHandle:
        try:
            raw = self._sdk.get(name)
        except vendor_sdk.SDKError as e:
            raise _translate(e, name=name) from e  # rule EH-03
        return _to_handle(raw)
```

**Shape B — `_translate()` raises directly (alternative; some teams prefer):**

```python
def _translate(e: vendor_sdk.SDKError, *, name: str | None = None) -> NoReturn:
    code = getattr(e, "error_code", None)
    if code == vendor_sdk.ERR_NOT_FOUND:
        raise ResourceNotFoundError(name or "unknown") from e
    raise ProviderError(f"vendor error {code}: {e}") from e
```

Both shapes uphold rule **EH-03** (cause chained). Choose Shape A for projects that value functional purity in the seam (the call site is explicit: `raise _translate(...) from e`, and the function itself is pure-data); choose Shape B if you find call sites cleaner without the wrapper `raise`.

#### Seam 2 — The shell wrapper

Every subprocess invocation MUST go through one wrapper that catches non-zero exit, timeout, and OS errors:

```python
# src/<app>/common/shell.py
import subprocess
from <app>.exceptions import CommandError

def run_command(
    argv: list[str],
    *,
    timeout: float | None = None,
    check: bool = True,
    capture: bool = True,
) -> subprocess.CompletedProcess:
    """Single subprocess wrapper.

    No `shell=True` ever — pass argv as a list, pre-validate any
    user input. Non-zero exit, timeout, and OS errors all raise
    typed CommandError.
    """
    try:
        result = subprocess.run(
            argv,
            timeout=timeout,
            capture_output=capture,
            text=True,
            shell=False,
            check=False,
        )
    except subprocess.TimeoutExpired as e:
        raise CommandError(argv=argv, returncode=-1,
                           stderr=f"timed out after {timeout}s: {e}") from e
    except OSError as e:
        # Binary not found, permission denied, ...
        raise CommandError(argv=argv, returncode=-2,
                           stderr=f"could not execute: {e}") from e

    if check and result.returncode != 0:
        raise CommandError(argv=argv,
                           returncode=result.returncode,
                           stderr=result.stderr or "")
    return result
```

**MUSTs:**
- `subprocess.run` MUST NOT be called outside this wrapper in core code. Lint rule (`flake8-bugbear` `B602` or a custom check).
- `shell=True` MUST NOT appear anywhere.

#### Seam 3 — The HTTP / SSH / external-IO client

Every external-IO client (HTTP via `httpx`, SSH via `paramiko`, etc.) MUST translate its protocol exceptions inside the client's own wrapper class:

```python
# src/<app>/common/ssh.py
import paramiko
from <app>.exceptions import GuestUnreachableError, GuestCommandError

class SSHClient:
    def connect(self) -> None:
        try:
            self._client.connect(...)
        except paramiko.AuthenticationException as e:
            raise GuestUnreachableError(
                f"SSH auth failed for {self.username}@{self.host}: {e}"
            ) from e
        except (paramiko.SSHException, OSError) as e:
            raise GuestUnreachableError(
                f"could not connect to {self.host}: {e}"
            ) from e
```

**Consumer pattern.** A typical project has three translation
seams: a provider seam (one per vendor SDK), a subprocess
seam (`run_command`), and an SSH seam (`SSHClient.connect/run`).
Each follows the Shape-A or Shape-B template above.

**Gap:** the libvirt translator drops `from e` in some branches (lines 102, 104). MUST be fixed — see Gap-Closure plan below.

---

## 4. Top-level handlers (rule EH-04)

Every entry point installs a top-level handler. This is a hard requirement because **uncaught exceptions silently kill threads and event loops**, leaving the app in a wedged state.

### 4.1 CLI handler

```python
# src/<app>/cli/__main__.py
import click

from <app>.exceptions import (
    AppError, ConfigError, ProviderError, OperationError,
    GuestError, HostEnvironmentError,
)

EXIT_SUCCESS = 0
EXIT_CONFIG_ERROR = 1
EXIT_OPERATION_ERROR = 2
EXIT_GUEST_ERROR = 3
EXIT_ENVIRONMENT_ERROR = 4
EXIT_UNKNOWN = 10

def _run_and_translate_exceptions(argv: list[str] | None = None) -> int:
    try:
        main.main(args=argv, standalone_mode=False, prog_name="<app>")
    except click.UsageError as e:
        e.show()
        return e.exit_code
    except click.ClickException as e:
        e.show()
        return e.exit_code
    except ConfigError as e:
        click.echo(f"config error: {e}", err=True)
        return EXIT_CONFIG_ERROR
    except ProviderError as e:
        click.echo(f"provider error: {e}", err=True)
        return EXIT_OPERATION_ERROR
    except OperationError as e:
        click.echo(f"operation failed: {e}", err=True)
        return EXIT_OPERATION_ERROR
    except GuestError as e:
        click.echo(f"guest error: {e}", err=True)
        return EXIT_GUEST_ERROR
    except HostEnvironmentError as e:
        click.echo(f"host environment error: {e}", err=True)
        return EXIT_ENVIRONMENT_ERROR
    except AppError as e:
        click.echo(f"error: {e}", err=True)
        return EXIT_UNKNOWN
    except click.exceptions.Exit as e:
        return int(e.exit_code)
    except Exception as e:  # noqa: BLE001 — top-level catch-all by design
        # Rule EH-05: an untyped exception is a bug. Crash-report it,
        # show the user a brief message that points at the report file.
        from <app>.observability.crash_reporter import report_now
        path = report_now(e, source="cli")
        click.echo(
            f"internal error: {e}\nA crash report was written to {path}.",
            err=True,
        )
        return EXIT_UNKNOWN
    return EXIT_SUCCESS
```

**MUSTs:**
- Order matters. Catch most-specific (`ConfigError`) before most-general (`AppError`).
- Every category MUST map to a documented exit code.
- The catch-all at the end MUST route to the crash reporter (rule EH-05).
- **`pyproject.toml` script entry point MUST point at `cli_main`, NOT at the Click group `main` directly.** When the script invokes `main` directly, Click handles its own `sys.exit` and `_run_and_translate_exceptions` never runs in production — the catch-all + excepthook installation are bypassed. The right shape is `app = "<app>.cli.__main__:cli_main"`.

**AS-IS in `mgf-common`.** `src/mgf/common/cli/_handler.py`
ships the catch-all + crash-reporter wiring; consumers wrap
their CLI entry point with the helper rather than re-implementing
the exit-code map.

### 4.2 Process-level: `sys.excepthook`

For desktop and mobile apps, install **before** the GUI loop starts:

```python
# src/<app>/observability/excepthook.py
import sys
from typing import Any
from <app>.observability.crash_reporter import report_now
from <app>.observability.logging_setup import LOG

def install_excepthook() -> None:
    """Install the process-wide unhandled-exception hook.

    Run this BEFORE any GUI loop or thread starts. Uncaught
    exceptions on the main thread land here.
    """
    prior = sys.excepthook

    def hook(exc_type: type[BaseException], exc: BaseException,
             tb: Any) -> None:
        if issubclass(exc_type, (KeyboardInterrupt, SystemExit)):
            prior(exc_type, exc, tb)
            return
        LOG.error("unhandled exception", exc_info=(exc_type, exc, tb))
        report_now(exc, source="excepthook")
        prior(exc_type, exc, tb)

    sys.excepthook = hook
```

### 4.3 Threads: `threading.excepthook`

```python
import threading
from <app>.observability.crash_reporter import report_now
from <app>.observability.logging_setup import LOG

def install_threading_excepthook() -> None:
    """Capture unhandled exceptions from non-main threads.

    Without this, a thread crash is logged to stderr by the Python
    runtime and otherwise lost — no crash report, no remote sink.
    """
    prior = threading.excepthook

    def hook(args: threading.ExceptHookArgs) -> None:
        if args.exc_type is SystemExit:
            return
        LOG.error("unhandled exception in thread %s",
                  args.thread.name if args.thread else "?",
                  exc_info=(args.exc_type, args.exc_value, args.exc_traceback))
        if args.exc_value is not None:
            report_now(args.exc_value, source=f"thread:{args.thread.name if args.thread else '?'}")
        prior(args)

    threading.excepthook = hook
```

### 4.4 asyncio handler

```python
import asyncio
from <app>.observability.crash_reporter import report_now
from <app>.observability.logging_setup import LOG

def install_asyncio_handler(loop: asyncio.AbstractEventLoop) -> None:
    def handler(loop: asyncio.AbstractEventLoop, ctx: dict) -> None:
        exc = ctx.get("exception")
        msg = ctx.get("message", "<no message>")
        if exc is not None:
            LOG.error("unhandled asyncio exception: %s", msg, exc_info=exc)
            report_now(exc, source="asyncio")
        else:
            LOG.error("asyncio error context: %s", ctx)

    loop.set_exception_handler(handler)
```

### 4.5 Qt handler

Qt is special — it has both Python-side and C++-side errors:

```python
# src/<app>/gui/qt_excepthook.py
import sys
from PySide6.QtCore import qInstallMessageHandler, QtMsgType
from <app>.observability.crash_reporter import report_now
from <app>.observability.logging_setup import LOG

def install_qt_handlers() -> None:
    """Install BOTH the Python excepthook (Qt does not call it on
    unhandled signal-slot exceptions on some platforms) AND the
    Qt-native message handler so qWarning/qCritical land in our log.
    """
    # Python side
    from <app>.observability.excepthook import install_excepthook
    install_excepthook()

    # Qt side
    def qt_message_handler(mode: QtMsgType, ctx, msg: str) -> None:
        if mode == QtMsgType.QtFatalMsg:
            LOG.error("Qt fatal: %s (%s:%d)", msg, ctx.file, ctx.line)
        elif mode == QtMsgType.QtCriticalMsg:
            LOG.error("Qt critical: %s", msg)
        elif mode == QtMsgType.QtWarningMsg:
            LOG.warning("Qt warning: %s", msg)
        else:
            LOG.debug("Qt: %s", msg)

    qInstallMessageHandler(qt_message_handler)
```

**MUSTs:**
- `install_qt_handlers()` MUST be called from `gui/app.py::run()` before `QApplication.exec()`.
- For Android (PySide6 on Android), the same hook applies; the iOS/macOS sandbox may require additional `NSExceptionHandler` integration via PyObjC — out of scope here, document per-project.

---

## 5. The GUI worker error pipeline (rule EH-06)

Every blocking operation invoked from a UI runs inside a worker (see `DESIGN_PRINCIPLES.md` §2.1). The base `CoreWorker` MUST catch every exception:

```python
# src/<app>/gui/workers/core_worker.py
class CoreWorker(QThread):
    # Canonical worker contract owned by GUI_CONSUMERS.md GUI-02.
    completed = Signal(object)  # terminal: success
    failed = Signal(object)     # terminal: typed AppError
    cancelled = Signal()        # terminal: cancellation != failure

    def run(self) -> None:
        try:
            result = self.do_work()
        except AppError as e:
            if self.cancellation.is_cancelled():
                self.cancelled.emit()  # GUI-02: cancellation != failure
                return
            # Typed: forward the typed error to the UI.
            self.failed.emit(e)
        except Exception as e:  # noqa: BLE001 — rule EH-06
            # Untyped: this is a bug. Crash-report it then surface a
            # typed error. The full traceback is in the log + report.
            from <app>.observability.crash_reporter import report_now
            report_now(e, source=f"worker:{type(self).__name__}")
            self.failed.emit(AppError(f"unexpected error: {e!r}"))
        else:
            self.completed.emit(result)
```

The UI slot connected to `failed` MUST display the message in a non-blocking way (status bar, log panel, inline label). It MUST NOT pop a modal dialog for every error — modal storms are the worst UX.

For *fatal* errors that block further interaction (e.g., the provider connection is dead), the slot MAY pop a single dialog and disable the affected controls.

### 5.1 Testing GUI workers

Two pitfalls worth documenting because they bite every project that adopts pytest-qt:

1. **`qtbot.addWidget(...)` rejects QThread.** It only accepts `QWidget` subclasses. `CoreWorker(QThread)` is NOT a `QWidget`. Use `worker.wait(timeout_ms)` directly after `worker.start()` to ensure the thread joined before the test exits — otherwise pytest-qt teardown fires before the worker's `do_work()` returns and the test sees flaky signal counts.

2. **Direct `sys.excepthook(...)` calls in tests chain to the prior hook**, which under pytest is often a hook that records the synthetic exception as a test failure (or surfaces it via `PytestUnraisableExceptionWarning`, which `filterwarnings = ["error"]` then escalates to a hard failure). Tests MUST replace the prior hook with a no-op before installing the standard one:

```python
# tests/.../test_excepthook.py
def test_excepthook_writes_crash_report(_restore_sys_excepthook):
    sys.excepthook = lambda *_args: None  # silence pytest's prior hook
    install_excepthook()
    try:
        raise RuntimeError("synthetic")
    except RuntimeError as e:
        sys.excepthook(type(e), e, e.__traceback__)
    # ... assertions on crash dir
```

Both are not standards-of-design — they are testing patterns that, if not learned, waste hours. They land in the standard so future projects skip the pain.

---

## 6. The crash reporter (rule EH-05)

Every project MUST ship a crash reporter. The reporter writes a structured JSON report locally and, if remote reporting is enabled, forwards it to the configured sink.

**Preferred shape: depend on `mgf-common`.** The reference
implementation lives in
[`mgf.common.observability.crash_reporter`](https://codeberg.org/magogi-admin/mgf-common/src/branch/main/src/mgf/common/observability/crash_reporter.py)
— `report_now()`, `install_excepthook()`,
`install_threading_excepthook()`, plus the `register_crash_sink`
decorator for custom sinks. Consumers SHOULD use this directly:
`from mgf.common.observability import report_now`. The shipped
implementation already covers every hardening described below.

If you genuinely need to roll your own (a niche-sink-only deploy,
a legacy codebase that cannot yet take the dependency), the
template in §6.1 is the v0.1 starting point. **Apply the §6.2
hardening** before shipping it — the v0.1 template is a sketch,
not a contract.

### 6.1 Drop-in template — `<app>/observability/crash_reporter.py`

```python
"""Local + optional remote crash reporting.

Local report: a JSON file under the application's state dir, one
file per crash, named with timestamp + short uuid. Survives
restarts; never ships to disk-full and crashes.

Remote: opt-in via env var or config. Supported sinks:
  - Sentry / GlitchTip   (sentry-sdk if installed)
  - OTLP                 (opentelemetry-sdk if installed)
  - HTTP webhook         (any URL accepting POST application/json)

Importing this module MUST NOT pull in any of the optional deps —
they are loaded lazily at first use.
"""

from __future__ import annotations

import json
import os
import platform
import sys
import traceback
import uuid
from datetime import UTC, datetime
from pathlib import Path
from typing import Any

from <app> import __version__
from <app>.observability.logging_setup import LOG

def _state_dir() -> Path:
    """Resolve the application's state dir lazily (rule CF-04)."""
    xdg = os.environ.get("XDG_STATE_HOME")
    base = Path(xdg) if xdg else Path.home() / ".local" / "state"
    return base / "<app>" / "crashes"

def _build_report(exc: BaseException, *, source: str) -> dict[str, Any]:
    return {
        "timestamp": datetime.now(UTC).isoformat(),
        "report_id": uuid.uuid4().hex,
        "app": "<app>",
        "version": __version__,
        "source": source,
        "exception": {
            "type": f"{type(exc).__module__}.{type(exc).__qualname__}",
            "message": str(exc),
            "args": [repr(a) for a in getattr(exc, "args", ())],
        },
        "traceback": traceback.format_exception(type(exc), exc, exc.__traceback__),
        "cause": (
            {
                "type": f"{type(exc.__cause__).__module__}.{type(exc.__cause__).__qualname__}",
                "message": str(exc.__cause__),
            }
            if exc.__cause__ is not None
            else None
        ),
        "platform": {
            "system": platform.system(),
            "release": platform.release(),
            "python": sys.version,
            "implementation": platform.python_implementation(),
        },
    }

def _write_local(report: dict[str, Any]) -> Path:
    target_dir = _state_dir()
    target_dir.mkdir(parents=True, exist_ok=True)
    fname = f"{report['timestamp'].replace(':', '-')}-{report['report_id'][:8]}.json"
    path = target_dir / fname
    # Atomic write (rule CF-04 generalised — any state file the app owns).
    tmp = path.with_suffix(".tmp")
    tmp.write_text(json.dumps(report, indent=2, default=str), encoding="utf-8")
    tmp.chmod(0o600)
    tmp.replace(path)
    return path

def _ship_remote(report: dict[str, Any]) -> None:
    """Best-effort remote shipping. Never raises.

    Reads <APP>_CRASH_SINK env var:
      - "sentry"   : use sentry-sdk if available
      - "otlp"     : use opentelemetry log exporter if available
      - "http://..": POST JSON to this URL
      - unset      : no-op
    """
    sink = os.environ.get("<APP>_CRASH_SINK", "").strip()
    if not sink:
        return
    try:
        if sink == "sentry":
            import sentry_sdk
            sentry_sdk.capture_exception(error=None, extra=report)
        elif sink == "otlp":
            from <app>.observability.otel_setup import emit_log_event
            emit_log_event("crash", attributes=report)
        elif sink.startswith(("http://", "https://")):
            import urllib.request
            req = urllib.request.Request(
                sink,
                data=json.dumps(report).encode("utf-8"),
                headers={"Content-Type": "application/json"},
                method="POST",
            )
            urllib.request.urlopen(req, timeout=5)  # noqa: S310 — sink is operator-controlled
        else:
            LOG.warning("unknown crash sink: %s", sink)
    except Exception as e:  # noqa: BLE001 — best-effort observer
        LOG.warning("crash report shipping failed: %s", e)

def report_now(exc: BaseException, *, source: str) -> Path:
    """Build, persist locally, and best-effort ship a crash report.

    Returns the local file path so callers can show it to the user.
    Never raises — failures during reporting are themselves reported
    via the logger only.
    """
    try:
        report = _build_report(exc, source=source)
        path = _write_local(report)
        _ship_remote(report)
        return path
    except Exception as e:  # noqa: BLE001 — see docstring
        LOG.error("crash reporter itself failed: %s", e)
        return Path("/dev/null")  # sentinel — caller checks
```

### 6.2 Wiring

```python
# src/<app>/cli/__main__.py
def cli_main() -> None:
    from <app>.observability.excepthook import install_excepthook
    from <app>.observability.threading_excepthook import install_threading_excepthook
    install_excepthook()
    install_threading_excepthook()
    sys.exit(_run_and_translate_exceptions(sys.argv[1:]))
```

```python
# src/<app>/gui/app.py
def run(...) -> int:
    from <app>.gui.qt_excepthook import install_qt_handlers
    from <app>.observability.threading_excepthook import install_threading_excepthook
    install_qt_handlers()
    install_threading_excepthook()
    ...
    return app.exec()
```

**MUSTs:**
- The crash reporter MUST be importable without any optional deps installed (lazy import for `sentry_sdk`, `opentelemetry`, etc.).
- The reporter MUST never raise — it is the last line of defense.
- The reporter MUST tolerate disk-full / read-only filesystem and degrade silently to logging.

### 6.3 Hardening on top of the v0.1 template

The §6.1 template is the v0.1 starting point. The shape covers
the EH-05 invariants (local file + remote ship + never raises),
but it is **not secret-safe out of the box**. Three hardening
layers MUST be added before shipping it to production. The
reference implementation in `mgf-common` already ships all
three:

**A. Bounded `repr()` on exception args.** The v0.1
template formats `exc.args` as `[repr(a) for a in ...]`. For
primitive args (str/int/bool/None) `repr()` is fine — its output
is structurally fixed. For a CUSTOM class, `repr()` may surface
secret state through `__repr__` (e.g., a connection wrapper
whose repr embeds `password=...`). Replace with a
type-allowlist:

```python
import datetime as _dt
import decimal
import uuid
from pathlib import PurePath
from typing import Any

_SAFE_REPR_TYPES: tuple[type, ...] = (
    str, int, float, complex, bool, bytes, bytearray, type(None),
    PurePath, uuid.UUID,
    _dt.datetime, _dt.date, _dt.time, _dt.timedelta,
    decimal.Decimal,
)

def _safe_repr(value: Any) -> str:
    if isinstance(value, _SAFE_REPR_TYPES):
        return repr(value)
    return f"<{type(value).__name__}>"

# In _build_report:
#     "args": [_safe_repr(a) for a in getattr(exc, "args", ())],
```

The allowlist contains types whose `__repr__` is structurally
fixed AND known not to surface arbitrary object state. Anything
else (custom classes AND stdlib containers like `dict`/`list`/
`tuple` whose reprs delegate to elements) lands as
`<TypeName>` — losing some diagnostic info, but bounding the
leak surface.

**B. Redactor pass before egress (rule SC-11).** The v0.1 template
ships the raw report. The hardened shape passes it through the
active `Redactor` BEFORE the local atomic-write AND BEFORE any
remote sink fires:

```python
from <app>.observability.redaction import Redactor

def _redact_report(report: dict[str, Any]) -> dict[str, Any]:
    """Scrub secret-shaped values before egress (SC-11)."""
    try:
        redactor = _get_active_redactor() or Redactor.default()
        redactor.redact(report)
        return report
    except Exception as redact_err:
        LOG.warning(
            "crash reporter redaction failed; emitting minimal payload",
            extra={
                "audit": True,
                "event": "crash.redaction_failed",
                "redaction_error_type": type(redact_err).__name__,
            },
        )
        return _minimal_report(report, error_type=type(redact_err).__name__)
```

**C. Fail-safe minimal payload (rule SC-11).** If the redactor
itself raises (a custom regex with catastrophic backtracking, a
malformed payload), the report is REPLACED with a minimal dict
preserving only the operator-visible structure:

```python
_SENTINEL = "[REDACTION FAILED — payload withheld]"

def _minimal_report(
    original: dict[str, Any], *, error_type: str
) -> dict[str, Any]:
    safe_exc_type = (
        original.get("exception", {}).get("type", "unknown")
    )
    return {
        "timestamp": original.get("timestamp", ""),
        "report_id": original.get("report_id", ""),
        "app": original.get("app", ""),
        "version": original.get("version", ""),
        "source": original.get("source", ""),
        "exception": {
            "type": safe_exc_type,  # type is safe — class name only
            "message": _SENTINEL,
            "args": [_SENTINEL],
        },
        "traceback": [_SENTINEL],
        "cause": None,
        "platform": original.get("platform", {}),
        "redaction_status": "failed",
        "redaction_error_type": error_type,
    }
```

The minimal payload deliberately does NOT include the redactor's
own error MESSAGE — only its TYPE. A redactor that crashed on a
secret-containing input could surface the secret in its own
exception message; type-only is the defensive shape.

**D. Use `mgf.common._io.atomic_write` for the local file.**
The v0.1 template uses inline `tmp.write_text` + `tmp.replace`,
which is *correct* but doesn't `fsync` the body before the
rename. The shared helper does. See `CONFIGURATION.md` §9.

**E. Document the `<APP>_CRASH_SINK` trust assumption.**
The v0.1 template accepts an arbitrary URL via the
`<APP>_CRASH_SINK` env var and POSTs the redacted report there.
This is **correct behaviour** — the operator who set the env var
is the trust anchor — but the assumption MUST be stated
explicitly in the crash-reporter reference doc. See
[`SECURITY_STANDARD.md`](SECURITY_STANDARD.md) §8.5
"Environment variables are a trusted input" for the framing.

Pin these hardening layers with the test shapes in
`TESTING.md` §11.5 (failure-injection) and `TESTING.md` §10.3
(redaction).

---

## 7. Best-effort observers (rule EH-10)

A "best-effort observer" is code that runs **inside a critical-path operation** but whose failure MUST NOT take down that operation. Examples: hook scripts, scheduler tick loops, libvirt event callbacks, alert evaluators.

The canonical shape:

```python
def fire_one(observer: Observer, event: Event) -> bool:
    """Run one observer. Never raises.

    Returns True on success, False on failure. Failures are logged at
    WARNING with the observer's name and the underlying error. The
    caller must continue to the next observer regardless of return.
    """
    try:
        observer.handle(event)
    except AppError as e:
        LOG.warning("observer %s: failed: %s", observer.name, e)
        return False
    except Exception as e:  # noqa: BLE001 — never-raises invariant
        LOG.warning("observer %s: unexpected error: %s", observer.name, e)
        return False
    return True

def fire_all(observers: list[Observer], event: Event) -> int:
    """Run every observer. Returns the count that succeeded."""
    return sum(fire_one(o, event) for o in observers)
```

**MUSTs:**
- The function docstring MUST state explicitly: "Never raises."
- A unit test MUST verify the never-raises contract (inject a raising observer; assert the loop continues).
- Logs MUST identify the observer by name so operators can find the misbehaving one.
- When the function is called from a path that has ALREADY
  committed externally-visible state (e.g.,
  `bootstrap()` has set `_GLOBAL_CONTEXT`), the helper MUST be
  best-effort by contract — propagating a raise leaves the
  caller in an inconsistent state no recovery path can fix.
  This is the same EH-10 shape, applied to one-shot helpers
  rather than a fan-out loop.

**The bootstrap-snapshot pattern (reference example).**
`mgf.common._lifecycle._maybe_emit_runtime_snapshot` is called
from `bootstrap()` AFTER `_GLOBAL_CONTEXT = new_context` has
been set. A propagating raise from inside (e.g., a consumer
`@computed_field` throwing during `settings.model_dump()`)
would leave the process with a fully-wired `AppContext` on the
global AND a propagating `BootstrapError` — an inconsistent
state callers cannot recover from. The fix (`RA-22`): the helper
body is wrapped in `try / except` that emits a WARNING with
`audit=true, event="bootstrap.runtime_snapshot_failed"` and
returns normally:

```python
def _maybe_emit_runtime_snapshot(ctx, settings) -> None:
    try:
        snapshot = {...}
        emit_runtime_snapshot(snapshot)
    except Exception as exc:
        LOG.warning(
            "runtime snapshot failed; bootstrap continues without it",
            extra={
                "audit": True,
                "event": "bootstrap.runtime_snapshot_failed",
                "error_type": type(exc).__name__,
                "error_message": str(exc),
            },
        )
```

The contract: any helper called from `bootstrap()` AFTER the
global is set MUST be best-effort. The audit log makes the
failure observable; the never-raises invariant keeps bootstrap
atomic.

**Consumer pattern.** Best-effort observers typically appear at
fan-out points: hook executors (`fire_one`), scheduler runners,
alert evaluators, event dispatchers. Each catches per-iteration
inside the loop and continues to the next iteration on failure.

---

## 8. Anti-patterns (forbidden)

| Anti-pattern | What's wrong | Right shape |
|---|---|---|
| `try: x() except Exception: pass` | Hides bugs. Permanently. | Translate to typed exception, OR log + re-raise, OR `contextlib.suppress(SpecificError)` in cleanup paths only |
| `try: x() except: ...` (bare except) | Catches `KeyboardInterrupt`, `SystemExit` | Always specify the exception type |
| `raise NewError(str(e))` (no `from e`) | Loses the cause; debugging becomes archaeology | `raise NewError(str(e)) from e` |
| `print(f"error: {e}")` | Bypasses logging; output mixes with user-facing CLI | Use `LOG.error` + `click.echo(..., err=True)` for user-facing |
| Translating in many places | Map drifts; inconsistent messages | One `_translate()` function per provider |
| Catching at the call site of a use case | Errors get swallowed before the top-level handler can categorize | Let typed errors propagate; only catch at the boundary you own |
| Modal dialog for every GUI error | UX disaster; users learn to dismiss without reading | Status bar / log panel by default; modal only for fatal blocking errors |
| `assert` for input validation | Compiled out by `-O` | Raise a typed exception instead |

---

## 9. Testing error paths

**MUSTs:**
- Every typed exception MUST have a unit test that constructs it and asserts: (a) the message format, (b) the field values are preserved, (c) the field values are copied (not aliased) where applicable.
- Every translation seam MUST have tests for: (a) success, (b) the typed translation of each known third-party error, (c) the catch-all "unknown vendor error" path.
- Every "best-effort observer" MUST have a test that injects a raising observer and asserts the loop continues.
- The CLI top-level handler MUST have integration tests asserting exit code per category.
- **Every actionable next-step suffix on a user-facing error message MUST be pinned by a substring test** (rule EH-09). Once `"run <app> <subcommand>"` is appended to a `ResourceNotFoundError` message, it's invisible to the rest of the test suite — a future refactor can drop it without anything failing. The fix is a small dedicated test file (e.g., `tests/unit/test_exceptions_actionable_hints.py`) that asserts each hint with a stable substring (NOT the whole message, so wording can still be tuned). The file's docstring tells the next contributor to add a row whenever they append a new hint. **Without this guard, EH-09 quietly degrades over time.**

---

## 10. Cross-references

- Logging integration (`LOG.exception`, structured fields, trace correlation): [`LOGGING.md`](LOGGING.md).
- Config-related errors (`AppConfigError`, `SchemaValidationError`): [`CONFIGURATION.md`](CONFIGURATION.md).
- The cross-cutting `fire_one` invariant: §7 above (best-effort observers).
- The module-level alias rule for testable defaults: [`DESIGN_PRINCIPLES.md`](DESIGN_PRINCIPLES.md) Appendix A.
- Async task lifecycle ownership (rule DP-11): [`DESIGN_PRINCIPLES.md`](DESIGN_PRINCIPLES.md) §11.
- Security-side traceback scrubbing (rule SC-11): [`SECURITY_STANDARD.md`](SECURITY_STANDARD.md) §9.2.

---

## 11. Async error handling (rule EH-04 + DP-11)

Async code multiplies the failure surface — a task that crashes
silently is far harder to find than a synchronous exception. The
rules below extend EH-* to async.

### 11.1 The asyncio loop exception handler

Rule **EH-04** requires a top-level handler at every entry point;
asyncio's analogue is `loop.set_exception_handler(...)`. It catches
every exception from a task that is **not** awaited (orphaned tasks,
fire-and-forget callbacks, scheduled coroutines).

The canonical install:

```python
# src/<app>/observability/asyncio_handler.py
import asyncio
import logging
from typing import Any

from <app>.observability.crash_reporter import report_now

LOG = logging.getLogger(__name__)

def install_asyncio_handler(loop: asyncio.AbstractEventLoop) -> None:
    """Wire the loop's exception handler into the crash reporter.

    Without this, an exception in an unawaited task is logged to
    stderr by asyncio's default and otherwise lost — no crash
    report, no remote sink. EH-04 + EH-05 require otherwise.
    """

    def handler(loop: asyncio.AbstractEventLoop, ctx: dict[str, Any]) -> None:
        exc = ctx.get("exception")
        msg = ctx.get("message", "<no message>")
        if exc is None:
            LOG.error("asyncio error context (no exception): %s", ctx)
            return
        LOG.error(
            "unhandled asyncio exception: %s",
            msg,
            exc_info=(type(exc), exc, exc.__traceback__),
        )
        report_now(exc, source="asyncio")

    loop.set_exception_handler(handler)
```

Wire from the entry point that creates the loop:

```python
# src/<app>/cli/__main__.py — async entry point
def cli_main() -> int:
    install_excepthook()
    install_threading_excepthook()
    asyncio.run(_async_main())  # default loop
    return 0

async def _async_main() -> None:
    loop = asyncio.get_running_loop()
    install_asyncio_handler(loop)
    await app.run()
```

### 11.2 `gather(return_exceptions=True)` discipline

Default `asyncio.gather(*coros)` propagates the **first** exception
and cancels the rest. That is correct for "all-must-succeed"
fan-out. But many real-world fan-outs are "best-effort" (poll N
hosts, collect what you can):

```python
# WRONG — first failure cancels every sibling
results = await asyncio.gather(*[probe(h) for h in hosts])

# RIGHT — every probe runs to completion; failures returned as exceptions
results = await asyncio.gather(*[probe(h) for h in hosts], return_exceptions=True)

successes: list[ProbeResult] = []
failures: list[tuple[str, BaseException]] = []
for host, result in zip(hosts, results, strict=True):
    if isinstance(result, BaseException):
        failures.append((host, result))
    else:
        successes.append(result)

if failures:
    LOG.warning("probe failed for %d/%d hosts", len(failures), len(hosts))
```

The rule:

- **All-must-succeed** → `gather(*coros)` (default).
- **Best-effort** → `gather(*coros, return_exceptions=True)` and
  partition the results explicitly.

`return_exceptions=True` MUST be paired with the partition — never
`results = await gather(..., return_exceptions=True)` followed by
treating `results` as if every item is a successful value.

### 11.3 `TaskGroup` (Python 3.11+) is the new default

`asyncio.TaskGroup` is the structured-concurrency primitive that
makes "every spawned task is awaited or cancelled at scope exit" a
language-level guarantee:

```python
async def fan_out_required(items: list[Item]) -> list[Result]:
    """All-must-succeed fan-out. Cancellation propagates correctly."""
    results: list[Result] = []

    async def _one(item: Item) -> None:
        results.append(await process(item))

    async with asyncio.TaskGroup() as tg:
        for item in items:
            tg.create_task(_one(item))

    return results
```

If any task raises, the group cancels its siblings, awaits them,
and re-raises an `ExceptionGroup` containing every exception. This
satisfies rule **DP-11** (task ownership) without manual
bookkeeping.

For best-effort fan-out, prefer `gather(return_exceptions=True)`
above — `TaskGroup` is structured around all-must-succeed
semantics.

### 11.4 `ExceptionGroup` handling

Python 3.11 introduced `ExceptionGroup` as the multi-exception
container. Catch via the `except*` syntax:

```python
try:
    async with asyncio.TaskGroup() as tg:
        ...
except* GuestUnreachableError as eg:
    # eg.exceptions contains every GuestUnreachableError; siblings
    # of other types pass through to other except* clauses.
    LOG.warning("hosts unreachable: %s", [str(e) for e in eg.exceptions])
except* OperationError as eg:
    LOG.error("operations failed: %s", [str(e) for e in eg.exceptions])
```

Translation seams (rule **EH-02**) running inside a `TaskGroup`
behave normally — each task's exception is translated as usual,
then aggregated by the group. Consumers handle the `ExceptionGroup`
at the boundary that owns the fan-out.

### 11.5 Cancellation re-raise rule

`asyncio.CancelledError` MUST re-raise unless the catching code is
the cancellation owner:

```python
# RIGHT — pass through
try:
    await long_op()
except asyncio.CancelledError:
    LOG.info("operation cancelled")
    raise

# RIGHT — owner converts to typed exception
try:
    await long_op()
except asyncio.CancelledError:
    raise OperationError("operation cancelled by user") from None

# WRONG — swallowing cancellation breaks structured concurrency
try:
    await long_op()
except asyncio.CancelledError:
    pass
```

Swallowing `CancelledError` defeats `TaskGroup`, `gather`, and any
upstream supervisor — the whole tree thinks the task succeeded
when it was actually cancelled.

### 11.6 Correlation across `await` points

The OpenTelemetry trace context is propagated automatically via
contextvars (asyncio honours contextvars per task). Custom
correlation (request ID, user ID) MUST use `contextvars.ContextVar`
rather than thread-local storage:

```python
import contextvars

request_id: contextvars.ContextVar[str] = contextvars.ContextVar("request_id")

async def handle_request(req_id: str) -> None:
    token = request_id.set(req_id)
    try:
        await do_work()  # do_work and its descendants see request_id.get() == req_id
    finally:
        request_id.reset(token)
```

A `JsonFormatter._add_otel_context` extension reads custom
contextvars and promotes them to log fields:

```python
def _add_correlation(payload: dict) -> None:
    try:
        payload["request_id"] = request_id.get()
    except LookupError:
        pass
```

### 11.7 Async testing references

See [`TESTING.md`](TESTING.md) §7 for the async test patterns:
per-test event loop, cancellation tests, no-`asyncio.sleep` for
synchronisation, etc.

---

## 12. Error message style guide (rule EH-09 expanded)

Rule EH-09 says user-facing messages MUST end in an actionable
next step. This section operationalises that for tone, structure,
and content.

### 12.1 Audience and tone

Error messages have three audiences:

| Audience | Where they read | Tone |
|---|---|---|
| **End user** (not a developer) | CLI output, GUI dialog | Plain language, no internal jargon, action-oriented |
| **Operator** (responds to alerts) | Log records, monitoring | Specific, with identifiers and context |
| **Developer** (debugging) | Log records, crash reports | Full traceback, every internal field |

The same exception serves all three because:
- The user sees the **message string** (rendered by the CLI / GUI).
- The operator sees the **structured fields** (logged via `extra={...}`).
- The developer sees the **traceback + cause chain** (in the crash report).

The message string is the user's lens. Make it count.

### 12.2 The four-part shape

A good user-facing message has four parts, in order:

1. **What failed** — the operation, named.
2. **Why** — the immediate cause, in plain language.
3. **Where** — the relevant identifier (resource name, file path,
   host).
4. **What to do** — the actionable next step.

Examples:

```
no VM named 'prod-1'; run 'myapp vm list' to see available VMs

config file /home/x/.config/myapp/config.yaml is malformed at line
17: expected mapping, got string. Re-edit the file and run
'myapp config validate' to retry.

SSH connection to 10.0.0.5 timed out after 30s; check that the
host is reachable (`ping 10.0.0.5`) and that sshd is running
(`systemctl status sshd` on the remote).

cannot start VM 'prod-1': required port 5901 is already in use by
PID 4711. Stop the conflicting process or change the VNC port in
the VM spec.
```

The pattern is consistent: **what + why + where + how to fix**.

### 12.3 Things to avoid

| Anti-pattern | Why bad | Better |
|---|---|---|
| `internal error` | User has no path forward | Name what failed; point at the crash report |
| `failed to do thing` | Already implied by an exception; no information | Say WHY |
| `error: error: error: ...` | Stacked prefixes from re-wrapping | Translate cleanly at the seam; one prefix |
| `Operation failed: %r` (with the dict's `__repr__`) | Unreadable | Format the relevant fields explicitly |
| Mentioning a Python class name (`KeyError`, `OSError`) | Internal implementation detail | Translate to a typed `AppError` with a human message |
| "Please contact support" | Useless if support means "open an issue against your own project" | Either describe the action or point at logs |
| Trailing period | Grammar choice; pick one and stay consistent | Project convention: NO trailing period in log messages, OPTIONAL period in user-facing messages |

### 12.4 Identifier formatting

| Identifier type | Format | Example |
|---|---|---|
| Resource name | Single quotes | `no VM named 'prod-1'` |
| File path | Bare path (will be tab-completable in most terminals) | `config file /home/x/.config/myapp/config.yaml is missing` |
| URL | Bare URL (terminal autolinking works) | `cannot reach https://api.example.com/v1/health` |
| IP / port | Bare | `port 5901 is already in use` |
| Command to run | Backticks | `run \`myapp vm list\` to see available VMs` |
| Multi-line technical detail | Indent under the message | (see EH-9.5) |

### 12.5 Structured fields

The same exception ALSO carries machine-readable fields (rule
**EH-08**) so log aggregators can filter without parsing the message:

```python
class VMNotFoundError(ResourceNotFoundError):
    """No VM with the given name exists.

    Carries .name so callers can route on the field rather than
    string-matching the message.
    """

    def __init__(self, name: str) -> None:
        self.name = name
        super().__init__(
            f"no VM named {name!r}; run 'myapp vm list' to see "
            f"available VMs"
        )
```

The CLI exit-code mapper logs the structured form:

```python
except VMNotFoundError as e:
    click.echo(f"vm error: {e}", err=True)
    LOG.error(
        "vm not found",
        extra={"vm_name": e.name, "operation": "vm.lookup"},
    )
    return EXIT_RESOURCE_NOT_FOUND
```

A SIEM operator can then filter `vm_name="prod-1"` without parsing
free-text.

### 12.6 Localization

The standard does NOT mandate localisation — every project decides
whether i18n is in scope. When it is:

- The message string lives in the exception's English form.
- The CLI / GUI translates at render time
  (`gettext.translation("myapp").gettext(str(e))`).
- The structured fields stay English (they're API contract).

For projects without i18n, English is the canonical form.

### 12.7 Pinning the actionable-hint suffix

A hint suffix like `; run 'myapp vm list'` is invisible to the rest
of the test suite — a refactor can drop it without anything failing.
Per rule **EH-09**, every hint MUST be pinned by a substring test
in a dedicated file:

```python
# tests/unit/test_exception_actionable_hints.py
"""Pin the actionable-hint suffix on every user-facing exception
message. EH-09: a hint that no test asserts is a hint that
silently rots.

When you add a new typed exception with a user-facing message,
add a row here.
"""

import pytest

from <app>.exceptions import VMNotFoundError, AppConfigError

HINTS = [
    (VMNotFoundError("x"), "myapp vm list"),
    (AppConfigError("..."), "myapp config validate"),
    # ... one row per typed exception with a user-facing message
]

@pytest.mark.parametrize("exc,hint_substring", HINTS)
def test_exception_carries_actionable_hint(exc, hint_substring):
    msg = str(exc)
    assert hint_substring in msg, (
        f"{type(exc).__name__} message lost its actionable hint. "
        f"Expected substring {hint_substring!r}, got: {msg}"
    )
```

The substring (not the whole message) lets wording be tuned without
test churn while still catching "the hint disappeared".

---

## 13. Resilience patterns (rules EH-11, EH-12, EH-13)

Production systems fail. The standard's job is to keep transient
failures from turning into outages: retry the right things, give
up on the wrong things, never amplify upstream load.

### 13.1 Retry with exponential backoff + full jitter (rule EH-11)

The canonical retry shape:

```python
# src/<app>/common/retry.py
"""Retry helper with exponential backoff and full jitter.

Reference: AWS Architecture Blog, "Exponential Backoff And Jitter"
(2015) — full jitter empirically minimises retry-storm contention.
"""

from __future__ import annotations

import logging
import random
import time
from collections.abc import Callable
from typing import TypeVar

from <app>.exceptions import AppError

LOG = logging.getLogger(__name__)
T = TypeVar("T")

def retry(
    fn: Callable[[], T],
    *,
    retryable: tuple[type[BaseException], ...] = (AppError,),
    max_attempts: int = 5,
    base_delay_s: float = 0.5,
    max_delay_s: float = 30.0,
    operation: str = "<unnamed>",
) -> T:
    """Call `fn` with exponential backoff + full jitter on failure.

    Re-raises the last exception when the budget is exhausted.
    Emits an audit log per attempt so retry behaviour is
    observable in production.

    Use ONLY for operations where retrying makes sense — typed
    transient failures (timeouts, 503s, lock contention). NEVER
    for validation errors (the input is wrong; retrying won't
    fix that).
    """
    last_exc: BaseException | None = None
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except retryable as e:
            last_exc = e
            if attempt >= max_attempts:
                LOG.error(
                    "retry budget exhausted",
                    extra={
                        "audit": True,
                        "event": "retry.exhausted",
                        "operation": operation,
                        "attempts": attempt,
                    },
                )
                raise
            # Full jitter: uniform[0, min(cap, base * 2^(attempt-1))]
            ceiling = min(max_delay_s, base_delay_s * (2 ** (attempt - 1)))
            delay = random.uniform(0, ceiling)
            LOG.warning(
                "retrying after failure",
                extra={
                    "audit": True,
                    "event": "retry.attempt",
                    "operation": operation,
                    "attempt": attempt,
                    "delay_s": delay,
                    "exception_type": type(e).__name__,
                },
            )
            time.sleep(delay)
    # Unreachable — the loop either returns or re-raises.
    assert last_exc is not None  # for the type checker
    raise last_exc
```

The async analogue uses `asyncio.sleep` — same shape, same MUSTs:

```python
async def aretry(
    fn: Callable[[], Awaitable[T]],
    *,
    retryable: tuple[type[BaseException], ...] = (AppError,),
    max_attempts: int = 5,
    base_delay_s: float = 0.5,
    max_delay_s: float = 30.0,
    operation: str = "<unnamed>",
) -> T:
    """Async retry with exponential backoff + full jitter."""
    # ... same logic, await fn() and await asyncio.sleep(delay)
```

**MUSTs:**

- **Full jitter** (uniform[0, ceiling]) is the right shape — see
  the AWS reference. "Equal jitter" and "decorrelated jitter" are
  acceptable variations; linear delay and constant delay are
  forbidden (synchronised retry storms hurt the upstream more
  than the original failure).
- **Finite cap** (`max_attempts`) is mandatory. A retry loop with
  no cap is a defect.
- **Per-attempt audit log** (rule SC-17) at WARNING for retries +
  ERROR for exhaustion makes retry behaviour debuggable in
  production.
- **Retryable type allow-list** is explicit. Default is the
  project's category that represents transient failures
  (`OperationError`, `GuestUnreachableError`). Validation errors
  (`SchemaValidationError`) MUST NOT be in the retryable set.

### 13.2 Circuit breaker (rule EH-13)

When an external call is failing repeatedly, retrying it amplifies
load on an already-struggling upstream. The **outcome EH-13 binds**
is that this amplification stays bounded; the circuit breaker below
is the **canonical mechanism** for it — it stops issuing calls for
a cooldown period. Where a finite retry budget (**EH-11**) plus a
documented call rate already cap the load an upstream can see, that
bound is an acceptable equivalent: record it per
[`non-adoption-is-valid`](principles/non-adoption-is-valid.md)
instead of standing up a breaker. The breaker is the reference
shape, not the only conformant one:

```python
# src/<app>/common/circuit_breaker.py
"""Three-state circuit breaker.

States:
  - CLOSED   — normal operation; failures count against threshold.
  - OPEN     — every call rejects immediately for `cooldown_s`.
  - HALF_OPEN — one test call allowed; success → CLOSED, failure → OPEN.
"""

from __future__ import annotations

import enum
import logging
import threading
import time
from collections.abc import Callable
from dataclasses import dataclass, field
from typing import TypeVar

from <app>.exceptions import OperationError

LOG = logging.getLogger(__name__)
T = TypeVar("T")

class CircuitBreakerOpenError(OperationError):
    """Raised when the breaker rejects a call without invoking fn."""

class _State(enum.Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

@dataclass
class CircuitBreaker:
    """Guards an external call with the three-state breaker pattern."""

    failure_threshold: int = 5      # CLOSED → OPEN after N consecutive failures
    cooldown_s: float = 60.0        # OPEN → HALF_OPEN after this many seconds
    name: str = "<unnamed>"

    _state: _State = field(default=_State.CLOSED, init=False)
    _failures: int = field(default=0, init=False)
    _opened_at: float = field(default=0.0, init=False)
    _lock: threading.Lock = field(default_factory=threading.Lock, init=False)

    def call(self, fn: Callable[[], T]) -> T:
        """Run `fn` through the breaker."""
        self._maybe_transition_to_half_open()

        with self._lock:
            if self._state is _State.OPEN:
                LOG.warning(
                    "circuit breaker rejected call",
                    extra={"audit": True, "event": "breaker.reject", "breaker": self.name},
                )
                raise CircuitBreakerOpenError(
                    f"circuit breaker {self.name!r} is OPEN; "
                    f"retry after the cooldown ({self.cooldown_s:.0f}s)"
                )

        try:
            result = fn()
        except Exception:
            self._record_failure()
            raise
        else:
            self._record_success()
            return result

    def _maybe_transition_to_half_open(self) -> None:
        with self._lock:
            if self._state is _State.OPEN and (
                time.monotonic() - self._opened_at >= self.cooldown_s
            ):
                self._state = _State.HALF_OPEN
                LOG.info(
                    "circuit breaker entering HALF_OPEN",
                    extra={"audit": True, "event": "breaker.half_open", "breaker": self.name},
                )

    def _record_failure(self) -> None:
        with self._lock:
            self._failures += 1
            if self._state is _State.HALF_OPEN or self._failures >= self.failure_threshold:
                self._state = _State.OPEN
                self._opened_at = time.monotonic()
                LOG.error(
                    "circuit breaker OPEN",
                    extra={
                        "audit": True,
                        "event": "breaker.open",
                        "breaker": self.name,
                        "failures": self._failures,
                    },
                )

    def _record_success(self) -> None:
        with self._lock:
            self._failures = 0
            if self._state is _State.HALF_OPEN:
                self._state = _State.CLOSED
                LOG.info(
                    "circuit breaker CLOSED",
                    extra={"audit": True, "event": "breaker.closed", "breaker": self.name},
                )
```

Combined with retry:

```python
breaker = CircuitBreaker(failure_threshold=5, cooldown_s=60.0, name="otel-collector")

def ship_telemetry(payload):
    return retry(
        lambda: breaker.call(lambda: client.post("/v1/traces", json=payload)),
        retryable=(GuestUnreachableError,),
        max_attempts=3,
    )
```

**MUSTs:**

- **State-change audit logs** (open / close / half-open) so an
  operator can see when the breaker tripped and recovered.
- **Per-call rejection log** at WARNING (not ERROR — rejection is
  the breaker's job).
- **Half-open MUST allow exactly one test call** — concurrent
  test calls would amplify load again.

### 13.3 Bulkhead (concurrency limiting)

A bulkhead caps the concurrent resource usage so a slow upstream
cannot exhaust the application's worker pool:

```python
import asyncio
from contextlib import asynccontextmanager

class Bulkhead:
    """Cap concurrent operations.

    Use one bulkhead per logical pool (e.g., one per upstream
    service). When the cap is reached, new acquirers wait — or
    fail fast if `nowait=True`.
    """

    def __init__(self, *, max_concurrent: int, name: str) -> None:
        self._sem = asyncio.Semaphore(max_concurrent)
        self._name = name

    @asynccontextmanager
    async def acquire(self, *, nowait: bool = False):
        if nowait and self._sem.locked():
            raise OperationError(
                f"bulkhead {self._name!r} is full; try again later"
            )
        await self._sem.acquire()
        try:
            yield
        finally:
            self._sem.release()
```

```python
otel_bulkhead = Bulkhead(max_concurrent=10, name="otel-collector")

async def ship_telemetry(payload):
    async with otel_bulkhead.acquire():
        await client.post("/v1/traces", json=payload)
```

A slow OTel collector can no longer back-pressure the entire
application — only the bulkhead's pool of 10 sees the slowness.

### 13.4 Timeout cascades

When operation A calls B which calls C, the timeouts MUST cascade
**downward** — C's timeout MUST be shorter than B's, which MUST be
shorter than A's. Otherwise B times out before C reports its
failure, and the failure mode is "I don't know what went wrong":

```
A (user request) budget: 30s
└── B (use case) budget: 25s ← 5s slack for A's wrap-up
    └── C (HTTP call)    budget: 10s    ← 15s slack for B's other work
        └── D (TCP/TLS)  budget: 5s
```

The cascade lets the inner failure surface clearly. The slack at
each level lets the outer layer translate, log, and clean up.

The pattern in code uses `asyncio.timeout` (Python 3.11+):

```python
async def use_case():
    async with asyncio.timeout(25):  # B's budget
        return await http_client.get("/api/v1/x")  # C uses its own 10s timeout
```

### 13.5 Idempotency (rule EH-12)

An operation that may be retried (or executed twice via at-least-
once semantics in a queue) MUST be idempotent: running it twice
produces the same result as running it once.

| Operation | Natural idempotency key |
|---|---|
| Create a VM with a given name | The VM name |
| Process a webhook event | The event's `delivery_id` |
| Send an email about an alert | (alert_id, version) |
| Charge a credit card | The merchant's `idempotency_key` header |
| Insert a row into a DB | A unique constraint on the natural key |

For DB inserts: `INSERT ... ON CONFLICT DO NOTHING` (Postgres) or
`INSERT OR IGNORE` (SQLite). For idempotent state machines: check
the current state first; transition only if not already in the
target state.

Operations that MUST NOT be retried until idempotency is
established: monetary transfers, irreversible side-effects,
notifications. In those cases, the right shape is
**at-most-once** delivery with a manual reconciliation step on
failure.

### 13.6 What NOT to retry

The retry / breaker / bulkhead patterns above are tools for
**transient** failures. Permanent failures MUST fail fast:

| Failure type | Retry? |
|---|---|
| Network timeout | Yes |
| `503 Service Unavailable` | Yes (with `Retry-After` if present) |
| `429 Too Many Requests` | Yes (with backoff at least matching `Retry-After`) |
| TCP reset, DNS failure | Yes |
| Lock contention (DB, file lock) | Yes (short backoff) |
| `400 Bad Request`, validation error | **NO** — the input is wrong; retry won't fix it |
| `401 Unauthorized`, `403 Forbidden` | **NO** — auth failures need credential refresh, not retry |
| `404 Not Found` (resource-not-found) | **NO** — the resource doesn't exist |
| `5xx` from a service that returns 5xx for invalid input | **NO** — the upstream lies; treat as 4xx |
| Programming errors (`AttributeError`, `KeyError`) | **NO** — these are bugs |

The retryable type allow-list (§13.1) MUST exclude the "NO" rows.

### 13.7 Anti-patterns

| Anti-pattern | What's wrong | Right shape |
|---|---|---|
| `for _ in range(3): try: do(); except: continue` | No backoff; tight retry storm | Use the `retry()` helper |
| `time.sleep(1); retry; time.sleep(1); retry` | Linear delay; synchronised storms | Exponential + jitter |
| `while True: try: do(); except: pass` | No cap; failure invisible | Cap + audit log |
| Retry on `ValueError` / `TypeError` | These are bugs, not transient failures | Restrict retryable types |
| Retry without idempotency | Two side-effects per "successful" operation | Establish idempotency first |
| Circuit breaker shared across logically distinct upstreams | One service's outage trips calls to a healthy one | One breaker per upstream |
| Per-call breaker (state not shared) | Defeats the breaker | Module-level instance, thread-safe |

---

## 14. Cross-references for resilience

- Async timeout primitive (`asyncio.timeout`):
  [`DESIGN_PRINCIPLES.md`](DESIGN_PRINCIPLES.md) §11.5.
- Cancellation-respecting retry: cancellation must propagate
  through the retry helper — see [`DESIGN_PRINCIPLES.md`](DESIGN_PRINCIPLES.md)
  §8 + the `CancellationToken` integration.
- Audit-log convention (rule SC-17) used by retry / breaker
  state-change logs: [`SECURITY_STANDARD.md`](SECURITY_STANDARD.md) §9.3.
- Performance budgets (a retry that loops past its caller's
  budget is a defect): [`DESIGN_PRINCIPLES.md`](DESIGN_PRINCIPLES.md) §6.
- Test patterns for resilience (injecting transient failures
  through the mock `FailurePlan`):
  [`TESTING.md`](TESTING.md) §4.

---

*End of `ERROR_HANDLING.md`.*
