Skip to content

App diagnostics — the /test standard

Every AltaForge app exposes a /test diagnostics page: live health of the deployment (config, secrets, and every dependency the app calls), plus a headless make doctor CLI that runs the same checks for CI and for agents (e.g. Claude) that need to verify an app is wired correctly without a browser.

It has three moving parts:

Part Where it lives What it is
Backend checks + HTTP routes altaforge.diagnostics (this repo) Pluggable Diagnostic classes mounted at GET /api/test/*
Frontend page altaforge-ui → <DiagnosticsPage/> One control that renders the checks as tabs; auto-hides what an app doesn't have
Headless runner app-side make doctor Runs the same checks from the CLI, non-zero exit on failure

1. Backend — altaforge.diagnostics

The contract

A check implements the Diagnostic protocol:

from typing import Protocol
from altaforge.diagnostics import ComponentStatus

class Diagnostic(Protocol):
    name: str  # route suffix: GET /api/test/{name}
    async def run(self) -> list[ComponentStatus]: ...

run() is best-effort: never raise for a single component — classify it as unreachable instead. Raise only if the check cannot run at all (e.g. a registry won't load); the router maps that to a synthetic failed component so the page still renders (never a 500).

Each component reports a check-agnostic ComponentStatus:

ComponentStatus(
    id="db",
    display_name="Primary database",
    reachable=True,
    latency_ms=12,
    detail="mssql, 1 pool",
    skipped=False,   # dev-only component → reported but excluded from the overall grade
)

Built-in checks

Importable from altaforge.diagnostics:

Class name Probes
EnvDiagnostic(manifest) env Required env vars present / well-formed (ENV framework manifest)
SecretsDiagnostic(contract) secrets Key Vault reachable and every env.secrets.toml key resolves
LlmGatewayDiagnostic() llm LiteLLM gateway reachable
DatabaseDiagnostic() db DB connection
AzureStorageDiagnostic() storage Blob storage
McpDiagnostic() mcp Each registered MCP server: reachable + RBAC + data-mode

Mounting the routes

from altaforge.diagnostics import create_diagnostics_router

app.include_router(create_diagnostics_router(diagnostics=build_diagnostics()))

This mounts, under prefix (default /api/test):

  • GET /api/test/{name} per check — returns DiagnosticsResponse with a 200 and an overall verdict (ok | fail | empty) so a UI always renders.
  • GET /api/test — aggregate of every check.
  • ?strict=1 on any of the above — returns 503 on a real failure, for exit-code / CI use.
  • (MCP only) POST /api/test/mcp/{server_id}/mode and .../call for the MCP panel's data-mode toggle and test-call.

create_app auto-mounts create_diagnostics_router() with just the MCP check, so every backend gets GET /api/test/mcp for free. Pass diagnostics=[...] to register the rest.

Security. These routes reveal internal component URLs, so they must run under the app's auth — the router does not add itself to DEFAULT_EXCLUDED_PATHS, and create_app mounts it after the auth middleware. What tabs the frontend renders is UX, not the security boundary.

Centralize the check set in a single factory so the HTTP router and the CLI can't drift:

# app/diagnostics_setup.py
from pathlib import Path
from altaforge.diagnostics import (
    Diagnostic, EnvDiagnostic, SecretsDiagnostic,
    LlmGatewayDiagnostic, DatabaseDiagnostic, AzureStorageDiagnostic,
)
from app.env_manifest import ENV_MANIFEST

SECRETS_CONTRACT = Path(__file__).resolve().parents[2] / "env.secrets.toml"

def build_diagnostics() -> list[Diagnostic]:
    return [
        EnvDiagnostic(ENV_MANIFEST),
        SecretsDiagnostic(SECRETS_CONTRACT),
        LlmGatewayDiagnostic(),
        DatabaseDiagnostic(),
        AzureStorageDiagnostic(),
    ]

2. Headless — make doctor (CI + agents)

A thin CLI over the same build_diagnostics(). Runs every check, prints a report, and exits non-zero on any real failure — so CI and agents can diagnose an app without a browser or auth session:

# app/doctor.py  →  python -m app.doctor
import asyncio, sys
from app.diagnostics_setup import build_diagnostics

async def main() -> int:
    failed = False
    for check in build_diagnostics():
        for c in await check.run():
            mark = "skip" if c.skipped else ("ok" if c.reachable else "FAIL")
            print(f"[{mark:>4}] {check.name}/{c.id} - {c.detail or ''}")
            failed = failed or (not c.reachable and not c.skipped)
    return 1 if failed else 0

if __name__ == "__main__":
    sys.exit(asyncio.run(main()))

doctor:  ## Run app health checks headless (CI / agents)
    cd backend && uv run python -m app.doctor

3. Frontend — <DiagnosticsPage/>

One control from altaforge-ui; no per-app tab wiring.

// app/test page
import { DiagnosticsPage } from "altaforge-ui/components";
export default function TestPage() {
  return <DiagnosticsPage />;   // defaults to endpointBase="/api/test"
}

It renders tabs App health (llm · db · storage) · Secrets · Environment, plus a dynamic MCP tab that appears only when the backend registers the MCP check (GET /api/test/mcp doesn't 404). Apps with no MCP registry get no MCP tab — no dead tabs, no wiring. Individual App-health rows also self-hide when their section endpoint 404s, so you register only the checks your app has.

Two things every consumer must get right:

  1. Import base.css once at the app root (token overrides), in order: 
    import "altaforge-ui/themes/styles.css";
import "altaforge-ui/base.css";
import "altaforge-ui/components/styles.css";
  2. Render /test even when org loading breaks. Diagnostics are most useful exactly when org/config resolution is failing, so keep /test under your layout/header but outside the org gate (still behind auth). Don't let a provider swallow the route by rendering a full-screen error in place of its children — move loading/error UI into the gate, not the provider.

Adoption checklist

  • [ ] Backend: add a build_diagnostics() returning your check set.
  • [ ] Backend: include_router(create_diagnostics_router(diagnostics=build_diagnostics())).
  • [ ] Backend: add doctor.py + a make doctor target over the same factory.
  • [ ] Frontend: point /test at <DiagnosticsPage/>; bump the altaforge-ui submodule.
  • [ ] Frontend: add the base.css import; keep /test outside the org gate but under auth.
  • [ ] Verify: make doctor locally, and load /test against a running backend.

Examples

  • SDP — altaml/altaforge-sdp#558: diagnostics_setup.build_diagnostics(), doctor.py + make doctor, and /test on the shared <DiagnosticsPage/>.
  • VS apps (contentgen · dealdesk · moderation) — carry the earlier per-app MCP /test; migrating to <DiagnosticsPage/> is in progress.