Skip to content

Environment, Secrets & Observability — Setup Guide

Audience: humans setting this up, and AI sessions wiring a new AltaForge solution the same way. Read §1–§3 for the model; follow §5 to set up a project; §7 is a terse checklist an agent can execute end-to-end.

This documents the pattern proven on DocGen (org Tomorrow Law, space dev, project docgen, Arize AX ca-central region).

1. The model in one rule

A name ending in _KEY / _TOKEN / _SECRET / _PASSWORD is a secret. Everything else is config. Secrets never get committed and never become a GitHub Variable. Config can live anywhere.

2. Three homes for settings

Tier Examples Local dev CI (GitHub Actions) Prod
Config (non-secret) endpoints, space/project IDs, flags, model names, DOCGEN_ENV .env.local GitHub Variables env / Variables
Secret (keys) ARIZE_API_KEY, ANTHROPIC_API_KEY, ALTAFORGE_LLM_API_KEY .env.local (gitignored) GitHub Secrets Azure Key Vault
Which environment dev / staging / prod n/a GitHub Environments the deploy target
  • A committed .env.example documents every variable name (with a CONFIG-vs-SECRET legend) and blank/placeholder values — never real values.
  • .gitignore must cover .env, .env.local, .env.*.local.

3. How a value reaches the running app

.env.local        (your machine)            ┐
GitHub Variables + Secrets   (CI)           ├──►  process env  ──►  configure_observability()  ──►  Arize / Phoenix
Azure Key Vault   (prod)                    ┘        ▲                  (reads ARIZE_* at boot)
                                                     │
                              server.py::_load_dotenv_files() loads .env.local
                              (docgen/ → demo/ → repo root) at startup

The app never hardcodes a value; it reads os.environ. Locally, the FastAPI server's _load_dotenv_files() merges .env.local into the env at boot (without overriding anything already set). In CI/prod the platform injects the env.

4. Observability specifics (Arize AX / Phoenix) — read before you wire it

The shared library altaforge.core.runtime provides the wiring: - Fail-loud init (preferred): require_tracing(service_name, project_name=None, observability_url=None, headers=None) and require_arize_ax(service_name, space_id, api_key, project_name=None, endpoint=…). These raise TracingNotConfiguredError if nothing is configured — a forgotten config crashes at startup instead of silently emitting nothing. The only quiet path is the explicit ALTAFORGE_TRACING_DISABLED opt-out. - Lenient primitives: configure_tracing(...) / configure_arize_ax(...) — same args, but silently no-op when unconfigured. The require_* wrappers call these after the loud check; reach for the primitives directly only when a silent no-op is genuinely what you want. - Non-graph instrumentation: @traced, traced_span, af_run, submit_with_context, SpanKind. - tracing_disabled() — True when ALTAFORGE_TRACING_DISABLED is set (1/true/yes/on).

A solution wraps these in a tiny observability.py::configure_observability() that reads env and calls them at boot (see docgen/src/docgen/observability.py for the reference).

Five hard-won facts (these cost real debugging):

  1. The Arize project = the OTel service.name. AX does not name the project from model_id or openinference.project.name. So a project named "Foo" requires service.name="Foo". We expose this as ARIZE_PROJECT_NAME → it becomes service.name.
  2. A trace with only service.name is rejected with HTTP 500. Arize AX requires a project/model identifier on the resource too. The lib sets model_id + openinference.project.name (= the project) alongside service.name, which satisfies ingestion. If you ever see a 500 on ingest, this is why.
  3. Use your region's OTLP endpoint. Your Arize app URL tells you the region: https://app.<region>.arize.com → OTLP is https://otlp.<region>.arize.com/v1/traces. The default otlp.arize.com is US-only and will 500/timeout for a non-US tenant. (Phoenix self-hosted: http://localhost:6006/v1/traces, no key.)
  4. Two different keys. The ingestion (service) key writes traces — it is write-only (a read attempt returns 401). To read/verify traces you need a separate developer key. They are not interchangeable.
  5. Tracing is loud-by-default. require_tracing/require_arize_ax (and therefore configure_observability) raise at startup when neither a full Arize credential pair nor an OTLP endpoint is present — because the failure mode that actually bites is "I thought it was tracing but nothing was exported." A half Arize config (space id but no api key) also raises. To run with no tracing on purpose — tests, local dev, a CI job that doesn't exercise it — set ALTAFORGE_TRACING_DISABLED=1 (the only sanctioned quiet path). The docgen test suite sets it in conftest.py; pnpm dev sets it for the Python server (override with ALTAFORGE_TRACING_DISABLED=0).

Arize hierarchy: Org → Space → Project. Example: org Tomorrow Law → space dev → project docgen. One space per environment is the clean split; the org/tenant is also stamped on every span (organization.slug) so you can filter within a project.

5. Step-by-step: set up a new project

5.1 Arize side (UI)

  1. Open your regional Arize app, e.g. https://app.ca-central-1a.arize.com (swap the region to match your account).
  2. Note the region from that URL — you need it for the OTLP endpoint (§5.3).
  3. Space → create or pick the environment space (e.g. dev). Its Space ID is in Settings. (It's a base64 identifier like U3BhY2U6… — an identifier, not a secret.)
  4. Settings → API Keys → Service Keys tab → + New API Key:
  5. Key Type: Service Key
  6. Space Role: Member (the minimum that can ingest; Read-Only cannot write)
  7. Scope to the single space only (not org-wide, not Admin)
  8. Copy the key immediately — shown once. This is your ingestion key (*_ARIZE_API_KEY).
  9. (Optional, for programmatic verification) create a developer/user key (Settings → API Keys, user key, or enable Developer Access) — this reads the GraphQL API. Store as ARIZE_DEVELOPER_KEY.
  10. The project doesn't need pre-creating — it's auto-created from service.name on first trace. Pick the name you want (e.g. the app name; avoid spaces).

Docs: Arize API & Service Keys · OpenTelemetry (arize-otel)

5.2 Decide the values

Variable Tier Example Notes
DOCGEN_ENV config dev per environment
ALTAFORGE_OBSERVABILITY_URL config https://otlp.ca-central-1a.arize.com/v1/traces your regional endpoint
ARIZE_SPACE_ID config U3BhY2U6… from Space settings
ARIZE_PROJECT_NAME config docgen becomes service.name → the project; no spaces
ARIZE_API_KEY secret … the ingestion service key
ARIZE_DEVELOPER_KEY secret … optional, read/verify only
ALTAFORGE_TRACING_DISABLED config true the explicit opt-out — set in tests/local/CI that run without tracing; leave blank in prod so a misconfig fails loud

5.3 Local dev

Put everything in <app>/.env.local (gitignored). The server loads it at boot. 

DOCGEN_ENV=dev
ALTAFORGE_OBSERVABILITY_URL=https://otlp.ca-central-1a.arize.com/v1/traces
ARIZE_SPACE_ID=U3BhY2U6…
ARIZE_PROJECT_NAME=docgen
ARIZE_API_KEY=<ingestion service key>          # secret
ARIZE_DEVELOPER_KEY=<read key>                  # optional, secret
Also commit a .env.example with these names and blank values + the CONFIG/SECRET legend.

5.4 GitHub — config in, keys out

UI: https://github.com/<org>/<repo>/settings/variables/actions (Variables), /settings/secrets/actions (Secrets), /settings/environments (Environments).

Repo-level Variables = global config: 

gh variable set WEBSEARCH_PROVIDER -R <org>/<repo> --body tavily
Environments (per env) for env-specific config — keys never go here: 
for e in dev staging prod; do gh api -X PUT repos/<org>/<repo>/environments/$e; done
gh variable set DOCGEN_ENV                  --env dev -R <org>/<repo> --body dev
gh variable set ARIZE_SPACE_ID              --env dev -R <org>/<repo> --body 'U3BhY2U6…'
gh variable set ARIZE_PROJECT_NAME          --env dev -R <org>/<repo> --body docgen
gh variable set ALTAFORGE_OBSERVABILITY_URL --env dev -R <org>/<repo> --body 'https://otlp.ca-central-1a.arize.com/v1/traces'
Secrets: keep keys out of GitHub entirely (this project's policy) — they live in .env.local locally and Azure Key Vault in prod. If your project does put CI secrets in GitHub, use Environment Secrets (gh secret set NAME --env dev), never Variables, and never across environments.

Verify nothing leaked: 

gh secret list -R <org>/<repo>            # expect empty (keys not in GH)
gh variable list -R <org>/<repo>          # config only
gh variable list --env dev -R <org>/<repo>

5.5 Wire the app

Add <app>/observability.py mirroring docgen/src/docgen/observability.py: 

from altaforge.core.runtime import require_arize_ax, require_tracing
import os
SERVICE_NAME = '<app>'

def configure_observability() -> bool:
    space_id = os.environ.get('ARIZE_SPACE_ID')
    api_key  = os.environ.get('ARIZE_API_KEY')
    url      = os.environ.get('ALTAFORGE_OBSERVABILITY_URL') or None
    project  = os.environ.get('ARIZE_PROJECT_NAME') or SERVICE_NAME   # = service.name = the Arize project
    # If either AX credential is present, require BOTH (a lone one is a misconfig).
    if space_id or api_key:
        return require_arize_ax(service_name=project, space_id=space_id, api_key=api_key,
                                **({'endpoint': url} if url else {}))
    return require_tracing(service_name=project, observability_url=url)
Call configure_observability() once at boot (app factory + any headless CLI entrypoint). It is idempotent and raises TracingNotConfiguredError when nothing is configured — so a forgotten Arize/OTLP setup fails loud instead of silently emitting nothing. Tests/local/CI that run without tracing opt out explicitly via ALTAFORGE_TRACING_DISABLED=1 (set it in conftest.py so it's in place before any import triggers configure_observability). Then instrument the pipeline with af_run(...) (root span, stamps org/run id) + @traced/traced_span on phases, and let LLM spans come free from the instrumented altaforge.llm client.

5.6 Verify it landed

  • Quick (human): open the Arize app → space → your project → Traces; look for the root span (af_run).
  • Programmatic (needs the developer key): query the GraphQL API and check the project + span count: 
    import httpx
gql = 'https://app.<region>.arize.com/graphql'
H = {'x-api-key': '<ARIZE_DEVELOPER_KEY>', 'Content-Type': 'application/json'}
q = '{ node(id:"<ARIZE_SPACE_ID>"){ ... on Space { name models(first:50){ edges { node { name modelType } } } } } }'
print(httpx.post(gql, headers=H, json={'query': q}).json())
    The project appears as a model (Arize calls projects "models") with modelType: generative_llm. Span rows: Model.spanRecordsPublic(dataset:{startTime,endTime,environmentName:tracing, externalModelVersionIds:[], externalBatchIds:[]}, includeRootSpans:true).

GitHub Variables/Secrets/Environments UI: https://github.com/<org>/<repo>/settings. Azure Key Vault for prod secrets: docs.

6. Gotchas (don't relearn these)

  • HTTP 500 on ingest → missing project identifier on the resource. The lib handles it; if you bypass the lib, set model_id + openinference.project.name, not just service.name.
  • HTTP 500/timeout that looks like a bad key → wrong region. Use otlp.<region>.arize.com.
  • Project shows a name you didn't expect → it's service.name, not model_id. Set ARIZE_PROJECT_NAME.
  • HTTP 403 after many quick sends → ingestion rate-limit from a burst; it clears on its own. Reads are unaffected.
  • 401 when reading → you used the ingestion key; reads need the developer key.
  • Don't put DOCGEN_ENV (or any env-specific value) as a repo-level Variable → it would apply to all environments. Use Environment Variables.

7. Checklist for an AI session (new project)

  1. Confirm the lib is available: from altaforge.core.runtime import require_arize_ax, require_tracing, traced, af_run.
  2. Get from the human: Arize app URL (→ region), space id, ingestion key (Member, single space), optional developer key, desired project name (no spaces).
  3. Create <app>/observability.py::configure_observability() (§5.5) using the require_* (fail-loud) wrappers; call it at every boot entrypoint; set ALTAFORGE_TRACING_DISABLED=1 in the test conftest + any tracing-free CI job.
  4. Instrument: af_run root + @traced phases + thread propagation (submit_with_context) on any pools.
  5. .env.local: all config + secrets (§5.3). Add/extend .env.example with names + the CONFIG/SECRET legend. Confirm .gitignore covers env files.
  6. GitHub: repo Variables (global config) + dev/staging/prod Environments with env config; keys stay out of GH (gh secret list empty). (§5.4)
  7. Verify: send a trace; confirm via the read API (§5.6) the project exists with ≥1 root span. Read the export result (SpanExportResult.SUCCESS) — never trust a piped exit.
  8. Gate before shipping any code change: run the repo's full test suite + (for the lib) ruff check + ruff format --check + mypy + pytest. Secrets never in chat, never committed.

Reference implementation: docgen/src/docgen/observability.py, docgen/.env.example, vendor/altaforge-libraries/.../core/runtime/tracing.py. Design: docs/design/specs/2026-06-13-altaforge-observability-tracing-evals-design.md.