TDD — BYOK LLM Keys for SDP Consumers¶

status: draft                # draft | reviewed | approved
date: 2026-07-06
author: Mark Ly
prd: n/a — standalone cross-cutting design (no parent PRD)
brief: n/a
written-by: TechLead
template-version: 2.2

Written by: TechLead. Reviewed by another senior engineer + the SDP owner before any consumer builds against it. Standalone — this is an architecture/design note, not a release TDD, so PRD cross-refs ([P-*]) are omitted; requirements are stated inline.

Change History¶

Date	Author	Change
2026-07-06	Mark Ly	Initial draft

TL;DR `[T-tldr]`¶

Problem: SDP is a shared, multi-tenant engine that makes LLM calls with its own key (ALTAFORGE_LLM_API_KEY from SDP's env/vault). Cost, quotas, rotation, and blast-radius are pooled on one SDP key instead of scoped per consumer.
Target: BYOK — each consuming app (CICI first) owns a LiteLLM gateway (virtual) key in its own vault and passes it to SDP per request; SDP runs that request on the caller's key.
The library already supports it: altaforge.llm's client(api_key=…) / LLMClient(api_key=…) take a per-call key (env is only the fallback). The work is an SDP app-layer seam, not new LLM plumbing.
Safe transport = server-side BFF only: the key rides consumer backend → SDP in a header over internal TLS; the browser is never in the key path. A Vite SPA calling SDP directly is disqualified for carrying the key.
SDP responsibilities: read X-AltaForge-LLM-Key, thread into LLMClient(api_key=…), redact it from logs + Arize/OTel spans, and authenticate the caller separately (Entra/JWT) — the LLM key is not an auth credential.
Phases: (1) SDP caller-key seam + redaction; (2) CICI consumer contract + BFF forwarding; (3) roll out to remaining consumers, then retire SDP's own-key fallback.

System Overview `[T-system]`¶

Consumers hold their own LiteLLM virtual key server-side and forward it to SDP per request. SDP uses the caller's key for that request's LLM calls and never persists it. Caller authentication (Entra) is a separate, parallel concern.

                      consumer org boundary                         shared engine
  ┌───────────────────────────────────────────────┐   ┌──────────────────────────────┐
  │  browser ──▶ consumer BFF (Next server route    │   │  SDP (FastAPI)               │
  │             OR existing FastAPI backend)        │   │                              │
  │             • loads ici--llm-api-key            │   │  • authN caller (Entra/JWT)  │
  │               from OWN vault (server-only env)  │   │  • reads X-AltaForge-LLM-Key │
  │             • attaches header, TLS ────────────────▶│  • LLMClient(api_key=<hdr>)  │──▶ LiteLLM ──▶ model
  │                                                 │   │  • redacts header in logs/   │    gateway
  │  (browser never sees the key)                   │   │    OTel; never stores it     │
  └───────────────────────────────────────────────┘   └──────────────────────────────┘

Key never leaves the consumer's org boundary at rest (its own vault); it transits, encrypted, only for the duration of a request and is used transiently in SDP memory.

Tech Stack `[T-stack]`¶

Layer	Choice	Rationale (only if non-default)
LLM client lib	`altaforge.llm` (`LLMClient`, `client()`)	Already supports per-call `api_key=`; no new dependency
Engine	SDP — FastAPI + LangGraph	Existing
Consumer BFF	Existing FastAPI backend or Next server route	Either is a valid server-side BFF; Next not required for this design (see `[T-decision-7]`)
Key type	LiteLLM virtual/gateway key	Revocable, rate-limited, per-consumer; not a raw `sk-ant-*`
Secrets	`altaforge.envsync` + per-consumer Key Vault	Consumer's key lives in its own vault (`icistgcanctrlkv01`), not SDP's
Caller auth	Entra / `JwtAuthBackend` (existing)	Separate from the LLM key
Observability	OTel → Arize / Loki (existing)	Must add header redaction (see `[T-decision-6]`)
Transport	internal HTTPS pod FQDNs	Existing; TLS required for the key hop

Data Model `[T-entity-N]`¶

n/a in v1. BYOK introduces no persistent entities. The key is transient — held at rest only in the consumer's vault (already managed by envsync), used in-memory per request in SDP, never written to any SDP table, log, or trace. (This "never persist the caller key" property is a hard requirement, enforced in [T-decision-6].)

API Contracts `[T-api-N]`¶

`[T-api-1]` SDP LLM-backed endpoints — new optional request header¶

Applies to every SDP endpoint that triggers an LLM call (extraction, assessment, generation).

New header:

X-AltaForge-LLM-Key: <consumer LiteLLM virtual key>

Semantics: - Precedence: header key wins; if absent, SDP falls back to its own ALTAFORGE_LLM_API_KEY only during migration ([T-decision-5]), then the fallback is removed and a missing header is an error. - After migration — missing header: 400 { "error": "missing X-AltaForge-LLM-Key" } on LLM-backed routes. - Caller auth is separate: an unauthenticated caller gets 401 regardless of the LLM header. A valid caller with a bad/rejected LLM key surfaces the upstream LiteLLM error (e.g. 402/429) mapped to a clean 502 { "error": "llm_key_rejected" }. - Never echoed: the key is never returned in any response body, error, or header.

`[T-api-2]` Consumer BFF → SDP (server-side)¶

The consumer's server route/proxy attaches the header from its server-only environment (ALTAFORGE_LLM_API_KEY, hydrated by envsync). Browser-originated requests never carry the key; the BFF injects it.

Component / Module Structure `[T-files]`¶

Distinctive touch-points only (not a full tree):

SDP (altaforge-sdp)
  <api layer>/deps.py         # extract X-AltaForge-LLM-Key from request -> request-scoped value
  llm integration:
    config.step_llm(step)     # resolve model+mode (existing)
    llm.step_client(step)     # MUST accept a per-request api_key and build/select client by (auth_mode, key)
                              #   — NOT reuse the process-cached env-key client (see [T-decision-3], [T-q-1])
  observability.py            # add X-AltaForge-LLM-Key to the redaction/deny list (httpx + logging + OTel)

consumer (e.g. client-canada-ici)
  backend/env.secrets.toml    # add ALTAFORGE_LLM_API_KEY = { kv = "ici--llm-api-key" }
  backend/<bff>               # attach header on server-side calls to SDP (BFF)
                              # OR, if converted, a Next server route does the same

Auth Model `[T-auth]`¶

Caller authentication is unchanged and independent: SDP authenticates the calling service/user via Entra / JwtAuthBackend (existing BFF path). This authorizes the request.
The LLM key authorizes nothing in SDP — it only scopes the downstream LLM spend/quota. It must never be accepted as, or conflated with, a caller credential. (Enforced by treating the two as separate inputs: reject on failed caller-auth before the LLM key is even read.)

Key Design Decisions `[T-decision-N]`¶

T-decision-1: Pass the key server-side (BFF → SDP) in a header — not from the browser, not seeded into SDP's vault. Decision: the consumer's server tier attaches the key; SDP reads it from the request header. Because: the browser must never see a long-lived key (a Vite SPA calling SDP directly would ship it to every user's network tab). And seeding each consumer's key into SDP's vault would copy credentials across the org boundary — the exact thing the KV standard ("vault = org boundary, never copy a credential across") forbids. Header-over-TLS from a server tier keeps the key in the consumer's boundary at rest and transient everywhere else.

T-decision-2: The key is a LiteLLM virtual/gateway key, never a raw provider key. Decision: consumers seed ici--llm-api-key = a LiteLLM gateway key. Because: a virtual key is revocable, rate-limited, and scoped per consumer — if it leaks, you rotate one key with bounded blast radius, versus a raw sk-ant-* with full account access.

T-decision-3: SDP uses the caller key via the existing LLMClient(api_key=…) — no new LLM plumbing. Decision: thread the header value into the client the library already exposes. Because: altaforge.llm already resolves api_key kwarg over env (resolved_key = api_key or os.environ.get('ALTAFORGE_LLM_API_KEY')). The seam is request→client wiring, not a new integration.

T-decision-4: Caller auth stays separate from the LLM key. Because: conflating them is privilege confusion — a valid LLM key is not proof of a valid caller, and vice-versa. Reject on caller-auth first; only then consume the LLM key.

T-decision-5: Keep SDP's own-key env fallback during migration, remove it after. Decision: header wins; absent header falls back to ALTAFORGE_LLM_API_KEY until all consumers forward a key, then the fallback is deleted and a missing header 400s. Because: lets consumers migrate one at a time without a flag-day; the final removal is what actually enforces BYOK.

T-decision-6: Redact X-AltaForge-LLM-Key everywhere, and never persist it. Decision: add the header to the deny/redact list for access logs, the httpx/logging instrumentors, and OTel/Arize span attributes; never write it to a DB row. Because: headers routinely leak into traces and error dumps; an unredacted key in Arize is a real disclosure. "Never persist" keeps the data-model surface at zero ([T-entity-N]).

T-decision-7: Do NOT require a Next conversion for this design. Decision: the BFF can be the consumer's existing FastAPI backend; Next is optional. Because: CICI already has a server tier (canada_ici_client, FastAPI) that can hold the key and forward it — the browser-exposure problem is solved by routing LLM-triggering calls through it. Converting Vite→Next is a separate strategic bet (auth posture / SSR / org parity), not a prerequisite here. If a consumer is on Next, its server route is the BFF instead — same contract.

Phase Decomposition `[T-phase-N]`¶

`[T-phase-1]` SDP caller-key seam + redaction¶

Issues: 1. Extract X-AltaForge-LLM-Key into a request-scoped dependency in SDP's API layer. 2. Make llm.step_client(step) accept a per-request api_key and build/select the client by (auth_mode, api_key) rather than reusing the process-cached env-key client ([T-q-1]). 3. Add the header to redaction lists (logging, httpx/OTel instrumentors, Arize span attrs). 4. Keep env-key fallback ([T-decision-5]).

Acceptance:

T-test-1.1: Given a request with X-AltaForge-LLM-Key: K, when SDP makes its LLM call, then the call authenticates with K (not SDP's env key). Verified via a stubbed LiteLLM asserting the received key.

T-test-1.2: Given a request with header K, when the run completes, then K appears in no log line, OTel span, Arize trace, or DB row.

T-test-1.3: Given two concurrent requests with different keys K1,K2, when both run, then neither leaks into the other's LLM call (no cross-request key bleed from client caching). Guards [T-q-1].

`[T-phase-2]` CICI consumer contract + BFF forwarding¶

Issues: 1. Seed ici--llm-api-key (LiteLLM virtual key) into icistgcanctrlkv01 (Dev Manager / DevOps). 2. Add ALTAFORGE_LLM_API_KEY = { kv = "ici--llm-api-key" } to CICI backend/env.secrets.toml; correct the "no secrets" comment. 3. Route LLM-triggering calls through CICI's backend (BFF); attach the header server-side from env.

Acceptance:

T-test-2.1: Given make env-backend on CICI, when it runs, then ALTAFORGE_LLM_API_KEY is hydrated into .env.local from the ICI vault (needs kv-secrets-readers-ici).

T-test-2.2: Given a browser-initiated action that triggers an LLM call, when the network is inspected, then the key is absent from all browser↔BFF traffic and present only on the BFF→SDP hop.

`[T-phase-3]` Roll out remaining consumers + retire SDP fallback¶

Issues: 1. Repeat phase-2 for each other SDP consumer. 2. Remove SDP's ALTAFORGE_LLM_API_KEY env fallback; missing header now 400s on LLM routes. 3. Remove SDP's own LLM key from its vault (no longer used for consumer traffic).

Acceptance:

T-test-3.1: Given the fallback is removed, when an LLM-backed SDP endpoint is called without the header, then it returns 400 missing X-AltaForge-LLM-Key and makes no LLM call.

T-test-3.2: Given all consumers migrated, when LLM spend is reviewed in the gateway, then usage is attributable per consumer virtual key.

Deployment `[T-deploy]`¶

Env vars (consumer): ALTAFORGE_LLM_API_KEY (gateway key, server-only, via envsync). Must not be exposed to the browser (no NEXT_PUBLIC_ / VITE_ prefix).
Env vars (SDP): ALTAFORGE_LLM_API_KEY retained only through phases 1–2 as fallback; removed in phase 3.
Vault seeding: ici--llm-api-key in icistgcanctrlkv01 (and equivalents per consumer). Access via kv-secrets-readers-<org>.
Redaction config: ship the header deny-list with phase 1 — before any real key is ever sent.
Rotation: rotate the virtual key in the gateway → reseed the consumer vault → make env-backend. SDP needs no redeploy (it uses whatever the caller sends).

Open Questions `[T-q-N]`¶

T-q-1: SDP's llm.get_llm_client(auth_mode) caches clients per auth mode (per current docs). BYOK requires the key to vary per request — so the cache key must include the api_key, or the client must be constructed per-request when a caller key is present. Need to confirm the current caching shape and that no key bleeds across requests (covered by [T-test-1.3]). Highest-risk item — verify before phase 1 build.

T-q-2: Is the request-letter / any current CICI feature actually LLM-backed, or docx-templated? Decides whether CICI needs the key now or only when LLM features land. (Requires reading the can-ici-backend branch of SDP.)

T-q-3: Rejected-key UX: how should a 429/402 from the gateway surface to the end user vs. an SDP-internal error? Proposed 502 llm_key_rejected — confirm with product.

T-q-4: Future option (not in scope): per-request short-lived virtual-key minting instead of forwarding a long-lived virtual key — only if a "no long-lived key in transit" requirement appears.

Sign-off¶

Role	Name	Date
Author	Mark Ly
Engineering reviewer
SDP owner
Approver

Reviewer's checklist¶

[ ] TL;DR conveys the engineering shape in ≤6 bullets
[ ] Every design decision has a because
[ ] Every phase has ≥1 Given/When/Then acceptance test
[ ] The key never touches the browser in any documented path ([T-decision-1], [T-test-2.2])
[ ] The key is never persisted/logged/traced ([T-decision-6], [T-test-1.2])
[ ] Caller auth is separate from the LLM key ([T-auth], [T-decision-4])
[ ] Client-cache/key-bleed risk is addressed before build ([T-q-1], [T-test-1.3])
[ ] Consumer contract change is scoped (env.secrets.toml + vault seed + BFF forwarding)
[ ] No Next conversion is assumed as a prerequisite ([T-decision-7])
[ ] Sections that don't apply are marked n/a (data model) ```

TDD — BYOK LLM Keys for SDP Consumers¶

Change History¶

TL;DR [T-tldr]¶

System Overview [T-system]¶

Tech Stack [T-stack]¶

Data Model [T-entity-N]¶

API Contracts [T-api-N]¶

[T-api-1] SDP LLM-backed endpoints — new optional request header¶

[T-api-2] Consumer BFF → SDP (server-side)¶

Component / Module Structure [T-files]¶

Auth Model [T-auth]¶

Key Design Decisions [T-decision-N]¶

Phase Decomposition [T-phase-N]¶

[T-phase-1] SDP caller-key seam + redaction¶

[T-phase-2] CICI consumer contract + BFF forwarding¶

[T-phase-3] Roll out remaining consumers + retire SDP fallback¶

Deployment [T-deploy]¶

Open Questions [T-q-N]¶

Sign-off¶

Reviewer's checklist¶

TL;DR `[T-tldr]`¶

System Overview `[T-system]`¶

Tech Stack `[T-stack]`¶

Data Model `[T-entity-N]`¶

API Contracts `[T-api-N]`¶

`[T-api-1]` SDP LLM-backed endpoints — new optional request header¶

`[T-api-2]` Consumer BFF → SDP (server-side)¶

Component / Module Structure `[T-files]`¶

Auth Model `[T-auth]`¶

Key Design Decisions `[T-decision-N]`¶

Phase Decomposition `[T-phase-N]`¶

`[T-phase-1]` SDP caller-key seam + redaction¶

`[T-phase-2]` CICI consumer contract + BFF forwarding¶

`[T-phase-3]` Roll out remaining consumers + retire SDP fallback¶

Deployment `[T-deploy]`¶

Open Questions `[T-q-N]`¶