TDD - DocGen Framework v0 Core (R1)¶

status: draft
date: 2026-05-26
author: AltaML TechLead
prd: ./r1-prd.md
domain-model: ../../reference/domain-model.md
brief: ../../explanation/brief.md
epic: ../roadmap.md
parent-release: E-release-1
written-by: TechLead

Written by: TechLead. The Phase Decomposition section is structured as atomic issues that agents can execute in a single focused pass. Each issue names the file paths to create/edit, the function signatures or schema to implement, and the acceptance test.

How to read this¶

Audience	Read	Skip
Engineering reviewer	TL;DR + Tech Stack + Data Model + Key Decisions + Reviewer Checklist	Detailed acceptance tests on first pass
Architect agent (issue decomposer)	Everything	—
Dev agent (implementer)	Everything; especially Phase Decomposition for the issue you're executing	Phases not in flight
Sponsor signing off	TL;DR + Phase Decomposition + Reviewer Checklist	Implementation detail

Length budget: longer than the template's 5-15 pages — the phase decomposition is the bulk and is intentionally explicit so agents can execute without ambiguity.

Change History¶

Date	Author	Change
2026-05-23	AltaML TechLead	Initial draft for R1 framework v0 core. Decomposed into 10 phases × ~3 issues each — each issue is an agent-actionable story with files + signatures + acceptance test.
2026-05-26	AltaML TechLead	Rearchitect to gather → synthesize → render pipeline. Engine no longer renders per-section in isolation; the graph collects all `data_needs`, runs gather (deduped), runs synthesize, then fans out per-section render. R1 ships prompt-only as the content recipe; texts / snippets / risk_blocks defer to R2. Tools elevated to first-class data sources with 4 built-ins + credentials in env vars (`.env` / Azure KV). Phase count goes 10 → 8.
2026-05-29	AltaML TechLead	Source traceability in R1.6. `content_outputs` gains `sources_json` (text/JSON, NOT NULL, default `'[]'`) carrying a list of `Source` records — `gathered_data_id` + `selector` pin — recorded from the INPUT side at render time. Adds `[T-api-9]` (`engine.record_sources` — render-internal source-recording helper), `[T-decision-17]` (input-side recording rule + optional LLM-citation validation), and an optional `content_outputs.output_schema` column. Multi-tenant isolation: Source records reference only org-scoped `gathered_data.id`s already filtered by `OrganizationScopedSession` — no cross-org leak surface. Span-level pinning (per-sentence within prose) defers to R2 texts. See `docs/design/specs/2026-05-29-source-traceability.md`.
2026-05-31	AltaML TechLead	Doc-owned actions refactor (R1.7). `data_needs` lives on the document, not the section. `content_outputs` table dropped; each section owns a single `prompt` + `render_mode` + `output_template` + `output_type`. `synthesize` is now a single structured mega-call that builds a dynamic Pydantic schema from the document's `from_synth` sections (no more mode picker). `render` dispatches per-section by `render_mode` (`from_synth` pulls from the synth payload, `template` does a Jinja2 render against texts/snippets/synth/gathered, `custom_render` makes its own LLM call). Adds two org-scoped registries: `texts` (key/value) and `snippets` (reusable Jinja2 templates). Adds `document.tools_available` for mid-flight tool use during the mega-synth + custom_render. Migration `20260531_2000_doc_owned_actions.py` is reversible; the demo seed collapses to one doc-level `research` data_need shared across all three sections. See Post-refactor architecture section below.
2026-06-05	AltaML TechLead	Fields + cited sources + verify phase + agentic-retrieval hardening (R1.8). Adds the `fields` table — per-document parameters (`subject_name`, etc.), distinct from the org-scoped `texts`/`snippets` registries; declared on a template (defaults), copied to instances on `instantiate_template`, overridable per instance. Jinja substitution precedence is org texts → document fields → generate-time variables, applied identically in gather, synthesize, render, and `bridge.compute_output_keys_by_hash` (mismatch → gathered rows mis-map). Adds section `output_type='cited'` (synth returns `{content, sources[]}`), a `sections.sources_json` column, mechanical URL verification (fabricated URLs stripped in synthesize), and an optional verify phase (`engine.verify_cited_sources`, gated by `documents.verify_cited_sources`) that semantically fact-checks each cited claim and stamps each source `confirmed`/`unsupported`/`unchecked`. Hardens the tool-use loop with five bounds (rounds, total tool calls, wall-clock, result-size, dedup cache), env-tunable via `DOCGEN_TOOL_LOOP_` and clamped to absolute ceilings. Parallelizes gather (`DOCGEN_GATHER_MAX_PARALLEL`, incremental per-row commit) and verify (`DOCGEN_VERIFY_MAX_PARALLEL`). Adds async generate (`bridge.start_render` fire-and-return + SSE progress feed `docgen.progress` at `/progress/{id}`) and `bridge.refresh_gather`. Splits models: gather uses `GATHER_MODEL` (Sonnet), synthesize/render/verify use `DEFAULT_MODEL` (Opus). `claude_web_search` gains `fetch_pages=True` + `max_results` cap (`DOCGEN_SEARCH_MAX_RESULTS`, default 20). `sections.last_generation_at` is now stamped on render and surfaced as `lastGeneratedAt`. Migration `20260605_1200_document_fields.py`. Demo seed: three walkthrough templates (Phoenix, DealDesk, Background Check) seeding four instances — Background Check stamps two (Brian Jean + Elon Musk) via field overrides; the Reference Check fixture is retained but no longer seeded. The uvicorn bridge now binds `127.0.0.1:8769` (was `8765`) and `pnpm dev` orchestrates via `scripts/dev.mjs`. See Post-refactor architecture (R1.8)* section below.
2026-06-05	AltaML TechLead	Substitution-layer vocabulary rename (docs). `atoms` → `texts` and `molecules` → `snippets` throughout (entity names, tables `atoms`/`molecules` → `texts`/`snippets`, identifiers `atoms_dict` → `texts_dict`, etc.), mirroring the code rename in PR #79. `Field` and `risk_block` are unchanged.
2026-06-07	AltaML TechLead	Environment-aware LLM auth + Opus 4.8. `_build_llm_client` selects auth by `DOCGEN_ENV`: dev → `oauth_anthropic` (Claude Code token), staging/uat/prod → `anthropic` (direct `ANTHROPIC_API_KEY`). New `anthropic` direct-key mode added to `altaforge.llm`. LiteLLM gateway bypassed in the completions path while under repair; `ClaudeWebSearchTool` always uses the direct key (web_search is gated off OAuth). `DEFAULT_MODEL` bumped to `claude-opus-4-8`. New `[T-decision-23]`; `[D-10]` + `[T-decision-22]` updated.

Post-refactor architecture (R1.7) `[T-refactor-r17]`¶

Supersedes parts of the original gather/synthesize/render contract while keeping the three-phase pipeline shape. Read this section first when reasoning about the current code; the older sections below remain accurate where they aren't contradicted here.

Data model deltas¶

Table	Change
`documents`	adds `tools_available` (JSON, default `[]`); drops `synthesis_mode_override` + `structured_schema_path`
`sections`	adds `render_mode` (enum: `from_synth` \| `template` \| `custom_render`), `prompt`, `output_template`, `output_type`, `content` (the persisted rendered text)
`data_needs`	drops `section_id`; gains `document_id` FK — data_needs are doc-owned
`content_outputs`	dropped entirely; each section's "what to render" lives on the section row
`texts` (new)	org-scoped `(key, value, value_type)` registry, unique on `(organization_id, key)`
`snippets` (new)	org-scoped `(key, template_body)` registry of reusable Jinja2 templates, unique on `(organization_id, key)`

The SynthesisMode enum is gone — the synthesize phase is unconditionally structured.

Pipeline shape¶

gather                                       synthesize                                       render
─────────                                    ──────────                                       ──────
iterate document.data_needs[]                build a dynamic Pydantic schema                  per section in position order:
  resolve args (Jinja, sandboxed)              one field per from_synth section,              dispatch on render_mode:
  hash → dedup + cache-hit                     keyed by f's_{section.id.hex}'                  - from_synth   → synth.payload_json[key]
  invoke (tool | LLM via altaforge.llm)        type from section.output_type                   - template     → Jinja2 render of
  persist gathered_data row                    description from section.prompt                                  output_template
returns {output_key: result_json}            ONE altaforge.llm.completion call                                  ctx = texts/snippets/
                                             with response_format=DynamicSchema                                 synth/gathered/document
                                             and tools=document.tools_available                - custom_render → engine_complete with
                                             returns SynthesizedContext row                                     section.prompt (Jinja-resolved),
                                                                                                                 tools_available inherited from
                                                                                                                 document
                                                                                              writes section.content directly

Render mode contract¶

`render_mode`	Uses LLM?	Required section fields	Notes
`from_synth` (default)	no	`prompt` (becomes the dynamic field description)	The mega-synth call already filled this section's content. Render just coerces and persists.
`template`	no	`output_template`	Pure Jinja2; references texts / snippets / synth / gathered / document.
`custom_render`	yes	`prompt` (Jinja-resolved before sending)	Has access to `document.tools_available`.

Texts + snippets¶

Texts are org-scoped key/value pairs the substitution layer can interpolate by key alone. Snippets are reusable Jinja2 templates that reference texts; the render layer renders each snippet with the texts namespace first, then exposes the flat {key: rendered} map to section templates as snippets.*. This is the deterministic substitution layer (no LLM); the LLM-driven path is the mega-synth.

All model calls go through altaforge.llm¶

The mega-synth, the per-section custom_render, the gather-phase kind='prompt' data_needs — every model call routes through altaforge.llm.LLMClient.completion(**kwargs) via the engine's engine_complete helper. Web-search-grounded calls go through altaforge.llm.anthropic_web_search(...) (direct Anthropic SDK; sidesteps the LiteLLM gateway's web_search streaming/citation bugs) for the ClaudeWebSearchTool path, and altaforge.llm.web_search(..., mode=OPENAI_*) for the Azure-OpenAI Responses-API path used by OpenAIWebSearchTool. Direct litellm / anthropic / httpx imports are still forbidden anywhere under docgen/src.

Post-refactor architecture (R1.8) `[T-refactor-r18]`¶

Builds on R1.7. Adds per-document parameters, cited sources + an optional semantic verify phase, agentic-retrieval bounds, gather/verify parallelism, async generate with progress streaming, and a per-phase model split. Where this section contradicts older ones, this wins.

Data model deltas¶

Table	Change
`fields` (new)	per-document parameter registry — `(id, organization_id, document_id, key, value, value_type, created_at, updated_at)`, unique on `(document_id, key)`. Distinct from org-scoped `texts`/`snippets`: a field is the blank you fill in for one document (`subject_name`). Declared on a template → copied to each instance on `instantiate_template`; also settable directly on an instance.
`documents`	adds `verify_cited_sources` (bool, default `false`) — opt-in gate for the verify phase.
`sections`	adds `sources_json` (JSON, default `'[]'`) — per-section citations for `output_type='cited'`; `last_generation_at` is now stamped by the render phase (surfaced to the UI as `lastGeneratedAt`).

Field is registered in the org-scoping model list (data/session.py) and carries organization_id, so cross-org reads are impossible by construction ([T-decision-2]).

Substitution scope precedence `[T-decision-18]`¶

Jinja resolution layers, lowest-to-highest precedence: org texts (base) → document fields → generate-time variables. The same merged namespace ({**texts_dict(session), **fields_dict(session, document_id), **variables}) is built in gather, synthesize, render, and bridge.compute_output_keys_by_hash. All four must agree — if the hash that maps a gathered row to a data_need is computed under a different scope than the gather that produced it, rows mis-map and the UI shows "not yet gathered" despite a successful run. Texts/snippets stay for genuine org reuse (firm boilerplate, severity_rubric); per-document subject values are fields, never texts.

Cited sources + verify phase `[T-decision-19]`¶

A section with output_type='cited' makes the mega-synth return {content, sources: [{label, url, verified, support, support_note}]} for that field, persisted to sections.sources_json. Two layers guard citations:

Mechanical URL verification (always, in synthesize). URLs the model emits that it never actually retrieved are stripped — verified reflects whether the URL appears in the gathered page set.
Semantic verify phase (optional, gated by documents.verify_cited_sources). engine.verify_cited_sources(document, synthesized_context, *, organization) runs between synthesize and render. Per cited section it fact-checks the claim against the full gathered page text (not snippets) and stamps each source's support: confirmed / unsupported / unchecked. It judges the claim holistically rather than by brittle inline-[n] index matching, and falls back to the search snippet when a fetched page is a dead shell (< _MIN_REAL_PAGE_CHARS). Parallelized per section (DOCGEN_VERIFY_MAX_PARALLEL); emits per-section progress events.

Inline [n] markers in the prose link to the matching sources_json entry in the UI; verify strips them before judging so the model reasons about the claim, not the marker.

Agentic-retrieval bounds `[T-decision-20]`¶

The tool-use loop (_llm_io.run_tool_use_loop) can never run unbounded. Five independent bounds, each an absolute ceiling that env config can only lower:

Bound	Env override	Guards against
max rounds	`DOCGEN_TOOL_LOOP_MAX_ROUNDS`	infinite back-and-forth
max total tool calls	`DOCGEN_TOOL_LOOP_MAX_CALLS`	fan-out within a round
wall-clock deadline	`DOCGEN_TOOL_LOOP_MAX_SECONDS`	a single slow tool stalling the run
per-result size cap	`DOCGEN_TOOL_LOOP_MAX_RESULT_CHARS`	a giant payload blowing the context window
dedup cache	—	the model re-issuing an identical call

Each _configured_limit reads the env var and clamps it to the absolute ceiling (env can tighten, never loosen). On any bound hit the loop finalizes gracefully — tools are stripped from the next call so the model must answer from what it has, rather than erroring.

Gather/verify parallelism + async generate `[T-decision-21]`¶

Gather dispatches data_needs through a ThreadPoolExecutor (DOCGEN_GATHER_MAX_PARALLEL, default 8, ceiling 16) with as_completed + incremental per-row commit, so a slow source never blocks rows that already finished. Verify is parallelized the same way (DOCGEN_VERIFY_MAX_PARALLEL).

Generate no longer couples to the HTTP request lifetime: bridge.start_render fires the pipeline and returns immediately; the UI subscribes to a Server-Sent-Events feed at /progress/{document_id} backed by an in-process docgen.progress store that the pipeline writes phase/section events to. bridge.refresh_gather bypasses the gather cache (delete rows + re-gather + re-synth). This is the same architecture R6's queue-backed SaaS API scales up — a deploy-target change, not an architecture change (consistent with [T-decision-15]).

Per-phase model split `[T-decision-22]`¶

_llm_io defines GATHER_MODEL (Claude Sonnet 4.6 — lightweight data collection) and DEFAULT_MODEL (Claude Opus 4.8 — reasoning-heavy synthesize / render / verify). The synthesized-context row records DEFAULT_MODEL (not organization.llm_model) so the inspector reflects the model that actually produced the content. claude_web_search runs with fetch_pages=True and caps results at DOCGEN_SEARCH_MAX_RESULTS (default 20, clamped ≤ 30); synthesize gets compacted snippets (_compact_for_prompt — drops page bodies, caps snippet length) while verify gets the full page text. (Opus was pinned to 4.6 only while the LiteLLM gateway — which didn't serve 4.8 — was the path; direct/OAuth both serve 4.8.)

Environment-aware LLM auth `[T-decision-23]`¶

_build_llm_client selects the auth mode by DOCGEN_ENV (default dev), so credentials match the deployment without per-call thought:

`DOCGEN_ENV`	Completions auth	Why
`dev`	`oauth_anthropic` (Claude Code CLI token)	developers are signed into Claude Code; no API key to manage. Falls back to `ANTHROPIC_API_KEY` if not signed in.
`staging` / `uat` / `prod`	`anthropic` — direct `ANTHROPIC_API_KEY`	servers have no CLI login; pay-per-use key

DOCGEN_FORCE_FAKE_LLM=1 short-circuits to the deterministic fake (tests only). The LiteLLM gateway is bypassed in the completions path while it's under repair (re-enable by restoring the gateway branch / ALTAFORGE_LLM_AUTH_MODE=litellm). The new anthropic mode lives in altaforge.llm (the env→mode policy is docgen's; the mechanism is the lib — modes are also directly selectable via ALTAFORGE_LLM_AUTH_MODE). The ClaudeWebSearchTool always uses the direct ANTHROPIC_API_KEY in every env — Anthropic feature-gates server-side web_search off OAuth tokens. Domain-model auth-mode table: [D-10].

TL;DR `[T-tldr]`¶

One Python package: docgen. Multi-tenant from day one. SQLite for local dev; same schema/code works against Postgres in R2+.
Engine is a LangGraph StateGraph built via altaforge.core.runtime.AltaForgeGraph — AltaForge's standard base, which auto-wires OpenTelemetry + OpenInference instrumentation pointing at Arize AX / Phoenix. Tracing is "for free": every node + LLM call becomes a span.
Pipeline shape: collect_data_needs → gather → synthesize → fan-out per-section render → persist. The engine pulls all data once per document (deduped), synthesizes a shared context, then renders sections in parallel against that context. Engine functions stay pure + unit-testable; graph nodes are thin wrappers that read/write shared state.
Sections own their actions. data_needs (gather) + content_outputs (output) are section-owned, prompt text inline, no shared prompts table ([T-decision-3] / [T-decision-4]). R1 output kind is 'prompt' / 'tool'; texts / snippets / risk_blocks land in R2 as added content_outputs.kind values. Document templates (is_template + snapshot instantiate_template) per [T-decision-16].
Tools are first-class data sources. DataTool ABC + built-ins (AddTool, AppendTool, WebSearchTool, PageScrapeTool, WebPageTool, FileIngestTool, MCPTool) + ClaudeWebSearchTool + OpenAIWebSearchTool (server-side web-search variants; both accept fetch_pages= to attach page bodies to each citation). A test-only StubTool lives under tests/ and is never exported from the production package. Credentials come from env vars; framework never persists secrets.
LLM access via altaforge.llm (vendored as submodule from altaforge-libraries). Auth is environment-aware (DOCGEN_ENV): dev → Claude Code OAuth token; staging/uat/prod → direct ANTHROPIC_API_KEY (auth_mode='anthropic'). LiteLLM gateway bypassed while under repair. See [T-decision-23].
8 phases, ~25-30 atomic issues, each shipping a vertical slice (Python + UI + demo + tests). See r1-stories.md.
Output proof: integration tests write real .md documents to test_outputs/ so reviewers can eyeball.

System Overview `[T-system]`¶

   Solution code (e.g., Odyssey, JBRK, VS, Demo)
   ┌───────────────────────────────────────────────────┐
   │  Python: imports docgen + altaforge.core + llm    │
   │  - registers solution-specific DataTool subclasses│
   │  - defines prompts + section data_needs           │
   │  - sets tool credentials from env vars            │
   │  - calls engine.generate_document(...)            │
   └─────────────────┬─────────────────────────────────┘
                     │
   ┌─────────────────▼─────────────────────────────────┐
   │  docgen (in-process Python library)               │
   │  - engine.generate_document(org_slug, spec)       │
   │       └─ builds an AltaForgeGraph(StateGraph)     │
   │          and invokes it                            │
   │                                                    │
   │   ┌──────────────────────────────────────────┐    │
   │   │  AltaForgeGraph (from altaforge.core)    │    │
   │   │   nodes:                                  │    │
   │   │     • collect_data_needs                  │    │
   │   │     • gather_node (parallel tool +       │    │
   │   │         prompt calls; persists           │    │
   │   │         gathered_data with dedup)        │    │
   │   │     • synthesize_node (one LLM call,     │    │
   │   │         raw_text/structured/none modes,  │    │
   │   │         persists synthesized_context)    │    │
   │   │     • render_node (per-section   │    │
   │   │         fan-out; prompt over context;    │    │
   │   │         render-time tool calls add       │    │
   │   │         render_time_tool_call=true span) │    │
   │   │     • persist_section_node                │    │
   │   │   each node = an OpenTelemetry span      │    │
   │   │   auto-exported to Arize AX / Phoenix    │    │
   │   └──────────────────────────────────────────┘    │
   │                                                    │
   │  - OrganizationScopedSession (data layer)         │
   │  - DataTool registry (Python objects, not DB)     │
   └────┬───────────────────────────────┬──────────────┘
        │                               │
        ▼                               ▼
   ┌──────────┐                  ┌──────────────────┐
   │ SQLite   │                  │ altaforge.llm    │
   │ (dev)    │                  │ (sibling lib)    │
   │ /        │                  │  - OAuth (dev,   │
   │ Postgres │                  │    via local CLI)│
   │ (R2+)    │                  │  - litellm mode  │
   │          │                  │    (CI)          │
   └──────────┘                  └────────┬─────────┘
                                          │
                                          ▼
                                ┌────────────────────┐
                                │ Provider APIs OR   │
                                │ AltaML LiteLLM     │
                                │ gateway → Claude / │
                                │ GPT / Gemini       │
                                └────────────────────┘

R1 is the entire docgen library. altaforge.core (runtime base + Arize wiring), altaforge.llm (LLM client), altaforge.logging, and providers are external.

External tool endpoints (HTTP, MCP, etc.) sit beside the LLM providers and are reached from gather_node (and occasionally render_node when prompts opt into render-time tool calls — flagged in Arize).

Tech Stack `[T-stack]`¶

Layer	Choice	Rationale (only if non-default)
Language	Python 3.11	(default)
Orchestration framework	LangGraph via `altaforge.core.runtime.AltaForgeGraph`	AltaForge-standard base — auto-wires OTel + OpenInference for Arize AX / Phoenix tracing on construction. See `[T-decision-1]`.
LLM access	`altaforge.llm.client(auth_mode=...).completion(...)` from `altaforge-libraries/altaforge` (vendored submodule; consumed as `altaforge[core,llm,logging]` extras off the unified v0.2 package)	4 auth modes (3 OAuth via vendor CLI tokens + 1 static-key for the gateway); centralizes LLM access org-wide. Web-search-grounded calls use `altaforge.llm.anthropic_web_search` (direct Anthropic SDK) or `altaforge.llm.web_search(mode=OPENAI_*)` (Azure OpenAI Responses API).
Tracing / observability	OpenTelemetry + OpenInference's `LangChainInstrumentor`, exported via OTLP-HTTP to Arize AX (or Phoenix self-hosted)	Wired automatically by `AltaForgeGraph`; endpoint resolved from `organization.observability_url`
Logging	`altaforge.logging` (Python stdlib `logging` + Loki forwarder via `AF_LOKI_URL`)	Standard across AltaForge solutions
Default model	Per-phase split (R1.8): gather → Claude Sonnet (`GATHER_MODEL`); synthesize / render / verify → Claude Opus (`DEFAULT_MODEL`)	Sonnet is enough for data collection; Opus does the reasoning-heavy passes. Per `[T-decision-22]`. Per-organization `organizations.llm_model` remains the fallback.
ORM / data layer	SQLAlchemy 2.x + Alembic migrations	(default)
DB (R1)	SQLite (single file, `./dev.db`) — multi-tenant via `organization_id`	Fast local dev; same models work against Postgres in R2+
Templating	Jinja2	Prompt body variables + tool args interpolation
Multi-tenant pattern	`organization_id` discriminator + SQLAlchemy `with_loader_criteria` at session creation	Per `[T-decision-2]`
Tool credentials	Env vars; `.env.example` shipped in repo; Azure Key Vault for staging+	12-factor pattern; no secrets in DB. See `[T-decision-9]`.
Built-in tools	`AddTool`, `AppendTool` (deterministic no-credential demos), `WebSearchTool`, `ClaudeWebSearchTool`, `OpenAIWebSearchTool` (both support `fetch_pages=` to attach `WebPage` bodies to citations), `PageScrapeTool`, `WebPageTool` (single-URL fetcher via `altaforge.llm.web_page`), `FileIngestTool`, `MCPTool` (+ `StubTool` test double under `tests/`, not exported)	Match brief's R1 promise; solution-specific tools subclass `DataTool`
Synthesis output modes	`none` / `raw_text` / `structured` (Pydantic). `templated` defers to R2.	Per `[T-decision-10]`
Library packaging	`uv` for development; `uv_build` build backend	matches sibling AltaForge libs
Test runner	pytest + pytest-asyncio	(default)
LLM-backed tests	Real LLM calls — via the dev's local vendor CLI token in dev, via the LiteLLM gateway in CI	Per memory rule: tests for agent behavior use real LLM, not mocks
Lint / format	ruff (lint + format) + mypy strict	matches sibling AltaForge libs
Pre-commit	pre-commit hooks: ruff, mypy, trailing-whitespace, end-of-file-fixer	(default)
CI	Out of scope for this release — owned by the AltaForge CI team; stories never include `.github/workflows/*` files	The CI team adds workflows on top of the conventions captured in this TDD
Markdown rendering	Jinja2 templates that emit Markdown directly	No external renderer needed; .md is the R1 output format
Output dir	Repo-root `/output/` is the canonical headless-`generate_document` output (rendered `.md` / `.docx`; gitignored). `docgen/test_outputs/` holds .md artifacts integration tests write.	Eyeball-able artifacts; reviewers open in any text editor
UX library (`docgen-ui/`)	React 18 + TypeScript + Vite (library mode, ESM + CJS) + `altaforge-ui` (Radix Themes) as peer	Built alongside Python framework per-phase; user-testable via demo
Demo app (`demo/`)	Next.js 14 (App Router) + `docgen-ui` + `altaforge-ui` via pnpm workspace	Consumes Python `docgen` via long-lived HTTP service (`docgen.server`, uvicorn @ `127.0.0.1:8769`). See [T-decision-15]
JS workspace	pnpm workspaces — members: `docgen-ui`, `demo`, `vendor/altaforge-ui`	Standard JS monorepo coordination
UI test runner	vitest + react-testing-library	(default for AltaML React work)

Data Model `[T-entity-N]`¶

All tables org-scoped. All FKs cascade on organization delete (defensive; organizations are not deleted in v0).

`[T-entity-1]` `organizations` ← references `[P-entity-1]`¶

Column	Type	Nullable	Default	Notes
id	text (uuid)	No	`gen_random_uuid()` (Postgres) / `uuid4()` (SQLite, app-generated)	PK
slug	text	No	—	Unique; URL-safe; e.g. `odyssey`, `jbrk`, `verticalscope`
display_name	text	No	—	Human-readable
llm_provider	text	No	`'claude'`	Routes through LiteLLM proxy or vendor CLI
llm_model	text	No	`'claude-sonnet-4-5'`	Specific model id
default_synthesis_mode	text	No	`'none'`	`'none'` / `'raw_text'` / `'structured'`; documents inherit unless overridden
eval_backend	text	No	`'ax'`	`'ax'` or `'phoenix'`; used in R2+
eval_endpoint	text	Yes	—	Phoenix URL when `eval_backend='phoenix'`; used in R2+
observability_url	text	Yes	—	OTLP-HTTP endpoint for Arize AX / Phoenix tracing
config_json	text (JSON)	No	`'{}'`	Per-organization overrides (kept as text in SQLite for portability; native jsonb in Postgres R2+)
created_at	timestamp	No	`now()`

Indexes: unique on slug.

`[T-entity-2]` `documents` ← references `[P-entity-2]`¶

Column	Type	Nullable	Default	Notes
id	text (uuid)	No	uuid4	PK
organization_id	text (uuid)	No	—	FK → `organizations.id`
title	text	No	—
state	text	No	`'draft'`	`draft` / `under-review` / `finalized`
is_template	bool	No	`false`	`true` = reusable template; `false` = instance / ad-hoc. See `[T-decision-16]`.
created_from_template_id	text (uuid)	Yes	—	FK → `documents.id`; provenance of an instance; null for templates + ad-hoc
synthesis_mode_override	text	Yes	—	If set, overrides `organizations.default_synthesis_mode` for this doc
structured_schema_path	text	Yes	—	When `synthesis_mode='structured'`, the Python dotted path to the Pydantic class
created_at	timestamp	No	`now()`
updated_at	timestamp	No	`now()`	App-managed (SQLAlchemy `onupdate=`)

Indexes: (organization_id, created_at DESC) for organization document lists; (organization_id, is_template) to split templates from instances.

`[T-entity-3]` `sections` ← references `[P-entity-3]`¶

Column	Type	Nullable	Default	Notes
id	text (uuid)	No	uuid4	PK
organization_id	text (uuid)	No	—	FK → `organizations.id` (denormalized for index efficiency)
document_id	text (uuid)	No	—	FK → `documents.id`, cascade delete
ordinal	int	No	—	Immutable creation order within document. Set once at insert (`max(ordinal) + 1`); never modified — stable audit identifier.
position	int	No	—	Mutable display order within document. Sort key for `list_sections`. Rewritten as `1..N` by `reorder_sections`; new rows append at `max(position) + 1`. App-layer guarantees no duplicates after every reorder; no DB-level uniqueness so a single-row update during reorder doesn't collide.
title	text	No	—
status	text	No	`'empty'`	Aggregate of the section's `content_outputs` statuses: `empty` / `generating` / `generated` / `failed` / `edited`
last_generation_at	timestamp	Yes	—	Most recent generation across the section's outputs

A section has no content-recipe columns and no stored content — what renders it lives in its content_outputs ([T-entity-5]) and its gather inputs in data_needs ([T-entity-6]); the section's rendered content is the ordered concatenation of content_outputs[].result, composed on read. See [T-decision-3].

Indexes: (document_id, ordinal) unique; (organization_id, status) for queue scans. No unique constraint on (document_id, position) by design — see [T-decision-13].

`[T-entity-4]` `tools` (registration) ← references `[P-entity-4]`¶

Column	Type	Nullable	Default	Notes
id	text (uuid)	No	uuid4	PK
organization_id	text (uuid)	No	—	FK → `organizations.id`
name	text	No	—	Organization-scoped unique
category	text	No	—	Free-form tool-category descriptor: `'web_search'`, `'page_scrape'`, `'file_ingest'`, `'mcp'`, `'math'`, `'text'`, solution-specific. Distinct from `data_needs.kind` — that's the action discriminator (`'tool'` vs `'prompt'`); this is the tool-type label.
class_path	text	No	—	Python dotted path to the `DataTool` subclass that backs this registration
config_json	text (JSON)	No	`'{}'`	Non-secret per-org config (e.g., default num_results); credentials never live here
created_at	timestamp	No	`now()`

Indexes: unique on (organization_id, name).

`[T-entity-5]` `content_outputs` (output actions) ← references `[P-entity-5]`¶

Section-owned output actions. Replaces the old shared prompts table — prompt text now lives inline on the action that uses it, so a template deep-copy carries it verbatim and templates/instances share nothing ([T-decision-4]). Same action core as data_needs; differs only in the tail (result, status).

Column	Type	Nullable	Default	Notes
id	text (uuid)	No	uuid4	PK
organization_id	text (uuid)	No	—	FK → `organizations.id` (denormalized)
section_id	text (uuid)	No	—	FK → `sections.id`, cascade delete
ordinal	int	No	—	Order within the section's outputs
kind	text	No	`'prompt'`	`'prompt'` \| `'tool'` in R1; R2 adds `'text'`, `'risk_block'`
prompt_body	text	Yes	—	Inline prompt text (Jinja vars) when `kind='prompt'`
tools_available	text (JSON)	No	`'[]'`	Tool names the prompt may call mid-render, when `kind='prompt'`
tool_ref	text	Yes	—	Tool name when `kind='tool'`; the tool's result is the output (deterministic, no LLM)
args_json	text (JSON)	No	`'{}'`	Args for the tool call, when `kind='tool'`
result	text	Yes	—	Rendered output for this action
status	text	No	`'empty'`	`empty` / `generating` / `generated` / `failed` / `edited`
sources_json	text (JSON)	No	`'[]'`	List of `Source` records — `{gathered_data_id, source_category, source_ref, output_key, used_as, selector}` — populated by the render phase from the INPUT side (which `gathered_data` rows fed this output). FK-by-value to `gathered_data.id` (no DB FK since the parent is JSON; integrity engine-enforced via `OrganizationScopedSession`). Per `[T-decision-17]` + `docs/design/specs/2026-05-29-source-traceability.md`.
output_schema	text (JSON)	Yes	—	Optional JSON Schema / Pydantic dotted path for the LLM's structured output (when `kind='prompt'`). If it declares a `citation: {gathered_data_id, selector}` field, render validates the LLM-emitted citation against context (`CitationValidationError` on hallucinated/out-of-context citation) and narrows the corresponding Source record's `selector`.

Indexes: (section_id, ordinal) unique.

`[T-entity-6]` `data_needs` (gather actions) ← references `[P-entity-6]`¶

Section-owned gather actions. Same action core as content_outputs ([T-entity-5]); differs only in the tail (output_key).

Column	Type	Nullable	Default	Notes
id	text (uuid)	No	uuid4	PK
organization_id	text (uuid)	No	—	FK → `organizations.id` (denormalized; org-scoped per `[T-decision-2]`)
section_id	text (uuid)	No	—	FK → `sections.id`, cascade delete
ordinal	int	No	—	Position within the section's gather actions
kind	text	No	—	`'tool'` or `'prompt'`
prompt_body	text	Yes	—	Inline prompt text when `kind='prompt'` (the prompt may itself call tools); null for `kind='tool'`
tools_available	text (JSON)	No	`'[]'`	Tool names the prompt may call mid-invocation, when `kind='prompt'`
tool_ref	text	Yes	—	Tool name when `kind='tool'`; null for `kind='prompt'`
args_json	text (JSON)	No	`'{}'`	Args passed at invocation time; Jinja-interpolated against document-level `variables`
output_key	text	No	—	Key the result is bound to in the gather/synthesize context

Indexes: (section_id, ordinal) unique.

`[T-entity-7]` `gathered_data` ← references `[P-entity-7]`¶

Column	Type	Nullable	Default	Notes
id	text (uuid)	No	uuid4	PK
organization_id	text (uuid)	No	—	FK → `organizations.id` (denormalized)
document_id	text (uuid)	No	—	FK → `documents.id`, cascade delete
source_category	text	No	—	Tool kind (`'web_search'`, `'file_ingest'`, …) or `'prompt'`
source_ref	text	No	—	Tool name or prompt name
args_hash	text	No	—	Stable hash of (source_ref, args_json) used for dedup
args_json	text (JSON)	No	`'{}'`	Resolved args
result_json	text (JSON)	No	`'null'`	Serialized result
fetched_at	timestamp	No	`now()`

Indexes: (document_id, args_hash) unique (the dedup key); (organization_id, fetched_at DESC) for audit scans.

`[T-entity-8]` `synthesized_context` ← references `[P-entity-8]`¶

Column	Type	Nullable	Default	Notes
id	text (uuid)	No	uuid4	PK
organization_id	text (uuid)	No	—	FK → `organizations.id` (denormalized)
document_id	text (uuid)	No	—	FK → `documents.id`, cascade delete
mode	text	No	—	`'none'` / `'raw_text'` / `'structured'` (`'templated'` is R2)
payload_json	text (JSON)	No	`'null'`	Text blob for `'raw_text'`; serialized Pydantic for `'structured'`
schema_path	text	Yes	—	When `mode='structured'`, the Pydantic dotted path used
model	text	Yes	—	LLM model used to produce this synthesis
created_at	timestamp	No	`now()`

Indexes: (document_id, created_at DESC) — most recent synthesis per document.

API Contracts `[T-api-N]`¶

R1 framework is a Python library. The public surface is module-level functions in docgen.

`[T-api-1]` `engine.generate_document(organization_slug, document_spec, variables=None)`¶

def generate_document(
    organization_slug: str,
    document_spec: DocumentSpec,
    variables: dict | None = None,
) -> Document:
    """
    Generate a full document for an organization from a spec.

    DocumentSpec is a Pydantic model with title, optional synthesis_mode_override,
    optional structured_schema_path, and a sections list. Each SectionSpec has a
    title, data_needs (list of gather actions), and content_outputs (list of
    output actions — kind 'prompt'/'tool', prompt text inline).

    Pipeline (one AltaForgeGraph invocation):
      collect_data_needs → gather → synthesize → fan-out render (per output) → persist

    Implementation: constructs an ``AltaForgeGraph`` keyed by the organization
    (so tracing spans carry the org's observability_url), registers the pipeline
    nodes, and invokes the compiled graph with the document spec + variables as
    initial state. Each output action's rendered text is persisted to its
    ``content_outputs.result`` from a terminal persist node; the section's
    content is the ordered concatenation of those results, composed on read.

    Returns the persisted Document with sections + outputs populated.

    Raises: OrganizationNotFound, ToolNotFound,
            GatherError, SynthesisError, RenderError.
    """

`[T-api-2]` `engine.gather(document, variables, *, organization)`¶

def gather(
    document: Document,
    variables: dict,
    *,
    organization: Organization,
) -> dict[str, Any]:
    """
    Pure function: invoke every (deduped) data need across the document's
    sections, return a dict keyed by data_need.output_key with the results.

    A `kind='tool'` gather action is dispatched through the registered DataTool
    named by `tool_ref`. A `kind='prompt'` gather action renders its inline
    `prompt_body` via altaforge.llm (and may call its `tools_available`
    mid-call).

    Each invocation persists a `gathered_data` row keyed by
    (document_id, args_hash). Dedup: two gather actions with the same
    (kind, tool_ref-or-prompt_body, args_json) produce one DB row and one
    invocation.

    Returns a dict like {"client_facts": {...}, "press_releases": [...], ...}.
    """

`[T-api-3]` `engine.synthesize(document, gathered, *, organization)`¶

def synthesize(
    document: Document,
    gathered: dict[str, Any],
    *,
    organization: Organization,
) -> SynthesizedContext:
    """
    Pure function: produce a SynthesizedContext for the document.

    Mode resolves from document.synthesis_mode_override or
    organization.default_synthesis_mode:

      'none'        → returns SynthesizedContext(mode='none', payload=gathered)
      'raw_text'    → one altaforge.llm call summarizing gathered into text;
                      returns SynthesizedContext(mode='raw_text', payload=text)
      'structured'  → one altaforge.llm call producing the document's
                      structured_schema_path Pydantic model;
                      returns SynthesizedContext(mode='structured', payload=model_dict)

    Persists one row to `synthesized_context` and returns the model.
    """

`[T-api-4]` `engine.render_output(content_output, context, *, organization)`¶

def render_output(
    content_output: ContentOutput,
    context: dict,
    *,
    organization: Organization,
) -> str:
    """
    Render one output action, dispatching on content_output.kind.

    kind='prompt': interpolate context (synthesized context + raw inputs) into
      content_output.prompt_body via Jinja, construct an LLM client via
      ``altaforge.llm.client(auth_mode=organization.llm_auth_mode)`` and call
      ``.completion(model=organization.llm_model, messages=[...], temperature=0,
      tools=content_output.tools_available)``. Three-attempt retry on transient
      failure. If a tool is invoked, the engine emits OTel span attribute
      ``render_time_tool_call=true`` so reviewers can spot drift in Arize.
    kind='tool': call the DataTool named by content_output.tool_ref with
      content_output.args_json; its return is the output (deterministic, no LLM).

    There is no shared prompts table — the prompt text is inline on the action,
    so the same output renders identically whether it lives on a template or a
    deep-copied instance.

    The ``auth_mode`` resolves per dev/CI context (dev = `oauth_anthropic`;
    CI = `litellm`); consuming code stays the same.

    Source recording: before commit, ``render_output`` calls
    ``engine.record_sources(content_output, gathered_index, llm_response)``
    (`[T-api-9]`) to populate ``content_output.sources_json`` from the
    INPUT side — which gathered_data rows were Jinja-referenced in
    prompt_body / args_json, plus any render-time tool calls. If the
    output_schema declares a citation field, the LLM's emitted citation
    is validated against context and narrows the matching Source's
    selector. Per `[T-decision-17]`.

    Returns: the rendered text (stored to ``content_output.result``).
    """

`[T-api-5]` `engine.export_markdown(document_id)`¶

def export_markdown(document_id: str) -> str:
    """
    Export a document's full content as Markdown.

    Iterates sections in `position` order, emits H2 with section.title +
    the section's composed content (its content_outputs' `result`s in
    ordinal order), joined with double newlines.

    Returns: complete markdown document.
    """

`[T-api-6]` `register_tool(tool: DataTool)`¶

def register_tool(tool: DataTool) -> None:
    """
    Register a DataTool instance in the in-process Python registry.

    Solution code calls this at startup, typically with credentials
    injected from env vars:

        register_tool(WebSearchTool(api_key=os.environ['WEBSEARCH_API_KEY']))

    The tool is looked up by ``tool.name`` when a data_need invokes it.
    No persistence; credentials never reach the database.
    """

`[T-api-7]` `templates.instantiate_template(organization_slug, template_id, *, title)` ← references `[T-decision-16]`¶

def instantiate_template(
    organization_slug: str,
    template_id: str,
    *,
    title: str,
) -> Document:
    """
    Deep-copy a template into a fresh instance.

    Loads the template (raises ValidationError if not is_template). Creates a new
    Document(is_template=False, created_from_template_id=template_id, title=...,
    copying synthesis settings), then deep-copies the subtree:
      document → sections → (data_needs + content_outputs)
    New ids + section_id on every copied row; organization_id inherited; prompt
    text copied verbatim. Each copied content_output's `result` is cleared and
    `status='empty'` — the instance starts empty, to be generated with the
    client's variables. Tool code is NOT copied; `tool_ref` (by name) + args
    travel as-is.

    Runs under OrganizationScopedSession, so a template from another org is
    not found (cross-org instantiate impossible).

    Raises: OrganizationNotFound, NotFoundError (template absent in org),
            ValidationError (target is not a template).
    """

`[T-api-8]` `templates.list_templates(organization_slug)` / `documents.list_documents`¶

list_templates returns documents where is_template=true; list_documents returns instances + ad-hoc (is_template=false). Both org-scoped.

`[T-api-9]` `engine.record_sources(content_output, gathered_index, llm_response)` ← references `[T-decision-17]`¶

def record_sources(
    content_output: ContentOutput,
    gathered_index: dict[str, GatheredData],
    llm_response: LLMResponse | None,
) -> list[Source]:
    """
    Render-internal: produce the list[Source] that lands in
    content_output.sources_json. Called by render_output before commit.

    Deterministic INPUT-side recording:
      kind='tool':   one Source(used_as='tool_input') per gathered_data row
                     whose output_key is Jinja-referenced in args_json.
                     selector=None.
      kind='prompt': one Source(used_as='prompt_context') per gathered_data
                     row whose output_key is Jinja-referenced in prompt_body.
                     selector=None initially.
                     Each render-time tool call (per [T-decision-11]) adds
                     a Source(used_as='tool_result', selector=None).

    Optional LLM-citation narrowing (kind='prompt' only):
      If content_output.output_schema declares a 'citation' field, the
      llm_response.parsed.citation MUST name a gathered_data_id present
      in the recorded Source list AND the citation.selector MUST resolve
      against that row's result_json — else CitationValidationError.
      A valid citation narrows the matching Source's selector in place
      (e.g. {"result_index": 2} for web_search). It NEVER adds or
      removes entries.

    Returns: list[Source] ready to JSON-serialize onto
             content_output.sources_json.

    Raises: CitationValidationError (LLM emitted an out-of-context
            gathered_data_id or an unresolvable selector).
    """

Selector shape is per source_category (engine-validated, not DB-enforced): - web_search: {"result_index": int} — result_json[i] - file_ingest: {"page": int, "char_range": [int, int]} — result_json.pages[p].text[a:b] - mcp: {"json_path": "$.x.y"} — jsonpath_extract(result_json, ...) - None — no fragment pin; whole row is the recorded source

The bridge (docgen.bridge + demo/lib/docgen-bridge.ts) resolves each persisted Source against its gathered_data.result_json and emits the enriched wire shape ({url, title, snippet, page, ...}) so the UI shows ONE citation per output without a JOIN at read time. Per source-traceability spec [5].

Component / Module Structure `[T-files]`¶

altaforge-docgen/                # this repo (polyglot monorepo)

  docgen/                        # Python package (R1 framework engine)
    pyproject.toml
    .env.example                 # required env var names (tool creds, LLM access)
    src/docgen/
      __init__.py                # public exports: gather, synthesize, render, register_tool, ...
      engine/                    # pure pipeline phases — invoked as direct calls from bridge._run_render_pipeline (the LangGraph node-wrapper layer in [T-decision-1/8] was never built; the phases run as plain functions)
        __init__.py              # re-exports gather / synthesize / verify_cited_sources / render
        gather.py                # gather — parallel data_needs fan-out (ThreadPoolExecutor, DOCGEN_GATHER_MAX_PARALLEL), dedup, incremental persist of gathered_data
        synthesize.py            # synthesize — single structured mega-call → SynthesizedContext; mechanical URL verification for output_type='cited' fields
        verify.py                # verify_cited_sources — optional semantic citation fact-check, gated by document.verify_cited_sources (R1.8)
        render.py                # render — per-section dispatch by render_mode (from_synth / template / custom_render); stamps section.content + sources_json + last_generation_at
        _llm_io.py               # engine_complete + bounded run_tool_use_loop ([T-decision-20]); GATHER_MODEL / DEFAULT_MODEL split; response-format helpers
        _jinja.py                # sandboxed Jinja env + scope merge (texts → fields → variables) per [T-decision-18]
        _args_hash.py            # stable hash of (source_ref, args) for gather dedup
      sections.py                # section CRUD
      documents.py               # document CRUD
      templates.py               # instantiate_template (deep-copy incl. field rows), list_templates
      texts.py                   # org-scoped text (key/value) CRUD + texts_dict
      snippets.py               # org-scoped snippet (Jinja template) CRUD
      fields.py                  # per-document field CRUD + fields_dict (R1.8)
      data_needs.py              # document-owned data_need CRUD
      sources.py                 # Source record helpers (denormalize gathered_data → wire shape)
      synthesized_context.py     # SynthesizedContext CRUD
      specs.py                   # Pydantic wire summaries (DocumentSummary, SectionSummary, ...)
      enums.py                   # shared enums (render_mode, status, ...)
      errors.py                  # DocGenError / NotFoundError / ValidationError / ConflictError
      progress.py                # in-process progress event store — the SSE feed source for start_render ([T-decision-21])
      bridge.py                  # _dispatch_command + start_render / refresh_gather / .docx export — the API the HTTP service, CLI, and tests all call
      server.py                  # FastAPI app (uvicorn @ 127.0.0.1:8769): POST /bridge + GET /progress/{id} (SSE)
      testing.py                 # importable test helpers (StubTool double, seed helpers)
      tools/
        __init__.py              # register_tool, get_tool
        base.py                  # DataTool ABC
        add.py                   # AddTool (math; deterministic)
        append.py                # AppendTool (text; deterministic)
        web_search.py            # WebSearchTool (Tavily / Brave)
        claude_web_search.py     # ClaudeWebSearchTool (Anthropic server-side; direct SDK via altaforge.llm.anthropic_web_search; fetch_pages= + max_results cap)
        openai_web_search.py     # OpenAIWebSearchTool (Azure OpenAI Responses-API web_search; OPENAI_FAST/OPENAI_AGENTIC/OPENAI_DEEP_RESEARCH modes; supports fetch_pages=)
        page_scrape.py           # PageScrapeTool
        web_page.py              # WebPageTool (single-URL fetcher via altaforge.llm.web_page; title + readable text + truncation flag)
        file_ingest.py           # FileIngestTool
        mcp.py                   # MCPTool
      fixtures/                  # demo seed data (R1.8: 3 walkthrough templates)
        seed.py                  # seed_demo_data — Phoenix + DealDesk + Background Check (Brian Jean + Elon Musk via field overrides)
        background_check.py      # Background Check template builder (+ field defaults)
        phoenix.py               # Phoenix template builder
        dealdesk.py              # DealDesk template builder
        reference_check.py       # Reference Check builder — retained, no longer seeded
        demo.py                  # demo-env tool registration (fetch_pages, DOCGEN_SEARCH_MAX_RESULTS)
        schemas.py               # structured-output Pydantic schemas
      data/
        __init__.py              # re-exports models + Field
        models.py                # SQLAlchemy: Organization, Document, Section, DataNeed, GatheredData, SynthesizedContext, Tool, Text, Snippet, Field
        session.py               # OrganizationScopedSession factory (org-scoping via do_orm_execute event)
        migrations/              # Alembic (head: 20260605_1200_document_fields)
          env.py
          versions/<ts>_*.py
    tests/                       # ~55 files — representative shape, not exhaustive
      conftest.py                # shared fixtures (org, session, tool registry)
      factories.py               # test-data factories
      _stub_tool.py              # StubTool double — never exported from the package
      scenarios/                 # reusable multi-step pipeline scenarios (gather.py, ...)
      unit/                      # tools + engine internals
        test_<tool>.py           # one per built-in: add, append, web_search, claude_web_search,
                                 #   openai_web_search, page_scrape, web_page, file_ingest, mcp
        test_engine_llm_io.py    # bounded tool-use loop + GATHER/DEFAULT model split
        test_jinja_resolve.py    # texts → fields → variables scope merge ([T-decision-18])
        test_<template>_fixture.py  # phoenix / dealdesk / background_check / reference_check seeds
        test_no_direct_llm_imports.py  # guards [T-decision-7]
        test_public_api.py / test_sources_model.py / test_args_hash.py / ...
      integration/               # CRUD, pipeline phases, migrations, isolation
        test_<entity>_crud.py    # documents / sections / data_needs / texts / snippets / tools / templates
        test_gather_phase.py / test_synthesize_phase.py / test_render_phase.py /
          test_verify_phase.py   # phase contracts (@pytest.mark.llm)
        test_start_render.py     # async fire-and-return + progress feed ([T-decision-21])
        test_bridge*.py          # dispatch / render / synthesize / download / lifecycle / llm
        test_migrations*.py      # alembic head (20260605_1200) + per-phase schema
        test_multi_org_isolation.py / test_gather_multi_tenant.py  # org-scoping
        test_document_1_pipeline.py  # end-to-end shape
    test_outputs/                # .md artifacts written by integration tests (gitignored except .gitkeep)
      .gitkeep
    README.md                    # Onboarding runbook
    alembic.ini

  docgen-ui/                     # React UX library (Vite library build; 'use client' preserved per STYLEGUIDE §3)
    package.json / vite.config.ts / tsconfig.json
    dist/                        # built bundle the demo imports (gitignored)
    src/
      index.ts                   # barrel — re-exports every component + its Props type
      DocumentsGrid/             # <DocumentsGrid> — af-ui DataGrid wrapper (document list)
      DocumentView/              # <DocumentView> — section composition + synthesis preview
      SectionCard/               # <SectionCard> — section display + status
      DataNeedForm/  DataNeedRow/            # gather-action editors
      ContentOutputForm/  ContentOutputRow/  # output-action editors
      SynthesisConfigForm/  SynthesizedContextPanel/  # synthesis config + inspector
      SourcesPanel/  RenderedSectionContent/  # cited-source chips + rendered prose
      LLMModeIndicator/          # auth-mode / model badge
                                 # (each folder: <Name>.tsx + <Name>.module.css + index.ts)
    tests/                       # vitest + react-testing-library

  demo/                          # Next.js 14 App Router (user-testable surface)
    package.json / next.config.js / playwright.config.ts
    scripts/                     # dev.mjs (uvicorn + Next runner) + dev-stop.mjs (port/orphan sweep)
    app/
      layout.tsx
      (main)/                    # list surface (route group)
        page.tsx                 # documents list (DocumentsPanel → DocumentsGrid)
        templates/page.tsx       # templates list
        DocGenShell.tsx / DocGenSidebar.tsx / HomeFrame.tsx / *Dialog.tsx / DebugTools.tsx
      docs/[id]/                 # document drill-in
        page.tsx / layout.tsx
        GenerateBar.tsx / SectionEditor.tsx / SectionSources.tsx / GatheredDataPanel.tsx /
        SynthesisInspector.tsx / DocumentActionBar.tsx / PreviewPanel.tsx / RenderPanel.tsx / ...
      api/                       # route handlers — thin proxies to docgen.server over HTTP
        documents/[id]/{route, render/start, progress, gather, gather/refresh, gathered,
          synthesize, synthesis, download/{md,docx}, sections, data-needs}/route.ts
        sections/[id]/... , texts/ , snippets/ , templates/[id]/instantiate ,
          dev/{reseed,refresh-seed-args} , health , llm-status , prompt-assist
    lib/
      docgen-bridge.ts           # async HTTP client for docgen.server
      auto-synth.ts              # generate-progress event channel (window CustomEvent)
      format.ts                  # timestamp formatting
    e2e/                         # Playwright specs

  vendor/
    altaforge-libraries/         # submodule → altaml/altaforge-libraries (consumes altaforge.llm + altaforge.core sub-packages)
    altaforge-ui/                # submodule → altaml/altaforge-ui (R1 USES it as a pnpm workspace member; consumed by docgen-ui + demo)
    altaml-claude/               # submodule → altaml/altaml-claude (template/skill source; not runtime)

  output/                        # canonical headless generate_document output (.md / .docx); gitignored
  package.json                   # root: minimal; workspace declaration only
  pnpm-workspace.yaml            # JS workspaces: docgen-ui, demo, vendor/altaforge-ui
  .pre-commit-config.yaml        # ruff, mypy, eof-fixer, trailing-whitespace (Python hooks for now)
  .gitignore                     # includes .env, node_modules, __pycache__, /output/, dist/, *.db, etc.
  CLAUDE.md
  README.md                      # repo root quickstart

Layout decision: subdir-per-package at repo root (Python docgen/, JS docgen-ui/, JS demo/). pnpm workspaces coordinate the JS packages; Python lives as a single package (no root pyproject.toml needed).

Auth Model `[T-auth]`¶

n/a in R1 — the framework is a library, not a service. Authentication is the solution's concern; the engine sees only an organization_slug. Auth becomes a framework concern in R6 when the SaaS HTTP API ships.

Key Design Decisions `[T-decision-N]`¶

T-decision-1: Engine is a LangGraph StateGraph built via AltaForgeGraph Decision: generate_document constructs an altaforge.core.runtime.AltaForgeGraph (a thin subclass of langgraph.graph.StateGraph), registers pipeline nodes (collect_data_needs, gather, synthesize, render_section fan-out, persist), and invokes the compiled graph synchronously. AltaForgeGraph's constructor wires OpenTelemetry + OpenInference's LangChainInstrumentor so every node and downstream LLM call becomes a span exported to Arize AX (endpoint resolved from the org's observability_url). Because: every AltaForge solution will run a LangGraph workflow + want Arize tracing, so standardizing the base earns its keep on day one. The graph stays in-process and synchronous — no queue, no Celery, no Redis — so R1's latency target (≤ 3 min p95 for a 5-section document with gather + synth + render) is comfortably handled. Pairs with [T-decision-8] which keeps the engine pipeline functions pure (nodes wrap them, not the other way around).

T-decision-2: Multi-tenant isolation at the session-factory layer, not per-query Decision: OrganizationScopedSession(organization_id) returns a SQLAlchemy session with with_loader_criteria(Model, lambda cls: cls.organization_id == organization_id) applied to every loadable model. The plain Session constructor is not exported. Because: The single most damaging bug class is a query forgetting to filter by organization. Enforcing at the session layer means a call site that wants to do the wrong thing can't accidentally — they'd have to explicitly construct an unscoped session. Per [P-risk-1].

T-decision-24: The caller's org is resolved per request from the authenticated user, carried in a ContextVar — never a process global Decision: The bridge HTTP service (docgen.server) mounts altaforge.auth's AuthMiddleware (real plumbing; FakeAuthBackend in dev/demo, JwtAuthBackend/Azure AD in prod — same middleware), so every request carries request.state.user with its organizations. The /bridge handler reads the target org from the X-Org-Slug header (the demo's per-user cookie selection), authorizes it against user["organizations"] (404 — not 403 — on mismatch, to avoid org enumeration), and binds it for the request via bridge.request_org_context(org, user_orgs), which sets two ContextVars. _demo_session() reads _current_org.get() and feeds it to OrganizationScopedSession ([T-decision-2]). No HTTP (CLI / tests / per-tenant deploy) → falls back to the DOCGEN_ORG_SLUG env var. The in-app org switcher sets the user's docgen_org cookie (per-user), NOT shared server state. Because: an "active org" held in a module-level mutable global races across concurrent requests and leaks one tenant's data into another's response. A ContextVar is isolated per async task/thread, so concurrent multi-tenant requests can't see each other (tests/integration/test_request_org_context.py proves it under a thread pool). A CI guard (tests/unit/test_no_request_state_globals.py) fails on any global in the request layer so the shortcut can't reappear. The org-wide rule lives in CLAUDE-python.md → "Request-scoped state is NEVER a process-global." Demos fake the credential only; the per-request resolution path is identical to production.

T-decision-3: A section owns its outputs (content_outputs); no content-recipe column, no stored content Decision: Sections carry no content_recipe_* column and no current_content. What renders a section lives in its content_outputs rows (output actions); the section's rendered content is the ordered concatenation of content_outputs[].result, composed on read, and its status is the aggregate of its outputs' statuses. Each output is independently regenerable / editable / auditable. Because: sections need many outputs (e.g. a deterministic block + an LLM paragraph), and a template deep-copy must carry each output verbatim. A single recipe FK couldn't express "many outputs" and a stored current_content would duplicate (and drift from) the outputs' results. Per-output result+status gives granular regenerate/edit/audit. See [T-decision-4] for the action model and [T-decision-16] for templates.

T-decision-4: Actions are section-owned with inline prompt text; no shared prompts table; tools referenced by name Decision: data_needs (gather) and content_outputs (output) are section-owned action tables sharing an action core (kind, prompt_body, tools_available, tool_ref, args_json). Prompt text lives inline on the action (prompt_body); there is no shared prompts table. A kind='tool' action references a registered DataTool by name (tool_ref) — tool code is shared, never copied; its args travel. kind='prompt' carries the prompt inline and may call tools_available mid-call. Because: a document template is instantiated by deep-copying the section subtree ([T-decision-16]); that's only correct if the subtree owns everything it needs. A shared prompt table would mean an instance's copy referenced shared rows, so editing one instance's prompt would mutate the template + every sibling — fighting the snapshot principle. Inlining prompt text makes the copy total and isolation free. The one thing not copyable is tool code (env-var creds, Python class), so tools stay referenced by name. (Deterministic shared boilerplate — R2 texts/risk_blocks — is a separate, deliberately-shared catalog.)

T-decision-5: SQLite for R1 dev; same models work for Postgres in R2+ Decision: R1 ships SQLite-only Alembic migrations. SQLAlchemy models use dialect-portable types (text uuids, text-JSON instead of jsonb). Because: Fast local dev, no DB container needed. R2 adds Postgres-specific migrations (jsonb column types, native uuid, etc.) without changing the SQLAlchemy model API.

T-decision-6: Real LLM calls in tests, no mocked SDK Decision: Integration tests use altaforge.llm.client(auth_mode=...) and hit a real provider. In local dev that's whichever vendor CLI the test runner has signed in (Claude Code / Codex CLI / Gemini CLI); in CI it's litellm mode against the AltaForge LiteLLM gateway. Because: Per memory rule: tests for agent behavior use real LLM. Mocking the LLM client tests the mock, not the system. Test latency is real but acceptable (tests tagged @pytest.mark.llm; CI runs them; local dev runs on-demand).

T-decision-7: altaforge.llm is the LLM client; litellm is not imported in docgen code Decision: docgen/engine/render.py imports from altaforge.llm import client. The litellm library is a transitive dep through altaforge.llm only. Because: Auth-mode dispatch (OAuth-via-CLI-tokens vs static-key gateway) is altaforge.llm's concern, not docgen's. If docgen imported litellm directly, the auth dispatch would scatter across docgen code.

T-decision-8: Engine functions stay pure; graph nodes are thin wrappers Decision: gather, synthesize, render_output, and export_markdown are plain Python functions with explicit args + return types — no DocGenState parameter, no global LangGraph dependency. The graph nodes (engine/nodes.py) read fields out of DocGenState, call the matching pure function, and write the result back into DocGenState. Because: the pure functions stay trivially unit-testable (no LangGraph runtime to spin up), composable from non-graph contexts (REPL, CLI scripts), and the engine's actual logic lives in one obvious place per concern rather than entangled with state-machine plumbing. Nodes become ~5 lines each and add observability + state mutation, nothing else.

T-decision-9: Tool credentials live in env vars, never in the database Decision: DataTool subclasses accept credentials as constructor args; solution code injects them at register_tool(...) time from env vars. The tools table holds registration metadata (name, category, class_path, non-secret config) only. Repo ships docgen/.env.example listing every env var name the built-in tools expect. Staging+ deployments use Azure Key Vault → AppService env-var references. Because: 12-factor app pattern; secrets never persist in the framework's DB; portable across dev / staging / client-cloud sovereignty (same env-var names, different KV backends). Per-org encrypted credential storage can land in R3 if shared-infra multi-tenant needs it.

T-decision-10: Synthesis is configurable per document, with 3 modes shipping in R1 Decision: Synthesis mode resolves from document.synthesis_mode_override (if set) or organization.default_synthesis_mode. R1 supports 'none' (skip; sections render directly against gathered_data), 'raw_text' (one LLM call → text), and 'structured' (one LLM call → Pydantic model defined at document.structured_schema_path). R2 adds 'templated' (synthesis produced via texts / snippets). Because: solutions vary in how much cross-section understanding is needed. Odyssey wants typed facts (structured); a quick reformatter wants no synthesis at all (none); a narrative summary fits raw_text. Locking one mode forces every solution into it; allowing all three keeps the contract per-document.

T-decision-11: Render-time tool calls allowed but emit a flagging OTel span attribute Decision: A kind='prompt' output action can invoke tools mid-render (via its inline tools_available). When that happens, render_output adds the OTel span attribute render_time_tool_call=true and includes the tool name + args in the span. Arize traces surface these so reviewers can promote the call to a data_need if it gets expensive. Because: matches the flexibility we want — prompts shouldn't be hobbled. But unflagged render-time tool calls would silently bloat per-document cost and latency. The Arize flag makes the drift visible without blocking it.

T-decision-12: Gather phase dedupes invocations by (source_ref, args_hash) Decision: When multiple sections declare the same data_need shape (same tool/prompt + same resolved args), the gather node invokes it once and binds the result to every output_key that needs it. gathered_data stores one row per unique invocation per document, keyed by args_hash. force_refetch=true on generate_document bypasses cached rows. Because: common-case efficiency — Odyssey's 5 paragraphs may all want the same efs_lookup(deal_id) data, and gather should run it once. Cached rows also serve as the audit trail for what was fed into synthesis.

T-decision-13: Section ordering splits into ordinal (immutable creation order) + position (mutable display order) Decision: sections carries two int columns. ordinal is set once at insert and (document_id, ordinal) is unique — it's the stable audit identifier ("Section 1 was created first"). position is the display sort key — list_sections orders by it, and reorder_sections(document_id, ordered_ids) rewrites positions as 1..N atomically. No DB-level uniqueness on (document_id, position) — the app layer guarantees it, sidestepping the 2-phase swap that uniqueness would force on every reorder. Because: a single column can't safely be both audit identity and reorderable. Conflating them means every reorder touches the unique-constrained column with cascading updates, risking partial-state corruption mid-swap. Splitting keeps ordinal stable for audit/joins/external refs while position carries the cheap, frequently-mutated display intent. UI drag-drop ships against position once reorder_sections is wired.

T-decision-14: R1 engine is synchronous; async variants defer to R6 Decision: All R1 engine + CRUD functions are synchronous (Session, not AsyncSession). Async variants are not added until R6 introduces an async-native web consumer (e.g. websocket streaming or per-request scoped DI). Today's HTTP bridge ([T-decision-15]) wraps the synchronous engine in a thread-pooled FastAPI handler, which is fine for our request volume. Because: Adding AsyncSession variants now would mean maintaining two code paths for zero benefit (speculative complexity). FastAPI calls sync def handlers in a worker thread, so the synchronous engine works fine behind the HTTP service. When R6 introduces a real async-native consumer, async variants land against it. CRUD functions already take an injected session, so the shape supports a Depends(get_session) style without rework.

T-decision-15: The demo↔Python boundary is a long-lived FastAPI HTTP service. Subprocess-per-call is permanently forbidden. Decision: Python docgen runs as a long-lived FastAPI process (docgen.server) under uvicorn, bound to 127.0.0.1:8769. The Next demo's TypeScript client (demo/lib/docgen-bridge.ts) talks to it via fetch. Every bridge function is async and returns Promise<T>. Dev orchestration: pnpm dev runs scripts/dev.mjs, which frees the ports then launches uvicorn (pnpm dev:py) + next dev (pnpm dev:web) together (Ctrl+C kills both). The R1.8 async-generate + SSE progress feed ([T-decision-21]) rides the same service: bridge.start_render returns immediately and the UI subscribes to /progress/{document_id}.

Forbidden: spawning a Python process per call from Node (child_process.spawnSync('python', ['-m', 'docgen.bridge', ...])). That pattern paid 500ms of cold-Python startup on every page render and is the reason the early demo felt slow. It is not to be reintroduced even as a "quick workaround" — fix the HTTP path instead. The previous CLI surface (python -m docgen.bridge <cmd> <args>) is retained for tests + manual CLI use, but production code paths go through HTTP.

Wire shape: one endpoint, POST /bridge, body {cmd: string, args: string[]}. Identical to the CLI dispatcher's shape so test code can call _dispatch_command(cmd, args) directly without re-mocking HTTP. Errors map to HTTP status (NotFoundError → 404, ValidationError → 422, unknown cmd → 400, other → 500). No auth; bind-to-loopback is the security boundary in dev.

Why this is the right transport (not just "faster bridge"): it matches the production deployment shape — Python docgen ships as its own service, Next is a separate process, they talk over HTTP. R6's SaaS HTTP API is the same architecture scaled up (auth, multi-tenant, queue-backed long jobs). Going from R1's local FastAPI to R6's hosted FastAPI is a deploy-target change, not an architecture change.

Logging: library modules use getLogger(__name__). The FastAPI service wires altaforge.logging.setup_application_logging at startup so stderr + Loki get every request. (The legacy CLI bridge's stderr-only logging policy still applies when invoking python -m docgen.bridge for tests.)

T-decision-17: Source traceability is recorded from the INPUT side at render time; LLM citations validated, never trusted; output-level granularity (span-level defers to R2) Decision: The render phase records the sources each content_output consumed by introspecting what the engine put into context (which gathered_data rows were Jinja-referenced in prompt_body / args_json; which tools fired mid-render per [T-decision-11]) — not by parsing the LLM's prose for citations. Recording is mandatory for every render and produces a list[Source] persisted to content_outputs.sources_json ([T-entity-5]). Optional per-output: if the action's output_schema declares a citation field, the LLM emits a citation alongside content AND engine.record_sources ([T-api-9]) validates it against the recorded sources + checks selector resolves against the underlying gathered_data.result_json; an unmatched gathered_data_id or unresolvable selector raises CitationValidationError (typed; surfaces as status='failed'). A validated citation only ever narrows an existing Source's selector — it never adds or removes entries. R1 granularity is output-level (a content_output records the rows it consumed; the optional citation pins a fragment within one row); span-level pinning (per-sentence within prose) defers to R2 texts where the binding is structural. Because: LLMs hallucinate citations confidently in the right format. Trusting LLM-emitted citations means the audit chain rests on the very model the citation exists to verify. Input-side recording is deterministic — the engine knows what it passed in. The optional LLM-citation path is a narrowing signal (which of the inputs?), not a trust signal, and is always validated; an unvalidated citation never lands. The lawyer-approved-content principle from [D-11] survives only if every link in the chain is established by code, not by the model. Span-level granularity in free prose would require either trusting the LLM (rejected) or post-hoc embedding-similarity matching (rejected as out-of-scope manufacturing) — defer to R2 where texts make pinning structural. Pairs with [T-decision-11] (render-time tool calls become Source(used_as='tool_result') entries on the triggering output, so drift remains traceable).

T-decision-16: Document templates via is_template + snapshot deep-copy instantiate_template Decision: A template is a Document with is_template=true — same documents / sections / action tables as any document, generatable for preview. instantiate_template(template_id, title) produces an instance (is_template=false, created_from_template_id set) by deep-copying the subtree — document → sections → (data_needs + content_outputs), prompt text and all — clearing each output's result so the instance starts empty. No new tables. Because: "configure once, stamp many" is DocGen's core reuse promise. Snapshot (copy), not reference: editing a template never changes existing instances — audit-correct for legal / underwriting output, and the deep copy is total because actions own their text inline ([T-decision-4]). One documents table (not a separate document_templates) keeps sections + actions uniform so the copy is a within-table subtree walk; divergent template-only metadata, if it ever grows, goes in a 1:1 side-table without moving the section-bearing core.

Phase Decomposition¶

The story-level implementation plan — atomic [T-issue-N.M] items with files, signatures, and acceptance tests — lives in a separate stories doc:

See ./r1-stories.md for the full phase decomposition.

Each story in r1-stories.md cites the [T-decision-N] row above that it implements. Architecture (this doc) and execution (r1-stories.md) update independently — architecture changes trigger story updates; story refactors stay local to r1-stories.md.

Deployment `[T-deploy]`¶

n/a in R1 — the framework is a library; no service deployment. The AltaForge LiteLLM gateway (external dependency) is operated by the platform team. R2 adds the client-cloud deployment template.

Open Questions `[T-q-N]`¶

T-q-1: ~~Variable resolution at generation time — where do {{client_name}}, {{deal_facts}}, etc. come from?~~ Resolved 2026-05-26: data_needs produce them via gather + synthesize. The variables kwarg on generate_document remains as the source of top-level document inputs (e.g., a deal_id) that data needs interpolate into their args_json.
T-q-2: Should engine.generate_document re-render a section whose content_outputs already have results? Default: yes — calling generate_document always regenerates all outputs. Per-output edit-vs-regenerate semantics live in R2's edit_output / regenerate_output APIs.
T-q-3: Error envelopes — when an LLM call fails after retries, the section's status becomes 'failed' and the rest of the document continues. Confirm this matches [P-sys] reliability rule.
T-q-4: Concurrency — R1 is synchronous; can two generate_document calls run in parallel against the same organization? Default: yes (SQLAlchemy session per call; no shared mutable state). Document this is safe.
T-q-5: Gather-phase parallelism — should data needs within one document run in parallel? Default: sequential in R1; asyncio gather is a low-risk follow-up if measured latency hurts.
T-q-6: gathered_data cache lifetime — when does an args_hash row count as stale? Default: per-document scope; rows tied to document_id so regenerating a document with force_refetch=true clears them. No global cache.

Sign-off¶

Role	Name	Date
Author	AltaML TechLead	2026-05-26
Engineering reviewer
Approver	Mark Ly (CTO)

Reviewer's checklist¶

[ ] TL;DR fits in ≤8 bullets and conveys the engineering shape
[ ] Every PRD [P-feat-N] has a corresponding TDD home (data model, API, file, or acceptance test) — see cross-ref check below
[ ] Tech Stack table is complete; deviations from AltaML defaults have a because
[ ] Every key decision has a because
[ ] Every [T-phase-N] has 2-5 [T-issue-N.M] items
[ ] Every [T-issue-N.M] names files, signatures (or schema), and an acceptance test
[ ] Every [T-issue-N.M] is sized for one agent session (1-3 hours of focused work)
[ ] Walkthrough tests cover R1 PRD's three R1 shapes (cooking sanity, Odyssey, render-time-drift)
[ ] Texts / snippets / risk_blocks deferred to R2 with forward-compat note
[ ] Open Questions are truly unresolved

Cross-ref check (PRD [P-feat-N] → TDD home):

P-feat-1 (schema + multi-tenant) → T-entity-1/2/3, T-decision-2, T-phase-1.1
P-feat-2 (document CRUD + DocumentView + demo) → T-phase-1.2
P-feat-3 (section CRUD + SectionCard + data_needs slot) → T-entity-3/6, T-decision-3/4, T-phase-1.3
P-feat-4 (tools + gather phase) → T-entity-4/7, T-api-2/6, T-decision-9/12, T-phase-1.4
P-feat-5 (synthesize phase) → T-entity-8, T-api-3, T-decision-10, T-phase-1.5
P-feat-6 (render phase + Arize-flagging) → T-api-4, T-decision-11, T-phase-1.6
P-feat-13 (source traceability — output-level) → T-entity-5 (sources_json + output_schema cols), T-api-9, T-decision-17, T-phase-1.6; eval P-eval-6
P-feat-7 (generate_document + markdown export) → T-api-1/5, T-decision-1/8, T-phase-1.7
P-feat-8 (walkthroughs + runbook + .env.example) → T-phase-1.8
P-feat-9 / P-feat-10 (docgen-ui + demo) → distributed across phases 1.2–1.8

When approved, the architect agent's job is done. Dev agents pick up [T-issue-N.M] items in order (or in parallel within a phase) and execute. Each issue has its own atomic acceptance test.

TDD - DocGen Framework v0 Core (R1)¶

How to read this¶

Change History¶

Post-refactor architecture (R1.7) [T-refactor-r17]¶

Data model deltas¶

Pipeline shape¶

Render mode contract¶

Texts + snippets¶

All model calls go through altaforge.llm¶

Post-refactor architecture (R1.8) [T-refactor-r18]¶

Data model deltas¶

Substitution scope precedence [T-decision-18]¶

Cited sources + verify phase [T-decision-19]¶

Agentic-retrieval bounds [T-decision-20]¶

Gather/verify parallelism + async generate [T-decision-21]¶

Per-phase model split [T-decision-22]¶

Environment-aware LLM auth [T-decision-23]¶

TL;DR [T-tldr]¶

System Overview [T-system]¶

Tech Stack [T-stack]¶

Data Model [T-entity-N]¶

[T-entity-1] organizations ← references [P-entity-1]¶

[T-entity-2] documents ← references [P-entity-2]¶

[T-entity-3] sections ← references [P-entity-3]¶

[T-entity-4] tools (registration) ← references [P-entity-4]¶

[T-entity-5] content_outputs (output actions) ← references [P-entity-5]¶

[T-entity-6] data_needs (gather actions) ← references [P-entity-6]¶

[T-entity-7] gathered_data ← references [P-entity-7]¶

[T-entity-8] synthesized_context ← references [P-entity-8]¶

API Contracts [T-api-N]¶

[T-api-1] engine.generate_document(organization_slug, document_spec, variables=None)¶

[T-api-2] engine.gather(document, variables, *, organization)¶

[T-api-3] engine.synthesize(document, gathered, *, organization)¶

[T-api-4] engine.render_output(content_output, context, *, organization)¶

[T-api-5] engine.export_markdown(document_id)¶

[T-api-6] register_tool(tool: DataTool)¶

[T-api-7] templates.instantiate_template(organization_slug, template_id, *, title) ← references [T-decision-16]¶

[T-api-8] templates.list_templates(organization_slug) / documents.list_documents¶

[T-api-9] engine.record_sources(content_output, gathered_index, llm_response) ← references [T-decision-17]¶

Component / Module Structure [T-files]¶

Auth Model [T-auth]¶

Key Design Decisions [T-decision-N]¶

Phase Decomposition¶

Deployment [T-deploy]¶

Open Questions [T-q-N]¶

Sign-off¶

Reviewer's checklist¶

Post-refactor architecture (R1.7) `[T-refactor-r17]`¶

Post-refactor architecture (R1.8) `[T-refactor-r18]`¶

Substitution scope precedence `[T-decision-18]`¶

Cited sources + verify phase `[T-decision-19]`¶

Agentic-retrieval bounds `[T-decision-20]`¶

Gather/verify parallelism + async generate `[T-decision-21]`¶

Per-phase model split `[T-decision-22]`¶

Environment-aware LLM auth `[T-decision-23]`¶

TL;DR `[T-tldr]`¶

System Overview `[T-system]`¶

Tech Stack `[T-stack]`¶

Data Model `[T-entity-N]`¶

`[T-entity-1]` `organizations` ← references `[P-entity-1]`¶

`[T-entity-2]` `documents` ← references `[P-entity-2]`¶

`[T-entity-3]` `sections` ← references `[P-entity-3]`¶

`[T-entity-4]` `tools` (registration) ← references `[P-entity-4]`¶

`[T-entity-5]` `content_outputs` (output actions) ← references `[P-entity-5]`¶

`[T-entity-6]` `data_needs` (gather actions) ← references `[P-entity-6]`¶

`[T-entity-7]` `gathered_data` ← references `[P-entity-7]`¶

`[T-entity-8]` `synthesized_context` ← references `[P-entity-8]`¶

API Contracts `[T-api-N]`¶

`[T-api-1]` `engine.generate_document(organization_slug, document_spec, variables=None)`¶

`[T-api-2]` `engine.gather(document, variables, *, organization)`¶

`[T-api-3]` `engine.synthesize(document, gathered, *, organization)`¶

`[T-api-4]` `engine.render_output(content_output, context, *, organization)`¶

`[T-api-5]` `engine.export_markdown(document_id)`¶

`[T-api-6]` `register_tool(tool: DataTool)`¶

`[T-api-7]` `templates.instantiate_template(organization_slug, template_id, *, title)` ← references `[T-decision-16]`¶

`[T-api-8]` `templates.list_templates(organization_slug)` / `documents.list_documents`¶

`[T-api-9]` `engine.record_sources(content_output, gathered_index, llm_response)` ← references `[T-decision-17]`¶

Component / Module Structure `[T-files]`¶

Auth Model `[T-auth]`¶

Key Design Decisions `[T-decision-N]`¶

Deployment `[T-deploy]`¶

Open Questions `[T-q-N]`¶