Skip to content

Domain Model - DocGen

status: draft
date: 2026-05-26
author: Mark Ly
release: releases/r1-framework-v0-core
written-by: AIA + TechLead
purpose: Canonical vocabulary for DocGen. Cited by every PRD/TDD/stories doc in this repo and by every consuming solution repo.

R1 ships this doc as part of the foundational deliverable. All downstream releases reference it.

Change History

Date Author Change
2026-05-23 Mark Ly Initial draft — texts / prompts / snippets / tools as orthogonal primitives; sections filled by inline content unit or snippet + role_filter.
2026-05-26 Mark Ly Rearchitect to gather → synthesize → render pipeline. Sections now declare data_needs; the engine gathers data once per document, synthesizes a shared context, then fans out per-section render. R1 ships prompts as the sole render recipe; texts / snippets / risk_blocks defer to R2. Tools are elevated to first-class data sources with 4 built-ins (web_search, page_scrape, file_ingest, MCP client). Credentials are env-var driven (.env for dev, Azure Key Vault for staging+).
2026-05-26 Mark Ly Section-owned actions + document templates. Removed the shared prompts table — prompt text now lives inline on section-owned actions: data_needs (gather, many) + content_outputs (output, many). Sections drop content_recipe_* and current_content (content derived from content_outputs[].result, composed on read). Added documents.is_template + created_from_template_id + instantiate_template (deep-copy snapshot; [D-13]). Tools referenced by name (code), not copied. Lands as two phases before R1.4: action-ownership refactor, then templates. See docs/design/specs/2026-05-26-document-templates-design.md.
2026-05-29 Mark Ly Source traceability (R1.6). Every content_output records the sources it consumed — sources_json column (list of Source records pinning each output back to gathered_data IDs + optional fragment selectors). Render records from the INPUT side at render time; LLM-emitted citations, when an output_schema.citation field is declared, are validated against context (never trusted). Bridge denormalizes selectors to URL/path/snippet for one clickable citation per output. Output-level granularity in R1; span-level (per-sentence in prose) defers to R2 texts. New [D-14]; [D-4] updated. See docs/design/specs/2026-05-29-source-traceability.md.
2026-06-05 Mark Ly Fields + cited sources + verify phase (R1.8). Adds fields — per-document parameters (the blank you fill in for one document, e.g. subject_name), distinct from the org-scoped texts/snippets registries. A field is declared on a template (default value), copied to each instance on instantiate_template, and overridable per instance — this is how one template serves many subjects (Brian Jean + Elon Musk background checks off one Background Check template). Jinja substitution precedence is org texts → document fields → generate-time variables. New [D-15]. Also: section output_type='cited' now returns {content, sources[]} (persisted to sections.sources_json), and an optional verify phase (gated by documents.verify_cited_sources) semantically fact-checks each cited claim and stamps sources confirmed/unsupported/unchecked; [D-14] extended. (Engine/transport detail — agentic-loop bounds, gather/verify parallelism, async generate + progress streaming, per-phase model split — lives in the TDD [T-refactor-r18].)
2026-06-05 Mark Ly Substitution-layer vocabulary rename (docs). atoms → texts and molecules → snippets throughout, mirroring the code rename in PR #79. Field and risk_block are unchanged. Text = org-scoped static/verbatim value; Snippet = org-scoped Jinja template; Field = per-document blank.

Concepts

This section is the source of truth for DocGen's vocabulary. Every term used in subsequent sections, in downstream PRDs/TDDs, and in code, means exactly what's defined here.

[D-1] Organizations and multi-tenancy

An organization is one client engagement — Odyssey is an organization, JBRK is an organization, VS Sales is an organization. Every persisted row in DocGen carries an organization_id that identifies which organization owns it.

DocGen runs in one of two deployment shapes:

  • AltaML-managed. DocGen has its own database (named docgen) on the shared AltaForge Postgres server. Multi-tenant within: all AltaML-managed organizations share the same database, separated by organization_id. Same pattern as other AltaForge products (SDE, RA).
  • Client-cloud. An organization requires data sovereignty (e.g., Odyssey). DocGen is deployed into the client's own cloud account with its own Postgres instance. The deployment runs as a single organization — same multi-tenant code, just with one row in organizations.

Local development uses SQLite — a single file (./dev.db) with multi-tenant via organization_id, mirroring the staging shape so models behave identically. SQLAlchemy abstracts the dialect.

Per-organization configuration (LLM provider, model, eval backend, synthesis mode default) lives in the organizations table.

[D-2] Documents and sections

A document is either a template (is_template=true) — a reusable, generatable configuration you stamp instances from — or an instance (is_template=false): one rendered output (a reporting letter, a firm-order memo, a sales letter) whose sections + actions were deep-copied from a template (or built ad-hoc). Documents belong to an organization and have a state (draft, under-review, finalized); an instance records created_from_template_id. See [D-13] for the template/instance model.

A section is a named region of a document's rendered output. Sections carry two ordering columns:

  • ordinal — immutable creation order. Set once at insert (max + 1); (document_id, ordinal) is unique. Stable identity for audit + external references.
  • position — mutable display order. Sort key for list_sections. Rewritten as 1..N by reorder_sections(document_id, ordered_ids). New sections append to the end (max + 1).

Sections also have a title. The reporting letter has sections "Section I", "Section II", "Section III". The firm-order memo has sections "Paragraph 1" through "Paragraph 5". Underwriters can reorder these in the UI (drag-drop on position) without disturbing ordinal — the audit trail keeps the original "who was created first" record while the displayed order is whatever the user wants.

Each section owns two ordered lists of actions — and owning them is what makes a template deep-copyable ([D-13]):

  1. data_needs — gather actions: what data this section needs before rendering. See [D-5].
  2. content_outputs — output actions: what produces the section's rendered content. See [D-4].

A section has no single content-recipe column and no stored current_content. Its rendered content is derived — the ordered concatenation of its content_outputs[].result, composed on read. Each output action carries its own status (empty → generating → generated / failed; edited after a human edit), so a single output can be regenerated or edited independently; the section's status is the aggregate.

[D-3] Tools — first-class data sources

A tool is an invokable function that returns data. Tools are the deterministic side of data acquisition: a SQL query, an HTTP call, an MCP server invocation, a file read. No LLM is involved in a tool call.

Tool fields:

  • name — identifier (e.g., web_search, client_db_lookup, pdf_parse)
  • kind — web_search / page_scrape / file_ingest / mcp / solution-specific
  • description — what this tool returns
  • args_schema — Pydantic model describing required + optional args
  • output_schema — Pydantic model describing the shape of the return value

Tool implementations live in code (Python classes subclassing DataTool), not in the database. The tools table stores registration — which tools an organization has enabled, plus per-org configuration that doesn't include secrets. Credentials never live in the database (see [D-9]).

R1 ships six built-in tool implementations plus a test stub:

Tool Purpose Credential requirement
AddTool Add two numbers ({a, b} → {sum}); deterministic None
AppendTool Concatenate N strings ({strings, separator?} → {result}); deterministic None
WebSearchTool HTTP call to a search provider (Tavily / Brave) Provider API key from env var
ClaudeWebSearchTool Anthropic server-side web search via altaforge.llm Anthropic API key (handled by altaforge.llm)
PageScrapeTool Fetch + parse a URL None typically
FileIngestTool Read a local or remote file (PDF, .docx, .md, .txt) None for local; depends on source for remote
MCPTool Call an MCP server MCP server URL + token from env vars
StubTool (test-only, under tests/) Canned-response double for unit tests None

Solution-specific tools (JBRK's PDF-to-graph extractor, Odyssey's EFS scraper) start as DataTool subclasses in solution repos and graduate into the framework's built-in set when they prove reusable across engagements.

[D-4] Content outputs — output actions

A content output is one output action on a section — it produces a piece of the section's rendered content. A section owns an ordered list of them (content_outputs). An output is either a prompt (an LLM renders it, and may call tools mid-render) or a tool (it emits a tool's result directly — deterministic, no LLM); R2 adds text / risk_block.

There is no shared prompts table. Prompt text lives inline on the action that uses it. An output action carries its own prompt body, so the action is fully self-contained and travels verbatim when a template is deep-copied ([D-13]). This is the deliberate inverse of a shared prompt library: per-document tuning, total isolation between a template and its instances.

Content-output fields (the action core, then output-specific tail):

  • section_id — FK to the owning section
  • ordinal — order within the section's outputs
  • kind — 'prompt' | 'tool' in R1; 'text' / 'risk_block' in R2
  • prompt_body — inline prompt text with Jinja variables ({{ context.client.name }}, etc.), when kind='prompt'
  • tools_available — tool names the prompt may call mid-render, when kind='prompt'
  • tool_ref — tool name when kind='tool'; the tool's result is the output (deterministic, no LLM)
  • args_json — args for the tool call, when kind='tool'
  • output_schema — (optional) JSON Schema / Pydantic dotted path for the LLM's structured output, when kind='prompt'. If it declares a citation field, the render phase validates the LLM-emitted citation against context. See [D-14].
  • result — (tail) the rendered output for this action
  • status — (tail) empty / generating / generated / failed / edited
  • sources_json — (tail) list of Source records pinning this output back to the gathered_data rows it consumed (and, optionally, to a fragment within one of them via selector). Populated by the render phase. See [D-14].

An output action may invoke tools mid-render (the prompt declares which). When it does, the engine emits an Arize span flagged render_time_tool_call=true so reviewers can promote the call into the gather phase if it gets expensive.

Why not a shared prompt catalog? Sharing fights the snapshot principle (a template edit would mutate every existing instance), and the reuse it promises is delivered instead by templates (copy on instantiate). The one place a shared, verbatim, fix-propagating catalog earns its keep is R2 texts / risk_blocks — deterministic, lawyer-approved boilerplate — which is a separate curated concept, not per-document LLM prompts.

[D-5] Data needs — gather actions

A data need is one gather action on a section — a declaration of one piece of data the section needs before rendering. A section owns an ordered list of them; the engine unions them across the whole document during gather.

Like content outputs ([D-4]), a data need is self-contained: a prompt-kind gather action carries its prompt text inline; a tool-kind references a tool by name (code, not copied). Both travel verbatim on template deep-copy ([D-13]).

Data need fields (the action core, then gather-specific tail):

  • section_id — FK to the owning section
  • ordinal — order within the section's gather actions
  • kind — 'tool' or 'prompt'
  • prompt_body — inline prompt text when kind='prompt' (the prompt may itself call tools); null for kind='tool'
  • tools_available — tool names the prompt may call mid-invocation, when kind='prompt'
  • tool_ref — tool name when kind='tool'; null for kind='prompt'
  • args_json — arguments to pass at invocation time (Jinja-interpolated with document-level inputs)
  • output_key — (tail) the name this data is bound to in the gathered context (client_facts, recent_news, etc.)

A section that says "I need the client's recent press releases" declares a data need pointing at WebSearchTool with args {query: "{{ client_name }} press releases", max_results: 5} and output_key='press_releases'. The engine runs the search once, persists the result under that key, and the synthesize + render phases see it.

If two sections declare the same gather action (same kind + tool_ref/prompt_body + args), the engine dedupes — one invocation, both sections see the same result.

The three phases that follow operate on this declaration. The full pipeline at a glance:

R1 pipeline

[D-6] Gather phase — produces gathered_data

The gather phase is the first step of generate_document. It:

  1. Walks the document's sections and unions their data_needs (dedupes on (kind, tool_ref-or-prompt_body, args_hash)).
  2. Invokes each unique gather action:
  3. kind='tool' → direct call into the registered DataTool named by tool_ref.
  4. kind='prompt' → LLM call via altaforge.llm with the action's inline prompt_body (the prompt may itself invoke tools mid-call).
  5. Persists every result to gathered_data keyed by (document_id, source_ref, args_hash).
  6. Returns a context dict where keys are the output_key values and values are the structured results.

gathered_data rows survive past generation — they're the audit trail. Regenerating a document with unchanged inputs hits cached results; changing args or re-running with force_refetch=true produces new rows.

[D-7] Synthesize phase — produces synthesized_context

The synthesize phase takes the gathered context and produces a SynthesizedContext — a single document-scoped object that every section's render step reads. R1 supports two output modes:

  • mode='raw_text' — a single LLM call digests the gathered context into a natural-language "understanding of the case." Sections render against a {{ context.summary }} Jinja variable.
  • mode='structured' — a single LLM call extracts typed facts into a Pydantic model defined per-organization (e.g., OdysseyDealFacts, JBRKClientFacts). Sections render against named fields ({{ context.deal.client_name }}).

A third mode — templated — uses texts / snippets to produce the synthesized context deterministically. Templated synthesis defers to R2 alongside texts.

The synthesize phase can also be skipped (mode='none') — sections render directly against gathered_data. Useful for documents where one section's data is one section's content (e.g., "show me the latest press release verbatim").

Synthesized context persists to the synthesized_context table per document. Regeneration re-runs synthesis with the (possibly new) gathered data.

[D-8] Render phase — produces output-action results

For each section's content_outputs, in ordinal order (parallel across sections where possible), dispatch on kind:

  • kind='prompt' — interpolate synthesized_context (and gathered_data as fallback) into prompt_body, invoke the LLM via altaforge.llm; the prompt may call its tools_available mid-render.
  • kind='tool' — call the tool named by tool_ref with args_json; its return is the output. Deterministic, no LLM. (Covers outputs assembled straight from data — e.g. Odyssey's deterministic firm-order paragraphs.)

Store the produced text in that output action's result; transition its status generating → generated (or failed). The section's content is the ordered concatenation of its outputs' results, composed on read.

In R1, an output action is kind='prompt' or kind='tool'. R2 adds kind='text' (deterministic Jinja over variables) and kind='risk_block' (Jinja + approval gate).

Render-time tool calls are allowed but flagged. A prompt output that calls a tool from its tools_available mid-render makes the engine emit an Arize OTel span flagged render_time_tool_call=true so reviewers can spot drift — every render-time call is something that could have been pre-fetched in the gather phase.

[D-9] Tool credentials — env vars, never the database

Credentials for tools (API keys, DB connection strings, MCP server tokens) are never persisted by the framework. Solutions inject credentials into tool instances at registration time:

import os
from docgen import register_tool
from docgen.tools import WebSearchTool, FileIngestTool

register_tool(WebSearchTool(api_key=os.environ['WEBSEARCH_API_KEY']))
register_tool(FileIngestTool())  # no creds needed for local files

The framework's contract is the env var name; what populates the env var differs per environment:

Env Source of env vars
Dev .env file (gitignored) loaded via uv run --env-file .env. Repo ships docgen/.env.example with every required var name.
Staging / UAT / Prod (AltaML-managed Azure) Azure Key Vault per environment; AppService / ContainerApp reads KV references → injects as env vars at boot
Client-cloud sovereignty (Odyssey) Client's own Key Vault in their Azure tenant; same env var names, different values; code is portable
Tests StubTool (in tests/) or hardcoded stubs; no real credentials touched in CI

If multi-tenant shared infra (R3+) needs per-org credential isolation at the framework level, an encrypted tool_credentials table can be added then. R1 doesn't need it (single-org for client-cloud; AltaML staff control AltaML-managed env vars).

[D-10] LLM access

DocGen accesses LLMs via altaforge.llm.client(auth_mode=...).completion(...) from the altaforge-libraries shared monorepo (altaforge-libraries/altaforge/). The wrapper sits over LiteLLM and supports five auth modes:

Mode When How
oauth_anthropic dev (default) with Claude Reads ~/.claude/.credentials.json (Claude Code's local token)
anthropic staging / uat / prod Direct sk-ant-* key via ANTHROPIC_API_KEY → api.anthropic.com (no gateway)
oauth_openai Local dev with OpenAI Reads ~/.codex/auth.json; routes to OpenAI's Codex backend
oauth_gemini Local dev with Gemini Reads ~/.gemini/oauth_creds.json; routes to Google's Code Assist endpoint
litellm AltaForge gateway (currently under repair, bypassed) Static API key + gateway URL via ALTAFORGE_LLM_API_KEY + ALTAFORGE_LLM_BASE_URL

Environment-driven selection. DocGen's _build_llm_client picks the mode by DOCGEN_ENV: dev → oauth_anthropic (developers' CLI sign-in, no key to manage), staging/uat/prod → anthropic (direct key; servers have no CLI login). The ClaudeWebSearchTool always uses the direct ANTHROPIC_API_KEY regardless of env (Anthropic feature-gates server-side web_search off OAuth tokens). The LiteLLM gateway is bypassed in the completions path while it's under repair. See TDD [T-decision-23].

Call sites stay simple regardless of auth mode:

from altaforge.llm import client

response = client(auth_mode='litellm').completion(
    model=organization.llm_model,    # e.g. "anthropic/claude-sonnet-4-5"
    messages=[{"role": "user", "content": rendered_prompt}],
    temperature=0,
)
text = response.choices[0].message.content

Either auth mode routes through providers (directly for the OAuth modes; via AltaML's LiteLLM proxy for litellm mode), which holds provider credentials and dispatches calls based on the model name.

Why the wrapper exists (vs using LiteLLM directly):

  • OAuth-token authentication for per-developer attribution (LiteLLM library doesn't ship this directly)
  • Single integration point for all AltaML products (DocGen, SDE, RA, etc.) using LLMs
  • Centralized place for future cross-cutting concerns (cost budgets, fallback routing, model-version pinning)

[D-11] Texts, snippets, risk blocks — deferred to R2

R1 ships prompts only as the output-action kind (content_outputs.kind='prompt', prompt text inline per [D-4]). R2 adds three richer output-action kinds alongside the workflow gates each one needs:

Recipe What it is Why R2
Text Deterministic Jinja-rendered content (heading + narrative + bullets). Same s85_transfer text works in a post-mortem pipeline, an estate freeze, or a crystallization. Adds determinism path + author workflow (texts ship via PR, once-reviewed-and-stable). Not required to ship the simplest complete pipeline in R1.
Snippet A named bundle of deterministic content units (texts + risk_blocks) arranged in phase_order. Encodes "standard ways of building a document for a recognized situation type." Prompts are not bundled into snippets — they live as kind='prompt' output actions on sections (content_outputs, [D-4]). This honors JBRK's "every word lawyer-approved" principle: anything inside a snippet is deterministic and reviewable. Composition is value-add over inline prompt outputs; defer until texts exist.
Risk block Deterministic Jinja shaped like a warning / analysis paragraph, with a stricter status='approved' workflow gate. Adds the approval workflow on top of the text render path.

R1 shipped a lightweight precursor. The org-scoped Text and Snippet in [D-15] are the simple forms — a Text is a flat value, a Snippet a single Jinja template. The structured composition described in this table (Text = heading/narrative/bullets unit; Snippet = an ordered composition of Texts + Risk Blocks with phase_order + entity_roles; Risk Block = approval-gated analysis) is the R2 upgrade, and is the design JBRK Legal OS requires — its Atom / Molecule / Risk Block model, captured in ../reference/JBRK_Legal_OS_Info_Guide.docx. R2 keeps the R1 names (Text / Snippet) and treats the flat R1 rows as the degenerate case (additive). Full design: ../releases/r2-framework-production-features/r2-prd.md [P2-composition] + r2-tdd.md [T2-composition].

R1's schema is designed so texts / snippets / risk_blocks land additively in R2 — new tables + new content_outputs.kind values, no R1 code changes. The content_outputs.kind discriminator is 'prompt' in R1; R2 extends it with 'text' / 'risk_block'.

[D-12] Concrete walkthroughs

R1's test suite runs walkthroughs against fixture data. Each walkthrough exercises the full gather → synthesize → render pipeline.

Walkthrough 1: cooking (sanity check, no synthesis)

Tools:
  file_ingest(path='fixtures/cookbook/carbonara.md') → ingredient list + steps

Section: "Recipe"
  data_needs:
    - source=tool:file_ingest, args={path:'fixtures/cookbook/carbonara.md'}, output_key='recipe'
  content_outputs:
    - kind=prompt, prompt_body: "Reformat the following recipe as numbered steps:\n\n{{ raw_inputs.recipe }}"

generate_document:
  → gather: one file_ingest call; result persists under 'recipe'
  → synthesize: mode='none' (skipped)
  → render: one LLM call to reformat_recipe
  → markdown export

Walkthrough 2: Odyssey-shape (5 prompts, structured synthesis)

Tools:
  efs_lookup(deal_id) → broker authority + pricing data (solution-specific `StubTool` stand-in for R1)
  pdf_parse(path='fixtures/odyssey/email_notice.pdf') → notice text

Sections (5):
  "Paragraph 1": data_needs=[pdf_parse],  content_outputs=[prompt pt_intro]
  "Paragraph 2": data_needs=[efs_lookup], content_outputs=[prompt pt_auth]
  "Paragraph 3": data_needs=[efs_lookup], content_outputs=[prompt pt_pricing]
  "Paragraph 4": data_needs=[],           content_outputs=[prompt pt_timing]
  "Paragraph 5": data_needs=[],           content_outputs=[prompt pt_signoff]

generate_document:
  → gather: two calls (efs_lookup deduped across paragraphs 2+3); persists 2 rows
  → synthesize: mode='structured' produces OdysseyDealFacts(client_name, authorizing_party,
       pricing, settlement_date, ...) — one LLM call
  → render: five LLM calls (one per paragraph), each reading {{ context.client_name }} etc.
  → markdown export

Five LLM calls for render + one for synthesis = six total. Pre-rearchitect, every prompt would have re-extracted its own facts.

Walkthrough 3: JBRK-shape (texts + snippets — deferred to R2)

JBRK's flow depends on texts (incorporate_newco, s85_transfer, ...) and snippets (post_mortem_pipeline, estate_freeze). Both deferred to R2 per [D-11]. R1's R2-readiness test asserts that adding the text + snippet recipe types is additive.

Walkthrough 4: texts + risk_blocks in one snippet — also R2

Proves the snippet_items.item_type discriminator. A firm_order_pipeline snippet contains text items (deterministic transaction steps) and risk_block items (deterministic Section III analyses, gated by status='approved'). Sections route via role_filter — one section pulls role='text' items, another pulls role='risk'. Both render mechanisms are Jinja; no LLM is invoked when rendering the snippet. R2.

Walkthrough 5 (R1-specific): all-prompts with structured synthesis + render-time tool drift

A document where one prompt has tools_available=['web_search'] set. The render step invokes the tool; the engine emits an Arize span tagged render_time_tool_call=true. The walkthrough asserts the span is recorded so reviewers can see the drift and promote the call to a data_need later.

[D-13] Document templates and instances

A template is a Document with is_template=true: a reusable, fully-configured document — sections, gather actions, output actions, synthesis settings — that you can also generate against sample data to preview it works. "Proposal", "Firm-order memo", "Reporting letter" are templates.

An instance is a Document with is_template=false produced by instantiate_template(template_id, title), which deep-copies the whole subtree — document → sections → (data_needs + content_outputs), prompt text and all — into a new document recording created_from_template_id. The copy clears each output action's result (status='empty'); the user then tweaks the copy and generates with that engagement's variables (e.g., "Proposal — Client A" vs "— Client B").

Key properties:

  • Snapshot, not reference. Editing a template never changes existing instances. Each instance froze the config that produced it — audit-correct for legal / underwriting output.
  • Nothing shared. Because actions own their prompt text inline ([D-4], [D-5]), the deep copy is total — templates and instances share no rows. The only thing referenced (not copied) is tool code (tool_ref by name); its args are copied.
  • Provenance is navigable; promotion is opt-in and additive. (R2 — proposed.) The created_from_template_id link is surfaced in the product — an instance shows "created from «template»", navigable to the source. The one-way rule (template→instance) has a single deliberate, opt-in exception: a user may promote an instance's section edit up to the source template. Promotion is a copy-up — it writes the edit to the template as a normal template revision and changes only future instantiations; it never retro-mutates existing instances and creates no live binding, so the snapshot guarantee above is intact. See R2 [T2-authoring-lifecycle] (A, B).
  • One concept. The template is the type — there's no separate DocumentType. Grouping templates into categories is a later, additive concern.
  • Instances start empty. Config copies; rendered results clear; regenerate fills them with the client's data.

list_templates() returns is_template=true; list_documents() returns instances + ad-hoc docs.

[D-14] Source traceability — output-level pinning to gathered_data

Every rendered content_output carries a sources_json column — a list of Source records that pin the output back to the gathered_data rows it consumed. Reviewers — lawyers, underwriters, brokers — can ask of any generated paragraph "where did this come from?" and follow the chain to a URL, file path + page, or MCP record.

A Source record is {gathered_data_id, source_category, source_ref, output_key, used_as, selector}:

  • gathered_data_id — FK by value to the gathered_data row that fed this output (no DB FK; the parent column is JSON, integrity is engine-enforced via the org-scoped session).
  • source_category / source_ref — mirrored from gathered_data so the UI labels each source without a JOIN.
  • output_key — the data_needs[].output_key this row bound to in the gather context.
  • used_as — how the engine used it: 'prompt_context' (interpolated into a prompt), 'tool_input' (an arg of a deterministic tool-kind output), or 'tool_result' (a render-time tool call per [D-8]).
  • selector — optional fragment pin within gathered_data.result_json (e.g. {result_index: 2} for a web_search hit; {page: 7, char_range: [120, 480]} for a file_ingest extract; {json_path: "$.x.y"} for MCP). None means "the whole row was relevant; no claim of pinning."

Recording is from the INPUT side at render time. The framework knows what it interpolated into the prompt (Jinja-referenced output_keys), so it records that — deterministically, in code, not by parsing the LLM's prose. LLM-emitted citations are never trusted as the source of truth. Optional per-output: when the action's output_schema declares a citation: {gathered_data_id, selector} field, the LLM emits a citation alongside content AND the engine validates it (the cited gathered_data_id must be in context; the selector must resolve against gathered_data.result_json). A validated citation narrows the matching Source's selector (None → {result_index: 2}); an invalid citation raises CitationValidationError and the output's status='failed'. The hallucinated-citation failure mode never persists.

The bridge layer denormalizes each persisted Source by loading its gathered_data row once and applying the selector, emitting an enriched wire payload {url, title, snippet, page, ...}. The UI shows ONE clickable citation per output (or a list, in wide-prompt cases) without a JOIN at read time. Persisted = lean, wire = consumer-friendly.

Granularity (R1): output-level. A content_output records the rows it consumed; one of them may be pinned to a fragment via selector. Span-level pinning (per-sentence within a prose blob) is deferred to R2 on top of texts — texts are deterministic Jinja blocks whose variable bindings are the source pins, by construction. Free-prose span pinning in R1 would require either trusting the LLM (rejected — failure mode this design exists to prevent) or post-hoc embedding-similarity matching (rejected — manufactures evidence the system didn't establish).

Solution-author pattern: if a citation must be defensible (lawyer-approved, regulator-reviewed), structure the section as narrow content_outputs — one output consumes one data_need row. The framework records exactly one Source and (with citation validation) pins the fragment. Wide outputs (one prompt consuming many rows) are also supported; the framework records all rows as sources with selector=None — the UI shows the list, the framework does not pretend to pin which row drove which sentence.

Multi-tenancy: Source records reference only org-scoped gathered_data.ids already filtered by OrganizationScopedSession. Cross-org leak risk is zero by construction.

Verify phase (R1.8, optional). When a document sets verify_cited_sources=true, a verify phase runs between synthesize and render. For each output_type='cited' section it fact-checks the claim against the full gathered page text and stamps every Source with a support verdict — confirmed, unsupported, or unchecked. This is a second, semantic guard on top of the mechanical URL check synthesize already performs (a URL the model never retrieved is stripped). It judges the claim holistically rather than by inline-[n] index matching, and falls back to the search snippet when a fetched page is a dead shell. The gate keeps the extra LLM spend opt-in.

See spec: docs/design/specs/2026-05-29-source-traceability.md.

[D-15] Fields — per-document parameters

A field is a per-document parameter — the blank you fill in for one document. subject_name="Brian Jean", subject_location="Fort McMurray, Alberta", background_check_date="2026-06-05" are fields on one Background Check instance. A field is {key, value, value_type} scoped to a single document_id (unique on (document_id, key)), org-stamped for tenancy.

Fields are deliberately distinct from texts and snippets ([D-11]):

Concept Scope What it is
Text organization A reusable value used verbatim — firm name, severity_rubric. Shared across every document the org generates.
Snippet organization A reusable Jinja template ({{ }}) that composes texts + fields — a disclaimer, a signature block.
Field document The per-document blank — subject_name. One template, filled differently per instance.

Why a separate object. Texts were briefly misused to carry per-document subjects, which collides the moment one template stamps two instances (subject_name can't be both "Brian Jean" and "Elon Musk" as a single org text). Fields fix that: the template carries field defaults, instantiate_template deep-copies them onto the instance (alongside sections + actions per [D-13]), and the instance overrides the values that differ. Texts/snippets stay for genuine org-level reuse.

Substitution precedence. The Jinja namespace every phase renders against is layered org texts (base) → document fields → generate-time variables (highest wins). The same merged namespace is built in gather, synthesize, render, and the bridge's output-key hashing — they must agree, or a gathered row computed under one scope won't map to the data_need resolved under another.

R1 → R2 forward compatibility

R1 ships:

  • tools table + registration mechanism (tools referenced by name; code, not copied)
  • data_needs table — section-owned gather actions, prompt text inline
  • content_outputs table — section-owned output actions, prompt text inline (kind='prompt'); carries sources_json for output-level source traceability per [D-14]
  • gathered_data table (per-document)
  • synthesized_context table (per-document)
  • documents.is_template + created_from_template_id + instantiate_template deep-copy — see [D-13]
  • No shared prompts table — prompt text lives inline on actions

R2 adds without touching R1 code:

  • texts, snippets, snippet_items, document_snippets, risk_blocks tables (the shared deterministic catalog)
  • content_outputs.kind gains 'text', 'risk_block'; snippet composition references the text catalog
  • Synthesis mode gains 'templated'
  • Span-level source pinning rides on texts — each text's variable bindings are its source pins by construction. Same sources_json shape; selector becomes structural (which text variable served this fragment) rather than LLM-emitted. If audit access patterns shift to reverse lookup ("every output that cited row X"), sources_json promotes to a content_output_sources join table — same fields, FK to content_outputs.id + (now hard) FK to gathered_data.id. Optional, not forced.

Cross-references

Sign-off

Role Name Date
Author Mark Ly 2026-05-26
Engineering reviewer
Approver Mark Ly (CTO)