Epic - DocGen¶
status: draft
date: 2026-05-23
author: Mark Ly
brief: ../explanation/brief.md
written-by: AIA + TechLead
stakeholders: Mark Ly (CTO/sponsor)
Length budget: 1-2 pages. One fact, one place - don't restate; cross-reference with [E-*] / [B-*] IDs.
Change History¶
| Date | Author | Change |
|---|---|---|
| 2026-05-20 | Mark Ly | Initial draft — 5 releases (multi-tenant-foundation-evals-odyssey, version-history-export-shared-infra, saas-api-and-demo, verticalscope-api-proofpoint, dealdesk-api-proofpoint). |
| 2026-05-23 | Mark Ly | Restructured to 4 releases. New R1 = framework-v0-core (foundational release with domain model + core framework + tests). Old R1 content shifted into new R2 (production-ready-plus-odyssey). API migrations collapsed into deferrable R4 (saas-api-plus-demo). Decomposition table also adopted the new merged-column format. |
| 2026-05-23 | Mark Ly | Re-decomposed to 6 releases after sizing audit. Two prior releases were too big: old R2 mixed framework features with Odyssey UAT (~10-14 weeks); old R3 mixed framework scale with 5+ solution rollouts (~12-20 weeks). Split so each release ships testable value in 4-8 weeks: R2 = framework production features only; R3 = Odyssey UAT only; R4 = framework scale only; R5 = multi-solution rollout (parallel streams); R6 = deferrable SaaS API + Demo. |
| 2026-05-24 | Mark Ly | altaforge-llm library is built first as a prerequisite — distributed to all AltaML devs via altaml/altaforge-libraries so OAuth-token-for-dev (no virtual-key sharing) works from day one. Tracked here as an R0 prerequisite; ships in altaforge-libraries, not in altaforge-docgen. R1 phases interleaved (Python + UI + demo together) so user testing rides alongside unit testing — see releases/r1-framework-v0-core/r1-stories.md for the 9-phase decomposition. docgen-ui + demo are explicitly in R1 scope (were always there in PRD/brief but missing from the stories doc until now). |
| 2026-05-26 | Mark Ly | R1 rearchitect to gather → synthesize → render pipeline. R1 ships prompts as the sole content recipe and elevates tools to first-class data sources (4 built-ins + Mock). Texts / snippets / risk_blocks + templated synthesis move to R2 as richer rendering primitives layered on the R1 foundation. R1 phase count drops 9 → 8 and story count drops 40-45 → 30-35; R2 grows correspondingly. |
| 2026-05-29 | Mark Ly | Source traceability folded into R1.6. Output-level source pinning (each content_outputs row records the gathered_data IDs + selectors it consumed) lands in the render phase as a cross-cutting concern — no new phase, no new release. Bridge denormalizes selectors to URL/title/snippet so the UI shows one clickable citation per output. Span-level pinning (per-sentence within prose) defers to R2 on top of texts. See ./releases/r1-framework-v0-core/r1-prd.md [P-feat-13] + docs/design/specs/2026-05-29-source-traceability.md. |
| 2026-06-25 | Mark Ly | Four near-term priority features inserted ahead of the remaining R4–R6 queue (CTO-requested): headless Platform API, parallel execution, built-in eval framework + improvement loop, and large-data/RAG ingestion. See [E-next-*]. |
| 2026-05-31 | Mark Ly | R1.7 = doc-owned actions refactor. Adds one final phase to R1: data needs move from sections to the document; content_outputs table is removed in favor of each section owning a single prompt + render_mode + output_template + output_type; synthesize collapses to one structured mega-call (dynamic Pydantic schema built from from_synth sections); render dispatches per-section by render_mode. Adds texts + snippets registries as the deterministic substitution layer (no LLM). Texts / snippets thus partially graduate from R2 into R1 — the minimal org-scoped key/value + reusable-template surface ships in R1.7 to support the new template render mode; richer R2 work (versioned texts / snippet composition rules / risk_blocks) still defers. |
Prerequisite — altaforge-llm (built in altaml/altaforge-libraries)¶
Before R1 starts, altaforge-llm is built and shipped from altaml/altaforge-libraries/altaforge-llm/ to the whole AltaML org. It's the wrapper over LiteLLM that supports both OAuth (default for local dev — every dev uses their own AltaML SSO token) and API key (CI / automation). Distributing it first means no LiteLLM virtual keys get passed around dev machines — every dev authenticates with their own identity from day one. Lives in altaforge-libraries; not an altaforge-docgen deliverable, but a hard prerequisite for R1 Phase 1.5 (prompt rendering).
Estimated effort: ~1-2 weeks (auth flow + LiteLLM wrap + tests). Parallelizable with anything that doesn't need LLM calls.
Decomposition [E-release-N]¶
| Release | One-line goal (what ships, why independent) | Maps to Brief |
|---|---|---|
E-release-1 framework-v0-core Blocked by: altaforge-llm shipped from altaforge-libraries (see Prerequisite above) |
Foundational release. Built in 8 interleaved phases (Python + UI + demo together — user testing rides alongside unit testing). Ships: domain model + schema (organizations, documents, sections, tools, prompts, data_needs, gathered_data, synthesized_context) + multi-tenant data layer + engine running the gather → synthesize → render pipeline (DataTool ABC + 4 built-ins, synthesize in raw_text + structured modes, prompt-only render with Arize-flagged render-time tool calls, output-level source traceability — every content_output records the gathered_data IDs + selectors it consumed; bridge surfaces URL/path/snippet so the UI shows one clickable citation per output; see docs/design/specs/2026-05-29-source-traceability.md) + docgen-ui React component library (Vite + altaforge-ui) + demo Next.js app + pytest + UI integration tests producing real markdown documents in test_outputs/ and a clickable demo. Texts / snippets / risk_blocks defer to R2 as richer rendering primitives layered on this foundation; span-level source pinning (per-sentence within prose) also defers to R2 texts. No deployment infra, no Arize dashboards, no .docx, no client work. ~30-35 atomic stories, 5-7 weeks. |
[B-tldr], [B-user], [B-outcome] |
E-release-2 framework-production-features Blocked by: E-release-1 |
Production-readiness framework features layered on R1: texts (deterministic Jinja content units) + snippets (composition of texts + prompts) + risk blocks (entity + approved workflow gate) + templated synthesis mode (synthesis output via texts / snippets), Arize eval adapter (AX hosted + Phoenix self-host), .docx export, version history (per-section version store + edit/rollback), client-cloud Docker Compose deployment template, Arize Phoenix self-host recipe. No client integration in this release. Framework is feature-complete for any client to operate in a sovereignty deployment. ~35 stories, 6-8 weeks. |
[B-success-1], Public-surface constraint |
E-release-3 odyssey-uat Blocked by: E-release-2 |
First real client deployment: Odyssey Phoenix firm-order-memo generator goes to UAT in Odyssey's cloud. Real phoenix_tool + efs_tool adapters (replacing R1's fixtures), eval suite vs Matthew's gold-standard outputs, deployment via R2's client-cloud Compose recipe, Arize Phoenix self-host colocated. Signoff by Matthew Nicolini + Mark Ly. ~20 stories + UAT iteration, 6-8 weeks. |
[B-success-1], Timeline constraint (Odyssey first paying client), Mandatory-tech constraint (sovereignty) |
E-release-4 framework-scale Blocked by: E-release-2 |
Framework features needed to scale beyond single-client deployments: version-pinning on content units (texts / prompts / risk blocks gain version columns; sections record which version produced their content), multi-format export expansion (.pdf via weasyprint, Google Docs API), shared-infra deployment template (one DocGen DB on shared AltaForge Postgres server, per-org credential scoping). No client integration. ~20 stories, 4-5 weeks. |
[B-success-1], [B-tldr] |
E-release-5 multi-solution-rollout Blocked by: E-release-3, E-release-4 |
Multi-client production rollout (parallel streams): Odyssey → prod (promotion from R3 UAT after a stabilization window). VS Sales → UAT + prod on shared infra. DealDesk → UAT + prod on shared infra. JBRK → UAT + JBRK lawyer-facing UX build (view + export + text-catalog browse) + prod. RWDI / Site / NAIT enter UAT paced by their own readiness. Release closes when the first wave (Odyssey prod + VS + DealDesk + JBRK in UAT) lands. ~80 stories across solution teams, 10-12 weeks. | [B-success-1], Public-surface constraint |
E-release-6 saas-api-plus-demo (deferrable) Blocked by: E-release-5 |
Ship the AltaML-hosted multi-tenant SaaS HTTP API + harden the R1 Demo app into the productized DocGen Demo. Demoable to all prospects independent of client work. Not a gate for any client engagement — all clients ship via lib import in R3/R5. Lands when commercial demand justifies. ~30 stories, 4-6 weeks. | [B-killer], [B-success-2], [B-tldr] (SaaS as default) |
Near-term priority features (CTO-requested 2026-06-25) [E-next-N]¶
Prioritized ahead of the remaining R4–R6 queue. Each is its own design → plan → build; several pull work forward from later releases.
| ID | Feature | What ships | Slots into / pulls from | Why now |
|---|---|---|---|---|
[E-next-1] |
Headless Platform API | Stable, authenticated, versioned HTTP/SDK API exposing all config + execution — template authoring, instance creation, field set, generate, readiness, retrieval — org-scoped via altaforge.auth. Promotes the internal docgen.server + bridge commands into a public API. |
Pulls the API surface forward from R6 (decoupled from the hosted demo). | Odyssey + SITE want to build their own UX and drive all config/exec via API. |
[E-next-2] |
Parallel / multi-threaded execution | Bounded concurrency in the engine: gather independent data_needs in parallel, render isolated sections in parallel, and batch-generate across instances (e.g. all trade packages for a project at once), with rate-limit backoff. |
R4 (framework-scale). | Per-project RFP batches (SITE: 4+ trades) and multi-instance generation are serial today. |
[E-next-3] |
Built-in eval framework + improvement loop | Generalize the per-solution harnesses (scripts/demo_phoenix_eval.py, scripts/site_rfp_eval.py) into a first-class capability: a declarative eval spec per template (gold set + metrics — byte-exact for templated, grounded recall/precision for AI, citation + parity checks), a runner, an HTML scorecard, and an improvement loop (capture misses → prompt/spec suggestions → re-run). |
Builds on R2's Arize adapter + the readiness gate. | Every solution re-implements eval; Phoenix + SITE proved the pattern. |
[E-next-4] |
Large-data / RAG ingestion | A retrieval-backed gather lane for sources too large for the context window: chunk + index + retrieve the trade/section-relevant passages before synthesis; pluggable to an AltaML RAG product. Extends file_ingest. |
New gather/tool capability (extends R1 gather). | SITE's DCC combined spec is 14.4 MB — unprocessable in one context window today; also lifts extraction recall generally. |
Each lands as its own docs/design/specs/ design + plan before implementation.
R5 streams (owners) [E-stream-N]¶
R5 is parallel-streamed by solution team. Per-stream owners + commitments:
| Stream | Owner | R5 deliverable |
|---|---|---|
[E-stream-1] Odyssey → prod |
Mark Gleason + Matthew Nicolini | Promotion from R3 UAT after a stabilization window. See ./releases/r5-multi-solution-rollout/r5-odyssey-prod-prd.md. |
[E-stream-2] VS Sales → UAT + prod |
VS-side AIA | UAT on R4 shared infra, then prod. Short timeline — first stream after Odyssey prod. |
[E-stream-3] DealDesk → UAT + prod |
DealDesk-side AIA | UAT on shared infra, then prod. Medium timeline; runs in parallel with VS. |
[E-stream-4] JBRK → UAT + prod |
Jon Gilbert + Richard Kirby + AltaML AIA | JBRK lawyer-facing UX build (view + export + text-catalog browse) + UAT (sovereignty TBD) + prod. Medium timeline. |
[E-stream-5..7] RWDI / Site / NAIT → UAT |
Per-engagement AIAs | Paced by each engagement's readiness; post-R5 acceptable. |
R5 closes when the first wave (Odyssey prod + VS prod + DealDesk prod + JBRK UAT) lands. Long-timeline streams (RWDI / Site / NAIT) may extend beyond R5.
Sequencing¶
R1 ──> R2 ──> R3 (Odyssey UAT)
│
└──> R4 (framework scale) ──┐
├──> R5 (multi-solution rollout) ──> R6 (SaaS API + Demo)
R3 (Odyssey UAT) ───┘
R3 and R4 can develop in parallel (both depend on R2; R3 is Odyssey integration, R4 is framework features). R5 needs both R3 (Odyssey at UAT) and R4 (shared-infra deploy) to land first. R6 is deferrable — lands when commercially needed.
Within-release parallelism: R5 is explicitly parallel-streamed by solution team (VS Sales, DealDesk, JBRK, Odyssey prod, paced clients).
Open questions [E-q-N]¶
None at the Epic level — release-internal unknowns surface in each child PRD.
Sign-off¶
| Role | Name | Date |
|---|---|---|
| Author | Mark Ly | 2026-05-23 |
| Sponsor | Mark Ly (CTO) | |
| Eng lead | ||
| Approver | Mark Ly (CTO) |
Reviewer's checklist¶
- [x] Each release ships value independently — not an engineering phase
- [x] Each release maps to ≥1 Brief item (
[B-success-N],[B-killer], or[B-not]-clarified) - [x] Dependencies stated; release count 3-7 (6 releases)
- [x] Each release sized for 4-8 weeks of focused work (R5 is multi-stream parallel)
- [x] Open questions are decomposition-level, not release-internal
- [x] Cross-ref IDs (
E-*) assigned - [x] Decomposition table uses the merged-column format per the altaml-claude epic template