Runbook: Migrate Agentic Markdown Blobs to Category-ID Paths¶

One-time data migration that accompanies the switch of agentic-search markdown blob paths from category/subcategory name to category/subcategory ID.

Run this once per environment after the backend and the rag_pipeline Function App have both been deployed with the ID-based path change. Until it runs, agentic search returns no results for documents whose markdown was generated under the old name-based layout.

Background¶

	Old layout (name-based)	New layout (ID-based)
Per-document markdown	`markdown/{slug(category)}/{slug(subcategory)}/{doc}.md`	`markdown/{category_id}/{subcategory_id}/{doc}.md`
Per-category corpus	`markdown/{slug(category)}/corpus.txt`	`markdown/{category_id}/corpus.txt`

Azure Blob Storage cannot rename folders or move files, so existing markdown must be regenerated under the new paths by re-running ingestion (reprocess from source). The orphaned name-based folders are then deleted.

Prerequisites¶

Backend and rag_pipeline deployed with the ID-based path change (same release).
An admin bearer token for the target environment (see token steps below).
The frontend URL for the target environment including /api (e.g. https://<frontend>.azurecontainerapps.io/api).
scripts/.env populated with MARKDOWN_CONTAINER_NAME and AZURE_STORAGE_* for the target environment (cleanup-legacy phase only — the reprocess phase uses the API, no DB access needed).
A maintenance window. Reprocessing re-runs chunking + embedding + markdown extraction for every document — run it outside working hours.

The migration script is scripts/migrate_agentic_blobs_to_category_id.py. Both phases are dry-run by default; add --apply to perform writes/deletes.

The rag_pipeline Function App is in a VNet and cannot be reached directly. The reprocess phase enumerates documents via GET /admin/documents/ and triggers each via POST /admin/documents/reprocess, both routed through the frontend URL that nginx proxies to the backend.

1. Dry-run the reprocess phase¶

export ADMIN_TOKEN=$(altaforge-ra-cli --profile <profile> auth ms get-token)
export FRONTEND_URL=$(altaforge-ra-cli config profile show <profile> | jq -r '.api_base_uri')

uv run --project scripts python scripts/migrate_agentic_blobs_to_category_id.py \
    reprocess --frontend-url "$FRONTEND_URL" --token "$ADMIN_TOKEN"

This prints the count and list of documents (ID + name) that would be reprocessed. Documents where any version is in processing/deleting/deleted state are skipped entirely. Review the list.

2. Run the reprocess phase (maintenance window)¶

uv run --project scripts python scripts/migrate_agentic_blobs_to_category_id.py \
    reprocess --frontend-url "$FRONTEND_URL" --token "$ADMIN_TOKEN" --apply

Each document is sent to POST /admin/documents/reprocess, which triggers the rag_pipeline to delete the document from the search index and restart the chunking pipeline; markdown is regenerated under the new ID-based paths. Progress is printed per document as it completes ([N/total] id: name — OK/FAILED). Tune throughput with --max-concurrent (default 3) and --delay. The call is idempotent — re-run to retry any documents reported as FAILED.

Monitor orchestrations to completion via the usual pipeline dashboards before continuing.

3. Verify¶

Run an agentic search and confirm documents are returned.
In the markdown container, confirm corpora now exist at markdown/{category_id}/corpus.txt.

4. Clean up legacy blobs¶

The orphaned name-based folders can be deleted either via Azure Storage Explorer or via the script.

Option A — Azure Storage Explorer (recommended): Open the markdown container, navigate to the markdown/ virtual directory, and delete any top-level folder whose name is non-numeric (e.g. hr-policy, benefits). Folders whose names are integers are the new ID-based paths — leave those untouched.

Option B — script:

# Dry run — lists the orphaned name-based folders that would be deleted
uv run --project scripts --env-file scripts/.env python scripts/migrate_agentic_blobs_to_category_id.py cleanup-legacy

# Delete them
uv run --project scripts --env-file scripts/.env python scripts/migrate_agentic_blobs_to_category_id.py cleanup-legacy --apply

The cleanup deletes blobs under markdown/ that match the legacy name-based layout — specifically:

any blob whose category segment is non-numeric (markdown/<category-slug>/...); or
a blob whose category segment is numeric but whose subcategory segment is non-numeric and is not the category-level corpus.txt (e.g. a category named "2024" → markdown/2024/benefits/...).

ID-based folders (markdown/<int>/corpus.txt and markdown/<int>/<int>/...) are left untouched. Note the deliberate limitation: a purely-numeric legacy path (markdown/2024/corpus.txt, markdown/2024/15/...) is indistinguishable from a real ID-based path, so it is kept rather than risk deleting live data — delete any such folder by hand if a category/subcategory had a purely numeric name. Only run this after step 3 confirms the ID-based corpora are in place.

Rollback¶

There is no in-place rollback — the migration regenerates rather than moves data. If agentic search regresses, redeploy the previous release (name-based paths). The old name-based folders remain intact until step 4 is applied, so deferring cleanup keeps the previous layout available as a safety net.