Import PDF/OCR Boundary Agent Tasks

Agent workflow: follow README.md for Audit -> Investigate -> Code -> Test -> Fix; keep this pack's writable scope and verification commands authoritative.

Status: P4-037 direct OCR foundation, opt-in durable Mistral batch hand-off, isolated direct/batch worker lanes, and opt-in usage/wallet accounting.

Current native status: POST /v1/import/ocr/jobs requires owner-scoped PDF, PNG, JPEG, and WebP sourceAssetId references from the non-default Gateway candidate (raw storageKey is rejected), calls Mistral POST /v1/ocr, and stores Markdown plus page/model/usage metadata in the import review state. The companion POST /v1/import/ocr/jobs/{id}/reprocess deliberately retries from the document-service source. Jobs found PROCESSING after restart are marked OCR_RECOVERY_REQUIRES_RETRY instead of automatically charging the provider a second time. ocrMode=mistral-batch is an explicit additive mode: it persists provider source/input/output file IDs, batch ID, lease, attempts, and next poll time. Restart resumes only provider polling after a batch ID is durable; an unfinished Start fails closed. Explicit reprocess reopens the same owner-scoped source asset and preserves the stored direct or batch engine. The DOCX Fast parser proof must not be used as OCR/Mistral batch, Word Equation, gateway, or browser parity evidence.

Dispatch type: orchestrator

Owner: import-service through services/docx-import-service.

Purpose: define the PDF/OCR job slice before implementation so agents do not mix OCR provider work into IAM/Profile, billing, wallet, file storage, or frontend routes.

Must read:

• docs/architecture/service-responsibility-matrix.md
• docs/agents/service-tasks/import-service.md
• docs/agents/service-tasks/file-service.md
• docs/agents/service-tasks/usage-service.md
• docs/agents/service-tasks/wallet-service.md
• Legacy OCR/PDF evidence under:
  • /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exam-paper-templates/mistral-document-ocr-converter.service.ts
  • /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exam-paper-templates/mistral-document-ocr-converter.helpers.ts
  • /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exam-paper-templates/mistral-document-ocr-converter.service.spec.ts
  • /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/ai-import/annotated-docx-import.service.ts
  • /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/ai-import/annotated-docx-import.helpers.ts
  • /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/app-data/app-data.import-jobs.ts
  • /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/app-data/app-data.import-operations.ts

Legacy source evidence:

• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exam-paper-templates/mistral-document-ocr-converter.service.ts
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exam-paper-templates/mistral-document-ocr-converter.helpers.ts
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exam-paper-templates/mistral-document-ocr-converter.service.spec.ts
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/ai-import/annotated-docx-import.service.ts
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/ai-import/annotated-docx-import.helpers.ts
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/app-data/app-data.import-jobs.ts
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/app-data/app-data.import-operations.ts

Writable files:

• services/docx-import-service/**
• contracts/openapi/services/docx-import-service.yaml
• docs/api/docx-import-service.md
• docs/agents/service-tasks/import-pdf-ocr-boundary.md

Orchestrator-owned files:

• Default gateway route table deploy/gateway/routes.json.
• Non-default import route examples, scripts/test/import-*.sh, root Makefile, VitePress config, Compose, K8s, and Helm unless explicitly delegated.
• Usage/wallet API contracts unless the orchestrator assigns the cross-service handoff.

Boundary:

• Source kind must be explicit, for example ocr-document or pdf-ocr; do not reuse DOCX Fast source kinds for OCR work.
• Uploaded binary ownership stays with file-service / document-service; import-service stores payload refs, derived page metadata, OCR text/layout output, review state, and materialized result refs.
• OCR provider settings and secrets are runtime config or secrets, not rows owned by IAM/Profile/Billing.
• When DOCX_IMPORT_OCR_USAGE_WALLET_ENABLED=true, import-service makes explicit owner-service calls to usage-service and wallet-service; it never moves credits locally. It preflights one ocr_pages_monthly unit and credit cost, then settles exact result pages with job-scoped idempotency keys. Other import workloads remain separate follow-up integrations.
• Durable statuses/events must cover queued, running, completed, failed, cancelled, retrying, and review-required states without depending on in-memory-only queues.
• Downloads must serve only materialized refs produced by the job and must not bypass file/document-service ownership for stored bytes.
• Public broad /api/exam-import*, /api/imports*, and upload routes stay legacy until route guards, live runtime evidence, browser proof, and rollback notes exist; default broad import routes legacy is the required cutover state.

Initial tasks:

1. Done: audit the legacy Mistral OCR converter and import job/read model code for request shape, provider timeout, page counting, result text/layout shape, warnings, and retry behavior.
2. Done for direct OCR: add service-owned payload, page metadata, provider/model/usage review metadata, status/events, warning, and explicit retry semantics. The source object remains owned by document-service.
3. Done: add provider, service, and HTTP tests before any route rehearsal. make test-import-pdf-ocr-workflow drives the stored document-service source through the real Mistral-compatible provider adapter, verifies review metadata, and verifies explicit reprocess. The routes stay internal-only.
4. Done for direct OCR: reprocess always reopens the original document-service object. Automatic restart recovery never retries an in-flight direct provider call.
5. Done for opt-in batch parity: ocrMode=mistral-batch uploads the source and JSONL input, submits endpoint=/v1/ocr, persists provider file/batch IDs plus lease/attempt/next-run state, polls the provider, and reads the output JSONL. Direct remains the omitted/default mode. Explicit batch reprocess reopens the same owner-scoped source asset, starts/polls a new batch, and emits the batch reprocess event; it never falls through to the direct engine. The public create-job contract does not state an idempotency guarantee, so automatic recovery resumes Poll only after a provider batch id is durable; a crash during Start fails closed for explicit reprocess or operator reconciliation instead of creating another batch.
6. Done for internal OCR: use versioned usage/wallet owner APIs behind an opt-in adapter. Preflight denial skips the provider; a usable result settles exact pages and persists only accounting metadata. Settlement runs its read-only exact-page quota and wallet-balance checks concurrently, then keeps usage failure/denial precedence before any idempotent debit/consume mutation. Enabled invalid configuration fails the worker closed. Do not apply this accounting keyspace to MathType or other import workloads.
7. Done for the candidate source route: upload through document-service returns only an opaque source id and import-service reopens the exact owner-scoped source with its internal token. OCR and MathType create reject raw storageKey; OCR detail/status/event/review/approve/reprocess and MathType reprocess require the job actor. make test-import-ocr-mathtype-routes validates the narrow route table. The isolated OCR_REVIEW_BROWSER_LOCAL_POSTGRES_CONFIRM=local-postgres make test-ocr-review-browser-local-postgres and OCR_REVIEW_BROWSER_LOCAL_POSTGRES_CONFIRM=local-postgres make test-ocr-batch-review-browser-local-postgres proofs cover direct and explicit batch OCR through Teacher Web, Gateway, local PostgreSQL, and a Mistral-compatible stub. After a Teacher browser save/reload, each harness also approves through candidate Gateway and reads the persisted Question Bank editor projection through candidate Gateway. It asserts the edited stem, SINGLE_CHOICE, answer key/option, solution, score rule, and source import job. They are not public-provider-quality, production, or default-route promotion evidence; default routes stay legacy.
8. Done for direct OCR latency isolation: direct Mistral requests run in the docx-import-ocr-direct lane, independently bounded by DOCX_IMPORT_OCR_WORKER_CONCURRENCY (default 1, maximum 10). A blocked direct provider call cannot consume a DOCX worker. This is not provider batch parity and does not authorize automatic replay after a restart.
9. Done for review approval: the OCR editor retains its sourceText and sections[] review shape. Approval converts only an explicitly saved OCR review into canonical questions[], retaining the teacher-edited stem_text/stem_html, type, options, reviewed answer labels, score and scoring_rule, solution, source spans, and ocr-document provenance for the Question Bank owner API. Completed OCR without ReviewedAt is rejected before any owner write. Direct and batch browser/PostgreSQL runs then prove the owner approval/readback rather than only an in-memory owner contract.

Acceptance:

• PDF/OCR import jobs are service-owned in docx-import-service and do not create a duplicate services/import-service.
• The job lifecycle survives service restart through persistent repository state and payload refs.
• A direct OCR call interrupted by a process restart is not automatically replayed. A batch job resumes only its provider poll after the provider batch ID is durable; a crash during batch Start remains a durable failed job for explicit reprocess or operator reconciliation.
• OCR output can be reviewed and approved through existing question/exam owner APIs, not direct cross-service DB writes. A completed OCR result is never approvable until an explicit review save creates ReviewedAt.
• The reviewed Question Bank readback must preserve the teacher's edited stem, type, answer, solution, and scoring rule in the owner version.
• Provider failures produce durable errors, warnings, status events, and retry-safe state.
• A blocked direct provider call cannot head-of-line block the native DOCX lane; queue snapshots and admin summaries identify direct and batch OCR lanes separately, including batch jobs delayed for their persisted next poll.
• Opt-in OCR usage/wallet calls are explicit and test-covered without local credit movement; the default disabled path does not call either service.
• File bytes and generated downloads stay behind file/document-service owned storage refs.

Verification:

• GOTOOLCHAIN=go1.25.11 go test ./services/docx-import-service/...
• make test-import-pdf-ocr-workflow covers the Mistral document/image data URL adapter, batch file/JSONL submit/poll/output adapter, durable review metadata, restart-safe poll recovery, document-service source re-open, preflight/settlement owner-service contract, and explicit reprocess. It is hermetic and does not make a billable Mistral call. It also proves a blocked OCR provider does not block a DOCX job, and proves that unreviewed OCR never calls Question Bank while a saved review sends canonical question payload fields through the owner HTTP contract.
• OpenAPI mirror check: diff -u services/docx-import-service/api/openapi.yaml contracts/openapi/services/docx-import-service.yaml
• Route proof before public promotion: a non-default OCR/MathType Gateway candidate has static/service checks and isolated direct and batch OCR browser proofs with Question Bank approval/readback. No public PDF/OCR route gate exists yet for default promotion; the isolated proof is a runnable candidate gate, not a public rollout. Keep default broad import routes legacy until a non-default deployed runtime, provider-quality evidence, and rollback proof are approved.
• Shared guards: make test-service-task-packs, pnpm docs:build, and git diff --check.

Rollback:

• Remove the non-default OCR route table entries or set them back to legacy_proxy.
• Set DOCX_IMPORT_OCR_USAGE_WALLET_ENABLED=false before rolling back OCR accounting, then leave existing DOCX Fast routes untouched.
• Keep legacy OCR/PDF code read-only; rollback must not edit legacy source.