import-service Agent Tasks

Agent workflow: follow README.md for Audit -> Investigate -> Code -> Test -> Fix; keep this pack's writable scope and verification commands authoritative.

Dispatch type: per-service

Dispatch ID: SVC-012

Current implementation: services/docx-import-service.

Owns: import jobs, parse orchestration, preview/draft state, review save, approval saga.

Must read:

• Phase 4 and 5 import handoffs under docs/agents/handoff-phase-4-*import*.md and handoff-phase-5-*
• docs/agents/service-tasks/import-pdf-ocr-boundary.md
• docs/agents/service-tasks/import-mathtype-conversion-boundary.md
• legacy import modules under /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/ai-import/, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exam-paper-templates/, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/app-data/app-data.exam-import-approval.ts, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/app-data/app-data.exam-import-lifecycle.ts, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/app-data/app-data.import-*.ts, and /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/formula-recognition/.
• legacy PDF/OCR/MathType files: /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exam-paper-templates/mistral-document-ocr-converter.service.ts, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exam-paper-templates/mistral-document-ocr-converter.helpers.ts, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exam-paper-templates/go-formula-docx-import.service.ts, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/ai-import/annotated-docx-import.helpers.ts, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/ai-import/ai-import-exam-review.ts, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/app-data/app-data.import-jobs.ts.

Legacy source evidence:

• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/ai-import/**
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exam-paper-templates/**
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/app-data/app-data.exam-import-*.ts
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/app-data/app-data.import-*.ts
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/formula-recognition/**

Writable files:

• services/docx-import-service/**
• contracts/openapi/services/docx-import-service.yaml
• docs/api/docx-import-service.md

Orchestrator-owned files:

• import route examples under deploy/gateway/routes.import-*.json, import route tests under scripts/test/import-*.sh, default deploy/gateway/routes.json, shared deploy manifests, and frontend routes unless explicitly delegated.

Delegated gateway verification scope:

• Only when the orchestrator assigns an import route/live/browser proof subtask, this service agent may update non-default deploy/gateway/routes.import-*.json examples, scripts/test/import-*.sh, docs/qa/import-*.md, and root Makefile test-import-* targets.
• Single-route review/detail examples and the combined review roundtrip table are guarded by make test-import-review-roundtrip-routes; do not leave import route examples without a named static target and rollback note.
• The default gateway route table and frontend routes remain orchestrator-owned; public promotion still requires browser/runtime proof and rollback notes.

Initial tasks:

• Done: approval uses the reviewed parse result as the sole source for both Question Bank ingest and EXAM_DRAFT snapshots. Regression covers manual correction/filtering -> save -> approve -> matching downstream ingest and snapshot content, answer keys, explanations, and source metadata.
• Done: expose owner queue/failure projection at GET /v1/admin/import/docx/queue-summary for admin operations queue.failures and queue.depths. The route returns service-local job status counts, native queue depth, optional BullMQ bridge depth, recent failures, and the owner retry command route without letting admin-service or BFF read import storage directly.
• Done: add DOCX Fast workflow coverage for create -> queue -> parse -> review save -> approve, including reviewed manual question-type overrides, source-location fields, and warning fields in the downstream ingest body.
• Done locally: real corpus proof now covers Math 22Q persistence through question-bank-service and EXAM_DRAFT approval. Imported option/media/formula reference primary keys are service-generated; correct_answer.labels from Go Formula DOCX is mapped to correct options and snapshot answer keys. See docs/qa/docx-fixture-persistence-runtime-proof.md. Do not regress this behavior by treating source option labels as global database ids.
• Done: EXAM_DRAFT approval persists an Import-owned operation keyed by (jobId,target). It freezes request/review digests and the outbound owner actor, checkpoints Question Bank IDs and the Exam ID, uses immutable Question Bank/Exam keys, and retries snapshot replacement without deleting owner data. A changed request or review after the first claim returns a conflict; a short lease prevents overlapping attempts. This is a checkpointed retry protocol, not a distributed transaction or default Gateway promotion.
• Current four-fixture approval contract: the DGNL 102-question fixture parses into non-empty stems, but direct approval is blocked before Question Bank because nine rows have no options and only 41 rows have a usable answer. Math 22Q and English 40Q are likewise blocked when their review-required answers remain incomplete; only the fully answered Physics 28Q fixture is forwarded unchanged by the isolated contract. The same contract saves a review-safe source-evidence subset and forwards exactly 13 Math rows, 34 English rows, or 37 DGNL rows with recomputed review stats; a nonnumeric Math placeholder such as Lời giải is not inferred as an answer key. Do not reintroduce automatic DGNL approval without reviewed answers and options.
• Done: add an internal, PDF/image-only Mistral OCR foundation at POST /v1/import/ocr/jobs and POST /v1/import/ocr/jobs/{id}/reprocess. It reuses the service-owned job/event/payload store, writes Mistral Markdown plus page/model/usage metadata into review state, and requires review before any question-bank approval. A direct OCR job found PROCESSING after a restart is failed as OCR_RECOVERY_REQUIRES_RETRY, not automatically sent to the provider again. This is not legacy batch OCR, Word Equation conversion, gateway promotion, or browser parity.
• Done: preserve the OCR editor's source-text/section review JSON, but convert it to canonical ImportResponse.questions[] only after the teacher saves a review. The approval bridge forwards reviewed stem/options/answer labels/solution/source-span metadata to Question Bank; the original completed OCR state remains blocked because it has no ReviewedAt timestamp.
• Done for internal OCR only: an opt-in usage/wallet adapter checks one ocr_pages_monthly unit and the AZ Credit balance before a Mistral call, then settles the exact result page count through idempotent owner-service calls. It records only accounting metadata in the review payload; it does not own quota counters or wallet rows. The flag is off by default and an enabled invalid configuration fails the service closed before it accepts OCR jobs.
• Preserve manual question-type overrides and source-location/warning fields for future PDF/OCR/MathType slices.
• Use import-pdf-ocr-boundary.md for the PDF/OCR job slice separate from DOCX Fast: upload payload refs, page count extraction, provider/runtime selection, durable status/events, retry semantics, download/materialized-asset refs, and no default route promotion.
• Use import-mathtype-conversion-boundary.md for the MathType/OLE/OMML/MTEF conversion slice: adapter contract to the current Go Formula runtime or a later ADR, deterministic formula-source metadata, review-only warnings, materialized media refs, and parser-version fields.
• Done locally for the provenance bridge: the import adapter calls Go Formula QAS once and preserves per-question formulas with LaTeX/MathML, MTEF/OMML source metadata, OLE/preview references, parser version, confidence, and review flags through canonical Question Bank formula refs and approval snapshots. It counts only per-question formulas and does not materialize OLE/WMF previews as browser images.
• Done: validate a new EXAM_DRAFT workflow (exam or tron-de) and delivery mode (ONLINE or OFFLINE) before the Question Bank call. Invalid values in those fields fail without creating canonical questions.
• Done: when EXAM_DRAFT snapshot persistence fails, retain the Question Bank and Exam owner writes and resume from the durable Import checkpoint. Exam snapshot routes resolve the scoped parent Exam first; Import must not issue owner-data deletes as compensation for an ambiguous remote response.
• Done locally: the isolated native PostgreSQL Physics fixture now creates a second job through the direct EXAM_DRAFT state machine and replays the same approval request. The replay returns the original Question Bank IDs and Exam ID with reused=true; owner reads retain one Exam and 28 canonical snapshots with the Question Bank's 18/4/6 type mapping and 16 true/false statements. A focused unit regression also injects one checkpoint-save failure after Question Bank ingest and after Exam create, then verifies the stable owner keys and frozen actor on retry. This validates replay/data parity, not a process-restart/PostgreSQL-outage proof or default Gateway promotion.
• Done: DOCX renderable package images use document-service's token-gated /v1/internal/media-assets boundary when INTERNAL_SERVICE_TOKEN is set. The document owner writes bytes and metadata in one request, avoiding container access to host-only presigned upload URLs. The legacy presigned path remains a tokenless fallback; import-service never writes MinIO/S3 or document-service tables directly.
• Done for the bounded internal conversion slice: POST /v1/import/mathtype/convert returns a direct *-word-equation.docx after replacing deterministic MTEF/OLE formulas with OMML. It is not a job or artifact workflow: it does not persist bytes, bypass document-service ownership, create a Gateway/frontend route, or claim visual equivalence for linear OMML fallback expressions.
• Done for the durable MathType slice: POST /v1/import/mathtype/jobs reads the source only through document-service, persists generated DOCX bytes only through its token-gated artifact endpoint, and stores artifact metadata plus conversion counters in the import job. Scoped download and explicit reprocess routes preserve document-service ownership; interrupted processing requires deliberate retry to avoid duplicate artifact creation.
• Done for the owner-scoped source candidate: OCR and MathType browser uploads enter document-service through POST /v1/import-sources and return only sourceAssetId. The create/reprocess paths reopen bytes through /v1/internal/import-sources/{id}/content with a shared internal token and exact organization/actor/purpose/source-kind checks. OCR and MathType job create reject raw storageKey; their explicit reprocess requires the job actor, and MathType download is actor-scoped. OCR detail/status/event/review/ approve are also actor-scoped. source_media_asset_id persists on the import job; isolated direct and opt-in batch OCR browser/PostgreSQL proofs are recorded in docs/qa/ocr-review-browser-local-postgres.md and docs/qa/ocr-batch-review-browser-local-postgres.md. Each saves the Teacher review, approves through candidate Gateway, and reads the canonical Question Bank editor projection back through candidate Gateway, preserving the edited stem/type/answer/solution/scoring rule. The default Gateway import route remains legacy.
• Done for opt-in DOCX Question Bank approval accounting: legacy-compatible questions entitlement preflight runs before every Question Bank ingest, including QUESTION_BANK, EXAM_DRAFT, the native Teacher review path, and trusted auto-approval. After the durable Question Bank checkpoint, import-service consumes the completed reviewed question count through usage-service with import-approval:usage:questions:{requestDigest} and persists APPROVAL_USAGE_CONSUMED. Denial calls no Question Bank API; consume retry resumes the checkpoint without re-ingest or a second counter write. This is quota parity, not a new AZ Credit wallet charge: legacy approval gated questions only. The flag remains off by default and enabled invalid usage configuration fails closed. DOCX_IMPORT_APPROVAL_USAGE_ENABLED is the canonical flag; the older Fast-only flag remains a compatibility alias. Compose, raw K8s, and Helm declare the canonical flag as false and the timeout as 3000, so an operator must explicitly opt into re-metering. docs/qa/docx-fast-auto-approval-usage-proof.md records the bounded proof.
• Add remaining usage/accounting and public-proof parity for import creation, MathType, and retry/reprocess before cutover. OCR has a bounded internal usage/wallet accounting contract, but list/detail filters must still distinguish docx-fast, ocr-document, and mathtype-equation sources without mixing ownership.
• Done: replace in-memory/limited import job list compatibility filters with organization-aware repository queries and accurate pagination totals for GET /v1/import/docx/teacher-library and the SSE snapshot path.
• Add quota/AZ Credit integration points for import creation, MathType, and retry/reprocess through explicit usage/wallet APIs. Keep the completed OCR adapter scoped to ocr_pages_monthly, and keep DOCX Fast approval on the separate questions entitlement with its own stable key; never reuse OCR idempotency keys for another import workload.
• Done for the bounded SVC-012-import-route-proof-bundle slice: matching status/approval/detail/review route rehearsals, Teacher-native local browser creation, and the review roundtrip self-test are documented; default import routes remain legacy-proxied until deployed rollback evidence exists.
• Done locally for native DOCX review media: the Teacher review surface reads an unapproved package asset only through the authenticated, job-scoped exam-import-docx-fast-asset-content Gateway adapter and renders its Blob URL. Reviewed output retains the canonical document-service media reference for Question Bank. The same local browser proof now exposes the first selected Physics formula as literal LaTeX with source_kind, deterministic/confidence state, and DOCX source part without injecting parser HTML; save/reload preserves its id, LaTeX, and provenance. This candidate exists only in the explicit review roundtrip route tables; local browser proof is passed, while deployment/default-route proof remains required.

Acceptance:

• Import-service stores draft/preview state, not canonical approved questions.
• Import-service owns import queue/failure projections. Admin-service/BFF may call GET /v1/admin/import/docx/queue-summary, but must not read import-service tables, spools, Redis keys, or in-memory queues directly.
• Approval writes through question-service/exam-service APIs.
• Parser/runtime failures produce durable status/events and retry-safe payload refs.
• Reviewed edits, especially questionTypeManualOverride, are the source of truth for approval after save/reload.
• Approval rejects every question that still needs an answer but has no parser/style-hint/review answer. Teachers must save a corrected review before question-bank or exam handoff; import-service must not infer an answer key.
• PDF/OCR/MathType work stays in services/docx-import-service or a documented adapter; do not create duplicate services/import-service or services/formula-docx-service directories.
• The opt-in OCR path calls usage/wallet owner APIs for quota and AZ Credit settlement while import-service records only workload/accounting metadata and idempotency keys. The opt-in DOCX Question Bank approval path is intentionally narrower: it uses usage-service questions preflight/consume only for all Question Bank approvals, matching legacy approval and never moving a wallet ledger into import-service. Other import workloads remain future explicit integrations.
• Download/content routes must serve only materialized refs produced by the job and must not bypass document-service/file-service ownership for stored bytes.

Verification:

• GOTOOLCHAIN=go1.25.11 go test ./services/docx-import-service/...
• Focused owner queue summary regression: GOTOOLCHAIN=go1.25.11 go test ./services/docx-import-service/internal/http -run TestAdminImportQueueSummaryReportsOwnerCounts -count=1
• make test-import-create-routes test-import-create-live test-import-create-browser
• make test-import-ocr-mathtype-routes proves the owner-source candidate tables while preserving default legacy routes; it is not live or browser promotion evidence.
• make test-import-pdf-ocr-workflow for the internal Mistral-compatible OCR create/reprocess workflow, owner-service accounting contract, durable review metadata, preflight denial, and settlement-failure behavior.
• OCR_REVIEW_BROWSER_LOCAL_POSTGRES_CONFIRM=local-postgres make test-ocr-review-browser-local-postgres and OCR_REVIEW_BROWSER_LOCAL_POSTGRES_CONFIRM=local-postgres make test-ocr-batch-review-browser-local-postgres prove the disposable direct and batch Teacher review -> candidate approval -> candidate Question Bank readback path. They are not default-route or real-provider evidence.
• make test-import-status-routes test-import-approval-routes
• make test-import-review-roundtrip-routes test-import-review-roundtrip-live
• IMPORT_REVIEW_BROWSER_SELF_TEST=1 make test-import-review-browser verifies the candidate media/formula route shape. The focused Teacher UI test verifies authenticated asset fetch, Blob rendering, and read-only formula provenance; neither is default-route proof.
• DOCX_PHYSICS_GATEWAY_ATTEMPT_LOCAL_POSTGRES_CONFIRM=local-postgres make test-docx-physics-attempt-local-postgres starts at the real Teacher native file input, proves source upload -> opaque asset -> DOCX Fast create -> native detail polling -> editor, records browser Resource Timing and file-selection latency, then runs formula/media review, approval, and attempt flow. It also checks exam-import-job-status through the dynamically served candidate after creation and completion, retaining only redacted route/correlation/counter and asset-hash artifacts. It is local non-default evidence only.
• Gateway demo route-table guard: make test-gateway-route-examples
• make test-docx-corpus test-docx-materialization test-docx-warning-capture test-docx-warning-parity test-docx-payload-persistence test-docx-bullmq-bridge test-docx-bullmq-status
• make test-docx-corpus uses the read-only four-fixture Go Formula runtime and then runs an in-process Import -> httptest Question Bank approval contract. It checks complete forwarded payload parity for the approval-safe Physics fixture and verifies Math, English, and DGNL cannot call Question Bank while review is required. It is not a shared database persistence or public-route proof.
• DOCX_MATERIALIZATION_SELF_TEST=1 make test-docx-materialization runs the Physics corpus through the real parser and materializer with an in-memory uploader, so the seven image-reference rewrites remain regression-covered without a user-owned import/document runtime.
• DOCX_MATERIALIZATION_COMPOSE_CONFIRM=isolated-runtime make test-docx-materialization-isolated creates a fresh Compose project with no reused host runtime, exposes only dynamically assigned loopback ports, and starts only the parser/document/import dependency slice (not Question Bank, Exam, IAM, or Classroom). It verifies the Physics 28-question parser response has seven materialized images, then reads every image through document-service and removes all project volumes. It is direct owner-runtime evidence, not Gateway/browser or default-route promotion proof.
• DOCX_REVIEW_GUARD_LIVE_CONFIRM=docx-review-guard-live HOCTAPAZ_DOCX_CORPUS_DIR=/Users/velikho/Desktop/test-hoctapaz make test-docx-review-guard-live runs all four local DOCX fixtures against the live import service. It checks parser counts, type/option repair, and that unresolved or multi-label single-choice answers block approval without writing any canonical question-bank rows.
• On an explicitly isolated Compose stack with the temporary BFF listener: DOCX_PHYSICS_EXAM_BLUEPRINT_ISOLATED_CONFIRM=docx-physics-exam-blueprint-isolated make test-docx-physics-exam-blueprint-isolated. It approves the review-safe Physics fixture to canonical Question Bank rows, then verifies the BFF/Exam matrix handoff, online/offline snapshots, and Word template output without changing a default Gateway route.
• DOCX_PHYSICS_EXAM_BLUEPRINT_ISOLATED_SELF_TEST=1 make test-docx-physics-exam-blueprint-isolated checks the isolated Compose topology and dynamic loopback ports without starting the runtime.
• DOCX_PHYSICS_EXAM_BLUEPRINT_LOCAL_POSTGRES_CONFIRM=local-postgres make test-docx-physics-exam-blueprint-local-postgres repeats the same canonical Physics handoff without Docker: Import sends seven source media assets to the temporary Document owner before Question Bank approval, then Exam resolves those assets while rendering Word. The native cluster uses separate owner databases and is removed after the artifact is written.
• DOCX_CORPUS_EXAM_BLUEPRINT_LOCAL_POSTGRES_CONFIRM=local-postgres make test-docx-corpus-exam-blueprint-local-postgres covers the remaining real Math, English, and DGNL fixtures with fresh native owner databases. It first requires the untouched EXAM_DRAFT approval to fail closed with no Question Bank rows, saves only the exact review-safe subset, then verifies canonical Question Bank rows, direct approval replay, BFF ONLINE/OFFLINE matrix snapshots, and a checksum-verified document-service Word export. Its fixed baselines are Math 22 -> 13 (12 single choice, 1 short numeric, two media, 28 formula refs), English 40 -> 34 (28 single choice, 6 passage), and DGNL 102 -> 37 single choice. It removes all temporary listeners and the PostgreSQL cluster after writing the artifact.
• docs/qa/docx-fixture-persistence-runtime-proof.md records the local persistence/runtime scope and explicit OLE/WMF/manual-review limits.
• The 2026-07-15 recheck artifacts are output/qa/docx-corpus-question-bank-local-postgres-20260715081210/summary.json and output/qa/docx-physics-gateway-attempt-local-postgres-20260715081327/summary.json. The 2026-07-15 revalidation of the cross-fixture EXAM_DRAFT workflow is output/qa/docx-corpus-exam-blueprint-local-postgres-20260715113710/summary.json. It re-ran the four-fixture parser/approval contract first, then retained only Math 13/22, English 34/40, and DGNL 37/102 review-safe rows for direct replay, BFF ONLINE/OFFLINE snapshots, and Word export. It is direct owner/BFF evidence only: it does not approve excluded rows, promote a default Gateway route, or replace browser/production proof.
• Focused approval regression: reviewed parse output overrides original parser output in the question-bank/exam approval payload: GOTOOLCHAIN=go1.25.11 go test ./services/docx-import-service/internal/usecase -run TestApproveUsesReviewedParseOutput -count=1
• Browser/runtime parity before route promotion: make test-import-review-browser or the matching live smoke named in the handoff.
• Future MathType and non-OCR accounting slices must add focused status/download/accounting tests plus route guards proving default broad import routes remain legacy until browser/runtime evidence exists.