search-service Agent Tasks

Agent workflow: follow README.md for Audit -> Investigate -> Code -> Test -> Fix; keep this pack's writable scope and verification commands authoritative.

Dispatch type: per-service

Dispatch ID: SVC-014

Current implementation: services/search-service foundation.

Owns: search indexes, sync checkpoints, replay receipts/version tombstones, and query APIs over indexed copies.

Must read:

• legacy search/filter code for questions, exams, courses, documents: /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/questions/questions.controller.ts, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/questions/question-bank.service.ts, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exams/exams.controller.ts, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/courses/courses.controller.ts, /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/storage/storage.controller.ts.
• question/course/document service task packs.

Legacy source evidence:

• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/questions/**
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exams/**
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/courses/**
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/storage/**

Writable files:

• services/search-service/**
• contracts/events/search.projection.changed.v1.schema.json
• contracts/events/README.md
• contracts/openapi/services/search-service.yaml
• docs/api/search-service.md

Orchestrator-owned files:

• owner-service canonical writes/backfill contracts, gateway route cutovers, and shared deploy manifests unless explicitly delegated.

Delegated gateway verification scope:

• Search-service agents must not add public gateway routes until a real legacy controller route is selected from docs/contracts/legacy/legacy-route-snapshot.json and mapped to search-service in docs/architecture/legacy-api-map.md.
• Only after that orchestrator assignment, this service agent may update non-default deploy/gateway/routes.search*.json examples, scripts/test/search-*.sh, docs/qa/search-*.md, and root Makefiletest-search-* targets for the named route rehearsal.
• Until a route is selected, docs/qa/search-route-rehearsals.md and make test-search-routes are the guard that proves no synthetic search-service public route has been introduced.
• The default gateway route table, owner-service canonical write contracts, public catalog ownership decisions, and shared deploy manifests remain orchestrator-owned. Public search route promotion requires browser/runtime parity and rollback proof.

Completed foundation:

• Defines rebuildable QUESTION, EXAM, COURSE, and DOCUMENT index documents.
• Persists sourceService with the owner implementation id used by the current rebuild APIs, not the planning alias. For example, question rebuild state uses question-bank-service, while the canonical planning alias remains question-service.
• Supports query/q, entity type, organization, owner, status, visibility, tag, taxonomy, pagination, and sort filters.
• Stores index documents and source sync checkpoints in service-owned Postgres tables.
• Exposes index upsert/bulk/delete, search, and sync-state APIs.
• Keeps canonical writes in owner services.
• Question-bank projection fixture coverage exists at services/search-service/internal/http/testdata/question_bank_projection_rebuild.json. It exercises bulk rebuild, search filters, idempotent re-indexing, and sync state through the HTTP routes backed by MemorySearchStore.
• Question-bank-service exposes the owner backfill route GET /v1/questions/search-projections for copied QUESTION documents. Search rebuild jobs should consume that route instead of reading question-bank tables.
• Search-service can pull one question-bank projection page through POST /v1/rebuild/question-bank/questions when QUESTION_BANK_SERVICE_BASE_URL is configured. The route indexes copied documents, writes sync-state, and returns PARTIAL while owner pages remain.
• scripts/migration/rebuild-question-search-index.py is the external migration workflow loop for question-bank, course, exam, and document sources. It repeatedly calls the matching search-service rebuild route, forwards org/actor scope, carries the returned cursor, verifies the owner sourceService/entityType, and stops when hasMore=false.
• Course-service exposes GET /v1/courses/search-projections for copied COURSE documents, and search-service can pull one page through POST /v1/rebuild/course-service/courses when COURSE_SERVICE_BASE_URL is configured.
• Exam-service exposes GET /v1/exams/search-projections for copied EXAM documents, and search-service can pull one page through POST /v1/rebuild/exam-service/exams when EXAM_SERVICE_BASE_URL is configured.
• Document-service exposes GET /v1/documents/search-projections for copied DOCUMENT teaching document projections, and search-service can pull one page through POST /v1/rebuild/document-service/documents when DOCUMENT_SERVICE_BASE_URL is configured. Fixture coverage exists at services/search-service/internal/http/testdata/document_service_projection_rebuild.json. Its owner now also has a durable per-document source revision/outbox and an opt-in JetStream publisher. Disposable PostgreSQL and broker proof cover archive DELETE tombstones, restart recovery, duplicate replay, and stale resurrection rejection. Course and Exam now have the same opt-in durable transport proof. Course archive is an ARCHIVED UPSERT; Exam CLOSED is an UPSERT; both use a tombstone only for physical deletion. Pull rebuild remains available only for a tuple without an accepted versioned event; an event-owned tuple is fenced from delayed pull UPSERTs and must recover through owner outbox retry/replay or a controlled version reset.
• Search and rebuild requests now require organization scope unless allOrganizations=true is requested by an ADMIN actor. HTTP routes default scope and actor context from X-Organization-Id, X-User-Id, and X-User-Role; explicit organizationId wins over allOrganizations so a scoped request cannot accidentally become global.
• Search-service now has an opt-in projection freshness worker controlled by SEARCH_REBUILD_WORKER_ENABLED=1. It cycles the configured question-bank, course, exam, and document owner sources, resumes from existing sync-state cursor and nextPage checkpoints, caps each tick with SEARCH_REBUILD_WORKER_MAX_PAGES, runs two independent owner sources in parallel by default through SEARCH_REBUILD_WORKER_MAX_CONCURRENT_SOURCES, and records owner-source failures as sync-state ERROR without making search-service unavailable. Set the concurrent-source limit to 1 for a strictly serial owner-load profile.
• Each worker source result records latencyMs from checkpoint read through owner fetch, copied-index mutation, and sync-state persistence. Run logs the same per-source latency with source, entity, page, requested, indexed, cursor, and error fields so cadence changes can be based on observed owner behavior rather than an assumed fixed interval.
• TestProjectionFreshnessWorkerFromEnvRebuildsOwnerFixturesConcurrently starts four disposable HTTP owner fixtures, configures the worker from its runtime environment, holds every owner response until all four requests are in flight, and verifies copied index rows plus SYNCED checkpoints. It also proves the worker forwards its internal ADMIN scope, page limit, and archive setting without creating a Gateway route.
• Search-service now accepts the internal search.projection.changed.v1 contract through its use case/repository boundary. The contract records (source,event id) receipts and an owner sourceVersion tombstone per copied entity in the same transaction as the index mutation. Duplicate deliveries are no-ops; a same-or-lower version is recorded stale, so an old UPSERT cannot resurrect a higher-version DELETE. Event updates preserve the existing pull-rebuild cursor while updating the SYNCED freshness checkpoint. There is deliberately no native HTTP or Gateway route. An opt-in JetStream subscriber starts only with SEARCH_PROJECTION_EVENT_TRANSPORT=nats, consumes SEARCH_PROJECTIONS / search.projection.changed.v1, and ACKs valid or stale deliveries while NAKing transient persistence failures. The default is disabled; Question Bank, Document, Course, and Exam publish only when their own opt-in transport is configured.

Next tasks:

• Collect owner/source freshness and recovery latency for the four opt-in producers before changing worker cadence or enabling any transport by default. Define a controlled version-reset runbook before using it; it must not become a blind pull overwrite.
• Use docs/agents/handoffs/SVC-014-search-projection-event-transport.md for the owner revision/outbox prerequisites. In particular, do not treat an updatedAt timestamp, a random event id, or question version alone as a sufficient sourceVersion without proving metadata and delete coverage.
• Do not add a search-service gateway rehearsal until the target is a real legacy route selected from docs/contracts/legacy/legacy-route-snapshot.json and mapped to search-service in docs/architecture/legacy-api-map.md. Current /api/public/catalog* entries are course-service-owned public catalog routes; they may feed copied COURSE index data, but they are not a search-service cutover target.
• When a real public search route is selected, add a non-default route-table rehearsal with scope guards, rollback evidence, and browser/runtime parity before any default route promotion. Do not invent synthetic /api/search compatibility routes without legacy evidence.
• Add browser/runtime parity evidence before any default route cutover.

Acceptance:

• Search rows are rebuildable from owner service data/events.
• Search does not become source of truth.
• Index freshness and sync state are observable.
• Public search/catalog promotion requires the existing organization-scope enforcement and admin/global search bypass rules to be paired with gateway route guards, browser/runtime proof, and rollback evidence.
• Public route promotion must cite the exact legacy controller and owner mapping; copied index ownership alone is not enough to move a catalog route to search-service.
• Search queries must not bypass owner-service visibility rules; copied index rows may be filtered, but canonical data and authorization decisions remain with owner services or gateway/BFF contracts.

Verification:

• GOTOOLCHAIN=go1.25.11 go test ./services/search-service/... -count=1
• GOTOOLCHAIN=go1.25.11 go test ./services/search-service/internal/http -run TestQuestionBankProjectionFixtureRebuild -count=1 -v
• GOTOOLCHAIN=go1.25.11 go test ./services/search-service/internal/usecase -run 'TestProjectionFreshnessWorker|TestRebuildFromProjection|TestHTTPProjectionSource' -count=1 -v
• GOTOOLCHAIN=go1.25.11 go test -race ./services/search-service/internal/usecase -run 'TestProjectionFreshnessWorker' -count=1 -v proves independent owner sources can start in parallel without a race while result ordering and per-source error/sync-state handling remain intact.
• GOTOOLCHAIN=go1.25.11 go test -race ./services/search-service/internal/usecase -run 'TestProjectionChange' -count=1 -v proves duplicate receipt handling, stale-version rejection, delete tombstone behavior, and pull cursor preservation in memory.
• SEARCH_PROJECTION_EVENTS_POSTGRES_CONFIRM=disposable-postgres make test-search-projection-events-postgres starts an isolated PostgreSQL container and proves the receipt, version tombstone, copied index mutation, stale replay outcome, and pull-rebuild fence transactionally.
• SEARCH_PROJECTION_PULL_WORKER_POSTGRES_CONFIRM=disposable-postgres make test-search-projection-pull-worker-postgres builds the real search-service binary with four local owner fixtures and one disposable Postgres database. It proves a one-page initial tick writes four PARTIAL cursor/nextPage=2 checkpoints, process restart resumes all four owner sources at page two, and the second tick writes four terminal SYNCEDnextPage=1 checkpoints, eight copied rows, native search/sync responses, and per-source latencyMs logs.
• SEARCH_PROJECTION_BROKER_SMOKE_CONFIRM=disposable-runtime make test-search-projection-broker-live starts isolated Question Bank/Search PostgreSQL and JetStream containers; it proves subscriber restart recovery, duplicate replay, stale ordering, and DELETE tombstones for Question Bank only, without a Gateway route.
• GOTOOLCHAIN=go1.25.11 go test -race ./services/search-service/internal/http -run TestProjectionFreshnessWorkerFromEnvRebuildsOwnerFixturesConcurrently -count=1 -v proves the environment-configured worker reaches all four disposable HTTP owner fixtures concurrently, then indexes their copied documents and writes SYNCED checkpoints.
• make test-search-projection-freshness-latency proves per-source latency includes the owner delay and the environment worker still rebuilds all four HTTP owner fixtures concurrently.
• DOCUMENT_SEARCH_PROJECTION_OUTBOX_POSTGRES_CONFIRM=disposable-postgres make test-document-search-projection-outbox-postgres proves Document owner create/archive/delete revision, rollback, lease/retry, and dead-letter state in an isolated PostgreSQL container.
• DOCUMENT_SEARCH_PROJECTION_BROKER_SMOKE_CONFIRM=disposable-runtime make test-document-search-projection-broker-live proves only Document-to-Search restart recovery, duplicate/stale replay, and the native archive DELETE tombstone through isolated JetStream/PostgreSQL.
• COURSE_SEARCH_PROJECTION_OUTBOX_POSTGRES_CONFIRM=disposable-postgres make test-course-search-projection-outbox-postgres proves Course owner count coalescing, rollback, organization-scope move, archive ARCHIVED UPSERT, physical-delete tombstone, and lease/retry/dead state in isolated PostgreSQL.
• COURSE_SEARCH_PROJECTION_BROKER_SMOKE_CONFIRM=disposable-runtime make test-course-search-projection-broker-live proves only Course-to-Search restart recovery, duplicate/stale replay, archive state, and the physical-delete tombstone through isolated JetStream/PostgreSQL.
• EXAM_SEARCH_PROJECTION_OUTBOX_POSTGRES_CONFIRM=disposable-postgres make test-exam-search-projection-outbox-postgres proves Exam owner revision/outbox creation, copied child-count mutation, rollback, redaction, and lease/retry/dead state in isolated PostgreSQL.
• EXAM_SEARCH_PROJECTION_BROKER_SMOKE_CONFIRM=disposable-runtime make test-exam-search-projection-broker-live proves only Exam-to-Search restart recovery, duplicate/stale replay, CLOSED UPSERT, and the physical-delete tombstone through isolated JetStream/PostgreSQL.
• GOTOOLCHAIN=go1.25.11 go test ./services/search-service/internal/http -run 'TestQuestionBankProjectionFixtureRebuild|TestDocumentServiceProjectionFixtureRebuild' -count=1 -v
• GOTOOLCHAIN=go1.25.11 go test ./services/search-service/internal/http -run 'TestSearchRebuildQuestionBankQuestionsRoute|TestSearchRebuildCourseServiceCoursesRoute|TestSearchRebuildExamServiceExamsRoute|TestSearchRebuildDocumentServiceDocumentsRoute' -count=1 -v
• make test-search-rebuild
• make test-search-routes
• SEARCH_REBUILD_SELF_TEST=1 python3 scripts/migration/rebuild-question-search-index.py covers the external rebuild helper for question-bank, course, exam, and document sources.
• Optional generic Postgres index/sync integration: SEARCH_SERVICE_POSTGRES_TEST_DATABASE_URL=... GOTOOLCHAIN=go1.25.11 go test ./services/search-service/internal/repository -run TestPostgresSearchIndexAndSyncIntegration -count=1 -v
• Backfill/rebuild slices must add and run a named rebuild fixture test before route promotion; current HTTP/repository tests only prove local index/search/sync.