Handoff: SVC-021-audit-replay-retention-proof audit-service replay retention proof

Scope

• Service/area: audit-service
• Goal: prove append-only audit/security persistence, source replay/idempotency, retention/legal-hold filtering, and legacy admin compatibility behavior.
• Non-goals: default /api/admin/operations/audit promotion, browser/runtime smoke, producer outbox retries, organization-membership producer semantics, or changes to legacy source.

Docs Read

• docs/agents/handoff-phase-9-admin-foundation.md
• docs/agents/service-tasks/audit-service.md
• docs/agents/handoffs/SVC-021-legacy-admin-audit-backfill-report.md
• docs/agents/service-agent-next-slice-queue.json
• docs/agents/service-agent-next-slice-closeout-ledger.json
• docs/agents/agent-handoff-format.md
• docs/qa/audit-admin-compat-route-rehearsals.md
• docs/qa/service-agent-goal-completion-audit.md
• docs/qa/service-agent-readiness-smoke.md

Legacy Evidence Read

• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/admin/operations.controller.ts
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/admin/feature-maintenance.service.ts
• /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/prisma/schema.prisma
• Legacy source and AuditLog rows were read-only evidence; no files under /Users/velikho/Desktop/WORKING/HOCTAPAZ/ were changed.

Files Changed

• docs/agents/handoffs/SVC-021-audit-replay-retention-proof.md
• services/audit-service/internal/repository/memory.go
• services/audit-service/internal/repository/memory_test.go
• docs/qa/audit-admin-compat-route-rehearsals.md
• No migration, gateway route, or legacy file was changed by this proof handoff.

Behavior Implemented

• The memory audit/security store now indexes replay identity as a (sourceService, sourceEventId) pair instead of a delimiter-joined string, preventing false replays when either component contains :.
• The existing audit-service foundation stores append-only audit/security events, deduplicates by sourceService plus sourceEventId, supports retention/legal-hold filters, projects legacy admin rows through a read-only batch endpoint, and exposes the compatibility read route.

Tests And Verification

• Command: GOTOOLCHAIN=go1.25.11 go test ./services/audit-service/...
• Result: passed with -count=1 after the bounded replay-index fix; the Postgres integration test was skipped because AUDIT_SERVICE_POSTGRES_TEST_DATABASE_URL was unset.
• Evidence: audit-service HTTP and memory-repository packages passed, including legacy projection, replay, retention filters, and security event behavior.
• Command: GOTOOLCHAIN=go1.25.11 go test ./services/audit-service/internal/repository -run TestMemoryStoreKeepsDelimitedSourceIdentitiesDistinct -count=1 -v
• Result: passed.
• Evidence: audit and security source pairs that would collide under delimiter joining remain distinct and list as two events.
• Command: GOTOOLCHAIN=go1.25.11 go test ./services/iam-service/internal/auditsink -run TestPublisherPublishesRedactedSecurityEvent -count=1 -v
• Result: passed.
• Evidence: IAM sends a best-effort security projection with sensitive metadata keys removed; this is self-test evidence only, not live sink evidence.
• Command: make test-audit-admin-compat-routes
• Result: passed.
• Evidence: default route table remained legacy; Compose and localhost non-default audit compatibility route tables passed method, ordering, auth, and target checks.
• Command: AUDIT_SERVICE_POSTGRES_TEST_DATABASE_URL=postgres://... GOTOOLCHAIN=go1.25.11 go test ./services/audit-service/internal/repository -run TestPostgresAuditAndSecurityIntegration -count=1 -v
• Result: passed at the Go test process level with TestPostgresAuditAndSecurityIntegration skipped because AUDIT_SERVICE_POSTGRES_TEST_DATABASE_URL was unset.
• Evidence: no current Postgres-backed or live audit-service evidence was produced in this pass.
• Prior isolated self-test retained from the earlier handoff: AUDIT_SERVICE_POSTGRES_TEST_DATABASE_URL=postgres://hoctapaz@localhost:55441/postgres?sslmode=disable GOTOOLCHAIN=go1.25.11 go test ./services/audit-service/internal/repository -run TestPostgresAuditAndSecurityIntegration -count=1 -v passed against a temporary local Postgres cluster with an isolated test schema; the cluster and temporary directory were removed after the run.
• Prior self-test evidence covered audit source replay without a second row, retention/action/entity filters, security filters, and append-only rejection for update/delete mutations; it is not live evidence.

Current Reverification (2026-07-13)

• Command: GOTOOLCHAIN=go1.25.11 go test ./services/audit-service/... -count=1, make test-audit-admin-compat-routes, and GOTOOLCHAIN=go1.25.11 go test -race ./services/audit-service/internal/usecase ./services/audit-service/internal/repository -count=1.
• Result: passed. This rechecks service HTTP/repository behavior, route-table ordering/auth/targets, and the repository's concurrent append/replay paths.
• Command: AUDIT_SERVICE_POSTGRES_TEST_DATABASE_URL='postgres://hoctapaz_audit:[email protected]:5433/hoctapaz_audit_db?sslmode=disable' GOTOOLCHAIN=go1.25.11 go test ./services/audit-service/internal/repository -run TestPostgresAuditAndSecurityIntegration -count=1 -v.
• Result: passed in the isolated hoctapaz-audit-proof-20260713 Compose project. The integration test verified audit source replay, audit update rejection, security filtering, and security delete rejection in its own temporary schema.
• Runtime: direct audit-service calls created an admin audit event plus a security event, replayed the audit source without changing its action, queried both with retentionBefore and legalHoldBefore, and projected a duplicate legacy batch with processed=2, created=1, replayed=1, failed=0. A direct SQL UPDATE audit_events failed with the append-only trigger. Artifact: output/qa/audit-replay-retention-proof-20260713/.
• Gateway compatibility: a temporary Gateway using deploy/gateway/routes.audit-admin-compat-native-localhost-example.json returned the projected admin audit page to an ADMIN token with route audit-service-admin-operations-audit and state native_read. Restarting it with GATEWAY_ROUTE_TABLE=deploy/gateway/routes.json returned the same path to route admin with state legacy_proxy; the expected 502 was from the deliberately unavailable proof legacy target.
• Reproducible owner-storage rerun (2026-07-15): AUDIT_REPLAY_RETENTION_POSTGRES_CONFIRM=disposable-postgres make test-audit-replay-retention-postgres starts a named PostgreSQL 16 container, runs the existing integration test in a random schema, and removes the container on exit. It verifies source replay, audit/security filters, and both append-only triggers without reading a configured service DSN. Artifact output/qa/audit-replay-retention-postgres-20260715-034640/ contains the focused Go test log and non-secret summary metadata. This is isolated database evidence only, not browser, default-route, or production proof.

Browser Candidate Recheck (2026-07-15)

• Command: make test-admin-audit-owner-browser.
• Result: passed. A fresh initdb audit-only database plus temporary audit-service, Gateway, and Admin Vite processes were removed on exit.
• Evidence: the Admin Operations audit tab rendered the seeded Bật Ví điện tử row through audit-service-admin-operations-audit with native_read; ADMIN received 200, STUDENT received 403, and anonymous received 401. The exact compatibility read now requires api-gateway, X-Internal-Token, and a forwarded ADMIN principal.
• Artifact: output/qa/admin-audit-owner-browser-20260715-034649/; the shell compatibility read took 2.19 ms, browser observation took 10 ms, and the stored request header is Bearer [redacted].

Required Evidence

• append-only proof: migrations define mutation-reject triggers for both append-only tables. The fresh Postgres integration rejected audit update and security delete, while the live audit owner path independently rejected a direct audit-event update.
• replay/idempotency proof: memory and Postgres paths resolve duplicate (sourceService, sourceEventId) to the original event and do not overwrite it or create a second row.
• replay key boundary proof: the memory path keeps colon-containing source-service and source-event pairs distinct for both audit and security events.
• retention and compatibility proof: fresh live audit/security owner calls selected the disposable records with both retention and legal-hold filters. The non-default compatibility Gateway returned the legacy-shaped page to an ADMIN principal; docs/qa/audit-admin-compat-route-rehearsals.md records the mapping and rollback confirmation.

Route State

• Legacy route: public /api/admin/operations/audit remains admin-service-backed in deploy/gateway/routes.json.
• Native route: non-default GET /api/admin/operations/audit maps to /v1/admin/operations/audit-compat and requires an ADMIN principal.
• Gateway state: audit producer/backfill changes require append-only, replay/idempotency, retention, compatibility, and rollback proof; no default route table was changed or promoted.
• Delegated gateway scope: none; audit-service owns no default gateway route changes.
• Rollback: keep GATEWAY_ROUTE_TABLE=deploy/gateway/routes.json; do not promote the audit-service route until browser/runtime, legacy backfill, response parity, and rollback evidence are reviewed.

Data Notes

• Tables touched: only the disposable audit-service audit_events and security_events tables in the named Compose project plus the integration test's temporary schema; legacy AuditLog remained read-only.
• Migration run: audit-service migrations ran in hoctapaz-audit-proof-20260713; the integration harness also applied them in its temporary schema.
• Validation report: docker compose -p hoctapaz-audit-proof-20260713 -f deploy/docker-compose.yml down -v --remove-orphans completed; no project container, volume, or listener remained on 5433, 8101, or 18094.

Risks / Blockers

• The compatibility route has fresh non-default API and local browser proof; response parity, production-like legacy backfill, and rollback evidence still remain default-promotion gates.
• The runtime proof writes only direct audit-service owner records; it does not verify a live IAM or production producer sink.
• A real or production-like legacy AuditLog backfill report is still required before public route promotion.
• Additional producer outbox retry semantics and any legacy security-event backfill remain separate tasks.
• This bounded handoff does not complete audit-service or the broader service rollout goal.

Next Step

• Run a browser recheck for the non-default admin audit view and produce a reviewed legacy AuditLog backfill report before considering any default route change.