HocTapAZ Docs

Appearance

Handoff: SVC-018-usage-billing-sync-replay-proof usage-service billing sync replay proof ​

Scope ​

  • Service/area: usage-service
  • Goal: prove replay-safe billing entitlement synchronization while keeping quota decisions inside usage-service and keeping money movement and workload execution outside it.
  • Non-goals: public/default route promotion, live browser proof, wallet AZ Credit debit/grant orchestration, billing renewal mutation, payment-provider reconciliation, workload execution, rolling-window parity, or changes to legacy source.

Docs Read ​

  • docs/architecture/service-responsibility-matrix.md
  • docs/architecture/legacy-api-map.md
  • docs/agents/service-tasks/usage-service.md
  • docs/agents/handoffs/SVC-018-quota-caller-contract-fixtures.md
  • docs/agents/service-tasks/billing-service.md
  • docs/agents/service-tasks/wallet-service.md
  • docs/agents/service-tasks/monetization-broker-transport.md
  • docs/agents/service-agent-next-slice-queue.json
  • docs/agents/service-agent-next-slice-closeout-ledger.json
  • docs/agents/agent-handoff-format.md
  • docs/api/usage-service.md
  • docs/qa/monetization-broker-live-smoke.md
  • docs/qa/service-agent-readiness-smoke.md
  • contracts/events/README.md

Legacy Evidence Read ​

  • /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/wallet/az-credit.service.ts
  • /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/wallet/az-credit-business.ts
  • /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/exams/exam-core.service.ts
  • /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/questions/question-generation.service.ts
  • /Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/questions/question-solution.service.ts
  • Legacy source was read-only evidence; no files under /Users/velikho/Desktop/WORKING/HOCTAPAZ/ were changed.

Files Changed ​

  • docs/agents/handoffs/SVC-018-usage-billing-sync-replay-proof.md
  • services/usage-service/internal/usecase/usage.go
  • services/usage-service/internal/http/server_test.go
  • No usage-service migration, OpenAPI, gateway route, or legacy file was changed by this proof handoff.

Behavior Implemented ​

  • Reused idempotency keys now require the same account, organization, resource, window, amount, and reference fields as the original usage event. A mismatched reuse returns ErrUsageConflict, maps to 409 USAGE_CONFLICT, and does not change the original caller's counter. Exact replays remain replay: true and do not increment counters.
  • The existing usage-service foundation materializes billing entitlement snapshots, rejects events without sourceSubscriptionEventId, rejects stale snapshots, preserves accepted snapshot IDs on replay, and exposes idempotent check/consume/reset decisions.

Tests And Verification ​

  • Command: GOTOOLCHAIN=go1.25.11 go test ./services/usage-service/... -count=1
  • Result: passed with -count=1.
  • Evidence: usage-service eventfanout, HTTP, repository, and use-case packages passed.
  • Command: GOTOOLCHAIN=go1.25.11 go test ./services/ai-classifier-service/... -count=1
  • Result: passed with the AI generation usage/wallet caller contract tests.
  • Command: GOTOOLCHAIN=go1.25.11 go test ./services/docx-import-service/... -count=1
  • Result: passed with the OCR/import usage/wallet caller contract tests.
  • Command: make test-import-pdf-ocr-workflow
  • Result: passed with the OCR workflow contract guard.
  • Command: make test-monetization-event-chain
  • Result: passed.
  • Evidence: payment, billing, wallet, and usage event-chain coverage passed.
  • Command: make test-monetization-broker-transport
  • Result: passed.
  • Evidence: broker transport coverage, broker self-test, all monetization eventfanout packages, and outbox tool tests passed.
  • Command: USAGE_SERVICE_POSTGRES_TEST_DATABASE_URL=postgres://... GOTOOLCHAIN=go1.25.11 go test ./services/usage-service/internal/repository -run TestPostgresUsageConsumeReplayIntegration -count=1 -v
  • Result: passed against a temporary local Postgres cluster on port 55439 with an isolated test schema; the cluster and temporary directory were removed after the run.
  • Actual command used: USAGE_SERVICE_POSTGRES_TEST_DATABASE_URL=postgres://hoctapaz@localhost:55439/postgres?sslmode=disable GOTOOLCHAIN=go1.25.11 go test ./services/usage-service/internal/repository -run TestPostgresUsageConsumeReplayIntegration -count=1 -v.
  • Evidence: the integration test passed entitlement materialization, first consume, idempotent replay without a second increment, counter listing, and reset.
  • Command: GOTOOLCHAIN=go1.25.11 go test ./services/usage-service/internal/usecase ./services/usage-service/internal/http -run 'TestUsageConsume(ReplayAndQuotaDenial|RejectsIdempotencyKeyReuseForDifferentCaller)$' -count=1 -v
  • Result: passed; the use-case package had no matching tests and the HTTP package passed both replay/denial and conflicting-reuse cases.
  • Command: MONETIZATION_BROKER_SMOKE_SELF_TEST=1 bash scripts/test/monetization-broker-live-smoke.sh
  • Result: passed. This is a parser/fixture self-test only, not live NATS evidence.

Required Evidence ​

  • billing sync replay proof: HandleBillingEntitlementsUpdated tests cover limit mapping, duplicate replay without entitlement ID churn, stale snapshot rejection, and required sourceSubscriptionEventId; NATS subscriber tests cover broker delivery and replay; the Postgres integration passed the persisted consume replay workflow.
  • quota caller contract: exact consume replays remain successful and idempotent, while a reused key with a different caller/debit shape returns 409 USAGE_CONFLICT without a second counter mutation.
  • no money movement in usage: usage-service owns only entitlement, counter, override, and usage-event persistence. Its domain and repository contain no wallet/payment ledger or VND movement operation; wallet debit/grant orchestration remains outside this service.
  • no workload execution in usage: usage-service exposes check/consume/reset decisions and consumes billing events only. It has no AI/import/file/question executor, worker runner, subprocess execution, or workload dispatch client; the NATS component is a billing-entitlement subscriber only.

Current Working-Tree Re-verification ​

On 2026-07-13, the usage package suite, monetization event-chain, and broker transport checks passed. The isolated Postgres consume-replay integration first exposed a stale test fixture: its replay omitted the original ImportJob reference fields, which correctly caused USAGE_CONFLICT under the current contract. The fixture now replays the same reference shape; the Postgres test and full usage suite pass again. The temporary Compose database and volume were removed after verification.

On 2026-07-14, a concurrent quota TOCTOU gap was fixed: ConsumeUsage now passes the resolved finite limit to the store, and both memory and Postgres stores recheck it while holding the counter lock before incrementing or writing an event. TestUsageConcurrentConsumeDoesNotExceedQuota gates two distinct six-page requests after their initial check against a ten-page quota; exactly one returns 200, the other returns 403 USAGE_DENIED, and the counter remains at six. It passed 100 times under -race.

  • GOTOOLCHAIN=go1.25.11 go test ./services/usage-service/... -count=1 and GOTOOLCHAIN=go1.25.11 go test -race ./services/usage-service/... -count=1 passed.
  • make test-monetization-event-chain test-monetization-routes passed.
  • GOTOOLCHAIN=go1.25.11 go test ./services/docx-import-service/... -count=1 and make test-import-pdf-ocr-workflow passed for the named OCR/import caller boundary.
  • The current Postgres integration was not rerun after this recheck because no isolated USAGE_SERVICE_POSTGRES_TEST_DATABASE_URL was supplied. The new Postgres branch compiles and is covered by the same locked-counter logic, but it must not be described as fresh live-Postgres concurrency proof.

Disposable Postgres Replay Runner ​

scripts/test/usage-entitlement-replay-postgres-isolated.sh now creates a disposable postgres:16-alpine container on a dynamic 127.0.0.1 port after explicit USAGE_ENTITLEMENT_REPLAY_POSTGRES_CONFIRM=disposable-postgres confirmation. It removes the container on exit, unsets DATABASE_URL, and replaces any caller test DSN with its own disposable DSN. The runner executes the repository group ^TestPostgresUsage.*(Entitlement|Replay).*$, so current consume replay coverage and later Postgres entitlement/replay tests with that naming contract run together.

This is reproducible isolated repository persistence proof. It does not prove live NATS delivery or redelivery, public/default gateway behavior, browser parity, production data behavior, or legacy rolling-window parity. It now does provide fresh Postgres serialization proof for billing entitlement snapshots: the advisory-lock integration test forces an older and newer snapshot to interleave and verifies the newer limit is the final entitlement state.

On 2026-07-14, the runner passed both TestPostgresUsageConsumeReplayIntegration and TestPostgresUsageEntitlementSnapshotsSerializeAndRejectStaleOverwrite with both DATABASE_URL and the usage test DSN deliberately set to invalid caller values. The latter uses a per-organization/account PostgreSQL advisory transaction lock to force old/new entitlement snapshot interleaving, then asserts the newer 20-page limit is retained. The dynamic 127.0.0.1 container was removed on exit. This confirms runner-owned database selection and fresh isolated Postgres replay/serialization proof, not the broader proof limits above.

Route State ​

  • Legacy route: public quota callers, AI/import/file/question routes, wallet routes, subscription pages, and payment hooks remain legacy-proxied by default.
  • Native route: service-to-service /v1/usage/check, /v1/usage/consume, /v1/entitlements/*, /v1/limit-overrides, /v1/usage/reset, and /v1/events/billing-entitlements-updated remain non-default/native rehearsal surfaces.
  • Gateway state: AI generation and OCR/import caller contracts are named through owner usage/wallet APIs; default public caller routes still need route-specific live/browser and rollback proof, and file/question caller ownership remains a separate decision. No gateway route edits were delegated or promoted.
  • Delegated gateway scope: none; usage-service does not own gateway route changes.
  • Rollback: keep GATEWAY_ROUTE_TABLE=deploy/gateway/routes.json; if a future caller rehearsal regresses, roll back the caller contract or usage-service behavior without moving debit logic into usage-service.

Data Notes ​

  • Tables touched: temporary Postgres test schema only; no shared or legacy database tables.
  • Migration run: services/usage-service/migrations/000001_usage_service.sql and 000002_usage_entitlement_sync_events.sql were applied by the integration test harness to the isolated schema.
  • Validation report: the integration test passed and cleanup left no /tmp/usage-pg.* directory or Postgres process on port 55439.

Risks / Blockers ​

  • The current native period policy is UTC bucketed windows, not legacy rolling-window parity; exact rolling behavior requires a separate route-cutover slice.
  • The prior local NATS chain artifact at output/qa/monetization-broker-live-smoke-20260709-091059/ proves payment -> billing -> usage entitlement materialization and wallet delivery, but this audit did not rerun live NATS, broker redelivery, or the new conflicting-replay behavior. The self-test above must not be treated as live broker evidence.
  • AI generation caller contract through usage and wallet owner APIs is covered by focused tests and the non-default runtime proof; route-specific browser and rollback evidence remain open.
  • OCR/import caller contract through usage and wallet owner APIs is covered by focused tests and the isolated workflow proof; route-specific browser and rollback evidence remain open.
  • Wallet debit/grant orchestration and workload execution must remain outside usage-service.
  • Fresh live Postgres concurrency proof for the atomic recheck remains open until an isolated usage-service test DSN is provided; no shared or legacy database should be used for that test.
  • This bounded handoff does not complete usage-service or the broader service rollout goal.

Next Step ​

  • Keep AI generation and OCR/import as the named caller contracts; attach their route-specific browser and rollback evidence before any public route promotion. Treat file/question caller ownership as a separate explicit contract.

Go-platform documentation is generated from repository Markdown.

Copyright 2026 HocTapAZ