AI Generation Settlement Reconciliation Proof

This proof covers the AI-owned recovery state after a provider has returned a generation preview but owner-service wallet or usage settlement does not finish. It does not promote a Gateway route or make AI the owner of quotas, credits, or canonical Question Bank writes.

Contract

When both AI_GENERATION_USAGE_WALLET_ENABLED=1 and AI_GENERATION_SETTLEMENT_WORKER_ENABLED=1 are configured, the service stores pending settlement metadata alongside the provider-result replay:

• pending records remain claimable even after the preview TTL expires;
• one worker claims a due record with FOR UPDATE SKIP LOCKED and a lease;
• attempts, retry time, worker lease, and last error are persisted in the AI database;
• a retry calls only the existing owner-service settlement operation with the original idempotency key; it never reruns provider, preflight, Question Bank context, or Question Bank writes.

This is forward reconciliation. It deliberately does not issue a compensating wallet credit when an earlier owner operation may already have committed.

Preflight Latency

The four read-only usage quota checks run concurrently, then the wallet balance is read only after every quota check allows the request. Error precedence stays in the documented resource order (credits_monthly, credits_weekly, credits_hourly, tokens_monthly), so concurrency reduces pre-provider latency without changing the rejection contract or introducing wallet reads on a denied quota.

After a successful wallet debit, the four independent, resource-keyed usage consumes run concurrently as well. Each keeps its existing idempotency key; partial completion remains recoverable by the settlement worker, while the returned error precedence stays in the same resource order.

Focused Verification

make test-ai-generation-settlement-reconciliation
make test-ai-generation-settlement-fault-injection
AI_GENERATION_SETTLEMENT_RUNTIME_CONFIRM=disposable-runtime \
  make test-ai-generation-settlement-runtime
AI_GENERATION_SETTLEMENT_LOCAL_POSTGRES_CONFIRM=local-postgres \
  make test-ai-generation-settlement-local-postgres

The target runs the worker, memory-store lease, HTTP runtime-config, and Postgres integration test packages. The Postgres test skips unless an isolated AI_CLASSIFIER_POSTGRES_DSN is set.

Disposable Cross-Service Runtime Recovery

make test-ai-generation-settlement-runtime starts a uniquely named Compose project with real ai-classifier-service, usage-service, wallet-service, and their owner Postgres stores. It exposes only random loopback ports and uses a loopback-only proxy to return one 503 for credits_weekly after the real wallet debit and three real usage mutations.

The runner disables the worker for the initial request, verifies one durable pending replay, recreates AI with the worker enabled, and verifies the durable row becomes settled after one claimed retry. It then asserts four preflight checks, eight total consumes, one effective debit, the four final owner counter values, a failed-plus-successful provider-usage audit pair, and an identical settled replay that makes no owner request. Its EXIT trap removes the Compose project volumes and terminates the proxy on both pass and failure.

The command requires an available Docker daemon and an explicit confirmation. It passed on 2026-07-14 with bundle output/qa/ai-generation-settlement-runtime-20260714-084542/. The runner observed 1:true:0 before the AI recreation and 1:false:1 after it, retained the wallet at 97, retained owner counters 3, 3, 3, and 3000, and recorded the failed-plus-successful audit pair (2:1:1). The summary records unique Compose volumes with no configured local Compose project or legacy database; post-run inspection found no project container, volume, or proxy process. This is local mock-provider owner-runtime evidence only, not Gateway, browser, deployed-target, credential, or default-route promotion evidence.

Local PostgreSQL Owner Runtime Recovery (2026-07-14)

The Docker-independent command below ran successfully:

AI_GENERATION_SETTLEMENT_LOCAL_POSTGRES_CONFIRM=local-postgres \
  make test-ai-generation-settlement-local-postgres

It built the current ai-classifier-service, usage-service, and wallet-service binaries, created a fresh initdb cluster under /tmp, and applied each service's current migrations to separate owner databases. No configured local, Docker, or legacy database was used. A loopback fault proxy returned one 503 for credits_weekly after the real wallet debit and three real usage writes. The initial durable replay was 1:true:0; after restarting the AI binary with its worker enabled, it became 1:false:1.

The recorded proxy counters were four preflight checks and eight consumes (two per resource). The settled replay left the wallet at 97, wrote one USAGE_DEBIT, retained usage counters 3, 3, 3, and 3000, and did not make another owner call. The AI audit database contains the expected failed pending-settlement event plus the successful event (2:1:1). The proof bundle is output/qa/ai-generation-settlement-local-postgres-20260714-084304/; its EXIT trap removed all test service processes and its local cluster.

The runner now also seeds a disposable non-secret default provider_settings row before the first AI binary start, updates it through the native ADMIN owner endpoint, and saves provider-status responses. The 2026-07-14 artifact output/qa/ai-generation-settlement-local-postgres-20260714-125226/ recorded configurationSource=database, provider mock, and apiKeyConfigured=false. The initial model was database-selected-mock-v1; the endpoint persisted database-selected-mock-v2, which the restarted AI binary then used. This proves current native startup reads the AI-owned setting and the owner write persists it; it does not make a provider-network call or validate a live credential.

A fresh recheck against the current worktree passed at output/qa/ai-generation-settlement-local-postgres-20260714-191927/. Its separate owner databases again recorded one effective three-credit USAGE_DEBIT, the four settled counters (3, 3, 3, and 3000), and no extra owner requests when the settled preview was replayed. The restarted AI binary read the persisted database-selected-mock-v2 setting. The runner removed its native processes and fresh initdb cluster on exit.

The 2026-07-15 recheck passed again at output/qa/ai-generation-settlement-local-postgres-20260715-050357/. It used fresh AI, Usage, and Wallet databases under /tmp, current owner binaries, the same one-time credits_weekly failure, and the database-selected mock provider setting. The runner completed its recovery assertions and removed its native processes and temporary PostgreSQL cluster on exit.

The current 2026-07-15 recheck passed at output/qa/ai-generation-settlement-local-postgres-20260715-053418/. It again used a fresh initdb cluster with separate AI, Usage, and Wallet databases, current native owner binaries, and the one-time credits_weekly failure. The provider runtime read the database-owned mock setting with database-selected-mock-v2; the runner removed its native processes, fault proxy, and temporary PostgreSQL cluster on exit. No configured local, Docker, or legacy database was used.

This is local mock-provider owner-runtime evidence. It does not prove provider credentials, Gateway rollback, browser behavior, deployed targets, or default route promotion.

AI_CLASSIFIER_POSTGRES_DSN='postgres://hoctapaz:[email protected]:15440/hoctapaz_ai_classifier_db?sslmode=disable' \
  GOTOOLCHAIN=go1.25.11 \
  go test ./services/ai-classifier-service/internal/repository \
  -run TestPostgresGenerationSettlementClaimPersistsAcrossStores -count=1 -v

Deployment Configuration

deploy/docker-compose.yml, Helm values, static K8s, and the service env example declare the same owner URLs, charging defaults, and worker controls. Both AI_GENERATION_USAGE_WALLET_ENABLED and AI_GENERATION_SETTLEMENT_WORKER_ENABLED remain false by default. A charged environment must turn on both flags only after its AI migration 000006 and the usage/wallet owner URLs are available; enabling the worker alone is a safe no-op because no settlement client is constructed.

make test-ai-generation-settlement-reconciliation includes a static deploy guard so future manifest changes cannot drop the worker controls or owner URLs.

Hermetic HTTP Fault Injection (2026-07-14)

make test-ai-generation-settlement-fault-injection sends one generation preview through the real AI HTTP handler and HTTP monetization client. Its usage fixture returns 503 for credits_weekly after the wallet debit and three usage mutations. The production worker loop then retries settlement with the stored provider result and original owner idempotency keys. The test proves one provider call, four quota preflight checks, two wallet ledger calls but one effective debit, eight usage-consume calls but four effective mutations, and a final HTTP replay that makes no further owner call.

Fixtures are in-process contract-shaped HTTP owners with memory-backed AI state. This proves the AI retry boundary, not real usage/wallet binaries or stores, a process restart, provider credentials, Gateway behavior, or route promotion.

Isolated Result (2026-07-14)

An isolated postgres:16 container received AI migrations 000001 through 000006. The test saved a pending replay with an expired preview TTL, reopened it through a new store instance, acquired one worker lease, rejected a second claim, then persisted retry time and error. The container was removed after the test.

The focused worker regression separately injects a partial owner sequence: the first settlement call has one effective wallet debit and only part of usage consumption, then the worker completes forward using the original idempotency keys. It observes one provider and preflight call, never a second provider run.

Boundaries

• usage-service remains the quota/counter owner.
• wallet-service remains the balance and immutable-ledger owner.
• ai-classifier-service owns only provider result replay, worker leases, and provider usage telemetry.
• Public AI routes, deploy/gateway/routes.json, provider credentials, and production-like data are unchanged.

Remaining Proof

The charged path remains opt-in. The local-Postgres or disposable-Compose cross-service runner must be rerun successfully against real usage/wallet owner APIs and stores whenever its inputs change. Gateway rollback and browser parity are still required before any default-route promotion review.