Appearance
Monetization Broker Transport Task
Agent workflow: follow README.md for Audit -> Investigate -> Code -> Test -> Fix; keep this pack's transport decision and verification commands authoritative.
Dispatch type: orchestrator
Owner: orchestrator with payment-service, billing-service, wallet-service, usage-service, deployment, and observability owners.
Problem:
The first monetization wave has replay-safe service foundations and durable payment/billing outboxes, but the current event delivery path is an internal HTTP sink. Payment, billing, wallet, and usage task packs all state that the HTTP sink must be replaced by the selected shared broker transport before route promotion. Without a dedicated handoff, each service agent may implement a different subscriber shape or mistake the rehearsal HTTP endpoints for the target architecture.
Goal:
Select and implement the shared broker transport for monetization events while preserving the existing outbox tables as the source of retry, replay, dead-letter, and operator evidence. Public payment, wallet, subscription, and quota routes stay legacy-proxied until broker proof, gateway rehearsal, browser/runtime parity, and rollback evidence exist.
Legacy source evidence:
/Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/wallet/**/Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/src/modules/payment/**/Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/api/prisma/schema.prisma- monetization frontend callers under
/Users/velikho/Desktop/WORKING/HOCTAPAZ/node-platform/apps/web - shared native event contracts in
contracts/events/README.md
Writable files:
services/payment-service/**services/billing-service/**services/wallet-service/**services/usage-service/**contracts/openapi/services/payment-service.yamlcontracts/openapi/services/billing-service.yamlcontracts/openapi/services/wallet-service.yamlcontracts/openapi/services/usage-service.yamldocs/api/payment-service.mddocs/api/billing-service.mddocs/api/wallet-service.mddocs/api/usage-service.mddocs/agents/service-tasks/monetization-broker-transport.mddocs/agents/service-tasks/monetization-scaffold-wave.md
Orchestrator-owned files:
- Broker ADR, shared transport package, Compose/Helm/K8s manifests, root
Makefile, migration/worker jobs, observability dashboards, VitePress navigation, gateway route tables, and browser QA artifacts unless explicitly delegated.
Transport contract:
- Selected transport: NATS JetStream, accepted in
docs/architecture/decision-log.mdas D-017. - Stream:
MONETIZATION. - Subjects:
monetization.payment.order.paid.v1,monetization.billing.entitlements.updated.v1, andmonetization.billing.subscription.activated.v1. - Keep existing service-owned outbox rows authoritative. Broker publishing is an outbox dispatcher mode, not a replacement for
pending,delivered,retry, anddeadstate. - Keep HTTP event sinks available only as rehearsal or rollback fallback while broker rollout is staged. Do not make the HTTP sink the final target architecture.
- Use stable event names and versioned payloads:
payment.order.paid.v1,billing.entitlements.updated.v1, andbilling.subscription.activated.v1. - Payment and billing publishers are opt-in with
MONETIZATION_EVENT_TRANSPORT=nats(or service-specificPAYMENT_EVENT_TRANSPORT/BILLING_EVENT_TRANSPORT). The default remainshttpuntil broker subscriber proof and route rollback evidence exist. - Use
sourceServiceplussourceEventIdidempotency at every consumer. Broker redelivery, HTTP fallback replay, and manual replay must all converge on the same processed-event rows and ledger/subscription/entitlement outcomes.
Service handoff:
payment-servicepublishespayment.order.paid.v1from its durable payment outbox to billing and wallet consumers. It must not activate subscriptions or mutate wallet ledgers directly.billing-serviceconsumespayment.order.paid.v1, activates subscriptions idempotently by source payment event id, and publishesbilling.entitlements.updated.v1plusbilling.subscription.activated.v1from its billing outbox.usage-serviceconsumesbilling.entitlements.updated.v1, rejects events withoutsourceSubscriptionEventId, ignores stale snapshots, and must not move wallet credits or call payment providers.wallet-serviceconsumespayment.order.paid.v1for wallet top-up credits andbilling.subscription.activated.v1for subscription credits. Positive subscription credits still requiresourcePaymentEventId.
Tasks:
- [x] Write or link the broker ADR that selects transport, local/offline deployment topology, subject names, retention, dead-letter policy, retry/backoff, and observability. D-017 selects NATS JetStream and the Compose/K8s/Helm runtime now enables JetStream file storage.
- [x] Add a payment outbox publisher adapter for the selected broker while preserving the existing internal HTTP sink as an explicit fallback mode.
services/payment-service/internal/eventfanout/nats_publisher.gopublishespayment.order.paid.v1envelopes tomonetization.payment.order.paid.v1through NATS JetStream with the payment outbox event id as the broker message id. - [x] Add a billing outbox publisher adapter for the selected broker while preserving existing usage/wallet HTTP sink fallback.
services/billing-service/internal/eventfanout/nats_publisher.gopublishesbilling.entitlements.updated.v1andbilling.subscription.activated.v1envelopes to their monetization subjects through NATS JetStream with the billing outbox event id as the broker message id. - [x] Add billing, wallet, and usage broker subscribers that reuse the same consumer usecases as the current HTTP event endpoints.
billing-serviceconsumesmonetization.payment.order.paid.v1,wallet-serviceconsumesmonetization.payment.order.paid.v1andmonetization.billing.subscription.activated.v1, andusage-serviceconsumesmonetization.billing.entitlements.updated.v1. The subscribers are opt-in with the same staged NATS transport mode, ack only after successful/ignored/replay handling, and keep HTTP endpoints as rollback fallback. - [x] Add replay tooling for broker messages by event id and for outbox rows by outbox id.
tools/monetization-outboxexposesinspect,list,list-dead, andreplay/retryfor payment and billing outboxes. The outbox id, event id, and JetStream message id are the same value, so operator replay resets the service-owned outbox row toPENDINGand lets the existing dispatcher republish with the same message id. Replay must not duplicate subscription activation, entitlement snapshots, wallet top-up credits, or subscription-credit ledger entries. - [x] Add dead-letter inspection and retry commands or runbook steps for both payment and billing outboxes. Use
go run ./tools/monetization-outbox/cmd/monetization-outbox list-dead --service payment|billing, theninspect --event-id <id>, andreplay --event-id <id> --reason "<why>" --confirmagainst the owning service DSN. - [x] Add local Compose and offline K8s wiring for the selected broker without using shared bootstrap database credentials or cross-service DB reads. D-017 plus P16-006 keeps JetStream storage in Compose, static K8s, and Helm; the operator CLI also takes only the selected service DSN.
- [x] Add an opt-in live broker smoke that publishes disposable paid and entitlement events through the broker, verifies payment-to-billing/wallet and billing-to-usage/wallet outcomes, records outbox delivery state, and tears down disposable data where possible.
scripts/test/monetization-broker-live-smoke.shruns only withMONETIZATION_EVENT_TRANSPORT=natsandMONETIZATION_BROKER_SMOKE_CONFIRM=broker-runtime; the cheap self-test is wired intomake test-monetization-broker-transport, and real runtime artifacts are expected underoutput/qa/monetization-broker-live-smoke-*. Fresh local broker proof was captured atoutput/qa/monetization-broker-live-smoke-20260714-075522/with delivered payment eventspayevt_b5389ccdbae67b1c025eb635andpayevt_b0f218cd5ed29effc7b3aef8, delivered billing eventsbevt_8d717d451add8a76714f0108andbevt_8d717d451add8a76714f0108_entitlements, usagecredits_monthlyallowed at limit1200, wallet balance1977, and a duplicate subscription webhook returningreplay=truewithout another payment transaction, billing event pair, or subscription credit grant. - [ ] Keep default public route tables legacy until broker proof is paired with route rehearsal, browser/runtime evidence, and rollback notes. The wallet-owned admin review read route now has a non-default gateway smoke harness in
scripts/test/wallet-admin-review-live-smoke.sh; this is route evidence only and does not promote public wallet, payment, subscription, or quota routes.
Acceptance:
- Broker transport is selected explicitly; service agents do not invent service-local broker choices.
- Existing HTTP event ingestion routes remain native-only rehearsal/fallback surfaces and do not imply public route promotion.
- Service-owned outboxes remain the retry/dead-letter source of truth even when delivery uses the broker.
- No monetization service reads another service database.
- Event delivery is replay-safe across broker redelivery, HTTP fallback, and manual outbox replay.
payment.order.paid.v1reaches billing and wallet; billing then emitsbilling.entitlements.updated.v1to usage andbilling.subscription.activated.v1to wallet.- Zero-price
FREE_TRIALactivation still emits entitlement updates but no subscription-credit wallet grant without payment evidence. - Broker deployment has Compose, offline K8s, observability, and rollback notes before any default payment, wallet, subscription, or quota route promotion.
Verification:
make test-monetization-event-chainmake test-monetization-routesmake test-monetization-broker-transportMONETIZATION_BROKER_SMOKE_SELF_TEST=1 bash scripts/test/monetization-broker-live-smoke.shGOTOOLCHAIN=go1.25.11 go test ./tools/monetization-outbox/... -count=1GOTOOLCHAIN=go1.25.11 go test ./services/payment-service/... ./services/billing-service/... ./services/wallet-service/... ./services/usage-service/... -count=1GOTOOLCHAIN=go1.25.11 go test ./services/payment-service/internal/eventfanout ./services/billing-service/internal/eventfanout ./services/wallet-service/internal/eventfanout ./services/usage-service/internal/eventfanout -count=1- Broker ADR/static config validation for the selected transport.
- Broker smoke target after implementation:
MONETIZATION_EVENT_TRANSPORT=nats MONETIZATION_BROKER_SMOKE_CONFIRM=broker-runtime make test-monetization-broker-live. - Latest local broker smoke artifact:
output/qa/monetization-broker-live-smoke-20260714-075522/summary.json. - OpenAPI YAML parse/diff for the four service contracts.
docker compose -f deploy/docker-compose.yml config --quiet- Helm render/lint and offline K8s manifest validation for the broker runtime and monetization workers.