Outbox relay — delivery and replay (T2.14)
Producer-side L1 transactional outbox for agreed P0 outcomes. Complements L2 sync HTTP (@cx-orbit/node in-memory queue) and L3 CDC reconciliation (T2.2c). Decision: ADR 0002.
Non-negotiable: the collector is the only event write ingress. The outbox lives in Simiaroom (or apps/producer-stub). Relays POST Authorization: Bearer to POST /v1/events:batch. Never @cx-orbit/db, never Orbit DATABASE_URL.
1. Agreed P0 set
The 12 T1.12 authoritative backend types (AGREED_P0_CANONICAL_TYPES in @cx-orbit/outbox):
authentication_succeeded, assessment_completed, therapist_selected, booking_confirmed, booking_cancelled, payment_succeeded, payment_failed, session_completed, session_no_show_recorded, therapy_pause_declared, treatment_completed, internal_case_resolved.
These must not rely on the in-memory SDK queue for durability. Non-P0 / generic track() may still use L2 and can drop after the 15-minute retry budget (stats().dropped).
2. Write path (same transaction as the business commit)
BEGIN
UPDATE bookings SET status = 'confirmed' WHERE id = …
INSERT INTO cx_outbox_events (event_id, envelope, …) -- pending
COMMIT
-- tracking failure must not roll back the booking
DDL: packages/outbox/sql/cx_outbox_events.postgres.sql.
Parameterized statements: POSTGRES_STATEMENTS in @cx-orbit/outbox.
Local / stub stand-in: FileOutboxStore + createOutboxProducer (atomic JSON file). withTransaction rolls back the file snapshot if the callback throws — analogue of “no outbox row if the business TX fails”.
Preserve a stable event_id from insert through every relay attempt (T0.4 transport key: project_id + event_id).
3. Delivery (relay)
relayOnce (CLI: pnpm --filter @cx-orbit/outbox relay -- --store <path>):
- Claim pending rows (or
deliveringwhose lease expired — crash safety, at-least-once). - POST the envelopes to the collector via
@cx-orbit/nodepostBatch. - Per item:
Collector outcome Outbox action accepted/accepted_with_warningsdeliveredduplicate(DUPLICATE_EVENT_IDorDUPLICATE_CANONICAL_KEY)delivered(success — not loss)Transient: network, 408, 429, 502, 503, 504 stay pending, backoff, sameevent_idPermanent: 400, 401, 403, 422, schema/quarantine retryable: falsedead_letter+ alert; do not retry
Unlike the in-memory SDK, the outbox does not drop P0 rows after 10 attempts or 15 minutes. They stay pending until delivered or dead-lettered.
Backoff: same as SDK (1s × 2^(n-1), full jitter, 60s cap), applied to next_attempt_at. Honor Retry-After when the collector sends it on a request-error retry.
4. Collector-outage drill (Test Agent)
# 1. Collector is stopped (do not start it)
# 2. Produce outbox events — exit 0, pending = 12
set -a && source .env && set +a
pnpm --filter @cx-orbit/producer-stub emit -- --outbox ./cx-outbox.json
# 3. Start collector
pnpm --filter @cx-orbit/collector dev
# 4. Relay → delivered
pnpm --filter @cx-orbit/outbox relay -- --store ./cx-outbox.json
# 5. Inspect — 12 canonical, no loss
curl -sS http://127.0.0.1:8080/v1/debug/store \
-H "Authorization: Bearer $COLLECTOR_SERVER_WRITE_KEY" \
| jq '{raw_count, canonical_count, types: [.events[].canonical_type]}'
Automated equivalent (no live process):
pnpm --filter @cx-orbit/outbox test
pnpm --filter @cx-orbit/producer-stub test
src/outage.test.ts stops the collector (fetch throws), enqueues all 12, reopens the file store (process crash), restarts collector, relays, asserts 12 canonical facts and 0 pending.
5. Replay and dedup (T0.4 / T2.2c intact)
At-least-once end-to-end. No exactly-once promise.
| Layer | Key | Replay of the same fact |
|---|---|---|
| Transport | project_id + event_id |
DUPLICATE_EVENT_ID — outbox marks delivered |
| Canonical | project_id + source + source_event_id |
DUPLICATE_CANONICAL_KEY — one canonical row |
Relay of an already-accepted envelope is success, not loss. Do not rotate event_id on retry. CDC / import copies of the same source fact must share source + source_event_id (they may use a new transport event_id).
Dead-letter rows are not auto-replayed. Fix payload/config, then insert a new pending row (new or same event_id after the bad row is removed) and relay.
6. Ops signals
| Signal | Meaning |
|---|---|
pending / oldest_pending_at growing |
Collector down or relay not running — page Simiaroom + Orbit |
dead_letter |
Permanent reject — do not retry blindly |
delivering stuck past lease |
Relay crash; next tick reclaims (at-least-once duplicate POST is OK) |
Collector /health ok + pending > 0 |
Run relay |
In-memory collector store still resets on collector process restart. Outbox replay after that restart re-delivers; expect DUPLICATE_* only if the collector process kept the first accept. A wiped in-memory collector + outbox replay is a clean re-accept (same event_id).
7. Residual
- Simiaroom product DB wiring (same-TX
INSERT) is in the Simiaroom repo; this workspace ships DDL + file-store stand-in + relay. - L3 CDC remains the catch-up for missed instrumentation, not a substitute for L1 on new P0 writes.
- Collector HTTP persist to Postgres is still not wired (T4.12 residual).