Transactional Outbox Relay

4 min read•docs/ops/OUTBOX-RELAY.md•

Outbox relay — delivery and replay (T2.14)

Producer-side L1 transactional outbox for agreed P0 outcomes. Complements L2 sync HTTP (@cx-orbit/node in-memory queue) and L3 CDC reconciliation (T2.2c). Decision: ADR 0002.

Non-negotiable: the collector is the only event write ingress. The outbox lives in Simiaroom (or apps/producer-stub). Relays POST Authorization: Bearer to POST /v1/events:batch. Never @cx-orbit/db, never Orbit DATABASE_URL.

1. Agreed P0 set

The 12 T1.12 authoritative backend types (AGREED_P0_CANONICAL_TYPES in @cx-orbit/outbox):

authentication_succeeded, assessment_completed, therapist_selected, booking_confirmed, booking_cancelled, payment_succeeded, payment_failed, session_completed, session_no_show_recorded, therapy_pause_declared, treatment_completed, internal_case_resolved.

These must not rely on the in-memory SDK queue for durability. Non-P0 / generic track() may still use L2 and can drop after the 15-minute retry budget (stats().dropped).

2. Write path (same transaction as the business commit)

BEGIN
  UPDATE bookings SET status = 'confirmed' WHERE id = …
  INSERT INTO cx_outbox_events (event_id, envelope, …)  -- pending
COMMIT
-- tracking failure must not roll back the booking

DDL: packages/outbox/sql/cx_outbox_events.postgres.sql.

Parameterized statements: POSTGRES_STATEMENTS in @cx-orbit/outbox.

Local / stub stand-in: FileOutboxStore + createOutboxProducer (atomic JSON file). withTransaction rolls back the file snapshot if the callback throws — analogue of “no outbox row if the business TX fails”.

Preserve a stable event_id from insert through every relay attempt (T0.4 transport key: project_id + event_id).

3. Delivery (relay)

relayOnce (CLI: pnpm --filter @cx-orbit/outbox relay -- --store <path>):

  1. Claim pending rows (or delivering whose lease expired — crash safety, at-least-once).
  2. POST the envelopes to the collector via @cx-orbit/node postBatch.
  3. Per item:
    Collector outcome Outbox action
    accepted / accepted_with_warnings delivered
    duplicate (DUPLICATE_EVENT_ID or DUPLICATE_CANONICAL_KEY) delivered (success — not loss)
    Transient: network, 408, 429, 502, 503, 504 stay pending, backoff, same event_id
    Permanent: 400, 401, 403, 422, schema/quarantine retryable: false dead_letter + alert; do not retry

Unlike the in-memory SDK, the outbox does not drop P0 rows after 10 attempts or 15 minutes. They stay pending until delivered or dead-lettered.

Backoff: same as SDK (1s × 2^(n-1), full jitter, 60s cap), applied to next_attempt_at. Honor Retry-After when the collector sends it on a request-error retry.

4. Collector-outage drill (Test Agent)

# 1. Collector is stopped (do not start it)

# 2. Produce outbox events — exit 0, pending = 12
set -a && source .env && set +a
pnpm --filter @cx-orbit/producer-stub emit -- --outbox ./cx-outbox.json

# 3. Start collector
pnpm --filter @cx-orbit/collector dev

# 4. Relay → delivered
pnpm --filter @cx-orbit/outbox relay -- --store ./cx-outbox.json

# 5. Inspect — 12 canonical, no loss
curl -sS http://127.0.0.1:8080/v1/debug/store \
  -H "Authorization: Bearer $COLLECTOR_SERVER_WRITE_KEY" \
  | jq '{raw_count, canonical_count, types: [.events[].canonical_type]}'

Automated equivalent (no live process):

pnpm --filter @cx-orbit/outbox test
pnpm --filter @cx-orbit/producer-stub test

src/outage.test.ts stops the collector (fetch throws), enqueues all 12, reopens the file store (process crash), restarts collector, relays, asserts 12 canonical facts and 0 pending.

5. Replay and dedup (T0.4 / T2.2c intact)

At-least-once end-to-end. No exactly-once promise.

Layer Key Replay of the same fact
Transport project_id + event_id DUPLICATE_EVENT_ID — outbox marks delivered
Canonical project_id + source + source_event_id DUPLICATE_CANONICAL_KEY — one canonical row

Relay of an already-accepted envelope is success, not loss. Do not rotate event_id on retry. CDC / import copies of the same source fact must share source + source_event_id (they may use a new transport event_id).

Dead-letter rows are not auto-replayed. Fix payload/config, then insert a new pending row (new or same event_id after the bad row is removed) and relay.

6. Ops signals

Signal Meaning
pending / oldest_pending_at growing Collector down or relay not running — page Simiaroom + Orbit
dead_letter Permanent reject — do not retry blindly
delivering stuck past lease Relay crash; next tick reclaims (at-least-once duplicate POST is OK)
Collector /health ok + pending > 0 Run relay

In-memory collector store still resets on collector process restart. Outbox replay after that restart re-delivers; expect DUPLICATE_* only if the collector process kept the first accept. A wiped in-memory collector + outbox replay is a clean re-accept (same event_id).

7. Residual

  • Simiaroom product DB wiring (same-TX INSERT) is in the Simiaroom repo; this workspace ships DDL + file-store stand-in + relay.
  • L3 CDC remains the catch-up for missed instrumentation, not a substitute for L1 on new P0 writes.
  • Collector HTTP persist to Postgres is still not wired (T4.12 residual).