#Collector operations runbook

On-call steps for CX Orbit ingest: rejects, duplicates, lag, and collector outage.

Companion: ../INTEGRATION-GUIDE.md (instrument + staging validate). Wire contract: ../SDK-AND-DATA-INGESTION.md §11. Failure ownership: ../adr/0002-collector-sdk-outbox.md §5.

Rule: no silent drop. Every processed batch item has a results[] row. Invalid items are quarantined when they can be sanitized. Duplicates are success, not loss.

#0. Surfaces (use these, not tribal knowledge)

Local defaults: collector http://127.0.0.1:8080, web http://localhost:3000. Staging: $COLLECTOR_PUBLIC_URL, $WEB_PUBLIC_URL (../deploy/railway-staging.md).

/dq polls /metrics every 30 seconds. Completeness and canonical duplicate rate come from the last reconcile report JSON (RECONCILE_REPORT_PATH, default docs/fixtures/dq/last-reconcile-report.json).

Pilot targets (fail the gate if breached) — source of truth DATA-QUALITY-BAR.md (T1.10):

#1. First 2 minutes (any ingest page)

set -a && source .env && set +a
COLLECTOR="${COLLECTOR_URL:-http://127.0.0.1:8080}"
WEB="${WEB_PUBLIC_URL:-http://localhost:3000}"

curl -sS -o /tmp/col-health.json -w "collector_http=%{http_code}\n" "$COLLECTOR/health"
curl -sS -o /tmp/web-health.json -w "web_http=%{http_code}\n" "$WEB/api/health"
curl -sS "$COLLECTOR/metrics" | jq '{
  accepted, rejected, quarantined, duplicate,
  rate_limited, auth_failures, late_events,
  latency_ms, event_lag_ms, quarantine
}'
curl -sS "$WEB/api/dq" | jq '{pilotGate: .view.pilotGate, widgets: [.view.widgets[] | {id, actual, status}]}'

Fixture drill (widgets move without live traffic): /dq?fixture=reject, dup, lag, missing.

#2. Troubleshoot rejects / quarantine

Rejects are permanent unless retryable: true (item-level transients are not used in MVP). Do not hammer the collector with the same bad payload.

#2.1 Get the code

From the producer flush() / HTTP body results[]:

#2.2 Confirm visibility (15-minute SLO)

curl -sS "$COLLECTOR/metrics" | jq '{rejected, quarantined, quarantine}'
# /dq rejects widget: actual "rejected N; quarantined M", status warn if N or M > 0

There is no public quarantine-payload HTTP inspector. Diagnosis is:

1. Ingest results[] (code, path, message, quarantine_id) — messages contain no PII.
2. /metrics + /dq counters.
3. Collector process logs (Railway: service collector).

Do not paste write keys, payloads with emails/phones, or therapy content into tickets.

#2.3 Replay after a schema/producer fix

Replay is an ops action. SDKs must not auto-retry quarantined schema errors.

1. Fix producer, schema, or decide discard
2. Re-POST the corrected envelope to POST /v1/events:batch
   (same write key, validation, privacy, dedup as live traffic)
3. Keep original event_id and source_event_id when those were valid
4. Success → quarantine.replayed increments; original row is not deleted
5. IDEMPOTENCY_CONFLICT as-is is refused — never replay to overwrite

Covered by collector contract tests CT-17 / CT-18 (pnpm --filter @cx-orbit/collector test).

#2.4 Rate limit (HTTP 429)

Default 120 requests/minute/write key (COLLECTOR_RATE_LIMIT_PER_MINUTE). Response includes Retry-After and retryable: true. SDK honors this. /dq queue widget shows rate_limited.

If 429s persist: split batches (max 100 events / 256 KB), slow the outbox relay, check for a retry storm of already-accepted items (safe but noisy — filter accepted/duplicate from the queue).

#3. Troubleshoot duplicates

Duplicates on ingest are success (outcome: duplicate). Clients must not retry them and must not page as “data loss.”

When duplicates are a problem: /dq duplicates widget fail, or reconcile duplicate_rate_pct > 0.1%. That means extra canonical facts (or a bad report), not healthy transport retries.

# Collector transport/canonical duplicate counter (process lifetime)
curl -sS "$COLLECTOR/metrics" | jq '.duplicate'

# Canonical vs source (needs events in this collector process)
set -a && source .env && set +a
# Script name is `run`; `pnpm … run` only lists scripts (exit 0). Use `run run`.
pnpm --filter @cx-orbit/reconcile-job run run
# inspect duplicate_rate_pct, raw_count, observed_count, mismatches

Reproduce locally:

# emit twice against the same collector process
pnpm --filter @cx-orbit/producer-stub emit
pnpm --filter @cx-orbit/producer-stub emit
# second flush: duplicate ≈ 12, accepted ≈ 0; debug store canonical_count still 12

If IDEMPOTENCY_CONFLICT appears: two producers (SDK + CDC, or confirm vs cancel) reused source_event_id with different material fields. Fix source_event_id construction (SDK-AND-DATA-INGESTION.md §7.4). Do not delete the first canonical row.

Missingness (completeness < 99%, missing_in_orbit): producer never emitted, wrong source/source_event_id, collector restarted (in-memory store empty), or SDK pointed at another environment. Re-emit from stub/outbox; re-run reconcile.

#4. Troubleshoot lag

Two clocks:

Late but valid events are stored (accepted_with_warnings / LATE_EVENT) — not dropped.

curl -sS "$COLLECTOR/metrics" | jq '{latency_ms, event_lag_ms, late_events}'

Staging fixtures in docs/fixtures/events/*/valid.json use 2026-08-03 timestamps; emitting them now will increment late_events. That is expected for those fixtures, not an outage.

#5. On-call: collector outage

Simiaroom booking / payment / session must keep working. Orbit ingest is best-effort after commit (ADR 0001 / 0002). Do not “fix” an outage by blocking checkout.

#5.1 Severity

#5.2 Confirm

COLLECTOR="${COLLECTOR_URL:-http://127.0.0.1:8080}"

curl -sS -w "\nhttp=%{http_code}\n" "$COLLECTOR/health"
# Staging Railway:
# curl -sS -w "\nhttp=%{http_code}\n" "$COLLECTOR_PUBLIC_URL/health"

# Optional: authenticated probe (expect 2xx or 422, not connection refused)
curl -sS -o /dev/null -w "batch_http=%{http_code}\n" -X POST "$COLLECTOR/v1/events:batch" \
  -H "Authorization: Bearer $COLLECTOR_SERVER_WRITE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"events":[]}'
# empty events → HTTP 400 BATCH_SCHEMA_INVALID if the process is up

HTTP 503 COLLECTOR_UNAVAILABLE is the load-shed / forced-unavailable shape (retryable: true). Drill locally:

COLLECTOR_FORCE_UNAVAILABLE=true pnpm --filter @cx-orbit/collector dev
# POST /v1/events:batch → 503, code COLLECTOR_UNAVAILABLE, retryable true

Railway collector: healthcheck path /health (apps/collector/railway.toml), restart policy ON_FAILURE. Check the collector service logs and last deploy. Web can be up while collector is down — /dq lag/rejects widgets go unknown.

#5.3 What Simiaroom should do (producer)

@cx-orbit/node flush() returns failed: true on transport failure — treat as visible, not success.

#5.4 What Orbit on-call should do

1. Do not tell Simiaroom to pause bookings.
2. Confirm Railway collector vs web vs Postgres independently (collector deploys alone).
3. If a bad deploy: roll back / redeploy collector only (../deploy/railway-staging.md).
4. If COLLECTOR_FORCE_UNAVAILABLE=true left on: unset and restart.
5. If write-key rotation mid-flight: 401s look like an outage to producers — publish the new key to Simiaroom staging/prod secrets before disabling the old one when possible.
6. Postgres down does not currently fail HTTP ingest (in-memory store). Still fix DB for web/jobs; do not assume ingest is durable across collector restart.

#5.5 After collector is healthy

curl -sS "$COLLECTOR/health" | jq .status   # ok

set -a && source .env && set +a
pnpm --filter @cx-orbit/producer-stub emit
# Script name is `run`; `pnpm … run` only lists scripts (exit 0). Use `run run`.
pnpm --filter @cx-orbit/reconcile-job run run
# expect within_dq_target true if this process received the slice

curl -sS "${WEB_PUBLIC_URL:-http://localhost:3000}/api/dq" | jq '.view.pilotGate'

Because the store is in-memory, a restart looks like total event loss until producers retry or you re-emit. That is expected for the collector process; it is not a reason to rebuild canonical facts by hand. Outbox relay (T2.14) is the durable catch-up path for agreed P0: OUTBOX-RELAY.md. CDC (L3) covers missed instrumentation.

If Simiaroom outbox replayed the same envelopes: expect DUPLICATE_EVENT_ID / DUPLICATE_CANONICAL_KEY — success.

#5.6 Escalate / residual

Page Orbit platform for collector 5xx; page Simiaroom eng for 4xx schema/auth and outbox depth. Joint triage for IDEMPOTENCY_CONFLICT and source-vs-Orbit count mismatch.