Data quality bar for pilot (T1.10)
Status: locked for test (implement agent attestation 2026-08-12)
Purpose: Numeric DQ checklist that gates Simiaroom pilot go-live. Every target is measurable on the T2.11 /dq dashboard.
Related: LIVE-VS-PERIODIC-FIELDS.md (T1.9 class SLOs), VOLUME-AND-RETENTION.md (T0.7 volume), SDK-AND-DATA-INGESTION.md §11 / §13, ops/COLLECTOR-RUNBOOK.md
This document is the source of truth for T1.10 numbers. Ingestion §13 suggested the same bar; T2.2c PILOT_DQ_TARGETS and T2.11 DQ_DASHBOARD_TARGETS apply it. Do not invent a second set of thresholds.
1. Scope
| In scope | Out of scope (explicit) |
|---|---|
| Evented P0 authoritative outcomes (T1.9 class) | Import weekly cadence as a go-live fail (T1.9: near-daily / weekly is the class contract) |
| Completeness vs producer-of-record source rows | CDC replica completeness until inventory #9 is granted (T2.2c residual) |
Ingest accept latency + event lag (received_at − occurred_at) |
Dashboard query p95 (VOLUME-AND-RETENTION §6 / T3.11) |
| Duplicate rate after canonicalization | Collector duplicate counter alone (retries are success; see §4) |
| Quarantine / reject visibility (no silent drop) | Zero rejects forever (instrumentation noise is expected) |
| T2.11 widgets: missingness, lag, rejects, duplicates, queue/retry | T2.5 session-orphan widget (additive; not this gate) |
| T5.1 Data Health scorecards (freshness/volume/schema/identity/derived overview) | |
| Collector availability ≥ 99.5% (ingestion §13 ops SLO; not a T2.11 field) | |
Staff auth on /dq (T2.10 / T5.5) |
P0 outcomes (completeness cohort): T2.2c p0_backend_slice — the 12 canonical types in docs/fixtures/staging/p0-backend-slice.json:
authentication_succeeded, assessment_completed, therapist_selected, booking_confirmed, booking_cancelled, payment_succeeded, payment_failed, session_completed, session_no_show_recorded, therapy_pause_declared, treatment_completed, internal_case_resolved.
Booking / payment / session facts stay evented. Feedback import emptiness does not fail this bar (empty.feedback = true is allowed on Client 360).
2. Completeness target for P0 outcomes
| Item | Value |
|---|---|
| Target | ≥ 99% matched vs expected source records |
| Formula | completeness_pct = 100 × matched_count / expected_count (packages/reconcile/src/metrics.ts) |
| Cohort | p0_backend_slice (section 1) |
| Match | Same source (producer of record) + source_event_id + canonical_type in Orbit canonical store |
| T2.11 widget | missingness |
| T2.11 fields | report.completeness_pct, report.matched_count, report.expected_count, report.mismatches[] where reason = missing_in_orbit |
| Pass | completeness_pct ≥ 99 |
| Fail go-live | completeness_pct < 99 (widget fail) |
One miss in the 12-event staging slice is 91.67% and fails (T2.11 fixture missing). Alertable mismatches (missing_in_orbit, missing_in_source, source_mismatch, source_event_id_mismatch) also set reconcile within_dq_target = false.
Why 99% (not 100%, not 95%): the P0 decision loop (journey, retention, at-risk, Client 360) cannot tolerate a 5% hole in bookings, payments, or sessions. 100% is not the pilot bar while CDC is blocked and producers are still instrumenting — 99% is the floor. At T0.7 Scenario A mid (~5,000 events/day) that allows on the order of 50 unmatched source rows/day before fail; the staging slice is stricter (12/12).
3. Max lag / freshness target
T1.9 class contracts stay in force. T1.10 tightens evented ingest into numbers T2.11 already scores.
| Class | T1.9 contract | T1.10 numeric gate | On T2.11? |
|---|---|---|---|
| evented | Same-day for P0 authoritative outcomes | (a) collector accept p95 < 5 s (v0 uses max as a conservative proxy) and (b) event lag < 24 h | Yes — lag widget |
| import | Near-daily; shipped ops_support_ticket weekly |
No go-live fail. Late-event flag default 7 d (MVP-EVENT-SCOPE.md). Empty feedback ≠ trigger. | No — not the v0 lag widget |
| manual | Immediate on staff save | Not ingest lag | No |
| Item | Value |
|---|---|
| Accept latency | metrics.latency_ms.max ≤ 5_000 ms (proxy for §13 p95 < 5 s) |
| Event lag | metrics.event_lag_ms.max ≤ 86_400_000 ms (received_at − occurred_at) |
| Late counter | metrics.late_events = count with lag > 24 h (diagnostic; max already fails the widget) |
| T2.11 widget | lag |
| T2.11 fields | metrics.latency_ms.{max,count,sum,last}, metrics.event_lag_ms.{max,count,last}, metrics.late_events |
| Pass | ingest max ≤ 5 s and event-lag max ≤ 24 h |
| Fail go-live | either threshold breached (T2.11 fixture lag: ingest 8 s / event 25 h) |
Late events are stored, not dropped. Freshness is measured on timestamps, not by rewriting occurred_at (DATA-MODEL.md §9, T1.9 §2).
Serving freshness (profile/risk minutes after SYS-01; screen p95 2–5 s) is not this gate.
4. Duplicate rate target
| Item | Value |
|---|---|
| Target | ≤ 0.1% after canonicalization |
| Formula | duplicate_rate_pct = 100 × max(0, raw_count − canonical_count) / raw_count |
| Meaning | Extra raw rows that did not create a new canonical fact (transport/canonical leak). Correct outcome: duplicate retries must not write a second raw or canonical row. |
| T2.11 widget | duplicates |
| T2.11 fields (gate) | report.duplicate_rate_pct, report.raw_count, report.observed_count |
| T2.11 fields (fallback) | If no report: metrics.duplicate / (accepted + duplicate + rejected) — still compared to 0.1% |
| Diagnostic (not the gate) | metrics.duplicate — at-least-once retries that the collector classified as success |
| Pass | duplicate_rate_pct ≤ 0.1 |
| Fail go-live | duplicate_rate_pct > 0.1 (T2.11 fixture dup: 50%) |
At-least-once delivery may retry HTTP; that is expected (TECHNICAL-ARCHITECTURE.md §5). The bar is no double-counted P0 outcomes, not “zero retries.” T0.7 notes retries may temporarily double accepted writes; after working dedup (project_id + source + source_event_id) the canonical rate stays ~0%. 0.1% of ~5,000 events/day is on the order of 5 leaked raw rows/day.
5. Quarantine / reject handling expectations
Contract: SDK-AND-DATA-INGESTION.md §11. Operator steps: ops/COLLECTOR-RUNBOOK.md §2. No silent drop.
| Expectation | Numeric / rule | T2.11 field |
|---|---|---|
| Invalid P0 items are visible | Within 15 minutes of ingest | rejects widget; /dq polls GET /metrics every 30 s (≪ 15 min) |
| Counters | rejected, quarantined (quarantined ≤ rejected) |
metrics.rejected, metrics.quarantined |
| Quarantine queue | Pending / replayed / discarded visible | metrics.quarantine.{pending,replayed,discarded} and queue_retry widget |
| Rate-limit / retry pressure | Visible, not hidden | metrics.rate_limited, metrics.auth_failures |
| Schema-invalid / privacy-invalid / conflict | Not auto-retried; fix producer, then replay from quarantine if eligible | §11.7; SDKs do not call replay |
| Retention | Quarantine 90 days (T0.7) | Not a dashboard gauge |
| Import CSV bad rows | import_row quarantined + error_reason; replay = new batch, collector-only emit |
CSV-IMPORT.md — not the T2.11 event widgets |
Widget status (locked): rejects and queue_retry are warn when counts are non-zero, pass when zero, unknown when /metrics is missing. They do not auto-fail the pilot gate. A non-zero reject count during instrumentation is expected. Unknown rejects cannot go-live (cannot prove “no silent drop”).
If invalid P0 outcomes never reach canonical, completeness (§2) fails the gate. Visibility without completeness is not enough.
6. Pilot gate: what fails go-live
Surface: /dq Pilot gate (T1.10) and GET /api/dq → view.pilotGate / view.pilotGateDetail.
Implementation: pilotGate() in apps/web/src/lib/dq/view-model.ts.
6.1 FAIL (do not go live)
Any of these T2.11 widgets is fail:
| Widget | Breach |
|---|---|
missingness |
Completeness < 99% |
duplicates |
Duplicate rate > 0.1% after canonicalization |
lag |
Accept max > 5 s or event lag max > 24 h |
6.2 UNKNOWN (do not go live)
Missing data on missingness, lag, duplicates, or rejects. Unknown cannot pass (no green-wash).
6.3 PASS (go-live allowed on this bar)
missingness,duplicates, andlagarepassrejectsispassorwarn(visible), notunknown
6.4 Does **not** fail this gate
| Condition | Why |
|---|---|
rejects / queue_retry = warn |
Visibility SLO, not a zero-reject SLO |
| Import feed not yet weekly | T1.9 class; empty feedback allowed |
session_orphans fail/warn |
T2.5 additive widget |
| T5.1 scorecard degraded | Later overview; not T1.10 |
| Durable SDK outbox depth unknown | T2.14 residual; v0 shows collector 429s + quarantine pending |
T4.1 applies this same FAIL/UNKNOWN/PASS table to the live pilot window.
7. T2.11 field map (Test Agent)
Every DoD numeric target has a dashboard field. Prove with fixtures (no live Simiaroom required):
| DoD item | Target | Widget id |
JSON path (/api/dq) |
Fixture that moves it |
|---|---|---|---|---|
| Completeness | ≥ 99% | missingness |
view.widgets[id=missingness].actual ← report.completeness_pct |
?fixture=missing → 91.67% fail |
| Max lag / freshness | ingest < 5 s and event lag < 24 h | lag |
metrics.latency_ms.max, metrics.event_lag_ms.max |
?fixture=lag → 8 s / 25 h fail |
| Duplicate rate | ≤ 0.1% | duplicates |
report.duplicate_rate_pct |
?fixture=dup → 50% fail |
| Quarantine / reject | visible within 15 min; no silent drop | rejects, queue_retry |
metrics.rejected, metrics.quarantined, metrics.quarantine.* |
?fixture=reject → warn (gate still pass if others pass) |
| Pilot gate | §6 | view.pilotGate |
view.pilotGate, view.pilotGateDetail |
healthy → pass; missing / dup / lag → fail |
Constants in code (must match this doc):
| Constant | Value | Path |
|---|---|---|
completenessMinPct |
99 | packages/reconcile/src/dq-targets.ts |
duplicateRateMaxPct |
0.1 | same |
ingestLatencyP95MaxMs |
5000 | apps/web/src/lib/dq/targets.ts |
eventLagMaxMs |
86400000 | same |
invalidVisibleWithinMinutes |
15 | same |
DQ_REFRESH_SECONDS |
30 | same |
8. Achievability (Approve Agent)
Ambitious for a decision-loop pilot; sized to T0.7 Scenario A (MVP P0 backend-only, 10⁴ events/day, mid ~3,000–6,000/day, peak average 0.07–0.19 events/s, bursts 10–40 events/s).
| Target | Why it is achievable at Scenario A | Why it is not soft |
|---|---|---|
| Completeness ≥ 99% | Backend SDK + outbox, collector-only writes, 12-type slice with producer-of-record keys. Staging already hits 100% on the fixture cohort (T2.2c). | 95% would hide missing bookings/payments/sessions on at-risk / 360. |
| Event lag < 24 h | T1.9 same-day evented contract; collector HTTP accept is seconds. | “Near-daily” is the import class, not P0 outcomes. |
| Accept p95 < 5 s | Ingest rate is ≪ 1 event/s average; 5 s is collector accept, not screen p95. v0 max proxy is stricter than true p95. | Still a hard fail on /dq if max > 5 s. |
| Dup ≤ 0.1% after canonicalization | Dedup keys are specified (T0.4 / T1.13) and proven on the slice (T2.2c). Retries that return duplicate do not inflate the rate. |
1% would allow double-counted sessions in retention. |
| Invalid visible ≤ 15 min | 30 s metrics poll. | Does not require zero rejects. |
Re-measure and publish volume_v2 in T4.1 / T4.12 if live Simiaroom traffic leaves the Scenario A band. Do not loosen this bar in T1.10 to chase a pass.
9. Out of scope (explicit)
| Item | Why deferred |
|---|---|
| Histogram p95 for ingest latency | T2.11 residual: v0 uses latency_ms.max |
Postgres source_reconciliation_run as the dashboard source |
v0 reads RECONCILE_REPORT_PATH JSON |
| CDC completeness vs live OLTP | Inventory #9 blocked |
| T5.1 scorecards as go-live | Separate task; this bar is T2.11 v0 widgets |
| Session orphan ≤ 5% | T2.5; widget exists, not in pilotGate() |
| Named data/eng sign-off | Same pattern as T1.9; Test/Approve against the checklist |
10. Sign-off
| Role | Name | Date | Status |
|---|---|---|---|
| Data lead (cx-orbit-data) | TBD — stakeholder sync | — | Pending named signatory |
| Eng lead (CX Orbit technical lead) | TBD — stakeholder sync | — | Pending named signatory |
| Implement agent (T1.10 deliverable) | cx-orbit-implement-task | 2026-08-12 | Checklist complete; numbers match T2.11 fields; ready for Test Agent |
References
- LIVE-VS-PERIODIC-FIELDS.md — class SLOs this bar tightens
- VOLUME-AND-RETENTION.md — Scenario A volume used for achievability
- SDK-AND-DATA-INGESTION.md §11 (quarantine) and §13 (same numbers)
- MVP-EVENT-SCOPE.md — 24 h P0 late threshold; 7 d import/CDC
- SOURCE-EVENT-MAPPING.md — producer of record / slice types
- ops/COLLECTOR-RUNBOOK.md — on-call for rejects / dupes / lag
- docs/fixtures/dq/README.md — T2.11 inject fixtures
- TECHNICAL-TASKS.md T1.10 / T2.11 / T4.1