# Measurement reliability

This page states what the score's precision actually is, including where it is not
good enough yet. It is published because a measurement product that hides its own
error bars is asking to be taken on faith, which is the opposite of the point.

## The measured spread

Three audits of the same brand — `brooklinen.com`, same code version, temperature 0,
run seconds apart — produced:

| Run | Total | Visibility | Invisible queries | Technical readiness | Coverage |
|---|---|---|---|---|---|
| 1 | **40** | 67 | 2 / 6 | 25 | 1.00 |
| 2 | **25** | 50 | 3 / 6 | 25 | 1.00 |
| 3 | **25** | 50 | 3 / 6 | 25 | 1.00 |

Range: **15 points**. Sample standard deviation ≈ 8.7 on n = 3, which is a small enough
sample that the true σ could plausibly sit anywhere in roughly [4.5, 54].

The cause is visible in the table: one shopper query flipped between visible and
invisible. At 15 points per invisible query, one flip *is* the entire spread.

This was later confirmed in production. On the first monthly Defend cycle, ten brands
were re-audited with no site changes; three moved by +15, +15, and −10.

## The noise floor

```
SCORE_NOISE_FLOOR = 15
```

A change of 15 points or less between two runs is reported as **not detectable**. The
comparison is inclusive — exactly 15 is treated as noise, not signal.

!!!danger The uncomfortable consequence
The smallest real single improvement is 10 points (fixing one missing signal) or 15
points (winning one query). Both sit at or below the noise floor. **No single fix is
currently detectable above the noise on its own.** Movement becomes meaningful when
several fixes land together, or across repeated runs.
!!!

This is a real limitation, not a caveat. It is why the product's unit of work is a
prioritized fix *list* rather than a single recommendation, and why Defend reruns
monthly rather than continuously.

## What is done about it

**Temperature 0 on every model call**, both the engine panel and the judge. This
reduces variance; it does not eliminate it, and it does not make the score reproducible
in the strict sense, because the model behind a given ID changes without notice. That
is what `score_version` exists for.

**The brand profile is pinned to the brand, not re-derived per run.** This was added
after two runs 11 hours apart scored 40 and 25, with the entire spread traceable to two
profile extractions disagreeing about what the company sold. A pinned profile means the
questions stay stable between runs. Re-extraction is explicit, via `refresh_profile`.

**Comparisons are guarded, in strict precedence:**

1. **No prior run** → not comparable.
2. **`cross_query_set`** → not comparable. Triggered when the two runs' question-set
   versions differ, *or* when the actual question text differs. The text-level check was
   added because two runs both labelled `tailored-v2` had asked different questions and
   reported a false 15-point movement.
3. **`cross_version`** → not comparable. The scoring method changed between the runs.
4. **Within the noise floor** → no detectable change.
5. Otherwise → a real movement, reported with its direction.

**Evidence gates withhold rather than guess.** A run that cannot see enough returns no
score at all, with named blockers.

## What this does not establish

Stated plainly, so nobody over-reads the table above:

- It does **not** show the technical-signal channel is deterministic. Technical readiness
  happened to be 25 in all three runs; three runs cannot establish stability.
- It does **not** validate temperature 0 as effective — there was no control run at a
  higher temperature to compare against.
- It cannot rule out silent model substitution by the gateway. The system records which
  models were *requested*; it does not yet read back which were *served*. That gap is known.
- An archetype flip between `ecommerce` and `service` swaps all four signal checks and
  can move the total by up to 40 points. Archetype stability across runs has not been
  measured over a large sample.

## Reading a score responsibly

- Treat the total as accurate to roughly ±15 points.
- Treat a single run as a snapshot, not a measurement.
- Treat `evidence_ok: false` as "no score", never as a low score.
- Compare runs only when the delta reports `detectable: true`.