Measurement reliability

This page states what the score's precision actually is, including where it is not good enough yet. It is published because a measurement product that hides its own error bars is asking to be taken on faith, which is the opposite of the point.

The measured spread

Three audits of the same brand — brooklinen.com, same code version, temperature 0, run seconds apart — produced:

Range: 15 points. Sample standard deviation ≈ 8.7 on n = 3, which is a small enough sample that the true σ could plausibly sit anywhere in roughly [4.5, 54].

The cause is visible in the table: one shopper query flipped between visible and invisible. At 15 points per invisible query, one flip is the entire spread.

This was later confirmed in production. On the first monthly Defend cycle, ten brands were re-audited with no site changes; three moved by +15, +15, and −10.

The noise floor

SCORE_NOISE_FLOOR = 15

A change of 15 points or less between two runs is reported as not detectable. The comparison is inclusive — exactly 15 is treated as noise, not signal.

This is a real limitation, not a caveat. It is why the product's unit of work is a prioritized fix list rather than a single recommendation, and why Defend reruns monthly rather than continuously.

What is done about it

Temperature 0 on every model call, both the engine panel and the judge. This reduces variance; it does not eliminate it, and it does not make the score reproducible in the strict sense, because the model behind a given ID changes without notice. That is what score_version exists for.

The brand profile is pinned to the brand, not re-derived per run. This was added after two runs 11 hours apart scored 40 and 25, with the entire spread traceable to two profile extractions disagreeing about what the company sold. A pinned profile means the questions stay stable between runs. Re-extraction is explicit, via refresh_profile.

Comparisons are guarded, in strict precedence:

1. No prior run → not comparable.

2. cross_query_set → not comparable. Triggered when the two runs' question-set versions differ, or when the actual question text differs. The text-level check was added because two runs both labelled tailored-v2 had asked different questions and reported a false 15-point movement.

3. cross_version → not comparable. The scoring method changed between the runs.

4. Within the noise floor → no detectable change.

5. Otherwise → a real movement, reported with its direction.

Evidence gates withhold rather than guess. A run that cannot see enough returns no score at all, with named blockers.

What this does not establish

Stated plainly, so nobody over-reads the table above:

• It does not show the technical-signal channel is deterministic. Technical readiness happened to be 25 in all three runs; three runs cannot establish stability.

• It does not validate temperature 0 as effective — there was no control run at a higher temperature to compare against.

• It cannot rule out silent model substitution by the gateway. The system records which models were requested; it does not yet read back which were served. That gap is known.

• An archetype flip between ecommerce and service swaps all four signal checks and can move the total by up to 40 points. Archetype stability across runs has not been measured over a large sample.

Reading a score responsibly

• Treat the total as accurate to roughly ±15 points.
• Treat a single run as a snapshot, not a measurement.
• Treat evidence_ok: false as "no score", never as a low score.
• Compare runs only when the delta reports detectable: true.