Model/docs/adr/0030-epc-prediction-validation-is-sap-version-aware-and-component-first.md

EPC Prediction validation is SAP-version-aware and component-first

Status: Accepted. Supersedes the Validation section of ADR-0029 and amends its decision 5 (cohort selection). All other ADR-0029 decisions stand (predict a picture and score it; deterministic neighbour synthesis; fetch-phase fallback; hybrid mode + template synthesis; dual gap-fill / anomaly use).

Why this ADR exists

ADR-0029's validation rested on a three-way SAP comparison, including a neighbour-mean-of-lodged-SAP baseline that predict-then-calculate was meant to beat. A second-order problem was invisible when that was written: the gov EPC register spans multiple SAP spec versions, and a property's neighbours are mostly older certs. In our development corpus only ~16% of certs are SAP 10.2 (sap_version == 10.2, schema 21.0.0 / 21.0.1); the rest were lodged under SAP 2012 (RdSAP 9.x). The same dwelling scores a different SAP under different spec versions, so:

• Averaging neighbours' lodged SAP is invalid — it blends 2012 and 10.2 numbers to estimate a 10.2 target. The ADR-0029 "baseline" was never a fair comparator; on the real corpus it appeared to beat prediction purely as an artifact of this mixing. It is removed.
• Comparing our calculator's output to a neighbour's lodged figure is only meaningful within a same-spec cohort — the existing SAP Spec Version / Validation Cohort rule (ADR-0010) already said this for calculator validation; it applies equally here.

Separately, measuring calc(predicted) against the held-out cert's lodged SAP conflates two unrelated errors: the prediction error and the calculator's own API-path residual (~3 SAP on random gov-API certs today — a known, separate workstream, since the calculator pins at 1e-4 only on the Elmhurst worksheets). A perfect prediction still scores ~3 off lodged. So lodged-SAP error is the wrong thing to tune prediction against.

Decisions

1. The source cohort keeps all cert vintages; only the validation target is SAP 10.2

A building's physical components (wall / roof / floor / heating fuel / age band) are agnostic of the SAP methodology that rated them — a pre-SAP10 neighbour is valid evidence about the street. Dropping pre-SAP10 certs from the cohort (ADR-0029 decision 5) would discard ~84% of neighbours and gut prediction. So: all vintages stay in the Comparable Properties cohort, with recency as a graduated weight (never a hard drop), mattering most for the one component that genuinely goes stale — the heating system, when a boiler is replaced. Only the held-out validation target is restricted to sap_version == 10.2, the only vintage with full-fidelity lodged components to check against. (Target selection uses the API sap_version field directly, not the inspection_date ≥ 2025-07-01 proxy.)

2. Component Accuracy is the primary, calculator-independent signal

Prediction is tuned against how closely the predicted EpcPropertyData components match the actual ones — not against any SAP score. Scored by leave-one-out over a 10.2 target: categoricals as a classification hit-rate (with None = not-applicable excluded from the denominator), numerics as a residual. Coverage spans the SAP-load-bearing components, led by heating (the proven dominant lever — ablating heating to the actual cuts the SAP error from ~7 to ~4.5):

• Heating — main fuel, main category, main control, water-heating fuel/code, has-cylinder, cylinder insulation, secondary heating
• Fabric — wall / roof / floor construction + insulation, age band (plus a ±1-band rate, since adjacent bands ≈ same U-value), room-in-roof
• Glazing — modal glazing type; window count + total-area residuals
• Counts / geometry — door count, building-parts count, floor area
• Renewables — PV presence, solar water heating

Load-bearing principle: predict the components well and correct SAP / carbon / PE fall out once calculator gaps close. Component Accuracy makes progress even while the calculator moves underneath us.

3. calc(predicted) vs API-lodged SAP / carbon / PE is a secondary, calculator-floored check — and two comparisons are rejected

The end-to-end number — does the predicted picture score like the official 10.2 EPC — is reported but not the thing we drive to zero: it is floored by the calculator's API-path residual and improves as both prediction and the calculator workstream land. Carbon and PE are more version-sensitive than SAP (grid factors shifted sharply between SAP 2012 and 10.2), so they too are compared only on 10.2 targets.

Rejected:

• calc(predicted) vs calc(actual) — cancels (and so hides) calculator error against a circular ground truth (our own calculator); a systematically wrong prediction in the calculator's blind spot would score perfectly. Not a validation signal; at most an internal attribution diagnostic.
• neighbour-mean-of-lodged-SAP baseline — mixes SAP versions (see above).

No synthetic SAP-weighted Component Accuracy index: weighting components by SAP impact reintroduces the calculator. The per-component table stays flat; the end-to-end MAE is the holistic rollup.

4. Two validation tiers, one shared scorer

• Tier 1 — committed CI gate. A small, anonymized, frozen fixture under tests/fixtures/ (addresses hashed — the predictor uses address only as a dedup key — post_town dropped; postcode + component fields retained; gov data is OGL). A ratcheting regression gate: each per-component floor / residual ceiling is the currently-measured value and only ever tightens (honouring the repo's no-tolerance-widening ethos); a regression fails the build. End-to-end SAP / carbon / PE thresholds are loose and explicitly calculator-floored — gross-regression guards, not targets. Gates when the fixture is present; skips with a message otherwise.
• Tier 1.5 — S3-hosted scale run (near-term). A few-thousand-cert anonymised corpus stored in S3 rather than committed to git (too large to commit, but far more statistically stable than the 36-target gate fixture). The integration test pulls it to a temp dir and runs the same load_corpus + evaluate_component_accuracy, then reports / asserts. This is the immediate realization of "validate at scale" — it reuses the committed-fixture machinery wholesale (only the data source differs) and needs no bulk-export streaming. Its numbers re-baseline the Tier-1 floors.
• Tier 2 — offline national battle-test (deferred). Built on harness/epc_bulk.py (streams the gov bulk export via HTTP range requests, filtered by sap_version) and harness/cohort.py (offline sweep that captures per-cert raises instead of aborting). Streams the register and buckets by postcode — because bulk is the whole register, every postcode is dense, giving national breadth and dense cohorts at once. Over tens of thousands of 10.2 targets it emits the Component Accuracy table, the end-to-end MAE, and a failure taxonomy (unsupported-schema, mapper raise, calculator raise, no-cohort, no-10.2-target) — the battle-test half. Not committed, not CI-gated; its numbers periodically re-baseline the Tier-1 floors.

All tiers run the same evaluate_component_accuracy / compare_prediction logic over load_corpus — one scorer, three data sources (committed fixture, S3 corpus, bulk stream).

Consequences

• ADR-0029's "Validation" section and its decision-5 clause "pre-SAP10 / very old certs are dropped" are superseded by the above. The CONTEXT terms Comparable Properties (all-vintage source) and Component Accuracy (new) are updated to match.
• The Tier-1 fixture is the first committed gov-EPC fixture sized for statistical stability rather than worksheet-exact pinning — a deliberate departure from the repo's 1e-4 pin convention, justified by prediction's irreducible error.