Model/docs/HANDOVER_REAL_LIFE_CERT_ACCURACY.md

Handoff — Real-life cert SAP accuracy (validate → fix → expand)

Purpose. Grow real_life_examples into a trustworthy regression corpus that validates this repo's SAP calculator against accredited Elmhurst Energy, one real certificate at a time — and use each cert to improve the mapper and calculator and add test coverage. This is the strategy/context doc; the step-by-step procedure is the /validate-cert-sap-accuracy skill.

---

TL;DR — the loop per cert

Run /validate-cert-sap-accuracy <uprn>. It drives:

1. scripts/fetch_real_life_epc_sample.py <uprn> → saves backend/epc_api/json_samples/real_life_examples/<schema>/uprn_<uprn>/epc.json, prints schema + lodged rating + our engine's SAP.
2. /epc-to-elmhurst-rdsap-inputs <uprn> → writes elmhurst_inputs.md (page-by-page Elmhurst entry sheet with code→value mappings).
3. You build it in Elmhurst, export the Summary and SAP-10.2 worksheet PDFs → save as elmhurst_summary.pdf / elmhurst_worksheet.pdf in the sample dir.
4. scripts/compare_epc_paths.py <uprn> → builds EpcPropertyData from BOTH the gov-API json and the Elmhurst summary, deep-diffs them, runs BOTH through Sap10Calculator, and prints Elmhurst's worksheet SAP.
5. Reconcile the field diffs to convergence (see "Reading the comparison").
6. Pin the agreed score: add a RealCertExpectation to tests/domain/sap10_calculator/test_real_cert_sap_accuracy.py; the sample dir is already the corpus entry, so the pin is what activates it.

Reading the comparison (the core skill)

• Our engine on Elmhurst inputs ≈ Elmhurst's worksheet SAP → the calculator is correct. (Proven repeatedly — it matches Elmhurst's fuel cost to the penny.)
• gov-API SAP vs Elmhurst-PDF SAP gap → input differences only. Triage each field diff into:
  • Elmhurst data-entry error (swapped floor dims, wrong cylinder/immersion, missing baths, wrong postcode) → fix in Elmhurst, re-export, re-compare.
  • gov-API mapper gap → a real per-cert-mapper fix (improve the mapper). Flag it; don't tune to mask it.
  • Ground-truth question (what the property actually is) → you settle it; align both sides to the lodged data.
• Ignore cosmetic diffs: codes vs strings (tenure, region), empty EnergyElement lists (the Elmhurst path stores construction in sap_building_parts).

Mental model (hard-won)

• The calculator is essentially exact. Fed identical inputs it reproduces accredited Elmhurst. So accuracy work is almost entirely mapper fidelity — making the gov-API EpcPropertyData match what an assessor would key in.
• Lodged energy_rating_current is NOT a clean target for pre-SAP10 schemas (17.x–19.0 lodge SAP-2012 ratings — a different methodology). Use Elmhurst on the lodged inputs as ground truth; cite lodged only as context.
• Pin the observed gov-API engine score, not the lodged or Elmhurst number — the test guards the production path. Record the Elmhurst-validated value + what reconciled it in the comment.

Code→value cheatsheet (the gotchas that bit us)

Full table: .claude/skills/epc-to-elmhurst-rdsap-inputs/reference/mapping.md. The ones that cost us time:

Field	Mapping	Note
immersion_heating_type	1 = DUAL, 2 = SINGLE	flips Table 13 eqn; swung cert 10002468137 by 4 SAP
cylinder_size	2 = 110 L, 3 = 160 L, 4 = 210 L	pick the litres in Elmhurst, not the label
party_wall_construction	1=Solid (U 0), 2=cavity unfilled (0.5), 3=filled (0.2), 4/5=unknown (0.25)	code 1 ≠ "unable to determine"
cylinder_insulation_type	1=Foam, 2=Jacket	—
Number of baths	rooms_with_bath_and_or_shower + rooms_with_bath_and_mixer_shower	Elmhurst WWHRS sub-tab, defaults to 0
Off-peak fuel (29)	space-heat 100% low rate (correct for storage heaters); water-heat = Table 13 split	meter = Economy-7/Dual
water_heating_code 903	Electric immersion off-peak → Elmhurst "Water Heater" category	not "Boiler Circulator" (901)
Windows (reduced-field)	area = 0.148 × TFA × band; raw U from glazing code via u_window (RdSAP Table 24)	not real geometry

Known mapper gaps to chase (improve the mapper)

• Lodged alt-wall dropped — sap_building_parts[].sap_alternative_wall_1 is None on the gov-API path even when the cert lodges one (Elmhurst keeps it). ~£1 / 0.06 SAP on cert 10002468137, but real. (per-cert-mapper / Khalim's domain.)
• Add more as new certs surface them — that's the point of expanding the corpus.

Picking certs for coverage

Maximise variety so each cert exercises new mapper/calculator paths:

• Heating: gas combi, gas boiler + cylinder, oil, LPG, solid fuel, heat pump (ASHP/GSHP), storage heaters (done), electric boiler, community/heat-network.
• Hot water: combi, cylinder (foam/jacket), immersion (single/dual), solar HW, WWHRS, instantaneous electric.
• Schema: 17.0, 17.1 (done), 18.0 (done), 19.0, 20.0.0, 21.0.0, 21.0.1.
• Geometry: flats (ground/mid/top floor), bungalow, extensions, room-in-roof, conservatory, basement.
• Tariff/region: mains gas, off-peak electric (done), 10/18/24-hour, varied regions.
• Tech: PV (export/non-export), wind, FGHRS.

Guardrails

• RdSAP-21.0.1 corpus gauge (tests/infrastructure/epc_client/test_sap_accuracy_corpus.py, currently 66.9% within-0.5 SAP) is the broad regression net for any mapper/calc change. Ratchet thresholds up, never loosen. Re-run it after every change.
• Don't tune the mapper to one cert — fix generically and confirm against the gauge. A single-cert tweak that regresses the corpus is net-negative.
• No tolerance widening in the real-cert test — pin the observed integer SAP; if a known engine bug blocks a cert, use known_bug_xfail="…" (strict xfail).

Current corpus

Sample	Schema	Pin	Status
uprn_100020450179	RdSAP-18.0	73	matches lodged 73
uprn_10002468137	RdSAP-17.1	61	Elmhurst-validated (dual immersion, 110 L, 2 baths); lodged 55 = old schema
uprn_10092973954	SAP-17.1 (full SAP)	77	full-SAP mapper partial; pinned to observed (not lodged 83)

Open threads

• Full-SAP mapper WIP (_sap_door_aggregates D2 door slice) is parked in git stash (hyde-wip-before-main-merge, stash@{0}) — not mine; the full-SAP effort should git stash show -p stash@{0} and re-apply. Full-SAP (SAP-Schema-*) support is incomplete; RdSAP schemas are the solid path.
• The off-peak water-heating (Table 13) fix and the per-cert-mapper accuracy work landed on main via PR #1217 — pull main before starting new certs.

Key files

Concern	Path
Per-cert loop (procedure)	skill validate-cert-sap-accuracy
Cert → Elmhurst input sheet	skill epc-to-elmhurst-rdsap-inputs (+ reference/mapping.md)
Capture a cert	scripts/fetch_real_life_epc_sample.py
Compare the two paths	scripts/compare_epc_paths.py
The accuracy test (pins)	tests/domain/sap10_calculator/test_real_cert_sap_accuracy.py
Corpus regression gauge	tests/infrastructure/epc_client/test_sap_accuracy_corpus.py
Mapper (improve here)	datatypes/epc/domain/mapper.py, domain/sap10_calculator/rdsap/cert_to_inputs.py
Calculator	domain/sap10_calculator/