Model/docs/HANDOVER_API_ACCURACY_S2.md

Handover — API SAP accuracy (session 2): fabric + tariff fixes, and why we now need worksheets

Branch: feature/per-cert-mapper-validation (long-lived working branch — NEVER PR to main; the user pushes/PRs when ready). HEAD 4d1a58b8, local-only ahead of origin.

READ ALSO: docs/HANDOVER_COST_DECOMPOSITION.md (the decomposition method + price calibration), and the auto-memory project_per_cert_mapper_validation_state (full slice log

• deproven approaches).

THE GOAL (unchanged, and we are FAR from it)

100% of API records with a lodged SAP must compute within 0.5 SAP of the API's energy_rating_current. scripts/eval_api_sap_accuracy.py headline (905 computed certs):

metric	session-2 start	now (4d1a58b8)
% |err| < 0.5	43.8%	45.0%
% |err| < 1.0	—	59.4%
% |err| < 2.0	—	77.6%
mean |err|	2.01	1.757
mean signed	−0.31	+0.019
p99 |err|	—	17.2
max |err|	—	61.4

Be honest about where this is: 45% within 0.5 is poor. The headline barely moved (+1.2pp) across 6 fixes because each clean cause is small (10-30 certs). What DID change decisively is the signed bias: −0.31 → +0.02. The systematic under-rating that defined the sample at session start is gone — the remaining error is bidirectional scatter, ~55% of certs are >0.5 off in BOTH directions, and there is no single lever left that moves the headline by more than ~0.3pp. Further progress is per-cause, and increasingly needs worksheet ground truth (see "Why we need worksheets" below).

WHAT SHIPPED THIS SESSION (7 commits, all green, pyright net-zero)

1. 98f71d25 decomposition tool scripts/decompose_api_cost_error.py — calibrates the consumer price from accurate gas certs (gas £0.0809, elec £0.2839/kWh), predicts each component cost, clusters by (component × direction). CAVEAT: it uses the STANDARD elec price, so it MIS-FLAGS off-peak-heated certs as heat:high. For electric certs compare against the cascade's own cost intermediates (SapResult.intermediate['main_heating_cost_gbp'] etc.), not the decomposition.
2. bb830741 sloping-ceiling — roof_construction=8 carries sloping_ceiling_insulation_thickness ("100mm"); the mapper dropped it. Now fed → Table 17 col (1a). 9884 −5.5 → +0.06.
3. 6b045146 gas-boiler fuel from §14.2 mains-gas meter (Summary/Elmhurst path) — a Table-4b gas boiler with a SEPARATE electric immersion (§15 "Electricity") used to raise MissingMainFuelType; now falls back to the "Main gas: Yes" meter flag → mains gas.
4. 3aed8f85 floor "another dwelling below" (code 6) — party floor, no heat loss (mirror of the roof's "another dwelling above" override). 2115 floor 47.85→0 W/K, −23→−4.
5. a64e857b roof "Unknown insulation" → Table 18 (§5.11.4) — "NI"=Not Indicated (undetermined), not zero; routes to age-band default not 2.30. Cluster mean|err| 7.8→1.8.
6. 678aa7af main-roof U ignores Room-in-Roof "no insulation" leak — _joined_descriptions concatenated ALL roofs[], so an RR "no insulation" contaminated the main-roof U. Now drops "Roof room(s)" entries for the main-roof U (RR shell unaffected; golden 6035 safe).
7. 4d1a58b8 Unknown-meter + storage/CPSU → off-peak tariff (§12) — storage heaters charge overnight; an Unknown (code-3) meter no longer bills their charge at standard 13.19p. rdsap_tariff_for_cert infers off-peak for Rule-1 CPSU/Rule-2 storage only; and _fuel_cost now uses _rdsap_tariff (not raw tariff_from_meter_type). 7336 −26 → −0.16.

DEPROVEN — do NOT retry (empirically failed this session)

• roof 'ND' (Not Determined) → Table 18. 'ND' is on ~305/905 certs and the lodged calc genuinely uses the description's high U for many; routing all 'ND' to age-default broke 9 certs (some 0 → +15) for zero net gain. The description is load-bearing even with 'ND'. (The narrow "unknown" word IS a clean signal — that's slice a64e857b.)
• broad "all §12 Rule-3 electric → off-peak on Unknown meters". Net-NEGATIVE (44.9→44.8, bias flipped +0.16). Room-heater dwellings (code 691) over-credit when forced off-peak (their electric-immersion HW goes off-peak). Direct-boiler 191 alone is +0.1 but requires a 191-vs-691 split that is NOT spec-grounded (both are Rule 3) — a population data-fit; left unshipped on purpose (the user's principle: RdSAP is deterministic, no overfitting).
• RR shell U Table-17-50mm (from session 1, still true): golden 6035 disproves it.

THE REMAINING CLUSTER MAP (where the error lives now)

Run scripts/decompose_api_cost_error.py for the live table. As of 4d1a58b8:

cluster	n	within 0.5	note
heat:high	319	39%	we over-state heating energy (or off-peak mis-priced)
heat:low	229	47%	we under-state heating energy
hw:low	161	50%
hw:high	120	43%
balanced	76	55%

By dwelling type / system (from _results.csv):

• Flats (prop 2): 283 certs, 31% within 0.5 — still the worst segment by far (houses 50%, bungalows 59%). Signed −0.24. The fabric/tariff fixes helped but flats remain hardest.
• Heat pumps (cat 4): 20 certs, 45% within 0.5, mean signed +1.43, mean|err| 3.81 — a distinct OVER-rating cluster, UNTOUCHED this session. These have PCDB indices (e.g. 9472 +15.0 idx 104351, 2789 +13.4 idx 104632, 4135 +10.0 idx 106465). Likely an Appendix-N / PCDB efficiency or HW-from-HP issue. Good next target — it's a coherent over-rate cluster, and HPs may be pinnable from a worksheet.
• Top single offenders (see eval TOP-40): 2100 −61 (n_bps=2, electric, prop 0), 2958 +32 (single-bp electric), 0390 −29 (flat, "Flat no insulation"+ND roof — the deproven path), 2080 −25 (electric direct-boiler flat — mixed cause), 7921 −23 (gas, PCDB idx 16814).

WHY WE NEED WORKSHEETS NOW (the user has accepted this)

The decomposition method got us the directional bias (under-rating → balanced). It is now exhausted for the bidirectional scatter because:

1. For electric/off-peak certs the consumer-price *_cost_current fields diverge from the SAP Table-12 prices the rating actually uses — the lodged total can EXCEED ours while the lodged SAP is HIGHER. So we cannot back-calculate a reliable kWh/cost target.
2. The remaining causes (HW immersion off-peak charge-vs-on-demand split; HP Appendix-N efficiency + HP-DHW; per-cert fabric like 2100's −61) are sub-component values that the ±10% calibration cannot resolve — they need a line-ref pin.

What to generate (in priority order): Elmhurst worksheets (P960 + Summary) for —

• A heat pump cat-4 cert that over-rates, e.g. 9472-3052-6202-0766-7200 (+15.0, idx 104351) or 2789-8331-7179-3314-1150 (+13.4). Pin §9b HP efficiency (Appendix N / Table 4a), the (206)/(207) seasonal eff, and HW-from-HP. This is the cleanest coherent cluster.
• A meter-3 electric flat with electric-immersion HW, e.g. 2474-3059-4202-4496-3200 (−13.3, cat-2 direct-boiler 191) or 2080 (−25.5). Pin EXACTLY how RdSAP bills the electric-immersion HW (§4 + Table 12a) and direct-acting heating on an off-peak tariff — this resolves whether Rule-3 electric on Unknown meters should be off-peak (the unshipped 191 question) and the HW-off-peak split.
• (Optional) 2100-5421-0922-1622-3463 (−61, the worst) — 2 building parts, electric; a worksheet would localise whether it's a §3 geometry or heating blowup.

The faithful-reproduction rule still holds: use the cert's OWN data (its API JSON is in /tmp/epc_2026_sample/<cert>.json; generate the Elmhurst worksheet from the same property), NOT a template-edited 001431. Template edits drift (session-1 lesson).

TOOLS & CONVENTIONS

• PYTHONPATH=/workspaces/model python scripts/eval_api_sap_accuracy.py — headline + TOP-40
  • per-cert /tmp/epc_2026_sample/_results.csv.
• PYTHONPATH=/workspaces/model python scripts/decompose_api_cost_error.py — component clusters + _cost_decomposition.csv (remember the off-peak caveat above).
• Sample: ~1009 cached API JSONs at /tmp/epc_2026_sample (override EPC_SAMPLE_CACHE).
• Conventions (non-negotiable): one cause = one slice = one commit; spec citation (page+line) in the message; AAA test headers; abs(x-y)<=tol not pytest.approx; SAP 10.2 only; no tolerance-widening / xfail; pyright strict net-zero (baseline- compare via git stash); stage files BY NAME (the tree carries unrelated scripts/
  • "sap worksheets/" changes — never git add -A); RdSAP is deterministic — every fix must be a spec rule, not a population data-fit (the user is firm on this); Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>.
• REGRESSION after any calculator change: tests/domain/sap10_calculator/, backend/documents_parser/tests/, datatypes/epc/, and the golden fixtures (esp. 6035).
• Pre-existing failures to IGNORE (fail on the stashed baseline too, NOT yours): test_from_rdsap_schema.py::…::test_total_floor_area, and the 2 stone-wall U tests in domain/sap10_ml/tests/test_rdsap_uvalues.py (…stone_granite_thin_wall_age_a_120mm…, …stone_sandstone…) — likely fallout from the §5.7 wall-U slice 27375d93; worth a separate fix but not yours to count against net-zero.

ARCHITECTURE NOTES THAT COST TIME (so you don't re-discover them)

• The API cost path uses inputs.fuel_cost (the Table-32/12a precompute, _fuel_cost), NOT the scalar space_heating_fuel_cost_gbp_per_kwh. calculator.py:540 picks the precompute when populated, ELSE the legacy scalar fields. _fuel_cost returns a ZERO sentinel for any off-peak tariff → the calculator then falls back to the legacy scalar fields (which DO carry the off-peak rate from _space_heating_fuel_cost_gbp_per_kwh). So a tariff change only bites if it flips _fuel_cost's tariff off STANDARD.
• _table_12a_system_for_main maps cat-10 room heaters → OTHER_DIRECT_ACTING_ELECTRIC but leaves storage (401-409, correct: → None → 100% low rate) and direct-boiler 191 / CPSU as TODO (→ None → pure low rate, which OVER-credits 191 on off-peak). Wiring 191/CPSU rows is a prerequisite if you ever revisit Rule-3-on-Unknown.
• Fuel codes stored on SapResult are the RAW API enum (26 = mains gas), not Table-12 codes — translate via table_12.API_FUEL_TO_TABLE_12 (the decomposition script does this).