Model/docs/HANDOVER_API_PROFILING.md

Handover — API SAP accuracy (session 3): raises cleared, now profile-driven

Branch: feature/per-cert-mapper-validation (long-lived working branch — NEVER PR to main; the user pushes/PRs when ready). HEAD a8e5563a+ (the profiler commit), local-only ahead of origin.

READ ALSO: the auto-memory project_per_cert_mapper_validation_state (full slice log + deproven approaches + the meter/shower data-fidelity findings), and the earlier docs/HANDOVER_API_ACCURACY_S2.md (session-2 method).

THE GOAL (unchanged)

100% of API records with a lodged SAP compute within 0.5 SAP of the API's energy_rating_current. Headline gauge: PYTHONPATH=/workspaces/model python scripts/eval_api_sap_accuracy.py.

metric	session-3 (a8e5563a)	session-4 (faf29942)	session-5 (19235d11)
% |err| < 0.5	45.1%	47.6%	53.1%
% |err| < 1.0	59.4%	62.6%	68.0%
% |err| < 2.0	77.7%	79.6%	~81%
mean |err|	1.702	1.586	1.329
median |err|	—	—	0.467
computed / raises	909 / 0	909 / 0	909 / 0
unsupported_schema	100 (deferred)	100 (deferred)	100 (deferred)

SESSION-5 — fuel-code collision (anthracite/coal), 19235d11

The re-audit traced the cohort's WORST cert (2100 anthracite, −61) + the −20/−21 coal cluster to the fuel-code collision (reference_fuel_code_collision): the shared price/CO2/PE lookups check Table-32/12-code membership BEFORE translating the gov-API fuel enum, so API-5 (anthracite) priced at the bulk-LPG rate (12.19 p) and API-33 (coal) at the electricity-10h rate (7.5 p). KEY constraint (goldens caught it): code 33 is ALSO the electricity-10h TARIFF code used by the dual-rate CO2/PE split — so the fix CANNOT live in the shared table functions (breaks golden 000565). Instead canonical_fuel_code (table_32) normalises the colliding SOLID-fuel enums at the fuel-TYPE boundary (_main_fuel_code/_water_heating_fuel_code): 5→15 anthracite 3.64 p, 33→11 house coal 3.67 p; also fixes is_electric_fuel_code(33) mis-flagging coal as electric. Scoped to {5, 33} (unambiguous large mispricings). mean|err| 1.424→1.329 (2100 −61→−11, residual now FABRIC: 110 905 kWh demand = a separate area over-statement); within-0.5 flat at 53.1%. DEFERRED follow-ups: dual-fuel API-9 (0.45 p delta, net-neutral, shifts certs in an un-understood direction — needs its own look); community API-20/25/31 (route through the heat-network standing/CO2 path, NOT unit_price — cert 8536 fuel-31 still mis-prices at 5.5 p electricity → −17). Method: the decompose_api_cost_error.py heat:high/low tail + field-by-field audit of the worst certs surfaced it; is_electric_fuel_code collisions are the tell.

THE UNIFYING PRINCIPLE (user, load-bearing) — "unknown insulation → as-built, NOT uninsulated"

An EPC insulation field that is UNDETERMINED (thickness 'ND'/'AB'/absent → parsed None, or description "as built / (assumed)") must map to the age-band default ("as built"), which is INSULATED at newer bands — never to the uninsulated row. The recurring bug shape: a fixed uninsulated U (cavity Filled-row, roof Table-16 2.30) is MASKED at old bands (where the age default coincides with uninsulated) and only diverges (catastrophic under-rate) at newer bands. All three session-5 fixes are instances. Review status across elements (all now conform):

• Flat roofs — FIXED 58cff932 (this slice). Pitched roofs — "unknown"→Table 18 (a64e857b); "no insulation" only appears at bands A/B where 2.30 IS the age default (verified, no new-band bug).
• Cavity walls — FIXED 2e466ed1. System/timber/solid walls — already on the as-built age row (verified: bidirectional scatter, not a one-cause under-rate). Floors — undetermined thickness already routes to the Table 19 age default (I=25/J=75/K=100 mm; verified).
• CAVEAT: do NOT broadly reroute PITCHED 'ND'/'NI'→Table 18 (the parsed-0 'NI' case) — that was empirically net-negative (pitched "no insulation" lodgements genuinely use 2.30 even at newer bands; the description is load-bearing for pitched lofts). The principle holds for flat roofs, cavity, floors; pitched lofts are the documented exception.

SESSION-5 UPDATE (HEAD 2e466ed1) — whc=903 immersion HW + as-built cavity-U both closed

Shipped (47.6 → 52.1%, two spec-grounded fixes):

(2) 2e466ed1 as-built "insulated (assumed)" cavity → Cavity-as-built row, not Filled cavity (48.6 → 52.1%, the bigger win). Robust-sweep lead: wall_desc="Cavity wall, as built, insulated (assumed)" median +0.26, n=145, but split by age band it was a CLEAN G/H signal (G +1.38 n37, H +1.61 n18; I-L neutral). RdSAP 10 Table 6 (England, p.41) "Filled cavity" row carries the † footnote ("assumed as built") ONLY at bands I-M, where it equals "Cavity as built"; at A-H the filled row is a GENUINE fill. An as-built cavity (type 4) must use "Cavity as built" at all bands (G/H 0.60 not 0.35). This was the SAME latent A-H bug slice S0380.210 fixed for "partial insulation (assumed)" but left for "insulated (assumed)" by a legacy convention. Retired _cavity_described_as_filled; genuine fills (wall_insulation_type=2) still hit the filled row. Per-band confirmation: I-M unchanged, G/H corrected exactly. Bucket within-0.5 47% → 66%; eval +32 net (36 improved, 4 regressed — offsetting-error electric-storage flats). 3 tests updated to the corrected behaviour (the legacy tests literally said "we follow the legacy convention for parity").

(1) 43d4c67d WHC-903 electric immersion off-peak HW → SAP 10.2 Table 13 high-rate fraction (47.6 → 48.6%). The session-4 robust-sweep whc=903 lead (median +0.87, n=84).

• 43d4c67d WHC-903 electric immersion off-peak HW → SAP 10.2 Table 13 high-rate fraction. Was billing 100% at the off-peak low rate; Table 12a "Immersion water heater" row (p.191) routes the WH column to Table 13 (p.197). New tables/table_13.py evaluates the Note-2 equations (f of cylinder volume V, occupancy N, single/dual immersion), clamped [0,1]; 18-/24-hour use the 10-hour column (Note 1). Wired into _hot_water_fuel_cost_gbp_per_kwh (threads V / N / immersion-single from the caller; absent any → old 100%-low fallback, no regression). Off-peak WHC-903 cohort (n=57): within-0.5 16% → 33%, median |err| 1.56 → 0.86.
• IMMERSION CODE MAPPING CORRECTED: immersion_heating_type 1 = DUAL, 2 = SINGLE. The session-3 handover lead #3's "(1=single, 2=dual)" was UNVERIFIED and BACKWARDS. Confirmed via RdSAP 10 §10.5 (p.54 — immersion "assumed dual" on a dual/off-peak meter) cross-checked with the cohort: code 1 sits 3.6:1 on dual meters (40 vs 11), code 2 on single meters (22 vs 16). Dual carries Table 13's small fraction → matches the over-rating direction; the single mapping overshot in a prototype (cohort within-0.5 16% → 14%). The description-vs-code-audit lesson again: skeptical of unverified handover code-semantics claims.
• Next robust leads (post-BOTH-fixes sweep, ranked by net directional skew + MEDIAN): every top bucket is now an UNDER-rate cluster (negative median = fabric/flat scatter, per-cert not one-bug): property_type=2 flats med −0.39 (n=283, netDir +75), roof "(another dwelling above)" −0.46 (n=182), wall_desc="Solid brick, as built, no insulation" −0.22 (n=114). No clean OVER-rate single-cause bucket remains (cavity-insulated dropped to −0.13, main_heat_cat=7 to −0.31, whc=903 off the top — all addressed). The flats under-rate is the biggest front but DIFFUSE (fabric/tariff per-cert) — likely needs worksheets, not one rule. The 100 unsupported-schema certs remain the deferred big ticket. METHOD NOTE: the cavity win came from splitting the +0.26 bucket BY AGE BAND — the mild median hid a sharp G/H spike. When a description bucket has a modest median but a plausible single mechanism, re-split by age band / sub-field before dismissing it as scatter.

SESSION-4 shipped (45.1 → 47.6%): four spec-grounded fixes + closed one false lead. See the ## SESSION-4 … blocks below and the auto-memory for full detail. The systematic bias is gone; the winning method this session was the description-vs-code audit + an outlier-robust categorical sweep (rank by net directional skew + MEDIAN, not mean — the mean-based metric is fooled by multi-cause outliers). 47.6% is still the target's halfway point.

WHAT SHIPPED THIS SESSION (7 slices, all green, pyright net-zero)

1. e41a0bc0 PCDB heat pump w/o SAP code → Table 12a ASHP_APP_N SH split (0.80 high-rate).
2. 2bc73fb0 HP-DHW (WHC 901/902/914 + PCDB HP) → Table 12a WH 0.70 split. Together (1)+(2) killed the cat-4 heat-pump over-rating bias (+1.43 → +0.06).
3. 449d8c5b direct-acting electric boiler (191) → zero primary circuit loss (SAP Table 3 p.160 zero list names it verbatim).
4. f4048588 wall_insulation_thermal_conductivity ignored → §5.8 default λ=0.04. (See KEY INSIGHT below — the gov field is RdSAP output, not an input.)
5. 1c5675a0 floor_heat_loss=8 → no floor heat loss (extension floor over a heated space; RdSAP §3, like code 6).
6. a8e5563a main_heating_category=9 (warm air) → Table 11 secondary fraction 0.10. (4)(5)(6) cleared all 4 raises — eval now has zero raises.
7. (profiler) scripts/profile_api_error.py — the new diagnostic (below).

SESSION-4 UPDATE (HEAD faf29942) — read before re-working the leads below

Shipped this session (45.1 → 47.6%)

1. b40e0f67 exposed-floor-on-flats (floor_heat_loss=1) — §3.12; per-BP override of the dwelling-level flat suppression.
2. 8741fbdf floor_heat_loss=3 → above partially heated space, U=0.7 (§3.12/§5.14) + re-pinned golden 7536 (its "irreducible residual" was THIS bug).
3. 5e7ef5c7 boiler interlock for TRVs+bypass controls 2107/2111 (§9.4.11) — biggest single win (+1.6pts). The no-interlock set was keyed off the wrong signal (the "+0.6 °C" annotation); 2107/2111 lack a room thermostat → −5pp + Table 4f ×1.3 pump.
4. faf29942 description-lodged secondary heating (§A.2.2/Table 11) — gas/oil boilers with an API-description-only secondary ("Portable electric heaters (assumed)", code field None) dropped the secondary (sec_kWh=0); now _has_lodged_secondary_description fires Table 11. Also added cat-8 (electric underfloor) Table-11 fraction 0.10.

• 560c912c/d0f57a0e docs: roof_construction=8 lead CLOSED as data-fidelity (not a bug — see the roof-8 section below; user worksheet sim-case-29 proved we ≡ Elmhurst).

The two methods that worked (reuse these)

• Description-vs-code audit: join each int code (floor_heat_loss, roof_construction, wall_construction, secondary type) to its authoritative …[].description, on single-element certs only (multi-element [] arrays are LOSSY). Mis-maps fall out (floor-3, secondary).
• Outlier-robust categorical sweep (/tmp/cat_audit2.py): rank field-values by net directional skew (#under−0.5 minus #over+0.5) + MEDIAN error. The mean-based directionality metric (/tmp/cat_audit.py) gets FOOLED by multi-cause outliers (e.g. "Solid brick no insulation" looked systematic at mean −1.07 but median is −0.22 = scatter; 2100 −61/RR drove it).

Open robust leads (verify with /tmp/cat_audit2.py — they shift; check MEDIAN not mean)

• whc=903 electric-immersion HW: median +0.87, n=84 — likely off-peak immersion handling (the handover noted WHC 903 raises NotImplementedError on the Table-12a off-peak-immersion row).
• main_heat_cat=7 electric storage: median +1.05, n=41 — over-rate (tariff/cost; partly artifact).
• immersion_type=2 dual: +1.50, n=43 — we OVER-credit (so §12 dual→off-peak would worsen it).
• dwelling_type=Top-floor flat: median −1.24, n=99 — under-rate, mostly fabric scatter/artifacts.
• Low-dir = SCATTER, do NOT single-fix: non-PCDB main / data_source=2 (n=242, 28% within-0.5), mains_gas=N electric (n=145), most flats. These are per-cert/data-fidelity, not one bug.

Resolved/closed this session (don't re-chase)

• Lead #1 floor_codes=3 RESOLVED — the code IS authoritative. The diagnostic that cracked it: join each single-BP cert's floor_heat_loss code to its independent floors[].description (the multi-BP tally was contaminated because a cert's floors[] summary is LOSSY — it drops some BPs' descriptions). Single-BP gives a perfect 1:1 enum: code 1↔"To external air"(exposed), 2↔"To unheated space"(semi-exposed), 3↔"(other premises below)" (9/9), 6↔"(another dwelling below)"(party), 7↔Solid/Suspended(ground). Per RdSAP §3.12 (p.25) code 3 = "above a partially heated space" (non-domestic premises below) → §5.14 constant U=0.7 (NOT Table-20 semi-exposed, NOT ground). SHIPPED 8741fbdf.
• SHIPPED b40e0f67: exposed-floor-on-flats (code 1) area fix — §3.12. A flat's code-1 floor was area-zeroed by _dwelling_exposure; now the per-BP is_exposed_floor overrides the flat suppression upward (mirrors the "another dwelling below" party override).
• SHIPPED 8741fbdf: code 3 → is_above_partially_heated_space (U=0.7) + area override. RE-PINNED golden 7536-3827 — its Ext2(bp3) code-3 floor was mis-read as "ground U=1.12" by a prior agent (the lossy floors[] dropped its description), who declared the residual an "irreducible register-rounding artifact, DO NOT chase". It was this bug: U 1.12→0.70, PE/CO2 residuals moved toward 0. LESSON: "irreducible residual" golden notes are suspect — a real mapper bug can hide there. Eval (both slices): 45.1→45.3%, mean|err| 1.702→1.659, <1.0 59.5→60.2%. User is generating a fresh 0380-2087-8190-2996-3075 worksheet to independently confirm U=0.7 (0380 now −0.63) — validate when it lands.
• Leads re-checked, NOT clean: immersion_type=2 (+1.86) is high-scatter (mean|err| 3.71, bidirectional). main_control=2107 (+1.63) is correctly mapped ("Programmer, TRVs and bypass" type 2 Table 4c(2)) — over-rate is diffuse gas-boiler/flat-fabric, not a dispatch bug. roof_codes=1 broad bucket is mean −0.15 (the −1.78 was top-floor-electric-flat outliers −29/−25). Remaining gains need per-cert worksheets (start code-3) or the unsupported-schema ticket.

SESSION-4 AUDIT — description-vs-code cross-check (other element types)

After floor-3, audited every API field that carries BOTH an integer code AND an independent …[].description (roofs, walls, floors, main_heating, controls), by joining code↔description on single-element certs (the multi-element [] summary is lossy).

• Walls / heating / controls: CLEAN — codes map 1:1 to their description families; residual error is per-cert scatter, not mis-mapping.
• Roof roof_construction=8: investigated hard, NOT a bug — DO NOT re-chase. It looked like a huge lead (n=57, ~189 |err|, mean −2.43) because code-8 sloping ceilings described "Pitched, insulated" with no numeric thickness compute U=2.30 (e.g. 7921-0052 roof_w 241 W/K, SAP 56.75 vs lodged 80). User worksheet (simulated case 29) settled it: at band C, RdSAP/Elmhurst assumes UNINSULATED for an "insulated (assumed)" roof+wall with no measured thickness → Elmhurst SAP 55, which MATCHES our 56.75. The lodged 80 needed real insulation the API record doesn't carry = data-fidelity artifact (meter-3 class). KEY LESSON (user): "insulated (assumed)" = the age-band DEFAULT insulation level, NOT "well insulated" — at old bands that default is ~uninsulated. Re-audit by band confirmed: old-band no-numeric = artifacts (we ≡ Elmhurst); newer-band code-8 already gets correct insulated U (2031 band I U≈0.18, 1436 band E 0.17, 0536 band B 0.09) — their under-rates are elsewhere; numeric- thickness = accurate (9884 +0.06). So the force-uninsulated-at-pre-1950 rule is CORRECT. A roof-8 "fix" would push us AWAY from Elmhurst's faithful 55 toward the unreproducible 80.
• Roof roof_construction=3: latent mis-map (inert). gov desc "(another dwelling above)" (party roof) but we map to "Pitched, no access to loft"; masked by dwelling exposure (mean −0.06). Correct for robustness only if touching the roof mapper; not worth chasing.
• Worksheet-gen constraints (user, for future repros): Elmhurst no longer lets you pick build form for a flat; and a band-C repro defaults to uninsulated walls+roof.

KEY INSIGHT (load-bearing, from the user)

The gov EPC API JSON is the published OUTPUT of RdSAP software (Elmhurst), not its input. So any API field Elmhurst doesn't expose as an input is register metadata the RdSAP10 method does not consume — route it to the spec default, don't try to "use" it. This is exactly why wall_insulation_thermal_conductivity (slice 4) → always λ=0.04. Apply the same lens to any new "extra" API field before wiring it.

THE NEW DIAGNOSTIC — scripts/profile_api_error.py (run this first)

PYTHONPATH=/workspaces/model python scripts/profile_api_error.py joins each computed cert's signed error with a rich feature set from its raw API JSON (not the mapped EpcPropertyData), and ranks (feature, value) buckets by error carried + by |mean signed| bias. This is how to find "silly API-path handling" gaps. --min-n N sets the bucket floor.

PRIORITISED LEADS (from the run at a8e5563a — verify with the profiler, they'll shift)

Cleanest "API-path handling" candidates first (small, biased buckets = likely a mapper/dispatch bug, not noise):

1. floor_codes=3 → mean signed +5.37 (n=10). We map API floor_heat_loss=3 → "To unheated space" (same as code 2). The +5.37 over-rate says that's wrong — code 3 likely isn't "unheated space" (or its U is wrong). Pull the n=10 certs, check what code 3 really is (ask the user the Elmhurst floor dropdown — the API=output lens). Highest bias, smallest scope = start here.
2. Control-code biases: main_control=2306 −2.96 (n=11), 2602 +2.49 (n=14), 2107 +1.65 (n=38), 2402 +1.14 (n=10), 2307 +0.74 (n=11). Several control codes carry systematic bias → Table 4c/4e control dispatch gaps. 2107/2602 are the biggest. Check _CONTROL_TYPE_BY_CODE + the Table 4c efficiency-adjustment / Table 4e control coverage.
3. immersion_type=2 (dual immersion) → +2.00 (n=43, mean|err| 3.85). RdSAP §12 lists "dual electric immersion" as an off-peak trigger; the cascade does NOT consume immersion_heating_type for tariff (verified — only comments reference it). Wiring the §12 dual-immersion → off-peak rule for Unknown meters is a clean spec slice. (1=single, 2=dual per the Elmhurst Summary.)
4. roof_codes=1 −1.78 (n=27) (flat roof under-rate) and roof_insulation_thickness=None −1.18 (n=52) — flat-roof / no-thickness roof handling.
5. main_data_source=2 / has_pcdb_main=False → 28% within 0.5, mean|err| 3.17 (n≈242). Non-PCDB heating systems (SAP-table efficiency) are a big under-rating cluster. Likely Table 4b default-efficiency or fabric, but worth a look — it's 1/4 of the sample.

Big scattered segments (need worksheets, NOT clean single fixes)

• whc=903 (electric immersion HW): 13% within 0.5, n=84 — looks like the worst bucket but it's the electric storage(cat-7)+room-heater(cat-10) segment compounding (worst certs span −29…+32, bidirectional). Not one bug.
• mains_gas=N (electric): 21% within 0.5, mean|err| 4.27 (n=145) — the hardest segment; per-cert fabric/tariff scatter.
• Flats (property_type=2): 31% within 0.5 (n=283) — still the worst dwelling type.
• cat-7 storage (+0.75) / cat-10 room heaters (+0.75) — both net over-rate; bidirectional.

DEPROVEN — do NOT retry (empirically failed in earlier sessions; details in memory)

• Routing roof 'ND' → Table 18 (description is load-bearing even with 'ND').
• Broad "all Unknown(meter 3) electric → off-peak" (over-credits room heaters). NOTE: the meter-3 under-rate is partly an irreducible data-fidelity artifact — the register stores meter_type=3 ("Unknown") on certs whose lodged rating actually used an off-peak meter (cert 2474: lodged 78 needs 18-hour, but API says Unknown → spec-faithful ~68). Don't chase those to the lodged value.
• RR shell U Table-17-50mm (golden 6035 disproves it).
• Shower enum is settled (non-bug): API shower_outlet_type 1=non-electric(mixer)/2=electric (cohort 2636/0330 validate at 1e-4); types 3/4/5 are finer gov-output sub-types (type 3 is all on unsupported schema 19.1.0; type 4 already accurate). shower_wwhrs 1/2/3/4 = none / inst- WWHRS-1 / inst-WWHRS-2 / storage. Low headline value — not worth pursuing.

THE 100 unsupported_schema CERTS (deferred — bigger ticket)

SAP-Schema-19.1.0 (and other pre-21). The user is planning a separate big piece: map old schemas → new + predict missing fields from similar-looking properties (needs an EPC-prediction method). That needs its own grilling session — do NOT start it here.

WORKSHEET WORKFLOW (the user generates them on request)

For per-cert scatter that needs ground truth, ask the user to generate P960 + Summary worksheets from the cert's OWN API JSON (/tmp/epc_2026_sample/<cert>.json). Describe the cert field-by-field first (the user reproduces in Elmhurst; their repros are approximate — confirm SAP matches lodged before pinning). Worksheets land under sap worksheets/golden fixture debugging/simulated case NN/ or sap worksheets/additional with api 2/<cert>/. Pin the cascade to the P960 §3/§4/§9a/§10a line refs at abs=1e-4. Caveat: the user's repros often diverge (wrong system / approximate inputs) — validate the BEHAVIOUR (e.g. λ, no-heat-loss) empirically against the lodged SAP, don't blindly pin to a non-faithful repro.

TOOLS & CONVENTIONS (non-negotiable)

• scripts/eval_api_sap_accuracy.py — headline + TOP-40 + _results.csv.
• scripts/profile_api_error.py — raw-API characteristic profiling (NEW, run first).
• scripts/decompose_api_cost_error.py — per-component cost decomposition (off-peak caveat: uses STANDARD elec price, mis-flags off-peak certs).
• ~1009 cached API JSONs at /tmp/epc_2026_sample (EPC_SAMPLE_CACHE overrides).
• one cause = one slice = one commit; spec citation (page+line) in the message; AAA test headers (# Arrange/# Act/# Assert); abs(x-y)<=tol not pytest.approx; SAP 10.2 only; no tolerance-widening / xfail; RdSAP is deterministic — every fix is a spec rule, not a population data-fit (the user is firm); pyright strict net-zero (baseline-compare via git stash); stage files BY NAME (tree carries unrelated scripts/ + sap worksheets/ changes — never git add -A); Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>.
• REGRESSION after any calc/mapper change: tests/domain/sap10_calculator/, backend/documents_parser/tests/, datatypes/epc/, golden fixtures (esp. 6035).
• Pre-existing failures to IGNORE (fail on the stashed baseline too): test_total_floor_area and the 2 stone-wall U tests in domain/sap10_ml/tests/test_rdsap_uvalues.py.

ARCHITECTURE NOTES (so you don't re-discover them)

• API path: EpcPropertyDataMapper.from_api_response(doc) → cert_to_inputs(epc, prices= SAP_10_2_SPEC_PRICES) → calculate_sap_from_inputs(...).sap_score_continuous.
• Cost path uses inputs.fuel_cost (Table-32/12a precompute); _fuel_cost returns a ZERO sentinel for off-peak → calculator falls back to the legacy scalar _space_heating_fuel_cost_ gbp_per_kwh (which DOES carry the off-peak rate). SapResult fuel codes are RAW API enums — translate via table_12.API_FUEL_TO_TABLE_12.
• Heating efficiency: _main_heating_detail_efficiency → PCDB Table 105 winter eff (if PCDB index) else seasonal_efficiency(code, cat, fuel) (Table 4a/4b, in domain/sap10_ml/ sap_efficiencies.py). Warm-air Table 4a code→eff map already covers 501-520.
• sap10_ml/ is marked for eventual migration to sap10_calculator/ but is still the live u-value/efficiency path.