Per review (Khalim): lift the full-SAP detection out of the SAP-Schema-16.x
branch into a single top-level gate in from_api_response, and key it on the
gov-API's own assessment_type declaration rather than the structural
sap_opening_types proxy.
- _is_full_sap_assessment(data): assessment_type == "SAP" — the authoritative
SAP-vs-RdSAP classification. Verified to separate the entire fixture corpus:
every full-SAP schema (SAP-Schema-17.x/18.x + the broken LIG 16.x) is "SAP";
every reduced cert (RdSAP-Schema-* and reduced SAP-Schema-16.x, incl.
sap_16_0.json) is "RdSAP". data_type / sap_opening_types agree but are derived
shape artifacts; assessment_type is the meaning.
- A cert that is full-SAP by assessment_type but whose schema_type LABEL is not a
recognised full-SAP schema is a *broken schema type* (label disagrees with the
assessment). _record_broken_schema_type logs it — visible, not silently
rerouted — so unreliable labels surface and coverage grows as new mislabels
appear. Generalises beyond 16.x to any future mislabel.
- _from_full_sap maps it via the full-SAP 17.1 mapper (real measured openings,
no defaulting; only `tenure` defaulted). Correctly-labelled full-SAP certs keep
their dedicated branches (one-mapper-per-schema convention); reduced certs are
unchanged.
Tests: broken cert routed AND recorded; correctly-labelled full-SAP not recorded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A full-SAP cert (assessment_type=SAP, the as-designed LIG-* new-builds) lodged
under a SAP-Schema-16.x version failed to map with:
ValueError: RdSapSchema17_1: missing required field 'door_count'
— crashing the property's modelling_e2e subtask. 89 distinct properties in the
2026-06-24 run hit this; ~22 of every 30 sampled are this full-SAP shape.
Root cause: from_api_response dispatches on the schema_type STRING only, and the
SAP-Schema-16.x branch assumed a single shape — "reduced-field (RdSAP-shaped)" —
sending every 16.x cert through _normalize_sap_schema_16_x -> RdSapSchema17_1.
But the SAP-Schema-16.x name covers two structurally different certs:
* reduced RdSAP (assessment_type=RdSAP): top-level door_count / glazed_area
band / construction-code building parts.
* full SAP (assessment_type=SAP): measured shape — sap_opening_types +
structured sap_building_parts carrying measured U-values and door/window
OPENINGS, with NO top-level door_count (the door is an opening). These omit
the reduced-only count fields, so the reduced normaliser failed loud on the
first one it checks (door_count) — and 14 others behind it.
Force-fitting a full-SAP cert to the reduced RdSAP schema was the bug: the data
was never missing, it was being validated against the wrong schema.
Fix: discriminate on shape (_is_full_sap_cert: assessment_type=SAP AND
sap_opening_types present) and route full-SAP 16.x certs to the existing
full-SAP 17.1 mapper, which reads the real measured openings (the lodged
0.9x2.1m door -> door_count 1) — no reduced-field defaulting. The only field the
16.x full-SAP shape omits that SapSchema17_1 needs is `tenure` (register
metadata, no SAP effect), defaulted. Reduced 16.x certs (assessment_type=RdSAP,
no opening types) are untouched — all pinned reduced 16.x fixtures stay on the
RdSAP path.
Regression test pins a real full-SAP-16.0 cert (0293-...-8795, lodged 83) mapping
via the full-SAP path + the reduced 16.2 cert staying on the RdSAP path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A predicted EPC is seeded by deep-copying one representative neighbour's
structure. _template chose the member whose floor area was closest to the
cohort median, ignoring building-part labels. When that member's only part
was lodged with a null identifier (mapped to OTHER), the prediction had no
MAIN part and the modelling_e2e handler rejected it as "not predictable" —
discarding an otherwise-rich same-type cohort.
Restrict the template to MAIN-bearing members (median still over the whole
cohort); fall back to closest-on-size only when none are MAIN-bearing, so an
all-unlabelled cohort is left for the handler's MAIN-part guard to reject
rather than silently relabelling real data.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Note that code 39 "any tariff" currently collapses to the standard electricity
rate; future work should resolve it to the dwelling's actual tariff (off-peak vs
standard) so off-peak electric heating prices correctly. Comment-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
modelling_e2e properties with main fuel 39 failed at the price boundary
(UnpricedFuelCode since #44fff767; previously mis-rated as non-electric →
the ~14-SAP over-rating flagged in earlier review).
Code 39 is SAP Table 12 "electricity, any tariff" (epc_codes.csv main_fuel 39 =
"electricity, unspecified tariff"; spec footnote (j): defines an electric system,
cost/CO2/PE = standard electricity). It was absent from API_FUEL_TO_TABLE_32, so
to_table_32_code(39) was None → is_electric_fuel_code(39) False and pricing
raised.
Fix: map API_FUEL_TO_TABLE_32[39] = 30 (standard electricity) — the canonical
place Khalim's fuel work added codes. One line makes classification, pricing,
CO2/PE and the billing carrier all agree (39 → 30 → ELECTRICITY).
Tests: to_table_32_code(39)==30, is_electric_fuel_code(39) True, price == standard
electricity, and the billing carrier resolves to ELECTRICITY. 0 corpus impact
(no lodged corpus cert uses 39); accuracy + mapper-corpus gates green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5 modelling_e2e cohort certs (full-SAP 19.1.0) were skipped with
"unmapped API sap_opening_type code: 6". Code 6 is "Ext Rooflight"; SAP 10.2
treats roof windows and rooflights as the same inclined-glazing family.
Fix: add 6 to the known opening-type taxonomy and route it onto the roof-window
path (`sap_roof_windows`) alongside code 5, via a `_SAP_ROOF_WINDOW_TYPES`
{5, 6} set. Genuinely-unknown codes (e.g. 99) still raise.
Cert 9878-3908-6309-6714-8200 now maps + calculates (sap 81). Regression test:
a type-6 opening maps onto sap_roof_windows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A dwelling's heating is one conceptual system, but its fields are scattered
across EpcPropertyData (a gov-API schema mirror): the cluster on sap_heating, the
electricity tariff on sap_energy_source.meter_type, hot-water flags loose at top
level. Three places synthesise a heating system — Measure Options, Landlord
Overrides, EPC Prediction's donor — and each hand-copied a different ad-hoc
subset. The override and donor both dropped meter_type, so an electric-storage
system landed on the template's single-rate meter and billed overnight heat at
the peak rate: property 713406 scored SAP 13 (G) vs ~50 (E), inflating the HHRSH
measure to +45.8 and overshooting the plan to band A.
Establish a single Coherent Heating System boundary (CONTEXT.md) that every
synthesiser must cover, with a source-appropriate fill policy (ADR-0035):
- Override overlay *completes* the partial system the landlord named. Companion
fields are now DERIVED from the SAP code, not hand-attached per archetype: the
off-peak meter from the calculator's single off-peak classification (new
OFF_PEAK_IMPLYING_HEATING_CODES = SAP §12 Rules 1-2), and an unobserved storage
charge control defaults to the conservative manual control (Table 4e 2401). So
adding a heating archetype is just adding its code — companions can't be
forgotten. A contract test guards it (every off-peak code drags a Dual meter).
- Prediction's heating donor now *carries* the donor's meter_type alongside its
sap_heating cluster — the donor is already coherent.
Coherence is a synthesis-time obligation only; the calculator still scores a real
lodged cert exactly as lodged.
Verified on 713406: baseline 13 -> 47.8 (E), matching its recorded rating; the
phantom HHRSH recommendation is gone and the plan no longer overshoots to A.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A SAP-Schema-17.0 full-SAP cohort cert (8265-7433-3220-9736-7902) omits the
top-level `has_hot_water_cylinder` and lodges it only under `sap_heating`. The
required top-level field on SapSchema17_1 made `from_dict` raise and skip the
cert.
Fix: make the top-level field Optional, and have the full-SAP mapper
(`from_sap_schema_17_1`, also used by 17.0/18.0.0 via delegation) fall back to
`sap_heating.has_hot_water_cylinder`. RdSAP mappers are unchanged (they keep the
top-level required field; their SapHeating has no such attribute).
Cert now maps + calculates (sap 76). Regression tests for the nested true/false
fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A SAP-16.x cohort cert (9258-4062-7265-2844-7954) lodges a bare int building-
part identifier (the second part as `1` after "Main Dwelling").
`BuildingPartIdentifier.from_api_string` regex-matched it assuming a string and
raised "TypeError: expected string or bytes-like object, got 'int'", failing the
whole property.
Fix: guard the match on `isinstance(api_identifier, str)` so a non-string
identifier falls to OTHER, matching the documented "anything unrecognised ->
OTHER" contract. The baseline SAP fabric sums all building parts regardless of
identifier, so OTHER is SAP-neutral; the identifier only labels parts for
measure targeting. Fixes every mapper (all route through from_api_string).
Cert now maps + calculates (sap 63). Regression test added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 modelling_e2e properties failed with KeyError: 'maxArrayPanelsCount'.
Google returns a `solarPotential` block with no array-level sizing fields
(`maxArrayPanelsCount` / `panelCapacityWatts`) for buildings with no usable
solar estimate. `SolarPotential.from_building_insights` hard-indexed those keys
and crashed the whole property.
Fix: the projection now returns Optional and yields None when those fields are
absent — the established "no solar potential" outcome (the orchestrator and
recommendation path already type it Optional and skip solar on None). Existing
callers (`_solar_potential_for`, harness) already assign to Optional.
Regression test + `assert is not None` narrowing on the valid-fixture tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
10 modelling_e2e properties failed with "unmapped SAP code in fuel_code: 10":
the billing layer (`sap_code_to_fuel`) had no carrier for Table-32 code 10
(dual fuel, mineral + wood) and raised rather than guess one.
SAP 10.2 treats dual fuel as its OWN fuel (its own Table-12 factors), so model
it as its own billing carrier rather than collapsing onto wood or coal:
- New `Fuel.DUAL_FUEL_MINERAL_AND_WOOD`.
- `_CODE_TO_FUEL[10]` -> that carrier.
- Fuel Rates snapshot prices it at 7.69 p/kWh — the midpoint of the COAL proxy
(7.13) and WOOD_LOGS (8.25). This mirrors SAP's own construction: Table-32
dual fuel (3.99) ~= midpoint of house coal (3.67) and wood logs (4.23).
Marked `derived` with a documented _note/_gap/_assumption (like the COAL and
HEAT_NETWORK proxies), since there is no retail blend price.
A dedicated carrier + rate (vs a one-line map to an existing carrier) keeps the
fuel identity faithful to SAP and avoids mispricing dual fuel as pure wood/coal.
Tests: code 10 -> DUAL_FUEL carrier; snapshot prices it at 7.69; grid-export
codes (36/60) still raise (the genuine no-carrier case).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TEMPORARY guard (remove once the SAP calculator's oil-heating under-score is
fixed): a predicted oil-boiler picture scores SAP 13/G against its own
synthesised recorded SAP of 50/E, so the optimiser overshoots goal C all the
way to band A and publishes nonsense.
A predicted EpcPropertyData carries its recorded SAP (energy_rating_current).
When the calculator baseline diverges from it by more than ~one band (20 SAP
points), withhold the Plan: raise inside the per-property loop so the existing
failure isolation drops just that property into `failures` and fails the
subtask, while every other property still models and persists. Lodged
Properties are untouched — they have a real recorded cert and the Rebaseliner
already owns this check.
Verified end-to-end against property 713406 (UPRN 100061849247): baseline 13.2
vs recorded 50 -> quarantined, no Plan written.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the mapper-driven e2e cascade pin for "simulated case 50" (000565 semi,
electric storage main SAP 402 + portable electric secondary + MVHR + whc-903
DUAL electric immersion + 160 L cylinder, Economy-7). Routes the Summary PDF
through extractor + mapper + calculator like the other 000565 fixtures.
Locks in two off-peak fixes this case ground-truthed:
- the Table 13 HW high/low split applied to CO2/PE (commit 39ae2cf0), and
- the Table 12a Grid 2 MVHR fan fraction 0.71/0.58 (commit cd5113ab).
All 11 SAP-result fields reconcile to the U985 worksheet EXACTLY, including
the (272) rating CO2 2397.1237 — SAP 38.8426 (=39), cost £1317.0116, water
1668.0788 kWh, fans 315.6384 kWh.
Summary mirrored to the tracked fixtures dir so the test doesn't depend on
the unstaged `sap worksheets/` workspace.
pyright strict gate not run locally (pyright not installed in this container).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SAP 10.2 Table 12d/12e: electric water heating on a 7-/10-hour tariff bills
CO2/PE at the high-rate code (32/34) and low-rate code (31/33), kWh-weighted
by the Table 13 high-rate fraction. The cost path already applied this split;
the CO2/PE factors did not — they used the flat annual Table 12 figure
(0.136 CO2 / 1.501 PE) for ALL dual-rate electric HW.
That flat-annual behaviour (slice S0380.163) was validated only against
HW-from-main "low-rate cost" certs (100% low, no high-rate split). It is NOT
how Elmhurst bills a whc-903 ELECTRIC IMMERSION: the hand-built case-50
worksheet (000565 + dual immersion, 7-hour) splits HW CO2/PE into "high rate
cost" (CO2 0.1475 / PE 1.5514) + "low rate cost" (CO2 0.1238 / PE 1.4429)
weighted by the Table 13 fraction 0.1009. So flat-0.136 for immersion HW was
a spec gap on our side, not an Elmhurst divergence.
Fix: `_electric_immersion_hw_high_rate_fraction` threads the Table 13 fraction
(scoped to whc-903, 7-/10-hour, cylinder data present) into the HW CO2 + PE
factor helpers, which then blend the Table 12d/12e high/low codes. The flat
rule is unchanged for HW-from-main and 18-/24-hour (no Table 12d split), so
the S0380.163 41-variant cases and the existing pin are untouched.
Case 50: rating CO2 2413.48 -> 2397.1237 = Elmhurst EXACT; demand CO2 2007.1384
EXACT; demand PE +111 -> +32.5 residual (within corpus PE noise). Corpus
unchanged 73.3% / MAE 0.774 / CO2 0.08 / PE 3.4 (62 whc-903 off-peak certs;
aggregate gauges hold). SAP unaffected (cost-based).
Pin: test_whc903_immersion_hw_co2_pe_factors_split_high_low_on_off_peak; doc
updated in SAP_CALCULATOR.md §8.1.
pyright strict gate not run locally (pyright not installed in this container).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
16 modelling_e2e properties failed with "Effective EPC is missing
renewable_heat_incentive; cannot read baseline space-heating / hot-water kWh".
Baseline runs for predicted properties too (ADR-0031), reading space/water-
heating kWh off the EPC's lodged RHI block. Predicted EPCs deep-copy a neighbour
template that may carry no RHI, so `_require_rhi` hard-failed the whole subtask.
Fix: when the EPC has no RHI, fall back to the property's OWN computed figures
from the scored SapResult (space_heating_kwh_per_yr / hot_water_kwh_per_yr) —
more representative than a neighbour's lodged numbers. Only when there is also no
SapResult (the rebaseliner scored nothing) is there genuinely no demand to
record, and we still fail noisily. Lodged certs are unchanged (RHI still wins).
Regression tests: fallback-to-computed, and the no-RHI/no-result raise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the mapper-driven e2e cascade pin for "simulated case 52" (000565
semi + regular non-combi mains-gas boiler SAP 102 + 160 L foam cylinder
heated from the main, no cylinder stat, uninsulated primary pipework,
standard tariff). Routes the Summary PDF through extractor + mapper +
calculator like the other 000565 / 001431_case* fixtures.
This closes the last untested branch of the cylinder/water chain: the
SAP 10.2 §4 cylinder storage loss (Table 2/2a/2b lines 51-55) + the
Table 3 PRIMARY circuit loss (59, uninsulated pipework + no stat) that
combi/immersion fixtures don't reach. All 11 SAP-result fields reconcile
to the U985 worksheet EXACTLY with no calculator change — SAP 57.2904
(=57), cost £911.1973, water 3929.7635 kWh — confirming the cylinder-loss
derivation is correct.
Summary mirrored to the tracked fixtures dir so the test doesn't depend
on the unstaged `sap worksheets/` workspace.
pyright strict gate not run locally (pyright not installed in this container).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
