Ports the §6.1 conservatory split (declare the four glazed fields; exclude the
glazed BP from the fabric loop; carry it as sap_conservatory) to the remaining
gov mappers that lacked it. Fixes the modelling_e2e construction_age_band
NOT-NULL failures: 5 type-4 conservatory certs on RdSAP-Schema-20.0.0 (e.g.
property 725213) mapped the conservatory to a null-age fabric part that crashed
on persist. 17.0/17.1/18.0 ported preventively (no-op for non-conservatory
certs). 21.0.0 deferred — it never adopted the ADR-0027 all-Optional
SapBuildingPart, so a sparse conservatory part needs that broader change first.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per review (Khalim): lift the full-SAP detection out of the SAP-Schema-16.x
branch into a single top-level gate in from_api_response, and key it on the
gov-API's own assessment_type declaration rather than the structural
sap_opening_types proxy.
- _is_full_sap_assessment(data): assessment_type == "SAP" — the authoritative
SAP-vs-RdSAP classification. Verified to separate the entire fixture corpus:
every full-SAP schema (SAP-Schema-17.x/18.x + the broken LIG 16.x) is "SAP";
every reduced cert (RdSAP-Schema-* and reduced SAP-Schema-16.x, incl.
sap_16_0.json) is "RdSAP". data_type / sap_opening_types agree but are derived
shape artifacts; assessment_type is the meaning.
- A cert that is full-SAP by assessment_type but whose schema_type LABEL is not a
recognised full-SAP schema is a *broken schema type* (label disagrees with the
assessment). _record_broken_schema_type logs it — visible, not silently
rerouted — so unreliable labels surface and coverage grows as new mislabels
appear. Generalises beyond 16.x to any future mislabel.
- _from_full_sap maps it via the full-SAP 17.1 mapper (real measured openings,
no defaulting; only `tenure` defaulted). Correctly-labelled full-SAP certs keep
their dedicated branches (one-mapper-per-schema convention); reduced certs are
unchanged.
Tests: broken cert routed AND recorded; correctly-labelled full-SAP not recorded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A full-SAP cert (assessment_type=SAP, the as-designed LIG-* new-builds) lodged
under a SAP-Schema-16.x version failed to map with:
ValueError: RdSapSchema17_1: missing required field 'door_count'
— crashing the property's modelling_e2e subtask. 89 distinct properties in the
2026-06-24 run hit this; ~22 of every 30 sampled are this full-SAP shape.
Root cause: from_api_response dispatches on the schema_type STRING only, and the
SAP-Schema-16.x branch assumed a single shape — "reduced-field (RdSAP-shaped)" —
sending every 16.x cert through _normalize_sap_schema_16_x -> RdSapSchema17_1.
But the SAP-Schema-16.x name covers two structurally different certs:
* reduced RdSAP (assessment_type=RdSAP): top-level door_count / glazed_area
band / construction-code building parts.
* full SAP (assessment_type=SAP): measured shape — sap_opening_types +
structured sap_building_parts carrying measured U-values and door/window
OPENINGS, with NO top-level door_count (the door is an opening). These omit
the reduced-only count fields, so the reduced normaliser failed loud on the
first one it checks (door_count) — and 14 others behind it.
Force-fitting a full-SAP cert to the reduced RdSAP schema was the bug: the data
was never missing, it was being validated against the wrong schema.
Fix: discriminate on shape (_is_full_sap_cert: assessment_type=SAP AND
sap_opening_types present) and route full-SAP 16.x certs to the existing
full-SAP 17.1 mapper, which reads the real measured openings (the lodged
0.9x2.1m door -> door_count 1) — no reduced-field defaulting. The only field the
16.x full-SAP shape omits that SapSchema17_1 needs is `tenure` (register
metadata, no SAP effect), defaulted. Reduced 16.x certs (assessment_type=RdSAP,
no opening types) are untouched — all pinned reduced 16.x fixtures stay on the
RdSAP path.
Regression test pins a real full-SAP-16.0 cert (0293-...-8795, lodged 83) mapping
via the full-SAP path + the reduced 16.2 cert staying on the RdSAP path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the merged 21.0.1 fix (d501535c): declare the four glazed conservatory
fields on the 19.0 SapBuildingPart so from_dict stops dropping them, exclude the
glazed BP from the fabric loop, and carry it as EpcPropertyData.sap_conservatory
(§6.1). Fixes cert 718138 (conservatory_type=4) mis-scoring as a fabric part and
failing to persist on the NOT-NULL construction_age_band.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5 modelling_e2e cohort certs (full-SAP 19.1.0) were skipped with
"unmapped API sap_opening_type code: 6". Code 6 is "Ext Rooflight"; SAP 10.2
treats roof windows and rooflights as the same inclined-glazing family.
Fix: add 6 to the known opening-type taxonomy and route it onto the roof-window
path (`sap_roof_windows`) alongside code 5, via a `_SAP_ROOF_WINDOW_TYPES`
{5, 6} set. Genuinely-unknown codes (e.g. 99) still raise.
Cert 9878-3908-6309-6714-8200 now maps + calculates (sap 81). Regression test:
a type-6 opening maps onto sap_roof_windows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A SAP-Schema-17.0 full-SAP cohort cert (8265-7433-3220-9736-7902) omits the
top-level `has_hot_water_cylinder` and lodges it only under `sap_heating`. The
required top-level field on SapSchema17_1 made `from_dict` raise and skip the
cert.
Fix: make the top-level field Optional, and have the full-SAP mapper
(`from_sap_schema_17_1`, also used by 17.0/18.0.0 via delegation) fall back to
`sap_heating.has_hot_water_cylinder`. RdSAP mappers are unchanged (they keep the
top-level required field; their SapHeating has no such attribute).
Cert now maps + calculates (sap 76). Regression tests for the nested true/false
fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A SAP-16.x cohort cert (9258-4062-7265-2844-7954) lodges a bare int building-
part identifier (the second part as `1` after "Main Dwelling").
`BuildingPartIdentifier.from_api_string` regex-matched it assuming a string and
raised "TypeError: expected string or bytes-like object, got 'int'", failing the
whole property.
Fix: guard the match on `isinstance(api_identifier, str)` so a non-string
identifier falls to OTHER, matching the documented "anything unrecognised ->
OTHER" contract. The baseline SAP fabric sums all building parts regardless of
identifier, so OTHER is SAP-neutral; the identifier only labels parts for
measure targeting. Fixes every mapper (all route through from_api_string).
Cert now maps + calculates (sap 63). Regression test added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review follow-up (Khalim): the first pass made far more optional than needed —
notably the whole SapBuildingPart block — and a buggy 21.0.0↔21.0.1 diff also
MISSED open_chimneys_count / cfl_/led_fixed_lighting_bulbs_count / suggested_
improvements, so the original change actually mapped only 3 of the 33 skipped
certs (the rest still failed on open_chimneys_count).
Re-derived the exact set empirically from all 33 skipped cohort certs:
widen only fields that are (a) required in 21.0.0, (b) already optional in
21.0.1, AND (c) genuinely omitted by ≥1 of those certs. Result:
- KEEP optional: the 4 SapWindow refinements, the top-level vent/lighting/
door/pressure-test block (incl. the 3 previously-missed fields), 2
SapEnergySource fields, Addendum.addendum_numbers, PhotovoltaicSupply.
none_or_no_details, and exactly ONE building-part field
(SapBuildingPart.roof_insulation_thickness — omitted by 7 certs).
- REVERT to required: the other 12 SapBuildingPart fields (construction_age_
band, wall_construction, …), MainHeatingDetail.emitter_temperature,
PvBatteries.pv_battery, ShowerOutlets.shower_outlet — none of the 33 certs
omit these, so they stay strict.
Mapper: coalesce the count fields (wet_rooms_count, open_chimneys_count,
cfl_/led_fixed_lighting_bulbs_count) to 0 like every other mapper, so the now-
optional values can't reach a NOT-NULL column (also drops 4 pyright ignores).
Now maps 32/33 (up from 3); the last cert hits a pre-existing pv_batteries-
shape AttributeError and degrades via the ADR-0031 skip path. pyright net
unchanged (43, no new errors); regression test rewritten to the real omitted set.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
37 modelling_e2e properties failed on the 2026-06-23 run with
`NotNullViolation: null value in column "wet_rooms_count" of relation
"epc_property"`.
Root cause: 21.0.1 lodges `wet_rooms_count` as Optional, and
`from_rdsap_schema_21_0_1` passed it straight through
(`wet_rooms_count=schema.wet_rooms_count`). A cert omitting it mapped to
`EpcPropertyData.wet_rooms_count=None`. When a predicted EPC (which deep-copies
a comparable template's EpcPropertyData) inherited that None and was persisted,
it violated the `epc_property.wet_rooms_count` NOT-NULL column — and the calc's
`wet_rooms_count > 0` check would also raise `TypeError` on None.
Fix: coalesce to 0, matching every other mapper (RdSAP "not lodged" → the
calc's minimum 1 wet room). Regression test added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The modelling_e2e cohort skipped ~35 real RdSAP-Schema-21.0.0 certs with
"SapWindow: missing required field 'frame_factor'" (and a cascade of
wet_rooms_count, the mechanical-vent duct block, wind_turbine_details, &c.).
These are complete certs — the `RdSapSchema21_0_0` dataclass was simply
modelled stricter than `RdSapSchema21_0_1`, which already treats every one of
these fields as optional. This aligns 21.0.0's optionality with 21.0.1 (the
proven path) so the certs map into the prediction donor pool instead of being
dropped:
- SapWindow: pvc_frame / glazing_gap / frame_factor / window_transmission_details
→ Optional (calc already falls back: Table 6c frame factor, SAP10 glazing
lookup, type-only U).
- 20 nested fields across Addendum / MainHeatingDetail / PhotovoltaicSupply /
PvBatteries / SapBuildingPart / SapEnergySource → Optional, mirroring 21.0.1
verbatim (classes made kw_only so defaults sit in place; parsed only via
from_dict, keyword construction).
- from_rdsap_schema_21_0_0: guard the window_transmission_details dereference
for None (mirrors the 21.0.1 `_api_sap_window` guard).
Existing 21.0.0 certs are unaffected (relaxing optionality does not change
parsing of certs that carry the fields); mapper-corpus + accuracy gates green.
Verified end-to-end against real cohort cert 2205-3036-3484-0400-5718:
maps + calculates (SAP 68). Regression test added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MVHR (24a) heat-recovery support, part 2: the mapper + cascade wiring.
Both source paths now resolve balanced whole-house MV with heat recovery
to the MVHR kind:
- gov-API: `_API_MECHANICAL_VENTILATION_TO_KIND` 4 → "MVHR" (was None /
treated as natural — under-stated ventilation heat loss, over-rating).
- Elmhurst Summary: `_ELMHURST_MV_TYPE_TO_KIND` "Mechanical ventilation
with heat recovery (MVHR)" → "MVHR" (was UnmappedElmhurstLabel, which
blocked the whole Summary for MVHR dwellings).
cert_to_inputs resolves the in-use heat-recovery efficiency + SFP for an
MVHR cert (`_mvhr_system_values`): pick the PCDB Table 323 data point by
the lodged wet-room count (SAP 10.2 §2.6.4), multiply the raw efficiency
by the Table 329 ducts-inside-envelope in-use factor (0.90) and the raw
SFP by the per-duct-type factor (rigid 1.4), and feed:
- the §2.6.6 eq (2) effective-air-change credit (23c) → (24a)/(25)m;
- the (230a) fan electricity (in-use SFP × 1.22 × V), costed but NOT
added to the Table 5a gains (its effect is in the efficiency).
An MVHR lodged with no PCDF index falls back to the SAP 10.2 Table 4g
default (raw efficiency 66% × 0.70, raw SFP 2.0 × 2.5).
Worksheet-proven on simulated case 49 (000565 semi + Vent Axia Sentinel
Kinetic B 500140 + gas combi → Elmhurst Current SAP 72): every MVHR line
matches Elmhurst exactly — (33) fabric heat loss 100.5923, (23c) in-use
efficiency 81.9% = 91 × 0.90, (25)m Jan 0.8571, (230a) fan electricity
415.9325, (231) total pumps/fans 501.9325. The residual SAP 71 vs 72 is
the known 000565-family space-heating-demand artifact (same -1/-2 seen on
cases 47/48), not the MVHR logic.
Corpus: within-0.5 72.6% -> 72.7%, MAE 0.788 -> 0.782, PE 3.6 -> 3.5.
The 3 gov-API MVHR certs: Flat 1 +6 -> 0 (Table 4g default path) and
12a Princes Gate +3 -> +1 (heat-recovery credit); Apartment 707 -4 -> -6
is a separate baseline under-rate (it under-rated as natural too — the
MVHR credit correctly adds ventilation loss per Elmhurst's method).
Ratcheted _MAX_SAP_MAE 0.79 -> 0.785, _MAX_PE_PER_M2_MAE 3.7 -> 3.6.
Note: pyright strict type gate not run locally (pyright not installed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the mapper-coverage gaps surfaced by the modelling_e2e prediction-cohort
failures (portfolio 796):
- built_form (SAP-16.0): derive from dwelling_type in _normalize_sap_schema_16_x
(Mid-terrace->4, End-terrace->3, Semi-detached->2, Detached->1; flats->modal 4).
ML-only field (SAP calc never reads it) so SAP- and gate-neutral. 5 flat certs
that omitted built_form now map.
- photovoltaic_supply as a measured-array LIST: routed all pre-21 RdSAP mappers
(17.0/17.1/18.0/19.0/20.0.0) through _map_schema_21_pv, whose list branch is now
dict-tolerant (_pv_array_field reads dict OR dataclass). They capture the PV
arrays like 21.0.x instead of raising "'list' object has no attribute
none_or_no_details" and sinking the whole cohort.
- windows-as-dict (16.x): handled in the normalizer (not just windows-as-list).
Genuinely-sparse certs (omit door_count/habitable/glazed_area) remain fail-loud;
the gate-regressing multiple_glazed_proportion default and the recursive
RdSAP-21.0.0 ADR-0028 alignment are left fail-loud + flagged for review (worklist).
+5 regression tests; component-accuracy gate 26/26; 0 new pyright errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
