Mapper-drop audit across the 9-fixture cohort: `percent_draughtproofed`
is lodged on 9/9 certs (raw values 85-100) but the schema-21.0.1
mapper never set it on EpcPropertyData. The site-notes mappers always
have (line 312 of mapper.py); only the API path was missing.
cert_to_inputs reads `epc.percent_draughtproofed` for the §2
ventilation cascade (window draught loss); with None → 0 default, the
calc was treating every API-routed cert as fully draughty —
over-counting draught infiltration on every fixture in the cohort.
Fix: `percent_draughtproofed=schema.percent_draughtproofed` in
`from_rdsap_schema_21_0_1`.
Cohort SAP / PE / CO2 shifts (all 9 fixtures move; many shift one
SAP point because the continuous SAP was near a rounding boundary):
cert old SAP new SAP PE shift CO2 shift
0240-0200-5706-2365-8010 -12 -10 -7.63 -0.39
0300-2747-7640-2526-2135 -9 -7 -6.36 -0.55
0390-2254-6420-2126-5561 (LN12) 0 +1 -9.10 -0.13
0390-2954-3640-2196-4175 -7 -4 -4.87 -0.44
2130-1033-4050-5007-8395 (DE22) +8 +9 -3.67 -0.04
6035-7729-2309-0879-2296 -6 -5 -8.90 -0.21
7536-3827-0600-0600-0276 +3 +4 -9.19 -0.24
8135-1728-8500-0511-3296 +1 +1 (cont -7.48 -0.14
72.7→73.5)
9390-2722-3520-2105-8715 +2 +3 -7.32 -0.01
LN12 lost its exact-SAP-match (0 → +1, continuous 65.47 → 66.28); the
other fixtures' rounded SAP residuals tightened or worsened by 1
depending on which side of the rounding boundary they sit. This is
spec-correctness over residual-tightness: the lodged value is correct,
our calc now reads it.
930/930 Elmhurst cascade green. 78/78 mapper tests + 14/14 golden
cohort + PCDB chain green. Pyright net-zero.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Audit of raw-JSON keys vs RdSapSchema21_0_1 across the 9-fixture
golden cohort surfaced 7 vent / draught fields silently dropped at
deserialization: blocked_chimneys_count, open_flues_count,
closed_flues_count, boilers_flues_count, other_flues_count, psv_count,
has_draught_lobby. cert_to_inputs reads all of them for the §2
infiltration cascade; without them the calc treats every dwelling as
flue-free / vent-free / no draught lobby and under-counts ACH.
Fix: declare the 7 fields on RdSapSchema21_0_1; extend the mapper to
surface blocked_chimneys_count on EpcPropertyData top-level (already
declared) and the other 6 on SapVentilation (extends the slice 37
extract_fans_count work). has_draught_lobby coerces "true"/"false"
strings to bool to match the SapVentilation type.
Cohort residual shifts after re-pinning:
- LN12 (0390-2254) — SAP +1 → 0 (FIRST CERT TO HIT LODGED SAP EXACTLY).
blocked_chimneys=2 reduces infiltration, tightens both SAP and PE
(PE −10.62 → −3.14, CO2 −0.11 → +0.04).
- 0300 — PE +18.92 → +17.34, CO2 −0.43 → −0.54 (open_flues=1 +
has_draught_lobby=true cross-cancel near-zero).
- 0390-2954 — PE −25.62 → −27.64, CO2 −2.45 → −2.58 (has_draught_lobby=true).
- 8135 — PE −17.58 → −14.37, CO2 −0.22 → −0.15 (blocked_chimneys=1).
- Other 5 fixtures (0240, DE22, 6035, 7536, plus retired 9390): no shift
— their certs lodge zeros or no vent fields beyond what Slice 37 plumbed.
Rounded-SAP cohort distribution post-slice:
0 (LN12), +1 (8135), +2 (9390), +3 (7536), +8 (DE22, spec-drift),
-6 (6035), -7 (0390-2954), -9 (0300), -12 (0240, RR-driven).
Schema scope: 21.0.1 only. 21.0.0 schema's SapBuildingPart shares the
same mapper code but no 21.0.0 fixtures live in the cohort to anchor
against; defer to a future slice if needed.
930/930 Elmhurst cascade green. 14/14 golden cohort green at new
pinned residuals. 77/77 mapper tests green. Pyright net-zero (34
errors before and after).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Schema-21.0.0/0.1's SapRoomInRoof dataclass declared only floor_area
and construction_age_band. Real certs lodge gable wall lengths under
sap_room_in_roof.room_in_roof_type_1 (RdSAP §3.9.1 Simplified Type 1).
from_dict silently dropped the whole block at deserialization, so the
mapper never had a chance to surface the lengths on EpcPropertyData.
Fix: add RoomInRoofType1 dataclass to both schema-21 variants;
extend SapRoomInRoof with `room_in_roof_type_1: Optional[...]`;
update the mapper to populate EpcPropertyData.SapRoomInRoof
gable_1_length_m / gable_2_length_m from the new field.
Calculator behaviour unchanged this slice: heat_transmission.py:243
requires BOTH length AND height to contribute gable area, and the
cert lodges length only (RdSAP §3.9.1 uses a default 2.45 m storey
height — not yet plumbed). Cert 0240's −12 SAP residual unchanged.
Schema scope: both 21.0.0 and 21.0.1 schemas (identical SapBuildingPart
mapper code, kept consistent). Older schemas (17/18/19/20) don't carry
this RR shape on their dataclasses and are out of scope per the prior
cohort scope decision.
Unblocks the follow-up slices that close the RR cascade: default
H_gable in calculator or mapper, parse "Roof room(s), insulated
(assumed)" description for the U-value override, etc.
930/930 Elmhurst cascade green. 14/14 golden cohort green at pinned
residuals (no shift, as expected). 76/76 mapper tests green.
Pyright net-zero (32 errors before and after).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 21.0.1 mapper produced EpcPropertyData with sap_ventilation=None,
so the cert→inputs cascade defaulted every ventilation count to zero
even when the cert lodged extract fans (most schema-21 certs do).
extract_fans_count was double-mapped — surfaced as a top-level field
the calculator never reads, but missing from the SapVentilation slice
the cascade does read.
Fix: populate sap_ventilation in from_rdsap_schema_21_0_1 with
extract_fans_count. Drives ~⅓ of the rating-cohort drift on a clean
no-PV no-secondary gas-combi cert.
Refactored test_golden_fixtures.py from global tolerance ceilings
(±13 SAP / ±35 PE) to per-cert pinned residuals at abs SAP=0,
PE=0.01 kWh/m², CO2=0.001 t/yr. Each cert's _GoldenExpectation now
records the actual current residual (SAP/PE/CO2 — CO2 newly pinned
via the postcode-cascade environmental section). Drift in either
direction fires the test: tighten the pin on improvement, document
on regression.
Recorded residuals reflect known remaining mapper gaps (RR room-in-
roof extraction on cert 0240, oil cascade on 0390, etc.) — tracked
in each cert's notes: field, not acceptance bounds.
930/930 Elmhurst cascade pins unchanged (site-notes EPCs already
populate sap_ventilation). 257/257 mapper tests green. 10/10 golden
cohort green under the new pins. Pyright net-zero (34 errors before
and after).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes 000516's §3 LINE_33 0.8215 W/K rooflight gap. Adds SapRoofWindow to
EpcPropertyData (area + raw U from RdSAP10 Table 24 "Roof window" column,
p.50/113) and iterates them in heat_transmission_from_cert alongside vertical
windows — same SAP10.2 §3.2 curtain transform R=0.04. Rooflight area is
subtracted from the main part's roof gross so net (30) + (27a) = original
gross, leaving (31) area aggregate invariant.
000516 LINE_33 residual: 0.8215 W/K → 0.0038 W/K. Remaining 0.0038 is the
same pre-existing wall-perimeter + per-window curtain precision drift biting
000474/477/480/490 (slice 27).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds `SapRoomInRoofSurface` dataclass (kind + area + insulation thickness
+ insulation type) and an optional `detailed_surfaces` list on
`SapRoomInRoof`. When `detailed_surfaces` is present, the Simplified
A_RR formula is bypassed and the calculator iterates each surface,
applying the appropriate Table 17 / Table 4 U-value:
slope → roof_w_per_k via u_rr_slope (Table 17 col 1)
flat_ceiling → roof_w_per_k via u_rr_flat_ceiling (Table 17 col 2)
stud_wall → roof_w_per_k via u_rr_stud_wall (Table 17 col 3)
gable_wall → party_walls_w_per_k at U=0.25 (Table 4 "as
common wall")
This mapping mirrors the U985 worksheet for 000477 where RR stud walls
+ slope + flat-ceiling lines sit under (30) and RR gable walls sit
under (32). The §3.9 deduction of `A_RR_floor` from the storey-below
roof area still applies.
Synthetic test pins a 1-storey + RR dwelling with 4 detailed surfaces
(slope/stud_wall/flat_ceiling/gable_wall) at hand-computed U-values
from Table 17 and Table 4, abs=0.001 tolerance.
Reference: RdSAP 10 (10-06-2025) §3.10 page 24-25; Figure 4; Table 17
page 44; Table 4 page 22.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extends `SapRoomInRoof` with six optional fields capturing the RdSAP10
§3.9.2 Simplified Type 2 lodgement: common_wall_length_m / height_m
plus two gable length/height pairs.
Type 2 fires when `common_wall_height_m` is set and < 1.8 m (otherwise
the space is a separate storey). Geometry per spec page 23:
A_common_wall = L × (0.25 + H)
A_gable = L × (0.25 + H_gable)
− Σ ((H_gable − H_common_wall_i)² / 2)
A_RR_final = A_RR − Σ A_common_wall − Σ A_gable
(− party / sheltered / connected when lodged, future
slice when a fixture exercises them)
Common walls and gables route to walls_w_per_k at U_main_wall (per spec:
"Common wall U-value is inferred from the U-value of the main wall in
the building part below"). A_RR_final routes to roof_w_per_k at
u_rr_default_all_elements (Table 18 col 4).
Synthetic test: 1-storey cavity-uninsulated dwelling at age B + RR
(floor 10 m², common_wall_length 5 m × 1 m height). Pins
walls_w_per_k = 60 × 1.5 + 6.25 × 1.5 = 99.375 W/K and
roof_w_per_k = 30 × 0.40 + 26.025 × 2.30 = 71.857 W/K at abs=0.001.
No production fixture exercises Type 2 yet — synthetic test is the
unit-level guard until a Type 2 cert lands in the corpus.
Reference: RdSAP 10 (10-06-2025) §3.9.2 page 22-23.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Surfaces four cert lodgements that the §2 ventilation cascade was
missing on the cert→inputs path. Without them, `cert_to_inputs` was
defaulting:
- extract_fans_count → 0 (PDF: 1-2 fans per fixture)
- percent_draughtproofed → 0 (PDF: 75-100% per fixture)
- sheltered_sides → 2 (PDF: 1-3 per fixture — hardcoded TODO)
- has_suspended_timber_floor → False (PDF: True on 000477/000487)
Net effect on (25)m monthly effective ACH ranged from -19% (000477)
to +5% (000490) → propagated 1:1 through HLC × ΔT → useful space heat
→ main + secondary fuel kWh → cost / SAP integer.
Schema:
- `SapVentilation` gains 4 new optional fields: `sheltered_sides`,
`has_suspended_timber_floor`, `suspended_timber_floor_sealed`,
`has_draught_lobby`. RdSAP cert lodges these but the type didn't
surface them.
- `cert_to_inputs.cert_to_inputs` reads them when set; falls back to
the SAP10.2 §2 worst-case defaults (sheltered=2, no timber floor,
no draught lobby) when the cert hasn't lodged. Removes the long-
standing `sheltered_sides=2` hardcode + 4 TODOs.
- `make_minimal_sap10_epc` accepts a `sap_ventilation` kwarg.
Per-fixture build_epc() updates lodge the U985 PDF values verbatim.
E2E pin: new parametrized test
`test_elmhurst_cert_to_inputs_monthly_infiltration_ach_matches_u985_
worksheet` asserts `inputs.monthly_infiltration_ach[m] == LINE_25_
EFFECTIVE_ACH[m]` at abs=1e-3 across all 6 fixtures + 12 months
(72 assertions). All pass.
Useful space heating drift:
000474: useful 10821.69 → 10765.85 (Δ -55.8 kWh vs PDF 10612.86 → +1.4% over, was +2.0%)
000490: useful 11262.05 → 11184.06 (Δ -78.0 kWh vs PDF 11183.28 → +0.007% — essentially exact)
SAP integer status:
000474: 62 = PDF 62 (delta 0) ✓
000490: 58 vs PDF 57 (delta 1; continuous 57.77 vs 57.40)
— remaining residual is pumps_fans hardcoded at 130 kWh
vs PDF 160 (Table 4f cascade not yet implemented → -£4 cost
+ 0.3 continuous SAP). Next slice.
Tightens `result.secondary_heating_fuel_kwh_per_yr` pin abs=10 → abs=0.1
(was loose to absorb the +0.7% useful overshoot which has now closed).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SapFloorDimension gains an is_exposed_floor flag (default False) signalling
that the floor sits over outside air or unheated space rather than soil —
typical for an extension that hangs off the main from the first storey
upward (Elmhurst 000490 Extension 1 is exactly this shape).
heat_transmission_from_cert now consults the flag on the part's ground
SapFloorDimension and dispatches to u_exposed_floor (Table 20) instead
of the BS EN ISO 13370 / Table 19 cascade. Basement floor still wins
priority (Table 23 § 5.17 overrides everything else for that part).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Lands the production code that the just-committed Elmhurst conformance
fixtures (6455d48b) exercise: the SAP10.3 calculator orchestrator
(domain.sap.calculator.Sap10Calculator), the RdSAP-driven cert→inputs
mapper (domain.sap.rdsap.cert_to_inputs), and the EpcPropertyData
strict-type pass that P6.1 starts.
calculator.py is the entry point. Two surfaces depending on the caller's
shape:
- Sap10Calculator().calculate(epc) — full RdSAP mapper + worksheet loop
- calculate_sap_from_inputs(inputs) — pure physics over typed inputs
P6.1 introduces BuildingPartIdentifier as a strictly-typed replacement
for bare-string matching on SapBuildingPart.identifier (motivated by
the pain point at worksheet/dimensions.py:74-82). Two boundary factories
canonicalise raw inputs: from_api_string for the gov-EPC API, and
extension(n) for site-notes / construction id flows.
Also catches up two transitive deps that 6455d48b implicitly required
but I missed:
- ml/rdsap_uvalues.py — party-wall U-value rows that heat_transmission
resolves; the U=0.0 branch the 000516 fixture exercises lands here.
- ml/tests/_fixtures.py — make_minimal_sap10_epc that every Elmhurst
fixture imports. Without this catch-up, checking out 6455d48b in
isolation would ImportError.
Out of scope (will commit separately): ml/transform.py legacy envelope
drift; backend/ FastAPI + documents_parser layer; etl/ scratch.
824 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
15 new features wired through schema -> domain -> mapper -> transform:
Main Dwelling fabric (11):
- wall_insulation_type, wall_insulation_thickness_mm, wall_dry_lined,
wall_thickness_mm, party_wall_construction
- roof_insulation_location, roof_insulation_thickness_mm
- floor_construction, floor_insulation, floor_insulation_thickness_mm,
floor_heat_loss
Dwelling-level scalars (4):
- multiple_glazed_proportion, number_baths, number_baths_wwhrs,
extract_fans_count
Thickness strings like '50mm'/'NI'/'ND' parsed via _parse_thickness_mm; NI
(no insulation) lands as 0mm so the model sees the physical zero rather than
a missing value. Categorical sentinels ('NA'/'NI'/'ND') become None.
Also fixed long-standing typo `multiple_glazed_propertion` -> `_proportion`
in domain dataclass + its lone DB-model usage.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two production fixes surfaced by the live run:
- mapper.from_rdsap_schema_21_0_1 now sets the three ML target scalars
(energy_rating_current, co2_emissions_current, energy_consumption_current).
They were silently None for every cert before, leaving the only labels as
the kWh fields from renewable_heat_incentive.
- train_baseline coerces object-dtype columns to numeric (None -> NaN) and
drops rows with null target per fit, so LightGBM accepts the frame.
E2E on 500 real certs (~1s):
sap_score R^2=0.604 MAPE=0.084
co2_emissions R^2=0.813 MAPE=0.130
peui_raw R^2=0.979 MAPE=0.026
space_heating_kwh R^2=0.823 MAPE=0.213
hot_water_kwh R^2=0.519 MAPE=0.115
peui_ucl excluded: UCL correction still needs wiring.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Currently fails on SapWindow.glazing_gap (first of ~30 fields the dataclass
incorrectly treats as required). Will go GREEN once 14j sweeps Optional.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SAP10 EPCs with measured PV carry photovoltaic_supply as a nested
list of arrays (peak_power, pitch, orientation, overshading) rather
than the legacy unmeasured wrapper {none_or_no_details:
{percent_roof_area: N}}. The schema-21 dataclasses now accept both
shapes via Union[PhotovoltaicSupply, List[List[PhotovoltaicArray]]],
and from_dict._coerce now dispatches list values onto list type
variants of multi-type Unions.
EpcPropertyData.SapEnergySource gains
photovoltaic_arrays: Optional[List[PhotovoltaicArray]] — populated
when the measured shape is present, otherwise None. The legacy
photovoltaic_supply field is preserved for the fallback case.
Both schema-21.0.0 and 21.0.1 mappers dispatch via the new
_map_schema_21_pv helper.
Unblocks Slice 11 (PV feature aggregation in EpcMlTransform).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Thirteen window-aggregate features land on the transform: count,
total area, eight SAP-octant area columns (N/NE/E/SE/S/SW/W/NW),
area-weighted draught-proofing pct, and area-weighted u_value +
solar transmittance (nullable, populated only when windows carry
transmission_details). Windows with orientation outside 1-8 (0,
NR) contribute to count and total area but no octant.
Also: epc codes CSV (gov api /api/codes export, RdSAP-Schema-21.x +
older versions) moved next to EpcPropertyData as epc_codes.csv —
canonical SAP enum source for upcoming categorical-share slices.
.gitignore exception added so the reference CSV is tracked.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
