Some gov-API RdSAP-21.0.1 certs (e.g. 2461-5385-9264-6821-5357, property_id=750232,
portfolio 824) omit `built_form` from the response entirely rather than sending a
code. `from_dict` hard-fails any dataclass field with no default that's absent
from the raw dict, so the whole cert failed to parse.
built_form: Optional[int] = None (kw_only, matching the has_fixed_air_conditioning
precedent already on this dataclass) — the downstream SAP-cascade consumer
(_api_sheltered_sides) already treats a non-int built_form as "no lodging" and
falls back to the documented default, so None here is an already-safe
degradation, not a new failure mode. Also guards mapper.py's
`built_form=str(schema.built_form)` against stringifying None into "None".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
datatypes/epc/domain/mapper.py's from_rdsap_schema_{17_0,17_1,18_0,19_0,
20_0_0,21_0_0} all stored a lodged sap_windows[].glazing_type verbatim
instead of routing it through _api_cascade_glazing_type (which
_api_sap_window, used by 21_0_1, already does correctly). The raw API code
is the RdSAP-21 glazing enum (e.g. code 1 = "DG pre-2002"), not the SAP
10.2 Table 6b cascade enum the calculator's U/g-value tables are keyed on
(cascade code 1 = single glazed) -- so any lodged window whose raw code
happened to collide with a different cascade meaning was silently
mis-rated.
Found while validating PR #1503 (property 753950 / uprn 100021969385)
against Elmhurst: our engine's windows_w_per_k was 82.27 vs Elmhurst's
51.44 -- the dwelling's larger window (91% of glazed area) lodges raw
glazing_type=1 and was being modelled as single glazed instead of double,
the dominant contributor to a SAP 42 vs Elmhurst 44 gap. Fixing this
narrows windows_w_per_k to 50.26 (Elmhurst 51.44, ~2% off) and moves the
engine's score to 43.
Extracted the 5 duplicated reduced-schema-window blocks (17.0/17.1/18.0/
19.0/20.0.0, which lodge a minimal window shape with no glazing_gap/
frame_factor/transmission fields) into a shared _reduced_field_api_sap_
window helper that also now populates window_transmission_details/
frame_factor via the SAP10 lookup instead of leaving them None. 21.0.0's
richer inline block gets the same one-line cascade fix.
+6 regression tests (one per affected schema seam + 21.0.0). Full domain/
schema/prediction/comparable-properties/sap10_calculator/modelling_e2e
suites pass (2746 passed, only the 2 pre-existing unrelated failures);
component-accuracy gate 26/26 unaffected.
Property 753950 (uprn 100021969385, cert 0141-2860-6891-9124-5625,
SAP-Schema-16.2) hard-failed modelling_e2e: RdSapSchema17_1 requires
multiple_glazed_proportion as a non-optional int, and this 16.x cert omits
the field entirely, lodging only the multiple_glazing_type="ND" sentinel.
The cert's own window.description ("Fully double glazed") states the
glazing extent unambiguously, so _normalize_sap_schema_16_x now derives
0/100 from "single"/"double" wording when the field is absent, mirroring
the existing single-glazed multiple_glazing_type cascade. This is
worklist P4 (.claude/skills/expand-sap-accuracy-corpus/worklist.md) —
a flat default was tried previously and reverted because making such
certs mappable at all pulls them into the EPC-prediction donor pool and
tips near-tie similarity matches; deriving from explicit text (rather
than a blind default) was the suggested unblock.
Re-measured the component-accuracy gate as the worklist asked: it drops
(has_hot_water_cylinder 0.8687->0.8586, cylinder_insulation_type
0.3333->0.1667, door_count residual 0.3131->0.3333) via the same
donor-pool-composition mechanism as the prior #1245/ADR-0037
re-baselines, not a prediction-logic loosening. Re-baselined the floors
with that rationale recorded inline.
Address PR review (dancafc):
- introduce UprnMatch NamedTuple (datatypes/address_match.py) for the
(uprn, address, lexiscore, certificate_number) return, replacing the bare
4-tuple in get_uprn_from_epc_df / get_uprn_from_historic_epc /
HistoricEpcResolver.resolve_uprn. Tuple-compatible, so unpacking is unchanged.
- rename get_uprn_with_epc_df -> get_uprn_from_epc_df (+ callers).
- type resolve_group_ambiguity via a GroupDecision NamedTuple and trim its
docstring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The row→domain mapper now names all 93 constructor arguments explicitly
instead of splatting a lowercased dict, takes a plain Mapping (a
DataFrame.to_dict("records") row) instead of a pandas Series, and ignores
columns the domain type doesn't know. A missing/renamed CSV column fails
loudly as a KeyError at the row. Both iterrows() call sites move to
to_dict("records") — pandas-stubs types iterrows' Series unparameterized,
which strict mode rejects. pandas-stubs + boto3-stubs[s3] make the stack
check clean: pyright strict is now 0 errors across the PR's files.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RdSAP 10 Specification (10-06-2025) §5.2.5: "If a dwelling or part of a
dwelling has commercial premises above record as another dwelling
above" — so code 9 ("another premises above") must get the same
zero-heat-loss treatment as code 7 ("another dwelling above"), per the
spec's own "there is no heat loss through the roof of a building part
that has the same dwelling above or another dwelling above."
The previous fix mapped both to None, which only avoided the mapper
crash — it didn't achieve zero heat loss. heat_transmission.py's
per-part suppression (part_roof_is_party) only fires when
roof_construction_type CONTAINS the literal string "another dwelling
above"; that string was only ever set on the Elmhurst ingestion path.
The API path mapped 7 (and, until now, 9) to None, so the suppression
never fired for any API-derived cert — has_exposed_roof (the
dwelling_type-label-only fallback) was the sole gate, with no way to
know about a per-part party-ceiling lodgement. Both codes now map to
"(another dwelling above)" so the existing suppression logic fires on
both ingestion paths, matching what
domain/epc/property_overrides/roof_party_ceiling_guard.py already does
independently for the landlord-override pipeline (same spec citation).
Verified against the motivating cert (uprn 10013320122): roof_w_per_k
now correctly computes to 0.0. Full RdSAP-21.0.1 corpus (1000 certs)
re-run: within-0.5 unchanged at 77.8%, MAE improves marginally
(0.6364 vs 0.637 ceiling) — no regression.
Same party-ceiling semantics as the already-mapped code 7 ("another
dwelling above" / "same dwelling above") — the unit has no exposed
roof, so there's nothing for the cascade to compute a U-value for.
Confirmed against the live cert (uprn 10013320122): roofs[].description
= "(another premises above)", roof_insulation_location = "ND" (the
codebase's established "no exposed roof" signal).
Surfaced by 1 failed modelling_e2e subtask on portfolio 817
(task_source='modelling_e2e', job_completed > '2026-07-03 12:50:37.555+01:00').
Some lodged certs omit the sap_heating.has_fixed_air_conditioning key,
which the generic from_dict loader treats as a required field and
rejects outright. Two prior commits (26651ca71, 8074f4152) fixed the
duplicate top-level field of the same name but left this nested copy
untouched; a third pass (44991bed0) missed it too. The mapper already
reads it defensively (== "true", which is False on None), so this is
a safe default.
Surfaced by 7 failed modelling_e2e subtasks on portfolio 817.
`from_sap_schema_17_1` hardcoded is_dwelling_export_capable=False, so
SAP 10.2 Appendix M1 zeroed the PV export credit on every export-capable
full-SAP cert. Cert 0380-3044-6070-2305-5925 (property 741840, PRD #1435
WS3) now reproduces its lodged 92 exactly (was 89); the existing
19.1.0 pin 10096028301 closes its documented -1 residual (84 -> 85 =
lodged). The full-SAP top-level pv_connection is deliberately NOT
carried: its enum disagrees with the RdSAP one the calculator gates on.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A pre-SAP10 RdSAP cert that omits per-window records lodges glazing as a
`multiple_glazing_type` code plus a `window.description` and a
`multiple_glazed_proportion`. When the numeric code is the "ND" (Not
Defined) sentinel, the reduced-field synthesis defaulted to the DG-modal
code 2 (double) — ignoring the two register signals that say otherwise.
`_normalize_sap_schema_16_x` already honours an explicit "Single glazed"
description for 16.x certs (rewriting multiple_glazing_type -> 5), but that
rule was never propagated to the 17.0 / 17.1 / 18.0 / 19.0 / 20.0.0
synthesis seams — they all blindly applied ND -> 2. So a genuinely
single-glazed dwelling was:
1. scored as double-glazed, overstating its baseline SAP, and
2. hidden from the glazing generator (which upgrades only single-glazed
windows), producing no glazing recommendation.
Reproduced on cert 2780-3922-3202-6042-9200 (uprn 10033526327), a
single-glazed community-heated flat lodging multiple_glazing_type="ND",
window "Single glazed", multiple_glazed_proportion 0 — mapped to four
double-glazed windows, no glazing measure offered.
Extend the 16.x rule to all five reduced-field seams via one shared
resolver: on the ND fallback, when the description says "single" or the
multiple-glazed proportion is 0, synthesise single glazing
(`_api_cascade_glazing_type(5) == 1`, SAP10 cascade single, Table 24
U~4.8) — the exact single code the glazing generator's {1, 15} set
matches — else keep each seam's DG-modal default. Refines ADR-0028's ND
handling.
Tests: all five seams synthesise single from a "Single glazed" + 0%
proportion cert; the double-glazed control keeps the DG-modal default;
plus unit pins on the resolver.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PRD #1385 phase 2. The 20.0.0 and 21.0.0 mappers dropped insulated_door_u_value
(top-level Optional float), unlike 21.0.1. heat_transmission reads it as the
measured door U-value override (cert_to_inputs -> heat_transmission:809), so
dropping it forced every door onto the RdSAP default U-value.
Verification: new test 3 passed (21.0.0 sample lodges 3; 20.0.0 latent -> inject
1.5 + absent -> None); corpus 6002 passed; SAP-accuracy regressions pass; pyright
unchanged (39 -> 39).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PRD #1385 phase 2. The 21.0.0 mapper dropped mechanical_ventilation_index_number
and mechanical_vent_duct_type, unlike 21.0.1. cert_to_inputs reads both (PCDB
Table 4f/329 heat-recovery/SFP lookup + the Table 329 in-use factor), so an
MVHR/MEV cert on the 21.0.0 path fell back to the SAP default efficiency.
Mirrors 21.0.1 exactly -- only the two fields it actually maps. The other four
mechanical_vent_duct_* schema fields have no mapper OR calculator consumer even
on 21.0.1 (verified), so they're deliberately left alone rather than populating
dead domain fields.
Verification: new test 1 passed (harvested 21.0.0 sample lodges index=12,
duct_type=3); corpus 6002 passed; SAP-accuracy regressions pass; pyright
unchanged (39 -> 39).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PRD #1385 phase 2. 21.0.0 was the last mapper still dropping the lodged
percent_draughtproofed (top-level int) — every other path (17.0/…/21.0.1) reads
it. Dropping it understated the SAP §2 draughtproofing/infiltration credit for
21.0.0 certs.
Verification: new test 1 passed (harvested 21.0.0 sample lodges 100); corpus
6002 passed; SAP-accuracy regressions pass; pyright unchanged (39 -> 39).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PRD #1385, building-part fabric family. The 17.0 mapper hardcoded
floor_insulation_thickness / sap_room_in_roof to None and omitted
flat_roof_insulation_thickness, all of which the 17.0 schema lodges and the
19.0/20.0/21.x paths read:
* floor_insulation_thickness: "NI" (no measured thickness) -> None so u_floor
reaches its age-band default, else pass through (mirrors the 19.0 note).
* flat_roof_insulation_thickness: pass through (was dropped).
* sap_room_in_roof: map to SapRoomInRoof(floor_area, construction_age_band),
None if absent (mirrors 19.0).
17.0-only: the 17.1 SapBuildingPart schema has none of these fields, so its None
there is correct (verified against the dataclass). roof_insulation_thickness is
intentionally left raw -- the sloping-ceiling resolve helper moves roof U-values
and is a separate follow-up.
Verification: new test 3 passed; test_mapper_corpus 6002 passed; SAP-accuracy /
RealCertExpectation regressions pass; pyright unchanged (39 -> 39, pre-existing).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PRD #1385, shower/WWHRS family. The 17.0 and 19.0 API mappers left
sap_heating.number_baths / mixer_shower_count as None, so the water-heating
demand (cert_to_inputs) fell back to defaults instead of the lodged bath/shower
room counts. Derive them from schema.sap_heating.instantaneous_wwhrs, mirroring
the 17.1/18.0/20.0 paths:
number_baths = rooms_with_bath_and_or_shower + rooms_with_bath_and_mixer_shower
mixer_shower_count = rooms_with_mixer_shower_no_bath + rooms_with_bath_and_mixer_shower
Guarded on instantaneous_wwhrs is not None (Optional in the schema), like 17.1.
Counts, same meaning across the reduced-field family.
Verification: new test 2 passed; test_mapper_corpus 6002 passed; SAP-accuracy /
RealCertExpectation regressions pass; pyright unchanged (39 -> 39, pre-existing).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PRD #1385, heating-controls family. The 17.0 API mapper omitted
main_heating_controls entirely, leaving the domain field None while the
19.0/20.0/21.x paths map the first lodged control via _map_energy_element. 17.0
lodges main_heating_controls (List[EnergyElement]) in all 23 local fixtures.
The mapped element is the same EnergyElement type 17.0 already carries for
main_heating / window, so this is straight parity (not new logic). Guarded on a
non-empty list -> None, matching the siblings.
Verification: new test 2 passed; test_mapper_corpus 6002 passed; recommendation
+ heating-control tests 85 passed (the domain EnergyElement field is distinct
from the subscriptable dict HeatingControlRecommender reads via the Property
wrapper, so no recommender destabilisation); SAP-accuracy regressions pass;
pyright unchanged (39 -> 39, pre-existing).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PRD #1385, lighting family. The 17.0 and 19.0 API mappers hardcoded every
fixed-lighting bulb count to 0, dropping the lodged lighting mix that the
reference-complete 17.1/18.0/20.0 paths read. The schema lodges total +
low-energy OUTLET counts (ADR-0028); mirror the reference split — low-energy
outlets → low_energy bulbs, the remainder → incandescent — so the SAP
lighting-energy calc sees the real low-energy credit instead of assuming none.
led/cfl stay 0 (reduced-field certs don't split those); 21.0.x use a different
bulb-based lighting schema and are out of scope. The `total - low_energy`
subtraction is safe: all 37 local 17.0/19.0 fixtures have low_energy <= total
(no negative incandescent), matching the reference paths that already do this
unclamped.
Verification: new test 2 passed; test_mapper_corpus 6002 passed; SAP-accuracy /
RealCertExpectation regressions 56 passed (no pinned shifts); pyright unchanged
(39 -> 39, pre-existing).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PRD #1385, ventilation family. The 17.0 and 19.0 API mappers dropped lodged
ventilation fields that the reference-complete 17.1/18.0/20.0 paths read:
* open_fireplaces_count -> open_chimneys_count: hardcoded 0 on both
(understating §2 infiltration).
* percent_draughtproofed: omitted on both.
* 17.0 additionally passed no sap_ventilation block, so the §2 cascade fell
back to NATURAL + its default sheltered_sides=2. Restored from the lodged
mechanical_ventilation + built_form via the existing _api_* helpers.
Semantic parity verified (not just type-equality): built_form and
mechanical_ventilation use the same gov code lists the 17.1/18.0/19.0/20.0/21.x
paths already map through these helpers; all 244 RdSAP fixtures resolve with zero
UnmappedApiCode, and the strict-coverage guard raises on any future divergence.
For the current corpus the 17.0 score-mover is sheltered_sides from built_form
(every 17.0 cert lodges mechanical_ventilation=0 -> NATURAL).
Adds a regression test asserting both paths carry the fields through
from_api_response on real certs.
Verification: new test 2 passed; test_mapper_corpus 6002 passed; SAP-accuracy /
RealCertExpectation regressions pass; pyright unchanged (39 -> 39, pre-existing).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`from_rdsap_schema_17_0` was an older copy that hardcoded a cluster of
lodged `sap_heating` fields to None while every other API path
(19.0/20.0/…) reads them from `schema.sap_heating`. PR #1380 restored the
secondary-heating pair; this restores the rest of the same block, all of
which `RdSapSchema17_0` lodges:
MainHeatingDetail: boiler_flue_type, fan_flue_present,
central_heating_pump_age, main_heating_index_number (PCDB efficiency index)
SapHeating: cylinder_thermostat, cylinder_insulation_thickness
Impact — cylinder_thermostat (12/23 local 17.0 fixtures) read as "no
thermostat", so the water-heating worksheet over-applied its ×1.3 penalty
and `recommend_cylinder_thermostat` could fire on a dwelling that already
has one; cylinder_insulation_thickness (13/23) discarded the measured value
so the cylinder-loss cascade fell back to age-band defaults;
main_heating_index_number lost the PCDB efficiency lookup.
Mirrors the 19.0 path exactly. secondary_*_type left untouched so this PR
stays independent of #1380 (no overlapping lines). Adds a regression test
asserting the whole block survives `from_api_response` on a real 17.0 cert.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `from_rdsap_schema_17_0` path hardcoded `sap_heating.secondary_heating_type`
and `secondary_fuel_type` to None, unlike every other API path (19.0/20.0/…),
which read them from `schema.sap_heating`. The gov EPC API *does* populate a
secondary heating SAP code for a genuine fixed secondary (e.g. 691 "electric
room heaters", 694), so the hardcode silently discarded it.
Because `recommend_secondary_heating_removal` gates purely on
`sap_heating.secondary_heating_type`, any 17.0 cert carrying a fixed secondary
would never get its removal recommendation. Surfaced while auditing portfolio
814's recommendations; 814 has 17.0 certs but none with a fixed secondary, so
the defect was latent there — this restores parity with the other paths so it
cannot bite.
Mirrors the 19.0 path exactly (secondary_fuel_type via `_api_secondary_fuel_type`).
Adds a regression guard that injects a fixed secondary into a real 17.0 cert and
asserts the code survives the mapping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HistoricEpcS3Repository reached into utils/s3.py (read_csv_gz_from_s3 +
parse_s3_uri), the legacy utility that self-constructs boto3 inside free
functions. The other S3 repositories deliberately depend on the
infrastructure/s3 layer instead (UnstandardisedAddressListCsvS3Repository
injects a CsvS3Client). Bring historic EPC into line.
- Add GzipCsvS3Client(S3Client) in infrastructure/s3: read_csv_gz(key) ->
DataFrame (get_object + gzip decode).
- Inject it into HistoricEpcS3Repository; the bucket lives in the client and
the repo only builds the per-postcode key + maps rows (no S3/HTTP code).
Add with_default_s3_client(s3_root) for composition roots.
- Update main.py and the match_addresses_for_postcode seam to the factory.
- Repo tests inject a real GzipCsvS3Client over a controlled boto stub
(exact key assertions + AccessDenied); add a moto-based client test and a
factory test covering s3_root -> bucket+key.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQE5TsSuQTeNSCSz9A9GQf
Confirming regression test for the prime-lead suspicion (SAP-16.x meter
over-rating). A full-SAP-shaped 16.x cert lodges the SAP energy_tariff code
space; the assessment_type=SAP gate routes it to the full-SAP mapper, which
translates the code (1=standard → STANDARD) rather than raw-passing it to the
RdSAP normaliser (where 1="dual" → SEVEN_HOUR, the over-rating bug). No fix
needed — the gate already prevents the drop; this pins that behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add code 5 → "off-peak 18 hour" to _SAP_TARIFF_TO_RDSAP_METER_TYPE so the
full-SAP meter translation covers the whole energy_tariff code space, not
just 1-4. Table 12a resolves the word alias to Tariff.EIGHTEEN_HOUR.
Real SAP-19.1.0/19.2.0 sample certs lodge tariff 5 and now resolve to the
18-hour off-peak split instead of defaulting to STANDARD (under/over-rated).
The same fix covers 17.0/17.1/18.0.0/19.0.0 — all route through the full-SAP
mapper via the assessment_type=SAP gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The full-SAP energy_tariff code space includes 5=off-peak-18-hour, but
_sap_17_1_meter_type only maps 1-4, dropping code 5 to "" → STANDARD.
Real SAP-19.1.0/19.2.0 certs lodge tariff 5, so the 18-hour off-peak
split is silently lost.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Crediting the full-SAP lodged PV closes the 2-point under-rating on uprn
10023444324 — the engine now reproduces the accredited lodged value exactly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full-SAP certs lodge measured PV under sap_energy_source.pv_arrays, but the
SAP-Schema-17.1 SapEnergySource modelled only wind, so from_dict dropped it at
parse and the mapper hardcoded no PV. A new SapPvArray schema field + the
_sap_17_1_pv_arrays mapper helper carry it to the domain photovoltaic_arrays the
Appendix-M generation credit reads — without it an all-electric dwelling the
array lifts to A/B is mis-modelled down a band or more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full-SAP energy_tariff (1=standard/2=7hr/3=10hr/4=24hr) and RdSAP meter_type
(1=dual/2=single/3=unknown/4=24hr) are different code spaces; the mapper passed
the code through untranslated, reading standard-tariff full-SAP certs as Economy 7
(over-rated) and Economy-7 certs as single (under-rated). Map onto the RdSAP word
aliases so the resolved Table 12a tariff is correct; absent/ND -> unknown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The port accepts a normalised Postcode and rejects malformed/empty ones via
Postcode.is_valid() (PostcodeNotFound) — dropping the per-lookup postcodes.io
HTTP call and the cross-module use of the private _sanitise_postcode. Mapper
helper promoted to public.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
