roof_construction_type was left None in from_site_notes, so the
calculator's "flat"/"sloping ceiling" substring routing in
heat_transmission couldn't detect flat roofs and every roof (flat or
pitched) fell through the pitched default. country_code was never set
on this path either, so the England/Wales/Scotland/NI U-value cascade
(u_floor, u_basement_floor, u_door) had no country to key off.
Sets roof_construction_type from the surveyed roof_space construction_type
(main building + extensions), and country_code="ENG" for this all-English
cohort, matching the existing from_elmhurst_site_notes path.
Fixes#1566
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`_map_sap_heating` left `main_heating_index_number` None, so the
calculator fell to a generic Table-4b seasonal efficiency instead of the
actual appliance. The site note lodges the boiler by its PCDB `product_id`
(and `summer_efficiency` is always 0), and those ids resolve directly in
the calculator's PCDB (`gas_oil_boiler_record`) — so carrying `product_id`
onto `main_heating_index_number` gives the real SEDBUK efficiency.
Gated on `product_id > 0` (0 = no product lodged → stays None; the
calculator keeps its default). `main_heating_category`/`sap_main_heating_code`
were evaluated and add nothing measurable — the PCDB index alone is the
efficiency source.
Guinness GMCA cohort: within-0.5 12.9% -> 13.4%, MAE 2.657 -> 2.621, no
regression (harness still 206-xfail, no new hard failures).
Closes#1563
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
secondary_fuel_type was hardcoded None in the from_site_notes mapper,
dropping the fuel for the ~67 Guinness GMCA 205 properties that lodge a
"Panel, convector or radiant heaters" secondary heater. Adds
_pashub_secondary_fuel_code (ADR-0015 pattern, mirrors
_pashub_main_fuel_code) mapping "Electricity" -> SAP10 code 30, with
"No Secondary Heating"/blank -> None and unknown labels strict-raising
UnmappedPasHubLabel.
Fixes#1564
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`_map_sap_heating` set cylinder geometry/insulation but left
`water_heating_code` / `water_heating_fuel` / `immersion_heating_type`
None (and immersion as a raw label). For cylinder dwellings that loses
the HW-system typing the calculator's water cascade keys off.
Resolve them at the mapper boundary, gated on cylinder presence (combi
dwellings keep the None default so HW inherits the main system, unchanged):
- `water_heating_code` from the `system` label — "From main heating 1" -> 901,
"Electric immersion" -> 903, community -> 950; strict-raises an unknown label.
- `water_heating_fuel` = standard electricity (30) for the electric-immersion
WHC 903; None otherwise (901 inherits the main fuel).
- `immersion_heating_type` via the Elmhurst map (Single -> 2), cylinder-gated.
Combi-dominated cohort so the accuracy impact is negligible by design
(Tier 3): within-0.5 and MAE unchanged (12.9% / 2.56), harness still
206-xfail with no new hard failures.
Closes#1565
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`_map_sap_window` copied the raw `glazing_type` survey label; the
calculator's isinstance-int guards read it as non-int and fell to the
double-pre-2002 U=2.8 default for every window — over-counting heat loss
on the ~852 windows the survey dates to 2002-2021 (RdSAP Table 24 U=2.0).
Add `_pashub_glazing_type_int` mapping the surveyed labels to the SAP10
cascade glazing codes the calculator reads for window U
(`_GLAZING_CODE_TO_UWINDOW`) and solar g (`_G_PERPENDICULAR_BY_GLAZING_TYPE`):
before 2002 -> 3, 2002-2021 -> 2, unknown install date -> 3 (pre-2002
default), post-2022 -> 13. Strict-raises `UnmappedPasHubLabel` on an unknown
label; blank passes through as "".
Completes the fabric stack (party-wall/wall-construction/insulation/glazing):
cohort MAE 2.79 -> 2.66, within-0.5 11.4% -> 12.9%, signed -0.04 -> +0.33.
Closes#1562
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reviewer pushback: the gov EPC code table (epc_codes.csv) defines
energy_efficiency_rating 0 generically as "N/A", not specifically
"party wall". The payload carries no other identifying field, so the
actual cause can't be determined from the data alone. The fix itself
is unaffected (still calc-neutral regardless of cause) — just
correcting comments/test naming to not assert an unconfirmed fact.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task 9d271e98-96e6-4be6-bb50-e9be6f954003 (portfolio 796) failed 21
"Top-floor flat" properties with "EnergyElement: missing required
field 'description'". Every one lodges a second walls[] entry that is
a bare zero-rated party-wall boundary element
({"energy_efficiency_rating": 0, "environmental_efficiency_rating": 0},
no description key at all) — unlike the sibling floors/roofs
"(other premises above/below)" sentinel convention, which does carry
text. Confirmed schema-wide across all 21 failing certs.
Default description to "" rather than fabricate construction text:
_joined_descriptions (heat_transmission.py) already filters out falsy
descriptions, so this is calc-neutral — it only unblocks the parse.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`from_site_notes` copied the raw `walls_insulation_type` survey label onto
the building part; the calculator's `_int_or_none` read it as `None`, so
the ~101 filled-cavity walls in the cohort lost their lower-U credit and
were over-counted.
Add `_pashub_wall_insulation_type_int` (mirroring the wall-construction
helper and the Elmhurst `_ELMHURST_WALL_INSULATION_TO_SAP10` sibling)
mapping the surveyed labels to the SAP10 wall-insulation codes `u_wall`
consumes ("As built" -> 4 assumed/default, "Filled Cavity" -> 2,
"External" -> 1), strict-raising `UnmappedPasHubLabel` on an unknown label.
Stacked on #1560, this is the fabric fix that removes the cohort's
systematic SAP bias: mean signed -0.79 -> -0.04, MAE 3.11 -> 2.79,
within-0.5 8.5% -> 11.4%.
Closes#1561
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`from_site_notes` copied the raw `walls_construction_type` survey label
onto the building part; the calculator's `_int_or_none` read it as `None`,
so every wall fell back to the age-band + thickness U-value and lost the
solid/cavity/timber/system-built distinction.
Add `_pashub_wall_construction_int` (mirroring `_pashub_party_wall_
construction_int`) mapping the four surveyed labels to the WALL_* codes
`u_wall` consumes, strict-raising `UnmappedPasHubLabel` on an unknown
label and passing a blank label through as the empty-string "no lodging"
sentinel (the field is `Union[int, str]`). On the Guinness GMCA cohort
this trims SAP MAE 3.22 -> 3.11 and lifts within-1.0 12.9% -> 14.9%.
Closes#1560
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task 9d271e98-96e6-4be6-bb50-e9be6f954003 (portfolio 796) failed property
715358 with "SapHeating: missing required field 'cylinder_size'": cert
0685-2881-6867-9029-1761 lodges has_hot_water_cylinder="false" but omits
sap_heating.cylinder_size entirely. Mirrors the already-handled inverse
gap in _normalize_sap_schema_16_x, so default to code 1 ("no cylinder")
rather than fail loud.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`from_site_notes` copied the raw `party_wall_construction_type` survey
label onto the building part; the calculator read `None` and applied the
U=0.25 house default to every party wall — phantom heat loss on the
~165/205 solid party walls that RdSAP 10 Table 15 rates U=0.0.
Add `_pashub_party_wall_construction_int` (mirroring `_pashub_main_fuel_code`)
mapping the six surveyed labels to the SAP10 codes `u_party_wall` consumes,
strict-raising `UnmappedPasHubLabel` on an unknown label, and wire it into
both site-note building-part mappers. On the Guinness GMCA cohort this
halves the systematic SAP under-rate (mean signed -1.58 -> -0.85).
Refs #1559
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_map_sap_heating resolves the raw emitter label (e.g. Radiators) to its
SAP10 emitter code via _pashub_heat_emitter_code, reusing the Elmhurst
emitter map. Blank passes through; an unrecognised label strict-raises
UnmappedPasHubLabel at the mapper boundary (ADR-0015) instead of
resurfacing as the calculator's UnmappedSapCode: heat_emitter_type.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Properties 741843/741872 (uprns 100060714155/100021944594) failed
modelling_e2e subtasks 6ac7841a-9338-4efe-97e5-7e0d19b3055d and
c5450f03-5b0d-4068-a3b3-d1470bc0af57 with a bare StopIteration: their
SAP-Schema-15.0 (2011-era LIG-lodged) certs identify the main dwelling's
building part as "Main building" rather than "Main Dwelling", so
from_api_string fell to OTHER and left the property with no MAIN part —
crashing wall_recommendation.py's unguarded next(...).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Thread the UPRN the PashubService has already resolved for a job into the
PasHub site-notes mapper so the EpcPropertyData aggregate is born with its
uprn set, and the existing save_epc_property_data path persists it.
- EpcPropertyDataMapper.from_site_notes gains uprn: Optional[int] = None and
sets it unconditionally (site notes never carry a UPRN natively).
- parse_site_notes_pdf / _parse_pashub forward the uprn; the Elmhurst branch
is untouched.
- PashubService coerces uprn str -> int in one place, carries it on the
internal upload record, and passes it into parse_site_notes_pdf.
- No new lookups; jobs with no known UPRN still persist null, unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The prior commit derived a Python bool (`cylinder_size != 1`), but
`from_rdsap_schema_17_1` reads the field via `schema.has_hot_water_cylinder
== "true"` — every real 15.0/16.x fixture lodges the lowercase string
"true"/"false", not a JSON boolean. A bare bool compares False against
that string check either way, so the derivation silently mapped every
cylinder-present cert to has_hot_water_cylinder=False too.
Caught by the requested true-branch test (mutating sap_16_2.json, which
lodges cylinder_size=2/has_hot_water_cylinder="true", to omit the field)
— it failed under the original bool-typed fix. Now emits "true"/"false"
strings to match the lodged convention.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task b9fcd354-2a33-4fc4-a64e-388c060a055c (portfolio 824 / scenario 1278)
also failed property_id 749229 (UPRN 100010349400) with "RdSapSchema17_1:
missing required field 'has_hot_water_cylinder'". Cert
9568-3034-6211-6089-5960 (SAP-Schema-15.0) lodges `sap_heating.cylinder_size`
but omits the separate top-level `has_hot_water_cylinder` boolean
RdSapSchema17_1 also requires.
Unlike `multiple_glazed_proportion` (deliberately left un-defaulted a few
lines above — a prior guessed default regressed the accuracy gate),
`cylinder_size` isn't a proxy needing inference: RdSAP 10 Table 28 defines
code 1 as "no cylinder" — literally the same fact `has_hot_water_cylinder`
encodes. Confirmed 1:1 across every real 15.0/16.x fixture that lodges both
fields (cylinder_size == 1 <-> false, every other code <-> true).
Added to the shared `_normalize_sap_schema_16_x`, so it covers the whole
15.0/16.0/16.1/16.2/16.3 reduced-field family, not just 15.0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task b9fcd354-2a33-4fc4-a64e-388c060a055c (portfolio 824 / scenario 1278)
still failed 5 of 128 batches with NotNullViolation on
epc_building_part.construction_age_band even after the SAP-Schema-15.0
mapper landed (PR #1531). Cert 8169-6926-5310-0447-8996 (UPRN
100010344955, SAP-Schema-15.0) lodges a bare alternative-wall record
(wall_area/wall_construction/wall_insulation_type, no identifier, age
band, or floor dimensions) as a second sap_building_parts element rather
than under a distinct sap_alternative_wall_1 key. RdSapSchema17_1 (which
the 15.0/16.x reduced-field mappers delegate to) has no alt-wall slot to
route it to, so it persisted as a phantom building part with a null age
band.
_drop_building_parts_without_age_band already solved exactly this shape
of problem for lodging artifacts (ae81a81e), but that commit sits on
fix/drop-building-part-without-age-band, an unmerged branch that has
since drifted far behind main. Reinstating the same filter (fresh, not
cherry-picked) fixes both the original artifact case and this SAP-15
alt-wall case, since RdSapSchema17_1 has nowhere else to put it either
way.
Verified against all 174 properties across the task's originally-failing
batches: zero remaining null-age-band building parts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
An unrecognised schema_type used to raise ValueError and abort the whole
call; now from_api_response logs a warning and returns None so one
unmapped cert doesn't break a batch. Schema 15.0 already has a mapper
(PR #1531) so it's unaffected; only genuinely unmapped versions skip.
Both flagged mismatches were Elmhurst input errors (same silent-stale-value
contamination class as the earlier chimneys/wall-thickness bugs), not
parser or calculator bugs:
- cylinder_size: build_100010086084.py's water_heating() selected
DropDownListCylinderSize by raw value "2", but this DOM's option values
ARE their visible litre-band text (no "2" among them) — Playwright
silently no-ops on a non-matching value, leaving a prior cert's "Medium
(131-170L)" selection in place. Fixed to match by text ("Normal"), and
added the missing mapper dict entries (Normal/Large litre-suffixed
labels) to _ELMHURST_CYLINDER_SIZE_LABEL_TO_SAP10 (a real mapper-coverage
gap — the calculator raises UnmappedElmhurstLabel rather than silently
mis-mapping).
- boiler_flue_type: the boiler-code search dialog's combined "Balanced/Open
Flue" Table 4b category doesn't drive the separate, independently
selectable RadioButtonListFlueType field, which was left at an inherited
"Balanced" from a prior cert. Fixed space_heating() to explicitly select
"Open" to match the lodged gov-API code. This field isn't consumed by
Sap10Calculator (ML/generator-only), so it had zero effect on the SAP
score.
Re-downloaded elmhurst_summary.pdf/elmhurst_worksheet.pdf after the fixes
(Recommendations page confirmed clean). Elmhurst-PDF-inputs path moved
46->47 SAP from the cylinder-volume correction; gov-API SAP (53) and
Elmhurst's own worksheet (51) are unchanged. Full accuracy suites re-run
clean (67 passed, 67 skipped, 1 xfailed, same as before).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Built UPRN 100010086084 (the largest gov-API-vs-lodged outlier from the
20-cert batch) in Elmhurst's accredited RdSAP10 tool to get real ground
truth, per feedback that lodged rating isn't a valid comparison target.
Elmhurst worksheet: SAP 46-48 (orientation-dependent), vs this engine's
53 on gov-API inputs -- a real ~5-7pt residual gap. Ruled out with
direct empirical tests: total_floor_area, missing party_wall_length,
region/climate (SAP rating always uses UK-average weather per Appendix
U -- confirmed inert), boiler/secondary efficiency (exact match, 66%/
63% both sides). Confirmed real but NOT blind-fixed (need more than one
cert's evidence, and a wall-U-value patch tested the wrong direction in
isolation): a wall-thickness-unknown U-value fallback mismatch (this
engine 1.70, Elmhurst's own "unknown" resolution 1.40) and a window
U-value divergence (1.85 vs 2.52 W/m2K for the same nominal glazing
pick). Single-window orientation confirmed a ~2pt partial contributor.
Also fixes a small, separate, evidenced gap the build surfaced: the
Elmhurst summary parser didn't recognise the live tool's "Medium
(131-170L)" cylinder-size label (only the bare "Medium").
Full write-up is in the RealCertExpectation comment for this cert in
test_real_cert_sap_accuracy.py and the worklist. Residual gap is
intentionally left open for follow-up, not force-closed.
Task a40e71c4-fd56-4a33-8677-e8643e6a5bb4 (portfolio 824, scenario 1278)
failed 2 of 29 properties with "Unsupported EPC schema: 'SAP-Schema-15.0'"
(property_id 748445/748451). Traced to two real 2011 lodgements
(schema_version "LIG-15.0", assessment_type "RdSAP") for UPRNs
100010359769/100010359788 — structurally the same reduced-field RdSAP-17.1
shape as the already-handled SAP-Schema-16.x family, one revision older.
from_sap_schema_15_0 reuses _normalize_sap_schema_16_x and adds three
schema-wide defaults this older generation never lodges at all (confirmed
absent on both real certs, not just sparse on one):
- door_count=1 (single external door, standard RdSAP assumption)
- instantaneous_wwhrs=0/0/0 and has_fghrs="N" (predates WWHRS/FGHRS
lodging — absence-means-none-fitted, same convention used elsewhere)
- percent_draughtproofed=0 (mirrors the existing has_draught_lobby
"assume none if unknown" convention in cert_to_inputs.py)
- energy_rating_average=60 (inert metadata, never read by the calculator)
Both real certs now map and run end-to-end through the SAP-10 engine,
matching (or within 1 point of) their lodged SAP score.
epb-data-warehouse's fixture set covers CEPC-7.0/7.1/8.0.0 (SBEM
non-domestic certs) alongside the RdSAP/SAP families we already map.
CEPC has no sensible EpcPropertyData shape (asset_rating/BER instead of
a SAP score, activities/hvac_systems instead of building elements), so
from_api_response now raises a typed NonDomesticSchema instead of
falling through to the generic "Unsupported EPC schema" ValueError.
Some gov-API RdSAP-21.0.1 certs (e.g. 2461-5385-9264-6821-5357, property_id=750232,
portfolio 824) omit `built_form` from the response entirely rather than sending a
code. `from_dict` hard-fails any dataclass field with no default that's absent
from the raw dict, so the whole cert failed to parse.
built_form: Optional[int] = None (kw_only, matching the has_fixed_air_conditioning
precedent already on this dataclass) — the downstream SAP-cascade consumer
(_api_sheltered_sides) already treats a non-int built_form as "no lodging" and
falls back to the documented default, so None here is an already-safe
degradation, not a new failure mode. Also guards mapper.py's
`built_form=str(schema.built_form)` against stringifying None into "None".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
datatypes/epc/domain/mapper.py's from_rdsap_schema_{17_0,17_1,18_0,19_0,
20_0_0,21_0_0} all stored a lodged sap_windows[].glazing_type verbatim
instead of routing it through _api_cascade_glazing_type (which
_api_sap_window, used by 21_0_1, already does correctly). The raw API code
is the RdSAP-21 glazing enum (e.g. code 1 = "DG pre-2002"), not the SAP
10.2 Table 6b cascade enum the calculator's U/g-value tables are keyed on
(cascade code 1 = single glazed) -- so any lodged window whose raw code
happened to collide with a different cascade meaning was silently
mis-rated.
Found while validating PR #1503 (property 753950 / uprn 100021969385)
against Elmhurst: our engine's windows_w_per_k was 82.27 vs Elmhurst's
51.44 -- the dwelling's larger window (91% of glazed area) lodges raw
glazing_type=1 and was being modelled as single glazed instead of double,
the dominant contributor to a SAP 42 vs Elmhurst 44 gap. Fixing this
narrows windows_w_per_k to 50.26 (Elmhurst 51.44, ~2% off) and moves the
engine's score to 43.
Extracted the 5 duplicated reduced-schema-window blocks (17.0/17.1/18.0/
19.0/20.0.0, which lodge a minimal window shape with no glazing_gap/
frame_factor/transmission fields) into a shared _reduced_field_api_sap_
window helper that also now populates window_transmission_details/
frame_factor via the SAP10 lookup instead of leaving them None. 21.0.0's
richer inline block gets the same one-line cascade fix.
+6 regression tests (one per affected schema seam + 21.0.0). Full domain/
schema/prediction/comparable-properties/sap10_calculator/modelling_e2e
suites pass (2746 passed, only the 2 pre-existing unrelated failures);
component-accuracy gate 26/26 unaffected.
Property 753950 (uprn 100021969385, cert 0141-2860-6891-9124-5625,
SAP-Schema-16.2) hard-failed modelling_e2e: RdSapSchema17_1 requires
multiple_glazed_proportion as a non-optional int, and this 16.x cert omits
the field entirely, lodging only the multiple_glazing_type="ND" sentinel.
The cert's own window.description ("Fully double glazed") states the
glazing extent unambiguously, so _normalize_sap_schema_16_x now derives
0/100 from "single"/"double" wording when the field is absent, mirroring
the existing single-glazed multiple_glazing_type cascade. This is
worklist P4 (.claude/skills/expand-sap-accuracy-corpus/worklist.md) —
a flat default was tried previously and reverted because making such
certs mappable at all pulls them into the EPC-prediction donor pool and
tips near-tie similarity matches; deriving from explicit text (rather
than a blind default) was the suggested unblock.
Re-measured the component-accuracy gate as the worklist asked: it drops
(has_hot_water_cylinder 0.8687->0.8586, cylinder_insulation_type
0.3333->0.1667, door_count residual 0.3131->0.3333) via the same
donor-pool-composition mechanism as the prior #1245/ADR-0037
re-baselines, not a prediction-logic loosening. Re-baselined the floors
with that rationale recorded inline.
Address PR review (dancafc):
- introduce UprnMatch NamedTuple (datatypes/address_match.py) for the
(uprn, address, lexiscore, certificate_number) return, replacing the bare
4-tuple in get_uprn_from_epc_df / get_uprn_from_historic_epc /
HistoricEpcResolver.resolve_uprn. Tuple-compatible, so unpacking is unchanged.
- rename get_uprn_with_epc_df -> get_uprn_from_epc_df (+ callers).
- type resolve_group_ambiguity via a GroupDecision NamedTuple and trim its
docstring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The row→domain mapper now names all 93 constructor arguments explicitly
instead of splatting a lowercased dict, takes a plain Mapping (a
DataFrame.to_dict("records") row) instead of a pandas Series, and ignores
columns the domain type doesn't know. A missing/renamed CSV column fails
loudly as a KeyError at the row. Both iterrows() call sites move to
to_dict("records") — pandas-stubs types iterrows' Series unparameterized,
which strict mode rejects. pandas-stubs + boto3-stubs[s3] make the stack
check clean: pyright strict is now 0 errors across the PR's files.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
