From 04b7c26f6790f5bfd844ca4790c669dc3db15592 Mon Sep 17 00:00:00 2001 From: KhalimCK Date: Thu, 16 Jul 2026 00:30:55 +0100 Subject: [PATCH] PasHub SAP accuracy: infiltration inputs + immersion DHW + lights/lobby/heat-network (#1615) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Electric-immersion DHW: bill at 100%, not the space boiler's efficiency 🟩 A separate electric immersion heater (SAP water-heating code 903) is 100% efficient (SAP 10.2 Table 4a) and the space-heating boiler provides no water heating, so Appendix D2.1 Eq D1 (the boiler seasonal-efficiency cascade) must not apply to it. But `_water_heating_main` resolves DHW to the SPACE main β€” a gas/oil boiler keeps its PCDB record β€” so three water-efficiency branches in cert_to_inputs billed the immersion-heated cylinder at the boiler's ~87% summer efficiency (a mongrel: electric fuel price x gas-boiler efficiency): 1. the scalar `water_eff = water_pcdb_main.summer_efficiency_pct / 100`, 2. the SAP Β§9.4.11 / Table 4c(2) -5pp no-interlock adjustment, and 3. the Eq D1 (winter, summer) seasonal pair from `pcdb_main`. Gate all three on `not dhw_is_electric_immersion`; the correct immersion path (`_water_efficiency_with_category_inherit` -> 1.0, Eq D1 off) then applies, mirroring the Table 3 zero-primary-loss gate already present for WHC 903. General bug β€” fires for any WHC-903 dwelling whose space main is a PCDB gas/oil boiler. It surfaced via the PasHub campaign's #1600 no-water-heating default (WHC 999 -> 903) on a gas-combi dwelling: 58 Hackle St M11 4WU, SAP 55.30 -> 57.67 (pre_sap 58, verified 59). Guardrails: the gov-API RdSAP corpus IMPROVES 78.8% -> 78.9% within-0.5, MAE 0.625 -> 0.622 (a handful of corpus certs with a boiler space main + electric immersion move closer to accredited) β€” MAE ceiling ratcheted 0.626 -> 0.625. Regression pinned in test_cert_to_inputs (RED before / GREEN after). pyright 0-new (cert_to_inputs baseline 30). Co-Authored-By: Claude Opus 4.8 (1M context) * PasHub SAP accuracy: thread low-energy lights + draught lobby (from_site_notes) 🟩 Two systematic silent-drops in the PAS Hub `from_site_notes` path, each a field the gov-API/Elmhurst mappers thread but site-notes dropped, letting a favourable default reach the SAP-10.2 calculator. Validated against the correct pashub oracle (property_baseline_performance.effective_sap_score, portfolio 838): cohort within-0.5 55.1% β†’ 62.6%, MAE 0.599 β†’ 0.521. - Low-energy lights: when "exact LED/CFL known = No", PAS Hub lodges an aggregate "Number of fixed low energy lights?" count. Previously dropped (no dataclass field / extractor key / mapper thread) β†’ calculator saw 0 low-energy bulbs and applied the pessimistic L5b/L8c no-data default, under-rating SAP. Now threaded into low_energy_fixed_lighting_bulbs_count, mirroring from_elmhurst_site_notes. ~6% of the cohort; resolves 58 Hackle (βˆ’0.43 vs oracle 59 with #1615). - Draught lobby: _map_sap_ventilation set only the legacy `draught_lobby` field, never the canonical `has_draught_lobby` Β§2 (13) gate the cascade reads, so the surveyed lobby was ignored and infiltration over-stated. Now mirrors Elmhurst. Two other audited levers were REJECTED against the pashub oracle (they matched the gov-cert/RdSAP convention but pashub does not apply them): percent_draughtproofed (cohort 30.8%) and the Β§A.2.2 assumed secondary heater (19.2%). A heat-network control-code fix (2306β†’2303 for 16 Bingley) is HELD pending spec/Elmhurst adjudication β€” it contradicts ADR-0053; see the NOTE on _PASHUB_HEAT_NETWORK_CONTROL_TO_SAP10. Co-Authored-By: Claude Opus 4.8 (1M context) * PasHub heat-network control: room-thermostat-only maps to 2308, not 2306 🟩 The Table 4e Group 3 label "charging linked to use of community heating, room thermostat only" was mapped to 2306 β€” the linked-to-use *with-TRVs* code (control type 3, space charging 1.00). The label explicitly excludes TRVs, so it is control type 2 / space 1.05 = code 2308/2309 (linked-to-use β†’ DHW 1.00). 2306 asserted TRVs the survey doesn't have and over-rated the cohort's sole heat-network dwelling (16 Bingley Close 56.89 β†’ 54.01 vs pashub oracle 52). Verified against SAP 10.2 Table 4e as encoded in cert_to_inputs.py (_CONTROL_TYPE_BY_CODE 2308β†’2, space-charging 2308β†’1.05, DHW 2308β†’1.00) and the RdSAP control-label vocabulary in MainheatControlAttributes.py. 2303 (the earlier audit's guess) is rejected: it is a flat-rate code (DHW 1.05) whose oracle match was a coincidence of two offsetting spec violations. The residual +2.0 to oracle 52 is the documented SAP-10.2-engine-vs-lodged offset, not a fuel/flags gap (those were threaded by the #1590 follow-up) β€” supersedes the ADR-0053 / HANDOVER_838 "community fuel/flags" attribution. 16 Bingley is the only heat-network fixture in portfolio 838, so this moves one fixture strictly toward its oracle. Harness ratcheted: within-0.5 0.51β†’0.58, MAE 0.625β†’0.521 (observed 58.5% / 0.520 with all three 2026-07-15 levers). Co-Authored-By: Claude Opus 4.8 (1M context) * PasHub SAP accuracy: restore 3 dropped Β§2 infiltration inputs (from_site_notes) `from_site_notes` uniquely dropped three ventilation inputs that the gov-API and Elmhurst sibling mappers all set, causing a systematic cohort-wide SAP under-rate vs the company's own accredited SAP-10.2 certs (property_baseline_performance.effective_sap_score, portfolio 838): 1. percent_draughtproofed β€” never set, so Β§2(15) window infiltration was pinned at its 0.25-ACH worst case on every dwelling. Reconstructed as the area-weighted % of draught-proofed windows (mirrors Elmhurst `draught_proofing_percent`). Dominant term. 2. Upper-storey +0.25 m joist void β€” omitted; now added, matching `_UPPER_FLOOR_HEIGHT_ADD_M` on the gov-API/Elmhurst paths. This RE-ADJUDICATES #1601 ("keep raw"): that call was confounded by the then-present draught-proofing drop suppressing every verified dwelling. 3. sheltered_sides β€” left None β†’ calculator's flat default of 2, which over-shelters end/semi/detached (RdSAP Β§S5 = 1/1/0). Now derived from built form. The three are NON-ADDITIVE β€” each overshoots alone (which is why all three were previously rejected individually) β€” but together they land all 7 verified ground-truth dwellings toward truth (none regress) and every built form near zero. Cohort 62.1% -> 83.3% within-0.5, MAE 0.507 -> 0.377. Guardrails: gov-API RdSAP corpus unchanged (78.9%, these are from_site_notes- only helpers). pashub harness 58.5% -> 79.5% / MAE 0.520 -> 0.389; ratchets tightened 0.58->0.78 and 0.521->0.40. pyright zero-new (mapper baseline 39). New focused tests in TestFromSiteNotesInfiltrationFixes cover all three fields incl. area-weighting and the built-form shelter map; goldens updated. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- backend/documents_parser/extractor.py | 9 + .../documents_parser/tests/test_end_to_end.py | 1 + .../documents_parser/tests/test_extractor.py | 30 +++ .../tests/test_pashub_sap_accuracy.py | 38 +++- datatypes/epc/domain/mapper.py | 126 +++++++++-- .../epc/domain/tests/test_from_site_notes.py | 200 +++++++++++++++++- .../epc/surveys/pashub_rdsap_site_notes.py | 1 + .../sap10_calculator/rdsap/cert_to_inputs.py | 29 ++- .../rdsap/test_cert_to_inputs.py | 55 +++++ .../epc_client/test_sap_accuracy_corpus.py | 7 +- 10 files changed, 453 insertions(+), 43 deletions(-) diff --git a/backend/documents_parser/extractor.py b/backend/documents_parser/extractor.py index fde4530e9..d06017dba 100644 --- a/backend/documents_parser/extractor.py +++ b/backend/documents_parser/extractor.py @@ -518,6 +518,15 @@ class PasHubRdSapSiteNotesExtractor: number_of_fixed_cfl_bulbs=int( self._get_in(rce_section, "Number of fixed CFL bulbs:") or 0 ), + number_of_fixed_low_energy_lights=( + int(low_energy_raw) + if ( + low_energy_raw := self._get_in( + rce_section, "Number of fixed low energy lights?" + ) + ) + else None + ), waste_water_heat_recovery=self._get_in( rce_section, "Are there any waste water heat recovery systems?" ) diff --git a/backend/documents_parser/tests/test_end_to_end.py b/backend/documents_parser/tests/test_end_to_end.py index b0858ddc6..6052e581c 100644 --- a/backend/documents_parser/tests/test_end_to_end.py +++ b/backend/documents_parser/tests/test_end_to_end.py @@ -279,6 +279,7 @@ class TestPdfToEpcPropertyData: sap_ventilation=SapVentilation( ventilation_type="Mechanical Extract - Decentralised", draught_lobby=False, + has_draught_lobby=False, pressure_test="No test", open_flues_count=0, closed_flues_count=0, diff --git a/backend/documents_parser/tests/test_extractor.py b/backend/documents_parser/tests/test_extractor.py index 0666f0d7c..3a57fcb56 100644 --- a/backend/documents_parser/tests/test_extractor.py +++ b/backend/documents_parser/tests/test_extractor.py @@ -1181,6 +1181,36 @@ class TestRoomCountElements: waste_water_heat_recovery="None", ) + def test_low_energy_lights_absent_when_exact_counts_known( + self, rce: RoomCountElements + ) -> None: + # Base fixture lodges the exact LED/CFL counts, so the aggregate + # "low energy lights" line is absent and stays None. + assert rce.number_of_fixed_low_energy_lights is None + + def test_low_energy_lights_parsed_when_exact_counts_unknown(self) -> None: + # Arrange β€” splice a survey that answers "No" to the exact-count + # question and lodges an aggregate low-energy count instead (the shape + # of ~6% of the PAS Hub cohort, e.g. 5 Bailey Street / 58 Hackle St). + lines = load_text_fixture() + idx = lines.index("Is the exact number of LED and CFL bulbs known?") + assert lines[idx + 1] == "Yes" + cfl_idx = lines.index("Number of fixed CFL bulbs:") + spliced = ( + lines[: idx + 1] + + ["No"] + + lines[idx + 2 : cfl_idx + 2] + + ["Number of fixed low energy lights?", "11"] + + lines[cfl_idx + 2 :] + ) + + # Act + rce = PasHubRdSapSiteNotesExtractor(spliced).extract_room_count_elements() + + # Assert + assert rce.exact_led_cfl_count_known is False + assert rce.number_of_fixed_low_energy_lights == 11 + class TestWaterUse: def test_full_water_use(self) -> None: diff --git a/backend/documents_parser/tests/test_pashub_sap_accuracy.py b/backend/documents_parser/tests/test_pashub_sap_accuracy.py index 09299d1cd..66524f6bd 100644 --- a/backend/documents_parser/tests/test_pashub_sap_accuracy.py +++ b/backend/documents_parser/tests/test_pashub_sap_accuracy.py @@ -75,8 +75,10 @@ _MANIFEST_PATH = _FIXTURES_DIR / "manifest.json" # 0.5. MAE rises 2.560 -> 2.701 because two of those four are large, isolated # divergences NOT caused by the mappings (which land the others at d=-1.7/+7.6): # - 499584755922 d=-33.0 Bulk-LPG main + house-coal secondary (34.0 vs pre_sap 67) -# - 507644414148 d=+18.2 community heating β€” pashub path maps only the control -# code, not the full heat-network fuel/flags (70.2 vs 52) +# - 507644414148 d=+18.2 community heating (70.2 vs 52) at the time. Since +# RESOLVED: fuel/flags threaded (#1590 follow-up) and the +# control code corrected 2306->2308 (2026-07-15); now +# +2.0, the SAP-10.2-engine-vs-lodged offset (not a gap). # Both are tracked for verification (pre_sap correctness vs a deeper modelling # gap) β€” see docs/HANDOVER_PASHUB_838_PROBLEM_PROPERTIES.md. Re-baselined to the # full-coverage aggregate; ratchet back DOWN as those two are resolved. @@ -98,11 +100,22 @@ _MANIFEST_PATH = _FIXTURES_DIR / "manifest.json" # ~58 vs 59). Ratcheted 0.50 -> 0.51 by #1602 (semi-exposed / exposed floor # exposure flags -> RdSAP Β§5.14 U=0.7 / Β§5.13 Table 20) + #1603 (alternative- # wall sub-areas + extension floor construction now threaded): observed 51.2% -# (105/205). #1601 (upper-floor +0.25 m) RESOLVED as keep-raw β€” adjudicated -# against verified truth it over-corrects every verified dwelling (raw beats -# +0.25 on all 7); pashub surveys internal dims and its upper-ground height -# gap is modal 0.00, so the joist void must not be added. -_MIN_WITHIN_HALF: float = 0.51 +# (105/205). Ratcheted 0.51 -> 0.58 by the 2026-07-15 levers (low-energy-lights +# + has_draught_lobby threading in from_site_notes + heat-network control code +# 2306->2308): observed 58.5%. +# Ratcheted 0.58 -> 0.78 by the 2026-07-15c INFILTRATION bundle: from_site_notes +# uniquely dropped THREE Β§2 inputs the gov-API/Elmhurst mappers set β€” +# `percent_draughtproofed` (window Β§2(15) pinned at the 0.25-ACH worst case), +# the +0.25 m upper-storey joist void, and built-form `sheltered_sides` +# (defaulted to a flat 2, over-sheltering end/semi/detached). The three are +# NON-ADDITIVE β€” each overshoots alone (which is why #1601 upper-floor +0.25 +# was previously RESOLVED as keep-raw: that call was confounded by the +# then-present draught-proofing drop suppressing every dwelling) β€” but together +# they land all 7 verified dwellings toward truth (none regress) and every +# built form near zero. Observed 79.5% here; vs the DB SAP-10.2 oracle +# (property_baseline_performance.effective_sap_score, portfolio 838) the cohort +# moves 62.1% -> 83.3% within-0.5. gov-API RdSAP corpus unchanged (78.9%). +_MIN_WITHIN_HALF: float = 0.78 # Ratcheted 2.71 -> 2.64 by #1590 bug 2 (roof "Insulation At: None" -> explicit # zero thickness): observed MAE 2.632 β€” the two ~7-SAP roof over-raters # (507665533138 / 507670893756) correcting is exactly 14/205 β‰ˆ 0.07 of MAE. @@ -167,9 +180,14 @@ _MIN_WITHIN_HALF: float = 0.51 # Bingley still +6.818). The three silent-drop fixes the gov-API/Elmhurst paths # already apply; the electric-shower drop (100/205) was the dominant lever. # Ratcheted 0.653 -> 0.625 by #1602 (floor exposure flags) + #1603 (alternative -# walls + extension floor construction): observed MAE 0.624. #1601 (upper-floor -# +0.25 m) RESOLVED as keep-raw β€” see the within-0.5 note above. -_MAX_SAP_MAE: float = 0.625 +# walls + extension floor construction): observed MAE 0.624. +# Ratcheted 0.625 -> 0.521 by the 2026-07-15 levers (low-energy-lights + +# has_draught_lobby + heat-network control 2306->2308): observed MAE 0.520. +# Ratcheted 0.521 -> 0.40 by the 2026-07-15c infiltration bundle +# (percent_draughtproofed + upper-floor +0.25 m + built-form sheltered_sides; +# #1601 upper-floor RE-ADJUDICATED, see the within-0.5 note above): observed +# MAE 0.389. +_MAX_SAP_MAE: float = 0.40 _KNOWN_GAP_REASON = ( "pashub `from_site_notes` mapper does not int-code a main-heating " diff --git a/datatypes/epc/domain/mapper.py b/datatypes/epc/domain/mapper.py index 62fe48069..3998b57eb 100644 --- a/datatypes/epc/domain/mapper.py +++ b/datatypes/epc/domain/mapper.py @@ -352,6 +352,31 @@ class EpcPropertyDataMapper: 2, ) # TODO: verify that is the correct approach + # Β§2 (15) draught-proofing of windows/doors. The gov-API and Elmhurst + # mappers set `percent_draughtproofed` (from their lodged percentage); + # from_site_notes uniquely dropped it β†’ the calculator read 0 and pinned + # the window-infiltration term at its 0.25-ACH worst case (0.25 βˆ’ 0.2 Γ— + # pct/100) on EVERY pashub dwelling β€” a uniform cohort-wide under-rate. + # PAS Hub surveys draught-proofing per window (SapWindow.draught_proofed; + # 198/205 dwellings are fully draught-proofed, consistent with sealed + # double-glazed units), so reconstruct the area-weighted percentage the + # sibling mappers lodge directly. See the #1601 note in + # `_map_floor_dimensions` for the non-additive height/shelter partners. + sap_windows = [_map_sap_window(w) for w in survey.windows] + _dp_total_area = sum( + (w.window_width or 0.0) * (w.window_height or 0.0) for w in sap_windows + ) + _dp_proofed_area = sum( + (w.window_width or 0.0) * (w.window_height or 0.0) + for w in sap_windows + if w.draught_proofed + ) + percent_draughtproofed = ( + int(round(100.0 * _dp_proofed_area / _dp_total_area)) + if _dp_total_area > 0 + else None + ) + return EpcPropertyData( uprn=uprn, dwelling_type=f"{general.detachment_type} {general.property_type.lower()}", @@ -371,7 +396,8 @@ class EpcPropertyDataMapper: main_heating=[], door_count=room_counts.number_of_external_doors, sap_heating=_map_sap_heating(heating, ventilation, survey.water_use), - sap_windows=[_map_sap_window(w) for w in survey.windows], + sap_windows=sap_windows, + percent_draughtproofed=percent_draughtproofed, sap_energy_source=SapEnergySource( gas_connection_available=general.mains_gas_available, meter_type=general.electric_meter_type, @@ -409,6 +435,16 @@ class EpcPropertyDataMapper: cfl_fixed_lighting_bulbs_count=room_counts.number_of_fixed_cfl_bulbs, led_fixed_lighting_bulbs_count=room_counts.number_of_fixed_led_bulbs, incandescent_fixed_lighting_bulbs_count=room_counts.number_of_fixed_incandescent_bulbs, + # When the assessor did not enumerate exact LED/CFL bulbs, PAS Hub + # records an aggregate "Number of fixed low energy lights?" instead; + # thread it so the Β§L calculation uses the surveyed low-energy count + # rather than falling back to the pessimistic no-data default. Mirrors + # the Elmhurst path (from_elmhurst_site_notes). + low_energy_fixed_lighting_bulbs_count=( + room_counts.number_of_fixed_low_energy_lights + if not room_counts.exact_led_cfl_count_known + else None + ), total_floor_area_m2=total_floor_area, built_form=general.detachment_type, property_type=general.property_type, @@ -424,7 +460,9 @@ class EpcPropertyDataMapper: waste_water_heat_recovery=room_counts.waste_water_heat_recovery, hydro=renewables.hydro, photovoltaic_array=renewables.photovoltaic_array, - sap_ventilation=_map_sap_ventilation(ventilation), + sap_ventilation=_map_sap_ventilation( + ventilation, built_form=general.detachment_type + ), ) @staticmethod @@ -5964,17 +6002,19 @@ def _extract_age_band(age_range: str) -> str: def _map_floor_dimensions( floors: List[FloorMeasurement], floor_type: str ) -> List[SapFloorDimension]: - # NOTE (#1601, RESOLVED β€” keep raw): the gov-API/Elmhurst mappers add a - # 0.25 m joist-void to UPPER-storey heights (their lodged height is a CLEAR - # internal ceiling). Adjudicated against VERIFIED truth (the reliable oracle, - # not pre_sap), pashub must NOT add it: RAW beats +0.25 on all 7 verified - # dwellings (16 Stillwater 75.6 vs +0.25's 75.2 against verified 76; every - # verified property is already slightly under truth and +0.25 only lowers - # SAP). pashub surveys internal dimensions (202/205 "Measurements Location: - # Internal") and its upperβˆ’ground height difference is modal at 0.00 β€” the - # joist void is not differentially present, and the absolute basis is - # inconsistent (β‰ˆhalf clear-ceiling ~2.3, β‰ˆhalf already storey-tall >2.6), - # so a blanket +0.25 is unsafe. Left as the raw surveyed value. + # #1601 RE-ADJUDICATED β€” add the 0.25 m upper-storey joist void, matching + # the gov-API/Elmhurst mappers (whose lodged upper height is a CLEAR internal + # ceiling; `_UPPER_FLOOR_HEIGHT_ADD_M` at mapper.py:5492 / :6697). The earlier + # "keep raw" call was made while from_site_notes still dropped + # `percent_draughtproofed` β€” that pinned the Β§2 (15) window-infiltration term + # at its 0.25-ACH worst case on every dwelling, suppressing the whole cohort, + # so omitting +0.25 (which lowers SAP) spuriously looked correct on the 7 + # verified dwellings. With draught-proofing restored the +0.25 is required: + # draught-proofing + upper-floor height + built-form sheltered_sides together + # move all 7 verified dwellings toward truth (none regress) and lift the + # cohort 62.1%β†’83.3% within-0.5 / MAE 0.507β†’0.377 vs the SAP-10.2 oracle + # (property_baseline_performance.effective_sap_score, portfolio 838). The + # three are non-additive β€” each overshoots alone β€” and MUST land together. is_exposed, is_above_partial = _pashub_floor_exposure(floor_type) dimensions: List[SapFloorDimension] = [] for floor in floors: @@ -5984,9 +6024,14 @@ def _map_floor_dimensions( # over heated space (#1602). Mirrors the gov-API/Elmhurst siblings, # which only flag the ground SapFloorDimension. is_ground = floor_number == 0 + room_height = ( + floor.height_m + if is_ground + else floor.height_m + _UPPER_FLOOR_HEIGHT_ADD_M + ) dimensions.append( SapFloorDimension( - room_height_m=floor.height_m, + room_height_m=room_height, total_floor_area_m2=floor.area_m2, party_wall_length_m=floor.pwl_m, heat_loss_perimeter_m=floor.heat_loss_perimeter_m, @@ -6625,13 +6670,38 @@ def _pashub_cylinder_insulation_type_code( return code -def _map_sap_ventilation(ventilation: Ventilation) -> SapVentilation: +# PAS Hub `detachment_type` label β†’ RdSAP Β§S5 sheltered-sides count (mirrors the +# gov-API `_API_BUILT_FORM_TO_SHELTERED_SIDES` int map). from_site_notes left +# `sheltered_sides` None β†’ the calculator's flat default of 2, which over-shelters +# end/semi/detached (spec 1/1/0) β€” an over-credit that masked the cohort-wide +# under-rate. Keyed case-insensitively; an unmapped label falls through to None +# (calculator default 2), the prior behaviour, rather than raising. +_PASHUB_BUILT_FORM_TO_SHELTERED_SIDES: Dict[str, int] = { + "detached": 0, + "semi-detached": 1, + "end-terrace": 1, + "mid-terrace": 2, + "enclosed end-terrace": 2, + "enclosed mid-terrace": 3, +} + + +def _map_sap_ventilation( + ventilation: Ventilation, built_form: Optional[str] = None +) -> SapVentilation: return SapVentilation( ventilation_type=ventilation.ventilation_type, mechanical_ventilation_kind=_pashub_mechanical_ventilation_kind( ventilation.ventilation_type ), + sheltered_sides=_PASHUB_BUILT_FORM_TO_SHELTERED_SIDES.get( + (built_form or "").strip().lower() + ), draught_lobby=ventilation.draught_lobby, + # `has_draught_lobby` is the canonical Β§2 (13) gate the cascade reads in + # preference to the legacy `draught_lobby` field; the site-notes path + # previously left it None, dropping the surveyed lobby. Mirror Elmhurst. + has_draught_lobby=ventilation.draught_lobby, pressure_test=ventilation.pressure_test, open_flues_count=ventilation.number_of_open_flues, closed_flues_count=ventilation.number_of_closed_flues, @@ -7817,16 +7887,26 @@ _PASHUB_HEAT_NETWORK_SYSTEM_TYPES: frozenset[str] = frozenset({ # PasHub Site-Notes main-heating "Controls" labels for a Table 4e Group 3 heat -# network β†’ their SAP code. All four "charging system linked to use of community -# heating" codes (2306/2310/2312/2314) are calculation-identical β€” control_type -# 3, DLF 1.00/1.00, temperature_adjustment 0.0 (`cert_to_inputs.py` -# `_CONTROL_TYPE_BY_CODE` / DLF / temp-adjustment maps) β€” so the cohort's -# "...room thermostat only" resolves to 2306, the domain's modal heat-network -# control code (`domain/epc/property_overlays/main_heating_system_overlay.py`, -# `HANDOVER_POST_S0380_176`). Every code here is in `_CONTROL_TYPE_BY_CODE`. +# network β†’ their SAP code. Table 4e Group 3 has two independent axes: +# * charging: flat-rate (DHW factor 1.05) vs linked-to-use (DHW factor 1.00); +# * control kit: no thermostat / room thermostat only (no TRVs) / thermostat +# + TRVs β†’ control_type 1 / 2 / 3 and space-charging factor 1.10 / 1.05 / 1.00. +# The label "charging system linked to use of community heating, room thermostat +# only" fixes BOTH axes: linked-to-use (DHW 1.00) AND thermostat-only/no-TRVs +# (type 2, space 1.05) β†’ code 2308/2309 (calculation-identical). It is NOT 2306, +# which is the linked-to-use WITH-TRVs code (type 3, space 1.00) β€” that asserts +# TRVs the label excludes and over-rated 16 Bingley (56.89 vs pashub 52; 2308 β†’ +# 54.01, the residual +2.0 being the documented SAP-10.2-engine-vs-lodged offset, +# NOT a fuel/flags gap β€” those were threaded by the #1590 follow-up). It is also +# NOT 2303, a FLAT-RATE code (DHW 1.05, type 1) whose oracle match (52.28) is a +# coincidence of two offsetting spec violations. Codes verified against +# `cert_to_inputs.py` `_CONTROL_TYPE_BY_CODE` (2308β†’2), space-charging (2308β†’1.05) +# and DHW (2308β†’1.00) maps + the RdSAP label vocab in +# `etl/epc_clean/epc_attributes/MainheatControlAttributes.py`. (Supersedes the +# earlier 2306 mapping and the ADR-0053 / HANDOVER_838 "fuel/flags" attribution.) _PASHUB_HEAT_NETWORK_CONTROL_TO_SAP10: Dict[str, int] = { "Charging system linked to use of community heating, room thermostat only": ( - 2306 + 2308 ), } diff --git a/datatypes/epc/domain/tests/test_from_site_notes.py b/datatypes/epc/domain/tests/test_from_site_notes.py index de16a931c..5a21a6db5 100644 --- a/datatypes/epc/domain/tests/test_from_site_notes.py +++ b/datatypes/epc/domain/tests/test_from_site_notes.py @@ -1,5 +1,6 @@ import json import os +from dataclasses import replace from datetime import date from typing import Any, Dict, Optional @@ -1677,9 +1678,11 @@ class TestFromSiteNotesExample1: ) def test_floor_height(self, result: EpcPropertyData) -> None: - # floors[0].height_m: 2.37 (Floor 1, upper) + # floors[0].height_m: 2.37 (Floor 1, upper) + 0.25 m joist-void = 2.62. + # The +0.25 m upper-storey add now mirrors the gov-API/Elmhurst mappers + # (see _map_floor_dimensions #1601 re-adjudication). assert ( - result.sap_building_parts[0].sap_floor_dimensions[0].room_height_m == 2.37 + result.sap_building_parts[0].sap_floor_dimensions[0].room_height_m == 2.62 ) def test_heat_loss_perimeter(self, result: EpcPropertyData) -> None: @@ -1913,7 +1916,7 @@ class TestFromSiteNotesExample1: party_wall_construction=4, # "Cavity Masonry, Unfilled" β†’ SAP10 code 4 (U=0.5) sap_floor_dimensions=[ SapFloorDimension( - room_height_m=2.37, + room_height_m=2.62, # 2.37 raw + 0.25 m upper-storey joist void total_floor_area_m2=24.78, party_wall_length_m=6.15, heat_loss_perimeter_m=14.21, @@ -1957,6 +1960,9 @@ class TestFromSiteNotesExample1: has_conservatory=False, blocked_chimneys_count=0, draughtproofed_door_count=2, + # Β§2 (15) draught-proofing: area-weighted % of draught-proofed + # windows (all 4 windows draught-proofed β†’ 100). Previously dropped. + percent_draughtproofed=100, report_reference="49D422A9-0779-44DD-9665-464D35DFF1A8", number_of_storeys=2, any_unheated_rooms=True, @@ -1974,11 +1980,139 @@ class TestFromSiteNotesExample1: flueless_gas_fires_count=0, pressure_test="No test", draught_lobby=False, + has_draught_lobby=False, + # RdSAP Β§S5 sheltered sides from built form: Mid-terrace β†’ 2. + # Previously left None (calculator default 2); now set explicitly + # so end/semi/detached get their correct 1/1/0 instead of 2. + sheltered_sides=2, ), ) assert result == expected +class TestFromSiteNotesInfiltrationFixes: + """Focused coverage for the three Β§2 infiltration inputs that + `from_site_notes` previously dropped β€” `percent_draughtproofed`, the + +0.25 m upper-storey joist void, and built-form `sheltered_sides`. All + three are set by the gov-API/Elmhurst sibling mappers; restoring them + together corrects a cohort-wide SAP under-rate vs the accredited SAP-10.2 + oracle (62.1%β†’83.3% within-0.5). Fixture example1 is a Mid-terrace house + with 4 all-draught-proofed windows and raw storey heights 2.35 (ground) / + 2.37 (upper). + """ + + @pytest.fixture + def survey(self) -> PasHubRdSapSiteNotes: + return from_dict( + PasHubRdSapSiteNotes, load("pashub_rdsap_site_notes_example1.json") + ) + + # --- percent_draughtproofed (Β§2 (15)) --- + + def test_all_windows_proofed_gives_100( + self, survey: PasHubRdSapSiteNotes + ) -> None: + result = EpcPropertyDataMapper.from_site_notes(survey) + assert result.percent_draughtproofed == 100 + + def test_percent_is_area_weighted_not_count_based( + self, survey: PasHubRdSapSiteNotes + ) -> None: + # Un-proof only the LARGEST window; the percentage must fall by that + # window's AREA share, which differs from a naive (N-1)/N count. + windows = survey.windows + big = max( + range(len(windows)), + key=lambda i: windows[i].width_m * windows[i].height_m, + ) + s = replace( + survey, + windows=[ + replace(w, draught_proofed=False) if i == big else w + for i, w in enumerate(windows) + ], + ) + result = EpcPropertyDataMapper.from_site_notes(s) + + total = sum(w.width_m * w.height_m for w in windows) + proofed = sum( + w.width_m * w.height_m for i, w in enumerate(windows) if i != big + ) + expected_area_pct = int(round(100.0 * proofed / total)) + count_pct = int(round(100.0 * (len(windows) - 1) / len(windows))) + assert result.percent_draughtproofed is not None + assert result.percent_draughtproofed == expected_area_pct + assert result.percent_draughtproofed != count_pct # proves AREA weighting + assert result.percent_draughtproofed < 100 + + def test_no_windows_proofed_gives_0( + self, survey: PasHubRdSapSiteNotes + ) -> None: + s = replace( + survey, windows=[replace(w, draught_proofed=False) for w in survey.windows] + ) + result = EpcPropertyDataMapper.from_site_notes(s) + assert result.percent_draughtproofed == 0 + + def test_no_windows_gives_none(self, survey: PasHubRdSapSiteNotes) -> None: + # No openings to weight β†’ None (not 0), so the calculator applies its + # own default rather than a spurious 0% draught-proofing. + s = replace(survey, windows=[]) + result = EpcPropertyDataMapper.from_site_notes(s) + assert result.percent_draughtproofed is None + + # --- +0.25 m upper-storey joist void --- + + def test_upper_floor_gets_joist_void_ground_does_not( + self, survey: PasHubRdSapSiteNotes + ) -> None: + result = EpcPropertyDataMapper.from_site_notes(survey) + parts = result.sap_building_parts + assert parts is not None + heights = {fd.floor: fd.room_height_m for fd in parts[0].sap_floor_dimensions} + assert heights[0] == 2.35 # ground floor: raw surveyed height + assert heights[1] == 2.62 # upper: 2.37 raw + 0.25 m joist void + + # --- built-form sheltered_sides (RdSAP Β§S5) --- + + @pytest.mark.parametrize( + "detachment_type, expected_sides", + [ + ("Detached", 0), + ("Semi-Detached", 1), + ("End-terrace", 1), + ("Mid-terrace", 2), + ("Enclosed End-terrace", 2), + ("Enclosed Mid-terrace", 3), + ], + ) + def test_sheltered_sides_from_built_form( + self, + survey: PasHubRdSapSiteNotes, + detachment_type: str, + expected_sides: int, + ) -> None: + s = replace( + survey, + general=replace(survey.general, detachment_type=detachment_type), + ) + result = EpcPropertyDataMapper.from_site_notes(s) + assert result.sap_ventilation is not None + assert result.sap_ventilation.sheltered_sides == expected_sides + + def test_unknown_built_form_leaves_sheltered_sides_none( + self, survey: PasHubRdSapSiteNotes + ) -> None: + # An unmapped label must fall through to None (calculator default 2), + # never silently mis-map to a wrong shelter count. + s = replace( + survey, general=replace(survey.general, detachment_type="Park home") + ) + result = EpcPropertyDataMapper.from_site_notes(s) + assert result.sap_ventilation is not None + assert result.sap_ventilation.sheltered_sides is None + + class TestFromSiteNotesVentilation: """ Fixture: pashub_rdsap_site_notes_example1.json @@ -2428,8 +2562,9 @@ class TestPasHubMainHeatingControlCoding: def test_heat_network_control_maps_to_group3_code(self) -> None: # Arrange β€” a Table 4e Group 3 heat network (`system_type = # "Community heating system"`). Its control label carries a Group 3 - # (23xx) code, not the boiler 21xx; the accredited Elmhurst certs for - # this same cohort lodge 2306 for it (HANDOVER_POST_S0380_176). + # (23xx) code, not the boiler 21xx. "charging linked to use … room + # thermostat only" is linked-to-use (DHW 1.00) + thermostat-only/no-TRVs + # β†’ code 2308, control type 2 (NOT 2306, the with-TRVs type-3 code). from domain.sap10_calculator.rdsap.cert_to_inputs import ( _control_type, # pyright: ignore[reportPrivateUsage] ) @@ -2452,9 +2587,10 @@ class TestPasHubMainHeatingControlCoding: survey ).sap_heating.main_heating_details[0] - # Assert β€” the Group 3 code, and its spec control type (3). - assert main.main_heating_control == 2306 - assert _control_type(main) == 3 + # Assert β€” the Group 3 linked-to-use, thermostat-only code and its + # spec control type (2). + assert main.main_heating_control == 2308 + assert _control_type(main) == 2 def test_heat_network_unmapped_control_label_strict_raises(self) -> None: # A community-system control label the Group 3 lookup does not cover @@ -2472,6 +2608,54 @@ class TestPasHubMainHeatingControlCoding: EpcPropertyDataMapper.from_site_notes(survey) +class TestPasHubLowEnergyLighting: + """When the assessor does not enumerate exact LED/CFL bulbs, PAS Hub lodges + an aggregate "Number of fixed low energy lights?" count. `from_site_notes` + must thread it into `low_energy_fixed_lighting_bulbs_count` (mirroring the + Elmhurst path) so Β§L uses the surveyed low-energy count rather than the + pessimistic no-lighting-data default that under-rates SAP.""" + + def test_aggregate_count_threaded_when_exact_unknown(self) -> None: + data = load("pashub_rdsap_site_notes_example1.json") + data["room_count_elements"]["exact_led_cfl_count_known"] = False + data["room_count_elements"]["number_of_fixed_low_energy_lights"] = 9 + survey = from_dict(PasHubRdSapSiteNotes, data) + + result = EpcPropertyDataMapper.from_site_notes(survey) + + assert result.low_energy_fixed_lighting_bulbs_count == 9 + + def test_aggregate_count_ignored_when_exact_counts_known(self) -> None: + # When exact LED/CFL counts are lodged, the per-type counts are + # authoritative and the aggregate field must not shadow them. + data = load("pashub_rdsap_site_notes_example1.json") + data["room_count_elements"]["exact_led_cfl_count_known"] = True + data["room_count_elements"]["number_of_fixed_low_energy_lights"] = 9 + survey = from_dict(PasHubRdSapSiteNotes, data) + + result = EpcPropertyDataMapper.from_site_notes(survey) + + assert result.low_energy_fixed_lighting_bulbs_count is None + + +class TestPasHubDraughtLobby: + """`from_site_notes` must set the canonical Β§2 (13) gate `has_draught_lobby` + (the cascade reads it in preference to the legacy `draught_lobby` field); + leaving it None dropped the surveyed lobby and over-stated infiltration.""" + + @pytest.mark.parametrize("surveyed, expected", [(True, True), (False, False)]) + def test_has_draught_lobby_threaded_from_survey( + self, surveyed: bool, expected: bool + ) -> None: + data = load("pashub_rdsap_site_notes_example1.json") + data["ventilation"]["draught_lobby"] = surveyed + survey = from_dict(PasHubRdSapSiteNotes, data) + + result = EpcPropertyDataMapper.from_site_notes(survey) + + assert result.sap_ventilation.has_draught_lobby is expected + + class TestPasHubUnmappedHeatEmitter: """A PasHub survey whose surveyed main-heating emitter label the mapper does not yet cover must strict-raise `UnmappedPasHubLabel` at the boundary, diff --git a/datatypes/epc/surveys/pashub_rdsap_site_notes.py b/datatypes/epc/surveys/pashub_rdsap_site_notes.py index 6b8c77de2..1a1135c8a 100644 --- a/datatypes/epc/surveys/pashub_rdsap_site_notes.py +++ b/datatypes/epc/surveys/pashub_rdsap_site_notes.py @@ -317,6 +317,7 @@ class RoomCountElements: number_of_fixed_cfl_bulbs: int waste_water_heat_recovery: str number_of_heated_rooms: Optional[int] = None + number_of_fixed_low_energy_lights: Optional[int] = None @dataclass diff --git a/domain/sap10_calculator/rdsap/cert_to_inputs.py b/domain/sap10_calculator/rdsap/cert_to_inputs.py index 1fc568e03..cd68d5099 100644 --- a/domain/sap10_calculator/rdsap/cert_to_inputs.py +++ b/domain/sap10_calculator/rdsap/cert_to_inputs.py @@ -7910,7 +7910,26 @@ def cert_to_inputs( if water_main is not None and water_main.main_heating_index_number is not None else None ) - if water_pcdb_main is not None and water_pcdb_main.summer_efficiency_pct is not None: + # DHW from a SEPARATE electric immersion heater (WHC 903) is not served by + # any boiler β€” SAP 10.2 Table 4a lists the immersion at 100% efficiency and + # SAP Β§9.4.11 / Appendix D2.1 Eq D1 (the boiler seasonal-efficiency cascade) + # applies only when "the boiler provides both space and water heating". But + # `_water_heating_main` resolves to the SPACE main (a gas/oil combi still has + # a PCDB record), so without this gate both the scalar `water_eff` and the + # Eq-D1 pair below would bill electric-immersion DHW at the boiler's ~87% + # summer efficiency instead of 100% (mongrel: electric fuel price Γ— gas-boiler + # efficiency). Fires for #1600's no-water-heating default (WHC 999 β†’ 903) with + # a gas-boiler main; the immersion path (`_water_efficiency_with_category_ + # inherit` β†’ 1.0) is the correct fallback, mirroring the Table 3 zero-primary- + # loss gate already applied for WHC 903. + dhw_is_electric_immersion = ( + epc.sap_heating.water_heating_code == _WHC_ELECTRIC_IMMERSION + ) + if ( + water_pcdb_main is not None + and water_pcdb_main.summer_efficiency_pct is not None + and not dhw_is_electric_immersion + ): water_eff = water_pcdb_main.summer_efficiency_pct / 100.0 else: water_eff = _water_efficiency_with_category_inherit( @@ -7972,6 +7991,10 @@ def cert_to_inputs( no_interlock and water_pcdb_main is not None and epc.has_hot_water_cylinder + # Not for a separate electric immersion (WHC 903): the -5pp boiler + # interlock adjustment has no meaning for a 100%-efficient immersion + # element (see the electric-immersion note above). + and not dhw_is_electric_immersion ): water_eff -= 0.05 # Resolve the (winter, summer) seasonal efficiency pair that feeds @@ -7993,6 +8016,10 @@ def cert_to_inputs( pcdb_main is not None and pcdb_main.winter_efficiency_pct is not None and pcdb_main.summer_efficiency_pct is not None + # Not when DHW is a separate electric immersion (WHC 903): the boiler + # does not provide water heating, so Eq D1 must not apply its seasonal + # efficiency to the immersion-heated cylinder (see the note above). + and not dhw_is_electric_immersion ): eq_d1_winter_summer_pct = ( pcdb_main.winter_efficiency_pct, diff --git a/tests/domain/sap10_calculator/rdsap/test_cert_to_inputs.py b/tests/domain/sap10_calculator/rdsap/test_cert_to_inputs.py index 578cb67de..a59aef297 100644 --- a/tests/domain/sap10_calculator/rdsap/test_cert_to_inputs.py +++ b/tests/domain/sap10_calculator/rdsap/test_cert_to_inputs.py @@ -962,6 +962,61 @@ def test_off_peak_immersion_unlodged_type_defaults_to_dual_table13_blend() -> No assert inputs.hot_water_high_rate_fraction == pytest.approx(_frac_from_cost) +def test_electric_immersion_dhw_not_billed_at_gas_boiler_efficiency() -> None: + # Arrange β€” a separate electric immersion heater (WHC 903) is 100% efficient + # (SAP 10.2 Table 4a) and the boiler provides no water heating, so Appendix + # D2.1 Eq D1 (the boiler seasonal-efficiency cascade) does not apply. But + # `_water_heating_main` resolves to the SPACE main β€” a gas combi keeps its + # PCDB record β€” so pre-fix both the scalar water efficiency and the Eq-D1 + # winter/summer pair billed the immersion-heated cylinder at the boiler's + # ~87% summer efficiency (a mongrel: electric fuel price Γ— gas-boiler + # efficiency). Fires for #1600's no-water-heating default (WHC 999 -> 903) + # on a gas-boiler main β€” 58 Hackle St M11 4WU (SAP 55.30 -> 57.67). + from dataclasses import replace + + pcdb_gas_boiler = replace( + _gas_boiler_detail(sap_main_heating_code=102), + # PCDB gas combi (winter 88.7% / summer 87.0%) β€” a real record so the + # buggy branches would have summer-efficiency data to (wrongly) apply. + main_heating_index_number=16839, + ) + boiler_epc = _cylinder_epc(cylinder_size=2, main=pcdb_gas_boiler) + boiler_epc = replace( + boiler_epc, + sap_heating=replace( + boiler_epc.sap_heating, water_heating_code=903, water_heating_fuel=29 + ), + ) + + # A non-boiler electric main (SAP code 401, no PCDB record) is the reference + # for a correctly-billed 100%-efficient immersion: Eq D1 cannot engage. + electric_main = MainHeatingDetail( + has_fghrs=False, + main_fuel_type=29, + heat_emitter_type=0, + emitter_temperature=1, + main_heating_control=2401, + main_heating_category=7, + sap_main_heating_code=401, + ) + electric_epc = replace( + boiler_epc, + sap_heating=replace(boiler_epc.sap_heating, main_heating_details=[electric_main]), + ) + + # Act + boiler_inputs = cert_to_inputs(boiler_epc) + electric_inputs = cert_to_inputs(electric_epc) + + # Assert β€” the immersion DHW energy is identical whether the SPACE main is a + # PCDB gas boiler or a non-boiler electric system: the boiler's efficiency + # never touches the immersion-heated cylinder. Pre-fix the boiler variant was + # inflated by ~1/0.878. + assert boiler_inputs.hot_water_kwh_per_yr == pytest.approx( + electric_inputs.hot_water_kwh_per_yr + ) + + def test_non_integrated_storage_heater_bills_100_percent_low_rate() -> None: # Arrange β€” same off-peak storage cert but SAP code 401 ("other storage # heaters"): Table 12a Grid 1 gives a 0.00 high-rate fraction β†’ the heat diff --git a/tests/infrastructure/epc_client/test_sap_accuracy_corpus.py b/tests/infrastructure/epc_client/test_sap_accuracy_corpus.py index a0a45a31e..8e90dac15 100644 --- a/tests/infrastructure/epc_client/test_sap_accuracy_corpus.py +++ b/tests/infrastructure/epc_client/test_sap_accuracy_corpus.py @@ -386,7 +386,12 @@ _MIN_WITHIN_HALF_SAP = 0.788 # The -0.672 overshoot is expected to move again when the C4 plant- # efficiency and C3.2 pumping siblings land. Unit-pinned in # test_cert_to_inputs. -_MAX_SAP_MAE = 0.626 +# Ratcheted 0.626 -> 0.625 by the electric-immersion water-efficiency fix +# (WHC 903 DHW no longer billed at the space boiler's PCDB seasonal +# efficiency / interlock -5pp β€” SAP Table 4a immersion = 100%): observed +# within-0.5 78.9%, MAE 0.622. A handful of corpus certs with a gas-boiler +# space main + separate electric immersion moved closer to accredited. +_MAX_SAP_MAE = 0.625 _MAX_CO2_MAE_TONNES = 0.072 # t CO2 / yr vs co2_emissions_current _MAX_PE_PER_M2_MAE = 3.0 # kWh / m2 / yr vs energy_consumption_current