A SAP-16.x cohort cert (9258-4062-7265-2844-7954) lodges a bare int building-
part identifier (the second part as `1` after "Main Dwelling").
`BuildingPartIdentifier.from_api_string` regex-matched it assuming a string and
raised "TypeError: expected string or bytes-like object, got 'int'", failing the
whole property.
Fix: guard the match on `isinstance(api_identifier, str)` so a non-string
identifier falls to OTHER, matching the documented "anything unrecognised ->
OTHER" contract. The baseline SAP fabric sums all building parts regardless of
identifier, so OTHER is SAP-neutral; the identifier only labels parts for
measure targeting. Fixes every mapper (all route through from_api_string).
Cert now maps + calculates (sap 63). Regression test added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 modelling_e2e properties failed with KeyError: 'maxArrayPanelsCount'.
Google returns a `solarPotential` block with no array-level sizing fields
(`maxArrayPanelsCount` / `panelCapacityWatts`) for buildings with no usable
solar estimate. `SolarPotential.from_building_insights` hard-indexed those keys
and crashed the whole property.
Fix: the projection now returns Optional and yields None when those fields are
absent — the established "no solar potential" outcome (the orchestrator and
recommendation path already type it Optional and skip solar on None). Existing
callers (`_solar_potential_for`, harness) already assign to Optional.
Regression test + `assert is not None` narrowing on the valid-fixture tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
10 modelling_e2e properties failed with "unmapped SAP code in fuel_code: 10":
the billing layer (`sap_code_to_fuel`) had no carrier for Table-32 code 10
(dual fuel, mineral + wood) and raised rather than guess one.
SAP 10.2 treats dual fuel as its OWN fuel (its own Table-12 factors), so model
it as its own billing carrier rather than collapsing onto wood or coal:
- New `Fuel.DUAL_FUEL_MINERAL_AND_WOOD`.
- `_CODE_TO_FUEL[10]` -> that carrier.
- Fuel Rates snapshot prices it at 7.69 p/kWh — the midpoint of the COAL proxy
(7.13) and WOOD_LOGS (8.25). This mirrors SAP's own construction: Table-32
dual fuel (3.99) ~= midpoint of house coal (3.67) and wood logs (4.23).
Marked `derived` with a documented _note/_gap/_assumption (like the COAL and
HEAT_NETWORK proxies), since there is no retail blend price.
A dedicated carrier + rate (vs a one-line map to an existing carrier) keeps the
fuel identity faithful to SAP and avoids mispricing dual fuel as pure wood/coal.
Tests: code 10 -> DUAL_FUEL carrier; snapshot prices it at 7.69; grid-export
codes (36/60) still raise (the genuine no-carrier case).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the mapper-driven e2e cascade pin for "simulated case 50" (000565 semi,
electric storage main SAP 402 + portable electric secondary + MVHR + whc-903
DUAL electric immersion + 160 L cylinder, Economy-7). Routes the Summary PDF
through extractor + mapper + calculator like the other 000565 fixtures.
Locks in two off-peak fixes this case ground-truthed:
- the Table 13 HW high/low split applied to CO2/PE (commit 39ae2cf0), and
- the Table 12a Grid 2 MVHR fan fraction 0.71/0.58 (commit cd5113ab).
All 11 SAP-result fields reconcile to the U985 worksheet EXACTLY, including
the (272) rating CO2 2397.1237 — SAP 38.8426 (=39), cost £1317.0116, water
1668.0788 kWh, fans 315.6384 kWh.
Summary mirrored to the tracked fixtures dir so the test doesn't depend on
the unstaged `sap worksheets/` workspace.
pyright strict gate not run locally (pyright not installed in this container).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SAP 10.2 Table 12d/12e: electric water heating on a 7-/10-hour tariff bills
CO2/PE at the high-rate code (32/34) and low-rate code (31/33), kWh-weighted
by the Table 13 high-rate fraction. The cost path already applied this split;
the CO2/PE factors did not — they used the flat annual Table 12 figure
(0.136 CO2 / 1.501 PE) for ALL dual-rate electric HW.
That flat-annual behaviour (slice S0380.163) was validated only against
HW-from-main "low-rate cost" certs (100% low, no high-rate split). It is NOT
how Elmhurst bills a whc-903 ELECTRIC IMMERSION: the hand-built case-50
worksheet (000565 + dual immersion, 7-hour) splits HW CO2/PE into "high rate
cost" (CO2 0.1475 / PE 1.5514) + "low rate cost" (CO2 0.1238 / PE 1.4429)
weighted by the Table 13 fraction 0.1009. So flat-0.136 for immersion HW was
a spec gap on our side, not an Elmhurst divergence.
Fix: `_electric_immersion_hw_high_rate_fraction` threads the Table 13 fraction
(scoped to whc-903, 7-/10-hour, cylinder data present) into the HW CO2 + PE
factor helpers, which then blend the Table 12d/12e high/low codes. The flat
rule is unchanged for HW-from-main and 18-/24-hour (no Table 12d split), so
the S0380.163 41-variant cases and the existing pin are untouched.
Case 50: rating CO2 2413.48 -> 2397.1237 = Elmhurst EXACT; demand CO2 2007.1384
EXACT; demand PE +111 -> +32.5 residual (within corpus PE noise). Corpus
unchanged 73.3% / MAE 0.774 / CO2 0.08 / PE 3.4 (62 whc-903 off-peak certs;
aggregate gauges hold). SAP unaffected (cost-based).
Pin: test_whc903_immersion_hw_co2_pe_factors_split_high_low_on_off_peak; doc
updated in SAP_CALCULATOR.md §8.1.
pyright strict gate not run locally (pyright not installed in this container).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
16 modelling_e2e properties failed with "Effective EPC is missing
renewable_heat_incentive; cannot read baseline space-heating / hot-water kWh".
Baseline runs for predicted properties too (ADR-0031), reading space/water-
heating kWh off the EPC's lodged RHI block. Predicted EPCs deep-copy a neighbour
template that may carry no RHI, so `_require_rhi` hard-failed the whole subtask.
Fix: when the EPC has no RHI, fall back to the property's OWN computed figures
from the scored SapResult (space_heating_kwh_per_yr / hot_water_kwh_per_yr) —
more representative than a neighbour's lodged numbers. Only when there is also no
SapResult (the rebaseliner scored nothing) is there genuinely no demand to
record, and we still fail noisily. Lodged certs are unchanged (RHI still wins).
Regression tests: fallback-to-computed, and the no-RHI/no-result raise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the mapper-driven e2e cascade pin for "simulated case 52" (000565
semi + regular non-combi mains-gas boiler SAP 102 + 160 L foam cylinder
heated from the main, no cylinder stat, uninsulated primary pipework,
standard tariff). Routes the Summary PDF through extractor + mapper +
calculator like the other 000565 / 001431_case* fixtures.
This closes the last untested branch of the cylinder/water chain: the
SAP 10.2 §4 cylinder storage loss (Table 2/2a/2b lines 51-55) + the
Table 3 PRIMARY circuit loss (59, uninsulated pipework + no stat) that
combi/immersion fixtures don't reach. All 11 SAP-result fields reconcile
to the U985 worksheet EXACTLY with no calculator change — SAP 57.2904
(=57), cost £911.1973, water 3929.7635 kWh — confirming the cylinder-loss
derivation is correct.
Summary mirrored to the tracked fixtures dir so the test doesn't depend
on the unstaged `sap worksheets/` workspace.
pyright strict gate not run locally (pyright not installed in this container).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fitting sealed glazing units changes two things beyond the pane's U/g
that the cascade reads, which the overlay didn't model — leaving the
double/secondary before→after pins ~0.7 SAP short (xfail):
1. Draught-proofing (RdSAP 10 §8.1). Sealed units draught-proof the panes
they replace, re-lodging the dwelling-level `percent_draughtproofed`
(cert 001431: 84 → 100). The §2 cascade reads that dwelling-level
value, so the overlay now carries it. `_recompute_percent_draughtproofed`
anchors on the lodged before-% — `after = round((round(before%/100 × N)
+ flips) / N × 100)`, N = openable windows (vertical + roof) + doors,
flips = upgraded panes that were not draught-proofed — so it's robust
to incomplete window extraction (unchanged openings are already in the
aggregate). ~0.3 SAP.
2. Frame factor (§6 solar gains). A replacement unit re-lodges its own
FF=0.70, overriding the pane it replaced — the two "single glazing,
known data" panes lodge FF 1.00 / 0.50 (one is 6.6 m²), so leaving them
unchanged understated solar gains by ~+150 kWh space heating. `WindowOverlay`
now carries `frame_factor`, written flat onto the window. ~0.4 SAP.
Wiring: `EpcSimulation.percent_draughtproofed` + `WindowOverlay.frame_factor`
new fields; `apply_simulations` / `_fold_window` write them; the glazing
generator computes both from the upgraded set and cert 001431's after.
Un-xfails `test_{double,secondary}_glazing_overlay_reproduces_the_relodged_after`
— both now pin SAP/CO2/PE to the relodged after within tolerance. Updates
the two `test_glazing_recommendation` overlay expectations for the new
`frame_factor`. 96 modelling tests pass; zero new pyright errors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SAP 10.2 Table 12a Grid 2 (PDF p.191) bills "Fans for mechanical ventilation
systems" at 0.71 (7-hour) / 0.58 (10-hour), distinct from "All other uses"
(0.90 / 0.80) which covers circulation pumps, flue fans and the solar HW pump.
The cost-split mech-vent kWh (`mev_kwh_for_cost_split`) only summed the
decentralised-MEV (230b) fans, not the (230a) MVHR fan electricity — even
though the total pumps/fans bucket adds both. So an MVHR dwelling on an
off-peak tariff billed its fan electricity at the 0.90/0.80 "all other uses"
rate instead of 0.71/0.58. The comment already said "MEV/MVHR-fan portion";
only the MEV term was wired when MVHR landed. Fixed to mirror both
mechanical-ventilation fan terms summed into the total.
Worksheet-proven on simulated case 50 (000565 semi + MVHR Vent Axia + dual
electric immersion, Unknown meter -> 7-hour via the §12 dual-immersion
trigger): the fan bucket (315.64 kWh, 100% MVHR per worksheet line 230a) was
billing at 14.311 p/kWh (0.90) vs Elmhurst's 12.451 p/kWh (0.71) — +£5.87/yr,
-0.23 SAP. After the fix our existing-dwelling rating reconciles to Elmhurst
EXACTLY: SAP value 38.8426 (=Elmhurst 38.8426 -> 39), total cost £1317.0116
(=Elmhurst £1317.0116 to the penny).
Same `mev_kwh_for_cost_split` feeds the CO2 + PE cascades, so all three split
consistently. 0 corpus impact (all 3 corpus MVHR certs are standard tariff);
gauge unchanged 73.3% / MAE 0.774 / CO2 0.08 / PE 3.4.
Pin: test_mvhr_fan_electricity_bills_at_grid2_fan_fraction_on_off_peak.
pyright strict gate not run locally (pyright not installed in this container).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SAP 10.2 Table 4a electric boilers (PDF p.170) split across three distinct
Table 12a Grid 1 SH rows (PDF p.191), not one "direct-acting" family as the
stale TODO in `_table_12a_system_for_main` implied:
- 191 Direct-acting electric boiler -> "Direct-acting electric boiler (a)"
row: 7-hour 0.90, 10-hour 0.50 (NOT the 1.00/0.50 "Other direct-acting
electric heating" room-heater row).
- 193/194/195/196 Electric dry core / water storage boiler -> "Electric dry
core or water storage boiler" row: 7-hour 0.00 (charged wholly off-peak =
100% low rate, identical to the None fallback).
- 192 Electric CPSU -> Appendix F; left falling through to None (off-peak
low) until the Appendix-F high-rate cascade is implemented.
The enum + fractions already existed in table_12a.py; only the code->enum
mapping was missing. Resolves the TODO and pins the spec-correct 0.00 for the
storage boilers so 195 can't be mis-"fixed" up to a direct-acting fraction.
Forward guard, 0 corpus impact: storage boilers already billed 100% low via
the None fallback, and all corpus 191 certs are on standard tariff (Table 12a
off-peak split never fires). Corpus gauge unchanged 73.3% / MAE 0.774.
Pin: test_electric_boilers_191_195_map_to_distinct_table_12a_grid1_rows.
pyright strict gate not run locally (pyright not installed in this container).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SAP 10.2 Table 3a (PDF p.160) additional combi loss (61)m. Two coupled
defects, both surfaced by simulated case 49 (000565 + gas combi, U985
"Combi keep hot type = None") sitting at SAP 71.43 vs the worksheet's 72:
1. The cascade defaulted EVERY non-PCDB combi to the flat keep-hot
time-clock row (600 × n/365). A combi WITHOUT a keep-hot facility uses
row 1 (600 × fu × n/365, fu = V_d/100 when daily HW < 100 L/day) —
over-counting (61)m for the no-keep-hot cohort. `water_heating_from_
cert` now defaults to the "without keep-hot" row.
2. `pcdb_combi_loss_override` returned None for keep_hot_facility=1/
timer=1, leaning on the OLD flat-600 default. So flipping the default
silently turned 190 corpus PCDB keep-hot-time-clock combis into
no-keep-hot. Fixed to return the flat keep-hot row EXPLICITLY.
Key insight (the Summary is the input echo; the U985 keep-hot line is a
computed OUTPUT, so it must be derivable): keep-hot rides on the PCDB
boiler record (Table 105 keep_hot_facility/timer), resolved by
`pcdb_combi_loss_override`. A generic SAP-code combi with no PCDB record
(case 49, PCDF ref 0) has no keep-hot by construction → row 1. So the
default is not a guess — it is the spec-correct value for non-PCDB combis.
Worksheet-proven: case 49 → cost £726.696, SAP 72 — matching the
accredited worksheet to the digit (continuous 71.6945 = the worksheet's
own 71.6945). 000516 (keep-hot None) also exact (£860.716, SAP 63);
000490 (PCDB 10328, keep_hot_facility=1/timer=1) keeps its flat-600 via
the PCDB path. Masked until now because every prior combi-loss worksheet
fixture was keep-hot (000490/000474/000480 time-clock) or had V_d >= 100
every month (001431, rows coincide); case 49 is the first no-keep-hot one.
Corpus within-0.5 72.7% -> 73.3%, MAE 0.781 -> 0.774, PE 3.5 -> 3.4;
ratcheted _MAX_SAP_MAE 0.785 -> 0.775, _MAX_PE_PER_M2_MAE 3.6 -> 3.5.
Note: pyright strict type gate not run locally (pyright not installed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the dMEV intermittent-fan fix (4db05e84) to MVHR. A balanced
whole-house MVHR system IS the dwelling's ventilation, so the lodged (7a)
intermittent-extract-fan count is explicit — a lodged 0 means 0, not the
RdSAP 10 Table 5 age-band "unknown" default. The cascade was substituting
the default (here 20 m³/h) into worksheet line (8) openings, inflating
(16/18) infiltration → (21) → (22b) → (25) effective ach → (38)
ventilation heat loss → the space-heating demand.
Worksheet-proven on simulated case 49 (000565 + Vent Axia 500140 MVHR,
lodged (7a)=0): our (8) openings 0.0723 -> 0.0000, (18) 0.7223 -> 0.6500,
(25)m Jan 0.9423 -> 0.8571, all now matching Elmhurst exactly; space-
heating demand 7857 -> 7528 kWh (worksheet 7546). SAP 70.90 -> 71.43
continuous. (The residual to the worksheet's 72 is its own continuous SAP
71.69 rounding up, driven by a separate gas-combi water-heating-loss gap,
not ventilation.)
Scoped to EXTRACT_OR_PIV_OUTSIDE + MVHR only — MV-without-HR
(mechanical_ventilation=1) stays on the default-substitution path
(forcing its lodged 0 regressed 47 Howsman / 18 Jutland and is not
worksheet-validated). Corpus within-0.5 holds 72.7%, MAE 0.782 -> 0.781.
Note: pyright strict type gate not run locally (pyright not installed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MVHR (24a) heat-recovery support, part 2: the mapper + cascade wiring.
Both source paths now resolve balanced whole-house MV with heat recovery
to the MVHR kind:
- gov-API: `_API_MECHANICAL_VENTILATION_TO_KIND` 4 → "MVHR" (was None /
treated as natural — under-stated ventilation heat loss, over-rating).
- Elmhurst Summary: `_ELMHURST_MV_TYPE_TO_KIND` "Mechanical ventilation
with heat recovery (MVHR)" → "MVHR" (was UnmappedElmhurstLabel, which
blocked the whole Summary for MVHR dwellings).
cert_to_inputs resolves the in-use heat-recovery efficiency + SFP for an
MVHR cert (`_mvhr_system_values`): pick the PCDB Table 323 data point by
the lodged wet-room count (SAP 10.2 §2.6.4), multiply the raw efficiency
by the Table 329 ducts-inside-envelope in-use factor (0.90) and the raw
SFP by the per-duct-type factor (rigid 1.4), and feed:
- the §2.6.6 eq (2) effective-air-change credit (23c) → (24a)/(25)m;
- the (230a) fan electricity (in-use SFP × 1.22 × V), costed but NOT
added to the Table 5a gains (its effect is in the efficiency).
An MVHR lodged with no PCDF index falls back to the SAP 10.2 Table 4g
default (raw efficiency 66% × 0.70, raw SFP 2.0 × 2.5).
Worksheet-proven on simulated case 49 (000565 semi + Vent Axia Sentinel
Kinetic B 500140 + gas combi → Elmhurst Current SAP 72): every MVHR line
matches Elmhurst exactly — (33) fabric heat loss 100.5923, (23c) in-use
efficiency 81.9% = 91 × 0.90, (25)m Jan 0.8571, (230a) fan electricity
415.9325, (231) total pumps/fans 501.9325. The residual SAP 71 vs 72 is
the known 000565-family space-heating-demand artifact (same -1/-2 seen on
cases 47/48), not the MVHR logic.
Corpus: within-0.5 72.6% -> 72.7%, MAE 0.788 -> 0.782, PE 3.6 -> 3.5.
The 3 gov-API MVHR certs: Flat 1 +6 -> 0 (Table 4g default path) and
12a Princes Gate +3 -> +1 (heat-recovery credit); Apartment 707 -4 -> -6
is a separate baseline under-rate (it under-rated as natural too — the
MVHR credit correctly adds ventilation loss per Elmhurst's method).
Ratcheted _MAX_SAP_MAE 0.79 -> 0.785, _MAX_PE_PER_M2_MAE 3.7 -> 3.6.
Note: pyright strict type gate not run locally (pyright not installed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MVHR (24a) heat-recovery support, part 1: the PCDB data layer.
PCDB Table 323 (PCDF Spec Rev 6b §A.18, Format 426; pcdb10.dat carries
Format 431, header `$323,431,...`) holds the per-wet-room SFP + heat-
exchanger efficiency for centralised MEV / MVHR units. Added
`MvhrRecord` / `MvhrDataPoint`, `parse_centralised_mv_row` /
`parse_table_323`, the ETL step, the committed jsonl, and the
`mvhr_record(pcdb_id)` runtime lookup (mirrors Table 322).
SAP 10.2 §2.6.4/§2.6.6: "MVHR ... SFP is a single value depending on the
number of wet rooms" — each test group's leading field is the wet-room
count; callers select the group matching the dwelling lodgement.
Worksheet-proven on simulated case 49 (000565, 2 wet rooms, Vent Axia
Sentinel Kinetic B 500140 → flow 21.0, SFP 0.88, efficiency 91%).
Also decoded the MVHR heat-recovery efficiency in-use factor from Table
329 (Format 432): system_type 3 ducts-inside-envelope = 0.90 (case-49
(23c) = 91 × 0.90 = 81.9%), cross-checked against system_type 10 = 0.70
(= SAP 10.2 Table 4g default heat-recovery in-use factor). "Table 4h is
no longer used – data now stored in the PCDB" (SAP 10.2 p.176).
The outside-envelope efficiency columns + with-scheme SFP blocks are
preserved verbatim in `raw` (no fixture exercises them yet).
Note: pyright strict type gate not run locally (pyright not installed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Skipped cohort certs were previously surfaced only as outputs.result on a
completed subtask, so they were easy to miss. Treat them as a failure too:
once the batch has run to completion (so every modellable property is already
written to DB), raise if there were any per-property errors OR any skipped
certs. The run gets flagged for debugging without discarding the work done.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A predicted Property (no lodged EPC) got a Plan but nothing else: the synthesised
EPC was never written to epc_property, and Baseline Performance was skipped — so
property 729529 (portfolio 796 / scenario 1268), predicted from its DA16 1QZ
cohort, was "missed" with no predicted-EPC row and no baseline row.
Persist the synthesised EPC in the predicted slot (uow.epc.save(..., source=
"predicted"), ADR-0031) inside the Plan UoW, then run the Baseline orchestrator
for predicted Properties too — it re-hydrates the predicted EPC and establishes
the baseline from it. The earlier "lodged only" guard is dropped: by the write
block the Property always has a persisted EPC (lodged or predicted); one that
could be neither fetched nor predicted raised earlier.
Verified against the DB by invoking the real handler for 729529: predicted
epc_property rows 0->1 and property_baseline_performance rows 0->1. Baseline on
the predicted picture builds cleanly (RHI present, reason pre_sap10). Tests
updated: prediction + broadening paths now assert the predicted-slot epc.save and
the baseline run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The handler wrote epc/spatial/solar/plan and marked the property modelled, but
never established its Baseline Performance — so no row was created in
property_baseline_performance for any property modelled through the Lambda
(noticed on portfolio 796 / scenario 1268 / property 727218, a lodged property).
Mirror the e2e runner: after the plan UoW commits (so the EPC is persisted for
the orchestrator to re-hydrate), run PropertyBaselineOrchestrator for lodged
properties. Predicted properties have no lodged figures and no persisted EPC, so
they are skipped — consistent with the e2e runner and the ara_first_run Baseline
stage.
Verified 727218's baseline pipeline builds end-to-end in-memory (lodged_performance
→ CalculatorRebaseliner → bill → PropertyBaselinePerformance, reason pre_sap10).
Tests: lodged path asserts the orchestrator runs once; prediction path asserts it
does not.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Landlord Override's building_part is a positional index (0=main, 1=extension
1…, ADR-0004), but the gov-API EPC can label that slot differently (e.g. lodge
the 2nd part as 'other', not 'extension_1'). The previous fix skipped such
orphaned overrides, silently discarding the landlord's correction. Now the
override falls back onto the EPC's part at that position (via _resolve_part), so
the correction lands; only a position the EPC models no part at is skipped
(no geometry to model a wholly-absent part). Replaces the skip-only behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes out the cohort-broadening work with its decision record and consolidates
the retry plumbing.
ADR-0034 documents broadening the EPC-Prediction cohort to the real unit
postcodes nearest the target (via postcodes.io) when its own postcode holds no
same-type comparable — extending ADR-0031 decision 5. Records why postcodes.io
was chosen over council[] (whole-LA, no property_type in rows), a bulk Code-Point
Open / ONSPD dataset, and the OS Places radius API, and the lazy / nearest-first
early-stop / soft-fail policy. Broadening-specific docstrings now cite 0034.
Retry consolidation: extract the EPC client's call_with_retry into a shared
infrastructure/http_retry.py keyed off a generic TransientHttpError marker, so
the mechanism (exponential backoff, Retry-After) is shared while each client
keeps its own transient policy. EpcRateLimitError now subclasses TransientHttpError
(still an EpcApiError); PostcodesIoClient routes through the same helper, raising
TransientHttpError on 429/5xx and soft-failing to the seed once exhausted (the EPC
client propagates instead). Direct tests for the shared helper; EPC + postcodes.io
suites repointed at the shared sleep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reconciliations to make the modelling_e2e Lambda handler production-ready.
1. Price through the off-catalogue overlay, drop the workarounds
The handler priced through a plain ProductPostgresRepository and excluded
secondary_heating_removal / system_tune_up / system_tune_up_zoned to dodge
ProductNotFound (and a poisoning pgEnum DataError). Those measures are now
priced by catalogue_with_off_catalogue_overrides (already used by the e2e
runner and PostgresUnitOfWork), so the exclusions are removed and ALL measure
types are considered. This also fixes gas-boiler / single-glazed properties,
which Dan's handler never excluded and so still crashed (the standard
system_tune_up option is built unconditionally — the considered-measures
exclusion never actually gated it).
2. Broaden the EPC-Prediction cohort to nearby real postcodes (ADR-0031)
A property with no lodged EPC and no same-type comparable in its own postcode
(e.g. the only flat among houses) used to gate out and fail the subtask. The
gov EPC API cannot search by radius/outcode, so we resolve the real unit
postcodes physically nearest the target via postcodes.io (keyless; already a
trusted in-repo dependency) and walk them nearest-first until enough same-type
comparables surface. New PostcodesIoClient (transient-failure retry with
exponential backoff, soft-failing to the seed so broadening never breaks
prediction) and EpcComparablePropertiesRepository.candidates_near. Wired into
the handler and e2e runner; broadening is lazy (only on gate-out) and memoised
per (postcode, property_type).
Validated live: property 728476 (gas boiler) prices system_tune_up at GBP295;
property 718580 (lone flat in BR6 6BS) now predicts via nearby BR6 postcodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Corpus validation of the modelling_e2e photovoltaic_supply-as-list fix. Cert
6102-6227-8000-0083-2292 (RdSAP-20.0.0 semi, gas combi + 2× 1.14 kW PV arrays)
crashed from_rdsap_schema_20_0_0 on the measured-array list; the fix routes it
through the dict-tolerant _map_schema_21_pv. PV correctly credited: engine 61
(no PV) → 66 (+5). Built in Elmhurst (evidence: epc.json + summary + worksheet,
fabric+heating; the PV "New Technologies" Panel-details grid deferred): worksheet
55 = engine-on-Elmhurst-inputs 55 exactly → calculator faithful. The +6 engine-vs-
Elmhurst base-dwelling residual is the documented RdSAP-default gap (band-C cavity-
uninsulated suspended-floor semi). Pinned engine 66.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@task_handler never built or passed cloud_logs_url, so every app using
it (incl. modelling_e2e) ran run_subtask with the None default and the
CloudWatch deep-link was never saved onto the SubTask. @subtask_handler
did this correctly.
Extract the URL builder into a shared utilities/aws_lambda/cloud_logs.py
(public cloudwatch_url()), use it from both handlers, and pass the URL
into run_subtask from @task_handler. Add regression tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@task_handler never built or passed cloud_logs_url, so every app using
it (incl. modelling_e2e) ran run_subtask with the None default and the
CloudWatch deep-link was never saved onto the SubTask. @subtask_handler
did this correctly.
Extract the URL builder into a shared utilities/aws_lambda/cloud_logs.py
(public cloudwatch_url()), use it from both handlers, and pass the URL
into run_subtask from @task_handler. Add regression tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Corpus validation of the modelling_e2e built_form fix. Cert 8742-6624-9300-2780-4926
(SAP-Schema-16.0, ground-floor electric-storage-heater flat) omits built_form; the
mapper now derives it from dwelling_type. built_form is ML-only so the fix is
SAP-neutral: engine 66 = lodged 66 exactly. Built in Elmhurst (evidence: epc.json +
summary + worksheet): worksheet 54, engine-on-Elmhurst-inputs 53 ≈ 54 → calculator
faithful. The +12 engine-vs-Elmhurst is a build/input gap (cert size-1 small cylinder
unrepresentable in Elmhurst's Normal/110L-minimum entry → higher HW + reduced-field
16.0 defaults). Pinned engine 66.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chasing the space-heating demand gap on "simulated case 48" (main 691 + Unknown
meter + 903 dual immersion): our SAP 55 vs Elmhurst 57. Every §10a cost line
already matched to the penny; the residual was demand — our space-heating
energy 3849.8 kWh vs Elmhurst 3513.8 (+9.6%). Traced through the worksheet: our
ventilation heat loss (38) ran ~35.5 W/K vs Elmhurst 27.76 — we were adding 20
m3/h of intermittent extract fans (the Table 5 age-band default) on a dwelling
with a decentralised mechanical extract (dMEV) system that lodges 0 fans.
SAP 10.2 §2 (PDF p.13): a whole-house mechanical EXTRACT system provides
extraction via the (23a) 0.5 system air-change rate; the lodged intermittent
extract-fan count (7a) is then explicit — a lodged 0 means 0 (the dMEV is the
ventilation), NOT "unknown". The Table 5 default is an unknown-fallback for
NATURALLY ventilated dwellings only, so it must not be substituted here.
Fix: for EXTRACT_OR_PIV_OUTSIDE, take vc.intermittent_fans as-is (no age-band
default). Worksheet-proven on two dMEV builds of cert 000565: "case 48" lodges
(7a)=0 -> our SAP 55 -> 57 EXACT; the original 000565 fixture lodges (7a)=2 and
keeps 2 (its e2e pins are unchanged). An earlier draft that forced fans=0 broke
000565 (which legitimately has 2) — corrected to "lodged as-is".
within-0.5 72.5% -> 72.6%, MAE 0.789 -> 0.788; CO2/PE unchanged. The fix also
reduces a systematic under-rating bias in the 21-cert dMEV cohort (median dSAP
-0.22 -> -0.08). Scoped to EXTRACT_OR_PIV_OUTSIDE; balanced MVHR/MV kinds left
untouched pending their own worksheet. SAP-schema regression
test_18_0_0 pin 80 -> 81 (closer to its lodged 84, same cause). Spec-pinned in
test_cert_to_inputs (dMEV-lodged-0 vs natural-default). pyright not installed
in this container -- strict type gate not run locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the follow-up from 22fe4f41. RdSAP 10 §12's third Unknown-meter
exception bullet — "main heating is ground source or water source heat pump" —
was unimplemented. Add `_GROUND_OR_WATER_SOURCE_HEAT_PUMP_CODES` (Table 4a,
SAP 10.2 PDF p.176-177: ground 211/215/221/225 + warm-air 521/525; water
213/216/223/226 + warm-air 523/526) to the Unknown-meter off-peak triggers;
once dual, Rules 1-4 (Rule 3) resolve it to 10-hour.
AIR-source heat pumps (214/217/224/227, 524/527) are deliberately EXCLUDED —
the spec names only ground/water source. Verified the only Unknown-meter heat
pumps in the corpus are "3/10 Bedford House" (main 214 = AIR source), which
correctly KEEP STANDARD. 0 corpus certs carry a GSHP/WSHP on an Unknown meter,
so this is a spec-completeness forward guard (gauge unchanged 72.5% / 0.789),
same family as the Scotland-J wall / rafters-M roof 0-impact spec fixes.
Coverage gap noted in-code: a database-index heat pump without a 211/213-style
SAP code can't have its source type read from the code alone (rare).
Spec-pinned (test_unknown_meter_ground_or_water_source_heat_pump_triggers_off_
peak). Gates green: corpus 72.5%/0.789, batch worksheet 0 raised/0 diverge,
suite 2989 passed (2 known pre-existing fails). pyright not installed locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MAE improved 0.793 -> 0.789 via the Unknown-meter + dual-electric-immersion
off-peak trigger (commit 22fe4f41). Ratchet the ceiling so the gain can't
silently regress. within-0.5 unchanged (72.5%).
pyright not installed in this container — strict type gate not run locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Supersedes the previous "verified non-fix" doc (3548f1f3): the spec DOES make
this a fix — Khalim was right that the Unknown-meter branch is driven by the
heating/water system, not a blanket STANDARD.
RdSAP 10 §12 (PDF p.62): "If the electricity meter is unknown, treat as single
meter EXCEPT where main heating OR WATER HEATING are intended to run off an
off-peak tariff (per systems listed in the text box above) ... If that results
in a dual meter, assign tariff per rules 1 to 4." The text-box off-peak systems
include DUAL ELECTRIC IMMERSION. Our `rdsap_tariff_for_cert` only triggered the
Unknown→off-peak exception on a storage/CPSU MAIN — it ignored the
dual-electric-immersion WATER-heating trigger, so an Unknown-meter dwelling
with a non-storage main (e.g. room heaters) + dual immersion was billed
STANDARD (13.19p flat) when it should be dual → Rules 1-4 on the main.
Fix: thread `water_is_off_peak_dual_immersion` (whc 903 + immersion lodged dual
via `_immersion_is_single is False`) into the Unknown-meter branch; when any
text-box trigger is present, resolve via the same Rules 1-4 dispatch (room
heaters → Rule 3 → 10-hour). Single-immersion / instantaneous (whc 909) certs
correctly stay STANDARD (no text-box system).
Worksheet-validated on "simulated case 48" (main 691 + Unknown meter + 903 dual
immersion): Elmhurst 10-Hour Off Peak, SAP 57; ours 45 → 55 (7-hour gives 45,
confirming 10-hour). Flips exactly ONE corpus cert — Apartment 241 (the genuine
-5.38 under-rater, main 691 + dual immersion) -5.38 → -1.05; every other
Unknown+dual-immersion cert already has a storage main (Rule 2). Corpus
within-0.5 holds 72.5%, MAE 0.793 → 0.789 (improved). CO2/PE unchanged.
GSHP/WSHP-main trigger (the other §12 Unknown exception bullet) is a separate
follow-up. Gates green: corpus 72.5%/0.789, batch worksheet 0 raised/0 diverge,
000565 e2e 11/11, suite 2987 passed (2 known pre-existing fails). pyright not
installed in this container — strict type gate not run locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two second-main fuel errors mis-cost a dual-main dwelling whose two main systems
burn different fuels (SAP 10.2 §10a worksheet (213) bills main 2 at its own fuel):
1. Off-peak/legacy scalar cost path (calculator.py + cert_to_inputs.py): main 2's
kWh was priced at main 1's `space_heating_fuel_cost_gbp_per_kwh` scalar. Split
main 1 vs main 2 and price main 2 at its OWN rate via the new
`_main_2_space_heating_fuel_cost_gbp_per_kwh` (+ CalculatorInputs field).
Scoped to a NON-electric second main (wood/oil/coal) — an electric second
main keeps main 1's scalar (its off-peak Table 12a split is the deferred §10a
slice; per-system splitting it regresses the off-peak electric cohort, certs
13 Parkers Hill / 34 Dunley Road). 0 corpus impact (no corpus cert has a
non-electric second main on an off-peak meter).
2. Elmhurst Summary mapper (mapper.py): when §14.1 omits the Fuel Type cell, a
fuel-fired second main (room-heater SAP code) inherited main 1's fuel. Derive
it from the SAP code's Table 4a category (solid 631-636 -> house coal, gas ->
mains gas, liquid -> oil) before the main-1 inherit, mirroring
`_elmhurst_secondary_fuel_from_sap_code` (same modal sub-fuel caveat). Boiler
codes (<601) still inherit main 1 (case 6 oil rads+UFH).
simulated case 47 (electric room heaters + solid room heaters 633): our SAP
37.81 -> 55.09 vs Elmhurst current 57 (residual is the wood-vs-coal sub-fuel the
Summary export does not carry). Corpus unchanged 72.5% / MAE 0.793; batch 0
raised / 0 diverge; 000565 e2e green. (mapper.py also carries an earlier,
behaviour-free roof-window doc comment.) Spec-cited unit pins added (AAA).
pyright not installed locally — strict type gate not run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`table_32.unit_price_p_per_kwh` silently returned the mains-gas default
(3.48 p/kWh) for any fuel code it could not resolve to a Table 32 price or a
translatable gov-API enum. An unhandled fuel billed at the gas rate mis-costs
the dwelling (same failure mode as the dual-main wood-vs-electric over-cost).
Raise `UnpricedFuelCode` (new, mirrors MissingMainFuelType / UnmappedSapCode)
so the gap surfaces at the price boundary. `None` (no fuel lodged) still
defaults — callers resolve "no system" upstream.
0 corpus impact: all 1000 certs compute (every lodged fuel resolves), so this
is a forward guard against future/unmapped fuels. Unit pin added; existing
None-default test docstring tightened. pyright not installed locally — strict
type gate not run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Main heating system 2's space-heating fuel cost (worksheet (213)) was billed
at main system 1's Table 32 unit price (`main_2_high_rate_gbp_per_kwh` reused
`main_1_high_rate_gbp_per_kwh`). For a dual-FUEL pair this grossly mis-costs the
second main: cert 10032957680 "Copse Cottage" (main 1 electric room heaters
fuel 30, main 2 wood logs fuel 6) charged its 9481 kWh of wood at 13.19 p/kWh
instead of 4.23 p/kWh — +£850/yr → SAP 21.75 vs lodged 45.
Route main 2 through its own fuel code (`_main_fuel_code(details[1])`), mirroring
the existing secondary-fuel handling. Copse Cottage 21.75 -> 45.94. Corpus
within-0.5 holds 72.5%, SAP MAE 0.815 -> 0.793 (ratcheted ceiling 0.82 -> 0.80);
CO2/PE unchanged. Same-fuel dual mains (gas+gas) unaffected. Off-peak-tariff
dual-fuel mains still defer to the legacy scalar path (separate slice).
Spec-cited unit pin added (AAA). pyright not installed locally — strict type
gate not run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make run_modelling_e2e the single script that does everything for a portfolio,
so the 291-property run needs one invocation with per-property recovery (no
all-or-nothing chunking):
- On --persist, a lodged-EPC Property now also gets its Baseline Performance
row written via PropertyBaselineOrchestrator (per Property, so one bad cert
does not abort the batch). Predicted (EPC-less) Properties have no lodged
figures, so they get a Plan but no baseline row.
- The run CSV gains api_sap (register) vs baseline_sap (calculator) + sap_delta,
so calculator-vs-API divergence is reviewable per property.
Fill the off-catalogue overlay for the measures the live material catalogue
cannot price, so they stop crashing the run:
- double_glazing (£550/window) and secondary_glazing (£400/window): priced
per window (the generator multiplies by single-glazed window count, matching
the legacy window_glazing). Grounded in 2025/26 UK installed costs; per-window
is the right unit for windows (fixed per-unit install dominates) — per-m2 fits
walls/floors, not glazing.
- gas_boiler_upgrade / system_tune_up / system_tune_up_zoned: these are priced
off the heating rate sheet (Products()), with get() reading the catalogue only
for an id — so the overlay entry exists to satisfy that lookup (material_id
stays None, as with ASHP); the rate sheet remains authoritative.
Validated on a 12-property sample (incl. a secondary-glazing case and a
SAP-Schema-16.2 cert): 12/12 baseline rows + plans, 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
