Two reconciliations to make the modelling_e2e Lambda handler production-ready.
1. Price through the off-catalogue overlay, drop the workarounds
The handler priced through a plain ProductPostgresRepository and excluded
secondary_heating_removal / system_tune_up / system_tune_up_zoned to dodge
ProductNotFound (and a poisoning pgEnum DataError). Those measures are now
priced by catalogue_with_off_catalogue_overrides (already used by the e2e
runner and PostgresUnitOfWork), so the exclusions are removed and ALL measure
types are considered. This also fixes gas-boiler / single-glazed properties,
which Dan's handler never excluded and so still crashed (the standard
system_tune_up option is built unconditionally — the considered-measures
exclusion never actually gated it).
2. Broaden the EPC-Prediction cohort to nearby real postcodes (ADR-0031)
A property with no lodged EPC and no same-type comparable in its own postcode
(e.g. the only flat among houses) used to gate out and fail the subtask. The
gov EPC API cannot search by radius/outcode, so we resolve the real unit
postcodes physically nearest the target via postcodes.io (keyless; already a
trusted in-repo dependency) and walk them nearest-first until enough same-type
comparables surface. New PostcodesIoClient (transient-failure retry with
exponential backoff, soft-failing to the seed so broadening never breaks
prediction) and EpcComparablePropertiesRepository.candidates_near. Wired into
the handler and e2e runner; broadening is lazy (only on gate-out) and memoised
per (postcode, property_type).
Validated live: property 728476 (gas boiler) prices system_tune_up at GBP295;
property 718580 (lone flat in BR6 6BS) now predicts via nearby BR6 postcodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Corpus validation of the modelling_e2e photovoltaic_supply-as-list fix. Cert
6102-6227-8000-0083-2292 (RdSAP-20.0.0 semi, gas combi + 2× 1.14 kW PV arrays)
crashed from_rdsap_schema_20_0_0 on the measured-array list; the fix routes it
through the dict-tolerant _map_schema_21_pv. PV correctly credited: engine 61
(no PV) → 66 (+5). Built in Elmhurst (evidence: epc.json + summary + worksheet,
fabric+heating; the PV "New Technologies" Panel-details grid deferred): worksheet
55 = engine-on-Elmhurst-inputs 55 exactly → calculator faithful. The +6 engine-vs-
Elmhurst base-dwelling residual is the documented RdSAP-default gap (band-C cavity-
uninsulated suspended-floor semi). Pinned engine 66.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tests/orchestration/test_postcode_splitter_orchestrator.py imports
`from moto import mock_aws` (moto 5.x) but moto was absent, so the file
errored at collection. Pin moto[s3,sqs]==5.0.28 (S3+SQS are the only mocked
services); resolves cleanly against the boto3 1.35.44 and cryptography 43.0.3
pins. All 4 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@task_handler never built or passed cloud_logs_url, so every app using
it (incl. modelling_e2e) ran run_subtask with the None default and the
CloudWatch deep-link was never saved onto the SubTask. @subtask_handler
did this correctly.
Extract the URL builder into a shared utilities/aws_lambda/cloud_logs.py
(public cloudwatch_url()), use it from both handlers, and pass the URL
into run_subtask from @task_handler. Add regression tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Landlord Override can reference a building part the lodged or predicted EPC
never carried (e.g. an extension_1 override on a property whose EPC has only
main). apply_simulations indexed parts_by_id[identifier] unguarded, raising
KeyError and aborting the whole property's modelling. Now the orphaned part is
skipped. Recovers 14 of the 22 modelling_e2e failures in portfolio 796.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@task_handler never built or passed cloud_logs_url, so every app using
it (incl. modelling_e2e) ran run_subtask with the None default and the
CloudWatch deep-link was never saved onto the SubTask. @subtask_handler
did this correctly.
Extract the URL builder into a shared utilities/aws_lambda/cloud_logs.py
(public cloudwatch_url()), use it from both handlers, and pass the URL
into run_subtask from @task_handler. Add regression tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Corpus validation of the modelling_e2e built_form fix. Cert 8742-6624-9300-2780-4926
(SAP-Schema-16.0, ground-floor electric-storage-heater flat) omits built_form; the
mapper now derives it from dwelling_type. built_form is ML-only so the fix is
SAP-neutral: engine 66 = lodged 66 exactly. Built in Elmhurst (evidence: epc.json +
summary + worksheet): worksheet 54, engine-on-Elmhurst-inputs 53 ≈ 54 → calculator
faithful. The +12 engine-vs-Elmhurst is a build/input gap (cert size-1 small cylinder
unrepresentable in Elmhurst's Normal/110L-minimum entry → higher HW + reduced-field
16.0 defaults). Pinned engine 66.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the mapper-coverage gaps surfaced by the modelling_e2e prediction-cohort
failures (portfolio 796):
- built_form (SAP-16.0): derive from dwelling_type in _normalize_sap_schema_16_x
(Mid-terrace->4, End-terrace->3, Semi-detached->2, Detached->1; flats->modal 4).
ML-only field (SAP calc never reads it) so SAP- and gate-neutral. 5 flat certs
that omitted built_form now map.
- photovoltaic_supply as a measured-array LIST: routed all pre-21 RdSAP mappers
(17.0/17.1/18.0/19.0/20.0.0) through _map_schema_21_pv, whose list branch is now
dict-tolerant (_pv_array_field reads dict OR dataclass). They capture the PV
arrays like 21.0.x instead of raising "'list' object has no attribute
none_or_no_details" and sinking the whole cohort.
- windows-as-dict (16.x): handled in the normalizer (not just windows-as-list).
Genuinely-sparse certs (omit door_count/habitable/glazed_area) remain fail-loud;
the gate-regressing multiple_glazed_proportion default and the recursive
RdSAP-21.0.0 ADR-0028 alignment are left fail-loud + flagged for review (worklist).
+5 regression tests; component-accuracy gate 26/26; 0 new pyright errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chasing the space-heating demand gap on "simulated case 48" (main 691 + Unknown
meter + 903 dual immersion): our SAP 55 vs Elmhurst 57. Every §10a cost line
already matched to the penny; the residual was demand — our space-heating
energy 3849.8 kWh vs Elmhurst 3513.8 (+9.6%). Traced through the worksheet: our
ventilation heat loss (38) ran ~35.5 W/K vs Elmhurst 27.76 — we were adding 20
m3/h of intermittent extract fans (the Table 5 age-band default) on a dwelling
with a decentralised mechanical extract (dMEV) system that lodges 0 fans.
SAP 10.2 §2 (PDF p.13): a whole-house mechanical EXTRACT system provides
extraction via the (23a) 0.5 system air-change rate; the lodged intermittent
extract-fan count (7a) is then explicit — a lodged 0 means 0 (the dMEV is the
ventilation), NOT "unknown". The Table 5 default is an unknown-fallback for
NATURALLY ventilated dwellings only, so it must not be substituted here.
Fix: for EXTRACT_OR_PIV_OUTSIDE, take vc.intermittent_fans as-is (no age-band
default). Worksheet-proven on two dMEV builds of cert 000565: "case 48" lodges
(7a)=0 -> our SAP 55 -> 57 EXACT; the original 000565 fixture lodges (7a)=2 and
keeps 2 (its e2e pins are unchanged). An earlier draft that forced fans=0 broke
000565 (which legitimately has 2) — corrected to "lodged as-is".
within-0.5 72.5% -> 72.6%, MAE 0.789 -> 0.788; CO2/PE unchanged. The fix also
reduces a systematic under-rating bias in the 21-cert dMEV cohort (median dSAP
-0.22 -> -0.08). Scoped to EXTRACT_OR_PIV_OUTSIDE; balanced MVHR/MV kinds left
untouched pending their own worksheet. SAP-schema regression
test_18_0_0 pin 80 -> 81 (closer to its lodged 84, same cause). Spec-pinned in
test_cert_to_inputs (dMEV-lodged-0 vs natural-default). pyright not installed
in this container -- strict type gate not run locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- run_modelling_e2e --from-db re-models from already-persisted inputs (reads
each Property's Effective EPC + planning protections + solar from the DB) and
skips every live fetcher — zero gov-API calls. With --persist it re-writes the
Plan and, for lodged-EPC Properties, the Baseline. Self-contained loop; the
live-fetch path is untouched. Makes local re-runs instant and avoids tripping
the gov API's per-IP rate limit (6000 req / 5 min) during iteration.
- EpcClientService.REQUEST_TIMEOUT 10s -> 30s: a cold per-UPRN search can exceed
10s and the old timeout turned it into a timeout-then-retry; 30s rides it out.
Note: an open perf question remains — modelling is fast in isolation (<0.5s/
property) but a long-lived --persist run shows ~1 min/property; suspected in the
persist path (plan.save / baseline) or connection handling, NOT the API. Left
mid-diagnosis for handover.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the follow-up from 22fe4f41. RdSAP 10 §12's third Unknown-meter
exception bullet — "main heating is ground source or water source heat pump" —
was unimplemented. Add `_GROUND_OR_WATER_SOURCE_HEAT_PUMP_CODES` (Table 4a,
SAP 10.2 PDF p.176-177: ground 211/215/221/225 + warm-air 521/525; water
213/216/223/226 + warm-air 523/526) to the Unknown-meter off-peak triggers;
once dual, Rules 1-4 (Rule 3) resolve it to 10-hour.
AIR-source heat pumps (214/217/224/227, 524/527) are deliberately EXCLUDED —
the spec names only ground/water source. Verified the only Unknown-meter heat
pumps in the corpus are "3/10 Bedford House" (main 214 = AIR source), which
correctly KEEP STANDARD. 0 corpus certs carry a GSHP/WSHP on an Unknown meter,
so this is a spec-completeness forward guard (gauge unchanged 72.5% / 0.789),
same family as the Scotland-J wall / rafters-M roof 0-impact spec fixes.
Coverage gap noted in-code: a database-index heat pump without a 211/213-style
SAP code can't have its source type read from the code alone (rare).
Spec-pinned (test_unknown_meter_ground_or_water_source_heat_pump_triggers_off_
peak). Gates green: corpus 72.5%/0.789, batch worksheet 0 raised/0 diverge,
suite 2989 passed (2 known pre-existing fails). pyright not installed locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MAE improved 0.793 -> 0.789 via the Unknown-meter + dual-electric-immersion
off-peak trigger (commit 22fe4f41). Ratchet the ceiling so the gain can't
silently regress. within-0.5 unchanged (72.5%).
pyright not installed in this container — strict type gate not run locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Supersedes the previous "verified non-fix" doc (3548f1f3): the spec DOES make
this a fix — Khalim was right that the Unknown-meter branch is driven by the
heating/water system, not a blanket STANDARD.
RdSAP 10 §12 (PDF p.62): "If the electricity meter is unknown, treat as single
meter EXCEPT where main heating OR WATER HEATING are intended to run off an
off-peak tariff (per systems listed in the text box above) ... If that results
in a dual meter, assign tariff per rules 1 to 4." The text-box off-peak systems
include DUAL ELECTRIC IMMERSION. Our `rdsap_tariff_for_cert` only triggered the
Unknown→off-peak exception on a storage/CPSU MAIN — it ignored the
dual-electric-immersion WATER-heating trigger, so an Unknown-meter dwelling
with a non-storage main (e.g. room heaters) + dual immersion was billed
STANDARD (13.19p flat) when it should be dual → Rules 1-4 on the main.
Fix: thread `water_is_off_peak_dual_immersion` (whc 903 + immersion lodged dual
via `_immersion_is_single is False`) into the Unknown-meter branch; when any
text-box trigger is present, resolve via the same Rules 1-4 dispatch (room
heaters → Rule 3 → 10-hour). Single-immersion / instantaneous (whc 909) certs
correctly stay STANDARD (no text-box system).
Worksheet-validated on "simulated case 48" (main 691 + Unknown meter + 903 dual
immersion): Elmhurst 10-Hour Off Peak, SAP 57; ours 45 → 55 (7-hour gives 45,
confirming 10-hour). Flips exactly ONE corpus cert — Apartment 241 (the genuine
-5.38 under-rater, main 691 + dual immersion) -5.38 → -1.05; every other
Unknown+dual-immersion cert already has a storage main (Rule 2). Corpus
within-0.5 holds 72.5%, MAE 0.793 → 0.789 (improved). CO2/PE unchanged.
GSHP/WSHP-main trigger (the other §12 Unknown exception bullet) is a separate
follow-up. Gates green: corpus 72.5%/0.789, batch worksheet 0 raised/0 diverge,
000565 e2e 11/11, suite 2987 passed (2 known pre-existing fails). pyright not
installed in this container — strict type gate not run locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Khalim's hand-built "simulated case 48" Elmhurst worksheet (main 691 room
heaters + Unknown meter + 903 dual electric immersion + cylinder) proves
Elmhurst resolves an Unknown meter + Rule-3 room-heater main to 10-Hour Off
Peak — Elmhurst SAP 57; ours (STANDARD) 45; routing Unknown+Rule3 to off-peak
(10h) gives 55, 7h gives 45 (so Elmhurst uses 10-hour, RdSAP 10 §12 Rule 3,
PDF p.62). So the prior "Rule 3 is not off-peak evidence" comment was wrong
about Elmhurst's behaviour.
But adopting it REGRESSES the lodged-register corpus 72.5%->71.8% / MAE
0.793->0.827: of 11 Unknown+Rule-3 corpus certs only Apartment 241 improves
(-5.38->-1.05); the other 10 overshoot +2.7..+9.1 (Flat 2 +9.11). The
register's meter_type=3 certs were lodged with STANDARD-tariff costing — the
gov-API "Unknown" is lossy and does not mean off-peak. North star is
reproducing the lodged register, not Elmhurst's deliberate-Unknown worksheet,
so KEEP STANDARD (same "Elmhurst != noisy register" family as
roof-windows/shutters).
Comment-only; no behaviour change (corpus gauge unchanged 72.5% / 0.793).
pyright not installed in this container — strict type gate not run locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
profile_corpus_error.py: print median signed error alongside the mean in each
feature bucket. The mean is dragged by fat-tail register anomalies (e.g. electric
room heaters mean -1.09 but median -0.01) — median is the outlier-resistant lens
for finding TRUE systematic slices, so hunt by |median|, not |signed|.
heat_transmission.py: document why permanent-shutter R is deliberately NOT applied
(Elmhurst uses R=0.04 curtains on every window incl. insulated shutters, proven on
case 46; API-path trial worsens MAE). Comment-only.
pyright not installed locally — strict type gate not run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two second-main fuel errors mis-cost a dual-main dwelling whose two main systems
burn different fuels (SAP 10.2 §10a worksheet (213) bills main 2 at its own fuel):
1. Off-peak/legacy scalar cost path (calculator.py + cert_to_inputs.py): main 2's
kWh was priced at main 1's `space_heating_fuel_cost_gbp_per_kwh` scalar. Split
main 1 vs main 2 and price main 2 at its OWN rate via the new
`_main_2_space_heating_fuel_cost_gbp_per_kwh` (+ CalculatorInputs field).
Scoped to a NON-electric second main (wood/oil/coal) — an electric second
main keeps main 1's scalar (its off-peak Table 12a split is the deferred §10a
slice; per-system splitting it regresses the off-peak electric cohort, certs
13 Parkers Hill / 34 Dunley Road). 0 corpus impact (no corpus cert has a
non-electric second main on an off-peak meter).
2. Elmhurst Summary mapper (mapper.py): when §14.1 omits the Fuel Type cell, a
fuel-fired second main (room-heater SAP code) inherited main 1's fuel. Derive
it from the SAP code's Table 4a category (solid 631-636 -> house coal, gas ->
mains gas, liquid -> oil) before the main-1 inherit, mirroring
`_elmhurst_secondary_fuel_from_sap_code` (same modal sub-fuel caveat). Boiler
codes (<601) still inherit main 1 (case 6 oil rads+UFH).
simulated case 47 (electric room heaters + solid room heaters 633): our SAP
37.81 -> 55.09 vs Elmhurst current 57 (residual is the wood-vs-coal sub-fuel the
Summary export does not carry). Corpus unchanged 72.5% / MAE 0.793; batch 0
raised / 0 diverge; 000565 e2e green. (mapper.py also carries an earlier,
behaviour-free roof-window doc comment.) Spec-cited unit pins added (AAA).
pyright not installed locally — strict type gate not run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
