Fitting sealed glazing units changes two things beyond the pane's U/g
that the cascade reads, which the overlay didn't model — leaving the
double/secondary before→after pins ~0.7 SAP short (xfail):
1. Draught-proofing (RdSAP 10 §8.1). Sealed units draught-proof the panes
they replace, re-lodging the dwelling-level `percent_draughtproofed`
(cert 001431: 84 → 100). The §2 cascade reads that dwelling-level
value, so the overlay now carries it. `_recompute_percent_draughtproofed`
anchors on the lodged before-% — `after = round((round(before%/100 × N)
+ flips) / N × 100)`, N = openable windows (vertical + roof) + doors,
flips = upgraded panes that were not draught-proofed — so it's robust
to incomplete window extraction (unchanged openings are already in the
aggregate). ~0.3 SAP.
2. Frame factor (§6 solar gains). A replacement unit re-lodges its own
FF=0.70, overriding the pane it replaced — the two "single glazing,
known data" panes lodge FF 1.00 / 0.50 (one is 6.6 m²), so leaving them
unchanged understated solar gains by ~+150 kWh space heating. `WindowOverlay`
now carries `frame_factor`, written flat onto the window. ~0.4 SAP.
Wiring: `EpcSimulation.percent_draughtproofed` + `WindowOverlay.frame_factor`
new fields; `apply_simulations` / `_fold_window` write them; the glazing
generator computes both from the upgraded set and cert 001431's after.
Un-xfails `test_{double,secondary}_glazing_overlay_reproduces_the_relodged_after`
— both now pin SAP/CO2/PE to the relodged after within tolerance. Updates
the two `test_glazing_recommendation` overlay expectations for the new
`frame_factor`. 96 modelling tests pass; zero new pyright errors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Skipped cohort certs were previously surfaced only as outputs.result on a
completed subtask, so they were easy to miss. Treat them as a failure too:
once the batch has run to completion (so every modellable property is already
written to DB), raise if there were any per-property errors OR any skipped
certs. The run gets flagged for debugging without discarding the work done.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A predicted Property (no lodged EPC) got a Plan but nothing else: the synthesised
EPC was never written to epc_property, and Baseline Performance was skipped — so
property 729529 (portfolio 796 / scenario 1268), predicted from its DA16 1QZ
cohort, was "missed" with no predicted-EPC row and no baseline row.
Persist the synthesised EPC in the predicted slot (uow.epc.save(..., source=
"predicted"), ADR-0031) inside the Plan UoW, then run the Baseline orchestrator
for predicted Properties too — it re-hydrates the predicted EPC and establishes
the baseline from it. The earlier "lodged only" guard is dropped: by the write
block the Property always has a persisted EPC (lodged or predicted); one that
could be neither fetched nor predicted raised earlier.
Verified against the DB by invoking the real handler for 729529: predicted
epc_property rows 0->1 and property_baseline_performance rows 0->1. Baseline on
the predicted picture builds cleanly (RHI present, reason pre_sap10). Tests
updated: prediction + broadening paths now assert the predicted-slot epc.save and
the baseline run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The handler wrote epc/spatial/solar/plan and marked the property modelled, but
never established its Baseline Performance — so no row was created in
property_baseline_performance for any property modelled through the Lambda
(noticed on portfolio 796 / scenario 1268 / property 727218, a lodged property).
Mirror the e2e runner: after the plan UoW commits (so the EPC is persisted for
the orchestrator to re-hydrate), run PropertyBaselineOrchestrator for lodged
properties. Predicted properties have no lodged figures and no persisted EPC, so
they are skipped — consistent with the e2e runner and the ara_first_run Baseline
stage.
Verified 727218's baseline pipeline builds end-to-end in-memory (lodged_performance
→ CalculatorRebaseliner → bill → PropertyBaselinePerformance, reason pre_sap10).
Tests: lodged path asserts the orchestrator runs once; prediction path asserts it
does not.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Landlord Override's building_part is a positional index (0=main, 1=extension
1…, ADR-0004), but the gov-API EPC can label that slot differently (e.g. lodge
the 2nd part as 'other', not 'extension_1'). The previous fix skipped such
orphaned overrides, silently discarding the landlord's correction. Now the
override falls back onto the EPC's part at that position (via _resolve_part), so
the correction lands; only a position the EPC models no part at is skipped
(no geometry to model a wholly-absent part). Replaces the skip-only behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes out the cohort-broadening work with its decision record and consolidates
the retry plumbing.
ADR-0034 documents broadening the EPC-Prediction cohort to the real unit
postcodes nearest the target (via postcodes.io) when its own postcode holds no
same-type comparable — extending ADR-0031 decision 5. Records why postcodes.io
was chosen over council[] (whole-LA, no property_type in rows), a bulk Code-Point
Open / ONSPD dataset, and the OS Places radius API, and the lazy / nearest-first
early-stop / soft-fail policy. Broadening-specific docstrings now cite 0034.
Retry consolidation: extract the EPC client's call_with_retry into a shared
infrastructure/http_retry.py keyed off a generic TransientHttpError marker, so
the mechanism (exponential backoff, Retry-After) is shared while each client
keeps its own transient policy. EpcRateLimitError now subclasses TransientHttpError
(still an EpcApiError); PostcodesIoClient routes through the same helper, raising
TransientHttpError on 429/5xx and soft-failing to the seed once exhausted (the EPC
client propagates instead). Direct tests for the shared helper; EPC + postcodes.io
suites repointed at the shared sleep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reconciliations to make the modelling_e2e Lambda handler production-ready.
1. Price through the off-catalogue overlay, drop the workarounds
The handler priced through a plain ProductPostgresRepository and excluded
secondary_heating_removal / system_tune_up / system_tune_up_zoned to dodge
ProductNotFound (and a poisoning pgEnum DataError). Those measures are now
priced by catalogue_with_off_catalogue_overrides (already used by the e2e
runner and PostgresUnitOfWork), so the exclusions are removed and ALL measure
types are considered. This also fixes gas-boiler / single-glazed properties,
which Dan's handler never excluded and so still crashed (the standard
system_tune_up option is built unconditionally — the considered-measures
exclusion never actually gated it).
2. Broaden the EPC-Prediction cohort to nearby real postcodes (ADR-0031)
A property with no lodged EPC and no same-type comparable in its own postcode
(e.g. the only flat among houses) used to gate out and fail the subtask. The
gov EPC API cannot search by radius/outcode, so we resolve the real unit
postcodes physically nearest the target via postcodes.io (keyless; already a
trusted in-repo dependency) and walk them nearest-first until enough same-type
comparables surface. New PostcodesIoClient (transient-failure retry with
exponential backoff, soft-failing to the seed so broadening never breaks
prediction) and EpcComparablePropertiesRepository.candidates_near. Wired into
the handler and e2e runner; broadening is lazy (only on gate-out) and memoised
per (postcode, property_type).
Validated live: property 728476 (gas boiler) prices system_tune_up at GBP295;
property 718580 (lone flat in BR6 6BS) now predicts via nearby BR6 postcodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Corpus validation of the modelling_e2e photovoltaic_supply-as-list fix. Cert
6102-6227-8000-0083-2292 (RdSAP-20.0.0 semi, gas combi + 2× 1.14 kW PV arrays)
crashed from_rdsap_schema_20_0_0 on the measured-array list; the fix routes it
through the dict-tolerant _map_schema_21_pv. PV correctly credited: engine 61
(no PV) → 66 (+5). Built in Elmhurst (evidence: epc.json + summary + worksheet,
fabric+heating; the PV "New Technologies" Panel-details grid deferred): worksheet
55 = engine-on-Elmhurst-inputs 55 exactly → calculator faithful. The +6 engine-vs-
Elmhurst base-dwelling residual is the documented RdSAP-default gap (band-C cavity-
uninsulated suspended-floor semi). Pinned engine 66.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@task_handler never built or passed cloud_logs_url, so every app using
it (incl. modelling_e2e) ran run_subtask with the None default and the
CloudWatch deep-link was never saved onto the SubTask. @subtask_handler
did this correctly.
Extract the URL builder into a shared utilities/aws_lambda/cloud_logs.py
(public cloudwatch_url()), use it from both handlers, and pass the URL
into run_subtask from @task_handler. Add regression tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@task_handler never built or passed cloud_logs_url, so every app using
it (incl. modelling_e2e) ran run_subtask with the None default and the
CloudWatch deep-link was never saved onto the SubTask. @subtask_handler
did this correctly.
Extract the URL builder into a shared utilities/aws_lambda/cloud_logs.py
(public cloudwatch_url()), use it from both handlers, and pass the URL
into run_subtask from @task_handler. Add regression tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Corpus validation of the modelling_e2e built_form fix. Cert 8742-6624-9300-2780-4926
(SAP-Schema-16.0, ground-floor electric-storage-heater flat) omits built_form; the
mapper now derives it from dwelling_type. built_form is ML-only so the fix is
SAP-neutral: engine 66 = lodged 66 exactly. Built in Elmhurst (evidence: epc.json +
summary + worksheet): worksheet 54, engine-on-Elmhurst-inputs 53 ≈ 54 → calculator
faithful. The +12 engine-vs-Elmhurst is a build/input gap (cert size-1 small cylinder
unrepresentable in Elmhurst's Normal/110L-minimum entry → higher HW + reduced-field
16.0 defaults). Pinned engine 66.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make run_modelling_e2e the single script that does everything for a portfolio,
so the 291-property run needs one invocation with per-property recovery (no
all-or-nothing chunking):
- On --persist, a lodged-EPC Property now also gets its Baseline Performance
row written via PropertyBaselineOrchestrator (per Property, so one bad cert
does not abort the batch). Predicted (EPC-less) Properties have no lodged
figures, so they get a Plan but no baseline row.
- The run CSV gains api_sap (register) vs baseline_sap (calculator) + sap_delta,
so calculator-vs-API divergence is reviewable per property.
Fill the off-catalogue overlay for the measures the live material catalogue
cannot price, so they stop crashing the run:
- double_glazing (£550/window) and secondary_glazing (£400/window): priced
per window (the generator multiplies by single-glazed window count, matching
the legacy window_glazing). Grounded in 2025/26 UK installed costs; per-window
is the right unit for windows (fixed per-unit install dominates) — per-m2 fits
walls/floors, not glazing.
- gas_boiler_upgrade / system_tune_up / system_tune_up_zoned: these are priced
off the heating rate sheet (Products()), with get() reading the catalogue only
for an id — so the overlay entry exists to satisfy that lookup (material_id
stays None, as with ASHP); the rate sheet remains authoritative.
Validated on a 12-property sample (incl. a secondary-glazing case and a
SAP-Schema-16.2 cert): 12/12 baseline rows + plans, 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Table 5 reads "Number of extract fans if known; if number is unknown:
[age-band default]" — the default is an UNKNOWN-fallback, NOT a floor. The
cascade applied `max(lodged, table_5_default)`, flooring a genuinely-lodged
count up to the age-band minimum: e.g. an age H-M dwelling lodging 2 extract
fans was billed at the 6-8-room default of 3, over-counting ventilation line
(8) and the heat-loss coefficient. Fixed to `lodged if lodged > 0 else
default` (a lodged 0 is the Elmhurst/RdSAP "unknown" form → default; any
positive count is taken literally).
Surfaced by Khalim's Elmhurst stress worksheet (simulated case 46): this was
its last ventilation residual — our Jan effective ACH 9.14 -> 9.0748 (exact
match to the accredited worksheet), SAP 29 -> 30 = Elmhurst, cost £1496 vs
£1493. Corpus IMPROVED: within-0.5 71.6% -> 72.5%, MAE 0.819 -> 0.815 (the
max-flooring over-counted ventilation on every cert lodging fans below its
age default). Floor ratcheted 0.71 -> 0.72. pyright not installed locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the main heating system does not heat every habitable room (heated rooms
< habitable rooms), SAP 10.2 Appendix A.2.2 assumes the unheated rooms are
served by a portable-electric secondary heater, so the Table 11 secondary
fraction (0.10 for a boiler main) must be costed at the electricity tariff —
even when the cert lodges no explicit secondary system.
`_secondary_fraction` previously returned 0 unless a secondary was lodged or
the main was a forced-secondary electric-storage code, dropping the assumed
secondary and billing 100% of space heat to the (cheaper) main fuel — an
over-rate. Added an `unheated_habitable_rooms` trigger plus
`_has_unheated_habitable_rooms(epc)`, which prefers the lodged
`any_unheated_rooms` flag and guards the gov-API `heated_rooms_count == 0`
"not provided" sentinel. The secondary fuel/efficiency cascade already
defaults to portable electric (code 693) when no secondary code is lodged.
Worksheet-validated on simulated case 46 (heated 4 < habitable 7, no lodged
secondary): the assumed 10% electric secondary (2289 kWh, ~£260) lifted our
SAP 39 -> 29.35 vs accredited Elmhurst 30 (cost £1502 vs £1493, within 0.6%).
Corpus UNCHANGED (71.6% / MAE 0.819): all 17 corpus certs with heated <
habitable already lodge an explicit secondary description, so the gov-API
path was already costing it; this only adds the assumed secondary where none
is lodged (Elmhurst / reduced-field path). pyright not installed locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The full RdSAP-Schema-21 label is "Double between 2002 and 2021" (double
glazing installed 2002-2021, SAP 10.2 Table 24 code 3). When the Elmhurst
Summary PDF wraps the trailing "and 2021" into an adjacent table cell that
the extractor joins away, the surfaced label truncates to "Double between
2002" — the same artifact already handled for "Triple post or during".
`_elmhurst_glazing_type_code` raised UnmappedElmhurstLabel on it, blocking
the whole Summary (surfaced on the simulated-case-46 multi-attribute
worksheet). Added the truncated form as a code-3 alias.
pyright not installed in this container.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The corrected stone branch (age-E-only cap, §3.5 Table-3 unknown-thickness
default, Scotland band-J override) improved the corpus gauge to SAP MAE 0.819
and PE MAE 3.6 kWh/m2/yr. Lock the gains in and record the corrected
mechanism in the provenance log; within-0.5 (71.6%) and CO2 (0.08) unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two-pass org_ref-matched builder for property_overrides (classify via ChatGPT
into the landlord ledger, validate+apply user edits, write idempotently);
ephemeral-Postgres smoke proves the one-property chain without creds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Caught live writing property_overrides on portfolio 796: the Python
override_component SAEnum lagged the DB enum, so reading a new-component row
back threw LookupError. Guard it with a consistency test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The FE-owned `material.type` pgEnum cannot carry `secondary_heating_removal`,
so pricing it through the DB catalogue raises a DataError that poisons the
session — the modelling pipeline crashed on any property with a lodged
secondary heater unless the measure was excluded on the Scenario.
Realise the `ProductRepository` docstring's intent (DB catalogue today, a JSON
file for costs the ETL does not yet supply, behind the same port): add a
`CompositeProductRepository` that resolves an override source first, then the
catalogue. Checking the override first keeps that Measure Type away from the DB
entirely; every other type misses the override and falls through unchanged.
- off_catalogue_costs.json prices it at £270 flat per-dwelling — the legacy
`Costs.heater_removal` ported to the new flat model (ADR-0028):
(£25 + £200 baseline) x 1.2 VAT, for the single fixed secondary a cert lodges.
Contingency (0.25) is joined from config, not the file.
- Wire the composite into PostgresUnitOfWork.product and run_modelling_e2e, so
the first-run pipeline and the local runner both honour the overlay.
- Integration test: drop the unrealistic seeded secondary_heating_removal DB
rows (the pgEnum can't hold the type) and assert it is JSON-sourced
(material_id is None, cost £270) end-to-end through a real Unit of Work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An uninsulated stone wall of lodged thickness, age bands A-E, is billed by
the RdSAP 10 §5.6 Table-12 formula on its measured thickness — sandstone /
limestone U = 54.876·W^-0.561, granite / whinstone U = 45.315·W^-0.513. Two
bugs suppressed it:
1. CAP: the as-built branch capped the formula result at the Table-6
typical-thickness default (`if u0 >= 1.7: return 1.7`). But the formula
only dips below 1.7 past ~488 mm (sandstone) / ~640 mm (granite), so the
cap nullified §5.6 for essentially every real-thickness stone wall,
under-counting fabric loss and over-rating. A measured 400 mm sandstone
age-B wall is 1.90 (Elmhurst-confirmed), not 1.70.
2. NO INSULATION-STATE GATE: the branch fired for any stone wall with a
lodged thickness, including ones whose insulation is "Unknown". RdSAP
treats an unknown-insulation wall via the Table-6 typical-thickness
default, NOT as bare stone of the lodged thickness — so a 50 mm granite
wall with Unknown insulation must read 1.70 (worksheet), not the formula's
6.09. Gated on `wall_insulation_type is not None`.
Fixes the 2 long-standing stone-U unit tests (granite 120 mm → 3.8871,
sandstone 120 mm → 3.7408 — they were correct red tests, not "known fails").
Corpus within-0.5 70.3% -> 71.6% (MAE 0.833 -> 0.822); ratcheted floors to
0.71 / 0.83. Worksheet fixture 000565 (granite 50 mm Unknown → 1.70) still
passes via the insulation gate. The two stone groups (sandstone/limestone vs
granite/whinstone) keep their distinct §5.6 coefficients.
pyright not installed in this codespace (strict gate not run locally).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
