Review follow-up (Khalim): the first pass made far more optional than needed —
notably the whole SapBuildingPart block — and a buggy 21.0.0↔21.0.1 diff also
MISSED open_chimneys_count / cfl_/led_fixed_lighting_bulbs_count / suggested_
improvements, so the original change actually mapped only 3 of the 33 skipped
certs (the rest still failed on open_chimneys_count).
Re-derived the exact set empirically from all 33 skipped cohort certs:
widen only fields that are (a) required in 21.0.0, (b) already optional in
21.0.1, AND (c) genuinely omitted by ≥1 of those certs. Result:
- KEEP optional: the 4 SapWindow refinements, the top-level vent/lighting/
door/pressure-test block (incl. the 3 previously-missed fields), 2
SapEnergySource fields, Addendum.addendum_numbers, PhotovoltaicSupply.
none_or_no_details, and exactly ONE building-part field
(SapBuildingPart.roof_insulation_thickness — omitted by 7 certs).
- REVERT to required: the other 12 SapBuildingPart fields (construction_age_
band, wall_construction, …), MainHeatingDetail.emitter_temperature,
PvBatteries.pv_battery, ShowerOutlets.shower_outlet — none of the 33 certs
omit these, so they stay strict.
Mapper: coalesce the count fields (wet_rooms_count, open_chimneys_count,
cfl_/led_fixed_lighting_bulbs_count) to 0 like every other mapper, so the now-
optional values can't reach a NOT-NULL column (also drops 4 pyright ignores).
Now maps 32/33 (up from 3); the last cert hits a pre-existing pv_batteries-
shape AttributeError and degrades via the ADR-0031 skip path. pyright net
unchanged (43, no new errors); regression test rewritten to the real omitted set.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fitting sealed glazing units changes two things beyond the pane's U/g
that the cascade reads, which the overlay didn't model — leaving the
double/secondary before→after pins ~0.7 SAP short (xfail):
1. Draught-proofing (RdSAP 10 §8.1). Sealed units draught-proof the panes
they replace, re-lodging the dwelling-level `percent_draughtproofed`
(cert 001431: 84 → 100). The §2 cascade reads that dwelling-level
value, so the overlay now carries it. `_recompute_percent_draughtproofed`
anchors on the lodged before-% — `after = round((round(before%/100 × N)
+ flips) / N × 100)`, N = openable windows (vertical + roof) + doors,
flips = upgraded panes that were not draught-proofed — so it's robust
to incomplete window extraction (unchanged openings are already in the
aggregate). ~0.3 SAP.
2. Frame factor (§6 solar gains). A replacement unit re-lodges its own
FF=0.70, overriding the pane it replaced — the two "single glazing,
known data" panes lodge FF 1.00 / 0.50 (one is 6.6 m²), so leaving them
unchanged understated solar gains by ~+150 kWh space heating. `WindowOverlay`
now carries `frame_factor`, written flat onto the window. ~0.4 SAP.
Wiring: `EpcSimulation.percent_draughtproofed` + `WindowOverlay.frame_factor`
new fields; `apply_simulations` / `_fold_window` write them; the glazing
generator computes both from the upgraded set and cert 001431's after.
Un-xfails `test_{double,secondary}_glazing_overlay_reproduces_the_relodged_after`
— both now pin SAP/CO2/PE to the relodged after within tolerance. Updates
the two `test_glazing_recommendation` overlay expectations for the new
`frame_factor`. 96 modelling tests pass; zero new pyright errors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment out the remaining workflows to cut GitHub Actions usage, per request:
- integration_tests.yml — rebaselining integration suite (PRs to main)
- deploy_fastapi_backend.yml — FastAPI backend deploy (push to dev/prod);
deploys must be run manually via `sls deploy` while disabled
- protect_releases.yml — main→dev PR-source guardrail
Fully commented (not deleted) so each restores by uncommenting.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment out the Docker-based unit-test workflow — it was consuming too many
GitHub Actions minutes. Fully commented (rather than deleted) so it can be
restored by uncommenting.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
37 modelling_e2e properties failed on the 2026-06-23 run with
`NotNullViolation: null value in column "wet_rooms_count" of relation
"epc_property"`.
Root cause: 21.0.1 lodges `wet_rooms_count` as Optional, and
`from_rdsap_schema_21_0_1` passed it straight through
(`wet_rooms_count=schema.wet_rooms_count`). A cert omitting it mapped to
`EpcPropertyData.wet_rooms_count=None`. When a predicted EPC (which deep-copies
a comparable template's EpcPropertyData) inherited that None and was persisted,
it violated the `epc_property.wet_rooms_count` NOT-NULL column — and the calc's
`wet_rooms_count > 0` check would also raise `TypeError` on None.
Fix: coalesce to 0, matching every other mapper (RdSAP "not lodged" → the
calc's minimum 1 wet room). Regression test added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The modelling_e2e cohort skipped ~35 real RdSAP-Schema-21.0.0 certs with
"SapWindow: missing required field 'frame_factor'" (and a cascade of
wet_rooms_count, the mechanical-vent duct block, wind_turbine_details, &c.).
These are complete certs — the `RdSapSchema21_0_0` dataclass was simply
modelled stricter than `RdSapSchema21_0_1`, which already treats every one of
these fields as optional. This aligns 21.0.0's optionality with 21.0.1 (the
proven path) so the certs map into the prediction donor pool instead of being
dropped:
- SapWindow: pvc_frame / glazing_gap / frame_factor / window_transmission_details
→ Optional (calc already falls back: Table 6c frame factor, SAP10 glazing
lookup, type-only U).
- 20 nested fields across Addendum / MainHeatingDetail / PhotovoltaicSupply /
PvBatteries / SapBuildingPart / SapEnergySource → Optional, mirroring 21.0.1
verbatim (classes made kw_only so defaults sit in place; parsed only via
from_dict, keyword construction).
- from_rdsap_schema_21_0_0: guard the window_transmission_details dereference
for None (mirrors the 21.0.1 `_api_sap_window` guard).
Existing 21.0.0 certs are unaffected (relaxing optionality does not change
parsing of certs that carry the fields); mapper-corpus + accuracy gates green.
Verified end-to-end against real cohort cert 2205-3036-3484-0400-5718:
maps + calculates (SAP 68). Regression test added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Skipped cohort certs were previously surfaced only as outputs.result on a
completed subtask, so they were easy to miss. Treat them as a failure too:
once the batch has run to completion (so every modellable property is already
written to DB), raise if there were any per-property errors OR any skipped
certs. The run gets flagged for debugging without discarding the work done.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a property failed, the handler recorded only its bare property_id and
raised RuntimeError("failed property_ids: [...]"). That string is what
SubTask.fail persists into the subtask outputs.error column, so a failed run
told you which property failed but never why — forcing a CloudWatch lookup.
The per-property catch now captures property_id, uprn, error_type, and the
error message, and the raised RuntimeError embeds those as JSON so the subtask
outputs column is parseable directly. query_failed_modelling_e2e.py reads that
outputs.error into a new Error column in its report.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A predicted Property (no lodged EPC) got a Plan but nothing else: the synthesised
EPC was never written to epc_property, and Baseline Performance was skipped — so
property 729529 (portfolio 796 / scenario 1268), predicted from its DA16 1QZ
cohort, was "missed" with no predicted-EPC row and no baseline row.
Persist the synthesised EPC in the predicted slot (uow.epc.save(..., source=
"predicted"), ADR-0031) inside the Plan UoW, then run the Baseline orchestrator
for predicted Properties too — it re-hydrates the predicted EPC and establishes
the baseline from it. The earlier "lodged only" guard is dropped: by the write
block the Property always has a persisted EPC (lodged or predicted); one that
could be neither fetched nor predicted raised earlier.
Verified against the DB by invoking the real handler for 729529: predicted
epc_property rows 0->1 and property_baseline_performance rows 0->1. Baseline on
the predicted picture builds cleanly (RHI present, reason pre_sap10). Tests
updated: prediction + broadening paths now assert the predicted-slot epc.save and
the baseline run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The handler wrote epc/spatial/solar/plan and marked the property modelled, but
never established its Baseline Performance — so no row was created in
property_baseline_performance for any property modelled through the Lambda
(noticed on portfolio 796 / scenario 1268 / property 727218, a lodged property).
Mirror the e2e runner: after the plan UoW commits (so the EPC is persisted for
the orchestrator to re-hydrate), run PropertyBaselineOrchestrator for lodged
properties. Predicted properties have no lodged figures and no persisted EPC, so
they are skipped — consistent with the e2e runner and the ara_first_run Baseline
stage.
Verified 727218's baseline pipeline builds end-to-end in-memory (lodged_performance
→ CalculatorRebaseliner → bill → PropertyBaselinePerformance, reason pre_sap10).
Tests: lodged path asserts the orchestrator runs once; prediction path asserts it
does not.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Landlord Override's building_part is a positional index (0=main, 1=extension
1…, ADR-0004), but the gov-API EPC can label that slot differently (e.g. lodge
the 2nd part as 'other', not 'extension_1'). The previous fix skipped such
orphaned overrides, silently discarding the landlord's correction. Now the
override falls back onto the EPC's part at that position (via _resolve_part), so
the correction lands; only a position the EPC models no part at is skipped
(no geometry to model a wholly-absent part). Replaces the skip-only behaviour.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes out the cohort-broadening work with its decision record and consolidates
the retry plumbing.
ADR-0034 documents broadening the EPC-Prediction cohort to the real unit
postcodes nearest the target (via postcodes.io) when its own postcode holds no
same-type comparable — extending ADR-0031 decision 5. Records why postcodes.io
was chosen over council[] (whole-LA, no property_type in rows), a bulk Code-Point
Open / ONSPD dataset, and the OS Places radius API, and the lazy / nearest-first
early-stop / soft-fail policy. Broadening-specific docstrings now cite 0034.
Retry consolidation: extract the EPC client's call_with_retry into a shared
infrastructure/http_retry.py keyed off a generic TransientHttpError marker, so
the mechanism (exponential backoff, Retry-After) is shared while each client
keeps its own transient policy. EpcRateLimitError now subclasses TransientHttpError
(still an EpcApiError); PostcodesIoClient routes through the same helper, raising
TransientHttpError on 429/5xx and soft-failing to the seed once exhausted (the EPC
client propagates instead). Direct tests for the shared helper; EPC + postcodes.io
suites repointed at the shared sleep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reconciliations to make the modelling_e2e Lambda handler production-ready.
1. Price through the off-catalogue overlay, drop the workarounds
The handler priced through a plain ProductPostgresRepository and excluded
secondary_heating_removal / system_tune_up / system_tune_up_zoned to dodge
ProductNotFound (and a poisoning pgEnum DataError). Those measures are now
priced by catalogue_with_off_catalogue_overrides (already used by the e2e
runner and PostgresUnitOfWork), so the exclusions are removed and ALL measure
types are considered. This also fixes gas-boiler / single-glazed properties,
which Dan's handler never excluded and so still crashed (the standard
system_tune_up option is built unconditionally — the considered-measures
exclusion never actually gated it).
2. Broaden the EPC-Prediction cohort to nearby real postcodes (ADR-0031)
A property with no lodged EPC and no same-type comparable in its own postcode
(e.g. the only flat among houses) used to gate out and fail the subtask. The
gov EPC API cannot search by radius/outcode, so we resolve the real unit
postcodes physically nearest the target via postcodes.io (keyless; already a
trusted in-repo dependency) and walk them nearest-first until enough same-type
comparables surface. New PostcodesIoClient (transient-failure retry with
exponential backoff, soft-failing to the seed so broadening never breaks
prediction) and EpcComparablePropertiesRepository.candidates_near. Wired into
the handler and e2e runner; broadening is lazy (only on gate-out) and memoised
per (postcode, property_type).
Validated live: property 728476 (gas boiler) prices system_tune_up at GBP295;
property 718580 (lone flat in BR6 6BS) now predicts via nearby BR6 postcodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Corpus validation of the modelling_e2e photovoltaic_supply-as-list fix. Cert
6102-6227-8000-0083-2292 (RdSAP-20.0.0 semi, gas combi + 2× 1.14 kW PV arrays)
crashed from_rdsap_schema_20_0_0 on the measured-array list; the fix routes it
through the dict-tolerant _map_schema_21_pv. PV correctly credited: engine 61
(no PV) → 66 (+5). Built in Elmhurst (evidence: epc.json + summary + worksheet,
fabric+heating; the PV "New Technologies" Panel-details grid deferred): worksheet
55 = engine-on-Elmhurst-inputs 55 exactly → calculator faithful. The +6 engine-vs-
Elmhurst base-dwelling residual is the documented RdSAP-default gap (band-C cavity-
uninsulated suspended-floor semi). Pinned engine 66.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tests/orchestration/test_postcode_splitter_orchestrator.py imports
`from moto import mock_aws` (moto 5.x) but moto was absent, so the file
errored at collection. Pin moto[s3,sqs]==5.0.28 (S3+SQS are the only mocked
services); resolves cleanly against the boto3 1.35.44 and cryptography 43.0.3
pins. All 4 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@task_handler never built or passed cloud_logs_url, so every app using
it (incl. modelling_e2e) ran run_subtask with the None default and the
CloudWatch deep-link was never saved onto the SubTask. @subtask_handler
did this correctly.
Extract the URL builder into a shared utilities/aws_lambda/cloud_logs.py
(public cloudwatch_url()), use it from both handlers, and pass the URL
into run_subtask from @task_handler. Add regression tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A Landlord Override can reference a building part the lodged or predicted EPC
never carried (e.g. an extension_1 override on a property whose EPC has only
main). apply_simulations indexed parts_by_id[identifier] unguarded, raising
KeyError and aborting the whole property's modelling. Now the orphaned part is
skipped. Recovers 14 of the 22 modelling_e2e failures in portfolio 796.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@task_handler never built or passed cloud_logs_url, so every app using
it (incl. modelling_e2e) ran run_subtask with the None default and the
CloudWatch deep-link was never saved onto the SubTask. @subtask_handler
did this correctly.
Extract the URL builder into a shared utilities/aws_lambda/cloud_logs.py
(public cloudwatch_url()), use it from both handlers, and pass the URL
into run_subtask from @task_handler. Add regression tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Corpus validation of the modelling_e2e built_form fix. Cert 8742-6624-9300-2780-4926
(SAP-Schema-16.0, ground-floor electric-storage-heater flat) omits built_form; the
mapper now derives it from dwelling_type. built_form is ML-only so the fix is
SAP-neutral: engine 66 = lodged 66 exactly. Built in Elmhurst (evidence: epc.json +
summary + worksheet): worksheet 54, engine-on-Elmhurst-inputs 53 ≈ 54 → calculator
faithful. The +12 engine-vs-Elmhurst is a build/input gap (cert size-1 small cylinder
unrepresentable in Elmhurst's Normal/110L-minimum entry → higher HW + reduced-field
16.0 defaults). Pinned engine 66.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the mapper-coverage gaps surfaced by the modelling_e2e prediction-cohort
failures (portfolio 796):
- built_form (SAP-16.0): derive from dwelling_type in _normalize_sap_schema_16_x
(Mid-terrace->4, End-terrace->3, Semi-detached->2, Detached->1; flats->modal 4).
ML-only field (SAP calc never reads it) so SAP- and gate-neutral. 5 flat certs
that omitted built_form now map.
- photovoltaic_supply as a measured-array LIST: routed all pre-21 RdSAP mappers
(17.0/17.1/18.0/19.0/20.0.0) through _map_schema_21_pv, whose list branch is now
dict-tolerant (_pv_array_field reads dict OR dataclass). They capture the PV
arrays like 21.0.x instead of raising "'list' object has no attribute
none_or_no_details" and sinking the whole cohort.
- windows-as-dict (16.x): handled in the normalizer (not just windows-as-list).
Genuinely-sparse certs (omit door_count/habitable/glazed_area) remain fail-loud;
the gate-regressing multiple_glazed_proportion default and the recursive
RdSAP-21.0.0 ADR-0028 alignment are left fail-loud + flagged for review (worklist).
+5 regression tests; component-accuracy gate 26/26; 0 new pyright errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
