Wire community heating fuel + CHP fraction (epc_main_heating_detail),
alt-wall is_sheltered + wall insulation thermal conductivity
(epc_building_part), and pv_diverter_present / measured cylinder volume /
AP50 air permeability (epc_property) through save + _compose/_to_*. All
deep-equal round-trip; coverage guard now enforces their reconstruction.
Columns live (FE migration applied).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add EpcPhotovoltaicArrayModel (epc_photovoltaic_array child table) and wire
save / delete / read so sap_energy_source.photovoltaic_arrays survives
load->save->load in order. Threaded through both the single get() and the
bulk _for_properties() paths via _compose -> _to_energy_source. Column
names match the FE migration (feature/epc-pv-and-floor-heatloss-schema).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add is_exposed_floor / is_above_partially_heated_space to
EpcFloorDimensionModel and wire from_domain + _to_floor_dimension. Column
names match the FE schema (feature/epc-pv-and-floor-heatloss-schema).
Live DB migration is run post-merge (drizzle-kit generate picks them up).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Route the Google Solar client through the shared call_with_retry with
full jitter (de-synchronises the 32 concurrent containers per Google's
"avoid synchronised requests" guidance), honouring Retry-After, a 60s max
backoff (rides out the 600 QPM per-minute window), and 6 bounded retries.
429/5xx/transport errors are transient; other 4xx propagate immediately;
404-entity-not-found stays BuildingInsightsNotFoundError. On exhaustion a
TransientHttpError surfaces so the subtask fails and is re-triggered (no
silent degrade).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Persist SapConservatory as five nullable conservatory_* columns on epc_property
(1:1 with the dwelling) and rebuild it in _compose, so the §6.1 fold survives
save -> reload -> score. Without this the scored (re-hydrated) EPC silently
dropped the conservatory (persist != score) — a latent gap shared with the
21.0.1 path. Adds a deep-equality round-trip test. ADR-0036.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes out the cohort-broadening work with its decision record and consolidates
the retry plumbing.
ADR-0034 documents broadening the EPC-Prediction cohort to the real unit
postcodes nearest the target (via postcodes.io) when its own postcode holds no
same-type comparable — extending ADR-0031 decision 5. Records why postcodes.io
was chosen over council[] (whole-LA, no property_type in rows), a bulk Code-Point
Open / ONSPD dataset, and the OS Places radius API, and the lazy / nearest-first
early-stop / soft-fail policy. Broadening-specific docstrings now cite 0034.
Retry consolidation: extract the EPC client's call_with_retry into a shared
infrastructure/http_retry.py keyed off a generic TransientHttpError marker, so
the mechanism (exponential backoff, Retry-After) is shared while each client
keeps its own transient policy. EpcRateLimitError now subclasses TransientHttpError
(still an EpcApiError); PostcodesIoClient routes through the same helper, raising
TransientHttpError on 429/5xx and soft-failing to the seed once exhausted (the EPC
client propagates instead). Direct tests for the shared helper; EPC + postcodes.io
suites repointed at the shared sleep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reconciliations to make the modelling_e2e Lambda handler production-ready.
1. Price through the off-catalogue overlay, drop the workarounds
The handler priced through a plain ProductPostgresRepository and excluded
secondary_heating_removal / system_tune_up / system_tune_up_zoned to dodge
ProductNotFound (and a poisoning pgEnum DataError). Those measures are now
priced by catalogue_with_off_catalogue_overrides (already used by the e2e
runner and PostgresUnitOfWork), so the exclusions are removed and ALL measure
types are considered. This also fixes gas-boiler / single-glazed properties,
which Dan's handler never excluded and so still crashed (the standard
system_tune_up option is built unconditionally — the considered-measures
exclusion never actually gated it).
2. Broaden the EPC-Prediction cohort to nearby real postcodes (ADR-0031)
A property with no lodged EPC and no same-type comparable in its own postcode
(e.g. the only flat among houses) used to gate out and fail the subtask. The
gov EPC API cannot search by radius/outcode, so we resolve the real unit
postcodes physically nearest the target via postcodes.io (keyless; already a
trusted in-repo dependency) and walk them nearest-first until enough same-type
comparables surface. New PostcodesIoClient (transient-failure retry with
exponential backoff, soft-failing to the seed so broadening never breaks
prediction) and EpcComparablePropertiesRepository.candidates_near. Wired into
the handler and e2e runner; broadening is lazy (only on gate-out) and memoised
per (postcode, property_type).
Validated live: property 728476 (gas boiler) prices system_tune_up at GBP295;
property 718580 (lone flat in BR6 6BS) now predicts via nearby BR6 postcodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- run_modelling_e2e --from-db re-models from already-persisted inputs (reads
each Property's Effective EPC + planning protections + solar from the DB) and
skips every live fetcher — zero gov-API calls. With --persist it re-writes the
Plan and, for lodged-EPC Properties, the Baseline. Self-contained loop; the
live-fetch path is untouched. Makes local re-runs instant and avoids tripping
the gov API's per-IP rate limit (6000 req / 5 min) during iteration.
- EpcClientService.REQUEST_TIMEOUT 10s -> 30s: a cold per-UPRN search can exceed
10s and the old timeout turned it into a timeout-then-retry; 30s rides it out.
Note: an open perf question remains — modelling is fast in isolation (<0.5s/
property) but a long-lived --persist run shows ~1 min/property; suspected in the
persist path (plan.save / baseline) or connection handling, NOT the API. Left
mid-diagnosis for handover.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Caught live writing property_overrides on portfolio 796: the Python
override_component SAEnum lagged the DB enum, so reading a new-component row
back threw LookupError. Guard it with a consistency test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SQLModel had drifted to a `bill_` prefix on the Bill Derivation block, but
the FE-owned Drizzle table uses unprefixed names (`heating_kwh`, `hot_water_kwh`
… `total_annual_bill_gbp`) plus a nullable `fuel_rates_period`. INSERTs failed
with UndefinedColumn. Rename the columns to mirror the live table column-for-
column (the prefix's anti-clash purpose is moot: `heating_kwh` != the recorded
`space_heating_kwh`), and add the `fuel_rates_period` column — left None until
Bill Derivation threads the snapshot period through.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new pipeline left no per-Property record of a run (the old engine set
property.has_recommendations and populated property_details_epc). Restore the
marker: PropertyRepository.mark_modelled sets has_recommendations (true when the
Plan carries measures, mirroring the old engine) and bumps updated_at, so a
first-run under the new process is identifiable as updated_at >= 2026-06-01.
ModellingOrchestrator marks each Property after its Scenarios (true if any
Scenario yielded a measure); run_modelling_e2e's --persist path marks it too
(its compute runs on in-memory fakes, so the DB UoW sets it directly). Adds the
has_recommendations/updated_at columns to the PropertyRow mirror.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The measures a run considers should come from the Scenario, not a CLI flag.
The live scenario table persists exclusions only (no inclusions column), as a
Postgres text-array of exact MeasureType values.
- Scenario gains `exclusions: frozenset[MeasureType]` + `considered_measures()`
(all measures minus the excluded ones, or None when none are excluded).
- ScenarioModel.to_domain parses the `{a,b,c}` exclusions array into
MeasureTypes, raising on a token that is not an exact MeasureType value
(no high-level category expansion), per the strict-enum convention.
- ModellingOrchestrator._plan_for derives the allowlist from the Scenario's
exclusions, combined (intersection) with any explicit considered_measures
override via the new `combine_considered_measures`.
- run_modelling_e2e sources the allowlist from the Scenario; --measures /
--exclude-measures become optional overlays (e.g. the technical
secondary_heating_removal exclusion the catalogue cannot yet stock).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `source` discriminator (lodged | predicted) to the EPC store so a Property
holds a lodged EPC and a predicted one (EPC Prediction gap-fill) at once
(ADR-0031). EpcRepository.save gains source="lodged"; idempotent delete is now
per-source (a predicted save no longer wipes lodged, and vice versa);
get_for_property/get_for_properties filter lodged; new get_predicted_for_property
/ get_predicted_for_properties read predicted. PropertyPostgresRepository.get +
get_many hydrate Property.predicted_epc, so the predicted picture reaches the
modelling read (both load via get_many). FakeEpcRepo mirrors the dual slot.
EpcPropertyModel gains `source` (default "lodged"); the test DB builds from the
SQLModel mirror so this is exercised without the prod migration. The matching
Drizzle change (column + per-(property_id,source) uniqueness) is the team's to
action before merge — docs/MIGRATION_NOTE_predicted_epc_source.md.
3 store tests (coexist, idempotent predicted re-save leaves lodged, lodged-only
has no predicted) + property-repo wiring; 85 pass across affected suites; new
code pyright-clean (2 pre-existing wwhrs errors in epc_property_table untouched).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Promote RdSAP-Schema-17.0 into SUPPORTED so all 1000 corpus certs are held to
the strict parse+map guard. Drop the now-redundant cert[0] tracer (subsumed by
the parametrised bucket); keep the reduced-field synthesis behavioural test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Promote RdSAP-Schema-19.0 into SUPPORTED so all 1000 corpus certs are held to
the strict parse+map guard. Drop the now-redundant cert[0] tracer (subsumed by
the parametrised bucket); keep the reduced-field synthesis behavioural test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Schema-fix delivered in the prior slice; this promotes the 1000-cert corpus
into the strict guard (matches 18.0/20.0.0).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drops the xfail now that Reduced-Field Synthesis (ADR-0027) maps every
20.0.0 cert; the corpus test holds the whole bucket to a strict 1000/1000.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pulls in 42 commits of calculator/mapper accuracy fixes from the per-cert
mapper-validation and floor/roof/heating fronts.
Conflict resolutions:
- mapper `_is_elmhurst_roof_window`: main dropped the branch's "wall location →
vertical" guard (it broke cert 000516's rooflight), but that re-broke cert
001431's two External-wall U>3.0 windows (which must stay vertical). The two
certs lodge a BYTE-IDENTICAL §11 row, so neither location nor U separates
them — the real discriminator is the room-in-roof context. Replaced the
unconditional U>3.0 backstop with one gated on the BP having a room-in-roof
(`_elmhurst_bp_has_room_in_roof`): 000516's Main BP has a "Room in roof type
1" (→ rooflight), 001431's does not (→ vertical). Validated against BOTH —
full Elmhurst worksheet suite 1038 pass + the 001431 window-extraction pin.
- property_postgres_repository: kept main's `ids_by_uprn` method + the branch's
`_restrictions_of` helper.
- sap_fuel.py: the branch relocated it to domain/billing/ (already carrying
main's to_table_32_code normalization), so kept the old path deleted.
Fallout from main's fabric fixes (validated by the boiler-3 real-cert pin which
still reproduces at delta 0):
- re-pinned the boiler-1 + boiler-instant-hw ASHP snapshot scores;
- main's §14.2 gas-boiler main-fuel derivation resolved the BGB/102 baseline
gap, so `test_gas_boiler_instant_hw_before_baselines` is now a passing test
(was an xfail tripwire).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
