The row→domain mapper now names all 93 constructor arguments explicitly
instead of splatting a lowercased dict, takes a plain Mapping (a
DataFrame.to_dict("records") row) instead of a pandas Series, and ignores
columns the domain type doesn't know. A missing/renamed CSV column fails
loudly as a KeyError at the row. Both iterrows() call sites move to
to_dict("records") — pandas-stubs types iterrows' Series unparameterized,
which strict mode rejects. pandas-stubs + boto3-stubs[s3] make the stack
check clean: pyright strict is now 0 errors across the PR's files.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The four lodged_* columns are nullable as a unit; from_domain writes them all
NULL when a predicted Property has no Lodged Performance, and to_domain
reconstructs lodged=None (lodged_sap_score is the read discriminator, mirroring
the bill block). The production lodged_* columns are FE-owned (Drizzle) and need
a companion ALTER ... DROP NOT NULL migration before the backend writes NULL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HistoricEpcS3Repository reached into utils/s3.py (read_csv_gz_from_s3 +
parse_s3_uri), the legacy utility that self-constructs boto3 inside free
functions. The other S3 repositories deliberately depend on the
infrastructure/s3 layer instead (UnstandardisedAddressListCsvS3Repository
injects a CsvS3Client). Bring historic EPC into line.
- Add GzipCsvS3Client(S3Client) in infrastructure/s3: read_csv_gz(key) ->
DataFrame (get_object + gzip decode).
- Inject it into HistoricEpcS3Repository; the bucket lives in the client and
the repo only builds the per-postcode key + maps rows (no S3/HTTP code).
Add with_default_s3_client(s3_root) for composition roots.
- Update main.py and the match_addresses_for_postcode seam to the factory.
- Repo tests inject a real GzipCsvS3Client over a controlled boto stub
(exact key assertions + AccessDenied); add a moto-based client test and a
factory test covering s3_root -> bucket+key.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQE5TsSuQTeNSCSz9A9GQf
Wire community heating fuel + CHP fraction (epc_main_heating_detail),
alt-wall is_sheltered + wall insulation thermal conductivity
(epc_building_part), and pv_diverter_present / measured cylinder volume /
AP50 air permeability (epc_property) through save + _compose/_to_*. All
deep-equal round-trip; coverage guard now enforces their reconstruction.
Columns live (FE migration applied).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add EpcPhotovoltaicArrayModel (epc_photovoltaic_array child table) and wire
save / delete / read so sap_energy_source.photovoltaic_arrays survives
load->save->load in order. Threaded through both the single get() and the
bulk _for_properties() paths via _compose -> _to_energy_source. Column
names match the FE migration (feature/epc-pv-and-floor-heatloss-schema).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add is_exposed_floor / is_above_partially_heated_space to
EpcFloorDimensionModel and wire from_domain + _to_floor_dimension. Column
names match the FE schema (feature/epc-pv-and-floor-heatloss-schema).
Live DB migration is run post-merge (drizzle-kit generate picks them up).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Route the Google Solar client through the shared call_with_retry with
full jitter (de-synchronises the 32 concurrent containers per Google's
"avoid synchronised requests" guidance), honouring Retry-After, a 60s max
backoff (rides out the 600 QPM per-minute window), and 6 bounded retries.
429/5xx/transport errors are transient; other 4xx propagate immediately;
404-entity-not-found stays BuildingInsightsNotFoundError. On exhaustion a
TransientHttpError surfaces so the subtask fails and is re-triggered (no
silent degrade).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Persist SapConservatory as five nullable conservatory_* columns on epc_property
(1:1 with the dwelling) and rebuild it in _compose, so the §6.1 fold survives
save -> reload -> score. Without this the scored (re-hydrated) EPC silently
dropped the conservatory (persist != score) — a latent gap shared with the
21.0.1 path. Adds a deep-equality round-trip test. ADR-0036.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes out the cohort-broadening work with its decision record and consolidates
the retry plumbing.
ADR-0034 documents broadening the EPC-Prediction cohort to the real unit
postcodes nearest the target (via postcodes.io) when its own postcode holds no
same-type comparable — extending ADR-0031 decision 5. Records why postcodes.io
was chosen over council[] (whole-LA, no property_type in rows), a bulk Code-Point
Open / ONSPD dataset, and the OS Places radius API, and the lazy / nearest-first
early-stop / soft-fail policy. Broadening-specific docstrings now cite 0034.
Retry consolidation: extract the EPC client's call_with_retry into a shared
infrastructure/http_retry.py keyed off a generic TransientHttpError marker, so
the mechanism (exponential backoff, Retry-After) is shared while each client
keeps its own transient policy. EpcRateLimitError now subclasses TransientHttpError
(still an EpcApiError); PostcodesIoClient routes through the same helper, raising
TransientHttpError on 429/5xx and soft-failing to the seed once exhausted (the EPC
client propagates instead). Direct tests for the shared helper; EPC + postcodes.io
suites repointed at the shared sleep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reconciliations to make the modelling_e2e Lambda handler production-ready.
1. Price through the off-catalogue overlay, drop the workarounds
The handler priced through a plain ProductPostgresRepository and excluded
secondary_heating_removal / system_tune_up / system_tune_up_zoned to dodge
ProductNotFound (and a poisoning pgEnum DataError). Those measures are now
priced by catalogue_with_off_catalogue_overrides (already used by the e2e
runner and PostgresUnitOfWork), so the exclusions are removed and ALL measure
types are considered. This also fixes gas-boiler / single-glazed properties,
which Dan's handler never excluded and so still crashed (the standard
system_tune_up option is built unconditionally — the considered-measures
exclusion never actually gated it).
2. Broaden the EPC-Prediction cohort to nearby real postcodes (ADR-0031)
A property with no lodged EPC and no same-type comparable in its own postcode
(e.g. the only flat among houses) used to gate out and fail the subtask. The
gov EPC API cannot search by radius/outcode, so we resolve the real unit
postcodes physically nearest the target via postcodes.io (keyless; already a
trusted in-repo dependency) and walk them nearest-first until enough same-type
comparables surface. New PostcodesIoClient (transient-failure retry with
exponential backoff, soft-failing to the seed so broadening never breaks
prediction) and EpcComparablePropertiesRepository.candidates_near. Wired into
the handler and e2e runner; broadening is lazy (only on gate-out) and memoised
per (postcode, property_type).
Validated live: property 728476 (gas boiler) prices system_tune_up at GBP295;
property 718580 (lone flat in BR6 6BS) now predicts via nearby BR6 postcodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- run_modelling_e2e --from-db re-models from already-persisted inputs (reads
each Property's Effective EPC + planning protections + solar from the DB) and
skips every live fetcher — zero gov-API calls. With --persist it re-writes the
Plan and, for lodged-EPC Properties, the Baseline. Self-contained loop; the
live-fetch path is untouched. Makes local re-runs instant and avoids tripping
the gov API's per-IP rate limit (6000 req / 5 min) during iteration.
- EpcClientService.REQUEST_TIMEOUT 10s -> 30s: a cold per-UPRN search can exceed
10s and the old timeout turned it into a timeout-then-retry; 30s rides it out.
Note: an open perf question remains — modelling is fast in isolation (<0.5s/
property) but a long-lived --persist run shows ~1 min/property; suspected in the
persist path (plan.save / baseline) or connection handling, NOT the API. Left
mid-diagnosis for handover.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Caught live writing property_overrides on portfolio 796: the Python
override_component SAEnum lagged the DB enum, so reading a new-component row
back threw LookupError. Guard it with a consistency test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SQLModel had drifted to a `bill_` prefix on the Bill Derivation block, but
the FE-owned Drizzle table uses unprefixed names (`heating_kwh`, `hot_water_kwh`
… `total_annual_bill_gbp`) plus a nullable `fuel_rates_period`. INSERTs failed
with UndefinedColumn. Rename the columns to mirror the live table column-for-
column (the prefix's anti-clash purpose is moot: `heating_kwh` != the recorded
`space_heating_kwh`), and add the `fuel_rates_period` column — left None until
Bill Derivation threads the snapshot period through.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new pipeline left no per-Property record of a run (the old engine set
property.has_recommendations and populated property_details_epc). Restore the
marker: PropertyRepository.mark_modelled sets has_recommendations (true when the
Plan carries measures, mirroring the old engine) and bumps updated_at, so a
first-run under the new process is identifiable as updated_at >= 2026-06-01.
ModellingOrchestrator marks each Property after its Scenarios (true if any
Scenario yielded a measure); run_modelling_e2e's --persist path marks it too
(its compute runs on in-memory fakes, so the DB UoW sets it directly). Adds the
has_recommendations/updated_at columns to the PropertyRow mirror.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The measures a run considers should come from the Scenario, not a CLI flag.
The live scenario table persists exclusions only (no inclusions column), as a
Postgres text-array of exact MeasureType values.
- Scenario gains `exclusions: frozenset[MeasureType]` + `considered_measures()`
(all measures minus the excluded ones, or None when none are excluded).
- ScenarioModel.to_domain parses the `{a,b,c}` exclusions array into
MeasureTypes, raising on a token that is not an exact MeasureType value
(no high-level category expansion), per the strict-enum convention.
- ModellingOrchestrator._plan_for derives the allowlist from the Scenario's
exclusions, combined (intersection) with any explicit considered_measures
override via the new `combine_considered_measures`.
- run_modelling_e2e sources the allowlist from the Scenario; --measures /
--exclude-measures become optional overlays (e.g. the technical
secondary_heating_removal exclusion the catalogue cannot yet stock).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `source` discriminator (lodged | predicted) to the EPC store so a Property
holds a lodged EPC and a predicted one (EPC Prediction gap-fill) at once
(ADR-0031). EpcRepository.save gains source="lodged"; idempotent delete is now
per-source (a predicted save no longer wipes lodged, and vice versa);
get_for_property/get_for_properties filter lodged; new get_predicted_for_property
/ get_predicted_for_properties read predicted. PropertyPostgresRepository.get +
get_many hydrate Property.predicted_epc, so the predicted picture reaches the
modelling read (both load via get_many). FakeEpcRepo mirrors the dual slot.
EpcPropertyModel gains `source` (default "lodged"); the test DB builds from the
SQLModel mirror so this is exercised without the prod migration. The matching
Drizzle change (column + per-(property_id,source) uniqueness) is the team's to
action before merge — docs/MIGRATION_NOTE_predicted_epc_source.md.
3 store tests (coexist, idempotent predicted re-save leaves lodged, lodged-only
has no predicted) + property-repo wiring; 85 pass across affected suites; new
code pyright-clean (2 pre-existing wwhrs errors in epc_property_table untouched).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Promote RdSAP-Schema-17.0 into SUPPORTED so all 1000 corpus certs are held to
the strict parse+map guard. Drop the now-redundant cert[0] tracer (subsumed by
the parametrised bucket); keep the reduced-field synthesis behavioural test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
