- run_modelling_e2e --from-db re-models from already-persisted inputs (reads
each Property's Effective EPC + planning protections + solar from the DB) and
skips every live fetcher — zero gov-API calls. With --persist it re-writes the
Plan and, for lodged-EPC Properties, the Baseline. Self-contained loop; the
live-fetch path is untouched. Makes local re-runs instant and avoids tripping
the gov API's per-IP rate limit (6000 req / 5 min) during iteration.
- EpcClientService.REQUEST_TIMEOUT 10s -> 30s: a cold per-UPRN search can exceed
10s and the old timeout turned it into a timeout-then-retry; 30s rides it out.
Note: an open perf question remains — modelling is fast in isolation (<0.5s/
property) but a long-lived --persist run shows ~1 min/property; suspected in the
persist path (plan.save / baseline) or connection handling, NOT the API. Left
mid-diagnosis for handover.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make run_modelling_e2e the single script that does everything for a portfolio,
so the 291-property run needs one invocation with per-property recovery (no
all-or-nothing chunking):
- On --persist, a lodged-EPC Property now also gets its Baseline Performance
row written via PropertyBaselineOrchestrator (per Property, so one bad cert
does not abort the batch). Predicted (EPC-less) Properties have no lodged
figures, so they get a Plan but no baseline row.
- The run CSV gains api_sap (register) vs baseline_sap (calculator) + sap_delta,
so calculator-vs-API divergence is reviewable per property.
Fill the off-catalogue overlay for the measures the live material catalogue
cannot price, so they stop crashing the run:
- double_glazing (£550/window) and secondary_glazing (£400/window): priced
per window (the generator multiplies by single-glazed window count, matching
the legacy window_glazing). Grounded in 2025/26 UK installed costs; per-window
is the right unit for windows (fixed per-unit install dominates) — per-m2 fits
walls/floors, not glazing.
- gas_boiler_upgrade / system_tune_up / system_tune_up_zoned: these are priced
off the heating rate sheet (Products()), with get() reading the catalogue only
for an id — so the overlay entry exists to satisfy that lookup (material_id
stays None, as with ASHP); the rate sheet remains authoritative.
Validated on a 12-property sample (incl. a secondary-glazing case and a
SAP-Schema-16.2 cert): 12/12 baseline rows + plans, 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two-pass org_ref-matched builder for property_overrides (classify via ChatGPT
into the landlord ledger, validate+apply user edits, write idempotently);
ephemeral-Postgres smoke proves the one-property chain without creds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The FE-owned `material.type` pgEnum cannot carry `secondary_heating_removal`,
so pricing it through the DB catalogue raises a DataError that poisons the
session — the modelling pipeline crashed on any property with a lodged
secondary heater unless the measure was excluded on the Scenario.
Realise the `ProductRepository` docstring's intent (DB catalogue today, a JSON
file for costs the ETL does not yet supply, behind the same port): add a
`CompositeProductRepository` that resolves an override source first, then the
catalogue. Checking the override first keeps that Measure Type away from the DB
entirely; every other type misses the override and falls through unchanged.
- off_catalogue_costs.json prices it at £270 flat per-dwelling — the legacy
`Costs.heater_removal` ported to the new flat model (ADR-0028):
(£25 + £200 baseline) x 1.2 VAT, for the single fixed secondary a cert lodges.
Contingency (0.25) is joined from config, not the file.
- Wire the composite into PostgresUnitOfWork.product and run_modelling_e2e, so
the first-run pipeline and the local runner both honour the overlay.
- Integration test: drop the unrealistic seeded secondary_heating_removal DB
rows (the pgEnum can't hold the type) and assert it is JSON-sourced
(material_id is None, cost £270) end-to-end through a real Unit of Work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`profile_corpus_error.py` and `dive_cert.py` compared our PE/CO2 against
the lodged EPC figures using the UK-average RATING cascade, but the EPC
lodges CO2/PE on the postcode DEMAND cascade (SAP 10.2 Appendix U p.124,
now wired into Sap10Calculator.calculate in fc7c4d2d). That confounded the
DEMAND-vs-COST triage: a cert whose demand actually reproduced on local
weather looked "PE off" purely from the climate difference and was
mislabelled DEMAND-side. Switching the PE/CO2 lens to `cert_to_demand_
inputs` (SAP still from the rating cascade) re-classifies the corpus
outside-0.5 set 261/42 -> 211/92 DEMAND/COST — ~50 certs are genuinely
cost-side (e.g. 10091578598: SAP +7.81 but PE +1.6 / CO2 -0.04). Sharpens
the hunt for the subtle widespread SAP term.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
profile_corpus_error.py buckets signed SAP error by raw-API feature and
lists worst over/under-raters with the PE/CO2-vs-cost split (COST-side vs
DEMAND-side triage). dive_cert.py dumps one cert's lodged-vs-ours
SAP/CO2/PE + full intermediate line refs + mapped inputs. Both run on the
committed RdSAP-21.0.1 corpus (no /tmp sample needed). Used to find the
stone-wall, per-part-roof, ground-floor-flat and HP-water fixes this session.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/run_first_run_e2e.py runs the real Ingestion -> Baseline -> Modelling
pipeline against the DB by composing build_first_run_pipeline + dispatch_first_run
with the live source clients (the Lambda handler can't run locally — its
_source_clients_from_env still raises, #1136). Unlike run_modelling_e2e it runs
real ingestion (persists EPC/spatial/solar) and has no inspect-only mode, so it's
gated behind --confirm (preview otherwise); measure scoping comes only from the
Scenario's exclusions (the pipeline threads no --measures), and the modelling
batch is all-or-nothing, both documented.
Extract the shared env/engine/S3 plumbing into scripts/e2e_common.py (public
load_env/build_engine/s3_parquet_reader) so both runners share one source and
neither imports the other's privates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The new pipeline left no per-Property record of a run (the old engine set
property.has_recommendations and populated property_details_epc). Restore the
marker: PropertyRepository.mark_modelled sets has_recommendations (true when the
Plan carries measures, mirroring the old engine) and bumps updated_at, so a
first-run under the new process is identifiable as updated_at >= 2026-06-01.
ModellingOrchestrator marks each Property after its Scenarios (true if any
Scenario yielded a measure); run_modelling_e2e's --persist path marks it too
(its compute runs on in-memory fakes, so the DB UoW sets it directly). Adds the
has_recommendations/updated_at columns to the PropertyRow mirror.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every worklist UPRN now carries schema · engine SAP / lodged · flag. Tally:
64 healthy, 19 MVHR-not-credited (🚩 flag B), 6 heat-pump fuel-39 (🚩 flag A),
4 sparse/NOT MAPPABLE (⛔), 3 Elmhurst-pinned. MVHR is the largest accuracy gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Autonomous-run triage of the moderate eng-vs-lodged gaps resolves them into two
patterns, both flagged for owner review (not auto-fixable):
- Heat-pump fuel code 39 mis-priced as gas (over-rates; both gap directions).
- MVHR heat recovery modelled as plain extract loss → systematic UNDER-rating
(~8-12 SAP) on every full-SAP cert carrying a mechanical_vent_system_index_number.
New memory mvhr-heat-recovery-not-modelled; needs the Appendix Q / PCDB MVHR
efficiency model.
findings doc updated with the classification.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Some SAP-Schema-17.x/18.0.0 certs lodge sap_openings width/height in MILLIMETRES
mixed with metre rows in the same array (e.g. a 2025x2100 mm window beside a
3.06x1 m one). The 17.1 mapper read them all as metres → a 4.25M m2 window →
HTC in the millions → SAP clamped to 1.
Fix (TDD, datatypes/epc/domain/mapper.py): _sanitise_opening_dimension_m treats
any dimension > 50 m as mm and divides by 1000; _sap_opening_area_m2 applies it
to areas. Wired into the window, roof-window, and door-area-weighting paths.
The 3 broken certs (uprn_10093117227 / 10090317693 / 10091636031) now score
90 / 81 / 79 instead of 1.
3 RED->GREEN slices + refactor; new test class
TestFromSapSchema17_1OpeningUnitSanitisation + sap_17_1_mm_openings.json fixture;
0 new pyright errors; no regressions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Schema coverage (datatypes/epc/domain/mapper.py):
- SAP-Schema-18.0.0: full-SAP shape ≡ 17.1 → from_sap_schema_17_1, no normalisation.
- SAP-Schema-16.0: same reduced-field 16.x path; default the omitted `tenure`
field in _normalize_sap_schema_16_x (metadata; SAP cascade never reads it).
Genuinely sparse 16.x certs (missing core fabric fields) still fail loud.
- Regression tests + sap_18_0_0.json / sap_16_0.json fixtures; 0 new pyright errors.
Autonomous triage of the worklist (scripts/hyde/autonomous_run_findings.md):
- Found + diagnosed 2 bugs (flagged, NOT fixed): (1) MAPPER — full-SAP openings
lodged in mm read as m → multi-million-m2 windows → SAP clamps to 1 (uprn_
10093117227 / 10090317693 / 10091636031); (2) CALCULATOR — database heat-pump
fuel code 39 mis-priced as gas, over-rates ~14 (uprn_10093114053).
- Most certs map within +/-4 of lodged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Routes run_modelling through prop.effective_epc and dumps each target's
property_overrides before the run, so a landlord wall override moves the
calculated SAP. Records the overlay design in ADR-0032.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update run_modelling_e2e's docstring so another dev can run it: the Scenario's
exclusions drive measure scoping (--measures/--exclude-measures are overlays),
and flag the secondary_heating_removal catalogue gap that currently requires
--exclude-measures. Replace the stale --measures examples with the real
scenario-driven inspect/persist commands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The measures a run considers should come from the Scenario, not a CLI flag.
The live scenario table persists exclusions only (no inclusions column), as a
Postgres text-array of exact MeasureType values.
- Scenario gains `exclusions: frozenset[MeasureType]` + `considered_measures()`
(all measures minus the excluded ones, or None when none are excluded).
- ScenarioModel.to_domain parses the `{a,b,c}` exclusions array into
MeasureTypes, raising on a token that is not an exact MeasureType value
(no high-level category expansion), per the strict-enum convention.
- ModellingOrchestrator._plan_for derives the allowlist from the Scenario's
exclusions, combined (intersection) with any explicit considered_measures
override via the new `combine_considered_measures`.
- run_modelling_e2e sources the allowlist from the Scenario; --measures /
--exclude-measures become optional overlays (e.g. the technical
secondary_heating_removal exclusion the catalogue cannot yet stock).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The run only showed the measures the Optimiser selected, so a candidate it
passed over (e.g. an ASHP it found too costly for the target band) and that
measure's cost were invisible.
Add `harness.console.candidate_recommendations` — every Generator Option
with its per-Option cost, before optimisation — and have run_modelling_e2e
print the full menu per property (flagging the selected Options), write a
"cost per measure" section into the markdown, and emit a per-Option
modelling_e2e_candidates.csv.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`restrict_to_considered_measures` filtered candidates only *after* every
generator had run, so an excluded measure still queried the catalogue.
That crashed properties with a lodged secondary heater: the live
`material.type` enum has no `secondary_heating_removal` value, so the
query raised a psycopg2 `InvalidTextRepresentation` before the allowlist
could drop it.
`_candidate_recommendations` now pairs each generator with the measure
types it can emit and runs it only when the allowlist admits one of them
(None = all), so an excluded measure never reaches the catalogue.
`restrict_to_considered_measures` still trims disallowed Options off the
multi-Option survivors. Add `--exclude-measures` to run_modelling_e2e
(allowlist minus the excluded set) for excluding one measure without
enumerating the rest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two review points from @dancafc:
1) Rename the `Comparable` dataclass → `ComparableProperty` (it models one
comparable *property*; the collection stays `ComparableProperties`). Applied
across domain, repositories, orchestration, harness, scripts, and tests with a
word-boundary rename so `ComparableProperties` is untouched.
2) Move `PredictionTarget` out of comparable_properties.py into prediction_target.py
(where `PredictionTargetAttributes` + `build_prediction_target` already live).
comparable_properties.py now imports it; no import cycle (prediction_target no
longer depends on comparable_properties). Importers updated.
92 tests pass across the touched suites; pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build a geographically DENSE postcode-clustered corpus to test cross-postcode
geo expansion (the handover's anticipated "real geo payoff"). The gov EPC API
has no area/prefix search (a partial postcode 400s; the old opendatacommunities
partial-search API is decommissioned), so neighbourhood enumeration is external:
seed K postcodes nationally, expand each via postcodes.io's nearest-postcode
endpoint into every unit within RADIUS_M, pull each one's full EPC cohort.
postcodes.io is a corpus-BUILD dependency only — the predictor stays pure. Same
on-disk layout as the scattered corpus, so load_corpus + the coords resolver
consume it unchanged.
MEASURE-FIRST RESULT — cross-postcode expansion is a NO-GO. On a 2-seed pilot
(York YO19 + Islington N51, 81 postcodes / 1558 certs, 140 SAP-10.2 targets),
pooling nearby postcodes regresses accuracy across the board:
same-postcode FA_MAE 9.53 wall 92% age 72% floor_con 85% cylinder 91%
cross <=0.3km FA_MAE 13.1 wall 80% age 61% floor_con 82% cylinder 79%
Even as a thin-cohort top-up it hurts (thin n=18: FA 5.24 -> 7.15). Root cause:
the postcode boundary is itself a strong homogeneity prior (a postcode is one
coherent street/development), so same-postcode neighbours beat geographically
near cross-boundary ones even when the home postcode is sparse (and they rarely
are — median same-postcode cohort here is 34). Geo-proximity helps WITHIN a
postcode (#1227) but does not survive crossing the boundary. Cross-postcode geo
closed; geo weighting stays intra-postcode. Tooling kept (reusable).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reduced-field window U: heat_transmission derived the synthesised-window
raw U from u_window(all None) -> the 2.5 placeholder regardless of glazing.
Now routes the (uniform) glazing_type code through u_window (RdSAP Table 24)
so e.g. double pre-2002 reads 2.8, not 2.5. Only the pre-SAP10 reduced-field
path is affected (21.0.1 certs carry per-window U upstream) — the RdSAP-21.0.1
corpus gauge is unchanged at 66.9% within-0.5.
test_real_cert_sap_accuracy: pin uprn_10002468137 (RdSAP-17.1, all-electric
storage heaters) at SAP 61, validated against Elmhurst on identical inputs
(dual off-peak immersion, 110 L cylinder, 2 baths). Our engine reproduces
Elmhurst's fuel cost to the penny; lodged 55 is the old SAP-2012 schema.
Tooling to grow the accuracy corpus:
- scripts/fetch_real_life_epc_sample.py — capture a cert by UPRN into the corpus.
- scripts/compare_epc_paths.py — diff gov-API vs Elmhurst-summary EpcPropertyData
and run both through the engine, localising mapper vs calculator differences.
- skill validate-cert-sap-accuracy — the end-to-end loop (capture -> Elmhurst
inputs -> human builds -> compare -> reconcile -> pin in the test).
- skill epc-to-elmhurst-rdsap-inputs reference: corrected immersion (code 1=dual),
cylinder size (code 2 = Normal/110 L), and bath-count (WWHRS sub-tab) mappings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a floor_area line giving MAE (m2), MAPE (% of actual), and the typical
(median actual) size, so the absolute error reads relative to dwelling size.
Corpus: MAE 10.48 m2 / MAPE 13.2% / typical 61 m2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Folds a haversine distance kernel into the categorical-mode weighting so a
nearer neighbour counts for more — applied ONLY to the components that showed
a clear distance signal in the corpus pre-check (age band, wall + floor
construction, glazing: homes built/retrofitted together cluster). Roof
construction showed no decay and is excluded; heating keeps its coherent
donor. Predictor stays pure: weights come from target.coordinates vs each
Comparable.coordinates (resolved at the boundary); geo is OFF when the target
has no coords, neutral for a neighbour with none.
Scale chosen on the harness: _GEO_SCALE_KM=0.1 is the gate-safe optimum
(0.05 lifts the corpus more but regresses fixture floor_construction).
Corpus (150pc/514, geo off->on): age 0.564->0.572, age_pm1 0.841->0.847,
wall 0.902->0.912, floor_con 0.786->0.796, glazing 0.667->0.673; roof
unchanged. Fixture: glazing 0.5278->0.5833 (floor ratcheted), all else held.
Refactored recency into a reusable _recency_weights vector composed via
_combine, so similarity/recency/geo factors multiply uniformly. Fixture ships
a committed _coordinates.json (OGL OS OpenData; build script carries it from
the corpus sidecar on rebuild) so the gate exercises geo without S3.
This is the per-component method applied to geography ([[feedback_per_component_best_method]]).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds coordinates: Optional[Coordinates] to Comparable and PredictionTarget
(data carriers — the pure predictor stays IO-free), and wires load_corpus to
read an optional _coordinates.json sidecar ({uprn: [lon, lat]}) and populate
each Comparable from its cert's uprn; iter_predictions threads the held-out
target's coordinates through. Absent sidecar -> geo-weighting stays off (no
behaviour change yet — weighting lands next slice). fetch_corpus_coordinates
now writes the sidecar into the corpus dir. load_corpus populates 99% of
corpus comparables.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One-time utility: resolves every corpus cert's uprn -> WGS84 lon/lat from the
OS Open-UPRN parquet (DATA_BUCKET/spatial/) via boto3, grouping UPRNs by their
covering partition so each ~1.7MB partition is read at most once (the efficient
batch lookup we intend to add to GeospatialRepository). Caches {uprn:[lon,lat]}
locally for the validation harness. Resolved 2609/2683 corpus UPRNs (97%).
Signal pre-check result (does intra-postcode proximity predict components?):
intra-postcode distances are non-trivial (median 44m, p90 138m, max ~1km),
and nearer neighbours match the target markedly better on age band (0.63 at
<20m -> 0.16 at >300m), wall, glazing and floor construction. Roof shows no
decay. => geo-proximity is worth building, per-component (strongest for age,
the weakest fabric component).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The validation report showed only the SAP calculator floor (calc(actual) vs
lodged), so the headline PEI MAE (~40 kWh/m2) read as prediction error when
much of it is the calculator's own API-path residual. Adds the CO2 + PEI
floors alongside SAP.
Diagnostic (150pc/514): PEI floor MAE 15.73 (calc(actual) vs lodged) vs SAP
floor 1.57; calc(actual)/lodged PEI ratio ~1.06 (mean +10.7, ~+6% over-
estimate). That RULES OUT the suspected gross unit/definition mismatch (a
unit bug would be ~2x/3.6x, not 1.06) and reframes #1228: the PEI gap is a
modest calculator bias (~16 floor, calc-branch) plus a larger prediction-
sensitivity term (~24) — PEI is far more prediction-sensitive than SAP.
CO2 floor 0.20 t. Script-only; no gate impact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
