scripts/run_first_run_e2e.py runs the real Ingestion -> Baseline -> Modelling
pipeline against the DB by composing build_first_run_pipeline + dispatch_first_run
with the live source clients (the Lambda handler can't run locally — its
_source_clients_from_env still raises, #1136). Unlike run_modelling_e2e it runs
real ingestion (persists EPC/spatial/solar) and has no inspect-only mode, so it's
gated behind --confirm (preview otherwise); measure scoping comes only from the
Scenario's exclusions (the pipeline threads no --measures), and the modelling
batch is all-or-nothing, both documented.
Extract the shared env/engine/S3 plumbing into scripts/e2e_common.py (public
load_env/build_engine/s3_parquet_reader) so both runners share one source and
neither imports the other's privates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The new pipeline left no per-Property record of a run (the old engine set
property.has_recommendations and populated property_details_epc). Restore the
marker: PropertyRepository.mark_modelled sets has_recommendations (true when the
Plan carries measures, mirroring the old engine) and bumps updated_at, so a
first-run under the new process is identifiable as updated_at >= 2026-06-01.
ModellingOrchestrator marks each Property after its Scenarios (true if any
Scenario yielded a measure); run_modelling_e2e's --persist path marks it too
(its compute runs on in-memory fakes, so the DB UoW sets it directly). Adds the
has_recommendations/updated_at columns to the PropertyRow mirror.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update run_modelling_e2e's docstring so another dev can run it: the Scenario's
exclusions drive measure scoping (--measures/--exclude-measures are overlays),
and flag the secondary_heating_removal catalogue gap that currently requires
--exclude-measures. Replace the stale --measures examples with the real
scenario-driven inspect/persist commands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The measures a run considers should come from the Scenario, not a CLI flag.
The live scenario table persists exclusions only (no inclusions column), as a
Postgres text-array of exact MeasureType values.
- Scenario gains `exclusions: frozenset[MeasureType]` + `considered_measures()`
(all measures minus the excluded ones, or None when none are excluded).
- ScenarioModel.to_domain parses the `{a,b,c}` exclusions array into
MeasureTypes, raising on a token that is not an exact MeasureType value
(no high-level category expansion), per the strict-enum convention.
- ModellingOrchestrator._plan_for derives the allowlist from the Scenario's
exclusions, combined (intersection) with any explicit considered_measures
override via the new `combine_considered_measures`.
- run_modelling_e2e sources the allowlist from the Scenario; --measures /
--exclude-measures become optional overlays (e.g. the technical
secondary_heating_removal exclusion the catalogue cannot yet stock).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The run only showed the measures the Optimiser selected, so a candidate it
passed over (e.g. an ASHP it found too costly for the target band) and that
measure's cost were invisible.
Add `harness.console.candidate_recommendations` — every Generator Option
with its per-Option cost, before optimisation — and have run_modelling_e2e
print the full menu per property (flagging the selected Options), write a
"cost per measure" section into the markdown, and emit a per-Option
modelling_e2e_candidates.csv.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`restrict_to_considered_measures` filtered candidates only *after* every
generator had run, so an excluded measure still queried the catalogue.
That crashed properties with a lodged secondary heater: the live
`material.type` enum has no `secondary_heating_removal` value, so the
query raised a psycopg2 `InvalidTextRepresentation` before the allowlist
could drop it.
`_candidate_recommendations` now pairs each generator with the measure
types it can emit and runs it only when the allowlist admits one of them
(None = all), so an excluded measure never reaches the catalogue.
`restrict_to_considered_measures` still trims disallowed Options off the
multi-Option survivors. Add `--exclude-measures` to run_modelling_e2e
(allowlist minus the excluded set) for excluding one measure without
enumerating the rest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ASHP bundle is priced from the rate sheet (ADR-0025); the catalogue
row is read only for its material id, which is nullable end-to-end. The
live `material` catalogue has no `air_source_heat_pump` row, so
`products.get` raised `ValueError: no active product` and aborted every
ASHP-eligible property.
Add `ProductNotFound(ValueError)` + a concrete `ProductRepository
.get_optional`, raise the typed error from both repos, and have
`_ashp_option` look the row up optionally — a missing row now yields an
ASHP Option with `material_id=None` rather than crashing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
build_first_run_pipeline now constructs epc_prediction=EpcPrediction() and accepts
comparables_repo + prediction_attributes_reader as optional params, threading them
into IngestionOrchestrator (ADR-0031). The on-switch is now just supplying those
two arguments — no orchestrator/handler edits — once they exist: the cohort repo
(its EPC client is the source client pending #1136) and the property_overrides
attributes reader (built separately). Both default None, so the feature stays OFF
and ingestion is unchanged until they're passed.
The epc_property.source migration is live, so the predicted-EPC persistence slot
(slice-5c) now works against the real DB. Handover updated to reflect the simpler
composition-root step.
pyright strict clean; handler + pipeline + ingestion-prediction tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two review points from @dancafc:
1) Rename the `Comparable` dataclass → `ComparableProperty` (it models one
comparable *property*; the collection stays `ComparableProperties`). Applied
across domain, repositories, orchestration, harness, scripts, and tests with a
word-boundary rename so `ComparableProperties` is untouched.
2) Move `PredictionTarget` out of comparable_properties.py into prediction_target.py
(where `PredictionTargetAttributes` + `build_prediction_target` already live).
comparable_properties.py now imports it; no import cycle (prediction_target no
longer depends on comparable_properties). Importers updated.
92 tests pass across the touched suites; pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Golden regression fixture for the multi-feature dwelling that surfaced the
two Elmhurst-extractor bugs in a33707f8. case 43 is a 2-BP mid-terrace with
a DETAILED room-in-roof (two slopes, two flat ceilings, party + exposed
gables, two common walls), a MIXED-insulation multi-section roof (Main
insulated + Extension uninsulated 2.30), a DRY-LINED extension solid wall,
a mains-gas boiler (102 / control 2106) and a House-coal solid-fuel
secondary (633).
Routes the Summary PDF through the WHOLE extractor + mapper + calculator
pipeline (no hand-built EpcPropertyData) and pins the §3 fabric + SAP-rating
block at abs=1e-4: (29a) walls 74.5800, (30) roof 38.5008, (33) fabric
172.7844, continuous SAP 73.2332 = (258), CO2 3518.3037 = (272). Guards the
detailed-RR slope/common_wall surfaces, the dry-lining R=0.17 adjustment,
and the per-part mixed-roof billing together. Summary mirrored to
backend/documents_parser/tests/fixtures/Summary_001431_case43.pdf; provider
module mirrors the _case6/_case21 pattern, assertion in
test_section_cascade_pins. Harness 47/47; regression = the 3 pre-existing
fails; pyright net-zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two compensating Summary-extractor bugs surfaced by simulated case 43 (a
2-BP mid-terrace with a detailed room-in-roof + a dry-lined extension wall).
Their fabric errors nearly cancelled (walls net −0.76 W/K), hiding both
behind a deceptively small +0.05 SAP delta.
Bug 1 — main/extension wall dry-lining never read. The §7 "Dry-lining:
Yes/No" line was parsed only for ALTERNATIVE walls; the main/extension
WallDetails dropped it, so a dry-lined solid wall was billed at its
un-adjusted base U. RdSAP 10 §5.8 + Table 14: a dry-lined uninsulated wall
adds R=0.17 → U = 1/(1/U_base + 0.17). Case 43 Ext1: solid brick 1.70 →
1.32. Added `WallDetails.dry_lined`, read it in the extractor (both the
main-wall builder and the As-Main copy), threaded it to the domain
`wall_dry_lined` (emit None when undried — cascade-equivalent to False,
keeps the field absent for the non-dry-lined majority).
Bug 2 — the LAST room-in-roof surface row's U over-read. The per-row token
scan stops at the next RIR-row name; the final surface (no successor) over-
read into the following section, shifting the trailing-token slotting and
silently zeroing its `default_u` (case 43 Common Wall 2: 1.90 → 0.00 → the
2.4 m² common wall billed at U=0 instead of the main-wall 1.90). Stop the
scan at the row's natural end — the "Yes"/"No" u_value_known flag plus the
trailing u_value numeric.
Case 43 now reproduces the P960 EXACTLY: (29a) walls 74.5800, (33) fabric
172.7844, continuous SAP 73.2332 = (258), CO2 3518.30 = (272), all <1e-4
(was SAP +0.0455 / CO2 −8.04). Harness 47/47 0 raised; regression = the 3
pre-existing fails; pyright net-zero (51=51).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-on to the slope/stud slice. A Detailed RR (RdSAP 10 §3.9.2) can also
lodge `common_wall_*` — the wall separating the room-in-roof from the rest of
the cold roof void. Those fields were undeclared → `from_dict` dropped them →
`_api_rir_detailed_surfaces` omitted the common walls → the RR undercounted
wall heat loss → over-rate.
Fix: declare `common_wall_length/height_1/2` on `RoomInRoofDetails`
(21_0_0 + 21_0_1) and build `kind="common_wall"` surfaces (raw L × H area to
2 d.p.). The cascade's Detailed-RR branch already bills common walls at the
storey-below main-wall U (Table 4 p.22 "Common wall") and deducts their area
from the §3.10.1 residual roof — no calculator change. No insulation thickness
is read: common walls take the main-wall U, not a Table 17 RR-element U.
6 /tmp certs carry detailed `common_wall_length_1`: cohort mean|err| 2.43 ->
1.25 (all were over-rating; e.g. 2877-3059 +4.55 -> +2.79). Gauges: corpus
within-0.5 67.5% -> 67.6% (MAE 0.987 -> 0.979); /tmp 71.6% -> 71.7%
(MAE 0.846 -> 0.838). Harness 47/47 0 raised; regression = the 3 pre-existing
fails; pyright net-zero (65=65).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lock in the detailed-RR slope + stud-wall gain (corpus within-0.5
67.3% -> 67.5%, MAE 1.020 -> 0.987). The corpus is a fixed 1000-cert
deterministic gauge, so the thresholds track measured HEAD with a small
margin per the ratchet convention.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gov-EPC API lodges a Detailed RR (RdSAP 10 §3.9, Figure 4) with up to
two sloping ceilings (`slope_*`) and two vertical stud/knee walls
(`stud_wall_*`) in addition to the gable + flat-ceiling surfaces. Those
slope/stud fields were undeclared on the 21.0.x schema, so `from_dict`
silently dropped them and `_api_rir_detailed_surfaces` built ONLY the gable
+ flat-ceiling surfaces. The (large) sloping roof and the knee walls
contributed ZERO heat loss → undercounted RR fabric loss → a systematic
over-rate.
Fix: declare `slope_*`/`stud_wall_*` on `RoomInRoofDetails`
(rdsap_schema_21_0_0 + _21_0_1) and build `kind="slope"` / `kind="stud_wall"`
surfaces in the mapper. The cascade's Detailed-RR branch already routes both
to the roof aggregate via `u_rr_slope` (Table 17 col 1) and `u_rr_stud_wall`
(Table 17 col 3) — RdSAP 10 §5.11.3, p.43-44 — so no calculator change is
needed (Summary path worksheet-validated by the 000565 detailed-RR fixtures).
insulation_type is left None to defer to the Table 17 col-(a) mineral-wool
default, mirroring the existing flat_ceiling branch.
15 /tmp certs carry `slope_height_1`: cohort mean|err| 4.26 -> 2.05, signed
+4.09 -> centred (14/15 were over-rating; e.g. 0390-2538 +5.95 -> +3.56).
Gauges: corpus within-0.5 67.3% -> 67.5% (MAE 1.020 -> 0.987); /tmp 71.4% ->
71.6% (MAE 0.882 -> 0.846). Harness 47/47 0 raised; regression = the 3
pre-existing fails; pyright net-zero (65=65).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gov API lodges a manufacturer's declared cylinder loss factor
(kWh/day) in `sap_heating.cylinder_heat_loss`, in which case it leaves
the cylinder volume / insulation type / thickness None. That field was
undeclared on the 21.0.x schemas, so `from_dict` dropped it — then
`_cylinder_storage_loss_override` hit its insulation-None / volume-None
guards and returned None, dropping the §4 storage loss ENTIRELY. The
dwelling over-rated (the declared loss is typically ~1.5 kWh/day ≈
550 kWh/yr).
SAP 10.2 §4 branch a) (PDF p.136): when the declared loss factor is
known, storage loss (50) = (48) declared loss × (49) Table-2b
temperature factor — replacing the Table 2 V×L×VF computation.
- declare `cylinder_heat_loss` on RdSapSchema21_0_0/21_0_1.SapHeating +
EpcPropertyData.SapHeating; thread through the 21.0.x mappers.
- `cylinder_storage_loss_monthly_kwh` gains `declared_loss_kwh_per_day`:
when set, combined_55 = declared × TF (volume/insulation unused).
- `_cylinder_storage_loss_override` resolves the declared loss BEFORE the
insulation/volume guards (the gov omits those when the loss is lodged).
12 /tmp certs carry it (mean |err| 3.00 -> 2.51; the clean ones close
hard, e.g. 2360 2.65 -> 0.30, 0245 2.25 -> 0.53). Corpus within-0.5
67.0% -> 67.3% (MAE 1.025 -> 1.020); /tmp 71.2% -> 71.4% (0.889 ->
0.882). Worksheet harness 47/47; regression = only the 3 pre-existing
fails; pyright net-zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Roofs lodged insulated at rafters carry their thickness in a DEDICATED
gov-EPC API field, `rafter_insulation_thickness` (e.g. "225mm"), while
`roof_insulation_thickness` stays None (rafters aren't loft joists). That
field was undeclared on the 21.0.x schemas, so `from_dict` silently
dropped it — the rafter certs only *looked* redacted (roof EER 2-4 =
insulated, yet no thickness), and the cascade fell to the Table 18 col (2)
unknown default (2.30), badly under-rating them.
- declare `rafter_insulation_thickness` on RdSapSchema21_0_0/21_0_1 +
EpcPropertyData.SapBuildingPart (mirrors the existing
sloping_ceiling_/flat_roof_insulation_thickness dropped-field handling).
- thread it through `from_rdsap_schema_21_0_0/21_0_1` (older schemas get
None via getattr).
- `heat_transmission` prefers `rafter_insulation_thickness` over
`roof_insulation_thickness` when the part is at-rafters, so the measured
RdSAP 10 §5.11.2 Table 16 column (2) row applies (225 mm → 0.25).
Completes the rafters roof fix: with the real thickness read, the rafter
certs are recovered rather than over-stated — cert 3100-8675-0922-8628
(band E, rafters 225mm) +8.93 → +0.43 SAP. Corpus within-0.5 67.0%
(MAE 1.025) and /tmp 71.2% (MAE 0.889) — both NET ABOVE the pre-rafters
baseline (66.9% / 70.6%). Worksheet harness 47/47; regression = only the
3 pre-existing fails; pyright net-zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`u_roof` only implemented the joist column, so roofs lodged insulated at
rafters (`roof_insulation_location == 1`) were mis-billed at the joist U
on both the API and Summary paths — under-stating loss, over-rating SAP.
RdSAP 10 §5.11.2 Table 16 (spec p.42-43) gives a distinct "insulation at
rafters" column (2): the rafter cavity is shallower than a loft void, so
the same depth yields a higher U (200 mm: rafters 0.29 vs joists 0.21).
§5.11 Table 18 (p.45) likewise carries a rafters column (2) for unknown /
as-built thickness (footnote (1): "The value from the table applies for
unknown and as built") — band A-D = 2.30, E = 1.50, F = 0.68, diverging
from the joist column's 100 mm-equivalent 0.40 default (footnote (4)).
- add `_ROOF_RAFTERS_BY_THICKNESS` (Table 16 col 2) + `_ROOF_RAFTERS_BY_AGE`
(Table 18 col 2) to rdsap_uvalues; `u_roof` selects them via a new
`insulation_at_rafters` flag (ignored for flat / sloping-ceiling roofs).
- `heat_transmission` derives the flag PER BUILDING PART from
`roof_insulation_location` (gov-API int 1 / Summary "R Rafters"), which
also fixes the multi-part dedup-roof-join problem: each part's own
location now drives its U, replacing the unattributable joined
`epc.roofs[]` description.
Worksheet-validated to 1e-4: simulated case 41 (4-bp — Ext1 rafters 200mm
→ 0.29, Ext3 rafters As-Built band F → 0.68; roof total 24.8350) and case
42 (6 variants — rafters 50mm → 0.88, rafters unknown band C → 2.30,
joists/none unchanged). Case 40 stays exact (roof 35.340, total 441.1606);
worksheet harness 47/47.
Corpus within-0.5 66.9% → 66.5% (gates 0.65/1.08 hold) — a spec-correct
shift, NOT a regression: all 15 corpus rafter certs carry redacted (None)
thickness yet lodge roof EER 2-4 (insulated), so the open API blanked a
specified thickness and the spec's unknown-rafter 2.30 default correctly
over-states them. Recovery needs a roof-EER→thickness inference on the
API path (follow-up), not a change to the U-table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
README at domain/epc_prediction/README.md — the flow diagram, where each piece
lives, links to the ADRs/CONTEXT/handover/migration note, and a runnable test
command. The team's entry point.
tests/e2e/test_epc_prediction_e2e.py — the whole gap-fill flow against the REAL
Postgres Unit of Work + EPC/Property repositories + EpcComparablePropertiesRepository
+ EpcPrediction, with only the three external HTTP clients faked (EPC API,
geospatial S3, Solar). Proves: EPC-less Property → Ingestion predicts from its
postcode cohort → persists to the predicted slot → reloaded Property resolves
effective_epc via source_path == "predicted". The canonical "see it in action".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gap-fill is wired end-to-end (slices 5a-5e) behind seams; this note is what's
left to switch it on in production: (1) implement the PredictionTargetAttributesReader
stub over property_overrides — with the override-value → API-code mapping
select_comparables needs; (2) run the epc_property.source Drizzle migration; (3)
pass the three optional collaborators at the IngestionOrchestrator composition
root. Plus the open Validation-Cohort exclusion (no code path exists yet — exclude
on source_path == "predicted" when one is built) and the anomaly dual-use pointer.
No code change: the validation exclusion has no consumer to attach to today, and
the structural signal (source_path == "predicted") already exists from slice-5a.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire EPC Prediction gap-fill into IngestionOrchestrator (ADR-0031). When the
predictor collaborators are injected (ComparablesRepo + PredictionAttributesReader
+ EpcPrediction), an EPC-less Property is predicted from its postcode cohort and
persisted to the predicted slot; the eligibility gate (unknown property_type) and
"a lodged EPC is never predicted over" both hold. The two-phase contract is kept:
prediction attributes (Landlord Overrides) resolve in the unit prep phase, the
cohort fetch + select + predict run in the no-unit IO phase, persistence in the
write phase. All three collaborators are OPTIONAL — unwired, ingestion behaves
exactly as before (existing tests unchanged).
3 tests (predict+persist, gate, lodged-wins); 228 pass across orchestration +
epc_prediction + repositories; pyright strict clean. Production composition-root
wiring (real ComparableProperties + override-attributes adapters) is part of the
Jun-te handover.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
build_prediction_target assembles an EPC-less Property's PredictionTarget from
its identity (postcode), resolved coordinates, and Landlord-Override attributes
(property_type / built_form / wall_construction). The eligibility GATE: a Property
whose property_type is unknown returns None — never sized from a mixed-type
cohort (ADR-0031). property_type is the hard cohort filter.
The override attributes are read through a PredictionTargetAttributesReader port
(stub seam) — the real adapter (a read over property_overrides) is being built
separately by the team; ingestion wiring depends on the abstraction and tests
substitute a fake. 2 tests (assembly + gate); pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `source` discriminator (lodged | predicted) to the EPC store so a Property
holds a lodged EPC and a predicted one (EPC Prediction gap-fill) at once
(ADR-0031). EpcRepository.save gains source="lodged"; idempotent delete is now
per-source (a predicted save no longer wipes lodged, and vice versa);
get_for_property/get_for_properties filter lodged; new get_predicted_for_property
/ get_predicted_for_properties read predicted. PropertyPostgresRepository.get +
get_many hydrate Property.predicted_epc, so the predicted picture reaches the
modelling read (both load via get_many). FakeEpcRepo mirrors the dual slot.
EpcPropertyModel gains `source` (default "lodged"); the test DB builds from the
SQLModel mirror so this is exercised without the prod migration. The matching
Drizzle change (column + per-(property_id,source) uniqueness) is the team's to
action before merge — docs/MIGRATION_NOTE_predicted_epc_source.md.
3 store tests (coexist, idempotent predicted re-save leaves lodged, lodged-only
has no predicted) + property-repo wiring; 85 pass across affected suites; new
code pyright-clean (2 pre-existing wwhrs errors in epc_property_table untouched).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build the cohort IO port ADR-0029 deferred (ADR-0031 slice-5b):
`ComparablePropertiesRepository.candidates_for(postcode) -> list[Comparable]`,
with an EPC-API + geospatial adapter that lists the postcode's lodged certs
(search_by_postcode), fetches + maps each (get_by_certificate_number), and
resolves their UPRNs to coordinates in ONE batched read. Register metadata the
cert doesn't carry (address, registration date) is threaded off the search row;
a UPRN-less or unparseable-date cert is kept, just uncoordinated / unweighted.
The domain select_comparables then filters these candidates into the cohort.
Thin CohortEpcClient / CohortGeospatial Protocols keep the adapter testable
against fakes; EpcClientService + GeospatialS3Repository satisfy them
structurally (no changes). 3 tests; pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `predicted_epc` slot to the Property aggregate and a "predicted" branch to
SourcePath / source_path / effective_epc (ADR-0031 decisions 1+3). A
neighbour-synthesised EpcPropertyData resolves as the Effective EPC ONLY when
there is neither a lodged EPC nor Site Notes — a real source always wins
(prediction is last-resort gap-fill). The slot is distinct from `epc` so a
predicted picture coexists with any lodged one (provenance is structural, not a
flag on EpcPropertyData); downstream consumers are untouched.
3 tests: predicted resolves when sole source; lodged EPC wins over predicted;
Site Notes win over predicted. 10/10 green, pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve the slice-5 design tree (grill-with-docs): estimation runs in Ingestion
(refines ADR-0029 dec-3; drops the #1227 "shift to Modelling" — no surviving
rationale, and stages communicate only via persisted state); predicted EPC is
persisted in a DISTINCT slot (EPC table + source discriminator) so lodged +
predicted coexist (enables EPC Anomaly Flags); provenance is structural (the
slot), not a field on EpcPropertyData; effective_epc/source_path gain a
"predicted" branch; slice-5 is gap-fill only; property_type is a REQUIRED input
(hard cohort filter) from Landlord Overrides, and Properties with unknown type
are gated out (no national defaults). OS postcode_search as a broader type
source is a noted follow-on. CONTEXT EPC Prediction entry gains the gating rule.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build a geographically DENSE postcode-clustered corpus to test cross-postcode
geo expansion (the handover's anticipated "real geo payoff"). The gov EPC API
has no area/prefix search (a partial postcode 400s; the old opendatacommunities
partial-search API is decommissioned), so neighbourhood enumeration is external:
seed K postcodes nationally, expand each via postcodes.io's nearest-postcode
endpoint into every unit within RADIUS_M, pull each one's full EPC cohort.
postcodes.io is a corpus-BUILD dependency only — the predictor stays pure. Same
on-disk layout as the scattered corpus, so load_corpus + the coords resolver
consume it unchanged.
MEASURE-FIRST RESULT — cross-postcode expansion is a NO-GO. On a 2-seed pilot
(York YO19 + Islington N51, 81 postcodes / 1558 certs, 140 SAP-10.2 targets),
pooling nearby postcodes regresses accuracy across the board:
same-postcode FA_MAE 9.53 wall 92% age 72% floor_con 85% cylinder 91%
cross <=0.3km FA_MAE 13.1 wall 80% age 61% floor_con 82% cylinder 79%
Even as a thin-cohort top-up it hurts (thin n=18: FA 5.24 -> 7.15). Root cause:
the postcode boundary is itself a strong homogeneity prior (a postcode is one
coherent street/development), so same-postcode neighbours beat geographically
near cross-boundary ones even when the home postcode is sparse (and they rarely
are — median same-postcode cohort here is 34). Geo-proximity helps WITHIN a
postcode (#1227) but does not survive crossing the boundary. Cross-postcode geo
closed; geo weighting stays intra-postcode. Tooling kept (reusable).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Size the predicted dwelling from the geo-proximity-weighted median of the
cohort's floor areas rather than the plain median: homes built together share a
footprint, so a nearer neighbour's area should count for more (the same street
signal #1227 already wired into age / wall / glazing). Reuses `_geo_weights` and
adds `_weighted_median`, which reduces exactly to `statistics.median` under
uniform weights (geo off / no target coordinates) — including the even-count
midpoint average — so the MAD-minimising guarantee is preserved.
Measured over the 514-target SAP-10.2 corpus (leave-one-out):
floor_area MAE 10.48 -> 9.73 m² MAPE 13.2% -> 12.2%
Re-baselines the n=36 fixture floor_area ceiling 11.8983 -> 12.0378 (a method
change, not a loosening; the small fixture subset moved +0.14 the other way as
sample noise while the population improved decisively). The ceiling still pins
the new deterministic value exactly, so the tighten-only ratchet resumes.
Investigation ruling out the adjacent floor-area levers (kept in the follow-up):
lowering minimum_cohort (9.78-10.03, worse), hard same-form filter (10.19),
mean instead of median (10.68), constant bias correction (10.47),
extension-conditioning (oracle 9.50, not worth the misclassification cost) and
room-in-roof conditioning/additive (RiR is a confound for large multi-part
outliers — RiR area is only ~21% of total, and the increment breaks the homes
already predicted exactly). Remaining cohort lever is built-form soft-weighting,
gated on a denser corpus.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Elmhurst Summary §15.1 lodges "Cylinder Size: Value known" with the
measured volume in the "Cylinder Volume (l)" line — the Summary-path
equivalent of the gov-API "Exact" descriptor. The mapper had no entry for
"Value known" so `_elmhurst_cylinder_size_code` raised UnmappedElmhurstLabel,
and even once mapped the measured volume was never threaded through, so the
cascade dropped the cylinder storage loss (~468 kWh/yr) from (219) water
heating on every measured-volume-cylinder Summary.
Per RdSAP 10 §10.5 Table 28 (p.55) a measured cylinder volume is used
directly. Map "Value known" → cascade code 6 (Exact) and thread the §15.1
"Cylinder Volume (l)" value into SapHeating.cylinder_volume_measured_l, which
`_cylinder_volume_l_from_code` (cert_to_inputs.py:5281) already reads for
code 6 — mirroring the gov-API path (mapper.py:1575/1885).
Pins simulated case 39 (P960-0001-001431): an age-A mid-terrace on direct-
acting electric room heaters (SAP code 691, cat 10, control 2602) with
electric-immersion DHW off a 117 L "Value known" cylinder. The full
extractor→mapper→calculator cascade now reproduces the worksheet's SAP-rating
block EXACTLY — SAP value 36.6365 (band F) and (272) CO2 2056.0731 kg/yr,
with (219) water heating 2637.5049 and (255) total energy cost 1802.0039.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a floor_area line giving MAE (m2), MAPE (% of actual), and the typical
(median actual) size, so the absolute error reads relative to dwelling size.
Corpus: MAE 10.48 m2 / MAPE 13.2% / typical 61 m2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
main's 'ND' multiple_glazing_type mapper fix (361abc12) changes the mapped
ground-truth glazing for one fixture cert, so modal_glazing_type re-baselines
0.5833 -> 0.5556 (21/36 -> 20/36). A mapper change shifts the deterministic
fixture rates like a fixture change does — re-baseline, not a prediction
regression. All other component floors + residual ceilings unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Folds a haversine distance kernel into the categorical-mode weighting so a
nearer neighbour counts for more — applied ONLY to the components that showed
a clear distance signal in the corpus pre-check (age band, wall + floor
construction, glazing: homes built/retrofitted together cluster). Roof
construction showed no decay and is excluded; heating keeps its coherent
donor. Predictor stays pure: weights come from target.coordinates vs each
Comparable.coordinates (resolved at the boundary); geo is OFF when the target
has no coords, neutral for a neighbour with none.
Scale chosen on the harness: _GEO_SCALE_KM=0.1 is the gate-safe optimum
(0.05 lifts the corpus more but regresses fixture floor_construction).
Corpus (150pc/514, geo off->on): age 0.564->0.572, age_pm1 0.841->0.847,
wall 0.902->0.912, floor_con 0.786->0.796, glazing 0.667->0.673; roof
unchanged. Fixture: glazing 0.5278->0.5833 (floor ratcheted), all else held.
Refactored recency into a reusable _recency_weights vector composed via
_combine, so similarity/recency/geo factors multiply uniformly. Fixture ships
a committed _coordinates.json (OGL OS OpenData; build script carries it from
the corpus sidecar on rebuild) so the gate exercises geo without S3.
This is the per-component method applied to geography ([[feedback_per_component_best_method]]).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds coordinates: Optional[Coordinates] to Comparable and PredictionTarget
(data carriers — the pure predictor stays IO-free), and wires load_corpus to
read an optional _coordinates.json sidecar ({uprn: [lon, lat]}) and populate
each Comparable from its cert's uprn; iter_predictions threads the held-out
target's coordinates through. Absent sidecar -> geo-weighting stays off (no
behaviour change yet — weighting lands next slice). fetch_corpus_coordinates
now writes the sidecar into the corpus dir. load_corpus populates 99% of
corpus comparables.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds GeospatialRepository.coordinates_for_uprns(uprns) -> dict — a batch
coordinate lookup returning only covered UPRNs. The S3 adapter overrides it
to read the meta once, group UPRNs by their covering partition, and read each
partition once for all the UPRNs it covers; co-located (closely-numbered)
UPRNs share a partition, so an EPC Prediction cohort is typically one or two
reads instead of one per neighbour. Default port impl is a per-UPRN loop.
Feeds the EPC Prediction geo-proximity work: a cohort's UPRNs resolve to
coordinates in a couple of reads (validated at corpus scale: 170 partition
reads for 2683 UPRNs).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One-time utility: resolves every corpus cert's uprn -> WGS84 lon/lat from the
OS Open-UPRN parquet (DATA_BUCKET/spatial/) via boto3, grouping UPRNs by their
covering partition so each ~1.7MB partition is read at most once (the efficient
batch lookup we intend to add to GeospatialRepository). Caches {uprn:[lon,lat]}
locally for the validation harness. Resolved 2609/2683 corpus UPRNs (97%).
Signal pre-check result (does intra-postcode proximity predict components?):
intra-postcode distances are non-trivial (median 44m, p90 138m, max ~1km),
and nearer neighbours match the target markedly better on age band (0.63 at
<20m -> 0.16 at >300m), wall, glazing and floor construction. Roof shows no
decay. => geo-proximity is worth building, per-component (strongest for age,
the weakest fabric component).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
