Routes run_modelling through prop.effective_epc and dumps each target's
property_overrides before the run, so a landlord wall override moves the
calculated SAP. Records the overlay design in ADR-0032.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update run_modelling_e2e's docstring so another dev can run it: the Scenario's
exclusions drive measure scoping (--measures/--exclude-measures are overlays),
and flag the secondary_heating_removal catalogue gap that currently requires
--exclude-measures. Replace the stale --measures examples with the real
scenario-driven inspect/persist commands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The measures a run considers should come from the Scenario, not a CLI flag.
The live scenario table persists exclusions only (no inclusions column), as a
Postgres text-array of exact MeasureType values.
- Scenario gains `exclusions: frozenset[MeasureType]` + `considered_measures()`
(all measures minus the excluded ones, or None when none are excluded).
- ScenarioModel.to_domain parses the `{a,b,c}` exclusions array into
MeasureTypes, raising on a token that is not an exact MeasureType value
(no high-level category expansion), per the strict-enum convention.
- ModellingOrchestrator._plan_for derives the allowlist from the Scenario's
exclusions, combined (intersection) with any explicit considered_measures
override via the new `combine_considered_measures`.
- run_modelling_e2e sources the allowlist from the Scenario; --measures /
--exclude-measures become optional overlays (e.g. the technical
secondary_heating_removal exclusion the catalogue cannot yet stock).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The run only showed the measures the Optimiser selected, so a candidate it
passed over (e.g. an ASHP it found too costly for the target band) and that
measure's cost were invisible.
Add `harness.console.candidate_recommendations` — every Generator Option
with its per-Option cost, before optimisation — and have run_modelling_e2e
print the full menu per property (flagging the selected Options), write a
"cost per measure" section into the markdown, and emit a per-Option
modelling_e2e_candidates.csv.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`restrict_to_considered_measures` filtered candidates only *after* every
generator had run, so an excluded measure still queried the catalogue.
That crashed properties with a lodged secondary heater: the live
`material.type` enum has no `secondary_heating_removal` value, so the
query raised a psycopg2 `InvalidTextRepresentation` before the allowlist
could drop it.
`_candidate_recommendations` now pairs each generator with the measure
types it can emit and runs it only when the allowlist admits one of them
(None = all), so an excluded measure never reaches the catalogue.
`restrict_to_considered_measures` still trims disallowed Options off the
multi-Option survivors. Add `--exclude-measures` to run_modelling_e2e
(allowlist minus the excluded set) for excluding one measure without
enumerating the rest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ASHP bundle is priced from the rate sheet (ADR-0025); the catalogue
row is read only for its material id, which is nullable end-to-end. The
live `material` catalogue has no `air_source_heat_pump` row, so
`products.get` raised `ValueError: no active product` and aborted every
ASHP-eligible property.
Add `ProductNotFound(ValueError)` + a concrete `ProductRepository
.get_optional`, raise the typed error from both repos, and have
`_ashp_option` look the row up optionally — a missing row now yields an
ASHP Option with `material_id=None` rather than crashing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
build_first_run_pipeline now constructs epc_prediction=EpcPrediction() and accepts
comparables_repo + prediction_attributes_reader as optional params, threading them
into IngestionOrchestrator (ADR-0031). The on-switch is now just supplying those
two arguments — no orchestrator/handler edits — once they exist: the cohort repo
(its EPC client is the source client pending #1136) and the property_overrides
attributes reader (built separately). Both default None, so the feature stays OFF
and ingestion is unchanged until they're passed.
The epc_property.source migration is live, so the predicted-EPC persistence slot
(slice-5c) now works against the real DB. Handover updated to reflect the simpler
composition-root step.
pyright strict clean; handler + pipeline + ingestion-prediction tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two review points from @dancafc:
1) Rename the `Comparable` dataclass → `ComparableProperty` (it models one
comparable *property*; the collection stays `ComparableProperties`). Applied
across domain, repositories, orchestration, harness, scripts, and tests with a
word-boundary rename so `ComparableProperties` is untouched.
2) Move `PredictionTarget` out of comparable_properties.py into prediction_target.py
(where `PredictionTargetAttributes` + `build_prediction_target` already live).
comparable_properties.py now imports it; no import cycle (prediction_target no
longer depends on comparable_properties). Importers updated.
92 tests pass across the touched suites; pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Golden regression fixture for the multi-feature dwelling that surfaced the
two Elmhurst-extractor bugs in a33707f8. case 43 is a 2-BP mid-terrace with
a DETAILED room-in-roof (two slopes, two flat ceilings, party + exposed
gables, two common walls), a MIXED-insulation multi-section roof (Main
insulated + Extension uninsulated 2.30), a DRY-LINED extension solid wall,
a mains-gas boiler (102 / control 2106) and a House-coal solid-fuel
secondary (633).
Routes the Summary PDF through the WHOLE extractor + mapper + calculator
pipeline (no hand-built EpcPropertyData) and pins the §3 fabric + SAP-rating
block at abs=1e-4: (29a) walls 74.5800, (30) roof 38.5008, (33) fabric
172.7844, continuous SAP 73.2332 = (258), CO2 3518.3037 = (272). Guards the
detailed-RR slope/common_wall surfaces, the dry-lining R=0.17 adjustment,
and the per-part mixed-roof billing together. Summary mirrored to
backend/documents_parser/tests/fixtures/Summary_001431_case43.pdf; provider
module mirrors the _case6/_case21 pattern, assertion in
test_section_cascade_pins. Harness 47/47; regression = the 3 pre-existing
fails; pyright net-zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two compensating Summary-extractor bugs surfaced by simulated case 43 (a
2-BP mid-terrace with a detailed room-in-roof + a dry-lined extension wall).
Their fabric errors nearly cancelled (walls net −0.76 W/K), hiding both
behind a deceptively small +0.05 SAP delta.
Bug 1 — main/extension wall dry-lining never read. The §7 "Dry-lining:
Yes/No" line was parsed only for ALTERNATIVE walls; the main/extension
WallDetails dropped it, so a dry-lined solid wall was billed at its
un-adjusted base U. RdSAP 10 §5.8 + Table 14: a dry-lined uninsulated wall
adds R=0.17 → U = 1/(1/U_base + 0.17). Case 43 Ext1: solid brick 1.70 →
1.32. Added `WallDetails.dry_lined`, read it in the extractor (both the
main-wall builder and the As-Main copy), threaded it to the domain
`wall_dry_lined` (emit None when undried — cascade-equivalent to False,
keeps the field absent for the non-dry-lined majority).
Bug 2 — the LAST room-in-roof surface row's U over-read. The per-row token
scan stops at the next RIR-row name; the final surface (no successor) over-
read into the following section, shifting the trailing-token slotting and
silently zeroing its `default_u` (case 43 Common Wall 2: 1.90 → 0.00 → the
2.4 m² common wall billed at U=0 instead of the main-wall 1.90). Stop the
scan at the row's natural end — the "Yes"/"No" u_value_known flag plus the
trailing u_value numeric.
Case 43 now reproduces the P960 EXACTLY: (29a) walls 74.5800, (33) fabric
172.7844, continuous SAP 73.2332 = (258), CO2 3518.30 = (272), all <1e-4
(was SAP +0.0455 / CO2 −8.04). Harness 47/47 0 raised; regression = the 3
pre-existing fails; pyright net-zero (51=51).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-on to the slope/stud slice. A Detailed RR (RdSAP 10 §3.9.2) can also
lodge `common_wall_*` — the wall separating the room-in-roof from the rest of
the cold roof void. Those fields were undeclared → `from_dict` dropped them →
`_api_rir_detailed_surfaces` omitted the common walls → the RR undercounted
wall heat loss → over-rate.
Fix: declare `common_wall_length/height_1/2` on `RoomInRoofDetails`
(21_0_0 + 21_0_1) and build `kind="common_wall"` surfaces (raw L × H area to
2 d.p.). The cascade's Detailed-RR branch already bills common walls at the
storey-below main-wall U (Table 4 p.22 "Common wall") and deducts their area
from the §3.10.1 residual roof — no calculator change. No insulation thickness
is read: common walls take the main-wall U, not a Table 17 RR-element U.
6 /tmp certs carry detailed `common_wall_length_1`: cohort mean|err| 2.43 ->
1.25 (all were over-rating; e.g. 2877-3059 +4.55 -> +2.79). Gauges: corpus
within-0.5 67.5% -> 67.6% (MAE 0.987 -> 0.979); /tmp 71.6% -> 71.7%
(MAE 0.846 -> 0.838). Harness 47/47 0 raised; regression = the 3 pre-existing
fails; pyright net-zero (65=65).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lock in the detailed-RR slope + stud-wall gain (corpus within-0.5
67.3% -> 67.5%, MAE 1.020 -> 0.987). The corpus is a fixed 1000-cert
deterministic gauge, so the thresholds track measured HEAD with a small
margin per the ratchet convention.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gov-EPC API lodges a Detailed RR (RdSAP 10 §3.9, Figure 4) with up to
two sloping ceilings (`slope_*`) and two vertical stud/knee walls
(`stud_wall_*`) in addition to the gable + flat-ceiling surfaces. Those
slope/stud fields were undeclared on the 21.0.x schema, so `from_dict`
silently dropped them and `_api_rir_detailed_surfaces` built ONLY the gable
+ flat-ceiling surfaces. The (large) sloping roof and the knee walls
contributed ZERO heat loss → undercounted RR fabric loss → a systematic
over-rate.
Fix: declare `slope_*`/`stud_wall_*` on `RoomInRoofDetails`
(rdsap_schema_21_0_0 + _21_0_1) and build `kind="slope"` / `kind="stud_wall"`
surfaces in the mapper. The cascade's Detailed-RR branch already routes both
to the roof aggregate via `u_rr_slope` (Table 17 col 1) and `u_rr_stud_wall`
(Table 17 col 3) — RdSAP 10 §5.11.3, p.43-44 — so no calculator change is
needed (Summary path worksheet-validated by the 000565 detailed-RR fixtures).
insulation_type is left None to defer to the Table 17 col-(a) mineral-wool
default, mirroring the existing flat_ceiling branch.
15 /tmp certs carry `slope_height_1`: cohort mean|err| 4.26 -> 2.05, signed
+4.09 -> centred (14/15 were over-rating; e.g. 0390-2538 +5.95 -> +3.56).
Gauges: corpus within-0.5 67.3% -> 67.5% (MAE 1.020 -> 0.987); /tmp 71.4% ->
71.6% (MAE 0.882 -> 0.846). Harness 47/47 0 raised; regression = the 3
pre-existing fails; pyright net-zero (65=65).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gov API lodges a manufacturer's declared cylinder loss factor
(kWh/day) in `sap_heating.cylinder_heat_loss`, in which case it leaves
the cylinder volume / insulation type / thickness None. That field was
undeclared on the 21.0.x schemas, so `from_dict` dropped it — then
`_cylinder_storage_loss_override` hit its insulation-None / volume-None
guards and returned None, dropping the §4 storage loss ENTIRELY. The
dwelling over-rated (the declared loss is typically ~1.5 kWh/day ≈
550 kWh/yr).
SAP 10.2 §4 branch a) (PDF p.136): when the declared loss factor is
known, storage loss (50) = (48) declared loss × (49) Table-2b
temperature factor — replacing the Table 2 V×L×VF computation.
- declare `cylinder_heat_loss` on RdSapSchema21_0_0/21_0_1.SapHeating +
EpcPropertyData.SapHeating; thread through the 21.0.x mappers.
- `cylinder_storage_loss_monthly_kwh` gains `declared_loss_kwh_per_day`:
when set, combined_55 = declared × TF (volume/insulation unused).
- `_cylinder_storage_loss_override` resolves the declared loss BEFORE the
insulation/volume guards (the gov omits those when the loss is lodged).
12 /tmp certs carry it (mean |err| 3.00 -> 2.51; the clean ones close
hard, e.g. 2360 2.65 -> 0.30, 0245 2.25 -> 0.53). Corpus within-0.5
67.0% -> 67.3% (MAE 1.025 -> 1.020); /tmp 71.2% -> 71.4% (0.889 ->
0.882). Worksheet harness 47/47; regression = only the 3 pre-existing
fails; pyright net-zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Roofs lodged insulated at rafters carry their thickness in a DEDICATED
gov-EPC API field, `rafter_insulation_thickness` (e.g. "225mm"), while
`roof_insulation_thickness` stays None (rafters aren't loft joists). That
field was undeclared on the 21.0.x schemas, so `from_dict` silently
dropped it — the rafter certs only *looked* redacted (roof EER 2-4 =
insulated, yet no thickness), and the cascade fell to the Table 18 col (2)
unknown default (2.30), badly under-rating them.
- declare `rafter_insulation_thickness` on RdSapSchema21_0_0/21_0_1 +
EpcPropertyData.SapBuildingPart (mirrors the existing
sloping_ceiling_/flat_roof_insulation_thickness dropped-field handling).
- thread it through `from_rdsap_schema_21_0_0/21_0_1` (older schemas get
None via getattr).
- `heat_transmission` prefers `rafter_insulation_thickness` over
`roof_insulation_thickness` when the part is at-rafters, so the measured
RdSAP 10 §5.11.2 Table 16 column (2) row applies (225 mm → 0.25).
Completes the rafters roof fix: with the real thickness read, the rafter
certs are recovered rather than over-stated — cert 3100-8675-0922-8628
(band E, rafters 225mm) +8.93 → +0.43 SAP. Corpus within-0.5 67.0%
(MAE 1.025) and /tmp 71.2% (MAE 0.889) — both NET ABOVE the pre-rafters
baseline (66.9% / 70.6%). Worksheet harness 47/47; regression = only the
3 pre-existing fails; pyright net-zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`u_roof` only implemented the joist column, so roofs lodged insulated at
rafters (`roof_insulation_location == 1`) were mis-billed at the joist U
on both the API and Summary paths — under-stating loss, over-rating SAP.
RdSAP 10 §5.11.2 Table 16 (spec p.42-43) gives a distinct "insulation at
rafters" column (2): the rafter cavity is shallower than a loft void, so
the same depth yields a higher U (200 mm: rafters 0.29 vs joists 0.21).
§5.11 Table 18 (p.45) likewise carries a rafters column (2) for unknown /
as-built thickness (footnote (1): "The value from the table applies for
unknown and as built") — band A-D = 2.30, E = 1.50, F = 0.68, diverging
from the joist column's 100 mm-equivalent 0.40 default (footnote (4)).
- add `_ROOF_RAFTERS_BY_THICKNESS` (Table 16 col 2) + `_ROOF_RAFTERS_BY_AGE`
(Table 18 col 2) to rdsap_uvalues; `u_roof` selects them via a new
`insulation_at_rafters` flag (ignored for flat / sloping-ceiling roofs).
- `heat_transmission` derives the flag PER BUILDING PART from
`roof_insulation_location` (gov-API int 1 / Summary "R Rafters"), which
also fixes the multi-part dedup-roof-join problem: each part's own
location now drives its U, replacing the unattributable joined
`epc.roofs[]` description.
Worksheet-validated to 1e-4: simulated case 41 (4-bp — Ext1 rafters 200mm
→ 0.29, Ext3 rafters As-Built band F → 0.68; roof total 24.8350) and case
42 (6 variants — rafters 50mm → 0.88, rafters unknown band C → 2.30,
joists/none unchanged). Case 40 stays exact (roof 35.340, total 441.1606);
worksheet harness 47/47.
Corpus within-0.5 66.9% → 66.5% (gates 0.65/1.08 hold) — a spec-correct
shift, NOT a regression: all 15 corpus rafter certs carry redacted (None)
thickness yet lodge roof EER 2-4 (insulated), so the open API blanked a
specified thickness and the spec's unknown-rafter 2.30 default correctly
over-states them. Recovery needs a roof-EER→thickness inference on the
API path (follow-up), not a change to the U-table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
README at domain/epc_prediction/README.md — the flow diagram, where each piece
lives, links to the ADRs/CONTEXT/handover/migration note, and a runnable test
command. The team's entry point.
tests/e2e/test_epc_prediction_e2e.py — the whole gap-fill flow against the REAL
Postgres Unit of Work + EPC/Property repositories + EpcComparablePropertiesRepository
+ EpcPrediction, with only the three external HTTP clients faked (EPC API,
geospatial S3, Solar). Proves: EPC-less Property → Ingestion predicts from its
postcode cohort → persists to the predicted slot → reloaded Property resolves
effective_epc via source_path == "predicted". The canonical "see it in action".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gap-fill is wired end-to-end (slices 5a-5e) behind seams; this note is what's
left to switch it on in production: (1) implement the PredictionTargetAttributesReader
stub over property_overrides — with the override-value → API-code mapping
select_comparables needs; (2) run the epc_property.source Drizzle migration; (3)
pass the three optional collaborators at the IngestionOrchestrator composition
root. Plus the open Validation-Cohort exclusion (no code path exists yet — exclude
on source_path == "predicted" when one is built) and the anomaly dual-use pointer.
No code change: the validation exclusion has no consumer to attach to today, and
the structural signal (source_path == "predicted") already exists from slice-5a.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire EPC Prediction gap-fill into IngestionOrchestrator (ADR-0031). When the
predictor collaborators are injected (ComparablesRepo + PredictionAttributesReader
+ EpcPrediction), an EPC-less Property is predicted from its postcode cohort and
persisted to the predicted slot; the eligibility gate (unknown property_type) and
"a lodged EPC is never predicted over" both hold. The two-phase contract is kept:
prediction attributes (Landlord Overrides) resolve in the unit prep phase, the
cohort fetch + select + predict run in the no-unit IO phase, persistence in the
write phase. All three collaborators are OPTIONAL — unwired, ingestion behaves
exactly as before (existing tests unchanged).
3 tests (predict+persist, gate, lodged-wins); 228 pass across orchestration +
epc_prediction + repositories; pyright strict clean. Production composition-root
wiring (real ComparableProperties + override-attributes adapters) is part of the
Jun-te handover.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
build_prediction_target assembles an EPC-less Property's PredictionTarget from
its identity (postcode), resolved coordinates, and Landlord-Override attributes
(property_type / built_form / wall_construction). The eligibility GATE: a Property
whose property_type is unknown returns None — never sized from a mixed-type
cohort (ADR-0031). property_type is the hard cohort filter.
The override attributes are read through a PredictionTargetAttributesReader port
(stub seam) — the real adapter (a read over property_overrides) is being built
separately by the team; ingestion wiring depends on the abstraction and tests
substitute a fake. 2 tests (assembly + gate); pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `source` discriminator (lodged | predicted) to the EPC store so a Property
holds a lodged EPC and a predicted one (EPC Prediction gap-fill) at once
(ADR-0031). EpcRepository.save gains source="lodged"; idempotent delete is now
per-source (a predicted save no longer wipes lodged, and vice versa);
get_for_property/get_for_properties filter lodged; new get_predicted_for_property
/ get_predicted_for_properties read predicted. PropertyPostgresRepository.get +
get_many hydrate Property.predicted_epc, so the predicted picture reaches the
modelling read (both load via get_many). FakeEpcRepo mirrors the dual slot.
EpcPropertyModel gains `source` (default "lodged"); the test DB builds from the
SQLModel mirror so this is exercised without the prod migration. The matching
Drizzle change (column + per-(property_id,source) uniqueness) is the team's to
action before merge — docs/MIGRATION_NOTE_predicted_epc_source.md.
3 store tests (coexist, idempotent predicted re-save leaves lodged, lodged-only
has no predicted) + property-repo wiring; 85 pass across affected suites; new
code pyright-clean (2 pre-existing wwhrs errors in epc_property_table untouched).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build the cohort IO port ADR-0029 deferred (ADR-0031 slice-5b):
`ComparablePropertiesRepository.candidates_for(postcode) -> list[Comparable]`,
with an EPC-API + geospatial adapter that lists the postcode's lodged certs
(search_by_postcode), fetches + maps each (get_by_certificate_number), and
resolves their UPRNs to coordinates in ONE batched read. Register metadata the
cert doesn't carry (address, registration date) is threaded off the search row;
a UPRN-less or unparseable-date cert is kept, just uncoordinated / unweighted.
The domain select_comparables then filters these candidates into the cohort.
Thin CohortEpcClient / CohortGeospatial Protocols keep the adapter testable
against fakes; EpcClientService + GeospatialS3Repository satisfy them
structurally (no changes). 3 tests; pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `predicted_epc` slot to the Property aggregate and a "predicted" branch to
SourcePath / source_path / effective_epc (ADR-0031 decisions 1+3). A
neighbour-synthesised EpcPropertyData resolves as the Effective EPC ONLY when
there is neither a lodged EPC nor Site Notes — a real source always wins
(prediction is last-resort gap-fill). The slot is distinct from `epc` so a
predicted picture coexists with any lodged one (provenance is structural, not a
flag on EpcPropertyData); downstream consumers are untouched.
3 tests: predicted resolves when sole source; lodged EPC wins over predicted;
Site Notes win over predicted. 10/10 green, pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve the slice-5 design tree (grill-with-docs): estimation runs in Ingestion
(refines ADR-0029 dec-3; drops the #1227 "shift to Modelling" — no surviving
rationale, and stages communicate only via persisted state); predicted EPC is
persisted in a DISTINCT slot (EPC table + source discriminator) so lodged +
predicted coexist (enables EPC Anomaly Flags); provenance is structural (the
slot), not a field on EpcPropertyData; effective_epc/source_path gain a
"predicted" branch; slice-5 is gap-fill only; property_type is a REQUIRED input
(hard cohort filter) from Landlord Overrides, and Properties with unknown type
are gated out (no national defaults). OS postcode_search as a broader type
source is a noted follow-on. CONTEXT EPC Prediction entry gains the gating rule.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build a geographically DENSE postcode-clustered corpus to test cross-postcode
geo expansion (the handover's anticipated "real geo payoff"). The gov EPC API
has no area/prefix search (a partial postcode 400s; the old opendatacommunities
partial-search API is decommissioned), so neighbourhood enumeration is external:
seed K postcodes nationally, expand each via postcodes.io's nearest-postcode
endpoint into every unit within RADIUS_M, pull each one's full EPC cohort.
postcodes.io is a corpus-BUILD dependency only — the predictor stays pure. Same
on-disk layout as the scattered corpus, so load_corpus + the coords resolver
consume it unchanged.
MEASURE-FIRST RESULT — cross-postcode expansion is a NO-GO. On a 2-seed pilot
(York YO19 + Islington N51, 81 postcodes / 1558 certs, 140 SAP-10.2 targets),
pooling nearby postcodes regresses accuracy across the board:
same-postcode FA_MAE 9.53 wall 92% age 72% floor_con 85% cylinder 91%
cross <=0.3km FA_MAE 13.1 wall 80% age 61% floor_con 82% cylinder 79%
Even as a thin-cohort top-up it hurts (thin n=18: FA 5.24 -> 7.15). Root cause:
the postcode boundary is itself a strong homogeneity prior (a postcode is one
coherent street/development), so same-postcode neighbours beat geographically
near cross-boundary ones even when the home postcode is sparse (and they rarely
are — median same-postcode cohort here is 34). Geo-proximity helps WITHIN a
postcode (#1227) but does not survive crossing the boundary. Cross-postcode geo
closed; geo weighting stays intra-postcode. Tooling kept (reusable).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Size the predicted dwelling from the geo-proximity-weighted median of the
cohort's floor areas rather than the plain median: homes built together share a
footprint, so a nearer neighbour's area should count for more (the same street
signal #1227 already wired into age / wall / glazing). Reuses `_geo_weights` and
adds `_weighted_median`, which reduces exactly to `statistics.median` under
uniform weights (geo off / no target coordinates) — including the even-count
midpoint average — so the MAD-minimising guarantee is preserved.
Measured over the 514-target SAP-10.2 corpus (leave-one-out):
floor_area MAE 10.48 -> 9.73 m² MAPE 13.2% -> 12.2%
Re-baselines the n=36 fixture floor_area ceiling 11.8983 -> 12.0378 (a method
change, not a loosening; the small fixture subset moved +0.14 the other way as
sample noise while the population improved decisively). The ceiling still pins
the new deterministic value exactly, so the tighten-only ratchet resumes.
Investigation ruling out the adjacent floor-area levers (kept in the follow-up):
lowering minimum_cohort (9.78-10.03, worse), hard same-form filter (10.19),
mean instead of median (10.68), constant bias correction (10.47),
extension-conditioning (oracle 9.50, not worth the misclassification cost) and
room-in-roof conditioning/additive (RiR is a confound for large multi-part
outliers — RiR area is only ~21% of total, and the increment breaks the homes
already predicted exactly). Remaining cohort lever is built-form soft-weighting,
gated on a denser corpus.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Elmhurst Summary §15.1 lodges "Cylinder Size: Value known" with the
measured volume in the "Cylinder Volume (l)" line — the Summary-path
equivalent of the gov-API "Exact" descriptor. The mapper had no entry for
"Value known" so `_elmhurst_cylinder_size_code` raised UnmappedElmhurstLabel,
and even once mapped the measured volume was never threaded through, so the
cascade dropped the cylinder storage loss (~468 kWh/yr) from (219) water
heating on every measured-volume-cylinder Summary.
Per RdSAP 10 §10.5 Table 28 (p.55) a measured cylinder volume is used
directly. Map "Value known" → cascade code 6 (Exact) and thread the §15.1
"Cylinder Volume (l)" value into SapHeating.cylinder_volume_measured_l, which
`_cylinder_volume_l_from_code` (cert_to_inputs.py:5281) already reads for
code 6 — mirroring the gov-API path (mapper.py:1575/1885).
Pins simulated case 39 (P960-0001-001431): an age-A mid-terrace on direct-
acting electric room heaters (SAP code 691, cat 10, control 2602) with
electric-immersion DHW off a 117 L "Value known" cylinder. The full
extractor→mapper→calculator cascade now reproduces the worksheet's SAP-rating
block EXACTLY — SAP value 36.6365 (band F) and (272) CO2 2056.0731 kg/yr,
with (219) water heating 2637.5049 and (255) total energy cost 1802.0039.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a floor_area line giving MAE (m2), MAPE (% of actual), and the typical
(median actual) size, so the absolute error reads relative to dwelling size.
Corpus: MAE 10.48 m2 / MAPE 13.2% / typical 61 m2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
main's 'ND' multiple_glazing_type mapper fix (361abc12) changes the mapped
ground-truth glazing for one fixture cert, so modal_glazing_type re-baselines
0.5833 -> 0.5556 (21/36 -> 20/36). A mapper change shifts the deterministic
fixture rates like a fixture change does — re-baseline, not a prediction
regression. All other component floors + residual ceilings unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Folds a haversine distance kernel into the categorical-mode weighting so a
nearer neighbour counts for more — applied ONLY to the components that showed
a clear distance signal in the corpus pre-check (age band, wall + floor
construction, glazing: homes built/retrofitted together cluster). Roof
construction showed no decay and is excluded; heating keeps its coherent
donor. Predictor stays pure: weights come from target.coordinates vs each
Comparable.coordinates (resolved at the boundary); geo is OFF when the target
has no coords, neutral for a neighbour with none.
Scale chosen on the harness: _GEO_SCALE_KM=0.1 is the gate-safe optimum
(0.05 lifts the corpus more but regresses fixture floor_construction).
Corpus (150pc/514, geo off->on): age 0.564->0.572, age_pm1 0.841->0.847,
wall 0.902->0.912, floor_con 0.786->0.796, glazing 0.667->0.673; roof
unchanged. Fixture: glazing 0.5278->0.5833 (floor ratcheted), all else held.
Refactored recency into a reusable _recency_weights vector composed via
_combine, so similarity/recency/geo factors multiply uniformly. Fixture ships
a committed _coordinates.json (OGL OS OpenData; build script carries it from
the corpus sidecar on rebuild) so the gate exercises geo without S3.
This is the per-component method applied to geography ([[feedback_per_component_best_method]]).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds coordinates: Optional[Coordinates] to Comparable and PredictionTarget
(data carriers — the pure predictor stays IO-free), and wires load_corpus to
read an optional _coordinates.json sidecar ({uprn: [lon, lat]}) and populate
each Comparable from its cert's uprn; iter_predictions threads the held-out
target's coordinates through. Absent sidecar -> geo-weighting stays off (no
behaviour change yet — weighting lands next slice). fetch_corpus_coordinates
now writes the sidecar into the corpus dir. load_corpus populates 99% of
corpus comparables.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
