Folds a haversine distance kernel into the categorical-mode weighting so a
nearer neighbour counts for more — applied ONLY to the components that showed
a clear distance signal in the corpus pre-check (age band, wall + floor
construction, glazing: homes built/retrofitted together cluster). Roof
construction showed no decay and is excluded; heating keeps its coherent
donor. Predictor stays pure: weights come from target.coordinates vs each
Comparable.coordinates (resolved at the boundary); geo is OFF when the target
has no coords, neutral for a neighbour with none.
Scale chosen on the harness: _GEO_SCALE_KM=0.1 is the gate-safe optimum
(0.05 lifts the corpus more but regresses fixture floor_construction).
Corpus (150pc/514, geo off->on): age 0.564->0.572, age_pm1 0.841->0.847,
wall 0.902->0.912, floor_con 0.786->0.796, glazing 0.667->0.673; roof
unchanged. Fixture: glazing 0.5278->0.5833 (floor ratcheted), all else held.
Refactored recency into a reusable _recency_weights vector composed via
_combine, so similarity/recency/geo factors multiply uniformly. Fixture ships
a committed _coordinates.json (OGL OS OpenData; build script carries it from
the corpus sidecar on rebuild) so the gate exercises geo without S3.
This is the per-component method applied to geography ([[feedback_per_component_best_method]]).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds coordinates: Optional[Coordinates] to Comparable and PredictionTarget
(data carriers — the pure predictor stays IO-free), and wires load_corpus to
read an optional _coordinates.json sidecar ({uprn: [lon, lat]}) and populate
each Comparable from its cert's uprn; iter_predictions threads the held-out
target's coordinates through. Absent sidecar -> geo-weighting stays off (no
behaviour change yet — weighting lands next slice). fetch_corpus_coordinates
now writes the sidecar into the corpus dir. load_corpus populates 99% of
corpus comparables.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds GeospatialRepository.coordinates_for_uprns(uprns) -> dict — a batch
coordinate lookup returning only covered UPRNs. The S3 adapter overrides it
to read the meta once, group UPRNs by their covering partition, and read each
partition once for all the UPRNs it covers; co-located (closely-numbered)
UPRNs share a partition, so an EPC Prediction cohort is typically one or two
reads instead of one per neighbour. Default port impl is a per-UPRN loop.
Feeds the EPC Prediction geo-proximity work: a cohort's UPRNs resolve to
coordinates in a couple of reads (validated at corpus scale: 170 partition
reads for 2683 UPRNs).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One-time utility: resolves every corpus cert's uprn -> WGS84 lon/lat from the
OS Open-UPRN parquet (DATA_BUCKET/spatial/) via boto3, grouping UPRNs by their
covering partition so each ~1.7MB partition is read at most once (the efficient
batch lookup we intend to add to GeospatialRepository). Caches {uprn:[lon,lat]}
locally for the validation harness. Resolved 2609/2683 corpus UPRNs (97%).
Signal pre-check result (does intra-postcode proximity predict components?):
intra-postcode distances are non-trivial (median 44m, p90 138m, max ~1km),
and nearer neighbours match the target markedly better on age band (0.63 at
<20m -> 0.16 at >300m), wall, glazing and floor construction. Roof shows no
decay. => geo-proximity is worth building, per-component (strongest for age,
the weakest fabric component).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds roof_insulation_thickness_pm1 (mirrors construction_age_band_pm1, issue
#1222): adjacent RdSAP thickness buckets (0/NI,12mm..400mm+) carry near-
identical roof U-values, so an off-by-one bucket is a SAP-neutral hit. 'ND'
(no-data) is off the ordered scale, so only an exact match counts there.
Honest measurement of SAP-relevant roof-insulation quality.
Corpus (150pc/514): exact 49.3% -> +/-1 53.7% (the misses are often multiple
buckets or ND, so the band gain is smaller than age's). Fixture: exact ==
+/-1 (0.4118) — its misses are all >1 bucket; gate floor added at 0.4118.
Also fixes two pre-existing pyright errors in the touched test file
(_epc main_fuel_type/main_heating_control were Optional but the
MainHeatingDetail attributes are non-optional Union[int, str]).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three more pre-existing failures (present at 9ee38211, before this branch's
recent commits; same family as the orchestration multi-measure re-pin) —
golden-cert plan expectations that predate the ASHP generator (ADR-0025)
and the optimiser folding forced dependencies into candidate gain (ADR-0016):
- test_console: a multi-measure plan now leads with air_source_heat_pump,
not cavity_wall_insulation (which is dropped — its forced ventilation makes
the pair net-negative). Assert a measure actually in the package.
- test_report 0330: package is now {solid_floor_insulation, air_source_heat_
pump}; cavity_wall + forced mechanical_ventilation correctly excluded.
- test_report 0036: gain-maximising package is now {solid_floor_insulation,
low_energy_lighting}.
Same verified-correct optimiser evolution as 077e3a39 (cavity_wall +2.9 SAP
alone but its forced fabric→ventilation dep drags the pair net-negative).
Re-pin to the actual packages + their trigger fields; the forced wall→vent
edge stays covered by test_measure_dependency / test_optimiser.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The validation report showed only the SAP calculator floor (calc(actual) vs
lodged), so the headline PEI MAE (~40 kWh/m2) read as prediction error when
much of it is the calculator's own API-path residual. Adds the CO2 + PEI
floors alongside SAP.
Diagnostic (150pc/514): PEI floor MAE 15.73 (calc(actual) vs lodged) vs SAP
floor 1.57; calc(actual)/lodged PEI ratio ~1.06 (mean +10.7, ~+6% over-
estimate). That RULES OUT the suspected gross unit/definition mismatch (a
unit bug would be ~2x/3.6x, not 1.06) and reframes #1228: the PEI gap is a
modest calculator bias (~16 floor, calc-branch) plus a larger prediction-
sensitivity term (~24) — PEI is far more prediction-sensitive than SAP.
CO2 floor 0.20 t. Script-only; no gate impact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Heating sub-fields can't be field-moded without breaking system coherence,
so the whole SapHeating cluster is now copied as a unit from a single
coherent donor rather than inherited from the structural template: the
neighbour matching the cohort's modal heating signature (main fuel +
category + cylinder presence), most recent among the matches (recent cert =
current system). Including cylinder presence in the signature is load-bearing
— it protects has_hot_water_cylinder + cylinder_insulation (a bare fuel+cat
signature regressed them).
Corpus (150pc/514): heating_main_control 66.3 -> 73.9% (+7.6, the target),
main_fuel 92.8 -> 96.9, category 90.7 -> 95.7, water_fuel 92.8 -> 96.3,
water_code 88.5 -> 95.3, has_cylinder 81.1 -> 89.7, secondary 36.2 -> 42.0.
SAP MAE vs lodged 7.08 -> 6.00 (calculator floor 1.57). cylinder_insulation
-13.6 corpus (tiny-n) but +33pp on the fixture; AC requires control up +
fuel/category hold + SAP not worsened, all met.
Gate (36-target fixture): zero regression; ratcheted main_category
0.8889->0.9444, main_control 0.7500->0.8056, water_fuel 0.9167->0.9722,
water_code 0.8889->0.9444, cylinder_insulation_type 0.1667->0.5000. This is
the per-component heating method ([[feedback_per_component_best_method]]):
coherent donor, never field-mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The optimiser-package expectation was stale: it predated the optimiser
folding a triggered measure's forced dependency into its candidate gain
(ADR-0016). The run considers ALL measures (considered_measures defaults
to None — no restriction), so once the ASHP bundle became SAP-beneficial
(ADR-0025) the gain-maximising package shifted.
Verified the new package is CORRECT, not a regression: on the test EPC,
cavity-wall insulation earns +2.9 SAP alone but its forced fabric→
ventilation dependency (ADR-0016) drags the wall+ventilation pair to a
NET −1.8 SAP (−0.9 on top of the ASHP package), so the gain-maximising
Optimiser correctly excludes the wall and its forced ventilation. Update
the expected set to {air_source_heat_pump, suspended_floor_insulation,
low_energy_lighting, secondary_heating_removal} and drop the wall/vent-
specific assertions — the forced wall→ventilation edge is covered by
test_measure_dependency / test_optimiser; this integration test keeps its
end-to-end optimise→persist→telescope coverage on the chosen package.
Pre-existing failure (present before this branch's recent commits), outside
the handover regression gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-component method: glazing type is now the recency-weighted cohort mode
applied to every predicted window, rather than copied from the template.
Glazing is retrofitted over a dwelling's life (single -> double), so a
recent neighbour reflects the current state — same family as roof-insulation
thickness. Recency is the CORRECT weighting here: plain moding regressed the
fixture (-5.6pp) and was previously reverted; similarity weighting also
regressed it; recency improves BOTH (window geometry stays on the template,
only the glazing categorical moves).
modal_glazing_type: corpus (150pc/514) 60.7 -> 66.7% (+6.0pp); fixture
0.5000 -> 0.5278 (floor ratcheted up). Heating, geometry residuals and all
other components unchanged. Refactored _recency_weighted_mode to a reusable
_recency_weighted_choice(value_of) shared by roof insulation + glazing.
Closes the #1223 per-component approach: floor-area (median estimate) +
glazing (recency) shipped as distinct best-fit methods rather than a global
recency template, which would have disturbed the coherence-coupled heating
cluster.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The realistic re-generation of case 37 (code-117 gas boiler, control 2102,
+ a MAINS-GAS condensing gas-fire secondary code 611, vs case 37's biogas
605). The full extractor -> mapper -> calculator pipeline reproduces the
worksheet's SAP-rating block EXACTLY: continuous SAP 60.9152 (Δ 2e-5) and
(272) CO2 5801.0770 (Δ ~0). This confirms the boiler-efficiency /
control-2102 −5pp interlock / secondary-fuel handling are all correct, and
that case 37's +7 gap was purely the biogas sub-fuel the Summary export
cannot carry.
Summary mirrored into backend/documents_parser/tests/fixtures so the pin
runs without the unstaged workspace. PE not pinned — it is a separate
DPER block (different scope) already guarded by the corpus PE gauge.
Worksheet harness 47/47 unchanged; pyright net-zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-component method, not a global template change: the predicted floor
area is now the cohort median (the MAD-minimising point estimate of the
target's size) rather than whichever structural template's own area. The
calculator derives heat loss from building-part geometry, not this scalar,
so decoupling them is safe and the scalar becomes a better size estimate.
floor_area mean|.|: corpus (150pc/514 targets) 10.62 -> 10.48; fixture
12.2175 -> 11.8983 (ceiling ratcheted down). No other component moves.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The HHR-storage HeatingOverlay (ADR-0024) added an off-peak electric
immersion cylinder but never set `immersion_heating_type`, so the overlaid
cert left it None. The calculator then could not resolve `immersion_single`
for the SAP 10.2 Table 13 HW high-rate split and billed hot water 100% at
the off-peak low rate — £127.41 vs the relodged after-cert's £169.39,
overstating the overlay's SAP by +1.26 (CO2/PE matched, isolating it to the
HW cost path).
Add `immersion_heating_type` to HeatingOverlay, route it through
`_fold_heating` (it lives on `sap_heating`), and set it to 1 (single
off-peak immersion) on the HHR overlay to match the relodged reference.
Closes both `test_hhr_storage_overlay_reproduces_the_relodged_after_*`
cascade pins (electric-storage and no-system befores share the after).
Pre-existing failure (present before this branch's recent commits), outside
the handover regression gate. Full modelling suite 220 pass, pyright net-
zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
