Commit graph

7023 commits

Author SHA1 Message Date
Khalim Conn-Kowlessar
7cfd54129b fix(mapper): read the dropped rafter_insulation_thickness API field
Roofs lodged insulated at rafters carry their thickness in a DEDICATED
gov-EPC API field, `rafter_insulation_thickness` (e.g. "225mm"), while
`roof_insulation_thickness` stays None (rafters aren't loft joists). That
field was undeclared on the 21.0.x schemas, so `from_dict` silently
dropped it — the rafter certs only *looked* redacted (roof EER 2-4 =
insulated, yet no thickness), and the cascade fell to the Table 18 col (2)
unknown default (2.30), badly under-rating them.

- declare `rafter_insulation_thickness` on RdSapSchema21_0_0/21_0_1 +
  EpcPropertyData.SapBuildingPart (mirrors the existing
  sloping_ceiling_/flat_roof_insulation_thickness dropped-field handling).
- thread it through `from_rdsap_schema_21_0_0/21_0_1` (older schemas get
  None via getattr).
- `heat_transmission` prefers `rafter_insulation_thickness` over
  `roof_insulation_thickness` when the part is at-rafters, so the measured
  RdSAP 10 §5.11.2 Table 16 column (2) row applies (225 mm → 0.25).

Completes the rafters roof fix: with the real thickness read, the rafter
certs are recovered rather than over-stated — cert 3100-8675-0922-8628
(band E, rafters 225mm) +8.93 → +0.43 SAP. Corpus within-0.5 67.0%
(MAE 1.025) and /tmp 71.2% (MAE 0.889) — both NET ABOVE the pre-rafters
baseline (66.9% / 70.6%). Worksheet harness 47/47; regression = only the
3 pre-existing fails; pyright net-zero.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 05:04:39 +00:00
Khalim Conn-Kowlessar
5d556faf86 fix(roof): bill at-rafters insulation on RdSAP 10 Table 16/18 column (2)
`u_roof` only implemented the joist column, so roofs lodged insulated at
rafters (`roof_insulation_location == 1`) were mis-billed at the joist U
on both the API and Summary paths — under-stating loss, over-rating SAP.

RdSAP 10 §5.11.2 Table 16 (spec p.42-43) gives a distinct "insulation at
rafters" column (2): the rafter cavity is shallower than a loft void, so
the same depth yields a higher U (200 mm: rafters 0.29 vs joists 0.21).
§5.11 Table 18 (p.45) likewise carries a rafters column (2) for unknown /
as-built thickness (footnote (1): "The value from the table applies for
unknown and as built") — band A-D = 2.30, E = 1.50, F = 0.68, diverging
from the joist column's 100 mm-equivalent 0.40 default (footnote (4)).

- add `_ROOF_RAFTERS_BY_THICKNESS` (Table 16 col 2) + `_ROOF_RAFTERS_BY_AGE`
  (Table 18 col 2) to rdsap_uvalues; `u_roof` selects them via a new
  `insulation_at_rafters` flag (ignored for flat / sloping-ceiling roofs).
- `heat_transmission` derives the flag PER BUILDING PART from
  `roof_insulation_location` (gov-API int 1 / Summary "R Rafters"), which
  also fixes the multi-part dedup-roof-join problem: each part's own
  location now drives its U, replacing the unattributable joined
  `epc.roofs[]` description.

Worksheet-validated to 1e-4: simulated case 41 (4-bp — Ext1 rafters 200mm
→ 0.29, Ext3 rafters As-Built band F → 0.68; roof total 24.8350) and case
42 (6 variants — rafters 50mm → 0.88, rafters unknown band C → 2.30,
joists/none unchanged). Case 40 stays exact (roof 35.340, total 441.1606);
worksheet harness 47/47.

Corpus within-0.5 66.9% → 66.5% (gates 0.65/1.08 hold) — a spec-correct
shift, NOT a regression: all 15 corpus rafter certs carry redacted (None)
thickness yet lodge roof EER 2-4 (insulated), so the open API blanked a
specified thickness and the spec's unknown-rafter 2.30 default correctly
over-states them. Recovery needs a roof-EER→thickness inference on the
API path (follow-up), not a change to the U-table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 04:42:44 +00:00
Khalim Conn-Kowlessar
f66e2cb020 docs(epc-prediction): module README + end-to-end showcase test
README at domain/epc_prediction/README.md — the flow diagram, where each piece
lives, links to the ADRs/CONTEXT/handover/migration note, and a runnable test
command. The team's entry point.

tests/e2e/test_epc_prediction_e2e.py — the whole gap-fill flow against the REAL
Postgres Unit of Work + EPC/Property repositories + EpcComparablePropertiesRepository
+ EpcPrediction, with only the three external HTTP clients faked (EPC API,
geospatial S3, Solar). Proves: EPC-less Property → Ingestion predicts from its
postcode cohort → persists to the predicted slot → reloaded Property resolves
effective_epc via source_path == "predicted". The canonical "see it in action".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 04:13:30 +00:00
Khalim Conn-Kowlessar
b677448fa0 docs(epc-prediction): slice-5f production-wiring handover for Jun-te
The gap-fill is wired end-to-end (slices 5a-5e) behind seams; this note is what's
left to switch it on in production: (1) implement the PredictionTargetAttributesReader
stub over property_overrides — with the override-value → API-code mapping
select_comparables needs; (2) run the epc_property.source Drizzle migration; (3)
pass the three optional collaborators at the IngestionOrchestrator composition
root. Plus the open Validation-Cohort exclusion (no code path exists yet — exclude
on source_path == "predicted" when one is built) and the anomaly dual-use pointer.

No code change: the validation exclusion has no consumer to attach to today, and
the structural signal (source_path == "predicted") already exists from slice-5a.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 04:05:00 +00:00
Khalim Conn-Kowlessar
5727ac53c1 feat(epc-prediction): slice-5e ingestion wiring (gate → predict → persist)
Wire EPC Prediction gap-fill into IngestionOrchestrator (ADR-0031). When the
predictor collaborators are injected (ComparablesRepo + PredictionAttributesReader
+ EpcPrediction), an EPC-less Property is predicted from its postcode cohort and
persisted to the predicted slot; the eligibility gate (unknown property_type) and
"a lodged EPC is never predicted over" both hold. The two-phase contract is kept:
prediction attributes (Landlord Overrides) resolve in the unit prep phase, the
cohort fetch + select + predict run in the no-unit IO phase, persistence in the
write phase. All three collaborators are OPTIONAL — unwired, ingestion behaves
exactly as before (existing tests unchanged).

3 tests (predict+persist, gate, lodged-wins); 228 pass across orchestration +
epc_prediction + repositories; pyright strict clean. Production composition-root
wiring (real ComparableProperties + override-attributes adapters) is part of the
Jun-te handover.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 04:03:02 +00:00
Khalim Conn-Kowlessar
f2f954f459 feat(epc-prediction): slice-5d target assembly + eligibility gate
build_prediction_target assembles an EPC-less Property's PredictionTarget from
its identity (postcode), resolved coordinates, and Landlord-Override attributes
(property_type / built_form / wall_construction). The eligibility GATE: a Property
whose property_type is unknown returns None — never sized from a mixed-type
cohort (ADR-0031). property_type is the hard cohort filter.

The override attributes are read through a PredictionTargetAttributesReader port
(stub seam) — the real adapter (a read over property_overrides) is being built
separately by the team; ingestion wiring depends on the abstraction and tests
substitute a fake. 2 tests (assembly + gate); pyright strict clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 03:56:57 +00:00
Khalim Conn-Kowlessar
fd43cf2d23 feat(epc-prediction): slice-5c predicted-EPC persistence slot
Add a `source` discriminator (lodged | predicted) to the EPC store so a Property
holds a lodged EPC and a predicted one (EPC Prediction gap-fill) at once
(ADR-0031). EpcRepository.save gains source="lodged"; idempotent delete is now
per-source (a predicted save no longer wipes lodged, and vice versa);
get_for_property/get_for_properties filter lodged; new get_predicted_for_property
/ get_predicted_for_properties read predicted. PropertyPostgresRepository.get +
get_many hydrate Property.predicted_epc, so the predicted picture reaches the
modelling read (both load via get_many). FakeEpcRepo mirrors the dual slot.

EpcPropertyModel gains `source` (default "lodged"); the test DB builds from the
SQLModel mirror so this is exercised without the prod migration. The matching
Drizzle change (column + per-(property_id,source) uniqueness) is the team's to
action before merge — docs/MIGRATION_NOTE_predicted_epc_source.md.

3 store tests (coexist, idempotent predicted re-save leaves lodged, lodged-only
has no predicted) + property-repo wiring; 85 pass across affected suites; new
code pyright-clean (2 pre-existing wwhrs errors in epc_property_table untouched).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 03:50:19 +00:00
Khalim Conn-Kowlessar
6979607ace feat(epc-prediction): slice-5b ComparableProperties repo port + adapter
Build the cohort IO port ADR-0029 deferred (ADR-0031 slice-5b):
`ComparablePropertiesRepository.candidates_for(postcode) -> list[Comparable]`,
with an EPC-API + geospatial adapter that lists the postcode's lodged certs
(search_by_postcode), fetches + maps each (get_by_certificate_number), and
resolves their UPRNs to coordinates in ONE batched read. Register metadata the
cert doesn't carry (address, registration date) is threaded off the search row;
a UPRN-less or unparseable-date cert is kept, just uncoordinated / unweighted.
The domain select_comparables then filters these candidates into the cohort.

Thin CohortEpcClient / CohortGeospatial Protocols keep the adapter testable
against fakes; EpcClientService + GeospatialS3Repository satisfy them
structurally (no changes). 3 tests; pyright strict clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 03:40:59 +00:00
Khalim Conn-Kowlessar
086187ddc7 feat(epc-prediction): slice-5a predicted source path on Property
Add a `predicted_epc` slot to the Property aggregate and a "predicted" branch to
SourcePath / source_path / effective_epc (ADR-0031 decisions 1+3). A
neighbour-synthesised EpcPropertyData resolves as the Effective EPC ONLY when
there is neither a lodged EPC nor Site Notes — a real source always wins
(prediction is last-resort gap-fill). The slot is distinct from `epc` so a
predicted picture coexists with any lodged one (provenance is structural, not a
flag on EpcPropertyData); downstream consumers are untouched.

3 tests: predicted resolves when sole source; lodged EPC wins over predicted;
Site Notes win over predicted. 10/10 green, pyright strict clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 03:33:47 +00:00
Khalim Conn-Kowlessar
d1227fd0c6 docs(epc-prediction): ADR-0031 production wiring + CONTEXT gating rule
Resolve the slice-5 design tree (grill-with-docs): estimation runs in Ingestion
(refines ADR-0029 dec-3; drops the #1227 "shift to Modelling" — no surviving
rationale, and stages communicate only via persisted state); predicted EPC is
persisted in a DISTINCT slot (EPC table + source discriminator) so lodged +
predicted coexist (enables EPC Anomaly Flags); provenance is structural (the
slot), not a field on EpcPropertyData; effective_epc/source_path gain a
"predicted" branch; slice-5 is gap-fill only; property_type is a REQUIRED input
(hard cohort filter) from Landlord Overrides, and Properties with unknown type
are gated out (no national defaults). OS postcode_search as a broader type
source is a noted follow-on. CONTEXT EPC Prediction entry gains the gating rule.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 03:23:12 +00:00
Khalim Conn-Kowlessar
58d5b17145 chore(epc-prediction): dense-corpus fetcher + cross-postcode geo no-go
Build a geographically DENSE postcode-clustered corpus to test cross-postcode
geo expansion (the handover's anticipated "real geo payoff"). The gov EPC API
has no area/prefix search (a partial postcode 400s; the old opendatacommunities
partial-search API is decommissioned), so neighbourhood enumeration is external:
seed K postcodes nationally, expand each via postcodes.io's nearest-postcode
endpoint into every unit within RADIUS_M, pull each one's full EPC cohort.
postcodes.io is a corpus-BUILD dependency only — the predictor stays pure. Same
on-disk layout as the scattered corpus, so load_corpus + the coords resolver
consume it unchanged.

MEASURE-FIRST RESULT — cross-postcode expansion is a NO-GO. On a 2-seed pilot
(York YO19 + Islington N51, 81 postcodes / 1558 certs, 140 SAP-10.2 targets),
pooling nearby postcodes regresses accuracy across the board:
  same-postcode  FA_MAE 9.53  wall 92%  age 72%  floor_con 85%  cylinder 91%
  cross <=0.3km  FA_MAE 13.1  wall 80%  age 61%  floor_con 82%  cylinder 79%
Even as a thin-cohort top-up it hurts (thin n=18: FA 5.24 -> 7.15). Root cause:
the postcode boundary is itself a strong homogeneity prior (a postcode is one
coherent street/development), so same-postcode neighbours beat geographically
near cross-boundary ones even when the home postcode is sparse (and they rarely
are — median same-postcode cohort here is 34). Geo-proximity helps WITHIN a
postcode (#1227) but does not survive crossing the boundary. Cross-postcode geo
closed; geo weighting stays intra-postcode. Tooling kept (reusable).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 03:03:15 +00:00
Khalim Conn-Kowlessar
be3e51bae9 feat(epc-prediction): geo-proximity-weighted floor-area median
Size the predicted dwelling from the geo-proximity-weighted median of the
cohort's floor areas rather than the plain median: homes built together share a
footprint, so a nearer neighbour's area should count for more (the same street
signal #1227 already wired into age / wall / glazing). Reuses `_geo_weights` and
adds `_weighted_median`, which reduces exactly to `statistics.median` under
uniform weights (geo off / no target coordinates) — including the even-count
midpoint average — so the MAD-minimising guarantee is preserved.

Measured over the 514-target SAP-10.2 corpus (leave-one-out):
  floor_area MAE  10.48 -> 9.73 m²   MAPE 13.2% -> 12.2%

Re-baselines the n=36 fixture floor_area ceiling 11.8983 -> 12.0378 (a method
change, not a loosening; the small fixture subset moved +0.14 the other way as
sample noise while the population improved decisively). The ceiling still pins
the new deterministic value exactly, so the tighten-only ratchet resumes.

Investigation ruling out the adjacent floor-area levers (kept in the follow-up):
lowering minimum_cohort (9.78-10.03, worse), hard same-form filter (10.19),
mean instead of median (10.68), constant bias correction (10.47),
extension-conditioning (oracle 9.50, not worth the misclassification cost) and
room-in-roof conditioning/additive (RiR is a confound for large multi-part
outliers — RiR area is only ~21% of total, and the increment breaks the homes
already predicted exactly). Remaining cohort lever is built-form soft-weighting,
gated on a denser corpus.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:08:05 +00:00
Khalim Conn-Kowlessar
b2b6f8e954 fix(mapper): map Elmhurst "Value known" cylinder to measured volume (code 6)
The Elmhurst Summary §15.1 lodges "Cylinder Size: Value known" with the
measured volume in the "Cylinder Volume (l)" line — the Summary-path
equivalent of the gov-API "Exact" descriptor. The mapper had no entry for
"Value known" so `_elmhurst_cylinder_size_code` raised UnmappedElmhurstLabel,
and even once mapped the measured volume was never threaded through, so the
cascade dropped the cylinder storage loss (~468 kWh/yr) from (219) water
heating on every measured-volume-cylinder Summary.

Per RdSAP 10 §10.5 Table 28 (p.55) a measured cylinder volume is used
directly. Map "Value known" → cascade code 6 (Exact) and thread the §15.1
"Cylinder Volume (l)" value into SapHeating.cylinder_volume_measured_l, which
`_cylinder_volume_l_from_code` (cert_to_inputs.py:5281) already reads for
code 6 — mirroring the gov-API path (mapper.py:1575/1885).

Pins simulated case 39 (P960-0001-001431): an age-A mid-terrace on direct-
acting electric room heaters (SAP code 691, cat 10, control 2602) with
electric-immersion DHW off a 117 L "Value known" cylinder. The full
extractor→mapper→calculator cascade now reproduces the worksheet's SAP-rating
block EXACTLY — SAP value 36.6365 (band F) and (272) CO2 2056.0731 kg/yr,
with (219) water heating 2637.5049 and (255) total energy cost 1802.0039.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 23:57:25 +00:00
Jun-te Kim
e289c1449b docs: handoff for expanding the real-life cert SAP-accuracy corpus
Strategy/context companion to the validate-cert-sap-accuracy skill: the
per-cert loop, how to read the gov-API-vs-Elmhurst comparison, the code->value
gotchas (immersion/cylinder/party-wall/baths/off-peak), known mapper gaps to
chase (alt-wall drop), cert-selection for coverage, guardrails (corpus gauge,
no tuning to one cert, no tolerance widening), and the current corpus state.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 15:28:40 +00:00
Jun-te Kim
5c11fd35c8 Validate SAP calculator vs Elmhurst; fix reduced-field window U; add accuracy harness
Reduced-field window U: heat_transmission derived the synthesised-window
raw U from u_window(all None) -> the 2.5 placeholder regardless of glazing.
Now routes the (uniform) glazing_type code through u_window (RdSAP Table 24)
so e.g. double pre-2002 reads 2.8, not 2.5. Only the pre-SAP10 reduced-field
path is affected (21.0.1 certs carry per-window U upstream) — the RdSAP-21.0.1
corpus gauge is unchanged at 66.9% within-0.5.

test_real_cert_sap_accuracy: pin uprn_10002468137 (RdSAP-17.1, all-electric
storage heaters) at SAP 61, validated against Elmhurst on identical inputs
(dual off-peak immersion, 110 L cylinder, 2 baths). Our engine reproduces
Elmhurst's fuel cost to the penny; lodged 55 is the old SAP-2012 schema.

Tooling to grow the accuracy corpus:
- scripts/fetch_real_life_epc_sample.py — capture a cert by UPRN into the corpus.
- scripts/compare_epc_paths.py — diff gov-API vs Elmhurst-summary EpcPropertyData
  and run both through the engine, localising mapper vs calculator differences.
- skill validate-cert-sap-accuracy — the end-to-end loop (capture -> Elmhurst
  inputs -> human builds -> compare -> reconcile -> pin in the test).
- skill epc-to-elmhurst-rdsap-inputs reference: corrected immersion (code 1=dual),
  cylinder size (code 2 = Normal/110 L), and bath-count (WWHRS sub-tab) mappings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 15:26:11 +00:00
Khalim Conn-Kowlessar
da3fc92d53 docs(epc-prediction): handover for the accuracy backlog + geo work
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 15:12:00 +00:00
Daniel Roth
1fe67fe814
Merge pull request #1235 from Hestia-Homes/feature/deploy-sharepoint-renamer
Sharepoint renamer: Remove breaking init file
2026-06-15 16:08:49 +01:00
Khalim Conn-Kowlessar
d8f015fb0e feat(epc-prediction): report floor-area MAE + MAPE vs typical size
Adds a floor_area line giving MAE (m2), MAPE (% of actual), and the typical
(median actual) size, so the absolute error reads relative to dwelling size.
Corpus: MAE 10.48 m2 / MAPE 13.2% / typical 61 m2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 15:07:22 +00:00
Khalim Conn-Kowlessar
aea2d7150f test(epc-prediction): re-baseline modal_glazing floor after main merge
main's 'ND' multiple_glazing_type mapper fix (361abc12) changes the mapped
ground-truth glazing for one fixture cert, so modal_glazing_type re-baselines
0.5833 -> 0.5556 (21/36 -> 20/36). A mapper change shifts the deterministic
fixture rates like a fixture change does — re-baseline, not a prediction
regression. All other component floors + residual ceilings unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 15:04:34 +00:00
Khalim Conn-Kowlessar
0b2827e9ff Merge remote-tracking branch 'origin/main' into feature/epc-prediction 2026-06-15 15:03:27 +00:00
Daniel Roth
03dc0a3eef add local handler and missing requirement 2026-06-15 15:03:07 +00:00
Khalim Conn-Kowlessar
1f26703dc5 feat(epc-prediction): geo-proximity weighting, per-component (#1227)
Folds a haversine distance kernel into the categorical-mode weighting so a
nearer neighbour counts for more — applied ONLY to the components that showed
a clear distance signal in the corpus pre-check (age band, wall + floor
construction, glazing: homes built/retrofitted together cluster). Roof
construction showed no decay and is excluded; heating keeps its coherent
donor. Predictor stays pure: weights come from target.coordinates vs each
Comparable.coordinates (resolved at the boundary); geo is OFF when the target
has no coords, neutral for a neighbour with none.

Scale chosen on the harness: _GEO_SCALE_KM=0.1 is the gate-safe optimum
(0.05 lifts the corpus more but regresses fixture floor_construction).
Corpus (150pc/514, geo off->on): age 0.564->0.572, age_pm1 0.841->0.847,
wall 0.902->0.912, floor_con 0.786->0.796, glazing 0.667->0.673; roof
unchanged. Fixture: glazing 0.5278->0.5833 (floor ratcheted), all else held.

Refactored recency into a reusable _recency_weights vector composed via
_combine, so similarity/recency/geo factors multiply uniformly. Fixture ships
a committed _coordinates.json (OGL OS OpenData; build script carries it from
the corpus sidecar on rebuild) so the gate exercises geo without S3.

This is the per-component method applied to geography ([[feedback_per_component_best_method]]).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:58:42 +00:00
Daniel Roth
9b21cc5512 remove breaking init file 2026-06-15 14:52:48 +00:00
Khalim Conn-Kowlessar
fdc314c857 feat(epc-prediction): thread coordinates onto Comparable + target (#1227)
Adds coordinates: Optional[Coordinates] to Comparable and PredictionTarget
(data carriers — the pure predictor stays IO-free), and wires load_corpus to
read an optional _coordinates.json sidecar ({uprn: [lon, lat]}) and populate
each Comparable from its cert's uprn; iter_predictions threads the held-out
target's coordinates through. Absent sidecar -> geo-weighting stays off (no
behaviour change yet — weighting lands next slice). fetch_corpus_coordinates
now writes the sidecar into the corpus dir. load_corpus populates 99% of
corpus comparables.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:46:01 +00:00
Jun-te Kim
140ad39898 Map full-SAP code-based heating systems via sap_main_heating_code 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 14:40:59 +00:00
Jun-te Kim
345154c6b7 Map full-SAP measured ventilation: air permeability, MV kind, sheltered sides 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 14:37:52 +00:00
Daniel Roth
9d56cd7c1e
Merge pull request #1234 from Hestia-Homes/feature/deploy-sharepoint-renamer
Deploy sharepoint renamer: Correct dockerfile imports
2026-06-15 15:35:55 +01:00
Khalim Conn-Kowlessar
95719dd587 feat(geospatial): batch coordinates_for_uprns lookup (#1227)
Adds GeospatialRepository.coordinates_for_uprns(uprns) -> dict — a batch
coordinate lookup returning only covered UPRNs. The S3 adapter overrides it
to read the meta once, group UPRNs by their covering partition, and read each
partition once for all the UPRNs it covers; co-located (closely-numbered)
UPRNs share a partition, so an EPC Prediction cohort is typically one or two
reads instead of one per neighbour. Default port impl is a per-UPRN loop.

Feeds the EPC Prediction geo-proximity work: a cohort's UPRNs resolve to
coordinates in a couple of reads (validated at corpus scale: 170 partition
reads for 2683 UPRNs).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:35:32 +00:00
Daniel Roth
b31db4b58b correct Dockerfile imports 2026-06-15 14:29:04 +00:00
Khalim Conn-Kowlessar
c0a1bcac95 feat(epc-prediction): resolve corpus UPRN coordinates from S3 (#1227 signal check)
One-time utility: resolves every corpus cert's uprn -> WGS84 lon/lat from the
OS Open-UPRN parquet (DATA_BUCKET/spatial/) via boto3, grouping UPRNs by their
covering partition so each ~1.7MB partition is read at most once (the efficient
batch lookup we intend to add to GeospatialRepository). Caches {uprn:[lon,lat]}
locally for the validation harness. Resolved 2609/2683 corpus UPRNs (97%).

Signal pre-check result (does intra-postcode proximity predict components?):
intra-postcode distances are non-trivial (median 44m, p90 138m, max ~1km),
and nearer neighbours match the target markedly better on age band (0.63 at
<20m -> 0.16 at >300m), wall, glazing and floor construction. Roof shows no
decay. => geo-proximity is worth building, per-component (strongest for age,
the weakest fabric component).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:28:39 +00:00
Jun-te Kim
c035d17f2b Map full-SAP certs end-to-end through the dispatch ladder and pin observed score 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 14:25:48 +00:00
Daniel Roth
0ed17cfd39 Merge branch 'main' into feature/deploy-sharepoint-renamer 2026-06-15 14:24:10 +00:00
Jun-te Kim
acd0ed485d Map full-SAP energy source, mains-gas inference and lighting bulbs 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 14:23:31 +00:00
Jun-te Kim
cb4d080da2 Map full-SAP heating systems onto the domain SapHeating model 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 14:18:01 +00:00
Jun-te Kim
6226575086
Merge pull request #1232 from Hestia-Homes/feature/deploy-sharepoint-renamer
Sharepoint renamer: fix terraform issue and add dry_run option
2026-06-15 15:13:02 +01:00
Jun-te Kim
125ff6f4dd Merge remote-tracking branch 'origin/main' into feature/hyde_make_it_more_accurate_with_tests
# Conflicts:
#	datatypes/epc/domain/mapper.py
2026-06-15 14:12:38 +00:00
Daniel Roth
8b27a5fda2 correct lambda name 2026-06-15 14:08:40 +00:00
Daniel Roth
1af9d84f94 Merge branch 'main' into feature/deploy-sharepoint-renamer 2026-06-15 14:07:27 +00:00
Daniel Roth
963b7d70fe fix terraform error and pass handler bool for dry runs 2026-06-15 14:06:54 +00:00
Khalim Conn-Kowlessar
4afab2c3d8 feat(epc-prediction): roof-insulation +/-1-bucket reporting
Adds roof_insulation_thickness_pm1 (mirrors construction_age_band_pm1, issue
#1222): adjacent RdSAP thickness buckets (0/NI,12mm..400mm+) carry near-
identical roof U-values, so an off-by-one bucket is a SAP-neutral hit. 'ND'
(no-data) is off the ordered scale, so only an exact match counts there.
Honest measurement of SAP-relevant roof-insulation quality.

Corpus (150pc/514): exact 49.3% -> +/-1 53.7% (the misses are often multiple
buckets or ND, so the band gain is smaller than age's). Fixture: exact ==
+/-1 (0.4118) — its misses are all >1 bucket; gate floor added at 0.4118.

Also fixes two pre-existing pyright errors in the touched test file
(_epc main_fuel_type/main_heating_control were Optional but the
MainHeatingDetail attributes are non-optional Union[int, str]).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:04:18 +00:00
Jun-te Kim
5a3228ab5e
Merge pull request #1217 from Hestia-Homes/feature/per-cert-mapper-validation
Feature/per cert mapper validation
2026-06-15 15:03:05 +01:00
Jun-te Kim
5ebeb71090 Back-solve habitable-room count from full-SAP measured living area 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 13:58:03 +00:00
Khalim Conn-Kowlessar
fffb07d04b test(harness): re-pin golden-cert plans to the gain-maximising packages
Three more pre-existing failures (present at 9ee38211, before this branch's
recent commits; same family as the orchestration multi-measure re-pin) —
golden-cert plan expectations that predate the ASHP generator (ADR-0025)
and the optimiser folding forced dependencies into candidate gain (ADR-0016):

- test_console: a multi-measure plan now leads with air_source_heat_pump,
  not cavity_wall_insulation (which is dropped — its forced ventilation makes
  the pair net-negative). Assert a measure actually in the package.
- test_report 0330: package is now {solid_floor_insulation, air_source_heat_
  pump}; cavity_wall + forced mechanical_ventilation correctly excluded.
- test_report 0036: gain-maximising package is now {solid_floor_insulation,
  low_energy_lighting}.

Same verified-correct optimiser evolution as 077e3a39 (cavity_wall +2.9 SAP
alone but its forced fabric→ventilation dep drags the pair net-negative).
Re-pin to the actual packages + their trigger fields; the forced wall→vent
edge stays covered by test_measure_dependency / test_optimiser.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:57:27 +00:00
Jun-te Kim
af26688846 Derive heat-loss perimeter and party-wall length from full-SAP measured wall areas 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 13:56:31 +00:00
Khalim Conn-Kowlessar
7f48495ed5 feat(epc-prediction): surface CO2 + PEI calculator floors in the report (#1228)
The validation report showed only the SAP calculator floor (calc(actual) vs
lodged), so the headline PEI MAE (~40 kWh/m2) read as prediction error when
much of it is the calculator's own API-path residual. Adds the CO2 + PEI
floors alongside SAP.

Diagnostic (150pc/514): PEI floor MAE 15.73 (calc(actual) vs lodged) vs SAP
floor 1.57; calc(actual)/lodged PEI ratio ~1.06 (mean +10.7, ~+6% over-
estimate). That RULES OUT the suspected gross unit/definition mismatch (a
unit bug would be ~2x/3.6x, not 1.06) and reframes #1228: the PEI gap is a
modest calculator bias (~16 floor, calc-branch) plus a larger prediction-
sensitivity term (~24) — PEI is far more prediction-sensitive than SAP.
CO2 floor 0.20 t. Script-only; no gate impact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:55:20 +00:00
Khalim Conn-Kowlessar
06a66b3dd9 feat(epc-prediction): coherent heating donor selection (#1225)
Heating sub-fields can't be field-moded without breaking system coherence,
so the whole SapHeating cluster is now copied as a unit from a single
coherent donor rather than inherited from the structural template: the
neighbour matching the cohort's modal heating signature (main fuel +
category + cylinder presence), most recent among the matches (recent cert =
current system). Including cylinder presence in the signature is load-bearing
— it protects has_hot_water_cylinder + cylinder_insulation (a bare fuel+cat
signature regressed them).

Corpus (150pc/514): heating_main_control 66.3 -> 73.9% (+7.6, the target),
main_fuel 92.8 -> 96.9, category 90.7 -> 95.7, water_fuel 92.8 -> 96.3,
water_code 88.5 -> 95.3, has_cylinder 81.1 -> 89.7, secondary 36.2 -> 42.0.
SAP MAE vs lodged 7.08 -> 6.00 (calculator floor 1.57). cylinder_insulation
-13.6 corpus (tiny-n) but +33pp on the fixture; AC requires control up +
fuel/category hold + SAP not worsened, all met.

Gate (36-target fixture): zero regression; ratcheted main_category
0.8889->0.9444, main_control 0.7500->0.8056, water_fuel 0.9167->0.9722,
water_code 0.8889->0.9444, cylinder_insulation_type 0.1667->0.5000. This is
the per-component heating method ([[feedback_per_component_best_method]]):
coherent donor, never field-mode.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:48:15 +00:00
Jun-te Kim
8746eabb70 Fail loud on unmapped full-SAP opening-type codes 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 13:48:14 +00:00
Jun-te Kim
dde98fb684 Collapse full-SAP roof-window openings onto sap_roof_windows 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 13:46:32 +00:00
Khalim Conn-Kowlessar
077e3a3947 test(orchestration): re-pin multi-measure plan to the gain-maximising package
The optimiser-package expectation was stale: it predated the optimiser
folding a triggered measure's forced dependency into its candidate gain
(ADR-0016). The run considers ALL measures (considered_measures defaults
to None — no restriction), so once the ASHP bundle became SAP-beneficial
(ADR-0025) the gain-maximising package shifted.

Verified the new package is CORRECT, not a regression: on the test EPC,
cavity-wall insulation earns +2.9 SAP alone but its forced fabric→
ventilation dependency (ADR-0016) drags the wall+ventilation pair to a
NET −1.8 SAP (−0.9 on top of the ASHP package), so the gain-maximising
Optimiser correctly excludes the wall and its forced ventilation. Update
the expected set to {air_source_heat_pump, suspended_floor_insulation,
low_energy_lighting, secondary_heating_removal} and drop the wall/vent-
specific assertions — the forced wall→ventilation edge is covered by
test_measure_dependency / test_optimiser; this integration test keeps its
end-to-end optimise→persist→telescope coverage on the chosen package.

Pre-existing failure (present before this branch's recent commits), outside
the handover regression gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:46:22 +00:00
Jun-te Kim
36929accf7 Collapse full-SAP door openings onto door count and area-weighted U-value 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 13:39:53 +00:00