Commit graph

6648 commits

Author SHA1 Message Date
Khalim Conn-Kowlessar
b677448fa0 docs(epc-prediction): slice-5f production-wiring handover for Jun-te
The gap-fill is wired end-to-end (slices 5a-5e) behind seams; this note is what's
left to switch it on in production: (1) implement the PredictionTargetAttributesReader
stub over property_overrides — with the override-value → API-code mapping
select_comparables needs; (2) run the epc_property.source Drizzle migration; (3)
pass the three optional collaborators at the IngestionOrchestrator composition
root. Plus the open Validation-Cohort exclusion (no code path exists yet — exclude
on source_path == "predicted" when one is built) and the anomaly dual-use pointer.

No code change: the validation exclusion has no consumer to attach to today, and
the structural signal (source_path == "predicted") already exists from slice-5a.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 04:05:00 +00:00
Khalim Conn-Kowlessar
5727ac53c1 feat(epc-prediction): slice-5e ingestion wiring (gate → predict → persist)
Wire EPC Prediction gap-fill into IngestionOrchestrator (ADR-0031). When the
predictor collaborators are injected (ComparablesRepo + PredictionAttributesReader
+ EpcPrediction), an EPC-less Property is predicted from its postcode cohort and
persisted to the predicted slot; the eligibility gate (unknown property_type) and
"a lodged EPC is never predicted over" both hold. The two-phase contract is kept:
prediction attributes (Landlord Overrides) resolve in the unit prep phase, the
cohort fetch + select + predict run in the no-unit IO phase, persistence in the
write phase. All three collaborators are OPTIONAL — unwired, ingestion behaves
exactly as before (existing tests unchanged).

3 tests (predict+persist, gate, lodged-wins); 228 pass across orchestration +
epc_prediction + repositories; pyright strict clean. Production composition-root
wiring (real ComparableProperties + override-attributes adapters) is part of the
Jun-te handover.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 04:03:02 +00:00
Khalim Conn-Kowlessar
f2f954f459 feat(epc-prediction): slice-5d target assembly + eligibility gate
build_prediction_target assembles an EPC-less Property's PredictionTarget from
its identity (postcode), resolved coordinates, and Landlord-Override attributes
(property_type / built_form / wall_construction). The eligibility GATE: a Property
whose property_type is unknown returns None — never sized from a mixed-type
cohort (ADR-0031). property_type is the hard cohort filter.

The override attributes are read through a PredictionTargetAttributesReader port
(stub seam) — the real adapter (a read over property_overrides) is being built
separately by the team; ingestion wiring depends on the abstraction and tests
substitute a fake. 2 tests (assembly + gate); pyright strict clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 03:56:57 +00:00
Khalim Conn-Kowlessar
fd43cf2d23 feat(epc-prediction): slice-5c predicted-EPC persistence slot
Add a `source` discriminator (lodged | predicted) to the EPC store so a Property
holds a lodged EPC and a predicted one (EPC Prediction gap-fill) at once
(ADR-0031). EpcRepository.save gains source="lodged"; idempotent delete is now
per-source (a predicted save no longer wipes lodged, and vice versa);
get_for_property/get_for_properties filter lodged; new get_predicted_for_property
/ get_predicted_for_properties read predicted. PropertyPostgresRepository.get +
get_many hydrate Property.predicted_epc, so the predicted picture reaches the
modelling read (both load via get_many). FakeEpcRepo mirrors the dual slot.

EpcPropertyModel gains `source` (default "lodged"); the test DB builds from the
SQLModel mirror so this is exercised without the prod migration. The matching
Drizzle change (column + per-(property_id,source) uniqueness) is the team's to
action before merge — docs/MIGRATION_NOTE_predicted_epc_source.md.

3 store tests (coexist, idempotent predicted re-save leaves lodged, lodged-only
has no predicted) + property-repo wiring; 85 pass across affected suites; new
code pyright-clean (2 pre-existing wwhrs errors in epc_property_table untouched).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 03:50:19 +00:00
Khalim Conn-Kowlessar
6979607ace feat(epc-prediction): slice-5b ComparableProperties repo port + adapter
Build the cohort IO port ADR-0029 deferred (ADR-0031 slice-5b):
`ComparablePropertiesRepository.candidates_for(postcode) -> list[Comparable]`,
with an EPC-API + geospatial adapter that lists the postcode's lodged certs
(search_by_postcode), fetches + maps each (get_by_certificate_number), and
resolves their UPRNs to coordinates in ONE batched read. Register metadata the
cert doesn't carry (address, registration date) is threaded off the search row;
a UPRN-less or unparseable-date cert is kept, just uncoordinated / unweighted.
The domain select_comparables then filters these candidates into the cohort.

Thin CohortEpcClient / CohortGeospatial Protocols keep the adapter testable
against fakes; EpcClientService + GeospatialS3Repository satisfy them
structurally (no changes). 3 tests; pyright strict clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 03:40:59 +00:00
Khalim Conn-Kowlessar
086187ddc7 feat(epc-prediction): slice-5a predicted source path on Property
Add a `predicted_epc` slot to the Property aggregate and a "predicted" branch to
SourcePath / source_path / effective_epc (ADR-0031 decisions 1+3). A
neighbour-synthesised EpcPropertyData resolves as the Effective EPC ONLY when
there is neither a lodged EPC nor Site Notes — a real source always wins
(prediction is last-resort gap-fill). The slot is distinct from `epc` so a
predicted picture coexists with any lodged one (provenance is structural, not a
flag on EpcPropertyData); downstream consumers are untouched.

3 tests: predicted resolves when sole source; lodged EPC wins over predicted;
Site Notes win over predicted. 10/10 green, pyright strict clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 03:33:47 +00:00
Khalim Conn-Kowlessar
d1227fd0c6 docs(epc-prediction): ADR-0031 production wiring + CONTEXT gating rule
Resolve the slice-5 design tree (grill-with-docs): estimation runs in Ingestion
(refines ADR-0029 dec-3; drops the #1227 "shift to Modelling" — no surviving
rationale, and stages communicate only via persisted state); predicted EPC is
persisted in a DISTINCT slot (EPC table + source discriminator) so lodged +
predicted coexist (enables EPC Anomaly Flags); provenance is structural (the
slot), not a field on EpcPropertyData; effective_epc/source_path gain a
"predicted" branch; slice-5 is gap-fill only; property_type is a REQUIRED input
(hard cohort filter) from Landlord Overrides, and Properties with unknown type
are gated out (no national defaults). OS postcode_search as a broader type
source is a noted follow-on. CONTEXT EPC Prediction entry gains the gating rule.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 03:23:12 +00:00
Khalim Conn-Kowlessar
58d5b17145 chore(epc-prediction): dense-corpus fetcher + cross-postcode geo no-go
Build a geographically DENSE postcode-clustered corpus to test cross-postcode
geo expansion (the handover's anticipated "real geo payoff"). The gov EPC API
has no area/prefix search (a partial postcode 400s; the old opendatacommunities
partial-search API is decommissioned), so neighbourhood enumeration is external:
seed K postcodes nationally, expand each via postcodes.io's nearest-postcode
endpoint into every unit within RADIUS_M, pull each one's full EPC cohort.
postcodes.io is a corpus-BUILD dependency only — the predictor stays pure. Same
on-disk layout as the scattered corpus, so load_corpus + the coords resolver
consume it unchanged.

MEASURE-FIRST RESULT — cross-postcode expansion is a NO-GO. On a 2-seed pilot
(York YO19 + Islington N51, 81 postcodes / 1558 certs, 140 SAP-10.2 targets),
pooling nearby postcodes regresses accuracy across the board:
  same-postcode  FA_MAE 9.53  wall 92%  age 72%  floor_con 85%  cylinder 91%
  cross <=0.3km  FA_MAE 13.1  wall 80%  age 61%  floor_con 82%  cylinder 79%
Even as a thin-cohort top-up it hurts (thin n=18: FA 5.24 -> 7.15). Root cause:
the postcode boundary is itself a strong homogeneity prior (a postcode is one
coherent street/development), so same-postcode neighbours beat geographically
near cross-boundary ones even when the home postcode is sparse (and they rarely
are — median same-postcode cohort here is 34). Geo-proximity helps WITHIN a
postcode (#1227) but does not survive crossing the boundary. Cross-postcode geo
closed; geo weighting stays intra-postcode. Tooling kept (reusable).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 03:03:15 +00:00
Khalim Conn-Kowlessar
be3e51bae9 feat(epc-prediction): geo-proximity-weighted floor-area median
Size the predicted dwelling from the geo-proximity-weighted median of the
cohort's floor areas rather than the plain median: homes built together share a
footprint, so a nearer neighbour's area should count for more (the same street
signal #1227 already wired into age / wall / glazing). Reuses `_geo_weights` and
adds `_weighted_median`, which reduces exactly to `statistics.median` under
uniform weights (geo off / no target coordinates) — including the even-count
midpoint average — so the MAD-minimising guarantee is preserved.

Measured over the 514-target SAP-10.2 corpus (leave-one-out):
  floor_area MAE  10.48 -> 9.73 m²   MAPE 13.2% -> 12.2%

Re-baselines the n=36 fixture floor_area ceiling 11.8983 -> 12.0378 (a method
change, not a loosening; the small fixture subset moved +0.14 the other way as
sample noise while the population improved decisively). The ceiling still pins
the new deterministic value exactly, so the tighten-only ratchet resumes.

Investigation ruling out the adjacent floor-area levers (kept in the follow-up):
lowering minimum_cohort (9.78-10.03, worse), hard same-form filter (10.19),
mean instead of median (10.68), constant bias correction (10.47),
extension-conditioning (oracle 9.50, not worth the misclassification cost) and
room-in-roof conditioning/additive (RiR is a confound for large multi-part
outliers — RiR area is only ~21% of total, and the increment breaks the homes
already predicted exactly). Remaining cohort lever is built-form soft-weighting,
gated on a denser corpus.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:08:05 +00:00
Khalim Conn-Kowlessar
da3fc92d53 docs(epc-prediction): handover for the accuracy backlog + geo work
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 15:12:00 +00:00
Khalim Conn-Kowlessar
d8f015fb0e feat(epc-prediction): report floor-area MAE + MAPE vs typical size
Adds a floor_area line giving MAE (m2), MAPE (% of actual), and the typical
(median actual) size, so the absolute error reads relative to dwelling size.
Corpus: MAE 10.48 m2 / MAPE 13.2% / typical 61 m2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 15:07:22 +00:00
Khalim Conn-Kowlessar
aea2d7150f test(epc-prediction): re-baseline modal_glazing floor after main merge
main's 'ND' multiple_glazing_type mapper fix (361abc12) changes the mapped
ground-truth glazing for one fixture cert, so modal_glazing_type re-baselines
0.5833 -> 0.5556 (21/36 -> 20/36). A mapper change shifts the deterministic
fixture rates like a fixture change does — re-baseline, not a prediction
regression. All other component floors + residual ceilings unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 15:04:34 +00:00
Khalim Conn-Kowlessar
0b2827e9ff Merge remote-tracking branch 'origin/main' into feature/epc-prediction 2026-06-15 15:03:27 +00:00
Khalim Conn-Kowlessar
1f26703dc5 feat(epc-prediction): geo-proximity weighting, per-component (#1227)
Folds a haversine distance kernel into the categorical-mode weighting so a
nearer neighbour counts for more — applied ONLY to the components that showed
a clear distance signal in the corpus pre-check (age band, wall + floor
construction, glazing: homes built/retrofitted together cluster). Roof
construction showed no decay and is excluded; heating keeps its coherent
donor. Predictor stays pure: weights come from target.coordinates vs each
Comparable.coordinates (resolved at the boundary); geo is OFF when the target
has no coords, neutral for a neighbour with none.

Scale chosen on the harness: _GEO_SCALE_KM=0.1 is the gate-safe optimum
(0.05 lifts the corpus more but regresses fixture floor_construction).
Corpus (150pc/514, geo off->on): age 0.564->0.572, age_pm1 0.841->0.847,
wall 0.902->0.912, floor_con 0.786->0.796, glazing 0.667->0.673; roof
unchanged. Fixture: glazing 0.5278->0.5833 (floor ratcheted), all else held.

Refactored recency into a reusable _recency_weights vector composed via
_combine, so similarity/recency/geo factors multiply uniformly. Fixture ships
a committed _coordinates.json (OGL OS OpenData; build script carries it from
the corpus sidecar on rebuild) so the gate exercises geo without S3.

This is the per-component method applied to geography ([[feedback_per_component_best_method]]).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:58:42 +00:00
Khalim Conn-Kowlessar
fdc314c857 feat(epc-prediction): thread coordinates onto Comparable + target (#1227)
Adds coordinates: Optional[Coordinates] to Comparable and PredictionTarget
(data carriers — the pure predictor stays IO-free), and wires load_corpus to
read an optional _coordinates.json sidecar ({uprn: [lon, lat]}) and populate
each Comparable from its cert's uprn; iter_predictions threads the held-out
target's coordinates through. Absent sidecar -> geo-weighting stays off (no
behaviour change yet — weighting lands next slice). fetch_corpus_coordinates
now writes the sidecar into the corpus dir. load_corpus populates 99% of
corpus comparables.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:46:01 +00:00
Daniel Roth
9d56cd7c1e
Merge pull request #1234 from Hestia-Homes/feature/deploy-sharepoint-renamer
Deploy sharepoint renamer: Correct dockerfile imports
2026-06-15 15:35:55 +01:00
Khalim Conn-Kowlessar
95719dd587 feat(geospatial): batch coordinates_for_uprns lookup (#1227)
Adds GeospatialRepository.coordinates_for_uprns(uprns) -> dict — a batch
coordinate lookup returning only covered UPRNs. The S3 adapter overrides it
to read the meta once, group UPRNs by their covering partition, and read each
partition once for all the UPRNs it covers; co-located (closely-numbered)
UPRNs share a partition, so an EPC Prediction cohort is typically one or two
reads instead of one per neighbour. Default port impl is a per-UPRN loop.

Feeds the EPC Prediction geo-proximity work: a cohort's UPRNs resolve to
coordinates in a couple of reads (validated at corpus scale: 170 partition
reads for 2683 UPRNs).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:35:32 +00:00
Daniel Roth
b31db4b58b correct Dockerfile imports 2026-06-15 14:29:04 +00:00
Khalim Conn-Kowlessar
c0a1bcac95 feat(epc-prediction): resolve corpus UPRN coordinates from S3 (#1227 signal check)
One-time utility: resolves every corpus cert's uprn -> WGS84 lon/lat from the
OS Open-UPRN parquet (DATA_BUCKET/spatial/) via boto3, grouping UPRNs by their
covering partition so each ~1.7MB partition is read at most once (the efficient
batch lookup we intend to add to GeospatialRepository). Caches {uprn:[lon,lat]}
locally for the validation harness. Resolved 2609/2683 corpus UPRNs (97%).

Signal pre-check result (does intra-postcode proximity predict components?):
intra-postcode distances are non-trivial (median 44m, p90 138m, max ~1km),
and nearer neighbours match the target markedly better on age band (0.63 at
<20m -> 0.16 at >300m), wall, glazing and floor construction. Roof shows no
decay. => geo-proximity is worth building, per-component (strongest for age,
the weakest fabric component).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:28:39 +00:00
Daniel Roth
0ed17cfd39 Merge branch 'main' into feature/deploy-sharepoint-renamer 2026-06-15 14:24:10 +00:00
Jun-te Kim
6226575086
Merge pull request #1232 from Hestia-Homes/feature/deploy-sharepoint-renamer
Sharepoint renamer: fix terraform issue and add dry_run option
2026-06-15 15:13:02 +01:00
Daniel Roth
8b27a5fda2 correct lambda name 2026-06-15 14:08:40 +00:00
Daniel Roth
1af9d84f94 Merge branch 'main' into feature/deploy-sharepoint-renamer 2026-06-15 14:07:27 +00:00
Daniel Roth
963b7d70fe fix terraform error and pass handler bool for dry runs 2026-06-15 14:06:54 +00:00
Khalim Conn-Kowlessar
4afab2c3d8 feat(epc-prediction): roof-insulation +/-1-bucket reporting
Adds roof_insulation_thickness_pm1 (mirrors construction_age_band_pm1, issue
#1222): adjacent RdSAP thickness buckets (0/NI,12mm..400mm+) carry near-
identical roof U-values, so an off-by-one bucket is a SAP-neutral hit. 'ND'
(no-data) is off the ordered scale, so only an exact match counts there.
Honest measurement of SAP-relevant roof-insulation quality.

Corpus (150pc/514): exact 49.3% -> +/-1 53.7% (the misses are often multiple
buckets or ND, so the band gain is smaller than age's). Fixture: exact ==
+/-1 (0.4118) — its misses are all >1 bucket; gate floor added at 0.4118.

Also fixes two pre-existing pyright errors in the touched test file
(_epc main_fuel_type/main_heating_control were Optional but the
MainHeatingDetail attributes are non-optional Union[int, str]).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:04:18 +00:00
Jun-te Kim
5a3228ab5e
Merge pull request #1217 from Hestia-Homes/feature/per-cert-mapper-validation
Feature/per cert mapper validation
2026-06-15 15:03:05 +01:00
Khalim Conn-Kowlessar
fffb07d04b test(harness): re-pin golden-cert plans to the gain-maximising packages
Three more pre-existing failures (present at 9ee38211, before this branch's
recent commits; same family as the orchestration multi-measure re-pin) —
golden-cert plan expectations that predate the ASHP generator (ADR-0025)
and the optimiser folding forced dependencies into candidate gain (ADR-0016):

- test_console: a multi-measure plan now leads with air_source_heat_pump,
  not cavity_wall_insulation (which is dropped — its forced ventilation makes
  the pair net-negative). Assert a measure actually in the package.
- test_report 0330: package is now {solid_floor_insulation, air_source_heat_
  pump}; cavity_wall + forced mechanical_ventilation correctly excluded.
- test_report 0036: gain-maximising package is now {solid_floor_insulation,
  low_energy_lighting}.

Same verified-correct optimiser evolution as 077e3a39 (cavity_wall +2.9 SAP
alone but its forced fabric→ventilation dep drags the pair net-negative).
Re-pin to the actual packages + their trigger fields; the forced wall→vent
edge stays covered by test_measure_dependency / test_optimiser.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:57:27 +00:00
Khalim Conn-Kowlessar
7f48495ed5 feat(epc-prediction): surface CO2 + PEI calculator floors in the report (#1228)
The validation report showed only the SAP calculator floor (calc(actual) vs
lodged), so the headline PEI MAE (~40 kWh/m2) read as prediction error when
much of it is the calculator's own API-path residual. Adds the CO2 + PEI
floors alongside SAP.

Diagnostic (150pc/514): PEI floor MAE 15.73 (calc(actual) vs lodged) vs SAP
floor 1.57; calc(actual)/lodged PEI ratio ~1.06 (mean +10.7, ~+6% over-
estimate). That RULES OUT the suspected gross unit/definition mismatch (a
unit bug would be ~2x/3.6x, not 1.06) and reframes #1228: the PEI gap is a
modest calculator bias (~16 floor, calc-branch) plus a larger prediction-
sensitivity term (~24) — PEI is far more prediction-sensitive than SAP.
CO2 floor 0.20 t. Script-only; no gate impact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:55:20 +00:00
Khalim Conn-Kowlessar
06a66b3dd9 feat(epc-prediction): coherent heating donor selection (#1225)
Heating sub-fields can't be field-moded without breaking system coherence,
so the whole SapHeating cluster is now copied as a unit from a single
coherent donor rather than inherited from the structural template: the
neighbour matching the cohort's modal heating signature (main fuel +
category + cylinder presence), most recent among the matches (recent cert =
current system). Including cylinder presence in the signature is load-bearing
— it protects has_hot_water_cylinder + cylinder_insulation (a bare fuel+cat
signature regressed them).

Corpus (150pc/514): heating_main_control 66.3 -> 73.9% (+7.6, the target),
main_fuel 92.8 -> 96.9, category 90.7 -> 95.7, water_fuel 92.8 -> 96.3,
water_code 88.5 -> 95.3, has_cylinder 81.1 -> 89.7, secondary 36.2 -> 42.0.
SAP MAE vs lodged 7.08 -> 6.00 (calculator floor 1.57). cylinder_insulation
-13.6 corpus (tiny-n) but +33pp on the fixture; AC requires control up +
fuel/category hold + SAP not worsened, all met.

Gate (36-target fixture): zero regression; ratcheted main_category
0.8889->0.9444, main_control 0.7500->0.8056, water_fuel 0.9167->0.9722,
water_code 0.8889->0.9444, cylinder_insulation_type 0.1667->0.5000. This is
the per-component heating method ([[feedback_per_component_best_method]]):
coherent donor, never field-mode.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:48:15 +00:00
Khalim Conn-Kowlessar
077e3a3947 test(orchestration): re-pin multi-measure plan to the gain-maximising package
The optimiser-package expectation was stale: it predated the optimiser
folding a triggered measure's forced dependency into its candidate gain
(ADR-0016). The run considers ALL measures (considered_measures defaults
to None — no restriction), so once the ASHP bundle became SAP-beneficial
(ADR-0025) the gain-maximising package shifted.

Verified the new package is CORRECT, not a regression: on the test EPC,
cavity-wall insulation earns +2.9 SAP alone but its forced fabric→
ventilation dependency (ADR-0016) drags the wall+ventilation pair to a
NET −1.8 SAP (−0.9 on top of the ASHP package), so the gain-maximising
Optimiser correctly excludes the wall and its forced ventilation. Update
the expected set to {air_source_heat_pump, suspended_floor_insulation,
low_energy_lighting, secondary_heating_removal} and drop the wall/vent-
specific assertions — the forced wall→ventilation edge is covered by
test_measure_dependency / test_optimiser; this integration test keeps its
end-to-end optimise→persist→telescope coverage on the chosen package.

Pre-existing failure (present before this branch's recent commits), outside
the handover regression gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:46:22 +00:00
Khalim Conn-Kowlessar
d762b25808 feat(epc-prediction): recency-weighted glazing mode (#1223)
Per-component method: glazing type is now the recency-weighted cohort mode
applied to every predicted window, rather than copied from the template.
Glazing is retrofitted over a dwelling's life (single -> double), so a
recent neighbour reflects the current state — same family as roof-insulation
thickness. Recency is the CORRECT weighting here: plain moding regressed the
fixture (-5.6pp) and was previously reverted; similarity weighting also
regressed it; recency improves BOTH (window geometry stays on the template,
only the glazing categorical moves).

modal_glazing_type: corpus (150pc/514) 60.7 -> 66.7% (+6.0pp); fixture
0.5000 -> 0.5278 (floor ratcheted up). Heating, geometry residuals and all
other components unchanged. Refactored _recency_weighted_mode to a reusable
_recency_weighted_choice(value_of) shared by roof insulation + glazing.

Closes the #1223 per-component approach: floor-area (median estimate) +
glazing (recency) shipped as distinct best-fit methods rather than a global
recency template, which would have disturbed the coherence-coupled heating
cluster.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:35:03 +00:00
Khalim Conn-Kowlessar
4fdc23f83d test(worksheet): pin simulated case 38 — mains-gas secondary reproduces worksheet exactly
The realistic re-generation of case 37 (code-117 gas boiler, control 2102,
+ a MAINS-GAS condensing gas-fire secondary code 611, vs case 37's biogas
605). The full extractor -> mapper -> calculator pipeline reproduces the
worksheet's SAP-rating block EXACTLY: continuous SAP 60.9152 (Δ 2e-5) and
(272) CO2 5801.0770 (Δ ~0). This confirms the boiler-efficiency /
control-2102 −5pp interlock / secondary-fuel handling are all correct, and
that case 37's +7 gap was purely the biogas sub-fuel the Summary export
cannot carry.

Summary mirrored into backend/documents_parser/tests/fixtures so the pin
runs without the unstaged workspace. PE not pinned — it is a separate
DPER block (different scope) already guarded by the corpus PE gauge.
Worksheet harness 47/47 unchanged; pyright net-zero.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:31:36 +00:00
Khalim Conn-Kowlessar
51cdc25ce8 feat(epc-prediction): cohort-median floor-area estimate (#1223)
Per-component method, not a global template change: the predicted floor
area is now the cohort median (the MAD-minimising point estimate of the
target's size) rather than whichever structural template's own area. The
calculator derives heat loss from building-part geometry, not this scalar,
so decoupling them is safe and the scalar becomes a better size estimate.

floor_area mean|.|: corpus (150pc/514 targets) 10.62 -> 10.48; fixture
12.2175 -> 11.8983 (ceiling ratcheted down). No other component moves.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:30:33 +00:00
Daniel Roth
17420408e4
Merge pull request #1230 from Hestia-Homes/feature/deploy-sharepoint-renamer
Deploy sharepoint renamer
2026-06-15 13:45:52 +01:00
Daniel Roth
b9cbea367d correct import in test file 2026-06-15 12:21:32 +00:00
Daniel Roth
a6050fc1c7 remove tests/ from pytest.ini 2026-06-15 12:04:33 +00:00
Daniel Roth
0fc81da4cf move input files out of scripts/ 2026-06-15 11:14:09 +00:00
Daniel Roth
5c314e2914 move tests out of scripts/ 2026-06-15 11:11:08 +00:00
Daniel Roth
38b9e63844 revert pytest.ini 2026-06-15 11:02:48 +00:00
Daniel Roth
beb4e5d0d9 Move SharePoint renamer logic from scripts/ into orchestrator and app-root handler 2026-06-15 11:01:51 +00:00
Daniel Roth
8cb0e986e6 Deploy SharePoint renamer as Lambda with SQS trigger 🟩 2026-06-15 10:52:52 +00:00
Daniel Roth
b3e9d858d9 SharePoint renamer Lambda handler stub created 🟥 2026-06-15 10:49:01 +00:00
Daniel Roth
383b8b0c37 SharePoint renamer build_canonical_filename behaviour verified by tests 🟩 2026-06-15 10:48:17 +00:00
Daniel Roth
9daf6a8668
Merge pull request #1221 from Hestia-Homes/improve-sharepoint-renamer
Sharepoint renamer recursively looks for files in subfolders
2026-06-15 11:18:03 +01:00
Khalim Conn-Kowlessar
c11eb46b8a fix(modelling): HHR overlay sets off-peak immersion type so HW Table 13 applies
The HHR-storage HeatingOverlay (ADR-0024) added an off-peak electric
immersion cylinder but never set `immersion_heating_type`, so the overlaid
cert left it None. The calculator then could not resolve `immersion_single`
for the SAP 10.2 Table 13 HW high-rate split and billed hot water 100% at
the off-peak low rate — £127.41 vs the relodged after-cert's £169.39,
overstating the overlay's SAP by +1.26 (CO2/PE matched, isolating it to the
HW cost path).

Add `immersion_heating_type` to HeatingOverlay, route it through
`_fold_heating` (it lives on `sap_heating`), and set it to 1 (single
off-peak immersion) on the HHR overlay to match the relodged reference.
Closes both `test_hhr_storage_overlay_reproduces_the_relodged_after_*`
cascade pins (electric-storage and no-system befores share the after).

Pre-existing failure (present before this branch's recent commits), outside
the handover regression gate. Full modelling suite 220 pass, pyright net-
zero.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 06:53:14 +00:00
Khalim Conn-Kowlessar
361abc1202 fix(mapper): handle 'ND' multiple_glazing_type on RdSAP-Schema-20.0.0
`_synthesise_20_0_0_sap_windows` passed `schema.multiple_glazing_type`
straight into `_api_cascade_glazing_type`, which raised UnmappedApiCode on
the "ND" (Not Defined) string that the 20.0.0 corpus lodges alongside the
1-8 integer codes — failing the mapper-coverage guard on every ND-glazed
20.0.0 cert. Mirror the existing 18.0/19.0/17.x seams: route integer codes
through the cascade, fall the "ND" string back to the DG-modal default
(cascade code 2 → daylight g_L 0.80). Also corrects the 20.0.0 schema
field type `int` → `Union[int, str]` to match the data (as 18.0 already
does), which keeps the isinstance guard pyright-clean.

Pre-existing failure (present before this branch's recent commits), not in
the handover regression gate. Fixes all 15 RdSAP-Schema-20.0.0 ND certs;
test_mapper_corpus 6002/6002 pass. pyright net-zero.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 06:43:55 +00:00
Khalim Conn-Kowlessar
6e9f831296 chore(epc-prediction): grow validation corpus to 150 postcodes
Bumps N_POSTCODES 40 -> 150 for the fetch script. Larger corpus (150
postcodes / 3719 certs) reduces leave-one-out variance and unblocks the
recency-template work (#1223), which regressed the noisier 36-target gate
fixture. Corpus itself stays out of git (gitignored /tmp + persistent
backup at /workspaces/home/epc_prediction_corpus_backup).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 06:42:19 +00:00
Khalim Conn-Kowlessar
0fae84d2b6 fix(elmhurst): map secondary room-heater SAP codes to Table 4a fuel category
Completes `_elmhurst_secondary_fuel_from_sap_code` per SAP 10.2 §12
(PDF p.34: "Secondary heating systems and applicable fuel types are taken
from the room heaters section of Table 4a") + RdSAP 10 §10.4.1. Each
Table 4a room-heater code now resolves to its fuel CATEGORY's modal fuel:

  - gas room heaters    601-613 → mains gas   (26 → Table 32 1, 3.48 p/kWh)
  - liquid room heaters 621-625 → heating oil (28 → Table 32 4, 5.44 p/kWh)
  - solid room heaters  631-636 → house coal  (11 → Table 32 11, 3.67 p/kWh)
  - electric room htrs   691-694/699/701 → None (cascade electricity default)

Previously only the gas (601-613→26) and solid (631-634→11) blocks were
mapped; liquid heaters (621-625) and 635-636 fell through to None →
silently billed as electricity (13.19 p/kWh), a large mis-price for an
oil/solid heater. The prior slice raised on those; this maps them to the
correct category fuel instead, and keeps the raise ONLY for codes inside
the room-heater range (601-701) that are not a recognised Table 4a row.

The specific sub-fuel within a category (mains gas vs LPG vs biogas) is a
SEPARATE lodgement per §10.4.1 and is NOT exported in the Summary, so the
gas block stays the modal mains gas — worksheet "simulated case 37" lodged
its 605 live-effect fire on biogas (7.60 p/kWh), unrecoverable from the
Summary code alone (this is the entire +7 SAP case-37 gap: secondary
energy £131 + a separate biogas standing charge £70; every other line
matches the worksheet exactly, incl. (206) main efficiency 61%).

5 AAA tests, harness 47/47 (0 raised), pyright net-zero, regression clean,
corpus gauge unchanged (Elmhurst-path only).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 06:27:37 +00:00
Khalim Conn-Kowlessar
718455e971 feat(epc-prediction): physical-similarity-weighted categorical mode (#1224)
ADR-0029 decision 5: survivors were treated equally; now each neighbour's
vote in the cohort mode decays with its distance from the cohort's physical
centre (floor area from the median, age band from the modal band), so the
mode leans on the most representative neighbours instead of being swayed by
size/era outliers. Scales (size 20 m^2, age weight 0.5) chosen on the
validation corpus; the tight size kernel is load-bearing (looser scales
regress floor_insulation on the fixture).

Corpus (181 SAP-10.2 targets): wall_insulation 83.4->86.2%,
roof_construction 86.2->87.3%, floor_construction 78.8->81.2%,
floor_insulation 92.9->94.1%; net +7.5pp gained vs -1.1pp (two 1-cert dips,
both held on the fixture). Geometry/residuals untouched (template unchanged).

Gate (36-target fixture): zero regression across all 24 floors/ceilings;
ratcheted wall_insulation_type 0.7778->0.8333, floor_construction
0.7500->0.8125, floor_insulation 0.9062->0.9375. Dead _mode/_int_mode
removed (superseded by the weighted variants).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 10:46:51 +00:00
Khalim Conn-Kowlessar
07051b9401 feat(epc-prediction): per-prediction confidence signal (#1226)
Adds PredictionConfidence (cohort size + per-component agreement = the
modal value's share among neighbours that lodge one) and
EpcPrediction.confidence(), a compute-only signal so downstream can flag
low-confidence components (ADR-0029 open item: 'confidence signal').

Sanity check on the 40-postcode corpus (1068 component predictions):
agreement is strongly predictive of correctness — pooled hit-rate 21.9%
(<0.5) / 46.7% (0.5-0.7) / 73.6% (0.7-0.9) / 95.5% (>=0.9); point-biserial
corr(agreement, correct) = 0.582. Cohort size tracks too (<6 -> 68.4%,
>=20 -> 96.0%). Surfacing / persistence is a separate HITL follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 10:35:59 +00:00