Commit graph

6536 commits

Author SHA1 Message Date
Khalim Conn-Kowlessar
c0a1bcac95 feat(epc-prediction): resolve corpus UPRN coordinates from S3 (#1227 signal check)
One-time utility: resolves every corpus cert's uprn -> WGS84 lon/lat from the
OS Open-UPRN parquet (DATA_BUCKET/spatial/) via boto3, grouping UPRNs by their
covering partition so each ~1.7MB partition is read at most once (the efficient
batch lookup we intend to add to GeospatialRepository). Caches {uprn:[lon,lat]}
locally for the validation harness. Resolved 2609/2683 corpus UPRNs (97%).

Signal pre-check result (does intra-postcode proximity predict components?):
intra-postcode distances are non-trivial (median 44m, p90 138m, max ~1km),
and nearer neighbours match the target markedly better on age band (0.63 at
<20m -> 0.16 at >300m), wall, glazing and floor construction. Roof shows no
decay. => geo-proximity is worth building, per-component (strongest for age,
the weakest fabric component).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:28:39 +00:00
Khalim Conn-Kowlessar
4afab2c3d8 feat(epc-prediction): roof-insulation +/-1-bucket reporting
Adds roof_insulation_thickness_pm1 (mirrors construction_age_band_pm1, issue
#1222): adjacent RdSAP thickness buckets (0/NI,12mm..400mm+) carry near-
identical roof U-values, so an off-by-one bucket is a SAP-neutral hit. 'ND'
(no-data) is off the ordered scale, so only an exact match counts there.
Honest measurement of SAP-relevant roof-insulation quality.

Corpus (150pc/514): exact 49.3% -> +/-1 53.7% (the misses are often multiple
buckets or ND, so the band gain is smaller than age's). Fixture: exact ==
+/-1 (0.4118) — its misses are all >1 bucket; gate floor added at 0.4118.

Also fixes two pre-existing pyright errors in the touched test file
(_epc main_fuel_type/main_heating_control were Optional but the
MainHeatingDetail attributes are non-optional Union[int, str]).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:04:18 +00:00
Khalim Conn-Kowlessar
7f48495ed5 feat(epc-prediction): surface CO2 + PEI calculator floors in the report (#1228)
The validation report showed only the SAP calculator floor (calc(actual) vs
lodged), so the headline PEI MAE (~40 kWh/m2) read as prediction error when
much of it is the calculator's own API-path residual. Adds the CO2 + PEI
floors alongside SAP.

Diagnostic (150pc/514): PEI floor MAE 15.73 (calc(actual) vs lodged) vs SAP
floor 1.57; calc(actual)/lodged PEI ratio ~1.06 (mean +10.7, ~+6% over-
estimate). That RULES OUT the suspected gross unit/definition mismatch (a
unit bug would be ~2x/3.6x, not 1.06) and reframes #1228: the PEI gap is a
modest calculator bias (~16 floor, calc-branch) plus a larger prediction-
sensitivity term (~24) — PEI is far more prediction-sensitive than SAP.
CO2 floor 0.20 t. Script-only; no gate impact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:55:20 +00:00
Khalim Conn-Kowlessar
06a66b3dd9 feat(epc-prediction): coherent heating donor selection (#1225)
Heating sub-fields can't be field-moded without breaking system coherence,
so the whole SapHeating cluster is now copied as a unit from a single
coherent donor rather than inherited from the structural template: the
neighbour matching the cohort's modal heating signature (main fuel +
category + cylinder presence), most recent among the matches (recent cert =
current system). Including cylinder presence in the signature is load-bearing
— it protects has_hot_water_cylinder + cylinder_insulation (a bare fuel+cat
signature regressed them).

Corpus (150pc/514): heating_main_control 66.3 -> 73.9% (+7.6, the target),
main_fuel 92.8 -> 96.9, category 90.7 -> 95.7, water_fuel 92.8 -> 96.3,
water_code 88.5 -> 95.3, has_cylinder 81.1 -> 89.7, secondary 36.2 -> 42.0.
SAP MAE vs lodged 7.08 -> 6.00 (calculator floor 1.57). cylinder_insulation
-13.6 corpus (tiny-n) but +33pp on the fixture; AC requires control up +
fuel/category hold + SAP not worsened, all met.

Gate (36-target fixture): zero regression; ratcheted main_category
0.8889->0.9444, main_control 0.7500->0.8056, water_fuel 0.9167->0.9722,
water_code 0.8889->0.9444, cylinder_insulation_type 0.1667->0.5000. This is
the per-component heating method ([[feedback_per_component_best_method]]):
coherent donor, never field-mode.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:48:15 +00:00
Khalim Conn-Kowlessar
d762b25808 feat(epc-prediction): recency-weighted glazing mode (#1223)
Per-component method: glazing type is now the recency-weighted cohort mode
applied to every predicted window, rather than copied from the template.
Glazing is retrofitted over a dwelling's life (single -> double), so a
recent neighbour reflects the current state — same family as roof-insulation
thickness. Recency is the CORRECT weighting here: plain moding regressed the
fixture (-5.6pp) and was previously reverted; similarity weighting also
regressed it; recency improves BOTH (window geometry stays on the template,
only the glazing categorical moves).

modal_glazing_type: corpus (150pc/514) 60.7 -> 66.7% (+6.0pp); fixture
0.5000 -> 0.5278 (floor ratcheted up). Heating, geometry residuals and all
other components unchanged. Refactored _recency_weighted_mode to a reusable
_recency_weighted_choice(value_of) shared by roof insulation + glazing.

Closes the #1223 per-component approach: floor-area (median estimate) +
glazing (recency) shipped as distinct best-fit methods rather than a global
recency template, which would have disturbed the coherence-coupled heating
cluster.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:35:03 +00:00
Khalim Conn-Kowlessar
51cdc25ce8 feat(epc-prediction): cohort-median floor-area estimate (#1223)
Per-component method, not a global template change: the predicted floor
area is now the cohort median (the MAD-minimising point estimate of the
target's size) rather than whichever structural template's own area. The
calculator derives heat loss from building-part geometry, not this scalar,
so decoupling them is safe and the scalar becomes a better size estimate.

floor_area mean|.|: corpus (150pc/514 targets) 10.62 -> 10.48; fixture
12.2175 -> 11.8983 (ceiling ratcheted down). No other component moves.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:30:33 +00:00
Khalim Conn-Kowlessar
6e9f831296 chore(epc-prediction): grow validation corpus to 150 postcodes
Bumps N_POSTCODES 40 -> 150 for the fetch script. Larger corpus (150
postcodes / 3719 certs) reduces leave-one-out variance and unblocks the
recency-template work (#1223), which regressed the noisier 36-target gate
fixture. Corpus itself stays out of git (gitignored /tmp + persistent
backup at /workspaces/home/epc_prediction_corpus_backup).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 06:42:19 +00:00
Khalim Conn-Kowlessar
718455e971 feat(epc-prediction): physical-similarity-weighted categorical mode (#1224)
ADR-0029 decision 5: survivors were treated equally; now each neighbour's
vote in the cohort mode decays with its distance from the cohort's physical
centre (floor area from the median, age band from the modal band), so the
mode leans on the most representative neighbours instead of being swayed by
size/era outliers. Scales (size 20 m^2, age weight 0.5) chosen on the
validation corpus; the tight size kernel is load-bearing (looser scales
regress floor_insulation on the fixture).

Corpus (181 SAP-10.2 targets): wall_insulation 83.4->86.2%,
roof_construction 86.2->87.3%, floor_construction 78.8->81.2%,
floor_insulation 92.9->94.1%; net +7.5pp gained vs -1.1pp (two 1-cert dips,
both held on the fixture). Geometry/residuals untouched (template unchanged).

Gate (36-target fixture): zero regression across all 24 floors/ceilings;
ratcheted wall_insulation_type 0.7778->0.8333, floor_construction
0.7500->0.8125, floor_insulation 0.9062->0.9375. Dead _mode/_int_mode
removed (superseded by the weighted variants).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 10:46:51 +00:00
Khalim Conn-Kowlessar
07051b9401 feat(epc-prediction): per-prediction confidence signal (#1226)
Adds PredictionConfidence (cohort size + per-component agreement = the
modal value's share among neighbours that lodge one) and
EpcPrediction.confidence(), a compute-only signal so downstream can flag
low-confidence components (ADR-0029 open item: 'confidence signal').

Sanity check on the 40-postcode corpus (1068 component predictions):
agreement is strongly predictive of correctness — pooled hit-rate 21.9%
(<0.5) / 46.7% (0.5-0.7) / 73.6% (0.7-0.9) / 95.5% (>=0.9); point-biserial
corr(agreement, correct) = 0.582. Cohort size tracks too (<6 -> 68.4%,
>=20 -> 96.0%). Surfacing / persistence is a separate HITL follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 10:35:59 +00:00
Khalim Conn-Kowlessar
ffaedd8d14 feat(epc-prediction): ±1-band age scoring + window_count cosmetic (#1222)
Measurement honesty so we optimise SAP-relevant accuracy, not SAP-neutral
misses (ADR-0030 Component Accuracy):
- Add construction_age_band_pm1: an exact-or-adjacent-band hit. Adjacent
  RdSAP age bands carry near-identical U-values, so an off-by-one is
  ~SAP-neutral. Full corpus: exact 78.5% but ±1-band 91.7% (fixture
  63.9% -> 83.3%) — most age misses are adjacent.
- Drop window_count from the gate's residual ceilings (cosmetic): the
  predicted picture clusters at a mapper-default 4 windows vs actuals 1-21,
  but total_window_area (the SAP-relevant signal) stays tight at ~3.4 m2.

Gate: + construction_age_band_pm1 floor 0.8333; window_count no longer gated.

Closes #1222

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 10:01:20 +00:00
Khalim Conn-Kowlessar
a5b7310911 feat(epc-prediction): recency-weighted mode for roof insulation (ADR-0029/0030)
Investigated recency-weighting (weight cohort votes by an exponential decay
in cert age). Key finding: it must be SELECTIVE. On the validation corpus it
HURTS permanent categoricals (wall 91.2->89.5, age 78.5->75.7 — discards
still-valid data) but clearly HELPS time-varying ones, where a recent
neighbour reflects the current physical state:
  roof_insulation_thickness  56.7 -> 60.7%  corpus   (+4pp)
                             29.4 -> 41.2%  fixture  (+12pp)

So apply a recency-weighted mode only to roof_insulation_thickness (loft
top-ups happen over time); keep the plain mode for permanent categoricals.
tau = 4yr (~2.8yr half-life); falls back to plain mode when no registration
dates are lodged. Gate floor ratcheted 0.2941 -> 0.4118.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 09:45:22 +00:00
Khalim Conn-Kowlessar
9dd23477ac feat(epc-prediction): cohort-mode roof + floor insulation (ADR-0030)
These independent fabric categoricals were template-copied; mode them like
the construction categoricals. Verified mode beats template before applying.
Big fixture win on roof insulation thickness (doubled), floor insulation
neutral-to-positive:
  roof_insulation_thickness  14.7% -> 29.4%  (gate floor ratcheted up)
  floor_insulation           90.6% (unchanged on the fixture)

Glazing type was tried too (+1.6pp on the 40-postcode corpus) but REGRESSED
the 36-target fixture (0.50 -> 0.44) — the gate caught it. Glazing moding is
marginal/noisy, so it's left on the template; revisit with a larger corpus.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 09:37:45 +00:00
Khalim Conn-Kowlessar
a622f97d27 docs(adr): ADR-0030 — record S3-hosted Tier-1.5 scale run
Tier-2 (full national bulk streaming) is deferred. The near-term scale
validation is a Tier-1.5: a few-thousand-cert anonymised corpus stored in
S3 (too large to commit, far more stable than the 36-target gate fixture),
pulled to a temp dir and run through the same load_corpus +
evaluate_component_accuracy. Reuses the committed-fixture machinery wholesale
— only the data source differs. One scorer, three data sources (committed
fixture / S3 corpus / bulk stream).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 09:25:58 +00:00
Khalim Conn-Kowlessar
e3a2720e5c feat(epc-prediction): Tier-1 ratcheting Component Accuracy gate (ADR-0030)
The committed CI gate: run the calculator-free leave-one-out scorer over the
frozen anonymised fixture (36 SAP-10.2 targets) and assert each per-component
classification rate / geometry residual is no worse than a committed baseline.
Prediction is deterministic + the fixture frozen, so the numbers reproduce
exactly — a failure is a real regression, never sample noise.

- 19 rate floors + 5 residual ceilings, seeded at the currently-measured
  values; they only ever tighten (no-widening ethos on an aggregate).
- Calculator-FREE — component floors are the real gate; the end-to-end
  SAP/carbon/PE guards stay out (their floor is the separate API-path
  calculator workstream).
- Skips with a message when the fixture is absent.

25 parametrized assertions, all green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 09:19:39 +00:00
Khalim Conn-Kowlessar
008c1922c4 feat(epc-prediction): anonymised Tier-1 fixture + builder (ADR-0030)
The committed gate needs frozen, reproducible data without dumping real UK
addresses into the repo. Add:
- harness anonymise_payload + stable_hash: hash street address + cert number
  into opaque, dedup-stable tokens; blank secondary address lines + post_town;
  keep postcode + all component/lodged fields (gov data is OGL). Unit-tested.
- scripts/build_epc_prediction_fixture.py: curate qualifying postcodes (>=1
  SAP 10.2 target + >=2 distinct addresses) from the local scratch corpus,
  anonymise, freeze under tests/fixtures/epc_prediction/.
- The frozen fixture: 15 postcodes / 280 certs / 36 SAP-10.2 targets.
  Verified no plaintext address_line_1 and post_town all blank.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 09:17:27 +00:00
Khalim Conn-Kowlessar
027ee1fba3 refactor(epc-prediction): extract shared leave-one-out scorer + corpus loader (ADR-0030)
"One scorer, two harnesses" (ADR-0030): the committed gate, the local script,
and the future battle-test must run the *same* scoring. Extract it:

- domain/epc_prediction/validation.py — `iter_predictions` (the single
  leave-one-out orchestration: latest-per-address hold-out, SAP-10.2 target
  filter, all-vintage source) + `evaluate_component_accuracy` (calculator-free
  ComponentAccuracy aggregation, the primary signal). Unit-tested.
- harness/epc_prediction_corpus.py — `load_corpus(dir)` IO: corpus dir ->
  Comparable cohorts (maps payloads, carries address + registration_date).

validate_epc_prediction.py now just loads + calls the scorer for the component
section and iterates iter_predictions for the calculator-floored end-to-end.
Identical numbers (181 targets, SAP MAE 6.34) — behaviour-preserving.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 09:12:08 +00:00
Khalim Conn-Kowlessar
65cb094abe feat(epc-prediction): SAP-10.2 target filter + carbon/PE end-to-end (ADR-0030)
Make the leave-one-out runner ADR-0030-compliant:
- Hold out only SAP 10.2 targets (sap_version == 10.2) — the source cohort
  keeps every vintage (components are methodology-agnostic).
- Label Component Accuracy as the PRIMARY, calculator-independent section.
- End-to-end vs API-lodged (SECONDARY, calculator-FLOORED): add CO2 (tonnes)
  and PEI (kWh/m2) alongside SAP, using the canonical performance.py mapping
  (co2_kg/1000; primary_energy_kwh_per_m2).
- Add the attribution readout calc(actual) vs lodged SAP — the calculator
  floor the end-to-end can reach.
- Drop the neighbour-mean-of-lodged-SAP baseline (mixes SAP versions —
  rejected by ADR-0030).

On the 181 SAP-10.2 targets: component rates are higher than the all-vintage
view (age band 60.9 -> 78.5%, floor_area mean|.| 12.7 -> 8.4). End-to-end SAP
MAE 6.34 vs the calc(actual) floor of 3.25 — ~half the gap is the known
API-path calculator residual, not prediction error.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 09:04:24 +00:00
Khalim Conn-Kowlessar
275a30a825 feat(epc-prediction): complete component coverage — fabric/glazing/renewables/doors (ADR-0030)
Finish the ADR-0030 Component Accuracy set: roof insulation thickness,
floor insulation, room-in-roof presence, modal glazing type, PV presence,
solar water heating (categoricals) + door count (residual). Presence flags
(room-in-roof, PV, solar) are always-applicable — predicting absence when
present is a real miss.

Template-copied baseline (40-postcode corpus), newly visible:
  floor_insulation         94.0%   solar_water_heating  99.7%
  has_pv                   98.6%   has_room_in_roof     91.9%
  modal_glazing_type       59.0%   <- weak
  roof_insulation_thickness 30.6%  <- weak
  door_count  mean|.| 0.40

compare_prediction now scores 19 categoricals + 5 residuals across every
SAP-load-bearing component group.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 09:00:30 +00:00
Khalim Conn-Kowlessar
cd43c52cf9 feat(epc-prediction): score the heating components (ADR-0030 Component Accuracy)
Heating is the dominant SAP lever (ablating it to actual cut the SAP error
~7 -> ~4.5) yet was entirely unscored. Add the heating group to
compare_prediction's categorical_hits: main fuel / category / control (off
the primary MainHeatingDetail), water-heating fuel / code, has-cylinder,
cylinder insulation, secondary heating (off SapHeating).

Template-copied baseline on the 40-postcode corpus (no predictor change
yet — this just makes the signal visible):
  heating_main_fuel        93.4%
  heating_main_category    92.7%
  water_heating_fuel/code  91.7% / 92.4%
  heating_main_control     62.1%   <- weak
  has_hot_water_cylinder   78.5%
  cylinder_insulation_type 35.8% (n=120)   <- weak
  secondary_heating_type   16.8% (n=125)   <- weak

Fuel/category predict well from the template; controls, cylinder, and
secondary heating are poor and now drive the next predictor slices.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 08:53:15 +00:00
Khalim Conn-Kowlessar
41b5ce5057 refactor(epc-prediction): name-keyed categorical_hits for Component Accuracy (ADR-0030)
ADR-0030 commits Component Accuracy to ~19 categorical components (5 today
+ 8 heating + glazing/renewables). Flat *_correct dataclass fields don't
scale — each needs manual runner wiring. Collapse them into a single
`categorical_hits: dict[str, Optional[bool]]` keyed by component name, which
also matches the runner's name-keyed aggregation (now generic: it tallies
whatever components the comparison reports). No behaviour change; the
classification rates are identical (wall n 578->575 is the 3 certs whose
actual wall is None, now correctly counted as not-applicable via _classify).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 08:50:34 +00:00
Khalim Conn-Kowlessar
35a7c07812 docs(adr): ADR-0030 — SAP-version-aware, component-first EPC Prediction validation
Records the grilling-session decisions amending ADR-0029's validation:
- Source cohort keeps all cert vintages (components are agnostic of the SAP
  methodology that rated them); only the held-out validation TARGET is
  restricted to SAP 10.2. Amends ADR-0029 decision 5 ("pre-SAP10 dropped").
- Component Accuracy (predicted vs API actual components) is the primary,
  calculator-independent signal. calc(predicted) vs calc(actual) rejected
  (circular ground truth, hides calculator error); neighbour-mean-lodged-SAP
  baseline rejected (mixes SAP versions). calc(predicted) vs API-lodged
  SAP/carbon/PE kept as a secondary, calculator-floored guard.
- Two tiers: committed anonymized fixture (ratcheting CI gate) + bulk-export
  national battle-test on harness/epc_bulk.py + harness/cohort.py, emitting
  accuracy + a failure taxonomy, re-baselining the gate floors.

CONTEXT.md: Comparable Properties corrected to all-vintage source; new
Component Accuracy term. ADR-0029 Validation section marked superseded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 08:47:58 +00:00
Khalim Conn-Kowlessar
c3d56b00dd chore(epc-prediction): grow validation corpus to 40 postcodes (ADR-0029)
Bump N_POSTCODES 150 -> 40 as the gradual-growth step from the 3-postcode
smoke. 40 postcodes / 1113 certs / 578 leave-one-out predictions is enough
for stable, trustworthy metrics (the smoke's 2 usable postcodes were
dominated by oddball flats — floor_area mean|.| 52.6 there vs 12.7 here).
Resumable + reproducible (random.seed(2026)); raise again to scale up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 01:52:44 +00:00
Khalim Conn-Kowlessar
fa11df56c2 fix(epc-prediction): dedupe re-lodgements + leak-free leave-one-out (ADR-0029)
The register lists every historical lodgement, so a postcode cohort
contains the same physical address many times (LS61AA: 15 certs / 11
addresses; NG71AA: 15 / 9 — "FLAT 3" appears 3x in each). Two
consequences:

  - Production: a re-lodged neighbour was counting up to 3x towards the
    cohort mode. select_comparables now dedupes candidates to the latest
    cert per address (one comparable per real neighbour) — Comparable
    gains address + registration_date (the register metadata its docstring
    already anticipated, read straight off the cached payload).

  - Validation: leave-one-out leaked — predicting a flat from a near-
    identical re-lodgement of itself. The harness now holds out a whole
    address (excludes every sibling cert) and evaluates on the latest cert
    per address (the best ground truth).

Removing the leak gives the honest numbers (19 distinct addresses):
  wall_construction      93.1% -> 89.5%
  construction_age_band  65.5% -> 52.6%
  roof_construction      79.3% -> 68.4%
  floor_area mean|.|     37.9  -> 52.6 m2
The earlier figures were inflated by self-leakage; these are the real
accuracy to beat.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 00:40:23 +00:00
Khalim Conn-Kowlessar
54a57363f8 feat(epc-prediction): cohort-mode the roof/floor/insulation/age categoricals (ADR-0029)
Only main wall_construction was set to the cohort mode; the other
homogeneous categoricals (wall insulation, construction age band, roof
construction, floor construction) were left as template-copied, so one
median-size template's quirks set them. Apply the same cohort-mode
mechanism to all of them per ADR-0029 decision 4 — the template still
supplies geometry, only the categorical codes move to the mode.

Verified mode beats (or ties) template-copy per categorical before
applying. Smoke corpus (29 leave-one-out) classification rates:
  construction_age_band  55.2% -> 65.5%
  roof_construction      72.4% -> 79.3%
  floor_construction     46.2% -> 84.6%
  wall_insulation_type   93.1% (tie — already template-strong)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 00:31:16 +00:00
Khalim Conn-Kowlessar
ed96df9315 feat(epc-prediction): classify roof/floor/insulation/age categoricals (ADR-0029)
The comparison only scored main wall_construction; everything else the
predictor produces (by template-copy) went unmeasured. Extend
compare_prediction to the rest of the ADR-0029 homogeneous categoricals —
wall insulation type, construction age band, roof construction, floor
construction — and aggregate per-categorical classification rates in the
runner. A categorical hit is "not applicable" (None, excluded from the
denominator) when the actual lodges no value, so absent-roof flats don't
score free wins.

Smoke corpus (29 leave-one-out, all but wall are template-copied today):
  wall_construction      93.1%
  wall_insulation_type   93.1%
  construction_age_band  55.2%   <- loud; candidate for cohort-mode
  roof_construction      72.4%
  floor_construction     46.2%   (n=13)

These numbers drive the next slice (extend cohort-mode coverage).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 00:10:56 +00:00
Khalim Conn-Kowlessar
4fa20ae76b fix(epc-prediction): size-representative template selection (ADR-0029)
Template (the comparable whose structure/geometry is copied wholesale)
was members[0] — an arbitrary draw from the API search order. With floor
area varying widely within a property_type cohort (NG71AA houses span
51-340 m2), this made the copied geometry noisy and systematically large.

Pick the member whose floor area is closest to the cohort median instead,
implementing ADR-0029 decision 4's unimplemented "closest on size"
criterion while keeping the structure coherent (it is still one real
property, so floor dims / windows / parts stay internally consistent for
the calculator).

Smoke corpus (29 leave-one-out predictions):
  floor_area  mean|.| 68.0 -> 37.9 m2  (bias +46.8 -> -3.9)
  window_area mean|.| 11.1 -> 7.3 m2
  parts       mean|.| 1.00 -> 0.38
  SAP |pred-calc - calc(actual)| MAE 7.19 -> 4.86

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 00:05:40 +00:00
Khalim Conn-Kowlessar
f3ad6343a3 feat(epc-prediction): leave-one-out validation harness (ADR-0029)
Pure compare_prediction (TDD): wall-construction classification hit + signed
residuals on floor area, window count, total window area, building-parts count.
Plus validate_epc_prediction.py (IO plumbing): drops each cert from its postcode
cohort, predicts from the rest on guaranteed inputs only, aggregates the metrics,
and reports SAP three ways (pred-calc vs lodged / vs calc-on-actual / vs the
neighbour-mean baseline). Smoke run: wall 90.9%, floor-area mean|·| 42.6 m2 (a
real signal — template-copied floor area is noisy), SAP pred-calc edges baseline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:55:05 +00:00
Khalim Conn-Kowlessar
5e6d2cff16 feat(epc-prediction): EpcPrediction hybrid synthesis (ADR-0029)
predict() copies a representative template comparable's structure (coherent for
the calculator), overrides the homogeneous categorical with the cohort mode
(robust to an atypical template), then applies known Landlord Overrides on top
(a known value wins over the estimate). Proven on wall construction; roof/floor/
insulation/age extend on the same mode+override mechanism, driven next by the
validation harness metrics.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:50:07 +00:00
Khalim Conn-Kowlessar
bf6b6fac17 feat(epc-prediction): Comparable Properties selection ladder (ADR-0029)
Pure-domain select_comparables: property type is an always-hard filter; built
form and known Landlord Overrides (e.g. solid brick) are conditioning filters on
the filter-then-relax ladder — applied while >= minimum_cohort survive, relaxed
otherwise (the mixed-street border case degrades gracefully). PredictionTarget
(known inputs) + Comparable (epc + register metadata) + ComparableProperties
(selected cohort). Weighting (recency x similarity) follows in the synthesis slice.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:44:57 +00:00
Khalim Conn-Kowlessar
80b525f0f4 feat(epc-prediction): postcode-clustered corpus fetch script (ADR-0029)
Builds the frozen validation corpus: samples postcodes from the register, then
caches each postcode's full cohort of raw cert payloads (the shape
from_api_response consumes), grouped by postcode, resumably. Reads the token
from backend/.env; cache dir /tmp/epc_prediction_corpus (EPC_PREDICTION_CORPUS
override). IO plumbing, not test-driven. Pairs with the leave-one-out harness.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:36:19 +00:00
Khalim Conn-Kowlessar
008a1b2783 docs(adr): EPC Prediction from Comparable Properties (ADR-0029)
Grill-with-docs outcome: deterministic neighbour synthesis (NOT ML) of an
EPC-less Property's EpcPropertyData picture, scored via Sap10Calculator.
Six decisions — predict-components-not-SAP; deterministic k-NN; fetch-phase
fallback behind a pure EpcPrediction service + ComparableProperties port;
hybrid synthesis (cohort-mode categoricals + coherent template structure +
overrides); filter-then-relax cohort weighted geo x recency x similarity;
dual-use gap-fill + anomaly flags. Frozen postcode-clustered corpus backs
leave-one-out validation. CONTEXT.md: new EPC Prediction term, Comparable
Properties refined, ML framing corrected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:36:19 +00:00
Jun-te Kim
015ab9d17b
Merge pull request #1219 from Hestia-Homes/feature/junte+khalim
rdSap 17, 18, 19, 20, now maps to EPCPropertyData
2026-06-12 17:14:52 +01:00
Jun-te Kim
1f40c3aeef fix engine dockerfile 2026-06-12 16:07:39 +00:00
Jun-te Kim
0159176772 python upgraded due to enum 2026-06-12 15:47:28 +00:00
Jun-te Kim
0c211f401f
Merge pull request #1220 from Hestia-Homes/feature/make_test_more_readable
added floats helper
2026-06-12 16:04:56 +01:00
Jun-te Kim
80ccec9b68 added floats helper 2026-06-12 14:28:41 +00:00
Jun-te Kim
a6123d762c Merge branch 'main' of https://github.com/Hestia-Homes/Model into feature/junte+khalim 2026-06-12 13:45:30 +00:00
Jun-te Kim
ff4a2e4242
Merge pull request #1198 from Hestia-Homes/feature/bill-derivation
Feature/bill derivation
2026-06-12 14:44:30 +01:00
Jun-te Kim
77c5f7da49 Merge branch 'feature/bill-derivation' of https://github.com/Hestia-Homes/Model into feature/junte+khalim 2026-06-12 12:52:40 +00:00
Jun-te Kim
32de7f6c3f 17.1 and 18 done by claude 2026-06-12 12:52:36 +00:00
Jun-te Kim
1ff50374e7 Record 17.0 band-4/5 synthesis transfer gaps at the seam (ADR-0028)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:50:29 +00:00
Jun-te Kim
6c03f3323c Guard all RdSAP-Schema-17.0 corpus certs in the strict parse+map bucket 🟩
Promote RdSAP-Schema-17.0 into SUPPORTED so all 1000 corpus certs are held to
the strict parse+map guard. Drop the now-redundant cert[0] tracer (subsumed by
the parametrised bucket); keep the reduced-field synthesis behavioural test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:48:47 +00:00
Jun-te Kim
9b01e1d0c9 Synthesise reduced-field windows for RdSAP-Schema-17.0 certs 🟩
Add the 17.0 synthesis seam over the shared _synthesise_reduced_field_windows
core (inherited 20.0.0 coefficients, ND glazing -> DG-modal default 2, per
ADR-0028). 17.0 glazed_type codes (1-4,7) are a subset of the verified 1-8
space. The 10 rich certs use lodged window_area directly; the windowless 990
synthesise a 4-way N/E/S/W split.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:47:40 +00:00
Jun-te Kim
887af58a25 Synthesise reduced-field windows for RdSAP-Schema-17.0 certs 🟥
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:46:27 +00:00
Jun-te Kim
26651ca71c Map RdSAP-Schema-17.0 certs to EpcPropertyData 🟩
Dispatch RdSAP-Schema-17.0 through from_api_response, parse-fix the schema
(data-driven required->optional, validated against the 1000-cert 17.0 corpus
per ADR-0028 — incl. SapHeating.cylinder_insulation_type and the
has_hot_water_cylinder / has_fixed_air_conditioning / has_heated_separate_
conservatory flags), and port the defensive mapper reads (dwelling_type
str/dict/number, photovoltaic_supply guard, sap_floor_dimensions guard). All
1000 corpus certs now parse and map.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:45:19 +00:00
Jun-te Kim
3995433816 Map RdSAP-Schema-17.0 certs to EpcPropertyData 🟥
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:40:04 +00:00
Jun-te Kim
24dcd9aa71 Record 19.0 band-4 synthesis transfer gap at the seam (ADR-0028)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:38:30 +00:00
Jun-te Kim
32eef951ee Add corpus profiler for the ADR-0028 seeing-the-data table
Reusable per-schema profiler: glazed_area band mix, Validation Cohort size,
observed-vs-predicted band glazing/floor ratio, and the ND/str sentinels that
drive schema widening. Regenerates the ADR-0028 transfer-check table from any
harvested corpus.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:36:08 +00:00
Jun-te Kim
99981e07e7 Guard all RdSAP-Schema-19.0 corpus certs in the strict parse+map bucket 🟩
Promote RdSAP-Schema-19.0 into SUPPORTED so all 1000 corpus certs are held to
the strict parse+map guard. Drop the now-redundant cert[0] tracer (subsumed by
the parametrised bucket); keep the reduced-field synthesis behavioural test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:34:29 +00:00
Jun-te Kim
1fcad454fa Synthesise reduced-field windows for RdSAP-Schema-19.0 certs 🟩
Add the 19.0 synthesis seam over the shared _synthesise_reduced_field_windows
core (inherited 20.0.0 coefficients, ND glazing -> DG-modal default 2, per
ADR-0028). 19.0 glazed_type codes (1-4,6,7) are a subset of the verified 1-8
space. The 6 rich certs use lodged window_area directly; the windowless 994
synthesise a 4-way N/E/S/W split.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:33:26 +00:00