Model/docs/adr/0029-epc-prediction-from-comparable-properties.md

EPC Prediction from Comparable Properties

~30% of UK homes (typically long-tenure) have no EPC. EPC Prediction produces a Property's EpcPropertyData picture from its Comparable Properties so an EPC-less Property flows through the rest of the pipeline (Rebaselining, Bill Derivation, Modelling) unchanged. This records the load-bearing design decisions taken in a grill-with-docs session.

Status

Accepted (design). Implementation pending.

Decisions

1. Predict the physical picture, score it with our calculator — never a SAP scalar

Prediction outputs a structured EpcPropertyData (building parts, windows, floor dimensions, construction + insulation, age band); SAP / CO2 / PEI / per-end-use kWh come from running Sap10Calculator on it. This is the same "assemble a picture, score once" mechanic as every other Effective EPC path (Landlord Overrides, Reduced-Field Synthesis), so a predicted Property is fully usable downstream (bills, measures, optimiser) — a directly-aggregated SAP scalar (legacy SearchEpc) would be a dead-end number. It also makes the component-classification accuracy metrics meaningful and keeps errors traceable to a wrong component rather than an opaque regression.

2. Deterministic neighbour synthesis, not ML

No trained model, no learned weights, no fit pipeline: filter a cohort, take categorical modes, copy a representative template, apply overrides. CONTEXT's prior "ML mechanism" framing is corrected — calling it ML invited the wrong architecture (training data, model artifacts, retraining). A future learned-weighting refinement is possible but separate, mirroring the calculator's flagged-future ML residual head. The domain class is EpcPrediction (no "Service" suffix, per the BillDerivation convention).

3. Fetch-phase fallback, behind a domain service + a cohort repository port

A pure EpcPrediction domain service (cohort of comparable EpcPropertyData in → predicted EpcPropertyData out) sits behind a ComparableProperties repository port that owns the cohort IO (postcode search → per-cert fetch, cached). It wires into IngestionOrchestrator._fetch: when epc_fetcher.get_by_uprn returns None, fetch the cohort and predict, persisting the picture marked as predicted (so the UI flags it and the Validation Cohort excludes it). Baseline and Modelling are untouched. Chosen over a fetcher-decorator (hides a heavy cohort fetch behind get_by_uprn) and a dedicated stage (a whole stage for "fill the gap when absent", duplicating IO ingestion already does). The heavy cohort IO stays visible in the no-unit IO phase.

4. Hybrid synthesis: cohort-mode categoricals + a coherent structural template

You cannot average a list of windows (counts differ; a mean orientation is meaningless) or building parts. So:

• Homogeneous categoricals (wall / roof / floor construction + insulation, age band) → cohort mode (robust to one oddball; drives the classification-rate metrics).
• Structure & geometry (building parts, per-window dimensions + orientations, floor dimensions) → copied wholesale from a single representative comparable chosen to be consistent with those modes and closest on geo + size (internally consistent for the calculator; drives the window-area / building-parts / floor-area residual metrics).
• Landlord Overrides and the known inputs are applied on top.

Rejected: field-by-field aggregation (legacy — incoherent, may not score sensibly) and single-nearest-neighbour copied wholesale (one atypical neighbour sets the categoricals → weaker classification).

5. Cohort selection: filter-then-relax ladder, weighted by geo × recency × similarity

Selection hard-filters on identity (property type, built form) and any known Landlord Override (e.g. solid brick — the mixed-street border case) while ≥ k comparables remain, widening the geographic scope (postcode → postcode-prefix) or demoting a known to a strong weight when sparse. Survivors are weighted by geographic proximity (true coordinates via GeospatialRepository, not the legacy house-number proxy) × recency (newer EPCs are higher quality) × physical similarity; pre-SAP10 / very old certs are dropped (amended by ADR-0030: all vintages are kept — components are methodology-agnostic — with recency as a graduated weight; only the validation target must be SAP 10.2). So a known field acts twice: upstream on cohort selection, and again as an override on the final picture.

6. Dual use: gap-fill (no EPC) and anomaly flags (has EPC)

The same cohort + comparison machinery produces EPC Anomaly Flags for Properties that do have an EPC (e.g. "all neighbours are 1930s; this lodges 1950 — correct?") — advisory, surfaced for user review. The no-EPC gap-fill lands first; the always-on anomaly-flag wiring is a follow-on increment.

Validation

Superseded by ADR-0030. The SAP-version mixing in the cohort makes the lodged-SAP comparisons below (esp. the neighbour-mean baseline) invalid; validation is now component-first over SAP-10.2-only targets. The frozen-corpus leave-one-out shape stands.

A frozen postcode-clustered corpus (a one-off fetch caches N postcodes × all their certs as EpcPropertyData) backs an offline, deterministic, repeatable leave-one-out harness over thousands of properties: drop a property with an EPC from its own cohort, predict it, compare predicted vs actual. Metrics: classification rate on wall / roof / floor construction + insulation and construction age band; residuals on SAP, total window area + window count, building-parts count, total floor area. SAP is reported three ways to attribute error — predicted-then-calculated vs (a) lodged SAP (end-to-end), (b) calculator-on-actual-components (isolates prediction error), (c) a direct neighbour-mean SAP baseline (proves predict-then-calculate beats naïve averaging).

Open (implementation-level)

• Provenance marker on the picture (predicted vs real) — exact representation TBD; needed for the UI flag and Validation Cohort exclusion.
• No-cohort fallback when zero comparables survive even after relaxing (low-confidence national property-type + age defaults, or skip-with-flag).
• Confidence signal (cohort size + agreement) carried for the UI and anomaly thresholds.