diff --git a/CONTEXT.md b/CONTEXT.md index 14189681..1435ea6a 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -66,7 +66,7 @@ The reference cohort matched to a target Property, used by **EPC Prediction** fo _Avoid_: neighbours, similar properties, peer set **EPC Prediction**: -Producing a Property's `EpcPropertyData` picture from its **Comparable Properties** when it has no EPC (~30% of UK homes, typically long-tenure). **Deterministic** neighbour synthesis (k-NN-style — *not* ML; no trained model): take the cohort **mode** for the homogeneous categoricals (wall / roof / floor construction + insulation, construction age band), copy a single representative comparable's **structure** wholesale (building parts, per-window dimensions + orientations, floor dimensions) so the picture stays internally consistent for the calculator, then apply **Landlord Overrides** and the known inputs on top. The result is scored through **SAP10 Calculation** like any other **Effective EPC**, so a predicted Property flows through Rebaselining, Bill Derivation, and Modelling unchanged — marked as predicted so the UI can flag it. The same cohort machinery also produces **EPC Anomaly Flags** for Properties that *do* have an EPC. A future learned-weighting refinement is possible but separate, as with the calculator's ML residual head. +Producing a Property's `EpcPropertyData` picture from its **Comparable Properties** when it has no EPC (~30% of UK homes, typically long-tenure). **Deterministic** neighbour synthesis (k-NN-style — *not* ML; no trained model): take the cohort **mode** for the homogeneous categoricals (wall / roof / floor construction + insulation, construction age band), copy a single representative comparable's **structure** wholesale (building parts, per-window dimensions + orientations, floor dimensions) so the picture stays internally consistent for the calculator, then apply **Landlord Overrides** and the known inputs on top. The result is scored through **SAP10 Calculation** like any other **Effective EPC**, so a predicted Property flows through Rebaselining, Bill Derivation, and Modelling unchanged — held in a **distinct predicted-EPC slot** that coexists with any lodged EPC (so provenance is structural and the UI can flag it; see ADR-0031). A **known property type is required** — the hard cohort filter (a flat is never sized from houses) — supplied by a **Landlord Override** (or, later, an Ordnance Survey lookup); a Property whose property type is genuinely unknown is **gated out**, never predicted from a mixed-type cohort and never given a national default. The same cohort machinery also produces **EPC Anomaly Flags** for Properties that *do* have an EPC. A future learned-weighting refinement is possible but separate, as with the calculator's ML residual head. _Avoid_: EpcPredictionService (no "service" suffix — name the operation), ML prediction (it is deterministic), EPC estimation ### Survey documents diff --git a/docs/adr/0031-epc-prediction-production-wiring.md b/docs/adr/0031-epc-prediction-production-wiring.md new file mode 100644 index 00000000..d6b21f35 --- /dev/null +++ b/docs/adr/0031-epc-prediction-production-wiring.md @@ -0,0 +1,87 @@ +# EPC Prediction production wiring + +ADR-0029 settled *how* EPC Prediction synthesises a Property's `EpcPropertyData` +from its **Comparable Properties**; this records *how it wires into the running +pipeline* — where estimation runs, how a predicted EPC is stored and told apart +from a lodged one, and which Properties are eligible. Resolved in a +grill-with-docs session after the algorithm + validation harness were built and +the accuracy backlog (#1222–1228) closed. + +## Status + +Accepted (design). Refines ADR-0029 decision 3. Implementation pending (slice 5). + +## Decisions + +### 1. Estimation runs in Ingestion — the #1227 "shift to Modelling" is dropped + +The cohort fetch + predict happens in `IngestionOrchestrator`, when +`epc_fetcher.get_by_uprn` returns `None` — upholding ADR-0029 decision 3. A +design note from issue #1227 had proposed moving estimation (and its distance +calcs) into `ModellingOrchestrator`; that is reversed here. Ingestion is already +the EPC-acquisition phase and *already resolves the Property's coordinates* +(`spatial`), so it can run the geo-weighted prediction with no new IO surface; +the First Run stages communicate **only through persisted state** (the pipeline +threads just `property_ids`, each stage reloads the `Property`), so a prediction +produced in Modelling would either have to be persisted there anyway or +recomputed every run. No rationale for the Modelling shift survived review. +Baseline and Modelling stay untouched — they read a populated `effective_epc`. + +### 2. The predicted EPC is persisted in a distinct slot, never overwriting the lodged one + +Because stages communicate via persisted state, the prediction **must be saved** +for Modelling to see it — in-memory-only would never reach stage 3. It is stored +as a **distinct predicted-EPC slot** on the Property (the EPC table reused with a +`source` discriminator — `lodged` / `predicted`), so a lodged EPC and a predicted +EPC can **coexist** on one Property. Coexistence is load-bearing: it is what lets +the same cohort machinery produce **EPC Anomaly Flags** for Properties that *do* +have an EPC (the dual-use named in ADR-0029), and it means a later real-EPC fetch +fills its own slot without the predicted one muddying provenance. Rejected: a +single EPC slot with an `is_predicted` flag — it cannot hold both, so it forecloses +anomaly detection and makes "lodged later arrives" ambiguous. + +### 3. Provenance is structural — on the Property, not on `EpcPropertyData` + +`EpcPropertyData` gains no `predicted` / `source` field. Which slot the picture +came from *is* its provenance. `Property.effective_epc` / `source_path` gain a +`"predicted"` branch, used only when there is no lodged EPC **and** no Site Notes +(the existing precedence is unchanged; a real source always wins). The +**Validation Cohort** then excludes any Property whose `effective_epc` resolves +via the predicted slot — it has no same-spec lodged ground truth — and the UI +flags it as predicted. Keeping `EpcPropertyData` clean means every downstream +consumer (calculator, generators, bill derivation) is unchanged and oblivious to +how the picture was sourced, exactly as for Landlord-Override and Reduced-Field +pictures. + +### 4. Slice 5 is gap-fill only; always-predict (anomaly) is a follow-on + +Prediction runs only when `epc is None`. Predicting for *every* Property to +compare against its lodged EPC (EPC Anomaly Flags) is real and the slot model +supports it, but it triples the ingestion cohort IO and needs its own +comparison + divergence-threshold + UI surface — so it does not ride in on the +wiring slice. The predicted-EPC slot and the `ComparableProperties` repository +this slice introduces are exactly what the anomaly capability reuses. + +### 5. A known property type is required — eligibility is gated, never defaulted + +A `PredictionTarget` needs `postcode` (from `PropertyIdentity`), `coordinates` +(geospatial), and `property_type` + `built_form` + `wall_construction` from +**Landlord Overrides**. `property_type` is the **hard** cohort filter (a flat +must not be sized from houses), so it is a **required input**: a Property whose +property type is genuinely unknown is **gated out** before prediction — flagged +un-predictable, never predicted from a mixed-type cohort and never given a +national default. (An Ordnance Survey `postcode_search` source can supply +property type more broadly than landlord input does; wiring it is a later +enhancement that widens the eligible population — out of scope here.) The +`ComparableProperties` repository port deferred by ADR-0029 is built in this +slice: it owns the cohort IO (postcode search → per-cert fetch → UPRN→coordinate +resolution) and returns candidate `Comparable`s for the domain +`select_comparables` to filter. + +## Consequences + +- A schema change: the EPC store gains a `source` discriminator (or equivalent), + and the Property repository a `get_predicted_for_property` read. +- Slice 5's reach is bounded to EPC-less Properties with a landlord-supplied + property type. The `postcode_search` integration is the lever to broaden it. +- `EpcPropertyData` stays unchanged, so no downstream consumer is touched.