From 2788849e16b504b5d5ef674fd3cbf7040a90ae09 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 6 Jul 2026 08:15:43 +0000 Subject: [PATCH] ADR-0054 + glossary: expired historic EPC conditions prediction with stable attributes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Settled in a grilling session with Khalim: the pre-2012 backup feeds EPC Prediction landlord-override-style (stable attrs as relax-ladder filters, TFA as a ±5% band, exact-UPRN lookup, source="expired"), never trusted as current state; scoped to the historic source; validated by a pairs harness. Co-Authored-By: Claude Opus 4.8 --- CONTEXT.md | 8 ++ ...tions-prediction-with-stable-attributes.md | 86 +++++++++++++++++++ 2 files changed, 94 insertions(+) create mode 100644 docs/adr/0054-expired-historic-epc-conditions-prediction-with-stable-attributes.md diff --git a/CONTEXT.md b/CONTEXT.md index 81b2716c3..cb977c4ff 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -81,6 +81,14 @@ _Avoid_: neighbours, similar properties, peer set Producing a Property's `EpcPropertyData` picture from its **Comparable Properties** when it has no EPC (~30% of UK homes, typically long-tenure). **Deterministic** neighbour synthesis (k-NN-style — *not* ML; no trained model): take the cohort **mode** for the homogeneous categoricals (wall / roof / floor construction + insulation, construction age band), copy a single representative comparable's **structure** wholesale (building parts, per-window dimensions + orientations, floor dimensions) so the picture stays internally consistent for the calculator, then apply **Landlord Overrides** and the known inputs on top. The result is scored through **SAP10 Calculation** like any other **Effective EPC**, so a predicted Property flows through Rebaselining, Bill Derivation, and Modelling unchanged — held in a **distinct predicted-EPC slot** that coexists with any lodged EPC (so provenance is structural and the UI can flag it; see ADR-0031). A **known property type is required** — the hard cohort filter (a flat is never sized from houses) — supplied by a **Landlord Override** (or, later, an Ordnance Survey lookup); a Property whose property type is genuinely unknown is **gated out**, never predicted from a mixed-type cohort and never given a national default. The same cohort machinery also produces **EPC Anomaly Flags** for Properties that *do* have an EPC. A future learned-weighting refinement is possible but separate, as with the calculator's ML residual head. _Avoid_: EpcPredictionService (no "service" suffix — name the operation), ML prediction (it is deterministic), EPC estimation +**Historic EPC**: +One certificate row from the final data dump of the shut-down old EPC register, held at `s3://retrofit-data-dev/historical_epc/{POSTCODE}/data.csv.gz` and read through the `HistoricEpcRepository` port. Partial, tabular, display-text data (the old API never exposed full SAP inputs), covering certificates the new gov API (registered ≥ 1 Jan 2012) cannot see. Consumed by `address2UPRN` (fuzzy address→UPRN resolution) and by **Expired-Enhanced Prediction** (exact-UPRN lookup only). +_Avoid_: old EPC (ambiguous with a pre-SAP10 cert from the new API), historical EPC API (the API is gone; only the backup exists) + +**Expired-Enhanced Prediction**: +**EPC Prediction** for a Property whose only certificate predates 2012: the expired **Historic EPC**, found by exact UPRN in its postcode shard, **conditions** cohort selection with its *stable* attributes (property type, built form, wall material, roof construction, age band, main fuel, and a ±5% floor-area band) exactly as a Landlord Override would. Volatile attributes (heating, hot water, glazing, PV, insulation, lighting) are excluded — 14+ years stale — and stay neighbour-predicted; the historic cert is **never copied into the Effective EPC as current state**. Persisted to the predicted slot with `source="expired"`. Scoped to the historic backup only; post-2012 expired certs from the new API keep their existing treatment (ADR-0054). +_Avoid_: historic override (it is conditioning, not an override the effective picture trusts), expired EPC path (names the input, not the operation) + ### Survey documents **Ventilation Audit**: diff --git a/docs/adr/0054-expired-historic-epc-conditions-prediction-with-stable-attributes.md b/docs/adr/0054-expired-historic-epc-conditions-prediction-with-stable-attributes.md new file mode 100644 index 000000000..58c869bc8 --- /dev/null +++ b/docs/adr/0054-expired-historic-epc-conditions-prediction-with-stable-attributes.md @@ -0,0 +1,86 @@ +# The expired historic EPC conditions prediction with its stable attributes; it is never trusted as current state + +## Status + +accepted + +## Context + +The old EPC register API has been shut down. Before it went, we captured its +final data dump to `s3://retrofit-data-dev/historical_epc/{POSTCODE}/data.csv.gz` +— one flat `HistoricEpc` row per certificate, partial tabular data (the old API +never exposed full SAP inputs). PR #1356 lifted that backup into a DDD stack: +`HistoricEpcRepository` (port) / `HistoricEpcS3Repository` (adapter) / +`HistoricEpcResolver`, so far consumed only by `address2UPRN`. + +The new gov EPC API (get-energy-performance-data.communities.gov.uk) only +covers certificates registered **since 1 January 2012**. A Property whose only +certificate predates 2012 is EPC-less to Ingestion today, so **EPC Prediction** +synthesises its picture blind from `property_type` + `built_form` (+ any +Landlord Overrides). But the historic dump holds *observed* attributes for that +exact dwelling — wall, roof, floor area, fuel, age band, heating, glazing, PV. + +The tension: a pre-2012 certificate is 14+ years stale. Construction and +geometry do not change (wall material, built form, age band, roughly the floor +area); the rest very plausibly has (heating system, hot water, glazing, PV, +insulation levels, lighting). Trusting a 2009 "back boiler, single glazed" +observation as override-grade current truth could make the prediction *worse* +than the neighbour-based default. + +## Decision + +An expired historic EPC **conditions** EPC Prediction the way a Landlord +Override does — it narrows and enriches the cohort — and is **never copied +into the Effective EPC as current state**. + +1. **Stable attributes only.** The historic certificate contributes: + `property_type` (the hard cohort filter, as today), `built_form`, wall + construction **material** (the RdSAP `wall_construction` code resolved from + the description's material prefix, per `wall_type_overlay.py`), roof + construction, `construction_age_band`, and `main_fuel`. Volatile attributes + — heating system, hot water, glazing, PV, insulation states, lighting — are + excluded and stay neighbour-predicted. +2. **Floor area is a tolerance band, not an override.** Comparables are + soft-filtered to within **±5%** of the historic certificate's total floor + area. The predicted floor area remains the cohort's geo-weighted median. +3. **Every historic filter rides the existing filter-then-relax ladder** + (ADR-0029): an attribute that cannot be resolved into the cohort's code + space maps to `None` and its filter is simply inactive; a filter that would + starve the cohort below the minimum is relaxed. Degradation is graceful by + construction. +4. **Exact-UPRN lookup.** The prediction path fetches the Property's postcode + shard and matches on UPRN equality (multiple rows for one UPRN → latest + lodgement date). Fuzzy address matching stays quarantined in `address2UPRN`; + a fuzzy hit will never import a neighbour's attributes as if observed. +5. **Provenance: `EpcSource` gains `"expired"`.** The enhanced prediction is + persisted to the predicted slot with `source="expired"` — enhanced by an + expired observation of this dwelling, rather than totally predicted. + `Property.source_path` / Effective-EPC precedence are unchanged. +6. **Scoped to the historic source.** Post-2012 certificates from the new API + — including expired or replaced ones — keep today's treatment. The + epistemic inconsistency (an expired 2014 cert is trusted as current; an + expired 2009 one only conditions) is acknowledged and deliberate: changing + the post-2012 path is a separate, larger-blast-radius decision. +7. **Validated by a pairs harness, not reasoning alone.** A repeatable script + finds properties holding a pre-2012 historic certificate *and* a + post-June-2025 (RdSAP 10 / SAP 10.2) lodged certificate, predicts from the + historic attributes, and reports Component Accuracy against the lodged + truth **per attribute** — so any whitelist member (main fuel is the + judgement call) can be promoted or demoted with evidence. A report, not a + CI gate. + +## Consequences + +- The lookup chain at Ingestion becomes: new EPC API → historic backup by UPRN + → plain prediction. A historic miss costs one S3 GET. +- `PredictionTarget` grows optional stable-attribute fields and + `select_comparables` gains their soft filters (including the first + numeric-tolerance filter); all default to `None`/inactive, so existing + callers and behaviour are untouched. +- `EpcSource` widens to `Literal["lodged", "predicted", "expired"]`; the + `epc_property.source` column is already TEXT, so no migration. +- Downstream consumers that branch on `source == "predicted"` treat + `"expired"` the same unless they opt into the distinction (it lives in the + predicted slot; `source_path` is unchanged). +- The whitelist is a hypothesis until the pairs harness reports; attribute + membership changes are cheap (one resolver each).