ADR-0054 + glossary: expired historic EPC conditions prediction with stable attributes

Settled in a grilling session with Khalim: the pre-2012 backup feeds EPC
Prediction landlord-override-style (stable attrs as relax-ladder filters,
TFA as a ±5% band, exact-UPRN lookup, source="expired"), never trusted as
current state; scoped to the historic source; validated by a pairs harness.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CONTEXT.md

@@ -81,6 +81,14 @@ _Avoid_: neighbours, similar properties, peer set
 Producing a Property's `EpcPropertyData` picture from its **Comparable Properties** when it has no EPC (~30% of UK homes, typically long-tenure). **Deterministic** neighbour synthesis (k-NN-style — *not* ML; no trained model): take the cohort **mode** for the homogeneous categoricals (wall / roof / floor construction + insulation, construction age band), copy a single representative comparable's **structure** wholesale (building parts, per-window dimensions + orientations, floor dimensions) so the picture stays internally consistent for the calculator, then apply **Landlord Overrides** and the known inputs on top. The result is scored through **SAP10 Calculation** like any other **Effective EPC**, so a predicted Property flows through Rebaselining, Bill Derivation, and Modelling unchanged — held in a **distinct predicted-EPC slot** that coexists with any lodged EPC (so provenance is structural and the UI can flag it; see ADR-0031). A **known property type is required** — the hard cohort filter (a flat is never sized from houses) — supplied by a **Landlord Override** (or, later, an Ordnance Survey lookup); a Property whose property type is genuinely unknown is **gated out**, never predicted from a mixed-type cohort and never given a national default. The same cohort machinery also produces **EPC Anomaly Flags** for Properties that *do* have an EPC. A future learned-weighting refinement is possible but separate, as with the calculator's ML residual head.
 _Avoid_: EpcPredictionService (no "service" suffix — name the operation), ML prediction (it is deterministic), EPC estimation
 
+**Historic EPC**:
+One certificate row from the final data dump of the shut-down old EPC register, held at `s3://retrofit-data-dev/historical_epc/{POSTCODE}/data.csv.gz` and read through the `HistoricEpcRepository` port. Partial, tabular, display-text data (the old API never exposed full SAP inputs), covering certificates the new gov API (registered ≥ 1 Jan 2012) cannot see. Consumed by `address2UPRN` (fuzzy address→UPRN resolution) and by **Expired-Enhanced Prediction** (exact-UPRN lookup only).
+_Avoid_: old EPC (ambiguous with a pre-SAP10 cert from the new API), historical EPC API (the API is gone; only the backup exists)
+
+**Expired-Enhanced Prediction**:
+**EPC Prediction** for a Property whose only certificate predates 2012: the expired **Historic EPC**, found by exact UPRN in its postcode shard, **conditions** cohort selection with its *stable* attributes (property type, built form, wall material, roof construction, age band, main fuel, and a ±5% floor-area band) exactly as a Landlord Override would. Volatile attributes (heating, hot water, glazing, PV, insulation, lighting) are excluded — 14+ years stale — and stay neighbour-predicted; the historic cert is **never copied into the Effective EPC as current state**. Persisted to the predicted slot with `source="expired"`. Scoped to the historic backup only; post-2012 expired certs from the new API keep their existing treatment (ADR-0054).
+_Avoid_: historic override (it is conditioning, not an override the effective picture trusts), expired EPC path (names the input, not the operation)
+
 ### Survey documents
 
 **Ventilation Audit**:

docs/adr/0054-expired-historic-epc-conditions-prediction-with-stable-attributes.md

@@ -0,0 +1,86 @@
+# The expired historic EPC conditions prediction with its stable attributes; it is never trusted as current state
+
+## Status
+
+accepted
+
+## Context
+
+The old EPC register API has been shut down. Before it went, we captured its
+final data dump to `s3://retrofit-data-dev/historical_epc/{POSTCODE}/data.csv.gz`
+— one flat `HistoricEpc` row per certificate, partial tabular data (the old API
+never exposed full SAP inputs). PR #1356 lifted that backup into a DDD stack:
+`HistoricEpcRepository` (port) / `HistoricEpcS3Repository` (adapter) /
+`HistoricEpcResolver`, so far consumed only by `address2UPRN`.
+
+The new gov EPC API (get-energy-performance-data.communities.gov.uk) only
+covers certificates registered **since 1 January 2012**. A Property whose only
+certificate predates 2012 is EPC-less to Ingestion today, so **EPC Prediction**
+synthesises its picture blind from `property_type` + `built_form` (+ any
+Landlord Overrides). But the historic dump holds *observed* attributes for that
+exact dwelling — wall, roof, floor area, fuel, age band, heating, glazing, PV.
+
+The tension: a pre-2012 certificate is 14+ years stale. Construction and
+geometry do not change (wall material, built form, age band, roughly the floor
+area); the rest very plausibly has (heating system, hot water, glazing, PV,
+insulation levels, lighting). Trusting a 2009 "back boiler, single glazed"
+observation as override-grade current truth could make the prediction *worse*
+than the neighbour-based default.
+
+## Decision
+
+An expired historic EPC **conditions** EPC Prediction the way a Landlord
+Override does — it narrows and enriches the cohort — and is **never copied
+into the Effective EPC as current state**.
+
+1. **Stable attributes only.** The historic certificate contributes:
+   `property_type` (the hard cohort filter, as today), `built_form`, wall
+   construction **material** (the RdSAP `wall_construction` code resolved from
+   the description's material prefix, per `wall_type_overlay.py`), roof
+   construction, `construction_age_band`, and `main_fuel`. Volatile attributes
+   — heating system, hot water, glazing, PV, insulation states, lighting — are
+   excluded and stay neighbour-predicted.
+2. **Floor area is a tolerance band, not an override.** Comparables are
+   soft-filtered to within **±5%** of the historic certificate's total floor
+   area. The predicted floor area remains the cohort's geo-weighted median.
+3. **Every historic filter rides the existing filter-then-relax ladder**
+   (ADR-0029): an attribute that cannot be resolved into the cohort's code
+   space maps to `None` and its filter is simply inactive; a filter that would
+   starve the cohort below the minimum is relaxed. Degradation is graceful by
+   construction.
+4. **Exact-UPRN lookup.** The prediction path fetches the Property's postcode
+   shard and matches on UPRN equality (multiple rows for one UPRN → latest
+   lodgement date). Fuzzy address matching stays quarantined in `address2UPRN`;
+   a fuzzy hit will never import a neighbour's attributes as if observed.
+5. **Provenance: `EpcSource` gains `"expired"`.** The enhanced prediction is
+   persisted to the predicted slot with `source="expired"` — enhanced by an
+   expired observation of this dwelling, rather than totally predicted.
+   `Property.source_path` / Effective-EPC precedence are unchanged.
+6. **Scoped to the historic source.** Post-2012 certificates from the new API
+   — including expired or replaced ones — keep today's treatment. The
+   epistemic inconsistency (an expired 2014 cert is trusted as current; an
+   expired 2009 one only conditions) is acknowledged and deliberate: changing
+   the post-2012 path is a separate, larger-blast-radius decision.
+7. **Validated by a pairs harness, not reasoning alone.** A repeatable script
+   finds properties holding a pre-2012 historic certificate *and* a
+   post-June-2025 (RdSAP 10 / SAP 10.2) lodged certificate, predicts from the
+   historic attributes, and reports Component Accuracy against the lodged
+   truth **per attribute** — so any whitelist member (main fuel is the
+   judgement call) can be promoted or demoted with evidence. A report, not a
+   CI gate.
+
+## Consequences
+
+- The lookup chain at Ingestion becomes: new EPC API → historic backup by UPRN
+  → plain prediction. A historic miss costs one S3 GET.
+- `PredictionTarget` grows optional stable-attribute fields and
+  `select_comparables` gains their soft filters (including the first
+  numeric-tolerance filter); all default to `None`/inactive, so existing
+  callers and behaviour are untouched.
+- `EpcSource` widens to `Literal["lodged", "predicted", "expired"]`; the
+  `epc_property.source` column is already TEXT, so no migration.
+- Downstream consumers that branch on `source == "predicted"` treat
+  `"expired"` the same unless they opt into the distinction (it lives in the
+  predicted slot; `source_path` is unchanged).
+- The whitelist is a hypothesis until the pairs harness reports; attribute
+  membership changes are cheap (one resolver each).