Merge pull request #1372 from Hestia-Homes/fix/1361-class-a-heating-classification-taxonomy

Fix PRD #1361 Class A: complete the landlord-heating classification taxonomy
This commit is contained in:
Daniel Roth 2026-07-01 08:16:45 +01:00 committed by GitHub
commit db66b3b273
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
11 changed files with 877 additions and 9 deletions

View file

@ -101,6 +101,10 @@ _Avoid_: energy assessment, site survey, field survey, Domna survey, Hestia surv
Property data supplied by a landlord that may correct or supplement the public EPC for a single Property; triggers Rebaselining when applied; not applicable when Site Notes are present.
_Avoid_: patches (deprecated), corrections, manual EPC, edits
**Landlord-Description Classification**:
Resolving a **Landlord Description** (unbounded free-text a landlord supplies for one component — "CWI" / "Cav filled" / "cavity insulated" all name one thing) onto a **Recognised Internal Description** via an LLM classifier, persisted in the `landlord_*_overrides` table (`source=classifier`) as a reviewed cache. Four vocabularies are kept **distinct** and must not be conflated: a **Landlord Description** (unbounded input); a **Recognised Internal Description** (the closed target taxonomy — e.g. a `MainHeatingSystemType` archetype — each binding to a Simulation Overlay); a **Lodged Description** (the gov-EPC `main_heating[].description` rendering, e.g. "Room heaters, electric" — only an example of which system *types* occur, never a map key); and the **SAP main heating code** (Table 4a/4b, what the calculator consumes). The classifier maps Landlord → Recognised Internal → SAP code. When it cannot confidently place the text it emits **`None`** (no overlay → the lodged EPC stands, surfaced to the user as "no suitable match"), **never the nearest wrong archetype** — the target taxonomy must be complete enough that a real system always has a correct home, so the classifier never overflows into a garbage-drawer archetype (ADR-0041).
_Avoid_: "the LLM mapper is unreliable" (the failure mode is a too-small target taxonomy, not LLM language ability); conflating the landlord input vocabulary with the gov-EPC lodged rendering or the RdSAP entry-tool catalogue; treating a deterministic dict as a *replacement* for the LLM rather than a reviewed cache of its output
### Modelling
**Effective EPC**:

View file

@ -108,7 +108,10 @@ from repositories.product.composite_product_repository import (
from repositories.property.in_memory_property_overrides_reader import (
InMemoryPropertyOverridesReader,
)
from repositories.property.landlord_override_overlays import overlays_from
from repositories.property.landlord_override_overlays import (
flag_fuel_mismatch,
overlays_from,
)
from repositories.property.override_backed_prediction_attributes_reader import (
OverrideBackedPredictionAttributesReader,
)
@ -561,7 +564,9 @@ def handler(
epc = (
None # no stored lodged EPC; prediction path handles this property
)
overrides = overlays_from(overrides_reader.overrides_for(pid))
resolved_overrides = overrides_reader.overrides_for(pid)
flag_fuel_mismatch(resolved_overrides)
overrides = overlays_from(resolved_overrides)
predicted_epc: Optional[EpcPropertyData] = None
predicted_epc_is_new = False

View file

@ -0,0 +1,129 @@
# Landlord-heating classification targets a complete, modellable taxonomy; unmapped input is no-override
## Status
accepted
## Context
Landlord supplementary data describes a dwelling's components in **unbounded
free-text** — a wall is "cavity insulated" / "CWI" / "Cav filled"; a heating
system is "communal gas boiler" / "Quantum storage" / a boiler make. An LLM
classifier maps that free-text onto a **closed internal taxonomy** of recognised
descriptions, each of which binds to a Simulation Overlay applied to
`EpcPropertyData` (ADR-0032). For heating the taxonomy is the
`MainHeatingSystemType` enum, mapped to a representative SAP Table 4a/4b code by
`main_heating_system_overlay`, with coherent companions dragged from the code
(ADR-0035). The LLM is the right tool for the unbounded-free-text problem; the
deterministic dicts in legacy `asset_list` were a *reviewed cache of LLM output*,
not a replacement, and the `landlord_*_overrides` table already serves as that
verified cache (keyed `portfolio_id + description`).
The taxonomy was **too small** — 9 heating archetypes against a ~13-family RdSAP
main-heating taxonomy. With no correct target the classifier force-picked the
nearest wrong archetype, and treated **"Gas CPSU" as a garbage drawer**: oil and
solid-fuel room heaters, community heating, and `"boiler: a rated na"` all
classified to Gas CPSU; the word "convector" in `"electric (direct acting) room
heaters: panel, convector or radiant heaters"` pulled it to `"Electric storage
heaters, convector"` (code 403, off-peak storage) instead of `"Electric room
heaters"` (691, direct-acting, single-rate).
Folded into the Effective EPC, a single-rate dwelling modelled as off-peak
storage scores **SAP ~10 (band G)**. This is the true cause of **PRD #1361
Class A** — 14 properties lodged in band C/D rebaselining to band G — which had
been mis-attributed to a fabric "no insulation (assumed)" assumption and the
pre-SAP10 rebaseline. The walls are correct (landlord asserted as-built); the
heating is the crater. Exemplar **property 718066**: lodged SAP 57, landlord
described "electric (direct acting) room heaters: panel, convector or radiant"
(correct code 691, single-rate), mis-stored as code 403 → live SAP **9.9**. The
stored override was classified `source=classifier` on 2026-06-20, **before** the
691 "Electric room heaters" archetype existed (commit ada2bc07, 2026-06-29) — so
it is also stale.
## Decision
The fix is the **target taxonomy, not the mapper**. The LLM stays.
1. **The classifier targets a complete, modellable taxonomy.** Expand
`MainHeatingSystemType` + `_MAIN_HEATING_CODES` to cover every RdSAP
main-heating family the calculator can score: room heaters by fuel
(gas / oil / solid), warm air, **heat pumps**, and **community heating**. Each
maps to a representative Table 4a/4b code; coherent companions drag from the
code per ADR-0035, so adding an archetype is still "just add its code".
2. **Unmapped free-text → `None`, never a forced archetype.** The overlay already
returns `None` (no overlay → the lodged EPC heating stands) for any value not
in `_MAIN_HEATING_CODES`; the classifier must *emit* Unknown/`None` when it
cannot confidently place the text, rather than the nearest wrong archetype.
"Gas CPSU" (or any other archetype) is never a fallback. The `None` mapping is
persisted and surfaced to the user as "no suitable match" for later edit. A
forced archetype overwrites a correct lodged cert; `None` (keep lodged) is the
only safe and honest behaviour.
3. **Heat pumps are modellable without a PCDB index** — a "model unknown" heat
pump maps to a default Table 4a heat-pump code (211224) with the table's
default seasonal efficiency; no `main_heating_index_number` is required.
4. **Community heating is modelled via its explicit codes** — 301 (boiler
community) / 302 (CHP + boilers) / 304 (electric heat-pump community), which
the catalogue strings disambiguate ("community boilers only" / "community CHP
and boilers" / "community heat pump"). The calculator (Appendix C §C3.2
distribution loss, Table 12c DLF, Table 12 source factors) and Bill Derivation
(`HEAT_NETWORK` indicative rate) already support these. **Load-bearing
assumption:** when the landlord text does not supply them, the **DLF defaults
to Table 12c 1.50** and the source to a boiler — community SAP is *sensitive*
to the DLF (1.0 vs 1.5 is a large swing), so this default is documented here
and surfaced as an assumption. Coarse "communal heating" with **no named
source** falls to `None` rather than inventing a heat-network model.
5. **Re-classify the stale `source=classifier` overrides** once the taxonomy is
complete, so the band-G properties pick up their correct code.
## Considered Options
- **Replace the LLM with a deterministic catalogue map** — rejected: landlord
input is unbounded free-text, not a closed vocabulary. The LLM is the right
front door; determinism belongs as a downstream *verified* cache (the override
table), built from LLM output + review, exactly as `asset_list` was.
- **Keep the 9-member enum, tune the LLM prompt** — rejected: with no correct
target the LLM must pick a wrong archetype regardless of prompt. Target
completeness is the fix; prompt quality is secondary.
- **Force unmapped input to a conservative archetype** (e.g. direct-acting
electric) — rejected: any forced archetype overwrites a correct lodged cert.
- **Defer community heating to `None`** (grill option b) — rejected in favour of
modelling the three explicit community codes, since calculator + bill already
support heat networks and the catalogue strings disambiguate the cases; only
coarse unnamed "communal heating" falls to `None`.
This extends [ADR-0032](0032-landlord-override-epc-overlay.md) (the override
overlay) and [ADR-0035](0035-coherent-heating-system-synthesis.md) (coherent
companions drag from the code). The visible baseline shift it produces is
correct Rebaselining per [ADR-0039](0039-override-aware-rebaselining.md). It
**corrects the stated cause of PRD #1361 Class A** (classifier taxonomy gap, not
fabric/rebaseline).
### Amendment: natural-fuel coherence
A room-heater / direct-acting archetype names a *system*, not its fuel — and fuel
arrives as its own composable `main_fuel` override. Two rules keep the system
self-coherent without coupling the two overlays:
- **Every archetype drags a natural fuel so the system self-coheres without a
`main_fuel` override.** Where the fuel is unambiguous it is exact — electric
room heaters / direct-acting / storage → electricity (RdSAP `main_fuel` 29), a
gas boiler → mains gas (26), an oil room heater → oil (28). **Solid fuel is
ambiguous** (coal / anthracite / smokeless / dual fuel / wood logs / pellets),
so it **defaults to house coal (33)** — the most common solid fuel — which a
specific `main_fuel` override then refines. (The `main_fuel` override's own
vocabulary must grow to carry the full solid-fuel list — a parallel taxonomy
gap, same shape as this one.)
- **A present `main_fuel` override wins** by the applicator's last-wins
composition (`apply_simulations` setattrs non-`None` fields in order), so the
`main_fuel` overlay is applied **after** the heating overlay. The agreeing
case (electric system + electricity fuel) is order-immaterial; only a
*disagreeing* one depends on the ordering.
- **A landlord fuel that contradicts the archetype's natural fuel is *logged*,
not raised** (e.g. "gas" fuel on a solid-fuel room heater) — an
override-plausibility signal (cf. the ADR-0039 scanner); what to do with it is
deferred. The override is still honoured (the landlord wins); we only record
the implausibility.

View file

@ -68,6 +68,61 @@ _ASSUMED_DUAL_METER_CODES = OFF_PEAK_IMPLYING_HEATING_CODES | _ROOM_HEATER_CODES
_MANUAL_CHARGE_CONTROL = 2401
_STORAGE_HEATER_CODES = frozenset(range(401, 410))
# SAP Table 4a category 10 ("Room heaters") and its conservative Table 4e Group 6
# control. A landlord names a room heater, not its control; code 2601 ("no
# thermostatic control of room temperature") is the lowest-SAP room-heater
# control — the modal one real solid-fuel room-heater certs lodge — so it never
# over-credits an unobserved control (the room-heater mirror of the storage
# manual-charge default). Scoped per family as archetypes land; solid-fuel room
# heaters are Table 4a 631-636.
_ROOM_HEATER_CATEGORY = 10
_ROOM_HEATER_CONTROL = 2601
_SOLID_FUEL_ROOM_HEATER_CODES = frozenset(range(631, 637))
# Oil room heaters (SAP Table 4a 621-625) — category 10, conservative room-heater
# control, natural fuel heating oil (RdSAP main_fuel 28).
_OIL_ROOM_HEATER_CODES = frozenset(range(621, 626))
_OIL_FUEL = 28
# Gas (incl. LPG) room heaters (SAP Table 4a 601-613) — category 10, conservative
# control, natural fuel mains gas (26); an LPG dwelling is refined by a main_fuel
# override (the overlay can't see the mains connection).
_GAS_ROOM_HEATER_CODES = frozenset(range(601, 614))
# Fuel-burning room heaters (solid + oil + gas) share category 10 + the
# conservative room-heater control; only the natural fuel differs by family.
# (Distinct from `_ROOM_HEATER_CODES` above, the electric-room-heater dual-meter
# set.)
_CATEGORY_10_ROOM_HEATER_CODES = (
_SOLID_FUEL_ROOM_HEATER_CODES | _OIL_ROOM_HEATER_CODES | _GAS_ROOM_HEATER_CODES
)
# Natural-fuel coherence (ADR-0041): where an archetype unambiguously implies a
# fuel, the overlay drags it so a system-only override is self-coherent on a cert
# that lodged a different fuel. A later `main_fuel` override still wins (last-wins
# composition); a contradicting landlord fuel is logged, not silently overridden.
# Electric room heaters (Table 4a 691-701) are unambiguously electricity (RdSAP
# main_fuel code 29). Solid fuel is ambiguous (coal / anthracite / smokeless /
# dual fuel / wood logs / pellets), so it defaults to house coal (33) — the most
# common solid fuel — when the landlord gives no `main_fuel` override; a specific
# solid-fuel override still wins by last-wins composition.
_ELECTRICITY_FUEL = 29
_HOUSE_COAL_FUEL = 33
_ELECTRIC_ROOM_HEATER_CODES = frozenset(range(691, 702))
# Heat pumps (SAP Table 4a 211-224 wet, 521-527 warm-air) are category 4 and
# unambiguously electric (natural fuel 29). Modellable on the default code's SPF
# without a PCDB index (ADR-0041).
_HEAT_PUMP_CATEGORY = 4
_HEAT_PUMP_CODES = frozenset(range(211, 225)) | frozenset(range(521, 528))
# Community / heat-network heating (SAP Table 4a 301-304) is category 6; the
# calculator's `_is_heat_network` keys off code OR category 6. The boiler-driven
# schemes (301/302/303) are dominantly mains gas (community) — RdSAP main_fuel 20
# — per real community certs; the heat-pump scheme (304) is electricity
# (community), added with its archetype (ADR-0041).
_HEAT_NETWORK_CATEGORY = 6
_HEAT_NETWORK_CODES = frozenset({301, 302, 303, 304})
_COMMUNITY_BOILER_CODES = frozenset({301, 302, 303})
_COMMUNITY_GAS_FUEL = 20
# SAP Table 4c full boiler-control code: programmer + room thermostat + TRVs. The
# landlord names the boiler, not its controls — but a gas boiler installed under
# modern Building Regs must carry compliant controls, and this overlay already
@ -109,6 +164,29 @@ _MAIN_HEATING_CODES: dict[str, int] = {
"Electric storage heaters, fan": 404,
"Direct-acting electric": 191,
"Electric room heaters": 691,
"Solid fuel room heater, closed": 633,
# Default air-source heat pump — SAP Table 4a 211 ("flow temp in other
# cases", default SPF 2.30). Modellable without a PCDB index; an actual
# product index would refine it (ADR-0041).
"Air source heat pump": 211,
# Boiler-driven community / heat-network heating — SAP Table 4a 301. The
# calculator derives the heat-source efficiency (80%) + DLF (1.50 default)
# from the code (cert_to_inputs). A generic "Community heating" with no named
# source stays unmapped (-> None) — the source can't be assumed (ADR-0041).
"Community heating, boilers": 301,
# Community CHP + boilers — SAP Table 4a 302 (heat network, category 6).
"Community heating, CHP and boilers": 302,
# Modern standalone oil room heater — SAP Table 4a 623 ("Oil room heater,
# 2000 or later", no boiler). Natural fuel heating oil (ADR-0041).
"Oil room heater, 2000 or later": 623,
# Gas room heaters — each catalogue variant to its own SAP Table 4a code
# (wide efficiency spread, so no single representative). Natural fuel mains
# gas; an LPG dwelling is refined by a `main_fuel` override (ADR-0041).
"Gas room heater, condensing fire": 611,
"Gas room heater, decorative fuel-effect": 612,
"Gas room heater, flush live-effect": 605,
"Gas room heater, open flue 1980 or later": 603,
"Gas room heater, open flue pre-1980": 601,
}
@ -129,6 +207,38 @@ def _control_for(code: int) -> Optional[int]:
return _MANUAL_CHARGE_CONTROL
if code in _GAS_BOILER_CODES:
return _FULL_BOILER_CONTROL
if code in _CATEGORY_10_ROOM_HEATER_CODES:
return _ROOM_HEATER_CONTROL
return None
def _category_for(code: int) -> Optional[int]:
"""The SAP Table 4a heating category a code implies, where the archetype
fixes it. Room heaters are category 10 set so a system-only override reads
as a room-heater system, not whatever category the replaced system carried."""
if code in _CATEGORY_10_ROOM_HEATER_CODES:
return _ROOM_HEATER_CATEGORY
if code in _HEAT_PUMP_CODES:
return _HEAT_PUMP_CATEGORY
if code in _HEAT_NETWORK_CODES:
return _HEAT_NETWORK_CATEGORY
return None
def _natural_fuel_for(code: int) -> Optional[int]:
"""The fuel an archetype unambiguously implies (a coherent default), or None
where the fuel is ambiguous and must come from the `main_fuel` override. A
present `main_fuel` override wins by last-wins composition."""
if code in _ELECTRIC_ROOM_HEATER_CODES or code in _HEAT_PUMP_CODES:
return _ELECTRICITY_FUEL
if code in _SOLID_FUEL_ROOM_HEATER_CODES:
return _HOUSE_COAL_FUEL
if code in _OIL_ROOM_HEATER_CODES:
return _OIL_FUEL
if code in _GAS_ROOM_HEATER_CODES:
return _MAINS_GAS_FUEL
if code in _COMMUNITY_BOILER_CODES:
return _COMMUNITY_GAS_FUEL
return None
@ -151,6 +261,16 @@ def _gas_boiler_overlay(code: int) -> HeatingOverlay:
)
def natural_fuel_for(main_heating_value: str) -> Optional[int]:
"""The RdSAP `main_fuel` code a heating archetype unambiguously implies — its
natural fuel (ADR-0041) or None for an unmapped archetype. Exposed so a
plausibility check can compare it against a landlord `main_fuel` override."""
code = _MAIN_HEATING_CODES.get(main_heating_value)
if code is None:
return None
return _natural_fuel_for(code)
def main_heating_overlay_for(
main_heating_value: str, building_part: int
) -> Optional[EpcSimulation]:
@ -162,6 +282,8 @@ def main_heating_overlay_for(
return EpcSimulation(
heating=HeatingOverlay(
sap_main_heating_code=code,
main_heating_category=_category_for(code),
main_fuel_type=_natural_fuel_for(code),
meter_type=_meter_for(code),
main_heating_control=_control_for(code),
# A landlord override describes the existing dwelling, so its assumed

View file

@ -25,4 +25,14 @@ class MainHeatingSystemType(Enum):
ELECTRIC_STORAGE_FAN = "Electric storage heaters, fan"
DIRECT_ELECTRIC = "Direct-acting electric"
ELECTRIC_ROOM_HEATERS = "Electric room heaters"
SOLID_FUEL_ROOM_HEATER_CLOSED = "Solid fuel room heater, closed"
AIR_SOURCE_HEAT_PUMP = "Air source heat pump"
COMMUNITY_BOILERS = "Community heating, boilers"
COMMUNITY_CHP_AND_BOILERS = "Community heating, CHP and boilers"
OIL_ROOM_HEATER_POST_2000 = "Oil room heater, 2000 or later"
GAS_FIRE_CONDENSING = "Gas room heater, condensing fire"
GAS_FIRE_DECORATIVE = "Gas room heater, decorative fuel-effect"
GAS_FIRE_FLUSH_LIVE_EFFECT = "Gas room heater, flush live-effect"
GAS_FIRE_OPEN_FLUE_POST_1980 = "Gas room heater, open flue 1980 or later"
GAS_FIRE_OPEN_FLUE_PRE_1980 = "Gas room heater, open flue pre-1980"
UNKNOWN = "Unknown"

View file

@ -28,6 +28,7 @@ values are live vs no-op before any write.
from __future__ import annotations
import logging
from typing import Callable, Optional
from domain.epc.property_overlays.attribute_overlay import (
@ -41,6 +42,7 @@ from domain.epc.property_overlays.glazing_overlay import glazing_overlay_for
from domain.epc.property_overlays.main_fuel_overlay import fuel_overlay_for
from domain.epc.property_overlays.main_heating_system_overlay import (
main_heating_overlay_for,
natural_fuel_for,
)
from domain.epc.property_overlays.water_heating_overlay import (
water_heating_overlay_for,
@ -50,6 +52,8 @@ from domain.epc.property_overlays.wall_type_overlay import wall_overlay_for
from domain.modelling.simulation import EpcSimulation
from repositories.property.property_overrides_reader import ResolvedPropertyOverrides
logger = logging.getLogger(__name__)
# Each override component maps its value (+ building part) to an overlay, or None
# when the value isn't resolvable. Fabric (wall/roof) folds onto building parts;
# property_type / built_form_type are whole-dwelling categorical corrections
@ -67,9 +71,22 @@ _COMPONENT_OVERLAYS: dict[str, Callable[[str, int], Optional[EpcSimulation]]] =
}
# Components whose overlay must be applied LAST so an explicit value wins a
# default another overlay dragged. `apply_simulations` is last-wins and override
# rows arrive in arbitrary order, so a `main_fuel` override must be applied after
# the `main_heating_system` archetype, whose natural-fuel default it overrides
# (ADR-0041) — e.g. "smokeless coal" must beat a solid-fuel room heater's coal
# default.
_APPLY_LAST: frozenset[str] = frozenset({"main_fuel"})
def overlays_from(overrides: ResolvedPropertyOverrides) -> list[EpcSimulation]:
overlays: list[EpcSimulation] = []
for row in overrides.rows:
# Stable sort: non-`_APPLY_LAST` rows keep their order, `main_fuel` goes last.
ordered_rows = sorted(
overrides.rows, key=lambda row: row.override_component in _APPLY_LAST
)
for row in ordered_rows:
mapper = _COMPONENT_OVERLAYS.get(row.override_component)
if mapper is None:
continue
@ -77,3 +94,55 @@ def overlays_from(overrides: ResolvedPropertyOverrides) -> list[EpcSimulation]:
if overlay is not None:
overlays.append(overlay)
return overlays
# Coarse fuel family per RdSAP `main_fuel` code (main_fuel_overlay._FUEL_CODES),
# for the plausibility check. The natural fuel a solid-fuel archetype drags
# (house coal) is a *default* across the ambiguous solid family, so a same-family
# override (smokeless / dual fuel / biomass) is a refinement, not a contradiction
# — only a different family (gas/electric on a solid heater) is flagged.
_FUEL_FAMILY: dict[int, str] = {
26: "gas", 20: "gas",
27: "lpg", 3: "lpg", 17: "lpg",
28: "oil",
29: "electricity", 25: "electricity",
33: "solid", 15: "solid", 10: "solid", 31: "solid",
}
def _override_value(overrides: ResolvedPropertyOverrides, component: str) -> Optional[str]:
for row in overrides.rows:
if row.override_component == component:
return row.override_value
return None
def flag_fuel_mismatch(overrides: ResolvedPropertyOverrides) -> None:
"""Log (not raise) when a landlord `main_fuel` override contradicts the
`main_heating_system` archetype's natural fuel — a plausibility signal (e.g.
a gas fuel on an electric room heater). The override is still honoured (it
wins, ADR-0041); we only record the implausibility. Deferred handling."""
heating_value = _override_value(overrides, "main_heating_system")
fuel_value = _override_value(overrides, "main_fuel")
if heating_value is None or fuel_value is None:
return
natural = natural_fuel_for(heating_value)
fuel_overlay = fuel_overlay_for(fuel_value, 0)
override_fuel = (
fuel_overlay.heating.main_fuel_type
if fuel_overlay is not None and fuel_overlay.heating is not None
else None
)
if natural is None or not isinstance(override_fuel, int):
return
natural_family = _FUEL_FAMILY.get(natural)
override_family = _FUEL_FAMILY.get(override_fuel)
if natural_family is None or override_family is None:
return
if natural_family != override_family:
logger.warning(
"Landlord main_fuel %r contradicts the natural fuel of heating "
"system %r (override is honoured; flagged for review)",
fuel_value,
heating_value,
)

View file

@ -14,7 +14,10 @@ from domain.property.properties import Properties
from domain.property.property import Property, PropertyIdentity
from infrastructure.postgres.property_table import PropertyRow
from repositories.epc.epc_repository import EpcRepository
from repositories.property.landlord_override_overlays import overlays_from
from repositories.property.landlord_override_overlays import (
flag_fuel_mismatch,
overlays_from,
)
from repositories.property.property_overrides_reader import PropertyOverridesReader
from repositories.property.property_repository import (
PropertyIdentityInsert,
@ -63,7 +66,9 @@ class PropertyPostgresRepository(PropertyRepository):
no reader is wired (the overlay stays off) or the Property has none."""
if self._overrides_reader is None:
return []
return overlays_from(self._overrides_reader.overrides_for(property_id))
overrides = self._overrides_reader.overrides_for(property_id)
flag_fuel_mismatch(overrides)
return overlays_from(overrides)
def get(self, property_id: int) -> Property:
row = self._session.get(PropertyRow, property_id)

View file

@ -0,0 +1,176 @@
"""One-time re-classification of stale landlord main-heating-system overrides.
PRD #1361 Class A: the LLM landlord-description classifier ran against a
too-small `MainHeatingSystemType` taxonomy (ADR-0041), so canonical RdSAP
heating descriptions were stored against the wrong archetype community heating
and oil/solid room heaters dumped into "Gas CPSU", and direct-acting
"panel, convector or radiant" room heaters mis-read as off-peak storage (the
band-G crater). The taxonomy is now complete, so these *canonical* descriptions
re-resolve deterministically no LLM, no nondeterminism.
This rewrites the resolved archetype on every affected row in BOTH the
per-Property fact layer (`property_overrides.override_value`) and the
per-portfolio classifier cache (`landlord_main_heating_system_overrides.value`),
keyed on the raw description (`original_spreadsheet_description` / `description`).
DRY-RUN BY DEFAULT: prints the row counts it would change and writes nothing.
Pass `--apply` to execute inside a transaction. Idempotent only rows whose
stored value differs from the corrected archetype are touched, so a second run
is a no-op. Descriptions with no confident archetype yet (solid-fuel open-fire /
with-boiler, warm air, electric underfloor) are deliberately NOT remapped here.
"""
from __future__ import annotations
import argparse
from sqlalchemy import text
from domain.epc.property_overrides.main_heating_system_type import (
MainHeatingSystemType,
)
from scripts.e2e_common import build_engine, load_env
# Canonical RdSAP main-heating description (lowercased, as stored in
# `original_spreadsheet_description` / `description`) → the corrected archetype.
# Only descriptions whose correct archetype now exists in the taxonomy appear
# here; the value is a `MainHeatingSystemType` so a typo can't slip a non-archetype
# string into the DB (the enum is the single source of truth — ADR-0041).
REMAP: dict[str, MainHeatingSystemType] = {
"community heating systems: community boilers only (rdsap)": (
MainHeatingSystemType.COMMUNITY_BOILERS
),
"community heating systems: community chp and boilers (rdsap)": (
MainHeatingSystemType.COMMUNITY_CHP_AND_BOILERS
),
"electric (direct acting) room heaters: panel, convector or radiant heaters": (
MainHeatingSystemType.ELECTRIC_ROOM_HEATERS
),
"oil room heaters: room heater, 2000 or later": (
MainHeatingSystemType.OIL_ROOM_HEATER_POST_2000
),
"solid fuel room heaters: closed room heater": (
MainHeatingSystemType.SOLID_FUEL_ROOM_HEATER_CLOSED
),
"gas (including lpg) room heaters: condensing gas fire": (
MainHeatingSystemType.GAS_FIRE_CONDENSING
),
"gas (including lpg) room heaters: decorative fuel effect gas fire, "
"open to chimney": MainHeatingSystemType.GAS_FIRE_DECORATIVE,
"gas (including lpg) room heaters: flush fitting live fuel effect gas fire "
"(open fronted), sealed to fireplace opening": (
MainHeatingSystemType.GAS_FIRE_FLUSH_LIVE_EFFECT
),
"gas (including lpg) room heaters: gas fire, open flue, 1980 or later "
"(open fronted), sitting proud of, and sealed to, fireplace opening": (
MainHeatingSystemType.GAS_FIRE_OPEN_FLUE_POST_1980
),
"gas (including lpg) room heaters: gas fire, open flue, pre-1980 "
"(open fronted)": MainHeatingSystemType.GAS_FIRE_OPEN_FLUE_PRE_1980,
}
# property_overrides keys the raw text on `original_spreadsheet_description`; the
# classifier cache keys it on `description`. Both compared case-insensitively.
_PROPERTY_OVERRIDES_UPDATE = text(
"""
UPDATE property_overrides
SET override_value = :new_value
WHERE override_component = 'main_heating_system'
AND lower(original_spreadsheet_description) = :description
AND override_value <> :new_value
"""
)
# The cache `value` column is the Drizzle-owned `main_heating_system` pgEnum;
# compare/update it as text (no CAST) so a target value the DB enum does not yet
# carry doesn't error the read. The UPDATE is only issued for values the live
# enum already has — `_db_enum_values` gates it (see main()).
_CACHE_UPDATE = text(
"""
UPDATE landlord_main_heating_system_overrides
SET value = :new_value, updated_at = now()
WHERE lower(description) = :description
AND value::text <> :new_value
"""
)
_PROPERTY_OVERRIDES_COUNT = text(
"""
SELECT count(*) FROM property_overrides
WHERE override_component = 'main_heating_system'
AND lower(original_spreadsheet_description) = :description
AND override_value <> :new_value
"""
)
_CACHE_COUNT = text(
"""
SELECT count(*) FROM landlord_main_heating_system_overrides
WHERE lower(description) = :description
AND value::text <> :new_value
"""
)
_ENUM_VALUES = text(
"SELECT e.enumlabel FROM pg_enum e JOIN pg_type t ON t.oid = e.enumtypid "
"WHERE t.typname = 'main_heating_system'"
)
def main() -> None:
load_env()
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--apply",
action="store_true",
help="execute the updates (default: dry-run, writes nothing)",
)
args = parser.parse_args()
engine = build_engine()
with engine.begin() as conn:
conn.execute(text("SET statement_timeout = 120000"))
# The cache `value` is the Drizzle-owned pgEnum; only update it for
# archetypes the live enum already carries. The rest need the Drizzle
# migration (FE repo) before the cache — and the classifier — can store
# them; property_overrides (TEXT, what the modelling reads) is updated
# regardless, which is the actual band-G fix.
enum_values = {r[0] for r in conn.execute(_ENUM_VALUES)}
prop_total = 0
cache_total = 0
deferred: list[str] = []
for description, archetype in REMAP.items():
params = {"description": description, "new_value": archetype.value}
prop_n = conn.execute(_PROPERTY_OVERRIDES_COUNT, params).scalar() or 0
prop_total += prop_n
in_enum = archetype.value in enum_values
cache_n = conn.execute(_CACHE_COUNT, params).scalar() or 0 if in_enum else 0
cache_total += cache_n
if not in_enum and prop_n:
deferred.append(archetype.value)
if prop_n or cache_n:
tag = "" if in_enum else " [cache deferred: enum value missing]"
print(
f" {prop_n:>6} property_overrides + {cache_n} cache "
f"-> {archetype.value!r} <= {description!r}{tag}"
)
if args.apply:
conn.execute(_PROPERTY_OVERRIDES_UPDATE, params)
if in_enum:
conn.execute(_CACHE_UPDATE, params)
verb = "updated" if args.apply else "would update"
print(
f"\n{verb} {prop_total} property_overrides rows + {cache_total} "
f"classifier-cache rows across {len(REMAP)} descriptions."
)
if deferred:
print(
f"\n{len(set(deferred))} archetype(s) NOT in the Drizzle "
"`main_heating_system` pgEnum — their cache rows are deferred "
"until the FE-repo enum migration lands (property_overrides was "
"still updated, which is what the modelling reads):"
)
for v in sorted(set(deferred)):
print(f" {v!r}")
if not args.apply:
print("\nDRY-RUN — nothing written. Re-run with --apply to execute.")
if __name__ == "__main__":
main()

View file

@ -245,13 +245,14 @@ def test_off_peak_archetypes_drag_dual_others_drag_single() -> None:
@pytest.mark.parametrize(
"main_heating_value",
["Unknown", "", "Air source heat pump", "Community heating"],
["Unknown", "", "Community heating"],
)
def test_unresolvable_or_unmodelled_heating_produces_no_overlay(
main_heating_value: str,
) -> None:
# Heat pumps (main_heating_index_number) and community heating (community
# codes) don't map to a Table 4b sap_main_heating_code yet — no overlay.
# Genuinely unrecognised values (Unknown / empty) and not-yet-modelled
# community heating produce no overlay — the lodged EPC heating stands
# (ADR-0041). Air source heat pumps ARE now modelled (Table 4a 211).
# Act
simulation = main_heating_overlay_for(main_heating_value, 0)
@ -383,3 +384,238 @@ def test_every_resolvable_main_heating_value_decodes(
# Assert
assert simulation is not None
def test_solid_fuel_room_heater_decodes_to_the_closed_room_heater_code() -> None:
# A landlord-named solid-fuel room heater (e.g. a closed stove) is a
# recognised archetype, not a gas wet system — it must decode to its SAP
# Table 4a code (633, closed solid-fuel room heater), not overflow into the
# nearest wrong archetype the way the under-populated taxonomy did, sending
# "solid fuel room heaters: closed room heater" to Gas CPSU (ADR-0041).
# Act
simulation = main_heating_overlay_for("Solid fuel room heater, closed", 0)
# Assert — SAP Table 4a code 633 (closed solid-fuel room heater).
assert simulation is not None
assert simulation.heating is not None
assert simulation.heating.sap_main_heating_code == 633
def test_solid_fuel_room_heater_drags_its_coherent_room_heater_companions() -> None:
# The landlord names the system, not its category/control/meter. A solid-fuel
# room heater is a room-heater system (Table 4a category 10) on a single-rate
# meter (not off-peak storage), under the conservative room-heater control
# real certs lodge (Table 4e Group 6 code 2601 — "no thermostatic control",
# the lowest-SAP room-heater control, so it never over-credits an unobserved
# control, mirroring the storage manual-charge default). ADR-0035.
# Act
simulation = main_heating_overlay_for("Solid fuel room heater, closed", 0)
# Assert
assert simulation is not None
assert simulation.heating is not None
heating = simulation.heating
assert heating.main_heating_category == 10
assert heating.main_heating_control == 2601
assert heating.meter_type == "Single"
def test_electric_room_heaters_drag_their_natural_electricity_fuel() -> None:
# A landlord names the system; an electric room heater's natural fuel is
# unambiguously electricity (RdSAP main_fuel code 29). The overlay drags it so
# a system-only override is self-coherent even on a cert that lodged a
# different fuel — a later `main_fuel` override still wins (last-wins
# composition), and a contradicting fuel is logged, not silently kept
# (ADR-0041 natural-fuel coherence).
# Act
simulation = main_heating_overlay_for("Electric room heaters", 0)
# Assert
assert simulation is not None
assert simulation.heating is not None
assert simulation.heating.main_fuel_type == 29
def test_solid_fuel_room_heater_defaults_to_house_coal_as_its_natural_fuel() -> None:
# Solid fuel is fuel-ambiguous (coal / anthracite / smokeless / dual fuel /
# wood logs / pellets), but the system must still self-cohere when no
# main_fuel override is given. Default to house coal (RdSAP main_fuel code
# 33), the most common solid fuel; a main_fuel override naming a specific
# solid fuel still wins by last-wins composition (ADR-0041).
# Act
simulation = main_heating_overlay_for("Solid fuel room heater, closed", 0)
# Assert
assert simulation is not None
assert simulation.heating is not None
assert simulation.heating.main_fuel_type == 33
def test_air_source_heat_pump_decodes_to_the_default_ashp_code() -> None:
# A landlord names "air source heat pump" without a PCDB model. It is
# modellable via the SAP Table 4a default ASHP code (211, "flow temp in other
# cases", default SPF 2.30) — no main_heating_index_number needed — so it must
# decode to 211, not produce no overlay / fall to Unknown (ADR-0041).
# Act
simulation = main_heating_overlay_for("Air source heat pump", 0)
# Assert — SAP Table 4a code 211 (default air-source heat pump).
assert simulation is not None
assert simulation.heating is not None
assert simulation.heating.sap_main_heating_code == 211
def test_air_source_heat_pump_drags_its_coherent_companions() -> None:
# A heat pump is unambiguously electric (natural fuel 29) and SAP Table 4a
# category 4. Unlike storage/CPSU it does NOT imply an off-peak tariff (SAP
# §12 Rule 3 is conditional, not forcing), so the coherent default meter is
# single-rate — forcing Dual would assume an off-peak split it may not have
# and mis-bill it. The overlay drags these code-derived companions so a
# system-only override is self-coherent (ADR-0035 / ADR-0041).
# Act
simulation = main_heating_overlay_for("Air source heat pump", 0)
# Assert
assert simulation is not None
assert simulation.heating is not None
heating = simulation.heating
assert heating.main_heating_category == 4
assert heating.main_fuel_type == 29
assert heating.meter_type == "Single"
def test_community_boilers_decode_to_the_heat_network_boiler_code() -> None:
# A community/heat-network scheme driven by boilers is SAP Table 4a code 301
# (the calculator derives the heat-source efficiency + DLF from it). It must
# decode to 301, not the Gas CPSU (120) the under-populated taxonomy forced —
# a single-dwelling gas wet boiler is the wrong picture for a heat network
# (ADR-0041). A generic "Community heating" with no named source stays None.
# Act
simulation = main_heating_overlay_for("Community heating, boilers", 0)
# Assert — SAP Table 4a code 301 (boiler-driven community heating).
assert simulation is not None
assert simulation.heating is not None
assert simulation.heating.sap_main_heating_code == 301
def test_community_boilers_drag_heat_network_category_and_a_community_gas_fuel() -> None:
# A community boiler scheme is SAP main_heating_category 6 (heat network), so
# the calculator treats it as a heat network (cert_to_inputs `_is_heat_network`
# checks code OR category 6). Its natural fuel is mains gas (community) — RdSAP
# main_fuel code 20, the dominant fuel real community-boiler certs lodge. A
# specific main_fuel override (e.g. biomass community) still wins (ADR-0041).
# Act
simulation = main_heating_overlay_for("Community heating, boilers", 0)
# Assert
assert simulation is not None
assert simulation.heating is not None
heating = simulation.heating
assert heating.main_heating_category == 6
assert heating.main_fuel_type == 20
def test_community_chp_and_boilers_decode_to_their_heat_network_code() -> None:
# A community scheme with CHP + boilers is SAP Table 4a code 302, still a heat
# network (category 6) on community mains gas (20). Real audit data: 10
# properties carry this, today mis-bucketed to Gas CPSU (ADR-0041).
# Act
simulation = main_heating_overlay_for("Community heating, CHP and boilers", 0)
# Assert
assert simulation is not None
assert simulation.heating is not None
heating = simulation.heating
assert heating.sap_main_heating_code == 302
assert heating.main_heating_category == 6
assert heating.main_fuel_type == 20
def test_oil_room_heater_decodes_to_the_post_2000_code() -> None:
# A modern standalone oil room heater is SAP Table 4a code 623 ("Oil room
# heater, 2000 or later", no boiler). It must decode to 623, not the Gas CPSU
# the under-populated taxonomy forced (ADR-0041).
# Act
simulation = main_heating_overlay_for("Oil room heater, 2000 or later", 0)
# Assert — SAP Table 4a code 623.
assert simulation is not None
assert simulation.heating is not None
assert simulation.heating.sap_main_heating_code == 623
def test_oil_room_heater_drags_its_coherent_room_heater_companions() -> None:
# An oil room heater is a room-heater system (category 10) on a single-rate
# meter, under the conservative room-heater control (2601), with its natural
# fuel heating oil (RdSAP main_fuel 28). The overlay drags these so a
# system-only override is self-coherent (ADR-0035 / ADR-0041).
# Act
simulation = main_heating_overlay_for("Oil room heater, 2000 or later", 0)
# Assert
assert simulation is not None
assert simulation.heating is not None
heating = simulation.heating
assert heating.main_heating_category == 10
assert heating.main_heating_control == 2601
assert heating.main_fuel_type == 28
assert heating.meter_type == "Single"
@pytest.mark.parametrize(
("value", "code"),
[
("Gas room heater, condensing fire", 611),
("Gas room heater, decorative fuel-effect", 612),
("Gas room heater, flush live-effect", 605),
("Gas room heater, open flue 1980 or later", 603),
("Gas room heater, open flue pre-1980", 601),
],
)
def test_gas_room_heater_variants_decode_to_their_sap_codes(
value: str, code: int
) -> None:
# Gas room heaters span a wide efficiency range (decorative 0.20 -> condensing
# 0.85), so each catalogue variant maps to its own SAP Table 4a code rather
# than collapsing to one representative — which would over-credit a decorative
# fire and under-credit a condensing one. Today they mis-bucket to "Gas
# boiler, regular" (ADR-0041).
# Act
simulation = main_heating_overlay_for(value, 0)
# Assert
assert simulation is not None
assert simulation.heating is not None
assert simulation.heating.sap_main_heating_code == code
def test_gas_room_heater_drags_its_coherent_companions() -> None:
# A gas room heater is a room-heater system (category 10) on a single-rate
# meter, under the conservative room-heater control (2601). Its natural fuel
# is mains gas (26) — an LPG dwelling is refined by a main_fuel override
# (the overlay can't see the mains connection) (ADR-0041).
# Act
simulation = main_heating_overlay_for("Gas room heater, condensing fire", 0)
# Assert
assert simulation is not None
assert simulation.heating is not None
heating = simulation.heating
assert heating.main_heating_category == 10
assert heating.main_heating_control == 2601
assert heating.main_fuel_type == 26
assert heating.meter_type == "Single"

View file

@ -6,12 +6,23 @@ that fold onto the lodged EPC — per component, partial, skipping unmapped rows
from __future__ import annotations
import logging
import pytest
from datatypes.epc.domain.epc_property_data import BuildingPartIdentifier
from repositories.property.landlord_override_overlays import overlays_from
from domain.modelling.scoring.overlay_applicator import apply_simulations
from repositories.property.landlord_override_overlays import (
flag_fuel_mismatch,
overlays_from,
)
from repositories.property.property_overrides_reader import (
ResolvedPropertyOverride,
ResolvedPropertyOverrides,
)
from tests.domain.sap10_calculator.worksheet._elmhurst_worksheet_000490 import (
build_epc,
)
def test_roof_type_row_produces_a_roof_overlay() -> None:
@ -143,3 +154,74 @@ def test_unresolvable_rows_are_skipped() -> None:
# Assert
assert overlays == []
def test_main_fuel_override_wins_over_a_heating_archetype_natural_fuel() -> None:
# A heating-system override drags a natural fuel (a solid-fuel room heater
# defaults to house coal, RdSAP main_fuel 33), but an explicit main_fuel
# override must win. apply_simulations is last-wins and the override rows
# arrive in arbitrary order, so overlays_from must apply main_fuel AFTER
# main_heating_system — otherwise the natural-fuel default clobbers the
# explicit fuel (ADR-0041). The rows here are in the order that exposes it.
baseline = build_epc()
overrides = ResolvedPropertyOverrides(
rows=(
ResolvedPropertyOverride("main_fuel", 0, "smokeless coal"),
ResolvedPropertyOverride(
"main_heating_system", 0, "Solid fuel room heater, closed"
),
)
)
# Act
result = apply_simulations(baseline, overlays_from(overrides))
# Assert — the explicit fuel (smokeless coal 15) wins over the coal default.
assert result.sap_heating.main_heating_details[0].main_fuel_type == 15
def test_a_fuel_override_contradicting_the_heating_archetype_is_logged(
caplog: pytest.LogCaptureFixture,
) -> None:
# An electric room heater is unambiguously electricity; a "mains gas" main_fuel
# override contradicts it (a different fuel family). The override still wins
# (ADR-0041), but the implausibility is logged for later review — not raised.
overrides = ResolvedPropertyOverrides(
rows=(
ResolvedPropertyOverride(
"main_heating_system", 0, "Electric room heaters"
),
ResolvedPropertyOverride("main_fuel", 0, "mains gas"),
)
)
# Act
with caplog.at_level(logging.WARNING):
flag_fuel_mismatch(overrides)
# Assert
assert any("fuel" in r.message.lower() for r in caplog.records)
def test_a_same_family_fuel_refinement_is_not_logged(
caplog: pytest.LogCaptureFixture,
) -> None:
# A solid-fuel room heater's natural fuel (house coal 33) is a *default* for
# the ambiguous solid family. A "smokeless coal" override refines it within
# the same family — that is expected, not a contradiction, so it must NOT log.
# Only a different fuel family (gas/electric on a solid heater) is flagged.
overrides = ResolvedPropertyOverrides(
rows=(
ResolvedPropertyOverride(
"main_heating_system", 0, "Solid fuel room heater, closed"
),
ResolvedPropertyOverride("main_fuel", 0, "smokeless coal"),
)
)
# Act
with caplog.at_level(logging.WARNING):
flag_fuel_mismatch(overrides)
# Assert
assert caplog.records == []

View file

@ -0,0 +1,30 @@
"""The one-time heating-override re-classification map is internally valid."""
from __future__ import annotations
from domain.epc.property_overlays.main_heating_system_overlay import (
main_heating_overlay_for,
)
from domain.epc.property_overrides.main_heating_system_type import (
MainHeatingSystemType,
)
from scripts.reclassify_heating_overrides import REMAP
def test_every_remap_target_is_a_resolvable_archetype() -> None:
# The remap must never point a stored description at an archetype the overlay
# can't model — that would replace one broken value with another. Every
# target must decode to a real SAP heating code (ADR-0041).
for description, archetype in REMAP.items():
simulation = main_heating_overlay_for(archetype.value, 0)
assert simulation is not None, description
assert simulation.heating is not None, description
assert simulation.heating.sap_main_heating_code is not None, description
def test_remap_targets_are_main_heating_system_members() -> None:
# Belt and braces: the values are enum members (a typo can't smuggle a
# non-archetype string into the DB).
for archetype in REMAP.values():
assert archetype in MainHeatingSystemType