Merge pull request #1457 from Hestia-Homes/fix/complete-heating-companion-taxonomy

Complete the heating-override companion taxonomy; an incomplete set now raises (closes #1444)

docs/adr/0053-heating-override-companion-taxonomy-complete-incomplete-set-raises.md

@@ -0,0 +1,109 @@
+# The heating-override companion taxonomy is complete; an incomplete set now raises
+
+## Status
+
+accepted
+
+## Context
+
+[ADR-0048](0048-heating-override-forces-category-and-control-defers-fuel-and-meter.md)
+set the invariant — a system-replacing heating override emits a complete
+category + control companion set, never inheriting the replaced system's — and
+intended, once the taxonomy was covered, to "flip the log to a raise, so the
+coherence invariant is enforced" (#1444 Part C).
+[ADR-0050](0050-electric-underfloor-override-drags-category-8-and-conservative-group-7-control.md)
+(underfloor) and
+[ADR-0052](0052-electric-boiler-and-cpsu-overrides-drag-category-2-and-conservative-group-1-control.md)
+(electric boilers) filled two families. Two archetype families remained on the
+gap-log, and with them Part C stayed blocked:
+
+- **"Air source heat pump" (SAP 211)** — category 4 already forced, control
+  `None` (Table 4e Group 2 unmapped). No live overrides today (census across
+  all portfolios: 0), so this is taxonomy completion, not a live defect.
+- **"Community heating, boilers" / "…, CHP and boilers" (301/302)** — category
+  6 already forced, control `None` (Group 3 unmapped). **2,220 live
+  overrides** — the largest population on the log. The inherited control is
+  doubly load-bearing for heat networks: it feeds the Table 4e temperature
+  adjustment *and* the SAP 10.2 Table 4c(3) charging factors (worksheet
+  (305)/(305a)).
+
+## Decision
+
+1. **Wet heat pumps (Table 4a 211-224) drag control 2201** — Table 4e Group 2
+   "no time or thermostatic control of room temperature", **+0.3 °C**, the
+   largest Group 2 adjustment: the conservative in-group default (the heat-pump
+   mirror of storage 2401 / room-heater 2601 / underfloor 2701 / boiler 2101).
+   Warm-air heat pumps (521-527) are deliberately left unmapped — no archetype
+   reaches them, and the raise below is the guard that a future warm-air
+   archetype cannot land without choosing its Group 5 control.
+
+   This is the *existing-system* fill policy: the ASHP **Measure** overlay
+   installs 2210 (time-and-temperature zone control) because a measure designs
+   a modern end-state (ADR-0024); a landlord override describes an existing
+   pump whose control was never observed, so the conservative default applies.
+
+2. **Community heating (301/302) defaults to control 2313, keeping an
+   assessor-observed Group 3 control** (decided with Khalim, revising the
+   first-cut 2301). The control encodes two independent unknowns and the
+   default is chosen per axis:
+   - **Kit axis** (thermostat/TRVs): community schemes skew modern, and the
+     portfolio's own lodged community certs are overwhelmingly thermostatted
+     (modal control 2306; none lodge 2301) — so 2313 ("flat rate charging,
+     room thermostat and TRVs", temperature adjustment 0.0) concedes this
+     axis to the evidence.
+   - **Charging axis** (flat rate vs usage-linked): the billing arrangement
+     is scheme paperwork an override genuinely cannot see, so flat rate
+     (Table 4c(3) space **1.05**, DHW **1.05**) is retained — the
+     usage-linked credit (1.0/1.0) is never assumed.
+   - **Keep-observed**: when the cert already lodges a Group 3 control the
+     override is confirming a community system, not replacing one — the
+     assessor observed that control, and stamping any default over it
+     destroys real information (a usage-linked 2306 is worth ~1-2 SAP over
+     flat-rate). The overlay opts in via
+     `keep_existing_heat_network_control` (the control mirror of ADR-0035's
+     `keep_existing_off_peak_meter`); only a stale cross-family control (the
+     boiler-2110 / storage-2401 inheritances) is replaced by 2313.
+
+3. **An incomplete companion set now raises** (`ValueError`) instead of
+   logging. Every archetype in `_MAIN_HEATING_CODES` resolves a complete
+   category + control (gas boilers via their dedicated overlay; all others via
+   `_category_for`/`_control_for`), so the log-and-continue mitigation has no
+   remaining population — and a silent log was only ever a stop-gap
+   (ADR-0048: "the intended end-state is to fill each archetype's
+   category/control … and then flip the log to a raise"). A future archetype
+   added without companions now fails loudly at overlay time rather than
+   shipping a silently-incoherent cert.
+
+## Consequences
+
+- The companion-gap error log is gone; #1444 closes.
+- Community overrides converge on defensible controls with minimal score
+  churn. Live sample (portfolio 796, 12 properties): the ten dwellings with
+  assessor-observed Group 3 controls (2306/2307/2309) shift **±0.0** — their
+  observed controls survive the override — and only the two group-incoherent
+  dwellings (a Group-1 *boiler* zone control 2110 riding on a heat network)
+  move, **−1.0** each onto the 2313 default. Contrast the first-cut 2301
+  design, which shifted the whole sample 0 to −3.
+- Adding a heating archetype now *requires* choosing its category + control
+  (or extending the family sets) — the raise turns the ADR-0048 checklist into
+  an enforced contract.
+
+### Alternatives rejected
+
+- **The maximally conservative community default (2301).** The first cut.
+  Rejected on evidence: no lodged community cert in the portfolio carries
+  2301, so it under-rates the typical (modern, thermostatted) scheme and
+  shifted the sampled cohort 0 to −3 SAP for no informational gain.
+- **A usage-linked community default (2306/2312/2314).** Rejected: credits a
+  billing arrangement the landlord never reported — the charging axis is the
+  one the override genuinely cannot observe, so it stays flat-rate.
+- **Stamping the default over an observed Group 3 control.** Rejected: the
+  ADR-0035 "always stamp" pattern exists to kill *stale* companions; an
+  in-group control on a confirmed community cert is observed data, and
+  overwriting it is information destruction, not coherence.
+- **Keep log-and-continue indefinitely.** Rejected: the log was the interim
+  mitigation while the taxonomy was incomplete; leaving it means the next
+  archetype ships the ADR-0048 inheritance bug silently again.
+- **Raise on warm-air heat-pump codes' missing control now.** Not applicable —
+  no archetype maps to 521-527; the raise fires exactly when one is added
+  without its Group 5 control, which is the point.

domain/epc/property_overlays/main_heating_system_overlay.py

@@ -28,7 +28,6 @@ are left UNKNOWN until modelled. Unresolvable values produce no overlay.
 
 from __future__ import annotations
 
-import logging
 from typing import Optional
 
 from domain.modelling.simulation import EpcSimulation, HeatingOverlay
@@ -36,8 +35,6 @@ from domain.sap10_calculator.tables.table_12a import (
     OFF_PEAK_IMPLYING_HEATING_CODES,
 )
 
-logger = logging.getLogger(__name__)
-
 # Off-peak (Economy 7) meter. Electric storage / CPSU systems charge overnight at
 # the low rate and cannot run economically on a single-rate meter; "Dual" lets
 # the §12 dispatch resolve the specific tariff (storage 7-hour, CPSU 10-hour).
@@ -168,6 +165,17 @@ _UNDERFLOOR_CONTROL = 2701
 # without a PCDB index (ADR-0041).
 _HEAT_PUMP_CATEGORY = 4
 _HEAT_PUMP_CODES = frozenset(range(211, 225)) | frozenset(range(521, 528))
+# SAP Table 4e Group 2 heat-pump control: 2201 ("no time or thermostatic
+# control of room temperature", +0.3 C) is the conservative default when the
+# landlord named only the system — the largest Group 2 adjustment, so an
+# unobserved control is never over-credited. Deliberately NOT the ASHP
+# Measure's 2210 zone control: a measure designs a modern end-state
+# (ADR-0024); an override describes an existing pump (ADR-0053). Scoped to the
+# WET heat pumps (211-224) — warm-air pumps (521-527) take Table 4e Group 5
+# and stay unmapped (no archetype maps to them; the incomplete-companion raise
+# guards that a future one cannot land without choosing its Group 5 control).
+_WET_HEAT_PUMP_CODES = frozenset(range(211, 225))
+_CONSERVATIVE_HEAT_PUMP_CONTROL = 2201
 
 # Community / heat-network heating (SAP Table 4a 301-304) is category 6; the
 # calculator's `_is_heat_network` keys off code OR category 6. The boiler-driven
@@ -177,6 +185,18 @@ _HEAT_PUMP_CODES = frozenset(range(211, 225)) | frozenset(range(521, 528))
 _HEAT_NETWORK_CATEGORY = 6
 _HEAT_NETWORK_CODES = frozenset({301, 302, 303, 304})
 _COMMUNITY_BOILER_CODES = frozenset({301, 302, 303})
+# SAP Table 4e Group 3 heat-network control: 2313 ("flat rate charging, room
+# thermostat and TRVs") is the default when the landlord named only the scheme
+# — a per-axis choice (ADR-0053, decided with Khalim). KIT axis: community
+# schemes skew modern and the portfolio's lodged community certs are
+# overwhelmingly thermostatted (modal 2306; none lodge 2301), so a room
+# thermostat + TRVs is the evidence-backed assumption (temperature adjustment
+# 0.0). CHARGING axis: flat rate (Table 4c(3) space 1.05 / DHW 1.05) — the
+# billing arrangement is scheme paperwork an override cannot see, so the
+# usage-linked credit (1.0/1.0) is never assumed. An assessor-OBSERVED Group 3
+# control on the cert is kept in preference to this default (the overlay sets
+# `keep_existing_heat_network_control`; the applicator honours it).
+_DEFAULT_HEAT_NETWORK_CONTROL = 2313
 _COMMUNITY_GAS_FUEL = 20
 
 # SAP Table 4c full boiler-control code: programmer + room thermostat + TRVs. The
@@ -305,6 +325,10 @@ def _control_for(code: int) -> Optional[int]:
         return _UNDERFLOOR_CONTROL
     if code in _ELECTRIC_BOILER_CODES:
         return _CONSERVATIVE_BOILER_CONTROL
+    if code in _WET_HEAT_PUMP_CODES:
+        return _CONSERVATIVE_HEAT_PUMP_CONTROL
+    if code in _HEAT_NETWORK_CODES:
+        return _DEFAULT_HEAT_NETWORK_CONTROL
     return None
 
 
@@ -395,21 +419,19 @@ def main_heating_overlay_for(
     category = _category_for(code)
     control = _control_for(code)
     if category is None or control is None:
-        # A system-replacing override should stamp a complete, coherent companion
+        # A system-replacing override must stamp a complete, coherent companion
         # set; an unmapped category or control leaves the field unset, so the
-        # effective cert inherits the REPLACED system's value (a silent mis-rating
-        # — e.g. a room heater keeping a storage heater's category/control). We do
-        # not yet have confident defaults for every archetype, so log-and-continue
-        # (matching `flag_fuel_mismatch`) to make the gap visible for review
-        # rather than raise mid-run.
-        logger.error(
-            "Landlord main_heating_system %r (SAP code %d) overlaid with an "
-            "incomplete companion set (category=%s, control=%s); the effective "
-            "cert may inherit the replaced system's value — known gap, review",
-            main_heating_value,
-            code,
-            category,
-            control,
+        # effective cert inherits the REPLACED system's value — the silent
+        # mis-rating class ADR-0048 exists to stop. Every mapped archetype now
+        # carries confident defaults (ADR-0050/0052/0053), so an incomplete set
+        # can only mean a new archetype landed without choosing its companions:
+        # fail loudly at overlay time rather than ship an incoherent cert
+        # (ADR-0048's intended end-state, #1444 Part C).
+        raise ValueError(
+            f"Landlord main_heating_system {main_heating_value!r} (SAP code "
+            f"{code}) resolves an incomplete companion set (category="
+            f"{category}, control={control}); add the archetype's SAP Table 4a "
+            f"category and conservative Table 4e control before mapping it"
         )
     return EpcSimulation(
         heating=HeatingOverlay(
@@ -423,5 +445,10 @@ def main_heating_overlay_for(
             # (e.g. a 24-hour all-low tariff). Measures, which re-meter for real,
             # build their HeatingOverlay directly and leave this False.
             keep_existing_off_peak_meter=True,
+            # Likewise a heat-network override's control default (2313) must
+            # not overwrite an assessor-OBSERVED Group 3 control on a community
+            # cert being confirmed — only a stale cross-family control is
+            # replaced (ADR-0053).
+            keep_existing_heat_network_control=code in _HEAT_NETWORK_CODES,
         )
     )

domain/modelling/scoring/overlay_applicator.py

@@ -152,6 +152,13 @@ _SAP_HEATING_FIELDS: tuple[str, ...] = (
 _ENERGY_SOURCE_FIELDS: tuple[str, ...] = ("meter_type", "gas_connection_available")
 
 
+def _is_heat_network_control(control: object) -> bool:
+    """True iff the lodged control is a SAP Table 4e Group 3 (heat network)
+    code — 2301-2314 — i.e. an assessor-observed community control a
+    heat-network override's default must not overwrite (ADR-0053)."""
+    return isinstance(control, int) and 2301 <= control <= 2314
+
+
 def _is_off_peak_meter(meter_type: object) -> bool:
     """True iff the meter resolves to an off-peak Table 12a tariff (not the
     STANDARD single-rate column). Unparseable / absent meters count as not
@@ -177,8 +184,21 @@ def _fold_heating(epc: EpcPropertyData, overlay: HeatingOverlay) -> None:
     main = epc.sap_heating.main_heating_details[0]
     for field_name in _MAIN_HEATING_FIELDS:
         value = getattr(overlay, field_name)
-        if value is not None:
-            setattr(main, field_name, value)
+        if value is None:
+            continue
+        # A community override's control is a coherent default, not an observed
+        # one: when it opts in (`keep_existing_heat_network_control`) and the
+        # cert already lodges a Table 4e GROUP 3 control (assessor-observed on
+        # a community cert being confirmed), keep the cert's. A cross-family
+        # control (boiler 2110, storage 2401) is stale and is still replaced
+        # (ADR-0053).
+        if (
+            field_name == "main_heating_control"
+            and overlay.keep_existing_heat_network_control
+            and _is_heat_network_control(main.main_heating_control)
+        ):
+            continue
+        setattr(main, field_name, value)
     # `main_heating_index_number` (PCDB-resolved, e.g. a heat pump) and
     # `sap_main_heating_code` (Table 4a-resolved, e.g. storage heaters) are
     # mutually-exclusive efficiency anchors: a whole-system replacement to one

domain/modelling/simulation.py

@@ -200,6 +200,15 @@ class HeatingOverlay:
     # MEASURE re-meters for real (Elmhurst re-lodges 18-hour → Dual on a storage
     # install), so it leaves this False. `_fold_heating` reads it.
     keep_existing_off_peak_meter: bool = False
+    # The community-heating mirror of the flag above (ADR-0053): a heat-network
+    # override's `main_heating_control` is a coherent DEFAULT, not an observed
+    # control — when the cert already lodges a Table 4e Group 3 control the
+    # assessor observed it (a community cert being confirmed, not replaced), so
+    # the default must not overwrite it (e.g. a usage-linked 2306 downgraded to
+    # flat-rate 2313 destroys ~1-2 real SAP). A cross-family control (a boiler
+    # 2110, storage 2401) is stale by definition and is still replaced.
+    # `_fold_heating` reads it.
+    keep_existing_heat_network_control: bool = False
 
 
 @dataclass(frozen=True)

tests/domain/epc/test_main_heating_system_overlay.py

@@ -92,20 +92,66 @@ def test_electric_room_heaters_overlay_the_room_heater_charge_control() -> None:
     assert simulation.heating.main_heating_control == 2601
 
 
-def test_overlay_logs_when_it_cannot_complete_a_companion_set(
-    caplog: pytest.LogCaptureFixture,
+def test_overlay_raises_on_an_incomplete_companion_set(
+    monkeypatch: pytest.MonkeyPatch,
 ) -> None:
-    # An archetype the overlay cannot yet fully companion — here an air-source
-    # heat pump, whose Table 4e Group 2 conservative control is not yet mapped —
-    # is still overlaid (log-and-continue), but the incomplete companion set is
-    # logged as an error so the known gap is visible (CloudWatch) rather than
-    # silently inheriting the replaced system's stale value.
-    # Act
-    with caplog.at_level(logging.ERROR):
-        main_heating_overlay_for("Air source heat pump", 0)
+    # Every real archetype resolves a complete companion set, so the
+    # log-and-continue mitigation has no remaining population and the
+    # ADR-0048 end-state applies: an incomplete set FAILS LOUDLY instead of
+    # shipping a silently-incoherent cert (ADR-0053, #1444 Part C). Simulated
+    # with a hypothetical warm-air heat-pump archetype (SAP 521 — category 4
+    # but its Table 4e Group 5 control is deliberately unmapped): a future
+    # archetype cannot land without choosing its companions.
+    from domain.epc.property_overlays import main_heating_system_overlay as mod
 
-    # Assert — the gap is reported and names the archetype.
-    assert "Air source heat pump" in caplog.text
+    monkeypatch.setitem(mod._MAIN_HEATING_CODES, "Warm-air heat pump", 521)
+
+    # Act / Assert — the raise names the archetype.
+    with pytest.raises(ValueError, match="Warm-air heat pump"):
+        main_heating_overlay_for("Warm-air heat pump", 0)
+
+
+@pytest.mark.parametrize(
+    "main_heating_value",
+    ["Community heating, boilers", "Community heating, CHP and boilers"],
+)
+def test_community_heating_drags_the_default_group_3_control(
+    main_heating_value: str,
+) -> None:
+    # The landlord names the scheme, not its control or charging arrangement.
+    # The Table 4e Group 3 default is 2313 ("flat rate charging, room
+    # thermostat and TRVs"): community schemes skew modern and the portfolio's
+    # own lodged community certs are overwhelmingly thermostatted (modal 2306;
+    # none lodge 2301), so the KIT axis concedes to the evidence — while the
+    # CHARGING axis stays flat-rate (space 1.05 / DHW 1.05): the billing
+    # arrangement is scheme paperwork an override genuinely cannot see, so
+    # usage-linked (1.0/1.0) is never assumed (ADR-0053, decided with Khalim).
+    # Leaving the control unset inherits the replaced system's instead (the
+    # ADR-0048 bug class).
+    # Act
+    simulation = main_heating_overlay_for(main_heating_value, 0)
+
+    # Assert — SAP Table 4e Group 3 default heat-network control.
+    assert simulation is not None
+    assert simulation.heating is not None
+    assert simulation.heating.main_heating_control == 2313
+
+
+def test_air_source_heat_pump_drags_the_conservative_group_2_control() -> None:
+    # The landlord names the system, not its control, so the overlay assumes
+    # the conservative Table 4e Group 2 default: 2201 ("no time or thermostatic
+    # control", +0.3 C — the largest Group 2 adjustment, so an unobserved
+    # control is never over-credited; the heat-pump mirror of storage 2401 /
+    # boiler 2101). Deliberately NOT the ASHP Measure's 2210 zone control —
+    # a measure designs a modern end-state (ADR-0024); an override describes an
+    # existing pump (ADR-0053).
+    # Act
+    simulation = main_heating_overlay_for("Air source heat pump", 0)
+
+    # Assert — SAP Table 4e Group 2 conservative heat-pump control.
+    assert simulation is not None
+    assert simulation.heating is not None
+    assert simulation.heating.main_heating_control == 2201
 
 
 def test_complete_room_heater_overlay_does_not_log_a_companion_gap(
@@ -601,6 +647,51 @@ def test_unresolvable_or_unmodelled_heating_produces_no_overlay(
     assert simulation is None
 
 
+def test_community_override_keeps_an_observed_group_3_control() -> None:
+    # A community override on a cert that ALREADY lodges community heating is a
+    # confirmation, not a system switch — the assessor observed the control
+    # (e.g. 2306, "charging linked to use, programmer and TRVs", worth ~1-2 SAP
+    # over the flat-rate default). Stamping the 2313 default over it would
+    # destroy real information, so the overlay opts into keeping an existing
+    # Table 4e GROUP 3 control (the heat-network mirror of
+    # `keep_existing_off_peak_meter`, ADR-0053).
+    # Arrange — a cert lodging community heating with an observed 2306.
+    baseline = build_epc()
+    main = baseline.sap_heating.main_heating_details[0]
+    main.sap_main_heating_code = 301
+    main.main_heating_category = 6
+    main.main_heating_control = 2306
+    overlay = main_heating_overlay_for("Community heating, boilers", 0)
+    assert overlay is not None
+
+    # Act
+    result = apply_simulations(baseline, [overlay])
+
+    # Assert — the observed usage-linked control survives.
+    assert result.sap_heating.main_heating_details[0].main_heating_control == 2306
+
+
+def test_community_override_replaces_a_cross_family_control() -> None:
+    # When the replaced system was a DIFFERENT family, its control is stale by
+    # definition — a Group-1 boiler zone control (2110) has no meaning on a
+    # heat network (two live portfolio-796 dwellings carried exactly that) —
+    # so the Group 3 default is stamped (ADR-0053).
+    # Arrange — a cert lodging a gas boiler with zone control.
+    baseline = build_epc()
+    main = baseline.sap_heating.main_heating_details[0]
+    main.sap_main_heating_code = 102
+    main.main_heating_category = 2
+    main.main_heating_control = 2110
+    overlay = main_heating_overlay_for("Community heating, boilers", 0)
+    assert overlay is not None
+
+    # Act
+    result = apply_simulations(baseline, [overlay])
+
+    # Assert — the stale boiler control is replaced by the Group 3 default.
+    assert result.sap_heating.main_heating_details[0].main_heating_control == 2313
+
+
 def test_main_heating_override_remaps_the_primary_system_code() -> None:
     # Arrange
     baseline = build_epc()