From 7ecb7eebd18e7b2fa50e8c9bd8ccbe78238c65b2 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 30 Jun 2026 22:07:27 +0000 Subject: [PATCH 1/6] Record that a predicted Property has no Lodged Performance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Amends ADR-0004's both-halves-always-populated invariant: EPC Prediction borrows a neighbour's recorded figures, so a predicted Property has no Lodged Performance — its lodged half is absent (NULL), only the Effective half is persisted. Sharpens the CONTEXT.md glossary for Lodged / Baseline Performance. Co-Authored-By: Claude Opus 4.8 (1M context) --- CONTEXT.md | 4 +- ...eline-performance-lodged-effective-pair.md | 55 +++++++++++++++++++ 2 files changed, 57 insertions(+), 2 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index fe6493ee4..70e9be18e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -116,11 +116,11 @@ Deterministically translating an **old / reduced-data EPC schema** into the curr _Avoid_: gap-fill (means the neighbour-ML path), reduced-data expansion (overloaded with the calculator's Table-5 step), remapping (the schema-translation part only) **Baseline Performance**: -A Property's current performance aggregate, holding both Lodged Performance and Effective Performance plus the energy block: delivered kWh **per end use** (heating, hot water, lighting, appliances, cooking, pumps/fans, cooling) and the **annual bill** composed into per-section costs plus a total, produced by **Bill Derivation** from SAP10 Calculation's per-end-use kWh × current Fuel Rates. Persisted as one row (flat typed columns, per-section kWh + cost + total); surfaced as one block in the UI. +A Property's current performance aggregate, holding both Lodged Performance and Effective Performance plus the energy block: delivered kWh **per end use** (heating, hot water, lighting, appliances, cooking, pumps/fans, cooling) and the **annual bill** composed into per-section costs plus a total, produced by **Bill Derivation** from SAP10 Calculation's per-end-use kWh × current Fuel Rates. Persisted as one row (flat typed columns, per-section kWh + cost + total); surfaced as one block in the UI. The **Lodged half is optional**: a predicted Property has no lodged record (see Lodged Performance), so its `lodged_*` are `NULL` and only the Effective half + energy block are populated (ADR-0004 amendment, #1361). _Avoid_: baseline predictions, predicted baseline, rebaselined values **Lodged Performance**: -The SAP / EPC Band / carbon emissions / Primary Energy Intensity recorded on the public EPC (or the Site Notes' as-surveyed values when Site Notes are the source) — unmodified by modelling. The half of Baseline Performance that says "what the government register says about this Property". +The SAP / EPC Band / carbon emissions / Primary Energy Intensity recorded on the public EPC (or the Site Notes' as-surveyed values when Site Notes are the source) — unmodified by modelling. The half of Baseline Performance that says "what the government register says about this Property". **Requires a record _of this Property_** — a lodged cert or a Site Notes survey — so it is **absent (`None`/NULL) for a predicted Property**: **EPC Prediction** synthesises the picture from neighbours, copying a comparable's recorded figures, so there is no government-register record of _this_ Property to lodge. Only the Effective half is persisted then (ADR-0004 amendment, #1361). Reading a predicted EPC's recorded fields as Lodged Performance manufactures a phantom — a neighbour's SAP presented as this Property's lodged figure. _Avoid_: original performance, raw EPC values, recorded baseline **Effective Performance**: diff --git a/docs/adr/0004-baseline-performance-lodged-effective-pair.md b/docs/adr/0004-baseline-performance-lodged-effective-pair.md index fc27be7da..a8b4ba476 100644 --- a/docs/adr/0004-baseline-performance-lodged-effective-pair.md +++ b/docs/adr/0004-baseline-performance-lodged-effective-pair.md @@ -39,3 +39,58 @@ than churning the table twice. The SQLModel row is defined in `infrastructure/postgres/` so the ephemeral-Postgres tests build it via `create_all`; the production migration is FE-owned (Drizzle ORM) and tracked in `docs/migrations/`. + +### Amendment (2026-06-30, #1361 Class B): the Lodged half is absent for a predicted Property + +The original invariant — **every** `PropertyBaselinePerformance` populates **both** halves, even when +equal — assumed every Property has a record *of its own* performance to lodge: a public EPC cert (the +`epc_with_overlay` path) or a Site Notes survey. It does **not** hold for the **EPC Prediction** path +(ADR-0029/0031). A predicted Property has no record of itself — its `EpcPropertyData` is deterministic +neighbour synthesis that copies a representative comparable's structure wholesale, so the +`energy_rating_current` / band / CO2 / Primary Energy Intensity on the synthesised picture are a +*different dwelling's* lodged figures. Reading **Lodged Performance** off it manufactures a **phantom**: +a borrowed neighbour's SAP presented as this Property's government-register figure. This affected +**every** predicted Property (~12,236 rows estate-wide), not the 8 the `effective-lodged-divergence` +audit surfaced — that audit only fires when the borrowed figure lands ≥15 SAP from the Effective, so it +under-counts the phantom by three orders of magnitude. + +**Decision.** The Lodged half is **optional**. When `source_path == "predicted"` there is no Lodged +Performance: `PropertyBaselinePerformance.lodged` is `None` and the four `lodged_*` columns are `NULL`. +The **Effective half is unchanged and still persisted** — a predicted Property is a first-class modelled +output (it flows through Rebaselining, Bill Derivation, and Modelling like any other; its Effective +Performance and bill block are correct and load-bearing for the FE and the plan-vs-effective audit +checks). `rebaseline_reason` stays `physical_state_changed` / `both`, which already records that the +Effective figure was scored from a changed picture. + +**Principle (the boundary).** Lodged Performance requires a record **of this Property** — a lodged cert +*or* a Site Notes survey (the as-surveyed values are a real observation of this dwelling). EPC +Prediction borrows a neighbour's, so it has none. The branch is therefore `source_path == "predicted"` — +**not** `physical_state_changed` (true for Site Notes and Landlord Overrides too) and **not** "no public +EPC" (Site Notes has none yet keeps a legitimate Lodged Performance). Whether the Site Notes as-surveyed +values are truly "lodged" is a separate question, deferred. + +**Rejected — a sentinel (`lodged = 0`).** Considered, to make "no record" visibly present rather than an +empty cell. Rejected: `0` is a valid-looking SAP score that (a) re-trips `effective-lodged-divergence` +for every predicted Property (`|effective − 0| ≥ 15`), (b) poisons every `AVG(lodged_*)` aggregate that +`NULL` is correctly excluded from, and (c) has no coherent `lodged_epc_band` enum member. The "no lodged +record" signal belongs in `NULL` (honest absence) plus the **structural provenance** already carried by +the distinct predicted-EPC slot — which the FE renders as a predicted badge — not overloaded onto the +score column. + +**Consequences.** +- The four `lodged_*` columns become **nullable**. The production table is **FE-owned (Drizzle)**, so the + migration (`ALTER … DROP NOT NULL`) lands in the FE repo and **must precede** any backend write of a + `NULL` lodged, or the predicted-Property INSERT violates the constraint and aborts the batch + (ADR-0012). The SQLModel mirror in `infrastructure/postgres/` is updated to `Optional` so the + ephemeral-Postgres tests build the nullable shape. +- The fix lives in `PropertyBaselineOrchestrator.run()` — the single chokepoint both the First Run + pipeline and `applications/modelling_e2e/handler.py` call — so one change repairs both entry points. + The Rebaseliner port takes `Optional[Performance]`; for a predicted Property `physical_state_changed` + is always true, so `CalculatorRebaseliner` adopts the calculator output and never reads the absent + lodged half (the divergence-log path is the pristine-cert case only, where lodged is non-null). +- A one-time backfill (`scripts/`, dry-run default + `--apply`, idempotent) NULLs the four `lodged_*` + columns on existing predicted-source rows (~12,236). It corrects only the Lodged half; Effective, the + bill block, and `rebaseline_reason` are left intact. +- `effective-lodged-divergence` already short-circuits on `lodged_sap IS NULL`, so it goes green for + predicted Properties once backfilled. Class C of #1361 (a floor that ignores implausibly-low lodged + scores on *real* certs) is an additive guard, independent of this change. From 5d63118d5ac9fcebf9f2773fea6a964aa50b348e Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 30 Jun 2026 22:07:27 +0000 Subject: [PATCH 2/6] =?UTF-8?q?Persist=20no=20Lodged=20Performance=20for?= =?UTF-8?q?=20a=20predicted=20Property=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- .../test_property_baseline_orchestrator.py | 76 ++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-) diff --git a/tests/orchestration/test_property_baseline_orchestrator.py b/tests/orchestration/test_property_baseline_orchestrator.py index 76ee42ce1..a848ddd22 100644 --- a/tests/orchestration/test_property_baseline_orchestrator.py +++ b/tests/orchestration/test_property_baseline_orchestrator.py @@ -1,6 +1,6 @@ from __future__ import annotations -from typing import TYPE_CHECKING +from typing import Optional, TYPE_CHECKING import pytest @@ -244,6 +244,80 @@ def test_run_raises_when_no_rhi_and_no_scored_result() -> None: assert uow.commits == 0 +class _PredictedRebaseliner(Rebaseliner): + """Scores a predicted picture into a fixed Effective Performance, ignoring the + (absent) lodged half — a predicted Property always has ``physical_state_changed``, + so the calculator output IS the Effective Performance (ADR-0039).""" + + def __init__(self, effective: Performance, result: SapResult) -> None: + self._effective = effective + self._result = result + + def rebaseline( + self, + property_id: int, + effective_epc: EpcPropertyData, + lodged: Optional[Performance], + *, + physical_state_changed: bool = False, + ) -> RebaselineResult: + return RebaselineResult( + effective=self._effective, + reason="physical_state_changed", + sap_result=self._result, + ) + + +def _predicted_property() -> Property: + """A predicted Property (ADR-0031): no lodged EPC, only a neighbour-synthesised + predicted EPC whose recorded performance fields are a *different dwelling's* — + the phantom Lodged Performance this fix must not propagate. Mirrors the #1361 + exemplar (uprn 34003067 / property 742072), predicted-only.""" + predicted = object.__new__(EpcPropertyData) + predicted.energy_rating_current = 14 # phantom: copied from a neighbour template + predicted.current_energy_efficiency_band = Epc.G + predicted.co2_emissions_current = 5.0 + predicted.energy_consumption_current = 400 + predicted.sap_version = 10.2 + predicted.renewable_heat_incentive = None + return Property( + identity=PropertyIdentity( + portfolio_id=1, postcode="A0 0AA", address="1 Some Street", uprn=34003067 + ), + epc=None, + predicted_epc=predicted, + ) + + +def test_run_persists_no_lodged_performance_for_a_predicted_property() -> None: + # Arrange — a predicted Property (no lodged cert); the rebaseliner scores its + # predicted picture into a known Effective Performance. + property_baseline_repo = FakePropertyBaselineRepo() + uow = FakeUnitOfWork( + property=FakePropertyRepo({742072: _predicted_property()}), + property_baseline=property_baseline_repo, + ) + effective = Performance( + sap_score=57, epc_band=Epc.D, co2_emissions=1.5, primary_energy_intensity=160 + ) + orchestrator = PropertyBaselineOrchestrator( + unit_of_work=lambda: uow, + rebaseliner=_PredictedRebaseliner( + effective, _sap_result_with_heating(space_kwh=4200.0, water_kwh=1600.0) + ), + fuel_rates=FuelRatesStaticFileRepository(), + ) + + # Act + orchestrator.run([742072]) + + # Assert — no lodged comparator was manufactured from the neighbour-synthesised + # EPC's recorded fields; the Effective half is still persisted. + (baseline, _) = property_baseline_repo.saved[0] + assert baseline.lodged is None + assert baseline.effective == effective + + def test_run_derives_and_persists_a_bill_when_the_rebaseliner_scores() -> None: # Arrange — a rebaseliner that hands back a SapResult with lighting energy, # so the orchestrator prices it into a Bill at the committed snapshot. From da805bce21b5c6cd8f47bde45ef5b38a74b15138 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 30 Jun 2026 22:10:57 +0000 Subject: [PATCH 3/6] =?UTF-8?q?Persist=20no=20Lodged=20Performance=20for?= =?UTF-8?q?=20a=20predicted=20Property=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The PropertyBaselineOrchestrator now reads no Lodged Performance off a predicted Property's neighbour-synthesised EPC: lodged is None when source_path is 'predicted', so no phantom lodged figure is manufactured. The Effective half is unchanged. Rebaseliner port + PropertyBaselinePerformance.lodged widen to Optional; the pristine-cert paths assert non-None (lodged is None only for a predicted Property, which is always physical_state_changed). Co-Authored-By: Claude Opus 4.8 (1M context) --- domain/property_baseline/calculator_rebaseliner.py | 7 ++++++- .../property_baseline_performance.py | 7 +++++-- domain/property_baseline/rebaseliner.py | 14 +++++++++++--- orchestration/property_baseline_orchestrator.py | 12 ++++++++++-- .../test_property_baseline_orchestrator.py | 3 ++- 5 files changed, 34 insertions(+), 9 deletions(-) diff --git a/domain/property_baseline/calculator_rebaseliner.py b/domain/property_baseline/calculator_rebaseliner.py index 04558ab75..0b9414fdf 100644 --- a/domain/property_baseline/calculator_rebaseliner.py +++ b/domain/property_baseline/calculator_rebaseliner.py @@ -57,7 +57,7 @@ class CalculatorRebaseliner(Rebaseliner): self, property_id: int, effective_epc: "EpcPropertyData", - lodged: Performance, + lodged: Optional[Performance], *, physical_state_changed: bool = False, ) -> RebaselineResult: @@ -87,6 +87,11 @@ class CalculatorRebaseliner(Rebaseliner): reason=reason, sap_result=result, ) + # The pristine-lodged path: not pre-SAP10 and not physical-state-changed, so + # there is a real lodged cert to validate against. ``lodged is None`` only + # happens for a predicted Property, which is physical_state_changed and so + # took the branch above — it never reaches here. + assert lodged is not None self._log_divergence( property_id=property_id, sap_version=sap_version, result=result, lodged=lodged ) diff --git a/domain/property_baseline/property_baseline_performance.py b/domain/property_baseline/property_baseline_performance.py index 6fee98582..a65289bca 100644 --- a/domain/property_baseline/property_baseline_performance.py +++ b/domain/property_baseline/property_baseline_performance.py @@ -15,7 +15,10 @@ class PropertyBaselinePerformance: Holds both halves — ``lodged`` (what the gov register says) and ``effective`` (what the modelling pipeline scored against) — plus the ``rebaseline_reason`` recording *why* they differ (``"none"`` when equal). - Both halves are always populated, even when equal. + Both halves are populated for a Property with a real record of itself; the + ``lodged`` half is ``None`` for a predicted Property, which has no lodged cert + to read a Lodged Performance off — only the Effective half is established then + (ADR-0004 amendment, #1361). Carries the part of the energy block that needs no derivation: annual ``space_heating_kwh`` / ``water_heating_kwh`` read off the EPC's RHI. @@ -25,7 +28,7 @@ class PropertyBaselinePerformance: ran (the stub path produced no ``SapResult`` to price). """ - lodged: Performance + lodged: Optional[Performance] effective: Performance rebaseline_reason: RebaselineReason space_heating_kwh: float diff --git a/domain/property_baseline/rebaseliner.py b/domain/property_baseline/rebaseliner.py index 9171b4bf6..daef0045c 100644 --- a/domain/property_baseline/rebaseliner.py +++ b/domain/property_baseline/rebaseliner.py @@ -62,11 +62,15 @@ class Rebaseliner(ABC): self, property_id: int, effective_epc: EpcPropertyData, - lodged: Performance, + lodged: Optional[Performance], *, physical_state_changed: bool = False, ) -> RebaselineResult: - """Produce Effective Performance. ``physical_state_changed`` is True when + """Produce Effective Performance. ``lodged`` is ``None`` for a predicted + Property — it has no lodged cert, so there is no Lodged Performance to + compare against (which is also why ``physical_state_changed`` is always + True there: the calculator output IS the Effective Performance). + ``physical_state_changed`` is True when the Effective EPC was assembled from something other than a pristine lodged cert — Landlord Overrides, Site Notes, or EPC Prediction moved the physical picture (Rebaselining trigger (b)/(c)) — so the accredited lodged @@ -91,7 +95,7 @@ class StubRebaseliner(Rebaseliner): self, property_id: int, effective_epc: EpcPropertyData, - lodged: Performance, + lodged: Optional[Performance], *, physical_state_changed: bool = False, ) -> RebaselineResult: @@ -108,4 +112,8 @@ class StubRebaseliner(Rebaseliner): "Property needs rebaselining (physical state changed by overrides " "/ prediction); this stub does not run the calculator" ) + # Only a pristine SAP10 cert reaches here, and that always has a Lodged + # Performance — ``lodged is None`` implies a predicted Property, which is + # ``physical_state_changed`` and so raised above. + assert lodged is not None return RebaselineResult(effective=lodged, reason="none", sap_result=None) diff --git a/orchestration/property_baseline_orchestrator.py b/orchestration/property_baseline_orchestrator.py index bc77605ff..0ee523479 100644 --- a/orchestration/property_baseline_orchestrator.py +++ b/orchestration/property_baseline_orchestrator.py @@ -8,7 +8,7 @@ from domain.billing.bill import EnergyBreakdown from domain.sap10_calculator.calculator import SapResult from domain.billing.bill_derivation import BillDerivation from domain.property_baseline.property_baseline_performance import PropertyBaselinePerformance -from domain.property_baseline.performance import lodged_performance +from domain.property_baseline.performance import Performance, lodged_performance from domain.property_baseline.rebaseliner import Rebaseliner from repositories.fuel_rates.fuel_rates_repository import FuelRatesRepository from repositories.unit_of_work import UnitOfWork @@ -50,7 +50,15 @@ class PropertyBaselineOrchestrator: properties = uow.property.get_many(property_ids) for property_id, prop in zip(property_ids, properties, strict=True): effective_epc = prop.effective_epc - lodged = lodged_performance(effective_epc) + # A predicted Property has no lodged cert — its Effective EPC is a + # neighbour-synthesised picture, so its recorded performance fields + # are a different dwelling's. There is no Lodged Performance to read + # (ADR-0004 amendment, #1361); only the Effective half is established. + lodged: Optional[Performance] = ( + None + if prop.source_path == "predicted" + else lodged_performance(effective_epc) + ) rebaselined = self._rebaseliner.rebaseline( property_id, effective_epc, diff --git a/tests/orchestration/test_property_baseline_orchestrator.py b/tests/orchestration/test_property_baseline_orchestrator.py index a848ddd22..912dc9085 100644 --- a/tests/orchestration/test_property_baseline_orchestrator.py +++ b/tests/orchestration/test_property_baseline_orchestrator.py @@ -161,10 +161,11 @@ class _ScoringRebaseliner(Rebaseliner): self, property_id: int, effective_epc: EpcPropertyData, - lodged: Performance, + lodged: Optional[Performance], *, physical_state_changed: bool = False, ) -> RebaselineResult: + assert lodged is not None # this stub is only used on the real-cert path return RebaselineResult( effective=lodged, reason="none", sap_result=self._result ) From b9bec18f4486caba1e9ffe59c6ee361e54bd62fd Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 30 Jun 2026 22:13:04 +0000 Subject: [PATCH 4/6] =?UTF-8?q?Round-trip=20a=20null=20Lodged=20Performanc?= =?UTF-8?q?e=20through=20persistence=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four lodged_* columns are nullable as a unit; from_domain writes them all NULL when a predicted Property has no Lodged Performance, and to_domain reconstructs lodged=None (lodged_sap_score is the read discriminator, mirroring the bill block). The production lodged_* columns are FE-owned (Drizzle) and need a companion ALTER ... DROP NOT NULL migration before the backend writes NULL. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../property_baseline_performance_table.py | 51 +++++++++++++------ ...t_property_baseline_postgres_repository.py | 32 ++++++++++++ 2 files changed, 68 insertions(+), 15 deletions(-) diff --git a/infrastructure/postgres/property_baseline_performance_table.py b/infrastructure/postgres/property_baseline_performance_table.py index 89019478b..ecbb3722e 100644 --- a/infrastructure/postgres/property_baseline_performance_table.py +++ b/infrastructure/postgres/property_baseline_performance_table.py @@ -48,12 +48,16 @@ class PropertyBaselinePerformanceModel(SQLModel, table=True): # Postgres won't coerce to the enum. Bind the native enums explicitly # (``create_type=False``: the types already exist). Values stay plain # strings on the domain side (``Epc(...)`` / the RebaselineReason Literal). - lodged_sap_score: int - lodged_epc_band: str = Field( - sa_column=Column(SAEnum(*_EPC_BANDS, name="epc", create_type=False), nullable=False) + # The four lodged_* columns are nullable as a unit: a predicted Property has no + # lodged cert, so it has no Lodged Performance (ADR-0004 amendment, #1361). They + # are written all-set or all-NULL; lodged_sap_score is the read discriminator. + lodged_sap_score: Optional[int] = None + lodged_epc_band: Optional[str] = Field( + default=None, + sa_column=Column(SAEnum(*_EPC_BANDS, name="epc", create_type=False), nullable=True), ) - lodged_co2_emissions_t_per_yr: float - lodged_primary_energy_intensity_kwh_per_m2_yr: int + lodged_co2_emissions_t_per_yr: Optional[float] = None + lodged_primary_energy_intensity_kwh_per_m2_yr: Optional[int] = None effective_sap_score: int effective_epc_band: str = Field( @@ -102,12 +106,17 @@ class PropertyBaselinePerformanceModel(SQLModel, table=True): def from_domain( cls, baseline: PropertyBaselinePerformance, property_id: int ) -> "PropertyBaselinePerformanceModel": + # A predicted Property has no Lodged Performance — the four lodged_* columns + # are all NULL then (ADR-0004 amendment, #1361). + lodged = baseline.lodged model = cls( property_id=property_id, - lodged_sap_score=baseline.lodged.sap_score, - lodged_epc_band=baseline.lodged.epc_band.value, - lodged_co2_emissions_t_per_yr=baseline.lodged.co2_emissions, - lodged_primary_energy_intensity_kwh_per_m2_yr=baseline.lodged.primary_energy_intensity, + lodged_sap_score=lodged.sap_score if lodged is not None else None, + lodged_epc_band=lodged.epc_band.value if lodged is not None else None, + lodged_co2_emissions_t_per_yr=lodged.co2_emissions if lodged is not None else None, + lodged_primary_energy_intensity_kwh_per_m2_yr=( + lodged.primary_energy_intensity if lodged is not None else None + ), effective_sap_score=baseline.effective.sap_score, effective_epc_band=baseline.effective.epc_band.value, effective_co2_emissions_t_per_yr=baseline.effective.co2_emissions, @@ -139,12 +148,7 @@ class PropertyBaselinePerformanceModel(SQLModel, table=True): def to_domain(self) -> PropertyBaselinePerformance: return PropertyBaselinePerformance( - lodged=Performance( - sap_score=self.lodged_sap_score, - epc_band=Epc(self.lodged_epc_band), - co2_emissions=self.lodged_co2_emissions_t_per_yr, - primary_energy_intensity=self.lodged_primary_energy_intensity_kwh_per_m2_yr, - ), + lodged=self._read_lodged(), effective=Performance( sap_score=self.effective_sap_score, epc_band=Epc(self.effective_epc_band), @@ -157,6 +161,23 @@ class PropertyBaselinePerformanceModel(SQLModel, table=True): bill=self._read_bill(), ) + def _read_lodged(self) -> Optional[Performance]: + """The Lodged half, or None for a predicted Property whose four lodged_* + columns are NULL (ADR-0004 amendment, #1361). They are written as a unit + (`from_domain`), so lodged_sap_score is the discriminator and the other + three are non-null alongside it.""" + if self.lodged_sap_score is None: + return None + assert self.lodged_epc_band is not None + assert self.lodged_co2_emissions_t_per_yr is not None + assert self.lodged_primary_energy_intensity_kwh_per_m2_yr is not None + return Performance( + sap_score=self.lodged_sap_score, + epc_band=Epc(self.lodged_epc_band), + co2_emissions=self.lodged_co2_emissions_t_per_yr, + primary_energy_intensity=self.lodged_primary_energy_intensity_kwh_per_m2_yr, + ) + def _read_bill(self) -> Optional[Bill]: """Reconstruct the Bill from the ``bill_*`` columns. The total is the not-None discriminator: a persisted bill always sets it, so its absence diff --git a/tests/repositories/property_baseline/test_property_baseline_postgres_repository.py b/tests/repositories/property_baseline/test_property_baseline_postgres_repository.py index de6de8f4d..7edb0a151 100644 --- a/tests/repositories/property_baseline/test_property_baseline_postgres_repository.py +++ b/tests/repositories/property_baseline/test_property_baseline_postgres_repository.py @@ -83,6 +83,38 @@ def test_resaving_baseline_for_a_property_replaces_rather_than_duplicating( assert loaded == rerun +def _predicted_baseline() -> PropertyBaselinePerformance: + """A predicted Property's baseline: no Lodged Performance (no cert to read one + off), only the Effective half established (ADR-0004 amendment, #1361).""" + return PropertyBaselinePerformance( + lodged=None, + effective=Performance( + sap_score=57, epc_band=Epc.D, co2_emissions=1.5, primary_energy_intensity=160 + ), + rebaseline_reason="physical_state_changed", + space_heating_kwh=4200.0, + water_heating_kwh=1600.0, + ) + + +def test_predicted_baseline_round_trips_with_no_lodged_half(db_engine: Engine) -> None: + # Arrange — a predicted Property has no Lodged Performance; the four lodged_* + # columns persist as NULL and reconstruct as a None lodged half. + baseline = _predicted_baseline() + with Session(db_engine) as session: + PropertyBaselinePostgresRepository(session).save(baseline, property_id=13) + session.commit() + + # Act + with Session(db_engine) as session: + loaded = PropertyBaselinePostgresRepository(session).get_for_property(13) + + # Assert — the lodged half round-trips as None; the Effective half is intact. + assert loaded == baseline + assert loaded is not None + assert loaded.lodged is None + + def test_get_for_property_returns_none_when_absent(db_engine: Engine) -> None: # Arrange / Act with Session(db_engine) as session: From 7e06aa63c62591792c12a12b64d6caaae7df3705 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 30 Jun 2026 22:17:57 +0000 Subject: [PATCH 5/6] =?UTF-8?q?Backfill=20the=20phantom=20Lodged=20Perform?= =?UTF-8?q?ance=20on=20predicted=20Properties=20to=20NULL=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One-time script (dry-run default, --apply in a transaction, idempotent) that NULLs the four lodged_* columns on every predicted-source baseline row — a predicted EPC exists and no lodged one does — leaving Effective, the bill block, and rebaseline_reason intact. Dry-run against the audited DB reports 12,236 rows. Must run after the FE-owned Drizzle ALTER ... DROP NOT NULL lands. Co-Authored-By: Claude Opus 4.8 (1M context) --- scripts/null_predicted_lodged_performance.py | 111 ++++++++++++++++ .../test_null_predicted_lodged_performance.py | 120 ++++++++++++++++++ 2 files changed, 231 insertions(+) create mode 100644 scripts/null_predicted_lodged_performance.py create mode 100644 tests/scripts/test_null_predicted_lodged_performance.py diff --git a/scripts/null_predicted_lodged_performance.py b/scripts/null_predicted_lodged_performance.py new file mode 100644 index 000000000..d25104132 --- /dev/null +++ b/scripts/null_predicted_lodged_performance.py @@ -0,0 +1,111 @@ +"""One-time backfill: NULL the phantom Lodged Performance on predicted Properties. + +#1361 Class B: ``PropertyBaselineOrchestrator`` used to read Lodged Performance +off a predicted Property's neighbour-synthesised EPC, persisting a phantom set of +``lodged_*`` figures on ``property_baseline_performance`` — a *different +dwelling's* SAP / band / carbon / Primary Energy Intensity presented as this +Property's government-register record. The orchestrator no longer does this +(``lodged`` is ``None`` when ``source_path == "predicted"``, ADR-0004 amendment), +but the ~12k rows written before the fix still carry the phantom. + +This NULLs the four ``lodged_*`` columns on every predicted-source baseline row, +leaving the **Effective** half, the **bill** block, and ``rebaseline_reason`` +untouched (a predicted Property's Effective Performance is correct — it is a +first-class modelled output). + +A predicted-source Property is identified exactly as the orchestrator's +``source_path == "predicted"``: it has a **predicted** EPC and **no lodged** EPC. +Site Notes keep their as-surveyed Lodged Performance and so are excluded — though +no Site-Notes-sourced EPC exists in ``epc_property`` today, the predicate (a +predicted EPC, no lodged EPC) holds regardless. + +DRY-RUN BY DEFAULT: prints the count it would change and writes nothing. Pass +``--apply`` to execute inside a transaction. **Idempotent** — only rows whose +``lodged_sap_score`` is still non-NULL are touched, so a second run is a no-op. + +Requires the FE-owned Drizzle ``ALTER ... DROP NOT NULL`` on the four ``lodged_*`` +columns to have landed first; without it the UPDATE to NULL violates the +constraint. +""" + +from __future__ import annotations + +import argparse + +from sqlalchemy import Connection, text + +from scripts.e2e_common import build_engine, load_env + +# A predicted-source baseline row: a predicted EPC exists for the property and no +# lodged one does (``source_path == "predicted"``). ``lodged_sap_score IS NOT +# NULL`` makes it idempotent — a row already nulled is skipped on a re-run. +_PREDICTED_PHANTOM_PREDICATE = """ + pbp.lodged_sap_score IS NOT NULL + AND EXISTS ( + SELECT 1 FROM epc_property e + WHERE e.property_id = pbp.property_id AND e.source = 'predicted' + ) + AND NOT EXISTS ( + SELECT 1 FROM epc_property e + WHERE e.property_id = pbp.property_id AND e.source = 'lodged' + ) +""" + +_COUNT = text( + f""" + SELECT count(*) FROM property_baseline_performance pbp + WHERE {_PREDICTED_PHANTOM_PREDICATE} + """ +) + +_NULL_LODGED = text( + f""" + UPDATE property_baseline_performance AS pbp + SET lodged_sap_score = NULL, + lodged_epc_band = NULL, + lodged_co2_emissions_t_per_yr = NULL, + lodged_primary_energy_intensity_kwh_per_m2_yr = NULL + WHERE {_PREDICTED_PHANTOM_PREDICATE} + """ +) + + +def backfill(conn: Connection, *, apply: bool) -> int: + """NULL the four ``lodged_*`` columns on predicted-source baseline rows. + + Returns the number of phantom rows found (those that ``--apply`` would / did + null). Reads the count first so the dry-run reports it without writing. + """ + found = conn.execute(_COUNT).scalar() or 0 + if apply: + conn.execute(_NULL_LODGED) + return found + + +def main() -> None: + load_env() + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--apply", + action="store_true", + help="execute the update (default: dry-run, writes nothing)", + ) + args = parser.parse_args() + + engine = build_engine() + with engine.begin() as conn: + conn.execute(text("SET statement_timeout = 120000")) + found = backfill(conn, apply=args.apply) + + verb = "NULLed" if args.apply else "would NULL" + print( + f"{verb} the Lodged Performance (lodged_* → NULL) on {found} " + "predicted-source baseline row(s); Effective / bill / rebaseline_reason " + "left intact." + ) + if not args.apply: + print("\nDRY-RUN — nothing written. Re-run with --apply to execute.") + + +if __name__ == "__main__": + main() diff --git a/tests/scripts/test_null_predicted_lodged_performance.py b/tests/scripts/test_null_predicted_lodged_performance.py new file mode 100644 index 000000000..0487359a7 --- /dev/null +++ b/tests/scripts/test_null_predicted_lodged_performance.py @@ -0,0 +1,120 @@ +"""The one-time backfill nulls a predicted Property's phantom Lodged Performance, +leaves a lodged Property's intact, and is idempotent (#1361 Class B).""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +from sqlalchemy import Engine +from sqlmodel import Session + +from datatypes.epc.domain.epc import Epc +from datatypes.epc.domain.epc_property_data import EpcPropertyData +from datatypes.epc.domain.mapper import EpcPropertyDataMapper +from domain.property_baseline.performance import Performance +from domain.property_baseline.property_baseline_performance import ( + PropertyBaselinePerformance, +) +from repositories.epc.epc_postgres_repository import EpcPostgresRepository +from repositories.property_baseline.property_baseline_postgres_repository import ( + PropertyBaselinePostgresRepository, +) +from scripts.null_predicted_lodged_performance import backfill + +_JSON_SAMPLES = Path(__file__).resolve().parents[2] / "backend/epc_api/json_samples" + +_PHANTOM_LODGED = Performance( + sap_score=14, epc_band=Epc.G, co2_emissions=5.0, primary_energy_intensity=400 +) +_EFFECTIVE = Performance( + sap_score=57, epc_band=Epc.D, co2_emissions=1.5, primary_energy_intensity=160 +) + + +def _epc() -> EpcPropertyData: + raw: dict[str, Any] = json.loads( + (_JSON_SAMPLES / "RdSAP-Schema-21.0.0" / "epc.json").read_text() + ) + return EpcPropertyDataMapper.from_api_response(raw) + + +def _baseline(lodged: Performance) -> PropertyBaselinePerformance: + return PropertyBaselinePerformance( + lodged=lodged, + effective=_EFFECTIVE, + rebaseline_reason="physical_state_changed", + space_heating_kwh=4200.0, + water_heating_kwh=1600.0, + ) + + +def _seed(db_engine: Engine) -> None: + """A predicted Property (id 1, predicted EPC only, phantom lodged) and a + lodged Property (id 2, lodged EPC, real lodged) — both with a baseline row + carrying a populated lodged half (the pre-fix state).""" + epc = _epc() + with Session(db_engine) as session: + epc_repo = EpcPostgresRepository(session) + epc_repo.save(epc, property_id=1, source="predicted") + epc_repo.save(epc, property_id=2, source="lodged") + baseline_repo = PropertyBaselinePostgresRepository(session) + baseline_repo.save(_baseline(_PHANTOM_LODGED), property_id=1) + baseline_repo.save(_baseline(_PHANTOM_LODGED), property_id=2) + session.commit() + + +def test_apply_nulls_only_the_predicted_propertys_lodged_half( + db_engine: Engine, +) -> None: + # Arrange + _seed(db_engine) + + # Act + with db_engine.begin() as conn: + found = backfill(conn, apply=True) + + # Assert — one phantom found and nulled; the predicted Property loses its + # lodged half but keeps its Effective half, while the lodged Property is + # untouched. + assert found == 1 + with Session(db_engine) as session: + repo = PropertyBaselinePostgresRepository(session) + predicted = repo.get_for_property(1) + lodged = repo.get_for_property(2) + assert predicted is not None + assert predicted.lodged is None + assert predicted.effective == _EFFECTIVE + assert lodged is not None + assert lodged.lodged == _PHANTOM_LODGED + + +def test_dry_run_reports_the_phantom_but_writes_nothing(db_engine: Engine) -> None: + # Arrange + _seed(db_engine) + + # Act + with db_engine.begin() as conn: + found = backfill(conn, apply=False) + + # Assert — the phantom is counted, but the row is left untouched. + assert found == 1 + with Session(db_engine) as session: + predicted = PropertyBaselinePostgresRepository(session).get_for_property(1) + assert predicted is not None + assert predicted.lodged == _PHANTOM_LODGED + + +def test_re_running_apply_is_a_no_op(db_engine: Engine) -> None: + # Arrange — the first apply nulls the phantom. + _seed(db_engine) + with db_engine.begin() as conn: + backfill(conn, apply=True) + + # Act — a second apply finds nothing left to null. + with db_engine.begin() as conn: + found = backfill(conn, apply=True) + + # Assert + assert found == 0 From 09b3bc98ba8a4e1f304320ef5665264b6395dae1 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 30 Jun 2026 22:19:54 +0000 Subject: [PATCH 6/6] Document the nullable lodged_* migration for the FE/Drizzle owner Specifies the ALTER ... DROP NOT NULL on the four lodged_* columns (#1361) and the ordering constraint: it must land before the backend writes NULL or runs the backfill, else the predicted-Property INSERT aborts the batch. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../property-baseline-performance-table.md | 23 +++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/docs/migrations/property-baseline-performance-table.md b/docs/migrations/property-baseline-performance-table.md index 0aebba832..efd1ec10b 100644 --- a/docs/migrations/property-baseline-performance-table.md +++ b/docs/migrations/property-baseline-performance-table.md @@ -18,10 +18,10 @@ straight lift-and-shift of the columns below. |---|---|---| | `id` | serial PK | | | `property_id` | int, FK → `property.id`, **unique** | one Baseline Performance per Property | -| `lodged_sap_score` | int | Lodged Performance — gov register, off the Effective EPC | -| `lodged_epc_band` | text | the `Epc` enum, stored as its string value (e.g. `"C"`) | -| `lodged_co2_emissions_t_per_yr` | float | tonnes CO₂/yr (whole dwelling) | -| `lodged_primary_energy_intensity_kwh_per_m2_yr` | int | PEUI (kWh/m²/yr); **not** "heat demand" — see CONTEXT.md | +| `lodged_sap_score` | int, **nullable** | Lodged Performance — gov register, off the Effective EPC. **NULL for a predicted Property** (no lodged cert — #1361, see below) | +| `lodged_epc_band` | text, **nullable** | the `Epc` enum, stored as its string value (e.g. `"C"`) | +| `lodged_co2_emissions_t_per_yr` | float, **nullable** | tonnes CO₂/yr (whole dwelling) | +| `lodged_primary_energy_intensity_kwh_per_m2_yr` | int, **nullable** | PEUI (kWh/m²/yr); **not** "heat demand" — see CONTEXT.md | | `effective_sap_score` | int | Effective Performance — what modelling scored against | | `effective_epc_band` | text | | | `effective_co2_emissions_t_per_yr` | float | tonnes CO₂/yr (whole dwelling) | @@ -30,6 +30,21 @@ straight lift-and-shift of the columns below. | `space_heating_kwh` | float | EPC `renewable_heat_incentive` recorded demand. **Superseded** by `heating_kwh` (delivered) when the bill block populates; kept until then to avoid an empty-kWh gap, dropped in the population slice. | | `water_heating_kwh` | float | EPC `renewable_heat_incentive`; **superseded** by `hot_water_kwh`. | +### Lodged half is nullable (#1361 Class B, 2026-06-30) + +The four `lodged_*` columns are **nullable**. A **predicted** Property (EPC Prediction, ADR-0029/0031: +no lodged cert, its `EpcPropertyData` synthesised from neighbours) has **no Lodged Performance** — the +backend now writes `lodged_* = NULL` for it and persists only the Effective half (ADR-0004 amendment). +Reading the synthesised EPC's recorded fields as a lodged figure had been manufacturing a phantom (a +neighbour's SAP), which the migration + a one-time backfill remove. + +**FE-owned Drizzle migration required:** `ALTER TABLE property_baseline_performance ALTER COLUMN +lodged_sap_score DROP NOT NULL` (and the same for `lodged_epc_band`, `lodged_co2_emissions_t_per_yr`, +`lodged_primary_energy_intensity_kwh_per_m2_yr`). This **must land before** the backend deploys the +orchestrator change or runs `scripts/null_predicted_lodged_performance.py` — a `NULL` write against a +`NOT NULL` column aborts the batch (ADR-0012). The backfill NULLs the four columns on the ~12,236 +existing predicted-source rows; Effective, the bill block, and `rebaseline_reason` are left intact. + ### Bill block (ADR-0014) — the energy bill, composed per section Produced by **Bill Derivation**: the calculator's **delivered** kWh per end use priced at current