Model/docs/adr/0004-baseline-performance-lodged-effective-pair.md
Khalim Conn-Kowlessar 7ecb7eebd1 Record that a predicted Property has no Lodged Performance
Amends ADR-0004's both-halves-always-populated invariant: EPC Prediction
borrows a neighbour's recorded figures, so a predicted Property has no Lodged
Performance — its lodged half is absent (NULL), only the Effective half is
persisted. Sharpens the CONTEXT.md glossary for Lodged / Baseline Performance.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 22:07:27 +00:00

96 lines
8.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# `PropertyBaselinePerformance` stores both lodged and effective values
A Property's current performance has two states we care about: the rating that was lodged on the government register (the "lodged" SAP / band / carbon / heat) and the rating produced by the modelling pipeline against the current Effective EPC (the "effective" values, which may have been rebaselined by ML when the EPC was pre-SAP10 or when Landlord Overrides / Site Notes changed physical state). We considered storing a single set of values — the rebaselined-if-needed-otherwise-lodged figures — and rejected that. Both are stored as a pair on every `PropertyBaselinePerformance`, equal when no rebaselining trigger fires.
The pair lets the FE show "this is what the gov register says vs this is the SAP10-equivalent we modelled against" side by side without a second query, and keeps the audit trail clean: a user looking at a property's plan can see exactly which figure drove the recommendation pipeline. Storing only one set forces a downstream consumer to recompute the missing one from raw EPC fields when it needs both, which is the kind of derivation creep we want to keep out of the FE.
The cost is a wider row + the discipline that **every** `PropertyBaselinePerformance` populates both halves, even when they're equal. Annual kWh, fuel split and bills are not paired — they are always derived deterministically by `EpcEnergyDerivationService` against the Effective state, because the EPC's recorded cost fields use fuel rates pinned to the inspection date and the UCL correction depends on the modelled band.
## Consequences
- Reversing this means rewriting every consumer that has learned to read both values. Hard to roll back once the FE depends on the pair.
- The rebaseline trigger has two reasons (`pre_sap10`, `physical_state_changed`, or `both`) — store the reason alongside so we know *why* a property was rebaselined when debugging.
### Amendment (2026-05-30, #1135): standalone `property_baseline_performance` table
The original consequence read *"`property_details_epc` (or its successor) carries 8 fields
instead of 4 for the SAP-equivalent block"* — i.e. the pair as columns on the EPC-details table.
That is superseded. `property_details_epc` is being **retired**: it is too tightly coupled to the
schema of the legacy EPC API, which the Ara rebuild is moving off. So the pair has no home there.
`PropertyBaselinePerformance` instead persists as its **own standalone `property_baseline_performance` table, one
row per Property**, behind a dedicated `PropertyBaselineRepository` port (`save` / `get_for_property`),
mirroring the EPC slice's repo shape. This is the cleaner model regardless of the retirement:
`PropertyBaselinePerformance` is its own aggregate (a Property's current performance), not a detail of any
single EPC.
The row is **flat typed columns**, not a JSONB blob, because the FE both surfaces the block and
queries the lodged-vs-effective pair: `lodged_{sap_score, epc_band, co2_emissions,
primary_energy_intensity}`, the four `effective_*` mirrors, `rebaseline_reason`, and (for the part
of the energy block that needs no derivation) `space_heating_kwh` / `water_heating_kwh`. The
fourth paired quantity is **Primary Energy Intensity**, not "heat demand" — see CONTEXT.md
(the prose above predates that term being sharpened).
Fuel split and bills — the rest of the EPC Energy Derivation block — are **deferred to a
follow-up**: bills require a current Fuel Rates source (Ofgem-cap ETL) that does not yet exist, and
fuel split is produced by the same `EpcEnergyDerivationService`, so the two land together rather
than churning the table twice.
The SQLModel row is defined in `infrastructure/postgres/` so the ephemeral-Postgres tests build it
via `create_all`; the production migration is FE-owned (Drizzle ORM) and tracked in
`docs/migrations/`.
### Amendment (2026-06-30, #1361 Class B): the Lodged half is absent for a predicted Property
The original invariant — **every** `PropertyBaselinePerformance` populates **both** halves, even when
equal — assumed every Property has a record *of its own* performance to lodge: a public EPC cert (the
`epc_with_overlay` path) or a Site Notes survey. It does **not** hold for the **EPC Prediction** path
(ADR-0029/0031). A predicted Property has no record of itself — its `EpcPropertyData` is deterministic
neighbour synthesis that copies a representative comparable's structure wholesale, so the
`energy_rating_current` / band / CO2 / Primary Energy Intensity on the synthesised picture are a
*different dwelling's* lodged figures. Reading **Lodged Performance** off it manufactures a **phantom**:
a borrowed neighbour's SAP presented as this Property's government-register figure. This affected
**every** predicted Property (~12,236 rows estate-wide), not the 8 the `effective-lodged-divergence`
audit surfaced — that audit only fires when the borrowed figure lands ≥15 SAP from the Effective, so it
under-counts the phantom by three orders of magnitude.
**Decision.** The Lodged half is **optional**. When `source_path == "predicted"` there is no Lodged
Performance: `PropertyBaselinePerformance.lodged` is `None` and the four `lodged_*` columns are `NULL`.
The **Effective half is unchanged and still persisted** — a predicted Property is a first-class modelled
output (it flows through Rebaselining, Bill Derivation, and Modelling like any other; its Effective
Performance and bill block are correct and load-bearing for the FE and the plan-vs-effective audit
checks). `rebaseline_reason` stays `physical_state_changed` / `both`, which already records that the
Effective figure was scored from a changed picture.
**Principle (the boundary).** Lodged Performance requires a record **of this Property** — a lodged cert
*or* a Site Notes survey (the as-surveyed values are a real observation of this dwelling). EPC
Prediction borrows a neighbour's, so it has none. The branch is therefore `source_path == "predicted"`
**not** `physical_state_changed` (true for Site Notes and Landlord Overrides too) and **not** "no public
EPC" (Site Notes has none yet keeps a legitimate Lodged Performance). Whether the Site Notes as-surveyed
values are truly "lodged" is a separate question, deferred.
**Rejected — a sentinel (`lodged = 0`).** Considered, to make "no record" visibly present rather than an
empty cell. Rejected: `0` is a valid-looking SAP score that (a) re-trips `effective-lodged-divergence`
for every predicted Property (`|effective 0| ≥ 15`), (b) poisons every `AVG(lodged_*)` aggregate that
`NULL` is correctly excluded from, and (c) has no coherent `lodged_epc_band` enum member. The "no lodged
record" signal belongs in `NULL` (honest absence) plus the **structural provenance** already carried by
the distinct predicted-EPC slot — which the FE renders as a predicted badge — not overloaded onto the
score column.
**Consequences.**
- The four `lodged_*` columns become **nullable**. The production table is **FE-owned (Drizzle)**, so the
migration (`ALTER … DROP NOT NULL`) lands in the FE repo and **must precede** any backend write of a
`NULL` lodged, or the predicted-Property INSERT violates the constraint and aborts the batch
(ADR-0012). The SQLModel mirror in `infrastructure/postgres/` is updated to `Optional` so the
ephemeral-Postgres tests build the nullable shape.
- The fix lives in `PropertyBaselineOrchestrator.run()` — the single chokepoint both the First Run
pipeline and `applications/modelling_e2e/handler.py` call — so one change repairs both entry points.
The Rebaseliner port takes `Optional[Performance]`; for a predicted Property `physical_state_changed`
is always true, so `CalculatorRebaseliner` adopts the calculator output and never reads the absent
lodged half (the divergence-log path is the pristine-cert case only, where lodged is non-null).
- A one-time backfill (`scripts/`, dry-run default + `--apply`, idempotent) NULLs the four `lodged_*`
columns on existing predicted-source rows (~12,236). It corrects only the Lodged half; Effective, the
bill block, and `rebaseline_reason` are left intact.
- `effective-lodged-divergence` already short-circuits on `lodged_sap IS NULL`, so it goes green for
predicted Properties once backfilled. Class C of #1361 (a floor that ignores implausibly-low lodged
scores on *real* certs) is an additive guard, independent of this change.