diff --git a/CONTEXT.md b/CONTEXT.md index fe6493ee4..70e9be18e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -116,11 +116,11 @@ Deterministically translating an **old / reduced-data EPC schema** into the curr _Avoid_: gap-fill (means the neighbour-ML path), reduced-data expansion (overloaded with the calculator's Table-5 step), remapping (the schema-translation part only) **Baseline Performance**: -A Property's current performance aggregate, holding both Lodged Performance and Effective Performance plus the energy block: delivered kWh **per end use** (heating, hot water, lighting, appliances, cooking, pumps/fans, cooling) and the **annual bill** composed into per-section costs plus a total, produced by **Bill Derivation** from SAP10 Calculation's per-end-use kWh × current Fuel Rates. Persisted as one row (flat typed columns, per-section kWh + cost + total); surfaced as one block in the UI. +A Property's current performance aggregate, holding both Lodged Performance and Effective Performance plus the energy block: delivered kWh **per end use** (heating, hot water, lighting, appliances, cooking, pumps/fans, cooling) and the **annual bill** composed into per-section costs plus a total, produced by **Bill Derivation** from SAP10 Calculation's per-end-use kWh × current Fuel Rates. Persisted as one row (flat typed columns, per-section kWh + cost + total); surfaced as one block in the UI. The **Lodged half is optional**: a predicted Property has no lodged record (see Lodged Performance), so its `lodged_*` are `NULL` and only the Effective half + energy block are populated (ADR-0004 amendment, #1361). _Avoid_: baseline predictions, predicted baseline, rebaselined values **Lodged Performance**: -The SAP / EPC Band / carbon emissions / Primary Energy Intensity recorded on the public EPC (or the Site Notes' as-surveyed values when Site Notes are the source) — unmodified by modelling. The half of Baseline Performance that says "what the government register says about this Property". +The SAP / EPC Band / carbon emissions / Primary Energy Intensity recorded on the public EPC (or the Site Notes' as-surveyed values when Site Notes are the source) — unmodified by modelling. The half of Baseline Performance that says "what the government register says about this Property". **Requires a record _of this Property_** — a lodged cert or a Site Notes survey — so it is **absent (`None`/NULL) for a predicted Property**: **EPC Prediction** synthesises the picture from neighbours, copying a comparable's recorded figures, so there is no government-register record of _this_ Property to lodge. Only the Effective half is persisted then (ADR-0004 amendment, #1361). Reading a predicted EPC's recorded fields as Lodged Performance manufactures a phantom — a neighbour's SAP presented as this Property's lodged figure. _Avoid_: original performance, raw EPC values, recorded baseline **Effective Performance**: diff --git a/docs/adr/0004-baseline-performance-lodged-effective-pair.md b/docs/adr/0004-baseline-performance-lodged-effective-pair.md index fc27be7da..a8b4ba476 100644 --- a/docs/adr/0004-baseline-performance-lodged-effective-pair.md +++ b/docs/adr/0004-baseline-performance-lodged-effective-pair.md @@ -39,3 +39,58 @@ than churning the table twice. The SQLModel row is defined in `infrastructure/postgres/` so the ephemeral-Postgres tests build it via `create_all`; the production migration is FE-owned (Drizzle ORM) and tracked in `docs/migrations/`. + +### Amendment (2026-06-30, #1361 Class B): the Lodged half is absent for a predicted Property + +The original invariant — **every** `PropertyBaselinePerformance` populates **both** halves, even when +equal — assumed every Property has a record *of its own* performance to lodge: a public EPC cert (the +`epc_with_overlay` path) or a Site Notes survey. It does **not** hold for the **EPC Prediction** path +(ADR-0029/0031). A predicted Property has no record of itself — its `EpcPropertyData` is deterministic +neighbour synthesis that copies a representative comparable's structure wholesale, so the +`energy_rating_current` / band / CO2 / Primary Energy Intensity on the synthesised picture are a +*different dwelling's* lodged figures. Reading **Lodged Performance** off it manufactures a **phantom**: +a borrowed neighbour's SAP presented as this Property's government-register figure. This affected +**every** predicted Property (~12,236 rows estate-wide), not the 8 the `effective-lodged-divergence` +audit surfaced — that audit only fires when the borrowed figure lands ≥15 SAP from the Effective, so it +under-counts the phantom by three orders of magnitude. + +**Decision.** The Lodged half is **optional**. When `source_path == "predicted"` there is no Lodged +Performance: `PropertyBaselinePerformance.lodged` is `None` and the four `lodged_*` columns are `NULL`. +The **Effective half is unchanged and still persisted** — a predicted Property is a first-class modelled +output (it flows through Rebaselining, Bill Derivation, and Modelling like any other; its Effective +Performance and bill block are correct and load-bearing for the FE and the plan-vs-effective audit +checks). `rebaseline_reason` stays `physical_state_changed` / `both`, which already records that the +Effective figure was scored from a changed picture. + +**Principle (the boundary).** Lodged Performance requires a record **of this Property** — a lodged cert +*or* a Site Notes survey (the as-surveyed values are a real observation of this dwelling). EPC +Prediction borrows a neighbour's, so it has none. The branch is therefore `source_path == "predicted"` — +**not** `physical_state_changed` (true for Site Notes and Landlord Overrides too) and **not** "no public +EPC" (Site Notes has none yet keeps a legitimate Lodged Performance). Whether the Site Notes as-surveyed +values are truly "lodged" is a separate question, deferred. + +**Rejected — a sentinel (`lodged = 0`).** Considered, to make "no record" visibly present rather than an +empty cell. Rejected: `0` is a valid-looking SAP score that (a) re-trips `effective-lodged-divergence` +for every predicted Property (`|effective − 0| ≥ 15`), (b) poisons every `AVG(lodged_*)` aggregate that +`NULL` is correctly excluded from, and (c) has no coherent `lodged_epc_band` enum member. The "no lodged +record" signal belongs in `NULL` (honest absence) plus the **structural provenance** already carried by +the distinct predicted-EPC slot — which the FE renders as a predicted badge — not overloaded onto the +score column. + +**Consequences.** +- The four `lodged_*` columns become **nullable**. The production table is **FE-owned (Drizzle)**, so the + migration (`ALTER … DROP NOT NULL`) lands in the FE repo and **must precede** any backend write of a + `NULL` lodged, or the predicted-Property INSERT violates the constraint and aborts the batch + (ADR-0012). The SQLModel mirror in `infrastructure/postgres/` is updated to `Optional` so the + ephemeral-Postgres tests build the nullable shape. +- The fix lives in `PropertyBaselineOrchestrator.run()` — the single chokepoint both the First Run + pipeline and `applications/modelling_e2e/handler.py` call — so one change repairs both entry points. + The Rebaseliner port takes `Optional[Performance]`; for a predicted Property `physical_state_changed` + is always true, so `CalculatorRebaseliner` adopts the calculator output and never reads the absent + lodged half (the divergence-log path is the pristine-cert case only, where lodged is non-null). +- A one-time backfill (`scripts/`, dry-run default + `--apply`, idempotent) NULLs the four `lodged_*` + columns on existing predicted-source rows (~12,236). It corrects only the Lodged half; Effective, the + bill block, and `rebaseline_reason` are left intact. +- `effective-lodged-divergence` already short-circuits on `lodged_sap IS NULL`, so it goes green for + predicted Properties once backfilled. Class C of #1361 (a floor that ignores implausibly-low lodged + scores on *real* certs) is an additive guard, independent of this change.