diff --git a/docs/baseline-downgrade-followups.md b/docs/baseline-downgrade-followups.md new file mode 100644 index 00000000..409db534 --- /dev/null +++ b/docs/baseline-downgrade-followups.md @@ -0,0 +1,56 @@ +# Follow-ups from the full-SAP baseline-downgrade work (ADR-0037) + +Open items surfaced while fixing the full-SAP mapper / portfolio-796 downgrades. +Each is **separate from** the full-SAP fix (`fix/baseline-downgrades`). + +## 1. Older RdSAP mappers drop `mechanical_ventilation_kind` (calc-facing) + +**Severity: accuracy bug for MEV/MVHR dwellings.** + +`mechanical_ventilation_kind` (the natural / MEV / MVHR ventilation **type**) is +mapped in **only `from_rdsap_schema_21_0_1`** ([mapper.py:2650], via +`_api_mechanical_ventilation_kind`). The other six API mappers — +`from_rdsap_schema_17_0/17_1/18_0/19_0/20_0_0/21_0_0` — build +`SapVentilation(sheltered_sides=…)` and **drop** the cert's `mechanical_ventilation` +field. (The full-SAP path maps it via `_sap_17_1_ventilation` — not affected.) + +- For **natural-ventilation** certs (`mechanical_ventilation = 0`, e.g. UPRN + 100020603823 / property 726605) it's **benign** — unmapped → `None` defaults to + natural in the §2 cascade. +- For **MEV/MVHR** certs (`mechanical_ventilation ≠ 0`) the calculator treats the + dwelling as **naturally ventilated**, getting the SAP §2 ventilation cascade + (and any heat recovery) wrong → mis-scored SAP. + +The granular **counts** (fans/flues/vents) are *not* a bug: older RdSAP open-data +certs don't lodge them, and the calc correctly uses RdSAP Table-5 age defaults. +`percent_draughtproofed` is mapped (top-level) and read by the calc. + +**Fix:** mirror `mechanical_ventilation_kind=_api_mechanical_ventilation_kind(schema.mechanical_ventilation)` +into the six older RdSAP mappers. **Calc-facing → validate** with the RdSAP-21.0.1 +corpus (must hold 73.3% / MAE 0.774) plus an Elmhurst-anchored MEV/MVHR cert (the +corpus is natural-vent-dominated, so add a mechanical-vent `RealCertExpectation`). +Quantify blast radius: count older-RdSAP certs with `mechanical_ventilation ≠ 0`. + +## 2. FE "Main Fuel: Unknown" is FE-side, not a Model mapper gap + +`main_fuel_type` (the gov fuel **code**) **is** populated Model-side — UPRN +10093412452 (709791) → `1`, UPRN 100020603823 (726605) → `26` — and is persisted +(`epc_main_heating_detail.main_fuel_type`). So the panel's "Main Fuel: Unknown" +is the **front-end** either not mapping the code → fuel name or reading a field we +don't populate. Needs an FE-repo (Drizzle/Next) check, not a Model change. + +## 3. Survivor clusters from the post-fix population sweep + +`scripts/hyde/case_a_population_sweep.py`: the representative sample rebaselined +cleanly (0 survivors), but the worst-old-drop sample held **28 survivors** +(lodged A/B → new C/D, 15–25 SAP) in tight UPRN clusters (new-build blocks), +spanning multiple schemas (16.0/16.1/17.1/18.0.0) and heating types. No single +mapper-gap signature → likely genuine SAP-2012→10.2 drops for very-high-lodged +new-builds, but **triage one in Elmhurst** to confirm genuine vs a residual calc +issue before trusting the cohort. + +## 4. Predicted-property display path (e.g. property 721167) + +721167 has **no lodged EPC** (predicted). Its Heating-Control / Main-Fuel / +Ventilation Unknowns come from the prediction + landlord-override **overlay** not +populating the display fields — a separate path from the lodged-cert mappers. diff --git a/scripts/hyde/case_a_population_sweep.py b/scripts/hyde/case_a_population_sweep.py new file mode 100644 index 00000000..f96c0af5 --- /dev/null +++ b/scripts/hyde/case_a_population_sweep.py @@ -0,0 +1,125 @@ +"""Local population sanity sweep for the full-SAP baseline-downgrade fix (ADR-0037). + +For a sample of Case A properties (portfolio 796, full-SAP path: epc_property +assessment_type+sap_version NULL), re-map the real cert with the NEW mapper, run +the SAP-10.2 calculator, and compute the post-fix Effective SAP/band. Compare to +the lodged figures to (a) confirm the fix flips Effective off the stale lodged +value and (b) flag survivors — drops too large to be a 2012→10.2 methodology +shift (>15 SAP or >=2 bands below lodged), candidate deeper-calc/cert bugs. + +Read-only. No DB writes. +""" +from __future__ import annotations + +import os +import random # noqa: F401 — only used with a fixed seed below + +from scripts.e2e_common import load_env, build_engine +from sqlalchemy import text + +load_env() + +from infrastructure.epc_client.epc_client_service import EpcClientService +from datatypes.epc.domain.mapper import EpcPropertyDataMapper +from domain.sap10_calculator.calculator import Sap10Calculator +from datatypes.epc.domain.epc import Epc + +_BAND_IDX = {b: i for i, b in enumerate(["G", "F", "E", "D", "C", "B", "A"])} + + +def _band_letter(epc_band: object) -> str: + s = str(epc_band) + return s.split(".")[-1] if "." in s else s + + +def main() -> None: + eng = build_engine() + with eng.connect() as c: + # 30 worst old-drops + 30 spread across the cohort (deterministic order). + worst = c.execute(text(""" + SELECT p.uprn, pbp.lodged_sap_score, pbp.lodged_epc_band, + pbp.effective_sap_score AS old_eff, pl.post_sap_points AS old_post + FROM property p + JOIN epc_property ep ON ep.uprn=p.uprn + JOIN property_baseline_performance pbp ON pbp.property_id=p.id + JOIN plan pl ON pl.property_id=p.id + WHERE p.portfolio_id=796 AND ep.assessment_type IS NULL + AND ep.sap_version IS NULL + ORDER BY (pbp.effective_sap_score - pl.post_sap_points) DESC + LIMIT 30 + """)).fetchall() + spread = c.execute(text(""" + SELECT p.uprn, pbp.lodged_sap_score, pbp.lodged_epc_band, + pbp.effective_sap_score AS old_eff, pl.post_sap_points AS old_post + FROM property p + JOIN epc_property ep ON ep.uprn=p.uprn + JOIN property_baseline_performance pbp ON pbp.property_id=p.id + JOIN plan pl ON pl.property_id=p.id + WHERE p.portfolio_id=796 AND ep.assessment_type IS NULL + AND ep.sap_version IS NULL AND (p.uprn % 7) = 0 + ORDER BY p.uprn LIMIT 30 + """)).fetchall() + + rows = {r._mapping["uprn"]: dict(r._mapping) for r in worst} + for r in spread: + rows.setdefault(r._mapping["uprn"], dict(r._mapping)) + + svc = EpcClientService(os.environ["OPEN_EPC_API_TOKEN"]) + calc = Sap10Calculator() + + n = 0 + flipped = 0 + calc_errors = 0 + survivors = [] + drops = [] + for uprn, row in rows.items(): + n += 1 + try: + results = svc._search(uprn=int(uprn)) + if not results: + print(f" uprn={uprn} NO_CERT") + continue + latest = max(results, key=lambda x: x.registration_date) + raw = svc._fetch_certificate(latest.certificate_number) + epc = EpcPropertyDataMapper.from_api_response(raw) + new_sap = calc.calculate(epc).sap_score + except Exception as e: # noqa: BLE001 — sweep tolerates per-cert failure + calc_errors += 1 + print(f" uprn={uprn} CALC_ERROR {type(e).__name__}: {str(e)[:90]}") + continue + new_band = _band_letter(Epc.from_sap_score(new_sap)) + lodged = row["lodged_sap_score"] + lodged_band = row["lodged_epc_band"] + drop = lodged - new_sap + drops.append(drop) + if epc.sap_version is not None and epc.sap_version < 10.2: + flipped += 1 + band_drop = _BAND_IDX.get(lodged_band, 0) - _BAND_IDX.get(new_band, 0) + survivor = drop > 15 or band_drop >= 2 + tag = " *** SURVIVOR" if survivor else "" + if survivor: + survivors.append((uprn, lodged, lodged_band, new_sap, new_band, drop)) + print( + f" uprn={uprn} lodged={lodged}/{lodged_band} " + f"new_eff={new_sap}/{new_band} drop={drop} " + f"(old_eff={row['old_eff']} old_post={round(float(row['old_post']),1)}){tag}" + ) + + computed = len(drops) + print("\n==== SUMMARY ====") + print(f"sampled={n} computed={computed} calc_errors={calc_errors}") + print(f"rebaseline flipped (sap_version<10.2 now set) = {flipped}/{computed}") + if drops: + drops_sorted = sorted(drops) + print( + f"lodged-new_eff drop: min={min(drops)} median=" + f"{drops_sorted[len(drops_sorted)//2]} max={max(drops)} " + f"mean={round(sum(drops)/len(drops),1)}" + ) + print(f"survivors (>15 SAP or >=2 bands below lodged) = {len(survivors)}") + for s in survivors: + print(f" SURVIVOR uprn={s[0]} lodged={s[1]}/{s[2]} new={s[3]}/{s[4]} drop={s[5]}") + + +if __name__ == "__main__": + main()