docs: full-SAP baseline-downgrade follow-ups + Case A population sweep tool

Captures the calc-facing ventilation-type gap (older RdSAP mappers drop
mechanical_ventilation_kind), the FE-side Main Fuel display, the sweep survivor
clusters, and the predicted-property display path — all separate from ADR-0037.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/baseline-downgrade-followups.md

@@ -0,0 +1,56 @@
+# Follow-ups from the full-SAP baseline-downgrade work (ADR-0037)
+
+Open items surfaced while fixing the full-SAP mapper / portfolio-796 downgrades.
+Each is **separate from** the full-SAP fix (`fix/baseline-downgrades`).
+
+## 1. Older RdSAP mappers drop `mechanical_ventilation_kind` (calc-facing)
+
+**Severity: accuracy bug for MEV/MVHR dwellings.**
+
+`mechanical_ventilation_kind` (the natural / MEV / MVHR ventilation **type**) is
+mapped in **only `from_rdsap_schema_21_0_1`** ([mapper.py:2650], via
+`_api_mechanical_ventilation_kind`). The other six API mappers —
+`from_rdsap_schema_17_0/17_1/18_0/19_0/20_0_0/21_0_0` — build
+`SapVentilation(sheltered_sides=…)` and **drop** the cert's `mechanical_ventilation`
+field. (The full-SAP path maps it via `_sap_17_1_ventilation` — not affected.)
+
+- For **natural-ventilation** certs (`mechanical_ventilation = 0`, e.g. UPRN
+  100020603823 / property 726605) it's **benign** — unmapped → `None` defaults to
+  natural in the §2 cascade.
+- For **MEV/MVHR** certs (`mechanical_ventilation ≠ 0`) the calculator treats the
+  dwelling as **naturally ventilated**, getting the SAP §2 ventilation cascade
+  (and any heat recovery) wrong → mis-scored SAP.
+
+The granular **counts** (fans/flues/vents) are *not* a bug: older RdSAP open-data
+certs don't lodge them, and the calc correctly uses RdSAP Table-5 age defaults.
+`percent_draughtproofed` is mapped (top-level) and read by the calc.
+
+**Fix:** mirror `mechanical_ventilation_kind=_api_mechanical_ventilation_kind(schema.mechanical_ventilation)`
+into the six older RdSAP mappers. **Calc-facing → validate** with the RdSAP-21.0.1
+corpus (must hold 73.3% / MAE 0.774) plus an Elmhurst-anchored MEV/MVHR cert (the
+corpus is natural-vent-dominated, so add a mechanical-vent `RealCertExpectation`).
+Quantify blast radius: count older-RdSAP certs with `mechanical_ventilation ≠ 0`.
+
+## 2. FE "Main Fuel: Unknown" is FE-side, not a Model mapper gap
+
+`main_fuel_type` (the gov fuel **code**) **is** populated Model-side — UPRN
+10093412452 (709791) → `1`, UPRN 100020603823 (726605) → `26` — and is persisted
+(`epc_main_heating_detail.main_fuel_type`). So the panel's "Main Fuel: Unknown"
+is the **front-end** either not mapping the code → fuel name or reading a field we
+don't populate. Needs an FE-repo (Drizzle/Next) check, not a Model change.
+
+## 3. Survivor clusters from the post-fix population sweep
+
+`scripts/hyde/case_a_population_sweep.py`: the representative sample rebaselined
+cleanly (0 survivors), but the worst-old-drop sample held **28 survivors**
+(lodged A/B → new C/D, 15–25 SAP) in tight UPRN clusters (new-build blocks),
+spanning multiple schemas (16.0/16.1/17.1/18.0.0) and heating types. No single
+mapper-gap signature → likely genuine SAP-2012→10.2 drops for very-high-lodged
+new-builds, but **triage one in Elmhurst** to confirm genuine vs a residual calc
+issue before trusting the cohort.
+
+## 4. Predicted-property display path (e.g. property 721167)
+
+721167 has **no lodged EPC** (predicted). Its Heating-Control / Main-Fuel /
+Ventilation Unknowns come from the prediction + landlord-override **overlay** not
+populating the display fields — a separate path from the lodged-cert mappers.

scripts/hyde/case_a_population_sweep.py

@@ -0,0 +1,125 @@
+"""Local population sanity sweep for the full-SAP baseline-downgrade fix (ADR-0037).
+
+For a sample of Case A properties (portfolio 796, full-SAP path: epc_property
+assessment_type+sap_version NULL), re-map the real cert with the NEW mapper, run
+the SAP-10.2 calculator, and compute the post-fix Effective SAP/band. Compare to
+the lodged figures to (a) confirm the fix flips Effective off the stale lodged
+value and (b) flag survivors — drops too large to be a 2012→10.2 methodology
+shift (>15 SAP or >=2 bands below lodged), candidate deeper-calc/cert bugs.
+
+Read-only. No DB writes.
+"""
+from __future__ import annotations
+
+import os
+import random  # noqa: F401 — only used with a fixed seed below
+
+from scripts.e2e_common import load_env, build_engine
+from sqlalchemy import text
+
+load_env()
+
+from infrastructure.epc_client.epc_client_service import EpcClientService
+from datatypes.epc.domain.mapper import EpcPropertyDataMapper
+from domain.sap10_calculator.calculator import Sap10Calculator
+from datatypes.epc.domain.epc import Epc
+
+_BAND_IDX = {b: i for i, b in enumerate(["G", "F", "E", "D", "C", "B", "A"])}
+
+
+def _band_letter(epc_band: object) -> str:
+    s = str(epc_band)
+    return s.split(".")[-1] if "." in s else s
+
+
+def main() -> None:
+    eng = build_engine()
+    with eng.connect() as c:
+        # 30 worst old-drops + 30 spread across the cohort (deterministic order).
+        worst = c.execute(text("""
+            SELECT p.uprn, pbp.lodged_sap_score, pbp.lodged_epc_band,
+                   pbp.effective_sap_score AS old_eff, pl.post_sap_points AS old_post
+            FROM property p
+            JOIN epc_property ep ON ep.uprn=p.uprn
+            JOIN property_baseline_performance pbp ON pbp.property_id=p.id
+            JOIN plan pl ON pl.property_id=p.id
+            WHERE p.portfolio_id=796 AND ep.assessment_type IS NULL
+              AND ep.sap_version IS NULL
+            ORDER BY (pbp.effective_sap_score - pl.post_sap_points) DESC
+            LIMIT 30
+        """)).fetchall()
+        spread = c.execute(text("""
+            SELECT p.uprn, pbp.lodged_sap_score, pbp.lodged_epc_band,
+                   pbp.effective_sap_score AS old_eff, pl.post_sap_points AS old_post
+            FROM property p
+            JOIN epc_property ep ON ep.uprn=p.uprn
+            JOIN property_baseline_performance pbp ON pbp.property_id=p.id
+            JOIN plan pl ON pl.property_id=p.id
+            WHERE p.portfolio_id=796 AND ep.assessment_type IS NULL
+              AND ep.sap_version IS NULL AND (p.uprn % 7) = 0
+            ORDER BY p.uprn LIMIT 30
+        """)).fetchall()
+
+    rows = {r._mapping["uprn"]: dict(r._mapping) for r in worst}
+    for r in spread:
+        rows.setdefault(r._mapping["uprn"], dict(r._mapping))
+
+    svc = EpcClientService(os.environ["OPEN_EPC_API_TOKEN"])
+    calc = Sap10Calculator()
+
+    n = 0
+    flipped = 0
+    calc_errors = 0
+    survivors = []
+    drops = []
+    for uprn, row in rows.items():
+        n += 1
+        try:
+            results = svc._search(uprn=int(uprn))
+            if not results:
+                print(f"  uprn={uprn} NO_CERT")
+                continue
+            latest = max(results, key=lambda x: x.registration_date)
+            raw = svc._fetch_certificate(latest.certificate_number)
+            epc = EpcPropertyDataMapper.from_api_response(raw)
+            new_sap = calc.calculate(epc).sap_score
+        except Exception as e:  # noqa: BLE001 — sweep tolerates per-cert failure
+            calc_errors += 1
+            print(f"  uprn={uprn} CALC_ERROR {type(e).__name__}: {str(e)[:90]}")
+            continue
+        new_band = _band_letter(Epc.from_sap_score(new_sap))
+        lodged = row["lodged_sap_score"]
+        lodged_band = row["lodged_epc_band"]
+        drop = lodged - new_sap
+        drops.append(drop)
+        if epc.sap_version is not None and epc.sap_version < 10.2:
+            flipped += 1
+        band_drop = _BAND_IDX.get(lodged_band, 0) - _BAND_IDX.get(new_band, 0)
+        survivor = drop > 15 or band_drop >= 2
+        tag = "  *** SURVIVOR" if survivor else ""
+        if survivor:
+            survivors.append((uprn, lodged, lodged_band, new_sap, new_band, drop))
+        print(
+            f"  uprn={uprn} lodged={lodged}/{lodged_band} "
+            f"new_eff={new_sap}/{new_band} drop={drop} "
+            f"(old_eff={row['old_eff']} old_post={round(float(row['old_post']),1)}){tag}"
+        )
+
+    computed = len(drops)
+    print("\n==== SUMMARY ====")
+    print(f"sampled={n}  computed={computed}  calc_errors={calc_errors}")
+    print(f"rebaseline flipped (sap_version<10.2 now set) = {flipped}/{computed}")
+    if drops:
+        drops_sorted = sorted(drops)
+        print(
+            f"lodged-new_eff drop: min={min(drops)} median="
+            f"{drops_sorted[len(drops_sorted)//2]} max={max(drops)} "
+            f"mean={round(sum(drops)/len(drops),1)}"
+        )
+    print(f"survivors (>15 SAP or >=2 bands below lodged) = {len(survivors)}")
+    for s in survivors:
+        print(f"   SURVIVOR uprn={s[0]} lodged={s[1]}/{s[2]} new={s[3]}/{s[4]} drop={s[5]}")
+
+
+if __name__ == "__main__":
+    main()