Re-pin Wythenshawe accuracy harness to the DB oracle (#1649)

`test_pashub_sap_accuracy_wythenshawe` graded against the manifest `pre_sap`
(`hubspot_deal_data.pre_sap`) — surveyor hand-entered and frequently wrong,
which floated a misleading 79% / MAE 0.75. Grade instead against the DB
PasHub-assessment SAP (`epc_property_energy_performance.energy_rating_current`,
`ep.source='lodged'`), matched by UPRN or postcode+house-number.

`scripts/enrich_pashub_wythenshawe_oracle.py` resolves each fixture's oracle SAP
once (DB required, not run in CI) and bakes `oracle_sap` / `oracle_match` into
`manifest.json`, so the harness stays DB-free. 148/168 resolve (39 UPRN + 109
address); the 20 unmatched carry `oracle_sap` null and are excluded from the
aggregate until PasHub sources their SAP.

With the six from_site_notes fixes, the 148 matched score within-0.5 89.2%
(132/148), MAE 0.297 (was, vs pre_sap, 79.2% / 0.754). Ratchets re-baselined
0.75→0.88 floor, 0.85→0.31 ceiling. The residual tail is the integer-oracle
rounding floor (spec-correct dwellings) plus a couple of spec-literal gaps, not
engine defects.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

backend/documents_parser/tests/test_pashub_sap_accuracy_wythenshawe.py

@@ -1,17 +1,30 @@
 """End-to-end accuracy test for the pashub site-notes extractor — WYTHENSHAWE.
 
 Sibling of `test_pashub_sap_accuracy.py` (the Guinness GMCA cohort); see that
-module's docstring for the full chain and the meaning of the hybrid `pre_sap`
-gauge. This module points at the `pashub_accuracy_wythenshawe/` fixtures built by
+module's docstring for the full chain. This module points at the
+`pashub_accuracy_wythenshawe/` fixtures built by
 
     python scripts/build_pashub_accuracy_fixtures.py \\
         --project-code "[Wythenshawe WH:SHF RAs - WH:SF - 240426 - 494493718748]" \\
         --out-dir pashub_accuracy_wythenshawe
 
 from `hubspot_deal_data` (the project_code above) + the `rd_sap_site_note` PDFs
-in S3. The ratchets below are calibrated to THIS cohort — do not copy the
-Guinness constants across; each portfolio discovers its own floor/ceiling by
-running.
+in S3.
+
+ORACLE: this module grades against the **DB PasHub-assessment SAP** — the
+`epc_property_energy_performance.energy_rating_current` (`ep.source='lodged'`)
+that PasHub's own assessment lodged for the dwelling — NOT the manifest
+`pre_sap` (`hubspot_deal_data.pre_sap`, surveyor hand-entered and frequently
+wrong; grading against it gave a misleading 79.2% / MAE 0.75 while the engine
+actually matched the DB in every pre_sap≠DB case). Each fixture's oracle SAP is
+resolved (by UPRN or postcode+house-number) and baked into `manifest.json` as
+`oracle_sap` by `scripts/enrich_pashub_wythenshawe_oracle.py`, so CI needs no DB
+access. 20 fixtures have no DB-oracle match yet (`oracle_sap` null) and are
+excluded from the accuracy aggregate until PasHub sources their SAP.
+
+The ratchets below are calibrated to THIS cohort against the DB oracle — do not
+copy the Guinness constants across; each portfolio discovers its own
+floor/ceiling by running.
 """
 
 from __future__ import annotations
@@ -35,39 +48,22 @@ _KNOWN_GAPS: tuple[type[Exception], ...] = (UnmappedSapCode, UnmappedPasHubLabel
 _FIXTURES_DIR = Path(__file__).parent / "fixtures" / "pashub_accuracy_wythenshawe"
 _MANIFEST_PATH = _FIXTURES_DIR / "manifest.json"
 
-# --- Ratchets (never loosen at fixed coverage), mirroring the Guinness module.
-# First full run (2026-07-16): **149 computed / 19 blocked / 0 errored**
-# (within-0.5 76.5% = 114/149, MAE 0.665). The 19 blocks are a spread of
-# separately-ticketed mapper gaps — floor type 'Same dwelling below' (×10),
-# room-in-roof gable 'None' (×2), 'No thermostatic control...' on 'Room heaters'
-# (×2), plus one-off secondary-heating / secondary-fuel / room-heater / HHR
-# storage-control (2404-on-402) labels — each its own field-by-field follow-up,
-# not tuning. The floor/ceiling below are the observed baseline with a hair of
-# headroom; treat them as a regression tripwire, NOT a cohort accuracy figure,
-# and re-baseline (coverage growth is not loosening) as further mapper fixes
-# unblock fixtures.
-# 2026-07-16 (#1622/#1624): room-heater control 2601 unblocked the two 'No
-# thermostatic control...' fixtures → **151 computed / 17 blocked** (within-0.5
-# 76.2% = 115/151, MAE 0.716; 500767954156 lands within 0.5, 500797465814 is a
-# −8.8 tail entry). MAE ceiling re-baselined 0.67 → 0.72 per the
-# coverage-growth convention above.
-# 2026-07-16 (#1622/#1625): floor type 'Same dwelling below' → "(another
-# dwelling below)" party-floor suppression (gov-API code-8 precedent) unblocked
-# all 10 fixtures → **161 computed / 7 blocked** (within-0.5 75.8% = 122/161,
-# MAE 0.740). 7/10 newly-computed land within 0.5 — the treatment validates;
-# −3.3/−5.5 tail entries dilute the aggregates. Floor 0.76 → 0.75, ceiling
-# 0.72 → 0.75.
-# 2026-07-16 (#1622/#1626-#1631): the remaining one-off labels (HHR-control
-# 2404→2402 downgrade, 'Loose jacket' cylinder, secondary 601/609, secondary
-# dual fuel, gas room-heater 603) → **165 computed / 3 blocked** (within-0.5
-# 75.2% = 124/165, MAE 0.844). The MAE step is one accepted pre_sap
-# divergence: 500797805807 (90 Brookfield Gardens) scores 49.5 vs pre_sap 31
-# (+18.5) — its own 2025 accredited cert 6890-0597-0722-0196-3943 lodges the
-# identical heating codes (603/26/2601) at band E, agreeing with US; the
-# PasHub pre_sap instead matches the dwelling's superseded electric
-# room-heater certs (2017/2023, band F). Ceiling 0.75 → 0.85.
-_MIN_WITHIN_HALF: float = 0.75
-_MAX_SAP_MAE: float = 0.85
+# --- Ratchets (never loosen at fixed coverage), against the DB oracle.
+# Re-pinned to the DB PasHub-assessment SAP (#1649): grading the earlier
+# pre_sap-based history (which floated 75-76% / MAE 0.67-0.85 purely on
+# surveyor data-entry noise) is retired — see the module docstring.
+# 2026-07-19 (#1649 A-F): with the six from_site_notes fixes in place, the
+# **148 DB-oracle-matched fixtures** (39 UPRN + 109 postcode+house-number; 20
+# unmatched, excluded) score **within-0.5 89.2% (132/148), MAE 0.297**. The
+# residual tail is not fixable engine defects: the DB oracle is a *rounded
+# integer* SAP graded against a continuous engine (a +0.5-to-band cluster of
+# spec-correct filled-cavity dwellings sits at that rounding floor), plus a
+# couple of spec-literal-vs-accredited-engine gaps (e.g. 6 Barry Road's
+# uninsulated solid-brick alternative wall at the RdSAP Table 6 U=1.7 default).
+# Treat floor/ceiling as a regression tripwire, not a target; re-baseline
+# (coverage growth is not loosening) as PasHub sources the 20 unmatched SAPs.
+_MIN_WITHIN_HALF: float = 0.88
+_MAX_SAP_MAE: float = 0.31
 
 _KNOWN_GAP_REASON = (
     "pashub `from_site_notes` mapper does not int-code a main-heating "
@@ -82,6 +78,13 @@ class PashubFixture:
     pre_sap_raw: str
     pre_sap_score: int
     pdf: str
+    # DB PasHub-assessment SAP (the grading oracle) —
+    # `epc_property_energy_performance.energy_rating_current`, `source='lodged'`,
+    # matched by UPRN or postcode+house-number and baked into the manifest by
+    # `scripts/enrich_pashub_wythenshawe_oracle.py` (see the module docstring).
+    # None for the 20 address-unmatched fixtures (excluded from the aggregate).
+    oracle_sap: Optional[int] = None
+    oracle_match: Optional[str] = None  # "uprn" | "address" | None
 
     @property
     def path(self) -> Path:
@@ -95,7 +98,7 @@ class Outcome:
     blocked: bool  # blocked on the known main-heating-control gap (xfail territory)
     reason: Optional[str]
     sap_continuous: Optional[float]
-    diff: Optional[float]  # abs(our SAP - pre_sap)
+    diff: Optional[float]  # abs(our SAP - DB oracle); None when unmatched/blocked
 
 
 def _load_manifest() -> list[PashubFixture]:
@@ -119,7 +122,14 @@ def _evaluate(deal_id: str) -> Outcome:
         result = Sap10Calculator().calculate(epc)
     except _KNOWN_GAPS as exc:
         return Outcome(blocked=True, reason=str(exc), sap_continuous=None, diff=None)
-    diff = abs(result.sap_score_continuous - fixture.pre_sap_score)
+    # Grade against the DB PasHub-assessment oracle, not the surveyor-entered
+    # `pre_sap`. A fixture with no DB-oracle match carries diff None and is
+    # excluded from the accuracy aggregate (it still asserts it computes).
+    diff = (
+        abs(result.sap_score_continuous - fixture.oracle_sap)
+        if fixture.oracle_sap is not None
+        else None
+    )
     return Outcome(
         blocked=False,
         reason=None,
@@ -141,10 +151,14 @@ def test_pashub_fixture_computes(deal_id: str) -> None:
 
 @pytest.mark.skipif(not _FIXTURES, reason="no pashub_accuracy_wythenshawe fixtures/manifest")
 def test_pashub_sap_accuracy_aggregate(capsys: pytest.CaptureFixture[str]) -> None:
-    """SAP within-0.5 of pashub's `pre_sap`, aggregated over computable fixtures."""
+    """SAP within-0.5 of the DB PasHub-assessment oracle, aggregated over the
+    address/UPRN-matched fixtures. Fixtures with no DB-oracle match are counted
+    (`unmatched`) but excluded from the accuracy figures — they cannot be graded
+    until PasHub sources their SAP."""
     diffs: list[float] = []
     blocked = 0
     errored = 0
+    unmatched = 0
     for deal_id in _IDS:
         try:
             outcome = _evaluate(deal_id)
@@ -154,13 +168,15 @@ def test_pashub_sap_accuracy_aggregate(capsys: pytest.CaptureFixture[str]) -> No
         if outcome.blocked:
             blocked += 1
             continue
-        assert outcome.diff is not None
+        if outcome.diff is None:  # computed, but no DB-oracle SAP to grade against
+            unmatched += 1
+            continue
         diffs.append(outcome.diff)
 
     if not diffs:
         pytest.xfail(
-            f"no fixture computes a SAP score "
-            f"({blocked} blocked on the main-heating-control gap, {errored} errored); "
+            f"no fixture grades against the DB oracle "
+            f"({blocked} blocked, {unmatched} address-unmatched, {errored} errored); "
             f"{_KNOWN_GAP_REASON}"
         )
 
@@ -170,8 +186,8 @@ def test_pashub_sap_accuracy_aggregate(capsys: pytest.CaptureFixture[str]) -> No
 
     with capsys.disabled():
         print(
-            f"\n[pashub WYTHENSHAWE | {n} computed / {blocked} blocked / "
-            f"{errored} errored]"
+            f"\n[pashub WYTHENSHAWE | {n} graded / {unmatched} unmatched / "
+            f"{blocked} blocked / {errored} errored | oracle=DB energy_rating_current]"
             f"\n  SAP within-0.5 = {within_half:.1%}   MAE = {sap_mae:.3f}"
         )
 

scripts/enrich_pashub_wythenshawe_oracle.py

@@ -0,0 +1,134 @@
+"""Enrich the PasHub Wythenshawe fixture manifest with the DB-oracle SAP.
+
+The accuracy harness must grade against the reliable oracle — the PasHub
+company-assessment result stored in the DB (`epc_property_energy_performance.
+energy_rating_current`, `ep.source='lodged'`) — NOT the manifest `pre_sap`
+(`hubspot_deal_data.pre_sap`, surveyor hand-entered and frequently wrong). This
+one-time / rebuild-able provenance script resolves each fixture to that DB SAP
+and writes `oracle_sap` (int or null) + `oracle_match` ("uprn" / "address" /
+null) back into `manifest.json`, so the committed manifest is what the test
+consumes and CI needs no DB access. Mirrors `build_pashub_accuracy_fixtures.py`.
+
+Match order: manifest `uprn` (bigint, exact) → else postcode + house-number
+parsed from the PDF "Property Address:" block. Latest cert per key by `ep.id`.
+
+Requires DB credentials (reads `backend/.env`); NOT run in CI.
+
+    python scripts/enrich_pashub_wythenshawe_oracle.py [--dry-run]
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+from typing import Optional
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(REPO_ROOT))
+
+from scripts.e2e_common import build_engine, load_env  # noqa: E402
+from sqlalchemy import text  # noqa: E402
+from sqlalchemy.engine import Connection  # noqa: E402
+
+from backend.documents_parser.parser import pdf_to_text_list  # noqa: E402
+
+_FIXTURES = (
+    REPO_ROOT
+    / "backend"
+    / "documents_parser"
+    / "tests"
+    / "fixtures"
+    / "pashub_accuracy_wythenshawe"
+)
+_MANIFEST = _FIXTURES / "manifest.json"
+_POSTCODE_RE = re.compile(r"^[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}$")
+_HOUSENUM_RE = re.compile(r"^(\d+[A-Za-z]?)\b")
+
+
+def _address_from_pdf(deal_id: str) -> tuple[Optional[str], Optional[str]]:
+    """(house_number, postcode) from the PDF "Property Address:" block."""
+    tl = pdf_to_text_list((_FIXTURES / f"{deal_id}.pdf").read_bytes())
+    try:
+        i = next(k for k, l in enumerate(tl) if l.strip() == "Property Address:")
+    except StopIteration:
+        return None, None
+    block = [l.strip().rstrip(",") for l in tl[i + 1 : i + 6] if l.strip()]
+    postcode = next((l for l in block if _POSTCODE_RE.match(l.upper())), None)
+    house = None
+    if block:
+        m = _HOUSENUM_RE.match(block[0])
+        if m:
+            house = m.group(1)
+    return house, postcode
+
+
+def _resolve(
+    conn: Connection, uprn: Optional[int], deal_id: str
+) -> tuple[Optional[int], str]:
+    if uprn:
+        row = conn.execute(
+            text(
+                "select eppe.energy_rating_current from epc_property ep "
+                "join epc_property_energy_performance eppe "
+                "on eppe.epc_property_id = ep.id "
+                "where ep.source='lodged' and ep.uprn=:u "
+                "and eppe.energy_rating_current is not null "
+                "order by ep.id desc limit 1"
+            ),
+            {"u": int(uprn)},
+        ).scalar()
+        if row is not None:
+            return int(row), "uprn"
+    house, postcode = _address_from_pdf(deal_id)
+    if house and postcode:
+        rows = conn.execute(
+            text(
+                "select ep.address_line_1, eppe.energy_rating_current "
+                "from epc_property ep join epc_property_energy_performance eppe "
+                "on eppe.epc_property_id = ep.id "
+                "where ep.source='lodged' "
+                "and replace(upper(ep.postcode),' ','')=:pc "
+                "and eppe.energy_rating_current is not null order by ep.id desc"
+            ),
+            {"pc": postcode.upper().replace(" ", "")},
+        ).fetchall()
+        for row_pair in rows:
+            a1 = str(row_pair[0] or "").strip()
+            m = _HOUSENUM_RE.match(a1)
+            if m and m.group(1).lower() == house.lower():
+                return int(row_pair[1]), "address"
+    return None, "unmatched"
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--dry-run", action="store_true")
+    args = ap.parse_args()
+
+    load_env(REPO_ROOT / "backend" / ".env")
+    engine = build_engine()
+    manifest = json.loads(_MANIFEST.read_text())
+
+    matched = 0
+    with engine.connect() as conn:
+        for fx in manifest["fixtures"]:
+            oracle_sap, kind = _resolve(conn, fx.get("uprn"), fx["deal_id"])
+            fx["oracle_sap"] = oracle_sap
+            fx["oracle_match"] = None if kind == "unmatched" else kind
+            if oracle_sap is not None:
+                matched += 1
+
+    n = len(manifest["fixtures"])
+    print(f"resolved {matched}/{n} fixtures to a DB oracle SAP")
+    if args.dry_run:
+        print("--dry-run: manifest NOT written")
+        return
+    _MANIFEST.write_text(json.dumps(manifest, indent=2) + "\n")
+    print(f"wrote {_MANIFEST}")
+
+
+if __name__ == "__main__":
+    main()