diff --git a/docs/HANDOVER_PASHUB_EXTRACTOR_ACCURACY.md b/docs/HANDOVER_PASHUB_EXTRACTOR_ACCURACY.md new file mode 100644 index 000000000..36e919967 --- /dev/null +++ b/docs/HANDOVER_PASHUB_EXTRACTOR_ACCURACY.md @@ -0,0 +1,99 @@ +# PasHub extractor accuracy — handover (2026-07-14 late) + +## Goal +Guinness GMCA 205-property cohort within 0.5 SAP of pashub's `pre_sap` +(PRD #1555). Khalim: "We can get to 100% within 0.5 for this cohort." The +bugs are **extraction** (survey detail not reaching the calculator), not the +calculator — proven repeatedly this session. + +## Where we are (branch `feat/pashub-extractor-fixes-and-runner`, pushed) +Harness `backend/documents_parser/tests/test_pashub_sap_accuracy.py`: +**205/205 compute, MAE 1.551, within-0.5 18.5% (38/205), bias ≈ −1.0** +(from 12.4%/2.56 at session start). Ratchets `_MIN_WITHIN_HALF=0.18`, +`_MAX_SAP_MAE=1.56` — tighten after every fix, comment each move. + +### Fixed this session (all TDD 🟥/🟩 pairs; see git log on the branch) +Issue **#1590** (all 7 subtasks done + 2 round-2 fixes; trajectory table in +its last comment): secondary fuel labels; Table 4e Group 3 control 2306; +roof "Insulation At: None"→explicit 0; **PV arrays extracted+mapped** +(`_pashub_pv_arrays`); `pv_connection`→int (1/2); **mechanical ventilation** +`_pashub_mechanical_ventilation_kind` (biggest win: 76 fixtures, MAE −0.8); +cylinder size Table 28; system-build/basement code-6 collision; +**room-in-roof** (survey `RoomInRoofDetail` → extractor `_parse_room_in_roof` +→ mapper `_pashub_room_in_roof`; commons are `stud_wall`, roof-space +insulation threads into slope/stud/flat surfaces); **community heating** +(extractor captures "Heating System (Other):" → `_pashub_community_heating` +→ `sap_main_heating_code=301` + Table 12 fuel 51 via +`_resolve_community_heating_fuel_code`). + +Issue **#1589** (separate, unfixed): modelling `refetch_epc=True` violates +ADR-0001 recency — gov API overrides newer stored site notes. + +## Ground truth — now verified +`pre_sap` in the manifest was hand-verified by Khalim in pashub for the 4 +worst deviations (2026-07-14): manifest corrected for 499584755922 (F33, was +stale D67 — we were right) and 499617935574 (accredited E43, was preliminary +53). **Confirmed correct** (so our deviation = extraction bug to hunt): + +| deal | address | pashub | ours | d | +|---|---|---|---|---| +| 499530733767 | 58 Hackle Street, M11 4WU | D58 | 65.8 | **+7.8** | +| 499617935574 | Brightholme, 14 North Road, M11 4WE | 43 | 50.3 | **+7.3** | +| 507644414148 | 16 Bingley Close, M11 3RF | E52 | 57.7 | **+5.7** | + +Rule: don't tune to `pre_sap` blindly, but after this verification treat it +as trustworthy unless a survey PDF plainly contradicts it (then ask Khalim to +check pashub — give him the ADDRESS, not the deal id). + +## The proven working loop (repeat until 100%) +1. Rank divergences: parse every manifest fixture (`parse_site_notes_pdf` → + `Sap10Calculator().calculate`) vs `pre_sap_score`; list worst signed. +2. Fan out 3-5 Sonnet sub-agents, one per worst case, with THE RECIPE: + dump extracted `EpcPropertyData` fields; dump PDF text (`fitz`); diff + field-by-field against the survey; attribute via `dataclasses.replace` + patch-and-rerun (record SAP delta per candidate fix); report the + mis-extracted field + concrete fix + deltas. Investigation only, no edits. +3. TDD each confirmed fix (`/tdd` skill): RED commit 🟥 → GREEN commit 🟩, + test in `datatypes/epc/domain/tests/test_from_site_notes.py` (mapper) and/or + `backend/documents_parser/tests/test_extractor.py` (extractor; splice + verbatim PDF lines into `load_text_fixture()`); update the golden + `test_full_mapping` when field coding changes. +4. Re-run harness; ratchet; commit; push. + +## Conventions (non-negotiable) +- `_pashub_*` helpers strict-raise `UnmappedPasHubLabel` on unknown non-empty + labels; blank passes through/None. Mirror `_ELMHURST_*`/`_api_*` siblings. +- pyright: zero NEW errors (baselines: mapper.py 39, extractor.py 34, + test_from_site_notes.py ≤13). Beware `Union[RoofSpaceDetail, + ExtensionRoofSpace]` attribute access — add fields to BOTH. +- Survey dataclasses hold RAW labels; coding happens at the mapper boundary + (ADR-0015). Ratchets never loosen at fixed coverage. + +## Known leads for the remaining gap +- **Solid ground floors**: worst-biased category (n=107, mean −1.53; 5/7 of + the −4..−5 Natural-ventilation cluster: 499529160912, 499538003156, + 499627236560, 499586692333, 507533541571, 499570197710, 499592219845). + Dig the ISO 13370 solid-floor U/perimeter cascade vs what the surveys lodge. +- 16 Bingley (+5.7): community-heating residual — water heating is + "Hot water only community scheme - boilers" (WHC 950 path?), community + standing charge/tariff detail, or DLF "Unknown" distribution type. +- 58 Hackle (+7.8): undug — full recipe pass needed. +- Brightholme (+7.3): RIR dwelling; the accredited 43 reflects a QA + correction ("no access to loft — hatch screwed shut") — possibly RIR + insulation should NOT thread from the roof-space block when access was + impossible, or slopes should default worse. Check `_pashub_room_in_roof`. +- Mechanical ventilation and PCDB efficiency are VERIFIED CORRECT — don't re-dig. + +## Also outstanding +- **Open the review PR** for the branch (metrics table = #1590's last + comment; ~26 commits, all green: 206 harness + ~294 unit tests). +- Portfolio 838 modelling: re-extract 4 properties in pashub (754772/9 + Philips Park Court, 754780/12 Seymour Rd S, 754844/16 Bingley — the + community fix now makes it extractable, 754816/130 Stanton St has no site + note at all); then `RUN_DRY=0 python scripts/run_pashub_modelling.py` + (env-parametrised: RUN_PORTFOLIO/RUN_SCENARIO/RUN_PIDS/RUN_DRY). +- `effective_sap_score` == the pashub rating for unchanged lodged dwellings + (rebaseliner pass-through) — NEVER validate the calculator against it. +- hubspot_deal_data still holds the stale pre_sap for the 2 corrected deals — + re-running `scripts/build_pashub_accuracy_fixtures.py` will REVERT the + manifest corrections unless the DB rows are fixed first.