Model/docs/HANDOVER_PASHUB_838_PROBLEM_PROPERTIES.md

PasHub portfolio 838 — problem properties & validation (2026-07-14)

Cohort: The Guinness Partnership GMCA (205 PasHub site-notes properties), portfolio 838, scenario 1297. Modelled locally through the real applications/modelling_e2e lambda handler using the stored PasHub site notes (refetch_epc=False) — see scripts/run_pashub_modelling.py.

Headline: our SAP calculator diverges from pashub's rating by ~MAE 2.7 — real work remains

Do not use property_baseline_performance.effective_sap_score to validate the calculator. For a SAP-10.2 lodged EPC whose physical state is unchanged, the rebaseliner passes Lodged Performance through as Effective (domain/property_baseline/rebaseliner.py), and Lodged Performance's sap_score is energy_rating_current — the fetched pashub rating (domain/property_baseline/performance.py:63). So effective_sap_score is the pashub rating; comparing it to energy_rating_current compares the rating to itself (circular — an earlier "99.5% match" was this mistake).

The real accuracy of our calculator is the test_pashub_sap_accuracy harness: ~12.2% within-0.5, MAE ~2.7 vs pre_sap (which ≈ the pashub rating, MAE 0.15 — same ground truth). Worked example: property 754881 — our extraction is byte-identical to the stored ingestion (fuel 26, PCDB index 18119, wall 4/2, roof 200mm, party-wall 4, floor solid), yet our calculator scores 71.9 where pashub lodged 85 — a genuine ~13-point gap to root-cause. These are the extraction/calculator bugs to hunt.

Image stripping does NOT lose data: the fixture (stripped) and DB (original PDF) parses of 754881 are identical field-for-field — so the harness fixtures are faithful and the ~2.7 MAE is a true calculator gap, not a fixture artifact. parse_site_notes_pdf never extracts the SAP rating from the PDF at all (energy_rating_current is populated by Dan's separate pashub-API fetch, pashub_service.py:45 ← preSapRating), which is why the rating is None on a raw parse but present in the DB — unrelated to stripping.

Properties that did not run (need PasHub re-extraction — data, not code)

These 3 have no fresh int-coded stored site note (only pre-fix rows with a string fuel), so modelling strict-raised and skipped them. Re-trigger PasHub extraction so a current-mapper site note is stored, then they model like the other 201.

property_id	UPRN	address	stored site-note fuel	fix
754772	77168847	9 Philips Park Court, Willdale Close, M11 4DH	"Mains gas" (string)	re-extract
754780	77180607	12 Seymour Road South, Clayton, M11 4PG	"Mains gas" (string)	re-extract
754844	77155031	16 Bingley Close, Beswick, M11 3RF	"" (blank)	re-extract; if the survey genuinely lodges no main fuel, that is a separate blank/residual-fuel mapper gap

Properties that ran but not from their own PasHub survey

property_id	UPRN	address	note
754816	77180053	130 Stanton Street, Clayton, M11 4PX	no PasHub site note exists at all — modelled via the prediction path. Needs extraction so it models from its own survey.
754778	77181049	(M11)	winner lodged EPC is not a PasHub site note (gov EPC / other); modelled SAP 60 vs stored rating 57 (d=+3, the cohort's only >0.5 divergence). Check why its PasHub site note isn't the winner.

Accuracy outliers surfaced by closing the extractor gaps

Closing the last 4 test_pashub_sap_accuracy xfails (see the PR) makes them compute; two then show a large gap vs pre_sap (not vs the stored rating — verify pre_sap first, do not tune to it):

fixture (deal)	ours	pre_sap	note
499584755922	34.0	67	main fuel extracted as Bulk LPG (27) + house-coal secondary → low SAP. Either a bad pre_sap or a main-fuel extraction issue; confirm the survey's main fuel.
507644414148	70.2	52	community heating. The PasHub path maps only the Table 4e Group 3 control code (2306); it does not yet set the full heat-network fuel/flags (main_fuel is 26, not a Table 12 community code). Deeper community-heating mapping gap.

Root-caused extractor bugs (issue #1590)

Five worst-divergence properties were deep-dived (extracted inputs diffed against the PDF text; gaps attributed via patch-and-rerun). 7 distinct extractor bugs — PV arrays never extracted (−13.5), roof "Insulation At: None" treated as unknown not zero (−7.6/−7.1, systematic), pv_connection string passthrough (−6.9), ventilation kind never mapped (−4.7), room-in-roof never built (−2.1), cylinder "No Access" passthrough (−0.6), system-build/ basement code-6 collision (latent) — are itemised with fixes and fixture deals in https://github.com/Hestia-Homes/Model/issues/1590. Also there: the ground-truth caveat that pre_sap is pashub's preliminary figure — the accredited lodgement can differ downstream of the site notes (754917: 53 → 43).

How to reproduce

python scripts/run_pashub_modelling.py           # dry-run, all 838 batches
RUN_DRY=0 python scripts/run_pashub_modelling.py  # real writes
RUN_PIDS="754772" RUN_DRY=0 python scripts/run_pashub_modelling.py  # one property

Validation sweep (ours vs stored rating) is in the PR description / this handover.