mirror of
https://github.com/Hestia-Homes/Model.git
synced 2026-07-27 23:35:01 +00:00
178af2f044
scripts/run_pashub_modelling.py invokes the modelling_e2e handler in-process over a portfolio using STORED EPCs (refetch_epc=False — the correct source for PasHub cohorts, issue #1589), parametrised by env (portfolio/scenario/pids/ dry-run/batch). Handover documents the 3 re-extraction stragglers, the rebaseliner pass-through gotcha (effective_sap_score IS the pashub rating for unchanged lodged EPCs — never validate the calculator against it), and the 7 root-caused extractor bugs now tracked in #1590. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
86 lines
5.3 KiB
Markdown
86 lines
5.3 KiB
Markdown
# PasHub portfolio 838 — problem properties & validation (2026-07-14)
|
||
|
||
Cohort: **The Guinness Partnership GMCA** (205 PasHub site-notes properties),
|
||
portfolio **838**, scenario **1297**. Modelled locally through the real
|
||
`applications/modelling_e2e` lambda handler using the **stored** PasHub site
|
||
notes (`refetch_epc=False`) — see `scripts/run_pashub_modelling.py`.
|
||
|
||
## Headline: our SAP calculator diverges from pashub's rating by ~MAE 2.7 — real work remains
|
||
|
||
**Do not use `property_baseline_performance.effective_sap_score` to validate the
|
||
calculator.** For a SAP-10.2 lodged EPC whose physical state is unchanged, the
|
||
rebaseliner passes **Lodged Performance through as Effective**
|
||
(`domain/property_baseline/rebaseliner.py`), and Lodged Performance's `sap_score`
|
||
is `energy_rating_current` — the fetched pashub rating
|
||
(`domain/property_baseline/performance.py:63`). So `effective_sap_score` **is**
|
||
the pashub rating; comparing it to `energy_rating_current` compares the rating to
|
||
itself (circular — an earlier "99.5% match" was this mistake).
|
||
|
||
The real accuracy of our **calculator** is the `test_pashub_sap_accuracy`
|
||
harness: **~12.2% within-0.5, MAE ~2.7** vs `pre_sap` (which ≈ the pashub
|
||
rating, MAE 0.15 — same ground truth). Worked example: property **754881** — our
|
||
extraction is **byte-identical** to the stored ingestion (fuel 26, PCDB index
|
||
18119, wall 4/2, roof 200mm, party-wall 4, floor solid), yet our calculator
|
||
scores **71.9** where pashub lodged **85** — a genuine ~13-point gap to
|
||
root-cause. These are the extraction/calculator bugs to hunt.
|
||
|
||
**Image stripping does NOT lose data:** the fixture (stripped) and DB (original
|
||
PDF) parses of 754881 are identical field-for-field — so the harness fixtures are
|
||
faithful and the ~2.7 MAE is a true calculator gap, not a fixture artifact.
|
||
`parse_site_notes_pdf` never extracts the SAP rating from the PDF at all
|
||
(`energy_rating_current` is populated by Dan's separate pashub-API fetch,
|
||
`pashub_service.py:45` ← `preSapRating`), which is why the rating is `None` on a
|
||
raw parse but present in the DB — unrelated to stripping.
|
||
|
||
## Properties that did not run (need PasHub re-extraction — data, not code)
|
||
|
||
These 3 have **no fresh int-coded stored site note** (only pre-fix rows with a
|
||
string fuel), so modelling strict-raised and skipped them. Re-trigger PasHub
|
||
extraction so a current-mapper site note is stored, then they model like the
|
||
other 201.
|
||
|
||
| property_id | UPRN | address | stored site-note fuel | fix |
|
||
|---|---|---|---|---|
|
||
| 754772 | 77168847 | 9 Philips Park Court, Willdale Close, M11 4DH | `"Mains gas"` (string) | re-extract |
|
||
| 754780 | 77180607 | 12 Seymour Road South, Clayton, M11 4PG | `"Mains gas"` (string) | re-extract |
|
||
| 754844 | 77155031 | 16 Bingley Close, Beswick, M11 3RF | `""` (blank) | re-extract; if the survey genuinely lodges no main fuel, that is a separate blank/residual-fuel mapper gap |
|
||
|
||
## Properties that ran but not from their own PasHub survey
|
||
|
||
| property_id | UPRN | address | note |
|
||
|---|---|---|---|
|
||
| 754816 | 77180053 | 130 Stanton Street, Clayton, M11 4PX | **no PasHub site note exists at all** — modelled via the prediction path. Needs extraction so it models from its own survey. |
|
||
| 754778 | 77181049 | (M11) | winner lodged EPC is **not** a PasHub site note (gov EPC / other); modelled SAP 60 vs stored rating 57 (**d=+3**, the cohort's only >0.5 divergence). Check why its PasHub site note isn't the winner. |
|
||
|
||
## Accuracy outliers surfaced by closing the extractor gaps
|
||
|
||
Closing the last 4 `test_pashub_sap_accuracy` xfails (see the PR) makes them
|
||
**compute**; two then show a large gap **vs `pre_sap`** (not vs the stored
|
||
rating — verify `pre_sap` first, do not tune to it):
|
||
|
||
| fixture (deal) | ours | pre_sap | note |
|
||
|---|---|---|---|
|
||
| 499584755922 | 34.0 | 67 | main fuel extracted as **Bulk LPG (27)** + house-coal secondary → low SAP. Either a bad `pre_sap` or a main-fuel extraction issue; confirm the survey's main fuel. |
|
||
| 507644414148 | 70.2 | 52 | **community heating**. The PasHub path maps only the Table 4e Group 3 control code (2306); it does **not** yet set the full heat-network fuel/flags (main_fuel is 26, not a Table 12 community code). Deeper community-heating mapping gap. |
|
||
|
||
## Root-caused extractor bugs (issue #1590)
|
||
|
||
Five worst-divergence properties were deep-dived (extracted inputs diffed
|
||
against the PDF text; gaps attributed via patch-and-rerun). **7 distinct
|
||
extractor bugs** — PV arrays never extracted (−13.5), roof "Insulation At:
|
||
None" treated as unknown not zero (−7.6/−7.1, systematic), `pv_connection`
|
||
string passthrough (−6.9), ventilation kind never mapped (−4.7), room-in-roof
|
||
never built (−2.1), cylinder "No Access" passthrough (−0.6), system-build/
|
||
basement code-6 collision (latent) — are itemised with fixes and fixture deals
|
||
in **https://github.com/Hestia-Homes/Model/issues/1590**. Also there: the
|
||
ground-truth caveat that `pre_sap` is pashub's *preliminary* figure — the
|
||
accredited lodgement can differ downstream of the site notes (754917: 53 → 43).
|
||
|
||
## How to reproduce
|
||
|
||
```
|
||
python scripts/run_pashub_modelling.py # dry-run, all 838 batches
|
||
RUN_DRY=0 python scripts/run_pashub_modelling.py # real writes
|
||
RUN_PIDS="754772" RUN_DRY=0 python scripts/run_pashub_modelling.py # one property
|
||
```
|
||
Validation sweep (ours vs stored rating) is in the PR description / this handover.
|