`test_pashub_sap_accuracy_wythenshawe` graded against the manifest `pre_sap`
(`hubspot_deal_data.pre_sap`) — surveyor hand-entered and frequently wrong,
which floated a misleading 79% / MAE 0.75. Grade instead against the DB
PasHub-assessment SAP (`epc_property_energy_performance.energy_rating_current`,
`ep.source='lodged'`), matched by UPRN or postcode+house-number.
`scripts/enrich_pashub_wythenshawe_oracle.py` resolves each fixture's oracle SAP
once (DB required, not run in CI) and bakes `oracle_sap` / `oracle_match` into
`manifest.json`, so the harness stays DB-free. 148/168 resolve (39 UPRN + 109
address); the 20 unmatched carry `oracle_sap` null and are excluded from the
aggregate until PasHub sources their SAP.
With the six from_site_notes fixes, the 148 matched score within-0.5 89.2%
(132/148), MAE 0.297 (was, vs pre_sap, 79.2% / 0.754). Ratchets re-baselined
0.75→0.88 floor, 0.85→0.31 ceiling. The residual tail is the integer-oracle
rounding floor (spec-correct dwellings) plus a couple of spec-literal gaps, not
engine defects.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deal's third_party_surveyor_identifier is now required by the log
and amend flows: the trigger contract rejects messages without it
(blank treated as missing, since a blank resource makes OpenHousing
silently drop the appointment), and LogJobRequest/AmendJobRequest carry
it as a required field. The ABRI_RELAY_DEFAULT_RESOURCE fallback and
its config/terraform/workflow wiring are removed as dead code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One-time ingest of Wythenshawe Community Housing Group Elmhurst *Summary*
PDFs: resolve each property's SharePoint folder from the WCHG xlsx, download
the summary, run the existing extractor -> mapper -> SAP10 pipeline, and
(with --commit) push the PDF to S3, record an uploaded_files row, and save
the EpcPropertyData to epc_property keyed on the national UPRN (property_id
NULL, replace-by-(uprn,source)). Dry-run by default; also emits a SAP-accuracy
CSV vs Elmhurst's reported current SAP (% within 0.5 + MAE).
Adds DomnaSites.SOCIAL_HOUSING (SOCIAL_HOUSING_SHAREPOINT_ID) for the site.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/run_pashub_modelling.py invokes the modelling_e2e handler in-process
over a portfolio using STORED EPCs (refetch_epc=False — the correct source for
PasHub cohorts, issue #1589), parametrised by env (portfolio/scenario/pids/
dry-run/batch). Handover documents the 3 re-extraction stragglers, the
rebaseliner pass-through gotcha (effective_sap_score IS the pashub rating for
unchanged lodged EPCs — never validate the calculator against it), and the 7
root-caused extractor bugs now tracked in #1590.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Foundation for PRD #1555: runs each PAS Hub site-note PDF through the
extractor -> EpcPropertyData -> Sap10Calculator and gauges the computed
SAP against pashub's own SAP-10.2 `pre_sap` (from hubspot_deal_data).
- test_pashub_sap_accuracy.py: hybrid gate. Per-fixture "must compute"
(xfail on the known in-progress mapper gaps MissingMainFuelType /
UnmappedSapCode / UnmappedPasHubLabel) + aggregate within-0.5 ratchet
floor, mirroring test_sap_accuracy_corpus.py.
- 205 image-stripped site-note PDFs + manifest.json. Images stripped so
the repo footprint stays ~52MB while the text layer the extractor reads
is byte-identical.
- build_pashub_accuracy_fixtures.py: provenance/rebuild from S3 +
hubspot_deal_data.
All 206 currently xfail on the known heating-string mapper gaps; each fix
(#1556-1568) flips its fixtures to computing and ratchets the floor.
Refs #1555#1568
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-attempted entering this SAP-15.0 cert's 4 real N/E/S/W windows (gov-API
sap_windows, 2.516 m^2 each) as separate Elmhurst Openings rows instead of
the accepted single combined South row, to test whether it closes the
residual ~2pt gap (engine 53 vs Elmhurst worksheet 51).
Reproduced live the documented elmhurst_lib.py window-grid limitation:
adding a 2nd row wipes the previously-added row's width/height to 0.00,
in both same-session and fresh-session-per-window attempts. Not fixable
via current Playwright automation. Restored the assessment to its original
single South 10.06 m^2 row (no PDFs re-downloaded, no worksheet change).
Separately confirmed our own engine already models this cert's windows
per-orientation correctly from gov-API sap_windows via
solar_gains.py's ORIENTATION_BY_SAP10_CODE cascade -- this is an Elmhurst
build-tooling ceiling, not an engine or mapper bug. sap_score=53 remains
unchanged; no calculator/mapper code was touched. Full accuracy suite
re-run clean (66 passed/67 skipped/1 xfailed + corpus test passed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both flagged mismatches were Elmhurst input errors (same silent-stale-value
contamination class as the earlier chimneys/wall-thickness bugs), not
parser or calculator bugs:
- cylinder_size: build_100010086084.py's water_heating() selected
DropDownListCylinderSize by raw value "2", but this DOM's option values
ARE their visible litre-band text (no "2" among them) — Playwright
silently no-ops on a non-matching value, leaving a prior cert's "Medium
(131-170L)" selection in place. Fixed to match by text ("Normal"), and
added the missing mapper dict entries (Normal/Large litre-suffixed
labels) to _ELMHURST_CYLINDER_SIZE_LABEL_TO_SAP10 (a real mapper-coverage
gap — the calculator raises UnmappedElmhurstLabel rather than silently
mis-mapping).
- boiler_flue_type: the boiler-code search dialog's combined "Balanced/Open
Flue" Table 4b category doesn't drive the separate, independently
selectable RadioButtonListFlueType field, which was left at an inherited
"Balanced" from a prior cert. Fixed space_heating() to explicitly select
"Open" to match the lodged gov-API code. This field isn't consumed by
Sap10Calculator (ML/generator-only), so it had zero effect on the SAP
score.
Re-downloaded elmhurst_summary.pdf/elmhurst_worksheet.pdf after the fixes
(Recommendations page confirmed clean). Elmhurst-PDF-inputs path moved
46->47 SAP from the cylinder-volume correction; gov-API SAP (53) and
Elmhurst's own worksheet (51) are unchanged. Full accuracy suites re-run
clean (67 passed, 67 skipped, 1 xfailed, same as before).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Elmhurst rebuild for this cert had two carryover bugs from the shared
Playwright session: TextBoxOpenChimneys silently inherited "2" from a prior
cert's build (this cert's register lodges open_fireplaces_count=0, which
the gov-API mapper already reads correctly), and wall thickness inherited
a stale 280mm. Also fixed a real bug in the space_heating() PCDB-ref-clearing
logic: input_value() returns the string "0" when already cleared, which is
truthy in Python, causing a spurious navigate-away that broke the next
E.goto() call.
Rebuilt cleanly (chimneys=0, wall thickness=220mm per RdSAP convention for
unrecorded solid-wall thickness, since CheckBoxWallThicknessUnknown does not
persist via Playwright in this tool version). Elmhurst's own worksheet score
moves from a contaminated 46/48 to 51, closing the gap against this engine's
53-on-gov-API-inputs from ~5-7pt to ~2pt. No mapper.py change was needed --
the gov-API mapper's open_chimneys_count=0 was correct all along.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Built UPRN 100010086084 (the largest gov-API-vs-lodged outlier from the
20-cert batch) in Elmhurst's accredited RdSAP10 tool to get real ground
truth, per feedback that lodged rating isn't a valid comparison target.
Elmhurst worksheet: SAP 46-48 (orientation-dependent), vs this engine's
53 on gov-API inputs -- a real ~5-7pt residual gap. Ruled out with
direct empirical tests: total_floor_area, missing party_wall_length,
region/climate (SAP rating always uses UK-average weather per Appendix
U -- confirmed inert), boiler/secondary efficiency (exact match, 66%/
63% both sides). Confirmed real but NOT blind-fixed (need more than one
cert's evidence, and a wall-U-value patch tested the wrong direction in
isolation): a wall-thickness-unknown U-value fallback mismatch (this
engine 1.70, Elmhurst's own "unknown" resolution 1.40) and a window
U-value divergence (1.85 vs 2.52 W/m2K for the same nominal glazing
pick). Single-window orientation confirmed a ~2pt partial contributor.
Also fixes a small, separate, evidenced gap the build surfaced: the
Elmhurst summary parser didn't recognise the live tool's "Medium
(131-170L)" cylinder-size label (only the bare "Medium").
Full write-up is in the RealCertExpectation comment for this cert in
test_real_cert_sap_accuracy.py and the worklist. Residual gap is
intentionally left open for follow-up, not force-closed.
Built the property-753950 cert (worklist P4, PR #1503) in the live RdSAP-10
Online tool via Playwright: SAP 44 (E), vs our engine's 42 and the lodged
2014 cert's 45 — a normal RdSAP-2012->SAP-10.2 recalculation spread, not a
defect surfaced by the multiple_glazed_proportion fix.
Also surfaced (and documented as a caveat, not fixed here) a pre-existing
SAP-16.2 gap: the cert lodges party_wall_length_m=0 on both floors despite
being semi-detached, which Elmhurst's validation rejects and this PR's
engine change doesn't touch.
Address PR review (dancafc):
- introduce UprnMatch NamedTuple (datatypes/address_match.py) for the
(uprn, address, lexiscore, certificate_number) return, replacing the bare
4-tuple in get_uprn_from_epc_df / get_uprn_from_historic_epc /
HistoricEpcResolver.resolve_uprn. Tuple-compatible, so unpacking is unchanged.
- rename get_uprn_with_epc_df -> get_uprn_from_epc_df (+ callers).
- type resolve_group_ambiguity via a GroupDecision NamedTuple and trim its
docstring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The move in 3934e2d3 left the trigger-builder test and local smoke script
importing the deleted etl.hubspot.abri_flow_triggers module; point them at
HubspotDealDiffer.check_abri_triggers_and_construct_message so they collect
and run again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
30 pairs (28 deterministically scoreable) from the 2,000-postcode sweep,
frozen as ONE anonymised raw-payload JSON (pairs + cohorts + actuals — a
thousand per-cert files would drown the PR diff). The gate replays the
whole conditioning path offline — mapper, conditioning, selection,
synthesis, comparison — in ~9s; floors are the measured values, tighten-
only. comparable_from_payload is extracted from the corpus loader so both
fixture formats share one payload->ComparableProperty path; the builder
(build_expired_pairs_corpus.py) refreezes from the raw-JSON disk cache.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
roof_construction codes group by FORM (empirical: 1=Flat 98%, 4/5/8=
Pitched 88-99%, 3=dwelling-above 100% over 7,974 certs; 7/9=premises-
above per #1452), so the filter matches families — an exact-code filter
would wrongly drop pitched neighbours lodged as 5/8. Historic prefixes
map to the same families; roof rooms and thatch stay unconditioned.
Harness ladder replay and telemetry mirror the new filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JsonCachingEpcClient caches at the raw layer (cert payloads verbatim,
search rows as JSON) so warm re-runs replay through retry+mapper+parsing
in minutes AND the cache doubles as integration-test fixture material —
the ADR-0030 corpus pattern (anonymised raw JSON through the mapper),
which pickled domain objects could never be. Wired into the pairs
harness (--cache-dir). roof_construction_code_sweep pins the RdSAP roof
code table the basement-code-6 way: single-building-part certs only,
code x description-prefix purity table. Also fixes the tfa telemetry key
rename that left the agreement row empty.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Evidence (439-pair harness, PR #1466): historic-vs-new age band agrees
52% exactly but 90% within one band (assessors re-band, skewing newer);
TFA agrees 45% within ±5% but 82% within ±20%. Equality/±5% steered the
cohort toward stale values where they engaged and relaxed everywhere
else. Band definitions are public so the harness's ladder replay shares
one source of truth.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 2000-postcode sweep died at ~884 on a strict-raising RdSAP-17.1 cert
(missing multiple_glazed_proportion) inside the pair-check, losing the
whole run's output. get_by_uprn and the cohort fetch now skip-and-log per
item; telemetry rows append as produced and carry raw values (age band,
TFA, fuel text) so band-width questions are answerable post hoc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per pair, replay the age->fuel->TFA conditioning ladder over the plain
arm's cohort to record ENGAGED vs RELAXED per filter, and record whether
each historic stable attribute still agrees with the newly lodged cert
(the direct staleness measurement). Emitted as JSONL (--telemetry) and
aggregated into two diagnosis tables appended to the report.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Classification hit-rates for every compare_prediction component, all five
numeric residuals, and the secondary calculator-floored SAP residual
(calc(predicted) − lodged), plain vs conditioned side by side — the same
metric shape as the prediction-corpus gate (ADR-0030). Pair-check now
precedes the cohort fetch so a national postcode sweep only pays the
expensive search-by-postcode for shards that actually hold a pair.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Finds pre-2012 (S3 backup) x SAP-10.2 (new API) cert pairs per postcode,
predicts each from its leave-one-out cohort with and without ADR-0054
conditioning, and reports compare_prediction hit rates side by side.
Evidence for the stable-attribute whitelist, not a CI gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_u_brick_thin_wall_age_a_to_e's `<= 280` check put an uninsulated solid-
brick wall of exactly 280mm in the "200 to 280mm -> 1.7" row; RdSAP 10
§5.7 Table 13 (spec PDF p.41) shares 280 as an unlabelled edge between
that row and "280 to 420mm -> 1.4", so the table text alone doesn't say
which row owns it.
Found by building cert 100031768368 (280mm exactly, band C) in Elmhurst
and comparing every worksheet line to the calculator: volume/ACH/floor/
doors matched exactly, but the wall didn't (1.70 vs Elmhurst's 1.40)
despite an identical input crosswalk. Initially reverted a first attempt
at this fix when it appeared to regress an already-pinned cert
(217091901, band A, also nominally 280mm, previously "confirmed" at
U=1.70) — but that cert's build script never actually set a wall
thickness field, so its shared Elmhurst assessment had silently
inherited a stale 260mm from an earlier build. Fixed that build script
and rebuilt with the correct 280mm entry: Elmhurst's worksheet now also
gives U=1.40, matching 100031768368 and confirming the fix rather than
contradicting it.
Both certs move to (near-)exact lodged matches: 100031768368 59.12 ->
61.21 (lodged 61, within 0.5); 217091901 60.82 -> 61.59 (lodged 62,
exact). Corpus gauge 77.8% -> 78.6% within-0.5, MAE 0.636 -> 0.627 (14
corpus certs lodge exactly 280mm solid brick). TDD'd, 46 pins + full
suite green (2310 passed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The printed "candidate measures considered" now matches what the
orchestrator actually offered — on 711795 the lodged cavity wall printed
a cavity-fill candidate the plan (timber-frame effective wall) never saw.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
