Corpus validation of the modelling_e2e built_form fix. Cert 8742-6624-9300-2780-4926
(SAP-Schema-16.0, ground-floor electric-storage-heater flat) omits built_form; the
mapper now derives it from dwelling_type. built_form is ML-only so the fix is
SAP-neutral: engine 66 = lodged 66 exactly. Built in Elmhurst (evidence: epc.json +
summary + worksheet): worksheet 54, engine-on-Elmhurst-inputs 53 ≈ 54 → calculator
faithful. The +12 engine-vs-Elmhurst is a build/input gap (cert size-1 small cylinder
unrepresentable in Elmhurst's Normal/110L-minimum entry → higher HW + reduced-field
16.0 defaults). Pinned engine 66.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the mapper-coverage gaps surfaced by the modelling_e2e prediction-cohort
failures (portfolio 796):
- built_form (SAP-16.0): derive from dwelling_type in _normalize_sap_schema_16_x
(Mid-terrace->4, End-terrace->3, Semi-detached->2, Detached->1; flats->modal 4).
ML-only field (SAP calc never reads it) so SAP- and gate-neutral. 5 flat certs
that omitted built_form now map.
- photovoltaic_supply as a measured-array LIST: routed all pre-21 RdSAP mappers
(17.0/17.1/18.0/19.0/20.0.0) through _map_schema_21_pv, whose list branch is now
dict-tolerant (_pv_array_field reads dict OR dataclass). They capture the PV
arrays like 21.0.x instead of raising "'list' object has no attribute
none_or_no_details" and sinking the whole cohort.
- windows-as-dict (16.x): handled in the normalizer (not just windows-as-list).
Genuinely-sparse certs (omit door_count/habitable/glazed_area) remain fail-loud;
the gate-regressing multiple_glazed_proportion default and the recursive
RdSAP-21.0.0 ADR-0028 alignment are left fail-loud + flagged for review (worklist).
+5 regression tests; component-accuracy gate 26/26; 0 new pyright errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- run_modelling_e2e --from-db re-models from already-persisted inputs (reads
each Property's Effective EPC + planning protections + solar from the DB) and
skips every live fetcher — zero gov-API calls. With --persist it re-writes the
Plan and, for lodged-EPC Properties, the Baseline. Self-contained loop; the
live-fetch path is untouched. Makes local re-runs instant and avoids tripping
the gov API's per-IP rate limit (6000 req / 5 min) during iteration.
- EpcClientService.REQUEST_TIMEOUT 10s -> 30s: a cold per-UPRN search can exceed
10s and the old timeout turned it into a timeout-then-retry; 30s rides it out.
Note: an open perf question remains — modelling is fast in isolation (<0.5s/
property) but a long-lived --persist run shows ~1 min/property; suspected in the
persist path (plan.save / baseline) or connection handling, NOT the API. Left
mid-diagnosis for handover.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make run_modelling_e2e the single script that does everything for a portfolio,
so the 291-property run needs one invocation with per-property recovery (no
all-or-nothing chunking):
- On --persist, a lodged-EPC Property now also gets its Baseline Performance
row written via PropertyBaselineOrchestrator (per Property, so one bad cert
does not abort the batch). Predicted (EPC-less) Properties have no lodged
figures, so they get a Plan but no baseline row.
- The run CSV gains api_sap (register) vs baseline_sap (calculator) + sap_delta,
so calculator-vs-API divergence is reviewable per property.
Fill the off-catalogue overlay for the measures the live material catalogue
cannot price, so they stop crashing the run:
- double_glazing (£550/window) and secondary_glazing (£400/window): priced
per window (the generator multiplies by single-glazed window count, matching
the legacy window_glazing). Grounded in 2025/26 UK installed costs; per-window
is the right unit for windows (fixed per-unit install dominates) — per-m2 fits
walls/floors, not glazing.
- gas_boiler_upgrade / system_tune_up / system_tune_up_zoned: these are priced
off the heating rate sheet (Products()), with get() reading the catalogue only
for an id — so the overlay entry exists to satisfy that lookup (material_id
stays None, as with ASHP); the rate sheet remains authoritative.
Validated on a 12-property sample (incl. a secondary-glazing case and a
SAP-Schema-16.2 cert): 12/12 baseline rows + plans, 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Table 5 reads "Number of extract fans if known; if number is unknown:
[age-band default]" — the default is an UNKNOWN-fallback, NOT a floor. The
cascade applied `max(lodged, table_5_default)`, flooring a genuinely-lodged
count up to the age-band minimum: e.g. an age H-M dwelling lodging 2 extract
fans was billed at the 6-8-room default of 3, over-counting ventilation line
(8) and the heat-loss coefficient. Fixed to `lodged if lodged > 0 else
default` (a lodged 0 is the Elmhurst/RdSAP "unknown" form → default; any
positive count is taken literally).
Surfaced by Khalim's Elmhurst stress worksheet (simulated case 46): this was
its last ventilation residual — our Jan effective ACH 9.14 -> 9.0748 (exact
match to the accredited worksheet), SAP 29 -> 30 = Elmhurst, cost £1496 vs
£1493. Corpus IMPROVED: within-0.5 71.6% -> 72.5%, MAE 0.819 -> 0.815 (the
max-flooring over-counted ventilation on every cert lodging fans below its
age default). Floor ratcheted 0.71 -> 0.72. pyright not installed locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the main heating system does not heat every habitable room (heated rooms
< habitable rooms), SAP 10.2 Appendix A.2.2 assumes the unheated rooms are
served by a portable-electric secondary heater, so the Table 11 secondary
fraction (0.10 for a boiler main) must be costed at the electricity tariff —
even when the cert lodges no explicit secondary system.
`_secondary_fraction` previously returned 0 unless a secondary was lodged or
the main was a forced-secondary electric-storage code, dropping the assumed
secondary and billing 100% of space heat to the (cheaper) main fuel — an
over-rate. Added an `unheated_habitable_rooms` trigger plus
`_has_unheated_habitable_rooms(epc)`, which prefers the lodged
`any_unheated_rooms` flag and guards the gov-API `heated_rooms_count == 0`
"not provided" sentinel. The secondary fuel/efficiency cascade already
defaults to portable electric (code 693) when no secondary code is lodged.
Worksheet-validated on simulated case 46 (heated 4 < habitable 7, no lodged
secondary): the assumed 10% electric secondary (2289 kWh, ~£260) lifted our
SAP 39 -> 29.35 vs accredited Elmhurst 30 (cost £1502 vs £1493, within 0.6%).
Corpus UNCHANGED (71.6% / MAE 0.819): all 17 corpus certs with heated <
habitable already lodge an explicit secondary description, so the gov-API
path was already costing it; this only adds the assumed secondary where none
is lodged (Elmhurst / reduced-field path). pyright not installed locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The full RdSAP-Schema-21 label is "Double between 2002 and 2021" (double
glazing installed 2002-2021, SAP 10.2 Table 24 code 3). When the Elmhurst
Summary PDF wraps the trailing "and 2021" into an adjacent table cell that
the extractor joins away, the surfaced label truncates to "Double between
2002" — the same artifact already handled for "Triple post or during".
`_elmhurst_glazing_type_code` raised UnmappedElmhurstLabel on it, blocking
the whole Summary (surfaced on the simulated-case-46 multi-attribute
worksheet). Added the truncated form as a code-3 alias.
pyright not installed in this container.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`u_window` hard-coded the 12 mm gap row for pre-2002 double/triple glazing
(double 2.8, triple 2.1), ignoring the lodged glazing gap. Table 24 splits
the pre-2002 rows by gap: double 6mm=3.1 / 12mm=2.8 / 16mm+=2.7; triple
6mm=2.4 / 12mm=2.1 / 16mm+=2.0 (PVC/wooden), with a metal-frame column
(+0.5/+0.5/+0.5 ish). Added a `glazing_gap` parameter + `_glazing_gap_row`
helper and wired `w.glazing_gap` through the synthesised-window caller in
heat_transmission.
Corpus impact nil by design: the gov-API mapper already resolves per-window
U gap-aware via `_API_GLAZING_TYPE_GAP_TO_TRANSMISSION` (e.g. code 3 + gap
"16+" → 2.7), so corpus certs use that lodged per-window U, not this fallback.
This aligns the reduced-field / worksheet fallback path with the mapper and
Table 24. Unknown gap still defaults to the 12 mm row.
(Metal frames are not distinguishable on the gov-API path — only a `pvc_frame`
boolean exists and Table 24 groups PVC+wooden — so the PVC/wooden U stands
there; the metal column applies only where frame material is lodged.)
Spec-pinned: pre-2002 double + triple gap-row tests. pyright not installed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
