`.claude/settings.json` was committed to main by mistake — it holds
per-developer permission allow-lists (npx cache paths, /tmp script paths,
and even hardcoded credentials), not shared project config. Mirror the
existing `.claude/settings.local.json` treatment: remove it from the index
and add it to .gitignore so each developer keeps their own local copy.
Claude Code merges settings.json + settings.local.json at runtime, so no
permissions are lost.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gov-API lodges secondary fuel as an enum whose value can COLLIDE with a
different same-valued RdSAP 10 Table 32 / SAP 10.2 Table 12 fuel code:
- enum 9 = "dual fuel (mineral and wood)" vs Table code 9 = LPG SC11F
- enum 5 = "anthracite" vs Table code 5 = LPG (bulk)
The main-fuel boundary already canonicalises these (`_GOV_API_COLLISION_
FUELS`), but the SECONDARY-heating cost + CO2/PE paths never did — they took
the bare same-value lookup, so a dual-fuel room heater was priced as LPG
(3.48 vs dual-fuel 3.99 p/kWh) and emitted as LPG (CO2 0.241 vs 0.087),
and an anthracite secondary as bulk LPG (12.19 vs 3.64 p/kWh). The price
under-count over-rates SAP; the CO2 over-count inflates emissions.
Fix: add enum 9 to `_GOV_API_COLLISION_FUELS` (5 and 33 were already there)
and canonicalise the secondary fuel code on both the cost
(`_secondary_fuel_cost_gbp_per_kwh`) and factor (`_secondary_fuel_code`)
paths, mirroring the main-fuel boundary. canonical_fuel_code only touches
{5,9,33}, so genuinely Table-coded secondaries (House coal 11, wood logs 20,
community fuels 30-32) are left unchanged — confirmed by a full-map audit.
Corpus: within-0.5 69.7% -> 70.2% (MAE 0.854 -> 0.845; dual-fuel-secondary
cohort 42.9% -> 49.0%, signed +0.55 -> +0.41) and CO2 MAE 0.12 -> 0.08 t/yr
(bias +0.04 -> 0.00). Ratcheted the corpus floors (within 0.70, MAE 0.85,
CO2 0.09, PE 4.0). A prior session deferred enum 9 ("direction not
understood") while the EPC PE/CO2 lens was confounded by the climate-cascade
bug (fc7c4d2d); on the corrected lens the over-rate direction is clear.
pyright not installed in this codespace (strict gate not run locally).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`profile_corpus_error.py` and `dive_cert.py` compared our PE/CO2 against
the lodged EPC figures using the UK-average RATING cascade, but the EPC
lodges CO2/PE on the postcode DEMAND cascade (SAP 10.2 Appendix U p.124,
now wired into Sap10Calculator.calculate in fc7c4d2d). That confounded the
DEMAND-vs-COST triage: a cert whose demand actually reproduced on local
weather looked "PE off" purely from the climate difference and was
mislabelled DEMAND-side. Switching the PE/CO2 lens to `cert_to_demand_
inputs` (SAP still from the rating cascade) re-classifies the corpus
outside-0.5 set 261/42 -> 211/92 DEMAND/COST — ~50 certs are genuinely
cost-side (e.g. 10091578598: SAP +7.81 but PE +1.6 / CO2 -0.04). Sharpens
the hunt for the subtle widespread SAP term.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SAP/EI rating is computed on UK-average weather (Appendix U Tables
U1-U3 region 0) so ratings are nationally comparable, but Appendix U
paragraph 1 (PDF p.124) requires that "other calculations (such as for
energy use and costs on EPCs) are done using local weather. Weather data
for each postcode district are taken from the PCDB". `Sap10Calculator.
calculate` ran ONE cascade (UK-average) and fed it to SAP, CO2 AND primary
energy, so every cert's EPC-displayed CO2/PE were computed on the wrong
climate. Because most of England is warmer than the UK-average, this
systematically OVER-counted heating demand on the emissions/PE outputs.
The two cascades (`cert_to_inputs` rating, `cert_to_demand_inputs`
postcode) already existed; this wires the demand cascade into the
production entry point and grafts its CO2/PE onto the rating result (SAP
unchanged). The corpus gauge's longstanding +5% CO2/PE over-estimate was
mostly this climate bug, NOT (as previously diagnosed) per-cert mapper
fidelity:
CO2 MAE 0.26 -> 0.12 t/yr (bias +0.18 -> +0.04)
PE MAE 13.6 -> 3.8 kWh/m2 (bias +9.0 -> +0.24)
SAP within-0.5 = 69.7% (rating cascade, unchanged)
Worksheet-validated to 1e-4 on simulated case 45 (heat-pump ground-floor
flat, postcode W6): the P960 prints the current dwelling twice — Block 1
on UK-average weather (SAP 60.5318, CO2 692.13) and Block 2 on postcode
weather (CO2 626.78, PE 6581.59). Both reproduce exactly. Added a tracked
case-45 Summary fixture + two-cascade cascade pin as a permanent guard,
and ratcheted the corpus CO2/PE ceilings to 0.13 / 4.2. The e2e Elmhurst
suite (Block-1 line refs) now pins the rating cascade directly; the two
Vaillant overlay snapshots refreshed to demand-cascade CO2/PE.
pyright not installed in this codespace (strict gate not run locally);
change is type-trivial (dataclasses.replace over SapResult).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
profile_corpus_error.py buckets signed SAP error by raw-API feature and
lists worst over/under-raters with the PE/CO2-vs-cost split (COST-side vs
DEMAND-side triage). dive_cert.py dumps one cert's lodged-vs-ours
SAP/CO2/PE + full intermediate line refs + mapped inputs. Both run on the
committed RdSAP-21.0.1 corpus (no /tmp sample needed). Used to find the
stone-wall, per-part-roof, ground-floor-flat and HP-water fixes this session.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a heat-pump cert lodges a PCDB Table 362 record, the APM override
set BOTH the space efficiency (N3.6) and the water efficiency (N3.7a)
from the heat pump unconditionally. But the PCDB η_water applies only
when the DHW is heated BY the heat pump (water-heating code "from main":
901/902/914). A separate electric immersion (WHC 903) heats the water at
100% regardless of the space system, so applying the HP's water SCOP
(187.5% × 0.6 in-use = 112.5%) under-counted the immersion's hot-water
fuel.
Gate the η_water override on the DHW-from-main codes; a separate immersion
keeps its own 100% efficiency. Space η_space still always uses the APM
value (the heat pump is the space main).
Worksheet-validated to 1e-4 on simulated case 45 (HP space + WHC-903
immersion): water fuel (62) 1893.57 -> 2130.2639, total cost (255)
619.7433, CO2 692.13 — all matching the P960 exactly; SAP 60.53 -> rounds
to the worksheet's 61. RdSAP-21.0.1 corpus unchanged (no HP+WHC903 certs
in it). Pinned in test_cert_to_inputs (immersion fuel is main-independent).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flat floor-exposure heuristic keys on dwelling_type: a flat defaults
to has_exposed_floor=False (assuming a heated dwelling below). The
Elmhurst Summary path lodges a ground-floor flat's vertical position as a
"Ground floor" floor_type rather than the API floor_heat_loss=1 exposed
code, and the mapper can label such a flat "Top-floor flat" — so the
cascade dropped the ground floor entirely (a ground floor is in contact
with the ground and carries heat loss).
Treat a "ground floor" floor_type as a heat-loss floor, overriding the
dwelling-level suppression upward — mirroring the existing "another
dwelling below" party override downward.
Worksheet-validated to 1e-4 on simulated case 45 (a ground-floor flat
the mapper labelled "Top-floor flat"): floor (28a) 0 -> 25.38 W/K,
fabric (33) 75.63 -> 101.0104, HTC (39) 112.93 -> 145.3579, all matching
the P960 exactly; SAP 67.81 -> 62.52. RdSAP-21.0.1 corpus within-0.5
69.5% -> 69.7% (MAE 0.859 -> 0.854). Floors ratcheted. Pinned in
test_heat_transmission (ground-floor billed + party-floor suppressed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds whole-dwelling property_type/built_form to EpcSimulation (folded by
apply_simulations) and maps those override components. property_type drives
party-wall heat loss + ASHP/solar/wall eligibility, so a landlord correction now
moves both the SAP calc and the measure menu; built_form has no calculator
consumer today (feeds the ML transform). Written as the landlord text value
(park-home check is text-only). Refines ADR-0032 dec-4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends WallType coverage to timber/stone/system-built/cob/park-home/curtain and
adds RoofType "Pitched, N mm loft insulation" -> roof_insulation_thickness. The
"(assumed) insulated"/"partial" wall states stay deferred (ambiguous code, needs
Elmhurst validation per ADR-0032); property_type/built_form carry no SAP weight.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SQLModel had drifted to a `bill_` prefix on the Bill Derivation block, but
the FE-owned Drizzle table uses unprefixed names (`heating_kwh`, `hot_water_kwh`
… `total_annual_bill_gbp`) plus a nullable `fuel_rates_period`. INSERTs failed
with UndefinedColumn. Rename the columns to mirror the live table column-for-
column (the prefix's anti-clash purpose is moot: `heating_kwh` != the recorded
`space_heating_kwh`), and add the `fuel_rates_period` column — left None until
Bill Derivation threads the snapshot period through.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Baseline stage is the first consumer to read these off a persisted EPC
end-to-end, surfacing three gaps that only manifest on real API data:
- Only the 21.0.1 mapper copied through the recorded current-performance
scalars (SAP rating, CO2, PEUI) and *no* mapper mapped the EPC band, so
Lodged Performance raised for 17.x/18.0/19.0/20.0.0 certs. Overlay all four
from the raw payload in `from_api_response`, once, for every schema version.
- Likewise the `renewable_heat_incentive` block (baseline space/water-heating
kWh) was only mapped by the 21.x paths. Gap-fill it centrally from the raw
payload when a mapper left it unset.
- The FE-owned `epc_property` date columns are Postgres `timestamp`s while the
SQLModel mirror types them `str`, so a read hands back a `datetime` and
`date.fromisoformat()` raised. Normalise via `_as_date()`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deduplicated `epc.roofs[]` list cannot be indexed 1:1 against the
building parts (190/329 multi-part certs have len(roofs) != len(parts)),
so every part's `u_roof` consumed a SINGLE join of all roof descriptions.
That leaked one part's insulation state onto another: a "Flat, no
insulation" extension dragged a "Pitched, insulated (assumed)" main roof
to the uninsulated 2.30, ~3x over-stating its heat loss. 3-part certs
systematically under-rated (56% within-0.5, mean -0.79 SAP).
Partition the non-RR roof descriptions into flat vs pitched/sloping and
match each part to its own kind (`_main_roof_descriptions_by_kind`),
falling back to the global join when a part's kind has no matching entry.
Corpus cert 100010129331: roof 110.5 -> 31.3 W/K, +13.10 -> -0.05 SAP.
RdSAP-21.0.1 within-0.5 68.8% -> 69.5% (MAE 0.888 -> 0.859; PE 13.9 ->
13.6); 3-part cohort 56% -> 61%. Floors/ceilings ratcheted. Pinned in
test_heat_transmission (by_kind split + mixed-roof no-contamination).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The §5.8 Table-14 added-insulation R-value adjustment was gated to
WALL_SOLID_BRICK, so a stone (granite/sandstone) wall lodging
wall_insulation_type 1/3 ("External"/"Internal") + a thickness fell
through the §5.6 thin-wall branch and was billed at its UNINSULATED U
(e.g. sandstone 520 mm + 100 mm internal: 1.64 instead of 0.30 → ~5×
the wall heat loss). Mirror the brick insulation branch into the stone
block, feeding the RAW §5.6 U₀ into the §5.8 chain per the same rule the
brick branch and the dry-lined granite pin 000565 already follow (the
Table-6 footnote (a) 1.7 cap does not apply on the insulated path).
Corpus cert 100052159386 (sandstone 520 mm + 100 mm internal): -26.20 ->
-4.08 SAP, walls 300 -> 55 W/K. RdSAP-21.0.1 corpus within-0.5 68.6% ->
68.8% (SAP MAE 0.942 -> 0.888; PE MAE 14.3 -> 13.9; CO2 0.27 -> 0.26);
floors/ceilings ratcheted. Unit-pinned in test_rdsap_uvalues.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/run_first_run_e2e.py runs the real Ingestion -> Baseline -> Modelling
pipeline against the DB by composing build_first_run_pipeline + dispatch_first_run
with the live source clients (the Lambda handler can't run locally — its
_source_clients_from_env still raises, #1136). Unlike run_modelling_e2e it runs
real ingestion (persists EPC/spatial/solar) and has no inspect-only mode, so it's
gated behind --confirm (preview otherwise); measure scoping comes only from the
Scenario's exclusions (the pipeline threads no --measures), and the modelling
batch is all-or-nothing, both documented.
Extract the shared env/engine/S3 plumbing into scripts/e2e_common.py (public
load_env/build_engine/s3_parquet_reader) so both runners share one source and
neither imports the other's privates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gov API lodges a NON-SEPARATED conservatory (conservatory_type=4) as a
glazed "building part" carrying only {floor_area, room_height,
double_glazed, glazed_perimeter} — no fabric, no floor dimensions. The
four fields were undeclared on the 21.0.1 SapBuildingPart, so `from_dict`
dropped them and the conservatory was silently lost: it billed no §6.1
window/rooflight/floor and added nothing to TFA (5 corpus certs over-rated
— too little heat loss → SAP too high).
Fix (21.0.1 schema + mapper):
- declare the four glazed fields on `SapBuildingPart`;
- `_api_sap_conservatory` builds `EpcPropertyData.sap_conservatory` from
the glazed BP (identified by a lodged `glazed_perimeter`; only type-4
conservatories lodge it — separated ones, §6.2, lodge nothing);
- exclude the glazed BP from the fabric building-part loop (it is billed
by the §6.1 cascade, not as a dwelling part);
- `_total_floor_area_from_building_parts` adds the conservatory floor area
to TFA (drives occupancy → §4/§5 demand).
Validation is cross-mapper parity, NOT a corpus back-solve: the API mapper
feeds the SAME worksheet-validated §6.1 cascade (`conservatory_geometry`,
pinned to 1e-4 against the case-44 Summary) as the Elmhurst path — so the
API conservatory fabric is correct by construction. `from_api_response`
on an injected type-4 cert reproduces the glazed wall (perimeter × ground-
floor room height = 22.05), glazed roof (floor/cos20 = 12.77) and Table 25
double U_eff (2.758 wall / 2.993 roof); a separated (type 2/3) cert lodges
no glazed BP → disregarded per §6.2.
Gauges: corpus within-0.5 67.9% → 68.6% (MAE 0.959 → 0.942; floor 0.67→0.68,
ceiling 0.97→0.95); /tmp eval mean|err| 0.822 → 0.817. Harness 47/47
0-raised; regression = the 3 pre-existing fails; pyright net-zero (65=65).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The new pipeline left no per-Property record of a run (the old engine set
property.has_recommendations and populated property_details_epc). Restore the
marker: PropertyRepository.mark_modelled sets has_recommendations (true when the
Plan carries measures, mirroring the old engine) and bumps updated_at, so a
first-run under the new process is identifiable as updated_at >= 2026-06-01.
ModellingOrchestrator marks each Property after its Scenarios (true if any
Scenario yielded a measure); run_modelling_e2e's --persist path marks it too
(its compute runs on in-memory fakes, so the DB UoW sets it directly). Adds the
has_recommendations/updated_at columns to the PropertyRow mirror.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Close the §6.1 conservatory demand cascade per RdSAP 10 §6.1 + Table 25.
Solar gains (§6, solar_gains.py) — Table 25 note (PDF p.51): "The
orientation of windows in a conservatory is not recorded, thus solar
gains are calculated using the default solar flux (East/West orientation,
with 20° pitch for roof windows)." The glazed wall bills onto the (76)
East line (vertical, average-overshading Z); the glazed roof onto the
(82) roof-window line (20° pitch, Z=1.0), both at Table 25 g=0.76, FF=0.70.
TFA-occupancy (mapper) — §6.1: the conservatory floor area is added to the
dwelling total floor area. TFA drives occupancy → §5 internal gains + §4
hot-water demand, so the non-separated conservatory's floor area now
enters `EpcPropertyData.total_floor_area_m2` (the worksheet's (4) = 95.38
carries it). Separated conservatories (§6.2) stay excluded.
Pinned against the case-44 P960 demand cascade at abs=1e-4: (73) internal
gains 625.1759, (83) solar gains 495.8655, (95) useful gains 1079.6510,
(99) space heating per m² 89.8073 — the full §6.1 chain reproduces EXACTLY.
The whole-dwelling SAP (72.9517) / CO2 (3241.8656) are not pinned: the
case-44 Summary omits the House-Coal secondary heater (SAP 633) the P960
descriptor carries (cf. case 43), so the cascade computes no secondary —
the entire residual (+349.77 kg CO2). A Summary-input defect, independent
of §6.1; every conservatory-affected line ref is exact. Worksheet harness
stays 47/47 0-raised; corpus unchanged (API path; mirror is the next slice).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SAP 10.2 §2 (17)-(18): a measured/design air permeability at 50 Pa from a
Blower Door test routes infiltration via `(18) = AP50/20 + (8)`, in
preference to the components-based (16) estimate. The Elmhurst extractor
read only the AP4 ("Pulse") column of §12.2, so a Blower Door result
(§12.2 "Pressure Test Result (AP50)") fell through to the structural-
infiltration default — over-counting ventilation heat loss.
Surfaced by simulated case 44 (AP50 4.50): effective air change rate was
0.81 vs the worksheet's 0.58 (+38% ventilation loss). The cascade already
supports `air_permeability_ap50` (preferred over AP4); this wires the read
end to end (extractor → ElmhurstSiteNotes → SapVentilation → cert_to_inputs).
Pinned against the case-44 P960 §2 at abs=1e-4: (18) infiltration 0.3417
(= 4.5/20 + 0.1167) and (25) Jan effective ach 0.5812. Worksheet harness
stays 47/47 0-raised.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every worklist UPRN now carries schema · engine SAP / lodged · flag. Tally:
64 healthy, 19 MVHR-not-credited (🚩 flag B), 6 heat-pump fuel-39 (🚩 flag A),
4 sparse/NOT MAPPABLE (⛔), 3 Elmhurst-pinned. MVHR is the largest accuracy gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Autonomous-run triage of the moderate eng-vs-lodged gaps resolves them into two
patterns, both flagged for owner review (not auto-fixable):
- Heat-pump fuel code 39 mis-priced as gas (over-rates; both gap directions).
- MVHR heat recovery modelled as plain extract loss → systematic UNDER-rating
(~8-12 SAP) on every full-SAP cert carrying a mechanical_vent_system_index_number.
New memory mvhr-heat-recovery-not-modelled; needs the Appendix Q / PCDB MVHR
efficiency model.
findings doc updated with the classification.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Some SAP-Schema-17.x/18.0.0 certs lodge sap_openings width/height in MILLIMETRES
mixed with metre rows in the same array (e.g. a 2025x2100 mm window beside a
3.06x1 m one). The 17.1 mapper read them all as metres → a 4.25M m2 window →
HTC in the millions → SAP clamped to 1.
Fix (TDD, datatypes/epc/domain/mapper.py): _sanitise_opening_dimension_m treats
any dimension > 50 m as mm and divides by 1000; _sap_opening_area_m2 applies it
to areas. Wired into the window, roof-window, and door-area-weighting paths.
The 3 broken certs (uprn_10093117227 / 10090317693 / 10091636031) now score
90 / 81 / 79 instead of 1.
3 RED->GREEN slices + refactor; new test class
TestFromSapSchema17_1OpeningUnitSanitisation + sap_17_1_mm_openings.json fixture;
0 new pyright errors; no regressions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
