#1672 (which includes #1666's main_heating_fraction normalisation at the mapper)
is merged into main. The decimal-fraction cohort now maps to integer percent
before save, so the INTEGER column no longer truncates and all 1000 corpus certs
round-trip their SAP to 1e-9. Remove the temporary _is_pre_1666_decimal_fraction
skip so the gate asserts a hard zero over the whole corpus.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The accuracy corpus scores straight off the mapped fixture and never saves, so it
is structurally blind to any field the calculator reads but the schema drops
(#1665). This gate maps -> saves to the ephemeral test DB -> reads back ->
re-scores every corpus cert and asserts the reloaded SAP equals the in-memory SAP
to 1e-9. It would have caught cylinder_heat_loss / room-in-roof / roof-windows /
wall_u_value on day one; a new drop now fails here immediately.
Save+read+rollback per cert keeps the graph off disk (bounded, ~18s over 1000
certs). One documented, precise skip: the pre-#1666 decimal main_heating_fraction
cohort, whose INTEGER column truncates the raw decimal on save (0.8 -> 1) — fixed
upstream by #1666 (mapper normalisation, on #1672). Remove the skip once #1672 is
in main so the gate asserts a hard zero.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SapBuildingPart.wall_u_value (the gov-EPC lodged wall U, authoritative over the
derived default) had no column and dropped on save, so the reloaded dwelling fell
back to the age-band default U — -2.32 SAP on the 2 affected RdSAP corpus certs.
Surfaced by the new DB-round-trip gate. The prior _UNPERSISTED_ALLOWLIST
justification ("deliberate — full-SAP only; RdSAP re-model uses default U-values")
was stale: the cascade DOES honour the lodged U on the API path.
Add wall_u_value to EpcBuildingPartModel, write in from_domain, reconstruct in
_to_building_part, drop the allowlist entry. Verified: certs 39036600 / 38137278
round-trip 72.43 / 69.11 exactly. Needs an added FE column
(epc_building_part.wall_u_value, double precision NULL) on assessment-model#443.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SapRoomInRoof.detailed_surfaces (+ the gable/common-wall scalars) had no columns,
so 107 corpus certs lost their whole §3.9/§3.10 room-in-roof geometry on the DB
round-trip and fell back to the Simplified all-elements age-band default —
worst case -15.21 SAP (#1665 / #1664).
Add EpcRoomInRoofModel (0..1 per building part, unique FK) + EpcRoomInRoofSurface
Model (0..n, surface_index order). Save nests via RETURNING (building part -> RIR
-> surfaces); delete clears bottom-up (surfaces -> room_in_roof -> parts) since
the FE FKs are ON DELETE no action; both read paths group + reconstruct in a new
_to_room_in_roof / _to_rir_surface, replacing the two-field flat rebuild in
_to_building_part. The flat room_in_roof_* columns on epc_building_part are
superseded and dropped in a follow-up (#1664). Nullable is preserved on
insulation_thickness_mm / u_value (null vs 0 is the Table 17-vs-18 branch); area
is unconstrained (a §3.9.2 absent-gable adjustment is signed). Drop the 8
_UNPERSISTED_ALLOWLIST entries. Verified: worst RIR cert 10012119141 round-trip
-15.21 -> 0.0000 with detailed_surfaces at deep equality.
Companion FE migration: assessment-model#443 (epc_room_in_roof +
epc_room_in_roof_surface). Deploy gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EpcPropertyData.sap_roof_windows (List[SapRoofWindow]) had no table, so 55
corpus certs lost their rooflights on the DB round-trip and were re-scored
without them (§3 (27a) heat transmission + §6 solar gain). Add EpcRoofWindowModel
(mirrors EpcPhotovoltaicArrayModel) with a roof_window_index order column and the
resolved SapRoofWindow fields; wire save (batch insert), delete (both delete
paths), both read paths (get + get_many) and _compose reconstruction; drop the
two _UNPERSISTED_ALLOWLIST entries.
u_value_raw / g_perpendicular / frame_factor are stored resolved, not re-derived:
the Table 24 lookup is keyed on glazing_type AND glazing_gap, and glazing_gap is
not retained on SapRoofWindow, so they can't be reconstructed from the persisted
fields. window_location is JSONB (int on API / str on site-notes). Verified: a
roof-window cert round-trips sap_roof_windows to deep equality.
Companion FE migration: assessment-model#443 (epc_roof_window). Deploy gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SapAlternativeWall.u_value (heat_transmission.py:1616), .wall_thickness_mm
(:1645) and .is_basement (:1618) were read by the calculator but had no column,
so they dropped on save. Latent today (0 gov-API certs lodge them — live only on
the Elmhurst/site-notes path), but is_basement is the nastiest shape: it does not
null out, it FLIPS meaning (True -> None -> is_basement_wall reads False),
silently switching off the RdSAP §5.17 / Table 23 basement-wall U path.
Add alt_wall_{1,2}_{u_value,thickness_mm,is_basement} to EpcBuildingPartModel,
write in from_domain, reconstruct in _to_alt_wall, and drop the three
_UNPERSISTED_ALLOWLIST entries so the structural guard enforces them. is_basement
is nullable (None "not stated" != False "not a basement"; a NOT NULL DEFAULT
false re-enables the code-6 heuristic — same reasoning as #1661's wall_is_basement).
Companion assessment-model migration: six nullable columns on epc_building_part
(deploy gate). Round-trip + field-coverage guard green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SapHeating.cylinder_heat_loss is read by the calculator (cert_to_inputs §4
cylinder loss) but had no column, so it was silently dropped on save: a dwelling
reloaded from Postgres lost its lodged cylinder loss and fell back to the Table
2b age-band default — worst case -15.53 SAP over the corpus round-trip (#1665).
Add heating_cylinder_heat_loss (nullable double precision) to EpcPropertyModel,
write it in from_domain and reconstruct it in the SapHeating compose path,
mirroring heating_cylinder_volume_measured_l. Remove the now-false
_UNPERSISTED_ALLOWLIST entry ("dormant — not read by the calculator") so the
ADR structural guard enforces it. Verified: worst cylinder cert 10091630692
round-trip -15.53 -> 0.0000.
Companion assessment-model migration (heating_cylinder_heat_loss column) must
land + be applied per-environment before the backend deploys (deploy gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ADR-0030 component-accuracy and expired-pairs gates re-derive the *actual*
EpcPropertyData through the same mapper the leave-one-out scorer predicts against.
#1669 (measurement_type=2 external->internal conversion, RdSAP §3.4 + Table 2)
gives the fixture's external certs sharper internal dimensions, which tips the
geo-proximity-weighted donor mode on three components by single-target amounts:
total_window_area 3.7184 -> 3.7484, door_count 0.3333 -> 0.3737, has_pv floor
0.8929 -> 0.8571 (one pair 25/28 -> 24/28).
This is a ground-truth-method change, not a prediction-logic loosening: the fix
is spec-correct, corpus MAE improved (0.571 -> 0.570) and the real-cert accuracy
pin held. Per the gate convention the affected floors/ceilings are re-baselined
to the measured values with the attribution recorded; tighten-only resumes here.
Bisected: #1666/#1667 leave the gates green; #1669 introduces all three shifts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous commit staged these two files at `git mv` time but never
staged the edits that followed, so they landed with their pre-refactor
content: `class PashubService` and `from backend.pashub_fetcher...`
imports that no longer resolve. Local runs passed because the working
tree was correct; CI collected the committed tree and failed with
ModuleNotFoundError: No module named 'backend.pashub_fetcher'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The service was the last one living wholly under backend/. It now follows
the same layering as abri and the other newer services:
domain/pashub_fetcher/ core file classification, subfolders
infrastructure/pashub_fetcher/ PasHub client, token getter, wire DTOs
orchestration/ PashubFetcherOrchestrator (was PashubService)
applications/pashub_fetcher/ lambda handler, trigger request, dev tooling
core_files.py is split along the layer boundary: the domain module keeps the
filename/evidence-category classification rules and no longer imports
infrastructure.postgres, while the CoreFiles -> FileTypeEnum translation moves
to infrastructure/pashub_fetcher/core_file_types.py.
Tests move into the tests/ tree by layer. Note this puts them in the only
suite CI currently runs (unit_tests.yml is disabled), so these 73 tests now
execute on PRs for the first time; they were previously reachable only via
the legacy pytest.ini testpaths.
sharepoint_renamer's image now copies just domain/pashub_fetcher/ rather than
the whole service, since SharepointSubfolders is all it needed.
Behaviour is unchanged. tests/ goes 9927 -> 10000 passed (+73, exactly the
tests that moved in); the legacy suite keeps its same 17 pre-existing failures
and 11 errors. tests/test_lambda_packaging.py confirms both changed
Dockerfiles still copy their handler's full import closure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The four calculator-accuracy fixes stack to move the RdSAP-21.0.1 corpus from
within-0.5 80.3% / MAE 0.584 to within-0.5 81.2% / MAE 0.565082. Ratchet the pins
(repo convention: tighten, never loosen): _MIN_WITHIN_HALF_SAP 0.803 -> 0.812
(measured floor), _MAX_SAP_MAE 0.584 -> 0.566 (MAE rounded up to 3 d.p. so the
`<=` assert holds).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wet underfloor responsiveness was pinned at R=0.75 for the lumped gov-EPC emitter
code 2, regardless of floor construction. SAP 10.2 Table 4d splits it three ways
and RdSAP 10 Table 29 (p.56-57) selects from floor construction + age band:
timber floor 1.0, screed above insulation 0.75, concrete slab 0.25. A
concrete-slab system (solid floor, age A-E) was billed as screed, over-stating
responsiveness (higher MIT) and over-rating the dwelling; R=0.25 was unreachable.
_responsiveness now takes the cert and, for the lumped underfloor code, derives
the Table 29 subtype via _underfloor_responsiveness from the main part's floor
construction + age band: suspended TIMBER -> 1.0; solid A-E -> 0.25 (concrete
slab); solid F-M / suspended-not-timber / unknown / no ground floor -> 0.75
(screed, the prior default, so ambiguous no-ground-floor certs do not move). The
mapper stays a pass-through.
Spec: SAP 10.2 Table 4d (p.170), RdSAP 10 Table 29 (p.56-57). Corpus: MAE
0.570 -> 0.565 (net toward lodged), within-0.5 81.2%. Cert 100100137438 (solid,
band B) 72 -> 69, exactly lodged. Closes#1668 on the API path.
NOTE: the issue's worksheet adjudication (run 100100137438 through accredited
Elmhurst to confirm concrete-slab R=0.25) is still outstanding — it corrects to
lodged 69 exactly, but is not yet Elmhurst-worksheet-confirmed. The Elmhurst
site-notes path (_resolve_elmhurst_underfloor_subtype still raises for solid A-E)
is untouched — no corpus/fixture exercises it; separate follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A cert lodging its horizontal dimensions measured externally
(measurement_type=2) must have its floor area and exposed perimeter converted to
internal dimensions before any SAP use (RdSAP 10 §3.4 + Table 2, p.17-18).
measurement_type was read nowhere (the `# What is this?` field), so external
certs carried a TFA ~11-15% too large — inflating occupancy and understating
every per-m2 output (CO2/m2, PE/m2).
Plumb measurement_type onto EpcPropertyData, and add a post-mapping
_with_internal_dimensions step: compute ONE whole-dwelling area/perimeter ratio
from the ground-floor footprint via the Table 2 built-form equation (§3.4 Note 4
— not a per-storey conversion, which over-corrects multi-part mid-terraces),
apply it to every storey's area/perimeter, reduce party walls by 2w (Note 5),
and refresh total_floor_area_m2 (read for occupancy N). Wall thickness w is the
lodged main-dwelling value, else the Table 3 default (p.19). Room-in-roof floor
area is always internal (§3.1) and left untouched. Scoped to the RdSAP-Schema
reduced-data family (17.0-21.x) where per-storey external dims are the lodgement
standard; legacy SAP-Schema-15.0/16.x are plumbed but not converted.
Corpus: within-0.5 81.0% -> 81.2%, MAE 0.571 -> 0.570, PE 2.5 -> 2.4. Cert
100091435353 TFA 128.7 -> 111.6 (lodged 112), SAP 66.23 -> 65.33 (+1.2 -> +0.3);
mean |dSAP vs lodged| over the 12 external certs 0.556 -> 0.429. Closes#1669.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On the gov-EPC API path a window's metal frame was unrepresentable: both
transmission tables were frameless (U_pvc, g, 0.70), so a cert lodging a metal
frame (pvc_frame:"false", RdSAP input 6-5) was billed at the PVC/wooden U and
frame factor 0.70 — over-rating metal-framed dwellings.
Add the RdSAP 10 Table 24 metal U-column (PDF p.50-51; mirrors u_window's metal
values) and a pvc_frame dimension to _api_glazing_transmission: an explicit
"false" routes the window to the metal U + SAP 10.2 Table 6c frame factor 0.8;
"true"/None/other keep the PVC column + 0.70, so no PVC cert regresses.
Secondary glazing keeps its frame-independent U (only FF changes). Wired into
_api_sap_window and _api_sap_roof_window (both full-schema, lodge pvc_frame);
the reduced-field path has no pvc_frame and is unchanged.
Spec: RdSAP 10 Table 24 metal column, SAP 10.2 Table 6c, RdSAP input 6-5 (p.80).
Corpus: within-0.5 80.6% -> 81.0%, MAE 0.578 -> 0.571. Cert 21067296
68.60 -> 66.96 (toward lodged 67). Closes#1667.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gov-EPC API lodges a two-main dwelling's main_heating_fraction pair in
two incompatible encodings — a decimal fraction summing to ~1.0 ([0.8, 0.2])
or an integer percent summing to ~100 ([80, 20]) — while every consumer
divides by 100 unconditionally. A decimal pair therefore collapsed the second
main's share to ~1% of its true value, billing almost the whole load to the
first system's fuel/efficiency.
Normalise once at the mapper via a shared pair-sum discriminator
(_normalised_main_heating_fraction): a genuine two-main split summing nearer
1.0 than 100 is scaled to percent; a percent pair and any single main are left
exactly as lodged, so the 3 percent-encoded corpus certs do not regress. Wired
across the seven from_rdsap_schema_* branches and the SAP-17.1 path, whose old
int(fraction) floor silently deleted a decimal-lodged second main (int(0.35)=0).
Spec: RdSAP 10 item 7-5 (percent is canonical), p.81. Corpus: within-0.5
80.3% -> 80.6%, MAE 0.584 -> 0.578. Example cert 10070086972 17.59 -> 18.73
(toward lodged 18). Closes#1666.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`plan.name` is a freeform, user-entered label that is unset on
modelling-generated Plans (0/205 populated on a real portfolio). The meaningful
"name of the plan chosen" is the Scenario the home's default Plan came from, so
`plan_name` now reads `scenario.name` via `plan.scenario_id` (LEFT JOIN scenario)
on both export selections. On the default-plan export this labels each home with
its source scenario (e.g. "Fabric Only" / "ASHP + PV"); on the scenario export it
equals that sheet's scenario.
Repository test seeds scenarios and asserts the source-scenario name; 62 tests
pass, pyright --strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A second export selection alongside the per-Scenario one, chosen by
`plan_selection` on the export request:
- "scenario" (default, unchanged): one sheet per Scenario, freshest Plan per
(property, scenario); scenario_ids required.
- "default" (new): a single "Default Plans" sheet of each home's is_default
Plan (one-per-property across all Scenarios, ADR-0012/0017) — the portfolio's
current state, one row per home; scenario_ids ignored.
Repository gains `default_rows_for(portfolio_id, property_ids)`: same read-model
and Effective-EPC join as `rows_for`, with the plan-selection WHERE swapped from
`scenario_id = :scenario_id` to `is_default = TRUE`. Orchestrator branches on
plan_selection; the router validates it and only requires scenario_ids for the
scenario selection. A recipe without plan_selection defaults to "scenario", so
the change is backward-compatible.
Both selections now surface the chosen Plan's `name` as a `plan_name` column.
Tests: repository default_rows_for (scenario-independent selection, plan_name,
non-default excluded); orchestrator default path (one "Default Plans" sheet;
empty selection is a recorded failure). 62 passed, pyright --strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 item 2-3(h) "another dwelling above" is U=0 (spec p.45: "There is no
heat loss through the roof of a building part that has the same dwelling above
or another dwelling above"). gov-API `roof_construction = 3` is that code, but
the mapper labelled it "Pitched (slates/tiles), no access to loft" (which is
code 5), so the per-part party override at heat_transmission.py:1172 never
matched and the part fell through to the dwelling_type-label-only
has_exposed_roof heuristic.
190 of the 200 corpus code-3 parts sit at index 0 and are already suppressed by
`_cert_lodges_exposed_roof` off the roof_insulation_location == "ND" signal. The
unmitigated set is the 10 parts at index > 0, where extensions default to
exposed and were billed a phantom pitched roof. Also removes a latent PV defect:
the bogus "Pitched" prefix satisfied `is_pitched` and divided PV array area by
cos(35 deg), inflating kWp.
Shipped with the stairwell fix so the corpus pin moves once: within-0.5
78.8% -> 79.6%, SAP MAE 0.626 -> 0.611, PE MAE 2.9 -> 2.7.
The roof change alone trades 4 certs out of the +-0.5 band for 2 in (one moving
4.4 SAP closer). Those 4 were investigated and are NOT a masked bug: the corpus
error distribution is symmetric (mean +0.076, median +0.03, sigma 1.606; 112
certs above +0.5 vs 102 below), ground-floor flats show 7 positive / 8 negative
with no excess over the null, and closing them needs HLC changes of 2.3-5.6% —
below the ~8.8% noise floor of the RHI anchor, calibrated on the 76 certs whose
SAP is exact (space-heating ratio 1.062 +/- 0.088).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §5.9 "U-values of sheltered walls" (PDF p.43): a wall between a
flat/maisonette and an unheated buffer takes SAP 10.2 §3.3's U = 1/(1/U0 + Ru),
and Ru depends on the BUFFER TYPE — "Use Ru = 0.5 m²K/W for corridors and
Ru = 2.1 m²K/W for stairwell."
The buffer type is lodged on `SapFlatDetails.heat_loss_corridor` (0-based per
item 1-11, PDF p.70: 0 none, 1 heated, 2 unheated corridor, 3 unheated
stairwell). That field was read NOWHERE under domain/ — corridor presence was
inferred only from a lodged sheltered alternative wall — so every sheltered wall
took the corridor 0.5 and every stairwell flat over-billed its sheltered wall,
under-rating the dwelling.
Distinct from `_SHELTERED_GABLE_ADDED_RESISTANCE_M2K_W`, which is RdSAP 10
Table 4 (p.22) for a room-in-roof sheltered gable — same 0.5, different rule,
deliberately untouched.
Corpus gauge ratcheted UP: within-0.5 78.8% -> 79.8% (+10 certs), SAP MAE
0.626 -> 0.613, PE MAE 2.9 -> 2.8. Failure set and pyright count identical to
main. Portfolio 824 is stairwell-heavy (60 properties at code 3); property
749898 over-bills 4.44 W/K on a 48 m2 flat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §3.12 (PDF p.26), the flats/maisonettes floor rules: "There is no heat
loss through the floor if there is another flat below. Otherwise the floor area
of the flat ... is ... a semi-exposed floor if there are unheated premises below
it (e.g. an enclosed garage) ... Semi-exposed (sheltered) floors are treated as
if they were fully exposed." §5.13 (p.48) confirms the U-value identity: Table 20
applies to exposed and semi-exposed alike.
API floor_heat_loss=2 -> floor_type "To unheated space" set no exposure signal,
so on a mid-/top-floor flat the dwelling-level heuristic (has_exposed_floor=False,
assuming a heated dwelling below) went unopposed and the ENTIRE floor was billed
at 0 W/K. 11 properties across portfolios 796+824, mean +4.02 SAP, 10 of 11
over-rating — we judged dwellings more efficient than the accredited assessment
and would withhold retrofit measures they need.
Exemplars: 750018 space heating -53.2% -> -8.1% vs the cert's own RHI figure,
SAP +8.22 -> +1.16; 749898 -43.0% -> +0.9%, SAP +6.29 -> -1.11.
SCOPED TO FLATS ON PURPOSE. §3.12 is the flats/maisonettes section, and the
accredited Elmhurst worksheet disagrees with Table 20 for a house: golden cert
7536-3827-0600-0600-0276 (detached) lodges floor_heat_loss=2 on its Main part and
the worksheet bills it on the BS EN ISO 13370 ground-floor cascade at U 0.97, not
Table 20's 1.20. An unscoped mapper-level fix broke that pin and the
uprn_100050881708 accuracy pin; the scoped fix leaves both untouched.
No regressions: identical failure set and pyright count vs main; the RdSAP-21.0.1
accuracy corpus gauge is unchanged at 78.8% within-0.5 / MAE 0.625 (correctly —
all 7 corpus certs lodging code 2 are houses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §5.17 / Table 23. `wall_is_basement` is not a lodged U-value — it is a
disambiguation flag selecting WHICH RdSAP default applies, so unlike
wall/roof/floor_u_value it must survive the round-trip.
Code 6 is canonically SYSTEM-BUILT and the gov-API path infers "basement" from a
code-6 heuristic; the Elmhurst site-notes mapper sets wall_is_basement=False
precisely to defeat it. Dropping the field inverted the answer — a system-built
wall came back as a basement wall, dragging the whole ground floor onto the
Table 23 basement-floor U-value via `has_basement`.
Declared on no RdSAP schema, so it never arrives via the gov API; the earlier
corpus scan showing 0 occurrences measured the wrong population. The Elmhurst
site-notes path sets it and those EPCs are persisted
(scripts/wchg_elmhurst_ingest.py).
Nullable is load-bearing: None = "not stated, use the heuristic", False =
"explicitly system-built, do not".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
