Compose the party-ceiling and sloping-ceiling guards in the roof
GuardedColumnClassifier so a PitchedWithSlopingCeiling override resolves to the
Pitched, sloping ceiling family instead of being routed onto Pitched, N mm loft
insulation by the LLM (ADR-0066).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Since #1665 landed, the full room-in-roof geometry is stored in and reconstructed
from the epc_room_in_roof / epc_room_in_roof_surface child tables; the two flat
columns on epc_building_part (room_in_roof_floor_area,
room_in_roof_construction_age_band) are redundant duplicates that are written but
never read. Remove the two SQLModel fields and their from_domain writes so the
backend stops populating them — the required step BEFORE the FE migration drops
the columns (dropping them while a backend that still writes them is live would
500 every save).
Reconstruction already reads the child table (_to_room_in_roof), so the
round-trip gate stays a hard zero. The columns remain in the DB (nullable,
unwritten) until the FE drop migration runs.
Deploy AFTER #1674 is stable in every environment; the FE column drop runs after
this deploys everywhere.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#1672 (which includes #1666's main_heating_fraction normalisation at the mapper)
is merged into main. The decimal-fraction cohort now maps to integer percent
before save, so the INTEGER column no longer truncates and all 1000 corpus certs
round-trip their SAP to 1e-9. Remove the temporary _is_pre_1666_decimal_fraction
skip so the gate asserts a hard zero over the whole corpus.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The accuracy corpus scores straight off the mapped fixture and never saves, so it
is structurally blind to any field the calculator reads but the schema drops
(#1665). This gate maps -> saves to the ephemeral test DB -> reads back ->
re-scores every corpus cert and asserts the reloaded SAP equals the in-memory SAP
to 1e-9. It would have caught cylinder_heat_loss / room-in-roof / roof-windows /
wall_u_value on day one; a new drop now fails here immediately.
Save+read+rollback per cert keeps the graph off disk (bounded, ~18s over 1000
certs). One documented, precise skip: the pre-#1666 decimal main_heating_fraction
cohort, whose INTEGER column truncates the raw decimal on save (0.8 -> 1) — fixed
upstream by #1666 (mapper normalisation, on #1672). Remove the skip once #1672 is
in main so the gate asserts a hard zero.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SapBuildingPart.wall_u_value (the gov-EPC lodged wall U, authoritative over the
derived default) had no column and dropped on save, so the reloaded dwelling fell
back to the age-band default U — -2.32 SAP on the 2 affected RdSAP corpus certs.
Surfaced by the new DB-round-trip gate. The prior _UNPERSISTED_ALLOWLIST
justification ("deliberate — full-SAP only; RdSAP re-model uses default U-values")
was stale: the cascade DOES honour the lodged U on the API path.
Add wall_u_value to EpcBuildingPartModel, write in from_domain, reconstruct in
_to_building_part, drop the allowlist entry. Verified: certs 39036600 / 38137278
round-trip 72.43 / 69.11 exactly. Needs an added FE column
(epc_building_part.wall_u_value, double precision NULL) on assessment-model#443.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SapRoomInRoof.detailed_surfaces (+ the gable/common-wall scalars) had no columns,
so 107 corpus certs lost their whole §3.9/§3.10 room-in-roof geometry on the DB
round-trip and fell back to the Simplified all-elements age-band default —
worst case -15.21 SAP (#1665 / #1664).
Add EpcRoomInRoofModel (0..1 per building part, unique FK) + EpcRoomInRoofSurface
Model (0..n, surface_index order). Save nests via RETURNING (building part -> RIR
-> surfaces); delete clears bottom-up (surfaces -> room_in_roof -> parts) since
the FE FKs are ON DELETE no action; both read paths group + reconstruct in a new
_to_room_in_roof / _to_rir_surface, replacing the two-field flat rebuild in
_to_building_part. The flat room_in_roof_* columns on epc_building_part are
superseded and dropped in a follow-up (#1664). Nullable is preserved on
insulation_thickness_mm / u_value (null vs 0 is the Table 17-vs-18 branch); area
is unconstrained (a §3.9.2 absent-gable adjustment is signed). Drop the 8
_UNPERSISTED_ALLOWLIST entries. Verified: worst RIR cert 10012119141 round-trip
-15.21 -> 0.0000 with detailed_surfaces at deep equality.
Companion FE migration: assessment-model#443 (epc_room_in_roof +
epc_room_in_roof_surface). Deploy gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EpcPropertyData.sap_roof_windows (List[SapRoofWindow]) had no table, so 55
corpus certs lost their rooflights on the DB round-trip and were re-scored
without them (§3 (27a) heat transmission + §6 solar gain). Add EpcRoofWindowModel
(mirrors EpcPhotovoltaicArrayModel) with a roof_window_index order column and the
resolved SapRoofWindow fields; wire save (batch insert), delete (both delete
paths), both read paths (get + get_many) and _compose reconstruction; drop the
two _UNPERSISTED_ALLOWLIST entries.
u_value_raw / g_perpendicular / frame_factor are stored resolved, not re-derived:
the Table 24 lookup is keyed on glazing_type AND glazing_gap, and glazing_gap is
not retained on SapRoofWindow, so they can't be reconstructed from the persisted
fields. window_location is JSONB (int on API / str on site-notes). Verified: a
roof-window cert round-trips sap_roof_windows to deep equality.
Companion FE migration: assessment-model#443 (epc_roof_window). Deploy gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SapAlternativeWall.u_value (heat_transmission.py:1616), .wall_thickness_mm
(:1645) and .is_basement (:1618) were read by the calculator but had no column,
so they dropped on save. Latent today (0 gov-API certs lodge them — live only on
the Elmhurst/site-notes path), but is_basement is the nastiest shape: it does not
null out, it FLIPS meaning (True -> None -> is_basement_wall reads False),
silently switching off the RdSAP §5.17 / Table 23 basement-wall U path.
Add alt_wall_{1,2}_{u_value,thickness_mm,is_basement} to EpcBuildingPartModel,
write in from_domain, reconstruct in _to_alt_wall, and drop the three
_UNPERSISTED_ALLOWLIST entries so the structural guard enforces them. is_basement
is nullable (None "not stated" != False "not a basement"; a NOT NULL DEFAULT
false re-enables the code-6 heuristic — same reasoning as #1661's wall_is_basement).
Companion assessment-model migration: six nullable columns on epc_building_part
(deploy gate). Round-trip + field-coverage guard green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EpcPropertyModel.uprn was `Optional[int]`, which SQLModel maps to INTEGER, so a
real 11/12-digit UPRN (e.g. 10000452538) overflowed int32 and hard-failed the
insert with NumericValueOutOfRange — a failure on EVERY real-cert save, masked
because the round-trip fixtures use small fake UPRNs. The FE `property.uprn` is
already bigint; pin the column to BigInteger so ours matches (backend-only, no
migration). Verified: uprn 10000452538 saves + reloads.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SapHeating.cylinder_heat_loss is read by the calculator (cert_to_inputs §4
cylinder loss) but had no column, so it was silently dropped on save: a dwelling
reloaded from Postgres lost its lodged cylinder loss and fell back to the Table
2b age-band default — worst case -15.53 SAP over the corpus round-trip (#1665).
Add heating_cylinder_heat_loss (nullable double precision) to EpcPropertyModel,
write it in from_domain and reconstruct it in the SapHeating compose path,
mirroring heating_cylinder_volume_measured_l. Remove the now-false
_UNPERSISTED_ALLOWLIST entry ("dormant — not read by the calculator") so the
ADR structural guard enforces it. Verified: worst cylinder cert 10091630692
round-trip -15.53 -> 0.0000.
Companion assessment-model migration (heating_cylinder_heat_loss column) must
land + be applied per-environment before the backend deploys (deploy gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ADR-0030 component-accuracy and expired-pairs gates re-derive the *actual*
EpcPropertyData through the same mapper the leave-one-out scorer predicts against.
#1669 (measurement_type=2 external->internal conversion, RdSAP §3.4 + Table 2)
gives the fixture's external certs sharper internal dimensions, which tips the
geo-proximity-weighted donor mode on three components by single-target amounts:
total_window_area 3.7184 -> 3.7484, door_count 0.3333 -> 0.3737, has_pv floor
0.8929 -> 0.8571 (one pair 25/28 -> 24/28).
This is a ground-truth-method change, not a prediction-logic loosening: the fix
is spec-correct, corpus MAE improved (0.571 -> 0.570) and the real-cert accuracy
pin held. Per the gate convention the affected floors/ceilings are re-baselined
to the measured values with the attribution recorded; tighten-only resumes here.
Bisected: #1666/#1667 leave the gates green; #1669 introduces all three shifts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous commit staged these two files at `git mv` time but never
staged the edits that followed, so they landed with their pre-refactor
content: `class PashubService` and `from backend.pashub_fetcher...`
imports that no longer resolve. Local runs passed because the working
tree was correct; CI collected the committed tree and failed with
ModuleNotFoundError: No module named 'backend.pashub_fetcher'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The service was the last one living wholly under backend/. It now follows
the same layering as abri and the other newer services:
domain/pashub_fetcher/ core file classification, subfolders
infrastructure/pashub_fetcher/ PasHub client, token getter, wire DTOs
orchestration/ PashubFetcherOrchestrator (was PashubService)
applications/pashub_fetcher/ lambda handler, trigger request, dev tooling
core_files.py is split along the layer boundary: the domain module keeps the
filename/evidence-category classification rules and no longer imports
infrastructure.postgres, while the CoreFiles -> FileTypeEnum translation moves
to infrastructure/pashub_fetcher/core_file_types.py.
Tests move into the tests/ tree by layer. Note this puts them in the only
suite CI currently runs (unit_tests.yml is disabled), so these 73 tests now
execute on PRs for the first time; they were previously reachable only via
the legacy pytest.ini testpaths.
sharepoint_renamer's image now copies just domain/pashub_fetcher/ rather than
the whole service, since SharepointSubfolders is all it needed.
Behaviour is unchanged. tests/ goes 9927 -> 10000 passed (+73, exactly the
tests that moved in); the legacy suite keeps its same 17 pre-existing failures
and 11 errors. tests/test_lambda_packaging.py confirms both changed
Dockerfiles still copy their handler's full import closure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The four calculator-accuracy fixes stack to move the RdSAP-21.0.1 corpus from
within-0.5 80.3% / MAE 0.584 to within-0.5 81.2% / MAE 0.565082. Ratchet the pins
(repo convention: tighten, never loosen): _MIN_WITHIN_HALF_SAP 0.803 -> 0.812
(measured floor), _MAX_SAP_MAE 0.584 -> 0.566 (MAE rounded up to 3 d.p. so the
`<=` assert holds).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wet underfloor responsiveness was pinned at R=0.75 for the lumped gov-EPC emitter
code 2, regardless of floor construction. SAP 10.2 Table 4d splits it three ways
and RdSAP 10 Table 29 (p.56-57) selects from floor construction + age band:
timber floor 1.0, screed above insulation 0.75, concrete slab 0.25. A
concrete-slab system (solid floor, age A-E) was billed as screed, over-stating
responsiveness (higher MIT) and over-rating the dwelling; R=0.25 was unreachable.
_responsiveness now takes the cert and, for the lumped underfloor code, derives
the Table 29 subtype via _underfloor_responsiveness from the main part's floor
construction + age band: suspended TIMBER -> 1.0; solid A-E -> 0.25 (concrete
slab); solid F-M / suspended-not-timber / unknown / no ground floor -> 0.75
(screed, the prior default, so ambiguous no-ground-floor certs do not move). The
mapper stays a pass-through.
Spec: SAP 10.2 Table 4d (p.170), RdSAP 10 Table 29 (p.56-57). Corpus: MAE
0.570 -> 0.565 (net toward lodged), within-0.5 81.2%. Cert 100100137438 (solid,
band B) 72 -> 69, exactly lodged. Closes#1668 on the API path.
NOTE: the issue's worksheet adjudication (run 100100137438 through accredited
Elmhurst to confirm concrete-slab R=0.25) is still outstanding — it corrects to
lodged 69 exactly, but is not yet Elmhurst-worksheet-confirmed. The Elmhurst
site-notes path (_resolve_elmhurst_underfloor_subtype still raises for solid A-E)
is untouched — no corpus/fixture exercises it; separate follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A cert lodging its horizontal dimensions measured externally
(measurement_type=2) must have its floor area and exposed perimeter converted to
internal dimensions before any SAP use (RdSAP 10 §3.4 + Table 2, p.17-18).
measurement_type was read nowhere (the `# What is this?` field), so external
certs carried a TFA ~11-15% too large — inflating occupancy and understating
every per-m2 output (CO2/m2, PE/m2).
Plumb measurement_type onto EpcPropertyData, and add a post-mapping
_with_internal_dimensions step: compute ONE whole-dwelling area/perimeter ratio
from the ground-floor footprint via the Table 2 built-form equation (§3.4 Note 4
— not a per-storey conversion, which over-corrects multi-part mid-terraces),
apply it to every storey's area/perimeter, reduce party walls by 2w (Note 5),
and refresh total_floor_area_m2 (read for occupancy N). Wall thickness w is the
lodged main-dwelling value, else the Table 3 default (p.19). Room-in-roof floor
area is always internal (§3.1) and left untouched. Scoped to the RdSAP-Schema
reduced-data family (17.0-21.x) where per-storey external dims are the lodgement
standard; legacy SAP-Schema-15.0/16.x are plumbed but not converted.
Corpus: within-0.5 81.0% -> 81.2%, MAE 0.571 -> 0.570, PE 2.5 -> 2.4. Cert
100091435353 TFA 128.7 -> 111.6 (lodged 112), SAP 66.23 -> 65.33 (+1.2 -> +0.3);
mean |dSAP vs lodged| over the 12 external certs 0.556 -> 0.429. Closes#1669.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
