The ADR-0030 component-accuracy and expired-pairs gates re-derive the *actual*
EpcPropertyData through the same mapper the leave-one-out scorer predicts against.
#1669 (measurement_type=2 external->internal conversion, RdSAP §3.4 + Table 2)
gives the fixture's external certs sharper internal dimensions, which tips the
geo-proximity-weighted donor mode on three components by single-target amounts:
total_window_area 3.7184 -> 3.7484, door_count 0.3333 -> 0.3737, has_pv floor
0.8929 -> 0.8571 (one pair 25/28 -> 24/28).
This is a ground-truth-method change, not a prediction-logic loosening: the fix
is spec-correct, corpus MAE improved (0.571 -> 0.570) and the real-cert accuracy
pin held. Per the gate convention the affected floors/ceilings are re-baselined
to the measured values with the attribution recorded; tighten-only resumes here.
Bisected: #1666/#1667 leave the gates green; #1669 introduces all three shifts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous commit staged these two files at `git mv` time but never
staged the edits that followed, so they landed with their pre-refactor
content: `class PashubService` and `from backend.pashub_fetcher...`
imports that no longer resolve. Local runs passed because the working
tree was correct; CI collected the committed tree and failed with
ModuleNotFoundError: No module named 'backend.pashub_fetcher'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The service was the last one living wholly under backend/. It now follows
the same layering as abri and the other newer services:
domain/pashub_fetcher/ core file classification, subfolders
infrastructure/pashub_fetcher/ PasHub client, token getter, wire DTOs
orchestration/ PashubFetcherOrchestrator (was PashubService)
applications/pashub_fetcher/ lambda handler, trigger request, dev tooling
core_files.py is split along the layer boundary: the domain module keeps the
filename/evidence-category classification rules and no longer imports
infrastructure.postgres, while the CoreFiles -> FileTypeEnum translation moves
to infrastructure/pashub_fetcher/core_file_types.py.
Tests move into the tests/ tree by layer. Note this puts them in the only
suite CI currently runs (unit_tests.yml is disabled), so these 73 tests now
execute on PRs for the first time; they were previously reachable only via
the legacy pytest.ini testpaths.
sharepoint_renamer's image now copies just domain/pashub_fetcher/ rather than
the whole service, since SharepointSubfolders is all it needed.
Behaviour is unchanged. tests/ goes 9927 -> 10000 passed (+73, exactly the
tests that moved in); the legacy suite keeps its same 17 pre-existing failures
and 11 errors. tests/test_lambda_packaging.py confirms both changed
Dockerfiles still copy their handler's full import closure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The four calculator-accuracy fixes stack to move the RdSAP-21.0.1 corpus from
within-0.5 80.3% / MAE 0.584 to within-0.5 81.2% / MAE 0.565082. Ratchet the pins
(repo convention: tighten, never loosen): _MIN_WITHIN_HALF_SAP 0.803 -> 0.812
(measured floor), _MAX_SAP_MAE 0.584 -> 0.566 (MAE rounded up to 3 d.p. so the
`<=` assert holds).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wet underfloor responsiveness was pinned at R=0.75 for the lumped gov-EPC emitter
code 2, regardless of floor construction. SAP 10.2 Table 4d splits it three ways
and RdSAP 10 Table 29 (p.56-57) selects from floor construction + age band:
timber floor 1.0, screed above insulation 0.75, concrete slab 0.25. A
concrete-slab system (solid floor, age A-E) was billed as screed, over-stating
responsiveness (higher MIT) and over-rating the dwelling; R=0.25 was unreachable.
_responsiveness now takes the cert and, for the lumped underfloor code, derives
the Table 29 subtype via _underfloor_responsiveness from the main part's floor
construction + age band: suspended TIMBER -> 1.0; solid A-E -> 0.25 (concrete
slab); solid F-M / suspended-not-timber / unknown / no ground floor -> 0.75
(screed, the prior default, so ambiguous no-ground-floor certs do not move). The
mapper stays a pass-through.
Spec: SAP 10.2 Table 4d (p.170), RdSAP 10 Table 29 (p.56-57). Corpus: MAE
0.570 -> 0.565 (net toward lodged), within-0.5 81.2%. Cert 100100137438 (solid,
band B) 72 -> 69, exactly lodged. Closes#1668 on the API path.
NOTE: the issue's worksheet adjudication (run 100100137438 through accredited
Elmhurst to confirm concrete-slab R=0.25) is still outstanding — it corrects to
lodged 69 exactly, but is not yet Elmhurst-worksheet-confirmed. The Elmhurst
site-notes path (_resolve_elmhurst_underfloor_subtype still raises for solid A-E)
is untouched — no corpus/fixture exercises it; separate follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A cert lodging its horizontal dimensions measured externally
(measurement_type=2) must have its floor area and exposed perimeter converted to
internal dimensions before any SAP use (RdSAP 10 §3.4 + Table 2, p.17-18).
measurement_type was read nowhere (the `# What is this?` field), so external
certs carried a TFA ~11-15% too large — inflating occupancy and understating
every per-m2 output (CO2/m2, PE/m2).
Plumb measurement_type onto EpcPropertyData, and add a post-mapping
_with_internal_dimensions step: compute ONE whole-dwelling area/perimeter ratio
from the ground-floor footprint via the Table 2 built-form equation (§3.4 Note 4
— not a per-storey conversion, which over-corrects multi-part mid-terraces),
apply it to every storey's area/perimeter, reduce party walls by 2w (Note 5),
and refresh total_floor_area_m2 (read for occupancy N). Wall thickness w is the
lodged main-dwelling value, else the Table 3 default (p.19). Room-in-roof floor
area is always internal (§3.1) and left untouched. Scoped to the RdSAP-Schema
reduced-data family (17.0-21.x) where per-storey external dims are the lodgement
standard; legacy SAP-Schema-15.0/16.x are plumbed but not converted.
Corpus: within-0.5 81.0% -> 81.2%, MAE 0.571 -> 0.570, PE 2.5 -> 2.4. Cert
100091435353 TFA 128.7 -> 111.6 (lodged 112), SAP 66.23 -> 65.33 (+1.2 -> +0.3);
mean |dSAP vs lodged| over the 12 external certs 0.556 -> 0.429. Closes#1669.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On the gov-EPC API path a window's metal frame was unrepresentable: both
transmission tables were frameless (U_pvc, g, 0.70), so a cert lodging a metal
frame (pvc_frame:"false", RdSAP input 6-5) was billed at the PVC/wooden U and
frame factor 0.70 — over-rating metal-framed dwellings.
Add the RdSAP 10 Table 24 metal U-column (PDF p.50-51; mirrors u_window's metal
values) and a pvc_frame dimension to _api_glazing_transmission: an explicit
"false" routes the window to the metal U + SAP 10.2 Table 6c frame factor 0.8;
"true"/None/other keep the PVC column + 0.70, so no PVC cert regresses.
Secondary glazing keeps its frame-independent U (only FF changes). Wired into
_api_sap_window and _api_sap_roof_window (both full-schema, lodge pvc_frame);
the reduced-field path has no pvc_frame and is unchanged.
Spec: RdSAP 10 Table 24 metal column, SAP 10.2 Table 6c, RdSAP input 6-5 (p.80).
Corpus: within-0.5 80.6% -> 81.0%, MAE 0.578 -> 0.571. Cert 21067296
68.60 -> 66.96 (toward lodged 67). Closes#1667.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gov-EPC API lodges a two-main dwelling's main_heating_fraction pair in
two incompatible encodings — a decimal fraction summing to ~1.0 ([0.8, 0.2])
or an integer percent summing to ~100 ([80, 20]) — while every consumer
divides by 100 unconditionally. A decimal pair therefore collapsed the second
main's share to ~1% of its true value, billing almost the whole load to the
first system's fuel/efficiency.
Normalise once at the mapper via a shared pair-sum discriminator
(_normalised_main_heating_fraction): a genuine two-main split summing nearer
1.0 than 100 is scaled to percent; a percent pair and any single main are left
exactly as lodged, so the 3 percent-encoded corpus certs do not regress. Wired
across the seven from_rdsap_schema_* branches and the SAP-17.1 path, whose old
int(fraction) floor silently deleted a decimal-lodged second main (int(0.35)=0).
Spec: RdSAP 10 item 7-5 (percent is canonical), p.81. Corpus: within-0.5
80.3% -> 80.6%, MAE 0.584 -> 0.578. Example cert 10070086972 17.59 -> 18.73
(toward lodged 18). Closes#1666.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`plan.name` is a freeform, user-entered label that is unset on
modelling-generated Plans (0/205 populated on a real portfolio). The meaningful
"name of the plan chosen" is the Scenario the home's default Plan came from, so
`plan_name` now reads `scenario.name` via `plan.scenario_id` (LEFT JOIN scenario)
on both export selections. On the default-plan export this labels each home with
its source scenario (e.g. "Fabric Only" / "ASHP + PV"); on the scenario export it
equals that sheet's scenario.
Repository test seeds scenarios and asserts the source-scenario name; 62 tests
pass, pyright --strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A second export selection alongside the per-Scenario one, chosen by
`plan_selection` on the export request:
- "scenario" (default, unchanged): one sheet per Scenario, freshest Plan per
(property, scenario); scenario_ids required.
- "default" (new): a single "Default Plans" sheet of each home's is_default
Plan (one-per-property across all Scenarios, ADR-0012/0017) — the portfolio's
current state, one row per home; scenario_ids ignored.
Repository gains `default_rows_for(portfolio_id, property_ids)`: same read-model
and Effective-EPC join as `rows_for`, with the plan-selection WHERE swapped from
`scenario_id = :scenario_id` to `is_default = TRUE`. Orchestrator branches on
plan_selection; the router validates it and only requires scenario_ids for the
scenario selection. A recipe without plan_selection defaults to "scenario", so
the change is backward-compatible.
Both selections now surface the chosen Plan's `name` as a `plan_name` column.
Tests: repository default_rows_for (scenario-independent selection, plan_name,
non-default excluded); orchestrator default path (one "Default Plans" sheet;
empty selection is a recorded failure). 62 passed, pyright --strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`SapRoomInRoof.detailed_surfaces` is read by the SAP calculator
(heat_transmission.py:590, :690-696, :1311-1470) but has no Postgres column, so
the per-surface geometry is silently discarded on save. Without it the cascade
abandons the RdSAP 10 §3.10 "Detailed" route — each surface at its own Table 17 /
Table 4 U-value — and bills the whole notional roof shell at the Table 18
column (4) age-band default, typically 2.30 W/m2K for pre-1976 stock.
Measured on the committed 1000-cert RdSAP-21.0.1 corpus: 108 certs (10.8%) lodge
RIR geometry across 118 building parts; 107 of 108 score differently when it is
dropped; 15 change EPC band; worst case -15.21 SAP (uprn 10012119141, top-floor
maisonette, band E -> F). Larger in both population and magnitude than the
rafter_insulation_thickness gap that opened #1656.
Doc only — the Drizzle tables are owned by Hestia-Homes/assessment-model and the
model-side wiring follows once they exist.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 item 2-3(h) "another dwelling above" is U=0 (spec p.45: "There is no
heat loss through the roof of a building part that has the same dwelling above
or another dwelling above"). gov-API `roof_construction = 3` is that code, but
the mapper labelled it "Pitched (slates/tiles), no access to loft" (which is
code 5), so the per-part party override at heat_transmission.py:1172 never
matched and the part fell through to the dwelling_type-label-only
has_exposed_roof heuristic.
190 of the 200 corpus code-3 parts sit at index 0 and are already suppressed by
`_cert_lodges_exposed_roof` off the roof_insulation_location == "ND" signal. The
unmitigated set is the 10 parts at index > 0, where extensions default to
exposed and were billed a phantom pitched roof. Also removes a latent PV defect:
the bogus "Pitched" prefix satisfied `is_pitched` and divided PV array area by
cos(35 deg), inflating kWp.
Shipped with the stairwell fix so the corpus pin moves once: within-0.5
78.8% -> 79.6%, SAP MAE 0.626 -> 0.611, PE MAE 2.9 -> 2.7.
The roof change alone trades 4 certs out of the +-0.5 band for 2 in (one moving
4.4 SAP closer). Those 4 were investigated and are NOT a masked bug: the corpus
error distribution is symmetric (mean +0.076, median +0.03, sigma 1.606; 112
certs above +0.5 vs 102 below), ground-floor flats show 7 positive / 8 negative
with no excess over the null, and closing them needs HLC changes of 2.3-5.6% —
below the ~8.8% noise floor of the RHI anchor, calibrated on the 76 certs whose
SAP is exact (space-heating ratio 1.062 +/- 0.088).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §5.9 "U-values of sheltered walls" (PDF p.43): a wall between a
flat/maisonette and an unheated buffer takes SAP 10.2 §3.3's U = 1/(1/U0 + Ru),
and Ru depends on the BUFFER TYPE — "Use Ru = 0.5 m²K/W for corridors and
Ru = 2.1 m²K/W for stairwell."
The buffer type is lodged on `SapFlatDetails.heat_loss_corridor` (0-based per
item 1-11, PDF p.70: 0 none, 1 heated, 2 unheated corridor, 3 unheated
stairwell). That field was read NOWHERE under domain/ — corridor presence was
inferred only from a lodged sheltered alternative wall — so every sheltered wall
took the corridor 0.5 and every stairwell flat over-billed its sheltered wall,
under-rating the dwelling.
Distinct from `_SHELTERED_GABLE_ADDED_RESISTANCE_M2K_W`, which is RdSAP 10
Table 4 (p.22) for a room-in-roof sheltered gable — same 0.5, different rule,
deliberately untouched.
Corpus gauge ratcheted UP: within-0.5 78.8% -> 79.8% (+10 certs), SAP MAE
0.626 -> 0.613, PE MAE 2.9 -> 2.8. Failure set and pyright count identical to
main. Portfolio 824 is stairwell-heavy (60 properties at code 3); property
749898 over-bills 4.44 W/K on a 48 m2 flat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §3.12 (PDF p.26), the flats/maisonettes floor rules: "There is no heat
loss through the floor if there is another flat below. Otherwise the floor area
of the flat ... is ... a semi-exposed floor if there are unheated premises below
it (e.g. an enclosed garage) ... Semi-exposed (sheltered) floors are treated as
if they were fully exposed." §5.13 (p.48) confirms the U-value identity: Table 20
applies to exposed and semi-exposed alike.
API floor_heat_loss=2 -> floor_type "To unheated space" set no exposure signal,
so on a mid-/top-floor flat the dwelling-level heuristic (has_exposed_floor=False,
assuming a heated dwelling below) went unopposed and the ENTIRE floor was billed
at 0 W/K. 11 properties across portfolios 796+824, mean +4.02 SAP, 10 of 11
over-rating — we judged dwellings more efficient than the accredited assessment
and would withhold retrofit measures they need.
Exemplars: 750018 space heating -53.2% -> -8.1% vs the cert's own RHI figure,
SAP +8.22 -> +1.16; 749898 -43.0% -> +0.9%, SAP +6.29 -> -1.11.
SCOPED TO FLATS ON PURPOSE. §3.12 is the flats/maisonettes section, and the
accredited Elmhurst worksheet disagrees with Table 20 for a house: golden cert
7536-3827-0600-0600-0276 (detached) lodges floor_heat_loss=2 on its Main part and
the worksheet bills it on the BS EN ISO 13370 ground-floor cascade at U 0.97, not
Table 20's 1.20. An unscoped mapper-level fix broke that pin and the
uprn_100050881708 accuracy pin; the scoped fix leaves both untouched.
No regressions: identical failure set and pyright count vs main; the RdSAP-21.0.1
accuracy corpus gauge is unchanged at 78.8% within-0.5 / MAE 0.625 (correctly —
all 7 corpus certs lodging code 2 are houses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §5.17 / Table 23. `wall_is_basement` is not a lodged U-value — it is a
disambiguation flag selecting WHICH RdSAP default applies, so unlike
wall/roof/floor_u_value it must survive the round-trip.
Code 6 is canonically SYSTEM-BUILT and the gov-API path infers "basement" from a
code-6 heuristic; the Elmhurst site-notes mapper sets wall_is_basement=False
precisely to defeat it. Dropping the field inverted the answer — a system-built
wall came back as a basement wall, dragging the whole ground floor onto the
Table 23 basement-floor U-value via `has_basement`.
Declared on no RdSAP schema, so it never arrives via the gov API; the earlier
corpus scan showing 0 occurrences measured the wrong population. The Elmhurst
site-notes path sets it and those EPCs are persisted
(scripts/wchg_elmhurst_ingest.py).
Nullable is load-bearing: None = "not stated, use the heuristic", False =
"explicitly system-built, do not".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add `rafter_insulation_thickness` to `EpcBuildingPartModel` (JSONB, matching the
sibling thickness columns so Union[str, int] survives) and wire it through
`from_domain` / `_to_building_part`.
Property 749719 lodges "250mm" -> Table 16 col (2) U 0.23. Dropped, the roof
billed at U 2.30 — a 10x error worth -9.8 SAP (lodged 73, calculated 63.2) that
moved the dwelling a whole band, C -> D, and drew a 9.63-point loft-insulation
recommendation on a roof rated 4/5 by the assessor.
The sibling wall/roof/floor_u_value and wall_is_basement fields have the same
drop but are deliberately NOT persisted: they are full-SAP artefacts and we
re-model as RdSAP, which uses default U-values. Their allow-list entries are
re-justified rather than removed — the previous "not read by the calculator" was
stale and is what hid this drop.
Requires the Drizzle migration in Hestia-Homes/assessment-model to land FIRST;
without the column, EPC reads fail outright.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §5.11.2 — a roof insulated at rafters lodges its thickness in
`rafter_insulation_thickness`, not `roof_insulation_thickness`. The field had no
DB column, so it was mapped from the gov-EPC API and silently dropped on save;
every at-rafters roof then fell back to the Table 18 col (2) unknown-thickness
default (2.30 W/m²K for bands A-D — the spec's *uninsulated* value).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
0390's floor+ASHP package reaches post 68.772, which publishes as 69 = band C,
so under ADR-0064 it now meets the goal and stops before low-energy lighting —
the same over-purchase the ADR removes, one level up at the package. 0380 fires
the same three measures with identical trigger attributes, so every assertion
stands unchanged, and it lands at post 74.4, clear of a band floor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The modelling_e2e handler models one property per run_modelling call, so a
per-property "no persisted baseline" line would be one per property across a
whole re-model. The harness has no Baseline stage by design and cannot drift
from one, so it now establishes its own the way the Baseline orchestrator
would; the Postgres-backed lane is where the assertion has something to catch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-0065 is claimed by Scenario Export on the open #1546; this branch is the
later of the two, so it moves.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
