The ADR-0030 component-accuracy and expired-pairs gates re-derive the *actual*
EpcPropertyData through the same mapper the leave-one-out scorer predicts against.
#1669 (measurement_type=2 external->internal conversion, RdSAP §3.4 + Table 2)
gives the fixture's external certs sharper internal dimensions, which tips the
geo-proximity-weighted donor mode on three components by single-target amounts:
total_window_area 3.7184 -> 3.7484, door_count 0.3333 -> 0.3737, has_pv floor
0.8929 -> 0.8571 (one pair 25/28 -> 24/28).
This is a ground-truth-method change, not a prediction-logic loosening: the fix
is spec-correct, corpus MAE improved (0.571 -> 0.570) and the real-cert accuracy
pin held. Per the gate convention the affected floors/ceilings are re-baselined
to the measured values with the attribution recorded; tighten-only resumes here.
Bisected: #1666/#1667 leave the gates green; #1669 introduces all three shifts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous commit staged these two files at `git mv` time but never
staged the edits that followed, so they landed with their pre-refactor
content: `class PashubService` and `from backend.pashub_fetcher...`
imports that no longer resolve. Local runs passed because the working
tree was correct; CI collected the committed tree and failed with
ModuleNotFoundError: No module named 'backend.pashub_fetcher'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The service was the last one living wholly under backend/. It now follows
the same layering as abri and the other newer services:
domain/pashub_fetcher/ core file classification, subfolders
infrastructure/pashub_fetcher/ PasHub client, token getter, wire DTOs
orchestration/ PashubFetcherOrchestrator (was PashubService)
applications/pashub_fetcher/ lambda handler, trigger request, dev tooling
core_files.py is split along the layer boundary: the domain module keeps the
filename/evidence-category classification rules and no longer imports
infrastructure.postgres, while the CoreFiles -> FileTypeEnum translation moves
to infrastructure/pashub_fetcher/core_file_types.py.
Tests move into the tests/ tree by layer. Note this puts them in the only
suite CI currently runs (unit_tests.yml is disabled), so these 73 tests now
execute on PRs for the first time; they were previously reachable only via
the legacy pytest.ini testpaths.
sharepoint_renamer's image now copies just domain/pashub_fetcher/ rather than
the whole service, since SharepointSubfolders is all it needed.
Behaviour is unchanged. tests/ goes 9927 -> 10000 passed (+73, exactly the
tests that moved in); the legacy suite keeps its same 17 pre-existing failures
and 11 errors. tests/test_lambda_packaging.py confirms both changed
Dockerfiles still copy their handler's full import closure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The four calculator-accuracy fixes stack to move the RdSAP-21.0.1 corpus from
within-0.5 80.3% / MAE 0.584 to within-0.5 81.2% / MAE 0.565082. Ratchet the pins
(repo convention: tighten, never loosen): _MIN_WITHIN_HALF_SAP 0.803 -> 0.812
(measured floor), _MAX_SAP_MAE 0.584 -> 0.566 (MAE rounded up to 3 d.p. so the
`<=` assert holds).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wet underfloor responsiveness was pinned at R=0.75 for the lumped gov-EPC emitter
code 2, regardless of floor construction. SAP 10.2 Table 4d splits it three ways
and RdSAP 10 Table 29 (p.56-57) selects from floor construction + age band:
timber floor 1.0, screed above insulation 0.75, concrete slab 0.25. A
concrete-slab system (solid floor, age A-E) was billed as screed, over-stating
responsiveness (higher MIT) and over-rating the dwelling; R=0.25 was unreachable.
_responsiveness now takes the cert and, for the lumped underfloor code, derives
the Table 29 subtype via _underfloor_responsiveness from the main part's floor
construction + age band: suspended TIMBER -> 1.0; solid A-E -> 0.25 (concrete
slab); solid F-M / suspended-not-timber / unknown / no ground floor -> 0.75
(screed, the prior default, so ambiguous no-ground-floor certs do not move). The
mapper stays a pass-through.
Spec: SAP 10.2 Table 4d (p.170), RdSAP 10 Table 29 (p.56-57). Corpus: MAE
0.570 -> 0.565 (net toward lodged), within-0.5 81.2%. Cert 100100137438 (solid,
band B) 72 -> 69, exactly lodged. Closes#1668 on the API path.
NOTE: the issue's worksheet adjudication (run 100100137438 through accredited
Elmhurst to confirm concrete-slab R=0.25) is still outstanding — it corrects to
lodged 69 exactly, but is not yet Elmhurst-worksheet-confirmed. The Elmhurst
site-notes path (_resolve_elmhurst_underfloor_subtype still raises for solid A-E)
is untouched — no corpus/fixture exercises it; separate follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A cert lodging its horizontal dimensions measured externally
(measurement_type=2) must have its floor area and exposed perimeter converted to
internal dimensions before any SAP use (RdSAP 10 §3.4 + Table 2, p.17-18).
measurement_type was read nowhere (the `# What is this?` field), so external
certs carried a TFA ~11-15% too large — inflating occupancy and understating
every per-m2 output (CO2/m2, PE/m2).
Plumb measurement_type onto EpcPropertyData, and add a post-mapping
_with_internal_dimensions step: compute ONE whole-dwelling area/perimeter ratio
from the ground-floor footprint via the Table 2 built-form equation (§3.4 Note 4
— not a per-storey conversion, which over-corrects multi-part mid-terraces),
apply it to every storey's area/perimeter, reduce party walls by 2w (Note 5),
and refresh total_floor_area_m2 (read for occupancy N). Wall thickness w is the
lodged main-dwelling value, else the Table 3 default (p.19). Room-in-roof floor
area is always internal (§3.1) and left untouched. Scoped to the RdSAP-Schema
reduced-data family (17.0-21.x) where per-storey external dims are the lodgement
standard; legacy SAP-Schema-15.0/16.x are plumbed but not converted.
Corpus: within-0.5 81.0% -> 81.2%, MAE 0.571 -> 0.570, PE 2.5 -> 2.4. Cert
100091435353 TFA 128.7 -> 111.6 (lodged 112), SAP 66.23 -> 65.33 (+1.2 -> +0.3);
mean |dSAP vs lodged| over the 12 external certs 0.556 -> 0.429. Closes#1669.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On the gov-EPC API path a window's metal frame was unrepresentable: both
transmission tables were frameless (U_pvc, g, 0.70), so a cert lodging a metal
frame (pvc_frame:"false", RdSAP input 6-5) was billed at the PVC/wooden U and
frame factor 0.70 — over-rating metal-framed dwellings.
Add the RdSAP 10 Table 24 metal U-column (PDF p.50-51; mirrors u_window's metal
values) and a pvc_frame dimension to _api_glazing_transmission: an explicit
"false" routes the window to the metal U + SAP 10.2 Table 6c frame factor 0.8;
"true"/None/other keep the PVC column + 0.70, so no PVC cert regresses.
Secondary glazing keeps its frame-independent U (only FF changes). Wired into
_api_sap_window and _api_sap_roof_window (both full-schema, lodge pvc_frame);
the reduced-field path has no pvc_frame and is unchanged.
Spec: RdSAP 10 Table 24 metal column, SAP 10.2 Table 6c, RdSAP input 6-5 (p.80).
Corpus: within-0.5 80.6% -> 81.0%, MAE 0.578 -> 0.571. Cert 21067296
68.60 -> 66.96 (toward lodged 67). Closes#1667.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gov-EPC API lodges a two-main dwelling's main_heating_fraction pair in
two incompatible encodings — a decimal fraction summing to ~1.0 ([0.8, 0.2])
or an integer percent summing to ~100 ([80, 20]) — while every consumer
divides by 100 unconditionally. A decimal pair therefore collapsed the second
main's share to ~1% of its true value, billing almost the whole load to the
first system's fuel/efficiency.
Normalise once at the mapper via a shared pair-sum discriminator
(_normalised_main_heating_fraction): a genuine two-main split summing nearer
1.0 than 100 is scaled to percent; a percent pair and any single main are left
exactly as lodged, so the 3 percent-encoded corpus certs do not regress. Wired
across the seven from_rdsap_schema_* branches and the SAP-17.1 path, whose old
int(fraction) floor silently deleted a decimal-lodged second main (int(0.35)=0).
Spec: RdSAP 10 item 7-5 (percent is canonical), p.81. Corpus: within-0.5
80.3% -> 80.6%, MAE 0.584 -> 0.578. Example cert 10070086972 17.59 -> 18.73
(toward lodged 18). Closes#1666.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`plan.name` is a freeform, user-entered label that is unset on
modelling-generated Plans (0/205 populated on a real portfolio). The meaningful
"name of the plan chosen" is the Scenario the home's default Plan came from, so
`plan_name` now reads `scenario.name` via `plan.scenario_id` (LEFT JOIN scenario)
on both export selections. On the default-plan export this labels each home with
its source scenario (e.g. "Fabric Only" / "ASHP + PV"); on the scenario export it
equals that sheet's scenario.
Repository test seeds scenarios and asserts the source-scenario name; 62 tests
pass, pyright --strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A second export selection alongside the per-Scenario one, chosen by
`plan_selection` on the export request:
- "scenario" (default, unchanged): one sheet per Scenario, freshest Plan per
(property, scenario); scenario_ids required.
- "default" (new): a single "Default Plans" sheet of each home's is_default
Plan (one-per-property across all Scenarios, ADR-0012/0017) — the portfolio's
current state, one row per home; scenario_ids ignored.
Repository gains `default_rows_for(portfolio_id, property_ids)`: same read-model
and Effective-EPC join as `rows_for`, with the plan-selection WHERE swapped from
`scenario_id = :scenario_id` to `is_default = TRUE`. Orchestrator branches on
plan_selection; the router validates it and only requires scenario_ids for the
scenario selection. A recipe without plan_selection defaults to "scenario", so
the change is backward-compatible.
Both selections now surface the chosen Plan's `name` as a `plan_name` column.
Tests: repository default_rows_for (scenario-independent selection, plan_name,
non-default excluded); orchestrator default path (one "Default Plans" sheet;
empty selection is a recorded failure). 62 passed, pyright --strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 item 2-3(h) "another dwelling above" is U=0 (spec p.45: "There is no
heat loss through the roof of a building part that has the same dwelling above
or another dwelling above"). gov-API `roof_construction = 3` is that code, but
the mapper labelled it "Pitched (slates/tiles), no access to loft" (which is
code 5), so the per-part party override at heat_transmission.py:1172 never
matched and the part fell through to the dwelling_type-label-only
has_exposed_roof heuristic.
190 of the 200 corpus code-3 parts sit at index 0 and are already suppressed by
`_cert_lodges_exposed_roof` off the roof_insulation_location == "ND" signal. The
unmitigated set is the 10 parts at index > 0, where extensions default to
exposed and were billed a phantom pitched roof. Also removes a latent PV defect:
the bogus "Pitched" prefix satisfied `is_pitched` and divided PV array area by
cos(35 deg), inflating kWp.
Shipped with the stairwell fix so the corpus pin moves once: within-0.5
78.8% -> 79.6%, SAP MAE 0.626 -> 0.611, PE MAE 2.9 -> 2.7.
The roof change alone trades 4 certs out of the +-0.5 band for 2 in (one moving
4.4 SAP closer). Those 4 were investigated and are NOT a masked bug: the corpus
error distribution is symmetric (mean +0.076, median +0.03, sigma 1.606; 112
certs above +0.5 vs 102 below), ground-floor flats show 7 positive / 8 negative
with no excess over the null, and closing them needs HLC changes of 2.3-5.6% —
below the ~8.8% noise floor of the RHI anchor, calibrated on the 76 certs whose
SAP is exact (space-heating ratio 1.062 +/- 0.088).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §5.9 "U-values of sheltered walls" (PDF p.43): a wall between a
flat/maisonette and an unheated buffer takes SAP 10.2 §3.3's U = 1/(1/U0 + Ru),
and Ru depends on the BUFFER TYPE — "Use Ru = 0.5 m²K/W for corridors and
Ru = 2.1 m²K/W for stairwell."
The buffer type is lodged on `SapFlatDetails.heat_loss_corridor` (0-based per
item 1-11, PDF p.70: 0 none, 1 heated, 2 unheated corridor, 3 unheated
stairwell). That field was read NOWHERE under domain/ — corridor presence was
inferred only from a lodged sheltered alternative wall — so every sheltered wall
took the corridor 0.5 and every stairwell flat over-billed its sheltered wall,
under-rating the dwelling.
Distinct from `_SHELTERED_GABLE_ADDED_RESISTANCE_M2K_W`, which is RdSAP 10
Table 4 (p.22) for a room-in-roof sheltered gable — same 0.5, different rule,
deliberately untouched.
Corpus gauge ratcheted UP: within-0.5 78.8% -> 79.8% (+10 certs), SAP MAE
0.626 -> 0.613, PE MAE 2.9 -> 2.8. Failure set and pyright count identical to
main. Portfolio 824 is stairwell-heavy (60 properties at code 3); property
749898 over-bills 4.44 W/K on a 48 m2 flat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §3.12 (PDF p.26), the flats/maisonettes floor rules: "There is no heat
loss through the floor if there is another flat below. Otherwise the floor area
of the flat ... is ... a semi-exposed floor if there are unheated premises below
it (e.g. an enclosed garage) ... Semi-exposed (sheltered) floors are treated as
if they were fully exposed." §5.13 (p.48) confirms the U-value identity: Table 20
applies to exposed and semi-exposed alike.
API floor_heat_loss=2 -> floor_type "To unheated space" set no exposure signal,
so on a mid-/top-floor flat the dwelling-level heuristic (has_exposed_floor=False,
assuming a heated dwelling below) went unopposed and the ENTIRE floor was billed
at 0 W/K. 11 properties across portfolios 796+824, mean +4.02 SAP, 10 of 11
over-rating — we judged dwellings more efficient than the accredited assessment
and would withhold retrofit measures they need.
Exemplars: 750018 space heating -53.2% -> -8.1% vs the cert's own RHI figure,
SAP +8.22 -> +1.16; 749898 -43.0% -> +0.9%, SAP +6.29 -> -1.11.
SCOPED TO FLATS ON PURPOSE. §3.12 is the flats/maisonettes section, and the
accredited Elmhurst worksheet disagrees with Table 20 for a house: golden cert
7536-3827-0600-0600-0276 (detached) lodges floor_heat_loss=2 on its Main part and
the worksheet bills it on the BS EN ISO 13370 ground-floor cascade at U 0.97, not
Table 20's 1.20. An unscoped mapper-level fix broke that pin and the
uprn_100050881708 accuracy pin; the scoped fix leaves both untouched.
No regressions: identical failure set and pyright count vs main; the RdSAP-21.0.1
accuracy corpus gauge is unchanged at 78.8% within-0.5 / MAE 0.625 (correctly —
all 7 corpus certs lodging code 2 are houses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §5.17 / Table 23. `wall_is_basement` is not a lodged U-value — it is a
disambiguation flag selecting WHICH RdSAP default applies, so unlike
wall/roof/floor_u_value it must survive the round-trip.
Code 6 is canonically SYSTEM-BUILT and the gov-API path infers "basement" from a
code-6 heuristic; the Elmhurst site-notes mapper sets wall_is_basement=False
precisely to defeat it. Dropping the field inverted the answer — a system-built
wall came back as a basement wall, dragging the whole ground floor onto the
Table 23 basement-floor U-value via `has_basement`.
Declared on no RdSAP schema, so it never arrives via the gov API; the earlier
corpus scan showing 0 occurrences measured the wrong population. The Elmhurst
site-notes path sets it and those EPCs are persisted
(scripts/wchg_elmhurst_ingest.py).
Nullable is load-bearing: None = "not stated, use the heuristic", False =
"explicitly system-built, do not".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add `rafter_insulation_thickness` to `EpcBuildingPartModel` (JSONB, matching the
sibling thickness columns so Union[str, int] survives) and wire it through
`from_domain` / `_to_building_part`.
Property 749719 lodges "250mm" -> Table 16 col (2) U 0.23. Dropped, the roof
billed at U 2.30 — a 10x error worth -9.8 SAP (lodged 73, calculated 63.2) that
moved the dwelling a whole band, C -> D, and drew a 9.63-point loft-insulation
recommendation on a roof rated 4/5 by the assessor.
The sibling wall/roof/floor_u_value and wall_is_basement fields have the same
drop but are deliberately NOT persisted: they are full-SAP artefacts and we
re-model as RdSAP, which uses default U-values. Their allow-list entries are
re-justified rather than removed — the previous "not read by the calculator" was
stale and is what hid this drop.
Requires the Drizzle migration in Hestia-Homes/assessment-model to land FIRST;
without the column, EPC reads fail outright.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §5.11.2 — a roof insulated at rafters lodges its thickness in
`rafter_insulation_thickness`, not `roof_insulation_thickness`. The field had no
DB column, so it was mapped from the gov-EPC API and silently dropped on save;
every at-rafters roof then fell back to the Table 18 col (2) unknown-thickness
default (2.30 W/m²K for bands A-D — the spec's *uninsulated* value).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
0390's floor+ASHP package reaches post 68.772, which publishes as 69 = band C,
so under ADR-0064 it now meets the goal and stops before low-energy lighting —
the same over-purchase the ADR removes, one level up at the package. 0380 fires
the same three measures with identical trigger attributes, so every assertion
stands unchanged, and it lands at post 74.4, clear of a band floor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The modelling_e2e handler models one property per run_modelling call, so a
per-property "no persisted baseline" line would be one per property across a
whole re-model. The harness has no Baseline stage by design and cannot drift
from one, so it now establishes its own the way the Baseline orchestrator
would; the Postgres-backed lane is where the assertion has something to catch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dan's review on #1546:
- Stream the workbook to a /tmp temp file and S3Client.upload_file it, instead
of render_workbook -> bytes -> put_object. render_workbook now saves straight
to a path; the orchestrator renders to a temp file, multipart-uploads it, and
always cleans it up. Restores ADR-0065's "never an in-memory BytesIO" decision
(the OOM path at the 100k-row cap).
- Move the raw `SELECT ... FROM scenario` out of the Lambda handler into
ScenarioNamesPostgresRepository, so the handler stays composition-only and all
SQL lives in repositories/.
- current_sap_points goes through _float, matching its Optional[float] read-model
type and the sibling numeric facts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "latest EPC per UPRN" selection ordered by e.id DESC, but row-id does not
track recency: a gov-EPC cert re-ingested after a PasHub survey lands with a
higher e.id, so the header/perf reads silently picked the stale gov cert (wrong
TFA, lodgement, property_type, current band) on 3 of portfolio 838. Order by the
effective lodgement date (registration -> completion -> inspection) DESC NULLS
LAST, e.id DESC tiebreak.
Compose the descriptive fabric from the structured SAP fields a re-survey lodges
(no gov-EPC prose): walls/roof/floor from the main epc_building_part, heating and
hot water from epc_main_heating_detail + the water-heating codes, decoded with
the SAP engine's own code space (fabric_description.py). Structured fabric wins
where present and falls back to the gov-EPC prose otherwise; windows and lighting
stay on prose (their per-element codes do not reconstruct the dwelling phrase).
Add built_form (epc_property, code-decoded) and property_age_band (building part
construction_age_band -> RdSAP date range) columns.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
6035 is built_form 4 (Mid-Terrace), so this PR routes its Exposed Type-1
RR gable to a party gable (U=0.25) instead of the main-wall U. Heat loss
drops: PE resid -0.14 -> -0.92, CO2 -0.0362 -> -0.0549 (SAP still exact 70).
Residuals stay within noise (PE <0.5% of lodged 191). All 113 golden
fixtures pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deal's third_party_surveyor_identifier is now required by the log
and amend flows: the trigger contract rejects messages without it
(blank treated as missing, since a blank resource makes OpenHousing
silently drop the appointment), and LogJobRequest/AmendJobRequest carry
it as a required field. The ABRI_RELAY_DEFAULT_RESOURCE fallback and
its config/terraform/workflow wiring are removed as dead code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Carry third_party_surveyor_identifier on the Abri trigger message and
through dispatch into the log and amend flows, booking the job against
that resource. LogJobRequest gains the optional resource field
AmendJobRequest already had; the client falls back to the configured
default surveyor when the deal names no third-party resource, so
existing deals behave unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A mid-terrace / enclosed-mid-terrace (built_form 4/6) has external walls on
its two opposite (front/rear) faces only (RdSAP 10 §1.1, p.7-8); both end/
gable elevations — and their room-in-roof continuations — abut a neighbour
and are party walls (§1.4.3, p.9). The gov-API path is built_form-blind, so a
cert lodging an "Exposed" RR gable (`gable_wall_type=1`) on such a form had it
billed at the masonry main-wall U (~2.5) instead of the party 0.25 (Table 4,
p.22), over-counting fabric heat loss and under-rating the dwelling.
`_api_type_1_gable_kind` now takes `built_form` and reclassifies an Exposed RR
gable to Party on the both-ends-party forms {4, 6} only; end-terrace {3, 5},
semi {2} and detached {1} keep their genuine exposed gable. The API path lodges
no per-gable U, so this only ever overrides a cascade fallback.
Corroborated three ways: RdSAP §1.1 + Table 4; S10TP-05 Appendix A (a
mid-terrace's roof-gable junction length is zero); and three accredited
Elmhurst mid-terrace worksheets (000477/000480/000516), which all lodge Party
gables. Those worksheets run the site-notes path (already correct) so the 195
Elmhurst pins are untouched.
Corpus (RdSAP-21.0.1, 1000 certs): within-0.5 78.8% -> 79.5%, SAP MAE
0.625 -> 0.599, PE MAE 2.85 -> 2.71, CO2 MAE 0.070 -> 0.068. Cert
100040550095 -10.13 -> +0.37; the 18-cert terraced-external-gable cohort mean
-1.94 -> +0.11; end-terrace/semi/detached untouched. Thresholds ratcheted.
Tests: new test_mapper_rir_gable_terraced (9); 195 Elmhurst pins, 288
cert_to_inputs/real-cert pins, 54 mapper tests, and the corpus gauge all green;
no new pyright errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two fixes so a multi-scenario export over a PasHub-fetched portfolio is correct:
1. Plan selection: take the NEWEST plan per (property, scenario), not the
is_default one. is_default is one-per-property (not per-scenario), so it
cannot scope a multi-scenario export — a property's default sits under a
single scenario, leaving every other scenario with zero rows. _ROWS_SQL and
_MEASURES_SQL now DISTINCT ON (property_id) ORDER BY created_at DESC, id DESC.
2. Effective-EPC descriptive block: match the header facts + current performance
by UPRN (the latest epc_property for the source), because the PasHub re-fetch
lands records with property_id = NULL — a property_id join silently fell back
to a stale gov-EPC cert (wrong 2013-2025 lodgement dates / int property_type)
or nothing. Lodgement date coalesces registration -> completion -> inspection
(PasHub surveys carry only inspection_date). Descriptive prose elements stay
on the property_id gov-EPC fallback (a survey lodges structured fields, not
gov-EPC prose).
Result on portfolio 838 / scenarios 1303+1304: both sheets populate 205 rows;
property_type text on 204/205; lodgement dates 2026 on 200/205. Repository tests
updated (uprn fixtures, newest-plan selection + a newest-wins case); 22 export
tests pass, pyright clean.
* Electric-immersion DHW: bill at 100%, not the space boiler's efficiency 🟩
A separate electric immersion heater (SAP water-heating code 903) is 100%
efficient (SAP 10.2 Table 4a) and the space-heating boiler provides no water
heating, so Appendix D2.1 Eq D1 (the boiler seasonal-efficiency cascade) must
not apply to it. But `_water_heating_main` resolves DHW to the SPACE main — a
gas/oil boiler keeps its PCDB record — so three water-efficiency branches in
cert_to_inputs billed the immersion-heated cylinder at the boiler's ~87% summer
efficiency (a mongrel: electric fuel price x gas-boiler efficiency):
1. the scalar `water_eff = water_pcdb_main.summer_efficiency_pct / 100`,
2. the SAP §9.4.11 / Table 4c(2) -5pp no-interlock adjustment, and
3. the Eq D1 (winter, summer) seasonal pair from `pcdb_main`.
Gate all three on `not dhw_is_electric_immersion`; the correct immersion path
(`_water_efficiency_with_category_inherit` -> 1.0, Eq D1 off) then applies,
mirroring the Table 3 zero-primary-loss gate already present for WHC 903.
General bug — fires for any WHC-903 dwelling whose space main is a PCDB gas/oil
boiler. It surfaced via the PasHub campaign's #1600 no-water-heating default
(WHC 999 -> 903) on a gas-combi dwelling: 58 Hackle St M11 4WU, SAP 55.30 ->
57.67 (pre_sap 58, verified 59).
Guardrails: the gov-API RdSAP corpus IMPROVES 78.8% -> 78.9% within-0.5, MAE
0.625 -> 0.622 (a handful of corpus certs with a boiler space main + electric
immersion move closer to accredited) — MAE ceiling ratcheted 0.626 -> 0.625.
Regression pinned in test_cert_to_inputs (RED before / GREEN after). pyright
0-new (cert_to_inputs baseline 30).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* PasHub SAP accuracy: thread low-energy lights + draught lobby (from_site_notes) 🟩
Two systematic silent-drops in the PAS Hub `from_site_notes` path, each a field
the gov-API/Elmhurst mappers thread but site-notes dropped, letting a favourable
default reach the SAP-10.2 calculator. Validated against the correct pashub
oracle (property_baseline_performance.effective_sap_score, portfolio 838):
cohort within-0.5 55.1% → 62.6%, MAE 0.599 → 0.521.
- Low-energy lights: when "exact LED/CFL known = No", PAS Hub lodges an aggregate
"Number of fixed low energy lights?" count. Previously dropped (no dataclass
field / extractor key / mapper thread) → calculator saw 0 low-energy bulbs and
applied the pessimistic L5b/L8c no-data default, under-rating SAP. Now threaded
into low_energy_fixed_lighting_bulbs_count, mirroring from_elmhurst_site_notes.
~6% of the cohort; resolves 58 Hackle (−0.43 vs oracle 59 with #1615).
- Draught lobby: _map_sap_ventilation set only the legacy `draught_lobby` field,
never the canonical `has_draught_lobby` §2 (13) gate the cascade reads, so the
surveyed lobby was ignored and infiltration over-stated. Now mirrors Elmhurst.
Two other audited levers were REJECTED against the pashub oracle (they matched
the gov-cert/RdSAP convention but pashub does not apply them): percent_draughtproofed
(cohort 30.8%) and the §A.2.2 assumed secondary heater (19.2%). A heat-network
control-code fix (2306→2303 for 16 Bingley) is HELD pending spec/Elmhurst
adjudication — it contradicts ADR-0053; see the NOTE on _PASHUB_HEAT_NETWORK_CONTROL_TO_SAP10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* PasHub heat-network control: room-thermostat-only maps to 2308, not 2306 🟩
The Table 4e Group 3 label "charging linked to use of community heating, room
thermostat only" was mapped to 2306 — the linked-to-use *with-TRVs* code
(control type 3, space charging 1.00). The label explicitly excludes TRVs, so
it is control type 2 / space 1.05 = code 2308/2309 (linked-to-use → DHW 1.00).
2306 asserted TRVs the survey doesn't have and over-rated the cohort's sole
heat-network dwelling (16 Bingley Close 56.89 → 54.01 vs pashub oracle 52).
Verified against SAP 10.2 Table 4e as encoded in cert_to_inputs.py
(_CONTROL_TYPE_BY_CODE 2308→2, space-charging 2308→1.05, DHW 2308→1.00) and the
RdSAP control-label vocabulary in MainheatControlAttributes.py. 2303 (the earlier
audit's guess) is rejected: it is a flat-rate code (DHW 1.05) whose oracle match
was a coincidence of two offsetting spec violations. The residual +2.0 to oracle
52 is the documented SAP-10.2-engine-vs-lodged offset, not a fuel/flags gap
(those were threaded by the #1590 follow-up) — supersedes the ADR-0053 /
HANDOVER_838 "community fuel/flags" attribution.
16 Bingley is the only heat-network fixture in portfolio 838, so this moves one
fixture strictly toward its oracle. Harness ratcheted: within-0.5 0.51→0.58,
MAE 0.625→0.521 (observed 58.5% / 0.520 with all three 2026-07-15 levers).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* PasHub SAP accuracy: restore 3 dropped §2 infiltration inputs (from_site_notes)
`from_site_notes` uniquely dropped three ventilation inputs that the gov-API
and Elmhurst sibling mappers all set, causing a systematic cohort-wide SAP
under-rate vs the company's own accredited SAP-10.2 certs
(property_baseline_performance.effective_sap_score, portfolio 838):
1. percent_draughtproofed — never set, so §2(15) window infiltration was
pinned at its 0.25-ACH worst case on every dwelling. Reconstructed as the
area-weighted % of draught-proofed windows (mirrors Elmhurst
`draught_proofing_percent`). Dominant term.
2. Upper-storey +0.25 m joist void — omitted; now added, matching
`_UPPER_FLOOR_HEIGHT_ADD_M` on the gov-API/Elmhurst paths. This
RE-ADJUDICATES #1601 ("keep raw"): that call was confounded by the
then-present draught-proofing drop suppressing every verified dwelling.
3. sheltered_sides — left None → calculator's flat default of 2, which
over-shelters end/semi/detached (RdSAP §S5 = 1/1/0). Now derived from
built form.
The three are NON-ADDITIVE — each overshoots alone (which is why all three
were previously rejected individually) — but together they land all 7
verified ground-truth dwellings toward truth (none regress) and every built
form near zero. Cohort 62.1% -> 83.3% within-0.5, MAE 0.507 -> 0.377.
Guardrails: gov-API RdSAP corpus unchanged (78.9%, these are from_site_notes-
only helpers). pashub harness 58.5% -> 79.5% / MAE 0.520 -> 0.389; ratchets
tightened 0.58->0.78 and 0.521->0.40. pyright zero-new (mapper baseline 39).
New focused tests in TestFromSiteNotesInfiltrationFixes cover all three
fields incl. area-weighting and the built-form shelter map; goldens updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
