Thread the UPRN the PashubService has already resolved for a job into the
PasHub site-notes mapper so the EpcPropertyData aggregate is born with its
uprn set, and the existing save_epc_property_data path persists it.
- EpcPropertyDataMapper.from_site_notes gains uprn: Optional[int] = None and
sets it unconditionally (site notes never carry a UPRN natively).
- parse_site_notes_pdf / _parse_pashub forward the uprn; the Elmhurst branch
is untouched.
- PashubService coerces uprn str -> int in one place, carries it on the
internal upload record, and passes it into parse_site_notes_pdf.
- No new lookups; jobs with no known UPRN still persist null, unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The prior commit derived a Python bool (`cylinder_size != 1`), but
`from_rdsap_schema_17_1` reads the field via `schema.has_hot_water_cylinder
== "true"` — every real 15.0/16.x fixture lodges the lowercase string
"true"/"false", not a JSON boolean. A bare bool compares False against
that string check either way, so the derivation silently mapped every
cylinder-present cert to has_hot_water_cylinder=False too.
Caught by the requested true-branch test (mutating sap_16_2.json, which
lodges cylinder_size=2/has_hot_water_cylinder="true", to omit the field)
— it failed under the original bool-typed fix. Now emits "true"/"false"
strings to match the lodged convention.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task b9fcd354-2a33-4fc4-a64e-388c060a055c (portfolio 824 / scenario 1278)
also failed property_id 749229 (UPRN 100010349400) with "RdSapSchema17_1:
missing required field 'has_hot_water_cylinder'". Cert
9568-3034-6211-6089-5960 (SAP-Schema-15.0) lodges `sap_heating.cylinder_size`
but omits the separate top-level `has_hot_water_cylinder` boolean
RdSapSchema17_1 also requires.
Unlike `multiple_glazed_proportion` (deliberately left un-defaulted a few
lines above — a prior guessed default regressed the accuracy gate),
`cylinder_size` isn't a proxy needing inference: RdSAP 10 Table 28 defines
code 1 as "no cylinder" — literally the same fact `has_hot_water_cylinder`
encodes. Confirmed 1:1 across every real 15.0/16.x fixture that lodges both
fields (cylinder_size == 1 <-> false, every other code <-> true).
Added to the shared `_normalize_sap_schema_16_x`, so it covers the whole
15.0/16.0/16.1/16.2/16.3 reduced-field family, not just 15.0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task b9fcd354-2a33-4fc4-a64e-388c060a055c (portfolio 824 / scenario 1278)
still failed 5 of 128 batches with NotNullViolation on
epc_building_part.construction_age_band even after the SAP-Schema-15.0
mapper landed (PR #1531). Cert 8169-6926-5310-0447-8996 (UPRN
100010344955, SAP-Schema-15.0) lodges a bare alternative-wall record
(wall_area/wall_construction/wall_insulation_type, no identifier, age
band, or floor dimensions) as a second sap_building_parts element rather
than under a distinct sap_alternative_wall_1 key. RdSapSchema17_1 (which
the 15.0/16.x reduced-field mappers delegate to) has no alt-wall slot to
route it to, so it persisted as a phantom building part with a null age
band.
_drop_building_parts_without_age_band already solved exactly this shape
of problem for lodging artifacts (ae81a81e), but that commit sits on
fix/drop-building-part-without-age-band, an unmerged branch that has
since drifted far behind main. Reinstating the same filter (fresh, not
cherry-picked) fixes both the original artifact case and this SAP-15
alt-wall case, since RdSapSchema17_1 has nowhere else to put it either
way.
Verified against all 174 properties across the task's originally-failing
batches: zero remaining null-age-band building parts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
from_api_response now returns None for an unsupported schema_type
instead of raising, so the harness must check before dereferencing
the mapped EpcPropertyData — otherwise an AttributeError replaces
the expected ValueError in calculator_error.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Review findings on PR #1527:
- The overlay constants, ScoredOption builder, ventilation dependency and
selected_types helper come from the shared _optimiser_fixtures module
(landed on the fabric-first base) instead of local copies; the boiler
overlay is the shared BOILER_OVERLAY (SAP Table 4a code 104, a mains-gas
combi) rather than code 201, which is neither a boiler nor a heat pump.
_IWI_OVERLAY (solid-wall internal, type 3) stays local — no shared
equivalent — and the carbon stubs stay bespoke (the shared StubScorer has
no CO2 knob).
- The optimise_package_fabric_first import is lifted to module scope.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review findings on PR #1527:
- The 'which goals are goal-aligned' set lived in two adjacent if-chains
(_objective_for and _require_budget_for_goal_aligned) that had to stay in
sync — a new goal-aligned goal added to one but not the other would slip
the budget guard. Both now read a single _GOAL_ALIGNED_OBJECTIVES table.
- The goal strings are the canonical PortfolioGoal enum values, not
re-declared string constants, so goal-value drift can't silently degrade
a goal to max-SAP; _target_sap reads the enum too.
- _scored_candidate_groups takes objective without a default (its only
caller passes it).
- scoring.py: 'cached: float | None' -> Optional[float] per the CLAUDE.md
'Use Optional over | None' rule.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review findings on PR #1527:
- The objective is required (no sap_rating default) on _repair_to_target,
_best_repair_candidate and _rescored_groups: every caller already passes
it, and a default would let a future call path silently optimise SAP for
a carbon/bill goal while pyright stayed green. The default stays on the
public optimise_package / optimise_package_fabric_first entry points.
- _best_repair_candidate hoists objective(current) out of the candidate
loop: current is loop-invariant, so for the Energy-Savings bill objective
this was one full BillDerivation.derive per candidate per repair iteration
for the same score.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An unrecognised schema_type used to raise ValueError and abort the whole
call; now from_api_response logs a warning and returns None so one
unmapped cert doesn't break a batch. Schema 15.0 already has a mapper
(PR #1531) so it's unaffected; only genuinely unmapped versions skip.
OpenHousing logs a job but silently drops the appointment when no bookable
resource is sent, so the survey date never lands. log_job already sends the
configured default_resource; amend_job omitted the parameter unless the
request carried an explicit surveyor. Default amend's resource to the
configured surveyor too, and fail loudly at config load on a blank
ABRI_RELAY_DEFAULT_RESOURCE so the misconfiguration can't ship an empty
resource again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review findings on PR #1526:
- tests/domain/modelling/_optimiser_fixtures.py is the one home for the
overlay constants, the ScoredOption builder, the additive per-kind
StubScorer and the forced ventilation dependency; test_optimiser.py and
test_optimiser_fabric_first.py had byte-identical copies of each
(and _StubScorer / _VentStubScorer fold into one parameterised stub).
- Fixture worlds are domain-plausible per team convention: the fabric-vs-
heating contrast is a £12,000 EWI against a £3,200 gas boiler rather
than a £500 heat pump undercutting a £1,000 cavity wall; heating
overlays carry real identities (SAP Table 4a code 104 for the boiler,
a PCDF index for the heat pump) instead of code 201 doubling as both;
whole-dwelling double glazing is £3,500, not £500.
- Dead knobs removed: the unused _ROOF_OVERLAY, the always-zero roof
gain, the duplicate _BOILER_OVERLAY, and the nested conditional
expressions in the interaction stubs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review findings on PR #1526, behaviour-preserving:
- No fabric groups, or phase 1 committed nothing → delegate to the plain
optimise_package: the phase-2 prefix would be empty and its re-scoring
would reproduce the role-1 signals the groups already carry, one full
SAP-calculator run per Option for zero effect (the common case on
already-insulated stock).
- Phase 1 consumed every group → return the fabric package instead of a
phase-2 pass that can only select the empty package.
- _rescored_groups takes start_sap from the caller: the post-fabric
baseline score is the phase-1 package score already in hand, not a
fresh calculator run.
- fabric_types → phase_one_types: the set holds everything phase 1
committed, injected dependencies included — that inclusion is what the
outstanding-dependencies filter relies on, so the name must not invite
narrowing it to FABRIC_MEASURE_TYPES.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-attempted entering this SAP-15.0 cert's 4 real N/E/S/W windows (gov-API
sap_windows, 2.516 m^2 each) as separate Elmhurst Openings rows instead of
the accepted single combined South row, to test whether it closes the
residual ~2pt gap (engine 53 vs Elmhurst worksheet 51).
Reproduced live the documented elmhurst_lib.py window-grid limitation:
adding a 2nd row wipes the previously-added row's width/height to 0.00,
in both same-session and fresh-session-per-window attempts. Not fixable
via current Playwright automation. Restored the assessment to its original
single South 10.06 m^2 row (no PDFs re-downloaded, no worksheet change).
Separately confirmed our own engine already models this cert's windows
per-orientation correctly from gov-API sap_windows via
solar_gains.py's ORIENTATION_BY_SAP10_CODE cascade -- this is an Elmhurst
build-tooling ceiling, not an engine or mapper bug. sap_score=53 remains
unchanged; no calculator/mapper code was touched. Full accuracy suite
re-run clean (66 passed/67 skipped/1 xfailed + corpus test passed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both flagged mismatches were Elmhurst input errors (same silent-stale-value
contamination class as the earlier chimneys/wall-thickness bugs), not
parser or calculator bugs:
- cylinder_size: build_100010086084.py's water_heating() selected
DropDownListCylinderSize by raw value "2", but this DOM's option values
ARE their visible litre-band text (no "2" among them) — Playwright
silently no-ops on a non-matching value, leaving a prior cert's "Medium
(131-170L)" selection in place. Fixed to match by text ("Normal"), and
added the missing mapper dict entries (Normal/Large litre-suffixed
labels) to _ELMHURST_CYLINDER_SIZE_LABEL_TO_SAP10 (a real mapper-coverage
gap — the calculator raises UnmappedElmhurstLabel rather than silently
mis-mapping).
- boiler_flue_type: the boiler-code search dialog's combined "Balanced/Open
Flue" Table 4b category doesn't drive the separate, independently
selectable RadioButtonListFlueType field, which was left at an inherited
"Balanced" from a prior cert. Fixed space_heating() to explicitly select
"Open" to match the lodged gov-API code. This field isn't consumed by
Sap10Calculator (ML/generator-only), so it had zero effect on the SAP
score.
Re-downloaded elmhurst_summary.pdf/elmhurst_worksheet.pdf after the fixes
(Recommendations page confirmed clean). Elmhurst-PDF-inputs path moved
46->47 SAP from the cylinder-volume correction; gov-API SAP (53) and
Elmhurst's own worksheet (51) are unchanged. Full accuracy suites re-run
clean (67 passed, 67 skipped, 1 xfailed, same as before).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Elmhurst rebuild for this cert had two carryover bugs from the shared
Playwright session: TextBoxOpenChimneys silently inherited "2" from a prior
cert's build (this cert's register lodges open_fireplaces_count=0, which
the gov-API mapper already reads correctly), and wall thickness inherited
a stale 280mm. Also fixed a real bug in the space_heating() PCDB-ref-clearing
logic: input_value() returns the string "0" when already cleared, which is
truthy in Python, causing a spurious navigate-away that broke the next
E.goto() call.
Rebuilt cleanly (chimneys=0, wall thickness=220mm per RdSAP convention for
unrecorded solid-wall thickness, since CheckBoxWallThicknessUnknown does not
persist via Playwright in this tool version). Elmhurst's own worksheet score
moves from a contaminated 46/48 to 51, closing the gap against this engine's
53-on-gov-API-inputs from ~5-7pt to ~2pt. No mapper.py change was needed --
the gov-API mapper's open_chimneys_count=0 was correct all along.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Built UPRN 100010086084 (the largest gov-API-vs-lodged outlier from the
20-cert batch) in Elmhurst's accredited RdSAP10 tool to get real ground
truth, per feedback that lodged rating isn't a valid comparison target.
Elmhurst worksheet: SAP 46-48 (orientation-dependent), vs this engine's
53 on gov-API inputs -- a real ~5-7pt residual gap. Ruled out with
direct empirical tests: total_floor_area, missing party_wall_length,
region/climate (SAP rating always uses UK-average weather per Appendix
U -- confirmed inert), boiler/secondary efficiency (exact match, 66%/
63% both sides). Confirmed real but NOT blind-fixed (need more than one
cert's evidence, and a wall-U-value patch tested the wrong direction in
isolation): a wall-thickness-unknown U-value fallback mismatch (this
engine 1.70, Elmhurst's own "unknown" resolution 1.40) and a window
U-value divergence (1.85 vs 2.52 W/m2K for the same nominal glazing
pick). Single-window orientation confirmed a ~2pt partial contributor.
Also fixes a small, separate, evidenced gap the build surfaced: the
Elmhurst summary parser didn't recognise the live tool's "Medium
(131-170L)" cylinder-size label (only the bare "Medium").
Full write-up is in the RealCertExpectation comment for this cert in
test_real_cert_sap_accuracy.py and the worklist. Residual gap is
intentionally left open for follow-up, not force-closed.
These are 2011 SAP-2009 lodgements; this calculator implements current
RdSAP10, which has since revised several U-value tables. Closeness to
lodged was never evidence of mapper correctness (nor is a gap evidence
against it) -- the prior commit's comments and worklist entries wrongly
framed 16/20 "matching lodged" as validation and the other 4 as
anomalies needing explanation. Corrected the comments to state what the
batch actually shows: all 20 structurally distinct certs map without
error and produce plausible scores. Real accuracy validation still
needs an Elmhurst RdSAP10 rebuild on the same methodology as the
engine, which remains open.
Sampled all 20 unique property_ids that failed with "Unsupported EPC
schema: 'SAP-Schema-15.0'" across the sub_task history (114 total across
72 subtask rows), captured 20 real live certs, and pinned each as a
RealCertExpectation regression in the accuracy corpus.
16/20 land exact or within 1-3 SAP of the lodged rating, confirming the
door_count/WWHRS/FGHRS/draught-proofing defaults generalise beyond the
original two properties. The other 4 (all solid-brick, uninsulated,
age band C) diverge +6 to +12 — traced to a legitimate SAP-2009-vintage-
lodgement vs RdSAP10-recalculation methodology gap (u_wall() checked
directly against RdSAP10's own Table 6, returns the correct 1.70 — no
mapper defect), not a mapper bug, so pinned to the engine's observed
value rather than tuned to lodged.
Task a40e71c4-fd56-4a33-8677-e8643e6a5bb4 (portfolio 824, scenario 1278)
failed 2 of 29 properties with "Unsupported EPC schema: 'SAP-Schema-15.0'"
(property_id 748445/748451). Traced to two real 2011 lodgements
(schema_version "LIG-15.0", assessment_type "RdSAP") for UPRNs
100010359769/100010359788 — structurally the same reduced-field RdSAP-17.1
shape as the already-handled SAP-Schema-16.x family, one revision older.
from_sap_schema_15_0 reuses _normalize_sap_schema_16_x and adds three
schema-wide defaults this older generation never lodges at all (confirmed
absent on both real certs, not just sparse on one):
- door_count=1 (single external door, standard RdSAP assumption)
- instantaneous_wwhrs=0/0/0 and has_fghrs="N" (predates WWHRS/FGHRS
lodging — absence-means-none-fitted, same convention used elsewhere)
- percent_draughtproofed=0 (mirrors the existing has_draught_lobby
"assume none if unknown" convention in cert_to_inputs.py)
- energy_rating_average=60 (inert metadata, never read by the calculator)
Both real certs now map and run end-to-end through the SAP-10 engine,
matching (or within 1 point of) their lodged SAP score.
epb-data-warehouse's fixture set covers CEPC-7.0/7.1/8.0.0 (SBEM
non-domestic certs) alongside the RdSAP/SAP families we already map.
CEPC has no sensible EpcPropertyData shape (asset_rating/BER instead of
a SAP score, activities/hvac_systems instead of building elements), so
from_api_response now raises a typed NonDomesticSchema instead of
falling through to the generic "Unsupported EPC schema" ValueError.
Review feedback: cavity and internal wall insulation are competing options,
so a package selecting both read as nonsense even in a synthetic fixture.
The behaviour under test (a dependency triggered in both phases injects
once) now uses an airtightness pair that genuinely coexists — cavity wall
in phase 1, double glazing re-entering in phase 2 — with the same numbers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documents behaviour already delivered: with no SAP target both phases run
max-gain, so the envelope still consumes the budget first. Test passed on
arrival.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
