The new gov EPC API (api.get-energy-performance-data..., Bearer auth) returns
403 "Bad authentication header" with EPC_AUTH_TOKEN but 200 with
OPEN_EPC_API_TOKEN — the token name is misleading (it is the Bearer token for
the new API, not the open-data API). Verified live against
/api/domestic/search. Unblocks the live EPC fetch in run_modelling_e2e.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 5 (local run sources the DB, read-only) + slice 6 (optional persist),
landing together as one script rewrite (the persist path is interleaved with
the compute path).
The same local computation now runs whether or not the result is stored:
- Both modes price against the live `material` catalogue (read-only
ProductPostgresRepository over one shared Session) and model against a real
Scenario read from the DB (--scenario-id; its goal_value drives the band,
rejected if null) — so the inspected recommendations are exactly what gets
stored. The JSON sample catalogue is no longer used by this script.
- --measures restricts the run to a comma-separated considered_measures
allowlist (e.g. high_heat_retention_storage_heaters,solar_pv).
- --persist writes the inputs (EPC + spatial + solar) and the *same* computed
Plan via the production repos in one PostgresUnitOfWork, then commits
(idempotent: PlanPostgresRepository replaces by (property_id, scenario_id)).
Gated: --persist requires --scenario-id and --portfolio-id. Default is
inspect-only — no DB writes.
harness.console.run_modelling gains `products` and `scenario` overrides (the
seam the script drives); defaults unchanged, so existing callers are
unaffected. Suite 257 pass + 3 xfail; pyright clean; --help/guard/measure
parsing verified. Not yet executed against the DB (awaiting property_ids +
write-confirm).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ProductPostgresRepository.get took .first() with no ORDER BY, so when a
measure type has several active material rows (the live catalogue holds 74
solar_pv, 5 high_heat_retention_storage_heaters) the chosen row — hence the
cost and material_id — depended on the database's physical row order. Order by
id so a re-seed prices the same product every time.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add domain/modelling/considered_measures.py::restrict_to_considered_measures —
the pure allowlist that limits a run to a chosen set of MeasureType (mirroring
the legacy engine's `inclusions`). It filters at the Option level, so a
multi-option Recommendation (e.g. Heating & Hot Water competing HHRSH against
an ASHP bundle) is kept with only its allowed Options; a Recommendation left
with none is dropped. None = consider everything (unrestricted default).
Thread `considered_measures: frozenset[MeasureType] | None` through
ModellingOrchestrator.run -> _plan_for -> _scored_candidate_groups /
_candidate_recommendations (applies the filter) and _measure_dependencies
(suppresses a forced dependency whose required measure is outside the
allowlist, so a restricted run forces nothing it is not considering). The
local-run seam (harness.console.run_modelling) gains the same param.
The Optimiser still freely chooses among survivors — including none. Tests:
the pure filter (3 cases) + an orchestrator-seam test proving a
{solar_pv}-restricted run yields only solar_pv options. 257 pass + 3 xfail;
pyright clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tighten the recommendation/plan vocabulary off generic str:
MeasureOption.measure_type and PlanMeasure.measure_type are now MeasureType
(also _GlazingTarget.measure_type, MeasureDependency.triggers ->
frozenset[MeasureType], and the optimiser's chosen/required-type locals).
Because MeasureType is a StrEnum the change is transparent to persistence
(the `recommendation` varchar column), the optimiser group-by key, and every
`== "solar_pv"`-style comparison — so pyright now enforces the enum at every
construction site with no runtime behaviour change.
The catalogue boundary stays str: ProductRepository.get(measure_type: str)
and Product.measure_type are unchanged (they map arbitrary DB/JSON rows), so
the fake product repos in tests need no edit. Test construction helpers coerce
their str arg via MeasureType(...); direct constructions use members.
Suite green: tests/domain/modelling + orchestration + harness 253 pass + 3
xfail; pyright clean on production + tests (pre-existing moto + property-
override-rowcount baselines untouched).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce domain/modelling/measure_type.py — a StrEnum with one member per
modelled measure (the 15 the generators emit). A StrEnum so each member *is*
its string value: it persists straight into the `recommendation` varchar
column, is the optimiser's group-by key, and compares equal to the catalogue /
EPC strings — so it replaces the per-generator string constants with no
persistence or optimiser change.
Repoint every generator's measure-type constant/literal to a MeasureType
member (wall, solid_wall, roof, floor, glazing, lighting, ventilation,
heating, solar). Field annotations stay `str` for now; tightening them to
MeasureType is the next slice.
This is the enum the historical engine deferred (engine.py:970
"TODO - formalise property measure types into an enum") and the vocabulary the
forthcoming `considered_measures` allowlist will speak (mirroring the legacy
`inclusions`).
Suite green: tests/domain/modelling + orchestration + harness 253 pass + 3
xfail; pyright clean on the enum + generators.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per Property the inspection script now resolves the UPRN's spatial
reference from the Ordnance Survey Open-UPRN parquet in S3
(GeospatialS3Repository over a boto3 ParquetReader) and threads both
levers into run_modelling:
- planning_restrictions: the conservation/listed/heritage flags that gate
the wall + solar measures (ADR-0019/0020).
- solar_insights: a live Google Solar buildingInsights fetch keyed on the
reference coordinates, so the Solar PV Options can fire (ADR-0026).
Mirrors IngestionOrchestrator._fetch's coords->solar flow. Degrades
gracefully per Property: a UPRN S3 doesn't cover -> unrestricted/no-solar;
a point Google has no coverage for (BuildingInsightsNotFoundError) ->
no-solar; both still modelled. --no-solar skips the Google leg. A context
note (restrictions; solar) is printed and written to the md/csv summary.
Verified live: spatial_for + solar fetch round-trip on real UPRNs (S3 via
ambient ~/.aws creds, pyarrow reads parquet bytes). pyright clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Revives the local recommendation-inspection flow for specific Properties.
`scripts/run_modelling_e2e.py` reads each Property's UPRN from the DB
(read-only), fetches the latest EPC live from the gov EPC API by UPRN, runs the
Modelling stage in memory (all Generators → Optimiser → costed, attributed
Plan), and prints a per-Property plan table + writes a Markdown/CSV summary.
Persists nothing — purely for inspection.
The local DB's Properties have no linked ingested EPC (epc_property.property_id
is NULL for all rows; Ingestion's source clients are stubbed, #1136), so the
EPC must be fetched inline rather than read back. Builds the connection from the
`DB_*` env vars in backend/.env and the EPC token from `EPC_AUTH_TOKEN`.
Threads optional solar insights through harness `run_modelling` (so Solar PV
Options can fire once coordinates are wired) and adds the `solar_pv` catalogue
row. Solar + planning restrictions + DB persistence are noted follow-ups.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Publish container ports without a fixed host side in the backend and
asset_list compose files, and switch forwardPorts to service-qualified
form. Lets a second worktree's devcontainer spin up without conflicting
on host 8000/5432 (backend) or 8081 (asset_list), matching the pattern
in assessment-model-sales-frontend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pulls in 42 commits of calculator/mapper accuracy fixes from the per-cert
mapper-validation and floor/roof/heating fronts.
Conflict resolutions:
- mapper `_is_elmhurst_roof_window`: main dropped the branch's "wall location →
vertical" guard (it broke cert 000516's rooflight), but that re-broke cert
001431's two External-wall U>3.0 windows (which must stay vertical). The two
certs lodge a BYTE-IDENTICAL §11 row, so neither location nor U separates
them — the real discriminator is the room-in-roof context. Replaced the
unconditional U>3.0 backstop with one gated on the BP having a room-in-roof
(`_elmhurst_bp_has_room_in_roof`): 000516's Main BP has a "Room in roof type
1" (→ rooflight), 001431's does not (→ vertical). Validated against BOTH —
full Elmhurst worksheet suite 1038 pass + the 001431 window-extraction pin.
- property_postgres_repository: kept main's `ids_by_uprn` method + the branch's
`_restrictions_of` helper.
- sap_fuel.py: the branch relocated it to domain/billing/ (already carrying
main's to_table_32_code normalization), so kept the old path deleted.
Fallout from main's fabric fixes (validated by the boiler-3 real-cert pin which
still reproduces at delta 0):
- re-pinned the boiler-1 + boiler-instant-hw ASHP snapshot scores;
- main's §14.2 gas-boiler main-fuel derivation resolved the BGB/102 baseline
gap, so `test_gas_boiler_instant_hw_before_baselines` is now a passing test
(was an xfail tripwire).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 9 of the Solar PV Recommendation Generator (ADR-0026). Pins the
overlay→calculator PV cascade against Elmhurst's before/after re-lodgements of
cert 001431, across the orientation/pitch/overshading config space (the certs
lodge synthetic 1.00 kWp test vectors):
- SE/SW, shaded (overshading 3/2), pitch 30°/45°
- E/W, unshaded, pitch 60°/45°
- NW/N, unshaded, pitch 60°/45° (the low-yield orientations)
Each hand-built SolarOverlay reproduces the relodged after at abs ≤ 1e-4 on
SAP / CO2 / primary energy.
Battery tripwire (per user): the "with battery" cert lodges a §19 5 kWh battery
the current extractor does NOT parse, so it scores identically to its
no-battery twin — the no-battery overlay reproduces it today, and the pin will
fail (alerting us to switch to the with-battery overlay) once the extractor
parses the battery. A companion test pins that the calculator already models
the 5 kWh battery (it raises SAP), so the fix target is established.
All five certs share an EES 'WGK' / SAP-code-502 main-heating lodgement the
mapper doesn't yet derive a fuel for; the pins patch the shared fuel (mains gas
26) identically on before+after to isolate the PV delta (the solar overlay
never touches heating), and `test_solar_before_baselines` xfails as the
forcing-function tripwire for that separate mapper-front gap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 8 of the Solar PV Recommendation Generator (ADR-0026). The
ModellingOrchestrator now reads each Property's persisted Google Solar
buildingInsights JSON (uow.solar), projects it once per Property into a typed
SolarPotential via `_solar_potential_for` (None for a missing or error
payload), and threads it into `recommend_solar` alongside planning_restrictions
— mirroring the ASHP wiring. Solar fires only when a feasible potential is
present, so dwellings without fetched solar data are unaffected.
FakeSolarRepo now returns None for an unseeded Property (was raising) and
supports `by_property` seeding, so the orchestrator's new solar read is exercised.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 6 of the Solar PV Recommendation Generator (ADR-0026). `recommend_solar`
emits one "Solar PV" Recommendation of up to five conservatively-sized configs
× {no battery, battery} = ≤10 competing Options (a free Optimiser candidate).
Each Option folds a SolarOverlay built from the chosen config: one
PhotovoltaicArray per non-north segment (peak_power = panels × panelCapacityW /
1000; orientation/pitch from geometry; generation-calibrated overshading),
is_dwelling_export_capable set True absolutely, a diverter when the dwelling
has a cylinder (None for a combi), a 5 kWh battery for the battery variant, and
the per-config composite cost from Products.solar_bundle_cost.
Eligibility = house/bungalow ∧ not listed/heritage (blocks_internal, the same
gate as ASHP — a conservation area does NOT block PV) ∧ no existing PV ∧ a
feasible SolarPotential. Flats and existing-PV top-up are deferred.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 7 of the Solar PV Recommendation Generator (ADR-0026). Adds the
composite per-dwelling Solar PV cost on the Products collection (ADR-0025
pattern): pv_system(kWp band, nearest of the ECOPV06-13 EA bands 1.0→4.5 kWp,
floor/cap at the ends) + scaffolding(£900 first elevation + £450 each
additional, default 2) + enabling base (EICR £150 + DNO £50 + 2-way consumer
unit £330) + [diverter £980 if cylinder] + [battery if the with-battery
variant] → Cost(total, contingency_rate 0.15).
Rates are data in the committed solar_rates.json (Southern Housing "SOLAR PV &
BATTERY" EA column), loaded via SolarRates.from_json/.default and injectable.
The £2,000 / 5 kWh battery is NOT on the rate sheet — a flagged estimate
(battery_estimate=true), confirmed with the user to stand in until a DB rate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 5 of the Solar PV Recommendation Generator (ADR-0026). Adds the flat
`SolarOverlay` and `_fold_solar`, the sixth Simulation Overlay surface: like
the ventilation/lighting overlays it targets no building part and folds its
fields onto `sap_energy_source` (home of the SAP Appendix M PV inputs) —
photovoltaic_arrays (absolute target, one PhotovoltaicArray per non-north
segment, replacing the dwelling's existing arrays), pv_diverter_present,
pv_connection, is_dwelling_export_capable (set True absolutely), pv_batteries.
Omitted fields leave the baseline unchanged (combi → no diverter); the
baseline is never mutated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 4 of the Solar PV Recommendation Generator (ADR-0026).
`select_conservative_configs` turns Google's full solarPanelConfigs ladder
into up to five competing array configs for the Optimiser: drop north-facing
planes (within 30° of due north, wrap-aware), cap usable panels at ~70% of
maxArrayPanelsCount (imagery misses obstructions; MCS edge setback), collapse
rungs that trim to the same usable size keeping the higher-generation layout,
then sample five spanning min→max by expected generation. Returns () when
nothing usable remains.
Real London example → 5 rungs at 4/12/19/26/34 panels (all ≤34.3 = 70% of
49); synthetic cases pin the north-drop and the 70% cap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3 of the Solar PV Recommendation Generator (ADR-0026). Per roof segment,
back-solve the effective overshading factor ZPV from Google's expected
generation against SAP's own unshaded annual output:
ZPV = (yearlyEnergyDcKwh × 0.955) / (0.8 × kWp × S)
reusing the calculator's Appendix U3.3 annual solar radiation S via a new
public seam `pv_annual_solar_radiation_kwh_per_m2`. Dividing Google's
generation by SAP's S cancels orientation/tilt and isolates shading; the
result snaps to the RdSAP bucket {1:1.0, 2:0.8, 3:0.5, 4:0.35} via the
ADR-0026 midpoint cutpoints (≥0.90→1, 0.65–0.90→2, 0.425–0.65→3, <0.425→4;
ZPV>1→1). The real London example's planes all back-solve to ZPV>1 → code 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 2 of the Solar PV Recommendation Generator (ADR-0026). Adds the
strictly-typed `SolarPotential` domain projection over the raw Google Solar
`buildingInsights` JSON that Ingestion persists (SolarRepository): the
`solarPanelConfigs` ladder, each rung broken into its roof segments with
Google's continuous azimuth/tilt mapped to the SAP octant
(`azimuth_to_sap_octant`, 0°=N clockwise → 1=N..8=NW, matching the
calculator's ORIENTATION_BY_SAP10_CODE) and RdSAP §11.1 pitch code
(`pitch_to_sap_code`, snap to {0→1,30→2,45→3,60→4,90→5}).
Pinned against the real London buildingInsights example (mirrored into
fixtures from the user-provided RTF): 400 W panels, maxArrayPanelsCount 49,
46-rung ladder, per-segment SE/NW/NE/SW octants at ~32° → pitch code 2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Elmhurst §15.1 "Insulated: No Insulation" label was lodged but absent
from `_ELMHURST_CYLINDER_INSULATION_LABEL_TO_SAP10`, so
`_elmhurst_cylinder_insulation_code` raised UnmappedElmhurstLabel — blocking
the parse of every Solar PV example cert (the solar `before` lodges "No
Insulation"). An uninsulated cylinder has no insulation *type*, so per the
no-misleading-insulation convention it maps to `cylinder_insulation_type =
None` rather than naming a material; the lodged 0 mm thickness carries the
storage-loss signal the SAP 10.2 Table 2 dispatch needs.
Slice 1 of the Solar PV Recommendation Generator (ADR-0026).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Designed via grill-with-docs. API-first (Google Solar -> typed SolarPotential):
one Solar PV Recommendation, up to 5 conservatively-sized configs x battery
on/off; new PV overlay surface (per-roof-segment arrays, diverter conditional on
cylinder, export-capable ensured); overshading generation-calibrated from the
API's expected generation; eligibility offers conservation areas (only listed/
heritage block); composite costing from EA rates (contingency 0.15). CONTEXT
gains Solar Potential, Solar PV Recommendation, Solar PV Eligibility.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code 3 = "(other premises below)" = above partially heated space (§3.12 →
U=0.7), confirmed 9/9 on single-BP certs (the diagnostic that dodged the
lossy-floors[] contamination). Records the 7536 re-pin and the lesson that
"irreducible residual" golden notes can mask a real mapper bug.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The API `floor_heat_loss` code is authoritative — confirmed by joining each
single-BP cert's code to its independent `floors[].description` (which the
gov register publishes alongside the code):
code 1 ↔ "To external air" (exposed, 9/9)
code 2 ↔ "To unheated space" (semi-exposed, 6/6)
code 3 ↔ "(other premises below)" (partially htd, 9/9)
code 6 ↔ "(another dwelling below)" (party, 176/176)
code 7 ↔ "Solid"/"Suspended …" (ground, all)
Code 3 was mis-mapped to "To unheated space" (semi-exposed) and, on
mid-/top-floor flats, had its floor area zeroed entirely by the
dwelling-level exposure heuristic. RdSAP 10 §3.12 (PDF p.25) classes a
flat's floor over non-domestic "other premises … heated, but at different
times" as "above a partially heated space" → the §5.14 (PDF p.47) constant
U=0.7 W/m²K — distinct from semi-exposed (Table 20) and party (no loss).
Fix: the mapper sets `is_above_partially_heated_space` on the floor=0
dimension for code 3 (string → "(other premises below)" for fidelity), and
the heat-transmission step lets that per-BP lodgement override the flat
suppression upward (mirroring the existing exposed / "another dwelling
below" overrides). The cascade already routes is_above_partial → U=0.7.
Re-pins golden cert 7536-3827: its Ext2 (bp3) lodges code 3, but the cert's
lossy `floors[]` summary dropped that description, so a prior agent guessed
"code 3 = ground" (U=1.12) and concluded the residual was an irreducible
"register-rounding" artifact. It was this bug: Ext2 floor U 1.12 → 0.70,
PE -6.1952 → -5.6414, CO2 -0.1639 → -0.1492 (both toward 0), SAP unchanged.
Eval: 909 computed, 45.1% → 45.3% within 0.5, mean|err| 1.702 → 1.659,
<1.0 59.5% → 60.2%. 13 code-3 certs improve (0380 +3.71 → -0.63, 0350
+7.82 → +0.83, 2610 +7.47 → -1.29); the few that overshoot were already
failing and carry independent fabric bugs (9763's walls = 8 W/K for 60 m²).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Record that profiler lead #1 (floor_codes=3) is not a clean single cause
(bimodal + confounded), that the paired worksheet certs confirm only
codes 1/6/7 (code 3 unmapped → needs a worksheet for 0380-2087), and that
immersion_type=2 / main_control=2107 / roof_codes=1 are scatter, not
dispatch bugs. The exposed-floor-on-flats fix (§3.12) shipped at b40e0f67.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A mid-/top-floor flat whose lowest floor is lodged as an exposed floor
(API floor_heat_loss=1) had its floor area zeroed by the dwelling-level
exposure heuristic, which keys only on the flat label and defaults
has_exposed_floor=False (assuming the floor sits over another *heated*
dwelling). RdSAP 10 §3.12 (PDF p.25) is explicit:
"Otherwise the floor area of the flat ... is:
- an exposed floor if there is an open space below"
i.e. a flat cantilevered over a passageway IS a heat-loss floor on
Table 20. The per-BP `is_exposed_floor` lodgement is authoritative and
now overrides the dwelling-level suppression upward, mirroring the
existing "another dwelling below" party override (which suppresses
downward). The code-1↔"E To external air" enum is confirmed by the
paired API+Summary worksheet certs (0350, 3800).
Eval: 45.1% → 45.3% within 0.5 (909 computed); cert 3836 +6.79 → +0.77,
5717 +1.31 → -0.07 and 0997 +0.76 → +0.05 cross into <0.5. Two
already-failing under-rated certs (7636, 2241) shift further — both are
dominated by independent cost-side over-counts the exposed floor merely
unmasks (7636 walls = 8.98 W/K for 33.87 m² is the real defect).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Joins each computed cert's signed error (eval _results.csv) with a rich
feature set extracted from its RAW API JSON (not the mapped
EpcPropertyData), then ranks (feature, value) buckets by error carried
and by |mean signed| bias. Surfaces systematic API-path handling gaps —
a field the mapper silently drops still shows as an error-correlated
bucket. Companion to eval_api_sap_accuracy.py / decompose_api_cost_error.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
main_heating_category=9 (warm-air systems, NOT heat pump) had no entry
in _SECONDARY_HEATING_FRACTION_BY_CATEGORY, so a warm-air main with a
lodged secondary raised UnmappedSapCode in
_secondary_heating_fraction_for_category — the last calc_raise in the
API sample (cert 0380-2197-2590-2996-2715: warm air mains gas code 506 +
electric room-heater secondary).
SAP 10.2 Table 11 (p.188): a gas/oil warm-air unit falls under "All gas,
liquid and solid fuel systems" (0.10), and electric warm air under
"Other electric systems" (also 0.10) — so 0.10 regardless of fuel. The
warm-air efficiency (Table 4a code→eff: 506→0.70) and Table 4f fan
energy were already wired; this was the only missing dispatch entry.
0380 now computes: SAP 78.1 vs lodged 77 (+1.1; the residual is per-cert
fabric/PV, not the warm-air dispatch — a faithful 0380 worksheet isn't
available, sim case 28 diverges at SAP 57 / code 502 / condensing unit).
Eval: zero raises remain, computed 908→909; mean|err| 1.703→1.702.
Regression green (2448 pass incl. golden 6035 + cohort); pyright
net-zero (44=44).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The boiler-1 and boiler-2 after-certs predate the Vaillant product swap (they
lodge the old aroTHERM index 101413), so rather than wait on regenerated certs
these now snapshot the Vaillant overlay's own output on the before (SAP/CO2/PE
at 1e-4), taken as correct because the same overlay reproduces the corrected
Vaillant cert at delta 0 in the boiler-3 (system-boiler) pin. Drops the xfail
markers and the two stale, now-unreferenced after-cert fixtures.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
