ADR-0027 Reduced-Field Synthesis: certs with no per-window array now get total
glazing = 0.148 x TFA x band-multiplier (median + quartile multipliers fit from
the 1000 real 21.0.1 certs), split 4-way across N/E/S/W with width=area/4,
height=1.0; glazing_type routed through the verified 21.0.1 cascade. Also guard
optional PhotovoltaicSupply.none_or_no_details (a parse straggler). Corpus maps
983/1000, up from 974.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Required->optional defaults (kw_only + data-driven from corpus presence) so
993/1000 certs that omit sap_windows parse, and honest Union[str, DescriptionV1]
typing for description/dwelling_type which the corpus lodges as localised dicts
in ~half the certs. The never-run 20.0.0 mapper path now produces EpcPropertyData;
974/1000 corpus certs map (xpass), up from 7.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sharpen the glossary to decouple deterministic old-schema re-mapping from
neighbour-prediction gap-fill (a separate, unimplemented ML path), and add
the Reduced-Field Synthesis term. ADR-0027 records the pre-SAP10 20.0.0
mapper's best-attempt synthesis (corpus-fit glazing 0.148xTFAxband, 4-way
orientation) and its trade-offs. Grill resume doc captures every resolved branch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new gov EPC API (api.get-energy-performance-data..., Bearer auth) returns
403 "Bad authentication header" with EPC_AUTH_TOKEN but 200 with
OPEN_EPC_API_TOKEN — the token name is misleading (it is the Bearer token for
the new API, not the open-data API). Verified live against
/api/domestic/search. Unblocks the live EPC fetch in run_modelling_e2e.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 5 (local run sources the DB, read-only) + slice 6 (optional persist),
landing together as one script rewrite (the persist path is interleaved with
the compute path).
The same local computation now runs whether or not the result is stored:
- Both modes price against the live `material` catalogue (read-only
ProductPostgresRepository over one shared Session) and model against a real
Scenario read from the DB (--scenario-id; its goal_value drives the band,
rejected if null) — so the inspected recommendations are exactly what gets
stored. The JSON sample catalogue is no longer used by this script.
- --measures restricts the run to a comma-separated considered_measures
allowlist (e.g. high_heat_retention_storage_heaters,solar_pv).
- --persist writes the inputs (EPC + spatial + solar) and the *same* computed
Plan via the production repos in one PostgresUnitOfWork, then commits
(idempotent: PlanPostgresRepository replaces by (property_id, scenario_id)).
Gated: --persist requires --scenario-id and --portfolio-id. Default is
inspect-only — no DB writes.
harness.console.run_modelling gains `products` and `scenario` overrides (the
seam the script drives); defaults unchanged, so existing callers are
unaffected. Suite 257 pass + 3 xfail; pyright clean; --help/guard/measure
parsing verified. Not yet executed against the DB (awaiting property_ids +
write-confirm).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ProductPostgresRepository.get took .first() with no ORDER BY, so when a
measure type has several active material rows (the live catalogue holds 74
solar_pv, 5 high_heat_retention_storage_heaters) the chosen row — hence the
cost and material_id — depended on the database's physical row order. Order by
id so a re-seed prices the same product every time.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add domain/modelling/considered_measures.py::restrict_to_considered_measures —
the pure allowlist that limits a run to a chosen set of MeasureType (mirroring
the legacy engine's `inclusions`). It filters at the Option level, so a
multi-option Recommendation (e.g. Heating & Hot Water competing HHRSH against
an ASHP bundle) is kept with only its allowed Options; a Recommendation left
with none is dropped. None = consider everything (unrestricted default).
Thread `considered_measures: frozenset[MeasureType] | None` through
ModellingOrchestrator.run -> _plan_for -> _scored_candidate_groups /
_candidate_recommendations (applies the filter) and _measure_dependencies
(suppresses a forced dependency whose required measure is outside the
allowlist, so a restricted run forces nothing it is not considering). The
local-run seam (harness.console.run_modelling) gains the same param.
The Optimiser still freely chooses among survivors — including none. Tests:
the pure filter (3 cases) + an orchestrator-seam test proving a
{solar_pv}-restricted run yields only solar_pv options. 257 pass + 3 xfail;
pyright clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tighten the recommendation/plan vocabulary off generic str:
MeasureOption.measure_type and PlanMeasure.measure_type are now MeasureType
(also _GlazingTarget.measure_type, MeasureDependency.triggers ->
frozenset[MeasureType], and the optimiser's chosen/required-type locals).
Because MeasureType is a StrEnum the change is transparent to persistence
(the `recommendation` varchar column), the optimiser group-by key, and every
`== "solar_pv"`-style comparison — so pyright now enforces the enum at every
construction site with no runtime behaviour change.
The catalogue boundary stays str: ProductRepository.get(measure_type: str)
and Product.measure_type are unchanged (they map arbitrary DB/JSON rows), so
the fake product repos in tests need no edit. Test construction helpers coerce
their str arg via MeasureType(...); direct constructions use members.
Suite green: tests/domain/modelling + orchestration + harness 253 pass + 3
xfail; pyright clean on production + tests (pre-existing moto + property-
override-rowcount baselines untouched).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce domain/modelling/measure_type.py — a StrEnum with one member per
modelled measure (the 15 the generators emit). A StrEnum so each member *is*
its string value: it persists straight into the `recommendation` varchar
column, is the optimiser's group-by key, and compares equal to the catalogue /
EPC strings — so it replaces the per-generator string constants with no
persistence or optimiser change.
Repoint every generator's measure-type constant/literal to a MeasureType
member (wall, solid_wall, roof, floor, glazing, lighting, ventilation,
heating, solar). Field annotations stay `str` for now; tightening them to
MeasureType is the next slice.
This is the enum the historical engine deferred (engine.py:970
"TODO - formalise property measure types into an enum") and the vocabulary the
forthcoming `considered_measures` allowlist will speak (mirroring the legacy
`inclusions`).
Suite green: tests/domain/modelling + orchestration + harness 253 pass + 3
xfail; pyright clean on the enum + generators.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per Property the inspection script now resolves the UPRN's spatial
reference from the Ordnance Survey Open-UPRN parquet in S3
(GeospatialS3Repository over a boto3 ParquetReader) and threads both
levers into run_modelling:
- planning_restrictions: the conservation/listed/heritage flags that gate
the wall + solar measures (ADR-0019/0020).
- solar_insights: a live Google Solar buildingInsights fetch keyed on the
reference coordinates, so the Solar PV Options can fire (ADR-0026).
Mirrors IngestionOrchestrator._fetch's coords->solar flow. Degrades
gracefully per Property: a UPRN S3 doesn't cover -> unrestricted/no-solar;
a point Google has no coverage for (BuildingInsightsNotFoundError) ->
no-solar; both still modelled. --no-solar skips the Google leg. A context
note (restrictions; solar) is printed and written to the md/csv summary.
Verified live: spatial_for + solar fetch round-trip on real UPRNs (S3 via
ambient ~/.aws creds, pyarrow reads parquet bytes). pyright clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Revives the local recommendation-inspection flow for specific Properties.
`scripts/run_modelling_e2e.py` reads each Property's UPRN from the DB
(read-only), fetches the latest EPC live from the gov EPC API by UPRN, runs the
Modelling stage in memory (all Generators → Optimiser → costed, attributed
Plan), and prints a per-Property plan table + writes a Markdown/CSV summary.
Persists nothing — purely for inspection.
The local DB's Properties have no linked ingested EPC (epc_property.property_id
is NULL for all rows; Ingestion's source clients are stubbed, #1136), so the
EPC must be fetched inline rather than read back. Builds the connection from the
`DB_*` env vars in backend/.env and the EPC token from `EPC_AUTH_TOKEN`.
Threads optional solar insights through harness `run_modelling` (so Solar PV
Options can fire once coordinates are wired) and adds the `solar_pv` catalogue
row. Solar + planning restrictions + DB persistence are noted follow-ups.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Publish container ports without a fixed host side in the backend and
asset_list compose files, and switch forwardPorts to service-qualified
form. Lets a second worktree's devcontainer spin up without conflicting
on host 8000/5432 (backend) or 8081 (asset_list), matching the pattern
in assessment-model-sales-frontend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pulls in 42 commits of calculator/mapper accuracy fixes from the per-cert
mapper-validation and floor/roof/heating fronts.
Conflict resolutions:
- mapper `_is_elmhurst_roof_window`: main dropped the branch's "wall location →
vertical" guard (it broke cert 000516's rooflight), but that re-broke cert
001431's two External-wall U>3.0 windows (which must stay vertical). The two
certs lodge a BYTE-IDENTICAL §11 row, so neither location nor U separates
them — the real discriminator is the room-in-roof context. Replaced the
unconditional U>3.0 backstop with one gated on the BP having a room-in-roof
(`_elmhurst_bp_has_room_in_roof`): 000516's Main BP has a "Room in roof type
1" (→ rooflight), 001431's does not (→ vertical). Validated against BOTH —
full Elmhurst worksheet suite 1038 pass + the 001431 window-extraction pin.
- property_postgres_repository: kept main's `ids_by_uprn` method + the branch's
`_restrictions_of` helper.
- sap_fuel.py: the branch relocated it to domain/billing/ (already carrying
main's to_table_32_code normalization), so kept the old path deleted.
Fallout from main's fabric fixes (validated by the boiler-3 real-cert pin which
still reproduces at delta 0):
- re-pinned the boiler-1 + boiler-instant-hw ASHP snapshot scores;
- main's §14.2 gas-boiler main-fuel derivation resolved the BGB/102 baseline
gap, so `test_gas_boiler_instant_hw_before_baselines` is now a passing test
(was an xfail tripwire).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 9 of the Solar PV Recommendation Generator (ADR-0026). Pins the
overlay→calculator PV cascade against Elmhurst's before/after re-lodgements of
cert 001431, across the orientation/pitch/overshading config space (the certs
lodge synthetic 1.00 kWp test vectors):
- SE/SW, shaded (overshading 3/2), pitch 30°/45°
- E/W, unshaded, pitch 60°/45°
- NW/N, unshaded, pitch 60°/45° (the low-yield orientations)
Each hand-built SolarOverlay reproduces the relodged after at abs ≤ 1e-4 on
SAP / CO2 / primary energy.
Battery tripwire (per user): the "with battery" cert lodges a §19 5 kWh battery
the current extractor does NOT parse, so it scores identically to its
no-battery twin — the no-battery overlay reproduces it today, and the pin will
fail (alerting us to switch to the with-battery overlay) once the extractor
parses the battery. A companion test pins that the calculator already models
the 5 kWh battery (it raises SAP), so the fix target is established.
All five certs share an EES 'WGK' / SAP-code-502 main-heating lodgement the
mapper doesn't yet derive a fuel for; the pins patch the shared fuel (mains gas
26) identically on before+after to isolate the PV delta (the solar overlay
never touches heating), and `test_solar_before_baselines` xfails as the
forcing-function tripwire for that separate mapper-front gap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 8 of the Solar PV Recommendation Generator (ADR-0026). The
ModellingOrchestrator now reads each Property's persisted Google Solar
buildingInsights JSON (uow.solar), projects it once per Property into a typed
SolarPotential via `_solar_potential_for` (None for a missing or error
payload), and threads it into `recommend_solar` alongside planning_restrictions
— mirroring the ASHP wiring. Solar fires only when a feasible potential is
present, so dwellings without fetched solar data are unaffected.
FakeSolarRepo now returns None for an unseeded Property (was raising) and
supports `by_property` seeding, so the orchestrator's new solar read is exercised.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 6 of the Solar PV Recommendation Generator (ADR-0026). `recommend_solar`
emits one "Solar PV" Recommendation of up to five conservatively-sized configs
× {no battery, battery} = ≤10 competing Options (a free Optimiser candidate).
Each Option folds a SolarOverlay built from the chosen config: one
PhotovoltaicArray per non-north segment (peak_power = panels × panelCapacityW /
1000; orientation/pitch from geometry; generation-calibrated overshading),
is_dwelling_export_capable set True absolutely, a diverter when the dwelling
has a cylinder (None for a combi), a 5 kWh battery for the battery variant, and
the per-config composite cost from Products.solar_bundle_cost.
Eligibility = house/bungalow ∧ not listed/heritage (blocks_internal, the same
gate as ASHP — a conservation area does NOT block PV) ∧ no existing PV ∧ a
feasible SolarPotential. Flats and existing-PV top-up are deferred.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 7 of the Solar PV Recommendation Generator (ADR-0026). Adds the
composite per-dwelling Solar PV cost on the Products collection (ADR-0025
pattern): pv_system(kWp band, nearest of the ECOPV06-13 EA bands 1.0→4.5 kWp,
floor/cap at the ends) + scaffolding(£900 first elevation + £450 each
additional, default 2) + enabling base (EICR £150 + DNO £50 + 2-way consumer
unit £330) + [diverter £980 if cylinder] + [battery if the with-battery
variant] → Cost(total, contingency_rate 0.15).
Rates are data in the committed solar_rates.json (Southern Housing "SOLAR PV &
BATTERY" EA column), loaded via SolarRates.from_json/.default and injectable.
The £2,000 / 5 kWh battery is NOT on the rate sheet — a flagged estimate
(battery_estimate=true), confirmed with the user to stand in until a DB rate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 5 of the Solar PV Recommendation Generator (ADR-0026). Adds the flat
`SolarOverlay` and `_fold_solar`, the sixth Simulation Overlay surface: like
the ventilation/lighting overlays it targets no building part and folds its
fields onto `sap_energy_source` (home of the SAP Appendix M PV inputs) —
photovoltaic_arrays (absolute target, one PhotovoltaicArray per non-north
segment, replacing the dwelling's existing arrays), pv_diverter_present,
pv_connection, is_dwelling_export_capable (set True absolutely), pv_batteries.
Omitted fields leave the baseline unchanged (combi → no diverter); the
baseline is never mutated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 4 of the Solar PV Recommendation Generator (ADR-0026).
`select_conservative_configs` turns Google's full solarPanelConfigs ladder
into up to five competing array configs for the Optimiser: drop north-facing
planes (within 30° of due north, wrap-aware), cap usable panels at ~70% of
maxArrayPanelsCount (imagery misses obstructions; MCS edge setback), collapse
rungs that trim to the same usable size keeping the higher-generation layout,
then sample five spanning min→max by expected generation. Returns () when
nothing usable remains.
Real London example → 5 rungs at 4/12/19/26/34 panels (all ≤34.3 = 70% of
49); synthetic cases pin the north-drop and the 70% cap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3 of the Solar PV Recommendation Generator (ADR-0026). Per roof segment,
back-solve the effective overshading factor ZPV from Google's expected
generation against SAP's own unshaded annual output:
ZPV = (yearlyEnergyDcKwh × 0.955) / (0.8 × kWp × S)
reusing the calculator's Appendix U3.3 annual solar radiation S via a new
public seam `pv_annual_solar_radiation_kwh_per_m2`. Dividing Google's
generation by SAP's S cancels orientation/tilt and isolates shading; the
result snaps to the RdSAP bucket {1:1.0, 2:0.8, 3:0.5, 4:0.35} via the
ADR-0026 midpoint cutpoints (≥0.90→1, 0.65–0.90→2, 0.425–0.65→3, <0.425→4;
ZPV>1→1). The real London example's planes all back-solve to ZPV>1 → code 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 2 of the Solar PV Recommendation Generator (ADR-0026). Adds the
strictly-typed `SolarPotential` domain projection over the raw Google Solar
`buildingInsights` JSON that Ingestion persists (SolarRepository): the
`solarPanelConfigs` ladder, each rung broken into its roof segments with
Google's continuous azimuth/tilt mapped to the SAP octant
(`azimuth_to_sap_octant`, 0°=N clockwise → 1=N..8=NW, matching the
calculator's ORIENTATION_BY_SAP10_CODE) and RdSAP §11.1 pitch code
(`pitch_to_sap_code`, snap to {0→1,30→2,45→3,60→4,90→5}).
Pinned against the real London buildingInsights example (mirrored into
fixtures from the user-provided RTF): 400 W panels, maxArrayPanelsCount 49,
46-rung ladder, per-segment SE/NW/NE/SW octants at ~32° → pitch code 2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Elmhurst §15.1 "Insulated: No Insulation" label was lodged but absent
from `_ELMHURST_CYLINDER_INSULATION_LABEL_TO_SAP10`, so
`_elmhurst_cylinder_insulation_code` raised UnmappedElmhurstLabel — blocking
the parse of every Solar PV example cert (the solar `before` lodges "No
Insulation"). An uninsulated cylinder has no insulation *type*, so per the
no-misleading-insulation convention it maps to `cylinder_insulation_type =
None` rather than naming a material; the lodged 0 mm thickness carries the
storage-loss signal the SAP 10.2 Table 2 dispatch needs.
Slice 1 of the Solar PV Recommendation Generator (ADR-0026).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Designed via grill-with-docs. API-first (Google Solar -> typed SolarPotential):
one Solar PV Recommendation, up to 5 conservatively-sized configs x battery
on/off; new PV overlay surface (per-roof-segment arrays, diverter conditional on
cylinder, export-capable ensured); overshading generation-calibrated from the
API's expected generation; eligibility offers conservation areas (only listed/
heritage block); composite costing from EA rates (contingency 0.15). CONTEXT
gains Solar Potential, Solar PV Recommendation, Solar PV Eligibility.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code 3 = "(other premises below)" = above partially heated space (§3.12 →
U=0.7), confirmed 9/9 on single-BP certs (the diagnostic that dodged the
lossy-floors[] contamination). Records the 7536 re-pin and the lesson that
"irreducible residual" golden notes can mask a real mapper bug.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The API `floor_heat_loss` code is authoritative — confirmed by joining each
single-BP cert's code to its independent `floors[].description` (which the
gov register publishes alongside the code):
code 1 ↔ "To external air" (exposed, 9/9)
code 2 ↔ "To unheated space" (semi-exposed, 6/6)
code 3 ↔ "(other premises below)" (partially htd, 9/9)
code 6 ↔ "(another dwelling below)" (party, 176/176)
code 7 ↔ "Solid"/"Suspended …" (ground, all)
Code 3 was mis-mapped to "To unheated space" (semi-exposed) and, on
mid-/top-floor flats, had its floor area zeroed entirely by the
dwelling-level exposure heuristic. RdSAP 10 §3.12 (PDF p.25) classes a
flat's floor over non-domestic "other premises … heated, but at different
times" as "above a partially heated space" → the §5.14 (PDF p.47) constant
U=0.7 W/m²K — distinct from semi-exposed (Table 20) and party (no loss).
Fix: the mapper sets `is_above_partially_heated_space` on the floor=0
dimension for code 3 (string → "(other premises below)" for fidelity), and
the heat-transmission step lets that per-BP lodgement override the flat
suppression upward (mirroring the existing exposed / "another dwelling
below" overrides). The cascade already routes is_above_partial → U=0.7.
Re-pins golden cert 7536-3827: its Ext2 (bp3) lodges code 3, but the cert's
lossy `floors[]` summary dropped that description, so a prior agent guessed
"code 3 = ground" (U=1.12) and concluded the residual was an irreducible
"register-rounding" artifact. It was this bug: Ext2 floor U 1.12 → 0.70,
PE -6.1952 → -5.6414, CO2 -0.1639 → -0.1492 (both toward 0), SAP unchanged.
Eval: 909 computed, 45.1% → 45.3% within 0.5, mean|err| 1.702 → 1.659,
<1.0 59.5% → 60.2%. 13 code-3 certs improve (0380 +3.71 → -0.63, 0350
+7.82 → +0.83, 2610 +7.47 → -1.29); the few that overshoot were already
failing and carry independent fabric bugs (9763's walls = 8 W/K for 60 m²).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Record that profiler lead #1 (floor_codes=3) is not a clean single cause
(bimodal + confounded), that the paired worksheet certs confirm only
codes 1/6/7 (code 3 unmapped → needs a worksheet for 0380-2087), and that
immersion_type=2 / main_control=2107 / roof_codes=1 are scatter, not
dispatch bugs. The exposed-floor-on-flats fix (§3.12) shipped at b40e0f67.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A mid-/top-floor flat whose lowest floor is lodged as an exposed floor
(API floor_heat_loss=1) had its floor area zeroed by the dwelling-level
exposure heuristic, which keys only on the flat label and defaults
has_exposed_floor=False (assuming the floor sits over another *heated*
dwelling). RdSAP 10 §3.12 (PDF p.25) is explicit:
"Otherwise the floor area of the flat ... is:
- an exposed floor if there is an open space below"
i.e. a flat cantilevered over a passageway IS a heat-loss floor on
Table 20. The per-BP `is_exposed_floor` lodgement is authoritative and
now overrides the dwelling-level suppression upward, mirroring the
existing "another dwelling below" party override (which suppresses
downward). The code-1↔"E To external air" enum is confirmed by the
paired API+Summary worksheet certs (0350, 3800).
Eval: 45.1% → 45.3% within 0.5 (909 computed); cert 3836 +6.79 → +0.77,
5717 +1.31 → -0.07 and 0997 +0.76 → +0.05 cross into <0.5. Two
already-failing under-rated certs (7636, 2241) shift further — both are
dominated by independent cost-side over-counts the exposed floor merely
unmasks (7636 walls = 8.98 W/K for 33.87 m² is the real defect).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
