Add Property.physical_state_changed (true on Site Notes / Landlord Overrides
/ Prediction — trigger (b)/(c)) and pass it from the
PropertyBaselineOrchestrator into the Rebaseliner. So an overridden or
predicted SAP>=10.2 property now stores calc(effective) as its Effective
baseline instead of echoing the lodged headline — closing the "81 in the DB"
divergence between the displayed baseline and the modelled plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fixture lodges is_exposed_floor=True, which was silently dropped on
save->reload before this branch — so the orchestrator (reads the DB) modelled
the floor as not-exposed. Now it round-trips, the exposed floor is honoured,
and the ASHP-led max-gain fallback on this gas dwelling is bill-neutral
(marginally negative) rather than bill-positive: large energy saving, but the
gas->electricity fuel switch offsets the GBP saving. Same optimal package,
same telescoping; only the bill-savings sign assumption changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Even after batching the data writes, the handler still wrote to the DB per
property through the orchestrator's SubTask bookkeeping: create + start +
complete each self-committed, and _cascade re-listed every sibling and re-saved
the parent on every transition — ~5 writes per property plus an O(N^2) cascade.
- TaskOrchestrator.run_subtasks: create all children in one INSERT, run each
(failures isolated per child), then persist all terminal states in one bulk
save and cascade the parent once. Children go WAITING -> terminal; the
transient IN_PROGRESS row is never written.
- SubTaskRepository.create_many / save_many (bulk INSERT / bulk fetch + update).
- _cascade short-circuits when the Task is already FAILED (terminal) — skips the
sibling roll-up entirely.
- modelling_e2e handler fans out via run_subtasks instead of per-property
create_child_subtask + run_subtask.
Per N-property batch the SubTask bookkeeping drops from ~5N writes + an O(N^2)
cascade to ~2 writes + 1 cascade.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
16 modelling_e2e properties failed with "Effective EPC is missing
renewable_heat_incentive; cannot read baseline space-heating / hot-water kWh".
Baseline runs for predicted properties too (ADR-0031), reading space/water-
heating kWh off the EPC's lodged RHI block. Predicted EPCs deep-copy a neighbour
template that may carry no RHI, so `_require_rhi` hard-failed the whole subtask.
Fix: when the EPC has no RHI, fall back to the property's OWN computed figures
from the scored SapResult (space_heating_kwh_per_yr / hot_water_kwh_per_yr) —
more representative than a neighbour's lodged numbers. Only when there is also no
SapResult (the rebaseliner scored nothing) is there genuinely no demand to
record, and we still fail noisily. Lodged certs are unchanged (RHI still wins).
Regression tests: fallback-to-computed, and the no-RHI/no-result raise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The FE-owned `material.type` pgEnum cannot carry `secondary_heating_removal`,
so pricing it through the DB catalogue raises a DataError that poisons the
session — the modelling pipeline crashed on any property with a lodged
secondary heater unless the measure was excluded on the Scenario.
Realise the `ProductRepository` docstring's intent (DB catalogue today, a JSON
file for costs the ETL does not yet supply, behind the same port): add a
`CompositeProductRepository` that resolves an override source first, then the
catalogue. Checking the override first keeps that Measure Type away from the DB
entirely; every other type misses the override and falls through unchanged.
- off_catalogue_costs.json prices it at £270 flat per-dwelling — the legacy
`Costs.heater_removal` ported to the new flat model (ADR-0028):
(£25 + £200 baseline) x 1.2 VAT, for the single fixed secondary a cert lodges.
Contingency (0.25) is joined from config, not the file.
- Wire the composite into PostgresUnitOfWork.product and run_modelling_e2e, so
the first-run pipeline and the local runner both honour the overlay.
- Integration test: drop the unrealistic seeded secondary_heating_removal DB
rows (the pgEnum can't hold the type) and assert it is JSON-sourced
(material_id is None, cost £270) end-to-end through a real Unit of Work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new pipeline left no per-Property record of a run (the old engine set
property.has_recommendations and populated property_details_epc). Restore the
marker: PropertyRepository.mark_modelled sets has_recommendations (true when the
Plan carries measures, mirroring the old engine) and bumps updated_at, so a
first-run under the new process is identifiable as updated_at >= 2026-06-01.
ModellingOrchestrator marks each Property after its Scenarios (true if any
Scenario yielded a measure); run_modelling_e2e's --persist path marks it too
(its compute runs on in-memory fakes, so the DB UoW sets it directly). Adds the
has_recommendations/updated_at columns to the PropertyRow mirror.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`restrict_to_considered_measures` filtered candidates only *after* every
generator had run, so an excluded measure still queried the catalogue.
That crashed properties with a lodged secondary heater: the live
`material.type` enum has no `secondary_heating_removal` value, so the
query raised a psycopg2 `InvalidTextRepresentation` before the allowlist
could drop it.
`_candidate_recommendations` now pairs each generator with the measure
types it can emit and runs it only when the allowlist admits one of them
(None = all), so an excluded measure never reaches the catalogue.
`restrict_to_considered_measures` still trims disallowed Options off the
multi-Option survivors. Add `--exclude-measures` to run_modelling_e2e
(allowlist minus the excluded set) for excluding one measure without
enumerating the rest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two review points from @dancafc:
1) Rename the `Comparable` dataclass → `ComparableProperty` (it models one
comparable *property*; the collection stays `ComparableProperties`). Applied
across domain, repositories, orchestration, harness, scripts, and tests with a
word-boundary rename so `ComparableProperties` is untouched.
2) Move `PredictionTarget` out of comparable_properties.py into prediction_target.py
(where `PredictionTargetAttributes` + `build_prediction_target` already live).
comparable_properties.py now imports it; no import cycle (prediction_target no
longer depends on comparable_properties). Importers updated.
92 tests pass across the touched suites; pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire EPC Prediction gap-fill into IngestionOrchestrator (ADR-0031). When the
predictor collaborators are injected (ComparablesRepo + PredictionAttributesReader
+ EpcPrediction), an EPC-less Property is predicted from its postcode cohort and
persisted to the predicted slot; the eligibility gate (unknown property_type) and
"a lodged EPC is never predicted over" both hold. The two-phase contract is kept:
prediction attributes (Landlord Overrides) resolve in the unit prep phase, the
cohort fetch + select + predict run in the no-unit IO phase, persistence in the
write phase. All three collaborators are OPTIONAL — unwired, ingestion behaves
exactly as before (existing tests unchanged).
3 tests (predict+persist, gate, lodged-wins); 228 pass across orchestration +
epc_prediction + repositories; pyright strict clean. Production composition-root
wiring (real ComparableProperties + override-attributes adapters) is part of the
Jun-te handover.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `source` discriminator (lodged | predicted) to the EPC store so a Property
holds a lodged EPC and a predicted one (EPC Prediction gap-fill) at once
(ADR-0031). EpcRepository.save gains source="lodged"; idempotent delete is now
per-source (a predicted save no longer wipes lodged, and vice versa);
get_for_property/get_for_properties filter lodged; new get_predicted_for_property
/ get_predicted_for_properties read predicted. PropertyPostgresRepository.get +
get_many hydrate Property.predicted_epc, so the predicted picture reaches the
modelling read (both load via get_many). FakeEpcRepo mirrors the dual slot.
EpcPropertyModel gains `source` (default "lodged"); the test DB builds from the
SQLModel mirror so this is exercised without the prod migration. The matching
Drizzle change (column + per-(property_id,source) uniqueness) is the team's to
action before merge — docs/MIGRATION_NOTE_predicted_epc_source.md.
3 store tests (coexist, idempotent predicted re-save leaves lodged, lodged-only
has no predicted) + property-repo wiring; 85 pass across affected suites; new
code pyright-clean (2 pre-existing wwhrs errors in epc_property_table untouched).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The optimiser-package expectation was stale: it predated the optimiser
folding a triggered measure's forced dependency into its candidate gain
(ADR-0016). The run considers ALL measures (considered_measures defaults
to None — no restriction), so once the ASHP bundle became SAP-beneficial
(ADR-0025) the gain-maximising package shifted.
Verified the new package is CORRECT, not a regression: on the test EPC,
cavity-wall insulation earns +2.9 SAP alone but its forced fabric→
ventilation dependency (ADR-0016) drags the wall+ventilation pair to a
NET −1.8 SAP (−0.9 on top of the ASHP package), so the gain-maximising
Optimiser correctly excludes the wall and its forced ventilation. Update
the expected set to {air_source_heat_pump, suspended_floor_insulation,
low_energy_lighting, secondary_heating_removal} and drop the wall/vent-
specific assertions — the forced wall→ventilation edge is covered by
test_measure_dependency / test_optimiser; this integration test keeps its
end-to-end optimise→persist→telescope coverage on the chosen package.
Pre-existing failure (present before this branch's recent commits), outside
the handover regression gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Orchestrator runs recommend_secondary_heating_removal; report._triggers_for
explains it via the lodged secondary_heating_type; harness catalogue + ARA seed
price it. Re-pins the golden/integration plans it shifts: it is a cheap (\£250)
SAP lever, so on gas-main certs lodging an electric secondary (691) it displaces
the \£12k ASHP (0330, 0036) or joins the all-beneficial-measures package (000490,
where its marginal SAP is 0 under the category-4 ASHP but the heater is still
physically removed). Consistent with the optimiser's existing kitchen-sink
package behaviour.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the system tune-up to the heating Recommendation: keep the existing wet
boiler but install better heating controls and fix the cylinder. Two competing
Options (the Optimiser picks <=1 across the whole heating rec) per the user's
two best control end-states:
- system_tune_up — standard controls (programmer + room thermostat +
TRVs, SAP 10.2 Table 4e code 2106)
- system_tune_up_zoned — time-and-temperature zone control (code 2110, type 3):
more SAP uplift for more cost
Both keep the boiler (no fuel / SAP code / flue change), set the control
ABSOLUTELY to their end-state, and apply the conditional cylinder fixes (an
80 mm jacket when under-insulated, a thermostat when absent — only when a
cylinder exists). Each control option is offered only when it genuinely improves
the existing control — standard is skipped when the control is already 2106 /
2110 / 2112, zone when already 2110 / 2112 — so neither is ever a downgrade or a
no-op.
Validated against the Elmhurst "system tune up" re-lodgements (cert 001431):
nine befores spanning controls 2101-2113 all converge to the two common afters,
proving the control overlay is absolute. The cascade pin is parametrised over
two starting controls (2101 "no control" + 2113 "room thermostat and TRVs") x
both afters, delta 0 (SAP/CO2/PE).
Wires the two MeasureTypes through contingencies (0.15), the offline catalogue
(500 / 900), the catalogue-coverage list, the report triggers, and the ARA
first-run seed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the first boiler-upgrade option to the single "Heating & Hot Water"
Recommendation (ADR-0024 expansion): a dwelling whose existing wet boiler heats
a hot-water cylinder is offered a new gas condensing boiler, with the cylinder
jacketed when under-insulated and given a thermostat when absent. One competing
Option (the Optimiser picks <=1), folded into one composite Plan line.
The end-state is read from the Elmhurst before/after re-lodgements (cert 001431,
gas boiler upgrade - with cylinder), which REVISE ADR-0024:
- Target is always a gas condensing boiler, not fuel-preserving: every after
lodges fuel 26. Gas->gas always; a non-gas wet boiler ->gas only with a
mains-gas connection; electric boilers are left alone (electrification is the
upgrade path). Eligibility = wet-boiler SAP code (Table 4a/4b 101-141 /
151-161 / 191-196) + not an electric boiler + mains gas present.
- End-state is a Table 4b SAP code, not a PCDB index: code 102 (regular boiler
+ cylinder). The calculator derives the condensing seasonal efficiency from
the code, so no efficiency input exists or is needed.
- A modern condensing boiler has a fanned flue: the after flips
`fan_flue_present` False->True on every cert (SAP 10.2 Table 4f flue-fan +
the Table 4b condensing-efficiency basis). Added as a new HeatingOverlay
field, routed to main_heating_details[0].
- Cylinder thermostat is always added when absent (user-locked); the jacket is
the 80 mm `cylinder_insulation_type=2` end-state, applied only when the
cylinder is below 80 mm (never downgrading a better one). Both are conditional
per-dwelling components, not a frozen overlay.
Cascade-pinned delta-0 (SAP/CO2/PE) against the relodged after via
`_assert_overlay_reproduces_after`. NB the absolute SAP on this dwelling is
subject to a separate Summary-path mapper roof-fidelity gap (we read the roof
better-insulated than Elmhurst, scoring ~75 vs the printed 56); the gap is
identical on before+after (the boiler measure never touches the roof) so it
cancels and the pin still proves the exact heating field-delta. Tracked on the
calculator branch.
Wires the new `gas_boiler_upgrade` MeasureType through contingencies (0.26),
the offline sample catalogue, the catalogue-coverage list, and the ARA
first-run integration seed (the option fires on any mains-gas boiler+cylinder
dwelling).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add domain/modelling/considered_measures.py::restrict_to_considered_measures —
the pure allowlist that limits a run to a chosen set of MeasureType (mirroring
the legacy engine's `inclusions`). It filters at the Option level, so a
multi-option Recommendation (e.g. Heating & Hot Water competing HHRSH against
an ASHP bundle) is kept with only its allowed Options; a Recommendation left
with none is dropped. None = consider everything (unrestricted default).
Thread `considered_measures: frozenset[MeasureType] | None` through
ModellingOrchestrator.run -> _plan_for -> _scored_candidate_groups /
_candidate_recommendations (applies the filter) and _measure_dependencies
(suppresses a forced dependency whose required measure is outside the
allowlist, so a restricted run forces nothing it is not considering). The
local-run seam (harness.console.run_modelling) gains the same param.
The Optimiser still freely chooses among survivors — including none. Tests:
the pure filter (3 cases) + an orchestrator-seam test proving a
{solar_pv}-restricted run yields only solar_pv options. 257 pass + 3 xfail;
pyright clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pulls in 42 commits of calculator/mapper accuracy fixes from the per-cert
mapper-validation and floor/roof/heating fronts.
Conflict resolutions:
- mapper `_is_elmhurst_roof_window`: main dropped the branch's "wall location →
vertical" guard (it broke cert 000516's rooflight), but that re-broke cert
001431's two External-wall U>3.0 windows (which must stay vertical). The two
certs lodge a BYTE-IDENTICAL §11 row, so neither location nor U separates
them — the real discriminator is the room-in-roof context. Replaced the
unconditional U>3.0 backstop with one gated on the BP having a room-in-roof
(`_elmhurst_bp_has_room_in_roof`): 000516's Main BP has a "Room in roof type
1" (→ rooflight), 001431's does not (→ vertical). Validated against BOTH —
full Elmhurst worksheet suite 1038 pass + the 001431 window-extraction pin.
- property_postgres_repository: kept main's `ids_by_uprn` method + the branch's
`_restrictions_of` helper.
- sap_fuel.py: the branch relocated it to domain/billing/ (already carrying
main's to_table_32_code normalization), so kept the old path deleted.
Fallout from main's fabric fixes (validated by the boiler-3 real-cert pin which
still reproduces at delta 0):
- re-pinned the boiler-1 + boiler-instant-hw ASHP snapshot scores;
- main's §14.2 gas-boiler main-fuel derivation resolved the BGB/102 baseline
gap, so `test_gas_boiler_instant_hw_before_baselines` is now a passing test
(was an xfail tripwire).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 8 of the Solar PV Recommendation Generator (ADR-0026). The
ModellingOrchestrator now reads each Property's persisted Google Solar
buildingInsights JSON (uow.solar), projects it once per Property into a typed
SolarPotential via `_solar_potential_for` (None for a missing or error
payload), and threads it into `recommend_solar` alongside planning_restrictions
— mirroring the ASHP wiring. Solar fires only when a feasible potential is
present, so dwellings without fetched solar data are unaffected.
FakeSolarRepo now returns None for an unseeded Property (was raising) and
supports `by_property` seeding, so the orchestrator's new solar read is exercised.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The efficient Vaillant aroTHERM plus 5 kW (ADR-0025) now raises SAP even on gas
dwellings, so the Optimiser selects the ASHP bundle far more often -- a deliberate
product shift (user-confirmed). Updated the integration/harness tests:
- ARA multi-measure: ASHP is additive to the kept fabric package -> add to set.
- ARA listed_uprn: is_listed blocks walls AND ASHP (blocks_internal); the gate is
now observable via ASHP (selected unrestricted, blocked listed).
- report three-measures: ASHP + solid-floor displace the fabric stack; assert
their triggers (property_type, main_heating_category / floor).
- console hhr + solid_wall coverage tests: assert the measure is OFFERED as a
candidate (the wiring/eligibility intent), since ASHP now out-competes them in
selection; also assert the optimised package leads with ASHP.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
recommend_heating now receives planning_restrictions in the orchestrator (the
ASHP planning gate); the ASHP bundle joins the free candidate pool for every
house/bungalow. Catalogue + contingency (legacy 0.25) gain air_source_heat_pump;
report.py _triggers_for explains the ASHP trigger; the harness forcing test
covers it. Integration tests seed an air_source_heat_pump MaterialRow (ASHP
fires on every house, the broadest trigger yet). NB the optimiser correctly does
NOT select ASHP for an EPC-band goal — gas->electric does not improve the SAP
cost-rating; ASHP is a CO2/PE measure, selectable once non-EPC goals land. ASHP
bundle COMPLETE (S5-S7). ADR-0024.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 4 of the lighting generator (ADR-0023): run recommend_lighting in
_candidate_recommendations (no planning gate). Price low_energy_lighting in the
offline catalogue + contingency table (0.26, the legacy rate); the
_GENERATOR_MEASURE_TYPES forcing test enforces both. A run_modelling test pins
the wiring end-to-end (an incandescent-lit dwelling gets the LED upgrade in the
optimised package).
Downstream updates, all because lighting now fires on any cert with non-LED
bulbs: report.py gains the low_energy_lighting trigger (the non-LED counts); the
two golden-cert report tests and the multi-measure integration test now expect
low_energy_lighting alongside the fabric measures (the sample/golden EPCs lodge
low-energy-unknown bulbs); first-run integration seeds a low_energy_lighting
MaterialRow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 4 of the glazing generator (ADR-0022): run recommend_glazing in
_candidate_recommendations, threading the Property's PlanningRestrictions so a
protected dwelling is offered secondary glazing instead of double (mirrors
recommend_solid_wall). Price both Measure Types in the offline catalogue
(double £600/window, secondary £510 -- the legacy 0.85x scaling) and the
contingency table (0.15, the legacy windows_glazing rate); the
_GENERATOR_MEASURE_TYPES forcing test enforces both entries exist.
run_modelling tests pin the wiring end-to-end on an all-single-glazed dwelling:
double when unrestricted, secondary when listed. The first-run integration test
seeds a double_glazing Product because its lodged EPC has a single-glazed
window. _single_glazed_epc() deep-copies build_epc() (which shares its window
objects) so the mutation can't leak into other tests' baselines.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3c.6. The integrating proof through real Postgres: two solid-brick
uninsulated dwellings, identical but for the planning status Ingestion caches
per UPRN. Ingestion writes the spatial reference; Modelling reads it back off
the Property and gates the wall measures — the listed dwelling gets neither
EWI nor IWI, the unrestricted one gets a wall measure. Closes slice 3c
(ADR-0019/ADR-0020).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3c.4. Ingestion now resolves the whole spatial reference in one lookup
(`spatial_for`) — the coordinates drive the Solar fetch as before, and the
reference (coordinates + planning protections) is persisted per-UPRN via
`uow.spatial` in the same write batch, so Modelling can read the protections
back off the Property (ADR-0020). `_Fetched` carries the UPRN and the reference
into the write phase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3c.3. Ingestion writes the OS spatial reference cache through the same
unit it persists the EPC/solar enrichments with, so `UnitOfWork` declares a
`spatial` repo, `PostgresUnitOfWork` binds a `SpatialPostgresRepository` to the
session, and `FakeUnitOfWork` gains a `FakeSpatialRepo` (seedable for read
tests, recording writes for ingestion-side assertions).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
