The new pipeline left no per-Property record of a run (the old engine set
property.has_recommendations and populated property_details_epc). Restore the
marker: PropertyRepository.mark_modelled sets has_recommendations (true when the
Plan carries measures, mirroring the old engine) and bumps updated_at, so a
first-run under the new process is identifiable as updated_at >= 2026-06-01.
ModellingOrchestrator marks each Property after its Scenarios (true if any
Scenario yielded a measure); run_modelling_e2e's --persist path marks it too
(its compute runs on in-memory fakes, so the DB UoW sets it directly). Adds the
has_recommendations/updated_at columns to the PropertyRow mirror.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`restrict_to_considered_measures` filtered candidates only *after* every
generator had run, so an excluded measure still queried the catalogue.
That crashed properties with a lodged secondary heater: the live
`material.type` enum has no `secondary_heating_removal` value, so the
query raised a psycopg2 `InvalidTextRepresentation` before the allowlist
could drop it.
`_candidate_recommendations` now pairs each generator with the measure
types it can emit and runs it only when the allowlist admits one of them
(None = all), so an excluded measure never reaches the catalogue.
`restrict_to_considered_measures` still trims disallowed Options off the
multi-Option survivors. Add `--exclude-measures` to run_modelling_e2e
(allowlist minus the excluded set) for excluding one measure without
enumerating the rest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two review points from @dancafc:
1) Rename the `Comparable` dataclass → `ComparableProperty` (it models one
comparable *property*; the collection stays `ComparableProperties`). Applied
across domain, repositories, orchestration, harness, scripts, and tests with a
word-boundary rename so `ComparableProperties` is untouched.
2) Move `PredictionTarget` out of comparable_properties.py into prediction_target.py
(where `PredictionTargetAttributes` + `build_prediction_target` already live).
comparable_properties.py now imports it; no import cycle (prediction_target no
longer depends on comparable_properties). Importers updated.
92 tests pass across the touched suites; pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire EPC Prediction gap-fill into IngestionOrchestrator (ADR-0031). When the
predictor collaborators are injected (ComparablesRepo + PredictionAttributesReader
+ EpcPrediction), an EPC-less Property is predicted from its postcode cohort and
persisted to the predicted slot; the eligibility gate (unknown property_type) and
"a lodged EPC is never predicted over" both hold. The two-phase contract is kept:
prediction attributes (Landlord Overrides) resolve in the unit prep phase, the
cohort fetch + select + predict run in the no-unit IO phase, persistence in the
write phase. All three collaborators are OPTIONAL — unwired, ingestion behaves
exactly as before (existing tests unchanged).
3 tests (predict+persist, gate, lodged-wins); 228 pass across orchestration +
epc_prediction + repositories; pyright strict clean. Production composition-root
wiring (real ComparableProperties + override-attributes adapters) is part of the
Jun-te handover.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `source` discriminator (lodged | predicted) to the EPC store so a Property
holds a lodged EPC and a predicted one (EPC Prediction gap-fill) at once
(ADR-0031). EpcRepository.save gains source="lodged"; idempotent delete is now
per-source (a predicted save no longer wipes lodged, and vice versa);
get_for_property/get_for_properties filter lodged; new get_predicted_for_property
/ get_predicted_for_properties read predicted. PropertyPostgresRepository.get +
get_many hydrate Property.predicted_epc, so the predicted picture reaches the
modelling read (both load via get_many). FakeEpcRepo mirrors the dual slot.
EpcPropertyModel gains `source` (default "lodged"); the test DB builds from the
SQLModel mirror so this is exercised without the prod migration. The matching
Drizzle change (column + per-(property_id,source) uniqueness) is the team's to
action before merge — docs/MIGRATION_NOTE_predicted_epc_source.md.
3 store tests (coexist, idempotent predicted re-save leaves lodged, lodged-only
has no predicted) + property-repo wiring; 85 pass across affected suites; new
code pyright-clean (2 pre-existing wwhrs errors in epc_property_table untouched).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The optimiser-package expectation was stale: it predated the optimiser
folding a triggered measure's forced dependency into its candidate gain
(ADR-0016). The run considers ALL measures (considered_measures defaults
to None — no restriction), so once the ASHP bundle became SAP-beneficial
(ADR-0025) the gain-maximising package shifted.
Verified the new package is CORRECT, not a regression: on the test EPC,
cavity-wall insulation earns +2.9 SAP alone but its forced fabric→
ventilation dependency (ADR-0016) drags the wall+ventilation pair to a
NET −1.8 SAP (−0.9 on top of the ASHP package), so the gain-maximising
Optimiser correctly excludes the wall and its forced ventilation. Update
the expected set to {air_source_heat_pump, suspended_floor_insulation,
low_energy_lighting, secondary_heating_removal} and drop the wall/vent-
specific assertions — the forced wall→ventilation edge is covered by
test_measure_dependency / test_optimiser; this integration test keeps its
end-to-end optimise→persist→telescope coverage on the chosen package.
Pre-existing failure (present before this branch's recent commits), outside
the handover regression gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Orchestrator runs recommend_secondary_heating_removal; report._triggers_for
explains it via the lodged secondary_heating_type; harness catalogue + ARA seed
price it. Re-pins the golden/integration plans it shifts: it is a cheap (\£250)
SAP lever, so on gas-main certs lodging an electric secondary (691) it displaces
the \£12k ASHP (0330, 0036) or joins the all-beneficial-measures package (000490,
where its marginal SAP is 0 under the category-4 ASHP but the heater is still
physically removed). Consistent with the optimiser's existing kitchen-sink
package behaviour.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the system tune-up to the heating Recommendation: keep the existing wet
boiler but install better heating controls and fix the cylinder. Two competing
Options (the Optimiser picks <=1 across the whole heating rec) per the user's
two best control end-states:
- system_tune_up — standard controls (programmer + room thermostat +
TRVs, SAP 10.2 Table 4e code 2106)
- system_tune_up_zoned — time-and-temperature zone control (code 2110, type 3):
more SAP uplift for more cost
Both keep the boiler (no fuel / SAP code / flue change), set the control
ABSOLUTELY to their end-state, and apply the conditional cylinder fixes (an
80 mm jacket when under-insulated, a thermostat when absent — only when a
cylinder exists). Each control option is offered only when it genuinely improves
the existing control — standard is skipped when the control is already 2106 /
2110 / 2112, zone when already 2110 / 2112 — so neither is ever a downgrade or a
no-op.
Validated against the Elmhurst "system tune up" re-lodgements (cert 001431):
nine befores spanning controls 2101-2113 all converge to the two common afters,
proving the control overlay is absolute. The cascade pin is parametrised over
two starting controls (2101 "no control" + 2113 "room thermostat and TRVs") x
both afters, delta 0 (SAP/CO2/PE).
Wires the two MeasureTypes through contingencies (0.15), the offline catalogue
(500 / 900), the catalogue-coverage list, the report triggers, and the ARA
first-run seed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the first boiler-upgrade option to the single "Heating & Hot Water"
Recommendation (ADR-0024 expansion): a dwelling whose existing wet boiler heats
a hot-water cylinder is offered a new gas condensing boiler, with the cylinder
jacketed when under-insulated and given a thermostat when absent. One competing
Option (the Optimiser picks <=1), folded into one composite Plan line.
The end-state is read from the Elmhurst before/after re-lodgements (cert 001431,
gas boiler upgrade - with cylinder), which REVISE ADR-0024:
- Target is always a gas condensing boiler, not fuel-preserving: every after
lodges fuel 26. Gas->gas always; a non-gas wet boiler ->gas only with a
mains-gas connection; electric boilers are left alone (electrification is the
upgrade path). Eligibility = wet-boiler SAP code (Table 4a/4b 101-141 /
151-161 / 191-196) + not an electric boiler + mains gas present.
- End-state is a Table 4b SAP code, not a PCDB index: code 102 (regular boiler
+ cylinder). The calculator derives the condensing seasonal efficiency from
the code, so no efficiency input exists or is needed.
- A modern condensing boiler has a fanned flue: the after flips
`fan_flue_present` False->True on every cert (SAP 10.2 Table 4f flue-fan +
the Table 4b condensing-efficiency basis). Added as a new HeatingOverlay
field, routed to main_heating_details[0].
- Cylinder thermostat is always added when absent (user-locked); the jacket is
the 80 mm `cylinder_insulation_type=2` end-state, applied only when the
cylinder is below 80 mm (never downgrading a better one). Both are conditional
per-dwelling components, not a frozen overlay.
Cascade-pinned delta-0 (SAP/CO2/PE) against the relodged after via
`_assert_overlay_reproduces_after`. NB the absolute SAP on this dwelling is
subject to a separate Summary-path mapper roof-fidelity gap (we read the roof
better-insulated than Elmhurst, scoring ~75 vs the printed 56); the gap is
identical on before+after (the boiler measure never touches the roof) so it
cancels and the pin still proves the exact heating field-delta. Tracked on the
calculator branch.
Wires the new `gas_boiler_upgrade` MeasureType through contingencies (0.26),
the offline sample catalogue, the catalogue-coverage list, and the ARA
first-run integration seed (the option fires on any mains-gas boiler+cylinder
dwelling).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add domain/modelling/considered_measures.py::restrict_to_considered_measures —
the pure allowlist that limits a run to a chosen set of MeasureType (mirroring
the legacy engine's `inclusions`). It filters at the Option level, so a
multi-option Recommendation (e.g. Heating & Hot Water competing HHRSH against
an ASHP bundle) is kept with only its allowed Options; a Recommendation left
with none is dropped. None = consider everything (unrestricted default).
Thread `considered_measures: frozenset[MeasureType] | None` through
ModellingOrchestrator.run -> _plan_for -> _scored_candidate_groups /
_candidate_recommendations (applies the filter) and _measure_dependencies
(suppresses a forced dependency whose required measure is outside the
allowlist, so a restricted run forces nothing it is not considering). The
local-run seam (harness.console.run_modelling) gains the same param.
The Optimiser still freely chooses among survivors — including none. Tests:
the pure filter (3 cases) + an orchestrator-seam test proving a
{solar_pv}-restricted run yields only solar_pv options. 257 pass + 3 xfail;
pyright clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pulls in 42 commits of calculator/mapper accuracy fixes from the per-cert
mapper-validation and floor/roof/heating fronts.
Conflict resolutions:
- mapper `_is_elmhurst_roof_window`: main dropped the branch's "wall location →
vertical" guard (it broke cert 000516's rooflight), but that re-broke cert
001431's two External-wall U>3.0 windows (which must stay vertical). The two
certs lodge a BYTE-IDENTICAL §11 row, so neither location nor U separates
them — the real discriminator is the room-in-roof context. Replaced the
unconditional U>3.0 backstop with one gated on the BP having a room-in-roof
(`_elmhurst_bp_has_room_in_roof`): 000516's Main BP has a "Room in roof type
1" (→ rooflight), 001431's does not (→ vertical). Validated against BOTH —
full Elmhurst worksheet suite 1038 pass + the 001431 window-extraction pin.
- property_postgres_repository: kept main's `ids_by_uprn` method + the branch's
`_restrictions_of` helper.
- sap_fuel.py: the branch relocated it to domain/billing/ (already carrying
main's to_table_32_code normalization), so kept the old path deleted.
Fallout from main's fabric fixes (validated by the boiler-3 real-cert pin which
still reproduces at delta 0):
- re-pinned the boiler-1 + boiler-instant-hw ASHP snapshot scores;
- main's §14.2 gas-boiler main-fuel derivation resolved the BGB/102 baseline
gap, so `test_gas_boiler_instant_hw_before_baselines` is now a passing test
(was an xfail tripwire).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 8 of the Solar PV Recommendation Generator (ADR-0026). The
ModellingOrchestrator now reads each Property's persisted Google Solar
buildingInsights JSON (uow.solar), projects it once per Property into a typed
SolarPotential via `_solar_potential_for` (None for a missing or error
payload), and threads it into `recommend_solar` alongside planning_restrictions
— mirroring the ASHP wiring. Solar fires only when a feasible potential is
present, so dwellings without fetched solar data are unaffected.
FakeSolarRepo now returns None for an unseeded Property (was raising) and
supports `by_property` seeding, so the orchestrator's new solar read is exercised.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The efficient Vaillant aroTHERM plus 5 kW (ADR-0025) now raises SAP even on gas
dwellings, so the Optimiser selects the ASHP bundle far more often -- a deliberate
product shift (user-confirmed). Updated the integration/harness tests:
- ARA multi-measure: ASHP is additive to the kept fabric package -> add to set.
- ARA listed_uprn: is_listed blocks walls AND ASHP (blocks_internal); the gate is
now observable via ASHP (selected unrestricted, blocked listed).
- report three-measures: ASHP + solid-floor displace the fabric stack; assert
their triggers (property_type, main_heating_category / floor).
- console hhr + solid_wall coverage tests: assert the measure is OFFERED as a
candidate (the wiring/eligibility intent), since ASHP now out-competes them in
selection; also assert the optimised package leads with ASHP.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
recommend_heating now receives planning_restrictions in the orchestrator (the
ASHP planning gate); the ASHP bundle joins the free candidate pool for every
house/bungalow. Catalogue + contingency (legacy 0.25) gain air_source_heat_pump;
report.py _triggers_for explains the ASHP trigger; the harness forcing test
covers it. Integration tests seed an air_source_heat_pump MaterialRow (ASHP
fires on every house, the broadest trigger yet). NB the optimiser correctly does
NOT select ASHP for an EPC-band goal — gas->electric does not improve the SAP
cost-rating; ASHP is a CO2/PE measure, selectable once non-EPC goals land. ASHP
bundle COMPLETE (S5-S7). ADR-0024.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 4 of the lighting generator (ADR-0023): run recommend_lighting in
_candidate_recommendations (no planning gate). Price low_energy_lighting in the
offline catalogue + contingency table (0.26, the legacy rate); the
_GENERATOR_MEASURE_TYPES forcing test enforces both. A run_modelling test pins
the wiring end-to-end (an incandescent-lit dwelling gets the LED upgrade in the
optimised package).
Downstream updates, all because lighting now fires on any cert with non-LED
bulbs: report.py gains the low_energy_lighting trigger (the non-LED counts); the
two golden-cert report tests and the multi-measure integration test now expect
low_energy_lighting alongside the fabric measures (the sample/golden EPCs lodge
low-energy-unknown bulbs); first-run integration seeds a low_energy_lighting
MaterialRow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 4 of the glazing generator (ADR-0022): run recommend_glazing in
_candidate_recommendations, threading the Property's PlanningRestrictions so a
protected dwelling is offered secondary glazing instead of double (mirrors
recommend_solid_wall). Price both Measure Types in the offline catalogue
(double £600/window, secondary £510 -- the legacy 0.85x scaling) and the
contingency table (0.15, the legacy windows_glazing rate); the
_GENERATOR_MEASURE_TYPES forcing test enforces both entries exist.
run_modelling tests pin the wiring end-to-end on an all-single-glazed dwelling:
double when unrestricted, secondary when listed. The first-run integration test
seeds a double_glazing Product because its lodged EPC has a single-glazed
window. _single_glazed_epc() deep-copies build_epc() (which shares its window
objects) so the mutation can't leak into other tests' baselines.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3c.6. The integrating proof through real Postgres: two solid-brick
uninsulated dwellings, identical but for the planning status Ingestion caches
per UPRN. Ingestion writes the spatial reference; Modelling reads it back off
the Property and gates the wall measures — the listed dwelling gets neither
EWI nor IWI, the unrestricted one gets a wall measure. Closes slice 3c
(ADR-0019/ADR-0020).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3c.4. Ingestion now resolves the whole spatial reference in one lookup
(`spatial_for`) — the coordinates drive the Solar fetch as before, and the
reference (coordinates + planning protections) is persisted per-UPRN via
`uow.spatial` in the same write batch, so Modelling can read the protections
back off the Property (ADR-0020). `_Fetched` carries the UPRN and the reference
into the write phase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3c.3. Ingestion writes the OS spatial reference cache through the same
unit it persists the EPC/solar enrichments with, so `UnitOfWork` declares a
`spatial` repo, `PostgresUnitOfWork` binds a `SpatialPostgresRepository` to the
session, and `FakeUnitOfWork` gains a `FakeSpatialRepo` (seedable for read
tests, recording writes for ingestion-side assertions).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 5a: the promotion. Replaces StubRebaseliner in production and collapses the
shadow runner into the rebaseliner (ADR-0013 amendment).
- CalculatorRebaseliner runs Sap10Calculator on every Property:
* sap_version < 10.2 -> Effective Performance IS the calculator output
(band via Epc.from_sap_score, CO2 kg->t, PEUI rounded), reason "pre_sap10".
* sap_version >= 10.2 -> Effective = lodged (API figures on-target), and the
calculator only logs divergence (SAP>0.5, PEUI/CO2 1%) as a validation signal.
* a calculator raise propagates -> batch aborts (ADR-0012); fix the cert at once.
- Rebaseliner.rebaseline gains property_id (for the divergence log).
- LoggingCalculatorShadow / the calculator_shadow seam removed from the
orchestrator; its divergence-comparison logic now lives in the rebaseliner.
- StubRebaseliner kept (signature updated) for orchestrator/repo unit tests.
The SapResult->EnergyBreakdown adapter + BillDerivation wiring (to populate the
bill block) follow once the appliances/cooking SapResult fields land.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire Sap10Calculator into PropertyBaselineOrchestrator as a non-load-bearing
shadow runner. For each property it scores the Effective EPC beside the
load-bearing Lodged/Effective write, catches any strict-raise -> log.error
(never aborts the batch), and on success log.warning's divergence from Lodged:
SAP |continuous - lodged| > 0.5; PEUI/CO2 > 1% relative (CO2 after kg->tonnes).
Every line is tagged with sap_version so SAP-10.2 signal separates from
older-spec drift (ADR-0010 Validation Cohort).
Per ADR-0013, Calculated SAP10 Performance is not a persisted third value-set:
effective = calculated in every baselining scenario, so the calculator IS the
mechanism that produces Effective Performance (the Rebaseliner). It runs in
shadow only while being hardened; when overrides/estimation land it is promoted
to drive Effective and the failure posture flips to abort (ADR-0012, calculator
now load-bearing). No table change.
- ADR-0013 + CONTEXT (Calculated SAP10 Performance / Effective Performance /
Rebaselining) record the decision.
- CalculatorShadow port + LoggingCalculatorShadow + Calculator protocol.
- FakeCalculatorShadow for orchestrator unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Plan derives its Valuation Uplift (ADR-0018) from its baseline -> post
band jump and works+contingency cost, given one external input — the
Property's current market value (a Property Valuation, mostly absent).
`Plan.valuation` / `Plan.baseline_epc_rating` are derived like the other
headline figures; `PlanModel.from_domain` maps the £ forms to the live
plan.valuation_* columns (NULL when no value — the percentage is not
persisted on those columns). `Property.current_market_value` is the new
optional source; the orchestrator threads it onto the Plan. `run_one`
takes a `current_market_value` so the harness can value the uplift, and
the sense-check table shows the average % (always) plus the £ forms when
known.
Sourcing the current market value (upload / default) remains deferred
(ADR-0018); it is None throughout until that lands, so the columns stay
NULL at scale.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expand half of the recommendation_materials retirement (ADR-0017). A
Plan Measure installs a single Product, so thread its catalogue id end to
end — Product.id -> MeasureOption.material_id -> PlanMeasure.material_id
-> recommendation.material_id — replacing the per-material BOM child
table with one nullable column on the row. ProductPostgresRepository
reads the id from MaterialRow; the four fabric generators set it on their
Option; the orchestrator carries it onto the Plan Measure; the mirror
declares + maps the column. Optional throughout (the JSON stopgap
catalogue carries no ids -> NULL).
The multi-measure integration test now pins each persisted measure's
material_id to its seeded MaterialRow id. Migration spec (live column
must be added before this deploys; contraction is the owner's next step)
in docs/migrations/recommendation-material-id.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3. `harness.console.run_one(epc, goal_band=...)` wires the full
AraFirstRunPipeline against in-memory fakes — no Postgres, no network —
runs one property, prints the sense-check table, and returns the Plan
for interactive poking from a REPL at the worktree root. Defaults to the
committed harness sample catalogue.
Refactors the slice-1 integration test to delegate to run_one (dropping
~70 lines of duplicated wiring + the now-unused test catalogue fixture),
so it exercises the shipped entrypoint rather than a parallel copy. The
new console test covers run_one's print/return contract.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
