Model/docs/HANDOVER_MODELLING.md

HANDOVER — Modelling stage rebuild

Branch: feature/bill-derivation (worktree /workspaces/home/hestia-worktrees/model-assemble-new-backend). HEAD: 84ec6da0. PRD: GitHub Hestia-Homes/Model#1152, sliced into #1153–#1161. All slices #1153–#1161 closed.

Issue status

Issue	What	State
#1153	Overlay Applicator + EpcSimulation	✅ closed
#1154	Package Scorer	✅ closed — Elmhurst cascade pin (4c0a907a)
#1155	wall Recommendation Generator	✅ closed; cascade-pinned
#1156	score Options + attribution	✅ closed
#1157	persist a Plan via ModellingOrchestrator	✅ closed this session (772cdd4f→c7e2aa37)
#1158	roof (loft) generator	✅ closed — 300 mm + cascade pin
#1159	floor generator	✅ closed — overlay insulation-type field + pins
#1160	Optimiser (knapsack + greedy repair)	✅ closed this session (77983cae→34d4748a)
#1161	Measure Dependency (ventilation)	✅ closed this session (7c59e919→0fec0699)

What this session did

1. Cascade pins for #1154/#1158/#1159 — tests/domain/modelling/test_elmhurst_cascade_pins.py. Parse Elmhurst before/after recommendation Summaries via the extractor chain (NOT parse_site_notes_pdf), apply the generator's overlay, score, assert delta 0 vs the after-cert. Found+fixed: loft 270→300 mm; suspended floor needs the overlay to also set floor_insulation_type_str='Retro-fitted'.
2. ProductJsonRepository (cc0bb8f9) — file-backed catalogue behind the ProductRepository port.
3. #1157 — persist a Plan. Design review (/grill-with-docs) + 5 TDD slices. See "Design decisions" below.
4. #1160 — the Optimiser. 4 TDD slices. See "Design decisions".

Design decisions locked this session (READ THESE)

• Multi-phase is DEFERRED (speculative prospective-client ask). ADR-0005 rewritten to "Deferred". No plan_phase table, no phase column. CONTEXT.md no longer has Scenario Phase / Plan Phase / Rolled-over Options. Everything is single-phase. Future: a migration adds plan_phase + back-fills live plans as 1-phase.
• Plan Measure is the new term (in CONTEXT.md): the persisted selected Option + its role-3 attributed impact + cost. Recommendation stays the candidate (never persisted; no stored impact).
• Reuse the LIVE tables (plan, recommendation) — they exist in the live product (backend/app/db/models/recommendations.py, SQLAlchemy Base) and the FE reads them. The rebuild writes the same physical tables via SQLModel mirrors (infrastructure/postgres/plan_table.py) — the established pattern (task_table.py→tasks, product_table.py→material). ADR-0017 records this.
• Added recommendation.plan_id (FK→plan, ON DELETE CASCADE); retire the plan_recommendations m2m for new writes. FE-owned Drizzle migration: docs/migrations/recommendation-plan-id.md.
• Tracer persists SAP + CO₂ (tonnes = calc kg ÷ 1000) + cost + derived post_epc_rating. Energy/bill columns deferred. Idempotent replace per (property_id, scenario_id).
• Optimiser = exact pure-Python multiple-choice knapsack, NOT mip. Recycles GainOptimiser/CostOptimiser's formulation (≤1/group, maximise gain s.t. budget) but not the dependency — mip's CBC backend does not load on this aarch64 container (NameError: cbclib), so the legacy solver can't run/be tested here. ADR-0016's MILP is only a warm-start signal, so exact small-scale enumeration is ample. Re-score + greedy-repair toward the goal's SAP target gives the truth.

domain/modelling/ layout (grouped 84ec6da0)

Behaviour lives in subpackages; shared value-object vocabulary stays flat at the top (imported everywhere): recommendation.py (Recommendation / MeasureOption / Cost), plan.py, scenario.py, product.py, contingencies.py, simulation.py (EpcSimulation overlay).

• generators/ — wall_recommendation / roof_recommendation / floor_recommendation.
• scoring/ — overlay_applicator (apply_simulations), package_scorer (role 2), scoring (role-1 independent_option_impacts + role-3 marginal_impacts). Note the path is domain.modelling.scoring.scoring for the role-1/3 module.
• optimisation/ — optimiser, measure_dependency.

What's built (all in domain/modelling/, infrastructure/postgres/, repositories/, orchestration/)

• Generators (generators/): recommend_cavity_wall / recommend_loft_insulation (300 mm) / recommend_floor_insulation (sets floor_insulation_type_str).
• simulation.py overlay + scoring/overlay_applicator.apply_simulations (generic field-fold) + scoring/package_scorer.PackageScorer.score (role 2) + scoring/scoring.py (marginal_impacts role 3, independent_option_impacts role 1).
• scenario.py Scenario(id, goal, goal_value, budget, is_default); plan.py Plan + PlanMeasure (derives cost_of_works/contingency_cost/co2_savings/post_epc_rating).
• optimisation/optimiser.py — optimise(groups, budget) (exact knapsack) + optimise_package(...) (re-score + greedy repair, Scorer Protocol, OptimisedPackage).
• infrastructure/postgres/: scenario_table.ScenarioRow, plan_table.{PlanRow,RecommendationRow} (mirrors of live tables; from_domain).
• repositories/: scenario/, plan/, product/ (Postgres + Json) — all on the UnitOfWork (uow.scenario/uow.product/uow.plan).
• ModellingOrchestrator.run(property_ids, scenario_ids, portfolio_id) — one UoW, commit once; generate (wall/roof/floor) → role-1 score → optimise_package → role-3 attribute → persist. Wired into AraFirstRunPipeline + handler.py.
• datatypes/epc/domain/epc.py::Epc.sap_lower_bound() (band → min SAP, target for INCREASING_EPC).

Gotchas (will bite a fresh agent)

• mip / CBC is broken on aarch64 here — never build runnable code on mip. The legacy recommendations/optimiser/ tests only "pass" because they avoid constructing a mip.Model.
• moto is not installed — tests/orchestration/test_postcode_splitter_orchestrator.py and tests/repositories/unstandardised_address/ fail at collection. Pre-existing, unrelated; --ignore them when sweeping.
• Run tests: python -m pytest <path> -q (do NOT pass -p no:cov). Ephemeral Postgres via the db_engine fixture builds only SQLModel.metadata — legacy Base tables are absent in tests, which is why mirrors work.
• Worktree import trap: python /tmp/foo.py imports /workspaces/model, not this worktree. Use pytest (rootdir handles it) or a python -c from the worktree root.
• Driving Modelling in an integration test: the calculator fixtures (_elmhurst_worksheet_000490.build_epc()) lack lodged recorded-performance fields, so the Baseline stage can't run on them. Drive ModellingOrchestrator directly off a repo-seeded EPC (EpcPostgresRepository(session).save(epc, property_id, portfolio_id)) — see test_modelling_optimises_and_persists_a_multi_measure_plan. The sample API EPC (_lodged_epc()) does go through the full pipeline.
• PortfolioGoal.INCREASING_EPC value is "Increasing EPC" (with a space) — the orchestrator compares scenario.goal == "Increasing EPC".
• A generator calls products.get(...) during candidate generation, so the integration test must seed a material row for every measure type that fires (e.g. the sample EPC's uninsulated solid floor needs solid_floor_insulation).
• Don't edit the SAP calculator's heat_transmission.py (another agent owns it).

Conventions

Commit per TDD slice; conventional-commit message ending Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>; stay on feature/bill-derivation. Tests use literal # Arrange / # Act / # Assert; assert with abs(x - y) <= tol (not pytest.approx); pyright strict, zero errors; annotate call-return locals. Cascade pins target the worksheet at delta 0.

#1161 — Measure Dependency (ventilation), as built (4 TDD slices, all green)

Forks resolved with the user (AskUserQuestion): guard now (skip when already MEV/MVHR), persist as a Plan Measure (cost + real negative marginal), forced but its cost counts toward spend (mandatory-when-triggered, never budget-gated; repair sees less headroom).

1. 7c59e919 — Simulation Overlay grows a dwelling-level segment: VentilationOverlay (all-optional partial of SapVentilation, field mechanical_ventilation_kind) + EpcSimulation.ventilation; apply_simulations folds it onto sap_ventilation (creating one if the baseline lodged none). Until now the overlay was building-part only — ventilation is whole-dwelling.
2. 6b11c902 — generic injection in the optimiser: MeasureDependency(triggers: frozenset[str], required: ScoredOption) lives in optimisation/optimiser.py (its input contract). optimise_package(..., dependencies=()) injects any dependency whose triggers ∩ selected-measure-types, before every re-score (initial and each repair). _inject dedups by required measure-type. Forced (injected even over budget) but its cost is in _package_cost, so repair headroom shrinks. _best_repair_candidate folds in any dependency a candidate newly triggers, so its marginal SAP and incremental cost are truthful; affordability gates on whole-package cost vs budget. Returned selected includes the injected deps. Optimiser stays domain-agnostic — no ventilation import.
3. 1bf5b410 — domain/modelling/optimisation/measure_dependency.py: MEASURES_NEEDING_VENTILATION (cavity/internal/external wall, cf. legacy assumptions.measures_needing_ventilation) + ventilation_dependency(epc, products) → MEV Option (mechanical_ventilation_kind="EXTRACT_OR_PIV_OUTSIDE", decentralised MEV = legacy "mechanical, extract only"), priced at 2 fully-loaded units. Returns None when sap_ventilation.mechanical_ventilation_kind is already set (= legacy has_ventilation — confirmed against backend/Property.py:1236). Note: builder fetches the Product up-front, so the catalogue needs a mechanical_ventilation row for every not-yet-ventilated dwelling, even if no wall is ultimately selected.
4. 0fec0699 — orchestrator wiring: _measure_dependencies builds the (≤1) dependency; _BEST_PRACTICE_ORDER gains "mechanical_ventilation" between loft and floors (role-3 cascade walls→roof→vent→floor); ventilation persists as a Plan Measure with its real negative marginal + cost. Added mechanical_ventilation: 0.26 contingency (legacy Costs.CONTINGENCIES). On 000490 the real calculator scores MEV at −1.275 SAP.

Gotchas for the next agent: the ventilation Product/contingency must exist for any not-yet-ventilated dwelling (build-time fetch, not inject-time); the stub scorer in test_optimiser.py indexes building_parts[MAIN], so vent-only overlays need the separate _VentStubScorer.

What's left

Deferred fronts (open, post-#1161): exclusion-filtering of the candidate pool (deferred from #1160); a Bill-Derivation slice that re-runs bills on the post-package EPC to fill the deferred energy/bill columns (plan.post_energy_consumption/post_energy_bill, recommendation.kwh_savings/energy_cost_savings); persist unselected alternatives (default=False rows linked via plan_id) for the swap-in UX — open ADR-0016 question: what impact figure they carry; promote ProductRepository to the DB+file composite; non-EPC goal objectives (Energy Savings, Reducing CO2) in the optimiser. Possible extension of the ventilation trigger set to roof insulation (now a one-line data edit in MEASURES_NEEDING_VENTILATION); and making the dependency builder lazy (thunk) so the Product is only fetched when a trigger is actually selected.

Key references

• ADRs: 0005 (multi-phase deferred), 0011/0012 (orchestrators + UoW), 0016 (three scoring roles + warm-start/re-score/repair), 0017 (Plan persistence — evolve live tables).
• CONTEXT.md: Plan, Plan Measure, Recommendation, Measure Option, Optimised Package, Scenario, Measure Dependency.
• Auto-memory project_modelling_stage_state has the running state.