Model/docs/HANDOVER_MODELLING.md

# HANDOVER — Modelling stage rebuild

**Branch:** `feature/bill-derivation` (worktree `/workspaces/home/hestia-worktrees/model-assemble-new-backend`). **HEAD:** `84ec6da0`.
**PRD:** GitHub `Hestia-Homes/Model#1152`, sliced into #1153–#1161. **All slices #1153–#1161 closed.**

## Issue status

| Issue | What | State |
|---|---|---|
| #1153 | Overlay Applicator + `EpcSimulation` | ✅ closed |
| #1154 | Package Scorer | ✅ closed — Elmhurst cascade pin (`4c0a907a`) |
| #1155 | wall Recommendation Generator | ✅ closed; cascade-pinned |
| #1156 | score Options + attribution | ✅ closed |
| #1157 | persist a Plan via `ModellingOrchestrator` | ✅ **closed this session** (`772cdd4f`→`c7e2aa37`) |
| #1158 | roof (loft) generator | ✅ closed — 300 mm + cascade pin |
| #1159 | floor generator | ✅ closed — overlay insulation-type field + pins |
| #1160 | Optimiser (knapsack + greedy repair) | ✅ **closed this session** (`77983cae`→`34d4748a`) |
| #1161 | Measure Dependency (ventilation) | ✅ **closed this session** (`7c59e919`→`0fec0699`) |

## What this session did

1. **Cascade pins for #1154/#1158/#1159** — `tests/domain/modelling/test_elmhurst_cascade_pins.py`. Parse Elmhurst before/after recommendation Summaries via the extractor chain (NOT `parse_site_notes_pdf`), apply the generator's overlay, score, assert delta 0 vs the after-cert. Found+fixed: loft 270→**300** mm; suspended floor needs the overlay to also set `floor_insulation_type_str='Retro-fitted'`.
2. **`ProductJsonRepository`** (`cc0bb8f9`) — file-backed catalogue behind the `ProductRepository` port.
3. **#1157 — persist a Plan.** Design review (`/grill-with-docs`) + 5 TDD slices. See "Design decisions" below.
4. **#1160 — the Optimiser.** 4 TDD slices. See "Design decisions".

## Design decisions locked this session (READ THESE)

- **Multi-phase is DEFERRED** (speculative prospective-client ask). **ADR-0005 rewritten to "Deferred".** No `plan_phase` table, no `phase` column. `CONTEXT.md` no longer has Scenario Phase / Plan Phase / Rolled-over Options. Everything is **single-phase**. Future: a migration adds `plan_phase` + back-fills live plans as 1-phase.
- **Plan Measure** is the new term (in `CONTEXT.md`): the persisted selected Option + its role-3 attributed impact + cost. **Recommendation** stays the *candidate* (never persisted; no stored impact).
- **Reuse the LIVE tables** (`plan`, `recommendation`) — they exist in the live product (`backend/app/db/models/recommendations.py`, SQLAlchemy `Base`) and the FE reads them. The rebuild writes the **same physical tables via SQLModel mirrors** (`infrastructure/postgres/plan_table.py`) — the established pattern (`task_table.py`→`tasks`, `product_table.py`→`material`). **ADR-0017** records this.
- Added **`recommendation.plan_id`** (FK→plan, ON DELETE CASCADE); retire the `plan_recommendations` m2m for new writes. FE-owned Drizzle migration: `docs/migrations/recommendation-plan-id.md`.
- Tracer persists **SAP + CO₂ (tonnes = calc kg ÷ 1000) + cost + derived `post_epc_rating`**. Energy/bill columns deferred. Idempotent replace per (property_id, scenario_id).
- **Optimiser = exact pure-Python multiple-choice knapsack**, NOT `mip`. Recycles `GainOptimiser`/`CostOptimiser`'s *formulation* (≤1/group, maximise gain s.t. budget) but not the dependency — **`mip`'s CBC backend does not load on this aarch64 container** (`NameError: cbclib`), so the legacy solver can't run/be tested here. ADR-0016's MILP is only a warm-start signal, so exact small-scale enumeration is ample. Re-score + greedy-repair toward the goal's SAP target gives the truth.

## `domain/modelling/` layout (grouped `84ec6da0`)

Behaviour lives in subpackages; shared value-object vocabulary stays flat at the top (imported everywhere): `recommendation.py` (Recommendation / MeasureOption / Cost), `plan.py`, `scenario.py`, `product.py`, `contingencies.py`, `simulation.py` (EpcSimulation overlay).
- `generators/` — `wall_recommendation` / `roof_recommendation` / `floor_recommendation`.
- `scoring/` — `overlay_applicator` (apply_simulations), `package_scorer` (role 2), `scoring` (role-1 `independent_option_impacts` + role-3 `marginal_impacts`). Note the path is `domain.modelling.scoring.scoring` for the role-1/3 module.
- `optimisation/` — `optimiser`, `measure_dependency`.

## What's built (all in `domain/modelling/`, `infrastructure/postgres/`, `repositories/`, `orchestration/`)

- Generators (`generators/`): `recommend_cavity_wall` / `recommend_loft_insulation` (300 mm) / `recommend_floor_insulation` (sets `floor_insulation_type_str`).
- `simulation.py` overlay + `scoring/overlay_applicator.apply_simulations` (generic field-fold) + `scoring/package_scorer.PackageScorer.score` (role 2) + `scoring/scoring.py` (`marginal_impacts` role 3, `independent_option_impacts` role 1).
- `scenario.py` `Scenario(id, goal, goal_value, budget, is_default)`; `plan.py` `Plan` + `PlanMeasure` (derives cost_of_works/contingency_cost/co2_savings/post_epc_rating).
- `optimisation/optimiser.py` — `optimise(groups, budget)` (exact knapsack) + `optimise_package(...)` (re-score + greedy repair, `Scorer` Protocol, `OptimisedPackage`).
- `infrastructure/postgres/`: `scenario_table.ScenarioRow`, `plan_table.{PlanRow,RecommendationRow}` (mirrors of live tables; `from_domain`).
- `repositories/`: `scenario/`, `plan/`, `product/` (Postgres + Json) — all on the `UnitOfWork` (`uow.scenario`/`uow.product`/`uow.plan`).
- `ModellingOrchestrator.run(property_ids, scenario_ids, portfolio_id)` — one UoW, commit once; generate (wall/roof/floor) → role-1 score → `optimise_package` → role-3 attribute → persist. Wired into `AraFirstRunPipeline` + `handler.py`.
- `datatypes/epc/domain/epc.py::Epc.sap_lower_bound()` (band → min SAP, target for INCREASING_EPC).

## Gotchas (will bite a fresh agent)

- **`mip` / CBC is broken on aarch64** here — never build runnable code on `mip`. The legacy `recommendations/optimiser/` tests only "pass" because they avoid constructing a `mip.Model`.
- **`moto` is not installed** — `tests/orchestration/test_postcode_splitter_orchestrator.py` and `tests/repositories/unstandardised_address/` fail at *collection*. Pre-existing, unrelated; `--ignore` them when sweeping.
- **Run tests:** `python -m pytest <path> -q` (do NOT pass `-p no:cov`). Ephemeral Postgres via the `db_engine` fixture builds **only `SQLModel.metadata`** — legacy `Base` tables are absent in tests, which is why mirrors work.
- **Worktree import trap:** `python /tmp/foo.py` imports `/workspaces/model`, not this worktree. Use `pytest` (rootdir handles it) or a `python -c` from the worktree root.
- **Driving Modelling in an integration test:** the calculator fixtures (`_elmhurst_worksheet_000490.build_epc()`) lack lodged recorded-performance fields, so the **Baseline stage can't run on them**. Drive `ModellingOrchestrator` directly off a repo-seeded EPC (`EpcPostgresRepository(session).save(epc, property_id, portfolio_id)`) — see `test_modelling_optimises_and_persists_a_multi_measure_plan`. The sample API EPC (`_lodged_epc()`) does go through the full pipeline.
- **`PortfolioGoal.INCREASING_EPC` value is `"Increasing EPC"`** (with a space) — the orchestrator compares `scenario.goal == "Increasing EPC"`.
- A generator calls `products.get(...)` during candidate generation, so the integration test must **seed a `material` row for every measure type that fires** (e.g. the sample EPC's uninsulated solid floor needs `solid_floor_insulation`).
- **Don't edit the SAP calculator's `heat_transmission.py`** (another agent owns it).

## Conventions

Commit per TDD slice; conventional-commit message ending `Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>`; stay on `feature/bill-derivation`. Tests use literal `# Arrange / # Act / # Assert`; assert with `abs(x - y) <= tol` (not `pytest.approx`); pyright strict, zero errors; annotate call-return locals. Cascade pins target the worksheet at delta 0.

## #1161 — Measure Dependency (ventilation), as built (4 TDD slices, all green)

Forks resolved with the user (AskUserQuestion): **guard now** (skip when already MEV/MVHR), **persist as a Plan Measure** (cost + real negative marginal), **forced but its cost counts toward spend** (mandatory-when-triggered, never budget-gated; repair sees less headroom).

1. **`7c59e919`** — Simulation Overlay grows a dwelling-level segment: `VentilationOverlay` (all-optional partial of `SapVentilation`, field `mechanical_ventilation_kind`) + `EpcSimulation.ventilation`; `apply_simulations` folds it onto `sap_ventilation` (creating one if the baseline lodged none). Until now the overlay was building-part only — ventilation is whole-dwelling.
2. **`6b11c902`** — generic injection in the optimiser: `MeasureDependency(triggers: frozenset[str], required: ScoredOption)` lives in `optimisation/optimiser.py` (its input contract). `optimise_package(..., dependencies=())` injects any dependency whose triggers ∩ selected-measure-types, before every re-score (initial **and** each repair). `_inject` dedups by required measure-type. Forced (injected even over budget) but its cost is in `_package_cost`, so repair headroom shrinks. `_best_repair_candidate` folds in any dependency a candidate newly triggers, so its marginal SAP and incremental cost are truthful; affordability gates on whole-package cost vs budget. Returned `selected` includes the injected deps. Optimiser stays domain-agnostic — no ventilation import.
3. **`1bf5b410`** — `domain/modelling/optimisation/measure_dependency.py`: `MEASURES_NEEDING_VENTILATION` (cavity/internal/external wall, cf. legacy `assumptions.measures_needing_ventilation`) + `ventilation_dependency(epc, products)` → MEV Option (`mechanical_ventilation_kind="EXTRACT_OR_PIV_OUTSIDE"`, decentralised MEV = legacy "mechanical, extract only"), priced at 2 fully-loaded units. Returns **None** when `sap_ventilation.mechanical_ventilation_kind` is already set (= legacy `has_ventilation` — confirmed against `backend/Property.py:1236`). Note: builder fetches the Product up-front, so the catalogue needs a `mechanical_ventilation` row for **every** not-yet-ventilated dwelling, even if no wall is ultimately selected.
4. **`0fec0699`** — orchestrator wiring: `_measure_dependencies` builds the (≤1) dependency; `_BEST_PRACTICE_ORDER` gains `"mechanical_ventilation"` between loft and floors (role-3 cascade walls→roof→**vent**→floor); ventilation persists as a Plan Measure with its real negative marginal + cost. Added `mechanical_ventilation: 0.26` contingency (legacy `Costs.CONTINGENCIES`). On 000490 the real calculator scores MEV at **−1.275 SAP**.

Gotchas for the next agent: the ventilation Product/contingency must exist for any not-yet-ventilated dwelling (build-time fetch, not inject-time); the stub scorer in `test_optimiser.py` indexes `building_parts[MAIN]`, so vent-only overlays need the separate `_VentStubScorer`.

## What's left

**Deferred fronts** (open, post-#1161): exclusion-filtering of the candidate pool (deferred from #1160); a **Bill-Derivation slice** that re-runs bills on the post-package EPC to fill the deferred energy/bill columns (`plan.post_energy_consumption`/`post_energy_bill`, `recommendation.kwh_savings`/`energy_cost_savings`); persist **unselected alternatives** (`default=False` rows linked via `plan_id`) for the swap-in UX — open ADR-0016 question: what impact figure they carry; promote `ProductRepository` to the DB+file composite; non-EPC goal objectives (Energy Savings, Reducing CO2) in the optimiser. Possible extension of the ventilation trigger set to roof insulation (now a one-line data edit in `MEASURES_NEEDING_VENTILATION`); and making the dependency builder lazy (thunk) so the Product is only fetched when a trigger is actually selected.

## Key references

- ADRs: **0005** (multi-phase deferred), **0011/0012** (orchestrators + UoW), **0016** (three scoring roles + warm-start/re-score/repair), **0017** (Plan persistence — evolve live tables).
- `CONTEXT.md`: Plan, Plan Measure, Recommendation, Measure Option, Optimised Package, Scenario, Measure Dependency.
- Auto-memory `project_modelling_stage_state` has the running state.