Model/docs/HANDOVER_MODELLING.md

# HANDOVER — Modelling stage rebuild

**Branch:** `feature/bill-derivation` (worktree `/workspaces/home/hestia-worktrees/model-assemble-new-backend`). **HEAD:** `34d4748a`.
**PRD:** GitHub `Hestia-Homes/Model#1152`, sliced into #1153–#1161.

## Issue status

| Issue | What | State |
|---|---|---|
| #1153 | Overlay Applicator + `EpcSimulation` | ✅ closed |
| #1154 | Package Scorer | ✅ closed — Elmhurst cascade pin (`4c0a907a`) |
| #1155 | wall Recommendation Generator | ✅ closed; cascade-pinned |
| #1156 | score Options + attribution | ✅ closed |
| #1157 | persist a Plan via `ModellingOrchestrator` | ✅ **closed this session** (`772cdd4f`→`c7e2aa37`) |
| #1158 | roof (loft) generator | ✅ closed — 300 mm + cascade pin |
| #1159 | floor generator | ✅ closed — overlay insulation-type field + pins |
| #1160 | Optimiser (knapsack + greedy repair) | ✅ **closed this session** (`77983cae`→`34d4748a`) |
| #1161 | Measure Dependency (ventilation) | **NOT STARTED — next** |

## What this session did

1. **Cascade pins for #1154/#1158/#1159** — `tests/domain/modelling/test_elmhurst_cascade_pins.py`. Parse Elmhurst before/after recommendation Summaries via the extractor chain (NOT `parse_site_notes_pdf`), apply the generator's overlay, score, assert delta 0 vs the after-cert. Found+fixed: loft 270→**300** mm; suspended floor needs the overlay to also set `floor_insulation_type_str='Retro-fitted'`.
2. **`ProductJsonRepository`** (`cc0bb8f9`) — file-backed catalogue behind the `ProductRepository` port.
3. **#1157 — persist a Plan.** Design review (`/grill-with-docs`) + 5 TDD slices. See "Design decisions" below.
4. **#1160 — the Optimiser.** 4 TDD slices. See "Design decisions".

## Design decisions locked this session (READ THESE)

- **Multi-phase is DEFERRED** (speculative prospective-client ask). **ADR-0005 rewritten to "Deferred".** No `plan_phase` table, no `phase` column. `CONTEXT.md` no longer has Scenario Phase / Plan Phase / Rolled-over Options. Everything is **single-phase**. Future: a migration adds `plan_phase` + back-fills live plans as 1-phase.
- **Plan Measure** is the new term (in `CONTEXT.md`): the persisted selected Option + its role-3 attributed impact + cost. **Recommendation** stays the *candidate* (never persisted; no stored impact).
- **Reuse the LIVE tables** (`plan`, `recommendation`) — they exist in the live product (`backend/app/db/models/recommendations.py`, SQLAlchemy `Base`) and the FE reads them. The rebuild writes the **same physical tables via SQLModel mirrors** (`infrastructure/postgres/plan_table.py`) — the established pattern (`task_table.py`→`tasks`, `product_table.py`→`material`). **ADR-0017** records this.
- Added **`recommendation.plan_id`** (FK→plan, ON DELETE CASCADE); retire the `plan_recommendations` m2m for new writes. FE-owned Drizzle migration: `docs/migrations/recommendation-plan-id.md`.
- Tracer persists **SAP + CO₂ (tonnes = calc kg ÷ 1000) + cost + derived `post_epc_rating`**. Energy/bill columns deferred. Idempotent replace per (property_id, scenario_id).
- **Optimiser = exact pure-Python multiple-choice knapsack**, NOT `mip`. Recycles `GainOptimiser`/`CostOptimiser`'s *formulation* (≤1/group, maximise gain s.t. budget) but not the dependency — **`mip`'s CBC backend does not load on this aarch64 container** (`NameError: cbclib`), so the legacy solver can't run/be tested here. ADR-0016's MILP is only a warm-start signal, so exact small-scale enumeration is ample. Re-score + greedy-repair toward the goal's SAP target gives the truth.

## What's built (all in `domain/modelling/`, `infrastructure/postgres/`, `repositories/`, `orchestration/`)

- Generators: `recommend_cavity_wall` / `recommend_loft_insulation` (300 mm) / `recommend_floor_insulation` (sets `floor_insulation_type_str`).
- `simulation.py` overlay + `overlay_applicator.apply_simulations` (generic field-fold) + `package_scorer.PackageScorer.score` (role 2) + `scoring.py` (`marginal_impacts` role 3, `independent_option_impacts` role 1).
- `scenario.py` `Scenario(id, goal, goal_value, budget, is_default)`; `plan.py` `Plan` + `PlanMeasure` (derives cost_of_works/contingency_cost/co2_savings/post_epc_rating).
- `optimiser.py` — `optimise(groups, budget)` (exact knapsack) + `optimise_package(...)` (re-score + greedy repair, `Scorer` Protocol, `OptimisedPackage`).
- `infrastructure/postgres/`: `scenario_table.ScenarioRow`, `plan_table.{PlanRow,RecommendationRow}` (mirrors of live tables; `from_domain`).
- `repositories/`: `scenario/`, `plan/`, `product/` (Postgres + Json) — all on the `UnitOfWork` (`uow.scenario`/`uow.product`/`uow.plan`).
- `ModellingOrchestrator.run(property_ids, scenario_ids, portfolio_id)` — one UoW, commit once; generate (wall/roof/floor) → role-1 score → `optimise_package` → role-3 attribute → persist. Wired into `AraFirstRunPipeline` + `handler.py`.
- `datatypes/epc/domain/epc.py::Epc.sap_lower_bound()` (band → min SAP, target for INCREASING_EPC).

## Gotchas (will bite a fresh agent)

- **`mip` / CBC is broken on aarch64** here — never build runnable code on `mip`. The legacy `recommendations/optimiser/` tests only "pass" because they avoid constructing a `mip.Model`.
- **`moto` is not installed** — `tests/orchestration/test_postcode_splitter_orchestrator.py` and `tests/repositories/unstandardised_address/` fail at *collection*. Pre-existing, unrelated; `--ignore` them when sweeping.
- **Run tests:** `python -m pytest <path> -q` (do NOT pass `-p no:cov`). Ephemeral Postgres via the `db_engine` fixture builds **only `SQLModel.metadata`** — legacy `Base` tables are absent in tests, which is why mirrors work.
- **Worktree import trap:** `python /tmp/foo.py` imports `/workspaces/model`, not this worktree. Use `pytest` (rootdir handles it) or a `python -c` from the worktree root.
- **Driving Modelling in an integration test:** the calculator fixtures (`_elmhurst_worksheet_000490.build_epc()`) lack lodged recorded-performance fields, so the **Baseline stage can't run on them**. Drive `ModellingOrchestrator` directly off a repo-seeded EPC (`EpcPostgresRepository(session).save(epc, property_id, portfolio_id)`) — see `test_modelling_optimises_and_persists_a_multi_measure_plan`. The sample API EPC (`_lodged_epc()`) does go through the full pipeline.
- **`PortfolioGoal.INCREASING_EPC` value is `"Increasing EPC"`** (with a space) — the orchestrator compares `scenario.goal == "Increasing EPC"`.
- A generator calls `products.get(...)` during candidate generation, so the integration test must **seed a `material` row for every measure type that fires** (e.g. the sample EPC's uninsulated solid floor needs `solid_floor_insulation`).
- **Don't edit the SAP calculator's `heat_transmission.py`** (another agent owns it).

## Conventions

Commit per TDD slice; conventional-commit message ending `Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>`; stay on `feature/bill-derivation`. Tests use literal `# Arrange / # Act / # Assert`; assert with `abs(x - y) <= tol` (not `pytest.approx`); pyright strict, zero errors; annotate call-return locals. Cascade pins target the worksheet at delta 0.

## What's left

1. **#1161 — Measure Dependency (ventilation).** Per ADR-0016: a forced dependency (wall/roof insulation requires adequate ventilation) is **excluded from the optimiser's candidate pool** but **injected into the Optimised Package BEFORE the re-score**, so its (negative) SAP contribution lands in the truthful figure and the repair decision. Trigger set held as data (cf. legacy `assumptions.measures_needing_ventilation`). The injection point is `optimise_package` in `domain/modelling/optimiser.py` (inject after warm-start, before/within the re-score).
2. **Deferred fronts** (open, post-#1161): exclusion-filtering of the candidate pool (deferred from #1160); a **Bill-Derivation slice** that re-runs bills on the post-package EPC to fill the deferred energy/bill columns (`plan.post_energy_consumption`/`post_energy_bill`, `recommendation.kwh_savings`/`energy_cost_savings`); persist **unselected alternatives** (`default=False` rows linked via `plan_id`) for the swap-in UX — open ADR-0016 question: what impact figure they carry; promote `ProductRepository` to the DB+file composite; non-EPC goal objectives (Energy Savings, Reducing CO2) in the optimiser.

## Key references

- ADRs: **0005** (multi-phase deferred), **0011/0012** (orchestrators + UoW), **0016** (three scoring roles + warm-start/re-score/repair), **0017** (Plan persistence — evolve live tables).
- `CONTEXT.md`: Plan, Plan Measure, Recommendation, Measure Option, Optimised Package, Scenario, Measure Dependency.
- Auto-memory `project_modelling_stage_state` has the running state.