ADR-0055 + ADR-0056: Modelling Run distributor decisions and glossary

The distributor pre-creates batch sub_tasks under the app-owned task and
sub_task status is a record, not retry fuel; filter resolution is a shared
contract with the app's preview. Scenario immutability replaces the
Scenario Snapshot fiction; Modelling Run, Distributor and First Run terms
pinned in CONTEXT.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

CONTEXT.md

@@ -199,8 +199,16 @@ _Avoid_: rebaseline (that is a specific ML trigger — see Rebaselining), enrich
 The third stage. Takes the baselined Property plus a set of **Scenarios** and produces **Recommendations** → an **Optimised Package** → **Plans**, persisted to repos. A separate orchestrator from Baseline so the single-property flow can stop after Baseline and only run Modelling when the user hits "play".
 _Avoid_: scoring (overloaded), recommendation engine
 
+**Modelling Run**:
+One triggered unit of modelling over a portfolio: a target set of **Properties** resolved from user-chosen filters (no filters = the whole portfolio) crossed with one or more **Scenarios**, producing one **Plan** per (Scenario, Property). Tracked as a single task; its batch sub_tasks are the units of execution, of failure, and of re-run. Re-runs append Plans; readers take the latest per Property.
+_Avoid_: modelling job (ambiguous with one lambda invocation), batch (that is one message-worth of a run), trigger run
+
+**Distributor**:
+The role of the Modelling Run entry point: validate the request, resolve the filters to a concrete Property set, pre-create the run's batch sub_tasks, and fan the work out to the modelling workers. It never models synchronously and owns nothing after the fan-out — progress and terminal state roll up from the workers' sub_task statuses.
+_Avoid_: trigger endpoint (names the URL, not the role), orchestrator (taken — stage orchestrators, TaskOrchestrator)
+
 **First Run**:
-The use case where a Property has only a row in the property table (post address→UPRN matching) and no existing **Plan**: the pipeline runs Ingestion → Baseline → Modelling end-to-end over a batch. The first sibling lambda being built (`ara_first_run`).
+The special case of a **Modelling Run** where a Property has only a row in the property table (post address→UPRN matching) and no existing **Plan**: the pipeline runs Ingestion → Baseline → Modelling end-to-end. Executed by the same `modelling_e2e` worker as re-runs — the lambda originally planned as `ara_first_run` serves both.
 _Avoid_: initial run, cold run
 
 ### ML training
@@ -238,12 +246,8 @@ _Avoid_: emission factors (ambiguous), CO2 rates
 ### Outputs
 
 **Scenario**:
-A named portfolio-level retrofit plan, built by a user in the scenario-builder UI and persisted before any modelling fires; carries the overall goal (e.g. Increasing EPC), budget, exclusions, housing type, and the set of measure types it permits. The model is triggered against one or more Scenarios at once; each Scenario yields one Plan per Property.
-_Avoid_: project, batch, run-set
-
-**Scenario Snapshot**:
-A frozen copy of a Scenario pinned at trigger time, keyed by (task, scenario); used by the modelling pipeline so mid-run edits to the live Scenario do not affect an in-flight job. Snapshots are read-only and may be garbage-collected after the task completes.
-_Avoid_: scenario version, frozen scenario, pinned scenario
+A named portfolio-level retrofit plan, built by a user in the scenario-builder UI and persisted before any modelling fires; carries the overall goal (e.g. Increasing EPC), budget, exclusions, housing type, and the set of measure types it permits. The model is triggered against one or more Scenarios at once; each Scenario yields one Plan per Property. Scenarios are **immutable after creation**: they may be renamed, and deleted only while no Plans reference them — their modelling-relevant values never change, so an in-flight run can safely read the live row (no snapshot/pinning machinery is needed or exists).
+_Avoid_: project, batch, run-set, scenario snapshot (described pinning machinery that immutability makes unnecessary; removed 2026-07)
 
 **Plan**:
 The per-Property output of one Scenario's modelling run; carries the **Optimised Package** selected for the Property (its **Plan Measures**) and the Property's post-retrofit figures (SAP / kWh / CO₂ / bills). A Property modelled against N Scenarios in one trigger ends up with N Plans.
@@ -443,7 +447,7 @@ addresses `<michaelduong@Michaels-MacBook-Pro.local>`, `<michael@11s-MacBook.loc
 - **Bill Derivation** derives **fuel split** and **bills** from kWh values (sourced from the EPC's `renewable_heat_incentive` fields for baseline SAP10 properties, or from ML when Rebaselining fires), reading current **Fuel Rates** and **Carbon Factors** from their respective repos.
 - **EPC Prediction** uses **Comparable Properties** for both gap-filling (the no-EPC path) and producing **EPC Anomaly Flags** (the has-EPC path).
 - Triggering the model against N **Scenarios** produces N **Plans** per Property. Each **Plan** holds one **Optimised Package** — its selected **Plan Measures** — plus the Property's post-retrofit figures.
-- A **Scenario Snapshot** is pinned at trigger time per (task, scenario) so mid-run edits to the live Scenario do not affect an in-flight modelling job.
+- **Scenarios** are immutable after creation (rename-only; deletable only while no Plans reference them), so in-flight modelling jobs read the live Scenario row safely.
 - A **Recommendation** references one **Measure Type** and carries property-specific cost and impact.
 - A **Property Valuation** (current market value) is a Baseline attribute and is mostly absent; a **Valuation Uplift** is a Plan output, always a percentage from the **EPC Band** jump and an absolute £ only when a Property Valuation exists.
 - **Address Matching** uses a **User Address** and **Postcode** to find a **UPRN** by scoring **UPRN Candidates** from an EPC search. A **Lexirank** of 1 with no **Ambiguous Match** and a **Lexiscore** ≥ the **Score Threshold** produces a **Best Match**.

docs/adr/0055-modelling-run-batches-attach-to-the-app-owned-task.md

@@ -0,0 +1,103 @@
+# A Modelling Run's batch sub_tasks are pre-created by the distributor and attach to the app-owned task; their status is a record, not retry logic
+
+## Status
+
+accepted
+
+## Context
+
+The Ara app is gaining a filter-scoped modelling trigger: `POST
+/v1/modelling/trigger-run` accepts a portfolio, a set of scenario ids, and
+property-group filters, and fans one plan-generation job per (Scenario,
+Property) out to the existing `modelling_e2e` workers. The app creates the
+task before calling (service `modelling_run`, status `in progress`, the full
+request JSON in the new `tasks.inputs` column) and renders progress as
+`count(completed)/count(all)` over the task's sub_tasks.
+
+Two prior patterns conflicted:
+
+- The legacy `/v1/plan/trigger` route creates the task **and** its per-chunk
+  sub_tasks in the API, passing `task_id`/`subtask_id` in each SQS message.
+- The `modelling_e2e` worker's `@task_handler` creates **its own** Task per
+  SQS message and fans per-property child sub_tasks under it
+  (`TaskOrchestrator.run_subtasks`); nothing can attach to a caller's task.
+
+The parent-status roll-up (`Task.recalculate_from_subtasks`) had three
+properties that break shared-parent fan-out: a complete+waiting mix rolls up
+to `waiting` (a status the app does not render); `SubTask.start()` refuses to
+start from `failed` (breaking any redelivery/re-run of a failed batch); and
+`TaskOrchestrator._cascade` short-circuits on a FAILED parent, making failure
+permanently sticky.
+
+## Decisions
+
+### 1. The distributor pre-creates one sub_task per SQS message at accept time
+
+One task spans the whole run. The distributor resolves the filters, batches
+the properties (50 per message, postcode-grouped so the workers' prediction
+cohort cache keeps paying, one message per (scenario, batch)), bulk-creates
+one `waiting` sub_task per message under the app's task, and sends each
+message carrying `task_id`, `subtask_id`, and the batch payload. Each
+sub_task's `inputs` is the **exact message payload** — the run's precise
+what-ran record, and a failed batch's re-run recipe (re-send its own inputs).
+
+The progress denominator is therefore fixed and correct the moment the
+endpoint returns 202, and it counts **batches, not properties**.
+
+### 2. The worker gains an attach mode; the script path is untouched
+
+When a message carries `task_id` + `subtask_id`, `task_handler` creates no
+Task and the handler runs the whole batch under the supplied sub_task —
+no per-property child sub_tasks. Without those fields, current behaviour
+(own task, per-property children) is unchanged, so
+`scripts/trigger_modelling_e2e_sqs.py` keeps working.
+
+### 3. Sub_task status is a database record, not retry fuel
+
+Per-property failures inside a batch (unresolvable type, degenerate
+prediction, …) stay isolated as today — the surviving properties model and
+persist. But if **any** property failed, the batch sub_task is marked
+`failed` with `outputs = {succeeded: n, failed: [{property_id, error}]}`,
+and the parent task rolls up `failed`. The lambda still succeeds to SQS:
+a `failed` sub_task **never** triggers redelivery. Recovery is deliberate —
+fix the cause, re-send the sub_task's own `inputs`, the sub_task completes,
+and the parent task un-fails to `complete`. Batch-level crashes (OOM,
+timeout, unhandled infra errors) keep normal SQS redelivery semantics; that
+is infrastructure retry, not status-driven retry.
+
+### 4. The roll-up rules change to make failure recoverable and progress honest
+
+- A complete/failed + waiting mix rolls up `in progress`, not `waiting`.
+- `SubTask.start()` may start from `failed` (redelivery and re-runs).
+- `_cascade` no longer short-circuits on a FAILED parent: the parent status
+  is always recomputed from the children, so completing a re-run batch
+  flips the task from `failed` to `complete`.
+
+### 5. Contract guards
+
+- `409` if the task already has sub_tasks (double-submit / blind retry
+  protection — the app checks progress instead of re-POSTing).
+- `400` if the filters resolve to zero properties (a zero-sub_task task
+  could never complete; the app's preview shows the same count first).
+- `4xx` if any scenario does not belong to the portfolio. Scenarios are
+  immutable after creation (rename-only; deletable only while no Plans
+  reference them), so workers read the live Scenario row safely.
+- Pipeline flags are pinned by the endpoint, not the caller:
+  `refetch_epc=True, repredict_epc=True, refetch_solar=True, dry_run=False`.
+  Note `refetch_solar` is a misnomer: it only gates the Google fetch for
+  UPRNs with **no stored row**; stored Solar is never re-fetched.
+
+## Consequences
+
+- The app's progress bar denominator is batch count (≈ `ceil(N/50)` ×
+  scenarios), not the preview's property count. Property-level failure
+  detail is summed from sub_task `outputs`.
+- A 10k-property run whose single bad batch is fixed and re-run ends
+  `complete` — no permanently red runs over recoverable data issues.
+- The roll-up changes are global to the task machinery; other task_handler
+  lambdas keep their semantics (their per-message tasks have no shared
+  parent), but FAILED-parent recomputation now costs one children read.
+- FastAPI needs the modelling_e2e queue wired: `MODELLING_E2E_SQS_URL` from
+  the modelling_e2e remote state (outputs already exported), its ARN in the
+  `fastapi-sqs-send` policy, and `deploy_terraform.yml`'s `fast_api_lambda`
+  job ordered after `modelling_e2e_lambda`.

docs/adr/0056-modelling-run-filter-resolution-is-a-shared-contract-with-the-app-preview.md

@@ -0,0 +1,59 @@
+# Modelling Run filter resolution is a shared contract with the app's preview; the precedence order is authoritative here
+
+## Status
+
+accepted
+
+## Context
+
+The trigger-run request carries filters (`postcodes`, `property_types`,
+`built_forms`) that the distributor resolves to a concrete property set. The
+Next.js app shows users "N properties will be modelled" **before** POSTing,
+computed by its own implementation of the same rule. If the two
+implementations drift, the preview lies — the run models a different set
+than the user approved. There is no shared code path between the two
+codebases, so the rule itself must be the contract, written down once.
+
+A property's type and built form have no single column of truth: they may be
+overridden by the landlord, derived from a lodged or predicted EPC (as text
+labels or RdSAP numeric codes), or sit in legacy `property` columns — or be
+unknowable.
+
+## Decision
+
+Both implementations resolve the filters with exactly this rule; any change
+lands in both codebases and amends this ADR.
+
+1. **Base set**: `property` rows where `portfolio_id` matches and
+   `marked_for_deletion = false`.
+2. **postcodes**: exact match on `property.postcode`, canonical form
+   (uppercase, single space). The app pre-normalises and caps the list at 40.
+3. **property_types / built_forms** — resolve each property's value by
+   precedence, then filter:
+   1. `property_overrides` snapshot where `building_part = 0` and
+      `override_component` is `property_type` / `built_form_type` → the
+      `override_value`;
+   2. else EPC-derived: `epc_property` for the property, `source='lodged'`
+      preferred over `'predicted'`. Values are text labels **or** RdSAP
+      numeric codes; codes map as — property_type: 0=House, 1=Bungalow,
+      2=Flat, 3=Maisonette, 4=Park home, falling back to `dwelling_type`;
+      built_form: 1=Detached, 2=Semi-Detached, 3=End-Terrace, 4=Mid-Terrace,
+      5=Enclosed End-Terrace, 6=Enclosed Mid-Terrace. Text passes through
+      as-is;
+   3. else the legacy `property.property_type` / `property.built_form`
+      columns;
+   4. else **"Unknown"** — a selectable filter value meaning exactly this
+      bucket (no resolvable value at any level).
+4. An absent filter key is unconstrained; present keys combine with **AND**.
+
+## Consequences
+
+- "N properties will be modelled" in the preview equals the number the
+  distributor fans out (the run's property count; the task's sub_task count
+  is batches, per ADR-0055).
+- The precedence means an override always beats a cert and a cert always
+  beats the legacy columns — consistent with how the modelling pipeline
+  itself treats Landlord Overrides as strongest truth.
+- Drift risk is accepted and mitigated by this document; a cross-repo
+  contract test (same fixture set, both implementations) is the natural
+  follow-on if drift ever bites.