From e58a90a17b081a4bcc7443a27c9c260aeb6755b8 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 6 Jul 2026 15:45:43 +0000 Subject: [PATCH 01/11] docs: ModellingRun + Run filter language and ADR-0008 (filters to distributor) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Grilling-session outcomes for the bulk-trigger-modelling feature: the ModellingRun/Run-filter glossary entries and the decision record — insert-only run rows, filters (not ids) to the distributor lambda, backend-owned filter resolution with a shared-code-path count endpoint, run_id correlation into task records for derived in-flight state, warn-don't-block concurrency. Co-Authored-By: Claude Fable 5 --- CONTEXT.md | 8 ++ ...8-modelling-runs-filters-to-distributor.md | 87 +++++++++++++++++++ 2 files changed, 95 insertions(+) create mode 100644 docs/adr/0008-modelling-runs-filters-to-distributor.md diff --git a/CONTEXT.md b/CONTEXT.md index 46f0f7a9..9d60223d 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -104,6 +104,14 @@ _Avoid_: invasive (unused here), wet trades (a narrower, overlapping set: plaste A Scenario constraint that takes a measure off the table for the optimiser. The *only* measure-constraint concept on a Scenario — there is no stored inclusion list; "only Solar PV + ASHP" is expressed by excluding everything else. An empty exclusion set is valid and means every measure is available. _Avoid_: inclusions, allowed measures (as a stored concept — UI may present "allowed/excluded" toggles, but what's captured is the exclusion set) +**Modelling run**: +One user-initiated act of triggering modelling: a set of Scenarios crossed with one filter-defined property selection within a Portfolio. An insert-only record of *what was asked* — the Run filters, the Scenarios, who triggered it, when, and the matched-property count shown at preview — so Plans can be traced back to the selection that produced them. Execution progress is a separate concern (the pipeline's task records own it); a Modelling run is the request and its provenance, not the work. A Scenario is the *question*; a Modelling run *asks* it (possibly again, possibly for a different property selection). Re-running an already-modelled property appends a newer Plan; latest wins on read. +_Avoid_: job, batch, trigger request, modelling task (the pipeline's execution records) + +**Run filter**: +The property-selecting constraints of a Modelling run: postcode, property type, and built form (each multi-select; tags later). An absent filter means *unconstrained* — "all postcodes" is the absence of a postcode filter, never an enumeration. Type and built form are the canonical enum vocabularies plus **Unknown**, which selects properties with no resolvable value at all. Resolution follows the Landlord-override rule: an override fact beats an EPC-derived value; the modelling backend is the single authority for resolving filters to properties (the app never re-implements the rule — matched counts come from the backend). +_Avoid_: property query, segment, search (the postcode-search journey is unrelated) + ## Lifecycle A **BulkUpload** moves through these statuses: diff --git a/docs/adr/0008-modelling-runs-filters-to-distributor.md b/docs/adr/0008-modelling-runs-filters-to-distributor.md new file mode 100644 index 00000000..dd95ee69 --- /dev/null +++ b/docs/adr/0008-modelling-runs-filters-to-distributor.md @@ -0,0 +1,87 @@ +# 8. Modelling runs pass filters to the distributor; the backend owns resolution + +Date: 2026-07-06 + +## Status + +Accepted + +Numbering note: 0004–0006 are referenced by CONTEXT.md but were lost to the old +`docs/adr/**` gitignore rule; 0008 avoids all collisions (0003 = app-authored +scenarios, 0007 = postcode search). + +## Context + +Users can now add properties (postcode search, bulk upload) and author +Scenarios (ADR-0003) before any modelling exists. The missing step is +triggering modelling for *chosen* properties: many Scenarios × a subset of the +portfolio, potentially 50,000 properties. The entry point on the Model side is +a **distributor** lambda — it fans work out to modelling workers, it does not +model synchronously. ADR-0003 recorded a known follow-up: the in-flight +"modelling running" state is not representable by derivation and needs an +explicit marker when trigger wiring lands. This is that wiring. + +## Decision + +- **A `ModellingRun` is recorded, insert-only, by Next.js at trigger time**: + portfolio, the selected scenario ids, the Run filters, the matched-property + count shown at preview, who triggered it, when. It is the request and its + provenance — Plans become traceable to the selection that produced them. + It carries no status column (mirrors ADR-0003's derived-state rule). +- **The distributor receives `{run_id, portfolio_id, scenario_ids, filters}`** + — filters, never property-id lists. The payload stays tiny at any portfolio + size; the run row is the durable copy of the same config. +- **The modelling backend is the single authority for resolving filters to + properties.** Resolution precedence: landlord-override fact → EPC-derived + value (new-approach EPC graph, lodged over predicted, or legacy property-row + columns) → **Unknown**. The app never re-implements this rule. +- **Preview counts come from a backend dry-run/count endpoint** taking the + same filters through the same resolution code path as the real run — the + preview is a promise, not an estimate. The response includes per-scenario + already-modelled counts so the UI can flag re-modelling ("120 of 214 will + be re-modelled"). Plan arithmetic is exact: one plan per + (scenario, property) per run. +- **Filter options are resolution-free in the app**: the postcode multi-select + is one `SELECT DISTINCT postcode` per portfolio; type/built-form options are + the static canonical enums plus Unknown. Filters are bounded (cap the + postcode multi-select) as a UX guard; "all" is an absent filter, never an + enumeration. +- **Run status is derived by correlation, not stored**: the distributor stamps + `run_id` on the task records the pipeline already writes + (`tasks`/`sub_tasks`, service `plan_engine`). Scenario badges gain a third + derived state — "Modelling in progress" — and a per-portfolio run history + lists each run's filters, counts, initiator and derived status. No live + progress bars in v1. +- **Concurrent runs warn, never block**: triggering a scenario with a run in + flight shows who started it and when, but proceeds. Plans append and + latest-per-property wins on read, so overlap is safe; a hard lock could leak + on a dead run. + +## Alternatives considered + +- **Pass resolved property ids to the distributor**: makes preview-vs-run + drift impossible by construction, but ~50k ids is a payload the async entry + point can't take, and it moves the resolution rule into the app — the + backend already owns it for modelling itself. +- **App-side count queries mirroring the resolution rule**: no backend + dependency for the preview, but the overrides-else-EPC rule would live in + two codebases and the shown count could disagree with what runs. +- **Status columns on the run row (two-writer, as bulk uploads)**: simple + reads, but a second source of truth that can silently never arrive; task + correlation reuses records the pipeline writes anyway. +- **Blocking concurrent runs**: prevents accidental double-spend but a stalled + pipeline would lock the scenario until someone clears task rows by hand. + +## Consequences + +- Two backend asks must land with the distributor: accept + persist `run_id` + into task records, and expose the dry-run/count endpoint sharing the + resolution code path. +- The preview's already-modelled callout and matched counts are only as fresh + as trigger time; a property added between preview and trigger is picked up + by the run (filters re-resolve) — the run row records the previewed count so + discrepancies are visible after the fact. +- Reporting/scenario UIs gain a third derived scenario state; anything + assuming "has a run row ⇒ has plans" is wrong (a dead run may produce none). +- The tagging system (future) extends Run filters without touching the + contract shape: tags become one more optional filter key. From adcc9b8b8315e79ba2c89d1ac78459c6a921b379 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 6 Jul 2026 16:11:08 +0000 Subject: [PATCH 02/11] docs(adr): record run-completion email notification as a known follow-up The dispatch confirmation sets a time expectation only; the email promise stays out of the UI until a completion notification exists (run_id -> task correlation provides the signal to build it on). Co-Authored-By: Claude Fable 5 --- docs/adr/0008-modelling-runs-filters-to-distributor.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/adr/0008-modelling-runs-filters-to-distributor.md b/docs/adr/0008-modelling-runs-filters-to-distributor.md index dd95ee69..9d72e06f 100644 --- a/docs/adr/0008-modelling-runs-filters-to-distributor.md +++ b/docs/adr/0008-modelling-runs-filters-to-distributor.md @@ -85,3 +85,8 @@ explicit marker when trigger wiring lands. This is that wiring. assuming "has a run row ⇒ has plans" is wrong (a dead run may produce none). - The tagging system (future) extends Run filters without touching the contract shape: tags become one more optional filter key. +- Known follow-up: **email notification on run completion**. The dispatch + confirmation sets a time expectation only; "we'll email you when results + are ready" was deliberately kept out of the UI copy until a completion + notification exists. The run_id → task correlation gives the completion + signal to hang it on. From c7dfe95e15ba28f68c28e2ad7e2a31b136d183aa Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 6 Jul 2026 16:27:21 +0000 Subject: [PATCH 03/11] docs(adr): run record is an app-created task, not a new table Amends ADR-0008 after challenge: the Modelling run reuses the BulkUpload trigger convention (app inserts the tasks row + config subtask with inputs JSON, passes task_id to the distributor) instead of a dedicated modelling_run table. No migration; status and progress come from the task system by construction; dispatch failure marks the task failed. Co-Authored-By: Claude Fable 5 --- CONTEXT.md | 2 +- ...8-modelling-runs-filters-to-distributor.md | 48 ++++++++++++------- 2 files changed, 33 insertions(+), 17 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 9d60223d..33bf900a 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -105,7 +105,7 @@ A Scenario constraint that takes a measure off the table for the optimiser. The _Avoid_: inclusions, allowed measures (as a stored concept — UI may present "allowed/excluded" toggles, but what's captured is the exclusion set) **Modelling run**: -One user-initiated act of triggering modelling: a set of Scenarios crossed with one filter-defined property selection within a Portfolio. An insert-only record of *what was asked* — the Run filters, the Scenarios, who triggered it, when, and the matched-property count shown at preview — so Plans can be traced back to the selection that produced them. Execution progress is a separate concern (the pipeline's task records own it); a Modelling run is the request and its provenance, not the work. A Scenario is the *question*; a Modelling run *asks* it (possibly again, possibly for a different property selection). Re-running an already-modelled property appends a newer Plan; latest wins on read. +One user-initiated act of triggering modelling: a set of Scenarios crossed with one filter-defined property selection within a Portfolio. A durable record of *what was asked* — the Run filters, the Scenarios, who triggered it, when, and the matched-property count shown at preview — so Plans can be traced back to the selection that produced them, alongside how the work is going. A Scenario is the *question*; a Modelling run *asks* it (possibly again, possibly for a different property selection). Re-running an already-modelled property appends a newer Plan; latest wins on read. _Avoid_: job, batch, trigger request, modelling task (the pipeline's execution records) **Run filter**: diff --git a/docs/adr/0008-modelling-runs-filters-to-distributor.md b/docs/adr/0008-modelling-runs-filters-to-distributor.md index 9d72e06f..dab4e11e 100644 --- a/docs/adr/0008-modelling-runs-filters-to-distributor.md +++ b/docs/adr/0008-modelling-runs-filters-to-distributor.md @@ -23,14 +23,23 @@ explicit marker when trigger wiring lands. This is that wiring. ## Decision -- **A `ModellingRun` is recorded, insert-only, by Next.js at trigger time**: - portfolio, the selected scenario ids, the Run filters, the matched-property - count shown at preview, who triggered it, when. It is the request and its - provenance — Plans become traceable to the selection that produced them. - It carries no status column (mirrors ADR-0003's derived-state rule). -- **The distributor receives `{run_id, portfolio_id, scenario_ids, filters}`** +- **A Modelling run is recorded as an app-created task** — the BulkUpload + trigger convention (`triggerAddressMatching`): Next.js inserts the `tasks` + row (service `modelling_run`, `source_id` = portfolio, status `waiting`) + plus one config `sub_task` whose `inputs` JSON carries + `{scenario_ids, filters, previewed_property_count, triggered_by}`, then + passes `task_id` to the distributor, which attaches its execution subtasks + to the same task. **No new table, no migration.** Run history and the + per-scenario in-flight badge read the portfolio's own `modelling_run` tasks + (portfolio-scoped, a handful of rows) and parse the app-authored inputs + JSON; execution progress ("61 of 104 ready") falls out of the existing + subtask-count summary pattern. +- **The distributor receives `{task_id, portfolio_id, scenario_ids, filters}`** — filters, never property-id lists. The payload stays tiny at any portfolio - size; the run row is the durable copy of the same config. + size; the config subtask is the durable copy of the same request. +- **Dispatch failure marks the task `failed`** (an improvement on the + bulk-upload flow, which leaves failed triggers at `waiting`), so history + never shows a ghost run as pending. - **The modelling backend is the single authority for resolving filters to properties.** Resolution precedence: landlord-override fact → EPC-derived value (new-approach EPC graph, lodged over predicted, or legacy property-row @@ -46,12 +55,11 @@ explicit marker when trigger wiring lands. This is that wiring. the static canonical enums plus Unknown. Filters are bounded (cap the postcode multi-select) as a UX guard; "all" is an absent filter, never an enumeration. -- **Run status is derived by correlation, not stored**: the distributor stamps - `run_id` on the task records the pipeline already writes - (`tasks`/`sub_tasks`, service `plan_engine`). Scenario badges gain a third - derived state — "Modelling in progress" — and a per-portfolio run history - lists each run's filters, counts, initiator and derived status. No live - progress bars in v1. +- **Run status is the task's status** — no correlation machinery needed: the + run and the execution share one record by construction. Scenario badges + gain a third derived state — "Modelling in progress" — and a per-portfolio + run history lists each run's filters, counts, initiator and derived status + with subtask-count progress. - **Concurrent runs warn, never block**: triggering a scenario with a run in flight shows who started it and when, but proceeds. Plans append and latest-per-property wins on read, so overlap is safe; a hard lock could leak @@ -59,6 +67,13 @@ explicit marker when trigger wiring lands. This is that wiring. ## Alternatives considered +- **A dedicated `modelling_run` table (+ scenarios join table)**: relational + scenario linkage and typed columns for who/filters/counts, but a migration, + a second record system beside tasks, and correlation machinery (either the + backend stamping run ids into tasks, or the app storing returned task ids). + The task system already records app-initiated pipeline work with inputs, + progress and status — and the badge/history queries are portfolio-scoped + over a handful of rows, so the JSON parse costs nothing that matters. - **Pass resolved property ids to the distributor**: makes preview-vs-run drift impossible by construction, but ~50k ids is a payload the async entry point can't take, and it moves the resolution rule into the app — the @@ -74,9 +89,10 @@ explicit marker when trigger wiring lands. This is that wiring. ## Consequences -- Two backend asks must land with the distributor: accept + persist `run_id` - into task records, and expose the dry-run/count endpoint sharing the - resolution code path. +- Two backend asks must land with the distributor: accept an app-created + `task_id` and attach execution subtasks to it (rather than creating its own + task, as the current `plan_engine` entrypoint does), and expose the + dry-run/count endpoint sharing the resolution code path. - The preview's already-modelled callout and matched counts are only as fresh as trigger time; a property added between preview and trigger is picked up by the run (filters re-resolve) — the run row records the previewed count so From 35eb681f8fe1b84a56229dc4e29ed848ef490736 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 6 Jul 2026 16:51:55 +0000 Subject: [PATCH 04/11] =?UTF-8?q?feat(modelling-runs):=20pure=20domain=20c?= =?UTF-8?q?ore=20=E2=80=94=20filters,=20run=20record,=20status,=20bounds?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TDD'd per ADR-0008: normaliseRunFilters (canonical postcodes, enum vocabularies + Unknown, 40-postcode cap, absence = all), buildRunRecord / parseRunConfig (app-authored task + config-subtask inputs, null-safe round-trip), selectionSummary, deriveRunStatus (dispatched / in-progress with subtask progress / complete / failed), isLargeRun (10k gate). Co-Authored-By: Claude Fable 5 --- src/lib/modellingRuns/model.test.ts | 134 +++++++++++++++++++ src/lib/modellingRuns/model.ts | 198 ++++++++++++++++++++++++++++ 2 files changed, 332 insertions(+) create mode 100644 src/lib/modellingRuns/model.test.ts create mode 100644 src/lib/modellingRuns/model.ts diff --git a/src/lib/modellingRuns/model.test.ts b/src/lib/modellingRuns/model.test.ts new file mode 100644 index 00000000..656a72fd --- /dev/null +++ b/src/lib/modellingRuns/model.test.ts @@ -0,0 +1,134 @@ +import { describe, expect, it } from "vitest"; +import { + buildRunRecord, + deriveRunStatus, + isLargeRun, + normaliseRunFilters, + parseRunConfig, + selectionSummary, +} from "./model"; + +describe("normaliseRunFilters", () => { + it("canonicalises postcodes, dedupes, and drops empty selections (absent = all)", () => { + const out = normaliseRunFilters({ + postcodes: ["m20 4tf", "M20 4TF", "b93 8su"], + propertyTypes: [], + builtForms: undefined, + }); + expect(out).toEqual({ + ok: true, + filters: { postcodes: ["B93 8SU", "M20 4TF"] }, + }); + }); + + it("accepts canonical types/built forms plus Unknown, rejects anything else", () => { + const ok = normaliseRunFilters({ + propertyTypes: ["House", "Unknown"], + builtForms: ["Semi-Detached"], + }); + expect(ok).toEqual({ + ok: true, + filters: { propertyTypes: ["House", "Unknown"], builtForms: ["Semi-Detached"] }, + }); + expect(normaliseRunFilters({ propertyTypes: ["Castle"] }).ok).toBe(false); + expect(normaliseRunFilters({ builtForms: ["Terraced"] }).ok).toBe(false); + }); + + it("caps the postcode selection at 40 — 'all postcodes' is absence, not enumeration", () => { + const many = Array.from({ length: 41 }, (_, i) => `M${Math.floor(i / 10)}${i % 10} 4TF`); + const out = normaliseRunFilters({ postcodes: many }); + expect(out.ok).toBe(false); + if (!out.ok) expect(out.error).toMatch(/40/); + expect(normaliseRunFilters({ postcodes: many.slice(0, 40) }).ok).toBe(true); + }); +}); + +describe("buildRunRecord", () => { + it("produces the app-authored task row and config-subtask inputs (ADR-0008)", () => { + const record = buildRunRecord({ + portfolioId: "814", + scenarioIds: ["12", "34"], + filters: { postcodes: ["B93 8SU"], propertyTypes: ["House"] }, + previewedPropertyCount: 214, + triggeredBy: "khalim@domna.homes", + }); + expect(record.task).toEqual({ + taskSource: "Modelling run – 2 scenarios", + service: "modelling_run", + source: "portfolio_id", + sourceId: "814", + status: "waiting", + }); + expect(record.configInputs).toEqual({ + portfolio_id: 814, + scenario_ids: [12, 34], + filters: { postcodes: ["B93 8SU"], property_types: ["House"] }, + previewed_property_count: 214, + triggered_by: "khalim@domna.homes", + }); + }); +}); + +describe("parseRunConfig", () => { + it("round-trips a stored run record back to the app's shape", () => { + const { configInputs } = buildRunRecord({ + portfolioId: "814", + scenarioIds: ["12"], + filters: { builtForms: ["Detached", "Unknown"] }, + previewedPropertyCount: 57, + triggeredBy: "priya@domna.homes", + }); + expect(parseRunConfig(JSON.stringify(configInputs))).toEqual({ + scenarioIds: ["12"], + filters: { builtForms: ["Detached", "Unknown"] }, + previewedPropertyCount: 57, + triggeredBy: "priya@domna.homes", + }); + }); + + it("returns null for missing or malformed inputs — never throws on old rows", () => { + expect(parseRunConfig(null)).toBeNull(); + expect(parseRunConfig("not json")).toBeNull(); + expect(parseRunConfig('{"unrelated":true}')).toBeNull(); + }); +}); + +describe("selectionSummary", () => { + it("describes the filters in the run-history format; no filters = All properties", () => { + expect(selectionSummary({})).toBe("All properties"); + expect( + selectionSummary({ + postcodes: ["B93 8SU", "M20 4TF"], + propertyTypes: ["House"], + builtForms: ["Detached", "Unknown"], + }), + ).toBe("Postcodes: B93 8SU, M20 4TF · Type: House · Built form: Detached, Unknown"); + }); +}); + +describe("deriveRunStatus", () => { + const base = { taskStatus: "waiting", jobCompleted: null, total: 0, completed: 0, failed: 0 }; + it("maps the task lifecycle to the four display states", () => { + // dispatched: accepted, no execution subtasks yet + expect(deriveRunStatus(base)).toEqual({ state: "dispatched" }); + // in progress once execution subtasks exist, with progress + expect( + deriveRunStatus({ ...base, taskStatus: "in progress", total: 104, completed: 61 }), + ).toEqual({ state: "in_progress", ready: 61, total: 104 }); + // complete when the task says so + expect( + deriveRunStatus({ ...base, taskStatus: "completed", jobCompleted: new Date(), total: 104, completed: 104 }), + ).toEqual({ state: "complete" }); + // failed statuses win regardless of counts + expect( + deriveRunStatus({ ...base, taskStatus: "Failed", total: 104, completed: 40, failed: 3 }), + ).toEqual({ state: "failed" }); + }); +}); + +describe("isLargeRun", () => { + it("gates runs of 10,000+ matched properties behind an extra confirm", () => { + expect(isLargeRun(9999)).toBe(false); + expect(isLargeRun(10000)).toBe(true); + }); +}); diff --git a/src/lib/modellingRuns/model.ts b/src/lib/modellingRuns/model.ts new file mode 100644 index 00000000..02cd9e41 --- /dev/null +++ b/src/lib/modellingRuns/model.ts @@ -0,0 +1,198 @@ +/** + * Modelling-run domain model — pure logic for the bulk-trigger-modelling + * journey. See CONTEXT.md (Modelling run, Run filter) and ADR-0008. + */ + +import { normalisePostcode } from "@/lib/postcodeSearch/model"; +import { + BuiltFormTypeValues, + PropertyTypeValues, +} from "@/app/db/schema/landlord_overrides"; + +/** An absent key means unconstrained — "all" is never an enumeration. */ +export interface RunFilters { + postcodes?: string[]; + propertyTypes?: string[]; + builtForms?: string[]; +} + +export type NormaliseFiltersResult = + | { ok: true; filters: RunFilters } + | { ok: false; error: string }; + +/** UX guard, not a transport limit — the config is stored, not carried. */ +export const MAX_FILTER_POSTCODES = 40; + +export function normaliseRunFilters(input: { + postcodes?: string[]; + propertyTypes?: string[]; + builtForms?: string[]; +}): NormaliseFiltersResult { + const filters: RunFilters = {}; + if (input.postcodes?.length) { + const seen = new Set(); + for (const raw of input.postcodes) { + const pc = normalisePostcode(raw); + if (!pc) return { ok: false, error: `Invalid postcode '${raw}'` }; + seen.add(pc); + } + filters.postcodes = [...seen].sort(); + if (filters.postcodes.length > MAX_FILTER_POSTCODES) { + return { + ok: false, + error: `Filters are limited to ${MAX_FILTER_POSTCODES} postcodes per run — leave the filter empty to include all postcodes`, + }; + } + } + // The filter vocabularies are the canonical enum lists (which already end + // in "Unknown" — the no-resolvable-value bucket, see CONTEXT.md Run filter). + for (const [key, allowed] of [ + ["propertyTypes", PropertyTypeValues], + ["builtForms", BuiltFormTypeValues], + ] as const) { + const values = input[key]; + if (!values?.length) continue; + const bad = values.find((v) => !allowed.includes(v)); + if (bad !== undefined) { + return { ok: false, error: `Invalid ${key === "propertyTypes" ? "property type" : "built form"} '${bad}'` }; + } + filters[key] = [...new Set(values)]; + } + return { ok: true, filters }; +} + +export interface RunConfigView { + scenarioIds: string[]; + filters: RunFilters; + previewedPropertyCount: number; + triggeredBy: string; +} + +/** + * Read a stored config-subtask `inputs` back into the app's shape. Null for + * anything that isn't a modelling-run config (malformed, legacy, absent) — + * history reads must never throw on old rows. + */ +export function parseRunConfig(inputs: string | null): RunConfigView | null { + if (!inputs) return null; + try { + const raw = JSON.parse(inputs) as Partial; + if (!Array.isArray(raw.scenario_ids) || typeof raw.triggered_by !== "string") { + return null; + } + const filters: RunFilters = {}; + if (raw.filters?.postcodes?.length) filters.postcodes = raw.filters.postcodes; + if (raw.filters?.property_types?.length) filters.propertyTypes = raw.filters.property_types; + if (raw.filters?.built_forms?.length) filters.builtForms = raw.filters.built_forms; + return { + scenarioIds: raw.scenario_ids.map(String), + filters, + previewedPropertyCount: raw.previewed_property_count ?? 0, + triggeredBy: raw.triggered_by, + }; + } catch { + return null; + } +} + +/** Above this, the UI asks for an explicit extra confirmation before running. */ +export const LARGE_RUN_THRESHOLD = 10_000; + +export function isLargeRun(matchedPropertyCount: number): boolean { + return matchedPropertyCount >= LARGE_RUN_THRESHOLD; +} + +export type RunStatus = + | { state: "dispatched" } + | { state: "in_progress"; ready: number; total: number } + | { state: "complete" } + | { state: "failed" }; + +/** + * Display state of a run from its task + execution-subtask counts (the config + * subtask is excluded by the caller). Status strings follow the task system's + * loose conventions (case-insensitive; completion also signalled by + * job_completed). + */ +export function deriveRunStatus(input: { + taskStatus: string; + jobCompleted: Date | null; + total: number; + completed: number; + failed: number; +}): RunStatus { + const s = input.taskStatus.toLowerCase(); + if (["failed", "failure", "error"].includes(s)) return { state: "failed" }; + if (input.jobCompleted || ["completed", "complete"].includes(s)) { + return { state: "complete" }; + } + if (input.total === 0) return { state: "dispatched" }; + return { state: "in_progress", ready: input.completed, total: input.total }; +} + +/** Human summary of a Run filter, as shown in run history ("All properties"). */ +export function selectionSummary(filters: RunFilters): string { + const bits: string[] = []; + if (filters.postcodes?.length) bits.push(`Postcodes: ${filters.postcodes.join(", ")}`); + if (filters.propertyTypes?.length) bits.push(`Type: ${filters.propertyTypes.join(", ")}`); + if (filters.builtForms?.length) bits.push(`Built form: ${filters.builtForms.join(", ")}`); + return bits.length ? bits.join(" · ") : "All properties"; +} + +export interface RunRequest { + portfolioId: string; + scenarioIds: string[]; + filters: RunFilters; + previewedPropertyCount: number; + triggeredBy: string; +} + +/** Backend-facing (snake_case) copy of a run's configuration. */ +export interface RunConfigInputs { + portfolio_id: number; + scenario_ids: number[]; + filters: { + postcodes?: string[]; + property_types?: string[]; + built_forms?: string[]; + }; + previewed_property_count: number; + triggered_by: string; +} + +/** + * The rows a Modelling run persists (ADR-0008): an app-authored task (the + * BulkUpload trigger convention) plus the config-subtask inputs that make the + * run's request durable and history-readable. + */ +export function buildRunRecord(req: RunRequest): { + task: { + taskSource: string; + service: "modelling_run"; + source: "portfolio_id"; + sourceId: string; + status: "waiting"; + }; + configInputs: RunConfigInputs; +} { + const filters: RunConfigInputs["filters"] = {}; + if (req.filters.postcodes) filters.postcodes = req.filters.postcodes; + if (req.filters.propertyTypes) filters.property_types = req.filters.propertyTypes; + if (req.filters.builtForms) filters.built_forms = req.filters.builtForms; + return { + task: { + taskSource: `Modelling run – ${req.scenarioIds.length} scenario${req.scenarioIds.length === 1 ? "" : "s"}`, + service: "modelling_run", + source: "portfolio_id", + sourceId: req.portfolioId, + status: "waiting", + }, + configInputs: { + portfolio_id: Number(req.portfolioId), + scenario_ids: req.scenarioIds.map(Number), + filters, + previewed_property_count: req.previewedPropertyCount, + triggered_by: req.triggeredBy, + }, + }; +} From 6c479542a0730e60be354ec83ee0db627c7968a2 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 6 Jul 2026 17:08:57 +0000 Subject: [PATCH 05/11] =?UTF-8?q?feat(modelling-runs):=20trigger=20journey?= =?UTF-8?q?=20=E2=80=94=20server=20module,=20routes,=20run=20page?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Builds the bulk-trigger-modelling journey per ADR-0008 on the TDD'd core: - Server module: triggerModellingRun (app-authored task + config subtask, BulkUpload trigger convention; dispatch to /v1/plan/trigger-run; failed dispatch marks the task failed), previewModellingRun (dry-run count proxy to /v1/plan/preview-run), listModellingRuns (one grouped portfolio-scoped query; config parsed back, progress from execution-subtask counts), scenarioIdsWithActiveRuns (third badge state). - Routes: POST/GET /api/portfolio/[id]/modelling-runs, POST .../preview, GET /api/portfolio/[id]/postcodes (grouped distinct + counts). - Page /portfolio/[slug]/modelling/run to the approved mockup in the scenarios-tab design language: scenario picker with the three badges and in-flight warning, three filter columns (postcodes from DB, canonical enums + Unknown), engine-counted summary tray with re-model callouts, large-run confirm (10k gate), done state with ETA, Recent runs table with live progress. TanStack Query v4; no useEffect/useMemo. - "Run modelling" button beside Add properties in PropertyTable. Backend contract (Model team): the distributor at /v1/plan/trigger-run accepts {task_id, portfolio_id, scenario_ids, filters, ...} and attaches execution subtasks to the given task; /v1/plan/preview-run returns {matched_properties, per_scenario[]} through the same resolution code path. Verified live against portfolio 814: postcodes, validation errors, unknown scenarios, dispatch failure marks the task failed and history shows it; simulated backend execution subtasks drove status to {in_progress, ready 2, total 4} and the scenario badge to "Modelling in progress" in the rendered page; portfolio page shows the button. tsc, lint and all 365 vitest tests pass. Co-Authored-By: Claude Fable 5 --- .../modelling-runs/preview/route.ts | 62 ++ .../[portfolioId]/modelling-runs/route.ts | 103 ++++ .../[portfolioId]/postcodes/route.ts | 42 ++ .../[slug]/components/PropertyTable.tsx | 10 + .../modelling/run/RunModellingClient.tsx | 582 ++++++++++++++++++ .../portfolio/[slug]/modelling/run/page.tsx | 38 ++ src/lib/modellingRuns/model.ts | 7 + src/lib/modellingRuns/server.ts | 244 ++++++++ 8 files changed, 1088 insertions(+) create mode 100644 src/app/api/portfolio/[portfolioId]/modelling-runs/preview/route.ts create mode 100644 src/app/api/portfolio/[portfolioId]/modelling-runs/route.ts create mode 100644 src/app/api/portfolio/[portfolioId]/postcodes/route.ts create mode 100644 src/app/portfolio/[slug]/modelling/run/RunModellingClient.tsx create mode 100644 src/app/portfolio/[slug]/modelling/run/page.tsx create mode 100644 src/lib/modellingRuns/server.ts diff --git a/src/app/api/portfolio/[portfolioId]/modelling-runs/preview/route.ts b/src/app/api/portfolio/[portfolioId]/modelling-runs/preview/route.ts new file mode 100644 index 00000000..ee8c768f --- /dev/null +++ b/src/app/api/portfolio/[portfolioId]/modelling-runs/preview/route.ts @@ -0,0 +1,62 @@ +import { NextRequest, NextResponse } from "next/server"; +import { getServerSession } from "next-auth"; +import { AuthOptions } from "@/app/api/auth/[...nextauth]/authOptions"; +import { z } from "zod"; +import { normaliseRunFilters } from "@/lib/modellingRuns/model"; +import { previewModellingRun } from "@/lib/modellingRuns/server"; + +const bodySchema = z.object({ + scenarioIds: z.array(z.string().regex(/^\d+$/)).min(1).max(50), + filters: z + .object({ + postcodes: z.array(z.string()).optional(), + propertyTypes: z.array(z.string()).optional(), + builtForms: z.array(z.string()).optional(), + }) + .default({}), +}); + +// POST — dry-run counts from the backend (same resolution code path as the +// real run; the preview is a promise, not an estimate — ADR-0008). +export async function POST( + request: NextRequest, + props: { params: Promise<{ portfolioId: string }> }, +) { + const session = await getServerSession(AuthOptions); + if (!session?.user?.email) { + return NextResponse.json({ error: "Unauthorised" }, { status: 401 }); + } + const { portfolioId } = await props.params; + if (!/^\d+$/.test(portfolioId)) { + return NextResponse.json({ error: "Invalid portfolio id" }, { status: 400 }); + } + + let body: z.infer; + try { + body = bodySchema.parse(await request.json()); + } catch { + return NextResponse.json({ error: "Invalid body" }, { status: 400 }); + } + const normalised = normaliseRunFilters(body.filters); + if (!normalised.ok) { + return NextResponse.json({ error: normalised.error }, { status: 400 }); + } + + try { + const result = await previewModellingRun({ + portfolioId, + scenarioIds: [...new Set(body.scenarioIds)], + filters: normalised.filters, + sessionToken: + request.cookies.get("__Secure-next-auth.session-token")?.value ?? + request.cookies.get("next-auth.session-token")?.value, + }); + if (!result.ok) { + return NextResponse.json({ error: result.message }, { status: result.status }); + } + return NextResponse.json(result.data); + } catch (err) { + console.error("POST /modelling-runs/preview error:", err); + return NextResponse.json({ error: "Internal server error" }, { status: 500 }); + } +} diff --git a/src/app/api/portfolio/[portfolioId]/modelling-runs/route.ts b/src/app/api/portfolio/[portfolioId]/modelling-runs/route.ts new file mode 100644 index 00000000..b62f8078 --- /dev/null +++ b/src/app/api/portfolio/[portfolioId]/modelling-runs/route.ts @@ -0,0 +1,103 @@ +import { NextRequest, NextResponse } from "next/server"; +import { getServerSession } from "next-auth"; +import { AuthOptions } from "@/app/api/auth/[...nextauth]/authOptions"; +import { z } from "zod"; +import { normaliseRunFilters } from "@/lib/modellingRuns/model"; +import { + listModellingRuns, + triggerModellingRun, +} from "@/lib/modellingRuns/server"; + +const filtersSchema = z.object({ + postcodes: z.array(z.string()).optional(), + propertyTypes: z.array(z.string()).optional(), + builtForms: z.array(z.string()).optional(), +}); + +const bodySchema = z.object({ + scenarioIds: z.array(z.string().regex(/^\d+$/)).min(1).max(50), + filters: filtersSchema.default({}), + previewedPropertyCount: z.number().int().nonnegative(), +}); + +function sessionToken(request: NextRequest): string | undefined { + return ( + request.cookies.get("__Secure-next-auth.session-token")?.value ?? + request.cookies.get("next-auth.session-token")?.value + ); +} + +// POST — record + dispatch a Modelling run (ADR-0008) +export async function POST( + request: NextRequest, + props: { params: Promise<{ portfolioId: string }> }, +) { + const session = await getServerSession(AuthOptions); + if (!session?.user?.email) { + return NextResponse.json({ error: "Unauthorised" }, { status: 401 }); + } + const { portfolioId } = await props.params; + if (!/^\d+$/.test(portfolioId)) { + return NextResponse.json({ error: "Invalid portfolio id" }, { status: 400 }); + } + + let body: z.infer; + try { + body = bodySchema.parse(await request.json()); + } catch { + return NextResponse.json({ error: "Invalid body" }, { status: 400 }); + } + const normalised = normaliseRunFilters(body.filters); + if (!normalised.ok) { + return NextResponse.json({ error: normalised.error }, { status: 400 }); + } + + try { + const outcome = await triggerModellingRun({ + portfolioId, + scenarioIds: [...new Set(body.scenarioIds)], + filters: normalised.filters, + previewedPropertyCount: body.previewedPropertyCount, + triggeredBy: session.user.email, + sessionToken: sessionToken(request), + }); + switch (outcome.kind) { + case "ok": + return NextResponse.json({ taskId: outcome.taskId }, { status: 202 }); + case "unknown_scenarios": + return NextResponse.json( + { error: `Unknown scenarios for this portfolio: ${outcome.missing.join(", ")}` }, + { status: 400 }, + ); + case "dispatch_failed": + return NextResponse.json( + { error: "Couldn't start modelling — please try again" }, + { status: outcome.status }, + ); + } + } catch (err) { + console.error("POST /modelling-runs error:", err); + return NextResponse.json({ error: "Internal server error" }, { status: 500 }); + } +} + +// GET — run history for the portfolio +export async function GET( + _request: NextRequest, + props: { params: Promise<{ portfolioId: string }> }, +) { + const session = await getServerSession(AuthOptions); + if (!session?.user?.email) { + return NextResponse.json({ error: "Unauthorised" }, { status: 401 }); + } + const { portfolioId } = await props.params; + if (!/^\d+$/.test(portfolioId)) { + return NextResponse.json({ error: "Invalid portfolio id" }, { status: 400 }); + } + try { + return NextResponse.json({ runs: await listModellingRuns(portfolioId) }); + } catch (err) { + console.error("GET /modelling-runs error:", err); + return NextResponse.json({ error: "Internal server error" }, { status: 500 }); + } +} diff --git a/src/app/api/portfolio/[portfolioId]/postcodes/route.ts b/src/app/api/portfolio/[portfolioId]/postcodes/route.ts new file mode 100644 index 00000000..0801470b --- /dev/null +++ b/src/app/api/portfolio/[portfolioId]/postcodes/route.ts @@ -0,0 +1,42 @@ +import { NextRequest, NextResponse } from "next/server"; +import { getServerSession } from "next-auth"; +import { AuthOptions } from "@/app/api/auth/[...nextauth]/authOptions"; +import { db } from "@/app/db/db"; +import { property } from "@/app/db/schema/property"; +import { and, asc, count, eq, isNotNull } from "drizzle-orm"; + +// GET — distinct postcodes (with counts) for the Run-filter options. One +// grouped, portfolio-scoped query (ix_property_portfolio). +export async function GET( + _request: NextRequest, + props: { params: Promise<{ portfolioId: string }> }, +) { + const session = await getServerSession(AuthOptions); + if (!session?.user?.email) { + return NextResponse.json({ error: "Unauthorised" }, { status: 401 }); + } + const { portfolioId } = await props.params; + if (!/^\d+$/.test(portfolioId)) { + return NextResponse.json({ error: "Invalid portfolio id" }, { status: 400 }); + } + try { + const rows = await db + .select({ postcode: property.postcode, n: count() }) + .from(property) + .where( + and( + eq(property.portfolioId, BigInt(portfolioId)), + eq(property.markedForDeletion, false), + isNotNull(property.postcode), + ), + ) + .groupBy(property.postcode) + .orderBy(asc(property.postcode)); + return NextResponse.json({ + postcodes: rows.map((r) => ({ postcode: r.postcode, count: r.n })), + }); + } catch (err) { + console.error("GET /postcodes error:", err); + return NextResponse.json({ error: "Internal server error" }, { status: 500 }); + } +} diff --git a/src/app/portfolio/[slug]/components/PropertyTable.tsx b/src/app/portfolio/[slug]/components/PropertyTable.tsx index 8fd7aed1..21f644b9 100644 --- a/src/app/portfolio/[slug]/components/PropertyTable.tsx +++ b/src/app/portfolio/[slug]/components/PropertyTable.tsx @@ -14,6 +14,7 @@ import { PlusIcon, MagnifyingGlassIcon, DocumentArrowUpIcon, + PlayIcon, } from "@heroicons/react/24/outline"; import { useRouter } from "next/navigation"; import { HomeIcon } from "@heroicons/react/24/outline"; @@ -672,6 +673,15 @@ export default function PropertyTable({ + + {/* Run modelling */} + diff --git a/src/app/portfolio/[slug]/modelling/run/RunModellingClient.tsx b/src/app/portfolio/[slug]/modelling/run/RunModellingClient.tsx new file mode 100644 index 00000000..34202925 --- /dev/null +++ b/src/app/portfolio/[slug]/modelling/run/RunModellingClient.tsx @@ -0,0 +1,582 @@ +"use client"; + +import { useState } from "react"; +import Link from "next/link"; +import { useRouter } from "next/navigation"; +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; +import { PlayIcon } from "@heroicons/react/24/solid"; +import { + RUN_FILTER_BUILT_FORMS, + RUN_FILTER_PROPERTY_TYPES, + RunFilters, + RunStatus, + isLargeRun, + MAX_FILTER_POSTCODES, +} from "@/lib/modellingRuns/model"; + +export interface ScenarioOption { + id: string; + name: string; + goal: string; + goalValue: string | null; + housingType: string; + budget: number | null; + status: "modelled" | "awaiting" | "running"; + modelledCount: number; +} + +interface PreviewResponse { + matchedProperties: number; + perScenario: { scenarioId: string; alreadyModelled: number }[]; +} + +interface RunHistoryRow { + taskId: string; + triggeredAt: string; + triggeredBy: string; + scenarios: { id: string; name: string | null }[]; + selection: string; + previewedPropertyCount: number; + planCount: number; + status: RunStatus; +} + +const STATUS_LABELS: Record = { + dispatched: "Dispatched", + in_progress: "In progress", + complete: "Complete", + failed: "Failed", +}; + +function RunBadge({ status }: { status: ScenarioOption["status"] }) { + if (status === "running") { + return ( + + + Modelling in progress + + ); + } + const modelled = status === "modelled"; + return ( + + + {modelled ? "Modelled" : "Awaiting modelling"} + + ); +} + +function FilterColumn({ + title, + options, + selected, + onToggle, +}: { + title: string; + options: { value: string; count?: number }[]; + selected: Set; + onToggle: (value: string) => void; +}) { + return ( +
+
+ + {title} + + + · {selected.size ? `${selected.size} selected` : "all"} + +
+
+ {options.map((o) => { + const on = selected.has(o.value); + return ( + + ); + })} +
+
+ ); +} + +export default function RunModellingClient({ + portfolioId, + portfolioName, + scenarios, +}: { + portfolioId: string; + portfolioName: string; + scenarios: ScenarioOption[]; +}) { + const router = useRouter(); + const queryClient = useQueryClient(); + const [selScenarios, setSelScenarios] = useState>(new Set()); + const [selPcs, setSelPcs] = useState>(new Set()); + const [selTypes, setSelTypes] = useState>(new Set()); + const [selForms, setSelForms] = useState>(new Set()); + const [confirmOpen, setConfirmOpen] = useState(false); + const [done, setDone] = useState<{ properties: number; scenarios: number } | null>(null); + + const filters: RunFilters = { + ...(selPcs.size ? { postcodes: [...selPcs].sort() } : {}), + ...(selTypes.size ? { propertyTypes: [...selTypes] } : {}), + ...(selForms.size ? { builtForms: [...selForms] } : {}), + }; + const scenarioIds = [...selScenarios].sort(); + + const postcodesQuery = useQuery<{ postcode: string; count: number }[], Error>({ + queryKey: ["portfolio-postcodes", portfolioId], + staleTime: 5 * 60 * 1000, + queryFn: async () => { + const res = await fetch(`/api/portfolio/${portfolioId}/postcodes`); + const body = await res.json(); + if (!res.ok) throw new Error(body.error ?? "Couldn't load postcodes"); + return body.postcodes; + }, + }); + + const preview = useQuery({ + queryKey: ["run-preview", portfolioId, scenarioIds, filters], + enabled: scenarioIds.length > 0 && !done, + keepPreviousData: true, + retry: false, + queryFn: async () => { + const res = await fetch(`/api/portfolio/${portfolioId}/modelling-runs/preview`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ scenarioIds, filters }), + }); + const body = await res.json(); + if (!res.ok) throw new Error(body.error ?? "Couldn't count the selection"); + return body; + }, + }); + + const history = useQuery({ + queryKey: ["modelling-runs", portfolioId], + refetchInterval: 30_000, + queryFn: async () => { + const res = await fetch(`/api/portfolio/${portfolioId}/modelling-runs`); + const body = await res.json(); + if (!res.ok) throw new Error(body.error ?? "Couldn't load run history"); + return body.runs; + }, + }); + + const run = useMutation({ + mutationFn: async () => { + const res = await fetch(`/api/portfolio/${portfolioId}/modelling-runs`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + scenarioIds, + filters, + previewedPropertyCount: preview.data?.matchedProperties ?? 0, + }), + }); + const body = await res.json(); + if (!res.ok) throw new Error(body.error ?? "Couldn't start modelling — please try again"); + return body; + }, + onSuccess: () => { + setDone({ properties: preview.data?.matchedProperties ?? 0, scenarios: scenarioIds.length }); + setSelScenarios(new Set()); + queryClient.invalidateQueries(["modelling-runs", portfolioId]); + }, + }); + + const toggle = (set: Set, setter: (s: Set) => void, value: string) => { + const next = new Set(set); + if (next.has(value)) next.delete(value); + else next.add(value); + setter(next); + }; + + const matched = preview.data?.matchedProperties; + const planTotal = matched != null ? matched * scenarioIds.length : null; + const remodel = (preview.data?.perScenario ?? []).filter((p) => p.alreadyModelled > 0); + const scenarioById = new Map(scenarios.map((s) => [s.id, s])); + const startRun = () => { + if (matched != null && isLargeRun(matched)) setConfirmOpen(true); + else run.mutate(); + }; + const eta = planTotal ? Math.max(5, Math.round(planTotal / 300) * 5) : 5; + + return ( +
+
+ + {portfolioName} / Portfolio + {" "} + / Run modelling +
+

+ Run your scenarios. +

+

+ Choose the scenarios you want answered, then the homes to include. The counts you see are + checked against your portfolio as you go — what you see is exactly what gets modelled. +

+ + {done ? ( +
+
+ +
+

Modelling started

+

+ We're modelling {done.properties.toLocaleString()}{" "} + {done.properties === 1 ? "property" : "properties"} against {done.scenarios}{" "} + scenario{done.scenarios > 1 ? "s" : ""} —{" "} + {(done.properties * done.scenarios).toLocaleString()} plan + {done.properties * done.scenarios === 1 ? "" : "s"} in total. Results for a run this + size are typically ready within ~{eta} minutes. You can check on this run any + time under Recent runs. +

+
+ + +
+
+ ) : ( + <> + {/* Step 1 — scenarios */} +
+ + Step 1 · Scenarios + +

+ Which questions are we asking? +

+

+ Every selected scenario runs against the same property selection. +

+ {scenarios.length === 0 ? ( +
+ No scenarios yet —{" "} + + create your first scenario + {" "} + and come back to run it. +
+ ) : ( + scenarios.map((s) => { + const on = selScenarios.has(s.id); + return ( +
+ + {on && s.status === "running" && ( +
+ This scenario is already being modelled. You can still include it — the + newest results are the ones you'll see. +
+ )} +
+ ); + }) + )} +
+ + {/* Step 2 — properties */} +
+ + Step 2 · Properties + +

+ Which homes? +

+

+ Leave a filter untouched to include everything. What you've told us about a home + takes priority over its EPC; “Unknown” picks out homes where we have + neither. +

+
+ ({ + value: p.postcode, + count: p.count, + }))} + selected={selPcs} + onToggle={(v) => toggle(selPcs, setSelPcs, v)} + /> + ({ value }))} + selected={selTypes} + onToggle={(v) => toggle(selTypes, setSelTypes, v)} + /> + ({ value }))} + selected={selForms} + onToggle={(v) => toggle(selForms, setSelForms, v)} + /> +
+

+ Filters are limited to {MAX_FILTER_POSTCODES} postcodes per run. Tags are coming — + you'll be able to save a selection and reuse it. +

+
+ + {/* Recent runs */} +
+
+

Recent runs

+ + what was asked, by whom, and how it went + +
+ {history.isLoading ? ( +

Loading…

+ ) : !history.data?.length ? ( +

+ No runs yet — your first appears here the moment you trigger it. +

+ ) : ( +
+ + + + {["When", "Scenarios", "Selection", "Properties", "Plans", "Status"].map( + (h) => ( + + ), + )} + + + + {history.data.map((r) => ( + + + + + + + + + ))} + +
+ {h} +
+ {new Date(r.triggeredAt).toLocaleString("en-GB", { + day: "numeric", + month: "short", + hour: "2-digit", + minute: "2-digit", + })} + {r.triggeredBy} + + {r.scenarios.map((s) => s.name ?? s.id).join(" · ") || "—"} + + {r.selection} + + {r.previewedPropertyCount.toLocaleString()} + + {r.status.state === "in_progress" ? ( + <> + + {r.status.ready.toLocaleString()} of {r.status.total.toLocaleString()} ready + + + + + + ) : ( + r.planCount.toLocaleString() + )} + + + {STATUS_LABELS[r.status.state]} + +
+
+ )} +
+ + )} + + {/* Summary tray */} + {!done && scenarioIds.length > 0 && ( +
+
+ + {matched != null ? matched.toLocaleString() : "—"} + properties + + × + + {scenarioIds.length} + + scenario{scenarioIds.length > 1 ? "s" : ""} + + + = + + {planTotal != null ? planTotal.toLocaleString() : "—"} + plans + + + + {preview.isFetching + ? "Counting…" + : preview.isError + ? preview.error.message + : "Checked against your portfolio just now"} + + +
+ {(remodel.length > 0 || run.isError) && ( +
+ {remodel.map((p) => ( + + {p.alreadyModelled.toLocaleString()} of{" "} + {matched?.toLocaleString() ?? "—"} have already been modelled in “ + {scenarioById.get(p.scenarioId)?.name ?? p.scenarioId}” — they'll be + modelled again and the newest results shown + + ))} + {run.isError && {run.error.message}} +
+ )} +
+ )} + + {/* Large-run confirm */} + {confirmOpen && ( +
+
+

+ This is a large run +

+

+ You're about to model {matched?.toLocaleString()} homes across{" "} + {scenarioIds.length} scenario{scenarioIds.length > 1 ? "s" : ""} —{" "} + {planTotal?.toLocaleString()} plans. A run this size can't be stopped + once it starts. +

+
+ + +
+
+
+ )} +
+ ); +} diff --git a/src/app/portfolio/[slug]/modelling/run/page.tsx b/src/app/portfolio/[slug]/modelling/run/page.tsx new file mode 100644 index 00000000..9a686cb9 --- /dev/null +++ b/src/app/portfolio/[slug]/modelling/run/page.tsx @@ -0,0 +1,38 @@ +import { getPortfolio } from "../../utils"; +import { listScenariosWithStatus } from "@/lib/scenarios/queries"; +import { scenarioIdsWithActiveRuns } from "@/lib/modellingRuns/server"; +import RunModellingClient, { ScenarioOption } from "./RunModellingClient"; + +export const dynamic = "force-dynamic"; + +export default async function Page(props: { params: Promise<{ slug: string }> }) { + const { slug } = await props.params; + const [{ name }, scenarios, activeIds] = await Promise.all([ + getPortfolio(slug), + listScenariosWithStatus(BigInt(slug)), + scenarioIdsWithActiveRuns(slug), + ]); + + const options: ScenarioOption[] = scenarios.map((s) => ({ + id: s.id.toString(), + name: s.name ?? `Scenario ${s.id}`, + goal: s.goal, + goalValue: s.goalValue, + housingType: s.housingType, + budget: s.budget, + status: activeIds.has(s.id.toString()) + ? "running" + : s.status === "modelled" + ? "modelled" + : "awaiting", + modelledCount: s.plansCount, + })); + + return ( + + ); +} diff --git a/src/lib/modellingRuns/model.ts b/src/lib/modellingRuns/model.ts index 02cd9e41..c00fe73a 100644 --- a/src/lib/modellingRuns/model.ts +++ b/src/lib/modellingRuns/model.ts @@ -9,6 +9,13 @@ import { PropertyTypeValues, } from "@/app/db/schema/landlord_overrides"; +/** + * The Run-filter vocabularies: the canonical enum lists, which already end in + * "Unknown" — the no-resolvable-value bucket (CONTEXT.md, Run filter). + */ +export const RUN_FILTER_PROPERTY_TYPES: readonly string[] = PropertyTypeValues; +export const RUN_FILTER_BUILT_FORMS: readonly string[] = BuiltFormTypeValues; + /** An absent key means unconstrained — "all" is never an enumeration. */ export interface RunFilters { postcodes?: string[]; diff --git a/src/lib/modellingRuns/server.ts b/src/lib/modellingRuns/server.ts new file mode 100644 index 00000000..b20a1940 --- /dev/null +++ b/src/lib/modellingRuns/server.ts @@ -0,0 +1,244 @@ +import { db } from "@/app/db/db"; +import { tasks } from "@/app/db/schema/tasks/tasks"; +import { subTasks } from "@/app/db/schema/tasks/subtask"; +import { scenario } from "@/app/db/schema/recommendations"; +import { and, desc, eq, inArray, sql } from "drizzle-orm"; +import { + RunFilters, + RunStatus, + buildRunRecord, + deriveRunStatus, + parseRunConfig, + selectionSummary, +} from "./model"; + +// The config subtask that makes a run's request durable (ADR-0008); execution +// subtasks attached by the distributor carry other service values. +export const RUN_CONFIG_SERVICE = "modelling_run_config"; + +// Distributor entry point on the Model backend. It fans the run out to +// workers, attaching execution subtasks to the task_id we pass (ADR-0008). +const DISTRIBUTOR_ENDPOINT = "/v1/plan/trigger-run"; +// Dry-run sibling: same filter resolution, counts only. +const PREVIEW_ENDPOINT = "/v1/plan/preview-run"; + +type BackendResult = { ok: true; data: T } | { ok: false; status: number; message: string }; + +async function callBackend( + endpoint: string, + payload: Record, + sessionToken: string | undefined, +): Promise> { + const url = process.env.FASTAPI_API_URL; + const key = process.env.FASTAPI_API_KEY; + if (!url || !key) { + console.error("FASTAPI_API_URL or FASTAPI_API_KEY not set"); + return { ok: false, status: 500, message: "Server misconfiguration" }; + } + try { + const res = await fetch(`${url}${endpoint}`, { + method: "POST", + headers: { + "Content-Type": "application/json", + "x-api-key": key, + Authorization: `Bearer ${sessionToken}`, + }, + body: JSON.stringify(payload), + }); + if (!res.ok) { + const errText = await res.text().catch(() => ""); + console.error(`FastAPI ${endpoint} failed:`, res.status, errText); + return { ok: false, status: 502, message: "Modelling backend request failed" }; + } + return { ok: true, data: (await res.json()) as T }; + } catch (err) { + console.error(`Failed to reach FastAPI ${endpoint}:`, err); + return { ok: false, status: 502, message: "Modelling backend request failed" }; + } +} + +export type TriggerRunOutcome = + | { kind: "ok"; taskId: string } + | { kind: "unknown_scenarios"; missing: string[] } + | { kind: "dispatch_failed"; status: number; message: string }; + +/** + * Record + dispatch a Modelling run: app-authored task and config subtask + * (BulkUpload trigger convention), then hand task_id to the distributor. A + * failed dispatch marks the task failed so history never shows a ghost run + * as pending (ADR-0008). + */ +export async function triggerModellingRun(args: { + portfolioId: string; + scenarioIds: string[]; + filters: RunFilters; + previewedPropertyCount: number; + triggeredBy: string; + sessionToken: string | undefined; +}): Promise { + const known = await db + .select({ id: scenario.id }) + .from(scenario) + .where( + and( + eq(scenario.portfolioId, BigInt(args.portfolioId)), + inArray(scenario.id, args.scenarioIds.map(BigInt)), + ), + ); + const knownIds = new Set(known.map((r) => r.id.toString())); + const missing = args.scenarioIds.filter((id) => !knownIds.has(id)); + if (missing.length > 0) return { kind: "unknown_scenarios", missing }; + + const record = buildRunRecord(args); + const now = new Date(); + const [task] = await db + .insert(tasks) + .values({ ...record.task, jobStarted: now }) + .returning(); + await db.insert(subTasks).values({ + taskId: task.id, + status: "waiting", + service: RUN_CONFIG_SERVICE, + inputs: JSON.stringify(record.configInputs), + }); + + const dispatch = await callBackend( + DISTRIBUTOR_ENDPOINT, + { task_id: task.id, ...record.configInputs }, + args.sessionToken, + ); + if (!dispatch.ok) { + await db.update(tasks).set({ status: "failed" }).where(eq(tasks.id, task.id)); + return { kind: "dispatch_failed", status: dispatch.status, message: dispatch.message }; + } + + await db.update(tasks).set({ status: "in progress" }).where(eq(tasks.id, task.id)); + return { kind: "ok", taskId: task.id }; +} + +export interface PreviewCounts { + matchedProperties: number; + perScenario: { scenarioId: string; alreadyModelled: number }[]; +} + +/** Proxy the backend dry-run: same resolution code path as the real run. */ +export async function previewModellingRun(args: { + portfolioId: string; + scenarioIds: string[]; + filters: RunFilters; + sessionToken: string | undefined; +}): Promise> { + const { configInputs } = buildRunRecord({ + ...args, + previewedPropertyCount: 0, + triggeredBy: "preview", + }); + const res = await callBackend<{ + matched_properties: number; + per_scenario?: { scenario_id: number; already_modelled: number }[]; + }>( + PREVIEW_ENDPOINT, + { + portfolio_id: configInputs.portfolio_id, + scenario_ids: configInputs.scenario_ids, + filters: configInputs.filters, + }, + args.sessionToken, + ); + if (!res.ok) return res; + return { + ok: true, + data: { + matchedProperties: res.data.matched_properties, + perScenario: (res.data.per_scenario ?? []).map((p) => ({ + scenarioId: String(p.scenario_id), + alreadyModelled: p.already_modelled, + })), + }, + }; +} + +export interface RunHistoryRow { + taskId: string; + triggeredAt: string; + triggeredBy: string; + scenarios: { id: string; name: string | null }[]; + selection: string; + previewedPropertyCount: number; + planCount: number; + status: RunStatus; +} + +/** + * Run history for a portfolio: its modelling_run tasks with app-authored + * config parsed back out and progress from execution-subtask counts. One + * grouped query — portfolio-scoped, never per-run probes. + */ +export async function listModellingRuns( + portfolioId: string, + limit = 20, +): Promise { + const rows = await db + .select({ + taskId: tasks.id, + jobStarted: tasks.jobStarted, + jobCompleted: tasks.jobCompleted, + status: tasks.status, + configInputs: sql`max(${subTasks.inputs}) filter (where ${subTasks.service} = ${RUN_CONFIG_SERVICE})`, + total: sql`count(${subTasks.id}) filter (where ${subTasks.service} is distinct from ${RUN_CONFIG_SERVICE})::int`, + completed: sql`count(case when lower(${subTasks.status}) in ('completed','complete') and ${subTasks.service} is distinct from ${RUN_CONFIG_SERVICE} then 1 end)::int`, + failed: sql`count(case when lower(${subTasks.status}) in ('failed','failure','error') and ${subTasks.service} is distinct from ${RUN_CONFIG_SERVICE} then 1 end)::int`, + }) + .from(tasks) + .leftJoin(subTasks, eq(subTasks.taskId, tasks.id)) + .where(and(eq(tasks.service, "modelling_run"), eq(tasks.sourceId, portfolioId))) + .groupBy(tasks.id) + .orderBy(desc(tasks.jobStarted)) + .limit(limit); + + const configs = rows.map((r) => parseRunConfig(r.configInputs)); + const scenarioIds = [...new Set(configs.flatMap((c) => c?.scenarioIds ?? []))]; + const names = scenarioIds.length + ? await db + .select({ id: scenario.id, name: scenario.name }) + .from(scenario) + .where(inArray(scenario.id, scenarioIds.map(BigInt))) + : []; + const nameById = new Map(names.map((n) => [n.id.toString(), n.name])); + + return rows.map((r, i) => { + const config = configs[i]; + const scenarios = (config?.scenarioIds ?? []).map((id) => ({ + id, + name: nameById.get(id) ?? null, + })); + return { + taskId: r.taskId, + triggeredAt: (r.jobStarted ?? new Date(0)).toISOString(), + triggeredBy: config?.triggeredBy ?? "unknown", + scenarios, + selection: config ? selectionSummary(config.filters) : "—", + previewedPropertyCount: config?.previewedPropertyCount ?? 0, + planCount: (config?.previewedPropertyCount ?? 0) * scenarios.length, + status: deriveRunStatus({ + taskStatus: r.status, + jobCompleted: r.jobCompleted, + total: r.total, + completed: r.completed, + failed: r.failed, + }), + }; + }); +} + +/** Scenario ids with a run currently in flight — drives the third badge state. */ +export async function scenarioIdsWithActiveRuns(portfolioId: string): Promise> { + const runs = await listModellingRuns(portfolioId, 50); + const active = new Set(); + for (const run of runs) { + if (run.status.state === "dispatched" || run.status.state === "in_progress") { + run.scenarios.forEach((s) => active.add(s.id)); + } + } + return active; +} From 285d8a38a9450c657dc02795845ccab3024c3710 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 6 Jul 2026 17:33:41 +0000 Subject: [PATCH 06/11] feat(modelling-runs): compute preview counts in-app, drop preview-run ask previewModellingRun now resolves Run filters in one grouped SQL statement (override snapshot -> EPC-derived via propertyTypeSql/new builtFormTypeSql -> Unknown) and intersects the matched set with plan for per-scenario already-modelled counts. The Model-backend ask shrinks to the distributor alone; the app's preview is the reference implementation of the resolution rule (ADR-0008 amended). Verified live against 814: all=340 matched / 338 already modelled in scenario 1271; B93 8SU=2 resolving House/Detached through override snapshots; Bungalow/Unknown=0. Co-Authored-By: Claude Fable 5 --- ...8-modelling-runs-filters-to-distributor.md | 21 ++-- .../modelling-runs/preview/route.ts | 8 +- src/lib/modellingRuns/server.ts | 95 +++++++++++++------ src/lib/services/epcSources.ts | 12 ++- 4 files changed, 89 insertions(+), 47 deletions(-) diff --git a/docs/adr/0008-modelling-runs-filters-to-distributor.md b/docs/adr/0008-modelling-runs-filters-to-distributor.md index dab4e11e..0fb7785b 100644 --- a/docs/adr/0008-modelling-runs-filters-to-distributor.md +++ b/docs/adr/0008-modelling-runs-filters-to-distributor.md @@ -44,11 +44,15 @@ explicit marker when trigger wiring lands. This is that wiring. properties.** Resolution precedence: landlord-override fact → EPC-derived value (new-approach EPC graph, lodged over predicted, or legacy property-row columns) → **Unknown**. The app never re-implements this rule. -- **Preview counts come from a backend dry-run/count endpoint** taking the - same filters through the same resolution code path as the real run — the - preview is a promise, not an estimate. The response includes per-scenario - already-modelled counts so the UI can flag re-modelling ("120 of 214 will - be re-modelled"). Plan arithmetic is exact: one plan per +- **Preview counts are computed in-app** (amended 2026-07-06; originally a + backend dry-run endpoint): one grouped SQL statement resolves the filters — + override snapshot → EPC-derived (`propertyTypeSql`/`builtFormTypeSql` in + epcSources: new-approach graph, lodged over predicted, RdSAP codes mapped; + legacy property-row columns) → Unknown — and intersects the matched set + with `plan` for per-scenario already-modelled counts ("120 of 214 will be + re-modelled"). Both the app and the distributor read this database and + implement this documented rule; divergence is a bug in whichever side moved + without the other. Plan arithmetic is exact: one plan per (scenario, property) per run. - **Filter options are resolution-free in the app**: the postcode multi-select is one `SELECT DISTINCT postcode` per portfolio; type/built-form options are @@ -89,10 +93,11 @@ explicit marker when trigger wiring lands. This is that wiring. ## Consequences -- Two backend asks must land with the distributor: accept an app-created +- One backend ask must land with the distributor: accept an app-created `task_id` and attach execution subtasks to it (rather than creating its own - task, as the current `plan_engine` entrypoint does), and expose the - dry-run/count endpoint sharing the resolution code path. + task, as the current `plan_engine` entrypoint does). Its filter resolution + must match the rule above — the app's preview is the reference + implementation. - The preview's already-modelled callout and matched counts are only as fresh as trigger time; a property added between preview and trigger is picked up by the run (filters re-resolve) — the run row records the previewed count so diff --git a/src/app/api/portfolio/[portfolioId]/modelling-runs/preview/route.ts b/src/app/api/portfolio/[portfolioId]/modelling-runs/preview/route.ts index ee8c768f..4817d91c 100644 --- a/src/app/api/portfolio/[portfolioId]/modelling-runs/preview/route.ts +++ b/src/app/api/portfolio/[portfolioId]/modelling-runs/preview/route.ts @@ -47,14 +47,8 @@ export async function POST( portfolioId, scenarioIds: [...new Set(body.scenarioIds)], filters: normalised.filters, - sessionToken: - request.cookies.get("__Secure-next-auth.session-token")?.value ?? - request.cookies.get("next-auth.session-token")?.value, }); - if (!result.ok) { - return NextResponse.json({ error: result.message }, { status: result.status }); - } - return NextResponse.json(result.data); + return NextResponse.json(result); } catch (err) { console.error("POST /modelling-runs/preview error:", err); return NextResponse.json({ error: "Internal server error" }, { status: 500 }); diff --git a/src/lib/modellingRuns/server.ts b/src/lib/modellingRuns/server.ts index b20a1940..62a74337 100644 --- a/src/lib/modellingRuns/server.ts +++ b/src/lib/modellingRuns/server.ts @@ -11,6 +11,7 @@ import { parseRunConfig, selectionSummary, } from "./model"; +import { builtFormTypeSql, propertyTypeSql } from "@/lib/services/epcSources"; // The config subtask that makes a run's request durable (ADR-0008); execution // subtasks attached by the distributor carry other service values. @@ -19,8 +20,6 @@ export const RUN_CONFIG_SERVICE = "modelling_run_config"; // Distributor entry point on the Model backend. It fans the run out to // workers, attaching execution subtasks to the task_id we pass (ADR-0008). const DISTRIBUTOR_ENDPOINT = "/v1/plan/trigger-run"; -// Dry-run sibling: same filter resolution, counts only. -const PREVIEW_ENDPOINT = "/v1/plan/preview-run"; type BackendResult = { ok: true; data: T } | { ok: false; status: number; message: string }; @@ -121,40 +120,74 @@ export interface PreviewCounts { perScenario: { scenarioId: string; alreadyModelled: number }[]; } -/** Proxy the backend dry-run: same resolution code path as the real run. */ +/** + * Count the properties a Run filter matches, plus how many of them already + * hold plans per selected scenario. Resolution precedence per ADR-0008: + * landlord-override snapshot → EPC-derived (new-approach graph, lodged over + * predicted, or legacy property-row columns) → Unknown. The distributor + * implements the same documented rule; both read this database. + * One grouped, portfolio-scoped statement — no per-scenario probes. + */ export async function previewModellingRun(args: { portfolioId: string; scenarioIds: string[]; filters: RunFilters; - sessionToken: string | undefined; -}): Promise> { - const { configInputs } = buildRunRecord({ - ...args, - previewedPropertyCount: 0, - triggeredBy: "preview", - }); - const res = await callBackend<{ - matched_properties: number; - per_scenario?: { scenario_id: number; already_modelled: number }[]; - }>( - PREVIEW_ENDPOINT, - { - portfolio_id: configInputs.portfolio_id, - scenario_ids: configInputs.scenario_ids, - filters: configInputs.filters, - }, - args.sessionToken, - ); - if (!res.ok) return res; +}): Promise { + const pid = BigInt(args.portfolioId); + const inList = (values: (string | number)[]) => + sql.join(values.map((v) => sql`${v}`), sql`, `); + const conditions = [sql`TRUE`]; + if (args.filters.postcodes) { + conditions.push(sql`resolved.postcode IN (${inList(args.filters.postcodes)})`); + } + if (args.filters.propertyTypes) { + conditions.push(sql`resolved.ptype IN (${inList(args.filters.propertyTypes)})`); + } + if (args.filters.builtForms) { + conditions.push(sql`resolved.bform IN (${inList(args.filters.builtForms)})`); + } + + const result = await db.execute<{ + matched: number; + per_scenario: { scenario_id: string; n: number }[] | null; + }>(sql` + WITH resolved AS ( + SELECT p.id, p.postcode, + COALESCE(pot.override_value, ${propertyTypeSql}, 'Unknown') AS ptype, + COALESCE(pobf.override_value, ${builtFormTypeSql}, 'Unknown') AS bform + FROM property p + LEFT JOIN property_overrides pot ON pot.property_id = p.id + AND pot.override_component = 'property_type' AND pot.building_part = 0 + LEFT JOIN property_overrides pobf ON pobf.property_id = p.id + AND pobf.override_component = 'built_form_type' AND pobf.building_part = 0 + LEFT JOIN epc_property epl ON epl.property_id = p.id AND epl.source = 'lodged' + LEFT JOIN epc_property epp ON epp.property_id = p.id AND epp.source = 'predicted' + WHERE p.portfolio_id = ${pid} AND p.marked_for_deletion = false + ), + matched AS ( + SELECT resolved.id FROM resolved + WHERE ${sql.join(conditions, sql` AND `)} + ), + already AS ( + SELECT scenario_id, COUNT(DISTINCT property_id)::int AS n + FROM plan + WHERE portfolio_id = ${pid} + AND scenario_id IN (${inList(args.scenarioIds.map(Number))}) + AND property_id IN (SELECT id FROM matched) + GROUP BY scenario_id + ) + SELECT + (SELECT COUNT(*)::int FROM matched) AS matched, + (SELECT json_agg(json_build_object('scenario_id', scenario_id::text, 'n', n)) FROM already) AS per_scenario + `); + + const row = result.rows[0]; return { - ok: true, - data: { - matchedProperties: res.data.matched_properties, - perScenario: (res.data.per_scenario ?? []).map((p) => ({ - scenarioId: String(p.scenario_id), - alreadyModelled: p.already_modelled, - })), - }, + matchedProperties: row?.matched ?? 0, + perScenario: (row?.per_scenario ?? []).map((p) => ({ + scenarioId: p.scenario_id, + alreadyModelled: p.n, + })), }; } diff --git a/src/lib/services/epcSources.ts b/src/lib/services/epcSources.ts index 8dd28206..3a75bfd9 100644 --- a/src/lib/services/epcSources.ts +++ b/src/lib/services/epcSources.ts @@ -211,7 +211,7 @@ export const PROPERTY_TYPE_LABELS: Record = { "3": "Maisonette", "4": "Park home", }; -const BUILT_FORM_LABELS: Record = { +export const BUILT_FORM_LABELS: Record = { "1": "Detached", "2": "Semi-Detached", "3": "End-Terrace", @@ -273,6 +273,16 @@ export const propertyTypeSql = sql`CASE ELSE p.property_type END`; +/** + * Built-form label. New: epc_property (lodged, else predicted) — RdSAP codes + * mapped via BUILT_FORM_LABELS, text passed through, mirroring + * resolvePropertyDescriptors. Legacy: property.built_form. + */ +export const builtFormTypeSql = sql`CASE + WHEN ${isNewApproachSql} THEN (CASE WHEN epl.id IS NOT NULL THEN ${codeLabelCaseSql(sql`epl.built_form`, BUILT_FORM_LABELS)} ELSE ${codeLabelCaseSql(sql`epp.built_form`, BUILT_FORM_LABELS)} END) + ELSE p.built_form +END`; + /** * The main building part's construction age band for one epc_property alias, as * a correlated scalar subquery — properties can also have extension parts, so a From 10888e3fff9d7391ae0f485852ee30de2d913912 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 6 Jul 2026 17:40:09 +0000 Subject: [PATCH 07/11] refactor(modelling-runs): minimal dispatch payload MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The distributor gets {task_id, portfolio_id, scenario_ids, filters} only — previewed count and triggered_by are app provenance, already durable on the config subtask keyed by the same task_id. Co-Authored-By: Claude Fable 5 --- src/lib/modellingRuns/server.ts | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/lib/modellingRuns/server.ts b/src/lib/modellingRuns/server.ts index 62a74337..d5ef3984 100644 --- a/src/lib/modellingRuns/server.ts +++ b/src/lib/modellingRuns/server.ts @@ -101,9 +101,16 @@ export async function triggerModellingRun(args: { inputs: JSON.stringify(record.configInputs), }); + // Minimal dispatch payload — previewed count and triggered_by are app + // provenance, already durable on the config subtask keyed by this task_id. const dispatch = await callBackend( DISTRIBUTOR_ENDPOINT, - { task_id: task.id, ...record.configInputs }, + { + task_id: task.id, + portfolio_id: record.configInputs.portfolio_id, + scenario_ids: record.configInputs.scenario_ids, + filters: record.configInputs.filters, + }, args.sessionToken, ); if (!dispatch.ok) { From 83117e90c5d09902fe90b7cd8470103299b7e4ee Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 7 Jul 2026 10:46:29 +0000 Subject: [PATCH 08/11] feat(modelling-runs): run config on tasks.inputs; backend contract updates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - The run's config now lives on the task row's new `inputs` column (migration PR #361) instead of a config subtask — sub_tasks belong entirely to the distributor's execution work. - Distributor path corrected to /v1/modelling/trigger-run. - Sub-task granularity is batches, not properties: history shows progress as a percentage under the status chip; the Plans column always shows the plan count (batch counts are never comparable to property/plan numbers). - ADR-0008 amended accordingly. Live verification of the task write waits for PR #361 to be applied. Co-Authored-By: Claude Fable 5 --- ...8-modelling-runs-filters-to-distributor.md | 29 +++++++----- src/app/db/schema/tasks/tasks.ts | 5 +++ .../modelling/run/RunModellingClient.tsx | 31 ++++++------- src/lib/modellingRuns/model.test.ts | 18 +++++--- src/lib/modellingRuns/model.ts | 26 +++++++---- src/lib/modellingRuns/server.ts | 45 +++++++------------ 6 files changed, 84 insertions(+), 70 deletions(-) diff --git a/docs/adr/0008-modelling-runs-filters-to-distributor.md b/docs/adr/0008-modelling-runs-filters-to-distributor.md index 0fb7785b..bbe0985c 100644 --- a/docs/adr/0008-modelling-runs-filters-to-distributor.md +++ b/docs/adr/0008-modelling-runs-filters-to-distributor.md @@ -26,17 +26,22 @@ explicit marker when trigger wiring lands. This is that wiring. - **A Modelling run is recorded as an app-created task** — the BulkUpload trigger convention (`triggerAddressMatching`): Next.js inserts the `tasks` row (service `modelling_run`, `source_id` = portfolio, status `waiting`) - plus one config `sub_task` whose `inputs` JSON carries - `{scenario_ids, filters, previewed_property_count, triggered_by}`, then - passes `task_id` to the distributor, which attaches its execution subtasks - to the same task. **No new table, no migration.** Run history and the - per-scenario in-flight badge read the portfolio's own `modelling_run` tasks - (portfolio-scoped, a handful of rows) and parse the app-authored inputs - JSON; execution progress ("61 of 104 ready") falls out of the existing - subtask-count summary pattern. -- **The distributor receives `{task_id, portfolio_id, scenario_ids, filters}`** - — filters, never property-id lists. The payload stays tiny at any portfolio - size; the config subtask is the durable copy of the same request. + whose **`inputs` column** (added for this — mirrors `sub_task.inputs`) + carries `{portfolio_id, scenario_ids, filters, previewed_property_count, + triggered_by}` as JSON, then passes `task_id` to the distributor, which + attaches its execution sub-tasks to the same task. **One new column, no new + table.** Run history and the per-scenario in-flight badge read the + portfolio's own `modelling_run` tasks (portfolio-scoped, a handful of rows) + and parse the app-authored inputs JSON; execution progress falls out of the + existing subtask-count summary pattern. Sub-task granularity is the + distributor's **batches**, so progress is shown as a percentage — batch + counts are internally consistent but never comparable to property or plan + numbers. +- **The distributor (`POST /v1/modelling/trigger-run`) receives + `{task_id, portfolio_id, scenario_ids, filters}`** — filters, never + property-id lists. The payload stays tiny at any portfolio size; the task's + `inputs` is the durable copy of the same request (plus provenance the + distributor doesn't need). - **Dispatch failure marks the task `failed`** (an improvement on the bulk-upload flow, which leaves failed triggers at `waiting`), so history never shows a ghost run as pending. @@ -63,7 +68,7 @@ explicit marker when trigger wiring lands. This is that wiring. run and the execution share one record by construction. Scenario badges gain a third derived state — "Modelling in progress" — and a per-portfolio run history lists each run's filters, counts, initiator and derived status - with subtask-count progress. + with batch-based percentage progress. - **Concurrent runs warn, never block**: triggering a scenario with a run in flight shows who started it and when, but proceeds. Plans append and latest-per-property wins on read, so overlap is safe; a hard lock could leak diff --git a/src/app/db/schema/tasks/tasks.ts b/src/app/db/schema/tasks/tasks.ts index badce0d0..e778d0d7 100644 --- a/src/app/db/schema/tasks/tasks.ts +++ b/src/app/db/schema/tasks/tasks.ts @@ -18,6 +18,11 @@ export const tasks = pgTable("tasks", { sourceId: text("source_id"), // identifier for the source (e.g., portfolio_id value) + // The task's request inputs, mirroring sub_task.inputs (JSON as text; could + // later change to JSONB). For a modelling run this is the run's config — + // portfolio id, scenario ids, filters (ADR-0008). + inputs: text("inputs"), + updatedAt: timestamp("updated_at", { precision: 6, withTimezone: true }) .defaultNow() .$onUpdate(() => new Date()) diff --git a/src/app/portfolio/[slug]/modelling/run/RunModellingClient.tsx b/src/app/portfolio/[slug]/modelling/run/RunModellingClient.tsx index 34202925..68b6c691 100644 --- a/src/app/portfolio/[slug]/modelling/run/RunModellingClient.tsx +++ b/src/app/portfolio/[slug]/modelling/run/RunModellingClient.tsx @@ -434,21 +434,7 @@ export default function RunModellingClient({ {r.previewedPropertyCount.toLocaleString()} - {r.status.state === "in_progress" ? ( - <> - - {r.status.ready.toLocaleString()} of {r.status.total.toLocaleString()} ready - - - - - - ) : ( - r.planCount.toLocaleString() - )} + {r.planCount.toLocaleString()} {STATUS_LABELS[r.status.state]} + {/* Work is chunked into batches, so show a percentage — + the counts never line up with property/plan numbers */} + {r.status.state === "in_progress" && ( + <> + + {Math.round((100 * r.status.ready) / Math.max(1, r.status.total))}% done + + + + + + )} ))} diff --git a/src/lib/modellingRuns/model.test.ts b/src/lib/modellingRuns/model.test.ts index 656a72fd..3767388f 100644 --- a/src/lib/modellingRuns/model.test.ts +++ b/src/lib/modellingRuns/model.test.ts @@ -44,7 +44,7 @@ describe("normaliseRunFilters", () => { }); describe("buildRunRecord", () => { - it("produces the app-authored task row and config-subtask inputs (ADR-0008)", () => { + it("produces the app-authored task row carrying the run config as its inputs (ADR-0008)", () => { const record = buildRunRecord({ portfolioId: "814", scenarioIds: ["12", "34"], @@ -58,27 +58,33 @@ describe("buildRunRecord", () => { source: "portfolio_id", sourceId: "814", status: "waiting", + inputs: JSON.stringify({ + portfolio_id: 814, + scenario_ids: [12, 34], + filters: { postcodes: ["B93 8SU"], property_types: ["House"] }, + previewed_property_count: 214, + triggered_by: "khalim@domna.homes", + }), }); - expect(record.configInputs).toEqual({ + // What the distributor receives — provenance fields stay on the task only + expect(record.dispatchPayload).toEqual({ portfolio_id: 814, scenario_ids: [12, 34], filters: { postcodes: ["B93 8SU"], property_types: ["House"] }, - previewed_property_count: 214, - triggered_by: "khalim@domna.homes", }); }); }); describe("parseRunConfig", () => { it("round-trips a stored run record back to the app's shape", () => { - const { configInputs } = buildRunRecord({ + const { task } = buildRunRecord({ portfolioId: "814", scenarioIds: ["12"], filters: { builtForms: ["Detached", "Unknown"] }, previewedPropertyCount: 57, triggeredBy: "priya@domna.homes", }); - expect(parseRunConfig(JSON.stringify(configInputs))).toEqual({ + expect(parseRunConfig(task.inputs)).toEqual({ scenarioIds: ["12"], filters: { builtForms: ["Detached", "Unknown"] }, previewedPropertyCount: 57, diff --git a/src/lib/modellingRuns/model.ts b/src/lib/modellingRuns/model.ts index c00fe73a..973bf2b5 100644 --- a/src/lib/modellingRuns/model.ts +++ b/src/lib/modellingRuns/model.ts @@ -168,9 +168,9 @@ export interface RunConfigInputs { } /** - * The rows a Modelling run persists (ADR-0008): an app-authored task (the - * BulkUpload trigger convention) plus the config-subtask inputs that make the - * run's request durable and history-readable. + * The row a Modelling run persists (ADR-0008): an app-authored task whose + * `inputs` carries the run's config — the durable, history-readable record. + * Sub-tasks are the distributor's: per-property execution work. */ export function buildRunRecord(req: RunRequest): { task: { @@ -179,13 +179,22 @@ export function buildRunRecord(req: RunRequest): { source: "portfolio_id"; sourceId: string; status: "waiting"; + inputs: string; }; - configInputs: RunConfigInputs; + /** What the distributor receives — provenance fields stay on the task. */ + dispatchPayload: Omit; } { const filters: RunConfigInputs["filters"] = {}; if (req.filters.postcodes) filters.postcodes = req.filters.postcodes; if (req.filters.propertyTypes) filters.property_types = req.filters.propertyTypes; if (req.filters.builtForms) filters.built_forms = req.filters.builtForms; + const configInputs: RunConfigInputs = { + portfolio_id: Number(req.portfolioId), + scenario_ids: req.scenarioIds.map(Number), + filters, + previewed_property_count: req.previewedPropertyCount, + triggered_by: req.triggeredBy, + }; return { task: { taskSource: `Modelling run – ${req.scenarioIds.length} scenario${req.scenarioIds.length === 1 ? "" : "s"}`, @@ -193,13 +202,12 @@ export function buildRunRecord(req: RunRequest): { source: "portfolio_id", sourceId: req.portfolioId, status: "waiting", + inputs: JSON.stringify(configInputs), }, - configInputs: { - portfolio_id: Number(req.portfolioId), - scenario_ids: req.scenarioIds.map(Number), + dispatchPayload: { + portfolio_id: configInputs.portfolio_id, + scenario_ids: configInputs.scenario_ids, filters, - previewed_property_count: req.previewedPropertyCount, - triggered_by: req.triggeredBy, }, }; } diff --git a/src/lib/modellingRuns/server.ts b/src/lib/modellingRuns/server.ts index d5ef3984..63f03c31 100644 --- a/src/lib/modellingRuns/server.ts +++ b/src/lib/modellingRuns/server.ts @@ -13,13 +13,11 @@ import { } from "./model"; import { builtFormTypeSql, propertyTypeSql } from "@/lib/services/epcSources"; -// The config subtask that makes a run's request durable (ADR-0008); execution -// subtasks attached by the distributor carry other service values. -export const RUN_CONFIG_SERVICE = "modelling_run_config"; - // Distributor entry point on the Model backend. It fans the run out to -// workers, attaching execution subtasks to the task_id we pass (ADR-0008). -const DISTRIBUTOR_ENDPOINT = "/v1/plan/trigger-run"; +// workers, attaching execution sub-tasks to the task_id we pass (ADR-0008). +// NOTE: sub-task granularity is BATCHES, not properties — progress counts are +// internally consistent but never comparable to the previewed property count. +const DISTRIBUTOR_ENDPOINT = "/v1/modelling/trigger-run"; type BackendResult = { ok: true; data: T } | { ok: false; status: number; message: string }; @@ -62,8 +60,9 @@ export type TriggerRunOutcome = | { kind: "dispatch_failed"; status: number; message: string }; /** - * Record + dispatch a Modelling run: app-authored task and config subtask - * (BulkUpload trigger convention), then hand task_id to the distributor. A + * Record + dispatch a Modelling run: one app-authored task whose `inputs` + * carries the run's config (BulkUpload trigger convention), then hand + * task_id to the distributor — sub_tasks are its, one per property. A * failed dispatch marks the task failed so history never shows a ghost run * as pending (ADR-0008). */ @@ -94,23 +93,12 @@ export async function triggerModellingRun(args: { .insert(tasks) .values({ ...record.task, jobStarted: now }) .returning(); - await db.insert(subTasks).values({ - taskId: task.id, - status: "waiting", - service: RUN_CONFIG_SERVICE, - inputs: JSON.stringify(record.configInputs), - }); // Minimal dispatch payload — previewed count and triggered_by are app - // provenance, already durable on the config subtask keyed by this task_id. + // provenance, already durable on the task's inputs. const dispatch = await callBackend( DISTRIBUTOR_ENDPOINT, - { - task_id: task.id, - portfolio_id: record.configInputs.portfolio_id, - scenario_ids: record.configInputs.scenario_ids, - filters: record.configInputs.filters, - }, + { task_id: task.id, ...record.dispatchPayload }, args.sessionToken, ); if (!dispatch.ok) { @@ -210,9 +198,10 @@ export interface RunHistoryRow { } /** - * Run history for a portfolio: its modelling_run tasks with app-authored - * config parsed back out and progress from execution-subtask counts. One - * grouped query — portfolio-scoped, never per-run probes. + * Run history for a portfolio: its modelling_run tasks with the app-authored + * config parsed from task inputs and progress from sub-task counts (one + * sub-task per property, distributor-attached). One grouped query — + * portfolio-scoped, never per-run probes. */ export async function listModellingRuns( portfolioId: string, @@ -224,10 +213,10 @@ export async function listModellingRuns( jobStarted: tasks.jobStarted, jobCompleted: tasks.jobCompleted, status: tasks.status, - configInputs: sql`max(${subTasks.inputs}) filter (where ${subTasks.service} = ${RUN_CONFIG_SERVICE})`, - total: sql`count(${subTasks.id}) filter (where ${subTasks.service} is distinct from ${RUN_CONFIG_SERVICE})::int`, - completed: sql`count(case when lower(${subTasks.status}) in ('completed','complete') and ${subTasks.service} is distinct from ${RUN_CONFIG_SERVICE} then 1 end)::int`, - failed: sql`count(case when lower(${subTasks.status}) in ('failed','failure','error') and ${subTasks.service} is distinct from ${RUN_CONFIG_SERVICE} then 1 end)::int`, + configInputs: tasks.inputs, + total: sql`count(${subTasks.id})::int`, + completed: sql`count(case when lower(${subTasks.status}) in ('completed','complete') then 1 end)::int`, + failed: sql`count(case when lower(${subTasks.status}) in ('failed','failure','error') then 1 end)::int`, }) .from(tasks) .leftJoin(subTasks, eq(subTasks.taskId, tasks.id)) From 8f7f5b3a88ed9d553740cb81fd2caa3f8beb6a49 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 7 Jul 2026 17:37:16 +0000 Subject: [PATCH 09/11] fix(modelling): set-and-forget trigger state, visible back link, filter clearing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-review UX feedback on the run journey: - The post-trigger card computed its ETA from the just-cleared scenario selection, so it always promised "~5 minutes" whatever the run size. ETA now comes from the captured run (estimateRunMinutes, TDD'd) and the done state is a compact status banner above a still-visible Recent runs table, which polls every 5s while a run is moving (historyRefetchInterval, TDD'd) — trigger, glance, leave. - The only way back was a text-xs gray-400 breadcrumb (~2.5:1, sub-AA); replaced with an accessible "← Back to {portfolio}" link. - "Remove all filters" button on Step 2. - Scenario rows were a checkbox nested inside a button (state invisible to screen readers); now a label wrapping a real checkbox. - Large-run confirm lists the scenario names and selection being committed, is labelled, closes on Escape, and starts focus on "Go back". - Postcode filter no longer fails silently: loading and error-with-retry states. - Run status "Dispatched" reads "Starting up"; scenarios page gains a "Run modelling" entry point. Co-Authored-By: Claude Fable 5 --- .../[slug]/(portfolio)/scenarios/page.tsx | 24 ++- .../modelling/run/RunModellingClient.tsx | 194 ++++++++++++------ src/lib/modellingRuns/model.test.ts | 35 ++++ src/lib/modellingRuns/model.ts | 24 +++ 4 files changed, 205 insertions(+), 72 deletions(-) diff --git a/src/app/portfolio/[slug]/(portfolio)/scenarios/page.tsx b/src/app/portfolio/[slug]/(portfolio)/scenarios/page.tsx index 05113760..ba1afc38 100644 --- a/src/app/portfolio/[slug]/(portfolio)/scenarios/page.tsx +++ b/src/app/portfolio/[slug]/(portfolio)/scenarios/page.tsx @@ -40,13 +40,23 @@ export default async function ScenariosPage(props: { }} /> - - + New scenario - +
+ {scenarios.length > 0 && ( + + Run modelling + + )} + + + New scenario + +

Each scenario is a question you ask the modelling engine. Its diff --git a/src/app/portfolio/[slug]/modelling/run/RunModellingClient.tsx b/src/app/portfolio/[slug]/modelling/run/RunModellingClient.tsx index 68b6c691..3d4808d4 100644 --- a/src/app/portfolio/[slug]/modelling/run/RunModellingClient.tsx +++ b/src/app/portfolio/[slug]/modelling/run/RunModellingClient.tsx @@ -2,16 +2,18 @@ import { useState } from "react"; import Link from "next/link"; -import { useRouter } from "next/navigation"; import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; -import { PlayIcon } from "@heroicons/react/24/solid"; +import { ArrowLeftIcon, CheckCircleIcon } from "@heroicons/react/24/solid"; import { RUN_FILTER_BUILT_FORMS, RUN_FILTER_PROPERTY_TYPES, RunFilters, RunStatus, + estimateRunMinutes, + historyRefetchInterval, isLargeRun, MAX_FILTER_POSTCODES, + selectionSummary, } from "@/lib/modellingRuns/model"; export interface ScenarioOption { @@ -42,7 +44,7 @@ interface RunHistoryRow { } const STATUS_LABELS: Record = { - dispatched: "Dispatched", + dispatched: "Starting up", in_progress: "In progress", complete: "Complete", failed: "Failed", @@ -128,7 +130,6 @@ export default function RunModellingClient({ portfolioName: string; scenarios: ScenarioOption[]; }) { - const router = useRouter(); const queryClient = useQueryClient(); const [selScenarios, setSelScenarios] = useState>(new Set()); const [selPcs, setSelPcs] = useState>(new Set()); @@ -174,7 +175,8 @@ export default function RunModellingClient({ const history = useQuery({ queryKey: ["modelling-runs", portfolioId], - refetchInterval: 30_000, + // v4 signature: (data, query). Fast while a run is moving, slow at rest. + refetchInterval: (data) => historyRefetchInterval(data), queryFn: async () => { const res = await fetch(`/api/portfolio/${portfolioId}/modelling-runs`); const body = await res.json(); @@ -220,16 +222,22 @@ export default function RunModellingClient({ if (matched != null && isLargeRun(matched)) setConfirmOpen(true); else run.mutate(); }; - const eta = planTotal ? Math.max(5, Math.round(planTotal / 300) * 5) : 5; + const filterCount = selPcs.size + selTypes.size + selForms.size; + const clearFilters = () => { + setSelPcs(new Set()); + setSelTypes(new Set()); + setSelForms(new Set()); + }; return (

-
- - {portfolioName} / Portfolio - {" "} - / Run modelling -
+ + + Back to {portfolioName} +

Run your scenarios.

@@ -239,34 +247,42 @@ export default function RunModellingClient({

{done ? ( -
-
- -
-

Modelling started

-

- We're modelling {done.properties.toLocaleString()}{" "} - {done.properties === 1 ? "property" : "properties"} against {done.scenarios}{" "} - scenario{done.scenarios > 1 ? "s" : ""} —{" "} - {(done.properties * done.scenarios).toLocaleString()} plan - {done.properties * done.scenarios === 1 ? "" : "s"} in total. Results for a run this - size are typically ready within ~{eta} minutes. You can check on this run any - time under Recent runs. -

-
- - +
+
+ +
+

+ Modelling started — you don't need to stay on this page +

+

+ We're modelling {done.properties.toLocaleString()}{" "} + {done.properties === 1 ? "home" : "homes"} against {done.scenarios}{" "} + scenario{done.scenarios === 1 ? "" : "s"} —{" "} + {(done.properties * done.scenarios).toLocaleString()} plan + {done.properties * done.scenarios === 1 ? "" : "s"}. Results for a run this size + are typically ready within{" "} + ~{estimateRunMinutes(done.properties * done.scenarios)} minutes — progress + updates under Recent runs below as it happens. +

+
+
+ + + Back to portfolio + +
) : ( @@ -298,9 +314,8 @@ export default function RunModellingClient({ const on = selScenarios.has(s.id); return (
- + {on && s.status === "running" && (
This scenario is already being modelled. You can still include it — the @@ -341,9 +355,19 @@ export default function RunModellingClient({ {/* Step 2 — properties */}
- - Step 2 · Properties - +
+ + Step 2 · Properties + + {filterCount > 0 && ( + + )} +

Which homes?

@@ -353,15 +377,39 @@ export default function RunModellingClient({ neither.

- ({ - value: p.postcode, - count: p.count, - }))} - selected={selPcs} - onToggle={(v) => toggle(selPcs, setSelPcs, v)} - /> + {postcodesQuery.isError ? ( +
+ + Postcode + +

+ We couldn't load your portfolio's postcodes.{" "} + +

+
+ ) : postcodesQuery.isLoading ? ( +
+ + Postcode + +

Loading postcodes…

+
+ ) : ( + ({ + value: p.postcode, + count: p.count, + }))} + selected={selPcs} + onToggle={(v) => toggle(selPcs, setSelPcs, v)} + /> + )} ({ value }))} @@ -381,8 +429,12 @@ export default function RunModellingClient({

- {/* Recent runs */} -
+ + )} + + {/* Recent runs — always visible: after a run is triggered this is where + its live status lands, so the banner above can say "watch below" */} +

Recent runs

@@ -470,9 +522,7 @@ export default function RunModellingClient({
)} -
- - )} +
{/* Summary tray */} {!done && scenarioIds.length > 0 && ( @@ -545,10 +595,14 @@ export default function RunModellingClient({
{ + if (e.key === "Escape") setConfirmOpen(false); + }} className="fixed inset-0 z-50 flex items-center justify-center bg-brandblue/35 p-5 backdrop-blur-sm" >
-

+

This is a large run

@@ -557,8 +611,18 @@ export default function RunModellingClient({ {planTotal?.toLocaleString()} plans. A run this size can't be stopped once it starts.

+
+ + Scenarios:{" "} + {scenarioIds.map((id) => scenarioById.get(id)?.name ?? id).join(" · ")} + + + Homes: {selectionSummary(filters)} + +