assessment-model/CONTEXT.md

Context

This document captures the domain language used in this project. Terms here are the canonical ones — when more than one word exists for a concept, we pick one and treat the others as aliases to avoid.

This file grows as terms are resolved during design conversations. Concepts that haven't been examined yet are not listed.

Language

Bulk upload

BulkUpload: A user-supplied spreadsheet of addresses for a Portfolio, transformed and matched to UPRNs before being inserted as Properties. Has an explicit lifecycle from upload through finalisation. Avoid: import, batch, file upload, ingest

ColumnMapping: The user's declaration of which spreadsheet column means what (e.g. column "Property Address" means address_1). Stored as JSON on the BulkUpload row. Avoid: schema, header map, field mapping

UPRN: Unique Property Reference Number — the UK national identifier for an address. Address matching attaches a UPRN to each row where possible.

Address matching: The pipeline stage that splits the source file by postcode, looks up UPRNs, and produces matched-address output. Triggered via FastAPI. Avoid: postcode lookup, address resolution, address lookup

Combiner: The pipeline stage that aggregates the per-postcode address-matching outputs into a single combined CSV in S3, ready for review. Avoid: aggregator, merger

Finalise: The terminal action that reads the combiner output, inserts rows as Properties on the Portfolio, and decides whether the BulkUpload needs further review. Avoid: import, commit, ingest

Landlord overrides

Landlord: The housing association supplying a Portfolio's BulkUploads. A Landlord knows facts about their properties that EPC data doesn't (e.g. that a cavity has been filled), and those facts take precedence when computing an assessment. Avoid: customer, client, owner, organisation (Organisation is a separate, broader entity)

Landlord override: A landlord-supplied fact about a property that takes precedence over EPC-derived defaults when computing an assessment. The end-to-end Landlord override journey has two layers — a VocabularyMapping layer (this glossary entry below) and a per-Property fact layer (the Property override, below). Avoid: customer data, manual override, landlord data

Property override: The per-Property fact layer — one resolved fact per (Property, Building part, component), where component is one of wall_type/roof_type/property_type/built_form_type. Holds a snapshot of the resolved enum value (a denormalised copy of the VocabularyMapping outcome at finalise time, so two Properties sharing a description can later diverge), plus the original spreadsheet text it resolved from. Materialised by the finaliser for UPRN-matched Properties only (v2); the resolved value is never UNKNOWN — the Verify step forces every UNKNOWN to be mapped before Finalise, and an unresolved description fails the run. See ADR-0005 (table) and ADR-0006 (population). Avoid: per-property mapping, property fact, override row

The Model backend consumes Property overrides at modelling time; property type (and built form when known) lets the predict-EPC service estimate a never-EPC'd property from surrounding homes of similar archetype.

Source row id: A synthetic UUID minted per source-file row at start-address-matching and written into both the address CSV and the classifier CSV. It is the stable join key that lets the finaliser tie a row's identity (combiner output → property_id) to that row's raw descriptions (classifier CSV), since neither file preserves row order and Internal Reference is absent from the classifier CSV. See ADR-0006. Avoid: row index, internal reference (a separate, optional landlord field)

VocabularyMapping: The translation from a free-text description to a canonical domain enum value (e.g. "cavity: filledcavity" → WallType.CAVITY). Two producers: the ColumnClassifier (an LLM in the Model service — needed because BulkUpload spreadsheets contain arbitrary landlord text that must be sanitised) and the deterministic OS-code mapping used by the postcode-search journey (a fixed OS classification code → enum table; no interpretation involved). Stored per-Portfolio, one row per (category, description). A row carries provenance (classifier, user, or os_places) so user overrides survive re-classification and OS-derived facts stay distinguishable. Avoid: column mapping (that's a separate concept — see ColumnMapping above), classification, dictionary

Postcode search

Postcode search: The journey that adds Properties to a Portfolio by searching a postcode: postcodes.io validates/normalises and supplies geography (local authority, constituency), OS Places supplies the addresses (UPRN, coordinates, classification code), and the user selects which addresses to add — across multiple postcode searches in one basket, submitted once. Raw OS responses are cached per-postcode (postcode_search) and re-served while fresh. Creates real Properties whose energy data is absent-until-modelled (derived state, no marker column); writes property type / built form facts through the Landlord-override layers via the deterministic OS-code mapping. Avoid: address import, quick add, single add

Building parts

Building part: One physically distinct part of a dwelling described by a single entry within a multi-valued cell. A dwelling is one Main building plus zero or more Extensions. Per-part descriptions appear as comma-separated entries in physical-element columns (e.g. Walls, Roofs); whole-dwelling columns (e.g. Property Type) carry a single entry and are not split per part. Avoid: annexe, unit, section, dwelling part

Main building: The principal building part of a dwelling — exactly one per address. The others are Extensions.

Extension: A building part that is not the Main building, numbered Extension 1 … Extension N-1 for an N-entry address. Avoid: annexe, addition, outbuilding

Multi-entry: The property of a BulkUpload row whose physical-element cells hold more than one comma-separated entry, one per Building part. Always intra-cell in our data — never multiple rows sharing one address/UPRN. Within a row, the multi-valued columns agree on entry-count, so position i is the same Building part across every multi-valued column. Avoid: multi-row, multi-record, duplicate address

Building-part ordering (a.k.a. ordering): The user's declaration, captured once per file, of which list-position maps to which Building part — because the entry order is a consistent per-file mistake ("A, B" could be [Main, Extension 1] or [Extension 1, Main]). Stored per entry-count as a permutation. See ADR-0004. Avoid: sort order, sequence, column mapping

Scenarios

Scenario: A named modelling configuration for a Portfolio — the question a modelling run answers (e.g. "what does it take to reach EPC C, excluding solid floor insulation?"). Captures the target, the measure constraints and the housing type (Social/Private — a genuine modelling input because it drives funding treatment, not portfolio metadata). The configuration is immutable once created — a change of mind is a new Scenario — but the name is a label, renameable at any time, and a Scenario that has no Plans yet may be deleted (an unasked question can be withdrawn; a modelled one cannot). Plans are generated against a Scenario by the modelling engine; impact/results live on the Plans (and Reporting), never on the Scenario itself. Avoid: plan (a Plan is one property's modelled answer within a Scenario), simulation, run

Goal: The optimisation objective of a Scenario. Increasing EPC targets a band (its goal value is the band letter, required). Every other goal (Reducing CO₂ emissions, Energy Savings, Valuation Improvement) means "improve that dimension as much as possible" — it carries no goal value; the only brake is an optional Budget, without which the engine recommends the maximum interventions. Avoid: target (reserve for the EPC band specifically), objective

Budget: An optional per-Scenario spending cap, denominated per property (£) — never a whole-portfolio envelope. Absent budget = no cap. Avoid: portfolio budget (a separate concept on the Portfolio row), envelope, cost limit

Disruptive measure: A measure whose installation involves major internal works for residents (internal wall insulation, solid/suspended floor insulation, room-in-roof insulation). Landlords frequently exclude them; the UI sign-posts them and the "Low disruption" quick-start excludes exactly this set. Canonical list: DISRUPTIVE_MEASURES in the Scenario domain model. Avoid: invasive (unused here), wet trades (a narrower, overlapping set: plaster-and-screed work)

Measure exclusion: A Scenario constraint that takes a measure off the table for the optimiser. The only measure-exclusion concept on a Scenario — there is no stored inclusion list; "only Solar PV + ASHP" is expressed by excluding everything else. An empty exclusion set is valid and means every measure is available. (Exclusions decide which measures are available; Fabric first decides the order they are considered — the two compose.) Avoid: inclusions, allowed measures (as a stored concept — UI may present "allowed/excluded" toggles, but what's captured is the exclusion set)

Fabric first: A Scenario constraint that forces the optimiser to apply every viable fabric measure (insulation, glazing, ventilation) before it will consider heating or renewables — the heating/renewables tier is reached only if the Scenario's upgrade target is still unmet once fabric is exhausted. Unlike a Measure exclusion (which removes a measure entirely), it constrains the order measures are considered, not which are on the table; the two compose (an excluded fabric measure simply isn't among those forced in). Off by default, set at creation and immutable like the rest of the config. The modelling backend owns the ordering logic and the fabric/heating split — the app only records the flag (trigger-payload field enforce_fabric_first). Avoid: fabric only (the quick-start that excludes heating/renewables outright — fabric-first still allows them as a fallback), sequencing/phasing (reserve for PIBI installation order), enforce fabric first (a UI verb on the toggle, not the domain noun)

Modelling run: One user-initiated act of triggering modelling: a set of Scenarios crossed with one filter-defined property selection within a Portfolio. A durable record of what was asked — the Run filters, the Scenarios, who triggered it, when, and the matched-property count shown at preview — so Plans can be traced back to the selection that produced them, alongside how the work is going. A Scenario is the question; a Modelling run asks it (possibly again, possibly for a different property selection). Re-running an already-modelled property appends a newer Plan; latest wins on read. Avoid: job, batch, trigger request, modelling task (the pipeline's execution records)

Run filter: The property-selecting constraints of a Modelling run: postcode, property type, built form, and Tags (each multi-select). An absent filter means unconstrained — "all postcodes" is the absence of a postcode filter, never an enumeration. Type and built form are the canonical enum vocabularies plus Unknown, which selects properties with no resolvable value at all; Tags are the Portfolio's own Tags (multi-select is any-of). Resolution follows the Landlord-override rule for type/built-form (an override fact beats an EPC-derived value) and a direct membership lookup for Tags — but in all cases the modelling backend is the single authority for resolving filters to properties (the app never re-implements the rule — matched counts come from the backend, which reads the same Tag membership the app writes). See ADR-0013. Avoid: property query, segment, search (the postcode-search journey is unrelated)

Reporting

Gross cost: The all-in price of delivering a Scenario's works: construction works + project delivery + contingency. Nothing a landlord pays for sits outside it. Avoid: total cost, capex, cost of works (that's the construction line only)

Net cost: Gross cost minus secured funding. The figure a board commits to. Avoid: net investment, cost after grants

Project delivery: The professional and management costs of delivering works, as distinct from the construction works themselves. Avoid: PC cost, prelims

Current stock: The reporting view of a Portfolio with no Scenario applied — every figure is Effective performance, aggregated. Also the "before" side of any scenario comparison ("Current → After scenario"). Avoid: baseline (reserved for property-level performance concepts), portfolio overview

Scenario overlay: The "After scenario" read model — the aggregate reporting picture of a Portfolio once a plan selection is applied: the portfolio-after averages and EPC-band distribution, the works-list ledger (Gross cost, Net cost, homes upgraded, funding) and the genuine SAP uplift. The after half of every Current → After comparison. A plan selection is either a specific Scenario's plans or the recommended-plans default (the latest is_default Plan per Property); the default view is the same overlay with no EPC target and the report-view filters (compliance window, lodged baseline — ADR-0010) inert. Computed once in getScenarioOverlay (src/lib/reporting/overlay.ts) and served by both scenario-metrics endpoints. See ADR-0015. Avoid: scenario metrics (the endpoint name, not the concept), after-distribution (only the band part), scenario results (Plans hold results; the overlay aggregates them)

Live projects

Project: A delivery programme within a Portfolio — a batch of Properties taken through retrofit together — identified by its HubSpot project code (hubspot_deal_data.project_code; a stable project id will replace the code once the HubSpot ops board is settled, so the code is the current identifier, not the essence). A Property whose deal carries no project code sits outside every Project (the "Unknown Project" grouping) and cannot be reached by selecting Projects. Distinct from the Ara Project (next section) — the app-owned works programme that will eventually supersede this feature; where "Project" alone is ambiguous, say HubSpot Project. See ADR-0018. Avoid: programme, scheme, deal group, batch (a Batch is a separate per-deal field)

Bulk document download: An app-triggered, asynchronous assembly of a single ZIP of the documents held against a chosen set of Properties — picked as a row selection in the Documents tab, scoped by the tab's current Project/group — delivered as a time-limited link emailed to the requester (and, within the session, retrievable in-app). A selection that matches no documents produces no ZIP: it fails rather than emailing an empty archive. Properties are matched to their documents by landlord property id. See ADR-0011. Avoid: bulk export, document export, zip download

Ara Projects

The works-management module at top-level /projects (schema in src/app/db/schema/projects/, merged in PR #404). The strategic successor to the HubSpot-sourced "Live projects" feature above — both coexist for now, with no migration in v1. See ADR-0018 and the schema's own docs at src/app/db/schema/projects/schema.md.

Project (Ara): An organisation-owned works programme — the aggregate root of the Ara Projects module. Belongs to an organisation and is many-to-many with Properties (via project_property), unlike the HubSpot Project above (externally sourced, at most one per Property). Owns its Project Workstreams, which in turn own Stages, Evidence requirements and Contractor assignments. Where "Project" alone is ambiguous, say Ara Project. Avoid: programme, scheme, works package

Workstream: A category of works (Windows, Doors, Roofs, Electrical, Heating, …) — reference data in workstream, instantiated for one Ara Project as a Project Workstream (project_workstream), which owns its own Stage ladder, Evidence requirements and Contractor assignments. Avoid: measure (Measure is taken by the Scenarios domain), trade, work type

Stage: One step in a Project Workstream's ordered ladder (project_workstream_stage, positioned by order). Each Project Workstream owns its own ladder — different workstreams may follow different workflows. The v1 default ladder is Ordered / In progress / Completed / Charged / Closed, and the terminal stage is the one with the highest order. A Work order's stage FK (work_order.project_workstream_stage_id) is the only lifecycle mechanism in v1 — work_order.status and project_workstream.current_stage_id exist in the schema but are untouched. work_order.priority is now used, but it is not a lifecycle mechanism: see Priority. Avoid: status (the unused text column), phase, step

Work order: One unit of work issued to a contractor: a (Property, Project Workstream Stage, contractor organisation, unique reference, forecast end date) row in work_order. Its position in the workflow is its Stage FK (see Stage). Avoid: job, ticket, task (a Task is the BulkUpload orchestration handle)

Priority: A flag on a Work order (work_order.priority, boolean), meaning "look at this one first". It is a stored fact — someone marks it — and so is deliberately independent of Overdue, which is derived from the forecast end date and the Stage ladder: a flagged work order may be well inside its dates, and a late one nobody flagged is not a priority. Not a scale: there are no priority levels in v1, only flagged and not. Avoid: urgency, severity, high/medium/low (there are no levels), escalation

Evidence requirement / Evidence: An Evidence requirement (project_workstream_evidence_requirement) is configuration: a required file_type (the existing enum) per Project Workstream, optionally narrowed to a single Stage. Evidence is a submission: an uploaded_files row with file_source = 'projects', linked to its Work order (work_order_id) and optionally to the requirement it satisfies. Evidence is advisory-only in v1 — badges and counts, never a gate: it does not block stage transitions and there is no approval state. Avoid: document (generic), attachment, proof

Contractor: An organisation assigned to a Project Workstream via project_workstream_contractor, carrying per-assignment update_stages / upload_documents permission flags — not a separate entity or table. Contractor users sign in through the existing team / team_members structures (each organisation gets a default team). Avoid: supplier, installer, vendor, subcontractor

Tags

Tag: A user-created, coloured label owned by a Portfolio, used to form an arbitrary group of its Properties for bulk actions (e.g. running Scenarios over the group, or excluding it from a Reporting view). Flat (no hierarchy) and many-to-many: a Property carries zero or more Tags, and a Tag labels zero or more Properties. Distinct from a HubSpot Project (a HubSpot-sourced grouping, at most one per Property) and from a Scenario (a modelling question): a Tag is an app-owned, hand-curated selection with no external source and no modelling semantics of its own. Avoid: label, category, group (the group is the effect of a Tag, not a separate entity), segment, tag group

Bulk tag assignment: Assigning one Tag to many Properties at once from a small uploaded file (CSV/Excel) of property identifiers — either landlord property id or UPRN, whichever column the file carries. Parsed and matched entirely in the app (no FastAPI pipeline); the target Tag is chosen in the upload dialog, not named in the file. An identifier that matches no Property is reported back (never silently dropped); an already-tagged match is a no-op. The parallel filter-driven path assigns a Tag to every Property matching the current property-table filter. Avoid: tag import, bulk tagging (verb is fine; the noun is Bulk tag assignment)

Lifecycle

A BulkUpload moves through these statuses:

ready_for_processing
  → mapping_complete         (user submits ColumnMapping; Next.js writes)
    → processing             (Address matching triggered; Next.js writes)
      → combining            (Combiner stage running; FastAPI writes directly)
        → awaiting_review    (Combiner output in S3; FastAPI writes directly)
          → finalising       (Finalise dispatched; Next.js writes via compare-and-swap)
            → complete       (Finaliser succeeded; FastAPI/Lambda writes directly)
            → failed         (Finaliser failed; FastAPI/Lambda writes directly)

complete and failed are terminal. finalising is the in-flight state of the async finaliser (mirrors combining); the UI renders it as "Uploading to ARA". See ADR-0005.

Re-mapping (PATCHing columnMapping) is legal only in ready_for_processing and mapping_complete. Any later state rejects with 409.

Two writers: Next.js owns transitions out of mapping_complete, into processing, and the awaiting_review → finalising compare-and-swap at Finalise dispatch. FastAPI/Lambda owns combining, awaiting_review, and the terminal finalising → complete/failed — writing them direct to the DB during the combiner and finaliser runs. The BulkUpload aggregate observes both. See ADR-0005.

At awaiting_review, Finalise is gated (not a new status — a precondition on the action): when classifier columns were mapped the user must acknowledge the classification-verification step, and when the file is Multi-entry they must confirm the Building-part ordering. See ADR-0004.

See ADR-0001 for the deliberate "not yet" decisions baked into this lifecycle.

Relationships

• A Portfolio has many BulkUploads.
• A BulkUpload produces zero or more Properties when finalised.
• A BulkUpload has at most one Task (the orchestration handle for the FastAPI pipeline run); a Task has many SubTasks (one per pipeline stage: address matching, combiner).
• A Portfolio has many VocabularyMappings — one row per (category, description) it has ever encountered across all its BulkUploads. See ADR-0002.
• A Recommendation belongs to exactly one Plan. Denormalised onto recommendation.plan_id; the plan_recommendations join table is being retired.
• A Recommendation has at most one Material. Denormalised onto recommendation.material_id (+ material_quantity, material_quantity_unit, material_depth). Historically (pre-~2023) a recommendation could carry multiple materials; ~128 such legacy rows were reconciled to one each on 2026-06-07. The cardinality guard in the backfill enforces this going forward.

Baseline performance

Lodged performance: The SAP score, EPC band, CO₂ emissions, and primary energy intensity as submitted to the government EPC register. Ground truth from the register; never modified. Avoid: original performance, registered performance

Effective performance: The SAP score (and associated metrics) that the modelling engine actually uses as its baseline. Usually equals Lodged performance, but differs when a Landlord override or data-quality issue makes the lodged certificate unreliable — triggering a Rebaseline. Avoid: current performance, adjusted performance

Rebaseline: The act of substituting a corrected set of performance metrics in place of the Lodged values. Recorded on property_baseline_performance with a rebaseline_reason enum value: none, pre_sap10, physical_state_changed, or both. Avoid: override, adjustment, correction

EPC provenance (epc_property.source): Whether a property's EPC picture is a real certificate or a gap-fill. Three values:

• lodged — a real, current public/landlord EPC exists.
• predicted — no certificate exists anywhere, so the EPC was gap-filled from nearby properties (EPC Prediction).
• expired — also gap-filled, but conditioned on this dwelling's own expired certificate from the historic register (Model ADR-0054). The gov EPC API only serves certificates registered since 2012, so a pre-2012 dwelling looks EPC-less to ingestion even though a real, stale certificate exists for it.

Independent of Rebaseline: a lodged property may still be rebaselined, and a gap-filled property still carries Lodged-performance figures (mirrored estimates), so the presence of lodged_* columns does not imply a real certificate — only source = lodged does. Avoid: estimated EPC (reserve "estimated" for the UI signal), source EPC

Predicted slot: predicted and expired are two flavours of one slot — a property holds at most one of them, and a re-ingestion may flip which. Every query must therefore address the slot (source IN ('predicted','expired')), never a single member: matching source = 'predicted' alone makes expired rows invisible, and since epc_property.source is plain text with no enum or CHECK constraint, nothing catches it. Mirrors PREDICTED_SLOT_SOURCES in the Model repo. See ADR-0014. Avoid: predicted source (it is two sources)

EPC coverage — does any certificate exist for this dwelling? lodged or expired ⇒ yes, one does (current, or out of date). predicted, or no epc_property rows at all ⇒ no, none does.

This is a different question from provenance's "was the picture modelled?" — an expired property is gap-filled and has a certificate. Conflating the two is what made the reporting cards wrong: the "Homes Without an EPC" card counted every gap-fill, so homes whose only fault was an out-of-date certificate were reported as having no EPC at all. The cards partition on coverage, not on provenance. Avoid: missing EPC (ambiguous — say "without an EPC" for coverage, "estimated" for provenance) Avoid: estimated EPC (reserve "estimated" for the UI signal), source EPC

Provenance signal (UI): What the user is told about an EPC's trustworthiness, derived from EPC provenance and Rebaseline together: Estimated (source IN (predicted, expired) — the whole predicted slot, dominant; "estimated based on nearby homes") › Re-modelled (source = lodged AND rebaseline_reason != none — effective diverged from the lodged certificate under SAP 10) › none (source = lodged AND rebaseline_reason = none — effective equals lodged). Estimated always wins when both could apply.

An expired property is Estimated: its picture was modelled, not read off a current certificate. That it also has an (out-of-date) certificate is a matter of EPC coverage, which the reporting cards use — not of this signal. Avoid: estimation notification, banner (those are component names)

Likely downgrade: A property whose Lodged band sits above its Effective band — a re-survey would likely certificate it lower. Judged on band movement only; SAP-point differences within a band don't qualify. Avoid: SAP downgrade (point-level, the retired signal), at-risk property

Likely upgrade: A property whose Effective band sits above its Lodged band — Landlord overrides or SAP 10.2 scoring improved the modelled picture, so a re-survey alone would likely certificate it higher, without works. The cheapest compliance move in a portfolio. Avoid: quick win, free upgrade

Example dialogue

Dev: "If the Combiner finishes but the user hasn't clicked Finalise, what does the user see?" Domain expert: "The BulkUpload sits in awaiting_review. The frontend polls and shows a 'review and confirm' button. Nothing's been written to Properties yet."

Dev: "And if Finalise runs and 30% of rows have no UPRN?" Domain expert: "Those still get imported as Properties — just without a UPRN — and the BulkUpload moves to complete. Manual cleanup happens later in the property table."

(Planned change — v3 / ADR-0006: no-UPRN rows will move to a separate staging table to be re-matched, so property holds only matched rows. v2 does not change this yet — and v2 writes Property overrides only for the UPRN-matched rows.)

Flagged ambiguities

• The portfolio Property table's Property Type, Built Form, Construction Age, Main Fuel, Wall Type, Roof Type and Heating System columns show the landlord-override-resolved value — the same precedence the modelling Run filter applies (override fact → EPC-derived → Unknown) — not the raw epc_property value or the legacy property-row column. See ADR-0012. All seven are also filters, and the per-part ones (wall/roof) read the main building part. Wall/Roof/Heating filter by coarse bucket rather than the full vocabulary — their resolved values are free-text (override vocab / EPC epc_energy_element.description / legacy column) with hundreds of variants, so a fixed set of prefix/substring buckets (Wall → Cavity / Solid Brick / Timber Frame / Stone / …; Roof → Pitched / Flat / Room-in-Roof / …; Heating → Gas Boiler / Community / Electric Storage / Heat Pump / …) matches them, while the columns still display the full description. classifyDescriptor is the pure twin of the SQL bucketing. Two label subtleties: Construction Age is an RdSAP band (e.g. "1900–1929", "before 1900"), never a single year (it was formerly the "Year Built" column); and Main Fuel for new-approach properties reads the EPC's epc_main_heating_detail.main_fuel_type (an RdSAP main_fuel code or an EPR string, folded to a coarse label) beneath any landlord override — it only reads Unknown when neither is present.
• The UI surfaces labelled "Current EPC" and "Current Efficiency State" (property table, building-passport card) show Effective performance, not Lodged — despite the glossary advising against "current performance" for Effective. The label is a product choice ("current" = the property's true present-day modelled state); the underlying datum is still Effective performance. The "Lodged EPC" column/badge is the only surface showing Lodged performance, and only when a real certificate exists (source = lodged).
• "Upload" is used in the codebase to mean both the file-on-S3 and the BulkUpload row. We standardise on BulkUpload for the row; the file is just "the source file."
• "Onboarding" appears in some route paths (bulk_onboarding_inputs/...) but isn't part of this glossary — we use BulkUpload end-to-end.