assessment-model/CONTEXT.md

# Context

This document captures the domain language used in this project. Terms here are the **canonical** ones — when more than one word exists for a concept, we pick one and treat the others as aliases to avoid.

This file grows as terms are resolved during design conversations. Concepts that haven't been examined yet are not listed.

## Language

### Bulk upload

**BulkUpload**:
A user-supplied spreadsheet of addresses for a Portfolio, transformed and matched to UPRNs before being inserted as Properties. Has an explicit lifecycle from upload through finalisation.
_Avoid_: import, batch, file upload, ingest

**ColumnMapping**:
The user's declaration of which spreadsheet column means what (e.g. column "Property Address" means `address_1`). Stored as JSON on the BulkUpload row.
_Avoid_: schema, header map, field mapping

**UPRN**:
Unique Property Reference Number — the UK national identifier for an address. Address matching attaches a UPRN to each row where possible.

**Address matching**:
The pipeline stage that splits the source file by postcode, looks up UPRNs, and produces matched-address output. Triggered via FastAPI.
_Avoid_: postcode lookup, address resolution, address lookup

**Combiner**:
The pipeline stage that aggregates the per-postcode address-matching outputs into a single combined CSV in S3, ready for review.
_Avoid_: aggregator, merger

**Finalise**:
The terminal action that reads the combiner output, inserts rows as Properties on the Portfolio, and decides whether the BulkUpload needs further review.
_Avoid_: import, commit, ingest

### Landlord overrides

**Landlord**:
The housing association supplying a Portfolio's BulkUploads. A Landlord knows facts about their properties that EPC data doesn't (e.g. that a cavity has been filled), and those facts take precedence when computing an assessment.
_Avoid_: customer, client, owner, organisation (Organisation is a separate, broader entity)

**Landlord override**:
A landlord-supplied fact about a property that takes precedence over EPC-derived defaults when computing an assessment. The end-to-end Landlord override journey has two layers — a **VocabularyMapping** layer (this glossary entry below) and a per-Property fact layer (the **Property override**, below).
_Avoid_: customer data, manual override, landlord data

**Property override**:
The per-Property fact layer — one resolved fact per `(Property, Building part, component)`, where component is one of `wall_type`/`roof_type`/`property_type`/`built_form_type`. Holds a **snapshot** of the resolved enum value (a denormalised copy of the VocabularyMapping outcome at finalise time, so two Properties sharing a description can later diverge), plus the original spreadsheet text it resolved from. Materialised by the finaliser **for UPRN-matched Properties only** (v2); the resolved value is never `UNKNOWN` — the Verify step forces every `UNKNOWN` to be mapped before Finalise, and an unresolved description fails the run. See [ADR-0005](./docs/adr/0005-async-bulk-upload-finaliser.md) (table) and [ADR-0006](./docs/adr/0006-property-overrides-join-and-no-uprn-defer.md) (population).
_Avoid_: per-property mapping, property fact, override row

The Model backend consumes Property overrides at modelling time; property type (and built form when known) lets the **predict-EPC service** estimate a never-EPC'd property from surrounding homes of similar archetype.

**Source row id**:
A synthetic UUID minted per source-file row at `start-address-matching` and written into **both** the address CSV and the classifier CSV. It is the stable join key that lets the finaliser tie a row's identity (combiner output → `property_id`) to that row's raw descriptions (classifier CSV), since neither file preserves row order and `Internal Reference` is absent from the classifier CSV. See [ADR-0006](./docs/adr/0006-property-overrides-join-and-no-uprn-defer.md).
_Avoid_: row index, internal reference (a separate, optional landlord field)

**VocabularyMapping**:
The translation from a free-text description to a canonical domain enum value (e.g. `"cavity: filledcavity"` → `WallType.CAVITY`). Two producers: the `ColumnClassifier` (an LLM in the Model service — needed because BulkUpload spreadsheets contain arbitrary landlord text that must be sanitised) and the **deterministic OS-code mapping** used by the postcode-search journey (a fixed OS classification code → enum table; no interpretation involved). Stored per-Portfolio, one row per `(category, description)`. A row carries provenance (`classifier`, `user`, or `os_places`) so user overrides survive re-classification and OS-derived facts stay distinguishable.
_Avoid_: column mapping (that's a separate concept — see `ColumnMapping` above), classification, dictionary

### Postcode search

**Postcode search**:
The journey that adds Properties to a Portfolio by searching a postcode: postcodes.io validates/normalises and supplies geography (local authority, constituency), OS Places supplies the addresses (UPRN, coordinates, classification code), and the user selects which addresses to add — across multiple postcode searches in one basket, submitted once. Raw OS responses are cached per-postcode (`postcode_search`) and re-served while fresh. Creates real Properties whose energy data is absent-until-modelled (derived state, no marker column); writes property type / built form facts through the Landlord-override layers via the deterministic OS-code mapping.
_Avoid_: address import, quick add, single add

### Building parts

**Building part**:
One physically distinct part of a dwelling described by a single entry within a multi-valued cell. A dwelling is one **Main building** plus zero or more **Extensions**. Per-part descriptions appear as comma-separated entries in physical-element columns (e.g. `Walls`, `Roofs`); whole-dwelling columns (e.g. `Property Type`) carry a single entry and are **not** split per part.
_Avoid_: annexe, unit, section, dwelling part

**Main building**:
The principal building part of a dwelling — exactly one per address. The others are **Extensions**.

**Extension**:
A building part that is not the Main building, numbered **Extension 1 … Extension N-1** for an N-entry address.
_Avoid_: annexe, addition, outbuilding

**Multi-entry**:
The property of a BulkUpload row whose physical-element cells hold **more than one comma-separated entry**, one per **Building part**. Always intra-cell in our data — never multiple rows sharing one address/UPRN. Within a row, the multi-valued columns agree on entry-count, so **position `i` is the same Building part across every multi-valued column**.
_Avoid_: multi-row, multi-record, duplicate address

**Building-part ordering** (a.k.a. **ordering**):
The user's declaration, captured once per file, of which list-position maps to which Building part — because the entry order is a consistent per-file mistake (`"A, B"` could be `[Main, Extension 1]` or `[Extension 1, Main]`). Stored per entry-count as a permutation. See [ADR-0004](./docs/adr/0004-multi-entry-building-part-ordering.md).
_Avoid_: sort order, sequence, column mapping

## Lifecycle

A **BulkUpload** moves through these statuses:

```
ready_for_processing
  → mapping_complete         (user submits ColumnMapping; Next.js writes)
    → processing             (Address matching triggered; Next.js writes)
      → combining            (Combiner stage running; FastAPI writes directly)
        → awaiting_review    (Combiner output in S3; FastAPI writes directly)
          → finalising       (Finalise dispatched; Next.js writes via compare-and-swap)
            → complete       (Finaliser succeeded; FastAPI/Lambda writes directly)
            → failed         (Finaliser failed; FastAPI/Lambda writes directly)
```

`complete` and `failed` are terminal. `finalising` is the in-flight state of the
async finaliser (mirrors `combining`); the UI renders it as "Uploading to ARA". See
[ADR-0005](./docs/adr/0005-async-bulk-upload-finaliser.md).

Re-mapping (PATCHing `columnMapping`) is legal only in `ready_for_processing` and `mapping_complete`. Any later state rejects with 409.

**Two writers**: Next.js owns transitions out of `mapping_complete`, into `processing`, and the `awaiting_review → finalising` compare-and-swap at Finalise dispatch. FastAPI/Lambda owns `combining`, `awaiting_review`, and the terminal `finalising → complete`/`failed` — writing them direct to the DB during the combiner and finaliser runs. The BulkUpload aggregate observes both. See [ADR-0005](./docs/adr/0005-async-bulk-upload-finaliser.md).

At `awaiting_review`, **Finalise is gated** (not a new status — a precondition on the action): when classifier columns were mapped the user must acknowledge the classification-verification step, and when the file is **Multi-entry** they must confirm the **Building-part ordering**. See [ADR-0004](./docs/adr/0004-multi-entry-building-part-ordering.md).

See [ADR-0001](./docs/adr/0001-bulk-upload-state-machine.md) for the deliberate "not yet" decisions baked into this lifecycle.

## Relationships

- A **Portfolio** has many **BulkUploads**.
- A **BulkUpload** produces zero or more **Properties** when finalised.
- A **BulkUpload** has at most one **Task** (the orchestration handle for the FastAPI pipeline run); a Task has many **SubTasks** (one per pipeline stage: address matching, combiner).
- A **Portfolio** has many **VocabularyMappings** — one row per `(category, description)` it has ever encountered across all its BulkUploads. See [ADR-0002](./docs/adr/0002-landlord-override-vocabulary.md).
- A **Recommendation** belongs to exactly one **Plan**. Denormalised onto `recommendation.plan_id`; the `plan_recommendations` join table is being retired.
- A **Recommendation** has at most one **Material**. Denormalised onto `recommendation.material_id` (+ `material_quantity`, `material_quantity_unit`, `material_depth`). Historically (pre-~2023) a recommendation could carry multiple materials; ~128 such legacy rows were reconciled to one each on 2026-06-07. The cardinality guard in the backfill enforces this going forward.

### Baseline performance

**Lodged performance**:
The SAP score, EPC band, CO₂ emissions, and primary energy intensity as submitted to the government EPC register. Ground truth from the register; never modified.
_Avoid_: original performance, registered performance

**Effective performance**:
The SAP score (and associated metrics) that the modelling engine actually uses as its baseline. Usually equals Lodged performance, but differs when a Landlord override or data-quality issue makes the lodged certificate unreliable — triggering a Rebaseline.
_Avoid_: current performance, adjusted performance

**Rebaseline**:
The act of substituting a corrected set of performance metrics in place of the Lodged values. Recorded on `property_baseline_performance` with a `rebaseline_reason` enum value: `none`, `pre_sap10`, `physical_state_changed`, or `both`.
_Avoid_: override, adjustment, correction

**EPC provenance** (`epc_property.source`):
Whether a property's EPC picture is a real certificate or a gap-fill. `lodged` = a real public/landlord EPC exists; `predicted` = no certificate, so the EPC was estimated from nearby properties (EPC Prediction gap-fill). Independent of Rebaseline: a `lodged` property may still be rebaselined, and a `predicted` property still carries Lodged-performance figures (mirrored estimates), so the presence of `lodged_*` columns does **not** imply a real certificate — only `source = lodged` does.
_Avoid_: estimated EPC (reserve "estimated" for the UI signal), source EPC

**Provenance signal** (UI):
What the user is told about an EPC's trustworthiness, derived from EPC provenance and Rebaseline together: **Estimated** (`source = predicted` — dominant; "estimated based on nearby homes") › **Re-modelled** (`source = lodged AND rebaseline_reason != none` — effective diverged from the lodged certificate under SAP 10) › **none** (`source = lodged AND rebaseline_reason = none` — effective equals lodged). Estimated always wins when both could apply.
_Avoid_: estimation notification, banner (those are component names)

## Example dialogue

> **Dev:** "If the **Combiner** finishes but the user hasn't clicked Finalise, what does the user see?"
> **Domain expert:** "The BulkUpload sits in `awaiting_review`. The frontend polls and shows a 'review and confirm' button. Nothing's been written to **Properties** yet."
>
> **Dev:** "And if **Finalise** runs and 30% of rows have no **UPRN**?"
> **Domain expert:** "Those still get imported as **Properties** — just without a UPRN — and the BulkUpload moves to `complete`. Manual cleanup happens later in the property table."
>
> _(Planned change — v3 / [ADR-0006](./docs/adr/0006-property-overrides-join-and-no-uprn-defer.md): no-UPRN rows will move to a separate staging table to be re-matched, so `property` holds only matched rows. v2 does **not** change this yet — and v2 writes **Property overrides** only for the UPRN-matched rows.)_

## Flagged ambiguities

- The UI surfaces labelled **"Current EPC"** and **"Current Efficiency State"** (property table, building-passport card) show **Effective performance**, not Lodged — despite the glossary advising against "current performance" for Effective. The label is a product choice ("current" = the property's true present-day modelled state); the underlying datum is still **Effective performance**. The **"Lodged EPC"** column/badge is the only surface showing **Lodged performance**, and only when a real certificate exists (`source = lodged`).
- "Upload" is used in the codebase to mean both the file-on-S3 and the BulkUpload row. We standardise on **BulkUpload** for the row; the file is just "the source file."
- "Onboarding" appears in some route paths (`bulk_onboarding_inputs/...`) but isn't part of this glossary — we use **BulkUpload** end-to-end.