mirror of
https://github.com/Hestia-Homes/Model.git
synced 2026-06-08 11:17:27 +00:00
Merge 8d6c770da8 into c98fc8452f
This commit is contained in:
KhalimCK
GitHub
commit
0cdace2a83
7 changed files with 1040 additions and 82 deletions
14
CLAUDE.md
14
CLAUDE.md
|
|
@ -36,28 +36,26 @@ Five Claude Code skills are installed in this repo's dev container. Each maps to
|
|||
|-------|--------|-------------|
|
||||
| **grill-me** | `/grill-me` | Before implementing — stress-tests a design through sequential questioning |
|
||||
| **to-prd** | `/to-prd` | After a planning conversation — formalises context into a GitHub issue PRD |
|
||||
| **ubiquitous-language** | `/ubiquitous-language` | When domain terms are drifting or ambiguous — builds/updates `UBIQUITOUS_LANGUAGE.md` |
|
||||
| **grill-with-docs** | `/grill-with-docs` | When domain terms are drifting or new concepts are landing — challenges plans against `CONTEXT.md`, sharpens terminology inline, and writes ADRs for load-bearing decisions in `docs/adr/`. Replaces the older `ubiquitous-language` skill. |
|
||||
| **tdd** | `/tdd` | During implementation — enforces vertical-slice TDD (one test → one impl → repeat) |
|
||||
| **improve-codebase-architecture** | `/improve-codebase-architecture` | During refactoring — surfaces shallow modules and proposes deepening opportunities |
|
||||
|
||||
Domain glossary lives at [CONTEXT.md](./CONTEXT.md); load-bearing decisions live at [docs/adr/](./docs/adr/). The legacy [UBIQUITOUS_LANGUAGE.md](./UBIQUITOUS_LANGUAGE.md) is a redirect.
|
||||
|
||||
### Typical session chains
|
||||
|
||||
**Feature planning:**
|
||||
`/grill-me` → `/to-prd` → `/ubiquitous-language`
|
||||
`/grill-me` → `/to-prd` → `/grill-with-docs`
|
||||
|
||||
**Implementation:**
|
||||
`/tdd` (+ `/grill-me` if a design fork appears mid-session)
|
||||
|
||||
**Refactoring:**
|
||||
`/improve-codebase-architecture` → `/grill-me` → `/tdd` → `/ubiquitous-language`
|
||||
`/improve-codebase-architecture` → `/grill-me` → `/tdd` → `/grill-with-docs`
|
||||
|
||||
### First time setting up?
|
||||
|
||||
New containers install all skills automatically via the Dockerfile. If you're in an existing container, run:
|
||||
|
||||
```bash
|
||||
bash .devcontainer/backend/install-claude-skills.sh
|
||||
```
|
||||
Skills are installed automatically when the dev container is built, via the postCreate step that pulls from `Hestia-Homes/agentic-toolkit` (see `.devcontainer/backend/Dockerfile`). If an existing container is missing skills, rebuild the dev container.
|
||||
|
||||
## Type Safety
|
||||
|
||||
|
|
|
|||
212
CONTEXT.md
Normal file
212
CONTEXT.md
Normal file
|
|
@ -0,0 +1,212 @@
|
|||
# Ara
|
||||
|
||||
The Domna product for domestic retrofit modelling: ingests open-source EPC data, lets users correct or supersede it with their own surveys, and produces optimised retrofit packages for each property in a portfolio.
|
||||
|
||||
## Language
|
||||
|
||||
### Product
|
||||
|
||||
**Ara**:
|
||||
The Domna product. Latin for "the altar"; named under Domna's classical-naming convention. Covers both the modelling product and the backend that powers it.
|
||||
_Avoid_: ARA (acronym style), v2 backend, the new backend
|
||||
|
||||
**Domna**:
|
||||
The company. Roman name; sibling to Ara in the same naming convention.
|
||||
|
||||
### Energy Performance Certificates
|
||||
|
||||
**EPC**:
|
||||
An Energy Performance Certificate — a government-issued document rating a dwelling's energy efficiency from A (best) to G (worst).
|
||||
_Avoid_: energy certificate, energy report
|
||||
|
||||
**Certificate Number**:
|
||||
The unique identifier assigned to an EPC by the government registry.
|
||||
_Avoid_: cert number, EPC ID
|
||||
|
||||
**Registration Date**:
|
||||
The date an EPC was lodged with the government register; used to identify the most recent certificate for a property.
|
||||
_Avoid_: assessment date, submission date
|
||||
|
||||
**EPC Band**:
|
||||
A single letter A–G representing a property's current or potential energy efficiency rating.
|
||||
_Avoid_: energy rating, EPC grade, EPC score
|
||||
|
||||
**Schema Type**:
|
||||
The versioned RdSAP or SAP schema that describes the structure of an EPC's raw data (e.g. `RdSAP-Schema-21.0.1`).
|
||||
_Avoid_: schema version, EPC format
|
||||
|
||||
**Domestic Certificate**:
|
||||
An EPC issued for a residential dwelling, as opposed to a commercial one.
|
||||
_Avoid_: residential EPC, home EPC
|
||||
|
||||
### Properties and addresses
|
||||
|
||||
**Property**:
|
||||
The Ara domain aggregate representing a single dwelling under modelling: its identity, source data, enrichments, and modelling outputs.
|
||||
_Avoid_: dwelling, unit, home, asset
|
||||
|
||||
**Properties**:
|
||||
A first-class collection of Property objects; the unit of bulk operation in services.
|
||||
_Avoid_: property list, batch (used for SQS chunks)
|
||||
|
||||
**UPRN**:
|
||||
Unique Property Reference Number — the government-issued permanent identifier for a physical address in the UK.
|
||||
_Avoid_: property ID, address ID, code
|
||||
|
||||
**Postcode**:
|
||||
A UK postal code used to group nearby addresses; the primary search key for finding EPC records.
|
||||
_Avoid_: zip code, postal code
|
||||
|
||||
**User Address**:
|
||||
A free-text address string provided by a user or imported from a customer dataset, before any normalisation or matching.
|
||||
_Avoid_: user input, raw address, user_inputed_address
|
||||
|
||||
**Comparable Properties**:
|
||||
The reference cohort matched to a target Property by both geographic proximity (postcode prefix / UPRN range) and physical similarity (property type, built form, age band); used by the EPC Prediction Service for gap-filling and anomaly detection.
|
||||
_Avoid_: neighbours, similar properties, peer set
|
||||
|
||||
### Source data
|
||||
|
||||
**Site Notes**:
|
||||
The full-coverage record produced by a Domna survey of a single Property; carries every EPC field the modelling pipeline requires, and when present supersedes the public EPC for that Property — except when the public EPC is newer.
|
||||
_Avoid_: energy assessment, site survey, field survey, Domna survey, Hestia survey
|
||||
|
||||
**Landlord Overrides**:
|
||||
Property data supplied by a landlord that may correct or supplement the public EPC for a single Property; triggers Rebaselining when applied; not applicable when Site Notes are present.
|
||||
_Avoid_: patches (deprecated), corrections, manual EPC, edits
|
||||
|
||||
### Modelling
|
||||
|
||||
**Effective EPC**:
|
||||
The EpcPropertyData scored by the modelling pipeline for a single Property, derived from either Site Notes alone or the public EPC with Landlord Overrides applied; carries source-derived physical fields and originally recorded performance values, with model-rebaselined performance held separately in Baseline Performance.
|
||||
_Avoid_: modelling EPC, working EPC, resolved EPC, derived EPC
|
||||
|
||||
**Rebaselining**:
|
||||
Re-predicting a Property's SAP, carbon emissions, and heat demand via ML when its Effective EPC's physical state diverges from the originally lodged public EPC (because Site Notes or Landlord Overrides have changed walls / heating / windows / etc.). Does not include kWh — that is always derived deterministically.
|
||||
_Avoid_: re-scoring, re-prediction, performance recomputation
|
||||
|
||||
**Baseline Performance**:
|
||||
A Property's current performance values — SAP, carbon emissions, heat demand, annual kWh, fuel split, bills — held against the Effective EPC. SAP / carbon / heat come directly from the Effective EPC's recorded values when no override applies, or from Rebaselining when an override changes physical state. Annual kWh and the fuel split are always derived deterministically by the EPC Energy Derivation Service.
|
||||
_Avoid_: baseline predictions, predicted baseline, rebaselined values
|
||||
|
||||
**EPC Energy Derivation**:
|
||||
The deterministic process that derives a Property's annual kWh, fuel split (gas / electric / other), and bills from the Effective EPC's energy fields — applying a UCL-style correction for known EPC over/under-prediction and deducing fuel type for heating + hot water from the SAP heating fields. No ML.
|
||||
_Avoid_: kWh prediction, baseline kWh, energy estimation
|
||||
|
||||
**EPC Anomaly Flag**:
|
||||
A per-field indicator that a Property's value for an EPC field differs significantly from Comparable Properties; advisory only — surfaces in the UI to prompt user review, does not block modelling.
|
||||
_Avoid_: outlier, mismatch, divergence flag
|
||||
|
||||
### Outputs
|
||||
|
||||
**Scenario**:
|
||||
A named portfolio-level container for a single modelling run, capturing the goal (e.g. Increasing EPC), budget, exclusions, and housing type; holds many Plans.
|
||||
_Avoid_: project, batch, run-set
|
||||
|
||||
**Plan**:
|
||||
The per-Property output of a single modelling run; belongs to one Scenario and carries the Property's full Recommendation list, Optimised Package, and post-retrofit predictions.
|
||||
_Avoid_: recommendation set, output, result
|
||||
|
||||
**Recommendation**:
|
||||
A single proposed retrofit measure for a Property, with its cost, SAP impact, kWh savings, carbon savings, and parts list.
|
||||
_Avoid_: suggestion, option
|
||||
|
||||
**Optimised Package**:
|
||||
The subset of a Property's Recommendations selected by the Optimiser Service for installation, chosen to satisfy the Scenario's goal subject to budget.
|
||||
_Avoid_: selected measures, default measures, optimal solution, recommended bundle
|
||||
|
||||
**Measure Type**:
|
||||
The catalogue classification of a retrofit measure (e.g. `solar_pv`, `loft_insulation`, `ashp`); one or more Recommendations reference the same Measure Type with property-specific cost and impact.
|
||||
_Avoid_: measure (ambiguous), category
|
||||
|
||||
### Address matching
|
||||
|
||||
**Lexiscore**:
|
||||
A similarity score in [0, 1] between a User Address and a candidate EPC address; combines token overlap and character-level similarity.
|
||||
_Avoid_: score, match score, similarity
|
||||
|
||||
**Lexirank**:
|
||||
Dense rank of candidates sorted by Lexiscore descending; rank 1 = best match.
|
||||
_Avoid_: rank, position
|
||||
|
||||
**UPRN Candidate**:
|
||||
An EPC Search Result that is a plausible match for a given User Address, before scoring decides the winner.
|
||||
_Avoid_: match candidate, result
|
||||
|
||||
**Score Threshold**:
|
||||
The minimum Lexiscore (currently 0.6) below which no match is returned even if a candidate exists.
|
||||
_Avoid_: minimum score, cutoff
|
||||
|
||||
**Ambiguous Match**:
|
||||
A matching outcome where two or more candidates share Lexirank 1, making it impossible to select a unique winner.
|
||||
_Avoid_: tie, draw, duplicate
|
||||
|
||||
**Best Match**:
|
||||
The single UPRN Candidate with Lexirank 1 that meets or exceeds the Score Threshold.
|
||||
_Avoid_: winner, top result
|
||||
|
||||
### API and integration
|
||||
|
||||
**EPC Search Result**:
|
||||
A lightweight record returned by the government domestic search endpoint — address lines, postcode, UPRN, band, and certificate number, but not full certificate data.
|
||||
_Avoid_: search row, EPC row, result
|
||||
|
||||
**EPC Property Data**:
|
||||
The fully mapped domain object produced after fetching and parsing a complete EPC certificate; the schema the modelling pipeline operates against.
|
||||
_Avoid_: EPC data, certificate data, parsed EPC
|
||||
|
||||
**Old EPC API**:
|
||||
The retired government API (`epc.opendatacommunities.org`) using HTTP Basic auth; decommissioned 30 May 2026.
|
||||
_Avoid_: legacy API
|
||||
|
||||
**New EPC API**:
|
||||
The replacement government API (`api.get-energy-performance-data.communities.gov.uk`) using Bearer Token auth.
|
||||
_Avoid_: new API, current API
|
||||
|
||||
**Bearer Token**:
|
||||
The auth credential required by the New EPC API; stored in the `EPC_AUTH_TOKEN` environment variable.
|
||||
_Avoid_: API key, auth token, secret
|
||||
|
||||
## Relationships
|
||||
|
||||
- A **Property** represents a single physical dwelling for modelling; identified by `(portfolio_id, UPRN)` or `(portfolio_id, landlord_property_id)`.
|
||||
- A **Property** has zero or more **EPCs** across time, exactly one **Effective EPC**, zero or one set of **Site Notes**, and zero or one set of **Landlord Overrides**.
|
||||
- An **EPC** belongs to exactly one **Property** and has one **Certificate Number**.
|
||||
- An **EPC** carries an **EPC Band** and is identifiable by its **Registration Date**; the most recent one is the current.
|
||||
- A **UPRN** identifies a physical dwelling permanently; it does not change when the property changes owner — but each portfolio gets its own **Property** keyed against it.
|
||||
- When a **Property** has both **Site Notes** and a public **EPC**, the newer of the two derives the **Effective EPC**. **Landlord Overrides** apply only when the **EPC** is the source — never when **Site Notes** are.
|
||||
- **Rebaselining** contributes the SAP / carbon / heat parts of **Baseline Performance** when the **Effective EPC** physical state diverges from the originally lodged EPC. **EPC Energy Derivation** contributes the kWh / fuel split / bills parts unconditionally for every Property.
|
||||
- The **EPC Prediction Service** uses **Comparable Properties** for both gap-filling and producing **EPC Anomaly Flags**.
|
||||
- A **Scenario** contains many **Plans** (one per Property). A **Plan** carries many **Recommendations**; the **Optimised Package** is the subset selected for installation.
|
||||
- A **Recommendation** references one **Measure Type** and carries property-specific cost and impact.
|
||||
- **Address Matching** uses a **User Address** and **Postcode** to find a **UPRN** by scoring **UPRN Candidates** from an EPC search. A **Lexirank** of 1 with no **Ambiguous Match** and a **Lexiscore** ≥ the **Score Threshold** produces a **Best Match**.
|
||||
|
||||
## Example dialogue
|
||||
|
||||
> **Dev:** "A landlord uploads a corrected boiler for one of their properties. What happens?"
|
||||
>
|
||||
> **Domain expert:** "That's a **Landlord Override** on the heating fields. Save it against the **Property**. The **Effective EPC** has changed, so **Rebaselining** runs to re-predict SAP / carbon / heat, and **EPC Energy Derivation** re-runs to update kWh / bills based on the new fuel deduction. With fresh **Baseline Performance** we regenerate **Recommendations**."
|
||||
|
||||
> **Dev:** "What if the same Property also has Site Notes?"
|
||||
>
|
||||
> **Domain expert:** "**Site Notes** supersede the public **EPC**, so **Landlord Overrides** don't apply. We model from the **Site Notes** version of the **Effective EPC**. If the public **EPC** is newer than the **Site Notes**, that's the one exception — we use the newer one."
|
||||
|
||||
> **Dev:** "After modelling we end up with a list of measures. Which ones get installed?"
|
||||
>
|
||||
> **Domain expert:** "The **Optimiser Service** picks the **Optimised Package** — a subset of **Recommendations** that hits the **Scenario** goal within budget. The rest stay in the **Plan** as alternatives the user can swap in."
|
||||
|
||||
> **Dev:** "I'm looking at a property where the EPC says cavity walls but every other house on the street has solid. Is that a bug?"
|
||||
>
|
||||
> **Domain expert:** "That's an **EPC Anomaly Flag**. We compute it against the **Comparable Properties** for that postcode. It's advisory — the UI surfaces it and the landlord can apply a **Landlord Override** if it's wrong."
|
||||
|
||||
## Flagged ambiguities
|
||||
|
||||
- **"property"** was historically warned against in favour of "dwelling"; that has been inverted. **Property** is now canonical for the Ara domain aggregate. Legacy code still uses "dwelling" in places — treat as alias.
|
||||
- **"energy assessment"** in the existing codebase (`energy_assessment_functions`, `energy_assessments_by_uprn`) refers to what is now canonically called **Site Notes**. New code uses **Site Notes**.
|
||||
- **"patch"** / `patch_epc` in the existing codebase has been merged into **Landlord Overrides**; the original concept is deprecated.
|
||||
- **"already_installed measures"** in the existing codebase is likely subsumed by **Landlord Overrides** ("we have a heat pump now" → override the heating fields). Final call deferred to implementation.
|
||||
- **"address"** appears as both the raw **User Address** (free-text) and a structured field on an **EPC Search Result** (normalised lines). Always qualify: "user address" vs "EPC address" or "address line 1".
|
||||
- **"score"** is used for `AddressMatch.score()` output, the `lexiscore` column, and informally. Prefer **Lexiscore** in domain discussions; reserve "score" for method-level code comments.
|
||||
- **"user_inputed_address"** in `backend/address2UPRN/main.py` is a misspelling and a synonym for **User Address** — the canonical term. New code should use `user_address`.
|
||||
- **"EPC"** is overloaded as both the document and the rating band letter. Use **EPC** for the document, **EPC Band** for the letter.
|
||||
- **"re-scoring"** has two meanings in the codebase — **Rebaselining** (re-predicting baseline performance after an EPC change) and post-optimisation measure re-prediction. Prefer **Rebaselining** for the former; for the latter, the **Optimiser Service** step does its own scoring without a special name.
|
||||
|
|
@ -1,78 +1,7 @@
|
|||
# Ubiquitous Language
|
||||
|
||||
Domain terminology glossary for this project. Generated and maintained by the `/ubiquitous-language` Claude Code skill.
|
||||
This file has been **superseded by [CONTEXT.md](./CONTEXT.md)**.
|
||||
|
||||
Invoke `/ubiquitous-language` in any session to extract new terms from the conversation, flag ambiguities, and update this file with canonical definitions.
|
||||
The project's domain glossary now lives at the repo root in `CONTEXT.md`, maintained by the `/grill-with-docs` skill (which replaced `/ubiquitous-language`).
|
||||
|
||||
---
|
||||
|
||||
## Energy Performance Certificates
|
||||
|
||||
| Term | Definition | Aliases to avoid |
|
||||
|------|------------|------------------|
|
||||
| **EPC** | An Energy Performance Certificate — a government-issued document rating a dwelling's energy efficiency from A (best) to G (worst). | "energy certificate", "energy report" |
|
||||
| **Certificate Number** | The unique identifier assigned to an EPC by the government registry. | "cert number", "EPC ID" |
|
||||
| **Registration Date** | The date an EPC was lodged with the government register; used to identify the most recent certificate for a property. | "assessment date", "submission date" |
|
||||
| **EPC Band** | A single letter A–G representing a property's current or potential energy efficiency rating. | "energy rating", "EPC grade", "EPC score" |
|
||||
| **Schema Type** | The versioned RdSAP or SAP schema that describes the structure of a certificate's raw data (e.g. `RdSAP-Schema-21.0.1`). | "schema version", "EPC format" |
|
||||
| **Domestic Certificate** | An EPC issued for a residential dwelling, as opposed to a commercial one. | "residential EPC", "home EPC" |
|
||||
|
||||
## Properties and Addresses
|
||||
|
||||
| Term | Definition | Aliases to avoid |
|
||||
|------|------------|------------------|
|
||||
| **UPRN** | Unique Property Reference Number — the government-issued permanent identifier for a physical address in the UK. | "property ID", "address ID", "code" |
|
||||
| **Postcode** | A UK postal code used to group nearby addresses; the primary search key for finding EPC records. | "zip code", "postal code" |
|
||||
| **User Address** | A free-text address string provided by a user or imported from a customer dataset, before any normalisation or matching. | "user input", "raw address", "user_inputed_address" |
|
||||
| **Dwelling** | A single residential unit that can hold an EPC — a house, flat, or maisonette. | "property", "unit", "home" |
|
||||
|
||||
## Address Matching
|
||||
|
||||
| Term | Definition | Aliases to avoid |
|
||||
|------|------------|------------------|
|
||||
| **Lexiscore** | A similarity score in [0, 1] between a user address and a candidate EPC address; combines token overlap and character-level similarity. | "score", "match score", "similarity" |
|
||||
| **Lexirank** | Dense rank of candidates sorted by lexiscore descending; rank 1 = best match. | "rank", "position" |
|
||||
| **UPRN Candidate** | An EPC search result that is a plausible match for a given user address, before scoring decides the winner. | "match candidate", "result" |
|
||||
| **Score Threshold** | The minimum lexiscore (currently 0.6) below which no match is returned even if a candidate exists. | "minimum score", "cutoff" |
|
||||
| **Ambiguous Match** | A matching outcome where two or more candidates share lexirank 1, making it impossible to select a unique winner. | "tie", "draw", "duplicate" |
|
||||
| **Best Match** | The single UPRN candidate with lexirank 1 that meets or exceeds the score threshold. | "winner", "top result" |
|
||||
|
||||
## API and Integration
|
||||
|
||||
| Term | Definition | Aliases to avoid |
|
||||
|------|------------|------------------|
|
||||
| **EPC Search Result** | A lightweight record returned by the government domestic search endpoint — contains address lines, postcode, UPRN, band, and certificate number but not the full certificate data. | "search row", "EPC row", "result" |
|
||||
| **EPC Property Data** | The fully mapped domain object produced after fetching and parsing a complete EPC certificate. | "EPC data", "certificate data", "parsed EPC" |
|
||||
| **Old EPC API** | The retired government API (`epc.opendatacommunities.org`) using HTTP Basic auth; decommissioned May 2026. | "legacy API" |
|
||||
| **New EPC API** | The replacement government API (`api.get-energy-performance-data.communities.gov.uk`) using Bearer token auth. | "new API", "current API" |
|
||||
| **Bearer Token** | The auth credential required by the new EPC API; stored in the `EPC_AUTH_TOKEN` environment variable. | "API key", "auth token", "secret" |
|
||||
|
||||
## Relationships
|
||||
|
||||
- An **EPC** belongs to exactly one **Dwelling** and has one **Certificate Number**.
|
||||
- A **Dwelling** may have multiple **EPCs** across time; the one with the most recent **Registration Date** is the current one.
|
||||
- A **UPRN** identifies a **Dwelling** permanently; it does not change when the property changes owner.
|
||||
- An **EPC Search Result** is a summary; it points to a full **EPC** via its **Certificate Number**.
|
||||
- **Address Matching** uses a **User Address** and **Postcode** to find a **UPRN** by scoring **UPRN Candidates** from an EPC search.
|
||||
- A **Lexirank** of 1 with no **Ambiguous Match** and a **Lexiscore** ≥ the **Score Threshold** produces a **Best Match**.
|
||||
|
||||
## Example dialogue
|
||||
|
||||
> **Dev:** "We have a user address and postcode. How do we find the UPRN?"
|
||||
|
||||
> **Domain expert:** "Search the **New EPC API** by **Postcode** — you get back a list of **EPC Search Results** for that area. Each one has an address and a **UPRN**. Score each against the **User Address** using the **Lexiscore**. If the top **UPRN Candidate** scores above the **Score Threshold** and there's no **Ambiguous Match**, that's your **Best Match**."
|
||||
|
||||
> **Dev:** "What if two results share the same address line 1?"
|
||||
|
||||
> **Domain expert:** "That's an **Ambiguous Match** — two candidates at **Lexirank** 1. Fall back to scoring on the full address using all address lines joined together. If that still ties, return nothing."
|
||||
|
||||
> **Dev:** "Once we have the best match, do we use the UPRN or fetch the full EPC?"
|
||||
|
||||
> **Domain expert:** "Depends on what you need. The **EPC Search Result** gives you the **EPC Band** and **Certificate Number**. If you need energy efficiency detail, use the **Certificate Number** to fetch the full **EPC Property Data**."
|
||||
|
||||
## Flagged ambiguities
|
||||
|
||||
- **"address"** appears as both the raw **User Address** (free-text from customer data) and a structured field on an **EPC Search Result** (normalised address lines). Always qualify: "user address" vs "EPC address" or "address line 1".
|
||||
- **"score"** is used for the `AddressMatch.score()` function output, the `lexiscore` DataFrame column, and informally in conversation. Prefer **Lexiscore** in domain discussions; reserve "score" for method-level code comments.
|
||||
- **"user_inputed_address"** in `backend/address2UPRN/main.py` is a misspelling and a synonym for **User Address** — the canonical term. New code should use `user_address`.
|
||||
- **"EPC"** is overloaded as both the document (an Energy Performance Certificate) and the rating band letter. Use **EPC** for the document and **EPC Band** for the letter.
|
||||
If you arrived here from a link in `CLAUDE.md` or older docs, follow the link above. This file is kept only to preserve git history and may be removed once internal references are updated.
|
||||
|
|
|
|||
782
ara_backend_design.md
Normal file
782
ara_backend_design.md
Normal file
|
|
@ -0,0 +1,782 @@
|
|||
# ARA Backend Redesign — Design PRD
|
||||
|
||||
**Status**: Draft for team review
|
||||
**Author**: Khalim Conn-Kowlessar (with Claude grill session)
|
||||
**Branch**: `ara-backend-design-prd`
|
||||
**Scope**: Service architecture + domain model + contracts for the new modelling backend. Linked sub-PRDs cover ML training pipeline, DB schema migration, and historical EPC re-mapping.
|
||||
|
||||
---
|
||||
|
||||
## 1. Context
|
||||
|
||||
### 1.1 The forcing function
|
||||
|
||||
The current modelling backend (`backend/engine/engine.py` — `model_engine`, 1331 LOC) was built as an MVP. It is:
|
||||
|
||||
- **Tightly coupled** to a specific gov EPC API that is being **decommissioned on 30 May 2026** (~17 days from today).
|
||||
- **A monolith** — one async function reaches into DB modules, HTTP clients, ML lambdas, S3, and queue infrastructure directly.
|
||||
- **Bottlenecked on a single person** — Khalim is the only contributor able to safely modify the engine because no one else can predict the blast radius of a change.
|
||||
- **Already returning erroneous data** from the old API (clients are aware). The replacement API is partially built (`backend/epc_client/epc_client_service.py`) on the current feature branch.
|
||||
|
||||
### 1.2 What needs to change
|
||||
|
||||
Beyond just swapping API clients, this is the moment to **rebuild the backend into a production-grade, contribute-able codebase**, with:
|
||||
|
||||
- A clear domain model rooted in the new EPC schema (`EpcPropertyData`).
|
||||
- Service boundaries that other team members can read, fix, and extend without needing the entire mental model.
|
||||
- Repository-mediated persistence so business logic can be tested without spinning up a database.
|
||||
- A separation between **data fetching** (slow, IO-heavy, external) and **modelling** (deterministic, fast, internal).
|
||||
- Baseline kWh and bills derived deterministically from the Effective EPC (SAP physics + UCL correction + per-fuel rates from a refreshable repo) rather than from the EPC's stale cost fields or from an ML kWh prediction.
|
||||
|
||||
### 1.3 Out of scope for this PRD
|
||||
|
||||
These ship as **linked sub-PRDs**:
|
||||
|
||||
- **Sub-PRD (ii) — ML training pipeline** (autogluon repo + parquet generation in this repo + scoring model retraining for the new EPC schema)
|
||||
- **Sub-PRD (iii) — DB schema migration** (new tables: `site_notes`, `landlord_overrides`, EPC cache, parallel write strategy)
|
||||
- **Sub-PRD (iv) — Historical EPC re-mapping** (one-off + ongoing batch job: legacy stored EPCs → new `EpcPropertyData` shape)
|
||||
|
||||
The contracts this PRD defines are the inputs each sub-PRD consumes.
|
||||
|
||||
---
|
||||
|
||||
## 2. Goals and non-goals
|
||||
|
||||
### 2.1 Goals
|
||||
|
||||
1. **Survive the 30 May API shutdown** — even if it means a brief degraded window, modelling continues to function against the new gov EPC API.
|
||||
2. **Decouple data fetching from modelling** — modelling never makes external HTTP calls; it reads everything from repositories.
|
||||
3. **Make every service unit-testable against fakes** — no test needs a real DB, a real gov API, or a real ML lambda to verify business logic.
|
||||
4. **Establish a single `Property` aggregate root** as the domain centrepiece; all 9 modelling concerns are slices of one aggregate.
|
||||
5. **Versioned ML data contract** — the EPC-to-features transform is the single shared artifact between this repo and the autogluon repo.
|
||||
6. **Per-property UI surfaces** — fetched data can be shown to users for review and override **before** modelling runs; modelling is triggered separately. This will enable a landlord facing version of the product where we fetch the open data, present back to the user for review and then perform the modelling.
|
||||
|
||||
### 2.2 Non-goals
|
||||
|
||||
- Multi-region deploy, GDPR-class data minimisation work, or compliance reporting — separate workstreams.
|
||||
- Replacement of the front-end. The new APIs preserve enough of the existing response shape that the FE migrates incrementally.
|
||||
- Removing pandas. The ML transform output is a parquet-friendly DataFrame-like shape; that stays.
|
||||
- A workflow engine (Prefect / Temporal / Airflow). Coordinator-class orchestration plus the existing SQS-fanout pattern is sufficient at the scale we serve.
|
||||
|
||||
---
|
||||
|
||||
## 3. Cutover plan
|
||||
|
||||
Forced cut-over, driven by the 30 May deadline. There is no strangler period because the Old EPC API death takes `model_engine` with it.
|
||||
|
||||
### 3.1 Phase 0 — Status quo (now → 30 May)
|
||||
|
||||
- `model_engine` keeps running against the Old EPC API for as long as it works.
|
||||
- Build of the 9 new services starts **this week**, in parallel to the old engine continuing to serve traffic.
|
||||
- The new `ara/` package lives alongside `backend/` but is not yet wired into any production endpoint.
|
||||
- Goal: keep the lights on until the API dies; start the build immediately so the dark period is short.
|
||||
|
||||
### 3.2 Phase 1 — Forced cut-over (30 May onwards)
|
||||
|
||||
- On 30 May the Old EPC API dies; `model_engine` ceases to function for any new modelling run.
|
||||
- Some downtime is expected and accepted. Clients are aware.
|
||||
- Modelling resumes when the new pipeline is ready end-to-end. Remains to be decided if we have a per-portfolio flag, purely for the front end to reference old tables where necessary. No parallel pipelines, no traffic split — the new pipeline is the only pipeline.
|
||||
- **Calico** and **Hyde** are the first live clients onto the new pipeline in June.
|
||||
- `model_engine`, `SearchEpc`, the legacy `Property`, and surrounding modules in `backend/` are deleted once the new pipeline is serving all traffic.
|
||||
|
||||
### 3.3 What is *not* done
|
||||
|
||||
- No strangler — there is nothing to strangle once the Old EPC API dies on 30 May.
|
||||
- No parallel-shadow run — would double compute and require diff tooling we don't have, while the old engine is already known to return bad data so diffs would be noise.
|
||||
- TBC per-portfolio feature flag. Without this, the cut-over is all-or-nothing. All old portfolios are broken.
|
||||
|
||||
---
|
||||
|
||||
## 4. Architecture overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Trigger endpoint(s) │
|
||||
│ (one or two — see §4.5; deferred decision) │
|
||||
└───────────┬──────────────────────────────────────────┬──────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────┐ ┌─────────────────┐
|
||||
│ IngestionPipe │ SQS, batches of N │ ModellingPipe │
|
||||
│ ----------- │ ◄─────────────────────│ ----------- │
|
||||
│ Fetchers run │ │ Reads via Repos │
|
||||
│ Persist via │ │ Calls Services │
|
||||
│ Repos │ │ ML predictions │
|
||||
└────────┬────────┘ └────────┬────────┘
|
||||
│ │
|
||||
└───────────────► Repos ◄─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ Postgres tables │
|
||||
│ (property, │
|
||||
│ epc_cache, │
|
||||
│ site_notes, │
|
||||
│ landlord_ │
|
||||
│ overrides, │
|
||||
│ plans, etc.) │
|
||||
└──────────────────┘
|
||||
|
||||
┌──────────────────────────┐
|
||||
│ RefreshOrchestrator │ triggers Ingestion → diff → conditionally Modelling
|
||||
└──────────────────────────┘
|
||||
```
|
||||
|
||||
### 4.1 Class taxonomy
|
||||
|
||||
Every class falls into exactly one of four roles:
|
||||
|
||||
| Role | Job | Examples |
|
||||
|------|-----|----------|
|
||||
| **Fetchers** | Call external APIs. Return raw response data. No DB. | `EpcClientService`, `GeospatialFetcher`, `SolarFetcher`, `SiteNotesIngester` |
|
||||
| **Repos** | Persist and load domain aggregates. SQL hidden inside. No external IO. | `PropertyRepo`, `EpcCacheRepo`, `SiteNotesRepo`, `LandlordOverridesRepo`, `RecommendationsRepo`, `GenericDataRepo`, `SubtaskRepo` |
|
||||
| **Services** | Business logic over domain objects. No external IO except via injected Fetchers / Repos. | `EpcRemappingService`, `EpcPredictionService`, `EpcEnergyDerivationService`, `KwhImpactService`, `ImpactPredictionService`, `RecommendationService`, `OptimiserService`, `FeatureBuilder`, `ResultsPersister` |
|
||||
| **Orchestrators** | Compose Fetchers + Services + Repos to produce an end-to-end result. The only place where step order is encoded. | `IngestionPipeline`, `ModellingPipeline`, `RefreshOrchestrator` |
|
||||
|
||||
This taxonomy is **strict**. A class that fetches *and* persists belongs in the Service layer and depends on a Fetcher + a Repo. No back-channels.
|
||||
|
||||
### 4.2 Two pipelines, one direction
|
||||
|
||||
Data flows one way only: **Ingestion → Repos → Modelling**.
|
||||
|
||||
- **Ingestion** writes; never calls Modelling.
|
||||
- **Modelling** reads; never calls Fetchers.
|
||||
|
||||
If Modelling needs fresh data, it returns "stale" and the caller decides whether to ingest first. This makes Modelling a pure function of repository state, which is the property that makes it reproducible, debuggable, and testable.
|
||||
|
||||
### 4.3 RefreshOrchestrator
|
||||
|
||||
Sits above both pipelines. Job:
|
||||
|
||||
1. Trigger `IngestionPipeline` for a portfolio.
|
||||
2. After ingestion completes, ask repos: "did anything change vs the last modelled snapshot?"
|
||||
3. If yes, trigger `ModellingPipeline`. If no, return early.
|
||||
|
||||
This avoids re-modelling 100k properties when only 200 had refreshed EPC data.
|
||||
|
||||
### 4.4 SQS fanout (preserved from current architecture)
|
||||
|
||||
The existing `trigger_plan_entrypoint` SQS-chunking pattern is kept. Both pipelines fan out per batch of ~30–100 properties (tuneable). Each consumer runs one batch end-to-end through the relevant pipeline.
|
||||
|
||||
UPRN partitioning: the trigger endpoint groups UPRNs by **locality** (postcode prefix / UPRN range) before chunking, so each batch maximises shared upstream fetches (one geospatial-range pull serves all 30 properties in the batch).
|
||||
|
||||
### 4.5 One endpoint for v1
|
||||
|
||||
For Phase 1 we ship **one trigger endpoint** that internally chains Ingestion → Modelling via `RefreshOrchestrator`. This matches the current FastAPI-fronted Lambda pattern (the FastAPI app in `services/<svc>/` is a thin entrypoint that invokes the modelling Lambda).
|
||||
|
||||
We can split into two endpoints later (refresh-only vs model-only) once a real workflow demands it — e.g. a Landlord-Override edit that should re-model without re-fetching open data. The class taxonomy and `RefreshOrchestrator` boundary allow this split without re-architecting.
|
||||
|
||||
### 4.6 Trigger contract
|
||||
|
||||
The trigger payload is reduced compared to today's `PlanTriggerRequest` ([backend/app/plan/schemas.py:98](../../backend/app/plan/schemas.py#L98)) — most of what's currently in the request body moves into the persisted `Scenario` aggregate.
|
||||
|
||||
```python
|
||||
class ModelTriggerRequest(BaseModel):
|
||||
portfolio_id: UUID
|
||||
property_ids: list[UUID] | S3Ref # inline up to ~10k, S3 ref above
|
||||
scenario_ids: list[UUID] # 1+; resolved + pinned to ScenarioSnapshot at fan-out
|
||||
task_id: UUID
|
||||
subtask_id: UUID # SQS state machine, preserved from today
|
||||
```
|
||||
|
||||
Everything that used to ride at the top level dies or moves:
|
||||
|
||||
- `goal`, `budget`, `goal_value`, `inclusions`, `exclusions`, `required_measures`, `enforce_fabric_first`, `scenario_name`, `housing_type` → into `Scenario` / `ScenarioPhase`.
|
||||
- `patches_file_path`, `already_installed_file_path`, `non_invasive_recommendations_file_path` → gone; Landlord Overrides covers all three.
|
||||
- `valuation_file_path` → gone; `ValuationService` derives it.
|
||||
- `ashp_cop`, `default_u_values` → `HeatingSystemAssumptionsRepo` / global config; not per-trigger.
|
||||
- `multi_plan` → gone; `scenario_ids: list[...]` handles N runs natively (one Plan per scenario per property).
|
||||
- `event_type`, `epc_certificate_number`, `lmk_key`, `file_format`, `sheet_name`, `index_start`/`index_end`, `file_type` → ingestion-side concerns; if needed, ride on a separate ingestion-trigger payload.
|
||||
|
||||
**Scenario snapshotting**: at fan-out time `RefreshOrchestrator` reads each requested `Scenario`, writes a `ScenarioSnapshot` keyed by `(task_id, scenario_id)`, and per-batch SQS messages reference the snapshot. Mid-run edits to the live `Scenario` do not affect an in-flight modelling job. Snapshots are read-only and can be garbage-collected after the task completes.
|
||||
|
||||
---
|
||||
|
||||
## 5. Domain model
|
||||
|
||||
### 5.1 Aggregate root: `Property`
|
||||
|
||||
`Property` is the centrepiece. Every service operates on one or more `Property` instances. Every repo writes one slice of `Property`. The aggregate carries all state for a single property's modelling run.
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class PropertyIdentity:
|
||||
portfolio_id: UUID
|
||||
uprn: Optional[int]
|
||||
landlord_property_id: Optional[str]
|
||||
address: AddressLines
|
||||
postcode: str
|
||||
|
||||
@dataclass
|
||||
class Property:
|
||||
identity: PropertyIdentity
|
||||
|
||||
# --- Source data — modelling path is determined by which of these are set ---
|
||||
epc: Optional[EpcPropertyData] # from gov API (or remapped historical)
|
||||
site_notes: Optional[SiteNotes] # our own survey; supersedes EPC when present
|
||||
landlord_overrides: Optional[LandlordOverrides] # sparse, only meaningful when epc set
|
||||
|
||||
# --- Enrichments ---
|
||||
geospatial: Optional[GeoSpatial]
|
||||
solar: Optional[SolarPotential]
|
||||
epc_anomaly_flags: Optional[EpcAnomalyFlags] # from EpcPredictionService vs neighbours
|
||||
|
||||
# --- Modelling outputs ---
|
||||
baseline_performance: Optional[BaselinePerformance] # carries lodged + effective pair; see §5.4
|
||||
recommendations: list[Recommendation]
|
||||
impact_predictions: Optional[ImpactPredictions]
|
||||
plans: list[Plan] # one per Scenario the property was modelled against
|
||||
|
||||
# --- Derived ---
|
||||
@property
|
||||
def source_path(self) -> Literal["site_notes", "epc_with_overlay"]: ...
|
||||
|
||||
@property
|
||||
def effective_epc(self) -> EpcPropertyData:
|
||||
"""The EPC the modelling pipeline actually scores against."""
|
||||
...
|
||||
```
|
||||
|
||||
### 5.2 `Properties` collection
|
||||
|
||||
A first-class iterable, so batch operations are obvious:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class Properties:
|
||||
items: list[Property]
|
||||
|
||||
def __iter__(self) -> Iterator[Property]: ...
|
||||
def __len__(self) -> int: ...
|
||||
def filter(self, pred: Callable[[Property], bool]) -> "Properties": ...
|
||||
def map(self, fn: Callable[[Property], Property]) -> "Properties": ...
|
||||
def with_landlord_overrides(self) -> "Properties": ...
|
||||
```
|
||||
|
||||
Services typically take and return `Properties`, not lists.
|
||||
|
||||
### 5.3 Other aggregates
|
||||
|
||||
| Aggregate | Owns | Repo |
|
||||
|---|---|---|
|
||||
| `Property` | property identity, epc, site_notes, landlord_overrides, enrichments, modelling results | `PropertyRepo` |
|
||||
| `Plan` | per-property modelling output for one Scenario: ordered `phases: list[PlanPhase]`, each carrying its `OptimisedPackage`, ending state snapshot, and rolled-over options | `RecommendationsRepo` |
|
||||
| `Scenario` | portfolio-wide scenario metadata (goal, budget, exclusions, housing type) plus ordered `phases: list[ScenarioPhase]`; each phase carries `measure_types_allowed`, phase budget, phase target | `RecommendationsRepo` |
|
||||
| `ScenarioSnapshot` | frozen copy of a `Scenario` pinned at trigger time, keyed by `(task_id, scenario_id)`, so mid-run scenario edits don't affect an in-flight modelling job | `RecommendationsRepo` |
|
||||
| `Subtask` / `Task` | SQS fanout state | `SubtaskRepo` |
|
||||
| `EpcCache` | gov-API responses keyed by UPRN, with freshness/TTL | `EpcCacheRepo` |
|
||||
| `GenericData` | UPRN-range geospatial, postcode lookups, shared static data | `GenericDataRepo` |
|
||||
| `FuelRates` | time-versioned, region-aware per-fuel rates (pence/kWh), standing charges, SEG export rate, calorific values | `FuelRatesRepo` |
|
||||
| `CarbonFactors` | time-versioned per-fuel CO2 emission factors (kgCO2e/kWh); Defra publishes annually | `CarbonFactorsRepo` |
|
||||
| `HeatingSystemAssumptions` | boiler efficiency tables, ASHP/GSHP COPs, solar-thermal coverage proportion; per-property physical assumptions, not fuel-market data | `HeatingSystemAssumptionsRepo` |
|
||||
|
||||
Aggregates are loaded **whole** — never half a `Property`. If a slice is too large to load eagerly (e.g. recommendation history), it lives in a separate aggregate.
|
||||
|
||||
A single-phase Scenario is `phases: [<one ScenarioPhase>]` with all measure types allowed and the full budget on it — no special-case path through the pipeline.
|
||||
|
||||
### 5.4 `BaselinePerformance` carries lodged + effective
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class BaselinePerformance:
|
||||
# As-lodged: unmodified EPC fields (or Site Notes' recorded values where Site Notes are the source).
|
||||
lodged_sap: int
|
||||
lodged_band: Epc
|
||||
lodged_carbon: float
|
||||
lodged_heat_demand: float
|
||||
|
||||
# Effective: what the modelling pipeline actually scored against.
|
||||
# Equals lodged when neither rebaselining trigger fires; equals ML output when rebaselined.
|
||||
effective_sap: int
|
||||
effective_band: Epc
|
||||
effective_carbon: float
|
||||
effective_heat_demand: float
|
||||
|
||||
# kWh / fuel split / bills — always derived deterministically from the Effective EPC by
|
||||
# EpcEnergyDerivationService (SAP physics + UCL correction + FuelRates lookup).
|
||||
# Lodged kWh / bills are not stored separately — the EPC's cost fields are stale by design.
|
||||
annual_kwh: float
|
||||
fuel_split: dict[Fuel, float]
|
||||
annual_bills: dict[Fuel, float]
|
||||
|
||||
rebaselined: bool
|
||||
rebaseline_reason: Optional[Literal["pre_sap10", "physical_state_changed", "both"]]
|
||||
```
|
||||
|
||||
The pair lets the FE show "lodged rating vs SAP10-equivalent rebaselined rating" side by side without a separate query. Both fields are always populated; when no rebaselining trigger fires, `effective_*` equals `lodged_*`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Source-of-truth and overlay precedence
|
||||
|
||||
There are exactly **two modelling paths**. The `Property.source_path` property selects.
|
||||
|
||||
### 6.1 Path 1 — Site notes
|
||||
|
||||
If a `Property` has `site_notes` and they are newer than any available EPC (or no EPC exists), site notes are the **complete** source of truth:
|
||||
|
||||
- `effective_epc` = `site_notes.to_epc_property_data()`.
|
||||
- EPC fields not covered by site notes — **none expected**. Site notes are committed to being a full-coverage survey. Treat any gap as a survey-quality bug, not a fallback signal.
|
||||
- `LandlordOverrides` are not applicable in Path 1 (the survey supersedes).
|
||||
|
||||
### 6.2 Path 2 — EPC with landlord overlay
|
||||
|
||||
If a `Property` has no site notes (or the EPC is newer):
|
||||
|
||||
- `effective_epc` = `epc` with `landlord_overrides` applied as a sparse field-level overlay (`landlord > epc`).
|
||||
- `LandlordOverrides` are sparse: each row represents one corrected field. Schema TBD at implementation time; assume flat input via Excel/CSV for v1, with a flag to revisit shape after first customer onboarding.
|
||||
|
||||
### 6.3 Recency tie-break
|
||||
|
||||
When a property has **both** site notes and a public EPC, the newer of the two wins. Rationale: a recent EPC may reflect retrofit work done after our survey; conversely a recent survey reflects on-site observations the EPC cannot capture.
|
||||
|
||||
This tie-break is implemented in `Property.source_path` and may be tuned later (e.g. always prefer surveys regardless of date, or per-portfolio policy).
|
||||
|
||||
### 6.4 Rebaselining trigger
|
||||
|
||||
ML re-predicts SAP / carbon / heat when **either** of these holds:
|
||||
|
||||
1. **Pre-SAP10 schema** — `effective_epc.sap_version < 10.0`. The EPC was rated under SAP 2012 (or earlier) and we want a SAP10-equivalent baseline so all properties are scored against the same model version. Canonical signal is the `sap_version: float` field; fall back to `schema_type` string, then to `lodgement_date` if both are absent. Site Notes are assumed SAP10 by construction (PasHub / ECMK produce them now) — Path 1 typically doesn't trigger this leg.
|
||||
2. **Physical state changed** — `effective_epc` differs from the lodged EPC's physical fields (walls / heating / windows / etc.). Triggered by Landlord Overrides changing physical state, or by Site Notes that contradict the lodged EPC.
|
||||
|
||||
When triggered, a single ML call re-predicts SAP/carbon/heat with the current Effective EPC state as input. Both reasons can fire together; the prediction is still one call.
|
||||
|
||||
kWh is **always** re-derived via `EpcEnergyDerivationService` — even when no ML rebaseline runs, because fuel rates change over time and the EPC's cost fields are stale by design.
|
||||
|
||||
The diff mechanism for "physical state changed" (content hash, dirty flag, etc.) is an implementation detail; start with a content hash of the physical-state subset of `EpcPropertyData` stored alongside the previous run.
|
||||
|
||||
### 6.5 Deprecated concepts
|
||||
|
||||
- **Patches** (`patch_epc`) — removed. Functionality subsumed by `LandlordOverrides`.
|
||||
- **Already-installed measures** — likely subsumed by `LandlordOverrides` ("we have a heat pump now" → override heating fields). Confirmed at implementation time.
|
||||
- **Non-invasive recommendations** — TBD whether this concept survives; not blocking.
|
||||
|
||||
---
|
||||
|
||||
## 7. Persistence: repositories and unit of work
|
||||
|
||||
### 7.1 What a repository is
|
||||
|
||||
A repository owns the SQL for one aggregate. Nothing else writes SQL for that aggregate. Callers see only domain objects.
|
||||
|
||||
```python
|
||||
class PropertyRepo(Protocol):
|
||||
def get(self, identity: PropertyIdentity) -> Optional[Property]: ...
|
||||
def bulk_save(self, uow: UnitOfWork, properties: Properties) -> None: ...
|
||||
def find_by_portfolio(self, portfolio_id: UUID) -> Properties: ...
|
||||
def find_stale(self, portfolio_id: UUID, threshold: timedelta) -> Properties: ...
|
||||
```
|
||||
|
||||
Implementation references current `db_funcs.*` modules during phase 0 to avoid a big-bang SQL rewrite, but the interface is fixed.
|
||||
|
||||
### 7.2 Unit of Work
|
||||
|
||||
Multi-table writes inside a single aggregate, or across aggregates that share a transaction (e.g. property + plan + recommendations) go through a `UnitOfWork`:
|
||||
|
||||
```python
|
||||
with self.uow_factory() as uow:
|
||||
self.property_repo.bulk_save(uow, properties)
|
||||
self.recommendations_repo.bulk_save(uow, plans)
|
||||
uow.commit()
|
||||
```
|
||||
|
||||
UoW owns the SQLAlchemy session lifecycle. Repos use the session passed in via the UoW. Outside a UoW, repos use a short-lived read session.
|
||||
|
||||
### 7.3 Repository inventory
|
||||
|
||||
| Repo | Tables it owns |
|
||||
|------|----------------|
|
||||
| `PropertyRepo` | `properties`, `property_details_epc`, `property_spatial` |
|
||||
| `EpcCacheRepo` | new table: `epc_api_cache` (TTL, raw API response, mapped `EpcPropertyData`) |
|
||||
| `SiteNotesRepo` | new table: `site_notes` (replaces current `energy_assessments`) |
|
||||
| `LandlordOverridesRepo` | new table: `landlord_overrides` (sparse, per-field rows for audit) |
|
||||
| `RecommendationsRepo` | `plans`, `plan_phases`, `recommendations`, `recommendation_parts`, `scenarios`, `scenario_phases`, `scenario_snapshots` |
|
||||
| `GenericDataRepo` | new table or S3-backed: UPRN-range geospatial + postcode-keyed shared static data |
|
||||
| `FuelRatesRepo` | new table: `fuel_rates` — `(fuel_type, rate_pence_per_kwh, standing_charge_pence_per_day, calorific_value_kwh_per_unit, unit, effective_from, effective_to, region_code Optional, source)`. SEG export rate is a row with `fuel_type = 'electricity_export'`. |
|
||||
| `CarbonFactorsRepo` | new table: `carbon_factors` — `(fuel_type, kgco2e_per_kwh, effective_from, effective_to, source)`. Defra publishes annually. |
|
||||
| `HeatingSystemAssumptionsRepo` | new table(s): boiler efficiency, ASHP/GSHP COP, solar-thermal coverage proportion. Static-ish, manual refresh. |
|
||||
| `SubtaskRepo` | `tasks`, `subtasks` (existing) |
|
||||
|
||||
DDL migrations are scoped to sub-PRD (iii).
|
||||
|
||||
### 7.4 Fakes
|
||||
|
||||
For tests, each repo has a `FakeXRepo` companion backed by a dict. Service unit tests inject fakes. No DB required.
|
||||
|
||||
---
|
||||
|
||||
## 8. ML contract
|
||||
|
||||
### 8.1 Where ML lives
|
||||
|
||||
| Concern | Owner |
|
||||
|---|---|
|
||||
| Defining the EPC → features transform | **This repo** (`ara.domain.ml.EpcMlTransform`) |
|
||||
| Loading data, applying transform, writing training parquet to S3 | **This repo** (sub-PRD (ii) batch job) |
|
||||
| Training, hyperparameter search, deployment | **Autogluon repo** |
|
||||
| Scoring at modelling time | **This repo** (`FeatureBuilder` calls `EpcMlTransform`, sends DataFrame to deployed lambda) |
|
||||
|
||||
The autogluon repo is intentionally **dumb**: it consumes parquet, knows which column is the target, knows which columns to ignore. It has no EPC semantics.
|
||||
|
||||
### 8.2 `EpcMlTransform`
|
||||
|
||||
A separate class (not a method on `EpcPropertyData`), because:
|
||||
|
||||
- The data class stays clean of training-infrastructure concerns.
|
||||
- Versioned transforms (`EpcMlTransformV1`, `EpcMlTransformV2`) swap easily.
|
||||
- Future need: injection of normalisation stats from the training set is straightforward on a class, awkward on a dataclass.
|
||||
|
||||
```python
|
||||
class EpcMlTransform:
|
||||
VERSION: str = "1.0.0" # semver
|
||||
|
||||
def to_row(self, epc: EpcPropertyData) -> dict[str, Any]: ...
|
||||
def to_rows(self, properties: Properties) -> pd.DataFrame: ...
|
||||
def schema(self) -> dict[str, type]: ... # for parquet emission + validation
|
||||
```
|
||||
|
||||
The interesting work — flattening `List[SapWindow]`, `List[SapBuildingPart]` into fixed-width columns — lives inside this class. Domain decisions (top-N windows, aggregate roofs, etc.) are encoded here and reviewed by Khalim. Sub-PRD (ii) goes into detail.
|
||||
|
||||
### 8.3 Versioning
|
||||
|
||||
- Transform class is **semver-tagged** (`VERSION = "1.0.0"`).
|
||||
- S3 path for training parquet includes the version: `s3://.../training/v1.0.0/...`.
|
||||
- Deployed scoring lambda is tagged with the transform version it was trained against.
|
||||
- Modelling pipeline asserts at startup that its `EpcMlTransform.VERSION` matches the deployed lambda's tag; mismatch = hard fail at deploy time.
|
||||
|
||||
Bump major when removing or renaming columns. Bump minor when adding optional columns (older models still scoreable; new models can be trained against new fields).
|
||||
|
||||
### 8.4 ML model families
|
||||
|
||||
Both ML calls (rebaselining + per-measure impact) use the same `EpcMlTransform`:
|
||||
|
||||
| Service | Lambda | Target |
|
||||
|---|---|---|
|
||||
| `RebaseliningService` (S4b) | `baseline-models-*` | SAP / carbon / heat demand under the current Effective EPC state (SAP10-equivalent) |
|
||||
| `ImpactPredictionService` (S6) | `impact-models-*` | SAP / carbon / heat demand impact per measure (and per battery option, using new EPC battery fields) |
|
||||
|
||||
Annual kWh and bills are never an ML target — derived deterministically by `EpcEnergyDerivationService` (S4a). Recommendation kWh delta is derived from the SAP delta predicted by S6 plus heating-system fuel + COP, not via a separate ML call.
|
||||
|
||||
The two families are trained against the same input feature schema; only target columns differ. Sub-PRD (ii) handles training-time details.
|
||||
|
||||
---
|
||||
|
||||
## 9. Service catalogue
|
||||
|
||||
The classes below implement the pipeline end-to-end. Detailed signatures are deliberately left for implementers — this PRD documents purpose, dependencies, and rough shape; per-service grill sessions produce the contracts.
|
||||
|
||||
**Out of the legacy engine** (deleted, not migrated): `PredictionMatrix` (debug-only, moves to test fixtures), `extract_portfolio_aggregation_data` (dead code, FE aggregates dynamically per §10), inspections plumbing (`inspections_map` is initialised but never populated in the current engine), patches / `already_installed` / `non_invasive_recommendations` (subsumed by Landlord Overrides), ECO4 / WHLG funding integration (`get_funding_data` and `optimise_with_scenarios`' funding paths), the pre-recommendation kWh ML lambda (`KWH_MODEL_PREFIXES`), and floor-count / heat-loss-perimeter estimation from geospatial (now on `EpcPropertyData`). Address matching (`address2UPRN`) lives as a separate service, not inside `EpcClientService`.
|
||||
|
||||
### 9.1 Fetchers (called by `IngestionPipeline`)
|
||||
|
||||
| # | Class | Purpose | Dependencies |
|
||||
|---|---|---|---|
|
||||
| F1 | `EpcClientService` | Fetches EPCs from new gov API. Already exists at `backend/epc_client/`. Scope narrows compared to current `SearchEpc` — address matching (`address2uprn`) and OS API estimation are not its concern. | httpx |
|
||||
| F2 | `GeospatialFetcher` | Fetches UPRN-range geospatial data. Replaces `OpenUprnClient`. **Floor count and heat-loss perimeter estimation are no longer needed** — both are now on `EpcPropertyData` directly (`number_of_storeys`, `SapFloorDimension.heat_loss_perimeter_m`). Scope reduces to building geometry and postcode-area context. | S3 / Ordnance Survey API |
|
||||
| F3 | `SolarFetcher` | Wraps Google Solar API; building-level + unit-level scenes. | Google Solar API |
|
||||
| F4 | `SiteNotesIngester` | Loads site notes from Excel uploads / structured input. Persists via `SiteNotesRepo`. | S3, repo |
|
||||
| F5 | `FuelRatesFetcher` | Scheduled ETL — scrapes Ofgem regional caps and per-fuel rates, writes timeseries rows to `FuelRatesRepo`. Manual CSV upload fallback for off-cycle corrections. | Ofgem feed, repo |
|
||||
| F6 | `CarbonFactorsFetcher` | Same shape as F5 against Defra's annual CO2 factor publication. | Defra feed, repo |
|
||||
|
||||
### 9.2 Domain services (called by `ModellingPipeline`)
|
||||
|
||||
| # | Class | Original-list # | Purpose | Reads | Writes |
|
||||
|---|---|---|---|---|---|
|
||||
| S1 | `EpcRemappingService` | 4 | Re-map legacy / historical EPCs into new `EpcPropertyData` shape. | `EpcCacheRepo` | `EpcCacheRepo` (mapped column) |
|
||||
| S2 | `EpcPredictionService` | 3 | For every property: produce predicted EPC + per-field anomaly flags vs neighbours. Used both for gap-fill (Path 2 if EPC missing) and UI surfacing. | `EpcCacheRepo`, `GenericDataRepo` | — |
|
||||
| S3 | `FeatureBuilder` | (new) | Wraps `EpcMlTransform`. Converts `Properties` → scoring DataFrame. | — | — |
|
||||
| S4a | `EpcEnergyDerivationService` | (new) | Derives annual kWh + fuel split + bills from the Effective EPC. Deterministic, no ML. Pipeline: (1) source regulated PEUI — either from `energy_consumption_current × floor_area` when EPC field present and no physical override, or from SAP physics (heat demand × area + SAP hot-water + SAP lighting) for Site Notes / overridden cases; (2) add appliance + cooking via SAP Appendix L formulas (port of [`AnnualBillSavings.estimate_appliances_energy_use`](../../backend/ml_models/AnnualBillSavings.py)); (3) apply UCL per-band correction (Few et al. 2023, Table 3), keyed on the **post-state Effective EPC's band** — not the lodged band; (4) decompose total PEUI into end-use shares via SAP-physics proportions; (5) primary→delivered per fuel using SAP primary factors; (6) bills = delivered kWh per fuel × current rate from `FuelRatesRepo` + standing charges + SEG credits. CO2 emissions from `CarbonFactorsRepo`. | `FuelRatesRepo`, `CarbonFactorsRepo`, `HeatingSystemAssumptionsRepo` | — |
|
||||
| S4b | `RebaseliningService` | (new, partial overlap with old "rebaselining" logic) | Triggered by §6.4 conditions (pre-SAP10 schema **or** physical state changed). Calls SAP/carbon/heat ML lambdas to produce SAP10-equivalent baseline against the current Effective EPC state. Both `BaselinePerformance.lodged_*` and `effective_*` are populated downstream — pair is always stored, equal when not rebaselined. kWh is re-derived via S4a, not ML. | `FeatureBuilder` | — |
|
||||
| S5 | `RecommendationService` | 6 | Generates per-property recommendations against the current rolling Effective EPC. Invoked **once per (scenario × phase)** — filters candidates to the phase's `measure_types_allowed`, returns candidates eligible against the post-prior-phase state. Replaces current `Recommendations` (1383 LOC). | `MaterialsRepo` | — |
|
||||
| S6 | `ImpactPredictionService` | 7 | Calls SAP / carbon / heat impact ML lambda for **every** candidate recommendation (FE displays all options to user). Invoked per (scenario × phase) with the rolling state's feature vector. Recommendation kWh delta is derived deterministically from SAP delta + heating-system fuel/COP, not from a separate ML call. Battery impact uses the new EPC battery fields (`energy_pv_battery_count`, `energy_pv_battery_capacity`) as ML inputs — the deterministic `BatterySAPScorer` from the legacy engine is replaced by ML prediction. | `FeatureBuilder` | — |
|
||||
| S7 | `OptimiserService` | 8 | Per-phase optimisation against rolling state. Reads `PlanPhase.state_at_end[n-1]` to honour cross-phase constraints (fabric-first, heat-pump-needs-insulation, ventilation). Wraps current `CostOptimiser` / `GainOptimiser` / `optimise_with_scenarios` minus the dead ECO-funding paths. Unselected candidates roll into phase n+1's candidate pool (auto vs user-marked TBD, §15). | — | — |
|
||||
| S8 | `ValuationService` | — | Estimates per-property valuation (current + post-retrofit) from academic-paper-based regression on EPC change, property type, region. Improvement on the existing `PropertyValuation.estimate` code — exact shape deferred to per-service grill. | — | — |
|
||||
| S9 | `ResultsPersister` | 9 | Final step: writes Plan (with `phases[]`) + Recommendations + Property updates via repos under one UoW, per scenario. | — | All write repos |
|
||||
|
||||
### 9.3 Orchestrators
|
||||
|
||||
| # | Class | Purpose |
|
||||
|---|---|---|
|
||||
| O1 | `IngestionPipeline` | Per-batch SQS consumer. Calls F1–F4, persists via repos. |
|
||||
| O2 | `ModellingPipeline` | Per-batch SQS consumer. Reads from repos, runs S1→S8 in order, ends with persistence. |
|
||||
| O3 | `RefreshOrchestrator` | Top-level: triggers Ingestion → diff → optionally Modelling. |
|
||||
|
||||
### 9.4 `ModellingPipeline` step order
|
||||
|
||||
For each `Property` in the batch, against each pinned `ScenarioSnapshot` from the trigger payload:
|
||||
|
||||
```
|
||||
Per-property setup (runs once regardless of scenario count):
|
||||
1. PropertyRepo.get() → Property (epc, site_notes, overrides, geospatial, solar)
|
||||
2. EpcRemappingService — if epc is in legacy schema, upgrade to current
|
||||
3. EpcPredictionService — predicted EPC + per-field anomaly flags (always runs)
|
||||
4. Compute Property.effective_epc (path-1 or path-2)
|
||||
5. RebaseliningService — IF §6.4 conditions hold (pre-SAP10 OR physical state changed),
|
||||
re-predict SAP/carbon/heat via ML against the Effective EPC state.
|
||||
Populate BaselinePerformance.lodged_* + effective_*.
|
||||
6. EpcEnergyDerivationService — SAP-physics + UCL (post-state band) + FuelRates → kWh, fuel split, bills.
|
||||
|
||||
Per-scenario loop:
|
||||
Per-phase loop (in scenario phase order):
|
||||
7. RecommendationService — generate candidate measures, restricted to phase's measure_types_allowed,
|
||||
against the rolling Effective EPC state (baseline for phase 1; updated for phase 2+).
|
||||
8. ImpactPredictionService — predict SAP/carbon/heat impact for those candidates, ML scored against
|
||||
the rolling state's feature vector. All candidates scored (FE shows options).
|
||||
9. OptimiserService — select package within phase budget + phase goal. Reads earlier-phase state to honour
|
||||
cross-phase constraints (fabric-first, heat-pump-needs-insulation, ventilation).
|
||||
10. Apply package → roll state forward (simulate post-package SAP / kWh / bills via S4a + impact predictions
|
||||
from step 8). Record `PlanPhase.state_at_end`. Unselected options become
|
||||
`PlanPhase.rolled_over_options` and are eligible candidates next phase.
|
||||
11. ResultsPersister — write Plan (phases[]) + Recommendations under one UoW for this scenario.
|
||||
```
|
||||
|
||||
Steps 1–6 run **once per property** regardless of scenario count.
|
||||
Steps 7–10 run **once per (scenario × phase)** per property.
|
||||
Step 11 runs once per scenario per property.
|
||||
|
||||
Batching: steps 5, 8 batch the whole batch into one ML call where possible. Step 8's cost scales with `N_phases × N_scenarios × N_candidate_measures`; multi-phase pays its own ML bill, single-phase scenarios cost the same as today.
|
||||
|
||||
Note vs the current `model_engine`: the **pre-recommendation** kWh ML call has been removed. Baseline kWh now comes from `EpcEnergyDerivationService` (SAP physics + UCL + FuelRates). ML is reserved for SAP/carbon/heat (rebaselining + impact prediction). Recommendation-level kWh delta is derived deterministically from the impact-predicted SAP delta plus heating-system fuel + COP from `HeatingSystemAssumptionsRepo`; no separate kWh ML lambda.
|
||||
|
||||
**Open future change** (flagged §15): SAP-impact-of-a-measure is not strictly additive — installing measure A changes the SAP impact of measure B. The current per-measure ML scoring + linear optimisation approximates this. A future iteration may pre-define candidate packages and ML-score whole packages, accepting the combinatorial cost in return for accuracy. Defer until implementation reveals where the approximation hurts.
|
||||
|
||||
### 9.5 Per-service contracts — deferred
|
||||
|
||||
Method signatures, return types, error semantics, and edge-case behaviour are **explicitly out of scope** for this PRD. The implementer of each service runs a `/grill-me` session against this document and produces a detailed sub-design before coding.
|
||||
|
||||
---
|
||||
|
||||
## 10. Cross-batch concerns
|
||||
|
||||
| Concern | Status | Approach |
|
||||
|---|---|---|
|
||||
| Building-level solar adjustment | Deferred — future TODO, not implemented today. | The current `building_ids` block in `model_engine` is dead-ish; it operates on the in-process batch only. New design preserves that limitation. Future feature: a post-modelling consolidation pass that groups results by `building_id` across batches and re-optimises. |
|
||||
| Portfolio aggregation | Dropped. | Front-end computes aggregations dynamically from per-property plans. `extract_portfolio_aggregation_data` in current engine is dead code (defined, never called) — deleting. |
|
||||
| Shared upstream data | Handled by orchestrator partitioning + `GenericDataRepo`. | Trigger endpoint groups UPRNs by postcode / UPRN-range before SQS chunking so each batch maximises intra-batch sharing. `GenericDataRepo` caches across batches so first batch pays, subsequent batches hit cache. |
|
||||
|
||||
---
|
||||
|
||||
## 11. Repository layout — monorepo via uv workspaces
|
||||
|
||||
The repo is restructured as a Python monorepo using **uv workspaces**. Shared types and shared infra live as workspace packages under `packages/`; each deployable Lambda or microservice lives as its own package under `services/`. Each `services/<svc>/` has its own `pyproject.toml`, `Dockerfile`, and Lambda image — the bundle contains only that service's deps + its workspace deps, keeping cold-start size and package weight contained.
|
||||
|
||||
```
|
||||
/
|
||||
├── pyproject.toml # workspace root
|
||||
├── uv.lock
|
||||
│
|
||||
├── packages/ # shared workspace packages — imported by services/
|
||||
│ ├── domain/ # "domna-domain"
|
||||
│ │ ├── pyproject.toml
|
||||
│ │ └── src/domain/
|
||||
│ │ ├── property.py # Property, Properties, PropertyIdentity
|
||||
│ │ ├── site_notes.py
|
||||
│ │ ├── landlord_overrides.py
|
||||
│ │ ├── baseline_performance.py # lodged + effective pair
|
||||
│ │ ├── plan.py # Plan, PlanPhase, OptimisedPackage
|
||||
│ │ ├── scenario.py # Scenario, ScenarioPhase, ScenarioSnapshot
|
||||
│ │ ├── recommendation.py
|
||||
│ │ ├── geospatial.py
|
||||
│ │ ├── solar.py
|
||||
│ │ ├── anomaly_flags.py
|
||||
│ │ └── ml/
|
||||
│ │ ├── transform.py # EpcMlTransform (versioned)
|
||||
│ │ └── schema.py
|
||||
│ │
|
||||
│ ├── repos/ # "domna-repos" — persistence, no business logic
|
||||
│ │ ├── pyproject.toml
|
||||
│ │ └── src/repos/
|
||||
│ │ ├── unit_of_work.py
|
||||
│ │ ├── property_repo.py
|
||||
│ │ ├── epc_cache_repo.py
|
||||
│ │ ├── site_notes_repo.py
|
||||
│ │ ├── landlord_overrides_repo.py
|
||||
│ │ ├── recommendations_repo.py
|
||||
│ │ ├── generic_data_repo.py
|
||||
│ │ ├── fuel_rates_repo.py
|
||||
│ │ ├── carbon_factors_repo.py
|
||||
│ │ ├── heating_system_assumptions_repo.py
|
||||
│ │ └── subtask_repo.py
|
||||
│ │
|
||||
│ ├── fetchers/ # "domna-fetchers" — external API clients
|
||||
│ │ ├── pyproject.toml
|
||||
│ │ └── src/fetchers/
|
||||
│ │ ├── epc_client.py # wraps backend/epc_client/
|
||||
│ │ ├── geospatial.py
|
||||
│ │ ├── solar.py
|
||||
│ │ ├── fuel_rates_fetcher.py
|
||||
│ │ └── carbon_factors_fetcher.py
|
||||
│ │
|
||||
│ └── utils/ # "domna-utils" — logging, AWS, S3, cloudwatch, subtasks
|
||||
│ ├── pyproject.toml
|
||||
│ └── src/utils/
|
||||
│
|
||||
├── services/ # deployable units, one Lambda image each
|
||||
│ ├── ara/ # the modelling backend
|
||||
│ │ ├── pyproject.toml # deps: domna-domain, domna-repos, domna-fetchers, domna-utils, ML libs
|
||||
│ │ ├── Dockerfile
|
||||
│ │ ├── src/ara/
|
||||
│ │ │ ├── services/ # EpcRemappingService, EpcPredictionService,
|
||||
│ │ │ │ # EpcEnergyDerivationService, RebaseliningService,
|
||||
│ │ │ │ # FeatureBuilder, RecommendationService,
|
||||
│ │ │ │ # ImpactPredictionService, OptimiserService,
|
||||
│ │ │ │ # ValuationService, ResultsPersister
|
||||
│ │ │ ├── orchestrators/ # IngestionPipeline, ModellingPipeline, RefreshOrchestrator
|
||||
│ │ │ └── lambdas/ # handler.py per Lambda + event-shape contracts
|
||||
│ │ └── tests/
|
||||
│ │ ├── fakes/ # FakePropertyRepo, FakeEpcClient, etc.
|
||||
│ │ ├── unit/ # service tests using fakes only
|
||||
│ │ └── integration/ # real DB + real SQS via localstack
|
||||
│ │
|
||||
│ ├── address2uprn/ # messy-address → UPRN matching, pre-modelling step
|
||||
│ │ ├── pyproject.toml
|
||||
│ │ ├── Dockerfile
|
||||
│ │ └── src/address2uprn/
|
||||
│ ├── hubspot/ # existing Hubspot ETL
|
||||
│ ├── pashub/ # PasHub survey ingestion
|
||||
│ ├── ecmk/ # ECMK assessment ingestion
|
||||
│ └── magicplan/ # MagicPlan integration
|
||||
│
|
||||
├── backend/ # legacy FastAPI app + microservices, kept until cut-over
|
||||
│ ├── app/ # FastAPI; thin entrypoints that invoke service Lambdas
|
||||
│ └── ... # legacy engine, SearchEpc, etc.; deleted after cut-over
|
||||
│
|
||||
├── datatypes/ # existing — EPC schemas; eventually folds into packages/domain/
|
||||
└── docs/
|
||||
└── adr/ # architectural decision records
|
||||
```
|
||||
|
||||
**Boundary properties** (enforced by package structure, not convention):
|
||||
- A `services/<svc>/` package can `import domain.*`, `import repos.*`, `import fetchers.*`, `import utils.*`. It **cannot** import another service's modules — they're separate distributions with no cross-import path.
|
||||
- ADR-0003 (Ingestion / Modelling separation) is preserved: modelling services in `services/ara/src/ara/services/` depend only on `repos.*` + `domain.*`, never on fetchers. Orchestrators are the only place fetchers and services meet.
|
||||
|
||||
**Migration** (incremental, not big-bang):
|
||||
1. Carve out `packages/domain/` first — fold `datatypes/epc/domain/` + the new aggregate types into it.
|
||||
2. Carve out `packages/utils/` from current `utils/` + `backend/utils/`.
|
||||
3. Carve out `packages/repos/` and `packages/fetchers/` once `services/ara/` is being built and needs them.
|
||||
4. `services/ara/` is greenfield — no legacy code lives in it.
|
||||
5. `services/address2uprn/`, `services/pashub/`, etc. are split out as their owners pick them up.
|
||||
6. `backend/` shrinks to the FastAPI entrypoint layer once everything else has moved.
|
||||
|
||||
**Reused intact** (no rewrite needed at carve-out time):
|
||||
- `backend/epc_client/` → folds into `packages/fetchers/src/fetchers/epc_client.py`.
|
||||
- `datatypes/epc/domain/` → folds into `packages/domain/src/domain/epc/`.
|
||||
- `recommendations/optimiser/` → wrapped by `services/ara/src/ara/services/optimiser.py`.
|
||||
- `backend/app/db/` → repos delegate into `db_funcs.*` until SQL is rewritten under sub-PRD (iii).
|
||||
|
||||
---
|
||||
|
||||
## 12. Testing strategy
|
||||
|
||||
### 12.1 Unit tests (the bulk)
|
||||
|
||||
Every service test injects fake fetchers and fake repos. No DB, no network, no ML lambda. A service test verifies one slice of logic in 5–30 lines.
|
||||
|
||||
Example:
|
||||
|
||||
```python
|
||||
def test_epc_prediction_flags_anomalous_wall_type():
|
||||
neighbours = [_make_epc(wall_construction="solid") for _ in range(5)]
|
||||
target = _make_property(epc=_make_epc(wall_construction="cavity"))
|
||||
repo = FakeGenericDataRepo(neighbours_by_postcode={target.identity.postcode: neighbours})
|
||||
|
||||
svc = EpcPredictionService(generic_repo=repo)
|
||||
result = svc.run(Properties([target]))
|
||||
|
||||
assert result[0].epc_anomaly_flags.wall_construction == "differs_from_neighbours"
|
||||
```
|
||||
|
||||
### 12.2 Integration tests
|
||||
|
||||
One per pipeline (Ingestion, Modelling, Refresh). Real Postgres (testcontainers or localstack), fake fetchers (hitting recorded fixtures), fake ML lambdas (returning canned predictions). Catches schema / SQL / transaction issues.
|
||||
|
||||
### 12.3 Contract tests
|
||||
|
||||
The transform (`EpcMlTransform`) has its own test suite:
|
||||
|
||||
- Golden file: given a fixed `Property`, output matches an expected DataFrame row exactly.
|
||||
- Schema test: the output columns exactly match a checked-in CSV header (so autogluon team sees breakage on PR).
|
||||
|
||||
### 12.4 What is NOT tested
|
||||
|
||||
- The autogluon repo's training code — owned there.
|
||||
- The gov EPC API behaviour — assumed via the official spec.
|
||||
- Front-end aggregation logic — owned there.
|
||||
|
||||
---
|
||||
|
||||
## 13. Observability
|
||||
|
||||
Each pipeline step emits a **structured log line** at start and end with:
|
||||
|
||||
```
|
||||
{step, property_id, uprn, portfolio_id, subtask_id, duration_ms, outcome, error?}
|
||||
```
|
||||
|
||||
Errors propagate with the `Property.identity` attached, so a portfolio of 100k can be triaged by grep.
|
||||
|
||||
The existing task/subtask state machine is preserved — `IngestionPipeline` and `ModellingPipeline` update subtask status at start (`in progress`), end (`complete` / `failed`), with the CloudWatch log URL attached as today.
|
||||
|
||||
CloudWatch alarms exist on subtask failure rate; thresholds remain unchanged.
|
||||
|
||||
---
|
||||
|
||||
## 14. Data flow: a worked example
|
||||
|
||||
A landlord uploads a corrected heating system for UPRN 12345 via the UI.
|
||||
|
||||
1. **UI** → `POST /properties/12345/overrides` → writes to `landlord_overrides` table via `LandlordOverridesRepo`.
|
||||
2. **RefreshOrchestrator** invoked (either automatically on override-write, or by a "re-model" button). Notes: ingestion is *not* triggered because no external state changed.
|
||||
3. **ModellingPipeline** invoked on a batch of `[12345]`:
|
||||
- Reads `Property(uprn=12345)` from `PropertyRepo`.
|
||||
- `Property.effective_epc` = epc + landlord_overrides → heating system fields differ from baseline.
|
||||
- `RebaseliningService` triggered: ML re-predicts SAP / carbon / heat against the new effective EPC.
|
||||
- `EpcEnergyDerivationService` re-runs over the new effective EPC to derive baseline kWh + fuel split + bills (no ML).
|
||||
- `RecommendationService` regenerates recommendations against the new baseline.
|
||||
- `OptimiserService` re-picks optimal package.
|
||||
- `ResultsPersister` writes new plan under one UoW (old plan is superseded; whether to soft-archive is a sub-PRD (iii) decision).
|
||||
|
||||
Total external calls: zero. The override write is the only thing that hit a network boundary, and that was the inbound HTTP from the UI.
|
||||
|
||||
---
|
||||
|
||||
## 15. Open questions for team review
|
||||
|
||||
1. **One endpoint vs two** (§4.5) — **resolved**: single endpoint for Phase 1; split later when a real workflow demands it.
|
||||
2. **`LandlordOverrides` shape** (§6.2) — flat-Excel-shape for v1, with a flag to revisit after first customer.
|
||||
3. **`already_installed` and `non_invasive_recommendations`** (§6.5) — both likely subsumed by overlay, but final call deferred.
|
||||
4. **Recency tie-break policy** (§6.3) — default "newer wins"; team to consider per-portfolio override.
|
||||
5. **`GenericDataRepo` storage backend** — Postgres table, S3, or DynamoDB. Postgres is the path of least infra change; recommend defaulting to that.
|
||||
6. **Soft-archive vs hard-overwrite** for superseded plans (§14) — affects audit / undo behaviour. Defer to sub-PRD (iii).
|
||||
7. **Building-level optimisation as a Phase 2 service** (§10) — agreed deferred; flag for roadmap discussion.
|
||||
8. **Transform versioning policy** (§8.3) — semver chosen; team to confirm bump conventions.
|
||||
9. **UCL EPC-correction model** (§9.2 S4a) — **resolved**: Few et al. 2023 (Energy & Buildings 288, 113024). Implementation pattern already in [`AnnualBillSavings.adjust_energy_to_metered`](../../backend/ml_models/AnnualBillSavings.py) — port the per-band gradients/intercepts (Table 3) into `EpcEnergyDerivationService`, keyed on the post-state Effective EPC band.
|
||||
10. **Fuel-price source for bill calculation** (§9.2 S4a) — **resolved**: `FuelRatesRepo` is a time-versioned, region-aware table; ETL by `FuelRatesFetcher` (Ofgem feed + manual upload fallback). Per-portfolio override deferred to v2 — confirm whether Calico / Hyde have bulk-buy contracts before first onboarding.
|
||||
11. **kWh handling under Rebaselining** (§9.4) — **resolved**: ML re-predicts SAP/carbon/heat only; `EpcEnergyDerivationService` re-derives kWh from the rebaselined Effective EPC. Heating-fuel-type change is handled naturally because S4a re-reads heating fields from the Effective EPC.
|
||||
12. **Phase rollover semantics** (§9.2 S7) — when a candidate measure isn't selected in phase n, does it auto-roll into phase n+1's candidate pool, or does the user mark which measure types can roll? Auto is simpler; user-marked is more flexible. Decide at scenario-builder UX time.
|
||||
13. **Package-level vs per-measure ML scoring** (§9.4) — SAP impact of a measure is not strictly additive; the current per-measure scoring + linear optimisation approximates this. A future iteration may pre-define candidate packages and ML-score whole packages. Defer until per-service grill on `OptimiserService`.
|
||||
14. **UCL extrapolation scope** (§9.2 S4a) — the Few et al. paper is gas-heated, no PV, England + Wales only. Current legacy code applies the correction to all properties regardless. Keep silent extrapolation for v1, or stratify (no correction for non-gas / PV) and surface uncertainty to FE? Defer to per-service grill.
|
||||
15. **`ValuationService` rebuild** (§9.2 S8) — existing `PropertyValuation.estimate` cites several papers; the rebuild should improve the regression. Shape deferred to per-service grill.
|
||||
16. **Battery-via-ML cutover** (§9.2 S6) — confirm the new ML model is trained against `energy_pv_battery_count` + `energy_pv_battery_capacity` and the legacy `BatterySAPScorer` can be retired without regression for battery-equipped properties.
|
||||
|
||||
---
|
||||
|
||||
## 16. Linked sub-PRDs (placeholders)
|
||||
|
||||
- **Sub-PRD (ii) — ML training pipeline** — `docs/sub-prds/ml-training-pipeline.md` (TBC)
|
||||
- **Sub-PRD (iii) — DB schema migration** — `docs/sub-prds/db-schema-migration.md` (TBC)
|
||||
- **Sub-PRD (iv) — Historical EPC re-mapping** — `docs/sub-prds/historical-epc-remap.md` (TBC)
|
||||
|
||||
Each sub-PRD owner: TBC. Each is independently reviewable but consumes the contracts defined in §5 (`Property` aggregate), §7 (repos), §8 (ML transform).
|
||||
|
||||
---
|
||||
|
||||
## 17. Next steps
|
||||
|
||||
1. Team review of this PRD (target: ~1 week).
|
||||
2. Open follow-up grill sessions per service (`/grill-me` on each of S1–S8 + F1–F4) before that service is implemented.
|
||||
3. Break into issues via `/to-issues` against the project tracker.
|
||||
4. Stand up the empty `ara/` package skeleton + fakes + first integration-test scaffold as PR-1.
|
||||
5. Land services in dependency order: domain → repos → fetchers → services → orchestrators → API.
|
||||
|
||||
Phase 1 milestone gate: first portfolio (Calico or Hyde) routed through the new pipeline end-to-end in June, with a manual spot-check on 5 representative properties to confirm outputs are reasonable. No parity-against-old-engine check — the old engine is dead by then.
|
||||
10
docs/adr/0001-two-source-paths.md
Normal file
10
docs/adr/0001-two-source-paths.md
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
# Two source paths for a Property, not layered precedence
|
||||
|
||||
For modelling a Property we considered a strict layered precedence stack — `patches > site_notes > energy_assessment > epc > predicted` — with per-field provenance tracking. We rejected that in favour of **two strictly disjoint source paths**: a Property is modelled either from its Site Notes alone, or from the public EPC with Landlord Overrides applied on top. Site Notes are committed to being full-coverage by the domain ([CONTEXT.md](../../CONTEXT.md): _Site Notes_), so once we have them the EPC is irrelevant; conversely, Landlord Overrides are only meaningful when the EPC is the source of physical state.
|
||||
|
||||
The trade-off: layered precedence is more flexible (it tolerates a partial Site Notes survey by falling through to EPC for missing fields), but mixed-source data muddles the audit trail and undermines the "if we surveyed it, trust the survey" promise. The two-path model gives a cleaner derivation rule and an unambiguous source-of-truth per Property, at the cost of treating survey gaps as a survey-quality bug rather than a fallback signal. A Recency Tie-Break covers the one case where both exist: the newer of the two wins.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Reversing this means rewriting `Property.effective_epc` and every service that reads it. Hard to roll back once 12 services depend on the two-path shape.
|
||||
- Future addition of a third path (e.g. partial-survey) is a real change, not just a config tweak — flag it as an ADR if proposed.
|
||||
14
docs/adr/0002-property-aggregate-root.md
Normal file
14
docs/adr/0002-property-aggregate-root.md
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
# `Property` is the aggregate root, not `EpcPropertyData`
|
||||
|
||||
The Ara modelling pipeline produces nine slices of per-property data (EPC, geospatial, solar, baseline performance, recommendations, optimised package, etc.). We considered making `EpcPropertyData` — the rich RdSAP-21-style EPC schema — the centrepiece, with other data hanging off it. We rejected that and introduced a new **`Property` aggregate root** that holds identity, all source data (EPC, Site Notes, Landlord Overrides), enrichments, and modelling outputs as named fields. Services take `Property` (or `Properties`) and return them with one slice populated.
|
||||
|
||||
Two reasons drove this:
|
||||
1. **Geospatial, solar, recommendations, and overrides are peers to the EPC**, not properties of it. Putting them on `EpcPropertyData` conflates physical-state schema with modelling-run state.
|
||||
2. **A typed `ModellingContext` dict-bag (the obvious alternative)** is exactly what the current legacy `Property` class became — 1259 lines of accumulated stuff, hard to read, hard to test, hard to extend. Named fields on a dataclass force the type system to keep us honest.
|
||||
|
||||
The cost is more domain types up front (`Property`, `Properties`, `PropertyIdentity`, `BaselinePerformance`, `OptimisedPackage`, etc.) and the discipline of one service writing one slice. The benefit is that every service has a single job and every test injects fake repos against a small, named structure.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Every service signature accepts or returns `Property` / `Properties`. Refactoring later means touching all of them.
|
||||
- `EpcPropertyData` stays a pure physical-state schema (defined in [datatypes/epc/domain/epc_property_data.py](../../datatypes/epc/domain/epc_property_data.py)) — no modelling outputs or run state on it.
|
||||
13
docs/adr/0003-strict-ingestion-modelling-separation.md
Normal file
13
docs/adr/0003-strict-ingestion-modelling-separation.md
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
# Strict separation between Ingestion and Modelling
|
||||
|
||||
Data flows one way only: **Ingestion → Repos → Modelling**. Modelling services never make external HTTP calls; Ingestion services never run business logic. If Modelling needs fresh data, it sees a stale record in a repo and returns; the caller (a refresh orchestrator or the FE) decides whether to ingest first. We considered allowing modelling services to call fetchers directly on cache miss — convenient — and rejected it.
|
||||
|
||||
The trade-off is that modelling cannot "self-heal" by going to the gov EPC API when it finds stale data. The benefit is that modelling becomes a deterministic function of repository state: same Property in the repos, same modelling output. That is the property that makes modelling unit-testable against fakes (no DB, no network, no ML lambda), reproducible, and debuggable. It also enables a per-property UI flow where fetched data is shown to the user for review and possible override **before** modelling runs.
|
||||
|
||||
Under the rushed timeline this constraint is more valuable, not less. Mixing fetchers into services is the easy thing to do when shipping fast; once it's done it's hard to extract.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Every modelling service depends only on Repos (and other Services / domain logic). No HTTP libraries in the modelling import graph.
|
||||
- A `RefreshOrchestrator` is the only thing that calls Ingestion then Modelling in sequence; nothing else may.
|
||||
- "Modelling is stale, refetch in-line" is a forbidden pattern — surface staleness, do not silently repair it.
|
||||
Loading…
Add table
Reference in a new issue