junte/Model
1
0
Fork 0
mirror of https://github.com/Hestia-Homes/Model.git synced 2026-07-27 23:35:01 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge remote-tracking branch 'origin/main' into fix/epc-persistence-dropped-fields

Browse source
This commit is contained in:
Khalim Conn-Kowlessar 2026-07-23 09:27:32 +00:00
commit 4a67857c9a
48 changed files with 4548 additions and 37 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

24
CONTEXT.md
View file

@ -105,12 +105,34 @@ _Avoid_: document category, file kind, doc class
**Download Package**:
The single ZIP archive a **Bulk Document Download** produces: one folder per property (named by human-readable address, enriched from the hubspot deals data, with `landlord_property_id` appended for uniqueness), each folder holding the latest file of each **Document Type** held for that property. Built best-effort — properties/documents that don't resolve are skipped and listed in the run's `sub_task.outputs`, not failed (ADR-0060). Streamed to a dedicated `DOCUMENT_EXPORTS_BUCKET` (never held whole in memory — a package can be several GB); delivered as a 60-minute presigned URL.
_Avoid_: export (that is the plan/scenario XLSX export), bundle, archive (ambiguous), zip (the format, not the concept)
_Avoid_: export (that is the **Scenario Export**), bundle, archive (ambiguous), zip (the format, not the concept)
**Bulk Document Download**:
The feature/job that assembles a **Download Package** for a chosen set of properties (the union of one or more whole HubSpot projects, selected by `project_code`, and a hand-picked `landlord_property_id` list) and emails the requester a link. Runs on the app-owned-task + attach-mode lane (ADR-0055): the FastAPI route resolves the selection to a distinct `landlord_property_id` set — project codes are expanded via `hubspot_deal_data` (which holds the project↔property grain), hand-picked ids are taken as given — pins that set and the recipient email onto the `sub_task`, and the `applications/bulk_document_download` Lambda builds the package, writes the URL to `sub_task.outputs`, and emails it (ADR-0059/0060). The Lambda matches files by `hubspot_deal_id`, bridging each selected `landlord_property_id` through `hubspot_deal_data` (`uploaded_files.landlord_property_id` is unpopulated in practice, so the property identity comes from the deal bridge); coverage is limited to deal-id-linked sources. Selection is capped by property count at the trigger.
_Avoid_: document export, bulk export, file dump
**Scenario Export** (canonical short form: **Export**):
The single branded `.xlsx` a user pulls from the front end to get a portfolio's
**modelling results** — property information plus each Property's **Plan** under
one or more **Scenarios** — with a link emailed to the requester. It is
**scenario-scoped**: the rows for a Scenario are the Properties that have a
**Plan** under it, so a Property with no Plan for that Scenario is absent (model
A, ADR-0065); a filter or a hand-picked id list only narrows that set. One
workbook holds **one sheet per selected Scenario** (`scenario_ids`), the same
filtered Property set rendered per Scenario — so sheets may legitimately differ
in row count. Property selection reuses the **Modelling Run**'s filter contract
(`PropertyGroupFilters` + `resolve_filtered_property_ids`, ADR-0056); an explicit
`property_ids` list is an **override** of the filters, not a co-filter. Built
**best-effort** from **persisted** data only (no live EPC calls), reading each
Property's **Effective EPC** (so `property_overrides` win over the lodged cert),
against a frozen 21-measure column contract. Runs on the app-owned-task +
attach-mode lane and is delivered by presigned URL + email exactly like a
**Download Package** (ADR-0055/0059), but its payload is plan/scenario data, not
survey documents. Distinct from a **Download Package** (the ZIP of survey
documents) and from a **Modelling Run** (which *produces* the Plans an Export
reads).
_Avoid_: data export / bulk export (ambiguous), download (that is the **Download Package**), report, dump, plan export (it is scenario-scoped, not per-Plan)
### Source data
**Site Notes**:

0
applications/ara_export/__init__.py Normal file
View file

19
applications/ara_export/ara_export_trigger_body.py Normal file
View file

@ -0,0 +1,19 @@
from uuid import UUID
from pydantic import BaseModel, ConfigDict
class AraExportTriggerBody(BaseModel):
"""SQS trigger for the ara_export Lambda (ADR-0055/0065).
Attach mode: the FastAPI route created the Task + SubTask and pinned the
recipe (``portfolio_id``, ``scenario_ids``, ``property_ids``,
``recipient_email``, ``export_name``) onto the SubTask's ``inputs``. The
message therefore carries only the identifiers — the (potentially large)
property set never travels through the 256 KB SQS body.
"""
model_config = ConfigDict(extra="allow")
task_id: UUID
subtask_id: UUID

106
applications/ara_export/handler.py Normal file
View file

@ -0,0 +1,106 @@
"""ara_export Lambda (ADR-0065).
Attach-mode (ADR-0055) handler. The FastAPI route created the app-owned Task +
SubTask and pinned the recipe onto the SubTask's ``inputs``; this reads that
recipe, builds the sheet-per-scenario branded workbook through the orchestrator,
and returns the result (presigned URL + counts), which ``TaskOrchestrator``
records on ``sub_task.outputs``. A selection that yields no rows raises
``SubTaskFailure`` from the orchestrator — recorded, not retried.
PostgresConfig-only (POSTGRES_*), like the bulk_document_download Lambda. The
workbook is written to the dedicated ``DOCUMENT_EXPORTS_BUCKET``; SES SMTP
settings arrive as env vars.
"""
from __future__ import annotations
import logging
import os
from typing import Any
import boto3
from applications.ara_export.ara_export_trigger_body import AraExportTriggerBody
from domain.tasks.tasks import Source
from infrastructure.email.ses_smtp_email_sender import SesSmtpEmailSender
from infrastructure.postgres.config import PostgresConfig
from infrastructure.postgres.engine import make_engine, make_session
from infrastructure.s3.s3_client import S3Client
from orchestration.scenario_export_orchestrator import ScenarioExportOrchestrator
from repositories.scenario_export.scenario_export_postgres_repository import (
ScenarioExportPostgresRepository,
)
from repositories.scenario_export.scenario_names_postgres_repository import (
ScenarioNamesPostgresRepository,
)
from repositories.tasks.subtask_postgres_repository import SubTaskPostgresRepository
from utilities.aws_lambda.task_handler import task_handler
logging.getLogger().setLevel(logging.INFO)
logger = logging.getLogger(__name__)
# The export link is valid for 60 minutes (ADR-0065).
_URL_TTL_SECONDS = 3600
def _email_sender() -> SesSmtpEmailSender:
return SesSmtpEmailSender(
host=os.environ["SES_SMTP_HOST"],
port=int(os.environ["SES_SMTP_PORT"]),
username=os.environ["SES_SMTP_USERNAME"],
password=os.environ["SES_SMTP_PASSWORD"],
from_address=os.environ["SES_SMTP_FROM_ADDRESS"],
)
@task_handler(task_source="ara_export", source=Source.PORTFOLIO)
def handler(body: dict[str, Any], context: Any) -> dict[str, Any]:
trigger = AraExportTriggerBody.model_validate(body)
engine = make_engine(PostgresConfig.from_env(os.environ))
session = make_session(engine)
try:
subtask = SubTaskPostgresRepository(session).get(trigger.subtask_id)
recipe: dict[str, Any] = subtask.inputs or {}
portfolio_id = int(recipe["portfolio_id"])
scenario_ids: list[int] = [int(x) for x in recipe["scenario_ids"]]
property_ids: list[int] = [int(x) for x in recipe["property_ids"]]
recipient_email = str(recipe["recipient_email"])
export_name = str(recipe["export_name"])
# ADR-0065 plan selection: "scenario" (default) or "default" (each
# home's is_default Plan). Absent on pre-existing recipes → "scenario".
plan_selection = str(recipe.get("plan_selection") or "scenario")
logger.info(
"ara_export: starting %s export for %d properties across %d "
"scenario(s) (subtask=%s)",
plan_selection,
len(property_ids),
len(scenario_ids),
trigger.subtask_id,
)
boto_s3: Any = boto3.client("s3") # pyright: ignore[reportUnknownMemberType]
orchestrator = ScenarioExportOrchestrator(
rows=ScenarioExportPostgresRepository(session),
scenarios=ScenarioNamesPostgresRepository(session),
exports=S3Client(boto_s3, os.environ["DOCUMENT_EXPORTS_BUCKET"]),
email=_email_sender(),
url_ttl_seconds=_URL_TTL_SECONDS,
)
result = orchestrator.run(
portfolio_id=portfolio_id,
scenario_ids=scenario_ids,
property_ids=property_ids,
recipient_email=recipient_email,
export_name=export_name,
plan_selection=plan_selection,
)
finally:
session.close()
return {
"presigned_url": result.presigned_url,
"export_s3_key": result.export_s3_key,
"sheet_count": result.sheet_count,
"row_count": result.row_count,
}

1
backend/app/config.py
View file

@ -47,6 +47,7 @@ class Settings(BaseSettings):
FINALISER_SQS_URL: str = "changeme"
MODELLING_E2E_SQS_URL: str = "changeme"
BULK_DOCUMENT_DOWNLOAD_SQS_URL: str = "changeme"
ARA_EXPORT_SQS_URL: str = "changeme"
# Third parties
EPC_AUTH_TOKEN: str = "changeme"

0
backend/app/exports/__init__.py Normal file
View file

104
backend/app/exports/export_tasks.py Normal file
View file

@ -0,0 +1,104 @@
"""The Scenario Export route's view of the app-owned task's sub_task (ADR-0065).
Parameterised SQL rather than the SQLModel mirrors: importing the
``infrastructure.postgres`` mirror of ``sub_task`` into the FastAPI process
double-registers the table and crashes at import (the same constraint the
Modelling Run distributor and the bulk-download route work around).
"""
from __future__ import annotations
import json
from datetime import datetime, timezone
from typing import Any, cast
from uuid import UUID
from sqlalchemy import text
from sqlmodel import Session
from backend.app.modelling.property_filters import (
PropertyGroupFilters,
resolve_filtered_property_ids,
)
class ScenarioExportTasks:
def __init__(self, session: Session) -> None:
self._session = session
def read_selection_config(self, task_id: UUID) -> dict[str, Any]:
"""The task's selection recipe, read from the FE-owned ``task.inputs``
JSON (ADR-0065) — ``{portfolio_id, scenario_ids, filters?, property_ids?,
recipient_email?}``. Empty dict when the task has no inputs."""
row = (
self._session.connection()
.execute(
text("SELECT inputs FROM tasks WHERE id = :task_id"),
{"task_id": task_id},
)
.first()
)
if row is None or row[0] is None:
return {}
parsed: Any = json.loads(row[0])
return cast(dict[str, Any], parsed) if isinstance(parsed, dict) else {}
def already_distributed(self, task_id: UUID) -> bool:
"""Whether the task already has a sub_task — the 409 guard against a
double-submit re-triggering the same export."""
row = (
self._session.connection()
.execute(
text("SELECT 1 FROM sub_task WHERE task_id = :task_id LIMIT 1"),
{"task_id": task_id},
)
.first()
)
return row is not None
def resolve_property_ids(
self,
*,
portfolio_id: int,
filters: PropertyGroupFilters,
property_ids: list[int],
) -> list[int]:
"""Resolve the selection to the distinct internal ``property.id`` set the
export is built from (ADR-0065). An explicit ``property_ids`` list is an
**override**, not a co-filter: when present it is used as given;
otherwise the ``filters`` are resolved against the portfolio through the
shared Modelling Run resolver (ADR-0056)."""
if property_ids:
return sorted(set(property_ids))
resolved = resolve_filtered_property_ids(
self._session, portfolio_id, filters
)
return sorted({filtered.property_id for filtered in resolved})
def create_export_subtask(
self, task_id: UUID, subtask_id: UUID, recipe: dict[str, Any]
) -> bool:
"""Pre-create the single ``waiting`` sub_task under the app-owned task,
pinning the recipe (resolved ``property_ids``, ``scenario_ids``,
``recipient_email``, ``export_name``) onto its inputs (ADR-0055/0065).
The insert is conditional on the task having no sub_task yet, so the DB
arbitrates a double-submit race. Returns whether this call created it."""
now = datetime.now(timezone.utc)
result = self._session.connection().execute(
text(
"INSERT INTO sub_task (id, task_id, status, inputs, updated_at)"
" SELECT :id, :task_id, 'waiting', :inputs, :updated_at"
" WHERE NOT EXISTS ("
" SELECT 1 FROM sub_task WHERE task_id = :task_id"
" )"
),
{
"id": subtask_id,
"task_id": task_id,
"inputs": json.dumps(recipe),
"updated_at": now,
},
)
self._session.commit()
return result.rowcount == 1

188
backend/app/exports/router.py Normal file
View file

@ -0,0 +1,188 @@
"""Scenario Export trigger: POST /v1/exports/scenario (ADR-0065).
Resolves the authenticated requester's email, reads the FE-written selection
recipe from ``task.inputs``, resolves + caps the property selection (an explicit
``property_ids`` list overrides the filters; otherwise the shared Modelling Run
filter resolver runs), pins the resolved recipe onto a single pre-created
sub_task under the app-owned task, and enqueues one message to the ara_export
worker. Never builds the workbook synchronously — the worker emails the link.
"""
import json
from collections.abc import Callable, Iterator
from typing import Any, cast
from uuid import uuid4
import boto3
from fastapi import APIRouter, Depends, HTTPException
from sqlmodel import Session
from backend.app.config import get_settings
from backend.app.db.connection import db_engine
from backend.app.dependencies import validate_jwt_token, validate_token
from backend.app.exports.export_tasks import ScenarioExportTasks
from backend.app.exports.schemas import ScenarioExportRequest
from backend.app.modelling.property_filters import PropertyGroupFilters
# The selection cap (ADR-0065) — rejected at the trigger so an oversized export
# never reaches the worker. Configurable ceiling; the JSON-column mechanism has
# no limit of its own.
MAX_PROPERTIES = 100_000
MessageSender = Callable[[str], None]
def get_session() -> Iterator[Session]:
with Session(db_engine) as session:
yield session
def get_message_sender() -> MessageSender:
settings = get_settings()
client: Any = cast(Any, boto3.client("sqs", settings.AWS_DEFAULT_REGION)) # pyright: ignore[reportUnknownMemberType]
queue_url = settings.ARA_EXPORT_SQS_URL
def send(body: str) -> None:
client.send_message(QueueUrl=queue_url, MessageBody=body)
return send
def get_requesting_user_email(user: Any = Depends(validate_jwt_token)) -> str:
"""The authenticated requester's email — the export recipient (ADR-0059)."""
email: Any = getattr(user, "email", None)
if email is None and isinstance(user, dict):
email = cast(dict[str, Any], user).get("email")
if not email and get_settings().ENVIRONMENT == "local":
email = "local-dev@example.com"
if not email:
raise HTTPException(
status_code=400,
detail="Could not determine the requesting user's email address.",
)
return str(email)
router = APIRouter(
prefix="/exports",
tags=["exports"],
dependencies=[Depends(validate_token)],
)
@router.post("/scenario", status_code=202)
async def scenario_export(
body: ScenarioExportRequest,
session: Session = Depends(get_session),
send_message: MessageSender = Depends(get_message_sender),
recipient_email: str = Depends(get_requesting_user_email),
) -> dict[str, str]:
tasks = ScenarioExportTasks(session)
if tasks.already_distributed(body.task_id):
raise HTTPException(
status_code=409,
detail=(
f"Task {body.task_id} already has a sub_task — an export has "
"already been started for it."
),
)
config = tasks.read_selection_config(body.task_id)
portfolio_id: Any = config.get("portfolio_id")
raw_scenario_ids: Any = config.get("scenario_ids") or []
raw_property_ids: Any = config.get("property_ids") or []
raw_filters: Any = config.get("filters") or {}
# ADR-0065 plan selection: "scenario" (per-Scenario sheets, the default) or
# "default" (one sheet of each home's is_default Plan). Absent → "scenario".
plan_selection: Any = config.get("plan_selection") or "scenario"
if not isinstance(portfolio_id, int):
raise HTTPException(
status_code=400, detail="task.inputs must provide an integer portfolio_id."
)
if plan_selection not in ("scenario", "default"):
raise HTTPException(
status_code=400,
detail='task.inputs plan_selection must be "scenario" or "default".',
)
# scenario_ids is required only for a per-Scenario export; the default-plan
# selection is scenario-independent (one default Plan per home).
if plan_selection == "scenario":
if not _is_int_list(raw_scenario_ids) or not raw_scenario_ids:
raise HTTPException(
status_code=400,
detail="task.inputs scenario_ids must be a non-empty list of "
"integers for a scenario export.",
)
elif raw_scenario_ids and not _is_int_list(raw_scenario_ids):
raise HTTPException(
status_code=400,
detail="task.inputs scenario_ids, if provided, must be integers.",
)
if not _is_int_list(raw_property_ids):
raise HTTPException(
status_code=400,
detail="task.inputs property_ids must be a list of integers.",
)
if not isinstance(raw_filters, dict):
raise HTTPException(
status_code=400, detail="task.inputs filters must be an object."
)
scenario_ids: list[int] = cast(list[int], raw_scenario_ids)
property_ids: list[int] = cast(list[int], raw_property_ids)
try:
filters = PropertyGroupFilters(**cast(dict[str, Any], raw_filters))
except Exception as exc: # pragma: no cover - pydantic validation detail
raise HTTPException(
status_code=400, detail=f"task.inputs filters are invalid: {exc}"
) from exc
resolved_property_ids = tasks.resolve_property_ids(
portfolio_id=portfolio_id, filters=filters, property_ids=property_ids
)
if not resolved_property_ids:
raise HTTPException(
status_code=400,
detail="The selection resolves to no properties.",
)
if len(resolved_property_ids) > MAX_PROPERTIES:
raise HTTPException(
status_code=400,
detail=(
f"The selection has {len(resolved_property_ids)} properties, over "
f"the {MAX_PROPERTIES} limit — narrow your selection."
),
)
subtask_id = uuid4()
recipe = {
"portfolio_id": portfolio_id,
"scenario_ids": scenario_ids,
"property_ids": resolved_property_ids,
"recipient_email": recipient_email,
"export_name": f"portfolio-{portfolio_id}-{body.task_id}",
"plan_selection": plan_selection,
}
created = tasks.create_export_subtask(body.task_id, subtask_id, recipe)
if not created:
raise HTTPException(
status_code=409,
detail=(
f"Task {body.task_id} already has a sub_task — an export has "
"already been started for it."
),
)
send_message(
json.dumps({"task_id": str(body.task_id), "subtask_id": str(subtask_id)})
)
return {"message": "Scenario export started"}
def _is_int_list(value: Any) -> bool:
# bool is an int subclass; exclude it so a stray True/False isn't a valid id.
return isinstance(value, list) and all(
isinstance(item, int) and not isinstance(item, bool)
for item in cast(list[Any], value)
)

12
backend/app/exports/schemas.py Normal file
View file

@ -0,0 +1,12 @@
from uuid import UUID
from pydantic import BaseModel
class ScenarioExportRequest(BaseModel):
"""The Scenario Export trigger body (ADR-0065). Carries only the app-owned
``task_id``: the FE wrote the selection recipe (portfolio, scenarios,
filters, optional property_ids) into ``task.inputs``, so the (potentially
large) selection never travels in the HTTP body."""
task_id: UUID

2
backend/app/main.py
View file

@ -12,6 +12,7 @@ from backend.app.plan import router as plan_router
from backend.app.tasks import router as tasks_router
from backend.app.bulk_uploads import router as bulk_uploads_router
from backend.app.documents import router as documents_router
from backend.app.exports import router as exports_router
from backend.app.modelling import router as modelling_router
from backend.app.dependencies import validate_api_key
from backend.app.config import get_settings
@ -67,6 +68,7 @@ app.include_router(whlg_router.router, prefix="/v1")
app.include_router(bulk_uploads_router.router, prefix="/v1")
app.include_router(modelling_router.router, prefix="/v1")
app.include_router(documents_router.router, prefix="/v1")
app.include_router(exports_router.router, prefix="/v1")
if get_settings().ENVIRONMENT == "local":
from backend.app.local import router as local_router

345
datatypes/epc/domain/mapper.py
View file

@ -1,5 +1,6 @@
import copy
import logging
import math
import re
from dataclasses import replace
from datetime import date
@ -212,6 +213,35 @@ def _sap_opening_area_m2(width: Any, height: Any) -> float:
)
def _normalised_main_heating_fraction(
raw: Optional[Union[int, float]],
all_fractions: Sequence[Optional[Union[int, float]]],
) -> Optional[int]:
"""Return `raw` as a canonical PERCENT (RdSAP 10 item 7-5, spec p.81).
The gov-EPC API lodges a two-main dwelling's `main_heating_fraction` pair in
two incompatible encodings — a DECIMAL fraction summing to ~1.0
(`[0.8, 0.2]`) or an integer PERCENT summing to ~100 (`[80, 20]`) — and every
consumer divides by 100 unconditionally. A decimal pair therefore collapses
the second system's share to ~1% of its true value, billing almost the whole
load to the first system's fuel and efficiency (#1666).
Discriminate on the pair sum: a genuine two-main split summing nearer 1.0
than 100 is decimal-encoded and scaled to percent; a percent-encoded pair
(sum ~100) and any single main are left exactly as lodged, so no
percent-encoded cert regresses. Returned as an int to match the domain
invariant (`MainHeatingDetail.main_heating_fraction` is a 0-100 percent).
"""
if raw is None:
return None
present = [float(f) for f in all_fractions if f is not None]
if len(present) >= 2:
pair_sum = sum(present)
if abs(pair_sum - 1.0) < abs(pair_sum - 100.0):
return round(float(raw) * 100.0)
return round(float(raw))
def _pv_array_field(array: Any, name: str) -> Any:
"""Read a PV-array field from either a `PhotovoltaicArray` dataclass (21.0.x,
where the schema Union parsed the nested list) or a raw dict (older schemas
@ -753,7 +783,13 @@ class EpcPropertyDataMapper:
main_heating_number=d.main_heating_number,
main_heating_control=d.main_heating_control,
main_heating_category=d.main_heating_category,
main_heating_fraction=d.main_heating_fraction,
main_heating_fraction=_normalised_main_heating_fraction(
d.main_heating_fraction,
[
x.main_heating_fraction
for x in schema.sap_heating.main_heating_details
],
),
sap_main_heating_code=d.sap_main_heating_code,
central_heating_pump_age=d.central_heating_pump_age,
main_heating_data_source=d.main_heating_data_source,
@ -1385,7 +1421,13 @@ class EpcPropertyDataMapper:
main_heating_number=d.main_heating_number,
main_heating_control=d.main_heating_control,
main_heating_category=d.main_heating_category,
main_heating_fraction=d.main_heating_fraction,
main_heating_fraction=_normalised_main_heating_fraction(
d.main_heating_fraction,
[
x.main_heating_fraction
for x in schema.sap_heating.main_heating_details
],
),
sap_main_heating_code=d.sap_main_heating_code,
central_heating_pump_age=None,
main_heating_data_source=d.main_heating_data_source,
@ -1573,7 +1615,13 @@ class EpcPropertyDataMapper:
main_heating_number=d.main_heating_number,
main_heating_control=d.main_heating_control,
main_heating_category=d.main_heating_category,
main_heating_fraction=d.main_heating_fraction,
main_heating_fraction=_normalised_main_heating_fraction(
d.main_heating_fraction,
[
x.main_heating_fraction
for x in schema.sap_heating.main_heating_details
],
),
sap_main_heating_code=d.sap_main_heating_code,
central_heating_pump_age=d.central_heating_pump_age,
main_heating_data_source=d.main_heating_data_source,
@ -1799,7 +1847,13 @@ class EpcPropertyDataMapper:
main_heating_number=d.main_heating_number,
main_heating_control=d.main_heating_control,
main_heating_category=d.main_heating_category,
main_heating_fraction=d.main_heating_fraction,
main_heating_fraction=_normalised_main_heating_fraction(
d.main_heating_fraction,
[
x.main_heating_fraction
for x in schema.sap_heating.main_heating_details
],
),
sap_main_heating_code=d.sap_main_heating_code,
central_heating_pump_age=d.central_heating_pump_age,
main_heating_data_source=d.main_heating_data_source,
@ -2052,7 +2106,13 @@ class EpcPropertyDataMapper:
main_heating_number=d.main_heating_number,
main_heating_control=d.main_heating_control,
main_heating_category=d.main_heating_category,
main_heating_fraction=d.main_heating_fraction,
main_heating_fraction=_normalised_main_heating_fraction(
d.main_heating_fraction,
[
x.main_heating_fraction
for x in schema.sap_heating.main_heating_details
],
),
sap_main_heating_code=d.sap_main_heating_code,
central_heating_pump_age=d.central_heating_pump_age,
main_heating_data_source=d.main_heating_data_source,
@ -2285,7 +2345,13 @@ class EpcPropertyDataMapper:
boiler_ignition_type=d.boiler_ignition_type,
main_heating_control=d.main_heating_control,
main_heating_category=d.main_heating_category,
main_heating_fraction=d.main_heating_fraction,
main_heating_fraction=_normalised_main_heating_fraction(
d.main_heating_fraction,
[
x.main_heating_fraction
for x in schema.sap_heating.main_heating_details
],
),
sap_main_heating_code=d.sap_main_heating_code,
central_heating_pump_age=d.central_heating_pump_age,
main_heating_data_source=d.main_heating_data_source,
@ -2634,7 +2700,13 @@ class EpcPropertyDataMapper:
boiler_ignition_type=d.boiler_ignition_type,
main_heating_control=d.main_heating_control,
main_heating_category=d.main_heating_category,
main_heating_fraction=d.main_heating_fraction,
main_heating_fraction=_normalised_main_heating_fraction(
d.main_heating_fraction,
[
x.main_heating_fraction
for x in schema.sap_heating.main_heating_details
],
),
sap_main_heating_code=d.sap_main_heating_code,
central_heating_pump_age=d.central_heating_pump_age,
main_heating_data_source=d.main_heating_data_source,
@ -3130,13 +3202,16 @@ class EpcPropertyDataMapper:
)
return None
return _clear_basement_flag_when_system_built(
_with_renewable_heat_incentive(
_with_recorded_performance(
_drop_building_parts_without_age_band(mapped), data
),
data,
)
return _with_internal_dimensions(
_clear_basement_flag_when_system_built(
_with_renewable_heat_incentive(
_with_recorded_performance(
_drop_building_parts_without_age_band(mapped), data
),
data,
)
),
data,
)
@ -3323,10 +3398,12 @@ def _sap_17_1_heating(schema: SapSchema17_1) -> SapHeating:
main_heating_index_number=d.main_heating_index_number,
main_heating_number=d.main_heating_number,
main_heating_category=d.main_heating_category,
main_heating_fraction=(
int(d.main_heating_fraction)
if d.main_heating_fraction is not None
else None
# #1666: normalise the encoding (and fix the old `int()` floor
# that silently deleted a decimal-lodged second main, e.g.
# int(0.35) -> 0) via the shared pair-sum discriminator.
main_heating_fraction=_normalised_main_heating_fraction(
d.main_heating_fraction,
[x.main_heating_fraction for x in sh.main_heating_details],
),
main_heating_data_source=d.main_heating_data_source,
)
@ -3351,6 +3428,181 @@ def _sap_back_solved_habitable_rooms(schema: SapSchema17_1) -> int:
)
# ---------------------------------------------------------------------------
# RdSAP 10 §3.4 conversion of external to internal dimensions (#1669)
# ---------------------------------------------------------------------------
# `measurement_type` lodged value: 1 = internal dimensions, 2 = external.
_EXTERNAL_MEASUREMENT: Final[int] = 2
def _table_3_row(*vals: int) -> Dict[str, int]:
"""Expand a Table 3 wall-thickness row's 10 spec columns to the 13 age
bands: bands I/J/K share the 9th column and L/M share the 10th."""
a, b, c, d, e, f, g, h, ijk, lm = vals
return {
"A": a, "B": b, "C": c, "D": d, "E": e, "F": f, "G": g, "H": h,
"I": ijk, "J": ijk, "K": ijk, "L": lm, "M": lm,
}
# RdSAP 10 Table 3 (PDF p.19): default main-dwelling wall thickness (mm) by wall
# construction row and age band, used only when the thickness is not lodged.
_TABLE_3_WALL_THICKNESS_MM: Final[Dict[str, Dict[str, int]]] = {
"solid": _table_3_row(220, 220, 220, 220, 240, 250, 270, 270, 300, 300),
"cavity": _table_3_row(250, 250, 250, 250, 250, 260, 270, 270, 300, 300),
"timber": _table_3_row(150, 150, 150, 250, 270, 270, 270, 270, 300, 300),
"cob": _table_3_row(540, 540, 540, 540, 540, 540, 560, 560, 590, 590),
"system": _table_3_row(250, 250, 250, 250, 250, 300, 300, 300, 300, 300),
}
# `wall_construction` code → Table 3 row. Table 3 has no stone row (stone is
# normally measured); stone (1/2) and any non-masonry/unknown fall back to the
# solid-masonry column.
_WALL_CONSTRUCTION_TO_TABLE_3_ROW: Final[Dict[int, str]] = {
1: "solid", 2: "solid", 3: "solid", # stone/granite, sandstone, solid brick
4: "cavity", 5: "timber", 6: "system", 7: "cob",
}
def _main_wall_thickness_m(epc: EpcPropertyData) -> Optional[float]:
"""The main dwelling's wall thickness in metres (RdSAP 10 §3.4 `w`): the
lodged value if present, else the Table 3 default for the main part's wall
construction + age band. None when neither is derivable."""
parts = epc.sap_building_parts
if not parts:
return None
main = parts[0]
if main.wall_thickness_mm is not None:
return main.wall_thickness_mm / 1000.0
construction = main.wall_construction
row = (
_WALL_CONSTRUCTION_TO_TABLE_3_ROW.get(construction, "solid")
if isinstance(construction, int)
else "solid"
)
band = (main.construction_age_band or "").strip().upper()[:1]
mm = _TABLE_3_WALL_THICKNESS_MM[row].get(band)
return mm / 1000.0 if mm is not None else None
def _table_2_ratios(
built_form: Optional[str], area_ext: float, perim_ext: float, w: float
) -> tuple[float, float]:
"""RdSAP 10 Table 2 (PDF p.18) whole-dwelling external→internal
(area_ratio, perimeter_ratio) by built form (1 detached, 2 semi, 3
end-terrace, 4 mid-terrace, 5 enclosed end-terrace, 6 enclosed mid-terrace).
`area_ext`/`perim_ext` are the ground-floor FOOTPRINT (§3.4 Note 4: one ratio
for the whole dwelling). Returns (1, 1) — no conversion — for an unrecognised
form or a degenerate footprint."""
if area_ext <= 0.0 or perim_ext <= 0.0:
return (1.0, 1.0)
bf = (built_form or "").strip()
if bf == "1": # detached
perim_int = perim_ext - 8.0 * w
area_int = area_ext - w * perim_int - 4.0 * w * w
elif bf in ("2", "3"): # semi-detached / end-terrace
if perim_ext**2 >= 8.0 * area_ext:
perim_int = perim_ext - 5.0 * w
a = 0.5 * (perim_ext - math.sqrt(perim_ext**2 - 8.0 * area_ext))
area_int = area_ext - w * (perim_ext + 0.5 * a) + 3.0 * w * w
else:
perim_int = perim_ext - 3.0 * w
area_int = area_ext - w * perim_ext + 3.0 * w * w
elif bf == "4": # mid-terrace
perim_int = perim_ext - 2.0 * w
area_int = area_ext - w * (perim_ext + 2.0 * area_ext / perim_ext) + 2.0 * w * w
elif bf == "5": # enclosed end-terrace
perim_int = perim_ext - 3.0 * w
area_int = area_ext - 1.5 * w * perim_ext + 2.25 * w * w
elif bf == "6": # enclosed mid-terrace
perim_int = perim_ext - w
area_int = area_ext - w * (area_ext / perim_ext + 1.5 * perim_ext) + 1.5 * w * w
else:
return (1.0, 1.0)
if area_int <= 0.0 or perim_int <= 0.0:
return (1.0, 1.0)
return (area_int / area_ext, perim_int / perim_ext)
def _domain_total_floor_area_m2(building_parts: List[SapBuildingPart]) -> float:
"""Sum the domain floor-dimension areas (plus each part's always-internal
room-in-roof floor area) — used to refresh `total_floor_area_m2` after a
§3.4 conversion, since `water_heating_from_cert` reads that scalar for
occupancy N."""
total = 0.0
for bp in building_parts:
for fd in bp.sap_floor_dimensions:
total += fd.total_floor_area_m2 or 0.0
rir = bp.sap_room_in_roof
if rir is not None and rir.floor_area:
total += rir.floor_area
return total
def _with_internal_dimensions(
epc: EpcPropertyData, data: Dict[str, Any]
) -> EpcPropertyData:
"""Plumb the lodged `measurement_type` onto `epc` and, when the horizontal
dimensions were measured externally (== 2), convert them to internal
dimensions per RdSAP 10 §3.4 + Table 2 before they reach the cascade (#1669).
§3.4 Note 4: ONE whole-dwelling area/perimeter ratio (from the ground-floor
footprint) is applied to every storey of every part — not a per-storey
conversion, which over-corrects multi-part mid-terraces. Room-in-roof floor
area is always measured internally (§3.1) and is left untouched; party wall
lengths are reduced by 2w (Table 2 Note 5). Internal certs are unchanged.
Scope: the conversion is applied only to the RdSAP-Schema reduced-data family
(17.0-21.x), where per-storey external dimensions are the lodgement standard
and the fix is corpus-validated (RdSAP-21.0.1). `measurement_type` is still
plumbed for every schema, but the legacy SAP-Schema-15.0/16.x certs — whose
per-storey areas do not reconcile to the lodged TFA — are left unconverted."""
epc.measurement_type = data.get("measurement_type")
schema_type = data.get("schema_type", "")
if (
epc.measurement_type != _EXTERNAL_MEASUREMENT
or not isinstance(schema_type, str)
or not schema_type.startswith("RdSAP-Schema-")
or not epc.sap_building_parts
):
return epc
w = _main_wall_thickness_m(epc)
if w is None:
return epc
ground_levels = [
fd.floor
for bp in epc.sap_building_parts
for fd in bp.sap_floor_dimensions
if fd.floor is not None
]
if not ground_levels:
return epc
ground = min(ground_levels)
area_ext = sum(
fd.total_floor_area_m2 or 0.0
for bp in epc.sap_building_parts
for fd in bp.sap_floor_dimensions
if fd.floor == ground
)
perim_ext = sum(
fd.heat_loss_perimeter_m or 0.0
for bp in epc.sap_building_parts
for fd in bp.sap_floor_dimensions
if fd.floor == ground
)
area_ratio, perim_ratio = _table_2_ratios(epc.built_form, area_ext, perim_ext, w)
if area_ratio == 1.0 and perim_ratio == 1.0:
return epc
for bp in epc.sap_building_parts:
for fd in bp.sap_floor_dimensions:
fd.total_floor_area_m2 *= area_ratio
fd.heat_loss_perimeter_m *= perim_ratio
if fd.party_wall_length_m:
fd.party_wall_length_m = max(0.0, fd.party_wall_length_m - 2.0 * w)
epc.total_floor_area_m2 = _domain_total_floor_area_m2(epc.sap_building_parts)
return epc
def _with_recorded_performance(
epc: EpcPropertyData, data: Dict[str, Any]
) -> EpcPropertyData:
@ -5269,24 +5521,67 @@ _API_GLAZING_TYPE_GAP_TO_TRANSMISSION: Dict[
}
# RdSAP 10 Table 24 METAL-frame U-column (PDF p.50-51) — #1667. Only the U
# differs from the PVC/wooden column above (the glazing g is frame-independent);
# the frame factor becomes 0.8 (SAP 10.2 Table 6c) instead of 0.70. Secondary
# glazing (types 4/11/12) has no separate metal U — its combined U is
# frame-independent — so it is absent here and keeps the PVC-column U. Values
# mirror `u_window`'s metal column (`rdsap_uvalues.py`): single 5.7; DG pre-2002
# 6/12/16 = 3.7/3.4/3.3; triple pre-2002 6/12/16 = 2.9/2.6/2.5; DG/triple 2002+
# = 2.2; 2022+ = 1.6.
_API_GLAZING_TYPE_TO_METAL_U: Dict[int, float] = {
1: 3.4, 3: 3.4, 7: 3.4, # DG pre-2002 / unknown-date, 12mm default
2: 2.2, 9: 2.2, # DG (2) / triple (9) 2002+ (pre-2022)
13: 1.6, 14: 1.6, # DG / DG-or-triple 2022+
5: 5.7, 15: 5.7, # single glazing
6: 2.6, 8: 2.6, 10: 2.6, # triple pre-2002 / unknown / known, 12mm default
}
_API_GLAZING_TYPE_GAP_TO_METAL_U: Dict[tuple[int, object], float] = {
(1, 6): 3.7, (1, 12): 3.4, (1, "16+"): 3.3,
(3, 6): 3.7, (3, 12): 3.4, (3, "16+"): 3.3,
(7, 6): 3.7, (7, 12): 3.4, (7, "16+"): 3.3,
(6, 6): 2.9, (6, 12): 2.6, (6, "16+"): 2.5,
(8, 6): 2.9, (8, 12): 2.6, (8, "16+"): 2.5,
(10, 6): 2.9, (10, 12): 2.6, (10, "16+"): 2.5,
}
# SAP 10.2 Table 6c window frame factor: 0.8 for metal, 0.70 for PVC/wooden.
_METAL_FRAME_FACTOR: Final[float] = 0.8
def _api_glazing_transmission(
glazing_type: Optional[int],
glazing_gap: object,
pvc_frame: object = None,
) -> Optional[tuple[float, float, float]]:
"""Resolve (U, g, frame_factor) for an API window from RdSAP 10 Table 24.
Per-gap override takes precedence over the type-only default. Returns None
only when `glazing_type` is absent (None → cascade default). A glazing_type
PRESENT but unmapped raises `UnmappedApiCode` rather than silently routing
the window to the u_window all-None default U=2.5 — the forcing function
that surfaced single-glazing (code 5 → 4.8) instead of letting it hide."""
that surfaced single-glazing (code 5 → 4.8) instead of letting it hide.
`pvc_frame` is the lodged RdSAP input 6-5 boolean (spec p.80): "true" =
wooden/PVC (grouped in Table 24), "false" = metal. Only an explicit "false"
routes the window to the Table 24 metal U-column + FF 0.8 (#1667); "true",
None or any other value keeps the PVC/wooden column + FF 0.70, so no
PVC-lodged cert regresses."""
if glazing_type is None:
return None
gap_key = (glazing_type, glazing_gap)
if gap_key in _API_GLAZING_TYPE_GAP_TO_TRANSMISSION:
return _API_GLAZING_TYPE_GAP_TO_TRANSMISSION[gap_key]
if glazing_type in _API_GLAZING_TYPE_TO_TRANSMISSION:
return _API_GLAZING_TYPE_TO_TRANSMISSION[glazing_type]
raise UnmappedApiCode("glazing_type", glazing_type)
base = _API_GLAZING_TYPE_GAP_TO_TRANSMISSION[gap_key]
elif glazing_type in _API_GLAZING_TYPE_TO_TRANSMISSION:
base = _API_GLAZING_TYPE_TO_TRANSMISSION[glazing_type]
else:
raise UnmappedApiCode("glazing_type", glazing_type)
if pvc_frame != "false":
return base
u_pvc, g_perp, _pvc_ff = base
metal_u = _API_GLAZING_TYPE_GAP_TO_METAL_U.get(
gap_key, _API_GLAZING_TYPE_TO_METAL_U.get(glazing_type, u_pvc)
)
return (metal_u, g_perp, _METAL_FRAME_FACTOR)
# GOV.UK RdSAP 21 `glazing_type` integer → SAP 10.2 Table 6b cascade
@ -5405,7 +5700,7 @@ def _api_sap_roof_window(w: Any) -> SapRoofWindow:
Table 6e Note 2 inclination adjustment (+0.30 W/m²K at 45° pitch) to
the inclined-position value the worksheet bills on (27a). Mirror of
the site-notes `_map_elmhurst_roof_window`."""
transmission = _api_glazing_transmission(w.glazing_type, w.glazing_gap)
transmission = _api_glazing_transmission(w.glazing_type, w.glazing_gap, w.pvc_frame)
vertical_u = transmission[0] if transmission is not None else 2.0
g_perp = transmission[1] if transmission is not None else 0.76
frame_factor = w.frame_factor
@ -5475,7 +5770,7 @@ def _api_sap_window(w: Any) -> SapWindow:
to the SAP 10.2 Table 6b cascade enum so the cascade's daylight g_L
lookup picks the spec-correct value (e.g. RdSAP 21 code 1 = DG
pre-2002, cascade g_L=0.80, not single-glazed 0.90)."""
transmission = _api_glazing_transmission(w.glazing_type, w.glazing_gap)
transmission = _api_glazing_transmission(w.glazing_type, w.glazing_gap, w.pvc_frame)
frame_factor: Optional[float] = w.frame_factor
if frame_factor is None and transmission is not None:
frame_factor = transmission[2]

84
deployment/terraform/lambda/ara_export/main.tf Normal file
View file

@ -0,0 +1,84 @@
############################################
# ara_export Lambda (ADR-0065) — builds one branded sheet-per-scenario .xlsx
# from persisted modelling data, uploads it to the exports bucket, and emails a
# presigned link. Reads only the DB (no source-bucket reads, unlike
# bulk_document_download), so the role needs only exports-bucket write/presign.
############################################
data "aws_secretsmanager_secret_version" "db_credentials" {
secret_id = "${var.stage}/assessment_model/db_credentials"
}
data "aws_secretsmanager_secret_version" "ses_smtp" {
secret_id = "${var.stage}/ses/smtp_credentials"
}
locals {
db_credentials = jsondecode(data.aws_secretsmanager_secret_version.db_credentials.secret_string)
ses_smtp = jsondecode(data.aws_secretsmanager_secret_version.ses_smtp.secret_string)
document_exports_bucket = "retrofit-document-exports-${var.stage}"
}
############################################
# Lambda + SQS queue + trigger
############################################
module "lambda" {
source = "../../modules/lambda_with_sqs"
name = var.lambda_name
stage = var.stage
image_uri = local.image_uri
reserved_concurrent_executions = var.reserved_concurrent_executions
batch_size = var.batch_size
maximum_concurrency = var.maximum_concurrency
timeout = 900
# Normal-mode openpyxl holds the workbook in memory (ADR-0065); 4 GB covers the
# 100k-property ceiling. Raise here (then switch to write_only) if it OOMs.
memory_size = 4096
environment = {
STAGE = var.stage
LOG_LEVEL = "info"
POSTGRES_USERNAME = local.db_credentials.db_assessment_model_username
POSTGRES_PASSWORD = local.db_credentials.db_assessment_model_password
POSTGRES_HOST = var.db_host
POSTGRES_DATABASE = var.db_name
POSTGRES_PORT = var.db_port
DOCUMENT_EXPORTS_BUCKET = local.document_exports_bucket
# SES SMTP — IAM-user credentials sourced from Secrets Manager (modules/ses).
SES_SMTP_HOST = "email-smtp.eu-west-2.amazonaws.com"
SES_SMTP_PORT = "587"
SES_SMTP_USERNAME = local.ses_smtp.username
SES_SMTP_PASSWORD = local.ses_smtp.password
SES_SMTP_FROM_ADDRESS = var.ses_from_address
}
}
############################################
# IAM: write + presign-read the workbook on the dedicated exports bucket.
# GetObject is required so the Lambda-signed presigned GET URL resolves.
############################################
module "s3_exports" {
source = "../../modules/s3_iam_policy"
policy_name = "AraExportS3Exports-${var.stage}"
policy_description = "Allow ara_export Lambda to write and presign-read Scenario Exports on the exports bucket"
bucket_arns = ["arn:aws:s3:::${local.document_exports_bucket}"]
actions = ["s3:PutObject", "s3:GetObject"]
resource_paths = ["/*"]
}
resource "aws_iam_role_policy_attachment" "s3_exports" {
role = module.lambda.role_name
policy_arn = module.s3_exports.policy_arn
}
# NOTE: no ses:* on the role — the handler sends over SMTP with the SES IAM-user
# credentials injected above.

9
deployment/terraform/lambda/ara_export/outputs.tf Normal file
View file

@ -0,0 +1,9 @@
output "ara_export_queue_url" {
value = module.lambda.queue_url
description = "URL of the ara-export SQS queue (FastAPI enqueues jobs here)"
}
output "ara_export_queue_arn" {
value = module.lambda.queue_arn
description = "ARN of the ara-export SQS queue"
}

20
deployment/terraform/lambda/ara_export/provider.tf Normal file
View file

@ -0,0 +1,20 @@
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.0"
}
}
backend "s3" {
bucket = "ara-export-terraform-state"
key = "terraform.tfstate"
region = "eu-west-2"
}
required_version = ">= 1.2.0"
}
provider "aws" {
region = "eu-west-2"
}

61
deployment/terraform/lambda/ara_export/variables.tf Normal file
View file

@ -0,0 +1,61 @@
variable "lambda_name" {
type = string
description = "Logical name of the lambda"
}
variable "stage" {
description = "Deployment stage (e.g. dev, prod)"
type = string
}
variable "ecr_repo_url" {
type = string
description = "ECR repository URL (no tag, no digest)"
}
variable "image_digest" {
type = string
description = "Image digest (sha256:...)"
}
variable "reserved_concurrent_executions" {
type = number
default = -1
description = "Reserved concurrency for the Lambda. -1 = unreserved."
}
variable "maximum_concurrency" {
type = number
default = 5
description = "Maximum concurrent Lambda invocations from the SQS trigger."
}
variable "batch_size" {
type = number
default = 1
}
variable "db_host" {
type = string
sensitive = true
}
variable "db_name" {
type = string
sensitive = true
}
variable "db_port" {
type = string
sensitive = true
}
variable "ses_from_address" {
type = string
description = "Verified SES sender for Scenario Export notifications (ADR-0059)."
default = "noreply@domna.homes"
}
locals {
image_uri = "${var.ecr_repo_url}@${var.image_digest}"
}

237
docs/adr/0065-ara-scenario-export.md Normal file
View file

@ -0,0 +1,237 @@
---
status: accepted (builds on ADR-0055, ADR-0056, ADR-0059, ADR-0060)
---
# ARA Scenario Export builds one branded workbook per request, sheet-per-scenario
Users need to pull a portfolio's modelling results — property information plus
each Property's **Plan** under one or more **Scenarios** — out of the product as
a single spreadsheet, initiated from the front end and delivered as a link
emailed to the requester. This is the **Export** the glossary already reserves
the word for ("the plan/scenario XLSX export"), distinct from a **Download
Package** (the ZIP of survey documents, ADR-0060).
The data layer already half-exists as `scripts/data_exports.py` (Jun-te Kim): a
CLI over the new DDD schema that reads `property` / `scenario` / `plan` /
`recommendation` / `material`, pivots measures onto a **stable 21-measure column
contract** (`EXPECTED_MEASURE_COLUMNS`), and emits one sheet per scenario. But
as written it (a) fetches EPC descriptive fields **live** from the gov API per
property (the retired `property_details_epc` table's replacement), (b) has **no
filters** and auto-derives every scenario with plans, (c) lives in `scripts/`
outside the DDD layout, and (d) has no styling, S3, or email. None of those
survive a front-end feature that must scope by scenario + filters and scale to
**tens of thousands** of properties inside a Lambda.
The trigger/email/large-selection plumbing is already solved by
**Bulk Document Download** (ADR-0055/0059/0060) and is reused wholesale.
## Decision
**A request produces exactly one branded `.xlsx` Export: one sheet per selected
Scenario, over a filtered, scenario-scoped Property set, built from persisted
data, on the app-owned-task + attach-mode lane (ADR-0055), delivered by
presigned URL + email (ADR-0059).**
- **Scenario-scoped selection (model A).** The unit of scope is *a Scenario's
results*. The base set for each sheet is the Properties that have a **Plan**
under that Scenario; filters and any hand-picked ids only **narrow** it. A
Property that filters in but has **no Plan for that Scenario drops out** of
that sheet — there is nothing to export for it. Scenario membership is applied
in the data layer (the `plan` join), not in the filter resolver.
- **Sheet-per-Scenario, one workbook (A2).** The recipe carries
`scenario_ids: list[int]`; the **same** filtered Property set is rendered once
per Scenario, each on its own sheet. Under model A the sheets may legitimately
have **different row counts** (a Property with a Plan under Scenario X but not
Y appears on the X sheet only). Excel-only for v1 — a sheet-per-scenario
workbook is an inherently Excel artifact; there is no `format` field (or it is
fixed to `"xlsx"`). CSV degradation is a fast-follow if a warehouse consumer
ever needs it.
- **Trigger & lifecycle (mirror ADR-0055).** The **front end** creates the
app-owned `tasks` row and writes the **selection recipe** —
`{portfolio_id, scenario_ids, filters, property_ids?, recipient_email}` — into
`tasks.inputs` (JSON, FE-owned), then calls the FastAPI route
(`backend/app/exports/`) with **only the `task_id`**. The route reads the
recipe from `tasks.inputs`, **resolves** it to the concrete internal
`property.id` set (below), **pins the resolved id list + recipe** onto **one
pre-created `sub_task`'s `inputs`**, and drops one SQS message
(`task_id`, `sub_task_id`) on a new `ARA_EXPORT_SQS_URL` queue. The new
`applications/ara_export` Lambda runs in **attach mode**, reading that recipe
from the sub_task; `TaskOrchestrator` owns status + roll-up. Nothing large
ever travels in an HTTP body **or** on SQS — the id list rides
`tasks.inputs` → `sub_task.inputs` (both Postgres JSON), so the 256 KB SQS
limit is irrelevant by construction and the "tens of thousands of ids"
concern is structurally impossible.
- **Filters are the ADR-0056 shared contract.** The route resolves `filters`
with the **same** `PropertyGroupFilters` model and
`resolve_filtered_property_ids(session, portfolio_id, filters)` function the
Modelling Run Distributor uses — one filter vocabulary, one resolver, two
triggers. New filter dimensions land on `PropertyGroupFilters` and both
triggers inherit them. `property_ids` is an **override, not a co-filter**:
present and non-empty ⇒ export exactly those ids; absent ⇒ resolve from
`filters`. Precedence (not intersection) avoids the "I ticked 5 rows, got 3"
surprise where a lingering filter silently drops hand-picked rows. Both paths
key on internal `property.id` (as `data_exports.py` and the modelling resolver
already do), so identifiers align.
- **Persisted data only — the Effective EPC, no live API.** Because model A
guarantees every exported Property has a Plan, it has been through Ingestion →
Baseline → Modelling and its cert / Effective EPC / Baseline Performance /
Plans are all persisted. The Export is a pure read. Descriptive fields follow
the **Effective EPC** precedence — `property_overrides` → lodged EPC →
predicted EPC — the same ladder `property_filters._resolved_component_values`
already implements, widened from `property_type`/`built_form` to the full
descriptive set. **No live `EpcClientService` calls**: tens of thousands of
sequential gov-API fetches would blow the Lambda's 15-minute ceiling and
hammer the register.
- **Promote the data layer into DDD, split by concern.** A `repositories/`
read repository returns the persisted, override-resolved rows for a
`(portfolio_id, scenario_id, property-id set)`; a **pure `domain/`
column-shaper** owns the frozen 21-measure contract and the
measure→cost-column pivot + savings summation (trivially unit-testable). The
`applications/ara_export` Lambda composes: repo read → domain shape → styling
→ S3 → email. `scripts/data_exports.py` becomes a thin CLI wrapper over the
same modules (or is retired) so the query/column logic exists **once** — this
also removes the `_PROPERTY_TYPE_CODES` map duplicated today in both
`data_exports.py` and `property_filters.py`.
- **Frozen column contract, mimic `data_exports.py`.** The 21-measure
`EXPECTED_MEASURE_COLUMNS` contract is **stable** — every measure column
always present, blank where a Property lacks that measure — rather than
emitting only the measures that happen to appear. Property identity + Effective
EPC descriptive fields + pivoted per-measure cost columns + summed savings +
Plan post-works figures (SAP / kWh / CO₂ / bill), per sheet.
- **Programmatic branding, palette in `infrastructure/`.** Styling is a
**programmatic openpyxl** pass (header fill + bold header font, frozen top
row, banded rows, coloured cost/band columns, column widths), not a populated
template. A2's **dynamic sheet count** defeats a fixed-sheet template (which
would force lossy per-sheet `copy_worksheet` cloning); a helper styles
whatever it generates. The brand palette + styling helper live in
`infrastructure/xlsx/` — filling the "no brand-colour constants exist
anywhere" gap once, reusable enough that `audit_generator` could later adopt
it. `domain/` stays pure and colour-agnostic. Monday.com-style **defaults**
ship now behind a `# TODO(brand)` marker until real Domna hex values land.
- **Caps & memory.** The bulk-download `MAX_PROPERTIES = 500` (a ZIP-of-GBs
policy) is dropped; the Export payload is one `.xlsx` of mostly numbers. The
route rejects `> ARA_EXPORT_MAX_PROPERTIES` (**100 000**, configurable)
synchronously with a legible "narrow your selection" error. The Lambda builds
in **normal openpyxl mode on a 4 GB-memory Lambda** to start, streamed to a
temp file on `/tmp` and multipart-uploaded (`S3Client.upload_file`), never an
in-memory `BytesIO`. The 100k ceiling in normal mode may need more memory
(openpyxl holds every cell as a Python object — file size is **not** the RAM
gauge); the fallback if it OOMs is to raise the Lambda memory, then switch to
`write_only` mode (which constrains the styling helper to style-at-write). The
common case (hundreds–low-thousands per portfolio) is comfortably within
4 GB.
- **Delivery — both channels (ADR-0059).** The workbook is written to the
dedicated `DOCUMENT_EXPORTS_BUCKET` (reused, not the shared `DATA_BUCKET`);
the presigned URL (60-minute expiry, `_URL_TTL_SECONDS = 3600`) and a
summary are written to `sub_task.outputs` **and** emailed via
`SesSmtpEmailSender` to `recipient_email`. Email send is **best-effort**
(wrapped in try/except — a delivery failure does not fail the task; the URL is
still on `sub_task.outputs`, which the FE polls).
- **Best-effort rows.** A Property/Plan that fails to resolve is skipped and
recorded in `sub_task.outputs`, not a hard failure. The run **fails only** on
an infrastructure error (DB / S3 / workbook / upload) or when the whole
selection yields **zero exportable rows**.
- **Idempotency.** The `sub_task` is created with the same idempotent
`INSERT ... WHERE NOT EXISTS` the DB arbitrates for double-submit
(`create_download_subtask` pattern).
## Deferred / future work
- **Tag-based filtering.** The next filter the front end wants is **tags**, but
there is no tag model in the backend (no `property_tags` table, no
override→EPC→predicted resolution ladder for it) — it is a new concept that
needs its own `/grill-with-docs` before landing on `PropertyGroupFilters`.
Once designed, both the Export and the Modelling Run triggers inherit it via
the ADR-0056 shared contract. A `# TODO(tags)` marker at the filter seam
points back to this ADR.
- **CSV output.** Only if a warehouse/finance consumer needs a flat feed;
would collapse the sheets to one table with a `scenario_id` column, unbranded.
- **`write_only` streaming** if the 100k ceiling proves too heavy for a
reasonably-sized Lambda in normal mode.
- **Exports-bucket retention** — the Exports are transient; an expiry lifecycle
rule is likely warranted but is left to a follow-up (as with ADR-0060).
## Considered options
- **Property-scoped export (model B)** — resolve Properties first, attach
scenario columns, keep no-Plan Properties with blank scenario cells. Rejected:
the feature is "give me the modelled results of Scenario X", so a Property with
no Plan for the Scenario has nothing to show; blanks would mislead.
- **One Scenario per export (A1)** — simplest contract, single flat result.
Rejected: the FE screen lets users tick several Scenarios for comparison, which
is exactly the sheet-per-Scenario workbook.
- **Import `scripts/data_exports.py` from the Lambda.** Rejected: re-buries raw
SQL in `scripts/`, keeps the dead live-EPC dependency, duplicates the
code-maps, and can't sit cleanly in the DDD layout.
- **Live EPC fetch (as the script does).** Rejected: tens of thousands of
sequential gov-API calls won't finish in a Lambda and abuse the register;
model A guarantees the data is already persisted.
- **`property_ids` intersects `filters`.** Rejected: a hand-picked selection
silently shrinking because of a stale filter is a surprising, hard-to-debug
UX; override precedence is predictable.
- **Populate a branded `.xlsx` template** (the `audit_generator` pattern).
Rejected for A2: a fixed-sheet template can't hold a dynamic number of
scenario sheets without lossy runtime `copy_worksheet` cloning, and couples the
deploy to a binary asset.
- **`write_only` from day one.** Deferred: it constrains styling to
style-at-write for no benefit at the common scale; adopt only if 4 GB normal
mode proves insufficient at the ceiling.
## Consequences
- New DDD pieces: `applications/ara_export/` (thin `@task_handler` +
trigger body carrying only `task_id`/`subtask_id`),
`orchestration/ara_export_orchestrator.py`, a `ScenarioExportRepository` in
`repositories/` returning an override-resolved read-model (so the orchestrator
never names `infrastructure.postgres.*`), a pure column-shaper in `domain/`,
and a brand palette + openpyxl styling helper in `infrastructure/xlsx/`. The
only `backend/` touches are the new `backend/app/exports/` route and reusing
`resolve_filtered_property_ids`.
- New infra: an `ARA_EXPORT_SQS_URL` queue (terraform), the Lambda at 4 GB
memory with `/tmp` for the workbook, reuse of `DOCUMENT_EXPORTS_BUCKET` and the
SES/S3 adapters from ADR-0059.
- The pinned, resolved `property.id` set makes a run reproducible even if the
portfolio changes after submission.
- `PropertyGroupFilters` / `resolve_filtered_property_ids` (ADR-0056) now has a
second production caller; changes to the shared contract affect both triggers —
the intended shareability, made explicit.
- `scripts/data_exports.py`'s query/column logic is de-duplicated into the new
modules; the standalone CLI either wraps them or is retired.
## Addendum — plan-selection option (default plan per home)
A second **plan selection** was added alongside the original per-Scenario export,
chosen by `plan_selection` on the export request:
- **`"scenario"` (default, unchanged):** one sheet per requested Scenario, the
freshest Plan per `(property, scenario)`. `scenario_ids` required.
- **`"default"` (new):** a single sheet titled **"Default Plans"** of each home's
**default Plan** (`plan.is_default = TRUE`), which is one-per-property across
all Scenarios (ADR-0012 / ADR-0017) — the portfolio's current state, one row
per home. `scenario_ids` is not required and is ignored.
The router (`backend/app/exports/router.py`) validates `plan_selection ∈
{"scenario", "default"}` and only requires a non-empty `scenario_ids` for the
scenario selection; the recipe carries `plan_selection` through to the handler
and orchestrator. The repository gains `default_rows_for(portfolio_id,
property_ids)` — the same read-model and Effective-EPC join as `rows_for`, with
the plan-selection `WHERE scenario_id = …` swapped for `WHERE is_default = TRUE`
(no scenario parameter). A recipe without `plan_selection` (pre-existing tasks)
defaults to `"scenario"`, so the change is backward-compatible.
**New column — `plan_name`.** Both selections surface the **source Scenario's
name** (`scenario.name`, via `plan.scenario_id`) as a `plan_name` column heading
the plan post-works block, so a reader sees which Scenario each home's Plan came
from (e.g. "Fabric Only" / "ASHP + PV"). The freeform `plan.name` is unset on
modelling-generated Plans, so the Scenario name is the meaningful identifier.

59
domain/sap10_calculator/rdsap/cert_to_inputs.py
View file

@ -2141,9 +2141,52 @@ def _control_type(main: Optional[MainHeatingDetail]) -> int:
raise UnmappedSapCode("main_heating_control", code)
# RdSAP 10 Table 29 (PDF p.56-57): a solid floor in age bands A-E carries no
# insulation layer, so wet underfloor pipes sit in a bare concrete slab (SAP 10.2
# Table 4d R=0.25). F-M solid floors carry insulation → screed above (0.75). #1668
_CONCRETE_SLAB_UNDERFLOOR_BANDS: Final[frozenset[str]] = frozenset("ABCDE")
# Lumped gov-EPC "underfloor" heat-emitter code — carries no screed/timber/slab
# subtype; the subtype is derived from floor construction + age band per Table 29.
_UNDERFLOOR_EMITTER_CODE: Final[int] = 2
def _underfloor_responsiveness(
floor_description: Optional[str], age_band: Optional[str]
) -> float:
"""SAP 10.2 Table 4d underfloor responsiveness selected via RdSAP 10 Table 29
from the main part's floor construction + age band (#1668):
- suspended TIMBER floor → 1.0 (fully responsive timber-floor UFH)
- solid floor, age A-E → 0.25 (bare concrete slab, no insulation)
- solid F-M / suspended-not-timber / unknown / no ground floor
→ 0.75 (screed above insulation — the prior
default, so ambiguous certs do not move).
"""
desc = (floor_description or "").lower()
is_timber = "timber" in desc and "not timber" not in desc
if is_timber:
return 1.0
band = (age_band or "").strip().upper()[:1]
if "solid" in desc and band in _CONCRETE_SLAB_UNDERFLOOR_BANDS:
return 0.25
return 0.75
def _main_underfloor_floor_context(
epc: Optional[EpcPropertyData],
) -> tuple[Optional[str], Optional[str]]:
"""(floor construction description, age band) of the main building part —
the Table 29 discriminator for the underfloor subtype in `_responsiveness`."""
if epc is None or not epc.sap_building_parts:
return (None, None)
main_bp = epc.sap_building_parts[0]
return (_effective_floor_description(epc, main_bp), main_bp.construction_age_band)
def _responsiveness(
main: Optional[MainHeatingDetail],
tariff: Optional[Tariff] = None,
epc: Optional[EpcPropertyData] = None,
) -> float:
"""SAP 10.2 responsiveness R ∈ [0, 1] per spec line 15271:
@ -2183,9 +2226,10 @@ def _responsiveness(
4 = Warm air → R = 1.0
5 = Fan coils → R = 1.0
"Concrete slab" UFH (Table 4d R=0.25) has no cert-side enum entry
yet — that variant would need a new mapper code before the cascade
can dispatch it.
The lumped underfloor code 2 carries no screed/timber/concrete-slab
subtype; its Table 4d responsiveness (0.25 / 0.75 / 1.0) is derived
from the main part's floor construction + age band per RdSAP 10
Table 29 via `_underfloor_responsiveness` (needs `epc`; #1668).
Strict-dispatch per [[reference-unmapped-sap-code]]: absent lodging
(None / 0 / "") returns modal default R=1.0 (radiators); lodging
@ -2210,6 +2254,9 @@ def _responsiveness(
emitter = main.heat_emitter_type
if not emitter:
return 1.0
if emitter == _UNDERFLOOR_EMITTER_CODE:
floor_description, age_band = _main_underfloor_floor_context(epc)
return _underfloor_responsiveness(floor_description, age_band)
if isinstance(emitter, int) and emitter in _RESPONSIVENESS_BY_EMITTER_CODE:
return _RESPONSIVENESS_BY_EMITTER_CODE[emitter]
raise UnmappedSapCode("heat_emitter_type", emitter)
@ -4958,7 +5005,7 @@ def mean_internal_temperature_section_from_cert(
thermal_mass_parameter_kj_per_m2_k=_thermal_mass_parameter_kj_per_m2_k(epc),
total_floor_area_m2=dim.total_floor_area_m2,
control_type=_control_type(main),
responsiveness=_responsiveness(main, tariff=tariff),
responsiveness=_responsiveness(main, tariff=tariff, epc=epc),
living_area_fraction=_living_area_fraction(
epc.habitable_rooms_count, dim.total_floor_area_m2
),
@ -8255,7 +8302,7 @@ def cert_to_inputs(
# for the Elmhurst corpus (cert-side mapping is a future slice).
control_type_value = _control_type(main)
_mit_tariff = tariff_from_meter_type(epc.sap_energy_source.meter_type)
responsiveness_value = _responsiveness(main, tariff=_mit_tariff)
responsiveness_value = _responsiveness(main, tariff=_mit_tariff, epc=epc)
living_area_fraction_value = _living_area_fraction(
epc.habitable_rooms_count, dim.total_floor_area_m2
)
@ -8279,7 +8326,7 @@ def cert_to_inputs(
main_2_control_type_value = _control_type(_mit_main_2)
main_2_fraction_value = _mit_main_2.main_heating_fraction / 100.0
main_2_responsiveness_value = _responsiveness(
_mit_main_2, tariff=_mit_tariff
_mit_main_2, tariff=_mit_tariff, epc=epc
)
monthly_total_gains_w = tuple(
internal_gains_monthly_w[m] + solar_gains_monthly_w[m] for m in range(12)

0
domain/scenario_export/__init__.py Normal file
View file

284
domain/scenario_export/fabric_description.py Normal file
View file

@ -0,0 +1,284 @@
"""Human-readable fabric descriptions from an EPC's structured SAP fields
(ADR-0065).
The PasHub re-survey lodges *structured* fabric — SAP building parts, a main
heating detail, coded fuel/water-heating — but no gov-EPC descriptive prose, so
the export's descriptive columns would be blank for a re-surveyed Property. This
module composes the client-sheet text for walls / roof / floor (from the main
``epc_building_part``) and heating / hot water (from ``epc_main_heating_detail``
and the ``epc_property`` water-heating fields), decoding the SAP integer codes
with the same code space the SAP engine uses. Windows and lighting are left on
the gov-EPC prose fallback (their per-element codes do not cleanly reconstruct
the dwelling-level phrase).
Pure logic: the repository reads the structured rows and calls these; each
renderer returns ``None`` when it has nothing to say, so the caller can fall
back to the gov-EPC prose.
"""
from __future__ import annotations
from typing import Any, Optional
# --- SAP integer-code → display-text maps ---------------------------------
# wall_construction (SAP10 codes, domain/sap10_ml/rdsap_uvalues.py:127-142).
_WALL_CONSTRUCTION: dict[int, str] = {
1: "Granite or whinstone",
2: "Sandstone",
3: "Solid brick",
4: "Cavity",
5: "Timber frame",
6: "System build",
7: "Cob",
8: "Park home",
9: "Curtain wall",
10: "Unknown",
11: "Cavity (filled party wall)",
}
# wall_insulation_type (RdSAP schema codes, rdsap_uvalues.py:145-158).
_WALL_INSULATION: dict[int, str] = {
1: "external insulation",
2: "filled cavity",
3: "internal insulation",
4: "as built",
5: "no insulation",
6: "filled cavity + external insulation",
7: "filled cavity + internal insulation",
}
# heat_emitter_type (SAP10.2, mapper.py _ELMHURST_HEAT_EMITTER_TO_SAP10).
_HEAT_EMITTER: dict[int, str] = {
1: "radiators",
2: "underfloor heating (in screed)",
3: "underfloor heating (timber floor)",
4: "warm air",
5: "fan coils",
}
# main_fuel_type / water_heating_fuel (RdSAP "(not community)" family; invert of
# main_fuel_overlay._FUEL_CODES).
_FUEL: dict[int, str] = {
3: "bottled LPG",
6: "wood logs",
10: "dual fuel (mineral and wood)",
15: "smokeless coal",
17: "LPG (special condition)",
20: "mains gas (community)",
25: "electricity (community)",
26: "mains gas",
27: "LPG (bulk)",
28: "oil",
29: "electricity",
31: "biomass (community)",
33: "house coal",
}
# main_heating_control (SAP10.2 Table 4e Group 1; comments in
# heating_recommendation._CONTROL_FEATURES_BY_CODE).
_HEATING_CONTROL: dict[int, str] = {
2101: "no time or thermostatic control",
2102: "programmer, no room thermostat",
2103: "room thermostat only",
2104: "programmer and room thermostat",
2105: "programmer and at least two room thermostats",
2106: "programmer, room thermostat and TRVs",
2107: "programmer, TRVs and bypass",
2108: "programmer, TRVs and flow switch",
2109: "programmer, TRVs and boiler energy manager",
2111: "TRVs and bypass",
2113: "room thermostat and TRVs",
}
# water_heating_code (SAP Table 4a; water_heating_overlay docstring).
_WATER_HEATING_SYSTEM: dict[int, str] = {
901: "from main system",
903: "electric immersion",
911: "gas boiler / circulator",
}
# RdSAP England-&-Wales construction age-band letters → the date range the EPC
# prints (domain/epc/property_overrides/construction_age_band.py). The lodged
# `epc_building_part.construction_age_band` carries the bare letter.
_AGE_BAND: dict[str, str] = {
"A": "before 1900",
"B": "1900-1929",
"C": "1930-1949",
"D": "1950-1966",
"E": "1967-1975",
"F": "1976-1982",
"G": "1983-1990",
"H": "1991-1995",
"I": "1996-2002",
"J": "2003-2006",
"K": "2007-2011",
"L": "2012-2022",
"M": "2023 onwards",
}
def _as_int(value: Any) -> Optional[int]:
"""A SAP code as ``int``. Accepts the raw int, a numeric string, or ``None``;
a non-numeric label (e.g. an insulation ``"As built"`` string in a numeric
slot) yields ``None``."""
if value is None:
return None
if isinstance(value, bool):
return None
if isinstance(value, int):
return value
try:
return int(str(value).strip())
except ValueError:
return None
def _clean_text(value: Any) -> Optional[str]:
"""A lodged free-text field trimmed to a non-empty string, else ``None``.
A bare integer (a code lodged in a text slot) is not display text."""
if value is None or isinstance(value, (int, float, bool)):
return None
text = str(value).strip()
return text or None
# built_form codes (epc_codes.csv; override_code_mapping._BUILT_FORM_CODES).
# The lodged value is usually the display text already; a minority of records
# carry the bare integer code instead.
_BUILT_FORM: dict[int, str] = {
1: "Detached",
2: "Semi-detached",
3: "End-terrace",
4: "Mid-terrace",
5: "Enclosed end-terrace",
6: "Enclosed mid-terrace",
}
def built_form_description(built_form: Any) -> Optional[str]:
"""The dwelling's built form. Decodes the integer code where a record
lodges one, otherwise passes the lodged display text through unchanged."""
code = _as_int(built_form)
if code is not None:
return _BUILT_FORM.get(code)
return _clean_text(built_form)
def age_band_description(band: Any) -> Optional[str]:
"""The construction age band as its date range, e.g. ``"1900-1929 (band B)"``.
The lodged value is a bare RdSAP letter; an unrecognised or ``"Unknown"``
band renders nothing."""
letter = _clean_text(band)
if letter is None:
return None
date_range = _AGE_BAND.get(letter.strip().upper())
return f"{date_range} (band {letter.strip().upper()})" if date_range else None
def wall_description(construction: Any, insulation: Any) -> Optional[str]:
"""A wall part's text, e.g. ``"Cavity wall, filled cavity"``. The
construction names the wall; the insulation qualifies it when known."""
cons = _WALL_CONSTRUCTION.get(_as_int(construction) or 0)
if cons is None:
return None
ins = _WALL_INSULATION.get(_as_int(insulation) or 0)
return f"{cons} wall, {ins}" if ins else f"{cons} wall"
def _insulation_thickness_mm(thickness: Any) -> Optional[int]:
"""A roof-insulation thickness in mm from a numeric code or a ``"225mm"``
string; a non-numeric label (``"As built"``, ``"ND"``) or ``None`` yields
``None``. Zero is a real value (no insulation) and is preserved."""
as_int = _as_int(thickness)
if as_int is not None:
return as_int
text = _clean_text(thickness)
if text is None:
return None
digits = "".join(ch for ch in text if ch.isdigit())
return int(digits) if digits else None
def roof_description(
construction_type: Any, insulation_location: Any, insulation_thickness: Any
) -> Optional[str]:
"""A roof part's text, e.g. ``"Pitched roof (Slates or tiles), Access to
loft; 270mm insulation at joists"``. The construction is already lodged as
prose; the insulation clause is appended when a thickness or a named
location is present."""
cons = _clean_text(construction_type)
if cons is None:
return None
thickness_mm = _insulation_thickness_mm(insulation_thickness)
location = _clean_text(insulation_location)
if location is not None and location.lower() in {"unknown", "none"}:
location = None
if thickness_mm == 0:
return f"{cons}; no insulation"
if thickness_mm is None and location is None:
return cons
clause = "insulation" if thickness_mm is None else f"{thickness_mm}mm insulation"
if location is not None:
clause = f"{clause} at {location.lower()}"
return f"{cons}; {clause}"
def floor_description(
floor_type: Any, construction_type: Any, insulation: Any
) -> Optional[str]:
"""A floor part's text, e.g. ``"Ground floor, suspended timber, insulation
as built"``. Composed from the lodged floor type, construction, and
insulation status (all lodged as prose)."""
parts: list[str] = []
location = _clean_text(floor_type)
if location is not None:
parts.append(location[0].upper() + location[1:].lower())
construction = _clean_text(construction_type)
if construction is not None:
parts.append(construction.lower())
ins = _clean_text(insulation)
if ins is not None:
parts.append(f"insulation {ins.lower()}")
return ", ".join(parts) or None
def heating_description(
fuel: Any, emitter: Any, control: Any, condensing: Any
) -> Optional[str]:
"""The main heating system's text from its SAP codes, e.g. ``"Mains gas,
radiators (condensing); programmer, room thermostat and TRVs"``. Reports
only what the codes state — the appliance archetype is not lodged on a
re-survey, so no boiler/heat-pump type is asserted."""
fuel_text = _FUEL.get(_as_int(fuel) or 0)
emitter_text = _HEAT_EMITTER.get(_as_int(emitter) or 0)
lead_parts = [p for p in (fuel_text, emitter_text) if p]
if not lead_parts:
return None
lead = ", ".join(p[0].upper() + p[1:] if i == 0 else p for i, p in enumerate(lead_parts))
if condensing is True:
lead = f"{lead} (condensing)"
control_text = _HEATING_CONTROL.get(_as_int(control) or 0)
return f"{lead}; {control_text}" if control_text else lead
def hot_water_description(
water_heating_code: Any,
water_heating_fuel: Any,
main_fuel: Any,
has_cylinder: Any,
solar: Any,
) -> Optional[str]:
"""The hot-water system's text from its SAP codes, e.g. ``"From main
system, mains gas"``. Where the water-heating code is not lodged on the
re-survey, it falls back to the SAP default (a dwelling with no cylinder
draws hot water from the main system) using the main heating fuel."""
system = _WATER_HEATING_SYSTEM.get(_as_int(water_heating_code) or 0)
fuel = _FUEL.get(_as_int(water_heating_fuel) or 0) or _FUEL.get(_as_int(main_fuel) or 0)
if system is None:
# No lodged water-heating code: a dwelling without a cylinder inherits
# hot water from the main system (SAP Table 4a code 901).
if has_cylinder is True:
system = "hot water cylinder"
elif has_cylinder is False:
system = "from main system"
else:
return None
text = f"{system}, {fuel}" if fuel else system
text = text[0].upper() + text[1:]
if solar is True:
text = f"{text}; with solar water heating"
return text

253
domain/scenario_export/scenario_sheet.py Normal file
View file

@ -0,0 +1,253 @@
"""The Scenario Export sheet shape (ADR-0065).
Pure logic: given the persisted, override-resolved data for the Properties in one
Scenario, lay out the wide export sheet — property identity + Effective-EPC
descriptive fields, each measure's cost pivoted onto its own column against a
frozen column contract, savings summed per Property, and the Plan's post-works
figures. No I/O: the repository reads the rows and the infrastructure layer
renders the workbook; this module only shapes one sheet's rows.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any, Optional, Sequence
# The measure cost columns always present on every sheet, in a stable order, so
# every scenario sheet in the workbook shares one column contract regardless of
# which measures a given selection happens to contain (ADR-0065). A Property that
# lacks a measure carries a blank in that column, never a missing column.
MEASURE_COLUMNS: tuple[str, ...] = (
"suspended_floor_insulation",
"solid_floor_insulation",
"external_wall_insulation",
"internal_wall_insulation",
"cavity_wall_insulation",
"loft_insulation",
"flat_roof_insulation",
"room_roof_insulation",
"secondary_glazing",
"double_glazing",
"solar_pv",
"high_heat_retention_storage_heaters",
"air_source_heat_pump",
"boiler_upgrade",
"gas_boiler_upgrade",
"roomstat_programmer_trvs",
"time_temperature_zone_control",
"low_energy_lighting",
"mechanical_ventilation",
"system_tune_up",
"system_tune_up_zoned",
)
# A solar_pv Option that installs a battery pivots to its own column, kept in the
# frozen contract so a sheet with battery-solar shares columns with one without.
_BATTERY_MEASURE_COLUMNS: tuple[str, ...] = ("solar_pv_with_battery",)
# Every measure cost column present on a sheet: the frozen base contract plus the
# battery variant.
_ALL_MEASURE_COLUMNS: tuple[str, ...] = MEASURE_COLUMNS + _BATTERY_MEASURE_COLUMNS
# Property identity and Effective-EPC descriptive columns, leading every sheet
# before the measure columns. The descriptive fields are already override-resolved
# by the repository (property_overrides → lodged EPC → predicted, ADR-0065).
_ID_COLUMNS: tuple[str, ...] = (
"property_id",
"landlord_property_id",
"uprn",
"address",
"postcode",
"property_type",
"built_form",
"property_age_band",
"walls",
"roof",
"floor",
"windows",
"heating",
"hot_water",
"lighting",
"total_floor_area",
"number_of_rooms",
"lodgement_date",
"is_expired",
"current_epc_rating",
"current_sap_points",
)
# Per-Property roll-ups of the selected measures' attributed impact, trailing the
# measure columns (ADR-0065).
_SAVINGS_COLUMNS: tuple[str, ...] = (
"sap_points",
"co2_equivalent_savings",
"kwh_savings",
"energy_cost_savings",
)
# Package-level totals, trailing the savings roll-ups.
_TOTAL_COLUMNS: tuple[str, ...] = ("total_retrofit_cost",)
# The default Plan's post-works figures, taken straight from the persisted Plan
# (the SAP calculator's output) with no recompute (ADR-0065).
_PLAN_COLUMNS: tuple[str, ...] = (
"plan_name",
"post_sap_points",
"post_epc_rating",
"cost_of_works",
"contingency_cost",
"co2_savings",
"energy_bill_savings",
"energy_consumption_savings",
"valuation_increase",
)
def _blank(value: Any) -> Any:
"""A missing value renders as a blank cell, not a zero or ``None``."""
return "" if value is None else value
def _sum(values: Sequence[Optional[float]]) -> float:
"""Sum optional numbers, treating an absent contribution as zero."""
return sum(value or 0.0 for value in values)
def _measure_column(measure: ExportMeasure) -> str:
"""The pivot column for a measure: a solar_pv Option carrying a battery lands
in its own ``solar_pv_with_battery`` column (ADR-0065)."""
if measure.measure_type == "solar_pv" and measure.includes_battery:
return "solar_pv_with_battery"
return measure.measure_type
@dataclass(frozen=True)
class ExportMeasure:
"""One selected measure on a Property's default Plan for the Scenario: the
measure type (the pivot column), its installed cost, and the SAP / carbon /
energy / bill savings attributed to it. ``includes_battery`` distinguishes a
solar-with-battery Option so it pivots to its own column (ADR-0065)."""
measure_type: str
estimated_cost: Optional[float] = None
sap_points: Optional[float] = None
co2_equivalent_savings: Optional[float] = None
kwh_savings: Optional[float] = None
energy_cost_savings: Optional[float] = None
includes_battery: bool = False
@dataclass(frozen=True)
class PropertyScenarioData:
"""One Property's persisted data for one Scenario: identity, the
Effective-EPC descriptive fields (already override-resolved by the
repository), the default Plan's post-works figures, and the selected
measures. The input row the shaper turns into one wide sheet row."""
property_id: int
landlord_property_id: Optional[str] = None
uprn: Optional[str] = None
address: Optional[str] = None
postcode: Optional[str] = None
property_type: Optional[str] = None
built_form: Optional[str] = None
property_age_band: Optional[str] = None
walls: Optional[str] = None
roof: Optional[str] = None
floor: Optional[str] = None
windows: Optional[str] = None
heating: Optional[str] = None
hot_water: Optional[str] = None
lighting: Optional[str] = None
total_floor_area: Optional[float] = None
number_of_rooms: Optional[int] = None
lodgement_date: Optional[str] = None
is_expired: Optional[bool] = None
current_epc_rating: Optional[str] = None
current_sap_points: Optional[float] = None
plan_name: Optional[str] = None
post_sap_points: Optional[float] = None
post_epc_rating: Optional[str] = None
cost_of_works: Optional[float] = None
contingency_cost: Optional[float] = None
co2_savings: Optional[float] = None
energy_bill_savings: Optional[float] = None
energy_consumption_savings: Optional[float] = None
valuation_increase: Optional[float] = None
measures: Sequence[ExportMeasure] = ()
@dataclass(frozen=True)
class ExportSheet:
"""One laid-out scenario sheet: the ordered column contract and the rows,
each a column-keyed mapping ready for the workbook renderer."""
columns: tuple[str, ...]
rows: tuple[dict[str, Any], ...]
_COLUMNS: tuple[str, ...] = (
_ID_COLUMNS + _ALL_MEASURE_COLUMNS + _SAVINGS_COLUMNS + _TOTAL_COLUMNS + _PLAN_COLUMNS
)
def _shape_row(prop: PropertyScenarioData) -> dict[str, Any]:
"""One Property's wide row: identity + Effective-EPC descriptive fields, the
measures pivoted onto their cost columns, the savings rolled up, and the
Plan's post-works figures (ADR-0065)."""
row: dict[str, Any] = {
"property_id": prop.property_id,
"landlord_property_id": _blank(prop.landlord_property_id),
"uprn": _blank(prop.uprn),
"address": _blank(prop.address),
"postcode": _blank(prop.postcode),
"property_type": _blank(prop.property_type),
"built_form": _blank(prop.built_form),
"property_age_band": _blank(prop.property_age_band),
"walls": _blank(prop.walls),
"roof": _blank(prop.roof),
"floor": _blank(prop.floor),
"windows": _blank(prop.windows),
"heating": _blank(prop.heating),
"hot_water": _blank(prop.hot_water),
"lighting": _blank(prop.lighting),
"total_floor_area": _blank(prop.total_floor_area),
"number_of_rooms": _blank(prop.number_of_rooms),
"lodgement_date": _blank(prop.lodgement_date),
"is_expired": _blank(prop.is_expired),
"current_epc_rating": _blank(prop.current_epc_rating),
"current_sap_points": _blank(prop.current_sap_points),
}
# Every measure column is present, blank unless this Property has it, so all
# scenario sheets share one contract (ADR-0065).
for column in _ALL_MEASURE_COLUMNS:
row[column] = ""
for measure in prop.measures:
column = _measure_column(measure)
if column in _ALL_MEASURE_COLUMNS:
row[column] = measure.estimated_cost
# Roll the selected measures' attributed impact up to the Property.
row["sap_points"] = _sum([m.sap_points for m in prop.measures])
row["co2_equivalent_savings"] = _sum([m.co2_equivalent_savings for m in prop.measures])
row["kwh_savings"] = _sum([m.kwh_savings for m in prop.measures])
row["energy_cost_savings"] = _sum([m.energy_cost_savings for m in prop.measures])
row["total_retrofit_cost"] = _sum([m.estimated_cost for m in prop.measures])
# The default Plan's post-works figures, passed through as persisted.
row["plan_name"] = _blank(prop.plan_name)
row["post_sap_points"] = _blank(prop.post_sap_points)
row["post_epc_rating"] = _blank(prop.post_epc_rating)
row["cost_of_works"] = _blank(prop.cost_of_works)
row["contingency_cost"] = _blank(prop.contingency_cost)
row["co2_savings"] = _blank(prop.co2_savings)
row["energy_bill_savings"] = _blank(prop.energy_bill_savings)
row["energy_consumption_savings"] = _blank(prop.energy_consumption_savings)
row["valuation_increase"] = _blank(prop.valuation_increase)
return row
def shape_scenario_sheet(
properties: Sequence[PropertyScenarioData],
) -> ExportSheet:
"""Shape one Scenario's Properties into a wide export sheet (ADR-0065)."""
rows = tuple(_shape_row(prop) for prop in properties)
return ExportSheet(columns=_COLUMNS, rows=rows)

3
infrastructure/postgres/product_table.py
View file

@ -22,3 +22,6 @@ class MaterialRow(SQLModel, table=True):
cost_unit: Optional[str] = Field(default=None)
description: Optional[str] = Field(default=None)
is_active: bool = Field(default=True)
# Whether a solar_pv Option's product bundle includes a battery; the Scenario
# Export reads it to pivot solar-with-battery to its own column (ADR-0065).
includes_battery: bool = Field(default=False)

0
infrastructure/xlsx/__init__.py Normal file
View file

51
infrastructure/xlsx/branding.py Normal file
View file

@ -0,0 +1,51 @@
"""Monday.com-style branding for the Scenario Export workbook (ADR-0065).
A programmatic openpyxl styling pass (no template): a bold indigo header band
with white text, a frozen header row, subtle alternating row bands, and roomy
column widths. The palette lives here as the single home for brand colour — the
"no brand-colour constants anywhere" gap the review flagged.
TODO(brand): swap the placeholder Monday-style hex for the real Domna brand
palette once design supplies it.
"""
from __future__ import annotations
from openpyxl.styles import Alignment, Font, PatternFill
from openpyxl.utils import get_column_letter
from openpyxl.worksheet.worksheet import Worksheet
# openpyxl colours are 8-char ARGB. Placeholder Monday-style palette.
HEADER_FILL: str = "FF6C6CFF" # indigo header band
HEADER_FONT: str = "FFFFFFFF" # white header text
ROW_BAND_FILL: str = "FFF1F1FF" # subtle alternating-row tint
_COLUMN_WIDTH: int = 18
def style_sheet(worksheet: Worksheet) -> None:
"""Paint the Monday-style branding onto a populated worksheet: header band +
font, frozen header row, banded rows, and column widths."""
header_fill = PatternFill(
start_color=HEADER_FILL, end_color=HEADER_FILL, fill_type="solid"
)
header_font = Font(bold=True, color=HEADER_FONT)
for cell in worksheet[1]:
cell.fill = header_fill
cell.font = header_font
cell.alignment = Alignment(horizontal="left", vertical="center")
# Keep the header visible while scrolling a long portfolio.
worksheet.freeze_panes = "A2"
band_fill = PatternFill(
start_color=ROW_BAND_FILL, end_color=ROW_BAND_FILL, fill_type="solid"
)
for row_index in range(3, worksheet.max_row + 1, 2):
for cell in worksheet[row_index]:
cell.fill = band_fill
for column_index in range(1, worksheet.max_column + 1):
worksheet.column_dimensions[get_column_letter(column_index)].width = (
_COLUMN_WIDTH
)

64
infrastructure/xlsx/scenario_export_workbook.py Normal file
View file

@ -0,0 +1,64 @@
"""Renders the Scenario Export workbook (ADR-0065).
One ``.xlsx`` with one sheet per Scenario (model A2), each sheet's rows shaped by
``domain.scenario_export`` and painted with the Monday-style branding in
``infrastructure.xlsx.branding``. Programmatic openpyxl (no template) so a
dynamic number of scenario sheets is handled natively.
"""
from __future__ import annotations
import re
from typing import Sequence
from openpyxl import Workbook
from domain.scenario_export.scenario_sheet import ExportSheet
from infrastructure.xlsx.branding import style_sheet
# Excel forbids these in a sheet title and caps titles at 31 characters.
_ILLEGAL_SHEET_CHARS = re.compile(r"[:\\/?*\[\]]")
_MAX_SHEET_NAME = 31
def _safe_sheet_name(name: str, used: set[str]) -> str:
"""An Excel-legal, unique sheet name: illegal characters stripped, capped at
31 chars, and suffixed on collision (two scenarios can share a name)."""
clean = _ILLEGAL_SHEET_CHARS.sub("", name or "scenario").strip() or "scenario"
clean = clean[:_MAX_SHEET_NAME]
base, index = clean, 1
while clean in used:
suffix = f" ({index})"
clean = base[: _MAX_SHEET_NAME - len(suffix)] + suffix
index += 1
used.add(clean)
return clean
def render_workbook(
sheets: Sequence[tuple[str, ExportSheet]], destination_path: str
) -> None:
"""Render the scenario sheets into a single branded ``.xlsx`` at
``destination_path``. Each tuple is a Scenario name and its shaped sheet;
sheet order follows the input (the requested ``scenario_ids`` order).
Saves straight to disk (``Workbook.save(path)``) so the caller can
``S3Client.upload_file`` it with multipart — the workbook is never also held
as an in-memory ``bytes`` copy (ADR-0065; the OOM path the 4 GB sizing was
against at the 100k-row cap). The further memory win is openpyxl
``write_only`` mode; that switch would live at this seam.
"""
workbook = Workbook()
default = workbook.active
if default is not None:
workbook.remove(default)
used_names: set[str] = set()
for name, sheet in sheets:
worksheet = workbook.create_sheet(title=_safe_sheet_name(name, used_names))
worksheet.append(list(sheet.columns))
for row in sheet.rows:
worksheet.append([row.get(column, "") for column in sheet.columns])
style_sheet(worksheet)
workbook.save(destination_path)

221
orchestration/scenario_export_orchestrator.py Normal file
View file

@ -0,0 +1,221 @@
"""The Scenario Export orchestrator (ADR-0065).
Ties one export together: for each requested Scenario, read its Properties'
persisted rows, shape them into a sheet, render the sheet-per-scenario branded
workbook, upload it to the exports bucket, mint a presigned URL, and email the
requester. Best-effort email (the link is also recorded on ``sub_task.outputs``);
a selection that yields no rows for any Scenario is a recorded failure — never an
empty workbook emailed to the user.
"""
from __future__ import annotations
import html
import logging
import os
import tempfile
from dataclasses import dataclass
from typing import Protocol, Sequence
from domain.scenario_export.scenario_sheet import (
ExportSheet,
PropertyScenarioData,
shape_scenario_sheet,
)
from domain.tasks.subtasks import SubTaskFailure
from infrastructure.s3.s3_client import S3Client
from infrastructure.xlsx.scenario_export_workbook import render_workbook
from repositories.email.email_sender import EmailSender
logger = logging.getLogger(__name__)
# The single sheet title for the default-plan selection (ADR-0065).
_DEFAULT_PLAN_SHEET_NAME = "Default Plans"
class ExportRowReader(Protocol):
def rows_for(
self,
*,
portfolio_id: int,
scenario_id: int,
property_ids: Sequence[int],
) -> list[PropertyScenarioData]: ...
def default_rows_for(
self,
*,
portfolio_id: int,
property_ids: Sequence[int],
) -> list[PropertyScenarioData]: ...
class ScenarioNames(Protocol):
def name_for(self, scenario_id: int) -> str: ...
@dataclass(frozen=True)
class ScenarioExportResult:
"""What a completed export reports (lands on ``sub_task.outputs``)."""
presigned_url: str
export_s3_key: str
sheet_count: int
row_count: int
class ScenarioExportOrchestrator:
def __init__(
self,
*,
rows: ExportRowReader,
scenarios: ScenarioNames,
exports: S3Client,
email: EmailSender,
url_ttl_seconds: int,
key_prefix: str = "scenario-exports",
) -> None:
self._rows = rows
self._scenarios = scenarios
self._exports = exports
self._email = email
self._url_ttl_seconds = url_ttl_seconds
self._key_prefix = key_prefix
def _scenario_sheets(
self,
portfolio_id: int,
scenario_ids: Sequence[int],
property_ids: Sequence[int],
) -> list[tuple[str, ExportSheet]]:
named_sheets: list[tuple[str, ExportSheet]] = []
for scenario_id in scenario_ids:
rows = self._rows.rows_for(
portfolio_id=portfolio_id,
scenario_id=scenario_id,
property_ids=property_ids,
)
named_sheets.append(
(self._scenarios.name_for(scenario_id), shape_scenario_sheet(rows))
)
return named_sheets
def _default_plan_sheets(
self, portfolio_id: int, property_ids: Sequence[int]
) -> list[tuple[str, ExportSheet]]:
# One sheet: each home's default Plan (`is_default`), across Scenarios.
rows = self._rows.default_rows_for(
portfolio_id=portfolio_id, property_ids=property_ids
)
return [(_DEFAULT_PLAN_SHEET_NAME, shape_scenario_sheet(rows))]
def run(
self,
*,
portfolio_id: int,
scenario_ids: Sequence[int],
property_ids: Sequence[int],
recipient_email: str,
export_name: str,
plan_selection: str = "scenario",
) -> ScenarioExportResult:
# ADR-0065 two selections:
# - "scenario": one sheet per requested Scenario, the freshest Plan per
# (property, scenario).
# - "default": ONE sheet of each home's default Plan (`is_default`),
# one-per-property across all Scenarios — the portfolio's current
# state. `scenario_ids` is ignored.
if plan_selection == "default":
named_sheets = self._default_plan_sheets(portfolio_id, property_ids)
else:
named_sheets = self._scenario_sheets(
portfolio_id, scenario_ids, property_ids
)
row_count = sum(len(sheet.rows) for _, sheet in named_sheets)
if row_count == 0:
# No Property had a Plan under the selection (model A) — a recorded
# failure, never an empty workbook emailed (ADR-0065).
raise SubTaskFailure(
"the scenario export selection produced no exportable rows",
details={
"portfolio_id": portfolio_id,
"plan_selection": plan_selection,
"scenario_ids": list(scenario_ids),
"property_count": len(property_ids),
},
)
key = f"{self._key_prefix}/{export_name}.xlsx"
# Render to a /tmp file and multipart-upload it, so the workbook is never
# also held in memory as a bytes copy (ADR-0065 — the OOM path at the
# 100k-row cap). The temp file is always cleaned up.
handle, workbook_path = tempfile.mkstemp(suffix=".xlsx")
os.close(handle)
try:
render_workbook(named_sheets, workbook_path)
self._exports.upload_file(workbook_path, key)
finally:
try:
os.remove(workbook_path)
except OSError:
pass
url = self._exports.generate_presigned_url(key, self._url_ttl_seconds)
subject, body, html_body = self._compose_email(
url=url, sheets=len(named_sheets), rows=row_count
)
try:
self._email.send(
to=recipient_email, subject=subject, body=body, html_body=html_body
)
logger.info("scenario_export: emailed %s", recipient_email)
except Exception:
# Best-effort delivery (ADR-0065): the link is also written to
# sub_task.outputs, so an email/transport failure must not lose an
# export that was already built and uploaded.
logger.warning(
"scenario_export: email delivery to %s failed; the link is still "
"recorded on sub_task.outputs",
recipient_email,
exc_info=True,
)
return ScenarioExportResult(
presigned_url=url,
export_s3_key=key,
sheet_count=len(named_sheets),
row_count=row_count,
)
def _compose_email(
self, *, url: str, sheets: int, rows: int
) -> tuple[str, str, str]:
"""The delivery email (ADR-0059/0065): a subject, a plain-text body
carrying the raw link, and an HTML alternative with a clean download
button so the long presigned URL isn't the visible content."""
minutes = self._url_ttl_seconds // 60
sheet_noun = "scenario" if sheets == 1 else "scenarios"
subject = f"Your scenario export is ready ({sheets} {sheet_noun})"
body = (
"Your scenario export is ready.\n\n"
f"It covers {sheets} {sheet_noun} across {rows} property row(s).\n\n"
f"Download it here (link valid for {minutes} minutes):\n{url}\n"
)
safe_url = html.escape(url, quote=True)
html_body = (
'<div style="font-family:Arial,Helvetica,sans-serif;font-size:15px;'
'color:#1a1a1a;line-height:1.5;">'
"<p>Your scenario export is ready.</p>"
f"<p>It covers <strong>{sheets}</strong> {sheet_noun} across "
f"<strong>{rows}</strong> property row(s).</p>"
'<p style="margin:24px 0;">'
f'<a href="{safe_url}" style="display:inline-block;padding:12px 22px;'
"background:#6C6CFF;color:#ffffff;text-decoration:none;border-radius:6px;"
'font-weight:bold;">Download export</a></p>'
f'<p style="color:#666;font-size:13px;">This link expires in {minutes} '
"minutes. If the button doesn't work, paste this URL into your browser:"
f'<br><span style="word-break:break-all;">{safe_url}</span></p>'
"</div>"
)
return subject, body, html_body

0
repositories/scenario_export/__init__.py Normal file
View file

414
repositories/scenario_export/scenario_export_postgres_repository.py Normal file
View file

@ -0,0 +1,414 @@
from __future__ import annotations
from datetime import date, datetime
from typing import Any, Optional, Sequence
from sqlalchemy import text
from sqlmodel import Session
from domain.scenario_export.fabric_description import (
age_band_description,
built_form_description,
floor_description,
heating_description,
hot_water_description,
roof_description,
wall_description,
)
from domain.scenario_export.scenario_sheet import ExportMeasure, PropertyScenarioData
from repositories.scenario_export.scenario_export_repository import (
ScenarioExportRepository,
)
# The NEWEST Plan per (property, scenario), scoped to the requested Properties.
# `is_default` is one-per-property (not per-scenario), so it cannot scope a
# multi-scenario export — the freshest Plan under the Scenario (created_at DESC,
# id tiebreak) is the export's selection. A Property with no Plan under the
# Scenario yields no row and drops out.
_ROWS_SQL = text(
"SELECT p.id, p.landlord_property_id, p.uprn, p.address, p.postcode,"
" np.post_sap_points, np.post_epc_rating, np.cost_of_works,"
" np.contingency_cost, np.co2_savings, np.energy_bill_savings,"
" np.energy_consumption_savings, np.valuation_increase, sc.name AS plan_name"
" FROM property p"
" JOIN ("
" SELECT DISTINCT ON (property_id) property_id, post_sap_points,"
" post_epc_rating, cost_of_works, contingency_cost, co2_savings,"
" energy_bill_savings, energy_consumption_savings, valuation_increase,"
" scenario_id"
" FROM plan"
" WHERE portfolio_id = :portfolio_id AND scenario_id = :scenario_id"
" AND property_id = ANY(:property_ids)"
" ORDER BY property_id, created_at DESC, id DESC"
" ) np ON np.property_id = p.id"
" LEFT JOIN scenario sc ON sc.id = np.scenario_id"
" WHERE p.portfolio_id = :portfolio_id"
" ORDER BY p.id"
)
# The selected measures on each Property's NEWEST Plan under the Scenario: the
# recommendations kept in the Optimised Package (``default`` and not
# ``already_installed``), with the installed material's battery flag for the
# solar pivot (ADR-0065). ``default`` is a reserved word, hence the quoting.
_MEASURES_SQL = text(
"SELECT r.property_id, r.measure_type, r.estimated_cost, r.sap_points,"
" r.co2_equivalent_savings, r.kwh_savings, r.energy_cost_savings,"
" COALESCE(m.includes_battery, FALSE) AS includes_battery"
" FROM recommendation r"
" JOIN ("
" SELECT DISTINCT ON (property_id) id"
" FROM plan"
" WHERE portfolio_id = :portfolio_id AND scenario_id = :scenario_id"
" AND property_id = ANY(:property_ids)"
" ORDER BY property_id, created_at DESC, id DESC"
" ) np ON np.id = r.plan_id"
" LEFT JOIN material m ON m.id = r.material_id"
' WHERE r."default" = TRUE'
" AND r.already_installed = FALSE"
)
# DEFAULT-PLAN variants (ADR-0065 "default plan per home" selection). Each home's
# ONE default Plan (`is_default = TRUE`), which is one-per-property across all
# Scenarios (ADR-0012 / ADR-0017) — so there is no `scenario_id` filter. The
# `DISTINCT ON (property_id) … created_at DESC` guard is retained as belt-and-
# braces against a transient double-default during a re-baseline write; steady
# state has exactly one. Otherwise identical to the scenario SQL above.
_DEFAULT_ROWS_SQL = text(
str(_ROWS_SQL)
.replace("scenario_id = :scenario_id", "is_default = TRUE")
)
_DEFAULT_MEASURES_SQL = text(
str(_MEASURES_SQL)
.replace("scenario_id = :scenario_id", "is_default = TRUE")
)
# The Effective-EPC descriptive picture (ADR-0065). Header facts and current
# performance are matched by UPRN (the latest epc_property for the source),
# because the PasHub re-fetch lands records with property_id = NULL — a
# property_id join would silently fall back to a stale gov-EPC cert (wrong
# lodgement dates / property_type) or nothing. "Latest" is ordered by the cert's
# effective lodgement date (registration → completion → inspection) DESC, NOT by
# `e.id`: row-id order does not track recency — a gov-EPC cert re-ingested after
# the PasHub survey lands with a HIGHER `e.id`, so an `e.id` sort silently picks
# the stale gov cert over the fresh 2026 survey (observed on 3 of portfolio 838).
# `e.id DESC` remains the tiebreak within one date (duplicate survey loads carry
# identical facts). The descriptive prose elements have no PasHub equivalent (a
# survey lodges structured fields, not gov-EPC prose), so they stay on the
# property_id join as the best-available gov-EPC fallback. All three are read
# separately and composed per Property.
_EPC_HEADER_SQL = text(
"WITH latest AS ("
" SELECT DISTINCT ON (p.id) p.id AS property_id, e.id AS epc_id"
" FROM property p JOIN epc_property e ON e.uprn = p.uprn"
" WHERE p.id = ANY(:property_ids) AND e.source = :source"
" ORDER BY p.id, COALESCE(e.registration_date, e.completion_date,"
" e.inspection_date) DESC NULLS LAST, e.id DESC)"
" SELECT l.property_id, e.property_type, e.total_floor_area_m2,"
" COALESCE(e.registration_date, e.completion_date, e.inspection_date),"
" e.habitable_rooms_count, e.built_form"
" FROM latest l JOIN epc_property e ON e.id = l.epc_id"
)
_EPC_ELEMENTS_SQL = text(
"SELECT ep.property_id, e.element_type, e.description"
" FROM epc_energy_element e"
" JOIN epc_property ep ON ep.id = e.epc_property_id"
" WHERE ep.property_id = ANY(:property_ids) AND ep.source = :source"
)
_EPC_PERF_SQL = text(
"WITH latest AS ("
" SELECT DISTINCT ON (p.id) p.id AS property_id, e.id AS epc_id"
" FROM property p JOIN epc_property e ON e.uprn = p.uprn"
" WHERE p.id = ANY(:property_ids) AND e.source = :source"
" ORDER BY p.id, COALESCE(e.registration_date, e.completion_date,"
" e.inspection_date) DESC NULLS LAST, e.id DESC)"
" SELECT l.property_id, perf.energy_rating_current,"
" perf.current_energy_efficiency_band"
" FROM latest l"
" JOIN epc_property_energy_performance perf ON perf.epc_property_id = l.epc_id"
)
# The structured SAP fabric on the same canonical (latest-by-date, UPRN-matched)
# cert the header reads: the MAIN building part (walls / roof / floor), the main
# heating detail (fuel / emitter / control / condensing) and the water-heating
# fields. A PasHub re-survey lodges these coded facts but no gov-EPC prose, so
# they supply walls/roof/floor/heating/hot_water where the prose fallback is
# blank. The main building part is chosen identifier='main' first, then the
# lowest part number / id (matching the SAP engine's `parts[0]` primary part).
# Windows and lighting are not read here (their per-element codes do not cleanly
# reconstruct the dwelling phrase); they stay on the gov-EPC prose fallback.
_EPC_STRUCTURED_FABRIC_SQL = text(
"WITH latest AS ("
" SELECT DISTINCT ON (p.id) p.id AS property_id, e.id AS epc_id"
" FROM property p JOIN epc_property e ON e.uprn = p.uprn"
" WHERE p.id = ANY(:property_ids) AND e.source = :source"
" ORDER BY p.id, COALESCE(e.registration_date, e.completion_date,"
" e.inspection_date) DESC NULLS LAST, e.id DESC),"
" main_part AS ("
" SELECT DISTINCT ON (l.property_id) l.property_id, bp.construction_age_band,"
" bp.wall_construction, bp.wall_insulation_type, bp.roof_construction_type,"
" bp.roof_insulation_location, bp.roof_insulation_thickness, bp.floor_type,"
" bp.floor_construction_type, bp.floor_insulation_type_str"
" FROM latest l JOIN epc_building_part bp ON bp.epc_property_id = l.epc_id"
" ORDER BY l.property_id,"
" (CASE WHEN lower(bp.identifier) = 'main' THEN 0 ELSE 1 END),"
" bp.building_part_number NULLS FIRST, bp.id),"
" main_heat AS ("
" SELECT DISTINCT ON (h.epc_property_id) h.epc_property_id, h.main_fuel_type,"
" h.heat_emitter_type, h.main_heating_control, h.condensing"
" FROM epc_main_heating_detail h"
" ORDER BY h.epc_property_id, h.main_heating_number NULLS FIRST, h.id)"
" SELECT l.property_id, mp.wall_construction, mp.wall_insulation_type,"
" mp.roof_construction_type, mp.roof_insulation_location,"
" mp.roof_insulation_thickness, mp.floor_type, mp.floor_construction_type,"
" mp.floor_insulation_type_str, mh.main_fuel_type, mh.heat_emitter_type,"
" mh.main_heating_control, mh.condensing, e.heating_water_heating_code,"
" e.heating_water_heating_fuel, e.has_hot_water_cylinder,"
" e.solar_water_heating, mp.construction_age_band"
" FROM latest l"
" JOIN epc_property e ON e.id = l.epc_id"
" LEFT JOIN main_part mp ON mp.property_id = l.property_id"
" LEFT JOIN main_heat mh ON mh.epc_property_id = l.epc_id"
)
# A landlord property_type override wins over any EPC-derived value (ADR-0065);
# the descriptive prose fields have no override prose, so only property_type is
# resolved this way (the override enum carries coded fabric facts, not prose).
_OVERRIDE_PROPERTY_TYPE_SQL = text(
"SELECT property_id, override_value FROM property_overrides"
" WHERE property_id = ANY(:property_ids) AND building_part = 0"
" AND override_component = 'property_type'"
)
# EPC energy-element type → export descriptive column.
_ELEMENT_TO_FIELD: dict[str, str] = {
"wall": "walls",
"roof": "roof",
"floor": "floor",
"window": "windows",
"main_heating": "heating",
"hot_water": "hot_water",
"lighting": "lighting",
}
# A cert older than this many years is flagged expired (as data_exports did).
_EXPIRY_DAYS = 365 * 10
def _is_expired(registration_date: Optional[str]) -> Optional[bool]:
if not registration_date:
return None
try:
lodged = datetime.fromisoformat(registration_date[:10]).date()
except ValueError:
return None
return (date.today() - lodged).days > _EXPIRY_DAYS
def _str(value: Any) -> Optional[str]:
"""Render a persisted scalar (int UPRN, Epc enum band) as its string form,
preserving absence as ``None``."""
if value is None:
return None
return str(getattr(value, "value", value))
def _float(value: Any) -> Optional[float]:
return None if value is None else float(value)
class ScenarioExportPostgresRepository(ScenarioExportRepository):
"""Reads Scenario Export rows from Postgres (ADR-0065). Hand-rolled SQL over
the live tables (the ``uploaded_file`` repository precedent), referencing only
the mirrored columns; returns the domain read-model, never ORM rows."""
def __init__(self, session: Session) -> None:
self._session = session
def rows_for(
self,
*,
portfolio_id: int,
scenario_id: int,
property_ids: Sequence[int],
) -> list[PropertyScenarioData]:
if not property_ids:
return []
params: dict[str, Any] = {
"portfolio_id": portfolio_id,
"scenario_id": scenario_id,
"property_ids": list(property_ids),
}
return self._assemble_rows(_ROWS_SQL, _MEASURES_SQL, params)
def default_rows_for(
self,
*,
portfolio_id: int,
property_ids: Sequence[int],
) -> list[PropertyScenarioData]:
if not property_ids:
return []
# No scenario_id: the default-plan SQL selects on `is_default = TRUE`.
params: dict[str, Any] = {
"portfolio_id": portfolio_id,
"property_ids": list(property_ids),
}
return self._assemble_rows(
_DEFAULT_ROWS_SQL, _DEFAULT_MEASURES_SQL, params
)
def _assemble_rows(
self, rows_sql: Any, measures_sql: Any, params: dict[str, Any]
) -> list[PropertyScenarioData]:
measures = self._measures_by_property(params, measures_sql)
epc = self._effective_epc_by_property(params)
result = self._session.connection().execute(rows_sql, params)
rows: list[PropertyScenarioData] = []
for row in result.all():
facts = epc.get(row[0], {})
rows.append(
PropertyScenarioData(
property_id=row[0],
landlord_property_id=_str(row[1]),
uprn=_str(row[2]),
address=_str(row[3]),
postcode=_str(row[4]),
post_sap_points=_float(row[5]),
post_epc_rating=_str(row[6]),
cost_of_works=_float(row[7]),
contingency_cost=_float(row[8]),
co2_savings=_float(row[9]),
energy_bill_savings=_float(row[10]),
energy_consumption_savings=_float(row[11]),
valuation_increase=_float(row[12]),
plan_name=_str(row[13]),
property_type=facts.get("property_type"),
built_form=facts.get("built_form"),
property_age_band=facts.get("property_age_band"),
walls=facts.get("walls"),
roof=facts.get("roof"),
floor=facts.get("floor"),
windows=facts.get("windows"),
heating=facts.get("heating"),
hot_water=facts.get("hot_water"),
lighting=facts.get("lighting"),
total_floor_area=facts.get("total_floor_area"),
number_of_rooms=facts.get("number_of_rooms"),
lodgement_date=facts.get("lodgement_date"),
is_expired=facts.get("is_expired"),
current_epc_rating=facts.get("current_epc_rating"),
current_sap_points=facts.get("current_sap_points"),
measures=measures.get(row[0], ()),
)
)
return rows
def _measures_by_property(
self, params: dict[str, Any], measures_sql: Any = _MEASURES_SQL
) -> dict[int, list[ExportMeasure]]:
result = self._session.connection().execute(measures_sql, params)
by_property: dict[int, list[ExportMeasure]] = {}
for row in result.all():
by_property.setdefault(row[0], []).append(
ExportMeasure(
measure_type=_str(row[1]) or "",
estimated_cost=_float(row[2]),
sap_points=_float(row[3]),
co2_equivalent_savings=_float(row[4]),
kwh_savings=_float(row[5]),
energy_cost_savings=_float(row[6]),
includes_battery=bool(row[7]),
)
)
return by_property
def _effective_epc_by_property(
self, params: dict[str, Any]
) -> dict[int, dict[str, Any]]:
"""The override-resolved Effective-EPC descriptive picture per Property:
override → lodged → predicted (ADR-0065)."""
lodged = self._epc_facts_for_source(params, "lodged")
predicted = self._epc_facts_for_source(params, "predicted")
facts: dict[int, dict[str, Any]] = {}
for property_id in set(lodged) | set(predicted):
merged = dict(predicted.get(property_id, {}))
# A lodged cert, where present, wins field by field (a null lodged
# field falls through to the predicted value).
for key, value in lodged.get(property_id, {}).items():
if value is not None:
merged[key] = value
facts[property_id] = merged
for row in self._session.connection().execute(
_OVERRIDE_PROPERTY_TYPE_SQL, params
).all():
facts.setdefault(row[0], {})["property_type"] = _str(row[1])
return facts
def _epc_facts_for_source(
self, params: dict[str, Any], source: str
) -> dict[int, dict[str, Any]]:
connection = self._session.connection()
source_params: dict[str, Any] = {**params, "source": source}
facts: dict[int, dict[str, Any]] = {}
for row in connection.execute(_EPC_HEADER_SQL, source_params).all():
registration_date = _str(row[3])
facts.setdefault(row[0], {}).update(
{
"property_type": _str(row[1]),
"total_floor_area": _float(row[2]),
"lodgement_date": registration_date,
"is_expired": _is_expired(registration_date),
"number_of_rooms": row[4],
"built_form": built_form_description(row[5]),
}
)
# A Property can lodge several rows of one element type (e.g. two walls);
# join their descriptions, as data_exports did.
descriptions: dict[tuple[int, str], list[str]] = {}
for row in connection.execute(_EPC_ELEMENTS_SQL, source_params).all():
element_type = _str(row[1]) or ""
descriptions.setdefault((row[0], element_type), []).append(
_str(row[2]) or ""
)
for (property_id, element_type), descs in descriptions.items():
field = _ELEMENT_TO_FIELD.get(element_type)
if field is not None:
facts.setdefault(property_id, {})[field] = "; ".join(
d for d in descs if d
)
for row in connection.execute(_EPC_PERF_SQL, source_params).all():
facts.setdefault(row[0], {}).update(
{
# _float to match the read-model's Optional[float] typing and
# the sibling numeric facts (the driver hands back an int).
"current_sap_points": _float(row[1]),
"current_epc_rating": _str(row[2]),
}
)
# Structured SAP fabric (walls/roof/floor/heating/hot_water) composed
# from the canonical cert's coded fields; overrides the gov-EPC prose
# where present (a re-survey lodges these but no prose), and leaves the
# prose in place where a field renders to nothing.
for row in connection.execute(_EPC_STRUCTURED_FABRIC_SQL, source_params).all():
rendered = {
"property_age_band": age_band_description(row[17]),
"walls": wall_description(row[1], row[2]),
"roof": roof_description(row[3], row[4], row[5]),
"floor": floor_description(row[6], row[7], row[8]),
"heating": heating_description(row[9], row[10], row[11], row[12]),
"hot_water": hot_water_description(
row[13], row[14], row[9], row[15], row[16]
),
}
facts.setdefault(row[0], {}).update(
{field: value for field, value in rendered.items() if value is not None}
)
return facts

39
repositories/scenario_export/scenario_export_repository.py Normal file
View file

@ -0,0 +1,39 @@
from __future__ import annotations
from abc import ABC, abstractmethod
from typing import Sequence
from domain.scenario_export.scenario_sheet import PropertyScenarioData
class ScenarioExportRepository(ABC):
"""Reads the persisted data one Scenario Export sheet needs (ADR-0065)."""
@abstractmethod
def rows_for(
self,
*,
portfolio_id: int,
scenario_id: int,
property_ids: Sequence[int],
) -> list[PropertyScenarioData]:
"""The export rows for the Properties in ``property_ids`` that have a
default Plan under the Scenario — identity, override-resolved
Effective-EPC descriptive fields, the Plan's post-works figures, and the
selected measures. A Property with no default Plan for the Scenario is
absent (scenario-scoped, model A)."""
...
@abstractmethod
def default_rows_for(
self,
*,
portfolio_id: int,
property_ids: Sequence[int],
) -> list[PropertyScenarioData]:
"""The export rows for the Properties in ``property_ids`` selected by
each home's **default Plan** (``plan.is_default = TRUE``), which is
one-per-property across all Scenarios (ADR-0012 / ADR-0017). Unlike
``rows_for`` this is NOT scenario-scoped — it is the portfolio's current
state, one row per home. A Property with no default Plan is absent."""
...

31
repositories/scenario_export/scenario_names_postgres_repository.py Normal file
View file

@ -0,0 +1,31 @@
from __future__ import annotations
from sqlalchemy import text
from sqlmodel import Session
# Scenario id → display name. The domain Scenario aggregate does not carry the
# name, so it is read from the ``scenario`` table; kept in the repository layer
# with the export's other reads (ADR-0065), so the Lambda handler stays
# composition-only.
_SCENARIO_NAME_SQL = text("SELECT name FROM scenario WHERE id = :scenario_id")
class ScenarioNamesPostgresRepository:
"""Reads a Scenario's display name from Postgres, satisfying the
orchestrator's ``ScenarioNames`` port. Looked up lazily and cached (an export
names only its handful of requested Scenarios)."""
def __init__(self, session: Session) -> None:
self._session = session
self._cache: dict[int, str] = {}
def name_for(self, scenario_id: int) -> str:
if scenario_id not in self._cache:
row = (
self._session.connection()
.execute(_SCENARIO_NAME_SQL, {"scenario_id": scenario_id})
.first()
)
name = row[0] if row is not None else None
self._cache[scenario_id] = name or f"scenario_{scenario_id}"
return self._cache[scenario_id]

0
tests/applications/ara_export/__init__.py Normal file
View file

24
tests/applications/ara_export/test_ara_export_trigger_body.py Normal file
View file

@ -0,0 +1,24 @@
"""The ara_export SQS trigger body (ADR-0065): carries only the task/subtask
identifiers (the selection rides sub_task.inputs), tolerating extra fields."""
from __future__ import annotations
from uuid import UUID
from applications.ara_export.ara_export_trigger_body import AraExportTriggerBody
def test_parses_the_task_and_subtask_ids_and_tolerates_extra_fields() -> None:
# arrange — a message body with the two ids and an extra AWS-added field.
raw = {
"task_id": "11111111-1111-1111-1111-111111111111",
"subtask_id": "22222222-2222-2222-2222-222222222222",
"Records": "ignored",
}
# act
body = AraExportTriggerBody.model_validate(raw)
# assert — the ids parse as UUIDs; the extra field is tolerated, not fatal.
assert body.task_id == UUID("11111111-1111-1111-1111-111111111111")
assert body.subtask_id == UUID("22222222-2222-2222-2222-222222222222")

0
tests/backend/app/exports/__init__.py Normal file
View file

49
tests/backend/app/exports/test_export_tasks.py Normal file
View file

@ -0,0 +1,49 @@
"""ScenarioExportTasks property-id resolution (ADR-0065): an explicit
property_ids list overrides the filters; otherwise the shared Modelling Run
filter resolver (ADR-0056) resolves the portfolio's filtered set."""
from __future__ import annotations
from sqlalchemy import Engine
from sqlmodel import Session
from backend.app.exports.export_tasks import ScenarioExportTasks
from backend.app.modelling.property_filters import PropertyGroupFilters
from infrastructure.postgres.property_table import PropertyRow # registers `property`
def test_explicit_property_ids_override_the_filters(db_engine: Engine) -> None:
# arrange — filters that match nothing, but an explicit hand-picked list.
with Session(db_engine) as session:
# act
result = ScenarioExportTasks(session).resolve_property_ids(
portfolio_id=100,
filters=PropertyGroupFilters(postcodes=["ZZ1 1ZZ"]),
property_ids=[3, 1, 2, 1],
)
# assert — the hand-picked ids win (deduplicated, sorted); filters ignored.
assert result == [1, 2, 3]
def test_resolves_from_filters_when_no_explicit_ids(db_engine: Engine) -> None:
# arrange — two properties in the portfolio, one matching the postcode filter.
with Session(db_engine) as session:
session.add(
PropertyRow(id=1, portfolio_id=100, postcode="AB1 2CD", landlord_property_id="LP1")
)
session.add(
PropertyRow(id=2, portfolio_id=100, postcode="XY9 9ZZ", landlord_property_id="LP2")
)
session.commit()
# act
with Session(db_engine) as session:
result = ScenarioExportTasks(session).resolve_property_ids(
portfolio_id=100,
filters=PropertyGroupFilters(postcodes=["AB1 2CD"]),
property_ids=[],
)
# assert — only the property matching the filter is resolved.
assert result == [1]

93
tests/datatypes/epc/domain/test_mapper_external_measurement_conversion.py Normal file
View file

@ -0,0 +1,93 @@
"""#1669 — when a cert lodges its horizontal dimensions measured externally
(`measurement_type == 2`), RdSAP 10 §3.4 + Table 2 (spec p.17-18) require the
external floor area and exposed perimeter to be converted to INTERNAL dimensions
before any SAP use. `measurement_type` was read nowhere, so external certs
carried a TFA ~11-15% too large — corrupting occupancy, reported floor area and
every per-m² output. The mapper must plumb `measurement_type` through and, for
external certs, apply the Table 2 whole-dwelling ratio (§3.4 Note 4) before the
values reach the cascade, while leaving internal (`== 1`) certs verbatim.
"""
from __future__ import annotations
import json
from copy import deepcopy
from pathlib import Path
from typing import Any
from datatypes.epc.domain.mapper import EpcPropertyDataMapper
from domain.sap10_calculator.calculator import Sap10Calculator
_CORPUS = Path("backend/epc_api/json_samples/RdSAP-Schema-21.0.1/corpus.jsonl")
def _corpus_cert(uprn: str) -> dict[str, Any]:
for line in _CORPUS.read_text().splitlines():
if uprn in line:
cert: dict[str, Any] = json.loads(line)
return cert
raise AssertionError(f"uprn {uprn} not found in the RdSAP-21.0.1 corpus")
def _with_measurement_type(payload: dict[str, Any], value: int) -> dict[str, Any]:
twin = deepcopy(payload)
twin["measurement_type"] = value
return twin
def _mapped_tfa(payload: dict[str, Any]) -> float:
epc = EpcPropertyDataMapper.from_api_response(payload)
assert epc is not None
assert epc.total_floor_area_m2 is not None
return epc.total_floor_area_m2
def test_measurement_type_is_plumbed_onto_the_domain_object() -> None:
# Arrange — cert 100091435353 lodges measurement_type 2; the field was
# dropped to None by every API mapper (`# What is this?`).
payload = _corpus_cert("100091435353")
# Act
epc = EpcPropertyDataMapper.from_api_response(payload)
# Assert
assert epc is not None
assert epc.measurement_type == 2
def test_external_cert_tfa_is_converted_to_internal_dimensions() -> None:
# Arrange — 100091435353 (detached, w=270mm) lodges a 128.7 m^2 EXTERNAL
# floor-area sum; its lodged internal TFA is 112.
payload = _corpus_cert("100091435353")
# Act
tfa = _mapped_tfa(payload)
# Assert — Table 2 detached conversion recovers the internal TFA to <0.5 m^2,
# not the raw external 128.7.
assert abs(tfa - 112.0) <= 0.5
def test_internal_cert_dimensions_are_left_verbatim() -> None:
# Arrange — the same cert forced to measurement_type 1 (internal); §3.4 does
# not apply, so the floor areas must be used exactly as lodged (128.7).
payload = _with_measurement_type(_corpus_cert("100091435353"), 1)
# Act
tfa = _mapped_tfa(payload)
# Assert — no conversion; the external sum is passed through unchanged.
assert abs(tfa - 128.7) <= 1e-6
def test_external_conversion_moves_sap_toward_lodged() -> None:
# Arrange — 100091435353 over-rated at 66.23 on HEAD (external dims), lodged 65.
payload = _corpus_cert("100091435353")
epc = EpcPropertyDataMapper.from_api_response(payload)
assert epc is not None
# Act
sap = Sap10Calculator().calculate(epc).sap_score_continuous
# Assert — the internal-dimension conversion pulls it to ~65.33, toward lodged.
assert abs(sap - 65.33) <= 0.05

116
tests/datatypes/epc/domain/test_mapper_main_heating_fraction.py Normal file
View file

@ -0,0 +1,116 @@
"""#1666 — a two-main dwelling's `main_heating_fraction` pair is lodged by the
gov-EPC API in two incompatible encodings: a DECIMAL fraction summing to ~1.0
(`[0.8, 0.2]`) or an integer PERCENT summing to ~100 (`[80, 20]`). Every consumer
divides by 100 unconditionally, so a decimal-encoded pair collapses the second
main's share to ~1% of its true value and bills almost the whole load to the
first system's fuel/efficiency. The mapper must normalise both encodings to the
spec-canonical percent (RdSAP 10 item 7-5, spec p.81) so the encoding is
invisible downstream, without regressing the percent-encoded certs.
"""
from __future__ import annotations
import json
from copy import deepcopy
from pathlib import Path
from typing import Any, Optional, cast
from datatypes.epc.domain.mapper import EpcPropertyDataMapper
from domain.sap10_calculator.calculator import Sap10Calculator
_RDSAP_CORPUS = Path("backend/epc_api/json_samples/RdSAP-Schema-21.0.1/corpus.jsonl")
_SAP_17_1_CORPUS = Path("backend/epc_api/json_samples/SAP-Schema-17.1/corpus.jsonl")
def _corpus_cert(corpus: Path, uprn: str) -> dict[str, Any]:
for line in corpus.read_text().splitlines():
if uprn in line:
cert: dict[str, Any] = json.loads(line)
return cert
raise AssertionError(f"uprn {uprn} not found in {corpus.name}")
def _mapped_fractions(payload: dict[str, Any]) -> list[Optional[int]]:
epc = EpcPropertyDataMapper.from_api_response(payload)
assert epc is not None
return [m.main_heating_fraction for m in epc.sap_heating.main_heating_details if m]
def _continuous_sap(payload: dict[str, Any]) -> float:
epc = EpcPropertyDataMapper.from_api_response(payload)
assert epc is not None
return Sap10Calculator().calculate(epc).sap_score_continuous
def _scaled_fractions(payload: dict[str, Any], factor: float) -> dict[str, Any]:
"""Deep-copy `payload`, multiplying every lodged `main_heating_fraction` by
`factor` — used to build a percent-encoded twin of a decimal-encoded cert."""
twin = deepcopy(payload)
def walk(node: object) -> None:
if isinstance(node, dict):
node_dict = cast("dict[str, Any]", node)
for key, value in node_dict.items():
if key == "main_heating_fraction" and value is not None:
node_dict[key] = value * factor
else:
walk(value)
elif isinstance(node, list):
for value in cast("list[Any]", node):
walk(value)
walk(twin)
return twin
def test_decimal_encoded_fraction_pair_is_normalised_to_percent() -> None:
# Arrange — corpus cert 10070086972 (semi-detached, lodged SAP 18) lodges its
# two mains as a DECIMAL pair [0.8, 0.2] summing to 1.0; the /100 consumers
# would otherwise bill the second main at 0.2% of the load, not 20%.
payload = _corpus_cert(_RDSAP_CORPUS, "10070086972")
# Act
fractions = _mapped_fractions(payload)
# Assert — scaled to the spec-canonical percent (RdSAP 10 item 7-5, p.81).
assert fractions == [80, 20]
def test_percent_encoded_fraction_pair_is_left_unchanged() -> None:
# Arrange — corpus cert 40037847 lodges a PERCENT pair [10, 90] summing to
# ~100; the pair-sum discriminator must leave it exactly as lodged so the 3
# percent-encoded corpus certs do not regress.
payload = _corpus_cert(_RDSAP_CORPUS, "40037847")
# Act
fractions = _mapped_fractions(payload)
# Assert
assert fractions == [10, 90]
def test_encoding_is_invisible_to_the_continuous_sap() -> None:
# Arrange — the decimal cert and a PERCENT twin built by scaling its lodged
# fractions x100 ([0.8, 0.2] -> [80, 20]); the encoding must not change SAP.
decimal_payload = _corpus_cert(_RDSAP_CORPUS, "10070086972")
percent_payload = _scaled_fractions(decimal_payload, 100)
# Act
sap_decimal = _continuous_sap(decimal_payload)
sap_percent = _continuous_sap(percent_payload)
# Assert — the two encodings must produce the same continuous SAP.
assert abs(sap_decimal - sap_percent) <= 1e-9
def test_sap_17_1_decimal_second_main_is_not_floored_to_zero() -> None:
# Arrange — SAP-Schema-17.1 cert 38334849 lodges two mains as a DECIMAL pair
# [0.5, 0.5]. The old `int(fraction)` floor mapped BOTH to 0, silently
# deleting the split; the shared normaliser must scale to percent instead.
payload = _corpus_cert(_SAP_17_1_CORPUS, "38334849")
# Act
fractions = _mapped_fractions(payload)
# Assert
assert fractions == [50, 50]

100
tests/datatypes/epc/domain/test_mapper_metal_window_frame.py Normal file
View file

@ -0,0 +1,100 @@
"""#1667 — on the gov-EPC API path a window's metal frame was unrepresentable:
the transmission lookup had no frame dimension, so every window was billed at
the PVC/wooden U-column and frame factor 0.70 even where the cert lodges a metal
frame (`pvc_frame:"false"`). RdSAP 10 Table 24 gives metal a higher U (+0.2 to
+0.9 W/m^2K) and SAP 10.2 Table 6c a frame factor of 0.8, so we over-rated
metal-framed dwellings. The mapper must route a metal-lodged window to the
Table 24 metal column + FF 0.8, leaving PVC-lodged windows unchanged.
"""
from __future__ import annotations
import json
from copy import deepcopy
from pathlib import Path
from typing import Any, cast
from datatypes.epc.domain.mapper import EpcPropertyDataMapper
from domain.sap10_calculator.calculator import Sap10Calculator
_CORPUS = Path("backend/epc_api/json_samples/RdSAP-Schema-21.0.1/corpus.jsonl")
def _corpus_cert(uprn: str) -> dict[str, Any]:
for line in _CORPUS.read_text().splitlines():
if uprn in line:
cert: dict[str, Any] = json.loads(line)
return cert
raise AssertionError(f"uprn {uprn} not found in the RdSAP-21.0.1 corpus")
def _forced_pvc_frame(payload: dict[str, Any], value: str) -> dict[str, Any]:
"""Deep-copy `payload`, setting every lodged `pvc_frame` to `value` — used to
build a PVC-framed twin of a metal-framed cert."""
twin = deepcopy(payload)
def walk(node: object) -> None:
if isinstance(node, dict):
node_dict = cast("dict[str, Any]", node)
for key, child in node_dict.items():
if key == "pvc_frame" and child is not None:
node_dict[key] = value
else:
walk(child)
elif isinstance(node, list):
for child in cast("list[Any]", node):
walk(child)
walk(twin)
return twin
def _window_u_and_ff(payload: dict[str, Any]) -> tuple[set[float], set[float]]:
epc = EpcPropertyDataMapper.from_api_response(payload)
assert epc is not None
u_values = {
w.window_transmission_details.u_value
for w in epc.sap_windows
if w.window_transmission_details is not None
}
frame_factors = {w.frame_factor for w in epc.sap_windows if w.frame_factor is not None}
return u_values, frame_factors
def test_metal_frame_window_bills_table_24_metal_u_and_frame_factor_0p8() -> None:
# Arrange — cert 21067296 lodges six pvc_frame:"false" (metal) double-glazed
# (type 3, 12 mm) windows; HEAD billed them on the PVC column (U 2.8, FF 0.70).
payload = _corpus_cert("21067296")
# Act
u_values, frame_factors = _window_u_and_ff(payload)
# Assert — Table 24 metal DG-12mm column (3.4) and SAP 10.2 Table 6c FF 0.8.
assert u_values == {3.4}
assert frame_factors == {0.8}
def test_pvc_frame_window_stays_on_the_pvc_column() -> None:
# Arrange — the same cert forced to pvc_frame:"true"; the fix must not
# regress PVC/wooden-framed windows (the discriminator is the boolean).
payload = _forced_pvc_frame(_corpus_cert("21067296"), "true")
# Act
u_values, frame_factors = _window_u_and_ff(payload)
# Assert — PVC DG-12mm column (2.8) and FF 0.70, exactly as before.
assert u_values == {2.8}
assert frame_factors == {0.7}
def test_metal_frame_fix_moves_cert_toward_lodged() -> None:
# Arrange — 21067296 scored 68.60 on HEAD (PVC assumption), lodged 67.
payload = _corpus_cert("21067296")
epc = EpcPropertyDataMapper.from_api_response(payload)
assert epc is not None
# Act
sap = Sap10Calculator().calculate(epc).sap_score_continuous
# Assert — the metal U-penalty pulls it to ~66.96, i.e. toward lodged 67.
assert abs(sap - 66.96) <= 0.05

13
tests/domain/epc_prediction/test_component_accuracy_gate.py
View file

@ -124,11 +124,20 @@ _RATE_FLOORS: dict[str, float] = {
# dimensional predictions); floor_area loosened 12.0378 -> 12.0586 as the one
# physical residual that fell (1-2 targets picking a new-build donor). See the
# _RATE_FLOORS note above.
# total_window_area 3.7184 -> 3.7484 and door_count 0.3333 -> 0.3737 re-baselined
# when the external-measurement conversion landed (#1669: measurement_type=2 certs
# now have their floor area + exposed perimeter converted external->internal per
# RdSAP 10 §3.4 + Table 2). The leave-one-out scorer re-derives the *actual*
# EpcPropertyData through the same mapper, so the sharper (internal) dimensions of
# the fixture's external certs shift the geo-proximity-weighted donor mode — a
# ground-truth-method change, not a prediction-logic loosening (spec-correct,
# corpus MAE 0.571 -> 0.570 and the real-cert pin held). The move is two
# single-target donor tips on the n=36 fixture. Tighten-only resumes from here.
_RESIDUAL_CEILINGS: dict[str, float] = {
"floor_area": 12.0586,
"total_window_area": 3.7184,
"total_window_area": 3.7484,
"building_parts": 0.1212,
"door_count": 0.3333,
"door_count": 0.3737,
}
_TOLERANCE = 1e-3

11
tests/domain/epc_prediction/test_expired_pairs_gate.py
View file

@ -55,6 +55,15 @@ _FIXTURE = (
# 28-scoreable fixture) and residual ceilings — the measured values; tighten,
# never loosen. The general (unconditioned) prediction floors live in
# test_component_accuracy_gate.py; this gate guards the CONDITIONING path.
#
# has_pv re-baselined 0.8929 -> 0.8571 when the external-measurement conversion
# landed (#1669: measurement_type=2 certs converted external->internal per RdSAP
# 10 §3.4 + Table 2). This gate re-derives the *actual* EpcPropertyData through
# the same mapper, so the sharper internal dimensions of the fixture's external
# certs shift the conditioned donor mode and one pair's has_pv agreement tips
# 25/28 -> 24/28 — a ground-truth-method change, not a prediction-logic loosening
# (spec-correct; corpus MAE improved and the real-cert pin held). Tighten-only
# resumes from here.
_CLASSIFICATION_FLOORS: dict[str, float] = {
"construction_age_band": 0.5357,
"construction_age_band_pm1": 0.8214,
@ -62,7 +71,7 @@ _CLASSIFICATION_FLOORS: dict[str, float] = {
"floor_construction": 0.8636,
"floor_insulation": 1.0000,
"has_hot_water_cylinder": 0.8214,
"has_pv": 0.8929,
"has_pv": 0.8571,
"has_room_in_roof": 0.8571,
"heating_main_category": 1.0000,
"heating_main_control": 0.6071,

78
tests/domain/sap10_calculator/rdsap/test_underfloor_responsiveness.py Normal file
View file

@ -0,0 +1,78 @@
"""#1668 — wet underfloor responsiveness was pinned at R=0.75 for the lumped
gov-EPC emitter code 2, regardless of floor construction. SAP 10.2 Table 4d
splits it three ways and RdSAP 10 Table 29 (p.56-57) selects which from floor
construction + age band: insulated timber floor R=1.0, screed above insulation
R=0.75, concrete slab R=0.25. A concrete-slab system (solid floor, age A-E) was
billed as screed (0.75), over-stating responsiveness and over-rating the
dwelling. The calculator must derive the Table 29 subtype from the main part's
floor construction + age band.
"""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
from datatypes.epc.domain.mapper import EpcPropertyDataMapper
from domain.sap10_calculator.calculator import Sap10Calculator
from domain.sap10_calculator.rdsap.cert_to_inputs import (
_underfloor_responsiveness, # pyright: ignore[reportPrivateUsage]
)
_CORPUS = Path("backend/epc_api/json_samples/RdSAP-Schema-21.0.1/corpus.jsonl")
def _corpus_cert(uprn: str) -> dict[str, Any]:
for line in _CORPUS.read_text().splitlines():
if uprn in line:
cert: dict[str, Any] = json.loads(line)
return cert
raise AssertionError(f"uprn {uprn} not found in the RdSAP-21.0.1 corpus")
def test_solid_floor_age_a_to_e_is_a_concrete_slab_r_0p25() -> None:
# Arrange / Act / Assert — RdSAP 10 Table 29: a solid floor in age bands A-E
# has no insulation layer, so the underfloor pipes sit in a concrete slab.
assert _underfloor_responsiveness("Solid", "B") == 0.25
assert _underfloor_responsiveness("Solid", "E") == 0.25
def test_solid_floor_age_f_onwards_is_screed_r_0p75() -> None:
# Arrange / Act / Assert — newer solid floors carry insulation, so the pipes
# are in screed above it (0.75), not a bare slab.
assert _underfloor_responsiveness("Solid", "F") == 0.75
assert _underfloor_responsiveness("Solid", "M") == 0.75
def test_suspended_timber_floor_is_r_1p0() -> None:
# Arrange / Act / Assert — a timber-floor underfloor system is fully
# responsive (Table 4d timber floor = 1.0).
assert _underfloor_responsiveness("Suspended timber", "D") == 1.0
def test_suspended_not_timber_is_screed_r_0p75_not_timber() -> None:
# Arrange / Act / Assert — "Suspended, not timber" must NOT be read as timber
# (the substring trap); it falls to the screed default 0.75.
assert _underfloor_responsiveness("Suspended, not timber", "C") == 0.75
def test_unknown_or_absent_floor_defaults_to_screed_r_0p75() -> None:
# Arrange / Act / Assert — a main part with no ground floor / unlodged
# construction keeps the prior 0.75, so ambiguous certs do not move.
assert _underfloor_responsiveness(None, "A") == 0.75
assert _underfloor_responsiveness("", None) == 0.75
def test_concrete_slab_cert_corrects_to_lodged() -> None:
# Arrange — cert 100100137438 (solid floor, band B, wet underfloor) over-rated
# at 72 on HEAD via the pinned 0.75; lodged 69.
payload = _corpus_cert("100100137438")
epc = EpcPropertyDataMapper.from_api_response(payload)
assert epc is not None
# Act
sap = Sap10Calculator().calculate(epc).sap_score
# Assert — Table 29 concrete-slab R=0.25 corrects it to exactly lodged 69.
assert sap == 69

0
tests/domain/scenario_export/__init__.py Normal file
View file

174
tests/domain/scenario_export/test_fabric_description.py Normal file
View file

@ -0,0 +1,174 @@
"""Behaviour of the structured-fabric renderers (ADR-0065): composing client
sheet text from an EPC's SAP integer codes, decoding with the SAP engine's own
code space and normalising the ragged shapes a re-survey lodges."""
from __future__ import annotations
from domain.scenario_export.fabric_description import (
age_band_description,
built_form_description,
floor_description,
heating_description,
hot_water_description,
roof_description,
wall_description,
)
class TestBuiltFormDescription:
def test_lodged_display_text_passes_through(self) -> None:
assert built_form_description("Mid-terrace") == "Mid-terrace"
assert built_form_description("Semi-Detached") == "Semi-Detached"
def test_integer_code_is_decoded(self) -> None:
assert built_form_description(4) == "Mid-terrace"
assert built_form_description("1") == "Detached"
def test_absent_or_unknown_code_renders_nothing(self) -> None:
assert built_form_description(None) is None
assert built_form_description(99) is None
class TestAgeBandDescription:
def test_rdsap_letter_decodes_to_date_range_and_band(self) -> None:
assert age_band_description("B") == "1900-1929 (band B)"
assert age_band_description("A") == "before 1900 (band A)"
assert age_band_description("M") == "2023 onwards (band M)"
def test_lowercase_and_whitespace_are_tolerated(self) -> None:
assert age_band_description(" k ") == "2007-2011 (band K)"
def test_unknown_or_absent_band_renders_nothing(self) -> None:
assert age_band_description("Unknown") is None
assert age_band_description(None) is None
assert age_band_description("Z") is None
class TestWallDescription:
def test_construction_and_insulation_codes_decode_to_text(self) -> None:
assert wall_description(4, 2) == "Cavity wall, filled cavity"
assert wall_description(3, 4) == "Solid brick wall, as built"
assert wall_description(5, 1) == "Timber frame wall, external insulation"
def test_construction_only_when_insulation_unknown(self) -> None:
assert wall_description(4, None) == "Cavity wall"
assert wall_description(4, 99) == "Cavity wall"
def test_numeric_string_codes_are_accepted(self) -> None:
# JSONB can round-trip a code as a string; it still decodes.
assert wall_description("4", "2") == "Cavity wall, filled cavity"
def test_unknown_construction_renders_nothing(self) -> None:
assert wall_description(None, 2) is None
assert wall_description(99, 2) is None
class TestRoofDescription:
def test_construction_prose_with_thickness_and_location(self) -> None:
assert (
roof_description("Pitched roof, Access to loft", "Joists", 270)
== "Pitched roof, Access to loft; 270mm insulation at joists"
)
def test_thickness_string_with_unit_is_normalised(self) -> None:
# A minority of records lodge the thickness as "225mm" rather than 225.
assert (
roof_description("Pitched", "Joists", "225mm")
== "Pitched; 225mm insulation at joists"
)
def test_non_numeric_thickness_and_coded_location_are_dropped(self) -> None:
# thickness "As built" is not a measurement; a bare int location is a
# code, not a place — with neither, only the construction survives.
assert roof_description("Pitched", 2, "As built") == "Pitched"
def test_named_location_without_a_thickness_still_notes_insulation(self) -> None:
assert (
roof_description("Pitched", "Joists", "As built")
== "Pitched; insulation at joists"
)
def test_zero_thickness_reads_as_no_insulation(self) -> None:
assert roof_description("Flat roof", "Joists", 0) == "Flat roof; no insulation"
def test_unknown_location_is_omitted(self) -> None:
assert roof_description("Pitched", "Unknown", 100) == "Pitched; 100mm insulation"
def test_construction_only_when_no_insulation_detail(self) -> None:
assert roof_description("Pitched", None, None) == "Pitched"
def test_no_construction_renders_nothing(self) -> None:
assert roof_description(None, "Joists", 270) is None
class TestFloorDescription:
def test_composes_type_construction_and_insulation(self) -> None:
assert (
floor_description("Ground floor", "Suspended, timber", "As Built")
== "Ground floor, suspended, timber, insulation as built"
)
def test_normalises_casing_of_the_floor_type(self) -> None:
assert floor_description("Ground Floor", "Solid", None) == "Ground floor, solid"
def test_retrofitted_insulation_is_carried(self) -> None:
assert (
floor_description("Ground floor", "Solid", "Retro-fitted")
== "Ground floor, solid, insulation retro-fitted"
)
def test_empty_when_nothing_lodged(self) -> None:
assert floor_description(None, None, None) is None
class TestHeatingDescription:
def test_fuel_emitter_control_and_condensing(self) -> None:
assert (
heating_description(26, 1, 2106, True)
== "Mains gas, radiators (condensing); programmer, room thermostat and TRVs"
)
def test_condensing_false_or_absent_omits_the_flag(self) -> None:
assert heating_description(26, 1, 2104, None) == (
"Mains gas, radiators; programmer and room thermostat"
)
assert heating_description(26, 1, None, False) == "Mains gas, radiators"
def test_unknown_control_is_dropped(self) -> None:
assert heating_description(26, 1, 9999, True) == "Mains gas, radiators (condensing)"
def test_empty_when_no_fuel_or_emitter(self) -> None:
assert heating_description(None, None, 2106, True) is None
class TestHotWaterDescription:
def test_lodged_code_and_fuel_decode(self) -> None:
assert (
hot_water_description(901, 26, 26, False, False) == "From main system, mains gas"
)
assert (
hot_water_description(903, 29, 26, True, False)
== "Electric immersion, electricity"
)
def test_falls_back_to_main_system_when_no_code_and_no_cylinder(self) -> None:
# SAP Table 4a: a dwelling with no cylinder draws hot water from the main
# system; the main heating fuel names it.
assert (
hot_water_description(None, None, 26, False, False)
== "From main system, mains gas"
)
def test_cylinder_without_a_code(self) -> None:
assert (
hot_water_description(None, None, 29, True, False)
== "Hot water cylinder, electricity"
)
def test_solar_water_heating_is_appended(self) -> None:
assert hot_water_description(901, 26, 26, False, True) == (
"From main system, mains gas; with solar water heating"
)
def test_empty_when_nothing_lodged(self) -> None:
assert hot_water_description(None, None, None, None, False) is None

226
tests/domain/scenario_export/test_scenario_sheet.py Normal file
View file

@ -0,0 +1,226 @@
"""The Scenario Export sheet shape (ADR-0065) — pure layout of one scenario
sheet: measure costs pivoted onto a frozen column contract, savings summed per
Property, and the Plan's post-works figures."""
from typing import Optional, Sequence
import pytest
from domain.scenario_export.scenario_sheet import (
MEASURE_COLUMNS,
ExportMeasure,
PropertyScenarioData,
shape_scenario_sheet,
)
def _measure(
*,
measure_type: str = "loft_insulation",
estimated_cost: Optional[float] = 1000.0,
sap_points: Optional[float] = None,
co2_equivalent_savings: Optional[float] = None,
kwh_savings: Optional[float] = None,
energy_cost_savings: Optional[float] = None,
includes_battery: bool = False,
) -> ExportMeasure:
return ExportMeasure(
measure_type=measure_type,
estimated_cost=estimated_cost,
sap_points=sap_points,
co2_equivalent_savings=co2_equivalent_savings,
kwh_savings=kwh_savings,
energy_cost_savings=energy_cost_savings,
includes_battery=includes_battery,
)
def _property(
*,
property_id: int = 1,
landlord_property_id: Optional[str] = "LP1",
measures: Sequence[ExportMeasure] = (),
**fields: object,
) -> PropertyScenarioData:
return PropertyScenarioData(
property_id=property_id,
landlord_property_id=landlord_property_id,
measures=measures,
**fields, # pyright: ignore[reportArgumentType]
)
def test_a_measures_cost_lands_in_its_own_column_with_the_property_identity() -> None:
# arrange — one Property with one loft-insulation measure priced at £1200.
prop = _property(
property_id=42,
landlord_property_id="LP42",
measures=[_measure(measure_type="loft_insulation", estimated_cost=1200.0)],
)
# act
sheet = shape_scenario_sheet([prop])
# assert — one row carrying the Property identity and the cost in the
# loft_insulation column.
assert len(sheet.rows) == 1
row = sheet.rows[0]
assert row["property_id"] == 42
assert row["landlord_property_id"] == "LP42"
assert row["loft_insulation"] == 1200.0
def test_every_measure_column_is_present_and_blank_where_the_property_lacks_it() -> None:
# arrange — a Property with a single measure. Every other measure column
# must still appear on the sheet (the frozen contract, ADR-0065), blank for
# this Property, so all scenario sheets share one column set.
prop = _property(
measures=[_measure(measure_type="cavity_wall_insulation", estimated_cost=800.0)]
)
# act
sheet = shape_scenario_sheet([prop])
# assert — the full frozen measure contract is present; the one selected
# measure carries its cost, every other measure column is blank.
assert set(MEASURE_COLUMNS).issubset(set(sheet.columns))
row = sheet.rows[0]
assert row["cavity_wall_insulation"] == 800.0
assert row["loft_insulation"] == ""
assert row["solar_pv"] == ""
def test_savings_and_sap_points_are_summed_across_a_propertys_measures() -> None:
# arrange — two measures, each pivoting to its own cost column and each
# contributing SAP points and carbon / energy / bill savings.
prop = _property(
measures=[
_measure(
measure_type="loft_insulation",
estimated_cost=1000.0,
sap_points=3.0,
co2_equivalent_savings=0.4,
kwh_savings=900.0,
energy_cost_savings=120.0,
),
_measure(
measure_type="solar_pv",
estimated_cost=5000.0,
sap_points=5.0,
co2_equivalent_savings=1.1,
kwh_savings=2500.0,
energy_cost_savings=300.0,
),
]
)
# act
sheet = shape_scenario_sheet([prop])
# assert — each cost pivots to its own column, and the savings roll up to the
# Property's totals.
row = sheet.rows[0]
assert row["loft_insulation"] == 1000.0
assert row["solar_pv"] == 5000.0
assert row["sap_points"] == 8.0
assert row["co2_equivalent_savings"] == pytest.approx(1.5)
assert row["kwh_savings"] == 3400.0
assert row["energy_cost_savings"] == 420.0
def test_total_retrofit_cost_sums_the_per_measure_costs() -> None:
# arrange — a Property with two priced measures.
prop = _property(
measures=[
_measure(measure_type="loft_insulation", estimated_cost=1200.0),
_measure(measure_type="cavity_wall_insulation", estimated_cost=800.0),
]
)
# act
sheet = shape_scenario_sheet([prop])
# assert — the package total is the sum of the measure costs.
assert sheet.rows[0]["total_retrofit_cost"] == 2000.0
def test_solar_pv_with_a_battery_pivots_to_its_own_battery_column() -> None:
# arrange — a solar_pv Option that includes a battery.
prop = _property(
measures=[
_measure(
measure_type="solar_pv", estimated_cost=6000.0, includes_battery=True
)
]
)
# act
sheet = shape_scenario_sheet([prop])
# assert — the cost lands in the battery variant column (a stable part of the
# contract), the plain solar_pv column stays blank, and the total still
# counts it.
assert "solar_pv_with_battery" in sheet.columns
row = sheet.rows[0]
assert row["solar_pv_with_battery"] == 6000.0
assert row["solar_pv"] == ""
assert row["total_retrofit_cost"] == 6000.0
def test_plan_post_works_figures_come_from_the_plan_row_blank_when_absent() -> None:
# arrange — a Property whose default Plan carries the SAP calculator's
# post-works figures; the contingency figure happens to be absent.
prop = _property(
post_sap_points=78.0,
post_epc_rating="C",
cost_of_works=4200.0,
co2_savings=1.8,
energy_bill_savings=350.0,
energy_consumption_savings=4100.0,
valuation_increase=15000.0,
)
# act
sheet = shape_scenario_sheet([prop])
# assert — the persisted Plan figures pass through directly (no recompute,
# ADR-0065); an absent figure is blank rather than zero.
row = sheet.rows[0]
assert row["post_sap_points"] == 78.0
assert row["post_epc_rating"] == "C"
assert row["cost_of_works"] == 4200.0
assert row["valuation_increase"] == 15000.0
assert row["contingency_cost"] == ""
def test_identity_and_effective_epc_descriptive_fields_pass_through() -> None:
# arrange — a Property with identity and (already override-resolved)
# Effective-EPC descriptive fields; the floor description is absent.
prop = _property(
property_id=7,
landlord_property_id="LP7",
uprn="100023",
address="12 Oak Street",
postcode="AB1 2CD",
property_type="House",
walls="Cavity wall, as built, insulated",
roof="Pitched, 250mm loft insulation",
current_epc_rating="D",
current_sap_points=58.0,
)
# act
sheet = shape_scenario_sheet([prop])
# assert — identity and descriptive fields appear as supplied; an unsupplied
# descriptive field is blank.
row = sheet.rows[0]
assert row["uprn"] == "100023"
assert row["address"] == "12 Oak Street"
assert row["postcode"] == "AB1 2CD"
assert row["property_type"] == "House"
assert row["walls"] == "Cavity wall, as built, insulated"
assert row["roof"] == "Pitched, 250mm loft insulation"
assert row["current_epc_rating"] == "D"
assert row["current_sap_points"] == 58.0
assert row["floor"] == ""

18
tests/infrastructure/epc_client/test_sap_accuracy_corpus.py
View file

@ -280,7 +280,18 @@ _CORPUS = Path(
# (U=0), previously mislabelled a pitched roof so the party override never fired.
# Combined with main's gable fix: within-0.5 79.5% -> 80.3%, MAE 0.599 -> 0.584,
# PE 2.71 -> 2.5. Ratcheted.
_MIN_WITHIN_HALF_SAP = 0.803
# SAP-10.2 CALCULATOR ACCURACY BATCH (#1656 backlog: #1666/#1667/#1669/#1668,
# all stacked on the merged 80.3% baseline above):
# #1666 normalise dual-encoded main_heating_fraction (decimal vs percent) ->
# 80.3% -> 80.6%, MAE 0.584 -> 0.578;
# #1667 bill metal-framed API windows on the Table 24 metal U-column + FF 0.8 ->
# 80.6% -> 81.0%, MAE 0.578 -> 0.571;
# #1669 convert measurement_type=2 external dims to internal (RdSAP §3.4/Table 2)
# -> 81.0% -> 81.2%, MAE 0.571 -> 0.570, PE 2.5 -> 2.4;
# #1668 derive wet-underfloor Table 29 subtype (concrete-slab R=0.25) ->
# 81.2%, MAE 0.570 -> 0.565.
# Measured within-0.5 = 0.812000, MAE = 0.565082. Ratcheted.
_MIN_WITHIN_HALF_SAP = 0.812
# 0.793 -> 0.789 via the §12 Unknown-meter + dual-electric-immersion off-peak
# trigger (RdSAP 10 PDF p.62): Apartment 241 (main 691 + 903 dual immersion)
# -5.38 -> -1.05. Worksheet-validated on "simulated case 48" (Elmhurst SAP 57,
@ -415,7 +426,10 @@ _MIN_WITHIN_HALF_SAP = 0.803
# ceiling roof-code-3 slice stacking on main's #1662 floor-to-unheated-space
# fix: merged-corpus MAE measured 0.583761 (within-0.5 80.3%), rounded up to
# 3 d.p. so the `<=` assert holds.
_MAX_SAP_MAE = 0.584
# Ratcheted 0.584 -> 0.566 by the #1666/#1667/#1669/#1668 calculator-accuracy
# batch (see the _MIN_WITHIN_HALF_SAP note above): merged-corpus MAE measured
# 0.565082, rounded up to 3 d.p.
_MAX_SAP_MAE = 0.566
_MAX_CO2_MAE_TONNES = 0.068 # t CO2 / yr vs co2_emissions_current
_MAX_PE_PER_M2_MAE = 2.72 # kWh / m2 / yr vs energy_consumption_current

0
tests/infrastructure/xlsx/__init__.py Normal file
View file

75
tests/infrastructure/xlsx/test_scenario_export_workbook.py Normal file
View file

@ -0,0 +1,75 @@
"""The Scenario Export workbook renderer (ADR-0065): one branded .xlsx with one
sheet per Scenario, headers then rows, verified by re-opening the bytes."""
from __future__ import annotations
from pathlib import Path
from openpyxl import load_workbook
from domain.scenario_export.scenario_sheet import ExportSheet
from infrastructure.xlsx.branding import HEADER_FILL
from infrastructure.xlsx.scenario_export_workbook import render_workbook
def test_renders_a_sheet_named_for_the_scenario_with_headers_then_rows(
tmp_path: Path,
) -> None:
# arrange — one scenario's shaped sheet.
sheet = ExportSheet(
columns=("property_id", "loft_insulation"),
rows=({"property_id": 1, "loft_insulation": 1200.0},),
)
# act
path = str(tmp_path / "export.xlsx")
render_workbook([("Fabric first", sheet)], path)
# assert — a real workbook with a sheet named for the scenario, the column
# contract as the header row, and the data beneath it.
workbook = load_workbook(path)
assert workbook.sheetnames == ["Fabric first"]
worksheet = workbook["Fabric first"]
assert [cell.value for cell in worksheet[1]] == ["property_id", "loft_insulation"]
assert [cell.value for cell in worksheet[2]] == [1, 1200.0]
def test_sanitises_and_deduplicates_scenario_sheet_names(tmp_path: Path) -> None:
# arrange — a name over Excel's 31-char limit (used twice, so it must be
# deduplicated) and one with characters Excel forbids in a sheet title.
empty = ExportSheet(columns=("property_id",), rows=())
long_name = "A really long scenario name that exceeds Excel's limit"
illegal_name = "Bad:/\\?*[]name"
# act
path = str(tmp_path / "export.xlsx")
render_workbook(
[(long_name, empty), (long_name, empty), (illegal_name, empty)], path
)
# assert — three distinct, Excel-legal sheet names.
names = load_workbook(path).sheetnames
assert len(names) == 3
assert len(set(names)) == 3
assert all(len(name) <= 31 for name in names)
assert not any(ch in name for name in names for ch in r":/\?*[]")
def test_brands_the_header_row_and_freezes_it(tmp_path: Path) -> None:
# arrange
sheet = ExportSheet(
columns=("property_id", "loft_insulation"),
rows=({"property_id": 1, "loft_insulation": 1200.0},),
)
# act
path = str(tmp_path / "export.xlsx")
render_workbook([("Fabric first", sheet)], path)
# assert — the header is a bold, brand-filled band and the header row is
# frozen so it stays visible while scrolling.
worksheet = load_workbook(path)["Fabric first"]
header_cell = worksheet["A1"]
assert header_cell.font.bold is True
assert header_cell.fill.fgColor.rgb == HEADER_FILL
assert worksheet.freeze_panes == "A2"

253
tests/orchestration/test_scenario_export_orchestrator.py Normal file
View file

@ -0,0 +1,253 @@
"""The Scenario Export orchestrator (ADR-0065): read → shape → render → upload →
presign → email, best-effort, verified with in-memory fakes and a moto S3."""
from __future__ import annotations
from io import BytesIO
from typing import Iterator, Optional, Sequence
import pytest
from moto import mock_aws
from openpyxl import load_workbook
from domain.scenario_export.scenario_sheet import ExportMeasure, PropertyScenarioData
from domain.tasks.subtasks import SubTaskFailure
from infrastructure.s3.s3_client import S3Client
from orchestration.scenario_export_orchestrator import ScenarioExportOrchestrator
from tests.infrastructure import make_boto_client
EXPORTS_BUCKET = "domna-exports-dev"
class _FakeRows:
"""Returns preset export rows per scenario id, and a preset default-plan set."""
def __init__(
self,
by_scenario: dict[int, list[PropertyScenarioData]],
default_rows: Optional[list[PropertyScenarioData]] = None,
) -> None:
self._by_scenario = by_scenario
self._default_rows = default_rows or []
def rows_for(
self,
*,
portfolio_id: int,
scenario_id: int,
property_ids: Sequence[int],
) -> list[PropertyScenarioData]:
return self._by_scenario.get(scenario_id, [])
def default_rows_for(
self,
*,
portfolio_id: int,
property_ids: Sequence[int],
) -> list[PropertyScenarioData]:
return self._default_rows
class _FakeScenarioNames:
def __init__(self, names: dict[int, str]) -> None:
self._names = names
def name_for(self, scenario_id: int) -> str:
return self._names[scenario_id]
class _RecordingEmail:
def __init__(self) -> None:
self.sent: list[tuple[str, str, str, Optional[str]]] = []
def send(
self, *, to: str, subject: str, body: str, html_body: Optional[str] = None
) -> None:
self.sent.append((to, subject, body, html_body))
def _row(property_id: int) -> PropertyScenarioData:
return PropertyScenarioData(
property_id=property_id,
landlord_property_id=f"LP{property_id}",
measures=[ExportMeasure(measure_type="loft_insulation", estimated_cost=1200.0)],
)
@pytest.fixture
def exports() -> Iterator[S3Client]:
with mock_aws():
boto_client = make_boto_client("s3")
boto_client.create_bucket(Bucket=EXPORTS_BUCKET)
yield S3Client(boto_client, EXPORTS_BUCKET)
def test_builds_and_uploads_a_sheet_per_scenario_workbook(exports: S3Client) -> None:
# arrange — two scenarios, each with a modelled property.
orchestrator = ScenarioExportOrchestrator(
rows=_FakeRows({5: [_row(1)], 9: [_row(1), _row(2)]}),
scenarios=_FakeScenarioNames({5: "Fabric first", 9: "Low carbon"}),
exports=exports,
email=_RecordingEmail(),
url_ttl_seconds=3600,
)
# act
result = orchestrator.run(
portfolio_id=100,
scenario_ids=[5, 9],
property_ids=[1, 2],
recipient_email="user@example.com",
export_name="portfolio-100",
)
# assert — the uploaded object is a real workbook with one sheet per scenario
# and the presigned URL points at it.
assert result.sheet_count == 2
assert result.export_s3_key in result.presigned_url
workbook = load_workbook(BytesIO(exports.get_object(result.export_s3_key)))
assert workbook.sheetnames == ["Fabric first", "Low carbon"]
def test_emails_the_requester_the_presigned_download_link(exports: S3Client) -> None:
# arrange
email = _RecordingEmail()
orchestrator = ScenarioExportOrchestrator(
rows=_FakeRows({5: [_row(1)]}),
scenarios=_FakeScenarioNames({5: "Fabric first"}),
exports=exports,
email=email,
url_ttl_seconds=3600,
)
# act
result = orchestrator.run(
portfolio_id=100,
scenario_ids=[5],
property_ids=[1],
recipient_email="user@example.com",
export_name="portfolio-100",
)
# assert — the requester is emailed the link (raw URL in the plain-text
# fallback, a clean button in the HTML alternative).
assert len(email.sent) == 1
to, _subject, body, html_body = email.sent[0]
assert to == "user@example.com"
assert result.presigned_url in body
assert html_body is not None
assert "Download" in html_body
class _FailingEmail:
def send(
self, *, to: str, subject: str, body: str, html_body: Optional[str] = None
) -> None:
raise RuntimeError("smtp down")
def test_an_email_failure_does_not_lose_the_built_export(exports: S3Client) -> None:
# arrange — the workbook uploads fine, but email delivery blows up.
orchestrator = ScenarioExportOrchestrator(
rows=_FakeRows({5: [_row(1)]}),
scenarios=_FakeScenarioNames({5: "Fabric first"}),
exports=exports,
email=_FailingEmail(),
url_ttl_seconds=3600,
)
# act — must not raise.
result = orchestrator.run(
portfolio_id=100,
scenario_ids=[5],
property_ids=[1],
recipient_email="user@example.com",
export_name="portfolio-100",
)
# assert — the export was still uploaded and its link returned (it also lands
# on sub_task.outputs, so the link isn't lost).
assert result.presigned_url
assert exports.get_object(result.export_s3_key)
def test_a_selection_that_yields_no_rows_is_a_recorded_failure(
exports: S3Client,
) -> None:
# arrange — no property has a default Plan under the scenario (model A), so
# the repo returns no rows for it.
email = _RecordingEmail()
orchestrator = ScenarioExportOrchestrator(
rows=_FakeRows({}),
scenarios=_FakeScenarioNames({5: "Fabric first"}),
exports=exports,
email=email,
url_ttl_seconds=3600,
)
# act / assert — a recorded failure, never an empty workbook emailed.
with pytest.raises(SubTaskFailure):
orchestrator.run(
portfolio_id=100,
scenario_ids=[5],
property_ids=[1],
recipient_email="user@example.com",
export_name="portfolio-100",
)
assert email.sent == []
def test_default_plan_selection_builds_one_sheet_from_each_homes_default_plan(
exports: S3Client,
) -> None:
# arrange — plan_selection="default" ignores scenario_ids and reads each
# home's default Plan (ADR-0065). The fake returns two default rows and
# would raise if `rows_for` (the scenario path) were called instead.
orchestrator = ScenarioExportOrchestrator(
rows=_FakeRows({}, default_rows=[_row(1), _row(2)]),
scenarios=_FakeScenarioNames({}),
exports=exports,
email=_RecordingEmail(),
url_ttl_seconds=3600,
)
# act — no scenario_ids needed for the default-plan selection.
result = orchestrator.run(
portfolio_id=100,
scenario_ids=[],
property_ids=[1, 2],
recipient_email="user@example.com",
export_name="portfolio-100-default",
plan_selection="default",
)
# assert — exactly one sheet, titled "Default Plans", with both homes' rows.
assert result.sheet_count == 1
assert result.row_count == 2
workbook = load_workbook(BytesIO(exports.get_object(result.export_s3_key)))
assert workbook.sheetnames == ["Default Plans"]
def test_default_selection_with_no_default_plans_is_a_recorded_failure(
exports: S3Client,
) -> None:
# arrange — no home has a default Plan.
orchestrator = ScenarioExportOrchestrator(
rows=_FakeRows({}, default_rows=[]),
scenarios=_FakeScenarioNames({}),
exports=exports,
email=_RecordingEmail(),
url_ttl_seconds=3600,
)
# act / assert — a selection that yields no rows is a failure, never an
# empty workbook emailed.
with pytest.raises(SubTaskFailure):
orchestrator.run(
portfolio_id=100,
scenario_ids=[],
property_ids=[1, 2],
recipient_email="user@example.com",
export_name="portfolio-100-default",
plan_selection="default",
)

0
tests/repositories/scenario_export/__init__.py Normal file
View file

724
tests/repositories/scenario_export/test_scenario_export_postgres_repository.py Normal file
View file

@ -0,0 +1,724 @@
"""Behaviour of the Postgres-backed ScenarioExportRepository (ADR-0065): reading
one Scenario's Properties — those with a default Plan under it — as the export
read-model, from persisted data only (no live EPC calls)."""
from __future__ import annotations
from typing import Optional, Union
from sqlalchemy import Engine
from sqlmodel import Session
from datatypes.epc.domain.epc import Epc
from domain.modelling.portfolio_goal import PortfolioGoal
from infrastructure.postgres.epc_property_table import ( # registers epc tables
EpcBuildingPartModel,
EpcEnergyElementModel,
EpcMainHeatingDetailModel,
EpcPropertyEnergyPerformanceModel,
EpcPropertyModel,
)
from infrastructure.postgres.modelling import ( # registers `plan` + `recommendation`
PlanModel,
RecommendationModel,
ScenarioModel,
)
from infrastructure.postgres.product_table import MaterialRow # registers `material`
from infrastructure.postgres.property_override_table import ( # registers overrides
PropertyOverrideRow,
)
from infrastructure.postgres.property_table import PropertyRow # registers `property`
from repositories.scenario_export.scenario_export_postgres_repository import (
ScenarioExportPostgresRepository,
)
def _add_epc(
session: Session,
*,
property_id: int,
epc_id: int,
uprn: int = 100001,
source: str = "lodged",
property_type: Optional[str] = None,
built_form: Optional[str] = None,
total_floor_area_m2: float = 90.0,
registration_date: str = "2005-01-01",
habitable_rooms_count: int = 4,
energy_rating_current: Optional[int] = 58,
band: Optional[str] = "D",
elements: Optional[dict[str, str]] = None,
) -> None:
"""Seed one EPC slot (lodged/predicted) with its energy-performance child and
descriptive energy elements. Fills every NOT-NULL column with a benign
default so the test only has to state the fields it cares about."""
session.add(
EpcPropertyModel(
id=epc_id,
property_id=property_id,
uprn=uprn,
source=source,
property_type=property_type,
built_form=built_form,
total_floor_area_m2=total_floor_area_m2,
registration_date=registration_date,
habitable_rooms_count=habitable_rooms_count,
dwelling_type="House",
tenure="owner-occupied",
transaction_type="none",
inspection_date=registration_date,
solar_water_heating=False,
has_hot_water_cylinder=False,
has_fixed_air_conditioning=False,
door_count=1,
wet_rooms_count=1,
extensions_count=0,
heated_rooms_count=1,
open_chimneys_count=0,
insulated_door_count=0,
cfl_fixed_lighting_bulbs_count=0,
led_fixed_lighting_bulbs_count=0,
incandescent_fixed_lighting_bulbs_count=0,
energy_gas_connection_available=False,
energy_meter_type="standard",
energy_pv_battery_count=0,
energy_wind_turbines_count=0,
energy_gas_smart_meter_present=False,
energy_is_dwelling_export_capable=False,
energy_wind_turbines_terrain_type="none",
energy_electricity_smart_meter_present=False,
)
)
session.flush() # parent epc_property before its FK children
session.add(
EpcPropertyEnergyPerformanceModel(
epc_property_id=epc_id,
energy_rating_current=energy_rating_current,
current_energy_efficiency_band=band,
)
)
for element_type, description in (elements or {}).items():
session.add(
EpcEnergyElementModel(
epc_property_id=epc_id,
element_type=element_type,
description=description,
energy_efficiency_rating=3,
environmental_efficiency_rating=3,
)
)
def _add_building_part(
session: Session,
*,
epc_id: int,
identifier: str = "main",
building_part_number: Optional[int] = None,
wall_construction: int = 4,
wall_insulation_type: int = 2,
roof_construction_type: Optional[str] = "Pitched roof, Access to loft",
roof_insulation_location: Optional[str] = "Joists",
roof_insulation_thickness: Optional[Union[str, int]] = 270,
floor_type: Optional[str] = "Ground floor",
floor_construction_type: Optional[str] = "Solid",
floor_insulation_type_str: Optional[str] = "As Built",
) -> None:
"""Seed one SAP building part on an EPC slot (the source of the structured
walls / roof / floor text)."""
session.add(
EpcBuildingPartModel(
epc_property_id=epc_id,
identifier=identifier,
construction_age_band="D",
wall_construction=wall_construction,
wall_insulation_type=wall_insulation_type,
wall_thickness_measured=False,
building_part_number=building_part_number,
roof_construction_type=roof_construction_type,
roof_insulation_location=roof_insulation_location,
roof_insulation_thickness=roof_insulation_thickness,
floor_type=floor_type,
floor_construction_type=floor_construction_type,
floor_insulation_type_str=floor_insulation_type_str,
)
)
def _add_main_heating(
session: Session,
*,
epc_id: int,
main_fuel_type: int = 26,
heat_emitter_type: int = 1,
main_heating_control: int = 2106,
condensing: Optional[bool] = True,
) -> None:
"""Seed the main heating detail on an EPC slot (the source of the structured
heating text)."""
session.add(
EpcMainHeatingDetailModel(
epc_property_id=epc_id,
has_fghrs=False,
main_fuel_type=main_fuel_type,
heat_emitter_type=heat_emitter_type,
emitter_temperature=1,
main_heating_control=main_heating_control,
condensing=condensing,
)
)
def test_returns_a_row_for_a_property_with_a_default_plan_under_the_scenario(
db_engine: Engine,
) -> None:
# arrange — a Property with a default Plan under scenario 5 carrying the SAP
# calculator's post-works figures.
with Session(db_engine) as session:
session.add(
PropertyRow(
id=1,
portfolio_id=100,
landlord_property_id="LP1",
uprn=100023,
address="12 Oak Street",
postcode="AB1 2CD",
)
)
session.add(
PlanModel(
portfolio_id=100,
property_id=1,
scenario_id=5,
is_default=True,
post_sap_points=78.0,
post_epc_rating=Epc.C,
cost_of_works=4200.0,
)
)
session.commit()
# act
with Session(db_engine) as session:
rows = ScenarioExportPostgresRepository(session).rows_for(
portfolio_id=100, scenario_id=5, property_ids=[1]
)
# assert — one read-model row carrying identity and the Plan's post-works
# figures.
assert len(rows) == 1
row = rows[0]
assert row.property_id == 1
assert row.landlord_property_id == "LP1"
assert row.uprn == "100023"
assert row.post_sap_points == 78.0
assert row.cost_of_works == 4200.0
def test_excludes_requested_properties_without_a_plan_under_the_scenario(
db_engine: Engine,
) -> None:
# arrange — four requested Properties: #1 and #2 both have a Plan under
# scenario 5 (#2's is non-default — is_default no longer gates, since it is
# one-per-property not per-scenario), #3 a Plan under a different scenario,
# #4 no Plan at all.
with Session(db_engine) as session:
for pid in (1, 2, 3, 4):
session.add(
PropertyRow(id=pid, portfolio_id=100, landlord_property_id=f"LP{pid}")
)
session.add(
PlanModel(portfolio_id=100, property_id=1, scenario_id=5, is_default=True)
)
session.add(
PlanModel(portfolio_id=100, property_id=2, scenario_id=5, is_default=False)
)
session.add(
PlanModel(portfolio_id=100, property_id=3, scenario_id=9, is_default=True)
)
session.commit()
# act — request all four.
with Session(db_engine) as session:
rows = ScenarioExportPostgresRepository(session).rows_for(
portfolio_id=100, scenario_id=5, property_ids=[1, 2, 3, 4]
)
# assert — #1 and #2 (both have a Plan under scenario 5) survive; #3 (wrong
# scenario) and #4 (no Plan) drop out.
assert [r.property_id for r in rows] == [1, 2]
def test_selects_the_newest_plan_per_property_under_the_scenario(
db_engine: Engine,
) -> None:
# arrange — a Property with an older and a newer Plan under scenario 5; the
# newest Plan wins regardless of is_default (a re-model appends a new Plan).
with Session(db_engine) as session:
session.add(
PropertyRow(id=1, portfolio_id=100, landlord_property_id="LP1", uprn=100001)
)
session.add(
PlanModel(
id=1, portfolio_id=100, property_id=1, scenario_id=5,
is_default=True, post_sap_points=60.0,
)
)
session.add(
PlanModel(
id=2, portfolio_id=100, property_id=1, scenario_id=5,
is_default=False, post_sap_points=72.0,
)
)
session.commit()
# act
with Session(db_engine) as session:
rows = ScenarioExportPostgresRepository(session).rows_for(
portfolio_id=100, scenario_id=5, property_ids=[1]
)
# assert — the newer Plan (id=2, non-default) wins over the older default.
assert len(rows) == 1
assert rows[0].post_sap_points == 72.0
def test_maps_the_default_plans_selected_measures_with_battery_and_exclusions(
db_engine: Engine,
) -> None:
# arrange — a default Plan with two selected measures (one solar_pv whose
# material carries a battery), plus a non-default and an already-installed
# recommendation that must be excluded.
with Session(db_engine) as session:
session.add(
PropertyRow(id=1, portfolio_id=100, landlord_property_id="LP1", uprn=100001)
)
session.add(
PlanModel(
id=10, portfolio_id=100, property_id=1, scenario_id=5, is_default=True
)
)
session.add(MaterialRow(id=1, type="loft_insulation", includes_battery=False))
session.add(MaterialRow(id=2, type="solar_pv", includes_battery=True))
session.add(
RecommendationModel(
plan_id=10,
property_id=1,
type="loft_insulation",
measure_type="loft_insulation",
description="Loft insulation",
estimated_cost=1200.0,
sap_points=3.0,
co2_equivalent_savings=0.4,
kwh_savings=900.0,
energy_cost_savings=120.0,
material_id=1,
default=True,
already_installed=False,
)
)
session.add(
RecommendationModel(
plan_id=10,
property_id=1,
type="solar_pv",
measure_type="solar_pv",
description="Solar PV with battery",
estimated_cost=6000.0,
sap_points=5.0,
material_id=2,
default=True,
already_installed=False,
)
)
session.add(
RecommendationModel(
plan_id=10,
property_id=1,
type="double_glazing",
measure_type="double_glazing",
description="Double glazing (not selected)",
estimated_cost=4000.0,
default=False,
already_installed=False,
)
)
session.add(
RecommendationModel(
plan_id=10,
property_id=1,
type="cavity_wall_insulation",
measure_type="cavity_wall_insulation",
description="CWI (already installed)",
estimated_cost=800.0,
default=True,
already_installed=True,
)
)
session.commit()
# act
with Session(db_engine) as session:
rows = ScenarioExportPostgresRepository(session).rows_for(
portfolio_id=100, scenario_id=5, property_ids=[1]
)
# assert — only the two selected measures map through, the battery flag is
# resolved from the material, and savings are carried per measure.
by_type = {m.measure_type: m for m in rows[0].measures}
assert set(by_type) == {"loft_insulation", "solar_pv"}
assert by_type["loft_insulation"].estimated_cost == 1200.0
assert by_type["loft_insulation"].kwh_savings == 900.0
assert by_type["loft_insulation"].includes_battery is False
assert by_type["solar_pv"].includes_battery is True
def test_effective_epc_descriptive_fields_come_from_the_lodged_epc(
db_engine: Engine,
) -> None:
# arrange — a Property with a default Plan under scenario 5 and a lodged EPC
# carrying descriptive energy elements and current performance.
with Session(db_engine) as session:
session.add(
PropertyRow(id=1, portfolio_id=100, landlord_property_id="LP1", uprn=100001)
)
session.add(
PlanModel(portfolio_id=100, property_id=1, scenario_id=5, is_default=True)
)
_add_epc(
session,
property_id=1,
epc_id=1,
source="lodged",
property_type="House",
total_floor_area_m2=92.5,
registration_date="2005-01-01",
habitable_rooms_count=5,
energy_rating_current=58,
band="D",
elements={
"wall": "Cavity wall, as built, insulated",
"roof": "Pitched, 250mm loft insulation",
"main_heating": "Boiler and radiators, mains gas",
"window": "Fully double glazed",
},
)
session.commit()
# act
with Session(db_engine) as session:
rows = ScenarioExportPostgresRepository(session).rows_for(
portfolio_id=100, scenario_id=5, property_ids=[1]
)
# assert — descriptive prose, current performance, floor area, lodgement and
# property_type all come from the persisted lodged EPC (no live calls).
row = rows[0]
assert row.walls == "Cavity wall, as built, insulated"
assert row.roof == "Pitched, 250mm loft insulation"
assert row.heating == "Boiler and radiators, mains gas"
assert row.windows == "Fully double glazed"
assert row.property_type == "House"
assert row.total_floor_area == 92.5
assert row.number_of_rooms == 5
assert row.current_epc_rating == "D"
assert row.current_sap_points == 58
assert row.lodgement_date == "2005-01-01"
assert row.is_expired is True
def test_header_picks_the_freshest_cert_by_date_not_by_row_id(
db_engine: Engine,
) -> None:
# arrange — two lodged certs on one UPRN: a fresh 2026 survey lodged with a
# LOWER epc_id, and a stale 2013 gov cert re-ingested afterwards with a
# HIGHER epc_id. An `e.id DESC` sort would wrongly pick the stale gov cert;
# the freshest lodgement date must win (the portfolio-838 TFA regression).
with Session(db_engine) as session:
session.add(
PropertyRow(id=1, portfolio_id=100, landlord_property_id="LP1", uprn=100001)
)
session.add(
PlanModel(portfolio_id=100, property_id=1, scenario_id=5, is_default=True)
)
_add_epc(
session,
property_id=1,
epc_id=10, # fresh survey, LOWER id
uprn=100001,
total_floor_area_m2=76.62,
registration_date="2026-05-18",
)
_add_epc(
session,
property_id=1,
epc_id=20, # stale gov cert re-ingested later, HIGHER id
uprn=100001,
total_floor_area_m2=77.0,
registration_date="2013-10-09",
)
session.commit()
# act
with Session(db_engine) as session:
rows = ScenarioExportPostgresRepository(session).rows_for(
portfolio_id=100, scenario_id=5, property_ids=[1]
)
# assert — the fresh 2026 survey's floor area and lodgement win, not the
# higher-id 2013 gov cert.
assert rows[0].total_floor_area == 76.62
assert rows[0].lodgement_date == "2026-05-18"
def test_structured_fabric_supplies_walls_roof_floor_heating_and_hot_water(
db_engine: Engine,
) -> None:
# arrange — a lodged EPC carrying gov prose for every descriptive field AND
# the structured SAP fabric (building part + heating detail) a re-survey
# lodges. The structured fabric must win for walls/roof/floor/heating/
# hot_water; windows/lighting have no structured source and keep the prose.
with Session(db_engine) as session:
session.add(
PropertyRow(id=1, portfolio_id=100, landlord_property_id="LP1", uprn=100001)
)
session.add(
PlanModel(portfolio_id=100, property_id=1, scenario_id=5, is_default=True)
)
_add_epc(
session,
property_id=1,
epc_id=1,
source="lodged",
built_form="Mid-terrace",
elements={
"wall": "PROSE walls",
"roof": "PROSE roof",
"floor": "PROSE floor",
"main_heating": "PROSE heating",
"hot_water": "PROSE hot water",
"window": "Fully double glazed",
"lighting": "Low energy lighting in all fixed outlets",
},
)
_add_building_part(session, epc_id=1)
_add_main_heating(session, epc_id=1)
session.commit()
# act
with Session(db_engine) as session:
rows = ScenarioExportPostgresRepository(session).rows_for(
portfolio_id=100, scenario_id=5, property_ids=[1]
)
# assert — structured fabric replaces the prose for its five fields; windows
# and lighting (no structured source) keep the gov prose.
row = rows[0]
assert row.built_form == "Mid-terrace"
assert row.property_age_band == "1950-1966 (band D)"
assert row.walls == "Cavity wall, filled cavity"
assert row.roof == "Pitched roof, Access to loft; 270mm insulation at joists"
assert row.floor == "Ground floor, solid, insulation as built"
assert row.heating == (
"Mains gas, radiators (condensing); programmer, room thermostat and TRVs"
)
assert row.hot_water == "From main system, mains gas"
assert row.windows == "Fully double glazed"
assert row.lighting == "Low energy lighting in all fixed outlets"
def test_structured_fabric_reads_the_main_building_part_over_an_extension(
db_engine: Engine,
) -> None:
# arrange — two building parts on the cert: an extension (solid brick) and
# the main dwelling (cavity). The main part must supply the wall text.
with Session(db_engine) as session:
session.add(
PropertyRow(id=1, portfolio_id=100, landlord_property_id="LP1", uprn=100001)
)
session.add(
PlanModel(portfolio_id=100, property_id=1, scenario_id=5, is_default=True)
)
_add_epc(session, property_id=1, epc_id=1, source="lodged")
_add_building_part(
session,
epc_id=1,
identifier="extension_1",
building_part_number=2,
wall_construction=3, # solid brick — the extension
)
_add_building_part(
session,
epc_id=1,
identifier="main",
building_part_number=1,
wall_construction=4, # cavity — the main dwelling
)
session.commit()
# act
with Session(db_engine) as session:
rows = ScenarioExportPostgresRepository(session).rows_for(
portfolio_id=100, scenario_id=5, property_ids=[1]
)
# assert — the main (cavity) part wins, not the extension (solid brick).
assert rows[0].walls == "Cavity wall, filled cavity"
def test_prose_remains_where_the_structured_fabric_is_absent(
db_engine: Engine,
) -> None:
# arrange — a lodged EPC with gov prose but no building part and no heating
# detail (an older gov cert, not a re-survey).
with Session(db_engine) as session:
session.add(
PropertyRow(id=1, portfolio_id=100, landlord_property_id="LP1", uprn=100001)
)
session.add(
PlanModel(portfolio_id=100, property_id=1, scenario_id=5, is_default=True)
)
_add_epc(
session,
property_id=1,
epc_id=1,
source="lodged",
elements={"wall": "Cavity wall, as built", "roof": "Pitched, 250mm"},
)
session.commit()
# act
with Session(db_engine) as session:
rows = ScenarioExportPostgresRepository(session).rows_for(
portfolio_id=100, scenario_id=5, property_ids=[1]
)
# assert — with no structured fabric, the gov prose is the fallback.
row = rows[0]
assert row.walls == "Cavity wall, as built"
assert row.roof == "Pitched, 250mm"
def test_property_type_override_beats_the_lodged_epc(db_engine: Engine) -> None:
# arrange — the lodged EPC says Flat, but a landlord override says House
# (Effective-EPC precedence: override → lodged, ADR-0065).
with Session(db_engine) as session:
session.add(
PropertyRow(id=1, portfolio_id=100, landlord_property_id="LP1", uprn=100001)
)
session.add(
PlanModel(portfolio_id=100, property_id=1, scenario_id=5, is_default=True)
)
_add_epc(session, property_id=1, epc_id=1, source="lodged", property_type="Flat")
session.add(
PropertyOverrideRow(
property_id=1,
portfolio_id=100,
building_part=0,
override_component="property_type",
override_value="House",
original_spreadsheet_description="detached house",
)
)
session.commit()
# act
with Session(db_engine) as session:
rows = ScenarioExportPostgresRepository(session).rows_for(
portfolio_id=100, scenario_id=5, property_ids=[1]
)
# assert — the override wins.
assert rows[0].property_type == "House"
def test_falls_back_to_the_predicted_epc_when_there_is_no_lodged_epc(
db_engine: Engine,
) -> None:
# arrange — the Property has only a predicted EPC slot, no lodged cert.
with Session(db_engine) as session:
session.add(
PropertyRow(id=1, portfolio_id=100, landlord_property_id="LP1", uprn=100001)
)
session.add(
PlanModel(portfolio_id=100, property_id=1, scenario_id=5, is_default=True)
)
_add_epc(
session,
property_id=1,
epc_id=1,
source="predicted",
property_type="Bungalow",
elements={"wall": "Predicted solid brick wall"},
)
session.commit()
# act
with Session(db_engine) as session:
rows = ScenarioExportPostgresRepository(session).rows_for(
portfolio_id=100, scenario_id=5, property_ids=[1]
)
# assert — the predicted slot supplies the descriptive picture (precedence
# falls through to predicted when no lodged EPC exists).
row = rows[0]
assert row.property_type == "Bungalow"
assert row.walls == "Predicted solid brick wall"
def test_default_rows_for_selects_each_homes_default_plan_across_scenarios(
db_engine: Engine,
) -> None:
# arrange — two homes, each with its default Plan under a DIFFERENT scenario
# (property 1 → scenario 5, property 2 → scenario 9), plus a non-default Plan
# on property 1 under a third scenario. The default-plan selection is
# scenario-independent (ADR-0012 / ADR-0017): it must return exactly the two
# `is_default=True` Plans, and `plan_name` carries the SOURCE SCENARIO's name
# (not the freeform `plan.name`, which is unset on modelling-generated Plans).
with Session(db_engine) as session:
session.add(ScenarioModel(id=5, portfolio_id=100, name="Fabric Only",
goal=PortfolioGoal.INCREASING_EPC, goal_value="C"))
session.add(ScenarioModel(id=7, portfolio_id=100, name="Low Carbon",
goal=PortfolioGoal.INCREASING_EPC, goal_value="C"))
session.add(ScenarioModel(id=9, portfolio_id=100, name="ASHP + PV",
goal=PortfolioGoal.INCREASING_EPC, goal_value="C"))
session.add(
PropertyRow(id=1, portfolio_id=100, landlord_property_id="LP1", uprn=1)
)
session.add(
PropertyRow(id=2, portfolio_id=100, landlord_property_id="LP2", uprn=2)
)
session.add(
PlanModel(
portfolio_id=100, property_id=1, scenario_id=5, is_default=True,
post_sap_points=78.0,
)
)
session.add(
PlanModel(
portfolio_id=100, property_id=1, scenario_id=7, is_default=False,
post_sap_points=81.0,
)
)
session.add(
PlanModel(
portfolio_id=100, property_id=2, scenario_id=9, is_default=True,
post_sap_points=69.0,
)
)
session.commit()
# act — no scenario_id.
with Session(db_engine) as session:
rows = ScenarioExportPostgresRepository(session).default_rows_for(
portfolio_id=100, property_ids=[1, 2]
)
# assert — one row per home, the default Plan, with its source scenario name;
# the non-default plan under "Low Carbon" (81.0) never appears.
by_property = {r.property_id: r for r in rows}
assert set(by_property) == {1, 2}
assert by_property[1].plan_name == "Fabric Only"
assert by_property[1].post_sap_points == 78.0
assert by_property[2].plan_name == "ASHP + PV"
assert by_property[2].post_sap_points == 69.0