Merge pull request #1544 from Hestia-Homes/feat/calico-stock-condition-ingestion

Calico roof-covering stock-condition ingestion (ADR-0064)
This commit is contained in:
KhalimCK 2026-07-24 14:31:54 +01:00 committed by GitHub
commit 206ac197ee
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
81 changed files with 1150 additions and 172 deletions

View file

@ -143,6 +143,18 @@ _Avoid_: energy assessment, site survey, field survey, Domna survey, Hestia surv
Property data supplied by a landlord that may correct or supplement the public EPC for a single Property; triggers Rebaselining when applied; not applicable when Site Notes are present.
_Avoid_: patches (deprecated), corrections, manual EPC, edits
**Stock Condition Data**:
Landlord-supplied per-**component** lifecycle records — one row per element (e.g. a dwelling's **roof covering**) carrying its material/type, fitted-or-renewed date, quantity, and a **Remaining Life**. It answers *"when is this component due for renewal?"*, not *"what is the dwelling's SAP picture?"* — so unlike **Site Notes** and **Landlord Overrides** it is **not read by the modelling engine** and never enters `EpcPropertyData`, Rebaselining, or a Recommendation (ADR-0064). Sourced from a landlord's stock-condition survey (asset-management data); ingested by this repo purely as a **producer** and persisted keyed by UPRN as `PropertyConditionSurvey → Element → AspectCondition`. The consumer is the Next.js/Drizzle front-end, which reads the tables directly (the DB schema is the contract — the `element_type` / `aspect_type` PG enums are an FE-owned surface, cf. [[main-heating-system-pgenum-is-fe-owned]]) to display component data against a Property and to raise the **Roof Replacement Flag**. **Remaining Life** derives from the component's **Decent Homes Standard** expected lifetime (`fitted date + standard lifespan report date`), so it is trusted as given, not recomputed here.
_Avoid_: condition survey (ambiguous with **Site Notes**), asset data, patches; treating it as a **Landlord Override** or fabric signal (the engine never reads it — this iteration)
**Component Renewal Year**:
The absolute calendar year a **Stock Condition Data** component is due for replacement — stored as `renewal_year` on the `AspectCondition`. Computed at load as **`export_year + Remaining Life`** (the export/report date the landlord's file was generated, not `now()`), so it is stable across re-runs — an absolute year does not drift, whereas a stored "years remaining" would. The fitted/renewed date is retained separately as `install_date`.
_Avoid_: remaining life (that is the input; the stored value is an absolute year), replacement date, end-of-life date
**Roof Replacement Flag**:
A **front-end-owned** presentation signal raised on a **Solar PV Recommendation** when a Property's roof-covering **Component Renewal Year** falls within (or near) the measure horizon — warning a human that installing solar onto a soon-to-be-replaced roof is unwise. It is **not** computed in this repo and does **not** change what the engine recommends (the optimiser's solar Option is unchanged): the Next.js/Drizzle app joins **Stock Condition Data** to the recommendation by UPRN and applies the "due soon" threshold itself (ADR-0064). A future engine upgrade that *acts* on this (bundling a re-roof, deferring solar) is explicitly out of scope for this iteration.
_Avoid_: roof warning (unspecific), re-roof recommendation (there is no engine measure yet), solar suppression
**Landlord-Description Classification**:
Resolving a **Landlord Description** (unbounded free-text a landlord supplies for one component — "CWI" / "Cav filled" / "cavity insulated" all name one thing) onto a **Recognised Internal Description** via an LLM classifier, persisted in the `landlord_*_overrides` table (`source=classifier`) as a reviewed cache. Four vocabularies are kept **distinct** and must not be conflated: a **Landlord Description** (unbounded input); a **Recognised Internal Description** (the closed target taxonomy — e.g. a `MainHeatingSystemType` archetype — each binding to a Simulation Overlay); a **Lodged Description** (the gov-EPC `main_heating[].description` rendering, e.g. "Room heaters, electric" — only an example of which system *types* occur, never a map key); and the **SAP main heating code** (Table 4a/4b, what the calculator consumes). The classifier maps Landlord → Recognised Internal → SAP code. When it cannot confidently place the text it emits **`None`** (no overlay → the lodged EPC stands, surfaced to the user as "no suitable match"), **never the nearest wrong archetype** — the target taxonomy must be complete enough that a real system always has a correct home, so the classifier never overflows into a garbage-drawer archetype (ADR-0041).

View file

@ -20,22 +20,26 @@ COPY backend/.env.test backend/.env
# -----------------------------
# Copy requirements FIRST (for Docker layer caching)
# -----------------------------
COPY backend/condition/handler/requirements.txt .
COPY applications/condition/requirements.txt .
# Install dependencies into Lambda runtime
RUN pip install --no-cache-dir -r requirements.txt
# -----------------------------
# Copy application code
# Copy application code (the handler's import closure — DDD layout)
# -----------------------------
COPY utils/ utils/
COPY backend/condition/ backend/condition/
COPY domain/ domain/
COPY infrastructure/ infrastructure/
COPY repositories/ repositories/
COPY applications/ applications/
COPY backend/app/db/models/condition.py backend/app/db/models/condition.py
# Legacy backend bridges still imported by the condition module:
# ConditionPostgres -> backend.app.db.connection (db_session);
# condition_tables -> backend.app.db.base (Base); connection -> backend.app.config.
COPY backend/app/config.py backend/app/config.py
COPY backend/app/db/base.py backend/app/db/base.py
COPY backend/app/db/connection.py backend/app/db/connection.py
COPY backend/app/config.py backend/app/config.py
COPY backend/__init__.py backend/__init__.py
COPY backend/app/__init__.py backend/app/__init__.py
COPY backend/app/db/__init__.py backend/app/db/__init__.py
@ -44,6 +48,6 @@ COPY backend/app/db/__init__.py backend/app/db/__init__.py
# -----------------------------
# Lambda handler
# -----------------------------
CMD ["backend/condition/handler/handler.handler"]
CMD ["applications/condition/handler.handler"]
# For local running
# CMD ["python", "-m", "backend.condition.handler.handler"]
# CMD ["python", "-m", "applications.condition.handler"]

View file

@ -12,7 +12,31 @@ The Condition Data Processor performs the following steps:
- **Load**
- Persist transformed data into the ARA database (not yet implemented)
The processor currently supports file formats provided by **Peabody** and **LBWF**.
The processor currently supports file formats provided by **Peabody**, **LBWF**,
and **Calico**.
---
## Calico (roof-covering, one-off)
Calico is a one-off bulk load of stock-condition **roof-covering** data
(ADR-0064), run through `applications/condition/calico_runner.py`
(`run_calico_s3_load`), not the SQS Lambda. It is **producer-only** — written to
the `property_condition_survey` tables for the FE to read; the modelling engine
never reads it.
- **Source:** `s3://condition-data-dev/input/calico/Roof Covering 2.6.26.xlsx`
- **Portfolio:** 824 · **export year:** 2026 (anchors renewal years).
- **Mapping:** `"Roof Covering"``ROOF` / `MATERIAL` (the external covering
material — tiles/slate — not the lining or structure).
- **Drops (reconciled, never silent):** `"N/A"`/blank covering rows on parse
(1,239), references that don't resolve to a UPRN, and null-UPRN properties are
reported. A `LoadReport` summarises `loaded / unmatched / null-uprn / blank`.
- **⚠ Open — UPRN resolution:** Calico's `Asset Reference` does **not** match our
`landlord_property_id` for portfolio 824 (confirmed: zero overlap; addresses do
match). The Postgres `PropertyUprnLookup` cannot resolve Calico yet — it needs a
Calico-supplied `Asset Reference → UPRN` table (use the CSV `UprnLookup`) or
address matching. See the ADR-0064 update.
---

View file

@ -0,0 +1,51 @@
from typing import BinaryIO, List
from applications.condition.load_report import LoadReport
from domain.condition.mapping.calico.calico_mapper import CalicoMapper
from domain.condition.property_condition_survey import PropertyConditionSurvey
from infrastructure.condition.parsing.calico_parser import CalicoParser
from repositories.condition.condition_writer import ConditionWriter
from repositories.condition.property_uprn_lookup import PropertyUprnLookup
from repositories.condition.uprn_lookup import UprnLookup
from utils.logger import setup_logger
logger = setup_logger()
def run_calico_load(
file_stream: BinaryIO,
uprn_lookup: UprnLookup,
export_year: int,
writer: ConditionWriter,
) -> LoadReport:
"""Parse a Calico export, map each roof-covering observation to a condition
survey keyed by UPRN, persist them, and return the reconciliation report
(ADR-0064). ``export_year`` anchors renewal years so they don't drift.
Works with any ``UprnLookup``. A CSV/S3 lookup carries only resolved UPRNs,
so the ``null_uprn`` count is only meaningful for the property-table lookup
(which surfaces portfolio properties missing a UPRN); it is zero otherwise."""
parser = CalicoParser(uprn_lookup)
properties = parser.parse(file_stream)
mapper = CalicoMapper()
surveys: List[PropertyConditionSurvey] = [
mapper.map_asset_conditions_for_property(prop, export_year)
for prop in properties
]
writer.bulk_insert_surveys(surveys)
null_uprn_references = (
uprn_lookup.null_uprn_references()
if isinstance(uprn_lookup, PropertyUprnLookup)
else []
)
report = LoadReport.from_calico_load(
properties=properties,
blank_placeholder_count=parser.blank_placeholder_count,
unmatched_reference_count=parser.unmatched_reference_count,
null_uprn_references=null_uprn_references,
)
logger.info(f"[calico_load] {report}")
return report

View file

@ -0,0 +1,65 @@
"""One-off runner for the Calico stock-condition (roof-covering) load.
Calico is a bulk one-off import, not a recurring feed, so it runs through this
runner rather than the SQS-Lambda path Peabody/LBWF use (ADR-0064). The raw
export and the Asset-Reference->UPRN lookup both live in S3.
Not exercised by the unit suite -- it is S3 + Postgres wiring around the
unit-tested ``run_calico_load``; run it where a database and S3 are available.
Calico Asset References do not match our stored ``landlord_property_id``, so the
UPRN is resolved from an address-matched CSV lookup (built against portfolio 824
and uploaded to S3), not the property table. Pass ``dry_run=True`` to reconcile
without writing.
"""
from io import BytesIO
from typing import List
from applications.condition.calico_load import run_calico_load
from applications.condition.load_report import LoadReport
from domain.condition.property_condition_survey import PropertyConditionSurvey
from infrastructure.condition.lookups.uprn_lookup_s3 import UprnLookupS3
from repositories.condition.condition_postgres import ConditionPostgres
from utils.logger import setup_logger
from utils.s3 import read_io_from_s3
logger = setup_logger()
# The export date the Calico file was generated (Roof Covering 2.6.26); anchors
# renewal years so they don't drift on re-run (ADR-0064).
CALICO_EXPORT_YEAR = 2026
CALICO_BUCKET = "condition-data-dev"
CALICO_KEY = "input/calico/Roof Covering 2.6.26.xlsx"
CALICO_UPRN_LOOKUP_KEY = "input/calico/uprn-lookup/calico_reference_to_uprn.csv"
class _NoOpWriter:
"""Discards surveys -- used for a dry run so the load can be reconciled
without touching the database."""
def bulk_insert_surveys(
self, surveys: List[PropertyConditionSurvey]
) -> None:
logger.info(f"[calico_runner] dry run -- would persist {len(surveys)} surveys")
def run_calico_s3_load(
bucket: str = CALICO_BUCKET,
key: str = CALICO_KEY,
uprn_lookup_key: str = CALICO_UPRN_LOOKUP_KEY,
export_year: int = CALICO_EXPORT_YEAR,
dry_run: bool = False,
) -> LoadReport:
file_bytes: BytesIO = read_io_from_s3(bucket_name=bucket, file_key=key)
lookup = UprnLookupS3(bucket=bucket, key=uprn_lookup_key)
writer = _NoOpWriter() if dry_run else ConditionPostgres()
report = run_calico_load(
file_stream=file_bytes,
uprn_lookup=lookup,
export_year=export_year,
writer=writer,
)
logger.info(f"[calico_runner] {'(dry run) ' if dry_run else ''}{report}")
return report

View file

@ -6,6 +6,7 @@ from pydantic import BaseModel
class ConditionFileType(Enum):
LBWF = "LBWF"
Peabody = "Peabody"
Calico = "Calico"
# TODO: make these asset management systems rather than client names

View file

@ -0,0 +1,47 @@
from typing import Optional
from applications.condition.condition_trigger_request import ConditionFileType
from domain.condition.mapping.calico.calico_mapper import CalicoMapper
from domain.condition.mapping.lbwf.lbwf_mapper import LbwfMapper
from domain.condition.mapping.mapper import Mapper
from domain.condition.mapping.peabody.peabody_mapper import PeabodyMapper
from repositories.condition.uprn_lookup import UprnLookup
from infrastructure.condition.parsing.parser import Parser
from infrastructure.condition.parsing.calico_parser import CalicoParser
from infrastructure.condition.parsing.lbwf_parser import LbwfParser
from infrastructure.condition.parsing.peabody_parser import PeabodyParser
def select_parser(
file_type: ConditionFileType, uprn_lookup: Optional[UprnLookup] = None
) -> Parser:
if file_type is ConditionFileType.LBWF:
return LbwfParser()
if file_type is ConditionFileType.Peabody:
if not uprn_lookup:
raise ValueError(
"Cannot instantiate Peabody Parser without UPRN lookup being provided"
)
return PeabodyParser(uprn_lookup=uprn_lookup)
if file_type is ConditionFileType.Calico:
if not uprn_lookup:
raise ValueError(
"Cannot instantiate Calico Parser without UPRN lookup being provided"
)
return CalicoParser(uprn_lookup=uprn_lookup)
raise ValueError("Unrecognised file type, unable to instantiate Parser")
def select_mapper(file_type: ConditionFileType) -> Mapper:
if file_type is ConditionFileType.LBWF:
return LbwfMapper()
if file_type is ConditionFileType.Peabody:
return PeabodyMapper()
if file_type is ConditionFileType.Calico:
return CalicoMapper()
raise ValueError("Unrecognised file type, unable to instantiate Mapper")

View file

@ -2,9 +2,9 @@ import json
from typing import Mapping, Any
from io import BytesIO
from backend.condition.condition_trigger_request import ConditionTriggerRequest
from backend.condition.lookups.uprn_lookup_s3 import UprnLookupS3
from backend.condition.processor import process_file
from applications.condition.condition_trigger_request import ConditionTriggerRequest
from infrastructure.condition.lookups.uprn_lookup_s3 import UprnLookupS3
from applications.condition.processor import process_file
from utils.logger import setup_logger
from utils.s3 import read_io_from_s3

View file

@ -0,0 +1,39 @@
from dataclasses import dataclass
from typing import List
from domain.condition.records.calico.calico_property import CalicoProperty
@dataclass(frozen=True)
class LoadReport:
"""The reconciliation a Calico load emits so nothing vanishes silently
(ADR-0064): how many observations were loaded versus dropped, and why.
``loaded``, ``unmatched_to_portfolio`` and ``blank_placeholder`` account for
the Calico rows. ``null_uprn`` is a portfolio-wide data-quality canary the
count of properties in the portfolio whose UPRN is null (expected: zero)
*not* a per-row drop reason, so it does not partition with the others: a
Calico reference pointing at a null-UPRN property is reported both here and
under ``unmatched_to_portfolio``.
"""
loaded: int
unmatched_to_portfolio: int
null_uprn: int
blank_placeholder: int
@classmethod
def from_calico_load(
cls,
properties: List[CalicoProperty],
blank_placeholder_count: int,
unmatched_reference_count: int,
null_uprn_references: List[str],
) -> "LoadReport":
loaded = sum(len(prop.assets) for prop in properties)
return cls(
loaded=loaded,
unmatched_to_portfolio=unmatched_reference_count,
null_uprn=len(null_uprn_references),
blank_placeholder=blank_placeholder_count,
)

View file

@ -1,8 +1,8 @@
from pathlib import Path
from backend.condition.condition_trigger_request import ConditionFileType
from backend.condition.lookups.uprn_lookup_csv import UprnLookupLocal
from backend.condition.processor import process_file
from applications.condition.condition_trigger_request import ConditionFileType
from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal
from applications.condition.processor import process_file
def main() -> None:

View file

@ -1,14 +1,14 @@
from typing import Any, BinaryIO, List, Optional
from datetime import datetime
from backend.condition.condition_trigger_request import ConditionFileType
from backend.condition.lookups.uprn_lookup import UprnLookup
from applications.condition.condition_trigger_request import ConditionFileType
from repositories.condition.uprn_lookup import UprnLookup
from utils.logger import setup_logger
from backend.condition.domain.mapping.mapper import Mapper
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
from backend.condition.parsing.parser import Parser
from backend.condition.persistence.condition_postgres import ConditionPostgres
from backend.condition.parsing.factory import select_parser, select_mapper
from domain.condition.mapping.mapper import Mapper
from domain.condition.property_condition_survey import PropertyConditionSurvey
from infrastructure.condition.parsing.parser import Parser
from repositories.condition.condition_postgres import ConditionPostgres
from applications.condition.factory import select_parser, select_mapper
logger = setup_logger()

View file

@ -3,7 +3,7 @@ from sqlalchemy import insert, delete
from sqlalchemy.orm import Session
from backend.app.db.connection import db_session, db_read_session
from backend.app.db.models.condition import PropertyConditionSurveyModel
from infrastructure.postgres.condition_tables import PropertyConditionSurveyModel
def bulk_insert_property_surveys(

View file

@ -1,35 +0,0 @@
from typing import Optional
from backend.condition.condition_trigger_request import ConditionFileType
from backend.condition.domain.mapping.lbwf.lbwf_mapper import LbwfMapper
from backend.condition.domain.mapping.mapper import Mapper
from backend.condition.domain.mapping.peabody.peabody_mapper import PeabodyMapper
from backend.condition.lookups.uprn_lookup import UprnLookup
from backend.condition.parsing.parser import Parser
from backend.condition.parsing.lbwf_parser import LbwfParser
from backend.condition.parsing.peabody_parser import PeabodyParser
def select_parser(
file_type: ConditionFileType, uprn_lookup: Optional[UprnLookup] = None
) -> Parser:
if file_type is ConditionFileType.LBWF:
return LbwfParser()
if file_type is ConditionFileType.Peabody:
if not uprn_lookup:
raise ValueError(
"Cannot instantiate Peabody Parser without UPRN lookup being provided"
)
return PeabodyParser(uprn_lookup=uprn_lookup)
raise ValueError("Unrecognised file type, unable to instantiate Parser")
def select_mapper(file_type: ConditionFileType) -> Mapper:
if file_type is ConditionFileType.LBWF:
return LbwfMapper()
if file_type is ConditionFileType.Peabody:
return PeabodyMapper()
raise ValueError("Unrecognised file type, unable to instantiate Mapper")

View file

@ -1,30 +0,0 @@
import pytest
from backend.condition.condition_trigger_request import ConditionFileType
from backend.condition.lookups.uprn_lookup_csv import UprnLookupLocal
from backend.condition.parsing.factory import select_parser
def test_selects_lbwf_parser():
# arrange
file_type = ConditionFileType.LBWF
expected_class_name = "LbwfParser"
# act
actual_class_name = select_parser(file_type).__class__.__name__
# assert
assert expected_class_name == actual_class_name
def test_selects_peabody_parser():
# arrange
file_type = ConditionFileType.Peabody
expected_class_name = "PeabodyParser"
uprn_lookup = UprnLookupLocal(csv_path="test")
# act
actual_class_name = select_parser(file_type, uprn_lookup).__class__.__name__
# assert
assert expected_class_name == actual_class_name

View file

@ -0,0 +1,127 @@
---
status: accepted
---
# Stock condition data is a producer-only, engine-isolated dataset; the roof-replacement warning on solar is a front-end overlay
We are ingesting a landlord's (Calico) **stock-condition survey** — for now, one
component: **roof covering**, ~5,346 rows of `Asset Reference, Type (material),
Fitted/Renewed Date, Quantity, Remaining Life`. The temptation is to treat any
landlord-supplied component fact like a **Landlord Override** and fold it into the
`EpcPropertyData` picture. That would be wrong here, and the whole design turns on
saying why.
**Stock Condition Data answers a different question from the modelling picture.**
A **Landlord Override** ("this wall is solid brick") and **Site Notes** (a full
Domna survey) both correct *what the dwelling is* so the SAP score is right — the
engine reads them. Stock Condition Data answers *when a component is due for
renewal*: the roof-covering material plus a **Remaining Life** derived from the
component's **Decent Homes Standard** expected lifetime. It is asset-management
data, not a fabric signal. Conflating the two would drag component-lifecycle rows
into `EpcPropertyData`, Rebaselining, and the Recommendation model for no scoring
benefit and real coupling cost.
**The consumer is the product, not the engine.** The value is: (1) show component
data against a Property in the front-end, and (2) warn a human, on a **Solar PV
Recommendation**, when the roof is near end-of-life — because installing solar
onto a roof due for replacement is a poor sequence. That warning is a *human
information* signal, not an optimiser input: the engine still recommends solar
unchanged; a badge says "mind the roof". The cross-reference (roof renewal year ↔
solar recommendation) and the "due soon" threshold live in the **Next.js/Drizzle
front-end**, which reads the condition tables directly. This repo is purely a
**producer** into a schema the front-end consumes — so the DB schema (the
`property_condition_survey` / `element` / `aspect_condition` tables and their
`element_type` / `aspect_type` PG enums) is an **FE-owned contract**, the same
pattern as `main_heating_system`.
Decided in a grill-with-docs session with Khalim, 2026-07-13.
## Decision
**Stock Condition Data is ingested producer-only and is engine-isolated. The
modelling engine never reads it (this iteration); the roof-replacement warning on
solar is a front-end presentation overlay, not an engine output.**
- **Engine isolation.** Condition rows are written only to
`PropertyConditionSurvey → Element → AspectCondition`, keyed by UPRN. They never
enter `EpcPropertyData`, Rebaselining, Bill Derivation, or a Recommendation. The
optimiser's Solar PV Option is unchanged whether or not the roof is near
end-of-life. A future engine upgrade that *acts* on condition data (bundling a
re-roof, deferring solar) is a separate, later decision — explicitly out of
scope now.
- **The flag lives above modelling.** The **Roof Replacement Flag** is computed by
the Next.js/Drizzle app, which joins condition data to the solar recommendation
by UPRN and owns the "due soon" threshold. This repo neither computes the flag
nor exposes a read API for it — it only writes the tables the FE reads. The DB
schema is therefore a shared FE contract; schema changes need a companion
Drizzle view.
- **UPRN resolution from the property table, not a CSV.** Calico's `Asset
Reference` maps to `property.landlord_property_id` scoped to `portfolio_id =
824`, which yields the `uprn`. The lookup is a Postgres-backed `UprnLookup`
reading the property table — fulfilling the existing "replace CSV with postgres"
TODO — not an exported file. A key-alignment test (Asset Reference == stored
`landlord_property_id`) is pinned during TDD implementation.
> **Update (confirmed against the DB, 2026-07-13).** The key alignment is
> **false**: Calico Asset References (`438…14029`) have **zero** overlap with
> portfolio 824's stored `landlord_property_id` (`690548…`), though the
> *addresses* match — portfolio 824 **is** Calico's Burnley stock, keyed under a
> different scheme, and all 5,276 of its properties have a non-null UPRN. So the
> Postgres property-table lookup cannot resolve Calico. An exact address-line
> join is also weak (555 / 3,990). Resolution is **open pending a Calico-supplied
> `Asset Reference → UPRN` (or → `landlord_property_id`) table** — which reverts
> this to the CSV `UprnLookup` path — or a proper address-normalisation match.
> The `PropertyUprnLookup` code stands; it is simply not the right resolver for
> Calico until the key exists.
- **Drop-and-report reconciliation; nothing vanishes silently.** Rows that do not
resolve to a portfolio-824 UPRN are dropped and counted; rows whose resolved
`property.uprn` is null are dropped **and reported** (none expected — we want to
see any that are); the ~1,239 blank-`Type` placeholder rows (Quantity 0,
sentinel date, nonsense Remaining Life) are dropped on parse. Every load emits a
reconciliation report: `loaded / unmatched-to-portfolio / null-uprn /
blank-placeholder`.
- **Renewal year is anchored to the export date, not `now()`.** `Remaining Life`
is years-from-report; the stored **Component Renewal Year** = `export_year +
Remaining Life` (export date passed explicitly as a load parameter — Calico's
file is dated ≈ 2026-06-02), so the absolute renewal year is reproducible across
re-runs. The fitted/renewed date is retained as `install_date`. This replaces
the module's `survey_year = datetime.now().year` TODO for this loader.
- **Delivery: one-off runner load, from S3.** Calico is a bulk one-off import, run
through the local/orchestration runner (not the SQS-Lambda path Peabody and LBWF
use). The raw `.xlsx` is stored in S3 under a documented Calico key and kept out
of git (it carries real addresses).
## Consequences
- The condition-ingestion module moves out of legacy `backend/condition/` into the
DDD layout (domain / repositories / infrastructure / applications /
orchestration) as a full lift-and-shift — shared core, the Peabody and LBWF
adapters, and the delivery path — with Calico added as a new adapter alongside
them. Peabody/LBWF keep their Lambda after the move; the relocation is
behaviour-preserving for them. **The move is deliberately relocation-only:** the
ORM models stay SQLAlchemy-declarative and `ConditionPostgres` keeps the legacy
`backend.app.db` `Base` / `db_session` bridges, so the existing test suite proves
behaviour is unchanged. Converting the tables to SQLModel and the repository to
constructor session-injection (matching the rest of `infrastructure/postgres`),
and clearing the module's pre-existing pyright-strict debt, are tracked
follow-ups to run where a database is available to verify them — not bundled into
the move.
- A new `Calico` `ConditionFileType`, parser + row DTO, roof-covering element map
(`"Roof Covering"` → roof `ElementType`, `MATERIAL` aspect = `Type`), and mapper
are added and registered in the factory. Roof covering is already a modelled
component type in `ElementType`, so no enum change is needed for this load.
- CONTEXT.md gains **Stock Condition Data**, **Component Renewal Year**, and
**Roof Replacement Flag**, kept distinct from **Site Notes** and **Landlord
Overrides** so the ubiquitous language records that this source is *not* read by
the engine.
- Because the FE reads these tables through Drizzle, any later schema change here
(a new component, a new aspect) is a cross-repo contract change requiring a
companion Drizzle migration — the `main_heating_system` FE-ownership lesson
applies.
- If the "mind the roof" signal later needs to *drive* modelling, that is a new
ADR that deliberately re-opens engine isolation — not an incremental tweak.

View file

View file

@ -2,7 +2,7 @@ from dataclasses import dataclass
from typing import Optional
from datetime import date
from backend.condition.domain.aspect_type import AspectType
from domain.condition.aspect_type import AspectType
@dataclass

View file

@ -1,8 +1,8 @@
from dataclasses import dataclass
from typing import List
from backend.condition.domain.aspect_condition import AspectCondition
from backend.condition.domain.element_type import ElementType
from domain.condition.aspect_condition import AspectCondition
from domain.condition.element_type import ElementType
@dataclass

View file

View file

@ -0,0 +1,12 @@
from domain.condition.aspect_type import AspectType
from domain.condition.element_type import ElementType
from domain.condition.mapping.element_mapping import ElementMapping
# Calico's component path -> the shared taxonomy. Held as data (like the LBWF /
# Peabody element maps) so extending Calico to further components is a data edit.
CALICO_ELEMENT_MAP: dict[str, ElementMapping] = {
"Roof Covering": ElementMapping(
elementType=ElementType.ROOF,
aspect_type=AspectType.MATERIAL,
),
}

View file

@ -0,0 +1,56 @@
from datetime import date
from typing import Any, List, Optional
from domain.condition.aspect_condition import AspectCondition
from domain.condition.element import Element
from domain.condition.mapping.calico.calico_element_map import CALICO_ELEMENT_MAP
from domain.condition.mapping.mapper import Mapper
from domain.condition.property_condition_survey import PropertyConditionSurvey
from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition
from domain.condition.records.calico.calico_property import CalicoProperty
SOURCE = "Calico"
class CalicoMapper(Mapper):
def map_asset_conditions_for_property(
self, client_property_data: Any, survey_year: Optional[int] = None
) -> PropertyConditionSurvey:
assert isinstance(client_property_data, CalicoProperty)
elements: List[Element] = []
for instance, asset in enumerate(client_property_data.assets, start=1):
mapping = CALICO_ELEMENT_MAP[asset.path]
elements.append(
Element(
element_type=mapping.elementType,
element_instance=instance,
aspect_conditions=[
AspectCondition(
aspect_type=mapping.aspect_type,
aspect_instance=1,
value=asset.covering_type,
quantity=asset.quantity,
install_date=asset.fitted_date,
renewal_year=CalicoMapper._renewal_year(
asset, survey_year
),
)
],
)
)
return PropertyConditionSurvey(
uprn=client_property_data.uprn,
elements=elements,
date=date(survey_year, 1, 1) if survey_year else date(2000, 1, 1),
source=SOURCE,
)
@staticmethod
def _renewal_year(
asset: CalicoAssetCondition, survey_year: Optional[int]
) -> Optional[int]:
if asset.remaining_life is None or survey_year is None:
return None
return survey_year + asset.remaining_life

View file

@ -1,8 +1,8 @@
from dataclasses import dataclass
from typing import Optional
from backend.condition.domain.aspect_type import AspectType
from backend.condition.domain.element_type import ElementType
from domain.condition.aspect_type import AspectType
from domain.condition.element_type import ElementType
@dataclass(frozen=True)

View file

@ -1,6 +1,6 @@
from backend.condition.domain.element_type import ElementType
from backend.condition.domain.aspect_type import AspectType
from backend.condition.domain.mapping.element_mapping import ElementMapping
from domain.condition.element_type import ElementType
from domain.condition.aspect_type import AspectType
from domain.condition.mapping.element_mapping import ElementMapping
LBWF_ELEMENT_MAP: dict[str, ElementMapping] = {

View file

@ -1,17 +1,17 @@
from typing import Any, Dict, Optional, Tuple
from datetime import date
from backend.condition.domain.aspect_condition import AspectCondition
from backend.condition.domain.element import Element
from backend.condition.domain.element_type import ElementType
from backend.condition.domain.mapping.element_mapping import ElementMapping
from backend.condition.domain.mapping.lbwf.lbwf_element_map import LBWF_ELEMENT_MAP
from backend.condition.domain.mapping.mapper import Mapper
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
from backend.condition.parsing.records.lbwf.lbwf_asset_condition import (
from domain.condition.aspect_condition import AspectCondition
from domain.condition.element import Element
from domain.condition.element_type import ElementType
from domain.condition.mapping.element_mapping import ElementMapping
from domain.condition.mapping.lbwf.lbwf_element_map import LBWF_ELEMENT_MAP
from domain.condition.mapping.mapper import Mapper
from domain.condition.property_condition_survey import PropertyConditionSurvey
from domain.condition.records.lbwf.lbwf_asset_condition import (
LbwfAssetCondition,
)
from backend.condition.parsing.records.lbwf.lbwf_house import LbwfHouse
from domain.condition.records.lbwf.lbwf_house import LbwfHouse
from utils.logger import setup_logger
logger = setup_logger()

View file

@ -1,8 +1,8 @@
from abc import ABC, abstractmethod
from typing import Any, List, Optional
from backend.condition.domain.element import Element
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
from domain.condition.element import Element
from domain.condition.property_condition_survey import PropertyConditionSurvey
class Mapper(ABC):

View file

@ -1,6 +1,6 @@
from backend.condition.domain.aspect_type import AspectType
from backend.condition.domain.element_type import ElementType
from backend.condition.domain.mapping.element_mapping import ElementMapping
from domain.condition.aspect_type import AspectType
from domain.condition.element_type import ElementType
from domain.condition.mapping.element_mapping import ElementMapping
PEABODY_ELEMENT_MAP = {

View file

@ -1,19 +1,19 @@
from typing import Any, Dict, Optional, Tuple
from datetime import date
from backend.condition.domain.aspect_condition import AspectCondition
from backend.condition.domain.element import Element
from backend.condition.domain.element_type import ElementType
from backend.condition.domain.mapping.element_mapping import ElementMapping
from backend.condition.domain.mapping.peabody.peabody_element_map import (
from domain.condition.aspect_condition import AspectCondition
from domain.condition.element import Element
from domain.condition.element_type import ElementType
from domain.condition.mapping.element_mapping import ElementMapping
from domain.condition.mapping.peabody.peabody_element_map import (
PEABODY_ELEMENT_MAP,
)
from backend.condition.domain.mapping.mapper import Mapper
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
from backend.condition.parsing.records.peabody.peabody_asset_condition import (
from domain.condition.mapping.mapper import Mapper
from domain.condition.property_condition_survey import PropertyConditionSurvey
from domain.condition.records.peabody.peabody_asset_condition import (
PeabodyAssetCondition,
)
from backend.condition.parsing.records.peabody.peabody_property import PeabodyProperty
from domain.condition.records.peabody.peabody_property import PeabodyProperty
from utils.logger import setup_logger
logger = setup_logger()

View file

@ -2,7 +2,7 @@ from dataclasses import dataclass
from typing import List
from datetime import date
from backend.condition.domain.element import Element
from domain.condition.element import Element
@dataclass

View file

View file

@ -0,0 +1,20 @@
from dataclasses import dataclass
from datetime import date
from typing import Optional
@dataclass
class CalicoAssetCondition:
"""One component row from a Calico stock-condition export.
Calico's current export carries a single component (``path == "Roof
Covering"``); ``covering_type`` is the material (e.g. "Concrete Tiles") and
``remaining_life`` is Calico's Decent-Homes-derived years-remaining figure.
"""
asset_reference: int
path: str
covering_type: Optional[str] = None
fitted_date: Optional[date] = None
quantity: Optional[int] = None
remaining_life: Optional[int] = None

View file

@ -0,0 +1,10 @@
from dataclasses import dataclass
from typing import List
from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition
@dataclass
class CalicoProperty:
uprn: int
assets: List[CalicoAssetCondition]

View file

@ -1,7 +1,7 @@
from dataclasses import dataclass
from typing import List
from backend.condition.parsing.records.lbwf.lbwf_asset_condition import LbwfAssetCondition
from domain.condition.records.lbwf.lbwf_asset_condition import LbwfAssetCondition
@dataclass
class LbwfHouse:

View file

@ -1,7 +1,7 @@
from dataclasses import dataclass
from typing import List
from backend.condition.parsing.records.peabody.peabody_asset_condition import PeabodyAssetCondition
from domain.condition.records.peabody.peabody_asset_condition import PeabodyAssetCondition
@dataclass
class PeabodyProperty:

View file

View file

@ -1,7 +1,7 @@
import csv
from io import TextIOWrapper
from typing import BinaryIO, Dict, TextIO
from backend.condition.lookups.uprn_lookup import UprnLookup
from repositories.condition.uprn_lookup import UprnLookup
class UprnLookupLocal(UprnLookup):

View file

@ -2,7 +2,7 @@ import csv
from io import BytesIO, TextIOWrapper
from typing import BinaryIO, Dict, TextIO
from backend.condition.lookups.uprn_lookup import UprnLookup
from repositories.condition.uprn_lookup import UprnLookup
from utils.s3 import read_io_from_s3

View file

@ -0,0 +1,114 @@
from collections import defaultdict
from typing import Any, BinaryIO, Dict, List, Optional, Tuple
from openpyxl import load_workbook
from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition
from domain.condition.records.calico.calico_property import CalicoProperty
from infrastructure.condition.date_utils import normalise_date
from infrastructure.condition.parsing.parser import Parser
from repositories.condition.uprn_lookup import UprnLookup
from utils.logger import setup_logger
logger = setup_logger()
class CalicoParser(Parser):
def __init__(self, uprn_lookup: UprnLookup) -> None:
self._uprn_lookup = uprn_lookup
self._blank_placeholder_count = 0
self._unmatched_reference_count = 0
@property
def blank_placeholder_count(self) -> int:
"""Rows dropped on parse for a blank / N/A covering type."""
return self._blank_placeholder_count
@property
def unmatched_reference_count(self) -> int:
"""Distinct asset references dropped for not resolving to a portfolio UPRN."""
return self._unmatched_reference_count
def parse(self, file_stream: BinaryIO) -> List[CalicoProperty]:
workbook = load_workbook(file_stream, data_only=True)
sheet = workbook[workbook.sheetnames[0]]
rows = sheet.iter_rows(values_only=True)
column_index = CalicoParser._column_indexes(next(rows))
assets = self._parse_assets(rows, column_index)
return self._group_assets_into_properties(assets)
def _parse_assets(
self, rows: Any, column_index: Dict[str, int]
) -> List[CalicoAssetCondition]:
assets: List[CalicoAssetCondition] = []
for row in rows:
covering_type = CalicoParser._clean_str(row[column_index["Type"]])
if covering_type is None:
# A blank covering type is a placeholder row (no roof-covering
# record captured), not an observation -- drop on parse.
self._blank_placeholder_count += 1
continue
assets.append(
CalicoAssetCondition(
asset_reference=int(row[column_index["Asset Reference"]]),
path=CalicoParser._clean_str(row[column_index["Path"]]) or "",
covering_type=covering_type,
fitted_date=normalise_date(row[column_index["Fitted / Renewed Date"]]),
quantity=CalicoParser._to_int(row[column_index["Quantity"]]),
remaining_life=CalicoParser._to_int(
row[column_index["Remaining Life"]]
),
)
)
return assets
def _group_assets_into_properties(
self, assets: List[CalicoAssetCondition]
) -> List[CalicoProperty]:
ref_to_uprn: Dict[str, int] = self._uprn_lookup.get_property_ref_to_uprn_lookup()
assets_by_ref: Dict[int, List[CalicoAssetCondition]] = defaultdict(list)
for asset in assets:
assets_by_ref[asset.asset_reference].append(asset)
properties: List[CalicoProperty] = []
for reference, grouped_assets in assets_by_ref.items():
uprn = ref_to_uprn.get(str(reference))
if uprn is None:
self._unmatched_reference_count += 1
logger.debug(
f"[CalicoParser] No UPRN for asset reference {reference}; dropping"
)
continue
properties.append(CalicoProperty(uprn=uprn, assets=grouped_assets))
return properties
@staticmethod
def _column_indexes(headers: Tuple[object | None, ...]) -> Dict[str, int]:
return {
header: index
for index, header in enumerate(headers)
if isinstance(header, str)
}
# Sentinels Calico lodges in the Type column for a placeholder row (no
# roof-covering record captured), treated as "no covering".
_BLANK_COVERING_SENTINELS = {"", "n/a", "na", "none"}
@staticmethod
def _clean_str(value: object) -> Optional[str]:
if not isinstance(value, str):
return None
stripped = value.strip()
if stripped.lower() in CalicoParser._BLANK_COVERING_SENTINELS:
return None
return stripped
@staticmethod
def _to_int(value: object) -> Optional[int]:
if isinstance(value, (int, float)):
return int(value)
return None

View file

@ -2,12 +2,12 @@ from typing import BinaryIO, Any, Dict, Iterator, List, Optional, Tuple
from openpyxl import Workbook, load_workbook
from collections import defaultdict
from backend.condition.parsing.parser import Parser
from backend.condition.parsing.records.lbwf.lbwf_asset_condition import (
from infrastructure.condition.parsing.parser import Parser
from domain.condition.records.lbwf.lbwf_asset_condition import (
LbwfAssetCondition,
)
from backend.condition.parsing.records.lbwf.lbwf_house import LbwfHouse
from backend.condition.utils.date_utils import normalise_date
from domain.condition.records.lbwf.lbwf_house import LbwfHouse
from infrastructure.condition.date_utils import normalise_date
from utils.logger import setup_logger
logger = setup_logger()

View file

@ -4,12 +4,12 @@ from typing import Any, BinaryIO, Dict, List, Optional, Tuple, DefaultDict
from openpyxl import Workbook, load_workbook
from collections import defaultdict
from backend.condition.lookups.uprn_lookup import UprnLookup
from backend.condition.parsing.parser import Parser
from backend.condition.parsing.records.peabody.peabody_asset_condition import (
from repositories.condition.uprn_lookup import UprnLookup
from infrastructure.condition.parsing.parser import Parser
from domain.condition.records.peabody.peabody_asset_condition import (
PeabodyAssetCondition,
)
from backend.condition.parsing.records.peabody.peabody_property import PeabodyProperty
from domain.condition.records.peabody.peabody_property import PeabodyProperty
from utils.logger import setup_logger
logger = setup_logger()

View file

@ -9,8 +9,8 @@ from sqlalchemy import (
)
from sqlalchemy.orm import relationship
from backend.condition.domain.aspect_type import AspectType
from backend.condition.domain.element_type import ElementType
from domain.condition.aspect_type import AspectType
from domain.condition.element_type import ElementType
from backend.app.db.base import Base

View file

View file

@ -3,12 +3,12 @@ from typing import List, Optional
from sqlmodel import Session
from utils.logger import setup_logger
from backend.app.db.models.condition import (
from infrastructure.postgres.condition_tables import (
AspectConditionModel,
ElementModel,
PropertyConditionSurveyModel,
)
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
from domain.condition.property_condition_survey import PropertyConditionSurvey
from backend.app.db.connection import db_session
logger = setup_logger()

View file

@ -0,0 +1,13 @@
from typing import List, Protocol
from domain.condition.property_condition_survey import PropertyConditionSurvey
class ConditionWriter(Protocol):
"""Persists condition surveys. ``ConditionPostgres`` is the production
implementation; a load orchestrator depends on this port so it can be
exercised without a database."""
def bulk_insert_surveys(
self, surveys: List[PropertyConditionSurvey]
) -> None: ...

View file

@ -0,0 +1,19 @@
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import List, Optional
@dataclass(frozen=True)
class PropertyReference:
"""A property's landlord reference and resolved UPRN, as held in the
``property`` table. ``uprn`` is nullable a property whose UPRN has not been
resolved yet cannot receive condition rows."""
landlord_property_id: Optional[str]
uprn: Optional[int]
class PropertyReferenceReader(ABC):
@abstractmethod
def references_for_portfolio(self, portfolio_id: int) -> List[PropertyReference]:
...

View file

@ -0,0 +1,28 @@
from typing import List
from sqlmodel import Session, select
from infrastructure.postgres.property_table import PropertyRow
from repositories.condition.property_reference_reader import (
PropertyReference,
PropertyReferenceReader,
)
class PropertyReferenceReaderPostgres(PropertyReferenceReader):
"""Reads landlord-reference -> UPRN pairs from the ``property`` table for a
portfolio. The reference key is ``landlord_property_id`` (ADR-0064)."""
def __init__(self, session: Session) -> None:
self._session = session
def references_for_portfolio(self, portfolio_id: int) -> List[PropertyReference]:
rows = self._session.exec(
select(PropertyRow.landlord_property_id, PropertyRow.uprn).where(
PropertyRow.portfolio_id == portfolio_id
)
).all()
return [
PropertyReference(landlord_property_id=landlord_property_id, uprn=uprn)
for landlord_property_id, uprn in rows
]

View file

@ -0,0 +1,33 @@
from typing import Dict, List
from repositories.condition.property_reference_reader import PropertyReferenceReader
from repositories.condition.uprn_lookup import UprnLookup
class PropertyUprnLookup(UprnLookup):
"""Resolves a landlord property reference to a UPRN from the ``property``
table, scoped to one portfolio (ADR-0064)."""
def __init__(
self, reader: PropertyReferenceReader, portfolio_id: int
) -> None:
self._reader = reader
self._portfolio_id = portfolio_id
def get_property_ref_to_uprn_lookup(self) -> Dict[str, int]:
references = self._reader.references_for_portfolio(self._portfolio_id)
return {
ref.landlord_property_id: ref.uprn
for ref in references
if ref.landlord_property_id is not None and ref.uprn is not None
}
def null_uprn_references(self) -> List[str]:
"""Landlord references present in the portfolio whose UPRN is null —
surfaced so a load can report them (none expected; ADR-0064)."""
references = self._reader.references_for_portfolio(self._portfolio_id)
return [
ref.landlord_property_id
for ref in references
if ref.landlord_property_id is not None and ref.uprn is None
]

View file

View file

@ -1,5 +1,5 @@
from backend.app.db.models.condition import PropertyConditionSurveyModel
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
from infrastructure.postgres.condition_tables import PropertyConditionSurveyModel
from domain.condition.property_condition_survey import PropertyConditionSurvey
class CustomAsserts:

View file

View file

@ -0,0 +1,49 @@
from typing import List
from repositories.condition.property_reference_reader import (
PropertyReference,
PropertyReferenceReader,
)
from repositories.condition.property_uprn_lookup import PropertyUprnLookup
class FakePropertyReferenceReader(PropertyReferenceReader):
def __init__(self, references: List[PropertyReference]) -> None:
self._references = references
def references_for_portfolio(self, portfolio_id: int) -> List[PropertyReference]:
return self._references
def test_property_uprn_lookup_excludes_rows_with_no_uprn():
# Arrange
reader = FakePropertyReferenceReader(
[
PropertyReference(landlord_property_id="443", uprn=100093305101),
PropertyReference(landlord_property_id="999", uprn=None),
]
)
lookup = PropertyUprnLookup(reader, portfolio_id=824)
# Act
mapping = lookup.get_property_ref_to_uprn_lookup()
# Assert
assert mapping == {"443": 100093305101}
def test_property_uprn_lookup_reports_references_with_null_uprn():
# Arrange
reader = FakePropertyReferenceReader(
[
PropertyReference(landlord_property_id="443", uprn=100093305101),
PropertyReference(landlord_property_id="999", uprn=None),
]
)
lookup = PropertyUprnLookup(reader, portfolio_id=824)
# Act
null_references = lookup.null_uprn_references()
# Assert
assert null_references == ["999"]

View file

@ -2,7 +2,7 @@ import pytest
from typing import Dict
from tempfile import NamedTemporaryFile
from backend.condition.lookups.uprn_lookup_csv import UprnLookupLocal
from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal
@pytest.fixture

View file

View file

@ -0,0 +1,36 @@
from datetime import date
from domain.condition.aspect_type import AspectType
from domain.condition.element_type import ElementType
from domain.condition.mapping.calico.calico_mapper import CalicoMapper
from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition
from domain.condition.records.calico.calico_property import CalicoProperty
def test_calico_mapper_sets_renewal_year_to_export_year_plus_remaining_life():
# Arrange
calico_property = CalicoProperty(
uprn=100093305101,
assets=[
CalicoAssetCondition(
asset_reference=443,
path="Roof Covering",
covering_type="Concrete Tiles",
fitted_date=date(1998, 4, 1),
quantity=55,
remaining_life=36,
)
],
)
export_year = 2026
# Act
survey = CalicoMapper().map_asset_conditions_for_property(
calico_property, export_year
)
# Assert
roof_material = survey.elements[0].aspect_conditions[0]
assert roof_material.aspect_type is AspectType.MATERIAL
assert survey.elements[0].element_type is ElementType.ROOF
assert roof_material.renewal_year == export_year + 36

View file

@ -1,16 +1,16 @@
from datetime import date
from backend.condition.domain.aspect_condition import AspectCondition
from backend.condition.domain.aspect_type import AspectType
from backend.condition.domain.element_type import ElementType
from backend.condition.domain.mapping.lbwf.lbwf_mapper import LbwfMapper
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
from backend.condition.parsing.records.lbwf.lbwf_house import LbwfHouse
from backend.condition.parsing.records.lbwf.lbwf_asset_condition import (
from domain.condition.aspect_condition import AspectCondition
from domain.condition.aspect_type import AspectType
from domain.condition.element_type import ElementType
from domain.condition.mapping.lbwf.lbwf_mapper import LbwfMapper
from domain.condition.property_condition_survey import PropertyConditionSurvey
from domain.condition.records.lbwf.lbwf_house import LbwfHouse
from domain.condition.records.lbwf.lbwf_asset_condition import (
LbwfAssetCondition,
)
from backend.condition.domain.element import Element
from backend.condition.tests.custom_asserts import CustomAsserts
from domain.condition.element import Element
from tests.condition.custom_asserts import CustomAsserts
def test_lbwf_mapper_maps_house():

View file

@ -1,16 +1,16 @@
from datetime import datetime, date
from backend.condition.domain.aspect_condition import AspectCondition
from backend.condition.domain.aspect_type import AspectType
from backend.condition.domain.element_type import ElementType
from backend.condition.domain.mapping.peabody.peabody_mapper import PeabodyMapper
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
from backend.condition.parsing.records.peabody.peabody_asset_condition import (
from domain.condition.aspect_condition import AspectCondition
from domain.condition.aspect_type import AspectType
from domain.condition.element_type import ElementType
from domain.condition.mapping.peabody.peabody_mapper import PeabodyMapper
from domain.condition.property_condition_survey import PropertyConditionSurvey
from domain.condition.records.peabody.peabody_asset_condition import (
PeabodyAssetCondition,
)
from backend.condition.parsing.records.peabody.peabody_property import PeabodyProperty
from backend.condition.domain.element import Element
from backend.condition.tests.custom_asserts import CustomAsserts
from domain.condition.records.peabody.peabody_property import PeabodyProperty
from domain.condition.element import Element
from tests.condition.custom_asserts import CustomAsserts
def test_peabody_mapper_maps_property():

View file

View file

@ -0,0 +1,106 @@
from datetime import date
from io import BytesIO
from typing import Dict, List, Optional
from openpyxl import Workbook
from infrastructure.condition.parsing.calico_parser import CalicoParser
from repositories.condition.uprn_lookup import UprnLookup
HEADERS = [
"Asset Reference",
"Asset Type",
"Address",
"Path",
"Type",
"Fitted / Renewed Date",
"Quantity",
"Remaining Life",
]
class FakeUprnLookup(UprnLookup):
def __init__(self, mapping: Dict[str, int]) -> None:
self._mapping = mapping
def get_property_ref_to_uprn_lookup(self) -> Dict[str, int]:
return self._mapping
def _workbook(rows: List[List[object]]) -> BytesIO:
wb = Workbook()
ws = wb.active
assert ws is not None
ws.append(HEADERS)
for row in rows:
ws.append(row)
buffer = BytesIO()
wb.save(buffer)
buffer.seek(0)
return buffer
def test_calico_parser_counts_blank_and_unmatched_drops():
# Arrange -- one loaded, one blank placeholder, one unmatched (not in lookup)
workbook = _workbook(
[
[443, "HOUSE", "31 Venice Avenue", "Roof Covering", "Concrete Tiles",
date(1998, 4, 1), 55, 36],
[438, "FLAT", "137 Brownhill Avenue", "Roof Covering", "N/A",
date(2000, 4, 1), 0, 973],
[777, "HOUSE", "8 Marine Avenue", "Roof Covering", "Natural Slate",
date(2010, 4, 1), 60, 50],
]
)
parser = CalicoParser(FakeUprnLookup({"443": 100093305101}))
# Act
parser.parse(workbook)
# Assert
assert parser.blank_placeholder_count == 1
assert parser.unmatched_reference_count == 1
def test_calico_parser_drops_na_sentinel_covering_rows():
# Arrange -- Calico writes the literal string "N/A" for a placeholder row
# (no roof-covering record captured), not an empty cell.
workbook = _workbook(
[
[443, "HOUSE", "31 Venice Avenue", "Roof Covering", "Concrete Tiles",
date(1998, 4, 1), 55, 36],
[438, "FLAT", "137 Brownhill Avenue", "Roof Covering", "N/A",
date(2000, 4, 1), 0, 973],
]
)
parser = CalicoParser(FakeUprnLookup({"443": 100093305101, "438": 100093388053}))
# Act
properties = parser.parse(workbook)
# Assert
references = [asset.asset_reference for prop in properties for asset in prop.assets]
assert references == [443]
def test_calico_parser_drops_rows_with_no_covering_type():
# Arrange
workbook = _workbook(
[
[443, "HOUSE", "31 Venice Avenue", "Roof Covering", "Concrete Tiles",
date(1998, 4, 1), 55, 36],
[438, "FLAT", "137 Brownhill Avenue", "Roof Covering", None,
date(2000, 4, 1), 0, 973],
]
)
parser = CalicoParser(FakeUprnLookup({"443": 100093305101, "438": 100093388053}))
# Act
properties = parser.parse(workbook)
# Assert
all_assets = [asset for prop in properties for asset in prop.assets]
references: List[int] = [asset.asset_reference for asset in all_assets]
coverings: List[Optional[str]] = [asset.covering_type for asset in all_assets]
assert references == [443]
assert coverings == ["Concrete Tiles"]

View file

@ -4,9 +4,9 @@ from io import BytesIO
from openpyxl import Workbook
from datetime import datetime
from backend.condition.parsing.lbwf_parser import LbwfParser
from backend.condition.parsing.records.lbwf.lbwf_asset_condition import LbwfAssetCondition
from backend.condition.parsing.records.lbwf.lbwf_house import LbwfHouse
from infrastructure.condition.parsing.lbwf_parser import LbwfParser
from domain.condition.records.lbwf.lbwf_asset_condition import LbwfAssetCondition
from domain.condition.records.lbwf.lbwf_house import LbwfHouse
@pytest.fixture
def lbwf_homes_xlsx_bytes() -> BytesIO:

View file

@ -0,0 +1,51 @@
import pytest
from applications.condition.condition_trigger_request import ConditionFileType
from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal
from applications.condition.factory import select_parser, select_mapper
def test_selects_lbwf_parser():
# arrange
file_type = ConditionFileType.LBWF
expected_class_name = "LbwfParser"
# act
actual_class_name = select_parser(file_type).__class__.__name__
# assert
assert expected_class_name == actual_class_name
def test_selects_peabody_parser():
# arrange
file_type = ConditionFileType.Peabody
expected_class_name = "PeabodyParser"
uprn_lookup = UprnLookupLocal(csv_path="test")
# act
actual_class_name = select_parser(file_type, uprn_lookup).__class__.__name__
# assert
assert expected_class_name == actual_class_name
def test_selects_calico_parser():
# arrange
uprn_lookup = UprnLookupLocal(csv_path="test")
# act
actual_class_name = select_parser(
ConditionFileType.Calico, uprn_lookup
).__class__.__name__
# assert
assert actual_class_name == "CalicoParser"
def test_selects_calico_mapper():
# act
actual_class_name = select_mapper(ConditionFileType.Calico).__class__.__name__
# assert
assert actual_class_name == "CalicoMapper"

View file

@ -5,12 +5,12 @@ from io import BytesIO
from openpyxl import Workbook
from datetime import datetime
from backend.condition.lookups.uprn_lookup_csv import UprnLookupLocal
from backend.condition.parsing.peabody_parser import PeabodyParser
from backend.condition.parsing.records.peabody.peabody_asset_condition import (
from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal
from infrastructure.condition.parsing.peabody_parser import PeabodyParser
from domain.condition.records.peabody.peabody_asset_condition import (
PeabodyAssetCondition,
)
from backend.condition.parsing.records.peabody.peabody_property import PeabodyProperty
from domain.condition.records.peabody.peabody_property import PeabodyProperty
@pytest.fixture

View file

View file

@ -1,14 +1,14 @@
import pytest
from datetime import date
from backend.condition.persistence.condition_postgres import ConditionPostgres
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
from backend.condition.domain.element import Element
from backend.condition.domain.element_type import ElementType
from backend.condition.domain.aspect_condition import AspectCondition
from backend.condition.domain.aspect_type import AspectType
from backend.app.db.models.condition import PropertyConditionSurveyModel
from backend.condition.tests.custom_asserts import CustomAsserts
from repositories.condition.condition_postgres import ConditionPostgres
from domain.condition.property_condition_survey import PropertyConditionSurvey
from domain.condition.element import Element
from domain.condition.element_type import ElementType
from domain.condition.aspect_condition import AspectCondition
from domain.condition.aspect_type import AspectType
from infrastructure.postgres.condition_tables import PropertyConditionSurveyModel
from tests.condition.custom_asserts import CustomAsserts
def test_map_survey_to_model() -> None:

View file

@ -0,0 +1,88 @@
from datetime import date
from io import BytesIO
from typing import List
from openpyxl import Workbook
from applications.condition.calico_load import run_calico_load
from applications.condition.load_report import LoadReport
from domain.condition.property_condition_survey import PropertyConditionSurvey
from repositories.condition.property_reference_reader import (
PropertyReference,
PropertyReferenceReader,
)
from repositories.condition.property_uprn_lookup import PropertyUprnLookup
HEADERS = [
"Asset Reference",
"Asset Type",
"Address",
"Path",
"Type",
"Fitted / Renewed Date",
"Quantity",
"Remaining Life",
]
class FakePropertyReferenceReader(PropertyReferenceReader):
def __init__(self, references: List[PropertyReference]) -> None:
self._references = references
def references_for_portfolio(self, portfolio_id: int) -> List[PropertyReference]:
return self._references
class FakeConditionWriter:
def __init__(self) -> None:
self.surveys: List[PropertyConditionSurvey] = []
def bulk_insert_surveys(self, surveys: List[PropertyConditionSurvey]) -> None:
self.surveys.extend(surveys)
def _workbook(rows: List[List[object]]) -> BytesIO:
wb = Workbook()
ws = wb.active
assert ws is not None
ws.append(HEADERS)
for row in rows:
ws.append(row)
buffer = BytesIO()
wb.save(buffer)
buffer.seek(0)
return buffer
def test_run_calico_load_persists_surveys_and_reports_reconciliation():
# Arrange -- one loaded (443), one blank placeholder (438), one unmatched (777)
workbook = _workbook(
[
[443, "HOUSE", "31 Venice Avenue", "Roof Covering", "Concrete Tiles",
date(1998, 4, 1), 55, 36],
[438, "FLAT", "137 Brownhill Avenue", "Roof Covering", "N/A",
date(2000, 4, 1), 0, 973],
[777, "HOUSE", "8 Marine Avenue", "Roof Covering", "Natural Slate",
date(2010, 4, 1), 60, 50],
]
)
reader = FakePropertyReferenceReader(
[
PropertyReference(landlord_property_id="443", uprn=100093305101),
PropertyReference(landlord_property_id="999", uprn=None),
]
)
lookup = PropertyUprnLookup(reader, portfolio_id=824)
writer = FakeConditionWriter()
# Act
report = run_calico_load(workbook, lookup, export_year=2026, writer=writer)
# Assert
assert report == LoadReport(
loaded=1, unmatched_to_portfolio=1, null_uprn=1, blank_placeholder=1
)
assert len(writer.surveys) == 1
persisted = writer.surveys[0]
assert persisted.uprn == 100093305101
assert persisted.elements[0].aspect_conditions[0].renewal_year == 2026 + 36

View file

@ -0,0 +1,38 @@
from applications.condition.load_report import LoadReport
from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition
from domain.condition.records.calico.calico_property import CalicoProperty
def _property(uprn: int, reference: int, covering: str) -> CalicoProperty:
return CalicoProperty(
uprn=uprn,
assets=[
CalicoAssetCondition(
asset_reference=reference, path="Roof Covering", covering_type=covering
)
],
)
def test_load_report_summarises_the_four_reconciliation_counts():
# Arrange
properties = [
_property(100093305101, 443, "Concrete Tiles"),
_property(100093388053, 444, "Natural Slate"),
]
# Act
report = LoadReport.from_calico_load(
properties=properties,
blank_placeholder_count=1239,
unmatched_reference_count=5,
null_uprn_references=["999", "888"],
)
# Assert
assert report == LoadReport(
loaded=2,
unmatched_to_portfolio=5,
null_uprn=2,
blank_placeholder=1239,
)