mirror of
https://github.com/Hestia-Homes/Model.git
synced 2026-07-27 23:35:01 +00:00
Merge pull request #1544 from Hestia-Homes/feat/calico-stock-condition-ingestion
Calico roof-covering stock-condition ingestion (ADR-0064)
This commit is contained in:
commit
206ac197ee
81 changed files with 1150 additions and 172 deletions
12
CONTEXT.md
12
CONTEXT.md
|
|
@ -143,6 +143,18 @@ _Avoid_: energy assessment, site survey, field survey, Domna survey, Hestia surv
|
|||
Property data supplied by a landlord that may correct or supplement the public EPC for a single Property; triggers Rebaselining when applied; not applicable when Site Notes are present.
|
||||
_Avoid_: patches (deprecated), corrections, manual EPC, edits
|
||||
|
||||
**Stock Condition Data**:
|
||||
Landlord-supplied per-**component** lifecycle records — one row per element (e.g. a dwelling's **roof covering**) carrying its material/type, fitted-or-renewed date, quantity, and a **Remaining Life**. It answers *"when is this component due for renewal?"*, not *"what is the dwelling's SAP picture?"* — so unlike **Site Notes** and **Landlord Overrides** it is **not read by the modelling engine** and never enters `EpcPropertyData`, Rebaselining, or a Recommendation (ADR-0064). Sourced from a landlord's stock-condition survey (asset-management data); ingested by this repo purely as a **producer** and persisted keyed by UPRN as `PropertyConditionSurvey → Element → AspectCondition`. The consumer is the Next.js/Drizzle front-end, which reads the tables directly (the DB schema is the contract — the `element_type` / `aspect_type` PG enums are an FE-owned surface, cf. [[main-heating-system-pgenum-is-fe-owned]]) to display component data against a Property and to raise the **Roof Replacement Flag**. **Remaining Life** derives from the component's **Decent Homes Standard** expected lifetime (`fitted date + standard lifespan − report date`), so it is trusted as given, not recomputed here.
|
||||
_Avoid_: condition survey (ambiguous with **Site Notes**), asset data, patches; treating it as a **Landlord Override** or fabric signal (the engine never reads it — this iteration)
|
||||
|
||||
**Component Renewal Year**:
|
||||
The absolute calendar year a **Stock Condition Data** component is due for replacement — stored as `renewal_year` on the `AspectCondition`. Computed at load as **`export_year + Remaining Life`** (the export/report date the landlord's file was generated, not `now()`), so it is stable across re-runs — an absolute year does not drift, whereas a stored "years remaining" would. The fitted/renewed date is retained separately as `install_date`.
|
||||
_Avoid_: remaining life (that is the input; the stored value is an absolute year), replacement date, end-of-life date
|
||||
|
||||
**Roof Replacement Flag**:
|
||||
A **front-end-owned** presentation signal raised on a **Solar PV Recommendation** when a Property's roof-covering **Component Renewal Year** falls within (or near) the measure horizon — warning a human that installing solar onto a soon-to-be-replaced roof is unwise. It is **not** computed in this repo and does **not** change what the engine recommends (the optimiser's solar Option is unchanged): the Next.js/Drizzle app joins **Stock Condition Data** to the recommendation by UPRN and applies the "due soon" threshold itself (ADR-0064). A future engine upgrade that *acts* on this (bundling a re-roof, deferring solar) is explicitly out of scope for this iteration.
|
||||
_Avoid_: roof warning (unspecific), re-roof recommendation (there is no engine measure yet), solar suppression
|
||||
|
||||
**Landlord-Description Classification**:
|
||||
Resolving a **Landlord Description** (unbounded free-text a landlord supplies for one component — "CWI" / "Cav filled" / "cavity insulated" all name one thing) onto a **Recognised Internal Description** via an LLM classifier, persisted in the `landlord_*_overrides` table (`source=classifier`) as a reviewed cache. Four vocabularies are kept **distinct** and must not be conflated: a **Landlord Description** (unbounded input); a **Recognised Internal Description** (the closed target taxonomy — e.g. a `MainHeatingSystemType` archetype — each binding to a Simulation Overlay); a **Lodged Description** (the gov-EPC `main_heating[].description` rendering, e.g. "Room heaters, electric" — only an example of which system *types* occur, never a map key); and the **SAP main heating code** (Table 4a/4b, what the calculator consumes). The classifier maps Landlord → Recognised Internal → SAP code. When it cannot confidently place the text it emits **`None`** (no overlay → the lodged EPC stands, surfaced to the user as "no suitable match"), **never the nearest wrong archetype** — the target taxonomy must be complete enough that a real system always has a correct home, so the classifier never overflows into a garbage-drawer archetype (ADR-0041).
|
||||
|
||||
|
|
|
|||
|
|
@ -20,22 +20,26 @@ COPY backend/.env.test backend/.env
|
|||
# -----------------------------
|
||||
# Copy requirements FIRST (for Docker layer caching)
|
||||
# -----------------------------
|
||||
COPY backend/condition/handler/requirements.txt .
|
||||
COPY applications/condition/requirements.txt .
|
||||
|
||||
# Install dependencies into Lambda runtime
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# -----------------------------
|
||||
# Copy application code
|
||||
# Copy application code (the handler's import closure — DDD layout)
|
||||
# -----------------------------
|
||||
COPY utils/ utils/
|
||||
COPY backend/condition/ backend/condition/
|
||||
COPY domain/ domain/
|
||||
COPY infrastructure/ infrastructure/
|
||||
COPY repositories/ repositories/
|
||||
COPY applications/ applications/
|
||||
|
||||
COPY backend/app/db/models/condition.py backend/app/db/models/condition.py
|
||||
# Legacy backend bridges still imported by the condition module:
|
||||
# ConditionPostgres -> backend.app.db.connection (db_session);
|
||||
# condition_tables -> backend.app.db.base (Base); connection -> backend.app.config.
|
||||
COPY backend/app/config.py backend/app/config.py
|
||||
COPY backend/app/db/base.py backend/app/db/base.py
|
||||
COPY backend/app/db/connection.py backend/app/db/connection.py
|
||||
COPY backend/app/config.py backend/app/config.py
|
||||
|
||||
COPY backend/__init__.py backend/__init__.py
|
||||
COPY backend/app/__init__.py backend/app/__init__.py
|
||||
COPY backend/app/db/__init__.py backend/app/db/__init__.py
|
||||
|
|
@ -44,6 +48,6 @@ COPY backend/app/db/__init__.py backend/app/db/__init__.py
|
|||
# -----------------------------
|
||||
# Lambda handler
|
||||
# -----------------------------
|
||||
CMD ["backend/condition/handler/handler.handler"]
|
||||
CMD ["applications/condition/handler.handler"]
|
||||
# For local running
|
||||
# CMD ["python", "-m", "backend.condition.handler.handler"]
|
||||
# CMD ["python", "-m", "applications.condition.handler"]
|
||||
|
|
@ -12,7 +12,31 @@ The Condition Data Processor performs the following steps:
|
|||
- **Load**
|
||||
- Persist transformed data into the ARA database (not yet implemented)
|
||||
|
||||
The processor currently supports file formats provided by **Peabody** and **LBWF**.
|
||||
The processor currently supports file formats provided by **Peabody**, **LBWF**,
|
||||
and **Calico**.
|
||||
|
||||
---
|
||||
|
||||
## Calico (roof-covering, one-off)
|
||||
|
||||
Calico is a one-off bulk load of stock-condition **roof-covering** data
|
||||
(ADR-0064), run through `applications/condition/calico_runner.py`
|
||||
(`run_calico_s3_load`), not the SQS Lambda. It is **producer-only** — written to
|
||||
the `property_condition_survey` tables for the FE to read; the modelling engine
|
||||
never reads it.
|
||||
|
||||
- **Source:** `s3://condition-data-dev/input/calico/Roof Covering 2.6.26.xlsx`
|
||||
- **Portfolio:** 824 · **export year:** 2026 (anchors renewal years).
|
||||
- **Mapping:** `"Roof Covering"` → `ROOF` / `MATERIAL` (the external covering
|
||||
material — tiles/slate — not the lining or structure).
|
||||
- **Drops (reconciled, never silent):** `"N/A"`/blank covering rows on parse
|
||||
(1,239), references that don't resolve to a UPRN, and null-UPRN properties are
|
||||
reported. A `LoadReport` summarises `loaded / unmatched / null-uprn / blank`.
|
||||
- **⚠ Open — UPRN resolution:** Calico's `Asset Reference` does **not** match our
|
||||
`landlord_property_id` for portfolio 824 (confirmed: zero overlap; addresses do
|
||||
match). The Postgres `PropertyUprnLookup` cannot resolve Calico yet — it needs a
|
||||
Calico-supplied `Asset Reference → UPRN` table (use the CSV `UprnLookup`) or
|
||||
address matching. See the ADR-0064 update.
|
||||
|
||||
---
|
||||
|
||||
51
applications/condition/calico_load.py
Normal file
51
applications/condition/calico_load.py
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
from typing import BinaryIO, List
|
||||
|
||||
from applications.condition.load_report import LoadReport
|
||||
from domain.condition.mapping.calico.calico_mapper import CalicoMapper
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
from infrastructure.condition.parsing.calico_parser import CalicoParser
|
||||
from repositories.condition.condition_writer import ConditionWriter
|
||||
from repositories.condition.property_uprn_lookup import PropertyUprnLookup
|
||||
from repositories.condition.uprn_lookup import UprnLookup
|
||||
from utils.logger import setup_logger
|
||||
|
||||
logger = setup_logger()
|
||||
|
||||
|
||||
def run_calico_load(
|
||||
file_stream: BinaryIO,
|
||||
uprn_lookup: UprnLookup,
|
||||
export_year: int,
|
||||
writer: ConditionWriter,
|
||||
) -> LoadReport:
|
||||
"""Parse a Calico export, map each roof-covering observation to a condition
|
||||
survey keyed by UPRN, persist them, and return the reconciliation report
|
||||
(ADR-0064). ``export_year`` anchors renewal years so they don't drift.
|
||||
|
||||
Works with any ``UprnLookup``. A CSV/S3 lookup carries only resolved UPRNs,
|
||||
so the ``null_uprn`` count is only meaningful for the property-table lookup
|
||||
(which surfaces portfolio properties missing a UPRN); it is zero otherwise."""
|
||||
parser = CalicoParser(uprn_lookup)
|
||||
properties = parser.parse(file_stream)
|
||||
|
||||
mapper = CalicoMapper()
|
||||
surveys: List[PropertyConditionSurvey] = [
|
||||
mapper.map_asset_conditions_for_property(prop, export_year)
|
||||
for prop in properties
|
||||
]
|
||||
|
||||
writer.bulk_insert_surveys(surveys)
|
||||
|
||||
null_uprn_references = (
|
||||
uprn_lookup.null_uprn_references()
|
||||
if isinstance(uprn_lookup, PropertyUprnLookup)
|
||||
else []
|
||||
)
|
||||
report = LoadReport.from_calico_load(
|
||||
properties=properties,
|
||||
blank_placeholder_count=parser.blank_placeholder_count,
|
||||
unmatched_reference_count=parser.unmatched_reference_count,
|
||||
null_uprn_references=null_uprn_references,
|
||||
)
|
||||
logger.info(f"[calico_load] {report}")
|
||||
return report
|
||||
65
applications/condition/calico_runner.py
Normal file
65
applications/condition/calico_runner.py
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
"""One-off runner for the Calico stock-condition (roof-covering) load.
|
||||
|
||||
Calico is a bulk one-off import, not a recurring feed, so it runs through this
|
||||
runner rather than the SQS-Lambda path Peabody/LBWF use (ADR-0064). The raw
|
||||
export and the Asset-Reference->UPRN lookup both live in S3.
|
||||
|
||||
Not exercised by the unit suite -- it is S3 + Postgres wiring around the
|
||||
unit-tested ``run_calico_load``; run it where a database and S3 are available.
|
||||
|
||||
Calico Asset References do not match our stored ``landlord_property_id``, so the
|
||||
UPRN is resolved from an address-matched CSV lookup (built against portfolio 824
|
||||
and uploaded to S3), not the property table. Pass ``dry_run=True`` to reconcile
|
||||
without writing.
|
||||
"""
|
||||
|
||||
from io import BytesIO
|
||||
from typing import List
|
||||
|
||||
from applications.condition.calico_load import run_calico_load
|
||||
from applications.condition.load_report import LoadReport
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
from infrastructure.condition.lookups.uprn_lookup_s3 import UprnLookupS3
|
||||
from repositories.condition.condition_postgres import ConditionPostgres
|
||||
from utils.logger import setup_logger
|
||||
from utils.s3 import read_io_from_s3
|
||||
|
||||
logger = setup_logger()
|
||||
|
||||
# The export date the Calico file was generated (Roof Covering 2.6.26); anchors
|
||||
# renewal years so they don't drift on re-run (ADR-0064).
|
||||
CALICO_EXPORT_YEAR = 2026
|
||||
CALICO_BUCKET = "condition-data-dev"
|
||||
CALICO_KEY = "input/calico/Roof Covering 2.6.26.xlsx"
|
||||
CALICO_UPRN_LOOKUP_KEY = "input/calico/uprn-lookup/calico_reference_to_uprn.csv"
|
||||
|
||||
|
||||
class _NoOpWriter:
|
||||
"""Discards surveys -- used for a dry run so the load can be reconciled
|
||||
without touching the database."""
|
||||
|
||||
def bulk_insert_surveys(
|
||||
self, surveys: List[PropertyConditionSurvey]
|
||||
) -> None:
|
||||
logger.info(f"[calico_runner] dry run -- would persist {len(surveys)} surveys")
|
||||
|
||||
|
||||
def run_calico_s3_load(
|
||||
bucket: str = CALICO_BUCKET,
|
||||
key: str = CALICO_KEY,
|
||||
uprn_lookup_key: str = CALICO_UPRN_LOOKUP_KEY,
|
||||
export_year: int = CALICO_EXPORT_YEAR,
|
||||
dry_run: bool = False,
|
||||
) -> LoadReport:
|
||||
file_bytes: BytesIO = read_io_from_s3(bucket_name=bucket, file_key=key)
|
||||
lookup = UprnLookupS3(bucket=bucket, key=uprn_lookup_key)
|
||||
writer = _NoOpWriter() if dry_run else ConditionPostgres()
|
||||
|
||||
report = run_calico_load(
|
||||
file_stream=file_bytes,
|
||||
uprn_lookup=lookup,
|
||||
export_year=export_year,
|
||||
writer=writer,
|
||||
)
|
||||
logger.info(f"[calico_runner] {'(dry run) ' if dry_run else ''}{report}")
|
||||
return report
|
||||
|
|
@ -6,6 +6,7 @@ from pydantic import BaseModel
|
|||
class ConditionFileType(Enum):
|
||||
LBWF = "LBWF"
|
||||
Peabody = "Peabody"
|
||||
Calico = "Calico"
|
||||
# TODO: make these asset management systems rather than client names
|
||||
|
||||
|
||||
47
applications/condition/factory.py
Normal file
47
applications/condition/factory.py
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
from typing import Optional
|
||||
from applications.condition.condition_trigger_request import ConditionFileType
|
||||
from domain.condition.mapping.calico.calico_mapper import CalicoMapper
|
||||
from domain.condition.mapping.lbwf.lbwf_mapper import LbwfMapper
|
||||
from domain.condition.mapping.mapper import Mapper
|
||||
from domain.condition.mapping.peabody.peabody_mapper import PeabodyMapper
|
||||
from repositories.condition.uprn_lookup import UprnLookup
|
||||
from infrastructure.condition.parsing.parser import Parser
|
||||
from infrastructure.condition.parsing.calico_parser import CalicoParser
|
||||
from infrastructure.condition.parsing.lbwf_parser import LbwfParser
|
||||
from infrastructure.condition.parsing.peabody_parser import PeabodyParser
|
||||
|
||||
|
||||
def select_parser(
|
||||
file_type: ConditionFileType, uprn_lookup: Optional[UprnLookup] = None
|
||||
) -> Parser:
|
||||
if file_type is ConditionFileType.LBWF:
|
||||
return LbwfParser()
|
||||
|
||||
if file_type is ConditionFileType.Peabody:
|
||||
if not uprn_lookup:
|
||||
raise ValueError(
|
||||
"Cannot instantiate Peabody Parser without UPRN lookup being provided"
|
||||
)
|
||||
return PeabodyParser(uprn_lookup=uprn_lookup)
|
||||
|
||||
if file_type is ConditionFileType.Calico:
|
||||
if not uprn_lookup:
|
||||
raise ValueError(
|
||||
"Cannot instantiate Calico Parser without UPRN lookup being provided"
|
||||
)
|
||||
return CalicoParser(uprn_lookup=uprn_lookup)
|
||||
|
||||
raise ValueError("Unrecognised file type, unable to instantiate Parser")
|
||||
|
||||
|
||||
def select_mapper(file_type: ConditionFileType) -> Mapper:
|
||||
if file_type is ConditionFileType.LBWF:
|
||||
return LbwfMapper()
|
||||
|
||||
if file_type is ConditionFileType.Peabody:
|
||||
return PeabodyMapper()
|
||||
|
||||
if file_type is ConditionFileType.Calico:
|
||||
return CalicoMapper()
|
||||
|
||||
raise ValueError("Unrecognised file type, unable to instantiate Mapper")
|
||||
|
|
@ -2,9 +2,9 @@ import json
|
|||
from typing import Mapping, Any
|
||||
from io import BytesIO
|
||||
|
||||
from backend.condition.condition_trigger_request import ConditionTriggerRequest
|
||||
from backend.condition.lookups.uprn_lookup_s3 import UprnLookupS3
|
||||
from backend.condition.processor import process_file
|
||||
from applications.condition.condition_trigger_request import ConditionTriggerRequest
|
||||
from infrastructure.condition.lookups.uprn_lookup_s3 import UprnLookupS3
|
||||
from applications.condition.processor import process_file
|
||||
from utils.logger import setup_logger
|
||||
from utils.s3 import read_io_from_s3
|
||||
|
||||
39
applications/condition/load_report.py
Normal file
39
applications/condition/load_report.py
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
from dataclasses import dataclass
|
||||
from typing import List
|
||||
|
||||
from domain.condition.records.calico.calico_property import CalicoProperty
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LoadReport:
|
||||
"""The reconciliation a Calico load emits so nothing vanishes silently
|
||||
(ADR-0064): how many observations were loaded versus dropped, and why.
|
||||
|
||||
``loaded``, ``unmatched_to_portfolio`` and ``blank_placeholder`` account for
|
||||
the Calico rows. ``null_uprn`` is a portfolio-wide data-quality canary — the
|
||||
count of properties in the portfolio whose UPRN is null (expected: zero) —
|
||||
*not* a per-row drop reason, so it does not partition with the others: a
|
||||
Calico reference pointing at a null-UPRN property is reported both here and
|
||||
under ``unmatched_to_portfolio``.
|
||||
"""
|
||||
|
||||
loaded: int
|
||||
unmatched_to_portfolio: int
|
||||
null_uprn: int
|
||||
blank_placeholder: int
|
||||
|
||||
@classmethod
|
||||
def from_calico_load(
|
||||
cls,
|
||||
properties: List[CalicoProperty],
|
||||
blank_placeholder_count: int,
|
||||
unmatched_reference_count: int,
|
||||
null_uprn_references: List[str],
|
||||
) -> "LoadReport":
|
||||
loaded = sum(len(prop.assets) for prop in properties)
|
||||
return cls(
|
||||
loaded=loaded,
|
||||
unmatched_to_portfolio=unmatched_reference_count,
|
||||
null_uprn=len(null_uprn_references),
|
||||
blank_placeholder=blank_placeholder_count,
|
||||
)
|
||||
|
|
@ -1,8 +1,8 @@
|
|||
from pathlib import Path
|
||||
|
||||
from backend.condition.condition_trigger_request import ConditionFileType
|
||||
from backend.condition.lookups.uprn_lookup_csv import UprnLookupLocal
|
||||
from backend.condition.processor import process_file
|
||||
from applications.condition.condition_trigger_request import ConditionFileType
|
||||
from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal
|
||||
from applications.condition.processor import process_file
|
||||
|
||||
|
||||
def main() -> None:
|
||||
|
|
@ -1,14 +1,14 @@
|
|||
from typing import Any, BinaryIO, List, Optional
|
||||
from datetime import datetime
|
||||
|
||||
from backend.condition.condition_trigger_request import ConditionFileType
|
||||
from backend.condition.lookups.uprn_lookup import UprnLookup
|
||||
from applications.condition.condition_trigger_request import ConditionFileType
|
||||
from repositories.condition.uprn_lookup import UprnLookup
|
||||
from utils.logger import setup_logger
|
||||
from backend.condition.domain.mapping.mapper import Mapper
|
||||
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
|
||||
from backend.condition.parsing.parser import Parser
|
||||
from backend.condition.persistence.condition_postgres import ConditionPostgres
|
||||
from backend.condition.parsing.factory import select_parser, select_mapper
|
||||
from domain.condition.mapping.mapper import Mapper
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
from infrastructure.condition.parsing.parser import Parser
|
||||
from repositories.condition.condition_postgres import ConditionPostgres
|
||||
from applications.condition.factory import select_parser, select_mapper
|
||||
|
||||
logger = setup_logger()
|
||||
|
||||
|
|
@ -3,7 +3,7 @@ from sqlalchemy import insert, delete
|
|||
from sqlalchemy.orm import Session
|
||||
|
||||
from backend.app.db.connection import db_session, db_read_session
|
||||
from backend.app.db.models.condition import PropertyConditionSurveyModel
|
||||
from infrastructure.postgres.condition_tables import PropertyConditionSurveyModel
|
||||
|
||||
|
||||
def bulk_insert_property_surveys(
|
||||
|
|
|
|||
|
|
@ -1,35 +0,0 @@
|
|||
from typing import Optional
|
||||
from backend.condition.condition_trigger_request import ConditionFileType
|
||||
from backend.condition.domain.mapping.lbwf.lbwf_mapper import LbwfMapper
|
||||
from backend.condition.domain.mapping.mapper import Mapper
|
||||
from backend.condition.domain.mapping.peabody.peabody_mapper import PeabodyMapper
|
||||
from backend.condition.lookups.uprn_lookup import UprnLookup
|
||||
from backend.condition.parsing.parser import Parser
|
||||
from backend.condition.parsing.lbwf_parser import LbwfParser
|
||||
from backend.condition.parsing.peabody_parser import PeabodyParser
|
||||
|
||||
|
||||
def select_parser(
|
||||
file_type: ConditionFileType, uprn_lookup: Optional[UprnLookup] = None
|
||||
) -> Parser:
|
||||
if file_type is ConditionFileType.LBWF:
|
||||
return LbwfParser()
|
||||
|
||||
if file_type is ConditionFileType.Peabody:
|
||||
if not uprn_lookup:
|
||||
raise ValueError(
|
||||
"Cannot instantiate Peabody Parser without UPRN lookup being provided"
|
||||
)
|
||||
return PeabodyParser(uprn_lookup=uprn_lookup)
|
||||
|
||||
raise ValueError("Unrecognised file type, unable to instantiate Parser")
|
||||
|
||||
|
||||
def select_mapper(file_type: ConditionFileType) -> Mapper:
|
||||
if file_type is ConditionFileType.LBWF:
|
||||
return LbwfMapper()
|
||||
|
||||
if file_type is ConditionFileType.Peabody:
|
||||
return PeabodyMapper()
|
||||
|
||||
raise ValueError("Unrecognised file type, unable to instantiate Mapper")
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
import pytest
|
||||
|
||||
from backend.condition.condition_trigger_request import ConditionFileType
|
||||
from backend.condition.lookups.uprn_lookup_csv import UprnLookupLocal
|
||||
from backend.condition.parsing.factory import select_parser
|
||||
|
||||
|
||||
def test_selects_lbwf_parser():
|
||||
# arrange
|
||||
file_type = ConditionFileType.LBWF
|
||||
expected_class_name = "LbwfParser"
|
||||
|
||||
# act
|
||||
actual_class_name = select_parser(file_type).__class__.__name__
|
||||
|
||||
# assert
|
||||
assert expected_class_name == actual_class_name
|
||||
|
||||
|
||||
def test_selects_peabody_parser():
|
||||
# arrange
|
||||
file_type = ConditionFileType.Peabody
|
||||
expected_class_name = "PeabodyParser"
|
||||
uprn_lookup = UprnLookupLocal(csv_path="test")
|
||||
|
||||
# act
|
||||
actual_class_name = select_parser(file_type, uprn_lookup).__class__.__name__
|
||||
|
||||
# assert
|
||||
assert expected_class_name == actual_class_name
|
||||
|
|
@ -0,0 +1,127 @@
|
|||
---
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# Stock condition data is a producer-only, engine-isolated dataset; the roof-replacement warning on solar is a front-end overlay
|
||||
|
||||
We are ingesting a landlord's (Calico) **stock-condition survey** — for now, one
|
||||
component: **roof covering**, ~5,346 rows of `Asset Reference, Type (material),
|
||||
Fitted/Renewed Date, Quantity, Remaining Life`. The temptation is to treat any
|
||||
landlord-supplied component fact like a **Landlord Override** and fold it into the
|
||||
`EpcPropertyData` picture. That would be wrong here, and the whole design turns on
|
||||
saying why.
|
||||
|
||||
**Stock Condition Data answers a different question from the modelling picture.**
|
||||
A **Landlord Override** ("this wall is solid brick") and **Site Notes** (a full
|
||||
Domna survey) both correct *what the dwelling is* so the SAP score is right — the
|
||||
engine reads them. Stock Condition Data answers *when a component is due for
|
||||
renewal*: the roof-covering material plus a **Remaining Life** derived from the
|
||||
component's **Decent Homes Standard** expected lifetime. It is asset-management
|
||||
data, not a fabric signal. Conflating the two would drag component-lifecycle rows
|
||||
into `EpcPropertyData`, Rebaselining, and the Recommendation model for no scoring
|
||||
benefit and real coupling cost.
|
||||
|
||||
**The consumer is the product, not the engine.** The value is: (1) show component
|
||||
data against a Property in the front-end, and (2) warn a human, on a **Solar PV
|
||||
Recommendation**, when the roof is near end-of-life — because installing solar
|
||||
onto a roof due for replacement is a poor sequence. That warning is a *human
|
||||
information* signal, not an optimiser input: the engine still recommends solar
|
||||
unchanged; a badge says "mind the roof". The cross-reference (roof renewal year ↔
|
||||
solar recommendation) and the "due soon" threshold live in the **Next.js/Drizzle
|
||||
front-end**, which reads the condition tables directly. This repo is purely a
|
||||
**producer** into a schema the front-end consumes — so the DB schema (the
|
||||
`property_condition_survey` / `element` / `aspect_condition` tables and their
|
||||
`element_type` / `aspect_type` PG enums) is an **FE-owned contract**, the same
|
||||
pattern as `main_heating_system`.
|
||||
|
||||
Decided in a grill-with-docs session with Khalim, 2026-07-13.
|
||||
|
||||
## Decision
|
||||
|
||||
**Stock Condition Data is ingested producer-only and is engine-isolated. The
|
||||
modelling engine never reads it (this iteration); the roof-replacement warning on
|
||||
solar is a front-end presentation overlay, not an engine output.**
|
||||
|
||||
- **Engine isolation.** Condition rows are written only to
|
||||
`PropertyConditionSurvey → Element → AspectCondition`, keyed by UPRN. They never
|
||||
enter `EpcPropertyData`, Rebaselining, Bill Derivation, or a Recommendation. The
|
||||
optimiser's Solar PV Option is unchanged whether or not the roof is near
|
||||
end-of-life. A future engine upgrade that *acts* on condition data (bundling a
|
||||
re-roof, deferring solar) is a separate, later decision — explicitly out of
|
||||
scope now.
|
||||
|
||||
- **The flag lives above modelling.** The **Roof Replacement Flag** is computed by
|
||||
the Next.js/Drizzle app, which joins condition data to the solar recommendation
|
||||
by UPRN and owns the "due soon" threshold. This repo neither computes the flag
|
||||
nor exposes a read API for it — it only writes the tables the FE reads. The DB
|
||||
schema is therefore a shared FE contract; schema changes need a companion
|
||||
Drizzle view.
|
||||
|
||||
- **UPRN resolution from the property table, not a CSV.** Calico's `Asset
|
||||
Reference` maps to `property.landlord_property_id` scoped to `portfolio_id =
|
||||
824`, which yields the `uprn`. The lookup is a Postgres-backed `UprnLookup`
|
||||
reading the property table — fulfilling the existing "replace CSV with postgres"
|
||||
TODO — not an exported file. A key-alignment test (Asset Reference == stored
|
||||
`landlord_property_id`) is pinned during TDD implementation.
|
||||
|
||||
> **Update (confirmed against the DB, 2026-07-13).** The key alignment is
|
||||
> **false**: Calico Asset References (`438…14029`) have **zero** overlap with
|
||||
> portfolio 824's stored `landlord_property_id` (`690548…`), though the
|
||||
> *addresses* match — portfolio 824 **is** Calico's Burnley stock, keyed under a
|
||||
> different scheme, and all 5,276 of its properties have a non-null UPRN. So the
|
||||
> Postgres property-table lookup cannot resolve Calico. An exact address-line
|
||||
> join is also weak (555 / 3,990). Resolution is **open pending a Calico-supplied
|
||||
> `Asset Reference → UPRN` (or → `landlord_property_id`) table** — which reverts
|
||||
> this to the CSV `UprnLookup` path — or a proper address-normalisation match.
|
||||
> The `PropertyUprnLookup` code stands; it is simply not the right resolver for
|
||||
> Calico until the key exists.
|
||||
|
||||
- **Drop-and-report reconciliation; nothing vanishes silently.** Rows that do not
|
||||
resolve to a portfolio-824 UPRN are dropped and counted; rows whose resolved
|
||||
`property.uprn` is null are dropped **and reported** (none expected — we want to
|
||||
see any that are); the ~1,239 blank-`Type` placeholder rows (Quantity 0,
|
||||
sentinel date, nonsense Remaining Life) are dropped on parse. Every load emits a
|
||||
reconciliation report: `loaded / unmatched-to-portfolio / null-uprn /
|
||||
blank-placeholder`.
|
||||
|
||||
- **Renewal year is anchored to the export date, not `now()`.** `Remaining Life`
|
||||
is years-from-report; the stored **Component Renewal Year** = `export_year +
|
||||
Remaining Life` (export date passed explicitly as a load parameter — Calico's
|
||||
file is dated ≈ 2026-06-02), so the absolute renewal year is reproducible across
|
||||
re-runs. The fitted/renewed date is retained as `install_date`. This replaces
|
||||
the module's `survey_year = datetime.now().year` TODO for this loader.
|
||||
|
||||
- **Delivery: one-off runner load, from S3.** Calico is a bulk one-off import, run
|
||||
through the local/orchestration runner (not the SQS-Lambda path Peabody and LBWF
|
||||
use). The raw `.xlsx` is stored in S3 under a documented Calico key and kept out
|
||||
of git (it carries real addresses).
|
||||
|
||||
## Consequences
|
||||
|
||||
- The condition-ingestion module moves out of legacy `backend/condition/` into the
|
||||
DDD layout (domain / repositories / infrastructure / applications /
|
||||
orchestration) as a full lift-and-shift — shared core, the Peabody and LBWF
|
||||
adapters, and the delivery path — with Calico added as a new adapter alongside
|
||||
them. Peabody/LBWF keep their Lambda after the move; the relocation is
|
||||
behaviour-preserving for them. **The move is deliberately relocation-only:** the
|
||||
ORM models stay SQLAlchemy-declarative and `ConditionPostgres` keeps the legacy
|
||||
`backend.app.db` `Base` / `db_session` bridges, so the existing test suite proves
|
||||
behaviour is unchanged. Converting the tables to SQLModel and the repository to
|
||||
constructor session-injection (matching the rest of `infrastructure/postgres`),
|
||||
and clearing the module's pre-existing pyright-strict debt, are tracked
|
||||
follow-ups to run where a database is available to verify them — not bundled into
|
||||
the move.
|
||||
- A new `Calico` `ConditionFileType`, parser + row DTO, roof-covering element map
|
||||
(`"Roof Covering"` → roof `ElementType`, `MATERIAL` aspect = `Type`), and mapper
|
||||
are added and registered in the factory. Roof covering is already a modelled
|
||||
component type in `ElementType`, so no enum change is needed for this load.
|
||||
- CONTEXT.md gains **Stock Condition Data**, **Component Renewal Year**, and
|
||||
**Roof Replacement Flag**, kept distinct from **Site Notes** and **Landlord
|
||||
Overrides** so the ubiquitous language records that this source is *not* read by
|
||||
the engine.
|
||||
- Because the FE reads these tables through Drizzle, any later schema change here
|
||||
(a new component, a new aspect) is a cross-repo contract change requiring a
|
||||
companion Drizzle migration — the `main_heating_system` FE-ownership lesson
|
||||
applies.
|
||||
- If the "mind the roof" signal later needs to *drive* modelling, that is a new
|
||||
ADR that deliberately re-opens engine isolation — not an incremental tweak.
|
||||
0
domain/condition/__init__.py
Normal file
0
domain/condition/__init__.py
Normal file
|
|
@ -2,7 +2,7 @@ from dataclasses import dataclass
|
|||
from typing import Optional
|
||||
from datetime import date
|
||||
|
||||
from backend.condition.domain.aspect_type import AspectType
|
||||
from domain.condition.aspect_type import AspectType
|
||||
|
||||
|
||||
@dataclass
|
||||
|
|
@ -1,8 +1,8 @@
|
|||
from dataclasses import dataclass
|
||||
from typing import List
|
||||
|
||||
from backend.condition.domain.aspect_condition import AspectCondition
|
||||
from backend.condition.domain.element_type import ElementType
|
||||
from domain.condition.aspect_condition import AspectCondition
|
||||
from domain.condition.element_type import ElementType
|
||||
|
||||
|
||||
@dataclass
|
||||
0
domain/condition/mapping/__init__.py
Normal file
0
domain/condition/mapping/__init__.py
Normal file
0
domain/condition/mapping/calico/__init__.py
Normal file
0
domain/condition/mapping/calico/__init__.py
Normal file
12
domain/condition/mapping/calico/calico_element_map.py
Normal file
12
domain/condition/mapping/calico/calico_element_map.py
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
from domain.condition.aspect_type import AspectType
|
||||
from domain.condition.element_type import ElementType
|
||||
from domain.condition.mapping.element_mapping import ElementMapping
|
||||
|
||||
# Calico's component path -> the shared taxonomy. Held as data (like the LBWF /
|
||||
# Peabody element maps) so extending Calico to further components is a data edit.
|
||||
CALICO_ELEMENT_MAP: dict[str, ElementMapping] = {
|
||||
"Roof Covering": ElementMapping(
|
||||
elementType=ElementType.ROOF,
|
||||
aspect_type=AspectType.MATERIAL,
|
||||
),
|
||||
}
|
||||
56
domain/condition/mapping/calico/calico_mapper.py
Normal file
56
domain/condition/mapping/calico/calico_mapper.py
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
from datetime import date
|
||||
from typing import Any, List, Optional
|
||||
|
||||
from domain.condition.aspect_condition import AspectCondition
|
||||
from domain.condition.element import Element
|
||||
from domain.condition.mapping.calico.calico_element_map import CALICO_ELEMENT_MAP
|
||||
from domain.condition.mapping.mapper import Mapper
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition
|
||||
from domain.condition.records.calico.calico_property import CalicoProperty
|
||||
|
||||
SOURCE = "Calico"
|
||||
|
||||
|
||||
class CalicoMapper(Mapper):
|
||||
def map_asset_conditions_for_property(
|
||||
self, client_property_data: Any, survey_year: Optional[int] = None
|
||||
) -> PropertyConditionSurvey:
|
||||
assert isinstance(client_property_data, CalicoProperty)
|
||||
|
||||
elements: List[Element] = []
|
||||
for instance, asset in enumerate(client_property_data.assets, start=1):
|
||||
mapping = CALICO_ELEMENT_MAP[asset.path]
|
||||
elements.append(
|
||||
Element(
|
||||
element_type=mapping.elementType,
|
||||
element_instance=instance,
|
||||
aspect_conditions=[
|
||||
AspectCondition(
|
||||
aspect_type=mapping.aspect_type,
|
||||
aspect_instance=1,
|
||||
value=asset.covering_type,
|
||||
quantity=asset.quantity,
|
||||
install_date=asset.fitted_date,
|
||||
renewal_year=CalicoMapper._renewal_year(
|
||||
asset, survey_year
|
||||
),
|
||||
)
|
||||
],
|
||||
)
|
||||
)
|
||||
|
||||
return PropertyConditionSurvey(
|
||||
uprn=client_property_data.uprn,
|
||||
elements=elements,
|
||||
date=date(survey_year, 1, 1) if survey_year else date(2000, 1, 1),
|
||||
source=SOURCE,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _renewal_year(
|
||||
asset: CalicoAssetCondition, survey_year: Optional[int]
|
||||
) -> Optional[int]:
|
||||
if asset.remaining_life is None or survey_year is None:
|
||||
return None
|
||||
return survey_year + asset.remaining_life
|
||||
|
|
@ -1,8 +1,8 @@
|
|||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
from backend.condition.domain.aspect_type import AspectType
|
||||
from backend.condition.domain.element_type import ElementType
|
||||
from domain.condition.aspect_type import AspectType
|
||||
from domain.condition.element_type import ElementType
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
0
domain/condition/mapping/lbwf/__init__.py
Normal file
0
domain/condition/mapping/lbwf/__init__.py
Normal file
|
|
@ -1,6 +1,6 @@
|
|||
from backend.condition.domain.element_type import ElementType
|
||||
from backend.condition.domain.aspect_type import AspectType
|
||||
from backend.condition.domain.mapping.element_mapping import ElementMapping
|
||||
from domain.condition.element_type import ElementType
|
||||
from domain.condition.aspect_type import AspectType
|
||||
from domain.condition.mapping.element_mapping import ElementMapping
|
||||
|
||||
|
||||
LBWF_ELEMENT_MAP: dict[str, ElementMapping] = {
|
||||
|
|
@ -1,17 +1,17 @@
|
|||
from typing import Any, Dict, Optional, Tuple
|
||||
from datetime import date
|
||||
|
||||
from backend.condition.domain.aspect_condition import AspectCondition
|
||||
from backend.condition.domain.element import Element
|
||||
from backend.condition.domain.element_type import ElementType
|
||||
from backend.condition.domain.mapping.element_mapping import ElementMapping
|
||||
from backend.condition.domain.mapping.lbwf.lbwf_element_map import LBWF_ELEMENT_MAP
|
||||
from backend.condition.domain.mapping.mapper import Mapper
|
||||
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
|
||||
from backend.condition.parsing.records.lbwf.lbwf_asset_condition import (
|
||||
from domain.condition.aspect_condition import AspectCondition
|
||||
from domain.condition.element import Element
|
||||
from domain.condition.element_type import ElementType
|
||||
from domain.condition.mapping.element_mapping import ElementMapping
|
||||
from domain.condition.mapping.lbwf.lbwf_element_map import LBWF_ELEMENT_MAP
|
||||
from domain.condition.mapping.mapper import Mapper
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
from domain.condition.records.lbwf.lbwf_asset_condition import (
|
||||
LbwfAssetCondition,
|
||||
)
|
||||
from backend.condition.parsing.records.lbwf.lbwf_house import LbwfHouse
|
||||
from domain.condition.records.lbwf.lbwf_house import LbwfHouse
|
||||
from utils.logger import setup_logger
|
||||
|
||||
logger = setup_logger()
|
||||
|
|
@ -1,8 +1,8 @@
|
|||
from abc import ABC, abstractmethod
|
||||
from typing import Any, List, Optional
|
||||
|
||||
from backend.condition.domain.element import Element
|
||||
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
|
||||
from domain.condition.element import Element
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
|
||||
|
||||
class Mapper(ABC):
|
||||
0
domain/condition/mapping/peabody/__init__.py
Normal file
0
domain/condition/mapping/peabody/__init__.py
Normal file
|
|
@ -1,6 +1,6 @@
|
|||
from backend.condition.domain.aspect_type import AspectType
|
||||
from backend.condition.domain.element_type import ElementType
|
||||
from backend.condition.domain.mapping.element_mapping import ElementMapping
|
||||
from domain.condition.aspect_type import AspectType
|
||||
from domain.condition.element_type import ElementType
|
||||
from domain.condition.mapping.element_mapping import ElementMapping
|
||||
|
||||
|
||||
PEABODY_ELEMENT_MAP = {
|
||||
|
|
@ -1,19 +1,19 @@
|
|||
from typing import Any, Dict, Optional, Tuple
|
||||
from datetime import date
|
||||
|
||||
from backend.condition.domain.aspect_condition import AspectCondition
|
||||
from backend.condition.domain.element import Element
|
||||
from backend.condition.domain.element_type import ElementType
|
||||
from backend.condition.domain.mapping.element_mapping import ElementMapping
|
||||
from backend.condition.domain.mapping.peabody.peabody_element_map import (
|
||||
from domain.condition.aspect_condition import AspectCondition
|
||||
from domain.condition.element import Element
|
||||
from domain.condition.element_type import ElementType
|
||||
from domain.condition.mapping.element_mapping import ElementMapping
|
||||
from domain.condition.mapping.peabody.peabody_element_map import (
|
||||
PEABODY_ELEMENT_MAP,
|
||||
)
|
||||
from backend.condition.domain.mapping.mapper import Mapper
|
||||
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
|
||||
from backend.condition.parsing.records.peabody.peabody_asset_condition import (
|
||||
from domain.condition.mapping.mapper import Mapper
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
from domain.condition.records.peabody.peabody_asset_condition import (
|
||||
PeabodyAssetCondition,
|
||||
)
|
||||
from backend.condition.parsing.records.peabody.peabody_property import PeabodyProperty
|
||||
from domain.condition.records.peabody.peabody_property import PeabodyProperty
|
||||
from utils.logger import setup_logger
|
||||
|
||||
logger = setup_logger()
|
||||
|
|
@ -2,7 +2,7 @@ from dataclasses import dataclass
|
|||
from typing import List
|
||||
from datetime import date
|
||||
|
||||
from backend.condition.domain.element import Element
|
||||
from domain.condition.element import Element
|
||||
|
||||
|
||||
@dataclass
|
||||
0
domain/condition/records/__init__.py
Normal file
0
domain/condition/records/__init__.py
Normal file
0
domain/condition/records/calico/__init__.py
Normal file
0
domain/condition/records/calico/__init__.py
Normal file
20
domain/condition/records/calico/calico_asset_condition.py
Normal file
20
domain/condition/records/calico/calico_asset_condition.py
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from typing import Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class CalicoAssetCondition:
|
||||
"""One component row from a Calico stock-condition export.
|
||||
|
||||
Calico's current export carries a single component (``path == "Roof
|
||||
Covering"``); ``covering_type`` is the material (e.g. "Concrete Tiles") and
|
||||
``remaining_life`` is Calico's Decent-Homes-derived years-remaining figure.
|
||||
"""
|
||||
|
||||
asset_reference: int
|
||||
path: str
|
||||
covering_type: Optional[str] = None
|
||||
fitted_date: Optional[date] = None
|
||||
quantity: Optional[int] = None
|
||||
remaining_life: Optional[int] = None
|
||||
10
domain/condition/records/calico/calico_property.py
Normal file
10
domain/condition/records/calico/calico_property.py
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
from dataclasses import dataclass
|
||||
from typing import List
|
||||
|
||||
from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition
|
||||
|
||||
|
||||
@dataclass
|
||||
class CalicoProperty:
|
||||
uprn: int
|
||||
assets: List[CalicoAssetCondition]
|
||||
0
domain/condition/records/lbwf/__init__.py
Normal file
0
domain/condition/records/lbwf/__init__.py
Normal file
|
|
@ -1,7 +1,7 @@
|
|||
from dataclasses import dataclass
|
||||
from typing import List
|
||||
|
||||
from backend.condition.parsing.records.lbwf.lbwf_asset_condition import LbwfAssetCondition
|
||||
from domain.condition.records.lbwf.lbwf_asset_condition import LbwfAssetCondition
|
||||
|
||||
@dataclass
|
||||
class LbwfHouse:
|
||||
0
domain/condition/records/peabody/__init__.py
Normal file
0
domain/condition/records/peabody/__init__.py
Normal file
|
|
@ -1,7 +1,7 @@
|
|||
from dataclasses import dataclass
|
||||
from typing import List
|
||||
|
||||
from backend.condition.parsing.records.peabody.peabody_asset_condition import PeabodyAssetCondition
|
||||
from domain.condition.records.peabody.peabody_asset_condition import PeabodyAssetCondition
|
||||
|
||||
@dataclass
|
||||
class PeabodyProperty:
|
||||
0
infrastructure/condition/__init__.py
Normal file
0
infrastructure/condition/__init__.py
Normal file
0
infrastructure/condition/lookups/__init__.py
Normal file
0
infrastructure/condition/lookups/__init__.py
Normal file
|
|
@ -1,7 +1,7 @@
|
|||
import csv
|
||||
from io import TextIOWrapper
|
||||
from typing import BinaryIO, Dict, TextIO
|
||||
from backend.condition.lookups.uprn_lookup import UprnLookup
|
||||
from repositories.condition.uprn_lookup import UprnLookup
|
||||
|
||||
|
||||
class UprnLookupLocal(UprnLookup):
|
||||
|
|
@ -2,7 +2,7 @@ import csv
|
|||
from io import BytesIO, TextIOWrapper
|
||||
from typing import BinaryIO, Dict, TextIO
|
||||
|
||||
from backend.condition.lookups.uprn_lookup import UprnLookup
|
||||
from repositories.condition.uprn_lookup import UprnLookup
|
||||
from utils.s3 import read_io_from_s3
|
||||
|
||||
|
||||
0
infrastructure/condition/parsing/__init__.py
Normal file
0
infrastructure/condition/parsing/__init__.py
Normal file
114
infrastructure/condition/parsing/calico_parser.py
Normal file
114
infrastructure/condition/parsing/calico_parser.py
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
from collections import defaultdict
|
||||
from typing import Any, BinaryIO, Dict, List, Optional, Tuple
|
||||
|
||||
from openpyxl import load_workbook
|
||||
|
||||
from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition
|
||||
from domain.condition.records.calico.calico_property import CalicoProperty
|
||||
from infrastructure.condition.date_utils import normalise_date
|
||||
from infrastructure.condition.parsing.parser import Parser
|
||||
from repositories.condition.uprn_lookup import UprnLookup
|
||||
from utils.logger import setup_logger
|
||||
|
||||
logger = setup_logger()
|
||||
|
||||
|
||||
class CalicoParser(Parser):
|
||||
def __init__(self, uprn_lookup: UprnLookup) -> None:
|
||||
self._uprn_lookup = uprn_lookup
|
||||
self._blank_placeholder_count = 0
|
||||
self._unmatched_reference_count = 0
|
||||
|
||||
@property
|
||||
def blank_placeholder_count(self) -> int:
|
||||
"""Rows dropped on parse for a blank / N/A covering type."""
|
||||
return self._blank_placeholder_count
|
||||
|
||||
@property
|
||||
def unmatched_reference_count(self) -> int:
|
||||
"""Distinct asset references dropped for not resolving to a portfolio UPRN."""
|
||||
return self._unmatched_reference_count
|
||||
|
||||
def parse(self, file_stream: BinaryIO) -> List[CalicoProperty]:
|
||||
workbook = load_workbook(file_stream, data_only=True)
|
||||
sheet = workbook[workbook.sheetnames[0]]
|
||||
|
||||
rows = sheet.iter_rows(values_only=True)
|
||||
column_index = CalicoParser._column_indexes(next(rows))
|
||||
|
||||
assets = self._parse_assets(rows, column_index)
|
||||
return self._group_assets_into_properties(assets)
|
||||
|
||||
def _parse_assets(
|
||||
self, rows: Any, column_index: Dict[str, int]
|
||||
) -> List[CalicoAssetCondition]:
|
||||
assets: List[CalicoAssetCondition] = []
|
||||
for row in rows:
|
||||
covering_type = CalicoParser._clean_str(row[column_index["Type"]])
|
||||
if covering_type is None:
|
||||
# A blank covering type is a placeholder row (no roof-covering
|
||||
# record captured), not an observation -- drop on parse.
|
||||
self._blank_placeholder_count += 1
|
||||
continue
|
||||
assets.append(
|
||||
CalicoAssetCondition(
|
||||
asset_reference=int(row[column_index["Asset Reference"]]),
|
||||
path=CalicoParser._clean_str(row[column_index["Path"]]) or "",
|
||||
covering_type=covering_type,
|
||||
fitted_date=normalise_date(row[column_index["Fitted / Renewed Date"]]),
|
||||
quantity=CalicoParser._to_int(row[column_index["Quantity"]]),
|
||||
remaining_life=CalicoParser._to_int(
|
||||
row[column_index["Remaining Life"]]
|
||||
),
|
||||
)
|
||||
)
|
||||
return assets
|
||||
|
||||
def _group_assets_into_properties(
|
||||
self, assets: List[CalicoAssetCondition]
|
||||
) -> List[CalicoProperty]:
|
||||
ref_to_uprn: Dict[str, int] = self._uprn_lookup.get_property_ref_to_uprn_lookup()
|
||||
|
||||
assets_by_ref: Dict[int, List[CalicoAssetCondition]] = defaultdict(list)
|
||||
for asset in assets:
|
||||
assets_by_ref[asset.asset_reference].append(asset)
|
||||
|
||||
properties: List[CalicoProperty] = []
|
||||
for reference, grouped_assets in assets_by_ref.items():
|
||||
uprn = ref_to_uprn.get(str(reference))
|
||||
if uprn is None:
|
||||
self._unmatched_reference_count += 1
|
||||
logger.debug(
|
||||
f"[CalicoParser] No UPRN for asset reference {reference}; dropping"
|
||||
)
|
||||
continue
|
||||
properties.append(CalicoProperty(uprn=uprn, assets=grouped_assets))
|
||||
|
||||
return properties
|
||||
|
||||
@staticmethod
|
||||
def _column_indexes(headers: Tuple[object | None, ...]) -> Dict[str, int]:
|
||||
return {
|
||||
header: index
|
||||
for index, header in enumerate(headers)
|
||||
if isinstance(header, str)
|
||||
}
|
||||
|
||||
# Sentinels Calico lodges in the Type column for a placeholder row (no
|
||||
# roof-covering record captured), treated as "no covering".
|
||||
_BLANK_COVERING_SENTINELS = {"", "n/a", "na", "none"}
|
||||
|
||||
@staticmethod
|
||||
def _clean_str(value: object) -> Optional[str]:
|
||||
if not isinstance(value, str):
|
||||
return None
|
||||
stripped = value.strip()
|
||||
if stripped.lower() in CalicoParser._BLANK_COVERING_SENTINELS:
|
||||
return None
|
||||
return stripped
|
||||
|
||||
@staticmethod
|
||||
def _to_int(value: object) -> Optional[int]:
|
||||
if isinstance(value, (int, float)):
|
||||
return int(value)
|
||||
return None
|
||||
|
|
@ -2,12 +2,12 @@ from typing import BinaryIO, Any, Dict, Iterator, List, Optional, Tuple
|
|||
from openpyxl import Workbook, load_workbook
|
||||
from collections import defaultdict
|
||||
|
||||
from backend.condition.parsing.parser import Parser
|
||||
from backend.condition.parsing.records.lbwf.lbwf_asset_condition import (
|
||||
from infrastructure.condition.parsing.parser import Parser
|
||||
from domain.condition.records.lbwf.lbwf_asset_condition import (
|
||||
LbwfAssetCondition,
|
||||
)
|
||||
from backend.condition.parsing.records.lbwf.lbwf_house import LbwfHouse
|
||||
from backend.condition.utils.date_utils import normalise_date
|
||||
from domain.condition.records.lbwf.lbwf_house import LbwfHouse
|
||||
from infrastructure.condition.date_utils import normalise_date
|
||||
from utils.logger import setup_logger
|
||||
|
||||
logger = setup_logger()
|
||||
|
|
@ -4,12 +4,12 @@ from typing import Any, BinaryIO, Dict, List, Optional, Tuple, DefaultDict
|
|||
from openpyxl import Workbook, load_workbook
|
||||
from collections import defaultdict
|
||||
|
||||
from backend.condition.lookups.uprn_lookup import UprnLookup
|
||||
from backend.condition.parsing.parser import Parser
|
||||
from backend.condition.parsing.records.peabody.peabody_asset_condition import (
|
||||
from repositories.condition.uprn_lookup import UprnLookup
|
||||
from infrastructure.condition.parsing.parser import Parser
|
||||
from domain.condition.records.peabody.peabody_asset_condition import (
|
||||
PeabodyAssetCondition,
|
||||
)
|
||||
from backend.condition.parsing.records.peabody.peabody_property import PeabodyProperty
|
||||
from domain.condition.records.peabody.peabody_property import PeabodyProperty
|
||||
from utils.logger import setup_logger
|
||||
|
||||
logger = setup_logger()
|
||||
|
|
@ -9,8 +9,8 @@ from sqlalchemy import (
|
|||
)
|
||||
from sqlalchemy.orm import relationship
|
||||
|
||||
from backend.condition.domain.aspect_type import AspectType
|
||||
from backend.condition.domain.element_type import ElementType
|
||||
from domain.condition.aspect_type import AspectType
|
||||
from domain.condition.element_type import ElementType
|
||||
|
||||
from backend.app.db.base import Base
|
||||
|
||||
0
repositories/condition/__init__.py
Normal file
0
repositories/condition/__init__.py
Normal file
|
|
@ -3,12 +3,12 @@ from typing import List, Optional
|
|||
from sqlmodel import Session
|
||||
|
||||
from utils.logger import setup_logger
|
||||
from backend.app.db.models.condition import (
|
||||
from infrastructure.postgres.condition_tables import (
|
||||
AspectConditionModel,
|
||||
ElementModel,
|
||||
PropertyConditionSurveyModel,
|
||||
)
|
||||
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
from backend.app.db.connection import db_session
|
||||
|
||||
logger = setup_logger()
|
||||
13
repositories/condition/condition_writer.py
Normal file
13
repositories/condition/condition_writer.py
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
from typing import List, Protocol
|
||||
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
|
||||
|
||||
class ConditionWriter(Protocol):
|
||||
"""Persists condition surveys. ``ConditionPostgres`` is the production
|
||||
implementation; a load orchestrator depends on this port so it can be
|
||||
exercised without a database."""
|
||||
|
||||
def bulk_insert_surveys(
|
||||
self, surveys: List[PropertyConditionSurvey]
|
||||
) -> None: ...
|
||||
19
repositories/condition/property_reference_reader.py
Normal file
19
repositories/condition/property_reference_reader.py
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PropertyReference:
|
||||
"""A property's landlord reference and resolved UPRN, as held in the
|
||||
``property`` table. ``uprn`` is nullable — a property whose UPRN has not been
|
||||
resolved yet cannot receive condition rows."""
|
||||
|
||||
landlord_property_id: Optional[str]
|
||||
uprn: Optional[int]
|
||||
|
||||
|
||||
class PropertyReferenceReader(ABC):
|
||||
@abstractmethod
|
||||
def references_for_portfolio(self, portfolio_id: int) -> List[PropertyReference]:
|
||||
...
|
||||
28
repositories/condition/property_reference_reader_postgres.py
Normal file
28
repositories/condition/property_reference_reader_postgres.py
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
from typing import List
|
||||
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from infrastructure.postgres.property_table import PropertyRow
|
||||
from repositories.condition.property_reference_reader import (
|
||||
PropertyReference,
|
||||
PropertyReferenceReader,
|
||||
)
|
||||
|
||||
|
||||
class PropertyReferenceReaderPostgres(PropertyReferenceReader):
|
||||
"""Reads landlord-reference -> UPRN pairs from the ``property`` table for a
|
||||
portfolio. The reference key is ``landlord_property_id`` (ADR-0064)."""
|
||||
|
||||
def __init__(self, session: Session) -> None:
|
||||
self._session = session
|
||||
|
||||
def references_for_portfolio(self, portfolio_id: int) -> List[PropertyReference]:
|
||||
rows = self._session.exec(
|
||||
select(PropertyRow.landlord_property_id, PropertyRow.uprn).where(
|
||||
PropertyRow.portfolio_id == portfolio_id
|
||||
)
|
||||
).all()
|
||||
return [
|
||||
PropertyReference(landlord_property_id=landlord_property_id, uprn=uprn)
|
||||
for landlord_property_id, uprn in rows
|
||||
]
|
||||
33
repositories/condition/property_uprn_lookup.py
Normal file
33
repositories/condition/property_uprn_lookup.py
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
from typing import Dict, List
|
||||
|
||||
from repositories.condition.property_reference_reader import PropertyReferenceReader
|
||||
from repositories.condition.uprn_lookup import UprnLookup
|
||||
|
||||
|
||||
class PropertyUprnLookup(UprnLookup):
|
||||
"""Resolves a landlord property reference to a UPRN from the ``property``
|
||||
table, scoped to one portfolio (ADR-0064)."""
|
||||
|
||||
def __init__(
|
||||
self, reader: PropertyReferenceReader, portfolio_id: int
|
||||
) -> None:
|
||||
self._reader = reader
|
||||
self._portfolio_id = portfolio_id
|
||||
|
||||
def get_property_ref_to_uprn_lookup(self) -> Dict[str, int]:
|
||||
references = self._reader.references_for_portfolio(self._portfolio_id)
|
||||
return {
|
||||
ref.landlord_property_id: ref.uprn
|
||||
for ref in references
|
||||
if ref.landlord_property_id is not None and ref.uprn is not None
|
||||
}
|
||||
|
||||
def null_uprn_references(self) -> List[str]:
|
||||
"""Landlord references present in the portfolio whose UPRN is null —
|
||||
surfaced so a load can report them (none expected; ADR-0064)."""
|
||||
references = self._reader.references_for_portfolio(self._portfolio_id)
|
||||
return [
|
||||
ref.landlord_property_id
|
||||
for ref in references
|
||||
if ref.landlord_property_id is not None and ref.uprn is None
|
||||
]
|
||||
0
tests/condition/__init__.py
Normal file
0
tests/condition/__init__.py
Normal file
|
|
@ -1,5 +1,5 @@
|
|||
from backend.app.db.models.condition import PropertyConditionSurveyModel
|
||||
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
|
||||
from infrastructure.postgres.condition_tables import PropertyConditionSurveyModel
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
|
||||
|
||||
class CustomAsserts:
|
||||
0
tests/condition/lookups/__init__.py
Normal file
0
tests/condition/lookups/__init__.py
Normal file
49
tests/condition/lookups/test_property_uprn_lookup.py
Normal file
49
tests/condition/lookups/test_property_uprn_lookup.py
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
from typing import List
|
||||
|
||||
from repositories.condition.property_reference_reader import (
|
||||
PropertyReference,
|
||||
PropertyReferenceReader,
|
||||
)
|
||||
from repositories.condition.property_uprn_lookup import PropertyUprnLookup
|
||||
|
||||
|
||||
class FakePropertyReferenceReader(PropertyReferenceReader):
|
||||
def __init__(self, references: List[PropertyReference]) -> None:
|
||||
self._references = references
|
||||
|
||||
def references_for_portfolio(self, portfolio_id: int) -> List[PropertyReference]:
|
||||
return self._references
|
||||
|
||||
|
||||
def test_property_uprn_lookup_excludes_rows_with_no_uprn():
|
||||
# Arrange
|
||||
reader = FakePropertyReferenceReader(
|
||||
[
|
||||
PropertyReference(landlord_property_id="443", uprn=100093305101),
|
||||
PropertyReference(landlord_property_id="999", uprn=None),
|
||||
]
|
||||
)
|
||||
lookup = PropertyUprnLookup(reader, portfolio_id=824)
|
||||
|
||||
# Act
|
||||
mapping = lookup.get_property_ref_to_uprn_lookup()
|
||||
|
||||
# Assert
|
||||
assert mapping == {"443": 100093305101}
|
||||
|
||||
|
||||
def test_property_uprn_lookup_reports_references_with_null_uprn():
|
||||
# Arrange
|
||||
reader = FakePropertyReferenceReader(
|
||||
[
|
||||
PropertyReference(landlord_property_id="443", uprn=100093305101),
|
||||
PropertyReference(landlord_property_id="999", uprn=None),
|
||||
]
|
||||
)
|
||||
lookup = PropertyUprnLookup(reader, portfolio_id=824)
|
||||
|
||||
# Act
|
||||
null_references = lookup.null_uprn_references()
|
||||
|
||||
# Assert
|
||||
assert null_references == ["999"]
|
||||
|
|
@ -2,7 +2,7 @@ import pytest
|
|||
from typing import Dict
|
||||
from tempfile import NamedTemporaryFile
|
||||
|
||||
from backend.condition.lookups.uprn_lookup_csv import UprnLookupLocal
|
||||
from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
0
tests/condition/mapping/__init__.py
Normal file
0
tests/condition/mapping/__init__.py
Normal file
36
tests/condition/mapping/test_calico_mapper.py
Normal file
36
tests/condition/mapping/test_calico_mapper.py
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
from datetime import date
|
||||
|
||||
from domain.condition.aspect_type import AspectType
|
||||
from domain.condition.element_type import ElementType
|
||||
from domain.condition.mapping.calico.calico_mapper import CalicoMapper
|
||||
from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition
|
||||
from domain.condition.records.calico.calico_property import CalicoProperty
|
||||
|
||||
|
||||
def test_calico_mapper_sets_renewal_year_to_export_year_plus_remaining_life():
|
||||
# Arrange
|
||||
calico_property = CalicoProperty(
|
||||
uprn=100093305101,
|
||||
assets=[
|
||||
CalicoAssetCondition(
|
||||
asset_reference=443,
|
||||
path="Roof Covering",
|
||||
covering_type="Concrete Tiles",
|
||||
fitted_date=date(1998, 4, 1),
|
||||
quantity=55,
|
||||
remaining_life=36,
|
||||
)
|
||||
],
|
||||
)
|
||||
export_year = 2026
|
||||
|
||||
# Act
|
||||
survey = CalicoMapper().map_asset_conditions_for_property(
|
||||
calico_property, export_year
|
||||
)
|
||||
|
||||
# Assert
|
||||
roof_material = survey.elements[0].aspect_conditions[0]
|
||||
assert roof_material.aspect_type is AspectType.MATERIAL
|
||||
assert survey.elements[0].element_type is ElementType.ROOF
|
||||
assert roof_material.renewal_year == export_year + 36
|
||||
|
|
@ -1,16 +1,16 @@
|
|||
from datetime import date
|
||||
|
||||
from backend.condition.domain.aspect_condition import AspectCondition
|
||||
from backend.condition.domain.aspect_type import AspectType
|
||||
from backend.condition.domain.element_type import ElementType
|
||||
from backend.condition.domain.mapping.lbwf.lbwf_mapper import LbwfMapper
|
||||
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
|
||||
from backend.condition.parsing.records.lbwf.lbwf_house import LbwfHouse
|
||||
from backend.condition.parsing.records.lbwf.lbwf_asset_condition import (
|
||||
from domain.condition.aspect_condition import AspectCondition
|
||||
from domain.condition.aspect_type import AspectType
|
||||
from domain.condition.element_type import ElementType
|
||||
from domain.condition.mapping.lbwf.lbwf_mapper import LbwfMapper
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
from domain.condition.records.lbwf.lbwf_house import LbwfHouse
|
||||
from domain.condition.records.lbwf.lbwf_asset_condition import (
|
||||
LbwfAssetCondition,
|
||||
)
|
||||
from backend.condition.domain.element import Element
|
||||
from backend.condition.tests.custom_asserts import CustomAsserts
|
||||
from domain.condition.element import Element
|
||||
from tests.condition.custom_asserts import CustomAsserts
|
||||
|
||||
|
||||
def test_lbwf_mapper_maps_house():
|
||||
|
|
@ -1,16 +1,16 @@
|
|||
from datetime import datetime, date
|
||||
|
||||
from backend.condition.domain.aspect_condition import AspectCondition
|
||||
from backend.condition.domain.aspect_type import AspectType
|
||||
from backend.condition.domain.element_type import ElementType
|
||||
from backend.condition.domain.mapping.peabody.peabody_mapper import PeabodyMapper
|
||||
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
|
||||
from backend.condition.parsing.records.peabody.peabody_asset_condition import (
|
||||
from domain.condition.aspect_condition import AspectCondition
|
||||
from domain.condition.aspect_type import AspectType
|
||||
from domain.condition.element_type import ElementType
|
||||
from domain.condition.mapping.peabody.peabody_mapper import PeabodyMapper
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
from domain.condition.records.peabody.peabody_asset_condition import (
|
||||
PeabodyAssetCondition,
|
||||
)
|
||||
from backend.condition.parsing.records.peabody.peabody_property import PeabodyProperty
|
||||
from backend.condition.domain.element import Element
|
||||
from backend.condition.tests.custom_asserts import CustomAsserts
|
||||
from domain.condition.records.peabody.peabody_property import PeabodyProperty
|
||||
from domain.condition.element import Element
|
||||
from tests.condition.custom_asserts import CustomAsserts
|
||||
|
||||
|
||||
def test_peabody_mapper_maps_property():
|
||||
0
tests/condition/parsing/__init__.py
Normal file
0
tests/condition/parsing/__init__.py
Normal file
106
tests/condition/parsing/test_calico_parser.py
Normal file
106
tests/condition/parsing/test_calico_parser.py
Normal file
|
|
@ -0,0 +1,106 @@
|
|||
from datetime import date
|
||||
from io import BytesIO
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from openpyxl import Workbook
|
||||
|
||||
from infrastructure.condition.parsing.calico_parser import CalicoParser
|
||||
from repositories.condition.uprn_lookup import UprnLookup
|
||||
|
||||
HEADERS = [
|
||||
"Asset Reference",
|
||||
"Asset Type",
|
||||
"Address",
|
||||
"Path",
|
||||
"Type",
|
||||
"Fitted / Renewed Date",
|
||||
"Quantity",
|
||||
"Remaining Life",
|
||||
]
|
||||
|
||||
|
||||
class FakeUprnLookup(UprnLookup):
|
||||
def __init__(self, mapping: Dict[str, int]) -> None:
|
||||
self._mapping = mapping
|
||||
|
||||
def get_property_ref_to_uprn_lookup(self) -> Dict[str, int]:
|
||||
return self._mapping
|
||||
|
||||
|
||||
def _workbook(rows: List[List[object]]) -> BytesIO:
|
||||
wb = Workbook()
|
||||
ws = wb.active
|
||||
assert ws is not None
|
||||
ws.append(HEADERS)
|
||||
for row in rows:
|
||||
ws.append(row)
|
||||
buffer = BytesIO()
|
||||
wb.save(buffer)
|
||||
buffer.seek(0)
|
||||
return buffer
|
||||
|
||||
|
||||
def test_calico_parser_counts_blank_and_unmatched_drops():
|
||||
# Arrange -- one loaded, one blank placeholder, one unmatched (not in lookup)
|
||||
workbook = _workbook(
|
||||
[
|
||||
[443, "HOUSE", "31 Venice Avenue", "Roof Covering", "Concrete Tiles",
|
||||
date(1998, 4, 1), 55, 36],
|
||||
[438, "FLAT", "137 Brownhill Avenue", "Roof Covering", "N/A",
|
||||
date(2000, 4, 1), 0, 973],
|
||||
[777, "HOUSE", "8 Marine Avenue", "Roof Covering", "Natural Slate",
|
||||
date(2010, 4, 1), 60, 50],
|
||||
]
|
||||
)
|
||||
parser = CalicoParser(FakeUprnLookup({"443": 100093305101}))
|
||||
|
||||
# Act
|
||||
parser.parse(workbook)
|
||||
|
||||
# Assert
|
||||
assert parser.blank_placeholder_count == 1
|
||||
assert parser.unmatched_reference_count == 1
|
||||
|
||||
|
||||
def test_calico_parser_drops_na_sentinel_covering_rows():
|
||||
# Arrange -- Calico writes the literal string "N/A" for a placeholder row
|
||||
# (no roof-covering record captured), not an empty cell.
|
||||
workbook = _workbook(
|
||||
[
|
||||
[443, "HOUSE", "31 Venice Avenue", "Roof Covering", "Concrete Tiles",
|
||||
date(1998, 4, 1), 55, 36],
|
||||
[438, "FLAT", "137 Brownhill Avenue", "Roof Covering", "N/A",
|
||||
date(2000, 4, 1), 0, 973],
|
||||
]
|
||||
)
|
||||
parser = CalicoParser(FakeUprnLookup({"443": 100093305101, "438": 100093388053}))
|
||||
|
||||
# Act
|
||||
properties = parser.parse(workbook)
|
||||
|
||||
# Assert
|
||||
references = [asset.asset_reference for prop in properties for asset in prop.assets]
|
||||
assert references == [443]
|
||||
|
||||
|
||||
def test_calico_parser_drops_rows_with_no_covering_type():
|
||||
# Arrange
|
||||
workbook = _workbook(
|
||||
[
|
||||
[443, "HOUSE", "31 Venice Avenue", "Roof Covering", "Concrete Tiles",
|
||||
date(1998, 4, 1), 55, 36],
|
||||
[438, "FLAT", "137 Brownhill Avenue", "Roof Covering", None,
|
||||
date(2000, 4, 1), 0, 973],
|
||||
]
|
||||
)
|
||||
parser = CalicoParser(FakeUprnLookup({"443": 100093305101, "438": 100093388053}))
|
||||
|
||||
# Act
|
||||
properties = parser.parse(workbook)
|
||||
|
||||
# Assert
|
||||
all_assets = [asset for prop in properties for asset in prop.assets]
|
||||
references: List[int] = [asset.asset_reference for asset in all_assets]
|
||||
coverings: List[Optional[str]] = [asset.covering_type for asset in all_assets]
|
||||
assert references == [443]
|
||||
assert coverings == ["Concrete Tiles"]
|
||||
|
|
@ -4,9 +4,9 @@ from io import BytesIO
|
|||
from openpyxl import Workbook
|
||||
from datetime import datetime
|
||||
|
||||
from backend.condition.parsing.lbwf_parser import LbwfParser
|
||||
from backend.condition.parsing.records.lbwf.lbwf_asset_condition import LbwfAssetCondition
|
||||
from backend.condition.parsing.records.lbwf.lbwf_house import LbwfHouse
|
||||
from infrastructure.condition.parsing.lbwf_parser import LbwfParser
|
||||
from domain.condition.records.lbwf.lbwf_asset_condition import LbwfAssetCondition
|
||||
from domain.condition.records.lbwf.lbwf_house import LbwfHouse
|
||||
|
||||
@pytest.fixture
|
||||
def lbwf_homes_xlsx_bytes() -> BytesIO:
|
||||
51
tests/condition/parsing/test_parsing_factory.py
Normal file
51
tests/condition/parsing/test_parsing_factory.py
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
import pytest
|
||||
|
||||
from applications.condition.condition_trigger_request import ConditionFileType
|
||||
from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal
|
||||
from applications.condition.factory import select_parser, select_mapper
|
||||
|
||||
|
||||
def test_selects_lbwf_parser():
|
||||
# arrange
|
||||
file_type = ConditionFileType.LBWF
|
||||
expected_class_name = "LbwfParser"
|
||||
|
||||
# act
|
||||
actual_class_name = select_parser(file_type).__class__.__name__
|
||||
|
||||
# assert
|
||||
assert expected_class_name == actual_class_name
|
||||
|
||||
|
||||
def test_selects_peabody_parser():
|
||||
# arrange
|
||||
file_type = ConditionFileType.Peabody
|
||||
expected_class_name = "PeabodyParser"
|
||||
uprn_lookup = UprnLookupLocal(csv_path="test")
|
||||
|
||||
# act
|
||||
actual_class_name = select_parser(file_type, uprn_lookup).__class__.__name__
|
||||
|
||||
# assert
|
||||
assert expected_class_name == actual_class_name
|
||||
|
||||
|
||||
def test_selects_calico_parser():
|
||||
# arrange
|
||||
uprn_lookup = UprnLookupLocal(csv_path="test")
|
||||
|
||||
# act
|
||||
actual_class_name = select_parser(
|
||||
ConditionFileType.Calico, uprn_lookup
|
||||
).__class__.__name__
|
||||
|
||||
# assert
|
||||
assert actual_class_name == "CalicoParser"
|
||||
|
||||
|
||||
def test_selects_calico_mapper():
|
||||
# act
|
||||
actual_class_name = select_mapper(ConditionFileType.Calico).__class__.__name__
|
||||
|
||||
# assert
|
||||
assert actual_class_name == "CalicoMapper"
|
||||
|
|
@ -5,12 +5,12 @@ from io import BytesIO
|
|||
from openpyxl import Workbook
|
||||
from datetime import datetime
|
||||
|
||||
from backend.condition.lookups.uprn_lookup_csv import UprnLookupLocal
|
||||
from backend.condition.parsing.peabody_parser import PeabodyParser
|
||||
from backend.condition.parsing.records.peabody.peabody_asset_condition import (
|
||||
from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal
|
||||
from infrastructure.condition.parsing.peabody_parser import PeabodyParser
|
||||
from domain.condition.records.peabody.peabody_asset_condition import (
|
||||
PeabodyAssetCondition,
|
||||
)
|
||||
from backend.condition.parsing.records.peabody.peabody_property import PeabodyProperty
|
||||
from domain.condition.records.peabody.peabody_property import PeabodyProperty
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
0
tests/condition/persistence/__init__.py
Normal file
0
tests/condition/persistence/__init__.py
Normal file
|
|
@ -1,14 +1,14 @@
|
|||
import pytest
|
||||
from datetime import date
|
||||
|
||||
from backend.condition.persistence.condition_postgres import ConditionPostgres
|
||||
from backend.condition.domain.property_condition_survey import PropertyConditionSurvey
|
||||
from backend.condition.domain.element import Element
|
||||
from backend.condition.domain.element_type import ElementType
|
||||
from backend.condition.domain.aspect_condition import AspectCondition
|
||||
from backend.condition.domain.aspect_type import AspectType
|
||||
from backend.app.db.models.condition import PropertyConditionSurveyModel
|
||||
from backend.condition.tests.custom_asserts import CustomAsserts
|
||||
from repositories.condition.condition_postgres import ConditionPostgres
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
from domain.condition.element import Element
|
||||
from domain.condition.element_type import ElementType
|
||||
from domain.condition.aspect_condition import AspectCondition
|
||||
from domain.condition.aspect_type import AspectType
|
||||
from infrastructure.postgres.condition_tables import PropertyConditionSurveyModel
|
||||
from tests.condition.custom_asserts import CustomAsserts
|
||||
|
||||
|
||||
def test_map_survey_to_model() -> None:
|
||||
88
tests/condition/test_calico_load.py
Normal file
88
tests/condition/test_calico_load.py
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
from datetime import date
|
||||
from io import BytesIO
|
||||
from typing import List
|
||||
|
||||
from openpyxl import Workbook
|
||||
|
||||
from applications.condition.calico_load import run_calico_load
|
||||
from applications.condition.load_report import LoadReport
|
||||
from domain.condition.property_condition_survey import PropertyConditionSurvey
|
||||
from repositories.condition.property_reference_reader import (
|
||||
PropertyReference,
|
||||
PropertyReferenceReader,
|
||||
)
|
||||
from repositories.condition.property_uprn_lookup import PropertyUprnLookup
|
||||
|
||||
HEADERS = [
|
||||
"Asset Reference",
|
||||
"Asset Type",
|
||||
"Address",
|
||||
"Path",
|
||||
"Type",
|
||||
"Fitted / Renewed Date",
|
||||
"Quantity",
|
||||
"Remaining Life",
|
||||
]
|
||||
|
||||
|
||||
class FakePropertyReferenceReader(PropertyReferenceReader):
|
||||
def __init__(self, references: List[PropertyReference]) -> None:
|
||||
self._references = references
|
||||
|
||||
def references_for_portfolio(self, portfolio_id: int) -> List[PropertyReference]:
|
||||
return self._references
|
||||
|
||||
|
||||
class FakeConditionWriter:
|
||||
def __init__(self) -> None:
|
||||
self.surveys: List[PropertyConditionSurvey] = []
|
||||
|
||||
def bulk_insert_surveys(self, surveys: List[PropertyConditionSurvey]) -> None:
|
||||
self.surveys.extend(surveys)
|
||||
|
||||
|
||||
def _workbook(rows: List[List[object]]) -> BytesIO:
|
||||
wb = Workbook()
|
||||
ws = wb.active
|
||||
assert ws is not None
|
||||
ws.append(HEADERS)
|
||||
for row in rows:
|
||||
ws.append(row)
|
||||
buffer = BytesIO()
|
||||
wb.save(buffer)
|
||||
buffer.seek(0)
|
||||
return buffer
|
||||
|
||||
|
||||
def test_run_calico_load_persists_surveys_and_reports_reconciliation():
|
||||
# Arrange -- one loaded (443), one blank placeholder (438), one unmatched (777)
|
||||
workbook = _workbook(
|
||||
[
|
||||
[443, "HOUSE", "31 Venice Avenue", "Roof Covering", "Concrete Tiles",
|
||||
date(1998, 4, 1), 55, 36],
|
||||
[438, "FLAT", "137 Brownhill Avenue", "Roof Covering", "N/A",
|
||||
date(2000, 4, 1), 0, 973],
|
||||
[777, "HOUSE", "8 Marine Avenue", "Roof Covering", "Natural Slate",
|
||||
date(2010, 4, 1), 60, 50],
|
||||
]
|
||||
)
|
||||
reader = FakePropertyReferenceReader(
|
||||
[
|
||||
PropertyReference(landlord_property_id="443", uprn=100093305101),
|
||||
PropertyReference(landlord_property_id="999", uprn=None),
|
||||
]
|
||||
)
|
||||
lookup = PropertyUprnLookup(reader, portfolio_id=824)
|
||||
writer = FakeConditionWriter()
|
||||
|
||||
# Act
|
||||
report = run_calico_load(workbook, lookup, export_year=2026, writer=writer)
|
||||
|
||||
# Assert
|
||||
assert report == LoadReport(
|
||||
loaded=1, unmatched_to_portfolio=1, null_uprn=1, blank_placeholder=1
|
||||
)
|
||||
assert len(writer.surveys) == 1
|
||||
persisted = writer.surveys[0]
|
||||
assert persisted.uprn == 100093305101
|
||||
assert persisted.elements[0].aspect_conditions[0].renewal_year == 2026 + 36
|
||||
38
tests/condition/test_load_report.py
Normal file
38
tests/condition/test_load_report.py
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
from applications.condition.load_report import LoadReport
|
||||
from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition
|
||||
from domain.condition.records.calico.calico_property import CalicoProperty
|
||||
|
||||
|
||||
def _property(uprn: int, reference: int, covering: str) -> CalicoProperty:
|
||||
return CalicoProperty(
|
||||
uprn=uprn,
|
||||
assets=[
|
||||
CalicoAssetCondition(
|
||||
asset_reference=reference, path="Roof Covering", covering_type=covering
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def test_load_report_summarises_the_four_reconciliation_counts():
|
||||
# Arrange
|
||||
properties = [
|
||||
_property(100093305101, 443, "Concrete Tiles"),
|
||||
_property(100093388053, 444, "Natural Slate"),
|
||||
]
|
||||
|
||||
# Act
|
||||
report = LoadReport.from_calico_load(
|
||||
properties=properties,
|
||||
blank_placeholder_count=1239,
|
||||
unmatched_reference_count=5,
|
||||
null_uprn_references=["999", "888"],
|
||||
)
|
||||
|
||||
# Assert
|
||||
assert report == LoadReport(
|
||||
loaded=2,
|
||||
unmatched_to_portfolio=5,
|
||||
null_uprn=2,
|
||||
blank_placeholder=1239,
|
||||
)
|
||||
Loading…
Add table
Reference in a new issue