From 89602eb8144f9aaa19cba835cabd36578bb86c58 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 14:34:22 +0000 Subject: [PATCH 01/26] Record stock-condition data as engine-isolated, FE-consumed (ADR-0064) Adds the Stock Condition Data / Component Renewal Year / Roof Replacement Flag glossary terms and ADR-0064, capturing that condition data is ingested producer-only and never read by the modelling engine. Co-Authored-By: Claude Opus 4.8 (1M context) --- CONTEXT.md | 12 ++ ...data-is-engine-isolated-and-fe-consumed.md | 108 ++++++++++++++++++ 2 files changed, 120 insertions(+) create mode 100644 docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md diff --git a/CONTEXT.md b/CONTEXT.md index 047087e97..3a7489b29 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -121,6 +121,18 @@ _Avoid_: energy assessment, site survey, field survey, Domna survey, Hestia surv Property data supplied by a landlord that may correct or supplement the public EPC for a single Property; triggers Rebaselining when applied; not applicable when Site Notes are present. _Avoid_: patches (deprecated), corrections, manual EPC, edits +**Stock Condition Data**: +Landlord-supplied per-**component** lifecycle records — one row per element (e.g. a dwelling's **roof covering**) carrying its material/type, fitted-or-renewed date, quantity, and a **Remaining Life**. It answers *"when is this component due for renewal?"*, not *"what is the dwelling's SAP picture?"* — so unlike **Site Notes** and **Landlord Overrides** it is **not read by the modelling engine** and never enters `EpcPropertyData`, Rebaselining, or a Recommendation (ADR-0064). Sourced from a landlord's stock-condition survey (asset-management data); ingested by this repo purely as a **producer** and persisted keyed by UPRN as `PropertyConditionSurvey → Element → AspectCondition`. The consumer is the Next.js/Drizzle front-end, which reads the tables directly (the DB schema is the contract — the `element_type` / `aspect_type` PG enums are an FE-owned surface, cf. [[main-heating-system-pgenum-is-fe-owned]]) to display component data against a Property and to raise the **Roof Replacement Flag**. **Remaining Life** derives from the component's **Decent Homes Standard** expected lifetime (`fitted date + standard lifespan − report date`), so it is trusted as given, not recomputed here. +_Avoid_: condition survey (ambiguous with **Site Notes**), asset data, patches; treating it as a **Landlord Override** or fabric signal (the engine never reads it — this iteration) + +**Component Renewal Year**: +The absolute calendar year a **Stock Condition Data** component is due for replacement — stored as `renewal_year` on the `AspectCondition`. Computed at load as **`export_year + Remaining Life`** (the export/report date the landlord's file was generated, not `now()`), so it is stable across re-runs — an absolute year does not drift, whereas a stored "years remaining" would. The fitted/renewed date is retained separately as `install_date`. +_Avoid_: remaining life (that is the input; the stored value is an absolute year), replacement date, end-of-life date + +**Roof Replacement Flag**: +A **front-end-owned** presentation signal raised on a **Solar PV Recommendation** when a Property's roof-covering **Component Renewal Year** falls within (or near) the measure horizon — warning a human that installing solar onto a soon-to-be-replaced roof is unwise. It is **not** computed in this repo and does **not** change what the engine recommends (the optimiser's solar Option is unchanged): the Next.js/Drizzle app joins **Stock Condition Data** to the recommendation by UPRN and applies the "due soon" threshold itself (ADR-0064). A future engine upgrade that *acts* on this (bundling a re-roof, deferring solar) is explicitly out of scope for this iteration. +_Avoid_: roof warning (unspecific), re-roof recommendation (there is no engine measure yet), solar suppression + **Landlord-Description Classification**: Resolving a **Landlord Description** (unbounded free-text a landlord supplies for one component — "CWI" / "Cav filled" / "cavity insulated" all name one thing) onto a **Recognised Internal Description** via an LLM classifier, persisted in the `landlord_*_overrides` table (`source=classifier`) as a reviewed cache. Four vocabularies are kept **distinct** and must not be conflated: a **Landlord Description** (unbounded input); a **Recognised Internal Description** (the closed target taxonomy — e.g. a `MainHeatingSystemType` archetype — each binding to a Simulation Overlay); a **Lodged Description** (the gov-EPC `main_heating[].description` rendering, e.g. "Room heaters, electric" — only an example of which system *types* occur, never a map key); and the **SAP main heating code** (Table 4a/4b, what the calculator consumes). The classifier maps Landlord → Recognised Internal → SAP code. When it cannot confidently place the text it emits **`None`** (no overlay → the lodged EPC stands, surfaced to the user as "no suitable match"), **never the nearest wrong archetype** — the target taxonomy must be complete enough that a real system always has a correct home, so the classifier never overflows into a garbage-drawer archetype (ADR-0041). diff --git a/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md b/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md new file mode 100644 index 000000000..43fdaf102 --- /dev/null +++ b/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md @@ -0,0 +1,108 @@ +--- +status: accepted +--- + +# Stock condition data is a producer-only, engine-isolated dataset; the roof-replacement warning on solar is a front-end overlay + +We are ingesting a landlord's (Calico) **stock-condition survey** — for now, one +component: **roof covering**, ~5,346 rows of `Asset Reference, Type (material), +Fitted/Renewed Date, Quantity, Remaining Life`. The temptation is to treat any +landlord-supplied component fact like a **Landlord Override** and fold it into the +`EpcPropertyData` picture. That would be wrong here, and the whole design turns on +saying why. + +**Stock Condition Data answers a different question from the modelling picture.** +A **Landlord Override** ("this wall is solid brick") and **Site Notes** (a full +Domna survey) both correct *what the dwelling is* so the SAP score is right — the +engine reads them. Stock Condition Data answers *when a component is due for +renewal*: the roof-covering material plus a **Remaining Life** derived from the +component's **Decent Homes Standard** expected lifetime. It is asset-management +data, not a fabric signal. Conflating the two would drag component-lifecycle rows +into `EpcPropertyData`, Rebaselining, and the Recommendation model for no scoring +benefit and real coupling cost. + +**The consumer is the product, not the engine.** The value is: (1) show component +data against a Property in the front-end, and (2) warn a human, on a **Solar PV +Recommendation**, when the roof is near end-of-life — because installing solar +onto a roof due for replacement is a poor sequence. That warning is a *human +information* signal, not an optimiser input: the engine still recommends solar +unchanged; a badge says "mind the roof". The cross-reference (roof renewal year ↔ +solar recommendation) and the "due soon" threshold live in the **Next.js/Drizzle +front-end**, which reads the condition tables directly. This repo is purely a +**producer** into a schema the front-end consumes — so the DB schema (the +`property_condition_survey` / `element` / `aspect_condition` tables and their +`element_type` / `aspect_type` PG enums) is an **FE-owned contract**, the same +pattern as `main_heating_system`. + +Decided in a grill-with-docs session with Khalim, 2026-07-13. + +## Decision + +**Stock Condition Data is ingested producer-only and is engine-isolated. The +modelling engine never reads it (this iteration); the roof-replacement warning on +solar is a front-end presentation overlay, not an engine output.** + +- **Engine isolation.** Condition rows are written only to + `PropertyConditionSurvey → Element → AspectCondition`, keyed by UPRN. They never + enter `EpcPropertyData`, Rebaselining, Bill Derivation, or a Recommendation. The + optimiser's Solar PV Option is unchanged whether or not the roof is near + end-of-life. A future engine upgrade that *acts* on condition data (bundling a + re-roof, deferring solar) is a separate, later decision — explicitly out of + scope now. + +- **The flag lives above modelling.** The **Roof Replacement Flag** is computed by + the Next.js/Drizzle app, which joins condition data to the solar recommendation + by UPRN and owns the "due soon" threshold. This repo neither computes the flag + nor exposes a read API for it — it only writes the tables the FE reads. The DB + schema is therefore a shared FE contract; schema changes need a companion + Drizzle view. + +- **UPRN resolution from the property table, not a CSV.** Calico's `Asset + Reference` maps to `property.landlord_property_id` scoped to `portfolio_id = + 824`, which yields the `uprn`. The lookup is a Postgres-backed `UprnLookup` + reading the property table — fulfilling the existing "replace CSV with postgres" + TODO — not an exported file. A key-alignment test (Asset Reference == stored + `landlord_property_id`) is pinned during TDD implementation. + +- **Drop-and-report reconciliation; nothing vanishes silently.** Rows that do not + resolve to a portfolio-824 UPRN are dropped and counted; rows whose resolved + `property.uprn` is null are dropped **and reported** (none expected — we want to + see any that are); the ~1,239 blank-`Type` placeholder rows (Quantity 0, + sentinel date, nonsense Remaining Life) are dropped on parse. Every load emits a + reconciliation report: `loaded / unmatched-to-portfolio / null-uprn / + blank-placeholder`. + +- **Renewal year is anchored to the export date, not `now()`.** `Remaining Life` + is years-from-report; the stored **Component Renewal Year** = `export_year + + Remaining Life` (export date passed explicitly as a load parameter — Calico's + file is dated ≈ 2026-06-02), so the absolute renewal year is reproducible across + re-runs. The fitted/renewed date is retained as `install_date`. This replaces + the module's `survey_year = datetime.now().year` TODO for this loader. + +- **Delivery: one-off runner load, from S3.** Calico is a bulk one-off import, run + through the local/orchestration runner (not the SQS-Lambda path Peabody and LBWF + use). The raw `.xlsx` is stored in S3 under a documented Calico key and kept out + of git (it carries real addresses). + +## Consequences + +- The condition-ingestion module moves out of legacy `backend/condition/` into the + DDD layout (domain / repositories / infrastructure / applications / + orchestration) as a full lift-and-shift — shared core, the Peabody and LBWF + adapters, and the delivery path — with Calico added as a new adapter alongside + them. Peabody/LBWF keep their Lambda after the move; the relocation is + behaviour-preserving for them. +- A new `Calico` `ConditionFileType`, parser + row DTO, roof-covering element map + (`"Roof Covering"` → roof `ElementType`, `MATERIAL` aspect = `Type`), and mapper + are added and registered in the factory. Roof covering is already a modelled + component type in `ElementType`, so no enum change is needed for this load. +- CONTEXT.md gains **Stock Condition Data**, **Component Renewal Year**, and + **Roof Replacement Flag**, kept distinct from **Site Notes** and **Landlord + Overrides** so the ubiquitous language records that this source is *not* read by + the engine. +- Because the FE reads these tables through Drizzle, any later schema change here + (a new component, a new aspect) is a cross-repo contract change requiring a + companion Drizzle migration — the `main_heating_system` FE-ownership lesson + applies. +- If the "mind the roof" signal later needs to *drive* modelling, that is a new + ADR that deliberately re-opens engine isolation — not an incremental tweak. From 735e83cef29230fc7b763323d6c462be6e03db61 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 14:51:25 +0000 Subject: [PATCH 02/26] =?UTF-8?q?Relocate=20condition-data=20ingestion=20i?= =?UTF-8?q?nto=20the=20DDD=20layout=20=F0=9F=9F=AA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Behaviour-preserving lift-and-shift of the condition module out of legacy backend/condition/ into domain/condition, infrastructure/condition, infrastructure/postgres/condition_tables.py, repositories/condition, applications/condition, and tests/condition. Imports rewritten to the DDD paths; ConditionPostgres and the ORM models keep the legacy backend.app.db Base/db_session bridges so the existing suite proves behaviour is unchanged (SQLModel + session-DI conversion tracked as a follow-up in ADR-0064). 16 condition tests pass at the new location. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../condition}/Dockerfile | 0 {backend => applications}/condition/README.md | 0 .../condition/__init__.py | 0 .../condition/condition_trigger_request.py | 0 .../condition}/factory.py | 16 ++++++++-------- .../condition}/handler.py | 6 +++--- .../condition/local_runner.py | 6 +++--- .../condition/processor.py | 14 +++++++------- .../condition}/requirements.txt | 0 .../app/db/functions/condition_functions.py | 2 +- ...-data-is-engine-isolated-and-fe-consumed.md | 9 ++++++++- domain/condition/__init__.py | 0 .../condition}/aspect_condition.py | 2 +- .../domain => domain/condition}/aspect_type.py | 0 .../domain => domain/condition}/element.py | 4 ++-- .../condition}/element_type.py | 0 domain/condition/mapping/__init__.py | 0 .../condition}/mapping/element_mapping.py | 4 ++-- domain/condition/mapping/lbwf/__init__.py | 0 .../mapping/lbwf/lbwf_element_map.py | 6 +++--- .../condition}/mapping/lbwf/lbwf_mapper.py | 18 +++++++++--------- .../condition}/mapping/mapper.py | 4 ++-- domain/condition/mapping/peabody/__init__.py | 0 .../mapping/peabody/peabody_element_map.py | 6 +++--- .../mapping/peabody/peabody_mapper.py | 18 +++++++++--------- .../condition}/property_condition_survey.py | 2 +- domain/condition/records/__init__.py | 0 domain/condition/records/lbwf/__init__.py | 0 .../records/lbwf/lbwf_asset_condition.py | 0 .../condition}/records/lbwf/lbwf_house.py | 2 +- domain/condition/records/peabody/__init__.py | 0 .../records/peabody/peabody_asset_condition.py | 0 .../records/peabody/peabody_property.py | 2 +- infrastructure/condition/__init__.py | 0 .../condition}/date_utils.py | 0 infrastructure/condition/lookups/__init__.py | 0 .../condition/lookups/uprn_lookup_csv.py | 2 +- .../condition/lookups/uprn_lookup_s3.py | 2 +- infrastructure/condition/parsing/__init__.py | 0 .../condition/parsing/lbwf_parser.py | 8 ++++---- .../condition/parsing/parser.py | 0 .../condition/parsing/peabody_parser.py | 8 ++++---- .../postgres/condition_tables.py | 4 ++-- repositories/condition/__init__.py | 0 .../condition}/condition_postgres.py | 4 ++-- .../condition}/uprn_lookup.py | 0 tests/condition/__init__.py | 0 .../condition}/custom_asserts.py | 4 ++-- tests/condition/lookups/__init__.py | 0 .../condition}/lookups/test_uprn_lookup_csv.py | 2 +- tests/condition/mapping/__init__.py | 0 .../condition}/mapping/test_lbwf_mapper.py | 18 +++++++++--------- .../condition}/mapping/test_peabody_mapper.py | 18 +++++++++--------- tests/condition/parsing/__init__.py | 0 .../condition}/parsing/test_lbwf_parser.py | 6 +++--- .../condition}/parsing/test_parsing_factory.py | 6 +++--- .../condition}/parsing/test_peabody_parser.py | 8 ++++---- tests/condition/persistence/__init__.py | 0 .../persistence/test_condition_postgres.py | 16 ++++++++-------- 59 files changed, 117 insertions(+), 110 deletions(-) rename {backend/condition/handler => applications/condition}/Dockerfile (100%) rename {backend => applications}/condition/README.md (100%) rename {backend => applications}/condition/__init__.py (100%) rename {backend => applications}/condition/condition_trigger_request.py (100%) rename {backend/condition/parsing => applications/condition}/factory.py (61%) rename {backend/condition/handler => applications/condition}/handler.py (88%) rename {backend => applications}/condition/local_runner.py (86%) rename {backend => applications}/condition/processor.py (73%) rename {backend/condition/handler => applications/condition}/requirements.txt (100%) create mode 100644 domain/condition/__init__.py rename {backend/condition/domain => domain/condition}/aspect_condition.py (85%) rename {backend/condition/domain => domain/condition}/aspect_type.py (100%) rename {backend/condition/domain => domain/condition}/element.py (58%) rename {backend/condition/domain => domain/condition}/element_type.py (100%) create mode 100644 domain/condition/mapping/__init__.py rename {backend/condition/domain => domain/condition}/mapping/element_mapping.py (67%) create mode 100644 domain/condition/mapping/lbwf/__init__.py rename {backend/condition/domain => domain/condition}/mapping/lbwf/lbwf_element_map.py (98%) rename {backend/condition/domain => domain/condition}/mapping/lbwf/lbwf_mapper.py (86%) rename {backend/condition/domain => domain/condition}/mapping/mapper.py (70%) create mode 100644 domain/condition/mapping/peabody/__init__.py rename {backend/condition/domain => domain/condition}/mapping/peabody/peabody_element_map.py (99%) rename {backend/condition/domain => domain/condition}/mapping/peabody/peabody_mapper.py (83%) rename {backend/condition/domain => domain/condition}/property_condition_survey.py (80%) create mode 100644 domain/condition/records/__init__.py create mode 100644 domain/condition/records/lbwf/__init__.py rename {backend/condition/parsing => domain/condition}/records/lbwf/lbwf_asset_condition.py (100%) rename {backend/condition/parsing => domain/condition}/records/lbwf/lbwf_house.py (73%) create mode 100644 domain/condition/records/peabody/__init__.py rename {backend/condition/parsing => domain/condition}/records/peabody/peabody_asset_condition.py (100%) rename {backend/condition/parsing => domain/condition}/records/peabody/peabody_property.py (74%) create mode 100644 infrastructure/condition/__init__.py rename {backend/condition/utils => infrastructure/condition}/date_utils.py (100%) create mode 100644 infrastructure/condition/lookups/__init__.py rename {backend => infrastructure}/condition/lookups/uprn_lookup_csv.py (92%) rename {backend => infrastructure}/condition/lookups/uprn_lookup_s3.py (93%) create mode 100644 infrastructure/condition/parsing/__init__.py rename {backend => infrastructure}/condition/parsing/lbwf_parser.py (96%) rename {backend => infrastructure}/condition/parsing/parser.py (100%) rename {backend => infrastructure}/condition/parsing/peabody_parser.py (94%) rename backend/app/db/models/condition.py => infrastructure/postgres/condition_tables.py (94%) create mode 100644 repositories/condition/__init__.py rename {backend/condition/persistence => repositories/condition}/condition_postgres.py (95%) rename {backend/condition/lookups => repositories/condition}/uprn_lookup.py (100%) create mode 100644 tests/condition/__init__.py rename {backend/condition/tests => tests/condition}/custom_asserts.py (96%) create mode 100644 tests/condition/lookups/__init__.py rename {backend/condition/tests => tests/condition}/lookups/test_uprn_lookup_csv.py (91%) create mode 100644 tests/condition/mapping/__init__.py rename {backend/condition/tests => tests/condition}/mapping/test_lbwf_mapper.py (95%) rename {backend/condition/tests => tests/condition}/mapping/test_peabody_mapper.py (91%) create mode 100644 tests/condition/parsing/__init__.py rename {backend/condition/tests => tests/condition}/parsing/test_lbwf_parser.py (93%) rename {backend/condition/tests => tests/condition}/parsing/test_parsing_factory.py (74%) rename {backend/condition/tests => tests/condition}/parsing/test_peabody_parser.py (94%) create mode 100644 tests/condition/persistence/__init__.py rename {backend/condition/tests => tests/condition}/persistence/test_condition_postgres.py (90%) diff --git a/backend/condition/handler/Dockerfile b/applications/condition/Dockerfile similarity index 100% rename from backend/condition/handler/Dockerfile rename to applications/condition/Dockerfile diff --git a/backend/condition/README.md b/applications/condition/README.md similarity index 100% rename from backend/condition/README.md rename to applications/condition/README.md diff --git a/backend/condition/__init__.py b/applications/condition/__init__.py similarity index 100% rename from backend/condition/__init__.py rename to applications/condition/__init__.py diff --git a/backend/condition/condition_trigger_request.py b/applications/condition/condition_trigger_request.py similarity index 100% rename from backend/condition/condition_trigger_request.py rename to applications/condition/condition_trigger_request.py diff --git a/backend/condition/parsing/factory.py b/applications/condition/factory.py similarity index 61% rename from backend/condition/parsing/factory.py rename to applications/condition/factory.py index b5d28e185..0a06811de 100644 --- a/backend/condition/parsing/factory.py +++ b/applications/condition/factory.py @@ -1,12 +1,12 @@ from typing import Optional -from backend.condition.condition_trigger_request import ConditionFileType -from backend.condition.domain.mapping.lbwf.lbwf_mapper import LbwfMapper -from backend.condition.domain.mapping.mapper import Mapper -from backend.condition.domain.mapping.peabody.peabody_mapper import PeabodyMapper -from backend.condition.lookups.uprn_lookup import UprnLookup -from backend.condition.parsing.parser import Parser -from backend.condition.parsing.lbwf_parser import LbwfParser -from backend.condition.parsing.peabody_parser import PeabodyParser +from applications.condition.condition_trigger_request import ConditionFileType +from domain.condition.mapping.lbwf.lbwf_mapper import LbwfMapper +from domain.condition.mapping.mapper import Mapper +from domain.condition.mapping.peabody.peabody_mapper import PeabodyMapper +from repositories.condition.uprn_lookup import UprnLookup +from infrastructure.condition.parsing.parser import Parser +from infrastructure.condition.parsing.lbwf_parser import LbwfParser +from infrastructure.condition.parsing.peabody_parser import PeabodyParser def select_parser( diff --git a/backend/condition/handler/handler.py b/applications/condition/handler.py similarity index 88% rename from backend/condition/handler/handler.py rename to applications/condition/handler.py index 2f3616a4c..71db0a90b 100644 --- a/backend/condition/handler/handler.py +++ b/applications/condition/handler.py @@ -2,9 +2,9 @@ import json from typing import Mapping, Any from io import BytesIO -from backend.condition.condition_trigger_request import ConditionTriggerRequest -from backend.condition.lookups.uprn_lookup_s3 import UprnLookupS3 -from backend.condition.processor import process_file +from applications.condition.condition_trigger_request import ConditionTriggerRequest +from infrastructure.condition.lookups.uprn_lookup_s3 import UprnLookupS3 +from applications.condition.processor import process_file from utils.logger import setup_logger from utils.s3 import read_io_from_s3 diff --git a/backend/condition/local_runner.py b/applications/condition/local_runner.py similarity index 86% rename from backend/condition/local_runner.py rename to applications/condition/local_runner.py index 4595b93b0..edce2501c 100644 --- a/backend/condition/local_runner.py +++ b/applications/condition/local_runner.py @@ -1,8 +1,8 @@ from pathlib import Path -from backend.condition.condition_trigger_request import ConditionFileType -from backend.condition.lookups.uprn_lookup_csv import UprnLookupLocal -from backend.condition.processor import process_file +from applications.condition.condition_trigger_request import ConditionFileType +from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal +from applications.condition.processor import process_file def main() -> None: diff --git a/backend/condition/processor.py b/applications/condition/processor.py similarity index 73% rename from backend/condition/processor.py rename to applications/condition/processor.py index ad5b42327..daefaeb3e 100644 --- a/backend/condition/processor.py +++ b/applications/condition/processor.py @@ -1,14 +1,14 @@ from typing import Any, BinaryIO, List, Optional from datetime import datetime -from backend.condition.condition_trigger_request import ConditionFileType -from backend.condition.lookups.uprn_lookup import UprnLookup +from applications.condition.condition_trigger_request import ConditionFileType +from repositories.condition.uprn_lookup import UprnLookup from utils.logger import setup_logger -from backend.condition.domain.mapping.mapper import Mapper -from backend.condition.domain.property_condition_survey import PropertyConditionSurvey -from backend.condition.parsing.parser import Parser -from backend.condition.persistence.condition_postgres import ConditionPostgres -from backend.condition.parsing.factory import select_parser, select_mapper +from domain.condition.mapping.mapper import Mapper +from domain.condition.property_condition_survey import PropertyConditionSurvey +from infrastructure.condition.parsing.parser import Parser +from repositories.condition.condition_postgres import ConditionPostgres +from applications.condition.factory import select_parser, select_mapper logger = setup_logger() diff --git a/backend/condition/handler/requirements.txt b/applications/condition/requirements.txt similarity index 100% rename from backend/condition/handler/requirements.txt rename to applications/condition/requirements.txt diff --git a/backend/app/db/functions/condition_functions.py b/backend/app/db/functions/condition_functions.py index d281b9a4b..fc62eda0b 100644 --- a/backend/app/db/functions/condition_functions.py +++ b/backend/app/db/functions/condition_functions.py @@ -3,7 +3,7 @@ from sqlalchemy import insert, delete from sqlalchemy.orm import Session from backend.app.db.connection import db_session, db_read_session -from backend.app.db.models.condition import PropertyConditionSurveyModel +from infrastructure.postgres.condition_tables import PropertyConditionSurveyModel def bulk_insert_property_surveys( diff --git a/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md b/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md index 43fdaf102..62e706dc6 100644 --- a/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md +++ b/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md @@ -91,7 +91,14 @@ solar is a front-end presentation overlay, not an engine output.** orchestration) as a full lift-and-shift — shared core, the Peabody and LBWF adapters, and the delivery path — with Calico added as a new adapter alongside them. Peabody/LBWF keep their Lambda after the move; the relocation is - behaviour-preserving for them. + behaviour-preserving for them. **The move is deliberately relocation-only:** the + ORM models stay SQLAlchemy-declarative and `ConditionPostgres` keeps the legacy + `backend.app.db` `Base` / `db_session` bridges, so the existing test suite proves + behaviour is unchanged. Converting the tables to SQLModel and the repository to + constructor session-injection (matching the rest of `infrastructure/postgres`), + and clearing the module's pre-existing pyright-strict debt, are tracked + follow-ups to run where a database is available to verify them — not bundled into + the move. - A new `Calico` `ConditionFileType`, parser + row DTO, roof-covering element map (`"Roof Covering"` → roof `ElementType`, `MATERIAL` aspect = `Type`), and mapper are added and registered in the factory. Roof covering is already a modelled diff --git a/domain/condition/__init__.py b/domain/condition/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/domain/aspect_condition.py b/domain/condition/aspect_condition.py similarity index 85% rename from backend/condition/domain/aspect_condition.py rename to domain/condition/aspect_condition.py index 75b46b09e..b768ff483 100644 --- a/backend/condition/domain/aspect_condition.py +++ b/domain/condition/aspect_condition.py @@ -2,7 +2,7 @@ from dataclasses import dataclass from typing import Optional from datetime import date -from backend.condition.domain.aspect_type import AspectType +from domain.condition.aspect_type import AspectType @dataclass diff --git a/backend/condition/domain/aspect_type.py b/domain/condition/aspect_type.py similarity index 100% rename from backend/condition/domain/aspect_type.py rename to domain/condition/aspect_type.py diff --git a/backend/condition/domain/element.py b/domain/condition/element.py similarity index 58% rename from backend/condition/domain/element.py rename to domain/condition/element.py index 4a154815b..78a51cf99 100644 --- a/backend/condition/domain/element.py +++ b/domain/condition/element.py @@ -1,8 +1,8 @@ from dataclasses import dataclass from typing import List -from backend.condition.domain.aspect_condition import AspectCondition -from backend.condition.domain.element_type import ElementType +from domain.condition.aspect_condition import AspectCondition +from domain.condition.element_type import ElementType @dataclass diff --git a/backend/condition/domain/element_type.py b/domain/condition/element_type.py similarity index 100% rename from backend/condition/domain/element_type.py rename to domain/condition/element_type.py diff --git a/domain/condition/mapping/__init__.py b/domain/condition/mapping/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/domain/mapping/element_mapping.py b/domain/condition/mapping/element_mapping.py similarity index 67% rename from backend/condition/domain/mapping/element_mapping.py rename to domain/condition/mapping/element_mapping.py index 95fd08b9f..ead1a8d21 100644 --- a/backend/condition/domain/mapping/element_mapping.py +++ b/domain/condition/mapping/element_mapping.py @@ -1,8 +1,8 @@ from dataclasses import dataclass from typing import Optional -from backend.condition.domain.aspect_type import AspectType -from backend.condition.domain.element_type import ElementType +from domain.condition.aspect_type import AspectType +from domain.condition.element_type import ElementType @dataclass(frozen=True) diff --git a/domain/condition/mapping/lbwf/__init__.py b/domain/condition/mapping/lbwf/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/domain/mapping/lbwf/lbwf_element_map.py b/domain/condition/mapping/lbwf/lbwf_element_map.py similarity index 98% rename from backend/condition/domain/mapping/lbwf/lbwf_element_map.py rename to domain/condition/mapping/lbwf/lbwf_element_map.py index bf54c5bbe..609c6d1ab 100644 --- a/backend/condition/domain/mapping/lbwf/lbwf_element_map.py +++ b/domain/condition/mapping/lbwf/lbwf_element_map.py @@ -1,6 +1,6 @@ -from backend.condition.domain.element_type import ElementType -from backend.condition.domain.aspect_type import AspectType -from backend.condition.domain.mapping.element_mapping import ElementMapping +from domain.condition.element_type import ElementType +from domain.condition.aspect_type import AspectType +from domain.condition.mapping.element_mapping import ElementMapping LBWF_ELEMENT_MAP: dict[str, ElementMapping] = { diff --git a/backend/condition/domain/mapping/lbwf/lbwf_mapper.py b/domain/condition/mapping/lbwf/lbwf_mapper.py similarity index 86% rename from backend/condition/domain/mapping/lbwf/lbwf_mapper.py rename to domain/condition/mapping/lbwf/lbwf_mapper.py index 9dbfcb17a..524a9960e 100644 --- a/backend/condition/domain/mapping/lbwf/lbwf_mapper.py +++ b/domain/condition/mapping/lbwf/lbwf_mapper.py @@ -1,17 +1,17 @@ from typing import Any, Dict, Optional, Tuple from datetime import date -from backend.condition.domain.aspect_condition import AspectCondition -from backend.condition.domain.element import Element -from backend.condition.domain.element_type import ElementType -from backend.condition.domain.mapping.element_mapping import ElementMapping -from backend.condition.domain.mapping.lbwf.lbwf_element_map import LBWF_ELEMENT_MAP -from backend.condition.domain.mapping.mapper import Mapper -from backend.condition.domain.property_condition_survey import PropertyConditionSurvey -from backend.condition.parsing.records.lbwf.lbwf_asset_condition import ( +from domain.condition.aspect_condition import AspectCondition +from domain.condition.element import Element +from domain.condition.element_type import ElementType +from domain.condition.mapping.element_mapping import ElementMapping +from domain.condition.mapping.lbwf.lbwf_element_map import LBWF_ELEMENT_MAP +from domain.condition.mapping.mapper import Mapper +from domain.condition.property_condition_survey import PropertyConditionSurvey +from domain.condition.records.lbwf.lbwf_asset_condition import ( LbwfAssetCondition, ) -from backend.condition.parsing.records.lbwf.lbwf_house import LbwfHouse +from domain.condition.records.lbwf.lbwf_house import LbwfHouse from utils.logger import setup_logger logger = setup_logger() diff --git a/backend/condition/domain/mapping/mapper.py b/domain/condition/mapping/mapper.py similarity index 70% rename from backend/condition/domain/mapping/mapper.py rename to domain/condition/mapping/mapper.py index 3479668a2..caf7f6e6c 100644 --- a/backend/condition/domain/mapping/mapper.py +++ b/domain/condition/mapping/mapper.py @@ -1,8 +1,8 @@ from abc import ABC, abstractmethod from typing import Any, List, Optional -from backend.condition.domain.element import Element -from backend.condition.domain.property_condition_survey import PropertyConditionSurvey +from domain.condition.element import Element +from domain.condition.property_condition_survey import PropertyConditionSurvey class Mapper(ABC): diff --git a/domain/condition/mapping/peabody/__init__.py b/domain/condition/mapping/peabody/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/domain/mapping/peabody/peabody_element_map.py b/domain/condition/mapping/peabody/peabody_element_map.py similarity index 99% rename from backend/condition/domain/mapping/peabody/peabody_element_map.py rename to domain/condition/mapping/peabody/peabody_element_map.py index ce344b9a9..0ef943c40 100644 --- a/backend/condition/domain/mapping/peabody/peabody_element_map.py +++ b/domain/condition/mapping/peabody/peabody_element_map.py @@ -1,6 +1,6 @@ -from backend.condition.domain.aspect_type import AspectType -from backend.condition.domain.element_type import ElementType -from backend.condition.domain.mapping.element_mapping import ElementMapping +from domain.condition.aspect_type import AspectType +from domain.condition.element_type import ElementType +from domain.condition.mapping.element_mapping import ElementMapping PEABODY_ELEMENT_MAP = { diff --git a/backend/condition/domain/mapping/peabody/peabody_mapper.py b/domain/condition/mapping/peabody/peabody_mapper.py similarity index 83% rename from backend/condition/domain/mapping/peabody/peabody_mapper.py rename to domain/condition/mapping/peabody/peabody_mapper.py index 92f1687f8..a493b3c39 100644 --- a/backend/condition/domain/mapping/peabody/peabody_mapper.py +++ b/domain/condition/mapping/peabody/peabody_mapper.py @@ -1,19 +1,19 @@ from typing import Any, Dict, Optional, Tuple from datetime import date -from backend.condition.domain.aspect_condition import AspectCondition -from backend.condition.domain.element import Element -from backend.condition.domain.element_type import ElementType -from backend.condition.domain.mapping.element_mapping import ElementMapping -from backend.condition.domain.mapping.peabody.peabody_element_map import ( +from domain.condition.aspect_condition import AspectCondition +from domain.condition.element import Element +from domain.condition.element_type import ElementType +from domain.condition.mapping.element_mapping import ElementMapping +from domain.condition.mapping.peabody.peabody_element_map import ( PEABODY_ELEMENT_MAP, ) -from backend.condition.domain.mapping.mapper import Mapper -from backend.condition.domain.property_condition_survey import PropertyConditionSurvey -from backend.condition.parsing.records.peabody.peabody_asset_condition import ( +from domain.condition.mapping.mapper import Mapper +from domain.condition.property_condition_survey import PropertyConditionSurvey +from domain.condition.records.peabody.peabody_asset_condition import ( PeabodyAssetCondition, ) -from backend.condition.parsing.records.peabody.peabody_property import PeabodyProperty +from domain.condition.records.peabody.peabody_property import PeabodyProperty from utils.logger import setup_logger logger = setup_logger() diff --git a/backend/condition/domain/property_condition_survey.py b/domain/condition/property_condition_survey.py similarity index 80% rename from backend/condition/domain/property_condition_survey.py rename to domain/condition/property_condition_survey.py index 6955e5faa..1a3491ba2 100644 --- a/backend/condition/domain/property_condition_survey.py +++ b/domain/condition/property_condition_survey.py @@ -2,7 +2,7 @@ from dataclasses import dataclass from typing import List from datetime import date -from backend.condition.domain.element import Element +from domain.condition.element import Element @dataclass diff --git a/domain/condition/records/__init__.py b/domain/condition/records/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/domain/condition/records/lbwf/__init__.py b/domain/condition/records/lbwf/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/parsing/records/lbwf/lbwf_asset_condition.py b/domain/condition/records/lbwf/lbwf_asset_condition.py similarity index 100% rename from backend/condition/parsing/records/lbwf/lbwf_asset_condition.py rename to domain/condition/records/lbwf/lbwf_asset_condition.py diff --git a/backend/condition/parsing/records/lbwf/lbwf_house.py b/domain/condition/records/lbwf/lbwf_house.py similarity index 73% rename from backend/condition/parsing/records/lbwf/lbwf_house.py rename to domain/condition/records/lbwf/lbwf_house.py index 3b472fbec..4c8a10e69 100644 --- a/backend/condition/parsing/records/lbwf/lbwf_house.py +++ b/domain/condition/records/lbwf/lbwf_house.py @@ -1,7 +1,7 @@ from dataclasses import dataclass from typing import List -from backend.condition.parsing.records.lbwf.lbwf_asset_condition import LbwfAssetCondition +from domain.condition.records.lbwf.lbwf_asset_condition import LbwfAssetCondition @dataclass class LbwfHouse: diff --git a/domain/condition/records/peabody/__init__.py b/domain/condition/records/peabody/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/parsing/records/peabody/peabody_asset_condition.py b/domain/condition/records/peabody/peabody_asset_condition.py similarity index 100% rename from backend/condition/parsing/records/peabody/peabody_asset_condition.py rename to domain/condition/records/peabody/peabody_asset_condition.py diff --git a/backend/condition/parsing/records/peabody/peabody_property.py b/domain/condition/records/peabody/peabody_property.py similarity index 74% rename from backend/condition/parsing/records/peabody/peabody_property.py rename to domain/condition/records/peabody/peabody_property.py index bfa6b65b4..49f11045a 100644 --- a/backend/condition/parsing/records/peabody/peabody_property.py +++ b/domain/condition/records/peabody/peabody_property.py @@ -1,7 +1,7 @@ from dataclasses import dataclass from typing import List -from backend.condition.parsing.records.peabody.peabody_asset_condition import PeabodyAssetCondition +from domain.condition.records.peabody.peabody_asset_condition import PeabodyAssetCondition @dataclass class PeabodyProperty: diff --git a/infrastructure/condition/__init__.py b/infrastructure/condition/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/utils/date_utils.py b/infrastructure/condition/date_utils.py similarity index 100% rename from backend/condition/utils/date_utils.py rename to infrastructure/condition/date_utils.py diff --git a/infrastructure/condition/lookups/__init__.py b/infrastructure/condition/lookups/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/lookups/uprn_lookup_csv.py b/infrastructure/condition/lookups/uprn_lookup_csv.py similarity index 92% rename from backend/condition/lookups/uprn_lookup_csv.py rename to infrastructure/condition/lookups/uprn_lookup_csv.py index 8b1c21a2c..e4cc7a9d3 100644 --- a/backend/condition/lookups/uprn_lookup_csv.py +++ b/infrastructure/condition/lookups/uprn_lookup_csv.py @@ -1,7 +1,7 @@ import csv from io import TextIOWrapper from typing import BinaryIO, Dict, TextIO -from backend.condition.lookups.uprn_lookup import UprnLookup +from repositories.condition.uprn_lookup import UprnLookup class UprnLookupLocal(UprnLookup): diff --git a/backend/condition/lookups/uprn_lookup_s3.py b/infrastructure/condition/lookups/uprn_lookup_s3.py similarity index 93% rename from backend/condition/lookups/uprn_lookup_s3.py rename to infrastructure/condition/lookups/uprn_lookup_s3.py index da725a2fc..b43033145 100644 --- a/backend/condition/lookups/uprn_lookup_s3.py +++ b/infrastructure/condition/lookups/uprn_lookup_s3.py @@ -2,7 +2,7 @@ import csv from io import BytesIO, TextIOWrapper from typing import BinaryIO, Dict, TextIO -from backend.condition.lookups.uprn_lookup import UprnLookup +from repositories.condition.uprn_lookup import UprnLookup from utils.s3 import read_io_from_s3 diff --git a/infrastructure/condition/parsing/__init__.py b/infrastructure/condition/parsing/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/parsing/lbwf_parser.py b/infrastructure/condition/parsing/lbwf_parser.py similarity index 96% rename from backend/condition/parsing/lbwf_parser.py rename to infrastructure/condition/parsing/lbwf_parser.py index a713b1ef2..a83ef5b78 100644 --- a/backend/condition/parsing/lbwf_parser.py +++ b/infrastructure/condition/parsing/lbwf_parser.py @@ -2,12 +2,12 @@ from typing import BinaryIO, Any, Dict, Iterator, List, Optional, Tuple from openpyxl import Workbook, load_workbook from collections import defaultdict -from backend.condition.parsing.parser import Parser -from backend.condition.parsing.records.lbwf.lbwf_asset_condition import ( +from infrastructure.condition.parsing.parser import Parser +from domain.condition.records.lbwf.lbwf_asset_condition import ( LbwfAssetCondition, ) -from backend.condition.parsing.records.lbwf.lbwf_house import LbwfHouse -from backend.condition.utils.date_utils import normalise_date +from domain.condition.records.lbwf.lbwf_house import LbwfHouse +from infrastructure.condition.date_utils import normalise_date from utils.logger import setup_logger logger = setup_logger() diff --git a/backend/condition/parsing/parser.py b/infrastructure/condition/parsing/parser.py similarity index 100% rename from backend/condition/parsing/parser.py rename to infrastructure/condition/parsing/parser.py diff --git a/backend/condition/parsing/peabody_parser.py b/infrastructure/condition/parsing/peabody_parser.py similarity index 94% rename from backend/condition/parsing/peabody_parser.py rename to infrastructure/condition/parsing/peabody_parser.py index 4620ba82c..e5b090d4d 100644 --- a/backend/condition/parsing/peabody_parser.py +++ b/infrastructure/condition/parsing/peabody_parser.py @@ -4,12 +4,12 @@ from typing import Any, BinaryIO, Dict, List, Optional, Tuple, DefaultDict from openpyxl import Workbook, load_workbook from collections import defaultdict -from backend.condition.lookups.uprn_lookup import UprnLookup -from backend.condition.parsing.parser import Parser -from backend.condition.parsing.records.peabody.peabody_asset_condition import ( +from repositories.condition.uprn_lookup import UprnLookup +from infrastructure.condition.parsing.parser import Parser +from domain.condition.records.peabody.peabody_asset_condition import ( PeabodyAssetCondition, ) -from backend.condition.parsing.records.peabody.peabody_property import PeabodyProperty +from domain.condition.records.peabody.peabody_property import PeabodyProperty from utils.logger import setup_logger logger = setup_logger() diff --git a/backend/app/db/models/condition.py b/infrastructure/postgres/condition_tables.py similarity index 94% rename from backend/app/db/models/condition.py rename to infrastructure/postgres/condition_tables.py index 96f601a73..4cff70c52 100644 --- a/backend/app/db/models/condition.py +++ b/infrastructure/postgres/condition_tables.py @@ -9,8 +9,8 @@ from sqlalchemy import ( ) from sqlalchemy.orm import relationship -from backend.condition.domain.aspect_type import AspectType -from backend.condition.domain.element_type import ElementType +from domain.condition.aspect_type import AspectType +from domain.condition.element_type import ElementType from backend.app.db.base import Base diff --git a/repositories/condition/__init__.py b/repositories/condition/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/persistence/condition_postgres.py b/repositories/condition/condition_postgres.py similarity index 95% rename from backend/condition/persistence/condition_postgres.py rename to repositories/condition/condition_postgres.py index e83df5406..fb3533e35 100644 --- a/backend/condition/persistence/condition_postgres.py +++ b/repositories/condition/condition_postgres.py @@ -3,12 +3,12 @@ from typing import List, Optional from sqlmodel import Session from utils.logger import setup_logger -from backend.app.db.models.condition import ( +from infrastructure.postgres.condition_tables import ( AspectConditionModel, ElementModel, PropertyConditionSurveyModel, ) -from backend.condition.domain.property_condition_survey import PropertyConditionSurvey +from domain.condition.property_condition_survey import PropertyConditionSurvey from backend.app.db.connection import db_session logger = setup_logger() diff --git a/backend/condition/lookups/uprn_lookup.py b/repositories/condition/uprn_lookup.py similarity index 100% rename from backend/condition/lookups/uprn_lookup.py rename to repositories/condition/uprn_lookup.py diff --git a/tests/condition/__init__.py b/tests/condition/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/tests/custom_asserts.py b/tests/condition/custom_asserts.py similarity index 96% rename from backend/condition/tests/custom_asserts.py rename to tests/condition/custom_asserts.py index 623dcf0cc..4749f1f9d 100644 --- a/backend/condition/tests/custom_asserts.py +++ b/tests/condition/custom_asserts.py @@ -1,5 +1,5 @@ -from backend.app.db.models.condition import PropertyConditionSurveyModel -from backend.condition.domain.property_condition_survey import PropertyConditionSurvey +from infrastructure.postgres.condition_tables import PropertyConditionSurveyModel +from domain.condition.property_condition_survey import PropertyConditionSurvey class CustomAsserts: diff --git a/tests/condition/lookups/__init__.py b/tests/condition/lookups/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/tests/lookups/test_uprn_lookup_csv.py b/tests/condition/lookups/test_uprn_lookup_csv.py similarity index 91% rename from backend/condition/tests/lookups/test_uprn_lookup_csv.py rename to tests/condition/lookups/test_uprn_lookup_csv.py index d01c52c2f..d211d5eb8 100644 --- a/backend/condition/tests/lookups/test_uprn_lookup_csv.py +++ b/tests/condition/lookups/test_uprn_lookup_csv.py @@ -2,7 +2,7 @@ import pytest from typing import Dict from tempfile import NamedTemporaryFile -from backend.condition.lookups.uprn_lookup_csv import UprnLookupLocal +from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal @pytest.fixture diff --git a/tests/condition/mapping/__init__.py b/tests/condition/mapping/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/tests/mapping/test_lbwf_mapper.py b/tests/condition/mapping/test_lbwf_mapper.py similarity index 95% rename from backend/condition/tests/mapping/test_lbwf_mapper.py rename to tests/condition/mapping/test_lbwf_mapper.py index 778901553..42c08c793 100644 --- a/backend/condition/tests/mapping/test_lbwf_mapper.py +++ b/tests/condition/mapping/test_lbwf_mapper.py @@ -1,16 +1,16 @@ from datetime import date -from backend.condition.domain.aspect_condition import AspectCondition -from backend.condition.domain.aspect_type import AspectType -from backend.condition.domain.element_type import ElementType -from backend.condition.domain.mapping.lbwf.lbwf_mapper import LbwfMapper -from backend.condition.domain.property_condition_survey import PropertyConditionSurvey -from backend.condition.parsing.records.lbwf.lbwf_house import LbwfHouse -from backend.condition.parsing.records.lbwf.lbwf_asset_condition import ( +from domain.condition.aspect_condition import AspectCondition +from domain.condition.aspect_type import AspectType +from domain.condition.element_type import ElementType +from domain.condition.mapping.lbwf.lbwf_mapper import LbwfMapper +from domain.condition.property_condition_survey import PropertyConditionSurvey +from domain.condition.records.lbwf.lbwf_house import LbwfHouse +from domain.condition.records.lbwf.lbwf_asset_condition import ( LbwfAssetCondition, ) -from backend.condition.domain.element import Element -from backend.condition.tests.custom_asserts import CustomAsserts +from domain.condition.element import Element +from tests.condition.custom_asserts import CustomAsserts def test_lbwf_mapper_maps_house(): diff --git a/backend/condition/tests/mapping/test_peabody_mapper.py b/tests/condition/mapping/test_peabody_mapper.py similarity index 91% rename from backend/condition/tests/mapping/test_peabody_mapper.py rename to tests/condition/mapping/test_peabody_mapper.py index 979258b08..db70de2a8 100644 --- a/backend/condition/tests/mapping/test_peabody_mapper.py +++ b/tests/condition/mapping/test_peabody_mapper.py @@ -1,16 +1,16 @@ from datetime import datetime, date -from backend.condition.domain.aspect_condition import AspectCondition -from backend.condition.domain.aspect_type import AspectType -from backend.condition.domain.element_type import ElementType -from backend.condition.domain.mapping.peabody.peabody_mapper import PeabodyMapper -from backend.condition.domain.property_condition_survey import PropertyConditionSurvey -from backend.condition.parsing.records.peabody.peabody_asset_condition import ( +from domain.condition.aspect_condition import AspectCondition +from domain.condition.aspect_type import AspectType +from domain.condition.element_type import ElementType +from domain.condition.mapping.peabody.peabody_mapper import PeabodyMapper +from domain.condition.property_condition_survey import PropertyConditionSurvey +from domain.condition.records.peabody.peabody_asset_condition import ( PeabodyAssetCondition, ) -from backend.condition.parsing.records.peabody.peabody_property import PeabodyProperty -from backend.condition.domain.element import Element -from backend.condition.tests.custom_asserts import CustomAsserts +from domain.condition.records.peabody.peabody_property import PeabodyProperty +from domain.condition.element import Element +from tests.condition.custom_asserts import CustomAsserts def test_peabody_mapper_maps_property(): diff --git a/tests/condition/parsing/__init__.py b/tests/condition/parsing/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/tests/parsing/test_lbwf_parser.py b/tests/condition/parsing/test_lbwf_parser.py similarity index 93% rename from backend/condition/tests/parsing/test_lbwf_parser.py rename to tests/condition/parsing/test_lbwf_parser.py index beb81a035..0b63161a1 100644 --- a/backend/condition/tests/parsing/test_lbwf_parser.py +++ b/tests/condition/parsing/test_lbwf_parser.py @@ -4,9 +4,9 @@ from io import BytesIO from openpyxl import Workbook from datetime import datetime -from backend.condition.parsing.lbwf_parser import LbwfParser -from backend.condition.parsing.records.lbwf.lbwf_asset_condition import LbwfAssetCondition -from backend.condition.parsing.records.lbwf.lbwf_house import LbwfHouse +from infrastructure.condition.parsing.lbwf_parser import LbwfParser +from domain.condition.records.lbwf.lbwf_asset_condition import LbwfAssetCondition +from domain.condition.records.lbwf.lbwf_house import LbwfHouse @pytest.fixture def lbwf_homes_xlsx_bytes() -> BytesIO: diff --git a/backend/condition/tests/parsing/test_parsing_factory.py b/tests/condition/parsing/test_parsing_factory.py similarity index 74% rename from backend/condition/tests/parsing/test_parsing_factory.py rename to tests/condition/parsing/test_parsing_factory.py index df01eaadc..f472f8c58 100644 --- a/backend/condition/tests/parsing/test_parsing_factory.py +++ b/tests/condition/parsing/test_parsing_factory.py @@ -1,8 +1,8 @@ import pytest -from backend.condition.condition_trigger_request import ConditionFileType -from backend.condition.lookups.uprn_lookup_csv import UprnLookupLocal -from backend.condition.parsing.factory import select_parser +from applications.condition.condition_trigger_request import ConditionFileType +from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal +from applications.condition.factory import select_parser def test_selects_lbwf_parser(): diff --git a/backend/condition/tests/parsing/test_peabody_parser.py b/tests/condition/parsing/test_peabody_parser.py similarity index 94% rename from backend/condition/tests/parsing/test_peabody_parser.py rename to tests/condition/parsing/test_peabody_parser.py index 5fb422047..ac3fbd609 100644 --- a/backend/condition/tests/parsing/test_peabody_parser.py +++ b/tests/condition/parsing/test_peabody_parser.py @@ -5,12 +5,12 @@ from io import BytesIO from openpyxl import Workbook from datetime import datetime -from backend.condition.lookups.uprn_lookup_csv import UprnLookupLocal -from backend.condition.parsing.peabody_parser import PeabodyParser -from backend.condition.parsing.records.peabody.peabody_asset_condition import ( +from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal +from infrastructure.condition.parsing.peabody_parser import PeabodyParser +from domain.condition.records.peabody.peabody_asset_condition import ( PeabodyAssetCondition, ) -from backend.condition.parsing.records.peabody.peabody_property import PeabodyProperty +from domain.condition.records.peabody.peabody_property import PeabodyProperty @pytest.fixture diff --git a/tests/condition/persistence/__init__.py b/tests/condition/persistence/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/condition/tests/persistence/test_condition_postgres.py b/tests/condition/persistence/test_condition_postgres.py similarity index 90% rename from backend/condition/tests/persistence/test_condition_postgres.py rename to tests/condition/persistence/test_condition_postgres.py index ca95eaaae..44a9417ac 100644 --- a/backend/condition/tests/persistence/test_condition_postgres.py +++ b/tests/condition/persistence/test_condition_postgres.py @@ -1,14 +1,14 @@ import pytest from datetime import date -from backend.condition.persistence.condition_postgres import ConditionPostgres -from backend.condition.domain.property_condition_survey import PropertyConditionSurvey -from backend.condition.domain.element import Element -from backend.condition.domain.element_type import ElementType -from backend.condition.domain.aspect_condition import AspectCondition -from backend.condition.domain.aspect_type import AspectType -from backend.app.db.models.condition import PropertyConditionSurveyModel -from backend.condition.tests.custom_asserts import CustomAsserts +from repositories.condition.condition_postgres import ConditionPostgres +from domain.condition.property_condition_survey import PropertyConditionSurvey +from domain.condition.element import Element +from domain.condition.element_type import ElementType +from domain.condition.aspect_condition import AspectCondition +from domain.condition.aspect_type import AspectType +from infrastructure.postgres.condition_tables import PropertyConditionSurveyModel +from tests.condition.custom_asserts import CustomAsserts def test_map_survey_to_model() -> None: From 99280b6a64f747dce58b873b2e02a55938f040ee Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 14:56:07 +0000 Subject: [PATCH 03/26] =?UTF-8?q?Map=20a=20Calico=20roof-covering=20row=20?= =?UTF-8?q?to=20a=20renewal=20year=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- domain/condition/mapping/calico/__init__.py | 0 .../mapping/calico/calico_element_map.py | 12 +++++++ .../condition/mapping/calico/calico_mapper.py | 11 ++++++ domain/condition/records/calico/__init__.py | 0 .../records/calico/calico_asset_condition.py | 20 +++++++++++ .../records/calico/calico_property.py | 10 ++++++ tests/condition/mapping/test_calico_mapper.py | 36 +++++++++++++++++++ 7 files changed, 89 insertions(+) create mode 100644 domain/condition/mapping/calico/__init__.py create mode 100644 domain/condition/mapping/calico/calico_element_map.py create mode 100644 domain/condition/mapping/calico/calico_mapper.py create mode 100644 domain/condition/records/calico/__init__.py create mode 100644 domain/condition/records/calico/calico_asset_condition.py create mode 100644 domain/condition/records/calico/calico_property.py create mode 100644 tests/condition/mapping/test_calico_mapper.py diff --git a/domain/condition/mapping/calico/__init__.py b/domain/condition/mapping/calico/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/domain/condition/mapping/calico/calico_element_map.py b/domain/condition/mapping/calico/calico_element_map.py new file mode 100644 index 000000000..36f8431ef --- /dev/null +++ b/domain/condition/mapping/calico/calico_element_map.py @@ -0,0 +1,12 @@ +from domain.condition.aspect_type import AspectType +from domain.condition.element_type import ElementType +from domain.condition.mapping.element_mapping import ElementMapping + +# Calico's component path -> the shared taxonomy. Held as data (like the LBWF / +# Peabody element maps) so extending Calico to further components is a data edit. +CALICO_ELEMENT_MAP: dict[str, ElementMapping] = { + "Roof Covering": ElementMapping( + elementType=ElementType.ROOF, + aspect_type=AspectType.MATERIAL, + ), +} diff --git a/domain/condition/mapping/calico/calico_mapper.py b/domain/condition/mapping/calico/calico_mapper.py new file mode 100644 index 000000000..446d8b8cb --- /dev/null +++ b/domain/condition/mapping/calico/calico_mapper.py @@ -0,0 +1,11 @@ +from typing import Any, Optional + +from domain.condition.mapping.mapper import Mapper +from domain.condition.property_condition_survey import PropertyConditionSurvey + + +class CalicoMapper(Mapper): + def map_asset_conditions_for_property( + self, client_property_data: Any, survey_year: Optional[int] = None + ) -> PropertyConditionSurvey: + raise NotImplementedError diff --git a/domain/condition/records/calico/__init__.py b/domain/condition/records/calico/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/domain/condition/records/calico/calico_asset_condition.py b/domain/condition/records/calico/calico_asset_condition.py new file mode 100644 index 000000000..0673fbba0 --- /dev/null +++ b/domain/condition/records/calico/calico_asset_condition.py @@ -0,0 +1,20 @@ +from dataclasses import dataclass +from datetime import date +from typing import Optional + + +@dataclass +class CalicoAssetCondition: + """One component row from a Calico stock-condition export. + + Calico's current export carries a single component (``path == "Roof + Covering"``); ``covering_type`` is the material (e.g. "Concrete Tiles") and + ``remaining_life`` is Calico's Decent-Homes-derived years-remaining figure. + """ + + asset_reference: int + path: str + covering_type: Optional[str] = None + fitted_date: Optional[date] = None + quantity: Optional[int] = None + remaining_life: Optional[int] = None diff --git a/domain/condition/records/calico/calico_property.py b/domain/condition/records/calico/calico_property.py new file mode 100644 index 000000000..ba0597d5f --- /dev/null +++ b/domain/condition/records/calico/calico_property.py @@ -0,0 +1,10 @@ +from dataclasses import dataclass +from typing import List + +from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition + + +@dataclass +class CalicoProperty: + uprn: int + assets: List[CalicoAssetCondition] diff --git a/tests/condition/mapping/test_calico_mapper.py b/tests/condition/mapping/test_calico_mapper.py new file mode 100644 index 000000000..c9e15c2e7 --- /dev/null +++ b/tests/condition/mapping/test_calico_mapper.py @@ -0,0 +1,36 @@ +from datetime import date + +from domain.condition.aspect_type import AspectType +from domain.condition.element_type import ElementType +from domain.condition.mapping.calico.calico_mapper import CalicoMapper +from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition +from domain.condition.records.calico.calico_property import CalicoProperty + + +def test_calico_mapper_sets_renewal_year_to_export_year_plus_remaining_life(): + # Arrange + calico_property = CalicoProperty( + uprn=100093305101, + assets=[ + CalicoAssetCondition( + asset_reference=443, + path="Roof Covering", + covering_type="Concrete Tiles", + fitted_date=date(1998, 4, 1), + quantity=55, + remaining_life=36, + ) + ], + ) + export_year = 2026 + + # Act + survey = CalicoMapper().map_asset_conditions_for_property( + calico_property, export_year + ) + + # Assert + roof_material = survey.elements[0].aspect_conditions[0] + assert roof_material.aspect_type is AspectType.MATERIAL + assert survey.elements[0].element_type is ElementType.ROOF + assert roof_material.renewal_year == export_year + 36 From 61a12c43a03f022e68dd8164159d0ced9842ba0a Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 14:56:53 +0000 Subject: [PATCH 04/26] =?UTF-8?q?Map=20a=20Calico=20roof-covering=20row=20?= =?UTF-8?q?to=20a=20renewal=20year=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CalicoMapper maps each roof-covering row onto a ROOF element with a MATERIAL aspect, setting renewal_year = export_year + remaining_life (Calico's Decent-Homes-derived years-remaining anchored to the export year, so the absolute renewal year is stable across re-runs). source="Calico". Co-Authored-By: Claude Opus 4.8 (1M context) --- .../condition/mapping/calico/calico_mapper.py | 49 ++++++++++++++++++- 1 file changed, 47 insertions(+), 2 deletions(-) diff --git a/domain/condition/mapping/calico/calico_mapper.py b/domain/condition/mapping/calico/calico_mapper.py index 446d8b8cb..fa127940d 100644 --- a/domain/condition/mapping/calico/calico_mapper.py +++ b/domain/condition/mapping/calico/calico_mapper.py @@ -1,11 +1,56 @@ -from typing import Any, Optional +from datetime import date +from typing import Any, List, Optional +from domain.condition.aspect_condition import AspectCondition +from domain.condition.element import Element +from domain.condition.mapping.calico.calico_element_map import CALICO_ELEMENT_MAP from domain.condition.mapping.mapper import Mapper from domain.condition.property_condition_survey import PropertyConditionSurvey +from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition +from domain.condition.records.calico.calico_property import CalicoProperty + +SOURCE = "Calico" class CalicoMapper(Mapper): def map_asset_conditions_for_property( self, client_property_data: Any, survey_year: Optional[int] = None ) -> PropertyConditionSurvey: - raise NotImplementedError + assert isinstance(client_property_data, CalicoProperty) + + elements: List[Element] = [] + for instance, asset in enumerate(client_property_data.assets, start=1): + mapping = CALICO_ELEMENT_MAP[asset.path] + elements.append( + Element( + element_type=mapping.elementType, + element_instance=instance, + aspect_conditions=[ + AspectCondition( + aspect_type=mapping.aspect_type, + aspect_instance=1, + value=asset.covering_type, + quantity=asset.quantity, + install_date=asset.fitted_date, + renewal_year=CalicoMapper._renewal_year( + asset, survey_year + ), + ) + ], + ) + ) + + return PropertyConditionSurvey( + uprn=client_property_data.uprn, + elements=elements, + date=date(survey_year, 1, 1) if survey_year else date(2000, 1, 1), + source=SOURCE, + ) + + @staticmethod + def _renewal_year( + asset: CalicoAssetCondition, survey_year: Optional[int] + ) -> Optional[int]: + if asset.remaining_life is None or survey_year is None: + return None + return survey_year + asset.remaining_life From 7308979825d8274190ac43d70d5e66b23102bdf1 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 14:59:04 +0000 Subject: [PATCH 05/26] =?UTF-8?q?Parse=20a=20Calico=20workbook=20and=20dro?= =?UTF-8?q?p=20rows=20with=20no=20covering=20type=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- .../condition/parsing/calico_parser.py | 13 ++++ tests/condition/parsing/test_calico_parser.py | 63 +++++++++++++++++++ 2 files changed, 76 insertions(+) create mode 100644 infrastructure/condition/parsing/calico_parser.py create mode 100644 tests/condition/parsing/test_calico_parser.py diff --git a/infrastructure/condition/parsing/calico_parser.py b/infrastructure/condition/parsing/calico_parser.py new file mode 100644 index 000000000..5d4edad0b --- /dev/null +++ b/infrastructure/condition/parsing/calico_parser.py @@ -0,0 +1,13 @@ +from typing import BinaryIO, List + +from domain.condition.records.calico.calico_property import CalicoProperty +from infrastructure.condition.parsing.parser import Parser +from repositories.condition.uprn_lookup import UprnLookup + + +class CalicoParser(Parser): + def __init__(self, uprn_lookup: UprnLookup) -> None: + self._uprn_lookup = uprn_lookup + + def parse(self, file_stream: BinaryIO) -> List[CalicoProperty]: + raise NotImplementedError diff --git a/tests/condition/parsing/test_calico_parser.py b/tests/condition/parsing/test_calico_parser.py new file mode 100644 index 000000000..563b7394e --- /dev/null +++ b/tests/condition/parsing/test_calico_parser.py @@ -0,0 +1,63 @@ +from datetime import date +from io import BytesIO +from typing import Dict, List, Optional + +from openpyxl import Workbook + +from infrastructure.condition.parsing.calico_parser import CalicoParser +from repositories.condition.uprn_lookup import UprnLookup + +HEADERS = [ + "Asset Reference", + "Asset Type", + "Address", + "Path", + "Type", + "Fitted / Renewed Date", + "Quantity", + "Remaining Life", +] + + +class FakeUprnLookup(UprnLookup): + def __init__(self, mapping: Dict[str, int]) -> None: + self._mapping = mapping + + def get_property_ref_to_uprn_lookup(self) -> Dict[str, int]: + return self._mapping + + +def _workbook(rows: List[List[object]]) -> BytesIO: + wb = Workbook() + ws = wb.active + assert ws is not None + ws.append(HEADERS) + for row in rows: + ws.append(row) + buffer = BytesIO() + wb.save(buffer) + buffer.seek(0) + return buffer + + +def test_calico_parser_drops_rows_with_no_covering_type(): + # Arrange + workbook = _workbook( + [ + [443, "HOUSE", "31 Venice Avenue", "Roof Covering", "Concrete Tiles", + date(1998, 4, 1), 55, 36], + [438, "FLAT", "137 Brownhill Avenue", "Roof Covering", None, + date(2000, 4, 1), 0, 973], + ] + ) + parser = CalicoParser(FakeUprnLookup({"443": 100093305101, "438": 100093388053})) + + # Act + properties = parser.parse(workbook) + + # Assert + all_assets = [asset for prop in properties for asset in prop.assets] + references: List[int] = [asset.asset_reference for asset in all_assets] + coverings: List[Optional[str]] = [asset.covering_type for asset in all_assets] + assert references == [443] + assert coverings == ["Concrete Tiles"] From 78e45f25a9b759201bece4430684557b4ddba9c5 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:00:55 +0000 Subject: [PATCH 06/26] =?UTF-8?q?Parse=20a=20Calico=20workbook=20and=20dro?= =?UTF-8?q?p=20rows=20with=20no=20covering=20type=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CalicoParser reads the workbook into CalicoAssetCondition rows, drops rows with no covering type, and groups them by asset reference into CalicoProperty, resolving each reference to a UPRN via the injected lookup (unmatched dropped). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../condition/parsing/calico_parser.py | 86 ++++++++++++++++++- 1 file changed, 84 insertions(+), 2 deletions(-) diff --git a/infrastructure/condition/parsing/calico_parser.py b/infrastructure/condition/parsing/calico_parser.py index 5d4edad0b..4532250a4 100644 --- a/infrastructure/condition/parsing/calico_parser.py +++ b/infrastructure/condition/parsing/calico_parser.py @@ -1,8 +1,16 @@ -from typing import BinaryIO, List +from collections import defaultdict +from typing import Any, BinaryIO, Dict, List, Optional, Tuple +from openpyxl import load_workbook + +from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition from domain.condition.records.calico.calico_property import CalicoProperty +from infrastructure.condition.date_utils import normalise_date from infrastructure.condition.parsing.parser import Parser from repositories.condition.uprn_lookup import UprnLookup +from utils.logger import setup_logger + +logger = setup_logger() class CalicoParser(Parser): @@ -10,4 +18,78 @@ class CalicoParser(Parser): self._uprn_lookup = uprn_lookup def parse(self, file_stream: BinaryIO) -> List[CalicoProperty]: - raise NotImplementedError + workbook = load_workbook(file_stream, data_only=True) + sheet = workbook[workbook.sheetnames[0]] + + rows = sheet.iter_rows(values_only=True) + column_index = CalicoParser._column_indexes(next(rows)) + + assets = CalicoParser._parse_assets(rows, column_index) + return self._group_assets_into_properties(assets) + + @staticmethod + def _parse_assets( + rows: Any, column_index: Dict[str, int] + ) -> List[CalicoAssetCondition]: + assets: List[CalicoAssetCondition] = [] + for row in rows: + covering_type = CalicoParser._clean_str(row[column_index["Type"]]) + if covering_type is None: + # A blank covering type is a placeholder row (no roof-covering + # record captured), not an observation -- drop on parse. + continue + assets.append( + CalicoAssetCondition( + asset_reference=int(row[column_index["Asset Reference"]]), + path=CalicoParser._clean_str(row[column_index["Path"]]) or "", + covering_type=covering_type, + fitted_date=normalise_date(row[column_index["Fitted / Renewed Date"]]), + quantity=CalicoParser._to_int(row[column_index["Quantity"]]), + remaining_life=CalicoParser._to_int( + row[column_index["Remaining Life"]] + ), + ) + ) + return assets + + def _group_assets_into_properties( + self, assets: List[CalicoAssetCondition] + ) -> List[CalicoProperty]: + ref_to_uprn: Dict[str, int] = self._uprn_lookup.get_property_ref_to_uprn_lookup() + + assets_by_ref: Dict[int, List[CalicoAssetCondition]] = defaultdict(list) + for asset in assets: + assets_by_ref[asset.asset_reference].append(asset) + + properties: List[CalicoProperty] = [] + for reference, grouped_assets in assets_by_ref.items(): + uprn = ref_to_uprn.get(str(reference)) + if uprn is None: + logger.debug( + f"[CalicoParser] No UPRN for asset reference {reference}; dropping" + ) + continue + properties.append(CalicoProperty(uprn=uprn, assets=grouped_assets)) + + return properties + + @staticmethod + def _column_indexes(headers: Tuple[object | None, ...]) -> Dict[str, int]: + return { + header: index + for index, header in enumerate(headers) + if isinstance(header, str) + } + + @staticmethod + def _clean_str(value: object) -> Optional[str]: + if not isinstance(value, str): + return None + stripped = value.strip() + return stripped or None + + @staticmethod + def _to_int(value: object) -> Optional[int]: + if isinstance(value, (int, float)): + return int(value) + return None From 44b39239a435f5ae77e3080b21beacf5d905e9cb Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:01:27 +0000 Subject: [PATCH 07/26] =?UTF-8?q?Drop=20Calico=20placeholder=20rows=20whos?= =?UTF-8?q?e=20covering=20type=20is=20the=20"N/A"=20sentinel=20?= =?UTF-8?q?=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/condition/parsing/test_calico_parser.py | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/tests/condition/parsing/test_calico_parser.py b/tests/condition/parsing/test_calico_parser.py index 563b7394e..f8ca9e557 100644 --- a/tests/condition/parsing/test_calico_parser.py +++ b/tests/condition/parsing/test_calico_parser.py @@ -40,6 +40,27 @@ def _workbook(rows: List[List[object]]) -> BytesIO: return buffer +def test_calico_parser_drops_na_sentinel_covering_rows(): + # Arrange -- Calico writes the literal string "N/A" for a placeholder row + # (no roof-covering record captured), not an empty cell. + workbook = _workbook( + [ + [443, "HOUSE", "31 Venice Avenue", "Roof Covering", "Concrete Tiles", + date(1998, 4, 1), 55, 36], + [438, "FLAT", "137 Brownhill Avenue", "Roof Covering", "N/A", + date(2000, 4, 1), 0, 973], + ] + ) + parser = CalicoParser(FakeUprnLookup({"443": 100093305101, "438": 100093388053})) + + # Act + properties = parser.parse(workbook) + + # Assert + references = [asset.asset_reference for prop in properties for asset in prop.assets] + assert references == [443] + + def test_calico_parser_drops_rows_with_no_covering_type(): # Arrange workbook = _workbook( From 56534617de941e5f50c4063e6d058113339c49c6 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:02:16 +0000 Subject: [PATCH 08/26] =?UTF-8?q?Drop=20Calico=20placeholder=20rows=20whos?= =?UTF-8?q?e=20covering=20type=20is=20the=20"N/A"=20sentinel=20?= =?UTF-8?q?=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The real Calico export lodges the literal string "N/A" (not an empty cell) for placeholder rows; _clean_str now treats N/A-family sentinels as no covering, so the 1,239 placeholders are dropped on parse (5,346 -> 4,107 real observations). Co-Authored-By: Claude Opus 4.8 (1M context) --- infrastructure/condition/parsing/calico_parser.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/infrastructure/condition/parsing/calico_parser.py b/infrastructure/condition/parsing/calico_parser.py index 4532250a4..30b5887a1 100644 --- a/infrastructure/condition/parsing/calico_parser.py +++ b/infrastructure/condition/parsing/calico_parser.py @@ -81,12 +81,18 @@ class CalicoParser(Parser): if isinstance(header, str) } + # Sentinels Calico lodges in the Type column for a placeholder row (no + # roof-covering record captured), treated as "no covering". + _BLANK_COVERING_SENTINELS = {"", "n/a", "na", "none"} + @staticmethod def _clean_str(value: object) -> Optional[str]: if not isinstance(value, str): return None stripped = value.strip() - return stripped or None + if stripped.lower() in CalicoParser._BLANK_COVERING_SENTINELS: + return None + return stripped @staticmethod def _to_int(value: object) -> Optional[int]: From 4e291e4184d37104195dac68c209511cf7d13c0f Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:03:55 +0000 Subject: [PATCH 09/26] =?UTF-8?q?Resolve=20landlord=20references=20to=20UP?= =?UTF-8?q?RNs,=20excluding=20unresolved=20rows=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- .../condition/property_reference_reader.py | 19 +++++++++++ .../condition/property_uprn_lookup.py | 18 +++++++++++ .../lookups/test_property_uprn_lookup.py | 32 +++++++++++++++++++ 3 files changed, 69 insertions(+) create mode 100644 repositories/condition/property_reference_reader.py create mode 100644 repositories/condition/property_uprn_lookup.py create mode 100644 tests/condition/lookups/test_property_uprn_lookup.py diff --git a/repositories/condition/property_reference_reader.py b/repositories/condition/property_reference_reader.py new file mode 100644 index 000000000..1eba7524e --- /dev/null +++ b/repositories/condition/property_reference_reader.py @@ -0,0 +1,19 @@ +from abc import ABC, abstractmethod +from dataclasses import dataclass +from typing import List, Optional + + +@dataclass(frozen=True) +class PropertyReference: + """A property's landlord reference and resolved UPRN, as held in the + ``property`` table. ``uprn`` is nullable — a property whose UPRN has not been + resolved yet cannot receive condition rows.""" + + landlord_property_id: Optional[str] + uprn: Optional[int] + + +class PropertyReferenceReader(ABC): + @abstractmethod + def references_for_portfolio(self, portfolio_id: int) -> List[PropertyReference]: + ... diff --git a/repositories/condition/property_uprn_lookup.py b/repositories/condition/property_uprn_lookup.py new file mode 100644 index 000000000..b9c8e4e2a --- /dev/null +++ b/repositories/condition/property_uprn_lookup.py @@ -0,0 +1,18 @@ +from typing import Dict + +from repositories.condition.property_reference_reader import PropertyReferenceReader +from repositories.condition.uprn_lookup import UprnLookup + + +class PropertyUprnLookup(UprnLookup): + """Resolves a landlord property reference to a UPRN from the ``property`` + table, scoped to one portfolio (ADR-0064).""" + + def __init__( + self, reader: PropertyReferenceReader, portfolio_id: int + ) -> None: + self._reader = reader + self._portfolio_id = portfolio_id + + def get_property_ref_to_uprn_lookup(self) -> Dict[str, int]: + raise NotImplementedError diff --git a/tests/condition/lookups/test_property_uprn_lookup.py b/tests/condition/lookups/test_property_uprn_lookup.py new file mode 100644 index 000000000..f303b55f1 --- /dev/null +++ b/tests/condition/lookups/test_property_uprn_lookup.py @@ -0,0 +1,32 @@ +from typing import List + +from repositories.condition.property_reference_reader import ( + PropertyReference, + PropertyReferenceReader, +) +from repositories.condition.property_uprn_lookup import PropertyUprnLookup + + +class FakePropertyReferenceReader(PropertyReferenceReader): + def __init__(self, references: List[PropertyReference]) -> None: + self._references = references + + def references_for_portfolio(self, portfolio_id: int) -> List[PropertyReference]: + return self._references + + +def test_property_uprn_lookup_excludes_rows_with_no_uprn(): + # Arrange + reader = FakePropertyReferenceReader( + [ + PropertyReference(landlord_property_id="443", uprn=100093305101), + PropertyReference(landlord_property_id="999", uprn=None), + ] + ) + lookup = PropertyUprnLookup(reader, portfolio_id=824) + + # Act + mapping = lookup.get_property_ref_to_uprn_lookup() + + # Assert + assert mapping == {"443": 100093305101} From 57b0a080f416599b177d87f50845058a3a8e82a4 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:04:39 +0000 Subject: [PATCH 10/26] =?UTF-8?q?Resolve=20landlord=20references=20to=20UP?= =?UTF-8?q?RNs,=20excluding=20unresolved=20rows=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PropertyUprnLookup builds a landlord_property_id -> uprn map for a portfolio via an injected PropertyReferenceReader, excluding rows whose UPRN (or reference) is null so unresolved properties never receive condition rows. Co-Authored-By: Claude Opus 4.8 (1M context) --- repositories/condition/property_uprn_lookup.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/repositories/condition/property_uprn_lookup.py b/repositories/condition/property_uprn_lookup.py index b9c8e4e2a..0c8082675 100644 --- a/repositories/condition/property_uprn_lookup.py +++ b/repositories/condition/property_uprn_lookup.py @@ -15,4 +15,9 @@ class PropertyUprnLookup(UprnLookup): self._portfolio_id = portfolio_id def get_property_ref_to_uprn_lookup(self) -> Dict[str, int]: - raise NotImplementedError + references = self._reader.references_for_portfolio(self._portfolio_id) + return { + ref.landlord_property_id: ref.uprn + for ref in references + if ref.landlord_property_id is not None and ref.uprn is not None + } From 43a0977fce987ae8ba9e71f09f0a1f6d723a71d8 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:05:43 +0000 Subject: [PATCH 11/26] =?UTF-8?q?Report=20landlord=20references=20whose=20?= =?UTF-8?q?portfolio=20UPRN=20is=20null=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- repositories/condition/property_uprn_lookup.py | 7 ++++++- .../lookups/test_property_uprn_lookup.py | 17 +++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/repositories/condition/property_uprn_lookup.py b/repositories/condition/property_uprn_lookup.py index 0c8082675..cf2b2a344 100644 --- a/repositories/condition/property_uprn_lookup.py +++ b/repositories/condition/property_uprn_lookup.py @@ -1,4 +1,4 @@ -from typing import Dict +from typing import Dict, List from repositories.condition.property_reference_reader import PropertyReferenceReader from repositories.condition.uprn_lookup import UprnLookup @@ -21,3 +21,8 @@ class PropertyUprnLookup(UprnLookup): for ref in references if ref.landlord_property_id is not None and ref.uprn is not None } + + def null_uprn_references(self) -> List[str]: + """Landlord references present in the portfolio whose UPRN is null — + surfaced so a load can report them (none expected; ADR-0064).""" + raise NotImplementedError diff --git a/tests/condition/lookups/test_property_uprn_lookup.py b/tests/condition/lookups/test_property_uprn_lookup.py index f303b55f1..265957b59 100644 --- a/tests/condition/lookups/test_property_uprn_lookup.py +++ b/tests/condition/lookups/test_property_uprn_lookup.py @@ -30,3 +30,20 @@ def test_property_uprn_lookup_excludes_rows_with_no_uprn(): # Assert assert mapping == {"443": 100093305101} + + +def test_property_uprn_lookup_reports_references_with_null_uprn(): + # Arrange + reader = FakePropertyReferenceReader( + [ + PropertyReference(landlord_property_id="443", uprn=100093305101), + PropertyReference(landlord_property_id="999", uprn=None), + ] + ) + lookup = PropertyUprnLookup(reader, portfolio_id=824) + + # Act + null_references = lookup.null_uprn_references() + + # Assert + assert null_references == ["999"] From c9c9463ac3b9fa6a5e3843b57b6b4a0d975ed7ad Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:06:05 +0000 Subject: [PATCH 12/26] =?UTF-8?q?Report=20landlord=20references=20whose=20?= =?UTF-8?q?portfolio=20UPRN=20is=20null=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- repositories/condition/property_uprn_lookup.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/repositories/condition/property_uprn_lookup.py b/repositories/condition/property_uprn_lookup.py index cf2b2a344..8a01b8c29 100644 --- a/repositories/condition/property_uprn_lookup.py +++ b/repositories/condition/property_uprn_lookup.py @@ -25,4 +25,9 @@ class PropertyUprnLookup(UprnLookup): def null_uprn_references(self) -> List[str]: """Landlord references present in the portfolio whose UPRN is null — surfaced so a load can report them (none expected; ADR-0064).""" - raise NotImplementedError + references = self._reader.references_for_portfolio(self._portfolio_id) + return [ + ref.landlord_property_id + for ref in references + if ref.landlord_property_id is not None and ref.uprn is None + ] From 19e278e6edb0335dd7d3c7924761a9453c706214 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:06:44 +0000 Subject: [PATCH 13/26] Read portfolio property references from Postgres for UPRN resolution Postgres adapter for PropertyReferenceReader: selects landlord_property_id and uprn from the property table for a portfolio, feeding PropertyUprnLookup. Thin I/O over the FE-owned property table; exercised end-to-end where a DB is available (no DB in this environment). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../property_reference_reader_postgres.py | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 repositories/condition/property_reference_reader_postgres.py diff --git a/repositories/condition/property_reference_reader_postgres.py b/repositories/condition/property_reference_reader_postgres.py new file mode 100644 index 000000000..4845ea4c5 --- /dev/null +++ b/repositories/condition/property_reference_reader_postgres.py @@ -0,0 +1,28 @@ +from typing import List + +from sqlmodel import Session, select + +from infrastructure.postgres.property_table import PropertyRow +from repositories.condition.property_reference_reader import ( + PropertyReference, + PropertyReferenceReader, +) + + +class PropertyReferenceReaderPostgres(PropertyReferenceReader): + """Reads landlord-reference -> UPRN pairs from the ``property`` table for a + portfolio. The reference key is ``landlord_property_id`` (ADR-0064).""" + + def __init__(self, session: Session) -> None: + self._session = session + + def references_for_portfolio(self, portfolio_id: int) -> List[PropertyReference]: + rows = self._session.exec( + select(PropertyRow.landlord_property_id, PropertyRow.uprn).where( + PropertyRow.portfolio_id == portfolio_id + ) + ).all() + return [ + PropertyReference(landlord_property_id=landlord_property_id, uprn=uprn) + for landlord_property_id, uprn in rows + ] From 5c9757ad890d1b8606c6ebb5e440b4ea46838581 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:08:18 +0000 Subject: [PATCH 14/26] =?UTF-8?q?Count=20Calico=20rows=20dropped=20as=20bl?= =?UTF-8?q?ank=20placeholders=20or=20unmatched=20references=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- .../condition/parsing/calico_parser.py | 12 ++++++++++ tests/condition/parsing/test_calico_parser.py | 22 +++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/infrastructure/condition/parsing/calico_parser.py b/infrastructure/condition/parsing/calico_parser.py index 30b5887a1..bc41a8c42 100644 --- a/infrastructure/condition/parsing/calico_parser.py +++ b/infrastructure/condition/parsing/calico_parser.py @@ -16,6 +16,18 @@ logger = setup_logger() class CalicoParser(Parser): def __init__(self, uprn_lookup: UprnLookup) -> None: self._uprn_lookup = uprn_lookup + self._blank_placeholder_count = 0 + self._unmatched_reference_count = 0 + + @property + def blank_placeholder_count(self) -> int: + """Rows dropped on parse for a blank / N/A covering type.""" + return self._blank_placeholder_count + + @property + def unmatched_reference_count(self) -> int: + """Distinct asset references dropped for not resolving to a portfolio UPRN.""" + return self._unmatched_reference_count def parse(self, file_stream: BinaryIO) -> List[CalicoProperty]: workbook = load_workbook(file_stream, data_only=True) diff --git a/tests/condition/parsing/test_calico_parser.py b/tests/condition/parsing/test_calico_parser.py index f8ca9e557..fed9812e0 100644 --- a/tests/condition/parsing/test_calico_parser.py +++ b/tests/condition/parsing/test_calico_parser.py @@ -40,6 +40,28 @@ def _workbook(rows: List[List[object]]) -> BytesIO: return buffer +def test_calico_parser_counts_blank_and_unmatched_drops(): + # Arrange -- one loaded, one blank placeholder, one unmatched (not in lookup) + workbook = _workbook( + [ + [443, "HOUSE", "31 Venice Avenue", "Roof Covering", "Concrete Tiles", + date(1998, 4, 1), 55, 36], + [438, "FLAT", "137 Brownhill Avenue", "Roof Covering", "N/A", + date(2000, 4, 1), 0, 973], + [777, "HOUSE", "8 Marine Avenue", "Roof Covering", "Natural Slate", + date(2010, 4, 1), 60, 50], + ] + ) + parser = CalicoParser(FakeUprnLookup({"443": 100093305101})) + + # Act + parser.parse(workbook) + + # Assert + assert parser.blank_placeholder_count == 1 + assert parser.unmatched_reference_count == 1 + + def test_calico_parser_drops_na_sentinel_covering_rows(): # Arrange -- Calico writes the literal string "N/A" for a placeholder row # (no roof-covering record captured), not an empty cell. From 8510ddf1614871c36d93e0a9dba7d203a0675c93 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:08:56 +0000 Subject: [PATCH 15/26] =?UTF-8?q?Count=20Calico=20rows=20dropped=20as=20bl?= =?UTF-8?q?ank=20placeholders=20or=20unmatched=20references=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CalicoParser now tracks how many rows it drops for a blank/N-A covering type and how many references it drops for not resolving to a portfolio UPRN, exposed for the load reconciliation report so nothing vanishes silently. Co-Authored-By: Claude Opus 4.8 (1M context) --- infrastructure/condition/parsing/calico_parser.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/infrastructure/condition/parsing/calico_parser.py b/infrastructure/condition/parsing/calico_parser.py index bc41a8c42..e6a106f78 100644 --- a/infrastructure/condition/parsing/calico_parser.py +++ b/infrastructure/condition/parsing/calico_parser.py @@ -36,12 +36,11 @@ class CalicoParser(Parser): rows = sheet.iter_rows(values_only=True) column_index = CalicoParser._column_indexes(next(rows)) - assets = CalicoParser._parse_assets(rows, column_index) + assets = self._parse_assets(rows, column_index) return self._group_assets_into_properties(assets) - @staticmethod def _parse_assets( - rows: Any, column_index: Dict[str, int] + self, rows: Any, column_index: Dict[str, int] ) -> List[CalicoAssetCondition]: assets: List[CalicoAssetCondition] = [] for row in rows: @@ -49,6 +48,7 @@ class CalicoParser(Parser): if covering_type is None: # A blank covering type is a placeholder row (no roof-covering # record captured), not an observation -- drop on parse. + self._blank_placeholder_count += 1 continue assets.append( CalicoAssetCondition( @@ -77,6 +77,7 @@ class CalicoParser(Parser): for reference, grouped_assets in assets_by_ref.items(): uprn = ref_to_uprn.get(str(reference)) if uprn is None: + self._unmatched_reference_count += 1 logger.debug( f"[CalicoParser] No UPRN for asset reference {reference}; dropping" ) From 87eecc64006c563b89068dc077132f523cbacb06 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:09:39 +0000 Subject: [PATCH 16/26] =?UTF-8?q?Summarise=20a=20Calico=20load=20as=20a=20?= =?UTF-8?q?four-count=20reconciliation=20report=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- applications/condition/load_report.py | 25 ++++++++++++++++++ tests/condition/test_load_report.py | 38 +++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) create mode 100644 applications/condition/load_report.py create mode 100644 tests/condition/test_load_report.py diff --git a/applications/condition/load_report.py b/applications/condition/load_report.py new file mode 100644 index 000000000..eedba9c78 --- /dev/null +++ b/applications/condition/load_report.py @@ -0,0 +1,25 @@ +from dataclasses import dataclass +from typing import List + +from domain.condition.records.calico.calico_property import CalicoProperty + + +@dataclass(frozen=True) +class LoadReport: + """The reconciliation a Calico load emits so nothing vanishes silently + (ADR-0064): how many observations were loaded versus dropped, and why.""" + + loaded: int + unmatched_to_portfolio: int + null_uprn: int + blank_placeholder: int + + @classmethod + def from_calico_load( + cls, + properties: List[CalicoProperty], + blank_placeholder_count: int, + unmatched_reference_count: int, + null_uprn_references: List[str], + ) -> "LoadReport": + raise NotImplementedError diff --git a/tests/condition/test_load_report.py b/tests/condition/test_load_report.py new file mode 100644 index 000000000..038ea0ab4 --- /dev/null +++ b/tests/condition/test_load_report.py @@ -0,0 +1,38 @@ +from applications.condition.load_report import LoadReport +from domain.condition.records.calico.calico_asset_condition import CalicoAssetCondition +from domain.condition.records.calico.calico_property import CalicoProperty + + +def _property(uprn: int, reference: int, covering: str) -> CalicoProperty: + return CalicoProperty( + uprn=uprn, + assets=[ + CalicoAssetCondition( + asset_reference=reference, path="Roof Covering", covering_type=covering + ) + ], + ) + + +def test_load_report_summarises_the_four_reconciliation_counts(): + # Arrange + properties = [ + _property(100093305101, 443, "Concrete Tiles"), + _property(100093388053, 444, "Natural Slate"), + ] + + # Act + report = LoadReport.from_calico_load( + properties=properties, + blank_placeholder_count=1239, + unmatched_reference_count=5, + null_uprn_references=["999", "888"], + ) + + # Assert + assert report == LoadReport( + loaded=2, + unmatched_to_portfolio=5, + null_uprn=2, + blank_placeholder=1239, + ) From 75c561e9dbd5ebbed218ffd8cf25e3ebd742e13f Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:10:08 +0000 Subject: [PATCH 17/26] =?UTF-8?q?Summarise=20a=20Calico=20load=20as=20a=20?= =?UTF-8?q?four-count=20reconciliation=20report=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LoadReport.from_calico_load reports loaded observations alongside the three drop reasons (unmatched-to-portfolio, null-uprn, blank-placeholder) so every Calico row is accounted for. Co-Authored-By: Claude Opus 4.8 (1M context) --- applications/condition/load_report.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/applications/condition/load_report.py b/applications/condition/load_report.py index eedba9c78..bf0f2fc73 100644 --- a/applications/condition/load_report.py +++ b/applications/condition/load_report.py @@ -22,4 +22,10 @@ class LoadReport: unmatched_reference_count: int, null_uprn_references: List[str], ) -> "LoadReport": - raise NotImplementedError + loaded = sum(len(prop.assets) for prop in properties) + return cls( + loaded=loaded, + unmatched_to_portfolio=unmatched_reference_count, + null_uprn=len(null_uprn_references), + blank_placeholder=blank_placeholder_count, + ) From a6b782d5b34355e9d8bbaf611ef5fdbd655b284e Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:11:17 +0000 Subject: [PATCH 18/26] =?UTF-8?q?Select=20the=20Calico=20parser=20and=20ma?= =?UTF-8?q?pper=20by=20file=20type=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- .../condition/condition_trigger_request.py | 1 + .../condition/parsing/test_parsing_factory.py | 23 ++++++++++++++++++- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/applications/condition/condition_trigger_request.py b/applications/condition/condition_trigger_request.py index daa82949c..04fdc03bc 100644 --- a/applications/condition/condition_trigger_request.py +++ b/applications/condition/condition_trigger_request.py @@ -6,6 +6,7 @@ from pydantic import BaseModel class ConditionFileType(Enum): LBWF = "LBWF" Peabody = "Peabody" + Calico = "Calico" # TODO: make these asset management systems rather than client names diff --git a/tests/condition/parsing/test_parsing_factory.py b/tests/condition/parsing/test_parsing_factory.py index f472f8c58..8b775fa25 100644 --- a/tests/condition/parsing/test_parsing_factory.py +++ b/tests/condition/parsing/test_parsing_factory.py @@ -2,7 +2,7 @@ import pytest from applications.condition.condition_trigger_request import ConditionFileType from infrastructure.condition.lookups.uprn_lookup_csv import UprnLookupLocal -from applications.condition.factory import select_parser +from applications.condition.factory import select_parser, select_mapper def test_selects_lbwf_parser(): @@ -28,3 +28,24 @@ def test_selects_peabody_parser(): # assert assert expected_class_name == actual_class_name + + +def test_selects_calico_parser(): + # arrange + uprn_lookup = UprnLookupLocal(csv_path="test") + + # act + actual_class_name = select_parser( + ConditionFileType.Calico, uprn_lookup + ).__class__.__name__ + + # assert + assert actual_class_name == "CalicoParser" + + +def test_selects_calico_mapper(): + # act + actual_class_name = select_mapper(ConditionFileType.Calico).__class__.__name__ + + # assert + assert actual_class_name == "CalicoMapper" From 188176d81ea917bbda2e1fd7073c1c80b6d5018e Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:11:56 +0000 Subject: [PATCH 19/26] =?UTF-8?q?Select=20the=20Calico=20parser=20and=20ma?= =?UTF-8?q?pper=20by=20file=20type=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Registers ConditionFileType.Calico in the parser/mapper factory; the Calico parser requires a UPRN lookup (like Peabody) since references resolve to UPRNs via the property table. Co-Authored-By: Claude Opus 4.8 (1M context) --- applications/condition/factory.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/applications/condition/factory.py b/applications/condition/factory.py index 0a06811de..1a609b4c8 100644 --- a/applications/condition/factory.py +++ b/applications/condition/factory.py @@ -1,10 +1,12 @@ from typing import Optional from applications.condition.condition_trigger_request import ConditionFileType +from domain.condition.mapping.calico.calico_mapper import CalicoMapper from domain.condition.mapping.lbwf.lbwf_mapper import LbwfMapper from domain.condition.mapping.mapper import Mapper from domain.condition.mapping.peabody.peabody_mapper import PeabodyMapper from repositories.condition.uprn_lookup import UprnLookup from infrastructure.condition.parsing.parser import Parser +from infrastructure.condition.parsing.calico_parser import CalicoParser from infrastructure.condition.parsing.lbwf_parser import LbwfParser from infrastructure.condition.parsing.peabody_parser import PeabodyParser @@ -22,6 +24,13 @@ def select_parser( ) return PeabodyParser(uprn_lookup=uprn_lookup) + if file_type is ConditionFileType.Calico: + if not uprn_lookup: + raise ValueError( + "Cannot instantiate Calico Parser without UPRN lookup being provided" + ) + return CalicoParser(uprn_lookup=uprn_lookup) + raise ValueError("Unrecognised file type, unable to instantiate Parser") @@ -32,4 +41,7 @@ def select_mapper(file_type: ConditionFileType) -> Mapper: if file_type is ConditionFileType.Peabody: return PeabodyMapper() + if file_type is ConditionFileType.Calico: + return CalicoMapper() + raise ValueError("Unrecognised file type, unable to instantiate Mapper") From a6a48dd33746b6f7fbf68ff414ab536506e519fb Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:13:06 +0000 Subject: [PATCH 20/26] =?UTF-8?q?Run=20a=20Calico=20load=20end=20to=20end?= =?UTF-8?q?=20and=20return=20its=20reconciliation=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- applications/condition/calico_load.py | 17 +++++ repositories/condition/condition_writer.py | 13 ++++ tests/condition/test_calico_load.py | 88 ++++++++++++++++++++++ 3 files changed, 118 insertions(+) create mode 100644 applications/condition/calico_load.py create mode 100644 repositories/condition/condition_writer.py create mode 100644 tests/condition/test_calico_load.py diff --git a/applications/condition/calico_load.py b/applications/condition/calico_load.py new file mode 100644 index 000000000..a9eb018e4 --- /dev/null +++ b/applications/condition/calico_load.py @@ -0,0 +1,17 @@ +from typing import BinaryIO + +from applications.condition.load_report import LoadReport +from repositories.condition.condition_writer import ConditionWriter +from repositories.condition.property_uprn_lookup import PropertyUprnLookup + + +def run_calico_load( + file_stream: BinaryIO, + uprn_lookup: PropertyUprnLookup, + export_year: int, + writer: ConditionWriter, +) -> LoadReport: + """Parse a Calico export, map each roof-covering observation to a condition + survey keyed by UPRN, persist them, and return the reconciliation report + (ADR-0064). ``export_year`` anchors renewal years so they don't drift.""" + raise NotImplementedError diff --git a/repositories/condition/condition_writer.py b/repositories/condition/condition_writer.py new file mode 100644 index 000000000..6434fcb36 --- /dev/null +++ b/repositories/condition/condition_writer.py @@ -0,0 +1,13 @@ +from typing import List, Protocol + +from domain.condition.property_condition_survey import PropertyConditionSurvey + + +class ConditionWriter(Protocol): + """Persists condition surveys. ``ConditionPostgres`` is the production + implementation; a load orchestrator depends on this port so it can be + exercised without a database.""" + + def bulk_insert_surveys( + self, surveys: List[PropertyConditionSurvey] + ) -> None: ... diff --git a/tests/condition/test_calico_load.py b/tests/condition/test_calico_load.py new file mode 100644 index 000000000..2735cef05 --- /dev/null +++ b/tests/condition/test_calico_load.py @@ -0,0 +1,88 @@ +from datetime import date +from io import BytesIO +from typing import List + +from openpyxl import Workbook + +from applications.condition.calico_load import run_calico_load +from applications.condition.load_report import LoadReport +from domain.condition.property_condition_survey import PropertyConditionSurvey +from repositories.condition.property_reference_reader import ( + PropertyReference, + PropertyReferenceReader, +) +from repositories.condition.property_uprn_lookup import PropertyUprnLookup + +HEADERS = [ + "Asset Reference", + "Asset Type", + "Address", + "Path", + "Type", + "Fitted / Renewed Date", + "Quantity", + "Remaining Life", +] + + +class FakePropertyReferenceReader(PropertyReferenceReader): + def __init__(self, references: List[PropertyReference]) -> None: + self._references = references + + def references_for_portfolio(self, portfolio_id: int) -> List[PropertyReference]: + return self._references + + +class FakeConditionWriter: + def __init__(self) -> None: + self.surveys: List[PropertyConditionSurvey] = [] + + def bulk_insert_surveys(self, surveys: List[PropertyConditionSurvey]) -> None: + self.surveys.extend(surveys) + + +def _workbook(rows: List[List[object]]) -> BytesIO: + wb = Workbook() + ws = wb.active + assert ws is not None + ws.append(HEADERS) + for row in rows: + ws.append(row) + buffer = BytesIO() + wb.save(buffer) + buffer.seek(0) + return buffer + + +def test_run_calico_load_persists_surveys_and_reports_reconciliation(): + # Arrange -- one loaded (443), one blank placeholder (438), one unmatched (777) + workbook = _workbook( + [ + [443, "HOUSE", "31 Venice Avenue", "Roof Covering", "Concrete Tiles", + date(1998, 4, 1), 55, 36], + [438, "FLAT", "137 Brownhill Avenue", "Roof Covering", "N/A", + date(2000, 4, 1), 0, 973], + [777, "HOUSE", "8 Marine Avenue", "Roof Covering", "Natural Slate", + date(2010, 4, 1), 60, 50], + ] + ) + reader = FakePropertyReferenceReader( + [ + PropertyReference(landlord_property_id="443", uprn=100093305101), + PropertyReference(landlord_property_id="999", uprn=None), + ] + ) + lookup = PropertyUprnLookup(reader, portfolio_id=824) + writer = FakeConditionWriter() + + # Act + report = run_calico_load(workbook, lookup, export_year=2026, writer=writer) + + # Assert + assert report == LoadReport( + loaded=1, unmatched_to_portfolio=1, null_uprn=1, blank_placeholder=1 + ) + assert len(writer.surveys) == 1 + persisted = writer.surveys[0] + assert persisted.uprn == 100093305101 + assert persisted.elements[0].aspect_conditions[0].renewal_year == 2026 + 36 From 773869ada4d0126ef3bdd29cc53b9b830298e210 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:13:33 +0000 Subject: [PATCH 21/26] =?UTF-8?q?Run=20a=20Calico=20load=20end=20to=20end?= =?UTF-8?q?=20and=20return=20its=20reconciliation=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit run_calico_load ties parse -> map -> persist together: it resolves references to UPRNs, drops blank/unmatched rows, persists the roof-covering surveys with export-year-anchored renewal years, and returns the four-count LoadReport. Co-Authored-By: Claude Opus 4.8 (1M context) --- applications/condition/calico_load.py | 28 +++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/applications/condition/calico_load.py b/applications/condition/calico_load.py index a9eb018e4..73fbca1df 100644 --- a/applications/condition/calico_load.py +++ b/applications/condition/calico_load.py @@ -1,8 +1,14 @@ -from typing import BinaryIO +from typing import BinaryIO, List from applications.condition.load_report import LoadReport +from domain.condition.mapping.calico.calico_mapper import CalicoMapper +from domain.condition.property_condition_survey import PropertyConditionSurvey +from infrastructure.condition.parsing.calico_parser import CalicoParser from repositories.condition.condition_writer import ConditionWriter from repositories.condition.property_uprn_lookup import PropertyUprnLookup +from utils.logger import setup_logger + +logger = setup_logger() def run_calico_load( @@ -14,4 +20,22 @@ def run_calico_load( """Parse a Calico export, map each roof-covering observation to a condition survey keyed by UPRN, persist them, and return the reconciliation report (ADR-0064). ``export_year`` anchors renewal years so they don't drift.""" - raise NotImplementedError + parser = CalicoParser(uprn_lookup) + properties = parser.parse(file_stream) + + mapper = CalicoMapper() + surveys: List[PropertyConditionSurvey] = [ + mapper.map_asset_conditions_for_property(prop, export_year) + for prop in properties + ] + + writer.bulk_insert_surveys(surveys) + + report = LoadReport.from_calico_load( + properties=properties, + blank_placeholder_count=parser.blank_placeholder_count, + unmatched_reference_count=parser.unmatched_reference_count, + null_uprn_references=uprn_lookup.null_uprn_references(), + ) + logger.info(f"[calico_load] {report}") + return report From 3a77a1a81163084752895cf917ad837671c8a1e8 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:15:38 +0000 Subject: [PATCH 22/26] Wire the one-off Calico S3 load runner run_calico_s3_load reads the Calico export from S3 and runs it through run_calico_load against portfolio 824, resolving references to UPRNs via the property table and persisting with ConditionPostgres. S3 + Postgres wiring around the unit-tested load; run where a database and S3 are available. Co-Authored-By: Claude Opus 4.8 (1M context) --- applications/condition/calico_runner.py | 56 +++++++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 applications/condition/calico_runner.py diff --git a/applications/condition/calico_runner.py b/applications/condition/calico_runner.py new file mode 100644 index 000000000..cd7ed2a37 --- /dev/null +++ b/applications/condition/calico_runner.py @@ -0,0 +1,56 @@ +"""One-off runner for the Calico stock-condition (roof-covering) load. + +Calico is a bulk one-off import, not a recurring feed, so it runs through this +runner rather than the SQS-Lambda path Peabody/LBWF use (ADR-0064). The raw +export lives in S3; references resolve to UPRNs via the ``property`` table for +Calico's portfolio. + +Not exercised by the unit suite -- it is S3 + Postgres wiring around the +unit-tested ``run_calico_load``; run it where a database and S3 are available. +""" + +import os +from io import BytesIO + +from applications.condition.calico_load import run_calico_load +from applications.condition.load_report import LoadReport +from infrastructure.postgres.config import PostgresConfig +from infrastructure.postgres.engine import make_engine, transactional_session +from repositories.condition.condition_postgres import ConditionPostgres +from repositories.condition.property_reference_reader_postgres import ( + PropertyReferenceReaderPostgres, +) +from repositories.condition.property_uprn_lookup import PropertyUprnLookup +from utils.logger import setup_logger +from utils.s3 import read_io_from_s3 + +logger = setup_logger() + +CALICO_PORTFOLIO_ID = 824 +# The export date the Calico file was generated (Roof Covering 2.6.26); anchors +# renewal years so they don't drift on re-run (ADR-0064). +CALICO_EXPORT_YEAR = 2026 + + +def run_calico_s3_load( + bucket: str, + key: str, + portfolio_id: int = CALICO_PORTFOLIO_ID, + export_year: int = CALICO_EXPORT_YEAR, +) -> LoadReport: + file_bytes: BytesIO = read_io_from_s3(bucket_name=bucket, file_key=key) + + engine = make_engine(PostgresConfig.from_env(dict(os.environ))) + with transactional_session(engine) as session: + lookup = PropertyUprnLookup( + PropertyReferenceReaderPostgres(session), portfolio_id + ) + report = run_calico_load( + file_stream=file_bytes, + uprn_lookup=lookup, + export_year=export_year, + writer=ConditionPostgres(), + ) + + logger.info(f"[calico_runner] {report}") + return report From 0588ae233a536af2a3831b8dcbdba98a147651a9 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:23:46 +0000 Subject: [PATCH 23/26] Document null_uprn as a portfolio-wide canary, not a partitioning drop count Records the agreed reconciliation semantics: null_uprn reports portfolio properties missing a UPRN (expected zero) and may overlap unmatched_to_portfolio, rather than strictly partitioning the Calico rows. Co-Authored-By: Claude Opus 4.8 (1M context) --- applications/condition/load_report.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/applications/condition/load_report.py b/applications/condition/load_report.py index bf0f2fc73..21bbf7553 100644 --- a/applications/condition/load_report.py +++ b/applications/condition/load_report.py @@ -7,7 +7,15 @@ from domain.condition.records.calico.calico_property import CalicoProperty @dataclass(frozen=True) class LoadReport: """The reconciliation a Calico load emits so nothing vanishes silently - (ADR-0064): how many observations were loaded versus dropped, and why.""" + (ADR-0064): how many observations were loaded versus dropped, and why. + + ``loaded``, ``unmatched_to_portfolio`` and ``blank_placeholder`` account for + the Calico rows. ``null_uprn`` is a portfolio-wide data-quality canary — the + count of properties in the portfolio whose UPRN is null (expected: zero) — + *not* a per-row drop reason, so it does not partition with the others: a + Calico reference pointing at a null-UPRN property is reported both here and + under ``unmatched_to_portfolio``. + """ loaded: int unmatched_to_portfolio: int From f1cf7babade469f28bd6f3e884801a5b6c27f871 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:34:23 +0000 Subject: [PATCH 24/26] Record the confirmed Calico UPRN-join mismatch and S3 source Confirmed against the DB: Calico Asset References have zero overlap with portfolio 824's landlord_property_id (addresses match; all 824 UPRNs non-null), so the property-table lookup cannot resolve Calico. Documents the open resolution (Calico-supplied ref->UPRN table or address matching) in ADR-0064, the runner, and the README, and pins the confirmed S3 source location. Co-Authored-By: Claude Opus 4.8 (1M context) --- applications/condition/README.md | 26 ++++++++++++++++++- applications/condition/calico_runner.py | 13 ++++++++-- ...data-is-engine-isolated-and-fe-consumed.md | 12 +++++++++ 3 files changed, 48 insertions(+), 3 deletions(-) diff --git a/applications/condition/README.md b/applications/condition/README.md index 46302cab0..15597bf23 100644 --- a/applications/condition/README.md +++ b/applications/condition/README.md @@ -12,7 +12,31 @@ The Condition Data Processor performs the following steps: - **Load** - Persist transformed data into the ARA database (not yet implemented) -The processor currently supports file formats provided by **Peabody** and **LBWF**. +The processor currently supports file formats provided by **Peabody**, **LBWF**, +and **Calico**. + +--- + +## Calico (roof-covering, one-off) + +Calico is a one-off bulk load of stock-condition **roof-covering** data +(ADR-0064), run through `applications/condition/calico_runner.py` +(`run_calico_s3_load`), not the SQS Lambda. It is **producer-only** — written to +the `property_condition_survey` tables for the FE to read; the modelling engine +never reads it. + +- **Source:** `s3://condition-data-dev/input/calico/Roof Covering 2.6.26.xlsx` +- **Portfolio:** 824 · **export year:** 2026 (anchors renewal years). +- **Mapping:** `"Roof Covering"` → `ROOF` / `MATERIAL` (the external covering + material — tiles/slate — not the lining or structure). +- **Drops (reconciled, never silent):** `"N/A"`/blank covering rows on parse + (1,239), references that don't resolve to a UPRN, and null-UPRN properties are + reported. A `LoadReport` summarises `loaded / unmatched / null-uprn / blank`. +- **⚠ Open — UPRN resolution:** Calico's `Asset Reference` does **not** match our + `landlord_property_id` for portfolio 824 (confirmed: zero overlap; addresses do + match). The Postgres `PropertyUprnLookup` cannot resolve Calico yet — it needs a + Calico-supplied `Asset Reference → UPRN` table (use the CSV `UprnLookup`) or + address matching. See the ADR-0064 update. --- diff --git a/applications/condition/calico_runner.py b/applications/condition/calico_runner.py index cd7ed2a37..4a5f7bead 100644 --- a/applications/condition/calico_runner.py +++ b/applications/condition/calico_runner.py @@ -7,6 +7,12 @@ Calico's portfolio. Not exercised by the unit suite -- it is S3 + Postgres wiring around the unit-tested ``run_calico_load``; run it where a database and S3 are available. + +NOTE (2026-07-13): the ``PropertyUprnLookup`` wired below does **not** resolve +Calico yet -- Calico Asset References have zero overlap with portfolio 824's +stored ``landlord_property_id`` (ADR-0064 update). Swap in a CSV ``UprnLookup`` +once Calico supplies an ``Asset Reference -> UPRN`` table, or add address +matching. Running as-is would drop every row as unmatched. """ import os @@ -30,11 +36,14 @@ CALICO_PORTFOLIO_ID = 824 # The export date the Calico file was generated (Roof Covering 2.6.26); anchors # renewal years so they don't drift on re-run (ADR-0064). CALICO_EXPORT_YEAR = 2026 +# Confirmed source location. +CALICO_BUCKET = "condition-data-dev" +CALICO_KEY = "input/calico/Roof Covering 2.6.26.xlsx" def run_calico_s3_load( - bucket: str, - key: str, + bucket: str = CALICO_BUCKET, + key: str = CALICO_KEY, portfolio_id: int = CALICO_PORTFOLIO_ID, export_year: int = CALICO_EXPORT_YEAR, ) -> LoadReport: diff --git a/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md b/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md index 62e706dc6..5507fcca7 100644 --- a/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md +++ b/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md @@ -64,6 +64,18 @@ solar is a front-end presentation overlay, not an engine output.** TODO — not an exported file. A key-alignment test (Asset Reference == stored `landlord_property_id`) is pinned during TDD implementation. + > **Update (confirmed against the DB, 2026-07-13).** The key alignment is + > **false**: Calico Asset References (`438…14029`) have **zero** overlap with + > portfolio 824's stored `landlord_property_id` (`690548…`), though the + > *addresses* match — portfolio 824 **is** Calico's Burnley stock, keyed under a + > different scheme, and all 5,276 of its properties have a non-null UPRN. So the + > Postgres property-table lookup cannot resolve Calico. An exact address-line + > join is also weak (555 / 3,990). Resolution is **open pending a Calico-supplied + > `Asset Reference → UPRN` (or → `landlord_property_id`) table** — which reverts + > this to the CSV `UprnLookup` path — or a proper address-normalisation match. + > The `PropertyUprnLookup` code stands; it is simply not the right resolver for + > Calico until the key exists. + - **Drop-and-report reconciliation; nothing vanishes silently.** Rows that do not resolve to a portfolio-824 UPRN are dropped and counted; rows whose resolved `property.uprn` is null are dropped **and reported** (none expected — we want to From 37745939b351f39dd931941f3f699518dee238d5 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 16:22:53 +0000 Subject: [PATCH 25/26] Resolve Calico UPRNs from the address-matched S3 CSV lookup Calico Asset References don't match our landlord_property_id, so the runner now resolves UPRNs from the address-matched CSV (UprnLookupS3) uploaded to S3, not the property-table PropertyUprnLookup. run_calico_load accepts any UprnLookup (null_uprn count applies only to the property-table lookup). Adds a dry_run option to reconcile without writing. Co-Authored-By: Claude Opus 4.8 (1M context) --- applications/condition/calico_load.py | 16 +++++-- applications/condition/calico_runner.py | 60 ++++++++++++------------- 2 files changed, 43 insertions(+), 33 deletions(-) diff --git a/applications/condition/calico_load.py b/applications/condition/calico_load.py index 73fbca1df..ea17687ef 100644 --- a/applications/condition/calico_load.py +++ b/applications/condition/calico_load.py @@ -6,6 +6,7 @@ from domain.condition.property_condition_survey import PropertyConditionSurvey from infrastructure.condition.parsing.calico_parser import CalicoParser from repositories.condition.condition_writer import ConditionWriter from repositories.condition.property_uprn_lookup import PropertyUprnLookup +from repositories.condition.uprn_lookup import UprnLookup from utils.logger import setup_logger logger = setup_logger() @@ -13,13 +14,17 @@ logger = setup_logger() def run_calico_load( file_stream: BinaryIO, - uprn_lookup: PropertyUprnLookup, + uprn_lookup: UprnLookup, export_year: int, writer: ConditionWriter, ) -> LoadReport: """Parse a Calico export, map each roof-covering observation to a condition survey keyed by UPRN, persist them, and return the reconciliation report - (ADR-0064). ``export_year`` anchors renewal years so they don't drift.""" + (ADR-0064). ``export_year`` anchors renewal years so they don't drift. + + Works with any ``UprnLookup``. A CSV/S3 lookup carries only resolved UPRNs, + so the ``null_uprn`` count is only meaningful for the property-table lookup + (which surfaces portfolio properties missing a UPRN); it is zero otherwise.""" parser = CalicoParser(uprn_lookup) properties = parser.parse(file_stream) @@ -31,11 +36,16 @@ def run_calico_load( writer.bulk_insert_surveys(surveys) + null_uprn_references = ( + uprn_lookup.null_uprn_references() + if isinstance(uprn_lookup, PropertyUprnLookup) + else [] + ) report = LoadReport.from_calico_load( properties=properties, blank_placeholder_count=parser.blank_placeholder_count, unmatched_reference_count=parser.unmatched_reference_count, - null_uprn_references=uprn_lookup.null_uprn_references(), + null_uprn_references=null_uprn_references, ) logger.info(f"[calico_load] {report}") return report diff --git a/applications/condition/calico_runner.py b/applications/condition/calico_runner.py index 4a5f7bead..26db31af7 100644 --- a/applications/condition/calico_runner.py +++ b/applications/condition/calico_runner.py @@ -2,64 +2,64 @@ Calico is a bulk one-off import, not a recurring feed, so it runs through this runner rather than the SQS-Lambda path Peabody/LBWF use (ADR-0064). The raw -export lives in S3; references resolve to UPRNs via the ``property`` table for -Calico's portfolio. +export and the Asset-Reference->UPRN lookup both live in S3. Not exercised by the unit suite -- it is S3 + Postgres wiring around the unit-tested ``run_calico_load``; run it where a database and S3 are available. -NOTE (2026-07-13): the ``PropertyUprnLookup`` wired below does **not** resolve -Calico yet -- Calico Asset References have zero overlap with portfolio 824's -stored ``landlord_property_id`` (ADR-0064 update). Swap in a CSV ``UprnLookup`` -once Calico supplies an ``Asset Reference -> UPRN`` table, or add address -matching. Running as-is would drop every row as unmatched. +Calico Asset References do not match our stored ``landlord_property_id``, so the +UPRN is resolved from an address-matched CSV lookup (built against portfolio 824 +and uploaded to S3), not the property table. Pass ``dry_run=True`` to reconcile +without writing. """ -import os from io import BytesIO +from typing import List from applications.condition.calico_load import run_calico_load from applications.condition.load_report import LoadReport -from infrastructure.postgres.config import PostgresConfig -from infrastructure.postgres.engine import make_engine, transactional_session +from domain.condition.property_condition_survey import PropertyConditionSurvey +from infrastructure.condition.lookups.uprn_lookup_s3 import UprnLookupS3 from repositories.condition.condition_postgres import ConditionPostgres -from repositories.condition.property_reference_reader_postgres import ( - PropertyReferenceReaderPostgres, -) -from repositories.condition.property_uprn_lookup import PropertyUprnLookup from utils.logger import setup_logger from utils.s3 import read_io_from_s3 logger = setup_logger() -CALICO_PORTFOLIO_ID = 824 # The export date the Calico file was generated (Roof Covering 2.6.26); anchors # renewal years so they don't drift on re-run (ADR-0064). CALICO_EXPORT_YEAR = 2026 -# Confirmed source location. CALICO_BUCKET = "condition-data-dev" CALICO_KEY = "input/calico/Roof Covering 2.6.26.xlsx" +CALICO_UPRN_LOOKUP_KEY = "input/calico/uprn-lookup/calico_reference_to_uprn.csv" + + +class _NoOpWriter: + """Discards surveys -- used for a dry run so the load can be reconciled + without touching the database.""" + + def bulk_insert_surveys( + self, surveys: List[PropertyConditionSurvey] + ) -> None: + logger.info(f"[calico_runner] dry run -- would persist {len(surveys)} surveys") def run_calico_s3_load( bucket: str = CALICO_BUCKET, key: str = CALICO_KEY, - portfolio_id: int = CALICO_PORTFOLIO_ID, + uprn_lookup_key: str = CALICO_UPRN_LOOKUP_KEY, export_year: int = CALICO_EXPORT_YEAR, + dry_run: bool = False, ) -> LoadReport: file_bytes: BytesIO = read_io_from_s3(bucket_name=bucket, file_key=key) + lookup = UprnLookupS3(bucket=bucket, key=uprn_lookup_key) + writer = _NoOpWriter() if dry_run else ConditionPostgres() - engine = make_engine(PostgresConfig.from_env(dict(os.environ))) - with transactional_session(engine) as session: - lookup = PropertyUprnLookup( - PropertyReferenceReaderPostgres(session), portfolio_id - ) - report = run_calico_load( - file_stream=file_bytes, - uprn_lookup=lookup, - export_year=export_year, - writer=ConditionPostgres(), - ) - - logger.info(f"[calico_runner] {report}") + report = run_calico_load( + file_stream=file_bytes, + uprn_lookup=lookup, + export_year=export_year, + writer=writer, + ) + logger.info(f"[calico_runner] {'(dry run) ' if dry_run else ''}{report}") return report From 689b0a5618c050c2c6f0f9b683feedc046bdf4f1 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 17:03:49 +0000 Subject: [PATCH 26/26] Fix the condition Lambda Dockerfile for the DDD layout The lift-and-shift moved the handler to applications/condition/handler.py and split the module across domain/infrastructure/repositories/applications, but the Dockerfile still COPYed backend/condition/ and ran the old CMD path. Update the COPYs to the handler's real import closure (the DDD dirs + the legacy backend Base/db_session/settings bridges) and the CMD to applications/condition/handler. Co-Authored-By: Claude Opus 4.8 (1M context) --- applications/condition/Dockerfile | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/applications/condition/Dockerfile b/applications/condition/Dockerfile index fa1305738..c672c4a6e 100644 --- a/applications/condition/Dockerfile +++ b/applications/condition/Dockerfile @@ -20,22 +20,26 @@ COPY backend/.env.test backend/.env # ----------------------------- # Copy requirements FIRST (for Docker layer caching) # ----------------------------- -COPY backend/condition/handler/requirements.txt . +COPY applications/condition/requirements.txt . # Install dependencies into Lambda runtime RUN pip install --no-cache-dir -r requirements.txt # ----------------------------- -# Copy application code +# Copy application code (the handler's import closure — DDD layout) # ----------------------------- COPY utils/ utils/ -COPY backend/condition/ backend/condition/ +COPY domain/ domain/ +COPY infrastructure/ infrastructure/ +COPY repositories/ repositories/ +COPY applications/ applications/ -COPY backend/app/db/models/condition.py backend/app/db/models/condition.py +# Legacy backend bridges still imported by the condition module: +# ConditionPostgres -> backend.app.db.connection (db_session); +# condition_tables -> backend.app.db.base (Base); connection -> backend.app.config. +COPY backend/app/config.py backend/app/config.py COPY backend/app/db/base.py backend/app/db/base.py COPY backend/app/db/connection.py backend/app/db/connection.py -COPY backend/app/config.py backend/app/config.py - COPY backend/__init__.py backend/__init__.py COPY backend/app/__init__.py backend/app/__init__.py COPY backend/app/db/__init__.py backend/app/db/__init__.py @@ -44,6 +48,6 @@ COPY backend/app/db/__init__.py backend/app/db/__init__.py # ----------------------------- # Lambda handler # ----------------------------- -CMD ["backend/condition/handler/handler.handler"] +CMD ["applications/condition/handler.handler"] # For local running -# CMD ["python", "-m", "backend.condition.handler.handler"] +# CMD ["python", "-m", "applications.condition.handler"]