From f1cf7babade469f28bd6f3e884801a5b6c27f871 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Mon, 13 Jul 2026 15:34:23 +0000 Subject: [PATCH] Record the confirmed Calico UPRN-join mismatch and S3 source Confirmed against the DB: Calico Asset References have zero overlap with portfolio 824's landlord_property_id (addresses match; all 824 UPRNs non-null), so the property-table lookup cannot resolve Calico. Documents the open resolution (Calico-supplied ref->UPRN table or address matching) in ADR-0064, the runner, and the README, and pins the confirmed S3 source location. Co-Authored-By: Claude Opus 4.8 (1M context) --- applications/condition/README.md | 26 ++++++++++++++++++- applications/condition/calico_runner.py | 13 ++++++++-- ...data-is-engine-isolated-and-fe-consumed.md | 12 +++++++++ 3 files changed, 48 insertions(+), 3 deletions(-) diff --git a/applications/condition/README.md b/applications/condition/README.md index 46302cab0..15597bf23 100644 --- a/applications/condition/README.md +++ b/applications/condition/README.md @@ -12,7 +12,31 @@ The Condition Data Processor performs the following steps: - **Load** - Persist transformed data into the ARA database (not yet implemented) -The processor currently supports file formats provided by **Peabody** and **LBWF**. +The processor currently supports file formats provided by **Peabody**, **LBWF**, +and **Calico**. + +--- + +## Calico (roof-covering, one-off) + +Calico is a one-off bulk load of stock-condition **roof-covering** data +(ADR-0064), run through `applications/condition/calico_runner.py` +(`run_calico_s3_load`), not the SQS Lambda. It is **producer-only** — written to +the `property_condition_survey` tables for the FE to read; the modelling engine +never reads it. + +- **Source:** `s3://condition-data-dev/input/calico/Roof Covering 2.6.26.xlsx` +- **Portfolio:** 824 · **export year:** 2026 (anchors renewal years). +- **Mapping:** `"Roof Covering"` → `ROOF` / `MATERIAL` (the external covering + material — tiles/slate — not the lining or structure). +- **Drops (reconciled, never silent):** `"N/A"`/blank covering rows on parse + (1,239), references that don't resolve to a UPRN, and null-UPRN properties are + reported. A `LoadReport` summarises `loaded / unmatched / null-uprn / blank`. +- **⚠ Open — UPRN resolution:** Calico's `Asset Reference` does **not** match our + `landlord_property_id` for portfolio 824 (confirmed: zero overlap; addresses do + match). The Postgres `PropertyUprnLookup` cannot resolve Calico yet — it needs a + Calico-supplied `Asset Reference → UPRN` table (use the CSV `UprnLookup`) or + address matching. See the ADR-0064 update. --- diff --git a/applications/condition/calico_runner.py b/applications/condition/calico_runner.py index cd7ed2a37..4a5f7bead 100644 --- a/applications/condition/calico_runner.py +++ b/applications/condition/calico_runner.py @@ -7,6 +7,12 @@ Calico's portfolio. Not exercised by the unit suite -- it is S3 + Postgres wiring around the unit-tested ``run_calico_load``; run it where a database and S3 are available. + +NOTE (2026-07-13): the ``PropertyUprnLookup`` wired below does **not** resolve +Calico yet -- Calico Asset References have zero overlap with portfolio 824's +stored ``landlord_property_id`` (ADR-0064 update). Swap in a CSV ``UprnLookup`` +once Calico supplies an ``Asset Reference -> UPRN`` table, or add address +matching. Running as-is would drop every row as unmatched. """ import os @@ -30,11 +36,14 @@ CALICO_PORTFOLIO_ID = 824 # The export date the Calico file was generated (Roof Covering 2.6.26); anchors # renewal years so they don't drift on re-run (ADR-0064). CALICO_EXPORT_YEAR = 2026 +# Confirmed source location. +CALICO_BUCKET = "condition-data-dev" +CALICO_KEY = "input/calico/Roof Covering 2.6.26.xlsx" def run_calico_s3_load( - bucket: str, - key: str, + bucket: str = CALICO_BUCKET, + key: str = CALICO_KEY, portfolio_id: int = CALICO_PORTFOLIO_ID, export_year: int = CALICO_EXPORT_YEAR, ) -> LoadReport: diff --git a/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md b/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md index 62e706dc6..5507fcca7 100644 --- a/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md +++ b/docs/adr/0064-stock-condition-data-is-engine-isolated-and-fe-consumed.md @@ -64,6 +64,18 @@ solar is a front-end presentation overlay, not an engine output.** TODO — not an exported file. A key-alignment test (Asset Reference == stored `landlord_property_id`) is pinned during TDD implementation. + > **Update (confirmed against the DB, 2026-07-13).** The key alignment is + > **false**: Calico Asset References (`438…14029`) have **zero** overlap with + > portfolio 824's stored `landlord_property_id` (`690548…`), though the + > *addresses* match — portfolio 824 **is** Calico's Burnley stock, keyed under a + > different scheme, and all 5,276 of its properties have a non-null UPRN. So the + > Postgres property-table lookup cannot resolve Calico. An exact address-line + > join is also weak (555 / 3,990). Resolution is **open pending a Calico-supplied + > `Asset Reference → UPRN` (or → `landlord_property_id`) table** — which reverts + > this to the CSV `UprnLookup` path — or a proper address-normalisation match. + > The `PropertyUprnLookup` code stands; it is simply not the right resolver for + > Calico until the key exists. + - **Drop-and-report reconciliation; nothing vanishes silently.** Rows that do not resolve to a portfolio-824 UPRN are dropped and counted; rows whose resolved `property.uprn` is null are dropped **and reported** (none expected — we want to