Record the confirmed Calico UPRN-join mismatch and S3 source

Confirmed against the DB: Calico Asset References have zero overlap with
portfolio 824's landlord_property_id (addresses match; all 824 UPRNs non-null),
so the property-table lookup cannot resolve Calico. Documents the open
resolution (Calico-supplied ref->UPRN table or address matching) in ADR-0064,
the runner, and the README, and pins the confirmed S3 source location.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Khalim Conn-Kowlessar 2026-07-13 15:34:23 +00:00
parent 0588ae233a
commit f1cf7babad
3 changed files with 48 additions and 3 deletions

View file

@ -12,7 +12,31 @@ The Condition Data Processor performs the following steps:
- **Load**
- Persist transformed data into the ARA database (not yet implemented)
The processor currently supports file formats provided by **Peabody** and **LBWF**.
The processor currently supports file formats provided by **Peabody**, **LBWF**,
and **Calico**.
---
## Calico (roof-covering, one-off)
Calico is a one-off bulk load of stock-condition **roof-covering** data
(ADR-0064), run through `applications/condition/calico_runner.py`
(`run_calico_s3_load`), not the SQS Lambda. It is **producer-only** — written to
the `property_condition_survey` tables for the FE to read; the modelling engine
never reads it.
- **Source:** `s3://condition-data-dev/input/calico/Roof Covering 2.6.26.xlsx`
- **Portfolio:** 824 · **export year:** 2026 (anchors renewal years).
- **Mapping:** `"Roof Covering"``ROOF` / `MATERIAL` (the external covering
material — tiles/slate — not the lining or structure).
- **Drops (reconciled, never silent):** `"N/A"`/blank covering rows on parse
(1,239), references that don't resolve to a UPRN, and null-UPRN properties are
reported. A `LoadReport` summarises `loaded / unmatched / null-uprn / blank`.
- **⚠ Open — UPRN resolution:** Calico's `Asset Reference` does **not** match our
`landlord_property_id` for portfolio 824 (confirmed: zero overlap; addresses do
match). The Postgres `PropertyUprnLookup` cannot resolve Calico yet — it needs a
Calico-supplied `Asset Reference → UPRN` table (use the CSV `UprnLookup`) or
address matching. See the ADR-0064 update.
---

View file

@ -7,6 +7,12 @@ Calico's portfolio.
Not exercised by the unit suite -- it is S3 + Postgres wiring around the
unit-tested ``run_calico_load``; run it where a database and S3 are available.
NOTE (2026-07-13): the ``PropertyUprnLookup`` wired below does **not** resolve
Calico yet -- Calico Asset References have zero overlap with portfolio 824's
stored ``landlord_property_id`` (ADR-0064 update). Swap in a CSV ``UprnLookup``
once Calico supplies an ``Asset Reference -> UPRN`` table, or add address
matching. Running as-is would drop every row as unmatched.
"""
import os
@ -30,11 +36,14 @@ CALICO_PORTFOLIO_ID = 824
# The export date the Calico file was generated (Roof Covering 2.6.26); anchors
# renewal years so they don't drift on re-run (ADR-0064).
CALICO_EXPORT_YEAR = 2026
# Confirmed source location.
CALICO_BUCKET = "condition-data-dev"
CALICO_KEY = "input/calico/Roof Covering 2.6.26.xlsx"
def run_calico_s3_load(
bucket: str,
key: str,
bucket: str = CALICO_BUCKET,
key: str = CALICO_KEY,
portfolio_id: int = CALICO_PORTFOLIO_ID,
export_year: int = CALICO_EXPORT_YEAR,
) -> LoadReport:

View file

@ -64,6 +64,18 @@ solar is a front-end presentation overlay, not an engine output.**
TODO — not an exported file. A key-alignment test (Asset Reference == stored
`landlord_property_id`) is pinned during TDD implementation.
> **Update (confirmed against the DB, 2026-07-13).** The key alignment is
> **false**: Calico Asset References (`438…14029`) have **zero** overlap with
> portfolio 824's stored `landlord_property_id` (`690548…`), though the
> *addresses* match — portfolio 824 **is** Calico's Burnley stock, keyed under a
> different scheme, and all 5,276 of its properties have a non-null UPRN. So the
> Postgres property-table lookup cannot resolve Calico. An exact address-line
> join is also weak (555 / 3,990). Resolution is **open pending a Calico-supplied
> `Asset Reference → UPRN` (or → `landlord_property_id`) table** — which reverts
> this to the CSV `UprnLookup` path — or a proper address-normalisation match.
> The `PropertyUprnLookup` code stands; it is simply not the right resolver for
> Calico until the key exists.
- **Drop-and-report reconciliation; nothing vanishes silently.** Rows that do not
resolve to a portfolio-824 UPRN are dropped and counted; rows whose resolved
`property.uprn` is null are dropped **and reported** (none expected — we want to