mirror of
https://github.com/Hestia-Homes/Model.git
synced 2026-07-27 23:35:01 +00:00
Record the confirmed Calico UPRN-join mismatch and S3 source
Confirmed against the DB: Calico Asset References have zero overlap with portfolio 824's landlord_property_id (addresses match; all 824 UPRNs non-null), so the property-table lookup cannot resolve Calico. Documents the open resolution (Calico-supplied ref->UPRN table or address matching) in ADR-0064, the runner, and the README, and pins the confirmed S3 source location. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
0588ae233a
commit
f1cf7babad
3 changed files with 48 additions and 3 deletions
|
|
@ -12,7 +12,31 @@ The Condition Data Processor performs the following steps:
|
|||
- **Load**
|
||||
- Persist transformed data into the ARA database (not yet implemented)
|
||||
|
||||
The processor currently supports file formats provided by **Peabody** and **LBWF**.
|
||||
The processor currently supports file formats provided by **Peabody**, **LBWF**,
|
||||
and **Calico**.
|
||||
|
||||
---
|
||||
|
||||
## Calico (roof-covering, one-off)
|
||||
|
||||
Calico is a one-off bulk load of stock-condition **roof-covering** data
|
||||
(ADR-0064), run through `applications/condition/calico_runner.py`
|
||||
(`run_calico_s3_load`), not the SQS Lambda. It is **producer-only** — written to
|
||||
the `property_condition_survey` tables for the FE to read; the modelling engine
|
||||
never reads it.
|
||||
|
||||
- **Source:** `s3://condition-data-dev/input/calico/Roof Covering 2.6.26.xlsx`
|
||||
- **Portfolio:** 824 · **export year:** 2026 (anchors renewal years).
|
||||
- **Mapping:** `"Roof Covering"` → `ROOF` / `MATERIAL` (the external covering
|
||||
material — tiles/slate — not the lining or structure).
|
||||
- **Drops (reconciled, never silent):** `"N/A"`/blank covering rows on parse
|
||||
(1,239), references that don't resolve to a UPRN, and null-UPRN properties are
|
||||
reported. A `LoadReport` summarises `loaded / unmatched / null-uprn / blank`.
|
||||
- **⚠ Open — UPRN resolution:** Calico's `Asset Reference` does **not** match our
|
||||
`landlord_property_id` for portfolio 824 (confirmed: zero overlap; addresses do
|
||||
match). The Postgres `PropertyUprnLookup` cannot resolve Calico yet — it needs a
|
||||
Calico-supplied `Asset Reference → UPRN` table (use the CSV `UprnLookup`) or
|
||||
address matching. See the ADR-0064 update.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -7,6 +7,12 @@ Calico's portfolio.
|
|||
|
||||
Not exercised by the unit suite -- it is S3 + Postgres wiring around the
|
||||
unit-tested ``run_calico_load``; run it where a database and S3 are available.
|
||||
|
||||
NOTE (2026-07-13): the ``PropertyUprnLookup`` wired below does **not** resolve
|
||||
Calico yet -- Calico Asset References have zero overlap with portfolio 824's
|
||||
stored ``landlord_property_id`` (ADR-0064 update). Swap in a CSV ``UprnLookup``
|
||||
once Calico supplies an ``Asset Reference -> UPRN`` table, or add address
|
||||
matching. Running as-is would drop every row as unmatched.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
|
@ -30,11 +36,14 @@ CALICO_PORTFOLIO_ID = 824
|
|||
# The export date the Calico file was generated (Roof Covering 2.6.26); anchors
|
||||
# renewal years so they don't drift on re-run (ADR-0064).
|
||||
CALICO_EXPORT_YEAR = 2026
|
||||
# Confirmed source location.
|
||||
CALICO_BUCKET = "condition-data-dev"
|
||||
CALICO_KEY = "input/calico/Roof Covering 2.6.26.xlsx"
|
||||
|
||||
|
||||
def run_calico_s3_load(
|
||||
bucket: str,
|
||||
key: str,
|
||||
bucket: str = CALICO_BUCKET,
|
||||
key: str = CALICO_KEY,
|
||||
portfolio_id: int = CALICO_PORTFOLIO_ID,
|
||||
export_year: int = CALICO_EXPORT_YEAR,
|
||||
) -> LoadReport:
|
||||
|
|
|
|||
|
|
@ -64,6 +64,18 @@ solar is a front-end presentation overlay, not an engine output.**
|
|||
TODO — not an exported file. A key-alignment test (Asset Reference == stored
|
||||
`landlord_property_id`) is pinned during TDD implementation.
|
||||
|
||||
> **Update (confirmed against the DB, 2026-07-13).** The key alignment is
|
||||
> **false**: Calico Asset References (`438…14029`) have **zero** overlap with
|
||||
> portfolio 824's stored `landlord_property_id` (`690548…`), though the
|
||||
> *addresses* match — portfolio 824 **is** Calico's Burnley stock, keyed under a
|
||||
> different scheme, and all 5,276 of its properties have a non-null UPRN. So the
|
||||
> Postgres property-table lookup cannot resolve Calico. An exact address-line
|
||||
> join is also weak (555 / 3,990). Resolution is **open pending a Calico-supplied
|
||||
> `Asset Reference → UPRN` (or → `landlord_property_id`) table** — which reverts
|
||||
> this to the CSV `UprnLookup` path — or a proper address-normalisation match.
|
||||
> The `PropertyUprnLookup` code stands; it is simply not the right resolver for
|
||||
> Calico until the key exists.
|
||||
|
||||
- **Drop-and-report reconciliation; nothing vanishes silently.** Rows that do not
|
||||
resolve to a portfolio-824 UPRN are dropped and counted; rows whose resolved
|
||||
`property.uprn` is null are dropped **and reported** (none expected — we want to
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue