mirror of
https://github.com/Hestia-Homes/Model.git
synced 2026-07-27 23:35:01 +00:00
f1cf7babad
Confirmed against the DB: Calico Asset References have zero overlap with portfolio 824's landlord_property_id (addresses match; all 824 UPRNs non-null), so the property-table lookup cannot resolve Calico. Documents the open resolution (Calico-supplied ref->UPRN table or address matching) in ADR-0064, the runner, and the README, and pins the confirmed S3 source location. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
99 lines
4.2 KiB
Markdown
99 lines
4.2 KiB
Markdown
# Condition Data Processor
|
||
|
||
The Condition Data Processor performs the following steps:
|
||
|
||
- **Extract**
|
||
- Ingest client Condition Survey data files (currently from local files; future support planned for S3 and internal survey sources)
|
||
- Parse input files into Data Transfer Objects (DTOs)
|
||
|
||
- **Transform**
|
||
- Map source data into the internal domain data model
|
||
|
||
- **Load**
|
||
- Persist transformed data into the ARA database (not yet implemented)
|
||
|
||
The processor currently supports file formats provided by **Peabody**, **LBWF**,
|
||
and **Calico**.
|
||
|
||
---
|
||
|
||
## Calico (roof-covering, one-off)
|
||
|
||
Calico is a one-off bulk load of stock-condition **roof-covering** data
|
||
(ADR-0064), run through `applications/condition/calico_runner.py`
|
||
(`run_calico_s3_load`), not the SQS Lambda. It is **producer-only** — written to
|
||
the `property_condition_survey` tables for the FE to read; the modelling engine
|
||
never reads it.
|
||
|
||
- **Source:** `s3://condition-data-dev/input/calico/Roof Covering 2.6.26.xlsx`
|
||
- **Portfolio:** 824 · **export year:** 2026 (anchors renewal years).
|
||
- **Mapping:** `"Roof Covering"` → `ROOF` / `MATERIAL` (the external covering
|
||
material — tiles/slate — not the lining or structure).
|
||
- **Drops (reconciled, never silent):** `"N/A"`/blank covering rows on parse
|
||
(1,239), references that don't resolve to a UPRN, and null-UPRN properties are
|
||
reported. A `LoadReport` summarises `loaded / unmatched / null-uprn / blank`.
|
||
- **⚠ Open — UPRN resolution:** Calico's `Asset Reference` does **not** match our
|
||
`landlord_property_id` for portfolio 824 (confirmed: zero overlap; addresses do
|
||
match). The Postgres `PropertyUprnLookup` cannot resolve Calico yet — it needs a
|
||
Calico-supplied `Asset Reference → UPRN` table (use the CSV `UprnLookup`) or
|
||
address matching. See the ADR-0064 update.
|
||
|
||
---
|
||
|
||
## Running Locally
|
||
|
||
The `local_runner` script allows the processor to be executed in a local environment.
|
||
|
||
1. Copy sample input file(s) into the `sample_data/` directory. If working with Peabody data, you'll need the Landlord Reference / UPRN lookup file as well.
|
||
2. Update `local_runner.py` as required, specifically the definitions of:
|
||
- `lbwf_path`
|
||
- `peabody_path`
|
||
- `file_paths`
|
||
3. Run `local_runner.py`.
|
||
Breakpoints may be added and the script run in debug mode if required.
|
||
|
||
---
|
||
|
||
## Known Data Issues
|
||
|
||
Some inconsistencies exist in the source datasets, primarily involving multiple representations of the same logical element within a single file. In these cases, assumptions have been made in order to normalise the data into the internal domain model.
|
||
|
||
### Peabody Data – Wall Finish Mapping
|
||
|
||
In the original Peabody sample dataset, multiple Element/Sub-Element combinations correspond to wall finishes:
|
||
|
||
| Element_Code | Element | Sub_Element_Code | Sub_Element |
|
||
|--------------|----------|------------------|-----------------------|
|
||
| 53 | External | 23 | Primary Wall Finish |
|
||
| 53 | External | 30 | Secondary Wall Finish |
|
||
| 120 | WALLS | 2 | Wall Finish |
|
||
|
||
A single property may contain records for all three combinations, and each combination may appear multiple times.
|
||
|
||
For example, the property at **55 Burnaby Street, London** contains entries for all three of the above combinations. However, it contains only a single entry for *“WALLS: Wall structure”*, indicating that the property has only one structure rather than multiple.
|
||
|
||
This pattern is also observed in other sampled properties. Based on this, the following assumption is applied:
|
||
|
||
- “Secondary” refers to a secondary **finish**, not a secondary **wall**.
|
||
|
||
As a result:
|
||
- The property is mapped to a single Wall element.
|
||
- That Wall element is assigned three Finish aspects:
|
||
- Two with `aspect_instance = 1`
|
||
- One with `aspect_instance = 2`
|
||
|
||
This means that the combination of
|
||
`UPRN / ElementType / ElementInstance / AspectType / AspectInstance`
|
||
is **not guaranteed to be unique**.
|
||
|
||
### LBWF Data – Wall Finish Mapping
|
||
|
||
In the LBWF dataset, the following element codes map to wall finishes:
|
||
|
||
- `EXTWALLFN1`
|
||
- `EXTWALLFN2`
|
||
|
||
These are similarly mapped as multiple instances of the **Finish** aspect for a single Wall element.
|
||
|
||
---
|
||
|