# Condition Data Processor

The Condition Data Processor performs the following steps:

- **Extract**
  - Ingest client Condition Survey data files (currently from local files; future support planned for S3 and internal survey sources)
  - Parse input files into Data Transfer Objects (DTOs)

- **Transform**
  - Map source data into the internal domain data model

- **Load**
  - Persist transformed data into the ARA database (not yet implemented)

The processor currently supports file formats provided by **Peabody**, **LBWF**,
and **Calico**.

---

## Calico (roof-covering, one-off)

Calico is a one-off bulk load of stock-condition **roof-covering** data
(ADR-0064), run through `applications/condition/calico_runner.py`
(`run_calico_s3_load`), not the SQS Lambda. It is **producer-only** — written to
the `property_condition_survey` tables for the FE to read; the modelling engine
never reads it.

- **Source:** `s3://condition-data-dev/input/calico/Roof Covering 2.6.26.xlsx`
- **Portfolio:** 824 · **export year:** 2026 (anchors renewal years).
- **Mapping:** `"Roof Covering"` → `ROOF` / `MATERIAL` (the external covering
  material — tiles/slate — not the lining or structure).
- **Drops (reconciled, never silent):** `"N/A"`/blank covering rows on parse
  (1,239), references that don't resolve to a UPRN, and null-UPRN properties are
  reported. A `LoadReport` summarises `loaded / unmatched / null-uprn / blank`.
- **⚠ Open — UPRN resolution:** Calico's `Asset Reference` does **not** match our
  `landlord_property_id` for portfolio 824 (confirmed: zero overlap; addresses do
  match). The Postgres `PropertyUprnLookup` cannot resolve Calico yet — it needs a
  Calico-supplied `Asset Reference → UPRN` table (use the CSV `UprnLookup`) or
  address matching. See the ADR-0064 update.

---

## Running Locally

The `local_runner` script allows the processor to be executed in a local environment.

1. Copy sample input file(s) into the `sample_data/` directory. If working with Peabody data, you'll need the Landlord Reference / UPRN lookup file as well. 
2. Update `local_runner.py` as required, specifically the definitions of:
   - `lbwf_path`
   - `peabody_path`
   - `file_paths`
3. Run `local_runner.py`.  
   Breakpoints may be added and the script run in debug mode if required.

---

## Known Data Issues

Some inconsistencies exist in the source datasets, primarily involving multiple representations of the same logical element within a single file. In these cases, assumptions have been made in order to normalise the data into the internal domain model.

### Peabody Data – Wall Finish Mapping

In the original Peabody sample dataset, multiple Element/Sub-Element combinations correspond to wall finishes:

| Element_Code | Element  | Sub_Element_Code | Sub_Element           |
|--------------|----------|------------------|-----------------------|
| 53           | External | 23               | Primary Wall Finish   |
| 53           | External | 30               | Secondary Wall Finish |
| 120          | WALLS    | 2                | Wall Finish           |

A single property may contain records for all three combinations, and each combination may appear multiple times.

For example, the property at **55 Burnaby Street, London** contains entries for all three of the above combinations. However, it contains only a single entry for *“WALLS: Wall structure”*, indicating that the property has only one structure rather than multiple.

This pattern is also observed in other sampled properties. Based on this, the following assumption is applied:

- “Secondary” refers to a secondary **finish**, not a secondary **wall**.

As a result:
- The property is mapped to a single Wall element.
- That Wall element is assigned three Finish aspects:
  - Two with `aspect_instance = 1`
  - One with `aspect_instance = 2`

This means that the combination of  
`UPRN / ElementType / ElementInstance / AspectType / AspectInstance`  
is **not guaranteed to be unique**.

### LBWF Data – Wall Finish Mapping

In the LBWF dataset, the following element codes map to wall finishes:

- `EXTWALLFN1`
- `EXTWALLFN2`

These are similarly mapped as multiple instances of the **Finish** aspect for a single Wall element.

---