Model/applications/condition

Condition Data Processor

The Condition Data Processor performs the following steps:

• Extract

  • Ingest client Condition Survey data files (currently from local files; future support planned for S3 and internal survey sources)
  • Parse input files into Data Transfer Objects (DTOs)

• Transform

  • Map source data into the internal domain data model

• Load

  • Persist transformed data into the ARA database (not yet implemented)

The processor currently supports file formats provided by Peabody, LBWF, and Calico.

---

Calico (roof-covering, one-off)

Calico is a one-off bulk load of stock-condition roof-covering data (ADR-0064), run through applications/condition/calico_runner.py (run_calico_s3_load), not the SQS Lambda. It is producer-only — written to the property_condition_survey tables for the FE to read; the modelling engine never reads it.

• Source: s3://condition-data-dev/input/calico/Roof Covering 2.6.26.xlsx
• Portfolio: 824 · export year: 2026 (anchors renewal years).
• Mapping: "Roof Covering" → ROOF / MATERIAL (the external covering material — tiles/slate — not the lining or structure).
• Drops (reconciled, never silent): "N/A"/blank covering rows on parse (1,239), references that don't resolve to a UPRN, and null-UPRN properties are reported. A LoadReport summarises loaded / unmatched / null-uprn / blank.
• ⚠ Open — UPRN resolution: Calico's Asset Reference does not match our landlord_property_id for portfolio 824 (confirmed: zero overlap; addresses do match). The Postgres PropertyUprnLookup cannot resolve Calico yet — it needs a Calico-supplied Asset Reference → UPRN table (use the CSV UprnLookup) or address matching. See the ADR-0064 update.

---

Running Locally

The local_runner script allows the processor to be executed in a local environment.

1. Copy sample input file(s) into the sample_data/ directory. If working with Peabody data, you'll need the Landlord Reference / UPRN lookup file as well.
2. Update local_runner.py as required, specifically the definitions of:
   • lbwf_path
   • peabody_path
   • file_paths
3. Run local_runner.py.
   Breakpoints may be added and the script run in debug mode if required.

---

Known Data Issues

Some inconsistencies exist in the source datasets, primarily involving multiple representations of the same logical element within a single file. In these cases, assumptions have been made in order to normalise the data into the internal domain model.

Peabody Data – Wall Finish Mapping

In the original Peabody sample dataset, multiple Element/Sub-Element combinations correspond to wall finishes:

Element_Code	Element	Sub_Element_Code	Sub_Element
53	External	23	Primary Wall Finish
53	External	30	Secondary Wall Finish
120	WALLS	2	Wall Finish

A single property may contain records for all three combinations, and each combination may appear multiple times.

For example, the property at 55 Burnaby Street, London contains entries for all three of the above combinations. However, it contains only a single entry for “WALLS: Wall structure”, indicating that the property has only one structure rather than multiple.

This pattern is also observed in other sampled properties. Based on this, the following assumption is applied:

• “Secondary” refers to a secondary finish, not a secondary wall.

As a result:

• The property is mapped to a single Wall element.
• That Wall element is assigned three Finish aspects:
  • Two with aspect_instance = 1
  • One with aspect_instance = 2

This means that the combination of
UPRN / ElementType / ElementInstance / AspectType / AspectInstance
is not guaranteed to be unique.

LBWF Data – Wall Finish Mapping

In the LBWF dataset, the following element codes map to wall finishes:

• EXTWALLFN1
• EXTWALLFN2

These are similarly mapped as multiple instances of the Finish aspect for a single Wall element.

---