Model/docs/migrations/epc-property-round-trip-fidelity.md

# EPC persistence schema gaps — migrations for round-trip fidelity

**Context:** Slice 1 (Hestia-Homes/Model#1129) of the `ara_first_run` rebuild. The round-trip
fidelity test (`EpcPropertyData → epc_property tables → reload → EpcPropertyData`, deep-equality)
surfaced that the current `epc_property` schema stores only a **partial, partly type-lossy
projection** of the `EpcPropertyData` domain object. This document lists every gap and the
migration needed to close it, so the schema (FE-owned for some tables) can be updated.

We can make the column/table changes on the **SQLModel definitions** in
`infrastructure/postgres/epc_property_table.py` directly — tests build their schema from those
models via `SQLModel.metadata.create_all`, so they don't need the live DB. The live migrations
listed here are what must be applied wherever the physical tables are owned.

**`epc_cache` relationship:** the raw gov-API JSON response is retained in the `epc_cache` table,
so the *source* is always recoverable even where the structured `epc_property` projection is
lossy. That makes these gaps "the structured store is incomplete" rather than "data is lost
forever" — but the modelling pipeline reads the structured `epc_property`, not the raw cache, so
the gaps below still block faithful modelling and must be closed.

Priority key: **P0** modelling needs it now · **P1** needed soon · **P2** completeness.

---

## Status after Slice 1 (#1129)

The round-trip test passes over the persisted projection for RdSAP-Schema-21.0.0 and 21.0.1.
The following were **applied on the SQLModel** (`infrastructure/postgres/epc_property_table.py`)
and **still require the matching DB migration** wherever the physical tables live:

- **§1 JSONB** — all `Union` code columns converted (`epc_property`: `heating_cylinder_size`,
  `heating_immersion_heating_type`, `heating_cylinder_insulation_type`,
  `heating_secondary_heating_type`, `heating_shower_outlet_type`, `energy_pv_connection`;
  `epc_main_heating_detail`: `main_fuel_type`, `heat_emitter_type`, `emitter_temperature`,
  `main_heating_control`; `epc_building_part`: `wall_construction`, `wall_insulation_type`,
  `party_wall_construction`, `flat_roof_insulation_thickness`, `roof_insulation_location`,
  `roof_insulation_thickness`; `epc_window`: `glazing_gap`, `orientation`, `window_type`,
  `glazing_type`, `window_location`, `window_wall_type`, `draught_proofed`,
  `permanent_shutters_present`, `transmission_data_source`).
- **New scalar columns** — `epc_property`: `heating_number_baths`, `heating_number_baths_wwhrs`,
  `heating_electric_shower_count`, `heating_mixer_shower_count`,
  `mechanical_vent_duct_insulation_level`, `addendum_stone_walls`, `addendum_system_build`,
  `addendum_numbers` (JSONB), `ventilation_present`, `ventilation_sheltered_sides`,
  `ventilation_has_suspended_timber_floor`, `ventilation_suspended_timber_floor_sealed`,
  `ventilation_has_draught_lobby`, `ventilation_air_permeability_ap4_m3_h_m2`,
  `ventilation_mechanical_ventilation_kind`; `epc_building_part`: `roof_construction_type`,
  `curtain_wall_age`.
- **§2.1 `epc_renewable_heat_incentive` table** (#1137) — now created on the SQLModel and wired
  into save/get; the round-trip test asserts **full deep-equality** (no exclusion). DB migration
  still required.

**Still open (follow-up issues):** the remaining §2 structural tables (room-in-roof detail, PV
arrays, roof windows) + §3 nested-wall fields (`SapAlternativeWall.u_value`/`wall_thickness_mm`) +
`SapFloorDimension` exposed-floor flags — none populated in the 21.0.0/21.0.1 fixtures, so latent
until a richer fixture exercises them.

---

## 1. Type fidelity — convert `Union[int, str]` code columns to JSONB

These columns hold SAP/RdSAP categorical codes that are **`int` from the gov API** and **`str`
from Site Notes** (`Union[int, str]` in the domain). The forward mapper currently coerces them
with `str(...)` (and `bool(...)` for two window flags), so an API `int` of `26` is stored as
`"26"` and cannot be recovered. Convert each to **JSONB** and drop the `str()`/`bool()` coercion
in the forward mapper so the Python type round-trips exactly (JSON scalars preserve `int` vs
`str` vs `bool` vs `null`). **P0** — these feed the SAP10 calculator's int-keyed dispatch.

| Table | Columns |
|---|---|
| `epc_property` | `heating_cylinder_size`, `heating_immersion_heating_type`, `heating_cylinder_insulation_type`, `heating_secondary_heating_type`, `heating_shower_outlet_type`, `energy_pv_connection` |
| `epc_main_heating_detail` | `main_fuel_type`, `heat_emitter_type`, `emitter_temperature`, `main_heating_control` |
| `epc_building_part` | `wall_construction`, `wall_insulation_type`, `party_wall_construction`, `flat_roof_insulation_thickness`, `roof_insulation_location`, `roof_insulation_thickness` |
| `epc_window` | `glazing_gap`, `orientation`, `window_type`, `glazing_type`, `window_location`, `window_wall_type`, `draught_proofed`, `permanent_shutters_present` |

(`energy_meter_type` and `energy_wind_turbines_terrain_type` are `str` in the domain — leave as
`TEXT`.)

---

## 2. Not stored at all — new tables

### 2.1 `epc_renewable_heat_incentive` — **P0**
Maps `EpcPropertyData.renewable_heat_incentive` (`RenewableHeatIncentive`). Carries the **baseline
space-heating and hot-water kWh** that EPC Energy Derivation consumes — the single most important
gap. One row per `epc_property`.

| Column | Type | Source |
|---|---|---|
| `epc_property_id` | FK → `epc_property.id`, unique | |
| `space_heating_kwh` | float | `space_heating_kwh` |
| `water_heating_kwh` | float | `water_heating_kwh` |
| `impact_of_loft_insulation_kwh` | float, null | `impact_of_loft_insulation_kwh` |
| `impact_of_cavity_insulation_kwh` | float, null | `impact_of_cavity_insulation_kwh` |
| `impact_of_solid_wall_insulation_kwh` | float, null | `impact_of_solid_wall_insulation_kwh` |

### 2.2 `epc_room_in_roof` (+ `epc_room_in_roof_surface`) — **P1**
`SapBuildingPart.sap_room_in_roof` (`SapRoomInRoof`) is currently flattened to just
`room_in_roof_floor_area` + `room_in_roof_construction_age_band` on `epc_building_part`, dropping
the Type-2 geometry and the Detailed-measurement surfaces. Replace with a child table of
`epc_building_part`:

`epc_room_in_roof`: `epc_building_part_id` (FK, unique), `floor_area`, `construction_age_band`,
`common_wall_length_m`, `common_wall_height_m`, `gable_1_length_m`, `gable_1_height_m`,
`gable_2_length_m`, `gable_2_height_m`.

`epc_room_in_roof_surface` (0..n per RIR, from `detailed_surfaces: List[SapRoomInRoofSurface]`):
`epc_room_in_roof_id` (FK), `kind`, `area_m2`, `insulation_thickness_mm` (null),
`insulation_type` (null), `u_value` (null).

### 2.3 `epc_photovoltaic_array` — **P1**
`SapEnergySource.photovoltaic_arrays: List[PhotovoltaicArray]` (measured PV) is not stored at all
— only the `percent_roof_area` fallback is. One row per array: `epc_property_id` (FK),
`peak_power`, `pitch`, `orientation`, `overshading`.

### 2.4 `epc_roof_window` — **P2**
`EpcPropertyData.sap_roof_windows: List[SapRoofWindow]` not stored. One row per roof window:
`epc_property_id` (FK), `area_m2`, `u_value_raw`, `orientation`, `pitch_deg`, `g_perpendicular`,
`frame_factor`.

---

## 3. Not stored at all — new columns

### 3.1 `epc_property` additions
| Column | Type | Source | Pri |
|---|---|---|---|
| `addendum_stone_walls` | bool, null | `addendum.stone_walls` | P2 |
| `addendum_system_build` | bool, null | `addendum.system_build` | P2 |
| `addendum_numbers` | JSONB, null | `addendum.addendum_numbers` (`List[int]`) | P2 |
| `lzc_energy_sources` | JSONB, null | `lzc_energy_sources` (`List[int]`) | P2 |
| `solar_hw_collector_orientation` | text, null | `solar_hw_collector_orientation` | P1 |
| `solar_hw_collector_pitch_deg` | int, null | `solar_hw_collector_pitch_deg` | P1 |
| `solar_hw_overshading` | text, null | `solar_hw_overshading` | P1 |
| `extract_fans_count` | int, null | top-level `extract_fans_count` (distinct from the `ventilation_*` one) | P2 |
| `mechanical_vent_duct_insulation_level` | int, null | `mechanical_vent_duct_insulation_level` | P2 |

### 3.2 `epc_building_part` additions
| Column | Type | Source | Pri |
|---|---|---|---|
| `roof_construction_type` | text, null | `roof_construction_type` (Site-Notes str) | P1 |
| `curtain_wall_age` | text, null | `curtain_wall_age` (RdSAP §5.18) | P1 |
| `alt_wall_1_u_value` | float, null | `sap_alternative_wall_1.u_value` | P1 |
| `alt_wall_1_thickness_mm` | int, null | `sap_alternative_wall_1.wall_thickness_mm` | P1 |
| `alt_wall_2_u_value` | float, null | `sap_alternative_wall_2.u_value` | P1 |
| `alt_wall_2_thickness_mm` | int, null | `sap_alternative_wall_2.wall_thickness_mm` | P1 |

### 3.3 `epc_floor_dimension` additions
| Column | Type | Source | Pri |
|---|---|---|---|
| `is_exposed_floor` | bool, default false | `SapFloorDimension.is_exposed_floor` | P1 |
| `is_above_partially_heated_space` | bool, default false | `SapFloorDimension.is_above_partially_heated_space` | P1 |

---

## 4. Mapper-only gaps (no schema change required)

The table can already hold these; the **save mapper** simply doesn't write them. Fix in the
forward mapper, not the DB:

- **`air_tightness`** (`EnergyElement`) — `epc_energy_element.element_type` is a free string, so add
  an `"air_tightness"` element type to the save loop. **P1.**

---

## 5. Scope note

Slice 1 (#1129) asserts faithful round-trip over the **projection the schema is meant to store**,
after applying §1 (JSONB) and the straightforward §3/§4 additions on the SQLModel. The structural
new tables in §2 (RHI, room-in-roof, PV arrays, roof windows) are tracked as their own follow-up
issues — `epc_renewable_heat_incentive` (§2.1) first, as it unblocks EPC Energy Derivation. Each
gap above should become a checkbox on the relevant issue so nothing is silently dropped.