junte/Model
1
0
Fork 0
mirror of https://github.com/Hestia-Homes/Model.git synced 2026-06-30 13:10:47 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Merge pull request #1251 from Hestia-Homes/feature/hyde_make_it_more_accurate_with_tests

Browse source 
Feature/hyde make it more accurate with tests
This commit is contained in:
Jun-te Kim 2026-06-22 10:04:28 +01:00 • committed by GitHub
commit 2afa7acea4
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
128 changed files with 6743 additions and 533 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

23
.claude/skills/epc-to-elmhurst-rdsap-inputs/reference/mapping.md
View file

@ -180,12 +180,23 @@ Table 32 unit costs, p/kWh (`domain/sap10_calculator/tables/table_32.py`):
**`main_fuel_type` / `water_heating_fuel` 29 = off-peak (7-hour) electricity** →
Elmhurst Electricity meter type = **Dual-rate / Economy 7 (7-hour)**, NOT Single.
⚠️ **Known over-rating bug:** the engine prices **100% of off-peak space heating
AND hot water at the 5.50p low rate** (`inputs.space_heating_fuel_cost_gbp_per_kwh`
= 0.055), instead of the SAP **Table 12a high/low split** (a portion at the 15.29p
high rate). This under-costs all-electric Economy-7 dwellings and inflates the SAP
score. Always surface this in the output's "Known divergences". Canonical case:
UPRN 10002468137 — lodged 55, engine 62.
✅ **Economy-7 high/low split — FIXED (PR #1217).** The engine now applies the SAP
**Table 12a Grid 1** (space) + **Table 13** (immersion DHW) high/low split rather
than pricing 100% at the 5.50p low rate. Electric STORAGE heaters legitimately get
a 0.00 SH high-rate fraction (100% low — spec value, not a bug); immersion HW takes
the cylinder-volume/occupancy/single-dual Table 13 blend (applied in
`cert_to_inputs._hot_water_fuel_cost_gbp_per_kwh` when volume + occupancy +
single/dual are all resolved; absent any of them it still falls back to 100% low —
a rarer edge). Verified: canonical UPRN 10002468137 engine 60.92 = Elmhurst 61 to
the penny (its lodged 55 is the OLD SAP-2012 schema, not comparable); UPRN
10022893721 engine 79 = lodged 79, Elmhurst (Dual meter) 81.
⚠️ **When building in Elmhurst you MUST set the Economy-7 meter** (`main_fuel_type`
/ `water_heating_fuel` 29 = off-peak 7-hour → Electricity meter type **Dual**, NOT
Single). Elmhurst silently defaults to Single/Standard and prices at the 13.19p
standard rate, collapsing the worksheet SAP ~13 points — which can masquerade as an
engine "over-rating". The control is a hidden Meters sub-tab on the SpaceHeating
page (`TabPanelMeters_RadioButtonListElectricityType`).
## Water Heating

253
.claude/skills/expand-sap-accuracy-corpus/SKILL.md
View file

@ -19,6 +19,74 @@ score the property would get unchanged. Elmhurst is the accredited ground truth;
its Input Summary (parsed back to `EpcPropertyData`) exposes mapper holes, and its
worksheet exposes calculator holes.
## ⏩ Resume in a fresh context (autonomous run)
If the user says "continue" / "next" / "keep going through the worklist", run this
loop **continuously without asking between certs** — report only `eng X / elm Y`
per cert and tick the worklist line as each is pinned. The build automation is
fully working (see `scripts/hyde/elmhurst_lib.py` helpers + the `build_<uprn>.py`
templates); most certs build unattended end-to-end.
**State (2026-06-19):** pinned this campaign — SAP-17.1 cohort + RdSAP-17.1/18.0
(older) plus: **16.1** `100021943298` (76/75), **19.1.0** `10096028301` (82/82),
**16.3** `44012843` (79/78), **17.0** `10023444324` (80/80) + `10023444320`
(81/82), **RdSAP-20.0.0** `10090844932` (78/77), **16.2** `100090182288` (69/71,
semi house). Latest run (2026-06-19): **16.2** `100021985993` (74/72, end-terrace
bungalow), **17.1** `10091568921` (82/80, full-SAP end-terrace house combi 17615)
+ `10093718424` (81/80, semi sibling), **RdSAP-18.0** `10022893721` (79/81, first
NON-BOILER cert — electric storage heaters + immersion; storage-heater automation
now SOLVED incl. the Economy-7 Dual-meter step, see banked findings; engine 79 =
lodged, NO bug), **RdSAP-21.0.1** `10023443426` (76/79, native schema, combi house;
engine 76 = lodged EXACTLY; Elmhurst +3 = omitted-secondary build gap). Next `[ ]`:
`10093412452` (SAP-17.1), then `10090343335` (17.0) / `10093115480` (17.1) /
`68151071` (RdSAP-17.0). Skip `100020933699` (user said skip), `[⛔]` (NOT
MAPPABLE), `[⚠]` (flagged engine bugs: MVHR / heat-pump fuel-39).
**Per-UPRN recipe** (all commands `DISPLAY=:99`, cwd `scripts/hyde`; run
`bash scripts/hyde/start_viewer.sh` once; creds in `.elmhurst-creds.json`; shared
assessment GUID `B44A0DB4-4C08-4241-B818-86F060172105`):
1. `PYTHONPATH=/workspaces/model python scripts/fetch_real_life_epc_sample.py <uprn>`
then scope: dwelling_type/built_form, age band, walls/roof/floor descriptions,
heating `main_heating_index_number`/category/`has_hot_water_cylinder`, window
total area (`sum(sap_windows w*h)`), party_wall_length, lighting %, MEV/AP50.
2. **Copy the closest `build_<uprn>.py` template** and adjust values:
- combi flat → `build_100021943298.py`; regular-boiler+cylinder flat →
`build_44012843.py`; full-SAP combi flat (MEV+AP50) → `build_10096028301.py`
/ `build_10023444324.py` (+party wall) / `build_10023444320.py` (mid-floor);
- combi house → `build_10090844932.py` (end-terrace, party wall) /
`build_100090182288.py` (semi, no party wall).
Adjust: property type/built-form, band (`_pick` by year, e.g. "1950"/"2012"/
"2023"), two-floor dims + party wall, wall insulation, roof, floor, window m²,
doors, lighting, boiler PCDB ref + search query, MEV/AP50 if present.
3. Run pages: `for p in property_description [flats] dimensions walls roofs floors
openings ventilation; do … build_<uprn>.py $p; done` (one Save&Close each,
~1 min/page; flats only for Flat property type). Then a window-verify/fix
snippet (re-add the combined window if the grid shows 0.00), then
`build_<uprn>.py space_heating` and `water_heating`.
4. Heating uses the `elmhurst_lib` helpers: `E.select_boiler(page, "<model>",
"<pcdb_id>")` (look up id/type in `domain/elmhurst/pcdb_gas_oil_boiler_codes.csv`;
the lodged `main_heating_index_number` IS the id); control
`E.set_heating_dialog(page, "…ButtonMainHeatingControls", "^Boilers",
"^Standard", "CBE Programmer, room thermostat and TRVs")` (=2106); water
`E.set_heating_dialog(page, "…ButtonWaterHeatingCode", "From Space Heating",
"From the primary heating system")`; combi → `E.clear_hot_water_cylinder(page)`.
5. Download: edit `elmhurst_download.py` `SAMPLE_DIR` to the cert's
`<schema>/uprn_<uprn>` dir; first confirm Recommendations is clean (parse
`[id*=ContentPlaceHolder1] a` link text — Summary silently redirects to Address
until zero errors); `python scripts/hyde/elmhurst_download.py` (retry once; the
nav goes Address→Recommendations→Summary).
6. `PYTHONPATH=/workspaces/model python scripts/compare_epc_paths.py <uprn>` →
read **"gov-API inputs → SAP"** (engine) and **"Elmhurst's OWN engine
(worksheet …)"** (Elmhurst ground truth). Target ≤0.5–1.
7. **Pin** the engine value: add a `RealCertExpectation(schema, sample=uprn_<uprn>,
cert_num, sap_score=<engine>)` in `tests/.../test_real_cert_sap_accuracy.py`;
run `…::test_real_cert_sap_score`.
8. Tick the worklist line `[x] … · eng X / elm Y (lodged Z) · PINNED …`. Next cert.
See **Banked findings** below for the modal-dialog mechanics (all already encoded
in the helpers). New schema not mappable → add a dedicated `from_*_schema_*`
mapper first (per-schema convention) + guard with the RdSAP-21.0.1 corpus gauge.
## The loop (one UPRN)
1. **Pick** the first `[ ]` UPRN in [worklist.md](worklist.md).
@ -93,6 +161,112 @@ Pattern: `with E.session() as (ctx,page): E.goto(...); E.set_text/set_select(...
lines 17/18). This drove the first campaign mapper fix — see Banked findings.
## Banked findings (fold new ones in here as the corpus grows)
- **MAPPER GAP — cylinder insulation thickness dropped (RdSAP-17.0+):** the mapper
carries `cylinder_size` + `cylinder_insulation_type` but NOT
`cylinder_insulation_thickness` → `EpcPropertyData.sap_heating.cylinder_
insulation_thickness_mm` stays None even when the cert lodges it (e.g. 50 mm).
The engine then assumes a poorly-insulated cylinder → over-counts HW cylinder
loss → under-rates. Confirmed uprn_68151071 (raw 50 mm → mapped None; engine HW
3446 vs Elmhurst 2911 kWh; engine 68 vs lodged 70 / Elmhurst 71). FIX: map the
thickness in the RdSAP per-schema mapper; check blast radius (any pinned cylinder
cert may shift) + regress the RdSAP-21.0.1 corpus gauge. Leverage point — likely
improves every cylinder-with-lodged-thickness cert. Flagged, not yet fixed.
- **Party-wall type `_pick` gotcha:** matching `"filled"` ALSO matches "Cavity
masonry **UN**filled" (CU, U≈0.5) — the wrong type for a cert whose party wall is
U≈0. Match `"masonry filled"` to hit CF (Cavity masonry filled, U≈0). Affects
cavity builds (10090844932 / 10091568921 / 10093718424 / 10093412452 used the
loose `"filled"` and may have got CU in Elmhurst — only the Elmhurst cross-check,
not the pinned engine value which is validated against lodged). Fixed for
uprn_10093115480.
- **Shared-assessment reset: storage/electric → boiler cert.** The shared
assessment carries the PRIOR cert's heating system. Going storage→combi, the
boiler search dialog won't open while a SAP-table `MainHeatingCode` (e.g. SEB) is
set. Fix: JS-clear `MH1.TextBoxMainHeatingCode` to `''` + dispatch change, Save &
Close, then `select_boiler` works. Also RESET the electricity meter on the Meters
sub-tab (a prior off-peak cert leaves it Dual; gas certs want Single) and the
SECONDARY heating (a prior cert's secondary persists even when the calc shows
presence=No — set it explicitly or it pollutes the worksheet).
- **Secondary heating must be built in Elmhurst when lodged.** Certs lodge a
secondary (`sap_heating.secondary_heating_type`, e.g. 612 = mains-gas room heater
@ Table 4a seasonal efficiency 0.20 — a low-eff decorative/old gas fire). The
engine models it (fraction 0.1 from Table 11 ÷ the secondary efficiency → e.g.
3065 kWh for code 612); omitting it in Elmhurst inflates the worksheet (uprn_
10023443426: omitted → 79 vs engine/lodged 76). Build via Secondary present=Yes →
`ButtonSecondaryHeatingCode` cascade (title "Select secondary heating"): fuel →
sub-fuel → appliance → type, e.g. Gas → Mains gas → Room Heaters → RGx (pick the
RGx whose efficiency matches the lodged code, ~0.20 = decorative/old gas fire).
⚠ For a NATIVE RdSAP-21.0.1 cert, engine = lodged (exact, all components) is the
authoritative validation — if the Elmhurst rebuild diverges, suspect an omitted
lodged feature (secondary / meter), confirm via engine-on-Elmhurst-inputs ≈
worksheet, and pin the engine = lodged value.
⚠ **`present=No` does NOT clear the shared assessment's leftover secondary** — the
prior cert's secondary SYSTEM (fuel + SAP code) persists server-side even when the
UI dropdown reads "No", and it silently re-enters the worksheet. On a storage-heater
cert (uprn_100020665611, RdSAP-20.0.0 end-terrace house) a leftover House-Coal
closed-room-heater (SAP 633, cheap solid fuel) inflated the Elmhurst worksheet to
44 vs engine 36 / lodged 37 until OVERWRITTEN. Always set the secondary EXPLICITLY
to the lodged appliance: storage-heater certs lodge "Portable electric heaters
(assumed)" → present=Yes → `ButtonSecondaryHeatingCode` cascade Electric → Electric
→ Room Heaters → "REA Panel, convector or radiant heaters" (= SAP 691, eff 100%).
That dropped the worksheet 44 → 35 ≡ engine-on-Elmhurst-inputs 35 (faithful).
- **Openings: standard double/triple-glazing bands REQUIRE Frame Type + Glazing Gap.**
The window grid's `DropDownListExtFrameType` (PVC/Wood/Metal) and
`DropDownListExtGlazingGap` (6 mm / 12 mm / 16 mm or more) are required for any
band other than the new-build "Double post or during 2022" default — leaving them
empty fails the Recommendations gate ("Openings: Frame Type / Glazing Gap must be
entered"). Set both in `_add_window` BEFORE clicking Add (cert `pvc_window_frames`
→ PVC; `glazing_gap` mm → the band). Glazing bands available: Single / Double|Triple
{pre 2002, between 2002 and 2021, post or during 2022, with unknown install date} /
Secondary / *…known data*. For RdSAP double-glazing with no lodged install year
(engine U 2.8) pick "Double pre 2002" (~U3.1, sub-1-SAP diff) — the "…known data"
options demand per-row U/g values.
- **Non-boiler (storage-heater) main heating — SOLVED** (uprn_10022893721, RdSAP-
18.0 GF flat, electric storage heaters + immersion). The SpaceHeating page has NO
inline system-type selector and a `ButtonMainHeatingCode` button only APPEARS
once the bound PCDF boiler is cleared. Two-pass recipe (one Save & Close each):
1. **Clear the leftover PCDB boiler**: set `MH1.TextBoxPCDFBoilerReference` to
`"0"` via JS + dispatch `change`, then `save_close`. (It doesn't AutoPostBack;
the Save commits it. After reload, boilerRef="0", the boiler fuel/flue fields
vanish, and `MH1.ButtonMainHeatingCode` is now present.)
2. **Select the SAP-table system** via `set_heating_dialog(..ButtonMainHeating
Code, "^Electric","^Electric","Storage","SEB Modern slimline")` (title "Select
heating code"; L1 Gas/Oil/Solid Fuel/Electric/Community/No heating; storage L4
= SEA old large-volume / SEB modern slimline / SED fan / SEJ integrated / SEK
high-heat-retention). **Match the cert's `sap_main_heating_code`**: 402 = SEB.
Then the CONTROL via `set_heating_dialog(..ButtonMainHeatingControls,
"Storage Radiator","CSA Manual charge control")` (= SAP 2401). Secondary No.
Water: `set_heating_dialog(..ButtonWaterHeatingCode, "Water Heater","^Electric",
"Immersion")` (→ code HEI) — then the immersion code REQUIRES a cylinder: CHECK
`WH.CheckBoxHotWaterCylinder` (JS click+change, AutoPostBacks) or Recommendations
errors "must have a Hot Water Cylinder"; set `WH.DropDownListCylinderSize`
(size 2→Normal/110L, 3→Medium/160L, 4→Large/210L), `WH.DropDownListInsulated`
(Foam/Jacket), `WH.DropDownListInsulationThickness`, and `WH.RadioButton
ListImmersionHeater` (off-peak meter → Dual). See `build_10022893721.py` as the
template. ⚠ **CRITICAL — set the electricity meter type.** All-electric off-peak
certs MUST set the Economy-7 meter or Elmhurst silently defaults to Single /
Standard and prices everything at the 13.19p standard rate (worksheet collapses
~13 SAP). The control is a HIDDEN sub-tab on the SpaceHeating page:
`E.click_tab(page,"TabContainer_TabPanelMeters")` then `E.set_select(page,
"TabContainer_TabPanelMeters_RadioButtonListElectricityType","Dual")` (options
Single/Dual/18 Hour/24 Hour/Unknown; cert `meter_type` 1→Dual=7-hour off-peak).
Verify in Elmhurst summary §14.2 "Electricity meter type" / worksheet
"Electricity Tariff" before trusting the worksheet SAP.
- **Economy-7 off-peak pricing is CORRECT — do NOT "fix" it** (was a real bug, FIXED
in PR #1217 via the Table 13 off-peak water-heating split + window-U fix). The
engine applies SAP Table 12a Grid 1 (space) + Table 13 (immersion DHW) high/low
splits properly: storage heaters' SH high-rate fraction is legitimately 0.00
(100% low rate), immersion HW takes the volume/occupancy/single-dual blend. Proof:
uprn_10002468137 (canonical) engine 60.92 = Elmhurst 61 to the penny;
uprn_10022893721 engine 79 = lodged 79, Elmhurst (Dual meter) 81. ⚠ If you see a
big engine-over-Elmhurst gap on an all-electric off-peak cert, SUSPECT THE BUILD
(Elmhurst meter left on Single — see meter step above), not the calculator. The
`reference/mapping.md` "known over-rating bug (engine 62)" note is STALE (pre-PR
#1217). `cert_to_inputs.py` `_hot_water_fuel_cost_gbp_per_kwh` applies the immersion
Table 13 blend when cylinder volume + occupancy resolve; when `immersion_heating_
type` is UNLODGED on an off-peak meter it now defaults to DUAL per RdSAP §10.5
(was a 100%-low-rate fallback that under-costs — fixed, +regression test
`test_off_peak_immersion_unlodged_type_defaults_to_dual_table13_blend`). Only an
unresolvable cylinder volume / occupancy still falls back to 100% low.
- **Air-permeability AP50 fix** (uprn_10093116528, the first campaign cert): the
full-SAP mapper was routing the lodged q50 to the engine's AP4/Pulse formula
(`0.263×AP4^0.924`) instead of the AP50 `/20` path — a big infiltration
@ -105,13 +279,78 @@ Pattern: `with E.session() as (ctx,page): E.goto(...); E.set_text/set_select(...
default. Pin the engine's observed value; document the Elmhurst delta (don't tune).
- **FGHRS** (full-SAP `has_fghrs`/`fghrs_index_number`) is dropped by the mapper
and not yet modelled — omit it on BOTH sides so the comparison stays clean.
- **Validation errors live on the Recommendations page as LINKS** (red ✗ anchors),
not coloured spans — parse `[id*=ContentPlaceHolder1] a` text, not CSS colour.
Until Recommendations shows ZERO errors the **Energy Report Summary nav silently
redirects to the Address page** and the worksheet/Results PDFs won't generate
(Results renders empty). Two errors that bit the 16.1/19.1.0 builds: *"Walls
(Main): Insulation Thickness must be entered"* (set the insulation-thickness
dropdown — "Unknown" is fine for a reduced cert) and *"Incorrect Controls (NNNN
ctrl-group-X) for Heating System (idx ctrl-group-Y)"* (the heating control's
system-type must match the boiler's — see the controls dialog below).
- **All Elmhurst modal dialogs sit above a `modalBackground` that defeats element
clicks** (Playwright sees it "intercepting pointer events", even with force).
The cracked pattern is now in `elmhurst_lib.py`: open via JS click, set cascade
`<select>`s by JS (`.value` + dispatch `change` → drives the AutoPostBack), and
COMMIT by `page.mouse.click(x,y)` at the OK/Select control's centre (coordinate
clicks hit the topmost element = the dialog). Helpers: `E.select_boiler(page,
query, ref)`, `E.set_heating_dialog(page, button_suffix, *level_regexes)` (water
method + main-heating controls), `E.dialog_commit(page, label)`,
`E.clear_hot_water_cylinder(page)`.
- **Boiler by exact PCDB code** (eliminates the boiler-efficiency divergence):
`E.select_boiler(page, "<brand/model search>", "<pcdb_id>")` drives the search
dialog two-step (click the result ROW to highlight → click the top **Select**
span). The lodged `main_heating_index_number` IS the `pcdb_id`. Look up the Ref/
type/efficiency in **`domain/elmhurst/pcdb_gas_oil_boiler_codes.csv`**. NB:
typing the number straight into the *PCDF boiler Reference* box is COSMETIC — it
doesn't re-resolve the boiler; only the search dialog sets type/efficiency. A
combi keeps the cylinder checked from a prior regular boiler → `clear_hot_water_
cylinder()`, then set water method "From Space Heating → primary" via
`set_heating_dialog(..ButtonWaterHeatingCode, 'From Space Heating', 'From the
primary heating system')`.
- **Reduced-field (16.x) semi/terraced party wall.** 16.x lodges no
`party_wall_length`, so the engine models NO party wall (defaults 0). But
Elmhurst REJECTS a semi/terraced with a zero party wall ("At least one building
part must have a party wall length of non zero"). Enter the geometry-derived
length (solve `2w+d=heat_loss_perimeter`, `w*d=floor_area`; party wall = the
joined side `d`) on every floor + a filled-cavity party-wall type (CF), and
EXPECT engine to over-read Elmhurst by ~1–2 SAP (engine's no-party-wall vs
Elmhurst's forced one). That delta is the documented reduced-field gap, not a
bug — confirm via engine-on-Elmhurst-inputs ≈ worksheet. Pin the engine value.
- **Main-heating control must match the boiler's system type.** Heat-pump leftover
control (e.g. CHJ/2210, ctrl-group-2) is invalid for a gas boiler (ctrl-group-1)
and blocks the report. For a boiler programmer+room-stat+TRVs (SAP 2106):
`set_heating_dialog(..ButtonMainHeatingControls, '^Boilers', '^Standard', 'CBE
Programmer, room thermostat and TRVs')`.
⚠ **Zone-control codes are under L2 "Zone control", NOT "Standard".** SAP 2110
"Time and temperature zone control" = `set_heating_dialog(.., '^Boilers', 'Zone
control', 'CBI Time and temperature zone control')` (Boilers→Standard L3 only has
CBA…CBH — no zone option, so a wrong L2 leaves the cascade incomplete and the OK
click is swallowed by the modalBackground, hanging the run). CBL is the PCDF
variant. Match the lodged `main_heating_control`: 2110 → CBI, 2106 → CBE (Standard).
- **Removing an inherited secondary when the cert lodges NONE** — `present=No` ALONE
does NOT remove it: a persisted secondary CODE (e.g. an electric REA/691 from a
prior cert) survives the dropdown=No and the worksheet still applies it at Table-11
fraction 0.1 (worth ~1.5 SAP on a gas-combi cert — the electric secondary is priced
at the 13.19p peak rate vs gas 3.48p). The flag and the code are independent; the
worksheet keys off the CODE. ✅ FIX: `present=Yes` (exposes `TextBoxSecondaryHeating
Code`) → JS-clear that textbox to `''` + dispatch change → `present=No` → Save&Close.
An EMPTY code drops the worksheet's "Secondary Heating" to None / fraction 0.0.
Verify by re-downloading: worksheet line (201) "Fraction of space heat from
secondary" must read 0.0000. Seen on uprn_10090944225 (worksheet 79→80; closed the
cross-check gap to the residual +1 floor-U difference).
## Limitations / next improvements (make the campaign scale)
- **Per-assessment GUID** — `elmhurst_lib.py` reuses one `ASSESSMENT_GUID`
(Khalim-test), overwritten per UPRN. For parallelism, create one assessment per
UPRN and parameterize the GUID.
- **Data-driven fill** — `elmhurst_lib.py` gives the primitives; a per-cert driver
that reads field values straight from `elmhurst_inputs.md` (or a JSON sidecar)
would remove the remaining hand-keying. `elmhurst_fill.py` is the older template.
- **More dialogs** — boiler cascade is automatable; the control-code (2110) and
FGHRS/MV-unit search dialogs still need the open→search→select→OK pattern.
(Khalim-test), overwritten per UPRN. ⚠ This is a SHARED assessment: a concurrent
user (Khalim) overwrote a build mid-session once — coordinate before long runs,
or (better) create one assessment per UPRN and parameterize the GUID.
- **Data-driven fill** — `build_<uprn>.py` per-cert scripts now drive every page
incl. all dialogs via the `elmhurst_lib` helpers (see build_100021943298.py /
build_10096028301.py as templates). A generic driver reading `elmhurst_inputs.md`
is the remaining step. `elmhurst_fill.py` is the older single-cert template.
- **Per-schema mappers** — every SAP/RdSAP schema gets its own dedicated
`from_*_schema_*` mapper in `datatypes/epc/domain/mapper.py` (the 16.x family
shares `_normalize_sap_schema_16_x`; full-SAP revisions delegate to
`from_sap_schema_17_1`). Add new schema coverage as a dedicated method + dispatch
branch, never by tuple-stuffing. Guard with the RdSAP-21.0.1 corpus gauge.

50
.claude/skills/expand-sap-accuracy-corpus/worklist.md
View file

@ -4,12 +4,26 @@ Process **one at a time**, top to bottom (see [SKILL.md](SKILL.md)). After each
UPRN, tick it and annotate: `— <schema> · eng <X> / elm <Y> · <note>`.
**Legend:** `[ ]` todo · `[x]` pinned (≤0.5) · `🔧` mapper extended · `⚠` xfail
(engine bug) · `⛔` blocked.
(engine bug) · `⛔` blocked · `[🔍]` flagged — engine vs Elmhurst-worksheet gap too
large, or a new/complex build pattern needed; NEEDS INVESTIGATION.
**Worked reference (not in the 100):** `uprn_10092973954` (SAP-Schema-17.1,
2020 new-build flat) — full loop proven: eng 77 / elm 78, engine-on-Elmhurst-
inputs 79 (calculator faithful within ~1). Use it to sanity-check the pipeline.
## Do next — new schemas (need a mapper)
These two were flagged NOT MAPPABLE in the UI (red ✗). Mapper coverage now ADDED
(dedicated per-schema methods in `EpcPropertyDataMapper`; corpus gauge green, 0
new pyright errors). Elmhurst build + pin still pending — run the normal loop.
- [x] 🔧 10096028301 — SAP-Schema-19.1.0 (full-SAP g/f FLAT, band M, combi PCDB 17929, MEV, AP50 3.5) · eng 82 / elm 82 (lodged 85) · PINNED engine 82. 🔧 mapper added: `from_sap_schema_19_1_0`. Built in Elmhurst (boiler 17929 via search, control CBE/2106, water from primary, MEV on, AP50 Blower Door 3.5+cert). Engine EXACTLY matches Elmhurst worksheet (82.11 vs 82); engine-on-Elmhurst-inputs 82.16 ≈ 82 → calculator faithful. −3 vs lodged = measured-U-vs-RdSAP-default + MEV extract-not-recovery (documented). No mapper change beyond coverage.
- [x] 🔧 100021943298 — SAP-Schema-16.1 (g/f FLAT, band B, solid-brick internal, combi PCDB 10328) · eng 76 / elm 75 (lodged 72) · PINNED engine 76. 🔧 mapper added: `from_sap_schema_16_1`. Built in Elmhurst (boiler 10328 via search, control CBE/2106, water from primary, wall insulation thickness Unknown); worksheet 75 → engine within ~1 (tightest agreement, reduced-field). Boiler-select + water-heating + control dialogs all driven via automation (two-step row→Select / cascade + coordinate-OK). No mapper change beyond coverage.
(Refactor: split the full-SAP + 16.x dispatch tuples into dedicated per-schema
mappers — from_sap_schema_17_0/18_0_0/19_1_0 + from_sap_schema_16_0/1/2/3 — so
each schema has one mapper home for future divergence; all delegate to shared logic.)
## E2E testing set
UPRNs needed for end-to-end testing (also tracked in The 100 below).
@ -41,28 +55,28 @@ Skip the 🚩 MVHR / 🚩 heat-pump-fuel and ⛔ sparse certs.
- [x] 10093116543 — SAP-17.1 (2017 gas-combi semi) · eng 81 / elm 77 (lodged 82) · PINNED engine 81. +4 = documented full-SAP→RdSAP residual, NOT a mapper bug: ~1.5 floor-U (cert lodges measured 0.11 vs Elmhurst RdSAP solid default 0.23; U-known override disabled) + ~1 boiler-eff (cert PCDB 17644 88.5% vs Elmhurst generic BGW combi 84%; PCDB search disabled, 89% cascade option is a regular boiler needing a cylinder) + ~0.5 roof band-L/infil. Conservatory leftover from prior cert cleared (worksheet 73→77). No mapper change.
- [x] 🔧 10093116529 — SAP-17.1 (2017 gas-combi ground-floor FLAT, TFA 49) · eng 81 / elm 78 (lodged 81) · PINNED engine 81. 🔧 FIXED a real calc bug: full-SAP flats took the 0.25 house party-wall default instead of the RdSAP Table-15 flat 0.0 (flatness is in dwelling_type, not property_type) — heat_transmission._is_flat_or_maisonette_dwelling; +regression test. Cert lodges party u_value 0; Elmhurst worksheet 0.0; fix 80→81. Residual +3 vs Elmhurst = documented full-SAP→RdSAP gap (measured wall 0.184/floor 0.12 + PCDB 88.5% vs generic 84%). Calculator faithful: fed Elmhurst's Us, HTC 93.4 vs ~94. House→Flat Elmhurst switch (storeys→1, roof→another-dwelling-above). No mapper change.
- [ ] 🔧 100020933699 — SAP-16.2 SCHEMA COVERAGE ADDED (end-terrace house, band G). 16.2 is structurally RdSAP-17.1 (reduced fields, glazed_area band, construction-code building parts) under a different name; mapped via `_normalize_sap_schema_16_2` (renames windows→window, main_gas→mains_gas, boiler_index_number→main_heating_index_number, wwhrs→instantaneous_wwhrs + defaults) → reuses from_rdsap_schema_17_1. 🔧 Also fixed: "Single glazed" description honoured when multiple_glazing_type="ND" (was defaulting to double; RdSAP-21 code 5) → eng 72→71. +4 regression tests, sap_16_2.json fixture, 0 new pyright errors. eng 71 / lodged 70. ⚠ Known gap: 16.2 lodges no party_wall_length → end-terrace party wall unmodelled (likely the residual +1). ⏳ Elmhurst build (partial: PropDesc/Dims/Walls/Roofs done) + pin still pending.
- [ ] 🔧 44012843 — SAP-16.3 schema coverage (same _normalize_sap_schema_16_x reduced-field path as 16.2) · eng 79 / lodged 81 · g/f flat band K · Elmhurst pin pending
- [ ] 🔧 10023444324 — SAP-17.0 schema coverage (full-SAP shape ≡ 17.1; dispatched to from_sap_schema_17_1, no normalization) · eng 80 / lodged 82 · Elmhurst pin pending
- [x] 🔧 44012843 — SAP-16.3 (g/f FLAT, band K, cavity insulated, REGULAR boiler PCDB 9895 + cylinder, double glazed) · eng 79 / elm 78 (lodged 81) · PINNED engine 79. Built in Elmhurst (boiler 9895 via search, control CBE/2106, cylinder Large/foam/50mm, water from primary). Engine within ~1 (78.82 vs 78); engine-on-Elmhurst-inputs 78.48 ≈ 78 → calculator faithful. −2 vs lodged = reduced-field defaults. No mapper change beyond coverage.
- [x] 🔧 10023444324 — SAP-17.0 (full-SAP g/f FLAT, band M, combi PCDB 16211, MEV, AP50 3.2, party wall 6.43) · eng 80 / elm 80 (lodged 82) · PINNED engine 80. Built in Elmhurst (boiler 16211 via search, control CBE/2106, water from primary, MEV on, AP50 Blower Door 3.2, party wall 6.43). Engine EXACTLY matches Elmhurst worksheet (80.13 vs 80); engine-on-Elmhurst-inputs 81.03 ≈ 80 → calculator faithful. −2 vs lodged = full-SAP→RdSAP residual (measured U + MEV). No mapper change beyond coverage.
- [⚠] 10092970673 — SAP-17.0 · eng 77 / lodged 86 · 🚩 MVHR idx 500418 not credited (flagged)
- [⚠] 10094601287 — SAP-18.0.0 · eng 80 / lodged 84 · 🚩 MVHR idx 500230 not credited (flagged)
- [ ] 10090844932 — RdSAP-20.0.0 · eng 78 / lodged 78
- [x] 10090844932 — RdSAP-20.0.0 (end-terrace HOUSE 2-storey, band L, combi PCDB 10327, party wall 4.93, 250mm loft) · eng 78 / elm 77 (lodged 78) · PINNED engine 78. Built in Elmhurst (House, boiler 10327 via search, control CBE/2106, water from primary, party wall 4.93 filled). Within ~1 (78.13 vs 77); engine-on-Elmhurst-inputs 77.24 ≈ 77 → faithful.
- [⛔] 10090844948 — SAP-16.3 · NOT MAPPABLE (ValueError: RdSapSchema17_1: missing required field )
- [ ] 100090182288 — SAP-16.2 · eng 71 / lodged 71
- [x] 100090182288 — SAP-16.2 (semi-detached HOUSE 2-storey, band D, filled cavity, combi PCDB 10327, double glazed) · eng 71 / elm 69 (lodged 71) · PINNED engine 71. Built in Elmhurst (semi, boiler 10327, control CBE/2106, water from primary, party wall 4.28 filled — Elmhurst requires non-zero for a semi). +2 vs Elmhurst = documented 16.2 reduced-field party-wall gap (gov-API lodges no party_wall_length → engine models none); engine-on-Elmhurst-inputs 69.35 ≈ worksheet 69 → calculator faithful.
- [⚠] 10093114053 — SAP-17.0 · eng 93 / lodged 79 · 🚩 heat-pump fuel-39 (flagged)
- [ ] 10091568921 — SAP-17.1 · eng 82 / lodged 85
- [ ] 10093718424 — SAP-17.1 · eng 81 / lodged 84
- [ ] 10022893721 — RdSAP-18.0 · eng 79 / lodged 79
- [ ] 10023443426 — RdSAP-21.0.1 · eng 76 / lodged 76
- [ ] 10093412452 — SAP-17.1 · eng 81 / lodged 84
- [x] 10091568921 — SAP-17.1 (full-SAP END-TERRACE HOUSE 2-storey, band L 2018, combi PCDB 17615 Potterton Promax Ultra, party wall 40.56m², natural vent + 3 extract fans, AP50 4.45, 20 LED, measured U 0.18/0.10/0.15) · eng 82 / elm 80 (lodged 85) · PINNED engine 82. Built in Elmhurst (end-terrace house, boiler 17615, CBE/2106, water from primary, party wall 8.45 filled, AP50 Blower Door 4.45+cert). +2 = documented full-SAP→RdSAP residual (measured U beats band-L defaults); engine-on-Elmhurst-inputs 80.16 ≈ worksheet 80 → faithful. has_hot_water_cylinder lodged true but combi PCDB + no cylinder detail → built combi. No mapper change.
- [x] 10093718424 — SAP-17.1 (full-SAP SEMI-DETACHED HOUSE 2-storey, band L, combi PCDB 17615, party wall 40.89m², natural vent + 3 extract fans, AP50 4.18, 20 LED, measured U 0.25/0.13/0.16; sibling of 10091568921) · eng 81 / elm 80 (lodged 84) · PINNED engine 81. +1 = documented full-SAP→RdSAP residual; engine-on-Elmhurst-inputs 80.12 ≈ worksheet 80 → faithful. Built combi (cylinder lodged true but combi PCDB). No mapper change.
- [x] 10022893721 — RdSAP-18.0 (GF FLAT, band I, cavity insulated, ELECTRIC STORAGE HEATERS SAP 402 SEB Modern slimline + manual charge CSA/2401, immersion off-peak dual + cylinder Normal/110L foam 50mm, party wall 21.48) · eng 79 / elm 81 (lodged 79) · PINNED engine 79 = lodged. Built in Elmhurst (storage SEB, CSA control, immersion dual + cylinder, **Dual electricity meter / Economy 7**). engine -2 vs Elmhurst, within tolerance; engine-on-Elmhurst-inputs 78.76 ≈ 79 → faithful. Economy-7 off-peak pricing is CORRECT (Table 12a/13, fixed PR #1217) — NOT a bug. ⚠ Earlier mis-build left Elmhurst meter on Single/Standard → bogus worksheet 66; fixed by setting Dual meter on the Meters sub-tab. 🔧 First non-boiler cert — SOLVED storage-heater automation (see banked findings: clear PCDB boiler → ButtonMainHeatingCode Electric→Electric→Storage→SEB; set Dual meter). build_10022893721.py complete.
- [x] 10023443426 — RdSAP-21.0.1 (END-TERRACE HOUSE 2-storey, band L, cavity insulated, combi PCDB 17045 Ideal Logic ES35, 300mm loft, party wall 9.2 lodged, 11 dbl-glazed windows ~10.3m², 9 LED, mains-gas room-heater SECONDARY code 612 eff 0.20) · eng 76 / elm 79 (lodged 76) · PINNED engine 76 = lodged EXACTLY (native RdSAP-21.0.1 schema; reproduces every component: space 6129/main 6247/HW 2752/CO2 2232). Elmhurst +3 is a BUILD gap — my build omitted the lodged secondary gas fire (engine models it 3065 kWh = 0.1÷0.20); engine-on-Elmhurst-inputs 79.29 ≈ worksheet 79 → calculator faithful. NB: shared-assessment storage→combi reset needed (clear leftover MainHeatingCode) + meter reset to Single. build_10023443426.py.
- [x] 10093412452 — SAP-17.1 (full-SAP END-TERRACE HOUSE 2-storey, band L, combi PCDB 17615, party wall 41.01m², natural vent + 3 extract fans, AP50 4.62, 20 LED, measured U 0.25/0.13/0.16; sibling of 10093718424/10091568921) · eng 81 / elm 80 (lodged 84) · PINNED engine 81. +1 = documented full-SAP→RdSAP residual; engine-on-Elmhurst-inputs 79.91 ≈ worksheet 80 → faithful. Built combi (cylinder lodged true but combi PCDB). No mapper change.
- [⛔] 10014314798 — SAP-16.2 · NOT MAPPABLE (ValueError: RdSapSchema17_1: missing required field )
- [⚠] 10094601294 — SAP-18.0.0 · eng 81 / lodged 84 · 🚩 MVHR idx 500230 not credited (flagged)
- [ ] 10090343335 — SAP-17.0 · eng 86 / lodged 88
- [ ] 10093115480 — SAP-17.1 · eng 81 / lodged 81
- [ ] 68151071 — RdSAP-17.0 · eng 68 / lodged 70
- [ ] 100021985993 — SAP-16.2 · eng 74 / lodged 70
- [ ] 100020665611 — RdSAP-20.0.0 · eng 36 / lodged 37
- [🔍] 10090343335 — SAP-17.0 (full-SAP SEMI-DETACHED HOUSE, 3-storey incl. ROOM-IN-ROOF top floor h1.95, combi PCDB 16841 Vaillant ecoTEC plus 824, party wall 48.51m², 10 windows 18.46m² + 1 ROOF WINDOW type-12 2.31m², 3 doors, AP50 3.98, 10 LED, measured U walls 0.18/roof 0.15/floor 0.13, TFA 122) · eng 86 / lodged 88 · 🔍 FLAGGED — NEEDS NEW BUILD PATTERN: room-in-roof + roof-window not covered by templates (Elmhurst DIM panel only exposes 2 floors; room-in-roof entered via RoomInRoofMain age-band + separate RiR dimensions TBD; roof window type-12 entry TBD). Elmhurst build + worksheet comparison pending. Skipped in autonomous sweep to keep momentum; revisit with a dedicated room-in-roof build.
- [x] 10093115480 — SAP-17.1 (full-SAP END-TERRACE BUNGALOW single-storey, band L 2016, combi PCDB 16841, party wall 10.99m² CF filled, natural vent + 2 fans, AP50 3.85, 9 LED, measured U 0.19/0.11/0.12, TFA 56) · eng 81 / elm 78 (lodged 81) · PINNED engine 81 = lodged EXACT. +3 = documented full-SAP→RdSAP residual (measured U beats band-L defaults); engine-on-Elmhurst-inputs 78.29 ≈ worksheet 78 → faithful. Built combi. No mapper change.
- [🔍] 68151071 — RdSAP-17.0 (semi-detached BUNGALOW single-storey, band H 1991-1995, cavity insulated, REGULAR boiler PCDB 17550 Worcester Greenstar 18Ri + cylinder size2/foam/50mm, pitched 200mm loft, suspended uninsulated floor, party wall 9.48 lodged, TFA 50) · eng 68 / elm 71 (lodged 70) · 🔍 FLAGGED — MAPPER GAP (cylinder insulation thickness dropped). engine -3 vs Elmhurst, -2 vs lodged; engine HW 3446 kWh vs Elmhurst 2911. ROOT CAUSE confirmed: raw cert lodges sap_heating.cylinder_insulation_thickness=50 but the RdSAP-17.0 mapper maps cylinder_size + cylinder_insulation_type only, leaving EpcPropertyData.sap_heating.cylinder_insulation_thickness_mm=None → engine assumes a poorly-insulated cylinder → over-counts HW cylinder loss → under-rates. FIX: carry cylinder_insulation_thickness → cylinder_insulation_thickness_mm in the RdSAP mapper (check blast radius across schemas + regression). engine-on-Elmhurst-inputs 71.34 ≈ worksheet 71 → calculator faithful (gap is purely the dropped input). build_68151071.py + evidence saved.
- [x] 100021985993 — SAP-16.2 (END-TERRACE BUNGALOW single-storey, band C, solid-brick internal insulation, combi PCDB 10328, double glazed, 100mm loft, suspended uninsulated floor) · eng 74 / elm 72 (lodged 70) · PINNED engine 74. Built in Elmhurst (end-terrace, boiler 10328, control CBE/2106, water from primary, party wall 6.89 solid masonry — Elmhurst requires non-zero for an end-terrace). +2 vs Elmhurst = documented 16.2 reduced-field party-wall gap (gov-API lodges no party_wall_length → engine models none; worksheet's only extra element is party wall 16.19m²×U0.25=4.05 W/K ≈ 2 SAP). engine-on-Elmhurst-PDF-inputs 67 is PDF-parser noise (HW over-parsed), not a calc bug.
- [x] 100020665611 — RdSAP-20.0.0 (END-TERRACE HOUSE 2-storey, band E 1967-1975, cavity UNINSULATED, ELECTRIC STORAGE HEATERS SAP 402 SEB + manual charge CSA/2401, immersion off-peak Economy-7 Dual + cylinder Normal/110L foam 38mm, double glazed PVC, party wall 9.5m/floor U0.25, secondary portable electric heaters/SAP 691, TFA 87.4) · eng 36 / elm 35 (lodged 37) · PINNED engine 36. Built in Elmhurst (storage SEB, CSA control, immersion Dual + cylinder, **Dual electricity meter**, secondary electric room heater). engine-on-Elmhurst-inputs 35 ≡ worksheet 35 → calculator faithful; eng 36 ≈ lodged 37 (−1). Cylinder thickness 38mm correctly carried by RdSAP-20.0.0 mapper (no cylinder-gap here). ⚠ Secondary had to be set EXPLICITLY: shared-assessment leftover House-Coal secondary (633) survived `present=No` and inflated worksheet to 44 until overwritten with electric room heater. build_100020665611.py. No mapper change.
- [⚠] 10093388044 — SAP-17.1 · eng 87 / lodged 93 · 🚩 heat-pump fuel-39 (flagged)
- [ ] 10090944225 — SAP-17.0 · eng 81 / lodged 82
- [x] 10090944225 — SAP-17.0 (full-SAP GROUND-FLOOR FLAT, band L 2015, combi PCDB 17045 Ideal Logic Combi ES35, control 2110 CBI time+temp zone, NATURAL vent + 2 extract fans, AP50 3.48, party wall 3.41m² U0, measured U walls 0.27/floor 0.14/window 1.41, TFA 49.97, no cylinder/no secondary) · eng 81 / elm 80 (lodged 82) · PINNED engine 81. Built in Elmhurst (boiler 17045 via search, control CBI/2110 under Boilers→Zone control, water from primary, AP50 Blower Door 3.48+cert, Single meter). engine-on-Elmhurst-inputs 80 = worksheet 80 (exact) → faithful; eng 81 ≈ lodged 82 (−1). Remaining +1 = documented full-SAP→RdSAP residual localised to the FLOOR (cert measured U 0.14 vs Elmhurst RdSAP solid default 0.23; walls/party match) — engine correctly uses measured 0.14, NOT a mapper bug. ⚠ Initial worksheet read 79 due to a spurious leftover REA/691 electric secondary (Table-11 frac 0.1, ~1.5 SAP) the shared assessment retained — REMOVED via present=Yes→clear secondary code to empty→present=No (79→80). 🔧 Build notes: storage→combi reset (clear leftover SEB MainHeatingCode pass-1, reset meter Single); control 2110 is Boilers→**Zone control**→CBI (not Standard). No mapper change.
- [⚠] 10090341811 — SAP-17.0 · eng 80 / lodged 89 · 🚩 MVHR idx 500352 not credited (flagged)
- [ ] 10010215568 — RdSAP-17.1 · eng 75 / lodged 74
- [⚠] 10093117227 — SAP-17.1 · eng 90 / lodged 80 · 🚩 heat-pump fuel-39 (flagged)
@ -86,7 +100,7 @@ Skip the 🚩 MVHR / 🚩 heat-pump-fuel and ⛔ sparse certs.
- [⚠] 10091636116 — SAP-17.0 · eng 80 / lodged 88 · 🚩 MVHR idx 500249 not credited (flagged)
- [ ] 10093049853 — SAP-17.0 · eng 82 / lodged 87
- [ ] 10093390790 — SAP-17.1 · eng 79 / lodged 82
- [ ] 10093116330 — SAP-17.1 · eng 82 / lodged 83
- [x] 10093116330 — SAP-17.1 (2017 gas-combi 2-storey semi HOUSE, TFA 73) · eng 82 / elm 78 (lodged 83) · PINNED engine 82. +4 = documented full-SAP→RdSAP residual. Build clean. No mapper change.
- [ ] 10093116326 — SAP-17.1 · eng 82 / lodged 82
- [ ] 10090317693 — SAP-17.0 · eng 81 / lodged 88
- [ ] 10090034872 — SAP-17.0 · eng 83 / lodged 85
@ -100,7 +114,7 @@ Skip the 🚩 MVHR / 🚩 heat-pump-fuel and ⛔ sparse certs.
- [⚠] 10093305101 — SAP-17.1 · eng 81 / lodged 85 · 🚩 MVHR idx 500140 not credited (flagged)
- [ ] 100020933894 — SAP-16.0 · eng 61 / lodged 56
- [ ] 100020937013 — RdSAP-20.0.0 · eng 70 / lodged 73
- [ ] 🔧 10023444320 — SAP-17.0 schema coverage (full-SAP ≡ 17.1) · eng 81 / lodged 81 · Elmhurst pin pending
- [x] 🔧 10023444320 — SAP-17.0 (full-SAP MID-FLOOR FLAT, sibling of 324, combi PCDB 16211, MEV, AP50 3.09, no party wall) · eng 81 / elm 82 (lodged 81) · PINNED engine 81. Built in Elmhurst (boiler 16211, control CBE/2106, water from primary, MEV, AP50 3.09, mid-floor=floor to dwelling below). Within ~1 (81.38 vs 82); engine-on-Elmhurst-inputs 82.46 ≈ 82 → faithful. No mapper change beyond coverage.
- [ ] 100062188801 — SAP-16.3 · eng 68 / lodged 70
- [ ] 10008048040 — SAP-16.2 · eng 77 / lodged 75
- [ ] 10093101966 — SAP-17.1 · eng 82 / lodged 84

5
.gitignore vendored
View file

@ -313,3 +313,8 @@ scripts/eon/epc_cache.pkl
scripts/hyde/.elmhurst-session/
scripts/hyde/elmhurst_downloads/
scripts/hyde/.elmhurst-creds.json
# Hyde property-overrides script artifacts
overrides_cache.json
overrides_unknowns.csv
overrides_edits.csv

80
applications/landlord_description_overrides/handler.py
View file

@ -7,11 +7,16 @@ import boto3
from applications.landlord_description_overrides.landlord_description_overrides_trigger_body import (
LandlordDescriptionOverridesTriggerBody,
)
from domain.epc.built_form_type import BuiltFormType
from domain.epc.property_type import PropertyType
from domain.epc.roof_type import RoofType
from domain.epc.wall_type import WallType
from domain.epc.wall_type_construction_dates import (
from domain.epc.property_overrides.built_form_type import BuiltFormType
from domain.epc.property_overrides.construction_age_band import ConstructionAgeBand
from domain.epc.property_overrides.glazing_type import GlazingType
from domain.epc.property_overrides.main_fuel_type import MainFuelType
from domain.epc.property_overrides.main_heating_system_type import MainHeatingSystemType
from domain.epc.property_overrides.property_type import PropertyType
from domain.epc.property_overrides.roof_type import RoofType
from domain.epc.property_overrides.water_heating_type import WaterHeatingType
from domain.epc.property_overrides.wall_type import WallType
from domain.epc.property_overrides.wall_type_construction_dates import (
wall_type_construction_date_prompt_hint,
)
from infrastructure.chatgpt.chatgpt import ChatGPT
@ -24,6 +29,21 @@ from infrastructure.postgres.engine import commit_scope, make_engine, make_sessi
from infrastructure.postgres.landlord_built_form_type_override_table import (
LandlordBuiltFormTypeOverrideRow,
)
from infrastructure.postgres.landlord_construction_age_band_override_table import (
LandlordConstructionAgeBandOverrideRow,
)
from infrastructure.postgres.landlord_glazing_override_table import (
LandlordGlazingOverrideRow,
)
from infrastructure.postgres.landlord_main_fuel_override_table import (
LandlordMainFuelOverrideRow,
)
from infrastructure.postgres.landlord_main_heating_system_override_table import (
LandlordMainHeatingSystemOverrideRow,
)
from infrastructure.postgres.landlord_water_heating_override_table import (
LandlordWaterHeatingOverrideRow,
)
from infrastructure.postgres.landlord_property_type_override_table import (
LandlordPropertyTypeOverrideRow,
)
@ -102,6 +122,56 @@ def _build_columns(
session, LandlordRoofTypeOverrideRow
),
),
"main_fuel": lambda src: ClassifiableColumn(
name="main_fuel",
source_column=src,
classifier=ChatGptColumnClassifier(
chat_gpt, MainFuelType, MainFuelType.UNKNOWN
),
repo=LandlordOverridesRepository[MainFuelType](
session, LandlordMainFuelOverrideRow
),
),
"glazing": lambda src: ClassifiableColumn(
name="glazing",
source_column=src,
classifier=ChatGptColumnClassifier(
chat_gpt, GlazingType, GlazingType.UNKNOWN
),
repo=LandlordOverridesRepository[GlazingType](
session, LandlordGlazingOverrideRow
),
),
"construction_age_band": lambda src: ClassifiableColumn(
name="construction_age_band",
source_column=src,
classifier=ChatGptColumnClassifier(
chat_gpt, ConstructionAgeBand, ConstructionAgeBand.UNKNOWN
),
repo=LandlordOverridesRepository[ConstructionAgeBand](
session, LandlordConstructionAgeBandOverrideRow
),
),
"water_heating": lambda src: ClassifiableColumn(
name="water_heating",
source_column=src,
classifier=ChatGptColumnClassifier(
chat_gpt, WaterHeatingType, WaterHeatingType.UNKNOWN
),
repo=LandlordOverridesRepository[WaterHeatingType](
session, LandlordWaterHeatingOverrideRow
),
),
"main_heating_system": lambda src: ClassifiableColumn(
name="main_heating_system",
source_column=src,
classifier=ChatGptColumnClassifier(
chat_gpt, MainHeatingSystemType, MainHeatingSystemType.UNKNOWN
),
repo=LandlordOverridesRepository[MainHeatingSystemType](
session, LandlordMainHeatingSystemOverrideRow
),
),
}
columns: list[ClassifiableColumn[Any]] = []

BIN
backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-17.0/uprn_68151071/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-17.0/uprn_68151071/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-18.0/uprn_10022893721/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-18.0/uprn_10022893721/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-20.0.0/uprn_10090844932/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-20.0.0/uprn_10090844932/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-21.0.1/uprn_10023443426/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-21.0.1/uprn_10023443426/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

98
backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.1/uprn_100021943298/elmhurst_inputs.md Normal file
View file

@ -0,0 +1,98 @@
# Elmhurst RdSAP inputs — UPRN 100021943298 (cert 0498-6963-7270-0822-1980, SAP-Schema-16.1)
**Lodged SAP:** 72 **Our engine:** 76 (continuous ~75.x) ← compare Elmhurst against the engine
**Property:** Ground-floor FLAT, age band B (1900–1929), mains-gas combi + radiators, TFA 78 m²
**Schema note:** SAP-Schema-16.1 is reduced-field (RdSAP-shaped) — mapped via the
new `from_sap_schema_16_1` → `_normalize_sap_schema_16_x` → `from_rdsap_schema_17_1`.
Descriptions + glazed-area band, no measured U-values, so engine and Elmhurst should
both use RdSAP age-band/description U-values — expect TIGHTER agreement (the +4 over
lodged is the reconciliation target).
**Known divergences / gaps to watch:**
- **Party wall:** 16.1 lodges no `party_wall_length`; normalizer defaults 0 → engine
models no party wall. For a ground-floor flat the side/party walls to other dwellings
are handled by Elmhurst's Flat position + corridor — keep party-wall length 0 and check
the worksheet party-wall line.
- Age band B → Elmhurst on-screen band **B 1900-1929**.
## Property Description
| Elmhurst field | Value | Notes |
|---|---|---|
| Property Type | **Flat** | property_type 2 |
| Built form | **End-Terrace** | built_form 3 |
| Age band | **B 1900-1929** | construction_age_band B |
| Storeys | 1 | ground-floor flat, single storey |
| Habitable rooms | 3 | habitable_room_count 3 |
## Flats
| Elmhurst field | Value | Notes |
|---|---|---|
| Position of flat | **Ground Floor** | dwelling_type "Ground-floor flat" |
| Floor | 0 | ground = storey 0 |
| Corridor | None | all-exposed (no unheated corridor lodged) |
## Dimensions
| Elmhurst field | Value | Notes |
|---|---|---|
| Floor area (lowest) | 77.69 m² | floor 0 |
| Room height | 2.65 m | room_height |
| Heat-loss perimeter | 28.9 m | heat_loss_perimeter |
| Party-wall length | **0** | 16.1 lodges none |
## Walls
| Elmhurst field | Value | Notes |
|---|---|---|
| Construction | **Solid Brick** | wall_construction 3 |
| Insulation | **Internal** | wall_insulation_type 3 = internal insulation ("Solid brick, with internal insulation") |
| Thickness | (as built) | wall_thickness_measured True |
## Roofs
| Elmhurst field | Value | Notes |
|---|---|---|
| Type | **Another dwelling above** | roof "(another dwelling above)" → no roof heat loss |
## Floors
| Elmhurst field | Value | Notes |
|---|---|---|
| Location / type | Ground floor / **Suspended** | "Suspended, no insulation (assumed)" |
| Insulation | **As built** (none) | no insulation |
## Openings
| Elmhurst field | Value | Notes |
|---|---|---|
| Windows | **Double glazed**, total 11.544 m² (4 × 2.886, N/E/S/W) | glazed_area band 1, multiple_glazing_type 2 (double). Enter ONE combined "Double …" row of 11.544 m². |
| Doors | **2, uninsulated** | door_count 2, insulated_door_count 0 |
## Ventilation & Lighting
| Elmhurst field | Value | Notes |
|---|---|---|
| Ventilation | Natural | no MV |
| Extract fans / flues | 0 / 0 | none lodged |
| Sheltered sides | 1 | sheltered_sides 1 |
| Air Pressure Test | **Not available** (no test lodged) | clear any Blower Door |
| Lighting | **100% low-energy** (6 of 6 outlets) | low_energy 6 / total 6 |
## Space Heating
| Elmhurst field | Value | Notes |
|---|---|---|
| Main heating | Mains gas **combi** + radiators | fuel 26, emitter 1, control 2106, fan flue, PCDB 10328, category 2, NO cylinder → BGW condensing combi cascade |
| Secondary | None | secondary "None" |
## Water Heating
| Elmhurst field | Value | Notes |
|---|---|---|
| Water heating | From main (901), gas | water_heating from combi |
| Hot water cylinder | **None** | has_hot_water_cylinder False (combi/instantaneous) |
| Solar / WWHRS / FGHRS | None | — |
## Fields to clear in Elmhurst (do NOT map)
| Elmhurst field | Set to | Why absent |
|---|---|---|
| Conservatory | none | not lodged |
| Air Pressure Test method | **Not available** | no test lodged (clear any prior Blower Door) |
| Mechanical ventilation | off | natural ventilation |
| Cylinder size/insulation | (gone — combi) | no cylinder |
| Main Heating 2 / Secondary | none | single combi system |
| PV / Wind / Hydro | none | none lodged |
| Room-in-roof / extensions | blank | single building part, no RiR |

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.1/uprn_100021943298/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.1/uprn_100021943298/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

232
backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.1/uprn_100021943298/epc.json Normal file
View file

@ -0,0 +1,232 @@
{
"uprn": 100021943298,
"roofs": [
{
"description": "(another dwelling above)",
"energy_efficiency_rating": 0,
"environmental_efficiency_rating": 0
}
],
"walls": [
{
"description": "Solid brick, with internal insulation",
"energy_efficiency_rating": 4,
"environmental_efficiency_rating": 4
}
],
"floors": [
{
"description": "Suspended, no insulation (assumed)",
"energy_efficiency_rating": 0,
"environmental_efficiency_rating": 0
}
],
"status": "entered",
"tenure": 2,
"windows": [
{
"description": "Fully double glazed",
"energy_efficiency_rating": 3,
"environmental_efficiency_rating": 3
}
],
"lighting": {
"description": "Low energy lighting in all fixed outlets",
"energy_efficiency_rating": 5,
"environmental_efficiency_rating": 5
},
"postcode": "SE4 1YN",
"hot_water": {
"description": "From main system",
"energy_efficiency_rating": 4,
"environmental_efficiency_rating": 4
},
"post_town": "LONDON",
"built_form": 3,
"created_at": "2012-10-02 23:16:59.000000",
"door_count": 2,
"glazed_area": 1,
"region_code": 17,
"report_type": 2,
"sap_heating": {
"wwhrs": {
"rooms_with_bath_and_or_shower": 1,
"rooms_with_mixer_shower_no_bath": 0,
"rooms_with_bath_and_mixer_shower": 0
},
"cylinder_size": 1,
"water_heating_code": 901,
"water_heating_fuel": 26,
"main_heating_details": [
{
"has_fghrs": "N",
"main_fuel_type": 26,
"boiler_flue_type": 2,
"fan_flue_present": "Y",
"heat_emitter_type": 1,
"boiler_index_number": 10328,
"main_heating_number": 1,
"main_heating_control": 2106,
"main_heating_category": 2,
"main_heating_fraction": 1,
"main_heating_data_source": 1
}
],
"has_fixed_air_conditioning": "false"
},
"sap_version": 9.91,
"schema_type": "SAP-Schema-16.1",
"uprn_source": "Energy Assessor",
"country_code": "EAW",
"main_heating": [
{
"description": "Boiler and radiators, mains gas",
"energy_efficiency_rating": 4,
"environmental_efficiency_rating": 4
}
],
"dwelling_type": "Ground-floor flat",
"language_code": 1,
"property_type": 2,
"address_line_1": "11a, Chalsey Road",
"schema_version": "LIG-16.1",
"assessment_type": "RdSAP",
"completion_date": "2012-10-02",
"inspection_date": "2012-10-01",
"extensions_count": 0,
"measurement_type": 1,
"sap_flat_details": {
"level": 1,
"top_storey": "N",
"flat_location": 0,
"heat_loss_corridor": 0
},
"total_floor_area": 78,
"transaction_type": 8,
"conservatory_type": 1,
"heated_room_count": 3,
"registration_date": "2012-10-02",
"restricted_access": 0,
"sap_energy_source": {
"main_gas": "Y",
"meter_type": 2,
"photovoltaic_supply": {
"percent_roof_area": 0
},
"wind_turbines_count": 0,
"wind_turbines_terrain_type": 2
},
"secondary_heating": {
"description": "None",
"energy_efficiency_rating": 0,
"environmental_efficiency_rating": 0
},
"sap_building_parts": [
{
"identifier": "Main Dwelling",
"wall_dry_lined": "N",
"wall_thickness": 300,
"floor_heat_loss": 7,
"roof_construction": 3,
"wall_construction": 3,
"building_part_number": 1,
"sap_floor_dimensions": [
{
"floor": 0,
"room_height": 2.65,
"floor_insulation": 1,
"total_floor_area": 77.69,
"floor_construction": 2,
"heat_loss_perimeter": 28.9
}
],
"wall_insulation_type": 3,
"construction_age_band": "B",
"wall_thickness_measured": "Y",
"roof_insulation_location": "ND",
"roof_insulation_thickness": "ND",
"wall_insulation_thickness": "50mm"
}
],
"low_energy_lighting": 100,
"solar_water_heating": "N",
"bedf_revision_number": 329,
"habitable_room_count": 3,
"heating_cost_current": {
"value": 421,
"currency": "GBP"
},
"insulated_door_count": 0,
"co2_emissions_current": 2.4,
"energy_rating_average": 60,
"energy_rating_current": 72,
"lighting_cost_current": {
"value": 46,
"currency": "GBP"
},
"main_heating_controls": [
{
"description": "Programmer, room thermostat and TRVs",
"energy_efficiency_rating": 4,
"environmental_efficiency_rating": 4
}
],
"multiple_glazing_type": 1,
"open_fireplaces_count": 0,
"has_hot_water_cylinder": "false",
"heating_cost_potential": {
"value": 360,
"currency": "GBP"
},
"hot_water_cost_current": {
"value": 85,
"currency": "GBP"
},
"mechanical_ventilation": 0,
"percent_draughtproofed": 100,
"suggested_improvements": [
{
"sequence": 1,
"typical_saving": {
"value": 61,
"currency": "GBP"
},
"indicative_cost": "\u00a3800 - \u00a31,200",
"improvement_type": "W",
"improvement_details": {
"improvement_number": 47
},
"improvement_category": 5,
"energy_performance_rating": 75,
"environmental_impact_rating": 77
}
],
"co2_emissions_potential": 2.0,
"energy_rating_potential": 75,
"lighting_cost_potential": {
"value": 46,
"currency": "GBP"
},
"hot_water_cost_potential": {
"value": 85,
"currency": "GBP"
},
"renewable_heat_incentive": {
"water_heating": 2121,
"space_heating_existing_dwelling": 7355
},
"seller_commission_report": "Y",
"energy_consumption_current": 160,
"has_fixed_air_conditioning": "false",
"multiple_glazed_proportion": 100,
"calculation_software_version": 8.0,
"energy_consumption_potential": 137,
"environmental_impact_current": 72,
"fixed_lighting_outlets_count": 6,
"current_energy_efficiency_band": "C",
"environmental_impact_potential": 77,
"has_heated_separate_conservatory": "false",
"potential_energy_efficiency_band": "C",
"co2_emissions_current_per_floor_area": 31,
"low_energy_fixed_lighting_outlets_count": 6
}

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.2/uprn_100020933699/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.2/uprn_100020933699/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.2/uprn_100021985993/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.2/uprn_100021985993/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.2/uprn_100090182288/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.2/uprn_100090182288/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.3/uprn_44012843/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.3/uprn_44012843/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.0/uprn_10023444320/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.0/uprn_10023444320/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.0/uprn_10023444324/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.0/uprn_10023444324/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.1/uprn_10091568921/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.1/uprn_10091568921/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.1/uprn_10093115480/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.1/uprn_10093115480/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.1/uprn_10093116330/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.1/uprn_10093116330/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.1/uprn_10093412452/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.1/uprn_10093412452/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.1/uprn_10093718424/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.1/uprn_10093718424/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

96
backend/epc_api/json_samples/real_life_examples/SAP-Schema-19.1.0/uprn_10096028301/elmhurst_inputs.md Normal file
View file

@ -0,0 +1,96 @@
# Elmhurst RdSAP inputs — UPRN 10096028301 (cert 0390-3321-6060-2405-7985, SAP-Schema-19.1.0)
**Lodged SAP:** 85 **Our engine:** 82 ← compare Elmhurst against the engine
**Property:** Ground-floor FLAT, age band M (2023 onwards), mains-gas combi + radiators, decentralised MEV, TFA 73 m²
**Schema note:** SAP-Schema-19.1.0 is FULL SAP (measured U-values, measured air
permeability) — mapped via the new `from_sap_schema_19_1_0` (parses with the 17.1
dataclass, delegates to `from_sap_schema_17_1`). Near-twin of the worked-ref flat
10092973954 (same PCDB combi 17929, MEV, AP50). Expect the documented full-SAP→RdSAP
residual: the engine uses the cert's MEASURED U (wall 0.24 / floor 0.13 — WORSE than
RdSAP band-M defaults), so here the engine *under*-rates vs RdSAP (eng 82 < lodged 85).
**Known divergences / gaps to watch:**
- **MEV (decentralised, EXTRACT_OR_PIV_OUTSIDE):** engine prices it as extract loss,
not heat recovery — the documented MVHR gap. Omit/parallel on both sides.
- **Measured vs RdSAP U:** Elmhurst can't set the U-value-known override → it uses band-M
age defaults (better than the cert's measured 0.24 wall). Note the divergence; don't tune.
- Age band M → Elmhurst on-screen band **M 2023 onwards**.
## Property Description
| Elmhurst field | Value | Notes |
|---|---|---|
| Property Type | **Flat** | ground-floor flat |
| Built form | **Detached** (block) | built_form not lodged (full SAP) — use Detached as for worked-ref flat |
| Age band | **M 2023 onwards** | construction_age_band M |
| Storeys | 1 | ground-floor flat |
| Habitable rooms | 3 | habitable_room_count 3 |
## Flats
| Elmhurst field | Value | Notes |
|---|---|---|
| Position of flat | **Ground Floor** | dwelling_type ground-floor flat |
| Floor | 0 | ground = storey 0 |
| Corridor | None | all-exposed |
## Dimensions
| Elmhurst field | Value | Notes |
|---|---|---|
| Floor area (lowest) | 72.99 m² | floor 0 |
| Room height | 2.36 m | room_height |
| Heat-loss perimeter | 36.04 m | heat_loss_perimeter |
| Party-wall length | **0** | not lodged |
## Walls
| Elmhurst field | Value | Notes |
|---|---|---|
| Construction | **Cavity** | wall_construction 4 |
| Insulation | **As Built** | measured U 0.24 W/m²K (U-known override not settable → accept band-M default; note divergence) |
## Roofs
| Elmhurst field | Value | Notes |
|---|---|---|
| Type | **Another dwelling above** | roof "(other premises above)" → no roof heat loss |
## Floors
| Elmhurst field | Value | Notes |
|---|---|---|
| Location / type | Ground floor / Solid | measured U 0.13 (accept band-M default) |
| Insulation | As built | — |
## Openings
| Elmhurst field | Value | Notes |
|---|---|---|
| Windows | total ~11.37 m² (4 windows), U 1.3, g 0.46 | glazing_type 7. Enter ONE combined band row ("Double … 2022" ≈ U1.4/g0.72) of 11.37 m² — `…known data` demands per-row frame-factor/source and fails validation. |
| Doors | **1, solid (uninsulated)** U 1.2 | sap_opening_types: 1 Solid Door, U 1.2 |
## Ventilation & Lighting
| Elmhurst field | Value | Notes |
|---|---|---|
| Ventilation | **Mechanical — decentralised MEV** (extract) | ventilation EXTRACT_OR_PIV_OUTSIDE. 🚩 engine models as extract loss not heat recovery. |
| Air Pressure Test | **Blower Door**, result **3.5** (AP50) | air_tightness "AP50 = 3.5 as tested"; Elmhurst (18)=AP50/20. Needs a certificate number to pass Recommendations validation. |
| Sheltered sides | (from built form) | — |
| Lighting | per cert low-energy % | sap_lighting |
## Space Heating
| Elmhurst field | Value | Notes |
|---|---|---|
| Main heating | Mains gas **combi** + radiators | fuel mains gas, PCDB 17929 (Vaillant combi), control 2110, category 2, NO cylinder → BGW condensing combi cascade |
| Flue | Balanced (room-sealed) | fan_flue_present; balanced flue |
| Secondary | None | — |
## Water Heating
| Elmhurst field | Value | Notes |
|---|---|---|
| Water heating | From main (901), gas | combi |
| Hot water cylinder | **None** | has_hot_water_cylinder False (combi) |
| Solar / WWHRS / FGHRS | None | — |
## Fields to clear in Elmhurst (do NOT map)
| Elmhurst field | Set to | Why absent |
|---|---|---|
| Conservatory | none | not lodged |
| Cylinder size/insulation | (gone — combi) | no cylinder |
| Main Heating 2 / Secondary | none | single combi system |
| PV / Wind / Hydro / WWHRS / FGHRS | none | none lodged |
| Room-in-roof / extensions | blank | single building part |

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-19.1.0/uprn_10096028301/elmhurst_summary.pdf Normal file
View file

Binary file not shown.

BIN
backend/epc_api/json_samples/real_life_examples/SAP-Schema-19.1.0/uprn_10096028301/elmhurst_worksheet.pdf Normal file
View file

Binary file not shown.

456
backend/epc_api/json_samples/real_life_examples/SAP-Schema-19.1.0/uprn_10096028301/epc.json Normal file
View file

@ -0,0 +1,456 @@
{
"der": 12.66,
"ter": 13.84,
"dfee": 36.3,
"dper": 69.44,
"tfee": 38.6,
"tper": 72.99,
"uprn": 10096028301,
"roofs": [
{
"description": {
"value": "(other premises above)",
"language": "1"
},
"energy_efficiency_rating": 0,
"environmental_efficiency_rating": 0
}
],
"walls": [
{
"description": {
"value": "Average thermal transmittance 0.24 W/m\u00b2K",
"language": "1"
},
"energy_efficiency_rating": 5,
"environmental_efficiency_rating": 5
}
],
"floors": [
{
"description": {
"value": "Average thermal transmittance 0.13 W/m\u00b2K",
"language": "1"
},
"energy_efficiency_rating": 5,
"environmental_efficiency_rating": 5
}
],
"status": "entered",
"tenure": "ND",
"windows": {
"description": {
"value": "High performance glazing",
"language": "1"
},
"energy_efficiency_rating": 4,
"environmental_efficiency_rating": 4
},
"lighting": {
"description": {
"value": "Good lighting efficiency",
"language": "1"
},
"energy_efficiency_rating": 4,
"environmental_efficiency_rating": 4
},
"postcode": "CT5 1TN",
"data_type": 2,
"hot_water": {
"description": {
"value": "From main system, waste water heat recovery",
"language": "1"
},
"energy_efficiency_rating": 5,
"environmental_efficiency_rating": 5
},
"post_town": "Whitstable",
"built_form": 1,
"created_at": "2025-06-09 10:49:27",
"living_area": 26.89,
"orientation": 3,
"region_code": 14,
"report_type": 3,
"sap_heating": {
"number_baths": 1,
"thermal_store": 1,
"shower_outlets": [
{
"shower_wwhrs": 2,
"shower_flow_rate": 9,
"shower_outlet_type": 3
}
],
"water_fuel_type": 1,
"water_heating_code": 901,
"instantaneous_wwhrs": {
"wwhrs_index_number1": 80137
},
"main_heating_details": [
{
"has_fghrs": "false",
"main_fuel_type": 1,
"heat_emitter_type": 1,
"is_flue_fan_present": "true",
"main_heating_number": 1,
"is_condensing_boiler": "true",
"main_heating_control": 2110,
"is_interlocked_system": "true",
"main_heating_category": 2,
"main_heating_fraction": 1,
"gas_or_oil_boiler_type": 2,
"main_heating_flue_type": 2,
"central_heating_pump_age": 2,
"main_heating_data_source": 1,
"main_heating_index_number": 17929,
"has_separate_delayed_start": "true",
"is_oil_pump_in_heated_space": "false",
"is_main_heating_hetas_approved": "false",
"is_central_heating_pump_in_heated_space": "true"
}
],
"has_hot_water_cylinder": "false",
"has_cylinder_thermostat": "false",
"has_fixed_air_conditioning": "false",
"secondary_heating_category": 1,
"is_immersion_for_summer_use": "false",
"is_hot_water_separately_timed": "false",
"is_heat_pump_assisted_by_immersion": "false"
},
"sap_version": 10.2,
"schema_type": "SAP-Schema-19.1.0",
"uprn_source": "Address Matched",
"country_code": "ENG",
"main_heating": [
{
"description": {
"value": "Boiler and radiators, mains gas",
"language": "1"
},
"energy_efficiency_rating": 5,
"environmental_efficiency_rating": 4
}
],
"sap_lighting": [
[
{
"lighting_power": 9,
"lighting_outlets": 6,
"lighting_efficacy": 80
}
]
],
"terrain_type": 2,
"air_tightness": {
"description": {
"value": "Air permeability [AP50] = 3.5 m\u00b3/h.m\u00b2 (as tested)",
"language": "1"
},
"energy_efficiency_rating": 4,
"environmental_efficiency_rating": 4
},
"dwelling_type": "Ground-floor flat",
"language_code": 1,
"property_type": 2,
"pv_connection": 1,
"address_line_1": "1 Ransome Way",
"assessment_date": "2025-06-09",
"assessment_type": "SAP",
"completion_date": "2025-06-09",
"inspection_date": "2025-06-09",
"sap_ventilation": {
"psv_count": 0,
"wall_type": 2,
"pressure_test": 1,
"wet_rooms_count": 2,
"air_permeability": 3.52,
"open_flues_count": 0,
"ventilation_type": 6,
"has_draught_lobby": "true",
"other_flues_count": 0,
"closed_flues_count": 0,
"extract_fans_count": 0,
"boilers_flues_count": 0,
"open_chimneys_count": 0,
"sheltered_sides_count": 2,
"blocked_chimneys_count": 0,
"kitchen_duct_fans_count": 0,
"kitchen_room_fans_count": 0,
"kitchen_wall_fans_count": 1,
"flueless_gas_fires_count": 0,
"mechanical_vent_duct_type": 1,
"non_kitchen_duct_fans_count": 0,
"non_kitchen_room_fans_count": 0,
"non_kitchen_wall_fans_count": 1,
"mechanical_ventilation_data_source": 1,
"mechanical_vent_system_index_number": 500769,
"is_mechanical_vent_approved_installer_scheme": "false"
},
"design_water_use": 1,
"sap_data_version": 10.2,
"sap_flat_details": {
"level": 1,
"storeys": 3
},
"total_floor_area": 73,
"transaction_type": 6,
"cold_water_source": 1,
"conservatory_type": 1,
"registration_date": "2025-06-09",
"sap_energy_source": {
"pv_arrays": [
{
"pitch": 3,
"peak_power": 0.38,
"orientation": 5,
"overshading": 1
}
],
"electricity_tariff": 1
},
"sap_opening_types": [
{
"name": "Solid Door",
"type": 1,
"u_value": 1.2,
"data_source": 2,
"glazing_type": 1,
"isargonfilled": "false"
},
{
"name": "Windows",
"type": 4,
"u_value": 1.3,
"data_source": 4,
"frame_factor": 1.0,
"glazing_type": 7,
"isargonfilled": "false",
"solar_transmittance": 0.46
}
],
"secondary_heating": {
"description": {
"value": "None",
"language": "1"
},
"energy_efficiency_rating": 0,
"environmental_efficiency_rating": 0
},
"lowest_storey_area": 72.99,
"lzc_energy_sources": [
11
],
"sap_building_parts": [
{
"sap_roofs": [
{
"name": "Party roof 1",
"u_value": 0,
"roof_type": 4,
"kappa_value": 30,
"total_roof_area": 72.99
}
],
"sap_walls": [
{
"name": "Walls (1)",
"u_value": 0.24,
"wall_type": 2,
"kappa_value": 60,
"total_wall_area": 68.76,
"is_curtain_walling": "false"
},
{
"name": "Walls (2)",
"u_value": 0.26,
"wall_type": 3,
"kappa_value": 110,
"total_wall_area": 16.3,
"is_curtain_walling": "false"
},
{
"name": "Internal Wall (1)",
"u_value": 0,
"wall_type": 5,
"kappa_value": 9,
"total_wall_area": 114.18,
"is_curtain_walling": "false"
}
],
"sap_openings": [
{
"name": "Front Door",
"type": "Solid Door",
"width": 2.15,
"height": 1,
"location": "Walls (1)",
"orientation": 3
},
{
"name": "Front Windows",
"type": "Windows",
"width": 4.77,
"height": 1,
"location": "Walls (1)",
"orientation": 3
},
{
"name": "RH Windows",
"type": "Windows",
"width": 4.39,
"height": 1,
"location": "Walls (1)",
"orientation": 1
},
{
"name": "Rear Window",
"type": "Windows",
"width": 0.62,
"height": 1,
"location": "Walls (1)",
"orientation": 7
},
{
"name": "LH Window",
"type": "Windows",
"width": 1.59,
"height": 1,
"location": "Walls (1)",
"orientation": 5
}
],
"construction_year": 2025,
"sap_thermal_bridges": {
"thermal_bridges": [
{
"length": 9.65,
"psi_value": 0.021,
"psi_value_source": 1,
"thermal_bridge_type": "E2"
},
{
"length": 8.16,
"psi_value": 0.017,
"psi_value_source": 1,
"thermal_bridge_type": "E3"
},
{
"length": 16.95,
"psi_value": 0.013,
"psi_value_source": 1,
"thermal_bridge_type": "E4"
},
{
"length": 29.14,
"psi_value": 0.054,
"psi_value_source": 1,
"thermal_bridge_type": "E5"
},
{
"length": 29.14,
"psi_value": 0.052,
"psi_value_source": 1,
"thermal_bridge_type": "E7"
},
{
"length": 11.61,
"psi_value": 0.046,
"psi_value_source": 1,
"thermal_bridge_type": "E16"
},
{
"length": 6.91,
"psi_value": 0.111,
"psi_value_source": 1,
"thermal_bridge_type": "E5"
},
{
"length": 6.91,
"psi_value": 0.056,
"psi_value_source": 1,
"thermal_bridge_type": "E7"
},
{
"length": 4.72,
"psi_value": -0.1,
"psi_value_source": 1,
"thermal_bridge_type": "E17"
}
],
"thermal_bridge_code": 5
},
"building_part_number": 1,
"sap_floor_dimensions": [
{
"storey": 0,
"u_value": 0.13,
"floor_type": 2,
"kappa_value": 75,
"storey_height": 2.36,
"heat_loss_area": 72.99,
"total_floor_area": 72.99
}
]
}
],
"user_interface_name": "Design SAP 10",
"windows_overshading": 2,
"heating_cost_current": {
"value": 218,
"currency": "GBP"
},
"co2_emissions_current": 0.8,
"energy_rating_average": 60,
"energy_rating_current": 85,
"lighting_cost_current": {
"value": 57,
"currency": "GBP"
},
"main_heating_controls": [
{
"description": {
"value": "Time and temperature zone control",
"language": "1"
},
"energy_efficiency_rating": 5,
"environmental_efficiency_rating": 5
}
],
"has_hot_water_cylinder": "false",
"heating_cost_potential": {
"value": 218,
"currency": "GBP"
},
"hot_water_cost_current": {
"value": 118,
"currency": "GBP"
},
"user_interface_version": "2.22.1",
"co2_emissions_potential": 0.8,
"energy_rating_potential": 85,
"gas_smart_meter_present": "false",
"lighting_cost_potential": {
"value": 57,
"currency": "GBP"
},
"schema_version_original": "SAP-Schema-19.1.0",
"hot_water_cost_potential": {
"value": 118,
"currency": "GBP"
},
"is_in_smoke_control_area": "unknown",
"seller_commission_report": "Y",
"energy_consumption_current": 59,
"has_fixed_air_conditioning": "false",
"is_dwelling_export_capable": "true",
"multiple_glazed_percentage": 100,
"calculation_software_version": "2.22.1",
"energy_consumption_potential": 59,
"environmental_impact_current": 90,
"current_energy_efficiency_band": "B",
"environmental_impact_potential": 90,
"electricity_smart_meter_present": "false",
"has_heated_separate_conservatory": "false",
"potential_energy_efficiency_band": "B",
"co2_emissions_current_per_floor_area": 10.7
}

111
datatypes/epc/domain/mapper.py
View file

@ -883,6 +883,74 @@ class EpcPropertyDataMapper:
)
return roof_windows
@staticmethod
def from_sap_schema_19_1_0(schema: SapSchema17_1) -> EpcPropertyData:
"""Map a full-SAP `SAP-Schema-19.1.0` cert.
19.1.0 is a later full-SAP revision whose load-bearing fabric — measured
`sap_opening_types`, construction-code `sap_building_parts`,
`sap_heating` / `sap_ventilation` — is structurally identical to
`SAP-Schema-17.1`, so it parses with the 17.1 dataclass and reuses the
tested `from_sap_schema_17_1` mapper. This thin wrapper gives 19.1.0 its
own dispatch entry point: 19.1.0-only fields (e.g. `air_tightness`,
`windows_overshading`) are not yet consumed, and when one needs modelling
it is handled here rather than by widening the 17.1 mapper.
"""
return EpcPropertyDataMapper.from_sap_schema_17_1(schema)
@staticmethod
def from_sap_schema_17_0(schema: SapSchema17_1) -> EpcPropertyData:
"""Map a full-SAP `SAP-Schema-17.0` cert. Structurally identical to the
17.1 measured shape, so it parses with the 17.1 dataclass and reuses
`from_sap_schema_17_1`. Dedicated entry point per the one-mapper-per-
schema convention (note: distinct from RdSAP `from_rdsap_schema_17_0`)."""
return EpcPropertyDataMapper.from_sap_schema_17_1(schema)
@staticmethod
def from_sap_schema_18_0_0(schema: SapSchema17_1) -> EpcPropertyData:
"""Map a full-SAP `SAP-Schema-18.0.0` cert. Structurally identical to the
17.1 measured shape, so it parses with the 17.1 dataclass and reuses
`from_sap_schema_17_1`. Dedicated entry point per the one-mapper-per-
schema convention."""
return EpcPropertyDataMapper.from_sap_schema_17_1(schema)
@staticmethod
def from_sap_schema_16_0(data: Dict[str, Any]) -> EpcPropertyData:
"""Map a reduced-field `SAP-Schema-16.0` cert (see `from_sap_schema_16_2`)."""
return EpcPropertyDataMapper._from_sap_schema_16_x(data)
@staticmethod
def from_sap_schema_16_1(data: Dict[str, Any]) -> EpcPropertyData:
"""Map a reduced-field `SAP-Schema-16.1` cert (see `from_sap_schema_16_2`)."""
return EpcPropertyDataMapper._from_sap_schema_16_x(data)
@staticmethod
def from_sap_schema_16_2(data: Dict[str, Any]) -> EpcPropertyData:
"""Map a reduced-field `SAP-Schema-16.2` cert.
The SAP-Schema-16.x family is structurally RdSAP-17.1 (reduced fields,
`glazed_area` band, construction-code building parts) under a different
name plus a handful of renamed/omitted fields. Each revision gets a
dedicated mapper that normalises onto the RdSAP-17.1 shape via the shared
`_normalize_sap_schema_16_x` and reuses the tested `from_rdsap_schema_17_1`.
"""
return EpcPropertyDataMapper._from_sap_schema_16_x(data)
@staticmethod
def from_sap_schema_16_3(data: Dict[str, Any]) -> EpcPropertyData:
"""Map a reduced-field `SAP-Schema-16.3` cert (see `from_sap_schema_16_2`)."""
return EpcPropertyDataMapper._from_sap_schema_16_x(data)
@staticmethod
def _from_sap_schema_16_x(data: Dict[str, Any]) -> EpcPropertyData:
"""Shared body for the SAP-Schema-16.x dedicated mappers: normalise the
reduced-field doc onto the RdSAP-17.1 shape and reuse that mapper."""
from datatypes.epc.schema.rdsap_schema_17_1 import RdSapSchema17_1
return EpcPropertyDataMapper.from_rdsap_schema_17_1(
from_dict(RdSapSchema17_1, _normalize_sap_schema_16_x(data))
)
@staticmethod
def from_rdsap_schema_17_1(schema: RdSapSchema17_1) -> EpcPropertyData:
es = schema.sap_energy_source
@ -2578,27 +2646,36 @@ class EpcPropertyDataMapper:
mapped = EpcPropertyDataMapper.from_rdsap_schema_17_0(
from_dict(RdSapSchema17_0, data)
)
elif schema in ("SAP-Schema-17.1", "SAP-Schema-17.0", "SAP-Schema-18.0.0"):
# Full SAP (not RdSAP). SAP-Schema-17.0 / 18.0.0 are structurally
# identical to 17.1 (same measured sap_opening_types / building
# parts), so they parse with the 17.1 dataclass and reuse the same
# mapper. The common post-processing below (incl.
# _clear_basement_flag_when_system_built) is a no-op for full SAP
# (explicit wall types — no RdSAP code-6 basement ambiguity).
# Full-SAP family: each revision gets a dedicated mapper (see the
# per-schema methods) even where it currently delegates to the 17.1
# logic — so divergences land in one place rather than a shared tuple.
elif schema == "SAP-Schema-17.1":
mapped = EpcPropertyDataMapper.from_sap_schema_17_1(
from_dict(SapSchema17_1, data)
)
elif schema in ("SAP-Schema-16.2", "SAP-Schema-16.3", "SAP-Schema-16.0"):
# The SAP-Schema-16.x family is structurally RdSAP-17.1 (reduced
# fields, glazed_area band, construction-code building parts) under a
# different name + a handful of renamed/omitted fields — normalise
# onto the RdSAP-17.1 shape and reuse that tested mapper. See
# `_normalize_sap_schema_16_x`.
from datatypes.epc.schema.rdsap_schema_17_1 import RdSapSchema17_1
mapped = EpcPropertyDataMapper.from_rdsap_schema_17_1(
from_dict(RdSapSchema17_1, _normalize_sap_schema_16_x(data))
elif schema == "SAP-Schema-17.0":
mapped = EpcPropertyDataMapper.from_sap_schema_17_0(
from_dict(SapSchema17_1, data)
)
elif schema == "SAP-Schema-18.0.0":
mapped = EpcPropertyDataMapper.from_sap_schema_18_0_0(
from_dict(SapSchema17_1, data)
)
elif schema == "SAP-Schema-19.1.0":
mapped = EpcPropertyDataMapper.from_sap_schema_19_1_0(
from_dict(SapSchema17_1, data)
)
# SAP-Schema-16.x family: reduced-field (RdSAP-shaped) certs. Each gets a
# dedicated mapper that normalises onto the RdSAP-17.1 shape via the
# shared `_normalize_sap_schema_16_x` and reuses `from_rdsap_schema_17_1`.
elif schema == "SAP-Schema-16.0":
mapped = EpcPropertyDataMapper.from_sap_schema_16_0(data)
elif schema == "SAP-Schema-16.1":
mapped = EpcPropertyDataMapper.from_sap_schema_16_1(data)
elif schema == "SAP-Schema-16.2":
mapped = EpcPropertyDataMapper.from_sap_schema_16_2(data)
elif schema == "SAP-Schema-16.3":
mapped = EpcPropertyDataMapper.from_sap_schema_16_3(data)
else:
raise ValueError(f"Unsupported EPC schema: {schema!r}")

2
deployment/terraform/lambda/hubspot_deal_etl/variables.tf
View file

@ -19,7 +19,7 @@ variable "image_digest" {
variable "maximum_concurrency" {
type = number
default = 2
default = 20
description = "Maximum number of concurrent Lambda invocations from SQS (2-1000). null = no limit."
}

136
docs/adr/0033-age-band-resolves-assumed-fabric-states.md Normal file
View file

@ -0,0 +1,136 @@
# Age band resolves the "(assumed)"/"as built"/"Unknown" fabric states
ADR-0032 dec-4 **deferred** the `WallType` "(assumed) insulated" / "partial
insulation (assumed)" states (and, transitively, flat roofs): their
`wall_insulation_type` is "age-inferred in RdSAP, so the code is ambiguous and
must be pinned against the Elmhurst accuracy harness rather than guessed." This
ADR closes that deferral. The harness has now run. Terms in CONTEXT.md (Landlord
Overrides, Effective EPC, Validation Cohort); supersedes ADR-0032 dec-4.
## Status
Accepted. Supersedes the deferral in ADR-0032 dec-4. Evidence:
`scripts/hyde/uvalue_probe_{walls,roofs}.csv` (full A–M Elmhurst sweep, produced
by `scripts/hyde/probe_uvalues.py`).
## Context: what the Elmhurst sweep showed
The SAP 10.2 spec defers RdSAP to "a separate document" (Appendix S, p.122), so
the authoritative source for "what U-value does enum X at age Y produce" is the
accredited Elmhurst RdSAP-10 entry tool itself. We drove it across all 13
construction-age bands (A–M) for the two components Khalim flagged and read the
tool's recomputed **Default U-value** for every option:
* **Cavity wall, As Built** `[A–M]`:
`1.5, 1.5, 1.5, 1.5, 1.5, 1.0, 0.6, 0.6, 0.45, 0.35, 0.3, 0.28, 0.26`
* **Cavity wall, Filled** `[A–M]`:
`0.7, 0.7, 0.7, 0.7, 0.7, 0.4, 0.35, 0.35, 0.45, 0.35, 0.3, 0.28, 0.26`
* **Flat roof, As Built / Unknown** `[A–M]`:
`2.3, 2.3, 2.3, 2.3, 1.5, 0.68, 0.4, 0.35, 0.35, 0.25, 0.25, 0.18, 0.15`
Two facts fall out:
1. **Every value already lives in our tables.** As Built ≡ `(WALL_CAVITY, 0)`,
Filled ≡ `_CAVITY_FILLED_ENG`, flat As Built/Unknown ≡ `_FLAT_ROOF_BY_AGE`
(`domain/sap10_ml/rdsap_uvalues.py`) — exact match, all 13 bands. The U-value
layer was never the gap.
2. **"Unknown" computes identically to "As Built"** at every band — confirming
Khalim's read that Unknown is handled like As Built.
## Decisions
### 1. The "(assumed)" qualifier is *not* a distinct insulation code — age band is
ADR-0032 dec-4 worried that "partial insulation (assumed)" needs a
`wall_insulation_type` code we couldn't name. The sweep shows why none exists:
Elmhurst has **no "partial" enum**. "Partial insulation (assumed)" is simply
*As Built (code 4) evaluated at band F (1976–1982)* → 1.0 W/m²K. Likewise
"no insulation (assumed)" is As Built at an old band (A–E → 1.5) and "insulated
(assumed)" is As Built at a newer band (G+ → ≤0.60). All three resolve to
`wall_insulation_type = 4`; the **construction-age-band overlay** (already a
separate, wired override component) supplies the U-value. This matches Khalim's
bands: cavity ≤1975 uninsulated / 1976–82 partial / later insulated is exactly
the As-Built-by-age curve above.
### 2. Resolve to int codes, not a computed U-value override
Because int codes + the existing age-band tables reproduce Elmhurst exactly,
overlays keep emitting `wall_construction` / `wall_insulation_type` (walls) and
`roof_construction` / `roof_insulation_thickness` (roofs) — no new
`wall_u_value` / `roof_u_value` override path. This keeps the landlord overlay on
the same code path as every Measure (ADR-0032 dec-1) and avoids a second way to
move the wall/roof U-value. (Considered and rejected: writing a computed U-value
override. It would duplicate the RdSAP tables already in the calculator and
diverge the moment those tables are revised.)
### 3. Walls — extend the state map; no new machinery
`wall_type_overlay._STATE_INSULATION` gains the three "as built" variants
("as built", "as built, insulated (assumed)", "as built, partial insulation
(assumed)") → code 4, alongside the existing "no insulation (assumed)" → 4.
These previously produced **no overlay** (silently dropping the landlord's
correction). The age-band overlay does the rest.
### 4. Flat roofs — the overlay must set `roof_construction_type`
The calculator's flat-roof age-band path (`u_roof`, `_FLAT_ROOF_BY_AGE`) fires
only when the roof is flat, which `heat_transmission.py` reads from the **string**
`part.roof_construction_type` (`"flat" in roof_type_lower`, line 999) — *not* the
int `roof_construction`. The roof overlay today sets only
`roof_insulation_thickness`, so a flat-roof override can't reach that path. This
ADR adds `roof_construction_type` to `BuildingPartOverlay` and has
`roof_type_overlay` set it to `"Flat"` for the "Flat, …" family, alongside the
existing `roof_insulation_thickness` (set when the description carries an explicit
mm depth).
The same shape-flag mechanism extends to pitched roofs with no clean depth:
* `"Pitched, Unknown loft insulation"` → `roof_construction_type="Pitched"`,
thickness left `None` → the pitched age-band default (`_ROOF_BY_AGE`).
* `"Pitched, no insulation*"` → `roof_construction_type="Pitched"` **plus**
`roof_insulation_thickness=0` → the uninsulated 2.30 (Table 16 row 0). Pitched
"no insulation" text is load-bearing in `u_roof` (§5.11 note) — it does NOT take
the age-band default, unlike flat. Setting `0` (not `None`) also lets this case
override a lodged numeric thickness, which the "Unknown" cases cannot.
Constraint: the applicator only folds *non-None* overlay fields, so it cannot
*clear* a lodged `roof_insulation_thickness` to `None`. For As Built / Unknown
(no depth) the age-band default fires only when the baseline part carries no
numeric thickness — true on the Hyde flow, where the landlord override IS the roof
source and genuine As Built/Unknown lodgements carry `AB`/`ND`/None, not a number.
A flat/pitched-Unknown override layered on a part that already has a numeric
thickness keeps that thickness; that case is out of scope here and surfaced by the
coverage audit (below) for the verify step.
## Consequences
* Previously-dropped Hyde rows now move the score: ~1,366 flat-roof overrides
(`Flat: As Built` + `Flat: Unknown`) and every cavity "as built/partial/
insulated (assumed)" wall. This is the point — a more accurate Effective EPC —
and (per ADR-0032 dec-5) such overlaid Properties stay excluded from the
Validation Cohort on the divergence signal.
* `BuildingPartOverlay` gains a `roof_construction` field; `apply_simulations`
must fold it like `wall_construction`.
* The decision is pinned by tests mirroring the probe CSVs (one row per
component × band), so a future RdSAP table revision that breaks parity trips a
regression rather than silently drifting from Elmhurst.
* `scripts/hyde/probe_uvalues.py` stays as the re-runnable oracle: re-sweep to
re-confirm parity whenever the RdSAP tables or Elmhurst change.
* A **coverage-audit pair** guards against the silent-no-op class this ADR fixes,
and is the reusable gate for every future portfolio (Hyde is the template):
- `scripts/hyde/audit_override_coverage.py` — enum-level: runs every override
value through its overlay mapper and flags those that produce no overlay or
only an inert field. Run it whenever an override enum or overlay changes.
- `scripts/hyde/audit_hyde_rows.py` — row-level: weights the verdict by actual
portfolio row counts and cross-checks that every "another/same dwelling above"
party-ceiling no-op is genuinely a Flat/Maisonette (a House/Bungalow with a
dwelling above is a data error, not a safe no-op — flagged to CSV for review).
Contradictory properties (party-ceiling roof on a non-flat type — Hyde had 50,
possibly houses split into flats) are **skipped entirely** by the `write` step:
no overrides are written for them and their org_refs are recorded to
`skipped_contradictory_properties.csv` for joint human review, rather than
guessing which of the two conflicting landlord fields is correct.
Known residue logged, not silently dropped: `PitchedWithSlopingCeiling: *`
(~272 Hyde rows, needs the col-3 path), `Roof room(s)` (needs `sap_room_in_roof`),
the `Renewables` column (no ComponentSpec — PV unmapped), and `Timber frame,
with additional insulation` (wall, ~7 rows).

28
domain/elmhurst/README.md Normal file
View file

@ -0,0 +1,28 @@
# Elmhurst reference data
Lookup tables to support re-keying certs into the Elmhurst RdSAP entry tool
(see `scripts/hyde/` and the `expand-sap-accuracy-corpus` skill).
## `pcdb_gas_oil_boiler_codes.csv`
Every gas/oil boiler in the SAP Product Characteristics Database (PCDB Table
105), one row per boiler, so you can look up the **Ref No** to type into
Elmhurst's *PCDF boiler Reference* field (Space Heating → 🔍 search → Select).
Columns: `pcdb_id` (the Ref No), `brand`, `model`, `qualifier`, `type`
(Regular / Combi / Combi-storage / CPSU), `winter_eff_pct`, `summer_eff_pct`,
`output_kw_max`, `final_year`.
- Generated from `domain/sap10_calculator/tables/pcdb/data/pcdb_table_105_gas_oil_boilers.jsonl`.
- `type` is decoded from the raw PCDB row's boiler-type code (`raw[14]`:
`1`→Regular, `2`→Combi, `3`→Combi-storage, `4`→CPSU). This is the field that
decides whether Elmhurst wants a hot-water cylinder (Regular) or not (Combi).
- Find a boiler: `grep -i "ecotec pro,28" pcdb_gas_oil_boiler_codes.csv` or
filter by brand/model; the cert's lodged `main_heating_index_number` is the
`pcdb_id`.
**Note on entering it in Elmhurst:** the on-screen *PCDF boiler Reference* text
box is editable and will *display* a number you type, but typing alone does NOT
re-resolve the boiler — its type/efficiency/cylinder behaviour stays from the
previously-selected boiler. Only selecting via the 🔍 **search dialog** genuinely
sets the boiler. This CSV tells you which Ref to pick.

39
domain/epc/property_overlays/construction_age_band_overlay.py Normal file
View file

@ -0,0 +1,39 @@
"""Map a Landlord-Override construction-age-band value to a fabric Simulation
Overlay.
A construction-age-band value is the RdSAP England-&-Wales letter code (A..M)
the calculator's U-value cascades key on (`SapBuildingPart.construction_age_band`,
read via `.strip().upper()` against the letter-code bands). The overlay targets
the override's building part and sets the band; an unrecognised code produces no
overlay. Re-dating a part re-derives its construction-default U-values, so this
is the highest-leverage fabric override.
"""
from __future__ import annotations
from typing import Optional
from datatypes.epc.domain.epc_property_data import BuildingPartIdentifier
from domain.modelling.simulation import BuildingPartOverlay, EpcSimulation
# RdSAP England-&-Wales construction age bands (letter codes A..M).
_VALID_AGE_BANDS: frozenset[str] = frozenset("ABCDEFGHIJKLM")
def age_band_overlay_for(
age_band_value: str, building_part: int
) -> Optional[EpcSimulation]:
band = age_band_value.strip().upper()
if band not in _VALID_AGE_BANDS:
return None
identifier = (
BuildingPartIdentifier.MAIN
if building_part == 0
else BuildingPartIdentifier.extension(building_part)
)
return EpcSimulation(
building_parts={
identifier: BuildingPartOverlay(construction_age_band=band)
}
)

36
domain/epc/property_overlays/glazing_overlay.py Normal file
View file

@ -0,0 +1,36 @@
"""Map a Landlord-Override glazing value to a glazing Simulation Overlay.
A glazing value is one canonical glazing description carrying type + era
("Double glazing, 2002 or later", "Single glazing", "Triple glazing, 2002 or
later"). The calculator derives each window's U-value from its SAP10
`glazing_type` code via the RdSAP Table 24 cascade, so the overlay decomposes
the value into that code and emits a whole-dwelling `GlazingOverlay` (a landlord
describes the dwelling's glazing as a whole, with no per-window geometry, so
`building_part` is ignored). `_fold_glazing` expands it across every window.
Unresolvable values produce no overlay.
"""
from __future__ import annotations
from typing import Optional
from domain.modelling.simulation import EpcSimulation, GlazingOverlay
# Canonical glazing description → SAP10 glazing-type code (the Table 24 /
# `u_window` cascade enum, `_GLAZING_CODE_TO_UWINDOW` in heat_transmission).
_GLAZING_CODES: dict[str, int] = {
"Single glazing": 1,
"Double glazing, 2002 or later": 2,
"Double glazing, pre-2002": 3,
"Triple glazing, pre-2002": 6,
"Triple glazing, 2002 or later": 9,
}
def glazing_overlay_for(
glazing_value: str, building_part: int
) -> Optional[EpcSimulation]:
code = _GLAZING_CODES.get(glazing_value)
if code is None:
return None
return EpcSimulation(glazing=GlazingOverlay(glazing_type=code))

41
domain/epc/property_overlays/main_fuel_overlay.py Normal file
View file

@ -0,0 +1,41 @@
"""Map a Landlord-Override main-fuel value to a heating Simulation Overlay.
A main-fuel value is one canonical gov-EPC `main_fuel` description ("mains gas",
"electricity", …). The calculator reads the dwelling's primary fuel from
`main_heating_details[0].main_fuel_type` as the RdSAP **int code**, so the
overlay decomposes the value into that code and emits a whole-dwelling
`HeatingOverlay` (fuel is not a per-building-part attribute, so `building_part`
is ignored). Codes follow the modern RdSAP-20/21 `(not community)` family the
gov-EPC API baseline uses. Unresolvable values produce no overlay.
"""
from __future__ import annotations
from typing import Optional
from domain.modelling.simulation import EpcSimulation, HeatingOverlay
# RdSAP-20/21 `main_fuel` `(not community)` codes (epc_codes.csv `main_fuel`).
_FUEL_CODES: dict[str, int] = {
"mains gas": 26,
"mains gas (community)": 20,
"LPG (bulk)": 27,
"bottled LPG": 3,
"LPG special condition": 17,
"oil": 28,
"electricity": 29,
"electricity (community)": 25,
"house coal": 33,
"smokeless coal": 15,
"dual fuel (mineral and wood)": 10,
"biomass (community)": 31,
}
def fuel_overlay_for(
main_fuel_value: str, building_part: int
) -> Optional[EpcSimulation]:
code = _FUEL_CODES.get(main_fuel_value)
if code is None:
return None
return EpcSimulation(heating=HeatingOverlay(main_fuel_type=code))

46
domain/epc/property_overlays/main_heating_system_overlay.py Normal file
View file

@ -0,0 +1,46 @@
"""Map a Landlord-Override main-heating-system value to a heating Simulation Overlay.
A main-heating-system value is one canonical system archetype ("Gas boiler,
combi", "Electric storage heaters, fan"). The calculator reads the primary
system's `sap_main_heating_code` (SAP Table 4a/4b), so the overlay maps the
archetype to a representative code and emits a whole-dwelling `HeatingOverlay`
targeting `main_heating_details[0]` (`building_part` is ignored). It composes
field-wise with the main_fuel / water_heating overlays.
The SEDBUK A-G efficiency band the Hyde "Heating" column carries is NOT honoured
yet (no efficiency slot on the overlay/MainHeatingDetail) -- archetypes map to
their modern/condensing Table 4b code, so an old low-rated boiler is currently
modelled at the condensing efficiency. Heat pumps and community heating (which
resolve via main_heating_index_number / community codes, not a Table 4b code)
are left UNKNOWN until modelled. Unresolvable values produce no overlay.
"""
from __future__ import annotations
from typing import Optional
from domain.modelling.simulation import EpcSimulation, HeatingOverlay
# Canonical system archetype → representative `sap_main_heating_code` (SAP Table
# 4b boiler rows / Table 4a). Codes map to the modern/condensing variant (A-G
# efficiency deferred): 102 regular condensing, 104 condensing combi, 120 CPSU,
# 404 fan storage heaters, 191 direct-acting electric boiler.
_MAIN_HEATING_CODES: dict[str, int] = {
"Gas boiler, combi": 104,
"Gas boiler, regular": 102,
"Gas CPSU": 120,
"Electric storage heaters, old": 401,
"Electric storage heaters, slimline": 402,
"Electric storage heaters, convector": 403,
"Electric storage heaters, fan": 404,
"Direct-acting electric": 191,
}
def main_heating_overlay_for(
main_heating_value: str, building_part: int
) -> Optional[EpcSimulation]:
code = _MAIN_HEATING_CODES.get(main_heating_value)
if code is None:
return None
return EpcSimulation(heating=HeatingOverlay(sap_main_heating_code=code))

61
domain/epc/property_overlays/roof_type_overlay.py
View file

@ -1,12 +1,21 @@
"""Map a Landlord-Override `RoofType` value to a roof Simulation Overlay (ADR-0032).
"""Map a Landlord-Override `RoofType` value to a roof Simulation Overlay (ADR-0032/0033).
The calculator derives the roof U-value from the building part's loft-insulation
depth, so a `roof_type` override moves the score only via
`BuildingPartOverlay.roof_insulation_thickness` (mm). The resolvable family is
the explicit `"Pitched, N mm loft insulation"` values — N is parsed out.
Everything else (flat roofs, room-in-roof, "Unknown loft insulation",
"Another Premises Above" — a flat with a dwelling above, no roof to insulate) has
no clean loft depth, so it produces no overlay.
Two resolvable families:
* `"Pitched, N mm loft insulation"` — the loft depth N maps to
`BuildingPartOverlay.roof_insulation_thickness` (mm); the calculator scores the
roof from that depth.
* `"Flat, …"` — a flat roof carries no loft depth, so its U-value comes from the
age-band default (`_FLAT_ROOF_BY_AGE`, ADR-0033). The calculator's flat path
keys on the `roof_construction_type` *string* (`"flat" in …`), so the overlay
sets that to `"Flat"` and leaves thickness `None` for the (separately overlaid)
construction age band to drive the U-value. No flat `RoofType` value carries an
explicit mm depth, confirmed by the Elmhurst sweep (As Built / Unknown ≡
age-band default).
Everything else (room-in-roof, "Unknown loft insulation", party-ceiling adjacency
markers like "Another Premises Above") has no clean depth or shape correction, so
it produces no overlay.
"""
from __future__ import annotations
@ -23,8 +32,8 @@ _LOFT_MM = re.compile(r"(\d+)\+?\s*mm loft insulation")
def roof_overlay_for(
roof_type_value: str, building_part: int
) -> Optional[EpcSimulation]:
match = _LOFT_MM.search(roof_type_value)
if match is None:
overlay = _overlay_for(roof_type_value)
if overlay is None:
return None
identifier = (
@ -32,10 +41,28 @@ def roof_overlay_for(
if building_part == 0
else BuildingPartIdentifier.extension(building_part)
)
return EpcSimulation(
building_parts={
identifier: BuildingPartOverlay(
roof_insulation_thickness=int(match.group(1))
)
}
)
return EpcSimulation(building_parts={identifier: overlay})
def _overlay_for(roof_type_value: str) -> Optional[BuildingPartOverlay]:
match = _LOFT_MM.search(roof_type_value)
if match is not None:
return BuildingPartOverlay(roof_insulation_thickness=int(match.group(1)))
if roof_type_value.startswith("Flat,"):
# Flat roof: U-value is the age-band default; flag the shape and let the
# age-band overlay drive `_FLAT_ROOF_BY_AGE` (ADR-0033).
return BuildingPartOverlay(roof_construction_type="Flat")
if roof_type_value == "Pitched, Unknown loft insulation":
# Unknown loft depth: U-value is the pitched age-band default. Assert the
# pitched shape (keeps `is_flat_roof`/`is_sloping_ceiling` False) and leave
# thickness None so the age-band overlay drives `_ROOF_BY_AGE` (ADR-0033).
return BuildingPartOverlay(roof_construction_type="Pitched")
if roof_type_value.startswith("Pitched, no insulation"):
# Genuinely uninsulated pitched roof → 2.30 (Table 16 row 0). Thickness 0
# drives the calculator's thickness path and, unlike "Unknown", overrides
# any lodged numeric thickness (ADR-0033). Pitched text is load-bearing
# here — it does NOT take the age-band default (rdsap_uvalues §5.11 note).
return BuildingPartOverlay(
roof_construction_type="Pitched", roof_insulation_thickness=0
)
return None

8
domain/epc/property_overlays/wall_type_overlay.py
View file

@ -34,8 +34,16 @@ _MATERIAL_CONSTRUCTION: dict[str, int] = {
# RdSAP `wall_insulation_type` codes by insulation-state suffix
# (domain/sap10_ml/rdsap_uvalues.py): external 1, filled-cavity 2, internal 3,
# as-built/uninsulated 4, cavity+external 6, cavity+internal 7.
# All three "as built (assumed)" variants resolve to the same as-built code (4);
# the construction-age-band overlay supplies the U-value, so "partial" / "insulated"
# need no distinct code (ADR-0033, confirmed by the full A–M Elmhurst sweep in
# scripts/hyde/uvalue_probe_walls.csv — cavity As Built ≡ (CAVITY,0) by age band,
# e.g. band F → 1.0 = "partial"). The bare "as built" covers Cob / Park home.
_STATE_INSULATION: dict[str, int] = {
"as built": 4,
"as built, no insulation (assumed)": 4,
"as built, partial insulation (assumed)": 4,
"as built, insulated (assumed)": 4,
"with internal insulation": 3,
"with external insulation": 1,
"filled cavity": 2,

47
domain/epc/property_overlays/water_heating_overlay.py Normal file
View file

@ -0,0 +1,47 @@
"""Map a Landlord-Override water-heating value to a heating Simulation Overlay.
A water-heating value is one canonical "<system>, <fuel>" description ("From main
system, mains gas", "Electric immersion, electricity"). The calculator reads the
hot-water arrangement from `sap_heating.water_heating_code` (the SAP Table 4a
system code) and `water_heating_fuel`, so the overlay decomposes the value into
those two int codes and emits a whole-dwelling `HeatingOverlay` (water heating is
not per-building-part, so `building_part` is ignored). It composes field-wise with
the main_fuel / main_heating overlays. Unresolvable values produce no overlay.
"""
from __future__ import annotations
from typing import Optional
from domain.modelling.simulation import EpcSimulation, HeatingOverlay
# Canonical "<system>, <fuel>" description → (water_heating_code, water_heating_fuel).
# water_heating_code: 901 "from main system" (SAP Table 4a inherit-from-main),
# 903 "electric immersion". Fuel codes are the modern RdSAP "(not community)"
# family (26 mains gas, 29 electricity), matching the main_fuel overlay.
_WATER_HEATING_CODES: dict[str, tuple[int, int]] = {
"From main system, mains gas": (901, 26),
"From main system, electricity": (901, 29),
"From main system, oil": (901, 28),
"From main system, LPG (bulk)": (901, 27),
"From main system, bottled LPG": (901, 3),
"From main system, house coal": (901, 33),
"Electric immersion, electricity": (903, 29),
# "boiler/circulator for water heating only" — SAP Table 4a code 911 (gas).
"Gas boiler/circulator, mains gas": (911, 26),
}
def water_heating_overlay_for(
water_heating_value: str, building_part: int
) -> Optional[EpcSimulation]:
codes = _WATER_HEATING_CODES.get(water_heating_value)
if codes is None:
return None
water_heating_code, water_heating_fuel = codes
return EpcSimulation(
heating=HeatingOverlay(
water_heating_code=water_heating_code,
water_heating_fuel=water_heating_fuel,
)
)

4
domain/epc/property_overrides/__init__.py Normal file
View file

@ -0,0 +1,4 @@
"""Landlord property-override classifier vocabulary — the category enums a
landlord description resolves into, plus their value→code helpers. The classifier
target for the property_overrides chain (mirrors the property_overrides table /
override_component pgEnum). Distinct from the EPC-context types of the same name."""

0
domain/epc/built_form_type.py → domain/epc/property_overrides/built_form_type.py
View file

31
domain/epc/property_overrides/construction_age_band.py Normal file
View file

@ -0,0 +1,31 @@
from enum import Enum
class ConstructionAgeBand(Enum):
"""A landlord-supplied construction age band, as resolved by the
landlord-description-overrides context.
Each member's value is the RdSAP England-&-Wales age-band **letter code**
(A..M) the calculator's U-value cascades read from
`SapBuildingPart.construction_age_band` — the same representation the gov-EPC
API lodges. The construction-age-band Simulation Overlay
(``domain/epc/property_overlays/construction_age_band_overlay.py``) sets the
letter directly, so these values MUST stay the bare letter codes. Member
names carry the year ranges for readability. ``UNKNOWN`` covers values the
classifier cannot resolve (it leaves the lodged cert's age band untouched).
"""
A_BEFORE_1900 = "A"
B_1900_1929 = "B"
C_1930_1949 = "C"
D_1950_1966 = "D"
E_1967_1975 = "E"
F_1976_1982 = "F"
G_1983_1990 = "G"
H_1991_1995 = "H"
I_1996_2002 = "I"
J_2003_2006 = "J"
K_2007_2011 = "K"
L_2012_2022 = "L"
M_2023_ONWARDS = "M"
UNKNOWN = "Unknown"

24
domain/epc/property_overrides/glazing_type.py Normal file
View file

@ -0,0 +1,24 @@
from enum import Enum
class GlazingType(Enum):
"""A landlord-supplied glazing description, as resolved by the
landlord-description-overrides context.
Each member's value is the canonical glazing description (type + era) that
the glazing Simulation Overlay
(``domain/epc/property_overlays/glazing_overlay.py``) decomposes into the
SAP10 ``glazing_type`` code the calculator's Table-24 cascade reads — so the
member values here MUST stay in lock-step with that overlay's
``_GLAZING_CODES`` keys. The era matters: double-glazing pre-2002 and
2002-onward resolve to different codes (and U-values). ``UNKNOWN`` covers
values the classifier cannot resolve, and any glazing not yet given a
verified overlay code (it leaves the lodged cert's glazing untouched).
"""
SINGLE = "Single glazing"
DOUBLE_POST_2002 = "Double glazing, 2002 or later"
DOUBLE_PRE_2002 = "Double glazing, pre-2002"
TRIPLE_PRE_2002 = "Triple glazing, pre-2002"
TRIPLE_POST_2002 = "Triple glazing, 2002 or later"
UNKNOWN = "Unknown"

29
domain/epc/property_overrides/main_fuel_type.py Normal file
View file

@ -0,0 +1,29 @@
from enum import Enum
class MainFuelType(Enum):
"""A landlord-supplied main-fuel description, as resolved by the
landlord-description-overrides context.
Each member's value is the canonical fuel description that the main-fuel
Simulation Overlay (``domain/epc/property_overlays/main_fuel_overlay.py``)
decomposes into the RdSAP ``main_fuel`` int code the calculator reads — so
the member values here MUST stay in lock-step with that overlay's
``_FUEL_CODES`` keys. ``UNKNOWN`` covers values the classifier cannot
resolve, and also any fuel not yet given a verified overlay code (it leaves
the lodged cert's fuel untouched rather than guessing).
"""
MAINS_GAS = "mains gas"
MAINS_GAS_COMMUNITY = "mains gas (community)"
ELECTRICITY = "electricity"
ELECTRICITY_COMMUNITY = "electricity (community)"
LPG_BULK = "LPG (bulk)"
LPG_BOTTLED = "bottled LPG"
LPG_SPECIAL_CONDITION = "LPG special condition"
OIL = "oil"
HOUSE_COAL = "house coal"
SMOKELESS_COAL = "smokeless coal"
DUAL_FUEL_MINERAL_WOOD = "dual fuel (mineral and wood)"
BIOMASS_COMMUNITY = "biomass (community)"
UNKNOWN = "Unknown"

27
domain/epc/property_overrides/main_heating_system_type.py Normal file
View file

@ -0,0 +1,27 @@
from enum import Enum
class MainHeatingSystemType(Enum):
"""A landlord-supplied main-heating-system description, as resolved by the
landlord-description-overrides context.
Each member's value is the canonical system archetype that the main-heating
Simulation Overlay
(``domain/epc/property_overlays/main_heating_system_overlay.py``) maps to a
representative SAP ``sap_main_heating_code`` — so the member values MUST stay
in lock-step with that overlay's ``_MAIN_HEATING_CODES`` keys. The SEDBUK A-G
efficiency band the Hyde "Heating" column carries is NOT modelled yet
(deferred), so archetypes map to their modern/condensing code. ``UNKNOWN``
covers values the classifier cannot resolve and the not-yet-modelled systems
(heat pumps, community heating).
"""
GAS_COMBI = "Gas boiler, combi"
GAS_REGULAR = "Gas boiler, regular"
GAS_CPSU = "Gas CPSU"
ELECTRIC_STORAGE_OLD = "Electric storage heaters, old"
ELECTRIC_STORAGE_SLIMLINE = "Electric storage heaters, slimline"
ELECTRIC_STORAGE_CONVECTOR = "Electric storage heaters, convector"
ELECTRIC_STORAGE_FAN = "Electric storage heaters, fan"
DIRECT_ELECTRIC = "Direct-acting electric"
UNKNOWN = "Unknown"

0
domain/epc/override_code_mapping.py → domain/epc/property_overrides/override_code_mapping.py
View file

0
domain/epc/property_type.py → domain/epc/property_overrides/property_type.py
View file

0
domain/epc/roof_type.py → domain/epc/property_overrides/roof_type.py
View file

0
domain/epc/wall_type.py → domain/epc/property_overrides/wall_type.py
View file

2
domain/epc/wall_type_construction_dates.py → domain/epc/property_overrides/wall_type_construction_dates.py
View file

@ -27,7 +27,7 @@ from __future__ import annotations
from dataclasses import dataclass
from typing import Mapping, Optional
from domain.epc.wall_type import WallType
from domain.epc.property_overrides.wall_type import WallType
@dataclass(frozen=True)

26
domain/epc/property_overrides/water_heating_type.py Normal file
View file

@ -0,0 +1,26 @@
from enum import Enum
class WaterHeatingType(Enum):
"""A landlord-supplied water-heating description, as resolved by the
landlord-description-overrides context.
Each member's value is the canonical "<system>, <fuel>" description that the
water-heating Simulation Overlay
(``domain/epc/property_overlays/water_heating_overlay.py``) decomposes into
the SAP ``water_heating_code`` + ``water_heating_fuel`` int codes the
calculator reads — so the member values MUST stay in lock-step with that
overlay's ``_WATER_HEATING_CODES`` keys. ``UNKNOWN`` covers values the
classifier cannot resolve, and any combination not yet given verified codes
(it leaves the lodged cert's hot-water arrangement untouched).
"""
FROM_MAIN_MAINS_GAS = "From main system, mains gas"
FROM_MAIN_ELECTRICITY = "From main system, electricity"
FROM_MAIN_OIL = "From main system, oil"
FROM_MAIN_LPG_BULK = "From main system, LPG (bulk)"
FROM_MAIN_BOTTLED_LPG = "From main system, bottled LPG"
FROM_MAIN_HOUSE_COAL = "From main system, house coal"
ELECTRIC_IMMERSION = "Electric immersion, electricity"
GAS_BOILER_CIRCULATOR_MAINS_GAS = "Gas boiler/circulator, mains gas"
UNKNOWN = "Unknown"

18
domain/modelling/scoring/overlay_applicator.py
View file

@ -19,6 +19,7 @@ from datatypes.epc.domain.epc_property_data import (
)
from domain.modelling.simulation import (
EpcSimulation,
GlazingOverlay,
HeatingOverlay,
LightingOverlay,
SecondaryHeatingOverlay,
@ -53,6 +54,8 @@ def apply_simulations(
)
if simulation.lighting is not None:
_fold_lighting(result, simulation.lighting)
if simulation.glazing is not None:
_fold_glazing(result, simulation.glazing)
if simulation.heating is not None:
_fold_heating(result, simulation.heating)
if simulation.secondary_heating is not None:
@ -202,6 +205,21 @@ def _fold_window(window: SapWindow, overlay: WindowOverlay) -> None:
details.solar_transmittance = overlay.solar_transmittance
def _fold_glazing(epc: EpcPropertyData, overlay: GlazingOverlay) -> None:
"""Expand a whole-dwelling `GlazingOverlay` across every window: set each
window's `glazing_type` to the corrected SAP10 code AND clear its lodged
transmission U, so `heat_transmission`'s Table-24 cascade re-derives U from
the new type (the lodged U was for the old, mis-recorded glazing). A landlord
glazing override carries no per-window geometry, so it applies uniformly —
the expansion lives here because the baseline window list is known only at
fold time."""
if overlay.glazing_type is None:
return
for window in epc.sap_windows:
window.glazing_type = overlay.glazing_type
window.window_transmission_details = None
def _fold_ventilation(
baseline: Optional[SapVentilation], overlay: VentilationOverlay
) -> SapVentilation:

35
domain/modelling/simulation.py
View file

@ -28,6 +28,12 @@ class BuildingPartOverlay:
# The wall material (RdSAP `wall_construction` code). Left `None` by Measures
# — insulating a wall doesn't change its material — but set by a Landlord
# Override that corrects the construction itself (ADR-0032).
# RdSAP England-&-Wales construction age band — the letter code A..M the
# calculator's U-value cascades key on (`SapBuildingPart.construction_age_band`).
# Left `None` by Measures (retrofits don't change build era); set by a Landlord
# Override that corrects the lodged age band, which re-derives this part's
# fabric U-value defaults. Folds onto the part via the generic field loop.
construction_age_band: Optional[str] = None
wall_construction: Optional[int] = None
wall_insulation_type: Optional[int] = None
# Added solid-wall insulation depth (mm) — drives the calculator's Table 6
@ -35,6 +41,12 @@ class BuildingPartOverlay:
# IWI (`wall_insulation_type=3`); λ defaults to 0.04 W/m·K in the calculator.
wall_insulation_thickness: Optional[int] = None
roof_insulation_thickness: Optional[int] = None
# The roof shape string the calculator's flat-roof path keys on
# (`heat_transmission.py`: `"flat" in roof_construction_type`). Left `None`
# by Measures (insulating a roof doesn't change its shape); set by a Landlord
# Override that corrects a roof to flat so the `_FLAT_ROOF_BY_AGE` age-band
# default applies (ADR-0033). Folds onto the part via the generic field loop.
roof_construction_type: Optional[str] = None
floor_insulation_thickness: Optional[int] = None
floor_insulation_type_str: Optional[str] = None
@ -73,6 +85,28 @@ class WindowOverlay:
solar_transmittance: Optional[float] = None
@dataclass(frozen=True)
class GlazingOverlay:
"""All-optional partial of the dwelling's whole-glazing state — the
correction a Landlord Override makes when the lodged glazing is wrong.
Unlike a per-window `WindowOverlay` (keyed by `sap_windows` index), this
targets no single window: a landlord describes the dwelling's glazing as a
whole ("Double glazing, 2002 or later") with no per-window geometry, so the
overlay builder (which never sees the baseline window list) emits one of
these and `_fold_glazing` expands it across every `sap_windows` entry.
`glazing_type` is the SAP10 glazing-type code (Table 24 / `u_window`
cascade: 1=single, 2=double 2002-2021, 3=double pre-2002, 9=triple 2002+,
…). The fold sets it on every window AND clears each window's lodged
transmission U-value, so the Table-24 cascade re-derives the corrected U
from the new type (the lodged U was for the OLD, mis-recorded glazing).
A `None` field means "leave the baseline value unchanged".
"""
glazing_type: Optional[int] = None
@dataclass(frozen=True)
class LightingOverlay:
"""All-optional partial of the dwelling's fixed-lighting bulb counts — the
@ -220,6 +254,7 @@ class EpcSimulation:
windows: Mapping[int, WindowOverlay] = field(default_factory=_no_windows)
ventilation: Optional[VentilationOverlay] = None
lighting: Optional[LightingOverlay] = None
glazing: Optional[GlazingOverlay] = None
heating: Optional[HeatingOverlay] = None
secondary_heating: Optional[SecondaryHeatingOverlay] = None
solar: Optional[SolarOverlay] = None

23
domain/sap10_calculator/rdsap/cert_to_inputs.py
View file

@ -2579,9 +2579,13 @@ def _hot_water_fuel_cost_gbp_per_kwh(
billed at the high rate, the remainder at the low rate. Without it
the immersion HW billed 100% at the off-peak low rate, under-costing
the dwelling and over-rating it (median +0.98 SAP across the off-peak
WHC-903 API cohort). Needs `cylinder_volume_l` + `occupancy_n` +
`immersion_single`; absent any of them (no cylinder / volume not
resolvable) it falls back to the 100%-low-rate scalar.
WHC-903 API cohort). Needs `cylinder_volume_l` + `occupancy_n`; when
`immersion_single` is unlodged (None) this branch is on an off-peak /
dual meter, so per RdSAP 10 §10.5 (PDF p.54) the immersion is assumed
DUAL rather than falling back to the 100%-low-rate scalar (Elmhurst
applies the dual blend, e.g. uprn_10022893721 high-rate fraction
0.1386). Only an unresolvable cylinder volume / occupancy now falls
back to the 100%-low-rate scalar.
`inherit_main_for_community_heating`: per S0380.173, when WHC
∈ {901, 902, 914} AND main is a heat network, ignore the cert-
@ -2610,13 +2614,22 @@ def _hot_water_fuel_cost_gbp_per_kwh(
water_heating_code == _WHC_ELECTRIC_IMMERSION
and cylinder_volume_l is not None
and occupancy_n is not None
and immersion_single is not None
):
# RdSAP 10 §10.5 (PDF p.54): an immersion is assumed DUAL on a dual /
# off-peak meter. This branch is only reached on an off-peak tariff
# (tariff is not STANDARD ⇒ the meter is dual / Economy-7), so when the
# cert does not lodge `immersion_heating_type` default to dual rather
# than dropping to the 100%-low-rate fallback. The fallback under-costs
# the DHW and over-rates the dwelling, whereas Elmhurst applies the
# Table 13 dual blend (e.g. uprn_10022893721 high-rate fraction 0.1386).
effective_single = (
immersion_single if immersion_single is not None else False
)
high_rate, low_rate = _tariff_high_low_rates_p_per_kwh(tariff)
high_frac = electric_dhw_high_rate_fraction(
cylinder_volume_l=cylinder_volume_l,
occupancy_n=occupancy_n,
single_immersion=immersion_single,
single_immersion=effective_single,
tariff=tariff,
)
blended = high_frac * high_rate + (1.0 - high_frac) * low_rate

20
infrastructure/landlord_overrides/landlord_override_reader_postgres_repository.py
View file

@ -25,9 +25,24 @@ from infrastructure.postgres.landlord_property_type_override_table import (
from infrastructure.postgres.landlord_roof_type_override_table import (
LandlordRoofTypeOverrideRow,
)
from infrastructure.postgres.landlord_construction_age_band_override_table import (
LandlordConstructionAgeBandOverrideRow,
)
from infrastructure.postgres.landlord_glazing_override_table import (
LandlordGlazingOverrideRow,
)
from infrastructure.postgres.landlord_main_fuel_override_table import (
LandlordMainFuelOverrideRow,
)
from infrastructure.postgres.landlord_main_heating_system_override_table import (
LandlordMainHeatingSystemOverrideRow,
)
from infrastructure.postgres.landlord_wall_type_override_table import (
LandlordWallTypeOverrideRow,
)
from infrastructure.postgres.landlord_water_heating_override_table import (
LandlordWaterHeatingOverrideRow,
)
from repositories.landlord_overrides.landlord_override_reader import (
LandlordOverrideReader,
)
@ -38,6 +53,11 @@ _ROW_TYPES: dict[str, type] = {
"built_form_type": LandlordBuiltFormTypeOverrideRow,
"wall_type": LandlordWallTypeOverrideRow,
"roof_type": LandlordRoofTypeOverrideRow,
"main_fuel": LandlordMainFuelOverrideRow,
"glazing": LandlordGlazingOverrideRow,
"construction_age_band": LandlordConstructionAgeBandOverrideRow,
"water_heating": LandlordWaterHeatingOverrideRow,
"main_heating_system": LandlordMainHeatingSystemOverrideRow,
}

2
infrastructure/postgres/landlord_built_form_type_override_table.py
View file

@ -16,7 +16,7 @@ from sqlalchemy import BigInteger, Column, UniqueConstraint
from sqlalchemy import Enum as SAEnum
from sqlmodel import Field, SQLModel
from domain.epc.built_form_type import BuiltFormType
from domain.epc.property_overrides.built_form_type import BuiltFormType
from infrastructure.postgres.landlord_override_enums import override_source_sa_enum

73
infrastructure/postgres/landlord_construction_age_band_override_table.py Normal file
View file

@ -0,0 +1,73 @@
"""SQLModel mirror of the ``landlord_construction_age_band_overrides`` Drizzle table.
The schema source of truth lives in the ``assessment-model`` TS repo
(`src/app/db/schema/landlord_overrides.ts`). The migrations are owned there;
this row class only mirrors the columns so the Python lambda can read/write.
See ADR-0003. Shape mirrors ``LandlordWallTypeOverrideRow`` -- the only
differences are the table name, the ``construction_age_band`` pgEnum on
``value``, and the unique-constraint name.
"""
from datetime import datetime, timezone
from typing import ClassVar
from uuid import UUID, uuid4
from sqlalchemy import BigInteger, Column, UniqueConstraint
from sqlalchemy import Enum as SAEnum
from sqlmodel import Field, SQLModel
from domain.epc.property_overrides.construction_age_band import ConstructionAgeBand
from infrastructure.postgres.landlord_override_enums import override_source_sa_enum
class LandlordConstructionAgeBandOverrideRow(SQLModel, table=True):
__tablename__: ClassVar[str] = "landlord_construction_age_band_overrides" # pyright: ignore[reportIncompatibleVariableOverride]
__table_args__: ClassVar[tuple[UniqueConstraint, ...]] = ( # pyright: ignore[reportIncompatibleVariableOverride]
# NB: shortened (drop the redundant ``_overrides``) to stay within
# PostgreSQL's 63-char identifier limit -- the full
# ``landlord_construction_age_band_overrides_portfolio_description_unique``
# is 68 chars and would be silently truncated, diverging from Drizzle.
UniqueConstraint(
"portfolio_id",
"description",
name="landlord_construction_age_band_portfolio_description_unique",
),
)
id: UUID = Field(default_factory=uuid4, primary_key=True)
# bigint to match the Drizzle ``portfolio_id`` FK; SQLModel's default int
# mapping is 32-bit Integer and would overflow once portfolio IDs exceed
# 2^31. The FK to ``portfolio.id`` is enforced by the Drizzle migration,
# not declared here -- the ``portfolio`` table is not modelled in Python.
portfolio_id: int = Field(
sa_column=Column(BigInteger, nullable=False, index=True),
)
description: str = Field(nullable=False)
value: ConstructionAgeBand = Field(
sa_column=Column(
SAEnum(
ConstructionAgeBand,
name="construction_age_band",
values_callable=lambda cls: [m.value for m in cls], # pyright: ignore[reportUnknownLambdaType, reportUnknownMemberType, reportUnknownVariableType]
),
nullable=False,
),
)
# Shared SAEnum -- see ``landlord_override_enums`` for why this single
# instance is reused by every ``landlord_*_overrides`` row class.
source: str = Field(
sa_column=Column(override_source_sa_enum, nullable=False),
)
created_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
nullable=False,
)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
nullable=False,
)

69
infrastructure/postgres/landlord_glazing_override_table.py Normal file
View file

@ -0,0 +1,69 @@
"""SQLModel mirror of the ``landlord_glazing_overrides`` Drizzle table.
The schema source of truth lives in the ``assessment-model`` TS repo
(`src/app/db/schema/landlord_overrides.ts`). The migrations are owned there;
this row class only mirrors the columns so the Python lambda can read/write.
See ADR-0003. Shape mirrors ``LandlordWallTypeOverrideRow`` -- the only
differences are the table name, the ``glazing`` pgEnum on ``value``, and the
unique-constraint name.
"""
from datetime import datetime, timezone
from typing import ClassVar
from uuid import UUID, uuid4
from sqlalchemy import BigInteger, Column, UniqueConstraint
from sqlalchemy import Enum as SAEnum
from sqlmodel import Field, SQLModel
from domain.epc.property_overrides.glazing_type import GlazingType
from infrastructure.postgres.landlord_override_enums import override_source_sa_enum
class LandlordGlazingOverrideRow(SQLModel, table=True):
__tablename__: ClassVar[str] = "landlord_glazing_overrides" # pyright: ignore[reportIncompatibleVariableOverride]
__table_args__: ClassVar[tuple[UniqueConstraint, ...]] = ( # pyright: ignore[reportIncompatibleVariableOverride]
UniqueConstraint(
"portfolio_id",
"description",
name="landlord_glazing_overrides_portfolio_description_unique",
),
)
id: UUID = Field(default_factory=uuid4, primary_key=True)
# bigint to match the Drizzle ``portfolio_id`` FK; SQLModel's default int
# mapping is 32-bit Integer and would overflow once portfolio IDs exceed
# 2^31. The FK to ``portfolio.id`` is enforced by the Drizzle migration,
# not declared here -- the ``portfolio`` table is not modelled in Python.
portfolio_id: int = Field(
sa_column=Column(BigInteger, nullable=False, index=True),
)
description: str = Field(nullable=False)
value: GlazingType = Field(
sa_column=Column(
SAEnum(
GlazingType,
name="glazing",
values_callable=lambda cls: [m.value for m in cls], # pyright: ignore[reportUnknownLambdaType, reportUnknownMemberType, reportUnknownVariableType]
),
nullable=False,
),
)
# Shared SAEnum -- see ``landlord_override_enums`` for why this single
# instance is reused by every ``landlord_*_overrides`` row class.
source: str = Field(
sa_column=Column(override_source_sa_enum, nullable=False),
)
created_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
nullable=False,
)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
nullable=False,
)

69
infrastructure/postgres/landlord_main_fuel_override_table.py Normal file
View file

@ -0,0 +1,69 @@
"""SQLModel mirror of the ``landlord_main_fuel_overrides`` Drizzle table.
The schema source of truth lives in the ``assessment-model`` TS repo
(`src/app/db/schema/landlord_overrides.ts`). The migrations are owned there;
this row class only mirrors the columns so the Python lambda can read/write.
See ADR-0003. Shape mirrors ``LandlordWallTypeOverrideRow`` -- the only
differences are the table name, the ``main_fuel`` pgEnum on ``value``, and
the unique-constraint name.
"""
from datetime import datetime, timezone
from typing import ClassVar
from uuid import UUID, uuid4
from sqlalchemy import BigInteger, Column, UniqueConstraint
from sqlalchemy import Enum as SAEnum
from sqlmodel import Field, SQLModel
from domain.epc.property_overrides.main_fuel_type import MainFuelType
from infrastructure.postgres.landlord_override_enums import override_source_sa_enum
class LandlordMainFuelOverrideRow(SQLModel, table=True):
__tablename__: ClassVar[str] = "landlord_main_fuel_overrides" # pyright: ignore[reportIncompatibleVariableOverride]
__table_args__: ClassVar[tuple[UniqueConstraint, ...]] = ( # pyright: ignore[reportIncompatibleVariableOverride]
UniqueConstraint(
"portfolio_id",
"description",
name="landlord_main_fuel_overrides_portfolio_description_unique",
),
)
id: UUID = Field(default_factory=uuid4, primary_key=True)
# bigint to match the Drizzle ``portfolio_id`` FK; SQLModel's default int
# mapping is 32-bit Integer and would overflow once portfolio IDs exceed
# 2^31. The FK to ``portfolio.id`` is enforced by the Drizzle migration,
# not declared here -- the ``portfolio`` table is not modelled in Python.
portfolio_id: int = Field(
sa_column=Column(BigInteger, nullable=False, index=True),
)
description: str = Field(nullable=False)
value: MainFuelType = Field(
sa_column=Column(
SAEnum(
MainFuelType,
name="main_fuel",
values_callable=lambda cls: [m.value for m in cls], # pyright: ignore[reportUnknownLambdaType, reportUnknownMemberType, reportUnknownVariableType]
),
nullable=False,
),
)
# Shared SAEnum -- see ``landlord_override_enums`` for why this single
# instance is reused by every ``landlord_*_overrides`` row class.
source: str = Field(
sa_column=Column(override_source_sa_enum, nullable=False),
)
created_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
nullable=False,
)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
nullable=False,
)

71
infrastructure/postgres/landlord_main_heating_system_override_table.py Normal file
View file

@ -0,0 +1,71 @@
"""SQLModel mirror of the ``landlord_main_heating_system_overrides`` Drizzle table.
The schema source of truth lives in the ``assessment-model`` TS repo
(`src/app/db/schema/landlord_overrides.ts`). The migrations are owned there;
this row class only mirrors the columns so the Python lambda can read/write.
See ADR-0003. Shape mirrors ``LandlordWallTypeOverrideRow`` -- the only
differences are the table name, the ``main_heating_system`` pgEnum on ``value``,
and the unique-constraint name.
"""
from datetime import datetime, timezone
from typing import ClassVar
from uuid import UUID, uuid4
from sqlalchemy import BigInteger, Column, UniqueConstraint
from sqlalchemy import Enum as SAEnum
from sqlmodel import Field, SQLModel
from domain.epc.property_overrides.main_heating_system_type import MainHeatingSystemType
from infrastructure.postgres.landlord_override_enums import override_source_sa_enum
class LandlordMainHeatingSystemOverrideRow(SQLModel, table=True):
__tablename__: ClassVar[str] = "landlord_main_heating_system_overrides" # pyright: ignore[reportIncompatibleVariableOverride]
__table_args__: ClassVar[tuple[UniqueConstraint, ...]] = ( # pyright: ignore[reportIncompatibleVariableOverride]
# Shortened (drop the redundant ``_overrides``) to stay within
# PostgreSQL's 63-char identifier limit; mirrors the Drizzle name.
UniqueConstraint(
"portfolio_id",
"description",
name="landlord_main_heating_system_portfolio_description_unique",
),
)
id: UUID = Field(default_factory=uuid4, primary_key=True)
# bigint to match the Drizzle ``portfolio_id`` FK; SQLModel's default int
# mapping is 32-bit Integer and would overflow once portfolio IDs exceed
# 2^31. The FK to ``portfolio.id`` is enforced by the Drizzle migration,
# not declared here -- the ``portfolio`` table is not modelled in Python.
portfolio_id: int = Field(
sa_column=Column(BigInteger, nullable=False, index=True),
)
description: str = Field(nullable=False)
value: MainHeatingSystemType = Field(
sa_column=Column(
SAEnum(
MainHeatingSystemType,
name="main_heating_system",
values_callable=lambda cls: [m.value for m in cls], # pyright: ignore[reportUnknownLambdaType, reportUnknownMemberType, reportUnknownVariableType]
),
nullable=False,
),
)
# Shared SAEnum -- see ``landlord_override_enums`` for why this single
# instance is reused by every ``landlord_*_overrides`` row class.
source: str = Field(
sa_column=Column(override_source_sa_enum, nullable=False),
)
created_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
nullable=False,
)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
nullable=False,
)

2
infrastructure/postgres/landlord_property_type_override_table.py
View file

@ -14,7 +14,7 @@ from sqlalchemy import BigInteger, Column, UniqueConstraint
from sqlalchemy import Enum as SAEnum
from sqlmodel import Field, SQLModel
from domain.epc.property_type import PropertyType
from domain.epc.property_overrides.property_type import PropertyType
from infrastructure.postgres.landlord_override_enums import override_source_sa_enum

2
infrastructure/postgres/landlord_roof_type_override_table.py
View file

@ -16,7 +16,7 @@ from sqlalchemy import BigInteger, Column, UniqueConstraint
from sqlalchemy import Enum as SAEnum
from sqlmodel import Field, SQLModel
from domain.epc.roof_type import RoofType
from domain.epc.property_overrides.roof_type import RoofType
from infrastructure.postgres.landlord_override_enums import override_source_sa_enum

2
infrastructure/postgres/landlord_wall_type_override_table.py
View file

@ -16,7 +16,7 @@ from sqlalchemy import BigInteger, Column, UniqueConstraint
from sqlalchemy import Enum as SAEnum
from sqlmodel import Field, SQLModel
from domain.epc.wall_type import WallType
from domain.epc.property_overrides.wall_type import WallType
from infrastructure.postgres.landlord_override_enums import override_source_sa_enum

69
infrastructure/postgres/landlord_water_heating_override_table.py Normal file
View file

@ -0,0 +1,69 @@
"""SQLModel mirror of the ``landlord_water_heating_overrides`` Drizzle table.
The schema source of truth lives in the ``assessment-model`` TS repo
(`src/app/db/schema/landlord_overrides.ts`). The migrations are owned there;
this row class only mirrors the columns so the Python lambda can read/write.
See ADR-0003. Shape mirrors ``LandlordWallTypeOverrideRow`` -- the only
differences are the table name, the ``water_heating`` pgEnum on ``value``, and
the unique-constraint name.
"""
from datetime import datetime, timezone
from typing import ClassVar
from uuid import UUID, uuid4
from sqlalchemy import BigInteger, Column, UniqueConstraint
from sqlalchemy import Enum as SAEnum
from sqlmodel import Field, SQLModel
from domain.epc.property_overrides.water_heating_type import WaterHeatingType
from infrastructure.postgres.landlord_override_enums import override_source_sa_enum
class LandlordWaterHeatingOverrideRow(SQLModel, table=True):
__tablename__: ClassVar[str] = "landlord_water_heating_overrides" # pyright: ignore[reportIncompatibleVariableOverride]
__table_args__: ClassVar[tuple[UniqueConstraint, ...]] = ( # pyright: ignore[reportIncompatibleVariableOverride]
UniqueConstraint(
"portfolio_id",
"description",
name="landlord_water_heating_overrides_portfolio_description_unique",
),
)
id: UUID = Field(default_factory=uuid4, primary_key=True)
# bigint to match the Drizzle ``portfolio_id`` FK; SQLModel's default int
# mapping is 32-bit Integer and would overflow once portfolio IDs exceed
# 2^31. The FK to ``portfolio.id`` is enforced by the Drizzle migration,
# not declared here -- the ``portfolio`` table is not modelled in Python.
portfolio_id: int = Field(
sa_column=Column(BigInteger, nullable=False, index=True),
)
description: str = Field(nullable=False)
value: WaterHeatingType = Field(
sa_column=Column(
SAEnum(
WaterHeatingType,
name="water_heating",
values_callable=lambda cls: [m.value for m in cls], # pyright: ignore[reportUnknownLambdaType, reportUnknownMemberType, reportUnknownVariableType]
),
nullable=False,
),
)
# Shared SAEnum -- see ``landlord_override_enums`` for why this single
# instance is reused by every ``landlord_*_overrides`` row class.
source: str = Field(
sa_column=Column(override_source_sa_enum, nullable=False),
)
created_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
nullable=False,
)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
nullable=False,
)

5
infrastructure/postgres/property_override_table.py
View file

@ -27,6 +27,11 @@ override_component_sa_enum = SAEnum(
"roof_type",
"property_type",
"built_form_type",
"main_fuel",
"glazing",
"construction_age_band",
"water_heating",
"main_heating_system",
name="override_component",
)

8
orchestration/bulk_upload_finaliser_orchestrator.py
View file

@ -14,10 +14,10 @@ from typing import Any, Optional
from uuid import UUID
from domain.epc.built_form_type import BuiltFormType
from domain.epc.property_type import PropertyType
from domain.epc.roof_type import RoofType
from domain.epc.wall_type import WallType
from domain.epc.property_overrides.built_form_type import BuiltFormType
from domain.epc.property_overrides.property_type import PropertyType
from domain.epc.property_overrides.roof_type import RoofType
from domain.epc.property_overrides.wall_type import WallType
from repositories.bulk_upload.bulk_upload_status_writer import BulkUploadStatusWriter
from repositories.landlord_overrides.landlord_override_reader import (
LandlordOverrideReader,

37
repositories/property/landlord_override_overlays.py
View file

@ -7,18 +7,23 @@ type — `domain/` never imports `repositories/`.
Per-component and partial — an override produces an overlay only where a
component mapping exists and the value resolves; anything else is left to the
lodged EPC. All four `override_component`s are mapped:
lodged EPC. Every `override_component` is mapped:
* `wall_type` → fabric overlay (`wall_construction` + `wall_insulation_type`)
* `roof_type` → fabric overlay (`roof_insulation_thickness`, loft-depth family)
* `roof_type` → fabric overlay (`roof_insulation_thickness` / `roof_construction_type`)
* `construction_age_band` → per-part `construction_age_band` — the U-value lever
for the "as built / assumed / Unknown" fabric states (ADR-0033)
* `property_type` / `built_form_type` → whole-dwelling categorical correction
* `main_fuel` / `glazing` / `water_heating` / `main_heating_system` → whole-dwelling
Two value families deliberately resolve to *no* overlay rather than a guess: the
`"(assumed) insulated"` / `"partial insulation (assumed)"` wall states (RdSAP
infers their U-value from the build-era age band, so there is no single
`wall_insulation_type` code for them — they need Elmhurst validation, ADR-0032),
and `"Unknown"` categorical values. Roofs with no clean loft depth (flat,
room-in-roof, "another premises above") likewise produce no overlay.
Since ADR-0033 the "as built (assumed)" wall states resolve to the as-built code
(4) and flat / pitched-Unknown / pitched-no-insulation roofs produce an overlay —
their U-value comes from the per-part `construction_age_band`, not a per-state
code. Values that still resolve to *no* overlay (left to the lodged EPC): bare
`"Unknown"` categoricals, party-ceiling roofs ("another/same dwelling above" — a
party ceiling has ≈0 heat loss), room-in-roof, and the residue logged in ADR-0033.
The `audit_override_coverage.py` / `audit_hyde_rows.py` pair reports exactly which
values are live vs no-op before any write.
"""
from __future__ import annotations
@ -29,6 +34,17 @@ from domain.epc.property_overlays.attribute_overlay import (
built_form_overlay_for,
property_type_overlay_for,
)
from domain.epc.property_overlays.construction_age_band_overlay import (
age_band_overlay_for,
)
from domain.epc.property_overlays.glazing_overlay import glazing_overlay_for
from domain.epc.property_overlays.main_fuel_overlay import fuel_overlay_for
from domain.epc.property_overlays.main_heating_system_overlay import (
main_heating_overlay_for,
)
from domain.epc.property_overlays.water_heating_overlay import (
water_heating_overlay_for,
)
from domain.epc.property_overlays.roof_type_overlay import roof_overlay_for
from domain.epc.property_overlays.wall_type_overlay import wall_overlay_for
from domain.modelling.simulation import EpcSimulation
@ -43,6 +59,11 @@ _COMPONENT_OVERLAYS: dict[str, Callable[[str, int], Optional[EpcSimulation]]] =
"roof_type": roof_overlay_for,
"property_type": property_type_overlay_for,
"built_form_type": built_form_overlay_for,
"main_fuel": fuel_overlay_for,
"glazing": glazing_overlay_for,
"construction_age_band": age_band_overlay_for,
"water_heating": water_heating_overlay_for,
"main_heating_system": main_heating_overlay_for,
}

2
repositories/property/override_backed_prediction_attributes_reader.py
View file

@ -13,7 +13,7 @@ from __future__ import annotations
from typing import Optional
from domain.epc.override_code_mapping import (
from domain.epc.property_overrides.override_code_mapping import (
built_form_to_code,
property_type_to_code,
)

353
scripts/fill_domna_addresses.py Normal file
View file

@ -0,0 +1,353 @@
"""Fill the DOMNA columns in the AddressProfilingResults spreadsheet.
Input: scripts/manipulation(2).xlsx, sheet "AddressProfilingResults", columns
Organisation Reference | UPRN | DOMNA FOUND UPRN | DOMNA FOUND ADDRESS | Address | Postcode
Per-row rule ("if there's a UPRN in the UPRN column we're done"):
* UPRN present AND Address present -> nothing to do (already sorted).
* UPRN present AND Address missing -> reverse-lookup the address from the UPRN
via the EPC API -> DOMNA FOUND ADDRESS.
* UPRN missing AND Address present -> resolve a UPRN from address + postcode
(EPC API, then Ordnance Survey) -> writes
DOMNA FOUND UPRN + DOMNA FOUND ADDRESS.
* not resolvable -> marked "NOT FOUND" and listed in the
unresolved report.
Relaxed matching (this batch only — production AddressMatch is untouched): the
landlord writes flats as "3 GLADYS COURT" while EPC stores "Flat 3 Gladys
Court", which the production matcher hard-rejects. So per address we try several
query variants — the full string, just the first comma-segment, and a
"Flat <n> ..." form — and keep the best-scoring, unambiguous match. The unit
number must still match exactly (AddressMatch zeroes mismatched numbers), so a
wrong-unit match stays unlikely. Each fill carries its score + source so you can
spot-check (DOMNA SCORE / DOMNA SOURCE).
Rows that already have a DOMNA FOUND UPRN are skipped (idempotent / resumable).
python -m scripts.fill_domna_addresses
python -m scripts.fill_domna_addresses --limit 200 # smoke test first N
Keys come from backend/.env (OPEN_EPC_API_TOKEN, ORDNANCE_SURVEY_API_KEY). Run
from the worktree root (import trap).
"""
from __future__ import annotations
import argparse
import os
import re
import sys
from pathlib import Path
from typing import Optional
import pandas as pd
_REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(_REPO_ROOT)) # worktree root first — avoid the import trap
from backend.address2UPRN.main import get_epc_data_with_postcode # noqa: E402
from backend.address2UPRN.scoring import all_uprns_match, rank_address_similarity # noqa: E402
from backend.ordnanceSurvey.helpers import ( # noqa: E402
lookup_os_places,
os_places_results_to_dataframe,
)
from backend.utils.addressMatch import AddressMatch # noqa: E402
from datatypes.epc.search import EpcSearchResult # noqa: E402
from infrastructure.epc_client.epc_client_service import EpcClientService # noqa: E402
from scripts.resolve_uprns_for_finaliser import clean_postcode, load_keys # noqa: E402
SHEET = "AddressProfilingResults"
UPRN_COL = "UPRN"
ADDRESS_COL = "Address"
POSTCODE_COL = "Postcode"
REF_COL = "Organisation Reference"
FOUND_UPRN_COL = "DOMNA FOUND UPRN"
FOUND_ADDRESS_COL = "DOMNA FOUND ADDRESS"
SCORE_COL = "DOMNA SCORE"
SOURCE_COL = "DOMNA SOURCE"
NOT_FOUND = "NOT FOUND"
# EPC matches are tight (short addresses) so we hold the production 0.7 bar; OS
# addresses carry more trailing tokens, so a slightly lower bar is appropriate.
EPC_THRESHOLD = 0.7
OS_THRESHOLD = 0.6
_DEFAULT_IN = _REPO_ROOT / "scripts" / "manipulation(2).xlsx"
_DEFAULT_OUT = _REPO_ROOT / "scripts" / "manipulation_filled.xlsx"
_DEFAULT_UNRESOLVED = _REPO_ROOT / "scripts" / "manipulation_unresolved.csv"
# A resolved hit: (uprn, matched_address, score, source).
Hit = tuple[str, str, float, str]
def cell_str(value: object) -> str:
"""Coerce a spreadsheet cell to a trimmed string ("" for NaN/None)."""
if value is None:
return ""
text = str(value).strip()
return "" if text.lower() == "nan" else text
def parse_uprn_cell(value: object) -> Optional[int]:
"""Read a UPRN cell that pandas loaded as float64 back into an int."""
text = cell_str(value)
if not text:
return None
try:
return int(float(text))
except ValueError:
return None
def address_variants(address: str) -> list[str]:
"""Query forms to try for one input address, best-discriminating first.
Landlord flats read "3 GLADYS COURT, 260 REIGATE ROAD" but EPC stores
"Flat 3 Gladys Court"; the full string scores low (extra tokens) and the
bare "3 ..." trips the flat guard. So we also try the first comma-segment
and a "Flat <segment>" form.
"""
address = address.strip()
first = address.split(",")[0].strip()
variants = [address, first]
if re.match(r"^\d", first): # starts with a unit/house number
variants.append("Flat " + first)
variants.append("Flat " + address)
seen: set[str] = set()
out: list[str] = []
for v in variants:
key = v.lower()
if v and key not in seen:
seen.add(key)
out.append(v)
return out
def resolve_epc_relaxed(
address: str,
postcode_clean: str,
epc_cache: dict[str, pd.DataFrame],
threshold: float = EPC_THRESHOLD,
) -> Optional[Hit]:
"""Best unambiguous EPC match across the address variants (cached per postcode)."""
epc_df = epc_cache.get(postcode_clean)
if epc_df is None:
epc_df = get_epc_data_with_postcode(postcode=postcode_clean)
epc_cache[postcode_clean] = epc_df
if epc_df.empty:
return None
best: Optional[Hit] = None
for variant in address_variants(address):
scored = rank_address_similarity(epc_df, user_address=variant)
if scored.empty:
continue
score = float(scored.iloc[0]["lexiscore"])
if best is not None and score <= best[2]:
continue
top_rank = scored[scored["lexirank"] == 1]
# rank-1 rows must agree on one UPRN, else it's ambiguous — skip.
if not all_uprns_match(top_rank, top_rank.iloc[0]["uprn"]):
continue
uprn = str(top_rank.iloc[0]["uprn"])
if uprn in ("", "nan"):
continue
best = (uprn, str(scored.iloc[0]["address"]), score, "epc")
return best if best is not None and best[2] >= threshold else None
def resolve_os_relaxed(
address: str,
postcode_clean: str,
os_api_key: str,
os_cache: dict[str, pd.DataFrame],
threshold: float = OS_THRESHOLD,
) -> Optional[Hit]:
"""Best OS Places match across the address variants (cached per postcode)."""
places_df = os_cache.get(postcode_clean)
if places_df is None:
response = lookup_os_places(postcode_clean, os_api_key)
if response.get("status") == 200 and "data" in response:
places_df = os_places_results_to_dataframe(response["data"])
else:
places_df = pd.DataFrame()
os_cache[postcode_clean] = places_df
if places_df.empty or "ADDRESS" not in places_df.columns:
return None
records: list[dict[str, object]] = places_df.to_dict(orient="records")
best: Optional[Hit] = None
for variant in address_variants(address):
for rec in records:
candidate = str(rec.get("ADDRESS", ""))
score = AddressMatch.score(variant, candidate)
if best is None or score > best[2]:
best = (str(rec.get("UPRN", "")), candidate, score, "ordnance_survey")
return best if best is not None and best[2] >= threshold else None
def _address_from_search(result: EpcSearchResult) -> str:
parts = [
result.address_line_1,
result.address_line_2,
result.address_line_3,
result.address_line_4,
result.post_town,
]
return ", ".join(p.strip() for p in parts if p and p.strip())
def reverse_address_from_uprn(
uprn: int,
postcode_clean: str,
service: EpcClientService,
search_cache: dict[str, list[EpcSearchResult]],
) -> Optional[str]:
"""Find the EPC address for a known UPRN by searching its postcode (cached)."""
results = search_cache.get(postcode_clean)
if results is None:
results = service.search_by_postcode(postcode_clean)
search_cache[postcode_clean] = results
for result in results:
if result.uprn is not None and int(result.uprn) == uprn:
return _address_from_search(result)
return None
def fill(df: pd.DataFrame, *, os_api_key: Optional[str]) -> list[dict[str, str]]:
"""Fill the DOMNA columns in place. Returns the unresolved rows."""
for col in (FOUND_UPRN_COL, FOUND_ADDRESS_COL, SCORE_COL, SOURCE_COL):
if col not in df.columns:
df[col] = ""
df[FOUND_UPRN_COL] = df[FOUND_UPRN_COL].astype("object")
df[FOUND_ADDRESS_COL] = df[FOUND_ADDRESS_COL].astype("object")
token = os.environ.get("OPEN_EPC_API_TOKEN")
service = EpcClientService(auth_token=token) if token else None
epc_cache: dict[str, pd.DataFrame] = {}
os_cache: dict[str, pd.DataFrame] = {}
search_cache: dict[str, list[EpcSearchResult]] = {}
unresolved: list[dict[str, str]] = []
resolved_uprn = resolved_addr = skipped = 0
total = len(df)
for n, idx in enumerate(df.index, start=1):
ref = cell_str(df.at[idx, REF_COL])
given_uprn = parse_uprn_cell(df.at[idx, UPRN_COL])
address = cell_str(df.at[idx, ADDRESS_COL])
postcode_raw = cell_str(df.at[idx, POSTCODE_COL])
postcode_clean = clean_postcode(postcode_raw)
# Already sorted (UPRN + address) or already filled by a prior run.
if given_uprn is not None and address:
skipped += 1
continue
if cell_str(df.at[idx, FOUND_UPRN_COL]) and cell_str(df.at[idx, FOUND_UPRN_COL]) != NOT_FOUND:
skipped += 1
continue
def mark_not_found(reason: str) -> None:
df.at[idx, FOUND_UPRN_COL] = NOT_FOUND if given_uprn is None else ""
df.at[idx, FOUND_ADDRESS_COL] = NOT_FOUND
df.at[idx, SOURCE_COL] = "not_found"
unresolved.append(
{
"Organisation Reference": ref,
"reason": reason,
"Address": address,
"Postcode": postcode_raw,
}
)
# Case B — UPRN present, address missing: reverse-lookup the address.
if given_uprn is not None and not address:
found: Optional[str] = None
if service is not None and postcode_clean:
try:
found = reverse_address_from_uprn(
given_uprn, postcode_clean, service, search_cache
)
except Exception as exc:
print(f" reverse failed {ref} {given_uprn}: {exc}")
if found:
df.at[idx, FOUND_ADDRESS_COL] = found
df.at[idx, SOURCE_COL] = "epc_reverse"
resolved_addr += 1
else:
mark_not_found("no address for UPRN")
continue
# Case A — no UPRN, has address: resolve a UPRN.
if given_uprn is None and address:
if not postcode_clean:
mark_not_found("no postcode")
continue
hit: Optional[Hit] = None
if token:
try:
hit = resolve_epc_relaxed(address, postcode_clean, epc_cache)
except Exception as exc:
print(f" EPC failed {ref} {postcode_clean}: {exc}")
if hit is None and os_api_key:
try:
hit = resolve_os_relaxed(address, postcode_clean, os_api_key, os_cache)
except Exception as exc:
print(f" OS failed {ref} {postcode_clean}: {exc}")
if hit is not None:
uprn, matched, score, source = hit
df.at[idx, FOUND_UPRN_COL] = uprn
df.at[idx, FOUND_ADDRESS_COL] = matched
df.at[idx, SCORE_COL] = round(score, 4)
df.at[idx, SOURCE_COL] = source
resolved_uprn += 1
else:
mark_not_found("no UPRN match")
if n % 100 == 0:
print(
f"[{n}/{total}] resolved={resolved_uprn} not_found={len(unresolved)}"
)
continue
# Case C — neither a UPRN nor an address.
mark_not_found("no UPRN and no address")
print(
f"\nResolved {resolved_uprn} UPRNs, {resolved_addr} addresses; "
f"{skipped} already sorted/done; {len(unresolved)} not found."
)
return unresolved
def _parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--in", dest="inp", type=Path, default=_DEFAULT_IN)
parser.add_argument("--out", type=Path, default=_DEFAULT_OUT)
parser.add_argument("--unresolved", type=Path, default=_DEFAULT_UNRESOLVED)
parser.add_argument("--limit", type=int, default=None, help="process first N rows")
return parser.parse_args()
def main() -> int:
args = _parse_args()
_epc_token, os_api_key = load_keys()
df = pd.read_excel(args.inp, sheet_name=SHEET)
if args.limit is not None:
df = df.head(args.limit).copy()
print(f"Loaded {len(df)} rows from {args.inp} [{SHEET}]")
unresolved = fill(df, os_api_key=os_api_key)
df.to_excel(args.out, sheet_name=SHEET, index=False)
print(f"Wrote filled sheet -> {args.out}")
if unresolved:
pd.DataFrame(unresolved).to_csv(args.unresolved, index=False)
print(f"Wrote {len(unresolved)} unresolved rows -> {args.unresolved}")
return 0
if __name__ == "__main__":
sys.exit(main())

331
scripts/finalise_to_property_table.py Normal file
View file

@ -0,0 +1,331 @@
"""Insert resolved manipulation_filled rows into the FE-owned ``property`` table.
Reuses the bulk_upload_finaliser's own row->PropertyIdentityInsert mapping
(``BulkUploadFinaliserOrchestrator._row_to_insert``) and the same
``PropertyPostgresRepository.insert_all`` the Lambda uses, so a row inserted here
is identical to one the real finaliser would write. The status-writer /
property_overrides path is skipped — this only populates ``property`` (no
BulkUpload task needed).
Insert is ON CONFLICT (portfolio_id, uprn) DO NOTHING, so re-running is safe.
# one random resolved row into portfolio 796, then read it back
python -m scripts.finalise_to_property_table --portfolio 796 --one
# a specific Organisation Reference
python -m scripts.finalise_to_property_table --portfolio 796 --ref 56100000101
# the whole sheet (resolved rows only by default; --include-unmatched to add
# null-UPRN rows too)
python -m scripts.finalise_to_property_table --portfolio 796 --all
Postgres target comes from the root .env (POSTGRES_*). Run from the worktree root.
"""
from __future__ import annotations
import argparse
import os
import sys
from pathlib import Path
from typing import Optional
import pandas as pd
from dotenv import load_dotenv
from sqlmodel import select
_REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(_REPO_ROOT)) # worktree root first — avoid the import trap
from infrastructure.postgres.config import PostgresConfig # noqa: E402
from infrastructure.postgres.engine import commit_scope, make_engine, make_session # noqa: E402
from infrastructure.postgres.property_table import PropertyRow # noqa: E402
from orchestration.bulk_upload_finaliser_orchestrator import ( # noqa: E402
BulkUploadFinaliserOrchestrator,
)
from repositories.property.property_postgres_repository import ( # noqa: E402
PropertyPostgresRepository,
)
from repositories.property.property_repository import PropertyIdentityInsert # noqa: E402
from scripts.fill_domna_addresses import ( # noqa: E402
ADDRESS_COL,
FOUND_ADDRESS_COL,
FOUND_UPRN_COL,
POSTCODE_COL,
REF_COL,
SCORE_COL,
SHEET,
UPRN_COL,
NOT_FOUND,
cell_str,
parse_uprn_cell,
)
_DEFAULT_IN = _REPO_ROOT / "scripts" / "manipulation_filled.xlsx"
def _final_uprn(row: pd.Series) -> Optional[int]:
"""The authoritative UPRN: the given one, else the DOMNA-found one."""
given = parse_uprn_cell(row.get(UPRN_COL))
if given is not None:
return given
found = cell_str(row.get(FOUND_UPRN_COL))
if found and found != NOT_FOUND:
return parse_uprn_cell(found)
return None
def to_combiner_row(row: pd.Series) -> dict[str, str]:
"""Map one spreadsheet row to the combiner-output shape the finaliser reads."""
given_uprn = parse_uprn_cell(row.get(UPRN_COL))
address = cell_str(row.get(ADDRESS_COL))
uprn = _final_uprn(row)
domna_addr = cell_str(row.get(FOUND_ADDRESS_COL))
if domna_addr == NOT_FOUND:
domna_addr = ""
# Matched address: the resolved one when we found it, else the given address
# (for rows that already had a UPRN + address).
matched = domna_addr or (address if given_uprn is not None else "")
score = cell_str(row.get(SCORE_COL))
return {
"Address 1": address,
"Address 2": "",
"Address 3": "",
"postcode": cell_str(row.get(POSTCODE_COL)),
"Internal Reference": cell_str(row.get(REF_COL)),
"address2uprn_uprn": "" if uprn is None else str(uprn),
"address2uprn_address": matched,
"address2uprn_lexiscore": score,
}
def load_rows(
path: Path, *, include_unmatched: bool
) -> tuple[pd.DataFrame, list[dict[str, str]]]:
"""Load the sheet and the combiner rows. By default drop rows with no UPRN."""
df = pd.read_excel(path, sheet_name=SHEET)
df = df.reset_index(drop=True)
if not include_unmatched:
keep = df.apply(lambda r: _final_uprn(r) is not None, axis=1)
df = df[keep].reset_index(drop=True)
rows = [to_combiner_row(r) for _, r in df.iterrows()]
return df, rows
def dedupe_by_uprn(
rows: list[dict[str, str]],
) -> tuple[list[dict[str, str]], list[dict[str, str]]]:
"""Keep the first row per UPRN; return (kept, dropped collisions).
The DB INSERT collapses duplicate (portfolio, uprn) via ON CONFLICT DO
NOTHING anyway, so this just makes the collision explicit (the dropped rows
are written out for review) rather than letting an arbitrary ref win silently.
"""
seen: set[str] = set()
kept: list[dict[str, str]] = []
dropped: list[dict[str, str]] = []
for row in rows:
uprn = row["address2uprn_uprn"]
if uprn in seen:
dropped.append(row)
else:
seen.add(uprn)
kept.append(row)
return kept, dropped
# Force-reload teardown order (bottom-up). property_overrides is ON DELETE
# CASCADE so it clears itself when the property goes; everything below is NO
# ACTION and must be deleted first, deepest child first.
# property -> epc_property -> {these children}
_EPC_CHILD_TABLES = (
"epc_energy_element",
"epc_window",
"epc_main_heating_detail",
"epc_renewable_heat_incentive",
"epc_building_part",
"epc_flat_details",
)
# property -> {these direct dependents}, deleted after the epc children
_PROPERTY_DEPENDENTS = ("epc_property", "plan")
_INSERT_CHUNK = 4000 # 9 cols/row -> well under psycopg2's 65535-param limit
def _reset_portfolio(session: object, portfolio_id: int) -> int:
"""Delete a portfolio's properties and their NO ACTION dependency tree.
Returns the number of property rows deleted (property_overrides cascade).
"""
from sqlalchemy import text
pids = "SELECT id FROM property WHERE portfolio_id = :pid"
epc_ids = f"SELECT id FROM epc_property WHERE property_id IN ({pids})"
for table in _EPC_CHILD_TABLES:
session.execute( # type: ignore[attr-defined]
text(f"DELETE FROM {table} WHERE epc_property_id IN ({epc_ids})"),
{"pid": portfolio_id},
)
for table in _PROPERTY_DEPENDENTS:
session.execute( # type: ignore[attr-defined]
text(f"DELETE FROM {table} WHERE property_id IN ({pids})"),
{"pid": portfolio_id},
)
result = session.execute( # type: ignore[attr-defined]
text("DELETE FROM property WHERE portfolio_id = :pid"), {"pid": portfolio_id}
)
return result.rowcount
def clean_reload(
rows: list[dict[str, str]], portfolio_id: int, *, reset: bool
) -> tuple[int, int]:
"""Optionally wipe the portfolio, then chunk-insert rows. One transaction.
Returns (properties_deleted, properties_inserted).
"""
inserts: list[PropertyIdentityInsert] = [
BulkUploadFinaliserOrchestrator._row_to_insert(r, portfolio_id) for r in rows
]
engine = _engine()
session = make_session(engine)
deleted = 0
inserted = 0
try:
repo = PropertyPostgresRepository(session)
with commit_scope(session):
if reset:
deleted = _reset_portfolio(session, portfolio_id)
for start in range(0, len(inserts), _INSERT_CHUNK):
inserted += repo.insert_all(inserts[start : start + _INSERT_CHUNK])
finally:
session.close()
return deleted, inserted
def _engine():
load_dotenv(_REPO_ROOT / ".env")
return make_engine(PostgresConfig.from_env(os.environ))
def insert_rows(rows: list[dict[str, str]], portfolio_id: int) -> int:
"""Insert via the finaliser's mapper + repository. Returns rows inserted."""
inserts: list[PropertyIdentityInsert] = [
BulkUploadFinaliserOrchestrator._row_to_insert(r, portfolio_id) for r in rows
]
engine = _engine()
session = make_session(engine)
try:
repo = PropertyPostgresRepository(session)
with commit_scope(session):
inserted = repo.insert_all(inserts)
finally:
session.close()
return inserted
def fetch_by_ref(portfolio_id: int, ref: str) -> list[PropertyRow]:
"""Read back inserted rows for one Organisation Reference (for verification)."""
engine = _engine()
session = make_session(engine)
try:
stmt = select(PropertyRow).where(
PropertyRow.portfolio_id == portfolio_id,
PropertyRow.landlord_property_id == ref,
)
return list(session.exec(stmt).all())
finally:
session.close()
def _show(row: dict[str, str], insert: PropertyIdentityInsert) -> None:
print("\nSource (combiner) row:")
for k, v in row.items():
print(f" {k}: {v!r}")
print("\nMapped PropertyIdentityInsert:")
for k, v in insert.__dict__.items():
print(f" {k}: {v!r}")
def _parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--in", dest="inp", type=Path, default=_DEFAULT_IN)
parser.add_argument("--portfolio", type=int, required=True)
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument("--one", action="store_true", help="one random resolved row")
group.add_argument("--ref", help="a specific Organisation Reference")
group.add_argument("--all", action="store_true", help="every row")
parser.add_argument(
"--include-unmatched",
action="store_true",
help="also insert rows with no UPRN (null-UPRN property rows)",
)
parser.add_argument(
"--reset",
action="store_true",
help="(with --all) DELETE all properties in the portfolio first "
"(cascades property_overrides; clears plan/epc_property)",
)
parser.add_argument(
"--collisions",
type=Path,
default=_REPO_ROOT / "scripts" / "manipulation_collisions.csv",
help="where to write rows dropped as duplicate-UPRN collisions",
)
parser.add_argument("--seed", type=int, default=0, help="random seed for --one")
return parser.parse_args()
def main() -> int:
args = _parse_args()
df, rows = load_rows(args.inp, include_unmatched=args.include_unmatched)
print(f"Loaded {len(rows)} candidate rows from {args.inp}")
if args.all:
kept, dropped = dedupe_by_uprn(rows)
if dropped:
pd.DataFrame(dropped).to_csv(args.collisions, index=False)
print(
f"{len(dropped)} duplicate-UPRN rows dropped -> {args.collisions} "
f"({len(kept)} unique to insert)"
)
deleted, inserted = clean_reload(kept, args.portfolio, reset=args.reset)
if args.reset:
print(f"Deleted {deleted} existing properties in portfolio {args.portfolio}.")
print(f"Inserted {inserted} properties into portfolio {args.portfolio}.")
return 0
# Single-row paths: pick the row, show the mapping, insert, read back.
if args.ref:
match = [r for r in rows if r["Internal Reference"] == args.ref]
if not match:
print(f"No resolved row with Organisation Reference {args.ref!r}.")
return 1
row = match[0]
else: # --one: deterministic "random" pick via seed
idx = (args.seed * 7919) % len(rows)
row = rows[idx]
ref = row["Internal Reference"]
insert = BulkUploadFinaliserOrchestrator._row_to_insert(row, args.portfolio)
_show(row, insert)
inserted = insert_rows([row], args.portfolio)
print(
f"\ninsert_all -> {inserted} new row(s) "
f"(0 means it already existed; ON CONFLICT DO NOTHING)."
)
print(f"\nproperty rows for portfolio {args.portfolio}, ref {ref!r}:")
for pr in fetch_by_ref(args.portfolio, ref):
print(
f" id={pr.id} uprn={pr.uprn} address={pr.address!r} "
f"postcode={pr.postcode!r} status={pr.creation_status} "
f"lexiscore={pr.lexiscore}"
)
return 0
if __name__ == "__main__":
sys.exit(main())

44
scripts/hyde/RESUME_AFTER_KHALIM.md Normal file
View file

@ -0,0 +1,44 @@
# Resume prompt — finish the Hyde portfolio-796 property_overrides run (after Khalim review)
Paste the block below to continue. It tells the assistant to review the unknown-override
decisions with me, verify them, confirm before writing, then run the remaining steps.
---
We paused the Hyde property-overrides bulk load to review the UNKNOWN classifications with
Khalim. Pick it back up.
**Context (already done):**
- Target is **portfolio 796** in DevAssessmentModelDB (NOT 795 — 795 is empty).
- Script: `scripts/hyde/build_property_overrides.py`. Pass 1 (`classify`) is DONE — the
`landlord_*_overrides` ledger is populated; re-running classify is free (cache hits).
- The 19 unresolved descriptions are documented in `scripts/hyde/unknowns_review.md`, with
proposed values already written to `overrides_edits.csv` (gitignored).
- Env (DB creds + `OPENAI_API_KEY`) is in `/workspaces/home/github/Model/.env`; load it with
python-dotenv and set `POSTGRES_DRIVER=psycopg2`. Writes are idempotent upserts (unique on
`property_id, override_component, building_part`) — safe to re-run, never duplicates.
**Do this, in order:**
1. **Ask me what Khalim decided** for the unknowns. The one real judgement call is the
flat-roof reading: `Flat: As Built` (1,172 rows) + `Flat: Unknown` (194) → which of
`Flat, no insulation (assumed)` / `Flat, insulated (assumed)` / `Flat, limited insulation
(assumed)`. The `construction_age_band` bands (29,829 rows) are deterministic (band = first
letter) — keep as-is unless I say otherwise. Confirm the other roof/wall proposals too.
2. **Update `overrides_edits.csv`** (`corrected_value` column) to match Khalim's decisions.
3. Run `validate --edits overrides_edits.csv` and fix anything it rejects.
4. **Show me the final edits + the planned write counts, and WAIT for my explicit go-ahead
before any `--apply`.** Do not write to the DB before I confirm.
5. On my go-ahead:
- `apply-edits --edits overrides_edits.csv --portfolio-id 796 --apply` (user corrections → ledger)
- `write --excel scripts/hyde/hyde_property_overrides.xlsx --portfolio-id 796` (DRY RUN —
report unmatched org_refs + unresolved across all 31,773 first)
- then the same `write ... --apply`
6. `verify --portfolio-id 796 --org-ref <a few org_refs>` to confirm property_overrides +
overlays landed.
7. Remind me about the deferred **age-classifier prompt-hint fix** for the production lambda
(the live frontend will hit the same `"D: 1950-1966"` → UNKNOWN until that lands).
Every DB command loads env from `/workspaces/home/github/Model/.env`. Read-only checks
(`verify`, dry-run `write`) are fine to run unprompted; anything `--apply` needs my confirm.
---

479
scripts/hyde/build_property_overrides.py Normal file
View file

@ -0,0 +1,479 @@
"""Build ``property_overrides`` for a portfolio from the Hyde Excel, bypassing the
frontend + lambdas, using the ``landlord_*_overrides`` tables as the durable
classification ledger.
Why the ledger (not a throwaway cache): ``landlord_*_overrides`` stores
``(portfolio_id, description) -> value`` with a ``source`` (classifier|user).
* Re-runs classify only descriptions NOT already stored -> saves ChatGPT calls.
* Human corrections are stored as ``source=user`` and the classifier is
forbidden from overwriting them (ADR-0003) -> edits are permanent.
Then we resolve the vocab + match each row to a ``property.id`` by **org_ref**
(Excel "Organisation Reference" -> property.landlord_property_id) and upsert
``property_overrides`` (the fact layer the SAP overlay reads).
Subcommands:
list-values print each component's valid override values (reference)
classify --excel f --portfolio-id 795
PASS 1: classify cache-misses via ChatGPT,
upsert to landlord tables, write
overrides_unknowns.csv (with allowed_values)
validate --edits overrides_edits.csv
check a hand-edited file: every corrected_value
must be a valid enum value (suggests fixes)
apply-edits --edits overrides_edits.csv --portfolio-id 795 [--apply]
upsert validated corrections as source=user
write --excel f --portfolio-id 795 [--apply]
PASS 2: build + upsert property_overrides from vocab
Env: POSTGRES_* (PostgresConfig.from_env) and OPENAI_API_KEY (ChatGPT).
"""
from __future__ import annotations
import argparse
import csv
import difflib
import logging
import os
from collections import Counter
from dataclasses import dataclass
from datetime import datetime, timezone
from enum import Enum
from typing import Any, Optional
import pandas as pd # pyright: ignore[reportMissingTypeStubs]
from sqlalchemy import Table, text
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlmodel import SQLModel
from domain.epc.property_overrides.built_form_type import BuiltFormType
from domain.epc.property_overrides.construction_age_band import ConstructionAgeBand
from domain.epc.property_overrides.glazing_type import GlazingType
from domain.epc.property_overrides.main_fuel_type import MainFuelType
from domain.epc.property_overrides.main_heating_system_type import MainHeatingSystemType
from domain.epc.property_overrides.property_type import PropertyType
from domain.epc.property_overrides.roof_type import RoofType
from domain.epc.property_overrides.wall_type import WallType
from domain.epc.property_overrides.wall_type_construction_dates import (
wall_type_construction_date_prompt_hint,
)
from domain.epc.property_overrides.water_heating_type import WaterHeatingType
from infrastructure.chatgpt.chatgpt import ChatGPT
from infrastructure.chatgpt.chatgpt_column_classifier import ChatGptColumnClassifier
from infrastructure.landlord_overrides.landlord_override_reader_postgres_repository import (
LandlordOverrideReaderPostgresRepository,
)
from infrastructure.landlord_overrides.landlord_overrides_postgres_repository import (
LandlordOverridesRepository,
)
from infrastructure.postgres.config import PostgresConfig
from infrastructure.postgres.engine import commit_scope, make_engine, make_session
from infrastructure.postgres.landlord_built_form_type_override_table import (
LandlordBuiltFormTypeOverrideRow,
)
from infrastructure.postgres.landlord_construction_age_band_override_table import (
LandlordConstructionAgeBandOverrideRow,
)
from infrastructure.postgres.landlord_glazing_override_table import (
LandlordGlazingOverrideRow,
)
from infrastructure.postgres.landlord_main_fuel_override_table import (
LandlordMainFuelOverrideRow,
)
from infrastructure.postgres.landlord_main_heating_system_override_table import (
LandlordMainHeatingSystemOverrideRow,
)
from infrastructure.postgres.landlord_override_enums import OverrideSource
from infrastructure.postgres.landlord_property_type_override_table import (
LandlordPropertyTypeOverrideRow,
)
from infrastructure.postgres.landlord_roof_type_override_table import (
LandlordRoofTypeOverrideRow,
)
from infrastructure.postgres.landlord_wall_type_override_table import (
LandlordWallTypeOverrideRow,
)
from infrastructure.postgres.landlord_water_heating_override_table import (
LandlordWaterHeatingOverrideRow,
)
from repositories.property.landlord_override_overlays import overlays_from
from repositories.property.property_override_postgres_repository import (
PropertyOverridePostgresRepository,
)
from repositories.property.property_override_repository import PropertyOverrideInsert
from repositories.property.property_overrides_postgres_reader import (
PropertyOverridesPostgresReader,
)
logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = logging.getLogger("build_property_overrides")
ORG_REF_COLUMN = "Organisation Reference"
UNKNOWNS_PATH = "overrides_unknowns.csv"
# A "party ceiling" roof (another/same dwelling or premises above) means the
# dwelling has no exposed roof, which is only physically valid for a Flat or
# Maisonette. When the property type says House/Bungalow the two landlord fields
# contradict (a house has nothing above it). Per Khalim these may be houses split
# into flats — leave them entirely as-is for joint review, so we SKIP the whole
# property (write NO overrides for it) and record the org_refs for that review
# (ADR-0033; surfaced by scripts/hyde/audit_hyde_rows.py).
_PARTY_CEILING_ROOF_VALUES: frozenset[str] = frozenset({
"(another dwelling above)", "(same dwelling above)",
"(other premises above)", "(another premises above)",
"Another Premises Above",
})
_FLAT_PTYPE_PREFIXES: tuple[str, ...] = ("flat", "maisonette")
SKIPPED_PATH = "skipped_contradictory_properties.csv"
@dataclass(frozen=True)
class ComponentSpec:
component: str
enum_cls: type[Enum]
unknown: Enum
row_type: type[SQLModel]
excel_header: str
per_building_part: bool # comma = building parts (wall/roof/age) vs whole-dwelling
extra_instructions: Optional[str] = None
def allowed_values(self) -> list[str]:
"""Valid override values a human may pick (excludes UNKNOWN)."""
return sorted(m.value for m in self.enum_cls if m is not self.unknown)
def _component_specs() -> list[ComponentSpec]:
return [
ComponentSpec("property_type", PropertyType, PropertyType.UNKNOWN, LandlordPropertyTypeOverrideRow, "Property Type", False),
ComponentSpec("built_form_type", BuiltFormType, BuiltFormType.UNKNOWN, LandlordBuiltFormTypeOverrideRow, "Property Type", False),
ComponentSpec("wall_type", WallType, WallType.UNKNOWN, LandlordWallTypeOverrideRow, "Walls", True, wall_type_construction_date_prompt_hint()),
ComponentSpec("roof_type", RoofType, RoofType.UNKNOWN, LandlordRoofTypeOverrideRow, "Roofs", True),
ComponentSpec("construction_age_band", ConstructionAgeBand, ConstructionAgeBand.UNKNOWN, LandlordConstructionAgeBandOverrideRow, "Age", True),
ComponentSpec("main_fuel", MainFuelType, MainFuelType.UNKNOWN, LandlordMainFuelOverrideRow, "Main Fuel", False),
ComponentSpec("glazing", GlazingType, GlazingType.UNKNOWN, LandlordGlazingOverrideRow, "Glazing", False),
ComponentSpec("water_heating", WaterHeatingType, WaterHeatingType.UNKNOWN, LandlordWaterHeatingOverrideRow, "Hot Water", False),
ComponentSpec("main_heating_system", MainHeatingSystemType, MainHeatingSystemType.UNKNOWN, LandlordMainHeatingSystemOverrideRow, "Heating", False),
]
def _specs_by_component() -> dict[str, ComponentSpec]:
return {s.component: s for s in _component_specs()}
def _norm(s: Any) -> str:
"""Vocab key normalisation — mirrors the orchestrator (strip + lower)."""
return str(s or "").strip().lower()
def _split_entries(cell: Any, per_building_part: bool) -> list[str]:
raw = "" if cell is None else str(cell)
if not raw.strip():
return []
if not per_building_part:
return [raw.strip()]
return [part.strip() for part in raw.split(",") if part.strip()]
def _load_rows(excel: str, sheet: str) -> list[dict[str, Any]]:
return pd.read_excel(excel, sheet_name=sheet).to_dict(orient="records") # type: ignore[return-value]
def _filter_rows(rows: list[dict[str, Any]], org_ref: Optional[str],
limit: Optional[int]) -> list[dict[str, Any]]:
"""Narrow to one property (--org-ref) or the first N rows (--limit) for a
cheap smoke test before the full run."""
if org_ref:
rows = [r for r in rows if str(r.get(ORG_REF_COLUMN, "")).strip() == org_ref]
if limit:
rows = rows[:limit]
return rows
def _distinct_entries(rows: list[dict[str, Any]], spec: ComponentSpec) -> Counter[str]:
counts: Counter[str] = Counter()
for row in rows:
for entry in _split_entries(row.get(spec.excel_header), spec.per_building_part):
counts[entry] += 1
return counts
# --------------------------------------------------------------------------- #
def list_values(_: argparse.Namespace) -> None:
"""Print the valid override values per component (the reference for edits)."""
for spec in _component_specs():
print(f"\n## {spec.component} (Excel: {spec.excel_header})")
for v in spec.allowed_values():
print(f" {v}")
def validate(args: argparse.Namespace) -> None:
"""Check a hand-edited CSV: every corrected_value must be a valid enum value."""
specs = _specs_by_component()
bad = 0
with open(args.edits, newline="") as f:
for i, r in enumerate(csv.DictReader(f), start=2):
val = (r.get("corrected_value") or "").strip()
if not val:
continue
comp = (r.get("component") or "").strip()
spec = specs.get(comp)
if spec is None:
logger.error("row %d: unknown component %r", i, comp)
bad += 1
continue
if val not in spec.allowed_values():
hint = difflib.get_close_matches(val, spec.allowed_values(), n=2)
logger.error("row %d [%s]: %r is not a valid value.%s",
i, comp, val,
f" Did you mean: {hint}?" if hint else
" Run 'list-values' for the allowed set.")
bad += 1
if bad:
raise SystemExit(f"{bad} invalid corrected_value(s) — fix them before apply-edits.")
logger.info("All corrected values are valid enum values. ✓")
def _db_session() -> Any:
return make_session(make_engine(PostgresConfig.from_env(os.environ)))
def classify(args: argparse.Namespace) -> None:
rows = _filter_rows(_load_rows(args.excel, args.sheet), args.org_ref, args.limit)
logger.info("Classifying over %d row(s).", len(rows))
chat_gpt = ChatGPT()
session = _db_session()
reader = LandlordOverrideReaderPostgresRepository(session)
try:
vocab = reader.load_for_portfolio(args.portfolio_id) # {component: {desc: value}}
unknown_rows: list[tuple[str, str, int, str]] = []
for spec in _component_specs():
counts = _distinct_entries(rows, spec)
known = vocab.get(spec.component, {}) # already-classified (cache)
to_classify = {d for d in counts if _norm(d) not in known}
logger.info("%-22s %4d distinct | %4d cached | %4d to classify",
spec.component, len(counts), len(counts) - len(to_classify), len(to_classify))
resolved: dict[str, Enum] = {}
if to_classify:
classifier: ChatGptColumnClassifier[Any] = ChatGptColumnClassifier(
chat_gpt, spec.enum_cls, spec.unknown, extra_instructions=spec.extra_instructions)
resolved = classifier.classify(to_classify)
repo: LandlordOverridesRepository[Any] = LandlordOverridesRepository(session, spec.row_type)
with commit_scope(session):
# store keyed on the normalised description (matches the reader/finaliser lookup)
repo.upsert_all(args.portfolio_id, {_norm(d): m for d, m in resolved.items()})
# collect UNKNOWNs (freshly classified + anything cached as UNKNOWN) for review
unk = spec.unknown.value
for desc, n in counts.items():
v = resolved.get(desc).value if desc in resolved and resolved[desc] else known.get(_norm(desc)) # type: ignore[union-attr]
if v is None or v == unk:
allowed = " | ".join(spec.allowed_values())
unknown_rows.append((spec.component, desc, n, allowed))
with open(UNKNOWNS_PATH, "w", newline="") as f:
w = csv.writer(f)
w.writerow(["component", "description", "count", "corrected_value", "allowed_values"])
for comp, desc, n, allowed in sorted(unknown_rows, key=lambda r: (-r[2])):
w.writerow([comp, desc, n, "", allowed])
logger.info("\nWrote %s — fill 'corrected_value' (must match 'allowed_values'), "
"then: validate -> apply-edits -> write.", UNKNOWNS_PATH)
finally:
session.close()
def _upsert_user_corrections(session: Any, portfolio_id: int,
by_component: dict[str, dict[str, str]]) -> int:
"""Upsert validated human corrections as source=user (always wins on conflict)."""
specs = _specs_by_component()
n = 0
now = datetime.now(timezone.utc)
for comp, mapping in by_component.items():
spec = specs[comp]
table: Table = getattr(spec.row_type, "__table__")
rows = [{"portfolio_id": portfolio_id, "description": _norm(d), "value": v,
"source": OverrideSource.USER, "created_at": now, "updated_at": now}
for d, v in mapping.items()]
if not rows:
continue
stmt = pg_insert(table).values(rows)
stmt = stmt.on_conflict_do_update(
index_elements=["portfolio_id", "description"],
set_={"value": stmt.excluded.value, "source": stmt.excluded.source,
"updated_at": stmt.excluded.updated_at})
session.execute(stmt)
n += len(rows)
return n
def apply_edits(args: argparse.Namespace) -> None:
validate(args) # fail before touching the DB
specs = _specs_by_component()
by_component: dict[str, dict[str, str]] = {}
with open(args.edits, newline="") as f:
for r in csv.DictReader(f):
val = (r.get("corrected_value") or "").strip()
if val and r["component"] in specs:
by_component.setdefault(r["component"], {})[r["description"]] = val
session = _db_session()
try:
if not args.apply:
total = sum(len(m) for m in by_component.values())
logger.info("DRY RUN — %d user corrections ready. Re-run with --apply.", total)
return
with commit_scope(session):
n = _upsert_user_corrections(session, args.portfolio_id, by_component)
logger.info("Upserted %d user corrections (source=user).", n)
finally:
session.close()
def _org_ref_to_property_id(session: Any, portfolio_id: int) -> dict[str, int]:
stmt = text("SELECT landlord_property_id, id FROM property "
"WHERE portfolio_id = :pid AND landlord_property_id IS NOT NULL")
return {str(ref).strip(): int(pid) for ref, pid in session.execute(stmt, {"pid": portfolio_id})}
def write(args: argparse.Namespace) -> None:
rows = _filter_rows(_load_rows(args.excel, args.sheet), args.org_ref, args.limit)
logger.info("Writing over %d row(s).", len(rows))
session = _db_session()
reader = LandlordOverrideReaderPostgresRepository(session)
try:
vocab = reader.load_for_portfolio(args.portfolio_id)
org_ref_map = _org_ref_to_property_id(session, args.portfolio_id)
logger.info("Portfolio %d: %d properties with org_ref.", args.portfolio_id, len(org_ref_map))
ptype_header = _specs_by_component()["property_type"].excel_header
roof_header = _specs_by_component()["roof_type"].excel_header
roof_vocab = vocab.get("roof_type", {})
inserts: list[PropertyOverrideInsert] = []
unmatched: Counter[str] = Counter()
unresolved: Counter[str] = Counter()
skipped: list[tuple[str, str, str]] = [] # (org_ref, property_type, roof)
for row in rows:
org_ref = str(row.get(ORG_REF_COLUMN, "")).strip()
property_id = org_ref_map.get(org_ref)
if property_id is None:
unmatched[org_ref] += 1
continue
ptype_raw = str(row.get(ptype_header) or "").strip()
roof_raw = str(row.get(roof_header) or "").strip()
row_is_flat = ptype_raw.lower().startswith(_FLAT_PTYPE_PREFIXES)
roof_resolved = {
roof_vocab.get(_norm(e))
for e in _split_entries(row.get(roof_header), True)
}
if not row_is_flat and roof_resolved & _PARTY_CEILING_ROOF_VALUES:
# Party-ceiling roof on a non-flat dwelling — contradictory source
# data (maybe a house split into flats). Leave the property fully
# as-is for joint review: write NO overrides, record it (see above).
skipped.append((org_ref, ptype_raw, roof_raw))
continue
for spec in _component_specs():
comp_vocab = vocab.get(spec.component, {})
for building_part, entry in enumerate(
_split_entries(row.get(spec.excel_header), spec.per_building_part)):
value = comp_vocab.get(_norm(entry))
if not value or value == spec.unknown.value:
unresolved[f"{spec.component}: {entry}"] += 1
continue
inserts.append(PropertyOverrideInsert(
property_id=property_id, portfolio_id=args.portfolio_id,
building_part=building_part, override_component=spec.component,
override_value=value, original_spreadsheet_description=entry))
logger.info("Built %d rows | %d unmatched org_refs | %d unresolved | "
"%d contradictory properties skipped (left as-is)",
len(inserts), sum(unmatched.values()), sum(unresolved.values()),
len(skipped))
if skipped:
out = os.path.join(os.path.dirname(args.excel) or ".", SKIPPED_PATH)
with open(out, "w", newline="") as f:
w = csv.writer(f)
w.writerow(["org_ref", "property_type", "roof"])
w.writerows(sorted(skipped))
logger.info("Recorded %d skipped properties -> %s (for joint review)",
len(skipped), out)
if unresolved:
logger.info("Top unresolved (need apply-edits): %s", unresolved.most_common(10))
if not args.apply:
logger.info("DRY RUN — not writing. Re-run with --apply.")
for ins in inserts[:10]:
logger.info(" %s", ins)
return
with commit_scope(session):
affected = PropertyOverridePostgresRepository(session).upsert_all(inserts)
logger.info("Upserted %d property_overrides.", affected)
finally:
session.close()
def verify(args: argparse.Namespace) -> None:
"""For one property (by org_ref): show the persisted property_overrides rows
and the EpcSimulation overlays they produce — the end-to-end proof that the
chain reaches the SAP overlay surface."""
session = _db_session()
try:
org_ref_map = _org_ref_to_property_id(session, args.portfolio_id)
property_id = org_ref_map.get(args.org_ref)
if property_id is None:
raise SystemExit(f"org_ref {args.org_ref!r} not found in portfolio {args.portfolio_id}.")
reader = PropertyOverridesPostgresReader(lambda: session)
resolved = reader.overrides_for(property_id)
logger.info("property_id %d — %d property_overrides rows:", property_id, len(resolved.rows))
for r in resolved.rows:
logger.info(" part %d | %-22s = %s", r.building_part, r.override_component, r.override_value)
overlays = overlays_from(resolved)
logger.info("\n-> %d EpcSimulation overlay(s) produced (what the SAP calc applies):", len(overlays))
for o in overlays:
logger.info(" %s", o)
finally:
session.close()
def main() -> None:
p = argparse.ArgumentParser(description=__doc__)
sub = p.add_subparsers(dest="cmd", required=True)
sub.add_parser("list-values").set_defaults(func=list_values)
v = sub.add_parser("validate")
v.add_argument("--edits", required=True)
v.set_defaults(func=validate)
c = sub.add_parser("classify")
c.add_argument("--excel", required=True)
c.add_argument("--sheet", default="AddressProfilingResults")
c.add_argument("--portfolio-id", type=int, required=True)
c.add_argument("--org-ref", default=None, help="smoke test: only this property's org_ref")
c.add_argument("--limit", type=int, default=None, help="smoke test: first N rows")
c.set_defaults(func=classify)
a = sub.add_parser("apply-edits")
a.add_argument("--edits", required=True)
a.add_argument("--portfolio-id", type=int, required=True)
a.add_argument("--apply", action="store_true")
a.set_defaults(func=apply_edits)
w = sub.add_parser("write")
w.add_argument("--excel", required=True)
w.add_argument("--sheet", default="AddressProfilingResults")
w.add_argument("--portfolio-id", type=int, required=True)
w.add_argument("--org-ref", default=None, help="smoke test: only this property's org_ref")
w.add_argument("--limit", type=int, default=None, help="smoke test: first N rows")
w.add_argument("--apply", action="store_true")
w.set_defaults(func=write)
vf = sub.add_parser("verify")
vf.add_argument("--portfolio-id", type=int, required=True)
vf.add_argument("--org-ref", required=True)
vf.set_defaults(func=verify)
args = p.parse_args()
args.func(args)
if __name__ == "__main__":
main()

6
scripts/hyde/elmhurst_dom/conservatories.json
View file

@ -87,12 +87,6 @@
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionConservatories_Link"
},
{
"text": "Flats",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionFlats$Link\", \"\", true, \"\", \"\", false, true))",
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionFlats_Link"
},
{
"text": "Walls",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionWalls$Link\", \"\", true, \"\", \"\", false, true))",

54
scripts/hyde/elmhurst_dom/dimensions.json
View file

@ -9,7 +9,7 @@
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$RadioButtonDimensionsType",
"label": "Type",
"value": "Internal",
"disabled": true,
"disabled": false,
"autopostback": false,
"options": [
{
@ -29,13 +29,53 @@
}
]
},
{
"tag": "input",
"type": "text",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMain_WebUserControlDimensionsMain_TextBoxFloorArea1stFloor",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMain$WebUserControlDimensionsMain$TextBoxFloorArea1stFloor",
"label": "",
"value": "31.06",
"disabled": false,
"autopostback": true
},
{
"tag": "input",
"type": "text",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMain_WebUserControlDimensionsMain_TextBoxRoomHeight1stFloor",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMain$WebUserControlDimensionsMain$TextBoxRoomHeight1stFloor",
"label": "",
"value": "2.65",
"disabled": false,
"autopostback": true
},
{
"tag": "input",
"type": "text",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMain_WebUserControlDimensionsMain_TextBoxWallPerimeter1stFloor",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMain$WebUserControlDimensionsMain$TextBoxWallPerimeter1stFloor",
"label": "",
"value": "15.80",
"disabled": false,
"autopostback": true
},
{
"tag": "input",
"type": "text",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMain_WebUserControlDimensionsMain_TextBoxPartyWallLength1stFloor",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMain$WebUserControlDimensionsMain$TextBoxPartyWallLength1stFloor",
"label": "",
"value": "0.00",
"disabled": false,
"autopostback": true
},
{
"tag": "input",
"type": "text",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMain_WebUserControlDimensionsMain_TextBoxFloorAreaLowestFloor",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMain$WebUserControlDimensionsMain$TextBoxFloorAreaLowestFloor",
"label": "",
"value": "59.50",
"value": "31.06",
"disabled": false,
"autopostback": true
},
@ -55,7 +95,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMain_WebUserControlDimensionsMain_TextBoxWallPerimeterLowestFloor",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMain$WebUserControlDimensionsMain$TextBoxWallPerimeterLowestFloor",
"label": "",
"value": "13.86",
"value": "15.80",
"disabled": false,
"autopostback": true
},
@ -65,7 +105,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMain_WebUserControlDimensionsMain_TextBoxPartyWallLengthLowestFloor",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMain$WebUserControlDimensionsMain$TextBoxPartyWallLengthLowestFloor",
"label": "",
"value": "13.86",
"value": "0.00",
"disabled": false,
"autopostback": true
},
@ -154,12 +194,6 @@
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionConservatories_Link"
},
{
"text": "Flats",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionFlats$Link\", \"\", true, \"\", \"\", false, true))",
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionFlats_Link"
},
{
"text": "Walls",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionWalls$Link\", \"\", true, \"\", \"\", false, true))",

6
scripts/hyde/elmhurst_dom/floors.json
View file

@ -198,12 +198,6 @@
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionConservatories_Link"
},
{
"text": "Flats",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionFlats$Link\", \"\", true, \"\", \"\", false, true))",
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionFlats_Link"
},
{
"text": "Walls",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionWalls$Link\", \"\", true, \"\", \"\", false, true))",

6
scripts/hyde/elmhurst_dom/newtechnologies.json
View file

@ -184,12 +184,6 @@
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionConservatories_Link"
},
{
"text": "Flats",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionFlats$Link\", \"\", true, \"\", \"\", false, true))",
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionFlats_Link"
},
{
"text": "Walls",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionWalls$Link\", \"\", true, \"\", \"\", false, true))",

172
scripts/hyde/elmhurst_dom/openings.json
View file

@ -22,73 +22,13 @@
"disabled": false,
"autopostback": false
},
{
"tag": "input",
"type": "image",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_GridViewExtendedWidows_DeleteButton_1",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$GridViewExtendedWidows$ctl03$DeleteButton",
"label": "2",
"value": "",
"disabled": false,
"autopostback": false
},
{
"tag": "input",
"type": "image",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_GridViewExtendedWidows_CopyButton_1",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$GridViewExtendedWidows$ctl03$CopyButton",
"label": "2",
"value": "",
"disabled": false,
"autopostback": false
},
{
"tag": "input",
"type": "image",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_GridViewExtendedWidows_DeleteButton_2",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$GridViewExtendedWidows$ctl04$DeleteButton",
"label": "3",
"value": "",
"disabled": false,
"autopostback": false
},
{
"tag": "input",
"type": "image",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_GridViewExtendedWidows_CopyButton_2",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$GridViewExtendedWidows$ctl04$CopyButton",
"label": "3",
"value": "",
"disabled": false,
"autopostback": false
},
{
"tag": "input",
"type": "image",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_GridViewExtendedWidows_DeleteButton_3",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$GridViewExtendedWidows$ctl05$DeleteButton",
"label": "4",
"value": "",
"disabled": false,
"autopostback": false
},
{
"tag": "input",
"type": "image",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_GridViewExtendedWidows_CopyButton_3",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$GridViewExtendedWidows$ctl05$CopyButton",
"label": "4",
"value": "",
"disabled": false,
"autopostback": false
},
{
"tag": "input",
"type": "text",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_TextBoxExtWidth",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$TextBoxExtWidth",
"label": "",
"value": "1.44",
"value": "0.00",
"disabled": false,
"autopostback": true
},
@ -98,7 +38,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_TextBoxExtHeight",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$TextBoxExtHeight",
"label": "",
"value": "1.00",
"value": "0.00",
"disabled": false,
"autopostback": true
},
@ -108,7 +48,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_TextBoxExtArea",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$TextBoxExtArea",
"label": "",
"value": "1.44",
"value": "0.00",
"disabled": true,
"autopostback": false
},
@ -140,14 +80,14 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_DropDownListExtGlazing",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$DropDownListExtGlazing",
"label": "",
"value": "Double with unknown install date",
"value": "",
"disabled": false,
"autopostback": true,
"options": [
{
"value": "",
"text": "",
"selected": false
"selected": true
},
{
"value": "Single glazing",
@ -172,7 +112,7 @@
{
"value": "Double with unknown install date",
"text": "Double with unknown install date",
"selected": true
"selected": false
},
{
"value": "Secondary glazing",
@ -216,89 +156,25 @@
}
]
},
{
"tag": "select",
"type": "select-one",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_DropDownListExtFrameType",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$DropDownListExtFrameType",
"label": "",
"value": "PVC",
"disabled": false,
"autopostback": true,
"options": [
{
"value": "",
"text": "",
"selected": false
},
{
"value": "PVC",
"text": "PVC",
"selected": true
},
{
"value": "Wood",
"text": "Wood",
"selected": false
},
{
"value": "Metal",
"text": "Metal",
"selected": false
}
]
},
{
"tag": "select",
"type": "select-one",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_DropDownListExtGlazingGap",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$DropDownListExtGlazingGap",
"label": "",
"value": "12 mm",
"disabled": false,
"autopostback": true,
"options": [
{
"value": "",
"text": "",
"selected": false
},
{
"value": "6 mm",
"text": "6 mm",
"selected": false
},
{
"value": "12 mm",
"text": "12 mm",
"selected": true
},
{
"value": "16 mm or more",
"text": "16 mm or more",
"selected": false
}
]
},
{
"tag": "select",
"type": "select-one",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_DropDownListExtBuildingPartId",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$DropDownListExtBuildingPartId",
"label": "",
"value": "Main",
"value": "",
"disabled": false,
"autopostback": true,
"options": [
{
"value": "",
"text": "",
"selected": false
"selected": true
},
{
"value": "Main",
"text": "Main",
"selected": true
"selected": false
}
]
},
@ -308,24 +184,14 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_DropDownListExtLocation",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$DropDownListExtLocation",
"label": "",
"value": "External wall",
"value": "",
"disabled": false,
"autopostback": true,
"options": [
{
"value": "",
"text": "",
"selected": false
},
{
"value": "External wall",
"text": "External wall",
"selected": true
},
{
"value": "Alternative wall 1",
"text": "Alternative wall 1",
"selected": false
}
]
},
@ -335,19 +201,19 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_DropDownListExtOrientation",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$DropDownListExtOrientation",
"label": "",
"value": "North",
"value": "",
"disabled": false,
"autopostback": true,
"options": [
{
"value": "",
"text": "",
"selected": false
"selected": true
},
{
"value": "North",
"text": "North",
"selected": true
"selected": false
},
{
"value": "North East",
@ -397,7 +263,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_TextBoxExtUValue",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$TextBoxExtUValue",
"label": "",
"value": "2.80",
"value": "0.00",
"disabled": true,
"autopostback": true
},
@ -407,7 +273,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWindowsPanel_TextBoxExtGValue",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWindowsPanel$TextBoxExtGValue",
"label": "",
"value": "0.76",
"value": "0.00",
"disabled": true,
"autopostback": true
},
@ -420,7 +286,7 @@
"value": "on",
"disabled": false,
"autopostback": false,
"checked": true
"checked": false
},
{
"tag": "select",
@ -582,12 +448,6 @@
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionConservatories_Link"
},
{
"text": "Flats",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionFlats$Link\", \"\", true, \"\", \"\", false, true))",
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionFlats_Link"
},
{
"text": "Walls",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionWalls$Link\", \"\", true, \"\", \"\", false, true))",

71
scripts/hyde/elmhurst_dom/propertydescription.json
View file

@ -8,7 +8,7 @@
"id": "ContentBody_ContentPlaceHolder1_DropDownListPropertyType1",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$DropDownListPropertyType1",
"label": "",
"value": "F Flat",
"value": "H House",
"disabled": false,
"autopostback": true,
"options": [
@ -20,7 +20,7 @@
{
"value": "H House",
"text": "H House",
"selected": false
"selected": true
},
{
"value": "B Bungalow",
@ -30,7 +30,7 @@
{
"value": "F Flat",
"text": "F Flat",
"selected": true
"selected": false
},
{
"value": "M Maisonette",
@ -44,13 +44,60 @@
}
]
},
{
"tag": "select",
"type": "select-one",
"id": "ContentBody_ContentPlaceHolder1_DropDownListPropertyType2",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$DropDownListPropertyType2",
"label": "",
"value": "E End-Terrace",
"disabled": false,
"autopostback": true,
"options": [
{
"value": "",
"text": "",
"selected": false
},
{
"value": "D Detached",
"text": "D Detached",
"selected": false
},
{
"value": "S Semi-Detached",
"text": "S Semi-Detached",
"selected": false
},
{
"value": "M Mid-Terrace",
"text": "M Mid-Terrace",
"selected": false
},
{
"value": "E End-Terrace",
"text": "E End-Terrace",
"selected": true
},
{
"value": "EM Enclosed Mid-Terrace",
"text": "EM Enclosed Mid-Terrace",
"selected": false
},
{
"value": "EE Enclosed End-Terrace",
"text": "EE Enclosed End-Terrace",
"selected": false
}
]
},
{
"tag": "input",
"type": "text",
"id": "ContentBody_ContentPlaceHolder1_TextBoxStoreys",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TextBoxStoreys",
"label": "Storeys",
"value": "1",
"value": "2",
"disabled": false,
"autopostback": true
},
@ -60,7 +107,7 @@
"id": "ContentBody_ContentPlaceHolder1_TextBoxHabitableRooms",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TextBoxHabitableRooms",
"label": "Habitable Rooms",
"value": "3",
"value": "4",
"disabled": false,
"autopostback": false
},
@ -70,7 +117,7 @@
"id": "ContentBody_ContentPlaceHolder1_TextBoxHeatedHabitableRooms",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TextBoxHeatedHabitableRooms",
"label": "Heated Habitable Rooms",
"value": "3",
"value": "4",
"disabled": false,
"autopostback": true
},
@ -80,7 +127,7 @@
"id": "ContentBody_ContentPlaceHolder1_DropDownListDateBuiltMain",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$DropDownListDateBuiltMain",
"label": "Main Property",
"value": "L 2012-2022",
"value": "G 1983-1990",
"disabled": false,
"autopostback": true,
"options": [
@ -122,7 +169,7 @@
{
"value": "G 1983-1990",
"text": "G 1983-1990",
"selected": false
"selected": true
},
{
"value": "H 1991-1995",
@ -147,7 +194,7 @@
{
"value": "L 2012-2022",
"text": "L 2012-2022",
"selected": true
"selected": false
},
{
"value": "M 2023 onwards",
@ -968,12 +1015,6 @@
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionConservatories_Link"
},
{
"text": "Flats",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionFlats$Link\", \"\", true, \"\", \"\", false, true))",
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionFlats_Link"
},
{
"text": "Walls",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionWalls$Link\", \"\", true, \"\", \"\", false, true))",

6
scripts/hyde/elmhurst_dom/propertydetails.json
View file

@ -228,12 +228,6 @@
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionConservatories_Link"
},
{
"text": "Flats",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionFlats$Link\", \"\", true, \"\", \"\", false, true))",
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionFlats_Link"
},
{
"text": "Walls",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionWalls$Link\", \"\", true, \"\", \"\", false, true))",

12
scripts/hyde/elmhurst_dom/roofs.json
View file

@ -97,7 +97,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMain_WebUserControlRoofMain_DropDownListThickness",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMain$WebUserControlRoofMain$DropDownListThickness",
"label": "Insulation Thickness",
"value": "270 mm",
"value": "100 mm",
"disabled": false,
"autopostback": true,
"options": [
@ -129,7 +129,7 @@
{
"value": "100 mm",
"text": "100 mm",
"selected": false
"selected": true
},
{
"value": "125 mm",
@ -164,7 +164,7 @@
{
"value": "270 mm",
"text": "270 mm",
"selected": true
"selected": false
},
{
"value": "300 mm",
@ -273,12 +273,6 @@
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionConservatories_Link"
},
{
"text": "Flats",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionFlats$Link\", \"\", true, \"\", \"\", false, true))",
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionFlats_Link"
},
{
"text": "Walls",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionWalls$Link\", \"\", true, \"\", \"\", false, true))",

36
scripts/hyde/elmhurst_dom/spaceheating.json
View file

@ -18,7 +18,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMainHeating1_WebUserControlMainHeating1_TextBoxMainHeatingCode",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMainHeating1$WebUserControlMainHeating1$TextBoxMainHeatingCode",
"label": "Main Heating EES Code",
"value": "BGB",
"value": "BGW",
"disabled": false,
"autopostback": true
},
@ -114,7 +114,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMainHeating1_WebUserControlMainHeating1_RadioButtonListFlueType",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMainHeating1$WebUserControlMainHeating1$RadioButtonListFlueType",
"label": "Flue Type",
"value": "Open",
"value": "Balanced",
"disabled": false,
"autopostback": true,
"options": [
@ -126,12 +126,12 @@
{
"value": "Balanced",
"text": "Balanced",
"selected": false
"selected": true
},
{
"value": "Open",
"text": "Open",
"selected": true
"selected": false
}
]
},
@ -335,7 +335,7 @@
"id": "ContentBody_ContentPlaceHolder1_DropDownListSecondaryHeatingPresent",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$DropDownListSecondaryHeatingPresent",
"label": "",
"value": "Yes",
"value": "No",
"disabled": false,
"autopostback": true,
"options": [
@ -347,25 +347,15 @@
{
"value": "No",
"text": "No",
"selected": false
"selected": true
},
{
"value": "Yes",
"text": "Yes",
"selected": true
"selected": false
}
]
},
{
"tag": "input",
"type": "text",
"id": "ContentBody_ContentPlaceHolder1_TextBoxSecondaryHeatingCode",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TextBoxSecondaryHeatingCode",
"label": "Secondary Heating EES Code",
"value": "RWM",
"disabled": false,
"autopostback": true
},
{
"tag": "input",
"type": "image",
@ -845,12 +835,6 @@
"onclick": null,
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMainHeating2_WebUserControlMainHeating2_ButtonMainHeatingCode"
},
{
"text": "",
"href": "javascript:__doPostBack('ctl00$ctl00$ContentBody$ContentPlaceHolder1$ButtonSecondaryHeatingCode','')",
"onclick": null,
"id": "ContentBody_ContentPlaceHolder1_ButtonSecondaryHeatingCode"
},
{
"text": "Boiler search",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$ContentPlaceHolder1$SelectBoilerDialog$HyperLinkActionBoilerSearch\", \"\", true, \"\", \"\", false, true))",
@ -911,12 +895,6 @@
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionConservatories_Link"
},
{
"text": "Flats",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionFlats$Link\", \"\", true, \"\", \"\", false, true))",
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionFlats_Link"
},
{
"text": "Walls",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionWalls$Link\", \"\", true, \"\", \"\", false, true))",

42
scripts/hyde/elmhurst_dom/ventilationandcooling.json
View file

@ -152,19 +152,19 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelAirPressureTest_DropDownListTestMethod",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelAirPressureTest$DropDownListTestMethod",
"label": "Test Method",
"value": "Not available",
"value": "Blower Door",
"disabled": false,
"autopostback": true,
"options": [
{
"value": "Not available",
"text": "Not available",
"selected": true
"selected": false
},
{
"value": "Blower Door",
"text": "Blower Door",
"selected": false
"selected": true
},
{
"value": "Pulse",
@ -173,13 +173,33 @@
}
]
},
{
"tag": "input",
"type": "text",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelAirPressureTest_TextBoxTestResult",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelAirPressureTest$TextBoxTestResult",
"label": "Pressure Test Result",
"value": "3.59",
"disabled": false,
"autopostback": true
},
{
"tag": "input",
"type": "text",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelAirPressureTest_TextBoxTestCertificateNumber",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelAirPressureTest$TextBoxTestCertificateNumber",
"label": "Certificate Number",
"value": "APT-10093116330",
"disabled": false,
"autopostback": true
},
{
"tag": "input",
"type": "text",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelLighting_TextBoxLightsTotal",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelLighting$TextBoxLightsTotal",
"label": "",
"value": "6",
"value": "9",
"disabled": false,
"autopostback": true
},
@ -200,7 +220,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelLighting_TextBoxLedLightsTotal",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelLighting$TextBoxLedLightsTotal",
"label": "",
"value": "6",
"value": "0",
"disabled": false,
"autopostback": true
},
@ -210,7 +230,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelLighting_TextBoxCflLightsTotal",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelLighting$TextBoxCflLightsTotal",
"label": "",
"value": "0",
"value": "2",
"disabled": false,
"autopostback": true
},
@ -220,7 +240,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelLighting_TextBoxLelLightsTotal",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelLighting$TextBoxLelLightsTotal",
"label": "",
"value": "6",
"value": "2",
"disabled": true,
"autopostback": true
},
@ -230,7 +250,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelLighting_TextBoxIncandescentLights",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelLighting$TextBoxIncandescentLights",
"label": "",
"value": "0",
"value": "7",
"disabled": true,
"autopostback": false
},
@ -402,12 +422,6 @@
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionConservatories_Link"
},
{
"text": "Flats",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionFlats$Link\", \"\", true, \"\", \"\", false, true))",
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionFlats_Link"
},
{
"text": "Walls",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionWalls$Link\", \"\", true, \"\", \"\", false, true))",

48
scripts/hyde/elmhurst_dom/walls.json
View file

@ -8,7 +8,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMain_InnerTabContainerMain_TabPanelExternalWallMain_WebUserControlWallMain_DropDownListType",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMain$InnerTabContainerMain$TabPanelExternalWallMain$WebUserControlWallMain$DropDownListType",
"label": "Type",
"value": "TI Timber Frame",
"value": "CA Cavity",
"disabled": false,
"autopostback": true,
"options": [
@ -40,12 +40,12 @@
{
"value": "CA Cavity",
"text": "CA Cavity",
"selected": false
"selected": true
},
{
"value": "TI Timber Frame",
"text": "TI Timber Frame",
"selected": true
"selected": false
},
{
"value": "SY System Build",
@ -65,7 +65,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMain_InnerTabContainerMain_TabPanelExternalWallMain_WebUserControlWallMain_DropDownListInsulation",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMain$InnerTabContainerMain$TabPanelExternalWallMain$WebUserControlWallMain$DropDownListInsulation",
"label": "Insulation",
"value": "A As Built",
"value": "F Filled Cavity",
"disabled": false,
"autopostback": true,
"options": [
@ -74,6 +74,36 @@
"text": "",
"selected": false
},
{
"value": "E External",
"text": "E External",
"selected": false
},
{
"value": "F Filled Cavity",
"text": "F Filled Cavity",
"selected": true
},
{
"value": "FI Filled Cavity + Internal",
"text": "FI Filled Cavity + Internal",
"selected": false
},
{
"value": "FE Filled Cavity + External",
"text": "FE Filled Cavity + External",
"selected": false
},
{
"value": "UI Unfilled Cavity + Internal",
"text": "UI Unfilled Cavity + Internal",
"selected": false
},
{
"value": "UE Unfilled Cavity + External",
"text": "UE Unfilled Cavity + External",
"selected": false
},
{
"value": "I Internal",
"text": "I Internal",
@ -82,7 +112,7 @@
{
"value": "A As Built",
"text": "A As Built",
"selected": true
"selected": false
},
{
"value": "U Unknown",
@ -97,7 +127,7 @@
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMain_InnerTabContainerMain_TabPanelExternalWallMain_WebUserControlWallMain_TextBoxWallThickness",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelMain$InnerTabContainerMain$TabPanelExternalWallMain$WebUserControlWallMain$TextBoxWallThickness",
"label": "Wall Thickness [mm]",
"value": "400",
"value": "280",
"disabled": false,
"autopostback": true
},
@ -217,12 +247,6 @@
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionConservatories_Link"
},
{
"text": "Flats",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionFlats$Link\", \"\", true, \"\", \"\", false, true))",
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionFlats_Link"
},
{
"text": "Walls",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionWalls$Link\", \"\", true, \"\", \"\", false, true))",

155
scripts/hyde/elmhurst_dom/waterheating.json
View file

@ -21,154 +21,7 @@
"value": "on",
"disabled": true,
"autopostback": false,
"checked": true
},
{
"tag": "select",
"type": "select-one",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWaterHeating_DropDownListCylinderSize",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWaterHeating$DropDownListCylinderSize",
"label": "",
"value": "Large",
"disabled": false,
"autopostback": true,
"options": [
{
"value": "",
"text": "",
"selected": false
},
{
"value": "No Access",
"text": "No Access",
"selected": false
},
{
"value": "Normal",
"text": "Normal",
"selected": false
},
{
"value": "Medium",
"text": "Medium",
"selected": false
},
{
"value": "Large",
"text": "Large",
"selected": true
},
{
"value": "Value known",
"text": "Value known",
"selected": false
}
]
},
{
"tag": "select",
"type": "select-one",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWaterHeating_DropDownListInsulated",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWaterHeating$DropDownListInsulated",
"label": "",
"value": "Foam",
"disabled": false,
"autopostback": true,
"options": [
{
"value": "",
"text": "",
"selected": false
},
{
"value": "No Insulation",
"text": "No Insulation",
"selected": false
},
{
"value": "Jacket",
"text": "Jacket",
"selected": false
},
{
"value": "Foam",
"text": "Foam",
"selected": true
},
{
"value": "Measured Loss",
"text": "Measured Loss",
"selected": false
}
]
},
{
"tag": "select",
"type": "select-one",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWaterHeating_DropDownListInsulationThickness",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWaterHeating$DropDownListInsulationThickness",
"label": "",
"value": "120 mm",
"disabled": false,
"autopostback": true,
"options": [
{
"value": "",
"text": "",
"selected": false
},
{
"value": "None",
"text": "None",
"selected": false
},
{
"value": "12 mm",
"text": "12 mm",
"selected": false
},
{
"value": "25 mm",
"text": "25 mm",
"selected": false
},
{
"value": "38 mm",
"text": "38 mm",
"selected": false
},
{
"value": "50 mm",
"text": "50 mm",
"selected": false
},
{
"value": "80 mm",
"text": "80 mm",
"selected": false
},
{
"value": "120 mm",
"text": "120 mm",
"selected": true
},
{
"value": "160 mm",
"text": "160 mm",
"selected": false
}
]
},
{
"tag": "input",
"type": "checkbox",
"id": "ContentBody_ContentPlaceHolder1_TabContainer_TabPanelWaterHeating_CheckBoxCylinderThermostat",
"name": "ctl00$ctl00$ContentBody$ContentPlaceHolder1$TabContainer$TabPanelWaterHeating$CheckBoxCylinderThermostat",
"label": "",
"value": "on",
"disabled": false,
"autopostback": false,
"checked": true
"checked": false
},
{
"tag": "input",
@ -811,12 +664,6 @@
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionConservatories_Link"
},
{
"text": "Flats",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionFlats$Link\", \"\", true, \"\", \"\", false, true))",
"onclick": "waitForAction(this);",
"id": "ContentBody_buttonActionFlats_Link"
},
{
"text": "Walls",
"href": "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(\"ctl00$ctl00$ContentBody$buttonActionWalls$Link\", \"\", true, \"\", \"\", false, true))",

8
scripts/hyde/elmhurst_download.py
View file

@ -28,7 +28,7 @@ SESSION_DIR = HERE / ".elmhurst-session"
SAMPLE_DIR = (
HERE.parent.parent
/ "backend/epc_api/json_samples/real_life_examples"
/ "SAP-Schema-17.1/uprn_10092973954"
/ "SAP-Schema-17.0/uprn_10090944225"
)
ASSESSMENT_GUID = "B44A0DB4-4C08-4241-B818-86F060172105"
@ -85,6 +85,12 @@ def download_reports() -> int:
print(f"entering assessment {ASSESSMENT_GUID} ...", flush=True)
page.goto(ENTRY_URL, wait_until="networkidle", timeout=60_000)
# The Summary nav only fires AFTER the Recommendations validation gate has
# been visited — clicking Summary straight from the Address page is a no-op.
print("navigating via Recommendations ...", flush=True)
with page.expect_navigation(wait_until="networkidle", timeout=60_000):
page.click("#ContentBody_buttonActionRecommendations_Link", timeout=15_000)
page.wait_for_timeout(600)
print("navigating to Energy Report Summary ...", flush=True)
with page.expect_navigation(wait_until="networkidle", timeout=60_000):
page.click("#ContentBody_buttonActionSummary_Link", timeout=15_000)

Some files were not shown because too many files have changed in this diff Show more