User reverted the transaction_type drop after noting that it doesn't help
detect full-SAP assessments (that's `assessment_type` on the bulk-register
record, filtered out at build_features.py:37).
tenure removal stays; v2.0.0 still MAJOR (a column was removed).
Neither field physically affects SAP rating; they're dataset-side metadata
(owner-occupied vs rented, sale vs marketed) and any correlation with
sap_score is confounded with age/condition that the model already sees
through built_form / property_type / construction_age_band.
Dropping reduces feature count and removes a source of spurious split-gain.
MAJOR per ADR-0007 versioning policy (column removal): 1.0.0 -> 2.0.0.
Per ADR-0008: the v15 baseline reports MAPE but optimises MSE, which
under-weights tail rows. Switching to objective='mape' applies gradient
proportional to 1/|y| and lets the model focus where MAPE penalises.
Targets co2_emissions, space_heating_kwh, hot_water_kwh, and peui_raw
retain the default 'regression' objective (some rows have ~zero CO2 from
heavy PV; MAPE objective destabilises near zero).
Sample weights deferred to slice 16i if slice 16h's per-decile residuals
still show tail bias after the objective switch.
12 columns renamed; extension_2_* not added (88% null on 250k corpus;
envelope_heat_loss_w_per_k already sums extension_2+ via part-iterator).
ADR-0008.
VERSION 0.4.0 -> 1.0.0 (MAJOR per ADR-0007 versioning policy). Coordinated
cutover with AutoGluon repo + scoring lambda required at deploy time.
features_v16.txt is regenerated from transform.schema() at write-parquet time
(data/ml_training is gitignored; not committed).
ECF reconstruction per SAP10 §20.1 (Mid physics, ADR-0008):
total_cost_gbp = (space_kwh*p_space + dhw_kwh*p_dhw + light_kwh*p_elec) / 100
ECF = 0.42 * total_cost / (TFA + 45)
log10_ecf = log10(ECF) [0 for non-positive]
p_* are Table 32 unit prices via fuel_unit_price_p_per_kwh. Standing
charges deliberately omitted (constant fuel-mix offset; ADR-0008).
predicted_sap_score is NOT emitted as a feature (ADR-0008 Mid not Deep):
the model is left to learn the piecewise log/linear transform from
log10_ecf -> SAP itself, keeping the data layer SAP-version-agnostic.
VERSION 0.3.0 -> 0.4.0 (MINOR).
New module domain.ml.envelope sums Sigma(U*A) + y*A_exposed across every
sap_building_part on a cert. U-values come from rdsap_uvalues' cascade
defaults, so the feature is never null.
Per-part inputs: wall / roof / floor / party-wall / windows / doors.
Windows + doors are apportioned to the main part (first in the list)
per RdSAP10 convention.
Wired into EpcMlTransform.to_row; transform VERSION 0.1.0 -> 0.2.0
(MINOR bump for an additive column per the ADR-0007 policy).
7 envelope unit tests + 2 transform-level tests, all AAA. Reference
geometry: 100 m^2 age-G mid-terrace -> ~208 W/K; doubles for two
storeys; drops with better insulation; sums across extensions.
Adds `_per_decile_residuals` and writes `residuals_<target>.json` next to
metrics.json. Buckets test-set rows by deciles of the true target value;
each bucket carries count + MAPE + MAE + mean residual + true_min/max.
Lets us tell whether errors concentrate in the tails of the true distribution
(e.g. SAP<40 / SAP>85) vs the mid-band — which the global MAPE alone hides.
Baseline for slice 16's MAPE-improvement ablations.
Previously kept the full list of EpcPropertyData in memory before calling
EpcMlTransform.to_rows. For the 25k slice that's ~30 MB; for the 580k
full-2026 corpus it OOM-killed the process silently. Now: parse cert ->
to_row -> append dict -> drop EpcPropertyData reference, so memory is
O(row-dict * n) instead of O(EpcPropertyData * n). Same end-of-frame
post-processing (categorical casts, column-order pin).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
15 new features wired through schema -> domain -> mapper -> transform:
Main Dwelling fabric (11):
- wall_insulation_type, wall_insulation_thickness_mm, wall_dry_lined,
wall_thickness_mm, party_wall_construction
- roof_insulation_location, roof_insulation_thickness_mm
- floor_construction, floor_insulation, floor_insulation_thickness_mm,
floor_heat_loss
Dwelling-level scalars (4):
- multiple_glazed_proportion, number_baths, number_baths_wwhrs,
extract_fans_count
Thickness strings like '50mm'/'NI'/'ND' parsed via _parse_thickness_mm; NI
(no insulation) lands as 0mm so the model sees the physical zero rather than
a missing value. Categorical sentinels ('NA'/'NI'/'ND') become None.
Also fixed long-standing typo `multiple_glazed_propertion` -> `_proportion`
in domain dataclass + its lone DB-model usage.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two production fixes surfaced by the live run:
- mapper.from_rdsap_schema_21_0_1 now sets the three ML target scalars
(energy_rating_current, co2_emissions_current, energy_consumption_current).
They were silently None for every cert before, leaving the only labels as
the kWh fields from renewable_heat_incentive.
- train_baseline coerces object-dtype columns to numeric (None -> NaN) and
drops rows with null target per fit, so LightGBM accepts the frame.
E2E on 500 real certs (~1s):
sap_score R^2=0.604 MAPE=0.084
co2_emissions R^2=0.813 MAPE=0.130
peui_raw R^2=0.979 MAPE=0.026
space_heating_kwh R^2=0.823 MAPE=0.213
hot_water_kwh R^2=0.519 MAPE=0.115
peui_ucl excluded: UCL correction still needs wiring.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Currently fails on SapWindow.glazing_gap (first of ~30 fields the dataclass
incorrectly treats as required). Will go GREEN once 14j sweeps Optional.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bulk entries are NDJSON of wrapper records, not a JSON array. Each wrapper
carries certificate_number, assessment_type, and a stringified document with
the actual EPC schema payload. Filter to RdSAP, unwrap document, then map.
remote_bulk_fetcher: per-entry presigned-URL refresh (30s S3 TTL).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ijson use_float fixes Decimal/float coercion when streaming JSON.
pyright extraPaths so the new pkg type-checks against domna-domain.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Four ventilation features: mechanical_ventilation (categorical
SAP10 code, 0=natural through 6=positive-input-from-outside per
epc_codes.csv mechanical_ventilation enum), mechanical_vent_duct_type
(categorical), blocked_chimneys_count (int), and pressure_test
(int — air-tightness SAP10 code).
Pulled from top-level EpcPropertyData fields; ventilation on SAP10
API EPCs sits on the certificate directly, not on the
sap_ventilation block (which is site-notes-only).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Nine more energy-source features land: has_pv_battery,
pv_battery_count, pv_battery_capacity_kwh (count × per-unit
capacity from pv_batteries.pv_battery, nullable when count=0),
has_wind_turbine, wind_turbine_count, mains_gas (the dominant
fuel-deduction signal), and the three smart-meter / export
booleans (electricity_smart_meter_present, gas_smart_meter_present,
is_dwelling_export_capable).
Closes the PV/solar feature group started in slice 11a.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fifteen PV features land: has_pv (bool), pv_capacity_source (str
categorical: measured / estimated_from_roof_area / none),
pv_array_count, pv_total_peak_power_kw, eight peak-power-by-octant
columns (pv_peak_power_kw_{N..NW}), peak-power-weighted
pv_avg_pitch and pv_avg_overshading (nullable), and
pv_percent_roof_area (nullable — populated only on the estimated
branch).
Dispatches on the SAP10 EpcPropertyData.SapEnergySource shapes added
in slice 10.5: photovoltaic_arrays populates → measured;
photovoltaic_supply.none_or_no_details.percent_roof_area > 0 →
estimated; everything else → none. percent_roof_area == 0 is the
canonical no-PV payload and surfaces as 'none', not 'estimated'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SAP10 EPCs with measured PV carry photovoltaic_supply as a nested
list of arrays (peak_power, pitch, orientation, overshading) rather
than the legacy unmeasured wrapper {none_or_no_details:
{percent_roof_area: N}}. The schema-21 dataclasses now accept both
shapes via Union[PhotovoltaicSupply, List[List[PhotovoltaicArray]]],
and from_dict._coerce now dispatches list values onto list type
variants of multi-type Unions.
EpcPropertyData.SapEnergySource gains
photovoltaic_arrays: Optional[List[PhotovoltaicArray]] — populated
when the measured shape is present, otherwise None. The legacy
photovoltaic_supply field is preserved for the fallback case.
Both schema-21.0.0 and 21.0.1 mappers dispatch via the new
_map_schema_21_pv helper.
Unblocks Slice 11 (PV feature aggregation in EpcMlTransform).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fifteen heating features land via hybrid Top-1 + flat fields: the
primary heating slot from main_heating_details[0] gives
main_fuel_type, heat_emitter_type, main_heating_control,
main_heating_category, has_fghrs, fan_flue_present, boiler_flue_type
and central_heating_pump_age (all int-categorical for the SAP10
codes); main_heating_count carries the aggregate. Water heating
adds water_heating_code, water_heating_fuel, cylinder_size, and
cylinder_insulation_thickness_mm. Secondary heating is summarised
by has_secondary_heating (derived) and secondary_fuel_type.
Fuel codes follow the gov api enums in epc_codes.csv (44 main_fuel
values shared with water_heating_fuel). Union[int, str] fields
coerce to int when the value is int, else None.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Thirteen building-parts features land: five cross-all-parts physical
aggregates (count, total_heat_loss_perimeter_m,
total_party_wall_length_m, total_floor_area_from_parts_m2,
avg_room_height_m) and eight Main-Dwelling-specific columns
(heat_loss_perimeter, party_wall_length, total_floor_area,
avg_room_height, has_room_in_roof, construction_age_band,
wall_construction, roof_construction). Main-Dwelling columns are
None when no part has identifier == 'Main Dwelling' — honest about
data quality rather than silently falling back to the first part.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds seventeen window-categorical-share features: one float per
SAP10 glazed_type code (1-15) plus a `_other` bucket for anything
outside the enum, and a single `window_pct_pvc_frame` for the
area-weighted PVC-frame share. All shares are area-weighted over
total window area; null pvc_frame share for window-less properties.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Thirteen window-aggregate features land on the transform: count,
total area, eight SAP-octant area columns (N/NE/E/SE/S/SW/W/NW),
area-weighted draught-proofing pct, and area-weighted u_value +
solar transmittance (nullable, populated only when windows carry
transmission_details). Windows with orientation outside 1-8 (0,
NR) contribute to count and total area but no octant.
Also: epc codes CSV (gov api /api/codes export, RdSAP-Schema-21.x +
older versions) moved next to EpcPropertyData as epc_codes.csv —
canonical SAP enum source for upcoming categorical-share slices.
.gitignore exception added so the reference CSV is tracked.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds seven flat categorical features (dwelling_type, tenure,
transaction_type, property_type, built_form, region_code,
country_code) emitted as raw strings. New ColumnSpec.categorical
bool tells the parquet writer to cast these to pd.Categorical at the
I/O boundary, keeping pandas out of the domain/schema module.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds three non-nullable booleans (solar_water_heating,
has_hot_water_cylinder, has_fixed_air_conditioning) and three
optional integer indicators (percent_draughtproofed,
energy_rating_average, environmental_impact_current). All direct
EpcPropertyData field reads.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Ten flat int counts added to the transform — door_count,
habitable/heated/wet/insulated_door counts, extensions, open
chimneys, and the three fixed-lighting bulb counts (CFL/LED/
incandescent). All non-nullable; direct EpcPropertyData field reads.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First feature column lands on the transform: schema() advertises
total_floor_area_m2 as a non-nullable float; to_row() emits the value
from EpcPropertyData.total_floor_area_m2 alongside the six targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previous slice commits used -a-style and missed these new files;
imports in transform.py and test_transform.py would dangle on a
fresh checkout. Re-running pytest after this commit covers all four
EpcMlTransform tests cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
