Promotes ADR-0009 from Proposed to Accepted after the grill-with-docs
session resolved all seven open questions. Bundles the SAP 10.3 and
RdSAP 10 specifications under docs/sap-spec/ plus a calculator design
sketch (module layout, monthly-loop pseudo-code, status table).
CONTEXT.md adds three new domain terms parallel to existing performance
language:
- Calculated SAP10 Performance (parallel to Effective / Lodged)
- SAP10 Calculation (process; implemented by Sap10Calculator)
- Measure Application (process; implemented by MeasureApplicator)
ML pipeline is NOT retired — it stays as the residual head once the
calculator reaches parity in Session B. ADR-0009 §"Grill outcomes" carries
the seven binding scope decisions plus three Session-A-scope changes
discovered during the grill (RdSAP §19 EER formula, SAP 10.2 Appendix A
cross-reference, RdSAP Table 29 cascade defaults).
v20a added ventilation_heat_loss_w_per_k as a standalone feature but never
connected it to the HLC inside predicted_space_heating_kwh, so the
downstream physics aggregates (predicted_ecf, predicted_total_fuel_cost,
predicted_log10_ecf — the top-10 model features) never saw the
infiltration signal. Importance for ventilation_heat_loss_w_per_k was rank
58/196 (importance 30) vs envelope's rank 21 (86).
Adds the ventilation column to the envelope-conduction HLC before
applying HDH and efficiency, so chimney + draught-proofing signals flow
through the physics aggregates the model actually uses. Default 0 keeps
backwards compatibility.
Adds SAP10.2 §C tracer-bullet infiltration model as a new physics-as-feature
column alongside envelope_heat_loss_w_per_k. ACH = structural baseline
(0.35 masonry / 0.25 timber-or-system-built) + open chimneys at 40 m³/h each
minus a draught-proofing reduction scaled by window_pct_draught_proofed,
then volumed and converted to W/K. Targets the d0 catastrophic-low-SAP tail
where chimney + leakage signals dominate but envelope conduction alone
under-counts heat loss.
Scope deferred to follow-ups: MVHR/MEV factors (mechanical_ventilation is
100% null in the corpus), pressure-test override (pressure_test also 100%
null - slice 18e mapper fix), open flues / passive vents / flueless gas
fires (sap_ventilation sparsely populated).
Many real certs carry main_heating_category=4 (heat pump) but null
sap_main_heating_code, so seasonal_efficiency() was returning the 0.80
gas-boiler default — a 3x COP under-count that dragged the high-SAP
heat-pump tail. Adds main_heating_category + main_fuel_type fallbacks:
cat=4 -> 2.30, cat=7 -> 1.00, cat=10 routes by fuel
(electric=1.00, gas=0.55, oil=0.65), cat=5 warm air -> 0.76.
Explicit SAP codes still win.
When wall_construction integer is missing or WALL_UNKNOWN, u_wall now
parses the top-level walls[i].description for material keywords
(sandstone/limestone/granite/whinstone/cob/system built/timber frame/
solid brick/cavity) before falling through to the cavity-by-age default.
Explicit construction codes still win. Threaded through
envelope_heat_loss_w_per_k via a joined wall description string off the
top-level walls list.
Table 18 age-band roof defaults assume joist insulation >= 100mm, which
mis-rates heritage roofs the surveyor explicitly described as
uninsulated. u_roof now reads roofs[i].description and routes
"no insulation" / "uninsulated" -> 2.30 W/m^2K and "limited insulation"
-> 1.50 W/m^2K, threaded through envelope_heat_loss_w_per_k via a single
joined description string off the top-level roofs list.
Explicit insulation_thickness_mm still wins over description.
predicted_total_fuel_cost_gbp was silently mispricing every non-gas
property because primary_main_fuel_type / water_heating_fuel store the
gov EPC API enum (26=mains gas, 27=LPG, 28=oil, 29=electricity) and our
_FUEL_UNIT_PRICE dict is keyed by Table 32 codes (1=gas, 4=oil, 30=elec).
Codes 26-29 hit the dict's default 3.48 p/kWh -- silently treating
electric immersion as gas.
Concrete impact on OX1 5LR Sep 2025 cert (worst-predicted SAP=41, model
84): water_heating_fuel=29 (electric immersion). Real DHW cost 2941 kWh
* 13.19p = £388/yr; we computed 2941 * 3.48 = £102 (4x under). Net
predicted_total_fuel_cost £292 vs implied real £2513 -- predicted_ecf
0.49 (~SAP 93) vs real ECF 4.24 (SAP 41).
Effect: every off-gas property's predicted_ecf was systematically too
low, dragging the model's catastrophic-low-SAP predictions toward
mid-band. Expected to substantially reduce decile-0 bias on retrain.
New _API_TO_TABLE32 map covers codes 0-29. 4 new AAA tests; VERSION
2.2.0 -> 2.3.0 (MINOR; behavioural fix to existing column values).
The 17a-baseline residuals showed cylinder_insulation_thickness_mm,
cylinder_size and cylinder_insulation_type at ranks 3/6/9 for hot_water_kwh
because the crude 16d formula didn't use them -- the model had to learn
storage physics from raw features.
Now predicted_hot_water_kwh sums:
useful_demand (existing, unchanged)
+ distribution_loss = useful * 0.15
+ storage_loss = volume * insulation_factor * 365 * 0.6
(volume from cylinder_size, factor from
cylinder_insulation_thickness_mm or age-default)
+ primary_circuit_loss = 245 (age A-J) / 60 (age K-M)
- wwhrs_credit = useful * 0.12 if number_baths_wwhrs > 0
- solar_hw_credit = 250 if solar_water_heating
all / efficiency_water = delivered kWh
Same inputs we already extract; just plumbed through. Expected:
predicted_hot_water_kwh feature usage jumps from rank 10 to top tier,
hot_water_kwh MAPE drops from 7.17%, and predicted_ecf gets tighter for
gas-heat + electric-DHW mid-band homes -> SAP MAPE marginally better.
5 new AAA tests; VERSION 2.1.0 -> 2.2.0 (MINOR; column semantics enriched).
Closes the high-SAP under-prediction gap diagnosed in 16h. 40% of SAP-85+
properties have PV; predicted_ecf was 1.74 mean at that band -> SAP ~88
via the formula, vs label SAP 90+. Inverse: PV homes had HIGHER predicted_ecf
than non-PV at the same band because cost reconstruction had zero export
credit.
New helper: predicted_pv_generation_kwh(kWp, region) -> kWh/yr from a
SAP10.2 Table 6e regional yield factor (UK avg 850 kWh/kWp/yr; Highland
650; Thames 920).
predicted_total_fuel_cost_gbp now subtracts pv_kwh * standard electricity
price (Table 32 code 30, both self-consumption and export at 13.19 p/kWh).
New feature column predicted_pv_generation_kwh exposed alongside the
adjusted cost so the model sees both signals.
VERSION 2.0.0 -> 2.1.0 (MINOR: column added; existing column semantics
shifted but pre-deploy so no consumer break).
train_baseline now returns mae + rmse alongside mape/smape/r2. MAE is the
user-facing metric ("predicted SAP within N points"); RMSE the quadratic
counterpart. Both come straight from sklearn.
New sample_weight_fn parameter: callable(y_train) -> per-row weights.
Threads into LGBMRegressor.fit's sample_weight argument. Default None
preserves existing behaviour.
Default tail strategy exposed as low_sap_tail_weight(y, threshold=58,
weight=3): 3x weight where SAP < 58. Threshold picked from slice 16h's
per-decile residuals — decile 0 (SAP 1-58) carries 17% MAPE vs <5% body.
Three TDD tracers, all AAA.
250k retrain showed objective='mape' loses ~0.6 percentage points of
global sap_score MAPE (3.92% with regression vs 4.50% with mape) and
~0.7 pts on peui_ucl. The mape objective over-weights the low-SAP tail
(weight ~1/y) and drags the body MAPE up by more than it gains in the
tail.
Body MAPE on v16 features is already strong (2.38% on deciles 1-8); the
remaining tail bias at decile 0 (SAP<58, +3.1 bias) needs a different
fix -- sample weights or stratified loss -- queued as slice 16i.
User reverted the transaction_type drop after noting that it doesn't help
detect full-SAP assessments (that's `assessment_type` on the bulk-register
record, filtered out at build_features.py:37).
tenure removal stays; v2.0.0 still MAJOR (a column was removed).
Neither field physically affects SAP rating; they're dataset-side metadata
(owner-occupied vs rented, sale vs marketed) and any correlation with
sap_score is confounded with age/condition that the model already sees
through built_form / property_type / construction_age_band.
Dropping reduces feature count and removes a source of spurious split-gain.
MAJOR per ADR-0007 versioning policy (column removal): 1.0.0 -> 2.0.0.
Per ADR-0008: the v15 baseline reports MAPE but optimises MSE, which
under-weights tail rows. Switching to objective='mape' applies gradient
proportional to 1/|y| and lets the model focus where MAPE penalises.
Targets co2_emissions, space_heating_kwh, hot_water_kwh, and peui_raw
retain the default 'regression' objective (some rows have ~zero CO2 from
heavy PV; MAPE objective destabilises near zero).
Sample weights deferred to slice 16i if slice 16h's per-decile residuals
still show tail bias after the objective switch.
12 columns renamed; extension_2_* not added (88% null on 250k corpus;
envelope_heat_loss_w_per_k already sums extension_2+ via part-iterator).
ADR-0008.
VERSION 0.4.0 -> 1.0.0 (MAJOR per ADR-0007 versioning policy). Coordinated
cutover with AutoGluon repo + scoring lambda required at deploy time.
features_v16.txt is regenerated from transform.schema() at write-parquet time
(data/ml_training is gitignored; not committed).
ECF reconstruction per SAP10 §20.1 (Mid physics, ADR-0008):
total_cost_gbp = (space_kwh*p_space + dhw_kwh*p_dhw + light_kwh*p_elec) / 100
ECF = 0.42 * total_cost / (TFA + 45)
log10_ecf = log10(ECF) [0 for non-positive]
p_* are Table 32 unit prices via fuel_unit_price_p_per_kwh. Standing
charges deliberately omitted (constant fuel-mix offset; ADR-0008).
predicted_sap_score is NOT emitted as a feature (ADR-0008 Mid not Deep):
the model is left to learn the piecewise log/linear transform from
log10_ecf -> SAP itself, keeping the data layer SAP-version-agnostic.
VERSION 0.3.0 -> 0.4.0 (MINOR).
New module domain.ml.envelope sums Sigma(U*A) + y*A_exposed across every
sap_building_part on a cert. U-values come from rdsap_uvalues' cascade
defaults, so the feature is never null.
Per-part inputs: wall / roof / floor / party-wall / windows / doors.
Windows + doors are apportioned to the main part (first in the list)
per RdSAP10 convention.
Wired into EpcMlTransform.to_row; transform VERSION 0.1.0 -> 0.2.0
(MINOR bump for an additive column per the ADR-0007 policy).
7 envelope unit tests + 2 transform-level tests, all AAA. Reference
geometry: 100 m^2 age-G mid-terrace -> ~208 W/K; doubles for two
storeys; drops with better insulation; sums across extensions.
Adds `_per_decile_residuals` and writes `residuals_<target>.json` next to
metrics.json. Buckets test-set rows by deciles of the true target value;
each bucket carries count + MAPE + MAE + mean residual + true_min/max.
Lets us tell whether errors concentrate in the tails of the true distribution
(e.g. SAP<40 / SAP>85) vs the mid-band — which the global MAPE alone hides.
Baseline for slice 16's MAPE-improvement ablations.
Previously kept the full list of EpcPropertyData in memory before calling
EpcMlTransform.to_rows. For the 25k slice that's ~30 MB; for the 580k
full-2026 corpus it OOM-killed the process silently. Now: parse cert ->
to_row -> append dict -> drop EpcPropertyData reference, so memory is
O(row-dict * n) instead of O(EpcPropertyData * n). Same end-of-frame
post-processing (categorical casts, column-order pin).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
15 new features wired through schema -> domain -> mapper -> transform:
Main Dwelling fabric (11):
- wall_insulation_type, wall_insulation_thickness_mm, wall_dry_lined,
wall_thickness_mm, party_wall_construction
- roof_insulation_location, roof_insulation_thickness_mm
- floor_construction, floor_insulation, floor_insulation_thickness_mm,
floor_heat_loss
Dwelling-level scalars (4):
- multiple_glazed_proportion, number_baths, number_baths_wwhrs,
extract_fans_count
Thickness strings like '50mm'/'NI'/'ND' parsed via _parse_thickness_mm; NI
(no insulation) lands as 0mm so the model sees the physical zero rather than
a missing value. Categorical sentinels ('NA'/'NI'/'ND') become None.
Also fixed long-standing typo `multiple_glazed_propertion` -> `_proportion`
in domain dataclass + its lone DB-model usage.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two production fixes surfaced by the live run:
- mapper.from_rdsap_schema_21_0_1 now sets the three ML target scalars
(energy_rating_current, co2_emissions_current, energy_consumption_current).
They were silently None for every cert before, leaving the only labels as
the kWh fields from renewable_heat_incentive.
- train_baseline coerces object-dtype columns to numeric (None -> NaN) and
drops rows with null target per fit, so LightGBM accepts the frame.
E2E on 500 real certs (~1s):
sap_score R^2=0.604 MAPE=0.084
co2_emissions R^2=0.813 MAPE=0.130
peui_raw R^2=0.979 MAPE=0.026
space_heating_kwh R^2=0.823 MAPE=0.213
hot_water_kwh R^2=0.519 MAPE=0.115
peui_ucl excluded: UCL correction still needs wiring.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Currently fails on SapWindow.glazing_gap (first of ~30 fields the dataclass
incorrectly treats as required). Will go GREEN once 14j sweeps Optional.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bulk entries are NDJSON of wrapper records, not a JSON array. Each wrapper
carries certificate_number, assessment_type, and a stringified document with
the actual EPC schema payload. Filter to RdSAP, unwrap document, then map.
remote_bulk_fetcher: per-entry presigned-URL refresh (30s S3 TTL).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ijson use_float fixes Decimal/float coercion when streaming JSON.
pyright extraPaths so the new pkg type-checks against domna-domain.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Four ventilation features: mechanical_ventilation (categorical
SAP10 code, 0=natural through 6=positive-input-from-outside per
epc_codes.csv mechanical_ventilation enum), mechanical_vent_duct_type
(categorical), blocked_chimneys_count (int), and pressure_test
(int — air-tightness SAP10 code).
Pulled from top-level EpcPropertyData fields; ventilation on SAP10
API EPCs sits on the certificate directly, not on the
sap_ventilation block (which is site-notes-only).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Nine more energy-source features land: has_pv_battery,
pv_battery_count, pv_battery_capacity_kwh (count × per-unit
capacity from pv_batteries.pv_battery, nullable when count=0),
has_wind_turbine, wind_turbine_count, mains_gas (the dominant
fuel-deduction signal), and the three smart-meter / export
booleans (electricity_smart_meter_present, gas_smart_meter_present,
is_dwelling_export_capable).
Closes the PV/solar feature group started in slice 11a.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fifteen PV features land: has_pv (bool), pv_capacity_source (str
categorical: measured / estimated_from_roof_area / none),
pv_array_count, pv_total_peak_power_kw, eight peak-power-by-octant
columns (pv_peak_power_kw_{N..NW}), peak-power-weighted
pv_avg_pitch and pv_avg_overshading (nullable), and
pv_percent_roof_area (nullable — populated only on the estimated
branch).
Dispatches on the SAP10 EpcPropertyData.SapEnergySource shapes added
in slice 10.5: photovoltaic_arrays populates → measured;
photovoltaic_supply.none_or_no_details.percent_roof_area > 0 →
estimated; everything else → none. percent_roof_area == 0 is the
canonical no-PV payload and surfaces as 'none', not 'estimated'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SAP10 EPCs with measured PV carry photovoltaic_supply as a nested
list of arrays (peak_power, pitch, orientation, overshading) rather
than the legacy unmeasured wrapper {none_or_no_details:
{percent_roof_area: N}}. The schema-21 dataclasses now accept both
shapes via Union[PhotovoltaicSupply, List[List[PhotovoltaicArray]]],
and from_dict._coerce now dispatches list values onto list type
variants of multi-type Unions.
EpcPropertyData.SapEnergySource gains
photovoltaic_arrays: Optional[List[PhotovoltaicArray]] — populated
when the measured shape is present, otherwise None. The legacy
photovoltaic_supply field is preserved for the fallback case.
Both schema-21.0.0 and 21.0.1 mappers dispatch via the new
_map_schema_21_pv helper.
Unblocks Slice 11 (PV feature aggregation in EpcMlTransform).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fifteen heating features land via hybrid Top-1 + flat fields: the
primary heating slot from main_heating_details[0] gives
main_fuel_type, heat_emitter_type, main_heating_control,
main_heating_category, has_fghrs, fan_flue_present, boiler_flue_type
and central_heating_pump_age (all int-categorical for the SAP10
codes); main_heating_count carries the aggregate. Water heating
adds water_heating_code, water_heating_fuel, cylinder_size, and
cylinder_insulation_thickness_mm. Secondary heating is summarised
by has_secondary_heating (derived) and secondary_fuel_type.
Fuel codes follow the gov api enums in epc_codes.csv (44 main_fuel
values shared with water_heating_fuel). Union[int, str] fields
coerce to int when the value is int, else None.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Thirteen building-parts features land: five cross-all-parts physical
aggregates (count, total_heat_loss_perimeter_m,
total_party_wall_length_m, total_floor_area_from_parts_m2,
avg_room_height_m) and eight Main-Dwelling-specific columns
(heat_loss_perimeter, party_wall_length, total_floor_area,
avg_room_height, has_room_in_roof, construction_age_band,
wall_construction, roof_construction). Main-Dwelling columns are
None when no part has identifier == 'Main Dwelling' — honest about
data quality rather than silently falling back to the first part.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds seventeen window-categorical-share features: one float per
SAP10 glazed_type code (1-15) plus a `_other` bucket for anything
outside the enum, and a single `window_pct_pvc_frame` for the
area-weighted PVC-frame share. All shares are area-weighted over
total window area; null pvc_frame share for window-less properties.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Thirteen window-aggregate features land on the transform: count,
total area, eight SAP-octant area columns (N/NE/E/SE/S/SW/W/NW),
area-weighted draught-proofing pct, and area-weighted u_value +
solar transmittance (nullable, populated only when windows carry
transmission_details). Windows with orientation outside 1-8 (0,
NR) contribute to count and total area but no octant.
Also: epc codes CSV (gov api /api/codes export, RdSAP-Schema-21.x +
older versions) moved next to EpcPropertyData as epc_codes.csv —
canonical SAP enum source for upcoming categorical-share slices.
.gitignore exception added so the reference CSV is tracked.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds seven flat categorical features (dwelling_type, tenure,
transaction_type, property_type, built_form, region_code,
country_code) emitted as raw strings. New ColumnSpec.categorical
bool tells the parquet writer to cast these to pd.Categorical at the
I/O boundary, keeping pandas out of the domain/schema module.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds three non-nullable booleans (solar_water_heating,
has_hot_water_cylinder, has_fixed_air_conditioning) and three
optional integer indicators (percent_draughtproofed,
energy_rating_average, environmental_impact_current). All direct
EpcPropertyData field reads.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
