Commit graph

10 commits

Author SHA1 Message Date
Khalim Conn-Kowlessar
8fddd25b9a slice 14k: E2E pipeline runs on real 2026 RdSAP certs
Two production fixes surfaced by the live run:
- mapper.from_rdsap_schema_21_0_1 now sets the three ML target scalars
  (energy_rating_current, co2_emissions_current, energy_consumption_current).
  They were silently None for every cert before, leaving the only labels as
  the kWh fields from renewable_heat_incentive.
- train_baseline coerces object-dtype columns to numeric (None -> NaN) and
  drops rows with null target per fit, so LightGBM accepts the frame.

E2E on 500 real certs (~1s):
  sap_score             R^2=0.604  MAPE=0.084
  co2_emissions         R^2=0.813  MAPE=0.130
  peui_raw              R^2=0.979  MAPE=0.026
  space_heating_kwh     R^2=0.823  MAPE=0.213
  hot_water_kwh         R^2=0.519  MAPE=0.115

peui_ucl excluded: UCL correction still needs wiring.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 20:47:41 +00:00
Khalim Conn-Kowlessar
611c07de94 slice 14h: handle real bulk-JSON shape (NDJSON wrappers + document payload)
Bulk entries are NDJSON of wrapper records, not a JSON array. Each wrapper
carries certificate_number, assessment_type, and a stringified document with
the actual EPC schema payload. Filter to RdSAP, unwrap document, then map.

remote_bulk_fetcher: per-entry presigned-URL refresh (30s S3 TTL).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 19:45:52 +00:00
Khalim Conn-Kowlessar
9eb70cede1 slice 14g: remote_bulk_fetcher extracts ZIP entries via HTTP Range (no full download)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 19:16:52 +00:00
Khalim Conn-Kowlessar
b676e05d49 slice 14f: train_baseline fits LightGBM per target, emits MAPE/R^2 + importance
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 18:47:49 +00:00
Khalim Conn-Kowlessar
23ba2ef271 slice 14e: write_training_dataset emits parquet + schema.json + manifest.json
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 18:43:31 +00:00
Khalim Conn-Kowlessar
20fd55d5a1 slice 14d: build_features wires bulk reader -> mapper -> EpcMlTransform
ijson use_float fixes Decimal/float coercion when streaming JSON.
pyright extraPaths so the new pkg type-checks against domna-domain.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 18:38:41 +00:00
Khalim Conn-Kowlessar
0ff9d546b8 slice 14c: BulkZipReader streams certs from gov bulk JSON ZIP
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 18:27:24 +00:00
Khalim Conn-Kowlessar
7a6c8b4f24 slice 14b: Storage protocol + LocalStorage impl
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 17:52:54 +00:00
Khalim Conn-Kowlessar
eb42cb88a1 slice 14a: ml_training_data pkg + sample.py (CSV filter + random sample)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 17:39:43 +00:00
Khalim Conn-Kowlessar
dfe9e3ddbe added potential file scaffolding: 2026-05-15 10:56:53 +00:00