Commit graph

57 commits

Author SHA1 Message Date
Khalim Conn-Kowlessar
1f26703dc5 feat(epc-prediction): geo-proximity weighting, per-component (#1227)
Folds a haversine distance kernel into the categorical-mode weighting so a
nearer neighbour counts for more — applied ONLY to the components that showed
a clear distance signal in the corpus pre-check (age band, wall + floor
construction, glazing: homes built/retrofitted together cluster). Roof
construction showed no decay and is excluded; heating keeps its coherent
donor. Predictor stays pure: weights come from target.coordinates vs each
Comparable.coordinates (resolved at the boundary); geo is OFF when the target
has no coords, neutral for a neighbour with none.

Scale chosen on the harness: _GEO_SCALE_KM=0.1 is the gate-safe optimum
(0.05 lifts the corpus more but regresses fixture floor_construction).
Corpus (150pc/514, geo off->on): age 0.564->0.572, age_pm1 0.841->0.847,
wall 0.902->0.912, floor_con 0.786->0.796, glazing 0.667->0.673; roof
unchanged. Fixture: glazing 0.5278->0.5833 (floor ratcheted), all else held.

Refactored recency into a reusable _recency_weights vector composed via
_combine, so similarity/recency/geo factors multiply uniformly. Fixture ships
a committed _coordinates.json (OGL OS OpenData; build script carries it from
the corpus sidecar on rebuild) so the gate exercises geo without S3.

This is the per-component method applied to geography ([[feedback_per_component_best_method]]).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:58:42 +00:00
Khalim Conn-Kowlessar
fdc314c857 feat(epc-prediction): thread coordinates onto Comparable + target (#1227)
Adds coordinates: Optional[Coordinates] to Comparable and PredictionTarget
(data carriers — the pure predictor stays IO-free), and wires load_corpus to
read an optional _coordinates.json sidecar ({uprn: [lon, lat]}) and populate
each Comparable from its cert's uprn; iter_predictions threads the held-out
target's coordinates through. Absent sidecar -> geo-weighting stays off (no
behaviour change yet — weighting lands next slice). fetch_corpus_coordinates
now writes the sidecar into the corpus dir. load_corpus populates 99% of
corpus comparables.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:46:01 +00:00
Khalim Conn-Kowlessar
c0a1bcac95 feat(epc-prediction): resolve corpus UPRN coordinates from S3 (#1227 signal check)
One-time utility: resolves every corpus cert's uprn -> WGS84 lon/lat from the
OS Open-UPRN parquet (DATA_BUCKET/spatial/) via boto3, grouping UPRNs by their
covering partition so each ~1.7MB partition is read at most once (the efficient
batch lookup we intend to add to GeospatialRepository). Caches {uprn:[lon,lat]}
locally for the validation harness. Resolved 2609/2683 corpus UPRNs (97%).

Signal pre-check result (does intra-postcode proximity predict components?):
intra-postcode distances are non-trivial (median 44m, p90 138m, max ~1km),
and nearer neighbours match the target markedly better on age band (0.63 at
<20m -> 0.16 at >300m), wall, glazing and floor construction. Roof shows no
decay. => geo-proximity is worth building, per-component (strongest for age,
the weakest fabric component).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:28:39 +00:00
Khalim Conn-Kowlessar
7f48495ed5 feat(epc-prediction): surface CO2 + PEI calculator floors in the report (#1228)
The validation report showed only the SAP calculator floor (calc(actual) vs
lodged), so the headline PEI MAE (~40 kWh/m2) read as prediction error when
much of it is the calculator's own API-path residual. Adds the CO2 + PEI
floors alongside SAP.

Diagnostic (150pc/514): PEI floor MAE 15.73 (calc(actual) vs lodged) vs SAP
floor 1.57; calc(actual)/lodged PEI ratio ~1.06 (mean +10.7, ~+6% over-
estimate). That RULES OUT the suspected gross unit/definition mismatch (a
unit bug would be ~2x/3.6x, not 1.06) and reframes #1228: the PEI gap is a
modest calculator bias (~16 floor, calc-branch) plus a larger prediction-
sensitivity term (~24) — PEI is far more prediction-sensitive than SAP.
CO2 floor 0.20 t. Script-only; no gate impact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:55:20 +00:00
Khalim Conn-Kowlessar
6e9f831296 chore(epc-prediction): grow validation corpus to 150 postcodes
Bumps N_POSTCODES 40 -> 150 for the fetch script. Larger corpus (150
postcodes / 3719 certs) reduces leave-one-out variance and unblocks the
recency-template work (#1223), which regressed the noisier 36-target gate
fixture. Corpus itself stays out of git (gitignored /tmp + persistent
backup at /workspaces/home/epc_prediction_corpus_backup).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 06:42:19 +00:00
Khalim Conn-Kowlessar
008c1922c4 feat(epc-prediction): anonymised Tier-1 fixture + builder (ADR-0030)
The committed gate needs frozen, reproducible data without dumping real UK
addresses into the repo. Add:
- harness anonymise_payload + stable_hash: hash street address + cert number
  into opaque, dedup-stable tokens; blank secondary address lines + post_town;
  keep postcode + all component/lodged fields (gov data is OGL). Unit-tested.
- scripts/build_epc_prediction_fixture.py: curate qualifying postcodes (>=1
  SAP 10.2 target + >=2 distinct addresses) from the local scratch corpus,
  anonymise, freeze under tests/fixtures/epc_prediction/.
- The frozen fixture: 15 postcodes / 280 certs / 36 SAP-10.2 targets.
  Verified no plaintext address_line_1 and post_town all blank.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 09:17:27 +00:00
Khalim Conn-Kowlessar
027ee1fba3 refactor(epc-prediction): extract shared leave-one-out scorer + corpus loader (ADR-0030)
"One scorer, two harnesses" (ADR-0030): the committed gate, the local script,
and the future battle-test must run the *same* scoring. Extract it:

- domain/epc_prediction/validation.py — `iter_predictions` (the single
  leave-one-out orchestration: latest-per-address hold-out, SAP-10.2 target
  filter, all-vintage source) + `evaluate_component_accuracy` (calculator-free
  ComponentAccuracy aggregation, the primary signal). Unit-tested.
- harness/epc_prediction_corpus.py — `load_corpus(dir)` IO: corpus dir ->
  Comparable cohorts (maps payloads, carries address + registration_date).

validate_epc_prediction.py now just loads + calls the scorer for the component
section and iterates iter_predictions for the calculator-floored end-to-end.
Identical numbers (181 targets, SAP MAE 6.34) — behaviour-preserving.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 09:12:08 +00:00
Khalim Conn-Kowlessar
65cb094abe feat(epc-prediction): SAP-10.2 target filter + carbon/PE end-to-end (ADR-0030)
Make the leave-one-out runner ADR-0030-compliant:
- Hold out only SAP 10.2 targets (sap_version == 10.2) — the source cohort
  keeps every vintage (components are methodology-agnostic).
- Label Component Accuracy as the PRIMARY, calculator-independent section.
- End-to-end vs API-lodged (SECONDARY, calculator-FLOORED): add CO2 (tonnes)
  and PEI (kWh/m2) alongside SAP, using the canonical performance.py mapping
  (co2_kg/1000; primary_energy_kwh_per_m2).
- Add the attribution readout calc(actual) vs lodged SAP — the calculator
  floor the end-to-end can reach.
- Drop the neighbour-mean-of-lodged-SAP baseline (mixes SAP versions —
  rejected by ADR-0030).

On the 181 SAP-10.2 targets: component rates are higher than the all-vintage
view (age band 60.9 -> 78.5%, floor_area mean|.| 12.7 -> 8.4). End-to-end SAP
MAE 6.34 vs the calc(actual) floor of 3.25 — ~half the gap is the known
API-path calculator residual, not prediction error.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 09:04:24 +00:00
Khalim Conn-Kowlessar
275a30a825 feat(epc-prediction): complete component coverage — fabric/glazing/renewables/doors (ADR-0030)
Finish the ADR-0030 Component Accuracy set: roof insulation thickness,
floor insulation, room-in-roof presence, modal glazing type, PV presence,
solar water heating (categoricals) + door count (residual). Presence flags
(room-in-roof, PV, solar) are always-applicable — predicting absence when
present is a real miss.

Template-copied baseline (40-postcode corpus), newly visible:
  floor_insulation         94.0%   solar_water_heating  99.7%
  has_pv                   98.6%   has_room_in_roof     91.9%
  modal_glazing_type       59.0%   <- weak
  roof_insulation_thickness 30.6%  <- weak
  door_count  mean|.| 0.40

compare_prediction now scores 19 categoricals + 5 residuals across every
SAP-load-bearing component group.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 09:00:30 +00:00
Khalim Conn-Kowlessar
41b5ce5057 refactor(epc-prediction): name-keyed categorical_hits for Component Accuracy (ADR-0030)
ADR-0030 commits Component Accuracy to ~19 categorical components (5 today
+ 8 heating + glazing/renewables). Flat *_correct dataclass fields don't
scale — each needs manual runner wiring. Collapse them into a single
`categorical_hits: dict[str, Optional[bool]]` keyed by component name, which
also matches the runner's name-keyed aggregation (now generic: it tallies
whatever components the comparison reports). No behaviour change; the
classification rates are identical (wall n 578->575 is the 3 certs whose
actual wall is None, now correctly counted as not-applicable via _classify).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 08:50:34 +00:00
Khalim Conn-Kowlessar
c3d56b00dd chore(epc-prediction): grow validation corpus to 40 postcodes (ADR-0029)
Bump N_POSTCODES 150 -> 40 as the gradual-growth step from the 3-postcode
smoke. 40 postcodes / 1113 certs / 578 leave-one-out predictions is enough
for stable, trustworthy metrics (the smoke's 2 usable postcodes were
dominated by oddball flats — floor_area mean|.| 52.6 there vs 12.7 here).
Resumable + reproducible (random.seed(2026)); raise again to scale up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 01:52:44 +00:00
Khalim Conn-Kowlessar
fa11df56c2 fix(epc-prediction): dedupe re-lodgements + leak-free leave-one-out (ADR-0029)
The register lists every historical lodgement, so a postcode cohort
contains the same physical address many times (LS61AA: 15 certs / 11
addresses; NG71AA: 15 / 9 — "FLAT 3" appears 3x in each). Two
consequences:

  - Production: a re-lodged neighbour was counting up to 3x towards the
    cohort mode. select_comparables now dedupes candidates to the latest
    cert per address (one comparable per real neighbour) — Comparable
    gains address + registration_date (the register metadata its docstring
    already anticipated, read straight off the cached payload).

  - Validation: leave-one-out leaked — predicting a flat from a near-
    identical re-lodgement of itself. The harness now holds out a whole
    address (excludes every sibling cert) and evaluates on the latest cert
    per address (the best ground truth).

Removing the leak gives the honest numbers (19 distinct addresses):
  wall_construction      93.1% -> 89.5%
  construction_age_band  65.5% -> 52.6%
  roof_construction      79.3% -> 68.4%
  floor_area mean|.|     37.9  -> 52.6 m2
The earlier figures were inflated by self-leakage; these are the real
accuracy to beat.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 00:40:23 +00:00
Khalim Conn-Kowlessar
ed96df9315 feat(epc-prediction): classify roof/floor/insulation/age categoricals (ADR-0029)
The comparison only scored main wall_construction; everything else the
predictor produces (by template-copy) went unmeasured. Extend
compare_prediction to the rest of the ADR-0029 homogeneous categoricals —
wall insulation type, construction age band, roof construction, floor
construction — and aggregate per-categorical classification rates in the
runner. A categorical hit is "not applicable" (None, excluded from the
denominator) when the actual lodges no value, so absent-roof flats don't
score free wins.

Smoke corpus (29 leave-one-out, all but wall are template-copied today):
  wall_construction      93.1%
  wall_insulation_type   93.1%
  construction_age_band  55.2%   <- loud; candidate for cohort-mode
  roof_construction      72.4%
  floor_construction     46.2%   (n=13)

These numbers drive the next slice (extend cohort-mode coverage).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 00:10:56 +00:00
Khalim Conn-Kowlessar
f3ad6343a3 feat(epc-prediction): leave-one-out validation harness (ADR-0029)
Pure compare_prediction (TDD): wall-construction classification hit + signed
residuals on floor area, window count, total window area, building-parts count.
Plus validate_epc_prediction.py (IO plumbing): drops each cert from its postcode
cohort, predicts from the rest on guaranteed inputs only, aggregates the metrics,
and reports SAP three ways (pred-calc vs lodged / vs calc-on-actual / vs the
neighbour-mean baseline). Smoke run: wall 90.9%, floor-area mean|·| 42.6 m2 (a
real signal — template-copied floor area is noisy), SAP pred-calc edges baseline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:55:05 +00:00
Khalim Conn-Kowlessar
80b525f0f4 feat(epc-prediction): postcode-clustered corpus fetch script (ADR-0029)
Builds the frozen validation corpus: samples postcodes from the register, then
caches each postcode's full cohort of raw cert payloads (the shape
from_api_response consumes), grouped by postcode, resumably. Reads the token
from backend/.env; cache dir /tmp/epc_prediction_corpus (EPC_PREDICTION_CORPUS
override). IO plumbing, not test-driven. Pairs with the leave-one-out harness.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:36:19 +00:00
Jun-te Kim
80ccec9b68 added floats helper 2026-06-12 14:28:41 +00:00
Jun-te Kim
a6123d762c Merge branch 'main' of https://github.com/Hestia-Homes/Model into feature/junte+khalim 2026-06-12 13:45:30 +00:00
Jun-te Kim
ff4a2e4242
Merge pull request #1198 from Hestia-Homes/feature/bill-derivation
Feature/bill derivation
2026-06-12 14:44:30 +01:00
Jun-te Kim
32de7f6c3f 17.1 and 18 done by claude 2026-06-12 12:52:36 +00:00
Jun-te Kim
3995433816 Map RdSAP-Schema-17.0 certs to EpcPropertyData 🟥
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:40:04 +00:00
Jun-te Kim
32eef951ee Add corpus profiler for the ADR-0028 seeing-the-data table
Reusable per-schema profiler: glazed_area band mix, Validation Cohort size,
observed-vs-predicted band glazing/floor ratio, and the ND/str sentinels that
drive schema widening. Regenerates the ADR-0028 transfer-check table from any
harvested corpus.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:36:08 +00:00
Jun-te Kim
5178197dc2 Map RdSAP-Schema-19.0 certs to EpcPropertyData 🟥
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:19:16 +00:00
Jun-te Kim
cfc337f04a Dispatch and map RdSAP-Schema-18.0 certs end-to-end 🟥
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 11:12:53 +00:00
Daniel Roth
9a42eaf243 empty commit to trigger workflows 2026-06-11 09:32:14 +00:00
Jun-te Kim
362cd20f11 scripts? 2026-06-11 07:07:27 +00:00
Daniel Roth
b9eb23f6df allow write to real s3 when running locally 2026-06-09 13:52:39 +00:00
Daniel Roth
f8c955b2d3 local runner and correct template path 2026-06-09 13:03:05 +00:00
Jun-te Kim
06cb4f7b6e Merge branch 'feature/bill-derivation' into feature/junte+khalim 2026-06-09 10:06:40 +00:00
Khalim Conn-Kowlessar
4006753620 fix(scripts): authenticate the EPC client with OPEN_EPC_API_TOKEN
The new gov EPC API (api.get-energy-performance-data..., Bearer auth) returns
403 "Bad authentication header" with EPC_AUTH_TOKEN but 200 with
OPEN_EPC_API_TOKEN — the token name is misleading (it is the Bearer token for
the new API, not the open-data API). Verified live against
/api/domestic/search. Unblocks the live EPC fetch in run_modelling_e2e.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-08 20:58:22 +00:00
Khalim Conn-Kowlessar
0f6077a830 feat(scripts): DB-catalogue local run + optional --persist for run_modelling_e2e
Slice 5 (local run sources the DB, read-only) + slice 6 (optional persist),
landing together as one script rewrite (the persist path is interleaved with
the compute path).

The same local computation now runs whether or not the result is stored:
- Both modes price against the live `material` catalogue (read-only
  ProductPostgresRepository over one shared Session) and model against a real
  Scenario read from the DB (--scenario-id; its goal_value drives the band,
  rejected if null) — so the inspected recommendations are exactly what gets
  stored. The JSON sample catalogue is no longer used by this script.
- --measures restricts the run to a comma-separated considered_measures
  allowlist (e.g. high_heat_retention_storage_heaters,solar_pv).
- --persist writes the inputs (EPC + spatial + solar) and the *same* computed
  Plan via the production repos in one PostgresUnitOfWork, then commits
  (idempotent: PlanPostgresRepository replaces by (property_id, scenario_id)).
  Gated: --persist requires --scenario-id and --portfolio-id. Default is
  inspect-only — no DB writes.

harness.console.run_modelling gains `products` and `scenario` overrides (the
seam the script drives); defaults unchanged, so existing callers are
unaffected. Suite 257 pass + 3 xfail; pyright clean; --help/guard/measure
parsing verified. Not yet executed against the DB (awaiting property_ids +
write-confirm).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-08 20:45:50 +00:00
Jun-te Kim
b48700e964 Merge branch 'main' into feature/junte+khalim 2026-06-08 16:56:15 +00:00
Khalim Conn-Kowlessar
1b4806f8e4 feat(scripts): wire S3 geospatial + Google Solar into run_modelling_e2e
Per Property the inspection script now resolves the UPRN's spatial
reference from the Ordnance Survey Open-UPRN parquet in S3
(GeospatialS3Repository over a boto3 ParquetReader) and threads both
levers into run_modelling:

- planning_restrictions: the conservation/listed/heritage flags that gate
  the wall + solar measures (ADR-0019/0020).
- solar_insights: a live Google Solar buildingInsights fetch keyed on the
  reference coordinates, so the Solar PV Options can fire (ADR-0026).

Mirrors IngestionOrchestrator._fetch's coords->solar flow. Degrades
gracefully per Property: a UPRN S3 doesn't cover -> unrestricted/no-solar;
a point Google has no coverage for (BuildingInsightsNotFoundError) ->
no-solar; both still modelled. --no-solar skips the Google leg. A context
note (restrictions; solar) is printed and written to the md/csv summary.

Verified live: spatial_for + solar fetch round-trip on real UPRNs (S3 via
ambient ~/.aws creds, pyarrow reads parquet bytes). pyright clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-08 14:55:33 +00:00
Khalim Conn-Kowlessar
0918dd37ec feat(scripts): run_modelling_e2e — inspect recommendations per property_id
Revives the local recommendation-inspection flow for specific Properties.
`scripts/run_modelling_e2e.py` reads each Property's UPRN from the DB
(read-only), fetches the latest EPC live from the gov EPC API by UPRN, runs the
Modelling stage in memory (all Generators → Optimiser → costed, attributed
Plan), and prints a per-Property plan table + writes a Markdown/CSV summary.
Persists nothing — purely for inspection.

The local DB's Properties have no linked ingested EPC (epc_property.property_id
is NULL for all rows; Ingestion's source clients are stubbed, #1136), so the
EPC must be fetched inline rather than read back. Builds the connection from the
`DB_*` env vars in backend/.env and the EPC token from `EPC_AUTH_TOKEN`.

Threads optional solar insights through harness `run_modelling` (so Solar PV
Options can fire once coordinates are wired) and adds the `solar_pv` catalogue
row. Solar + planning restrictions + DB persistence are noted follow-ups.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-08 14:25:33 +00:00
Khalim Conn-Kowlessar
24492aa4ba Merge origin/main into feature/bill-derivation (calculator + mapper fixes)
Pulls in 42 commits of calculator/mapper accuracy fixes from the per-cert
mapper-validation and floor/roof/heating fronts.

Conflict resolutions:
- mapper `_is_elmhurst_roof_window`: main dropped the branch's "wall location →
  vertical" guard (it broke cert 000516's rooflight), but that re-broke cert
  001431's two External-wall U>3.0 windows (which must stay vertical). The two
  certs lodge a BYTE-IDENTICAL §11 row, so neither location nor U separates
  them — the real discriminator is the room-in-roof context. Replaced the
  unconditional U>3.0 backstop with one gated on the BP having a room-in-roof
  (`_elmhurst_bp_has_room_in_roof`): 000516's Main BP has a "Room in roof type
  1" (→ rooflight), 001431's does not (→ vertical). Validated against BOTH —
  full Elmhurst worksheet suite 1038 pass + the 001431 window-extraction pin.
- property_postgres_repository: kept main's `ids_by_uprn` method + the branch's
  `_restrictions_of` helper.
- sap_fuel.py: the branch relocated it to domain/billing/ (already carrying
  main's to_table_32_code normalization), so kept the old path deleted.

Fallout from main's fabric fixes (validated by the boiler-3 real-cert pin which
still reproduces at delta 0):
- re-pinned the boiler-1 + boiler-instant-hw ASHP snapshot scores;
- main's §14.2 gas-boiler main-fuel derivation resolved the BGB/102 baseline
  gap, so `test_gas_boiler_instant_hw_before_baselines` is now a passing test
  (was an xfail tripwire).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-08 13:12:21 +00:00
Daniel Roth
bd4ad9022c Merge branch 'main' into feature/handle-new-magicplan-response-structure 2026-06-08 12:36:27 +00:00
Khalim Conn-Kowlessar
28b1da1e06 feat(diag): profile API SAP error against raw-API characteristics
Joins each computed cert's signed error (eval _results.csv) with a rich
feature set extracted from its RAW API JSON (not the mapped
EpcPropertyData), then ranks (feature, value) buckets by error carried
and by |mean signed| bias. Surfaces systematic API-path handling gaps —
a field the mapper silently drops still shows as an error-correlated
bucket. Companion to eval_api_sap_accuracy.py / decompose_api_cost_error.py.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-07 20:38:19 +00:00
Khalim Conn-Kowlessar
98f71d2554 feat(diag): per-component cost decomposition for API SAP errors
Mirror of eval_api_sap_accuracy.py that decomposes each cert's SAP error
into per-component energy/cost deltas WITHOUT generating an Elmhurst
worksheet. Calibrates the consumer price from the certs we already get
right (gas £0.0809/kWh n=291, elec £0.2839/kWh n=326 over |SAP err|<0.4),
then for every cert compares our_component_kWh × price to the lodged
heating_cost_current / hot_water_cost_current / lighting_cost_current and
back-calculates a numeric energy target (lodged_cost / price).

Clusters errors by (component × direction). On the 905-cert sample this
reveals heat:high (we over-state heating energy → under-rate SAP) as the
dominant broken cluster: 332 certs, only 36.7% within 0.5. Output CSV at
<cache>/_cost_decomposition.csv.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-06 17:37:05 +00:00
Khalim Conn-Kowlessar
7e9231b36b fix(debug-tool): read the domain field names, not the schema ones
The first cut of elmhurst_input_sheet.py introspected the `schema`
dataclasses (rdsap_schema_*.py) but the mapper emits the `epc_property_data`
domain types, whose fields differ (wall_thickness_mm not wall_thickness;
total_floor_area_m2 not total_floor_area; frame_material not pvc_frame;
cylinder_insulation_thickness_mm; SapRoomInRoof has gable_*_length_m not
insulation/roof_room_connected). Worse, the getattr-with-None-default helper
printed None over real data, nearly sending a debug session chasing a
non-existent "dimensions dropped" mapper bug on cert 2100 (the dims map
fine; that cert's error is elsewhere). Switched to direct attribute access
so a future rename fails loudly, fixed every field name against the live
domain objects, and added roof_construction_type / floor_type for context.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-05 19:48:02 +00:00
Khalim Conn-Kowlessar
2c126b2a62 tooling(debug): add scripts/elmhurst_input_sheet.py worksheet-input dumper
Reconstructs the per-cert "Elmhurst SAP input sheet" generator that the
API-accuracy debugging loop relied on (the worked example survives at
'sap worksheets/golden fixture debugging/6035_elmhurst_input_sheet.md'); the
original was a throwaway and never committed. Companion to
eval_api_sap_accuracy.py: once that names a worst-offender cert, this dumps
the codes the mapper hands the calculator (from_api_response → EpcPropertyData)
in the 6035 layout — header, lodged element descriptions, building parts +
dimensions, windows, doors/heating/water/vent — plus the lodged reference
outputs and OUR continuous SAP next to the lodged value, to read side-by-side
with the Elmhurst Summary / P960 worksheet PDF.

Reads the fetch_2026_epc_sample.py cache (EPC_SAMPLE_CACHE, default
/tmp/epc_2026_sample). `--out-dir` writes <cert>_elmhurst_input_sheet.md.
Pyright strict clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-05 19:39:31 +00:00
Daniel Roth
641420be00 [UNRELATED] update sero address list for sharepoint file renaming 2026-06-05 14:22:09 +00:00
Khalim Conn-Kowlessar
8323d9cf07 Merge branch 'feature/per-cert-mapper-validation' of https://github.com/Hestia-Homes/Model into feature/bill-derivation 2026-06-05 09:38:40 +00:00
Khalim Conn-Kowlessar
3b442f9606 scripts: promote the API SAP-accuracy toolkit from /tmp
Three reusable scripts (each with a purpose/usage docstring) for wide-scale
testing of the calculator's API front-end against the GOV.UK EPB register —
the toolkit behind the 1000-cert study (docs/HANDOVER_API_SAMPLE_ACCURACY.md):

  fetch_2026_epc_sample.py    — sample cert numbers across a date window
                                (random pages) + download full schema-21 JSON
                                to a cache; resumable, 429/5xx backoff.
  eval_api_sap_accuracy.py    — % within 0.5 SAP, error histogram, worst-40,
                                and the mapper/calculator raise breakdown.
  analyse_api_sap_clusters.py — error grouped by property + heating type to
                                locate clusters (electric heating, flats, PV).

Cache dir defaults to /tmp/epc_2026_sample, overridable via EPC_SAMPLE_CACHE.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 16:52:09 +00:00
Khalim Conn-Kowlessar
afabfa0147 feat(modelling): sample a year from the EPC bulk export, offline-ready
fetch_epc_bulk_sample streams certificates-<year>.json out of the bulk ZIP via
range requests, keeps the first N SAP-version matches, and writes each cert's
inner document to <out>/<cert>.json for run_property_report. Stops after N, so
only the member prefix transfers, not the 15.7 GB archive (RangeFile.bytes_read
reports the true transfer vs the absolute ZIP offset). Verified on 2026: 100
SAP-10.2 certs -> report ran 81 scorable (MAE 2.03), 46 flagged, 19 raises
(11 full-SAP schema 19.1.0, 7 unmapped floor_construction 0/3, 1 missing
post_town) — real shadow-validation signal vs the curated golden 57.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 12:20:57 +00:00
Khalim Conn-Kowlessar
ea3af8d2f4 feat(modelling): CLI to fetch an EPC dump + build the inspection report
run_property_report builds the three-section Markdown+CSV report over a dir of
API-shaped EPC JSON, offline (defaults to the golden 57: 57/57 scorable, MAE
0.54, 6 flagged |Δ|>0.5). fetch_epc_dump pulls raw cert JSON from the live API
by --uprn/--postcode (picking the latest cert per match, skipping existing
files), mirroring fetch_cohort2's proven HTTP shape and reading
OPEN_EPC_API_TOKEN. Report artifacts + epc_dump/ are gitignored.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 11:26:17 +00:00
Khalim Conn-Kowlessar
8b5ab1c59e feat(modelling): turnkey offline cohort script (tables + CSV)
CertResult now carries its Plan (with flat baseline/post-SAP/measures
properties), and `format_cohort_csv` renders one browsable row per cert
(SAP transition, band, measures, cost, bill saving, valuation %, error).
`scripts/run_modelling_cohort.py` is turnkey: no args runs the committed
golden cohort, prints a sense-check table for the first measure-bearing
certs (a capped preview so a large dump doesn't flood the terminal), the
summary, and writes modelling_cohort.csv (gitignored). Point it at the
EPC dump when it lands.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 09:30:53 +00:00
Khalim Conn-Kowlessar
d8ef40c745 feat(modelling): offline cohort runner over an EPC-JSON dump
`harness.cohort.run_cohort(paths)` parses each API-shaped EPC JSON with
from_api_response and models it via run_modelling — no database, no
network — capturing per-cert errors instead of aborting the sweep, plus
`format_cohort_summary`. A thin `scripts/run_modelling_cohort.py` CLI
points it at a directory. Proven over the 57 golden API certs: 56 ran
offline, 15 produced measures, 1 errored (COAL has no Fuel Rates entry —
a BillDerivation coverage gap, not a harness one). Ready for the EPC dump.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 09:23:32 +00:00
Khalim Conn-Kowlessar
d7d5084f90 Move sap10_calculator tests to tests/domain/sap10_calculator/ for CI
The calculator tests lived under domain/sap10_calculator/{tests,worksheet/
tests,rdsap/tests,climate/tests,validation/tests}, none of which are in
pytest.ini testpaths — so CI (which collects tests/) never ran them. Relocate
all five dirs to tests/domain/sap10_calculator/{,worksheet,rdsap,climate,
validation}, mirroring the tests/domain/property_baseline/ convention, so the
cascade-pin / golden / e2e conformance suites run in CI.

Mechanics:
- git mv preserves history (110 files).
- Flattening the trailing /tests keeps each file's depth-to-repo-root
  identical, so all 16 repo-root parents[4] fixture refs stay valid. Only
  test_pcdb_etl.py's parents[1] (→ pcdb data) and one hardcoded absolute
  golden-fixture path in test_cert_to_inputs.py needed rebasing.
- Cross-imports rewritten domain.sap10_calculator.worksheet.tests →
  tests.domain.sap10_calculator.worksheet (21 files incl. the external
  importer backend/documents_parser/tests/test_summary_pdf_mapper_chain.py).
- Golden-fixture path strings in test_summary_pdf_mapper_chain.py +
  scripts/fetch_cohort2_api_jsons.py updated to the new location (the JSONs
  moved with the rdsap tests).

load_cells / gitignored worksheet xlsx: the xlsx-pinned tests (test_dimensions
/ ventilation / water_heating) read 2026-05-19-17-18 RdSap10Worksheet.xlsx,
which is gitignored (.gitignore `*.xlsx`) and so absent in CI. _xlsx_loader.
load_cells now pytest.skip()s when the file is absent, so those tests run
locally and skip cleanly in CI instead of erroring — no new CI failures from
the move, and the gitignore policy is respected.

Verified: tests/domain/sap10_calculator + backend/documents_parser +
tests/domain/property_baseline = 2248 pass, 1 skipped; pyright resolves the
new import paths with zero import-resolution errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-02 16:58:00 +00:00
Khalim Conn-Kowlessar
caee4de2f4 feat(ingestion): relocate EpcClientService to infrastructure + SolarRepo (#1133)
Move the EpcClientService package (client + _retry + exceptions + tests) from
the dying backend/ tree to infrastructure/epc_client/ as the New-EPC-API Fetcher;
update the two callers (address2UPRN, a script). All 14 client tests pass.

Add SolarRepository port + SolarPostgresRepository persisting Google Solar
building insights as JSONB (solar_building_insights table), one row per Property.
The EPC repo half of this slice already landed in #1129. pyright strict clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-30 19:45:26 +00:00
Khalim Conn-Kowlessar
22ae6f4d77 Slice S0380.39: bulk-fetch 38 cohort-2 EPC API JSONs for cross-mapper parity
Adds scripts/fetch_cohort2_api_jsons.py (throwaway one-off) plus 38
golden fixtures under domain/sap10_calculator/rdsap/tests/fixtures/golden/
covering every cert in "sap worksheets/additional with api 2/".

Each JSON is the inner `data` payload from the gov.uk EPB
/api/certificate endpoint — the same shape EpcPropertyDataMapper
.from_api_response consumes today.

Required prerequisite for Slice B (parametrized API-path chain test
that mirrors the cohort-2 Summary-path sweep at 1e-4 vs worksheet).
Per the cross-mapper-parity primitive: API EPC and Elmhurst EPC must
produce SAP within 1e-4 of each other and of the worksheet — the SAP
cascade is the load-bearing equivalence check.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 16:40:58 +00:00
Daniel Roth
9f7c16ccbd add address list 2026-05-21 15:30:03 +00:00