Commit graph

7193 commits

Author SHA1 Message Date
Jun-te Kim
7aea692521 historic EPC: read via infrastructure/s3, not the utils.s3 utility
HistoricEpcS3Repository reached into utils/s3.py (read_csv_gz_from_s3 +
parse_s3_uri), the legacy utility that self-constructs boto3 inside free
functions. The other S3 repositories deliberately depend on the
infrastructure/s3 layer instead (UnstandardisedAddressListCsvS3Repository
injects a CsvS3Client). Bring historic EPC into line.

- Add GzipCsvS3Client(S3Client) in infrastructure/s3: read_csv_gz(key) ->
  DataFrame (get_object + gzip decode).
- Inject it into HistoricEpcS3Repository; the bucket lives in the client and
  the repo only builds the per-postcode key + maps rows (no S3/HTTP code).
  Add with_default_s3_client(s3_root) for composition roots.
- Update main.py and the match_addresses_for_postcode seam to the factory.
- Repo tests inject a real GzipCsvS3Client over a controlled boto stub
  (exact key assertions + AccessDenied); add a moto-based client test and a
  factory test covering s3_root -> bucket+key.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQE5TsSuQTeNSCSz9A9GQf
2026-06-30 09:19:57 +00:00
Jun-te Kim
1ebced2c42 Merge remote-tracking branch 'origin/main' into feature/historic-epc-repository 2026-06-29 15:52:05 +00:00
Jun-te Kim
cc4b40fd95 Package repositories/ into the address2UPRN lambda image so it imports at cold start
main.py imports repositories.historic_epc.* after the re-plug; without the COPY
the lambda fails at init with Runtime.ImportModuleError (caught by
test_lambda_image_copies_full_import_closure).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 15:46:35 +00:00
Daniel Roth
6c42065304
Merge pull request #1358 from Hestia-Homes/feature/batch-save-and-delete-epc
Batch saving and deleting of epc property data, bulk saving of plans
2026-06-29 16:46:19 +01:00
Daniel Roth
c202f15d8c fix broken unit tests 2026-06-29 15:40:22 +00:00
Jun-te Kim
0536f70162 Take a Postcode value object at the historic EPC repository boundary 🟩
The port accepts a normalised Postcode and rejects malformed/empty ones via
Postcode.is_valid() (PostcodeNotFound) — dropping the per-lookup postcodes.io
HTTP call and the cross-module use of the private _sanitise_postcode. Mapper
helper promoted to public.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 15:24:18 +00:00
Jun-te Kim
23dfb3f899 Add UK-postcode format validity to the Postcode value object 🟩
A pure, offline regex check (no network) so a frozen value object stays pure.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 15:24:18 +00:00
Jun-te Kim
d732ec3fa3
Merge pull request #1357 from Hestia-Homes/chore/devcontainer-pyright
Add pyright + pandas-stubs to backend devcontainer
2026-06-29 16:14:39 +01:00
Jun-te Kim
c2e6b88744 Add pyright + pandas-stubs to backend devcontainer for the strict type gate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 15:10:40 +00:00
Daniel Roth
4764bc7c15 Batch plan saves reduce RDS CPU during bulk modelling runs 🟪
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 15:08:47 +00:00
Daniel Roth
46ca714ef9 Batch plan saves reduce RDS CPU during bulk modelling runs 🟩
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 15:06:47 +00:00
Daniel Roth
9c6b477025 Batch plan saves reduce RDS CPU during bulk modelling runs 🟥
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 15:04:54 +00:00
Jun-te Kim
fa6357a5cf Re-plug historic EPC lookup onto the repository; a missing postcode now yields empty matches 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:58:23 +00:00
Jun-te Kim
03687041c9 Return no historic UPRN for missing, ambiguous, or zero-score matches 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:52:42 +00:00
Jun-te Kim
b47c90f9cd Resolve an address to its unambiguous historic EPC UPRN 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:52:20 +00:00
Jun-te Kim
b8c376bfd6 Resolve an address to its unambiguous historic EPC UPRN 🟥
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:52:06 +00:00
Jun-te Kim
a30ab59f3c Compose repository and matcher into scored historic EPC matches 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:51:45 +00:00
Jun-te Kim
d7fd093c4f Compose repository and matcher into scored historic EPC matches 🟥
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:51:33 +00:00
Jun-te Kim
d084a0a785 Rank historic EPC records by address similarity 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:50:03 +00:00
Jun-te Kim
d41eb84530 Rank historic EPC records by address similarity 🟥
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:49:49 +00:00
Jun-te Kim
b059d55dd2 Strip trailing .0 from historic EPC UPRN 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:49:08 +00:00
Jun-te Kim
be2b71c235 Strip trailing .0 from historic EPC UPRN 🟥
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:48:46 +00:00
Jun-te Kim
66b776e0aa Raise PostcodeNotFound for an unusable postcode 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:48:06 +00:00
Jun-te Kim
49becf1ada Raise PostcodeNotFound for an unusable postcode 🟥
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:47:50 +00:00
Jun-te Kim
cd503411d2 Build S3 key from sanitised postcode and propagate non-missing read errors 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:47:18 +00:00
Jun-te Kim
88f4739ce8 Return empty list for a postcode with no historic EPC shard 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:46:42 +00:00
Jun-te Kim
6a095a2cf2 Return empty list for a postcode with no historic EPC shard 🟥
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:46:22 +00:00
Jun-te Kim
a372f2995f Map historic EPC S3 shard to domain records 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:45:25 +00:00
Jun-te Kim
74b8ad86b1 Map historic EPC S3 shard to domain records 🟥
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:44:56 +00:00
Daniel Roth
9f0dd067f8 Merge branch 'main' into feature/batch-save-and-delete-epc 2026-06-29 14:38:57 +00:00
Daniel Roth
597510b996
Merge pull request #1353 from Hestia-Homes/feature/dont-resave-epc-if-refetch-false
Don't re-save EPC data which has been read from database rather than fetched. Also turn on database insights
2026-06-29 15:14:14 +01:00
Daniel Roth
0718533d3a turn on rds performance insights 2026-06-29 14:09:24 +00:00
KhalimCK
8b0b5c0387
Merge pull request #1352 from Hestia-Homes/fix/ground-floor-flat-default-heating
fix(sap): credit PV only when connected to the dwelling's meter
2026-06-29 15:06:29 +01:00
Daniel Roth
b1406bb482 Skip re-saving EPC data when it was read from DB unchanged
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 14:03:23 +00:00
Khalim Conn-Kowlessar
08d9e46481 docs(pv): document gov-API pv_connection enum on the domain field 🟪
0 = no PV, 1 = not connected to dwelling meter, 2 = connected. Decoded
empirically from the corpus + accredited Elmhurst (the gov RdSAP schema
types it as a bare int). Points readers to the crediting gate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 13:50:01 +00:00
Khalim Conn-Kowlessar
9927169e23 test(accuracy): ratchet SAP gauge after PV connection gate 🟪
within-0.5 floor 0.73->0.74 (now 0.741), MAE ceiling 0.762->0.740 (now
0.7397) on the fixed RdSAP-21.0.1 corpus. Log entry appended.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 13:43:30 +00:00
Khalim Conn-Kowlessar
8606cab5f0 fix(pv): credit PV only when connected to the dwelling's meter 🟩
Gate PV generation/credit in cert_to_inputs on gov-API pv_connection:
credit only when ==2 ('connected'); ==1 ('present but not connected to the
dwelling's meter') contributes zero to the dwelling's cost/CO2/PE per
RdSAP 10 §11.1 / SAP 10.2 Appendix M. Non-int (None / site-notes str) keeps
the credit-if-array behaviour, so the Elmhurst/Summary + synthetic paths are
unchanged (no regression).

Corpus: all 5 pv_connection=1 PV certs move inside ±0.5 (e.g. 100051118081
+6.5→+0.5); MAE 0.760→0.740, within-0.5 73.8→74.3%, no regression
(pv_connection=2 certs keep their credit).

Also corrects a now-load-bearing latent bug: the solar-recommendation
overlay tagged recommended arrays pv_connection=1 ('not connected') — which
the new gate would zero. A new install connects to the dwelling's meter, so
it must be 2; pinned by the overlay test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 13:40:51 +00:00
Daniel Roth
b426940136
Merge pull request #1350 from Hestia-Homes/modelling-e2e-12-concurrency
Decrease e2e modelling concurrency from 32 to 12
2026-06-29 14:36:01 +01:00
Daniel Roth
66fa08af19 update comment 2026-06-29 13:33:48 +00:00
Daniel Roth
71b304fa47 decrease lambda concurrency from 32 to 12 2026-06-29 13:32:53 +00:00
Khalim Conn-Kowlessar
775232b4e7 test(pv): credit PV only when connected to dwelling meter (pv_connection) 🟥
RdSAP 10 §11.1 / SAP 10.2 Appendix M: PV is included in a dwelling's
assessment only if connected to the dwelling's own electricity meter. The
gov-API pv_connection enum encodes this — 0=no PV, 1=present-but-not-
connected, 2=connected. Corpus-validated (57 PV certs: pv_connection=1 MAE
4.48->1.22 without credit, 0/5 need it; pv_connection=2 needs it, MAE 0.98
vs 10.29) and Elmhurst-proven (connected SAP 87 vs not-connected 74).

cert_to_inputs currently credits a pv_connection=1 array; the test pins that
it must contribute zero generation. Adds pv_connection to make_minimal_sap10_epc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 13:32:20 +00:00
Daniel Roth
f27d1e21bb Batch EPC writes in EpcPostgresRepository pass pyright strict 🟪
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 13:17:29 +00:00
Daniel Roth
0fa1b9001c Batch EPC writes in _flush_writes: two save_batch() calls instead of N save() calls 🟩
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 13:09:56 +00:00
Daniel Roth
587465bff7 Batch EPC writes via save_batch() on EpcPostgresRepository 🟩
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 13:04:30 +00:00
Daniel Roth
fe69bccf22 Batch EPC writes via save_batch() on EpcPostgresRepository 🟥
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 12:19:49 +00:00
KhalimCK
98ba8d89dc
Merge pull request #1343 from Hestia-Homes/fix/effective-baseline-overrides-and-epc-fidelity
fix(baseline): honour EPC + overrides as the authoritative Effective baseline
2026-06-29 11:19:52 +01:00
KhalimCK
29a0cc7f47
Merge pull request #1344 from Hestia-Homes/tooling/modelling-anomaly-audit
feat(audit): pluggable modelling-anomaly audit + audit-ara-portfolio skill
2026-06-29 11:18:26 +01:00
KhalimCK
31e8b93ea0
Merge pull request #1345 from Hestia-Homes/fix/storage-heater-space-heating-demand
fix(sap): expose roof for mislabelled top-floor flats via roof_insulation_location
2026-06-29 11:18:03 +01:00
KhalimCK
d55f520f83
Merge pull request #1346 from Hestia-Homes/fix/elmhurst-summary-roof-window-classification
fix(elmhurst): party-ceiling Main BP window is vertical, not a rooflight
2026-06-29 11:17:49 +01:00
Khalim Conn-Kowlessar
02ef67fd8f chore(debug): summary_to_sap.py — Elmhurst Summary PDF -> our SAP + trail 🟪
Runs a Summary PDF through the chain-test path (pdftotext ->
ElmhurstSiteNotesExtractor -> from_elmhurst_site_notes) into Sap10Calculator
and dumps SAP + per-end-use kWh + the intermediate worksheet trail, for
diffing our calc against the accompanying Elmhurst U985 worksheet PDF.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 21:26:30 +00:00