Commit graph

111 commits

Author SHA1 Message Date
Jun-te Kim
dbcdf29bd9 refactor(address2uprn): name the match/decision return types; rename helper
Address PR review (dancafc):
- introduce UprnMatch NamedTuple (datatypes/address_match.py) for the
  (uprn, address, lexiscore, certificate_number) return, replacing the bare
  4-tuple in get_uprn_from_epc_df / get_uprn_from_historic_epc /
  HistoricEpcResolver.resolve_uprn. Tuple-compatible, so unpacking is unchanged.
- rename get_uprn_with_epc_df -> get_uprn_from_epc_df (+ callers).
- type resolve_group_ambiguity via a GroupDecision NamedTuple and trim its
  docstring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 16:00:00 +00:00
Jun-te Kim
4e0134dd27 Merge branch 'feat/address2uprn-certificate-number' into feature/uprn-confirmation-before-finalise
# Conflicts:
#	backend/address2UPRN/main.py
2026-07-07 15:17:47 +00:00
Jun-te Kim
daa1cd7967 feat(address2uprn): withhold ambiguous cross-row UPRN matches (ADR-0057)
Phase 1 of confirming UPRNs before finalise. address2uprn matched each
row independently, so one UPRN could be the best match for two distinct
addresses (a coarse EPC record absorbing several real addresses, e.g.
flats in a block). Those distinct addresses were then silently merged by
the property identity insert, and collided in property_overrides.

resolve_group_ambiguity() withholds a UPRN claimed by >=2 distinct
normalised addresses within a postcode group (keeps genuine same-address
re-listings), and the handler now emits an address2uprn_status column
(matched | ambiguous_duplicate | unmatched | invalid_postcode | error).
Withheld rows drop to a null UPRN but keep their lexiscore for triage on
the (upcoming) confirmation page.

Also adds the ADR-0057 backstop dedup in property_overrides upsert_all so
the ON CONFLICT statement can never double-touch a row.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 15:15:59 +00:00
Jun-te Kim
3718743801 Carry the EPC certificate number through address2uprn to property
EpcClientService.search_by_postcode already returns the matched
certificate number alongside the UPRN, but it was dropped before
persistence. Thread it through get_epc_data_with_postcode ->
get_uprn_with_epc_df / get_uprn_from_historic_epc (using the historic
dataset's lmk_key) -> the address2uprn_certificate_number result
column -> PropertyIdentityInsert -> the property table's new
certificate_number column (assessment-model PR #362).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 13:26:49 +00:00
Jun-te Kim
7aea692521 historic EPC: read via infrastructure/s3, not the utils.s3 utility
HistoricEpcS3Repository reached into utils/s3.py (read_csv_gz_from_s3 +
parse_s3_uri), the legacy utility that self-constructs boto3 inside free
functions. The other S3 repositories deliberately depend on the
infrastructure/s3 layer instead (UnstandardisedAddressListCsvS3Repository
injects a CsvS3Client). Bring historic EPC into line.

- Add GzipCsvS3Client(S3Client) in infrastructure/s3: read_csv_gz(key) ->
  DataFrame (get_object + gzip decode).
- Inject it into HistoricEpcS3Repository; the bucket lives in the client and
  the repo only builds the per-postcode key + maps rows (no S3/HTTP code).
  Add with_default_s3_client(s3_root) for composition roots.
- Update main.py and the match_addresses_for_postcode seam to the factory.
- Repo tests inject a real GzipCsvS3Client over a controlled boto stub
  (exact key assertions + AccessDenied); add a moto-based client test and a
  factory test covering s3_root -> bucket+key.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQE5TsSuQTeNSCSz9A9GQf
2026-06-30 09:19:57 +00:00
Jun-te Kim
cc4b40fd95 Package repositories/ into the address2UPRN lambda image so it imports at cold start
main.py imports repositories.historic_epc.* after the re-plug; without the COPY
the lambda fails at init with Runtime.ImportModuleError (caught by
test_lambda_image_copies_full_import_closure).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 15:46:35 +00:00
Jun-te Kim
fa6357a5cf Re-plug historic EPC lookup onto the repository; a missing postcode now yields empty matches 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:58:23 +00:00
Jun-te Kim
0159176772 python upgraded due to enum 2026-06-12 15:47:28 +00:00
Daniel Roth
d571ccaee5 remove number of address2uprn tests 2026-06-10 16:26:54 +00:00
Jun-te Kim
e60ca6ee5d source of the problem in address2uprn 2026-06-05 19:03:33 +00:00
Jun-te Kim
4e02eb7c77 more tests to ensure we don't deploy something that is brokern 2026-06-02 15:03:20 +00:00
Jun-te Kim
144233a5f3 backend was missing a dependency 2026-06-02 10:46:29 +00:00
Khalim Conn-Kowlessar
5a3be9d672 feat(ingestion): relocate EpcClientService to infrastructure + SolarRepo (#1133)
Move the EpcClientService package (client + _retry + exceptions + tests) from
the dying backend/ tree to infrastructure/epc_client/ as the New-EPC-API Fetcher;
update the two callers (address2UPRN, a script). All 14 client tests pass.

Add SolarRepository port + SolarPostgresRepository persisting Google Solar
building insights as JSONB (solar_building_insights table), one row per Property.
The EPC repo half of this slice already landed in #1129. pyright strict clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-01 16:28:48 +00:00
Jun-te Kim
53b211e951 epc token added 2026-05-20 15:43:41 +00:00
Jun-te Kim
914a8ed51e postcode splliter working e2e 2026-05-20 11:07:40 +00:00
Jun-te Kim
fce1e1008a added more test cases 2026-05-15 16:00:02 +00:00
Jun-te Kim
6afd076005 added 5 second rest every 100 tests 2026-05-15 11:28:04 +00:00
Jun-te Kim
2fb6a99956 throttle added 2026-05-13 14:02:36 +00:00
Jun-te Kim
dfc100f78b rank address similiarity 2026-05-12 16:02:01 +00:00
Jun-te Kim
27f2ef5e83 get rid of duplicate function and make better sensible variable name 2026-05-12 13:46:02 +00:00
Jun-te Kim
5cd21d8522 get rid of khalim's json 2026-05-12 12:55:50 +00:00
Jun-te Kim
18ea95b67d added env variables for boto 2026-05-12 12:34:17 +00:00
Jun-te Kim
35fea20fc7 changed function name 2026-05-12 10:54:45 +00:00
Jun-te Kim
bec5c4f3c3 one place to have df_has_single_uprn 2026-05-12 10:51:27 +00:00
Jun-te Kim
b364df89ad forgot to add tuple typing 2026-05-12 10:31:54 +00:00
Jun-te Kim
8635e2a1aa change file name of epc client service 2026-05-12 10:08:00 +00:00
Jun-te Kim
2c5c8337cc added more type hints 2026-05-12 10:01:25 +00:00
Jun-te Kim
e06ead55d0 add more type hint 2026-05-12 09:48:21 +00:00
Jun-te Kim
b72d5fbf42 fix nitpick 2026-05-12 09:43:40 +00:00
Jun-te Kim
c22528299c added type hinting to uprn 2026-05-12 09:40:12 +00:00
Jun-te Kim
5edae06a65 added imports at the top of the file instead of function 2026-05-12 09:37:23 +00:00
Jun-te Kim
1934c889b0 refactored test to deal with flats better 2026-05-11 16:23:03 +00:00
Jun-te Kim
9aae5bf482 added logic to deal with flats 2026-05-11 15:20:17 +00:00
Jun-te Kim
6504785e7c merged from main 2026-05-11 12:30:29 +00:00
Jun-te Kim
c9c43f178c demo generated for use in address2uprn 2026-05-08 14:48:15 +00:00
Khalim Conn-Kowlessar
8ec6eecc4d reverting manually tweaked tests 2026-04-28 12:00:19 +00:00
Khalim Conn-Kowlessar
8f2885474b fixing address2uprn tests 2026-04-28 11:53:52 +00:00
Khalim Conn-Kowlessar
1af6bc6748 creating lodgment dates data, using old EPC api, to verify test failures 2026-04-27 12:15:30 +00:00
Khalim Conn-Kowlessar
0955862973 working on integrating new EPC api into address2UPRN 2026-04-27 11:32:44 +00:00
Khalim Conn-Kowlessar
3ed25030d4 added new api call for new epc api 2026-04-25 22:17:38 +00:00
Jun-te Kim
d7b8ca34bf made everything complete not compelted 2026-04-21 20:37:34 +00:00
Jun-te Kim
98c9a1df74 added fix for utf 2026-04-13 15:15:42 +00:00
Jun-te Kim
17b8f22840 fix BOM issue 2026-04-13 14:59:00 +00:00
Jun-te Kim
d730a90246 added coordination comments 2026-04-08 14:47:31 +00:00
Jun-te Kim
23d7b22b54 save for easter weekend 2026-04-02 17:30:38 +00:00
Jun-te Kim
53ef72faab add change and put it to production 2026-04-02 13:05:16 +00:00
Jun-te Kim
e01b7225bb save 2026-03-24 13:01:56 +00:00
Jun-te Kim
f69a615140 revert to old one 2026-03-17 17:13:49 +00:00
Jun-te Kim
6bfeeeb1b1 go back to origional 2026-03-17 13:14:16 +00:00
Jun-te Kim
1b53b47048 add this in a sensible branch 2026-03-17 12:37:50 +00:00