Commit graph

68 commits

Author SHA1 Message Date
Jun-te Kim
dbcdf29bd9 refactor(address2uprn): name the match/decision return types; rename helper
Address PR review (dancafc):
- introduce UprnMatch NamedTuple (datatypes/address_match.py) for the
  (uprn, address, lexiscore, certificate_number) return, replacing the bare
  4-tuple in get_uprn_from_epc_df / get_uprn_from_historic_epc /
  HistoricEpcResolver.resolve_uprn. Tuple-compatible, so unpacking is unchanged.
- rename get_uprn_with_epc_df -> get_uprn_from_epc_df (+ callers).
- type resolve_group_ambiguity via a GroupDecision NamedTuple and trim its
  docstring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 16:00:00 +00:00
Jun-te Kim
4e0134dd27 Merge branch 'feat/address2uprn-certificate-number' into feature/uprn-confirmation-before-finalise
# Conflicts:
#	backend/address2UPRN/main.py
2026-07-07 15:17:47 +00:00
Jun-te Kim
daa1cd7967 feat(address2uprn): withhold ambiguous cross-row UPRN matches (ADR-0057)
Phase 1 of confirming UPRNs before finalise. address2uprn matched each
row independently, so one UPRN could be the best match for two distinct
addresses (a coarse EPC record absorbing several real addresses, e.g.
flats in a block). Those distinct addresses were then silently merged by
the property identity insert, and collided in property_overrides.

resolve_group_ambiguity() withholds a UPRN claimed by >=2 distinct
normalised addresses within a postcode group (keeps genuine same-address
re-listings), and the handler now emits an address2uprn_status column
(matched | ambiguous_duplicate | unmatched | invalid_postcode | error).
Withheld rows drop to a null UPRN but keep their lexiscore for triage on
the (upcoming) confirmation page.

Also adds the ADR-0057 backstop dedup in property_overrides upsert_all so
the ON CONFLICT statement can never double-touch a row.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 15:15:59 +00:00
Jun-te Kim
3718743801 Carry the EPC certificate number through address2uprn to property
EpcClientService.search_by_postcode already returns the matched
certificate number alongside the UPRN, but it was dropped before
persistence. Thread it through get_epc_data_with_postcode ->
get_uprn_with_epc_df / get_uprn_from_historic_epc (using the historic
dataset's lmk_key) -> the address2uprn_certificate_number result
column -> PropertyIdentityInsert -> the property table's new
certificate_number column (assessment-model PR #362).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 13:26:49 +00:00
Jun-te Kim
7aea692521 historic EPC: read via infrastructure/s3, not the utils.s3 utility
HistoricEpcS3Repository reached into utils/s3.py (read_csv_gz_from_s3 +
parse_s3_uri), the legacy utility that self-constructs boto3 inside free
functions. The other S3 repositories deliberately depend on the
infrastructure/s3 layer instead (UnstandardisedAddressListCsvS3Repository
injects a CsvS3Client). Bring historic EPC into line.

- Add GzipCsvS3Client(S3Client) in infrastructure/s3: read_csv_gz(key) ->
  DataFrame (get_object + gzip decode).
- Inject it into HistoricEpcS3Repository; the bucket lives in the client and
  the repo only builds the per-postcode key + maps rows (no S3/HTTP code).
  Add with_default_s3_client(s3_root) for composition roots.
- Update main.py and the match_addresses_for_postcode seam to the factory.
- Repo tests inject a real GzipCsvS3Client over a controlled boto stub
  (exact key assertions + AccessDenied); add a moto-based client test and a
  factory test covering s3_root -> bucket+key.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQE5TsSuQTeNSCSz9A9GQf
2026-06-30 09:19:57 +00:00
Jun-te Kim
fa6357a5cf Re-plug historic EPC lookup onto the repository; a missing postcode now yields empty matches 🟩
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 14:58:23 +00:00
Jun-te Kim
e60ca6ee5d source of the problem in address2uprn 2026-06-05 19:03:33 +00:00
Khalim Conn-Kowlessar
caee4de2f4 feat(ingestion): relocate EpcClientService to infrastructure + SolarRepo (#1133)
Move the EpcClientService package (client + _retry + exceptions + tests) from
the dying backend/ tree to infrastructure/epc_client/ as the New-EPC-API Fetcher;
update the two callers (address2UPRN, a script). All 14 client tests pass.

Add SolarRepository port + SolarPostgresRepository persisting Google Solar
building insights as JSONB (solar_building_insights table), one row per Property.
The EPC repo half of this slice already landed in #1129. pyright strict clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-30 19:45:26 +00:00
Jun-te Kim
dfc100f78b rank address similiarity 2026-05-12 16:02:01 +00:00
Jun-te Kim
27f2ef5e83 get rid of duplicate function and make better sensible variable name 2026-05-12 13:46:02 +00:00
Jun-te Kim
35fea20fc7 changed function name 2026-05-12 10:54:45 +00:00
Jun-te Kim
bec5c4f3c3 one place to have df_has_single_uprn 2026-05-12 10:51:27 +00:00
Jun-te Kim
b364df89ad forgot to add tuple typing 2026-05-12 10:31:54 +00:00
Jun-te Kim
8635e2a1aa change file name of epc client service 2026-05-12 10:08:00 +00:00
Jun-te Kim
2c5c8337cc added more type hints 2026-05-12 10:01:25 +00:00
Jun-te Kim
e06ead55d0 add more type hint 2026-05-12 09:48:21 +00:00
Jun-te Kim
b72d5fbf42 fix nitpick 2026-05-12 09:43:40 +00:00
Jun-te Kim
c22528299c added type hinting to uprn 2026-05-12 09:40:12 +00:00
Jun-te Kim
5edae06a65 added imports at the top of the file instead of function 2026-05-12 09:37:23 +00:00
Jun-te Kim
9aae5bf482 added logic to deal with flats 2026-05-11 15:20:17 +00:00
Khalim Conn-Kowlessar
8ec6eecc4d reverting manually tweaked tests 2026-04-28 12:00:19 +00:00
Khalim Conn-Kowlessar
0955862973 working on integrating new EPC api into address2UPRN 2026-04-27 11:32:44 +00:00
Khalim Conn-Kowlessar
3ed25030d4 added new api call for new epc api 2026-04-25 22:17:38 +00:00
Jun-te Kim
d7b8ca34bf made everything complete not compelted 2026-04-21 20:37:34 +00:00
Jun-te Kim
98c9a1df74 added fix for utf 2026-04-13 15:15:42 +00:00
Jun-te Kim
17b8f22840 fix BOM issue 2026-04-13 14:59:00 +00:00
Jun-te Kim
53ef72faab add change and put it to production 2026-04-02 13:05:16 +00:00
Jun-te Kim
147982cb7c optional none 2026-03-09 13:28:16 +00:00
Jun-te Kim
4d013f3295 dan's pr 2026-03-09 13:23:20 +00:00
Jun-te Kim
071a67e501 ordnancesurvey deployment 2026-03-06 13:29:25 +00:00
Jun-te Kim
815ce01082 lambda code works locally 2026-03-06 12:51:08 +00:00
Jun-te Kim
1b3a942c30 ordance survey logic basically finsihed 2026-03-04 16:58:23 +00:00
Jun-te Kim
db251c1857 removed duplicate code 2026-03-04 16:14:27 +00:00
0a87ba786c local run stuff 2026-02-16 14:14:01 +00:00
61377497ff get rid of unneccsary variable declartion 2026-02-16 14:07:23 +00:00
ed8d562917 added logger and verbose 2026-02-16 13:49:49 +00:00
62a8f543f6 get rid of comments 2026-02-16 13:04:27 +00:00
c1f784b87f address 2uprn and postcode splitter works locally 2026-02-16 12:13:16 +00:00
8e574c2401 post code splitter works 2026-02-13 18:30:47 +00:00
8152dc5166 deploy with new address2uprn handling 2026-02-13 11:15:15 +00:00
c2f29e86df made tests pass and redploy 2026-02-12 15:11:20 +00:00
3bdd4a4a97 test first with just 5 2026-02-12 15:03:38 +00:00
958ab72e0a deploy to main with new policy 2026-02-12 12:24:47 +00:00
c9ec097a43 pr review 2026-02-11 17:55:43 +00:00
b1164ffd90 get rid of local 2026-02-11 17:50:47 +00:00
e7691570fd merge 2026-02-11 17:42:30 +00:00
a7509aecdc added very serious logs 2026-02-11 16:59:57 +00:00
538f207d2f env variables added 2026-02-11 16:57:27 +00:00
762dccde01 run this end to end 2026-02-11 16:44:08 +00:00
9b414924d0 run this end to end 2026-02-11 16:37:55 +00:00