7aea692521
HistoricEpcS3Repository reached into utils/s3.py (read_csv_gz_from_s3 + parse_s3_uri), the legacy utility that self-constructs boto3 inside free functions. The other S3 repositories deliberately depend on the infrastructure/s3 layer instead (UnstandardisedAddressListCsvS3Repository injects a CsvS3Client). Bring historic EPC into line. - Add GzipCsvS3Client(S3Client) in infrastructure/s3: read_csv_gz(key) -> DataFrame (get_object + gzip decode). - Inject it into HistoricEpcS3Repository; the bucket lives in the client and the repo only builds the per-postcode key + maps rows (no S3/HTTP code). Add with_default_s3_client(s3_root) for composition roots. - Update main.py and the match_addresses_for_postcode seam to the factory. - Repo tests inject a real GzipCsvS3Client over a controlled boto stub (exact key assertions + AccessDenied); add a moto-based client test and a factory test covering s3_root -> bucket+key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQE5TsSuQTeNSCSz9A9GQf |
||
|---|---|---|
| .. | ||
| handler | ||
| tests | ||
| __init__.py | ||
| main.py | ||
| README.md | ||
| scoring.py | ||
| script.py | ||
So you want to fetch UPRN for an address list?
Before you run:
Step 1) Get the list and ensure the following columns exists
I believe lower and upper case matter:
- Address 1
- Address 2
- Address 3
- postcode
And save it as a .csv file
Step 2)
Before we run this, we need to upload it into S3 as well as put initiate a subtask + task
- S3 upload I'll recommend somewhere in retrofit-data-dev and get the s3_uri
For this example I'll be using "s3://retrofit-data-dev/ara_raw_inputs/calico/Calico Homes Full list EPC Properties(Sheet2) (1) (1).csv"
Go to Ara DB and make a new task_id with a randomly generated uuid as the primarily key
task_id = ea615ac3-ac28-46c4-8bff-2431c5b9c13d sub_task_id = 85a23b67-8f18-4299-9bf0-69bfb87adbc7 s3 => s3://retrofit-data-dev/ara_raw_inputs/eon/North Tyneside Council.csv
Step 3) Alright, now lets make the input for postcode-splitter sqs to get the ball rolling postcode-splitter-sqs => https://eu-west-2.console.aws.amazon.com/sqs/v3/home?region=eu-west-2#/queues/https%3A%2F%2Fsqs.eu-west-2.amazonaws.com%2F337213553626%2Fpostcode-splitter-queue-dev
{ "sub_task_id": "c5afbd49-f0cd-4930-82bf-bafc5243a34a", "task_id": "67a4b3f0-cc7a-4e8a-b314-deb783e0eedb", "s3_uri": "s3://retrofit-data-dev/ara_raw_inputs/eon/pickering ferens/Pickering Ferens - SHDF W3 Post Bid Stage MDS - Template - Vr4(in).csv" }
Each batch of csv should be saved in retrofit-data-dev/ara_postcode_splitter_batches///timestamp:uuid4.csv
outputs of address2uprn ( which is automatically triggered on postcodesplitter) will be saved on retrofit-data-dev/ara_raw_outputs///timestamp:uuid4.csv
Run the script in backend/scripts/combine_address2uprn_outputs.py with . This will combine all the outputs of the files for each address2uprn into one big file
Find out which ones have missing uprn and save that as a seperate sheet and save it somewhere in s3://retrofit-data-dev
I uploaded the missing uprn here: s3://retrofit-data-dev/ara_raw_inputs/calico/missinguprn.csv
ordnance_survey sqs is => https://eu-west-2.console.aws.amazon.com/sqs/v3/home?region=eu-west-2#/queues/https%3A%2F%2Fsqs.eu-west-2.amazonaws.com%2F337213553626%2FordnanceSurvey-queue-dev
{ "s3_uri": "s3://retrofit-data-dev/ara_raw_inputs/eon/beyond_housing/Book(Sheet1).csv", "task_id": "ccdec0d1-ebf3-484f-b2ae-397200dd25da", "sub_task_id": "569d41f6-45cd-4e64-a586-eb8c2097375d" }
outputs are at s3://retrofit-data-dev/ara_ordnance_survey_outputs//