mirror of
https://github.com/Hestia-Homes/Model.git
synced 2026-07-19 08:53:17 +00:00
9ec7987e97
Addresses PR #1498 review: - A per-file read failure (missing/deleted object, or a bucket the role can't reach) is now a SkippedDocument(reason=unreadable), not a whole-run abort; an all-unreadable selection still fails (no empty package). Best-effort on the read path (ADR-0060). - Each ZIP member is streamed to a temp file (S3DocumentDownloader.download, boto download_file) and added from disk, so a single multi-GB member never hits the heap — the 'never held whole in memory' claim is now true. - The 409 double-submit guard is DB-arbitrated: the sub_task insert is conditional on the task having none, so a race creates only one. - Local env gets a placeholder recipient email so the route is exercisable. - PackageEntry carries landlord_property_id so a read failure can be reported. - TODO noting the UploadedFile.s3_upload_timestamp typing fix. Two new tests: per-file read failure skips-and-reports; address fallback flows to the folder name. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
20 lines
754 B
Python
20 lines
754 B
Python
"""Streams uploaded documents from arbitrary S3 buckets to local disk (ADR-0060).
|
|
|
|
Uploaded files live across many source buckets (each `uploaded_files` row
|
|
carries its own `s3_file_bucket`), so — unlike the bucket-bound `S3Client` —
|
|
this takes the bucket per call. Downloading to a file (rather than reading the
|
|
object into memory) keeps a single multi-GB member off the heap when it is added
|
|
to the on-disk ZIP.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Any
|
|
|
|
|
|
class S3DocumentDownloader:
|
|
def __init__(self, boto_s3_client: Any) -> None:
|
|
self._client = boto_s3_client
|
|
|
|
def download(self, bucket: str, key: str, dest_path: str) -> None:
|
|
self._client.download_file(Bucket=bucket, Key=key, Filename=dest_path)
|