junte/Model
1
0
Fork 0
mirror of https://github.com/Hestia-Homes/Model.git synced 2026-07-12 13:29:04 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

ADR-0060: dedicated exports bucket + multi-GB streaming (drop DATA_BUCKET lifecycle) 🟪

Browse source 
Per review: packages go to a separate DOCUMENT_EXPORTS_BUCKET (not DATA_BUCKET),
no 7-day lifecycle imposed on the shared data bucket; packages can be several GB
so they stream via /tmp + multipart upload with raised ephemeral storage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Khalim Conn-Kowlessar 2026-07-08 09:52:43 +00:00
parent 8c0b015923
commit 6db6bbd98f
2 changed files with 23 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
CONTEXT.md
View file

@ -104,7 +104,7 @@ The category of an uploaded document — the `uploaded_files.file_type` enum (ph
_Avoid_: document category, file kind, doc class
**Download Package**:
The single ZIP archive a **Bulk Document Download** produces: one folder per property (named by human-readable address, enriched from the hubspot deals data, with `landlord_property_id` appended for uniqueness), each folder holding the latest file of each **Document Type** held for that property. Built best-effort — properties/documents that don't resolve are skipped and listed in the run's `sub_task.outputs`, not failed (ADR-0060). Stored under `DATA_BUCKET/bulk-downloads/` with a ~7-day lifecycle; delivered as a 60-minute presigned URL.
The single ZIP archive a **Bulk Document Download** produces: one folder per property (named by human-readable address, enriched from the hubspot deals data, with `landlord_property_id` appended for uniqueness), each folder holding the latest file of each **Document Type** held for that property. Built best-effort — properties/documents that don't resolve are skipped and listed in the run's `sub_task.outputs`, not failed (ADR-0060). Streamed to a dedicated `DOCUMENT_EXPORTS_BUCKET` (never held whole in memory — a package can be several GB); delivered as a 60-minute presigned URL.
_Avoid_: export (that is the plan/scenario XLSX export), bundle, archive (ambiguous), zip (the format, not the concept)
**Bulk Document Download**:

31
docs/adr/0060-bulk-document-download-package-model.md
View file

@ -29,9 +29,13 @@ lane (ADR-0055).**
not the 256 KB SQS body), pre-creates **one** `sub_task`, and drops one SQS
message (`task_id`, `sub_task_id`). The new `applications/bulk_document_download`
Lambda runs in **attach mode**; `TaskOrchestrator` owns status + roll-up.
- **One package = one sub_task.** No fan-out: the job streams each document out
of its own `s3_file_bucket` into a single ZIP in `/tmp` and multipart-uploads
it. One artifact, one URL, one email.
- **One package = one sub_task, streamed.** No fan-out. A Download Package can
be **several GB**, so it is never held whole in memory: each document is read
from its own `s3_file_bucket`, written into an on-disk ZIP in `/tmp`, and
released; the finished archive is **multipart-uploaded from disk**
(`S3Client.upload_file`). The Lambda's ephemeral storage (`/tmp`) is raised
accordingly (up to 10 GB) — see Consequences. One artifact, one URL, one
email.
- **Selection cap at the route.** Reject `> N` properties synchronously with a
legible "narrow your selection" error (property count, not a document/byte
query — cheap, no S3 on the hot path). A `/tmp` size budget inside the Lambda
@ -55,10 +59,14 @@ lane (ADR-0055).**
- **Delivery — both channels.** The presigned URL (60-minute expiry) and the
skip-summary are written to `sub_task.outputs` (the FE already polls task
status and can show the link) **and** emailed (ADR-0059).
- **Retention.** Packages are written to `DATA_BUCKET` under a
`bulk-downloads/` prefix with an S3 lifecycle rule expiring objects after
~7 days — longer than the 60-minute URL (so a link can be re-issued without
rebuilding) but short enough that bundles never accumulate.
- **A dedicated exports bucket.** Packages are written to a **separate
`DOCUMENT_EXPORTS_BUCKET`**, not `DATA_BUCKET` — the generated archives are a
distinct class of artifact (transient, user-facing, wide-read source but
single-writer) and keeping them out of the shared data bucket keeps IAM and
any future retention policy cleanly scoped to exports. No lifecycle rule is
imposed on `DATA_BUCKET`. Retention on the exports bucket is an open decision
(the packages are transient, so an expiry is likely warranted, but it is left
to a follow-up rather than baked in here).
## Considered options
@ -86,7 +94,12 @@ lane (ADR-0055).**
port/adapter from ADR-0059. The only `backend/` touch is the trigger route.
- The stored, pinned `property_id` set makes a run reproducible even if the
portfolio changes between trigger and execution.
- `N`, the `/tmp` size budget, and the 7-day lifecycle are tunable knobs, not
load-bearing invariants.
- `N`, the Lambda's ephemeral-storage (`/tmp`) size, and its memory are tunable
knobs, not load-bearing invariants. `/tmp` must be raised beyond the 512 MB
default (up to 10 GB) to hold a multi-GB archive — a new terraform knob on the
shared Lambda modules, backward-compatible (every other Lambda keeps 512 MB).
- A **new S3 bucket** (`DOCUMENT_EXPORTS_BUCKET`) is provisioned for the
packages, with its own IAM (single-writer, presign-read). Its retention policy
is a deliberate follow-up, not set here.
- Closing the `uploaded_files.property_id` gap later would let matching move off
`landlord_property_id` without changing the package model.