diff --git a/CONTEXT.md b/CONTEXT.md index 26c65a00c..33d79f62e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -104,7 +104,7 @@ The category of an uploaded document — the `uploaded_files.file_type` enum (ph _Avoid_: document category, file kind, doc class **Download Package**: -The single ZIP archive a **Bulk Document Download** produces: one folder per property (named by human-readable address, enriched from the hubspot deals data, with `landlord_property_id` appended for uniqueness), each folder holding the latest file of each **Document Type** held for that property. Built best-effort — properties/documents that don't resolve are skipped and listed in the run's `sub_task.outputs`, not failed (ADR-0060). Stored under `DATA_BUCKET/bulk-downloads/` with a ~7-day lifecycle; delivered as a 60-minute presigned URL. +The single ZIP archive a **Bulk Document Download** produces: one folder per property (named by human-readable address, enriched from the hubspot deals data, with `landlord_property_id` appended for uniqueness), each folder holding the latest file of each **Document Type** held for that property. Built best-effort — properties/documents that don't resolve are skipped and listed in the run's `sub_task.outputs`, not failed (ADR-0060). Streamed to a dedicated `DOCUMENT_EXPORTS_BUCKET` (never held whole in memory — a package can be several GB); delivered as a 60-minute presigned URL. _Avoid_: export (that is the plan/scenario XLSX export), bundle, archive (ambiguous), zip (the format, not the concept) **Bulk Document Download**: diff --git a/docs/adr/0060-bulk-document-download-package-model.md b/docs/adr/0060-bulk-document-download-package-model.md index 5e234bc0a..a220033ff 100644 --- a/docs/adr/0060-bulk-document-download-package-model.md +++ b/docs/adr/0060-bulk-document-download-package-model.md @@ -29,9 +29,13 @@ lane (ADR-0055).** not the 256 KB SQS body), pre-creates **one** `sub_task`, and drops one SQS message (`task_id`, `sub_task_id`). The new `applications/bulk_document_download` Lambda runs in **attach mode**; `TaskOrchestrator` owns status + roll-up. -- **One package = one sub_task.** No fan-out: the job streams each document out - of its own `s3_file_bucket` into a single ZIP in `/tmp` and multipart-uploads - it. One artifact, one URL, one email. +- **One package = one sub_task, streamed.** No fan-out. A Download Package can + be **several GB**, so it is never held whole in memory: each document is read + from its own `s3_file_bucket`, written into an on-disk ZIP in `/tmp`, and + released; the finished archive is **multipart-uploaded from disk** + (`S3Client.upload_file`). The Lambda's ephemeral storage (`/tmp`) is raised + accordingly (up to 10 GB) — see Consequences. One artifact, one URL, one + email. - **Selection cap at the route.** Reject `> N` properties synchronously with a legible "narrow your selection" error (property count, not a document/byte query — cheap, no S3 on the hot path). A `/tmp` size budget inside the Lambda @@ -55,10 +59,14 @@ lane (ADR-0055).** - **Delivery — both channels.** The presigned URL (60-minute expiry) and the skip-summary are written to `sub_task.outputs` (the FE already polls task status and can show the link) **and** emailed (ADR-0059). -- **Retention.** Packages are written to `DATA_BUCKET` under a - `bulk-downloads/` prefix with an S3 lifecycle rule expiring objects after - ~7 days — longer than the 60-minute URL (so a link can be re-issued without - rebuilding) but short enough that bundles never accumulate. +- **A dedicated exports bucket.** Packages are written to a **separate + `DOCUMENT_EXPORTS_BUCKET`**, not `DATA_BUCKET` — the generated archives are a + distinct class of artifact (transient, user-facing, wide-read source but + single-writer) and keeping them out of the shared data bucket keeps IAM and + any future retention policy cleanly scoped to exports. No lifecycle rule is + imposed on `DATA_BUCKET`. Retention on the exports bucket is an open decision + (the packages are transient, so an expiry is likely warranted, but it is left + to a follow-up rather than baked in here). ## Considered options @@ -86,7 +94,12 @@ lane (ADR-0055).** port/adapter from ADR-0059. The only `backend/` touch is the trigger route. - The stored, pinned `property_id` set makes a run reproducible even if the portfolio changes between trigger and execution. -- `N`, the `/tmp` size budget, and the 7-day lifecycle are tunable knobs, not - load-bearing invariants. +- `N`, the Lambda's ephemeral-storage (`/tmp`) size, and its memory are tunable + knobs, not load-bearing invariants. `/tmp` must be raised beyond the 512 MB + default (up to 10 GB) to hold a multi-GB archive — a new terraform knob on the + shared Lambda modules, backward-compatible (every other Lambda keeps 512 MB). +- A **new S3 bucket** (`DOCUMENT_EXPORTS_BUCKET`) is provisioned for the + packages, with its own IAM (single-writer, presign-read). Its retention policy + is a deliberate follow-up, not set here. - Closing the `uploaded_files.property_id` gap later would let matching move off `landlord_property_id` without changing the package model.