Model/docs/adr/0060-bulk-document-download-package-model.md

status
accepted (builds on ADR-0055, ADR-0059)

Bulk Document Download builds one capped, best-effort Download Package per request

Users need to pull the documents held in uploaded_files for many properties at once — per property, the latest file of each Document Type — as a single archive, without clicking through the UI file by file. The request is initiated in the front end, can span a hand-picked set of properties or one or more whole HubSpot projects (by project code), and finishes with a link emailed to the requester (ADR-0059).

uploaded_files links to a property by uprn or landlord_property_id (both nullable) and has no property_id column; file_type (the Document Type) is itself nullable; and files live across arbitrary buckets (s3_file_bucket per row).

Decision

A request produces exactly one Download Package: a ZIP of one folder per property, each holding the latest file of each Document Type — built best-effort, size-capped at the trigger, on the app-owned-task + attach-mode lane (ADR-0055).

• Trigger & lifecycle (ADR-0055). The front end creates the app-owned tasks row and writes the selection config — {project_codes?: str[], landlord_property_ids?: str[], portfolio_id?: int} — into tasks.inputs (TEXT/JSON, FE-owned), then calls the FastAPI route with only the task_id. A large selection therefore never travels in an HTTP body. The route reads the selection from tasks.inputs and resolves it to the distinct landlord_property_id set — the union of every property in the named HubSpot project_codes (read from hubspot_deal_data, where the project↔property grain lives) and the hand-picked landlord_property_ids. landlord_property_id is the selection key throughout because uploaded_files is matched on it (it has no property_id). At least one of the two must be provided; portfolio_id is optional and used only to name the package. The route then caps the set, resolves the recipient email from the authenticated user (ADR-0059), pins the resolved recipe (landlord_property_ids, recipient_email, package_name) onto one pre-created sub_task's inputs, and drops one SQS message (task_id, sub_task_id). The new applications/bulk_document_download Lambda runs in attach mode, reading that recipe from the sub_task; TaskOrchestrator owns status + roll-up. (task.inputs = the FE's selection; sub_task.inputs = the backend's resolved recipe.)
• One package = one sub_task, streamed. No fan-out. A Download Package can be several GB, so it is never held whole in memory: each document is read from its own s3_file_bucket, written into an on-disk ZIP in /tmp, and released; the finished archive is multipart-uploaded from disk (S3Client.upload_file). The Lambda's ephemeral storage (/tmp) is raised accordingly (up to 10 GB) — see Consequences. One artifact, one URL, one email.
• Selection cap at the route. Reject > N properties synchronously with a legible "narrow your selection" error (property count, not a document/byte query — cheap, no S3 on the hot path). A /tmp size budget inside the Lambda is a backstop that fails the sub_task with a clear reason if a pathological selection still overflows. N starts at a conservative, tunable value.
• Matching via hubspot_deal_id, bridged through hubspot_deal_data. The selection is a set of landlord_property_ids, but uploaded_files is matched on hubspot_deal_id: in practice no upload source populates uploaded_files.landlord_property_id (pashub, magic plan and the audit generator set hubspot_deal_id; ECMK sets hubspot_listing_id), so the worker joins uploaded_files → hubspot_deal_data (on deal_id) → landlord_property_id and takes the property identity from the bridge, not the file row. Coverage limitation: files linked only by hubspot_listing_id (ECMK) or uprn are not reached; extending the bridge to those keys is a follow-up. Property display info (the folder name) is enriched from the property record (repositories/property/).
• Layout. One folder per property named by human-readable address (hubspot-enriched), with landlord_property_id appended for uniqueness; inside, one file per Document Type = the newest by s3_upload_timestamp. Rows with a null Document Type are skipped and listed in the run output.
• Best-effort failure contract. Build from whatever resolves. Skipped properties (no documents) and skipped null-type files are recorded in sub_task.outputs. The run fails only on an infrastructure error (S3/DB/zip/upload/email) or when the whole selection yields zero documents. The email carries an "N properties, M documents, X skipped" summary.
• Delivery — both channels. The presigned URL (60-minute expiry) and the skip-summary are written to sub_task.outputs (the FE already polls task status and can show the link) and emailed (ADR-0059).
• A dedicated exports bucket. Packages are written to a separate DOCUMENT_EXPORTS_BUCKET, not DATA_BUCKET — the generated archives are a distinct class of artifact (transient, user-facing, wide-read source but single-writer) and keeping them out of the shared data bucket keeps IAM and any future retention policy cleanly scoped to exports. No lifecycle rule is imposed on DATA_BUCKET. Retention on the exports bucket is an open decision (the packages are transient, so an expiry is likely warranted, but it is left to a follow-up rather than baked in here).

Considered options

• Fan out into per-batch sub_tasks / multiple ZIPs (like the modelling run). Rejected: the user would receive N links / N emails and "the download" would stop being one file; the capped-single-package model keeps the artifact and the notification singular.
• Cap on resolved document count / bytes. Rejected for v1: it adds synchronous DB + S3 HEAD work to the trigger and couples the route to the packaging logic; a property-count cap is cheap and predictable.
• Strict "all-or-nothing" packaging. Rejected: one unreadable row or a property with no documents would deny the entire package; best-effort + reporting matches "include documents and properties where they exist".
• Match on landlord_property_id directly. Rejected once tested against real data: uploaded_files.landlord_property_id is unpopulated, so a direct match returns nothing. hubspot_deal_id (bridged via hubspot_deal_data) is the key that upload sources actually set.
• Match on uprn / hubspot_listing_id as well. Deferred: hubspot_deal_id covers the bulk of sources; add uprn/listing_id to the bridge if a source keyed only by those needs including.

Consequences

• New DDD pieces: applications/bulk_document_download/ (thin handler + trigger body), orchestration/bulk_document_download_orchestrator.py, packaging rules in domain/, a hubspot_deal_id-bridged query on the uploaded-file repository that returns a small PropertyDocument read-model (so the orchestrator never names infrastructure.postgres.*), a generate_presigned_url + multipart upload on S3Client, and the email port/adapter from ADR-0059. The only backend/ touch is the trigger route.
• The stored, pinned property_id set makes a run reproducible even if the portfolio changes between trigger and execution.
• N, the Lambda's ephemeral-storage (/tmp) size, and its memory are tunable knobs, not load-bearing invariants. /tmp must be raised beyond the 512 MB default (up to 10 GB) to hold a multi-GB archive — a new terraform knob on the shared Lambda modules, backward-compatible (every other Lambda keeps 512 MB).
• A new S3 bucket (DOCUMENT_EXPORTS_BUCKET) is provisioned for the packages, with its own IAM (single-writer, presign-read). Its retention policy is a deliberate follow-up, not set here.
• Because matching bridges through hubspot_deal_data, a file is only reachable once its deal has a hubspot_deal_data row (and the file carries that hubspot_deal_id). Populating uploaded_files.landlord_property_id at upload time, or closing the uploaded_files.property_id gap, would let matching drop the bridge without changing the package model.