Model/docs/adr/0060-bulk-document-download-package-model.md

status
accepted (builds on ADR-0055, ADR-0059)

Bulk Document Download builds one capped, best-effort Download Package per request

Users need to pull the documents held in uploaded_files for many properties at once — per property, the latest file of each Document Type — as a single archive, without clicking through the UI file by file. The request is initiated in the front end, can span a hand-picked set of properties or a whole portfolio, and finishes with a link emailed to the requester (ADR-0059).

uploaded_files links to a property by uprn or landlord_property_id (both nullable) and has no property_id column; file_type (the Document Type) is itself nullable; and files live across arbitrary buckets (s3_file_bucket per row).

Decision

A request produces exactly one Download Package: a ZIP of one folder per property, each holding the latest file of each Document Type — built best-effort, size-capped at the trigger, on the app-owned-task + attach-mode lane (ADR-0055).

• Trigger & lifecycle (ADR-0055). The FastAPI route creates the tasks row, resolves the selection to a concrete property_id set (expanding "all properties" of a portfolio), pins that set plus the recipient email into tasks.inputs (TEXT/JSON — a large id list is why this lives in inputs, not the 256 KB SQS body), pre-creates one sub_task, and drops one SQS message (task_id, sub_task_id). The new applications/bulk_document_download Lambda runs in attach mode; TaskOrchestrator owns status + roll-up.
• One package = one sub_task. No fan-out: the job streams each document out of its own s3_file_bucket into a single ZIP in /tmp and multipart-uploads it. One artifact, one URL, one email.
• Selection cap at the route. Reject > N properties synchronously with a legible "narrow your selection" error (property count, not a document/byte query — cheap, no S3 on the hot path). A /tmp size budget inside the Lambda is a backstop that fails the sub_task with a clear reason if a pathological selection still overflows. N starts at a conservative, tunable value.
• Matching on landlord_property_id. Files are gathered by landlord_property_id; the missing property_id on uploaded_files is a known future gap, out of scope here. Property display info (the folder name) is enriched from the hubspot deals data (repositories/hubspot_deals/).
• Layout. One folder per property named by human-readable address (hubspot-enriched), with landlord_property_id appended for uniqueness; inside, one file per Document Type = the newest by s3_upload_timestamp. Rows with a null Document Type are skipped and listed in the run output.
• Best-effort failure contract. Build from whatever resolves. Skipped properties (no documents) and skipped null-type files are recorded in sub_task.outputs. The run fails only on an infrastructure error (S3/DB/zip/upload/email) or when the whole selection yields zero documents. The email carries an "N properties, M documents, X skipped" summary.
• Delivery — both channels. The presigned URL (60-minute expiry) and the skip-summary are written to sub_task.outputs (the FE already polls task status and can show the link) and emailed (ADR-0059).
• Retention. Packages are written to DATA_BUCKET under a bulk-downloads/ prefix with an S3 lifecycle rule expiring objects after ~7 days — longer than the 60-minute URL (so a link can be re-issued without rebuilding) but short enough that bundles never accumulate.

Considered options

• Fan out into per-batch sub_tasks / multiple ZIPs (like the modelling run). Rejected: the user would receive N links / N emails and "the download" would stop being one file; the capped-single-package model keeps the artifact and the notification singular.
• Cap on resolved document count / bytes. Rejected for v1: it adds synchronous DB + S3 HEAD work to the trigger and couples the route to the packaging logic; a property-count cap is cheap and predictable.
• Strict "all-or-nothing" packaging. Rejected: one unreadable row or a property with no documents would deny the entire package; best-effort + reporting matches "include documents and properties where they exist".
• Match on uprn. Deferred: landlord_property_id is the agreed key and hubspot enrichment is keyed to the deal/property; revisit if/when the property_id gap on uploaded_files is closed.

Consequences

• New DDD pieces: applications/bulk_document_download/ (thin handler + trigger body), orchestration/bulk_document_download_orchestrator.py, packaging rules in domain/, a landlord_property_id-keyed "latest-per-Document-Type" query on the uploaded-file repository, a generate_presigned_url + multipart upload on S3Client, and the email port/adapter from ADR-0059. The only backend/ touch is the trigger route.
• The stored, pinned property_id set makes a run reproducible even if the portfolio changes between trigger and execution.
• N, the /tmp size budget, and the 7-day lifecycle are tunable knobs, not load-bearing invariants.
• Closing the uploaded_files.property_id gap later would let matching move off landlord_property_id without changing the package model.