diff --git a/CONTEXT.md b/CONTEXT.md index caaf3029e..26c65a00c 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -99,6 +99,18 @@ _Avoid_: ventilation report, audit report, PAS ventilation (that is the external An externally-uploaded ventilation survey document produced by a human assessor and ingested from an external source (e.g. Coordination Hub). Recorded in `uploaded_files` with `file_type = PAS_2023_VENTILATION`. Distinct from a **Ventilation Audit**, which is machine-generated from MagicPlan floor plan data. _Avoid_: ventilation audit (that is the generated output) +**Document Type**: +The category of an uploaded document — the `uploaded_files.file_type` enum (photo_pack, site_note, the pas_2023_* / ecmk_* families, magic_plan_json, mcs_compliance_certificate, ventilation_audit, other, …). Nullable on the row. In a **Download Package** exactly one file per Document Type is included per property — the newest by `s3_upload_timestamp`; a row with a null Document Type is skipped and reported (ADR-0060). +_Avoid_: document category, file kind, doc class + +**Download Package**: +The single ZIP archive a **Bulk Document Download** produces: one folder per property (named by human-readable address, enriched from the hubspot deals data, with `landlord_property_id` appended for uniqueness), each folder holding the latest file of each **Document Type** held for that property. Built best-effort — properties/documents that don't resolve are skipped and listed in the run's `sub_task.outputs`, not failed (ADR-0060). Stored under `DATA_BUCKET/bulk-downloads/` with a ~7-day lifecycle; delivered as a 60-minute presigned URL. +_Avoid_: export (that is the plan/scenario XLSX export), bundle, archive (ambiguous), zip (the format, not the concept) + +**Bulk Document Download**: +The feature/job that assembles a **Download Package** for a chosen set of properties (a hand-picked list or a whole portfolio) and emails the requester a link. Runs on the app-owned-task + attach-mode lane (ADR-0055): the FastAPI route pins the resolved `property_id` set and the recipient email into `tasks.inputs`, and the `applications/bulk_document_download` Lambda builds the package, writes the URL to `sub_task.outputs`, and emails it (ADR-0059/0060). Files are matched by `landlord_property_id` (the missing `property_id` on `uploaded_files` is a known future gap). Selection is capped by property count at the trigger. +_Avoid_: document export, bulk export, file dump + ### Source data **Site Notes**: diff --git a/docs/adr/0059-outbound-email-via-ses-smtp-adapter.md b/docs/adr/0059-outbound-email-via-ses-smtp-adapter.md new file mode 100644 index 000000000..fb2946eae --- /dev/null +++ b/docs/adr/0059-outbound-email-via-ses-smtp-adapter.md @@ -0,0 +1,79 @@ +--- +status: accepted +--- + +# Outbound email is sent from the backend via an SES-SMTP adapter, with the recipient threaded from the trigger route + +The Bulk Document Download feature (ADR-0060) must email a completed package's +link to the user who requested it. This is the **first outbound email in the +Python backend** — there is no `smtplib`/SES/SendGrid code anywhere today. Two +facts force a decision rather than a copy: + +1. **Where the email is sent from.** The front end can send email, but the + package is produced **asynchronously in a Lambda**, long after the trigger + request has returned `202`. The completion event exists only on the backend. +2. **Who to email.** Auth today only *gates* the FastAPI routes + (`validate_token` returns the raw token, not the user); neither the handler + nor the `Task`/`SubTask` model captures a user. The requesting identity is + provable (the NextAuth JWT carries `dbId` → `UserModel.email`) but is not + currently threaded anywhere. + +## Decision + +**The backend sends the email, through a DDD port + SES-SMTP adapter, to a +recipient the trigger route resolves and pins into `tasks.inputs`.** + +- **Send from the backend, on completion.** The job that finishes the work + owns the notification — the orchestrator calls an injected email port after + the package is uploaded and the URL minted. The front end is not asked to + poll-then-send. +- **Port + adapter (hexagonal).** A `repositories/email/` port + (`EmailSender` — `send(to, subject, body/html)`) with an + `infrastructure/email/ses_smtp_email_sender.py` adapter over `smtplib`, using + the **existing SES SMTP** setup (terraform `modules/ses`, domain + `domna.homes`, credentials already in Secrets Manager + `${stage}/ses/smtp_credentials`). SMTP, not the SES SendEmail API, because + that is what the provisioned infrastructure exposes. The port is injected into + the orchestrator so the send is testable behind an interface and the transport + is swappable. +- **Recipient threaded at the route, pinned in `inputs`.** The FastAPI trigger + route resolves the authenticated user (`dbId` → `UserModel.email`, via the + existing `get_user`) and writes the email into the `tasks.inputs` JSON at task + creation. The Lambda reads it from `inputs` and never touches auth. This + graduates the route from *gating* the token to *injecting* the user — a + deliberate, contained change to `backend/app/dependencies.py` usage at the one + new route. +- **Config follows the Lambda convention.** SMTP host/username/password reach + the Lambda as environment variables (sourced from the Secrets Manager entry in + terraform), read via `os.environ`/`PostgresConfig`-style config — not + `backend/app/config.py get_settings()`, which is the legacy app boundary. + +## Considered options + +- **Front end sends the email.** Rejected: the completion moment lives in the + Lambda; making the FE responsible means it must poll task status to + completion and then send — more moving parts, and the notification would lag + or be missed if the user closes the tab. Backend-sends keeps the trigger and + the notification on the same side of the queue. +- **SES SendEmail API (boto3 `ses`) instead of SMTP.** Rejected for now: the + provisioned infrastructure issues SMTP credentials; using the API would need + new IAM + identity wiring. Revisit if a templated/bulk API becomes worthwhile. +- **Resolve the recipient in the Lambda from `portfolio_id` → `PortfolioUsers`.** + Rejected: a portfolio has many users, so the join cannot identify *the + requester* — real risk of emailing the wrong person or several. The requester + is only unambiguous at the authenticated route. +- **FE passes the recipient email in the trigger body.** Rejected: trusts the + client for an identity the JWT already proves, and can drift from the + logged-in user. + +## Consequences + +- A reusable `EmailSender` capability now exists for future backend + notifications (not just this feature). +- The one new FastAPI route must inject the user (not merely gate) and persist + the email in `inputs`; `Task`/`SubTask` still gain no user column — the + recipient lives in the request payload, scoped to the job. +- Email content is plain for v1 (a link, a 60-minute expiry note, and the + skipped-document summary from `sub_task.outputs`); templating can follow. +- A new terraform env-var surface (SMTP creds from the existing Secrets Manager + entry) is added to the Lambda; no new SES infrastructure is required. diff --git a/docs/adr/0060-bulk-document-download-package-model.md b/docs/adr/0060-bulk-document-download-package-model.md new file mode 100644 index 000000000..5e234bc0a --- /dev/null +++ b/docs/adr/0060-bulk-document-download-package-model.md @@ -0,0 +1,92 @@ +--- +status: accepted (builds on ADR-0055, ADR-0059) +--- + +# Bulk Document Download builds one capped, best-effort Download Package per request + +Users need to pull the documents held in `uploaded_files` for many properties +at once — per property, the latest file of each **Document Type** — as a single +archive, without clicking through the UI file by file. The request is initiated +in the front end, can span a hand-picked set of properties or a whole +portfolio, and finishes with a link emailed to the requester (ADR-0059). + +`uploaded_files` links to a property by `uprn` **or** `landlord_property_id` +(both nullable) and has **no `property_id`** column; `file_type` (the Document +Type) is itself nullable; and files live across arbitrary buckets +(`s3_file_bucket` per row). + +## Decision + +**A request produces exactly one Download Package: a ZIP of one folder per +property, each holding the latest file of each Document Type — built +best-effort, size-capped at the trigger, on the app-owned-task + attach-mode +lane (ADR-0055).** + +- **Trigger & lifecycle (ADR-0055).** The FastAPI route creates the `tasks` + row, resolves the selection to a concrete `property_id` set (expanding "all + properties" of a portfolio), **pins that set plus the recipient email into + `tasks.inputs`** (TEXT/JSON — a large id list is why this lives in `inputs`, + not the 256 KB SQS body), pre-creates **one** `sub_task`, and drops one SQS + message (`task_id`, `sub_task_id`). The new `applications/bulk_document_download` + Lambda runs in **attach mode**; `TaskOrchestrator` owns status + roll-up. +- **One package = one sub_task.** No fan-out: the job streams each document out + of its own `s3_file_bucket` into a single ZIP in `/tmp` and multipart-uploads + it. One artifact, one URL, one email. +- **Selection cap at the route.** Reject `> N` properties synchronously with a + legible "narrow your selection" error (property count, not a document/byte + query — cheap, no S3 on the hot path). A `/tmp` size budget inside the Lambda + is a backstop that fails the sub_task with a clear reason if a pathological + selection still overflows. `N` starts at a conservative, tunable value. +- **Matching on `landlord_property_id`.** Files are gathered by + `landlord_property_id`; the missing `property_id` on `uploaded_files` is a + **known future gap**, out of scope here. Property display info (the folder + name) is enriched from the hubspot deals data + (`repositories/hubspot_deals/`). +- **Layout.** One folder per property named by human-readable **address** + (hubspot-enriched), with `landlord_property_id` appended for uniqueness; + inside, one file per Document Type = the newest by `s3_upload_timestamp`. + Rows with a **null Document Type are skipped** and listed in the run output. +- **Best-effort failure contract.** Build from whatever resolves. Skipped + properties (no documents) and skipped null-type files are recorded in + `sub_task.outputs`. The run **fails only** on an infrastructure error + (S3/DB/zip/upload/email) or when the whole selection yields **zero + documents**. The email carries an "N properties, M documents, X skipped" + summary. +- **Delivery — both channels.** The presigned URL (60-minute expiry) and the + skip-summary are written to `sub_task.outputs` (the FE already polls task + status and can show the link) **and** emailed (ADR-0059). +- **Retention.** Packages are written to `DATA_BUCKET` under a + `bulk-downloads/` prefix with an S3 lifecycle rule expiring objects after + ~7 days — longer than the 60-minute URL (so a link can be re-issued without + rebuilding) but short enough that bundles never accumulate. + +## Considered options + +- **Fan out into per-batch sub_tasks / multiple ZIPs** (like the modelling + run). Rejected: the user would receive N links / N emails and "the download" + would stop being one file; the capped-single-package model keeps the artifact + and the notification singular. +- **Cap on resolved document count / bytes.** Rejected for v1: it adds + synchronous DB + S3 HEAD work to the trigger and couples the route to the + packaging logic; a property-count cap is cheap and predictable. +- **Strict "all-or-nothing" packaging.** Rejected: one unreadable row or a + property with no documents would deny the entire package; best-effort + + reporting matches "include documents and properties where they exist". +- **Match on `uprn`.** Deferred: `landlord_property_id` is the agreed key and + hubspot enrichment is keyed to the deal/property; revisit if/when the + `property_id` gap on `uploaded_files` is closed. + +## Consequences + +- New DDD pieces: `applications/bulk_document_download/` (thin handler + + trigger body), `orchestration/bulk_document_download_orchestrator.py`, + packaging rules in `domain/`, a `landlord_property_id`-keyed + "latest-per-Document-Type" query on the uploaded-file repository, a + `generate_presigned_url` + multipart upload on `S3Client`, and the email + port/adapter from ADR-0059. The only `backend/` touch is the trigger route. +- The stored, pinned `property_id` set makes a run reproducible even if the + portfolio changes between trigger and execution. +- `N`, the `/tmp` size budget, and the 7-day lifecycle are tunable knobs, not + load-bearing invariants. +- Closing the `uploaded_files.property_id` gap later would let matching move off + `landlord_property_id` without changing the package model.