Model/docs/adr/0065-ara-scenario-export.md

status
accepted (builds on ADR-0055, ADR-0056, ADR-0059, ADR-0060)

ARA Scenario Export builds one branded workbook per request, sheet-per-scenario

Users need to pull a portfolio's modelling results — property information plus each Property's Plan under one or more Scenarios — out of the product as a single spreadsheet, initiated from the front end and delivered as a link emailed to the requester. This is the Export the glossary already reserves the word for ("the plan/scenario XLSX export"), distinct from a Download Package (the ZIP of survey documents, ADR-0060).

The data layer already half-exists as scripts/data_exports.py (Jun-te Kim): a CLI over the new DDD schema that reads property / scenario / plan / recommendation / material, pivots measures onto a stable 21-measure column contract (EXPECTED_MEASURE_COLUMNS), and emits one sheet per scenario. But as written it (a) fetches EPC descriptive fields live from the gov API per property (the retired property_details_epc table's replacement), (b) has no filters and auto-derives every scenario with plans, (c) lives in scripts/ outside the DDD layout, and (d) has no styling, S3, or email. None of those survive a front-end feature that must scope by scenario + filters and scale to tens of thousands of properties inside a Lambda.

The trigger/email/large-selection plumbing is already solved by Bulk Document Download (ADR-0055/0059/0060) and is reused wholesale.

Decision

A request produces exactly one branded .xlsx Export: one sheet per selected Scenario, over a filtered, scenario-scoped Property set, built from persisted data, on the app-owned-task + attach-mode lane (ADR-0055), delivered by presigned URL + email (ADR-0059).

• Scenario-scoped selection (model A). The unit of scope is a Scenario's results. The base set for each sheet is the Properties that have a Plan under that Scenario; filters and any hand-picked ids only narrow it. A Property that filters in but has no Plan for that Scenario drops out of that sheet — there is nothing to export for it. Scenario membership is applied in the data layer (the plan join), not in the filter resolver.

• Sheet-per-Scenario, one workbook (A2). The recipe carries scenario_ids: list[int]; the same filtered Property set is rendered once per Scenario, each on its own sheet. Under model A the sheets may legitimately have different row counts (a Property with a Plan under Scenario X but not Y appears on the X sheet only). Excel-only for v1 — a sheet-per-scenario workbook is an inherently Excel artifact; there is no format field (or it is fixed to "xlsx"). CSV degradation is a fast-follow if a warehouse consumer ever needs it.

• Trigger & lifecycle (mirror ADR-0055). The front end creates the app-owned tasks row and writes the selection recipe — {portfolio_id, scenario_ids, filters, property_ids?, recipient_email} — into tasks.inputs (JSON, FE-owned), then calls the FastAPI route (backend/app/exports/) with only the task_id. The route reads the recipe from tasks.inputs, resolves it to the concrete internal property.id set (below), pins the resolved id list + recipe onto one pre-created sub_task's inputs, and drops one SQS message (task_id, sub_task_id) on a new ARA_EXPORT_SQS_URL queue. The new applications/ara_export Lambda runs in attach mode, reading that recipe from the sub_task; TaskOrchestrator owns status + roll-up. Nothing large ever travels in an HTTP body or on SQS — the id list rides tasks.inputs → sub_task.inputs (both Postgres JSON), so the 256 KB SQS limit is irrelevant by construction and the "tens of thousands of ids" concern is structurally impossible.

• Filters are the ADR-0056 shared contract. The route resolves filters with the same PropertyGroupFilters model and resolve_filtered_property_ids(session, portfolio_id, filters) function the Modelling Run Distributor uses — one filter vocabulary, one resolver, two triggers. New filter dimensions land on PropertyGroupFilters and both triggers inherit them. property_ids is an override, not a co-filter: present and non-empty ⇒ export exactly those ids; absent ⇒ resolve from filters. Precedence (not intersection) avoids the "I ticked 5 rows, got 3" surprise where a lingering filter silently drops hand-picked rows. Both paths key on internal property.id (as data_exports.py and the modelling resolver already do), so identifiers align.

• Persisted data only — the Effective EPC, no live API. Because model A guarantees every exported Property has a Plan, it has been through Ingestion → Baseline → Modelling and its cert / Effective EPC / Baseline Performance / Plans are all persisted. The Export is a pure read. Descriptive fields follow the Effective EPC precedence — property_overrides → lodged EPC → predicted EPC — the same ladder property_filters._resolved_component_values already implements, widened from property_type/built_form to the full descriptive set. No live EpcClientService calls: tens of thousands of sequential gov-API fetches would blow the Lambda's 15-minute ceiling and hammer the register.

• Promote the data layer into DDD, split by concern. A repositories/ read repository returns the persisted, override-resolved rows for a (portfolio_id, scenario_id, property-id set); a pure domain/ column-shaper owns the frozen 21-measure contract and the measure→cost-column pivot + savings summation (trivially unit-testable). The applications/ara_export Lambda composes: repo read → domain shape → styling → S3 → email. scripts/data_exports.py becomes a thin CLI wrapper over the same modules (or is retired) so the query/column logic exists once — this also removes the _PROPERTY_TYPE_CODES map duplicated today in both data_exports.py and property_filters.py.

• Frozen column contract, mimic data_exports.py. The 21-measure EXPECTED_MEASURE_COLUMNS contract is stable — every measure column always present, blank where a Property lacks that measure — rather than emitting only the measures that happen to appear. Property identity + Effective EPC descriptive fields + pivoted per-measure cost columns + summed savings + Plan post-works figures (SAP / kWh / CO₂ / bill), per sheet.

• Programmatic branding, palette in infrastructure/. Styling is a programmatic openpyxl pass (header fill + bold header font, frozen top row, banded rows, coloured cost/band columns, column widths), not a populated template. A2's dynamic sheet count defeats a fixed-sheet template (which would force lossy per-sheet copy_worksheet cloning); a helper styles whatever it generates. The brand palette + styling helper live in infrastructure/xlsx/ — filling the "no brand-colour constants exist anywhere" gap once, reusable enough that audit_generator could later adopt it. domain/ stays pure and colour-agnostic. Monday.com-style defaults ship now behind a # TODO(brand) marker until real Domna hex values land.

• Caps & memory. The bulk-download MAX_PROPERTIES = 500 (a ZIP-of-GBs policy) is dropped; the Export payload is one .xlsx of mostly numbers. The route rejects > ARA_EXPORT_MAX_PROPERTIES (100 000, configurable) synchronously with a legible "narrow your selection" error. The Lambda builds in normal openpyxl mode on a 4 GB-memory Lambda to start, streamed to a temp file on /tmp and multipart-uploaded (S3Client.upload_file), never an in-memory BytesIO. The 100k ceiling in normal mode may need more memory (openpyxl holds every cell as a Python object — file size is not the RAM gauge); the fallback if it OOMs is to raise the Lambda memory, then switch to write_only mode (which constrains the styling helper to style-at-write). The common case (hundreds–low-thousands per portfolio) is comfortably within 4 GB.

• Delivery — both channels (ADR-0059). The workbook is written to the dedicated DOCUMENT_EXPORTS_BUCKET (reused, not the shared DATA_BUCKET); the presigned URL (60-minute expiry, _URL_TTL_SECONDS = 3600) and a summary are written to sub_task.outputs and emailed via SesSmtpEmailSender to recipient_email. Email send is best-effort (wrapped in try/except — a delivery failure does not fail the task; the URL is still on sub_task.outputs, which the FE polls).

• Best-effort rows. A Property/Plan that fails to resolve is skipped and recorded in sub_task.outputs, not a hard failure. The run fails only on an infrastructure error (DB / S3 / workbook / upload) or when the whole selection yields zero exportable rows.

• Idempotency. The sub_task is created with the same idempotent INSERT ... WHERE NOT EXISTS the DB arbitrates for double-submit (create_download_subtask pattern).

Deferred / future work

• Tag-based filtering. The next filter the front end wants is tags, but there is no tag model in the backend (no property_tags table, no override→EPC→predicted resolution ladder for it) — it is a new concept that needs its own /grill-with-docs before landing on PropertyGroupFilters. Once designed, both the Export and the Modelling Run triggers inherit it via the ADR-0056 shared contract. A # TODO(tags) marker at the filter seam points back to this ADR.
• CSV output. Only if a warehouse/finance consumer needs a flat feed; would collapse the sheets to one table with a scenario_id column, unbranded.
• write_only streaming if the 100k ceiling proves too heavy for a reasonably-sized Lambda in normal mode.
• Exports-bucket retention — the Exports are transient; an expiry lifecycle rule is likely warranted but is left to a follow-up (as with ADR-0060).

Considered options

• Property-scoped export (model B) — resolve Properties first, attach scenario columns, keep no-Plan Properties with blank scenario cells. Rejected: the feature is "give me the modelled results of Scenario X", so a Property with no Plan for the Scenario has nothing to show; blanks would mislead.
• One Scenario per export (A1) — simplest contract, single flat result. Rejected: the FE screen lets users tick several Scenarios for comparison, which is exactly the sheet-per-Scenario workbook.
• Import scripts/data_exports.py from the Lambda. Rejected: re-buries raw SQL in scripts/, keeps the dead live-EPC dependency, duplicates the code-maps, and can't sit cleanly in the DDD layout.
• Live EPC fetch (as the script does). Rejected: tens of thousands of sequential gov-API calls won't finish in a Lambda and abuse the register; model A guarantees the data is already persisted.
• property_ids intersects filters. Rejected: a hand-picked selection silently shrinking because of a stale filter is a surprising, hard-to-debug UX; override precedence is predictable.
• Populate a branded .xlsx template (the audit_generator pattern). Rejected for A2: a fixed-sheet template can't hold a dynamic number of scenario sheets without lossy runtime copy_worksheet cloning, and couples the deploy to a binary asset.
• write_only from day one. Deferred: it constrains styling to style-at-write for no benefit at the common scale; adopt only if 4 GB normal mode proves insufficient at the ceiling.

Consequences

• New DDD pieces: applications/ara_export/ (thin @task_handler + trigger body carrying only task_id/subtask_id), orchestration/ara_export_orchestrator.py, a ScenarioExportRepository in repositories/ returning an override-resolved read-model (so the orchestrator never names infrastructure.postgres.*), a pure column-shaper in domain/, and a brand palette + openpyxl styling helper in infrastructure/xlsx/. The only backend/ touches are the new backend/app/exports/ route and reusing resolve_filtered_property_ids.
• New infra: an ARA_EXPORT_SQS_URL queue (terraform), the Lambda at 4 GB memory with /tmp for the workbook, reuse of DOCUMENT_EXPORTS_BUCKET and the SES/S3 adapters from ADR-0059.
• The pinned, resolved property.id set makes a run reproducible even if the portfolio changes after submission.
• PropertyGroupFilters / resolve_filtered_property_ids (ADR-0056) now has a second production caller; changes to the shared contract affect both triggers — the intended shareability, made explicit.
• scripts/data_exports.py's query/column logic is de-duplicated into the new modules; the standalone CLI either wraps them or is retired.

Addendum — plan-selection option (default plan per home)

A second plan selection was added alongside the original per-Scenario export, chosen by plan_selection on the export request:

• "scenario" (default, unchanged): one sheet per requested Scenario, the freshest Plan per (property, scenario). scenario_ids required.
• "default" (new): a single sheet titled "Default Plans" of each home's default Plan (plan.is_default = TRUE), which is one-per-property across all Scenarios (ADR-0012 / ADR-0017) — the portfolio's current state, one row per home. scenario_ids is not required and is ignored.

The router (backend/app/exports/router.py) validates plan_selection ∈ {"scenario", "default"} and only requires a non-empty scenario_ids for the scenario selection; the recipe carries plan_selection through to the handler and orchestrator. The repository gains default_rows_for(portfolio_id, property_ids) — the same read-model and Effective-EPC join as rows_for, with the plan-selection WHERE scenario_id = … swapped for WHERE is_default = TRUE (no scenario parameter). A recipe without plan_selection (pre-existing tasks) defaults to "scenario", so the change is backward-compatible.

New column — plan_name. Both selections surface the source Scenario's name (scenario.name, via plan.scenario_id) as a plan_name column heading the plan post-works block, so a reader sees which Scenario each home's Plan came from (e.g. "Fabric Only" / "ASHP + PV"). The freeform plan.name is unset on modelling-generated Plans, so the Scenario name is the meaningful identifier.