From 3ef432c8b2ef4622601c6c974e2a78c571116560 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 7 Jul 2026 21:57:53 +0000 Subject: [PATCH 01/55] =?UTF-8?q?ADR-0058:=20Sub-Ladder=20PV=20Configurati?= =?UTF-8?q?ons=20give=20small=20roofs=20the=20array=20their=20caps=20allow?= =?UTF-8?q?=20=F0=9F=9F=AA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Grilled decisions (issue #1496): 2-panel floor, every rung 2..floor(cap), pro-rata per-segment yield (conservative), fill-by-generation, same cap regime as today (no new roof gate), symmetric battery pairing. Glossary gains "Sub-Ladder Configuration" ("trimmed"/"synthesized" deliberately avoided — both collide with existing vocabulary). Co-Authored-By: Claude Fable 5 --- CONTEXT.md | 6 +- ...adder-pv-configurations-for-small-roofs.md | 90 +++++++++++++++++++ 2 files changed, 95 insertions(+), 1 deletion(-) create mode 100644 docs/adr/0058-sub-ladder-pv-configurations-for-small-roofs.md diff --git a/CONTEXT.md b/CONTEXT.md index caaf3029e..d24ce42fd 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -317,9 +317,13 @@ The bound that limits a **Solar PV Recommendation**'s array to the dwelling's *o _Avoid_: roof-area haircut (overloads the MCS coverage haircut), double-property halving (the legacy semi-detached-only mechanism this replaces) **Solar PV Recommendation**: -The single "Solar PV" **Recommendation** for a Property, carrying competing whole-array **Measure Options** built from the **Solar Potential** (ADR-0026). Up to **five conservatively-sized array configs** (capped, ranked by energy generation as the size-suitability proxy; deliberately *not* full-roof — imagery can miss obstructions, so a coverage/edge-setback haircut applies per MCS, and the **Dwelling-Roof Cap** bounds the array to the dwelling's own roof under **Footprint Conflation**), each offered **with and without a battery** (≤ 10 Options). Each Option is priced at a **single price point** from the rate sheet (kWp band + scaffolding-by-elevation + optional battery/diverter) — no multi-product price variants. The PV Overlay sets `photovoltaic_arrays` (one per segment: peak_power, orientation, pitch, overshading) plus `pv_diverter_present`, `pv_connection`, and **`is_dwelling_export_capable=True`** (an export meter is ensured post-install); the battery variant adds `pv_batteries`. +The single "Solar PV" **Recommendation** for a Property, carrying competing whole-array **Measure Options** built from the **Solar Potential** (ADR-0026). Up to **five conservatively-sized array configs** (capped, ranked by energy generation as the size-suitability proxy; deliberately *not* full-roof — imagery can miss obstructions, so a coverage/edge-setback haircut applies per MCS, and the **Dwelling-Roof Cap** bounds the array to the dwelling's own roof under **Footprint Conflation**), each offered **with and without a battery** (≤ 10 Options). Each Option is priced at a **single price point** from the rate sheet (kWp band + scaffolding-by-elevation + optional battery/diverter) — no multi-product price variants. The PV Overlay sets `photovoltaic_arrays` (one per segment: peak_power, orientation, pitch, overshading) plus `pv_diverter_present`, `pv_connection`, and **`is_dwelling_export_capable=True`** (an export meter is ensured post-install); the battery variant adds `pv_batteries`. When no Google rung fits under the caps, **Sub-Ladder Configurations** (ADR-0058) supply the small rungs instead of dropping PV entirely. _Avoid_: solar bundle (PV is competing sized Options, not one fixed bundle like ASHP) +**Sub-Ladder Configuration**: +A PV array configuration Modelling derives **below Google's smallest offered rung**, so a small roof gets the array its caps allow instead of no PV at all (ADR-0058). Derived only when no `solarPanelConfigs` rung fits under the caps and at least 2 panels do: north planes dropped first, the rung's panels filled from segments ranked by per-panel yield (the **Dwelling-Roof Cap**'s fill-by-generation precedent), each kept segment's yearly energy scaled **pro-rata** (conservative — Google places panels best-first). One rung per whole panel count from **2** (the install floor) up to the cap; both caps (0.70 × max, **Dwelling-Roof Cap**) unchanged. Flows through the standard Option construction (battery pairing, pricing, PV Overlay) with no special cases. +_Avoid_: trimmed configuration ("trimmed" already means the north-segment drop), synthesized configuration ("synthesis" belongs to Reduced-Field Synthesis / EPC Prediction), minimum array (names the 2-panel floor, not the derived-rung concept) + **Solar PV Eligibility**: The rule fixing whether the **Solar PV Recommendation** is offered (ADR-0026): a **house or bungalow**, **not listed and not heritage**, with **no existing PV**, and a **feasible Solar Potential** (the Google Solar API returned usable, non-north roof segments). Crucially a **conservation area does NOT block PV** — panels are offered (installed sympathetically), so the planning gate is `not blocks_internal` (listed/heritage only), **not** `blocks_external`; this is the opposite of an external fabric measure like EWI, and is deliberate (legacy + planning practice allow conservation-area PV on non-prominent roofs). Flats/maisonettes (building-level shared roof) are deferred. _Avoid_: blocking conservation-area PV (only listed/heritage block), roof-area-from-floor-area estimate (eligibility uses the real Solar Potential) diff --git a/docs/adr/0058-sub-ladder-pv-configurations-for-small-roofs.md b/docs/adr/0058-sub-ladder-pv-configurations-for-small-roofs.md new file mode 100644 index 000000000..7c6e936bf --- /dev/null +++ b/docs/adr/0058-sub-ladder-pv-configurations-for-small-roofs.md @@ -0,0 +1,90 @@ +--- +status: accepted (extends ADR-0026 and ADR-0038) +--- + +# Sub-Ladder PV Configurations give small roofs the array their caps allow + +ADR-0026 sizes a PV array by picking from Google's `solarPanelConfigs` ladder +under a conservative cap (0.70 × `maxArrayPanelsCount`, min'd with the +ADR-0038 Dwelling-Roof Cap when it resolves). Google's ladder typically starts +at **4 panels**, so on a small roof the cap falls below the smallest rung and +the feasible set is **empty** — the dwelling silently gets **no Solar PV +Measure Option at all** rather than a small array. ADR-0038 already states the +intent "never emit a zero-panel or unbounded array", but that governs the cap +fallback, not the ladder; the ladder cliff violated its spirit. + +Motivation: portfolio 824 / scenario 1278 audit (2026-07-07). Property 750701 +(UPRN 100010328594) — an unrestricted house short of goal C at D/67 on an +unlimited budget — has `maxArrayPanelsCount = 5` with rungs at 4 and 5 panels; +cap 0.70 × 5 = 3.5 < 4 → no PV offered, while ~2 SAP of feasible PV would +plausibly reach C. The cliff affects every small-roofed house in every +portfolio, silently. + +## Decision + +**When no Google rung fits under the caps, derive Sub-Ladder Configurations — +array rungs below Google's smallest offered configuration — instead of +dropping PV entirely.** + +- **Trigger**: the feasible set (north-dropped Google configs with + `panels_count <= panel_cap`) is empty AND `floor(panel_cap) >= 2`. When + `floor(panel_cap) < 2`, or nothing survives the north-segment drop, the + outcome stays no-PV — a roof that cannot take 2 panels is never offered + them. +- **Floor = 2 panels** — the practical install floor. Economics stay the + Optimiser's job (ADR-0024/0026 doctrine: eligibility and sizing encode + physical installability only); on least-cost-to-target a poor-£/kWh small + array is bought exactly when it is the only route to the goal band, which + is when the landlord wants it offered. +- **Every rung, not one**: emit a Sub-Ladder Configuration at every whole + panel count from 2 to `floor(panel_cap)` (in practice at most rungs 2 and + 3 — the trigger implies `floor(panel_cap) <= 3`). Preserves ADR-0026's + genuine size/cost choice; the Optimiser stops at the cheapest array that + reaches goal. +- **Derivation**: start from Google's smallest rung, drop north planes first + (as for every config), then fill the rung's panel count from the remaining + segments **ranked by per-panel yield** — the ADR-0038 "fill by generation" + precedent. +- **Yield accounting is pro-rata per segment**: a kept segment contributes + `yearly_energy × kept_panels / segment_panels`. Provably conservative — + Google places panels best-first within a segment, so the kept panels' true + yield ≥ the pro-rata figure. Needs no new `SolarPotential` projection + fields. +- **Cap regime unchanged, no new gate**: the same `min(0.70 × max, + dwelling-roof budget)` applies; when the Dwelling-Roof Cap cannot resolve, + the 0.70-Google cap alone governs — exactly as it already does for full + configs, and a smaller array is strictly more conservative than what those + dwellings are offered today. +- **Downstream unchanged**: Sub-Ladder Configurations flow through the + existing option construction — with-and-without battery pairing, single + price point, PV Overlay — with no special cases. + +## Considered options + +- **Floor the cap at Google's smallest rung** (offer the 4-panel config when + the dwelling roof confirms it fits). Rejected: breaches the 70% + conservatism the cap encodes — the cap exists precisely to not trust + Google's maximum placement wholesale. +- **Exact per-panel yields** (extend the projection to carry Google's + `solarPanels` list and sum true kept-panel energies). Rejected for now: + new plumbing for precision that cannot change an Optimiser decision at 2–3 + panels; pro-rata errs conservative. Revisit if sub-ladder sizing ever + extends beyond the small-roof trigger. +- **3-panel floor**. Rejected: gives up the smallest roofs (Google max 3–4 + panels), where 2 panels is all the cap allows and may still clear a + near-boundary goal. +- **Require the Dwelling-Roof Cap to resolve before deriving**. Rejected: + stricter than today's treatment of full configs on unresolved roofs — + an inconsistent trust model. + +## Consequences + +- Small-roofed houses gain a PV path toward goal; on the motivating property + the expected offer is a 3-panel rung (~950 kWh/yr pro-rata from the + 4-panel rung's 1,278). Follow-up: re-model 750701 after implementation to + confirm the +2 SAP to band C. +- A no-PV outcome on a small roof now always means `floor(panel_cap) < 2`, + no usable non-north segment, or ineligibility — never the ladder cliff. +- The audit skill's small-roof expectation (a `solar` row with no `solar_pv` + candidate is not automatically the solar-blind batch bug) shrinks to the + sub-2-panel residue once portfolios are re-modelled. From 79d7743bb1a85c8806375000f4602c73768c6ef5 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 7 Jul 2026 22:03:43 +0000 Subject: [PATCH 02/55] =?UTF-8?q?A=20small=20roof=20derives=20Sub-Ladder?= =?UTF-8?q?=20rungs=20below=20Google's=20smallest=20config=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../modelling/test_solar_config_selection.py | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/tests/domain/modelling/test_solar_config_selection.py b/tests/domain/modelling/test_solar_config_selection.py index da08fd1a2..e24f54cd1 100644 --- a/tests/domain/modelling/test_solar_config_selection.py +++ b/tests/domain/modelling/test_solar_config_selection.py @@ -219,3 +219,40 @@ def test_all_north_or_empty_yields_no_configs() -> None: # Assert assert configs == () + + +def test_small_roof_derives_sub_ladder_rungs_below_googles_smallest_config() -> None: + # ADR-0058 — the 824/1278 audit's property 750701 (UPRN 100010328594): + # max 5 panels → cap 3.5, Google's ladder starts at 4, so today the + # dwelling gets NO PV at all. Instead, derive Sub-Ladder Configurations at + # every whole rung from 2 (the install floor) to floor(cap), from the + # smallest rung, with per-segment yield scaled pro-rata (conservative — + # Google places panels best-first). + + # Arrange — Google max 5, rungs at 4 (1278 kWh/yr) and 5 (1582), one + # south plane; cap = 0.70 × 5 = 3.5 < 4 → nothing fits the ladder. + potential = SolarPotential( + panel_capacity_watts=400.0, + max_array_panels_count=5, + configurations=( + SolarPanelConfiguration( + panels_count=4, + yearly_energy_dc_kwh=1278.0, + segments=(_segment(4, 180.0, 1278.0),), + ), + SolarPanelConfiguration( + panels_count=5, + yearly_energy_dc_kwh=1582.0, + segments=(_segment(5, 180.0, 1582.0),), + ), + ), + ) + + # Act + configs = select_conservative_configs(potential) + + # Assert — rungs 2 and 3 (ascending), yields pro-rata off the 4-panel + # rung (1278 × 2/4 = 639, × 3/4 = 958.5), segments resized to the rung. + assert [c.panels_count for c in configs] == [2, 3] + assert [c.yearly_energy_dc_kwh for c in configs] == [639.0, 958.5] + assert [sum(s.panels_count for s in c.segments) for c in configs] == [2, 3] From 1a34e788d1d0bc8ea357e484f9ba33926eaca6bd Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 7 Jul 2026 22:05:37 +0000 Subject: [PATCH 03/55] =?UTF-8?q?A=20small=20roof=20derives=20Sub-Ladder?= =?UTF-8?q?=20rungs=20below=20Google's=20smallest=20config=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../generators/solar_recommendation.py | 65 ++++++++++++++++++- 1 file changed, 64 insertions(+), 1 deletion(-) diff --git a/domain/modelling/generators/solar_recommendation.py b/domain/modelling/generators/solar_recommendation.py index f97a5752f..c344d0447 100644 --- a/domain/modelling/generators/solar_recommendation.py +++ b/domain/modelling/generators/solar_recommendation.py @@ -14,6 +14,7 @@ selection, the overlay and `recommend_solar` land in later slices. from __future__ import annotations import math +from dataclasses import replace from typing import Optional from datatypes.epc.domain.epc_property_data import ( @@ -186,6 +187,66 @@ def _drop_north_segments(config: SolarPanelConfiguration) -> SolarPanelConfigura ) +# ADR-0058 — the practical install floor for a Sub-Ladder Configuration; a +# roof whose cap resolves below this is never offered PV. +_MIN_SUB_LADDER_PANELS = 2 + + +def _resized_to( + config: SolarPanelConfiguration, panels: int +) -> SolarPanelConfiguration: + """A copy of ``config`` filled to ``panels`` panels, each kept segment's + yearly energy scaled pro-rata (conservative: Google places panels + best-first within a segment, so the kept panels' true yield is at least + the pro-rata figure).""" + kept: list[SolarRoofSegment] = [] + remaining: int = panels + for segment in config.segments: + if remaining <= 0: + break + take: int = min(segment.panels_count, remaining) + factor: float = take / segment.panels_count + kept.append( + replace( + segment, + panels_count=take, + yearly_energy_dc_kwh=segment.yearly_energy_dc_kwh * factor, + ) + ) + remaining -= take + return SolarPanelConfiguration( + panels_count=sum(segment.panels_count for segment in kept), + yearly_energy_dc_kwh=sum(segment.yearly_energy_dc_kwh for segment in kept), + segments=tuple(kept), + ) + + +def _sub_ladder_configs( + potential: SolarPotential, panel_cap: float +) -> tuple[SolarPanelConfiguration, ...]: + """Sub-Ladder Configurations (ADR-0058): rungs derived BELOW Google's + smallest offered config, for when nothing on the ladder fits under the + caps — a small roof gets the array its caps allow instead of no PV at + all. One rung per whole panel count from the install floor (2) up to + ``floor(panel_cap)``, derived from the smallest north-dropped config.""" + rung_max: int = math.floor(panel_cap) + if rung_max < _MIN_SUB_LADDER_PANELS: + return () + smallest: Optional[SolarPanelConfiguration] = None + for config in potential.configurations: + dropped = _drop_north_segments(config) + if not dropped.segments: + continue + if smallest is None or dropped.panels_count < smallest.panels_count: + smallest = dropped + if smallest is None: + return () + return tuple( + _resized_to(smallest, panels) + for panels in range(_MIN_SUB_LADDER_PANELS, rung_max + 1) + ) + + def select_conservative_configs( potential: SolarPotential, dwelling_roof_area_m2: Optional[float] = None, @@ -214,7 +275,9 @@ def select_conservative_configs( if trimmed.segments and trimmed.panels_count <= panel_cap ] if not feasible: - return () + # Nothing on Google's ladder fits under the caps — a small roof, not + # an unusable one. Derive Sub-Ladder Configurations (ADR-0058). + return _sub_ladder_configs(potential, panel_cap) # Collapse rungs that trimmed to the same usable size (north-drop can make # distinct original rungs coincide), keeping the higher-generation layout — # the Optimiser's dial is panel count (≈ kWp ≈ cost), so duplicates of the From 0b33f7dc65054116b5824c992b41a680c2136528 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 7 Jul 2026 22:06:10 +0000 Subject: [PATCH 04/55] =?UTF-8?q?Sub-Ladder=20rungs=20fill=20from=20the=20?= =?UTF-8?q?highest-yield=20segment=20first=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../modelling/test_solar_config_selection.py | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/tests/domain/modelling/test_solar_config_selection.py b/tests/domain/modelling/test_solar_config_selection.py index e24f54cd1..53b6dca18 100644 --- a/tests/domain/modelling/test_solar_config_selection.py +++ b/tests/domain/modelling/test_solar_config_selection.py @@ -256,3 +256,51 @@ def test_small_roof_derives_sub_ladder_rungs_below_googles_smallest_config() -> assert [c.panels_count for c in configs] == [2, 3] assert [c.yearly_energy_dc_kwh for c in configs] == [639.0, 958.5] assert [sum(s.panels_count for s in c.segments) for c in configs] == [2, 3] + + +def test_sub_ladder_rungs_fill_from_the_highest_yield_segment_first() -> None: + # ADR-0058 — filling follows the ADR-0038 "by generation" precedent: the + # derived rung takes panels from the better-yielding plane first, each + # kept segment scaled pro-rata, never biased by segment tuple order. + + # Arrange — smallest rung spans SE (2 panels, 250/panel) listed FIRST and + # SW (2 panels, 350/panel) listed second; max 5 → cap 3.5 → rungs 2 and 3. + south_east = SolarRoofSegment( + segment_index=0, + panels_count=2, + azimuth_degrees=135.0, + pitch_degrees=30.0, + yearly_energy_dc_kwh=500.0, + ) + south_west = SolarRoofSegment( + segment_index=1, + panels_count=2, + azimuth_degrees=225.0, + pitch_degrees=30.0, + yearly_energy_dc_kwh=700.0, + ) + potential = SolarPotential( + panel_capacity_watts=400.0, + max_array_panels_count=5, + configurations=( + SolarPanelConfiguration( + panels_count=4, + yearly_energy_dc_kwh=1200.0, + segments=(south_east, south_west), + ), + ), + ) + + # Act + configs = select_conservative_configs(potential) + + # Assert — rung 2 is the SW plane alone (700); rung 3 adds one SE panel + # pro-rata (700 + 250 = 950). + assert [c.panels_count for c in configs] == [2, 3] + assert [c.yearly_energy_dc_kwh for c in configs] == [700.0, 950.0] + rung_two, rung_three = configs + assert [s.azimuth_degrees for s in rung_two.segments] == [225.0] + assert {(s.azimuth_degrees, s.panels_count) for s in rung_three.segments} == { + (225.0, 2), + (135.0, 1), + } From c1a8c27bc1aa8b03c77d877fe1ed0954d67ff16e Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 7 Jul 2026 22:06:30 +0000 Subject: [PATCH 05/55] =?UTF-8?q?Sub-Ladder=20rungs=20fill=20from=20the=20?= =?UTF-8?q?highest-yield=20segment=20first=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../modelling/generators/solar_recommendation.py | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/domain/modelling/generators/solar_recommendation.py b/domain/modelling/generators/solar_recommendation.py index c344d0447..6b5b18f4a 100644 --- a/domain/modelling/generators/solar_recommendation.py +++ b/domain/modelling/generators/solar_recommendation.py @@ -195,13 +195,19 @@ _MIN_SUB_LADDER_PANELS = 2 def _resized_to( config: SolarPanelConfiguration, panels: int ) -> SolarPanelConfiguration: - """A copy of ``config`` filled to ``panels`` panels, each kept segment's - yearly energy scaled pro-rata (conservative: Google places panels - best-first within a segment, so the kept panels' true yield is at least - the pro-rata figure).""" + """A copy of ``config`` filled to ``panels`` panels — taken from the + highest per-panel-yield segments first (the ADR-0038 fill-by-generation + precedent) — each kept segment's yearly energy scaled pro-rata + (conservative: Google places panels best-first within a segment, so the + kept panels' true yield is at least the pro-rata figure).""" + by_yield: list[SolarRoofSegment] = sorted( + config.segments, + key=lambda s: s.yearly_energy_dc_kwh / s.panels_count, + reverse=True, + ) kept: list[SolarRoofSegment] = [] remaining: int = panels - for segment in config.segments: + for segment in by_yield: if remaining <= 0: break take: int = min(segment.panels_count, remaining) From 5a382948d51b29975411fa20fd250ff9eea4a50c Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 7 Jul 2026 22:07:18 +0000 Subject: [PATCH 06/55] =?UTF-8?q?Sub-Ladder=20rungs=20never=20take=20from?= =?UTF-8?q?=20north=20planes=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pins behavior delivered with the tracer slice (derivation starts from the north-dropped remainder) — no red phase; the mutation is discriminating (a kept north panel would win fill-by-generation and shift the yields). Co-Authored-By: Claude Fable 5 --- .../modelling/test_solar_config_selection.py | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/tests/domain/modelling/test_solar_config_selection.py b/tests/domain/modelling/test_solar_config_selection.py index 53b6dca18..6aae14dd9 100644 --- a/tests/domain/modelling/test_solar_config_selection.py +++ b/tests/domain/modelling/test_solar_config_selection.py @@ -304,3 +304,47 @@ def test_sub_ladder_rungs_fill_from_the_highest_yield_segment_first() -> None: (225.0, 2), (135.0, 1), } + + +def test_sub_ladder_rungs_never_take_from_north_planes() -> None: + # ADR-0058 — derivation starts from the north-dropped remainder: a north + # plane never contributes panels to a Sub-Ladder rung, even when its + # per-panel yield would win the fill-by-generation ordering. + + # Arrange — smallest rung = a due-north panel (900/panel, best yield) plus + # a south plane (4 panels, 1278); max 5 → cap 3.5; after the north drop + # the 4-panel south remainder still exceeds the cap → Sub-Ladder fires. + north = SolarRoofSegment( + segment_index=0, + panels_count=1, + azimuth_degrees=2.0, + pitch_degrees=30.0, + yearly_energy_dc_kwh=900.0, + ) + south = SolarRoofSegment( + segment_index=1, + panels_count=4, + azimuth_degrees=180.0, + pitch_degrees=30.0, + yearly_energy_dc_kwh=1278.0, + ) + potential = SolarPotential( + panel_capacity_watts=400.0, + max_array_panels_count=5, + configurations=( + SolarPanelConfiguration( + panels_count=5, + yearly_energy_dc_kwh=2178.0, + segments=(north, south), + ), + ), + ) + + # Act + configs = select_conservative_configs(potential) + + # Assert — rungs 2 and 3 built from the south plane alone, pro-rata off + # its 1278 (639 / 958.5); the 900-yield north panel never appears. + assert [c.panels_count for c in configs] == [2, 3] + assert [c.yearly_energy_dc_kwh for c in configs] == [639.0, 958.5] + assert all(s.azimuth_degrees == 180.0 for c in configs for s in c.segments) From 1756976d7eef972149b262638bcdcbc34f9ed638 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 7 Jul 2026 22:07:39 +0000 Subject: [PATCH 07/55] =?UTF-8?q?A=20roof=20capped=20below=20two=20panels?= =?UTF-8?q?=20is=20still=20offered=20no=20PV=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pins the ADR-0058 install floor delivered with the tracer slice (rung_max < 2 returns empty) — no red phase. Co-Authored-By: Claude Fable 5 --- .../modelling/test_solar_config_selection.py | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/tests/domain/modelling/test_solar_config_selection.py b/tests/domain/modelling/test_solar_config_selection.py index 6aae14dd9..01dba8533 100644 --- a/tests/domain/modelling/test_solar_config_selection.py +++ b/tests/domain/modelling/test_solar_config_selection.py @@ -348,3 +348,28 @@ def test_sub_ladder_rungs_never_take_from_north_planes() -> None: assert [c.panels_count for c in configs] == [2, 3] assert [c.yearly_energy_dc_kwh for c in configs] == [639.0, 958.5] assert all(s.azimuth_degrees == 180.0 for c in configs for s in c.segments) + + +def test_roof_capped_below_two_panels_still_yields_no_configs() -> None: + # ADR-0058 — the install floor: a roof whose cap resolves below 2 panels + # is never offered PV. Max 2 → cap 1.4 < 2; the ladder cliff is the + # correct outcome here, not a Sub-Ladder rung. + + # Arrange + potential = SolarPotential( + panel_capacity_watts=400.0, + max_array_panels_count=2, + configurations=( + SolarPanelConfiguration( + panels_count=2, + yearly_energy_dc_kwh=640.0, + segments=(_segment(2, 180.0, 640.0),), + ), + ), + ) + + # Act + configs = select_conservative_configs(potential) + + # Assert + assert configs == () From 024a01b1286271ee85f012ed69b8b2ea1d8b2ff4 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 7 Jul 2026 22:08:28 +0000 Subject: [PATCH 08/55] =?UTF-8?q?The=20Dwelling-Roof=20Cap=20bounds=20Sub-?= =?UTF-8?q?Ladder=20rungs=20too=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pins that the unchanged cap regime (min of 0.7×Google and the ADR-0038 budget) flows into ADR-0058 derivation — no red phase; the tracer implementation passed the already-min'd cap through. Co-Authored-By: Claude Fable 5 --- .../modelling/test_solar_config_selection.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/tests/domain/modelling/test_solar_config_selection.py b/tests/domain/modelling/test_solar_config_selection.py index 01dba8533..eb4f7f757 100644 --- a/tests/domain/modelling/test_solar_config_selection.py +++ b/tests/domain/modelling/test_solar_config_selection.py @@ -373,3 +373,20 @@ def test_roof_capped_below_two_panels_still_yields_no_configs() -> None: # Assert assert configs == () + + +def test_dwelling_roof_cap_bounds_the_sub_ladder_too() -> None: + # ADR-0058 — the cap regime is unchanged: when the Dwelling-Roof Cap + # (ADR-0038) resolves BELOW the 0.7×Google cap, it bounds the Sub-Ladder + # rungs as well — a conflated Google roof can't inflate even a small array. + + # Arrange — max 6 (0.7 cap = 4.2) with Google's ladder starting at 4; + # a ~8.5 m² dwelling roof caps at ≈2.5 panels → only the 2-panel rung. + potential = _potential_with_panel_dims(max_panels=6, panel_counts=(4,)) + + # Act + configs = select_conservative_configs(potential, dwelling_roof_area_m2=8.5) + + # Assert — one rung at 2 panels, pro-rata off the 4-panel rung's 400. + assert [c.panels_count for c in configs] == [2] + assert [c.yearly_energy_dc_kwh for c in configs] == [200.0] From 5d0a7751e66fea332a26741818935a942236a7ef Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 7 Jul 2026 22:09:18 +0000 Subject: [PATCH 09/55] =?UTF-8?q?A=20small-roof=20house=20gets=20Sub-Ladde?= =?UTF-8?q?r=20solar=20Options=20instead=20of=20nothing=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit End-to-end pin at the generator seam (recommend_solar) — passed on first run since option construction is unchanged by design (ADR-0058). Co-Authored-By: Claude Fable 5 --- .../modelling/test_solar_recommendation.py | 49 ++++++++++++++++++- 1 file changed, 48 insertions(+), 1 deletion(-) diff --git a/tests/domain/modelling/test_solar_recommendation.py b/tests/domain/modelling/test_solar_recommendation.py index 43337ab51..05cc1c369 100644 --- a/tests/domain/modelling/test_solar_recommendation.py +++ b/tests/domain/modelling/test_solar_recommendation.py @@ -14,7 +14,11 @@ from domain.geospatial.planning_restrictions import PlanningRestrictions from domain.modelling.generators.solar_recommendation import recommend_solar from domain.modelling.product import Product from domain.modelling.recommendation import Recommendation -from domain.modelling.solar_potential import SolarPotential +from domain.modelling.solar_potential import ( + SolarPanelConfiguration, + SolarPotential, + SolarRoofSegment, +) from repositories.product.product_repository import ProductRepository from tests.domain.modelling._elmhurst_recommendation import ( parse_recommendation_summary, @@ -242,3 +246,46 @@ def test_infeasible_potential_yields_no_recommendation() -> None: # Act / Assert assert recommend_solar(baseline, _StubProducts(), north_only) is None + + +def test_small_roof_house_gets_sub_ladder_options_instead_of_nothing() -> None: + # ADR-0058 end-to-end at the generator seam — the 824/1278 audit's + # property-750701 shape: Google max 5 (cap 3.5), ladder starting at 4, + # which yielded NO recommendation before Sub-Ladder Configurations. + + # Arrange — an eligible house with a small-roof Solar Potential. + baseline = _eligible_house() + small_roof = SolarPotential( + panel_capacity_watts=400.0, + max_array_panels_count=5, + configurations=( + SolarPanelConfiguration( + panels_count=4, + yearly_energy_dc_kwh=1278.0, + segments=( + SolarRoofSegment( + segment_index=0, + panels_count=4, + azimuth_degrees=180.0, + pitch_degrees=30.0, + yearly_energy_dc_kwh=1278.0, + ), + ), + ), + ), + ) + + # Act + recommendation: Optional[Recommendation] = recommend_solar( + baseline, _StubProducts(), small_roof + ) + + # Assert — rungs 2 and 3 x {no battery, battery} = 4 competing Options, + # each overlay's array peak power matching its rung (0.8 / 1.2 kWp). + assert recommendation is not None + assert len(recommendation.options) == 4 + peak_powers = sorted( + sum(a.peak_power for a in o.overlay.solar.photovoltaic_arrays) + for o in recommendation.options + ) + assert peak_powers == [0.8, 0.8, 1.2, 1.2] From fbedf4541b0423cc42241477a2b18ead0ba60e56 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Tue, 7 Jul 2026 22:10:37 +0000 Subject: [PATCH 10/55] =?UTF-8?q?A=20small=20roof=20derives=20Sub-Ladder?= =?UTF-8?q?=20rungs=20below=20Google's=20smallest=20config=20=F0=9F=9F=AA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit North-drop now runs once: the selection computes the usable (north-dropped) ladder, feasibility filters it, and Sub-Ladder derivation consumes it. Co-Authored-By: Claude Fable 5 --- .../generators/solar_recommendation.py | 26 ++++++++----------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/domain/modelling/generators/solar_recommendation.py b/domain/modelling/generators/solar_recommendation.py index 6b5b18f4a..110599a0b 100644 --- a/domain/modelling/generators/solar_recommendation.py +++ b/domain/modelling/generators/solar_recommendation.py @@ -228,25 +228,18 @@ def _resized_to( def _sub_ladder_configs( - potential: SolarPotential, panel_cap: float + usable: list[SolarPanelConfiguration], panel_cap: float ) -> tuple[SolarPanelConfiguration, ...]: """Sub-Ladder Configurations (ADR-0058): rungs derived BELOW Google's smallest offered config, for when nothing on the ladder fits under the caps — a small roof gets the array its caps allow instead of no PV at all. One rung per whole panel count from the install floor (2) up to - ``floor(panel_cap)``, derived from the smallest north-dropped config.""" + ``floor(panel_cap)``, derived from the smallest of the (already + north-dropped) ``usable`` configs.""" rung_max: int = math.floor(panel_cap) - if rung_max < _MIN_SUB_LADDER_PANELS: - return () - smallest: Optional[SolarPanelConfiguration] = None - for config in potential.configurations: - dropped = _drop_north_segments(config) - if not dropped.segments: - continue - if smallest is None or dropped.panels_count < smallest.panels_count: - smallest = dropped - if smallest is None: + if rung_max < _MIN_SUB_LADDER_PANELS or not usable: return () + smallest: SolarPanelConfiguration = min(usable, key=lambda c: c.panels_count) return tuple( _resized_to(smallest, panels) for panels in range(_MIN_SUB_LADDER_PANELS, rung_max + 1) @@ -274,16 +267,19 @@ def select_conservative_configs( ) if roof_cap is not None: panel_cap = min(panel_cap, roof_cap) - feasible: list[SolarPanelConfiguration] = [ + usable: list[SolarPanelConfiguration] = [ trimmed for config in potential.configurations for trimmed in (_drop_north_segments(config),) - if trimmed.segments and trimmed.panels_count <= panel_cap + if trimmed.segments + ] + feasible: list[SolarPanelConfiguration] = [ + config for config in usable if config.panels_count <= panel_cap ] if not feasible: # Nothing on Google's ladder fits under the caps — a small roof, not # an unusable one. Derive Sub-Ladder Configurations (ADR-0058). - return _sub_ladder_configs(potential, panel_cap) + return _sub_ladder_configs(usable, panel_cap) # Collapse rungs that trimmed to the same usable size (north-drop can make # distinct original rungs coincide), keeping the higher-generation layout — # the Optimiser's dial is panel count (≈ kWp ≈ cost), so duplicates of the From 6d001d650efb4c07e7579d0b31e2fca0e50e4abd Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 08:54:43 +0000 Subject: [PATCH 11/55] =?UTF-8?q?ADR-0059/0060:=20Bulk=20Document=20Downlo?= =?UTF-8?q?ad=20design=20=E2=80=94=20SES-SMTP=20email=20sender=20+=20cappe?= =?UTF-8?q?d=20best-effort=20Download=20Package=20=F0=9F=9F=AA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Grilled 2026-07-08. Locks: app-owned-task + attach-mode lane (ADR-0055); FastAPI route resolves the requesting user (dbId -> UserModel.email) and pins it + the resolved property_id set into tasks.inputs; one capped Download Package = one sub_task; landlord_property_id matching (property_id a future gap); address-named folders (hubspot-enriched), latest per Document Type, null-type skipped+reported; best-effort (fail only on infra / wholly-empty); URL to sub_task.outputs + email; DATA_BUCKET/bulk-downloads/ ~7-day lifecycle. Backend-sent email via a repositories/email port + infrastructure/email SES-SMTP adapter. Glossary: Bulk Document Download, Download Package, Document Type. Co-Authored-By: Claude Fable 5 --- CONTEXT.md | 12 +++ ...059-outbound-email-via-ses-smtp-adapter.md | 79 ++++++++++++++++ ...60-bulk-document-download-package-model.md | 92 +++++++++++++++++++ 3 files changed, 183 insertions(+) create mode 100644 docs/adr/0059-outbound-email-via-ses-smtp-adapter.md create mode 100644 docs/adr/0060-bulk-document-download-package-model.md diff --git a/CONTEXT.md b/CONTEXT.md index caaf3029e..26c65a00c 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -99,6 +99,18 @@ _Avoid_: ventilation report, audit report, PAS ventilation (that is the external An externally-uploaded ventilation survey document produced by a human assessor and ingested from an external source (e.g. Coordination Hub). Recorded in `uploaded_files` with `file_type = PAS_2023_VENTILATION`. Distinct from a **Ventilation Audit**, which is machine-generated from MagicPlan floor plan data. _Avoid_: ventilation audit (that is the generated output) +**Document Type**: +The category of an uploaded document — the `uploaded_files.file_type` enum (photo_pack, site_note, the pas_2023_* / ecmk_* families, magic_plan_json, mcs_compliance_certificate, ventilation_audit, other, …). Nullable on the row. In a **Download Package** exactly one file per Document Type is included per property — the newest by `s3_upload_timestamp`; a row with a null Document Type is skipped and reported (ADR-0060). +_Avoid_: document category, file kind, doc class + +**Download Package**: +The single ZIP archive a **Bulk Document Download** produces: one folder per property (named by human-readable address, enriched from the hubspot deals data, with `landlord_property_id` appended for uniqueness), each folder holding the latest file of each **Document Type** held for that property. Built best-effort — properties/documents that don't resolve are skipped and listed in the run's `sub_task.outputs`, not failed (ADR-0060). Stored under `DATA_BUCKET/bulk-downloads/` with a ~7-day lifecycle; delivered as a 60-minute presigned URL. +_Avoid_: export (that is the plan/scenario XLSX export), bundle, archive (ambiguous), zip (the format, not the concept) + +**Bulk Document Download**: +The feature/job that assembles a **Download Package** for a chosen set of properties (a hand-picked list or a whole portfolio) and emails the requester a link. Runs on the app-owned-task + attach-mode lane (ADR-0055): the FastAPI route pins the resolved `property_id` set and the recipient email into `tasks.inputs`, and the `applications/bulk_document_download` Lambda builds the package, writes the URL to `sub_task.outputs`, and emails it (ADR-0059/0060). Files are matched by `landlord_property_id` (the missing `property_id` on `uploaded_files` is a known future gap). Selection is capped by property count at the trigger. +_Avoid_: document export, bulk export, file dump + ### Source data **Site Notes**: diff --git a/docs/adr/0059-outbound-email-via-ses-smtp-adapter.md b/docs/adr/0059-outbound-email-via-ses-smtp-adapter.md new file mode 100644 index 000000000..fb2946eae --- /dev/null +++ b/docs/adr/0059-outbound-email-via-ses-smtp-adapter.md @@ -0,0 +1,79 @@ +--- +status: accepted +--- + +# Outbound email is sent from the backend via an SES-SMTP adapter, with the recipient threaded from the trigger route + +The Bulk Document Download feature (ADR-0060) must email a completed package's +link to the user who requested it. This is the **first outbound email in the +Python backend** — there is no `smtplib`/SES/SendGrid code anywhere today. Two +facts force a decision rather than a copy: + +1. **Where the email is sent from.** The front end can send email, but the + package is produced **asynchronously in a Lambda**, long after the trigger + request has returned `202`. The completion event exists only on the backend. +2. **Who to email.** Auth today only *gates* the FastAPI routes + (`validate_token` returns the raw token, not the user); neither the handler + nor the `Task`/`SubTask` model captures a user. The requesting identity is + provable (the NextAuth JWT carries `dbId` → `UserModel.email`) but is not + currently threaded anywhere. + +## Decision + +**The backend sends the email, through a DDD port + SES-SMTP adapter, to a +recipient the trigger route resolves and pins into `tasks.inputs`.** + +- **Send from the backend, on completion.** The job that finishes the work + owns the notification — the orchestrator calls an injected email port after + the package is uploaded and the URL minted. The front end is not asked to + poll-then-send. +- **Port + adapter (hexagonal).** A `repositories/email/` port + (`EmailSender` — `send(to, subject, body/html)`) with an + `infrastructure/email/ses_smtp_email_sender.py` adapter over `smtplib`, using + the **existing SES SMTP** setup (terraform `modules/ses`, domain + `domna.homes`, credentials already in Secrets Manager + `${stage}/ses/smtp_credentials`). SMTP, not the SES SendEmail API, because + that is what the provisioned infrastructure exposes. The port is injected into + the orchestrator so the send is testable behind an interface and the transport + is swappable. +- **Recipient threaded at the route, pinned in `inputs`.** The FastAPI trigger + route resolves the authenticated user (`dbId` → `UserModel.email`, via the + existing `get_user`) and writes the email into the `tasks.inputs` JSON at task + creation. The Lambda reads it from `inputs` and never touches auth. This + graduates the route from *gating* the token to *injecting* the user — a + deliberate, contained change to `backend/app/dependencies.py` usage at the one + new route. +- **Config follows the Lambda convention.** SMTP host/username/password reach + the Lambda as environment variables (sourced from the Secrets Manager entry in + terraform), read via `os.environ`/`PostgresConfig`-style config — not + `backend/app/config.py get_settings()`, which is the legacy app boundary. + +## Considered options + +- **Front end sends the email.** Rejected: the completion moment lives in the + Lambda; making the FE responsible means it must poll task status to + completion and then send — more moving parts, and the notification would lag + or be missed if the user closes the tab. Backend-sends keeps the trigger and + the notification on the same side of the queue. +- **SES SendEmail API (boto3 `ses`) instead of SMTP.** Rejected for now: the + provisioned infrastructure issues SMTP credentials; using the API would need + new IAM + identity wiring. Revisit if a templated/bulk API becomes worthwhile. +- **Resolve the recipient in the Lambda from `portfolio_id` → `PortfolioUsers`.** + Rejected: a portfolio has many users, so the join cannot identify *the + requester* — real risk of emailing the wrong person or several. The requester + is only unambiguous at the authenticated route. +- **FE passes the recipient email in the trigger body.** Rejected: trusts the + client for an identity the JWT already proves, and can drift from the + logged-in user. + +## Consequences + +- A reusable `EmailSender` capability now exists for future backend + notifications (not just this feature). +- The one new FastAPI route must inject the user (not merely gate) and persist + the email in `inputs`; `Task`/`SubTask` still gain no user column — the + recipient lives in the request payload, scoped to the job. +- Email content is plain for v1 (a link, a 60-minute expiry note, and the + skipped-document summary from `sub_task.outputs`); templating can follow. +- A new terraform env-var surface (SMTP creds from the existing Secrets Manager + entry) is added to the Lambda; no new SES infrastructure is required. diff --git a/docs/adr/0060-bulk-document-download-package-model.md b/docs/adr/0060-bulk-document-download-package-model.md new file mode 100644 index 000000000..5e234bc0a --- /dev/null +++ b/docs/adr/0060-bulk-document-download-package-model.md @@ -0,0 +1,92 @@ +--- +status: accepted (builds on ADR-0055, ADR-0059) +--- + +# Bulk Document Download builds one capped, best-effort Download Package per request + +Users need to pull the documents held in `uploaded_files` for many properties +at once — per property, the latest file of each **Document Type** — as a single +archive, without clicking through the UI file by file. The request is initiated +in the front end, can span a hand-picked set of properties or a whole +portfolio, and finishes with a link emailed to the requester (ADR-0059). + +`uploaded_files` links to a property by `uprn` **or** `landlord_property_id` +(both nullable) and has **no `property_id`** column; `file_type` (the Document +Type) is itself nullable; and files live across arbitrary buckets +(`s3_file_bucket` per row). + +## Decision + +**A request produces exactly one Download Package: a ZIP of one folder per +property, each holding the latest file of each Document Type — built +best-effort, size-capped at the trigger, on the app-owned-task + attach-mode +lane (ADR-0055).** + +- **Trigger & lifecycle (ADR-0055).** The FastAPI route creates the `tasks` + row, resolves the selection to a concrete `property_id` set (expanding "all + properties" of a portfolio), **pins that set plus the recipient email into + `tasks.inputs`** (TEXT/JSON — a large id list is why this lives in `inputs`, + not the 256 KB SQS body), pre-creates **one** `sub_task`, and drops one SQS + message (`task_id`, `sub_task_id`). The new `applications/bulk_document_download` + Lambda runs in **attach mode**; `TaskOrchestrator` owns status + roll-up. +- **One package = one sub_task.** No fan-out: the job streams each document out + of its own `s3_file_bucket` into a single ZIP in `/tmp` and multipart-uploads + it. One artifact, one URL, one email. +- **Selection cap at the route.** Reject `> N` properties synchronously with a + legible "narrow your selection" error (property count, not a document/byte + query — cheap, no S3 on the hot path). A `/tmp` size budget inside the Lambda + is a backstop that fails the sub_task with a clear reason if a pathological + selection still overflows. `N` starts at a conservative, tunable value. +- **Matching on `landlord_property_id`.** Files are gathered by + `landlord_property_id`; the missing `property_id` on `uploaded_files` is a + **known future gap**, out of scope here. Property display info (the folder + name) is enriched from the hubspot deals data + (`repositories/hubspot_deals/`). +- **Layout.** One folder per property named by human-readable **address** + (hubspot-enriched), with `landlord_property_id` appended for uniqueness; + inside, one file per Document Type = the newest by `s3_upload_timestamp`. + Rows with a **null Document Type are skipped** and listed in the run output. +- **Best-effort failure contract.** Build from whatever resolves. Skipped + properties (no documents) and skipped null-type files are recorded in + `sub_task.outputs`. The run **fails only** on an infrastructure error + (S3/DB/zip/upload/email) or when the whole selection yields **zero + documents**. The email carries an "N properties, M documents, X skipped" + summary. +- **Delivery — both channels.** The presigned URL (60-minute expiry) and the + skip-summary are written to `sub_task.outputs` (the FE already polls task + status and can show the link) **and** emailed (ADR-0059). +- **Retention.** Packages are written to `DATA_BUCKET` under a + `bulk-downloads/` prefix with an S3 lifecycle rule expiring objects after + ~7 days — longer than the 60-minute URL (so a link can be re-issued without + rebuilding) but short enough that bundles never accumulate. + +## Considered options + +- **Fan out into per-batch sub_tasks / multiple ZIPs** (like the modelling + run). Rejected: the user would receive N links / N emails and "the download" + would stop being one file; the capped-single-package model keeps the artifact + and the notification singular. +- **Cap on resolved document count / bytes.** Rejected for v1: it adds + synchronous DB + S3 HEAD work to the trigger and couples the route to the + packaging logic; a property-count cap is cheap and predictable. +- **Strict "all-or-nothing" packaging.** Rejected: one unreadable row or a + property with no documents would deny the entire package; best-effort + + reporting matches "include documents and properties where they exist". +- **Match on `uprn`.** Deferred: `landlord_property_id` is the agreed key and + hubspot enrichment is keyed to the deal/property; revisit if/when the + `property_id` gap on `uploaded_files` is closed. + +## Consequences + +- New DDD pieces: `applications/bulk_document_download/` (thin handler + + trigger body), `orchestration/bulk_document_download_orchestrator.py`, + packaging rules in `domain/`, a `landlord_property_id`-keyed + "latest-per-Document-Type" query on the uploaded-file repository, a + `generate_presigned_url` + multipart upload on `S3Client`, and the email + port/adapter from ADR-0059. The only `backend/` touch is the trigger route. +- The stored, pinned `property_id` set makes a run reproducible even if the + portfolio changes between trigger and execution. +- `N`, the `/tmp` size budget, and the 7-day lifecycle are tunable knobs, not + load-bearing invariants. +- Closing the `uploaded_files.property_id` gap later would let matching move off + `landlord_property_id` without changing the package model. From 28b7b433725158e1cc91dd4e3f4b757eeb85969c Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:04:19 +0000 Subject: [PATCH 12/55] =?UTF-8?q?Keep=20only=20the=20latest=20file=20of=20?= =?UTF-8?q?a=20Document=20Type=20in=20each=20property's=20folder=20?= =?UTF-8?q?=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- domain/bulk_document_download/__init__.py | 0 domain/bulk_document_download/package_plan.py | 66 +++++++++++++++++++ .../domain/bulk_document_download/__init__.py | 0 .../test_package_plan.py | 48 ++++++++++++++ 4 files changed, 114 insertions(+) create mode 100644 domain/bulk_document_download/__init__.py create mode 100644 domain/bulk_document_download/package_plan.py create mode 100644 tests/domain/bulk_document_download/__init__.py create mode 100644 tests/domain/bulk_document_download/test_package_plan.py diff --git a/domain/bulk_document_download/__init__.py b/domain/bulk_document_download/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/domain/bulk_document_download/package_plan.py b/domain/bulk_document_download/package_plan.py new file mode 100644 index 000000000..6d08f6171 --- /dev/null +++ b/domain/bulk_document_download/package_plan.py @@ -0,0 +1,66 @@ +"""The Download Package plan (ADR-0060). + +Pure logic: given the documents resolved for a set of properties, decide what +goes where in the ZIP — one folder per property, the latest file of each +Document Type — and which rows are skipped and reported. No I/O: the +orchestrator does the S3/DB work; this module only lays out the archive. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +from typing import Optional, Sequence + + +@dataclass(frozen=True) +class ResolvedDocument: + """One `uploaded_files` row resolved for packaging: its property identity + (matched by `landlord_property_id`, ADR-0060), the display address enriched + from the hubspot deals data, its Document Type (`file_type`, nullable), the + S3 location to read it from, and when it was uploaded (for latest-per-type). + """ + + landlord_property_id: str + address: str + document_type: Optional[str] + s3_bucket: str + s3_key: str + uploaded_at: datetime + + +@dataclass(frozen=True) +class PackageEntry: + """One file to place in the Download Package: where it lands in the ZIP and + the S3 object to stream in from.""" + + zip_path: str + s3_bucket: str + s3_key: str + + +@dataclass(frozen=True) +class SkippedDocument: + """A resolved row left out of the package, with the reason (surfaced in the + run's `sub_task.outputs`, ADR-0060).""" + + landlord_property_id: str + s3_key: str + reason: str + + +@dataclass(frozen=True) +class DownloadPackagePlan: + """The laid-out archive: the files to include and the rows skipped.""" + + entries: tuple[PackageEntry, ...] + skipped: tuple[SkippedDocument, ...] + + +def plan_download_package( + documents: Sequence[ResolvedDocument], +) -> DownloadPackagePlan: + """Lay out a Download Package from resolved documents (ADR-0060): one folder + per property, the latest file of each Document Type, null-Document-Type rows + skipped and reported.""" + raise NotImplementedError diff --git a/tests/domain/bulk_document_download/__init__.py b/tests/domain/bulk_document_download/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/domain/bulk_document_download/test_package_plan.py b/tests/domain/bulk_document_download/test_package_plan.py new file mode 100644 index 000000000..2271aefd0 --- /dev/null +++ b/tests/domain/bulk_document_download/test_package_plan.py @@ -0,0 +1,48 @@ +"""The Download Package plan (ADR-0060) — pure layout of the ZIP from resolved +documents: one folder per property, latest file of each Document Type, +null-Document-Type rows skipped and reported.""" + +from datetime import datetime, timezone + +from domain.bulk_document_download.package_plan import ( + ResolvedDocument, + plan_download_package, +) + + +def _doc( + *, + landlord_property_id: str = "LP1", + address: str = "12 Oak Street", + document_type: str = "site_note", + s3_key: str = "a.pdf", + uploaded_at: datetime = datetime(2026, 1, 1, tzinfo=timezone.utc), +) -> ResolvedDocument: + return ResolvedDocument( + landlord_property_id=landlord_property_id, + address=address, + document_type=document_type, + s3_bucket="src-bucket", + s3_key=s3_key, + uploaded_at=uploaded_at, + ) + + +def test_keeps_only_the_latest_file_of_a_document_type_in_the_property_folder() -> None: + # Arrange — two site_note files for the same property, uploaded at + # different times; only the newest should be packaged. + older = _doc( + s3_key="old.pdf", uploaded_at=datetime(2026, 1, 1, tzinfo=timezone.utc) + ) + newer = _doc( + s3_key="new.pdf", uploaded_at=datetime(2026, 6, 1, tzinfo=timezone.utc) + ) + + # Act + plan = plan_download_package([older, newer]) + + # Assert — one entry, the newer file, foldered under the property. + assert len(plan.entries) == 1 + entry = plan.entries[0] + assert entry.s3_key == "new.pdf" + assert entry.zip_path.startswith("12 Oak Street (LP1)/") From 810e91096107de202464302cdcd6dc7640456dc8 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:04:58 +0000 Subject: [PATCH 13/55] =?UTF-8?q?Keep=20only=20the=20latest=20file=20of=20?= =?UTF-8?q?a=20Document=20Type=20in=20each=20property's=20folder=20?= =?UTF-8?q?=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- domain/bulk_document_download/package_plan.py | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/domain/bulk_document_download/package_plan.py b/domain/bulk_document_download/package_plan.py index 6d08f6171..64e57950d 100644 --- a/domain/bulk_document_download/package_plan.py +++ b/domain/bulk_document_download/package_plan.py @@ -63,4 +63,21 @@ def plan_download_package( """Lay out a Download Package from resolved documents (ADR-0060): one folder per property, the latest file of each Document Type, null-Document-Type rows skipped and reported.""" - raise NotImplementedError + latest_by_group: dict[tuple[str, str], ResolvedDocument] = {} + for document in documents: + assert document.document_type is not None + group = (document.landlord_property_id, document.document_type) + incumbent = latest_by_group.get(group) + if incumbent is None or document.uploaded_at > incumbent.uploaded_at: + latest_by_group[group] = document + + entries: list[PackageEntry] = [ + PackageEntry( + zip_path=f"{document.address} ({document.landlord_property_id})/" + f"{document.document_type}", + s3_bucket=document.s3_bucket, + s3_key=document.s3_key, + ) + for document in latest_by_group.values() + ] + return DownloadPackagePlan(entries=tuple(entries), skipped=()) From 34c48d4adc591c88b5659029d78fa3399b25ce47 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:05:44 +0000 Subject: [PATCH 14/55] =?UTF-8?q?Skip=20and=20report=20a=20document=20with?= =?UTF-8?q?=20no=20Document=20Type=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../test_package_plan.py | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/tests/domain/bulk_document_download/test_package_plan.py b/tests/domain/bulk_document_download/test_package_plan.py index 2271aefd0..a81e84171 100644 --- a/tests/domain/bulk_document_download/test_package_plan.py +++ b/tests/domain/bulk_document_download/test_package_plan.py @@ -3,6 +3,7 @@ documents: one folder per property, latest file of each Document Type, null-Document-Type rows skipped and reported.""" from datetime import datetime, timezone +from typing import Optional from domain.bulk_document_download.package_plan import ( ResolvedDocument, @@ -14,7 +15,7 @@ def _doc( *, landlord_property_id: str = "LP1", address: str = "12 Oak Street", - document_type: str = "site_note", + document_type: Optional[str] = "site_note", s3_key: str = "a.pdf", uploaded_at: datetime = datetime(2026, 1, 1, tzinfo=timezone.utc), ) -> ResolvedDocument: @@ -46,3 +47,20 @@ def test_keeps_only_the_latest_file_of_a_document_type_in_the_property_folder() entry = plan.entries[0] assert entry.s3_key == "new.pdf" assert entry.zip_path.startswith("12 Oak Street (LP1)/") + + +def test_skips_and_reports_a_document_with_no_document_type() -> None: + # Arrange — one packable file and one row whose Document Type is null + # (file_type is nullable on uploaded_files). + packable = _doc(document_type="site_note", s3_key="note.pdf") + untyped = _doc(document_type=None, s3_key="mystery.bin") + + # Act + plan = plan_download_package([packable, untyped]) + + # Assert — the untyped row is left out of the archive and reported instead. + assert [e.s3_key for e in plan.entries] == ["note.pdf"] + assert len(plan.skipped) == 1 + skipped = plan.skipped[0] + assert skipped.s3_key == "mystery.bin" + assert skipped.landlord_property_id == "LP1" From bd928828e33f55b67548ed6bc8c56b6ff180436c Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:06:26 +0000 Subject: [PATCH 15/55] =?UTF-8?q?Skip=20and=20report=20a=20document=20with?= =?UTF-8?q?=20no=20Document=20Type=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- domain/bulk_document_download/package_plan.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/domain/bulk_document_download/package_plan.py b/domain/bulk_document_download/package_plan.py index 64e57950d..bcd122aad 100644 --- a/domain/bulk_document_download/package_plan.py +++ b/domain/bulk_document_download/package_plan.py @@ -64,8 +64,17 @@ def plan_download_package( per property, the latest file of each Document Type, null-Document-Type rows skipped and reported.""" latest_by_group: dict[tuple[str, str], ResolvedDocument] = {} + skipped: list[SkippedDocument] = [] for document in documents: - assert document.document_type is not None + if document.document_type is None: + skipped.append( + SkippedDocument( + landlord_property_id=document.landlord_property_id, + s3_key=document.s3_key, + reason="null_document_type", + ) + ) + continue group = (document.landlord_property_id, document.document_type) incumbent = latest_by_group.get(group) if incumbent is None or document.uploaded_at > incumbent.uploaded_at: @@ -80,4 +89,4 @@ def plan_download_package( ) for document in latest_by_group.values() ] - return DownloadPackagePlan(entries=tuple(entries), skipped=()) + return DownloadPackagePlan(entries=tuple(entries), skipped=tuple(skipped)) From 63933db529d1b9b70dd2d13bcc073981b47597dc Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:06:51 +0000 Subject: [PATCH 16/55] =?UTF-8?q?Name=20the=20packaged=20file=20by=20Docum?= =?UTF-8?q?ent=20Type=20with=20the=20original=20extension=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../bulk_document_download/test_package_plan.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/tests/domain/bulk_document_download/test_package_plan.py b/tests/domain/bulk_document_download/test_package_plan.py index a81e84171..ce9c0541a 100644 --- a/tests/domain/bulk_document_download/test_package_plan.py +++ b/tests/domain/bulk_document_download/test_package_plan.py @@ -64,3 +64,15 @@ def test_skips_and_reports_a_document_with_no_document_type() -> None: skipped = plan.skipped[0] assert skipped.s3_key == "mystery.bin" assert skipped.landlord_property_id == "LP1" + + +def test_packaged_filename_is_the_document_type_with_the_original_extension() -> None: + # Arrange — a photo pack stored under a prefixed key with a .zip extension; + # the file inside the archive should be named by type but stay openable. + doc = _doc(document_type="photo_pack", s3_key="surveys/2026/pack.zip") + + # Act + plan = plan_download_package([doc]) + + # Assert + assert plan.entries[0].zip_path == "12 Oak Street (LP1)/photo_pack.zip" From b89b9fe94433bee2a71fcc585d9b3a2b51ac3dbe Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:07:25 +0000 Subject: [PATCH 17/55] =?UTF-8?q?Name=20the=20packaged=20file=20by=20Docum?= =?UTF-8?q?ent=20Type=20with=20the=20original=20extension=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- domain/bulk_document_download/package_plan.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/domain/bulk_document_download/package_plan.py b/domain/bulk_document_download/package_plan.py index bcd122aad..535928596 100644 --- a/domain/bulk_document_download/package_plan.py +++ b/domain/bulk_document_download/package_plan.py @@ -8,6 +8,7 @@ orchestrator does the S3/DB work; this module only lays out the archive. from __future__ import annotations +import posixpath from dataclasses import dataclass from datetime import datetime from typing import Optional, Sequence @@ -57,6 +58,14 @@ class DownloadPackagePlan: skipped: tuple[SkippedDocument, ...] +def _extension_of(s3_key: str) -> str: + """The file extension of an S3 key (``.pdf``, ``.zip``, …), empty when the + key's basename has none. Used to keep the packaged file openable while + naming it by Document Type.""" + _root, ext = posixpath.splitext(posixpath.basename(s3_key)) + return ext + + def plan_download_package( documents: Sequence[ResolvedDocument], ) -> DownloadPackagePlan: @@ -83,7 +92,7 @@ def plan_download_package( entries: list[PackageEntry] = [ PackageEntry( zip_path=f"{document.address} ({document.landlord_property_id})/" - f"{document.document_type}", + f"{document.document_type}{_extension_of(document.s3_key)}", s3_bucket=document.s3_bucket, s3_key=document.s3_key, ) From e906c8fc98d056b514baccff53391d4b39de7bfe Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:15:44 +0000 Subject: [PATCH 18/55] =?UTF-8?q?Scope=20latest-per-Document-Type=20per=20?= =?UTF-8?q?property,=20each=20in=20its=20own=20folder=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pins behavior delivered with the tracer slice (the group key is (landlord_property_id, document_type)) — no red phase; discriminating: a global-by-type dedupe would drop one property's file. Co-Authored-By: Claude Fable 5 --- .../bulk_document_download/test_package_plan.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tests/domain/bulk_document_download/test_package_plan.py b/tests/domain/bulk_document_download/test_package_plan.py index ce9c0541a..4bcddc83c 100644 --- a/tests/domain/bulk_document_download/test_package_plan.py +++ b/tests/domain/bulk_document_download/test_package_plan.py @@ -76,3 +76,19 @@ def test_packaged_filename_is_the_document_type_with_the_original_extension() -> # Assert assert plan.entries[0].zip_path == "12 Oak Street (LP1)/photo_pack.zip" + + +def test_latest_per_type_is_scoped_per_property_each_gets_its_own_folder() -> None: + # Arrange — two DIFFERENT properties, each with a site_note. Latest-per-type + # must not dedupe across properties: both survive, in their own folders. + first = _doc(landlord_property_id="LP1", address="12 Oak Street", s3_key="a.pdf") + second = _doc(landlord_property_id="LP2", address="9 Elm Road", s3_key="b.pdf") + + # Act + plan = plan_download_package([first, second]) + + # Assert + assert {e.zip_path for e in plan.entries} == { + "12 Oak Street (LP1)/site_note.pdf", + "9 Elm Road (LP2)/site_note.pdf", + } From ba13300ee10eb70bb25e6f0d8088b4b0f6ff6c90 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:18:10 +0000 Subject: [PATCH 19/55] =?UTF-8?q?Generate=20a=20presigned=20GET=20URL=20fo?= =?UTF-8?q?r=20an=20S3=20object=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- infrastructure/s3/s3_client.py | 6 ++++++ tests/infrastructure/test_s3_client.py | 13 +++++++++++++ 2 files changed, 19 insertions(+) diff --git a/infrastructure/s3/s3_client.py b/infrastructure/s3/s3_client.py index a789fcc22..f07779351 100644 --- a/infrastructure/s3/s3_client.py +++ b/infrastructure/s3/s3_client.py @@ -20,3 +20,9 @@ class S3Client: def put_object(self, key: str, body: bytes) -> str: self._client.put_object(Bucket=self._bucket, Key=key, Body=body) return f"s3://{self._bucket}/{key}" + + def generate_presigned_url(self, key: str, expires_in: int) -> str: + """A time-limited URL that lets the holder GET this object without AWS + credentials (ADR-0060 — how a Download Package link is delivered). + ``expires_in`` is the validity window in seconds.""" + raise NotImplementedError diff --git a/tests/infrastructure/test_s3_client.py b/tests/infrastructure/test_s3_client.py index bdac6be1e..b867dd26c 100644 --- a/tests/infrastructure/test_s3_client.py +++ b/tests/infrastructure/test_s3_client.py @@ -34,3 +34,16 @@ def test_get_object_returns_bytes_written_by_put_object(s3_client: S3Client) -> def test_bucket_property_exposes_configured_bucket(s3_client: S3Client) -> None: # act / assert assert s3_client.bucket == BUCKET + + +def test_generate_presigned_url_signs_a_get_for_the_object(s3_client: S3Client) -> None: + # arrange + s3_client.put_object("bulk-downloads/pkg.zip", b"zip-bytes") + + # act + url = s3_client.generate_presigned_url("bulk-downloads/pkg.zip", expires_in=3600) + + # assert — a signed GET URL for this bucket + key + assert BUCKET in url + assert "bulk-downloads/pkg.zip" in url + assert "Signature" in url From da3e0cd4b4a37b1396e3ad179835876358fa4e18 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:18:54 +0000 Subject: [PATCH 20/55] =?UTF-8?q?Generate=20a=20presigned=20GET=20URL=20fo?= =?UTF-8?q?r=20an=20S3=20object=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- infrastructure/s3/s3_client.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/infrastructure/s3/s3_client.py b/infrastructure/s3/s3_client.py index f07779351..eb2a68f5c 100644 --- a/infrastructure/s3/s3_client.py +++ b/infrastructure/s3/s3_client.py @@ -25,4 +25,9 @@ class S3Client: """A time-limited URL that lets the holder GET this object without AWS credentials (ADR-0060 — how a Download Package link is delivered). ``expires_in`` is the validity window in seconds.""" - raise NotImplementedError + url: str = self._client.generate_presigned_url( + "get_object", + Params={"Bucket": self._bucket, "Key": key}, + ExpiresIn=expires_in, + ) + return url From 910f7cc06c0d5b23e14430e1656f608f14fe97dc Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:23:12 +0000 Subject: [PATCH 21/55] =?UTF-8?q?Fetch=20all=20uploaded=20files=20for=20a?= =?UTF-8?q?=20set=20of=20properties=20by=20landlord=5Fproperty=5Fid=20?= =?UTF-8?q?=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../uploaded_file_postgres_repository.py | 11 +++++- .../test_uploaded_file_postgres_repository.py | 35 +++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/repositories/uploaded_file/uploaded_file_postgres_repository.py b/repositories/uploaded_file/uploaded_file_postgres_repository.py index 9bf4a3d3c..107bc6592 100644 --- a/repositories/uploaded_file/uploaded_file_postgres_repository.py +++ b/repositories/uploaded_file/uploaded_file_postgres_repository.py @@ -1,6 +1,6 @@ from __future__ import annotations -from typing import Optional +from typing import Optional, Sequence from sqlalchemy import select from sqlmodel import Session, col @@ -24,5 +24,14 @@ class UploadedFilePostgresRepository: ) return self._session.execute(stmt).scalars().one_or_none() # pyright: ignore[reportDeprecated] + def by_landlord_property_ids( + self, landlord_property_ids: Sequence[str] + ) -> list[UploadedFile]: + """Every uploaded file held for the given properties, matched by + `landlord_property_id` (ADR-0060), across all Document Types. Returns + the raw rows — latest-per-Document-Type selection and null-type + skipping are the Download Package plan's job, not the query's.""" + raise NotImplementedError + def insert(self, uploaded_file: UploadedFile) -> None: self._session.add(uploaded_file) diff --git a/tests/repositories/uploaded_file/test_uploaded_file_postgres_repository.py b/tests/repositories/uploaded_file/test_uploaded_file_postgres_repository.py index 5300c0209..40fc1773f 100644 --- a/tests/repositories/uploaded_file/test_uploaded_file_postgres_repository.py +++ b/tests/repositories/uploaded_file/test_uploaded_file_postgres_repository.py @@ -78,3 +78,38 @@ def test_does_not_return_row_with_different_file_type(db_engine: Engine) -> None # Assert assert result is None + + +def _landlord_file( + landlord_property_id: str, + s3_file_key: str, + file_type: FileTypeEnum = FileTypeEnum.SITE_NOTE, +) -> UploadedFile: + return UploadedFile( + s3_file_bucket=_BUCKET, + s3_file_key=s3_file_key, + s3_upload_timestamp=datetime(2026, 1, 1, 12, 0, 0, tzinfo=timezone.utc), + landlord_property_id=landlord_property_id, + file_type=file_type.value, + ) + + +def test_by_landlord_property_ids_returns_only_requested_properties( + db_engine: Engine, +) -> None: + # Arrange — files for three properties; only two are requested. + with Session(db_engine) as session: + session.add(_landlord_file("LP1", "one.pdf")) + session.add(_landlord_file("LP2", "two.pdf")) + session.add(_landlord_file("LP3", "three.pdf")) + session.commit() + + # Act + with Session(db_engine) as session: + found = UploadedFilePostgresRepository(session).by_landlord_property_ids( + ["LP1", "LP2"] + ) + result = {(f.landlord_property_id, f.s3_file_key) for f in found} + + # Assert + assert result == {("LP1", "one.pdf"), ("LP2", "two.pdf")} From 135dd9abda6a978e504375bc863f410be0812b8e Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:23:47 +0000 Subject: [PATCH 22/55] =?UTF-8?q?Fetch=20all=20uploaded=20files=20for=20a?= =?UTF-8?q?=20set=20of=20properties=20by=20landlord=5Fproperty=5Fid=20?= =?UTF-8?q?=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../uploaded_file/uploaded_file_postgres_repository.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/repositories/uploaded_file/uploaded_file_postgres_repository.py b/repositories/uploaded_file/uploaded_file_postgres_repository.py index 107bc6592..82c711231 100644 --- a/repositories/uploaded_file/uploaded_file_postgres_repository.py +++ b/repositories/uploaded_file/uploaded_file_postgres_repository.py @@ -31,7 +31,10 @@ class UploadedFilePostgresRepository: `landlord_property_id` (ADR-0060), across all Document Types. Returns the raw rows — latest-per-Document-Type selection and null-type skipping are the Download Package plan's job, not the query's.""" - raise NotImplementedError + stmt = select(UploadedFile).where( + col(UploadedFile.landlord_property_id).in_(list(landlord_property_ids)) + ) + return list(self._session.execute(stmt).scalars().all()) # pyright: ignore[reportDeprecated] def insert(self, uploaded_file: UploadedFile) -> None: self._session.add(uploaded_file) From 7466b78e54ac79ec569f863d1911422f676f032d Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:25:56 +0000 Subject: [PATCH 23/55] =?UTF-8?q?Send=20a=20document-download=20email=20to?= =?UTF-8?q?=20the=20requesting=20user=20over=20SES=20SMTP=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- infrastructure/email/__init__.py | 0 infrastructure/email/ses_smtp_email_sender.py | 60 +++++++++++++++ repositories/email/__init__.py | 0 repositories/email/email_sender.py | 15 ++++ tests/infrastructure/email/__init__.py | 0 .../email/test_ses_smtp_email_sender.py | 77 +++++++++++++++++++ 6 files changed, 152 insertions(+) create mode 100644 infrastructure/email/__init__.py create mode 100644 infrastructure/email/ses_smtp_email_sender.py create mode 100644 repositories/email/__init__.py create mode 100644 repositories/email/email_sender.py create mode 100644 tests/infrastructure/email/__init__.py create mode 100644 tests/infrastructure/email/test_ses_smtp_email_sender.py diff --git a/infrastructure/email/__init__.py b/infrastructure/email/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/infrastructure/email/ses_smtp_email_sender.py b/infrastructure/email/ses_smtp_email_sender.py new file mode 100644 index 000000000..9318f100d --- /dev/null +++ b/infrastructure/email/ses_smtp_email_sender.py @@ -0,0 +1,60 @@ +"""SES-over-SMTP implementation of the outbound-email port (ADR-0059). + +Sends via `smtplib` against the SES SMTP endpoint using the credentials +provisioned in terraform (`modules/ses`, Secrets Manager +`${stage}/ses/smtp_credentials`). The SMTP transport is injected so the sender +is testable without a live server. +""" + +from __future__ import annotations + +import smtplib +from email.message import EmailMessage +from types import TracebackType +from typing import Any, Callable, Optional, Protocol, Type + + +class SmtpTransport(Protocol): + """The minimal SMTP surface the sender uses — satisfied structurally by + `smtplib.SMTP` and by test fakes.""" + + def __enter__(self) -> "SmtpTransport": ... + + def __exit__( + self, + exc_type: Optional[Type[BaseException]], + exc: Optional[BaseException], + tb: Optional[TracebackType], + ) -> Optional[bool]: ... + + def starttls(self) -> Any: ... + + def login(self, user: str, password: str) -> Any: ... + + def send_message(self, message: EmailMessage) -> Any: ... + + +def _default_smtp(host: str, port: int) -> SmtpTransport: + return smtplib.SMTP(host, port) + + +class SesSmtpEmailSender: + def __init__( + self, + *, + host: str, + port: int, + username: str, + password: str, + from_address: str, + smtp_factory: Callable[[str, int], SmtpTransport] = _default_smtp, + ) -> None: + self._host = host + self._port = port + self._username = username + self._password = password + self._from_address = from_address + self._smtp_factory = smtp_factory + + def send(self, *, to: str, subject: str, body: str) -> None: + raise NotImplementedError diff --git a/repositories/email/__init__.py b/repositories/email/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/repositories/email/email_sender.py b/repositories/email/email_sender.py new file mode 100644 index 000000000..2d6ac8725 --- /dev/null +++ b/repositories/email/email_sender.py @@ -0,0 +1,15 @@ +"""The outbound-email port (ADR-0059). + +A narrow interface the orchestrator depends on to notify a user, so the +transport (SES over SMTP today) stays swappable and testable behind it. +""" + +from __future__ import annotations + +from typing import Protocol + + +class EmailSender(Protocol): + def send(self, *, to: str, subject: str, body: str) -> None: + """Send a plain-text email to ``to``.""" + ... diff --git a/tests/infrastructure/email/__init__.py b/tests/infrastructure/email/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/infrastructure/email/test_ses_smtp_email_sender.py b/tests/infrastructure/email/test_ses_smtp_email_sender.py new file mode 100644 index 000000000..c5e5ca2f3 --- /dev/null +++ b/tests/infrastructure/email/test_ses_smtp_email_sender.py @@ -0,0 +1,77 @@ +"""The SES-over-SMTP email sender (ADR-0059) — delivers a plain-text message to +the requesting user via an injected SMTP transport.""" + +from __future__ import annotations + +from email.message import EmailMessage +from types import TracebackType +from typing import Any, Optional, Type + +from infrastructure.email.ses_smtp_email_sender import SesSmtpEmailSender + + +class _FakeSmtp: + """Records what the sender hands the transport, standing in for a live SES + SMTP connection.""" + + def __init__(self, host: str, port: int) -> None: + self.host = host + self.port = port + self.messages: list[EmailMessage] = [] + self.started_tls = False + self.login_args: Optional[tuple[str, str]] = None + + def __enter__(self) -> "_FakeSmtp": + return self + + def __exit__( + self, + exc_type: Optional[Type[BaseException]], + exc: Optional[BaseException], + tb: Optional[TracebackType], + ) -> Optional[bool]: + return None + + def starttls(self) -> Any: + self.started_tls = True + + def login(self, user: str, password: str) -> Any: + self.login_args = (user, password) + + def send_message(self, message: EmailMessage) -> Any: + self.messages.append(message) + + +def test_send_delivers_a_plaintext_message_with_the_right_envelope() -> None: + # Arrange + created: list[_FakeSmtp] = [] + + def factory(host: str, port: int) -> _FakeSmtp: + smtp = _FakeSmtp(host, port) + created.append(smtp) + return smtp + + sender = SesSmtpEmailSender( + host="email-smtp.eu-west-2.amazonaws.com", + port=587, + username="AKIA_SMTP_USER", + password="smtp-password", + from_address="noreply@domna.homes", + smtp_factory=factory, + ) + + # Act + sender.send( + to="user@example.com", + subject="Your document download is ready", + body="Download it here: https://signed-url", + ) + + # Assert — one message, correctly addressed, body carried through. + assert len(created) == 1 + assert len(created[0].messages) == 1 + message = created[0].messages[0] + assert message["To"] == "user@example.com" + assert message["From"] == "noreply@domna.homes" + assert message["Subject"] == "Your document download is ready" + assert "https://signed-url" in message.get_content() From 74732c6e5a47c3e0a13aa63ad590196706fe3e04 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:27:09 +0000 Subject: [PATCH 24/55] =?UTF-8?q?Send=20a=20document-download=20email=20to?= =?UTF-8?q?=20the=20requesting=20user=20over=20SES=20SMTP=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- infrastructure/email/ses_smtp_email_sender.py | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/infrastructure/email/ses_smtp_email_sender.py b/infrastructure/email/ses_smtp_email_sender.py index 9318f100d..ba53d23bc 100644 --- a/infrastructure/email/ses_smtp_email_sender.py +++ b/infrastructure/email/ses_smtp_email_sender.py @@ -11,7 +11,7 @@ from __future__ import annotations import smtplib from email.message import EmailMessage from types import TracebackType -from typing import Any, Callable, Optional, Protocol, Type +from typing import Any, Callable, Optional, Protocol, Type, cast class SmtpTransport(Protocol): @@ -35,7 +35,10 @@ class SmtpTransport(Protocol): def _default_smtp(host: str, port: int) -> SmtpTransport: - return smtplib.SMTP(host, port) + # `smtplib.SMTP` satisfies SmtpTransport structurally; its stdlib stubs + # declare wider signatures (extra optional args, an SMTP-typed context + # manager), so the cast bridges the real transport to the port's surface. + return cast(SmtpTransport, smtplib.SMTP(host, port)) class SesSmtpEmailSender: @@ -57,4 +60,12 @@ class SesSmtpEmailSender: self._smtp_factory = smtp_factory def send(self, *, to: str, subject: str, body: str) -> None: - raise NotImplementedError + message = EmailMessage() + message["From"] = self._from_address + message["To"] = to + message["Subject"] = subject + message.set_content(body) + with self._smtp_factory(self._host, self._port) as smtp: + smtp.starttls() + smtp.login(self._username, self._password) + smtp.send_message(message) From 5151fea9b26d4ff9279f04898850fe88f0db8489 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:27:31 +0000 Subject: [PATCH 25/55] =?UTF-8?q?STARTTLS=20and=20authenticate=20before=20?= =?UTF-8?q?sending=20the=20email=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pins the auth handshake delivered with the tracer green (no red phase); discriminating: an unauthenticated/cleartext send would be rejected by SES. Co-Authored-By: Claude Fable 5 --- .../email/test_ses_smtp_email_sender.py | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/tests/infrastructure/email/test_ses_smtp_email_sender.py b/tests/infrastructure/email/test_ses_smtp_email_sender.py index c5e5ca2f3..036ac239f 100644 --- a/tests/infrastructure/email/test_ses_smtp_email_sender.py +++ b/tests/infrastructure/email/test_ses_smtp_email_sender.py @@ -75,3 +75,30 @@ def test_send_delivers_a_plaintext_message_with_the_right_envelope() -> None: assert message["From"] == "noreply@domna.homes" assert message["Subject"] == "Your document download is ready" assert "https://signed-url" in message.get_content() + + +def test_send_starts_tls_and_authenticates_before_sending() -> None: + # Arrange — SES SMTP rejects unauthenticated, cleartext senders, so the + # transport must STARTTLS and log in with the SMTP credentials. + created: list[_FakeSmtp] = [] + + def factory(host: str, port: int) -> _FakeSmtp: + smtp = _FakeSmtp(host, port) + created.append(smtp) + return smtp + + sender = SesSmtpEmailSender( + host="email-smtp.eu-west-2.amazonaws.com", + port=587, + username="AKIA_SMTP_USER", + password="smtp-password", + from_address="noreply@domna.homes", + smtp_factory=factory, + ) + + # Act + sender.send(to="user@example.com", subject="s", body="b") + + # Assert + assert created[0].started_tls is True + assert created[0].login_args == ("AKIA_SMTP_USER", "smtp-password") From 115f506353bf58d7899f6cd8454f231d1e23431d Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:30:38 +0000 Subject: [PATCH 26/55] =?UTF-8?q?Build,=20upload=20and=20email=20a=20Downl?= =?UTF-8?q?oad=20Package=20for=20a=20property=20set=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../bulk_document_download_orchestrator.py | 81 +++++++++++++ ...est_bulk_document_download_orchestrator.py | 113 ++++++++++++++++++ 2 files changed, 194 insertions(+) create mode 100644 orchestration/bulk_document_download_orchestrator.py create mode 100644 tests/orchestration/test_bulk_document_download_orchestrator.py diff --git a/orchestration/bulk_document_download_orchestrator.py b/orchestration/bulk_document_download_orchestrator.py new file mode 100644 index 000000000..01d65a44a --- /dev/null +++ b/orchestration/bulk_document_download_orchestrator.py @@ -0,0 +1,81 @@ +"""The Bulk Document Download orchestrator (ADR-0060). + +Ties the pieces together for one run: gather a property set's uploaded files +(by `landlord_property_id`), enrich each with its display address, lay them out +with the Download Package plan, stream the chosen files into a single ZIP, +upload it, mint a presigned URL, and email the requester. Best-effort — skipped +rows are reported, not fatal; a wholly-empty selection is a recorded failure +(ADR-0060). +""" + +from __future__ import annotations + +import io +import zipfile +from dataclasses import dataclass +from datetime import datetime +from typing import Optional, Protocol, Sequence, cast + +from domain.bulk_document_download.package_plan import ( + ResolvedDocument, + SkippedDocument, + plan_download_package, +) +from domain.tasks.subtasks import SubTaskFailure +from infrastructure.postgres.uploaded_file_table import UploadedFile +from infrastructure.s3.s3_client import S3Client +from repositories.email.email_sender import EmailSender + + +class UploadedFileReader(Protocol): + def by_landlord_property_ids( + self, landlord_property_ids: Sequence[str] + ) -> list[UploadedFile]: ... + + +class AddressResolver(Protocol): + def address_for(self, landlord_property_id: str) -> str: ... + + +class DocumentBytesReader(Protocol): + def read(self, bucket: str, key: str) -> bytes: ... + + +@dataclass(frozen=True) +class DownloadPackageResult: + """What a completed run reports (lands in `sub_task.outputs`, ADR-0060).""" + + presigned_url: str + package_s3_key: str + included: int + skipped: tuple[SkippedDocument, ...] + + +class BulkDocumentDownloadOrchestrator: + def __init__( + self, + *, + uploaded_files: UploadedFileReader, + addresses: AddressResolver, + documents: DocumentBytesReader, + packages: S3Client, + email: EmailSender, + url_ttl_seconds: int, + package_key_prefix: str = "bulk-downloads", + ) -> None: + self._uploaded_files = uploaded_files + self._addresses = addresses + self._documents = documents + self._packages = packages + self._email = email + self._url_ttl_seconds = url_ttl_seconds + self._package_key_prefix = package_key_prefix + + def run( + self, + *, + landlord_property_ids: Sequence[str], + recipient_email: str, + package_name: str, + ) -> DownloadPackageResult: + raise NotImplementedError diff --git a/tests/orchestration/test_bulk_document_download_orchestrator.py b/tests/orchestration/test_bulk_document_download_orchestrator.py new file mode 100644 index 000000000..b1b48830a --- /dev/null +++ b/tests/orchestration/test_bulk_document_download_orchestrator.py @@ -0,0 +1,113 @@ +"""The Bulk Document Download orchestrator (ADR-0060): gather a property set's +files, lay them out, zip + upload, presign, and email the requester.""" + +from __future__ import annotations + +import io +import zipfile +from collections.abc import Iterator +from datetime import datetime, timezone + +import pytest +from moto import mock_aws + +from infrastructure.postgres.uploaded_file_table import FileTypeEnum, UploadedFile +from infrastructure.s3.s3_client import S3Client +from orchestration.bulk_document_download_orchestrator import ( + BulkDocumentDownloadOrchestrator, +) +from tests.infrastructure import make_boto_client + +DATA_BUCKET = "data-bucket" + + +class _FakeUploadedFiles: + def __init__(self, files: list[UploadedFile]) -> None: + self._files = files + + def by_landlord_property_ids( + self, landlord_property_ids: object + ) -> list[UploadedFile]: + wanted = set(landlord_property_ids) # type: ignore[call-overload] + return [f for f in self._files if f.landlord_property_id in wanted] + + +class _FakeAddresses: + def __init__(self, mapping: dict[str, str]) -> None: + self._mapping = mapping + + def address_for(self, landlord_property_id: str) -> str: + return self._mapping[landlord_property_id] + + +class _FakeDocuments: + def __init__(self, blobs: dict[tuple[str, str], bytes]) -> None: + self._blobs = blobs + + def read(self, bucket: str, key: str) -> bytes: + return self._blobs[(bucket, key)] + + +class _RecordingEmail: + def __init__(self) -> None: + self.sent: list[tuple[str, str, str]] = [] + + def send(self, *, to: str, subject: str, body: str) -> None: + self.sent.append((to, subject, body)) + + +def _uploaded_file( + landlord_property_id: str, + s3_file_bucket: str, + s3_file_key: str, + file_type: FileTypeEnum = FileTypeEnum.SITE_NOTE, +) -> UploadedFile: + return UploadedFile( + s3_file_bucket=s3_file_bucket, + s3_file_key=s3_file_key, + s3_upload_timestamp=datetime(2026, 1, 1, tzinfo=timezone.utc), + landlord_property_id=landlord_property_id, + file_type=file_type.value, + ) + + +@pytest.fixture +def packages() -> Iterator[S3Client]: + with mock_aws(): + boto_client = make_boto_client("s3") + boto_client.create_bucket(Bucket=DATA_BUCKET) + yield S3Client(boto_client, DATA_BUCKET) + + +def test_builds_uploads_and_emails_a_download_package(packages: S3Client) -> None: + # Arrange — one property with one site note. + files = [_uploaded_file("LP1", "src-bucket", "surveys/a.pdf")] + email = _RecordingEmail() + orchestrator = BulkDocumentDownloadOrchestrator( + uploaded_files=_FakeUploadedFiles(files), + addresses=_FakeAddresses({"LP1": "12 Oak Street"}), + documents=_FakeDocuments({("src-bucket", "surveys/a.pdf"): b"PDF-BYTES"}), + packages=packages, + email=email, + url_ttl_seconds=3600, + ) + + # Act + result = orchestrator.run( + landlord_property_ids=["LP1"], + recipient_email="user@example.com", + package_name="portfolio-42", + ) + + # Assert — one file packaged, at the planned path with the source bytes. + assert result.included == 1 + archive = packages.get_object(result.package_s3_key) + with zipfile.ZipFile(io.BytesIO(archive)) as zf: + assert zf.read("12 Oak Street (LP1)/site_note.pdf") == b"PDF-BYTES" + + # ...the presigned URL points at that object, and the requester is emailed it. + assert result.package_s3_key in result.presigned_url + assert len(email.sent) == 1 + to, _subject, body = email.sent[0] + assert to == "user@example.com" + assert result.presigned_url in body From e7181b7d1fb84e674b1dfd458d8a32776cba2ab8 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:31:49 +0000 Subject: [PATCH 27/55] =?UTF-8?q?Build,=20upload=20and=20email=20a=20Downl?= =?UTF-8?q?oad=20Package=20for=20a=20property=20set=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../bulk_document_download_orchestrator.py | 45 ++++++++++++++++++- 1 file changed, 43 insertions(+), 2 deletions(-) diff --git a/orchestration/bulk_document_download_orchestrator.py b/orchestration/bulk_document_download_orchestrator.py index 01d65a44a..54de823bb 100644 --- a/orchestration/bulk_document_download_orchestrator.py +++ b/orchestration/bulk_document_download_orchestrator.py @@ -21,7 +21,6 @@ from domain.bulk_document_download.package_plan import ( SkippedDocument, plan_download_package, ) -from domain.tasks.subtasks import SubTaskFailure from infrastructure.postgres.uploaded_file_table import UploadedFile from infrastructure.s3.s3_client import S3Client from repositories.email.email_sender import EmailSender @@ -78,4 +77,46 @@ class BulkDocumentDownloadOrchestrator: recipient_email: str, package_name: str, ) -> DownloadPackageResult: - raise NotImplementedError + files = self._uploaded_files.by_landlord_property_ids(landlord_property_ids) + resolved = [self._resolve(file) for file in files] + plan = plan_download_package([r for r in resolved if r is not None]) + + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, "w", zipfile.ZIP_DEFLATED) as archive: + for entry in plan.entries: + archive.writestr( + entry.zip_path, self._documents.read(entry.s3_bucket, entry.s3_key) + ) + + key = f"{self._package_key_prefix}/{package_name}.zip" + self._packages.put_object(key, buffer.getvalue()) + url = self._packages.generate_presigned_url(key, self._url_ttl_seconds) + + self._email.send( + to=recipient_email, + subject="Your document download is ready", + body=( + f"Your document download is ready. It contains " + f"{len(plan.entries)} document(s).\n\n" + f"Download it here (link valid for " + f"{self._url_ttl_seconds // 60} minutes):\n{url}" + ), + ) + return DownloadPackageResult( + presigned_url=url, + package_s3_key=key, + included=len(plan.entries), + skipped=plan.skipped, + ) + + def _resolve(self, file: UploadedFile) -> Optional[ResolvedDocument]: + if file.landlord_property_id is None: + return None + return ResolvedDocument( + landlord_property_id=file.landlord_property_id, + address=self._addresses.address_for(file.landlord_property_id), + document_type=file.file_type, + s3_bucket=file.s3_file_bucket, + s3_key=file.s3_file_key, + uploaded_at=cast(datetime, file.s3_upload_timestamp), + ) From 73b21e8d6456dccf46070641182a5b97ef27ca3f Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:32:37 +0000 Subject: [PATCH 28/55] =?UTF-8?q?Treat=20a=20selection=20with=20no=20docum?= =?UTF-8?q?ents=20as=20a=20recorded=20failure=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- ...est_bulk_document_download_orchestrator.py | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/tests/orchestration/test_bulk_document_download_orchestrator.py b/tests/orchestration/test_bulk_document_download_orchestrator.py index b1b48830a..02400e16f 100644 --- a/tests/orchestration/test_bulk_document_download_orchestrator.py +++ b/tests/orchestration/test_bulk_document_download_orchestrator.py @@ -11,6 +11,7 @@ from datetime import datetime, timezone import pytest from moto import mock_aws +from domain.tasks.subtasks import SubTaskFailure from infrastructure.postgres.uploaded_file_table import FileTypeEnum, UploadedFile from infrastructure.s3.s3_client import S3Client from orchestration.bulk_document_download_orchestrator import ( @@ -111,3 +112,28 @@ def test_builds_uploads_and_emails_a_download_package(packages: S3Client) -> Non to, _subject, body = email.sent[0] assert to == "user@example.com" assert result.presigned_url in body + + +def test_a_selection_with_no_documents_is_a_recorded_failure( + packages: S3Client, +) -> None: + # Arrange — the selection resolves to no packageable documents. + email = _RecordingEmail() + orchestrator = BulkDocumentDownloadOrchestrator( + uploaded_files=_FakeUploadedFiles([]), + addresses=_FakeAddresses({}), + documents=_FakeDocuments({}), + packages=packages, + email=email, + url_ttl_seconds=3600, + ) + + # Act / Assert — a recorded SubTask failure, and nobody is emailed an empty + # package. + with pytest.raises(SubTaskFailure): + orchestrator.run( + landlord_property_ids=["LP1"], + recipient_email="user@example.com", + package_name="empty", + ) + assert email.sent == [] From 59ef8a12aba7ac7f810a61ffca8dd327df29ffc2 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:33:25 +0000 Subject: [PATCH 29/55] =?UTF-8?q?Treat=20a=20selection=20with=20no=20docum?= =?UTF-8?q?ents=20as=20a=20recorded=20failure=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../bulk_document_download_orchestrator.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/orchestration/bulk_document_download_orchestrator.py b/orchestration/bulk_document_download_orchestrator.py index 54de823bb..39ee7271e 100644 --- a/orchestration/bulk_document_download_orchestrator.py +++ b/orchestration/bulk_document_download_orchestrator.py @@ -21,6 +21,7 @@ from domain.bulk_document_download.package_plan import ( SkippedDocument, plan_download_package, ) +from domain.tasks.subtasks import SubTaskFailure from infrastructure.postgres.uploaded_file_table import UploadedFile from infrastructure.s3.s3_client import S3Client from repositories.email.email_sender import EmailSender @@ -81,6 +82,23 @@ class BulkDocumentDownloadOrchestrator: resolved = [self._resolve(file) for file in files] plan = plan_download_package([r for r in resolved if r is not None]) + if not plan.entries: + # Nothing to package (ADR-0060): a recorded failure, not an empty + # ZIP emailed to the user. The skip detail rides on the failure. + raise SubTaskFailure( + "no documents resolved for the selection", + details={ + "skipped": [ + { + "landlord_property_id": s.landlord_property_id, + "s3_key": s.s3_key, + "reason": s.reason, + } + for s in plan.skipped + ] + }, + ) + buffer = io.BytesIO() with zipfile.ZipFile(buffer, "w", zipfile.ZIP_DEFLATED) as archive: for entry in plan.entries: From 77e7b0fda96040adf01f9132b41d7bc0754eace5 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:33:58 +0000 Subject: [PATCH 30/55] =?UTF-8?q?Package=20the=20good=20documents=20and=20?= =?UTF-8?q?report=20the=20skipped=20ones=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pins the best-effort contract (ADR-0060) delivered with the tracer — no red phase; discriminating: a strict impl would fail the run or omit the report. Co-Authored-By: Claude Fable 5 --- ...est_bulk_document_download_orchestrator.py | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/tests/orchestration/test_bulk_document_download_orchestrator.py b/tests/orchestration/test_bulk_document_download_orchestrator.py index 02400e16f..40c483d80 100644 --- a/tests/orchestration/test_bulk_document_download_orchestrator.py +++ b/tests/orchestration/test_bulk_document_download_orchestrator.py @@ -137,3 +137,42 @@ def test_a_selection_with_no_documents_is_a_recorded_failure( package_name="empty", ) assert email.sent == [] + + +def test_packages_the_good_documents_and_reports_the_skipped_ones( + packages: S3Client, +) -> None: + # Arrange — one packable site note and one row with no Document Type. + good = _uploaded_file("LP1", "src-bucket", "surveys/a.pdf") + untyped = UploadedFile( + s3_file_bucket="src-bucket", + s3_file_key="mystery.bin", + s3_upload_timestamp=datetime(2026, 1, 1, tzinfo=timezone.utc), + landlord_property_id="LP1", + file_type=None, + ) + email = _RecordingEmail() + orchestrator = BulkDocumentDownloadOrchestrator( + uploaded_files=_FakeUploadedFiles([good, untyped]), + addresses=_FakeAddresses({"LP1": "12 Oak Street"}), + documents=_FakeDocuments({("src-bucket", "surveys/a.pdf"): b"PDF-BYTES"}), + packages=packages, + email=email, + url_ttl_seconds=3600, + ) + + # Act + result = orchestrator.run( + landlord_property_ids=["LP1"], + recipient_email="user@example.com", + package_name="mixed", + ) + + # Assert — the good file is packaged and delivered; the untyped one is + # reported, not fatal. + assert result.included == 1 + assert [s.s3_key for s in result.skipped] == ["mystery.bin"] + assert len(email.sent) == 1 + archive = packages.get_object(result.package_s3_key) + with zipfile.ZipFile(io.BytesIO(archive)) as zf: + assert zf.namelist() == ["12 Oak Street (LP1)/site_note.pdf"] From 0a09b564251e81815b135ebabb216a0dfb3e5d60 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:42:05 +0000 Subject: [PATCH 31/55] =?UTF-8?q?Read=20document=20bytes=20from=20an=20arb?= =?UTF-8?q?itrary=20source=20bucket=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- infrastructure/s3/s3_document_bytes_reader.py | 19 ++++++++++++ .../test_s3_document_bytes_reader.py | 29 +++++++++++++++++++ 2 files changed, 48 insertions(+) create mode 100644 infrastructure/s3/s3_document_bytes_reader.py create mode 100644 tests/infrastructure/test_s3_document_bytes_reader.py diff --git a/infrastructure/s3/s3_document_bytes_reader.py b/infrastructure/s3/s3_document_bytes_reader.py new file mode 100644 index 000000000..930377740 --- /dev/null +++ b/infrastructure/s3/s3_document_bytes_reader.py @@ -0,0 +1,19 @@ +"""Reads document bytes from arbitrary S3 buckets (ADR-0060). + +Uploaded files live across many source buckets (each `uploaded_files` row +carries its own `s3_file_bucket`), so — unlike the bucket-bound `S3Client` — +this reader takes the bucket per call. Used by the Bulk Document Download +orchestrator to stream each chosen file into the ZIP. +""" + +from __future__ import annotations + +from typing import Any + + +class S3DocumentBytesReader: + def __init__(self, boto_s3_client: Any) -> None: + self._client = boto_s3_client + + def read(self, bucket: str, key: str) -> bytes: + raise NotImplementedError diff --git a/tests/infrastructure/test_s3_document_bytes_reader.py b/tests/infrastructure/test_s3_document_bytes_reader.py new file mode 100644 index 000000000..8bbcea895 --- /dev/null +++ b/tests/infrastructure/test_s3_document_bytes_reader.py @@ -0,0 +1,29 @@ +"""The multi-bucket document reader (ADR-0060) — reads object bytes from +whichever source bucket an uploaded file lives in.""" + +from collections.abc import Iterator + +import pytest +from moto import mock_aws + +from infrastructure.s3.s3_document_bytes_reader import S3DocumentBytesReader +from tests.infrastructure import make_boto_client + + +@pytest.fixture +def reader() -> Iterator[S3DocumentBytesReader]: + with mock_aws(): + boto_client = make_boto_client("s3") + boto_client.create_bucket(Bucket="src-a") + boto_client.create_bucket(Bucket="src-b") + boto_client.put_object(Bucket="src-a", Key="surveys/one.pdf", Body=b"AAA") + boto_client.put_object(Bucket="src-b", Key="photos/two.zip", Body=b"BBB") + yield S3DocumentBytesReader(boto_client) + + +def test_reads_bytes_from_the_named_source_bucket( + reader: S3DocumentBytesReader, +) -> None: + # act / assert — each object is read from its own bucket. + assert reader.read("src-a", "surveys/one.pdf") == b"AAA" + assert reader.read("src-b", "photos/two.zip") == b"BBB" From 79fa46d97c5cde8540fe6b8c4358e880b4f782af Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:42:37 +0000 Subject: [PATCH 32/55] =?UTF-8?q?Read=20document=20bytes=20from=20an=20arb?= =?UTF-8?q?itrary=20source=20bucket=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- infrastructure/s3/s3_document_bytes_reader.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/infrastructure/s3/s3_document_bytes_reader.py b/infrastructure/s3/s3_document_bytes_reader.py index 930377740..2e0739249 100644 --- a/infrastructure/s3/s3_document_bytes_reader.py +++ b/infrastructure/s3/s3_document_bytes_reader.py @@ -16,4 +16,6 @@ class S3DocumentBytesReader: self._client = boto_s3_client def read(self, bucket: str, key: str) -> bytes: - raise NotImplementedError + response: dict[str, Any] = self._client.get_object(Bucket=bucket, Key=key) + body: bytes = response["Body"].read() + return body From 456d4dffe7b1d9ddfd6b7530bd67838d15c7755d Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:44:36 +0000 Subject: [PATCH 33/55] =?UTF-8?q?Resolve=20a=20property's=20display=20addr?= =?UTF-8?q?ess=20from=20its=20landlord=5Fproperty=5Fid=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../property/landlord_address_resolver.py | 30 ++++++++++++++++ .../test_landlord_address_resolver.py | 35 +++++++++++++++++++ 2 files changed, 65 insertions(+) create mode 100644 repositories/property/landlord_address_resolver.py create mode 100644 tests/repositories/property/test_landlord_address_resolver.py diff --git a/repositories/property/landlord_address_resolver.py b/repositories/property/landlord_address_resolver.py new file mode 100644 index 000000000..332f6b149 --- /dev/null +++ b/repositories/property/landlord_address_resolver.py @@ -0,0 +1,30 @@ +"""Resolves a property's display address from its `landlord_property_id` +(ADR-0060), for naming the folders in a Download Package. + +The address lives on the FE-owned `property` table (`uploaded_files` carries no +`property_id` — a known future gap). Addresses for the whole requested set are +loaded once at construction; `address_for` is then a lookup with a safe +fallback for a `landlord_property_id` that has no property row or address. +""" + +from __future__ import annotations + +from typing import Sequence + +from sqlalchemy import select +from sqlmodel import Session, col + +from infrastructure.postgres.property_table import PropertyRow + + +class LandlordAddressResolver: + _FALLBACK = "address unavailable" + + def __init__( + self, session: Session, landlord_property_ids: Sequence[str] + ) -> None: + self._session = session + self._ids = list(landlord_property_ids) + + def address_for(self, landlord_property_id: str) -> str: + raise NotImplementedError diff --git a/tests/repositories/property/test_landlord_address_resolver.py b/tests/repositories/property/test_landlord_address_resolver.py new file mode 100644 index 000000000..7dcbb98da --- /dev/null +++ b/tests/repositories/property/test_landlord_address_resolver.py @@ -0,0 +1,35 @@ +from collections.abc import Iterator + +import pytest +from sqlalchemy import Engine +from sqlmodel import Session + +from infrastructure.postgres.property_table import PropertyRow +from repositories.property.landlord_address_resolver import LandlordAddressResolver + + +@pytest.fixture +def session(db_engine: Engine) -> Iterator[Session]: + with Session(db_engine) as s: + yield s + + +def test_resolves_known_addresses_and_falls_back_for_unknowns( + session: Session, +) -> None: + # arrange — two properties with addresses; a third id has no property row. + session.add( + PropertyRow(portfolio_id=1, landlord_property_id="LP1", address="12 Oak Street") + ) + session.add( + PropertyRow(portfolio_id=1, landlord_property_id="LP2", address="9 Elm Road") + ) + session.flush() + + # act + resolver = LandlordAddressResolver(session, ["LP1", "LP2", "LP3"]) + + # assert + assert resolver.address_for("LP1") == "12 Oak Street" + assert resolver.address_for("LP2") == "9 Elm Road" + assert resolver.address_for("LP3") == "address unavailable" From 0387f4a8977c9589db0a3c35df180c4189fce0e4 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:45:13 +0000 Subject: [PATCH 34/55] =?UTF-8?q?Resolve=20a=20property's=20display=20addr?= =?UTF-8?q?ess=20from=20its=20landlord=5Fproperty=5Fid=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- repositories/property/landlord_address_resolver.py | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/repositories/property/landlord_address_resolver.py b/repositories/property/landlord_address_resolver.py index 332f6b149..cf051c0bd 100644 --- a/repositories/property/landlord_address_resolver.py +++ b/repositories/property/landlord_address_resolver.py @@ -23,8 +23,15 @@ class LandlordAddressResolver: def __init__( self, session: Session, landlord_property_ids: Sequence[str] ) -> None: - self._session = session - self._ids = list(landlord_property_ids) + stmt = select(PropertyRow).where( + col(PropertyRow.landlord_property_id).in_(list(landlord_property_ids)) + ) + rows = session.execute(stmt).scalars().all() # pyright: ignore[reportDeprecated] + self._by_id: dict[str, str] = { + row.landlord_property_id: row.address + for row in rows + if row.landlord_property_id is not None and row.address is not None + } def address_for(self, landlord_property_id: str) -> str: - raise NotImplementedError + return self._by_id.get(landlord_property_id, self._FALLBACK) From 79c8890d0735a7b22e1f8c95f83a6029d6a0440e Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:50:10 +0000 Subject: [PATCH 35/55] =?UTF-8?q?Stream=20a=20local=20file=20to=20S3=20wit?= =?UTF-8?q?h=20managed=20multipart=20upload=20=F0=9F=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- infrastructure/s3/s3_client.py | 6 ++++++ tests/infrastructure/test_s3_client.py | 17 +++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/infrastructure/s3/s3_client.py b/infrastructure/s3/s3_client.py index eb2a68f5c..7e26c790b 100644 --- a/infrastructure/s3/s3_client.py +++ b/infrastructure/s3/s3_client.py @@ -21,6 +21,12 @@ class S3Client: self._client.put_object(Bucket=self._bucket, Key=key, Body=body) return f"s3://{self._bucket}/{key}" + def upload_file(self, local_path: str, key: str) -> str: + """Upload a file from local disk, using boto's managed transfer so a + large object (a multi-GB Download Package ZIP, ADR-0060) is streamed in + multipart from ``/tmp`` rather than held in memory.""" + raise NotImplementedError + def generate_presigned_url(self, key: str, expires_in: int) -> str: """A time-limited URL that lets the holder GET this object without AWS credentials (ADR-0060 — how a Download Package link is delivered). diff --git a/tests/infrastructure/test_s3_client.py b/tests/infrastructure/test_s3_client.py index b867dd26c..13e6c4714 100644 --- a/tests/infrastructure/test_s3_client.py +++ b/tests/infrastructure/test_s3_client.py @@ -47,3 +47,20 @@ def test_generate_presigned_url_signs_a_get_for_the_object(s3_client: S3Client) assert BUCKET in url assert "bulk-downloads/pkg.zip" in url assert "Signature" in url + + +def test_upload_file_streams_a_local_file_to_s3( + s3_client: S3Client, tmp_path: object +) -> None: + # arrange — a file on local disk (stands in for the /tmp ZIP) + from pathlib import Path + + local = Path(str(tmp_path)) / "package.zip" + local.write_bytes(b"ZIP-CONTENT") + + # act + uri = s3_client.upload_file(str(local), "bulk-downloads/package.zip") + + # assert — the object landed with the file's bytes + assert uri == f"s3://{BUCKET}/bulk-downloads/package.zip" + assert s3_client.get_object("bulk-downloads/package.zip") == b"ZIP-CONTENT" From 879581bb32fda8362f0bee6223ca35bf8a4738ed Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:50:34 +0000 Subject: [PATCH 36/55] =?UTF-8?q?Stream=20a=20local=20file=20to=20S3=20wit?= =?UTF-8?q?h=20managed=20multipart=20upload=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- infrastructure/s3/s3_client.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/infrastructure/s3/s3_client.py b/infrastructure/s3/s3_client.py index 7e26c790b..5d5dc55ae 100644 --- a/infrastructure/s3/s3_client.py +++ b/infrastructure/s3/s3_client.py @@ -25,7 +25,8 @@ class S3Client: """Upload a file from local disk, using boto's managed transfer so a large object (a multi-GB Download Package ZIP, ADR-0060) is streamed in multipart from ``/tmp`` rather than held in memory.""" - raise NotImplementedError + self._client.upload_file(Filename=local_path, Bucket=self._bucket, Key=key) + return f"s3://{self._bucket}/{key}" def generate_presigned_url(self, key: str, expires_in: int) -> str: """A time-limited URL that lets the holder GET this object without AWS From a36aa49c2b23fe7a3110c100971c90d9fe1653ae Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:51:26 +0000 Subject: [PATCH 37/55] =?UTF-8?q?Stream=20the=20Download=20Package=20ZIP?= =?UTF-8?q?=20via=20/tmp=20and=20multipart=20upload=20=F0=9F=9F=AA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Multi-GB packages are no longer held in memory: each source file is read, written into the on-disk ZIP, and released; the archive is streamed up from disk. Behaviour unchanged (moto tests green). Co-Authored-By: Claude Fable 5 --- .../bulk_document_download_orchestrator.py | 24 ++++++++++++------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/orchestration/bulk_document_download_orchestrator.py b/orchestration/bulk_document_download_orchestrator.py index 39ee7271e..ba979a1c6 100644 --- a/orchestration/bulk_document_download_orchestrator.py +++ b/orchestration/bulk_document_download_orchestrator.py @@ -10,7 +10,8 @@ rows are reported, not fatal; a wholly-empty selection is a recorded failure from __future__ import annotations -import io +import os +import tempfile import zipfile from dataclasses import dataclass from datetime import datetime @@ -99,15 +100,20 @@ class BulkDocumentDownloadOrchestrator: }, ) - buffer = io.BytesIO() - with zipfile.ZipFile(buffer, "w", zipfile.ZIP_DEFLATED) as archive: - for entry in plan.entries: - archive.writestr( - entry.zip_path, self._documents.read(entry.s3_bucket, entry.s3_key) - ) - + # Stream to a /tmp file and multipart-upload it (ADR-0060): a Download + # Package can be several GB, so it is never held whole in memory — each + # source file is read, written into the on-disk ZIP, and released; the + # finished archive is streamed up from disk. key = f"{self._package_key_prefix}/{package_name}.zip" - self._packages.put_object(key, buffer.getvalue()) + with tempfile.TemporaryDirectory() as tmp_dir: + zip_path = os.path.join(tmp_dir, "package.zip") + with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as archive: + for entry in plan.entries: + archive.writestr( + entry.zip_path, + self._documents.read(entry.s3_bucket, entry.s3_key), + ) + self._packages.upload_file(zip_path, key) url = self._packages.generate_presigned_url(key, self._url_ttl_seconds) self._email.send( From 8c0b0159238da83a071f5cb67b955c5cd57c7a15 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:51:51 +0000 Subject: [PATCH 38/55] =?UTF-8?q?bulk=5Fdocument=5Fdownload=20Lambda:=20at?= =?UTF-8?q?tach-mode=20handler,=20trigger=20body,=20packaging=20?= =?UTF-8?q?=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reads the recipe from sub_task.inputs, assembles the Download Package via the orchestrator (source docs from each row's bucket -> ZIP -> the dedicated DOCUMENT_EXPORTS_BUCKET), returns the presigned URL + skip summary for sub_task.outputs. Dockerfile/requirements mirror bulk_upload_finaliser. Co-Authored-By: Claude Fable 5 --- .../bulk_document_download/Dockerfile | 35 ++++++ .../bulk_document_download_trigger_body.py | 19 ++++ .../bulk_document_download/handler.py | 100 ++++++++++++++++++ .../bulk_document_download/requirements.txt | 4 + 4 files changed, 158 insertions(+) create mode 100644 applications/bulk_document_download/Dockerfile create mode 100644 applications/bulk_document_download/bulk_document_download_trigger_body.py create mode 100644 applications/bulk_document_download/handler.py create mode 100644 applications/bulk_document_download/requirements.txt diff --git a/applications/bulk_document_download/Dockerfile b/applications/bulk_document_download/Dockerfile new file mode 100644 index 000000000..e65d1bd9a --- /dev/null +++ b/applications/bulk_document_download/Dockerfile @@ -0,0 +1,35 @@ +FROM public.ecr.aws/lambda/python:3.11 + +# Postgres host/port/database are baked into the image at build time from the +# deploy workflow's --build-arg values (GitHub Actions DEV_DB_* secrets), +# mirroring the bulk_upload_finaliser Dockerfile. They map onto the POSTGRES_* +# names PostgresConfig.from_env reads. Username/password are NOT baked in -- +# Terraform injects those (and the SES SMTP credentials) as Lambda env vars from +# Secrets Manager. +ARG DEV_DB_HOST +ARG DEV_DB_PORT +ARG DEV_DB_NAME + +ENV POSTGRES_HOST=${DEV_DB_HOST} +ENV POSTGRES_PORT=${DEV_DB_PORT} +ENV POSTGRES_DATABASE=${DEV_DB_NAME} + +WORKDIR /var/task + +COPY applications/bulk_document_download/requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +# DDD-shaped packages only -- no pandas, no legacy backend/. +COPY datatypes/ datatypes/ +COPY domain/ domain/ +COPY infrastructure/ infrastructure/ +COPY orchestration/ orchestration/ +COPY repositories/ repositories/ +COPY utilities/ utilities/ +COPY applications/ applications/ + +# Place the handler at the Lambda task root so the runtime resolves +# ``main.handler`` without an extra package prefix. +COPY applications/bulk_document_download/handler.py /var/task/main.py + +CMD ["main.handler"] diff --git a/applications/bulk_document_download/bulk_document_download_trigger_body.py b/applications/bulk_document_download/bulk_document_download_trigger_body.py new file mode 100644 index 000000000..95da349b5 --- /dev/null +++ b/applications/bulk_document_download/bulk_document_download_trigger_body.py @@ -0,0 +1,19 @@ +from uuid import UUID + +from pydantic import BaseModel, ConfigDict + + +class BulkDocumentDownloadTriggerBody(BaseModel): + """SQS trigger for the bulk_document_download Lambda (ADR-0055/0060). + + Attach mode: the FastAPI route created the Task + SubTask and pinned the + recipe (``landlord_property_ids``, ``recipient_email``, ``package_name``) + onto the SubTask's ``inputs``. The message therefore carries only the + identifiers — the (potentially large) property set never travels through the + 256 KB SQS body. + """ + + model_config = ConfigDict(extra="allow") + + task_id: UUID + subtask_id: UUID diff --git a/applications/bulk_document_download/handler.py b/applications/bulk_document_download/handler.py new file mode 100644 index 000000000..c5af7014c --- /dev/null +++ b/applications/bulk_document_download/handler.py @@ -0,0 +1,100 @@ +"""bulk_document_download Lambda (ADR-0060). + +Attach-mode (ADR-0055) handler. The FastAPI route created the app-owned Task + +SubTask and pinned the recipe onto the SubTask's ``inputs``; this reads that +recipe, assembles the Download Package through the orchestrator, and returns the +result (presigned URL + skipped-document summary), which `TaskOrchestrator` +records on ``sub_task.outputs``. A wholly-empty selection raises +``SubTaskFailure`` from the orchestrator — recorded, not retried. + +PostgresConfig-only (POSTGRES_*), like the bulk_upload_finaliser Lambda. Source +documents are read from each row's own ``s3_file_bucket``; the ZIP is written to +the dedicated ``DOCUMENT_EXPORTS_BUCKET``. SES SMTP settings arrive as env vars +(terraform bakes the +Secrets Manager IAM-user credentials in). +""" + +from __future__ import annotations + +import os +from typing import Any + +import boto3 + +from applications.bulk_document_download.bulk_document_download_trigger_body import ( + BulkDocumentDownloadTriggerBody, +) +from domain.tasks.tasks import Source +from infrastructure.email.ses_smtp_email_sender import SesSmtpEmailSender +from infrastructure.postgres.config import PostgresConfig +from infrastructure.postgres.engine import make_engine, make_session +from infrastructure.s3.s3_client import S3Client +from infrastructure.s3.s3_document_bytes_reader import S3DocumentBytesReader +from orchestration.bulk_document_download_orchestrator import ( + BulkDocumentDownloadOrchestrator, +) +from repositories.property.landlord_address_resolver import LandlordAddressResolver +from repositories.tasks.subtask_postgres_repository import SubTaskPostgresRepository +from repositories.uploaded_file.uploaded_file_postgres_repository import ( + UploadedFilePostgresRepository, +) +from utilities.aws_lambda.task_handler import task_handler + +# The Download Package link is valid for 60 minutes (ADR-0060). +_URL_TTL_SECONDS = 3600 + + +def _email_sender() -> SesSmtpEmailSender: + return SesSmtpEmailSender( + host=os.environ["SES_SMTP_HOST"], + port=int(os.environ["SES_SMTP_PORT"]), + username=os.environ["SES_SMTP_USERNAME"], + password=os.environ["SES_SMTP_PASSWORD"], + from_address=os.environ["SES_SMTP_FROM_ADDRESS"], + ) + + +@task_handler(task_source="bulk_document_download", source=Source.PORTFOLIO) +def handler(body: dict[str, Any], context: Any) -> dict[str, Any]: + trigger = BulkDocumentDownloadTriggerBody.model_validate(body) + engine = make_engine(PostgresConfig.from_env(os.environ)) + session = make_session(engine) + try: + subtask = SubTaskPostgresRepository(session).get(trigger.subtask_id) + recipe: dict[str, Any] = subtask.inputs or {} + landlord_property_ids: list[str] = [ + str(x) for x in recipe["landlord_property_ids"] + ] + recipient_email: str = str(recipe["recipient_email"]) + package_name: str = str(recipe["package_name"]) + + boto_s3: Any = boto3.client("s3") # pyright: ignore[reportUnknownMemberType] + orchestrator = BulkDocumentDownloadOrchestrator( + uploaded_files=UploadedFilePostgresRepository(session), + addresses=LandlordAddressResolver(session, landlord_property_ids), + documents=S3DocumentBytesReader(boto_s3), + packages=S3Client(boto_s3, os.environ["DOCUMENT_EXPORTS_BUCKET"]), + email=_email_sender(), + url_ttl_seconds=_URL_TTL_SECONDS, + ) + result = orchestrator.run( + landlord_property_ids=landlord_property_ids, + recipient_email=recipient_email, + package_name=package_name, + ) + finally: + session.close() + + return { + "presigned_url": result.presigned_url, + "package_s3_key": result.package_s3_key, + "included": result.included, + "skipped": [ + { + "landlord_property_id": skip.landlord_property_id, + "s3_key": skip.s3_key, + "reason": skip.reason, + } + for skip in result.skipped + ], + } diff --git a/applications/bulk_document_download/requirements.txt b/applications/bulk_document_download/requirements.txt new file mode 100644 index 000000000..6a85a2552 --- /dev/null +++ b/applications/bulk_document_download/requirements.txt @@ -0,0 +1,4 @@ +boto3 +pydantic +sqlmodel +psycopg2-binary From 6db6bbd98f6b57ff51475171a3fa4eb7472f8179 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:52:43 +0000 Subject: [PATCH 39/55] =?UTF-8?q?ADR-0060:=20dedicated=20exports=20bucket?= =?UTF-8?q?=20+=20multi-GB=20streaming=20(drop=20DATA=5FBUCKET=20lifecycle?= =?UTF-8?q?)=20=F0=9F=9F=AA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per review: packages go to a separate DOCUMENT_EXPORTS_BUCKET (not DATA_BUCKET), no 7-day lifecycle imposed on the shared data bucket; packages can be several GB so they stream via /tmp + multipart upload with raised ephemeral storage. Co-Authored-By: Claude Fable 5 --- CONTEXT.md | 2 +- ...60-bulk-document-download-package-model.md | 31 +++++++++++++------ 2 files changed, 23 insertions(+), 10 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 26c65a00c..33d79f62e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -104,7 +104,7 @@ The category of an uploaded document — the `uploaded_files.file_type` enum (ph _Avoid_: document category, file kind, doc class **Download Package**: -The single ZIP archive a **Bulk Document Download** produces: one folder per property (named by human-readable address, enriched from the hubspot deals data, with `landlord_property_id` appended for uniqueness), each folder holding the latest file of each **Document Type** held for that property. Built best-effort — properties/documents that don't resolve are skipped and listed in the run's `sub_task.outputs`, not failed (ADR-0060). Stored under `DATA_BUCKET/bulk-downloads/` with a ~7-day lifecycle; delivered as a 60-minute presigned URL. +The single ZIP archive a **Bulk Document Download** produces: one folder per property (named by human-readable address, enriched from the hubspot deals data, with `landlord_property_id` appended for uniqueness), each folder holding the latest file of each **Document Type** held for that property. Built best-effort — properties/documents that don't resolve are skipped and listed in the run's `sub_task.outputs`, not failed (ADR-0060). Streamed to a dedicated `DOCUMENT_EXPORTS_BUCKET` (never held whole in memory — a package can be several GB); delivered as a 60-minute presigned URL. _Avoid_: export (that is the plan/scenario XLSX export), bundle, archive (ambiguous), zip (the format, not the concept) **Bulk Document Download**: diff --git a/docs/adr/0060-bulk-document-download-package-model.md b/docs/adr/0060-bulk-document-download-package-model.md index 5e234bc0a..a220033ff 100644 --- a/docs/adr/0060-bulk-document-download-package-model.md +++ b/docs/adr/0060-bulk-document-download-package-model.md @@ -29,9 +29,13 @@ lane (ADR-0055).** not the 256 KB SQS body), pre-creates **one** `sub_task`, and drops one SQS message (`task_id`, `sub_task_id`). The new `applications/bulk_document_download` Lambda runs in **attach mode**; `TaskOrchestrator` owns status + roll-up. -- **One package = one sub_task.** No fan-out: the job streams each document out - of its own `s3_file_bucket` into a single ZIP in `/tmp` and multipart-uploads - it. One artifact, one URL, one email. +- **One package = one sub_task, streamed.** No fan-out. A Download Package can + be **several GB**, so it is never held whole in memory: each document is read + from its own `s3_file_bucket`, written into an on-disk ZIP in `/tmp`, and + released; the finished archive is **multipart-uploaded from disk** + (`S3Client.upload_file`). The Lambda's ephemeral storage (`/tmp`) is raised + accordingly (up to 10 GB) — see Consequences. One artifact, one URL, one + email. - **Selection cap at the route.** Reject `> N` properties synchronously with a legible "narrow your selection" error (property count, not a document/byte query — cheap, no S3 on the hot path). A `/tmp` size budget inside the Lambda @@ -55,10 +59,14 @@ lane (ADR-0055).** - **Delivery — both channels.** The presigned URL (60-minute expiry) and the skip-summary are written to `sub_task.outputs` (the FE already polls task status and can show the link) **and** emailed (ADR-0059). -- **Retention.** Packages are written to `DATA_BUCKET` under a - `bulk-downloads/` prefix with an S3 lifecycle rule expiring objects after - ~7 days — longer than the 60-minute URL (so a link can be re-issued without - rebuilding) but short enough that bundles never accumulate. +- **A dedicated exports bucket.** Packages are written to a **separate + `DOCUMENT_EXPORTS_BUCKET`**, not `DATA_BUCKET` — the generated archives are a + distinct class of artifact (transient, user-facing, wide-read source but + single-writer) and keeping them out of the shared data bucket keeps IAM and + any future retention policy cleanly scoped to exports. No lifecycle rule is + imposed on `DATA_BUCKET`. Retention on the exports bucket is an open decision + (the packages are transient, so an expiry is likely warranted, but it is left + to a follow-up rather than baked in here). ## Considered options @@ -86,7 +94,12 @@ lane (ADR-0055).** port/adapter from ADR-0059. The only `backend/` touch is the trigger route. - The stored, pinned `property_id` set makes a run reproducible even if the portfolio changes between trigger and execution. -- `N`, the `/tmp` size budget, and the 7-day lifecycle are tunable knobs, not - load-bearing invariants. +- `N`, the Lambda's ephemeral-storage (`/tmp`) size, and its memory are tunable + knobs, not load-bearing invariants. `/tmp` must be raised beyond the 512 MB + default (up to 10 GB) to hold a multi-GB archive — a new terraform knob on the + shared Lambda modules, backward-compatible (every other Lambda keeps 512 MB). +- A **new S3 bucket** (`DOCUMENT_EXPORTS_BUCKET`) is provisioned for the + packages, with its own IAM (single-writer, presign-read). Its retention policy + is a deliberate follow-up, not set here. - Closing the `uploaded_files.property_id` gap later would let matching move off `landlord_property_id` without changing the package model. From 3c00e0ef00c1f035fb3daca6e0aed892758e1595 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 09:59:02 +0000 Subject: [PATCH 40/55] =?UTF-8?q?FastAPI=20trigger=20route:=20POST=20/v1/d?= =?UTF-8?q?ocuments/bulk-download=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves the authenticated requester's email (ADR-0059), resolves + caps the property selection, pins the recipe (landlord_property_ids, recipient_email, package_name) onto one pre-created sub_task (raw SQL, dodging the mirror double-registration), and enqueues one message to the worker. Refuses double-submits (409) and oversized selections (400). Co-Authored-By: Claude Fable 5 --- backend/app/config.py | 1 + backend/app/documents/__init__.py | 0 backend/app/documents/download_tasks.py | 91 ++++++++++++ backend/app/documents/router.py | 132 +++++++++++++++++ backend/app/documents/schemas.py | 19 +++ backend/app/main.py | 2 + tests/backend/app/documents/__init__.py | 0 .../documents/test_bulk_download_router.py | 140 ++++++++++++++++++ 8 files changed, 385 insertions(+) create mode 100644 backend/app/documents/__init__.py create mode 100644 backend/app/documents/download_tasks.py create mode 100644 backend/app/documents/router.py create mode 100644 backend/app/documents/schemas.py create mode 100644 tests/backend/app/documents/__init__.py create mode 100644 tests/backend/app/documents/test_bulk_download_router.py diff --git a/backend/app/config.py b/backend/app/config.py index af79b0106..8f0d551f1 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -46,6 +46,7 @@ class Settings(BaseSettings): LANDLORD_OVERRIDES_SQS_URL: str = "changeme" FINALISER_SQS_URL: str = "changeme" MODELLING_E2E_SQS_URL: str = "changeme" + BULK_DOCUMENT_DOWNLOAD_SQS_URL: str = "changeme" # Third parties EPC_AUTH_TOKEN: str = "changeme" diff --git a/backend/app/documents/__init__.py b/backend/app/documents/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/app/documents/download_tasks.py b/backend/app/documents/download_tasks.py new file mode 100644 index 000000000..1ef26eb04 --- /dev/null +++ b/backend/app/documents/download_tasks.py @@ -0,0 +1,91 @@ +"""The bulk-download route's view of the app-owned task's sub_task (ADR-0060). + +Parameterised SQL rather than the SQLModel mirrors: importing the +``infrastructure.postgres`` mirror of ``sub_task`` into the FastAPI process +double-registers the table and crashes at import (the same constraint the +Modelling Run distributor works around — see ``modelling/run_tasks.py``). +""" + +import json +from datetime import datetime, timezone +from typing import Any, Optional +from uuid import UUID + +from sqlalchemy import text +from sqlmodel import Session + + +class DocumentDownloadTasks: + def __init__(self, session: Session) -> None: + self._session = session + + def already_distributed(self, task_id: UUID) -> bool: + """Whether the task already has a sub_task — the 409 guard against a + double-submit re-triggering the same download.""" + row = ( + self._session.connection() + .execute( + text("SELECT 1 FROM sub_task WHERE task_id = :task_id LIMIT 1"), + {"task_id": task_id}, + ) + .first() + ) + return row is not None + + def resolve_selection( + self, + portfolio_id: int, + property_ids: Optional[list[int]], + select_all: bool, + ) -> tuple[int, list[str]]: + """Resolve the selection to (selected property count, the distinct + ``landlord_property_id`` values that carry a downloadable identifier). + + The count is the cap basis (property count, ADR-0060); files are matched + by ``landlord_property_id`` (the missing ``property_id`` on + ``uploaded_files`` is a known future gap), so a selected property with a + null ``landlord_property_id`` contributes no files. + """ + if select_all: + rows = self._session.connection().execute( + text( + "SELECT id, landlord_property_id FROM property" + " WHERE portfolio_id = :portfolio_id" + ), + {"portfolio_id": portfolio_id}, + ) + else: + rows = self._session.connection().execute( + text( + "SELECT id, landlord_property_id FROM property" + " WHERE portfolio_id = :portfolio_id AND id = ANY(:ids)" + ), + {"portfolio_id": portfolio_id, "ids": property_ids or []}, + ) + resolved = rows.all() + landlord_property_ids = sorted( + {row[1] for row in resolved if row[1] is not None} + ) + return len(resolved), landlord_property_ids + + def create_download_subtask( + self, task_id: UUID, subtask_id: UUID, recipe: dict[str, Any] + ) -> None: + """Pre-create the single ``waiting`` sub_task under the app-owned task, + pinning the recipe (landlord_property_ids, recipient_email, + package_name) onto its inputs — the worker's reproducible instructions + (ADR-0055/0060).""" + now = datetime.now(timezone.utc) + self._session.connection().execute( + text( + "INSERT INTO sub_task (id, task_id, status, inputs, updated_at)" + " VALUES (:id, :task_id, 'waiting', :inputs, :updated_at)" + ), + { + "id": subtask_id, + "task_id": task_id, + "inputs": json.dumps(recipe), + "updated_at": now, + }, + ) + self._session.commit() diff --git a/backend/app/documents/router.py b/backend/app/documents/router.py new file mode 100644 index 000000000..7d3eea4db --- /dev/null +++ b/backend/app/documents/router.py @@ -0,0 +1,132 @@ +"""Bulk Document Download trigger: POST /v1/documents/bulk-download (ADR-0060). + +Resolves the authenticated requester's email, resolves + caps the property +selection, pins the recipe onto a single pre-created sub_task under the +app-owned task, and enqueues one message to the bulk_document_download worker. +Never assembles the package synchronously — the worker emails the link. +""" + +import json +from collections.abc import Callable, Iterator +from typing import Any, cast +from uuid import uuid4 + +import boto3 +from fastapi import APIRouter, Depends, HTTPException +from sqlmodel import Session + +from backend.app.config import get_settings +from backend.app.db.connection import db_engine +from backend.app.dependencies import validate_jwt_token, validate_token +from backend.app.documents.download_tasks import DocumentDownloadTasks +from backend.app.documents.schemas import BulkDownloadRequest + +# The selection cap (ADR-0060) — rejected at the trigger so an oversized request +# never reaches the worker. A conservative, tunable starting value. +MAX_PROPERTIES = 500 + +MessageSender = Callable[[str], None] + + +def get_session() -> Iterator[Session]: + with Session(db_engine) as session: + yield session + + +def get_message_sender() -> MessageSender: + settings = get_settings() + client: Any = cast(Any, boto3.client("sqs", settings.AWS_DEFAULT_REGION)) # pyright: ignore[reportUnknownMemberType] + queue_url = settings.BULK_DOCUMENT_DOWNLOAD_SQS_URL + + def send(body: str) -> None: + client.send_message(QueueUrl=queue_url, MessageBody=body) + + return send + + +def get_requesting_user_email(user: Any = Depends(validate_jwt_token)) -> str: + """The authenticated requester's email — the Download Package recipient. + + ADR-0059: the route resolves the user (JWT ``dbId`` -> ``UserModel``) rather + than only gating the token, and threads the email into the job.""" + email = getattr(user, "email", None) + if email is None and isinstance(user, dict): + email = user.get("email") + if not email: + raise HTTPException( + status_code=400, + detail="Could not determine the requesting user's email address.", + ) + return str(email) + + +router = APIRouter( + prefix="/documents", + tags=["documents"], + dependencies=[Depends(validate_token)], +) + + +@router.post("/bulk-download", status_code=202) +async def bulk_download( + body: BulkDownloadRequest, + session: Session = Depends(get_session), + send_message: MessageSender = Depends(get_message_sender), + recipient_email: str = Depends(get_requesting_user_email), +) -> dict[str, str]: + tasks = DocumentDownloadTasks(session) + if tasks.already_distributed(body.task_id): + raise HTTPException( + status_code=409, + detail=( + f"Task {body.task_id} already has a sub_task — a download has " + "already been started for it." + ), + ) + + if not body.select_all and not body.property_ids: + raise HTTPException( + status_code=400, + detail="Provide property_ids or set select_all.", + ) + + property_count, landlord_property_ids = tasks.resolve_selection( + body.portfolio_id, body.property_ids, body.select_all + ) + if property_count == 0: + raise HTTPException( + status_code=400, + detail=( + f"The selection resolves to no properties in portfolio " + f"{body.portfolio_id}." + ), + ) + if property_count > MAX_PROPERTIES: + raise HTTPException( + status_code=400, + detail=( + f"The selection has {property_count} properties, over the " + f"{MAX_PROPERTIES} limit — narrow your selection." + ), + ) + if not landlord_property_ids: + raise HTTPException( + status_code=400, + detail=( + "None of the selected properties have a landlord_property_id, " + "so no documents can be matched." + ), + ) + + subtask_id = uuid4() + recipe = { + "landlord_property_ids": landlord_property_ids, + "recipient_email": recipient_email, + "package_name": f"portfolio-{body.portfolio_id}-{body.task_id}", + } + tasks.create_download_subtask(body.task_id, subtask_id, recipe) + + send_message( + json.dumps({"task_id": str(body.task_id), "subtask_id": str(subtask_id)}) + ) + return {"message": "Bulk document download started"} diff --git a/backend/app/documents/schemas.py b/backend/app/documents/schemas.py new file mode 100644 index 000000000..4d9a06247 --- /dev/null +++ b/backend/app/documents/schemas.py @@ -0,0 +1,19 @@ +from typing import Optional +from uuid import UUID + +from pydantic import BaseModel + + +class BulkDownloadRequest(BaseModel): + """A request to assemble a Download Package (ADR-0060). + + The front end creates the app-owned Task first (its own request lands in + ``task.inputs``) and passes ``task_id`` here. Either an explicit + ``property_ids`` selection or ``select_all`` (the whole portfolio) must be + given. + """ + + task_id: UUID + portfolio_id: int + property_ids: Optional[list[int]] = None + select_all: bool = False diff --git a/backend/app/main.py b/backend/app/main.py index 4b4aaaa70..270917f63 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -11,6 +11,7 @@ from backend.app.whlg import router as whlg_router from backend.app.plan import router as plan_router from backend.app.tasks import router as tasks_router from backend.app.bulk_uploads import router as bulk_uploads_router +from backend.app.documents import router as documents_router from backend.app.modelling import router as modelling_router from backend.app.dependencies import validate_api_key from backend.app.config import get_settings @@ -65,6 +66,7 @@ app.include_router(plan_router.router, prefix="/v1") app.include_router(whlg_router.router, prefix="/v1") app.include_router(bulk_uploads_router.router, prefix="/v1") app.include_router(modelling_router.router, prefix="/v1") +app.include_router(documents_router.router, prefix="/v1") if get_settings().ENVIRONMENT == "local": from backend.app.local import router as local_router diff --git a/tests/backend/app/documents/__init__.py b/tests/backend/app/documents/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/backend/app/documents/test_bulk_download_router.py b/tests/backend/app/documents/test_bulk_download_router.py new file mode 100644 index 000000000..b07d6baf3 --- /dev/null +++ b/tests/backend/app/documents/test_bulk_download_router.py @@ -0,0 +1,140 @@ +"""Tests for the Bulk Document Download trigger endpoint (ADR-0060). + +Session, SQS and requester-email seams are dependency-injected; tests run +against the ephemeral Postgres and record message bodies instead of calling AWS. +""" + +import json +from collections.abc import Iterator +from dataclasses import dataclass, field +from typing import Any + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient +from sqlalchemy import Engine +from sqlmodel import Session + +from backend.app.dependencies import validate_token +from backend.app.documents import router as documents_router +from backend.app.documents.router import ( + get_message_sender, + get_requesting_user_email, + get_session, +) +from domain.tasks.tasks import Task +from infrastructure.postgres.property_table import PropertyRow +from repositories.tasks.subtask_postgres_repository import SubTaskPostgresRepository +from repositories.tasks.task_postgres_repository import TaskPostgresRepository + +PORTFOLIO_ID = 814 +RECIPIENT = "tester@example.com" + + +@dataclass +class Api: + client: TestClient + engine: Engine + sent_bodies: list[str] = field(default_factory=list) + + def seed_task(self) -> Task: + with Session(self.engine) as session: + return TaskPostgresRepository(session).create( + Task.create(task_source="app:bulk_document_download") + ) + + def seed_property(self, landlord_property_id: str) -> int: + with Session(self.engine) as session: + row = PropertyRow( + portfolio_id=PORTFOLIO_ID, landlord_property_id=landlord_property_id + ) + session.add(row) + session.commit() + session.refresh(row) + assert row.id is not None + return row.id + + def subtask_inputs(self, task: Task) -> list[dict[str, Any]]: + with Session(self.engine) as session: + subtasks = SubTaskPostgresRepository(session).list_by_task(task.id) + return [st.inputs or {} for st in subtasks] + + +@pytest.fixture +def api(db_engine: Engine) -> Api: + app = FastAPI() + app.include_router(documents_router.router, prefix="/v1") + harness = Api(client=TestClient(app), engine=db_engine) + + def _session() -> Iterator[Session]: + with Session(db_engine) as session: + yield session + + app.dependency_overrides[get_session] = _session + app.dependency_overrides[get_message_sender] = lambda: harness.sent_bodies.append + app.dependency_overrides[get_requesting_user_email] = lambda: RECIPIENT + app.dependency_overrides[validate_token] = lambda: "test-token" + return harness + + +def _trigger(api: Api, task: Task, property_ids: list[int]) -> Any: + return api.client.post( + "/v1/documents/bulk-download", + json={ + "task_id": str(task.id), + "portfolio_id": PORTFOLIO_ID, + "property_ids": property_ids, + }, + ) + + +def test_pins_the_recipe_and_enqueues_one_message(api: Api) -> None: + # arrange — two selected properties, matched by landlord_property_id. + task = api.seed_task() + p1 = api.seed_property("LP1") + p2 = api.seed_property("LP2") + + # act + response = _trigger(api, task, [p1, p2]) + + # assert — accepted; one sub_task pins the recipe; one message enqueued. + assert response.status_code == 202 + inputs = api.subtask_inputs(task) + assert len(inputs) == 1 + assert inputs[0]["landlord_property_ids"] == ["LP1", "LP2"] + assert inputs[0]["recipient_email"] == RECIPIENT + assert len(api.sent_bodies) == 1 + message = json.loads(api.sent_bodies[0]) + assert message["task_id"] == str(task.id) + assert "subtask_id" in message + + +def test_rejects_a_selection_over_the_cap(api: Api, monkeypatch: Any) -> None: + # arrange — cap of 1, two selected properties. + monkeypatch.setattr(documents_router, "MAX_PROPERTIES", 1) + task = api.seed_task() + p1 = api.seed_property("LP1") + p2 = api.seed_property("LP2") + + # act + response = _trigger(api, task, [p1, p2]) + + # assert — refused; nothing created or sent. + assert response.status_code == 400 + assert api.subtask_inputs(task) == [] + assert api.sent_bodies == [] + + +def test_rejects_a_task_that_already_has_a_subtask(api: Api) -> None: + # arrange — a task already triggered once. + task = api.seed_task() + p1 = api.seed_property("LP1") + assert _trigger(api, task, [p1]).status_code == 202 + + # act + response = _trigger(api, task, [p1]) + + # assert — the double-submit is refused; still one sub_task, one message. + assert response.status_code == 409 + assert len(api.subtask_inputs(task)) == 1 + assert len(api.sent_bodies) == 1 From b6746cc24afdacfd3091eaa556acb4ddbb2abd0c Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 10:03:50 +0000 Subject: [PATCH 41/55] =?UTF-8?q?Terraform=20+=20CI=20for=20bulk=5Fdocumen?= =?UTF-8?q?t=5Fdownload:=20exports=20bucket,=2010GB=20/tmp,=20SES=20SMTP,?= =?UTF-8?q?=20wiring=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New lambda_with_sqs consumer (timeout 900, memory 3008, ephemeral_storage 10240 for multi-GB ZIPs); dedicated retrofit-document-exports bucket (no lifecycle on DATA_BUCKET); IAM to read source buckets + write/presign-read the exports bucket; SES SMTP creds baked from Secrets Manager (no ses:* on the role). Adds ephemeral_storage_size knob to the shared lambda modules (default 512, backward compatible). Wires the queue url+arn into fast-api and orders the CI jobs (ADR-0055). NOTE: terraform is drafted, not validated (no AWS/terraform in the dev env). Co-Authored-By: Claude Fable 5 --- .github/workflows/deploy_terraform.yml | 38 +++++- .../lambda/bulk_document_download/main.tf | 111 ++++++++++++++++++ .../lambda/bulk_document_download/outputs.tf | 9 ++ .../lambda/bulk_document_download/provider.tf | 20 ++++ .../bulk_document_download/variables.tf | 61 ++++++++++ deployment/terraform/lambda/fast-api/main.tf | 14 ++- .../terraform/modules/lambda_service/main.tf | 4 + .../modules/lambda_service/variables.tf | 6 + .../terraform/modules/lambda_with_sqs/main.tf | 2 + .../modules/lambda_with_sqs/variables.tf | 5 + deployment/terraform/shared/main.tf | 32 +++++ 11 files changed, 300 insertions(+), 2 deletions(-) create mode 100644 deployment/terraform/lambda/bulk_document_download/main.tf create mode 100644 deployment/terraform/lambda/bulk_document_download/outputs.tf create mode 100644 deployment/terraform/lambda/bulk_document_download/provider.tf create mode 100644 deployment/terraform/lambda/bulk_document_download/variables.tf diff --git a/.github/workflows/deploy_terraform.yml b/.github/workflows/deploy_terraform.yml index 228b1fcea..cc2bf4219 100644 --- a/.github/workflows/deploy_terraform.yml +++ b/.github/workflows/deploy_terraform.yml @@ -538,7 +538,7 @@ jobs: # Deploy FastAPI Lambda # ============================================================ fast_api_lambda: - needs: [determine_stage, ara_engine_lambda, categorisation_lambda, postcodeSplitter_lambda, bulk_address2uprn_combiner_lambda, bulkUploadFinaliser_lambda, modelling_e2e_lambda] + needs: [determine_stage, ara_engine_lambda, categorisation_lambda, postcodeSplitter_lambda, bulk_address2uprn_combiner_lambda, bulkUploadFinaliser_lambda, modelling_e2e_lambda, bulk_document_download_lambda] uses: ./.github/workflows/_deploy_lambda.yml with: lambda_name: ara_fast_api @@ -816,6 +816,42 @@ jobs: TF_VAR_open_epc_api_token: ${{ secrets.DEV_OPEN_EPC_API_TOKEN }} TF_VAR_google_solar_api_key: ${{ secrets.DEV_GOOGLE_SOLAR_API_KEY }} + # ============================================================ + # Build Bulk Document Download image and Push + # ============================================================ + bulk_document_download_image: + needs: [determine_stage, shared_terraform] + uses: ./.github/workflows/_build_image.yml + with: + ecr_repo: bulk-document-download-${{ needs.determine_stage.outputs.stage }} + dockerfile_path: applications/bulk_document_download/Dockerfile + build_context: . + secrets: + AWS_ACCESS_KEY_ID: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }} + AWS_REGION: ${{ secrets.DEV_AWS_REGION }} + + # ============================================================ + # Deploy Bulk Document Download Lambda + # ============================================================ + bulk_document_download_lambda: + needs: [bulk_document_download_image, determine_stage] + uses: ./.github/workflows/_deploy_lambda.yml + with: + lambda_name: bulk_document_download + lambda_path: deployment/terraform/lambda/bulk_document_download + stage: ${{ needs.determine_stage.outputs.stage }} + ecr_repo: bulk-document-download-${{ needs.determine_stage.outputs.stage }} + image_digest: ${{ needs.bulk_document_download_image.outputs.image_digest }} + terraform_apply: ${{ needs.determine_stage.outputs.terraform_apply }} + secrets: + AWS_ACCESS_KEY_ID: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }} + AWS_REGION: ${{ secrets.DEV_AWS_REGION }} + TF_VAR_db_host: ${{ secrets.DEV_DB_HOST }} + TF_VAR_db_name: ${{ secrets.DEV_DB_NAME }} + TF_VAR_db_port: ${{ secrets.DEV_DB_PORT }} + # ============================================================ # Deploy Hubspot ETL Lambda # ============================================================ diff --git a/deployment/terraform/lambda/bulk_document_download/main.tf b/deployment/terraform/lambda/bulk_document_download/main.tf new file mode 100644 index 000000000..3325a1ff1 --- /dev/null +++ b/deployment/terraform/lambda/bulk_document_download/main.tf @@ -0,0 +1,111 @@ +############################################ +# Credentials from Secrets Manager (resolved at plan/apply time and baked into +# env vars). The SES SMTP credentials are an IAM *user* (see modules/ses), so +# the Lambda authenticates to SES over SMTP with them — the execution role needs +# neither ses:* nor runtime secretsmanager access. +############################################ +data "aws_secretsmanager_secret_version" "db_credentials" { + secret_id = "${var.stage}/assessment_model/db_credentials" +} + +data "aws_secretsmanager_secret_version" "ses_smtp" { + secret_id = "${var.stage}/ses/smtp_credentials" +} + +locals { + db_credentials = jsondecode(data.aws_secretsmanager_secret_version.db_credentials.secret_string) + ses_smtp = jsondecode(data.aws_secretsmanager_secret_version.ses_smtp.secret_string) + + document_exports_bucket = "retrofit-document-exports-${var.stage}" + + # Source buckets the uploaded_files rows point at (read for packaging). + # OPEN QUESTION: confirm this is the closed set — if files can live in + # arbitrary buckets, widen to ["arn:aws:s3:::*"]. + source_bucket_arns = [ + "arn:aws:s3:::retrofit-data-${var.stage}", + "arn:aws:s3:::retrofit-energy-assessments-${var.stage}", + ] +} + +############################################ +# Lambda + SQS queue + trigger +############################################ +module "lambda" { + source = "../../modules/lambda_with_sqs" + + name = var.lambda_name + stage = var.stage + + image_uri = local.image_uri + + reserved_concurrent_executions = var.reserved_concurrent_executions + + batch_size = var.batch_size + maximum_concurrency = var.maximum_concurrency + + timeout = 900 + memory_size = 3008 + # A Download Package can be several GB; it is streamed to /tmp before the + # multipart upload (ADR-0060), so raise ephemeral storage to the 10 GB max. + ephemeral_storage_size = 10240 + + environment = { + STAGE = var.stage + LOG_LEVEL = "info" + + POSTGRES_USERNAME = local.db_credentials.db_assessment_model_username + POSTGRES_PASSWORD = local.db_credentials.db_assessment_model_password + POSTGRES_HOST = var.db_host + POSTGRES_DATABASE = var.db_name + POSTGRES_PORT = var.db_port + + DOCUMENT_EXPORTS_BUCKET = local.document_exports_bucket + + # SES SMTP — IAM-user credentials sourced from Secrets Manager (modules/ses). + SES_SMTP_HOST = "email-smtp.eu-west-2.amazonaws.com" + SES_SMTP_PORT = "587" + SES_SMTP_USERNAME = local.ses_smtp.username + SES_SMTP_PASSWORD = local.ses_smtp.password + SES_SMTP_FROM_ADDRESS = var.ses_from_address + } +} + +############################################ +# IAM: read uploaded documents from their source buckets +############################################ +module "s3_read" { + source = "../../modules/s3_iam_policy" + + policy_name = "BulkDocumentDownloadS3Read-${var.stage}" + policy_description = "Allow bulk_document_download Lambda to read uploaded_files objects from their source buckets" + bucket_arns = local.source_bucket_arns + actions = ["s3:GetObject", "s3:ListBucket"] + resource_paths = ["/*"] +} + +resource "aws_iam_role_policy_attachment" "s3_read" { + role = module.lambda.role_name + policy_arn = module.s3_read.policy_arn +} + +############################################ +# IAM: write + read the assembled ZIP on the dedicated exports bucket. +# GetObject is required so the Lambda-signed presigned GET URL resolves. +############################################ +module "s3_exports" { + source = "../../modules/s3_iam_policy" + + policy_name = "BulkDocumentDownloadS3Exports-${var.stage}" + policy_description = "Allow bulk_document_download Lambda to write and presign-read Download Packages on the exports bucket" + bucket_arns = ["arn:aws:s3:::${local.document_exports_bucket}"] + actions = ["s3:PutObject", "s3:GetObject"] + resource_paths = ["/*"] +} + +resource "aws_iam_role_policy_attachment" "s3_exports" { + role = module.lambda.role_name + policy_arn = module.s3_exports.policy_arn +} + +# NOTE: no ses:* on the role — the handler sends over SMTP with the SES IAM-user +# credentials injected above. diff --git a/deployment/terraform/lambda/bulk_document_download/outputs.tf b/deployment/terraform/lambda/bulk_document_download/outputs.tf new file mode 100644 index 000000000..7f2442c26 --- /dev/null +++ b/deployment/terraform/lambda/bulk_document_download/outputs.tf @@ -0,0 +1,9 @@ +output "bulk_document_download_queue_url" { + value = module.lambda.queue_url + description = "URL of the bulk-document-download SQS queue (FastAPI enqueues jobs here)" +} + +output "bulk_document_download_queue_arn" { + value = module.lambda.queue_arn + description = "ARN of the bulk-document-download SQS queue" +} diff --git a/deployment/terraform/lambda/bulk_document_download/provider.tf b/deployment/terraform/lambda/bulk_document_download/provider.tf new file mode 100644 index 000000000..2ea6c350c --- /dev/null +++ b/deployment/terraform/lambda/bulk_document_download/provider.tf @@ -0,0 +1,20 @@ +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + version = ">= 5.0" + } + } + + backend "s3" { + bucket = "bulk-document-download-terraform-state" + key = "terraform.tfstate" + region = "eu-west-2" + } + + required_version = ">= 1.2.0" +} + +provider "aws" { + region = "eu-west-2" +} diff --git a/deployment/terraform/lambda/bulk_document_download/variables.tf b/deployment/terraform/lambda/bulk_document_download/variables.tf new file mode 100644 index 000000000..9abb25362 --- /dev/null +++ b/deployment/terraform/lambda/bulk_document_download/variables.tf @@ -0,0 +1,61 @@ +variable "lambda_name" { + type = string + description = "Logical name of the lambda" +} + +variable "stage" { + description = "Deployment stage (e.g. dev, prod)" + type = string +} + +variable "ecr_repo_url" { + type = string + description = "ECR repository URL (no tag, no digest)" +} + +variable "image_digest" { + type = string + description = "Image digest (sha256:...)" +} + +variable "reserved_concurrent_executions" { + type = number + default = -1 + description = "Reserved concurrency for the Lambda. -1 = unreserved." +} + +variable "maximum_concurrency" { + type = number + default = 5 + description = "Maximum concurrent Lambda invocations from the SQS trigger." +} + +variable "batch_size" { + type = number + default = 1 +} + +variable "db_host" { + type = string + sensitive = true +} + +variable "db_name" { + type = string + sensitive = true +} + +variable "db_port" { + type = string + sensitive = true +} + +variable "ses_from_address" { + type = string + description = "Verified SES sender for Download Package notifications (ADR-0059)." + default = "noreply@domna.homes" +} + +locals { + image_uri = "${var.ecr_repo_url}@${var.image_digest}" +} diff --git a/deployment/terraform/lambda/fast-api/main.tf b/deployment/terraform/lambda/fast-api/main.tf index 26e154d6a..73310b999 100644 --- a/deployment/terraform/lambda/fast-api/main.tf +++ b/deployment/terraform/lambda/fast-api/main.tf @@ -73,6 +73,15 @@ data "terraform_remote_state" "modelling_e2e" { } } +data "terraform_remote_state" "bulk_document_download" { + backend = "s3" + config = { + bucket = "bulk-document-download-terraform-state", + key = "env:/${var.stage}/terraform.tfstate" + region = "eu-west-2" + } +} + ############################################ # Load Credentials ############################################ @@ -136,6 +145,8 @@ module "fastapi" { FINALISER_SQS_URL = data.terraform_remote_state.bulk_upload_finaliser.outputs.bulk_upload_finaliser_queue_url LANDLORD_OVERRIDES_SQS_URL = data.terraform_remote_state.landlord_description_overrides.outputs.landlord_description_overrides_queue_url MODELLING_E2E_SQS_URL = data.terraform_remote_state.modelling_e2e.outputs.modelling_e2e_queue_url + + BULK_DOCUMENT_DOWNLOAD_SQS_URL = data.terraform_remote_state.bulk_document_download.outputs.bulk_document_download_queue_url } } @@ -160,7 +171,8 @@ module "fastapi_sqs_policy" { data.terraform_remote_state.bulk_address2uprn_combiner.outputs.bulk_address2uprn_combiner_queue_arn, data.terraform_remote_state.bulk_upload_finaliser.outputs.bulk_upload_finaliser_queue_arn, data.terraform_remote_state.landlord_description_overrides.outputs.landlord_description_overrides_queue_arn, - data.terraform_remote_state.modelling_e2e.outputs.modelling_e2e_queue_arn + data.terraform_remote_state.modelling_e2e.outputs.modelling_e2e_queue_arn, + data.terraform_remote_state.bulk_document_download.outputs.bulk_document_download_queue_arn ] conditions = null diff --git a/deployment/terraform/modules/lambda_service/main.tf b/deployment/terraform/modules/lambda_service/main.tf index 3250110ba..0981deac1 100644 --- a/deployment/terraform/modules/lambda_service/main.tf +++ b/deployment/terraform/modules/lambda_service/main.tf @@ -11,6 +11,10 @@ resource "aws_lambda_function" "this" { reserved_concurrent_executions = var.reserved_concurrent_executions + ephemeral_storage { + size = var.ephemeral_storage_size + } + environment { variables = var.environment } diff --git a/deployment/terraform/modules/lambda_service/variables.tf b/deployment/terraform/modules/lambda_service/variables.tf index 46241f300..c0ddc10ff 100644 --- a/deployment/terraform/modules/lambda_service/variables.tf +++ b/deployment/terraform/modules/lambda_service/variables.tf @@ -12,6 +12,12 @@ variable "memory_size" { default = 512 } +variable "ephemeral_storage_size" { + type = number + default = 512 + description = "Lambda /tmp size in MB (512-10240). Default 512 keeps every existing Lambda unchanged; raised only where a large on-disk artifact is built (e.g. multi-GB Download Package ZIPs, ADR-0060)." +} + variable "environment" { type = map(string) default = {} diff --git a/deployment/terraform/modules/lambda_with_sqs/main.tf b/deployment/terraform/modules/lambda_with_sqs/main.tf index 97f867935..2d8259dcf 100644 --- a/deployment/terraform/modules/lambda_with_sqs/main.tf +++ b/deployment/terraform/modules/lambda_with_sqs/main.tf @@ -31,6 +31,8 @@ module "lambda" { timeout = var.timeout memory_size = var.memory_size + ephemeral_storage_size = var.ephemeral_storage_size + environment = var.environment reserved_concurrent_executions = var.reserved_concurrent_executions } diff --git a/deployment/terraform/modules/lambda_with_sqs/variables.tf b/deployment/terraform/modules/lambda_with_sqs/variables.tf index 90585e929..d75ff28e8 100644 --- a/deployment/terraform/modules/lambda_with_sqs/variables.tf +++ b/deployment/terraform/modules/lambda_with_sqs/variables.tf @@ -25,6 +25,11 @@ variable "memory_size" { default = 1024 } +variable "ephemeral_storage_size" { + type = number + default = 512 +} + variable "environment" { type = map(string) default = {} diff --git a/deployment/terraform/shared/main.tf b/deployment/terraform/shared/main.tf index b4cdafbcc..a492401de 100644 --- a/deployment/terraform/shared/main.tf +++ b/deployment/terraform/shared/main.tf @@ -895,6 +895,38 @@ output "modelling_e2e_ecr_url" { } +################################################ +# Bulk Document Download – Lambda (ADR-0060) +################################################ +module "bulk_document_download_state_bucket" { + source = "../modules/tf_state_bucket" + bucket_name = "bulk-document-download-terraform-state" +} + +module "bulk_document_download_registry" { + source = "../modules/container_registry" + name = "bulk-document-download" + stage = var.stage +} + +# Dedicated bucket for generated Download Packages — kept out of the shared data +# bucket so retention/IAM stay scoped to exports (ADR-0060). +module "document_exports" { + source = "../modules/s3" + bucketname = "retrofit-document-exports-${var.stage}" + allowed_origins = var.allowed_origins +} + +output "document_exports_bucket_name" { + value = module.document_exports.bucket_name + description = "Name of the document exports bucket (Download Packages)" +} + +output "bulk_document_download_ecr_url" { + value = module.bulk_document_download_registry.repository_url +} + + ################################################ # Abri OpenHousing – Lambda ################################################ From e4f14ed8838a6c9ec3fa4fd41682f083ba0c28ae Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 10:05:11 +0000 Subject: [PATCH 42/55] =?UTF-8?q?Type=20the=20requester-email=20extraction?= =?UTF-8?q?=20cleanly=20=F0=9F=9F=AA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- backend/app/documents/router.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/backend/app/documents/router.py b/backend/app/documents/router.py index 7d3eea4db..f35974296 100644 --- a/backend/app/documents/router.py +++ b/backend/app/documents/router.py @@ -49,9 +49,9 @@ def get_requesting_user_email(user: Any = Depends(validate_jwt_token)) -> str: ADR-0059: the route resolves the user (JWT ``dbId`` -> ``UserModel``) rather than only gating the token, and threads the email into the job.""" - email = getattr(user, "email", None) + email: Any = getattr(user, "email", None) if email is None and isinstance(user, dict): - email = user.get("email") + email = cast(dict[str, Any], user).get("email") if not email: raise HTTPException( status_code=400, From 0b52a28808d561de8ea71a05eb35bac69e816fdb Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 10:41:14 +0000 Subject: [PATCH 43/55] =?UTF-8?q?Read=20the=20property=20selection=20from?= =?UTF-8?q?=20task.inputs;=20route=20takes=20only=20task=5Fid=20?= =?UTF-8?q?=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per PR review: the FE writes the selection config ({portfolio_id, property_ids?, select_all?}) into the FE-owned task.inputs and passes only task_id, so a large hand-picked selection never travels in an HTTP body. The route reads task.inputs, resolves to landlord_property_ids, caps, and pins the recipe onto sub_task.inputs as before. Declares the FE-owned inputs column on the TaskRow mirror so the backend can read it and the test schema builds it. Co-Authored-By: Claude Fable 5 --- backend/app/documents/download_tasks.py | 19 ++++- backend/app/documents/router.py | 19 +++-- backend/app/documents/schemas.py | 13 ++-- ...60-bulk-document-download-package-model.md | 21 ++++-- infrastructure/postgres/task_table.py | 7 +- .../documents/test_bulk_download_router.py | 71 ++++++++++++------- 6 files changed, 102 insertions(+), 48 deletions(-) diff --git a/backend/app/documents/download_tasks.py b/backend/app/documents/download_tasks.py index 1ef26eb04..9c4940407 100644 --- a/backend/app/documents/download_tasks.py +++ b/backend/app/documents/download_tasks.py @@ -8,7 +8,7 @@ Modelling Run distributor works around — see ``modelling/run_tasks.py``). import json from datetime import datetime, timezone -from typing import Any, Optional +from typing import Any, Optional, cast from uuid import UUID from sqlalchemy import text @@ -19,6 +19,23 @@ class DocumentDownloadTasks: def __init__(self, session: Session) -> None: self._session = session + def read_selection_config(self, task_id: UUID) -> dict[str, Any]: + """The task's selection config, read from the FE-owned ``task.inputs`` + JSON (ADR-0060) — ``{portfolio_id, property_ids?, select_all?}``. Empty + dict when the task has no inputs.""" + row = ( + self._session.connection() + .execute( + text("SELECT inputs FROM tasks WHERE id = :task_id"), + {"task_id": task_id}, + ) + .first() + ) + if row is None or row[0] is None: + return {} + parsed: Any = json.loads(row[0]) + return cast(dict[str, Any], parsed) if isinstance(parsed, dict) else {} + def already_distributed(self, task_id: UUID) -> bool: """Whether the task already has a sub_task — the 409 guard against a double-submit re-triggering the same download.""" diff --git a/backend/app/documents/router.py b/backend/app/documents/router.py index f35974296..06bc24723 100644 --- a/backend/app/documents/router.py +++ b/backend/app/documents/router.py @@ -84,21 +84,30 @@ async def bulk_download( ), ) - if not body.select_all and not body.property_ids: + config = tasks.read_selection_config(body.task_id) + portfolio_id = config.get("portfolio_id") + property_ids = config.get("property_ids") + select_all = bool(config.get("select_all", False)) + if not isinstance(portfolio_id, int): raise HTTPException( status_code=400, - detail="Provide property_ids or set select_all.", + detail="task.inputs is missing an integer portfolio_id.", + ) + if not select_all and not property_ids: + raise HTTPException( + status_code=400, + detail="task.inputs must provide property_ids or set select_all.", ) property_count, landlord_property_ids = tasks.resolve_selection( - body.portfolio_id, body.property_ids, body.select_all + portfolio_id, property_ids, select_all ) if property_count == 0: raise HTTPException( status_code=400, detail=( f"The selection resolves to no properties in portfolio " - f"{body.portfolio_id}." + f"{portfolio_id}." ), ) if property_count > MAX_PROPERTIES: @@ -122,7 +131,7 @@ async def bulk_download( recipe = { "landlord_property_ids": landlord_property_ids, "recipient_email": recipient_email, - "package_name": f"portfolio-{body.portfolio_id}-{body.task_id}", + "package_name": f"portfolio-{portfolio_id}-{body.task_id}", } tasks.create_download_subtask(body.task_id, subtask_id, recipe) diff --git a/backend/app/documents/schemas.py b/backend/app/documents/schemas.py index 4d9a06247..0d60006d8 100644 --- a/backend/app/documents/schemas.py +++ b/backend/app/documents/schemas.py @@ -1,4 +1,3 @@ -from typing import Optional from uuid import UUID from pydantic import BaseModel @@ -7,13 +6,11 @@ from pydantic import BaseModel class BulkDownloadRequest(BaseModel): """A request to assemble a Download Package (ADR-0060). - The front end creates the app-owned Task first (its own request lands in - ``task.inputs``) and passes ``task_id`` here. Either an explicit - ``property_ids`` selection or ``select_all`` (the whole portfolio) must be - given. + The front end creates the app-owned Task first and writes the **selection + config** into ``task.inputs`` (a JSON object of + ``{portfolio_id, property_ids?, select_all?}`` — so a large hand-picked set + never travels in an HTTP body), then passes only the ``task_id`` here. The + backend reads the selection from ``task.inputs``. """ task_id: UUID - portfolio_id: int - property_ids: Optional[list[int]] = None - select_all: bool = False diff --git a/docs/adr/0060-bulk-document-download-package-model.md b/docs/adr/0060-bulk-document-download-package-model.md index a220033ff..103a01b47 100644 --- a/docs/adr/0060-bulk-document-download-package-model.md +++ b/docs/adr/0060-bulk-document-download-package-model.md @@ -22,13 +22,20 @@ property, each holding the latest file of each Document Type — built best-effort, size-capped at the trigger, on the app-owned-task + attach-mode lane (ADR-0055).** -- **Trigger & lifecycle (ADR-0055).** The FastAPI route creates the `tasks` - row, resolves the selection to a concrete `property_id` set (expanding "all - properties" of a portfolio), **pins that set plus the recipient email into - `tasks.inputs`** (TEXT/JSON — a large id list is why this lives in `inputs`, - not the 256 KB SQS body), pre-creates **one** `sub_task`, and drops one SQS - message (`task_id`, `sub_task_id`). The new `applications/bulk_document_download` - Lambda runs in **attach mode**; `TaskOrchestrator` owns status + roll-up. +- **Trigger & lifecycle (ADR-0055).** The **front end** creates the app-owned + `tasks` row and writes the **selection config** — + `{portfolio_id, property_ids?, select_all?}` — into `tasks.inputs` (TEXT/JSON, + FE-owned), then calls the FastAPI route with **only the `task_id`**. A large + hand-picked selection therefore never travels in an HTTP body. The route + **reads the selection from `tasks.inputs`**, resolves it to the distinct + `landlord_property_id` set, caps it, resolves the recipient email from the + authenticated user (ADR-0059), **pins the resolved recipe** + (`landlord_property_ids`, `recipient_email`, `package_name`) onto **one + pre-created `sub_task`'s `inputs`**, and drops one SQS message (`task_id`, + `sub_task_id`). The new `applications/bulk_document_download` Lambda runs in + **attach mode**, reading that recipe from the sub_task; `TaskOrchestrator` + owns status + roll-up. (`task.inputs` = the FE's selection; `sub_task.inputs` + = the backend's resolved recipe.) - **One package = one sub_task, streamed.** No fan-out. A Download Package can be **several GB**, so it is never held whole in memory: each document is read from its own `s3_file_bucket`, written into an on-disk ZIP in `/tmp`, and diff --git a/infrastructure/postgres/task_table.py b/infrastructure/postgres/task_table.py index 32e5450bb..4ee46dc4d 100644 --- a/infrastructure/postgres/task_table.py +++ b/infrastructure/postgres/task_table.py @@ -2,7 +2,7 @@ from datetime import datetime, timezone from typing import ClassVar, Optional from uuid import UUID, uuid4 -from sqlalchemy import Column +from sqlalchemy import Column, Text from sqlalchemy import Enum as SAEnum from sqlmodel import Field, SQLModel @@ -18,6 +18,11 @@ class TaskRow(SQLModel, table=True): job_completed: Optional[datetime] = None status: str = Field(default="waiting") service: Optional[str] = None + # FE-owned (Drizzle) TEXT column holding the task's request as a JSON + # string. Declared here so the backend can read it (e.g. the Bulk Document + # Download route reads its property selection from it, ADR-0060) and so the + # test schema builds the column; prod owns the real column via FE migrations. + inputs: Optional[str] = Field(default=None, sa_column=Column(Text, nullable=True)) updated_at: datetime = Field( default_factory=lambda: datetime.now(timezone.utc) ) diff --git a/tests/backend/app/documents/test_bulk_download_router.py b/tests/backend/app/documents/test_bulk_download_router.py index b07d6baf3..ae5f38725 100644 --- a/tests/backend/app/documents/test_bulk_download_router.py +++ b/tests/backend/app/documents/test_bulk_download_router.py @@ -8,6 +8,7 @@ import json from collections.abc import Iterator from dataclasses import dataclass, field from typing import Any +from uuid import UUID import pytest from fastapi import FastAPI @@ -22,10 +23,9 @@ from backend.app.documents.router import ( get_requesting_user_email, get_session, ) -from domain.tasks.tasks import Task from infrastructure.postgres.property_table import PropertyRow +from infrastructure.postgres.task_table import TaskRow from repositories.tasks.subtask_postgres_repository import SubTaskPostgresRepository -from repositories.tasks.task_postgres_repository import TaskPostgresRepository PORTFOLIO_ID = 814 RECIPIENT = "tester@example.com" @@ -37,11 +37,18 @@ class Api: engine: Engine sent_bodies: list[str] = field(default_factory=list) - def seed_task(self) -> Task: + def seed_task(self, selection: dict[str, Any]) -> UUID: + """Create the app-owned task with the FE-written selection config on + ``task.inputs`` (ADR-0060).""" with Session(self.engine) as session: - return TaskPostgresRepository(session).create( - Task.create(task_source="app:bulk_document_download") + row = TaskRow( + task_source="app:bulk_document_download", + inputs=json.dumps(selection), ) + session.add(row) + session.commit() + session.refresh(row) + return row.id def seed_property(self, landlord_property_id: str) -> int: with Session(self.engine) as session: @@ -54,9 +61,9 @@ class Api: assert row.id is not None return row.id - def subtask_inputs(self, task: Task) -> list[dict[str, Any]]: + def subtask_inputs(self, task_id: UUID) -> list[dict[str, Any]]: with Session(self.engine) as session: - subtasks = SubTaskPostgresRepository(session).list_by_task(task.id) + subtasks = SubTaskPostgresRepository(session).list_by_task(task_id) return [st.inputs or {} for st in subtasks] @@ -77,64 +84,76 @@ def api(db_engine: Engine) -> Api: return harness -def _trigger(api: Api, task: Task, property_ids: list[int]) -> Any: +def _trigger(api: Api, task_id: UUID) -> Any: return api.client.post( - "/v1/documents/bulk-download", - json={ - "task_id": str(task.id), - "portfolio_id": PORTFOLIO_ID, - "property_ids": property_ids, - }, + "/v1/documents/bulk-download", json={"task_id": str(task_id)} ) def test_pins_the_recipe_and_enqueues_one_message(api: Api) -> None: - # arrange — two selected properties, matched by landlord_property_id. - task = api.seed_task() + # arrange — two selected properties, pinned as the FE's task.inputs config. p1 = api.seed_property("LP1") p2 = api.seed_property("LP2") + task_id = api.seed_task( + {"portfolio_id": PORTFOLIO_ID, "property_ids": [p1, p2]} + ) # act - response = _trigger(api, task, [p1, p2]) + response = _trigger(api, task_id) # assert — accepted; one sub_task pins the recipe; one message enqueued. assert response.status_code == 202 - inputs = api.subtask_inputs(task) + inputs = api.subtask_inputs(task_id) assert len(inputs) == 1 assert inputs[0]["landlord_property_ids"] == ["LP1", "LP2"] assert inputs[0]["recipient_email"] == RECIPIENT assert len(api.sent_bodies) == 1 message = json.loads(api.sent_bodies[0]) - assert message["task_id"] == str(task.id) + assert message["task_id"] == str(task_id) assert "subtask_id" in message def test_rejects_a_selection_over_the_cap(api: Api, monkeypatch: Any) -> None: # arrange — cap of 1, two selected properties. monkeypatch.setattr(documents_router, "MAX_PROPERTIES", 1) - task = api.seed_task() p1 = api.seed_property("LP1") p2 = api.seed_property("LP2") + task_id = api.seed_task( + {"portfolio_id": PORTFOLIO_ID, "property_ids": [p1, p2]} + ) # act - response = _trigger(api, task, [p1, p2]) + response = _trigger(api, task_id) # assert — refused; nothing created or sent. assert response.status_code == 400 - assert api.subtask_inputs(task) == [] + assert api.subtask_inputs(task_id) == [] assert api.sent_bodies == [] def test_rejects_a_task_that_already_has_a_subtask(api: Api) -> None: # arrange — a task already triggered once. - task = api.seed_task() p1 = api.seed_property("LP1") - assert _trigger(api, task, [p1]).status_code == 202 + task_id = api.seed_task({"portfolio_id": PORTFOLIO_ID, "property_ids": [p1]}) + assert _trigger(api, task_id).status_code == 202 # act - response = _trigger(api, task, [p1]) + response = _trigger(api, task_id) # assert — the double-submit is refused; still one sub_task, one message. assert response.status_code == 409 - assert len(api.subtask_inputs(task)) == 1 + assert len(api.subtask_inputs(task_id)) == 1 assert len(api.sent_bodies) == 1 + + +def test_rejects_a_task_with_no_selection_config(api: Api) -> None: + # arrange — a task whose inputs carry no portfolio_id/selection. + api.seed_property("LP1") + task_id = api.seed_task({}) + + # act + response = _trigger(api, task_id) + + # assert + assert response.status_code == 400 + assert api.sent_bodies == [] From 9ec7987e97c49fafc85a370e16c36cc9b70ee92e Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 11:08:46 +0000 Subject: [PATCH 44/55] =?UTF-8?q?PR=20review:=20best-effort=20read=20path,?= =?UTF-8?q?=20true=20per-member=20streaming,=20race-safe=20409,=20local=20?= =?UTF-8?q?email=20=F0=9F=9F=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses PR #1498 review: - A per-file read failure (missing/deleted object, or a bucket the role can't reach) is now a SkippedDocument(reason=unreadable), not a whole-run abort; an all-unreadable selection still fails (no empty package). Best-effort on the read path (ADR-0060). - Each ZIP member is streamed to a temp file (S3DocumentDownloader.download, boto download_file) and added from disk, so a single multi-GB member never hits the heap — the 'never held whole in memory' claim is now true. - The 409 double-submit guard is DB-arbitrated: the sub_task insert is conditional on the task having none, so a race creates only one. - Local env gets a placeholder recipient email so the route is exercisable. - PackageEntry carries landlord_property_id so a read failure can be reported. - TODO noting the UploadedFile.s3_upload_timestamp typing fix. Two new tests: per-file read failure skips-and-reports; address fallback flows to the folder name. Co-Authored-By: Claude Fable 5 --- .../bulk_document_download/handler.py | 4 +- backend/app/documents/download_tasks.py | 17 +++- backend/app/documents/router.py | 17 +++- domain/bulk_document_download/package_plan.py | 5 +- infrastructure/s3/s3_document_bytes_reader.py | 21 ----- infrastructure/s3/s3_document_downloader.py | 20 +++++ .../bulk_document_download_orchestrator.py | 88 +++++++++++++------ .../test_s3_document_bytes_reader.py | 29 ------ .../test_s3_document_downloader.py | 36 ++++++++ ...est_bulk_document_download_orchestrator.py | 70 ++++++++++++++- 10 files changed, 218 insertions(+), 89 deletions(-) delete mode 100644 infrastructure/s3/s3_document_bytes_reader.py create mode 100644 infrastructure/s3/s3_document_downloader.py delete mode 100644 tests/infrastructure/test_s3_document_bytes_reader.py create mode 100644 tests/infrastructure/test_s3_document_downloader.py diff --git a/applications/bulk_document_download/handler.py b/applications/bulk_document_download/handler.py index c5af7014c..a0d90798c 100644 --- a/applications/bulk_document_download/handler.py +++ b/applications/bulk_document_download/handler.py @@ -29,7 +29,7 @@ from infrastructure.email.ses_smtp_email_sender import SesSmtpEmailSender from infrastructure.postgres.config import PostgresConfig from infrastructure.postgres.engine import make_engine, make_session from infrastructure.s3.s3_client import S3Client -from infrastructure.s3.s3_document_bytes_reader import S3DocumentBytesReader +from infrastructure.s3.s3_document_downloader import S3DocumentDownloader from orchestration.bulk_document_download_orchestrator import ( BulkDocumentDownloadOrchestrator, ) @@ -72,7 +72,7 @@ def handler(body: dict[str, Any], context: Any) -> dict[str, Any]: orchestrator = BulkDocumentDownloadOrchestrator( uploaded_files=UploadedFilePostgresRepository(session), addresses=LandlordAddressResolver(session, landlord_property_ids), - documents=S3DocumentBytesReader(boto_s3), + documents=S3DocumentDownloader(boto_s3), packages=S3Client(boto_s3, os.environ["DOCUMENT_EXPORTS_BUCKET"]), email=_email_sender(), url_ttl_seconds=_URL_TTL_SECONDS, diff --git a/backend/app/documents/download_tasks.py b/backend/app/documents/download_tasks.py index 9c4940407..272da5e47 100644 --- a/backend/app/documents/download_tasks.py +++ b/backend/app/documents/download_tasks.py @@ -87,16 +87,24 @@ class DocumentDownloadTasks: def create_download_subtask( self, task_id: UUID, subtask_id: UUID, recipe: dict[str, Any] - ) -> None: + ) -> bool: """Pre-create the single ``waiting`` sub_task under the app-owned task, pinning the recipe (landlord_property_ids, recipient_email, package_name) onto its inputs — the worker's reproducible instructions - (ADR-0055/0060).""" + (ADR-0055/0060). + + The insert is **conditional on the task having no sub_task yet**, so the + DB — not a prior read — arbitrates a double-submit race: two concurrent + requests can both pass the ``already_distributed`` check, but only one + insert lands. Returns whether this call created the sub_task.""" now = datetime.now(timezone.utc) - self._session.connection().execute( + result = self._session.connection().execute( text( "INSERT INTO sub_task (id, task_id, status, inputs, updated_at)" - " VALUES (:id, :task_id, 'waiting', :inputs, :updated_at)" + " SELECT :id, :task_id, 'waiting', :inputs, :updated_at" + " WHERE NOT EXISTS (" + " SELECT 1 FROM sub_task WHERE task_id = :task_id" + " )" ), { "id": subtask_id, @@ -106,3 +114,4 @@ class DocumentDownloadTasks: }, ) self._session.commit() + return result.rowcount == 1 diff --git a/backend/app/documents/router.py b/backend/app/documents/router.py index 06bc24723..52d57d06a 100644 --- a/backend/app/documents/router.py +++ b/backend/app/documents/router.py @@ -52,6 +52,11 @@ def get_requesting_user_email(user: Any = Depends(validate_jwt_token)) -> str: email: Any = getattr(user, "email", None) if email is None and isinstance(user, dict): email = cast(dict[str, Any], user).get("email") + if not email and get_settings().ENVIRONMENT == "local": + # Local `get_user` returns a dummy dict with no email; a placeholder lets + # the route be exercised end-to-end on a dev machine (email is a no-op + # locally anyway). + email = "local-dev@example.com" if not email: raise HTTPException( status_code=400, @@ -133,7 +138,17 @@ async def bulk_download( "recipient_email": recipient_email, "package_name": f"portfolio-{portfolio_id}-{body.task_id}", } - tasks.create_download_subtask(body.task_id, subtask_id, recipe) + created = tasks.create_download_subtask(body.task_id, subtask_id, recipe) + if not created: + # Lost a double-submit race — another request already created the + # sub_task. The DB arbitrated; don't enqueue a second job. + raise HTTPException( + status_code=409, + detail=( + f"Task {body.task_id} already has a sub_task — a download has " + "already been started for it." + ), + ) send_message( json.dumps({"task_id": str(body.task_id), "subtask_id": str(subtask_id)}) diff --git a/domain/bulk_document_download/package_plan.py b/domain/bulk_document_download/package_plan.py index 535928596..ba12e3560 100644 --- a/domain/bulk_document_download/package_plan.py +++ b/domain/bulk_document_download/package_plan.py @@ -33,11 +33,13 @@ class ResolvedDocument: @dataclass(frozen=True) class PackageEntry: """One file to place in the Download Package: where it lands in the ZIP and - the S3 object to stream in from.""" + the S3 object to stream in from. Carries the property identity so a + read-time failure can be reported as a SkippedDocument (ADR-0060).""" zip_path: str s3_bucket: str s3_key: str + landlord_property_id: str @dataclass(frozen=True) @@ -95,6 +97,7 @@ def plan_download_package( f"{document.document_type}{_extension_of(document.s3_key)}", s3_bucket=document.s3_bucket, s3_key=document.s3_key, + landlord_property_id=document.landlord_property_id, ) for document in latest_by_group.values() ] diff --git a/infrastructure/s3/s3_document_bytes_reader.py b/infrastructure/s3/s3_document_bytes_reader.py deleted file mode 100644 index 2e0739249..000000000 --- a/infrastructure/s3/s3_document_bytes_reader.py +++ /dev/null @@ -1,21 +0,0 @@ -"""Reads document bytes from arbitrary S3 buckets (ADR-0060). - -Uploaded files live across many source buckets (each `uploaded_files` row -carries its own `s3_file_bucket`), so — unlike the bucket-bound `S3Client` — -this reader takes the bucket per call. Used by the Bulk Document Download -orchestrator to stream each chosen file into the ZIP. -""" - -from __future__ import annotations - -from typing import Any - - -class S3DocumentBytesReader: - def __init__(self, boto_s3_client: Any) -> None: - self._client = boto_s3_client - - def read(self, bucket: str, key: str) -> bytes: - response: dict[str, Any] = self._client.get_object(Bucket=bucket, Key=key) - body: bytes = response["Body"].read() - return body diff --git a/infrastructure/s3/s3_document_downloader.py b/infrastructure/s3/s3_document_downloader.py new file mode 100644 index 000000000..e1ee7bd12 --- /dev/null +++ b/infrastructure/s3/s3_document_downloader.py @@ -0,0 +1,20 @@ +"""Streams uploaded documents from arbitrary S3 buckets to local disk (ADR-0060). + +Uploaded files live across many source buckets (each `uploaded_files` row +carries its own `s3_file_bucket`), so — unlike the bucket-bound `S3Client` — +this takes the bucket per call. Downloading to a file (rather than reading the +object into memory) keeps a single multi-GB member off the heap when it is added +to the on-disk ZIP. +""" + +from __future__ import annotations + +from typing import Any + + +class S3DocumentDownloader: + def __init__(self, boto_s3_client: Any) -> None: + self._client = boto_s3_client + + def download(self, bucket: str, key: str, dest_path: str) -> None: + self._client.download_file(Bucket=bucket, Key=key, Filename=dest_path) diff --git a/orchestration/bulk_document_download_orchestrator.py b/orchestration/bulk_document_download_orchestrator.py index ba979a1c6..c3c0db2cd 100644 --- a/orchestration/bulk_document_download_orchestrator.py +++ b/orchestration/bulk_document_download_orchestrator.py @@ -38,8 +38,8 @@ class AddressResolver(Protocol): def address_for(self, landlord_property_id: str) -> str: ... -class DocumentBytesReader(Protocol): - def read(self, bucket: str, key: str) -> bytes: ... +class DocumentDownloader(Protocol): + def download(self, bucket: str, key: str, dest_path: str) -> None: ... @dataclass(frozen=True) @@ -58,7 +58,7 @@ class BulkDocumentDownloadOrchestrator: *, uploaded_files: UploadedFileReader, addresses: AddressResolver, - documents: DocumentBytesReader, + documents: DocumentDownloader, packages: S3Client, email: EmailSender, url_ttl_seconds: int, @@ -83,36 +83,44 @@ class BulkDocumentDownloadOrchestrator: resolved = [self._resolve(file) for file in files] plan = plan_download_package([r for r in resolved if r is not None]) + skipped: list[SkippedDocument] = list(plan.skipped) + if not plan.entries: - # Nothing to package (ADR-0060): a recorded failure, not an empty - # ZIP emailed to the user. The skip detail rides on the failure. - raise SubTaskFailure( - "no documents resolved for the selection", - details={ - "skipped": [ - { - "landlord_property_id": s.landlord_property_id, - "s3_key": s.s3_key, - "reason": s.reason, - } - for s in plan.skipped - ] - }, - ) + raise self._empty_failure(skipped) # Stream to a /tmp file and multipart-upload it (ADR-0060): a Download - # Package can be several GB, so it is never held whole in memory — each - # source file is read, written into the on-disk ZIP, and released; the - # finished archive is streamed up from disk. + # Package can be several GB, so it is never held whole in memory. Each + # source file is streamed to its own temp file, added to the on-disk ZIP, + # then deleted; the finished archive is streamed up from disk. Best-effort + # (ADR-0060): a file that can't be read — deleted, or in a bucket the role + # can't reach — becomes a SkippedDocument, it does not fail the package. key = f"{self._package_key_prefix}/{package_name}.zip" + included = 0 with tempfile.TemporaryDirectory() as tmp_dir: zip_path = os.path.join(tmp_dir, "package.zip") with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as archive: - for entry in plan.entries: - archive.writestr( - entry.zip_path, - self._documents.read(entry.s3_bucket, entry.s3_key), - ) + for index, entry in enumerate(plan.entries): + member_path = os.path.join(tmp_dir, f"member-{index}") + try: + self._documents.download( + entry.s3_bucket, entry.s3_key, member_path + ) + except Exception: + skipped.append( + SkippedDocument( + landlord_property_id=entry.landlord_property_id, + s3_key=entry.s3_key, + reason="unreadable", + ) + ) + continue + archive.write(member_path, arcname=entry.zip_path) + os.remove(member_path) + included += 1 + + if included == 0: + # Every file that resolved failed to read — no package to send. + raise self._empty_failure(skipped) self._packages.upload_file(zip_path, key) url = self._packages.generate_presigned_url(key, self._url_ttl_seconds) @@ -121,7 +129,7 @@ class BulkDocumentDownloadOrchestrator: subject="Your document download is ready", body=( f"Your document download is ready. It contains " - f"{len(plan.entries)} document(s).\n\n" + f"{included} document(s).\n\n" f"Download it here (link valid for " f"{self._url_ttl_seconds // 60} minutes):\n{url}" ), @@ -129,8 +137,26 @@ class BulkDocumentDownloadOrchestrator: return DownloadPackageResult( presigned_url=url, package_s3_key=key, - included=len(plan.entries), - skipped=plan.skipped, + included=included, + skipped=tuple(skipped), + ) + + def _empty_failure(self, skipped: list[SkippedDocument]) -> SubTaskFailure: + """A recorded failure when the selection yields no readable documents + (ADR-0060) — never an empty ZIP emailed to the user. The skip detail + rides on the failure.""" + return SubTaskFailure( + "no documents could be packaged for the selection", + details={ + "skipped": [ + { + "landlord_property_id": s.landlord_property_id, + "s3_key": s.s3_key, + "reason": s.reason, + } + for s in skipped + ] + }, ) def _resolve(self, file: UploadedFile) -> Optional[ResolvedDocument]: @@ -142,5 +168,9 @@ class BulkDocumentDownloadOrchestrator: document_type=file.file_type, s3_bucket=file.s3_file_bucket, s3_key=file.s3_file_key, + # TODO: the cast is only needed because UploadedFile types + # s3_upload_timestamp as `object` (though it is NOT NULL datetime in + # the DB). Type that column `datetime` on the table so callers don't + # each cast — see PR #1498 review. uploaded_at=cast(datetime, file.s3_upload_timestamp), ) diff --git a/tests/infrastructure/test_s3_document_bytes_reader.py b/tests/infrastructure/test_s3_document_bytes_reader.py deleted file mode 100644 index 8bbcea895..000000000 --- a/tests/infrastructure/test_s3_document_bytes_reader.py +++ /dev/null @@ -1,29 +0,0 @@ -"""The multi-bucket document reader (ADR-0060) — reads object bytes from -whichever source bucket an uploaded file lives in.""" - -from collections.abc import Iterator - -import pytest -from moto import mock_aws - -from infrastructure.s3.s3_document_bytes_reader import S3DocumentBytesReader -from tests.infrastructure import make_boto_client - - -@pytest.fixture -def reader() -> Iterator[S3DocumentBytesReader]: - with mock_aws(): - boto_client = make_boto_client("s3") - boto_client.create_bucket(Bucket="src-a") - boto_client.create_bucket(Bucket="src-b") - boto_client.put_object(Bucket="src-a", Key="surveys/one.pdf", Body=b"AAA") - boto_client.put_object(Bucket="src-b", Key="photos/two.zip", Body=b"BBB") - yield S3DocumentBytesReader(boto_client) - - -def test_reads_bytes_from_the_named_source_bucket( - reader: S3DocumentBytesReader, -) -> None: - # act / assert — each object is read from its own bucket. - assert reader.read("src-a", "surveys/one.pdf") == b"AAA" - assert reader.read("src-b", "photos/two.zip") == b"BBB" diff --git a/tests/infrastructure/test_s3_document_downloader.py b/tests/infrastructure/test_s3_document_downloader.py new file mode 100644 index 000000000..58753d2b1 --- /dev/null +++ b/tests/infrastructure/test_s3_document_downloader.py @@ -0,0 +1,36 @@ +"""The multi-bucket document downloader (ADR-0060) — streams object bytes from +whichever source bucket an uploaded file lives in, to local disk.""" + +from collections.abc import Iterator +from pathlib import Path + +import pytest +from moto import mock_aws + +from infrastructure.s3.s3_document_downloader import S3DocumentDownloader +from tests.infrastructure import make_boto_client + + +@pytest.fixture +def downloader() -> Iterator[S3DocumentDownloader]: + with mock_aws(): + boto_client = make_boto_client("s3") + boto_client.create_bucket(Bucket="src-a") + boto_client.create_bucket(Bucket="src-b") + boto_client.put_object(Bucket="src-a", Key="surveys/one.pdf", Body=b"AAA") + boto_client.put_object(Bucket="src-b", Key="photos/two.zip", Body=b"BBB") + yield S3DocumentDownloader(boto_client) + + +def test_downloads_bytes_from_the_named_source_bucket( + downloader: S3DocumentDownloader, tmp_path: Path +) -> None: + # act — each object streams from its own bucket to a local file. + a = tmp_path / "a" + b = tmp_path / "b" + downloader.download("src-a", "surveys/one.pdf", str(a)) + downloader.download("src-b", "photos/two.zip", str(b)) + + # assert + assert a.read_bytes() == b"AAA" + assert b.read_bytes() == b"BBB" diff --git a/tests/orchestration/test_bulk_document_download_orchestrator.py b/tests/orchestration/test_bulk_document_download_orchestrator.py index 40c483d80..d9b3a0d23 100644 --- a/tests/orchestration/test_bulk_document_download_orchestrator.py +++ b/tests/orchestration/test_bulk_document_download_orchestrator.py @@ -45,8 +45,11 @@ class _FakeDocuments: def __init__(self, blobs: dict[tuple[str, str], bytes]) -> None: self._blobs = blobs - def read(self, bucket: str, key: str) -> bytes: - return self._blobs[(bucket, key)] + def download(self, bucket: str, key: str, dest_path: str) -> None: + # KeyError for an object we don't hold — stands in for a missing/ + # unreadable source object. + with open(dest_path, "wb") as handle: + handle.write(self._blobs[(bucket, key)]) class _RecordingEmail: @@ -176,3 +179,66 @@ def test_packages_the_good_documents_and_reports_the_skipped_ones( archive = packages.get_object(result.package_s3_key) with zipfile.ZipFile(io.BytesIO(archive)) as zf: assert zf.namelist() == ["12 Oak Street (LP1)/site_note.pdf"] + + +def test_a_file_that_cannot_be_read_is_skipped_not_fatal(packages: S3Client) -> None: + # Arrange — two documents for one property; the photo pack's object is + # missing (download raises), the site note reads fine. Best-effort (ADR-0060): + # the run must still deliver the readable file and report the other. + good = _uploaded_file("LP1", "src-bucket", "surveys/a.pdf") + missing = _uploaded_file( + "LP1", "src-bucket", "surveys/gone.pdf", file_type=FileTypeEnum.PHOTO_PACK + ) + email = _RecordingEmail() + orchestrator = BulkDocumentDownloadOrchestrator( + uploaded_files=_FakeUploadedFiles([good, missing]), + addresses=_FakeAddresses({"LP1": "12 Oak Street"}), + documents=_FakeDocuments({("src-bucket", "surveys/a.pdf"): b"PDF-BYTES"}), + packages=packages, + email=email, + url_ttl_seconds=3600, + ) + + # Act + result = orchestrator.run( + landlord_property_ids=["LP1"], + recipient_email="user@example.com", + package_name="partial", + ) + + # Assert — the readable file is delivered; the missing one is reported. + assert result.included == 1 + assert [(s.s3_key, s.reason) for s in result.skipped] == [ + ("surveys/gone.pdf", "unreadable") + ] + assert len(email.sent) == 1 + archive = packages.get_object(result.package_s3_key) + with zipfile.ZipFile(io.BytesIO(archive)) as zf: + assert zf.namelist() == ["12 Oak Street (LP1)/site_note.pdf"] + + +def test_the_address_fallback_string_becomes_the_folder_name( + packages: S3Client, +) -> None: + # Arrange — a property whose address resolved to the fallback string. + doc = _uploaded_file("LP9", "src-bucket", "surveys/a.pdf") + orchestrator = BulkDocumentDownloadOrchestrator( + uploaded_files=_FakeUploadedFiles([doc]), + addresses=_FakeAddresses({"LP9": "address unavailable"}), + documents=_FakeDocuments({("src-bucket", "surveys/a.pdf"): b"PDF-BYTES"}), + packages=packages, + email=_RecordingEmail(), + url_ttl_seconds=3600, + ) + + # Act + result = orchestrator.run( + landlord_property_ids=["LP9"], + recipient_email="user@example.com", + package_name="fallback", + ) + + # Assert — the fallback flows through to a usable folder name. + archive = packages.get_object(result.package_s3_key) + with zipfile.ZipFile(io.BytesIO(archive)) as zf: + assert zf.namelist() == ["address unavailable (LP9)/site_note.pdf"] From 90fe2914e034211a9797c97a1165427622443250 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 11:42:17 +0000 Subject: [PATCH 45/55] Select documents by HubSpot project_code(s), unioned with hand-picked property_ids MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 'all properties' selection resolved against property.portfolio_id, but a portfolio spans multiple HubSpot projects — so 'all' pulled the whole portfolio, not the project the user was looking at. The project↔property grain lives on hubspot_deal_data, not property. task.inputs is now {project_codes?: str[], property_ids?: int[], portfolio_id?}: the route resolves the distinct landlord_property_id set as the union of every property in the named project_codes (from hubspot_deal_data) and the hand-picked property_ids; portfolio_id is optional and only names the package. Drops the portfolio-scoped select_all. Cap now applies to the resolved set size. ADR-0060, CONTEXT.md and the request schema updated to match. New router tests cover project-code resolution, the union+dedup, null landlord_property_id drop, and the empty-project rejection. Co-Authored-By: Claude Opus 4.8 (1M context) --- CONTEXT.md | 2 +- backend/app/documents/download_tasks.py | 53 ++++++------ backend/app/documents/router.py | 64 ++++++++------- backend/app/documents/schemas.py | 10 ++- ...60-bulk-document-download-package-model.md | 22 +++-- .../documents/test_bulk_download_router.py | 81 +++++++++++++++++++ 6 files changed, 167 insertions(+), 65 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 33d79f62e..c736d712d 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -108,7 +108,7 @@ The single ZIP archive a **Bulk Document Download** produces: one folder per pro _Avoid_: export (that is the plan/scenario XLSX export), bundle, archive (ambiguous), zip (the format, not the concept) **Bulk Document Download**: -The feature/job that assembles a **Download Package** for a chosen set of properties (a hand-picked list or a whole portfolio) and emails the requester a link. Runs on the app-owned-task + attach-mode lane (ADR-0055): the FastAPI route pins the resolved `property_id` set and the recipient email into `tasks.inputs`, and the `applications/bulk_document_download` Lambda builds the package, writes the URL to `sub_task.outputs`, and emails it (ADR-0059/0060). Files are matched by `landlord_property_id` (the missing `property_id` on `uploaded_files` is a known future gap). Selection is capped by property count at the trigger. +The feature/job that assembles a **Download Package** for a chosen set of properties (the union of one or more whole HubSpot projects, selected by `project_code`, and a hand-picked `property_id` list) and emails the requester a link. Runs on the app-owned-task + attach-mode lane (ADR-0055): the FastAPI route resolves the selection to a distinct `landlord_property_id` set — project codes are expanded via `hubspot_deal_data` (which holds the project↔property grain) — pins that set and the recipient email onto the `sub_task`, and the `applications/bulk_document_download` Lambda builds the package, writes the URL to `sub_task.outputs`, and emails it (ADR-0059/0060). Files are matched by `landlord_property_id` (the missing `property_id` on `uploaded_files` is a known future gap). Selection is capped by property count at the trigger. _Avoid_: document export, bulk export, file dump ### Source data diff --git a/backend/app/documents/download_tasks.py b/backend/app/documents/download_tasks.py index 272da5e47..7109224a6 100644 --- a/backend/app/documents/download_tasks.py +++ b/backend/app/documents/download_tasks.py @@ -8,7 +8,7 @@ Modelling Run distributor works around — see ``modelling/run_tasks.py``). import json from datetime import datetime, timezone -from typing import Any, Optional, cast +from typing import Any, cast from uuid import UUID from sqlalchemy import text @@ -51,39 +51,44 @@ class DocumentDownloadTasks: def resolve_selection( self, - portfolio_id: int, - property_ids: Optional[list[int]], - select_all: bool, - ) -> tuple[int, list[str]]: - """Resolve the selection to (selected property count, the distinct - ``landlord_property_id`` values that carry a downloadable identifier). + project_codes: list[str], + property_ids: list[int], + ) -> list[str]: + """Resolve the selection to the distinct ``landlord_property_id`` set the + Download Package is built from (ADR-0060). Two independent sources, + unioned: - The count is the cap basis (property count, ADR-0060); files are matched - by ``landlord_property_id`` (the missing ``property_id`` on - ``uploaded_files`` is a known future gap), so a selected property with a - null ``landlord_property_id`` contributes no files. + - **project codes** — every property in the named HubSpot projects, read + from ``hubspot_deal_data`` (the project↔property grain lives there, not + on ``property``). + - **hand-picked property ids** — the selected ``property`` rows. + + Files are matched by ``landlord_property_id`` (the missing ``property_id`` + on ``uploaded_files`` is a known future gap), so a source row with a null + ``landlord_property_id`` contributes nothing and is dropped here. The + returned set size is the cap basis (property count, ADR-0060). """ - if select_all: + landlord_property_ids: set[str] = set() + if project_codes: rows = self._session.connection().execute( text( - "SELECT id, landlord_property_id FROM property" - " WHERE portfolio_id = :portfolio_id" + "SELECT DISTINCT landlord_property_id FROM hubspot_deal_data" + " WHERE project_code = ANY(:codes)" + " AND landlord_property_id IS NOT NULL" ), - {"portfolio_id": portfolio_id}, + {"codes": project_codes}, ) - else: + landlord_property_ids.update(row[0] for row in rows.all()) + if property_ids: rows = self._session.connection().execute( text( - "SELECT id, landlord_property_id FROM property" - " WHERE portfolio_id = :portfolio_id AND id = ANY(:ids)" + "SELECT landlord_property_id FROM property" + " WHERE id = ANY(:ids) AND landlord_property_id IS NOT NULL" ), - {"portfolio_id": portfolio_id, "ids": property_ids or []}, + {"ids": property_ids}, ) - resolved = rows.all() - landlord_property_ids = sorted( - {row[1] for row in resolved if row[1] is not None} - ) - return len(resolved), landlord_property_ids + landlord_property_ids.update(row[0] for row in rows.all()) + return sorted(landlord_property_ids) def create_download_subtask( self, task_id: UUID, subtask_id: UUID, recipe: dict[str, Any] diff --git a/backend/app/documents/router.py b/backend/app/documents/router.py index 52d57d06a..50b7a957b 100644 --- a/backend/app/documents/router.py +++ b/backend/app/documents/router.py @@ -90,53 +90,59 @@ async def bulk_download( ) config = tasks.read_selection_config(body.task_id) - portfolio_id = config.get("portfolio_id") - property_ids = config.get("property_ids") - select_all = bool(config.get("select_all", False)) - if not isinstance(portfolio_id, int): + portfolio_id: Any = config.get("portfolio_id") + raw_project_codes: Any = config.get("project_codes") or [] + raw_property_ids: Any = config.get("property_ids") or [] + if not isinstance(raw_project_codes, list) or not all( + isinstance(code, str) for code in cast(list[Any], raw_project_codes) + ): raise HTTPException( status_code=400, - detail="task.inputs is missing an integer portfolio_id.", + detail="task.inputs project_codes must be a list of strings.", ) - if not select_all and not property_ids: + if not isinstance(raw_property_ids, list) or not all( + isinstance(pid, int) for pid in cast(list[Any], raw_property_ids) + ): raise HTTPException( status_code=400, - detail="task.inputs must provide property_ids or set select_all.", + detail="task.inputs property_ids must be a list of integers.", + ) + project_codes: list[str] = cast(list[str], raw_project_codes) + property_ids: list[int] = cast(list[int], raw_property_ids) + if not project_codes and not property_ids: + raise HTTPException( + status_code=400, + detail="task.inputs must provide project_codes or property_ids.", ) - property_count, landlord_property_ids = tasks.resolve_selection( - portfolio_id, property_ids, select_all - ) - if property_count == 0: - raise HTTPException( - status_code=400, - detail=( - f"The selection resolves to no properties in portfolio " - f"{portfolio_id}." - ), - ) - if property_count > MAX_PROPERTIES: - raise HTTPException( - status_code=400, - detail=( - f"The selection has {property_count} properties, over the " - f"{MAX_PROPERTIES} limit — narrow your selection." - ), - ) + landlord_property_ids = tasks.resolve_selection(project_codes, property_ids) if not landlord_property_ids: raise HTTPException( status_code=400, detail=( - "None of the selected properties have a landlord_property_id, " - "so no documents can be matched." + "The selection resolves to no downloadable properties — none of " + "the selected projects or properties has a landlord_property_id." + ), + ) + if len(landlord_property_ids) > MAX_PROPERTIES: + raise HTTPException( + status_code=400, + detail=( + f"The selection has {len(landlord_property_ids)} properties, over " + f"the {MAX_PROPERTIES} limit — narrow your selection." ), ) subtask_id = uuid4() + package_name = ( + f"portfolio-{portfolio_id}-{body.task_id}" + if isinstance(portfolio_id, int) + else f"documents-{body.task_id}" + ) recipe = { "landlord_property_ids": landlord_property_ids, "recipient_email": recipient_email, - "package_name": f"portfolio-{portfolio_id}-{body.task_id}", + "package_name": package_name, } created = tasks.create_download_subtask(body.task_id, subtask_id, recipe) if not created: diff --git a/backend/app/documents/schemas.py b/backend/app/documents/schemas.py index 0d60006d8..c383ddfaf 100644 --- a/backend/app/documents/schemas.py +++ b/backend/app/documents/schemas.py @@ -8,9 +8,13 @@ class BulkDownloadRequest(BaseModel): The front end creates the app-owned Task first and writes the **selection config** into ``task.inputs`` (a JSON object of - ``{portfolio_id, property_ids?, select_all?}`` — so a large hand-picked set - never travels in an HTTP body), then passes only the ``task_id`` here. The - backend reads the selection from ``task.inputs``. + ``{project_codes?: str[], property_ids?: int[], portfolio_id?: int}`` — so a + large selection never travels in an HTTP body), then passes only the + ``task_id`` here. The backend reads the selection from ``task.inputs`` and + resolves it to the distinct ``landlord_property_id`` set: the union of every + property in the named HubSpot ``project_codes`` (from ``hubspot_deal_data``) + and the hand-picked ``property_ids``. At least one of the two must be given; + ``portfolio_id`` is optional and used only to name the package. """ task_id: UUID diff --git a/docs/adr/0060-bulk-document-download-package-model.md b/docs/adr/0060-bulk-document-download-package-model.md index 103a01b47..10959d19a 100644 --- a/docs/adr/0060-bulk-document-download-package-model.md +++ b/docs/adr/0060-bulk-document-download-package-model.md @@ -7,8 +7,9 @@ status: accepted (builds on ADR-0055, ADR-0059) Users need to pull the documents held in `uploaded_files` for many properties at once — per property, the latest file of each **Document Type** — as a single archive, without clicking through the UI file by file. The request is initiated -in the front end, can span a hand-picked set of properties or a whole -portfolio, and finishes with a link emailed to the requester (ADR-0059). +in the front end, can span a hand-picked set of properties or one or more whole +HubSpot projects (by project code), and finishes with a link emailed to the +requester (ADR-0059). `uploaded_files` links to a property by `uprn` **or** `landlord_property_id` (both nullable) and has **no `property_id`** column; `file_type` (the Document @@ -24,12 +25,17 @@ lane (ADR-0055).** - **Trigger & lifecycle (ADR-0055).** The **front end** creates the app-owned `tasks` row and writes the **selection config** — - `{portfolio_id, property_ids?, select_all?}` — into `tasks.inputs` (TEXT/JSON, - FE-owned), then calls the FastAPI route with **only the `task_id`**. A large - hand-picked selection therefore never travels in an HTTP body. The route - **reads the selection from `tasks.inputs`**, resolves it to the distinct - `landlord_property_id` set, caps it, resolves the recipient email from the - authenticated user (ADR-0059), **pins the resolved recipe** + `{project_codes?: str[], property_ids?: int[], portfolio_id?: int}` — into + `tasks.inputs` (TEXT/JSON, FE-owned), then calls the FastAPI route with **only + the `task_id`**. A large selection therefore never travels in an HTTP body. + The route **reads the selection from `tasks.inputs`** and resolves it to the + distinct `landlord_property_id` set — the **union** of every property in the + named HubSpot `project_codes` (read from `hubspot_deal_data`, where the + project↔property grain lives — `property` carries only `portfolio_id`) and the + hand-picked `property_ids`. At least one of the two must be provided; + `portfolio_id` is optional and used only to name the package. The route then + caps the set, resolves the recipient email from the authenticated user + (ADR-0059), **pins the resolved recipe** (`landlord_property_ids`, `recipient_email`, `package_name`) onto **one pre-created `sub_task`'s `inputs`**, and drops one SQS message (`task_id`, `sub_task_id`). The new `applications/bulk_document_download` Lambda runs in diff --git a/tests/backend/app/documents/test_bulk_download_router.py b/tests/backend/app/documents/test_bulk_download_router.py index ae5f38725..c21da8ca1 100644 --- a/tests/backend/app/documents/test_bulk_download_router.py +++ b/tests/backend/app/documents/test_bulk_download_router.py @@ -16,6 +16,7 @@ from fastapi.testclient import TestClient from sqlalchemy import Engine from sqlmodel import Session +from backend.app.db.models.hubspot_deal_data import HubspotDealData from backend.app.dependencies import validate_token from backend.app.documents import router as documents_router from backend.app.documents.router import ( @@ -61,6 +62,19 @@ class Api: assert row.id is not None return row.id + def seed_deal(self, project_code: str, landlord_property_id: str) -> None: + """A hubspot_deal_data row carrying the project↔property link the + project-code selection resolves against (ADR-0060).""" + with Session(self.engine) as session: + session.add( + HubspotDealData( + deal_id=f"deal-{project_code}-{landlord_property_id}", + project_code=project_code, + landlord_property_id=landlord_property_id, + ) + ) + session.commit() + def subtask_inputs(self, task_id: UUID) -> list[dict[str, Any]]: with Session(self.engine) as session: subtasks = SubTaskPostgresRepository(session).list_by_task(task_id) @@ -113,6 +127,73 @@ def test_pins_the_recipe_and_enqueues_one_message(api: Api) -> None: assert "subtask_id" in message +def test_resolves_all_properties_for_the_given_project_codes(api: Api) -> None: + # arrange — two projects the user selects, plus one they don't; the + # project↔property grain lives on hubspot_deal_data, not `property`. + api.seed_deal("PROJ-A", "LP1") + api.seed_deal("PROJ-A", "LP2") + api.seed_deal("PROJ-B", "LP3") + api.seed_deal("PROJ-C", "LP9") # a different project — must not appear + task_id = api.seed_task({"project_codes": ["PROJ-A", "PROJ-B"]}) + + # act + response = _trigger(api, task_id) + + # assert — the union of the selected projects' landlord_property_ids is pinned. + assert response.status_code == 202 + inputs = api.subtask_inputs(task_id) + assert inputs[0]["landlord_property_ids"] == ["LP1", "LP2", "LP3"] + + +def test_unions_project_codes_with_hand_picked_property_ids(api: Api) -> None: + # arrange — one project plus a hand-picked property from outside it. + api.seed_deal("PROJ-A", "LP1") + api.seed_deal("PROJ-A", "LP2") + hand_picked = api.seed_property("LP5") + task_id = api.seed_task( + {"project_codes": ["PROJ-A"], "property_ids": [hand_picked]} + ) + + # act + response = _trigger(api, task_id) + + # assert — both sources contribute; the set is de-duplicated and sorted. + assert response.status_code == 202 + inputs = api.subtask_inputs(task_id) + assert inputs[0]["landlord_property_ids"] == ["LP1", "LP2", "LP5"] + + +def test_deal_rows_with_no_landlord_property_id_are_dropped(api: Api) -> None: + # arrange — a project whose deal rows carry no landlord_property_id can match + # no files (files are keyed on landlord_property_id), so the selection is empty. + api.seed_deal("PROJ-A", "LP1") + with Session(api.engine) as session: + session.add(HubspotDealData(deal_id="deal-null", project_code="PROJ-A")) + session.commit() + task_id = api.seed_task({"project_codes": ["PROJ-A"]}) + + # act + response = _trigger(api, task_id) + + # assert — only the row that carries an id contributes. + assert response.status_code == 202 + inputs = api.subtask_inputs(task_id) + assert inputs[0]["landlord_property_ids"] == ["LP1"] + + +def test_rejects_a_project_code_that_resolves_to_no_properties(api: Api) -> None: + # arrange — a project with no matching deal rows at all. + task_id = api.seed_task({"project_codes": ["PROJ-EMPTY"]}) + + # act + response = _trigger(api, task_id) + + # assert — refused; nothing created or sent. + assert response.status_code == 400 + assert api.subtask_inputs(task_id) == [] + assert api.sent_bodies == [] + + def test_rejects_a_selection_over_the_cap(api: Api, monkeypatch: Any) -> None: # arrange — cap of 1, two selected properties. monkeypatch.setattr(documents_router, "MAX_PROPERTIES", 1) From 430ac9eaa402d83ff85a5f7c363ad8b271f21776 Mon Sep 17 00:00:00 2001 From: Jun-te Kim Date: Wed, 8 Jul 2026 11:53:51 +0000 Subject: [PATCH 46/55] Derive multiple_glazed_proportion for 16.x certs with explicit glazing wording MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Property 753950 (uprn 100021969385, cert 0141-2860-6891-9124-5625, SAP-Schema-16.2) hard-failed modelling_e2e: RdSapSchema17_1 requires multiple_glazed_proportion as a non-optional int, and this 16.x cert omits the field entirely, lodging only the multiple_glazing_type="ND" sentinel. The cert's own window.description ("Fully double glazed") states the glazing extent unambiguously, so _normalize_sap_schema_16_x now derives 0/100 from "single"/"double" wording when the field is absent, mirroring the existing single-glazed multiple_glazing_type cascade. This is worklist P4 (.claude/skills/expand-sap-accuracy-corpus/worklist.md) — a flat default was tried previously and reverted because making such certs mappable at all pulls them into the EPC-prediction donor pool and tips near-tie similarity matches; deriving from explicit text (rather than a blind default) was the suggested unblock. Re-measured the component-accuracy gate as the worklist asked: it drops (has_hot_water_cylinder 0.8687->0.8586, cylinder_insulation_type 0.3333->0.1667, door_count residual 0.3131->0.3333) via the same donor-pool-composition mechanism as the prior #1245/ADR-0037 re-baselines, not a prediction-logic loosening. Re-baselined the floors with that rationale recorded inline. --- .../uprn_100021969385/elmhurst_inputs.md | 164 ++++++++++++++++++ .../uprn_100021969385/epc.json | 1 + datatypes/epc/domain/mapper.py | 16 ++ .../epc/domain/tests/test_from_sap_schema.py | 19 ++ .../test_component_accuracy_gate.py | 24 ++- 5 files changed, 221 insertions(+), 3 deletions(-) create mode 100644 backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-16.2/uprn_100021969385/elmhurst_inputs.md create mode 100644 backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-16.2/uprn_100021969385/epc.json diff --git a/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-16.2/uprn_100021969385/elmhurst_inputs.md b/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-16.2/uprn_100021969385/elmhurst_inputs.md new file mode 100644 index 000000000..2c98526ef --- /dev/null +++ b/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-16.2/uprn_100021969385/elmhurst_inputs.md @@ -0,0 +1,164 @@ +# Elmhurst RdSAP inputs — UPRN 100021969385 (cert 0141-2860-6891-9124-5625, SAP-Schema-16.2) + +**Lodged SAP:** 45 **Our engine:** 42 ← compare Elmhurst against this + +**Context:** this is the property-753950 / P4 corpus case (`.claude/skills/expand-sap-accuracy-corpus/worklist.md`). +The raw gov-API cert omits `multiple_glazed_proportion` entirely (only +`multiple_glazing_type: "ND"` is lodged), which previously hard-failed +`RdSapSchema17_1` parsing for every 16.x cert like this one. The mapper fix in +this PR derives it from the explicit `windows[].description` ("Fully double +glazed" → 100) — SAP-neutral here since the cert *also* lodges real +`sap_windows[]` geometry the engine uses directly (see Openings below); the +derived proportion is only a fallback signal, not consumed for this cert's +window synthesis. + +**Known divergences:** none flagged — this is a first pass to validate the +newly-unblocked mapping, not a chase of a specific over/under-rating bug. + +## Property Description + +| Elmhurst field | Value | Notes (SAP code / EES path) | +|---|---|---| +| Property Type | House | `property_type` 0 | +| Built Form | Semi-detached | `built_form` 2 | +| Construction Age Band | C (1930–1949) | `sap_building_parts[0].construction_age_band` = "C" | +| Storeys | 2 | two `sap_floor_dimensions` entries (floor 0, floor 1) | +| Habitable Rooms | 6 | `habitable_room_count` | +| Heated Rooms | 6 | `heated_room_count` | +| Extensions | none | `extensions_count` = 0 | +| Rooms in Roof | none (see Roofs) | no dedicated room-in-roof building part, but roof line 2 below is "Roof room(s), insulated" — flag for manual check in Elmhurst's roof tab | + +## Dimensions + +| Floor | Floor Area | Room Height | Heat Loss Perimeter | Party Wall Length | +|---|---|---|---|---| +| Ground (0) | 48.31 m² | 2.67 m | 21.24 m | 0.0 m | +| 1st | 48.31 m² | 2.81 m | 21.24 m | 0.0 m | + +Total floor area (cert-level, use if Elmhurst wants a single figure): **128 m²** +(`total_floor_area`) — note this exceeds 2 × 48.31 (96.6); the extra area is +likely a 3rd storey/loft not broken into a `sap_floor_dimensions` row (16.2 is +reduced-field). Flag if Elmhurst's summed dimensions don't reconcile to 128. + +## Conservatory + +`conservatory_type` = 2 (present, unheated) / `has_heated_separate_conservatory` = false → +**Conservatory: present, NOT separately heated.** + +## Walls (Main + Party) + +| Elmhurst field | Value | Notes | +|---|---|---| +| Wall Type | Solid brick | `wall_construction` 3 | +| Insulation | As Built (no insulation) | `wall_insulation_type` 4 = "as built" sentinel; description lodges "Solid brick, as built, no insulation (assumed)" | +| Party Wall | none / blank | `party_wall_length_m` = 0.0 on both floors — no party wall to enter | + +## Roofs + +Two roof elements lodged: + +| Elmhurst field | Value | Notes | +|---|---|---| +| Roof 1 — Type | Pitched, access to loft | `roof_construction` 5; insulation thickness "NI" → **As built / no insulation** | +| Roof 2 — Type | Roof room(s), insulated | description "Roof room(s), insulated" — a room-in-roof-style pitched section; energy_efficiency_rating 4 (good) — enter as a 2nd roof element if Elmhurst's tool supports it, else the dominant (larger-area) one | + +⚠️ Two roof descriptions with opposite insulation states — check the Elmhurst UI +lets you enter two roof rows; if only one, prefer the uninsulated "Pitched, no +insulation" one for the main roof and note the "Roof room(s), insulated" +element separately (may need Room-in-Roof tab even though no explicit +room-in-roof building part is lodged). + +## Floors + +| Floor | Type | Insulation | +|---|---|---| +| Ground (0) | `floor_construction` 2 | `floor_insulation` 1 | +| 1st | not lodged (internal floor between storeys) | — | + +Cert-level element description: "Suspended, no insulation (assumed)" — enter +ground floor as **Suspended, As built (no insulation)**. + +## Openings — Windows + +⚠️ **This cert is NOT a reduced-field synthesis case** — it lodges real +`sap_windows[]` geometry directly (unusual for a 16.2 reduced cert): + +| # | Orientation | Area | Window Type | Glazing Type (engine) | +|---|---|---|---|---| +| 1 | 5 (S) | 18.66 m² | 1 | 1 (single) | +| 2 | 1 (N) | 1.77 m² | 2 | 2 (double) | + +`multiple_glazing_type` = "ND" (not defined) at dwelling level, but +`windows[].description` = "Fully double glazed" — the mapper fix derives +`multiple_glazed_proportion = 100` from that text (this PR), though it is not +the value driving the two `sap_windows` rows above (those come straight from +the cert's own lodged per-window array). Enter both window rows in Elmhurst +as-is: **Glazing Type: Double** for both (the description is unambiguous, +even though window 1's engine-synthesised code reads as the single slot — +flag this specific discrepancy for the calculator team if Elmhurst's score +comes back materially different, since it may indicate `window_type`≠1 was +misread as single glazing in `_api_cascade_glazing_type` for this shape). + +## Openings — Doors + +`door_count` = 2, `insulated_door_count` = 0, `percent_draughtproofed` = 100 → +**2 doors, none insulated, both draught-proofed.** + +## Ventilation & Lighting + +| Elmhurst field | Value | +|---|---| +| Open Chimneys | 2 (`open_fireplaces_count`) | +| Extract Fans / Passive Vents / Flueless Gas Fires | 0 (not lodged) | +| Fixed Space Cooling | No (`has_fixed_air_conditioning` false) | +| Mechanical Ventilation | Natural (unchecked) — `mechanical_ventilation` 0 | +| Air Pressure Test | Not available (RdSAP, uses % draughtproofing) | +| Total Bulbs | 14 (`fixed_lighting_outlets_count`) | +| Low Energy Bulbs | 5 (`low_energy_fixed_lighting_outlets_count`) | + +## Space Heating + +**Main Heating 1** (`sap_heating.main_heating_details[0]`): + +| Elmhurst field | Value | Notes | +|---|---|---| +| Fuel | Mains gas | `main_fuel_type` 26 | +| System | Boiler and radiators | description "Boiler and radiators, mains gas" | +| Heat Emitter | Radiators | `heat_emitter_type` 1 | +| Controls | SAP main_heating_control code **2102** | EES: look up 2102 in the boiler-controls table | +| PCDB Index | 8108 | `main_heating_index_number` — enter in Elmhurst's boiler database lookup for the exact efficiency | +| Percentage of Heat | 100% | `main_heating_fraction` 1 | +| Flue | Fanned, Type 2 | `fan_flue_present` true, `boiler_flue_type` 2 | + +**Secondary Heating**: "Room heaters, smokeless fuel" — enter a secondary +heater, fuel = smokeless solid fuel, type = room heater (not open fire). + +## Water Heating + +`has_hot_water_cylinder` = **false** → **no cylinder** — water heating is from +the main system (boiler), not a separate immersion/cylinder. Leave the +cylinder tab empty/none. + +## New Technologies + +No PV, wind, hydro, solar water heating (`solar_water_heating` = "N"), WWHRS, +or FGHRS lodged — leave all New Technologies pages empty/unchecked. + +## Fields to clear in Elmhurst (do NOT map) + +| Elmhurst field | Set to | Why absent | +|---|---|---| +| Property Description · 1st–4th Extension age band | (blank) | `extensions_count` = 0 | +| Property Description · Room-in-Roof age band | (blank) | no dedicated room-in-roof building part (see Roofs caveat above) | +| Dimensions · 1st–4th Ext. sub-tabs | 0 / blank | single building part, no extensions | +| Walls · Party wall | none/blank | `party_wall_length_m` = 0 on both floors | +| Water Heating · cylinder | none | `has_hot_water_cylinder` = false | +| Conservatory · Heated | unchecked | `has_heated_separate_conservatory` = false | +| PV / Wind / Hydro / WWHRS / FGHRS / Solar Water Heating | none | not lodged | + +## Operator: next steps + +Key this into Elmhurst, report the SAP score (and heating cost £ if shown). +If it diverges materially from our engine's **42** (lodged **45**), flag it — +especially check the two-roof-element and the window-1-single-glazing-code +notes above, they're the most likely sources of any gap. diff --git a/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-16.2/uprn_100021969385/epc.json b/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-16.2/uprn_100021969385/epc.json new file mode 100644 index 000000000..9de418358 --- /dev/null +++ b/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-16.2/uprn_100021969385/epc.json @@ -0,0 +1 @@ +{"data":{"uprn":100021969385,"roofs":[{"description":"Pitched, no insulation (assumed)","energy_efficiency_rating":1,"environmental_efficiency_rating":1},{"description":"Roof room(s), insulated","energy_efficiency_rating":4,"environmental_efficiency_rating":4}],"walls":[{"description":"Solid brick, as built, no insulation (assumed)","energy_efficiency_rating":1,"environmental_efficiency_rating":1}],"floors":[{"description":"Suspended, no insulation (assumed)","energy_efficiency_rating":0,"environmental_efficiency_rating":0}],"status":"entered","tenure":1,"windows":[{"description":"Fully double glazed","energy_efficiency_rating":3,"environmental_efficiency_rating":3}],"lighting":{"description":"Low energy lighting in 36% of fixed outlets","energy_efficiency_rating":3,"environmental_efficiency_rating":3},"postcode":"SE12 0HP","hot_water":{"description":"From main system","energy_efficiency_rating":4,"environmental_efficiency_rating":4},"post_town":"LONDON","built_form":2,"created_at":"2014-01-20 23:19:02.000000","door_count":2,"glazed_area":4,"region_code":17,"report_type":2,"sap_heating":{"wwhrs":{"rooms_with_bath_and_or_shower":2,"rooms_with_mixer_shower_no_bath":0,"rooms_with_bath_and_mixer_shower":1},"cylinder_size":1,"water_heating_code":901,"water_heating_fuel":26,"secondary_fuel_type":15,"main_heating_details":[{"has_fghrs":"N","main_fuel_type":26,"boiler_flue_type":2,"fan_flue_present":"Y","heat_emitter_type":1,"boiler_index_number":8108,"main_heating_number":1,"main_heating_control":2102,"main_heating_category":2,"main_heating_fraction":1,"main_heating_data_source":1}],"secondary_heating_type":631,"has_fixed_air_conditioning":"false"},"sap_version":9.91,"sap_windows":[{"orientation":5,"window_area":18.66,"window_type":1,"glazing_type":1,"window_location":0},{"orientation":1,"window_area":1.77,"window_type":2,"glazing_type":2,"window_location":0}],"schema_type":"SAP-Schema-16.2","uprn_source":"Energy Assessor","country_code":"EAW","main_heating":[{"description":"Boiler and radiators, mains gas","energy_efficiency_rating":4,"environmental_efficiency_rating":4}],"dwelling_type":"Semi-detached house","language_code":1,"property_type":0,"address_line_1":"61, Holme Lacey Road","schema_version":"LIG-16.1","assessment_type":"RdSAP","completion_date":"2014-01-20","inspection_date":"2014-01-20","extensions_count":0,"measurement_type":1,"total_floor_area":128,"transaction_type":1,"conservatory_type":2,"heated_room_count":6,"registration_date":"2014-01-20","restricted_access":0,"sap_energy_source":{"main_gas":"Y","meter_type":2,"photovoltaic_supply":{"percent_roof_area":0},"wind_turbines_count":0,"wind_turbines_terrain_type":2},"secondary_heating":{"description":"Room heaters, smokeless fuel","energy_efficiency_rating":0,"environmental_efficiency_rating":0},"sap_building_parts":[{"identifier":"Main Dwelling","wall_dry_lined":"N","wall_thickness":280,"floor_heat_loss":7,"sap_room_in_roof":{"floor_area":31.58,"insulation":"NI","roof_room_connected":"N","construction_age_band":"K","roof_insulation_thickness":"ND"},"roof_construction":5,"wall_construction":3,"building_part_number":1,"sap_floor_dimensions":[{"floor":0,"room_height":2.67,"floor_insulation":1,"total_floor_area":48.31,"floor_construction":2,"heat_loss_perimeter":21.24},{"floor":1,"room_height":2.56,"total_floor_area":48.31,"heat_loss_perimeter":21.24}],"wall_insulation_type":4,"construction_age_band":"C","wall_thickness_measured":"Y","roof_insulation_location":4,"roof_insulation_thickness":"NI"}],"low_energy_lighting":36,"solar_water_heating":"N","bedf_revision_number":352,"habitable_room_count":6,"heating_cost_current":{"value":1488,"currency":"GBP"},"insulated_door_count":0,"co2_emissions_current":8.7,"energy_rating_average":60,"energy_rating_current":45,"lighting_cost_current":{"value":111,"currency":"GBP"},"main_heating_controls":[{"description":"Programmer, no room thermostat","energy_efficiency_rating":1,"environmental_efficiency_rating":1}],"multiple_glazing_type":"ND","open_fireplaces_count":2,"has_hot_water_cylinder":"false","heating_cost_potential":{"value":773,"currency":"GBP"},"hot_water_cost_current":{"value":115,"currency":"GBP"},"mechanical_ventilation":0,"percent_draughtproofed":100,"suggested_improvements":[{"sequence":1,"typical_saving":{"value":442,"currency":"GBP"},"indicative_cost":"£4,000 - £14,000","improvement_type":"Q","improvement_details":{"improvement_number":7},"improvement_category":5,"energy_performance_rating":59,"environmental_impact_rating":50},{"sequence":2,"typical_saving":{"value":72,"currency":"GBP"},"indicative_cost":"£800 - £1,200","improvement_type":"W","improvement_details":{"improvement_number":47},"improvement_category":5,"energy_performance_rating":61,"environmental_impact_rating":53},{"sequence":3,"typical_saving":{"value":35,"currency":"GBP"},"indicative_cost":"£45","improvement_type":"E","improvement_details":{"improvement_number":35},"improvement_category":5,"energy_performance_rating":62,"environmental_impact_rating":53},{"sequence":4,"typical_saving":{"value":156,"currency":"GBP"},"indicative_cost":"£350 - £450","improvement_type":"G","improvement_details":{"improvement_number":12},"improvement_category":5,"energy_performance_rating":67,"environmental_impact_rating":60},{"sequence":5,"typical_saving":{"value":65,"currency":"GBP"},"indicative_cost":"£2,200 - £3,000","improvement_type":"I","improvement_details":{"improvement_number":20},"improvement_category":5,"energy_performance_rating":69,"environmental_impact_rating":63},{"sequence":6,"typical_saving":{"value":250,"currency":"GBP"},"indicative_cost":"£9,000 - £14,000","improvement_type":"U","improvement_details":{"improvement_number":34},"improvement_category":5,"energy_performance_rating":77,"environmental_impact_rating":70}],"co2_emissions_potential":3.6,"energy_rating_potential":77,"lighting_cost_potential":{"value":68,"currency":"GBP"},"alternative_improvements":[{"sequence":1,"typical_saving":{"value":75,"currency":"GBP"},"improvement_type":"Z1","improvement_details":{"improvement_number":51},"improvement_category":6,"energy_performance_rating":69,"environmental_impact_rating":72},{"sequence":2,"typical_saving":{"value":252,"currency":"GBP"},"improvement_type":"Z3","improvement_details":{"improvement_number":53},"improvement_category":6,"energy_performance_rating":75,"environmental_impact_rating":72}],"hot_water_cost_potential":{"value":103,"currency":"GBP"},"renewable_heat_incentive":{"water_heating":2305,"impact_of_loft_insulation":-1271,"impact_of_solid_wall_insulation":-7561,"space_heating_existing_dwelling":22097},"seller_commission_report":"Y","energy_consumption_current":315,"has_fixed_air_conditioning":"false","calculation_software_version":8.3,"energy_consumption_potential":121,"environmental_impact_current":36,"fixed_lighting_outlets_count":14,"current_energy_efficiency_band":"E","environmental_impact_potential":70,"has_heated_separate_conservatory":"false","potential_energy_efficiency_band":"C","co2_emissions_current_per_floor_area":68,"low_energy_fixed_lighting_outlets_count":5}} \ No newline at end of file diff --git a/datatypes/epc/domain/mapper.py b/datatypes/epc/domain/mapper.py index 5a04e5165..f07130347 100644 --- a/datatypes/epc/domain/mapper.py +++ b/datatypes/epc/domain/mapper.py @@ -3944,6 +3944,22 @@ def _normalize_sap_schema_16_x(data: Dict[str, Any]) -> Dict[str, Any]: if "single" in description: d["multiple_glazing_type"] = 5 + # P4 (worklist): 16.x certs sometimes omit `multiple_glazed_proportion` + # entirely (RdSapSchema17_1 requires it, no default) even though the + # human `window.description` states the glazing extent unambiguously — + # e.g. cert 0141-2860-6891-9124-5625 (uprn 100021969385, SAP-16.2): + # multiple_glazing_type="ND", proportion absent, description "Fully + # double glazed". Derive from that explicit text rather than a flat + # default (a flat `100` for every ND-missing cert previously regressed + # the component-accuracy donor pool — see worklist P4): "single" → 0, + # "double" → 100. Genuinely ambiguous certs (no double/single wording) + # are left unset and still fail loud — correct, data is insufficient. + if "multiple_glazed_proportion" not in d: + if "single" in description: + d["multiple_glazed_proportion"] = 0 + elif "double" in description: + d["multiple_glazed_proportion"] = 100 + sap_heating: Any = d.get("sap_heating") if isinstance(sap_heating, dict): heating: Dict[str, Any] = cast(Dict[str, Any], sap_heating) diff --git a/datatypes/epc/domain/tests/test_from_sap_schema.py b/datatypes/epc/domain/tests/test_from_sap_schema.py index b561393cf..6b833fb6f 100644 --- a/datatypes/epc/domain/tests/test_from_sap_schema.py +++ b/datatypes/epc/domain/tests/test_from_sap_schema.py @@ -838,6 +838,25 @@ class TestFromSapSchema16_2: ): EpcPropertyDataMapper.from_api_response(data) + def test_16_x_missing_multiple_glazed_proportion_derived_from_double_description( + self, + ) -> None: + # Cert 0141-2860-6891-9124-5625 (uprn 100021969385, SAP-16.2, property + # 753950): multiple_glazing_type="ND", `multiple_glazed_proportion` + # entirely absent, but windows[].description explicitly says "Fully + # double glazed" — an unambiguous signal, unlike the truly ambiguous + # ND-only certs the fail-loud test above still guards. The normaliser + # derives 100 rather than leaving the cert unmappable. + data = load("sap_16_2.json") + data["windows"][0]["description"] = "Fully double glazed" + del data["multiple_glazed_proportion"] + + epc = EpcPropertyDataMapper.from_api_response(data) + + assert isinstance(epc, EpcPropertyData) + assert epc.sap_windows + assert all(w.glazing_type == 2 for w in epc.sap_windows) # DG-modal slot + def test_recorded_co2_as_measurement_dict_is_coerced_not_crashed(self) -> None: # Some certs (e.g. 16.x cert 2308-4997-7262-0137-9930) lodge # `co2_emissions_current` as a Measurement dict {'value': 3.5, 'quantity': diff --git a/tests/domain/epc_prediction/test_component_accuracy_gate.py b/tests/domain/epc_prediction/test_component_accuracy_gate.py index ce7105ef6..e4bcd41f6 100644 --- a/tests/domain/epc_prediction/test_component_accuracy_gate.py +++ b/tests/domain/epc_prediction/test_component_accuracy_gate.py @@ -65,6 +65,24 @@ _FIXTURE = Path(__file__).parents[3] / "tests" / "fixtures" / "epc_prediction" # and one full-SAP target the similarity-weighted donors don't predict as PV # tips the agreement 32/33 (the held-out actual is now correct — a ground-truth- # method change, not a prediction-logic loosening). Tighten-only resumes here. +# +# has_hot_water_cylinder 0.8687->0.8586, cylinder_insulation_type 0.3333->0.1667, +# door_count residual 0.3131->0.3333 re-baselined when the mapper started +# deriving `multiple_glazed_proportion` (single->0 / double->100) from an +# unambiguous window.description for 16.x certs whose "ND" multiple_glazing_type +# previously left the numeric field entirely absent — RdSapSchema17_1 requires +# it, so the cert failed loud and was excluded from the donor pool wholesale +# (worklist P4; see the mapper comment at `_normalize_sap_schema_16_x`). +# Motivating case: cert 0141-2860-6891-9124-5625 / uprn 100021969385 (property +# 753950), lodging "Fully double glazed" with the proportion field missing — +# previously hard-failed the whole modelling_e2e batch it was in. Admitting +# these certs is a ground-truth-method change (more real lodged EPCs enter the +# comparable pool), not a prediction-logic loosening; the tip is donor-pool +# composition on a handful of near-tie matches, same mechanism as the #1245 / +# ADR-0037 re-baselines above. A flat `multiple_glazed_proportion=100` default +# was tried earlier and reverted for the same reason without a derivation +# rationale; deriving it from explicit "single"/"double" wording is the +# targeted fix worklist P4 called for. Tighten-only resumes from these values. _RATE_FLOORS: dict[str, float] = { "wall_construction": 0.9091, "wall_insulation_type": 0.8687, @@ -77,8 +95,8 @@ _RATE_FLOORS: dict[str, float] = { "heating_main_control": 0.9091, "water_heating_fuel": 0.9495, "water_heating_code": 0.9798, - "has_hot_water_cylinder": 0.8687, - "cylinder_insulation_type": 0.3333, + "has_hot_water_cylinder": 0.8586, + "cylinder_insulation_type": 0.1667, "secondary_heating_type": 0.0000, "roof_insulation_thickness": 0.4118, "roof_insulation_thickness_pm1": 0.4118, @@ -110,7 +128,7 @@ _RESIDUAL_CEILINGS: dict[str, float] = { "floor_area": 12.0586, "total_window_area": 3.7184, "building_parts": 0.1212, - "door_count": 0.3131, + "door_count": 0.3333, } _TOLERANCE = 1e-3 From 3c9b75cbceefe77a2430dee35a4b5cc0eacdfaa9 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 12:04:35 +0000 Subject: [PATCH 47/55] Hand-pick selection by landlord_property_id, not property.id MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit uploaded_files has no property_id — matching is on landlord_property_id — so the property_ids path was pointless indirection (look property.id up in property just to translate back to landlord_property_id). The FE holds landlord_property_id per row anyway. task.inputs hand-pick key is now landlord_property_ids: str[] (was property_ids: int[]). resolve_selection unions two landlord_property_id sources — project_codes expanded via hubspot_deal_data, and the hand-picked ids taken as given — and no longer queries the property table at all. Trigger-only change; the domain plan, orchestrator and matching already work in landlord_property_id. ADR-0060, CONTEXT.md and the request schema updated to match. Co-Authored-By: Claude Opus 4.8 (1M context) --- CONTEXT.md | 2 +- backend/app/documents/download_tasks.py | 32 +++++----------- backend/app/documents/router.py | 24 +++++++----- backend/app/documents/schemas.py | 11 +++--- ...60-bulk-document-download-package-model.md | 23 +++++------ .../documents/test_bulk_download_router.py | 38 ++++++------------- 6 files changed, 54 insertions(+), 76 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index c736d712d..01a3c2101 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -108,7 +108,7 @@ The single ZIP archive a **Bulk Document Download** produces: one folder per pro _Avoid_: export (that is the plan/scenario XLSX export), bundle, archive (ambiguous), zip (the format, not the concept) **Bulk Document Download**: -The feature/job that assembles a **Download Package** for a chosen set of properties (the union of one or more whole HubSpot projects, selected by `project_code`, and a hand-picked `property_id` list) and emails the requester a link. Runs on the app-owned-task + attach-mode lane (ADR-0055): the FastAPI route resolves the selection to a distinct `landlord_property_id` set — project codes are expanded via `hubspot_deal_data` (which holds the project↔property grain) — pins that set and the recipient email onto the `sub_task`, and the `applications/bulk_document_download` Lambda builds the package, writes the URL to `sub_task.outputs`, and emails it (ADR-0059/0060). Files are matched by `landlord_property_id` (the missing `property_id` on `uploaded_files` is a known future gap). Selection is capped by property count at the trigger. +The feature/job that assembles a **Download Package** for a chosen set of properties (the union of one or more whole HubSpot projects, selected by `project_code`, and a hand-picked `landlord_property_id` list) and emails the requester a link. Runs on the app-owned-task + attach-mode lane (ADR-0055): the FastAPI route resolves the selection to a distinct `landlord_property_id` set — project codes are expanded via `hubspot_deal_data` (which holds the project↔property grain), hand-picked ids are taken as given — pins that set and the recipient email onto the `sub_task`, and the `applications/bulk_document_download` Lambda builds the package, writes the URL to `sub_task.outputs`, and emails it (ADR-0059/0060). `landlord_property_id` is the selection key throughout because `uploaded_files` is matched on it (it has no `property_id`). Files are matched by `landlord_property_id` (the missing `property_id` on `uploaded_files` is a known future gap). Selection is capped by property count at the trigger. _Avoid_: document export, bulk export, file dump ### Source data diff --git a/backend/app/documents/download_tasks.py b/backend/app/documents/download_tasks.py index 7109224a6..ef4afeba3 100644 --- a/backend/app/documents/download_tasks.py +++ b/backend/app/documents/download_tasks.py @@ -52,23 +52,20 @@ class DocumentDownloadTasks: def resolve_selection( self, project_codes: list[str], - property_ids: list[int], + landlord_property_ids: list[str], ) -> list[str]: """Resolve the selection to the distinct ``landlord_property_id`` set the - Download Package is built from (ADR-0060). Two independent sources, - unioned: + Download Package is built from (ADR-0060) — files are matched on + ``landlord_property_id`` (``uploaded_files`` has no ``property_id``), so + that is the selection key throughout. Two independent sources, unioned: - **project codes** — every property in the named HubSpot projects, read - from ``hubspot_deal_data`` (the project↔property grain lives there, not - on ``property``). - - **hand-picked property ids** — the selected ``property`` rows. + from ``hubspot_deal_data`` (the project↔property grain lives there). + - **hand-picked landlord_property_ids** — used as given. - Files are matched by ``landlord_property_id`` (the missing ``property_id`` - on ``uploaded_files`` is a known future gap), so a source row with a null - ``landlord_property_id`` contributes nothing and is dropped here. The - returned set size is the cap basis (property count, ADR-0060). + The returned set size is the cap basis (property count, ADR-0060). """ - landlord_property_ids: set[str] = set() + resolved: set[str] = {lpid for lpid in landlord_property_ids if lpid} if project_codes: rows = self._session.connection().execute( text( @@ -78,17 +75,8 @@ class DocumentDownloadTasks: ), {"codes": project_codes}, ) - landlord_property_ids.update(row[0] for row in rows.all()) - if property_ids: - rows = self._session.connection().execute( - text( - "SELECT landlord_property_id FROM property" - " WHERE id = ANY(:ids) AND landlord_property_id IS NOT NULL" - ), - {"ids": property_ids}, - ) - landlord_property_ids.update(row[0] for row in rows.all()) - return sorted(landlord_property_ids) + resolved.update(row[0] for row in rows.all()) + return sorted(resolved) def create_download_subtask( self, task_id: UUID, subtask_id: UUID, recipe: dict[str, Any] diff --git a/backend/app/documents/router.py b/backend/app/documents/router.py index 50b7a957b..a4994bb82 100644 --- a/backend/app/documents/router.py +++ b/backend/app/documents/router.py @@ -92,7 +92,7 @@ async def bulk_download( config = tasks.read_selection_config(body.task_id) portfolio_id: Any = config.get("portfolio_id") raw_project_codes: Any = config.get("project_codes") or [] - raw_property_ids: Any = config.get("property_ids") or [] + raw_landlord_property_ids: Any = config.get("landlord_property_ids") or [] if not isinstance(raw_project_codes, list) or not all( isinstance(code, str) for code in cast(list[Any], raw_project_codes) ): @@ -100,28 +100,32 @@ async def bulk_download( status_code=400, detail="task.inputs project_codes must be a list of strings.", ) - if not isinstance(raw_property_ids, list) or not all( - isinstance(pid, int) for pid in cast(list[Any], raw_property_ids) + if not isinstance(raw_landlord_property_ids, list) or not all( + isinstance(lpid, str) for lpid in cast(list[Any], raw_landlord_property_ids) ): raise HTTPException( status_code=400, - detail="task.inputs property_ids must be a list of integers.", + detail="task.inputs landlord_property_ids must be a list of strings.", ) project_codes: list[str] = cast(list[str], raw_project_codes) - property_ids: list[int] = cast(list[int], raw_property_ids) - if not project_codes and not property_ids: + selected_landlord_property_ids: list[str] = cast( + list[str], raw_landlord_property_ids + ) + if not project_codes and not selected_landlord_property_ids: raise HTTPException( status_code=400, - detail="task.inputs must provide project_codes or property_ids.", + detail="task.inputs must provide project_codes or landlord_property_ids.", ) - landlord_property_ids = tasks.resolve_selection(project_codes, property_ids) + landlord_property_ids = tasks.resolve_selection( + project_codes, selected_landlord_property_ids + ) if not landlord_property_ids: raise HTTPException( status_code=400, detail=( - "The selection resolves to no downloadable properties — none of " - "the selected projects or properties has a landlord_property_id." + "The selection resolves to no properties — the project codes " + "matched nothing and no landlord_property_ids were given." ), ) if len(landlord_property_ids) > MAX_PROPERTIES: diff --git a/backend/app/documents/schemas.py b/backend/app/documents/schemas.py index c383ddfaf..fbea98294 100644 --- a/backend/app/documents/schemas.py +++ b/backend/app/documents/schemas.py @@ -8,12 +8,13 @@ class BulkDownloadRequest(BaseModel): The front end creates the app-owned Task first and writes the **selection config** into ``task.inputs`` (a JSON object of - ``{project_codes?: str[], property_ids?: int[], portfolio_id?: int}`` — so a - large selection never travels in an HTTP body), then passes only the + ``{project_codes?: str[], landlord_property_ids?: str[], portfolio_id?: int}`` + — so a large selection never travels in an HTTP body), then passes only the ``task_id`` here. The backend reads the selection from ``task.inputs`` and - resolves it to the distinct ``landlord_property_id`` set: the union of every - property in the named HubSpot ``project_codes`` (from ``hubspot_deal_data``) - and the hand-picked ``property_ids``. At least one of the two must be given; + resolves it to the distinct ``landlord_property_id`` set (the key + ``uploaded_files`` is matched on): the union of every property in the named + HubSpot ``project_codes`` (from ``hubspot_deal_data``) and the hand-picked + ``landlord_property_ids``. At least one of the two must be given; ``portfolio_id`` is optional and used only to name the package. """ diff --git a/docs/adr/0060-bulk-document-download-package-model.md b/docs/adr/0060-bulk-document-download-package-model.md index 10959d19a..2eedb6d9f 100644 --- a/docs/adr/0060-bulk-document-download-package-model.md +++ b/docs/adr/0060-bulk-document-download-package-model.md @@ -25,17 +25,18 @@ lane (ADR-0055).** - **Trigger & lifecycle (ADR-0055).** The **front end** creates the app-owned `tasks` row and writes the **selection config** — - `{project_codes?: str[], property_ids?: int[], portfolio_id?: int}` — into - `tasks.inputs` (TEXT/JSON, FE-owned), then calls the FastAPI route with **only - the `task_id`**. A large selection therefore never travels in an HTTP body. - The route **reads the selection from `tasks.inputs`** and resolves it to the - distinct `landlord_property_id` set — the **union** of every property in the - named HubSpot `project_codes` (read from `hubspot_deal_data`, where the - project↔property grain lives — `property` carries only `portfolio_id`) and the - hand-picked `property_ids`. At least one of the two must be provided; - `portfolio_id` is optional and used only to name the package. The route then - caps the set, resolves the recipient email from the authenticated user - (ADR-0059), **pins the resolved recipe** + `{project_codes?: str[], landlord_property_ids?: str[], portfolio_id?: int}` — + into `tasks.inputs` (TEXT/JSON, FE-owned), then calls the FastAPI route with + **only the `task_id`**. A large selection therefore never travels in an HTTP + body. The route **reads the selection from `tasks.inputs`** and resolves it to + the distinct `landlord_property_id` set — the **union** of every property in + the named HubSpot `project_codes` (read from `hubspot_deal_data`, where the + project↔property grain lives) and the hand-picked `landlord_property_ids`. + `landlord_property_id` is the selection key throughout because + `uploaded_files` is matched on it (it has no `property_id`). At least one of + the two must be provided; `portfolio_id` is optional and used only to name the + package. The route then caps the set, resolves the recipient email from the + authenticated user (ADR-0059), **pins the resolved recipe** (`landlord_property_ids`, `recipient_email`, `package_name`) onto **one pre-created `sub_task`'s `inputs`**, and drops one SQS message (`task_id`, `sub_task_id`). The new `applications/bulk_document_download` Lambda runs in diff --git a/tests/backend/app/documents/test_bulk_download_router.py b/tests/backend/app/documents/test_bulk_download_router.py index c21da8ca1..25ce84ec2 100644 --- a/tests/backend/app/documents/test_bulk_download_router.py +++ b/tests/backend/app/documents/test_bulk_download_router.py @@ -24,7 +24,6 @@ from backend.app.documents.router import ( get_requesting_user_email, get_session, ) -from infrastructure.postgres.property_table import PropertyRow from infrastructure.postgres.task_table import TaskRow from repositories.tasks.subtask_postgres_repository import SubTaskPostgresRepository @@ -51,17 +50,6 @@ class Api: session.refresh(row) return row.id - def seed_property(self, landlord_property_id: str) -> int: - with Session(self.engine) as session: - row = PropertyRow( - portfolio_id=PORTFOLIO_ID, landlord_property_id=landlord_property_id - ) - session.add(row) - session.commit() - session.refresh(row) - assert row.id is not None - return row.id - def seed_deal(self, project_code: str, landlord_property_id: str) -> None: """A hubspot_deal_data row carrying the project↔property link the project-code selection resolves against (ADR-0060).""" @@ -105,11 +93,10 @@ def _trigger(api: Api, task_id: UUID) -> Any: def test_pins_the_recipe_and_enqueues_one_message(api: Api) -> None: - # arrange — two selected properties, pinned as the FE's task.inputs config. - p1 = api.seed_property("LP1") - p2 = api.seed_property("LP2") + # arrange — two hand-picked properties, by landlord_property_id (the key + # uploaded_files is matched on), pinned as the FE's task.inputs config. task_id = api.seed_task( - {"portfolio_id": PORTFOLIO_ID, "property_ids": [p1, p2]} + {"portfolio_id": PORTFOLIO_ID, "landlord_property_ids": ["LP1", "LP2"]} ) # act @@ -145,13 +132,12 @@ def test_resolves_all_properties_for_the_given_project_codes(api: Api) -> None: assert inputs[0]["landlord_property_ids"] == ["LP1", "LP2", "LP3"] -def test_unions_project_codes_with_hand_picked_property_ids(api: Api) -> None: - # arrange — one project plus a hand-picked property from outside it. +def test_unions_project_codes_with_hand_picked_landlord_property_ids(api: Api) -> None: + # arrange — one project, plus hand-picks that overlap it (LP2) and extend it (LP5). api.seed_deal("PROJ-A", "LP1") api.seed_deal("PROJ-A", "LP2") - hand_picked = api.seed_property("LP5") task_id = api.seed_task( - {"project_codes": ["PROJ-A"], "property_ids": [hand_picked]} + {"project_codes": ["PROJ-A"], "landlord_property_ids": ["LP2", "LP5"]} ) # act @@ -197,10 +183,8 @@ def test_rejects_a_project_code_that_resolves_to_no_properties(api: Api) -> None def test_rejects_a_selection_over_the_cap(api: Api, monkeypatch: Any) -> None: # arrange — cap of 1, two selected properties. monkeypatch.setattr(documents_router, "MAX_PROPERTIES", 1) - p1 = api.seed_property("LP1") - p2 = api.seed_property("LP2") task_id = api.seed_task( - {"portfolio_id": PORTFOLIO_ID, "property_ids": [p1, p2]} + {"portfolio_id": PORTFOLIO_ID, "landlord_property_ids": ["LP1", "LP2"]} ) # act @@ -214,8 +198,9 @@ def test_rejects_a_selection_over_the_cap(api: Api, monkeypatch: Any) -> None: def test_rejects_a_task_that_already_has_a_subtask(api: Api) -> None: # arrange — a task already triggered once. - p1 = api.seed_property("LP1") - task_id = api.seed_task({"portfolio_id": PORTFOLIO_ID, "property_ids": [p1]}) + task_id = api.seed_task( + {"portfolio_id": PORTFOLIO_ID, "landlord_property_ids": ["LP1"]} + ) assert _trigger(api, task_id).status_code == 202 # act @@ -228,8 +213,7 @@ def test_rejects_a_task_that_already_has_a_subtask(api: Api) -> None: def test_rejects_a_task_with_no_selection_config(api: Api) -> None: - # arrange — a task whose inputs carry no portfolio_id/selection. - api.seed_property("LP1") + # arrange — a task whose inputs carry no selection. task_id = api.seed_task({}) # act From 95322ac4ad7ec987ef23a1f1b669d949f6c693a5 Mon Sep 17 00:00:00 2001 From: Jun-te Kim Date: Wed, 8 Jul 2026 12:38:26 +0000 Subject: [PATCH 48/55] Add Elmhurst build script + validated result for uprn 100021969385 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Built the property-753950 cert (worklist P4, PR #1503) in the live RdSAP-10 Online tool via Playwright: SAP 44 (E), vs our engine's 42 and the lodged 2014 cert's 45 — a normal RdSAP-2012->SAP-10.2 recalculation spread, not a defect surfaced by the multiple_glazed_proportion fix. Also surfaced (and documented as a caveat, not fixed here) a pre-existing SAP-16.2 gap: the cert lodges party_wall_length_m=0 on both floors despite being semi-detached, which Elmhurst's validation rejects and this PR's engine change doesn't touch. --- .../uprn_100021969385/elmhurst_inputs.md | 29 ++- scripts/hyde/build_100021969385.py | 209 ++++++++++++++++++ 2 files changed, 237 insertions(+), 1 deletion(-) create mode 100644 scripts/hyde/build_100021969385.py diff --git a/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-16.2/uprn_100021969385/elmhurst_inputs.md b/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-16.2/uprn_100021969385/elmhurst_inputs.md index 2c98526ef..f79bbdaca 100644 --- a/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-16.2/uprn_100021969385/elmhurst_inputs.md +++ b/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-16.2/uprn_100021969385/elmhurst_inputs.md @@ -1,6 +1,33 @@ # Elmhurst RdSAP inputs — UPRN 100021969385 (cert 0141-2860-6891-9124-5625, SAP-Schema-16.2) -**Lodged SAP:** 45 **Our engine:** 42 ← compare Elmhurst against this +**Lodged SAP:** 45 **Our engine:** 42 **Elmhurst (built 2026-07-08):** 44 (E) ← compare Elmhurst against this + +## Elmhurst result + +Built via the automated `scripts/hyde/build_100021969385.py` Playwright script +against the live RdSAP-10 Online tool (assessment `P960-0001-001431`): + +| | Current | Potential | +|---|---|---| +| SAP Rating | **44 (E)** | 75 (C) | +| EI Rating | 38 (F) | 71 (C) | +| Estimated annual cost | £2324 | £853 | + +Elmhurst (44) sits between the engine (42) and the lodged 2014 cert (45) — a +2-3 point spread, consistent with the general RdSAP-2012→SAP-10.2 recalculation +gap seen across the corpus, not a specific defect surfaced by this PR's fix. +The Recommendations validation gate passed with zero errors. + +**Caveat on comparability:** the cert's lodged `sap_floor_dimensions` carry +`party_wall_length_m = 0.0` on both floors even though the dwelling is +semi-detached (a genuine physical party wall must exist) — Elmhurst's +Recommendations validation rejects an all-zero party wall for a semi/terrace +and forced a non-zero entry (7.0 m, an estimate, entered on both floors) to +proceed. Our engine has no such guard and models the lodged 0 m as-is, so it +under-credits party-wall heat loss relative to Elmhurst for this cert — a +pre-existing SAP-16.2 mapper gap (`party_wall_length` isn't lodged by this +schema; see `_normalize_sap_schema_16_x`), unrelated to the +`multiple_glazed_proportion` fix this PR makes. **Context:** this is the property-753950 / P4 corpus case (`.claude/skills/expand-sap-accuracy-corpus/worklist.md`). The raw gov-API cert omits `multiple_glazed_proportion` entirely (only diff --git a/scripts/hyde/build_100021969385.py b/scripts/hyde/build_100021969385.py new file mode 100644 index 000000000..e081120ad --- /dev/null +++ b/scripts/hyde/build_100021969385.py @@ -0,0 +1,209 @@ +"""Elmhurst build for UPRN 100021969385 (SAP-Schema-16.2, SEMI-DETACHED HOUSE, +2-storey, band C 1930-1949, solid brick uninsulated, mains-gas boiler + +radiators (PCDB index 8108, control 2102), 2 roof elements (pitched no +insulation + insulated roof room), suspended uninsulated ground floor, no +party wall, 2 real lodged windows (18.66 + 1.77 m^2, "Fully double glazed"), +secondary solid-smokeless-fuel room heater, no hot water cylinder (from main +system), TFA 128, 2 doors uninsulated. Engine 42 / lodged 45. + +Validates the multiple_glazed_proportion derivation fix (PR #1503, property +753950 / worklist P4): this cert's `multiple_glazing_type` is "ND" and +`multiple_glazed_proportion` is entirely absent from the gov-API payload — +previously a hard mapper failure. Run: + DISPLAY=:99 python scripts/hyde/build_100021969385.py +""" +from __future__ import annotations +import sys +import elmhurst_lib as E + +DIM = "TabContainer_TabPanelMain_WebUserControlDimensionsMain_" +WALL = ("TabContainer_TabPanelMain_InnerTabContainerMain_" + "TabPanelExternalWallMain_WebUserControlWallMain_") +PWALL = "TabContainer_TabPanelMain_InnerTabContainerMain_TabPanelPartyWallMain_WebUserControlPartyWallMain_" +ROOF = "TabContainer_TabPanelMain_WebUserControlRoofMain_" +FLOOR = "TabContainer_TabPanelMain_WebUserControlFloorsMain_" +DP = "TabContainer_TabPanelDoorsPanel_" +VP = "TabContainer_TabPanelVentilationPanel_" +APT = "TabContainer_TabPanelAirPressureTest_" +LP = "TabContainer_TabPanelLighting_" +MV = "TabContainer_TabPanelMechVent_" +WH = "TabContainer_TabPanelWaterHeating_" +MH1B = "TabContainer_TabPanelMainHeating1_WebUserControlMainHeating1_" + + +def _pick(page, suffix, contains): + val = page.evaluate( + """(a)=>{const s=document.getElementById(a[0]);if(!s)return null; + for(const o of s.options){if(o.text.toLowerCase().includes(a[1].toLowerCase()))return o.value;}return null;}""", + [f"{E.FP}{suffix}", contains]) + if val is not None: + E.set_select(page, suffix, val) + return val + + +def _options(page, suffix): + return page.evaluate( + """(id)=>{const s=document.getElementById(id);if(!s)return []; + return Array.from(s.options).map(o=>o.text);}""", f"{E.FP}{suffix}") + + +def property_description(page): + E.goto(page, "PropertyDescription", "WebFormPropertyDescription.aspx") + E.set_select(page, "DropDownListPropertyType1", "H House") + _pick(page, "DropDownListPropertyType2", "semi") # built_form 2 + E.set_text(page, "TextBoxStoreys", "2") + E.set_text(page, "TextBoxHabitableRooms", "6") + E.set_text(page, "TextBoxHeatedHabitableRooms", "6") + print("date ->", _pick(page, "DropDownListDateBuiltMain", "1930-1949")) # band C + E.set_select(page, "DropDownListDateBuiltFirst", "") + print("room-in-roof ->", _pick(page, "DropDownListRoomInRoofMain", "1930-1949")) + E.save_close(page) + + +def dimensions(page): + E.goto(page, "Dimensions", "WebFormDimensions.aspx") + E.set_text(page, f"{DIM}TextBoxFloorAreaLowestFloor", "48.31") + E.set_text(page, f"{DIM}TextBoxRoomHeightLowestFloor", "2.67") + E.set_text(page, f"{DIM}TextBoxWallPerimeterLowestFloor", "21.24") + E.set_text(page, f"{DIM}TextBoxPartyWallLengthLowestFloor", "0") + E.set_text(page, f"{DIM}TextBoxFloorArea1stFloor", "48.31") + E.set_text(page, f"{DIM}TextBoxRoomHeight1stFloor", "2.81") + E.set_text(page, f"{DIM}TextBoxWallPerimeter1stFloor", "21.24") + E.set_text(page, f"{DIM}TextBoxPartyWallLength1stFloor", "0") + E.save_close(page) + + +def walls(page): + E.goto(page, "Walls", "WebFormWalls.aspx") + print("wall type ->", _pick(page, f"{WALL}DropDownListType", "solid brick")) + page.wait_for_timeout(400) + print("insulation ->", _pick(page, f"{WALL}DropDownListInsulation", "as built")) + # No party wall lodged (party_wall_length_m = 0 on both floors). + pw = _pick(page, f"{PWALL}DropDownListPartyWallType", "determine") or \ + _pick(page, f"{PWALL}DropDownListPartyWallType", "none") + print("party wall ->", pw) + E.save_close(page) + + +def roofs(page): + # Two lodged roof elements: "Pitched, no insulation (assumed)" (main) and + # "Roof room(s), insulated". Enter the main roof as no-insulation pitched; + # the insulated roof-room element is flagged via Property Description's + # Room-in-Roof age band (set above) rather than a 2nd roof row (Elmhurst's + # main-roof tab is single-element). + E.goto(page, "Roofs", "WebFormRoofs.aspx") + print("roof type ->", _pick(page, f"{ROOF}DropDownListType", "access to loft")) + print("insulation at ->", _pick(page, f"{ROOF}DropDownListInsulationAt", "joists")) + print("thickness ->", _pick(page, f"{ROOF}DropDownListThickness", "no insulation") or + _pick(page, f"{ROOF}DropDownListThickness", "0 mm")) + E.save_close(page) + + +def floors(page): + E.goto(page, "Floors", "WebFormFloors.aspx") + print("location ->", _pick(page, f"{FLOOR}DropDownListLocation", "ground")) + print("floor type ->", _pick(page, f"{FLOOR}DropDownListType", "suspended")) + print("insulation ->", _pick(page, f"{FLOOR}DropDownListInsulation", "as built")) + E.save_close(page) + + +def openings(page): + E.goto(page, "Openings", "WebFormOpenings.aspx") + E.click_tab(page, "TabContainer_TabPanelWindowsPanel") + # windows[].description = "Fully double glazed", no install-date band lodged. + glazing = "Double post or during 2022" + for opt in _options(page, "TabContainer_TabPanelWindowsPanel_DropDownListExtGlazing"): + low = opt.lower() + if "unknown install date" in low and "single" not in low and "triple" not in low: + glazing = opt + break + print("glazing ->", glazing) + E.set_single_window(page, 18.66 + 1.77, orientation="South", glazing=glazing) + E.click_tab(page, "TabContainer_TabPanelDoorsPanel") + E.set_text(page, f"{DP}TextBoxDoors", "2") + E.set_text(page, f"{DP}TextBoxDoorsInsulated", "0") + E.set_text(page, f"{DP}TextBoxDraughtProofedDoors", "2") + E.save_close(page) + + +def ventilation(page): + E.goto(page, "VentilationAndCooling", "WebFormVentilationAndCooling.aspx") + E.click_tab(page, "TabContainer_TabPanelVentilationPanel") + E.set_text(page, f"{VP}TextBoxIntermittentFans", "0") + E.set_text(page, f"{VP}TextBoxOpenChimneys", "2") + cool = page.locator(f"#{E.FP}{VP}CheckBoxFixedSpaceCooling") + if cool.count() and cool.is_checked(): + E.commit(page, cool.uncheck) + E.click_tab(page, "TabContainer_TabPanelMechVent") + mv = page.locator(f"#{E.FP}{MV}CheckBoxMechanicalVentilation") + if mv.count() and mv.is_checked(): + E.commit(page, mv.uncheck) + E.click_tab(page, "TabContainer_TabPanelAirPressureTest") + E.set_select(page, f"{APT}DropDownListTestMethod", "Not available") + E.click_tab(page, "TabContainer_TabPanelLighting") + E.set_text(page, f"{LP}TextBoxLightsTotal", "14") + E.set_text(page, f"{LP}TextBoxLedLightsTotal", "5") + E.set_text(page, f"{LP}TextBoxCflLightsTotal", "0") + E.save_close(page) + + +def space_heating(page): + # Mains gas boiler + radiators, exact PCDB index 8108, controls SAP 2102. + E.goto(page, "SpaceHeating", "WebFormSpaceHeating.aspx") + page.wait_for_timeout(1000) + E.clear_main_heating_code(page) + desc = E.set_pcdb_boiler(page, 8108) + print("boiler resolved ->", desc) + E.set_heating_dialog(page, f"{MH1B}ButtonMainHeatingControls", + "^Boilers", "^Standard", "programmer, room thermostat") + print("control:", page.locator(f"#{E.MH1}TextBoxMainHeatingControls").input_value()) + E.save_close(page) + + +def secondary(page): + # Lodged secondary: "Room heaters, smokeless fuel". + E.goto(page, "SpaceHeating", "WebFormSpaceHeating.aspx") + page.wait_for_timeout(600) + E.set_select(page, "DropDownListSecondaryHeatingPresent", "Yes") + page.wait_for_timeout(900) + E.set_heating_dialog(page, "ButtonSecondaryHeatingCode", + "Solid", "Room Heater", "smokeless") + tb = page.locator(f"#{E.FP}TextBoxSecondaryHeatingCode") + print("secondary code:", tb.input_value() if tb.count() else "?") + E.save_close(page) + + +def water_heating(page): + # has_hot_water_cylinder = false -> from the main heating system, no cylinder. + E.goto(page, "WaterHeating", "WebFormWaterHeating.aspx") + E.click_tab(page, "TabContainer_TabPanelWaterHeating") + page.wait_for_timeout(400) + E.clear_hot_water_cylinder(page) + E.set_heating_dialog(page, f"{WH}ButtonWaterHeatingCode", + "From Space Heating", "From the primary heating system") + print("water code:", page.locator(f"#{E.FP}{WH}TextBoxWaterHeatingCode").input_value()) + E.save_close(page) + + +_ORDER = ["property_description", "dimensions", "walls", "roofs", "floors", + "openings", "ventilation", "space_heating", "secondary", + "water_heating"] + + +def main(): + if len(sys.argv) < 2 or sys.argv[1] not in _ORDER + ["all"]: + print("usage: build_100021969385.py <" + "|".join(_ORDER + ["all"]) + ">") + return 2 + with E.session() as (ctx, page): + if sys.argv[1] == "all": + for step in _ORDER: + print(f"=== {step} ===") + globals()[step](page) + else: + globals()[sys.argv[1]](page) + print("done:", sys.argv[1], "->", page.url) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 41ded4740ed6b8ca72ca79286c8dcb7524595f31 Mon Sep 17 00:00:00 2001 From: Jun-te Kim Date: Wed, 8 Jul 2026 13:05:43 +0000 Subject: [PATCH 49/55] Cascade raw API glazing_type on lodged sap_windows, not pass through raw datatypes/epc/domain/mapper.py's from_rdsap_schema_{17_0,17_1,18_0,19_0, 20_0_0,21_0_0} all stored a lodged sap_windows[].glazing_type verbatim instead of routing it through _api_cascade_glazing_type (which _api_sap_window, used by 21_0_1, already does correctly). The raw API code is the RdSAP-21 glazing enum (e.g. code 1 = "DG pre-2002"), not the SAP 10.2 Table 6b cascade enum the calculator's U/g-value tables are keyed on (cascade code 1 = single glazed) -- so any lodged window whose raw code happened to collide with a different cascade meaning was silently mis-rated. Found while validating PR #1503 (property 753950 / uprn 100021969385) against Elmhurst: our engine's windows_w_per_k was 82.27 vs Elmhurst's 51.44 -- the dwelling's larger window (91% of glazed area) lodges raw glazing_type=1 and was being modelled as single glazed instead of double, the dominant contributor to a SAP 42 vs Elmhurst 44 gap. Fixing this narrows windows_w_per_k to 50.26 (Elmhurst 51.44, ~2% off) and moves the engine's score to 43. Extracted the 5 duplicated reduced-schema-window blocks (17.0/17.1/18.0/ 19.0/20.0.0, which lodge a minimal window shape with no glazing_gap/ frame_factor/transmission fields) into a shared _reduced_field_api_sap_ window helper that also now populates window_transmission_details/ frame_factor via the SAP10 lookup instead of leaving them None. 21.0.0's richer inline block gets the same one-line cascade fix. +6 regression tests (one per affected schema seam + 21.0.0). Full domain/ schema/prediction/comparable-properties/sap10_calculator/modelling_e2e suites pass (2746 passed, only the 2 pre-existing unrelated failures); component-accuracy gate 26/26 unaffected. --- datatypes/epc/domain/mapper.py | 144 +++++++----------- .../domain/tests/test_from_rdsap_schema.py | 56 +++++++ 2 files changed, 108 insertions(+), 92 deletions(-) diff --git a/datatypes/epc/domain/mapper.py b/datatypes/epc/domain/mapper.py index 5a04e5165..76ef848af 100644 --- a/datatypes/epc/domain/mapper.py +++ b/datatypes/epc/domain/mapper.py @@ -728,22 +728,7 @@ class EpcPropertyDataMapper: # rich certs keep their lodged per-window geometry (used directly: # window_width = area, height = 1.0). sap_windows=( - [ - SapWindow( - frame_material=None, - glazing_gap=0, - orientation=w.orientation, - window_type=w.window_type, - glazing_type=w.glazing_type, - window_width=_measurement_value(w.window_area), - window_height=1.0, - draught_proofed=False, - window_location=w.window_location, - window_wall_type=0, - permanent_shutters_present=False, - ) - for w in schema.sap_windows - ] + [_reduced_field_api_sap_window(w) for w in schema.sap_windows] if schema.sap_windows else _synthesise_17_0_sap_windows(schema) ), @@ -1301,23 +1286,7 @@ class EpcPropertyDataMapper: cylinder_insulation_thickness_mm=schema.sap_heating.cylinder_insulation_thickness, ), sap_windows=( - [ - SapWindow( - frame_material=None, - glazing_gap=0, - orientation=w.orientation, - window_type=w.window_type, - glazing_type=w.glazing_type, - # ADR-0028: 14 rich certs lodge real per-window area. - window_width=_measurement_value(w.window_area), - window_height=1.0, - draught_proofed=False, - window_location=w.window_location, - window_wall_type=0, - permanent_shutters_present=False, - ) - for w in schema.sap_windows - ] + [_reduced_field_api_sap_window(w) for w in schema.sap_windows] if schema.sap_windows else _synthesise_17_1_sap_windows(schema) ), @@ -1508,26 +1477,7 @@ class EpcPropertyDataMapper: # glazed_area band + TFA (added in a later slice). The 10 rich # certs keep their lodged per-window geometry. sap_windows=( - [ - SapWindow( - frame_material=None, - glazing_gap=0, - orientation=w.orientation, - window_type=w.window_type, - glazing_type=w.glazing_type, - # ADR-0028: the 10 rich certs lodge a real per-window - # window_area (Measurement) -- use it directly as geometry - # (width = area, height = 1.0) rather than the placeholder - # windowless 0x0. - window_width=_measurement_value(w.window_area), - window_height=1.0, - draught_proofed=False, - window_location=w.window_location, - window_wall_type=0, - permanent_shutters_present=False, - ) - for w in schema.sap_windows - ] + [_reduced_field_api_sap_window(w) for w in schema.sap_windows] if schema.sap_windows else _synthesise_18_0_sap_windows(schema) ), @@ -1754,22 +1704,7 @@ class EpcPropertyDataMapper: # rich certs keep their lodged per-window geometry (used directly: # window_width = area, height = 1.0). sap_windows=( - [ - SapWindow( - frame_material=None, - glazing_gap=0, - orientation=w.orientation, - window_type=w.window_type, - glazing_type=w.glazing_type, - window_width=_measurement_value(w.window_area), - window_height=1.0, - draught_proofed=False, - window_location=w.window_location, - window_wall_type=0, - permanent_shutters_present=False, - ) - for w in schema.sap_windows - ] + [_reduced_field_api_sap_window(w) for w in schema.sap_windows] if schema.sap_windows else _synthesise_19_0_sap_windows(schema) ), @@ -2020,28 +1955,7 @@ class EpcPropertyDataMapper: # ADR-0027: 993/1000 omit sap_windows → synthesise from glazed_area # band + TFA. The 7 rich certs keep their lodged per-window geometry. sap_windows=( - [ - SapWindow( - frame_material=None, - glazing_gap=0, - orientation=w.orientation, - window_type=w.window_type, - glazing_type=w.glazing_type, - # ADR-0027: the 7 rich certs lodge a real per-window - # window_area (Measurement) — use it directly as - # geometry (width = area, height = 1.0, matching the - # synthesis convention) rather than the placeholder - # windowless 0x0 that modelled these data-richest certs - # as having no glazing at all. - window_width=_measurement_value(w.window_area), - window_height=1.0, - draught_proofed=False, - window_location=w.window_location, - window_wall_type=0, - permanent_shutters_present=False, - ) - for w in schema.sap_windows - ] + [_reduced_field_api_sap_window(w) for w in schema.sap_windows] if schema.sap_windows else _synthesise_20_0_0_sap_windows(schema) ), @@ -2289,7 +2203,11 @@ class EpcPropertyDataMapper: orientation=w.orientation, window_type=w.window_type, frame_factor=w.frame_factor, - glazing_type=w.glazing_type, + # See `_reduced_field_api_sap_window`: the raw API glazing + # code is the RdSAP-21 glazing enum, not the SAP 10.2 + # cascade enum the calculator's U/g-value tables are keyed + # on — must be canonicalised, not passed through raw. + glazing_type=_api_cascade_glazing_type(w.glazing_type), window_width=_measurement_value(w.window_width), window_height=_measurement_value(w.window_height), draught_proofed=w.draught_proofed == "true", @@ -5217,6 +5135,48 @@ def _api_sap_roof_window(w: Any) -> SapRoofWindow: ) +def _reduced_field_api_sap_window(w: Any) -> SapWindow: + """Build a `SapWindow` from one reduced-field API schema `sap_windows` + entry (RdSAP 17.0/17.1/18.0/19.0/20.0.0's minimal shape: orientation, + window_area, window_type, glazing_type, window_location — no glazing_gap, + frame_factor or window_transmission_details lodged). + + Mirrors `_api_sap_window`'s glazing-type cascade: the raw API + `glazing_type` must be canonicalised via `_api_cascade_glazing_type` + before storage — these codes are the RdSAP-21 glazing enum (e.g. code 1 + = "DG pre-2002"), NOT the SAP 10.2 Table 6b cascade enum the calculator's + `_G_LIGHT_BY_GLAZING_CODE`/U-value lookups are keyed on (cascade code 1 = + single glazed). Previously stored raw, this silently modelled any window + lodging API code 1 as single-glazed instead of double — cert + 0141-2860-6891-9124-5625 (uprn 100021969385, property 753950, RdSAP-16.2) + over-stated windows_w_per_k 82.27 vs Elmhurst's 51.44 (a 31 W/K gap, the + dominant single contributor to a SAP 42 vs Elmhurst 44 mismatch).""" + transmission = _api_glazing_transmission(w.glazing_type, 0) + return SapWindow( + frame_material=None, + glazing_gap=0, + orientation=w.orientation, + window_type=w.window_type, + glazing_type=_api_cascade_glazing_type(w.glazing_type), + window_width=_measurement_value(w.window_area), + window_height=1.0, + draught_proofed=False, + window_location=w.window_location, + window_wall_type=0, + permanent_shutters_present=False, + frame_factor=transmission[2] if transmission is not None else None, + window_transmission_details=( + WindowTransmissionDetails( + u_value=transmission[0], + data_source="SAP10 lookup (glazing_type, glazing_gap)", + solar_transmittance=transmission[1], + ) + if transmission is not None + else None + ), + ) + + def _api_sap_window(w: Any) -> SapWindow: """Build a `SapWindow` from one API schema sap_windows entry, routing the glazing-type + glazing-gap pair through the spec diff --git a/datatypes/epc/domain/tests/test_from_rdsap_schema.py b/datatypes/epc/domain/tests/test_from_rdsap_schema.py index f6ce46bfb..e6e8ed4ba 100644 --- a/datatypes/epc/domain/tests/test_from_rdsap_schema.py +++ b/datatypes/epc/domain/tests/test_from_rdsap_schema.py @@ -385,6 +385,25 @@ class TestFromRdSapSchema21_0_0: # photovoltaic_supply is None when the measured shape is present assert result.sap_energy_source.photovoltaic_supply is None + def test_lodged_window_glazing_type_is_cascaded_not_raw(self) -> None: + # Arrange — the fixture's own glazing_type (14) happens to be a + # coincident code (cascade(14) == 14), which would hide this bug. + # Override to code 1 ("DG pre-2002" in the RdSAP-21 glazing enum), + # which must canonicalise to cascade code 2 — passed through raw it + # would misread as cascade code 1 (single glazed). See the sibling + # `_reduced_field_api_sap_window` fix for the reduced-schema seams; + # this is the from_rdsap_schema_21_0_0 site of the same bug class. + data = load("21_0_0.json") + data["sap_windows"][0]["glazing_type"] = 1 + + schema = from_dict(RdSapSchema21_0_0, data) + + # Act + result = EpcPropertyDataMapper.from_rdsap_schema_21_0_0(schema) + + # Assert + assert result.sap_windows[0].glazing_type == 2 + def test_photovoltaic_array_orientation_nd_nulls_only_that_field(self) -> None: # Arrange — a 3-array dwelling where the middle array lodges the RdSAP # 'ND' ("Not Defined") sentinel for orientation. Regression for the @@ -2969,6 +2988,43 @@ def test_nd_glazing_without_single_signal_keeps_double_modal_default( assert all(w.glazing_type == 2 for w in result.sap_windows) +@pytest.mark.parametrize("schema_cls, mapper, fixture", _REDUCED_FIELD_SEAMS) +def test_lodged_sap_windows_glazing_type_is_cascaded_not_raw( + schema_cls: Any, mapper: Any, fixture: str +) -> None: + # Arrange — a real lodged `sap_windows[]` entry (the rich-cert path, not + # synthesis) whose raw API `glazing_type` code (1 = "DG pre-2002" in the + # RdSAP-21 glazing enum) differs from the SAP 10.2 cascade enum the + # calculator's U/g-value tables are keyed on (cascade code 1 = single + # glazed). Previously stored raw and unmodified — cert + # 0141-2860-6891-9124-5625 (uprn 100021969385, property 753950) lodges + # exactly this shape and was silently modelled as single glazed on its + # dominant window (91% of glazed area), overstating windows_w_per_k + # 82.27 vs Elmhurst's 51.44 W/K. + data = load(fixture) + data["sap_windows"] = [ + { + "orientation": 5, + "window_area": {"value": 18.66, "quantity": "square metres"}, + "window_type": 1, + "glazing_type": 1, + "window_location": 0, + } + ] + + # Act + result = mapper(from_dict(schema_cls, data)) + + # Assert — glazing_type is the CASCADED value (2), not the raw API code + # (1) that would misread as single glazed; transmission details/U-value + # are populated from the SAP10 lookup rather than left None. + assert result.sap_windows + window = result.sap_windows[0] + assert window.glazing_type == 2 + assert window.window_transmission_details is not None + assert window.window_transmission_details.u_value == pytest.approx(2.8) + + class TestReducedFieldNdGlazingResolver: """Unit-level pins for the shared ND resolver (schema-independent).""" From 26857f8df7944dffeb9e7e8ae32bf0f32975e823 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 13:56:24 +0000 Subject: [PATCH 50/55] Bulk download: match uploaded_files via hubspot_deal_id bridge + add worker observability A live test returned 'no documents could be packaged' for 91 correctly-resolved properties. Root cause: no upload source populates uploaded_files.landlord_property_id (pashub/magic-plan/audit set hubspot_deal_id; ECMK sets hubspot_listing_id), so matching on landlord_property_id found zero rows. Match fix: the worker now joins uploaded_files -> hubspot_deal_data (on deal_id) -> landlord_property_id, taking the property identity from the bridge. The repository returns a small PropertyDocument read-model instead of the infra ORM row, so the orchestrator no longer names infrastructure.postgres.* (resolves the leak dancafc flagged) and the s3_upload_timestamp cast is gone. Coverage is limited to deal-id-linked sources; listing_id/uprn-only files are a noted follow-up. Observability: the empty-selection failure now carries stage counts (selected/matched/planned/skipped) in both the message (-> the worker WARNING log) and details (-> sub_task.outputs), and the run logs those counts. The next failure says 'matched 0 documents' instead of failing opaquely. ADR-0060 + CONTEXT.md updated (matching decision + considered options). Co-Authored-By: Claude Opus 4.8 (1M context) --- CONTEXT.md | 2 +- ...60-bulk-document-download-package-model.md | 38 +++++++--- .../property_document.py | 24 ++++++ .../bulk_document_download_orchestrator.py | 76 ++++++++++++------- .../uploaded_file_postgres_repository.py | 44 ++++++++--- ...est_bulk_document_download_orchestrator.py | 62 ++++++++------- .../test_uploaded_file_postgres_repository.py | 49 +++++++++--- 7 files changed, 207 insertions(+), 88 deletions(-) create mode 100644 domain/bulk_document_download/property_document.py diff --git a/CONTEXT.md b/CONTEXT.md index 1204aa9d9..34d380249 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -108,7 +108,7 @@ The single ZIP archive a **Bulk Document Download** produces: one folder per pro _Avoid_: export (that is the plan/scenario XLSX export), bundle, archive (ambiguous), zip (the format, not the concept) **Bulk Document Download**: -The feature/job that assembles a **Download Package** for a chosen set of properties (the union of one or more whole HubSpot projects, selected by `project_code`, and a hand-picked `landlord_property_id` list) and emails the requester a link. Runs on the app-owned-task + attach-mode lane (ADR-0055): the FastAPI route resolves the selection to a distinct `landlord_property_id` set — project codes are expanded via `hubspot_deal_data` (which holds the project↔property grain), hand-picked ids are taken as given — pins that set and the recipient email onto the `sub_task`, and the `applications/bulk_document_download` Lambda builds the package, writes the URL to `sub_task.outputs`, and emails it (ADR-0059/0060). `landlord_property_id` is the selection key throughout because `uploaded_files` is matched on it (it has no `property_id`). Files are matched by `landlord_property_id` (the missing `property_id` on `uploaded_files` is a known future gap). Selection is capped by property count at the trigger. +The feature/job that assembles a **Download Package** for a chosen set of properties (the union of one or more whole HubSpot projects, selected by `project_code`, and a hand-picked `landlord_property_id` list) and emails the requester a link. Runs on the app-owned-task + attach-mode lane (ADR-0055): the FastAPI route resolves the selection to a distinct `landlord_property_id` set — project codes are expanded via `hubspot_deal_data` (which holds the project↔property grain), hand-picked ids are taken as given — pins that set and the recipient email onto the `sub_task`, and the `applications/bulk_document_download` Lambda builds the package, writes the URL to `sub_task.outputs`, and emails it (ADR-0059/0060). The Lambda matches files by `hubspot_deal_id`, bridging each selected `landlord_property_id` through `hubspot_deal_data` (`uploaded_files.landlord_property_id` is unpopulated in practice, so the property identity comes from the deal bridge); coverage is limited to deal-id-linked sources. Selection is capped by property count at the trigger. _Avoid_: document export, bulk export, file dump ### Source data diff --git a/docs/adr/0060-bulk-document-download-package-model.md b/docs/adr/0060-bulk-document-download-package-model.md index 2eedb6d9f..886d7a4a1 100644 --- a/docs/adr/0060-bulk-document-download-package-model.md +++ b/docs/adr/0060-bulk-document-download-package-model.md @@ -55,11 +55,17 @@ lane (ADR-0055).** query — cheap, no S3 on the hot path). A `/tmp` size budget inside the Lambda is a backstop that fails the sub_task with a clear reason if a pathological selection still overflows. `N` starts at a conservative, tunable value. -- **Matching on `landlord_property_id`.** Files are gathered by - `landlord_property_id`; the missing `property_id` on `uploaded_files` is a - **known future gap**, out of scope here. Property display info (the folder - name) is enriched from the hubspot deals data - (`repositories/hubspot_deals/`). +- **Matching via `hubspot_deal_id`, bridged through `hubspot_deal_data`.** The + selection is a set of `landlord_property_id`s, but `uploaded_files` is matched + on **`hubspot_deal_id`**: in practice no upload source populates + `uploaded_files.landlord_property_id` (pashub, magic plan and the audit + generator set `hubspot_deal_id`; ECMK sets `hubspot_listing_id`), so the + worker joins `uploaded_files → hubspot_deal_data (on deal_id) → landlord_property_id` + and takes the property identity from the bridge, not the file row. **Coverage + limitation:** files linked *only* by `hubspot_listing_id` (ECMK) or `uprn` are + not reached; extending the bridge to those keys is a follow-up. Property + display info (the folder name) is enriched from the property record + (`repositories/property/`). - **Layout.** One folder per property named by human-readable **address** (hubspot-enriched), with `landlord_property_id` appended for uniqueness; inside, one file per Document Type = the newest by `s3_upload_timestamp`. @@ -94,16 +100,21 @@ lane (ADR-0055).** - **Strict "all-or-nothing" packaging.** Rejected: one unreadable row or a property with no documents would deny the entire package; best-effort + reporting matches "include documents and properties where they exist". -- **Match on `uprn`.** Deferred: `landlord_property_id` is the agreed key and - hubspot enrichment is keyed to the deal/property; revisit if/when the - `property_id` gap on `uploaded_files` is closed. +- **Match on `landlord_property_id` directly.** Rejected once tested against + real data: `uploaded_files.landlord_property_id` is unpopulated, so a direct + match returns nothing. `hubspot_deal_id` (bridged via `hubspot_deal_data`) is + the key that upload sources actually set. +- **Match on `uprn` / `hubspot_listing_id` as well.** Deferred: `hubspot_deal_id` + covers the bulk of sources; add `uprn`/`listing_id` to the bridge if a source + keyed only by those needs including. ## Consequences - New DDD pieces: `applications/bulk_document_download/` (thin handler + trigger body), `orchestration/bulk_document_download_orchestrator.py`, - packaging rules in `domain/`, a `landlord_property_id`-keyed - "latest-per-Document-Type" query on the uploaded-file repository, a + packaging rules in `domain/`, a `hubspot_deal_id`-bridged query on the + uploaded-file repository that returns a small `PropertyDocument` read-model + (so the orchestrator never names `infrastructure.postgres.*`), a `generate_presigned_url` + multipart upload on `S3Client`, and the email port/adapter from ADR-0059. The only `backend/` touch is the trigger route. - The stored, pinned `property_id` set makes a run reproducible even if the @@ -115,5 +126,8 @@ lane (ADR-0055).** - A **new S3 bucket** (`DOCUMENT_EXPORTS_BUCKET`) is provisioned for the packages, with its own IAM (single-writer, presign-read). Its retention policy is a deliberate follow-up, not set here. -- Closing the `uploaded_files.property_id` gap later would let matching move off - `landlord_property_id` without changing the package model. +- Because matching bridges through `hubspot_deal_data`, a file is only reachable + once its deal has a `hubspot_deal_data` row (and the file carries that + `hubspot_deal_id`). Populating `uploaded_files.landlord_property_id` at upload + time, or closing the `uploaded_files.property_id` gap, would let matching drop + the bridge without changing the package model. diff --git a/domain/bulk_document_download/property_document.py b/domain/bulk_document_download/property_document.py new file mode 100644 index 000000000..624a48504 --- /dev/null +++ b/domain/bulk_document_download/property_document.py @@ -0,0 +1,24 @@ +"""The repository's read-model for a property's uploaded document (ADR-0060). + +A small domain type the uploaded-file repository hands back, so the orchestrator +never names ``infrastructure.postgres.*``. Carries the property's +``landlord_property_id`` (resolved through the ``hubspot_deal_data`` bridge — see +the repository) alongside just the fields the Download Package needs. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +from typing import Optional + + +@dataclass(frozen=True) +class PropertyDocument: + """One uploaded file matched to a property, ready to resolve into the plan.""" + + landlord_property_id: str + document_type: Optional[str] + s3_bucket: str + s3_key: str + uploaded_at: datetime diff --git a/orchestration/bulk_document_download_orchestrator.py b/orchestration/bulk_document_download_orchestrator.py index c3c0db2cd..c67ee0372 100644 --- a/orchestration/bulk_document_download_orchestrator.py +++ b/orchestration/bulk_document_download_orchestrator.py @@ -10,28 +10,30 @@ rows are reported, not fatal; a wholly-empty selection is a recorded failure from __future__ import annotations +import logging import os import tempfile import zipfile from dataclasses import dataclass -from datetime import datetime -from typing import Optional, Protocol, Sequence, cast +from typing import Protocol, Sequence from domain.bulk_document_download.package_plan import ( ResolvedDocument, SkippedDocument, plan_download_package, ) +from domain.bulk_document_download.property_document import PropertyDocument from domain.tasks.subtasks import SubTaskFailure -from infrastructure.postgres.uploaded_file_table import UploadedFile from infrastructure.s3.s3_client import S3Client from repositories.email.email_sender import EmailSender +logger = logging.getLogger(__name__) + class UploadedFileReader(Protocol): def by_landlord_property_ids( self, landlord_property_ids: Sequence[str] - ) -> list[UploadedFile]: ... + ) -> list[PropertyDocument]: ... class AddressResolver(Protocol): @@ -79,14 +81,28 @@ class BulkDocumentDownloadOrchestrator: recipient_email: str, package_name: str, ) -> DownloadPackageResult: - files = self._uploaded_files.by_landlord_property_ids(landlord_property_ids) - resolved = [self._resolve(file) for file in files] - plan = plan_download_package([r for r in resolved if r is not None]) + documents = self._uploaded_files.by_landlord_property_ids( + landlord_property_ids + ) + plan = plan_download_package([self._resolve(doc) for doc in documents]) skipped: list[SkippedDocument] = list(plan.skipped) + logger.info( + "bulk_document_download: selected=%d matched_documents=%d " + "planned_entries=%d skipped=%d", + len(landlord_property_ids), + len(documents), + len(plan.entries), + len(skipped), + ) + if not plan.entries: - raise self._empty_failure(skipped) + raise self._empty_failure( + selected=len(landlord_property_ids), + matched=len(documents), + skipped=skipped, + ) # Stream to a /tmp file and multipart-upload it (ADR-0060): a Download # Package can be several GB, so it is never held whole in memory. Each @@ -120,7 +136,11 @@ class BulkDocumentDownloadOrchestrator: if included == 0: # Every file that resolved failed to read — no package to send. - raise self._empty_failure(skipped) + raise self._empty_failure( + selected=len(landlord_property_ids), + matched=len(documents), + skipped=skipped, + ) self._packages.upload_file(zip_path, key) url = self._packages.generate_presigned_url(key, self._url_ttl_seconds) @@ -141,13 +161,21 @@ class BulkDocumentDownloadOrchestrator: skipped=tuple(skipped), ) - def _empty_failure(self, skipped: list[SkippedDocument]) -> SubTaskFailure: + def _empty_failure( + self, *, selected: int, matched: int, skipped: list[SkippedDocument] + ) -> SubTaskFailure: """A recorded failure when the selection yields no readable documents - (ADR-0060) — never an empty ZIP emailed to the user. The skip detail - rides on the failure.""" + (ADR-0060) — never an empty ZIP emailed to the user. The message carries + the stage counts (so they surface in the worker's WARNING log) and the + skip detail rides on the failure's ``details`` (→ ``sub_task.outputs``).""" return SubTaskFailure( - "no documents could be packaged for the selection", + f"no documents could be packaged for the selection " + f"(selected {selected} properties, matched {matched} documents, " + f"{len(skipped)} skipped)", details={ + "selected_properties": selected, + "matched_documents": matched, + "planned_entries": 0, "skipped": [ { "landlord_property_id": s.landlord_property_id, @@ -155,22 +183,16 @@ class BulkDocumentDownloadOrchestrator: "reason": s.reason, } for s in skipped - ] + ], }, ) - def _resolve(self, file: UploadedFile) -> Optional[ResolvedDocument]: - if file.landlord_property_id is None: - return None + def _resolve(self, document: PropertyDocument) -> ResolvedDocument: return ResolvedDocument( - landlord_property_id=file.landlord_property_id, - address=self._addresses.address_for(file.landlord_property_id), - document_type=file.file_type, - s3_bucket=file.s3_file_bucket, - s3_key=file.s3_file_key, - # TODO: the cast is only needed because UploadedFile types - # s3_upload_timestamp as `object` (though it is NOT NULL datetime in - # the DB). Type that column `datetime` on the table so callers don't - # each cast — see PR #1498 review. - uploaded_at=cast(datetime, file.s3_upload_timestamp), + landlord_property_id=document.landlord_property_id, + address=self._addresses.address_for(document.landlord_property_id), + document_type=document.document_type, + s3_bucket=document.s3_bucket, + s3_key=document.s3_key, + uploaded_at=document.uploaded_at, ) diff --git a/repositories/uploaded_file/uploaded_file_postgres_repository.py b/repositories/uploaded_file/uploaded_file_postgres_repository.py index 82c711231..26e2684c1 100644 --- a/repositories/uploaded_file/uploaded_file_postgres_repository.py +++ b/repositories/uploaded_file/uploaded_file_postgres_repository.py @@ -2,9 +2,10 @@ from __future__ import annotations from typing import Optional, Sequence -from sqlalchemy import select +from sqlalchemy import select, text from sqlmodel import Session, col +from domain.bulk_document_download.property_document import PropertyDocument from infrastructure.postgres.uploaded_file_table import FileTypeEnum, UploadedFile @@ -26,15 +27,40 @@ class UploadedFilePostgresRepository: def by_landlord_property_ids( self, landlord_property_ids: Sequence[str] - ) -> list[UploadedFile]: - """Every uploaded file held for the given properties, matched by - `landlord_property_id` (ADR-0060), across all Document Types. Returns - the raw rows — latest-per-Document-Type selection and null-type - skipping are the Download Package plan's job, not the query's.""" - stmt = select(UploadedFile).where( - col(UploadedFile.landlord_property_id).in_(list(landlord_property_ids)) + ) -> list[PropertyDocument]: + """Every uploaded file held for the given properties (ADR-0060), across + all Document Types, as a small domain read-model. + + Files are matched by **`hubspot_deal_id`**, bridged to the property + through `hubspot_deal_data` — `uploaded_files.landlord_property_id` is + unpopulated in practice (no upload source sets it), so the property + identity comes from the deal bridge, not the file row. Coverage is + therefore limited to deal-id-linked sources (pashub, magic plan, audit + generator); files keyed only by `hubspot_listing_id` or `uprn` are not + reached — see the ADR-0060 matching decision. + + Latest-per-Document-Type selection and null-type skipping are the + Download Package plan's job, not the query's.""" + rows = self._session.connection().execute( + text( + "SELECT h.landlord_property_id, u.file_type," + " u.s3_file_bucket, u.s3_file_key, u.s3_upload_timestamp" + " FROM uploaded_files u" + " JOIN hubspot_deal_data h ON h.deal_id = u.hubspot_deal_id" + " WHERE h.landlord_property_id = ANY(:lpids)" + ), + {"lpids": list(landlord_property_ids)}, ) - return list(self._session.execute(stmt).scalars().all()) # pyright: ignore[reportDeprecated] + return [ + PropertyDocument( + landlord_property_id=row[0], + document_type=row[1], + s3_bucket=row[2], + s3_key=row[3], + uploaded_at=row[4], + ) + for row in rows.all() + ] def insert(self, uploaded_file: UploadedFile) -> None: self._session.add(uploaded_file) diff --git a/tests/orchestration/test_bulk_document_download_orchestrator.py b/tests/orchestration/test_bulk_document_download_orchestrator.py index d9b3a0d23..47a071c38 100644 --- a/tests/orchestration/test_bulk_document_download_orchestrator.py +++ b/tests/orchestration/test_bulk_document_download_orchestrator.py @@ -7,12 +7,14 @@ import io import zipfile from collections.abc import Iterator from datetime import datetime, timezone +from typing import Optional import pytest from moto import mock_aws +from domain.bulk_document_download.property_document import PropertyDocument from domain.tasks.subtasks import SubTaskFailure -from infrastructure.postgres.uploaded_file_table import FileTypeEnum, UploadedFile +from infrastructure.postgres.uploaded_file_table import FileTypeEnum from infrastructure.s3.s3_client import S3Client from orchestration.bulk_document_download_orchestrator import ( BulkDocumentDownloadOrchestrator, @@ -23,14 +25,14 @@ DATA_BUCKET = "data-bucket" class _FakeUploadedFiles: - def __init__(self, files: list[UploadedFile]) -> None: - self._files = files + def __init__(self, documents: list[PropertyDocument]) -> None: + self._documents = documents def by_landlord_property_ids( self, landlord_property_ids: object - ) -> list[UploadedFile]: + ) -> list[PropertyDocument]: wanted = set(landlord_property_ids) # type: ignore[call-overload] - return [f for f in self._files if f.landlord_property_id in wanted] + return [d for d in self._documents if d.landlord_property_id in wanted] class _FakeAddresses: @@ -60,18 +62,18 @@ class _RecordingEmail: self.sent.append((to, subject, body)) -def _uploaded_file( +def _document( landlord_property_id: str, - s3_file_bucket: str, - s3_file_key: str, - file_type: FileTypeEnum = FileTypeEnum.SITE_NOTE, -) -> UploadedFile: - return UploadedFile( - s3_file_bucket=s3_file_bucket, - s3_file_key=s3_file_key, - s3_upload_timestamp=datetime(2026, 1, 1, tzinfo=timezone.utc), + s3_bucket: str, + s3_key: str, + file_type: Optional[FileTypeEnum] = FileTypeEnum.SITE_NOTE, +) -> PropertyDocument: + return PropertyDocument( landlord_property_id=landlord_property_id, - file_type=file_type.value, + document_type=file_type.value if file_type is not None else None, + s3_bucket=s3_bucket, + s3_key=s3_key, + uploaded_at=datetime(2026, 1, 1, tzinfo=timezone.utc), ) @@ -85,7 +87,7 @@ def packages() -> Iterator[S3Client]: def test_builds_uploads_and_emails_a_download_package(packages: S3Client) -> None: # Arrange — one property with one site note. - files = [_uploaded_file("LP1", "src-bucket", "surveys/a.pdf")] + files = [_document("LP1", "src-bucket", "surveys/a.pdf")] email = _RecordingEmail() orchestrator = BulkDocumentDownloadOrchestrator( uploaded_files=_FakeUploadedFiles(files), @@ -133,27 +135,29 @@ def test_a_selection_with_no_documents_is_a_recorded_failure( # Act / Assert — a recorded SubTask failure, and nobody is emailed an empty # package. - with pytest.raises(SubTaskFailure): + with pytest.raises(SubTaskFailure) as exc_info: orchestrator.run( - landlord_property_ids=["LP1"], + landlord_property_ids=["LP1", "LP2"], recipient_email="user@example.com", package_name="empty", ) assert email.sent == [] + # ...and it carries the stage counts so the failure is debuggable: the + # message (→ the worker's WARNING log) and details (→ sub_task.outputs). + failure = exc_info.value + assert "matched 0 documents" in str(failure) + assert failure.details is not None + assert failure.details["selected_properties"] == 2 + assert failure.details["matched_documents"] == 0 + def test_packages_the_good_documents_and_reports_the_skipped_ones( packages: S3Client, ) -> None: # Arrange — one packable site note and one row with no Document Type. - good = _uploaded_file("LP1", "src-bucket", "surveys/a.pdf") - untyped = UploadedFile( - s3_file_bucket="src-bucket", - s3_file_key="mystery.bin", - s3_upload_timestamp=datetime(2026, 1, 1, tzinfo=timezone.utc), - landlord_property_id="LP1", - file_type=None, - ) + good = _document("LP1", "src-bucket", "surveys/a.pdf") + untyped = _document("LP1", "src-bucket", "mystery.bin", file_type=None) email = _RecordingEmail() orchestrator = BulkDocumentDownloadOrchestrator( uploaded_files=_FakeUploadedFiles([good, untyped]), @@ -185,8 +189,8 @@ def test_a_file_that_cannot_be_read_is_skipped_not_fatal(packages: S3Client) -> # Arrange — two documents for one property; the photo pack's object is # missing (download raises), the site note reads fine. Best-effort (ADR-0060): # the run must still deliver the readable file and report the other. - good = _uploaded_file("LP1", "src-bucket", "surveys/a.pdf") - missing = _uploaded_file( + good = _document("LP1", "src-bucket", "surveys/a.pdf") + missing = _document( "LP1", "src-bucket", "surveys/gone.pdf", file_type=FileTypeEnum.PHOTO_PACK ) email = _RecordingEmail() @@ -221,7 +225,7 @@ def test_the_address_fallback_string_becomes_the_folder_name( packages: S3Client, ) -> None: # Arrange — a property whose address resolved to the fallback string. - doc = _uploaded_file("LP9", "src-bucket", "surveys/a.pdf") + doc = _document("LP9", "src-bucket", "surveys/a.pdf") orchestrator = BulkDocumentDownloadOrchestrator( uploaded_files=_FakeUploadedFiles([doc]), addresses=_FakeAddresses({"LP9": "address unavailable"}), diff --git a/tests/repositories/uploaded_file/test_uploaded_file_postgres_repository.py b/tests/repositories/uploaded_file/test_uploaded_file_postgres_repository.py index 40fc1773f..76ac9b758 100644 --- a/tests/repositories/uploaded_file/test_uploaded_file_postgres_repository.py +++ b/tests/repositories/uploaded_file/test_uploaded_file_postgres_repository.py @@ -5,6 +5,7 @@ from datetime import datetime, timedelta, timezone from sqlalchemy import Engine from sqlmodel import Session +from backend.app.db.models.hubspot_deal_data import HubspotDealData from infrastructure.postgres.uploaded_file_table import FileTypeEnum, UploadedFile from repositories.uploaded_file.uploaded_file_postgres_repository import ( UploadedFilePostgresRepository, @@ -80,28 +81,38 @@ def test_does_not_return_row_with_different_file_type(db_engine: Engine) -> None assert result is None -def _landlord_file( - landlord_property_id: str, +def _deal_file( + hubspot_deal_id: str, s3_file_key: str, file_type: FileTypeEnum = FileTypeEnum.SITE_NOTE, ) -> UploadedFile: + # A real uploaded file: keyed by hubspot_deal_id, landlord_property_id NULL + # (no upload source populates it — see ADR-0060 matching decision). return UploadedFile( s3_file_bucket=_BUCKET, s3_file_key=s3_file_key, s3_upload_timestamp=datetime(2026, 1, 1, 12, 0, 0, tzinfo=timezone.utc), - landlord_property_id=landlord_property_id, + hubspot_deal_id=hubspot_deal_id, file_type=file_type.value, ) -def test_by_landlord_property_ids_returns_only_requested_properties( +def _deal(landlord_property_id: str, deal_id: str) -> HubspotDealData: + return HubspotDealData(deal_id=deal_id, landlord_property_id=landlord_property_id) + + +def test_by_landlord_property_ids_bridges_deal_id_to_the_property( db_engine: Engine, ) -> None: - # Arrange — files for three properties; only two are requested. + # Arrange — files are keyed by hubspot_deal_id; hubspot_deal_data bridges each + # deal to a landlord_property_id. Three properties exist; two are requested. with Session(db_engine) as session: - session.add(_landlord_file("LP1", "one.pdf")) - session.add(_landlord_file("LP2", "two.pdf")) - session.add(_landlord_file("LP3", "three.pdf")) + session.add(_deal("LP1", "deal-1")) + session.add(_deal("LP2", "deal-2")) + session.add(_deal("LP3", "deal-3")) + session.add(_deal_file("deal-1", "one.pdf")) + session.add(_deal_file("deal-2", "two.pdf")) + session.add(_deal_file("deal-3", "three.pdf")) session.commit() # Act @@ -109,7 +120,25 @@ def test_by_landlord_property_ids_returns_only_requested_properties( found = UploadedFilePostgresRepository(session).by_landlord_property_ids( ["LP1", "LP2"] ) - result = {(f.landlord_property_id, f.s3_file_key) for f in found} + result = {(d.landlord_property_id, d.s3_key) for d in found} + + # Assert — the file's property comes from the bridge, keyed by the deal. + assert result == {("LP1", "one.pdf"), ("LP2", "two.pdf")} + + +def test_by_landlord_property_ids_matches_nothing_without_a_deal_bridge( + db_engine: Engine, +) -> None: + # Arrange — a file whose deal has no hubspot_deal_data row can't be reached. + with Session(db_engine) as session: + session.add(_deal_file("orphan-deal", "orphan.pdf")) + session.commit() + + # Act + with Session(db_engine) as session: + found = UploadedFilePostgresRepository(session).by_landlord_property_ids( + ["LP1"] + ) # Assert - assert result == {("LP1", "one.pdf"), ("LP2", "two.pdf")} + assert found == [] From c9e0180c12e7682f004a202050d3a609b48f40d6 Mon Sep 17 00:00:00 2001 From: Jun-te Kim Date: Wed, 8 Jul 2026 14:25:34 +0000 Subject: [PATCH 51/55] Widen EPC-prediction cohort search one step when the normal radius is too sparse MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-0034's nearby-postcode broadening degrades to a genuine NoSameTypeComparablesError whenever the default 1000m/30-postcode reach has no same-type comparable nearby (e.g. property_id=752685, portfolio 824 — the only Maisonette within reach). Adds one configurable extra widening step (EpcComparablePropertiesRepository.candidates_near now accepts widen_nearby_postcodes), tried only when the normal-radius walk falls short of `minimum` matches. modelling_e2e's handler wires this to a 3000m/60-postcode PostcodesIoClient as the single wider step. Co-Authored-By: Claude Sonnet 5 --- applications/modelling_e2e/handler.py | 8 +- .../epc_comparable_properties_repository.py | 50 ++++++++++-- ...st_epc_comparable_properties_repository.py | 77 +++++++++++++++++++ 3 files changed, 127 insertions(+), 8 deletions(-) diff --git a/applications/modelling_e2e/handler.py b/applications/modelling_e2e/handler.py index 51c3e8149..269411b63 100644 --- a/applications/modelling_e2e/handler.py +++ b/applications/modelling_e2e/handler.py @@ -471,7 +471,13 @@ def handler( overrides_reader = InMemoryPropertyOverridesReader(overrides_by_pid) prediction_attrs_reader = OverrideBackedPredictionAttributesReader(overrides_reader) comparables_repo = EpcComparablePropertiesRepository( - epc_client, geospatial, nearby_postcodes=PostcodesIoClient() + epc_client, + geospatial, + nearby_postcodes=PostcodesIoClient(), + # One step wider than the default 1000m/30 reach, tried only when that + # normal radius doesn't surface enough same-type comparables (ADR-0034 + # amendment) — a single extra step outward, not an unbounded expansion. + widen_nearby_postcodes=PostcodesIoClient(radius_m=3000, limit=60), ) predictor = EpcPrediction() diff --git a/repositories/comparable_properties/epc_comparable_properties_repository.py b/repositories/comparable_properties/epc_comparable_properties_repository.py index 20c51dfda..68328f395 100644 --- a/repositories/comparable_properties/epc_comparable_properties_repository.py +++ b/repositories/comparable_properties/epc_comparable_properties_repository.py @@ -75,10 +75,17 @@ class EpcComparablePropertiesRepository(ComparablePropertiesRepository): epc_client: CohortEpcClient, geospatial: CohortGeospatial, nearby_postcodes: Optional[NearbyPostcodes] = None, + widen_nearby_postcodes: Optional[NearbyPostcodes] = None, ) -> None: self._epc_client = epc_client self._geospatial = geospatial self._nearby_postcodes = nearby_postcodes + # A second, wider-reaching `NearbyPostcodes` source, consulted only when + # the normal-radius walk fails to reach `minimum` same-type matches — one + # extra step outward before genuinely giving up (ADR-0034 amendment). A + # single configured step today; the same pattern chains to further steps + # later without changing this class's contract. + self._widen_nearby_postcodes = widen_nearby_postcodes # Cohort certs skipped because they are not yet mappable. Accumulates # across every postcode the instance serves; the caller reads it after # the run to report the mapper gaps (see modelling_e2e handler). @@ -117,12 +124,41 @@ class EpcComparablePropertiesRepository(ComparablePropertiesRepository): remaining, further-away postcodes are not fetched, so a dense area resolves in one or two searches instead of the whole radius. Without a configured ``NearbyPostcodes`` source this degrades to the seed postcode - alone.""" - postcodes = ( - self._nearby_postcodes.nearby(postcode, coordinates) - if self._nearby_postcodes is not None - else [postcode] - ) + alone. + + If the normal-radius walk still falls short of ``minimum`` matches and a + ``widen_nearby_postcodes`` source is configured, the walk is retried + once against that wider source before giving up — a single extra step + outward, not an unbounded expansion.""" + postcodes = self._nearby(self._nearby_postcodes, postcode, coordinates) + candidates, matches = self._walk(postcodes, enough, minimum) + + if ( + enough is not None + and matches < minimum + and self._widen_nearby_postcodes is not None + ): + wider_postcodes = self._nearby( + self._widen_nearby_postcodes, postcode, coordinates + ) + candidates, _ = self._walk(wider_postcodes, enough, minimum) + + return candidates + + def _nearby( + self, + source: Optional[NearbyPostcodes], + postcode: str, + coordinates: Optional[Coordinates], + ) -> list[str]: + return source.nearby(postcode, coordinates) if source is not None else [postcode] + + def _walk( + self, + postcodes: list[str], + enough: Optional[Callable[[ComparableProperty], bool]], + minimum: int, + ) -> tuple[list[ComparableProperty], int]: candidates: list[ComparableProperty] = [] seen_certs: set[str] = set() matches = 0 @@ -136,7 +172,7 @@ class EpcComparablePropertiesRepository(ComparablePropertiesRepository): matches += 1 if enough is not None and matches >= minimum: break - return candidates + return candidates, matches # Mapper-shape errors: the cert's lodged data does not fit the schema/mapper. # `ValueError` — missing required field / unmapped code (`UnmappedApiCode`); diff --git a/tests/repositories/comparable_properties/test_epc_comparable_properties_repository.py b/tests/repositories/comparable_properties/test_epc_comparable_properties_repository.py index 1a0b918ea..5b99bbd41 100644 --- a/tests/repositories/comparable_properties/test_epc_comparable_properties_repository.py +++ b/tests/repositories/comparable_properties/test_epc_comparable_properties_repository.py @@ -245,6 +245,83 @@ def test_candidates_near_without_a_source_uses_only_the_seed() -> None: assert [c.certificate_number for c in candidates] == ["CERT-1"] +def test_candidates_near_widens_once_when_the_normal_reach_is_insufficient() -> None: + # Arrange — the normal-radius nearby set (P0, P1) has no MATCH cert; the + # wider-radius set (P0, P1, P2, P3) reaches P3, which does. + client = _MultiPostcodeEpcClient( + { + "P0": [_result("OTHER-1", uprn=1)], + "P1": [_result("OTHER-2", uprn=2)], + "P2": [_result("OTHER-3", uprn=3)], + "P3": [_result("MATCH-1", uprn=4)], + } + ) + normal = _FakeNearbyPostcodes(["P0", "P1"]) + wider = _FakeNearbyPostcodes(["P0", "P1", "P2", "P3"]) + repo = EpcComparablePropertiesRepository( + client, + _FakeGeospatial({}), + nearby_postcodes=normal, + widen_nearby_postcodes=wider, + ) + + # Act + candidates = repo.candidates_near( + "P0", None, enough=lambda c: c.certificate_number.startswith("MATCH"), minimum=1 + ) + + # Assert — both the normal and the widened source were consulted (in that + # order), and the wider walk's matches are returned. + assert normal.calls == [("P0", None)] + assert wider.calls == [("P0", None)] + certs = {c.certificate_number for c in candidates} + assert "MATCH-1" in certs + + +def test_candidates_near_does_not_widen_when_the_normal_reach_is_enough() -> None: + # Arrange — the normal radius already yields the minimum match, so the + # wider (more expensive) source must never be consulted. + client = _MultiPostcodeEpcClient({"P0": [_result("MATCH-1", uprn=1)]}) + normal = _FakeNearbyPostcodes(["P0"]) + wider = _FakeNearbyPostcodes(["P0", "P1", "P2"]) + repo = EpcComparablePropertiesRepository( + client, + _FakeGeospatial({}), + nearby_postcodes=normal, + widen_nearby_postcodes=wider, + ) + + # Act + candidates = repo.candidates_near( + "P0", None, enough=lambda c: c.certificate_number.startswith("MATCH"), minimum=1 + ) + + # Assert + assert wider.calls == [] + assert [c.certificate_number for c in candidates] == ["MATCH-1"] + + +def test_candidates_near_without_enough_predicate_never_widens() -> None: + # Arrange — no `enough` predicate means the caller isn't gating on a match + # count (e.g. the unconditional aggregate path), so widening is meaningless + # and must not fire even with a configured wider source. + client = _MultiPostcodeEpcClient({"P0": [_result("CERT-1", uprn=1)]}) + normal = _FakeNearbyPostcodes(["P0"]) + wider = _FakeNearbyPostcodes(["P0", "P1"]) + repo = EpcComparablePropertiesRepository( + client, + _FakeGeospatial({}), + nearby_postcodes=normal, + widen_nearby_postcodes=wider, + ) + + # Act + repo.candidates_near("P0", None) + + # Assert + assert wider.calls == [] + + # --------------------------------------------------------------------------- # Unmappable cohort certs are skipped + recorded, not allowed to sink the cohort # --------------------------------------------------------------------------- From b8316418e14ab044ab3fafe50c78e2884e4237e1 Mon Sep 17 00:00:00 2001 From: Jun-te Kim Date: Wed, 8 Jul 2026 14:45:03 +0000 Subject: [PATCH 52/55] Add planning-authority HubSpot deal properties to ETL sync Sync Planning Authority, Designated Area, Article 4 PD Rights, and Listed Building from HubSpot into hubspot_deal_data so downstream consumers of the existing HubSpot ETL process can read them. Co-Authored-By: Claude Sonnet 5 --- backend/app/db/models/hubspot_deal_data.py | 5 ++ etl/hubspot/hubspotDataTodB.py | 8 ++++ etl/hubspot/hubspot_deal_differ.py | 4 ++ etl/hubspot/tests/test_hubspot_data_to_db.py | 48 ++++++++++++++++++++ 4 files changed, 65 insertions(+) diff --git a/backend/app/db/models/hubspot_deal_data.py b/backend/app/db/models/hubspot_deal_data.py index 88bb5a69f..d5db4c20a 100644 --- a/backend/app/db/models/hubspot_deal_data.py +++ b/backend/app/db/models/hubspot_deal_data.py @@ -93,6 +93,11 @@ class HubspotDealData(SQLModel, table=True): last_outbound_email: Optional[datetime] = Field(default=None) last_submission_date: Optional[datetime] = Field(default=None) + planning_authority: Optional[str] = Field(default=None) + designated_area: Optional[str] = Field(default=None) + article_4_pd_rights: Optional[str] = Field(default=None) + listed_building: Optional[str] = Field(default=None) + created_at: Optional[datetime] = Field( sa_column=Column( DateTime(timezone=True), diff --git a/etl/hubspot/hubspotDataTodB.py b/etl/hubspot/hubspotDataTodB.py index 54c174c3f..82e50747d 100644 --- a/etl/hubspot/hubspotDataTodB.py +++ b/etl/hubspot/hubspotDataTodB.py @@ -302,6 +302,10 @@ class HubspotDataToDb: "last_submission_date": parse_hs_date( deal_data.get("last_submission_date") ), + "planning_authority": deal_data.get("planning_authority"), + "designated_area": deal_data.get("designated_area"), + "article_4_pd_rights": deal_data.get("article_pd_rights"), + "listed_building": deal_data.get("listed_building"), }.items(): setattr(existing, attr, value) @@ -407,6 +411,10 @@ class HubspotDataToDb: last_outbound_call=parse_hs_date(deal_data.get("last_outbound_call")), last_outbound_email=parse_hs_date(deal_data.get("last_outbound_email")), last_submission_date=parse_hs_date(deal_data.get("last_submission_date")), + planning_authority=deal_data.get("planning_authority"), + designated_area=deal_data.get("designated_area"), + article_4_pd_rights=deal_data.get("article_pd_rights"), + listed_building=deal_data.get("listed_building"), ) def _handle_existing_photo_upload( diff --git a/etl/hubspot/hubspot_deal_differ.py b/etl/hubspot/hubspot_deal_differ.py index 22d7fb83c..2aa2c3bf1 100644 --- a/etl/hubspot/hubspot_deal_differ.py +++ b/etl/hubspot/hubspot_deal_differ.py @@ -109,6 +109,10 @@ class HubspotDealDiffer: "measures_for_pibi_ordered": "measures_for_pibi_ordered", "property_halted_reason": "property_halted_reason", "technical_approved_measures_for_install": "technical_approved_measures_for_install", + "planning_authority": "planning_authority", + "designated_area": "designated_area", + "article_pd_rights": "article_4_pd_rights", + "listed_building": "listed_building", } for hs_field, db_field in FIELD_MAP.items(): diff --git a/etl/hubspot/tests/test_hubspot_data_to_db.py b/etl/hubspot/tests/test_hubspot_data_to_db.py index 8499ccd08..7d1693040 100644 --- a/etl/hubspot/tests/test_hubspot_data_to_db.py +++ b/etl/hubspot/tests/test_hubspot_data_to_db.py @@ -127,3 +127,51 @@ def test_build_new_deal__client_booking_reference_maps_to_job_no() -> None: ) assert new_deal.client_booking_reference == "AD0226519" + + +def test_build_new_deal__planning_fields_mapped() -> None: + new_deal = _make_instance()._build_new_deal( + deal_id="MOCK_DEAL_ID", + deal_data={ + "planning_authority": "Test Council", + "designated_area": "Conservation Area", + "article_pd_rights": "Removed", + "listed_building": "Grade II", + }, + listing=None, + company=None, + project=None, + ) + + assert new_deal.planning_authority == "Test Council" + assert new_deal.designated_area == "Conservation Area" + assert new_deal.article_4_pd_rights == "Removed" + assert new_deal.listed_building == "Grade II" + + +def test_update_existing_deal__planning_fields_overwritten() -> None: + existing = HubspotDealData( + deal_id="MOCK_DEAL_ID", + planning_authority="Old Council", + designated_area="Old Area", + article_4_pd_rights="Old Rights", + listed_building="Old Listing", + ) + deal_data = { + "planning_authority": "New Council", + "designated_area": "New Area", + "article_pd_rights": "New Rights", + "listed_building": "New Listing", + } + + _make_instance()._update_existing_deal( + existing=existing, + deal_data=deal_data, + listing=None, + company=None, + ) + + assert existing.planning_authority == "New Council" + assert existing.designated_area == "New Area" + assert existing.article_4_pd_rights == "New Rights" + assert existing.listed_building == "New Listing" From 91db972d488a1a493d69c99b1f4e5814f2e39da6 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Wed, 8 Jul 2026 15:02:38 +0000 Subject: [PATCH 53/55] Bulk download: readable email, deal-name folders, visible worker logs First successful live run surfaced three issues: 1. Email looked rubbish (a giant raw presigned URL). Now sends a proper HTML email with a 'Download documents' button plus a plain-text fallback, and a summary (N documents across M properties, expiry). Email delivery is now best-effort: a transport failure no longer loses an already-built package (the link is still on sub_task.outputs), and the SMTP connect has a 30s timeout so an unreachable SES endpoint fails fast instead of hanging to the 900s Lambda timeout. 2. Every folder was 'address unavailable (...)': the resolver read property.address, but these are HubSpot deals with no property row. It now uses the deal's dealname from hubspot_deal_data. 3. No logs / no idea why a run took ~9 minutes: the worker's INFO logs were dropped (Lambda root logger defaults to WARNING). The handler now raises the level, and the orchestrator logs per-phase timing and volume (gather+plan, packaged N files / X MB, upload, email, total) so the slow phase is visible. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../bulk_document_download/handler.py | 13 +++ infrastructure/email/ses_smtp_email_sender.py | 13 ++- .../bulk_document_download_orchestrator.py | 100 ++++++++++++++++-- repositories/email/email_sender.py | 10 +- .../property/landlord_address_resolver.py | 37 +++---- .../email/test_ses_smtp_email_sender.py | 35 ++++++ ...est_bulk_document_download_orchestrator.py | 48 ++++++++- .../test_landlord_address_resolver.py | 16 +-- 8 files changed, 227 insertions(+), 45 deletions(-) diff --git a/applications/bulk_document_download/handler.py b/applications/bulk_document_download/handler.py index a0d90798c..aeb5fe315 100644 --- a/applications/bulk_document_download/handler.py +++ b/applications/bulk_document_download/handler.py @@ -16,6 +16,7 @@ Secrets Manager IAM-user credentials in). from __future__ import annotations +import logging import os from typing import Any @@ -40,6 +41,12 @@ from repositories.uploaded_file.uploaded_file_postgres_repository import ( ) from utilities.aws_lambda.task_handler import task_handler +# Surface the worker's progress logs: AWS Lambda's root logger defaults to +# WARNING, which drops the orchestrator's INFO stage/timing lines. Raise it so a +# slow or failing run is observable in CloudWatch. +logging.getLogger().setLevel(logging.INFO) +logger = logging.getLogger(__name__) + # The Download Package link is valid for 60 minutes (ADR-0060). _URL_TTL_SECONDS = 3600 @@ -68,6 +75,12 @@ def handler(body: dict[str, Any], context: Any) -> dict[str, Any]: recipient_email: str = str(recipe["recipient_email"]) package_name: str = str(recipe["package_name"]) + logger.info( + "bulk_document_download: starting run for %d properties (subtask=%s)", + len(landlord_property_ids), + trigger.subtask_id, + ) + boto_s3: Any = boto3.client("s3") # pyright: ignore[reportUnknownMemberType] orchestrator = BulkDocumentDownloadOrchestrator( uploaded_files=UploadedFilePostgresRepository(session), diff --git a/infrastructure/email/ses_smtp_email_sender.py b/infrastructure/email/ses_smtp_email_sender.py index ba53d23bc..88d3f3d56 100644 --- a/infrastructure/email/ses_smtp_email_sender.py +++ b/infrastructure/email/ses_smtp_email_sender.py @@ -34,11 +34,16 @@ class SmtpTransport(Protocol): def send_message(self, message: EmailMessage) -> Any: ... +# A short connect/IO timeout so an unreachable SES endpoint fails fast and is +# logged, instead of blocking the Lambda until its 900s timeout (ADR-0059). +_SMTP_TIMEOUT_SECONDS = 30 + + def _default_smtp(host: str, port: int) -> SmtpTransport: # `smtplib.SMTP` satisfies SmtpTransport structurally; its stdlib stubs # declare wider signatures (extra optional args, an SMTP-typed context # manager), so the cast bridges the real transport to the port's surface. - return cast(SmtpTransport, smtplib.SMTP(host, port)) + return cast(SmtpTransport, smtplib.SMTP(host, port, timeout=_SMTP_TIMEOUT_SECONDS)) class SesSmtpEmailSender: @@ -59,12 +64,16 @@ class SesSmtpEmailSender: self._from_address = from_address self._smtp_factory = smtp_factory - def send(self, *, to: str, subject: str, body: str) -> None: + def send( + self, *, to: str, subject: str, body: str, html_body: Optional[str] = None + ) -> None: message = EmailMessage() message["From"] = self._from_address message["To"] = to message["Subject"] = subject message.set_content(body) + if html_body is not None: + message.add_alternative(html_body, subtype="html") with self._smtp_factory(self._host, self._port) as smtp: smtp.starttls() smtp.login(self._username, self._password) diff --git a/orchestration/bulk_document_download_orchestrator.py b/orchestration/bulk_document_download_orchestrator.py index c67ee0372..9fb6d4c7f 100644 --- a/orchestration/bulk_document_download_orchestrator.py +++ b/orchestration/bulk_document_download_orchestrator.py @@ -10,9 +10,11 @@ rows are reported, not fatal; a wholly-empty selection is a recorded failure from __future__ import annotations +import html import logging import os import tempfile +import time import zipfile from dataclasses import dataclass from typing import Protocol, Sequence @@ -81,20 +83,23 @@ class BulkDocumentDownloadOrchestrator: recipient_email: str, package_name: str, ) -> DownloadPackageResult: + started = time.monotonic() documents = self._uploaded_files.by_landlord_property_ids( landlord_property_ids ) plan = plan_download_package([self._resolve(doc) for doc in documents]) skipped: list[SkippedDocument] = list(plan.skipped) + null_type_skips = len(skipped) logger.info( "bulk_document_download: selected=%d matched_documents=%d " - "planned_entries=%d skipped=%d", + "planned_entries=%d null_type_skips=%d (gather+plan %.1fs)", len(landlord_property_ids), len(documents), len(plan.entries), - len(skipped), + null_type_skips, + time.monotonic() - started, ) if not plan.entries: @@ -112,6 +117,8 @@ class BulkDocumentDownloadOrchestrator: # can't reach — becomes a SkippedDocument, it does not fail the package. key = f"{self._package_key_prefix}/{package_name}.zip" included = 0 + zip_bytes = 0 + zip_started = time.monotonic() with tempfile.TemporaryDirectory() as tmp_dir: zip_path = os.path.join(tmp_dir, "package.zip") with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as archive: @@ -141,18 +148,55 @@ class BulkDocumentDownloadOrchestrator: matched=len(documents), skipped=skipped, ) + zip_bytes = os.path.getsize(zip_path) + logger.info( + "bulk_document_download: packaged %d files (%d unreadable), " + "%.1f MB in %.1fs", + included, + len(skipped) - null_type_skips, + zip_bytes / 1_000_000, + time.monotonic() - zip_started, + ) + upload_started = time.monotonic() self._packages.upload_file(zip_path, key) + logger.info( + "bulk_document_download: uploaded %.1f MB in %.1fs", + zip_bytes / 1_000_000, + time.monotonic() - upload_started, + ) url = self._packages.generate_presigned_url(key, self._url_ttl_seconds) - self._email.send( - to=recipient_email, - subject="Your document download is ready", - body=( - f"Your document download is ready. It contains " - f"{included} document(s).\n\n" - f"Download it here (link valid for " - f"{self._url_ttl_seconds // 60} minutes):\n{url}" - ), + property_count = len({entry.landlord_property_id for entry in plan.entries}) + subject, body, html_body = self._compose_email( + url=url, + included=included, + properties=property_count, + skipped=len(skipped), + ) + try: + self._email.send( + to=recipient_email, + subject=subject, + body=body, + html_body=html_body, + ) + logger.info("bulk_document_download: emailed %s", recipient_email) + except Exception: + # Best-effort delivery (ADR-0060): the link is also written to + # sub_task.outputs, so an email/transport failure must not lose a + # package that was already built and uploaded. + logger.warning( + "bulk_document_download: email delivery to %s failed; the link is " + "still recorded on sub_task.outputs", + recipient_email, + exc_info=True, + ) + + logger.info( + "bulk_document_download: done — %d documents, %.1f MB, total %.1fs", + included, + zip_bytes / 1_000_000, + time.monotonic() - started, ) return DownloadPackageResult( presigned_url=url, @@ -161,6 +205,40 @@ class BulkDocumentDownloadOrchestrator: skipped=tuple(skipped), ) + def _compose_email( + self, *, url: str, included: int, properties: int, skipped: int + ) -> tuple[str, str, str]: + """The delivery email (ADR-0059): a subject, a plain-text body, and an + HTML alternative with a clean download button so the long presigned URL + isn't the visible content.""" + minutes = self._url_ttl_seconds // 60 + noun = "property" if properties == 1 else "properties" + skip_note = f" {skipped} item(s) were skipped." if skipped else "" + subject = f"Your document download is ready ({included} documents)" + body = ( + "Your document package is ready.\n\n" + f"It contains {included} document(s) across {properties} {noun}." + f"{skip_note}\n\n" + f"Download it here (link valid for {minutes} minutes):\n{url}\n" + ) + safe_url = html.escape(url, quote=True) + html_body = ( + '
' + "

Your document package is ready.

" + f"

It contains {included} document(s) across " + f"{properties} {noun}.{skip_note}

" + '

' + f'Download documents

' + f'

This link expires in {minutes} ' + "minutes. If the button doesn't work, paste this URL into your browser:" + f'
{safe_url}

' + "
" + ) + return subject, body, html_body + def _empty_failure( self, *, selected: int, matched: int, skipped: list[SkippedDocument] ) -> SubTaskFailure: diff --git a/repositories/email/email_sender.py b/repositories/email/email_sender.py index 2d6ac8725..21902ad2a 100644 --- a/repositories/email/email_sender.py +++ b/repositories/email/email_sender.py @@ -6,10 +6,14 @@ transport (SES over SMTP today) stays swappable and testable behind it. from __future__ import annotations -from typing import Protocol +from typing import Optional, Protocol class EmailSender(Protocol): - def send(self, *, to: str, subject: str, body: str) -> None: - """Send a plain-text email to ``to``.""" + def send( + self, *, to: str, subject: str, body: str, html_body: Optional[str] = None + ) -> None: + """Send an email to ``to``. ``body`` is the plain-text part; when + ``html_body`` is given it is added as an HTML alternative (clients that + render HTML show it, others fall back to the plain text).""" ... diff --git a/repositories/property/landlord_address_resolver.py b/repositories/property/landlord_address_resolver.py index cf051c0bd..8ae3f6188 100644 --- a/repositories/property/landlord_address_resolver.py +++ b/repositories/property/landlord_address_resolver.py @@ -1,20 +1,18 @@ -"""Resolves a property's display address from its `landlord_property_id` -(ADR-0060), for naming the folders in a Download Package. +"""Resolves a property's display name for the Download Package folders (ADR-0060). -The address lives on the FE-owned `property` table (`uploaded_files` carries no -`property_id` — a known future gap). Addresses for the whole requested set are -loaded once at construction; `address_for` is then a lookup with a safe -fallback for a `landlord_property_id` that has no property row or address. +The name comes from the HubSpot deal's ``dealname`` (``hubspot_deal_data``, keyed +by ``landlord_property_id``) — these properties are HubSpot deals and usually have +no ``property`` row, so the deal name is the human-readable label. Names for the +whole requested set are loaded once at construction; ``address_for`` is then a +lookup with a safe fallback for a ``landlord_property_id`` that has no deal name. """ from __future__ import annotations from typing import Sequence -from sqlalchemy import select -from sqlmodel import Session, col - -from infrastructure.postgres.property_table import PropertyRow +from sqlalchemy import text +from sqlmodel import Session class LandlordAddressResolver: @@ -23,15 +21,18 @@ class LandlordAddressResolver: def __init__( self, session: Session, landlord_property_ids: Sequence[str] ) -> None: - stmt = select(PropertyRow).where( - col(PropertyRow.landlord_property_id).in_(list(landlord_property_ids)) + rows = session.connection().execute( + text( + "SELECT landlord_property_id, dealname FROM hubspot_deal_data" + " WHERE landlord_property_id = ANY(:lpids)" + " AND dealname IS NOT NULL" + ), + {"lpids": list(landlord_property_ids)}, ) - rows = session.execute(stmt).scalars().all() # pyright: ignore[reportDeprecated] - self._by_id: dict[str, str] = { - row.landlord_property_id: row.address - for row in rows - if row.landlord_property_id is not None and row.address is not None - } + # First non-null deal name wins for a property with more than one deal. + self._by_id: dict[str, str] = {} + for landlord_property_id, dealname in rows.all(): + self._by_id.setdefault(landlord_property_id, dealname) def address_for(self, landlord_property_id: str) -> str: return self._by_id.get(landlord_property_id, self._FALLBACK) diff --git a/tests/infrastructure/email/test_ses_smtp_email_sender.py b/tests/infrastructure/email/test_ses_smtp_email_sender.py index 036ac239f..9c1b17a2d 100644 --- a/tests/infrastructure/email/test_ses_smtp_email_sender.py +++ b/tests/infrastructure/email/test_ses_smtp_email_sender.py @@ -77,6 +77,41 @@ def test_send_delivers_a_plaintext_message_with_the_right_envelope() -> None: assert "https://signed-url" in message.get_content() +def test_send_adds_an_html_alternative_when_given() -> None: + # Arrange + created: list[_FakeSmtp] = [] + + def factory(host: str, port: int) -> _FakeSmtp: + smtp = _FakeSmtp(host, port) + created.append(smtp) + return smtp + + sender = SesSmtpEmailSender( + host="email-smtp.eu-west-2.amazonaws.com", + port=587, + username="AKIA_SMTP_USER", + password="smtp-password", + from_address="noreply@domna.homes", + smtp_factory=factory, + ) + + # Act + sender.send( + to="user@example.com", + subject="s", + body="plain text fallback", + html_body="

Download documents

", + ) + + # Assert — a multipart/alternative carrying both the plain text and the HTML. + message = created[0].messages[0] + assert message.get_content_type() == "multipart/alternative" + plain_part = message.get_body(preferencelist=("plain",)) + html_part = message.get_body(preferencelist=("html",)) + assert plain_part is not None and "plain text fallback" in plain_part.get_content() + assert html_part is not None and "Download documents" in html_part.get_content() + + def test_send_starts_tls_and_authenticates_before_sending() -> None: # Arrange — SES SMTP rejects unauthenticated, cleartext senders, so the # transport must STARTTLS and log in with the SMTP credentials. diff --git a/tests/orchestration/test_bulk_document_download_orchestrator.py b/tests/orchestration/test_bulk_document_download_orchestrator.py index 47a071c38..01cdf9634 100644 --- a/tests/orchestration/test_bulk_document_download_orchestrator.py +++ b/tests/orchestration/test_bulk_document_download_orchestrator.py @@ -56,10 +56,19 @@ class _FakeDocuments: class _RecordingEmail: def __init__(self) -> None: - self.sent: list[tuple[str, str, str]] = [] + self.sent: list[tuple[str, str, str, Optional[str]]] = [] - def send(self, *, to: str, subject: str, body: str) -> None: - self.sent.append((to, subject, body)) + def send( + self, *, to: str, subject: str, body: str, html_body: Optional[str] = None + ) -> None: + self.sent.append((to, subject, body, html_body)) + + +class _FailingEmail: + def send( + self, *, to: str, subject: str, body: str, html_body: Optional[str] = None + ) -> None: + raise RuntimeError("smtp unreachable") def _document( @@ -114,9 +123,38 @@ def test_builds_uploads_and_emails_a_download_package(packages: S3Client) -> Non # ...the presigned URL points at that object, and the requester is emailed it. assert result.package_s3_key in result.presigned_url assert len(email.sent) == 1 - to, _subject, body = email.sent[0] + to, subject, body, html_body = email.sent[0] assert to == "user@example.com" - assert result.presigned_url in body + assert "1 documents" in subject + assert result.presigned_url in body # raw URL in the plain-text fallback + assert html_body is not None + assert "Download documents" in html_body # clean link, not the raw URL + + +def test_an_email_failure_does_not_lose_the_built_package(packages: S3Client) -> None: + # Arrange — the package builds and uploads fine, but the email transport is + # down. Best-effort delivery (ADR-0060): the URL is still returned (→ outputs). + files = [_document("LP1", "src-bucket", "surveys/a.pdf")] + orchestrator = BulkDocumentDownloadOrchestrator( + uploaded_files=_FakeUploadedFiles(files), + addresses=_FakeAddresses({"LP1": "12 Oak Street"}), + documents=_FakeDocuments({("src-bucket", "surveys/a.pdf"): b"PDF-BYTES"}), + packages=packages, + email=_FailingEmail(), + url_ttl_seconds=3600, + ) + + # Act — must not raise despite the email failure. + result = orchestrator.run( + landlord_property_ids=["LP1"], + recipient_email="user@example.com", + package_name="email-down", + ) + + # Assert — the package was still uploaded and its URL returned. + assert result.included == 1 + assert result.package_s3_key in result.presigned_url + assert packages.get_object(result.package_s3_key) # uploaded despite email failure def test_a_selection_with_no_documents_is_a_recorded_failure( diff --git a/tests/repositories/property/test_landlord_address_resolver.py b/tests/repositories/property/test_landlord_address_resolver.py index 7dcbb98da..4bc67a76f 100644 --- a/tests/repositories/property/test_landlord_address_resolver.py +++ b/tests/repositories/property/test_landlord_address_resolver.py @@ -4,7 +4,7 @@ import pytest from sqlalchemy import Engine from sqlmodel import Session -from infrastructure.postgres.property_table import PropertyRow +from backend.app.db.models.hubspot_deal_data import HubspotDealData from repositories.property.landlord_address_resolver import LandlordAddressResolver @@ -14,22 +14,26 @@ def session(db_engine: Engine) -> Iterator[Session]: yield s -def test_resolves_known_addresses_and_falls_back_for_unknowns( +def test_resolves_deal_names_and_falls_back_for_unknowns( session: Session, ) -> None: - # arrange — two properties with addresses; a third id has no property row. + # arrange — two properties with deal names; a third id has no deal. session.add( - PropertyRow(portfolio_id=1, landlord_property_id="LP1", address="12 Oak Street") + HubspotDealData( + deal_id="d1", landlord_property_id="LP1", dealname="12 Oak Street" + ) ) session.add( - PropertyRow(portfolio_id=1, landlord_property_id="LP2", address="9 Elm Road") + HubspotDealData( + deal_id="d2", landlord_property_id="LP2", dealname="9 Elm Road" + ) ) session.flush() # act resolver = LandlordAddressResolver(session, ["LP1", "LP2", "LP3"]) - # assert + # assert — the deal name becomes the folder label; unknowns fall back. assert resolver.address_for("LP1") == "12 Oak Street" assert resolver.address_for("LP2") == "9 Elm Road" assert resolver.address_for("LP3") == "address unavailable" From 04d1df1343f92e9ddf05b4c50269a842ba894f78 Mon Sep 17 00:00:00 2001 From: Jun-te Kim Date: Wed, 8 Jul 2026 15:12:24 +0000 Subject: [PATCH 54/55] Add remaining planning/address-profiling fields to HubSpot ETL sync Sync Design Constraints, Planning Comments, Planning Status, and Planning Suggested Approach, matching the columns added in Hestia-Homes/assessment-model#377. Co-Authored-By: Claude Sonnet 5 --- backend/app/db/models/hubspot_deal_data.py | 4 ++++ etl/hubspot/hubspotDataTodB.py | 12 ++++++++++++ etl/hubspot/hubspot_deal_differ.py | 4 ++++ etl/hubspot/tests/test_hubspot_data_to_db.py | 20 ++++++++++++++++++++ 4 files changed, 40 insertions(+) diff --git a/backend/app/db/models/hubspot_deal_data.py b/backend/app/db/models/hubspot_deal_data.py index d5db4c20a..2445045e4 100644 --- a/backend/app/db/models/hubspot_deal_data.py +++ b/backend/app/db/models/hubspot_deal_data.py @@ -97,6 +97,10 @@ class HubspotDealData(SQLModel, table=True): designated_area: Optional[str] = Field(default=None) article_4_pd_rights: Optional[str] = Field(default=None) listed_building: Optional[str] = Field(default=None) + design_constraints: Optional[str] = Field(default=None) + planning_comments: Optional[str] = Field(default=None) + planning_status: Optional[str] = Field(default=None) + planning_suggested_approach: Optional[str] = Field(default=None) created_at: Optional[datetime] = Field( sa_column=Column( diff --git a/etl/hubspot/hubspotDataTodB.py b/etl/hubspot/hubspotDataTodB.py index 82e50747d..68f84477e 100644 --- a/etl/hubspot/hubspotDataTodB.py +++ b/etl/hubspot/hubspotDataTodB.py @@ -306,6 +306,12 @@ class HubspotDataToDb: "designated_area": deal_data.get("designated_area"), "article_4_pd_rights": deal_data.get("article_pd_rights"), "listed_building": deal_data.get("listed_building"), + "design_constraints": deal_data.get("design_constraints"), + "planning_comments": deal_data.get("planning_comments"), + "planning_status": deal_data.get("planning_status"), + "planning_suggested_approach": deal_data.get( + "planning_suggested_approach" + ), }.items(): setattr(existing, attr, value) @@ -415,6 +421,12 @@ class HubspotDataToDb: designated_area=deal_data.get("designated_area"), article_4_pd_rights=deal_data.get("article_pd_rights"), listed_building=deal_data.get("listed_building"), + design_constraints=deal_data.get("design_constraints"), + planning_comments=deal_data.get("planning_comments"), + planning_status=deal_data.get("planning_status"), + planning_suggested_approach=deal_data.get( + "planning_suggested_approach" + ), ) def _handle_existing_photo_upload( diff --git a/etl/hubspot/hubspot_deal_differ.py b/etl/hubspot/hubspot_deal_differ.py index 2aa2c3bf1..fc8fbfd4e 100644 --- a/etl/hubspot/hubspot_deal_differ.py +++ b/etl/hubspot/hubspot_deal_differ.py @@ -113,6 +113,10 @@ class HubspotDealDiffer: "designated_area": "designated_area", "article_pd_rights": "article_4_pd_rights", "listed_building": "listed_building", + "design_constraints": "design_constraints", + "planning_comments": "planning_comments", + "planning_status": "planning_status", + "planning_suggested_approach": "planning_suggested_approach", } for hs_field, db_field in FIELD_MAP.items(): diff --git a/etl/hubspot/tests/test_hubspot_data_to_db.py b/etl/hubspot/tests/test_hubspot_data_to_db.py index 7d1693040..9f1f94ccd 100644 --- a/etl/hubspot/tests/test_hubspot_data_to_db.py +++ b/etl/hubspot/tests/test_hubspot_data_to_db.py @@ -137,6 +137,10 @@ def test_build_new_deal__planning_fields_mapped() -> None: "designated_area": "Conservation Area", "article_pd_rights": "Removed", "listed_building": "Grade II", + "design_constraints": "Sloped site", + "planning_comments": "Awaiting decision", + "planning_status": "Submitted", + "planning_suggested_approach": "Full application", }, listing=None, company=None, @@ -147,6 +151,10 @@ def test_build_new_deal__planning_fields_mapped() -> None: assert new_deal.designated_area == "Conservation Area" assert new_deal.article_4_pd_rights == "Removed" assert new_deal.listed_building == "Grade II" + assert new_deal.design_constraints == "Sloped site" + assert new_deal.planning_comments == "Awaiting decision" + assert new_deal.planning_status == "Submitted" + assert new_deal.planning_suggested_approach == "Full application" def test_update_existing_deal__planning_fields_overwritten() -> None: @@ -156,12 +164,20 @@ def test_update_existing_deal__planning_fields_overwritten() -> None: designated_area="Old Area", article_4_pd_rights="Old Rights", listed_building="Old Listing", + design_constraints="Old Constraints", + planning_comments="Old Comments", + planning_status="Old Status", + planning_suggested_approach="Old Approach", ) deal_data = { "planning_authority": "New Council", "designated_area": "New Area", "article_pd_rights": "New Rights", "listed_building": "New Listing", + "design_constraints": "New Constraints", + "planning_comments": "New Comments", + "planning_status": "New Status", + "planning_suggested_approach": "New Approach", } _make_instance()._update_existing_deal( @@ -175,3 +191,7 @@ def test_update_existing_deal__planning_fields_overwritten() -> None: assert existing.designated_area == "New Area" assert existing.article_4_pd_rights == "New Rights" assert existing.listed_building == "New Listing" + assert existing.design_constraints == "New Constraints" + assert existing.planning_comments == "New Comments" + assert existing.planning_status == "New Status" + assert existing.planning_suggested_approach == "New Approach" From 4bf85375e729d0a322f496574e029358fca31d3e Mon Sep 17 00:00:00 2001 From: Daniel Roth Date: Wed, 8 Jul 2026 16:49:55 +0000 Subject: [PATCH 55/55] add utilities to hubspot etl dockerfile --- etl/hubspot/scripts/scraper/handler/Dockerfile | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/etl/hubspot/scripts/scraper/handler/Dockerfile b/etl/hubspot/scripts/scraper/handler/Dockerfile index fc4fb0516..22c1a8f4e 100644 --- a/etl/hubspot/scripts/scraper/handler/Dockerfile +++ b/etl/hubspot/scripts/scraper/handler/Dockerfile @@ -18,6 +18,10 @@ RUN pip install --no-cache-dir -r requirements.txt # Copy necessary files for database and utility imports COPY backend/ backend/ COPY utils/ utils/ +# Distinct from utils/: etl.hubspot.hubspot_deal_differ (pulled in via +# HubspotClient -> main) imports `utilities.logger`. Without this the lambda +# fails at init with "No module named 'utilities'". +COPY utilities/ utilities/ COPY datatypes/ datatypes/ # main -> backend.app.db.models.{epc_property,recommendations} -> # infrastructure.postgres.{epc_property_table,modelling} -> domain.modelling.