The service was the last one living wholly under backend/. It now follows
the same layering as abri and the other newer services:
domain/pashub_fetcher/ core file classification, subfolders
infrastructure/pashub_fetcher/ PasHub client, token getter, wire DTOs
orchestration/ PashubFetcherOrchestrator (was PashubService)
applications/pashub_fetcher/ lambda handler, trigger request, dev tooling
core_files.py is split along the layer boundary: the domain module keeps the
filename/evidence-category classification rules and no longer imports
infrastructure.postgres, while the CoreFiles -> FileTypeEnum translation moves
to infrastructure/pashub_fetcher/core_file_types.py.
Tests move into the tests/ tree by layer. Note this puts them in the only
suite CI currently runs (unit_tests.yml is disabled), so these 73 tests now
execute on PRs for the first time; they were previously reachable only via
the legacy pytest.ini testpaths.
sharepoint_renamer's image now copies just domain/pashub_fetcher/ rather than
the whole service, since SharepointSubfolders is all it needed.
Behaviour is unchanged. tests/ goes 9927 -> 10000 passed (+73, exactly the
tests that moved in); the legacy suite keeps its same 17 pre-existing failures
and 11 errors. tests/test_lambda_packaging.py confirms both changed
Dockerfiles still copy their handler's full import closure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`plan.name` is a freeform, user-entered label that is unset on
modelling-generated Plans (0/205 populated on a real portfolio). The meaningful
"name of the plan chosen" is the Scenario the home's default Plan came from, so
`plan_name` now reads `scenario.name` via `plan.scenario_id` (LEFT JOIN scenario)
on both export selections. On the default-plan export this labels each home with
its source scenario (e.g. "Fabric Only" / "ASHP + PV"); on the scenario export it
equals that sheet's scenario.
Repository test seeds scenarios and asserts the source-scenario name; 62 tests
pass, pyright --strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A second export selection alongside the per-Scenario one, chosen by
`plan_selection` on the export request:
- "scenario" (default, unchanged): one sheet per Scenario, freshest Plan per
(property, scenario); scenario_ids required.
- "default" (new): a single "Default Plans" sheet of each home's is_default
Plan (one-per-property across all Scenarios, ADR-0012/0017) — the portfolio's
current state, one row per home; scenario_ids ignored.
Repository gains `default_rows_for(portfolio_id, property_ids)`: same read-model
and Effective-EPC join as `rows_for`, with the plan-selection WHERE swapped from
`scenario_id = :scenario_id` to `is_default = TRUE`. Orchestrator branches on
plan_selection; the router validates it and only requires scenario_ids for the
scenario selection. A recipe without plan_selection defaults to "scenario", so
the change is backward-compatible.
Both selections now surface the chosen Plan's `name` as a `plan_name` column.
Tests: repository default_rows_for (scenario-independent selection, plan_name,
non-default excluded); orchestrator default path (one "Default Plans" sheet;
empty selection is a recorded failure). 62 passed, pyright --strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`SapRoomInRoof.detailed_surfaces` is read by the SAP calculator
(heat_transmission.py:590, :690-696, :1311-1470) but has no Postgres column, so
the per-surface geometry is silently discarded on save. Without it the cascade
abandons the RdSAP 10 §3.10 "Detailed" route — each surface at its own Table 17 /
Table 4 U-value — and bills the whole notional roof shell at the Table 18
column (4) age-band default, typically 2.30 W/m2K for pre-1976 stock.
Measured on the committed 1000-cert RdSAP-21.0.1 corpus: 108 certs (10.8%) lodge
RIR geometry across 118 building parts; 107 of 108 score differently when it is
dropped; 15 change EPC band; worst case -15.21 SAP (uprn 10012119141, top-floor
maisonette, band E -> F). Larger in both population and magnitude than the
rafter_insulation_thickness gap that opened #1656.
Doc only — the Drizzle tables are owned by Hestia-Homes/assessment-model and the
model-side wiring follows once they exist.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §5.17 / Table 23. `wall_is_basement` is not a lodged U-value — it is a
disambiguation flag selecting WHICH RdSAP default applies, so unlike
wall/roof/floor_u_value it must survive the round-trip.
Code 6 is canonically SYSTEM-BUILT and the gov-API path infers "basement" from a
code-6 heuristic; the Elmhurst site-notes mapper sets wall_is_basement=False
precisely to defeat it. Dropping the field inverted the answer — a system-built
wall came back as a basement wall, dragging the whole ground floor onto the
Table 23 basement-floor U-value via `has_basement`.
Declared on no RdSAP schema, so it never arrives via the gov API; the earlier
corpus scan showing 0 occurrences measured the wrong population. The Elmhurst
site-notes path sets it and those EPCs are persisted
(scripts/wchg_elmhurst_ingest.py).
Nullable is load-bearing: None = "not stated, use the heuristic", False =
"explicitly system-built, do not".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add `rafter_insulation_thickness` to `EpcBuildingPartModel` (JSONB, matching the
sibling thickness columns so Union[str, int] survives) and wire it through
`from_domain` / `_to_building_part`.
Property 749719 lodges "250mm" -> Table 16 col (2) U 0.23. Dropped, the roof
billed at U 2.30 — a 10x error worth -9.8 SAP (lodged 73, calculated 63.2) that
moved the dwelling a whole band, C -> D, and drew a 9.63-point loft-insulation
recommendation on a roof rated 4/5 by the assessor.
The sibling wall/roof/floor_u_value and wall_is_basement fields have the same
drop but are deliberately NOT persisted: they are full-SAP artefacts and we
re-model as RdSAP, which uses default U-values. Their allow-list entries are
re-justified rather than removed — the previous "not read by the calculator" was
stale and is what hid this drop.
Requires the Drizzle migration in Hestia-Homes/assessment-model to land FIRST;
without the column, EPC reads fail outright.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-0065 is claimed by Scenario Export on the open #1546; this branch is the
later of the two, so it moves.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/run_pashub_modelling.py invokes the modelling_e2e handler in-process
over a portfolio using STORED EPCs (refetch_epc=False — the correct source for
PasHub cohorts, issue #1589), parametrised by env (portfolio/scenario/pids/
dry-run/batch). Handover documents the 3 re-extraction stragglers, the
rebaseliner pass-through gotcha (effective_sap_score IS the pashub rating for
unchanged lodged EPCs — never validate the calculator against it), and the 7
root-caused extractor bugs now tracked in #1590.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confirmed against the DB: Calico Asset References have zero overlap with
portfolio 824's landlord_property_id (addresses match; all 824 UPRNs non-null),
so the property-table lookup cannot resolve Calico. Documents the open
resolution (Calico-supplied ref->UPRN table or address matching) in ADR-0064,
the runner, and the README, and pins the confirmed S3 source location.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behaviour-preserving lift-and-shift of the condition module out of legacy
backend/condition/ into domain/condition, infrastructure/condition,
infrastructure/postgres/condition_tables.py, repositories/condition,
applications/condition, and tests/condition. Imports rewritten to the DDD
paths; ConditionPostgres and the ORM models keep the legacy backend.app.db
Base/db_session bridges so the existing suite proves behaviour is unchanged
(SQLModel + session-DI conversion tracked as a follow-up in ADR-0064).
16 condition tests pass at the new location.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the Stock Condition Data / Component Renewal Year / Roof Replacement
Flag glossary terms and ADR-0064, capturing that condition data is ingested
producer-only and never read by the modelling engine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The roof generator offered loft insulation only when genuinely uninsulated
(ADR-0021), so a loft with a shallow lodged depth (e.g. 724702's 50mm) got
nothing and never entered the optimiser pool. Grilled with Khalim: the loft
branch becomes eligible when the lodged depth is a known numeric value below
the 270mm building-regs compliance gate, topped up to the 300mm install depth.
Numeric-lodged-depths only; sentinels keep their ADR-0047 resolution. Loft
branch only. CONTEXT 'Roof Insulation Eligibility' updated to match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A live test returned 'no documents could be packaged' for 91 correctly-resolved
properties. Root cause: no upload source populates uploaded_files.landlord_property_id
(pashub/magic-plan/audit set hubspot_deal_id; ECMK sets hubspot_listing_id), so
matching on landlord_property_id found zero rows.
Match fix: the worker now joins uploaded_files -> hubspot_deal_data (on deal_id) ->
landlord_property_id, taking the property identity from the bridge. The repository
returns a small PropertyDocument read-model instead of the infra ORM row, so the
orchestrator no longer names infrastructure.postgres.* (resolves the leak dancafc
flagged) and the s3_upload_timestamp cast is gone. Coverage is limited to
deal-id-linked sources; listing_id/uprn-only files are a noted follow-up.
Observability: the empty-selection failure now carries stage counts
(selected/matched/planned/skipped) in both the message (-> the worker WARNING log)
and details (-> sub_task.outputs), and the run logs those counts. The next failure
says 'matched 0 documents' instead of failing opaquely.
ADR-0060 + CONTEXT.md updated (matching decision + considered options).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
uploaded_files has no property_id — matching is on landlord_property_id — so the
property_ids path was pointless indirection (look property.id up in property just
to translate back to landlord_property_id). The FE holds landlord_property_id per
row anyway.
task.inputs hand-pick key is now landlord_property_ids: str[] (was property_ids:
int[]). resolve_selection unions two landlord_property_id sources — project_codes
expanded via hubspot_deal_data, and the hand-picked ids taken as given — and no
longer queries the property table at all. Trigger-only change; the domain plan,
orchestrator and matching already work in landlord_property_id.
ADR-0060, CONTEXT.md and the request schema updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 'all properties' selection resolved against property.portfolio_id, but a
portfolio spans multiple HubSpot projects — so 'all' pulled the whole portfolio,
not the project the user was looking at. The project↔property grain lives on
hubspot_deal_data, not property.
task.inputs is now {project_codes?: str[], property_ids?: int[], portfolio_id?}:
the route resolves the distinct landlord_property_id set as the union of every
property in the named project_codes (from hubspot_deal_data) and the hand-picked
property_ids; portfolio_id is optional and only names the package. Drops the
portfolio-scoped select_all. Cap now applies to the resolved set size.
ADR-0060, CONTEXT.md and the request schema updated to match. New router tests
cover project-code resolution, the union+dedup, null landlord_property_id drop,
and the empty-project rejection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per PR review: the FE writes the selection config
({portfolio_id, property_ids?, select_all?}) into the FE-owned task.inputs and
passes only task_id, so a large hand-picked selection never travels in an HTTP
body. The route reads task.inputs, resolves to landlord_property_ids, caps, and
pins the recipe onto sub_task.inputs as before. Declares the FE-owned inputs
column on the TaskRow mirror so the backend can read it and the test schema
builds it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per review: packages go to a separate DOCUMENT_EXPORTS_BUCKET (not DATA_BUCKET),
no 7-day lifecycle imposed on the shared data bucket; packages can be several GB
so they stream via /tmp + multipart upload with raised ephemeral storage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Grilled 2026-07-08. Locks: app-owned-task + attach-mode lane (ADR-0055);
FastAPI route resolves the requesting user (dbId -> UserModel.email) and pins
it + the resolved property_id set into tasks.inputs; one capped Download
Package = one sub_task; landlord_property_id matching (property_id a future
gap); address-named folders (hubspot-enriched), latest per Document Type,
null-type skipped+reported; best-effort (fail only on infra / wholly-empty);
URL to sub_task.outputs + email; DATA_BUCKET/bulk-downloads/ ~7-day lifecycle.
Backend-sent email via a repositories/email port + infrastructure/email SES-SMTP
adapter. Glossary: Bulk Document Download, Download Package, Document Type.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Grilled decisions (issue #1496): 2-panel floor, every rung 2..floor(cap),
pro-rata per-segment yield (conservative), fill-by-generation, same cap
regime as today (no new roof gate), symmetric battery pairing. Glossary
gains "Sub-Ladder Configuration" ("trimmed"/"synthesized" deliberately
avoided — both collide with existing vocabulary).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- unit tests for resolve_group_ambiguity (distinct addresses withheld,
same-address re-listing kept, order preserved)
- Postgres integration tests for upsert_all's backstop dedup
- ADR-0057 recording the "confirm UPRNs before finalise" decision
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-0056 amended: override and EPC own type/form facts; a property with
neither is Unknown whatever the legacy columns say. Mirror columns removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The distributor pre-creates batch sub_tasks under the app-owned task and
sub_task status is a record, not retry fuel; filter resolution is a shared
contract with the app's preview. Scenario immutability replaces the
Scenario Snapshot fiction; Modelling Run, Distributor and First Run terms
pinned in CONTEXT.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Age band equality -> ±1-band neighbourhood; TFA ±5% -> ±20%; whitelist
core validated (fuel 95% / type 91% / wall 79% stable); roof-insulation
collateral flagged for follow-up. CONTEXT.md term updated to match.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Settled in a grilling session with Khalim: the pre-2012 backup feeds EPC
Prediction landlord-override-style (stable attrs as relax-ladder filters,
TFA as a ±5% band, exact-UPRN lookup, source="expired"), never trusted as
current state; scoped to the historic source; validated by a pairs harness.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
