The service was the last one living wholly under backend/. It now follows
the same layering as abri and the other newer services:
domain/pashub_fetcher/ core file classification, subfolders
infrastructure/pashub_fetcher/ PasHub client, token getter, wire DTOs
orchestration/ PashubFetcherOrchestrator (was PashubService)
applications/pashub_fetcher/ lambda handler, trigger request, dev tooling
core_files.py is split along the layer boundary: the domain module keeps the
filename/evidence-category classification rules and no longer imports
infrastructure.postgres, while the CoreFiles -> FileTypeEnum translation moves
to infrastructure/pashub_fetcher/core_file_types.py.
Tests move into the tests/ tree by layer. Note this puts them in the only
suite CI currently runs (unit_tests.yml is disabled), so these 73 tests now
execute on PRs for the first time; they were previously reachable only via
the legacy pytest.ini testpaths.
sharepoint_renamer's image now copies just domain/pashub_fetcher/ rather than
the whole service, since SharepointSubfolders is all it needed.
Behaviour is unchanged. tests/ goes 9927 -> 10000 passed (+73, exactly the
tests that moved in); the legacy suite keeps its same 17 pre-existing failures
and 11 errors. tests/test_lambda_packaging.py confirms both changed
Dockerfiles still copy their handler's full import closure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`plan.name` is a freeform, user-entered label that is unset on
modelling-generated Plans (0/205 populated on a real portfolio). The meaningful
"name of the plan chosen" is the Scenario the home's default Plan came from, so
`plan_name` now reads `scenario.name` via `plan.scenario_id` (LEFT JOIN scenario)
on both export selections. On the default-plan export this labels each home with
its source scenario (e.g. "Fabric Only" / "ASHP + PV"); on the scenario export it
equals that sheet's scenario.
Repository test seeds scenarios and asserts the source-scenario name; 62 tests
pass, pyright --strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A second export selection alongside the per-Scenario one, chosen by
`plan_selection` on the export request:
- "scenario" (default, unchanged): one sheet per Scenario, freshest Plan per
(property, scenario); scenario_ids required.
- "default" (new): a single "Default Plans" sheet of each home's is_default
Plan (one-per-property across all Scenarios, ADR-0012/0017) — the portfolio's
current state, one row per home; scenario_ids ignored.
Repository gains `default_rows_for(portfolio_id, property_ids)`: same read-model
and Effective-EPC join as `rows_for`, with the plan-selection WHERE swapped from
`scenario_id = :scenario_id` to `is_default = TRUE`. Orchestrator branches on
plan_selection; the router validates it and only requires scenario_ids for the
scenario selection. A recipe without plan_selection defaults to "scenario", so
the change is backward-compatible.
Both selections now surface the chosen Plan's `name` as a `plan_name` column.
Tests: repository default_rows_for (scenario-independent selection, plan_name,
non-default excluded); orchestrator default path (one "Default Plans" sheet;
empty selection is a recorded failure). 62 passed, pyright --strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`SapRoomInRoof.detailed_surfaces` is read by the SAP calculator
(heat_transmission.py:590, :690-696, :1311-1470) but has no Postgres column, so
the per-surface geometry is silently discarded on save. Without it the cascade
abandons the RdSAP 10 §3.10 "Detailed" route — each surface at its own Table 17 /
Table 4 U-value — and bills the whole notional roof shell at the Table 18
column (4) age-band default, typically 2.30 W/m2K for pre-1976 stock.
Measured on the committed 1000-cert RdSAP-21.0.1 corpus: 108 certs (10.8%) lodge
RIR geometry across 118 building parts; 107 of 108 score differently when it is
dropped; 15 change EPC band; worst case -15.21 SAP (uprn 10012119141, top-floor
maisonette, band E -> F). Larger in both population and magnitude than the
rafter_insulation_thickness gap that opened #1656.
Doc only — the Drizzle tables are owned by Hestia-Homes/assessment-model and the
model-side wiring follows once they exist.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RdSAP 10 §5.17 / Table 23. `wall_is_basement` is not a lodged U-value — it is a
disambiguation flag selecting WHICH RdSAP default applies, so unlike
wall/roof/floor_u_value it must survive the round-trip.
Code 6 is canonically SYSTEM-BUILT and the gov-API path infers "basement" from a
code-6 heuristic; the Elmhurst site-notes mapper sets wall_is_basement=False
precisely to defeat it. Dropping the field inverted the answer — a system-built
wall came back as a basement wall, dragging the whole ground floor onto the
Table 23 basement-floor U-value via `has_basement`.
Declared on no RdSAP schema, so it never arrives via the gov API; the earlier
corpus scan showing 0 occurrences measured the wrong population. The Elmhurst
site-notes path sets it and those EPCs are persisted
(scripts/wchg_elmhurst_ingest.py).
Nullable is load-bearing: None = "not stated, use the heuristic", False =
"explicitly system-built, do not".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add `rafter_insulation_thickness` to `EpcBuildingPartModel` (JSONB, matching the
sibling thickness columns so Union[str, int] survives) and wire it through
`from_domain` / `_to_building_part`.
Property 749719 lodges "250mm" -> Table 16 col (2) U 0.23. Dropped, the roof
billed at U 2.30 — a 10x error worth -9.8 SAP (lodged 73, calculated 63.2) that
moved the dwelling a whole band, C -> D, and drew a 9.63-point loft-insulation
recommendation on a roof rated 4/5 by the assessor.
The sibling wall/roof/floor_u_value and wall_is_basement fields have the same
drop but are deliberately NOT persisted: they are full-SAP artefacts and we
re-model as RdSAP, which uses default U-values. Their allow-list entries are
re-justified rather than removed — the previous "not read by the calculator" was
stale and is what hid this drop.
Requires the Drizzle migration in Hestia-Homes/assessment-model to land FIRST;
without the column, EPC reads fail outright.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-0065 is claimed by Scenario Export on the open #1546; this branch is the
later of the two, so it moves.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/run_pashub_modelling.py invokes the modelling_e2e handler in-process
over a portfolio using STORED EPCs (refetch_epc=False — the correct source for
PasHub cohorts, issue #1589), parametrised by env (portfolio/scenario/pids/
dry-run/batch). Handover documents the 3 re-extraction stragglers, the
rebaseliner pass-through gotcha (effective_sap_score IS the pashub rating for
unchanged lodged EPCs — never validate the calculator against it), and the 7
root-caused extractor bugs now tracked in #1590.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confirmed against the DB: Calico Asset References have zero overlap with
portfolio 824's landlord_property_id (addresses match; all 824 UPRNs non-null),
so the property-table lookup cannot resolve Calico. Documents the open
resolution (Calico-supplied ref->UPRN table or address matching) in ADR-0064,
the runner, and the README, and pins the confirmed S3 source location.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behaviour-preserving lift-and-shift of the condition module out of legacy
backend/condition/ into domain/condition, infrastructure/condition,
infrastructure/postgres/condition_tables.py, repositories/condition,
applications/condition, and tests/condition. Imports rewritten to the DDD
paths; ConditionPostgres and the ORM models keep the legacy backend.app.db
Base/db_session bridges so the existing suite proves behaviour is unchanged
(SQLModel + session-DI conversion tracked as a follow-up in ADR-0064).
16 condition tests pass at the new location.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the Stock Condition Data / Component Renewal Year / Roof Replacement
Flag glossary terms and ADR-0064, capturing that condition data is ingested
producer-only and never read by the modelling engine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The roof generator offered loft insulation only when genuinely uninsulated
(ADR-0021), so a loft with a shallow lodged depth (e.g. 724702's 50mm) got
nothing and never entered the optimiser pool. Grilled with Khalim: the loft
branch becomes eligible when the lodged depth is a known numeric value below
the 270mm building-regs compliance gate, topped up to the 300mm install depth.
Numeric-lodged-depths only; sentinels keep their ADR-0047 resolution. Loft
branch only. CONTEXT 'Roof Insulation Eligibility' updated to match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A live test returned 'no documents could be packaged' for 91 correctly-resolved
properties. Root cause: no upload source populates uploaded_files.landlord_property_id
(pashub/magic-plan/audit set hubspot_deal_id; ECMK sets hubspot_listing_id), so
matching on landlord_property_id found zero rows.
Match fix: the worker now joins uploaded_files -> hubspot_deal_data (on deal_id) ->
landlord_property_id, taking the property identity from the bridge. The repository
returns a small PropertyDocument read-model instead of the infra ORM row, so the
orchestrator no longer names infrastructure.postgres.* (resolves the leak dancafc
flagged) and the s3_upload_timestamp cast is gone. Coverage is limited to
deal-id-linked sources; listing_id/uprn-only files are a noted follow-up.
Observability: the empty-selection failure now carries stage counts
(selected/matched/planned/skipped) in both the message (-> the worker WARNING log)
and details (-> sub_task.outputs), and the run logs those counts. The next failure
says 'matched 0 documents' instead of failing opaquely.
ADR-0060 + CONTEXT.md updated (matching decision + considered options).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
uploaded_files has no property_id — matching is on landlord_property_id — so the
property_ids path was pointless indirection (look property.id up in property just
to translate back to landlord_property_id). The FE holds landlord_property_id per
row anyway.
task.inputs hand-pick key is now landlord_property_ids: str[] (was property_ids:
int[]). resolve_selection unions two landlord_property_id sources — project_codes
expanded via hubspot_deal_data, and the hand-picked ids taken as given — and no
longer queries the property table at all. Trigger-only change; the domain plan,
orchestrator and matching already work in landlord_property_id.
ADR-0060, CONTEXT.md and the request schema updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 'all properties' selection resolved against property.portfolio_id, but a
portfolio spans multiple HubSpot projects — so 'all' pulled the whole portfolio,
not the project the user was looking at. The project↔property grain lives on
hubspot_deal_data, not property.
task.inputs is now {project_codes?: str[], property_ids?: int[], portfolio_id?}:
the route resolves the distinct landlord_property_id set as the union of every
property in the named project_codes (from hubspot_deal_data) and the hand-picked
property_ids; portfolio_id is optional and only names the package. Drops the
portfolio-scoped select_all. Cap now applies to the resolved set size.
ADR-0060, CONTEXT.md and the request schema updated to match. New router tests
cover project-code resolution, the union+dedup, null landlord_property_id drop,
and the empty-project rejection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per PR review: the FE writes the selection config
({portfolio_id, property_ids?, select_all?}) into the FE-owned task.inputs and
passes only task_id, so a large hand-picked selection never travels in an HTTP
body. The route reads task.inputs, resolves to landlord_property_ids, caps, and
pins the recipe onto sub_task.inputs as before. Declares the FE-owned inputs
column on the TaskRow mirror so the backend can read it and the test schema
builds it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per review: packages go to a separate DOCUMENT_EXPORTS_BUCKET (not DATA_BUCKET),
no 7-day lifecycle imposed on the shared data bucket; packages can be several GB
so they stream via /tmp + multipart upload with raised ephemeral storage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Grilled 2026-07-08. Locks: app-owned-task + attach-mode lane (ADR-0055);
FastAPI route resolves the requesting user (dbId -> UserModel.email) and pins
it + the resolved property_id set into tasks.inputs; one capped Download
Package = one sub_task; landlord_property_id matching (property_id a future
gap); address-named folders (hubspot-enriched), latest per Document Type,
null-type skipped+reported; best-effort (fail only on infra / wholly-empty);
URL to sub_task.outputs + email; DATA_BUCKET/bulk-downloads/ ~7-day lifecycle.
Backend-sent email via a repositories/email port + infrastructure/email SES-SMTP
adapter. Glossary: Bulk Document Download, Download Package, Document Type.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Grilled decisions (issue #1496): 2-panel floor, every rung 2..floor(cap),
pro-rata per-segment yield (conservative), fill-by-generation, same cap
regime as today (no new roof gate), symmetric battery pairing. Glossary
gains "Sub-Ladder Configuration" ("trimmed"/"synthesized" deliberately
avoided — both collide with existing vocabulary).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- unit tests for resolve_group_ambiguity (distinct addresses withheld,
same-address re-listing kept, order preserved)
- Postgres integration tests for upsert_all's backstop dedup
- ADR-0057 recording the "confirm UPRNs before finalise" decision
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-0056 amended: override and EPC own type/form facts; a property with
neither is Unknown whatever the legacy columns say. Mirror columns removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The distributor pre-creates batch sub_tasks under the app-owned task and
sub_task status is a record, not retry fuel; filter resolution is a shared
contract with the app's preview. Scenario immutability replaces the
Scenario Snapshot fiction; Modelling Run, Distributor and First Run terms
pinned in CONTEXT.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Age band equality -> ±1-band neighbourhood; TFA ±5% -> ±20%; whitelist
core validated (fuel 95% / type 91% / wall 79% stable); roof-insulation
collateral flagged for follow-up. CONTEXT.md term updated to match.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Settled in a grilling session with Khalim: the pre-2012 backup feeds EPC
Prediction landlord-override-style (stable attrs as relax-ladder filters,
TFA as a ±5% band, exact-UPRN lookup, source="expired"), never trusted as
current state; scoped to the historic source; validated by a pairs harness.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#1444 Parts A+B for codes 191/192, diagnosed on portfolio 796: 30 of the
first 40 live overrides inherit a replaced system's category (6/4/7) —
suppressing HHRSH or billing direct-acting heat as storage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
WS2 of PRD #1435, diagnosed on property 711795: an electric-underfloor
override inherits the replaced community-heating category (no pathway,
baseline mis-rated 51 vs 67), and wall generators offer £4k insulation
that cannot change an age-band-L wall's derived U (0.280 → 0.280).
Glossary: Wall U-Value Gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ADR-0048 records why a system-replacing heating override must emit a
complete companion set — category and control archetype-forced for
coherence, fuel and meter deferred where multiple values are valid, and
an unmapped companion logged rather than silently inherited. Adds the
Heating Companion Set glossary term to CONTEXT.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
