The previous commit staged these two files at `git mv` time but never
staged the edits that followed, so they landed with their pre-refactor
content: `class PashubService` and `from backend.pashub_fetcher...`
imports that no longer resolve. Local runs passed because the working
tree was correct; CI collected the committed tree and failed with
ModuleNotFoundError: No module named 'backend.pashub_fetcher'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The service was the last one living wholly under backend/. It now follows
the same layering as abri and the other newer services:
domain/pashub_fetcher/ core file classification, subfolders
infrastructure/pashub_fetcher/ PasHub client, token getter, wire DTOs
orchestration/ PashubFetcherOrchestrator (was PashubService)
applications/pashub_fetcher/ lambda handler, trigger request, dev tooling
core_files.py is split along the layer boundary: the domain module keeps the
filename/evidence-category classification rules and no longer imports
infrastructure.postgres, while the CoreFiles -> FileTypeEnum translation moves
to infrastructure/pashub_fetcher/core_file_types.py.
Tests move into the tests/ tree by layer. Note this puts them in the only
suite CI currently runs (unit_tests.yml is disabled), so these 73 tests now
execute on PRs for the first time; they were previously reachable only via
the legacy pytest.ini testpaths.
sharepoint_renamer's image now copies just domain/pashub_fetcher/ rather than
the whole service, since SharepointSubfolders is all it needed.
Behaviour is unchanged. tests/ goes 9927 -> 10000 passed (+73, exactly the
tests that moved in); the legacy suite keeps its same 17 pre-existing failures
and 11 errors. tests/test_lambda_packaging.py confirms both changed
Dockerfiles still copy their handler's full import closure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A second export selection alongside the per-Scenario one, chosen by
`plan_selection` on the export request:
- "scenario" (default, unchanged): one sheet per Scenario, freshest Plan per
(property, scenario); scenario_ids required.
- "default" (new): a single "Default Plans" sheet of each home's is_default
Plan (one-per-property across all Scenarios, ADR-0012/0017) — the portfolio's
current state, one row per home; scenario_ids ignored.
Repository gains `default_rows_for(portfolio_id, property_ids)`: same read-model
and Effective-EPC join as `rows_for`, with the plan-selection WHERE swapped from
`scenario_id = :scenario_id` to `is_default = TRUE`. Orchestrator branches on
plan_selection; the router validates it and only requires scenario_ids for the
scenario selection. A recipe without plan_selection defaults to "scenario", so
the change is backward-compatible.
Both selections now surface the chosen Plan's `name` as a `plan_name` column.
Tests: repository default_rows_for (scenario-independent selection, plan_name,
non-default excluded); orchestrator default path (one "Default Plans" sheet;
empty selection is a recorded failure). 62 passed, pyright --strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The modelling_e2e handler models one property per run_modelling call, so a
per-property "no persisted baseline" line would be one per property across a
whole re-model. The harness has no Baseline stage by design and cannot drift
from one, so it now establishes its own the way the Baseline orchestrator
would; the Postgres-backed lane is where the assertion has something to catch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dan's review on #1546:
- Stream the workbook to a /tmp temp file and S3Client.upload_file it, instead
of render_workbook -> bytes -> put_object. render_workbook now saves straight
to a path; the orchestrator renders to a temp file, multipart-uploads it, and
always cleans it up. Restores ADR-0065's "never an in-memory BytesIO" decision
(the OOM path at the 100k-row cap).
- Move the raw `SELECT ... FROM scenario` out of the Lambda handler into
ScenarioNamesPostgresRepository, so the handler stays composition-only and all
SQL lives in repositories/.
- current_sap_points goes through _float, matching its Optional[float] read-model
type and the sibling numeric facts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deal's third_party_surveyor_identifier is now required by the log
and amend flows: the trigger contract rejects messages without it
(blank treated as missing, since a blank resource makes OpenHousing
silently drop the appointment), and LogJobRequest/AmendJobRequest carry
it as a required field. The ABRI_RELAY_DEFAULT_RESOURCE fallback and
its config/terraform/workflow wiring are removed as dead code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Carry third_party_surveyor_identifier on the Abri trigger message and
through dispatch into the log and amend flows, booking the job against
that resource. LogJobRequest gains the optional resource field
AmendJobRequest already had; the client falls back to the configured
default surveyor when the deal names no third-party resource, so
existing deals behave unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review findings on PR #1527:
- The 'which goals are goal-aligned' set lived in two adjacent if-chains
(_objective_for and _require_budget_for_goal_aligned) that had to stay in
sync — a new goal-aligned goal added to one but not the other would slip
the budget guard. Both now read a single _GOAL_ALIGNED_OBJECTIVES table.
- The goal strings are the canonical PortfolioGoal enum values, not
re-declared string constants, so goal-value drift can't silently degrade
a goal to max-SAP; _target_sap reads the enum too.
- _scored_candidate_groups takes objective without a default (its only
caller passes it).
- scoring.py: 'cached: float | None' -> Optional[float] per the CLAUDE.md
'Use Optional over | None' rule.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First successful live run surfaced three issues:
1. Email looked rubbish (a giant raw presigned URL). Now sends a proper HTML
email with a 'Download documents' button plus a plain-text fallback, and a
summary (N documents across M properties, expiry). Email delivery is now
best-effort: a transport failure no longer loses an already-built package
(the link is still on sub_task.outputs), and the SMTP connect has a 30s
timeout so an unreachable SES endpoint fails fast instead of hanging to the
900s Lambda timeout.
2. Every folder was 'address unavailable (...)': the resolver read property.address,
but these are HubSpot deals with no property row. It now uses the deal's
dealname from hubspot_deal_data.
3. No logs / no idea why a run took ~9 minutes: the worker's INFO logs were
dropped (Lambda root logger defaults to WARNING). The handler now raises the
level, and the orchestrator logs per-phase timing and volume (gather+plan,
packaged N files / X MB, upload, email, total) so the slow phase is visible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A live test returned 'no documents could be packaged' for 91 correctly-resolved
properties. Root cause: no upload source populates uploaded_files.landlord_property_id
(pashub/magic-plan/audit set hubspot_deal_id; ECMK sets hubspot_listing_id), so
matching on landlord_property_id found zero rows.
Match fix: the worker now joins uploaded_files -> hubspot_deal_data (on deal_id) ->
landlord_property_id, taking the property identity from the bridge. The repository
returns a small PropertyDocument read-model instead of the infra ORM row, so the
orchestrator no longer names infrastructure.postgres.* (resolves the leak dancafc
flagged) and the s3_upload_timestamp cast is gone. Coverage is limited to
deal-id-linked sources; listing_id/uprn-only files are a noted follow-up.
Observability: the empty-selection failure now carries stage counts
(selected/matched/planned/skipped) in both the message (-> the worker WARNING log)
and details (-> sub_task.outputs), and the run logs those counts. The next failure
says 'matched 0 documents' instead of failing opaquely.
ADR-0060 + CONTEXT.md updated (matching decision + considered options).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses PR #1498 review:
- A per-file read failure (missing/deleted object, or a bucket the role can't
reach) is now a SkippedDocument(reason=unreadable), not a whole-run abort;
an all-unreadable selection still fails (no empty package). Best-effort on
the read path (ADR-0060).
- Each ZIP member is streamed to a temp file (S3DocumentDownloader.download,
boto download_file) and added from disk, so a single multi-GB member never
hits the heap — the 'never held whole in memory' claim is now true.
- The 409 double-submit guard is DB-arbitrated: the sub_task insert is
conditional on the task having none, so a race creates only one.
- Local env gets a placeholder recipient email so the route is exercisable.
- PackageEntry carries landlord_property_id so a read failure can be reported.
- TODO noting the UploadedFile.s3_upload_timestamp typing fix.
Two new tests: per-file read failure skips-and-reports; address fallback flows
to the folder name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Multi-GB packages are no longer held in memory: each source file is read,
written into the on-disk ZIP, and released; the archive is streamed up from
disk. Behaviour unchanged (moto tests green).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
date_abandoned now resolves to the third failed attempt's confirmed
survey date, falling back to its last submission date, rather than the
processing date. This dates the OpenHousing cancellation to when the
job actually lapsed even if the trigger message lags or redelivers.
- domain: abandonment_date() encodes the confirmed-survey -> last-
submission fallback
- DealAbandonment carries both dates; abandon_job raises
AbandonmentDateUnknownError when a deal has neither
- last_submission_date now flows through the trigger message and request
- the injected clock is dropped: nothing reads now() any more
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
