date_abandoned now resolves to the third failed attempt's confirmed
survey date, falling back to its last submission date, rather than the
processing date. This dates the OpenHousing cancellation to when the
job actually lapsed even if the trigger message lags or redelivers.
- domain: abandonment_date() encodes the confirmed-survey -> last-
submission fallback
- DealAbandonment carries both dates; abandon_job raises
AbandonmentDateUnknownError when a deal has neither
- last_submission_date now flows through the trigger message and request
- the injected clock is dropped: nothing reads now() any more
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When refetch_epc=False and no stored lodged EPC exists, the handler no longer
falls back to a live EPC API call — it treats the property as EPC-less and
hands it to the prediction path. This keeps REFETCH_EPC (lodged path) and
REPREDICT_EPC (prediction path) cleanly independent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Even after batching the data writes, the handler still wrote to the DB per
property through the orchestrator's SubTask bookkeeping: create + start +
complete each self-committed, and _cascade re-listed every sibling and re-saved
the parent on every transition — ~5 writes per property plus an O(N^2) cascade.
- TaskOrchestrator.run_subtasks: create all children in one INSERT, run each
(failures isolated per child), then persist all terminal states in one bulk
save and cascade the parent once. Children go WAITING -> terminal; the
transient IN_PROGRESS row is never written.
- SubTaskRepository.create_many / save_many (bulk INSERT / bulk fetch + update).
- _cascade short-circuits when the Task is already FAILED (terminal) — skips the
sibling roll-up entirely.
- modelling_e2e handler fans out via run_subtasks instead of per-property
create_child_subtask + run_subtask.
Per N-property batch the SubTask bookkeeping drops from ~5N writes + an O(N^2)
cascade to ~2 writes + 1 cascade.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The handler fired ~2+2N read round-trips and N+N write transactions per
SQS batch, pinning RDS CPU under ~32 concurrent containers on pool_size=1.
Reads: merge the duplicate property query and add overrides_for_many /
SolarRepository.get_many so overrides, solar, and property rows each load
in one query (2+2N -> 3).
Writes: buffer each modelled property's persistence intent in memory
(_PropertyWrite) during the loop, then flush the whole batch in one
PostgresUnitOfWork with a single commit, and run the baseline orchestrator
once for all written ids (N+N -> 2 transactions). Per-property modelling
failures stay isolated in the loop; the batch write is all-or-nothing and
retried via SQS (saves are idempotent upserts).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The modelling_e2e Lambda held up to ~4 concurrent Postgres connections per
invocation: the read Session stayed open across the write loop (the catalogue
was queried live and overrides were read per-Property), each per-Property Unit
of Work opened a second, and the TaskOrchestrator ran on its own NullPool
engine — so the pool needed pool_size=2 + max_overflow=1 just for the modelling
work. Under 32 concurrent containers that approached RDS max_connections.
Restructure the handler to read everything up front — overrides, Scenario, an
in-memory catalogue snapshot, and stored Solar — through one short-lived read
Session, close it, then write each Property in a sequential Unit of Work. The
read and write Sessions no longer overlap, so the engine drops to pool_size=1,
max_overflow=0. Fold the orchestrator onto the same pooled engine: its repos
commit on every save, releasing the connection between bookkeeping calls, so it
holds none during the work. One invocation now uses one connection at a time.
The catalogue becomes a per-invocation snapshot (MaterialSnapshotRepository),
mirroring ProductPostgresRepository.get exactly — same drift mapping, lowest-id
pick, and errors — but priced after the Session closes. Transaction isolation
is preserved: per-Property writes and orchestrator bookkeeping keep their own
independent transactions, just drawn sequentially from a single connection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_predict_epc returned None for three unrelated causes — unresolved
property_type, an empty same-type cohort, and a degenerate (no MAIN part)
prediction — which the handler collapsed into one generic "not predictable"
string. The SubTask output could not say which cause fired or which data to
fix.
Raise a specific PropertyNotModellableError subclass per cause, each carrying
the property's identity (property_id, uprn, postcode, portfolio_id) and
cause-specific context. The unresolved-property-type message points at the
likely missing/contradictory Landlord Override. All subclass ValueError, so the
per-property failure boundary keeps catching them and records str(exc).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Skipped cohort certs were previously surfaced only as outputs.result on a
completed subtask, so they were easy to miss. Treat them as a failure too:
once the batch has run to completion (so every modellable property is already
written to DB), raise if there were any per-property errors OR any skipped
certs. The run gets flagged for debugging without discarding the work done.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A predicted Property (no lodged EPC) got a Plan but nothing else: the synthesised
EPC was never written to epc_property, and Baseline Performance was skipped — so
property 729529 (portfolio 796 / scenario 1268), predicted from its DA16 1QZ
cohort, was "missed" with no predicted-EPC row and no baseline row.
Persist the synthesised EPC in the predicted slot (uow.epc.save(..., source=
"predicted"), ADR-0031) inside the Plan UoW, then run the Baseline orchestrator
for predicted Properties too — it re-hydrates the predicted EPC and establishes
the baseline from it. The earlier "lodged only" guard is dropped: by the write
block the Property always has a persisted EPC (lodged or predicted); one that
could be neither fetched nor predicted raised earlier.
Verified against the DB by invoking the real handler for 729529: predicted
epc_property rows 0->1 and property_baseline_performance rows 0->1. Baseline on
the predicted picture builds cleanly (RHI present, reason pre_sap10). Tests
updated: prediction + broadening paths now assert the predicted-slot epc.save and
the baseline run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The handler wrote epc/spatial/solar/plan and marked the property modelled, but
never established its Baseline Performance — so no row was created in
property_baseline_performance for any property modelled through the Lambda
(noticed on portfolio 796 / scenario 1268 / property 727218, a lodged property).
Mirror the e2e runner: after the plan UoW commits (so the EPC is persisted for
the orchestrator to re-hydrate), run PropertyBaselineOrchestrator for lodged
properties. Predicted properties have no lodged figures and no persisted EPC, so
they are skipped — consistent with the e2e runner and the ara_first_run Baseline
stage.
Verified 727218's baseline pipeline builds end-to-end in-memory (lodged_performance
→ CalculatorRebaseliner → bill → PropertyBaselinePerformance, reason pre_sap10).
Tests: lodged path asserts the orchestrator runs once; prediction path asserts it
does not.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reconciliations to make the modelling_e2e Lambda handler production-ready.
1. Price through the off-catalogue overlay, drop the workarounds
The handler priced through a plain ProductPostgresRepository and excluded
secondary_heating_removal / system_tune_up / system_tune_up_zoned to dodge
ProductNotFound (and a poisoning pgEnum DataError). Those measures are now
priced by catalogue_with_off_catalogue_overrides (already used by the e2e
runner and PostgresUnitOfWork), so the exclusions are removed and ALL measure
types are considered. This also fixes gas-boiler / single-glazed properties,
which Dan's handler never excluded and so still crashed (the standard
system_tune_up option is built unconditionally — the considered-measures
exclusion never actually gated it).
2. Broaden the EPC-Prediction cohort to nearby real postcodes (ADR-0031)
A property with no lodged EPC and no same-type comparable in its own postcode
(e.g. the only flat among houses) used to gate out and fail the subtask. The
gov EPC API cannot search by radius/outcode, so we resolve the real unit
postcodes physically nearest the target via postcodes.io (keyless; already a
trusted in-repo dependency) and walk them nearest-first until enough same-type
comparables surface. New PostcodesIoClient (transient-failure retry with
exponential backoff, soft-failing to the seed so broadening never breaks
prediction) and EpcComparablePropertiesRepository.candidates_near. Wired into
the handler and e2e runner; broadening is lazy (only on gate-out) and memoised
per (postcode, property_type).
Validated live: property 728476 (gas boiler) prices system_tune_up at GBP295;
property 718580 (lone flat in BR6 6BS) now predicts via nearby BR6 postcodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Aligns the composition with its entry point (the `ara_first_run` lambda +
`AraFirstRunTriggerBody`): clearer what the file does.
- orchestration/first_run_pipeline.py → ara_first_run_pipeline.py
- FirstRunPipeline → AraFirstRunPipeline; FirstRunCommand → AraFirstRunCommand
- test files renamed to match
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage-2 entry point for the First Run use case. Adds the
`ara_first_run` Lambda package mirroring the `postcode_splitter`
template, its typed trigger contract, and a stub `FirstRunPipeline`.
- `AraFirstRunTriggerBody`: thin command of five fields — `task_id`,
`sub_task_id` (UUID, lifecycle), `portfolio_id`, `property_ids`,
`scenario_ids` (int business IDs). No `model_config` override, so
Pydantic's default `extra="ignore"` lets the FastAPI backend add
fields without breaking deployed lambdas. UPRNs / Scenario defs are
deliberately off the event — read from source-of-truth tables.
- Thin `handler.py`: validate-and-delegate only, via a named
`dispatch_first_run` seam (testable without the Lambda runtime).
Subtask status (in-progress/complete/failed) + CloudWatch log URL
come for free from the existing `@subtask_handler()` decorator.
- `FirstRunPipeline` (orchestration/) stub: `run(command)` receives the
validated command. Declares a structural `FirstRunCommand` Protocol
(the three business fields) that `AraFirstRunTriggerBody` satisfies,
so orchestration needs no application-layer import — rhymes with the
`EpcFetcher`/`SolarFetcher` Protocols on IngestionOrchestrator
(ADR-0011). Full Ingestion→Baseline→Modelling composition lands in
#1136.
- Dockerfile / requirements.txt / local_handler/ mirror postcode_splitter.
TDD: 7 new tests (trigger-body validation incl. forward-compat +
id-types, pipeline seam, handler delegation). pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
