When refetch_epc=False and no stored lodged EPC exists, the handler no longer
falls back to a live EPC API call — it treats the property as EPC-less and
hands it to the prediction path. This keeps REFETCH_EPC (lodged path) and
REPREDICT_EPC (prediction path) cleanly independent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Even after batching the data writes, the handler still wrote to the DB per
property through the orchestrator's SubTask bookkeeping: create + start +
complete each self-committed, and _cascade re-listed every sibling and re-saved
the parent on every transition — ~5 writes per property plus an O(N^2) cascade.
- TaskOrchestrator.run_subtasks: create all children in one INSERT, run each
(failures isolated per child), then persist all terminal states in one bulk
save and cascade the parent once. Children go WAITING -> terminal; the
transient IN_PROGRESS row is never written.
- SubTaskRepository.create_many / save_many (bulk INSERT / bulk fetch + update).
- _cascade short-circuits when the Task is already FAILED (terminal) — skips the
sibling roll-up entirely.
- modelling_e2e handler fans out via run_subtasks instead of per-property
create_child_subtask + run_subtask.
Per N-property batch the SubTask bookkeeping drops from ~5N writes + an O(N^2)
cascade to ~2 writes + 1 cascade.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The handler fired ~2+2N read round-trips and N+N write transactions per
SQS batch, pinning RDS CPU under ~32 concurrent containers on pool_size=1.
Reads: merge the duplicate property query and add overrides_for_many /
SolarRepository.get_many so overrides, solar, and property rows each load
in one query (2+2N -> 3).
Writes: buffer each modelled property's persistence intent in memory
(_PropertyWrite) during the loop, then flush the whole batch in one
PostgresUnitOfWork with a single commit, and run the baseline orchestrator
once for all written ids (N+N -> 2 transactions). Per-property modelling
failures stay isolated in the loop; the batch write is all-or-nothing and
retried via SQS (saves are idempotent upserts).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The modelling_e2e Lambda held up to ~4 concurrent Postgres connections per
invocation: the read Session stayed open across the write loop (the catalogue
was queried live and overrides were read per-Property), each per-Property Unit
of Work opened a second, and the TaskOrchestrator ran on its own NullPool
engine — so the pool needed pool_size=2 + max_overflow=1 just for the modelling
work. Under 32 concurrent containers that approached RDS max_connections.
Restructure the handler to read everything up front — overrides, Scenario, an
in-memory catalogue snapshot, and stored Solar — through one short-lived read
Session, close it, then write each Property in a sequential Unit of Work. The
read and write Sessions no longer overlap, so the engine drops to pool_size=1,
max_overflow=0. Fold the orchestrator onto the same pooled engine: its repos
commit on every save, releasing the connection between bookkeeping calls, so it
holds none during the work. One invocation now uses one connection at a time.
The catalogue becomes a per-invocation snapshot (MaterialSnapshotRepository),
mirroring ProductPostgresRepository.get exactly — same drift mapping, lowest-id
pick, and errors — but priced after the Session closes. Transaction isolation
is preserved: per-Property writes and orchestrator bookkeeping keep their own
independent transactions, just drawn sequentially from a single connection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_predict_epc returned None for three unrelated causes — unresolved
property_type, an empty same-type cohort, and a degenerate (no MAIN part)
prediction — which the handler collapsed into one generic "not predictable"
string. The SubTask output could not say which cause fired or which data to
fix.
Raise a specific PropertyNotModellableError subclass per cause, each carrying
the property's identity (property_id, uprn, postcode, portfolio_id) and
cause-specific context. The unresolved-property-type message points at the
likely missing/contradictory Landlord Override. All subclass ValueError, so the
per-property failure boundary keeps catching them and records str(exc).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Skipped cohort certs were previously surfaced only as outputs.result on a
completed subtask, so they were easy to miss. Treat them as a failure too:
once the batch has run to completion (so every modellable property is already
written to DB), raise if there were any per-property errors OR any skipped
certs. The run gets flagged for debugging without discarding the work done.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A predicted Property (no lodged EPC) got a Plan but nothing else: the synthesised
EPC was never written to epc_property, and Baseline Performance was skipped — so
property 729529 (portfolio 796 / scenario 1268), predicted from its DA16 1QZ
cohort, was "missed" with no predicted-EPC row and no baseline row.
Persist the synthesised EPC in the predicted slot (uow.epc.save(..., source=
"predicted"), ADR-0031) inside the Plan UoW, then run the Baseline orchestrator
for predicted Properties too — it re-hydrates the predicted EPC and establishes
the baseline from it. The earlier "lodged only" guard is dropped: by the write
block the Property always has a persisted EPC (lodged or predicted); one that
could be neither fetched nor predicted raised earlier.
Verified against the DB by invoking the real handler for 729529: predicted
epc_property rows 0->1 and property_baseline_performance rows 0->1. Baseline on
the predicted picture builds cleanly (RHI present, reason pre_sap10). Tests
updated: prediction + broadening paths now assert the predicted-slot epc.save and
the baseline run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The handler wrote epc/spatial/solar/plan and marked the property modelled, but
never established its Baseline Performance — so no row was created in
property_baseline_performance for any property modelled through the Lambda
(noticed on portfolio 796 / scenario 1268 / property 727218, a lodged property).
Mirror the e2e runner: after the plan UoW commits (so the EPC is persisted for
the orchestrator to re-hydrate), run PropertyBaselineOrchestrator for lodged
properties. Predicted properties have no lodged figures and no persisted EPC, so
they are skipped — consistent with the e2e runner and the ara_first_run Baseline
stage.
Verified 727218's baseline pipeline builds end-to-end in-memory (lodged_performance
→ CalculatorRebaseliner → bill → PropertyBaselinePerformance, reason pre_sap10).
Tests: lodged path asserts the orchestrator runs once; prediction path asserts it
does not.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reconciliations to make the modelling_e2e Lambda handler production-ready.
1. Price through the off-catalogue overlay, drop the workarounds
The handler priced through a plain ProductPostgresRepository and excluded
secondary_heating_removal / system_tune_up / system_tune_up_zoned to dodge
ProductNotFound (and a poisoning pgEnum DataError). Those measures are now
priced by catalogue_with_off_catalogue_overrides (already used by the e2e
runner and PostgresUnitOfWork), so the exclusions are removed and ALL measure
types are considered. This also fixes gas-boiler / single-glazed properties,
which Dan's handler never excluded and so still crashed (the standard
system_tune_up option is built unconditionally — the considered-measures
exclusion never actually gated it).
2. Broaden the EPC-Prediction cohort to nearby real postcodes (ADR-0031)
A property with no lodged EPC and no same-type comparable in its own postcode
(e.g. the only flat among houses) used to gate out and fail the subtask. The
gov EPC API cannot search by radius/outcode, so we resolve the real unit
postcodes physically nearest the target via postcodes.io (keyless; already a
trusted in-repo dependency) and walk them nearest-first until enough same-type
comparables surface. New PostcodesIoClient (transient-failure retry with
exponential backoff, soft-failing to the seed so broadening never breaks
prediction) and EpcComparablePropertiesRepository.candidates_near. Wired into
the handler and e2e runner; broadening is lazy (only on gate-out) and memoised
per (postcode, property_type).
Validated live: property 728476 (gas boiler) prices system_tune_up at GBP295;
property 718580 (lone flat in BR6 6BS) now predicts via nearby BR6 postcodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Aligns the composition with its entry point (the `ara_first_run` lambda +
`AraFirstRunTriggerBody`): clearer what the file does.
- orchestration/first_run_pipeline.py → ara_first_run_pipeline.py
- FirstRunPipeline → AraFirstRunPipeline; FirstRunCommand → AraFirstRunCommand
- test files renamed to match
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage-2 entry point for the First Run use case. Adds the
`ara_first_run` Lambda package mirroring the `postcode_splitter`
template, its typed trigger contract, and a stub `FirstRunPipeline`.
- `AraFirstRunTriggerBody`: thin command of five fields — `task_id`,
`sub_task_id` (UUID, lifecycle), `portfolio_id`, `property_ids`,
`scenario_ids` (int business IDs). No `model_config` override, so
Pydantic's default `extra="ignore"` lets the FastAPI backend add
fields without breaking deployed lambdas. UPRNs / Scenario defs are
deliberately off the event — read from source-of-truth tables.
- Thin `handler.py`: validate-and-delegate only, via a named
`dispatch_first_run` seam (testable without the Lambda runtime).
Subtask status (in-progress/complete/failed) + CloudWatch log URL
come for free from the existing `@subtask_handler()` decorator.
- `FirstRunPipeline` (orchestration/) stub: `run(command)` receives the
validated command. Declares a structural `FirstRunCommand` Protocol
(the three business fields) that `AraFirstRunTriggerBody` satisfies,
so orchestration needs no application-layer import — rhymes with the
`EpcFetcher`/`SolarFetcher` Protocols on IngestionOrchestrator
(ADR-0011). Full Ingestion→Baseline→Modelling composition lands in
#1136.
- Dockerfile / requirements.txt / local_handler/ mirror postcode_splitter.
TDD: 7 new tests (trigger-body validation incl. forward-compat +
id-types, pipeline seam, handler delegation). pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
