Off by default like the other two prediction collaborators; the pipeline
turns on Expired-Enhanced Prediction by passing a resolver over the
historic S3 backup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The landlord property-type description is a "<dwelling type>: <built form>: <floor>"
split whose leading token IS the dwelling type; the built-form tail is not. The
LLM occasionally over-read the tail and flipped the type — a handful of
"Bungalow: EndTerrace" / "Bungalow: MidTerrace" dwellings were stored as House.
Adds property_type_guard (claims the recognised leading token: House / Bungalow /
Flat / Maisonette / Park home; defers unrecognised phrasings to the LLM) and wires
property_type through a GuardedColumnClassifier, so the built-form tail can never
flip the type and the live path is deterministic.
Applied the scoped backfill to portfolio 796 (Hyde): 3 rows corrected from House
back to Bungalow. No enum migration needed — the targets are original members.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MainFuelType had no individual wood-logs member — only "biomass (community)" —
so the LLM classifier funnelled "Solid Fuel: Wood Logs" into the community fuel,
inventing a community heat network the dwelling isn't on (and mislabelling the
connection). main_fuel had no deterministic guard at all, so nothing caught it.
Verified against domain/sap10_calculator/docs/specs: RdSAP 10 Specification
Table 32 lists "wood logs" as a solid fuel (code 20, 0.028 kgCO2e/kWh); the
calculator's input scheme (the gov EPC API fuel enum) codes it 6 -> Table 32 20
(sap_efficiencies._API_TO_TABLE32), and water_heating_overlay already pins the
same fuel to 6. So _FUEL_CODES["wood logs"] = 6 is confirmed, not guessed.
Adds MainFuelType.WOOD_LOGS + the _FUEL_CODES entry, a main_fuel_guard mirroring
water_heating_guard (claims the "wood log" token; dual fuel keeps its own member
since it has no "wood log" substring), and wires main_fuel through a
GuardedColumnClassifier so the live path is deterministic.
Applied the scoped backfill to portfolio 796 (Hyde): 21 rows off
"biomass (community)" -> "wood logs". property_overrides (TEXT) only; the
classifier-cache pgEnum member is deferred to the FE Drizzle migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires GuardedColumnClassifier(glazing_mix_guard, <llm>) into the glazing column so a
structured percentage mix deterministically resolves to MIXED (no overlay), while
uniform and varied phrasings fall through to the LLM (#1376, ADR-0042).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires GuardedColumnClassifier(roof_party_ceiling_guard, <llm>) into the roof_type
column so a party-ceiling marker (another/same dwelling or premises above), with or
without a trailing depth, deterministically resolves to its party-ceiling member
(no overlay, ~0 heat loss) instead of the LLM occasionally reading it as a pitched
loft (#1376).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire flag_fuel_mismatch into the two override-resolution paths (the property
repository and the modelling-e2e handler), keeping overlays_from a pure mapping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When refetch_epc=False and no stored lodged EPC exists, the handler no longer
falls back to a live EPC API call — it treats the property as EPC-less and
hands it to the prediction path. This keeps REFETCH_EPC (lodged path) and
REPREDICT_EPC (prediction path) cleanly independent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Even after batching the data writes, the handler still wrote to the DB per
property through the orchestrator's SubTask bookkeeping: create + start +
complete each self-committed, and _cascade re-listed every sibling and re-saved
the parent on every transition — ~5 writes per property plus an O(N^2) cascade.
- TaskOrchestrator.run_subtasks: create all children in one INSERT, run each
(failures isolated per child), then persist all terminal states in one bulk
save and cascade the parent once. Children go WAITING -> terminal; the
transient IN_PROGRESS row is never written.
- SubTaskRepository.create_many / save_many (bulk INSERT / bulk fetch + update).
- _cascade short-circuits when the Task is already FAILED (terminal) — skips the
sibling roll-up entirely.
- modelling_e2e handler fans out via run_subtasks instead of per-property
create_child_subtask + run_subtask.
Per N-property batch the SubTask bookkeeping drops from ~5N writes + an O(N^2)
cascade to ~2 writes + 1 cascade.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The handler fired ~2+2N read round-trips and N+N write transactions per
SQS batch, pinning RDS CPU under ~32 concurrent containers on pool_size=1.
Reads: merge the duplicate property query and add overrides_for_many /
SolarRepository.get_many so overrides, solar, and property rows each load
in one query (2+2N -> 3).
Writes: buffer each modelled property's persistence intent in memory
(_PropertyWrite) during the loop, then flush the whole batch in one
PostgresUnitOfWork with a single commit, and run the baseline orchestrator
once for all written ids (N+N -> 2 transactions). Per-property modelling
failures stay isolated in the loop; the batch write is all-or-nothing and
retried via SQS (saves are idempotent upserts).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The invocation is architecturally one DB connection at a time (read up front,
sequential write Units of Work, overrides resolved on the unit's own session).
Keep that as the design intent, but back it with NullPool instead of a fixed
pool_size=1 pool: each checkout opens a fresh connection and closes it on return,
so there is no pool slot to exhaust.
The difference is the failure mode if a path ever regresses and holds two
Sessions at once. A pool_size=1/max_overflow=0 pool turns that into a hard
30s dead-lock that fails the whole invocation ("QueuePool limit of size 1
overflow 0 reached, connection timed out"). NullPool instead opens a transient
second connection for that instant and the Lambda keeps running. The design
target stays one connection; NullPool just keeps it alive if we slip.
The single-connection invariant itself is still enforced in the Unit of Work
(overrides read on the unit's own session) and pinned by the regression test,
which uses its own strict pool_size=1 engine so it asserts the architecture
regardless of the production NullPool choice.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The modelling_e2e Lambda held up to ~4 concurrent Postgres connections per
invocation: the read Session stayed open across the write loop (the catalogue
was queried live and overrides were read per-Property), each per-Property Unit
of Work opened a second, and the TaskOrchestrator ran on its own NullPool
engine — so the pool needed pool_size=2 + max_overflow=1 just for the modelling
work. Under 32 concurrent containers that approached RDS max_connections.
Restructure the handler to read everything up front — overrides, Scenario, an
in-memory catalogue snapshot, and stored Solar — through one short-lived read
Session, close it, then write each Property in a sequential Unit of Work. The
read and write Sessions no longer overlap, so the engine drops to pool_size=1,
max_overflow=0. Fold the orchestrator onto the same pooled engine: its repos
commit on every save, releasing the connection between bookkeeping calls, so it
holds none during the work. One invocation now uses one connection at a time.
The catalogue becomes a per-invocation snapshot (MaterialSnapshotRepository),
mirroring ProductPostgresRepository.get exactly — same drift mapping, lowest-id
pick, and errors — but priced after the Session closes. Transaction isolation
is preserved: per-Property writes and orchestrator bookkeeping keep their own
independent transactions, just drawn sequentially from a single connection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_predict_epc returned None for three unrelated causes — unresolved
property_type, an empty same-type cohort, and a degenerate (no MAIN part)
prediction — which the handler collapsed into one generic "not predictable"
string. The SubTask output could not say which cause fired or which data to
fix.
Raise a specific PropertyNotModellableError subclass per cause, each carrying
the property's identity (property_id, uprn, postcode, portfolio_id) and
cause-specific context. The unresolved-property-type message points at the
likely missing/contradictory Landlord Override. All subclass ValueError, so the
per-property failure boundary keeps catching them and records str(exc).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
