The modelling_e2e Lambda held up to ~4 concurrent Postgres connections per
invocation: the read Session stayed open across the write loop (the catalogue
was queried live and overrides were read per-Property), each per-Property Unit
of Work opened a second, and the TaskOrchestrator ran on its own NullPool
engine — so the pool needed pool_size=2 + max_overflow=1 just for the modelling
work. Under 32 concurrent containers that approached RDS max_connections.
Restructure the handler to read everything up front — overrides, Scenario, an
in-memory catalogue snapshot, and stored Solar — through one short-lived read
Session, close it, then write each Property in a sequential Unit of Work. The
read and write Sessions no longer overlap, so the engine drops to pool_size=1,
max_overflow=0. Fold the orchestrator onto the same pooled engine: its repos
commit on every save, releasing the connection between bookkeeping calls, so it
holds none during the work. One invocation now uses one connection at a time.
The catalogue becomes a per-invocation snapshot (MaterialSnapshotRepository),
mirroring ProductPostgresRepository.get exactly — same drift mapping, lowest-id
pick, and errors — but priced after the Session closes. Transaction isolation
is preserved: per-Property writes and orchestrator bookkeeping keep their own
independent transactions, just drawn sequentially from a single connection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make run_modelling_e2e the single script that does everything for a portfolio,
so the 291-property run needs one invocation with per-property recovery (no
all-or-nothing chunking):
- On --persist, a lodged-EPC Property now also gets its Baseline Performance
row written via PropertyBaselineOrchestrator (per Property, so one bad cert
does not abort the batch). Predicted (EPC-less) Properties have no lodged
figures, so they get a Plan but no baseline row.
- The run CSV gains api_sap (register) vs baseline_sap (calculator) + sap_delta,
so calculator-vs-API divergence is reviewable per property.
Fill the off-catalogue overlay for the measures the live material catalogue
cannot price, so they stop crashing the run:
- double_glazing (£550/window) and secondary_glazing (£400/window): priced
per window (the generator multiplies by single-glazed window count, matching
the legacy window_glazing). Grounded in 2025/26 UK installed costs; per-window
is the right unit for windows (fixed per-unit install dominates) — per-m2 fits
walls/floors, not glazing.
- gas_boiler_upgrade / system_tune_up / system_tune_up_zoned: these are priced
off the heating rate sheet (Products()), with get() reading the catalogue only
for an id — so the overlay entry exists to satisfy that lookup (material_id
stays None, as with ASHP); the rate sheet remains authoritative.
Validated on a 12-property sample (incl. a secondary-glazing case and a
SAP-Schema-16.2 cert): 12/12 baseline rows + plans, 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The FE-owned `material.type` pgEnum cannot carry `secondary_heating_removal`,
so pricing it through the DB catalogue raises a DataError that poisons the
session — the modelling pipeline crashed on any property with a lodged
secondary heater unless the measure was excluded on the Scenario.
Realise the `ProductRepository` docstring's intent (DB catalogue today, a JSON
file for costs the ETL does not yet supply, behind the same port): add a
`CompositeProductRepository` that resolves an override source first, then the
catalogue. Checking the override first keeps that Measure Type away from the DB
entirely; every other type misses the override and falls through unchanged.
- off_catalogue_costs.json prices it at £270 flat per-dwelling — the legacy
`Costs.heater_removal` ported to the new flat model (ADR-0028):
(£25 + £200 baseline) x 1.2 VAT, for the single fixed secondary a cert lodges.
Contingency (0.25) is joined from config, not the file.
- Wire the composite into PostgresUnitOfWork.product and run_modelling_e2e, so
the first-run pipeline and the local runner both honour the overlay.
- Integration test: drop the unrealistic seeded secondary_heating_removal DB
rows (the pgEnum can't hold the type) and assert it is JSON-sourced
(material_id is None, cost £270) end-to-end through a real Unit of Work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ASHP bundle is priced from the rate sheet (ADR-0025); the catalogue
row is read only for its material id, which is nullable end-to-end. The
live `material` catalogue has no `air_source_heat_pump` row, so
`products.get` raised `ValueError: no active product` and aborted every
ASHP-eligible property.
Add `ProductNotFound(ValueError)` + a concrete `ProductRepository
.get_optional`, raise the typed error from both repos, and have
`_ashp_option` look the row up optionally — a missing row now yields an
ASHP Option with `material_id=None` rather than crashing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ProductPostgresRepository.get took .first() with no ORDER BY, so when a
measure type has several active material rows (the live catalogue holds 74
solar_pv, 5 high_heat_retention_storage_heaters) the chosen row — hence the
cost and material_id — depended on the database's physical row order. Order by
id so a re-seed prices the same product every time.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expand half of the recommendation_materials retirement (ADR-0017). A
Plan Measure installs a single Product, so thread its catalogue id end to
end — Product.id -> MeasureOption.material_id -> PlanMeasure.material_id
-> recommendation.material_id — replacing the per-material BOM child
table with one nullable column on the row. ProductPostgresRepository
reads the id from MaterialRow; the four fabric generators set it on their
Option; the orchestrator carries it onto the Plan Measure; the mirror
declares + maps the column. Optional throughout (the JSON stopgap
catalogue carries no ids -> NULL).
The multi-measure integration test now pins each persisted measure's
material_id to its seeded MaterialRow id. Migration spec (live column
must be added before this deploys; contraction is the owner's next step)
in docs/migrations/recommendation-material-id.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the file-backed Product catalogue — the stopgap source for costs
the ETL does not yet supply, behind the same ProductRepository port as
ProductPostgresRepository. The JSON file maps each Measure Type to its
fully-loaded unit cost; the per-Measure-Type contingency is joined from
config (not stored in the file), so config stays the single source of
truth for contingency — mirroring the Postgres repo's mapping.
Strict-raises (ValueError) on an absent measure type, a non-object
entry, or a missing/non-numeric unit_cost_per_m2, matching the
repo-wide strict-no-silent-default convention. tmp_path-backed tests,
no DB fixture needed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Product(measure_type, unit_cost_per_m2, contingency_rate). ProductRepository
is the DDD port abstracting the catalogue source; ProductPostgresRepository
reads the externally-owned material table (defensive SQLModel view
MaterialRow) and maps an active row to a Product — total_cost becomes the
fully-loaded unit_cost_per_m2 — joining the per-measure-type contingency
(contingencies.py, mirrors Costs.CONTINGENCIES; cavity 0.10). Strict-raise
on missing/inactive row. A JSON-backed impl will follow behind the same
port for ETL-gap costs.
Two DB tests against an ephemeral Postgres (map active row; raise on
inactive-only). Toward #1155 cost (4b). Also generalises the CONTEXT
Simulation Overlay wording: windows are targeted by index, building-part
association carried via window_location (_window_bp_index). pyright clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
