The Dwelling-Roof Cap (ADR-0038) sizes by usable roof area and ranks segments
by distance from the dwelling, so the projection must carry each panel's
footprint and each segment's centre + area (from roofSegmentStats).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full-SAP certs mapped property_type=None, so the hard cohort filter silently
excluded them as comparables. Correctly typing them admits real lodged EPCs as
donors — a ground-truth-method change (cf #1245). Net over the n=36 fixture: 16
components better, 4 worse, 6 unchanged; gains concentrated in the physical
characteristics full-SAP certs measure (window_count 3.83->1.69, building_parts,
total_window_area, floor_construction, construction_age_band, glazing, walls).
The 4 that fell are new-build-vs-old-stock service mismatch on 1-2 targets each
(heating/water fuel, cylinder insulation) + floor_area. Tighten 16, loosen 4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Route the Google Solar client through the shared call_with_retry with
full jitter (de-synchronises the 32 concurrent containers per Google's
"avoid synchronised requests" guidance), honouring Retry-After, a 60s max
backoff (rides out the 600 QPM per-minute window), and 6 bounded retries.
429/5xx/transport errors are transient; other 4xx propagate immediately;
404-entity-not-found stays BuildingInsightsNotFoundError. On exhaustion a
TransientHttpError surfaces so the subtask fails and is re-triggered (no
silent degrade).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Even after batching the data writes, the handler still wrote to the DB per
property through the orchestrator's SubTask bookkeeping: create + start +
complete each self-committed, and _cascade re-listed every sibling and re-saved
the parent on every transition — ~5 writes per property plus an O(N^2) cascade.
- TaskOrchestrator.run_subtasks: create all children in one INSERT, run each
(failures isolated per child), then persist all terminal states in one bulk
save and cascade the parent once. Children go WAITING -> terminal; the
transient IN_PROGRESS row is never written.
- SubTaskRepository.create_many / save_many (bulk INSERT / bulk fetch + update).
- _cascade short-circuits when the Task is already FAILED (terminal) — skips the
sibling roll-up entirely.
- modelling_e2e handler fans out via run_subtasks instead of per-property
create_child_subtask + run_subtask.
Per N-property batch the SubTask bookkeeping drops from ~5N writes + an O(N^2)
cascade to ~2 writes + 1 cascade.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The handler fired ~2+2N read round-trips and N+N write transactions per
SQS batch, pinning RDS CPU under ~32 concurrent containers on pool_size=1.
Reads: merge the duplicate property query and add overrides_for_many /
SolarRepository.get_many so overrides, solar, and property rows each load
in one query (2+2N -> 3).
Writes: buffer each modelled property's persistence intent in memory
(_PropertyWrite) during the loop, then flush the whole batch in one
PostgresUnitOfWork with a single commit, and run the baseline orchestrator
once for all written ids (N+N -> 2 transactions). Per-property modelling
failures stay isolated in the loop; the batch write is all-or-nothing and
retried via SQS (saves are idempotent upserts).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An Economy-7 storage dwelling now prices heating at the 0.20-day/0.80-
night blend through cert -> calculator -> bill, instead of raising
UnpricedFuel and aborting the modelling_e2e batch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface the hot-water (Table 13 / HP-DHW), secondary (direct-acting),
main-2 and ALL_OTHER_USES High-Rate Fractions on CalculatorInputs from
the same Table 12a helpers the SAP cost path uses, so Bill Derivation's
day/night split matches the rating's exactly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The modelling_e2e Lambda runs on a single-connection pool (pool_size=1,
max_overflow=0) so one invocation uses one Postgres connection. But re-hydrating
a Property through PostgresUnitOfWork resolved its Landlord Overrides through a
PropertyOverridesPostgresReader built from the unit's session *factory* — which
opens a brand-new Session per call. While the unit's own read transaction was
still open (PropertyPostgresRepository.get_many had checked out the connection),
that second Session asked the pool for a second connection, found none, and timed
out after 30s:
QueuePool limit of size 1 overflow 0 reached, connection timed out, timeout 30.00
The baseline stage (PropertyBaselineOrchestrator.run -> uow.property.get_many ->
landlord overrides) hit this on every invocation.
Read the overrides on the unit's OWN session instead. property_overrides is
committed reference data, so reading it inside the unit's transaction sees the
same rows and keeps the invocation on one connection. Extract the query/mapping
into a shared helper and add OpenSessionPropertyOverridesReader (reads on a
caller-owned, already-open session without closing it) for the unit; the
standalone PropertyOverridesPostgresReader still opens its own short session for
use outside a unit.
Regression test pins the invariant with a real pool_size=1/max_overflow=0 engine:
without the fix it reproduces the exact QueuePool timeout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The modelling_e2e Lambda held up to ~4 concurrent Postgres connections per
invocation: the read Session stayed open across the write loop (the catalogue
was queried live and overrides were read per-Property), each per-Property Unit
of Work opened a second, and the TaskOrchestrator ran on its own NullPool
engine — so the pool needed pool_size=2 + max_overflow=1 just for the modelling
work. Under 32 concurrent containers that approached RDS max_connections.
Restructure the handler to read everything up front — overrides, Scenario, an
in-memory catalogue snapshot, and stored Solar — through one short-lived read
Session, close it, then write each Property in a sequential Unit of Work. The
read and write Sessions no longer overlap, so the engine drops to pool_size=1,
max_overflow=0. Fold the orchestrator onto the same pooled engine: its repos
commit on every save, releasing the connection between bookkeeping calls, so it
holds none during the work. One invocation now uses one connection at a time.
The catalogue becomes a per-invocation snapshot (MaterialSnapshotRepository),
mirroring ProductPostgresRepository.get exactly — same drift mapping, lowest-id
pick, and errors — but priced after the Session closes. Transaction isolation
is preserved: per-Property writes and orchestrator bookkeeping keep their own
independent transactions, just drawn sequentially from a single connection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_predict_epc returned None for three unrelated causes — unresolved
property_type, an empty same-type cohort, and a degenerate (no MAIN part)
prediction — which the handler collapsed into one generic "not predictable"
string. The SubTask output could not say which cause fired or which data to
fix.
Raise a specific PropertyNotModellableError subclass per cause, each carrying
the property's identity (property_id, uprn, postcode, portfolio_id) and
cause-specific context. The unresolved-property-type message points at the
likely missing/contradictory Landlord Override. All subclass ValueError, so the
per-property failure boundary keeps catching them and records str(exc).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fail if any EpcPropertyData field is neither reconstructed by _compose nor on a
documented allow-list, turning latent persistence gaps into explicit decisions
(would have caught the conservatory and roof-window drops). ADR-0036.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Persist SapConservatory as five nullable conservatory_* columns on epc_property
(1:1 with the dwelling) and rebuild it in _compose, so the §6.1 fold survives
save -> reload -> score. Without this the scored (re-hydrated) EPC silently
dropped the conservatory (persist != score) — a latent gap shared with the
21.0.1 path. Adds a deep-equality round-trip test. ADR-0036.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A predicted EPC is seeded by deep-copying one representative neighbour's
structure. _template chose the member whose floor area was closest to the
cohort median, ignoring building-part labels. When that member's only part
was lodged with a null identifier (mapped to OTHER), the prediction had no
MAIN part and the modelling_e2e handler rejected it as "not predictable" —
discarding an otherwise-rich same-type cohort.
Restrict the template to MAIN-bearing members (median still over the whole
cohort); fall back to closest-on-size only when none are MAIN-bearing, so an
all-unlabelled cohort is left for the handler's MAIN-part guard to reject
rather than silently relabelling real data.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
modelling_e2e properties with main fuel 39 failed at the price boundary
(UnpricedFuelCode since #44fff767; previously mis-rated as non-electric →
the ~14-SAP over-rating flagged in earlier review).
Code 39 is SAP Table 12 "electricity, any tariff" (epc_codes.csv main_fuel 39 =
"electricity, unspecified tariff"; spec footnote (j): defines an electric system,
cost/CO2/PE = standard electricity). It was absent from API_FUEL_TO_TABLE_32, so
to_table_32_code(39) was None → is_electric_fuel_code(39) False and pricing
raised.
Fix: map API_FUEL_TO_TABLE_32[39] = 30 (standard electricity) — the canonical
place Khalim's fuel work added codes. One line makes classification, pricing,
CO2/PE and the billing carrier all agree (39 → 30 → ELECTRICITY).
Tests: to_table_32_code(39)==30, is_electric_fuel_code(39) True, price == standard
electricity, and the billing carrier resolves to ELECTRICITY. 0 corpus impact
(no lodged corpus cert uses 39); accuracy + mapper-corpus gates green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A dwelling's heating is one conceptual system, but its fields are scattered
across EpcPropertyData (a gov-API schema mirror): the cluster on sap_heating, the
electricity tariff on sap_energy_source.meter_type, hot-water flags loose at top
level. Three places synthesise a heating system — Measure Options, Landlord
Overrides, EPC Prediction's donor — and each hand-copied a different ad-hoc
subset. The override and donor both dropped meter_type, so an electric-storage
system landed on the template's single-rate meter and billed overnight heat at
the peak rate: property 713406 scored SAP 13 (G) vs ~50 (E), inflating the HHRSH
measure to +45.8 and overshooting the plan to band A.
Establish a single Coherent Heating System boundary (CONTEXT.md) that every
synthesiser must cover, with a source-appropriate fill policy (ADR-0035):
- Override overlay *completes* the partial system the landlord named. Companion
fields are now DERIVED from the SAP code, not hand-attached per archetype: the
off-peak meter from the calculator's single off-peak classification (new
OFF_PEAK_IMPLYING_HEATING_CODES = SAP §12 Rules 1-2), and an unobserved storage
charge control defaults to the conservative manual control (Table 4e 2401). So
adding a heating archetype is just adding its code — companions can't be
forgotten. A contract test guards it (every off-peak code drags a Dual meter).
- Prediction's heating donor now *carries* the donor's meter_type alongside its
sap_heating cluster — the donor is already coherent.
Coherence is a synthesis-time obligation only; the calculator still scores a real
lodged cert exactly as lodged.
Verified on 713406: baseline 13 -> 47.8 (E), matching its recorded rating; the
phantom HHRSH recommendation is gone and the plan no longer overshoots to A.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
