- run_modelling_e2e --from-db re-models from already-persisted inputs (reads
each Property's Effective EPC + planning protections + solar from the DB) and
skips every live fetcher — zero gov-API calls. With --persist it re-writes the
Plan and, for lodged-EPC Properties, the Baseline. Self-contained loop; the
live-fetch path is untouched. Makes local re-runs instant and avoids tripping
the gov API's per-IP rate limit (6000 req / 5 min) during iteration.
- EpcClientService.REQUEST_TIMEOUT 10s -> 30s: a cold per-UPRN search can exceed
10s and the old timeout turned it into a timeout-then-retry; 30s rides it out.
Note: an open perf question remains — modelling is fast in isolation (<0.5s/
property) but a long-lived --persist run shows ~1 min/property; suspected in the
persist path (plan.save / baseline) or connection handling, NOT the API. Left
mid-diagnosis for handover.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Caught live writing property_overrides on portfolio 796: the Python
override_component SAEnum lagged the DB enum, so reading a new-component row
back threw LookupError. Guard it with a consistency test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SQLModel had drifted to a `bill_` prefix on the Bill Derivation block, but
the FE-owned Drizzle table uses unprefixed names (`heating_kwh`, `hot_water_kwh`
… `total_annual_bill_gbp`) plus a nullable `fuel_rates_period`. INSERTs failed
with UndefinedColumn. Rename the columns to mirror the live table column-for-
column (the prefix's anti-clash purpose is moot: `heating_kwh` != the recorded
`space_heating_kwh`), and add the `fuel_rates_period` column — left None until
Bill Derivation threads the snapshot period through.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new pipeline left no per-Property record of a run (the old engine set
property.has_recommendations and populated property_details_epc). Restore the
marker: PropertyRepository.mark_modelled sets has_recommendations (true when the
Plan carries measures, mirroring the old engine) and bumps updated_at, so a
first-run under the new process is identifiable as updated_at >= 2026-06-01.
ModellingOrchestrator marks each Property after its Scenarios (true if any
Scenario yielded a measure); run_modelling_e2e's --persist path marks it too
(its compute runs on in-memory fakes, so the DB UoW sets it directly). Adds the
has_recommendations/updated_at columns to the PropertyRow mirror.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The measures a run considers should come from the Scenario, not a CLI flag.
The live scenario table persists exclusions only (no inclusions column), as a
Postgres text-array of exact MeasureType values.
- Scenario gains `exclusions: frozenset[MeasureType]` + `considered_measures()`
(all measures minus the excluded ones, or None when none are excluded).
- ScenarioModel.to_domain parses the `{a,b,c}` exclusions array into
MeasureTypes, raising on a token that is not an exact MeasureType value
(no high-level category expansion), per the strict-enum convention.
- ModellingOrchestrator._plan_for derives the allowlist from the Scenario's
exclusions, combined (intersection) with any explicit considered_measures
override via the new `combine_considered_measures`.
- run_modelling_e2e sources the allowlist from the Scenario; --measures /
--exclude-measures become optional overlays (e.g. the technical
secondary_heating_removal exclusion the catalogue cannot yet stock).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `source` discriminator (lodged | predicted) to the EPC store so a Property
holds a lodged EPC and a predicted one (EPC Prediction gap-fill) at once
(ADR-0031). EpcRepository.save gains source="lodged"; idempotent delete is now
per-source (a predicted save no longer wipes lodged, and vice versa);
get_for_property/get_for_properties filter lodged; new get_predicted_for_property
/ get_predicted_for_properties read predicted. PropertyPostgresRepository.get +
get_many hydrate Property.predicted_epc, so the predicted picture reaches the
modelling read (both load via get_many). FakeEpcRepo mirrors the dual slot.
EpcPropertyModel gains `source` (default "lodged"); the test DB builds from the
SQLModel mirror so this is exercised without the prod migration. The matching
Drizzle change (column + per-(property_id,source) uniqueness) is the team's to
action before merge — docs/MIGRATION_NOTE_predicted_epc_source.md.
3 store tests (coexist, idempotent predicted re-save leaves lodged, lodged-only
has no predicted) + property-repo wiring; 85 pass across affected suites; new
code pyright-clean (2 pre-existing wwhrs errors in epc_property_table untouched).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Promote RdSAP-Schema-17.0 into SUPPORTED so all 1000 corpus certs are held to
the strict parse+map guard. Drop the now-redundant cert[0] tracer (subsumed by
the parametrised bucket); keep the reduced-field synthesis behavioural test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Promote RdSAP-Schema-19.0 into SUPPORTED so all 1000 corpus certs are held to
the strict parse+map guard. Drop the now-redundant cert[0] tracer (subsumed by
the parametrised bucket); keep the reduced-field synthesis behavioural test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Schema-fix delivered in the prior slice; this promotes the 1000-cert corpus
into the strict guard (matches 18.0/20.0.0).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drops the xfail now that Reduced-Field Synthesis (ADR-0027) maps every
20.0.0 cert; the corpus test holds the whole bucket to a strict 1000/1000.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pulls in 42 commits of calculator/mapper accuracy fixes from the per-cert
mapper-validation and floor/roof/heating fronts.
Conflict resolutions:
- mapper `_is_elmhurst_roof_window`: main dropped the branch's "wall location →
vertical" guard (it broke cert 000516's rooflight), but that re-broke cert
001431's two External-wall U>3.0 windows (which must stay vertical). The two
certs lodge a BYTE-IDENTICAL §11 row, so neither location nor U separates
them — the real discriminator is the room-in-roof context. Replaced the
unconditional U>3.0 backstop with one gated on the BP having a room-in-roof
(`_elmhurst_bp_has_room_in_roof`): 000516's Main BP has a "Room in roof type
1" (→ rooflight), 001431's does not (→ vertical). Validated against BOTH —
full Elmhurst worksheet suite 1038 pass + the 001431 window-extraction pin.
- property_postgres_repository: kept main's `ids_by_uprn` method + the branch's
`_restrictions_of` helper.
- sap_fuel.py: the branch relocated it to domain/billing/ (already carrying
main's to_table_32_code normalization), so kept the old path deleted.
Fallout from main's fabric fixes (validated by the boiler-3 real-cert pin which
still reproduces at delta 0):
- re-pinned the boiler-1 + boiler-instant-hw ASHP snapshot scores;
- main's §14.2 gas-boiler main-fuel derivation resolved the BGB/102 baseline
gap, so `test_gas_boiler_instant_hw_before_baselines` is now a passing test
(was an xfail tripwire).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR feedback (dancafc): the SQLModel column was Optional[str], but the
domain `SapBuildingPart.wall_insulation_thickness` is Optional[Union[str,
int]] — `_api_resolve_wall_insulation_thickness` returns an int mm when the
API lodges `wall_insulation_thickness == "measured"` (SAP 10.2 §5.7 /
Table 8). The plain str column round-trips that int back as the string
"100", corrupting the Table 8 insulated-wall U-value lookup.
This column was missed in the round-trip-fidelity §1 JSONB sweep
(#1129) — its `Union[str, int]` sibling `roof_insulation_thickness` was
converted, but `wall_insulation_thickness` was not, and no 21.0.0/21.0.1
fixture lodges "measured" so the gap stayed latent. Convert to JSONB
(matching `roof_insulation_thickness` / `flat_roof_insulation_thickness`),
align the column type to Optional[Union[str, int]] (also removes a pyright
type-mismatch), record it in the migration doc §1, and add a round-trip
guard test asserting an int survives as an int (fails as '100' == 100 on
the old str column).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3c.2. The OS Open-UPRN reference set is too large to host in Postgres, so
it lives in S3 and is cached per-UPRN in the existing `property_details_spatial`
table (ADR-0020). `PropertyDetailsSpatialRow` mirrors that table (uprn unique);
`SpatialRepository` / `SpatialPostgresRepository` upsert one shared row per UPRN
and read the planning protections back by UPRN (a null flag reads as
unrestricted; absent UPRNs are omitted so the caller defaults them).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Plan derives its Valuation Uplift (ADR-0018) from its baseline -> post
band jump and works+contingency cost, given one external input — the
Property's current market value (a Property Valuation, mostly absent).
`Plan.valuation` / `Plan.baseline_epc_rating` are derived like the other
headline figures; `PlanModel.from_domain` maps the £ forms to the live
plan.valuation_* columns (NULL when no value — the percentage is not
persisted on those columns). `Property.current_market_value` is the new
optional source; the orchestrator threads it onto the Plan. `run_one`
takes a `current_market_value` so the harness can value the uplift, and
the sense-check table shows the average % (always) plus the £ forms when
known.
Sourcing the current market value (upload / default) remains deferred
(ADR-0018); it is None throughout until that lands, so the columns stay
NULL at scale.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expand half of the recommendation_materials retirement (ADR-0017). A
Plan Measure installs a single Product, so thread its catalogue id end to
end — Product.id -> MeasureOption.material_id -> PlanMeasure.material_id
-> recommendation.material_id — replacing the per-material BOM child
table with one nullable column on the row. ProductPostgresRepository
reads the id from MaterialRow; the four fabric generators set it on their
Option; the orchestrator carries it onto the Plan Measure; the mirror
declares + maps the column. Optional throughout (the JSON stopgap
catalogue carries no ids -> NULL).
The multi-measure integration test now pins each persisted measure's
material_id to its seeded MaterialRow id. Migration spec (live column
must be added before this deploys; contraction is the owner's next step)
in docs/migrations/recommendation-material-id.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the scenario and installed_measure tables into
infrastructure/postgres/modelling/ as full-parity SQLModel definitions
(ScenarioModel, InstalledMeasureModel + MeasureType), completing the cluster
consolidation. backend/app/db/models/recommendations.py is now a pure
re-export shim.
ScenarioModel.goal is the PortfolioGoal enum (legacy planning branches on it),
sourced from domain/modelling/portfolio_goal.py; the repo's to_domain maps it to
its value string, so domain Scenario.goal is now the value ("Increasing EPC")
consistent with the orchestrator's check — fixing the latent name-vs-value
inconsistency the old str column masked (the scenario repo test stored the enum
*name*). Parity columns are nullable (mirror convention; live NOT-NULLs owned by
Drizzle).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Standardise the modelling persistence classes on the …Model suffix (PlanModel,
RecommendationModel, RecommendationMaterialModel) — matching the epc_property
precedent and the legacy names the rest of backend/ already imports, so the
shim's plan re-export becomes literal (no alias) and the eventual shim deletion
needs zero renames. The …Row→…Model sweep for the non-cluster tables
(Property/Task/Material/…) waits until their live legacy …Model counterparts
are retired, to avoid reintroducing dual-definition collisions. No behaviour
change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
