ADR-0014 BillDerivation prices a per-end-use EnergyBreakdown
(HEATING / HOT_WATER / LIGHTING / PUMPS_FANS / APPLIANCES / COOKING).
SapResult already carried the first four but not appliances or cooking,
so a downstream SapResult→EnergyBreakdown adapter had to stub those two
at 0 kWh — understating the bill by the whole unregulated electricity
load. Surface them so the property_baseline side can wire the sections.
Adds two output-only fields to CalculatorInputs + SapResult, threaded
exactly like lighting_kwh_per_yr:
appliances_kwh_per_yr — SAP 10.2 Appendix L L13/L14/L16a annual E_A
(sum of the §5 (68) monthly appliances kWh)
cooking_kwh_per_yr — SAP 10.2 Appendix L L20 (p.91) ELECTRICITY
estimate E_cook = 138 + 28×N
Both values already existed in cert_to_inputs.py (appliances_monthly_kwh,
cooking_monthly_kwh) — reused, not recomputed.
Fuel attribution: cooking_kwh_per_yr is the L20 ELECTRICITY figure (the
field docstring says so), distinct from the L18 cooking heat GAIN
(35 + 7N W) the §5 internal-gains cascade uses. The bill adapter should
treat cooking as an electricity carrier; a gas-cooker split, if ever
needed, is a separate follow-up.
HARD CONSTRAINT honoured — output-only, zero rating drift. Appliances +
cooking are unregulated and are NOT fed into ECF / total_fuel_cost /
CO2 / primary energy / sap_score. Every golden-fixture, Elmhurst e2e
SapResult pin, section cascade pin, and heating-corpus residual stays
byte-identical (1165 rated pins green). The synthetic CalculatorInputs
fixtures set the new fields non-zero on purpose so the existing cost/PE
reconciliation assertions act as leak detectors.
New focused test asserts both fields are populated (non-zero) and
threaded unchanged onto SapResult, with cooking equal to the L20
electricity figure (138 + 28×occupancy) to 1e-9. pyright net-zero
111 → 111.
Note: 11 pre-existing failures in test_appendix_u.py / test_table_32.py
arrived with the recently absorbed PR and are unrelated to this change
(they fail identically on the clean branch); flagged separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR feedback: prefer an abstract base the calculator inherits from over a
structural Protocol. Define `SapCalculator(ABC)` in the calculator package
(the engine owns its own contract) and have `Sap10Calculator` inherit it;
a future methodology is another subclass. Placing the ABC with the engine —
not in property_baseline — keeps the dependency pointing consumer -> engine
(sap10_calculator imports nothing from property_baseline). Consistent with
the repo's existing port convention (FuelRatesRepository(ABC)).
CalculatorRebaseliner keeps its reference to SapCalculator type-only (under
TYPE_CHECKING), so the module still does not import the calculator at
runtime. Test fakes now inherit the ABC since structural conformance no
longer applies.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 5a: the promotion. Replaces StubRebaseliner in production and collapses the
shadow runner into the rebaseliner (ADR-0013 amendment).
- CalculatorRebaseliner runs Sap10Calculator on every Property:
* sap_version < 10.2 -> Effective Performance IS the calculator output
(band via Epc.from_sap_score, CO2 kg->t, PEUI rounded), reason "pre_sap10".
* sap_version >= 10.2 -> Effective = lodged (API figures on-target), and the
calculator only logs divergence (SAP>0.5, PEUI/CO2 1%) as a validation signal.
* a calculator raise propagates -> batch aborts (ADR-0012); fix the cert at once.
- Rebaseliner.rebaseline gains property_id (for the divergence log).
- LoggingCalculatorShadow / the calculator_shadow seam removed from the
orchestrator; its divergence-comparison logic now lives in the rebaseliner.
- StubRebaseliner kept (signature updated) for orchestrator/repo unit tests.
The SapResult->EnergyBreakdown adapter + BillDerivation wiring (to populate the
bill block) follow once the appliances/cooking SapResult fields land.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3 of Bill Derivation. sap_code_to_fuel(code) maps a SAP 10.2 / Table 32
fuel code to the canonical billing Fuel — bounded to the ~47 Table 32 codes (the
carrier, orthogonal to the PCDB product index, so all PCDB heat pumps share one
electricity code). Mains gas / LPG / oil+bioliquids / coal / smokeless / wood /
electricity (standard + off-peak) / heat-network groupings; an unmapped code
(dual fuel, grid-export) raises UnmappedSapCode rather than guessing.
Also: ADR-0014 deferred/TODO section records the stubbed appliances+cooking
(pending the SapResult fields), the off-peak day/night split, the heat-network
rate gap, and regional rates / ETL.
The SapResult -> EnergyBreakdown adapter (next slice) is gated on the
appliances/cooking fields landing on SapResult.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 2 of Bill Derivation. BillDerivation(fuel_rates).derive(breakdown) takes a
delivered-energy breakdown (per-section EnergyLine(section, fuel, kwh) +
exported_kwh) and produces a Bill: per-section kWh + cost, standing charges,
SEG credit, and total.
- Each end-use line billed at its fuel's unit rate.
- Standing charge added ONCE per distinct fuel used (a meter, not an end use);
off-gas fuels carry 0 so contribute nothing — no metered/unmetered special case.
- SEG export credit subtracted.
- Deterministic (ADR-0006); raises UnpricedFuel (via FuelRates) on an unpriced
fuel (e.g. heat network) rather than billing at a wrong default.
Pure domain — no calculator dependency; the SapResult->EnergyBreakdown adapter
is slice 3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 1 of Bill Derivation — the reference-data foundation that later slices
price the calculator's per-end-use kWh against:
- Fuel enum (canonical billing fuels; the join key between the calculator's
SAP-code fuels and the rates snapshot). COAL + HEAT_NETWORK are members with
no national rate.
- FuelRates value object: unit_rate_p_per_kwh / standing_charge_p_per_day /
seg_export_p_per_kwh; raises UnpricedFuel on a fuel it has no rate for rather
than billing at a wrong default.
- FuelRatesRepository port (ADR-0011 Repo-reads-stored-reference-data) +
StaticFileFuelRatesRepository reading a committed JSON snapshot.
- Snapshot fuel_rates_2026_q2.json: GB national, Apr-Jun 2026 Ofgem cap
(gas/electricity) + DESNZ/NEP May 2026 (off-gas). Carries the full researched
data; the value object exposes single-rate fuels this slice. Off-peak
(day/night), house coal and heat network raise UnpricedFuel until later slices.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire Sap10Calculator into PropertyBaselineOrchestrator as a non-load-bearing
shadow runner. For each property it scores the Effective EPC beside the
load-bearing Lodged/Effective write, catches any strict-raise -> log.error
(never aborts the batch), and on success log.warning's divergence from Lodged:
SAP |continuous - lodged| > 0.5; PEUI/CO2 > 1% relative (CO2 after kg->tonnes).
Every line is tagged with sap_version so SAP-10.2 signal separates from
older-spec drift (ADR-0010 Validation Cohort).
Per ADR-0013, Calculated SAP10 Performance is not a persisted third value-set:
effective = calculated in every baselining scenario, so the calculator IS the
mechanism that produces Effective Performance (the Rebaseliner). It runs in
shadow only while being hardened; when overrides/estimation land it is promoted
to drive Effective and the failure posture flips to abort (ADR-0012, calculator
now load-bearing). No table change.
- ADR-0013 + CONTEXT (Calculated SAP10 Performance / Effective Performance /
Rebaselining) record the decision.
- CalculatorShadow port + LoggingCalculatorShadow + Calculator protocol.
- FakeCalculatorShadow for orchestrator unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This branch's objective is the SAL ingestion handler
(applications/SAL/handler.py) and its dependency tree. Drop work
that crept in but is unreferenced by it:
- EPC feature: domain/epc, infrastructure/epc (gov_uk + historical
clients), tests/infrastructure/epc
- datatypes/epc edits (instantaneous_wwhrs Optional) reverted to main
- asset_list/app.py local data-file/column tweak reverted to main
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Aligns the composition with its entry point (the `ara_first_run` lambda +
`AraFirstRunTriggerBody`): clearer what the file does.
- orchestration/first_run_pipeline.py → ara_first_run_pipeline.py
- FirstRunPipeline → AraFirstRunPipeline; FirstRunCommand → AraFirstRunCommand
- test files renamed to match
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Final slice of ADR-0012: collapse the per-property read round-trips a batch
made (Baseline hydrated ~8 queries x 30 properties one at a time) into a
handful of per-table IN queries.
- EpcPostgresRepository: extracted a shared `_compose(rows)` from `get` (the
windows + floor-dim fetches are now passed in, not fetched inline), so both
`get` and the new `get_for_properties(property_ids)` build EpcPropertyData
from pre-fetched rows. `get_for_properties` fetches each child table once
(`WHERE epc_property_id IN ...`), groups in memory, and composes — load-whole
per ADR-0002.
- PropertyRepository.get_many(property_ids) -> Properties: one query for the
property rows + one bulk EPC hydration, composed in input order.
- BaselineOrchestrator / IngestionOrchestrator read the batch via get_many
instead of N x get.
- Ports + fakes gain the bulk methods.
The #1129 round-trip fidelity test stays green (the compose extraction is
behaviour-preserving). New tests: bulk hydration correctness + round-trips are
constant w.r.t. batch size (one-per-table, proven by query count). 123 pass;
pyright strict clean; AAA.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the handler's whole-pipeline Session (one transaction across all
three stages, connection pinned during Ingestion's external IO) with a
Unit-of-Work per stage (ADR-0012, added here). Each stage runs its batch in
one unit and commits once; any property raising aborts the batch and the
subtask fails noisily.
- BaselineOrchestrator(unit_of_work, rebaseliner): one unit for the batch,
commit once. Raise on a pre-SAP10 property leaves the unit uncommitted.
- IngestionOrchestrator(unit_of_work, epc_fetcher, geospatial_repo,
solar_fetcher): fetch/write split — phase 1 fetches the whole batch (EPC /
coords / solar) with NO unit open; phase 2 writes in one unit and commits.
The connection is never held during external IO. Geospatial S3 repo stays
injected (reference data, not transactional).
- Handler: module-scoped engine (pool reused across warm invocations) + a UoW
factory; whole-pipeline `with Session` gone. `build_first_run_pipeline`
composes on the factory. Source clients still behind the raising seam.
- ADR-0012 records the decision (per-stage boundary, all-or-nothing batch,
idempotent re-run, fetch/write split, module-scoped engine). Modelling stub
left untouched (no-op, no DB) per the ADR.
Tests: orchestrators on a shared FakeUnitOfWork (assert persisted batch +
exactly-once commit + no-commit-on-raise). New real-DB E2E integration test:
real PostgresUnitOfWork, Ingestion writes the EPC → Baseline reads it back
through the repo → re-run replaces, not duplicates (1 EPC row, 1 baseline row
after two runs). 121 pass in tests/; pyright strict clean; AAA.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-runs of a First Run batch re-save a property's data; that must replace,
not duplicate (ADR-0012 idempotent batch writes).
- `EpcPostgresRepository.save` deletes the property's existing EPC graph
(parent + all child tables, floor-dims via their building parts) before
inserting, when a `property_id` is given. Anonymous saves still insert.
- `BaselinePostgresRepository.save` deletes the existing row for the
`property_id` before inserting — no more unique-constraint violation on
re-save; also what the re-score-on-override path needs.
- Solar already upserts, so it's unchanged.
The #1129 round-trip fidelity test stays green (delete-first is a no-op on
a first save). 2 new tests (re-save replaces, not duplicates). pyright
strict clean; AAA.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First slice of the per-stage batch-transaction refactor (ADR-0012). A
UnitOfWork is the single transaction a stage runs its batch in: a context
manager exposing the DB repos bound to one session, committing once on
`commit()` and rolling back on exception or exit-without-commit
(all-or-nothing per batch, fail noisily).
- `UnitOfWork` (port): `property` / `epc` / `solar` / `baseline` repos +
`commit()` / `rollback()`; `__exit__` rolls back uncommitted work.
- `PostgresUnitOfWork(session_factory)`: opens a Session from an injected
factory (a module-scoped engine + sessionmaker in prod, so the pool is
reused across warm invocations), binds the Postgres repos to it, closes
on exit.
Not yet wired into any orchestrator — that lands in the Baseline /
Ingestion refactor slices. 3 tests against ephemeral PG (commit durable
across units; exception rolls back; no-commit persists nothing). pyright
strict clean; AAA.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes the First Run spine. Replaces the #1130 stub FirstRunPipeline
with the real three-stage composition and wires it into the handler.
- `FirstRunPipeline.run(command)` sequences Ingestion → Baseline →
Modelling, threading **only** `property_ids` between stages (and
`scenario_ids` into Modelling, off the command — never a prior stage's
output). Stages are injected behind thin `IngestionStage` /
`BaselineStage` / `ModellingStage` Protocols (the EpcFetcher/SolarFetcher
idiom), so the handler owns wiring and tests substitute fakes (ADR-0011).
- `ModellingOrchestrator` stub + `ScenarioRepository` / `MaterialsRepository`
seam ports — `run(property_ids, scenario_ids)` reads through repos, does
no scoring yet. Method shapes deferred to the Modelling per-service grills
(Scenario / Scenario Phase / Snapshot / Optimised Package / Plans are rich
— not pre-empted here).
- Handler delegates to the real pipeline via `build_first_run_pipeline`
(Postgres-backed repos off the session). The Ingestion source clients
(EPC API / Google Solar / geospatial S3) are isolated behind one
`_source_clients_from_env` seam that raises until the deploy/Terraform
config settles — out of scope for this slice. Subtask complete/failed +
CloudWatch URL still come from `@subtask_handler`.
Integration test (the criterion's centrepiece): wires REAL Ingestion +
REAL Baseline + stub Modelling through a shared fake EPC repo, with a
repo-backed PropertyRepo composing the Property from that slice. Proves
Baseline reads the very EPC Ingestion persisted — the through-repos
hand-off, no in-memory coupling. Plus a composition test pinning stage
order + only-property_ids threading.
TDD, one test → one impl. pyright strict clean; AAA layout. 116 pass in
the tests/ tree, no regressions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage 2 of First Run. Establishes each Property's Baseline Performance
from persisted source data and writes it back — reads only from repos,
never a Fetcher or HTTP (ADR-0003), so it is byte-identical whether
Ingestion ran milliseconds ago or last week.
Domain (`domain/baseline/`):
- `Performance` VO — the four rated quantities: SAP / EPC Band / CO2 /
Primary Energy Intensity. `lodged_performance(epc)` reads them off the
EPC's recorded fields (PEUI = `energy_consumption_current`).
- `BaselinePerformance` (ADR-0004) — the paired `lodged` + `effective`
Performance + `rebaseline_reason`, plus the no-derivation part of the
energy block (`space_heating_kwh` / `water_heating_kwh`, off the RHI,
deterministic per ADR-0006). Both halves always populated.
- `Rebaseliner` port + `StubRebaseliner`: the re-score-on-override seam
(ADR-0011). SAP10 certs pass through (effective == lodged, reason
"none"); a pre-SAP10 cert raises `RebaselineNotImplemented` rather
than fabricating a plausible-but-wrong "none" — ML rebaselining is not
wired yet. Mirrors the repo's strict-raise culture.
Persistence: new `BaselineRepository` port + `BaselinePostgresRepository`
+ flat-column `baseline_performance` SQLModel (one row per Property). Per
ADR-0004's amendment this is a standalone table, NOT columns on the
retiring `property_details_epc`. Production migration is FE-owned
(Drizzle) — docs/migrations/baseline-performance-table.md.
Docs (grill-with-docs): corrected CONTEXT.md Lodged/Effective Performance
to Primary Energy Intensity (the term collided with its own _Avoid_ entry
under "heat demand") + fixed stale RHI field names; amended ADR-0004
Consequences for the standalone-table decision.
Fuel split + bills (rest of EPC Energy Derivation) deferred to a
follow-up — they need a Fuel Rates source (Ofgem-cap ETL) that does not
exist yet.
TDD, one test -> one impl: 7 tests (lodged read, rebaseliner pass-through
+ raise, orchestrator establish-and-persist + pre-SAP10 raise, Postgres
round-trip + absent). pyright strict clean; AAA layout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage-2 entry point for the First Run use case. Adds the
`ara_first_run` Lambda package mirroring the `postcode_splitter`
template, its typed trigger contract, and a stub `FirstRunPipeline`.
- `AraFirstRunTriggerBody`: thin command of five fields — `task_id`,
`sub_task_id` (UUID, lifecycle), `portfolio_id`, `property_ids`,
`scenario_ids` (int business IDs). No `model_config` override, so
Pydantic's default `extra="ignore"` lets the FastAPI backend add
fields without breaking deployed lambdas. UPRNs / Scenario defs are
deliberately off the event — read from source-of-truth tables.
- Thin `handler.py`: validate-and-delegate only, via a named
`dispatch_first_run` seam (testable without the Lambda runtime).
Subtask status (in-progress/complete/failed) + CloudWatch log URL
come for free from the existing `@subtask_handler()` decorator.
- `FirstRunPipeline` (orchestration/) stub: `run(command)` receives the
validated command. Declares a structural `FirstRunCommand` Protocol
(the three business fields) that `AraFirstRunTriggerBody` satisfies,
so orchestration needs no application-layer import — rhymes with the
`EpcFetcher`/`SolarFetcher` Protocols on IngestionOrchestrator
(ADR-0011). Full Ingestion→Baseline→Modelling composition lands in
#1136.
- Dockerfile / requirements.txt / local_handler/ mirror postcode_splitter.
TDD: 7 new tests (trigger-body validation incl. forward-compat +
id-types, pipeline seam, handler delegation). pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage 1 of the pipeline: per property, read its UPRN from the property row,
fetch its EPC, resolve coordinates from the Geospatial reference repo, thread
those into the Solar fetcher, and persist EPC + solar via repos. Fetchers never
call each other — the orchestrator threads the coordinate (ADR-0011). Coordinates
are reference data (deterministic from UPRN), resolved transiently to drive the
solar fetch rather than persisted per-property.
Depends on thin EpcFetcher/SolarFetcher Protocols (EpcClientService and
GoogleSolarApiClient satisfy them structurally). Unit-tested against fakes — no
DB, gov API, or network: persists EPC, threads coords into solar, skips
UPRN-less properties and skips solar when coordinates are absent. pyright clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add Coordinates value object + GeospatialRepository port + GeospatialS3Repository
adapter. Resolves a Property's lon/lat from the partitioned Ordnance Survey
Open-UPRN parquet (filename_meta -> partition -> UPRN row). A Repo, not a
Fetcher (ADR-0011): no live OS API call. The parquet reader is injected, so it's
unit-tested against fixture parquets with no S3/network; returns None when the
UPRN is uncovered or absent. pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the EpcClientService package (client + _retry + exceptions + tests) from
the dying backend/ tree to infrastructure/epc_client/ as the New-EPC-API Fetcher;
update the two callers (address2UPRN, a script). All 14 client tests pass.
Add SolarRepository port + SolarPostgresRepository persisting Google Solar
building insights as JSONB (solar_building_insights table), one row per Property.
The EPC repo half of this slice already landed in #1129. pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the Ara modelling aggregate root (ADR-0002): domain/property/ with
PropertyIdentity, SiteNotes, Property, Properties. Property.source_path
implements the two disjoint source paths + Recency Tie-Break (ADR-0001;
survey wins on an equal date); effective_epc resolves to the surveyed data
(Site Notes path) or the public EPC (epc_with_overlay path — Landlord
Overrides overlay is a later slice). Pure dataclasses, no infrastructure imports.
PropertyRepository port + PropertyPostgresRepository hydrate the aggregate
whole from a defensive view of the FE-owned 'property' table (identity columns)
plus the EPC slice via EpcRepository.get_for_property. Reads only from repos
(ADR-0003). 8 domain + 1 hydration test; pyright strict clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add epc_renewable_heat_incentive table (space_heating_kwh, water_heating_kwh +
the three insulation-impact kWh fields), wired into EpcPostgresRepository
save/get. This is the P0 gap: RenewableHeatIncentive carries the baseline
space-heating/hot-water kWh that EPC Energy Derivation consumes.
The round-trip test now asserts full deep-equality (dropped the
renewable_heat_incentive exclusion) and passes for RdSAP 21.0.0 + 21.0.1.
DB migration for the new table documented in
docs/migrations/epc-property-round-trip-fidelity.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Relocate EpcPropertyModel + child tables from the dying backend/ tree to
infrastructure/postgres/epc_property_table.py (re-export shim keeps
documents_parser working). Add EpcRepository port + EpcPostgresRepository with
a full reverse mapper (epc_property tables -> EpcPropertyData).
Round-trip test surfaced two fidelity gaps:
1. Union[int,str] SAP code fields were str()-coerced on save, losing the int
(API) vs str (Site Notes) distinction. Now stored as JSONB (type-preserving).
2. The schema was a partial projection. Closed the cheap gaps on the model
(heating shower/bath counts, roof_construction_type, curtain_wall_age,
addendum, mechanical_vent_duct_insulation_level, SAP 10.2 §2 ventilation
fields + a ventilation_present flag). Structural gaps tracked as follow-ups;
renewable_heat_incentive (P0, #1137) excluded from the assertion until landed.
Round-trip passes for RdSAP-Schema-21.0.0 and 21.0.1; pyright strict clean.
Migration inventory for the DB: docs/migrations/epc-property-round-trip-fidelity.md
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This branch's objective is the SAL ingestion handler
(applications/SAL/handler.py) and its dependency tree. Drop work
that crept in but is unreferenced by it:
- EPC feature: domain/epc, infrastructure/epc (gov_uk + historical
clients), tests/infrastructure/epc
- datatypes/epc edits (instantaneous_wwhrs Optional) reverted to main
- asset_list/app.py local data-file/column tweak reverted to main
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
