Code 3 = "(other premises below)" = above partially heated space (§3.12 →
U=0.7), confirmed 9/9 on single-BP certs (the diagnostic that dodged the
lossy-floors[] contamination). Records the 7536 re-pin and the lesson that
"irreducible residual" golden notes can mask a real mapper bug.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Record that profiler lead #1 (floor_codes=3) is not a clean single cause
(bimodal + confounded), that the paired worksheet certs confirm only
codes 1/6/7 (code 3 unmapped → needs a worksheet for 0380-2087), and that
immersion_type=2 / main_control=2107 / roof_codes=1 are scatter, not
dispatch bugs. The exposed-floor-on-flats fix (§3.12) shipped at b40e0f67.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR feedback (dancafc): the SQLModel column was Optional[str], but the
domain `SapBuildingPart.wall_insulation_thickness` is Optional[Union[str,
int]] — `_api_resolve_wall_insulation_thickness` returns an int mm when the
API lodges `wall_insulation_thickness == "measured"` (SAP 10.2 §5.7 /
Table 8). The plain str column round-trips that int back as the string
"100", corrupting the Table 8 insulated-wall U-value lookup.
This column was missed in the round-trip-fidelity §1 JSONB sweep
(#1129) — its `Union[str, int]` sibling `roof_insulation_thickness` was
converted, but `wall_insulation_thickness` was not, and no 21.0.0/21.0.1
fixture lodges "measured" so the gap stayed latent. Convert to JSONB
(matching `roof_insulation_thickness` / `flat_roof_insulation_thickness`),
align the column type to Optional[Union[str, int]] (also removes a pyright
type-mismatch), record it in the migration doc §1, and add a round-trip
guard test asserting an int survives as an int (fails as '100' == 100 on
the old str column).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flags the SY/B disambiguation change and the field-vs-property merge
landmine (raises AttributeError at first EpcPropertyData instantiation,
not at import; git merges silently) for the feature/bill-derivation
reviewer, with the recommended reconciliation and the strict-xfail
tripwire they own.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 5b: update the FE-owned migration spec so the other repo can create the
bill columns in parallel.
- Bill block: per-section delivered kWh + cost (heating, hot water, lighting,
appliances, cooking, pumps/fans, cooling) + standing_charges_gbp,
seg_credit_gbp, total_annual_bill_gbp, fuel_rates_period.
- space_heating_kwh / water_heating_kwh (RHI recorded demand) marked SUPERSEDED
by heating_kwh / hot_water_kwh (calculator delivered fuel); kept until the bill
populates, then dropped.
- Cooling section kept (mostly 0 but affects the bill, cheap to store).
- Records the calculator-load-bearing posture (effective_* may differ from
lodged_* for pre-10.2) and that columns are defined now / populated when the
SapResult->EnergyBreakdown adapter + BillDerivation wiring land.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3 of Bill Derivation. sap_code_to_fuel(code) maps a SAP 10.2 / Table 32
fuel code to the canonical billing Fuel — bounded to the ~47 Table 32 codes (the
carrier, orthogonal to the PCDB product index, so all PCDB heat pumps share one
electricity code). Mains gas / LPG / oil+bioliquids / coal / smokeless / wood /
electricity (standard + off-peak) / heat-network groupings; an unmapped code
(dual fuel, grid-export) raises UnmappedSapCode rather than guessing.
Also: ADR-0014 deferred/TODO section records the stubbed appliances+cooking
(pending the SapResult fields), the off-peak day/night split, the heat-network
rate gap, and regional rates / ETL.
The SapResult -> EnergyBreakdown adapter (next slice) is gated on the
appliances/cooking fields landing on SapResult.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pin the bills design from a /grill-with-docs session:
- ADR-0014: whole-home annual bill from SAP10 Calculation's delivered kWh per
end use, re-priced at real Fuel Rates (NOT the calculator's SAP-notional
total_fuel_cost_gbp, which is RdSAP Table 32 standardised prices ~half real
electricity). Fuel enum + FuelRates + FuelRatesRepository static snapshot;
per-section + total flat columns; raise on unpriced fuel (house coal /
heat network are the named gaps).
- ADR-0013 amendment: the shadow stepping-stone is collapsed — the calculator
is load-bearing now. effective=calculated for sap_version<10.2 (StubRebaseliner
floor 10.0->10.2); >=10.2 keeps lodged + logs divergence; a strict-raise
aborts the batch (load-bearing for bills regardless of version).
- CONTEXT: EPC Energy Derivation -> Bill Derivation (no "service" suffix);
Baseline Performance energy block = per-end-use kWh + per-section bill + total;
Fuel Rates = committed static snapshot; Rebaselining trigger threshold 10.2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire Sap10Calculator into PropertyBaselineOrchestrator as a non-load-bearing
shadow runner. For each property it scores the Effective EPC beside the
load-bearing Lodged/Effective write, catches any strict-raise -> log.error
(never aborts the batch), and on success log.warning's divergence from Lodged:
SAP |continuous - lodged| > 0.5; PEUI/CO2 > 1% relative (CO2 after kg->tonnes).
Every line is tagged with sap_version so SAP-10.2 signal separates from
older-spec drift (ADR-0010 Validation Cohort).
Per ADR-0013, Calculated SAP10 Performance is not a persisted third value-set:
effective = calculated in every baselining scenario, so the calculator IS the
mechanism that produces Effective Performance (the Rebaseliner). It runs in
shadow only while being hardened; when overrides/estimation land it is promoted
to drive Effective and the failure posture flips to abort (ADR-0012, calculator
now load-bearing). No table change.
- ADR-0013 + CONTEXT (Calculated SAP10 Performance / Effective Performance /
Rebaselining) record the decision.
- CalculatorShadow port + LoggingCalculatorShadow + Calculator protocol.
- FakeCalculatorShadow for orchestrator unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Factual staleness fix flagged in the handover; the calculator lives in
domain/sap10_calculator/calculator.py. Glossary term 'Baseline Performance'
deliberately left unchanged (concept vs PropertyBaselinePerformance class).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Orientation for the next chat picking up the two open fronts after the
ara_first_run rebuild shipped:
- where things stand (merged to main via per-cert; branch/worktree layout;
PRs into per-cert), authoritative ADRs/CONTEXT to read,
- current architecture + key files (post baseline→property_baseline /
FirstRun→AraFirstRun rename),
- conventions + gotchas (TDD, ephemeral PG, FakeUnitOfWork, pyright noise to
ignore, gh-credential push workaround),
- Task 1: wire Sap10Calculator into PropertyBaselineOrchestrator (Calculated
SAP10 Performance as a third value-set; failure-posture decision),
- Task 2: Modelling (stubs to build out; MaterialsRepository naming open;
needs a UoW when writing Plans),
- the raising/no-op seams not to mistake for done,
- known doc drift flagged (CONTEXT term vs PropertyBaselinePerformance class;
stale domain/sap/ path → domain/sap10_calculator).
Also banners ara_backend_design.md as superseded (architecture) by ADR-0011/0012.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 5b: update the FE-owned migration spec so the other repo can create the
bill columns in parallel.
- Bill block: per-section delivered kWh + cost (heating, hot water, lighting,
appliances, cooking, pumps/fans, cooling) + standing_charges_gbp,
seg_credit_gbp, total_annual_bill_gbp, fuel_rates_period.
- space_heating_kwh / water_heating_kwh (RHI recorded demand) marked SUPERSEDED
by heating_kwh / hot_water_kwh (calculator delivered fuel); kept until the bill
populates, then dropped.
- Cooling section kept (mostly 0 but affects the bill, cheap to store).
- Records the calculator-load-bearing posture (effective_* may differ from
lodged_* for pre-10.2) and that columns are defined now / populated when the
SapResult->EnergyBreakdown adapter + BillDerivation wiring land.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice 3 of Bill Derivation. sap_code_to_fuel(code) maps a SAP 10.2 / Table 32
fuel code to the canonical billing Fuel — bounded to the ~47 Table 32 codes (the
carrier, orthogonal to the PCDB product index, so all PCDB heat pumps share one
electricity code). Mains gas / LPG / oil+bioliquids / coal / smokeless / wood /
electricity (standard + off-peak) / heat-network groupings; an unmapped code
(dual fuel, grid-export) raises UnmappedSapCode rather than guessing.
Also: ADR-0014 deferred/TODO section records the stubbed appliances+cooking
(pending the SapResult fields), the off-peak day/night split, the heat-network
rate gap, and regional rates / ETL.
The SapResult -> EnergyBreakdown adapter (next slice) is gated on the
appliances/cooking fields landing on SapResult.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pin the bills design from a /grill-with-docs session:
- ADR-0014: whole-home annual bill from SAP10 Calculation's delivered kWh per
end use, re-priced at real Fuel Rates (NOT the calculator's SAP-notional
total_fuel_cost_gbp, which is RdSAP Table 32 standardised prices ~half real
electricity). Fuel enum + FuelRates + FuelRatesRepository static snapshot;
per-section + total flat columns; raise on unpriced fuel (house coal /
heat network are the named gaps).
- ADR-0013 amendment: the shadow stepping-stone is collapsed — the calculator
is load-bearing now. effective=calculated for sap_version<10.2 (StubRebaseliner
floor 10.0->10.2); >=10.2 keeps lodged + logs divergence; a strict-raise
aborts the batch (load-bearing for bills regardless of version).
- CONTEXT: EPC Energy Derivation -> Bill Derivation (no "service" suffix);
Baseline Performance energy block = per-end-use kWh + per-section bill + total;
Fuel Rates = committed static snapshot; Rebaselining trigger threshold 10.2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire Sap10Calculator into PropertyBaselineOrchestrator as a non-load-bearing
shadow runner. For each property it scores the Effective EPC beside the
load-bearing Lodged/Effective write, catches any strict-raise -> log.error
(never aborts the batch), and on success log.warning's divergence from Lodged:
SAP |continuous - lodged| > 0.5; PEUI/CO2 > 1% relative (CO2 after kg->tonnes).
Every line is tagged with sap_version so SAP-10.2 signal separates from
older-spec drift (ADR-0010 Validation Cohort).
Per ADR-0013, Calculated SAP10 Performance is not a persisted third value-set:
effective = calculated in every baselining scenario, so the calculator IS the
mechanism that produces Effective Performance (the Rebaseliner). It runs in
shadow only while being hardened; when overrides/estimation land it is promoted
to drive Effective and the failure posture flips to abort (ADR-0012, calculator
now load-bearing). No table change.
- ADR-0013 + CONTEXT (Calculated SAP10 Performance / Effective Performance /
Rebaselining) record the decision.
- CalculatorShadow port + LoggingCalculatorShadow + Calculator protocol.
- FakeCalculatorShadow for orchestrator unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Factual staleness fix flagged in the handover; the calculator lives in
domain/sap10_calculator/calculator.py. Glossary term 'Baseline Performance'
deliberately left unchanged (concept vs PropertyBaselinePerformance class).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make the stored units explicit on the property_baseline_performance columns:
- `*_co2_emissions` → `*_co2_emissions_t_per_yr` (tonnes CO₂/yr, whole dwelling)
- `*_primary_energy_intensity` → `*_primary_energy_intensity_kwh_per_m2_yr`
Column names only; the domain `Performance` VO stays unit-suffix-free (units are
a storage concern, mapped in from_domain/to_domain). Migration doc updated.
Round-trip stays green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the handler's whole-pipeline Session (one transaction across all
three stages, connection pinned during Ingestion's external IO) with a
Unit-of-Work per stage (ADR-0012, added here). Each stage runs its batch in
one unit and commits once; any property raising aborts the batch and the
subtask fails noisily.
- BaselineOrchestrator(unit_of_work, rebaseliner): one unit for the batch,
commit once. Raise on a pre-SAP10 property leaves the unit uncommitted.
- IngestionOrchestrator(unit_of_work, epc_fetcher, geospatial_repo,
solar_fetcher): fetch/write split — phase 1 fetches the whole batch (EPC /
coords / solar) with NO unit open; phase 2 writes in one unit and commits.
The connection is never held during external IO. Geospatial S3 repo stays
injected (reference data, not transactional).
- Handler: module-scoped engine (pool reused across warm invocations) + a UoW
factory; whole-pipeline `with Session` gone. `build_first_run_pipeline`
composes on the factory. Source clients still behind the raising seam.
- ADR-0012 records the decision (per-stage boundary, all-or-nothing batch,
idempotent re-run, fetch/write split, module-scoped engine). Modelling stub
left untouched (no-op, no DB) per the ADR.
Tests: orchestrators on a shared FakeUnitOfWork (assert persisted batch +
exactly-once commit + no-commit-on-raise). New real-DB E2E integration test:
real PostgresUnitOfWork, Ingestion writes the EPC → Baseline reads it back
through the repo → re-run replaces, not duplicates (1 EPC row, 1 baseline row
after two runs). 121 pass in tests/; pyright strict clean; AAA.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage 2 of First Run. Establishes each Property's Baseline Performance
from persisted source data and writes it back — reads only from repos,
never a Fetcher or HTTP (ADR-0003), so it is byte-identical whether
Ingestion ran milliseconds ago or last week.
Domain (`domain/baseline/`):
- `Performance` VO — the four rated quantities: SAP / EPC Band / CO2 /
Primary Energy Intensity. `lodged_performance(epc)` reads them off the
EPC's recorded fields (PEUI = `energy_consumption_current`).
- `BaselinePerformance` (ADR-0004) — the paired `lodged` + `effective`
Performance + `rebaseline_reason`, plus the no-derivation part of the
energy block (`space_heating_kwh` / `water_heating_kwh`, off the RHI,
deterministic per ADR-0006). Both halves always populated.
- `Rebaseliner` port + `StubRebaseliner`: the re-score-on-override seam
(ADR-0011). SAP10 certs pass through (effective == lodged, reason
"none"); a pre-SAP10 cert raises `RebaselineNotImplemented` rather
than fabricating a plausible-but-wrong "none" — ML rebaselining is not
wired yet. Mirrors the repo's strict-raise culture.
Persistence: new `BaselineRepository` port + `BaselinePostgresRepository`
+ flat-column `baseline_performance` SQLModel (one row per Property). Per
ADR-0004's amendment this is a standalone table, NOT columns on the
retiring `property_details_epc`. Production migration is FE-owned
(Drizzle) — docs/migrations/baseline-performance-table.md.
Docs (grill-with-docs): corrected CONTEXT.md Lodged/Effective Performance
to Primary Energy Intensity (the term collided with its own _Avoid_ entry
under "heat demand") + fixed stale RHI field names; amended ADR-0004
Consequences for the standalone-table decision.
Fuel split + bills (rest of EPC Energy Derivation) deferred to a
follow-up — they need a Fuel Rates source (Ofgem-cap ETL) that does not
exist yet.
TDD, one test -> one impl: 7 tests (lodged read, rebaseliner pass-through
+ raise, orchestrator establish-and-persist + pre-SAP10 raise, Postgres
round-trip + absent). pyright strict clean; AAA layout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add epc_renewable_heat_incentive table (space_heating_kwh, water_heating_kwh +
the three insulation-impact kWh fields), wired into EpcPostgresRepository
save/get. This is the P0 gap: RenewableHeatIncentive carries the baseline
space-heating/hot-water kWh that EPC Energy Derivation consumes.
The round-trip test now asserts full deep-equality (dropped the
renewable_heat_incentive exclusion) and passes for RdSAP 21.0.0 + 21.0.1.
DB migration for the new table documented in
docs/migrations/epc-property-round-trip-fidelity.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Relocate EpcPropertyModel + child tables from the dying backend/ tree to
infrastructure/postgres/epc_property_table.py (re-export shim keeps
documents_parser working). Add EpcRepository port + EpcPostgresRepository with
a full reverse mapper (epc_property tables -> EpcPropertyData).
Round-trip test surfaced two fidelity gaps:
1. Union[int,str] SAP code fields were str()-coerced on save, losing the int
(API) vs str (Site Notes) distinction. Now stored as JSONB (type-preserving).
2. The schema was a partial projection. Closed the cheap gaps on the model
(heating shower/bath counts, roof_construction_type, curtain_wall_age,
addendum, mechanical_vent_duct_insulation_level, SAP 10.2 §2 ventilation
fields + a ventilation_present flag). Structural gaps tracked as follow-ups;
renewable_heat_incentive (P0, #1137) excluded from the assertion until landed.
Round-trip passes for RdSAP-Schema-21.0.0 and 21.0.1; pyright strict clean.
Migration inventory for the DB: docs/migrations/epc-property-round-trip-fidelity.md
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Records the grill-with-docs outcomes for the ara_first_run rebuild: three
composable stage orchestrators (Ingestion/Baseline/Modelling), one lambda per
use case chaining them through repos (not in-memory), and the Fetcher-vs-Repo
data-source taxonomy. Amends ADR-0003's chaining rule to generalise beyond
RefreshOrchestrator. Adds the pipeline-composition + First Run vocabulary to
CONTEXT.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Orientation for the next chat picking up the two open fronts after the
ara_first_run rebuild shipped:
- where things stand (merged to main via per-cert; branch/worktree layout;
PRs into per-cert), authoritative ADRs/CONTEXT to read,
- current architecture + key files (post baseline→property_baseline /
FirstRun→AraFirstRun rename),
- conventions + gotchas (TDD, ephemeral PG, FakeUnitOfWork, pyright noise to
ignore, gh-credential push workaround),
- Task 1: wire Sap10Calculator into PropertyBaselineOrchestrator (Calculated
SAP10 Performance as a third value-set; failure-posture decision),
- Task 2: Modelling (stubs to build out; MaterialsRepository naming open;
needs a UoW when writing Plans),
- the raising/no-op seams not to mistake for done,
- known doc drift flagged (CONTEXT term vs PropertyBaselinePerformance class;
stale domain/sap/ path → domain/sap10_calculator).
Also banners ara_backend_design.md as superseded (architecture) by ADR-0011/0012.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make the stored units explicit on the property_baseline_performance columns:
- `*_co2_emissions` → `*_co2_emissions_t_per_yr` (tonnes CO₂/yr, whole dwelling)
- `*_primary_energy_intensity` → `*_primary_energy_intensity_kwh_per_m2_yr`
Column names only; the domain `Performance` VO stays unit-suffix-free (units are
a storage concern, mapped in from_domain/to_domain). Migration doc updated.
Round-trip stays green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the handler's whole-pipeline Session (one transaction across all
three stages, connection pinned during Ingestion's external IO) with a
Unit-of-Work per stage (ADR-0012, added here). Each stage runs its batch in
one unit and commits once; any property raising aborts the batch and the
subtask fails noisily.
- BaselineOrchestrator(unit_of_work, rebaseliner): one unit for the batch,
commit once. Raise on a pre-SAP10 property leaves the unit uncommitted.
- IngestionOrchestrator(unit_of_work, epc_fetcher, geospatial_repo,
solar_fetcher): fetch/write split — phase 1 fetches the whole batch (EPC /
coords / solar) with NO unit open; phase 2 writes in one unit and commits.
The connection is never held during external IO. Geospatial S3 repo stays
injected (reference data, not transactional).
- Handler: module-scoped engine (pool reused across warm invocations) + a UoW
factory; whole-pipeline `with Session` gone. `build_first_run_pipeline`
composes on the factory. Source clients still behind the raising seam.
- ADR-0012 records the decision (per-stage boundary, all-or-nothing batch,
idempotent re-run, fetch/write split, module-scoped engine). Modelling stub
left untouched (no-op, no DB) per the ADR.
Tests: orchestrators on a shared FakeUnitOfWork (assert persisted batch +
exactly-once commit + no-commit-on-raise). New real-DB E2E integration test:
real PostgresUnitOfWork, Ingestion writes the EPC → Baseline reads it back
through the repo → re-run replaces, not duplicates (1 EPC row, 1 baseline row
after two runs). 121 pass in tests/; pyright strict clean; AAA.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage 2 of First Run. Establishes each Property's Baseline Performance
from persisted source data and writes it back — reads only from repos,
never a Fetcher or HTTP (ADR-0003), so it is byte-identical whether
Ingestion ran milliseconds ago or last week.
Domain (`domain/baseline/`):
- `Performance` VO — the four rated quantities: SAP / EPC Band / CO2 /
Primary Energy Intensity. `lodged_performance(epc)` reads them off the
EPC's recorded fields (PEUI = `energy_consumption_current`).
- `BaselinePerformance` (ADR-0004) — the paired `lodged` + `effective`
Performance + `rebaseline_reason`, plus the no-derivation part of the
energy block (`space_heating_kwh` / `water_heating_kwh`, off the RHI,
deterministic per ADR-0006). Both halves always populated.
- `Rebaseliner` port + `StubRebaseliner`: the re-score-on-override seam
(ADR-0011). SAP10 certs pass through (effective == lodged, reason
"none"); a pre-SAP10 cert raises `RebaselineNotImplemented` rather
than fabricating a plausible-but-wrong "none" — ML rebaselining is not
wired yet. Mirrors the repo's strict-raise culture.
Persistence: new `BaselineRepository` port + `BaselinePostgresRepository`
+ flat-column `baseline_performance` SQLModel (one row per Property). Per
ADR-0004's amendment this is a standalone table, NOT columns on the
retiring `property_details_epc`. Production migration is FE-owned
(Drizzle) — docs/migrations/baseline-performance-table.md.
Docs (grill-with-docs): corrected CONTEXT.md Lodged/Effective Performance
to Primary Energy Intensity (the term collided with its own _Avoid_ entry
under "heat demand") + fixed stale RHI field names; amended ADR-0004
Consequences for the standalone-table decision.
Fuel split + bills (rest of EPC Energy Derivation) deferred to a
follow-up — they need a Fuel Rates source (Ofgem-cap ETL) that does not
exist yet.
TDD, one test -> one impl: 7 tests (lodged read, rebaseliner pass-through
+ raise, orchestrator establish-and-persist + pre-SAP10 raise, Postgres
round-trip + absent). pyright strict clean; AAA layout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add epc_renewable_heat_incentive table (space_heating_kwh, water_heating_kwh +
the three insulation-impact kWh fields), wired into EpcPostgresRepository
save/get. This is the P0 gap: RenewableHeatIncentive carries the baseline
space-heating/hot-water kWh that EPC Energy Derivation consumes.
The round-trip test now asserts full deep-equality (dropped the
renewable_heat_incentive exclusion) and passes for RdSAP 21.0.0 + 21.0.1.
DB migration for the new table documented in
docs/migrations/epc-property-round-trip-fidelity.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Relocate EpcPropertyModel + child tables from the dying backend/ tree to
infrastructure/postgres/epc_property_table.py (re-export shim keeps
documents_parser working). Add EpcRepository port + EpcPostgresRepository with
a full reverse mapper (epc_property tables -> EpcPropertyData).
Round-trip test surfaced two fidelity gaps:
1. Union[int,str] SAP code fields were str()-coerced on save, losing the int
(API) vs str (Site Notes) distinction. Now stored as JSONB (type-preserving).
2. The schema was a partial projection. Closed the cheap gaps on the model
(heating shower/bath counts, roof_construction_type, curtain_wall_age,
addendum, mechanical_vent_duct_insulation_level, SAP 10.2 §2 ventilation
fields + a ventilation_present flag). Structural gaps tracked as follow-ups;
renewable_heat_incentive (P0, #1137) excluded from the assertion until landed.
Round-trip passes for RdSAP-Schema-21.0.0 and 21.0.1; pyright strict clean.
Migration inventory for the DB: docs/migrations/epc-property-round-trip-fidelity.md
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Records the grill-with-docs outcomes for the ara_first_run rebuild: three
composable stage orchestrators (Ingestion/Baseline/Modelling), one lambda per
use case chaining them through repos (not in-memory), and the Fetcher-vs-Repo
data-source taxonomy. Amends ADR-0003's chaining rule to generalise beyond
RefreshOrchestrator. Adds the pipeline-composition + First Run vocabulary to
CONTEXT.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Locality of reference — SAP-specific docs, specs, and runtime data
now live alongside the calculator that consumes them, mirroring the
prior packages→domain layout moves.
Move targets:
- Narrative MDs → domain/sap10_calculator/docs/
NEXT_AGENT_PROMPT.md, HANDOVER_NEXT.md, SAP_CALCULATOR.md
- Spec PDFs → domain/sap10_calculator/docs/specs/
RdSAP 10 Specification 10-06-2025.pdf
PCDF_Spec_Rev-06b_12_May_2021.pdf
sap-10-2-full-specification-2025-03-14.pdf
sap-10-3-full-specification-2026-01-13.pdf
- PCDB runtime data → domain/sap10_calculator/tables/pcdb/data/
pcdb10.dat (8.3MB) + 7× pcdb_table_*.jsonl (18MB total)
Path code rewrites (load-bearing):
- tables/pcdb/__init__.py: replaced parents[4]/'docs'/'sap-spec' with
Path(__file__).resolve().parent/'data' for Table 105 JSONL loading.
- tables/pcdb/postcode_weather.py: same rebase for the pcdb10.dat path
read by _postcode_climate_table().
- tables/pcdb/etl.py __main__: same rebase for the manual ETL invocation
(source + output_dir both now point inside the package).
- tests/test_pcdb_etl.py: _PCDB_DAT_PATH now derives from
parents[1]/'tables'/'pcdb'/'data' (was parents[3]/'docs'/'sap-spec').
Citation rewrites:
- 12 .py docstrings and 4 .md docs (ADRs + READMEs + narrative docs)
had `docs/sap-spec/<file>` strings rewritten to their new locations.
- Two cases where the catch-all sed misfired (an ADR-0009 line about a
PCDB extract; the pcdb __init__.py docstring about ETL output) were
hand-corrected to point at tables/pcdb/data/ rather than docs/specs/.
docs/sap-spec/ is now empty (will be removed in a follow-up sweep or
left as a vestigial empty dir for future repurposing). ADRs 0009 and
0010 remain at docs/adr/ — they're part of the chronological
cross-cutting decision log, not calculator-specific narrative.
Verified:
- Calculator's 1e-4 production gate
(test_api_001479_full_chain_sap_matches_worksheet_pdf_exactly) GREEN.
- Wider sweep (domain/sap10_calculator/ + domain/sap10_ml/): 1654
passed / 20 failed — exact pre-move baseline. All 20 failures
pre-existing (10 hand-built skeleton + 4 cohort chain + 6 cohort
diff).
- Pyright net-zero on the 4 touched runtime/test files (0 errors)
and unchanged on heat_transmission.py (13) / cert_to_inputs.py (35) /
mapper.py (33).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sibling migration to the sap10_calculator move — `domain.ml` now lives
at the root-level layout (`domain/sap10_ml/`) matching the pattern
already used by `domain.addresses`, `domain.tasks`, `domain.postcode`,
and `domain.sap10_calculator`.
Changes:
- `git mv packages/domain/src/domain/ml → domain/sap10_ml` (19 files;
history preserved).
- Subpackage rename: `domain.ml` → `domain.sap10_ml`. 32 references
rewritten across .py and .md files: 11 internal + 21 external
(datatypes/epc/domain/mapper.py, 14 files in domain/sap10_calculator,
2 backend tests, 2 ADRs, 1 README, 1 design doc).
- Path-string updates: `pytest.ini` testpath
`packages/domain/src/domain/ml/tests` → `domain/sap10_ml/tests` so
ML tests stay in the default auto-discovered sweep. `CONTEXT.md`
also updated.
`packages/domain/src/domain/` is now empty — the workspace `domain/`
tree has been fully migrated. Together with the `domain/__init__.py`
deletions from the sap10_calculator commit (29ac35cc), `domain` is
now a single root-level namespace package with subpackages
{addresses, sap10_calculator, sap10_ml, tasks} + the standalone
`postcode.py` module.
Verified:
- Focused sweep (backend mapper-chain + sap10_calculator worksheet
e2e + golden fixtures): 99 passed / 19 failed — identical baseline.
- Wider sweep (all sap10_calculator + sap10_ml): 1654 passed / 20
failed (same pre-existing failures).
- domain/sap10_ml/tests: 210/210 PASSED at new path.
- Pyright net-zero: heat_transmission.py 13, cert_to_inputs.py 35,
mapper.py 33, rdsap_uvalues.py 1 (all unchanged from baseline).
Note: `packages/domain/pyproject.toml` still declares
`packages = ["src/domain"]` for the hatchling wheel — that target
directory is now empty and the wheel build is effectively a no-op.
Retiring the workspace package or repointing the wheel is a follow-up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migration of the SAP 10.2 calculator package from the uv-workspace
src-layout (`packages/domain/src/domain/sap`) to the root-level layout
(`domain/sap10_calculator`), matching the pattern already used by
`domain.addresses` / `domain.tasks` / `domain.postcode`.
Changes:
- `git mv packages/domain/src/domain/sap → domain/sap10_calculator`
(92 files; git auto-detected all as renames so blame/history is
preserved).
- Subpackage rename: `domain.sap` → `domain.sap10_calculator`. 48
Python files rewritten (`from domain.sap.X` → `from domain.sap10_
calculator.X`); zero remaining `domain.sap` refs after the sed pass.
- Path-string updates: 3 .py files (test fixtures + xlsx loader) +
6 markdown docs (CONTEXT.md, 2 ADRs, 3 sap-spec docs, sap10_
calculator/README.md) had hard-coded `packages/domain/src/domain/
sap/...` paths rewritten to `domain/sap10_calculator/...`.
- `Path(__file__).parents[N]` rebasing: the old tree was 3 levels
deeper than the new one (`packages/domain/src/`), so 4× `parents[7]`
became `parents[4]` and 1× `parents[6]` became `parents[3]` across
`tables/pcdb/{__init__.py, postcode_weather.py, etl.py}`,
`worksheet/tests/_xlsx_loader.py`, and `tests/test_pcdb_etl.py`.
- PEP 420 namespace package: deleted both `domain/__init__.py`
(root + workspace, both load-bearing only as empty/docstring) so
Python combines `domain.sap10_calculator` (root) and `domain.ml`
(workspace) into one namespace package. Confirmed via
`domain.__path__ == ['/workspaces/model/domain',
'/workspaces/model/packages/domain/src/domain']`. Without this,
the root `domain/__init__.py` shadowed the workspace one and
`domain.ml` was unreachable.
Verified:
- Full sweep (`backend/documents_parser/tests/test_summary_pdf_
mapper_chain.py + domain/sap10_calculator/worksheet/tests/test_
e2e_elmhurst_sap_score.py + domain/sap10_calculator/rdsap/tests/
test_golden_fixtures.py`): 99 passed / 19 failed — exact same
counts as pre-refactor. All 19 failures pre-existing (9 hand-built
001479 + 6 cohort diff + 4 cohort chain non-spec).
- Wider sweep (all sap10_calculator + domain.ml): 1654 passed /
20 failed (the +1 vs the focused sweep is the pre-existing
`test_roof_insulated_assumed_with_ni_thickness_uses_50mm_per_
section_5_11_4` which was already failing on the previous baseline).
- Pyright net-zero on the three load-bearing baselines:
`heat_transmission.py` 13, `cert_to_inputs.py` 35, `mapper.py` 33.
Lift-and-shift only — no semantic renames (`Sap10Calculator` stays
`Sap10Calculator`), no testpaths edits in pytest.ini (sap tests
continue to be invoked by explicit pytest paths).
Note: `domain.ml` still lives at `packages/domain/src/domain/ml/`.
Migrating it would close out the dual-`domain/` layout but is
out of scope for this commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three load-bearing files that the post-Slice-95 tests and docs cite
but were never tracked:
1. `packages/domain/src/domain/sap/rdsap/tests/fixtures/golden/
0535-9020-6509-0821-6222.json` — API JSON for cert 001479
(Elmhurst worksheet P960-0001-001479, lodged 31 Oct 2025).
Required by `test_api_001479_full_chain_sap_matches_worksheet_pdf_
exactly` (Slice 95's Layer 4 1e-4 gate) and by
`test_golden_cert_residual_matches_pin` (residual-from-integer
pin path). Without this committed, both tests fail to find the
fixture file.
2. `docs/sap-spec/RdSAP 10 Specification 10-06-2025.pdf` — replaces
the previously-tracked `rdsap-10-specification-2025-06-10.pdf`
(same content, cleaner filename). Cited from 5 source files
(`table_32.py`, `pcdb/parser.py`, README.md, SAP_CALCULATOR.md,
NEXT_AGENT_PROMPT.md) and every spec-citation commit message
in Slices 87-95. Git auto-detected the rename.
3. `docs/sap-spec/PCDF_Spec_Rev-06b_12_May_2021.pdf` — cited from
`pcdb/parser.py:69` and the §4-water-heating combi-loss
docstrings; needed to validate the PCDB Table 3a/3b/3c routing
logic.
Also fixes the one stale reference in `test_dimensions.py:471`
that still pointed to the old `rdsap-10-specification-2025-06-10
.pdf` filename — now points to the renamed file.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Status: Slice 95 closed Layer 4 (API → cascade SAP) on cert 001479 at
< 1e-4 vs worksheet 69.0094. Production goal MET; the
`test_api_001479_full_chain_sap_matches_worksheet_pdf_exactly` test
formalises this gate. Updates to keep the next agent honest:
- NEXT_AGENT_PROMPT: header + status table + cumulative SAP delta table
+ "First action" + epilogue all reflect Slice 95's close-out.
- NEXT_AGENT_PROMPT §4 (Outlier golden cert investigations): rewrote
the cert 0240 entry. The earlier "Type-1 RR gable_wall_lengths not
extracted" claim is stale — mapper.py:1349-1369 already extracts
them (Slices 71-86). The -15 SAP residual is a mix, dominated by
the windows subsystem (11 windows × 18.28 m² with default U≈2.27
because Slice 93's `_API_GLAZING_TYPE_TO_TRANSMISSION` only covers
glazing codes 3 and 13; cert 0240 lodges code 2). Surfacing
glazing_type=2 (and likely other unmapped codes) is the biggest
single-slice leverage point — and would touch 6035 too.
- test_golden_fixtures.py cert 0240 `notes:` field: replaced the
stale RR hypothesis with the actual cascade subsystem breakdown
and the glazing_type-2 surfacing recommendation.
No production code changed; docs and a `_GoldenExpectation.notes`
string only. test_golden_fixtures.py stays GREEN (14 passed).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cert 001479 API path closed from +3.08 → +0.0006 SAP delta vs
worksheet 69.0094 in Slices 87-94. Fabric heat loss is now EXACT
across all 6 components. Replaced the prior handover (which assumed
the Elmhurst path was still RED with a 0.26 SAP gap on cohort 000474)
with the current state:
- Acceptance criterion corrected: 1e-4 against worksheet continuous
SAP (not ±0.5 against API integer) when a worksheet is available.
- Validation layer status table reflects current GREEN/RED state.
- Slice 87-94 progression captured with each fix's SAP delta impact.
- Diagnostic probe + queue documented for next agent: close 001479's
residual +0.0006 (HW + gains), write Layer 3 diff test, then
process new cert pairs as user sources them.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
User reframed the end goal explicitly: the production flow is
`API JSON → EpcPropertyDataMapper.from_api_response → SAP calculator`
landing within ±0.5 of the API-published SAP. The Elmhurst-site-notes
work is the cross-validation route — same dwelling, independent path
into EpcPropertyData. Once both routes agree on cert 001479, the API
mapper is validated by transitivity.
Restructure the handover around four nested validation layers:
Layer 1 (hand-built cascade pin): 6 cohort certs GREEN; 001479 partial
Layer 2 (Elmhurst ≡ hand-built): cohort 000474 GREEN; 5 others pending
Layer 3 (API ≡ Elmhurst): test doesn't exist yet
Layer 4 (API cascade ±0.5): 72.08 vs 69 (delta +3.08)
Each layer validates the one below. Closing inner-most first means
upper layers can lean on it as reference.
Documents tools/patterns built in slices 63-70:
- `_LOAD_BEARING_FIELDS` allow-list (~40 cascade/semantic fields)
- `_NON_LOAD_BEARING_WINDOW_SUBFIELDS` deny-list (descriptive int/str
encoding noise)
- `_diff_load_bearing` recursive helper (strict-pyright-clean)
- `test_from_elmhurst_site_notes_matches_hand_built_NNNNNN` tracer-
bullet pattern (000474 is the worked example)
Next-step ordering: parametrize over 5 other cohort certs, complete
001479 hand-built (currently 2/11 cascade pins green; gap −3.02 SAP),
add cert 001479 to diff test, then add API mapper → hand-built diff
test, then the production-flow acceptance pin in test_golden_fixtures
for cert 001479.
Lists source-data caveats (the M-vs-L Ext1 age discrepancy on 001479).
Conventions to honour (AAA, abs(diff)<=tol, one slice=one commit,
1e-4 Elmhurst / 0.5 API, no widening, pyright net-zero). Cached
artefacts (golden JSON, Summary PDF, worksheet PDF) noted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Update NEXT_AGENT_PROMPT.md with the pivot to the rigorous cohort
pattern: cert 001479's hand-built `_elmhurst_worksheet_001479.py`
becomes the ground-truth EpcPropertyData. Cross-mapper parity work
then collapses to "both mappers produce hand-built-equivalent
EpcPropertyData".
Two parallel workstreams documented:
1. Iterate the hand-built skeleton (Slice 62) until all 11 cascade
pins hit 1e-4. Current state: 2/11 green (pumps_fans, lighting);
sap_score_continuous gap −3.02 SAP. Likely next slices: HW demand
routing, §2 ventilation tuning, thermal mass parameter, multiple-
glazed proportion.
2. Once hand-built is GREEN, add `test_elmhurst_mapper_matches_hand_
built` + `test_api_mapper_matches_hand_built` over the 7-cert
cohort (000474..000516 + 001479). Every field diff = mapper bug
to close. Cross-parity collapses to "both mappers produce
hand-built-equivalent".
Documents the M-vs-L Ext1 age-band source-data conflict (hand-built
uses worksheet's L; Elmhurst mapper trusts Summary's M) — surfaces
as a known caveat in cross-mapper diff.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Update NEXT_AGENT_PROMPT.md for the TDD session that landed 3 more
slices on top of Session 1's fabric work:
58: secondary fuel cost routes through lodged secondary_fuel_type
(closes the biggest single gap on cert 001479 — 9 SAP)
59: heat_transmission apportions windows per bp via window_location
60: thermal bridging y uses primary bp's age (dwelling-wide)
Chain pin `test_summary_001479_full_chain_sap_matches_worksheet_pdf_
exactly` is committed RED as the load-bearing TDD forcing function:
Pre-workstream: delta +5.84 SAP (cascade 63.17 vs target 69.0094)
Post-Slice 60: delta −1.19 SAP (cascade 70.20 vs target 69.0094)
Per-bp fabric U-values all match the worksheet exactly. Remaining
1.19 SAP overshoot maps to ~3 W/K of HLC undercount in roof + floor:
- Ext2 PS sloping-ceiling roof area uses floor projection (1.92 m²)
instead of slant area (2.22 m²). −0.81 W/K.
- Main ground-floor U: `u_floor` Table 19 returns 0.60 for age C;
worksheet expects 0.65 (same as age B). −1.52 W/K.
- (31) external area under-count drives bridging gap. −2.08 W/K.
Slice 61 (SapFloorDimension.floor_lodged_u_value override using
Summary §9 "Default U-value") was attempted and reverted: closed
001479 floor gap exactly but broke 000474 cohort's 1e-4 pin (its
cascade calibration uses u_floor age-B 0.77 vs Summary's lodged
0.75). Next session needs a different fix — Table 19 audit for
age C, or selective override.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
