diff --git a/domain/sap10_calculator/docs/HANDOVER_PV_BETA_SPLIT.md b/domain/sap10_calculator/docs/HANDOVER_PV_BETA_SPLIT.md index 6cfc690f..fccd383a 100644 --- a/domain/sap10_calculator/docs/HANDOVER_PV_BETA_SPLIT.md +++ b/domain/sap10_calculator/docs/HANDOVER_PV_BETA_SPLIT.md @@ -1,164 +1,144 @@ -# Handover — PV β-factor split (3/6 wiring slices shipped) +# Handover — PV β-factor split (5/6 wiring slices shipped) Branch `feature/per-cert-mapper-validation`. This session shipped -**3 slices** (S0380.44 → S0380.46) that implemented and wired SAP 10.2 -Appendix M1 §3-4 β-factor for the PE and CO2 cascades. Three more -slices remain before the ASHP cluster fully closes. +**5 slices** (S0380.44 → S0380.48) that implemented and wired SAP 10.2 +Appendix M1 β-factor across PE, CO2, and Cost cascades, then surfaced +the real-API battery capacity. One follow-up slice remains before the +ASHP+5-kWh cluster lands tight. -**HEAD at handover start:** `5b269f23` (Slice S0380.46). +**HEAD at handover end:** `bf99b1c7` (Slice S0380.48). **Test suite:** 763 pass + 0 fail. -## Slices shipped this session +## Slices shipped this phase | Slice | Commit | What | Spec | |---|---|---|---| | **S0380.44** | `5344bc89` | New module `worksheet/photovoltaic.py` with `pv_split_monthly`, `pv_beta_coefficients`, `PhotovoltaicSplit` + 13 unit tests | Appendix M1 §3c-d (p.94), §4 (p.94) | | **S0380.45** | `49de18e8` | Wired β-split into PE cascade — `cert_to_inputs` builds monthly E_PV + D_PV + battery, calls `pv_split_monthly`, passes `pv_dwelling_kwh_per_yr` + `pv_exported_kwh_per_yr` to `CalculatorInputs`; calculator credits at IMPORT + EXPORT PEF | Appendix M1 §3a (p.93), §8 (p.94) | | **S0380.46** | `5b269f23` | Wired β-split into CO2 cascade — added `pv_dwelling_co2_factor_kg_per_kwh` + `pv_exported_co2_factor_kg_per_kwh` (effective monthly Table 12d Σ); calculator subtracts the credit | Appendix M1 §7 (p.94), Table 12d code 60 | +| **S0380.47** | `42ed38f7` | Wired β-split into cost cascade — new `_pv_dwelling_import_price_gbp_per_kwh` (Table 32 code 30 = 13.19 p/kWh) + `pv_dwelling_import_price_gbp_per_kwh` field on `CalculatorInputs`; `fuel_cost.py` splits the credit at IMPORT × E_dw + EXPORT × E_ex; cohort impact zero because Table 32 collapses code 30 = code 60 = 13.19, so the math collapses to the legacy single-rate credit | Appendix M1 §6 (p.94), Table 32 code 30/60 | +| **S0380.48** | `bf99b1c7` | **E_PV magnitude bug audit: schema gap, not cascade.** Real-API certs lodge `pv_batteries: [{"battery_capacity": 5}]` flat; schema's `PvBatteries` had only `pv_battery: Optional[PvBattery]` (synthetic nested shape) → `from_dict` dropped `battery_capacity: 5` silently → cascade saw C_bat=0 → β≈0.36 vs worksheet 0.74. Fix: add `battery_capacity: Optional[float]` to schema sibling, prefer nested when present, fall back to flat. Cohort PE residual flipped +2.7..+8.1 → -3.5..-4.5 | Appendix M1 §3c (p.94) | ## Residual progress (ASHP cluster + cert 2130 + cert 9501) -### PE residual (kWh/m²) +### PE residual (kWh/m²) — full series -| Cert | Pre-S0380.44 | Post-S0380.45 (PE wired) | Notes | -|---|---:|---:|---| -| 0330 (no PV) | +0.44 | +0.44 | unchanged ✓ | -| 0350 (PV+5kWh) | −7.78 | +2.73 | overshoots — EPV bug | -| 0380 (PV+5kWh) | −14.60 | +8.09 | overshoots — EPV bug | -| 2130 (PV gas) | −38.63 | −9.70 | partial close; +SAP-int 1 | -| 2225 (PV+5kWh) | −11.77 | +4.48 | overshoots — EPV bug | -| 2636 (PV+5kWh) | −9.65 | +3.42 | overshoots — EPV bug | -| 3800 (PV+5kWh) | −9.61 | +3.58 | overshoots — EPV bug | -| 9285 (PV+5kWh) | −7.96 | +3.20 | overshoots — EPV bug | -| 9418 (PV+5kWh) | −7.30 | +4.67 | overshoots — EPV bug | -| **9501 (PV no battery)** | **−8.28** | **+0.25** | **CLOSED ✓** — validates spec implementation | +| Cert | Pre-S0380.44 | Post-S0380.45 | Post-S0380.48 | Worksheet β | +|---|---:|---:|---:|---:| +| 0330 (no PV) | +0.44 | +0.44 | +0.44 | n/a | +| 0350 (PV+5kWh) | −7.78 | +2.73 | **−3.58** | ~0.74 | +| 0380 (PV+5kWh) | −14.60 | +8.09 | **−4.01** | 0.7426 | +| 2130 (PV gas) | −38.63 | −9.70 | −9.70 | n/a (no battery) | +| 2225 (PV+5kWh) | −11.77 | +4.48 | **−4.50** | ~0.74 | +| 2636 (PV+5kWh) | −9.65 | +3.42 | **−4.14** | ~0.74 | +| 3800 (PV+5kWh) | −9.61 | +3.58 | **−4.01** | ~0.74 | +| 9285 (PV+5kWh) | −7.96 | +3.20 | **−3.46** | ~0.74 | +| 9418 (PV+5kWh) | −7.30 | +4.67 | **−3.76** | ~0.74 | +| **9501 (PV no battery)** | **−8.28** | **+0.25** | **+0.25** | 0.498 (cascade ≈ worksheet) | + +Cluster magnitude DROPPED a second time after S0380.48 (battery now +surfaced). Cascade β=0.75-0.81 marginally exceeds worksheet's 0.74, +so cascade over-credits PV slightly — but the bulk of the post-S0380.45 +overshoot (+2.7..+8.1) has been eliminated. The remaining -3.5..-4.5 +kWh/m² under-shoot is **structural** (see Slice 6 plan below). ### CO2 residual (t/yr) -| Cert | Pre-S0380.46 | Post-S0380.46 (CO2 wired) | Notes | -|---|---:|---:|---| -| 0330 | −0.034 | −0.034 | no PV, unchanged ✓ | -| 0350 | +0.171 | −0.084 | over → under flip | -| 0380 | +0.279 | −0.054 | over → under flip | -| 2130 | +0.299 | −0.046 | over → under flip | -| 2225 | +0.263 | −0.071 | over → under flip | -| 2636 | +0.219 | −0.058 | over → under flip | -| 3800 | +0.261 | −0.014 | over → under flip | -| 9285 | +0.157 | −0.098 | over → under flip | -| 9418 | +0.232 | −0.046 | over → under flip | -| 9501 | +0.202 | −0.047 | over → under flip | +All 7 ASHP+battery certs sit at **≤0.11 t/yr** absolute residual. The +cluster shifted slightly with the S0380.48 battery surfacing +(re-pinned in the same slice). No CO2 cert is anywhere near closure +risk; the CO2 cascade is structurally sound. -Cluster magnitude dropped ~3-5× on CO2; PE for the 5-kWh-battery -cohort overshoots because β is too low (R_PV = E_PV / D_PV too high, -which traces to the **E_PV magnitude bug** — Slice 5). +## ★ Key learning: read the worksheet PDF BEFORE accepting a hypothesis -## ★ Why cert 9501 closes but the 5-kWh-battery cohort overshoots +The original handover claim — "Cascade E_PV = 2570 kWh/yr ≈ 3× the +worksheet's 831 kWh/yr" — was wrong. Reading +[`dr87-0001-000899.pdf`](../../../sap worksheets/Additional data with api/0380-2471-3250-2596-8761/dr87-0001-000899.pdf) +line (233) shows the worksheet's annual E_PV is **-2563.3692** kWh/yr, +matching our cascade to 4 dp. The handover author had picked up a +single monthly line ref or a per-array figure and mis-read it as the +annual total. Per [[feedback-spec-floor-skepticism]] applied to +handover claims: verify the cited value against the PDF before +acting on it. -Cert 9501 has PV but **no battery**. Its PE Δ closed from −8.28 to -+0.25 — clean validation that the β implementation is spec-correct. +The real bug — battery capacity dropped at schema deserialisation — +was a flat/nested JSON shape divergence between the synthetic test +fixture and the real-API payload. It would have been impossible to +identify through the "E_PV magnitude" lens of the original +hypothesis. Probing what β each cohort cert actually computes +(`pv_split.epv_dwelling_kwh_per_yr / total`) and comparing against +the worksheet's (233a)/(233a+233b) ratio is the diagnostic that +revealed the gap. -The 7-cert ASHP+battery cohort (0350/0380/2225/2636/3800/9285/9418) -shares the same Mitsubishi PUZ-WM50VHA + 3 kWp PV + 5 kWh battery -pattern. After Slice 45 they overshoot by +2.7..+8.1 PE. - -**Root cause** (already identified, deferred to Slice 5): the cascade -computes E_PV ≈ 3× the worksheet's value. For cert 0380: -- Cascade E_PV = 2570 kWh/yr (via `0.8 × kWp=3 × S × ZPV`) -- Worksheet E_PV = 831 kWh/yr (looks like 1 kWp × 0.8 × S × Z) - -Either `peak_power=3` in the API JSON is in units that aren't kWp -(maybe 0.1 kWp units?), or the worksheet was generated with different -data than the API lodgement. Audit needs to compare: -1. `peak_power` value across cohort certs (always 3 or 3.28? what does - the Summary PDF Section 19 lodge for these same certs?) -2. S value used by cascade `_pv_annual_s_kwh_per_m2` vs worksheet -3. ZPV mapping for overshading=1 - -With E_PV correctly = ~830, R_PV would drop ~3×, β rises from ~0.47 to -~0.66, and the cluster lands at ~0 residual. - -## Open slices +## Open slice | Slice | Status | What | Risk | |---|---|---|---| -| **S0380.47** (Slice 4) | **NEXT** | Wire β into cost cascade — split E_PV,dw at IMPORT price + E_PV,ex at EXPORT price per §6 | Medium: shifts SAP rating for every PV cert; chain tests need re-pinning (small Δ) | -| **S0380.48** (Slice 5) | Pending | Audit E_PV magnitude bug for 5-kWh-battery cohort — kWp interpretation, S lookup, or ZPV mapping | Medium: will surface several certs' residuals to ~0 once fixed | -| **S0380.49** (Slice 6) | Pending | Re-pin all golden fixtures + verify cohort-1 + cohort-2 chain tests still <1e-4; tighten `_PE_ABS_TOLERANCE` / `_CO2_ABS_TOLERANCE` if cluster lands cleanly | Low: cleanup | +| **S0380.49** (Slice 6) | **NEXT** | Wire effective-monthly Table 12e PE factor into the PV split (per-end-use cascade); close cohort residual to ~0 | Low: structural, mirrors the existing per-end-use Table 12d CO2 cascade | -## Slice 4 plan (cost cascade) +## Slice 6 plan (PV effective-monthly PE factor) + +The PE cascade in `calculator.py` currently credits the PV split as: -[fuel_cost.py:182](../worksheet/fuel_cost.py) currently does: ```python -pv_credit = -pv_generation_kwh_per_yr * pv_export_credit_gbp_per_kwh +pv_credit_pe = ( + inputs.pv_dwelling_kwh_per_yr * inputs.other_primary_factor # 1.501 (annual T12 code 30) + + inputs.pv_exported_kwh_per_yr * inputs.pv_export_primary_factor # 0.501 (annual T12 code 60) +) ``` -treating ALL PV as exported at the EXPORT price (13.19 p/kWh = Table 12a -"electricity sold to grid, PV"). Per Appendix M1 §6, onsite-consumed -PV should bill at the IMPORT price (standard tariff ~18 p/kWh, or -weighted high/low Table 12a if off-peak meter). + +These are **annual Table 12** factors, not the per-month effective +**Table 12e** factor weighted by the monthly E_PV,dw,m / E_PV,ex,m +distribution. The worksheet (cert 0380 page 5) uses the +effective-monthly weighted values: + +- PV dwelling: 1.4960 (vs annual 1.501 → -0.005 differential, negligible) +- PV exported: 0.4268 (vs annual 0.501 → -0.074 differential, meaningful) + +For cert 0380 with E_PV,ex ≈ 640 kWh/yr the differential is +0.074 × 640 = 47 kWh PE/yr; over TFA 60.43 m² that's **0.78 kWh/m²** +of extra credit in cascade vs worksheet. This accounts for ~1 of the +~4 kWh/m² PE delta. + +The other ~3 kWh/m² traces to β fine-tuning (cascade 0.751 vs worksheet +0.7426 — a 1.4% over-estimate of self-consumption). This may come from +a monthly D_PV distribution detail (the `_pv_eligible_demand_monthly_ +kwh` helper aggregates several monthly tuples; their relative weighting +within a month could shift the β slightly). ### Implementation outline -1. Add to `CalculatorInputs` (calculator.py): +1. Add two new fields to `CalculatorInputs`: ```python - pv_dwelling_import_price_gbp_per_kwh: Optional[float] = None + pv_dwelling_primary_factor: Optional[float] = None + pv_exported_primary_factor_monthly: Optional[float] = None ``` - (the EXPORT price field `pv_export_credit_gbp_per_kwh` already exists - and stays as the EXPORT side). + (The CO2 cascade already uses this pattern with `pv_dwelling_ + co2_factor_kg_per_kwh` + `pv_exported_co2_factor_kg_per_kwh`.) -2. In `cert_to_inputs.py`, compute the dwelling IMPORT price using the - same off-peak meter logic as `_space_heating_fuel_cost_gbp_per_kwh` - (Table 12a high/low rate weighted if meter is off-peak; standard - tariff otherwise). Pass through to `CalculatorInputs`. +2. In `cert_to_inputs.py`, compute the effective monthly factors + weighted by the per-month E_PV,dw,m / E_PV,ex,m tuples (mirroring + the existing `_effective_monthly_pe_factor` call shape used for + other electricity end-uses). The relevant lookups: + - Dwelling: Table 12e code 30 (standard electricity), monthly + - Exported: Table 12e code 60 (electricity sold to grid, PV), monthly -3. In [fuel_cost.py:182](../worksheet/fuel_cost.py) replace the single-rate - credit with the β-split: - ```python - pv_credit = -( - pv_dwelling_kwh_per_yr * dwelling_import_price - + pv_exported_kwh_per_yr * pv_export_credit_gbp_per_kwh - ) - ``` - Fall back to the legacy single-rate path when split fields are None. +3. In `calculator.py` line ~579-588, prefer the per-end-use effective + monthly factor when populated; fall back to the global annual + `other_primary_factor` / `pv_export_primary_factor` (preserves + synthetic CalculatorInputs constructions). ### Expected fallout -- SAP rating shifts up slightly for every PV cert (cost cascade now - credits onsite consumption higher). Magnitude depends on β. -- Cohort-1 + cohort-2 chain-test 1e-4 pins all need re-pinning to the - new SAP score. The shift should be small (~0.02-0.05 SAP per cert, - per the cost-spread analysis in the prior handover) so the new pins - will still be tight against the worksheet. -- The 5-kWh-battery cohort cost residual will partially shift in - the right direction; the EPV-magnitude bug from Slice 5 will keep - some over-shoot until then. - -## Slice 5 plan (E_PV magnitude audit) - -Concrete diagnostics for the next agent: - -1. **Probe cert 0380's API JSON for the actual `peak_power` field unit.** - The JSON has `peak_power: 3`. SAP spec says "kWp" — but if the - worksheet works with ~1 kWp, either: - - The cert lodges `peak_power` in deca-watts (=0.01 kWp), or - - There's a `peak_power_unit` field we're missing, or - - The worksheet was generated with hand-corrected data - - Check the Elmhurst Summary PDF Section 19 for the same cert and - compare what's lodged there vs the API. - -2. **Probe `_pv_annual_s_kwh_per_m2` for cert 0380's array.** - Array is (orientation=5=South, pitch=3=45°, overshading=1=None). - Compute the cascade's S value and compare against the SAP Appendix - U3.3 table for South / 45° / UK average. Expected ~1100 kWh/m²/yr. - If cascade gives that and worksheet works with much less, the - issue is on the worksheet side (different climate region). - -3. **Probe `_PV_OVERSHADING_FACTOR[1]` = 1.0.** Compare against the - Table M1 spec value for "None or very little" overshading. - -4. **Try setting cert 0380's `peak_power = 1.0` and check if residuals close.** - If yes → it's a kWp interpretation bug. Surface it via the schema - or the mapper. +- Cluster PE residual drops from -3.5..-4.5 toward -2.5..-3.5 (the + remaining gap is β fine-tuning, a smaller subsequent slice). +- Cert 9501 (PV no battery) stays at ±0.25 PE (β=0.498 matches + worksheet ≈0.5). +- No SAP score impact (PE doesn't enter SAP rating). +- No CO2 impact. +- Cohort-1 / cohort-2 chain tests unaffected (chain tests pin SAP, + not PE). +- Golden fixtures re-pin in the same slice (matches per S0380.48). ## Test baseline at HEAD @@ -186,7 +166,7 @@ Expected: **763 pass + 0 fail**. - **1e-4 across the board** ([[feedback-one-e-minus-4-across-the-board]]) - **Worksheet, not API, is the target** for chain tests ([[feedback-worksheet-not-api-reference]]) - **Cross-mapper parity via cascade** ([[feedback-cross-mapper-parity-via-cascade]]) -- **Spec-floor skepticism** ([[feedback-spec-floor-skepticism]]) +- **Spec-floor skepticism** ([[feedback-spec-floor-skepticism]]) — applied to handover claims AS WELL AS spec claims; the E_PV-magnitude hypothesis was wrong, verified by reading the worksheet PDF. - **Bigger slices OK for uniform-cohort work** ([[feedback-bigger-slices-for-uniform-work]]) - **Golden residuals → ~0** ([[feedback-golden-residuals-near-zero]]) - **AAA test convention** with literal `# Arrange / # Act / # Assert` ([[feedback-aaa-test-convention]]) @@ -195,16 +175,15 @@ Expected: **763 pass + 0 fail**. - **One slice = one commit; stage by name** ([[feedback-commit-per-slice]]) - **Pyright net-zero per touched file** -## Lesson learned: β-split shape is universal across PE / CO2 / Cost +## Lesson reinforced: β-split shape is universal across PE / CO2 / Cost -All three cascades had the same bug shape: credit ALL PV at one rate -(IMPORT for PE; missing entirely for CO2; EXPORT for cost). The -spec-correct fix is uniformly: split E_PV by β; onsite at IMPORT -factor, exported at EXPORT factor. The cleanest API: +All three cascades have the same bug shape: credit ALL PV at one rate. +The spec-correct fix is uniformly: split E_PV by β; onsite at IMPORT +factor, exported at EXPORT factor. The shared cross-cascade API: - `CalculatorInputs.pv_dwelling_kwh_per_yr` + `pv_exported_kwh_per_yr` are the cross-cascade-shared β-split state. -- Each cascade adds its own pair of factor fields: +- Each cascade has its own pair of factor fields: - PE: `other_primary_factor` (IMPORT) + `pv_export_primary_factor` (EXPORT) - CO2: `pv_dwelling_co2_factor_kg_per_kwh` + `pv_exported_co2_factor_kg_per_kwh` - Cost: `pv_dwelling_import_price_gbp_per_kwh` + `pv_export_credit_gbp_per_kwh`