diff --git a/.claude/skills/expand-sap-accuracy-corpus/worklist.md b/.claude/skills/expand-sap-accuracy-corpus/worklist.md index dbb1cea00..d5c40292c 100644 --- a/.claude/skills/expand-sap-accuracy-corpus/worklist.md +++ b/.claude/skills/expand-sap-accuracy-corpus/worklist.md @@ -387,30 +387,39 @@ Skip the ๐Ÿšฉ MVHR / ๐Ÿšฉ heat-pump-fuel and โ›” sparse certs. - [ ] 10013924858 โ€” SAP-17.0 ยท eng 81 / lodged 84 # SAP-Schema-15.0 mapper validation (task a40e71c4-fd56-4a33-8677-e8643e6a5bb4) -๐Ÿ”ง New schema โ€” `from_sap_schema_15_0` added in mapper.py (PR #1531). No full -Elmhurst rebuild done for this batch: 16/20 land exact or within 1-3 of lodged -(strong signal alone, given each other case in this file needed a full rebuild -to reach that confidence); the other 4 share a diagnosed, non-mapper cause -(solid-brick U-value confirmed correct against RdSAP10 Table 6 directly โ€” see -comment in test_real_cert_sap_accuracy.py) so pinned without a rebuild too. All -20 pinned as `RealCertExpectation`s. -- [x] ๐Ÿ”ง 100010359769 โ€” SAP-15.0 ยท eng 68 (lodged 68) -- [x] ๐Ÿ”ง 100010359788 โ€” SAP-15.0 ยท eng 72 (lodged 73) -- [x] ๐Ÿ” 100010086084 โ€” SAP-15.0 ยท eng 53 (lodged 45) ยท solid-brick vintage-methodology gap, not mapper โ€” see test file comment -- [x] ๐Ÿ” 100010090369 โ€” SAP-15.0 ยท eng 53 (lodged 47) ยท ditto -- [x] ๐Ÿ” 100010074470 โ€” SAP-15.0 ยท eng 46 (lodged 39) ยท ditto -- [x] ๐Ÿ” 100010100813 โ€” SAP-15.0 ยท eng 59 (lodged 47) ยท ditto (largest gap, +12) -- [x] ๐Ÿ”ง 77179816 โ€” SAP-15.0 ยท eng 76 (lodged 75) -- [x] ๐Ÿ”ง 10070400619 โ€” SAP-15.0 ยท eng 77 (lodged 77) -- [x] ๐Ÿ”ง 77179820 โ€” SAP-15.0 ยท eng 77 (lodged 77) -- [x] ๐Ÿ”ง 10070394806 โ€” SAP-15.0 ยท eng 78 (lodged 77) -- [x] ๐Ÿ”ง 77181233 โ€” SAP-15.0 ยท eng 68 (lodged 69) -- [x] ๐Ÿ”ง 77180626 โ€” SAP-15.0 ยท eng 68 (lodged 68) -- [x] ๐Ÿ”ง 10070400618 โ€” SAP-15.0 ยท eng 70 (lodged 69) -- [x] ๐Ÿ”ง 100010350459 โ€” SAP-15.0 ยท eng 65 (lodged 65) -- [x] ๐Ÿ”ง 100010350468 โ€” SAP-15.0 ยท eng 66 (lodged 66) -- [x] ๐Ÿ”ง 100010333531 โ€” SAP-15.0 ยท eng 73 (lodged 74) -- [x] ๐Ÿ”ง 100010341489 โ€” SAP-15.0 ยท eng 65 (lodged 64) -- [x] ๐Ÿ”ง 100010341491 โ€” SAP-15.0 ยท eng 62 (lodged 59) -- [x] ๐Ÿ”ง 100010347646 โ€” SAP-15.0 ยท eng 75 (lodged 75) -- [x] ๐Ÿ”ง 100010347637 โ€” SAP-15.0 ยท eng 72 (lodged 72) +๐Ÿ”ง New schema โ€” `from_sap_schema_15_0` added in mapper.py (PR #1531). No +Elmhurst rebuild done for this batch โ€” and unlike every other entry in this +file, **lodged is NOT the comparison target here**: these are 2011 SAP-2009 +lodgements, but this calculator implements current RdSAP10, which has since +revised several U-value tables. A gap from lodged is expected either way and +is not, by itself, evidence of a mapper bug โ€” nor is closeness to lodged +evidence of correctness. What this batch actually validates: all 20 +structurally distinct real certs (different fuels, meters, secondary-heating +types, wall constructions) map without error and produce plausible scores +through `from_sap_schema_15_0`'s three schema-wide defaults. The 4 solid-brick +cases below show the largest gap from lodged; `u_wall` was spot-checked +directly against RdSAP10 Table 6 for that construction/age band and returns +the spec-correct value, so at minimum the wall U-value lookup isn't the +source. Real accuracy validation (engine vs Elmhurst on the SAME RdSAP10 +methodology) is still open โ€” see comment in test_real_cert_sap_accuracy.py. +All 20 pinned as `RealCertExpectation`s to the engine's OWN observed output. +- [x] ๐Ÿ”ง 100010359769 โ€” SAP-15.0 ยท eng 68 (2011 lodged 68, not a target) +- [x] ๐Ÿ”ง 100010359788 โ€” SAP-15.0 ยท eng 72 (2011 lodged 73, not a target) +- [x] ๐Ÿ”ง 100010086084 โ€” SAP-15.0 ยท eng 53 (2011 lodged 45, not a target) ยท solid-brick, largest-gap group; u_wall spot-checked correct +- [x] ๐Ÿ”ง 100010090369 โ€” SAP-15.0 ยท eng 53 (2011 lodged 47, not a target) ยท ditto +- [x] ๐Ÿ”ง 100010074470 โ€” SAP-15.0 ยท eng 46 (2011 lodged 39, not a target) ยท ditto +- [x] ๐Ÿ”ง 100010100813 โ€” SAP-15.0 ยท eng 59 (2011 lodged 47, not a target) ยท ditto +- [x] ๐Ÿ”ง 77179816 โ€” SAP-15.0 ยท eng 76 (2011 lodged 75, not a target) +- [x] ๐Ÿ”ง 10070400619 โ€” SAP-15.0 ยท eng 77 (2011 lodged 77, not a target) +- [x] ๐Ÿ”ง 77179820 โ€” SAP-15.0 ยท eng 77 (2011 lodged 77, not a target) +- [x] ๐Ÿ”ง 10070394806 โ€” SAP-15.0 ยท eng 78 (2011 lodged 77, not a target) +- [x] ๐Ÿ”ง 77181233 โ€” SAP-15.0 ยท eng 68 (2011 lodged 69, not a target) +- [x] ๐Ÿ”ง 77180626 โ€” SAP-15.0 ยท eng 68 (2011 lodged 68, not a target) +- [x] ๐Ÿ”ง 10070400618 โ€” SAP-15.0 ยท eng 70 (2011 lodged 69, not a target) +- [x] ๐Ÿ”ง 100010350459 โ€” SAP-15.0 ยท eng 65 (2011 lodged 65, not a target) +- [x] ๐Ÿ”ง 100010350468 โ€” SAP-15.0 ยท eng 66 (2011 lodged 66, not a target) +- [x] ๐Ÿ”ง 100010333531 โ€” SAP-15.0 ยท eng 73 (2011 lodged 74, not a target) +- [x] ๐Ÿ”ง 100010341489 โ€” SAP-15.0 ยท eng 65 (2011 lodged 64, not a target) +- [x] ๐Ÿ”ง 100010341491 โ€” SAP-15.0 ยท eng 62 (2011 lodged 59, not a target) +- [x] ๐Ÿ”ง 100010347646 โ€” SAP-15.0 ยท eng 75 (2011 lodged 75, not a target) +- [x] ๐Ÿ”ง 100010347637 โ€” SAP-15.0 ยท eng 72 (2011 lodged 72, not a target) diff --git a/tests/domain/sap10_calculator/test_real_cert_sap_accuracy.py b/tests/domain/sap10_calculator/test_real_cert_sap_accuracy.py index 8e5ef0ff8..b3c029dff 100644 --- a/tests/domain/sap10_calculator/test_real_cert_sap_accuracy.py +++ b/tests/domain/sap10_calculator/test_real_cert_sap_accuracy.py @@ -1091,24 +1091,36 @@ _EXPECTATIONS: Final[tuple[RealCertExpectation, ...]] = ( # SAP-Schema-15.0 campaign (task a40e71c4-fd56-4a33-8677-e8643e6a5bb4, # portfolio 824 / scenario 1278 โ€” the failing task that surfaced this # schema gap; see `from_sap_schema_15_0` in mapper.py). 20 real 2011-era - # LIG-15.0 lodgements sampled across the failing portfolio. 16 of 20 land - # exact or within 1-3 of the lodged score, confirming the mapper's three - # schema-wide defaults (door_count=1, WWHRS/FGHRS "none fitted", - # percent_draughtproofed=0) are sound. + # LIG-15.0 lodgements sampled across the failing portfolio. # - # The remaining 4 (086084/090369/074470/100813) share solid-brick - # UNINSULATED walls, age band C, and diverge from lodged by +6 to +12 โ€” - # every other affected field (fuel, meter type, secondary heating type) - # varies independently across them, so wall construction is the only - # correlate. Checked `u_wall(Country.ENG, "C", 3, None, - # wall_insulation_type=4)` directly: it returns 1.70, RdSAP10's own - # published Table 6 value for solid 220-280mm brick in that age band โ€” no - # mapper/U-value bug found. These are 2011 lodgements, assessed under the - # older SAP-2009 methodology's (higher/more conservative) solid-wall - # U-values; re-scoring under current RdSAP10 legitimately lands higher. - # Same category as the documented full-SAP-through-RdSAP and reduced- - # field party-wall vintage gaps elsewhere in this file โ€” PINNED to the - # engine's observed value, not lodged; not tuned. + # NOTE on what these pins do and don't validate: `sap_score` here is the + # ENGINE'S OWN observed output, not an accuracy claim against the lodged + # `energy_rating_current`. These are 2011 certs, assessed under the SAP + # 2009 methodology of the day; this calculator implements the CURRENT + # RdSAP10 methodology, which has since revised several U-value tables + # (solid-wall thickness bands among them โ€” see `u_wall` ยง5.7 Table 13). + # A 2011-vs-2026-methodology delta is therefore EXPECTED and tells you + # nothing about mapper correctness either way โ€” closeness to lodged is + # not evidence for the mapper, and a gap from lodged is not evidence + # against it. Real accuracy validation needs an Elmhurst RdSAP10 rebuild + # (the `validate-cert-sap-accuracy` skill) on the SAME methodology as the + # engine, which this batch did not do. + # + # What this batch DOES validate: all 20 structurally distinct real certs + # (different fuels, meter types, secondary-heating types, wall + # constructions, portfolios) map without error through the new + # `from_sap_schema_15_0` path and produce plausible SAP scores โ€” i.e. the + # three schema-wide defaults (door_count=1, WWHRS/FGHRS "none fitted", + # percent_draughtproofed=0) don't crash or produce nonsense on a diverse + # sample. Four of the twenty (086084/090369/074470/100813, sharing + # solid-brick UNINSULATED walls / age band C) show the largest gap from + # lodged; spot-checked `u_wall(Country.ENG, "C", 3, None, + # wall_insulation_type=4)` directly and it returns 1.70, matching RdSAP10's + # own published Table 6 for that construction/age band โ€” so at minimum + # the wall U-value lookup itself is not the source of any divergence. + # PINNED to the engine's observed value on every case (the file's + # standing convention) โ€” not tuned to lodged, and not claimed as an + # accuracy match either. RealCertExpectation( schema="SAP-Schema-15.0", sample="uprn_100010359769",