Model/docs/grill-sessions/2026-06-10-pre-sap10-mapper-generalization.md

Grill spec — generalise Reduced-Field Synthesis to the rest of the pre-SAP10 RdSAP family

Date: 2026-06-10 · Branch: feature/junte+khalim · Status: SPEC — READY TO GRILL.

Grill this by running /grill-me and feeding it this file. Start at Q1 (ROOT).

---

Why this exists

The RdSAP 20.0.0 mapper now works end-to-end: all 1000 corpus certs parse, map via Reduced-Field Synthesis (ADR-0027), and score through Sap10Calculator without crashing. scripts/eon/find_epc_data.py shows lodged-vs-our-calculated SAP side by side and the deltas are sane (mostly ±7, same band). The pattern is proven.

The goal now: apply the same playbook to the other pre-SAP10 RdSAP specs so historical EPC data across more lodgement years can be Rebaselined. This is pure leverage — the hard design thinking (synthesis coefficients, Validation-Cohort rule, schema-fix mechanism) is already done; what remains is per-spec drift and a decision about how much to share vs copy.

What we already hold (the repeatable 20.0.0 playbook)

Each step below is proven for 20.0.0. The grill is about which steps change per spec.

1. Harvest a corpus — scripts/eon/harvest_certs.py streams a local bulk dump (downloads/certificates-YYYY.json) for the year that spec dominates, caps at 1000, writes backend/epc_api/json_samples/<schema>/corpus.jsonl. No API token needed.
2. Fix the placeholder schema — every rdsap_schema_*.py was generated from ONE example so it over-constrains. Make it @dataclass(kw_only=True) + data-driven required→optional (any field present in <100% of the corpus gets a default; [] for lists, None otherwise) → all certs parse.
3. Synthesise the measured fields the reduced schema only records categorically: windows (glazed_area band × floor area, 4-way N/E/S/W split), lighting LEL, hot-water bath/mixer counts, ventilation/chimneys/sheltered-sides, glazing cascade.
4. Leave calculator defaults to the calculator — cert_to_inputs is the RdSAP Table-5 expansion engine; the mapper supplies raw reduced data only.
5. Wire dispatch + flip a strict guard — add the schema_type branch to from_api_response, promote the corpus into the strict parse+map bucket in infrastructure/epc_client/tests/test_mapper_corpus.py.
6. Record every synthesis assumption in code comments + test names (Validation-Cohort rule: no same-spec ground truth).

Ground truth about the targets (verified 2026-06-10)

Spec	Schema module	Mapper method	Dispatched?	Corpus?	Notes
21.0.1	✅	from_rdsap_schema_21_0_1	✅	✅ 1000	reference (rich, measured)
21.0.0	✅	from_rdsap_schema_21_0_0	✅	❌	dispatched but unguarded
20.0.0	✅	from_rdsap_schema_20_0_0	✅	✅ 1000	DONE — the template
19.0	✅	from_rdsap_schema_19_0	❌	❌	orphaned; sap_windows=[] hardcoded
18.0	✅	from_rdsap_schema_18_0	❌	❌	orphaned
17.1	✅	from_rdsap_schema_17_1	❌	❌	orphaned
17.0	✅	from_rdsap_schema_17_0	❌	❌	orphaned

• 19.0 confirmed same reduced-field shape as 20.0.0: glazed_area: int band + multiple_glazing_type: int, and the mapper currently hardcodes sap_windows=[] — i.e. the exact windowless-corruption bug that 20.0.0's synthesis fixed. 18.0/17.1/17.0 are almost certainly the same family.
• The 17–19 mapper methods exist but are unreachable: from_api_response only branches 21.0.1 / 21.0.0 / 20.0.0; everything else hits raise ValueError(f"Unsupported EPC schema").
• Corpora are harvestable. downloads/README.txt schema-by-year: 2020 → RdSAP-Schema-19.0 (1632), 2021–2024 → 20.0.0, 2025–2026 → 21.0.1. Older RdSAP (17.x/18.0) live in the 2012–2019 dumps (all present locally). SAP-Schema-1x (full/design SAP) and CEPC-* (commercial) are different families with no RdSAP mapper.

---

Decision tree to grill (each has a recommended answer)

Q1 (ROOT) — Target set and order. What are we generalising to, and in what order?

Recommend: the pre-SAP10 RdSAP family only, one spec at a time, 19.0 first (dominant in the 2020 dump, closest sibling to 20.0.0, mapper already stubbed), then 18.0 → 17.1 → 17.0 as their dumps confirm volume. Exclude SAP-Schema-1x (full/design SAP — new-build, not reduced; a separate mapper family and ADR) and CEPC-* (non-domestic). Carve out 21.0.0 as a quick win: it's already dispatched, it just needs a harvested corpus to join the strict guard. Sub-question: do we batch all four 17–19 in one branch sweep, or land 19.0 fully (corpus → schema → synthesis → dispatch → guard) before starting 18.0? Recommend: land 19.0 end-to-end first — it either confirms the playbook transfers cleanly (then 18.0/17.x are fast) or surfaces drift early.

Q2 — Coefficient reuse vs re-fit (the load-bearing, ADR-worthy one).

20.0.0's glazing synthesis uses 0.148 × TFA × band_multiplier, fit from the 21.0.1 corpus's glazing/floor ratio. For 19.0/18.0/17.x: reuse the same coefficients, or re-fit per spec? DIRECTION (user, 2026-06-10): re-work the coefficients from each new corpus's own data — do not inherit the 21.0.1 fit by default. Treat 0.148 + the band multipliers as a starting hypothesis to confirm or replace against what the new corpus actually shows, per spec. The empirical numbers lead; we only keep the 20.0.0 values if the new corpus reproduces them.

The constraint this hits (must resolve while grilling): a reduced schema does not measure per-window area (that's the whole reason synthesis exists), so a 19.0/18.0/17.x corpus cannot self-fit the glazing/floor ratio — there's no measured glazing column in it to regress on. So "work it out from the new corpus" splits into two parts:

• What the reduced corpus can give us directly → re-derive per spec: the glazed_area band distribution (how many Normal/More/Less), total_floor_area distribution, and whether the band codes/semantics match 20.0.0. This validates (or breaks) the band-multiplier assumption empirically.
• The base ratio itself (the 0.148) → needs a measured reference from the same stock/era. Options to grill: (a) use the contemporaneous measured corpus if one exists for that year (e.g. a rich-window spec lodged alongside), (b) fit from the handful of rich certs the reduced corpus does carry (20.0.0 had 7/1000 with lodged sap_windows — check the count per spec), or (c) fall back to the 21.0.1 fit only if (a)/(b) yield too little signal, and say so explicitly.

This moves every rebaselined score for the spec, so the per-spec fit + its evidence wants an ADR (extends ADR-0027). Record the derivation (corpus, sample size, quartiles) the same way 20.0.0 did.

Q3 — Code-space drift across versions.

Do 17–19 use the same integer code spaces as 20.0.0 (glazing_type, built_form, orientation, fuel, heat-emitter, party-wall, roof/floor construction)? 20.0.0's codes turned out identical to 21.0.1's, so we routed through the existing cascades verbatim. Recommend: assume identical within the RdSAP family; cross-check each version against epc_codes.csv during grilling and add a per-spec cascade override only where the corpus proves a code diverged. Don't pre-build translation layers.

Q4 — Schema-fix mechanism. Same kw_only + data-driven required→optional?

Recommend: yes, unchanged. Each placeholder schema over-constrains identically (single-example generation). Run the one-pass corpus scan to enumerate all missing-required fields at once (not whack-a-mole), then default them. Mechanical, low-risk, proven.

Q5 — Shared synthesis helper vs per-mapper copy (the architecture fork).

20.0.0's synthesis lives in _synthesise_20_0_0_sap_windows + inline mapper blocks. With 19.0 we'll have a second instance — the classic extract trigger. Recommend: once 19.0 is green, extract a single spec-parameterised _synthesise_reduced_field_windows(glazed_area, tfa, glazing_type) (and shared lighting/hot-water/ventilation helpers) so 18.0/17.x are near-free and the coefficients live in exactly one place. Defer the extraction until 19.0 confirms the shape (avoid abstracting from one example). This is the /improve-codebase-architecture hook — a deep module behind a small interface.

Q6 — Per-spec field availability.

Do 17–19 actually lodge the synthesis inputs 20.0.0 relies on — instantaneous_wwhrs (bath room counts), low_energy_fixed_lighting_outlets_count, percent_draughtproofed, open_fireplaces_count, multiple_glazing_type? Older specs may omit or rename some. Recommend: profile each corpus up front (one-pass field-presence scan); where a 20.0.0 input is absent, degrade gracefully to the calculator's own default rather than fabricating — and record the gap in a test name.

Q7 — Dispatch wiring + acceptance bar.

Recommend: per spec, add the schema_type branch to from_api_response (wrapped in _clear_basement_flag_when_system_built like the others) and promote its corpus into the strict parse+map bucket in test_mapper_corpus.py. Smoke-check with scripts/eon/find_epc_data.py (extend the UPRN list with that spec's certs) — our re-score should track the lodged figure within a sane band. The formal SAP-score value test stays deferred (same as 20.0.0) until we choose to land it.

Q8 — Validation-Cohort / is there ANY cross-check?

Same rule as 20.0.0: a pre-SAP10 cert has no same-spec lodged figure to validate against, so synthesis assumptions go in code/test names. But probe one opportunistic anchor: a single UPRN re-lodged across spec versions (e.g. a dwelling with both a 19.0 and a 20.0.0 cert, unchanged between) — our re-score of both should roughly agree. Recommend: if dual-lodged UPRNs surface during harvest, keep a handful as a cross-spec regression anchor; don't block on it.

---

How to reproduce / kick off (19.0 first)

# 1. Confirm 19.0 volume + reduced-field shape in the 2020 dump
python - <<'EOF'
import json
from collections import Counter
# stream the first N lines of certificates-2020.json, bucket by schema_type,
# and dump one RdSAP-Schema-19.0 document to inspect glazed_area / sap_windows
EOF

# 2. Add a harvest source row and run it
#    scripts/eon/harvest_certs.py SOURCES += ("certificates-2020.json","RdSAP-Schema-19.0",1000)

# 3. Drive the (orphaned) 19.0 mapper against the new corpus to bucket parse failures
python - <<'EOF'
import json, collections
from pathlib import Path
from datatypes.epc.schema.rdsap_schema_19_0 import RdSapSchema19_0
from datatypes.epc.schema.helpers import from_dict
certs=[json.loads(l) for l in Path("backend/epc_api/json_samples/RdSAP-Schema-19.0/corpus.jsonl").read_text().splitlines() if l.strip()]
b=collections.Counter()
for c in certs:
    try: from_dict(RdSapSchema19_0, c)
    except Exception as e: b[f"{type(e).__name__}: {str(e)[:70]}"]+=1
for k,v in b.most_common(): print(v,k)
EOF

References

• ADR-0027 (docs/adr/0027-rdsap-20-0-0-reduced-field-synthesis.md) — the synthesis decision, coefficients, rejected alternatives. Extend (not replace) for the family-wide coefficient choice (Q2).
• ADR-0015 (mappers own cert normalization), ADR-0004 (lodged-vs-effective pair).
• CONTEXT.md — Reduced-Field Synthesis, Rebaselining, Lodged / Effective Performance, Validation Cohort, pre-SAP10.
• 20.0.0 resume doc — docs/grill-sessions/2026-06-09-rdsap-20-0-0-remapper.md (the worked example).