Model/scripts/hyde/full_sap_17_1_remapper_handoff.md

Handoff: build a full-SAP (SAP-Schema-17.1) → EpcPropertyData mapper

You are picking up a focused piece of work in /workspaces/model (a Python repo, branch feature/hyde_make_it_more_accurate_with_tests). Read this whole prompt before starting.

Goal

Add support for full-SAP certificates (schema SAP-Schema-17.1) to the EPC mapper so a lodged full-SAP cert can flow through the existing chain:

EpcClientService.get_by_certificate_number(cert)
  → EpcPropertyDataMapper.from_api_response(raw)   # <-- the work is here
    → EpcPropertyData
      → Sap10Calculator().calculate(epc) → SapResult

Today from_api_response raises ValueError: Unsupported EPC schema: 'SAP-Schema-17.1'. There is a captured real cert and a strict-xfail test waiting to flip to green (see Acceptance).

Essential domain context (do not skip)

The GOV.UK EPB register holds two different certificate families:

• RdSAP (RdSAP-Schema-*) — Reduced data SAP, for existing dwellings. The assessor records a reduced input set; the schema carries proxy/count fields (door_count, window, percent_draughtproofed, low_energy_fixed_lighting_outlets_count, …) and the mapper applies RdSAP table defaults downstream.
• full SAP (SAP-Schema-*) — the full SAP calculation input set, typically for new-build / on-construction assessments (assessment_type: "SAP"). It carries richer, measured geometry instead of proxies: explicit windows, sap_opening_types, air_tightness, living_area, orientation, sap_ventilation, sap_flat_details.

All 7 currently-supported schemas are RdSAP. No full-SAP path exists anywhere yet. This is net-new, not a tweak.

EpcPropertyData (the target domain aggregate) and Sap10Calculator already consume RdSAP-derived data fine — your job is the mapping, not the calculator. Where full SAP gives a measured value that RdSAP only approximated, prefer the measured value; where full SAP omits something RdSAP supplied, you'll need a sensible default (mirror the RdSAP mapper's defaulting and document each with an # ADR-...-style comment as the existing methods do).

Current state — exact files & line refs

• Dispatch: datatypes/epc/domain/mapper.py, from_api_response (the if schema == "...": ladder, ~lines 2200–2261, ending in the raise ValueError(f"Unsupported EPC schema: {schema!r}")). Each branch does from_dict(SchemaClass, data) (dacite) then a from_<schema> static method.
• RdSAP mapper methods: same file — from_rdsap_schema_17_1 (~line 623) is the closest analogue; from_rdsap_schema_18_0 (~814) etc. Read 17.1 end-to-end first — it's your template. Note AnyRdSapSchema union (~lines 98–105) and shared helpers _map_energy_elements, _map_energy_element, _api_sheltered_sides, _clear_basement_flag_when_system_built.
• Schema models: datatypes/epc/schema/ — one dataclass module per schema (rdsap_schema_17_1.py, …). You'll add sap_schema_17_1.py here, modelling the full-SAP payload as frozen-ish dataclasses (from dataclasses import dataclass), reusing common.py types where shapes match. from_dict (dacite) builds it from the raw JSON, so the dataclass must match the JSON field names exactly (with Optional[...] for anything not always present).
• Captured sample (your fixture + shape reference): backend/epc_api/json_samples/real_life_examples/SAP-Schema-17.1/uprn_10092973954/epc.json (cert 0862-3892-7875-2690-2325, lodged energy_rating_current = 83).
• The waiting test: tests/domain/sap10_calculator/test_real_cert_sap_accuracy.py — the uprn_10092973954 case is a strict xfail(raises=ValueError) flagged via unsupported_schema=True on its RealCertExpectation.

The structural delta (full-SAP 17.1 vs RdSAP 18.0 top-level keys)

Full-SAP has, RdSAP doesn't: windows (dict), sap_opening_types (list), air_tightness (dict), sap_ventilation (dict), sap_flat_details (dict), living_area, orientation, multiple_glazed_percentage, design_water_use, data_type, sap_data_version, is_in_smoke_control_area, seller_commission_report, assessment_date, address_line_2/3.

RdSAP has, full-SAP doesn't: window (singular proxy), door_count, extensions_count, habitable_room_count, heated_room_count, fixed_lighting_outlets_count, low_energy_fixed_lighting_outlets_count, open_fireplaces_count, percent_draughtproofed, solar_water_heating, mechanical_ventilation, glazed_area, glazing_gap, multiple_glazed_proportion, pvc_window_frames, measurement_type, insulated_door_count, suggested_improvements.

schema_version_original is LIG-17.0; assessment_type is SAP.

Map the rich full-SAP fields onto the EpcPropertyData / Sap* fields the RdSAP path populates. Inspect the nested shapes (windows, sap_opening_types, air_tightness, sap_ventilation) directly in the JSON sample and against what Sap10Calculator/cert_to_inputs (domain/sap10_calculator/rdsap/cert_to_inputs.py) actually reads.

Task

1. Inspect the sample JSON thoroughly; enumerate every full-SAP field and nested object.
2. Add datatypes/epc/schema/sap_schema_17_1.py — dataclasses matching the full-SAP payload (dacite-compatible, Optional where appropriate).
3. Add EpcPropertyDataMapper.from_sap_schema_17_1(schema) -> EpcPropertyData in mapper.py, modelled on from_rdsap_schema_17_1. Comment every place you default or reinterpret a field vs the RdSAP path.
4. Wire a if schema == "SAP-Schema-17.1": branch into from_api_response (decide whether _clear_basement_flag_when_system_built applies — check what it does).
5. Flip the test: in test_real_cert_sap_accuracy.py remove unsupported_schema=True from the uprn_10092973954 case and set its sap_score to whatever the calculator now produces — BUT first sanity-check that figure against the lodged rating (83) and, ideally, a domain expert. If the calc score diverges materially from lodged, that's a finding to report, not silently pin. Do NOT widen tolerances to force a pass.

Constraints

• Type safety: all new code must pass pyright under typeCheckingMode = strict, zero errors. Use Optional[X] not X | None. Annotate all return types. No bare dict — use dict[str, Any] for raw payloads.
• TDD: this repo uses vertical-slice TDD (/tdd skill). Prefer one failing test → one impl increment → repeat. The end-to-end accuracy test is the outer loop; add focused mapper unit tests under datatypes/epc/domain/tests/ as you go (see test_from_rdsap_schema.py for the pattern).
• Domain language / ADRs: load-bearing mapping decisions (e.g. how full-SAP windows collapse onto the calculator's window model, how air_tightness feeds infiltration) should be recorded as ADRs in docs/adr/ — the existing mapper methods reference ADR-0028 etc. Use the /grill-with-docs skill if terminology is drifting.
• Run the suite with: python -m pytest tests/domain/sap10_calculator/test_real_cert_sap_accuracy.py -q --no-cov -rxs

Acceptance criteria

• from_api_response maps a SAP-Schema-17.1 payload to a valid EpcPropertyData with no ValueError.
• Sap10Calculator().calculate(epc) runs end-to-end on cert 0862-3892-7875-2690-2325 and produces a SAP score.
• The uprn_10092973954 accuracy case is no longer xfail — it's a real pin, with its score reconciled against the lodged 83 (divergence explained if any).
• pyright --strict clean; new mapper unit tests cover the full-SAP field mappings; no tolerance widening anywhere.

Capturing more full-SAP samples

scripts/fetch_real_life_epc_sample.py <uprn> resolves a UPRN to its latest cert, writes it under backend/epc_api/json_samples/real_life_examples/<schema_type>/uprn_<uprn>/epc.json, and prints schema + lodged rating + current calc output (or NOT MAPPABLE). Use it to gather a small full-SAP cohort to harden the mapper against shape variation before pinning scores.