Commit graph

6202 commits

Author SHA1 Message Date
Daniel Roth
3d2a21d9d9 All downloaded PasHub files uploaded to SharePoint property folder 🟥 2026-06-04 15:40:25 +00:00
Daniel Roth
815173bc79 tweak local trigger 2026-06-04 15:40:25 +00:00
Daniel Roth
35b65d0d8d adjust comment about evidence type 2026-06-04 15:40:25 +00:00
Daniel Roth
5fa6f32644 Service persists mcs_certificate file_type when evidence_category is MCS cert 🟩 2026-06-04 15:40:25 +00:00
Daniel Roth
367f6f6cf9 Downloaded files carry evidence_category as DownloadedFile 🟩 2026-06-04 15:40:25 +00:00
Daniel Roth
1e2b997f2f Downloaded files carry evidence_category as DownloadedFile 🟥 2026-06-04 15:40:25 +00:00
Daniel Roth
e18c300deb MCS cert identified by evidence_category in get_core_file_type 🟩 2026-06-04 15:40:25 +00:00
Daniel Roth
1af1d4a82c MCS cert identified by evidence_category in get_core_file_type 🟥 2026-06-04 15:40:25 +00:00
Daniel Roth
5ed3bf73e8 evidence categories plus typehinting 2026-06-04 15:40:25 +00:00
Daniel Roth
020a24d345 run() returns core and other file paths 🟩 2026-06-04 15:40:25 +00:00
Daniel Roth
d8ec12065f run() returns core and other file paths 🟥 2026-06-04 15:40:25 +00:00
Daniel Roth
0aa6a4fc30 Other files persisted to DB with file_type OTHER 🟩 2026-06-04 15:40:25 +00:00
Daniel Roth
1650762ae2 Other files persisted to DB with file_type OTHER 🟥 2026-06-04 15:40:25 +00:00
Daniel Roth
c86dbeb4a1 Upload other files to S3 when get_other_files is True 🟩 2026-06-04 15:40:25 +00:00
Daniel Roth
098f60ecfd Upload other files to S3 when get_other_files is True 🟥 2026-06-04 15:40:25 +00:00
Daniel Roth
9c38f45c98 tidying for readability 2026-06-04 15:40:25 +00:00
Daniel Roth
c9a2ce4921 Service deletes other-file temp paths after run 🟩 2026-06-04 15:40:25 +00:00
Daniel Roth
f8d2bb8049 Service deletes other-file temp paths after run 🟥 2026-06-04 15:40:25 +00:00
Daniel Roth
49e7b7fea6 Wire service to get_evidence_files_by_job_id; retire get_core_evidence_files_by_job_id 🟪 2026-06-04 15:40:25 +00:00
Daniel Roth
662f6de0ab get_evidence_files_by_job_id downloads other files when include_other=True 🟩 2026-06-04 15:40:25 +00:00
Daniel Roth
c4ffaaa069 get_evidence_files_by_job_id downloads other files when include_other=True 🟥 2026-06-04 15:40:25 +00:00
Daniel Roth
f95b6bdd7d get_evidence_files_by_job_id returns DownloadedFiles with empty other when include_other=False 🟩 2026-06-04 15:40:25 +00:00
Daniel Roth
665dc69ad5 get_evidence_files_by_job_id returns DownloadedFiles with empty other when include_other=False 🟥 2026-06-04 15:40:25 +00:00
Daniel Roth
e7c679e0db Group evidence into core and other via _group_into_core_and_other_files 🟪 2026-06-04 15:40:25 +00:00
Daniel Roth
99229844b5 _select_other_files returns non-core evidence files 🟩 2026-06-04 15:40:25 +00:00
Daniel Roth
db796747d9 _select_other_files returns non-core evidence files 🟥 2026-06-04 15:40:25 +00:00
Daniel Roth
6cb6c8c756 allow for missing deal stage column when triggering sqs from file 2026-06-04 15:40:25 +00:00
Daniel Roth
790e430aff rename local handler trigger script 2026-06-04 15:40:25 +00:00
Khalim Conn-Kowlessar
69fdbf9f1d S0380.223: complete _part_geometry early-return key contract (RR KeyError)
5 certs in a 2026 API sample raised `KeyError: 'rr_common_wall_area_m2'`
and were blocked from computing. Root cause: `_part_geometry`'s early
return (taken when a building part lodges no sap_floor_dimensions —
e.g. a party-wall-only or RR-only extension as bp[0]) returned only 6 of
the 9 keys the full return exposes, omitting rr_common_wall_area_m2,
rr_gable_area_m2 and cantilever_floor_area_m2. The §3.9 RR contribution
block reads geom["rr_common_wall_area_m2"] / ["rr_gable_area_m2"] for
EVERY part, so the floorless part's truncated dict raised KeyError at
heat_transmission.py:974.

Fix: the early return now exposes all 9 keys, the three RR/cantilever
geometry values defaulting to 0.0 — correct, since a part with no floor
dimensions has no derivable RR shell or cantilever (no floor area).
Pure contract-completion bug; no spec/U-value change.

Regression test pins the invariant directly: a floorless part's
_part_geometry keys must equal a with-floors part's keys. Validated: all
5 certs now compute (4 within ~2 SAP of lodged; the 5th, 8536, has a
separate residual). §4 suite 2393 passed; heat_transmission.py pyright
unchanged at 12, test file at 71.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 15:33:52 +00:00
Jun-te Kim
a4670f8bc0 deploy lambda 2026-06-04 15:32:51 +00:00
Khalim Conn-Kowlessar
28634e8ae5 S0380.222: map API roof_construction codes 6 (thatched) + 7 (dwelling
above) → None

The 2026 sample lodges roof_construction=6 (1 cert, "Thatched, with
additional insulation") and =7 (6 certs, "(same dwelling above)" /
"(another dwelling above)"), both raising UnmappedApiCode and blocking
the cert. roof_construction_type is read ONLY for the §3 "sloping
ceiling" cos(30°) inclined-surface factor (Slice 89); the base roof
U-value comes from the global roofs[].description. Neither code is a
sloping ceiling:
  - 6 = thatched — U set by the description, not this field;
  - 7 = same/another dwelling above — an internal ceiling with no roof
    heat loss (the roof-side analogue of floor_construction code 0,
    governed by the roof_heat_loss / description path).

Map both to None: carries no information the cascade consumes here and
correctly avoids the cos(30°) false-trigger. Empirically inert and
validated — roof W/K is byte-identical whether 6/7 map to None or to an
explicit pitched string across all code-6/7 certs in the sample. 5 of
the 7 now compute (e.g. thatched cert 2276 SAP 62.8 vs lodged 63); the
other 2 also carry a gable_wall_type 2/3 raise (separate, worksheet-
backed slice).

Dict value type widened to Optional[str]. §4 suite 2392 passed; mapper.py
pyright unchanged at 32; new tests suppress reportPrivateUsage (net-zero).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 15:16:49 +00:00
Jun-te Kim
384e3da92a
Merge pull request #1172 from Hestia-Homes/feature/landlord_data
Feature/landlord data
2026-06-04 16:14:39 +01:00
Daniel Roth
6e986b76cc
Merge pull request #1173 from Hestia-Homes/bug/pashub-fetcher-correctly-writes-to-sharepoint
Increase pashub fetcher timeout to 5 mins
2026-06-04 16:14:35 +01:00
Daniel Roth
ab465bf010 increase pashub fetcher timeout to 5 mins 2026-06-04 15:12:29 +00:00
Khalim Conn-Kowlessar
aac3f0690a S0380.221: default a missing API post_town so the cert stays mappable
A 2026-register cert (4519-9056-4002-0222-4802) omits the top-level
post_town entirely (its town sits only in address_line_3 "BARNSTAPLE").
RdSapSchema21_0_x declares post_town as a required no-default field, so
from_dict raised "missing required field 'post_town'" and blocked the
whole cert from computing.

post_town is address metadata the SAP cascade never reads (no consumer
in domain/sap10_calculator/), so default an absent post_town to "" in a
from_api_response pre-processor (mirroring _normalize_shower_outlets) —
inert for the calculation, keeps the cert mappable. The schema dataclass
can't simply give post_town a default: it is a plain (non-kw_only)
dataclass with 57 required fields after post_town, so a mid-list default
would break field ordering.

Validated: cert 4519 now maps (post_town="") and computes SAP cont 74.68
vs lodged 75. §4 suite 2392 passed; mapper.py pyright unchanged at 32;
new tests suppress reportPrivateUsage (net-zero).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 14:59:37 +00:00
Jun-te Kim
261fae2e79 reformatted to be DDD structure 2026-06-04 14:50:04 +00:00
Khalim Conn-Kowlessar
d164850dd3 S0380.220: map API floor_construction code 0 → None (unknown/N-A)
The 2026 sample's second-largest mapper raise: 37 certs lodge
sap_floor_dimensions.floor_construction=0, which raised UnmappedApiCode
and blocked the cert. Code 0 is the "not recorded / not applicable"
sentinel — 33/37 pair it with floor_heat_loss=6 ("another dwelling
below", an upper-floor flat with no ground floor to describe); the rest
carry mixed Solid / unheated-space descriptions. There is no single
construction to assert.

Map code 0 → None, which defers to RdSAP 10 Table 19 ("where floor
construction is unknown" → age-band default) — identical to how an
unlodged floor_construction (the 993 None certs) is already handled, and
honest about the absence (cf. the no-misleading-insulation_type rule).

Empirically inert and validated: across all 37 code-0 certs the cascade
floor W/K is byte-identical whether code 0 maps to None or to an explicit
"Solid" string — the another-dwelling-below floors compute to 0.0 W/K
(handled via floor_heat_loss + property_type=Flat + floors[].description,
per the _API_FLOOR_HEAT_LOSS_TO_FLOOR_TYPE code-6 note), and the few
genuine ground/unheated floors hit the same age-band default either way.
All 37 now compute (were raising).

Dict value type widened to Optional[str] for the None entry; helper
already returns Optional[str]. §4 suite + schema-mapper tests green
(pre-existing test_total_floor_area failure unrelated); mapper.py pyright
unchanged at 32; new test suppresses reportPrivateUsage (net-zero).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 14:37:50 +00:00
Khalim Conn-Kowlessar
c89bec42cb S0380.219: map API floor_construction code 3 → "Suspended, not timber"
A random 1000-cert Jan–May 2026 EPB-register sample surfaced 53 certs
lodging sap_floor_dimensions.floor_construction=3, which raised
UnmappedApiCode and blocked the whole cert from computing (~44 of the
sample's mapper raises). RdSAP 10 field 3-1 "Floor construction"
enumerates the lowest-floor construction as solid / suspended timber /
suspended, not timber, and the spec's "Suspended not timber (structural
infiltration 0)" makes the split load-bearing.

Map code 3 to the canonical "Suspended, not timber" string (the same
value the site-notes mapper already emits — cross-mapper parity):
  - u_floor takes the suspended BS EN ISO 13370 branch via the
    "Suspended" prefix (_floor_is_suspended_from_description), and
  - _has_suspended_timber_floor_per_spec's exact-match
    `!= "Suspended timber"` gate correctly does NOT fire, so the §5 (12)
    0.1/0.2 floor-infiltration adjustment is skipped (structural
    infiltration 0) — exactly the spec rule for not-timber suspended.

Validated: all 5 sampled code-3 certs now compute (e.g.
0340-2877-5570-2606-5965 floor_construction_type="Suspended, not
timber", SAP cont 60.12 vs lodged 60). Confirmed against the cert's own
global floor descriptions ("Suspended, …", floor_heat_loss=7).

Code semantics established from the RdSAP 10 spec + the lodged certs'
human-readable floor descriptions (the EPB /api/codes endpoint carries
no floor_construction enum). §4 suite + schema-mapper tests green
(the pre-existing test_total_floor_area failure is unrelated). mapper.py
pyright unchanged at 32; new test suppresses reportPrivateUsage to keep
net-zero new errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 14:02:43 +00:00
Daniel Roth
d9124ee3b4
Merge pull request #1170 from Hestia-Homes/bug/pashub-fetcher-correctly-writes-to-sharepoint
Pashub fetcher bugs: Playwright /tmp cleanup, and sharepoint folder pagination
2026-06-04 13:59:55 +01:00
Daniel Roth
bdae83b715 SharePoint folder listing requests 500 items per page to cover alphabetically-late properties 🟩 2026-06-04 12:52:07 +00:00
Daniel Roth
428a5db513 Playwright tmp dirs are cleaned up after browser close 🟩 2026-06-04 12:51:14 +00:00
Daniel Roth
bd1fa09dc2 Playwright tmp dirs are cleaned up after browser close 🟥 2026-06-04 12:50:16 +00:00
Khalim Conn-Kowlessar
d3def1e254 docs: handover — S0380.218 closed the "with api 3" pair (both clean)
Record S0380.218 shipped, bump HEAD/next-slice, note both certs are
0-residual cross-validated golden fixtures and flag the optional
Summary-path regression guard as the cheap follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 12:36:23 +00:00
Khalim Conn-Kowlessar
6d9ef1143c S0380.218: golden fixtures for 2 fresh API+Summary+worksheet triples
Two certs fetched fresh from the GOV.UK EPB register, each with an
Elmhurst Summary PDF (input) and a dr87 worksheet PDF (the (1)..(286)
ground truth):

  0340-2467-9260-2006-6521  (Summary_000922 / dr87-0001-000922)
  5500-5070-0822-0201-3663  (Summary_000920 / dr87-0001-000920)

Both run through BOTH front-ends — from_api_response and
from_elmhurst_site_notes — and through the rating + demand cascades.
Cross-mapper parity holds: the two paths agree to <1e-4 on continuous
SAP, fuel cost, CO2 and PE. Both paths reproduce the worksheet exactly:

  0340: (255) cost 776.4295, (272) CO2 2875.0498, (286) PE 16474.5616;
        fabric (33) 171.6188, (37) 205.9358; SAP int 70 = lodged.
  5500: (255) cost 751.8295, (272) CO2 2423.4547, (286) PE 14397.0118;
        fabric (33) 141.1226, (37) 167.3696; SAP int 66 = lodged.

Pinned in two tables of test_golden_fixtures.py:
  - _EXPECTATIONS / test_golden_cert_residual_matches_pin — SAP/PE/CO2
    residual vs the integer-rounded lodged register (SAP resid +0 both).
  - _WORKSHEET_PE_CO2 / test_golden_cert_pe_co2_matches_worksheet —
    PE (286)/(4) and CO2 (272) vs the worksheet at +0.0000 (the
    load-bearing 1e-4 check; lodged register is integer-rounded).

Dropped-field audit (raw JSON keys vs the schema-21.0.1 dataclass
fields consumed by from_dict) re-run on both fresh JSONs: no new
silently-dropped fields — only created_at metadata and the
shower_outlet_type/shower_wwhrs keys already handled by
_normalize_shower_outlets (mapper.py:2047). No calculator or mapper
change required; this is pure validation + regression-pinning.

Full §4 suite: 2392 passed, 1 skipped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 12:35:50 +00:00
Khalim Conn-Kowlessar
1085842395 docs: handover — flagged numbers were stale (different branch), Part 1 is the task
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 12:11:12 +00:00
Khalim Conn-Kowlessar
8bd8ff8e5c docs: handover — fresh-API cross-comparison + flagged-cert debugging
Next-agent brief: fetch certs fresh from the EPC API (two new API+Summary+
worksheet triples for cross-mapper parity, plus six dashboard-flagged certs).
Flags the critical reconciliation: the user's flagged numbers don't match the
golden-fixtures cascade (0390-2954-3640 pinned +0 but flagged -6.85; 7536/2130
flags are pre-this-session), so fresh-raw-JSON-vs-curated-fixture or a
different engine must be reconciled before debugging. Documents the EPC API
fetch mechanism, the dropped-field audit method, this session's 4 fixes, and
the conventions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 12:08:28 +00:00
Khalim Conn-Kowlessar
f895dd3ab7 S0380.217: capture wall_insulation_thermal_conductivity (was dropped)
Second silently-dropped field from the 2130 audit: the schema-21
SapBuildingPart never declared `wall_insulation_thermal_conductivity`, so
`from_dict` discarded it. Captured it through schema 21.0.0/21.0.1 → domain
SapBuildingPart → API mapper, and wired it into u_wall's RdSAP 10 §5.8
documentary-evidence R-value calc (both the solid-brick §5.7/§5.8 path and
the cavity-composite path), replacing the bare 0.04 λ constant with a
resolved λ.

Resolver: absent / "Unknown" → the §5.8 default 0.04 W/m·K (mineral wool /
EPS); a mapped code → its λ; an unmapped integer code RAISES so the enum is
confirmed against a worksheet rather than silently mis-factored (same
incremental-coverage discipline as the glazing-type map). Only code 1
(= the default 0.04) is mapped — the sole observed value (cert 2130 Ext1).

Zero cascade effect today: the λ path fires only for solid-brick/cavity
walls with a *measured* wall thickness, and 2130 Ext1 lodges no wall
thickness, so its conductivity is captured-but-unused; all existing §5.8
certs lodge no conductivity → 0.04 default unchanged. The point is to stop
dropping lodged data and make λ correct when a future cert exercises it.

Suite: 2523 passed (1 pre-existing TFA fail); sap10_ml 237 passed (2
pre-existing stone-formula fails). Zero new pyright errors (46=46).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 11:57:00 +00:00
Jun-te Kim
dfd05ba28b tests files 2026-06-04 11:47:42 +00:00
Khalim Conn-Kowlessar
6b4f1aec44 docs: finalise 2130 as 0240-like — PV β-split proven exact
Updated 2130's pin notes with the case-18 finding: our cascade reproduces
the worksheet PV split to the decimal (gen 2684.17 / onsite 970.77 / export
1713.40), so the Appendix M1 β-split is exact, not the suspected bug. With
the gas PE factor also exact (1.13) and the wall measurement now wired
(S0380.215), 2130's +2/-11.72 is the irreducible API-only lodged residual
(0240-like), not a closable calculator bug. Notes-only; pin unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 11:04:47 +00:00
Khalim Conn-Kowlessar
712cc6f3f8 S0380.216: extractor — handle wrapped glazing-gap column in §11 labels
pdftotext dumps of hand-entered Elmhurst worksheets wrap the §11 glazing-
GAP column ("16 mm or more") onto the glazing-TYPE token, yielding labels
like "Double between 2002 and 2021 16 mm or [1st]" that
`_elmhurst_glazing_type_code` didn't recognise → UnmappedElmhurstLabel,
blocking the whole Summary from parsing.

Added a fallback: when the lightly-cleaned label isn't a known key, strip a
trailing wrapped gap descriptor (`\s+\d+\s*mm\b.*$`) and retry. Applied
AFTER the direct lookup so explicitly-mapped interleaved variants (e.g.
"Double with unknown 16 mm or install date more", where the gap splits into
the middle) are unaffected. The gap drives the API-path U-value lookup, not
the site-notes glazing-type enum, so dropping it is loss-free for the
cascade.

Unblocks running our cascade on hand-entered worksheet Summaries — used to
validate the PV β-split against simulated case 18 (our split matches the
P960 worksheet exactly: gen 2684.17, onsite 970.77, export 1713.40).

Suite: 2391 passed, 1 skipped. Zero new pyright errors (mapper 32=32).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 11:03:32 +00:00