Commit graph

8582 commits

Author SHA1 Message Date
Daniel Roth
8e9c1c257d Pin the AWS provider to 6.56.0 across every Terraform stack
v6.57.0 (released 29 July 2026) breaks ordinary reads across unrelated AWS
services — SSM GetParameter returns SerializationException, IAM GetPolicy
returns a 302, ECR returns InvalidSignatureException — with no config
change. Reported upstream as hashicorp/terraform-provider-aws#49170, open
with no root cause identified; 6.56.0 is confirmed good.

Every stack declared `>= 5.0` with no upper bound and no lock files are
committed, so each CI run silently resolved whatever HashiCorp had shipped
most recently. That is how a provider released today broke a pipeline
nobody had touched. Pinning exactly makes the deployed version a reviewed
decision rather than a discovery.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 10:46:45 +00:00
Daniel Roth
72944dccab
Merge pull request #1706 from Hestia-Homes/feature/renamer-task-handler
SharePoint Renamer: put the run on the app-owned-task lane
2026-07-29 11:22:49 +01:00
Daniel Roth
292bd7e17d Restore the Abri ops guide to main
An unrelated in-progress edit that was already in the working tree when
this branch's work started, swept in by a bare `git add -A`. It belongs to
its author, not to this PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 10:09:39 +00:00
Daniel Roth
3e6464fb15 Bind a property's rename callback without a default-argument trick 🟪
partial binds plan eagerly and without introducing a parameter the caller
never passes, so there is no late-binding question to reason about.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 10:05:29 +00:00
Daniel Roth
35a3ebe0e3 Complete a property whose file SharePoint rejected 🟩
Green on arrival — rename() records rejections rather than raising.
Pinned after verifying it bites: letting the rejection escape fails the
child sub_task, which turns the whole Task red.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:53:58 +00:00
Daniel Roth
5b1b319c2d Give each property that needs renaming its own sub_task 🟩
Properties are planned and renamed one at a time, so only those with work
get a row and a run killed by the timeout still leaves every property it
reached both renamed and recorded.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:52:30 +00:00
Daniel Roth
cc07cc0114 Plan each property's renames before applying any of them 🟩
The walk now yields a PropertyPlan per address-list row, and rename()
applies one. run() composes the two, so single-pass callers are unchanged.
Knowing a property's size before the first rename is what makes "when is
this property finished?" answerable.

Tests that reached through the old private _process_folder are rewritten
against the public run(), so they survive this kind of change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:50:28 +00:00
Daniel Roth
737d07a951 Plan each property's renames before applying any of them 🟥
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:47:20 +00:00
Daniel Roth
bf1f5693be Build the handler test's address list without a shared mutable default 🟪
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:21:11 +00:00
Daniel Roth
bbb3c9e057 Verify a local rename run the way the queue delivers it 🟪
The local invoke now sends an SQS-shaped payload — a direct invoke is the
one shape that cannot reproduce the dry_run bug class. Test modules route
protected-method calls through typed helpers so both pass pyright strict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:19:51 +00:00
Daniel Roth
9870005230 Ship the Renamer image and infrastructure with database access 🟩
The task lane pulls the task domain types, both task repositories and four
Postgres modules into the image, plus SQLAlchemy, SQLModel and a driver.
Terraform gains the DB credentials block and five Postgres env vars; the
deploy job gains the three DB secrets the shared workflow already declares.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:11:41 +00:00
Daniel Roth
be1d6faea0 Complete a rename run that hit locked files or missing folders, and reject an unusable site 🟩
Green on arrival: both fall out of the summary the run now returns and of
the guarded site lookup. Pinned after verifying they bite — an unguarded
DomnaSites[name] completes an "ECO" run that renames files under a
different site's alias.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:03:43 +00:00
Daniel Roth
0836b18d5b Attribute a rename run to its request and target site 🟩
Green on arrival: @task_handler records inputs and reads source_id from
the body key matching the source's value. Pinned because it is the only
place the new sharepoint_site member is proven against a real Postgres
enum rather than a Python one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:01:51 +00:00
Daniel Roth
1213b80e3f Honour dry_run on a queue-delivered rename request 🟩
Green on arrival: reading the request from the SQS record body is what
taking the task lane gave us. Verified the guard bites by restoring the
raw-event read, under which the "dry" run renames a live file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:59:45 +00:00
Daniel Roth
694881c110 Record a rename run's result summary on its sub_task 🟩
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:58:47 +00:00
Daniel Roth
e7785f6b76 Record a rename run's result summary on its sub_task 🟥
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:58:18 +00:00
Daniel Roth
5309c1f097 Merge subfolder rename results into their parent folder's summary 🟩
Passed on arrival: threading the summary back up through the traversal
already carried subfolder results. Pinned so the merge cannot regress.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:51:36 +00:00
Daniel Roth
b738840c84 Count files skipped as already canonical apart from skipped images 🟩
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:51:02 +00:00
Daniel Roth
2aa698da1e Count files skipped as already canonical apart from skipped images 🟥
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:50:45 +00:00
Daniel Roth
ae7c9cdd46 Record each file SharePoint refused to rename, with its error 🟩
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:49:51 +00:00
Daniel Roth
a69d603cde Record each file SharePoint refused to rename, with its error 🟥
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:49:17 +00:00
Daniel Roth
9248aee139 List the UPRNs with no SharePoint folder under any root 🟩
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:45:46 +00:00
Daniel Roth
9e48f39fa5 List the UPRNs with no SharePoint folder under any root 🟥
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:45:20 +00:00
Daniel Roth
371df4d54f Count dry-run rename intents separately from real renames 🟩
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:44:37 +00:00
Daniel Roth
0135e71590 Count dry-run rename intents separately from real renames 🟥
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:44:20 +00:00
Daniel Roth
391e7bee37 Report how many files a rename run renamed 🟩
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:43:31 +00:00
Daniel Roth
5dc524be8f Report how many files a rename run renamed 🟥
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:42:33 +00:00
Jun-te Kim
82e1dd555a
Merge pull request #1700 from Hestia-Homes/investigatin/bugs_for_esh
Recognise "complete" as a MAIN building part, and anchor lone parts
2026-07-28 18:25:50 +01:00
KhalimCK
3afc43f484
Merge pull request #1691 from Hestia-Homes/fix/default-plan-refresh-on-rerun
Refresh a property's default Plan when its owning scenario is re-run
2026-07-28 18:13:02 +01:00
Jun-te Kim
2aba56c28e
Merge pull request #1696 from Hestia-Homes/fix/wall-construction-enum-mapping
Offer EWI/IWI on gov-API system-built walls (wall_construction code 8)
2026-07-28 18:04:46 +01:00
Jun-te Kim
bb5746c9b5 Recognise "complete" as a MAIN building part, and anchor lone parts 🟪
Six properties failed task 4d006fab (portfolio 854 / scenario 1334) with
DegeneratePredictionError. None has a lodged EPC, so each was predicted from a
neighbouring-cert cohort — and every cohort cert lodges its single building part
as `identifier: "complete"`. from_api_string accepted only "Main Dwelling" /
"Main building", so "complete" fell to OTHER, _has_main_part returned False, and
prediction refused to proceed.

"complete" is what SAP 9.91/9.92-era software writes for an unextended dwelling:
semantically identical to "Main Dwelling". Casing varies within a single postcode
(cert 8824-7422-1180-6934-1902 lodges "complete", 199 Highfield Road "Complete"),
so matching is now case-folded.

The API's `identifier` is free text with no schema enum — api.yml declares the
whole cert body `additionalProperties: true` and documents no field of it, and
`identifier` is absent from the 17 vocabularies at /api/codes. So the recognised
set can only grow by discovery, and each miss has cost an incident ("Main
building" → task a40e71c4; "complete" → task 4d006fab). Hence three changes, not
one:

- "complete" → MAIN, case-insensitively.
- Bare "Extension" → EXTENSION_1. The regex required a digit, so 668 corpus
  occurrences silently dropped a real extension from the structure. This reverses
  a prior deliberate pin whose stated rationale (RdSAP10 §1.2's 4-extension cap)
  does not apply — there is no out-of-range number in a bare "Extension".
- _anchor_lone_building_part: a cert lodging exactly ONE part describes the whole
  dwelling whatever it is called, so an unclassified lone part becomes MAIN.
  Multi-part certs are untouched — there the identifier carries real information
  and guessing would invent structure the cert does not state.

Also removes a false claim from DegeneratePredictionError's message: it asserted
the template was "lodged with a null part identifier", which was hardcoded, never
checked against a real cert, and wrong. It misdirected this investigation.

Verified against the live gov API: all 13 cohort certs across BL1 8EB, BL3 6XJ
and BL4 0RA now resolve a MAIN part. The full modelling e2e was NOT run (no local
AWS creds for the geospatial lookup) — this clears the blocking error, it does not
prove the predictions are good.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 15:12:01 +00:00
KhalimCK
4c0ff31454
Merge pull request #1693 from Hestia-Homes/debug/pashub-lrha-sap-gaps
PasHub LRHA WAVE 3: three spec-correct SAP fixes (58.3% → 62.1% within-0.5)
2026-07-28 12:14:58 +01:00
Jun-te Kim
aa4446757f Amend ADR-0019: gov-API code 8 is system built, not park home 🟪
Record why the generator now keys system-built on both code 6 and the
gov-EPC API code 8, citing the GOV.UK RdSAP WallConstructionCode XSD, and
flag the remaining mapper-level enum-normalisation work as open.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DX8oAGsGkBHq3U4dsxYRzz
2026-07-28 10:33:52 +00:00
Jun-te Kim
d1f064433c Offer EWI/IWI on a gov-API system-built wall (code 8) 🟩
Key the solid-wall Recommendation Generator's system-built branch on the
gov-EPC API code 8 (`_WALL_SYSTEM_BUILT_GOV_API`) alongside the internal
code 6, so a precast/no-fines-concrete system-built wall gets its EWI+IWI
Options. This mirrors `u_wall`, which already resolves code 8 as
system-built via `rdsap_uvalues._GOV_API_WALL_CODE_TO_TYPE[8] =
WALL_SYSTEM_BUILT`. Repoint the explicit park-home exclusion to the
gov-canonical park-home code 10. Supersedes ADR-0019's "do not key
system-built on 8", which read code 8 as the Elmhurst park-home label.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DX8oAGsGkBHq3U4dsxYRzz
2026-07-28 10:33:16 +00:00
Jun-te Kim
79bf28d8cb Offer EWI/IWI on a gov-API system-built wall (code 8) 🟥
The gov-EPC API lodges a system-built wall as `wall_construction == 8`
per the authoritative GOV.UK RdSAP `WallConstructionCode` enumeration
(communitiesuk/epb-data-warehouse `.../SAP-Domains.xsd`: 8 = "system
built" in every schema version 17.0-21.0.1; park home is 10). The solid-
wall Recommendation Generator instead read code 8 as the Elmhurst
park-home label and suppressed the whole cohort's EWI/IWI Recommendation
(a 2026 DB sweep finds ~5.1k certs at code 8, ~2.1k uninsulated main
walls). Rewrite the mislabelled park-home test to pin the correct
behaviour (code 8 → EWI+IWI), and add a genuine park-home test on the
gov code 10 so the fix doesn't start over-offering on real park homes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DX8oAGsGkBHq3U4dsxYRzz
2026-07-28 10:33:06 +00:00
Daniel Roth
7f0056c700
Merge pull request #1694 from Hestia-Homes/fix/pashub-evidence-pagination
Paginate PasHub evidence list instead of capping at one page
2026-07-28 11:21:19 +01:00
Daniel Roth
fdccad9c37 Fetch every evidence page when a job has more files than one page holds 🟪
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 10:12:25 +00:00
Daniel Roth
3c52a0aad6 Raise truncation error when PasHub stops serving pages before the full evidence set arrives 🟩
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 10:12:03 +00:00
Daniel Roth
f02441b6c4 Raise truncation error when PasHub stops serving pages before the full evidence set arrives 🟥
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 10:11:44 +00:00
Daniel Roth
4efb7eccbc Fetch every evidence page when a job has more files than one page holds 🟩
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 10:07:28 +00:00
Daniel Roth
5a85403c3d Fetch every evidence page when a job has more files than one page holds 🟥
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 10:06:00 +00:00
Khalim Conn-Kowlessar
7b50e5b7f5 docs(pashub): Elmhurst input sheets for the Varsity Rise cluster
The 12 Varsity Rise (Louth) homes are the only community-heated dwellings in
the LRHA WAVE 3 cohort and the only residual over-rate cluster (~+1 SAP). Two
candidate causes cannot be adjudicated from PasHub's `pre_sap` alone and need
accredited Elmhurst worksheets: (A) an electric-immersion DHW fuel-code
collision that bills immersion as community biomass (code 42), and (B) a
suspected community-DHW solar over-credit (the parser drops solar-collector
geometry, so the engine takes Table-29 defaults).

Add page-by-page Elmhurst RdSAP input sheets for one cert of each type
(461386632388 group A, 497671579889 group B) with a "what to read back" list
of the worksheet lines needed to diff term-by-term. Also correct the immersion
single → dual note on the 497655371974 build sheet (the cert lodges Dual).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-28 09:23:11 +00:00
Khalim Conn-Kowlessar
4d9dd9aaf3 fix(sap): (203)-weight the dual-main Table 4e MIT adjustment
For a dwelling with two main systems heating different parts, the (93)m
Table 4e mean-internal-temperature adjustment was taken from system 1 alone
(`details[0]`). SAP 10.2 p.186 requires the (203)-weighted mean of each
system's control adjustment — (1 - (203)) for system 1, (203) for system 2 —
mirroring the Table 9b responsiveness weighting already applied two lines
above. Keyed to system 1, a dwelling led by a small-fraction system (e.g. a
20% HHRSH, adj 0.0 °C, ahead of an 80% manual-charge storage heater, adj
+0.7 °C) was modelled ~0.7 °C too cool → space heat under-counted → over-rated.

Compute the weighted adjustment at the call site. Shared `cert_to_inputs`
path, so it also lifts the accredited gov-API RdSAP corpus (81.5% → 81.7%,
guardrail). 5 Edmund Close +2.74 → +0.59; 4 Edmund Close -2.51 → -0.55.
LRHA cohort within-0.5 60.2% → 62.1%, MAE 0.496 → 0.445; ratchets re-based to
0.62 / 0.45. Also corrects an oil-combi docstring (462051619031 is a Grant
Vortex 10599, not the Worcester 18415).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-28 09:22:59 +00:00
Khalim Conn-Kowlessar
138f5688e0 fix(pashub): wire the surveyed measured cylinder heat-loss factor
A PasHub survey can lodge a manufacturer's measured cylinder loss ("What is
the cylinder measured heat loss: 0.94 kWh/24 hours") — the SAP 10.2 §4
(48)-(50) declared-loss factor that must override the Table 2 V×L×VF
insulation computation. The site-notes path dropped it two ways: the
extractor looked for "Cylinder Measured Heat Loss:" (matches 0 fixtures) while
the PDF label is "What is the cylinder measured heat loss:" (92 fixtures), and
the mapper never wired `cylinder_heat_loss` even when captured. So the
calculator fell back to the age-band insulation default — a lossier cylinder
than surveyed — over-costing hot water and under-rating SAP.

Fix the extractor label and add `_pashub_cylinder_measured_heat_loss` (parses
the leading float, returns None for "Not known"), mirroring the gov-API and
Elmhurst paths that already pass `cylinder_heat_loss` through. 3 Woodmans
Court -1.33 → -0.09; LRHA cohort within-0.5 58.3% → 60.2%, MAE 0.515 → 0.496.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-28 09:22:05 +00:00
Khalim Conn-Kowlessar
451d0ceb73 fix(sap): bill mixed per-window glazing at each window's own U
The window channel collapsed a dwelling with genuinely mixed per-window
glazing (distinct `glazing_type` codes, no lodged per-window U) onto
windows[0]'s U, billing the entire opening area at one window's value.
SAP 10.2 worksheet (26)-(27) sums each window's own RdSAP Table 24 U over
its own area. The PasHub site-notes path is the only source that lodges
heterogeneous per-window codes with no per-window U (the gov-API/Elmhurst
paths carry a uniform code or a lodged U), so it was the only one mis-billed.

Add a `windows_have_mixed_codes` branch (factoring `_window_u_raw_from_code`
out of `_synthesised_window_u_raw`) that bills each window individually with
the per-window curtain transform. Scoped to lodge no average → uniform-glazing
dwellings stay byte-identical. Symmetric: corrects over- and under-raters
(e.g. 8 Loveden View -1.39 → window channel 32.31 → 26.67 W/K). LRHA cohort
within-0.5 58.3% → 58.3%, MAE 0.531 → 0.515.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-28 09:21:40 +00:00
Khalim Conn-Kowlessar
0ca8ca31a0 Refresh a property's default Plan when its owning scenario is re-run
A Plan became default only when its Scenario was flagged default or the
property had no Plan yet. Scenarios are created is_default=False, so in
practice only the first-ever Plan was ever promoted — re-modelling a
property appended a new Plan but left the default frozen on the first run,
so a re-model never surfaced on the FE (property 792168 kept its 2026-07-22
scenario-1328 default through three same-scenario re-runs).

Also promote when the re-run models the SAME scenario that currently owns
the property's default, so a re-run refreshes the shown Plan in place (the
save demotes the stale one). A re-run of a different scenario still does not
hijack the default. New PlanRepository.default_plan_scenario_by_property
surfaces each property's default scenario for the decision.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-27 16:53:24 +00:00
KhalimCK
035d2f5136
Merge pull request #1687 from Hestia-Homes/fix/epc-property-per-property-persistence
EPC persistence: write a per-property lodged copy, and refresh a stale stored cert on refetch
2026-07-27 16:44:55 +01:00
Jun-te Kim
dc2913b7db
Merge pull request #1688 from Hestia-Homes/fix/sap17-1-community-heating
Map full-SAP community heating (heat networks) instead of dropping it
2026-07-27 15:53:11 +01:00
Khalim Conn-Kowlessar
de08d93369 Refresh a stale stored gov cert on a refetch when its content differs
refetch_epc=True fetched a fresh cert but the recency tie-break (strict >)
dropped it whenever the stored row shared the cert's inspection date — so a
row written before a schema field/table existed (e.g. room-in-roof geometry)
could never be healed by re-fetching. The dwelling was modelled on stale
fabric indefinitely.

Reconcile instead of just comparing dates: on an equal inspection date,
refresh a public gov cert (uploaded_file_id NULL) when the fetched cert
differs from the stored one, so a plain re-fetch re-ingests it. One of our
own surveys (PasHub / ECMK / Elmhurst — uploaded_file_id set) is preserved on
the tie, keeping the survey-wins rule (ADR-0001 / #1589); an unchanged public
cert is still not re-persisted.

`_reconcile_lodged` layers the content refresh over `_newer_lodged` (recency
untouched). New `EpcRepository.survey_lodged_uprns` surfaces the row-level
uploaded_file_id provenance the hydrated EpcPropertyData drops. Together with
the property_id-anchored persist, a refetch_epc=True re-run now heals both a
missing epc_property row and a stale one.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-27 14:34:26 +00:00
Daniel Roth
7680d9346e
Merge pull request #1686 from Hestia-Homes/feature/abri-api-void-job
Abri API: Log empty properties in hubspot as Void
2026-07-27 15:21:07 +01:00