Sync Planning Authority, Designated Area, Article 4 PD Rights, and
Listed Building from HubSpot into hubspot_deal_data so downstream
consumers of the existing HubSpot ETL process can read them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Built the property-753950 cert (worklist P4, PR #1503) in the live RdSAP-10
Online tool via Playwright: SAP 44 (E), vs our engine's 42 and the lodged
2014 cert's 45 — a normal RdSAP-2012->SAP-10.2 recalculation spread, not a
defect surfaced by the multiple_glazed_proportion fix.
Also surfaced (and documented as a caveat, not fixed here) a pre-existing
SAP-16.2 gap: the cert lodges party_wall_length_m=0 on both floors despite
being semi-detached, which Elmhurst's validation rejects and this PR's
engine change doesn't touch.
uploaded_files has no property_id — matching is on landlord_property_id — so the
property_ids path was pointless indirection (look property.id up in property just
to translate back to landlord_property_id). The FE holds landlord_property_id per
row anyway.
task.inputs hand-pick key is now landlord_property_ids: str[] (was property_ids:
int[]). resolve_selection unions two landlord_property_id sources — project_codes
expanded via hubspot_deal_data, and the hand-picked ids taken as given — and no
longer queries the property table at all. Trigger-only change; the domain plan,
orchestrator and matching already work in landlord_property_id.
ADR-0060, CONTEXT.md and the request schema updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Property 753950 (uprn 100021969385, cert 0141-2860-6891-9124-5625,
SAP-Schema-16.2) hard-failed modelling_e2e: RdSapSchema17_1 requires
multiple_glazed_proportion as a non-optional int, and this 16.x cert omits
the field entirely, lodging only the multiple_glazing_type="ND" sentinel.
The cert's own window.description ("Fully double glazed") states the
glazing extent unambiguously, so _normalize_sap_schema_16_x now derives
0/100 from "single"/"double" wording when the field is absent, mirroring
the existing single-glazed multiple_glazing_type cascade. This is
worklist P4 (.claude/skills/expand-sap-accuracy-corpus/worklist.md) —
a flat default was tried previously and reverted because making such
certs mappable at all pulls them into the EPC-prediction donor pool and
tips near-tie similarity matches; deriving from explicit text (rather
than a blind default) was the suggested unblock.
Re-measured the component-accuracy gate as the worklist asked: it drops
(has_hot_water_cylinder 0.8687->0.8586, cylinder_insulation_type
0.3333->0.1667, door_count residual 0.3131->0.3333) via the same
donor-pool-composition mechanism as the prior #1245/ADR-0037
re-baselines, not a prediction-logic loosening. Re-baselined the floors
with that rationale recorded inline.
The 'all properties' selection resolved against property.portfolio_id, but a
portfolio spans multiple HubSpot projects — so 'all' pulled the whole portfolio,
not the project the user was looking at. The project↔property grain lives on
hubspot_deal_data, not property.
task.inputs is now {project_codes?: str[], property_ids?: int[], portfolio_id?}:
the route resolves the distinct landlord_property_id set as the union of every
property in the named project_codes (from hubspot_deal_data) and the hand-picked
property_ids; portfolio_id is optional and only names the package. Drops the
portfolio-scoped select_all. Cap now applies to the resolved set size.
ADR-0060, CONTEXT.md and the request schema updated to match. New router tests
cover project-code resolution, the union+dedup, null landlord_property_id drop,
and the empty-project rejection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses PR #1498 review:
- A per-file read failure (missing/deleted object, or a bucket the role can't
reach) is now a SkippedDocument(reason=unreadable), not a whole-run abort;
an all-unreadable selection still fails (no empty package). Best-effort on
the read path (ADR-0060).
- Each ZIP member is streamed to a temp file (S3DocumentDownloader.download,
boto download_file) and added from disk, so a single multi-GB member never
hits the heap — the 'never held whole in memory' claim is now true.
- The 409 double-submit guard is DB-arbitrated: the sub_task insert is
conditional on the task having none, so a race creates only one.
- Local env gets a placeholder recipient email so the route is exercisable.
- PackageEntry carries landlord_property_id so a read failure can be reported.
- TODO noting the UploadedFile.s3_upload_timestamp typing fix.
Two new tests: per-file read failure skips-and-reports; address fallback flows
to the folder name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per PR review: the FE writes the selection config
({portfolio_id, property_ids?, select_all?}) into the FE-owned task.inputs and
passes only task_id, so a large hand-picked selection never travels in an HTTP
body. The route reads task.inputs, resolves to landlord_property_ids, caps, and
pins the recipe onto sub_task.inputs as before. Declares the FE-owned inputs
column on the TaskRow mirror so the backend can read it and the test schema
builds it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolves the authenticated requester's email (ADR-0059), resolves + caps the
property selection, pins the recipe (landlord_property_ids, recipient_email,
package_name) onto one pre-created sub_task (raw SQL, dodging the mirror
double-registration), and enqueues one message to the worker. Refuses
double-submits (409) and oversized selections (400).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Address PR review (dancafc):
- introduce UprnMatch NamedTuple (datatypes/address_match.py) for the
(uprn, address, lexiscore, certificate_number) return, replacing the bare
4-tuple in get_uprn_from_epc_df / get_uprn_from_historic_epc /
HistoricEpcResolver.resolve_uprn. Tuple-compatible, so unpacking is unchanged.
- rename get_uprn_with_epc_df -> get_uprn_from_epc_df (+ callers).
- type resolve_group_ambiguity via a GroupDecision NamedTuple and trim its
docstring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 1 of confirming UPRNs before finalise. address2uprn matched each
row independently, so one UPRN could be the best match for two distinct
addresses (a coarse EPC record absorbing several real addresses, e.g.
flats in a block). Those distinct addresses were then silently merged by
the property identity insert, and collided in property_overrides.
resolve_group_ambiguity() withholds a UPRN claimed by >=2 distinct
normalised addresses within a postcode group (keeps genuine same-address
re-listings), and the handler now emits an address2uprn_status column
(matched | ambiguous_duplicate | unmatched | invalid_postcode | error).
Withheld rows drop to a null UPRN but keep their lexiscore for triage on
the (upcoming) confirmation page.
Also adds the ADR-0057 backstop dedup in property_overrides upsert_all so
the ON CONFLICT statement can never double-touch a row.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EpcClientService.search_by_postcode already returns the matched
certificate number alongside the UPRN, but it was dropped before
persistence. Thread it through get_epc_data_with_postcode ->
get_uprn_with_epc_df / get_uprn_from_historic_epc (using the historic
dataset's lmk_key) -> the address2uprn_certificate_number result
column -> PropertyIdentityInsert -> the property table's new
certificate_number column (assessment-model PR #362).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Review feedback (#1481): the address batcher and the Modelling Run batcher
implemented the same greedy packing; the core moves to
utilities/grouped_batching.py and both become thin wrappers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ADR-0056 amended: override and EPC own type/form facts; a property with
neither is Unknown whatever the legacy columns say. Mirror columns removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DB access via parameterised SQL, not the infrastructure SQLModel mirrors:
importing those alongside the legacy backend.app.db.models mirrors of the
same tables double-registers them in the shared metadata and crashes the
app at import. Contained to the modelling package until the DDD cut-over.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/health already returned GITHUB_SHA in its response but never logged
it, and the fastapi lambda's Terraform environment never actually set
GITHUB_SHA — so every health check response contained "unknown" in
production. Wire var.github_sha through the fastapi lambda module
(default "unknown" for local/other invocations) via a new
TF_VAR_github_sha env var set from `github.sha` in
_deploy_lambda.yml's Terraform Plan step, and log it on every /health
call so a request in CloudWatch can be tied back to the deploy that
served it.
Also fix the zip-size gate added for PR #1469: putting it in
tests/test_lambda_zip_size.py (run via the Docker-based ddd_tests.yml
suite) broke CI, because Dockerfile.test's build context excludes
deployment/* (.dockerignore), so check_lambda_zip_size.py couldn't
find variables.tf to read zip_excludes from inside that container.
Move the check to its own lightweight workflow,
check_lambda_zip_size.yml, triggered on pull_request into main — a
plain checkout (no Docker build) has the full repo, so the check
works, runs fast, and still gates merges to main before a regression
can roll into the dev deploy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Relocate scripts/check_lambda_zip_size.py to
backend/app/requirements/, right beside the requirements.txt it's
built to measure, since it's specific to the fastapi lambda rather
than a generic repo script.
While moving it, also make the excludes list self-updating: the
script now reads zip_excludes straight out of
deployment/terraform/modules/lambda_with_api_gateway/variables.tf
by default instead of duplicating it as CLI flags, so the CI check
and the actual Terraform packaging can't drift apart. requirements.txt
defaults to the sibling file too, so both the CI step and local runs
now collapse to a bare `check_lambda_zip_size.py` invocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
