partial binds plan eagerly and without introducing a parameter the caller
never passes, so there is no late-binding question to reason about.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Properties are planned and renamed one at a time, so only those with work
get a row and a run killed by the timeout still leaves every property it
reached both renamed and recorded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The local invoke now sends an SQS-shaped payload — a direct invoke is the
one shape that cannot reproduce the dry_run bug class. Test modules route
protected-method calls through typed helpers so both pass pyright strict.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The task lane pulls the task domain types, both task repositories and four
Postgres modules into the image, plus SQLAlchemy, SQLModel and a driver.
Terraform gains the DB credentials block and five Postgres env vars; the
deploy job gains the three DB secrets the shared workflow already declares.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six properties failed task 4d006fab (portfolio 854 / scenario 1334) with
DegeneratePredictionError. None has a lodged EPC, so each was predicted from a
neighbouring-cert cohort — and every cohort cert lodges its single building part
as `identifier: "complete"`. from_api_string accepted only "Main Dwelling" /
"Main building", so "complete" fell to OTHER, _has_main_part returned False, and
prediction refused to proceed.
"complete" is what SAP 9.91/9.92-era software writes for an unextended dwelling:
semantically identical to "Main Dwelling". Casing varies within a single postcode
(cert 8824-7422-1180-6934-1902 lodges "complete", 199 Highfield Road "Complete"),
so matching is now case-folded.
The API's `identifier` is free text with no schema enum — api.yml declares the
whole cert body `additionalProperties: true` and documents no field of it, and
`identifier` is absent from the 17 vocabularies at /api/codes. So the recognised
set can only grow by discovery, and each miss has cost an incident ("Main
building" → task a40e71c4; "complete" → task 4d006fab). Hence three changes, not
one:
- "complete" → MAIN, case-insensitively.
- Bare "Extension" → EXTENSION_1. The regex required a digit, so 668 corpus
occurrences silently dropped a real extension from the structure. This reverses
a prior deliberate pin whose stated rationale (RdSAP10 §1.2's 4-extension cap)
does not apply — there is no out-of-range number in a bare "Extension".
- _anchor_lone_building_part: a cert lodging exactly ONE part describes the whole
dwelling whatever it is called, so an unclassified lone part becomes MAIN.
Multi-part certs are untouched — there the identifier carries real information
and guessing would invent structure the cert does not state.
Also removes a false claim from DegeneratePredictionError's message: it asserted
the template was "lodged with a null part identifier", which was hardcoded, never
checked against a real cert, and wrong. It misdirected this investigation.
Verified against the live gov API: all 13 cohort certs across BL1 8EB, BL3 6XJ
and BL4 0RA now resolve a MAIN part. The full modelling e2e was NOT run (no local
AWS creds for the geospatial lookup) — this clears the blocking error, it does not
prove the predictions are good.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Plan became default only when its Scenario was flagged default or the
property had no Plan yet. Scenarios are created is_default=False, so in
practice only the first-ever Plan was ever promoted — re-modelling a
property appended a new Plan but left the default frozen on the first run,
so a re-model never surfaced on the FE (property 792168 kept its 2026-07-22
scenario-1328 default through three same-scenario re-runs).
Also promote when the re-run models the SAME scenario that currently owns
the property's default, so a re-run refreshes the shown Plan in place (the
save demotes the stale one). A re-run of a different scenario still does not
hijack the default. New PlanRepository.default_plan_scenario_by_property
surfaces each property's default scenario for the decision.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
refetch_epc=True fetched a fresh cert but the recency tie-break (strict >)
dropped it whenever the stored row shared the cert's inspection date — so a
row written before a schema field/table existed (e.g. room-in-roof geometry)
could never be healed by re-fetching. The dwelling was modelled on stale
fabric indefinitely.
Reconcile instead of just comparing dates: on an equal inspection date,
refresh a public gov cert (uploaded_file_id NULL) when the fetched cert
differs from the stored one, so a plain re-fetch re-ingests it. One of our
own surveys (PasHub / ECMK / Elmhurst — uploaded_file_id set) is preserved on
the tie, keeping the survey-wins rule (ADR-0001 / #1589); an unchanged public
cert is still not re-persisted.
`_reconcile_lodged` layers the content refresh over `_newer_lodged` (recency
untouched). New `EpcRepository.survey_lodged_uprns` surfaces the row-level
uploaded_file_id provenance the hydrated EpcPropertyData drops. Together with
the property_id-anchored persist, a refetch_epc=True re-run now heals both a
missing epc_property row and a stale one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The lodged EPC read keys on UPRN (ADR-0001 recency tie-break), but writes
and the front end both key on property_id. When a dwelling is onboarded
under two portfolios, the shared UPRN surfaces the sibling's lodged row,
so `lodged_epc_is_new` stays False and the modelling run skips persisting
a copy for this property — leaving it with no epc_property row the FE can
read. The EPC is modelled but invisible (portfolio 850 / property 792235,
uprn 100021979925, also under property 753896 in portfolio 830).
Drive the lodged persist off "does THIS property own a lodged row?"
(property_id-scoped) rather than "is the cert new?" (UPRN-scoped): add
`EpcRepository.property_ids_with_lodged_epc` and persist a per-property
copy whenever the cert is new OR the property has no lodged row of its
own. Idempotent (save_batch is replace-by-property_id), and the recency
skip still holds for a property that already owns its row.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Compose the party-ceiling and sloping-ceiling guards in the roof
GuardedColumnClassifier so a PitchedWithSlopingCeiling override resolves to the
Pitched, sloping ceiling family instead of being routed onto Pitched, N mm loft
insulation by the LLM (ADR-0066).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The service was the last one living wholly under backend/. It now follows
the same layering as abri and the other newer services:
domain/pashub_fetcher/ core file classification, subfolders
infrastructure/pashub_fetcher/ PasHub client, token getter, wire DTOs
orchestration/ PashubFetcherOrchestrator (was PashubService)
applications/pashub_fetcher/ lambda handler, trigger request, dev tooling
core_files.py is split along the layer boundary: the domain module keeps the
filename/evidence-category classification rules and no longer imports
infrastructure.postgres, while the CoreFiles -> FileTypeEnum translation moves
to infrastructure/pashub_fetcher/core_file_types.py.
Tests move into the tests/ tree by layer. Note this puts them in the only
suite CI currently runs (unit_tests.yml is disabled), so these 73 tests now
execute on PRs for the first time; they were previously reachable only via
the legacy pytest.ini testpaths.
sharepoint_renamer's image now copies just domain/pashub_fetcher/ rather than
the whole service, since SharepointSubfolders is all it needed.
Behaviour is unchanged. tests/ goes 9927 -> 10000 passed (+73, exactly the
tests that moved in); the legacy suite keeps its same 17 pre-existing failures
and 11 errors. tests/test_lambda_packaging.py confirms both changed
Dockerfiles still copy their handler's full import closure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A second export selection alongside the per-Scenario one, chosen by
`plan_selection` on the export request:
- "scenario" (default, unchanged): one sheet per Scenario, freshest Plan per
(property, scenario); scenario_ids required.
- "default" (new): a single "Default Plans" sheet of each home's is_default
Plan (one-per-property across all Scenarios, ADR-0012/0017) — the portfolio's
current state, one row per home; scenario_ids ignored.
Repository gains `default_rows_for(portfolio_id, property_ids)`: same read-model
and Effective-EPC join as `rows_for`, with the plan-selection WHERE swapped from
`scenario_id = :scenario_id` to `is_default = TRUE`. Orchestrator branches on
plan_selection; the router validates it and only requires scenario_ids for the
scenario selection. A recipe without plan_selection defaults to "scenario", so
the change is backward-compatible.
Both selections now surface the chosen Plan's `name` as a `plan_name` column.
Tests: repository default_rows_for (scenario-independent selection, plan_name,
non-default excluded); orchestrator default path (one "Default Plans" sheet;
empty selection is a recorded failure). 62 passed, pyright --strict clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dan's review on #1546:
- Stream the workbook to a /tmp temp file and S3Client.upload_file it, instead
of render_workbook -> bytes -> put_object. render_workbook now saves straight
to a path; the orchestrator renders to a temp file, multipart-uploads it, and
always cleans it up. Restores ADR-0065's "never an in-memory BytesIO" decision
(the OOM path at the 100k-row cap).
- Move the raw `SELECT ... FROM scenario` out of the Lambda handler into
ScenarioNamesPostgresRepository, so the handler stays composition-only and all
SQL lives in repositories/.
- current_sap_points goes through _float, matching its Optional[float] read-model
type and the sibling numeric facts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deal's third_party_surveyor_identifier is now required by the log
and amend flows: the trigger contract rejects messages without it
(blank treated as missing, since a blank resource makes OpenHousing
silently drop the appointment), and LogJobRequest/AmendJobRequest carry
it as a required field. The ABRI_RELAY_DEFAULT_RESOURCE fallback and
its config/terraform/workflow wiring are removed as dead code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Carry third_party_surveyor_identifier on the Abri trigger message and
through dispatch into the log and amend flows, booking the job against
that resource. LogJobRequest gains the optional resource field
AmendJobRequest already had; the client falls back to the configured
default surveyor when the deal names no third-party resource, so
existing deals behave unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Read stored lodged EPCs whatever refetch_epc says, so a stored assessment
can win the ADR-0001 Recency Tie-Break against the gov register.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_lambda_image_copies_full_import_closure caught this: importing the Historic
EPC S3 repo drags datatypes/epc/domain/historic_epc_matching.py into the handler's
init-time closure, and that reaches back into the legacy address matcher —
backend/address2UPRN/scoring.py and utils/pandas_utils.py. The image COPYed
neither, so the Lambda would have died at cold start with Runtime.ImportModuleError.
Copied file-by-file rather than `COPY backend/ backend/`: backend/ is the whole
legacy engine and the closure needs only these seven files. Their third-party
deps (pandas, requests) are already in requirements.txt, so no new pip installs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-0054 gives a prediction conditioned by an expired Historic EPC the source
"expired", so reporting can tell "no EPC at all" apart from "only an expired
one". It was implemented in IngestionOrchestrator — but the pipeline that
actually runs is modelling_e2e, which predicts through its own _predict_epc and
never adopted it. Result: zero `expired` rows have ever been written, and
`epc_property.source` holds only 'lodged' and 'predicted'.
Two things kept the flavour unreachable here, and both had to go:
- _predict_epc never looked at the historic backup at all, so no prediction
was ever conditioned.
- _flush_writes passed the literal source="predicted", and _PropertyWrite had
no field to carry a flavour, so one would have been dropped before the write
even if computed.
_predict_epc now mirrors IngestionOrchestrator._predict: the expired cert's
stable attributes fill the gaps Landlord Overrides left (overrides still win
where both speak) and condition the cohort, and it returns the source alongside
the EPC. _PropertyWrite carries it; _flush_writes persists it. save_batch
already groups deletes by source family, so a mixed predicted/expired batch
clears the shared slot correctly.
Ships dark: the reader is built only when HISTORIC_EPC_S3_ROOT is set, so
without it every prediction stays plain "predicted", exactly as today.
DEFAULT_S3_ROOT names the dev bucket, so defaulting to it would have a prod
lambda silently reading dev data — Terraform sets the var per environment.
Note this also rescues properties that currently fail to model outright: an
expired cert can supply the property_type no override resolved, where today
that raises UnresolvedPropertyTypeError.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An epc_property row can carry a null property_id (FE/historic ingestion
persists EPC rows never linked to a property row), so a property_id-keyed
read silently misses them. UPRN is the durable key both the property row
and the epc_property row share. Repoint the lodged reads (get_for_property
+ get_for_properties) onto UPRN; the predicted reads stay on property_id
because a predicted EPC deep-copies a neighbour's UPRN, so its UPRN column
is never the property's own.
UPRN is not unique on epc_property, so the read tie-break is now load-
bearing (it was dormant under property_id, where the write path guarantees
one lodged row per property): the most recently ingested (highest-id) row
wins. Write path stays keyed on (property_id, source).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires POST /v1/exports/scenario -> tasks.inputs recipe -> pinned sub_task ->
ARA_EXPORT_SQS_URL -> ara_export Lambda -> orchestrator. Route resolution and the
trigger body are covered by tests; the Lambda handler mirrors bulk_document_download.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The lift-and-shift moved the handler to applications/condition/handler.py and
split the module across domain/infrastructure/repositories/applications, but the
Dockerfile still COPYed backend/condition/ and ran the old CMD path. Update the
COPYs to the handler's real import closure (the DDD dirs + the legacy backend
Base/db_session/settings bridges) and the CMD to applications/condition/handler.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Calico Asset References don't match our landlord_property_id, so the runner now
resolves UPRNs from the address-matched CSV (UprnLookupS3) uploaded to S3, not
the property-table PropertyUprnLookup. run_calico_load accepts any UprnLookup
(null_uprn count applies only to the property-table lookup). Adds a dry_run
option to reconcile without writing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confirmed against the DB: Calico Asset References have zero overlap with
portfolio 824's landlord_property_id (addresses match; all 824 UPRNs non-null),
so the property-table lookup cannot resolve Calico. Documents the open
resolution (Calico-supplied ref->UPRN table or address matching) in ADR-0064,
the runner, and the README, and pins the confirmed S3 source location.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the agreed reconciliation semantics: null_uprn reports portfolio
properties missing a UPRN (expected zero) and may overlap unmatched_to_portfolio,
rather than strictly partitioning the Calico rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
run_calico_s3_load reads the Calico export from S3 and runs it through
run_calico_load against portfolio 824, resolving references to UPRNs via the
property table and persisting with ConditionPostgres. S3 + Postgres wiring
around the unit-tested load; run where a database and S3 are available.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
run_calico_load ties parse -> map -> persist together: it resolves references to
UPRNs, drops blank/unmatched rows, persists the roof-covering surveys with
export-year-anchored renewal years, and returns the four-count LoadReport.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Registers ConditionFileType.Calico in the parser/mapper factory; the Calico
parser requires a UPRN lookup (like Peabody) since references resolve to UPRNs
via the property table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LoadReport.from_calico_load reports loaded observations alongside the three drop
reasons (unmatched-to-portfolio, null-uprn, blank-placeholder) so every Calico
row is accounted for.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behaviour-preserving lift-and-shift of the condition module out of legacy
backend/condition/ into domain/condition, infrastructure/condition,
infrastructure/postgres/condition_tables.py, repositories/condition,
applications/condition, and tests/condition. Imports rewritten to the DDD
paths; ConditionPostgres and the ORM models keep the legacy backend.app.db
Base/db_session bridges so the existing suite proves behaviour is unchanged
(SQLModel + session-DI conversion tracked as a follow-up in ADR-0064).
16 condition tests pass at the new location.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
