Green on arrival — rename() records rejections rather than raising.
Pinned after verifying it bites: letting the rejection escape fails the
child sub_task, which turns the whole Task red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Properties are planned and renamed one at a time, so only those with work
get a row and a run killed by the timeout still leaves every property it
reached both renamed and recorded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The local invoke now sends an SQS-shaped payload — a direct invoke is the
one shape that cannot reproduce the dry_run bug class. Test modules route
protected-method calls through typed helpers so both pass pyright strict.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Green on arrival: both fall out of the summary the run now returns and of
the guarded site lookup. Pinned after verifying they bite — an unguarded
DomnaSites[name] completes an "ECO" run that renames files under a
different site's alias.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Green on arrival: @task_handler records inputs and reads source_id from
the body key matching the source's value. Pinned because it is the only
place the new sharepoint_site member is proven against a real Postgres
enum rather than a Python one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Green on arrival: reading the request from the SQS record body is what
taking the task lane gave us. Verified the guard bites by restoring the
raw-event read, under which the "dry" run renames a live file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six properties failed task 4d006fab (portfolio 854 / scenario 1334) with
DegeneratePredictionError. None has a lodged EPC, so each was predicted from a
neighbouring-cert cohort — and every cohort cert lodges its single building part
as `identifier: "complete"`. from_api_string accepted only "Main Dwelling" /
"Main building", so "complete" fell to OTHER, _has_main_part returned False, and
prediction refused to proceed.
"complete" is what SAP 9.91/9.92-era software writes for an unextended dwelling:
semantically identical to "Main Dwelling". Casing varies within a single postcode
(cert 8824-7422-1180-6934-1902 lodges "complete", 199 Highfield Road "Complete"),
so matching is now case-folded.
The API's `identifier` is free text with no schema enum — api.yml declares the
whole cert body `additionalProperties: true` and documents no field of it, and
`identifier` is absent from the 17 vocabularies at /api/codes. So the recognised
set can only grow by discovery, and each miss has cost an incident ("Main
building" → task a40e71c4; "complete" → task 4d006fab). Hence three changes, not
one:
- "complete" → MAIN, case-insensitively.
- Bare "Extension" → EXTENSION_1. The regex required a digit, so 668 corpus
occurrences silently dropped a real extension from the structure. This reverses
a prior deliberate pin whose stated rationale (RdSAP10 §1.2's 4-extension cap)
does not apply — there is no out-of-range number in a bare "Extension".
- _anchor_lone_building_part: a cert lodging exactly ONE part describes the whole
dwelling whatever it is called, so an unclassified lone part becomes MAIN.
Multi-part certs are untouched — there the identifier carries real information
and guessing would invent structure the cert does not state.
Also removes a false claim from DegeneratePredictionError's message: it asserted
the template was "lodged with a null part identifier", which was hardcoded, never
checked against a real cert, and wrong. It misdirected this investigation.
Verified against the live gov API: all 13 cohort certs across BL1 8EB, BL3 6XJ
and BL4 0RA now resolve a MAIN part. The full modelling e2e was NOT run (no local
AWS creds for the geospatial lookup) — this clears the blocking error, it does not
prove the predictions are good.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Plan became default only when its Scenario was flagged default or the
property had no Plan yet. Scenarios are created is_default=False, so in
practice only the first-ever Plan was ever promoted — re-modelling a
property appended a new Plan but left the default frozen on the first run,
so a re-model never surfaced on the FE (property 792168 kept its 2026-07-22
scenario-1328 default through three same-scenario re-runs).
Also promote when the re-run models the SAME scenario that currently owns
the property's default, so a re-run refreshes the shown Plan in place (the
save demotes the stale one). A re-run of a different scenario still does not
hijack the default. New PlanRepository.default_plan_scenario_by_property
surfaces each property's default scenario for the decision.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
refetch_epc=True fetched a fresh cert but the recency tie-break (strict >)
dropped it whenever the stored row shared the cert's inspection date — so a
row written before a schema field/table existed (e.g. room-in-roof geometry)
could never be healed by re-fetching. The dwelling was modelled on stale
fabric indefinitely.
Reconcile instead of just comparing dates: on an equal inspection date,
refresh a public gov cert (uploaded_file_id NULL) when the fetched cert
differs from the stored one, so a plain re-fetch re-ingests it. One of our
own surveys (PasHub / ECMK / Elmhurst — uploaded_file_id set) is preserved on
the tie, keeping the survey-wins rule (ADR-0001 / #1589); an unchanged public
cert is still not re-persisted.
`_reconcile_lodged` layers the content refresh over `_newer_lodged` (recency
untouched). New `EpcRepository.survey_lodged_uprns` surfaces the row-level
uploaded_file_id provenance the hydrated EpcPropertyData drops. Together with
the property_id-anchored persist, a refetch_epc=True re-run now heals both a
missing epc_property row and a stale one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The lodged EPC read keys on UPRN (ADR-0001 recency tie-break), but writes
and the front end both key on property_id. When a dwelling is onboarded
under two portfolios, the shared UPRN surfaces the sibling's lodged row,
so `lodged_epc_is_new` stays False and the modelling run skips persisting
a copy for this property — leaving it with no epc_property row the FE can
read. The EPC is modelled but invisible (portfolio 850 / property 792235,
uprn 100021979925, also under property 753896 in portfolio 830).
Drive the lodged persist off "does THIS property own a lodged row?"
(property_id-scoped) rather than "is the cert new?" (UPRN-scoped): add
`EpcRepository.property_ids_with_lodged_epc` and persist a per-property
copy whenever the cert is new OR the property has no lodged row of its
own. Idempotent (save_batch is replace-by-property_id), and the recency
skip still holds for a property that already owns its row.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The service was the last one living wholly under backend/. It now follows
the same layering as abri and the other newer services:
domain/pashub_fetcher/ core file classification, subfolders
infrastructure/pashub_fetcher/ PasHub client, token getter, wire DTOs
orchestration/ PashubFetcherOrchestrator (was PashubService)
applications/pashub_fetcher/ lambda handler, trigger request, dev tooling
core_files.py is split along the layer boundary: the domain module keeps the
filename/evidence-category classification rules and no longer imports
infrastructure.postgres, while the CoreFiles -> FileTypeEnum translation moves
to infrastructure/pashub_fetcher/core_file_types.py.
Tests move into the tests/ tree by layer. Note this puts them in the only
suite CI currently runs (unit_tests.yml is disabled), so these 73 tests now
execute on PRs for the first time; they were previously reachable only via
the legacy pytest.ini testpaths.
sharepoint_renamer's image now copies just domain/pashub_fetcher/ rather than
the whole service, since SharepointSubfolders is all it needed.
Behaviour is unchanged. tests/ goes 9927 -> 10000 passed (+73, exactly the
tests that moved in); the legacy suite keeps its same 17 pre-existing failures
and 11 errors. tests/test_lambda_packaging.py confirms both changed
Dockerfiles still copy their handler's full import closure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deal's third_party_surveyor_identifier is now required by the log
and amend flows: the trigger contract rejects messages without it
(blank treated as missing, since a blank resource makes OpenHousing
silently drop the appointment), and LogJobRequest/AmendJobRequest carry
it as a required field. The ABRI_RELAY_DEFAULT_RESOURCE fallback and
its config/terraform/workflow wiring are removed as dead code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Carry third_party_surveyor_identifier on the Abri trigger message and
through dispatch into the log and amend flows, booking the job against
that resource. LogJobRequest gains the optional resource field
AmendJobRequest already had; the client falls back to the configured
default surveyor when the deal names no third-party resource, so
existing deals behave unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Read stored lodged EPCs whatever refetch_epc says, so a stored assessment
can win the ADR-0001 Recency Tie-Break against the gov register.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-0054 gives a prediction conditioned by an expired Historic EPC the source
"expired", so reporting can tell "no EPC at all" apart from "only an expired
one". It was implemented in IngestionOrchestrator — but the pipeline that
actually runs is modelling_e2e, which predicts through its own _predict_epc and
never adopted it. Result: zero `expired` rows have ever been written, and
`epc_property.source` holds only 'lodged' and 'predicted'.
Two things kept the flavour unreachable here, and both had to go:
- _predict_epc never looked at the historic backup at all, so no prediction
was ever conditioned.
- _flush_writes passed the literal source="predicted", and _PropertyWrite had
no field to carry a flavour, so one would have been dropped before the write
even if computed.
_predict_epc now mirrors IngestionOrchestrator._predict: the expired cert's
stable attributes fill the gaps Landlord Overrides left (overrides still win
where both speak) and condition the cohort, and it returns the source alongside
the EPC. _PropertyWrite carries it; _flush_writes persists it. save_batch
already groups deletes by source family, so a mixed predicted/expired batch
clears the shared slot correctly.
Ships dark: the reader is built only when HISTORIC_EPC_S3_ROOT is set, so
without it every prediction stays plain "predicted", exactly as today.
DEFAULT_S3_ROOT names the dev bucket, so defaulting to it would have a prod
lambda silently reading dev data — Terraform sets the var per environment.
Note this also rescues properties that currently fail to model outright: an
expired cert can supply the property_type no override resolved, where today
that raises UnresolvedPropertyTypeError.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An epc_property row can carry a null property_id (FE/historic ingestion
persists EPC rows never linked to a property row), so a property_id-keyed
read silently misses them. UPRN is the durable key both the property row
and the epc_property row share. Repoint the lodged reads (get_for_property
+ get_for_properties) onto UPRN; the predicted reads stay on property_id
because a predicted EPC deep-copies a neighbour's UPRN, so its UPRN column
is never the property's own.
UPRN is not unique on epc_property, so the read tie-break is now load-
bearing (it was dormant under property_id, where the write path guarantees
one lodged row per property): the most recently ingested (highest-id) row
wins. Write path stays keyed on (property_id, source).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires POST /v1/exports/scenario -> tasks.inputs recipe -> pinned sub_task ->
ARA_EXPORT_SQS_URL -> ara_export Lambda -> orchestrator. Route resolution and the
trigger body are covered by tests; the Lambda handler mirrors bulk_document_download.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
date_abandoned now resolves to the third failed attempt's confirmed
survey date, falling back to its last submission date, rather than the
processing date. This dates the OpenHousing cancellation to when the
job actually lapsed even if the trigger message lags or redelivers.
- domain: abandonment_date() encodes the confirmed-survey -> last-
submission fallback
- DealAbandonment carries both dates; abandon_job raises
AbandonmentDateUnknownError when a deal has neither
- last_submission_date now flows through the trigger message and request
- the injected clock is dropped: nothing reads now() any more
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When refetch_epc=False and no stored lodged EPC exists, the handler no longer
falls back to a live EPC API call — it treats the property as EPC-less and
hands it to the prediction path. This keeps REFETCH_EPC (lodged path) and
REPREDICT_EPC (prediction path) cleanly independent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Even after batching the data writes, the handler still wrote to the DB per
property through the orchestrator's SubTask bookkeeping: create + start +
complete each self-committed, and _cascade re-listed every sibling and re-saved
the parent on every transition — ~5 writes per property plus an O(N^2) cascade.
- TaskOrchestrator.run_subtasks: create all children in one INSERT, run each
(failures isolated per child), then persist all terminal states in one bulk
save and cascade the parent once. Children go WAITING -> terminal; the
transient IN_PROGRESS row is never written.
- SubTaskRepository.create_many / save_many (bulk INSERT / bulk fetch + update).
- _cascade short-circuits when the Task is already FAILED (terminal) — skips the
sibling roll-up entirely.
- modelling_e2e handler fans out via run_subtasks instead of per-property
create_child_subtask + run_subtask.
Per N-property batch the SubTask bookkeeping drops from ~5N writes + an O(N^2)
cascade to ~2 writes + 1 cascade.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
