Model/docs/adr/0057-uprn-confirmation-precedes-finalise.md

UPRN confirmation precedes finalise; the bulk-upload review is one two-tab page and the portfolio "unmatched" tab is retired

Status

proposed

Context

The bulk-upload pipeline is a chain of SQS-driven Lambda stages: address2uprn → landlord-overrides → combiner → (review) → finaliser. address2uprn matches each input row independently against per-postcode EPC candidates, keeping the best candidate above a lexiscore >= 0.7 floor. There is no cross-row uniqueness guard: nothing stops one UPRN being the best match for two different addresses. The textbook case is a coarse EPC record 42 Moreton Road (no flat token) that Flat 1, 42 Moreton Road and Flat 2, 42 Moreton Road both clear — the building-number guard passes and the flat guard never fires, so both distinct flats collapse onto one UPRN.

Downstream this is corrosive, not just noisy:

• The property identity insert is on_conflict_do_nothing on the partial unique index (portfolio_id, uprn) WHERE uprn IS NOT NULL. Two distinct addresses carrying the same UPRN therefore merge into a single property — one address silently loses its identity.
• _build_overrides (the finaliser) keys property_overrides on (property_id, override_component, building_part). Both merged rows emit the same tuple, so the single INSERT ... ON CONFLICT DO UPDATE raises CardinalityViolation ("cannot affect row a second time").

The classifier and finaliser were built assuming a UPRN is always intact and unique; it is not. Separately, a post-onboarding portfolio-level "Unmatched properties" tab let users fix no-UPRN properties after the fact — too late (the identity merge has already happened) and in a different place from the rest of review.

Decision

1. address2uprn emits a UPRN only when confident and unambiguous. Within a postcode group, a UPRN that is the best match for two or more rows whose normalised input addresses differ is withheld (dropped to null) on every such row; a UPRN shared only by rows with the same normalised address is a genuine re-listing and is kept. address2uprn emits an address2uprn_status column (matched | ambiguous_duplicate | unmatched | invalid_postcode | error); withheld rows keep their lexiscore for triage. A lexiscore review band [0.7, X) that also withholds shaky matches is a follow-on once X is calibrated (see Consequences).
2. One pre-finalise review page, two tabs, styled like the retired unmatched tab:
   • Addresses — the flagged rows (unmatched / ambiguous_duplicate / low-score); the user resolves each via OS Places (reusing the existing MatchAddress / searchAddresses / useAssignUprn flow + postcode_search cache), producing a valid residential UPRN, or explicitly marks no-UPRN.
   • Classification — the existing description→value verify gate, semantics unchanged (every Unknown must be mapped; the finaliser still fails loud on an unresolved value). Finalise is enabled only when both tabs are clear.
3. The finaliser runs only after confirmation, so every row carries a confirmed UPRN or an explicit no-UPRN (a first-class terminal state: property identity with uprn NULL, no property_overrides, building-passport shows the "not matched yet" card).
4. Retire the portfolio-level "Unmatched properties" tab — properties now arrive pre-matched, so post-onboarding UPRN cleanup is unnecessary.
5. PropertyOverridePostgresRepository.upsert_all keeps a defensive last-write-wins dedup as a backstop for the single-statement invariant, but it is no longer the primary fix.

Consequences

• Distinct addresses can no longer silently merge into one property; the CardinalityViolation can no longer arise from real data.
• Onboarding reaching "complete" now implies confirmed identities, so portfolio pages need no unmatched tab.
• The ambiguous/low-score subset adds bounded user work at review time; confident rows pass untouched.
• The lexiscore review-band threshold X needs calibrating against a real portfolio's score distribution before it is switched on; until then only the ambiguous-duplicate rule (which needs no threshold) is active.
• Migration is safe stage-by-stage: withheld UPRNs are caught by the existing unmatched tab until the two-tab page ships, and the tab is removed only after it does — so an ambiguous property always has somewhere to be resolved.