merged from main

This commit is contained in:
Jun-te Kim 2026-06-08 09:36:26 +00:00
parent 4981b0ee12
commit 052ab446e0
12 changed files with 641 additions and 239 deletions

View file

@ -1,8 +1,28 @@
{
"permissions": {
"allow": [
"Read(//home/vscode/.claude/skills/grill-me/**)",
"Bash(grep \"\\\\.py$\")",
"Bash(git fetch *)",
"Bash(git add *)",
"Bash(git commit *)",
"Bash(git merge *)",
"Read(//home/vscode/.claude/skills/to-issues/**)",
"Read(//home/vscode/.claude/skills/triage/**)",
"Read(//home/vscode/.claude/skills/**)",
"Bash(echo \"tsc exit: $?\")",
"Bash(npm install *)",
"Bash(npx drizzle-kit *)",
"Bash(echo \"frontend tsc exit: $?\")"
],
"deny": [
"Bash(npx drizzle-kit generate)",
"Bash(npx drizzle-kit push)"
],
"additionalDirectories": [
"/workspaces/home/github/Model/backend/app/bulk_uploads",
"/workspaces/home/github/Model/applications/landlord_description_overrides",
"/workspaces/home/github/Model/orchestration"
]
}
}

View file

@ -118,10 +118,13 @@ framing, which was wrong once classifier header-sharing became a requirement.
POST-back.
- **State machine:** the classifier runs as a **subtask under the same address
task** (not a separate task, not a new `BulkUpload` status). Both subtasks
fire together at the **"Start address matching"** action. Safe: the combiner
globs S3 `ara_raw_outputs/{task_id}/`, which the classifier never writes to,
so the combined address output is not affected; `subtask_handler` does no
sibling-completion gating.
fire together at the **"Start address matching"** action. The combined address
*output* is not affected (the classifier writes Postgres, not
`ara_raw_outputs/{task_id}/`), **but** the parent Task *status* IS recomputed
from all subtasks (`TaskOrchestrator._cascade`), so a classifier failure fails
the onboarding task and gates the combiner. This coupling was knowingly
accepted (2026-05-28), superseding Q1 non-blocking — see ADR-0003's Amended
note. (Corrects an earlier wrong claim that the envelope does no gating.)
- **Progress honesty:** add a nullable **`service`/`kind` discriminator to
`sub_task`** (existing rows = address/legacy) so the progress view shows
address batches vs classification separately and attributes failures
@ -164,9 +167,9 @@ ADR-0002 anticipates.
Grilling is complete (Q1Q7). Suggested follow-ups:
1. Promote Q4 (trigger + state-machine integration) to a **frontend ADR-0003**;
cross-link the backend's ADR-0003 (which already superseded ADR-0002's
Next.js-writes clause).
1. **Done** — Q4 promoted to
[ADR-0003](../adr/0003-classifier-triggers-as-address-subtask.md) (trigger +
state-machine integration), cross-linking the backend's ADR-0003.
2. Break the work into issues (`/to-issues`): (a) invert the mapping shape + UI
+ migration + validation; (b) `sub_task` discriminator + progress view;
(c) classifier trigger (new FastAPI endpoint + payload, fire at "Start");

View file

@ -0,0 +1,88 @@
# Landlord override e2e — verification & deploy checklist
**Created:** 2026-05-28
**Branch:** `feature/frontend_landlord_overrides` (this repo) + `feature/landlord_data` (Model repo)
**Plan:** [landlord-override-frontend-plan.md](./landlord-override-frontend-plan.md) · **ADR:** [0003-classifier-triggers-as-address-subtask.md](../adr/0003-classifier-triggers-as-address-subtask.md)
## Context for picking this up cold
The landlord-classifier e2e is **implemented across both repos but uncommitted**, and
**statically verified only** (frontend `tsc` 0 errors, `next lint` clean, backend
`py_compile` clean). It has **not** been run live — that needs the steps below
(migrations applied, SQS queue + env, FastAPI endpoint + lambda deployed with
OpenAI/S3/Postgres access). Two migration files are **generated but not applied**:
`0215_invert_column_mapping.sql` (data) and `0216_add_subtask_service.sql` (schema).
Work through the sections in order — each step's prerequisites come first.
---
## A. Before you start
- [ ] Use dev/preview first, not prod.
- [ ] Confirm `.env.local` DB creds (`DB_HOST/PORT/USERNAME/PASSWORD/NAME`) point at the target DB.
- [ ] **Back up** `column_mapping` — the 0215 inversion is one-shot/irreversible:
```sql
CREATE TABLE _bak_bulk_mapping AS
SELECT id, column_mapping FROM bulk_address_uploads WHERE column_mapping IS NOT NULL;
```
## B. Database migrations ⚠️ read the gotcha
`0215` = data (inverts `header→field``field→header`); `0216` = schema (`ADD COLUMN sub_task.service`).
⚠️ package.json only has `migration:push` (`drizzle-kit push`). **`push` diffs schema and will NOT run the 0215 data `UPDATE`** — it would add the column but silently skip the inversion. Use `migrate`:
- [ ] ```bash
npx drizzle-kit migrate # runs 0215 then 0216 in order
```
(If the team only uses `push`: run `push` for 0216, then execute `0215`'s SQL manually.)
- [ ] ⚠️ Run **0215 exactly once, on old-shape data**. Re-running re-inverts and corrupts. `migrate` guards via the journal; manual runs don't.
- [ ] Verify 0215 — values should now be headers:
```sql
SELECT column_mapping FROM bulk_address_uploads WHERE column_mapping IS NOT NULL LIMIT 5;
-- expect {"address_1":"Addr 1","postcode":"PCode", ...}
```
- [ ] Verify 0216:
```sql
SELECT 1 FROM information_schema.columns
WHERE table_name='sub_task' AND column_name='service';
```
## C. Backend deploy (Model service)
- [ ] Create an SQS queue for the classifier (e.g. `landlord-description-overrides`).
- [ ] Set **`LANDLORD_OVERRIDES_SQS_URL`** in the FastAPI env to that queue.
- [ ] Deploy FastAPI so `/v1/bulk-uploads/trigger-landlord-overrides` is live.
- [ ] Deploy the lambda (`applications/landlord_description_overrides`) + event-source mapping queue → lambda.
- [ ] Lambda env/IAM: `OPENAI_API_KEY`, Postgres creds, **S3 read on the original-upload bucket** (it reads `upload.s3Bucket/s3Key`, not `retrofit-data-dev`).
## D. Frontend deploy
- [ ] Deploy `assessment-model` with the new code (`FASTAPI_API_URL` / `FASTAPI_API_KEY` already set).
## E. Verify the UI (Column Remapper)
- [ ] Map-columns page shows **one row per field with a header dropdown**, split into **Address fields** + **Landlord description fields**.
- [ ] Leaving Address 1 / Postcode unset blocks submit.
- [ ] Two **address** fields → one column is blocked; the **same** column → Property Type + Built Form is allowed.
- [ ] Existing `mapping_complete` uploads open with their mapping intact (confirms 0215).
## F. Verify trigger → classify → persist
- [ ] Map ≥1 landlord-description field, click **Start address matching**.
- [ ] `sub_task` has two rows under the task: `service='address2uprn'` and `service='landlord_description_overrides'`.
- [ ] SQS message enqueued + lambda ran (CloudWatch).
- [ ] Rows appear with `source='classifier'`:
```sql
SELECT description, value FROM landlord_property_type_overrides WHERE portfolio_id = <id> LIMIT 10;
```
## G. Verify the results view
- [ ] `/portfolio/<id>/landlord-overrides` lists `description → value` per category with a "classifier" badge. (No nav link yet — reach by URL.)
## H. Regression — address matching unaffected
- [ ] The same upload's address pipeline still emits the canonical CSV (`Address 1`/`postcode`) and combines normally.
## I. Watch-out (by design — ADR-0003 "accepted coupling")
- [ ] If the classifier subtask **fails**, the shared onboarding task goes FAILED and **"Run Combiner" is blocked**; the task only COMPLETEs once classification finishes. If painful, switch to the separate-task design in the ADR.
---
## Still open / not done
- [ ] Commit the work (this repo + Model repo, separately) — currently uncommitted.
- [ ] Nav link to `/portfolio/<id>/landlord-overrides` (reachable by URL only).
- [ ] User-edit write-back for overrides (deferred — Q7 "read-only this iteration").

View file

@ -3,20 +3,13 @@ import { getServerSession } from "next-auth";
import { AuthOptions } from "@/app/api/auth/[...nextauth]/authOptions";
import { createS3Client, createRetrofitDataS3Client, retrofitDataS3Bucket } from "@/app/utils/s3";
import * as XLSX from "xlsx";
import { loadForAddressMatching, triggerAddressMatching } from "@/lib/bulkUpload/server";
import { loadForAddressMatching, triggerAddressMatching, triggerClassifier } from "@/lib/bulkUpload/server";
import { readSessionToken } from "@/lib/session";
const FIELD_RENAME: Record<string, string> = {
address_1: "Address 1",
address_2: "Address 2",
address_3: "Address 3",
postcode: "postcode",
internal_reference: "Internal Reference",
};
import { ADDRESS_FIELDS } from "@/lib/bulkUpload/columnFields";
function transformFile(
buffer: Buffer,
columnMapping: Record<string, string>
columnMapping: Record<string, string> // field → source header
): { csv: string; error?: never } | { csv?: never; error: string } {
const wb = XLSX.read(buffer, { type: "buffer" });
const sheet = wb.Sheets[wb.SheetNames[0]];
@ -24,16 +17,13 @@ function transformFile(
if (rows.length === 0) return { error: "Empty file" };
const sourceHeaders = Object.keys(rows[0]);
const outputHeaders: string[] = [];
const sourceToOutput: Record<string, string> = {};
for (const src of sourceHeaders) {
const mapped = columnMapping[src];
if (!mapped || mapped === "skip") continue;
const renamed = FIELD_RENAME[mapped] ?? mapped;
outputHeaders.push(renamed);
sourceToOutput[src] = renamed;
const outputToSource: Record<string, string> = {};
for (const field of ADDRESS_FIELDS) {
const src = columnMapping[field.value];
if (!src || !field.outputHeader) continue;
outputHeaders.push(field.outputHeader);
outputToSource[field.outputHeader] = src;
}
if (!outputHeaders.includes("Address 1"))
@ -43,8 +33,8 @@ function transformFile(
const outputRows = rows.map((row) => {
const out: Record<string, unknown> = {};
for (const [src, renamed] of Object.entries(sourceToOutput)) {
out[renamed] = row[src] ?? "";
for (const [outName, src] of Object.entries(outputToSource)) {
out[outName] = row[src] ?? "";
}
return out;
});
@ -112,13 +102,13 @@ export async function POST(
const s3Uri = `s3://${outputBucket}/${transformedKey}`;
const trigger = await triggerAddressMatching({
uploadId,
s3Uri,
sessionToken: readSessionToken(request),
});
const sessionToken = readSessionToken(request);
const trigger = await triggerAddressMatching({ uploadId, s3Uri, sessionToken });
if (trigger.kind === "trigger_failed")
return NextResponse.json({ error: trigger.message }, { status: trigger.status });
// Co-fire the landlord classifier (non-blocking) under the same task.
await triggerClassifier({ taskId: trigger.taskId, uploadId, sessionToken });
return NextResponse.json({ taskId: trigger.taskId }, { status: 200 });
}

View file

@ -1513,69 +1513,6 @@
"when": 1779992128370,
"tag": "0216_add_subtask_service",
"breakpoints": true
},
{
"idx": 217,
"version": "7",
"when": 1780404222902,
"tag": "0217_gray_hellion",
"breakpoints": true
},
{
"idx": 218,
"version": "7",
"when": 1780408378351,
"tag": "0218_natural_umar",
"breakpoints": true
},
{
"idx": 219,
"version": "7",
"when": 1780419959831,
"tag": "0219_add_verify_ack",
"breakpoints": true
},
{
"idx": 220,
"version": "7",
"when": 1780491109956,
"tag": "0220_round_retro_girl",
"breakpoints": true
},
{
"idx": 221,
"version": "7",
"when": 1780566543108,
"tag": "0221_nice_sumo",
"breakpoints": true
},
{
"idx": 222,
"version": "7",
"when": 1780647165601,
"tag": "0222_nifty_hellcat",
"breakpoints": true
},
{
"idx": 223,
"version": "7",
"when": 1780647248894,
"tag": "0223_recommendation_plan_id_backfill",
"breakpoints": true
},
{
"idx": 224,
"version": "7",
"when": 1780653770494,
"tag": "0224_busy_nitro",
"breakpoints": true
},
{
"idx": 225,
"version": "7",
"when": 1780654800000,
"tag": "0225_recommendation_material_id_backfill",
"breakpoints": true
}
]
}

View file

@ -41,11 +41,16 @@ export default function OnboardingProgress({
}
const { task, upload } = progress.data;
const total = task?.totalSubtasks ?? 0;
const completedSubtasks = task?.completedSubtasks ?? 0;
const failedSubtasks = task?.failedSubtasks ?? 0;
// Address-matching batches drive the bar; classification is shown separately.
const total = task?.addressTotal ?? 0;
const completedSubtasks = task?.addressCompleted ?? 0;
const failedSubtasks = task?.addressFailed ?? 0;
const percent = total > 0 ? Math.round((completedSubtasks / total) * 100) : 0;
const classifierTotal = task?.classifierTotal ?? 0;
const classifierCompleted = task?.classifierCompleted ?? 0;
const classifierFailed = task?.classifierFailed ?? 0;
const taskStatus = task?.status.toLowerCase() ?? "";
const taskDone = TASK_TERMINAL_STATUSES.has(taskStatus);
const taskFailed = TASK_FAILED_STATUSES.has(taskStatus);
@ -76,6 +81,24 @@ export default function OnboardingProgress({
{failedSubtasks} failed
</span>
)}
{classifierTotal > 0 && (
<span className="flex items-center gap-1">
<span className="text-gray-400">Classification:</span>
{classifierFailed > 0 ? (
<span className="flex items-center gap-1 text-red-500 font-semibold">
<span className="w-1.5 h-1.5 rounded-full bg-red-400" />
failed
</span>
) : classifierCompleted >= classifierTotal ? (
<span className="font-semibold text-green-600">complete</span>
) : (
<span className="flex items-center gap-1 text-blue-500">
<span className="w-1.5 h-1.5 rounded-full bg-blue-400 animate-pulse" />
running
</span>
)}
</span>
)}
{!taskDone && (
<span className="flex items-center gap-1 text-blue-500">
<span className="w-1.5 h-1.5 rounded-full bg-blue-400 animate-pulse" />

View file

@ -10,38 +10,14 @@ import {
ArrowsRightLeftIcon,
} from "@heroicons/react/24/outline";
import { useSetColumnMapping } from "@/lib/bulkUpload/client";
const INTERNAL_FIELDS = [
{ value: "address_1", label: "Address 1", required: true },
{ value: "address_2", label: "Address 2", required: false },
{ value: "address_3", label: "Address 3", required: false },
{ value: "postcode", label: "Postcode", required: true },
{ value: "internal_reference", label: "Internal Reference (Optional)", required: false },
{ value: "skip", label: "Skip this column", required: false },
];
const REQUIRED_VALUES = ["address_1", "postcode"];
function autoDetect(header: string): string {
const h = header.toLowerCase().replace(/[\s_\-]/g, "");
if (/^(address|addr)(line)?(1|one)?$/.test(h)) return "address_1";
if (/^(address|addr)(line)?(2|two)|^street$/.test(h)) return "address_2";
if (/^(address|addr)(line)?(3|three)|^locality$|^town$|^city$/.test(h)) return "address_3";
if (/^post(al)?code$|^postcode$|^pcode$/.test(h)) return "postcode";
if (/^(internal)?ref(erence)?$|^id$/.test(h)) return "internal_reference";
return "skip";
}
function buildInitialMapping(
headers: string[],
existing?: Record<string, string>
): Record<string, string> {
const mapping: Record<string, string> = {};
for (const h of headers) {
mapping[h] = existing?.[h] ?? autoDetect(h);
}
return mapping;
}
import {
ADDRESS_FIELDS,
CLASSIFIER_FIELDS,
NOT_PROVIDED,
buildInitialMapping,
validateColumnMapping,
type InternalField,
} from "@/lib/bulkUpload/columnFields";
interface Props {
portfolioId: string;
@ -59,19 +35,24 @@ export default function MapColumnsClient({
existingMapping,
}: Props) {
const router = useRouter();
// mapping: internal field → source CSV header. Unmapped fields are absent.
const [mapping, setMapping] = useState<Record<string, string>>(
buildInitialMapping(sourceHeaders, existingMapping)
);
const setMappingMutation = useSetColumnMapping(portfolioId, uploadId);
const mappedValues = Object.values(mapping).filter((v) => v !== "skip");
const missingRequired = REQUIRED_VALUES.filter((r) => !mappedValues.includes(r));
const validationError = validateColumnMapping(mapping);
const submitting = setMappingMutation.isPending;
const error = setMappingMutation.error?.message ?? null;
const canSubmit = missingRequired.length === 0 && !submitting;
const requestError = setMappingMutation.error?.message ?? null;
const canSubmit = validationError === null && !submitting;
function setField(header: string, value: string) {
setMapping((prev) => ({ ...prev, [header]: value }));
function setField(field: string, header: string) {
setMapping((prev) => {
const next = { ...prev };
if (header === NOT_PROVIDED) delete next[field];
else next[field] = header;
return next;
});
}
function handleSubmit() {
@ -86,6 +67,86 @@ export default function MapColumnsClient({
);
}
function renderRow(field: InternalField) {
const value = mapping[field.value] ?? NOT_PROVIDED;
const isMapped = value !== NOT_PROVIDED;
return (
<div
key={field.value}
className="grid grid-cols-12 items-center px-6 py-4 hover:bg-gray-50/50 transition-colors"
>
{/* Internal field */}
<div className="col-span-4 flex items-center gap-3">
<div className="w-8 h-8 rounded-lg bg-gray-100 flex items-center justify-center shrink-0">
<TableCellsIcon className="h-4 w-4 text-gray-400" />
</div>
<div>
<p className="text-sm font-semibold text-gray-900">
{field.label}
{field.required && <span className="text-amber-600"> *</span>}
</p>
<p className="text-xs text-gray-400">
{field.kind === "classifier" ? "Landlord description" : "Internal field"}
</p>
</div>
</div>
{/* Arrow */}
<div className="col-span-1 flex justify-center">
<ArrowsRightLeftIcon className="h-4 w-4 text-gray-300" />
</div>
{/* Header picker */}
<div className="col-span-5">
<select
value={value}
onChange={(e) => setField(field.value, e.target.value)}
className="w-full text-sm border border-gray-200 rounded-lg px-3 py-2 bg-white text-gray-800 focus:outline-none focus:ring-2 focus:ring-[#14163d]/20 focus:border-[#14163d]"
>
<option value={NOT_PROVIDED}>Not provided</option>
{sourceHeaders.map((header) => (
<option key={header} value={header}>
{header}
</option>
))}
</select>
</div>
{/* Status badge */}
<div className="col-span-2 flex justify-end">
<span
className={`inline-flex items-center gap-1 px-2.5 py-1 rounded-full text-xs font-semibold ${
isMapped ? "bg-amber-50 text-amber-700" : "bg-gray-100 text-gray-400"
}`}
>
<span className="w-1.5 h-1.5 rounded-full bg-current opacity-70" />
{isMapped ? "Mapped" : "Not provided"}
</span>
</div>
</div>
);
}
function renderSection(title: string, subtitle: string, fields: InternalField[]) {
return (
<div className="bg-white border border-gray-100 rounded-2xl overflow-hidden shadow-sm mb-6">
<div className="px-6 py-3 bg-gray-50 border-b border-gray-100">
<p className="text-xs font-semibold text-gray-500 uppercase tracking-wider">
{title}
</p>
<p className="text-xs text-gray-400 mt-0.5">{subtitle}</p>
</div>
{sourceHeaders.length === 0 ? (
<div className="px-6 py-12 text-center text-sm text-gray-400">
No headers found in this file.
</div>
) : (
<div className="divide-y divide-gray-50">{fields.map(renderRow)}</div>
)}
</div>
);
}
return (
<div className="max-w-4xl mx-auto px-6 py-10">
{/* Breadcrumb + step */}
@ -116,102 +177,27 @@ export default function MapColumnsClient({
Column Remapper
</h1>
<p className="text-sm text-gray-500 max-w-lg">
Align your spreadsheet headers with our internal property data structure to
ensure accurate address processing.
Tell us which spreadsheet column feeds each field. Address fields drive
matching; landlord-description fields are classified into property facts.
</p>
</div>
{/* Table */}
<div className="bg-white border border-gray-100 rounded-2xl overflow-hidden shadow-sm mb-6">
{/* Column headers */}
<div className="grid grid-cols-12 items-center px-6 py-3 bg-gray-50 border-b border-gray-100">
<span className="col-span-4 text-xs font-semibold text-gray-400 uppercase tracking-wider">
Spreadsheet Header
</span>
<span className="col-span-1" />
<span className="col-span-5 text-xs font-semibold text-gray-400 uppercase tracking-wider">
Internal Field Mapping
</span>
<span className="col-span-2 text-xs font-semibold text-gray-400 uppercase tracking-wider text-right">
Status
</span>
</div>
{sourceHeaders.length === 0 ? (
<div className="px-6 py-12 text-center text-sm text-gray-400">
No headers found in this file.
</div>
) : (
<div className="divide-y divide-gray-50">
{sourceHeaders.map((header) => {
const value = mapping[header] ?? "skip";
const isMapped = value !== "skip";
return (
<div
key={header}
className="grid grid-cols-12 items-center px-6 py-4 hover:bg-gray-50/50 transition-colors"
>
{/* Source header */}
<div className="col-span-4 flex items-center gap-3">
<div className="w-8 h-8 rounded-lg bg-gray-100 flex items-center justify-center shrink-0">
<TableCellsIcon className="h-4 w-4 text-gray-400" />
</div>
<div>
<p className="text-sm font-semibold text-gray-900">{header}</p>
<p className="text-xs text-gray-400">Source column</p>
</div>
</div>
{/* Arrow */}
<div className="col-span-1 flex justify-center">
<ArrowsRightLeftIcon className="h-4 w-4 text-gray-300" />
</div>
{/* Dropdown */}
<div className="col-span-5">
<select
value={value}
onChange={(e) => setField(header, e.target.value)}
className="w-full text-sm border border-gray-200 rounded-lg px-3 py-2 bg-white text-gray-800 focus:outline-none focus:ring-2 focus:ring-[#14163d]/20 focus:border-[#14163d]"
>
{INTERNAL_FIELDS.map((f) => (
<option key={f.value} value={f.value}>
{f.label}
</option>
))}
</select>
</div>
{/* Status badge */}
<div className="col-span-2 flex justify-end">
<span
className={`inline-flex items-center gap-1 px-2.5 py-1 rounded-full text-xs font-semibold ${
isMapped
? "bg-amber-50 text-amber-700"
: "bg-gray-100 text-gray-400"
}`}
>
<span className="w-1.5 h-1.5 rounded-full bg-current opacity-70" />
{isMapped ? "Mapped" : "Skipped"}
</span>
</div>
</div>
);
})}
</div>
)}
</div>
{/* Validation error */}
{missingRequired.length > 0 && (
<p className="text-xs text-amber-600 mb-4">
Required fields not yet mapped:{" "}
{missingRequired
.map((r) => INTERNAL_FIELDS.find((f) => f.value === r)?.label)
.join(", ")}
</p>
{renderSection(
"Address fields",
"Used for address matching. A column can feed only one address field.",
ADDRESS_FIELDS
)}
{error && <p className="text-xs text-red-500 mb-4">{error}</p>}
{renderSection(
"Landlord description fields (optional)",
"Classified into property facts. Several fields may share one column.",
CLASSIFIER_FIELDS
)}
{/* Validation / request error */}
{validationError && (
<p className="text-xs text-amber-600 mb-4">{validationError}</p>
)}
{requestError && <p className="text-xs text-red-500 mb-4">{requestError}</p>}
{/* Footer */}
<div className="flex items-center justify-between">
@ -249,8 +235,10 @@ export default function MapColumnsClient({
Pro Tip
</p>
<p className="text-sm text-gray-500 italic">
&ldquo;Ensure your source file doesn&apos;t have blank headers. Any column mapped to
&ldquo;Skip&rdquo; will be ignored during import.&rdquo;
&ldquo;Fields left as &ldquo;Not provided&rdquo; are ignored. The same
column can feed several landlord-description fields e.g. one
&ldquo;Property Type&rdquo; column can drive both Property Type and Built
Form.&rdquo;
</p>
</div>
</div>

View file

@ -0,0 +1,101 @@
import { getServerSession } from "next-auth";
import { redirect } from "next/navigation";
import { AuthOptions } from "@/app/api/auth/[...nextauth]/authOptions";
import {
getLandlordOverrides,
type OverrideRow,
type LandlordOverrideCategory,
} from "@/lib/landlordOverrides/server";
import { CLASSIFIER_FIELDS } from "@/lib/bulkUpload/columnFields";
export default async function LandlordOverridesPage(props: {
params: Promise<{ slug: string }>;
}) {
const { slug } = await props.params;
const session = await getServerSession(AuthOptions);
if (!session) redirect("/login");
const results = await getLandlordOverrides(slug);
const total = Object.values(results).reduce((n, rows) => n + rows.length, 0);
return (
<div className="max-w-4xl mx-auto px-6 py-10">
<div className="mb-8">
<h1 className="text-3xl font-extrabold text-gray-900 tracking-tight mb-1">
Landlord overrides
</h1>
<p className="text-sm text-gray-500 max-w-lg">
Property facts classified from your bulk-upload descriptions. Read-only
editing comes later.
</p>
</div>
{total === 0 ? (
<div className="bg-white border border-gray-100 rounded-2xl px-6 py-12 text-center text-sm text-gray-400 shadow-sm">
No classified values yet. They appear here once a bulk upload with
landlord-description columns has been processed.
</div>
) : (
CLASSIFIER_FIELDS.map((field) => (
<OverrideSection
key={field.value}
title={field.label}
rows={results[field.value as LandlordOverrideCategory]}
/>
))
)}
</div>
);
}
function OverrideSection({ title, rows }: { title: string; rows: OverrideRow[] }) {
return (
<div className="bg-white border border-gray-100 rounded-2xl overflow-hidden shadow-sm mb-6">
<div className="flex items-center justify-between px-6 py-3 bg-gray-50 border-b border-gray-100">
<p className="text-xs font-semibold text-gray-500 uppercase tracking-wider">
{title}
</p>
<span className="text-xs text-gray-400">
{rows.length} {rows.length === 1 ? "value" : "values"}
</span>
</div>
{rows.length === 0 ? (
<div className="px-6 py-8 text-center text-sm text-gray-400">
No values for this category.
</div>
) : (
<div className="divide-y divide-gray-50">
{rows.map((row, i) => (
<div key={i} className="grid grid-cols-12 items-center px-6 py-3 gap-2">
<p
className="col-span-6 text-sm text-gray-700 truncate"
title={row.description}
>
{row.description}
</p>
<p className="col-span-4 text-sm font-semibold text-gray-900">
{row.value}
</p>
<div className="col-span-2 flex justify-end">
<SourceBadge source={row.source} />
</div>
</div>
))}
</div>
)}
</div>
);
}
function SourceBadge({ source }: { source: string }) {
const isUser = source === "user";
return (
<span
className={`inline-flex items-center px-2.5 py-1 rounded-full text-xs font-semibold ${
isUser ? "bg-emerald-50 text-emerald-700" : "bg-indigo-50 text-indigo-700"
}`}
>
{isUser ? "user" : "classifier"}
</span>
);
}

View file

@ -0,0 +1,100 @@
// Single source of truth for BulkUpload column mapping.
//
// The mapping is stored as `field → source CSV header` (one entry per mapped
// internal field). One source header may feed several CLASSIFIER fields (e.g.
// "Property Type" → both property_type and built_form_type) but at most one
// ADDRESS field — see docs/adr/0003-classifier-triggers-as-address-subtask.md
// and docs/wip/landlord-override-frontend-plan.md (Q2.2, Q3).
//
// Classifier field `value`s mirror the Model service's ClassifiableColumn names
// (property_type / built_form_type / wall_type / roof_type) so the mapping can
// be forwarded to the lambda trigger verbatim.
export type InternalFieldKind = "address" | "classifier";
export interface InternalField {
value: string;
label: string;
kind: InternalFieldKind;
required: boolean;
// Canonical header written into the address-matching CSV (address fields only).
outputHeader?: string;
}
export const INTERNAL_FIELDS: InternalField[] = [
{ value: "address_1", label: "Address 1", kind: "address", required: true, outputHeader: "Address 1" },
{ value: "address_2", label: "Address 2", kind: "address", required: false, outputHeader: "Address 2" },
{ value: "address_3", label: "Address 3", kind: "address", required: false, outputHeader: "Address 3" },
{ value: "postcode", label: "Postcode", kind: "address", required: true, outputHeader: "postcode" },
{ value: "internal_reference", label: "Internal Reference", kind: "address", required: false, outputHeader: "Internal Reference" },
{ value: "property_type", label: "Property Type", kind: "classifier", required: false },
{ value: "built_form_type", label: "Built Form", kind: "classifier", required: false },
{ value: "wall_type", label: "Wall Type", kind: "classifier", required: false },
{ value: "roof_type", label: "Roof Type", kind: "classifier", required: false },
];
export const ADDRESS_FIELDS = INTERNAL_FIELDS.filter((f) => f.kind === "address");
export const CLASSIFIER_FIELDS = INTERNAL_FIELDS.filter((f) => f.kind === "classifier");
export const CLASSIFIER_FIELD_VALUES = CLASSIFIER_FIELDS.map((f) => f.value);
export const REQUIRED_FIELD_VALUES = INTERNAL_FIELDS.filter((f) => f.required).map((f) => f.value);
// Sentinel for an unmapped field in the UI dropdown ("Not provided").
export const NOT_PROVIDED = "";
// header → address field detection. Classifier fields are never auto-detected
// (Q2.1): mapping them is always an explicit user choice.
export function autoDetectField(header: string): string | null {
const h = header.toLowerCase().replace(/[\s_\-]/g, "");
if (/^(address|addr)(line)?(1|one)?$/.test(h)) return "address_1";
if (/^(address|addr)(line)?(2|two)|^street$/.test(h)) return "address_2";
if (/^(address|addr)(line)?(3|three)|^locality$|^town$|^city$/.test(h)) return "address_3";
if (/^post(al)?code$|^postcode$|^pcode$/.test(h)) return "postcode";
if (/^(internal)?ref(erence)?$|^id$/.test(h)) return "internal_reference";
return null;
}
// Build the initial field→header mapping: keep any existing choices, then
// auto-fill address fields from the headers (first matching header wins).
export function buildInitialMapping(
sourceHeaders: string[],
existing?: Record<string, string>,
): Record<string, string> {
const mapping: Record<string, string> = { ...(existing ?? {}) };
for (const header of sourceHeaders) {
const field = autoDetectField(header);
if (!field) continue;
if (mapping[field] === undefined) mapping[field] = header;
}
return mapping;
}
// Validation shared by the client (live) and the server (authoritative).
// Returns the first problem as a message, or null when the mapping is valid.
export function validateColumnMapping(mapping: Record<string, string>): string | null {
for (const field of REQUIRED_FIELD_VALUES) {
if (!mapping[field]) {
const label = INTERNAL_FIELDS.find((f) => f.value === field)?.label ?? field;
return `${label} must be mapped to a column.`;
}
}
const usedAddressHeaders = new Set<string>();
for (const field of ADDRESS_FIELDS) {
const header = mapping[field.value];
if (!header) continue;
if (usedAddressHeaders.has(header)) {
return `Column "${header}" is mapped to more than one address field.`;
}
usedAddressHeaders.add(header);
}
return null;
}
// The classifier subset of the mapping (category → source header) that gets
// forwarded to the lambda trigger. Address fields are intentionally excluded.
export function classifierMapping(mapping: Record<string, string>): Record<string, string> {
const out: Record<string, string> = {};
for (const field of CLASSIFIER_FIELD_VALUES) {
if (mapping[field]) out[field] = mapping[field];
}
return out;
}

View file

@ -4,6 +4,8 @@ import { tasks } from "@/app/db/schema/tasks/tasks";
import { subTasks } from "@/app/db/schema/tasks/subtask";
import { count, desc, eq, sql } from "drizzle-orm";
import type { BulkUpload, BulkUploadStatus, ProgressView, TaskSummary } from "./types";
import { validateColumnMapping, classifierMapping } from "./columnFields";
import { SUBTASK_SERVICE } from "./types";
const REMAP_ALLOWED: ReadonlySet<BulkUploadStatus> = new Set([
"ready_for_processing",
@ -78,6 +80,12 @@ async function loadTaskSummary(taskId: string): Promise<TaskSummary | null> {
totalSubtasks: count(subTasks.id),
completedSubtasks: sql<number>`count(case when lower(${subTasks.status}) in ('completed', 'complete') then 1 end)::int`,
failedSubtasks: sql<number>`count(case when lower(${subTasks.status}) in ('failed', 'failure', 'error') then 1 end)::int`,
addressTotal: sql<number>`count(case when (${subTasks.service} = 'address2uprn' or ${subTasks.service} is null) and ${subTasks.id} is not null then 1 end)::int`,
addressCompleted: sql<number>`count(case when (${subTasks.service} = 'address2uprn' or ${subTasks.service} is null) and lower(${subTasks.status}) in ('completed', 'complete') then 1 end)::int`,
addressFailed: sql<number>`count(case when (${subTasks.service} = 'address2uprn' or ${subTasks.service} is null) and lower(${subTasks.status}) in ('failed', 'failure', 'error') then 1 end)::int`,
classifierTotal: sql<number>`count(case when ${subTasks.service} = 'landlord_description_overrides' then 1 end)::int`,
classifierCompleted: sql<number>`count(case when ${subTasks.service} = 'landlord_description_overrides' and lower(${subTasks.status}) in ('completed', 'complete') then 1 end)::int`,
classifierFailed: sql<number>`count(case when ${subTasks.service} = 'landlord_description_overrides' and lower(${subTasks.status}) in ('failed', 'failure', 'error') then 1 end)::int`,
})
.from(tasks)
.leftJoin(subTasks, eq(subTasks.taskId, tasks.id))
@ -94,13 +102,6 @@ export async function getProgressView(uploadId: string): Promise<ProgressView |
return { upload, task };
}
function validateMapping(mapping: Record<string, string>): string | null {
const values = Object.values(mapping);
if (!values.includes("address_1")) return "Mapping must include address_1.";
if (!values.includes("postcode")) return "Mapping must include postcode.";
return null;
}
export type SetMappingOutcome =
| { kind: "ok"; upload: BulkUpload }
| { kind: "not_found" }
@ -116,7 +117,7 @@ export async function setColumnMapping(
if (!REMAP_ALLOWED.has(upload.status as BulkUploadStatus))
return { kind: "invalid_status", current: upload.status };
const reason = validateMapping(mapping);
const reason = validateColumnMapping(mapping);
if (reason) return { kind: "invalid_mapping", reason };
const [updated] = await db
@ -174,6 +175,7 @@ export async function triggerAddressMatching(args: {
.values({
taskId: task.id,
status: "waiting",
service: SUBTASK_SERVICE.address,
inputs: JSON.stringify({ bulk_upload_id: args.uploadId }),
})
.returning();
@ -209,6 +211,62 @@ export async function triggerAddressMatching(args: {
return { kind: "ok", taskId: task.id };
}
// Co-fires the landlord classifier as a subtask under the address task. Reads
// the ORIGINAL upload (the address-matching CSV strips the description columns)
// and is non-blocking: a trigger failure marks only the classifier subtask, so
// address matching is unaffected. See ADR-0003.
export async function triggerClassifier(args: {
taskId: string;
uploadId: string;
sessionToken: string | undefined;
}): Promise<void> {
const upload = await loadById(args.uploadId);
if (!upload || !upload.columnMapping) return;
const columnMapping = classifierMapping(upload.columnMapping);
if (Object.keys(columnMapping).length === 0) return;
const [subTask] = await db
.insert(subTasks)
.values({
taskId: args.taskId,
status: "waiting",
service: SUBTASK_SERVICE.classifier,
inputs: JSON.stringify({ bulk_upload_id: args.uploadId }),
})
.returning();
const payload = {
task_id: args.taskId,
sub_task_id: subTask.id,
s3_uri: `s3://${upload.s3Bucket}/${upload.s3Key}`,
portfolio_id: Number(upload.portfolioId),
column_mapping: columnMapping,
};
const trigger = await triggerFastApiPipeline({
endpoint: "/v1/bulk-uploads/trigger-landlord-overrides",
payload,
sessionToken: args.sessionToken,
});
if (!trigger.ok) {
await db
.update(subTasks)
.set({
status: "failed",
outputs: JSON.stringify({ error: trigger.message }),
})
.where(eq(subTasks.id, subTask.id));
return;
}
await db
.update(subTasks)
.set({ status: "in progress", inputs: JSON.stringify(payload) })
.where(eq(subTasks.id, subTask.id));
}
export type CombineRetriggerOutcome =
| { kind: "triggered"; taskId: string; subTaskId: string }
| { kind: "already_combined" }

View file

@ -14,6 +14,13 @@ export type BulkUploadStatus = (typeof BULK_UPLOAD_STATUSES)[number];
export type BulkUpload = typeof bulkAddressUploads.$inferSelect;
// sub_task.service values. NULL is treated as address (legacy rows + the
// backend-spawned postcode-split children, which don't set it). See ADR-0003.
export const SUBTASK_SERVICE = {
address: "address2uprn",
classifier: "landlord_description_overrides",
} as const;
export type TaskSummary = {
id: string;
taskSource: string;
@ -25,6 +32,14 @@ export type TaskSummary = {
totalSubtasks: number;
completedSubtasks: number;
failedSubtasks: number;
// Per-pipeline breakdown so onboarding progress can separate address
// matching from landlord classification (ADR-0003).
addressTotal: number;
addressCompleted: number;
addressFailed: number;
classifierTotal: number;
classifierCompleted: number;
classifierFailed: number;
};
export type ProgressView = {

View file

@ -0,0 +1,79 @@
import { db } from "@/app/db/db";
import {
landlordPropertyTypeOverrides,
landlordBuiltFormTypeOverrides,
landlordWallTypeOverrides,
landlordRoofTypeOverrides,
} from "@/app/db/schema/landlord_overrides";
import { asc, eq } from "drizzle-orm";
export interface OverrideRow {
description: string;
value: string;
source: string;
}
export type LandlordOverrideCategory =
| "property_type"
| "built_form_type"
| "wall_type"
| "roof_type";
export type LandlordOverrideResults = Record<LandlordOverrideCategory, OverrideRow[]>;
const EMPTY: LandlordOverrideResults = {
property_type: [],
built_form_type: [],
wall_type: [],
roof_type: [],
};
// Reads the four landlord_*_overrides tables for a portfolio. The portfolio id
// is a bigint FK; the bulk-upload flow carries it as a numeric string.
export async function getLandlordOverrides(
portfolioId: string
): Promise<LandlordOverrideResults> {
if (!/^\d+$/.test(portfolioId)) return EMPTY;
const pid = BigInt(portfolioId);
const [property_type, built_form_type, wall_type, roof_type] = await Promise.all([
db
.select({
description: landlordPropertyTypeOverrides.description,
value: landlordPropertyTypeOverrides.value,
source: landlordPropertyTypeOverrides.source,
})
.from(landlordPropertyTypeOverrides)
.where(eq(landlordPropertyTypeOverrides.portfolioId, pid))
.orderBy(asc(landlordPropertyTypeOverrides.description)),
db
.select({
description: landlordBuiltFormTypeOverrides.description,
value: landlordBuiltFormTypeOverrides.value,
source: landlordBuiltFormTypeOverrides.source,
})
.from(landlordBuiltFormTypeOverrides)
.where(eq(landlordBuiltFormTypeOverrides.portfolioId, pid))
.orderBy(asc(landlordBuiltFormTypeOverrides.description)),
db
.select({
description: landlordWallTypeOverrides.description,
value: landlordWallTypeOverrides.value,
source: landlordWallTypeOverrides.source,
})
.from(landlordWallTypeOverrides)
.where(eq(landlordWallTypeOverrides.portfolioId, pid))
.orderBy(asc(landlordWallTypeOverrides.description)),
db
.select({
description: landlordRoofTypeOverrides.description,
value: landlordRoofTypeOverrides.value,
source: landlordRoofTypeOverrides.source,
})
.from(landlordRoofTypeOverrides)
.where(eq(landlordRoofTypeOverrides.portfolioId, pid))
.orderBy(asc(landlordRoofTypeOverrides.description)),
]);
return { property_type, built_form_type, wall_type, roof_type };
}