mirror of
https://github.com/Hestia-Homes/assessment-model.git
synced 2026-07-27 22:45:03 +00:00
Merge pull request #448 from Hestia-Homes/issue-415-import-column-mapping
feat(ara-projects): work-order import 2/3 — column mapping + value mapping
This commit is contained in:
Daniel Roth
GitHub
commit
bf3f8aeb20
17 changed files with 3611 additions and 12 deletions
91
docs/adr/0020-import-session-and-mapping-are-client-held.md
Normal file
91
docs/adr/0020-import-session-and-mapping-are-client-held.md
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
# 20. The work-order import session — and the mapping persisted alongside it — is client-held
|
||||
|
||||
Date: 2026-07-23
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
Numbering note: two ADRs carry 0019 (the UK date format one, and the
|
||||
work-order-import one from #414's branch). 0020 follows both.
|
||||
|
||||
## Context
|
||||
|
||||
Issue #415 (import step 2 of 3) asks for two mapping layers over the parsed
|
||||
file — column mapping and value mapping — and for the result to be
|
||||
"persist[ed] alongside the import session".
|
||||
|
||||
The import journey has no session row. #414 decided that the parse route holds
|
||||
nothing: it reads the uploaded file, returns **every** row to the browser, and
|
||||
writes nothing. The mapping, validation and commit steps all read that array
|
||||
from the import screen's React state, and commit (#417) is the first and only
|
||||
write. There is no `import_session` table, no staged file in S3, and no
|
||||
`work_order` row until the user commits.
|
||||
|
||||
So "alongside the import session" had to be given a meaning. Three options:
|
||||
|
||||
1. **A database session.** Add `import_session` (file identity, status, a JSONB
|
||||
mapping), write it when the file is parsed, PATCH it as the user maps.
|
||||
2. **Browser storage.** Keep the session in React state and mirror it to
|
||||
`sessionStorage` so a refresh can restore it.
|
||||
3. **A client-held session object.** Define `ImportSession` — file identity plus
|
||||
both mapping layers — hold it in the import screen's state next to the parsed
|
||||
rows, and hand it to commit.
|
||||
|
||||
## Decision
|
||||
|
||||
**Option 3.** `src/lib/projects/import/session.ts` defines `ImportSession`; the
|
||||
import screen owns one, the mapping component edits it, and #417 will POST it
|
||||
with the rows at commit.
|
||||
|
||||
The shape is deliberately **JSON-clean end to end** — plain objects, string ids
|
||||
(every `bigserial` is stringified at `mappingTargets`), no `bigint`, no `Date`,
|
||||
no class instances. It can be POSTed verbatim and stored in a JSONB column
|
||||
unchanged, which is how `bulk_address_uploads.column_mapping` already holds the
|
||||
older BulkUpload journey's equivalent.
|
||||
|
||||
`sessionMatchesFile` guards resumption: a mapping is a set of **column
|
||||
indices**, so a session is meaningless against a file whose headings differ, and
|
||||
restoring one would silently map the wrong columns.
|
||||
|
||||
### Why not the database session
|
||||
|
||||
Persisting the mapping without the rows buys the user nothing. The rows are
|
||||
client-held, so a refresh loses them regardless; the user must re-upload, and a
|
||||
restored mapping cannot be applied to a file that is no longer in hand. A
|
||||
session row would therefore be durable state that no journey can currently
|
||||
resume from — and it would cost a migration, a status lifecycle, a PATCH
|
||||
endpoint and an expiry story to earn it.
|
||||
|
||||
There is also a constraint worth recording plainly: this branch works against a
|
||||
shared production database and was scoped to write nothing — no migrations. That
|
||||
ruled option 1 out for *this* ticket, but it is not the reason it was rejected;
|
||||
the reason is that the rows and the mapping have to be durable together or
|
||||
neither is useful.
|
||||
|
||||
### Why not `sessionStorage`
|
||||
|
||||
It survives a refresh, but only for the mapping — the rows still do not, so the
|
||||
restored mapping has no file to apply to. It also has to be read at mount, which
|
||||
in a server-rendered client component means either a hydration mismatch or an
|
||||
effect, and this codebase avoids `useEffect` by convention.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A page refresh loses the import in progress, and the user re-uploads. This is
|
||||
unchanged from #414, and the mapping does not make it worse.
|
||||
- Stepping back to the drop zone and forward again does **not** lose the
|
||||
mapping: the session lives in `ImportUpload`, above the mapping component, so
|
||||
the mapping component may unmount freely. This is why every edit reports the
|
||||
whole session upward rather than only living in the table's own state.
|
||||
- No server-side record exists of who mapped what, or of a mapping that was
|
||||
never committed. Nothing in v1 asks for one; if audit becomes a requirement,
|
||||
it lands with the session row, not before.
|
||||
- Two mapping layers on 20,000 rows are re-derived on every edit rather than
|
||||
cached. That is affordable (a few passes over three cells per row) and is what
|
||||
keeps the tables from disagreeing with the mapping the user is editing.
|
||||
- **What would change the answer:** moving the file server-side. If the parsed
|
||||
rows are ever staged (S3, or a rows table) — which is the natural fix for
|
||||
files too large to hold in the browser, and the shape the BulkUpload journey
|
||||
already uses — then a session row becomes worth its cost, and `ImportSession`
|
||||
is already the payload it would store.
|
||||
|
|
@ -0,0 +1,176 @@
|
|||
"use client";
|
||||
|
||||
import {
|
||||
assignColumn,
|
||||
columnMappingRows,
|
||||
IMPORT_FIELDS,
|
||||
IMPORT_FIELDS_BY_KEY,
|
||||
IMPORT_REQUIREMENTS,
|
||||
summariseColumnMapping,
|
||||
type ImportColumnMapping,
|
||||
type ImportFieldKey,
|
||||
} from "@/lib/projects/import/columnMapping";
|
||||
import { MappingSelect, MappingStatusIcon } from "./MappingSelect";
|
||||
|
||||
/** Fields named by a requirement carry the "required" tag in the picker. */
|
||||
const REQUIRED_KEYS = new Set<ImportFieldKey>(
|
||||
IMPORT_REQUIREMENTS.flatMap((requirement) => requirement.keys),
|
||||
);
|
||||
|
||||
/**
|
||||
* Layer 1 of the mapping step (issue #415): one row per **file column**, each
|
||||
* pointed at an Ara field, at "don't import", or at nothing yet.
|
||||
*
|
||||
* Rows are file columns rather than Ara fields because that is the direction
|
||||
* the wireframe reads (file → Ara) and the direction the user thinks in — they
|
||||
* are looking at their own spreadsheet. Which *fields* are still missing is a
|
||||
* separate question, answered by the blocker list beneath the table, since a
|
||||
* missing field has no row of its own to complain from.
|
||||
*
|
||||
* Stateless: it renders the mapping it is handed and reports edits upward, so
|
||||
* the whole screen re-derives from one value.
|
||||
*/
|
||||
export function ColumnMappingTable({
|
||||
headers,
|
||||
preview,
|
||||
mapping,
|
||||
onChange,
|
||||
}: {
|
||||
headers: string[];
|
||||
/** Sample rows, for the "e.g. …" hint under each heading. */
|
||||
preview: string[][];
|
||||
mapping: ImportColumnMapping;
|
||||
onChange: (next: ImportColumnMapping) => void;
|
||||
}) {
|
||||
const rows = columnMappingRows(headers, mapping, preview);
|
||||
const summary = summariseColumnMapping(headers, mapping);
|
||||
|
||||
return (
|
||||
<section
|
||||
data-testid="column-mapping"
|
||||
className="rounded-xl border border-gray-200 bg-white overflow-hidden"
|
||||
>
|
||||
<header className="flex items-center justify-between border-b border-gray-200 bg-gray-50 px-4 py-3">
|
||||
<div>
|
||||
<h2 className="text-sm font-semibold text-gray-800">Column mapping</h2>
|
||||
<p className="text-xs text-gray-500 mt-0.5">
|
||||
Match each column in your file to the field it holds.
|
||||
</p>
|
||||
</div>
|
||||
<span className="rounded-full bg-white border border-gray-200 px-2 py-1 text-xs font-medium text-gray-500">
|
||||
File → Ara
|
||||
</span>
|
||||
</header>
|
||||
|
||||
<div className="overflow-x-auto">
|
||||
<table className="min-w-full text-sm">
|
||||
<thead className="bg-white">
|
||||
<tr className="text-left text-xs font-semibold uppercase tracking-wide text-gray-500">
|
||||
<th className="px-4 py-2 w-5/12">File column</th>
|
||||
<th className="px-4 py-2 w-1/12 text-center">Status</th>
|
||||
<th className="px-4 py-2 w-6/12">Ara field</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody className="divide-y divide-gray-100">
|
||||
{rows.map((row) => (
|
||||
<tr
|
||||
key={row.columnIndex}
|
||||
data-testid="column-mapping-row"
|
||||
data-status={row.status}
|
||||
className={
|
||||
row.status === "needs_review"
|
||||
? "bg-red-50/40 border-l-4 border-l-red-400"
|
||||
: ""
|
||||
}
|
||||
>
|
||||
<td className="px-4 py-2.5 align-top">
|
||||
<div className="font-medium text-gray-800">{row.header}</div>
|
||||
{row.sample !== "" && (
|
||||
<div className="text-xs text-gray-500 truncate max-w-[16rem]">
|
||||
e.g. {row.sample}
|
||||
</div>
|
||||
)}
|
||||
</td>
|
||||
<td className="px-4 py-2.5 text-center align-top">
|
||||
<MappingStatusIcon status={row.status} />
|
||||
</td>
|
||||
<td className="px-4 py-2 align-top">
|
||||
<MappingSelect
|
||||
ariaLabel={`Ara field for the ${row.header} column`}
|
||||
testId={`column-field-${row.columnIndex}`}
|
||||
value={row.field ?? (row.status === "ignored" ? "__ignore__" : null)}
|
||||
invalid={row.status === "needs_review"}
|
||||
placeholder="Choose a field…"
|
||||
noneLabel="Not mapped yet"
|
||||
onChange={(next) =>
|
||||
onChange(
|
||||
assignColumn(
|
||||
mapping,
|
||||
row.columnIndex,
|
||||
next === "__ignore__"
|
||||
? "ignore"
|
||||
: (next as ImportFieldKey | null),
|
||||
),
|
||||
)
|
||||
}
|
||||
options={[
|
||||
...IMPORT_FIELDS.map((field) => {
|
||||
const heldBy = mapping.fields[field.key];
|
||||
return {
|
||||
value: field.key,
|
||||
label: REQUIRED_KEYS.has(field.key)
|
||||
? `${field.label} (required)`
|
||||
: field.label,
|
||||
// Say so when picking this field would take it off
|
||||
// another column, rather than silently moving it.
|
||||
hint:
|
||||
heldBy !== undefined && heldBy !== row.columnIndex
|
||||
? `currently ${headers[heldBy]}`
|
||||
: field.hint,
|
||||
};
|
||||
}),
|
||||
{ value: "__ignore__", label: "Don't import this column" },
|
||||
]}
|
||||
/>
|
||||
</td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
{summary.missingRequirements.length > 0 && (
|
||||
<div
|
||||
role="alert"
|
||||
data-testid="column-mapping-blockers"
|
||||
className="border-t border-red-200 bg-red-50 px-4 py-3"
|
||||
>
|
||||
<p className="text-sm font-semibold text-red-900">
|
||||
{summary.missingRequirements.length} required field
|
||||
{summary.missingRequirements.length === 1 ? "" : "s"} still to map
|
||||
</p>
|
||||
<ul className="mt-1.5 space-y-1">
|
||||
{summary.missingRequirements.map((requirement) => (
|
||||
<li key={requirement.id} className="text-xs text-red-800">
|
||||
<span className="font-semibold">{requirement.label}</span> —{" "}
|
||||
{requirement.reason}
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{summary.complete && summary.unmappedOptionalFields.length > 0 && (
|
||||
<div className="border-t border-gray-100 bg-gray-50/60 px-4 py-2.5">
|
||||
<p className="text-xs text-gray-500">
|
||||
Not mapped (optional):{" "}
|
||||
{summary.unmappedOptionalFields
|
||||
.map((key) => IMPORT_FIELDS_BY_KEY[key].label)
|
||||
.join(", ")}
|
||||
.
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
);
|
||||
}
|
||||
242
src/app/projects/[projectId]/import/components/ImportMapping.tsx
Normal file
242
src/app/projects/[projectId]/import/components/ImportMapping.tsx
Normal file
|
|
@ -0,0 +1,242 @@
|
|||
"use client";
|
||||
|
||||
import { useState } from "react";
|
||||
import {
|
||||
ArrowLeftIcon,
|
||||
DocumentTextIcon,
|
||||
ExclamationTriangleIcon,
|
||||
} from "@heroicons/react/24/outline";
|
||||
import { Button } from "@/app/shadcn_components/ui/button";
|
||||
import {
|
||||
autoMatchColumns,
|
||||
summariseColumnMapping,
|
||||
} from "@/lib/projects/import/columnMapping";
|
||||
import type { ImportMappingTargets } from "@/lib/projects/import/mappingTargets";
|
||||
import {
|
||||
buildImportSession,
|
||||
sessionMatchesFile,
|
||||
type ImportSession,
|
||||
} from "@/lib/projects/import/session";
|
||||
import {
|
||||
buildValueMappingModel,
|
||||
EMPTY_VALUE_MAPPING,
|
||||
} from "@/lib/projects/import/valueMapping";
|
||||
import { ColumnMappingTable } from "./ColumnMappingTable";
|
||||
import { ValueMappingTables } from "./ValueMappingTables";
|
||||
import type { ParsedImportResponse } from "./ImportUpload";
|
||||
|
||||
/** What the mapping step hands to validation (#417) when the user continues. */
|
||||
export interface MappingHandoff {
|
||||
session: ImportSession;
|
||||
/** Rows that will fail validation as mapped — a warning, never a blocker. */
|
||||
warningRows: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Work-order import, step 2 of 3 — column mapping and value mapping (#415).
|
||||
*
|
||||
* Wireframes: `08-map-columns-rebuilt.html` (value mapping locked) and
|
||||
* `09-map-columns-workstream-unlocked.html` (unlocked).
|
||||
*
|
||||
* Holds the two mapping layers and nothing else. Every figure on the screen —
|
||||
* statuses, counts, which values need review, whether Continue is enabled — is
|
||||
* **derived inline** from `(file, columnMapping, valueMapping, targets)` on each
|
||||
* render rather than stored, so the tables cannot disagree with the mapping the
|
||||
* user is looking at. That is also why re-pointing the workstream column
|
||||
* immediately re-groups the contractor and stage tables: they were never a
|
||||
* separate copy of the answer.
|
||||
*
|
||||
* The initial column mapping is auto-matched once, in the `useState`
|
||||
* initialiser — unless a session for this same file comes back in, which is
|
||||
* what returning from a later step restores. It is a *suggestion*, so it
|
||||
* belongs in state (the user edits it) rather than being recomputed each render.
|
||||
*
|
||||
* Every edit also reports the whole session upward. That is what "persist the
|
||||
* mapping alongside the import session" amounts to while the session is
|
||||
* client-held (ADR-0020): the parent owns the session, this component owns the
|
||||
* editing of it, and stepping away and back does not lose the work.
|
||||
*/
|
||||
export function ImportMapping({
|
||||
file,
|
||||
targets,
|
||||
initialSession,
|
||||
onSessionChange,
|
||||
onBack,
|
||||
onContinue,
|
||||
}: {
|
||||
file: ParsedImportResponse;
|
||||
targets: ImportMappingTargets;
|
||||
/** A previously-edited session to resume; ignored unless it fits this file. */
|
||||
initialSession?: ImportSession | null;
|
||||
/** Called with the full session after every edit, so the parent can keep it. */
|
||||
onSessionChange?: (session: ImportSession) => void;
|
||||
onBack: () => void;
|
||||
onContinue: (handoff: MappingHandoff) => void;
|
||||
}) {
|
||||
// A mapping is a set of column indices, so a session from a *different* file
|
||||
// would map the wrong columns; `sessionMatchesFile` is what refuses it.
|
||||
const resumed =
|
||||
initialSession && sessionMatchesFile(initialSession, file)
|
||||
? initialSession
|
||||
: null;
|
||||
|
||||
const [columnMapping, setColumnMapping] = useState(
|
||||
() => resumed?.columnMapping ?? autoMatchColumns(file.headers),
|
||||
);
|
||||
const [valueMapping, setValueMapping] = useState(
|
||||
() => resumed?.valueMapping ?? EMPTY_VALUE_MAPPING,
|
||||
);
|
||||
|
||||
function updateColumns(next: typeof columnMapping) {
|
||||
setColumnMapping(next);
|
||||
onSessionChange?.(buildImportSession(file, next, valueMapping));
|
||||
}
|
||||
|
||||
function updateValues(next: typeof valueMapping) {
|
||||
setValueMapping(next);
|
||||
onSessionChange?.(buildImportSession(file, columnMapping, next));
|
||||
}
|
||||
|
||||
const summary = summariseColumnMapping(file.headers, columnMapping);
|
||||
const valueModel = buildValueMappingModel(
|
||||
file.rows,
|
||||
columnMapping,
|
||||
valueMapping,
|
||||
targets,
|
||||
);
|
||||
|
||||
const valuesNeedingReview =
|
||||
valueModel.needsReview.workstreams +
|
||||
valueModel.needsReview.contractors +
|
||||
valueModel.needsReview.stages;
|
||||
|
||||
return (
|
||||
<div data-testid="import-mapping">
|
||||
<div className="rounded-xl border border-gray-200 bg-white px-4 py-3 flex flex-wrap items-center justify-between gap-4">
|
||||
<div className="flex items-center min-w-0">
|
||||
<DocumentTextIcon className="h-5 w-5 text-gray-400 mr-2 shrink-0" />
|
||||
<div className="min-w-0">
|
||||
<p className="text-sm font-semibold text-gray-800 truncate">
|
||||
{file.filename}
|
||||
</p>
|
||||
<p className="text-xs text-gray-500">
|
||||
{file.rowCount.toLocaleString("en-GB")} row
|
||||
{file.rowCount === 1 ? "" : "s"} · {file.headers.length} column
|
||||
{file.headers.length === 1 ? "" : "s"}
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
<div className="flex items-center gap-6">
|
||||
<Figure label="Mapped" value={summary.matched} />
|
||||
<Figure
|
||||
label="Needs review"
|
||||
value={summary.needsReview + valuesNeedingReview}
|
||||
tone={
|
||||
summary.needsReview + valuesNeedingReview > 0 ? "warning" : "plain"
|
||||
}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="mt-4">
|
||||
<ColumnMappingTable
|
||||
headers={file.headers}
|
||||
preview={file.preview}
|
||||
mapping={columnMapping}
|
||||
onChange={updateColumns}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<ValueMappingTables
|
||||
model={valueModel}
|
||||
targets={targets}
|
||||
valueMapping={valueMapping}
|
||||
onChange={updateValues}
|
||||
// Locked until every required column is mapped — until then we do not
|
||||
// reliably know which column holds the workstream, and every value
|
||||
// table hangs off that one answer.
|
||||
locked={!summary.complete}
|
||||
/>
|
||||
|
||||
{summary.complete && valueModel.warningRows > 0 && (
|
||||
<div
|
||||
role="status"
|
||||
data-testid="value-mapping-warning"
|
||||
className="mt-4 flex items-start rounded-lg border border-amber-200 bg-amber-50 p-4 text-sm text-amber-900"
|
||||
>
|
||||
<ExclamationTriangleIcon className="h-5 w-5 mr-2.5 shrink-0 text-amber-500" />
|
||||
<div>
|
||||
<p className="font-semibold">
|
||||
{valueModel.warningRows.toLocaleString("en-GB")} of{" "}
|
||||
{file.rowCount.toLocaleString("en-GB")} rows will fail validation
|
||||
as mapped
|
||||
</p>
|
||||
<p className="mt-0.5 text-amber-800">
|
||||
Rows with an unmapped workstream, a missing or unmapped contractor,
|
||||
or a stage that matches nothing cannot be imported. You can
|
||||
continue and deal with them at validation, or map the values above
|
||||
first.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className="mt-6 flex items-center justify-between">
|
||||
<Button variant="outline" onClick={onBack} data-testid="mapping-back">
|
||||
<ArrowLeftIcon className="h-4 w-4 mr-1.5" />
|
||||
Back to upload
|
||||
</Button>
|
||||
|
||||
<div className="flex items-center gap-3">
|
||||
{!summary.complete && (
|
||||
<p className="text-xs text-red-700 max-w-xs text-right">
|
||||
Map{" "}
|
||||
{summary.missingRequirements.map((r) => r.label).join(", ")} to
|
||||
continue.
|
||||
</p>
|
||||
)}
|
||||
<Button
|
||||
data-testid="mapping-continue"
|
||||
disabled={!summary.complete}
|
||||
onClick={() =>
|
||||
onContinue({
|
||||
session: buildImportSession(file, columnMapping, valueMapping),
|
||||
warningRows: valueModel.warningRows,
|
||||
})
|
||||
}
|
||||
>
|
||||
Continue to validation
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/** One figure in the file summary card. */
|
||||
function Figure({
|
||||
label,
|
||||
value,
|
||||
tone = "plain",
|
||||
}: {
|
||||
label: string;
|
||||
value: number;
|
||||
tone?: "plain" | "warning";
|
||||
}) {
|
||||
return (
|
||||
<div className="text-right">
|
||||
<p className="text-[11px] font-semibold uppercase tracking-wider text-gray-400">
|
||||
{label}
|
||||
</p>
|
||||
<p
|
||||
data-testid={`mapping-figure-${label.toLowerCase().replace(/\s+/g, "-")}`}
|
||||
className={[
|
||||
"text-xl font-bold tabular-nums",
|
||||
tone === "warning" ? "text-red-600" : "text-gray-900",
|
||||
].join(" ")}
|
||||
>
|
||||
{value.toLocaleString("en-GB")}
|
||||
</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
@ -5,6 +5,7 @@ import { useMutation } from "@tanstack/react-query";
|
|||
import {
|
||||
ArrowDownTrayIcon,
|
||||
ArrowPathIcon,
|
||||
CheckCircleIcon,
|
||||
CloudArrowUpIcon,
|
||||
DocumentTextIcon,
|
||||
ExclamationTriangleIcon,
|
||||
|
|
@ -15,6 +16,9 @@ import {
|
|||
hasAcceptedImportExtension,
|
||||
MAX_IMPORT_FILE_BYTES,
|
||||
} from "@/lib/projects/import/parse";
|
||||
import type { ImportMappingTargets } from "@/lib/projects/import/mappingTargets";
|
||||
import type { ImportSession } from "@/lib/projects/import/session";
|
||||
import { ImportMapping, type MappingHandoff } from "./ImportMapping";
|
||||
|
||||
/** The parse route's response — the whole file, not just the preview. */
|
||||
export interface ParsedImportResponse {
|
||||
|
|
@ -27,21 +31,44 @@ export interface ParsedImportResponse {
|
|||
|
||||
const MAX_MB = MAX_IMPORT_FILE_BYTES / (1024 * 1024);
|
||||
|
||||
/** Which of the wizard's steps this component is showing. */
|
||||
type ImportStep = "upload" | "mapping" | "mapped";
|
||||
|
||||
/**
|
||||
* Work-order import drop zone (issue #414).
|
||||
* Work-order import drop zone (issue #414) and the step-2 mapping screen it
|
||||
* hands off to (issue #415).
|
||||
*
|
||||
* Uploads the file to the parse route and renders the returned headers and
|
||||
* preview. The full `rows` array stays here in `parse.data` — the mapping
|
||||
* (#415), validation (#416) and commit (#417) steps read it from this
|
||||
* component's state rather than re-fetching or re-parsing, which is why the
|
||||
* route hands back every row and not only the preview.
|
||||
*
|
||||
* The step is client state rather than a route because the parsed file lives
|
||||
* here: navigating to a `/import/mapping` URL would drop the rows on the floor
|
||||
* and force a re-upload on every Back. Going back to the upload step therefore
|
||||
* keeps the parsed file, and only choosing a different file discards it.
|
||||
*/
|
||||
export function ImportUpload({ projectId }: { projectId: string }) {
|
||||
export function ImportUpload({
|
||||
projectId,
|
||||
targets,
|
||||
}: {
|
||||
projectId: string;
|
||||
/** The project's workstreams, stages and contractors — the value-mapping targets. */
|
||||
targets: ImportMappingTargets;
|
||||
}) {
|
||||
const fileInputRef = useRef<HTMLInputElement>(null);
|
||||
const [isDragging, setIsDragging] = useState(false);
|
||||
const [fileName, setFileName] = useState<string | null>(null);
|
||||
// Size/extension failures caught before the upload; the route re-checks both.
|
||||
const [clientError, setClientError] = useState<string | null>(null);
|
||||
const [step, setStep] = useState<ImportStep>("upload");
|
||||
// The import session: the file's mapping, held here so stepping back to the
|
||||
// drop zone and forward again does not throw the user's work away. Nothing is
|
||||
// written server-side until commit (#417) — see ADR-0020.
|
||||
const [session, setSession] = useState<ImportSession | null>(null);
|
||||
// Set when the user finishes mapping — the artefact validation (#417) takes.
|
||||
const [handoff, setHandoff] = useState<MappingHandoff | null>(null);
|
||||
|
||||
const parse = useMutation<ParsedImportResponse, Error, File>({
|
||||
mutationFn: async (file: File) => {
|
||||
|
|
@ -61,6 +88,11 @@ export function ImportUpload({ projectId }: { projectId: string }) {
|
|||
setClientError(null);
|
||||
parse.reset();
|
||||
setFileName(file.name);
|
||||
// A different file means a different set of columns, so any mapping made
|
||||
// against the old one is meaningless (see `sessionMatchesFile`).
|
||||
setStep("upload");
|
||||
setSession(null);
|
||||
setHandoff(null);
|
||||
|
||||
if (!hasAcceptedImportExtension(file.name)) {
|
||||
setClientError(
|
||||
|
|
@ -93,12 +125,44 @@ export function ImportUpload({ projectId }: { projectId: string }) {
|
|||
setFileName(null);
|
||||
setClientError(null);
|
||||
parse.reset();
|
||||
setStep("upload");
|
||||
setSession(null);
|
||||
setHandoff(null);
|
||||
if (fileInputRef.current) fileInputRef.current.value = "";
|
||||
}
|
||||
|
||||
const error = clientError ?? (parse.isError ? parse.error.message : null);
|
||||
const parsed = parse.data;
|
||||
|
||||
if (parsed && step === "mapping") {
|
||||
return (
|
||||
<ImportMapping
|
||||
// Keyed on the file so a fresh upload starts from its own auto-match
|
||||
// rather than inheriting the previous file's mapping state.
|
||||
key={`${parsed.filename}:${parsed.rowCount}`}
|
||||
file={parsed}
|
||||
targets={targets}
|
||||
initialSession={session}
|
||||
onSessionChange={setSession}
|
||||
onBack={() => setStep("upload")}
|
||||
onContinue={(next) => {
|
||||
setHandoff(next);
|
||||
setStep("mapped");
|
||||
}}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
if (parsed && step === "mapped" && handoff) {
|
||||
return (
|
||||
<MappingSaved
|
||||
handoff={handoff}
|
||||
onEdit={() => setStep("mapping")}
|
||||
onStartOver={reset}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div data-testid="import-upload">
|
||||
<label
|
||||
|
|
@ -241,8 +305,10 @@ export function ImportUpload({ projectId }: { projectId: string }) {
|
|||
)}
|
||||
|
||||
<div className="mt-6 flex justify-end">
|
||||
{/* Enabled by the column-mapping step (#415). */}
|
||||
<Button disabled title="Column mapping arrives in #415">
|
||||
<Button
|
||||
data-testid="import-continue-to-mapping"
|
||||
onClick={() => setStep("mapping")}
|
||||
>
|
||||
Continue to mapping
|
||||
</Button>
|
||||
</div>
|
||||
|
|
@ -251,3 +317,64 @@ export function ImportUpload({ projectId }: { projectId: string }) {
|
|||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* The end of step 2: the mapping is settled and held in the import session,
|
||||
* waiting for validation and commit (#417) to consume it.
|
||||
*
|
||||
* It states what was captured rather than claiming anything was saved to the
|
||||
* server, because nothing was — an import writes nothing until commit
|
||||
* (ADR-0020). Both ways back are offered: edit the mapping, or start again with
|
||||
* a different file.
|
||||
*/
|
||||
function MappingSaved({
|
||||
handoff,
|
||||
onEdit,
|
||||
onStartOver,
|
||||
}: {
|
||||
handoff: MappingHandoff;
|
||||
onEdit: () => void;
|
||||
onStartOver: () => void;
|
||||
}) {
|
||||
const { session, warningRows } = handoff;
|
||||
const mappedFields = Object.keys(session.columnMapping.fields).length;
|
||||
|
||||
return (
|
||||
<div
|
||||
data-testid="import-mapping-saved"
|
||||
className="rounded-2xl border border-gray-200 bg-white p-6"
|
||||
>
|
||||
<div className="flex items-start">
|
||||
<CheckCircleIcon className="h-6 w-6 mr-3 shrink-0 text-green-500" />
|
||||
<div className="min-w-0">
|
||||
<h2 className="text-lg font-semibold text-gray-900">
|
||||
Mapping complete
|
||||
</h2>
|
||||
<p className="mt-1 text-sm text-gray-600">
|
||||
{mappedFields} field{mappedFields === 1 ? "" : "s"} mapped across{" "}
|
||||
{session.rowCount.toLocaleString("en-GB")} row
|
||||
{session.rowCount === 1 ? "" : "s"} of {session.filename}. Nothing
|
||||
has been written yet — validation and commit arrive in #417.
|
||||
</p>
|
||||
|
||||
{warningRows > 0 && (
|
||||
<p className="mt-3 rounded-lg border border-amber-200 bg-amber-50 px-3 py-2 text-xs text-amber-900">
|
||||
{warningRows.toLocaleString("en-GB")} row
|
||||
{warningRows === 1 ? "" : "s"} still carry values that map to
|
||||
nothing and will fail validation.
|
||||
</p>
|
||||
)}
|
||||
|
||||
<div className="mt-5 flex gap-3">
|
||||
<Button variant="outline" onClick={onEdit} data-testid="mapping-edit">
|
||||
Edit mapping
|
||||
</Button>
|
||||
<Button variant="outline" onClick={onStartOver}>
|
||||
Choose a different file
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
|
|||
126
src/app/projects/[projectId]/import/components/MappingSelect.tsx
Normal file
126
src/app/projects/[projectId]/import/components/MappingSelect.tsx
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
"use client";
|
||||
|
||||
import {
|
||||
Select,
|
||||
SelectContent,
|
||||
SelectItem,
|
||||
SelectTrigger,
|
||||
SelectValue,
|
||||
} from "@/app/shadcn_components/ui/select";
|
||||
|
||||
/**
|
||||
* The one picker both mapping layers use (issue #415): "this file thing is that
|
||||
* Ara thing", with an explicit way to say *neither*.
|
||||
*
|
||||
* Radix reserves the empty string as a value, so "not mapped" travels as a
|
||||
* sentinel and is translated at this boundary — callers only ever see
|
||||
* `string | null`, which is what the mapping modules store.
|
||||
*/
|
||||
const NONE = "__none__";
|
||||
|
||||
export interface MappingOption {
|
||||
value: string;
|
||||
label: string;
|
||||
/** Muted second line — what a column is currently used for, a contractor's org, … */
|
||||
hint?: string;
|
||||
}
|
||||
|
||||
export function MappingSelect({
|
||||
value,
|
||||
options,
|
||||
onChange,
|
||||
noneLabel,
|
||||
placeholder,
|
||||
disabled = false,
|
||||
invalid = false,
|
||||
testId,
|
||||
ariaLabel,
|
||||
}: {
|
||||
/** The chosen option's value, or null when nothing is mapped. */
|
||||
value: string | null;
|
||||
options: MappingOption[];
|
||||
/** Called with null when the user picks the "none" entry. */
|
||||
onChange: (value: string | null) => void;
|
||||
/** What choosing nothing means here — "Don't import", "Leave unmapped", … */
|
||||
noneLabel: string;
|
||||
placeholder: string;
|
||||
disabled?: boolean;
|
||||
/** Draws the needs-review treatment the wireframe gives unmapped rows. */
|
||||
invalid?: boolean;
|
||||
testId?: string;
|
||||
ariaLabel: string;
|
||||
}) {
|
||||
return (
|
||||
<Select
|
||||
value={value ?? NONE}
|
||||
onValueChange={(next) => onChange(next === NONE ? null : next)}
|
||||
disabled={disabled}
|
||||
>
|
||||
<SelectTrigger
|
||||
aria-label={ariaLabel}
|
||||
data-testid={testId}
|
||||
className={[
|
||||
"h-9 text-sm",
|
||||
invalid ? "border-red-300 text-red-700" : "",
|
||||
].join(" ")}
|
||||
>
|
||||
<SelectValue placeholder={placeholder} />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
<SelectItem value={NONE}>
|
||||
<span className="text-gray-500">{noneLabel}</span>
|
||||
</SelectItem>
|
||||
{options.map((option) => (
|
||||
<SelectItem key={option.value} value={option.value}>
|
||||
<span className="font-medium">{option.label}</span>
|
||||
{option.hint && (
|
||||
<span className="ml-2 text-xs text-gray-500">{option.hint}</span>
|
||||
)}
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
);
|
||||
}
|
||||
|
||||
/** The wireframe's per-row status affordance, shared by both tables. */
|
||||
export function MappingStatusIcon({
|
||||
status,
|
||||
}: {
|
||||
status: "matched" | "needs_review" | "ignored";
|
||||
}) {
|
||||
if (status === "matched") {
|
||||
return (
|
||||
<span
|
||||
title="Matched"
|
||||
data-testid="mapping-status-matched"
|
||||
className="inline-flex h-5 w-5 items-center justify-center rounded-full bg-green-100 text-green-700 text-xs font-bold"
|
||||
aria-label="Matched"
|
||||
>
|
||||
✓
|
||||
</span>
|
||||
);
|
||||
}
|
||||
if (status === "ignored") {
|
||||
return (
|
||||
<span
|
||||
title="Not imported"
|
||||
data-testid="mapping-status-ignored"
|
||||
className="inline-flex h-5 w-5 items-center justify-center rounded-full bg-gray-100 text-gray-400 text-xs font-bold"
|
||||
aria-label="Not imported"
|
||||
>
|
||||
–
|
||||
</span>
|
||||
);
|
||||
}
|
||||
return (
|
||||
<span
|
||||
title="Needs review"
|
||||
data-testid="mapping-status-needs-review"
|
||||
className="inline-flex h-5 w-5 items-center justify-center rounded-full bg-red-100 text-red-700 text-xs font-bold"
|
||||
aria-label="Needs review"
|
||||
>
|
||||
?
|
||||
</span>
|
||||
);
|
||||
}
|
||||
|
|
@ -0,0 +1,390 @@
|
|||
"use client";
|
||||
|
||||
import { LockClosedIcon } from "@heroicons/react/24/outline";
|
||||
import type { ImportMappingTargets } from "@/lib/projects/import/mappingTargets";
|
||||
import {
|
||||
setContractorValue,
|
||||
setStageValue,
|
||||
setWorkstreamValue,
|
||||
type ImportValueMapping,
|
||||
type ScopedValueEntry,
|
||||
type ValueMappingModel,
|
||||
} from "@/lib/projects/import/valueMapping";
|
||||
import { MappingSelect, MappingStatusIcon } from "./MappingSelect";
|
||||
|
||||
/**
|
||||
* Layer 2 of the mapping step (issue #415): the file's *values* against the
|
||||
* project's own workstreams, contractors and stages.
|
||||
*
|
||||
* Three tables rather than one, because they answer different questions at
|
||||
* different scopes — and the two lower ones are scoped **by** the first:
|
||||
* contractors and stages are offered per workstream, since an assignment
|
||||
* belongs to a workstream and a ladder belongs to a workstream (ADR-0019).
|
||||
* That is why they are grouped under workstream headings rather than listed
|
||||
* flat: the same contractor name under two workstreams is two questions.
|
||||
*
|
||||
* Locked until column mapping is complete, exactly as wireframe 08 shows —
|
||||
* there is nothing to list until we know which column holds the workstream.
|
||||
*/
|
||||
export function ValueMappingTables({
|
||||
model,
|
||||
targets,
|
||||
valueMapping,
|
||||
onChange,
|
||||
locked,
|
||||
}: {
|
||||
model: ValueMappingModel;
|
||||
targets: ImportMappingTargets;
|
||||
valueMapping: ImportValueMapping;
|
||||
onChange: (next: ImportValueMapping) => void;
|
||||
locked: boolean;
|
||||
}) {
|
||||
const contractorOptions = (entry: ScopedValueEntry) =>
|
||||
(targets.workstreams.find((w) => w.id === entry.workstreamId)?.contractors ?? [])
|
||||
.map((contractor) => ({ value: contractor.id, label: contractor.name }));
|
||||
|
||||
const stageOptions = (entry: ScopedValueEntry) =>
|
||||
(targets.workstreams.find((w) => w.id === entry.workstreamId)?.stages ?? []).map(
|
||||
(stage, index) => ({
|
||||
value: stage.id,
|
||||
label: stage.name,
|
||||
hint: index === 0 ? "first stage" : undefined,
|
||||
}),
|
||||
);
|
||||
|
||||
return (
|
||||
<section data-testid="value-mapping" className="relative">
|
||||
{locked && (
|
||||
<div
|
||||
data-testid="value-mapping-locked"
|
||||
className="absolute inset-0 z-20 flex items-start justify-center rounded-xl bg-white/80 backdrop-blur-[2px] pt-16"
|
||||
>
|
||||
<div className="rounded-xl border border-gray-200 bg-white px-6 py-5 text-center shadow-sm">
|
||||
<LockClosedIcon className="mx-auto h-6 w-6 text-gray-400 mb-2" />
|
||||
<p className="text-sm font-semibold text-gray-800">
|
||||
Complete column mapping to map cell values
|
||||
</p>
|
||||
<p className="mt-1 text-xs text-gray-500 max-w-xs">
|
||||
We can only list the values in your file once we know which column
|
||||
each field comes from.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className={locked ? "pointer-events-none select-none opacity-40" : ""}>
|
||||
<ValuePanel
|
||||
title="Workstream values"
|
||||
subtitle="Each distinct value in your workstream column, matched to one of this project's workstreams."
|
||||
testId="workstream-value-mapping"
|
||||
valueHeader="File value"
|
||||
targetHeader="Ara workstream"
|
||||
empty={
|
||||
model.hasWorkstreamColumn
|
||||
? "No workstream values found in the file."
|
||||
: "Map the workstream column first."
|
||||
}
|
||||
notes={[
|
||||
model.blankRows.workstream > 0
|
||||
? `${rows(model.blankRows.workstream)} have no workstream — those rows will fail validation.`
|
||||
: null,
|
||||
]}
|
||||
>
|
||||
{model.workstreams.map((entry) => (
|
||||
<ValueRow
|
||||
key={entry.key}
|
||||
value={entry.value}
|
||||
rowCount={entry.rowCount}
|
||||
status={entry.status}
|
||||
matchedBy={entry.matchedBy}
|
||||
>
|
||||
<MappingSelect
|
||||
ariaLabel={`Ara workstream for the value ${entry.value}`}
|
||||
testId={`workstream-value-${entry.key}`}
|
||||
value={entry.targetId}
|
||||
invalid={entry.status === "needs_review"}
|
||||
placeholder="Select workstream…"
|
||||
noneLabel="Leave unmapped"
|
||||
disabled={locked}
|
||||
onChange={(next) =>
|
||||
onChange(setWorkstreamValue(valueMapping, entry.key, next))
|
||||
}
|
||||
options={targets.workstreams.map((workstream) => ({
|
||||
value: workstream.id,
|
||||
label: workstream.name,
|
||||
}))}
|
||||
/>
|
||||
</ValueRow>
|
||||
))}
|
||||
</ValuePanel>
|
||||
|
||||
<ScopedPanel
|
||||
title="Contractor values"
|
||||
subtitle="Contractors are offered per workstream — only the companies assigned to that workstream can receive its work."
|
||||
testId="contractor-value-mapping"
|
||||
targetHeader="Assigned contractor"
|
||||
entries={model.contractors}
|
||||
targets={targets}
|
||||
placeholder="Select contractor…"
|
||||
noneTargetsNote="No contractor is assigned to this workstream, so its rows cannot be imported."
|
||||
optionsFor={contractorOptions}
|
||||
onSelect={(entry, next) =>
|
||||
onChange(setContractorValue(valueMapping, entry.key, next))
|
||||
}
|
||||
locked={locked}
|
||||
empty={
|
||||
model.hasContractorColumn
|
||||
? "No contractor values to map yet."
|
||||
: "Map the contractor column first."
|
||||
}
|
||||
notes={[
|
||||
model.blankRows.contractor > 0
|
||||
? `${rows(model.blankRows.contractor)} have no contractor — those rows will fail validation.`
|
||||
: null,
|
||||
model.unscopedRows > 0
|
||||
? `${rows(model.unscopedRows)} are waiting on their workstream value — map it above and their contractors appear here.`
|
||||
: null,
|
||||
]}
|
||||
/>
|
||||
|
||||
<ScopedPanel
|
||||
title="Stage values"
|
||||
subtitle="Stages are offered per workstream, from that workstream's own ladder."
|
||||
testId="stage-value-mapping"
|
||||
targetHeader="Workstream stage"
|
||||
entries={model.stages}
|
||||
targets={targets}
|
||||
placeholder="Select stage…"
|
||||
noneTargetsNote="This workstream has no stages configured."
|
||||
optionsFor={stageOptions}
|
||||
onSelect={(entry, next) =>
|
||||
onChange(setStageValue(valueMapping, entry.key, next))
|
||||
}
|
||||
locked={locked}
|
||||
empty={
|
||||
model.hasStageColumn
|
||||
? "No stage values in the file — every row will start at its workstream's first stage."
|
||||
: "No stage column mapped — every row will start at its workstream's first stage."
|
||||
}
|
||||
notes={[
|
||||
model.blankRows.stage > 0
|
||||
? `${rows(model.blankRows.stage)} have no stage — they will start at their workstream's first stage.`
|
||||
: null,
|
||||
model.needsReview.stages > 0
|
||||
? "A stage that matches nothing is an error — leave the cell blank instead if the work has not started."
|
||||
: null,
|
||||
]}
|
||||
/>
|
||||
</div>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
|
||||
/** "1 row" / "1,248 rows", en-GB grouped. */
|
||||
function rows(count: number): string {
|
||||
return `${count.toLocaleString("en-GB")} row${count === 1 ? "" : "s"}`;
|
||||
}
|
||||
|
||||
/** The shared chrome of a value table: header, column headings, notes. */
|
||||
function ValuePanel({
|
||||
title,
|
||||
subtitle,
|
||||
testId,
|
||||
valueHeader = "File value",
|
||||
targetHeader,
|
||||
notes,
|
||||
empty,
|
||||
children,
|
||||
}: {
|
||||
title: string;
|
||||
subtitle: string;
|
||||
testId: string;
|
||||
valueHeader?: string;
|
||||
targetHeader: string;
|
||||
notes: Array<string | null>;
|
||||
empty: string;
|
||||
children: React.ReactNode;
|
||||
}) {
|
||||
const hasRows = Array.isArray(children) ? children.length > 0 : Boolean(children);
|
||||
const shown = notes.filter((note): note is string => note !== null);
|
||||
|
||||
return (
|
||||
<div
|
||||
data-testid={testId}
|
||||
className="mt-4 rounded-xl border border-gray-200 bg-white overflow-hidden"
|
||||
>
|
||||
<header className="border-b border-gray-200 bg-gray-50 px-4 py-3">
|
||||
<h2 className="text-sm font-semibold text-gray-800">{title}</h2>
|
||||
<p className="text-xs text-gray-500 mt-0.5">{subtitle}</p>
|
||||
</header>
|
||||
|
||||
{hasRows ? (
|
||||
<div className="overflow-x-auto">
|
||||
<table className="min-w-full text-sm">
|
||||
<thead>
|
||||
<tr className="text-left text-xs font-semibold uppercase tracking-wide text-gray-500">
|
||||
<th className="px-4 py-2 w-5/12">{valueHeader}</th>
|
||||
<th className="px-4 py-2 w-1/12 text-center">Status</th>
|
||||
<th className="px-4 py-2 w-6/12">{targetHeader}</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody className="divide-y divide-gray-100">{children}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
) : (
|
||||
<p className="px-4 py-6 text-sm text-gray-500">{empty}</p>
|
||||
)}
|
||||
|
||||
{shown.length > 0 && (
|
||||
<div className="border-t border-gray-100 bg-gray-50/60 px-4 py-2.5 space-y-1">
|
||||
{shown.map((note) => (
|
||||
<p key={note} className="text-xs text-gray-600">
|
||||
{note}
|
||||
</p>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/** One value row: the file's text, its row count, its status, its picker. */
|
||||
function ValueRow({
|
||||
value,
|
||||
rowCount,
|
||||
status,
|
||||
matchedBy,
|
||||
scope,
|
||||
children,
|
||||
}: {
|
||||
value: string;
|
||||
rowCount: number;
|
||||
status: "matched" | "needs_review";
|
||||
matchedBy: "user" | "auto" | null;
|
||||
/** The workstream a contractor/stage value sits under, when it has one. */
|
||||
scope?: string;
|
||||
children: React.ReactNode;
|
||||
}) {
|
||||
return (
|
||||
<tr
|
||||
data-testid="value-mapping-row"
|
||||
data-status={status}
|
||||
className={
|
||||
status === "needs_review" ? "bg-red-50/40 border-l-4 border-l-red-400" : ""
|
||||
}
|
||||
>
|
||||
<td className="px-4 py-2.5 align-top">
|
||||
<span className="inline-block rounded bg-gray-100 px-2 py-0.5 font-medium text-gray-800">
|
||||
{value}
|
||||
</span>
|
||||
<div className="text-xs text-gray-500 mt-1">
|
||||
{rows(rowCount)}
|
||||
{scope && <span className="ml-1.5 text-gray-400">· {scope}</span>}
|
||||
{matchedBy === "auto" && (
|
||||
<span className="ml-1.5 text-gray-400">· auto-matched</span>
|
||||
)}
|
||||
</div>
|
||||
</td>
|
||||
<td className="px-4 py-2.5 text-center align-top">
|
||||
<MappingStatusIcon status={status} />
|
||||
</td>
|
||||
<td className="px-4 py-2 align-top">{children}</td>
|
||||
</tr>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* A contractor or stage table: the same rows, grouped under the workstream they
|
||||
* are scoped to, so it is never ambiguous *which* "Acme" is being mapped.
|
||||
*/
|
||||
function ScopedPanel({
|
||||
title,
|
||||
subtitle,
|
||||
testId,
|
||||
targetHeader,
|
||||
entries,
|
||||
targets,
|
||||
optionsFor,
|
||||
onSelect,
|
||||
placeholder,
|
||||
noneTargetsNote,
|
||||
locked,
|
||||
notes,
|
||||
empty,
|
||||
}: {
|
||||
title: string;
|
||||
subtitle: string;
|
||||
testId: string;
|
||||
targetHeader: string;
|
||||
entries: ScopedValueEntry[];
|
||||
targets: ImportMappingTargets;
|
||||
optionsFor: (entry: ScopedValueEntry) => Array<{
|
||||
value: string;
|
||||
label: string;
|
||||
hint?: string;
|
||||
}>;
|
||||
onSelect: (entry: ScopedValueEntry, next: string | null) => void;
|
||||
placeholder: string;
|
||||
noneTargetsNote: string;
|
||||
locked: boolean;
|
||||
notes: Array<string | null>;
|
||||
empty: string;
|
||||
}) {
|
||||
// Grouped in the targets' own (alphabetical) order so the grouping does not
|
||||
// reshuffle as values are mapped; within a group the biggest value leads.
|
||||
const groups = targets.workstreams
|
||||
.map((workstream) => ({
|
||||
workstream,
|
||||
entries: entries.filter((entry) => entry.workstreamId === workstream.id),
|
||||
}))
|
||||
.filter((group) => group.entries.length > 0);
|
||||
|
||||
return (
|
||||
<ValuePanel
|
||||
title={title}
|
||||
subtitle={subtitle}
|
||||
testId={testId}
|
||||
targetHeader={targetHeader}
|
||||
notes={notes}
|
||||
empty={empty}
|
||||
>
|
||||
{groups.flatMap((group) => [
|
||||
<tr key={`group-${group.workstream.id}`} className="bg-gray-50">
|
||||
<td
|
||||
colSpan={3}
|
||||
className="px-4 py-1.5 text-xs font-semibold uppercase tracking-wide text-gray-500"
|
||||
>
|
||||
{group.workstream.name}
|
||||
{group.entries[0].noTargets && (
|
||||
<span className="ml-2 font-normal normal-case text-red-700">
|
||||
{noneTargetsNote}
|
||||
</span>
|
||||
)}
|
||||
</td>
|
||||
</tr>,
|
||||
...group.entries.map((entry) => (
|
||||
<ValueRow
|
||||
key={entry.key}
|
||||
value={entry.value}
|
||||
rowCount={entry.rowCount}
|
||||
status={entry.status}
|
||||
matchedBy={entry.matchedBy}
|
||||
scope={group.workstream.name}
|
||||
>
|
||||
<MappingSelect
|
||||
ariaLabel={`${targetHeader} for the value ${entry.value} in ${group.workstream.name}`}
|
||||
testId={`scoped-value-${entry.key}`}
|
||||
value={entry.targetId}
|
||||
invalid={entry.status === "needs_review"}
|
||||
placeholder={placeholder}
|
||||
noneLabel="Leave unmapped"
|
||||
disabled={locked || entry.noTargets}
|
||||
onChange={(next) => onSelect(entry, next)}
|
||||
options={optionsFor(entry)}
|
||||
/>
|
||||
</ValueRow>
|
||||
)),
|
||||
])}
|
||||
</ValuePanel>
|
||||
);
|
||||
}
|
||||
|
|
@ -2,6 +2,10 @@ import { notFound } from "next/navigation";
|
|||
import { requireProjectAccess } from "../../guards";
|
||||
import { canManageProject } from "@/lib/projects/authz";
|
||||
import { loadImportReadiness } from "@/lib/projects/import/readiness";
|
||||
import {
|
||||
EMPTY_MAPPING_TARGETS,
|
||||
loadMappingTargets,
|
||||
} from "@/lib/projects/import/mappingTargets";
|
||||
import { ImportUpload } from "./components/ImportUpload";
|
||||
import { ImportSetupIncomplete } from "./components/ImportSetupIncomplete";
|
||||
|
||||
|
|
@ -10,11 +14,14 @@ export const metadata = {
|
|||
};
|
||||
|
||||
/**
|
||||
* Work-order import, step 1 of 3 — upload and parse (issue #414).
|
||||
* Work-order import, steps 1 and 2 of 3 — upload and parse (issue #414), then
|
||||
* column mapping and value mapping (issue #415).
|
||||
*
|
||||
* Wireframe: `docs/wireframes/ara-projects/1-admin-import/07-import-upload.html`.
|
||||
* The mapping step (#415), validation and commit (#417) follow; this page ends
|
||||
* at a parsed preview and writes nothing.
|
||||
* Wireframes: `07-import-upload.html`, then `08-map-columns-rebuilt.html` and
|
||||
* `09-map-columns-workstream-unlocked.html`. Both steps live behind this one
|
||||
* route because the parsed file is held client-side (ADR-0020) — a separate
|
||||
* mapping URL would drop it. Validation and commit (#417) follow; this page
|
||||
* still writes nothing.
|
||||
*
|
||||
* Only internal and client-org managers may import, so this checks
|
||||
* `canManageProject` rather than settling for the layout's view-level guard —
|
||||
|
|
@ -52,14 +59,21 @@ export default async function ImportPage(props: {
|
|||
// zone at all. 404 rather than 403, so project existence stays private.
|
||||
if (!canManageProject(user, project)) notFound();
|
||||
|
||||
// Read-only: gather whether the project is set up enough to import against.
|
||||
// Read-only: gather whether the project is set up enough to import against,
|
||||
// and — when it is — what the file's values may be mapped to. The targets are
|
||||
// loaded here, on the server, rather than fetched by the mapping step: they
|
||||
// are the project's own configuration, they cannot change mid-import, and a
|
||||
// gated project has nothing to map to anyway.
|
||||
const readiness = await loadImportReadiness(project.id);
|
||||
const targets = readiness.ready
|
||||
? await loadMappingTargets(project.id)
|
||||
: EMPTY_MAPPING_TARGETS;
|
||||
|
||||
return (
|
||||
<div className="max-w-4xl mx-auto px-6 py-10">
|
||||
<div className="max-w-5xl mx-auto px-6 py-10">
|
||||
<div className="mb-8">
|
||||
<p className="text-xs font-semibold text-gray-400 uppercase tracking-widest mb-1">
|
||||
Work-order import · Step 1 of 3
|
||||
Work-order import · Steps 1–2 of 3
|
||||
</p>
|
||||
<h1
|
||||
className="text-3xl font-extrabold text-gray-900 tracking-tight"
|
||||
|
|
@ -76,7 +90,7 @@ export default async function ImportPage(props: {
|
|||
</div>
|
||||
|
||||
{readiness.ready ? (
|
||||
<ImportUpload projectId={projectId} />
|
||||
<ImportUpload projectId={projectId} targets={targets} />
|
||||
) : (
|
||||
<ImportSetupIncomplete
|
||||
readiness={readiness}
|
||||
|
|
|
|||
230
src/lib/projects/import/columnMapping.test.ts
Normal file
230
src/lib/projects/import/columnMapping.test.ts
Normal file
|
|
@ -0,0 +1,230 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
assignColumn,
|
||||
autoMatchColumns,
|
||||
cellFor,
|
||||
columnMappingRows,
|
||||
EMPTY_COLUMN_MAPPING,
|
||||
fieldForColumn,
|
||||
IMPORT_REQUIREMENTS,
|
||||
summariseColumnMapping,
|
||||
} from "./columnMapping";
|
||||
import { IMPORT_TEMPLATE_COLUMNS } from "./parse";
|
||||
|
||||
/** A realistic export from an asset-management system, not our own template. */
|
||||
const CLIENT_HEADERS = [
|
||||
"Asset Ref",
|
||||
"Works Order Number",
|
||||
"Work Type",
|
||||
"Contractor Name",
|
||||
"Forecast End",
|
||||
"Priority",
|
||||
];
|
||||
|
||||
describe("autoMatchColumns", () => {
|
||||
it("maps every column of our own template", () => {
|
||||
const mapping = autoMatchColumns([...IMPORT_TEMPLATE_COLUMNS]);
|
||||
|
||||
expect(mapping.fields).toEqual({
|
||||
landlordPropertyId: 0,
|
||||
workstream: 1,
|
||||
contractor: 2,
|
||||
stage: 3,
|
||||
workOrderReference: 4,
|
||||
forecastEndDate: 5,
|
||||
});
|
||||
});
|
||||
|
||||
it("maps a client's own headings by similarity", () => {
|
||||
const mapping = autoMatchColumns(CLIENT_HEADERS);
|
||||
|
||||
expect(mapping.fields.landlordPropertyId).toBe(0); // Asset Ref
|
||||
expect(mapping.fields.workOrderReference).toBe(1); // Works Order Number
|
||||
expect(mapping.fields.workstream).toBe(2); // Work Type
|
||||
expect(mapping.fields.contractor).toBe(3); // Contractor Name
|
||||
expect(mapping.fields.forecastEndDate).toBe(4); // Forecast End
|
||||
});
|
||||
|
||||
it("leaves a column it cannot place unmapped rather than guessing", () => {
|
||||
// v1 has no Priority field (#426 owns priority), so it must stay unmapped
|
||||
// and show as needs review — not be forced into the nearest field.
|
||||
const mapping = autoMatchColumns(CLIENT_HEADERS);
|
||||
expect(fieldForColumn(mapping, 5)).toBeNull();
|
||||
expect(Object.values(mapping.fields)).not.toContain(5);
|
||||
});
|
||||
|
||||
it("does not read a Status column as the Stage field", () => {
|
||||
// A contractor's own status vocabulary is not this project's stage ladder,
|
||||
// so the user has to say so (CONTEXT.md keeps "status" away from Stage).
|
||||
const mapping = autoMatchColumns(["Property ID", "Workstream", "Status"]);
|
||||
expect(mapping.fields.stage).toBeUndefined();
|
||||
});
|
||||
|
||||
it("gives the identifier fields a column each when the file carries both", () => {
|
||||
const mapping = autoMatchColumns(["Property Reference", "UPRN"]);
|
||||
expect(mapping.fields.landlordPropertyId).toBe(0);
|
||||
expect(mapping.fields.uprn).toBe(1);
|
||||
});
|
||||
|
||||
it("maps nothing, and blames nothing, for headings it has never seen", () => {
|
||||
const mapping = autoMatchColumns(["Col1", "Col2", "Col3"]);
|
||||
expect(mapping.fields).toEqual({});
|
||||
expect(mapping.ignoredColumns).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("assignColumn", () => {
|
||||
it("returns a new mapping and leaves the original untouched", () => {
|
||||
const before = autoMatchColumns([...IMPORT_TEMPLATE_COLUMNS]);
|
||||
const after = assignColumn(before, 3, null);
|
||||
|
||||
expect(before.fields.stage).toBe(3);
|
||||
expect(after.fields.stage).toBeUndefined();
|
||||
});
|
||||
|
||||
it("moves a field off the column that held it, so two columns never share one", () => {
|
||||
const mapping = assignColumn(
|
||||
autoMatchColumns([...IMPORT_TEMPLATE_COLUMNS]),
|
||||
5,
|
||||
"workstream",
|
||||
);
|
||||
|
||||
expect(mapping.fields.workstream).toBe(5);
|
||||
// Column 1 was the workstream; it must now supply nothing rather than
|
||||
// silently continuing to claim the field.
|
||||
expect(fieldForColumn(mapping, 1)).toBeNull();
|
||||
});
|
||||
|
||||
it("clears the column's previous field when it takes a new one", () => {
|
||||
const mapping = assignColumn(
|
||||
autoMatchColumns([...IMPORT_TEMPLATE_COLUMNS]),
|
||||
1,
|
||||
"stage",
|
||||
);
|
||||
|
||||
expect(mapping.fields.stage).toBe(1);
|
||||
expect(mapping.fields.workstream).toBeUndefined();
|
||||
});
|
||||
|
||||
it("records an explicit 'don't import' distinctly from never having looked", () => {
|
||||
const mapping = assignColumn(EMPTY_COLUMN_MAPPING, 2, "ignore");
|
||||
|
||||
expect(mapping.ignoredColumns).toEqual([2]);
|
||||
expect(
|
||||
columnMappingRows(["a", "b", "c"], mapping).map((r) => r.status),
|
||||
).toEqual(["needs_review", "needs_review", "ignored"]);
|
||||
});
|
||||
|
||||
it("un-ignores a column when it is given a field", () => {
|
||||
const ignored = assignColumn(EMPTY_COLUMN_MAPPING, 2, "ignore");
|
||||
const mapped = assignColumn(ignored, 2, "workstream");
|
||||
|
||||
expect(mapped.ignoredColumns).toEqual([]);
|
||||
expect(mapped.fields.workstream).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("columnMappingRows", () => {
|
||||
it("returns one row per file column, in file order, with a sample cell", () => {
|
||||
const rows = columnMappingRows(
|
||||
["Asset Ref", "Work Type"],
|
||||
autoMatchColumns(["Asset Ref", "Work Type"]),
|
||||
[
|
||||
["", "Window Programme"],
|
||||
["100245", "Window Programme"],
|
||||
],
|
||||
);
|
||||
|
||||
expect(rows.map((r) => r.header)).toEqual(["Asset Ref", "Work Type"]);
|
||||
// The first *non-blank* cell, so a gap in row 1 does not blank the hint.
|
||||
expect(rows[0].sample).toBe("100245");
|
||||
expect(rows[0].field).toBe("landlordPropertyId");
|
||||
expect(rows[0].status).toBe("matched");
|
||||
});
|
||||
|
||||
it("reports an empty sample when the column is blank throughout", () => {
|
||||
const rows = columnMappingRows(["Stage"], EMPTY_COLUMN_MAPPING, [[""], [""]]);
|
||||
expect(rows[0].sample).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("summariseColumnMapping", () => {
|
||||
it("counts matched, ignored and needs-review columns", () => {
|
||||
const mapping = assignColumn(autoMatchColumns(CLIENT_HEADERS), 5, "ignore");
|
||||
const summary = summariseColumnMapping(CLIENT_HEADERS, mapping);
|
||||
|
||||
expect(summary.matched).toBe(5);
|
||||
expect(summary.ignored).toBe(1);
|
||||
expect(summary.needsReview).toBe(0);
|
||||
});
|
||||
|
||||
it("blocks while a required field has no column, and says which", () => {
|
||||
// Contractor is required (ADR-0019) and this file does not carry it.
|
||||
const headers = ["Property ID", "Work Type"];
|
||||
const summary = summariseColumnMapping(headers, autoMatchColumns(headers));
|
||||
|
||||
expect(summary.complete).toBe(false);
|
||||
expect(summary.missingRequirements.map((r) => r.id)).toEqual(["contractor"]);
|
||||
});
|
||||
|
||||
it("treats the property identifier as satisfied by either column", () => {
|
||||
const withUprn = summariseColumnMapping(
|
||||
["UPRN", "Workstream", "Contractor"],
|
||||
autoMatchColumns(["UPRN", "Workstream", "Contractor"]),
|
||||
);
|
||||
const withLandlordId = summariseColumnMapping(
|
||||
["Property Ref", "Workstream", "Contractor"],
|
||||
autoMatchColumns(["Property Ref", "Workstream", "Contractor"]),
|
||||
);
|
||||
|
||||
expect(withUprn.complete).toBe(true);
|
||||
expect(withLandlordId.complete).toBe(true);
|
||||
});
|
||||
|
||||
it("blocks when neither identifier column is mapped", () => {
|
||||
const headers = ["Workstream", "Contractor"];
|
||||
const summary = summariseColumnMapping(headers, autoMatchColumns(headers));
|
||||
|
||||
expect(summary.complete).toBe(false);
|
||||
expect(summary.missingRequirements.map((r) => r.id)).toEqual([
|
||||
"propertyIdentifier",
|
||||
]);
|
||||
});
|
||||
|
||||
it("never blocks on an optional field", () => {
|
||||
// No Stage, no reference, no forecast end — all optional, so this proceeds.
|
||||
const headers = ["Property ID", "Workstream", "Contractor"];
|
||||
const summary = summariseColumnMapping(headers, autoMatchColumns(headers));
|
||||
|
||||
expect(summary.complete).toBe(true);
|
||||
expect(summary.missingRequirements).toEqual([]);
|
||||
expect(summary.unmappedOptionalFields).toEqual([
|
||||
"stage",
|
||||
"workOrderReference",
|
||||
"forecastEndDate",
|
||||
]);
|
||||
});
|
||||
|
||||
it("names all three requirements when nothing is mapped", () => {
|
||||
const summary = summariseColumnMapping(["a", "b"], EMPTY_COLUMN_MAPPING);
|
||||
expect(summary.missingRequirements).toHaveLength(IMPORT_REQUIREMENTS.length);
|
||||
expect(summary.complete).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("cellFor", () => {
|
||||
const mapping = autoMatchColumns([...IMPORT_TEMPLATE_COLUMNS]);
|
||||
|
||||
it("reads the cell the field's column points at", () => {
|
||||
const row = ["PROP-1", "Windows", "Acme", "Ordered", "WO-1", "2026-09-30"];
|
||||
expect(cellFor(row, mapping, "contractor")).toBe("Acme");
|
||||
});
|
||||
|
||||
it("reads an unmapped field as blank rather than undefined", () => {
|
||||
expect(cellFor(["PROP-1"], EMPTY_COLUMN_MAPPING, "contractor")).toBe("");
|
||||
});
|
||||
|
||||
it("reads a short row as blank rather than undefined", () => {
|
||||
expect(cellFor(["PROP-1"], mapping, "contractor")).toBe("");
|
||||
});
|
||||
});
|
||||
396
src/lib/projects/import/columnMapping.ts
Normal file
396
src/lib/projects/import/columnMapping.ts
Normal file
|
|
@ -0,0 +1,396 @@
|
|||
/**
|
||||
* Work-order import — column mapping (issue #415, step 2 of 3, layer 1 of 2).
|
||||
*
|
||||
* Layer 1 answers "which file column is which Ara field?". Layer 2
|
||||
* (`./valueMapping`) answers "which of this column's values is which of the
|
||||
* project's workstreams / contractors / stages?".
|
||||
*
|
||||
* Pure and DB-free like `./parse`: it takes the headers the parser found and
|
||||
* returns data. Every function is total and immutable — the component holds one
|
||||
* `ImportColumnMapping` in state and swaps it for a new one on each edit, so
|
||||
* the whole screen (statuses, counts, the value-mapping tables) re-derives from
|
||||
* that single value with no effect and no cached copy to fall out of step.
|
||||
*
|
||||
* ## Required vs optional (ADR-0019)
|
||||
*
|
||||
* The parser is header-agnostic on purpose; requiredness is enforced *here*.
|
||||
* Property identifier, Workstream and Contractor are required; Stage, Work
|
||||
* order reference and Forecast end date are optional. Property identifier is
|
||||
* required as a **group**: a file may carry a landlord property id, a UPRN or
|
||||
* both, and mapping either one satisfies it (which one a row matches on is
|
||||
* validation's business at #417).
|
||||
*
|
||||
* Priority and cost columns are deliberately absent from the field list — v1
|
||||
* ignores them (#426 owns priority), so they land as unmapped columns the user
|
||||
* can dismiss.
|
||||
*/
|
||||
|
||||
import {
|
||||
AUTO_MATCH_THRESHOLD,
|
||||
bestSimilarity,
|
||||
greedyPairs,
|
||||
} from "./similarity";
|
||||
|
||||
/** The Ara fields an import file column can be mapped to. */
|
||||
export type ImportFieldKey =
|
||||
| "landlordPropertyId"
|
||||
| "uprn"
|
||||
| "workstream"
|
||||
| "contractor"
|
||||
| "stage"
|
||||
| "workOrderReference"
|
||||
| "forecastEndDate";
|
||||
|
||||
export interface ImportFieldDef {
|
||||
key: ImportFieldKey;
|
||||
/** What the field is called in the UI, and the first phrase auto-match scores against. */
|
||||
label: string;
|
||||
/** One line under the label in the field picker. */
|
||||
hint: string;
|
||||
/** Extra phrases auto-match scores against — the headings real files use. */
|
||||
aliases: readonly string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Every mappable field, in the order the template ships its columns.
|
||||
*
|
||||
* The alias lists are drawn from the headings asset-management exports actually
|
||||
* use ("Asset Ref", "Works Order Number", "Work Type"). Two deliberate
|
||||
* omissions:
|
||||
* - `stage` does **not** alias "status". A file's Status column is usually a
|
||||
* contractor's own workflow state rather than a stage on this project's
|
||||
* ladder, and CONTEXT.md keeps "status" away from Stage for that reason.
|
||||
* It shows as *needs review* so the user makes the call.
|
||||
* - nothing aliases "priority" or "cost" — v1 has no field for them.
|
||||
*/
|
||||
export const IMPORT_FIELDS: readonly ImportFieldDef[] = [
|
||||
{
|
||||
key: "landlordPropertyId",
|
||||
label: "Landlord property id",
|
||||
hint: "Your own reference for the property",
|
||||
aliases: [
|
||||
"property id",
|
||||
"property identifier",
|
||||
"property reference",
|
||||
"property ref",
|
||||
"landlord property id",
|
||||
"landlord reference",
|
||||
"asset id",
|
||||
"asset ref",
|
||||
"asset reference",
|
||||
"asset number",
|
||||
],
|
||||
},
|
||||
{
|
||||
key: "uprn",
|
||||
label: "UPRN",
|
||||
hint: "Unique Property Reference Number",
|
||||
aliases: ["uprn", "unique property reference number"],
|
||||
},
|
||||
{
|
||||
key: "workstream",
|
||||
label: "Workstream",
|
||||
hint: "The category of works — mapped to this project's workstreams",
|
||||
aliases: [
|
||||
"workstream",
|
||||
"work stream",
|
||||
"work type",
|
||||
"works type",
|
||||
"programme",
|
||||
"work programme",
|
||||
"works programme",
|
||||
"job type",
|
||||
"trade",
|
||||
],
|
||||
},
|
||||
{
|
||||
key: "contractor",
|
||||
label: "Contractor",
|
||||
hint: "Who delivers the work — mapped to this workstream's contractors",
|
||||
aliases: [
|
||||
"contractor",
|
||||
"contractor name",
|
||||
"supplier",
|
||||
"supplier name",
|
||||
"installer",
|
||||
"delivery partner",
|
||||
"company",
|
||||
],
|
||||
},
|
||||
{
|
||||
key: "stage",
|
||||
label: "Stage",
|
||||
hint: "Optional — blank rows start at the workstream's first stage",
|
||||
aliases: ["stage", "stage name", "current stage", "work stage", "phase"],
|
||||
},
|
||||
{
|
||||
key: "workOrderReference",
|
||||
label: "Work order reference",
|
||||
hint: "Optional — your reference for the work order",
|
||||
aliases: [
|
||||
"work order reference",
|
||||
"work order ref",
|
||||
"work order number",
|
||||
"works order number",
|
||||
"wo reference",
|
||||
"wo ref",
|
||||
"wo number",
|
||||
"order number",
|
||||
"job number",
|
||||
"job reference",
|
||||
],
|
||||
},
|
||||
{
|
||||
key: "forecastEndDate",
|
||||
label: "Forecast end date",
|
||||
hint: "Optional — when the work is expected to finish",
|
||||
aliases: [
|
||||
"forecast end",
|
||||
"forecast end date",
|
||||
"forecast completion",
|
||||
"forecast completion date",
|
||||
"planned end date",
|
||||
"target completion date",
|
||||
"completion date",
|
||||
"due date",
|
||||
"end date",
|
||||
],
|
||||
},
|
||||
] as const;
|
||||
|
||||
export const IMPORT_FIELDS_BY_KEY: Record<ImportFieldKey, ImportFieldDef> =
|
||||
Object.fromEntries(IMPORT_FIELDS.map((f) => [f.key, f])) as Record<
|
||||
ImportFieldKey,
|
||||
ImportFieldDef
|
||||
>;
|
||||
|
||||
/**
|
||||
* A requirement is satisfied when **any** of its keys is mapped.
|
||||
*
|
||||
* Single-key requirements are the ordinary case; `propertyIdentifier` is the
|
||||
* reason the concept exists — landlord property id *or* UPRN (or both) will do.
|
||||
*/
|
||||
export interface ImportRequirement {
|
||||
id: string;
|
||||
label: string;
|
||||
keys: readonly ImportFieldKey[];
|
||||
/** Shown when the requirement is unmet, so the blocker explains itself. */
|
||||
reason: string;
|
||||
}
|
||||
|
||||
export const IMPORT_REQUIREMENTS: readonly ImportRequirement[] = [
|
||||
{
|
||||
id: "propertyIdentifier",
|
||||
label: "Property identifier",
|
||||
keys: ["landlordPropertyId", "uprn"],
|
||||
reason:
|
||||
"Map a landlord property id or a UPRN column (or both) so rows can be matched to properties.",
|
||||
},
|
||||
{
|
||||
id: "workstream",
|
||||
label: "Workstream",
|
||||
keys: ["workstream"],
|
||||
reason: "Every row must say which workstream the work belongs to.",
|
||||
},
|
||||
{
|
||||
id: "contractor",
|
||||
label: "Contractor",
|
||||
keys: ["contractor"],
|
||||
reason: "Every row must name the contractor delivering the work.",
|
||||
},
|
||||
] as const;
|
||||
|
||||
/** Field keys that are optional — everything not named by a requirement. */
|
||||
export const OPTIONAL_FIELD_KEYS: readonly ImportFieldKey[] = IMPORT_FIELDS.map(
|
||||
(f) => f.key,
|
||||
).filter((key) => !IMPORT_REQUIREMENTS.some((r) => r.keys.includes(key)));
|
||||
|
||||
/**
|
||||
* The user's declaration of which column is which field.
|
||||
*
|
||||
* `fields` is keyed by field rather than by column because that is the question
|
||||
* every downstream consumer asks ("which column holds the contractor?") and
|
||||
* because it makes the one-column-per-field invariant unrepresentable-otherwise
|
||||
* rather than merely enforced.
|
||||
*
|
||||
* `ignoredColumns` records columns the user has *decided* not to import, which
|
||||
* is a different state from a column they have not looked at yet — the first
|
||||
* shows as settled, the second as needs review.
|
||||
*/
|
||||
export interface ImportColumnMapping {
|
||||
fields: Partial<Record<ImportFieldKey, number>>;
|
||||
ignoredColumns: number[];
|
||||
}
|
||||
|
||||
export const EMPTY_COLUMN_MAPPING: ImportColumnMapping = {
|
||||
fields: {},
|
||||
ignoredColumns: [],
|
||||
};
|
||||
|
||||
/** Per-file-column status, as the wireframe's column-mapping table renders it. */
|
||||
export type ColumnMappingStatus = "matched" | "ignored" | "needs_review";
|
||||
|
||||
export interface ColumnMappingRow {
|
||||
columnIndex: number;
|
||||
header: string;
|
||||
/** The field this column supplies, or null when unmapped. */
|
||||
field: ImportFieldKey | null;
|
||||
status: ColumnMappingStatus;
|
||||
/** A non-blank sample cell for this column, for the "e.g. …" hint. Empty when none. */
|
||||
sample: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Suggest a mapping from the file's headings alone.
|
||||
*
|
||||
* Scores every (column, field) pair on heading similarity and takes the best
|
||||
* pairs first, so a file carrying both "Property Ref" and "UPRN" gives each its
|
||||
* own field instead of letting the first column win both. Columns that match
|
||||
* nothing are left unmapped — *needs review*, never a guess.
|
||||
*/
|
||||
export function autoMatchColumns(headers: string[]): ImportColumnMapping {
|
||||
const pairs = greedyPairs(
|
||||
headers,
|
||||
IMPORT_FIELDS,
|
||||
(header, field) => bestSimilarity(header, [field.label, ...field.aliases]),
|
||||
AUTO_MATCH_THRESHOLD,
|
||||
);
|
||||
|
||||
const fields: Partial<Record<ImportFieldKey, number>> = {};
|
||||
for (const pair of pairs) {
|
||||
fields[IMPORT_FIELDS[pair.targetIndex].key] = pair.candidateIndex;
|
||||
}
|
||||
return { fields, ignoredColumns: [] };
|
||||
}
|
||||
|
||||
/**
|
||||
* Point a column at a field, at nothing, or explicitly at "don't import".
|
||||
*
|
||||
* Returns a new mapping; the caller's copy is untouched. Assigning a field that
|
||||
* another column already supplies moves it, because two columns cannot both be
|
||||
* the contractor and silently dropping one would be worse than saying so.
|
||||
*/
|
||||
export function assignColumn(
|
||||
mapping: ImportColumnMapping,
|
||||
columnIndex: number,
|
||||
target: ImportFieldKey | "ignore" | null,
|
||||
): ImportColumnMapping {
|
||||
const fields = { ...mapping.fields };
|
||||
|
||||
// This column is being repurposed, so it stops supplying whatever it did.
|
||||
for (const [key, index] of Object.entries(fields)) {
|
||||
if (index === columnIndex) delete fields[key as ImportFieldKey];
|
||||
}
|
||||
|
||||
const ignoredColumns = mapping.ignoredColumns.filter((i) => i !== columnIndex);
|
||||
|
||||
if (target === "ignore") {
|
||||
ignoredColumns.push(columnIndex);
|
||||
ignoredColumns.sort((a, b) => a - b);
|
||||
} else if (target !== null) {
|
||||
// One column per field: whoever held it before gives it up.
|
||||
fields[target] = columnIndex;
|
||||
}
|
||||
|
||||
return { fields, ignoredColumns };
|
||||
}
|
||||
|
||||
/** The field a column supplies, or null. */
|
||||
export function fieldForColumn(
|
||||
mapping: ImportColumnMapping,
|
||||
columnIndex: number,
|
||||
): ImportFieldKey | null {
|
||||
for (const field of IMPORT_FIELDS) {
|
||||
if (mapping.fields[field.key] === columnIndex) return field.key;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* One row per file column, in file order — the left-hand table.
|
||||
*
|
||||
* `rows` supplies the "e.g. …" sample under each heading; pass the preview
|
||||
* rather than all 20,000, the first non-blank cell is all it looks for.
|
||||
*/
|
||||
export function columnMappingRows(
|
||||
headers: string[],
|
||||
mapping: ImportColumnMapping,
|
||||
rows: string[][] = [],
|
||||
): ColumnMappingRow[] {
|
||||
return headers.map((header, columnIndex) => {
|
||||
const field = fieldForColumn(mapping, columnIndex);
|
||||
const ignored = mapping.ignoredColumns.includes(columnIndex);
|
||||
return {
|
||||
columnIndex,
|
||||
header,
|
||||
field,
|
||||
status: field ? "matched" : ignored ? "ignored" : "needs_review",
|
||||
sample: firstNonBlank(rows, columnIndex),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/** The first non-blank cell in a column, for the sample hint. */
|
||||
function firstNonBlank(rows: string[][], columnIndex: number): string {
|
||||
for (const row of rows) {
|
||||
const cell = row[columnIndex];
|
||||
if (cell !== undefined && cell !== "") return cell;
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
export interface ColumnMappingSummary {
|
||||
/** Columns pointed at a field — the "Auto-matched" figure once nothing has been edited. */
|
||||
matched: number;
|
||||
/** Columns the user has explicitly dismissed. */
|
||||
ignored: number;
|
||||
/** Columns neither mapped nor dismissed — the "Needs review" figure. */
|
||||
needsReview: number;
|
||||
/** Requirements with no column behind them. Empty means the step may proceed. */
|
||||
missingRequirements: ImportRequirement[];
|
||||
/** Optional fields with no column — informational, never a blocker. */
|
||||
unmappedOptionalFields: ImportFieldKey[];
|
||||
/**
|
||||
* True when every required field has a column. This — and only this — is what
|
||||
* gates progression to validation; unmapped *values* are warnings whose rows
|
||||
* fail downstream (#417), not blockers here.
|
||||
*/
|
||||
complete: boolean;
|
||||
}
|
||||
|
||||
export function summariseColumnMapping(
|
||||
headers: string[],
|
||||
mapping: ImportColumnMapping,
|
||||
): ColumnMappingSummary {
|
||||
const rows = columnMappingRows(headers, mapping);
|
||||
const missingRequirements = IMPORT_REQUIREMENTS.filter(
|
||||
(requirement) =>
|
||||
!requirement.keys.some((key) => mapping.fields[key] !== undefined),
|
||||
);
|
||||
|
||||
return {
|
||||
matched: rows.filter((r) => r.status === "matched").length,
|
||||
ignored: rows.filter((r) => r.status === "ignored").length,
|
||||
needsReview: rows.filter((r) => r.status === "needs_review").length,
|
||||
missingRequirements,
|
||||
unmappedOptionalFields: OPTIONAL_FIELD_KEYS.filter(
|
||||
(key) => mapping.fields[key] === undefined,
|
||||
),
|
||||
complete: missingRequirements.length === 0,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* A row's cell for a field, or `""` when the field is unmapped or the row is
|
||||
* short. Shared by the value-mapping layer and (at #417) by validation, so
|
||||
* "unmapped reads as blank" is decided in exactly one place.
|
||||
*/
|
||||
export function cellFor(
|
||||
row: string[],
|
||||
mapping: ImportColumnMapping,
|
||||
field: ImportFieldKey,
|
||||
): string {
|
||||
const columnIndex = mapping.fields[field];
|
||||
if (columnIndex === undefined) return "";
|
||||
return row[columnIndex] ?? "";
|
||||
}
|
||||
164
src/lib/projects/import/mappingTargets.test.ts
Normal file
164
src/lib/projects/import/mappingTargets.test.ts
Normal file
|
|
@ -0,0 +1,164 @@
|
|||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
// The module imports `@/app/db/db` for its loader. Stub it so importing the
|
||||
// module opens no connection and the production database is never touched; the
|
||||
// pure assembly tests never reach it and the loader tests drive it explicitly.
|
||||
const mockSelect = vi.fn();
|
||||
vi.mock("@/app/db/db", () => ({ db: { select: () => mockSelect() } }));
|
||||
|
||||
import {
|
||||
assembleMappingTargets,
|
||||
loadMappingTargets,
|
||||
UNNAMED_CONTRACTOR,
|
||||
type ContractorTargetFacts,
|
||||
type StageTargetFacts,
|
||||
type WorkstreamTargetFacts,
|
||||
} from "./mappingTargets";
|
||||
|
||||
const WORKSTREAMS: WorkstreamTargetFacts[] = [
|
||||
{ projectWorkstreamId: 2n, workstreamId: 11n, workstreamName: "Windows" },
|
||||
{ projectWorkstreamId: 1n, workstreamId: 10n, workstreamName: "Doors" },
|
||||
];
|
||||
|
||||
const STAGES: StageTargetFacts[] = [
|
||||
{ id: 203n, projectWorkstreamId: 2n, name: "Completed", order: 3 },
|
||||
{ id: 201n, projectWorkstreamId: 2n, name: "Ordered", order: 1 },
|
||||
{ id: 202n, projectWorkstreamId: 2n, name: "In progress", order: 2 },
|
||||
];
|
||||
|
||||
const CONTRACTORS: ContractorTargetFacts[] = [
|
||||
{
|
||||
id: 403n,
|
||||
projectWorkstreamId: 2n,
|
||||
organisationId: "org-b",
|
||||
organisationName: "Sunrise Glazing",
|
||||
},
|
||||
{
|
||||
id: 402n,
|
||||
projectWorkstreamId: 2n,
|
||||
organisationId: "org-a",
|
||||
organisationName: "Acme Windows Ltd",
|
||||
},
|
||||
];
|
||||
|
||||
describe("assembleMappingTargets", () => {
|
||||
it("nests stages and contractors under their workstream", () => {
|
||||
const targets = assembleMappingTargets(WORKSTREAMS, STAGES, CONTRACTORS);
|
||||
const windows = targets.workstreams.find((w) => w.name === "Windows")!;
|
||||
|
||||
expect(windows.id).toBe("2");
|
||||
expect(windows.stages).toHaveLength(3);
|
||||
expect(windows.contractors).toHaveLength(2);
|
||||
expect(targets.workstreams.find((w) => w.name === "Doors")!.stages).toEqual([]);
|
||||
});
|
||||
|
||||
it("stringifies every bigint id, so the result can cross to a client component", () => {
|
||||
const targets = assembleMappingTargets(WORKSTREAMS, STAGES, CONTRACTORS);
|
||||
|
||||
expect(() => JSON.stringify(targets)).not.toThrow();
|
||||
const windows = targets.workstreams.find((w) => w.name === "Windows")!;
|
||||
expect(windows.workstreamId).toBe("11");
|
||||
expect(windows.stages[0].id).toBe("201");
|
||||
expect(windows.contractors[0].id).toBe("402");
|
||||
});
|
||||
|
||||
it("orders stages by their ladder order, not by name or id", () => {
|
||||
const windows = assembleMappingTargets(WORKSTREAMS, STAGES, [])
|
||||
.workstreams.find((w) => w.name === "Windows")!;
|
||||
|
||||
// stages[0] is load-bearing: a blank Stage cell falls back to it (ADR-0019).
|
||||
expect(windows.stages.map((s) => s.name)).toEqual([
|
||||
"Ordered",
|
||||
"In progress",
|
||||
"Completed",
|
||||
]);
|
||||
});
|
||||
|
||||
it("orders workstreams and contractors alphabetically", () => {
|
||||
const targets = assembleMappingTargets(WORKSTREAMS, STAGES, CONTRACTORS);
|
||||
|
||||
expect(targets.workstreams.map((w) => w.name)).toEqual(["Doors", "Windows"]);
|
||||
expect(
|
||||
targets.workstreams.find((w) => w.name === "Windows")!.contractors.map((c) => c.name),
|
||||
).toEqual(["Acme Windows Ltd", "Sunrise Glazing"]);
|
||||
});
|
||||
|
||||
it("labels an organisation with no name rather than showing a blank option", () => {
|
||||
const targets = assembleMappingTargets(
|
||||
WORKSTREAMS,
|
||||
[],
|
||||
[
|
||||
{
|
||||
id: 404n,
|
||||
projectWorkstreamId: 1n,
|
||||
organisationId: "org-x",
|
||||
organisationName: null,
|
||||
},
|
||||
],
|
||||
);
|
||||
|
||||
expect(
|
||||
targets.workstreams.find((w) => w.name === "Doors")!.contractors[0].name,
|
||||
).toBe(UNNAMED_CONTRACTOR);
|
||||
});
|
||||
|
||||
it("drops children belonging to no workstream on this project", () => {
|
||||
// Only reachable if a caller mixes projects — attaching them elsewhere
|
||||
// would put a contractor in front of work they are not assigned to.
|
||||
const targets = assembleMappingTargets(
|
||||
WORKSTREAMS,
|
||||
[{ id: 999n, projectWorkstreamId: 77n, name: "Elsewhere", order: 1 }],
|
||||
[
|
||||
{
|
||||
id: 998n,
|
||||
projectWorkstreamId: 77n,
|
||||
organisationId: "org-z",
|
||||
organisationName: "Other Project Ltd",
|
||||
},
|
||||
],
|
||||
);
|
||||
|
||||
expect(targets.workstreams.flatMap((w) => w.stages)).toEqual([]);
|
||||
expect(targets.workstreams.flatMap((w) => w.contractors)).toEqual([]);
|
||||
});
|
||||
|
||||
it("returns no workstreams for a project that has selected none", () => {
|
||||
expect(assembleMappingTargets([], [], []).workstreams).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("loadMappingTargets", () => {
|
||||
/** A Drizzle select chain that resolves to `rows` when awaited. */
|
||||
function chainResolving(rows: unknown[]) {
|
||||
const chain: Record<string, unknown> = {};
|
||||
for (const method of ["from", "innerJoin", "where", "orderBy"]) {
|
||||
chain[method] = () => chain;
|
||||
}
|
||||
chain.then = (resolve: (value: unknown) => unknown) => resolve(rows);
|
||||
return chain;
|
||||
}
|
||||
|
||||
it("assembles the three read-only queries into one target set", async () => {
|
||||
mockSelect
|
||||
.mockReturnValueOnce(chainResolving(WORKSTREAMS))
|
||||
.mockReturnValueOnce(chainResolving(STAGES))
|
||||
.mockReturnValueOnce(chainResolving(CONTRACTORS));
|
||||
|
||||
const targets = await loadMappingTargets(42n);
|
||||
|
||||
expect(mockSelect).toHaveBeenCalledTimes(3);
|
||||
expect(targets.workstreams.map((w) => w.name)).toEqual(["Doors", "Windows"]);
|
||||
expect(
|
||||
targets.workstreams.find((w) => w.name === "Windows")!.stages.map((s) => s.id),
|
||||
).toEqual(["201", "202", "203"]);
|
||||
});
|
||||
|
||||
it("returns an empty target set for a project with no workstreams", async () => {
|
||||
mockSelect
|
||||
.mockReturnValueOnce(chainResolving([]))
|
||||
.mockReturnValueOnce(chainResolving([]))
|
||||
.mockReturnValueOnce(chainResolving([]));
|
||||
|
||||
expect((await loadMappingTargets(7n)).workstreams).toEqual([]);
|
||||
});
|
||||
});
|
||||
223
src/lib/projects/import/mappingTargets.ts
Normal file
223
src/lib/projects/import/mappingTargets.ts
Normal file
|
|
@ -0,0 +1,223 @@
|
|||
/**
|
||||
* Work-order import — what the file's values may be mapped *to* (issue #415).
|
||||
*
|
||||
* The value-mapping layer offers the project's own configuration as the target
|
||||
* set: its selected workstreams (`project_workstream`), and per workstream its
|
||||
* stage ladder (`project_workstream_stage`) and its assigned contractors
|
||||
* (`project_workstream_contractor` → `organisation`). Nothing else is on offer
|
||||
* — import never creates a workstream, a stage or a contractor (ADR-0019).
|
||||
*
|
||||
* Split the way `./readiness` splits, for the same reason:
|
||||
* - `assembleMappingTargets` is **pure** — three flat fact lists in, one
|
||||
* nested, sorted, client-safe structure out. Unit-tested with fixtures and
|
||||
* no connection.
|
||||
* - `loadMappingTargets` is the read-only round trip that gathers them.
|
||||
*
|
||||
* ## Why the ids come back as strings
|
||||
*
|
||||
* Every id here is a `bigserial`, which Drizzle hands back as a `bigint`, and a
|
||||
* `bigint` cannot cross the server→client boundary (it is not JSON-serialisable
|
||||
* and React refuses to serialise it into a client component's props). They are
|
||||
* stringified once, here, so the mapping UI and the persisted mapping hold the
|
||||
* same textual ids and #417 parses them back exactly once at commit.
|
||||
*
|
||||
* ## Population
|
||||
*
|
||||
* These three tables are filled in by the setup wizard (#410/#411/#413), which
|
||||
* is in flight. Against a project whose setup has not run, this returns an empty
|
||||
* workstream list — which is precisely the state the import page's readiness
|
||||
* gate (ADR-0019) refuses to show the drop zone for, so the mapping step never
|
||||
* sees it in practice.
|
||||
*/
|
||||
|
||||
import { db } from "@/app/db/db";
|
||||
import { asc, eq } from "drizzle-orm";
|
||||
import {
|
||||
projectWorkstream,
|
||||
projectWorkstreamContractor,
|
||||
projectWorkstreamStage,
|
||||
workstream,
|
||||
} from "@/app/db/schema/projects/projects";
|
||||
import { organisation } from "@/app/db/schema/organisation";
|
||||
|
||||
/** One stage on a workstream's ladder. */
|
||||
export interface StageTarget {
|
||||
/** `project_workstream_stage.id`, as a string. */
|
||||
id: string;
|
||||
name: string;
|
||||
/** `project_workstream_stage.order` — ascending, the first is the ladder's start. */
|
||||
order: number;
|
||||
}
|
||||
|
||||
/** One contractor assigned to a workstream. */
|
||||
export interface ContractorTarget {
|
||||
/**
|
||||
* `project_workstream_contractor.id`, as a string — the *assignment*, not the
|
||||
* organisation. `work_order.project_workstream_contractor_id` points at the
|
||||
* assignment, so that is what a mapped value must resolve to.
|
||||
*/
|
||||
id: string;
|
||||
/** `organisation.id` (a uuid), carried for display and for #417's checks. */
|
||||
organisationId: string;
|
||||
name: string;
|
||||
}
|
||||
|
||||
/** One workstream the project selected, with everything a row may map into. */
|
||||
export interface WorkstreamTarget {
|
||||
/** `project_workstream.id`, as a string. */
|
||||
id: string;
|
||||
/** `workstream.id` (the reference-data row), as a string. */
|
||||
workstreamId: string;
|
||||
name: string;
|
||||
/** Ascending by `order`; `stages[0]` is the fallback for a blank Stage cell. */
|
||||
stages: StageTarget[];
|
||||
/** Alphabetical by name. Only these may receive this workstream's work. */
|
||||
contractors: ContractorTarget[];
|
||||
}
|
||||
|
||||
export interface ImportMappingTargets {
|
||||
/** Alphabetical by name. */
|
||||
workstreams: WorkstreamTarget[];
|
||||
}
|
||||
|
||||
export const EMPTY_MAPPING_TARGETS: ImportMappingTargets = { workstreams: [] };
|
||||
|
||||
/** Facts as the database yields them, before assembly. */
|
||||
export interface WorkstreamTargetFacts {
|
||||
projectWorkstreamId: bigint;
|
||||
workstreamId: bigint;
|
||||
workstreamName: string;
|
||||
}
|
||||
|
||||
export interface StageTargetFacts {
|
||||
id: bigint;
|
||||
projectWorkstreamId: bigint;
|
||||
name: string;
|
||||
order: number;
|
||||
}
|
||||
|
||||
export interface ContractorTargetFacts {
|
||||
id: bigint;
|
||||
projectWorkstreamId: bigint;
|
||||
organisationId: string;
|
||||
/** `organisation.name` is nullable in the schema. */
|
||||
organisationName: string | null;
|
||||
}
|
||||
|
||||
/** Shown when an assigned organisation has no name recorded. */
|
||||
export const UNNAMED_CONTRACTOR = "Unnamed organisation";
|
||||
|
||||
/**
|
||||
* Nest and sort the facts into the target structure, stringifying every id.
|
||||
*
|
||||
* Stages and contractors whose `projectWorkstreamId` names no workstream on
|
||||
* this project are dropped rather than silently attached elsewhere — that can
|
||||
* only happen if the caller mixes projects, and inventing a home for such a row
|
||||
* would put a contractor in front of work they are not assigned to.
|
||||
*/
|
||||
export function assembleMappingTargets(
|
||||
workstreams: WorkstreamTargetFacts[],
|
||||
stages: StageTargetFacts[],
|
||||
contractors: ContractorTargetFacts[],
|
||||
): ImportMappingTargets {
|
||||
const byId = new Map<string, WorkstreamTarget>();
|
||||
for (const w of workstreams) {
|
||||
byId.set(String(w.projectWorkstreamId), {
|
||||
id: String(w.projectWorkstreamId),
|
||||
workstreamId: String(w.workstreamId),
|
||||
name: w.workstreamName,
|
||||
stages: [],
|
||||
contractors: [],
|
||||
});
|
||||
}
|
||||
|
||||
for (const stage of stages) {
|
||||
byId.get(String(stage.projectWorkstreamId))?.stages.push({
|
||||
id: String(stage.id),
|
||||
name: stage.name,
|
||||
order: stage.order,
|
||||
});
|
||||
}
|
||||
|
||||
for (const contractor of contractors) {
|
||||
byId.get(String(contractor.projectWorkstreamId))?.contractors.push({
|
||||
id: String(contractor.id),
|
||||
organisationId: contractor.organisationId,
|
||||
name: contractor.organisationName ?? UNNAMED_CONTRACTOR,
|
||||
});
|
||||
}
|
||||
|
||||
const sorted = [...byId.values()].sort((a, b) =>
|
||||
a.name.localeCompare(b.name, "en-GB"),
|
||||
);
|
||||
for (const target of sorted) {
|
||||
// The ladder's order is meaning, not presentation: stages[0] is the stage a
|
||||
// blank Stage cell falls back to at commit (ADR-0019).
|
||||
target.stages.sort((a, b) => a.order - b.order || a.name.localeCompare(b.name, "en-GB"));
|
||||
target.contractors.sort((a, b) => a.name.localeCompare(b.name, "en-GB"));
|
||||
}
|
||||
|
||||
return { workstreams: sorted };
|
||||
}
|
||||
|
||||
/**
|
||||
* Gather one project's mapping targets. Read-only: three SELECTs, no writes.
|
||||
*
|
||||
* Three queries rather than one join: joining stages and contractors to
|
||||
* workstreams together yields a stages×contractors product per workstream that
|
||||
* would have to be de-duplicated in JS anyway, and the three are independent so
|
||||
* they run concurrently.
|
||||
*/
|
||||
export async function loadMappingTargets(
|
||||
projectId: bigint,
|
||||
): Promise<ImportMappingTargets> {
|
||||
const [workstreams, stages, contractors] = await Promise.all([
|
||||
db
|
||||
.select({
|
||||
projectWorkstreamId: projectWorkstream.id,
|
||||
workstreamId: projectWorkstream.workstreamId,
|
||||
workstreamName: workstream.name,
|
||||
})
|
||||
.from(projectWorkstream)
|
||||
.innerJoin(workstream, eq(projectWorkstream.workstreamId, workstream.id))
|
||||
.where(eq(projectWorkstream.projectId, projectId)),
|
||||
|
||||
db
|
||||
.select({
|
||||
id: projectWorkstreamStage.id,
|
||||
projectWorkstreamId: projectWorkstreamStage.projectWorkstreamId,
|
||||
name: projectWorkstreamStage.name,
|
||||
order: projectWorkstreamStage.order,
|
||||
})
|
||||
.from(projectWorkstreamStage)
|
||||
.innerJoin(
|
||||
projectWorkstream,
|
||||
eq(projectWorkstreamStage.projectWorkstreamId, projectWorkstream.id),
|
||||
)
|
||||
.where(eq(projectWorkstream.projectId, projectId))
|
||||
.orderBy(asc(projectWorkstreamStage.order)),
|
||||
|
||||
db
|
||||
.select({
|
||||
id: projectWorkstreamContractor.id,
|
||||
projectWorkstreamId: projectWorkstreamContractor.projectWorkstreamId,
|
||||
organisationId: projectWorkstreamContractor.organisationId,
|
||||
organisationName: organisation.name,
|
||||
})
|
||||
.from(projectWorkstreamContractor)
|
||||
.innerJoin(
|
||||
projectWorkstream,
|
||||
eq(
|
||||
projectWorkstreamContractor.projectWorkstreamId,
|
||||
projectWorkstream.id,
|
||||
),
|
||||
)
|
||||
.innerJoin(
|
||||
organisation,
|
||||
eq(projectWorkstreamContractor.organisationId, organisation.id),
|
||||
)
|
||||
.where(eq(projectWorkstream.projectId, projectId)),
|
||||
]);
|
||||
|
||||
return assembleMappingTargets(workstreams, stages, contractors);
|
||||
}
|
||||
72
src/lib/projects/import/session.test.ts
Normal file
72
src/lib/projects/import/session.test.ts
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import { autoMatchColumns } from "./columnMapping";
|
||||
import {
|
||||
EMPTY_VALUE_MAPPING,
|
||||
setWorkstreamValue,
|
||||
valueKey,
|
||||
} from "./valueMapping";
|
||||
import {
|
||||
buildImportSession,
|
||||
IMPORT_SESSION_VERSION,
|
||||
sessionMatchesFile,
|
||||
} from "./session";
|
||||
|
||||
const FILE = {
|
||||
filename: "stonewater_planned_works.xlsx",
|
||||
headers: ["Asset Ref", "Work Type", "Contractor Name"],
|
||||
rowCount: 1248,
|
||||
};
|
||||
|
||||
function session() {
|
||||
return buildImportSession(
|
||||
FILE,
|
||||
autoMatchColumns(FILE.headers),
|
||||
setWorkstreamValue(EMPTY_VALUE_MAPPING, valueKey("Window Programme"), "2"),
|
||||
);
|
||||
}
|
||||
|
||||
describe("buildImportSession", () => {
|
||||
it("carries the file's identity and both mapping layers", () => {
|
||||
const built = session();
|
||||
|
||||
expect(built.version).toBe(IMPORT_SESSION_VERSION);
|
||||
expect(built.filename).toBe(FILE.filename);
|
||||
expect(built.rowCount).toBe(1248);
|
||||
expect(built.columnMapping.fields.workstream).toBe(1);
|
||||
expect(built.valueMapping.workstreams["window programme"]).toBe("2");
|
||||
});
|
||||
|
||||
it("does not carry the rows — they stay where the parse step put them", () => {
|
||||
expect(session()).not.toHaveProperty("rows");
|
||||
});
|
||||
|
||||
it("survives a JSON round trip unchanged, so it can be POSTed at commit", () => {
|
||||
const built = session();
|
||||
expect(JSON.parse(JSON.stringify(built))).toEqual(built);
|
||||
});
|
||||
});
|
||||
|
||||
describe("sessionMatchesFile", () => {
|
||||
it("accepts the file it was built against", () => {
|
||||
expect(sessionMatchesFile(session(), FILE)).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects a file whose headings moved, since the mapping is column indices", () => {
|
||||
expect(
|
||||
sessionMatchesFile(session(), {
|
||||
headers: ["Work Type", "Asset Ref", "Contractor Name"],
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects a file with a different number of columns", () => {
|
||||
expect(
|
||||
sessionMatchesFile(session(), { headers: ["Asset Ref", "Work Type"] }),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects a session written by a different version of the shape", () => {
|
||||
const stale = { ...session(), version: 0 as typeof IMPORT_SESSION_VERSION };
|
||||
expect(sessionMatchesFile(stale, FILE)).toBe(false);
|
||||
});
|
||||
});
|
||||
88
src/lib/projects/import/session.ts
Normal file
88
src/lib/projects/import/session.ts
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
/**
|
||||
* Work-order import — the import session (issue #415).
|
||||
*
|
||||
* The session is everything the wizard knows about one import in progress: the
|
||||
* file it parsed, and the two mapping layers the user declared over it. It is
|
||||
* the artefact the ticket means by "persist the mapping alongside the import
|
||||
* session".
|
||||
*
|
||||
* ## Where it lives
|
||||
*
|
||||
* Client-side, in the import screen's React state — the same place #414 put the
|
||||
* parsed rows, and for the same reason: nothing about an import is written
|
||||
* until commit (#417). There is no `import_session` table to hang the mapping
|
||||
* off; adding one is a migration, and the mapping alone is not worth a table
|
||||
* that the file's rows would still not fit in. So "alongside the session" is
|
||||
* taken literally: `ImportSession` carries the mappings in the same object as
|
||||
* the file identity, and that object is what #417 POSTs at commit. See
|
||||
* ADR-0020 for the trade-off and what would change the answer.
|
||||
*
|
||||
* The shape is therefore deliberately **JSON-serialisable end to end** — plain
|
||||
* objects, string ids, no `bigint`, no `Date`, no class instances — so it can
|
||||
* be POSTed verbatim, and so a future server-side session can persist it as a
|
||||
* JSON column without reshaping — which is how
|
||||
* `bulk_address_uploads.column_mapping` already stores the older BulkUpload
|
||||
* journey's equivalent.
|
||||
*/
|
||||
|
||||
import type { ImportColumnMapping } from "./columnMapping";
|
||||
import type { ImportValueMapping } from "./valueMapping";
|
||||
|
||||
/**
|
||||
* Bumped whenever the persisted shape changes meaning, so a session written by
|
||||
* an older client can be recognised rather than silently misread.
|
||||
*/
|
||||
export const IMPORT_SESSION_VERSION = 1;
|
||||
|
||||
export interface ImportSession {
|
||||
version: typeof IMPORT_SESSION_VERSION;
|
||||
/** The uploaded file's name, as the user sees it in the header. */
|
||||
filename: string;
|
||||
/** The file's headings, in file order — what the column mapping's indices point into. */
|
||||
headers: string[];
|
||||
/** Data rows in the file, for the counts the summary shows. */
|
||||
rowCount: number;
|
||||
/** Which column is which field. */
|
||||
columnMapping: ImportColumnMapping;
|
||||
/** Which of those columns' values are which of the project's entities. */
|
||||
valueMapping: ImportValueMapping;
|
||||
}
|
||||
|
||||
/**
|
||||
* Assemble the session. Takes the parsed file's identity rather than its rows:
|
||||
* the rows stay where the parse step put them, and re-sending 20,000 of them
|
||||
* with every mapping keystroke is exactly what holding them client-side avoids.
|
||||
*/
|
||||
export function buildImportSession(
|
||||
file: { filename: string; headers: string[]; rowCount: number },
|
||||
columnMapping: ImportColumnMapping,
|
||||
valueMapping: ImportValueMapping,
|
||||
): ImportSession {
|
||||
return {
|
||||
version: IMPORT_SESSION_VERSION,
|
||||
filename: file.filename,
|
||||
headers: file.headers,
|
||||
rowCount: file.rowCount,
|
||||
columnMapping,
|
||||
valueMapping,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* True when this session was written against the file currently in hand.
|
||||
*
|
||||
* A mapping is a set of *column indices*, so it is meaningless against a
|
||||
* different file — restoring one over a new upload would silently map the wrong
|
||||
* columns. Headers are compared in order, which is the only thing the indices
|
||||
* depend on.
|
||||
*/
|
||||
export function sessionMatchesFile(
|
||||
session: ImportSession,
|
||||
file: { headers: string[] },
|
||||
): boolean {
|
||||
return (
|
||||
session.version === IMPORT_SESSION_VERSION &&
|
||||
session.headers.length === file.headers.length &&
|
||||
session.headers.every((header, i) => header === file.headers[i])
|
||||
);
|
||||
}
|
||||
146
src/lib/projects/import/similarity.test.ts
Normal file
146
src/lib/projects/import/similarity.test.ts
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
AUTO_MATCH_THRESHOLD,
|
||||
bestSimilarity,
|
||||
greedyPairs,
|
||||
normaliseForMatch,
|
||||
similarity,
|
||||
VALUE_MATCH_THRESHOLD,
|
||||
valueSimilarity,
|
||||
} from "./similarity";
|
||||
|
||||
describe("normaliseForMatch", () => {
|
||||
it("lowercases, turns punctuation into spaces and collapses whitespace", () => {
|
||||
expect(normaliseForMatch(" Work_Order / REFERENCE ")).toBe(
|
||||
"work order reference",
|
||||
);
|
||||
});
|
||||
|
||||
it("collapses a value that is only punctuation to the empty string", () => {
|
||||
expect(normaliseForMatch(" -- ")).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("similarity", () => {
|
||||
it("scores an exact match, ignoring case and punctuation, as 1", () => {
|
||||
expect(similarity("Work Type", "work_type")).toBe(1);
|
||||
});
|
||||
|
||||
it("scores a blank against anything as 0", () => {
|
||||
expect(similarity("", "Workstream")).toBe(0);
|
||||
expect(similarity("—", "Workstream")).toBe(0);
|
||||
});
|
||||
|
||||
it("ranks word containment above letter overlap", () => {
|
||||
// "Contractor" ⊂ "Contractor Name" is a much stronger signal than the
|
||||
// bigram overlap of "Stage"/"Storage", and must outrank it.
|
||||
expect(similarity("Contractor", "Contractor Name")).toBe(0.9);
|
||||
expect(similarity("Stage", "Storage")).toBeLessThan(0.9);
|
||||
});
|
||||
|
||||
it("scores near-misses between 0 and 1", () => {
|
||||
const score = similarity("Forecast End", "Forecast end date");
|
||||
expect(score).toBeGreaterThan(AUTO_MATCH_THRESHOLD);
|
||||
expect(score).toBeLessThan(1);
|
||||
});
|
||||
|
||||
it("keeps unrelated headings below the auto-match threshold", () => {
|
||||
expect(similarity("Postcode", "Contractor")).toBeLessThan(
|
||||
AUTO_MATCH_THRESHOLD,
|
||||
);
|
||||
expect(similarity("Priority", "Workstream")).toBeLessThan(
|
||||
AUTO_MATCH_THRESHOLD,
|
||||
);
|
||||
// The one that matters most: a Status column is not a Stage column.
|
||||
expect(similarity("Status", "Stage")).toBeLessThan(AUTO_MATCH_THRESHOLD);
|
||||
});
|
||||
});
|
||||
|
||||
describe("valueSimilarity", () => {
|
||||
it("matches a project's name buried in a longer file value", () => {
|
||||
// The cases the wireframe shows: the file's vocabulary is wordier than the
|
||||
// project's, and plain bigram overlap punishes it for the extra words.
|
||||
expect(valueSimilarity("Window Programme", "Windows")).toBeGreaterThan(
|
||||
VALUE_MATCH_THRESHOLD,
|
||||
);
|
||||
expect(valueSimilarity("Door Replacement", "Doors")).toBeGreaterThan(
|
||||
VALUE_MATCH_THRESHOLD,
|
||||
);
|
||||
// This one the heading rule already handles — "Decorations" is a whole word
|
||||
// of the file's value — so it keeps the word-containment score.
|
||||
expect(valueSimilarity("Cyclical Decorations", "Decorations")).toBe(0.9);
|
||||
});
|
||||
|
||||
it("refuses two contractor names that share only their filler words", () => {
|
||||
expect(
|
||||
valueSimilarity("Acme Windows Ltd", "Acme Doors Ltd"),
|
||||
).toBeLessThan(VALUE_MATCH_THRESHOLD);
|
||||
expect(
|
||||
valueSimilarity("Someone Else Ltd", "Acme Windows Ltd"),
|
||||
).toBeLessThan(VALUE_MATCH_THRESHOLD);
|
||||
expect(
|
||||
valueSimilarity("Sunrise Glazing", "Sunrise Roofing Ltd"),
|
||||
).toBeLessThan(VALUE_MATCH_THRESHOLD);
|
||||
});
|
||||
|
||||
it("is never stricter than the heading rule it builds on", () => {
|
||||
for (const [a, b] of [
|
||||
["Windows", "Windows"],
|
||||
["Contractor", "Contractor Name"],
|
||||
["Stage", "Storage"],
|
||||
]) {
|
||||
expect(valueSimilarity(a, b)).toBeGreaterThanOrEqual(similarity(a, b));
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("bestSimilarity", () => {
|
||||
it("takes the best score across a field's label and aliases", () => {
|
||||
expect(bestSimilarity("Asset Ref", ["Landlord property id", "asset ref"])).toBe(1);
|
||||
});
|
||||
|
||||
it("is 0 when there are no phrases to score against", () => {
|
||||
expect(bestSimilarity("Anything", [])).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("greedyPairs", () => {
|
||||
const score = (a: string, b: string) => similarity(a, b);
|
||||
|
||||
it("gives each target to its best candidate, one apiece", () => {
|
||||
const pairs = greedyPairs(
|
||||
["Property Ref", "UPRN"],
|
||||
["Property reference", "UPRN"],
|
||||
score,
|
||||
);
|
||||
|
||||
expect(pairs).toHaveLength(2);
|
||||
expect(pairs.map((p) => [p.candidateIndex, p.targetIndex])).toEqual(
|
||||
expect.arrayContaining([
|
||||
[0, 0],
|
||||
[1, 1],
|
||||
]),
|
||||
);
|
||||
});
|
||||
|
||||
it("does not let one candidate take two targets, or one target two candidates", () => {
|
||||
// Both headings look like the same target; only the better one may have it.
|
||||
const pairs = greedyPairs(
|
||||
["Workstream", "Workstream 2"],
|
||||
["Workstream"],
|
||||
score,
|
||||
);
|
||||
|
||||
expect(pairs).toHaveLength(1);
|
||||
expect(pairs[0].candidateIndex).toBe(0);
|
||||
});
|
||||
|
||||
it("drops pairs below the threshold rather than matching them weakly", () => {
|
||||
expect(greedyPairs(["Postcode"], ["Contractor"], score)).toEqual([]);
|
||||
});
|
||||
|
||||
it("breaks ties on candidate order, so a given file always maps the same way", () => {
|
||||
const pairs = greedyPairs(["Stage", "Stage"], ["Stage"], score);
|
||||
expect(pairs).toEqual([{ candidateIndex: 0, targetIndex: 0, score: 1 }]);
|
||||
});
|
||||
});
|
||||
226
src/lib/projects/import/similarity.ts
Normal file
226
src/lib/projects/import/similarity.ts
Normal file
|
|
@ -0,0 +1,226 @@
|
|||
/**
|
||||
* Header/value similarity — the auto-match primitive shared by both mapping
|
||||
* layers of the work-order import (issue #415).
|
||||
*
|
||||
* Column mapping matches a file *heading* to an Ara field; value mapping
|
||||
* matches a file *cell value* to one of the project's workstreams, contractors
|
||||
* or stages. Both are "the user typed something close to what we call it", so
|
||||
* both score the same way rather than each growing its own fuzzy matcher.
|
||||
*
|
||||
* Pure and dependency-free: no database, no React, no I/O.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Fold a heading or value to its comparable form: lowercase, punctuation and
|
||||
* underscores to spaces, runs of whitespace collapsed.
|
||||
*
|
||||
* Deliberately *not* stemming or dropping stop-words — "Work order reference"
|
||||
* and "Work order number" must stay distinguishable from each other, and a
|
||||
* stemmer would need a dictionary this module has no business carrying.
|
||||
*/
|
||||
export function normaliseForMatch(value: string): string {
|
||||
return value
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]+/g, " ")
|
||||
.trim()
|
||||
.replace(/\s+/g, " ");
|
||||
}
|
||||
|
||||
/** The normalised form's words, empty array for a blank input. */
|
||||
function tokens(normalised: string): string[] {
|
||||
return normalised === "" ? [] : normalised.split(" ");
|
||||
}
|
||||
|
||||
/** Character bigrams of a normalised string with its spaces removed. */
|
||||
function bigrams(normalised: string): string[] {
|
||||
const compact = normalised.replace(/ /g, "");
|
||||
const out: string[] = [];
|
||||
for (let i = 0; i < compact.length - 1; i++) out.push(compact.slice(i, i + 2));
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Sørensen–Dice coefficient over character bigrams, in `[0, 1]`.
|
||||
*
|
||||
* Chosen over Levenshtein because it is length-insensitive in the way headings
|
||||
* need: "WO Ref" vs "Work order reference" is a *containment* relationship that
|
||||
* an edit-distance ratio punishes for the length difference alone.
|
||||
*/
|
||||
function diceCoefficient(a: string, b: string): number {
|
||||
const left = bigrams(a);
|
||||
const right = bigrams(b);
|
||||
if (left.length === 0 || right.length === 0) return a === b ? 1 : 0;
|
||||
|
||||
// Multiset intersection: each bigram in `left` consumes one occurrence in
|
||||
// `right`, so a repeated bigram cannot be matched twice.
|
||||
const counts = new Map<string, number>();
|
||||
for (const gram of right) counts.set(gram, (counts.get(gram) ?? 0) + 1);
|
||||
|
||||
let shared = 0;
|
||||
for (const gram of left) {
|
||||
const remaining = counts.get(gram) ?? 0;
|
||||
if (remaining > 0) {
|
||||
shared++;
|
||||
counts.set(gram, remaining - 1);
|
||||
}
|
||||
}
|
||||
|
||||
return (2 * shared) / (left.length + right.length);
|
||||
}
|
||||
|
||||
/** True when every word of the shorter phrase appears in the longer one. */
|
||||
function isTokenSubset(a: string[], b: string[]): boolean {
|
||||
if (a.length === 0 || b.length === 0) return false;
|
||||
const [short, long] = a.length <= b.length ? [a, b] : [b, a];
|
||||
const pool = new Set(long);
|
||||
return short.every((token) => pool.has(token));
|
||||
}
|
||||
|
||||
/**
|
||||
* How alike two headings/values are, in `[0, 1]`.
|
||||
*
|
||||
* Three tiers, highest first:
|
||||
* 1. `1` — identical once normalised ("Work Type" vs "work type").
|
||||
* 2. `0.9` — one phrase's words are a subset of the other's ("Contractor" vs
|
||||
* "Contractor Name"). Ranked above any Dice score because word containment
|
||||
* is a much stronger signal than shared letters: "Stage" and "Storage"
|
||||
* score well on bigrams and mean nothing to each other.
|
||||
* 3. the Dice coefficient — everything else, including near-misses and
|
||||
* typos.
|
||||
*/
|
||||
export function similarity(a: string, b: string): number {
|
||||
const left = normaliseForMatch(a);
|
||||
const right = normaliseForMatch(b);
|
||||
if (left === "" || right === "") return 0;
|
||||
if (left === right) return 1;
|
||||
if (isTokenSubset(tokens(left), tokens(right))) return 0.9;
|
||||
return diceCoefficient(left, right);
|
||||
}
|
||||
|
||||
/**
|
||||
* How much of `from`'s text finds a counterpart in `to`, weighted by word
|
||||
* length so a long distinguishing word counts for more than a short filler one.
|
||||
*
|
||||
* Directional on purpose — see `valueSimilarity`.
|
||||
*/
|
||||
function tokenCoverage(from: string[], to: string[]): number {
|
||||
let matched = 0;
|
||||
let total = 0;
|
||||
for (const token of from) {
|
||||
total += token.length;
|
||||
let best = 0;
|
||||
for (const other of to) {
|
||||
const score = token === other ? 1 : diceCoefficient(token, other);
|
||||
if (score > best) best = score;
|
||||
}
|
||||
matched += token.length * best;
|
||||
}
|
||||
return total === 0 ? 0 : matched / total;
|
||||
}
|
||||
|
||||
/**
|
||||
* Similarity for *cell values*, which are messier than headings.
|
||||
*
|
||||
* A file's workstream column says "Window Programme" where the project calls it
|
||||
* "Windows", and "Door Replacement" where it says "Doors" — the project's name
|
||||
* is one word of a longer phrase, in a form `similarity` scores at 0.5 because
|
||||
* the extra words dilute the bigram overlap. So this takes the best *directional*
|
||||
* word coverage as well: "Windows" is almost entirely present in "Window
|
||||
* Programme" (0.91) even though the reverse is not true (0.36).
|
||||
*
|
||||
* Kept apart from `similarity` rather than folded into it, because the same
|
||||
* leniency is wrong for headings: "Forecast Start" would cover "Forecast end
|
||||
* date" well enough to be auto-mapped to the wrong field, and a heading mapped
|
||||
* to the wrong field mis-reads every row in the file.
|
||||
*/
|
||||
export function valueSimilarity(a: string, b: string): number {
|
||||
const base = similarity(a, b);
|
||||
if (base >= 0.9) return base;
|
||||
const left = tokens(normaliseForMatch(a));
|
||||
const right = tokens(normaliseForMatch(b));
|
||||
return Math.max(base, tokenCoverage(left, right), tokenCoverage(right, left));
|
||||
}
|
||||
|
||||
/**
|
||||
* The bar a *value* auto-match must clear — higher than the heading bar,
|
||||
* because the coverage rule above is more generous and because contractor names
|
||||
* share filler words ("Ltd", "Group", "Services") that must not be enough on
|
||||
* their own. "Acme Windows Ltd" vs "Acme Doors Ltd" scores 0.67 and is
|
||||
* correctly refused.
|
||||
*/
|
||||
export const VALUE_MATCH_THRESHOLD = 0.75;
|
||||
|
||||
/**
|
||||
* The best score of `candidate` against any of `phrases` — a field's label plus
|
||||
* its aliases, or a target's name.
|
||||
*/
|
||||
export function bestSimilarity(
|
||||
candidate: string,
|
||||
phrases: readonly string[],
|
||||
): number {
|
||||
let best = 0;
|
||||
for (const phrase of phrases) {
|
||||
const score = similarity(candidate, phrase);
|
||||
if (score > best) best = score;
|
||||
}
|
||||
return best;
|
||||
}
|
||||
|
||||
/**
|
||||
* The bar an auto-match must clear.
|
||||
*
|
||||
* 0.62 sits above the noise floor of unrelated four-to-six-letter headings
|
||||
* (which land around 0.3–0.5 on Dice) and below "Forecast End" vs "Forecast end
|
||||
* date" (0.83). A wrong auto-match is worse than none — an unmatched column
|
||||
* shows as *needs review* and asks the user, whereas a confidently wrong one
|
||||
* has to be *noticed* first.
|
||||
*/
|
||||
export const AUTO_MATCH_THRESHOLD = 0.62;
|
||||
|
||||
/**
|
||||
* Greedily pair candidates to targets, best score first, one target per
|
||||
* candidate and one candidate per target.
|
||||
*
|
||||
* Greedy rather than optimal (Hungarian) assignment: the score matrix is tiny,
|
||||
* the user can override any row, and "the best match wins first" is the rule a
|
||||
* person would predict when they look at the result. Ties break on candidate
|
||||
* order then target order, so the outcome is stable for a given file.
|
||||
*/
|
||||
export function greedyPairs<C, T>(
|
||||
candidates: readonly C[],
|
||||
targets: readonly T[],
|
||||
score: (candidate: C, target: T) => number,
|
||||
threshold: number = AUTO_MATCH_THRESHOLD,
|
||||
): Array<{ candidateIndex: number; targetIndex: number; score: number }> {
|
||||
const scored: Array<{
|
||||
candidateIndex: number;
|
||||
targetIndex: number;
|
||||
score: number;
|
||||
}> = [];
|
||||
|
||||
candidates.forEach((candidate, candidateIndex) => {
|
||||
targets.forEach((target, targetIndex) => {
|
||||
const value = score(candidate, target);
|
||||
if (value >= threshold) scored.push({ candidateIndex, targetIndex, score: value });
|
||||
});
|
||||
});
|
||||
|
||||
scored.sort(
|
||||
(a, b) =>
|
||||
b.score - a.score ||
|
||||
a.candidateIndex - b.candidateIndex ||
|
||||
a.targetIndex - b.targetIndex,
|
||||
);
|
||||
|
||||
const usedCandidates = new Set<number>();
|
||||
const usedTargets = new Set<number>();
|
||||
const pairs: typeof scored = [];
|
||||
for (const pair of scored) {
|
||||
if (usedCandidates.has(pair.candidateIndex)) continue;
|
||||
if (usedTargets.has(pair.targetIndex)) continue;
|
||||
usedCandidates.add(pair.candidateIndex);
|
||||
usedTargets.add(pair.targetIndex);
|
||||
pairs.push(pair);
|
||||
}
|
||||
return pairs;
|
||||
}
|
||||
395
src/lib/projects/import/valueMapping.test.ts
Normal file
395
src/lib/projects/import/valueMapping.test.ts
Normal file
|
|
@ -0,0 +1,395 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import { autoMatchColumns } from "./columnMapping";
|
||||
import type { ImportMappingTargets } from "./mappingTargets";
|
||||
import {
|
||||
buildValueMappingModel,
|
||||
EMPTY_VALUE_MAPPING,
|
||||
scopedValueKey,
|
||||
setContractorValue,
|
||||
setStageValue,
|
||||
setWorkstreamValue,
|
||||
valueKey,
|
||||
} from "./valueMapping";
|
||||
|
||||
const HEADERS = ["Property ID", "Work Type", "Contractor", "Stage"];
|
||||
const COLUMNS = autoMatchColumns(HEADERS);
|
||||
|
||||
/**
|
||||
* Windows has two contractors and a three-stage ladder; Doors has one
|
||||
* contractor that is *also* called Acme — the case the scoping rule exists for.
|
||||
*/
|
||||
const TARGETS: ImportMappingTargets = {
|
||||
workstreams: [
|
||||
{
|
||||
id: "1",
|
||||
workstreamId: "10",
|
||||
name: "Doors",
|
||||
stages: [
|
||||
{ id: "301", name: "Ordered", order: 1 },
|
||||
{ id: "302", name: "Completed", order: 2 },
|
||||
],
|
||||
contractors: [{ id: "401", organisationId: "org-c", name: "Acme Doors Ltd" }],
|
||||
},
|
||||
{
|
||||
id: "2",
|
||||
workstreamId: "11",
|
||||
name: "Windows",
|
||||
stages: [
|
||||
{ id: "201", name: "Ordered", order: 1 },
|
||||
{ id: "202", name: "In progress", order: 2 },
|
||||
{ id: "203", name: "Completed", order: 3 },
|
||||
],
|
||||
contractors: [
|
||||
{ id: "402", organisationId: "org-a", name: "Acme Windows Ltd" },
|
||||
{ id: "403", organisationId: "org-b", name: "Sunrise Glazing" },
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
function model(
|
||||
rows: string[][],
|
||||
valueMapping = EMPTY_VALUE_MAPPING,
|
||||
columns = COLUMNS,
|
||||
) {
|
||||
return buildValueMappingModel(rows, columns, valueMapping, TARGETS);
|
||||
}
|
||||
|
||||
describe("buildValueMappingModel — workstream values", () => {
|
||||
it("lists distinct values with their row counts, biggest first", () => {
|
||||
const result = model([
|
||||
["P1", "Windows", "Acme Windows Ltd", "Ordered"],
|
||||
["P2", "Windows", "Acme Windows Ltd", "Ordered"],
|
||||
["P3", "Doors", "Acme Doors Ltd", "Ordered"],
|
||||
]);
|
||||
|
||||
expect(result.workstreams.map((e) => [e.value, e.rowCount])).toEqual([
|
||||
["Windows", 2],
|
||||
["Doors", 1],
|
||||
]);
|
||||
});
|
||||
|
||||
it("auto-matches a value that is close to a workstream's name", () => {
|
||||
const result = model([["P1", "Window Programme", "Acme Windows Ltd", ""]]);
|
||||
|
||||
expect(result.workstreams[0]).toMatchObject({
|
||||
targetId: "2",
|
||||
matchedBy: "auto",
|
||||
status: "matched",
|
||||
});
|
||||
});
|
||||
|
||||
it("treats differently-cased spellings of one value as one decision", () => {
|
||||
const result = model([
|
||||
["P1", "WINDOWS", "Acme Windows Ltd", ""],
|
||||
["P2", "windows", "Acme Windows Ltd", ""],
|
||||
]);
|
||||
|
||||
expect(result.workstreams).toHaveLength(1);
|
||||
expect(result.workstreams[0].rowCount).toBe(2);
|
||||
});
|
||||
|
||||
it("leaves a value matching nothing as needs review", () => {
|
||||
const result = model([["P1", "Bathroom Programme", "Acme Doors Ltd", ""]]);
|
||||
|
||||
expect(result.workstreams[0]).toMatchObject({
|
||||
targetId: null,
|
||||
status: "needs_review",
|
||||
});
|
||||
expect(result.needsReview.workstreams).toBe(1);
|
||||
});
|
||||
|
||||
it("prefers the user's choice over the auto-match, and records that it did", () => {
|
||||
const rows = [["P1", "Window Programme", "Acme Windows Ltd", ""]];
|
||||
const chosen = setWorkstreamValue(
|
||||
EMPTY_VALUE_MAPPING,
|
||||
valueKey("Window Programme"),
|
||||
"1",
|
||||
);
|
||||
|
||||
expect(model(rows, chosen).workstreams[0]).toMatchObject({
|
||||
targetId: "1",
|
||||
matchedBy: "user",
|
||||
});
|
||||
});
|
||||
|
||||
it("does not put the suggestion back when the user clears a value", () => {
|
||||
const rows = [["P1", "Windows", "Acme Windows Ltd", ""]];
|
||||
const cleared = setWorkstreamValue(EMPTY_VALUE_MAPPING, valueKey("Windows"), null);
|
||||
|
||||
expect(model(rows, cleared).workstreams[0]).toMatchObject({
|
||||
targetId: null,
|
||||
matchedBy: null,
|
||||
status: "needs_review",
|
||||
});
|
||||
});
|
||||
|
||||
it("counts blank workstream cells rather than listing them as a value", () => {
|
||||
const result = model([
|
||||
["P1", "", "Acme Windows Ltd", ""],
|
||||
["P2", "Windows", "Acme Windows Ltd", ""],
|
||||
]);
|
||||
|
||||
expect(result.workstreams).toHaveLength(1);
|
||||
expect(result.blankRows.workstream).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildValueMappingModel — contractor values are scoped to the workstream", () => {
|
||||
it("offers only the contractors assigned to that row's workstream", () => {
|
||||
// "Acme" reads as Acme Windows under Windows and Acme Doors under Doors.
|
||||
const result = model([
|
||||
["P1", "Windows", "Acme Windows Ltd", ""],
|
||||
["P2", "Doors", "Acme Doors Ltd", ""],
|
||||
]);
|
||||
|
||||
expect(result.contractors).toHaveLength(2);
|
||||
const windows = result.contractors.find((e) => e.workstreamName === "Windows");
|
||||
const doors = result.contractors.find((e) => e.workstreamName === "Doors");
|
||||
expect(windows).toMatchObject({ targetId: "402", status: "matched" });
|
||||
expect(doors).toMatchObject({ targetId: "401", status: "matched" });
|
||||
});
|
||||
|
||||
it("keeps the same contractor text as two decisions under two workstreams", () => {
|
||||
const result = model([
|
||||
["P1", "Windows", "Sunrise Glazing", ""],
|
||||
["P2", "Doors", "Sunrise Glazing", ""],
|
||||
]);
|
||||
|
||||
expect(result.contractors).toHaveLength(2);
|
||||
// Assigned to Windows only, so under Doors it resolves to nothing.
|
||||
expect(
|
||||
result.contractors.find((e) => e.workstreamName === "Windows"),
|
||||
).toMatchObject({ targetId: "403", status: "matched" });
|
||||
expect(
|
||||
result.contractors.find((e) => e.workstreamName === "Doors"),
|
||||
).toMatchObject({ targetId: null, status: "needs_review" });
|
||||
});
|
||||
|
||||
it("merges two spellings of one workstream into a single contractor decision", () => {
|
||||
const result = model([
|
||||
["P1", "Windows", "Sunrise Glazing", ""],
|
||||
["P2", "Window Programme", "Sunrise Glazing", ""],
|
||||
]);
|
||||
|
||||
expect(result.workstreams).toHaveLength(2);
|
||||
// Both spellings are the Windows workstream, so "Sunrise Glazing" beneath
|
||||
// them is one question, asked once, over two rows.
|
||||
expect(result.contractors).toHaveLength(1);
|
||||
expect(result.contractors[0].rowCount).toBe(2);
|
||||
});
|
||||
|
||||
it("does not list contractor values whose workstream is not mapped yet", () => {
|
||||
const result = model([
|
||||
["P1", "Bathroom Programme", "Some Bathroom Co", ""],
|
||||
["P2", "Windows", "Sunrise Glazing", ""],
|
||||
]);
|
||||
|
||||
expect(result.contractors.map((e) => e.value)).toEqual(["Sunrise Glazing"]);
|
||||
expect(result.unscopedRows).toBe(1);
|
||||
});
|
||||
|
||||
it("does not count a blank workstream as merely waiting to be scoped", () => {
|
||||
// A blank workstream cell cannot be rescued by mapping, so it is reported
|
||||
// as blank rather than as "waiting on its workstream value".
|
||||
const result = model([["P1", "", "Sunrise Glazing", ""]]);
|
||||
|
||||
expect(result.blankRows.workstream).toBe(1);
|
||||
expect(result.unscopedRows).toBe(0);
|
||||
});
|
||||
|
||||
it("surfaces them as soon as the workstream is mapped", () => {
|
||||
const rows = [["P1", "Bathroom Programme", "Sunrise Glazing", ""]];
|
||||
const mapped = setWorkstreamValue(
|
||||
EMPTY_VALUE_MAPPING,
|
||||
valueKey("Bathroom Programme"),
|
||||
"2",
|
||||
);
|
||||
|
||||
const result = model(rows, mapped);
|
||||
expect(result.unscopedRows).toBe(0);
|
||||
expect(result.contractors[0]).toMatchObject({
|
||||
value: "Sunrise Glazing",
|
||||
targetId: "403",
|
||||
});
|
||||
});
|
||||
|
||||
it("counts a blank contractor rather than listing it", () => {
|
||||
const result = model([["P1", "Windows", "", ""]]);
|
||||
|
||||
expect(result.contractors).toEqual([]);
|
||||
expect(result.blankRows.contractor).toBe(1);
|
||||
});
|
||||
|
||||
it("drops a user's choice that the workstream no longer offers", () => {
|
||||
// Mapped Acme Doors under Doors, then re-pointed the value at Windows —
|
||||
// assignment 401 is not a Windows contractor, so the answer lapses.
|
||||
const rows = [["P1", "Doors", "Acme Doors Ltd", ""]];
|
||||
const stale = setContractorValue(
|
||||
setWorkstreamValue(EMPTY_VALUE_MAPPING, valueKey("Doors"), "2"),
|
||||
scopedValueKey("2", "Acme Doors Ltd"),
|
||||
"401",
|
||||
);
|
||||
|
||||
expect(model(rows, stale).contractors[0]).toMatchObject({
|
||||
targetId: null,
|
||||
status: "needs_review",
|
||||
});
|
||||
});
|
||||
|
||||
it("flags a workstream with no contractor assigned at all", () => {
|
||||
const barren: ImportMappingTargets = {
|
||||
workstreams: [
|
||||
{ id: "9", workstreamId: "19", name: "Roofs", stages: [], contractors: [] },
|
||||
],
|
||||
};
|
||||
const result = buildValueMappingModel(
|
||||
[["P1", "Roofs", "Any Co", "Ordered"]],
|
||||
COLUMNS,
|
||||
EMPTY_VALUE_MAPPING,
|
||||
barren,
|
||||
);
|
||||
|
||||
expect(result.contractors[0].noTargets).toBe(true);
|
||||
expect(result.stages[0].noTargets).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildValueMappingModel — stage values", () => {
|
||||
it("matches a stage against that workstream's own ladder", () => {
|
||||
const result = model([["P1", "Windows", "Sunrise Glazing", "In progress"]]);
|
||||
|
||||
expect(result.stages[0]).toMatchObject({
|
||||
targetId: "202",
|
||||
status: "matched",
|
||||
});
|
||||
});
|
||||
|
||||
it("does not offer another workstream's stage", () => {
|
||||
// "In progress" is on the Windows ladder, not the Doors one.
|
||||
const result = model([["P1", "Doors", "Acme Doors Ltd", "In progress"]]);
|
||||
|
||||
expect(result.stages[0]).toMatchObject({ targetId: null, status: "needs_review" });
|
||||
});
|
||||
|
||||
it("leaves blank stages out of the table — they are legitimate", () => {
|
||||
const result = model([
|
||||
["P1", "Windows", "Sunrise Glazing", ""],
|
||||
["P2", "Windows", "Sunrise Glazing", "Ordered"],
|
||||
]);
|
||||
|
||||
expect(result.stages).toHaveLength(1);
|
||||
expect(result.blankRows.stage).toBe(1);
|
||||
});
|
||||
|
||||
it("accepts the user's own stage choice", () => {
|
||||
const rows = [["P1", "Windows", "Sunrise Glazing", "Fitted"]];
|
||||
const chosen = setStageValue(
|
||||
EMPTY_VALUE_MAPPING,
|
||||
scopedValueKey("2", "Fitted"),
|
||||
"203",
|
||||
);
|
||||
|
||||
expect(model(rows, chosen).stages[0]).toMatchObject({
|
||||
targetId: "203",
|
||||
matchedBy: "user",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildValueMappingModel — warning rows", () => {
|
||||
it("is zero when everything resolves", () => {
|
||||
const result = model([
|
||||
["P1", "Windows", "Sunrise Glazing", "Ordered"],
|
||||
["P2", "Doors", "Acme Doors Ltd", ""],
|
||||
]);
|
||||
|
||||
expect(result.warningRows).toBe(0);
|
||||
});
|
||||
|
||||
it("counts a row once even when several of its values are broken", () => {
|
||||
// Unmapped workstream *and* an unmappable contractor: one broken row.
|
||||
const result = model([["P1", "Bathroom Programme", "", ""]]);
|
||||
expect(result.warningRows).toBe(1);
|
||||
});
|
||||
|
||||
it("counts blank and unmapped contractors", () => {
|
||||
const result = model([
|
||||
["P1", "Windows", "", ""],
|
||||
["P2", "Windows", "Someone Else Ltd", ""],
|
||||
["P3", "Windows", "Sunrise Glazing", ""],
|
||||
]);
|
||||
|
||||
expect(result.warningRows).toBe(2);
|
||||
});
|
||||
|
||||
it("counts a provided-but-unmatched stage, but never a blank one", () => {
|
||||
const result = model([
|
||||
["P1", "Windows", "Sunrise Glazing", ""],
|
||||
["P2", "Windows", "Sunrise Glazing", "Snagging"],
|
||||
]);
|
||||
|
||||
expect(result.warningRows).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildValueMappingModel — the counts match the file", () => {
|
||||
it("accounts for every row exactly once across the workstream table", () => {
|
||||
const rows = [
|
||||
["P1", "Windows", "Sunrise Glazing", "Ordered"],
|
||||
["P2", "WINDOWS", "Acme Windows Ltd", ""],
|
||||
["P3", "Window Programme", "", "Snagging"],
|
||||
["P4", "Doors", "Acme Doors Ltd", ""],
|
||||
["P5", "Bathroom Programme", "Anyone Ltd", ""],
|
||||
["P6", "", "Anyone Ltd", ""],
|
||||
];
|
||||
const result = model(rows);
|
||||
|
||||
const counted =
|
||||
result.workstreams.reduce((sum, e) => sum + e.rowCount, 0) +
|
||||
result.blankRows.workstream;
|
||||
expect(counted).toBe(rows.length);
|
||||
});
|
||||
|
||||
it("counts a value's rows however the file spells it", () => {
|
||||
const result = model([
|
||||
["P1", "Windows", "Sunrise Glazing", ""],
|
||||
["P2", " windows ", "Sunrise Glazing", ""],
|
||||
["P3", "WINDOWS", "Sunrise Glazing", ""],
|
||||
]);
|
||||
|
||||
expect(result.workstreams).toHaveLength(1);
|
||||
expect(result.workstreams[0].rowCount).toBe(3);
|
||||
expect(result.contractors[0].rowCount).toBe(3);
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildValueMappingModel — unmapped columns", () => {
|
||||
it("reports which value columns exist at all", () => {
|
||||
const noStage = autoMatchColumns(["Property ID", "Work Type", "Contractor"]);
|
||||
const result = buildValueMappingModel(
|
||||
[["P1", "Windows", "Sunrise Glazing"]],
|
||||
noStage,
|
||||
EMPTY_VALUE_MAPPING,
|
||||
TARGETS,
|
||||
);
|
||||
|
||||
expect(result.hasWorkstreamColumn).toBe(true);
|
||||
expect(result.hasContractorColumn).toBe(true);
|
||||
expect(result.hasStageColumn).toBe(false);
|
||||
expect(result.stages).toEqual([]);
|
||||
});
|
||||
|
||||
it("lists nothing at all when the workstream column is unmapped", () => {
|
||||
const result = buildValueMappingModel(
|
||||
[["P1", "Windows", "Sunrise Glazing", "Ordered"]],
|
||||
{ fields: {}, ignoredColumns: [] },
|
||||
EMPTY_VALUE_MAPPING,
|
||||
TARGETS,
|
||||
);
|
||||
|
||||
expect(result.workstreams).toEqual([]);
|
||||
expect(result.contractors).toEqual([]);
|
||||
expect(result.blankRows.workstream).toBe(1);
|
||||
});
|
||||
});
|
||||
493
src/lib/projects/import/valueMapping.ts
Normal file
493
src/lib/projects/import/valueMapping.ts
Normal file
|
|
@ -0,0 +1,493 @@
|
|||
/**
|
||||
* Work-order import — value mapping (issue #415, step 2 of 3, layer 2 of 2).
|
||||
*
|
||||
* Layer 1 (`./columnMapping`) said which column is the workstream. This layer
|
||||
* says which of *that column's values* is which of the project's workstreams —
|
||||
* and then, scoped to each of those, which contractor and which stage.
|
||||
*
|
||||
* ## The scoping rule (ADR-0019)
|
||||
*
|
||||
* A contractor value does not map to "a contractor"; it maps to a contractor
|
||||
* **assigned to that row's workstream** (`project_workstream_contractor`), and
|
||||
* a stage maps to a stage on **that workstream's own ladder**
|
||||
* (`project_workstream_stage`). The same text can therefore need two different
|
||||
* answers — "Acme" under Windows is a different assignment from "Acme" under
|
||||
* Doors, and may not exist at all under one of them. So contractor and stage
|
||||
* entries are keyed by `(resolved workstream, value)`, never by value alone,
|
||||
* and each offers only that workstream's assignments.
|
||||
*
|
||||
* A consequence worth stating: rows whose workstream value is blank or not yet
|
||||
* mapped have no scope, so their contractor and stage values cannot be listed
|
||||
* yet. They are counted (`unscopedRows`) rather than guessed at, and appear as
|
||||
* soon as their workstream is mapped — which is why the whole model is derived
|
||||
* fresh from `(rows, columnMapping, valueMapping, targets)` on every change
|
||||
* instead of being accumulated in state.
|
||||
*
|
||||
* ## One-to-one vs many-to-one
|
||||
*
|
||||
* Column mapping is one-to-one (one column per field), so it assigns greedily.
|
||||
* Value mapping is deliberately **many-to-one**: "Window Programme" and
|
||||
* "Windows - Phase 2" may both be the Windows workstream, so each distinct
|
||||
* value takes its own best match independently.
|
||||
*
|
||||
* ## What blocks what
|
||||
*
|
||||
* Nothing here blocks progression to validation — that is column mapping's job.
|
||||
* An unmapped value is a *warning* whose rows become validation errors at #417,
|
||||
* except a blank Stage, which is legitimate and falls back to the workstream's
|
||||
* first stage at commit.
|
||||
*
|
||||
* Pure and DB-free.
|
||||
*/
|
||||
|
||||
import {
|
||||
normaliseForMatch,
|
||||
VALUE_MATCH_THRESHOLD,
|
||||
valueSimilarity,
|
||||
} from "./similarity";
|
||||
import { cellFor, type ImportColumnMapping } from "./columnMapping";
|
||||
import type { ImportMappingTargets, WorkstreamTarget } from "./mappingTargets";
|
||||
|
||||
/**
|
||||
* The user's value-level decisions.
|
||||
*
|
||||
* A key that is **absent** has never been touched, so auto-match speaks for it.
|
||||
* A key mapped to **null** was explicitly cleared, and auto-match must not
|
||||
* quietly put its suggestion back — "I looked at this and none of them is
|
||||
* right" is an answer.
|
||||
*
|
||||
* Contractor and stage keys carry their workstream scope (see `scopedValueKey`).
|
||||
*/
|
||||
export interface ImportValueMapping {
|
||||
/** value key → `project_workstream.id`. */
|
||||
workstreams: Record<string, string | null>;
|
||||
/** scoped key → `project_workstream_contractor.id`. */
|
||||
contractors: Record<string, string | null>;
|
||||
/** scoped key → `project_workstream_stage.id`. */
|
||||
stages: Record<string, string | null>;
|
||||
}
|
||||
|
||||
export const EMPTY_VALUE_MAPPING: ImportValueMapping = {
|
||||
workstreams: {},
|
||||
contractors: {},
|
||||
stages: {},
|
||||
};
|
||||
|
||||
/**
|
||||
* The key a file value is remembered by: its normalised form, so "ACME LTD",
|
||||
* "Acme Ltd" and "acme ltd" are one decision rather than three.
|
||||
*/
|
||||
export function valueKey(value: string): string {
|
||||
return normaliseForMatch(value);
|
||||
}
|
||||
|
||||
/** The key a contractor/stage value is remembered by, within one workstream. */
|
||||
export function scopedValueKey(
|
||||
projectWorkstreamId: string,
|
||||
value: string,
|
||||
): string {
|
||||
// The workstream id is all digits (a `bigserial`), so the first space is
|
||||
// unambiguously the separator however many spaces the value itself carries.
|
||||
return `${projectWorkstreamId} ${valueKey(value)}`;
|
||||
}
|
||||
|
||||
export type ValueMappingStatus = "matched" | "needs_review";
|
||||
|
||||
/** Where a mapped value's answer came from — the UI badges auto-matches. */
|
||||
export type ValueMatchSource = "user" | "auto" | null;
|
||||
|
||||
export interface WorkstreamValueEntry {
|
||||
/** The value as it first appears in the file, for display. */
|
||||
value: string;
|
||||
/** Its `valueKey`, for reading and writing `ImportValueMapping.workstreams`. */
|
||||
key: string;
|
||||
/** How many file rows carry this value. */
|
||||
rowCount: number;
|
||||
/** `project_workstream.id`, or null when unmapped. */
|
||||
targetId: string | null;
|
||||
matchedBy: ValueMatchSource;
|
||||
status: ValueMappingStatus;
|
||||
}
|
||||
|
||||
export interface ScopedValueEntry {
|
||||
/** The workstream this decision is scoped to — the row's mapped workstream. */
|
||||
workstreamId: string;
|
||||
workstreamName: string;
|
||||
value: string;
|
||||
/** Its `scopedValueKey`, for reading and writing the mapping. */
|
||||
key: string;
|
||||
rowCount: number;
|
||||
/** `project_workstream_contractor.id` / `project_workstream_stage.id`, or null. */
|
||||
targetId: string | null;
|
||||
matchedBy: ValueMatchSource;
|
||||
status: ValueMappingStatus;
|
||||
/**
|
||||
* True when the workstream this value sits under has nothing to offer — no
|
||||
* contractor assigned, or no stage on its ladder. The readiness gate
|
||||
* (ADR-0019) makes this unreachable from the import page, but a caller that
|
||||
* skipped the gate would otherwise show an empty picker with no explanation.
|
||||
*/
|
||||
noTargets: boolean;
|
||||
}
|
||||
|
||||
export interface ValueMappingModel {
|
||||
/** Distinct workstream values, biggest row count first. */
|
||||
workstreams: WorkstreamValueEntry[];
|
||||
/** Distinct (workstream, contractor value) pairs, biggest row count first. */
|
||||
contractors: ScopedValueEntry[];
|
||||
/** Distinct (workstream, stage value) pairs. Blank stage cells are not listed. */
|
||||
stages: ScopedValueEntry[];
|
||||
/** Whether the column behind each value table is mapped at all. */
|
||||
hasWorkstreamColumn: boolean;
|
||||
hasContractorColumn: boolean;
|
||||
hasStageColumn: boolean;
|
||||
/** Rows whose cell is blank, per field. A blank stage is legitimate; the others are not. */
|
||||
blankRows: { workstream: number; contractor: number; stage: number };
|
||||
/**
|
||||
* Rows carrying a workstream value that is not yet mapped, so their
|
||||
* contractor and stage values cannot be scoped and are not listed yet. They
|
||||
* appear as soon as that workstream value is mapped — which is what separates
|
||||
* them from rows whose workstream cell is simply blank (`blankRows`).
|
||||
*/
|
||||
unscopedRows: number;
|
||||
/** Entries still needing review, per table. */
|
||||
needsReview: { workstreams: number; contractors: number; stages: number };
|
||||
/**
|
||||
* Rows that will fail validation at #417 as things stand: a blank or unmapped
|
||||
* workstream, a blank or unmapped contractor, or a *provided* stage that
|
||||
* matches nothing. Counted per row, so a row with two problems counts once.
|
||||
*/
|
||||
warningRows: number;
|
||||
}
|
||||
|
||||
/** Row-level view of one file row's three mappable values. */
|
||||
interface RowValues {
|
||||
workstream: string;
|
||||
contractor: string;
|
||||
stage: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the whole value-mapping picture from the file, the column mapping, the
|
||||
* user's value decisions and the project's targets.
|
||||
*
|
||||
* One pass over the rows to count distinct values, then a second, tiny pass
|
||||
* over the distinct sets to resolve each. It is re-run on every edit — the
|
||||
* inputs are small enough (20,000 rows at most, three cells each) that deriving
|
||||
* beats caching, and caching is what would let the tables disagree with the
|
||||
* mapping the user is looking at.
|
||||
*/
|
||||
export function buildValueMappingModel(
|
||||
rows: string[][],
|
||||
columnMapping: ImportColumnMapping,
|
||||
valueMapping: ImportValueMapping,
|
||||
targets: ImportMappingTargets,
|
||||
): ValueMappingModel {
|
||||
const hasWorkstreamColumn = columnMapping.fields.workstream !== undefined;
|
||||
const hasContractorColumn = columnMapping.fields.contractor !== undefined;
|
||||
const hasStageColumn = columnMapping.fields.stage !== undefined;
|
||||
|
||||
const blankRows = { workstream: 0, contractor: 0, stage: 0 };
|
||||
|
||||
// Pass 1 — distinct workstream values, and the contractor/stage values seen
|
||||
// beneath each of them (keyed by the *file's* workstream value, since the
|
||||
// workstream it resolves to is not known until pass 2).
|
||||
const workstreamValues = new Map<string, { value: string; rowCount: number }>();
|
||||
const rowsByWorkstreamKey = new Map<string, RowValues[]>();
|
||||
|
||||
for (const row of rows) {
|
||||
const values: RowValues = {
|
||||
workstream: cellFor(row, columnMapping, "workstream"),
|
||||
contractor: cellFor(row, columnMapping, "contractor"),
|
||||
stage: cellFor(row, columnMapping, "stage"),
|
||||
};
|
||||
|
||||
if (values.workstream === "") blankRows.workstream++;
|
||||
if (values.contractor === "") blankRows.contractor++;
|
||||
if (values.stage === "") blankRows.stage++;
|
||||
|
||||
if (values.workstream === "") continue;
|
||||
|
||||
const key = valueKey(values.workstream);
|
||||
const seen = workstreamValues.get(key);
|
||||
if (seen) seen.rowCount++;
|
||||
else workstreamValues.set(key, { value: values.workstream, rowCount: 1 });
|
||||
|
||||
const bucket = rowsByWorkstreamKey.get(key);
|
||||
if (bucket) bucket.push(values);
|
||||
else rowsByWorkstreamKey.set(key, [values]);
|
||||
}
|
||||
|
||||
// Pass 2 — resolve each distinct workstream value.
|
||||
const workstreams: WorkstreamValueEntry[] = [];
|
||||
const resolvedByKey = new Map<string, WorkstreamTarget>();
|
||||
const targetsById = new Map(targets.workstreams.map((w) => [w.id, w]));
|
||||
|
||||
for (const [key, { value, rowCount }] of workstreamValues) {
|
||||
const resolved = resolveValue(
|
||||
key,
|
||||
value,
|
||||
valueMapping.workstreams,
|
||||
targets.workstreams,
|
||||
(target) => target.name,
|
||||
(target) => target.id,
|
||||
);
|
||||
workstreams.push({ value, key, rowCount, ...resolved });
|
||||
const target = resolved.targetId
|
||||
? targetsById.get(resolved.targetId)
|
||||
: undefined;
|
||||
if (target) resolvedByKey.set(key, target);
|
||||
}
|
||||
|
||||
// Pass 3 — contractor and stage values, grouped by the workstream their rows
|
||||
// resolved to. Two file spellings of the same workstream merge here, which is
|
||||
// right: they are one workstream, so "Acme" beneath either is one decision.
|
||||
const contractorTally = new Map<
|
||||
string,
|
||||
{ workstream: WorkstreamTarget; value: string; rowCount: number }
|
||||
>();
|
||||
const stageTally = new Map<
|
||||
string,
|
||||
{ workstream: WorkstreamTarget; value: string; rowCount: number }
|
||||
>();
|
||||
// Rows with a *blank* workstream are reported as blank, not as unscoped:
|
||||
// mapping cannot rescue them, so promising that their contractors will appear
|
||||
// once the workstream is mapped would be a lie.
|
||||
let unscopedRows = 0;
|
||||
|
||||
for (const [key, bucket] of rowsByWorkstreamKey) {
|
||||
const workstreamTarget = resolvedByKey.get(key);
|
||||
if (!workstreamTarget) {
|
||||
// Unmapped workstream: no scope, so nothing beneath it can be listed yet.
|
||||
unscopedRows += bucket.length;
|
||||
continue;
|
||||
}
|
||||
for (const values of bucket) {
|
||||
if (hasContractorColumn && values.contractor !== "") {
|
||||
tally(contractorTally, workstreamTarget, values.contractor);
|
||||
}
|
||||
// A blank stage is legitimate — it falls back to the workstream's first
|
||||
// stage at commit — so it is counted as blank, never listed for mapping.
|
||||
if (hasStageColumn && values.stage !== "") {
|
||||
tally(stageTally, workstreamTarget, values.stage);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const contractors = [...contractorTally.entries()].map(
|
||||
([key, { workstream, value, rowCount }]) => ({
|
||||
workstreamId: workstream.id,
|
||||
workstreamName: workstream.name,
|
||||
value,
|
||||
key,
|
||||
rowCount,
|
||||
noTargets: workstream.contractors.length === 0,
|
||||
...resolveValue(
|
||||
key,
|
||||
value,
|
||||
valueMapping.contractors,
|
||||
workstream.contractors,
|
||||
(target) => target.name,
|
||||
(target) => target.id,
|
||||
),
|
||||
}),
|
||||
);
|
||||
|
||||
const stages = [...stageTally.entries()].map(
|
||||
([key, { workstream, value, rowCount }]) => ({
|
||||
workstreamId: workstream.id,
|
||||
workstreamName: workstream.name,
|
||||
value,
|
||||
key,
|
||||
rowCount,
|
||||
noTargets: workstream.stages.length === 0,
|
||||
...resolveValue(
|
||||
key,
|
||||
value,
|
||||
valueMapping.stages,
|
||||
workstream.stages,
|
||||
(target) => target.name,
|
||||
(target) => target.id,
|
||||
),
|
||||
}),
|
||||
);
|
||||
|
||||
byRowCount(workstreams);
|
||||
byRowCount(contractors);
|
||||
byRowCount(stages);
|
||||
|
||||
return {
|
||||
workstreams,
|
||||
contractors,
|
||||
stages,
|
||||
hasWorkstreamColumn,
|
||||
hasContractorColumn,
|
||||
hasStageColumn,
|
||||
blankRows,
|
||||
unscopedRows,
|
||||
needsReview: {
|
||||
workstreams: workstreams.filter((e) => e.status === "needs_review").length,
|
||||
contractors: contractors.filter((e) => e.status === "needs_review").length,
|
||||
stages: stages.filter((e) => e.status === "needs_review").length,
|
||||
},
|
||||
warningRows: countWarningRows(
|
||||
rowsByWorkstreamKey,
|
||||
resolvedByKey,
|
||||
blankRows.workstream,
|
||||
new Set(
|
||||
contractors.filter((e) => e.status === "needs_review").map((e) => e.key),
|
||||
),
|
||||
new Set(
|
||||
stages.filter((e) => e.status === "needs_review").map((e) => e.key),
|
||||
),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
/** Add one row's contribution to a scoped tally. */
|
||||
function tally(
|
||||
into: Map<
|
||||
string,
|
||||
{ workstream: WorkstreamTarget; value: string; rowCount: number }
|
||||
>,
|
||||
workstream: WorkstreamTarget,
|
||||
value: string,
|
||||
): void {
|
||||
const key = scopedValueKey(workstream.id, value);
|
||||
const seen = into.get(key);
|
||||
if (seen) seen.rowCount++;
|
||||
else into.set(key, { workstream, value, rowCount: 1 });
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide one value: the user's answer if they gave one, otherwise the best
|
||||
* target above the auto-match threshold, otherwise nothing.
|
||||
*
|
||||
* A user answer that names a target no longer on offer (they mapped a
|
||||
* contractor, then re-pointed the workstream at one that contractor is not
|
||||
* assigned to) is discarded rather than kept as a dangling id — the value falls
|
||||
* back to needing review, which is the truth of the situation.
|
||||
*/
|
||||
function resolveValue<T>(
|
||||
key: string,
|
||||
value: string,
|
||||
decisions: Record<string, string | null>,
|
||||
candidates: readonly T[],
|
||||
nameOf: (target: T) => string,
|
||||
idOf: (target: T) => string,
|
||||
): { targetId: string | null; matchedBy: ValueMatchSource; status: ValueMappingStatus } {
|
||||
if (Object.prototype.hasOwnProperty.call(decisions, key)) {
|
||||
const chosen = decisions[key];
|
||||
if (chosen === null) {
|
||||
return { targetId: null, matchedBy: null, status: "needs_review" };
|
||||
}
|
||||
if (candidates.some((target) => idOf(target) === chosen)) {
|
||||
return { targetId: chosen, matchedBy: "user", status: "matched" };
|
||||
}
|
||||
return { targetId: null, matchedBy: null, status: "needs_review" };
|
||||
}
|
||||
|
||||
let best: { id: string; score: number } | null = null;
|
||||
let tied = false;
|
||||
for (const target of candidates) {
|
||||
const score = valueSimilarity(value, nameOf(target));
|
||||
if (score < VALUE_MATCH_THRESHOLD) continue;
|
||||
if (best === null || score > best.score) {
|
||||
best = { id: idOf(target), score };
|
||||
tied = false;
|
||||
} else if (score === best.score) {
|
||||
tied = true;
|
||||
}
|
||||
}
|
||||
|
||||
// Two candidates the value fits equally well is not a match, it is a
|
||||
// question: "Acme" beneath a workstream carrying both Acme Windows and Acme
|
||||
// Doors must be asked, not guessed. (ADR-0019 leaves same-org disambiguation
|
||||
// to #417; this keeps the guess from being made here in the meantime.)
|
||||
return best && !tied
|
||||
? { targetId: best.id, matchedBy: "auto", status: "matched" }
|
||||
: { targetId: null, matchedBy: null, status: "needs_review" };
|
||||
}
|
||||
|
||||
/** Biggest first, then alphabetical — stable as the user maps, since neither key changes. */
|
||||
function byRowCount(entries: Array<{ rowCount: number; value: string }>): void {
|
||||
entries.sort(
|
||||
(a, b) => b.rowCount - a.rowCount || a.value.localeCompare(b.value, "en-GB"),
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* How many rows will fail validation at #417 as things stand.
|
||||
*
|
||||
* Counted **per row**, by walking the rows again rather than by summing the
|
||||
* tables' row counts: a row with an unmapped workstream *and* a blank
|
||||
* contractor is one broken row, not two, and summing the tables would report it
|
||||
* twice.
|
||||
*
|
||||
* A row is broken when its workstream is blank or unmapped, its contractor is
|
||||
* blank or unmapped, or it carries a stage that matches nothing. A *blank*
|
||||
* stage is not broken — it falls back to the workstream's first stage.
|
||||
*/
|
||||
function countWarningRows(
|
||||
rowsByWorkstreamKey: Map<string, RowValues[]>,
|
||||
resolvedByKey: Map<string, WorkstreamTarget>,
|
||||
blankWorkstreamRows: number,
|
||||
unmappedContractorKeys: Set<string>,
|
||||
unmappedStageKeys: Set<string>,
|
||||
): number {
|
||||
// A blank workstream cell is unmappable, so every such row is already broken.
|
||||
let broken = blankWorkstreamRows;
|
||||
|
||||
for (const [key, bucket] of rowsByWorkstreamKey) {
|
||||
const workstream = resolvedByKey.get(key);
|
||||
if (!workstream) {
|
||||
// The whole bucket shares one unmapped workstream value.
|
||||
broken += bucket.length;
|
||||
continue;
|
||||
}
|
||||
for (const values of bucket) {
|
||||
const contractorBroken =
|
||||
values.contractor === "" ||
|
||||
unmappedContractorKeys.has(scopedValueKey(workstream.id, values.contractor));
|
||||
const stageBroken =
|
||||
values.stage !== "" &&
|
||||
unmappedStageKeys.has(scopedValueKey(workstream.id, values.stage));
|
||||
if (contractorBroken || stageBroken) broken++;
|
||||
}
|
||||
}
|
||||
|
||||
return broken;
|
||||
}
|
||||
|
||||
/** Record one workstream-value decision. `null` means "none of these". */
|
||||
export function setWorkstreamValue(
|
||||
mapping: ImportValueMapping,
|
||||
key: string,
|
||||
targetId: string | null,
|
||||
): ImportValueMapping {
|
||||
return {
|
||||
...mapping,
|
||||
workstreams: { ...mapping.workstreams, [key]: targetId },
|
||||
};
|
||||
}
|
||||
|
||||
/** Record one contractor decision, scoped to its workstream. */
|
||||
export function setContractorValue(
|
||||
mapping: ImportValueMapping,
|
||||
key: string,
|
||||
targetId: string | null,
|
||||
): ImportValueMapping {
|
||||
return {
|
||||
...mapping,
|
||||
contractors: { ...mapping.contractors, [key]: targetId },
|
||||
};
|
||||
}
|
||||
|
||||
/** Record one stage decision, scoped to its workstream. */
|
||||
export function setStageValue(
|
||||
mapping: ImportValueMapping,
|
||||
key: string,
|
||||
targetId: string | null,
|
||||
): ImportValueMapping {
|
||||
return { ...mapping, stages: { ...mapping.stages, [key]: targetId } };
|
||||
}
|
||||
Loading…
Add table
Reference in a new issue