4422cebe0b
The pure half of import step 2 of 3, built on #414's parsed-rows-held-client-side model. No database writes, no migrations; four new modules under src/lib/projects/import, all pure except one read-only loader. similarity.ts — the auto-match primitive Normalisation plus a Sørensen–Dice bigram score, with word containment ranked above letter overlap ("Contractor" ⊂ "Contractor Name" beats "Stage"/"Storage"). Headings and cell values match by *different* rules: valueSimilarity adds directional, length-weighted word coverage so "Window Programme" reaches "Windows" (0.91) and "Door Replacement" reaches "Doors", at a higher threshold (0.75 vs 0.62) so "Acme Windows Ltd" does not reach "Acme Doors Ltd" (0.67). The leniency is deliberately kept out of heading matching, where it would map "Forecast Start" onto "Forecast end date" and mis-read every row in the file. columnMapping.ts — file column -> Ara field Seven fields (landlord property id, UPRN, workstream, contractor, stage, WO reference, forecast end date) with alias lists drawn from real asset-management exports. Requirements are modelled as groups: property identifier is satisfied by landlord property id *or* UPRN, so a file may carry either or both. autoMatchColumns assigns greedily — one column per field, best pair first, and no guess below the threshold. Two deliberate non-matches: a Status column is not the Stage field (CONTEXT.md keeps them apart), and nothing matches Priority or cost (v1 ignores them; #426 owns priority). Per-column status is matched / ignored / needs_review, where "ignored" is the user's explicit "don't import" and is distinct from not having looked yet. valueMapping.ts — cell values -> the project's own entities Contractor and stage values are keyed by (resolved workstream, value), never by value alone: "Acme" under Windows is a different assignment from "Acme" under Doors and may not exist under one of them (ADR-0019). Two file spellings of one workstream merge into a single contractor question. Rows whose workstream is blank or unmapped have no scope, so their contractor/stage values are counted (unscopedRows) rather than guessed, and appear the moment the workstream is mapped. Value matching is many-to-one (several spellings, one workstream) where column matching is one-to-one, and a tie between two equally-good candidates is refused rather than guessed. A user's answer that the workstream no longer offers lapses to needs-review instead of persisting as a dangling id. warningRows is counted per row by a second pass, not by summing the tables — a row with an unmapped workstream and a blank contractor is one broken row. mappingTargets.ts — what values may map to Pure assembleMappingTargets + one read-only loadMappingTargets (three SELECTs). Every bigserial id is stringified once, here, because bigint cannot cross the server->client boundary. Stages keep ladder order (stages[0] is the blank-stage fallback at commit); workstreams and contractors sort alphabetically. session.ts — the artefact the mapping is persisted alongside ImportSession carries the file identity plus both mapping layers, JSON-clean end to end. sessionMatchesFile refuses a session whose headers have moved, since a mapping is column indices. Blocking rule (AC): only unmapped *required* columns block progression to validation. Unmapped values are warnings whose rows fail at #417; a blank stage is not a warning at all (it falls back to the workstream's first stage). 112 tests pass in src/lib/projects/import (66 new); tsc clean. Tests mock @/app/db/db and open no connection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .agents | ||
| .claude | ||
| .devcontainer | ||
| .github/workflows | ||
| .impeccable/critique | ||
| .vscode | ||
| backlog | ||
| cypress | ||
| docs | ||
| drizzle/meta | ||
| public | ||
| src | ||
| vitest | ||
| .db-env | ||
| .eslintrc.json | ||
| .gitignore | ||
| CLAUDE.md | ||
| components.json | ||
| CONTEXT.md | ||
| cypress.config.ts | ||
| devcontainer.sh | ||
| drizzle.config.ts | ||
| generate_migration.sh | ||
| migrate_to_db.sh | ||
| next.config.js | ||
| package-lock.json | ||
| package.json | ||
| postcss.config.js | ||
| PRODUCT.md | ||
| README.md | ||
| run_build.sh | ||
| run_local.sh | ||
| skills-lock.json | ||
| tailwind.config.js | ||
| tsconfig.json | ||
| vitest.config.ts | ||
This is a Next.js project bootstrapped with create-next-app.
Getting Started
When first getting set up you'll firstly want to install the existing dependencies. To do this, simply run
npm install
# or
yarn install
First, run the development server:
npm run dev
# or
yarn dev
# or
pnpm dev
Open http://localhost:3000 with your browser to see the result.
You can start editing the page by modifying app/page.tsx. The page auto-updates as you edit the file.
This project uses next/font to automatically optimize and load Inter, a custom Google Font.
Learn More
To learn more about Next.js, take a look at the following resources:
- Next.js Documentation - learn about Next.js features and API.
- Learn Next.js - an interactive Next.js tutorial.
You can check out the Next.js GitHub repository - your feedback and contributions are welcome!
Deploy on Vercel
The easiest way to deploy your Next.js app is to use the Vercel Platform from the creators of Next.js.
Check out our Next.js deployment documentation for more details.
We currently have a development version, found at https://assessment-model-dev.vercel.app, and a production version at https://assessment-model.vercel.app, however the production version is missing a significant number of environmental variables, which will need to be added.
Drizzle ORM
We're using Drizzle ORM to interface with our AWS Postgres database. Documentation on getting set up can be found here
Schemas
In order to get started with Drizzle, a schema needs to be created. Schemas can be added src/app/db/schema as typescript files. See the documentation on how to set up schemas but effectively, Drizzle allows you to define schemas as typescript code, which allows for simple, type safe schema definitions.
Creating Migrations
To create a migration, a command has been set up in package.json. Simply run
npm run migration:generate
Or with yarn/pnmp accordingly.
Note, there seems to be a bug with Drizzle which is documented here.
The workaround is to open up tsconfig.json and comment out "target": "es5", and replace it with "target": "ESNext". This should hopefully only
be a temporary workaround required.
Pushing migrations
To push migrations, another command has been set up in package.json, since drizzle-kit currently does not support pushing for Postgres out of the box.
Run
npm run migration:push
Which will commit changes to the database. The database changes will be pushed to the public schema, whereas a meta record will be pushed to the __drizzle_migrations schema.
Inserting users into the database
In order to insert a user into the database, simply run
npm run create_users -- {email} {firstName}
Since we're using just the built in process arguments to read command line arguments, the ordering of arguments needs to be email address and then name
Cypress Testing Documentation
This document provides an overview of how to perform end-to-end testing using Cypress for the login functionality of the application. The testing code is based on the provided code snippets.
Prerequisites
Node.js installed on your machine Cypress installed as a dev dependency in your project
Test Execution
To execute the login tests, follow these steps:
- Run
npm run test:e2e:open
Which will open Crypress test runner
- Select the tests that you want to run. At the time of writing, only login tests have been completed
Key files
The key files that are at play for testing are documented here. Because of some issues testing next-auth and setting cookies, a custom command to set
the JWT, and avoid the functionality defined in the signIn function as defined in src/app/api/auth/[...nextauth]/route.ts
cypress/plugins/index.js is a standard file, required by cypress to journey through the Google Oauth flow
cypress/figtures/session.json is a user fixture that is used to log in a user in the login tests
cypress/support/commands.ts creates a custom login user command which sets a JWT and allows us to actually authenticate. cy.intercept only mocks the client side behaviour of the apis and therefore does not set any cookies, do this function does this manually.
Generating pre-signed urls
In our terraform stack, we have a module called s3_presignable_bucket which contains the definition for our bucket which we will use to store retrofit plan input csv's in.
We will generate a pre-signed url and then make a post request to that endpoint to store that data to s3. Part of that process is the creation of an AWS IAM role which contains
the permission set to access the bucket, rerofit-plan-inputs-<stage>. The name of this IAM role is s3_presign_role_<stage> and for our NextJS application, as it's hosted outside of AWS (for the moment), we need to generate a set of access credentials to give the application access to this bucket. The access key and secret key are automatically generated and stored in AWS secrets manager under dev/presign_frontend/access_key and dev/presign_frontend/secret_key and need to be set in the environment for the pre-sign api to store csv data to aws.