Repository for the assessment model application
Find a file
Daniel Roth 9cfde2767f feat(ara-projects): work-order import step 1 — upload and parse
Adds `/projects/[projectId]/import`: a CSV/XLSX drop zone that uploads to a
route handler, parses server-side with SheetJS (already a dependency — the
Bulk tag assignment flow is the in-app parsing precedent, not the FastAPI
BulkUpload pipeline), and returns headers plus a first-N preview.

Nothing is written to the database at this step. Insert happens at commit
(#417).

Files:
  src/lib/projects/import/parse.ts       pure, DB-free parser + template
  src/lib/projects/import/parse.test.ts  23 unit tests, no DB, no fixtures
  .../api/projects/[projectId]/import/parse/route.ts     POST multipart
  .../api/projects/[projectId]/import/template/route.ts  GET template CSV
  .../projects/[projectId]/import/page.tsx               server component
  .../import/components/ImportUpload.tsx                 drop zone + preview

Persistence choice: hold rows client-side, not S3-and-reparse
-------------------------------------------------------------
The ticket asks for a choice between uploading the raw file to S3 via the
presigned-URL pattern in `src/lib/bulkUpload/client.ts` and re-parsing it per
step, versus holding the parsed rows in the client and POSTing them at commit.

Chosen: hold the rows client-side. The parse route therefore returns every
row, not only the preview, and the mapping (#415) and validation (#416) steps
read them from `ImportUpload`'s mutation state.

Reasoning:

1. S3-and-reparse needs somewhere to keep the object key and the import's
   status between steps, and the Ara Projects schema (PR #404) has no import
   table. Adding one means designing a second BulkUpload-shaped lifecycle —
   hard to reverse, and ADR-worthy — for a v1 that is insert-only with no
   diffing (deferred to #425). Not worth it to carry a file across three steps
   of one sitting.

2. The scale does not call for it. This import declares which workstreams the
   properties of a single project take — thousands of rows, not the
   hundreds-of-thousands the BulkUpload property pipeline handles. The 20k row
   cap bounds the commit POST to a few MB of JSON.

3. Re-parsing per step parses the same bytes three or four times and lets the
   steps disagree if the parser ever changes underneath them. Client-held rows
   are by construction the exact rows the user previewed and will validate.

4. It matches the existing precedent: Bulk tag assignment (ADR-0013) parses a
   small spreadsheet and POSTs the extracted identifiers at commit, with no
   S3 round trip.

The cost is that a page refresh loses progress and the commit POST carries the
payload. Both are acceptable for a single-sitting, insert-only v1; if the
import later needs to be resumable or to exceed this scale, S3 staging plus an
import table is the upgrade path, and the parser is already isolated behind
`parseImportFile` so only the transport changes.

Notes
-----
- The parser is deliberately header-agnostic: it reports the headers it finds
  rather than demanding the template's. Mapping arbitrary client headings is
  #415's job, and rejecting unrecognised headings here would make the mapping
  step unreachable for exactly the files that need it.
- Authorization uses `canManageProject` from `src/lib/projects/authz.ts` with
  facts from the #408 repository, so importing is internal/client-org only and
  never a contractor. Unreachable projects 404 rather than 403. The page
  applies the same check so a contractor never sees the drop zone.
- `raw: false` on the SheetJS read keeps leading zeros on numeric-looking
  property identifiers (covered by a test).
- Size is capped at 10MB client- and server-side. The wireframe says 50MB;
  10MB is far past what four narrow columns can plausibly reach.
- Known limitation, documented in a characterisation test: SheetJS does not
  throw on loose bytes, so a multi-line non-spreadsheet renamed to .csv parses
  into nonsense headers rather than being rejected. Nothing is written, the
  extension allowlist catches the honestly-named case, and the user sees the
  garbage in the preview. Content sniffing here would be guesswork.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 16:43:48 +00:00
.claude save 2026-06-29 10:54:48 +00:00
.devcontainer pull latest agentic-toolkit version 2026-07-21 10:05:23 +00:00
.github/workflows updating CI 2026-05-05 19:36:33 +00:00
.impeccable/critique feat(portfolio): address impeccable critique — no-plans state, contrast, focus 2026-07-13 21:50:27 +00:00
.vscode added db filtereing 2026-01-05 17:30:58 +00:00
backlog update instruction doc and context 2026-06-03 10:33:52 +00:00
cypress fix(ara-projects): require user onboarding before /projects 2026-07-21 16:08:47 +00:00
docs docs(ara-projects): ADR-0018 + CONTEXT.md Ara Projects domain language 2026-07-21 08:30:32 +00:00
drizzle/meta Got drizzle orm working 2023-07-10 17:48:37 +01:00
public added robots and security txt files 2026-03-10 18:56:53 +00:00
src feat(ara-projects): work-order import step 1 — upload and parse 2026-07-21 16:43:48 +00:00
.db-env added files for reports 2025-10-30 17:15:30 +00:00
.eslintrc.json Initial commit from Create Next App 2023-05-25 06:07:00 +01:00
.gitignore fix(live): flag major condition issues by the issue, not a retired stage / S3 URL 2026-07-20 16:42:55 +01:00
CLAUDE.md updated claude.md 2026-05-06 16:11:00 +00:00
components.json Added layout of toolbar n portfolio page 2023-07-12 15:41:58 +01:00
CONTEXT.md docs(ara-projects): ADR-0018 + CONTEXT.md Ara Projects domain language 2026-07-21 08:30:32 +00:00
cypress.config.ts Creating automated testing - blocked until we have a test account 2023-07-11 12:14:48 +01:00
devcontainer.sh pr for claude skills 2026-05-05 19:53:21 +00:00
drizzle.config.ts Fix drizzle-kit schema glob and generate Projects migration 2026-07-17 15:36:08 +00:00
generate_migration.sh added migration scripts and ability to add a deal in hubspot@ 2025-10-22 15:44:00 +00:00
migrate_to_db.sh new migration files 2026-06-03 12:55:49 +00:00
next.config.js revamping the plan page 2026-04-10 21:54:26 +00:00
package-lock.json Surface coordinator damp & mould commentary in the risk drill-down 2026-05-28 14:58:23 +00:00
package.json perf(backfill): faster, resumable recommendation denormalization + ops tooling 2026-07-02 10:00:46 +00:00
postcss.config.js Initial commit from Create Next App 2023-05-25 06:07:00 +01:00
PRODUCT.md docs(design): home-page redesign context — Stitch refs, PRODUCT.md, impeccable trial 2026-07-07 15:12:07 +00:00
README.md deploy? 2026-05-28 13:52:01 +00:00
run_build.sh addded changes to let build work 2025-08-21 11:30:51 +00:00
run_local.sh automated script that ran for next 15 2025-07-22 10:50:50 +00:00
skills-lock.json Updated survey request UI for Devon County Council 2026-05-06 20:37:30 +00:00
tailwind.config.js re-vamping ara ui 2026-04-08 10:29:14 +00:00
tsconfig.json minor visual tweaks 2026-04-10 22:49:00 +00:00
vitest.config.ts fixing multi company bug 2026-05-07 20:08:04 +00:00

This is a Next.js project bootstrapped with create-next-app.

Getting Started

When first getting set up you'll firstly want to install the existing dependencies. To do this, simply run

npm install
# or
yarn install

First, run the development server:

npm run dev
# or
yarn dev
# or
pnpm dev

Open http://localhost:3000 with your browser to see the result.

You can start editing the page by modifying app/page.tsx. The page auto-updates as you edit the file.

This project uses next/font to automatically optimize and load Inter, a custom Google Font.

Learn More

To learn more about Next.js, take a look at the following resources:

You can check out the Next.js GitHub repository - your feedback and contributions are welcome!

Deploy on Vercel

The easiest way to deploy your Next.js app is to use the Vercel Platform from the creators of Next.js.

Check out our Next.js deployment documentation for more details.

We currently have a development version, found at https://assessment-model-dev.vercel.app, and a production version at https://assessment-model.vercel.app, however the production version is missing a significant number of environmental variables, which will need to be added.

Drizzle ORM

We're using Drizzle ORM to interface with our AWS Postgres database. Documentation on getting set up can be found here

Schemas

In order to get started with Drizzle, a schema needs to be created. Schemas can be added src/app/db/schema as typescript files. See the documentation on how to set up schemas but effectively, Drizzle allows you to define schemas as typescript code, which allows for simple, type safe schema definitions.

Creating Migrations

To create a migration, a command has been set up in package.json. Simply run

npm run migration:generate

Or with yarn/pnmp accordingly.

Note, there seems to be a bug with Drizzle which is documented here.

The workaround is to open up tsconfig.json and comment out "target": "es5", and replace it with "target": "ESNext". This should hopefully only be a temporary workaround required.

Pushing migrations

To push migrations, another command has been set up in package.json, since drizzle-kit currently does not support pushing for Postgres out of the box.

Run

npm run migration:push

Which will commit changes to the database. The database changes will be pushed to the public schema, whereas a meta record will be pushed to the __drizzle_migrations schema.

Inserting users into the database

In order to insert a user into the database, simply run

npm run create_users -- {email} {firstName}

Since we're using just the built in process arguments to read command line arguments, the ordering of arguments needs to be email address and then name

Cypress Testing Documentation

This document provides an overview of how to perform end-to-end testing using Cypress for the login functionality of the application. The testing code is based on the provided code snippets.

Prerequisites

Node.js installed on your machine Cypress installed as a dev dependency in your project

Test Execution

To execute the login tests, follow these steps:

  1. Run
npm run test:e2e:open

Which will open Crypress test runner

  1. Select the tests that you want to run. At the time of writing, only login tests have been completed

Key files

The key files that are at play for testing are documented here. Because of some issues testing next-auth and setting cookies, a custom command to set the JWT, and avoid the functionality defined in the signIn function as defined in src/app/api/auth/[...nextauth]/route.ts

cypress/plugins/index.js is a standard file, required by cypress to journey through the Google Oauth flow

cypress/figtures/session.json is a user fixture that is used to log in a user in the login tests

cypress/support/commands.ts creates a custom login user command which sets a JWT and allows us to actually authenticate. cy.intercept only mocks the client side behaviour of the apis and therefore does not set any cookies, do this function does this manually.

Generating pre-signed urls

In our terraform stack, we have a module called s3_presignable_bucket which contains the definition for our bucket which we will use to store retrofit plan input csv's in.

We will generate a pre-signed url and then make a post request to that endpoint to store that data to s3. Part of that process is the creation of an AWS IAM role which contains the permission set to access the bucket, rerofit-plan-inputs-<stage>. The name of this IAM role is s3_presign_role_<stage> and for our NextJS application, as it's hosted outside of AWS (for the moment), we need to generate a set of access credentials to give the application access to this bucket. The access key and secret key are automatically generated and stored in AWS secrets manager under dev/presign_frontend/access_key and dev/presign_frontend/secret_key and need to be set in the environment for the pre-sign api to store csv data to aws.