Adds `/projects/[projectId]/import`: a CSV/XLSX drop zone that uploads to a route handler, parses server-side with SheetJS (already a dependency — the Bulk tag assignment flow is the in-app parsing precedent, not the FastAPI BulkUpload pipeline), and returns headers plus a first-N preview. Nothing is written to the database at this step. Insert happens at commit (#417). Files: src/lib/projects/import/parse.ts pure, DB-free parser + template src/lib/projects/import/parse.test.ts 23 unit tests, no DB, no fixtures .../api/projects/[projectId]/import/parse/route.ts POST multipart .../api/projects/[projectId]/import/template/route.ts GET template CSV .../projects/[projectId]/import/page.tsx server component .../import/components/ImportUpload.tsx drop zone + preview Persistence choice: hold rows client-side, not S3-and-reparse ------------------------------------------------------------- The ticket asks for a choice between uploading the raw file to S3 via the presigned-URL pattern in `src/lib/bulkUpload/client.ts` and re-parsing it per step, versus holding the parsed rows in the client and POSTing them at commit. Chosen: hold the rows client-side. The parse route therefore returns every row, not only the preview, and the mapping (#415) and validation (#416) steps read them from `ImportUpload`'s mutation state. Reasoning: 1. S3-and-reparse needs somewhere to keep the object key and the import's status between steps, and the Ara Projects schema (PR #404) has no import table. Adding one means designing a second BulkUpload-shaped lifecycle — hard to reverse, and ADR-worthy — for a v1 that is insert-only with no diffing (deferred to #425). Not worth it to carry a file across three steps of one sitting. 2. The scale does not call for it. This import declares which workstreams the properties of a single project take — thousands of rows, not the hundreds-of-thousands the BulkUpload property pipeline handles. The 20k row cap bounds the commit POST to a few MB of JSON. 3. Re-parsing per step parses the same bytes three or four times and lets the steps disagree if the parser ever changes underneath them. Client-held rows are by construction the exact rows the user previewed and will validate. 4. It matches the existing precedent: Bulk tag assignment (ADR-0013) parses a small spreadsheet and POSTs the extracted identifiers at commit, with no S3 round trip. The cost is that a page refresh loses progress and the commit POST carries the payload. Both are acceptable for a single-sitting, insert-only v1; if the import later needs to be resumable or to exceed this scale, S3 staging plus an import table is the upgrade path, and the parser is already isolated behind `parseImportFile` so only the transport changes. Notes ----- - The parser is deliberately header-agnostic: it reports the headers it finds rather than demanding the template's. Mapping arbitrary client headings is #415's job, and rejecting unrecognised headings here would make the mapping step unreachable for exactly the files that need it. - Authorization uses `canManageProject` from `src/lib/projects/authz.ts` with facts from the #408 repository, so importing is internal/client-org only and never a contractor. Unreachable projects 404 rather than 403. The page applies the same check so a contractor never sees the drop zone. - `raw: false` on the SheetJS read keeps leading zeros on numeric-looking property identifiers (covered by a test). - Size is capped at 10MB client- and server-side. The wireframe says 50MB; 10MB is far past what four narrow columns can plausibly reach. - Known limitation, documented in a characterisation test: SheetJS does not throw on loose bytes, so a multi-line non-spreadsheet renamed to .csv parses into nonsense headers rather than being rejected. Nothing is written, the extension allowlist catches the honestly-named case, and the user sees the garbage in the preview. Content sniffing here would be guesswork. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .claude | ||
| .devcontainer | ||
| .github/workflows | ||
| .impeccable/critique | ||
| .vscode | ||
| backlog | ||
| cypress | ||
| docs | ||
| drizzle/meta | ||
| public | ||
| src | ||
| .db-env | ||
| .eslintrc.json | ||
| .gitignore | ||
| CLAUDE.md | ||
| components.json | ||
| CONTEXT.md | ||
| cypress.config.ts | ||
| devcontainer.sh | ||
| drizzle.config.ts | ||
| generate_migration.sh | ||
| migrate_to_db.sh | ||
| next.config.js | ||
| package-lock.json | ||
| package.json | ||
| postcss.config.js | ||
| PRODUCT.md | ||
| README.md | ||
| run_build.sh | ||
| run_local.sh | ||
| skills-lock.json | ||
| tailwind.config.js | ||
| tsconfig.json | ||
| vitest.config.ts | ||
This is a Next.js project bootstrapped with create-next-app.
Getting Started
When first getting set up you'll firstly want to install the existing dependencies. To do this, simply run
npm install
# or
yarn install
First, run the development server:
npm run dev
# or
yarn dev
# or
pnpm dev
Open http://localhost:3000 with your browser to see the result.
You can start editing the page by modifying app/page.tsx. The page auto-updates as you edit the file.
This project uses next/font to automatically optimize and load Inter, a custom Google Font.
Learn More
To learn more about Next.js, take a look at the following resources:
- Next.js Documentation - learn about Next.js features and API.
- Learn Next.js - an interactive Next.js tutorial.
You can check out the Next.js GitHub repository - your feedback and contributions are welcome!
Deploy on Vercel
The easiest way to deploy your Next.js app is to use the Vercel Platform from the creators of Next.js.
Check out our Next.js deployment documentation for more details.
We currently have a development version, found at https://assessment-model-dev.vercel.app, and a production version at https://assessment-model.vercel.app, however the production version is missing a significant number of environmental variables, which will need to be added.
Drizzle ORM
We're using Drizzle ORM to interface with our AWS Postgres database. Documentation on getting set up can be found here
Schemas
In order to get started with Drizzle, a schema needs to be created. Schemas can be added src/app/db/schema as typescript files. See the documentation on how to set up schemas but effectively, Drizzle allows you to define schemas as typescript code, which allows for simple, type safe schema definitions.
Creating Migrations
To create a migration, a command has been set up in package.json. Simply run
npm run migration:generate
Or with yarn/pnmp accordingly.
Note, there seems to be a bug with Drizzle which is documented here.
The workaround is to open up tsconfig.json and comment out "target": "es5", and replace it with "target": "ESNext". This should hopefully only
be a temporary workaround required.
Pushing migrations
To push migrations, another command has been set up in package.json, since drizzle-kit currently does not support pushing for Postgres out of the box.
Run
npm run migration:push
Which will commit changes to the database. The database changes will be pushed to the public schema, whereas a meta record will be pushed to the __drizzle_migrations schema.
Inserting users into the database
In order to insert a user into the database, simply run
npm run create_users -- {email} {firstName}
Since we're using just the built in process arguments to read command line arguments, the ordering of arguments needs to be email address and then name
Cypress Testing Documentation
This document provides an overview of how to perform end-to-end testing using Cypress for the login functionality of the application. The testing code is based on the provided code snippets.
Prerequisites
Node.js installed on your machine Cypress installed as a dev dependency in your project
Test Execution
To execute the login tests, follow these steps:
- Run
npm run test:e2e:open
Which will open Crypress test runner
- Select the tests that you want to run. At the time of writing, only login tests have been completed
Key files
The key files that are at play for testing are documented here. Because of some issues testing next-auth and setting cookies, a custom command to set
the JWT, and avoid the functionality defined in the signIn function as defined in src/app/api/auth/[...nextauth]/route.ts
cypress/plugins/index.js is a standard file, required by cypress to journey through the Google Oauth flow
cypress/figtures/session.json is a user fixture that is used to log in a user in the login tests
cypress/support/commands.ts creates a custom login user command which sets a JWT and allows us to actually authenticate. cy.intercept only mocks the client side behaviour of the apis and therefore does not set any cookies, do this function does this manually.
Generating pre-signed urls
In our terraform stack, we have a module called s3_presignable_bucket which contains the definition for our bucket which we will use to store retrofit plan input csv's in.
We will generate a pre-signed url and then make a post request to that endpoint to store that data to s3. Part of that process is the creation of an AWS IAM role which contains
the permission set to access the bucket, rerofit-plan-inputs-<stage>. The name of this IAM role is s3_presign_role_<stage> and for our NextJS application, as it's hosted outside of AWS (for the moment), we need to generate a set of access credentials to give the application access to this bucket. The access key and secret key are automatically generated and stored in AWS secrets manager under dev/presign_frontend/access_key and dev/presign_frontend/secret_key and need to be set in the environment for the pre-sign api to store csv data to aws.