From 9bf36ad4b6e04c04af3905a0fc7f103250ab38b7 Mon Sep 17 00:00:00 2001 From: Khalim Conn-Kowlessar Date: Thu, 2 Jul 2026 10:00:46 +0000 Subject: [PATCH] perf(backfill): faster, resumable recommendation denormalization + ops tooling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Overhaul the recommendation plan_id/material_* backfill for the 21M-row prod table (see docs/runbooks/recommendation-denormalization-backfill.md): - Single combined pass: set plan_id + all four material_* columns in one write per row (was two full passes, rewriting every row twice). Guard + COALESCE keep it idempotent and resumable — done rows are never rewritten. - Session tuning: synchronous_commit=off, tunable work_mem / maintenance_work_mem. - Range + phase controls (BACKFILL_START_ID/END_ID/SKIP_CARDINALITY/ONLY_FINALIZE) to run disjoint id-range workers in parallel when the disk isn't the bottleneck. - Percent-complete + ETA in the progress log. - Finalize builds the two new indexes plain by default (faster in a quiet window) with a CONCURRENTLY toggle. The real speed-up in prod was dropping the 3 secondary indexes for the backfill then rebuilding — the table is packed (fillfactor 100) so updates can't go HOT and otherwise maintain every index per row. Index drop/rebuild + vacuum are kept as standalone SQL so ops controls exactly when the app loses/regains them: - sql/drop-recommendation-backfill-indexes.sql - sql/rebuild-recommendation-backfill-indexes.sql (1GB maintenance_work_mem — safe on the 4GB instance) - sql/vacuum-recommendation.sql run-sql.ts runs .sql files statement-by-statement, autocommitted (so VACUUM / CREATE INDEX CONCURRENTLY work), for environments without the psql CLI. Wired up as npm run db:run-sql and backfill:recommendation-restore. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...recommendation-denormalization-backfill.md | 140 +++++++++ package.json | 5 +- ...backfill-recommendation-denormalization.ts | 291 +++++++++++------- src/app/db/run-sql.ts | 68 ++++ .../drop-recommendation-backfill-indexes.sql | 26 ++ ...ebuild-recommendation-backfill-indexes.sql | 31 ++ src/app/db/sql/vacuum-recommendation.sql | 7 + 7 files changed, 461 insertions(+), 107 deletions(-) create mode 100644 docs/runbooks/recommendation-denormalization-backfill.md create mode 100644 src/app/db/run-sql.ts create mode 100644 src/app/db/sql/drop-recommendation-backfill-indexes.sql create mode 100644 src/app/db/sql/rebuild-recommendation-backfill-indexes.sql create mode 100644 src/app/db/sql/vacuum-recommendation.sql diff --git a/docs/runbooks/recommendation-denormalization-backfill.md b/docs/runbooks/recommendation-denormalization-backfill.md new file mode 100644 index 00000000..916ff601 --- /dev/null +++ b/docs/runbooks/recommendation-denormalization-backfill.md @@ -0,0 +1,140 @@ +# Runbook: `recommendation` denormalisation backfill + +Populates `recommendation.plan_id` and `recommendation.material_*` from the +`plan_recommendations` / `recommendation_materials` join tables, as the first +step toward making those columns the source of truth and dropping the join +tables. See [ADR 0001](../adr/0001-data-backfills-outside-drizzle.md) for why +this runs outside drizzle. + +The table is ~21m rows. This runbook is the **fast** procedure. + +## Why it's slow by default, and the lever + +`recommendation` is packed (fillfactor 100), so an UPDATE can't do a HOT update — +Postgres writes a new row version and inserts into **every index on the table** +for all ~21m rows. That index write-amplification (and its WAL) is what drains IO. + +The lever: **drop the secondary indexes, backfill, rebuild them.** With them gone, +each updated row only maintains the primary key (~4 index inserts → 1). Combined +with the single-pass rewrite (plan_id + material in one UPDATE, not two) and +`synchronous_commit = off`, this turns a ~1-day run into a few hours. + +`DROP INDEX` is near-instant — the cost people mean is the *rebuild*, which is why +we do it once, at the end, in a low-usage window. + +## What the app needs to keep working + +Only the 3 secondary indexes. The half-filled `plan_id`/`material_id` columns are +NULL-tolerant (the read paths already handle NULLs), and the *new* +plan_id/material_id indexes aren't used by anything yet. So **at any stopping +point, rebuild the 3 secondary indexes and the app is back to today's behaviour** +— a 50%-done backfill is invisible to it. + +## Prerequisites + +- `npm run migration:migrate` has applied 0222/0224 (the column + `NOT VALID` FK adds). +- `.env.local` points at the target DB (`DB_HOST`/`DB_PORT`/`DB_USERNAME`/`DB_PASSWORD`/`DB_NAME`). +- A low-usage window. + +## Files + +- Script: `src/app/db/backfill-recommendation-denormalization.ts` (`npm run backfill:recommendation-denormalization`) +- Drop: `src/app/db/sql/drop-recommendation-backfill-indexes.sql` +- Rebuild:`src/app/db/sql/rebuild-recommendation-backfill-indexes.sql` + +--- + +## Procedure A — one window (a few hours, start to finish) + +```bash +# 1. Drop the secondary indexes (app reads degrade to seq scans until step 4) +psql "$DATABASE_URL" -f src/app/db/sql/drop-recommendation-backfill-indexes.sql + +# 2. Run the full backfill + finalize (builds the 2 new indexes, validates FKs) +npm run backfill:recommendation-denormalization + +# 3. Rebuild the secondary indexes — app back to normal +psql "$DATABASE_URL" -f src/app/db/sql/rebuild-recommendation-backfill-indexes.sql + +# 4. Reclaim the ~2x bloat in-place updates leave behind, refresh stats +psql "$DATABASE_URL" -c "VACUUM (ANALYZE) recommendation;" +``` + +`DATABASE_URL` is illustrative — use whatever psql connection matches `.env.local`. + +--- + +## Procedure B — split across windows (50% now, finish later) + +Each window is self-contained; between windows the app is 100% normal. + +**Every window except the last:** + +```bash +psql "$DATABASE_URL" -f src/app/db/sql/drop-recommendation-backfill-indexes.sql +BACKFILL_FINALIZE=false npm run backfill:recommendation-denormalization # Ctrl-C any time; it's resumable +psql "$DATABASE_URL" -f src/app/db/sql/rebuild-recommendation-backfill-indexes.sql +``` + +`BACKFILL_FINALIZE=false` skips the new-index build + FK validate — there's no +point building the plan_id/material_id indexes until the data is all in. + +**The final window** (when the per-batch logs / the NULL summary show plan_id is +down to ~0 and only genuinely material-less rows remain): + +```bash +psql "$DATABASE_URL" -f src/app/db/sql/drop-recommendation-backfill-indexes.sql +npm run backfill:recommendation-denormalization # FINALIZE defaults on: builds new indexes, validates FKs +psql "$DATABASE_URL" -f src/app/db/sql/rebuild-recommendation-backfill-indexes.sql +psql "$DATABASE_URL" -c "VACUUM (ANALYZE) recommendation;" +``` + +Cost of splitting: you rebuild the secondary indexes once per window. Two of the +three are partial (tiny), so each restore is effectively one full-size index +build (`recommendation_property_id_idx`, a few minutes). + +### Interrupting mid-run + +Ctrl-C is safe. The script commits per batch, so completed batches persist. Re-run +to continue — the `IS NULL` guards skip every row already done. If you're worried +about app performance while stopped, rebuild the secondary indexes first. + +--- + +## Alternative — no index juggling (simpler, slower) + +If you'd rather not touch indexes at all, just run the tuned single pass with the +indexes left in place and stop/resume freely: + +```bash +BACKFILL_FINALIZE=false npm run backfill:recommendation-denormalization +``` + +Slower per row (index write-amplification stays), but the app is always fully +indexed and there's zero drop/rebuild ceremony. Finalize on the last run. + +--- + +## Tunables (env) + +| Var | Default | Notes | +| --- | --- | --- | +| `BACKFILL_BATCH_SIZE` | `100000` | rows scanned per committed batch | +| `BACKFILL_SLEEP_MS` | `0` | inter-batch pause; raise to throttle IO if not in a quiet window | +| `BACKFILL_SYNC_COMMIT` | `off` | `on` to keep synchronous_commit (safe to leave off — run is resumable) | +| `BACKFILL_MAINTENANCE_WORK_MEM` | `1GB` | memory for the finalize index builds | +| `BACKFILL_WORK_MEM` | `128MB` | memory for the batch joins | +| `BACKFILL_INDEX_CONCURRENTLY` | *(off)* | `true` builds the new indexes CONCURRENTLY (non-blocking, slower) | +| `BACKFILL_FINALIZE` | *(on)* | `false` skips new-index build + FK validate (partial windows) | + +## Verifying completion + +The script prints remaining NULLs at the end: + +- `plan_id` NULL → should trend to ~0 (investigate any residue: recommendations with no `plan_recommendations` row). +- `material_id` NULL → **expected** for recommendations that have no material; not an error. + +## After the backfill is fully done + +A later migration can `SET NOT NULL` on `plan_id` and drop the join tables, gated +on this backfill having completed with zero unexpected NULLs. Out of scope here. diff --git a/package.json b/package.json index e205a7e5..92fda474 100644 --- a/package.json +++ b/package.json @@ -14,7 +14,10 @@ "migration:generate": "drizzle-kit generate", "migration:migrate": "drizzle-kit migrate", "create_user": "tsx src/app/db/create_user.ts", - "backfill:recommendation-denormalization": "tsx src/app/db/backfill-recommendation-denormalization.ts" + "backfill:recommendation-denormalization": "tsx src/app/db/backfill-recommendation-denormalization.ts", + "backfill:recommendation-denormalization:pass": "BACKFILL_FINALIZE=false tsx src/app/db/backfill-recommendation-denormalization.ts", + "db:run-sql": "tsx src/app/db/run-sql.ts", + "backfill:recommendation-restore": "tsx src/app/db/run-sql.ts src/app/db/sql/rebuild-recommendation-backfill-indexes.sql src/app/db/sql/vacuum-recommendation.sql" }, "dependencies": { "@aws-sdk/client-s3": "^3.971.0", diff --git a/src/app/db/backfill-recommendation-denormalization.ts b/src/app/db/backfill-recommendation-denormalization.ts index ce8b74d2..b0854571 100644 --- a/src/app/db/backfill-recommendation-denormalization.ts +++ b/src/app/db/backfill-recommendation-denormalization.ts @@ -1,7 +1,8 @@ /** * Backfills the denormalised plan_id / material_* columns on `recommendation` - * from the plan_recommendations and recommendation_materials join tables, then - * builds the supporting indexes and validates the foreign keys — all ONLINE. + * from the plan_recommendations and recommendation_materials join tables in a + * SINGLE keyset-paginated pass, then (optionally) builds the supporting indexes + * and validates the foreign keys — all ONLINE and resumable. * * Why this lives outside drizzle: * drizzle-kit migrate wraps every pending migration in ONE transaction, so it @@ -14,21 +15,55 @@ * Run AFTER `npm run migration:migrate` has applied 0222/0224 (the column adds): * npm run backfill:recommendation-denormalization * + * ── Making it fast (see docs/runbooks/recommendation-denormalization-backfill.md) + * The dominant cost is index write-amplification: `recommendation` is packed + * (fillfactor 100), so each UPDATE can't do a HOT update and must insert into + * every index on the table for every one of ~21m rows. To go roughly an order + * of magnitude faster in a low-usage window: + * 1. DROP the 3 secondary indexes (src/app/db/sql/drop-recommendation-backfill-indexes.sql) + * 2. run this script (BACKFILL_FINALIZE=false for a partial/interrupted window) + * 3. REBUILD them after (src/app/db/sql/rebuild-recommendation-backfill-indexes.sql) + * The backfill is resumable: stop any time, rebuild the indexes, and the app is + * back to today's behaviour (the half-filled columns are NULL-tolerant). Re-run + * to continue — the IS NULL guards never rewrite a row that's already done. + * + * This pass sets plan_id AND the four material_* columns in ONE write per row + * (the previous version did two full passes, rewriting every row twice). + * * Safe to re-run: every step is idempotent (IS NULL guards, IF NOT EXISTS, - * VALIDATE on an already-valid constraint is a no-op). If interrupted, just run - * it again — it resumes from wherever it left off. + * VALIDATE on an already-valid constraint is a no-op). * * Tunables via env: - * BACKFILL_BATCH_SIZE rows scanned per committed batch (default 25000) - * BACKFILL_SLEEP_MS pause between batches, eases IO pressure (default 50) + * BACKFILL_BATCH_SIZE rows scanned per committed batch (default 100000) + * BACKFILL_SLEEP_MS pause between batches, eases IO pressure (default 0) + * BACKFILL_SYNC_COMMIT 'on' keeps synchronous_commit; default 'off' (faster + * batch commits — safe here because the run is resumable) + * BACKFILL_MAINTENANCE_WORK_MEM session maintenance_work_mem for index builds (default '1GB') + * BACKFILL_WORK_MEM session work_mem for the batch joins (default '128MB') + * BACKFILL_INDEX_CONCURRENTLY 'true' builds the new indexes CONCURRENTLY (default plain + * CREATE INDEX — faster, but briefly blocks writes) + * BACKFILL_FINALIZE 'false' skips the index build + FK validate; use this for + * partial/interrupted windows, finalize on the last one */ import dotenv from "dotenv"; import { Pool, type PoolClient } from "pg"; dotenv.config({ path: ".env.local" }); -const BATCH_SIZE = Number(process.env.BACKFILL_BATCH_SIZE ?? 25_000); -const SLEEP_MS = Number(process.env.BACKFILL_SLEEP_MS ?? 50); +const BATCH_SIZE = Number(process.env.BACKFILL_BATCH_SIZE ?? 100_000); +const SLEEP_MS = Number(process.env.BACKFILL_SLEEP_MS ?? 0); +const SYNC_COMMIT = process.env.BACKFILL_SYNC_COMMIT === "on" ? "on" : "off"; +const MAINTENANCE_WORK_MEM = process.env.BACKFILL_MAINTENANCE_WORK_MEM ?? "1GB"; +const WORK_MEM = process.env.BACKFILL_WORK_MEM ?? "128MB"; +const INDEX_CONCURRENTLY = process.env.BACKFILL_INDEX_CONCURRENTLY === "true"; +const FINALIZE = process.env.BACKFILL_FINALIZE !== "false"; + +// Range + phase controls, for running several workers over disjoint id ranges in +// parallel (near-linear speed-up — the ranges never contend for the same rows). +const START_ID = process.env.BACKFILL_START_ID ?? "0"; +const END_ID = process.env.BACKFILL_END_ID ?? null; // inclusive upper bound; null = unbounded +const SKIP_CARDINALITY = process.env.BACKFILL_SKIP_CARDINALITY === "true"; +const ONLY_FINALIZE = process.env.BACKFILL_ONLY_FINALIZE === "true"; const pool = new Pool({ host: process.env.DB_HOST, @@ -65,28 +100,79 @@ async function assertSingleCardinality( /** * Keyset-paginate the whole `recommendation` table by id and, for each batch, - * set the target columns from the join table where a match exists. Each batch - * is its own autocommitted statement (no surrounding BEGIN), so locks are held - * only for the rows in that batch. + * set plan_id and the four material_* columns from the join tables in a single + * UPDATE. Each batch is its own autocommitted statement (no surrounding BEGIN), + * so locks are held only for the rows in that batch. * - * We scan by id (not `WHERE col IS NULL LIMIT n`) so rows with no match don't - * get re-selected forever — they're simply left NULL and we move past them. + * We scan by id (not `WHERE col IS NULL LIMIT n`) so rows with no match — or a + * legitimately NULL material_id — don't get re-selected forever; they're left as + * they are and we move past them. + * + * The guard only touches a row when there is real work to do (a NULL column that + * has a matching source row), and COALESCE never overwrites an already-set value. + * So a row is written at most once, and a re-run after interruption skips every + * row already done — the pass is idempotent and resumable. + * + * The single-cardinality assertion above guarantees the LEFT JOINs cannot fan a + * row out into multiple update targets. */ -async function backfillColumns( - client: PoolClient, - label: string, - updateSql: (idFrom: string, limit: string) => string, -): Promise { - let lastId = "0"; +const fmtDuration = (secs: number): string => { + const s = Math.max(0, Math.round(secs)); + const h = Math.floor(s / 3600); + const m = Math.floor((s % 3600) / 60); + return h > 0 ? `${h}h${m}m` : m > 0 ? `${m}m${s % 60}s` : `${s}s`; +}; + +async function backfillDenormalized(client: PoolClient): Promise { + let lastId = START_ID; let scannedTotal = 0; let updatedTotal = 0; const startedAt = Date.now(); + // Upper bound of the id keyset, for percent-complete + ETA. When END_ID is set + // (a parallel worker), use it; otherwise ask the table (fast, PK). + let maxId: number; + if (END_ID != null) { + maxId = Number(END_ID); + } else { + const { rows: maxRows } = await client.query( + `SELECT max(id) AS max_id FROM recommendation`, + ); + maxId = Number(maxRows[0].max_id ?? 0); + } + console.log( + `Backfilling id ${Number(START_ID).toLocaleString()} → ${maxId.toLocaleString()}...`, + ); + for (;;) { - const { rows } = await client.query(updateSql("$1", "$2"), [ - lastId, - BATCH_SIZE, - ]); + const { rows } = await client.query( + `WITH batch AS ( + SELECT id FROM recommendation + WHERE id > $1 + AND ($3::bigint IS NULL OR id <= $3::bigint) + ORDER BY id + LIMIT $2 + ), + upd AS ( + UPDATE recommendation r + SET plan_id = COALESCE(r.plan_id, pr.plan_id), + material_id = COALESCE(r.material_id, rm.material_id), + material_quantity = COALESCE(r.material_quantity, rm.quantity), + material_quantity_unit = COALESCE(r.material_quantity_unit, rm.quantity_unit), + material_depth = COALESCE(r.material_depth, rm.depth) + FROM batch b + LEFT JOIN plan_recommendations pr ON pr.recommendation_id = b.id + LEFT JOIN recommendation_materials rm ON rm.recommendation_id = b.id + WHERE r.id = b.id + AND ( (r.plan_id IS NULL AND pr.plan_id IS NOT NULL) + OR (r.material_id IS NULL AND rm.material_id IS NOT NULL) ) + RETURNING r.id + ) + SELECT (SELECT max(id) FROM batch) AS max_id, + (SELECT count(*) FROM batch) AS scanned, + (SELECT count(*) FROM upd) AS updated`, + [lastId, BATCH_SIZE, END_ID], + ); const { max_id, scanned, updated } = rows[0] as { max_id: string | null; scanned: number; @@ -99,17 +185,22 @@ async function backfillColumns( updatedTotal += Number(updated); lastId = max_id; - const rate = Math.round(scannedTotal / ((Date.now() - startedAt) / 1000)); + const elapsed = (Date.now() - startedAt) / 1000; + const rate = Math.round(scannedTotal / elapsed); + const from = Number(START_ID); + const span = maxId - from; + const pct = span > 0 ? Math.min(100, ((Number(lastId) - from) / span) * 100) : 0; + const eta = pct > 0 ? fmtDuration((elapsed / pct) * (100 - pct)) : "?"; console.log( - `[${label}] up to id ${lastId} — scanned ${scannedTotal.toLocaleString()}, ` + - `updated ${updatedTotal.toLocaleString()} (${rate.toLocaleString()} rows/s scan)`, + `${pct.toFixed(1)}% (id ${lastId}) — scanned ${scannedTotal.toLocaleString()}, ` + + `updated ${updatedTotal.toLocaleString()} — ${rate.toLocaleString()} rows/s, ETA ${eta}`, ); if (SLEEP_MS > 0) await sleep(SLEEP_MS); } console.log( - `[${label}] done: scanned ${scannedTotal.toLocaleString()}, ` + + `Backfill pass done: scanned ${scannedTotal.toLocaleString()}, ` + `updated ${updatedTotal.toLocaleString()} in ` + `${Math.round((Date.now() - startedAt) / 1000)}s`, ); @@ -119,91 +210,77 @@ async function main(): Promise { const client = await pool.connect(); try { console.log( - `Config: batch=${BATCH_SIZE.toLocaleString()} rows, sleep=${SLEEP_MS}ms\n`, + `Config: batch=${BATCH_SIZE.toLocaleString()} rows, sleep=${SLEEP_MS}ms, ` + + `synchronous_commit=${SYNC_COMMIT}, work_mem=${WORK_MEM}, ` + + `maintenance_work_mem=${MAINTENANCE_WORK_MEM}, finalize=${FINALIZE}, ` + + `range=(${Number(START_ID).toLocaleString()}, ${END_ID ?? "∞"}]` + + `${ONLY_FINALIZE ? ", ONLY_FINALIZE" : ""}\n`, ); - // 1. Guard the 1:1 assumption before mutating anything. - console.log("Checking cardinality..."); - await assertSingleCardinality(client, "plan_recommendations"); - await assertSingleCardinality(client, "recommendation_materials"); + // Session tuning. synchronous_commit=off is safe here: the backfill is + // idempotent and resumable, so a crash that loses the last few batch commits + // just means the next run continues from there. work_mem feeds the batch + // joins; maintenance_work_mem feeds the CREATE INDEX in the finalize step. + await client.query(`SET synchronous_commit = ${SYNC_COMMIT}`); + await client.query(`SET work_mem = '${WORK_MEM}'`); + await client.query(`SET maintenance_work_mem = '${MAINTENANCE_WORK_MEM}'`); - // 2. Backfill plan_id. - await backfillColumns( - client, - "plan_id", - (idFrom, limit) => ` - WITH batch AS ( - SELECT id FROM recommendation - WHERE id > ${idFrom} - ORDER BY id - LIMIT ${limit} - ), - upd AS ( - UPDATE recommendation r - SET plan_id = pr.plan_id - FROM batch b - JOIN plan_recommendations pr ON pr.recommendation_id = b.id - WHERE r.id = b.id - AND r.plan_id IS NULL - RETURNING r.id - ) - SELECT (SELECT max(id) FROM batch) AS max_id, - (SELECT count(*) FROM batch) AS scanned, - (SELECT count(*) FROM upd) AS updated`, - ); + if (ONLY_FINALIZE) { + // Parallel workflow: the data was written by range workers; this run just + // builds the indexes + validates the FKs. No cardinality check, no pass. + console.log("ONLY_FINALIZE: skipping cardinality check + backfill pass."); + } else { + // 1. Guard the 1:1 assumption before mutating anything. Skippable on + // secondary parallel workers (one worker having checked is enough). + if (SKIP_CARDINALITY) { + console.log("Skipping cardinality check (BACKFILL_SKIP_CARDINALITY=true)."); + } else { + console.log("Checking cardinality..."); + await assertSingleCardinality(client, "plan_recommendations"); + await assertSingleCardinality(client, "recommendation_materials"); + } - // 3. Backfill the four material_* columns in one pass. - await backfillColumns( - client, - "material", - (idFrom, limit) => ` - WITH batch AS ( - SELECT id FROM recommendation - WHERE id > ${idFrom} - ORDER BY id - LIMIT ${limit} - ), - upd AS ( - UPDATE recommendation r - SET material_id = rm.material_id, - material_quantity = rm.quantity, - material_quantity_unit = rm.quantity_unit, - material_depth = rm.depth - FROM batch b - JOIN recommendation_materials rm ON rm.recommendation_id = b.id - WHERE r.id = b.id - AND r.material_id IS NULL - RETURNING r.id - ) - SELECT (SELECT max(id) FROM batch) AS max_id, - (SELECT count(*) FROM batch) AS scanned, - (SELECT count(*) FROM upd) AS updated`, - ); + // 2. Single combined pass: plan_id + material_* in one write per row. + console.log("Backfilling plan_id + material_* (single pass)..."); + await backfillDenormalized(client); + } - // 4. Build indexes CONCURRENTLY (no write lock). Must NOT be in a txn — - // a pooled client with no BEGIN runs each statement autocommitted. - console.log("Creating indexes concurrently..."); - await client.query( - `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_recommendation_plan_id - ON recommendation USING btree (plan_id)`, - ); - await client.query( - `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_recommendation_material_id - ON recommendation USING btree (material_id)`, - ); + if (FINALIZE || ONLY_FINALIZE) { + // 3. Build the two new indexes. Plain CREATE INDEX is faster in a quiet + // window; BACKFILL_INDEX_CONCURRENTLY=true avoids the brief write lock. + // Autocommitted (no surrounding BEGIN), which CONCURRENTLY requires. + const cc = INDEX_CONCURRENTLY ? "CONCURRENTLY " : ""; + console.log( + `Creating indexes (${INDEX_CONCURRENTLY ? "concurrently" : "plain"})...`, + ); + await client.query( + `CREATE INDEX ${cc}IF NOT EXISTS idx_recommendation_plan_id + ON recommendation USING btree (plan_id)`, + ); + await client.query( + `CREATE INDEX ${cc}IF NOT EXISTS idx_recommendation_material_id + ON recommendation USING btree (material_id)`, + ); - // 5. Validate the FKs online (ShareUpdateExclusiveLock — allows reads/writes). - console.log("Validating foreign keys..."); - await client.query( - `ALTER TABLE recommendation - VALIDATE CONSTRAINT recommendation_plan_id_plan_id_fk`, - ); - await client.query( - `ALTER TABLE recommendation - VALIDATE CONSTRAINT recommendation_material_id_material_id_fk`, - ); + // 4. Validate the FKs online (ShareUpdateExclusiveLock — allows reads/writes). + console.log("Validating foreign keys..."); + await client.query( + `ALTER TABLE recommendation + VALIDATE CONSTRAINT recommendation_plan_id_plan_id_fk`, + ); + await client.query( + `ALTER TABLE recommendation + VALIDATE CONSTRAINT recommendation_material_id_material_id_fk`, + ); + } else { + console.log( + "Skipping index build + FK validate (BACKFILL_FINALIZE=false). " + + "Run once more with finalize enabled in your final window.", + ); + } - // 6. Report any rows still unlinked (expected for material; investigate for plan). + // 5. Report any rows still unlinked. material_id NULL is expected (not every + // recommendation has a material); plan_id NULL should trend toward ~0. const { rows } = await client.query( `SELECT count(*) FILTER (WHERE plan_id IS NULL) AS plan_null, count(*) FILTER (WHERE material_id IS NULL) AS material_null @@ -213,7 +290,9 @@ async function main(): Promise { `\nRemaining NULLs — plan_id: ${Number(rows[0].plan_null).toLocaleString()}, ` + `material_id: ${Number(rows[0].material_null).toLocaleString()}`, ); - console.log("Backfill complete."); + console.log( + FINALIZE ? "Backfill complete." : "Backfill pass complete (not finalized).", + ); } finally { client.release(); await pool.end(); diff --git a/src/app/db/run-sql.ts b/src/app/db/run-sql.ts new file mode 100644 index 00000000..2b491b8c --- /dev/null +++ b/src/app/db/run-sql.ts @@ -0,0 +1,68 @@ +/** + * Runs one or more .sql files against the DB from .env.local, executing each + * statement autocommitted (no wrapping transaction) so statements that cannot run + * inside a transaction block — VACUUM, CREATE INDEX CONCURRENTLY — work. For + * environments without the psql CLI. + * + * tsx src/app/db/run-sql.ts [more.sql ...] + * + * Statement splitting is deliberately simple: it strips full-line `--` comments + * and splits on `;`. That's correct for the hand-written files in + * src/app/db/sql, but it is NOT a general SQL parser — don't point it at files + * with `;` inside string literals or dollar-quoted function bodies. + */ +import { readFileSync } from "fs"; +import dotenv from "dotenv"; +import { Pool } from "pg"; + +dotenv.config({ path: ".env.local" }); + +const files = process.argv.slice(2); +if (files.length === 0) { + console.error("Usage: tsx src/app/db/run-sql.ts [more.sql ...]"); + process.exit(1); +} + +const pool = new Pool({ + host: process.env.DB_HOST, + port: Number(process.env.DB_PORT), + user: process.env.DB_USERNAME, + password: process.env.DB_PASSWORD, + database: process.env.DB_NAME, +}); + +function statementsOf(path: string): string[] { + const raw = readFileSync(path, "utf8"); + return raw + .split("\n") + .filter((line) => !line.trim().startsWith("--")) + .join("\n") + .split(";") + .map((s) => s.trim()) + .filter((s) => s.length > 0); +} + +async function main(): Promise { + const client = await pool.connect(); + try { + for (const file of files) { + console.log(`\n=== ${file} ===`); + for (const sql of statementsOf(file)) { + const preview = sql.replace(/\s+/g, " ").slice(0, 70); + const startedAt = Date.now(); + console.log(`> ${preview}${sql.length > 70 ? "…" : ""}`); + await client.query(sql); + console.log(` ok (${Math.round((Date.now() - startedAt) / 1000)}s)`); + } + } + console.log("\nAll statements complete."); + } finally { + client.release(); + await pool.end(); + } +} + +main().catch((err) => { + console.error("SQL run failed:", err); + process.exit(1); +}); diff --git a/src/app/db/sql/drop-recommendation-backfill-indexes.sql b/src/app/db/sql/drop-recommendation-backfill-indexes.sql new file mode 100644 index 00000000..2fd28dd1 --- /dev/null +++ b/src/app/db/sql/drop-recommendation-backfill-indexes.sql @@ -0,0 +1,26 @@ +-- Drop the 3 secondary indexes on `recommendation` so the denormalisation +-- backfill can run without per-row index write-amplification. +-- +-- `recommendation` is packed (fillfactor 100), so each backfill UPDATE cannot do +-- a HOT update and must insert a new entry into every index on the table for all +-- ~21m rows. Dropping these first cuts index inserts per updated row from ~4 to 1 +-- (just the primary key), which is the bulk of the speed-up. +-- +-- These serve production reads (fetch-by-property, the active-defaults paths), so +-- ONLY run this in a low-usage window and rebuild them +-- (rebuild-recommendation-backfill-indexes.sql) as soon as the window's backfill +-- pass finishes. DROP INDEX is near-instant (catalog change + file unlink); it +-- takes a brief ACCESS EXCLUSIVE lock, so if long-running queries are in flight +-- use `DROP INDEX CONCURRENTLY` (cannot run inside a transaction) instead. +-- +-- Kept OUT of the backfill script on purpose so you control exactly when the app +-- loses and regains these indexes. See +-- docs/runbooks/recommendation-denormalization-backfill.md +-- +-- NOTE: idx_recommendation_active_id_property was originally built CONCURRENTLY; +-- if a prior CONCURRENTLY build was ever interrupted it can linger as INVALID — +-- the DROP below clears it either way. + +DROP INDEX IF EXISTS recommendation_property_id_idx; +DROP INDEX IF EXISTS idx_recommendation_active_defaults; +DROP INDEX IF EXISTS idx_recommendation_active_id_property; diff --git a/src/app/db/sql/rebuild-recommendation-backfill-indexes.sql b/src/app/db/sql/rebuild-recommendation-backfill-indexes.sql new file mode 100644 index 00000000..37714e3b --- /dev/null +++ b/src/app/db/sql/rebuild-recommendation-backfill-indexes.sql @@ -0,0 +1,31 @@ +-- Rebuild the 3 secondary indexes on `recommendation` after a backfill window, +-- restoring the app to its normal (pre-backfill) query behaviour. +-- +-- The definitions below reproduce exactly what the migrations created (partial +-- WHERE clauses included). Run any time — the backfill does NOT need to be +-- complete first: these indexes don't reference plan_id/material_id, so a +-- half-finished backfill is invisible to them and to the app. +-- +-- Two of the three are PARTIAL (only rows where default = true and not yet +-- installed), so they cover a small slice of the table and rebuild in seconds; +-- only recommendation_property_id_idx spans all rows and takes a few minutes. +-- +-- Plain CREATE INDEX is fastest but briefly blocks writes while each builds. If +-- you need writes to keep flowing, replace CREATE INDEX with +-- CREATE INDEX CONCURRENTLY (slower, cannot run inside a transaction, and each +-- statement must be sent on its own). + +-- 1GB comfortably holds the ~500MB sort for a 21M-row single-column btree. +-- Keep it well under instance RAM (this DB is 4GB) to avoid memory pressure. +SET maintenance_work_mem = '1GB'; + +CREATE INDEX IF NOT EXISTS recommendation_property_id_idx + ON recommendation USING btree (property_id); + +CREATE INDEX IF NOT EXISTS idx_recommendation_active_defaults + ON recommendation USING btree (id) + WHERE "default" = true AND already_installed = false; + +CREATE INDEX IF NOT EXISTS idx_recommendation_active_id_property + ON recommendation USING btree (id, property_id) + WHERE "default" = true AND already_installed = false; diff --git a/src/app/db/sql/vacuum-recommendation.sql b/src/app/db/sql/vacuum-recommendation.sql new file mode 100644 index 00000000..d5495446 --- /dev/null +++ b/src/app/db/sql/vacuum-recommendation.sql @@ -0,0 +1,7 @@ +-- Reclaim the dead tuples the backfill left behind (it updated ~every row, so the +-- table is ~2x bloated) and refresh planner stats so queries pick up the new +-- indexes. Standard VACUUM (not FULL) — online, no exclusive lock, but it does a +-- full heap + index scan so it can take a while. No progress output; watch +-- pg_stat_progress_vacuum in another session. + +VACUUM (ANALYZE) recommendation;