perf(backfill): faster, resumable recommendation denormalization + ops tooling

Overhaul the recommendation plan_id/material_* backfill for the 21M-row prod table (see docs/runbooks/recommendation-denormalization-backfill.md): - Single combined pass: set plan_id + all four material_* columns in one write per row (was two full passes, rewriting every row twice). Guard + COALESCE keep it idempotent and resumable — done rows are never rewritten. - Session tuning: synchronous_commit=off, tunable work_mem / maintenance_work_mem. - Range + phase controls (BACKFILL_START_ID/END_ID/SKIP_CARDINALITY/ONLY_FINALIZE) to run disjoint id-range workers in parallel when the disk isn't the bottleneck. - Percent-complete + ETA in the progress log. - Finalize builds the two new indexes plain by default (faster in a quiet window) with a CONCURRENTLY toggle. The real speed-up in prod was dropping the 3 secondary indexes for the backfill then rebuilding — the table is packed (fillfactor 100) so updates can't go HOT and otherwise maintain every index per row. Index drop/rebuild + vacuum are kept as standalone SQL so ops controls exactly when the app loses/regains them: - sql/drop-recommendation-backfill-indexes.sql - sql/rebuild-recommendation-backfill-indexes.sql (1GB maintenance_work_mem — safe on the 4GB instance) - sql/vacuum-recommendation.sql run-sql.ts runs .sql files statement-by-statement, autocommitted (so VACUUM / CREATE INDEX CONCURRENTLY work), for environments without the psql CLI. Wired up as npm run db:run-sql and backfill:recommendation-restore. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 08:48:34 +00:00 · 2026-07-02 10:00:46 +00:00 · 2026-07-02 10:00:46 +00:00 · 9bf36ad4b6
commit 9bf36ad4b6
parent 1dfb1e1534
7 changed files with 461 additions and 107 deletions
--- a/docs/runbooks/recommendation-denormalization-backfill.md
+++ b/docs/runbooks/recommendation-denormalization-backfill.md
@ -0,0 +1,140 @@
+# Runbook: `recommendation` denormalisation backfill
+
+Populates `recommendation.plan_id` and `recommendation.material_*` from the
+`plan_recommendations` / `recommendation_materials` join tables, as the first
+step toward making those columns the source of truth and dropping the join
+tables. See [ADR 0001](../adr/0001-data-backfills-outside-drizzle.md) for why
+this runs outside drizzle.
+
+The table is ~21m rows. This runbook is the **fast** procedure.
+
+## Why it's slow by default, and the lever
+
+`recommendation` is packed (fillfactor 100), so an UPDATE can't do a HOT update —
+Postgres writes a new row version and inserts into **every index on the table**
+for all ~21m rows. That index write-amplification (and its WAL) is what drains IO.
+
+The lever: **drop the secondary indexes, backfill, rebuild them.** With them gone,
+each updated row only maintains the primary key (~4 index inserts → 1). Combined
+with the single-pass rewrite (plan_id + material in one UPDATE, not two) and
+`synchronous_commit = off`, this turns a ~1-day run into a few hours.
+
+`DROP INDEX` is near-instant — the cost people mean is the *rebuild*, which is why
+we do it once, at the end, in a low-usage window.
+
+## What the app needs to keep working
+
+Only the 3 secondary indexes. The half-filled `plan_id`/`material_id` columns are
+NULL-tolerant (the read paths already handle NULLs), and the *new*
+plan_id/material_id indexes aren't used by anything yet. So **at any stopping
+point, rebuild the 3 secondary indexes and the app is back to today's behaviour**
+— a 50%-done backfill is invisible to it.
+
+## Prerequisites
+
+- `npm run migration:migrate` has applied 0222/0224 (the column + `NOT VALID` FK adds).
+- `.env.local` points at the target DB (`DB_HOST`/`DB_PORT`/`DB_USERNAME`/`DB_PASSWORD`/`DB_NAME`).
+- A low-usage window.
+
+## Files
+
+- Script: `src/app/db/backfill-recommendation-denormalization.ts` (`npm run backfill:recommendation-denormalization`)
+- Drop:   `src/app/db/sql/drop-recommendation-backfill-indexes.sql`
+- Rebuild:`src/app/db/sql/rebuild-recommendation-backfill-indexes.sql`
+
+---
+
+## Procedure A — one window (a few hours, start to finish)
+
+```bash
+# 1. Drop the secondary indexes (app reads degrade to seq scans until step 4)
+psql "$DATABASE_URL" -f src/app/db/sql/drop-recommendation-backfill-indexes.sql
+
+# 2. Run the full backfill + finalize (builds the 2 new indexes, validates FKs)
+npm run backfill:recommendation-denormalization
+
+# 3. Rebuild the secondary indexes — app back to normal
+psql "$DATABASE_URL" -f src/app/db/sql/rebuild-recommendation-backfill-indexes.sql
+
+# 4. Reclaim the ~2x bloat in-place updates leave behind, refresh stats
+psql "$DATABASE_URL" -c "VACUUM (ANALYZE) recommendation;"
+```
+
+`DATABASE_URL` is illustrative — use whatever psql connection matches `.env.local`.
+
+---
+
+## Procedure B — split across windows (50% now, finish later)
+
+Each window is self-contained; between windows the app is 100% normal.
+
+**Every window except the last:**
+
+```bash
+psql "$DATABASE_URL" -f src/app/db/sql/drop-recommendation-backfill-indexes.sql
+BACKFILL_FINALIZE=false npm run backfill:recommendation-denormalization   # Ctrl-C any time; it's resumable
+psql "$DATABASE_URL" -f src/app/db/sql/rebuild-recommendation-backfill-indexes.sql
+```
+
+`BACKFILL_FINALIZE=false` skips the new-index build + FK validate — there's no
+point building the plan_id/material_id indexes until the data is all in.
+
+**The final window** (when the per-batch logs / the NULL summary show plan_id is
+down to ~0 and only genuinely material-less rows remain):
+
+```bash
+psql "$DATABASE_URL" -f src/app/db/sql/drop-recommendation-backfill-indexes.sql
+npm run backfill:recommendation-denormalization       # FINALIZE defaults on: builds new indexes, validates FKs
+psql "$DATABASE_URL" -f src/app/db/sql/rebuild-recommendation-backfill-indexes.sql
+psql "$DATABASE_URL" -c "VACUUM (ANALYZE) recommendation;"
+```
+
+Cost of splitting: you rebuild the secondary indexes once per window. Two of the
+three are partial (tiny), so each restore is effectively one full-size index
+build (`recommendation_property_id_idx`, a few minutes).
+
+### Interrupting mid-run
+
+Ctrl-C is safe. The script commits per batch, so completed batches persist. Re-run
+to continue — the `IS NULL` guards skip every row already done. If you're worried
+about app performance while stopped, rebuild the secondary indexes first.
+
+---
+
+## Alternative — no index juggling (simpler, slower)
+
+If you'd rather not touch indexes at all, just run the tuned single pass with the
+indexes left in place and stop/resume freely:
+
+```bash
+BACKFILL_FINALIZE=false npm run backfill:recommendation-denormalization
+```
+
+Slower per row (index write-amplification stays), but the app is always fully
+indexed and there's zero drop/rebuild ceremony. Finalize on the last run.
+
+---
+
+## Tunables (env)
+
+| Var | Default | Notes |
+| --- | --- | --- |
+| `BACKFILL_BATCH_SIZE` | `100000` | rows scanned per committed batch |
+| `BACKFILL_SLEEP_MS` | `0` | inter-batch pause; raise to throttle IO if not in a quiet window |
+| `BACKFILL_SYNC_COMMIT` | `off` | `on` to keep synchronous_commit (safe to leave off — run is resumable) |
+| `BACKFILL_MAINTENANCE_WORK_MEM` | `1GB` | memory for the finalize index builds |
+| `BACKFILL_WORK_MEM` | `128MB` | memory for the batch joins |
+| `BACKFILL_INDEX_CONCURRENTLY` | *(off)* | `true` builds the new indexes CONCURRENTLY (non-blocking, slower) |
+| `BACKFILL_FINALIZE` | *(on)* | `false` skips new-index build + FK validate (partial windows) |
+
+## Verifying completion
+
+The script prints remaining NULLs at the end:
+
+- `plan_id` NULL → should trend to ~0 (investigate any residue: recommendations with no `plan_recommendations` row).
+- `material_id` NULL → **expected** for recommendations that have no material; not an error.
+
+## After the backfill is fully done
+
+A later migration can `SET NOT NULL` on `plan_id` and drop the join tables, gated
+on this backfill having completed with zero unexpected NULLs. Out of scope here.
--- a/package.json
+++ b/package.json
@ -14,7 +14,10 @@
    "migration:generate": "drizzle-kit generate",
    "migration:migrate": "drizzle-kit migrate",
    "create_user": "tsx src/app/db/create_user.ts",
-    "backfill:recommendation-denormalization": "tsx src/app/db/backfill-recommendation-denormalization.ts"
+    "backfill:recommendation-denormalization": "tsx src/app/db/backfill-recommendation-denormalization.ts",
+    "backfill:recommendation-denormalization:pass": "BACKFILL_FINALIZE=false tsx src/app/db/backfill-recommendation-denormalization.ts",
+    "db:run-sql": "tsx src/app/db/run-sql.ts",
+    "backfill:recommendation-restore": "tsx src/app/db/run-sql.ts src/app/db/sql/rebuild-recommendation-backfill-indexes.sql src/app/db/sql/vacuum-recommendation.sql"
  },
  "dependencies": {
    "@aws-sdk/client-s3": "^3.971.0",
--- a/src/app/db/backfill-recommendation-denormalization.ts
+++ b/src/app/db/backfill-recommendation-denormalization.ts
@ -1,7 +1,8 @@
 /**
 * Backfills the denormalised plan_id / material_* columns on `recommendation`
- * from the plan_recommendations and recommendation_materials join tables, then
- * builds the supporting indexes and validates the foreign keys — all ONLINE.
+ * from the plan_recommendations and recommendation_materials join tables in a
+ * SINGLE keyset-paginated pass, then (optionally) builds the supporting indexes
+ * and validates the foreign keys — all ONLINE and resumable.
 *
 * Why this lives outside drizzle:
 *   drizzle-kit migrate wraps every pending migration in ONE transaction, so it
@ -14,21 +15,55 @@
 * Run AFTER `npm run migration:migrate` has applied 0222/0224 (the column adds):
 *   npm run backfill:recommendation-denormalization
 *
+ * ── Making it fast (see docs/runbooks/recommendation-denormalization-backfill.md)
+ * The dominant cost is index write-amplification: `recommendation` is packed
+ * (fillfactor 100), so each UPDATE can't do a HOT update and must insert into
+ * every index on the table for every one of ~21m rows. To go roughly an order
+ * of magnitude faster in a low-usage window:
+ *   1. DROP the 3 secondary indexes  (src/app/db/sql/drop-recommendation-backfill-indexes.sql)
+ *   2. run this script                (BACKFILL_FINALIZE=false for a partial/interrupted window)
+ *   3. REBUILD them after             (src/app/db/sql/rebuild-recommendation-backfill-indexes.sql)
+ * The backfill is resumable: stop any time, rebuild the indexes, and the app is
+ * back to today's behaviour (the half-filled columns are NULL-tolerant). Re-run
+ * to continue — the IS NULL guards never rewrite a row that's already done.
+ *
+ * This pass sets plan_id AND the four material_* columns in ONE write per row
+ * (the previous version did two full passes, rewriting every row twice).
+ *
 * Safe to re-run: every step is idempotent (IS NULL guards, IF NOT EXISTS,
- * VALIDATE on an already-valid constraint is a no-op). If interrupted, just run
- * it again — it resumes from wherever it left off.
+ * VALIDATE on an already-valid constraint is a no-op).
 *
 * Tunables via env:
- *   BACKFILL_BATCH_SIZE   rows scanned per committed batch (default 25000)
- *   BACKFILL_SLEEP_MS     pause between batches, eases IO pressure (default 50)
+ *   BACKFILL_BATCH_SIZE            rows scanned per committed batch (default 100000)
+ *   BACKFILL_SLEEP_MS             pause between batches, eases IO pressure (default 0)
+ *   BACKFILL_SYNC_COMMIT          'on' keeps synchronous_commit; default 'off' (faster
+ *                                 batch commits — safe here because the run is resumable)
+ *   BACKFILL_MAINTENANCE_WORK_MEM session maintenance_work_mem for index builds (default '1GB')
+ *   BACKFILL_WORK_MEM             session work_mem for the batch joins (default '128MB')
+ *   BACKFILL_INDEX_CONCURRENTLY   'true' builds the new indexes CONCURRENTLY (default plain
+ *                                 CREATE INDEX — faster, but briefly blocks writes)
+ *   BACKFILL_FINALIZE             'false' skips the index build + FK validate; use this for
+ *                                 partial/interrupted windows, finalize on the last one
 */
 import dotenv from "dotenv";
 import { Pool, type PoolClient } from "pg";

 dotenv.config({ path: ".env.local" });

-const BATCH_SIZE = Number(process.env.BACKFILL_BATCH_SIZE ?? 25_000);
-const SLEEP_MS = Number(process.env.BACKFILL_SLEEP_MS ?? 50);
+const BATCH_SIZE = Number(process.env.BACKFILL_BATCH_SIZE ?? 100_000);
+const SLEEP_MS = Number(process.env.BACKFILL_SLEEP_MS ?? 0);
+const SYNC_COMMIT = process.env.BACKFILL_SYNC_COMMIT === "on" ? "on" : "off";
+const MAINTENANCE_WORK_MEM = process.env.BACKFILL_MAINTENANCE_WORK_MEM ?? "1GB";
+const WORK_MEM = process.env.BACKFILL_WORK_MEM ?? "128MB";
+const INDEX_CONCURRENTLY = process.env.BACKFILL_INDEX_CONCURRENTLY === "true";
+const FINALIZE = process.env.BACKFILL_FINALIZE !== "false";
+
+// Range + phase controls, for running several workers over disjoint id ranges in
+// parallel (near-linear speed-up — the ranges never contend for the same rows).
+const START_ID = process.env.BACKFILL_START_ID ?? "0";
+const END_ID = process.env.BACKFILL_END_ID ?? null; // inclusive upper bound; null = unbounded
+const SKIP_CARDINALITY = process.env.BACKFILL_SKIP_CARDINALITY === "true";
+const ONLY_FINALIZE = process.env.BACKFILL_ONLY_FINALIZE === "true";

 const pool = new Pool({
  host: process.env.DB_HOST,
@ -65,28 +100,79 @@ async function assertSingleCardinality(

 /**
 * Keyset-paginate the whole `recommendation` table by id and, for each batch,
- * set the target columns from the join table where a match exists. Each batch
- * is its own autocommitted statement (no surrounding BEGIN), so locks are held
- * only for the rows in that batch.
+ * set plan_id and the four material_* columns from the join tables in a single
+ * UPDATE. Each batch is its own autocommitted statement (no surrounding BEGIN),
+ * so locks are held only for the rows in that batch.
 *
- * We scan by id (not `WHERE col IS NULL LIMIT n`) so rows with no match don't
- * get re-selected forever — they're simply left NULL and we move past them.
+ * We scan by id (not `WHERE col IS NULL LIMIT n`) so rows with no match — or a
+ * legitimately NULL material_id — don't get re-selected forever; they're left as
+ * they are and we move past them.
+ *
+ * The guard only touches a row when there is real work to do (a NULL column that
+ * has a matching source row), and COALESCE never overwrites an already-set value.
+ * So a row is written at most once, and a re-run after interruption skips every
+ * row already done — the pass is idempotent and resumable.
+ *
+ * The single-cardinality assertion above guarantees the LEFT JOINs cannot fan a
+ * row out into multiple update targets.
 */
-async function backfillColumns(
-  client: PoolClient,
-  label: string,
-  updateSql: (idFrom: string, limit: string) => string,
-): Promise<void> {
-  let lastId = "0";
+const fmtDuration = (secs: number): string => {
+  const s = Math.max(0, Math.round(secs));
+  const h = Math.floor(s / 3600);
+  const m = Math.floor((s % 3600) / 60);
+  return h > 0 ? `${h}h${m}m` : m > 0 ? `${m}m${s % 60}s` : `${s}s`;
+};
+
+async function backfillDenormalized(client: PoolClient): Promise<void> {
+  let lastId = START_ID;
  let scannedTotal = 0;
  let updatedTotal = 0;
  const startedAt = Date.now();

+  // Upper bound of the id keyset, for percent-complete + ETA. When END_ID is set
+  // (a parallel worker), use it; otherwise ask the table (fast, PK).
+  let maxId: number;
+  if (END_ID != null) {
+    maxId = Number(END_ID);
+  } else {
+    const { rows: maxRows } = await client.query(
+      `SELECT max(id) AS max_id FROM recommendation`,
+    );
+    maxId = Number(maxRows[0].max_id ?? 0);
+  }
+  console.log(
+    `Backfilling id ${Number(START_ID).toLocaleString()} → ${maxId.toLocaleString()}...`,
+  );
+
  for (;;) {
-    const { rows } = await client.query(updateSql("$1", "$2"), [
-      lastId,
-      BATCH_SIZE,
-    ]);
+    const { rows } = await client.query(
+      `WITH batch AS (
+         SELECT id FROM recommendation
+          WHERE id > $1
+            AND ($3::bigint IS NULL OR id <= $3::bigint)
+          ORDER BY id
+          LIMIT $2
+       ),
+       upd AS (
+         UPDATE recommendation r
+            SET plan_id                = COALESCE(r.plan_id, pr.plan_id),
+                material_id            = COALESCE(r.material_id, rm.material_id),
+                material_quantity      = COALESCE(r.material_quantity, rm.quantity),
+                material_quantity_unit = COALESCE(r.material_quantity_unit, rm.quantity_unit),
+                material_depth         = COALESCE(r.material_depth, rm.depth)
+           FROM batch b
+           LEFT JOIN plan_recommendations pr     ON pr.recommendation_id = b.id
+           LEFT JOIN recommendation_materials rm ON rm.recommendation_id = b.id
+          WHERE r.id = b.id
+            AND ( (r.plan_id     IS NULL AND pr.plan_id     IS NOT NULL)
+               OR (r.material_id IS NULL AND rm.material_id IS NOT NULL) )
+         RETURNING r.id
+       )
+       SELECT (SELECT max(id) FROM batch)  AS max_id,
+              (SELECT count(*) FROM batch) AS scanned,
+              (SELECT count(*) FROM upd)   AS updated`,
+      [lastId, BATCH_SIZE, END_ID],
+    );
    const { max_id, scanned, updated } = rows[0] as {
      max_id: string | null;
      scanned: number;
@ -99,17 +185,22 @@ async function backfillColumns(
    updatedTotal += Number(updated);
    lastId = max_id;

-    const rate = Math.round(scannedTotal / ((Date.now() - startedAt) / 1000));
+    const elapsed = (Date.now() - startedAt) / 1000;
+    const rate = Math.round(scannedTotal / elapsed);
+    const from = Number(START_ID);
+    const span = maxId - from;
+    const pct = span > 0 ? Math.min(100, ((Number(lastId) - from) / span) * 100) : 0;
+    const eta = pct > 0 ? fmtDuration((elapsed / pct) * (100 - pct)) : "?";
    console.log(
-      `[${label}] up to id ${lastId} — scanned ${scannedTotal.toLocaleString()}, ` +
-        `updated ${updatedTotal.toLocaleString()} (${rate.toLocaleString()} rows/s scan)`,
+      `${pct.toFixed(1)}% (id ${lastId}) — scanned ${scannedTotal.toLocaleString()}, ` +
+        `updated ${updatedTotal.toLocaleString()} — ${rate.toLocaleString()} rows/s, ETA ${eta}`,
    );

    if (SLEEP_MS > 0) await sleep(SLEEP_MS);
  }

  console.log(
-    `[${label}] done: scanned ${scannedTotal.toLocaleString()}, ` +
+    `Backfill pass done: scanned ${scannedTotal.toLocaleString()}, ` +
      `updated ${updatedTotal.toLocaleString()} in ` +
      `${Math.round((Date.now() - startedAt) / 1000)}s`,
  );
@ -119,91 +210,77 @@ async function main(): Promise<void> {
  const client = await pool.connect();
  try {
    console.log(
-      `Config: batch=${BATCH_SIZE.toLocaleString()} rows, sleep=${SLEEP_MS}ms\n`,
+      `Config: batch=${BATCH_SIZE.toLocaleString()} rows, sleep=${SLEEP_MS}ms, ` +
+        `synchronous_commit=${SYNC_COMMIT}, work_mem=${WORK_MEM}, ` +
+        `maintenance_work_mem=${MAINTENANCE_WORK_MEM}, finalize=${FINALIZE}, ` +
+        `range=(${Number(START_ID).toLocaleString()}, ${END_ID ?? "∞"}]` +
+        `${ONLY_FINALIZE ? ", ONLY_FINALIZE" : ""}\n`,
    );

-    // 1. Guard the 1:1 assumption before mutating anything.
-    console.log("Checking cardinality...");
-    await assertSingleCardinality(client, "plan_recommendations");
-    await assertSingleCardinality(client, "recommendation_materials");
+    // Session tuning. synchronous_commit=off is safe here: the backfill is
+    // idempotent and resumable, so a crash that loses the last few batch commits
+    // just means the next run continues from there. work_mem feeds the batch
+    // joins; maintenance_work_mem feeds the CREATE INDEX in the finalize step.
+    await client.query(`SET synchronous_commit = ${SYNC_COMMIT}`);
+    await client.query(`SET work_mem = '${WORK_MEM}'`);
+    await client.query(`SET maintenance_work_mem = '${MAINTENANCE_WORK_MEM}'`);

-    // 2. Backfill plan_id.
-    await backfillColumns(
-      client,
-      "plan_id",
-      (idFrom, limit) => `
-        WITH batch AS (
-          SELECT id FROM recommendation
-           WHERE id > ${idFrom}
-           ORDER BY id
-           LIMIT ${limit}
-        ),
-        upd AS (
-          UPDATE recommendation r
-             SET plan_id = pr.plan_id
-            FROM batch b
-            JOIN plan_recommendations pr ON pr.recommendation_id = b.id
-           WHERE r.id = b.id
-             AND r.plan_id IS NULL
-          RETURNING r.id
-        )
-        SELECT (SELECT max(id) FROM batch)   AS max_id,
-               (SELECT count(*) FROM batch)  AS scanned,
-               (SELECT count(*) FROM upd)    AS updated`,
-    );
+    if (ONLY_FINALIZE) {
+      // Parallel workflow: the data was written by range workers; this run just
+      // builds the indexes + validates the FKs. No cardinality check, no pass.
+      console.log("ONLY_FINALIZE: skipping cardinality check + backfill pass.");
+    } else {
+      // 1. Guard the 1:1 assumption before mutating anything. Skippable on
+      //    secondary parallel workers (one worker having checked is enough).
+      if (SKIP_CARDINALITY) {
+        console.log("Skipping cardinality check (BACKFILL_SKIP_CARDINALITY=true).");
+      } else {
+        console.log("Checking cardinality...");
+        await assertSingleCardinality(client, "plan_recommendations");
+        await assertSingleCardinality(client, "recommendation_materials");
+      }

-    // 3. Backfill the four material_* columns in one pass.
-    await backfillColumns(
-      client,
-      "material",
-      (idFrom, limit) => `
-        WITH batch AS (
-          SELECT id FROM recommendation
-           WHERE id > ${idFrom}
-           ORDER BY id
-           LIMIT ${limit}
-        ),
-        upd AS (
-          UPDATE recommendation r
-             SET material_id            = rm.material_id,
-                 material_quantity      = rm.quantity,
-                 material_quantity_unit = rm.quantity_unit,
-                 material_depth         = rm.depth
-            FROM batch b
-            JOIN recommendation_materials rm ON rm.recommendation_id = b.id
-           WHERE r.id = b.id
-             AND r.material_id IS NULL
-          RETURNING r.id
-        )
-        SELECT (SELECT max(id) FROM batch)   AS max_id,
-               (SELECT count(*) FROM batch)  AS scanned,
-               (SELECT count(*) FROM upd)    AS updated`,
-    );
+      // 2. Single combined pass: plan_id + material_* in one write per row.
+      console.log("Backfilling plan_id + material_* (single pass)...");
+      await backfillDenormalized(client);
+    }

-    // 4. Build indexes CONCURRENTLY (no write lock). Must NOT be in a txn —
-    //    a pooled client with no BEGIN runs each statement autocommitted.
-    console.log("Creating indexes concurrently...");
-    await client.query(
-      `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_recommendation_plan_id
-         ON recommendation USING btree (plan_id)`,
-    );
-    await client.query(
-      `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_recommendation_material_id
-         ON recommendation USING btree (material_id)`,
-    );
+    if (FINALIZE || ONLY_FINALIZE) {
+      // 3. Build the two new indexes. Plain CREATE INDEX is faster in a quiet
+      //    window; BACKFILL_INDEX_CONCURRENTLY=true avoids the brief write lock.
+      //    Autocommitted (no surrounding BEGIN), which CONCURRENTLY requires.
+      const cc = INDEX_CONCURRENTLY ? "CONCURRENTLY " : "";
+      console.log(
+        `Creating indexes (${INDEX_CONCURRENTLY ? "concurrently" : "plain"})...`,
+      );
+      await client.query(
+        `CREATE INDEX ${cc}IF NOT EXISTS idx_recommendation_plan_id
+           ON recommendation USING btree (plan_id)`,
+      );
+      await client.query(
+        `CREATE INDEX ${cc}IF NOT EXISTS idx_recommendation_material_id
+           ON recommendation USING btree (material_id)`,
+      );

-    // 5. Validate the FKs online (ShareUpdateExclusiveLock — allows reads/writes).
-    console.log("Validating foreign keys...");
-    await client.query(
-      `ALTER TABLE recommendation
-         VALIDATE CONSTRAINT recommendation_plan_id_plan_id_fk`,
-    );
-    await client.query(
-      `ALTER TABLE recommendation
-         VALIDATE CONSTRAINT recommendation_material_id_material_id_fk`,
-    );
+      // 4. Validate the FKs online (ShareUpdateExclusiveLock — allows reads/writes).
+      console.log("Validating foreign keys...");
+      await client.query(
+        `ALTER TABLE recommendation
+           VALIDATE CONSTRAINT recommendation_plan_id_plan_id_fk`,
+      );
+      await client.query(
+        `ALTER TABLE recommendation
+           VALIDATE CONSTRAINT recommendation_material_id_material_id_fk`,
+      );
+    } else {
+      console.log(
+        "Skipping index build + FK validate (BACKFILL_FINALIZE=false). " +
+          "Run once more with finalize enabled in your final window.",
+      );
+    }

-    // 6. Report any rows still unlinked (expected for material; investigate for plan).
+    // 5. Report any rows still unlinked. material_id NULL is expected (not every
+    //    recommendation has a material); plan_id NULL should trend toward ~0.
    const { rows } = await client.query(
      `SELECT count(*) FILTER (WHERE plan_id IS NULL)     AS plan_null,
              count(*) FILTER (WHERE material_id IS NULL) AS material_null
@ -213,7 +290,9 @@ async function main(): Promise<void> {
      `\nRemaining NULLs — plan_id: ${Number(rows[0].plan_null).toLocaleString()}, ` +
        `material_id: ${Number(rows[0].material_null).toLocaleString()}`,
    );
-    console.log("Backfill complete.");
+    console.log(
+      FINALIZE ? "Backfill complete." : "Backfill pass complete (not finalized).",
+    );
  } finally {
    client.release();
    await pool.end();
--- a/src/app/db/run-sql.ts
+++ b/src/app/db/run-sql.ts
@ -0,0 +1,68 @@
+/**
+ * Runs one or more .sql files against the DB from .env.local, executing each
+ * statement autocommitted (no wrapping transaction) so statements that cannot run
+ * inside a transaction block — VACUUM, CREATE INDEX CONCURRENTLY — work. For
+ * environments without the psql CLI.
+ *
+ *   tsx src/app/db/run-sql.ts <file.sql> [more.sql ...]
+ *
+ * Statement splitting is deliberately simple: it strips full-line `--` comments
+ * and splits on `;`. That's correct for the hand-written files in
+ * src/app/db/sql, but it is NOT a general SQL parser — don't point it at files
+ * with `;` inside string literals or dollar-quoted function bodies.
+ */
+import { readFileSync } from "fs";
+import dotenv from "dotenv";
+import { Pool } from "pg";
+
+dotenv.config({ path: ".env.local" });
+
+const files = process.argv.slice(2);
+if (files.length === 0) {
+  console.error("Usage: tsx src/app/db/run-sql.ts <file.sql> [more.sql ...]");
+  process.exit(1);
+}
+
+const pool = new Pool({
+  host: process.env.DB_HOST,
+  port: Number(process.env.DB_PORT),
+  user: process.env.DB_USERNAME,
+  password: process.env.DB_PASSWORD,
+  database: process.env.DB_NAME,
+});
+
+function statementsOf(path: string): string[] {
+  const raw = readFileSync(path, "utf8");
+  return raw
+    .split("\n")
+    .filter((line) => !line.trim().startsWith("--"))
+    .join("\n")
+    .split(";")
+    .map((s) => s.trim())
+    .filter((s) => s.length > 0);
+}
+
+async function main(): Promise<void> {
+  const client = await pool.connect();
+  try {
+    for (const file of files) {
+      console.log(`\n=== ${file} ===`);
+      for (const sql of statementsOf(file)) {
+        const preview = sql.replace(/\s+/g, " ").slice(0, 70);
+        const startedAt = Date.now();
+        console.log(`> ${preview}${sql.length > 70 ? "…" : ""}`);
+        await client.query(sql);
+        console.log(`  ok (${Math.round((Date.now() - startedAt) / 1000)}s)`);
+      }
+    }
+    console.log("\nAll statements complete.");
+  } finally {
+    client.release();
+    await pool.end();
+  }
+}
+
+main().catch((err) => {
+  console.error("SQL run failed:", err);
+  process.exit(1);
+});
--- a/src/app/db/sql/drop-recommendation-backfill-indexes.sql
+++ b/src/app/db/sql/drop-recommendation-backfill-indexes.sql
@ -0,0 +1,26 @@
+-- Drop the 3 secondary indexes on `recommendation` so the denormalisation
+-- backfill can run without per-row index write-amplification.
+--
+-- `recommendation` is packed (fillfactor 100), so each backfill UPDATE cannot do
+-- a HOT update and must insert a new entry into every index on the table for all
+-- ~21m rows. Dropping these first cuts index inserts per updated row from ~4 to 1
+-- (just the primary key), which is the bulk of the speed-up.
+--
+-- These serve production reads (fetch-by-property, the active-defaults paths), so
+-- ONLY run this in a low-usage window and rebuild them
+-- (rebuild-recommendation-backfill-indexes.sql) as soon as the window's backfill
+-- pass finishes. DROP INDEX is near-instant (catalog change + file unlink); it
+-- takes a brief ACCESS EXCLUSIVE lock, so if long-running queries are in flight
+-- use `DROP INDEX CONCURRENTLY` (cannot run inside a transaction) instead.
+--
+-- Kept OUT of the backfill script on purpose so you control exactly when the app
+-- loses and regains these indexes. See
+-- docs/runbooks/recommendation-denormalization-backfill.md
+--
+-- NOTE: idx_recommendation_active_id_property was originally built CONCURRENTLY;
+-- if a prior CONCURRENTLY build was ever interrupted it can linger as INVALID —
+-- the DROP below clears it either way.
+
+DROP INDEX IF EXISTS recommendation_property_id_idx;
+DROP INDEX IF EXISTS idx_recommendation_active_defaults;
+DROP INDEX IF EXISTS idx_recommendation_active_id_property;
--- a/src/app/db/sql/rebuild-recommendation-backfill-indexes.sql
+++ b/src/app/db/sql/rebuild-recommendation-backfill-indexes.sql
@ -0,0 +1,31 @@
+-- Rebuild the 3 secondary indexes on `recommendation` after a backfill window,
+-- restoring the app to its normal (pre-backfill) query behaviour.
+--
+-- The definitions below reproduce exactly what the migrations created (partial
+-- WHERE clauses included). Run any time — the backfill does NOT need to be
+-- complete first: these indexes don't reference plan_id/material_id, so a
+-- half-finished backfill is invisible to them and to the app.
+--
+-- Two of the three are PARTIAL (only rows where default = true and not yet
+-- installed), so they cover a small slice of the table and rebuild in seconds;
+-- only recommendation_property_id_idx spans all rows and takes a few minutes.
+--
+-- Plain CREATE INDEX is fastest but briefly blocks writes while each builds. If
+-- you need writes to keep flowing, replace CREATE INDEX with
+-- CREATE INDEX CONCURRENTLY (slower, cannot run inside a transaction, and each
+-- statement must be sent on its own).
+
+-- 1GB comfortably holds the ~500MB sort for a 21M-row single-column btree.
+-- Keep it well under instance RAM (this DB is 4GB) to avoid memory pressure.
+SET maintenance_work_mem = '1GB';
+
+CREATE INDEX IF NOT EXISTS recommendation_property_id_idx
+  ON recommendation USING btree (property_id);
+
+CREATE INDEX IF NOT EXISTS idx_recommendation_active_defaults
+  ON recommendation USING btree (id)
+  WHERE "default" = true AND already_installed = false;
+
+CREATE INDEX IF NOT EXISTS idx_recommendation_active_id_property
+  ON recommendation USING btree (id, property_id)
+  WHERE "default" = true AND already_installed = false;
--- a/src/app/db/sql/vacuum-recommendation.sql
+++ b/src/app/db/sql/vacuum-recommendation.sql
@ -0,0 +1,7 @@
+-- Reclaim the dead tuples the backfill left behind (it updated ~every row, so the
+-- table is ~2x bloated) and refresh planner stats so queries pick up the new
+-- indexes. Standard VACUUM (not FULL) — online, no exclusive lock, but it does a
+-- full heap + index scan so it can take a while. No progress output; watch
+-- pg_stat_progress_vacuum in another session.
+
+VACUUM (ANALYZE) recommendation;