diff --git a/CONTEXT.md b/CONTEXT.md
index c2073354..cdd69ab7 100644
--- a/CONTEXT.md
+++ b/CONTEXT.md
@@ -124,6 +124,14 @@ _Avoid_: current performance, adjusted performance
The act of substituting a corrected set of performance metrics in place of the Lodged values. Recorded on `property_baseline_performance` with a `rebaseline_reason` enum value: `none`, `pre_sap10`, `physical_state_changed`, or `both`.
_Avoid_: override, adjustment, correction
+**EPC provenance** (`epc_property.source`):
+Whether a property's EPC picture is a real certificate or a gap-fill. `lodged` = a real public/landlord EPC exists; `predicted` = no certificate, so the EPC was estimated from nearby properties (EPC Prediction gap-fill). Independent of Rebaseline: a `lodged` property may still be rebaselined, and a `predicted` property still carries Lodged-performance figures (mirrored estimates), so the presence of `lodged_*` columns does **not** imply a real certificate — only `source = lodged` does.
+_Avoid_: estimated EPC (reserve "estimated" for the UI signal), source EPC
+
+**Provenance signal** (UI):
+What the user is told about an EPC's trustworthiness, derived from EPC provenance and Rebaseline together: **Estimated** (`source = predicted` — dominant; "estimated based on nearby homes") › **Re-modelled** (`source = lodged AND rebaseline_reason != none` — effective diverged from the lodged certificate under SAP 10) › **none** (`source = lodged AND rebaseline_reason = none` — effective equals lodged). Estimated always wins when both could apply.
+_Avoid_: estimation notification, banner (those are component names)
+
## Example dialogue
> **Dev:** "If the **Combiner** finishes but the user hasn't clicked Finalise, what does the user see?"
@@ -136,5 +144,6 @@ _Avoid_: override, adjustment, correction
## Flagged ambiguities
+- The UI surfaces labelled **"Current EPC"** and **"Current Efficiency State"** (property table, building-passport card) show **Effective performance**, not Lodged — despite the glossary advising against "current performance" for Effective. The label is a product choice ("current" = the property's true present-day modelled state); the underlying datum is still **Effective performance**. The **"Lodged EPC"** column/badge is the only surface showing **Lodged performance**, and only when a real certificate exists (`source = lodged`).
- "Upload" is used in the codebase to mean both the file-on-S3 and the BulkUpload row. We standardise on **BulkUpload** for the row; the file is just "the source file."
- "Onboarding" appears in some route paths (`bulk_onboarding_inputs/...`) but isn't part of this glossary — we use **BulkUpload** end-to-end.
diff --git a/docs/runbooks/recommendation-denormalization-backfill.md b/docs/runbooks/recommendation-denormalization-backfill.md
new file mode 100644
index 00000000..916ff601
--- /dev/null
+++ b/docs/runbooks/recommendation-denormalization-backfill.md
@@ -0,0 +1,140 @@
+# Runbook: `recommendation` denormalisation backfill
+
+Populates `recommendation.plan_id` and `recommendation.material_*` from the
+`plan_recommendations` / `recommendation_materials` join tables, as the first
+step toward making those columns the source of truth and dropping the join
+tables. See [ADR 0001](../adr/0001-data-backfills-outside-drizzle.md) for why
+this runs outside drizzle.
+
+The table is ~21m rows. This runbook is the **fast** procedure.
+
+## Why it's slow by default, and the lever
+
+`recommendation` is packed (fillfactor 100), so an UPDATE can't do a HOT update —
+Postgres writes a new row version and inserts into **every index on the table**
+for all ~21m rows. That index write-amplification (and its WAL) is what drains IO.
+
+The lever: **drop the secondary indexes, backfill, rebuild them.** With them gone,
+each updated row only maintains the primary key (~4 index inserts → 1). Combined
+with the single-pass rewrite (plan_id + material in one UPDATE, not two) and
+`synchronous_commit = off`, this turns a ~1-day run into a few hours.
+
+`DROP INDEX` is near-instant — the cost people mean is the *rebuild*, which is why
+we do it once, at the end, in a low-usage window.
+
+## What the app needs to keep working
+
+Only the 3 secondary indexes. The half-filled `plan_id`/`material_id` columns are
+NULL-tolerant (the read paths already handle NULLs), and the *new*
+plan_id/material_id indexes aren't used by anything yet. So **at any stopping
+point, rebuild the 3 secondary indexes and the app is back to today's behaviour**
+— a 50%-done backfill is invisible to it.
+
+## Prerequisites
+
+- `npm run migration:migrate` has applied 0222/0224 (the column + `NOT VALID` FK adds).
+- `.env.local` points at the target DB (`DB_HOST`/`DB_PORT`/`DB_USERNAME`/`DB_PASSWORD`/`DB_NAME`).
+- A low-usage window.
+
+## Files
+
+- Script: `src/app/db/backfill-recommendation-denormalization.ts` (`npm run backfill:recommendation-denormalization`)
+- Drop: `src/app/db/sql/drop-recommendation-backfill-indexes.sql`
+- Rebuild:`src/app/db/sql/rebuild-recommendation-backfill-indexes.sql`
+
+---
+
+## Procedure A — one window (a few hours, start to finish)
+
+```bash
+# 1. Drop the secondary indexes (app reads degrade to seq scans until step 4)
+psql "$DATABASE_URL" -f src/app/db/sql/drop-recommendation-backfill-indexes.sql
+
+# 2. Run the full backfill + finalize (builds the 2 new indexes, validates FKs)
+npm run backfill:recommendation-denormalization
+
+# 3. Rebuild the secondary indexes — app back to normal
+psql "$DATABASE_URL" -f src/app/db/sql/rebuild-recommendation-backfill-indexes.sql
+
+# 4. Reclaim the ~2x bloat in-place updates leave behind, refresh stats
+psql "$DATABASE_URL" -c "VACUUM (ANALYZE) recommendation;"
+```
+
+`DATABASE_URL` is illustrative — use whatever psql connection matches `.env.local`.
+
+---
+
+## Procedure B — split across windows (50% now, finish later)
+
+Each window is self-contained; between windows the app is 100% normal.
+
+**Every window except the last:**
+
+```bash
+psql "$DATABASE_URL" -f src/app/db/sql/drop-recommendation-backfill-indexes.sql
+BACKFILL_FINALIZE=false npm run backfill:recommendation-denormalization # Ctrl-C any time; it's resumable
+psql "$DATABASE_URL" -f src/app/db/sql/rebuild-recommendation-backfill-indexes.sql
+```
+
+`BACKFILL_FINALIZE=false` skips the new-index build + FK validate — there's no
+point building the plan_id/material_id indexes until the data is all in.
+
+**The final window** (when the per-batch logs / the NULL summary show plan_id is
+down to ~0 and only genuinely material-less rows remain):
+
+```bash
+psql "$DATABASE_URL" -f src/app/db/sql/drop-recommendation-backfill-indexes.sql
+npm run backfill:recommendation-denormalization # FINALIZE defaults on: builds new indexes, validates FKs
+psql "$DATABASE_URL" -f src/app/db/sql/rebuild-recommendation-backfill-indexes.sql
+psql "$DATABASE_URL" -c "VACUUM (ANALYZE) recommendation;"
+```
+
+Cost of splitting: you rebuild the secondary indexes once per window. Two of the
+three are partial (tiny), so each restore is effectively one full-size index
+build (`recommendation_property_id_idx`, a few minutes).
+
+### Interrupting mid-run
+
+Ctrl-C is safe. The script commits per batch, so completed batches persist. Re-run
+to continue — the `IS NULL` guards skip every row already done. If you're worried
+about app performance while stopped, rebuild the secondary indexes first.
+
+---
+
+## Alternative — no index juggling (simpler, slower)
+
+If you'd rather not touch indexes at all, just run the tuned single pass with the
+indexes left in place and stop/resume freely:
+
+```bash
+BACKFILL_FINALIZE=false npm run backfill:recommendation-denormalization
+```
+
+Slower per row (index write-amplification stays), but the app is always fully
+indexed and there's zero drop/rebuild ceremony. Finalize on the last run.
+
+---
+
+## Tunables (env)
+
+| Var | Default | Notes |
+| --- | --- | --- |
+| `BACKFILL_BATCH_SIZE` | `100000` | rows scanned per committed batch |
+| `BACKFILL_SLEEP_MS` | `0` | inter-batch pause; raise to throttle IO if not in a quiet window |
+| `BACKFILL_SYNC_COMMIT` | `off` | `on` to keep synchronous_commit (safe to leave off — run is resumable) |
+| `BACKFILL_MAINTENANCE_WORK_MEM` | `1GB` | memory for the finalize index builds |
+| `BACKFILL_WORK_MEM` | `128MB` | memory for the batch joins |
+| `BACKFILL_INDEX_CONCURRENTLY` | *(off)* | `true` builds the new indexes CONCURRENTLY (non-blocking, slower) |
+| `BACKFILL_FINALIZE` | *(on)* | `false` skips new-index build + FK validate (partial windows) |
+
+## Verifying completion
+
+The script prints remaining NULLs at the end:
+
+- `plan_id` NULL → should trend to ~0 (investigate any residue: recommendations with no `plan_recommendations` row).
+- `material_id` NULL → **expected** for recommendations that have no material; not an error.
+
+## After the backfill is fully done
+
+A later migration can `SET NOT NULL` on `plan_id` and drop the join tables, gated
+on this backfill having completed with zero unexpected NULLs. Out of scope here.
diff --git a/package.json b/package.json
index e205a7e5..92fda474 100644
--- a/package.json
+++ b/package.json
@@ -14,7 +14,10 @@
"migration:generate": "drizzle-kit generate",
"migration:migrate": "drizzle-kit migrate",
"create_user": "tsx src/app/db/create_user.ts",
- "backfill:recommendation-denormalization": "tsx src/app/db/backfill-recommendation-denormalization.ts"
+ "backfill:recommendation-denormalization": "tsx src/app/db/backfill-recommendation-denormalization.ts",
+ "backfill:recommendation-denormalization:pass": "BACKFILL_FINALIZE=false tsx src/app/db/backfill-recommendation-denormalization.ts",
+ "db:run-sql": "tsx src/app/db/run-sql.ts",
+ "backfill:recommendation-restore": "tsx src/app/db/run-sql.ts src/app/db/sql/rebuild-recommendation-backfill-indexes.sql src/app/db/sql/vacuum-recommendation.sql"
},
"dependencies": {
"@aws-sdk/client-s3": "^3.971.0",
diff --git a/src/app/api/portfolio/[portfolioId]/scenario/[scenarioId]/measures/route.ts b/src/app/api/portfolio/[portfolioId]/scenario/[scenarioId]/measures/route.ts
index dccf47d9..21aaed37 100644
--- a/src/app/api/portfolio/[portfolioId]/scenario/[scenarioId]/measures/route.ts
+++ b/src/app/api/portfolio/[portfolioId]/scenario/[scenarioId]/measures/route.ts
@@ -20,62 +20,11 @@ export async function GET(
const pid = BigInt(portfolioId);
const sid = BigInt(scenarioId);
- // TEMP: Remove batteries as underspecified
- // const result = await db.execute(sql`
- // WITH latest_plans AS (
- // SELECT DISTINCT ON (property_id)
- // *
- // FROM plan
- // WHERE portfolio_id = ${pid}
- // AND scenario_id = ${sid}
- // ORDER BY property_id, created_at DESC
- // ),
-
- // recommendation_flags AS (
- // SELECT
- // r.id AS recommendation_id,
- // r.measure_type AS measure_type,
- // r.property_id AS property_id,
- // r.estimated_cost AS estimated_cost,
- // BOOL_OR(m.includes_battery) AS includes_battery
-
- // FROM latest_plans lp
- // JOIN plan_recommendations pr
- // ON pr.plan_id = lp.id
- // JOIN recommendation r
- // ON r.id = pr.recommendation_id
-
- // LEFT JOIN recommendation_materials rm
- // ON rm.recommendation_id = r.id
- // LEFT JOIN material m
- // ON m.id = rm.material_id
- // AND m.is_active = true
-
- // WHERE r.default = true
- // AND r.already_installed = false
-
- // GROUP BY
- // r.id,
- // r.measure_type,
- // r.property_id,
- // r.estimated_cost
- // )
-
- // SELECT
- // measure_type,
- // COALESCE(includes_battery, false) AS includes_battery,
-
- // COUNT(DISTINCT property_id)::int AS homes_count,
- // SUM(estimated_cost)::float AS total_cost,
- // AVG(estimated_cost)::float AS average_cost
-
- // FROM recommendation_flags
- // GROUP BY
- // measure_type,
- // includes_battery
- // ORDER BY total_cost DESC;
- // `);
-
+ // Latest plan per property for this scenario, then its recommendations read
+ // through the DENORMALISED link: join by the indexed recommendation.property_id
+ // and scope to the plan via recommendation.plan_id. The plan_recommendations
+ // join table is retired (no rows for new-approach plans), so the old EXISTS
+ // against it returned zero measures. See the handover / ADR notes.
const result = await db.execute(sql`
SELECT
r.measure_type,
@@ -83,23 +32,19 @@ export async function GET(
COUNT(DISTINCT r.property_id)::int AS homes_count,
SUM(r.estimated_cost)::float AS total_cost,
AVG(r.estimated_cost)::float AS average_cost
- FROM recommendation r
- WHERE r.default = true
+ FROM (
+ SELECT DISTINCT ON (property_id)
+ id, property_id
+ FROM plan
+ WHERE portfolio_id = ${pid}
+ AND scenario_id = ${sid}
+ ORDER BY property_id, created_at DESC
+ ) lp
+ JOIN recommendation r
+ ON r.property_id = lp.property_id
+ AND r.plan_id = lp.id
+ AND r.default = true
AND r.already_installed = false
- AND EXISTS (
- SELECT 1
- FROM (
- SELECT DISTINCT ON (p.property_id)
- p.id
- FROM plan p
- WHERE p.portfolio_id = ${pid}
- AND p.scenario_id = ${sid}
- ORDER BY p.property_id, p.created_at DESC
- ) lp
- JOIN plan_recommendations pr
- ON pr.plan_id = lp.id
- WHERE pr.recommendation_id = r.id
- )
GROUP BY
r.measure_type,
r.type
diff --git a/src/app/api/portfolio/[portfolioId]/scenario/[scenarioId]/metrics/route.ts b/src/app/api/portfolio/[portfolioId]/scenario/[scenarioId]/metrics/route.ts
index 8fc292b6..d56943e8 100644
--- a/src/app/api/portfolio/[portfolioId]/scenario/[scenarioId]/metrics/route.ts
+++ b/src/app/api/portfolio/[portfolioId]/scenario/[scenarioId]/metrics/route.ts
@@ -3,6 +3,12 @@ import { sql } from "drizzle-orm";
import { NextRequest, NextResponse } from "next/server";
import { sapToEpc } from "@/app/utils";
import type { PortfolioGoalType } from "@/app/db/schema/portfolio";
+import {
+ newApproachJoins,
+ carbonSql,
+ billsSql,
+ effectiveSapSql,
+} from "@/lib/services/epcSources";
/* =======================
Types
@@ -124,14 +130,18 @@ export async function GET(
SUM(post_energy_bill)::float AS total_bills,
SUM(
CASE
+ -- TEMP (demo): baseline is the effective SAP, not p.current_sap_points
+ -- (NULL for new-approach → uplift summed to 0 → £/SAP showed 0). Count
+ -- genuine gains only (post > baseline); sub-baseline plans excluded.
WHEN cost_of_works > 0
- AND post_sap_points IS NOT NULL
- THEN post_sap_points - p.current_sap_points
+ AND post_sap_points > (${effectiveSapSql})
+ THEN post_sap_points - (${effectiveSapSql})
ELSE 0
END
)::float AS total_sap_uplift
FROM latest_plans lp
JOIN property p ON p.id = lp.property_id
+ LEFT JOIN property_baseline_performance bp ON bp.property_id = p.id
-- Conditional filter: only restrict by original_sap_points when the toggle is on
-- AND the scenario has an EPC target. Written as an OR chain so Postgres evaluates
-- it as a single WHERE clause — avoiding the need to dynamically build the query
@@ -176,8 +186,13 @@ export async function GET(
)::float AS total_funding
FROM latest_plans lp
JOIN property p ON p.id = lp.property_id
+ LEFT JOIN property_baseline_performance bp ON bp.property_id = p.id
LEFT JOIN funding_package fp ON fp.plan_id = lp.id
WHERE lp.cost_of_works > 0
+ -- TEMP (demo): exclude plans whose post SAP is below the effective baseline
+ -- (target-level post_sap on already-compliant homes) — not real upgrades.
+ -- COALESCE keeps rows we can't compare. See ADR-0002.
+ AND COALESCE(lp.post_sap_points >= (${effectiveSapSql}), true)
AND (
${useOriginalBaseline} = false
OR ${minSap}::float IS NULL
@@ -208,26 +223,19 @@ export async function GET(
/* ---------- Carbon ---------- */
CASE
WHEN lp.id IS NOT NULL THEN lp.post_co2_emissions
- ELSE e.co2_emissions
+ ELSE ${carbonSql(sql`e`)}
END AS effective_carbon,
/* ---------- Bills ---------- */
CASE
WHEN lp.id IS NOT NULL THEN lp.post_energy_bill
- ELSE (
- e.heating_cost_current +
- e.hot_water_cost_current +
- e.lighting_cost_current +
- e.appliances_cost_current +
- e.gas_standing_charge +
- e.electricity_standing_charge -
- COALESCE(e.installed_measures_total_energy_bill_adjustment, 0)
- )
+ ELSE ${billsSql(sql`e`)}
END AS effective_bills
FROM property p
LEFT JOIN property_details_epc e
ON e.property_id = p.id
+ ${newApproachJoins}
LEFT JOIN LATERAL (
SELECT *
@@ -263,10 +271,18 @@ export async function GET(
const epcRows = await db.execute(sql`
SELECT
CASE
- WHEN lp.id IS NOT NULL THEN lp.post_sap_points
- ELSE p.current_sap_points
+ -- A retrofit scenario can't make a property worse. The engine writes a
+ -- target-level post_sap (e.g. ~C) even for properties already above the
+ -- target, so post_sap can sit BELOW the baseline — which made the chart
+ -- show e.g. B properties "improving" down to C. Clamp to the baseline.
+ WHEN lp.id IS NOT NULL THEN GREATEST(${effectiveSapSql}, lp.post_sap_points)
+ -- No qualifying plan → unchanged property. Use the effective
+ -- (re-baselined) baseline to match the "before" distribution; NOT
+ -- p.current_sap_points (NULL for new-approach → "Unknown"). See ADR-0002.
+ ELSE ${effectiveSapSql}
END AS effective_sap
FROM property p
+ LEFT JOIN property_baseline_performance bp ON bp.property_id = p.id
LEFT JOIN LATERAL (
SELECT *
FROM plan
diff --git a/src/app/api/portfolio/[portfolioId]/scenario/default/measures/route.ts b/src/app/api/portfolio/[portfolioId]/scenario/default/measures/route.ts
index 1e3d2bc1..3640c773 100644
--- a/src/app/api/portfolio/[portfolioId]/scenario/default/measures/route.ts
+++ b/src/app/api/portfolio/[portfolioId]/scenario/default/measures/route.ts
@@ -19,6 +19,11 @@ export async function GET(
const pid = BigInt(portfolioId);
+ // Latest default plan per property, then its recommendations read through the
+ // DENORMALISED link: join by the indexed recommendation.property_id and scope
+ // to the plan via recommendation.plan_id. The plan_recommendations join table
+ // is retired (no rows for new-approach plans), so the old EXISTS against it
+ // returned zero measures.
const result = await db.execute(sql`
SELECT
r.measure_type,
@@ -26,23 +31,19 @@ export async function GET(
COUNT(DISTINCT r.property_id)::int AS homes_count,
SUM(r.estimated_cost)::float AS total_cost,
AVG(r.estimated_cost)::float AS average_cost
- FROM recommendation r
- WHERE r.default = true
+ FROM (
+ SELECT DISTINCT ON (property_id)
+ id, property_id
+ FROM plan
+ WHERE portfolio_id = ${pid}
+ AND is_default = true
+ ORDER BY property_id, created_at DESC
+ ) lp
+ JOIN recommendation r
+ ON r.property_id = lp.property_id
+ AND r.plan_id = lp.id
+ AND r.default = true
AND r.already_installed = false
- AND EXISTS (
- SELECT 1
- FROM (
- SELECT DISTINCT ON (p.property_id)
- p.id
- FROM plan p
- WHERE p.portfolio_id = ${pid}
- AND p.is_default = true
- ORDER BY p.property_id, p.created_at DESC
- ) lp
- JOIN plan_recommendations pr
- ON pr.plan_id = lp.id
- WHERE pr.recommendation_id = r.id
- )
GROUP BY
r.measure_type,
r.type
diff --git a/src/app/api/portfolio/[portfolioId]/scenario/default/metrics/route.ts b/src/app/api/portfolio/[portfolioId]/scenario/default/metrics/route.ts
index 6d5df36b..b5f56284 100644
--- a/src/app/api/portfolio/[portfolioId]/scenario/default/metrics/route.ts
+++ b/src/app/api/portfolio/[portfolioId]/scenario/default/metrics/route.ts
@@ -3,6 +3,12 @@ import { sql } from "drizzle-orm";
import { NextRequest, NextResponse } from "next/server";
import { sapToEpc } from "@/app/utils";
import type { PortfolioGoalType } from "@/app/db/schema/portfolio";
+import {
+ newApproachJoins,
+ carbonSql,
+ billsSql,
+ effectiveSapSql,
+} from "@/lib/services/epcSources";
/* =======================
Types
@@ -90,14 +96,19 @@ export async function GET(
SUM(post_energy_bill)::float AS total_bills,
SUM(
CASE
+ -- TEMP (demo): baseline is the effective SAP, not p.current_sap_points
+ -- (NULL for new-approach → uplift summed to 0 → £/SAP showed 0). Count
+ -- genuine gains only (post > baseline); sub-baseline "downgrade" plans
+ -- are excluded rather than dragging the uplift negative. See ADR-0002.
WHEN cost_of_works > 0
- AND post_sap_points IS NOT NULL
- THEN post_sap_points - p.current_sap_points
+ AND post_sap_points > (${effectiveSapSql})
+ THEN post_sap_points - (${effectiveSapSql})
ELSE 0
END
)::float AS total_sap_uplift
FROM latest_plans lp
- JOIN property p ON p.id = lp.property_id;
+ JOIN property p ON p.id = lp.property_id
+ LEFT JOIN property_baseline_performance bp ON bp.property_id = p.id;
`);
const scenarioAgg = scenarioMetricsResult.rows[0] as ScenarioAggregates;
@@ -123,8 +134,15 @@ export async function GET(
COALESCE(fp.total_uplift, 0)
)::float AS total_funding
FROM latest_plans lp
+ JOIN property p ON p.id = lp.property_id
+ LEFT JOIN property_baseline_performance bp ON bp.property_id = p.id
LEFT JOIN funding_package fp ON fp.plan_id = lp.id
- WHERE lp.cost_of_works > 0;
+ WHERE lp.cost_of_works > 0
+ -- TEMP (demo): a plan whose post SAP is below the effective baseline isn't
+ -- a real upgrade (target-level post_sap on already-compliant homes), so
+ -- exclude it from the count + cost. COALESCE keeps rows we can't compare
+ -- (NULL post or NULL baseline). See ADR-0002.
+ AND COALESCE(lp.post_sap_points >= (${effectiveSapSql}), true);
`);
const upgraded = upgradedResult.rows[0] as UpgradedAggregates;
@@ -150,26 +168,19 @@ export async function GET(
/* ---------- Carbon ---------- */
CASE
WHEN lp.id IS NOT NULL THEN lp.post_co2_emissions
- ELSE e.co2_emissions
+ ELSE ${carbonSql(sql`e`)}
END AS effective_carbon,
/* ---------- Bills ---------- */
CASE
WHEN lp.id IS NOT NULL THEN lp.post_energy_bill
- ELSE (
- e.heating_cost_current +
- e.hot_water_cost_current +
- e.lighting_cost_current +
- e.appliances_cost_current +
- e.gas_standing_charge +
- e.electricity_standing_charge -
- COALESCE(e.installed_measures_total_energy_bill_adjustment, 0)
- )
+ ELSE ${billsSql(sql`e`)}
END AS effective_bills
FROM property p
LEFT JOIN property_details_epc e
ON e.property_id = p.id
+ ${newApproachJoins}
LEFT JOIN LATERAL (
SELECT *
@@ -193,10 +204,19 @@ export async function GET(
const epcRows = await db.execute(sql`
SELECT
CASE
- WHEN lp.id IS NOT NULL THEN lp.post_sap_points
- ELSE p.current_sap_points
+ -- A retrofit scenario can't make a property worse. The engine writes a
+ -- target-level post_sap (e.g. ~C) even for properties already above the
+ -- target, so post_sap can sit BELOW the baseline — which made the chart
+ -- show e.g. B properties "improving" down to C. Clamp to the baseline so
+ -- the post-scenario band is never worse than before.
+ WHEN lp.id IS NOT NULL THEN GREATEST(${effectiveSapSql}, lp.post_sap_points)
+ -- No plan → unchanged property. Use the effective (re-baselined) baseline
+ -- so this matches the "before" distribution — NOT p.current_sap_points,
+ -- which is NULL for new-approach properties. See ADR-0002.
+ ELSE ${effectiveSapSql}
END AS effective_sap
FROM property p
+ LEFT JOIN property_baseline_performance bp ON bp.property_id = p.id
LEFT JOIN LATERAL (
SELECT *
FROM plan
diff --git a/src/app/api/property-meta/[propertyId]/route.ts b/src/app/api/property-meta/[propertyId]/route.ts
index 81f47ae7..84ccc2c9 100644
--- a/src/app/api/property-meta/[propertyId]/route.ts
+++ b/src/app/api/property-meta/[propertyId]/route.ts
@@ -1,46 +1,16 @@
-import { db } from "@/app/db/db";
-import { property } from "@/app/db/schema/property";
import { serializeBigInt } from "@/app/utils";
-import { eq } from "drizzle-orm";
+import { resolvePropertyMeta } from "@/lib/services/epcSources";
import { NextRequest, NextResponse } from "next/server";
export async function GET(request: NextRequest, props: { params: Promise<{ propertyId: string }> }) {
const params = await props.params;
const propertyId = params.propertyId;
- const propertyMeta = await db.query.property.findFirst({
- columns: {
- id: true,
- uprn: true,
- address: true,
- postcode: true,
- hasPreConditionReport: true,
- hasRecommendations: true,
- createdAt: true,
- propertyType: true,
- builtForm: true,
- localAuthority: true,
- constituency: true,
- numberOfRooms: true,
- yearBuilt: true,
- tenure: true,
- currentEpcRating: true,
- currentSapPoints: true,
- originalSapPoints: true,
- updatedAt: true,
- currentValuation: true,
- },
- where: eq(property.id, BigInt(propertyId)),
- with: {
- detailsEpc: {
- columns: {
- currentEnergyDemand: true,
- co2Emissions: true,
- estimated: true,
- },
- },
- },
- });
+ // `detailsEpc` and the headline SAP/EPC are resolved via the cutoff: legacy
+ // properties read property_details_epc / the property row; new-approach ones
+ // read property_baseline_performance (the property-row headline columns aren't
+ // written yet). See src/lib/services/epcSources.ts.
+ const propertyMeta = await resolvePropertyMeta(propertyId);
if (!propertyMeta) {
return new NextResponse(null, { status: 404 });
diff --git a/src/app/components/building-passport/CurrentEfficiencyCard.tsx b/src/app/components/building-passport/CurrentEfficiencyCard.tsx
index 723cb630..7aad49fc 100644
--- a/src/app/components/building-passport/CurrentEfficiencyCard.tsx
+++ b/src/app/components/building-passport/CurrentEfficiencyCard.tsx
@@ -5,6 +5,9 @@ import { LodgedEpcTooltip } from "./LodgedEpcTooltip";
interface CurrentEfficiencyCardProps {
epcRating: string | null;
sapPoints: number;
+ provenanceSignal?: "estimated" | "remodelled" | "none";
+ lodgedEpcRating?: string | null;
+ lodgedSapPoints?: number | null;
originalSapPoints?: number | null;
}
@@ -58,11 +61,24 @@ function getEpcDescription(letter: string | null | undefined): string {
export function CurrentEfficiencyCard({
epcRating,
sapPoints,
+ provenanceSignal = "none",
+ lodgedEpcRating,
+ lodgedSapPoints,
originalSapPoints,
}: CurrentEfficiencyCardProps) {
const epcHex = getEpcHex(epcRating);
- const lodgedLetter = originalSapPoints ? sapToEpc(originalSapPoints) : null;
- const showLodgedBadge = lodgedLetter !== null && lodgedLetter !== epcRating;
+ // New-approach: show the lodged certificate whenever the property was
+ // re-modelled (effective diverged from lodged), even within the same band.
+ // Legacy: fall back to the row's originalSapPoints, only when the band differs.
+ const legacyLodgedLetter = originalSapPoints ? sapToEpc(originalSapPoints) : null;
+ const remodelled = provenanceSignal === "remodelled";
+ const lodgedLetter = remodelled
+ ? lodgedEpcRating ?? null
+ : legacyLodgedLetter !== epcRating
+ ? legacyLodgedLetter
+ : null;
+ const lodgedBadgeSap = remodelled ? lodgedSapPoints : originalSapPoints;
+ const showLodgedBadge = lodgedLetter !== null;
const lodgedHex = getEpcHex(lodgedLetter);
return (
@@ -87,9 +103,9 @@ export function CurrentEfficiencyCard({
>
{lodgedLetter}
- {originalSapPoints != null && (
+ {lodgedBadgeSap != null && (
- / {Math.round(originalSapPoints)}
+ / {Math.round(lodgedBadgeSap)}
)}
diff --git a/src/app/db/backfill-recommendation-denormalization.ts b/src/app/db/backfill-recommendation-denormalization.ts
index ce8b74d2..b0854571 100644
--- a/src/app/db/backfill-recommendation-denormalization.ts
+++ b/src/app/db/backfill-recommendation-denormalization.ts
@@ -1,7 +1,8 @@
/**
* Backfills the denormalised plan_id / material_* columns on `recommendation`
- * from the plan_recommendations and recommendation_materials join tables, then
- * builds the supporting indexes and validates the foreign keys — all ONLINE.
+ * from the plan_recommendations and recommendation_materials join tables in a
+ * SINGLE keyset-paginated pass, then (optionally) builds the supporting indexes
+ * and validates the foreign keys — all ONLINE and resumable.
*
* Why this lives outside drizzle:
* drizzle-kit migrate wraps every pending migration in ONE transaction, so it
@@ -14,21 +15,55 @@
* Run AFTER `npm run migration:migrate` has applied 0222/0224 (the column adds):
* npm run backfill:recommendation-denormalization
*
+ * ── Making it fast (see docs/runbooks/recommendation-denormalization-backfill.md)
+ * The dominant cost is index write-amplification: `recommendation` is packed
+ * (fillfactor 100), so each UPDATE can't do a HOT update and must insert into
+ * every index on the table for every one of ~21m rows. To go roughly an order
+ * of magnitude faster in a low-usage window:
+ * 1. DROP the 3 secondary indexes (src/app/db/sql/drop-recommendation-backfill-indexes.sql)
+ * 2. run this script (BACKFILL_FINALIZE=false for a partial/interrupted window)
+ * 3. REBUILD them after (src/app/db/sql/rebuild-recommendation-backfill-indexes.sql)
+ * The backfill is resumable: stop any time, rebuild the indexes, and the app is
+ * back to today's behaviour (the half-filled columns are NULL-tolerant). Re-run
+ * to continue — the IS NULL guards never rewrite a row that's already done.
+ *
+ * This pass sets plan_id AND the four material_* columns in ONE write per row
+ * (the previous version did two full passes, rewriting every row twice).
+ *
* Safe to re-run: every step is idempotent (IS NULL guards, IF NOT EXISTS,
- * VALIDATE on an already-valid constraint is a no-op). If interrupted, just run
- * it again — it resumes from wherever it left off.
+ * VALIDATE on an already-valid constraint is a no-op).
*
* Tunables via env:
- * BACKFILL_BATCH_SIZE rows scanned per committed batch (default 25000)
- * BACKFILL_SLEEP_MS pause between batches, eases IO pressure (default 50)
+ * BACKFILL_BATCH_SIZE rows scanned per committed batch (default 100000)
+ * BACKFILL_SLEEP_MS pause between batches, eases IO pressure (default 0)
+ * BACKFILL_SYNC_COMMIT 'on' keeps synchronous_commit; default 'off' (faster
+ * batch commits — safe here because the run is resumable)
+ * BACKFILL_MAINTENANCE_WORK_MEM session maintenance_work_mem for index builds (default '1GB')
+ * BACKFILL_WORK_MEM session work_mem for the batch joins (default '128MB')
+ * BACKFILL_INDEX_CONCURRENTLY 'true' builds the new indexes CONCURRENTLY (default plain
+ * CREATE INDEX — faster, but briefly blocks writes)
+ * BACKFILL_FINALIZE 'false' skips the index build + FK validate; use this for
+ * partial/interrupted windows, finalize on the last one
*/
import dotenv from "dotenv";
import { Pool, type PoolClient } from "pg";
dotenv.config({ path: ".env.local" });
-const BATCH_SIZE = Number(process.env.BACKFILL_BATCH_SIZE ?? 25_000);
-const SLEEP_MS = Number(process.env.BACKFILL_SLEEP_MS ?? 50);
+const BATCH_SIZE = Number(process.env.BACKFILL_BATCH_SIZE ?? 100_000);
+const SLEEP_MS = Number(process.env.BACKFILL_SLEEP_MS ?? 0);
+const SYNC_COMMIT = process.env.BACKFILL_SYNC_COMMIT === "on" ? "on" : "off";
+const MAINTENANCE_WORK_MEM = process.env.BACKFILL_MAINTENANCE_WORK_MEM ?? "1GB";
+const WORK_MEM = process.env.BACKFILL_WORK_MEM ?? "128MB";
+const INDEX_CONCURRENTLY = process.env.BACKFILL_INDEX_CONCURRENTLY === "true";
+const FINALIZE = process.env.BACKFILL_FINALIZE !== "false";
+
+// Range + phase controls, for running several workers over disjoint id ranges in
+// parallel (near-linear speed-up — the ranges never contend for the same rows).
+const START_ID = process.env.BACKFILL_START_ID ?? "0";
+const END_ID = process.env.BACKFILL_END_ID ?? null; // inclusive upper bound; null = unbounded
+const SKIP_CARDINALITY = process.env.BACKFILL_SKIP_CARDINALITY === "true";
+const ONLY_FINALIZE = process.env.BACKFILL_ONLY_FINALIZE === "true";
const pool = new Pool({
host: process.env.DB_HOST,
@@ -65,28 +100,79 @@ async function assertSingleCardinality(
/**
* Keyset-paginate the whole `recommendation` table by id and, for each batch,
- * set the target columns from the join table where a match exists. Each batch
- * is its own autocommitted statement (no surrounding BEGIN), so locks are held
- * only for the rows in that batch.
+ * set plan_id and the four material_* columns from the join tables in a single
+ * UPDATE. Each batch is its own autocommitted statement (no surrounding BEGIN),
+ * so locks are held only for the rows in that batch.
*
- * We scan by id (not `WHERE col IS NULL LIMIT n`) so rows with no match don't
- * get re-selected forever — they're simply left NULL and we move past them.
+ * We scan by id (not `WHERE col IS NULL LIMIT n`) so rows with no match — or a
+ * legitimately NULL material_id — don't get re-selected forever; they're left as
+ * they are and we move past them.
+ *
+ * The guard only touches a row when there is real work to do (a NULL column that
+ * has a matching source row), and COALESCE never overwrites an already-set value.
+ * So a row is written at most once, and a re-run after interruption skips every
+ * row already done — the pass is idempotent and resumable.
+ *
+ * The single-cardinality assertion above guarantees the LEFT JOINs cannot fan a
+ * row out into multiple update targets.
*/
-async function backfillColumns(
- client: PoolClient,
- label: string,
- updateSql: (idFrom: string, limit: string) => string,
-): Promise {
- let lastId = "0";
+const fmtDuration = (secs: number): string => {
+ const s = Math.max(0, Math.round(secs));
+ const h = Math.floor(s / 3600);
+ const m = Math.floor((s % 3600) / 60);
+ return h > 0 ? `${h}h${m}m` : m > 0 ? `${m}m${s % 60}s` : `${s}s`;
+};
+
+async function backfillDenormalized(client: PoolClient): Promise {
+ let lastId = START_ID;
let scannedTotal = 0;
let updatedTotal = 0;
const startedAt = Date.now();
+ // Upper bound of the id keyset, for percent-complete + ETA. When END_ID is set
+ // (a parallel worker), use it; otherwise ask the table (fast, PK).
+ let maxId: number;
+ if (END_ID != null) {
+ maxId = Number(END_ID);
+ } else {
+ const { rows: maxRows } = await client.query(
+ `SELECT max(id) AS max_id FROM recommendation`,
+ );
+ maxId = Number(maxRows[0].max_id ?? 0);
+ }
+ console.log(
+ `Backfilling id ${Number(START_ID).toLocaleString()} → ${maxId.toLocaleString()}...`,
+ );
+
for (;;) {
- const { rows } = await client.query(updateSql("$1", "$2"), [
- lastId,
- BATCH_SIZE,
- ]);
+ const { rows } = await client.query(
+ `WITH batch AS (
+ SELECT id FROM recommendation
+ WHERE id > $1
+ AND ($3::bigint IS NULL OR id <= $3::bigint)
+ ORDER BY id
+ LIMIT $2
+ ),
+ upd AS (
+ UPDATE recommendation r
+ SET plan_id = COALESCE(r.plan_id, pr.plan_id),
+ material_id = COALESCE(r.material_id, rm.material_id),
+ material_quantity = COALESCE(r.material_quantity, rm.quantity),
+ material_quantity_unit = COALESCE(r.material_quantity_unit, rm.quantity_unit),
+ material_depth = COALESCE(r.material_depth, rm.depth)
+ FROM batch b
+ LEFT JOIN plan_recommendations pr ON pr.recommendation_id = b.id
+ LEFT JOIN recommendation_materials rm ON rm.recommendation_id = b.id
+ WHERE r.id = b.id
+ AND ( (r.plan_id IS NULL AND pr.plan_id IS NOT NULL)
+ OR (r.material_id IS NULL AND rm.material_id IS NOT NULL) )
+ RETURNING r.id
+ )
+ SELECT (SELECT max(id) FROM batch) AS max_id,
+ (SELECT count(*) FROM batch) AS scanned,
+ (SELECT count(*) FROM upd) AS updated`,
+ [lastId, BATCH_SIZE, END_ID],
+ );
const { max_id, scanned, updated } = rows[0] as {
max_id: string | null;
scanned: number;
@@ -99,17 +185,22 @@ async function backfillColumns(
updatedTotal += Number(updated);
lastId = max_id;
- const rate = Math.round(scannedTotal / ((Date.now() - startedAt) / 1000));
+ const elapsed = (Date.now() - startedAt) / 1000;
+ const rate = Math.round(scannedTotal / elapsed);
+ const from = Number(START_ID);
+ const span = maxId - from;
+ const pct = span > 0 ? Math.min(100, ((Number(lastId) - from) / span) * 100) : 0;
+ const eta = pct > 0 ? fmtDuration((elapsed / pct) * (100 - pct)) : "?";
console.log(
- `[${label}] up to id ${lastId} — scanned ${scannedTotal.toLocaleString()}, ` +
- `updated ${updatedTotal.toLocaleString()} (${rate.toLocaleString()} rows/s scan)`,
+ `${pct.toFixed(1)}% (id ${lastId}) — scanned ${scannedTotal.toLocaleString()}, ` +
+ `updated ${updatedTotal.toLocaleString()} — ${rate.toLocaleString()} rows/s, ETA ${eta}`,
);
if (SLEEP_MS > 0) await sleep(SLEEP_MS);
}
console.log(
- `[${label}] done: scanned ${scannedTotal.toLocaleString()}, ` +
+ `Backfill pass done: scanned ${scannedTotal.toLocaleString()}, ` +
`updated ${updatedTotal.toLocaleString()} in ` +
`${Math.round((Date.now() - startedAt) / 1000)}s`,
);
@@ -119,91 +210,77 @@ async function main(): Promise {
const client = await pool.connect();
try {
console.log(
- `Config: batch=${BATCH_SIZE.toLocaleString()} rows, sleep=${SLEEP_MS}ms\n`,
+ `Config: batch=${BATCH_SIZE.toLocaleString()} rows, sleep=${SLEEP_MS}ms, ` +
+ `synchronous_commit=${SYNC_COMMIT}, work_mem=${WORK_MEM}, ` +
+ `maintenance_work_mem=${MAINTENANCE_WORK_MEM}, finalize=${FINALIZE}, ` +
+ `range=(${Number(START_ID).toLocaleString()}, ${END_ID ?? "∞"}]` +
+ `${ONLY_FINALIZE ? ", ONLY_FINALIZE" : ""}\n`,
);
- // 1. Guard the 1:1 assumption before mutating anything.
- console.log("Checking cardinality...");
- await assertSingleCardinality(client, "plan_recommendations");
- await assertSingleCardinality(client, "recommendation_materials");
+ // Session tuning. synchronous_commit=off is safe here: the backfill is
+ // idempotent and resumable, so a crash that loses the last few batch commits
+ // just means the next run continues from there. work_mem feeds the batch
+ // joins; maintenance_work_mem feeds the CREATE INDEX in the finalize step.
+ await client.query(`SET synchronous_commit = ${SYNC_COMMIT}`);
+ await client.query(`SET work_mem = '${WORK_MEM}'`);
+ await client.query(`SET maintenance_work_mem = '${MAINTENANCE_WORK_MEM}'`);
- // 2. Backfill plan_id.
- await backfillColumns(
- client,
- "plan_id",
- (idFrom, limit) => `
- WITH batch AS (
- SELECT id FROM recommendation
- WHERE id > ${idFrom}
- ORDER BY id
- LIMIT ${limit}
- ),
- upd AS (
- UPDATE recommendation r
- SET plan_id = pr.plan_id
- FROM batch b
- JOIN plan_recommendations pr ON pr.recommendation_id = b.id
- WHERE r.id = b.id
- AND r.plan_id IS NULL
- RETURNING r.id
- )
- SELECT (SELECT max(id) FROM batch) AS max_id,
- (SELECT count(*) FROM batch) AS scanned,
- (SELECT count(*) FROM upd) AS updated`,
- );
+ if (ONLY_FINALIZE) {
+ // Parallel workflow: the data was written by range workers; this run just
+ // builds the indexes + validates the FKs. No cardinality check, no pass.
+ console.log("ONLY_FINALIZE: skipping cardinality check + backfill pass.");
+ } else {
+ // 1. Guard the 1:1 assumption before mutating anything. Skippable on
+ // secondary parallel workers (one worker having checked is enough).
+ if (SKIP_CARDINALITY) {
+ console.log("Skipping cardinality check (BACKFILL_SKIP_CARDINALITY=true).");
+ } else {
+ console.log("Checking cardinality...");
+ await assertSingleCardinality(client, "plan_recommendations");
+ await assertSingleCardinality(client, "recommendation_materials");
+ }
- // 3. Backfill the four material_* columns in one pass.
- await backfillColumns(
- client,
- "material",
- (idFrom, limit) => `
- WITH batch AS (
- SELECT id FROM recommendation
- WHERE id > ${idFrom}
- ORDER BY id
- LIMIT ${limit}
- ),
- upd AS (
- UPDATE recommendation r
- SET material_id = rm.material_id,
- material_quantity = rm.quantity,
- material_quantity_unit = rm.quantity_unit,
- material_depth = rm.depth
- FROM batch b
- JOIN recommendation_materials rm ON rm.recommendation_id = b.id
- WHERE r.id = b.id
- AND r.material_id IS NULL
- RETURNING r.id
- )
- SELECT (SELECT max(id) FROM batch) AS max_id,
- (SELECT count(*) FROM batch) AS scanned,
- (SELECT count(*) FROM upd) AS updated`,
- );
+ // 2. Single combined pass: plan_id + material_* in one write per row.
+ console.log("Backfilling plan_id + material_* (single pass)...");
+ await backfillDenormalized(client);
+ }
- // 4. Build indexes CONCURRENTLY (no write lock). Must NOT be in a txn —
- // a pooled client with no BEGIN runs each statement autocommitted.
- console.log("Creating indexes concurrently...");
- await client.query(
- `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_recommendation_plan_id
- ON recommendation USING btree (plan_id)`,
- );
- await client.query(
- `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_recommendation_material_id
- ON recommendation USING btree (material_id)`,
- );
+ if (FINALIZE || ONLY_FINALIZE) {
+ // 3. Build the two new indexes. Plain CREATE INDEX is faster in a quiet
+ // window; BACKFILL_INDEX_CONCURRENTLY=true avoids the brief write lock.
+ // Autocommitted (no surrounding BEGIN), which CONCURRENTLY requires.
+ const cc = INDEX_CONCURRENTLY ? "CONCURRENTLY " : "";
+ console.log(
+ `Creating indexes (${INDEX_CONCURRENTLY ? "concurrently" : "plain"})...`,
+ );
+ await client.query(
+ `CREATE INDEX ${cc}IF NOT EXISTS idx_recommendation_plan_id
+ ON recommendation USING btree (plan_id)`,
+ );
+ await client.query(
+ `CREATE INDEX ${cc}IF NOT EXISTS idx_recommendation_material_id
+ ON recommendation USING btree (material_id)`,
+ );
- // 5. Validate the FKs online (ShareUpdateExclusiveLock — allows reads/writes).
- console.log("Validating foreign keys...");
- await client.query(
- `ALTER TABLE recommendation
- VALIDATE CONSTRAINT recommendation_plan_id_plan_id_fk`,
- );
- await client.query(
- `ALTER TABLE recommendation
- VALIDATE CONSTRAINT recommendation_material_id_material_id_fk`,
- );
+ // 4. Validate the FKs online (ShareUpdateExclusiveLock — allows reads/writes).
+ console.log("Validating foreign keys...");
+ await client.query(
+ `ALTER TABLE recommendation
+ VALIDATE CONSTRAINT recommendation_plan_id_plan_id_fk`,
+ );
+ await client.query(
+ `ALTER TABLE recommendation
+ VALIDATE CONSTRAINT recommendation_material_id_material_id_fk`,
+ );
+ } else {
+ console.log(
+ "Skipping index build + FK validate (BACKFILL_FINALIZE=false). " +
+ "Run once more with finalize enabled in your final window.",
+ );
+ }
- // 6. Report any rows still unlinked (expected for material; investigate for plan).
+ // 5. Report any rows still unlinked. material_id NULL is expected (not every
+ // recommendation has a material); plan_id NULL should trend toward ~0.
const { rows } = await client.query(
`SELECT count(*) FILTER (WHERE plan_id IS NULL) AS plan_null,
count(*) FILTER (WHERE material_id IS NULL) AS material_null
@@ -213,7 +290,9 @@ async function main(): Promise {
`\nRemaining NULLs — plan_id: ${Number(rows[0].plan_null).toLocaleString()}, ` +
`material_id: ${Number(rows[0].material_null).toLocaleString()}`,
);
- console.log("Backfill complete.");
+ console.log(
+ FINALIZE ? "Backfill complete." : "Backfill pass complete (not finalized).",
+ );
} finally {
client.release();
await pool.end();
diff --git a/src/app/db/run-sql.ts b/src/app/db/run-sql.ts
new file mode 100644
index 00000000..2b491b8c
--- /dev/null
+++ b/src/app/db/run-sql.ts
@@ -0,0 +1,68 @@
+/**
+ * Runs one or more .sql files against the DB from .env.local, executing each
+ * statement autocommitted (no wrapping transaction) so statements that cannot run
+ * inside a transaction block — VACUUM, CREATE INDEX CONCURRENTLY — work. For
+ * environments without the psql CLI.
+ *
+ * tsx src/app/db/run-sql.ts [more.sql ...]
+ *
+ * Statement splitting is deliberately simple: it strips full-line `--` comments
+ * and splits on `;`. That's correct for the hand-written files in
+ * src/app/db/sql, but it is NOT a general SQL parser — don't point it at files
+ * with `;` inside string literals or dollar-quoted function bodies.
+ */
+import { readFileSync } from "fs";
+import dotenv from "dotenv";
+import { Pool } from "pg";
+
+dotenv.config({ path: ".env.local" });
+
+const files = process.argv.slice(2);
+if (files.length === 0) {
+ console.error("Usage: tsx src/app/db/run-sql.ts [more.sql ...]");
+ process.exit(1);
+}
+
+const pool = new Pool({
+ host: process.env.DB_HOST,
+ port: Number(process.env.DB_PORT),
+ user: process.env.DB_USERNAME,
+ password: process.env.DB_PASSWORD,
+ database: process.env.DB_NAME,
+});
+
+function statementsOf(path: string): string[] {
+ const raw = readFileSync(path, "utf8");
+ return raw
+ .split("\n")
+ .filter((line) => !line.trim().startsWith("--"))
+ .join("\n")
+ .split(";")
+ .map((s) => s.trim())
+ .filter((s) => s.length > 0);
+}
+
+async function main(): Promise {
+ const client = await pool.connect();
+ try {
+ for (const file of files) {
+ console.log(`\n=== ${file} ===`);
+ for (const sql of statementsOf(file)) {
+ const preview = sql.replace(/\s+/g, " ").slice(0, 70);
+ const startedAt = Date.now();
+ console.log(`> ${preview}${sql.length > 70 ? "…" : ""}`);
+ await client.query(sql);
+ console.log(` ok (${Math.round((Date.now() - startedAt) / 1000)}s)`);
+ }
+ }
+ console.log("\nAll statements complete.");
+ } finally {
+ client.release();
+ await pool.end();
+ }
+}
+
+main().catch((err) => {
+ console.error("SQL run failed:", err);
+ process.exit(1);
+});
diff --git a/src/app/db/schema/property.ts b/src/app/db/schema/property.ts
index 870cd025..58ed1d4c 100644
--- a/src/app/db/schema/property.ts
+++ b/src/app/db/schema/property.ts
@@ -39,6 +39,14 @@ export interface PropertyMeta {
currentEpcRating: string;
currentSapPoints: number;
originalSapPoints: number | null;
+ // Provenance of the displayed "current" figures (see ADR-0002 / provenance.ts).
+ // "estimated" = no certificate (predicted); "remodelled" = lodged but
+ // rebaselined; "none" = lodged == effective, or legacy.
+ provenanceSignal: "estimated" | "remodelled" | "none";
+ // Lodged (gov-register) headline, only when a real certificate exists; feeds
+ // the "Lodged EPC" badge alongside the effective "current".
+ lodgedEpcRating: string | null;
+ lodgedSapPoints: number | null;
updatedAt: string;
currentValuation: number | null;
detailsEpc: {
@@ -385,6 +393,13 @@ export interface PropertyWithRelations extends Record {
// New fields
landlordPropertyId: string | null;
originalSapPoints: number | null;
+ // Effective "current" (ADR-0002) is in currentEpcRating/currentSapPoints; these
+ // carry the separate lodged certificate + the default plan's post-retrofit
+ // rating + the provenance signal that drives the "Predicted" pill.
+ lodgedEpcRating: string | null;
+ postEpcRating: string | null;
+ postSapPoints: number | null;
+ provenanceSignal: "estimated" | "remodelled" | "none";
epcLodgementDate: string | null;
epcIsExpired: boolean | null;
// Optional columns (hidden by default)
diff --git a/src/app/db/sql/drop-recommendation-backfill-indexes.sql b/src/app/db/sql/drop-recommendation-backfill-indexes.sql
new file mode 100644
index 00000000..2fd28dd1
--- /dev/null
+++ b/src/app/db/sql/drop-recommendation-backfill-indexes.sql
@@ -0,0 +1,26 @@
+-- Drop the 3 secondary indexes on `recommendation` so the denormalisation
+-- backfill can run without per-row index write-amplification.
+--
+-- `recommendation` is packed (fillfactor 100), so each backfill UPDATE cannot do
+-- a HOT update and must insert a new entry into every index on the table for all
+-- ~21m rows. Dropping these first cuts index inserts per updated row from ~4 to 1
+-- (just the primary key), which is the bulk of the speed-up.
+--
+-- These serve production reads (fetch-by-property, the active-defaults paths), so
+-- ONLY run this in a low-usage window and rebuild them
+-- (rebuild-recommendation-backfill-indexes.sql) as soon as the window's backfill
+-- pass finishes. DROP INDEX is near-instant (catalog change + file unlink); it
+-- takes a brief ACCESS EXCLUSIVE lock, so if long-running queries are in flight
+-- use `DROP INDEX CONCURRENTLY` (cannot run inside a transaction) instead.
+--
+-- Kept OUT of the backfill script on purpose so you control exactly when the app
+-- loses and regains these indexes. See
+-- docs/runbooks/recommendation-denormalization-backfill.md
+--
+-- NOTE: idx_recommendation_active_id_property was originally built CONCURRENTLY;
+-- if a prior CONCURRENTLY build was ever interrupted it can linger as INVALID —
+-- the DROP below clears it either way.
+
+DROP INDEX IF EXISTS recommendation_property_id_idx;
+DROP INDEX IF EXISTS idx_recommendation_active_defaults;
+DROP INDEX IF EXISTS idx_recommendation_active_id_property;
diff --git a/src/app/db/sql/rebuild-recommendation-backfill-indexes.sql b/src/app/db/sql/rebuild-recommendation-backfill-indexes.sql
new file mode 100644
index 00000000..37714e3b
--- /dev/null
+++ b/src/app/db/sql/rebuild-recommendation-backfill-indexes.sql
@@ -0,0 +1,31 @@
+-- Rebuild the 3 secondary indexes on `recommendation` after a backfill window,
+-- restoring the app to its normal (pre-backfill) query behaviour.
+--
+-- The definitions below reproduce exactly what the migrations created (partial
+-- WHERE clauses included). Run any time — the backfill does NOT need to be
+-- complete first: these indexes don't reference plan_id/material_id, so a
+-- half-finished backfill is invisible to them and to the app.
+--
+-- Two of the three are PARTIAL (only rows where default = true and not yet
+-- installed), so they cover a small slice of the table and rebuild in seconds;
+-- only recommendation_property_id_idx spans all rows and takes a few minutes.
+--
+-- Plain CREATE INDEX is fastest but briefly blocks writes while each builds. If
+-- you need writes to keep flowing, replace CREATE INDEX with
+-- CREATE INDEX CONCURRENTLY (slower, cannot run inside a transaction, and each
+-- statement must be sent on its own).
+
+-- 1GB comfortably holds the ~500MB sort for a 21M-row single-column btree.
+-- Keep it well under instance RAM (this DB is 4GB) to avoid memory pressure.
+SET maintenance_work_mem = '1GB';
+
+CREATE INDEX IF NOT EXISTS recommendation_property_id_idx
+ ON recommendation USING btree (property_id);
+
+CREATE INDEX IF NOT EXISTS idx_recommendation_active_defaults
+ ON recommendation USING btree (id)
+ WHERE "default" = true AND already_installed = false;
+
+CREATE INDEX IF NOT EXISTS idx_recommendation_active_id_property
+ ON recommendation USING btree (id, property_id)
+ WHERE "default" = true AND already_installed = false;
diff --git a/src/app/db/sql/vacuum-recommendation.sql b/src/app/db/sql/vacuum-recommendation.sql
new file mode 100644
index 00000000..d5495446
--- /dev/null
+++ b/src/app/db/sql/vacuum-recommendation.sql
@@ -0,0 +1,7 @@
+-- Reclaim the dead tuples the backfill left behind (it updated ~every row, so the
+-- table is ~2x bloated) and refresh planner stats so queries pick up the new
+-- indexes. Standard VACUUM (not FULL) — online, no exclusive lock, but it does a
+-- full heap + index scan so it can take a while. No progress output; watch
+-- pg_stat_progress_vacuum in another session.
+
+VACUUM (ANALYZE) recommendation;
diff --git a/src/app/portfolio/[slug]/(portfolio)/reporting/BreakdownChart.tsx b/src/app/portfolio/[slug]/(portfolio)/reporting/BreakdownChart.tsx
index 9b660ed0..976ecdec 100644
--- a/src/app/portfolio/[slug]/(portfolio)/reporting/BreakdownChart.tsx
+++ b/src/app/portfolio/[slug]/(portfolio)/reporting/BreakdownChart.tsx
@@ -51,15 +51,27 @@ export function BreakdownChart({
const rows: any[] = [];
- for (const d of epcBands) {
- const epc = d.epc ?? "Unknown";
+ // The baseline query only returns bands that have properties, so a band
+ // reached ONLY after the scenario (e.g. A/B when nothing starts that high)
+ // is absent from epcBands — looping it alone drops those scenario bars.
+ // Iterate a canonical order over baseline ∪ scenario instead.
+ const baselineByBand = new Map(
+ epcBands.map((d) => [d.epc ?? "Unknown", d]),
+ );
+ const BAND_ORDER = ["A", "B", "C", "D", "E", "F", "G", "Unknown"];
+ const presentBands = BAND_ORDER.filter(
+ (b) => baselineByBand.has(b) || (scenarioEpcBands?.[b] ?? 0) > 0,
+ );
+
+ for (const epc of presentBands) {
+ const d = baselineByBand.get(epc);
const scenarioValue = scenarioEpcBands?.[epc] ?? 0;
// Baseline (stacked)
rows.push({
label: `${epc}`,
- [friendlyKeys.actual]: d.actual ?? 0,
- [friendlyKeys.estimated]: d.estimated ?? 0,
+ [friendlyKeys.actual]: d?.actual ?? 0,
+ [friendlyKeys.estimated]: d?.estimated ?? 0,
[friendlyKeys.scenario]: 0,
});
diff --git a/src/app/portfolio/[slug]/(portfolio)/reporting/databaseFunctions.ts b/src/app/portfolio/[slug]/(portfolio)/reporting/databaseFunctions.ts
index 7dbf1216..44d349c3 100644
--- a/src/app/portfolio/[slug]/(portfolio)/reporting/databaseFunctions.ts
+++ b/src/app/portfolio/[slug]/(portfolio)/reporting/databaseFunctions.ts
@@ -1,6 +1,19 @@
import { db } from "@/app/db/db";
import { sql, eq } from "drizzle-orm";
import { scenario } from "@/app/db/schema/recommendations";
+import {
+ NEW_APPROACH_CUTOFF,
+ newApproachJoins,
+ carbonSql,
+ billsSql,
+ energyConsumptionSql,
+ estimatedSql,
+ isExpiredSql,
+ effectiveSapSql,
+ effectiveEpcBandSql,
+ propertyTypeSql,
+ constructionYearSql,
+} from "@/lib/services/epcSources";
import type {
AverageMetrics,
@@ -27,20 +40,13 @@ export async function getAverages(
): Promise {
const result = await db.execute(sql`
SELECT
- AVG(p.current_sap_points)::float AS avg_sap,
- AVG(e.co2_emissions)::float AS avg_carbon,
- AVG(
- e.heating_cost_current +
- e.hot_water_cost_current +
- e.lighting_cost_current +
- e.appliances_cost_current +
- e.gas_standing_charge +
- e.electricity_standing_charge -
- COALESCE(e.installed_measures_total_energy_bill_adjustment, 0)
- )::float AS avg_bills,
- AVG(e.primary_energy_consumption)::float AS avg_energy_consumption
+ AVG(${effectiveSapSql})::float AS avg_sap,
+ AVG(${carbonSql(sql`e`)})::float AS avg_carbon,
+ AVG(${billsSql(sql`e`)})::float AS avg_bills,
+ AVG(${energyConsumptionSql(sql`e`)})::float AS avg_energy_consumption
FROM property p
LEFT JOIN property_details_epc e ON e.property_id = p.id
+ ${newApproachJoins}
WHERE p.portfolio_id = ${portfolioId};
`);
@@ -50,18 +56,11 @@ export async function getAverages(
export async function getTotals(portfolioId: number): Promise {
const result = await db.execute(sql`
SELECT
- SUM(e.co2_emissions)::float AS total_carbon,
- SUM(
- e.heating_cost_current +
- e.hot_water_cost_current +
- e.lighting_cost_current +
- e.appliances_cost_current +
- e.gas_standing_charge +
- e.electricity_standing_charge -
- COALESCE(e.installed_measures_total_energy_bill_adjustment, 0)
- )::float AS total_bills
+ SUM(${carbonSql(sql`e`)})::float AS total_carbon,
+ SUM(${billsSql(sql`e`)})::float AS total_bills
FROM property p
LEFT JOIN property_details_epc e ON e.property_id = p.id
+ ${newApproachJoins}
WHERE p.portfolio_id = ${portfolioId};
`);
@@ -74,20 +73,21 @@ export async function getCountByAgeBand(
const result = await db.execute(sql`
SELECT
CASE
- WHEN year_built ~ '^[0-9]+$' THEN
- CASE
- WHEN CAST(year_built AS int) < 1900 THEN '<1900'
- WHEN CAST(year_built AS int) BETWEEN 1900 AND 1929 THEN '1900–1929'
- WHEN CAST(year_built AS int) BETWEEN 1930 AND 1949 THEN '1930–1949'
- WHEN CAST(year_built AS int) BETWEEN 1950 AND 1975 THEN '1950–1975'
- WHEN CAST(year_built AS int) BETWEEN 1976 AND 1999 THEN '1976–1999'
- ELSE '2000+'
- END
+ WHEN q.built_year < 1900 THEN '<1900'
+ WHEN q.built_year BETWEEN 1900 AND 1929 THEN '1900–1929'
+ WHEN q.built_year BETWEEN 1930 AND 1949 THEN '1930–1949'
+ WHEN q.built_year BETWEEN 1950 AND 1975 THEN '1950–1975'
+ WHEN q.built_year BETWEEN 1976 AND 1999 THEN '1976–1999'
+ WHEN q.built_year >= 2000 THEN '2000+'
ELSE 'Unknown'
END AS age_band,
COUNT(*)::int AS count
- FROM property
- WHERE portfolio_id = ${portfolioId}
+ FROM (
+ SELECT ${constructionYearSql} AS built_year
+ FROM property p
+ ${newApproachJoins}
+ WHERE p.portfolio_id = ${portfolioId}
+ ) q
GROUP BY age_band
ORDER BY age_band;
`);
@@ -102,16 +102,17 @@ export async function getCountByEpcBand(
SELECT *
FROM (
SELECT
- COALESCE(p.current_epc_rating::text, 'Unknown') AS epc,
+ COALESCE((${effectiveEpcBandSql})::text, 'Unknown') AS epc,
COUNT(*) FILTER (
- WHERE e.estimated = false OR e.estimated IS NULL
+ WHERE ${estimatedSql(sql`e`)} = false
)::int AS actual,
COUNT(*) FILTER (
- WHERE e.estimated = true
+ WHERE ${estimatedSql(sql`e`)} = true
)::int AS estimated
FROM property p
LEFT JOIN property_details_epc e
ON e.property_id = p.id
+ ${newApproachJoins}
WHERE p.portfolio_id = ${portfolioId}
GROUP BY epc
) q
@@ -136,10 +137,12 @@ export async function getEstimatedCounts(
): Promise {
const result = await db.execute(sql`
SELECT
- SUM(CASE WHEN e.estimated = true THEN 1 ELSE 0 END)::int AS estimated,
- SUM(CASE WHEN e.estimated = false THEN 1 ELSE 0 END)::int AS actual
- FROM property_details_epc e
- WHERE e.portfolio_id = ${portfolioId};
+ SUM(CASE WHEN ${estimatedSql(sql`e`)} = true THEN 1 ELSE 0 END)::int AS estimated,
+ SUM(CASE WHEN ${estimatedSql(sql`e`)} = false THEN 1 ELSE 0 END)::int AS actual
+ FROM property p
+ LEFT JOIN property_details_epc e ON e.property_id = p.id
+ ${newApproachJoins}
+ WHERE p.portfolio_id = ${portfolioId};
`);
return result.rows[0];
@@ -149,10 +152,11 @@ export async function getCountByPropertyType(
portfolioId: number,
): Promise {
const result = await db.execute(sql`
- SELECT property_type AS type, COUNT(*)::int AS count
- FROM property
- WHERE portfolio_id = ${portfolioId}
- GROUP BY property_type
+ SELECT ${propertyTypeSql} AS type, COUNT(*)::int AS count
+ FROM property p
+ ${newApproachJoins}
+ WHERE p.portfolio_id = ${portfolioId}
+ GROUP BY type
ORDER BY count DESC;
`);
@@ -163,14 +167,16 @@ export async function getExpiredEpcCount(portfolioId: number): Promise {
const result = await db.execute<{ expired: number }>(sql`
SELECT
SUM(
- CASE
- WHEN is_expired = true AND estimated = false
- THEN 1
- ELSE 0
+ CASE
+ WHEN ${isExpiredSql(sql`e`)} = true AND ${estimatedSql(sql`e`)} = false
+ THEN 1
+ ELSE 0
END
)::int AS expired
- FROM property_details_epc
- WHERE portfolio_id = ${portfolioId};
+ FROM property p
+ LEFT JOIN property_details_epc e ON e.property_id = p.id
+ ${newApproachJoins}
+ WHERE p.portfolio_id = ${portfolioId};
`);
return result.rows[0].expired;
@@ -186,6 +192,9 @@ export async function getLikelyDowngrades(
JOIN property_details_epc e
ON e.property_id = p.id
WHERE p.portfolio_id = ${portfolioId}
+ -- GAP: the SAP-05 "likely downgrade" signal has no equivalent in the new
+ -- pipeline (handled there via Rebaseline). Restrict to legacy properties.
+ AND p.updated_at < ${NEW_APPROACH_CUTOFF}::date
AND e.sap_05_overwritten = true
AND p.current_sap_points IS NOT NULL
AND e.sap_05_score IS NOT NULL
diff --git a/src/app/portfolio/[slug]/building-passport/[propertyId]/assessment/page.tsx b/src/app/portfolio/[slug]/building-passport/[propertyId]/assessment/page.tsx
index a2123cb9..dacb6205 100644
--- a/src/app/portfolio/[slug]/building-passport/[propertyId]/assessment/page.tsx
+++ b/src/app/portfolio/[slug]/building-passport/[propertyId]/assessment/page.tsx
@@ -238,6 +238,9 @@ export default async function PreAssessmentReport(props: {
diff --git a/src/app/portfolio/[slug]/building-passport/[propertyId]/layout.tsx b/src/app/portfolio/[slug]/building-passport/[propertyId]/layout.tsx
index a4c68a42..e60c2f85 100644
--- a/src/app/portfolio/[slug]/building-passport/[propertyId]/layout.tsx
+++ b/src/app/portfolio/[slug]/building-passport/[propertyId]/layout.tsx
@@ -65,7 +65,7 @@ export default async function DashboardLayout(props: {
decentHomes={decentHomes}
/>
- {propertyMeta.detailsEpc.estimated && }
+ {propertyMeta.detailsEpc?.estimated && }
{children}
diff --git a/src/app/portfolio/[slug]/building-passport/[propertyId]/page.tsx b/src/app/portfolio/[slug]/building-passport/[propertyId]/page.tsx
index 1035edfe..92c12eed 100644
--- a/src/app/portfolio/[slug]/building-passport/[propertyId]/page.tsx
+++ b/src/app/portfolio/[slug]/building-passport/[propertyId]/page.tsx
@@ -86,6 +86,9 @@ export default async function BuildingPassportHome(props: {
@@ -218,9 +221,15 @@ export default async function BuildingPassportHome(props: {
{/* Location & Status */}
Location & Status
-
-
+
+
{/* Annual Energy Costs */}
diff --git a/src/app/portfolio/[slug]/building-passport/[propertyId]/utils.ts b/src/app/portfolio/[slug]/building-passport/[propertyId]/utils.ts
index c56b477c..89875a9b 100644
--- a/src/app/portfolio/[slug]/building-passport/[propertyId]/utils.ts
+++ b/src/app/portfolio/[slug]/building-passport/[propertyId]/utils.ts
@@ -1,6 +1,7 @@
import S3 from "aws-sdk/clients/s3";
import {
Recommendation,
+ recommendation,
planRecommendations,
plan,
Plan,
@@ -13,15 +14,18 @@ import { surveyDB } from "@/app/db/surveyDB/connection";
import {
Feature,
GeneralFeature,
- PropertyDetailsEpc,
PropertyDetailsSpatial,
PropertyMeta,
- propertyDetailsEpc,
propertyDetailsSpatial,
NonIntrusiveSurveyData,
nonInstrusiveSurvey,
} from "@/app/db/schema/property";
-import { getRating } from "@/app/utils";
+import {
+ ConditionReport,
+ resolveConditionReport,
+ resolvePropertyMeta,
+} from "@/lib/services/epcSources";
+import { getRating, serializeBigInt } from "@/app/utils";
import { eq, desc, and } from "drizzle-orm";
import {
energyAssessment,
@@ -161,23 +165,26 @@ export async function getEnergyAssessmentDocuments(
export async function getRecommendations(
planId: string
): Promise {
- const data = (await db.query.planRecommendations.findMany({
- where: eq(planRecommendations.planId, BigInt(planId)),
- columns: {},
- with: {
- recommendation: true,
- },
- })) as RecommendationList;
-
- if (!data) {
- throw new Error("Network response was not ok");
+ // Recommendations link to their plan via recommendation.plan_id (the
+ // plan_recommendations join table is retired). recommendation.plan_id is NOT
+ // indexed and the table has tens of millions of rows, so we key off the plan's
+ // property_id (indexed) and filter plan_id in memory. We drop already-installed
+ // measures, matching the previous behaviour.
+ const planRow = await db.query.plan.findFirst({
+ columns: { propertyId: true },
+ where: eq(plan.id, BigInt(planId)),
+ });
+ if (!planRow?.propertyId) {
+ return [];
}
- // unnest the recommendations
- // We drop measures that are already installed
- const recommendations = data
- .map((item) => item.recommendation)
- .filter((rec) => !rec.alreadyInstalled);
+ const recommendations = await db.query.recommendation.findMany({
+ where: and(
+ eq(recommendation.propertyId, planRow.propertyId),
+ eq(recommendation.planId, BigInt(planId)),
+ eq(recommendation.alreadyInstalled, false)
+ ),
+ });
return recommendations;
}
@@ -234,32 +241,26 @@ export async function getPlans(propertyId: string): Promise {
export async function getPropertyMeta(
propertyId: string
): Promise {
- const url = process.env.URL + `/api/property-meta/${propertyId}`;
-
- // Note, for the moment, we don't cache the data
- const response = await fetch(url, {
- method: "GET",
- headers: { "Content-Type": "application/json" },
- next: { revalidate: 60 },
- });
- if (!response.ok) {
+ // Resolve in-process instead of self-fetching /api/property-meta. The HTTP
+ // round-trip went to process.env.URL, which can point at a different
+ // deployment (e.g. production) and was serving stale/old-code responses — so
+ // the backfilled headline SAP/EPC never reached the page. We JSON round-trip
+ // through serializeBigInt to keep the exact shape the fetch used to return
+ // (bigints as strings, dates as ISO strings).
+ const meta = await resolvePropertyMeta(propertyId);
+ if (!meta) {
throw new Error("Network response was not ok");
}
- return response.json();
+ return JSON.parse(JSON.stringify(meta, serializeBigInt)) as PropertyMeta;
}
export async function getConditionReport(
propertyId: string
-): Promise {
- const data = await db.query.propertyDetailsEpc.findFirst({
- where: eq(propertyDetailsEpc.propertyId, BigInt(propertyId)),
- });
-
- if (!data) {
- throw new Error("Network response was not ok");
- }
-
- return data;
+): Promise {
+ // Branches on the new-approach cutoff: legacy properties read
+ // property_details_epc; new-approach ones read the epc_property graph +
+ // property_baseline_performance. See src/lib/services/epcSources.ts.
+ return resolveConditionReport(propertyId);
}
export async function getSpatialData(
@@ -298,7 +299,7 @@ export async function getNonIntrusiveSurvey(
}
export function formatGeneralFeatures(
- conditionReportData: PropertyDetailsEpc,
+ conditionReportData: ConditionReport,
propertyType: string
): GeneralFeature[] {
// if a property is a flat/maisonette, we show heat loss coridoor information, otherwise we won't show it
@@ -315,26 +316,27 @@ export function formatGeneralFeatures(
},
{
feature: "Number of storeys",
- description: conditionReportData.numberStoreys || "Unknown",
+ description: conditionReportData.numberStoreys ?? "Unknown",
},
];
const generealFeatures: GeneralFeature[] = [
{
feature: "Floor Height",
- description: conditionReportData.floorHeight || "Unknown",
+ description: conditionReportData.floorHeight ?? "Unknown",
},
{
feature: "Number of heated rooms",
- description: conditionReportData.numberHeatedRooms || "Unknown",
+ description: conditionReportData.numberHeatedRooms ?? "Unknown",
},
{
feature: "Number of open fire places",
- description: conditionReportData.numberOpenFireplaces || "Unknown",
+ // ?? so a genuine 0 (no fireplaces) shows as 0, not "Unknown".
+ description: conditionReportData.numberOpenFireplaces ?? "Unknown",
},
{
feature: "Number of extensions",
- description: conditionReportData.numberExtensions || "Unknown",
+ description: conditionReportData.numberExtensions ?? "Unknown",
},
{
feature: "Mains gas",
@@ -353,7 +355,7 @@ export function formatGeneralFeatures(
}
export function formatRetrofitFeatures(
- conditionReportData: PropertyDetailsEpc
+ conditionReportData: ConditionReport
): Feature[] {
// Helper function to convert rating number to string
@@ -437,7 +439,7 @@ export function formatRetrofitFeatures(
}
export function formatHeatDemandFeatures(
- conditionReportData: PropertyDetailsEpc
+ conditionReportData: ConditionReport
): GeneralFeature[] {
return [
{
diff --git a/src/app/portfolio/[slug]/components/PropertyFilters.tsx b/src/app/portfolio/[slug]/components/PropertyFilters.tsx
index 546b2b75..7add0920 100644
--- a/src/app/portfolio/[slug]/components/PropertyFilters.tsx
+++ b/src/app/portfolio/[slug]/components/PropertyFilters.tsx
@@ -16,6 +16,7 @@ import {
TENURE_OPTIONS,
YEAR_BUILT_OPTIONS,
MAINFUEL_OPTIONS,
+ PROVENANCE_OPTIONS,
} from "@/app/utils/propertyFilters";
/* -----------------------------------------------------------------------
@@ -33,6 +34,7 @@ const FIELD_OPTIONS: { value: FilterField; label: string }[] = [
{ value: "builtForm", label: "Built Form" },
{ value: "tenure", label: "Tenure" },
{ value: "yearBuilt", label: "Year Built" },
+ { value: "provenance", label: "Provenance" },
{ value: "floorArea", label: "Floor Area (m²)" },
{ value: "co2Emissions", label: "CO₂ Emissions (kg/m²/yr)" },
{ value: "mainfuel", label: "Main Fuel" },
@@ -74,6 +76,7 @@ const ENUM_FIELD_OPTIONS: Record = {
builtForm: BUILT_FORM_OPTIONS,
tenure: TENURE_OPTIONS,
yearBuilt: YEAR_BUILT_OPTIONS,
+ provenance: PROVENANCE_OPTIONS,
mainfuel: MAINFUEL_OPTIONS,
};
diff --git a/src/app/portfolio/[slug]/components/expectedEpc.test.ts b/src/app/portfolio/[slug]/components/expectedEpc.test.ts
new file mode 100644
index 00000000..50a09aac
--- /dev/null
+++ b/src/app/portfolio/[slug]/components/expectedEpc.test.ts
@@ -0,0 +1,20 @@
+import { describe, expect, it } from "vitest";
+import { expectedEpcRating } from "./expectedEpc";
+
+describe("expectedEpcRating", () => {
+ it("uses the default plan's post EPC rating when present", () => {
+ expect(
+ expectedEpcRating({ postEpcRating: "C", postSapPoints: 71.4 }),
+ ).toBe("C");
+ });
+
+ it("derives the band from post SAP when the plan has no stored rating", () => {
+ expect(
+ expectedEpcRating({ postEpcRating: null, postSapPoints: 64 }),
+ ).toBe("D");
+ });
+
+ it("is blank when the property has no default plan", () => {
+ expect(expectedEpcRating(null)).toBe("");
+ });
+});
diff --git a/src/app/portfolio/[slug]/components/expectedEpc.ts b/src/app/portfolio/[slug]/components/expectedEpc.ts
new file mode 100644
index 00000000..686e5cf1
--- /dev/null
+++ b/src/app/portfolio/[slug]/components/expectedEpc.ts
@@ -0,0 +1,19 @@
+import { sapToEpc } from "@/app/utils";
+
+/**
+ * The portfolio "Expected EPC" — the post-retrofit band of a property's default
+ * plan. The plan engine already models interactions, SAP 10 and the effective
+ * baseline, so we trust `plan.post_epc_rating` (falling back to deriving it from
+ * `post_sap_points`) rather than adding raw recommendation SAP onto the current
+ * score, which double-counts across plans and overshoots. See ADR-0002 and the
+ * grilling notes for property 732385 (read A, should read C).
+ *
+ * Returns "" when there is no default plan — the column renders blank, not the
+ * current rating.
+ */
+export function expectedEpcRating(
+ plan: { postEpcRating: string | null; postSapPoints: number | null } | null,
+): string {
+ if (!plan) return "";
+ return plan.postEpcRating ?? sapToEpc(plan.postSapPoints);
+}
diff --git a/src/app/portfolio/[slug]/components/propertyTableColumns.tsx b/src/app/portfolio/[slug]/components/propertyTableColumns.tsx
index 7ef022eb..9d2955d5 100644
--- a/src/app/portfolio/[slug]/components/propertyTableColumns.tsx
+++ b/src/app/portfolio/[slug]/components/propertyTableColumns.tsx
@@ -15,6 +15,7 @@ import { FunnelIcon } from "@heroicons/react/24/outline";
import { formatNumber, getEpcColorClass, sapToEpc } from "@/app/utils";
import { cn } from "@/lib/utils";
import { PropertyWithRelations } from "@/app/db/schema/property";
+import { expectedEpcRating } from "./expectedEpc";
import { X } from "lucide-react";
import {
EnumOption,
@@ -72,6 +73,20 @@ const EpcLetterBubble = ({ letter }: { letter: string }) => {
);
};
+/**
+ * Marks a property whose EPC was estimated from nearby homes (no certificate) —
+ * the "trust this less" signal. Re-modelled properties aren't marked (the
+ * Lodged-EPC column differing from Current already conveys that). See ADR-0002.
+ */
+const PredictedPill = () => (
+
+ Predicted
+
+);
+
/* -----------------------------------------------------------------------
Column header with dropdown filter
------------------------------------------------------------------------ */
@@ -229,19 +244,27 @@ const coreColumns: ColumnDef[] = [
),
cell: ({ row }) => (
-
);
},
diff --git a/src/app/portfolio/[slug]/utils.ts b/src/app/portfolio/[slug]/utils.ts
index e3676c89..6014cc25 100644
--- a/src/app/portfolio/[slug]/utils.ts
+++ b/src/app/portfolio/[slug]/utils.ts
@@ -19,14 +19,29 @@ import {
ScenarioSelect,
} from "@/app/db/schema/recommendations";
import { sql } from "drizzle-orm";
+import {
+ newApproachJoins,
+ carbonSql,
+ totalFloorAreaSql,
+ lodgementDateSql,
+ isExpiredSql,
+ mainfuelSql,
+ effectiveSapSql,
+ effectiveEpcBandSql,
+ lodgedEpcBandSql,
+ lodgedSapSql,
+ provenanceSignalSql,
+} from "@/lib/services/epcSources";
import {
FilterGroups,
+ FilterField,
PropertyFilter,
PROPERTY_TYPE_OPTIONS,
BUILT_FORM_OPTIONS,
TENURE_OPTIONS,
YEAR_BUILT_OPTIONS,
MAINFUEL_OPTIONS,
+ PROVENANCE_OPTIONS,
EnumOption,
} from "@/app/utils/propertyFilters";
import { EPC_TO_SAP_MIN, EPC_TO_SAP_MAX } from "@/app/utils/epc";
@@ -36,6 +51,7 @@ const ENUM_FIELD_DB_OPTIONS: Record = {
builtForm: BUILT_FORM_OPTIONS,
tenure: TENURE_OPTIONS,
yearBuilt: YEAR_BUILT_OPTIONS,
+ provenance: PROVENANCE_OPTIONS,
mainfuel: MAINFUEL_OPTIONS,
};
@@ -513,10 +529,11 @@ function buildConditionSql(filter: PropertyFilter): ReturnType | nul
return null;
case "currentEpc":
- return buildEpcSapCondition(sql`p.current_sap_points`, filter.operator, filter.value);
+ // "Current" is the effective baseline (ADR-0002), not the null row column.
+ return buildEpcSapCondition(effectiveSapSql, filter.operator, filter.value);
case "lodgedEpc":
- return buildEpcSapCondition(sql`p.original_sap_points`, filter.operator, filter.value);
+ return buildEpcSapCondition(lodgedSapSql, filter.operator, filter.value);
case "expectedEpc": {
if (filter.operator === "epc_at_least") {
@@ -528,8 +545,11 @@ function buildConditionSql(filter: PropertyFilter): ReturnType | nul
}
case "epcExpiryDate": {
- const expiryExpr = sql`(epc.lodgement_date + INTERVAL '10 years')`;
- const guard = sql`epc.lodgement_date IS NOT NULL`;
+ // Branches on the cutoff: new-approach properties derive the lodgement
+ // date from the lodged epc_property row. See epcSources.ts.
+ const lodgementExpr = lodgementDateSql(sql`epc`);
+ const expiryExpr = sql`(${lodgementExpr} + INTERVAL '10 years')`;
+ const guard = sql`${lodgementExpr} IS NOT NULL`;
if (filter.operator === "date_before") {
return sql`${guard} AND ${expiryExpr} < ${filter.value}::date`;
@@ -543,7 +563,7 @@ function buildConditionSql(filter: PropertyFilter): ReturnType | nul
if (filter.operator === "date_preset") {
switch (filter.value) {
case "expired":
- return sql`epc.is_expired = true`;
+ return sql`${isExpiredSql(sql`epc`)} = true`;
case "expires_this_year":
return sql`${guard} AND EXTRACT(YEAR FROM ${expiryExpr}) = EXTRACT(YEAR FROM CURRENT_DATE)`;
case "expires_within_1_year":
@@ -561,6 +581,7 @@ function buildConditionSql(filter: PropertyFilter): ReturnType | nul
case "builtForm":
case "tenure":
case "yearBuilt":
+ case "provenance":
case "mainfuel": {
if (filter.operator !== "enum_one_of") return null;
@@ -578,7 +599,10 @@ function buildConditionSql(filter: PropertyFilter): ReturnType | nul
builtForm: sql`p.built_form`,
tenure: sql`p.tenure`,
yearBuilt: sql`p.year_built`,
- mainfuel: sql`epc.mainfuel`,
+ provenance: provenanceSignalSql,
+ // GAP: new-approach properties have no main-fuel string → NULL, so they
+ // only match the "no value" filter branch. See epcSources.ts.
+ mainfuel: mainfuelSql(sql`epc`),
};
const col = colMap[filter.field];
@@ -616,18 +640,20 @@ function buildConditionSql(filter: PropertyFilter): ReturnType | nul
case "floorArea": {
const n = parseFloat(filter.value);
if (isNaN(n)) return null;
- if (filter.operator === "num_gte") return sql`epc.total_floor_area >= ${n}`;
- if (filter.operator === "num_lte") return sql`epc.total_floor_area <= ${n}`;
- if (filter.operator === "num_equals") return sql`epc.total_floor_area = ${n}`;
+ const area = totalFloorAreaSql(sql`epc`);
+ if (filter.operator === "num_gte") return sql`${area} >= ${n}`;
+ if (filter.operator === "num_lte") return sql`${area} <= ${n}`;
+ if (filter.operator === "num_equals") return sql`${area} = ${n}`;
return null;
}
case "co2Emissions": {
const n = parseFloat(filter.value);
if (isNaN(n)) return null;
- if (filter.operator === "num_gte") return sql`epc.co2_emissions >= ${n}`;
- if (filter.operator === "num_lte") return sql`epc.co2_emissions <= ${n}`;
- if (filter.operator === "num_equals") return sql`epc.co2_emissions = ${n}`;
+ const carbon = carbonSql(sql`epc`);
+ if (filter.operator === "num_gte") return sql`${carbon} >= ${n}`;
+ if (filter.operator === "num_lte") return sql`${carbon} <= ${n}`;
+ if (filter.operator === "num_equals") return sql`${carbon} = ${n}`;
return null;
}
}
@@ -656,17 +682,62 @@ function buildWhereClause(filterGroups: FilterGroups): ReturnType {
: sql``;
}
+// Filter fields whose SQL references the EPC graph (epc/bp/epl/epp) vs the
+// default-plan LATERAL. The count query only needs a join when an active filter
+// references it — otherwise joining is pure cost. The `pl` LATERAL in particular
+// runs once per property (31k+ correlated plan lookups for a large portfolio),
+// which alone pushed the unfiltered count past Vercel's 15s limit.
+const EPC_JOIN_FILTER_FIELDS = new Set([
+ "currentEpc",
+ "lodgedEpc",
+ "provenance",
+ "co2Emissions",
+ "floorArea",
+ "epcExpiryDate",
+ "mainfuel",
+]);
+const PLAN_JOIN_FILTER_FIELDS = new Set(["expectedEpc"]);
+
+function filterFieldsInUse(filterGroups: FilterGroups): Set {
+ const fields = new Set();
+ for (const group of filterGroups) {
+ for (const cond of group.conditions) fields.add(cond.field);
+ }
+ return fields;
+}
+
export async function getPropertiesCount(
portfolioId: string,
filterGroups: FilterGroups = []
): Promise {
const combinedWhere = buildWhereClause(filterGroups);
+ const fields = filterFieldsInUse(filterGroups);
+
+ // Only join what an active filter needs. COUNT is unaffected by the LEFT JOINs
+ // otherwise (none multiply rows), so omitting them is purely a speed-up.
+ const needsEpcJoins = [...fields].some((f) => EPC_JOIN_FILTER_FIELDS.has(f));
+ const needsPlanJoin = [...fields].some((f) => PLAN_JOIN_FILTER_FIELDS.has(f));
+
+ const epcJoins = needsEpcJoins
+ ? sql`LEFT JOIN property_details_epc epc ON epc.property_id = p.id
+ ${newApproachJoins}`
+ : sql``;
+ const planJoin = needsPlanJoin
+ ? sql`LEFT JOIN LATERAL (
+ SELECT id, post_sap_points FROM plan
+ WHERE property_id = p.id
+ AND portfolio_id = p.portfolio_id
+ AND is_default = true
+ ORDER BY created_at DESC
+ LIMIT 1
+ ) pl ON true`
+ : sql``;
const result = await db.execute<{ count: string }>(sql`
SELECT COUNT(DISTINCT p.id)::int AS count
FROM property p
- LEFT JOIN property_details_epc epc ON epc.property_id = p.id
- LEFT JOIN plan pl ON pl.property_id = p.id AND pl.is_default = true
+ ${epcJoins}
+ ${planJoin}
WHERE p.portfolio_id = ${portfolioId}
${combinedWhere}
`);
@@ -693,67 +764,74 @@ export async function getProperties(
p.postcode AS postcode,
p.status AS status,
p.creation_status AS "creationStatus",
- p.current_epc_rating AS "currentEpcRating",
- p.current_sap_points AS "currentSapPoints",
- t.epc AS "targetEpc",
+ -- "Current" is the effective (re-baselined) figure — the canonical current
+ -- per ADR-0002. The lodged certificate is a separate column. New-approach
+ -- properties have null headline columns on the property row, so both source
+ -- from property_baseline_performance.
+ ${effectiveEpcBandSql} AS "currentEpcRating",
+ ${effectiveSapSql} AS "currentSapPoints",
+ ${lodgedEpcBandSql} AS "lodgedEpcRating",
+ ${provenanceSignalSql} AS "provenanceSignal",
+ -- "Expected EPC" is the default plan's modelled post-retrofit rating, not
+ -- current + Σ recommendation SAP (which double-counts across plans and
+ -- overshoots). See expectedEpc.ts / ADR-0002.
+ NULL AS "targetEpc",
+ pl.post_epc_rating AS "postEpcRating",
+ pl.post_sap_points AS "postSapPoints",
pl.id AS "planId",
- fp.scheme AS "fundingScheme",
- COALESCE(SUM(r.sap_points), 0) AS "totalRecommendationSapPoints",
- COALESCE(SUM(r.estimated_cost), 0) AS "totalRecommendationCost",
+ -- funding_package is no longer read here (the proposals table fetches it
+ -- separately). Kept as NULL to preserve the PropertyWithRelations shape.
+ NULL AS "fundingScheme",
+ COALESCE(rec.sap_points, 0) AS "totalRecommendationSapPoints",
+ COALESCE(rec.cost, 0) AS "totalRecommendationCost",
p.landlord_property_id AS "landlordPropertyId",
p.original_sap_points AS "originalSapPoints",
p.property_type AS "propertyType",
p.built_form AS "builtForm",
p.tenure AS tenure,
p.year_built AS "yearBuilt",
- epc.lodgement_date::text AS "epcLodgementDate",
- epc.is_expired AS "epcIsExpired",
- epc.total_floor_area AS "totalFloorArea",
- epc.co2_emissions AS "co2Emissions",
- epc.mainfuel AS mainfuel,
+ ${lodgementDateSql(sql`epc`)}::text AS "epcLodgementDate",
+ ${isExpiredSql(sql`epc`)} AS "epcIsExpired",
+ ${totalFloorAreaSql(sql`epc`)} AS "totalFloorArea",
+ ${carbonSql(sql`epc`)} AS "co2Emissions",
+ ${mainfuelSql(sql`epc`)} AS mainfuel,
p.lexiscore AS lexiscore
FROM property p
- LEFT JOIN property_targets t
- ON t.property_id = p.id
- LEFT JOIN plan pl
- ON pl.property_id = p.id
- AND pl.is_default = true
- LEFT JOIN funding_package fp
- ON fp.plan_id = pl.id
- LEFT JOIN plan_recommendations pr
- ON pr.plan_id = pl.id
- LEFT JOIN recommendation r
- ON r.id = pr.recommendation_id
- AND r.default = true
- AND r.already_installed = false
+ -- LATERAL one-row lookups keep this query at "N properties × index probe"
+ -- instead of hash-joining the multi-million-row plan/recommendation tables.
+ LEFT JOIN LATERAL (
+ SELECT id, post_sap_points, post_epc_rating FROM plan
+ WHERE property_id = p.id
+ AND portfolio_id = p.portfolio_id
+ AND is_default = true
+ ORDER BY created_at DESC
+ LIMIT 1
+ ) pl ON true
+ -- Active (default, not-yet-installed) recommendations for the property,
+ -- read denormalised. We aggregate by recommendation.property_id — the only
+ -- indexed access path (there is NO index on recommendation.plan_id, and the
+ -- table has tens of millions of rows) — and rely on the partial index on
+ -- (default, already_installed). The retired plan_recommendations join table
+ -- is intentionally not used.
+ LEFT JOIN LATERAL (
+ SELECT
+ COALESCE(SUM(sap_points), 0) AS sap_points,
+ COALESCE(SUM(estimated_cost), 0) AS cost
+ FROM recommendation
+ WHERE property_id = p.id
+ -- Scope to the default plan. property_id is the indexed access path, so
+ -- this only filters the property's own (tiny) rec set in memory — no
+ -- unindexed plan_id scan — while stopping the SUM double-counting a
+ -- measure once per plan when a property has several plans.
+ AND plan_id = pl.id
+ AND "default" = true
+ AND already_installed = false
+ ) rec ON true
LEFT JOIN property_details_epc epc
ON epc.property_id = p.id
+ ${newApproachJoins}
WHERE p.portfolio_id = ${portfolioId}
${combinedWhere}
- GROUP BY
- p.id,
- p.portfolio_id,
- p.address,
- p.postcode,
- p.status,
- p.creation_status,
- p.current_epc_rating,
- p.current_sap_points,
- t.epc,
- pl.id,
- fp.scheme,
- p.landlord_property_id,
- p.original_sap_points,
- p.property_type,
- p.built_form,
- p.tenure,
- p.year_built,
- epc.lodgement_date,
- epc.is_expired,
- epc.total_floor_area,
- epc.co2_emissions,
- epc.mainfuel,
- p.lexiscore
LIMIT ${limit} OFFSET ${offset};
`);
diff --git a/src/app/shadcn_components/ui/tooltip.tsx b/src/app/shadcn_components/ui/tooltip.tsx
index 30fc44d9..25576f54 100644
--- a/src/app/shadcn_components/ui/tooltip.tsx
+++ b/src/app/shadcn_components/ui/tooltip.tsx
@@ -15,15 +15,17 @@ const TooltipContent = React.forwardRef<
React.ElementRef,
React.ComponentPropsWithoutRef
>(({ className, sideOffset = 4, ...props }, ref) => (
-
+
+
+
))
TooltipContent.displayName = TooltipPrimitive.Content.displayName
diff --git a/src/app/utils/propertyFilters.ts b/src/app/utils/propertyFilters.ts
index e67309ae..1f85675e 100644
--- a/src/app/utils/propertyFilters.ts
+++ b/src/app/utils/propertyFilters.ts
@@ -10,6 +10,7 @@ export type FilterField =
| "builtForm"
| "tenure"
| "yearBuilt"
+ | "provenance"
| "floorArea"
| "co2Emissions"
| "mainfuel";
@@ -89,6 +90,14 @@ export const TENURE_OPTIONS: EnumOption[] = [
{ label: "Not Recorded", dbValues: ["__null__"] },
];
+// Provenance signal (ADR-0002). dbValues are the raw signal strings emitted by
+// provenanceSignalSql / resolveProvenanceSignal.
+export const PROVENANCE_OPTIONS: EnumOption[] = [
+ { label: "Predicted", dbValues: ["estimated"] },
+ { label: "Re-modelled", dbValues: ["remodelled"] },
+ { label: "Standard", dbValues: ["none"] },
+];
+
export const YEAR_BUILT_OPTIONS: EnumOption[] = [
"1900","1930","1950","1967","1976","1983","1991","1996",
"2003","2007","2008","2009","2010","2011","2012","2013",
diff --git a/src/lib/services/epcSources.test.ts b/src/lib/services/epcSources.test.ts
new file mode 100644
index 00000000..1d85a6ab
--- /dev/null
+++ b/src/lib/services/epcSources.test.ts
@@ -0,0 +1,68 @@
+import { describe, expect, it } from "vitest";
+import {
+ CONSTRUCTION_AGE_BANDS,
+ PROPERTY_TYPE_LABELS,
+} from "./epcSources";
+import { PROPERTY_TYPE_OPTIONS } from "@/app/utils/propertyFilters";
+
+// The reporting age-band bucketing in getCountByAgeBand (databaseFunctions.ts),
+// mirrored so the CONSTRUCTION_AGE_BANDS representative years stay anchored to
+// the buckets the chart renders.
+function reportingAgeBand(year: number): string {
+ if (year < 1900) return "<1900";
+ if (year <= 1929) return "1900–1929";
+ if (year <= 1949) return "1930–1949";
+ if (year <= 1975) return "1950–1975";
+ if (year <= 1999) return "1976–1999";
+ return "2000+";
+}
+
+describe("CONSTRUCTION_AGE_BANDS", () => {
+ it("keeps each representative year consistent with the band's label", () => {
+ for (const { label, year } of Object.values(CONSTRUCTION_AGE_BANDS)) {
+ if (label.startsWith("before")) {
+ expect(year).toBeLessThan(1900);
+ } else {
+ const start = Number(label.match(/\d{4}/)![0]);
+ expect(year).toBe(start);
+ }
+ }
+ });
+
+ it("buckets every RdSAP band into the expected reporting age band", () => {
+ const buckets = Object.fromEntries(
+ Object.entries(CONSTRUCTION_AGE_BANDS).map(([code, { year }]) => [
+ code,
+ reportingAgeBand(year),
+ ]),
+ );
+ expect(buckets).toEqual({
+ A: "<1900",
+ B: "1900–1929",
+ C: "1930–1949",
+ D: "1950–1975",
+ E: "1950–1975",
+ F: "1976–1999",
+ G: "1976–1999",
+ H: "1976–1999",
+ // I (1996–2002) straddles the 1976–1999 / 2000+ boundary; it buckets by
+ // its start year.
+ I: "1976–1999",
+ J: "2000+",
+ K: "2000+",
+ L: "2000+",
+ M: "2000+",
+ });
+ });
+});
+
+describe("PROPERTY_TYPE_LABELS", () => {
+ it("maps codes to the labels the property-type filter options use", () => {
+ const optionLabels = new Set(PROPERTY_TYPE_OPTIONS.map((o) => o.label));
+ for (const label of Object.values(PROPERTY_TYPE_LABELS)) {
+ // Park home has no filter option — it's rare and charts render it as-is.
+ if (label === "Park home") continue;
+ expect(optionLabels).toContain(label);
+ }
+ });
+});
diff --git a/src/lib/services/epcSources.ts b/src/lib/services/epcSources.ts
new file mode 100644
index 00000000..b50cdd78
--- /dev/null
+++ b/src/lib/services/epcSources.ts
@@ -0,0 +1,841 @@
+/**
+ * EPC read-model resolver.
+ *
+ * The legacy plan engine wrote a flat per-property table `property_details_epc`.
+ * The new modelling pipeline (Hestia-Homes/Model) does NOT write that table —
+ * it writes the normalized EPC graph (`epc_property` + children), the modelling
+ * output (`plan` / `recommendation`) and `property_baseline_performance`.
+ *
+ * A property is "new approach" when `property.updated_at >= 2026-06-01` (the
+ * cutoff — the old pipeline predates it). For such properties the app must read
+ * the new sources and must NOT touch `property_details_epc`. Legacy properties
+ * keep reading the old table unchanged.
+ *
+ * This module is the single place that knows the cutoff and the field→source
+ * mapping, so callers (building-passport, property-meta, portfolio list,
+ * reporting + scenario metrics) branch consistently.
+ *
+ * Mapping applied (see the handover report for the full table and flagged gaps):
+ * - EPC fabric descriptions + 1–5 ratings → `epc_energy_element`
+ * - whole-dwelling counts / flags → `epc_property` (+ `epc_flat_details`)
+ * - baseline carbon / energy / bills → `property_baseline_performance`
+ * - post-retrofit data → `plan` / `recommendation`
+ * - "estimated" flag → property has only a `source='predicted'`
+ * EPC (no `source='lodged'` row)
+ *
+ * Gracefully-omitted gaps for new properties (no clean equivalent — reported to
+ * the owner): `mainfuel`, `energyTariff`, `floorHeight`, the installed-measures
+ * bill adjustment, and the SAP-05 "likely downgrade" metric.
+ */
+import { db } from "@/app/db/db";
+import { and, eq, sql } from "drizzle-orm";
+import { resolveProvenanceSignal, type ProvenanceSignal } from "./provenance";
+import {
+ epcProperty,
+ epcEnergyElement,
+ epcFlatDetails,
+ epcBuildingPart,
+ epcFloorDimension,
+ propertyBaselinePerformance,
+ propertyDetailsEpc,
+ property,
+} from "@/app/db/schema/property";
+
+/** The discriminator: properties updated on/after this date use the new sources. */
+export const NEW_APPROACH_CUTOFF = "2026-06-01";
+
+export function isNewApproach(updatedAt: Date | string | null | undefined): boolean {
+ if (!updatedAt) return false;
+ const d = updatedAt instanceof Date ? updatedAt : new Date(updatedAt);
+ return d >= new Date(`${NEW_APPROACH_CUTOFF}T00:00:00.000Z`);
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Shared SQL fragments for set-based aggregate queries.
+//
+// These assume the standard aliases:
+// p = property
+// bp = property_baseline_performance (new — baseline carbon/energy/bills)
+// epl = epc_property WHERE source='lodged'
+// epp = epc_property WHERE source='predicted'
+// and take the legacy `property_details_epc` alias as a fragment so each query
+// can keep its own (`e` vs `epc`).
+//
+// Add `newApproachJoins` to any query that uses one of the *Sql expressions.
+// ─────────────────────────────────────────────────────────────────────────────
+
+type Alias = ReturnType;
+
+/** TRUE for new-approach rows. */
+export const isNewApproachSql = sql`p.updated_at >= ${NEW_APPROACH_CUTOFF}::date`;
+
+/** LEFT JOINs that expose the new sources alongside the legacy table. */
+export const newApproachJoins = sql`
+ LEFT JOIN property_baseline_performance bp ON bp.property_id = p.id
+ LEFT JOIN epc_property epl ON epl.property_id = p.id AND epl.source = 'lodged'
+ LEFT JOIN epc_property epp ON epp.property_id = p.id AND epp.source = 'predicted'
+`;
+
+/**
+ * Baseline carbon (t CO₂/yr). New: the lodged (gov-registry) value. Legacy:
+ * e.co2_emissions. Swap to `effective_co2_emissions_t_per_yr` if the
+ * modelling-adjusted baseline is wanted instead.
+ */
+export const carbonSql = (e: Alias) =>
+ sql`CASE WHEN ${isNewApproachSql} THEN bp.lodged_co2_emissions_t_per_yr ELSE ${e}.co2_emissions END`;
+
+/** Primary energy intensity (kWh/m²/yr). New: lodged value. Legacy: e.primary_energy_consumption. */
+export const energyConsumptionSql = (e: Alias) =>
+ sql`CASE WHEN ${isNewApproachSql} THEN bp.lodged_primary_energy_intensity_kwh_per_m2_yr ELSE ${e}.primary_energy_consumption END`;
+
+/**
+ * Baseline SAP score. New: the lodged (gov-registry) score from
+ * property_baseline_performance (the property-row current_sap_points isn't
+ * written for new-approach properties). Legacy: property.current_sap_points.
+ * Swap to `effective_sap_score` for the modelling-adjusted baseline.
+ */
+export const sapSql = sql`CASE WHEN ${isNewApproachSql} THEN bp.lodged_sap_score ELSE p.current_sap_points END`;
+
+/** Baseline EPC band. New: lodged band from property_baseline_performance. Legacy: property.current_epc_rating. */
+export const epcBandSql = sql`CASE WHEN ${isNewApproachSql} THEN bp.lodged_epc_band ELSE p.current_epc_rating END`;
+
+/**
+ * Effective (re-baselined) SAP score / EPC band — the canonical "current" for
+ * display and reporting (ADR-0002). New: bp.effective_*; legacy: the property
+ * row. These replaced the lodged `sapSql`/`epcBandSql` above as the table +
+ * reporting source; those lodged fragments are intentionally retained (unused
+ * for now) for a possible future register-fidelity reporting view (ADR-0002).
+ * The "Lodged EPC" column itself uses the source-gated `lodgedEpcBandSql` below.
+ */
+export const effectiveSapSql = sql`CASE WHEN ${isNewApproachSql} THEN bp.effective_sap_score ELSE p.current_sap_points END`;
+export const effectiveEpcBandSql = sql`CASE WHEN ${isNewApproachSql} THEN bp.effective_epc_band ELSE p.current_epc_rating END`;
+
+/**
+ * Lodged EPC band for the "Lodged EPC" column — only when a real certificate
+ * exists (`source = lodged`, i.e. the lodged epc_property row `epl`). A
+ * predicted property carries lodged_* (mirrored estimates) that must not
+ * surface, so it renders NULL. Legacy: derive from the property row's
+ * original_sap_points (handled in the column, kept as the row value here).
+ */
+export const lodgedEpcBandSql = sql`CASE WHEN ${isNewApproachSql} THEN (CASE WHEN epl.id IS NOT NULL THEN bp.lodged_epc_band ELSE NULL END) ELSE p.current_epc_rating END`;
+
+/** Lodged SAP score for the "Lodged EPC" filter — same source/gate as lodgedEpcBandSql. Legacy: the row's original_sap_points. */
+export const lodgedSapSql = sql`CASE WHEN ${isNewApproachSql} THEN (CASE WHEN epl.id IS NOT NULL THEN bp.lodged_sap_score ELSE NULL END) ELSE p.original_sap_points END`;
+
+/**
+ * Provenance signal for the portfolio table (mirrors resolveProvenanceSignal /
+ * provenance.ts). New-approach only; legacy → 'none'.
+ */
+export const provenanceSignalSql = sql`CASE
+ WHEN ${isNewApproachSql} THEN (
+ CASE
+ WHEN epp.id IS NOT NULL AND epl.id IS NULL THEN 'estimated'
+ WHEN epl.id IS NOT NULL AND bp.rebaseline_reason IS NOT NULL AND bp.rebaseline_reason <> 'none' THEN 'remodelled'
+ ELSE 'none'
+ END
+ )
+ ELSE 'none'
+END`;
+
+/**
+ * Annual energy bill (£). New: sum of the individual cost columns on
+ * property_baseline_performance (NULL when no baseline row exists, so it's
+ * excluded from AVG/SUM rather than counted as £0). Legacy: the old
+ * component-sum formula incl. standing charges and the installed-measures
+ * adjustment.
+ */
+export const billsSql = (e: Alias) =>
+ sql`CASE WHEN ${isNewApproachSql} THEN (
+ CASE WHEN bp.id IS NULL THEN NULL ELSE (
+ COALESCE(bp.heating_cost_gbp, 0) +
+ COALESCE(bp.hot_water_cost_gbp, 0) +
+ COALESCE(bp.lighting_cost_gbp, 0) +
+ COALESCE(bp.appliances_cost_gbp, 0) +
+ COALESCE(bp.cooking_cost_gbp, 0) +
+ COALESCE(bp.pumps_fans_cost_gbp, 0) +
+ COALESCE(bp.cooling_cost_gbp, 0) +
+ COALESCE(bp.standing_charges_gbp, 0) -
+ COALESCE(bp.seg_credit_gbp, 0)
+ ) END
+ ) ELSE (
+ ${e}.heating_cost_current +
+ ${e}.hot_water_cost_current +
+ ${e}.lighting_cost_current +
+ ${e}.appliances_cost_current +
+ ${e}.gas_standing_charge +
+ ${e}.electricity_standing_charge -
+ COALESCE(${e}.installed_measures_total_energy_bill_adjustment, 0)
+ ) END`;
+
+/** Total floor area (m²). New: lodged epc_property. Legacy: e.total_floor_area. */
+export const totalFloorAreaSql = (e: Alias) =>
+ sql`CASE WHEN ${isNewApproachSql} THEN epl.total_floor_area_m2 ELSE ${e}.total_floor_area END`;
+
+/** EPC lodgement date. New: lodged epc_property registration/inspection date. Legacy: e.lodgement_date. */
+export const lodgementDateSql = (e: Alias) =>
+ sql`CASE WHEN ${isNewApproachSql} THEN COALESCE(epl.registration_date, epl.inspection_date) ELSE ${e}.lodgement_date END`;
+
+/**
+ * "estimated" boolean. New: the property only has a predicted EPC (no lodged
+ * row). Legacy: e.estimated.
+ */
+export const estimatedSql = (e: Alias) =>
+ sql`CASE WHEN ${isNewApproachSql} THEN (epl.id IS NULL AND epp.id IS NOT NULL) ELSE COALESCE(${e}.estimated, false) END`;
+
+/**
+ * "expired" boolean. New: derived from the lodged EPC age (>10 years), since the
+ * new graph stores no is_expired flag. Legacy: e.is_expired.
+ */
+export const isExpiredSql = (e: Alias) =>
+ sql`CASE WHEN ${isNewApproachSql} THEN (epl.registration_date IS NOT NULL AND epl.registration_date < (CURRENT_DATE - INTERVAL '10 years')) ELSE ${e}.is_expired END`;
+
+/** Main fuel. GAP for new properties (no clean equivalent) → NULL. Legacy: e.mainfuel. */
+export const mainfuelSql = (e: Alias) =>
+ sql`CASE WHEN ${isNewApproachSql} THEN NULL ELSE ${e}.mainfuel END`;
+
+// ─────────────────────────────────────────────────────────────────────────────
+// RdSAP code → label maps for the descriptive fields the new pipeline stores as
+// codes on epc_property / epc_building_part (the property-row text columns
+// aren't written for new-approach properties). Unknown codes fall back to the
+// raw value. Shared by the SQL fragments below and resolvePropertyDescriptors —
+// the single source of truth for code→label mapping.
+// ─────────────────────────────────────────────────────────────────────────────
+
+export const PROPERTY_TYPE_LABELS: Record = {
+ "0": "House",
+ "1": "Bungalow",
+ "2": "Flat",
+ "3": "Maisonette",
+ "4": "Park home",
+};
+const BUILT_FORM_LABELS: Record = {
+ "1": "Detached",
+ "2": "Semi-Detached",
+ "3": "End-Terrace",
+ "4": "Mid-Terrace",
+ "5": "Enclosed End-Terrace",
+ "6": "Enclosed Mid-Terrace",
+};
+// SAP tenure codes → the text labels already used in the data. NOTE: best-effort
+// (standard SAP 1/2/3); most rows store the text label directly, only a few use
+// codes. Confirm against the backend tenure enum.
+const TENURE_LABELS: Record = {
+ "1": "Owner Occupied",
+ "2": "Rented Social",
+ "3": "Rented Private",
+};
+/**
+ * Matches the backend ConstructionAgeBand enum (RdSAP England & Wales letters).
+ * `year` is a representative construction year (the band's start) for reporting
+ * queries that bucket by year — band I (1996–2002) straddles the reporting
+ * 1976–1999 / 2000+ boundary, so it buckets with its start year.
+ */
+export const CONSTRUCTION_AGE_BANDS: Record<
+ string,
+ { label: string; year: number }
+> = {
+ A: { label: "before 1900", year: 1899 }, // no start year — any pre-1900 value
+ B: { label: "1900–1929", year: 1900 },
+ C: { label: "1930–1949", year: 1930 },
+ D: { label: "1950–1966", year: 1950 },
+ E: { label: "1967–1975", year: 1967 },
+ F: { label: "1976–1982", year: 1976 },
+ G: { label: "1983–1990", year: 1983 },
+ H: { label: "1991–1995", year: 1991 },
+ I: { label: "1996–2002", year: 1996 },
+ J: { label: "2003–2006", year: 2003 },
+ K: { label: "2007–2011", year: 2007 },
+ L: { label: "2012–2022", year: 2012 },
+ M: { label: "2023 onwards", year: 2023 },
+};
+
+/** `CASE