diff --git a/CONTEXT.md b/CONTEXT.md index aa87bc9a..4016fdcc 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -206,6 +206,7 @@ _Avoid_: estimation notification, banner (those are component names) ## Flagged ambiguities +- The portfolio **Property table**'s **Property Type**, **Built Form**, **Construction Age**, **Main Fuel**, **Wall Type**, **Roof Type** and **Heating System** columns show the **landlord-override-resolved** value — the same precedence the modelling **Run filter** applies (override fact → EPC-derived → **Unknown**) — not the raw `epc_property` value or the legacy property-row column. See [ADR-0012](./docs/adr/0012-portfolio-list-resolves-descriptors-via-override-precedence.md). All seven are also **filters**, and the per-part ones (wall/roof) read the **main building part**. Wall/Roof/Heating filter by **coarse bucket** rather than the full vocabulary — their resolved values are free-text (override vocab / EPC `epc_energy_element.description` / legacy column) with hundreds of variants, so a fixed set of prefix/substring buckets (Wall → Cavity / Solid Brick / Timber Frame / Stone / …; Roof → Pitched / Flat / Room-in-Roof / …; Heating → Gas Boiler / Community / Electric Storage / Heat Pump / …) matches them, while the columns still **display the full description**. `classifyDescriptor` is the pure twin of the SQL bucketing. Two label subtleties: **Construction Age** is an RdSAP band (e.g. "1900–1929", "before 1900"), never a single year (it was formerly the "Year Built" column); and **Main Fuel** for new-approach properties reads the EPC's `epc_main_heating_detail.main_fuel_type` (an RdSAP `main_fuel` code or an EPR string, folded to a coarse label) beneath any landlord override — it only reads **Unknown** when neither is present. - The UI surfaces labelled **"Current EPC"** and **"Current Efficiency State"** (property table, building-passport card) show **Effective performance**, not Lodged — despite the glossary advising against "current performance" for Effective. The label is a product choice ("current" = the property's true present-day modelled state); the underlying datum is still **Effective performance**. The **"Lodged EPC"** column/badge is the only surface showing **Lodged performance**, and only when a real certificate exists (`source = lodged`). - "Upload" is used in the codebase to mean both the file-on-S3 and the BulkUpload row. We standardise on **BulkUpload** for the row; the file is just "the source file." - "Onboarding" appears in some route paths (`bulk_onboarding_inputs/...`) but isn't part of this glossary — we use **BulkUpload** end-to-end. diff --git a/docs/adr/0012-portfolio-list-resolves-descriptors-via-override-precedence.md b/docs/adr/0012-portfolio-list-resolves-descriptors-via-override-precedence.md new file mode 100644 index 00000000..08f4c461 --- /dev/null +++ b/docs/adr/0012-portfolio-list-resolves-descriptors-via-override-precedence.md @@ -0,0 +1,89 @@ +# 12. The portfolio property-list resolves descriptors through the landlord-override precedence + +Date: 2026-07-10 + +## Status + +Accepted + +## Context + +The portfolio property-list's **Property Type**, **Built Form**, **Year Built** +and **Main Fuel** columns and filters read raw `property` row columns (and, for +main fuel, the legacy `property_details_epc` table). Those columns are NULL for +**new-approach** properties (`property.updated_at >= 2026-06-01`, the EPC-graph +cutoff — see `epcSources.ts`), so for any new portfolio the columns render blank +and the filters match nothing. The already-migrated columns (current/lodged EPC, +floor area, CO₂, provenance, expiry) route through `epcSources` fragments and +work; these four were simply never migrated. + +Separately, [ADR-0008](./0008-modelling-runs-filters-to-distributor.md) +established the resolution precedence for the modelling **Run filter**: +**landlord-override fact → EPC-derived value (new-approach graph, lodged over +predicted, RdSAP codes mapped; legacy property-row fallback) → Unknown**. It is +implemented in-app inline in `previewModellingRun` and mirrored by the +distributor; divergence between the two is defined as a bug. + +## Decision + +The portfolio list resolves Property Type, Built Form, Construction Age and Main +Fuel through the **same override precedence as the Run filter**, so the browse +surface agrees with what a modelling run would actually select. Concretely: + +- **Shared fragments, one home for the rule.** `epcSources.ts` gains + `overrideJoins` (LEFT JOINs of `property_overrides` aliased per component at + `building_part = 0`) and `resolvedPropertyTypeSql` / `resolvedBuiltFormSql` / + `resolvedConstructionBandSql` / `resolvedMainFuelSql`. Both the list query and + `previewModellingRun` consume them; the inline COALESCE in + `previewModellingRun` is refactored onto the shared fragments. +- **Never-null.** Each fragment is `COALESCE(override, epcSql, 'Unknown')`. Every + property resolves to a concrete, filterable string; the **Unknown** filter chip + matches "no resolvable value at all" — character-for-character the semantics + CONTEXT.md already gives the Run filter's Unknown. The list columns render an + explicit **Unknown** pill instead of a blank `—`. +- **Construction Age is band-native, not a year.** Override and EPC both store + the RdSAP band **letter** (`A..M`, `Unknown`); the field resolves in letter + space and displays the band **label** ("before 1900", "1900–1929", … "2023 + onwards"). A numeric year cannot represent an `Unknown` override and misreads a + band as a precise year. The column is **renamed "Year Built" → "Construction + Age"** and its filter options become the bands, not years; legacy raw years are + bucketed into bands. +- **Main Fuel is override-only for new-approach.** The EPC graph exposes no clean + main-fuel string (a documented `epcSources` gap → NULL), so for new-approach + properties the override is the *only* source; unoverridden properties read + **Unknown**. The override vocabulary (`"mains gas"`, `"LPG (bulk)"`, …) is + folded into the existing collapsed filter labels (Mains Gas, LPG, …). +- **`tenure` is removed from the list**, not fixed. It has no override component, + and its `epc_property.tenure` values don't cleanly reconcile with the filter's + `dbValues`. It is dropped from the list column and filter only; + `resolvePropertyMeta`/`resolvePropertyDescriptors` still resolve it for the + per-property building passport, and `property.tenure` is retained. + +## Alternatives considered + +- **EPC-derived only, no overrides.** Smaller, and consistent with how the other + already-migrated columns behave. Rejected: the list would then disagree with + the modelling Run filter — e.g. show "Flat" for a property a landlord override + makes (and modelling treats as) "Maisonette" — reintroducing exactly the + browse-vs-model mismatch this work set out to remove. +- **Construction Age as a numeric year.** Familiar and a smaller diff. Rejected: + cannot represent an `Unknown` override (collapses to `—`, breaking the + never-null contract) and misrepresents a band as a point year. +- **Keep the precedence rule inline in `previewModellingRun` and duplicate it in + the list.** Avoids touching a working feature. Rejected: it re-creates the + two-homes divergence ADR-0008 explicitly warns against — now within the app. + +## Consequences + +- The count query (`getPropertiesCount`) must join `property_overrides` when any + of these fields is filtered — `EPC_JOIN_FILTER_FIELDS` (or a sibling set) is + extended. Joins are perf-sensitive (the count path already brushes Vercel's + 15s limit), so the joins stay conditional on an active filter. +- Every new-approach property without a main-fuel override shows **Unknown** — a + lot of Unknown pills, but an honest signal of the EPC gap rather than a blank + implying "not applicable". +- Filter option lists grow: Property Type gains **Park home** + **Unknown**; + Built Form gains **Unknown** (keeping **Not Recorded** for the explicit + override); Main Fuel folds the override vocab into existing labels. +- `previewModellingRun` and the portfolio list are now provably in sync — the + shared fragment is the single reference implementation ADR-0008 asked for. diff --git a/src/app/api/properties/route.ts b/src/app/api/properties/route.ts index bf0239a5..6726e8be 100644 --- a/src/app/api/properties/route.ts +++ b/src/app/api/properties/route.ts @@ -2,7 +2,10 @@ import { NextRequest, NextResponse } from "next/server"; import { getProperties, getPropertiesCount } from "@/app/portfolio/[slug]/utils"; import { FilterGroups } from "@/app/utils/propertyFilters"; -const DEFAULT_LIMIT = 1000; +// The property list paginates at 7 rows/page and isn't scrolled through in bulk, +// so a small initial page keeps the (join-heavy) query fast. Export fetches its +// own larger batch on demand (see EXPORT_LIMIT in PropertyTable). +const DEFAULT_LIMIT = 250; export async function POST(req: NextRequest) { const body = await req.json(); diff --git a/src/app/db/schema/property.ts b/src/app/db/schema/property.ts index d05097c8..9158103b 100644 --- a/src/app/db/schema/property.ts +++ b/src/app/db/schema/property.ts @@ -410,11 +410,13 @@ export interface PropertyWithRelations extends Record { // Optional columns (hidden by default) propertyType: string | null; builtForm: string | null; - tenure: string | null; yearBuilt: string | null; totalFloorArea: number | null; co2Emissions: number | null; mainfuel: string | null; + wallType: string | null; + roofType: string | null; + heatingSystem: string | null; lexiscore: number | null; // Portfolio Tags on this property (ADR-0013), name-ordered; [] when untagged. tags: PropertyTag[]; diff --git a/src/app/portfolio/[slug]/components/PropertyFilters.tsx b/src/app/portfolio/[slug]/components/PropertyFilters.tsx index b6551978..f4a9dc35 100644 --- a/src/app/portfolio/[slug]/components/PropertyFilters.tsx +++ b/src/app/portfolio/[slug]/components/PropertyFilters.tsx @@ -12,13 +12,14 @@ import { FilterField, FilterOperator, DatePreset, - EnumOption, PROPERTY_TYPE_OPTIONS, BUILT_FORM_OPTIONS, - TENURE_OPTIONS, YEAR_BUILT_OPTIONS, MAINFUEL_OPTIONS, PROVENANCE_OPTIONS, + WALL_TYPE_OPTIONS, + ROOF_TYPE_OPTIONS, + HEATING_SYSTEM_OPTIONS, } from "@/app/utils/propertyFilters"; /* ----------------------------------------------------------------------- @@ -34,12 +35,14 @@ const FIELD_OPTIONS: { value: FilterField; label: string }[] = [ { value: "epcExpiryDate", label: "EPC Expiry Date" }, { value: "propertyType", label: "Property Type" }, { value: "builtForm", label: "Built Form" }, - { value: "tenure", label: "Tenure" }, - { value: "yearBuilt", label: "Year Built" }, + { value: "yearBuilt", label: "Construction Age" }, { value: "provenance", label: "Provenance" }, { value: "floorArea", label: "Floor Area (m²)" }, { value: "co2Emissions", label: "CO₂ Emissions (kg/m²/yr)" }, { value: "mainfuel", label: "Main Fuel" }, + { value: "wallType", label: "Wall Type" }, + { value: "roofType", label: "Roof Type" }, + { value: "heatingSystem", label: "Heating System" }, { value: "tags", label: "Tags" }, ]; @@ -77,13 +80,18 @@ const NUM_OPERATOR_OPTIONS: { value: FilterOperator; label: string }[] = [ { value: "num_equals", label: "= (equals)" }, ]; -const ENUM_FIELD_OPTIONS: Record = { - propertyType: PROPERTY_TYPE_OPTIONS, - builtForm: BUILT_FORM_OPTIONS, - tenure: TENURE_OPTIONS, - yearBuilt: YEAR_BUILT_OPTIONS, - provenance: PROVENANCE_OPTIONS, - mainfuel: MAINFUEL_OPTIONS, +// Options need only a `label` for the dropdown: the exact-enum fields carry +// EnumOption[] (dbValues used server-side), the free-text descriptor fields carry +// DescriptorBucket[] (needles used server-side). Both satisfy `{ label }`. +const ENUM_FIELD_OPTIONS: Record = { + propertyType: PROPERTY_TYPE_OPTIONS, + builtForm: BUILT_FORM_OPTIONS, + yearBuilt: YEAR_BUILT_OPTIONS, + provenance: PROVENANCE_OPTIONS, + mainfuel: MAINFUEL_OPTIONS, + wallType: WALL_TYPE_OPTIONS, + roofType: ROOF_TYPE_OPTIONS, + heatingSystem: HEATING_SYSTEM_OPTIONS, }; /* ----------------------------------------------------------------------- @@ -304,7 +312,7 @@ function EnumMultiDropdown({ selectedLabels, onChange, }: { - options: EnumOption[]; + options: { label: string }[]; selectedLabels: string[]; onChange: (labels: string[]) => void; }) { diff --git a/src/app/portfolio/[slug]/components/PropertyTable.tsx b/src/app/portfolio/[slug]/components/PropertyTable.tsx index 27ef002d..15c0a9a0 100644 --- a/src/app/portfolio/[slug]/components/PropertyTable.tsx +++ b/src/app/portfolio/[slug]/components/PropertyTable.tsx @@ -22,11 +22,7 @@ import BulkUploadComingSoonModal from "@/app/components/portfolio/BulkUploadComi import { sapToEpc } from "@/app/utils"; import { columns } from "@/app/portfolio/[slug]/components/propertyTableColumns"; import { PropertyWithRelations } from "@/app/db/schema/property"; -import { - TENURE_OPTIONS, - MAINFUEL_OPTIONS, - EnumOption, -} from "@/app/utils/propertyFilters"; +import { MAINFUEL_OPTIONS, EnumOption } from "@/app/utils/propertyFilters"; import { OPTIONAL_COLUMN_IDS, OPTIONAL_COLUMN_LABELS, @@ -62,7 +58,7 @@ import { Button } from "@/app/shadcn_components/ui/button"; /* ---------------------------------------- Export helpers ----------------------------------------- */ -const EXPORT_LIMIT = 1000; +const EXPORT_LIMIT = 6000; function resolveEnumLabel( options: EnumOption[], @@ -86,11 +82,13 @@ function exportToCsv(data: PropertyWithRelations[]) { "Plan Cost (£)", "Property Type", "Built Form", - "Tenure", - "Year Built", + "Construction Age", "Floor Area (m²)", "CO₂ Emissions (kg/m²/yr)", "Main Fuel", + "Wall Type", + "Roof Type", + "Heating System", ]; const rows = data.map((p) => { @@ -120,11 +118,13 @@ function exportToCsv(data: PropertyWithRelations[]) { p.totalRecommendationCost ? p.totalRecommendationCost.toFixed(2) : "", p.propertyType ?? "", p.builtForm ?? "", - resolveEnumLabel(TENURE_OPTIONS, p.tenure), p.yearBuilt ?? "", p.totalFloorArea != null ? p.totalFloorArea.toFixed(1) : "", p.co2Emissions != null ? p.co2Emissions.toFixed(1) : "", resolveEnumLabel(MAINFUEL_OPTIONS, p.mainfuel), + p.wallType ?? "", + p.roofType ?? "", + p.heatingSystem ?? "", ]; }); @@ -416,7 +416,7 @@ export default function PropertyTable({ }); const totalCount = allResponse?.total ?? 0; - const DISPLAY_LIMIT = 1000; + const DISPLAY_LIMIT = 250; const LOAD_MORE_SIZE = 100; // ── Extra rows (lazy-loaded pages beyond the initial 1 000) ────────────── @@ -509,6 +509,31 @@ export default function PropertyTable({ if (isOnLastPage && hasMore) loadMore(); }, [isOnLastPage, hasMore, loadMore]); + // Export fetches its own batch (up to EXPORT_LIMIT) for the current filters, + // independent of the on-screen page — the display fetch is deliberately small. + const [isExporting, setIsExporting] = useState(false); + const handleExport = useCallback(async () => { + if (isExporting) return; + setIsExporting(true); + try { + const res = await fetch("/api/properties", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + portfolioId, + filters: allFilterGroups, + limit: EXPORT_LIMIT, + offset: 0, + }), + }); + if (!res.ok) return; + const json: { data: PropertyWithRelations[] } = await res.json(); + exportToCsv(json.data); + } finally { + setIsExporting(false); + } + }, [isExporting, portfolioId, allFilterGroups]); + /* ---------------------------------------- Delete preview state ----------------------------------------- */ @@ -630,12 +655,12 @@ export default function PropertyTable({ ) : ( )} diff --git a/src/app/portfolio/[slug]/components/propertyTableColumns.tsx b/src/app/portfolio/[slug]/components/propertyTableColumns.tsx index 6630b0c8..3142be4a 100644 --- a/src/app/portfolio/[slug]/components/propertyTableColumns.tsx +++ b/src/app/portfolio/[slug]/components/propertyTableColumns.tsx @@ -17,11 +17,7 @@ import { cn } from "@/lib/utils"; import { PropertyWithRelations } from "@/app/db/schema/property"; import { expectedEpcRating } from "./expectedEpc"; import { X } from "lucide-react"; -import { - EnumOption, - TENURE_OPTIONS, - MAINFUEL_OPTIONS, -} from "@/app/utils/propertyFilters"; +import { EnumOption, MAINFUEL_OPTIONS } from "@/app/utils/propertyFilters"; import { CurrentEpcTooltip } from "./CurrentEpcTooltip"; import { TagsCell } from "./TagsCell"; import { Checkbox } from "@/app/shadcn_components/ui/checkbox"; @@ -38,13 +34,6 @@ function resolveEnumLabel( return opt?.label ?? dbValue; } -function tenureBadgeClass(label: string): string { - if (label.toLowerCase().includes("owner")) return "bg-blue-50 text-blue-700"; - if (label.toLowerCase().includes("private")) return "bg-violet-50 text-violet-700"; - if (label.toLowerCase().includes("social")) return "bg-emerald-50 text-emerald-700"; - return "bg-slate-100 text-slate-500"; -} - function Pill({ children, className = "bg-slate-100 text-slate-600", @@ -171,11 +160,13 @@ export const OPTIONAL_COLUMN_IDS = [ "tags", "propertyType", "builtForm", - "tenure", "yearBuilt", "totalFloorArea", "co2Emissions", "mainfuel", + "wallType", + "roofType", + "heatingSystem", ] as const; export type OptionalColumnId = (typeof OPTIONAL_COLUMN_IDS)[number]; @@ -184,11 +175,13 @@ const OPTIONAL_COLUMN_LABELS: Record = { tags: "Tags", propertyType: "Property Type", builtForm: "Built Form", - tenure: "Tenure", - yearBuilt: "Year Built", + yearBuilt: "Construction Age", totalFloorArea: "Floor Area (m²)", co2Emissions: "CO₂ Emissions", mainfuel: "Main Fuel", + wallType: "Wall Type", + roofType: "Roof Type", + heatingSystem: "Heating System", }; export { OPTIONAL_COLUMN_LABELS }; @@ -458,20 +451,10 @@ const optionalColumns: ColumnDef[] = [ return val ? {val} : ; }, }, - { - id: "tenure", - accessorKey: "tenure", - header: () =>
Tenure
, - cell: ({ row }) => { - const label = resolveEnumLabel(TENURE_OPTIONS, row.original.tenure); - if (!label) return ; - return {label}; - }, - }, { id: "yearBuilt", accessorKey: "yearBuilt", - header: () =>
Year Built
, + header: () =>
Construction Age
, cell: ({ row }) => (
{row.original.yearBuilt ?? "—"}
), @@ -511,8 +494,42 @@ const optionalColumns: ColumnDef[] = [ return label ? {label} : ; }, }, + { + id: "wallType", + accessorKey: "wallType", + header: () =>
Wall Type
, + cell: ({ row }) => , + }, + { + id: "roofType", + accessorKey: "roofType", + header: () =>
Roof Type
, + cell: ({ row }) => , + }, + { + id: "heatingSystem", + accessorKey: "heatingSystem", + header: () =>
Heating System
, + cell: ({ row }) => , + }, ]; +/** Long free-text descriptor (wall/roof/heating): truncate with a full-value tooltip; muted when Unknown. */ +function DescriptorCell({ value }: { value: string | null }) { + const val = value ?? "Unknown"; + return ( +
+ {val} +
+ ); +} + export const columns: ColumnDef[] = [ ...coreColumns, ...optionalColumns, diff --git a/src/app/portfolio/[slug]/utils.ts b/src/app/portfolio/[slug]/utils.ts index c6f5f326..745a0e4f 100644 --- a/src/app/portfolio/[slug]/utils.ts +++ b/src/app/portfolio/[slug]/utils.ts @@ -25,12 +25,25 @@ import { totalFloorAreaSql, lodgementDateSql, isExpiredSql, - mainfuelSql, effectiveSapSql, effectiveEpcBandSql, lodgedEpcBandSql, lodgedSapSql, provenanceSignalSql, + propertyTypeOverrideJoin, + resolvedPropertyTypeSql, + builtFormOverrideJoin, + resolvedBuiltFormSql, + constructionAgeOverrideJoin, + resolvedConstructionAgeSql, + mainFuelOverrideJoin, + resolvedMainFuelSql, + wallTypeOverrideJoin, + resolvedWallTypeSql, + roofTypeOverrideJoin, + resolvedRoofTypeSql, + mainHeatingOverrideJoin, + resolvedHeatingSystemSql, } from "@/lib/services/epcSources"; import { FilterGroups, @@ -38,18 +51,19 @@ import { PropertyFilter, PROPERTY_TYPE_OPTIONS, BUILT_FORM_OPTIONS, - TENURE_OPTIONS, YEAR_BUILT_OPTIONS, MAINFUEL_OPTIONS, PROVENANCE_OPTIONS, EnumOption, + DESCRIPTOR_FILTER_CONFIG, + DescriptorBucket, + DescriptorMatch, } from "@/app/utils/propertyFilters"; import { EPC_TO_SAP_MIN, EPC_TO_SAP_MAX } from "@/app/utils/epc"; const ENUM_FIELD_DB_OPTIONS: Record = { propertyType: PROPERTY_TYPE_OPTIONS, builtForm: BUILT_FORM_OPTIONS, - tenure: TENURE_OPTIONS, yearBuilt: YEAR_BUILT_OPTIONS, provenance: PROVENANCE_OPTIONS, mainfuel: MAINFUEL_OPTIONS, @@ -508,6 +522,50 @@ function buildEpcSapCondition( return null; } +/** An ILIKE pattern for a needle: prefix ("cavity%") or substring ("%mains gas%"). */ +function descriptorLikePattern(needle: string, match: DescriptorMatch): string { + return match === "prefix" ? `${needle}%` : `%${needle}%`; +} + +/** + * WHERE fragment matching the free-text wall/roof/heating column against the + * selected coarse buckets (mirrors classifyDescriptor). A non-empty bucket + * matches any of its needles; the "Unknown" bucket matches the complement — + * rows matching NO known needle (which includes the resolved 'Unknown' + * terminal). NULL is impossible here (resolvedXSql COALESCEs to 'Unknown'). + */ +function buildDescriptorBucketCondition( + col: ReturnType, + buckets: DescriptorBucket[], + match: DescriptorMatch, + selectedLabels: string[], +): ReturnType | null { + const allNeedles = buckets.flatMap((b) => b.needles); + const parts: ReturnType[] = []; + + for (const label of selectedLabels) { + const bucket = buckets.find((b) => b.label === label); + if (!bucket) continue; + if (bucket.needles.length === 0) { + // "Unknown" = the complement of every known needle. + if (allNeedles.length === 0) return null; + const nots = allNeedles.map( + (n) => sql`${col} NOT ILIKE ${descriptorLikePattern(n, match)}`, + ); + parts.push(sql`(${sql.join(nots, sql` AND `)})`); + } else { + const likes = bucket.needles.map( + (n) => sql`${col} ILIKE ${descriptorLikePattern(n, match)}`, + ); + parts.push(sql`(${sql.join(likes, sql` OR `)})`); + } + } + + if (parts.length === 0) return null; + if (parts.length === 1) return parts[0]; + return sql`(${sql.join(parts, sql` OR `)})`; +} + function buildConditionSql(filter: PropertyFilter): ReturnType | null { switch (filter.field) { case "address": @@ -579,7 +637,6 @@ function buildConditionSql(filter: PropertyFilter): ReturnType | nul case "propertyType": case "builtForm": - case "tenure": case "yearBuilt": case "provenance": case "mainfuel": { @@ -595,14 +652,15 @@ function buildConditionSql(filter: PropertyFilter): ReturnType | nul const options = ENUM_FIELD_DB_OPTIONS[filter.field]; const colMap: Record> = { - propertyType: sql`p.property_type`, - builtForm: sql`p.built_form`, - tenure: sql`p.tenure`, - yearBuilt: sql`p.year_built`, + // Override-resolved to match the column + the Run filter (ADR-0012). + propertyType: resolvedPropertyTypeSql, + builtForm: resolvedBuiltFormSql, + yearBuilt: resolvedConstructionAgeSql, provenance: provenanceSignalSql, - // GAP: new-approach properties have no main-fuel string → NULL, so they - // only match the "no value" filter branch. See epcSources.ts. - mainfuel: mainfuelSql(sql`epc`), + // Override-resolved (ADR-0012): the new EPC graph has no main fuel, so a + // new-approach property matches a fuel filter only via a landlord override; + // otherwise it resolves to 'Unknown'. + mainfuel: resolvedMainFuelSql(sql`epc`), }; const col = colMap[filter.field]; @@ -688,6 +746,36 @@ function buildConditionSql(filter: PropertyFilter): ReturnType | nul if (parts.length === 1) return parts[0]; return sql`(${sql.join(parts, sql` OR `)})`; } + + case "wallType": + case "roofType": + case "heatingSystem": { + // Free-text descriptors: bucket the resolved value by prefix/substring + // needles (ADR-0012), mirroring classifyDescriptor. The resolved SQL folds + // override → EPC element description → legacy → 'Unknown'. + if (filter.operator !== "enum_one_of") return null; + + let selectedLabels: string[]; + try { + selectedLabels = JSON.parse(filter.value); + } catch { + return null; + } + if (selectedLabels.length === 0) return null; + + const { buckets, match } = DESCRIPTOR_FILTER_CONFIG[filter.field]; + const colMap: Record> = { + wallType: resolvedWallTypeSql(sql`epc`), + roofType: resolvedRoofTypeSql(sql`epc`), + heatingSystem: resolvedHeatingSystemSql(sql`epc`), + }; + return buildDescriptorBucketCondition( + colMap[filter.field], + buckets, + match, + selectedLabels, + ); + } } return null; } @@ -727,6 +815,15 @@ const EPC_JOIN_FILTER_FIELDS = new Set([ "floorArea", "epcExpiryDate", "mainfuel", + // Override-resolved (ADR-0012): needs the EPC graph joins + the override join. + "propertyType", + "builtForm", + "yearBuilt", + // Free-text descriptors (ADR-0012): resolved SQL reads the EPC element + // description subquery + the override join. + "wallType", + "roofType", + "heatingSystem", ]); const PLAN_JOIN_FILTER_FIELDS = new Set(["expectedEpc"]); @@ -752,7 +849,14 @@ export async function getPropertiesCount( const epcJoins = needsEpcJoins ? sql`LEFT JOIN property_details_epc epc ON epc.property_id = p.id - ${newApproachJoins}` + ${newApproachJoins} + ${propertyTypeOverrideJoin} + ${builtFormOverrideJoin} + ${constructionAgeOverrideJoin} + ${mainFuelOverrideJoin} + ${wallTypeOverrideJoin} + ${roofTypeOverrideJoin} + ${mainHeatingOverrideJoin}` : sql``; const planJoin = needsPlanJoin ? sql`LEFT JOIN LATERAL ( @@ -864,15 +968,26 @@ export async function getProperties( COALESCE(rec.cost, 0) AS "totalRecommendationCost", p.landlord_property_id AS "landlordPropertyId", p.original_sap_points AS "originalSapPoints", - p.property_type AS "propertyType", - p.built_form AS "builtForm", - p.tenure AS tenure, - p.year_built AS "yearBuilt", + -- Property Type is resolved through the landlord-override precedence + -- (override → EPC-derived → 'Unknown'), matching the modelling Run filter. + -- See resolvePropertyType / ADR-0012. + ${resolvedPropertyTypeSql} AS "propertyType", + ${resolvedBuiltFormSql} AS "builtForm", + -- Construction Age is band-native and override-resolved (ADR-0012); the + -- "yearBuilt" key is retained to avoid a wide rename. See resolveConstructionAge. + ${resolvedConstructionAgeSql} AS "yearBuilt", ${lodgementDateSql(sql`epc`)}::text AS "epcLodgementDate", ${isExpiredSql(sql`epc`)} AS "epcIsExpired", ${totalFloorAreaSql(sql`epc`)} AS "totalFloorArea", ${carbonSql(sql`epc`)} AS "co2Emissions", - ${mainfuelSql(sql`epc`)} AS mainfuel, + -- Main Fuel: override → EPC-derived → 'Unknown'. The new EPC graph has no + -- main fuel, so overrides are the only new-approach source (ADR-0012). + ${resolvedMainFuelSql(sql`epc`)} AS mainfuel, + -- Wall / roof / heating: override → EPC element description → legacy → 'Unknown' + -- (ADR-0012). Per-part components (wall/roof) read the main building part. + ${resolvedWallTypeSql(sql`epc`)} AS "wallType", + ${resolvedRoofTypeSql(sql`epc`)} AS "roofType", + ${resolvedHeatingSystemSql(sql`epc`)} AS "heatingSystem", p.lexiscore AS lexiscore, -- Portfolio Tags on this property, name-ordered, as JSON (ADR-0013). A -- correlated subquery keeps the main query at one row per property (a JOIN @@ -920,6 +1035,13 @@ export async function getProperties( LEFT JOIN property_details_epc epc ON epc.property_id = p.id ${newApproachJoins} + ${propertyTypeOverrideJoin} + ${builtFormOverrideJoin} + ${constructionAgeOverrideJoin} + ${mainFuelOverrideJoin} + ${wallTypeOverrideJoin} + ${roofTypeOverrideJoin} + ${mainHeatingOverrideJoin} WHERE p.portfolio_id = ${portfolioId} -- Unmatched (no-UPRN) properties live in the "Needs attention" tab until -- resolved, so they're excluded from the main table. diff --git a/src/app/utils/propertyFilters.ts b/src/app/utils/propertyFilters.ts index d85609ef..56b178a6 100644 --- a/src/app/utils/propertyFilters.ts +++ b/src/app/utils/propertyFilters.ts @@ -8,12 +8,14 @@ export type FilterField = | "propertyRef" | "propertyType" | "builtForm" - | "tenure" | "yearBuilt" | "provenance" | "floorArea" | "co2Emissions" | "mainfuel" + | "wallType" + | "roofType" + | "heatingSystem" | "tags"; export type FilterOperator = @@ -65,13 +67,23 @@ export interface EnumOption { dbValues: string[]; } +// dbValues are the label vocabulary emitted by resolvedPropertyTypeSql (override +// snapshot / EPC-derived label). "Park home" is reachable from an RdSAP code (4) +// or an override; "Unknown" is the never-null terminal + an explicit override — +// it selects properties with no resolvable value at all (ADR-0012). export const PROPERTY_TYPE_OPTIONS: EnumOption[] = [ { label: "House", dbValues: ["House"] }, { label: "Flat", dbValues: ["Flat"] }, { label: "Bungalow", dbValues: ["Bungalow"] }, { label: "Maisonette", dbValues: ["Maisonette"] }, + { label: "Park home", dbValues: ["Park home"] }, + { label: "Unknown", dbValues: ["Unknown"] }, ]; +// dbValues are the label vocabulary emitted by resolvedBuiltFormSql (override +// snapshot / EPC-derived label). "Not Recorded" matches only an explicit +// override of that literal; "Unknown" is the never-null terminal + an explicit +// override — it selects properties with no resolvable value at all (ADR-0012). export const BUILT_FORM_OPTIONS: EnumOption[] = [ { label: "Detached", dbValues: ["Detached"] }, { label: "Semi-Detached", dbValues: ["Semi-Detached"] }, @@ -80,15 +92,7 @@ export const BUILT_FORM_OPTIONS: EnumOption[] = [ { label: "Enclosed End-Terrace", dbValues: ["Enclosed End-Terrace"] }, { label: "Enclosed Mid-Terrace", dbValues: ["Enclosed Mid-Terrace"] }, { label: "Not Recorded", dbValues: ["Not Recorded"] }, -]; - -export const TENURE_OPTIONS: EnumOption[] = [ - { label: "Owner-occupied", dbValues: ["Owner-occupied", "owner-occupied"] }, - { label: "Rented (Private)", dbValues: ["Rented (private)", "rental (private)", "rented (private)"] }, - { label: "Rented (Social)", dbValues: ["Rented (social)", "rental (social)"] }, - { label: "Not Defined", dbValues: ["Not defined - use in the case of a new dwelling for which the intended tenure in not known. It is not to be used for an existing dwelling"] }, - { label: "Unknown", dbValues: ["unknown"] }, - { label: "Not Recorded", dbValues: ["__null__"] }, + { label: "Unknown", dbValues: ["Unknown"] }, ]; // Provenance signal (ADR-0002). dbValues are the raw signal strings emitted by @@ -99,26 +103,210 @@ export const PROVENANCE_OPTIONS: EnumOption[] = [ { label: "Standard", dbValues: ["none"] }, ]; +// Construction Age is band-native (ADR-0012): labels + dbValues are the RdSAP +// band labels emitted by resolvedConstructionAgeSql. These MUST match +// CONSTRUCTION_AGE_BANDS[*].label in epcSources.ts exactly (note the en-dash); +// epcSources.test.ts guards against drift. "Unknown" is the never-null terminal. export const YEAR_BUILT_OPTIONS: EnumOption[] = [ - "1900","1930","1950","1967","1976","1983","1991","1996", - "2003","2007","2008","2009","2010","2011","2012","2013", - "2014","2015","2016","2017","2018","2019","2020","2021", - "2022","2023","2024","2025", -].map((y) => ({ label: y, dbValues: [y] })); + "before 1900", "1900–1929", "1930–1949", "1950–1966", "1967–1975", + "1976–1982", "1983–1990", "1991–1995", "1996–2002", "2003–2006", + "2007–2011", "2012–2022", "2023 onwards", "Unknown", +].map((label) => ({ label, dbValues: [label] })); -export const MAINFUEL_OPTIONS: EnumOption[] = [ - { label: "Mains Gas", dbValues: ["Gas mains gas", "Mains gas community", "Mains gas not community", "Mains gas this is for backwards compatibility only and should not be used"] }, - { label: "LPG", dbValues: ["Bottled lpg", "Lpg community", "Lpg not community", "Lpg this is for backwards compatibility only and should not be used"] }, - { label: "Oil", dbValues: ["Oil heating oil", "Oil community", "Oil not community", "Oil this is for backwards compatibility only and should not be used"] }, - { label: "Electricity", dbValues: ["Electricity electricity unspecified tariff", "Electricity community", "Electricity not community", "Electricity this is for backwards compatibility only and should not be used"] }, - { label: "Biomass", dbValues: ["Bulk wood pellets", "Wood chips", "Wood logs", "Biomass community", "Biomass this is for backwards compatibility only and should not be used"] }, - { label: "Coal", dbValues: ["Anthracite", "House coal not community", "House coal this is for backwards compatibility only and should not be used", "Smokeless coal", "Coal community"] }, - { label: "Dual Fuel (Mineral + Wood)", dbValues: ["Dual fuel mineral wood"] }, +// RdSAP `main_fuel` code → coarse label. The new EPC graph stores the main fuel +// as this numeric code on epc_main_heating_detail.main_fuel_type (see mainfuelSql +// / resolveMainFuel), so every code that can appear must fold into a filter +// option or an overridden/EPC-derived fuel would display but be unfilterable +// (ADR-0012). Codes are the authoritative Model `epc_codes.csv` `main_fuel` table +// (RdSAP), cross-checked against production epc rows: 26→mains gas, 29/39→electric, +// 0/20/25/31/57→"Community scheme", 5/15/33→coal, 6→wood logs, 9→dual fuel. All +// "(community)"/heat-network codes bucket to "Community Heat Network" — being on a +// communal scheme is the salient fact; the scheme's own fuel is a secondary detail. +export const RDSAP_MAIN_FUEL_CODE_LABELS: Record = { + // Mains gas + "1": "Mains Gas", "26": "Mains Gas", + // LPG + "2": "LPG", "3": "LPG", "17": "LPG", "27": "LPG", "38": "LPG", + // Oil + "4": "Oil", "28": "Oil", + // Electricity + "10": "Electricity", "29": "Electricity", "39": "Electricity", + // Coal / mineral solid fuel (anthracite, smokeless, house coal) + "5": "Coal", "14": "Coal", "15": "Coal", "33": "Coal", + // Biomass / wood (logs, pellets, chips) + "6": "Biomass", "7": "Biomass", "8": "Biomass", "12": "Biomass", "16": "Biomass", + // Dual fuel + "9": "Dual Fuel (Mineral + Wood)", + // Biogas + "13": "Biogas", "51": "Biogas", + // Bioethanol + "19": "Bioethanol", "76": "Bioethanol", + // Biodiesel (from biomass / used cooking oil / vegetable / rape seed oil) + "34": "Biodiesel", "35": "Biodiesel", "36": "Biodiesel", "37": "Biodiesel", + "71": "Biodiesel", "72": "Biodiesel", "73": "Biodiesel", "74": "Biodiesel", + // Biodiesel blend + "18": "B30k (Biodiesel blend)", "75": "B30k (Biodiesel blend)", + // Waste combustion + "11": "Waste Combustion", + // Community / heat-network (the dwelling is on a communal scheme) + "0": "Community Heat Network", "20": "Community Heat Network", "21": "Community Heat Network", + "22": "Community Heat Network", "23": "Community Heat Network", "24": "Community Heat Network", + "25": "Community Heat Network", "30": "Community Heat Network", "31": "Community Heat Network", + "32": "Community Heat Network", "41": "Community Heat Network", "42": "Community Heat Network", + "43": "Community Heat Network", "44": "Community Heat Network", "45": "Community Heat Network", + "46": "Community Heat Network", "47": "Community Heat Network", "48": "Community Heat Network", + "49": "Community Heat Network", "50": "Community Heat Network", "52": "Community Heat Network", + "53": "Community Heat Network", "54": "Community Heat Network", "55": "Community Heat Network", + "56": "Community Heat Network", "57": "Community Heat Network", "58": "Community Heat Network", + "99": "Community Heat Network", +}; + +// EPR free-text main_fuel_type strings (Elmhurst/PASHub, ~5% of new-approach +// rows) folded to the same labels. The empty-string sentinel is NULLIF'd to NULL +// upstream and resolves to 'Unknown'. +export const EPR_MAIN_FUEL_STRING_LABELS: Record = { + "Mains gas": "Mains Gas", + "Electricity": "Electricity", + "Bulk LPG": "LPG", + "Heating oil": "Oil", +}; + +// dbValues fold several vocabularies onto one user-facing label: the raw legacy +// EPC mainfuel strings (property_details_epc), the landlord main_fuel override +// values ("mains gas", "LPG (bulk)", … — see MainFuelValues), the new-approach +// RdSAP numeric codes (RDSAP_MAIN_FUEL_CODE_LABELS) and the EPR strings +// (EPR_MAIN_FUEL_STRING_LABELS). resolvedMainFuelSql emits whichever applies; all +// must resolve to the same chip (ADR-0012). descriptorResolution.test asserts +// every override value, RdSAP code and EPR string is covered here. +const MAINFUEL_OPTIONS_BASE: EnumOption[] = [ + { label: "Mains Gas", dbValues: ["Gas mains gas", "Mains gas community", "Mains gas not community", "Mains gas this is for backwards compatibility only and should not be used", "mains gas", "mains gas (community)"] }, + { label: "LPG", dbValues: ["Bottled lpg", "Lpg community", "Lpg not community", "Lpg this is for backwards compatibility only and should not be used", "LPG (bulk)", "bottled LPG", "LPG special condition"] }, + { label: "Oil", dbValues: ["Oil heating oil", "Oil community", "Oil not community", "Oil this is for backwards compatibility only and should not be used", "oil"] }, + { label: "Electricity", dbValues: ["Electricity electricity unspecified tariff", "Electricity community", "Electricity not community", "Electricity this is for backwards compatibility only and should not be used", "electricity", "electricity (community)"] }, + { label: "Biomass", dbValues: ["Bulk wood pellets", "Wood chips", "Wood logs", "Biomass community", "Biomass this is for backwards compatibility only and should not be used", "biomass (community)", "wood logs"] }, + { label: "Coal", dbValues: ["Anthracite", "House coal not community", "House coal this is for backwards compatibility only and should not be used", "Smokeless coal", "Coal community", "house coal", "smokeless coal"] }, + { label: "Dual Fuel (Mineral + Wood)", dbValues: ["Dual fuel mineral wood", "dual fuel (mineral and wood)"] }, { label: "Biogas", dbValues: ["Biogas not community"] }, + { label: "Bioethanol", dbValues: [] }, { label: "Biodiesel", dbValues: ["Heat from boilers using biodiesel from any biomass source community"] }, { label: "B30d (Biodiesel blend)", dbValues: ["B30d community"] }, { label: "B30k (Biodiesel blend)", dbValues: ["B30k not community"] }, + { label: "Waste Combustion", dbValues: [] }, { label: "Community Heat Network", dbValues: ["From heat network data community"] }, { label: "No Heating System", dbValues: ["To be used only when there is no heatinghotwater system", "To be used only when there is no heatinghotwater system or data is from a community network"] }, - { label: "Unknown / No Data", dbValues: ["UNKNOWN", "NO DATA!"] }, + { label: "Unknown / No Data", dbValues: ["UNKNOWN", "NO DATA!", "Unknown"] }, ]; + +/** Fold the RdSAP code + EPR string vocabularies into each label's dbValues. */ +export const MAINFUEL_OPTIONS: EnumOption[] = MAINFUEL_OPTIONS_BASE.map((opt) => { + const codes = Object.entries(RDSAP_MAIN_FUEL_CODE_LABELS) + .filter(([, label]) => label === opt.label) + .map(([code]) => code); + const strings = Object.entries(EPR_MAIN_FUEL_STRING_LABELS) + .filter(([, label]) => label === opt.label) + .map(([str]) => str); + return { ...opt, dbValues: [...opt.dbValues, ...codes, ...strings] }; +}); + +/* ----------------------------------------------------------------------- + Free-text descriptor buckets: Wall Type / Roof Type / Heating System (ADR-0012) + + Unlike the enum fields above (exact dbValues), the resolved wall/roof/heating + value is a free-text description (override vocabulary, EPC + epc_energy_element.description, or legacy property_details_epc column) with + HIGH cardinality — 88+ distinct EPC wall descriptions, 245 legacy, plus + insulation-thickness variants, "Average thermal transmittance …" U-values and + encoding corruption. Exact matching can't scale, so each coarse bucket is a set + of case-insensitive NEEDLES matched by prefix (wall/roof) or substring + (heating, whose fuel follows a comma). classifyDescriptor is the pure twin of + the SQL bucketing in the portfolio query's buildConditionSql — keep in sync. + Ordering matters: the first bucket whose needle matches wins. +------------------------------------------------------------------------ */ + +export interface DescriptorBucket { + label: string; + /** Lowercase needles, matched per the field's DescriptorMatch mode. Empty = the Unknown terminal (matches the complement). */ + needles: string[]; +} + +export type DescriptorMatch = "prefix" | "contains"; + +// Wall construction. Prefix-matched on the leading construction term. "Stone" +// folds sandstone/limestone + granite/whinstone; "Average thermal transmittance …" +// (a U-value, not a construction type) and corruption fall through to Unknown. +export const WALL_TYPE_OPTIONS: DescriptorBucket[] = [ + { label: "Cavity", needles: ["cavity"] }, + { label: "Solid Brick", needles: ["solid brick"] }, + { label: "Timber Frame", needles: ["timber frame"] }, + { label: "Stone", needles: ["sandstone", "granite"] }, + { label: "System Built", needles: ["system built"] }, + { label: "Park Home", needles: ["park home"] }, + { label: "Cob", needles: ["cob"] }, + { label: "Curtain Wall", needles: ["curtain"] }, + { label: "Basement", needles: ["basement"] }, + { label: "Unknown", needles: [] }, +]; + +// Roof form. Prefix-matched. "Premises Above" folds the "(another dwelling +// above)" / "(other premises above)" family (a flat with a dwelling above it, so +// no own roof). U-values fall through to Unknown. +export const ROOF_TYPE_OPTIONS: DescriptorBucket[] = [ + { label: "Pitched", needles: ["pitched"] }, + { label: "Flat", needles: ["flat"] }, + { label: "Room-in-Roof", needles: ["roof room"] }, + { label: "Thatched", needles: ["thatched"] }, + { label: "Premises Above", needles: ["(another", "(other", "(same", "another dwelling", "another premises", "other premises"] }, + { label: "Unknown", needles: [] }, +]; + +// Heating system. Substring-matched (the fuel follows a comma, e.g. "Boiler and +// radiators, mains gas"). Order is load-bearing: Heat Pump / Community / Solid +// Fuel / Oil / LPG are checked before the gas + electric buckets so a +// heat-pump/community/wood/oil/LPG system isn't miscaught by "gas"/"electric"; +// Gas Room Heater before Gas Boiler; Electric Storage before Direct Electric. +export const HEATING_SYSTEM_OPTIONS: DescriptorBucket[] = [ + { label: "Heat Pump", needles: ["heat pump"] }, + { label: "Community", needles: ["community"] }, + { label: "Solid Fuel", needles: ["coal", "anthracite", "smokeless", "wood", "dual fuel", "biomass", "solid fuel"] }, + // NB: bare "oil" is a substring of "b(oil)er" — match the fuel token only (", oil" + // as in "Boiler and radiators, oil") and the leading override "Oil room heater". + { label: "Oil", needles: [", oil", "oil room heater"] }, + { label: "LPG", needles: ["lpg"] }, + { label: "Gas Room Heater", needles: ["gas room heater", "room heaters, mains gas", "room heaters, gas"] }, + { label: "Gas Boiler", needles: ["mains gas", "gas boiler", "gas cpsu"] }, + { label: "Electric Storage", needles: ["storage heater"] }, + { label: "Direct Electric", needles: ["electric"] }, + { label: "Unknown", needles: [] }, +]; + +/** Filter config per descriptor field: its buckets + how needles are matched. */ +export const DESCRIPTOR_FILTER_CONFIG: Record< + "wallType" | "roofType" | "heatingSystem", + { buckets: DescriptorBucket[]; match: DescriptorMatch } +> = { + wallType: { buckets: WALL_TYPE_OPTIONS, match: "prefix" }, + roofType: { buckets: ROOF_TYPE_OPTIONS, match: "prefix" }, + heatingSystem: { buckets: HEATING_SYSTEM_OPTIONS, match: "contains" }, +}; + +/** + * Classify a resolved free-text descriptor into its coarse filter bucket: the + * first bucket with a matching needle, else "Unknown". Pure twin of the SQL + * bucketing in buildConditionSql (portfolio query) — keep in sync. + */ +export function classifyDescriptor( + value: string | null | undefined, + buckets: DescriptorBucket[], + match: DescriptorMatch, +): string { + if (value == null) return "Unknown"; + const v = value.toLowerCase(); + for (const bucket of buckets) { + for (const needle of bucket.needles) { + if (match === "prefix" ? v.startsWith(needle) : v.includes(needle)) { + return bucket.label; + } + } + } + return "Unknown"; +} diff --git a/src/lib/modellingRuns/server.ts b/src/lib/modellingRuns/server.ts index 990915a7..f6c77b01 100644 --- a/src/lib/modellingRuns/server.ts +++ b/src/lib/modellingRuns/server.ts @@ -11,7 +11,12 @@ import { parseRunConfig, selectionSummary, } from "./model"; -import { builtFormTypeSql, propertyTypeSql } from "@/lib/services/epcSources"; +import { + builtFormOverrideJoin, + propertyTypeOverrideJoin, + resolvedBuiltFormSql, + resolvedPropertyTypeSql, +} from "@/lib/services/epcSources"; // Distributor entry point on the Model backend. It fans the run out to // workers, attaching execution sub-tasks to the task_id we pass (ADR-0008). @@ -154,14 +159,14 @@ export async function previewModellingRun(args: { per_scenario: { scenario_id: string; n: number }[] | null; }>(sql` WITH resolved AS ( + -- Resolution rule shared with the portfolio list via epcSources fragments + -- so the preview and the list can never drift (ADR-0008, ADR-0012). SELECT p.id, p.postcode, - COALESCE(pot.override_value, ${propertyTypeSql}, 'Unknown') AS ptype, - COALESCE(pobf.override_value, ${builtFormTypeSql}, 'Unknown') AS bform + ${resolvedPropertyTypeSql} AS ptype, + ${resolvedBuiltFormSql} AS bform FROM property p - LEFT JOIN property_overrides pot ON pot.property_id = p.id - AND pot.override_component = 'property_type' AND pot.building_part = 0 - LEFT JOIN property_overrides pobf ON pobf.property_id = p.id - AND pobf.override_component = 'built_form_type' AND pobf.building_part = 0 + ${propertyTypeOverrideJoin} + ${builtFormOverrideJoin} LEFT JOIN epc_property epl ON epl.property_id = p.id AND epl.source = 'lodged' LEFT JOIN epc_property epp ON epp.property_id = p.id AND epp.source = 'predicted' WHERE p.portfolio_id = ${pid} AND p.marked_for_deletion = false diff --git a/src/lib/services/descriptorResolution.test.ts b/src/lib/services/descriptorResolution.test.ts new file mode 100644 index 00000000..1c487586 --- /dev/null +++ b/src/lib/services/descriptorResolution.test.ts @@ -0,0 +1,521 @@ +import { describe, expect, it } from "vitest"; +import { + resolveBuiltForm, + resolveConstructionAge, + resolveMainFuel, + resolveOverrideDescriptor, + resolvePropertyType, +} from "./descriptorResolution"; +import { + MAINFUEL_OPTIONS, + RDSAP_MAIN_FUEL_CODE_LABELS, + EPR_MAIN_FUEL_STRING_LABELS, + WALL_TYPE_OPTIONS, + ROOF_TYPE_OPTIONS, + HEATING_SYSTEM_OPTIONS, + DESCRIPTOR_FILTER_CONFIG, + classifyDescriptor, +} from "@/app/utils/propertyFilters"; +import { + MainFuelValues, + WallTypeValues, + RoofTypeValues, + MainHeatingSystemValues, +} from "@/app/db/schema/landlord_overrides"; + +describe("resolvePropertyType", () => { + it("prefers a landlord override over the EPC-derived value", () => { + expect( + resolvePropertyType({ + override: "Maisonette", + isNewApproach: true, + lodgedPropertyType: "2", // EPC code for Flat + lodgedDwellingType: null, + predictedPropertyType: null, + predictedDwellingType: null, + legacyPropertyType: null, + }), + ).toBe("Maisonette"); + }); + + it("maps an RdSAP property-type code to its label when there is no override", () => { + expect( + resolvePropertyType({ + override: null, + isNewApproach: true, + lodgedPropertyType: "0", // RdSAP code for House + lodgedDwellingType: null, + predictedPropertyType: null, + predictedDwellingType: null, + legacyPropertyType: null, + }), + ).toBe("House"); + }); + + it("falls back to the predicted EPC when there is no lodged value", () => { + expect( + resolvePropertyType({ + override: null, + isNewApproach: true, + lodgedPropertyType: null, + lodgedDwellingType: null, + predictedPropertyType: "2", // RdSAP code for Flat + predictedDwellingType: null, + legacyPropertyType: null, + }), + ).toBe("Flat"); + }); + + it("prefers the lodged EPC over the predicted EPC when both exist", () => { + expect( + resolvePropertyType({ + override: null, + isNewApproach: true, + lodgedPropertyType: "0", // House + lodgedDwellingType: null, + predictedPropertyType: "2", // Flat + predictedDwellingType: null, + legacyPropertyType: null, + }), + ).toBe("House"); + }); + + it("falls back to dwelling_type when the lodged property_type is absent", () => { + expect( + resolvePropertyType({ + override: null, + isNewApproach: true, + lodgedPropertyType: null, + lodgedDwellingType: "Flat", + predictedPropertyType: null, + predictedDwellingType: null, + legacyPropertyType: null, + }), + ).toBe("Flat"); + }); + + it("reads the legacy property-row value for a legacy property", () => { + expect( + resolvePropertyType({ + override: null, + isNewApproach: false, + lodgedPropertyType: "0", // must be ignored for legacy + lodgedDwellingType: null, + predictedPropertyType: null, + predictedDwellingType: null, + legacyPropertyType: "Bungalow", + }), + ).toBe("Bungalow"); + }); + + it("never reads the EPC graph nor the legacy row across the cutoff (new-approach ignores the legacy row)", () => { + expect( + resolvePropertyType({ + override: null, + isNewApproach: true, + lodgedPropertyType: null, + lodgedDwellingType: null, + predictedPropertyType: null, + predictedDwellingType: null, + legacyPropertyType: "Bungalow", // legacy row must NOT surface for a new-approach property + }), + ).toBe("Unknown"); + }); + + // Spec locks for decisions taken in ADR-0012 — an override always wins (even an + // explicit "Unknown"), and an unrecognised RdSAP code is passed through as-is. + it("lets an explicit 'Unknown' override win over a known EPC value", () => { + expect( + resolvePropertyType({ + override: "Unknown", + isNewApproach: true, + lodgedPropertyType: "0", // EPC knows "House" — the override still wins + lodgedDwellingType: null, + predictedPropertyType: null, + predictedDwellingType: null, + legacyPropertyType: null, + }), + ).toBe("Unknown"); + }); + + it("passes an unrecognised RdSAP code through unchanged", () => { + expect( + resolvePropertyType({ + override: null, + isNewApproach: true, + lodgedPropertyType: "9", // not in PROPERTY_TYPE_LABELS + lodgedDwellingType: null, + predictedPropertyType: null, + predictedDwellingType: null, + legacyPropertyType: null, + }), + ).toBe("9"); + }); +}); + +describe("resolveBuiltForm", () => { + it("prefers a landlord override over the EPC-derived value", () => { + expect( + resolveBuiltForm({ + override: "Detached", + isNewApproach: true, + lodgedBuiltForm: "4", // EPC code for Mid-Terrace + predictedBuiltForm: null, + legacyBuiltForm: null, + }), + ).toBe("Detached"); + }); + + it("maps an RdSAP built-form code to its label when there is no override", () => { + expect( + resolveBuiltForm({ + override: null, + isNewApproach: true, + lodgedBuiltForm: "2", // RdSAP code for Semi-Detached + predictedBuiltForm: null, + legacyBuiltForm: null, + }), + ).toBe("Semi-Detached"); + }); + + it("falls back to the predicted EPC when there is no lodged value", () => { + expect( + resolveBuiltForm({ + override: null, + isNewApproach: true, + lodgedBuiltForm: null, + predictedBuiltForm: "1", // Detached + legacyBuiltForm: null, + }), + ).toBe("Detached"); + }); + + it("reads the legacy property-row value for a legacy property", () => { + expect( + resolveBuiltForm({ + override: null, + isNewApproach: false, + lodgedBuiltForm: "1", // must be ignored for legacy + predictedBuiltForm: null, + legacyBuiltForm: "End-Terrace", + }), + ).toBe("End-Terrace"); + }); + + it("resolves to 'Unknown' when nothing is available (never null)", () => { + expect( + resolveBuiltForm({ + override: null, + isNewApproach: true, + lodgedBuiltForm: null, + predictedBuiltForm: null, + legacyBuiltForm: "Detached", // legacy row must NOT surface for new-approach + }), + ).toBe("Unknown"); + }); +}); + +describe("resolveConstructionAge", () => { + it("maps a landlord override band letter to its band label", () => { + expect( + resolveConstructionAge({ + override: "B", + isNewApproach: true, + lodgedAgeBand: "F", // EPC says a different band — override still wins + predictedAgeBand: null, + legacyYearBuilt: null, + }), + ).toBe("1900–1929"); + }); + + it("maps the lodged EPC band letter to its label when there is no override", () => { + expect( + resolveConstructionAge({ + override: null, + isNewApproach: true, + lodgedAgeBand: "C", + predictedAgeBand: null, + legacyYearBuilt: null, + }), + ).toBe("1930–1949"); + }); + + it("falls back to the predicted EPC band when there is no lodged band", () => { + expect( + resolveConstructionAge({ + override: null, + isNewApproach: true, + lodgedAgeBand: null, + predictedAgeBand: "M", + legacyYearBuilt: null, + }), + ).toBe("2023 onwards"); + }); + + it("buckets a legacy year_built into its band (mid-range)", () => { + expect(legacyAge("1985")).toBe("1983–1990"); + }); + + it("buckets a legacy year before 1900 into band A", () => { + expect(legacyAge("1899")).toBe("before 1900"); + }); + + it("buckets the first year of a band correctly (boundary)", () => { + expect(legacyAge("1900")).toBe("1900–1929"); + }); + + it("buckets a recent legacy year into the open-ended final band", () => { + expect(legacyAge("2025")).toBe("2023 onwards"); + }); + + it("resolves a non-numeric legacy year_built to 'Unknown'", () => { + expect(legacyAge("N/A")).toBe("Unknown"); + }); + + it("resolves to 'Unknown' when nothing is available; ignores the legacy row for new-approach", () => { + expect( + resolveConstructionAge({ + override: null, + isNewApproach: true, + lodgedAgeBand: null, + predictedAgeBand: null, + legacyYearBuilt: "1985", // must NOT surface for a new-approach property + }), + ).toBe("Unknown"); + }); + + it("lets an explicit 'Unknown' override win", () => { + expect( + resolveConstructionAge({ + override: "Unknown", + isNewApproach: true, + lodgedAgeBand: "C", + predictedAgeBand: null, + legacyYearBuilt: null, + }), + ).toBe("Unknown"); + }); +}); + +describe("resolveMainFuel", () => { + const base = { + override: null, + isNewApproach: true, + lodgedMainFuel: null, + predictedMainFuel: null, + legacyMainFuel: null, + }; + + it("prefers a landlord override over the EPC-derived value", () => { + expect( + resolveMainFuel({ ...base, override: "mains gas", lodgedMainFuel: "26" }), + ).toBe("mains gas"); + }); + + it("reads the new-approach EPC main-fuel code (lodged) as a raw value", () => { + expect(resolveMainFuel({ ...base, lodgedMainFuel: "26" })).toBe("26"); + }); + + it("falls back to the predicted EPC main fuel when there is no lodged one", () => { + expect(resolveMainFuel({ ...base, predictedMainFuel: "29" })).toBe("29"); + }); + + it("prefers the lodged EPC main fuel over the predicted one", () => { + expect( + resolveMainFuel({ ...base, lodgedMainFuel: "26", predictedMainFuel: "29" }), + ).toBe("26"); + }); + + it("passes an EPR string main fuel through unchanged", () => { + expect(resolveMainFuel({ ...base, lodgedMainFuel: "Mains gas" })).toBe("Mains gas"); + }); + + it("resolves to 'Unknown' for a new-approach property with no EPC main fuel; ignores the legacy row", () => { + expect( + resolveMainFuel({ ...base, legacyMainFuel: "Gas mains gas" }), + ).toBe("Unknown"); + }); + + it("reads the legacy EPC main fuel for a legacy property", () => { + expect( + resolveMainFuel({ + ...base, + isNewApproach: false, + lodgedMainFuel: "26", // must be ignored for legacy + legacyMainFuel: "Gas mains gas", + }), + ).toBe("Gas mains gas"); + }); + + it("resolves to 'Unknown' when nothing is available (never null)", () => { + expect(resolveMainFuel({ ...base, isNewApproach: false })).toBe("Unknown"); + }); +}); + +describe("MAINFUEL_OPTIONS coverage", () => { + const covered = new Set(MAINFUEL_OPTIONS.flatMap((o) => o.dbValues)); + const labels = new Set(MAINFUEL_OPTIONS.map((o) => o.label)); + + // Every landlord main-fuel override value must fold into a filter option, or an + // overridden fuel would display but be unfilterable (ADR-0012). + it("covers every landlord main-fuel override value", () => { + for (const value of MainFuelValues) { + expect(covered).toContain(value); + } + }); + + // Every RdSAP main_fuel code the new EPC graph can store must fold into a + // filter option under a real label (a typo'd label would be silently dropped + // by the MAINFUEL_OPTIONS merge — assert both the code AND its label). + it("folds every RdSAP main_fuel code under a real label", () => { + for (const [code, label] of Object.entries(RDSAP_MAIN_FUEL_CODE_LABELS)) { + expect(labels).toContain(label); + expect(covered).toContain(code); + } + }); + + // Every EPR free-text main_fuel string must fold in too. + it("folds every EPR main-fuel string under a real label", () => { + for (const [str, label] of Object.entries(EPR_MAIN_FUEL_STRING_LABELS)) { + expect(labels).toContain(label); + expect(covered).toContain(str); + } + }); +}); + +describe("resolveOverrideDescriptor (wall / roof / heating free-text)", () => { + const base = { + override: null, + isNewApproach: true, + lodgedDescription: null, + predictedDescription: null, + legacyDescription: null, + }; + + it("prefers a landlord override over the EPC description", () => { + expect( + resolveOverrideDescriptor({ + ...base, + override: "Cavity wall, filled cavity", + lodgedDescription: "Solid brick, as built, no insulation (assumed)", + }), + ).toBe("Cavity wall, filled cavity"); + }); + + it("uses the lodged EPC description when there is no override", () => { + expect( + resolveOverrideDescriptor({ + ...base, + lodgedDescription: "Pitched, 250 mm loft insulation", + }), + ).toBe("Pitched, 250 mm loft insulation"); + }); + + it("falls back to the predicted EPC description when there is no lodged one", () => { + expect( + resolveOverrideDescriptor({ ...base, predictedDescription: "Flat, insulated (assumed)" }), + ).toBe("Flat, insulated (assumed)"); + }); + + it("reads the legacy description for a legacy property", () => { + expect( + resolveOverrideDescriptor({ + ...base, + isNewApproach: false, + lodgedDescription: "ignored for legacy", + legacyDescription: "Boiler and radiators, mains gas", + }), + ).toBe("Boiler and radiators, mains gas"); + }); + + it("resolves to 'Unknown' when nothing is available; ignores legacy for new-approach", () => { + expect( + resolveOverrideDescriptor({ ...base, legacyDescription: "must not surface" }), + ).toBe("Unknown"); + }); +}); + +describe("classifyDescriptor — wall / roof / heating buckets", () => { + const wall = (v: string) => + classifyDescriptor(v, WALL_TYPE_OPTIONS, DESCRIPTOR_FILTER_CONFIG.wallType.match); + const roof = (v: string) => + classifyDescriptor(v, ROOF_TYPE_OPTIONS, DESCRIPTOR_FILTER_CONFIG.roofType.match); + const heat = (v: string) => + classifyDescriptor(v, HEATING_SYSTEM_OPTIONS, DESCRIPTOR_FILTER_CONFIG.heatingSystem.match); + + // Every landlord override value must fall into a real (non-Unknown) bucket, or + // an overridden component would display but be unfilterable (ADR-0012). The + // literal "Unknown" is the sole exception — it maps to the Unknown bucket. + const assertVocabCovered = ( + values: readonly string[], + classify: (v: string) => string, + labels: Set, + ) => { + for (const value of values) { + const bucket = classify(value); + expect(labels).toContain(bucket); + if (value !== "Unknown") expect(bucket).not.toBe("Unknown"); + } + }; + + it("buckets every WallType override value", () => { + assertVocabCovered(WallTypeValues, wall, new Set(WALL_TYPE_OPTIONS.map((b) => b.label))); + }); + it("buckets every RoofType override value", () => { + assertVocabCovered(RoofTypeValues, roof, new Set(ROOF_TYPE_OPTIONS.map((b) => b.label))); + }); + it("buckets every MainHeatingSystem override value", () => { + assertVocabCovered( + MainHeatingSystemValues, + heat, + new Set(HEATING_SYSTEM_OPTIONS.map((b) => b.label)), + ); + }); + + // Representative EPC / legacy descriptions (from production) bucket as expected. + it("buckets representative EPC/legacy WALL descriptions", () => { + expect(wall("Cavity wall, filled cavity")).toBe("Cavity"); + expect(wall("Solid brick, as built, no insulation (assumed)")).toBe("Solid Brick"); + expect(wall("Timber frame, as built, insulated (assumed)")).toBe("Timber Frame"); + expect(wall("Sandstone or limestone, as built, no insulation (assumed)")).toBe("Stone"); + expect(wall("Granite or whinstone, as built, no insulation (assumed)")).toBe("Stone"); + expect(wall("System built, with external insulation")).toBe("System Built"); + // A bare U-value is not a construction type → Unknown. + expect(wall("Average thermal transmittance 0.21 W/m²K")).toBe("Unknown"); + }); + + it("buckets representative EPC/legacy ROOF descriptions", () => { + expect(roof("Pitched, 200 mm loft insulation")).toBe("Pitched"); + expect(roof("Flat, no insulation (assumed)")).toBe("Flat"); + expect(roof("Roof room(s), insulated (assumed)")).toBe("Room-in-Roof"); + expect(roof("(another dwelling above)")).toBe("Premises Above"); + expect(roof("(other premises above)")).toBe("Premises Above"); + expect(roof("Another Premises Above")).toBe("Premises Above"); + expect(roof("Average thermal transmittance 0.11 W/m²K")).toBe("Unknown"); + }); + + it("buckets representative EPC/legacy HEATING descriptions", () => { + expect(heat("Boiler and radiators, mains gas")).toBe("Gas Boiler"); + expect(heat("Room heaters, mains gas")).toBe("Gas Room Heater"); + expect(heat("Community scheme")).toBe("Community"); + expect(heat("Community scheme, biomass")).toBe("Community"); + expect(heat("Electric storage heaters")).toBe("Electric Storage"); + expect(heat("Room heaters, electric")).toBe("Direct Electric"); + expect(heat("Air source heat pump, radiators, electric")).toBe("Heat Pump"); + // "oil" is a substring of "boiler" — a gas boiler must NOT bucket as Oil. + expect(heat("Boiler and radiators, oil")).toBe("Oil"); + expect(heat("Boiler and radiators, dual fuel (mineral and wood)")).toBe("Solid Fuel"); + expect(heat("Boiler and radiators, lpg")).toBe("LPG"); + }); +}); + +/** Legacy-property helper: only year_built is set, cutoff is false. */ +function legacyAge(year: string): string { + return resolveConstructionAge({ + override: null, + isNewApproach: false, + lodgedAgeBand: null, + predictedAgeBand: null, + legacyYearBuilt: year, + }); +} diff --git a/src/lib/services/descriptorResolution.ts b/src/lib/services/descriptorResolution.ts new file mode 100644 index 00000000..da166b64 --- /dev/null +++ b/src/lib/services/descriptorResolution.ts @@ -0,0 +1,179 @@ +/** + * Pure precedence resolvers for the portfolio-list property descriptors. + * + * Mirrors the provenance.ts split: the DB query gathers the raw inputs (landlord + * override snapshot, lodged/predicted EPC values, legacy property-row value) and + * these functions own the precedence + RdSAP code→label mapping. The SQL twins + * in epcSources.ts (resolvedPropertyTypeSql, …) mirror this logic — keep in sync. + * + * Precedence (ADR-0008, extended to the list by ADR-0012): + * landlord override → EPC-derived (lodged over predicted, codes mapped; + * legacy property-row fallback) → 'Unknown' (never null) + */ +import { + BUILT_FORM_LABELS, + CONSTRUCTION_AGE_BANDS, + PROPERTY_TYPE_LABELS, +} from "./epcSources"; + +export interface PropertyTypeInputs { + /** property_overrides.override_value for component 'property_type', building_part 0. */ + override: string | null; + /** Whether the property reads the new EPC graph (updated_at >= cutoff). */ + isNewApproach: boolean; + /** epc_property(source='lodged').property_type — RdSAP code or text. */ + lodgedPropertyType: string | null; + /** epc_property(source='lodged').dwelling_type — fallback when property_type is absent. */ + lodgedDwellingType: string | null; + predictedPropertyType: string | null; + predictedDwellingType: string | null; + /** property.property_type — the legacy row value. */ + legacyPropertyType: string | null; +} + +export function resolvePropertyType(i: PropertyTypeInputs): string { + if (i.override) return i.override; + const derived = i.isNewApproach + ? epcPropertyType(i.lodgedPropertyType, i.lodgedDwellingType) ?? + epcPropertyType(i.predictedPropertyType, i.predictedDwellingType) + : i.legacyPropertyType; + return derived ?? "Unknown"; +} + +/** + * One EPC source's property-type label: the RdSAP code mapped to a label (text + * passes through), falling back to dwelling_type. Mirrors epcPropertyTypeSql. + */ +function epcPropertyType( + propertyType: string | null, + dwellingType: string | null, +): string | null { + if (propertyType != null) return PROPERTY_TYPE_LABELS[propertyType] ?? propertyType; + return dwellingType; +} + +export interface BuiltFormInputs { + /** property_overrides.override_value for component 'built_form_type', building_part 0. */ + override: string | null; + isNewApproach: boolean; + /** epc_property(source='lodged').built_form — RdSAP code or text. */ + lodgedBuiltForm: string | null; + predictedBuiltForm: string | null; + /** property.built_form — the legacy row value. */ + legacyBuiltForm: string | null; +} + +export function resolveBuiltForm(i: BuiltFormInputs): string { + if (i.override) return i.override; + const derived = i.isNewApproach + ? epcBuiltForm(i.lodgedBuiltForm) ?? epcBuiltForm(i.predictedBuiltForm) + : i.legacyBuiltForm; + return derived ?? "Unknown"; +} + +/** One EPC source's built-form label: RdSAP code → label, text passes through. Mirrors builtFormTypeSql. */ +function epcBuiltForm(builtForm: string | null): string | null { + if (builtForm == null) return null; + return BUILT_FORM_LABELS[builtForm] ?? builtForm; +} + +export interface ConstructionAgeInputs { + /** property_overrides.override_value for 'construction_age_band', building_part 0 — an RdSAP letter (A..M) or "Unknown". */ + override: string | null; + isNewApproach: boolean; + /** The main building part's epc_building_part.construction_age_band — an RdSAP letter. */ + lodgedAgeBand: string | null; + predictedAgeBand: string | null; + /** property.year_built — a raw year string (legacy). Bucketed into a band. */ + legacyYearBuilt: string | null; +} + +/** + * Construction age as an RdSAP band LABEL ("before 1900" … "2023 onwards", + * "Unknown") — the band is the underlying datum, never a single year (ADR-0012). + * Override and EPC both store the band letter; legacy year_built is bucketed into + * a band. Mirrors resolvedConstructionAgeSql. + */ +export function resolveConstructionAge(i: ConstructionAgeInputs): string { + if (i.override) return bandLetterToLabel(i.override); + const band = i.isNewApproach + ? i.lodgedAgeBand ?? i.predictedAgeBand + : bucketLegacyYearToBand(i.legacyYearBuilt); + return band != null ? bandLetterToLabel(band) : "Unknown"; +} + +/** RdSAP band letter → band label; unrecognised values (e.g. "Unknown") pass through. */ +function bandLetterToLabel(letter: string): string { + return CONSTRUCTION_AGE_BANDS[letter]?.label ?? letter; +} + +/** + * Bucket a legacy raw year_built into its RdSAP band letter: the band with the + * greatest start year not after the given year. Non-numeric text → null (Unknown); + * a year below the earliest start is pre-1900 → band A. Mirrors the legacy branch + * of resolvedConstructionAgeSql. + */ +export interface OverrideDescriptorInputs { + /** property_overrides.override_value for this component, building_part 0. */ + override: string | null; + isNewApproach: boolean; + /** epc_energy_element.description for the element on the lodged EPC. */ + lodgedDescription: string | null; + predictedDescription: string | null; + /** The legacy property_details_epc column (walls / roof / heating). */ + legacyDescription: string | null; +} + +/** + * Free-text component descriptor (wall type / roof type / heating system): + * override → EPC-derived (lodged over predicted, from epc_energy_element) → + * legacy property_details_epc column → 'Unknown'. No code→label mapping — the + * value is the description string itself. Mirrors resolvedWallTypeSql etc. + */ +export function resolveOverrideDescriptor(i: OverrideDescriptorInputs): string { + if (i.override) return i.override; + const derived = i.isNewApproach + ? i.lodgedDescription ?? i.predictedDescription + : i.legacyDescription; + return derived ?? "Unknown"; +} + +export interface MainFuelInputs { + /** property_overrides.override_value for component 'main_fuel', building_part 0. */ + override: string | null; + isNewApproach: boolean; + /** epc_main_heating_detail.main_fuel_type scalar on the lodged EPC — an RdSAP code ("26") or an EPR string ("Mains gas"). */ + lodgedMainFuel: string | null; + predictedMainFuel: string | null; + /** property_details_epc.mainfuel — the raw legacy EPC string. */ + legacyMainFuel: string | null; +} + +/** + * Main fuel raw value (folded to a label by the cell's resolveEnumLabel): + * override → EPC-derived (new: epc_main_heating_detail scalar, lodged over + * predicted; legacy: the property_details_epc string) → 'Unknown'. The + * new-approach scalar is an RdSAP `main_fuel` code or an EPR string; both fold + * into MAINFUEL_OPTIONS. Never null → 'Unknown'. Mirrors resolvedMainFuelSql. + */ +export function resolveMainFuel(i: MainFuelInputs): string { + if (i.override) return i.override; + const derived = i.isNewApproach + ? i.lodgedMainFuel ?? i.predictedMainFuel + : i.legacyMainFuel; + return derived ?? "Unknown"; +} + +function bucketLegacyYearToBand(yearBuilt: string | null): string | null { + if (yearBuilt == null || !/^[0-9]+$/.test(yearBuilt)) return null; + const year = parseInt(yearBuilt, 10); + let best: string | null = null; + let bestStart = -Infinity; + for (const [letter, { year: start }] of Object.entries(CONSTRUCTION_AGE_BANDS)) { + if (start <= year && start > bestStart) { + best = letter; + bestStart = start; + } + } + return best ?? "A"; +} diff --git a/src/lib/services/epcSources.test.ts b/src/lib/services/epcSources.test.ts index 1d85a6ab..5b859fd9 100644 --- a/src/lib/services/epcSources.test.ts +++ b/src/lib/services/epcSources.test.ts @@ -3,7 +3,10 @@ import { CONSTRUCTION_AGE_BANDS, PROPERTY_TYPE_LABELS, } from "./epcSources"; -import { PROPERTY_TYPE_OPTIONS } from "@/app/utils/propertyFilters"; +import { + PROPERTY_TYPE_OPTIONS, + YEAR_BUILT_OPTIONS, +} from "@/app/utils/propertyFilters"; // The reporting age-band bucketing in getCountByAgeBand (databaseFunctions.ts), // mirrored so the CONSTRUCTION_AGE_BANDS representative years stay anchored to @@ -57,12 +60,26 @@ describe("CONSTRUCTION_AGE_BANDS", () => { }); describe("PROPERTY_TYPE_LABELS", () => { - it("maps codes to the labels the property-type filter options use", () => { + it("maps every code to a label the property-type filter offers as an option", () => { const optionLabels = new Set(PROPERTY_TYPE_OPTIONS.map((o) => o.label)); for (const label of Object.values(PROPERTY_TYPE_LABELS)) { - // Park home has no filter option — it's rare and charts render it as-is. - if (label === "Park home") continue; expect(optionLabels).toContain(label); } }); }); + +describe("YEAR_BUILT_OPTIONS (Construction Age)", () => { + // Guards the band-native filter (ADR-0012) against label drift — a mismatched + // en-dash or reworded band would silently make the filter match nothing. + it("is exactly the RdSAP band labels in order, plus Unknown", () => { + const bandLabels = Object.values(CONSTRUCTION_AGE_BANDS).map((b) => b.label); + expect(YEAR_BUILT_OPTIONS.map((o) => o.label)).toEqual([ + ...bandLabels, + "Unknown", + ]); + }); + + it("uses each band label as its own dbValue", () => { + for (const o of YEAR_BUILT_OPTIONS) expect(o.dbValues).toEqual([o.label]); + }); +}); diff --git a/src/lib/services/epcSources.ts b/src/lib/services/epcSources.ts index 3a75bfd9..42174e97 100644 --- a/src/lib/services/epcSources.ts +++ b/src/lib/services/epcSources.ts @@ -192,9 +192,35 @@ export const estimatedSql = (e: Alias) => export const isExpiredSql = (e: Alias) => sql`CASE WHEN ${isNewApproachSql} THEN (epl.registration_date IS NOT NULL AND epl.registration_date < (CURRENT_DATE - INTERVAL '10 years')) ELSE ${e}.is_expired END`; -/** Main fuel. GAP for new properties (no clean equivalent) → NULL. Legacy: e.mainfuel. */ +/** + * The main-fuel scalar for one epc_property alias, read from the JSONB + * `epc_main_heating_detail.main_fuel_type` as a correlated subquery. The stored + * scalar is either a numeric RdSAP `main_fuel` code ("26", "29", … — ~95% of + * rows, from the gov-EPC API) or an EPR text string ("Mains gas", "Bulk LPG", … + * from Elmhurst/PASHub). `#>> '{}'` extracts the JSON scalar as text; NULLIF + * drops the empty-string sentinel to NULL. Codes/strings are folded to a label + * downstream by MAINFUEL_OPTIONS — the fragment emits the raw value, like + * resolvedMainFuelSql's legacy branch. (No index on epc_property_id yet — see + * ix_epc_main_heating_detail_epc_property in the schema.) + */ +const epcMainFuelSql = (ep: Alias) => sql`( + SELECT NULLIF(mhd.main_fuel_type #>> '{}', '') + FROM epc_main_heating_detail mhd + WHERE mhd.epc_property_id = ${ep}.id + LIMIT 1 +)`; + +/** + * Main fuel raw value. New: the epc_main_heating_detail scalar (lodged over + * predicted) — an RdSAP code or an EPR string, folded to a label by + * MAINFUEL_OPTIONS. Legacy: e.mainfuel. Mirrors the new-approach branch of + * resolveMainFuel. + */ export const mainfuelSql = (e: Alias) => - sql`CASE WHEN ${isNewApproachSql} THEN NULL ELSE ${e}.mainfuel END`; + sql`CASE + WHEN ${isNewApproachSql} THEN (CASE WHEN epl.id IS NOT NULL THEN ${epcMainFuelSql(sql`epl`)} ELSE ${epcMainFuelSql(sql`epp`)} END) + ELSE ${e}.mainfuel + END`; // ───────────────────────────────────────────────────────────────────────────── // RdSAP code → label maps for the descriptive fields the new pipeline stores as @@ -283,6 +309,108 @@ export const builtFormTypeSql = sql`CASE ELSE p.built_form END`; +// ───────────────────────────────────────────────────────────────────────────── +// Landlord-override-resolved descriptors (ADR-0012). Each fragment wraps the +// EPC-derived expression above with the property_overrides snapshot and a +// never-null 'Unknown' terminal — the same precedence previewModellingRun +// applies (override → EPC-derived → Unknown). The TS twins in +// descriptorResolution.ts (resolvePropertyType, …) own and unit-test this +// precedence; keep the two in sync. +// +// A resolved fragment references an override alias that its join provides — add +// the matching *OverrideJoin to any query that selects/filters on the fragment. +// Overrides are whole-dwelling for these components, so the join pins +// building_part = 0 (the main building) and never multiplies rows. +// ───────────────────────────────────────────────────────────────────────────── + +/** LEFT JOIN exposing the property_type override snapshot (alias `pot`). */ +export const propertyTypeOverrideJoin = sql` + LEFT JOIN property_overrides pot + ON pot.property_id = p.id + AND pot.override_component = 'property_type' + AND pot.building_part = 0 +`; + +/** property_type: override → EPC-derived → 'Unknown'. Mirrors resolvePropertyType. */ +export const resolvedPropertyTypeSql = sql`COALESCE(pot.override_value, ${propertyTypeSql}, 'Unknown')`; + +/** LEFT JOIN exposing the built_form_type override snapshot (alias `pobf`). */ +export const builtFormOverrideJoin = sql` + LEFT JOIN property_overrides pobf + ON pobf.property_id = p.id + AND pobf.override_component = 'built_form_type' + AND pobf.building_part = 0 +`; + +/** built_form: override → EPC-derived → 'Unknown'. Mirrors resolveBuiltForm. */ +export const resolvedBuiltFormSql = sql`COALESCE(pobf.override_value, ${builtFormTypeSql}, 'Unknown')`; + +/** LEFT JOIN exposing the main_fuel override snapshot (alias `pmf`). */ +export const mainFuelOverrideJoin = sql` + LEFT JOIN property_overrides pmf + ON pmf.property_id = p.id + AND pmf.override_component = 'main_fuel' + AND pmf.building_part = 0 +`; + +/** + * main fuel (raw value; folded to a label by the cell's resolveEnumLabel): + * override → EPC-derived → 'Unknown'. The new EPC graph has no main fuel, so + * mainfuelSql is NULL for new-approach — the override is the only new-approach + * source. Takes the legacy `property_details_epc` alias. Mirrors resolveMainFuel. + */ +export const resolvedMainFuelSql = (e: Alias) => + sql`COALESCE(pmf.override_value, ${mainfuelSql(e)}, 'Unknown')`; + +// ─── Free-text component descriptors: wall / roof / heating (ADR-0012) ──────── +// override → EPC-derived (epc_energy_element description, lodged over predicted) +// → legacy property_details_epc column → 'Unknown'. Mirrors resolveOverrideDescriptor. + +/** One EPC source's description for an energy element, as a correlated subquery. */ +const epcElementDescSql = (ep: Alias, elementType: string) => sql`( + SELECT ee.description FROM epc_energy_element ee + WHERE ee.epc_property_id = ${ep}.id AND ee.element_type = ${elementType} + LIMIT 1 +)`; + +/** EPC-or-legacy element description: new = lodged over predicted; legacy = the property_details_epc column. */ +const epcElementSql = (elementType: string, legacyCol: Alias) => sql`CASE + WHEN ${isNewApproachSql} THEN (CASE WHEN epl.id IS NOT NULL THEN ${epcElementDescSql(sql`epl`, elementType)} ELSE ${epcElementDescSql(sql`epp`, elementType)} END) + ELSE ${legacyCol} +END`; + +/** LEFT JOIN exposing the wall_type override snapshot at the main building part (alias `pwall`). */ +export const wallTypeOverrideJoin = sql` + LEFT JOIN property_overrides pwall + ON pwall.property_id = p.id + AND pwall.override_component = 'wall_type' + AND pwall.building_part = 0 +`; +/** LEFT JOIN exposing the roof_type override snapshot at the main building part (alias `proof`). */ +export const roofTypeOverrideJoin = sql` + LEFT JOIN property_overrides proof + ON proof.property_id = p.id + AND proof.override_component = 'roof_type' + AND proof.building_part = 0 +`; +/** LEFT JOIN exposing the main_heating_system override snapshot (alias `pheat`). */ +export const mainHeatingOverrideJoin = sql` + LEFT JOIN property_overrides pheat + ON pheat.property_id = p.id + AND pheat.override_component = 'main_heating_system' + AND pheat.building_part = 0 +`; + +/** wall type: override → EPC 'wall' description → legacy walls → 'Unknown'. Mirrors resolveOverrideDescriptor. */ +export const resolvedWallTypeSql = (e: Alias) => + sql`COALESCE(pwall.override_value, ${epcElementSql("wall", sql`${e}.walls`)}, 'Unknown')`; +/** roof type: override → EPC 'roof' description → legacy roof → 'Unknown'. */ +export const resolvedRoofTypeSql = (e: Alias) => + sql`COALESCE(proof.override_value, ${epcElementSql("roof", sql`${e}.roof`)}, 'Unknown')`; +/** heating system: override → EPC 'main_heating' description → legacy heating → 'Unknown'. */ +export const resolvedHeatingSystemSql = (e: Alias) => + sql`COALESCE(pheat.override_value, ${epcElementSql("main_heating", sql`${e}.heating`)}, 'Unknown')`; + /** * The main building part's construction age band for one epc_property alias, as * a correlated scalar subquery — properties can also have extension parts, so a @@ -316,6 +444,52 @@ export const constructionYearSql = sql`CASE ELSE NULL END`; +// ─── Construction age as a band LABEL (ADR-0012) ───────────────────────────── +// Unlike constructionYearSql (which flattens the band to a representative year +// for reporting buckets), the portfolio list keeps the band itself. Mirrors +// resolveConstructionAge in descriptorResolution.ts — keep in sync. + +/** RdSAP band letter → band label (e.g. B → "1900–1929"). */ +const AGE_BAND_LABELS: Record = Object.fromEntries( + Object.entries(CONSTRUCTION_AGE_BANDS).map(([letter, { label }]) => [letter, label]), +); + +/** Bucket a legacy year_built expression into its band label; non-numeric → NULL. */ +const legacyYearBandLabelSql = (yearExpr: Alias) => { + const y = sql`CAST(${yearExpr} AS int)`; + const thresholds = Object.entries(CONSTRUCTION_AGE_BANDS) + .filter(([letter]) => letter !== "A") + .sort((a, b) => b[1].year - a[1].year) + .map(([, { year, label }]) => sql`WHEN ${y} >= ${year} THEN ${label}`); + // The numeric guard fires the CAST only on digit strings; below the earliest + // band start falls through to band A. + return sql`CASE WHEN ${yearExpr} ~ '^[0-9]+$' THEN (CASE ${sql.join( + thresholds, + sql` `, + )} ELSE ${CONSTRUCTION_AGE_BANDS.A.label} END) ELSE NULL END`; +}; + +/** EPC-or-legacy band label: new-approach main-part band (lodged over predicted); legacy year bucketed. */ +const constructionAgeBandLabelSql = sql`CASE + WHEN ${isNewApproachSql} THEN (CASE WHEN epl.id IS NOT NULL THEN ${codeLabelCaseSql(mainPartAgeBandSql(sql`epl`), AGE_BAND_LABELS)} ELSE ${codeLabelCaseSql(mainPartAgeBandSql(sql`epp`), AGE_BAND_LABELS)} END) + ELSE ${legacyYearBandLabelSql(sql`p.year_built`)} +END`; + +/** LEFT JOIN exposing the construction_age_band override snapshot (alias `pab`). */ +export const constructionAgeOverrideJoin = sql` + LEFT JOIN property_overrides pab + ON pab.property_id = p.id + AND pab.override_component = 'construction_age_band' + AND pab.building_part = 0 +`; + +/** construction age: override → EPC band → legacy year bucket → 'Unknown'. Mirrors resolveConstructionAge. */ +export const resolvedConstructionAgeSql = sql`COALESCE( + ${codeLabelCaseSql(sql`pab.override_value`, AGE_BAND_LABELS)}, + ${constructionAgeBandLabelSql}, + 'Unknown' +)`; + // ───────────────────────────────────────────────────────────────────────────── // Per-property resolvers (TypeScript object building). // ─────────────────────────────────────────────────────────────────────────────