The neighbour SAP-divergence cohort was (postcode, property_type, built_form),
which mixed electric- and gas-heated dwellings. Electricity scores materially
lower in SAP than mains gas for identical fabric, so a mixed-fuel postcode
produced cross-fuel false outliers — an electric dwelling flagged only for being
electric among gas neighbours.
Add the main-heating fuel class to the cohort key (electricity variants — 29/30
+ off-peak 31-40 — collapse to one class; every other code is its own bucket).
Now every flagged divergence is a same-fuel comparison worth investigating. On
portfolio 814 this refined 20 raw divergences to 17 same-fuel ones while keeping
the genuine within-fuel outliers (e.g. the all-electric WC2B 4AW flats at SAP
26/38 vs a cohort median of 67).
Reaches the fuel via epc_main_heating_detail through the indexed property_id —
still never touches the recommendation table. Surfaced by the portfolio-814
recommendation audit (Work item B, #1388).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One-time script (dry-run default, --apply in a transaction, idempotent) reusing the
live glazing_mix_guard so the backfill and classifier cannot drift. Maps every
genuine percentage-mix override (neither type >= 90%) off a flattened single type
onto MIXED (no overlay -> the cert's per-window glazing is kept). property_overrides
TEXT updated now; the glazing pgEnum cache write for 'Mixed glazing' is deferred
until the FE adds the member. Dry-run against the audited DB reports 647 rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A focused sibling to audit-ara-portfolio: that skill audits baselines/plans/SAP;
this one audits the *recommendations themselves* — why a measure was or wasn't
offered. Motivated by the portfolio-814 review (Khalim's HHRSH-on-community-
heating, missing-HHRSH, missing-secondary-heating-removal, and a neighbour split).
Adds:
- .claude/skills/find-weird-recommendations/SKILL.md — scan -> neighbour scan ->
live re-model deep-dive -> root-cause -> codify, with a seeded known-bug
catalogue and the query-safety rules inherited from audit-ara-portfolio.
- scripts/audit/anomalies.py: new `plan-stops-short-of-goal` HIGH check — the
default plan ends below the goal band on an unlimited-budget scenario (the
deterministic worklist for "why didn't this get recommended X"). Adds
scenario_budget to the bundle/query so budget-capped scenarios are excluded.
- scripts/audit/neighbour_divergence.py: groups a portfolio by (postcode,
property_type, built_form) and flags effective-SAP outliers vs the cohort
median. Never touches the 26m-row recommendation table, so it is safe
portfolio-wide.
- Tests for both (12 passing).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One-time script (dry-run default, --apply in a transaction, idempotent) mapping a
flat roof's stated depth (flat: Nmm) onto the nearest Table 16 rung member so the
overlay scores it by thickness, not the age-band default. Updates property_overrides
(TEXT) immediately; the roof_type pgEnum cache writes are deferred until the FE adds
the FLAT_*MM members. Dry-run against the audited DB reports 102 rows, deferring
Flat 50/100/150 mm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One-time script (dry-run default, --apply in a transaction, idempotent) that maps
every party-ceiling override (another/same dwelling or premises above) currently
stored as an external roof value onto the party-ceiling member, reusing the live
roof_party_ceiling_guard so the backfill and the classifier cannot drift. Updates
property_overrides (TEXT, the modelling read path) and the roof classifier cache;
dry-run against the audited DB reports 106 rows. No FE migration — the party-
ceiling values already exist in the roof pgEnum.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One-time script (dry-run default, --apply in a transaction, idempotent) that
NULLs the four lodged_* columns on every predicted-source baseline row — a
predicted EPC exists and no lodged one does — leaving Effective, the bill block,
and rebaseline_reason intact. Dry-run against the audited DB reports 12,236 rows.
Must run after the FE-owned Drizzle ALTER ... DROP NOT NULL lands.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deterministic catalogue remap (no LLM) that re-resolves the stored archetype for
the canonical RdSAP heating descriptions the under-populated taxonomy
mis-classified. Dry-run by default; --apply writes inside a transaction;
idempotent. Updates property_overrides.override_value (TEXT — what the modelling
reads, the actual band-G fix: 3,028 rows incl. 2,210 community boilers + 770
panel/convector room heaters). Cache (the Drizzle-owned main_heating_system
pgEnum) updates are deferred per-value until the FE-repo enum migration adds the
new archetypes — surfaced explicitly by the script.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces sfr/principal_pitch/2_export_data.py, which read the retired
plan_recommendations m2m and recommendation_materials table. The new model
links a recommendation to its plan directly (recommendation.plan_id), keeps
materials inline on the recommendation (material_id), marks the chosen plan
per (scenario, property) with is_default, and stores post-works SAP/EPC and
savings on the plan row (the new SAP calculator's output).
Takes a portfolio id, resolves every modelled scenario (those with plans),
and writes one workbook with a properties sheet per scenario. EPC descriptive
fields are sourced live from the EPC service (property_details_epc is dead);
property_type falls back override -> cert.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `recommendation` table (~26m rows) has no index on `plan_id`, so any query
reaching it via `plan_id` — including the audit's own rollup — seq-scans the
whole table and saturates the shared DB (it blocked the DB during the
portfolio-796 audit). Make the audit safe-by-default:
- statement_timeout (120s) on the audit connection — a hard ceiling so a bad
plan aborts instead of hammering the DB.
- The recommendation rollup (the two solar checks) is now opt-in via
--with-recommendations, and EXPLAIN-gated: it refuses to run (raising
RecommendationScanError) when the plan contains a Seq Scan on recommendation,
which it does on any large portfolio until idx_recommendation_plan_id exists.
- SKILL.md documents the plan_id-no-index trap, the reach-via-property_id /
EXPLAIN-first / confirm-with-user rules, and the index as the real fix.
Verified on 796/1268: default run is bounded and completes (2,952 anomalies over
31,919 properties); --with-recommendations aborts pre-scan portfolio-wide but is
allowed for a single property.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Runs a Summary PDF through the chain-test path (pdftotext ->
ElmhurstSiteNotesExtractor -> from_elmhurst_site_notes) into Sap10Calculator
and dumps SAP + per-end-use kWh + the intermediate worksheet trail, for
diffing our calc against the accompanying Elmhurst U985 worksheet PDF.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add Phase 6 (self-improve) to audit-ara-portfolio: when a run confirms a
novel systematic problem, codify it as a check — gated on systematic (>=5
props, root-caused), not-already-covered, and /grill-me-pressure-tested.
Each check records provenance (motivating cause + example properties) so the
registry stays sharp and compounds every run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Script takes an optional --scenario to restrict to one scenario's plans. New
skill drives the full loop: run the deterministic scan, review groups,
deep-dive samples via run_modelling_e2e, characterise sub-classes, and
cross-reference open PRs/ADRs — then proposes new checks to codify.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A check-registry that reads property/baseline/plan/scenario and flags odd
results (plan-below-baseline, already-meets-goal-with-works, band/score
mismatch, zero-works-post-differs, effective-lodged divergence, negative
bill savings). Writes modelling_audit.md/.csv. Adding a check = one
decorated function.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When refetch_epc=False and no stored lodged EPC exists, the handler no longer
falls back to a live EPC API call — it treats the property as EPC-less and
hands it to the prediction path. This keeps REFETCH_EPC (lodged path) and
REPREDICT_EPC (prediction path) cleanly independent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Captures the calc-facing ventilation-type gap (older RdSAP mappers drop
mechanical_ventilation_kind), the FE-side Main Fuel display, the sweep survivor
clusters, and the predicted-property display path — all separate from ADR-0037.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Group pending property IDs by postcode and pack them into ~BATCH_SIZE
messages, never splitting a postcode, so each SQS message drives one
batched modelling_e2e Lambda invocation. Adds a completed-since skip
filter and a properties cap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In-container Playwright runs were flaky: the chromium renderer crashed
mid-build ("Target crashed") on the 64M /dev/shm, and login intermittently
hung. Added `--disable-dev-shm-usage` + `--no-sandbox` launch args, a
4-attempt login retry loop (domcontentloaded + explicit selector wait),
and an `ELM_GUID` env override so a per-UPRN assessment can be targeted
without editing the module. Tooling only — no calculator impact.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
