/health already returned GITHUB_SHA in its response but never logged
it, and the fastapi lambda's Terraform environment never actually set
GITHUB_SHA — so every health check response contained "unknown" in
production. Wire var.github_sha through the fastapi lambda module
(default "unknown" for local/other invocations) via a new
TF_VAR_github_sha env var set from `github.sha` in
_deploy_lambda.yml's Terraform Plan step, and log it on every /health
call so a request in CloudWatch can be tied back to the deploy that
served it.
Also fix the zip-size gate added for PR #1469: putting it in
tests/test_lambda_zip_size.py (run via the Docker-based ddd_tests.yml
suite) broke CI, because Dockerfile.test's build context excludes
deployment/* (.dockerignore), so check_lambda_zip_size.py couldn't
find variables.tf to read zip_excludes from inside that container.
Move the check to its own lightweight workflow,
check_lambda_zip_size.yml, triggered on pull_request into main — a
plain checkout (no Docker build) has the full repo, so the check
works, runs fast, and still gates merges to main before a regression
can roll into the dev deploy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
deploy_terraform.yml only triggers on push to dev/prod — by the time
it runs, the change has already merged to main and rolled into dev.
That's exactly how the size regression sat undetected for weeks: it
only ever surfaced deep inside terraform apply on dev.
Drop the fast_api_lambda_zip_size_check job from deploy_terraform.yml
and instead add tests/test_lambda_zip_size.py, which runs
backend/app/requirements/check_lambda_zip_size.py as part of the
existing pytest tests/ suite. ddd_tests.yml already runs that suite
on `pull_request: branches: ["**"]`, so this now fails the PR before
anything merges to main, rather than failing the dev deploy after.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
30 pairs (28 deterministically scoreable) from the 2,000-postcode sweep,
frozen as ONE anonymised raw-payload JSON (pairs + cohorts + actuals — a
thousand per-cert files would drown the PR diff). The gate replays the
whole conditioning path offline — mapper, conditioning, selection,
synthesis, comparison — in ~9s; floors are the measured values, tighten-
only. comparable_from_payload is extracted from the corpus loader so both
fixture formats share one payload->ComparableProperty path; the builder
(build_expired_pairs_corpus.py) refreezes from the raw-JSON disk cache.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Relocate scripts/check_lambda_zip_size.py to
backend/app/requirements/, right beside the requirements.txt it's
built to measure, since it's specific to the fastapi lambda rather
than a generic repo script.
While moving it, also make the excludes list self-updating: the
script now reads zip_excludes straight out of
deployment/terraform/modules/lambda_with_api_gateway/variables.tf
by default instead of duplicating it as CLI flags, so the CI check
and the actual Terraform packaging can't drift apart. requirements.txt
defaults to the sibling file too, so both the CI step and local runs
now collapse to a bare `check_lambda_zip_size.py` invocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dev deploys have failed since PR #1231 (2026-06-15) with
InvalidParameterValueException: Unzipped size must be smaller than
262144000 bytes. The lambda zip is built from the whole repo root
(deployment/terraform/modules/lambda_with_api_gateway) and only
excluded tests/deployment/pycache — so .git (206MB packed), the
per-schema corpus.jsonl fixtures under backend/epc_api/json_samples/
(~50MB), and the sap10_calculator PDF specs (~13MB) were all being
zipped up even though none of them are read at runtime.
Also add scripts/check_lambda_zip_size.py, which mirrors the
terraform module's pip install + zip-with-excludes behaviour to
report the projected unzipped size without mutating the repo, and
wire it into deploy_terraform.yml as a fast_api_lambda_zip_size_check
job that gates the real deploy so this fails fast with a clear
message instead of surfacing as an opaque Terraform/AWS error.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
roof_construction codes group by FORM (empirical: 1=Flat 98%, 4/5/8=
Pitched 88-99%, 3=dwelling-above 100% over 7,974 certs; 7/9=premises-
above per #1452), so the filter matches families — an exact-code filter
would wrongly drop pitched neighbours lodged as 5/8. Historic prefixes
map to the same families; roof rooms and thatch stay unconditioned.
Harness ladder replay and telemetry mirror the new filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JsonCachingEpcClient caches at the raw layer (cert payloads verbatim,
search rows as JSON) so warm re-runs replay through retry+mapper+parsing
in minutes AND the cache doubles as integration-test fixture material —
the ADR-0030 corpus pattern (anonymised raw JSON through the mapper),
which pickled domain objects could never be. Wired into the pairs
harness (--cache-dir). roof_construction_code_sweep pins the RdSAP roof
code table the basement-code-6 way: single-building-part certs only,
code x description-prefix purity table. Also fixes the tfa telemetry key
rename that left the agreement row empty.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Age band equality -> ±1-band neighbourhood; TFA ±5% -> ±20%; whitelist
core validated (fuel 95% / type 91% / wall 79% stable); roof-insulation
collateral flagged for follow-up. CONTEXT.md term updated to match.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Evidence (439-pair harness, PR #1466): historic-vs-new age band agrees
52% exactly but 90% within one band (assessors re-band, skewing newer);
TFA agrees 45% within ±5% but 82% within ±20%. Equality/±5% steered the
cohort toward stale values where they engaged and relaxed everywhere
else. Band definitions are public so the harness's ladder replay shares
one source of truth.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Strip the pre-RdSAP-17 deprecation rider and alias the SAP-prefixed
forms before the code-table lookup. Raises the historic fuel resolution
rate from ~22% of pairs to ~92% of dump rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 2000-postcode sweep died at ~884 on a strict-raising RdSAP-17.1 cert
(missing multiple_glazed_proportion) inside the pair-check, losing the
whole run's output. get_by_uprn and the cohort fetch now skip-and-log per
item; telemetry rows append as produced and carry raw values (age band,
TFA, fuel text) so band-width questions are answerable post hoc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per pair, replay the age->fuel->TFA conditioning ladder over the plain
arm's cohort to record ENGAGED vs RELAXED per filter, and record whether
each historic stable attribute still agrees with the newly lodged cert
(the direct staleness measurement). Emitted as JSONL (--telemetry) and
aggregated into two diagnosis tables appended to the report.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Classification hit-rates for every compare_prediction component, all five
numeric residuals, and the secondary calculator-floored SAP residual
(calc(predicted) − lodged), plain vs conditioned side by side — the same
metric shape as the prediction-corpus gate (ADR-0030). Pair-check now
precedes the cohort fetch so a national postcode sweep only pays the
expensive search-by-postcode for shards that actually hold a pair.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Finds pre-2012 (S3 backup) x SAP-10.2 (new API) cert pairs per postcode,
predicts each from its leave-one-out cohort with and without ADR-0054
conditioning, and reports compare_prediction hit rates side by side.
Evidence for the stable-attribute whitelist, not a CI gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Off by default like the other two prediction collaborators; the pipeline
turns on Expired-Enhanced Prediction by passing a resolver over the
historic S3 backup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ingestion's lookup chain becomes: new EPC API -> historic backup by exact
UPRN -> plain prediction. A found record's stable attributes fill the
gaps Landlord Overrides left (overrides always win) and enrich the target
with age band / fuel / TFA band; the persisted predicted-slot row carries
source="expired". Reader unwired or shard miss -> behaviour unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
EpcSource widens to "expired" (ADR-0054; the column is TEXT — no
migration). "predicted"/"expired" form one slot family: _slot_sources
routes every slot read and slot-clearing delete through the family, so a
re-ingestion flipping the flavour replaces the row instead of stranding
its sibling. FakeEpcRepo mirrors the family.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Age band (Table S1 letter), main fuel code, and a ±5% floor-area band —
the first numeric-tolerance filter — each ride _maybe_filter, so an
unresolved attribute (None) is inactive and a starving filter relaxes.
Existing callers pass no new fields and are behaviourally untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pins behaviour the exact-match slice already carried (max by
lodgement_date; ISO dates sort lexicographically).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
