diff --git a/.claude/skills/expand-sap-accuracy-corpus/worklist.md b/.claude/skills/expand-sap-accuracy-corpus/worklist.md index cdd3abbd..58f3b984 100644 --- a/.claude/skills/expand-sap-accuracy-corpus/worklist.md +++ b/.claude/skills/expand-sap-accuracy-corpus/worklist.md @@ -120,6 +120,53 @@ malformed cert (e.g. PV-as-list AttributeError) is skipped+reported too, not jus missing-field ValueErrors; transient `EpcApiError` (subclasses `Exception`) still propagates. +regression test. +### πŸ”§πŸ” modelling_e2e failure sweep β€” later run (2026-06-23 14:48 BST, portfolio 796, scenario 1268) +A second `modelling_e2e` run (filter `task_source='modelling_e2e' AND job_started > +'2026-06-23 14:48:01.806 +0100'`): **200 tasks, 30 FAILED (15%) β†’ 22 unique props** +(8 retried once). The DB only stores bare `property_id` in subtask `outputs` +(`cloud_logs_url` empty) β€” root causes recovered by replaying each through +`scripts.run_modelling_e2e --scenario-id 1268`. Four root-cause clusters: + +**β‘  βœ… FIXED β€” `KeyError: BuildingPartIdentifier.EXTENSION_1` (14 props, the dominant +failure).** A **Landlord Override** targeted `extension_1` but the lodged/predicted EPC +had no such part; `overlay_applicator.apply_simulations` indexed `parts_by_id[identifier]` +unguarded β†’ crashed the whole property. Root cause is a **numbering mismatch**: the +override's `building_part` is a positional index (0=main, 1=extension 1…, ADR-0004), but +the gov-API EPC can label that slot differently (720142's 2nd part lodged as `other`, no +`extension_1`). Fix (per product decision β€” *honour the override, don't drop it*): +`_resolve_part` falls back to the EPC's part **at that position** when the semantic label +is absent, so the landlord's correction lands; only a position the EPC models no part at +is skipped (no geometry to model a wholly-absent part). +regression tests +`test_override_for_an_absent_semantic_part_lands_on_the_part_at_that_position` / +`test_override_with_no_part_at_that_position_is_skipped`. Not a mapper gap β€” a +modelling-overlay bug. The 14 (pid / uprn): +710295/100020458237, 710482/100020450179, 713040/100020404702, 715575/100020397529, +715894/100020604961, 717435/22010468, 720142/100020383544, 720560/100021921443, +721241/100020453651, 724945/100021915421, 725415/100020404036, 726517/100020631307, +726592/100021918195, 730800/100021920273. (Observed post-fix eng: 720142 69.2, +710295 63.2β†’69.8, 720560 61.8β†’70.4 predicted.) + +**β‘‘ πŸ” mapper/cascade gap β€” `UnmappedSapCode: fuel_code 10`** (1 prop, 730259 / uprn +100061905741). EPC-less β†’ prediction synthesises a cohort EPC carrying `fuel_code 10`, +absent from the calculator's cascade dispatch dict β†’ FATAL (it's the target prop's own +fuel, not a skippable cohort cert). FOR KHALIM: add fuel-code-10 to the cascade. + +**β‘’ β›” not-predictable β€” empty/unresolved postcode cohort** (3 props): 714585/100020612517 +(`CR0 0DD`), 718580/10013149015 (`BR6 6BS`), 723881/22005280 (`BN41 2TP`). EPC-less AND +cohort empty after filtering β†’ Prediction Path 3 can't fire. Coverage gap, not a crash bug. + +**β‘£ ⚠ transient β€” pass clean on replay** (4 props): 712401/100020394694 (eng 75.3), +718138/100020397707 (eng 63.5β†’69.2), 720844/100020472603 (eng 71.8), 723648/100020480302 +(eng 70.8). Likely a flaky EPC/Solar API call during the batch β€” no code defect. + +**E2E candidates to pin (best = the lodged-EPC ones, clean to build in Elmhurst):** +`[ ]` 100020383544 (pid 720142, eng 69.2) Β· `[ ]` 100020458237 (710295) Β· `[ ]` +100020404702 (713040) Β· `[ ]` 100020450179 (710482) Β· `[ ]` 100021915421 (724945) Β· +`[ ]` 100021920273 (730800) Β· `[ ]` 100020394694 (712401, eng 75.3) Β· `[ ]` +100020397707 (718138) Β· `[ ]` 100020472603 (720844, eng 71.8) Β· `[ ]` 100020480302 +(723648, eng 70.8). (Schema not populated on the gov-API objects β€” confirm when keying +each into Elmhurst.) + ### πŸ“‹ PLAN β€” close the 8 modelling_e2e mapping gaps (2026-06-23 run, portfolio 796) The 8 failed prediction targets reduce to **5 distinct mapper-gap classes** (the fix targets). Per class: fix the mapper GENERICALLY, guard with BOTH the RdSAP-21.0.1 diff --git a/.devcontainer/backend/requirements.txt b/.devcontainer/backend/requirements.txt index 2db3710a..78131981 100644 --- a/.devcontainer/backend/requirements.txt +++ b/.devcontainer/backend/requirements.txt @@ -24,6 +24,7 @@ ipykernel>=6.25,<7 dotenv psycopg[binary] pytest-postgresql +moto[s3,sqs]==5.0.28 # mock_aws (moto 5.x) for S3/SQS in orchestration tests # Formatting black==26.1.0 boto3-stubs diff --git a/.github/workflows/deploy_fastapi_backend.yml b/.github/workflows/deploy_fastapi_backend.yml index 6ebb6cef..645d6bc1 100644 --- a/.github/workflows/deploy_fastapi_backend.yml +++ b/.github/workflows/deploy_fastapi_backend.yml @@ -1,170 +1,173 @@ -name: Fast Api Backend Deploy - -on: - push: - branches: [ dev, prod ] - -jobs: - deploy: - runs-on: ubuntu-latest - - steps: - - name: Checkout code - uses: actions/checkout@v3 - - - name: Set up Python - uses: actions/setup-python@v2 - with: - python-version: 3.10.12 - - - name: Install Serverless and plugins - run: | - npm install -g serverless@^3.38.0 - npm install -g serverless-domain-manager@^7.3.8 - npm install -g serverless-python-requirements - - - name: Install dependencies - run: | - python -m pip install --upgrade pip - - - name: AWS credentials for dev - if: github.ref == 'refs/heads/dev' - uses: aws-actions/configure-aws-credentials@v1 - with: - aws-access-key-id: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }} - aws-region: eu-west-2 - - - name: AWS credentials for prod - if: github.ref == 'refs/heads/prod' - uses: aws-actions/configure-aws-credentials@v1 - with: - aws-access-key-id: ${{ secrets.PROD_AWS_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.PROD_AWS_SECRET_ACCESS_KEY }} - aws-region: eu-west-2 - - - name: Set domain name - id: set_domain - run: echo "::set-output name=domain::${{ secrets[format('{0}_DOMAIN_NAME', github.ref_name)] }}" - - - name: Set EPC auth token - id: set_auth_token - run: echo "::set-output name=auth_token::${{ secrets[format('{0}_EPC_AUTH_TOKEN', github.ref_name)] }}" - - - name: Set Open EPC API token - id: set_open_epc_token - run: echo "::set-output name=open_epc_token::${{ secrets[format('{0}_OPEN_EPC_API_TOKEN', github.ref_name)] }}" - - # Store port, name and host in github secrets - - name: Set DB credentials - id: set_db_credentials - run: | - echo "::set-output name=db_host::${{ secrets[format('{0}_DB_HOST', github.ref_name)] }}" - echo "::set-output name=db_port::${{ secrets[format('{0}_DB_PORT', github.ref_name)] }}" - echo "::set-output name=db_name::${{ secrets[format('{0}_DB_NAME', github.ref_name)] }}" - - - name: Set ECR credentials - id: set_ecr_credentials - run: | - echo "::set-output name=ecr_uri::${{ secrets[format('{0}_ECR_URI', github.ref_name)] }}" - - - name: Set Secrets - id: set_api_secrets - run: | - echo "::set-output name=sap_predictions_bucket::${{ secrets[format('{0}_SAP_PREDICTIONS_BUCKET', github.ref_name)] }}" - echo "::set-output name=carbon_predictions_bucket::${{ secrets[format('{0}_CARBON_PREDICTIONS_BUCKET', github.ref_name)] }}" - echo "::set-output name=heat_predictions_bucket::${{ secrets[format('{0}_HEAT_PREDICTIONS_BUCKET', github.ref_name)] }}" - echo "::set-output name=lighting_cost_predictions_bucket::${{ secrets[format('{0}_LIGHTING_COST_PREDICTIONS_BUCKET', github.ref_name)] }}" - echo "::set-output name=heating_cost_predictions_bucket::${{ secrets[format('{0}_HEATING_COST_PREDICTIONS_BUCKET', github.ref_name)] }}" - echo "::set-output name=hot_water_cost_predictions_bucket::${{ secrets[format('{0}_HOT_WATER_COST_PREDICTIONS_BUCKET', github.ref_name)] }}" - echo "::set-output name=heating_kwh_predictions_bucket::${{ secrets[format('{0}_HEATING_KWH_PREDICTIONS_BUCKET', github.ref_name)] }}" - echo "::set-output name=hotwater_kwh_predictions_bucket::${{ secrets[format('{0}_HOTWATER_KWH_PREDICTIONS_BUCKET', github.ref_name)] }}" - echo "::set-output name=energy_asessments_bucket::${{ secrets[format('{0}_ENERGY_ASSESSMENTS_BUCKET', github.ref_name)] }}" - echo "::set-output name=google_solar_api_key::${{ secrets[format('{0}_GOOGLE_SOLAR_API_KEY', github.ref_name)] }}" - echo "::set-output name=sap_baseline_predictions_bucket::${{ secrets[format('{0}_SAP_BASELINE_PREDICTIONS_BUCKET', github.ref_name)] }}" - echo "::set-output name=carbon_baseline_predictions_bucket::${{ secrets[format('{0}_CARBON_BASELINE_PREDICTIONS_BUCKET', github.ref_name)] }}" - echo "::set-output name=heat_baseline_predictions_bucket::${{ secrets[format('{0}_HEAT_BASELINE_PREDICTIONS_BUCKET', github.ref_name)] }}" - - - name: Setup Docker - uses: docker/setup-buildx-action@v1 - - # - name: Setup Docker Buildx - # run: | - # docker buildx create --use - - - name: Build Docker Image For Engine - run: | - docker buildx build \ - --platform linux/amd64 \ - --provenance=false \ - --output=type=docker \ - -t fastapi-lambda-image:${{ github.sha }} \ - -f backend/docker/engine.Dockerfile \ - . - - - name: Login to ECR - run: | - aws ecr get-login-password --region eu-west-2 | docker login --username AWS --password-stdin ${{ steps.set_ecr_credentials.outputs.ecr_uri }} - - - name: Tag and Push Docker Image to ECR - run: | - docker tag fastapi-lambda-image:${{ github.sha }} ${{ steps.set_ecr_credentials.outputs.ecr_uri }}:${{ github.sha }} - docker push ${{ steps.set_ecr_credentials.outputs.ecr_uri }}:${{ github.sha }} - - - name: Deploy to AWS Lambda via Serverless - env: - API_KEY: ${{ secrets.FASTAPI_API_KEY }} - ENVIRONMENT: ${{ github.ref_name }} - SECRET_KEY: ${{ secrets.NEXTAUTH_SECRET }} - PLAN_TRIGGER_BUCKET: 'retrofit-plan-inputs-${{ github.ref_name }}' - DATA_BUCKET: 'retrofit-data-${{ github.ref_name }}' - PREDICTIONS_BUCKET: 'retrofit-sap-predictions-${{ github.ref_name }}' - SAP_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.sap_predictions_bucket }} - CARBON_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.carbon_predictions_bucket }} - HEAT_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.heat_predictions_bucket }} - LIGHTING_COST_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.lighting_cost_predictions_bucket }} - HEATING_COST_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.heating_cost_predictions_bucket }} - HOT_WATER_COST_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.hot_water_cost_predictions_bucket }} - HEATING_KWH_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.heating_kwh_predictions_bucket }} - HOTWATER_KWH_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.hotwater_kwh_predictions_bucket }} - ENERGY_ASSESSMENTS_BUCKET: ${{ steps.set_api_secrets.outputs.energy_asessments_bucket }} - GOOGLE_SOLAR_API_KEY: ${{ steps.set_api_secrets.outputs.google_solar_api_key }} - DOMAIN_NAME: ${{ steps.set_domain.outputs.domain }} - EPC_AUTH_TOKEN: ${{ steps.set_auth_token.outputs.auth_token }} - OPEN_EPC_API_TOKEN: ${{ steps.set_open_epc_token.outputs.open_epc_token }} - DB_HOST: ${{ steps.set_db_credentials.outputs.db_host }} - DB_PORT: ${{ steps.set_db_credentials.outputs.db_port }} - DB_NAME: ${{ steps.set_db_credentials.outputs.db_name }} - ECR_URI: ${{ steps.set_ecr_credentials.outputs.ecr_uri }} - GITHUB_SHA: ${{ github.sha }} - SAP_BASELINE_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.sap_baseline_predictions_bucket }} - CARBON_BASELINE_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.carbon_baseline_predictions_bucket }} - HEAT_BASELINE_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.heat_baseline_predictions_bucket }} - run: | - # Fetch database credentials from AWS Secrets Manager - SECRET_VALUE=$(aws secretsmanager get-secret-value --secret-id ${{ github.ref_name }}/assessment_model/db_credentials --query SecretString) - DB_USERNAME=$(echo "$SECRET_VALUE" | jq -r '. | fromjson | .db_assessment_model_username') - DB_PASSWORD=$(echo "$SECRET_VALUE" | jq -r '. | fromjson | .db_assessment_model_password') - - # Set the database credentials as environment variables - export DB_USERNAME - export DB_PASSWORD - - # Deploy to AWS Lambda via Serverless - sls deploy --stage ${{ github.ref_name }} --verbose - - - name: Smoke test deployed /health - env: - EXPECTED_SHA: ${{ github.sha }} - HEALTH_URL: https://api.${{ steps.set_domain.outputs.domain }}/health - run: | - set -euo pipefail - echo "Probing $HEALTH_URL" - RESPONSE=$(curl -fsSL --max-time 30 --retry 3 --retry-delay 5 --retry-connrefused "$HEALTH_URL") - echo "Response: $RESPONSE" - ACTUAL_SHA=$(echo "$RESPONSE" | jq -r '.sha') - if [[ "$ACTUAL_SHA" != "$EXPECTED_SHA" ]]; then - echo "::error::SHA mismatch. expected=$EXPECTED_SHA actual=$ACTUAL_SHA" - exit 1 - fi - echo "Health check passed. sha=$ACTUAL_SHA" +# Temporarily disabled β€” the FastAPI backend deploy pipeline (deploys must be run manually via `sls deploy` while disabled). +# Commented out to cut GitHub Actions minutes; uncomment to re-enable. +# +# name: Fast Api Backend Deploy +# +# on: +# push: +# branches: [ dev, prod ] +# +# jobs: +# deploy: +# runs-on: ubuntu-latest +# +# steps: +# - name: Checkout code +# uses: actions/checkout@v3 +# +# - name: Set up Python +# uses: actions/setup-python@v2 +# with: +# python-version: 3.10.12 +# +# - name: Install Serverless and plugins +# run: | +# npm install -g serverless@^3.38.0 +# npm install -g serverless-domain-manager@^7.3.8 +# npm install -g serverless-python-requirements +# +# - name: Install dependencies +# run: | +# python -m pip install --upgrade pip +# +# - name: AWS credentials for dev +# if: github.ref == 'refs/heads/dev' +# uses: aws-actions/configure-aws-credentials@v1 +# with: +# aws-access-key-id: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }} +# aws-secret-access-key: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }} +# aws-region: eu-west-2 +# +# - name: AWS credentials for prod +# if: github.ref == 'refs/heads/prod' +# uses: aws-actions/configure-aws-credentials@v1 +# with: +# aws-access-key-id: ${{ secrets.PROD_AWS_ACCESS_KEY_ID }} +# aws-secret-access-key: ${{ secrets.PROD_AWS_SECRET_ACCESS_KEY }} +# aws-region: eu-west-2 +# +# - name: Set domain name +# id: set_domain +# run: echo "::set-output name=domain::${{ secrets[format('{0}_DOMAIN_NAME', github.ref_name)] }}" +# +# - name: Set EPC auth token +# id: set_auth_token +# run: echo "::set-output name=auth_token::${{ secrets[format('{0}_EPC_AUTH_TOKEN', github.ref_name)] }}" +# +# - name: Set Open EPC API token +# id: set_open_epc_token +# run: echo "::set-output name=open_epc_token::${{ secrets[format('{0}_OPEN_EPC_API_TOKEN', github.ref_name)] }}" +# +# # Store port, name and host in github secrets +# - name: Set DB credentials +# id: set_db_credentials +# run: | +# echo "::set-output name=db_host::${{ secrets[format('{0}_DB_HOST', github.ref_name)] }}" +# echo "::set-output name=db_port::${{ secrets[format('{0}_DB_PORT', github.ref_name)] }}" +# echo "::set-output name=db_name::${{ secrets[format('{0}_DB_NAME', github.ref_name)] }}" +# +# - name: Set ECR credentials +# id: set_ecr_credentials +# run: | +# echo "::set-output name=ecr_uri::${{ secrets[format('{0}_ECR_URI', github.ref_name)] }}" +# +# - name: Set Secrets +# id: set_api_secrets +# run: | +# echo "::set-output name=sap_predictions_bucket::${{ secrets[format('{0}_SAP_PREDICTIONS_BUCKET', github.ref_name)] }}" +# echo "::set-output name=carbon_predictions_bucket::${{ secrets[format('{0}_CARBON_PREDICTIONS_BUCKET', github.ref_name)] }}" +# echo "::set-output name=heat_predictions_bucket::${{ secrets[format('{0}_HEAT_PREDICTIONS_BUCKET', github.ref_name)] }}" +# echo "::set-output name=lighting_cost_predictions_bucket::${{ secrets[format('{0}_LIGHTING_COST_PREDICTIONS_BUCKET', github.ref_name)] }}" +# echo "::set-output name=heating_cost_predictions_bucket::${{ secrets[format('{0}_HEATING_COST_PREDICTIONS_BUCKET', github.ref_name)] }}" +# echo "::set-output name=hot_water_cost_predictions_bucket::${{ secrets[format('{0}_HOT_WATER_COST_PREDICTIONS_BUCKET', github.ref_name)] }}" +# echo "::set-output name=heating_kwh_predictions_bucket::${{ secrets[format('{0}_HEATING_KWH_PREDICTIONS_BUCKET', github.ref_name)] }}" +# echo "::set-output name=hotwater_kwh_predictions_bucket::${{ secrets[format('{0}_HOTWATER_KWH_PREDICTIONS_BUCKET', github.ref_name)] }}" +# echo "::set-output name=energy_asessments_bucket::${{ secrets[format('{0}_ENERGY_ASSESSMENTS_BUCKET', github.ref_name)] }}" +# echo "::set-output name=google_solar_api_key::${{ secrets[format('{0}_GOOGLE_SOLAR_API_KEY', github.ref_name)] }}" +# echo "::set-output name=sap_baseline_predictions_bucket::${{ secrets[format('{0}_SAP_BASELINE_PREDICTIONS_BUCKET', github.ref_name)] }}" +# echo "::set-output name=carbon_baseline_predictions_bucket::${{ secrets[format('{0}_CARBON_BASELINE_PREDICTIONS_BUCKET', github.ref_name)] }}" +# echo "::set-output name=heat_baseline_predictions_bucket::${{ secrets[format('{0}_HEAT_BASELINE_PREDICTIONS_BUCKET', github.ref_name)] }}" +# +# - name: Setup Docker +# uses: docker/setup-buildx-action@v1 +# +# # - name: Setup Docker Buildx +# # run: | +# # docker buildx create --use +# +# - name: Build Docker Image For Engine +# run: | +# docker buildx build \ +# --platform linux/amd64 \ +# --provenance=false \ +# --output=type=docker \ +# -t fastapi-lambda-image:${{ github.sha }} \ +# -f backend/docker/engine.Dockerfile \ +# . +# +# - name: Login to ECR +# run: | +# aws ecr get-login-password --region eu-west-2 | docker login --username AWS --password-stdin ${{ steps.set_ecr_credentials.outputs.ecr_uri }} +# +# - name: Tag and Push Docker Image to ECR +# run: | +# docker tag fastapi-lambda-image:${{ github.sha }} ${{ steps.set_ecr_credentials.outputs.ecr_uri }}:${{ github.sha }} +# docker push ${{ steps.set_ecr_credentials.outputs.ecr_uri }}:${{ github.sha }} +# +# - name: Deploy to AWS Lambda via Serverless +# env: +# API_KEY: ${{ secrets.FASTAPI_API_KEY }} +# ENVIRONMENT: ${{ github.ref_name }} +# SECRET_KEY: ${{ secrets.NEXTAUTH_SECRET }} +# PLAN_TRIGGER_BUCKET: 'retrofit-plan-inputs-${{ github.ref_name }}' +# DATA_BUCKET: 'retrofit-data-${{ github.ref_name }}' +# PREDICTIONS_BUCKET: 'retrofit-sap-predictions-${{ github.ref_name }}' +# SAP_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.sap_predictions_bucket }} +# CARBON_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.carbon_predictions_bucket }} +# HEAT_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.heat_predictions_bucket }} +# LIGHTING_COST_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.lighting_cost_predictions_bucket }} +# HEATING_COST_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.heating_cost_predictions_bucket }} +# HOT_WATER_COST_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.hot_water_cost_predictions_bucket }} +# HEATING_KWH_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.heating_kwh_predictions_bucket }} +# HOTWATER_KWH_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.hotwater_kwh_predictions_bucket }} +# ENERGY_ASSESSMENTS_BUCKET: ${{ steps.set_api_secrets.outputs.energy_asessments_bucket }} +# GOOGLE_SOLAR_API_KEY: ${{ steps.set_api_secrets.outputs.google_solar_api_key }} +# DOMAIN_NAME: ${{ steps.set_domain.outputs.domain }} +# EPC_AUTH_TOKEN: ${{ steps.set_auth_token.outputs.auth_token }} +# OPEN_EPC_API_TOKEN: ${{ steps.set_open_epc_token.outputs.open_epc_token }} +# DB_HOST: ${{ steps.set_db_credentials.outputs.db_host }} +# DB_PORT: ${{ steps.set_db_credentials.outputs.db_port }} +# DB_NAME: ${{ steps.set_db_credentials.outputs.db_name }} +# ECR_URI: ${{ steps.set_ecr_credentials.outputs.ecr_uri }} +# GITHUB_SHA: ${{ github.sha }} +# SAP_BASELINE_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.sap_baseline_predictions_bucket }} +# CARBON_BASELINE_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.carbon_baseline_predictions_bucket }} +# HEAT_BASELINE_PREDICTIONS_BUCKET: ${{ steps.set_api_secrets.outputs.heat_baseline_predictions_bucket }} +# run: | +# # Fetch database credentials from AWS Secrets Manager +# SECRET_VALUE=$(aws secretsmanager get-secret-value --secret-id ${{ github.ref_name }}/assessment_model/db_credentials --query SecretString) +# DB_USERNAME=$(echo "$SECRET_VALUE" | jq -r '. | fromjson | .db_assessment_model_username') +# DB_PASSWORD=$(echo "$SECRET_VALUE" | jq -r '. | fromjson | .db_assessment_model_password') +# +# # Set the database credentials as environment variables +# export DB_USERNAME +# export DB_PASSWORD +# +# # Deploy to AWS Lambda via Serverless +# sls deploy --stage ${{ github.ref_name }} --verbose +# +# - name: Smoke test deployed /health +# env: +# EXPECTED_SHA: ${{ github.sha }} +# HEALTH_URL: https://api.${{ steps.set_domain.outputs.domain }}/health +# run: | +# set -euo pipefail +# echo "Probing $HEALTH_URL" +# RESPONSE=$(curl -fsSL --max-time 30 --retry 3 --retry-delay 5 --retry-connrefused "$HEALTH_URL") +# echo "Response: $RESPONSE" +# ACTUAL_SHA=$(echo "$RESPONSE" | jq -r '.sha') +# if [[ "$ACTUAL_SHA" != "$EXPECTED_SHA" ]]; then +# echo "::error::SHA mismatch. expected=$EXPECTED_SHA actual=$ACTUAL_SHA" +# exit 1 +# fi +# echo "Health check passed. sha=$ACTUAL_SHA" diff --git a/.github/workflows/integration_tests.yml b/.github/workflows/integration_tests.yml index 6093e249..374bf94e 100644 --- a/.github/workflows/integration_tests.yml +++ b/.github/workflows/integration_tests.yml @@ -1,35 +1,38 @@ -name: Rebaselining Integration Test - -on: - pull_request: - branches: - - main - -jobs: - rebaselining-integration-test: - runs-on: ubuntu-latest - steps: - - name: Checkout code - uses: actions/checkout@v4 - - - name: Set up Python 3.11 - uses: actions/setup-python@v4 - with: - python-version: '3.11' - - - name: Install tox via Makefile - run: | - make setup - - - name: Configure AWS credentials for dev - uses: aws-actions/configure-aws-credentials@v1 - with: - aws-access-key-id: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }} - aws-region: eu-west-2 - - - name: Run only rebaselining integration test - env: - EPC_AUTH_TOKEN: ${{ secrets.DEV_EPC_AUTH_TOKEN }} - run: | - make test ARGS="-m integration" +# Temporarily disabled β€” the rebaselining integration suite. +# Commented out to cut GitHub Actions minutes; uncomment to re-enable. +# +# name: Rebaselining Integration Test +# +# on: +# pull_request: +# branches: +# - main +# +# jobs: +# rebaselining-integration-test: +# runs-on: ubuntu-latest +# steps: +# - name: Checkout code +# uses: actions/checkout@v4 +# +# - name: Set up Python 3.11 +# uses: actions/setup-python@v4 +# with: +# python-version: '3.11' +# +# - name: Install tox via Makefile +# run: | +# make setup +# +# - name: Configure AWS credentials for dev +# uses: aws-actions/configure-aws-credentials@v1 +# with: +# aws-access-key-id: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }} +# aws-secret-access-key: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }} +# aws-region: eu-west-2 +# +# - name: Run only rebaselining integration test +# env: +# EPC_AUTH_TOKEN: ${{ secrets.DEV_EPC_AUTH_TOKEN }} +# run: | +# make test ARGS="-m integration" diff --git a/.github/workflows/protect_releases.yml b/.github/workflows/protect_releases.yml index cbd08e2f..7c8c7258 100644 --- a/.github/workflows/protect_releases.yml +++ b/.github/workflows/protect_releases.yml @@ -1,17 +1,20 @@ -name: Restrict PR source - -on: - pull_request: - branches: - - dev - -jobs: - check-source-branch: - runs-on: ubuntu-latest - steps: - - name: Fail if PR is not from main - run: | - if [[ "${{ github.head_ref }}" != "main" ]]; then - echo "Only PRs from main are allowed into dev" - exit 1 - fi \ No newline at end of file +# Temporarily disabled β€” the mainβ†’dev PR-source guardrail. +# Commented out to cut GitHub Actions minutes; uncomment to re-enable. +# +# name: Restrict PR source +# +# on: +# pull_request: +# branches: +# - dev +# +# jobs: +# check-source-branch: +# runs-on: ubuntu-latest +# steps: +# - name: Fail if PR is not from main +# run: | +# if [[ "${{ github.head_ref }}" != "main" ]]; then +# echo "Only PRs from main are allowed into dev" +# exit 1 +# fi diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml index 12d836d0..bd022a2a 100644 --- a/.github/workflows/unit_tests.yml +++ b/.github/workflows/unit_tests.yml @@ -1,67 +1,70 @@ -name: Run unit tests - -on: - pull_request: - branches: - - "**" - - -jobs: - test-docker: - name: Tests (Docker) - runs-on: ubuntu-latest - - services: - postgres: - image: postgres:15 - env: - POSTGRES_USER: test - POSTGRES_PASSWORD: test - POSTGRES_DB: test - ports: - - 5432:5432 - options: >- - --health-cmd pg_isready - --health-interval 10s - --health-timeout 5s - --health-retries 5 - - steps: - - name: Checkout code - uses: actions/checkout@v4 - - - name: Build test image - run: docker build -f Dockerfile.test -t model-test . - - - name: Initialise database schema - run: | - docker run --rm \ - --network host \ - -e DB_HOST=localhost \ - -e DB_NAME=test \ - -e DB_USERNAME=test \ - -e DB_PASSWORD=test \ - -e DB_PORT=5432 \ - model-test python scripts/init_db.py - - - name: Run tests - run: | - docker run --rm \ - --network host \ - -e EPC_AUTH_TOKEN=${{ secrets.DEV_EPC_AUTH_TOKEN }} \ - -e OPEN_EPC_API_TOKEN=${{ secrets.DEV_OPEN_EPC_API_TOKEN }} \ - -e HUBSPOT_API_KEY=${{ secrets.HUBSPOT_API_KEY }} \ - -e AWS_ACCESS_KEY_ID=${{ secrets.DEV_AWS_ACCESS_KEY_ID }} \ - -e AWS_SECRET_ACCESS_KEY=${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }} \ - -e AWS_DEFAULT_REGION=${{ secrets.DEV_AWS_REGION }} \ - -e DB_HOST=localhost \ - -e DB_NAME=test \ - -e DB_USERNAME=test \ - -e DB_PASSWORD=test \ - -e DB_PORT=5432 \ - model-test pytest -vv -m 'not integration' - - # The DDD rewrite (tests/) runs in its own workflow (ddd_tests.yml): its - # SQLModel table classes map to the same physical tables as the legacy - # backend models and share the one global SQLModel.metadata, so the two - # suites cannot be imported into the same pytest process. +# Temporarily disabled β€” this Docker-based suite was consuming too many GitHub +# Actions minutes. Uncomment to re-enable. +# +# name: Run unit tests +# +# on: +# pull_request: +# branches: +# - "**" +# +# +# jobs: +# test-docker: +# name: Tests (Docker) +# runs-on: ubuntu-latest +# +# services: +# postgres: +# image: postgres:15 +# env: +# POSTGRES_USER: test +# POSTGRES_PASSWORD: test +# POSTGRES_DB: test +# ports: +# - 5432:5432 +# options: >- +# --health-cmd pg_isready +# --health-interval 10s +# --health-timeout 5s +# --health-retries 5 +# +# steps: +# - name: Checkout code +# uses: actions/checkout@v4 +# +# - name: Build test image +# run: docker build -f Dockerfile.test -t model-test . +# +# - name: Initialise database schema +# run: | +# docker run --rm \ +# --network host \ +# -e DB_HOST=localhost \ +# -e DB_NAME=test \ +# -e DB_USERNAME=test \ +# -e DB_PASSWORD=test \ +# -e DB_PORT=5432 \ +# model-test python scripts/init_db.py +# +# - name: Run tests +# run: | +# docker run --rm \ +# --network host \ +# -e EPC_AUTH_TOKEN=${{ secrets.DEV_EPC_AUTH_TOKEN }} \ +# -e OPEN_EPC_API_TOKEN=${{ secrets.DEV_OPEN_EPC_API_TOKEN }} \ +# -e HUBSPOT_API_KEY=${{ secrets.HUBSPOT_API_KEY }} \ +# -e AWS_ACCESS_KEY_ID=${{ secrets.DEV_AWS_ACCESS_KEY_ID }} \ +# -e AWS_SECRET_ACCESS_KEY=${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }} \ +# -e AWS_DEFAULT_REGION=${{ secrets.DEV_AWS_REGION }} \ +# -e DB_HOST=localhost \ +# -e DB_NAME=test \ +# -e DB_USERNAME=test \ +# -e DB_PASSWORD=test \ +# -e DB_PORT=5432 \ +# model-test pytest -vv -m 'not integration' +# +# # The DDD rewrite (tests/) runs in its own workflow (ddd_tests.yml): its +# # SQLModel table classes map to the same physical tables as the legacy +# # backend models and share the one global SQLModel.metadata, so the two +# # suites cannot be imported into the same pytest process. diff --git a/applications/modelling_e2e/handler.py b/applications/modelling_e2e/handler.py index 30c26f82..c5aa189b 100644 --- a/applications/modelling_e2e/handler.py +++ b/applications/modelling_e2e/handler.py @@ -25,6 +25,7 @@ from __future__ import annotations import dataclasses import io +import json import os from collections.abc import Callable from typing import Any, Optional, cast @@ -94,6 +95,7 @@ from repositories.property.property_overrides_postgres_reader import ( from repositories.scenario.scenario_postgres_repository import ( ScenarioPostgresRepository, ) +from repositories.solar.solar_postgres_repository import SolarPostgresRepository from utilities.aws_lambda.task_handler import task_handler from utilities.logger import setup_logger @@ -288,8 +290,9 @@ def handler(body: dict[str, Any], context: Any) -> Optional[dict[str, Any]]: try: scenario = ScenarioPostgresRepository(read_session).get_many([scenario_id])[0] products = catalogue_with_off_catalogue_overrides(read_session) + solar_reader = SolarPostgresRepository(read_session) - errors: list[int] = [] + failures: list[dict[str, Any]] = [] for property_id in property_ids: try: @@ -356,9 +359,18 @@ def handler(body: dict[str, Any], context: Any) -> Optional[dict[str, Any]]: landlord_overrides=overrides, ).effective_epc - solar_insights: Optional[dict[str, Any]] = ( - None if no_solar else _solar_insights_for(solar_client, spatial) - ) + # Read-before-fetch: the Google Solar call is paid, so skip it + # when this UPRN's insights are already persisted. Only a cache + # miss hits Google β€” re-runs cost nothing for solar. + solar_insights: Optional[dict[str, Any]] + solar_was_fetched = False + if no_solar: + solar_insights = None + else: + solar_insights = solar_reader.get(uprn) + if solar_insights is None: + solar_insights = _solar_insights_for(solar_client, spatial) + solar_was_fetched = solar_insights is not None # All Measure Types are considered: the off-catalogue overlay # (catalogue_with_off_catalogue_overrides) prices the measures the @@ -405,7 +417,8 @@ def handler(body: dict[str, Any], context: Any) -> Optional[dict[str, Any]]: if spatial is not None: uow.spatial.save(uprn, spatial) if ( - solar_insights is not None + solar_was_fetched + and solar_insights is not None and spatial is not None and spatial.coordinates is not None ): @@ -437,10 +450,18 @@ def handler(body: dict[str, Any], context: Any) -> Optional[dict[str, Any]]: except Exception as error: # noqa: BLE001 logger.error( - f"property={property_id}: {type(error).__name__}: {error}", + f"property={property_id} uprn={uprns.get(property_id)}: " + f"{type(error).__name__}: {error}", exc_info=True, ) - errors.append(property_id) + failures.append( + { + "property_id": property_id, + "uprn": uprns.get(property_id), + "error_type": type(error).__name__, + "error": str(error), + } + ) # Cohort certs the mapper could not consume were skipped (not aborted on) # so prediction could proceed; surface them β€” with cert numbers β€” in the @@ -455,12 +476,29 @@ def handler(body: dict[str, Any], context: Any) -> Optional[dict[str, Any]]: f"{[s['certificate_number'] for s in skipped_certs]}" ) - if errors: - message = f"failed property_ids: {errors}" + # A property that errored AND a cohort cert the mapper could not consume + # are both surfaced as failures, so the subtask is marked failed and + # shows up for debugging. The whole batch has already run by this point β€” + # every property that could be modelled was written to DB above β€” so + # failing here flags the run without discarding the work done so far. + if failures or skipped_certs: + parts: list[str] = [] + if failures: + failed_ids = [f["property_id"] for f in failures] + # Persisted verbatim into the subtask's outputs.error (via + # SubTask.fail): include each property's error type + message, + # not just the IDs, so failed runs are diagnosable without + # cross-referencing CloudWatch. + parts.append( + f"failed property_ids: {failed_ids}; " + f"details: {json.dumps(failures)}" + ) if skipped_certs: - message += f"; skipped_unmappable_cohort_certs: {skipped_certs}" - raise RuntimeError(message) + parts.append( + f"skipped_unmappable_cohort_certs: {json.dumps(skipped_certs)}" + ) + raise RuntimeError("; ".join(parts)) - return {"skipped_unmappable_cohort_certs": skipped_certs} if skipped_certs else None + return None finally: read_session.close() diff --git a/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-20.0.0/uprn_22086693/elmhurst_summary.pdf b/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-20.0.0/uprn_22086693/elmhurst_summary.pdf new file mode 100644 index 00000000..56569fde Binary files /dev/null and b/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-20.0.0/uprn_22086693/elmhurst_summary.pdf differ diff --git a/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-20.0.0/uprn_22086693/elmhurst_worksheet.pdf b/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-20.0.0/uprn_22086693/elmhurst_worksheet.pdf new file mode 100644 index 00000000..eb7b60b6 Binary files /dev/null and b/backend/epc_api/json_samples/real_life_examples/RdSAP-Schema-20.0.0/uprn_22086693/elmhurst_worksheet.pdf differ diff --git a/backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.0/uprn_10070004512/elmhurst_summary.pdf b/backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.0/uprn_10070004512/elmhurst_summary.pdf new file mode 100644 index 00000000..eddc4314 Binary files /dev/null and b/backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.0/uprn_10070004512/elmhurst_summary.pdf differ diff --git a/backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.0/uprn_10070004512/elmhurst_worksheet.pdf b/backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.0/uprn_10070004512/elmhurst_worksheet.pdf new file mode 100644 index 00000000..b71391c8 Binary files /dev/null and b/backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.0/uprn_10070004512/elmhurst_worksheet.pdf differ diff --git a/backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.0/uprn_10070004512/epc.json b/backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.0/uprn_10070004512/epc.json new file mode 100644 index 00000000..63dbcfab --- /dev/null +++ b/backend/epc_api/json_samples/real_life_examples/SAP-Schema-16.0/uprn_10070004512/epc.json @@ -0,0 +1,281 @@ +{ + "uprn": 10070004512, + "roofs": [ + { + "description": "(another dwelling above)", + "energy_efficiency_rating": 0, + "environmental_efficiency_rating": 0 + } + ], + "walls": [ + { + "description": "Cavity wall, filled cavity", + "energy_efficiency_rating": 4, + "environmental_efficiency_rating": 4 + } + ], + "floors": [ + { + "description": "To external air, no insulation (assumed)", + "energy_efficiency_rating": 0, + "environmental_efficiency_rating": 0 + } + ], + "status": "entered", + "windows": [ + { + "description": "Fully double glazed", + "energy_efficiency_rating": 3, + "environmental_efficiency_rating": 3 + } + ], + "lighting": { + "description": "Low energy lighting in all fixed outlets", + "energy_efficiency_rating": 5, + "environmental_efficiency_rating": 5 + }, + "postcode": "BR1 4QF", + "hot_water": { + "description": "Electric immersion, off-peak", + "energy_efficiency_rating": 3, + "environmental_efficiency_rating": 1 + }, + "post_town": "BROMLEY", + "created_at": "2012-04-20 10:59:22.000000", + "door_count": 1, + "glazed_area": 1, + "region_code": 14, + "report_type": 2, + "sap_heating": { + "wwhrs": { + "rooms_with_bath_and_or_shower": 1, + "rooms_with_mixer_shower_no_bath": 0, + "rooms_with_bath_and_mixer_shower": 0 + }, + "cylinder_size": 1, + "water_heating_code": 903, + "water_heating_fuel": 29, + "main_heating_details": [ + { + "has_fghrs": "N", + "main_fuel_type": 29, + "heat_emitter_type": 0, + "main_heating_number": 1, + "main_heating_control": 2401, + "main_heating_category": 7, + "main_heating_fraction": 1, + "sap_main_heating_code": 402, + "main_heating_data_source": 2 + } + ], + "immersion_heating_type": 1, + "has_fixed_air_conditioning": "false" + }, + "sap_version": 9.91, + "schema_type": "SAP-Schema-16.0", + "uprn_source": "Energy Assessor", + "country_code": "EAW", + "main_heating": [ + { + "description": "Electric storage heaters", + "energy_efficiency_rating": 3, + "environmental_efficiency_rating": 1 + } + ], + "dwelling_type": "Ground-floor flat", + "language_code": 1, + "property_type": 2, + "address_line_1": "54a, Boyland Road", + "schema_version": "LIG-16.0", + "assessment_type": "RdSAP", + "completion_date": "2012-04-20", + "inspection_date": "2012-04-20", + "extensions_count": 0, + "measurement_type": 1, + "sap_flat_details": { + "level": 1, + "top_storey": "N", + "flat_location": 0, + "heat_loss_corridor": 0 + }, + "total_floor_area": 33, + "transaction_type": 3, + "conservatory_type": 1, + "heated_room_count": 2, + "registration_date": "2012-04-20", + "restricted_access": 1, + "sap_energy_source": { + "main_gas": "N", + "meter_type": 1, + "photovoltaic_supply": { + "percent_roof_area": 0 + }, + "wind_turbines_count": 0, + "wind_turbines_terrain_type": 1 + }, + "secondary_heating": { + "description": "Portable electric heaters (assumed)", + "energy_efficiency_rating": 0, + "environmental_efficiency_rating": 0 + }, + "sap_building_parts": [ + { + "identifier": "Main Dwelling", + "wall_dry_lined": "N", + "floor_heat_loss": 1, + "roof_construction": 3, + "wall_construction": 4, + "building_part_number": 1, + "sap_floor_dimensions": [ + { + "floor": 0, + "room_height": 2.42, + "floor_insulation": 1, + "total_floor_area": 33.24, + "floor_construction": 0, + "heat_loss_perimeter": 10.39 + } + ], + "wall_insulation_type": 2, + "construction_age_band": "B", + "wall_thickness_measured": "N", + "roof_insulation_location": "ND", + "roof_insulation_thickness": "ND" + } + ], + "low_energy_lighting": 100, + "solar_water_heating": "N", + "bedf_revision_number": 321, + "habitable_room_count": 2, + "heating_cost_current": { + "value": 303, + "currency": "GBP" + }, + "insulated_door_count": 0, + "co2_emissions_current": { + "value": 2.9, + "quantity": "tonnes per year" + }, + "energy_rating_average": 60, + "energy_rating_current": 66, + "lighting_cost_current": { + "value": 23, + "currency": "GBP" + }, + "main_heating_controls": [ + { + "description": "Manual charge control", + "energy_efficiency_rating": 2, + "environmental_efficiency_rating": 2 + } + ], + "multiple_glazing_type": 3, + "open_fireplaces_count": 0, + "has_hot_water_cylinder": "false", + "heating_cost_potential": { + "value": 190, + "currency": "GBP" + }, + "hot_water_cost_current": { + "value": 106, + "currency": "GBP" + }, + "mechanical_ventilation": 0, + "percent_draughtproofed": 100, + "suggested_improvements": [ + { + "sequence": 1, + "typical_saving": { + "value": 86, + "currency": "GBP" + }, + "indicative_cost": "\u00a3800 - \u00a31,200", + "improvement_type": "W", + "improvement_details": { + "improvement_number": 47 + }, + "improvement_category": 5, + "energy_performance_rating": 73, + "environmental_impact_rating": 58 + }, + { + "sequence": 2, + "typical_saving": { + "value": 27, + "currency": "GBP" + }, + "indicative_cost": "\u00a3600 - \u00a3800", + "improvement_type": "L", + "improvement_details": { + "improvement_number": 25 + }, + "improvement_category": 5, + "energy_performance_rating": 75, + "environmental_impact_rating": 60 + } + ], + "co2_emissions_potential": { + "value": 2.1, + "quantity": "tonnes per year" + }, + "energy_rating_potential": 75, + "lighting_cost_potential": { + "value": 23, + "currency": "GBP" + }, + "alternative_improvements": [ + { + "sequence": 1, + "typical_saving": { + "value": 24, + "currency": "GBP" + }, + "improvement_type": "J2", + "improvement_details": { + "improvement_number": 54 + }, + "improvement_category": 6, + "energy_performance_rating": 76, + "environmental_impact_rating": 93 + }, + { + "sequence": 2, + "typical_saving": { + "value": 84, + "currency": "GBP" + }, + "improvement_type": "Z1", + "improvement_details": { + "improvement_number": 51 + }, + "improvement_category": 6, + "energy_performance_rating": 80, + "environmental_impact_rating": 81 + } + ], + "hot_water_cost_potential": { + "value": 106, + "currency": "GBP" + }, + "renewable_heat_incentive": { + "water_heating": 1434, + "space_heating_existing_dwelling": 4064 + }, + "seller_commission_report": "Y", + "energy_consumption_current": 497, + "has_fixed_air_conditioning": "false", + "multiple_glazed_proportion": 100, + "calculation_software_version": 4.1, + "energy_consumption_potential": 365, + "environmental_impact_current": 47, + "fixed_lighting_outlets_count": 5, + "current_energy_efficiency_band": "D", + "environmental_impact_potential": 60, + "has_heated_separate_conservatory": "false", + "potential_energy_efficiency_band": "C", + "co2_emissions_current_per_floor_area": { + "value": 88, + "quantity": "kg/m2 per year" + }, + "low_energy_fixed_lighting_outlets_count": 5 +} \ No newline at end of file diff --git a/datatypes/epc/domain/mapper.py b/datatypes/epc/domain/mapper.py index 2f2bd00b..1658383f 100644 --- a/datatypes/epc/domain/mapper.py +++ b/datatypes/epc/domain/mapper.py @@ -2150,7 +2150,10 @@ class EpcPropertyDataMapper: door_count=schema.door_count, habitable_rooms_count=schema.habitable_room_count, heated_rooms_count=schema.heated_room_count, - wet_rooms_count=schema.wet_rooms_count, + # 21.0.1 lodges wet_rooms_count as Optional; None violates the + # epc_property NOT-NULL column (and the calc's `> 0` check). Coalesce + # to 0 like every other mapper (RdSAP "not lodged" β†’ calc minimum 1). + wet_rooms_count=schema.wet_rooms_count or 0, extensions_count=schema.extensions_count, open_chimneys_count=schema.open_chimneys_count, insulated_door_count=schema.insulated_door_count, diff --git a/datatypes/epc/domain/tests/test_from_rdsap_schema.py b/datatypes/epc/domain/tests/test_from_rdsap_schema.py index e0c6b1ba..023f7c60 100644 --- a/datatypes/epc/domain/tests/test_from_rdsap_schema.py +++ b/datatypes/epc/domain/tests/test_from_rdsap_schema.py @@ -379,6 +379,22 @@ class TestFromRdSapSchema21_0_1: # --- general --- + def test_omitted_wet_rooms_count_defaults_to_zero_not_none(self) -> None: + # 21.0.1 lodges wet_rooms_count as Optional, so a cert that omits it + # mapped to EpcPropertyData.wet_rooms_count=None. That None then + # violated the epc_property.wet_rooms_count NOT-NULL constraint when a + # predicted EPC (which deep-copies a comparable template) was persisted, + # and would crash the calc's `wet_rooms_count > 0` check. 37 modelling_e2e + # properties failed on the 2026-06-23 run for this reason. Default to 0, + # matching every other mapper (RdSAP 0 β†’ calc's Table-?? minimum 1 fan). + data = load("21_0_1.json") + data.pop("wet_rooms_count", None) + schema = from_dict(RdSapSchema21_0_1, data) + + result = EpcPropertyDataMapper.from_rdsap_schema_21_0_1(schema) + + assert result.wet_rooms_count == 0 + def test_uprn(self, result: EpcPropertyData) -> None: assert result.uprn == 12457 diff --git a/domain/modelling/generators/glazing_recommendation.py b/domain/modelling/generators/glazing_recommendation.py index 897b011c..53a77bac 100644 --- a/domain/modelling/generators/glazing_recommendation.py +++ b/domain/modelling/generators/glazing_recommendation.py @@ -21,7 +21,7 @@ scoring (ADR-0016). from dataclasses import dataclass from typing import Final, Optional -from datatypes.epc.domain.epc_property_data import EpcPropertyData +from datatypes.epc.domain.epc_property_data import EpcPropertyData, SapWindow from domain.geospatial.planning_restrictions import PlanningRestrictions from domain.modelling.measure_type import MeasureType from domain.modelling.recommendation import Cost, MeasureOption, Recommendation @@ -48,6 +48,7 @@ class _GlazingTarget: glazing_type: int u_value: float solar_transmittance: float + frame_factor: float # Unrestricted: replace the units with double glazing (gt=5 "Double post 2022"; @@ -58,6 +59,10 @@ _DOUBLE: Final[_GlazingTarget] = _GlazingTarget( glazing_type=5, u_value=1.40, solar_transmittance=0.72, + # Replacement double units re-lodge a standard FF=0.70 (cert 001431's + # after), overriding the panes they replace (e.g. FF 1.00 / 0.50 on the + # "single glazing, known data" windows) β€” feeds Β§6 solar gains. + frame_factor=0.70, ) # Protected (conservation/listed/heritage): fit an internal secondary pane # (gt=11 "Secondary glazing - Normal emissivity", what cert 001431 re-lodges; @@ -70,9 +75,61 @@ _SECONDARY: Final[_GlazingTarget] = _GlazingTarget( glazing_type=11, u_value=2.90, solar_transmittance=0.85, + frame_factor=0.70, # cert 001431's after re-lodges FF=0.70. ) +def _is_draught_proofed(window: SapWindow) -> bool: + """`SapWindow.draught_proofed` is `Union[bool, str]` (bool from the + site-notes mapper, the string "true"/"false" from the API). Normalise + to a bool.""" + flag = window.draught_proofed + if isinstance(flag, bool): + return flag + return flag.strip().lower() in {"true", "yes"} + + +def _recompute_percent_draughtproofed( + epc: EpcPropertyData, upgraded_indices: tuple[int, ...] +) -> Optional[int]: + """RdSAP 10 Β§8.1 draught-proofing percentage after a glazing upgrade. + + Β§8.1: "[(number of draughtproofed openable windows & doors) / (total + number of openable windows & doors)] Γ— 100", as an integer. Sealed + double/secondary units are draught-proofed, so every upgraded window + that was NOT draught-proofed flips into the numerator. + + Anchored on the lodged dwelling-level `percent_draughtproofed` (the + value the Β§2 cascade reads) rather than re-deriving the before-count + from per-window flags: the unchanged openings are already folded into + that aggregate, so this is robust to incomplete window extraction. + + d0 = round(before% / 100 Γ— N) # draught-proofed before + after = round((d0 + flips) / N Γ— 100) # RdSAP 10 Β§8.1, integer + + N counts every openable window (vertical + roof) plus external doors. + Returns None when no before% is lodged or there are no openings. + """ + before = epc.percent_draughtproofed + if before is None: + return None + n_openings = ( + len(epc.sap_windows) + + len(epc.sap_roof_windows or []) + + (epc.door_count or 0) + ) + if n_openings <= 0: + return None + flips = sum( + 1 + for index in upgraded_indices + if not _is_draught_proofed(epc.sap_windows[index]) + ) + draught_proofed_before = round(before / 100 * n_openings) + after = round((draught_proofed_before + flips) / n_openings * 100) + return max(0, min(100, after)) + + def recommend_glazing( epc: EpcPropertyData, products: ProductRepository, @@ -99,9 +156,15 @@ def recommend_glazing( glazing_type=target.glazing_type, u_value=target.u_value, solar_transmittance=target.solar_transmittance, + frame_factor=target.frame_factor, ) for index in single_indices - } + }, + # Sealed units draught-proof the panes they replace β€” RdSAP 10 + # Β§8.1 re-lodges the dwelling's percentage (cert 001431: 84 β†’ 100). + percent_draughtproofed=_recompute_percent_draughtproofed( + epc, single_indices + ), ) cost = Cost( total=len(single_indices) * product.unit_cost_per_m2, diff --git a/domain/modelling/scoring/overlay_applicator.py b/domain/modelling/scoring/overlay_applicator.py index 961f4e02..93875bbf 100644 --- a/domain/modelling/scoring/overlay_applicator.py +++ b/domain/modelling/scoring/overlay_applicator.py @@ -9,10 +9,12 @@ then discarded). See ADR-0016. import copy from dataclasses import fields -from typing import Optional, Sequence +from typing import Final, Optional, Sequence from datatypes.epc.domain.epc_property_data import ( + BuildingPartIdentifier, EpcPropertyData, + SapBuildingPart, SapVentilation, SapWindow, WindowTransmissionDetails, @@ -29,6 +31,39 @@ from domain.modelling.simulation import ( ) +# A Landlord Override's building part is a POSITIONAL index (0=main, 1=extension +# 1…, ADR-0004), translated to a `BuildingPartIdentifier` upstream. This recovers +# that position so an override can fall back onto the part the EPC actually models +# at that slot when the gov-API labelled it differently (e.g. lodged a 2nd part as +# `other` rather than `extension_1`). +_POSITION_BY_IDENTIFIER: Final[dict[BuildingPartIdentifier, int]] = { + BuildingPartIdentifier.MAIN: 0, + BuildingPartIdentifier.EXTENSION_1: 1, + BuildingPartIdentifier.EXTENSION_2: 2, + BuildingPartIdentifier.EXTENSION_3: 3, + BuildingPartIdentifier.EXTENSION_4: 4, +} + + +def _resolve_part( + epc: EpcPropertyData, + parts_by_id: dict[BuildingPartIdentifier, SapBuildingPart], + identifier: BuildingPartIdentifier, +) -> Optional[SapBuildingPart]: + """The building part an overlay targets: the EPC's part with that identifier + when present, else the part at the override's positional index (so a + correction for `extension_1` still lands on the EPC's 2nd part even when the + gov-API lodged it under a different label). ``None`` when the EPC models no + part at that position.""" + part = parts_by_id.get(identifier) + if part is not None: + return part + position = _POSITION_BY_IDENTIFIER.get(identifier) + if position is None or position >= len(epc.sap_building_parts): + return None + return epc.sap_building_parts[position] + + def apply_simulations( baseline: EpcPropertyData, simulations: Sequence[EpcSimulation] ) -> EpcPropertyData: @@ -41,7 +76,12 @@ def apply_simulations( for simulation in simulations: for identifier, overlay in simulation.building_parts.items(): - part = parts_by_id[identifier] + part = _resolve_part(result, parts_by_id, identifier) + # No part at this position β€” the EPC models fewer parts than the + # override's index. We have no geometry to model the missing part, so + # skip it rather than crash the whole property's modelling. + if part is None: + continue for overlay_field in fields(overlay): value = getattr(overlay, overlay_field.name) if value is not None: @@ -66,6 +106,8 @@ def apply_simulations( result.property_type = simulation.property_type if simulation.built_form is not None: result.built_form = simulation.built_form + if simulation.percent_draughtproofed is not None: + result.percent_draughtproofed = simulation.percent_draughtproofed return result @@ -178,11 +220,13 @@ def _fold_lighting(epc: EpcPropertyData, overlay: LightingOverlay) -> None: def _fold_window(window: SapWindow, overlay: WindowOverlay) -> None: """Write a `WindowOverlay`'s non-``None`` fields onto a (copied) window: - ``glazing_type`` flat on the window, ``u_value`` / ``solar_transmittance`` - into its `WindowTransmissionDetails` (where the cascade reads them), starting - a fresh one when the window lodged none.""" + ``glazing_type`` and ``frame_factor`` flat on the window, ``u_value`` / + ``solar_transmittance`` into its `WindowTransmissionDetails` (where the + cascade reads them), starting a fresh one when the window lodged none.""" if overlay.glazing_type is not None: window.glazing_type = overlay.glazing_type + if overlay.frame_factor is not None: + window.frame_factor = overlay.frame_factor if overlay.u_value is None and overlay.solar_transmittance is None: return details: Optional[WindowTransmissionDetails] = window.window_transmission_details diff --git a/domain/modelling/simulation.py b/domain/modelling/simulation.py index 77ce2537..2f6fc94d 100644 --- a/domain/modelling/simulation.py +++ b/domain/modelling/simulation.py @@ -77,12 +77,16 @@ class WindowOverlay: are written into the window's `WindowTransmissionDetails` β€” where the calculator reads heat loss and solar gain from β€” because our calculator consumes the lodged values directly rather than deriving them from - `glazing_type`. A `None` field means "leave the baseline value unchanged". + `glazing_type`. `frame_factor` is written flat on the window (the Β§6 + solar-gain area factor); a replacement unit re-lodges its own FF, which + can differ from the pane it replaced. A `None` field means "leave the + baseline value unchanged". """ glazing_type: Optional[int] = None u_value: Optional[float] = None solar_transmittance: Optional[float] = None + frame_factor: Optional[float] = None @dataclass(frozen=True) @@ -263,3 +267,8 @@ class EpcSimulation: # built form (property_type drives party-wall heat loss + measure eligibility). property_type: Optional[str] = None built_form: Optional[str] = None + # Whole-dwelling RdSAP 10 Β§8.1 draught-proofing percentage, when a + # Measure changes it (e.g. glazing: sealed units draught-proof the + # panes they replace). The Β§2 cascade reads this dwelling-level value, + # so the overlay sets it directly. `None` leaves the baseline's value. + percent_draughtproofed: Optional[int] = None diff --git a/scripts/hyde/build_10070004512.py b/scripts/hyde/build_10070004512.py new file mode 100644 index 00000000..2210242a --- /dev/null +++ b/scripts/hyde/build_10070004512.py @@ -0,0 +1,288 @@ +"""Elmhurst build for UPRN 10070004512 (SAP-Schema-16.0, GROUND-FLOOR FLAT, +band B, cavity FILLED, ELECTRIC STORAGE HEATERS (SAP 402 SEB, manual charge +control CSA/2401) + electric immersion off-peak (Economy-7 Dual meter) with a +cylinder (size 1), roof = another dwelling above, floor to EXTERNAL AIR, double +glazed, TFA 33.24, window 4.88 mΒ². Engine 66 = lodged 66. + +P1 of the modelling_e2e corpus validation β€” the built_form fix cert (16.0 omitted +built_form; mapper derives it from dwelling_type β†’ flatβ†’modal 4). built_form is +ML-only so SAP-neutral; engine reproduces lodged exactly. Storage-heater build +(see build_10022893721.py). Engine models NO secondary (sap_heating. +secondary_heating_type is None) β†’ secondary present=No to match. Run: + DISPLAY=:99 python scripts/hyde/build_10070004512.py +""" +from __future__ import annotations +import sys +import elmhurst_lib as E + +DIM = "TabContainer_TabPanelMain_WebUserControlDimensionsMain_" +WALL = ("TabContainer_TabPanelMain_InnerTabContainerMain_" + "TabPanelExternalWallMain_WebUserControlWallMain_") +ROOF = "TabContainer_TabPanelMain_WebUserControlRoofMain_" +FLOOR = "TabContainer_TabPanelMain_WebUserControlFloorsMain_" +WP = "TabContainer_TabPanelWindowsPanel_" +DP = "TabContainer_TabPanelDoorsPanel_" +VP = "TabContainer_TabPanelVentilationPanel_" +APT = "TabContainer_TabPanelAirPressureTest_" +LP = "TabContainer_TabPanelLighting_" +MV = "TabContainer_TabPanelMechVent_" +WH = "TabContainer_TabPanelWaterHeating_" + + +def _pick(page, suffix, contains): + val = page.evaluate( + """(a)=>{const s=document.getElementById(a[0]);if(!s)return null; + for(const o of s.options){if(o.text.toLowerCase().includes(a[1].toLowerCase()))return o.value;}return null;}""", + [f"{E.FP}{suffix}", contains]) + if val is not None: + E.set_select(page, suffix, val) + return val + + +def _options(page, suffix): + return page.evaluate( + """(id)=>{const s=document.getElementById(id);if(!s)return []; + return Array.from(s.options).map(o=>o.text);}""", f"{E.FP}{suffix}") + + +def property_description(page): + E.goto(page, "PropertyDescription", "WebFormPropertyDescription.aspx") + E.set_select(page, "DropDownListPropertyType1", "F Flat") + _pick(page, "DropDownListPropertyType2", "mid-terrace") # built_form 4 + E.set_text(page, "TextBoxStoreys", "1") + E.set_text(page, "TextBoxHabitableRooms", "2") + E.set_text(page, "TextBoxHeatedHabitableRooms", "2") + print("date ->", _pick(page, "DropDownListDateBuiltMain", "1900-1929")) # band B + E.set_select(page, "DropDownListDateBuiltFirst", "") + E.set_select(page, "DropDownListRoomInRoofMain", "") + E.save_close(page) + + +def flats(page): + E.goto(page, "Flats", "WebFormFlats.aspx") + E.set_select(page, "DropDownListPositionOfFlat", "Ground Floor") + E.set_text(page, "TextBoxFloor", "0") + E.set_select(page, "RadioButtonListFlatCoridor", "None") + E.save_close(page) + + +def dimensions(page): + E.goto(page, "Dimensions", "WebFormDimensions.aspx") + E.set_text(page, f"{DIM}TextBoxFloorAreaLowestFloor", "33.24") + E.set_text(page, f"{DIM}TextBoxRoomHeightLowestFloor", "2.42") + E.set_text(page, f"{DIM}TextBoxWallPerimeterLowestFloor", "10.39") + # 16.0 lodges no party_wall_length; a flat's side party walls are unmodelled by + # the engine. Try 0 (Elmhurst may require non-zero β€” adjust if Recommendations + # complains). + E.set_text(page, f"{DIM}TextBoxPartyWallLengthLowestFloor", "0") + E.save_close(page) + + +def walls(page): + E.goto(page, "Walls", "WebFormWalls.aspx") + E.set_select(page, f"{WALL}DropDownListType", "CA Cavity") + page.wait_for_timeout(400) + print("insulation ->", _pick(page, f"{WALL}DropDownListInsulation", "filled")) + E.save_close(page) + + +def roofs(page): + E.goto(page, "Roofs", "WebFormRoofs.aspx") + _pick(page, f"{ROOF}DropDownListType", "another dwelling above") + E.save_close(page) + + +def floors(page): + E.goto(page, "Floors", "WebFormFloors.aspx") + # Floor is "to external air" β€” that is the LOCATION/exposure, not a TYPE. + _pick(page, f"{FLOOR}DropDownListLocation", "external air") # E To external air + page.wait_for_timeout(400) + _pick(page, f"{FLOOR}DropDownListType", "solid") # construction; U from exposure + ins = page.locator(f"#{E.FP}{FLOOR}DropDownListInsulation") + if ins.count(): + E.set_select(page, f"{FLOOR}DropDownListInsulation", "A As built") + E.save_close(page) + + +def openings(page): + E.goto(page, "Openings", "WebFormOpenings.aspx") + E.click_tab(page, "TabContainer_TabPanelWindowsPanel") + _add_window(page, 4.88, "North", _glazing(page)) + _delete_zero_rows(page) + E.click_tab(page, "TabContainer_TabPanelDoorsPanel") + E.set_text(page, f"{DP}TextBoxDoors", "1") + E.set_text(page, f"{DP}TextBoxDoorsInsulated", "0") + E.set_text(page, f"{DP}TextBoxDraughtProofedDoors", "0") + E.save_close(page) + + +def _glazing(page): + for needle in ("unknown install date", "before 2002", "pre 2002"): + for opt in _options(page, f"{WP}DropDownListExtGlazing"): + low = opt.lower() + if needle in low and "triple" not in low and "single" not in low and "known data" not in low: + return opt + return "Double post or during 2022" + + +def _add_window(page, area, orientation, glazing): + print("glazing ->", glazing) + E.set_select(page, f"{WP}DropDownListExtGlazing", glazing) + page.wait_for_timeout(400) + ft = page.locator(f"#{E.FP}{WP}DropDownListExtFrameType") + if ft.count(): + ft.select_option("PVC") + gg = page.locator(f"#{E.FP}{WP}DropDownListExtGlazingGap") + if gg.count(): + gg.select_option("12 mm") + wid = f"{E.FP}{WP}TextBoxExtWidth" + page.evaluate( + """(a)=>{const e=document.getElementById(a[0]);if(e){e.value=a[1]; + e.dispatchEvent(new Event('input',{bubbles:true})); + e.dispatchEvent(new Event('change',{bubbles:true})); + e.dispatchEvent(new Event('blur',{bubbles:true}));}}""", [wid, str(area)]) + page.locator(f"#{E.FP}{WP}TextBoxExtHeight").fill("1.00") + page.locator(f"#{E.FP}{WP}DropDownListExtOrientation").select_option(orientation) + page.locator(f"#{E.FP}{WP}DropDownListExtBuildingPartId").select_option("Main") + page.locator(f"#{E.FP}{WP}DropDownListExtLocation").select_option("External wall") + page.wait_for_timeout(300) + before = E.window_row_count(page) + page.evaluate("(id)=>{const e=document.getElementById(id); if(e)e.click();}", f"{E.FP}{WP}ButtonAddWindow") + for _ in range(25): + page.wait_for_timeout(200) + if E.window_row_count(page) > before: + break + + +def _grid_rows(page): + return page.evaluate( + """()=>{const t=document.querySelector("[id*=GridViewExtendedWidows]"); + if(!t)return[];return Array.from(t.querySelectorAll('tr')).slice(1) + .map(r=>Array.from(r.querySelectorAll('td')).map(c=>c.innerText.trim()));}""") + + +def _delete_zero_rows(page): + g = 0 + while g < 6 and E.window_row_count(page) > 1: + g += 1 + rows = _grid_rows(page) + bad = next((i for i, c in enumerate(rows) if len(c) > 1 and c[1] in ("0.00", "0", "0.0")), None) + if bad is None: + break + _delete_row(page, bad) + page.wait_for_timeout(400) + + +def _delete_row(page, idx): + before = E.window_row_count(page) + btn = page.evaluate( + """(i)=>{const b=document.querySelectorAll("[id*='GridViewExtendedWidows_DeleteButton_']");return b[i]?b[i].id:null;}""", idx) + if not btn: + return + page.evaluate("(id)=>{const e=document.getElementById(id); if(e)e.click();}", btn) + page.wait_for_selector(f"#{E.FP}DeleteWindowDialog_LinkButtonYes", state="visible", timeout=5000) + page.evaluate("(id)=>{const e=document.getElementById(id); if(e)e.click();}", f"{E.FP}DeleteWindowDialog_LinkButtonYes") + for _ in range(20): + page.wait_for_timeout(200) + if E.window_row_count(page) < before: + break + + +def ventilation(page): + E.goto(page, "VentilationAndCooling", "WebFormVentilationAndCooling.aspx") + E.click_tab(page, "TabContainer_TabPanelVentilationPanel") + E.set_text(page, f"{VP}TextBoxIntermittentFans", "0") + cool = page.locator(f"#{E.FP}{VP}CheckBoxFixedSpaceCooling") + if cool.count() and cool.is_checked(): + E.commit(page, cool.uncheck) + E.click_tab(page, "TabContainer_TabPanelMechVent") + mv = page.locator(f"#{E.FP}{MV}CheckBoxMechanicalVentilation") + if mv.count() and mv.is_checked(): + E.commit(page, mv.uncheck) + E.click_tab(page, "TabContainer_TabPanelAirPressureTest") + E.set_select(page, f"{APT}DropDownListTestMethod", "Not available") + E.click_tab(page, "TabContainer_TabPanelLighting") + E.set_text(page, f"{LP}TextBoxLightsTotal", "5") + E.set_text(page, f"{LP}TextBoxLedLightsTotal", "5") # 100% low energy + E.set_text(page, f"{LP}TextBoxCflLightsTotal", "0") + E.save_close(page) + + +def space_heating(page): + # Electric storage heaters (SAP 402 = SEB), manual charge control (SAP 2401 = + # CSA). Two passes: clear bound PCDB boiler first, then set the SAP-table code. + E.goto(page, "SpaceHeating", "WebFormSpaceHeating.aspx") + page.wait_for_timeout(1000) + rid = f"{E.MH1}TextBoxPCDFBoilerReference" + ref = page.locator(f"#{rid}").input_value() + if ref not in ("0", ""): + print(f"clearing bound PCDB boiler {ref} -> 0 (rerun space_heating)") + page.evaluate("""(rid)=>{const r=document.getElementById(rid);r.value='0'; + r.dispatchEvent(new Event('change',{bubbles:true}));}""", rid) + page.wait_for_timeout(500) + E.save_close(page) + return + E.set_heating_dialog(page, "TabContainer_TabPanelMainHeating1_WebUserControlMainHeating1_ButtonMainHeatingCode", + "^Electric", "^Electric", "Storage", "SEB Modern slimline") + print("code:", page.locator(f"#{E.MH1}TextBoxMainHeatingCode").input_value()) + E.set_heating_dialog(page, "TabContainer_TabPanelMainHeating1_WebUserControlMainHeating1_ButtonMainHeatingControls", + "Storage Radiator", "CSA Manual charge control") + print("control:", page.locator(f"#{E.MH1}TextBoxMainHeatingControls").input_value()) + E.set_select(page, "DropDownListSecondaryHeatingPresent", "No") + # Economy-7 Dual meter (cert meter_type 1) β€” hidden Meters sub-tab. + E.click_tab(page, "TabContainer_TabPanelMeters") + E.set_select(page, "TabContainer_TabPanelMeters_RadioButtonListElectricityType", "Dual") + print("meter:", page.locator("#ContentBody_ContentPlaceHolder1_TabContainer_TabPanelMeters_RadioButtonListElectricityType").input_value()) + E.save_close(page) + + +def water_heating(page): + # Electric immersion off-peak (Dual) WITH cylinder (size 1 = small). The + # immersion code REQUIRES a cylinder. + E.goto(page, "WaterHeating", "WebFormWaterHeating.aspx") + E.click_tab(page, "TabContainer_TabPanelWaterHeating") + page.wait_for_timeout(600) + E.set_heating_dialog(page, f"{WH}ButtonWaterHeatingCode", + "Water Heater", "^Electric", "Immersion") + print("water code:", page.locator(f"#{E.FP}{WH}TextBoxWaterHeatingCode").input_value()) + cid = f"{E.FP}{WH}CheckBoxHotWaterCylinder" + cyl = page.locator(f"#{cid}") + if cyl.count() and not cyl.is_checked(): + try: + with page.expect_navigation(wait_until="load", timeout=8000): + page.evaluate( + """(id)=>{const c=document.getElementById(id);if(c){c.checked=true; + c.dispatchEvent(new Event('click',{bubbles:true})); + c.dispatchEvent(new Event('change',{bubbles:true}));}}""", cid) + except Exception: + page.wait_for_timeout(2000) + print("cylinder present:", cyl.is_checked() if cyl.count() else "n/a") + print("cyl sizes:", _options(page, f"{WH}DropDownListCylinderSize")) + _pick(page, f"{WH}DropDownListCylinderSize", "small") or \ + E.set_select(page, f"{WH}DropDownListCylinderSize", "Normal") + E.set_select(page, f"{WH}DropDownListInsulated", "Foam") + isuf = f"{WH}DropDownListInsulationThickness" + if page.locator(f"#{E.FP}{isuf}").count(): + _pick(page, isuf, "Unknown") or E.set_select(page, isuf, "25 mm") + imm = page.locator(f"#{E.FP}{WH}RadioButtonListImmersionHeater") + if imm.count(): + E.set_select(page, f"{WH}RadioButtonListImmersionHeater", "Dual") + E.save_close(page) + + +_ORDER = ["property_description", "flats", "dimensions", "walls", "roofs", + "floors", "openings", "ventilation", "space_heating", "water_heating"] + + +def main(): + if len(sys.argv) < 2 or sys.argv[1] not in _ORDER: + print("usage: build_10070004512.py <" + "|".join(_ORDER) + ">") + return 2 + with E.session() as (ctx, page): + globals()[sys.argv[1]](page) + print("done:", sys.argv[1], "->", page.url) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/hyde/build_22086693.py b/scripts/hyde/build_22086693.py new file mode 100644 index 00000000..5f102a45 --- /dev/null +++ b/scripts/hyde/build_22086693.py @@ -0,0 +1,307 @@ +"""Elmhurst build for UPRN 22086693 (RdSAP-Schema-20.0.0, SEMI-DETACHED HOUSE, +2-storey, band C 1930-1949, cavity UNINSULATED, mains-gas COMBI (no PCDB index β†’ +generic SAP Table 4b combi), control 2106 (CBE), pitched 200 mm loft, suspended +uninsulated floor, party wall 6.8 m, double glazed, 2Γ— PV ARRAYS @1.14 kW +(orientations SE/SW, pitch 30Β°, overshading modest), electric SECONDARY room +heater (SAP 691 REA), TFA ~74, window 11.84 mΒ². Engine 66 / lodged 72. + +P2 of the modelling_e2e corpus validation β€” the photovoltaic_supply-as-list fix +cert. The PV adds +5 (engine 61β†’66) so it IS credited; the βˆ’6 vs lodged is a +fabric/heating gap to localise. NEW build elements: generic (no-PCDB) combi + +the Renewables/PV page. Run: + DISPLAY=:99 python scripts/hyde/build_22086693.py +""" +from __future__ import annotations +import sys +import elmhurst_lib as E + +DIM = "TabContainer_TabPanelMain_WebUserControlDimensionsMain_" +WALL = ("TabContainer_TabPanelMain_InnerTabContainerMain_" + "TabPanelExternalWallMain_WebUserControlWallMain_") +PWALL = "TabContainer_TabPanelMain_InnerTabContainerMain_TabPanelPartyWallMain_WebUserControlPartyWallMain_" +ROOF = "TabContainer_TabPanelMain_WebUserControlRoofMain_" +FLOOR = "TabContainer_TabPanelMain_WebUserControlFloorsMain_" +WP = "TabContainer_TabPanelWindowsPanel_" +DP = "TabContainer_TabPanelDoorsPanel_" +VP = "TabContainer_TabPanelVentilationPanel_" +APT = "TabContainer_TabPanelAirPressureTest_" +LP = "TabContainer_TabPanelLighting_" +MV = "TabContainer_TabPanelMechVent_" +WH = "TabContainer_TabPanelWaterHeating_" +MH1B = "TabContainer_TabPanelMainHeating1_WebUserControlMainHeating1_" + + +def _pick(page, suffix, contains): + val = page.evaluate( + """(a)=>{const s=document.getElementById(a[0]);if(!s)return null; + for(const o of s.options){if(o.text.toLowerCase().includes(a[1].toLowerCase()))return o.value;}return null;}""", + [f"{E.FP}{suffix}", contains]) + if val is not None: + E.set_select(page, suffix, val) + return val + + +def _options(page, suffix): + return page.evaluate( + """(id)=>{const s=document.getElementById(id);if(!s)return []; + return Array.from(s.options).map(o=>o.text);}""", f"{E.FP}{suffix}") + + +def property_description(page): + E.goto(page, "PropertyDescription", "WebFormPropertyDescription.aspx") + E.set_select(page, "DropDownListPropertyType1", "H House") + _pick(page, "DropDownListPropertyType2", "semi") # built_form 2 + E.set_text(page, "TextBoxStoreys", "2") + E.set_text(page, "TextBoxHabitableRooms", "4") + E.set_text(page, "TextBoxHeatedHabitableRooms", "4") + print("date ->", _pick(page, "DropDownListDateBuiltMain", "1930-1949")) # band C + E.set_select(page, "DropDownListDateBuiltFirst", "") + E.set_select(page, "DropDownListRoomInRoofMain", "") + E.save_close(page) + + +def dimensions(page): + E.goto(page, "Dimensions", "WebFormDimensions.aspx") + E.set_text(page, f"{DIM}TextBoxFloorAreaLowestFloor", "36.86") + E.set_text(page, f"{DIM}TextBoxRoomHeightLowestFloor", "2.30") + E.set_text(page, f"{DIM}TextBoxWallPerimeterLowestFloor", "13.4") + E.set_text(page, f"{DIM}TextBoxPartyWallLengthLowestFloor", "6.8") + E.set_text(page, f"{DIM}TextBoxFloorArea1stFloor", "36.86") + E.set_text(page, f"{DIM}TextBoxRoomHeight1stFloor", "2.30") + E.set_text(page, f"{DIM}TextBoxWallPerimeter1stFloor", "17.4") + E.set_text(page, f"{DIM}TextBoxPartyWallLength1stFloor", "6.8") + E.save_close(page) + + +def walls(page): + E.goto(page, "Walls", "WebFormWalls.aspx") + E.set_select(page, f"{WALL}DropDownListType", "CA Cavity") + page.wait_for_timeout(400) + print("insulation ->", _pick(page, f"{WALL}DropDownListInsulation", "as built")) # uninsulated + # Semi party wall: cavity. Match "masonry filled" β†’ CF (Uβ‰ˆ0); avoid loose "filled". + pw = _pick(page, f"{PWALL}DropDownListPartyWallType", "masonry filled") \ + or _pick(page, f"{PWALL}DropDownListPartyWallType", "determine") + print("party wall ->", pw) + E.save_close(page) + + +def roofs(page): + E.goto(page, "Roofs", "WebFormRoofs.aspx") + _pick(page, f"{ROOF}DropDownListType", "access to loft") + _pick(page, f"{ROOF}DropDownListInsulationAt", "joists") + E.set_select(page, f"{ROOF}DropDownListThickness", "200 mm") + E.save_close(page) + + +def floors(page): + E.goto(page, "Floors", "WebFormFloors.aspx") + E.set_select(page, f"{FLOOR}DropDownListLocation", "G Ground floor") + _pick(page, f"{FLOOR}DropDownListType", "suspended timber") + E.set_select(page, f"{FLOOR}DropDownListInsulation", "A As built") + E.save_close(page) + + +def openings(page): + E.goto(page, "Openings", "WebFormOpenings.aspx") + E.click_tab(page, "TabContainer_TabPanelWindowsPanel") + _add_window(page, 11.84, "North", _glazing(page)) + _delete_zero_rows(page) + E.click_tab(page, "TabContainer_TabPanelDoorsPanel") + E.set_text(page, f"{DP}TextBoxDoors", "1") + E.set_text(page, f"{DP}TextBoxDoorsInsulated", "0") + E.set_text(page, f"{DP}TextBoxDraughtProofedDoors", "0") + E.save_close(page) + + +def _glazing(page): + for needle in ("unknown install date", "before 2002", "pre 2002"): + for opt in _options(page, f"{WP}DropDownListExtGlazing"): + low = opt.lower() + if needle in low and "triple" not in low and "single" not in low and "known data" not in low: + return opt + return "Double post or during 2022" + + +def _add_window(page, area, orientation, glazing): + print("glazing ->", glazing) + E.set_select(page, f"{WP}DropDownListExtGlazing", glazing) + page.wait_for_timeout(400) + ft = page.locator(f"#{E.FP}{WP}DropDownListExtFrameType") + if ft.count(): + ft.select_option("PVC") + gg = page.locator(f"#{E.FP}{WP}DropDownListExtGlazingGap") + if gg.count(): + gg.select_option("12 mm") + wid = f"{E.FP}{WP}TextBoxExtWidth" + page.evaluate( + """(a)=>{const e=document.getElementById(a[0]);if(e){e.value=a[1]; + e.dispatchEvent(new Event('input',{bubbles:true})); + e.dispatchEvent(new Event('change',{bubbles:true})); + e.dispatchEvent(new Event('blur',{bubbles:true}));}}""", [wid, str(area)]) + page.locator(f"#{E.FP}{WP}TextBoxExtHeight").fill("1.00") + page.locator(f"#{E.FP}{WP}DropDownListExtOrientation").select_option(orientation) + page.locator(f"#{E.FP}{WP}DropDownListExtBuildingPartId").select_option("Main") + page.locator(f"#{E.FP}{WP}DropDownListExtLocation").select_option("External wall") + page.wait_for_timeout(300) + before = E.window_row_count(page) + page.evaluate("(id)=>{const e=document.getElementById(id); if(e)e.click();}", f"{E.FP}{WP}ButtonAddWindow") + for _ in range(25): + page.wait_for_timeout(200) + if E.window_row_count(page) > before: + break + + +def fix_window(page): + """Edit-in-place: overwrite the shared assessment's leftover window width to ours.""" + E.goto(page, "Openings", "WebFormOpenings.aspx") + E.click_tab(page, "TabContainer_TabPanelWindowsPanel") + page.wait_for_timeout(800) + wid = f"{E.FP}{WP}TextBoxExtWidth" + page.evaluate("""(a)=>{const e=document.getElementById(a[0]);if(e){e.value=a[1]; + e.dispatchEvent(new Event('input',{bubbles:true})); + e.dispatchEvent(new Event('change',{bubbles:true})); + e.dispatchEvent(new Event('blur',{bubbles:true}));}}""", [wid, "11.84"]) + print("window width now:", page.locator(f"#{wid}").input_value()) + E.save_close(page) + + +def _grid_rows(page): + return page.evaluate( + """()=>{const t=document.querySelector("[id*=GridViewExtendedWidows]"); + if(!t)return[];return Array.from(t.querySelectorAll('tr')).slice(1) + .map(r=>Array.from(r.querySelectorAll('td')).map(c=>c.innerText.trim()));}""") + + +def _delete_zero_rows(page): + g = 0 + while g < 6 and E.window_row_count(page) > 1: + g += 1 + rows = _grid_rows(page) + bad = next((i for i, c in enumerate(rows) if len(c) > 1 and c[1] in ("0.00", "0", "0.0")), None) + if bad is None: + break + _delete_row(page, bad) + page.wait_for_timeout(400) + + +def _delete_row(page, idx): + before = E.window_row_count(page) + btn = page.evaluate( + """(i)=>{const b=document.querySelectorAll("[id*='GridViewExtendedWidows_DeleteButton_']");return b[i]?b[i].id:null;}""", idx) + if not btn: + return + page.evaluate("(id)=>{const e=document.getElementById(id); if(e)e.click();}", btn) + page.wait_for_selector(f"#{E.FP}DeleteWindowDialog_LinkButtonYes", state="visible", timeout=5000) + page.evaluate("(id)=>{const e=document.getElementById(id); if(e)e.click();}", f"{E.FP}DeleteWindowDialog_LinkButtonYes") + for _ in range(20): + page.wait_for_timeout(200) + if E.window_row_count(page) < before: + break + + +def ventilation(page): + E.goto(page, "VentilationAndCooling", "WebFormVentilationAndCooling.aspx") + E.click_tab(page, "TabContainer_TabPanelVentilationPanel") + E.set_text(page, f"{VP}TextBoxIntermittentFans", "0") + cool = page.locator(f"#{E.FP}{VP}CheckBoxFixedSpaceCooling") + if cool.count() and cool.is_checked(): + E.commit(page, cool.uncheck) + E.click_tab(page, "TabContainer_TabPanelMechVent") + mv = page.locator(f"#{E.FP}{MV}CheckBoxMechanicalVentilation") + if mv.count() and mv.is_checked(): + E.commit(page, mv.uncheck) + E.click_tab(page, "TabContainer_TabPanelAirPressureTest") + E.set_select(page, f"{APT}DropDownListTestMethod", "Not available") + E.click_tab(page, "TabContainer_TabPanelLighting") + E.set_text(page, f"{LP}TextBoxLightsTotal", "10") + E.set_text(page, f"{LP}TextBoxLedLightsTotal", "10") # 100% LED + E.set_text(page, f"{LP}TextBoxCflLightsTotal", "0") + E.save_close(page) + + +def space_heating(page): + # Generic (no-PCDB) mains-gas combi. The shared assessment may carry a prior + # cert's storage SEB / PCDB boiler β€” clear it first if present, then select a + # generic gas combi via the boiler dialog. Two passes if a SAP code is bound. + E.goto(page, "SpaceHeating", "WebFormSpaceHeating.aspx") + page.wait_for_timeout(1000) + # Pass 1: clear a leftover SAP-table MainHeatingCode (e.g. SEB) so the boiler + # search/dialog is usable. + mhc = page.locator(f"#{E.MH1}TextBoxMainHeatingCode") + code = mhc.input_value() if mhc.count() else "" + if code and code not in ("0",): + print(f"clearing leftover MainHeatingCode {code}") + page.evaluate("""(id)=>{const e=document.getElementById(id);if(e){e.value=''; + e.dispatchEvent(new Event('change',{bubbles:true}));}}""", f"{E.MH1}TextBoxMainHeatingCode") + page.wait_for_timeout(400) + E.save_close(page) + return + # Generic (no-PCDB) mains-gas boiler via the cascade: Gas β†’ Mains gas β†’ + # Boilers β†’ Post 1998 β†’ Condensing (combi vs regular is set by the water- + # heating page: "from primary" + no cylinder = combi). ~89% SAP Table 4b. + E.set_heating_dialog(page, f"{MH1B}ButtonMainHeatingCode", + "^Gas", "Mains gas", "Boilers", "Post 1998", "Condensing", + "Combi condens") # L6 = BGW Post 98 Combi condens. (SAP 4b ~89%) + print("code:", page.locator(f"#{E.MH1}TextBoxMainHeatingCode").input_value()) + E.set_heating_dialog(page, f"{MH1B}ButtonMainHeatingControls", + "^Boilers", "^Standard", "CBE Programmer, room thermostat and TRVs") + print("control:", page.locator(f"#{E.MH1}TextBoxMainHeatingControls").input_value()) + E.save_close(page) + + +def secondary(page): + # Lodged secondary: SAP 691 = REA electric panel/convector/radiant room heater. + E.goto(page, "SpaceHeating", "WebFormSpaceHeating.aspx") + page.wait_for_timeout(600) + E.set_select(page, "DropDownListSecondaryHeatingPresent", "Yes") + page.wait_for_timeout(900) + E.set_heating_dialog(page, "ButtonSecondaryHeatingCode", + "Electric", "Electric", "Room Heater", "REA Panel") + tb = page.locator(f"#{E.FP}TextBoxSecondaryHeatingCode") + print("secondary code:", tb.input_value() if tb.count() else "?") + E.save_close(page) + + +def water_heating(page): + E.goto(page, "WaterHeating", "WebFormWaterHeating.aspx") + E.click_tab(page, "TabContainer_TabPanelWaterHeating") + page.wait_for_timeout(400) + E.clear_hot_water_cylinder(page) + E.set_heating_dialog(page, f"{WH}ButtonWaterHeatingCode", + "From Space Heating", "From the primary heating system") + print("water code:", page.locator(f"#{E.FP}{WH}TextBoxWaterHeatingCode").input_value()) + E.save_close(page) + + +def renewables(page): + # 2Γ— PV arrays @1.14 kW. Discover the Renewables/PV page structure. + for url in ("WebFormRenewables.aspx", "WebFormPhotovoltaics.aspx", "WebFormSolar.aspx"): + try: + E.goto(page, url.replace("WebForm", "").replace(".aspx", ""), url) + page.wait_for_timeout(800) + print("PV page:", url, "->", page.url) + inputs = page.evaluate( + """()=>Array.from(document.querySelectorAll("[id*=ContentPlaceHolder1] input,[id*=ContentPlaceHolder1] select")).map(e=>e.id.replace('ContentBody_ContentPlaceHolder1_','')).filter(i=>/pv|photov|solar|peak|orient|pitch|oversh|panel/i.test(i)).slice(0,30)""") + print("PV-ish fields:", inputs) + return + except Exception as e: + print(" ", url, "->", type(e).__name__) + + +_ORDER = ["property_description", "dimensions", "walls", "roofs", "floors", + "openings", "fix_window", "ventilation", "space_heating", "secondary", + "water_heating", "renewables"] + + +def main(): + if len(sys.argv) < 2 or sys.argv[1] not in _ORDER: + print("usage: build_22086693.py <" + "|".join(_ORDER) + ">") + return 2 + with E.session() as (ctx, page): + globals()[sys.argv[1]](page) + print("done:", sys.argv[1], "->", page.url) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/hyde/elmhurst_download.py b/scripts/hyde/elmhurst_download.py index 0c317033..6e9e4805 100644 --- a/scripts/hyde/elmhurst_download.py +++ b/scripts/hyde/elmhurst_download.py @@ -28,7 +28,7 @@ SESSION_DIR = HERE / ".elmhurst-session" SAMPLE_DIR = ( HERE.parent.parent / "backend/epc_api/json_samples/real_life_examples" - / "SAP-Schema-16.0/uprn_10070004512" + / "RdSAP-Schema-20.0.0/uprn_22086693" ) ASSESSMENT_GUID = "B44A0DB4-4C08-4241-B818-86F060172105" diff --git a/scripts/query_failed_modelling_e2e.py b/scripts/query_failed_modelling_e2e.py index a4cca69f..db684023 100644 --- a/scripts/query_failed_modelling_e2e.py +++ b/scripts/query_failed_modelling_e2e.py @@ -11,6 +11,7 @@ from __future__ import annotations import json import sys from pathlib import Path +from typing import Any, cast _REPO_ROOT = Path(__file__).resolve().parents[1] sys.path.insert(0, str(_REPO_ROOT)) @@ -30,6 +31,7 @@ with engine.connect() as conn: st.id AS subtask_id, st.task_id, st.inputs, + st.outputs, st.updated_at FROM sub_task st JOIN tasks t ON t.id = st.task_id @@ -45,19 +47,32 @@ with engine.connect() as conn: ) exit(0) + def _error_text(outputs_raw: object) -> str: + """Pull the persisted failure reason out of the subtask outputs JSON.""" + try: + outputs: Any = ( + json.loads(outputs_raw) + if isinstance(outputs_raw, str) + else (outputs_raw or {}) + ) + except Exception: + return str(outputs_raw or "") + if isinstance(outputs, dict): + return str(cast("dict[str, Any]", outputs).get("error", "")) + return "" + # Collect all property_ids across all rows all_property_ids: list[int] = [] - parsed: list[tuple[str, str, list[int], str, str]] = [] - for subtask_id, task_id, inputs_raw, updated_at in subtask_rows: + parsed: list[tuple[str, str, list[int], str, str, str]] = [] + for subtask_id, task_id, inputs_raw, outputs_raw, updated_at in subtask_rows: try: - inputs = ( + inputs: Any = ( json.loads(inputs_raw) if isinstance(inputs_raw, str) else (inputs_raw or {}) ) - property_ids: list[int] = [ - int(p) for p in (inputs.get("property_ids") or []) - ] + raw_ids = cast("list[Any]", cast("dict[str, Any]", inputs).get("property_ids") or []) + property_ids: list[int] = [int(p) for p in raw_ids] except Exception: property_ids = [] parsed.append( @@ -67,6 +82,7 @@ with engine.connect() as conn: property_ids, str(updated_at), inputs_raw or "", + _error_text(outputs_raw), ) ) all_property_ids.extend(property_ids) @@ -80,23 +96,29 @@ with engine.connect() as conn: ).fetchall() uprn_map = {int(r[0]): int(r[1]) for r in uprn_rows} +def _cell(value: str) -> str: + """Escape table-breaking characters so multi-line errors stay on one row.""" + return value.replace("|", "\\|").replace("\n", "
") + + lines: list[str] = [ "# Failed modelling_e2e Subtasks\n", - f"| Subtask ID | Task ID | Updated At | Property ID | UPRN | Inputs |", - f"|-----------|---------|------------|-------------|------|--------|", + f"| Subtask ID | Task ID | Updated At | Property ID | UPRN | Error | Inputs |", + f"|-----------|---------|------------|-------------|------|-------|--------|", ] -for subtask_id, task_id, property_ids, updated_at, inputs_raw in parsed: - inputs_cell = (inputs_raw or "").replace("|", "\\|") +for subtask_id, task_id, property_ids, updated_at, inputs_raw, error in parsed: + inputs_cell = _cell(inputs_raw or "") + error_cell = _cell(error or "") if property_ids: for pid in property_ids: uprn = uprn_map.get(pid, "unknown") lines.append( - f"| {subtask_id} | {task_id} | {updated_at} | {pid} | {uprn} | {inputs_cell} |" + f"| {subtask_id} | {task_id} | {updated_at} | {pid} | {uprn} | {error_cell} | {inputs_cell} |" ) else: lines.append( - f"| {subtask_id} | {task_id} | {updated_at} | β€” | β€” | {inputs_cell} |" + f"| {subtask_id} | {task_id} | {updated_at} | β€” | β€” | {error_cell} | {inputs_cell} |" ) _OUTPUT.write_text("\n".join(lines) + "\n", encoding="utf-8") diff --git a/scripts/trigger_modelling_e2e_sqs.py b/scripts/trigger_modelling_e2e_sqs.py index f0eaed38..76105422 100644 --- a/scripts/trigger_modelling_e2e_sqs.py +++ b/scripts/trigger_modelling_e2e_sqs.py @@ -20,9 +20,10 @@ PORTFOLIO_ID: int = 796 SCENARIO_ID: int = 1268 SQS_QUEUE_NAME: str = "modelling_e2e-queue-dev" -# Number of postcodes to process this run (postcodes where all properties are -# already completed are skipped and do not count toward this limit). -POSTCODES_LIMIT: int = 1000 +# Max number of properties to process this run (cost cap). Postcodes are added +# whole, smallest-group-first, until adding the next would exceed this β€” so the +# actual count lands at or just under the limit. +PROPERTIES_LIMIT: int = 5000 # True β†’ Lambda runs the full pipeline but skips all DB writes (safe for testing). DRY_RUN: bool = False @@ -94,21 +95,31 @@ def _completed_property_ids() -> set[int]: def main() -> None: postcode_map = _load_postcode_map() - completed = _completed_property_ids() - logger.info(f"{len(completed)} property IDs already completed β€” skipping") + # Pending filter disabled β€” re-run every property regardless of whether it + # already has a completed modelling_e2e sub_task for this scenario. batches: list[tuple[str, list[int]]] = [] for postcode, ids in postcode_map.items(): - pending = [pid for pid in ids if pid not in completed] - if pending: - batches.append((postcode, pending)) + if ids: + batches.append((postcode, ids)) - to_process = batches[:POSTCODES_LIMIT] + to_process: list[tuple[str, list[int]]] = [] + property_count = 0 + for postcode, ids in batches: + if property_count + len(ids) > PROPERTIES_LIMIT: + continue + to_process.append((postcode, ids)) + property_count += len(ids) if not to_process: logger.info("Nothing left to process.") return + logger.info( + f"selected {len(to_process)} postcodes / {property_count} properties " + f"(limit {PROPERTIES_LIMIT})" + ) + sqs: Any = cast( Any, boto3.client("sqs", region_name="eu-west-2") ) # pyright: ignore[reportUnknownMemberType] diff --git a/tests/applications/modelling_e2e/test_handler.py b/tests/applications/modelling_e2e/test_handler.py index 6478572a..baa20103 100644 --- a/tests/applications/modelling_e2e/test_handler.py +++ b/tests/applications/modelling_e2e/test_handler.py @@ -214,16 +214,19 @@ def test_lodged_epc_path_saves_epc_plan_and_marks_modelled( _baseline_orchestrator.return_value.run.assert_called_once_with([PROPERTY_ID]) -def test_skipped_cohort_certs_are_surfaced_in_the_outputs() -> None: +def test_skipped_cohort_certs_fail_the_subtask_but_the_plan_is_still_saved() -> None: """Cohort certs the mapper can't consume are skipped (so prediction is not - aborted) and surfaced β€” with cert numbers β€” in the subtask outputs, so the - mapper gaps can be reported and closed.""" + aborted), then surfaced as a failure β€” the subtask is marked failed (the + cert numbers land in outputs.error via the raised RuntimeError) so the + mapper gaps get debugged. The batch still ran to completion first, so the + property's plan was committed before the handler raised.""" from repositories.comparable_properties.epc_comparable_properties_repository import ( SkippedCohortCert, ) mock_engine = _engine_mock([PROPERTY_ID], [UPRN], [POSTCODE]) mock_plan = _plan_mock() + mock_uow = MagicMock() skipped = [ SkippedCohortCert( certificate_number="8257-7539-1649-0633-4992", @@ -287,24 +290,21 @@ def test_skipped_cohort_certs_are_surfaced_in_the_outputs() -> None: MockUoW = stack.enter_context( patch("applications.modelling_e2e.handler.PostgresUnitOfWork") ) - MockUoW.return_value.__enter__.return_value = MagicMock() + MockUoW.return_value.__enter__.return_value = mock_uow MockUoW.return_value.__exit__.return_value = False - # Act - result = _call_handler(_BODY) + # Act β€” the skipped cert fails the subtask, but only after the batch ran. + with pytest.raises(RuntimeError) as excinfo: + _call_handler(_BODY) - # Assert β€” the handler's return (β†’ subtask outputs.result) carries the cert - # numbers + errors of every skipped cohort cert. - assert result == { - "skipped_unmappable_cohort_certs": [ - { - "certificate_number": "8257-7539-1649-0633-4992", - "error": ( - "ValueError: RdSapSchema17_1: missing required field 'window'" - ), - } - ] - } + # Assert β€” the cert number + error reach outputs.error (the raised message), + # and the property's plan was still committed before the handler raised. + message = str(excinfo.value) + assert "skipped_unmappable_cohort_certs" in message + assert "8257-7539-1649-0633-4992" in message + assert "RdSapSchema17_1: missing required field 'window'" in message + mock_uow.plan.save.assert_called_once() + mock_uow.commit.assert_called_once() # --------------------------------------------------------------------------- diff --git a/tests/domain/modelling/test_elmhurst_cascade_pins.py b/tests/domain/modelling/test_elmhurst_cascade_pins.py index 19186eae..c99f7b6f 100644 --- a/tests/domain/modelling/test_elmhurst_cascade_pins.py +++ b/tests/domain/modelling/test_elmhurst_cascade_pins.py @@ -492,10 +492,11 @@ def test_suspended_floor_overlay_reproduces_the_relodged_after() -> None: def test_double_glazing_overlay_reproduces_the_relodged_after_windows() -> None: - # The full-SAP pin below is xfail (draught-proofing coupling), but the - # overlay's actual job β€” turning every single-glazed window into the - # relodged spec β€” is deterministic and must hold exactly: it proves the - # generator detects BOTH single-glazing codes (1 and 15) on the real cert. + # The full-SAP pin below now passes (the draught-proofing + frame-factor + # coupling is modelled), but this narrower pin isolates the overlay's actual + # job β€” turning every single-glazed window into the relodged spec β€” which is + # deterministic and must hold exactly: it proves the generator detects BOTH + # single-glazing codes (1 and 15) on the real cert. # Arrange before: EpcPropertyData = parse_recommendation_summary( "double_glazing_001431_before.pdf" @@ -528,24 +529,13 @@ def test_double_glazing_overlay_reproduces_the_relodged_after_windows() -> None: assert _window_spec(applied) == _window_spec(after) -_GLAZING_DRAUGHT_COUPLING_REASON: Final[str] = ( - "Blocked on the glazing measure's draught-proofing coupling. The window " - "U/g overlay reproduces the after's 14 windows EXACTLY (all four single-" - "glazed panes β€” codes 1 and 15 β€” become the relodged double/secondary " - "spec). The residual ~0.7 SAP is a secondary effect the overlay does not " - "model: replacing the single-glazed (lodged draught_proofed=No) windows " - "with sealed units re-lodges percent_draughtproofed 84->100 (~0.3 SAP) and " - "lowers fabric heat loss by ~+150 kWh space heating (~0.4 SAP) not yet " - "isolated. Flips green once the glazing overlay propagates draught-proofing " - "(and the residual fabric coupling is modelled)." -) - - -@pytest.mark.xfail(strict=True, reason=_GLAZING_DRAUGHT_COUPLING_REASON) def test_double_glazing_overlay_reproduces_the_relodged_after() -> None: # Arrange β€” cert 001431 lodges four single-glazed windows (codes 1 and 15, # "single glazing, known data"); the after re-lodges every one as double - # (gt=5, U=1.40, g=0.72). + # (gt=5, U=1.40, g=0.72). The overlay now also models the two secondary + # effects of fitting sealed units: RdSAP 10 Β§8.1 draught-proofing + # 84 β†’ 100 (`percent_draughtproofed`) and the re-lodged FF=0.70 on the + # panes that lodged FF 1.00 / 0.50 β€” so the full-SAP pin closes. before: EpcPropertyData = parse_recommendation_summary( "double_glazing_001431_before.pdf" ) @@ -561,7 +551,6 @@ def test_double_glazing_overlay_reproduces_the_relodged_after() -> None: ) -@pytest.mark.xfail(strict=True, reason=_GLAZING_DRAUGHT_COUPLING_REASON) def test_secondary_glazing_overlay_reproduces_the_relodged_after() -> None: # Arrange β€” a planning protection forces secondary glazing; the after # re-lodges every single-glazed window as secondary (gt=11, U=2.90, g=0.85). diff --git a/tests/domain/modelling/test_glazing_recommendation.py b/tests/domain/modelling/test_glazing_recommendation.py index 993052f0..52063b5e 100644 --- a/tests/domain/modelling/test_glazing_recommendation.py +++ b/tests/domain/modelling/test_glazing_recommendation.py @@ -67,8 +67,12 @@ def test_single_glazed_dwelling_yields_a_double_glazing_recommendation() -> None option = recommendation.options[0] assert option.measure_type == "double_glazing" assert dict(option.overlay.windows) == { - 0: WindowOverlay(glazing_type=5, u_value=1.40, solar_transmittance=0.72), - 2: WindowOverlay(glazing_type=5, u_value=1.40, solar_transmittance=0.72), + 0: WindowOverlay( + glazing_type=5, u_value=1.40, solar_transmittance=0.72, frame_factor=0.70 + ), + 2: WindowOverlay( + glazing_type=5, u_value=1.40, solar_transmittance=0.72, frame_factor=0.70 + ), } @@ -121,6 +125,10 @@ def test_planning_protection_picks_secondary_glazing_over_double() -> None: option = recommendation.options[0] assert option.measure_type == "secondary_glazing" assert dict(option.overlay.windows) == { - 0: WindowOverlay(glazing_type=11, u_value=2.90, solar_transmittance=0.85), - 2: WindowOverlay(glazing_type=11, u_value=2.90, solar_transmittance=0.85), + 0: WindowOverlay( + glazing_type=11, u_value=2.90, solar_transmittance=0.85, frame_factor=0.70 + ), + 2: WindowOverlay( + glazing_type=11, u_value=2.90, solar_transmittance=0.85, frame_factor=0.70 + ), } diff --git a/tests/domain/modelling/test_overlay_applicator.py b/tests/domain/modelling/test_overlay_applicator.py index 64a237d7..8f6c960f 100644 --- a/tests/domain/modelling/test_overlay_applicator.py +++ b/tests/domain/modelling/test_overlay_applicator.py @@ -55,6 +55,55 @@ def test_apply_writes_targeted_building_part_and_leaves_others_untouched() -> No ) +def test_override_for_an_absent_semantic_part_lands_on_the_part_at_that_position() -> ( + None +): + # `building_part` is a POSITIONAL index (0=main, 1=extension 1…, ADR-0004). The + # gov-API EPC can label its parts differently (e.g. a 2nd part lodged as `other` + # rather than `extension_1`). An `extension_1` override must still land on the + # part at position 1 β€” the landlord's correction is applied, not dropped. + # Arrange β€” build_epc() is [MAIN, EXTENSION_1]; relabel the 2nd part to OTHER so + # the EXTENSION_1 identifier is absent but position 1 still exists. + baseline: EpcPropertyData = build_epc() + baseline.sap_building_parts[1].identifier = BuildingPartIdentifier.OTHER + simulation = EpcSimulation( + building_parts={ + BuildingPartIdentifier.EXTENSION_1: BuildingPartOverlay( + wall_insulation_type=3 + ) + } + ) + + # Act + result: EpcPropertyData = apply_simulations(baseline, [simulation]) + + # Assert β€” the override folded onto the part at position 1 (the OTHER part). + assert _part(result, BuildingPartIdentifier.OTHER).wall_insulation_type == 3 + + +def test_override_with_no_part_at_that_position_is_skipped() -> None: + # When there is genuinely no part at the override's position (the EPC models + # fewer parts than the index), the override is skipped rather than crashing β€” + # we cannot model an extension we have no geometry for. + # Arrange β€” build_epc() has 2 parts (positions 0, 1); position 2 is absent. + baseline: EpcPropertyData = build_epc() + simulation = EpcSimulation( + building_parts={ + BuildingPartIdentifier.MAIN: BuildingPartOverlay(wall_insulation_type=1), + BuildingPartIdentifier.EXTENSION_2: BuildingPartOverlay( + wall_insulation_type=1 + ), + } + ) + + # Act + result: EpcPropertyData = apply_simulations(baseline, [simulation]) + + # Assert β€” the present part got its overlay; nothing was added for position 2. + assert _part(result, BuildingPartIdentifier.MAIN).wall_insulation_type == 1 + assert len(result.sap_building_parts) == len(baseline.sap_building_parts) + + def test_flat_roof_construction_type_folds_onto_the_part() -> None: # ADR-0033: a flat-roof landlord override sets `roof_construction_type` so the # calculator's flat path (`"flat" in roof_construction_type`) fires the diff --git a/tests/domain/sap10_calculator/test_real_cert_sap_accuracy.py b/tests/domain/sap10_calculator/test_real_cert_sap_accuracy.py index 56989c71..5575384f 100644 --- a/tests/domain/sap10_calculator/test_real_cert_sap_accuracy.py +++ b/tests/domain/sap10_calculator/test_real_cert_sap_accuracy.py @@ -603,6 +603,52 @@ _EXPECTATIONS: Final[tuple[RealCertExpectation, ...]] = ( "engine prices 100% at the low rate; see elmhurst_worksheet.pdf (243-246)" ), ), + # UPRN 10070004512 β†’ cert 8742-6624-9300-2780-4926. SAP-Schema-16.0, GROUND- + # FLOOR FLAT, band B, cavity FILLED, ELECTRIC STORAGE HEATERS (SAP 402 SEB, + # manual charge CSA/2401) + electric immersion off-peak (Economy-7 Dual meter) + # with a small cylinder (size 1), roof = another dwelling above, floor to + # EXTERNAL AIR, double glazed, TFA 33.24. Engine 66 = lodged 66 EXACTLY. + # This is the modelling_e2e built_form fix cert: 16.0 omits `built_form`, which + # RdSapSchema17_1 requires; the mapper derives it from dwelling_type (flat β†’ + # modal 4). built_form is ML-only (the SAP calculator never reads it) so the fix + # is SAP-NEUTRAL β€” the engine reproduces the lodged score regardless. Built in + # Elmhurst RdSAP10 (evidence saved: elmhurst_summary.pdf / elmhurst_worksheet.pdf): + # worksheet SAP 54, engine on Elmhurst's own parsed inputs 53 β‰ˆ 54 β†’ calculator + # faithful. The engine 66 vs Elmhurst 54 (+12) is an input/build gap dominated by + # HOT WATER (engine 1272 vs Elmhurst 1948 kWh): the cert lodges a size-1 (small) + # cylinder but Elmhurst's RdSAP entry has no "Small" option (Normal/110 L is the + # smallest), forcing a larger cylinder + more storage loss; plus the reduced-field + # 16.0 floor/party-wall defaults. PINNED to the observed engine 66 (= lodged 66). + RealCertExpectation( + schema="SAP-Schema-16.0", + sample="uprn_10070004512", + cert_num="8742-6624-9300-2780-4926", + sap_score=66, + ), + # UPRN 22086693 β†’ cert 6102-6227-8000-0083-2292. RdSAP-Schema-20.0.0, SEMI- + # DETACHED HOUSE 2-storey, band C, cavity UNINSULATED, mains-gas COMBI (no PCDB + # β†’ generic SAP Table-4b BGW post-98 condensing combi), control 2106 (CBE), + # pitched 200 mm loft, suspended uninsulated floor, party wall 6.8 m, double + # glazed, electric secondary room heater (SAP 691 REA), 2Γ— PV ARRAYS @1.14 kW, + # 100% LED, TFA ~74. Engine 66 / lodged 72. + # This is the modelling_e2e photovoltaic_supply-AS-LIST fix cert: 20.0.0 typed + # `photovoltaic_supply` as the wrapper only, so a measured-array LIST crashed + # `from_rdsap_schema_20_0_0` ("'list' object has no attribute none_or_no_details") + # and sank the whole prediction cohort. The fix routes it through the dict-tolerant + # `_map_schema_21_pv`, capturing the arrays. PV is correctly credited: engine + # WITHOUT pv = 61, WITH pv = 66 (+5). Built in Elmhurst RdSAP10 (evidence saved: + # elmhurst_summary.pdf / elmhurst_worksheet.pdf) β€” fabric+heating only (the PV + # is a separate "New Technologies" Panel-details grid, deferred): worksheet 55, + # engine on Elmhurst's own parsed inputs 55 = 55 EXACTLY β†’ calculator faithful. + # The engine-without-pv 61 vs Elmhurst 55 (+6) is the documented engine-vs- + # Elmhurst-RdSAP-default residual on a band-C cavity-uninsulated suspended-floor + # semi; lodged 72 vs engine 66 (βˆ’6) is that plus PV-credit method. PINNED engine 66. + RealCertExpectation( + schema="RdSAP-Schema-20.0.0", + sample="uprn_22086693", + cert_num="6102-6227-8000-0083-2292", + sap_score=66, + ), )