Merge pull request #67 from Hestia-Homes/heat-dev-model

Heat dev model

README.md

@@ -31,7 +31,7 @@ In order for this to be set up, some key environment variables needs to be inser
 secrets. Each different model and protected branch has its own set of secrets which allows for flexibility
 between different pipelines.
 
-For example, for the branch sap_change-dev, the prefix=SAP_CHANGE_DEV, and the following secrets are:
+For example, for the branch sap-dev, the prefix=SAP_DEV, and the following secrets are:
 
 - {prefix}_ECR_URI, which is the URI of the ECR repository to push to. For example, for the
    sap change model this is the lambda-sap-prediction-dev repository.
@@ -58,7 +58,7 @@ First, navigate to the root directory of the repository. Open a terminal and exe
 2. command to build the Docker image:
 
 ```bash
-docker build -t sap_change -f deployment/Dockerfile.prediction.lambda .
+docker build -t sap -f deployment/Dockerfile.prediction.lambda .
 ```
 
 This will build a Docker image tagged as sap_change using the Dockerfile.prediction.lambda located
@@ -68,7 +68,7 @@ in the deployment directory.
 Once the image is built, you can run it using the following command:
 
 ```bash
-docker run -p 9000:8080 -v ~/.aws/credentials:/root/.aws/credentials:ro -e RUNTIME_ENVIRONMENT=dev sap_change
+docker run -p 9000:8080 -v ~/.aws/credentials:/root/.aws/credentials:ro -e RUNTIME_ENVIRONMENT=dev -e PREDICTIONS_BUCKET=retrofit-sap-predictions-dev sap
 ```
 This command does the following:
 
@@ -79,6 +79,7 @@ Sets the RUNTIME_ENVIRONMENT variable to dev.
 To test the Lambda function, use the following curl command:
 
 ```json
-curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"body": "{\"file_location\": \"s3://retrofit-data-dev/model_build_data/change_data/rdsap_full/test_data_with_id.parquet\", \"property_id\": 1, \"portfolio_id\": 4, \"created_at\": \"now\"}"'
+curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"body": "{\"file_location\": \"s3://retrofit-data-dev/sap_change_model/one_sample_test_dataset.parquet\", \"property_id\": 1, \"portfolio_id\": 4, \"created_at\": \"now\"}"}'
 ```
+
 This will send a POST request to the running Lambda function and pass in the required data as JSON.

modules/ml-pipeline/src/pipeline/4_generate_metrics.py

@@ -4,9 +4,7 @@ After the model is built, we can evaluate its performance
 """
 
 import os
-import yaml
 import pandas as pd
-from pathlib import Path
 from core.interface.InterfaceModels import MLModel
 from core.interface.InterfaceMetrics import MLMetrics
 from core.interface.InterfaceDataClient import DataClient

modules/ml-pipeline/src/pipeline/configs/build_model.yaml

@@ -13,6 +13,6 @@ default:
       output_filepath: ./data/model/allmodels/
       problem_type: regression
       eval_metric: mean_squared_error #mean_absolute_error
-      time_limit: 1000
+      time_limit: 4000
       presets: medium_quality
       excluded_model_types: ['KNN', 'RF']

modules/ml-pipeline/src/pipeline/dvc.lock

@@ -29,8 +29,8 @@ stages:
     outs:
     - path: data/prepared_data/
       hash: md5
-      md5: 71e63a792f7723e2aea0709efde1a92b.dir
+      md5: e0be70d5025e40dd0d655d9949f72130.dir
-      size: 31751660
+      size: 31800776
       nfiles: 2
   build_model:
     cmd: python 2_build_model.py
@@ -41,8 +41,8 @@ stages:
       size: 5359
     - path: data/prepared_data
       hash: md5
-      md5: 71e63a792f7723e2aea0709efde1a92b.dir
+      md5: e0be70d5025e40dd0d655d9949f72130.dir
-      size: 31751660
+      size: 31800776
       nfiles: 2
     params:
       configs/build_model.yaml:
@@ -58,7 +58,7 @@ stages:
               output_filepath: ./data/model/allmodels/
               problem_type: regression
               eval_metric: mean_squared_error
-              time_limit: 1000
+              time_limit: 4000
               presets: medium_quality
               excluded_model_types:
               - KNN
@@ -66,13 +66,13 @@ stages:
     outs:
     - path: data/model/
       hash: md5
-      md5: 0ffc51be7c8381c9e4106309e3e05ca3.dir
+      md5: 14ca33cde5e86770135f768abaf84978.dir
-      size: 345904743
+      size: 422447808
       nfiles: 27
     - path: metrics/fit_metrics.json
       hash: md5
-      md5: 3d4ff3a3ca3c327e2c1e9aa1338c18ce
+      md5: 41bfb8d2da8f06d1864d73ce125cc6aa
-      size: 220
+      size: 221
   generate_predictions:
     cmd: python 3_generate_predictions.py
     deps:
@@ -82,13 +82,13 @@ stages:
       size: 3028
     - path: data/model
       hash: md5
-      md5: 0ffc51be7c8381c9e4106309e3e05ca3.dir
+      md5: 14ca33cde5e86770135f768abaf84978.dir
-      size: 345904743
+      size: 422447808
       nfiles: 27
     - path: data/prepared_data
       hash: md5
-      md5: 71e63a792f7723e2aea0709efde1a92b.dir
+      md5: e0be70d5025e40dd0d655d9949f72130.dir
-      size: 31751660
+      size: 31800776
       nfiles: 2
     params:
       configs/settings.yaml:
@@ -100,8 +100,8 @@ stages:
     outs:
     - path: data/predictions/
       hash: md5
-      md5: 00ff804016290d56e1490e59c098b060.dir
+      md5: 40d0c7a7fd4a15add0615e322cf341a0.dir
-      size: 351811
+      size: 352151
       nfiles: 1
   generate_metrics:
     cmd: python 4_generate_metrics.py
@@ -112,13 +112,13 @@ stages:
       size: 4487
     - path: data/predictions
       hash: md5
-      md5: 00ff804016290d56e1490e59c098b060.dir
+      md5: 40d0c7a7fd4a15add0615e322cf341a0.dir
-      size: 351811
+      size: 352151
       nfiles: 1
     - path: data/prepared_data
       hash: md5
-      md5: 71e63a792f7723e2aea0709efde1a92b.dir
+      md5: e0be70d5025e40dd0d655d9949f72130.dir
-      size: 31751660
+      size: 31800776
       nfiles: 2
     params:
       configs/settings.yaml:
@@ -128,8 +128,8 @@ stages:
     outs:
     - path: metrics/metrics.json
       hash: md5
-      md5: 63ef63e4fabe929b914a0059ceeddabc
+      md5: 4e023650240e78d6ad761f1db7aac922
-      size: 221
+      size: 220
   startup_cleanup:
     cmd: python 0_startup_cleanup.py
     deps: