Model/backend/README.md

# Backend

This is the api service that will supply the frontend with the insights that are driven by the machine
learning and data modelling services.

# Usage

## Prerequisites

Python 3.8+
Poetry for managing project dependencies and virtual environment.

## Installation and setup

1. Clone this directory and navigate into the project directory.

```commandline
git clone https://github.com/Hestia-Homes/Model.git
cd backend
```

2. For environment management, I'm using conda with pycharm which is a convenient setup for development
   on a mac M1 however using tools such as poetry or pipenv is also fine.

For example, to install conda and create a virtual environment for this project, run the following commands:

```commandline
conda create -n backend python=3.10
conda activate backend
```

then enter the virtual environment and install the dependencies using conda.

```commandline
conda install --file requirements/base.txt
```

3. Duplicate .env.example and rename it to .env

```commandline
cp .env.example .env
```

4. Open .env and fill in the required environment variables.

## Running the Application

from `model/backend/` you can run with the following command:

```commandline
uvicorn app.main:app --reload
```

Or run `sh run_local.sh`, which runs that same uvicorn command.

You application will be available at the designated url

## API Documentation

FastAPI automatically generates interactive API documentation for your application. To access the docs, start your
server and visit <yourlocalurl>/docs in your browser. Alternatively, you can go to
<yourlocalurl>/redoc to view the documentation in the ReDoc format.

## Building the lambda's backend docker image locally

To build the backend docker image locally, run the following command from the root of the project directory:

```commandline
docker build -t fastapi-lambda-image:latest -f backend/docker/lambda.Dockerfile .
```

To check the size of the resulting image, run the following command:

```bash
docker images | grep fastapi-lambda-image
```

To run a shell inside the Docker container to inspect its contents, run:

```commandline
docker run -it fastapi-lambda-image:latest /bin/bash
```

Running in lambda results in running in a slightly different format compared to running the fastapi
application locally. If you want to run the fastapi application locally, in docker, we have a docker
file which builds the same environment as in lambda but runs the fast api application with uvicorn.

Run

```commandline
docker build -t fastapi-local-image:latest -f backend/docker/Dockerfile .
```

This will be the image. To run it, simply run

```commandline
docker run -p 8000:8000 -v ~/.aws:/root/.aws fastapi-local-image:latest
```

This assumes you have a ~/.aws folder with your aws credentials in it. If you don't have this, you can
run the following command with your aws access token exported into your environment.

```commandline
docker run -p 8000:8000 -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION fastapi-local-image:latest
```

## Emulating the lambda locally

I have set up a script called `run_local_lambda.sh` which will allow you to emulate the lambda locally.
You need to have a .env file with the necessary environment variables at backend/env and also
and aws credentials file at ~/.aws/credentials, locally.

To run this, firstly run:

```bash
chmod +x run_lambda_local.sh
```

Now you can run the script with

```
./run_lambda_local.sh
```

In order to make a request to it, there is a specific format the request must be in, to
emuate lambda. If using postman, the url you want is `http://localhost:8000/2015-03-31/functions/function/invocations`
and you need to pass a body like this:

```json
{
  "httpMethod": "POST",
  "body": "{\"portfolio_id\": 4, \"housing_type\": \"Private\", \"goal\": \"Increase EPC\", \"goal_value\": \"C\", \"trigger_file_path\": \"2/4/portfolio_plan_properties-20230724T093542483Z.csv\"}",
  "path": "/v1/plan/trigger",
  "resource": "/",
  "headers": {
    "Accept": "*/*",
    "Content-Type": "application/json",
    "Authorization": "Bearer YOUR_TOKEN_HERE",
    "x-api-key": "YOUR_API_KEY_HERE"
  },
  "requestContext": {},
  "multiValueQueryStringParameters": null
}

```

Logs for the container can quickly be seen via Docker desktop

## Testing

To run tests, run the following command from the root of the project directory:

```commandline
pytest
```

## Local Development

During local development, you may need to generate and use a dummy JWT to
test protected endpoints of the application.

# Generating a Dummy JWT

FastAPI provides a convenient way to generate a dummy JWT for testing.
To generate a dummy JWT, follow the steps below:

Make sure your application is running in a local environment.
The dummy token endpoint is only available in a local environment.

While your application is running, visit the /dummy-token endpoint using a tool
like curl or any HTTP client like Postman.

For instance, if your server is running locally on port 8000, you can use curl
to get a dummy token:

```commandline
curl http://localhost:8000/local/dummy-token
```

You will receive a response containing the dummy JWT

```json
{
  "dummy_token": "<Your Dummy Token>"
}
```

### Using the Dummy JWT

Once you've obtained a dummy JWT, you can use it to make requests to
protected endpoints in your application:

1. When making a request, include an Authorization header with the value Bearer
   <Your Dummy Token>. Replace <Your Dummy Token> with the token you
   received from the /dummy-token endpoint.

2. Now you can make requests to the protected endpoints of the application.

Remember, the dummy JWT is meant for testing purposes only and should not be
used in production environments. The /dummy-token endpoint is not available
in non-local environments.

# Custom Domain Setup for AWS API Gateway

Before you deploy your Serverless application for the first time, you need to set up a custom domain for AWS API
Gateway. This is done using the sls create_domain command, which creates a custom domain in API Gateway that your
services can use.

To set up a custom domain, use the following command:

```bash
sls create_domain --stage dev --aws-profile DevAdmin --verbose
```

Replace dev with the name of the stage you're deploying to. This command only needs to be run once per custom domain,
and not every time you deploy your application. After running this command,
you can associate your AWS Lambda functions with this domain using the customDomain
configuration in your serverless.yml file.

This command requires the Serverless Domain Manager plugin,
so make sure you have it installed and properly configured in your serverless.yml file.

Please note that the process of creating and associating a custom domain can take up to 40 minutes.
Once the custom domain is created, it's immediately available for use in your Serverless applications.

Remember to replace DevAdmin with the profile that has appropriate permissions in your AWS account.
The `--verbose` flag is optional and is used to print detailed logs to the console.

# Creating a CNAME Record in Google Domains

After deploying the AWS Lambda function for the first time, you need to set up a
CNAME record in Google Domains to route traffic from your custom domain to
the CloudFront distribution created by API Gateway. This will re-route traffic from
your custom domain to the CloudFront distribution created by API Gateway, and
therefore to your lambda.
See [here](https://docs.aws.amazon.com/amplify/latest/userguide/to-add-a-custom-domain-managed-by-google-domains.html)
for AWS' documentation on this.

You can find the CloudFront
domain by going to the API Gateway console and clicking on Custom Domain Names.

Here are the steps to create a CNAME record:

1. Log in to Google Domains.
2. Select the name of your domain.
3. Open the menu, if it's not already open.
4. Click "DNS."
5. Scroll down to the "Custom resource records" section.
6. In the "Name" field, enter your subdomain (e.g., api if your API is available at api.example.com).
7. In the "Type" dropdown menu, select "CNAME."
8. In the "TTL" field, enter 1H to set it to 1 hour (or another suitable value).
9. In the "Data" field, enter the CloudFront domain that was created by API Gateway (you can find this in the API
   Gateway console, under Custom Domain Names).
10. Click "Add."

This will direct any traffic from your custom domain to your AWS CloudFront
distribution. Please note that DNS changes might take some time (up to 24-48
hours in some cases) to propagate across the internet.

Also, please make sure that your CloudFront distribution is configured to
accept your custom domain as a valid domain name. In AWS API Gateway,
under Custom Domain Names, make sure that your custom domain is listed and
mapped to the appropriate API stage.

Remember to replace api and the CloudFront domain with your actual subdomain
and CloudFront domain.

Certainly! Here's a detailed documentation for your README:

* * *

Deployment Troubleshooting for `fastapi-lambda`
-----------------------------------------------

### Context:

When deploying the `fastapi-lambda` using Serverless Framework, you may encounter issues related to domain management,
especially if you're using a custom domain for your API. This documentation provides troubleshooting steps and details
on how to resolve potential conflicts.

### Potential Issues & Solutions:

#### 1\. Conflict with Existing CloudFront Distribution:

**Error Message**:

csharpCopy code

`One or more aliases specified for the distribution includes an incorrectly configured DNS record that points to another CloudFront distribution.`

**Cause**: This can occur if there's an existing CNAME record in your DNS provider pointing to a CloudFront
distribution.

**Solution**:

* Check your DNS provider (e.g., Google Domains) and verify the CNAME record for `api.dev.hestia.homes`.
* Temporarily remove or update the conflicting CNAME record.
* Run the `sls create_domain` command again.
* Update the DNS settings in your DNS provider based on the new configuration provided by
  the `serverless-domain-manager` plugin.

#### 2\. Conflict with Route53:

**Error Message**:

csharpCopy code

`Deleting RestApi failed. Please remove all base path mappings related to the RestApi in your domains.`

**Cause**: This can occur if there are residual AWS configurations, especially in Route53, from previous deployments.

**Solution**:

* Navigate to the AWS Route53 Console.
* Identify and delete any residual Hosted Zones or Record Sets related to `api.dev.hestia.homes`.
* Ensure that you have backed up any necessary configurations before deleting.

#### 3\. Other AWS Resources Conflicts:

You might encounter issues where AWS resources, such as S3 buckets or CloudFront distributions, are not properly deleted
or are conflicting with new deployments.

**Solution**:

* Navigate to the respective AWS service dashboard.
* Manually identify and rectify any conflicting resources. This might involve emptying S3 buckets or deleting CloudFront
  distributions.
* Ensure backups and proper precautions before deleting any resources.

### Additional Notes:

* **Backup Configurations**: Always backup your configurations before making changes. This ensures that you can revert
  to a previous state if needed.
* **DNS Propagation**: Remember that DNS changes can take some time to propagate globally. After making DNS changes, you
  might not see immediate effects.
* **CloudFront Distributions**: If you can't find a CloudFront distribution in the AWS CloudFront console, it's possible
  that it was automatically created by another AWS service like API Gateway. It might need to be managed or deleted from
  that service's dashboard.

### After succesfully running creating the custom domain

After successfully creating the custom domain with the `serverless-domain-manager` plugin, you should add back the CNAME
record into Google Domains (or whatever platform is being used to manage domains now)
to ensure that the custom domain properly points to the CloudFront distribution managed by
AWS.

Here's what you should do:

1. **Log in to Google Domains**:

    * Go to [Google Domains](https://domains.google.com/).
    * Navigate to the management page for `hestia.homes`.
2. **Add/Update the CNAME Record**:

    * Find the section for custom resource records.
    * Add (or update if it already exists) a CNAME record for `api.dev`.
    * Point it to the CloudFront distribution domain name (e.g., `d2d269kjy1nyhz.cloudfront.net.`). Ensure you include
      the trailing dot at the end. This can be found in API gateway
3. **Check DNS Propagation**:

    * Keep in mind that DNS changes might take some time to propagate. You can use online tools
      like [DNS Checker](https://www.dnschecker.org/) to verify the propagation status worldwide.
    * Test your API endpoint `api.dev.hestia.homes` to ensure it's resolving correctly and accessing your Lambda
      function.

By following these steps, you should have your custom domain properly configured and pointing to your AWS Lambda
function via the CloudFront distribution