This project aims to forecast the energy consumption of the different regions of Denmark, allowing ML-driven decisions to allocate resources efficiently.

• ML Lifecycle
• Infrastructure
  • Architecture
• Configuration
  • Credentials
  • Build with Terraform
  • Configure with Ansible
• Local Debugging
• Project Structure
• Main Stack
• User Interface

The ML Lifecycle uses batch processing with an FTI(Feature-Training-Inference) methodoloy, allowing to decouple the different components of this pipeline, sticking to good MLOps practices. Also a monitoring step to calculate the error of the predictions on a hourly basis.

The data is served by two UIs or via REST APIs.

To configure the credentials values there are two possible ways:

1. Create .env files based on the .env.default values of the different folders and set values.

• Folders to check
• src/app-api
• batch-prediction-pipeline
• training-pipeline
• feature-pipeline

2. Set all necesary credentials as environment variables, a 'settings.py' file will automatically load the variables.

export FS_API_KEY=${value}
export FS_PROJECT_NAME=${value}  
export WANDB_API_KEY=${value}  
export WANDB_ENTITY=${value}  
export WANDB_PROJECT=${value}  
export AWS_ACCESS_KEY_ID=${value}  
export AWS_SECRET_ACCESS_KEY=${value}  
export AWS_BUCKET_NAME=${value}  

1. Configure credentials (if were not already defined as env variables)

export AWS_ACCESS_KEY_ID=${value}  
export AWS_SECRET_ACCESS_KEY=${value}  

2. Run the follwing command:

cd infrastructure/terraform
terraform plan
terraform apply -auto-approve

1. Upload your keypair

mkdir {WORKDIR}/infrastructure/ansible/keys                                             # Create key folder
mv {KEYPAIR_PATH}/TF_EnergyConsumption.pem {workdir}/infrastructure/ansible/keys        # Upload your key

2. Configure API keys There is a default file with all the necessary APIs keys

cat {ANSIBLE_PATH}/inventories/dev/group_vars/secrets.default.yml >> secrets.test.yml    # Configure all the API keys in a new file    
ansible-vault encrypt {ANSIBLE_PATH}/inventories/dev/group_vars/secrets.enc.yml

3. Configure ansible-vault password

{your_ansible_password} >> {ANSIBLE_PATH}/inventories/dev/group_vars/secrets-key

4. Configure IP addresses This can be done after deploying your infrastructure, because public IP addresses for the servers are randomly assigned from a pool. Open the following file: {ANSIBLE_PATH}/inventories/dev/hosts.ini and replaced the IP addresses.

5. Configurate servers. Once all previous steps are correctly done, configure the servers running the following command:

ansible-playbook system.yml

Configurations takes between 5-10 minutes

[NOTE!] Make sure the configure all the credentials before

To debug the application locally, run the following commands:

docker-compose build src/airflow/
docker-compose up src/airflow/

docker-compose build src/deploy/
docker-compose up src/deploy

The project main structure is as follows:

├── README.md                       
├── .github/workflows               
│   ├── CI/CD Pipelines             
├── infastructure                   
│   ├── ansible                     
│   │   ├── group_vars              
│   │   ├── playbooks               
│   │   └── hosts.ini               
│   ├── terraform                   
│   │   ├── modules
├── src               
│   ├── airflow                     
│   ├── app-api                     
│   ├── app-frontend                
│   ├── app-monitoring              
│   ├── batch-prediction-pipeline   
│   ├── deploy                      
│   ├── feature-pipeline            
│   ├── training-pipeline           

• Streamlit - The UI to visualize data metrics and predictions

• FastAPI - The API to communicate with website application

• Airflow - Scheduling, managing and running the different pipelines
• Hopsworks - The feature store used to versionize the data
• Weights&Biases - Logging and saving models metrics and model registry

• GitHubs Actions - The CI/CD Pipeline
• Docker / Docker-Compose - Contarize components
• Terraform - Provision and manage resources in the AWS
• Ansible - Automate configuration of AWS resources

Two different webpages UI were built, the fisrt one for visualizing the forecasting data of the differents zones, while the other allows to compare the predicted data with the truth data collected each hour.

PREDICTION

MONITORING