jimzucker / aws-forecast Goto Github PK

Lambda Function / Python Script to reproduce the forecast numbers we see at the top of AWS Cost Explorer and post a one-line update to Slack. Example output: Forecast: $7,275 (+7.19%)

License: Apache License 2.0

Python 97.82% Shell 2.18%

aws aws-boto3 cost-estimation cost-optimization cost-explorer python

aws-forecast's Introduction

aws-forecast

User Story

I found myself logging in daily check our AWS spend and change to prior month to keep an eye on our AWS bill and decided to create a script to slack it one time per day to save time.

So I set out to automate this as a slack post daily to save time. While doing this I found that the actual and forecast with % change from prior month that we see at the top of Cost Explorer are not directly available from the Cost Explorer API.

Acceptance Criteria

Numbers generated include percent change must be consistent with the numbers in Cost Explorer UI.
Application must produce a cleanly formatted one line output.
Code must be written as python functions that we can re-use to integrate into a slack-bot.
Post to slack if url is defined as an AWS secret (see below)
Provide example Lambda function that posts to slack on a cron 1 time per day

Solution

AWS Architecture

Environment Variables

We use these to make it compatible with running the same script from Lambda and the commandline for testing

GET_FORECAST_COLUMNS_DISPLAYED - specify columnns to display and the order 
    default: "Account,Forecast,Change"

GET_FORECAST_ACCOUNT_COLUMN_WIDTH - max width for account name for formatting
	default: 12

AWS_LAMBDA_FUNCTION_NAME - set if running in lambda(Automatically set in Lambda)
GET_FORECAST_AWS_PROFILE - set for testing on command line to pick a profile from your credentials file

Github Secrets for S3 Upload

This application uses a Github Workflow to upload the get-forecast.py lambda function to Amazon S3. To get this working properly, you need to configure your Github Secrets as follows, by providing it Credentials that have permission to access the S3 Bucket. For more information on giving permissions to access S3 Buckets through IAM Roles, see IAM Configuration.md Go to your repository settings -> Secrets and Variables, and create these 2 secrets. AWS_ACCESS_KEY = AWS_SECRET_KEY =

Cloud Formation

If you enter and slack URL and/or SNS and/or Teams URL it will publish in addition to logging.

File: get_forecast_cf.yaml

Note: The Cloud Formation loads the python script from a public S3 bucket, s3://jimzucker-github-getforecast/get_forecast.zip

Sample Output

Command line (for development/testing)

python3 get_forecast.py

Technical Notes

AWS API Used

get_cost_forecast - used to get current month forecast. (note we exclude credits)
get_cost_and_usage - used to get prior & current month actuals (note we exclude credits)

Boundary conditions handled

In testing I found several situations where the calls to get_cost_forecast would fail that we address in function calc_forecast:

Weekends - there is a sensitivity to the start date being on a weekend
Failure on new accounts or start of the month - on some days the calc fails due to insufficient data and we have to fall back to actuals

Manual instructions

If you dont want to use the Cloud Formation document, see these instructions: Click here

aws-forecast's People

Contributors

Stargazers

Watchers

Forkers

bishakha4265 serhiimalykhinriskfocus rsel99 klaxit subash-chandra subashc2023

aws-forecast's Issues

Enhance slack posting to be a graphic

Currently the slack messages is simple text we should look at improving the format.

boto3_session seems to error, possibly because Okta SSO or new aws cli version.

Running AWS SSO managed through Okta to get credentials and adding them to .aws/credentials file and manually running get_forecast.py with appropriate flags produces the following errors. Lambda is expected.

$ python3 get_forecast.py --profile redacted_AWSAdministratorAccess --type FORECAST
INFO:root:Not running as lambda
ERROR:root:Cannot connect to boto3
ERROR:root:Cannot connect to Cost Explorer with boto3

Will look into some sort of SSO or possible aws version issue but if any ideas would be helpful.

Re-create:

Login to SSO to get new credentials
Add new creds to ~/.aws/credentials files
export new creds to ensure term session is up-to-date
Run python3 get_forecast.py --profile redacted_AWSAdministratorAccess --type FORECAST

aws --version
aws-cli/2.0.26 Python/3.7.4 Darwin/19.6.0 botocore/2.0.0dev30

Create Cloud Formation for Installation

There is an open item to create cloud formation to install the stack.

error creating lambda outside us-east-1

After that i am trying to create the lambda function by using the cloud formation template by using the provided template from the git repo https://github.com/jimzucker/aws-forecast/blob/main/get_forecast_cf.yaml

While creating the cloud formation stack by using the template i am getting the following error:

getForecastLambda CREATE_FAILED

Resource handler returned message: "Error occurred while GetObject. S3 Error Code: PermanentRedirect. S3 Error Message: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: Lambda, Status Code: 400, Request ID: 93a27aa3-519f-4fc8-aff0-7431f1588832, Extended Request ID: null)" (RequestToken: 0b4cf6b3-9ff5-97e4-4a25-639d3784676b, HandlerErrorCode: InvalidRequest)

As per my understanding the Amazon S3 bucket in the same AWS Region as your function. The bucket can be in a different AWS account. I am trying to create this lambda function from us-west-2 region and I hope the https://s3.amazonaws.com/jimzucker-github-getforecast/get_forecast.zip is hosted in us-east-1