virtualzero / cloudfrontaccesslogstoexcel Goto Github PK

View Code? Open in Web Editor NEW

License: GNU Lesser General Public License v3.0

Python 98.01% CSS 0.27% TeX 0.01% JavaScript 0.06% C++ 0.48% C 1.12% Fortran 0.03% Smarty 0.01%

cloudfrontaccesslogstoexcel's Introduction

CloudFrontAccessLogsToExcel

When AWS CloudFront is configured to deliver access logs, it can be a chore to parse, aggregate, and organize all of the data. Each access log is ~5 KB and thousands can easily stack up within a week. CloudFrontAccessLogsToExcel offers 2 solutions: a cloud solution that will run without intervention on a schedule defined by a Cron expression and a CLI version for local use. Both versions download all of the access logs from a specified bucket, uncompress each log, read each log, append its data to the master log, convert the master log format from CSV to XLSX, save the XLSX file, delete the downloaded access logs, and delete the access logs from the S3 bucket that the logs are stored in. The CLI version allows for many of these operations to be performed individually, while the cloud version will save the generated XLSX file in a S3 bucket. Many of the operations performed in the script are threaded to increase performance and reduce run time.

Cloud Solution

1. Upload the zip file located in aws/lambda to a S3 bucket

2. Note the uploaded zip file's bucket name, key, and version ID

3. Upload the CloudFormation template located in aws/cloudformation to Cloudformation

4. Enter the required information into the form

5. CloudFormation will provision all of the resources and handle all permissions

Resources created by the CloudFormation template:

IAM Role
IAM Managed Policy
Lambda Function
CloudWatch Rule
CloudWatch to Lambda Permission

CLI Solution

1. Install the dependencies

cd CloudFrontAccessLogsToExcel && pipenv install

2. Execute the script

pipenv run python cli/access_logs_to_excel_cli.py -b

Usage

Flags

Flag	Operation
-h --help	Displays the help menu, flags, and flag descriptions
-d --download-logs	Download CloudFront access logs from S3
-x --convert-to-excel	Convert downloaded CloudFront access logs to Excel spreadsheet
-b --batch-operation	Performs all operations, most common usage
-r --delete-remote	Deletes all CloudFront access logs from S3 bucket
-l --delete-local	Deletes all downloaded CloudFront access logs and directory

Recommend Projects

virtualzero / cloudfrontaccesslogstoexcel Goto Github PK