A simple python quick start template to use with AWS Batch that helps you build a docker image through CI / CD .
This demo batch downloads a sample json and uploads to s3 destination.
- Install Python 3.6
- Optional: Install virtual environment https://virtualenv.pypa.io/en/latest/installation/ or conda https://conda.io/docs/installation.html
- Install python dependencies
pip install -r source/requirements.txt
export PYTHONPATH=./source
# To get help
python ./source/main.py -h
# Sample download data to current directory
python ./source/main.py .
# Sample download data to s3 path s3://mybucket/mydir/
python ./source/main.py . --s3 s3://mybucket/mydir/
-
Create an repository in ECR registry called "aws-batch-sample-python", as detailed here https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html
-
Setup AWS codebuild to build a docker container as detailed using cloudformation stack codebuild_cloudformation.json. This uses the buildspec.yaml. For more details on codebuild see https://docs.aws.amazon.com/codebuild/latest/userguide/sample-docker.html.
Note The ecr repositoryname to use is specified in the buildspec.yaml file -
Start a build in codebuild to push a new image into the ECS Repository "aws-batch-sample-python". When the build succeds, you will see a image in the repository
-
Register a job with AWS batch as detailed in aws_batch/README.md
This library is licensed under the MIT-0 License. See the LICENSE file.