A CDK stack to deploy Amazon EMR with Atlas.

1. Creates an AWS EMR cluster within a new VPC.
2. Creates an IAM service role for the EMR cluster to read scripts from s3 bucket.

Please refer to the following the link

Please refer to the following the link

git clone https://github.com/aws-samples/aws-cdk-emr-atlas
cd aws-cdk-emr-atlas/aws-emr-cdk/

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip3 install PyYAML
pip3 install aws_cdk.core
pip3 install aws_cdk.aws_emr
pip3 install aws_cdk.aws_ec2

Update the configurations in the app-config.yml file.

Before deploy, here is something you need to know:

1. You need a key pair config to EC2, which config in app-config.yaml file as emr->ec2->key_pair.
2. You need to create two S3 bucket, and config it in app-config.yaml for s3_log_bucket and s3_script_bucket.
3. Put file aws-cdk-emr-atlas/aws-emr-cdk/apache-atlas-emr.sh to the bucket, which is the value of 's3_script_bucket' key in app-config.yaml.
4. The IAM role and job flow role for EMR service, will be created automatically.
5. A VPC with public subnet will be created automatically.

cdk synth  # To review the cloudformation template
cdk diff  # To review the change set
cdk deploy  # To deploy the stack

After you deploy the stack, you could find a EMR cluster in the console, try to connect the master node in terminal to run a job, and test other services on it, or add a step on the console to test Hadoop and Spark.

1. AWS Best Practice: link
2. Resizing a cluster: link
3. Submit a job on console: link
4. Submit Hadoop jobs interactively: link
5. You can terminate a EMR cluster by console/CLI/APIA: link