Questions regarding environment variables and AWS credentials about aws-mwaa-local-runner HOT 2 CLOSED

I think if you comb through the commits responsible for that env file being added: #95 and #149, the original intention seems to be so that you can test boto3 and AWS related operators in mwaa-local-runner.

Does the examples for setting credentials as environment variables mean this can be done when using MWAA in AWS, too? I have seen no documentation for this. I've only seen documentation for setting variables either using airflow custom configuration, or aws secrets manager. No mention of environment variables anywhere except this repo's README.

Don't think you can set the AWS creds directly as env variables for MWAA.

If not possible, to keep code parity, should I be using airflow variables or airflow connections locally?

I think the difficulty with keeping parity between your dev (mwaa-local-runner) and prod (deployed MWAA) is going to be the IAM role. When you're running locally, you need some way to pass your AWS credentials to the running mwaa-local-runner whereas on the production/deployed MWAA you can attach an IAM role to it directly.

If using connections, reading https://docs.aws.amazon.com/mwaa/latest/userguide/manage-connection-types.html it mentions we should not set aws_default since it overrides the internal one created by MWAA, but looking at the connections using the UI in the aws cloud, there are 0 connections there. Should I use that one?

If you create an aws_default connection in your local runner with the correct fields that would work and you wouldn't need the .env.localrunner set but that would be a little annoying since you would have to change them each time (assuming you're using a temporary credential from STS) whereas you can likely create some script to fill out that env.localrunner file with the temporary credential data before you start mwaa-local-runner.

Have a look here for setting up the connection details : https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html
If you're interested in how the aws operators work and authenticate with aws, have a look at the source code for AWS Hook: https://airflow.apache.org/docs/apache-airflow/1.10.10/_modules/airflow/contrib/hooks/aws_hook.html#AwsHook, specifically the _get_credentials method.

For our use case on MWAA we just create a task specific role that the mwaa role can assume for any AWS related service/resource interactions. Lot easier in my opinion.

from aws-mwaa-local-runner.

I'd also like to add that you can use the local runner to define environment variables in MWAA: https://docs.aws.amazon.com/mwaa/latest/userguide/using-startup-script.html. However in many cases, the suggestions above are better practice especially for sensitive data.

from aws-mwaa-local-runner.