`airflow connections list` returns cryptography.fernet.InvalidToken about aws-mwaa-local-runner HOT 7 CLOSED

I would suggest that for local development the FERNET_KEY variable remains static and fixed in the docker-compose definition.

First, generate a cryptography key (fernet_key)

python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)"

then, copy-paste the printed result into the environment definition of your docker-compose.

        environment:
            - LOAD_EX=n
            - EXECUTOR=Local
            - FERNET_KEY=xxx

It is better to restart the environment for these changes to take place properly.

• docker-compose -f ./docker/docker-compose-local.yml down
• rm -r db-data
• ./mwaa-local-env start

if necessary, replace it in the airflow.cfg as well.

This should solve part of the issue.
The other part is the ability of aws-mwaa-local-runner to perform inner airflow cli commands.

This is possible to verify by simply going into the container and execute airflow version:

docker exec CONTAINER_ID airflow version

(to check your container_id do: docker ps)

The output should be something like:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/usr/local/lib/python3.7/site-packages/airflow/__init__.py", line 34, in <module>
    from airflow import settings
  File "/usr/local/lib/python3.7/site-packages/airflow/settings.py", line 37, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/usr/local/lib/python3.7/site-packages/airflow/configuration.py", line 1098, in <module>
    conf = initialize_config()
  File "/usr/local/lib/python3.7/site-packages/airflow/configuration.py", line 860, in initialize_config
    conf.validate()
  File "/usr/local/lib/python3.7/site-packages/airflow/configuration.py", line 199, in validate
    self._validate_config_dependencies()
  File "/usr/local/lib/python3.7/site-packages/airflow/configuration.py", line 240, in _validate_config_dependencies
    f"error: sqlite C library version too old (< {min_sqlite_version}). "
airflow.exceptions.AirflowConfigException: error: sqlite C library version too old (< 3.15.0). See https://airflow.apache.org/docs/apache-airflow/2.0.2/howto/set-up-database.rst#setting-up-a-sqlite-database

In fact, this simply means that CLI is trying to access to the default database (sqlite). We have to override this configuration in order to perform any CLI operations.

Simply add SqlAlchemy connection (AIRFLOW__CORE__SQL_ALCHEMY_CONN) to the environment in the docker-compose definition:

        environment:
            - LOAD_EX=n
            - EXECUTOR=Local
            - FERNET_KEY=xxx
            - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow

Other solution would be to add it in the airflow.cfg file by replacing this line:
#sql_alchemy_conn = sqlite:///{AIRFLOW_HOME}/airflow.db
by
sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@postgres:5432/airflow

Now, it should be possible to perform airflow connectios list.
Possibly helps with #27

This solution I propose should be only temporary waiting for the complete integration into the local-runner process.

from aws-mwaa-local-runner.

same problem here, I can add a connection by going to http://localhost:8080/connection/add and the connection works. But when I restart the environment, the connection doesn't work anymore and I get the same error when I execute my DAG.

from aws-mwaa-local-runner.

Possibly related to #27

from aws-mwaa-local-runner.

Possibly related to #29

from aws-mwaa-local-runner.

I have found a procedure that fixes this.

To Replicate

1. define fernet_key = ... in airflow.cfg
2. run ./mwaa-local-env build-image
3. run ./mwaa-local-env start
4. http://localhost:8080/home will run successfully
5. connect to local_runner container docker exec -it docker_local-runner_1 /bin/sh
6. run airflow connections list will return the same error listed above cryptography.fernet.InvalidToken

The Fix

7. run airflow db reset -y
8. airflow connections list will now work - and it will continue to work even when the container is stopped and restarted.

Break It Again

9. run airflow db init and this will cause the problem reported in number 6 above to return.

Can anyone (@o-nikolas @dhegberg) explain why this is the case?

from aws-mwaa-local-runner.

As a workaround:

1. delete db-data local folder
2. run ./mwaa-local-env build-image
3. run ./mwaa-local-env start

from aws-mwaa-local-runner.

As a workaround:

1. delete db-data local folder
2. run ./mwaa-local-env build-image
3. run ./mwaa-local-env start

For me, this still results in the same problem. The airflow cli still fails to display airflow connections list

from aws-mwaa-local-runner.