aws-samples / amazon-mwaa-examples Goto Github PK

Amazon Managed Workflows for Apache Airflow (MWAA) Examples repository contains example DAGs, requirements.txt, plugins, and CloudFormation templates focused on Amazon MWAA.

License: MIT No Attribution

Python 59.01% JavaScript 1.01% Shell 1.72% Makefile 0.82% Jupyter Notebook 0.25% TypeScript 24.55% HTML 0.21% HCL 4.58% Batchfile 0.05% Dockerfile 0.18% PLpgSQL 7.62%

amazon-mwaa-examples's Introduction

Amazon Managed Workflows for Apache Airflow (MWAA) Examples

This repository contains example DAGs, requirements.txt, plugins, and CloudFormation templates focused on Amazon MWAA. Since Amazon MWAA is running open-source Apache Airflow many of the contributions will be applicable for self-managed implementations as well.

Contributing

We want examples of as many use cases in this repository as possible! Please submit a Pull Request if you would like to add something.

DAG Files

Example Directed Acyclic Graph (DAG) workflows that have been tested to work on Amazon MWAA. Associated requirements.txt will be referenced to the entry in the next section. See Amazon MWAA documentation for details.

Use Cases

This folder contains complete set of sample use cases including documentation, infrastructure as code, and dependant resources. Follow the README.md in each use case to get started.

Requirements.txt

Sometimes getting the right combination of Python libraries is tricky. This repository is here to help. See Amazon MWAA documentation for details.

Plugins

Most Airflow community plugins will work fine on Amazon MWAA. This repository is for specific examples that have been testing on the service. See Amazon MWAA documentation for details.

CloudFormation

Example CloudFormation templates for Amazon MWAA. See AWS CloudFormation documentation for details.

IAM Policies

Example AWS IAM Policy definitions. See Amazon MWAA documentation for details.

Setup for the examples

These examples all assume a working Amazon MWAA environment.

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

amazon-mwaa-examples's People

Contributors

Stargazers

Watchers

amazon-mwaa-examples's Issues

start-stop-mwaa-environment - AWS_REGION env var needed to deploy outside of us-east-1

Hello guys,
I was testing the start-stop-mwaa-environment project for our MWAA in eu-west-3.
I just wanted to let you know, that at least in my case, to successfully run the npm run deploy step I had to set this env var:

export AWS_REGION=eu-west-3

Without this variable npm run deploy would output this warning (see the -us-east-1):

current credentials could not be used to assume 'arn:aws:iam::*****:role/cdk-hnb659fds-deploy-role-*****-us-east-1', but are for the right account. Proceeding anyway.

And then errors out with:

mwaa-pause-resume-airflow-internal: SSM parameter /cdk-bootstrap/hnb659fds/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html)

After setting:

export AWS_REGION=eu-west-3

It worked perfectly.
Maybe is something that will be useful to have in the readme.

We are on airflow 2.7.2 and I am using the latest code of this project available in main.

mwaa_export_data - Lambda: Invoke - Error - Notify_success

I'm getting the following error on the notify_success task:
"botocore.errorfactory.TaskTimedOut: An error occurred (TaskTimedOut) when calling the SendTaskFailure operation: Task Timed Out: 'Provided task does not exist anymore'"
Reading this Stack Overflow solution it looks like the a taskToken can only be used once.
Can anyone help me solve this problem, please?

Thanks

husky - .git can't be found (see https://typicode.github.io/husky/#/?id=custom-directory)

I'm trying to run the "npm i" command and i am prompted with the below error

nitin@DESKTOP-2IDUI08:~/amazon-mwaa-examples/usecases/start-stop-mwaa-environment$ npm i

> [email protected] postinstall
> (cd lib/lambda && npm install) && husky install


up to date, audited 90 packages in 600ms

2 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
husky - .git can't be found (see https://typicode.github.io/husky/#/?id=custom-directory)
npm ERR! code 1
npm ERR! path /home/nitin/amazon-mwaa-examples/usecases/start-stop-mwaa-environment
npm ERR! command failed
npm ERR! command sh -c (cd lib/lambda && npm install) && husky install

npm ERR! A complete log of this run can be found in: /home/nitin/.npm/_logs/2023-09-05T12_59_24_077Z-debug-0.log

Surprisingly! The CDK deployment was successful despite of some of the test suits have failed while running the "npm test" command.
the error log is attached below

 FAIL  lib/lambda/mwaa-status-poller.test.ts
  ● Test suite failed to run

    lib/lambda/mwaa-status-poller.test.ts:33:41 - error TS2345: Argument of type 'MWAAClient' is not assignable
 to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, any>>'.
      Type 'MWAAClient' is not assignable to type 'Client<ServiceInputTypes, MetadataBearer, any>'.
        The types of 'middlewareStack.concat' are incompatible between these types.
          Type '<InputType extends ServiceInputTypes, OutputType extends ServiceOutputTypes>(from: MiddlewareSt
ack<InputType, OutputType>) => MiddlewareStack<...>' is not assignable to type '<InputType extends ServiceInput
Types, OutputType extends MetadataBearer>(from: MiddlewareStack<InputType, OutputType>) => MiddlewareStack<Inpu
tType, OutputType>'.
            Types of parameters 'from' and 'from' are incompatible.
              Type 'MiddlewareStack<InputType, OutputType>' is not assignable to type 'MiddlewareStack<InputTyp
e, ServiceOutputTypes>'.
                Types of property 'addRelativeTo' are incompatible.
                  Type '(middleware: MiddlewareType<InputType, OutputType>, options: RelativeMiddlewareOptions)
 => void' is not assignable to type '(middleware: MiddlewareType<InputType, ServiceOutputTypes>, options: Relat
iveMiddlewareOptions) => void'.
                    Types of parameters 'middleware' and 'middleware' are incompatible.
                      Type 'MiddlewareType<InputType, ServiceOutputTypes>' is not assignable to type 'Middlewar
eType<InputType, OutputType>'.
                        Type 'InitializeMiddleware<InputType, ServiceOutputTypes>' is not assignable to type 'M
iddlewareType<InputType, OutputType>'.
                          Type 'InitializeMiddleware<InputType, ServiceOutputTypes>' is not assignable to type
'InitializeMiddleware<InputType, OutputType>'.
                            Call signature return types 'InitializeHandler<InputType, ServiceOutputTypes>' and
'InitializeHandler<InputType, OutputType>' are incompatible.
                              Type 'Promise<InitializeHandlerOutput<ServiceOutputTypes>>' is not assignable to
type 'Promise<InitializeHandlerOutput<OutputType>>'.
                                Type 'InitializeHandlerOutput<ServiceOutputTypes>' is not assignable to type 'I
nitializeHandlerOutput<OutputType>'.
                                  Types of property 'output' are incompatible.
                                    Type 'ServiceOutputTypes' is not assignable to type 'OutputType'.
                                      'ServiceOutputTypes' is assignable to the constraint of type 'OutputType'
, but 'OutputType' could be instantiated with a different subtype of constraint 'MetadataBearer'.
                                        Type 'CreateCliTokenCommandOutput' is not assignable to type 'OutputTyp
e'.
                                          'CreateCliTokenCommandOutput' is assignable to the constraint of type
 'OutputType', but 'OutputType' could be instantiated with a different subtype of constraint 'MetadataBearer'.

    33       const mwaaClientMock = mockClient(mwaaClient);
                                               ~~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:35:9 - error TS2345: Argument of type '{ Environment: { Status: strin
g; }; }' is not assignable to parameter of type 'CommandResponse<MetadataBearer>'.
      Object literal may only specify known properties, and 'Environment' does not exist in type 'CommandRespon
se<MetadataBearer>'.

    35         Environment: {
               ~~~~~~~~~~~~~~
    36           Status: 'AVAILABLE',
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    37         },
       ~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:48:41 - error TS2345: Argument of type 'MWAAClient' is not assignable
 to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, any>>'.

    48       const mwaaClientMock = mockClient(mwaaClient);
                                               ~~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:50:9 - error TS2345: Argument of type '{ Environment: { Status: strin
g; }; }' is not assignable to parameter of type 'CommandResponse<MetadataBearer>'.
      Object literal may only specify known properties, and 'Environment' does not exist in type 'CommandRespon
se<MetadataBearer>'.

    50         Environment: {
               ~~~~~~~~~~~~~~
    51           Status: 'CREATING',
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    52         },
       ~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:63:41 - error TS2345: Argument of type 'MWAAClient' is not assignable
 to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, any>>'.

    63       const mwaaClientMock = mockClient(mwaaClient);
                                               ~~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:74:41 - error TS2345: Argument of type 'MWAAClient' is not assignable
 to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, any>>'.

    74       const mwaaClientMock = mockClient(mwaaClient);
                                               ~~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:87:41 - error TS2345: Argument of type 'MWAAClient' is not assignable
 to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, any>>'.

    87       const mwaaClientMock = mockClient(mwaaClient);
                                               ~~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:102:41 - error TS2345: Argument of type 'MWAAClient' is not assignabl
e to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, any>>'.

    102       const mwaaClientMock = mockClient(mwaaClient);
                                                ~~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:104:9 - error TS2345: Argument of type '{ Environment: { Status: stri
ng; }; }' is not assignable to parameter of type 'CommandResponse<MetadataBearer>'.
      Object literal may only specify known properties, and 'Environment' does not exist in type 'CommandRespon
se<MetadataBearer>'.

    104         Environment: {
                ~~~~~~~~~~~~~~
    105           Status: 'DELETED',
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    106         },
        ~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:117:41 - error TS2345: Argument of type 'MWAAClient' is not assignabl
e to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, any>>'.

    117       const mwaaClientMock = mockClient(mwaaClient);
                                                ~~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:119:9 - error TS2345: Argument of type '{ Environment: { Status: stri
ng; }; }' is not assignable to parameter of type 'CommandResponse<MetadataBearer>'.
      Object literal may only specify known properties, and 'Environment' does not exist in type 'CommandRespon
se<MetadataBearer>'.

    119         Environment: {
                ~~~~~~~~~~~~~~
    120           Status: 'AVAILABLE',
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    121         },
        ~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:132:41 - error TS2345: Argument of type 'MWAAClient' is not assignabl
e to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, any>>'.

    132       const mwaaClientMock = mockClient(mwaaClient);
                                                ~~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:143:41 - error TS2345: Argument of type 'MWAAClient' is not assignabl
e to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, any>>'.

    143       const mwaaClientMock = mockClient(mwaaClient);
                                                ~~~~~~~~~~
    lib/lambda/mwaa-status-poller.test.ts:156:41 - error TS2345: Argument of type 'MWAAClient' is not assignabl
e to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, any>>'.

    156       const mwaaClientMock = mockClient(mwaaClient);
                                                ~~~~~~~~~~


  751.1kb2116f70b352f89c26a6269ccf1564e7343602a24a7b9a74d3f2\index.js
  ...2116f70b352f89c26a6269ccf1564e7343602a24a7b9a74d3f2\index.js.map    1.4mb

Done in 416ms

...7e8e1d06e543bc54c0369252189ace173091346061b7070fa7303ec\index.js  751.1kb
  ...1d06e543bc54c0369252189ace173091346061b7070fa7303ec\index.js.map    1.4mb

Done in 498ms
Bundling asset mwaa-main-stack/mwaa-polling-stack/mwaa-polling-polling-fn/Code/Stage...
Bundling asset mwaa-main-stack/mwaa-polling-stack/mwaa-polling-polling-fn/Code/Stage...

  751.1kb6612f946fc3e6a2f5eb2d66ff30fe390b85fa298f2824109e5e\index.js
  ...6612f946fc3e6a2f5eb2d66ff30fe390b85fa298f2824109e5e\index.js.map    1.4mb

Done in 595ms
 FAIL  lib/lambda/mwaa-new-environment-function.test.ts
  ● Test suite failed to run

    lib/lambda/mwaa-new-environment-function.test.ts:23:29 - error TS2344: Type 'MWAAClient' does not satisfy t
he constraint 'Client<any, any, any>'.
      The types of 'middlewareStack.concat' are incompatible between these types.
        Type '<InputType extends ServiceInputTypes, OutputType extends ServiceOutputTypes>(from: MiddlewareStac
k<InputType, OutputType>) => MiddlewareStack<...>' is not assignable to type '<InputType extends any, OutputTyp
e extends any>(from: MiddlewareStack<InputType, OutputType>) => MiddlewareStack<InputType, OutputType>'.
          Types of parameters 'from' and 'from' are incompatible.
            Type 'MiddlewareStack<InputType, OutputType>' is not assignable to type 'MiddlewareStack<InputType,
 ServiceOutputTypes>'.
              Types of property 'addRelativeTo' are incompatible.
                Type '(middleware: MiddlewareType<InputType, OutputType>, options: RelativeMiddlewareOptions) =
> void' is not assignable to type '(middleware: MiddlewareType<InputType, ServiceOutputTypes>, options: Relativ
eMiddlewareOptions) => void'.
                  Types of parameters 'middleware' and 'middleware' are incompatible.
                    Type 'MiddlewareType<InputType, ServiceOutputTypes>' is not assignable to type 'MiddlewareT
ype<InputType, OutputType>'.
                      Type 'InitializeMiddleware<InputType, ServiceOutputTypes>' is not assignable to type 'Mid
dlewareType<InputType, OutputType>'.
                        Type 'InitializeMiddleware<InputType, ServiceOutputTypes>' is not assignable to type 'I
nitializeMiddleware<InputType, OutputType>'.
                          Call signature return types 'InitializeHandler<InputType, ServiceOutputTypes>' and 'I
nitializeHandler<InputType, OutputType>' are incompatible.
                            Type 'Promise<InitializeHandlerOutput<ServiceOutputTypes>>' is not assignable to ty
pe 'Promise<InitializeHandlerOutput<OutputType>>'.
                              Type 'InitializeHandlerOutput<ServiceOutputTypes>' is not assignable to type 'Ini
tializeHandlerOutput<OutputType>'.
                                Types of property 'output' are incompatible.
                                  Type 'ServiceOutputTypes' is not assignable to type 'OutputType'.
                                    'ServiceOutputTypes' is assignable to the constraint of type 'OutputType',
but 'OutputType' could be instantiated with a different subtype of constraint 'any'.
                                      Type 'CreateCliTokenCommandOutput' is not assignable to type 'OutputType'
.
                                        'CreateCliTokenCommandOutput' is assignable to the constraint of type '
OutputType', but 'OutputType' could be instantiated with a different subtype of constraint 'any'.

    23 let mwaaMock: AwsClientStub<MWAAClient>;
                                   ~~~~~~~~~~
    lib/lambda/mwaa-new-environment-function.test.ts:27:5 - error TS2322: Type 'AwsStub<ServiceInputTypes, Meta
dataBearer>' is not assignable to type 'never'.

    27     mwaaMock = mockClient(MWAAClient);
           ~~~~~~~~
    lib/lambda/mwaa-new-environment-function.test.ts:27:27 - error TS2345: Argument of type 'typeof MWAAClient'
 is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, any>>'.
      Type 'typeof MWAAClient' is not assignable to type 'ClassType<Client<ServiceInputTypes, MetadataBearer, a
ny>>'.
        Types of property 'prototype' are incompatible.
          Type 'MWAAClient' is not assignable to type 'Client<ServiceInputTypes, MetadataBearer, any>'.

    27     mwaaMock = mockClient(MWAAClient);
                                 ~~~~~~~~~~
    lib/lambda/mwaa-new-environment-function.test.ts:31:14 - error TS2339: Property 'restore' does not exist on
 type 'never'.

    31     mwaaMock.restore();
                    ~~~~~~~
    lib/lambda/mwaa-new-environment-function.test.ts:35:14 - error TS2339: Property 'on' does not exist on type
 'never'.

    35     mwaaMock.on(CreateEnvironmentCommand).resolves({ Arn: 'my-env-arn' });
                    ~~
    lib/lambda/mwaa-new-environment-function.test.ts:48:44 - error TS2345: Argument of type 'typeof CreateEnvir
onmentCommand' is not assignable to parameter of type 'new (input: CreateEnvironmentCommandInput) => AwsCommand
<CreateEnvironmentCommandInput, MetadataBearer, any, any>'.
      Types of construct signatures are incompatible.
        Type 'new (input: CreateEnvironmentCommandInput) => CreateEnvironmentCommand' is not assignable to type
 'new (input: CreateEnvironmentCommandInput) => AwsCommand<CreateEnvironmentCommandInput, MetadataBearer, any,
any>'.
          Construct signature return types 'CreateEnvironmentCommand' and 'AwsCommand<CreateEnvironmentCommandI
nput, MetadataBearer, any, any>' are incompatible.
            The types of 'middlewareStack.concat' are incompatible between these types.
              Type '<InputType extends CreateEnvironmentCommandInput, OutputType extends CreateEnvironmentComma
ndOutput>(from: MiddlewareStack<InputType, OutputType>) => MiddlewareStack<...>' is not assignable to type '<In
putType extends CreateEnvironmentCommandInput, OutputType extends MetadataBearer>(from: MiddlewareStack<InputTy
pe, OutputType>) => MiddlewareStack<...>'.
                Types of parameters 'from' and 'from' are incompatible.
                  Type 'MiddlewareStack<InputType, OutputType>' is not assignable to type 'MiddlewareStack<Inpu
tType, CreateEnvironmentCommandOutput>'.
                    Types of property 'addRelativeTo' are incompatible.
                      Type '(middleware: MiddlewareType<InputType, OutputType>, options: RelativeMiddlewareOpti
ons) => void' is not assignable to type '(middleware: MiddlewareType<InputType, CreateEnvironmentCommandOutput>
, options: RelativeMiddlewareOptions) => void'.
                        Types of parameters 'middleware' and 'middleware' are incompatible.
                          Type 'MiddlewareType<InputType, CreateEnvironmentCommandOutput>' is not assignable to
 type 'MiddlewareType<InputType, OutputType>'.
                            Type 'InitializeMiddleware<InputType, CreateEnvironmentCommandOutput>' is not assig
nable to type 'MiddlewareType<InputType, OutputType>'.
                              Type 'InitializeMiddleware<InputType, CreateEnvironmentCommandOutput>' is not ass
ignable to type 'InitializeMiddleware<InputType, OutputType>'.
                                Call signature return types 'InitializeHandler<InputType, CreateEnvironmentComm
andOutput>' and 'InitializeHandler<InputType, OutputType>' are incompatible.
                                  Type 'Promise<InitializeHandlerOutput<CreateEnvironmentCommandOutput>>' is no
t assignable to type 'Promise<InitializeHandlerOutput<OutputType>>'.
                                    Type 'InitializeHandlerOutput<CreateEnvironmentCommandOutput>' is not assig
nable to type 'InitializeHandlerOutput<OutputType>'.
                                      Types of property 'output' are incompatible.
                                        Type 'CreateEnvironmentCommandOutput' is not assignable to type 'Output
Type'.
                                          'CreateEnvironmentCommandOutput' is assignable to the constraint of t
ype 'OutputType', but 'OutputType' could be instantiated with a different subtype of constraint 'MetadataBearer
'.

    48     expect(mwaaMock).toHaveReceivedCommand(CreateEnvironmentCommand);
                                                  ~~~~~~~~~~~~~~~~~~~~~~~~
    lib/lambda/mwaa-new-environment-function.test.ts:53:14 - error TS2339: Property 'on' does not exist on type
 'never'.

    53     mwaaMock.on(CreateEnvironmentCommand).resolves({ Arn: 'my-env-arn' });
                    ~~
    lib/lambda/mwaa-new-environment-function.test.ts:66:44 - error TS2345: Argument of type 'typeof CreateEnvir
onmentCommand' is not assignable to parameter of type 'new (input: CreateEnvironmentCommandInput) => AwsCommand
<CreateEnvironmentCommandInput, MetadataBearer, any, any>'.

    66     expect(mwaaMock).toHaveReceivedCommand(CreateEnvironmentCommand);
                                                  ~~~~~~~~~~~~~~~~~~~~~~~~
    lib/lambda/mwaa-new-environment-function.test.ts:71:14 - error TS2339: Property 'on' does not exist on type
 'never'.

    71     mwaaMock.on(CreateEnvironmentCommand).resolves({ Arn: 'my-env-arn' });
                    ~~
    lib/lambda/mwaa-new-environment-function.test.ts:83:44 - error TS2345: Argument of type 'typeof CreateEnvir
onmentCommand' is not assignable to parameter of type 'new (input: CreateEnvironmentCommandInput) => AwsCommand
<CreateEnvironmentCommandInput, MetadataBearer, any, any>'.

    83     expect(mwaaMock).toHaveReceivedCommand(CreateEnvironmentCommand);
                                                  ~~~~~~~~~~~~~~~~~~~~~~~~


index.js  751.1kba054fefd21e607937ea05e4a747d9332fde7ae9e27b\
  ...65a65c13a054fefd21e607937ea05e4a747d9332fde7ae9e27b\index.js.map    1.4mb

Done in 826ms

  ...3bfa0d44d2ded8898c8cd3b5072331489d93f5f6b6310a52effd6e5\index.js  751.1kb
  ...0d44d2ded8898c8cd3b5072331489d93f5f6b6310a52effd6e5\index.js.map    1.4mb

Done in 693ms

751.1kbba9ad7dc1df6a71b563b9d8398c3d9db20ab4ef86dc0e1ff8a4a0\index.js
  ...9ad7dc1df6a71b563b9d8398c3d9db20ab4ef86dc0e1ff8a4a0\index.js.map    1.4mb

Done in 641ms
Bundling asset mwaa-main-stack/mwaa-polling-stack/mwaa-polling-polling-fn/Code/Stage...
Bundling asset mwaa-main-stack/mwaa-polling-stack/mwaa-polling-polling-fn/Code/Stage...
Bundling asset mwaa-main-stack/mwaa-polling-stack/mwaa-polling-polling-fn/Code/Stage...
Bundling asset mwaa-main-stack/mwaa-polling-stack/mwaa-polling-polling-fn/Code/Stage...
 FAIL  lib/lambda/dags-cli.test.ts
  ● Test suite failed to run

    lib/lambda/dags-cli.test.ts:24:29 - error TS2344: Type 'MWAAClient' does not satisfy the constraint 'Client
<any, any, any>'.

    24 let mwaaMock: AwsClientStub<MWAAClient>;
                                   ~~~~~~~~~~
    lib/lambda/dags-cli.test.ts:49:7 - error TS2322: Type 'AwsStub<ServiceInputTypes, MetadataBearer>' is not a
ssignable to type 'never'.

    49       mwaaMock = mockClient(MWAAClient);
             ~~~~~~~~
    lib/lambda/dags-cli.test.ts:49:29 - error TS2345: Argument of type 'typeof MWAAClient' is not assignable to
 parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, any>>'.

    49       mwaaMock = mockClient(MWAAClient);
                                   ~~~~~~~~~~
    lib/lambda/dags-cli.test.ts:50:16 - error TS2339: Property 'on' does not exist on type 'never'.

    50       mwaaMock.on(CreateCliTokenCommand).resolves(token);
                      ~~
    lib/lambda/dags-cli.test.ts:54:16 - error TS2339: Property 'restore' does not exist on type 'never'.

    54       mwaaMock.restore();
                      ~~~~~~~
    lib/lambda/dags-cli.test.ts:75:48 - error TS2345: Argument of type 'typeof CreateCliTokenCommand' is not as
signable to parameter of type 'new (input: CreateCliTokenCommandInput) => AwsCommand<CreateCliTokenCommandInput
, MetadataBearer, any, any>'.
      Types of construct signatures are incompatible.
        Type 'new (input: CreateCliTokenCommandInput) => CreateCliTokenCommand' is not assignable to type 'new
(input: CreateCliTokenCommandInput) => AwsCommand<CreateCliTokenCommandInput, MetadataBearer, any, any>'.
          Construct signature return types 'CreateCliTokenCommand' and 'AwsCommand<CreateCliTokenCommandInput,
MetadataBearer, any, any>' are incompatible.
            The types of 'middlewareStack.concat' are incompatible between these types.
              Type '<InputType extends CreateCliTokenCommandInput, OutputType extends CreateCliTokenCommandOutp
ut>(from: MiddlewareStack<InputType, OutputType>) => MiddlewareStack<...>' is not assignable to type '<InputTyp
e extends CreateCliTokenCommandInput, OutputType extends MetadataBearer>(from: MiddlewareStack<InputType, Outpu
tType>) => MiddlewareStack<...>'.
                Types of parameters 'from' and 'from' are incompatible.
                  Type 'MiddlewareStack<InputType, OutputType>' is not assignable to type 'MiddlewareStack<Inpu
tType, CreateCliTokenCommandOutput>'.
                    Types of property 'addRelativeTo' are incompatible.
                      Type '(middleware: MiddlewareType<InputType, OutputType>, options: RelativeMiddlewareOpti
ons) => void' is not assignable to type '(middleware: MiddlewareType<InputType, CreateCliTokenCommandOutput>, o
ptions: RelativeMiddlewareOptions) => void'.
                        Types of parameters 'middleware' and 'middleware' are incompatible.
                          Type 'MiddlewareType<InputType, CreateCliTokenCommandOutput>' is not assignable to ty
pe 'MiddlewareType<InputType, OutputType>'.
                            Type 'InitializeMiddleware<InputType, CreateCliTokenCommandOutput>' is not assignab
le to type 'MiddlewareType<InputType, OutputType>'.
                              Type 'InitializeMiddleware<InputType, CreateCliTokenCommandOutput>' is not assign
able to type 'InitializeMiddleware<InputType, OutputType>'.
                                Call signature return types 'InitializeHandler<InputType, CreateCliTokenCommand
Output>' and 'InitializeHandler<InputType, OutputType>' are incompatible.
                                  Type 'Promise<InitializeHandlerOutput<CreateCliTokenCommandOutput>>' is not a
ssignable to type 'Promise<InitializeHandlerOutput<OutputType>>'.
                                    Type 'InitializeHandlerOutput<CreateCliTokenCommandOutput>' is not assignab
le to type 'InitializeHandlerOutput<OutputType>'.
                                      Types of property 'output' are incompatible.
                                        Type 'CreateCliTokenCommandOutput' is not assignable to type 'OutputTyp
e'.
                                          'CreateCliTokenCommandOutput' is assignable to the constraint of type
 'OutputType', but 'OutputType' could be instantiated with a different subtype of constraint 'MetadataBearer'.

    75         expect(mwaaMock).toHaveReceivedCommand(CreateCliTokenCommand);
                                                      ~~~~~~~~~~~~~~~~~~~~~
    lib/lambda/dags-cli.test.ts:85:53 - error TS2345: Argument of type 'typeof CreateCliTokenCommand' is not as
signable to parameter of type 'new (input: CreateCliTokenCommandInput) => AwsCommand<CreateCliTokenCommandInput
, MetadataBearer, any, any>'.
      Types of construct signatures are incompatible.
        Type 'new (input: CreateCliTokenCommandInput) => CreateCliTokenCommand' is not assignable to type 'new
(input: CreateCliTokenCommandInput) => AwsCommand<CreateCliTokenCommandInput, MetadataBearer, any, any>'.
          Type 'CreateCliTokenCommand' is not assignable to type 'AwsCommand<CreateCliTokenCommandInput, Metada
taBearer, any, any>'.

    85         expect(mwaaMock).toHaveReceivedCommandTimes(CreateCliTokenCommand, 1);
                                                           ~~~~~~~~~~~~~~~~~~~~~


index.js  384.8kb27b2213755c3705b8200742b12aca8e2315aeef9d35\
  ...795a606727b2213755c3705b8200742b12aca8e2315aeef9d35\index.js.map  800.3kb

Error: Account id of stack environment must be a 'string' but received 'number'

I'm getting the following error while trying to deploy the "start-stop-mwaa-environment" stack in cdk using "npm run deploy" command

Error: Account id of stack environment must be a 'string' but received 'number'
    at MwaaMainStack.parseEnvironment (/home/nitin/amazon-mwaa-examples/usecases/start-stop-mwaa-environment/node_modules/aws-cdk-lib/core/lib/stack.js:1:18573)
    at new Stack (/home/nitin/amazon-mwaa-examples/usecases/start-stop-mwaa-environment/node_modules/aws-cdk-lib/core/lib/stack.js:1:2885)
    at new MwaaMainStack (/home/nitin/amazon-mwaa-examples/usecases/start-stop-mwaa-environment/lib/infrastructure/mwaa-main-stack.ts:58:5)
    at Object.<anonymous> (/home/nitin/amazon-mwaa-examples/usecases/start-stop-mwaa-environment/bin/mwaa-pause-resume.ts:29:1)
    at Module._compile (node:internal/modules/cjs/loader:1233:14)
    at Module.m._compile (/home/nitin/amazon-mwaa-examples/usecases/start-stop-mwaa-environment/node_modules/ts-node/src/index.ts:1618:23)
    at Module._extensions..js (node:internal/modules/cjs/loader:1287:10)
    at Object.require.extensions.<computed> [as .ts] (/home/nitin/amazon-mwaa-examples/usecases/start-stop-mwaa-environment/node_modules/ts-node/src/index.ts:1621:12)
    at Module.load (node:internal/modules/cjs/loader:1091:32)
    at Function.Module._load (node:internal/modules/cjs/loader:938:12)

Any help would be appreciated

Export functionality is not working

Hi Team,

I am using mwaa with airflow version 2.5.1. When the pause state machine gets invoked, it is constantly getting failed in the Export metadata phase with the following error. Looking forward to a resolution

**{

"dag": "mwaa_export_data",
"dag_run": "manual__2024-04-08T18:53:53+00:00",
"tasks": [
"export_variable => queued",
"export_data => queued",
"export_connection => queued",
"export_active_dags => running",
"back_up_activedags => success",
"pause_dags => success",
"clean_up => None",
"notify_success => None",
"activate_dags_on_failure => None"
],
"status": "Fail"
}**

export data DAG running into errors

the export data DAG keeps failing at the def export_active_dags() step. the pause_dags() task completes and all the active DAGs are paused. and then the DAG fails with the error FileNotFoundError: [Errno 2] No such file or directory: 's3://mys3-bucket/mwaa-migration-test/active_dags.csv'.
As a result of this, when the DAG fails, all my active DAGs remain in a paused state and I have to unpause all of them. Am I missing something?

Error: Invalid assembly manifest

Hello,
when trying to deploy I get the following error.
I have configured the .env file as you indicate and I have also tried with export but it still gives me the same error

Error: Invalid assembly manifest:
instance.missing[0].props.account is not of a type(s) string
instance.missing[0].props requires property "filters"
instance.missing[0].props.account is not of a type(s) string
instance.missing[0].props.account is not of a type(s) string
instance.missing[0].props requires property "domainName"
instance.missing[0].props.account is not of a type(s) string
instance.missing[0].props requires property "parameterName"
instance.missing[0].props.account is not of a type(s) string
instance.missing[0].props.account is not of a type(s) string
instance.missing[0].props requires property "serviceName"
instance.missing[0].props.account is not of a type(s) string
instance.missing[0].props requires property "loadBalancerType"
instance.missing[0].props.account is not of a type(s) string
instance.missing[0].props requires property "loadBalancerType"
instance.missing[0].props.account is not of a type(s) string
instance.missing[0].props.account is not of a type(s) string
instance.missing[0].props requires property "aliasName"
instance.missing[0].props requires property "pluginName"
instance.missing[0].props is not any of <#/definitions/AmiContextQuery>,<#/definitions/AvailabilityZonesContextQuery>,<#/definitions/HostedZoneContextQuery>,<#/definitions/SSMParameterContextQuery>,<#/definitions/VpcContextQuery>,<#/definitions/EndpointServiceAvailabilityZonesContextQuery>,<#/definitions/LoadBalancerContextQuery>,<#/definitions/LoadBalancerListenerContextQuery>,<#/definitions/SecurityGroupContextQuery>,<#/definitions/KeyContextQuery>,<#/definitions/PluginContextQuery>
    at Function.validate (/home/jonatan/Escritorio/Desarrollos/start-stop-mwaa-environment/node_modules/aws-cdk-lib/cloud-assembly-schema/lib/manifest.js:1:3118)
    at Function.saveManifest (/home/jonatan/Escritorio/Desarrollos/start-stop-mwaa-environment/node_modules/aws-cdk-lib/cloud-assembly-schema/lib/manifest.js:3:130)
    at Function.saveAssemblyManifest (/home/jonatan/Escritorio/Desarrollos/start-stop-mwaa-environment/node_modules/aws-cdk-lib/cloud-assembly-schema/lib/manifest.js:1:979)
    at CloudAssemblyBuilder.buildAssembly (/home/jonatan/Escritorio/Desarrollos/start-stop-mwaa-environment/node_modules/aws-cdk-lib/cx-api/lib/cloud-assembly.js:1:5956)
    at Object.synthesize (/home/jonatan/Escritorio/Desarrollos/start-stop-mwaa-environment/node_modules/aws-cdk-lib/core/lib/private/synthesis.js:1:849)
    at App.synth (/home/jonatan/Escritorio/Desarrollos/start-stop-mwaa-environment/node_modules/aws-cdk-lib/core/lib/stage.js:1:1922)
    at process.<anonymous> (/home/jonatan/Escritorio/Desarrollos/start-stop-mwaa-environment/node_modules/aws-cdk-lib/core/lib/app.js:1:1355)
    at Object.onceWrapper (node:events:629:26)
    at process.emit (node:events:514:28)
    at process.emit (node:domain:489:12)

Subprocess exited with error 1

start-stop-mwaa-environment - mwaa_import_data.py - task_instance.csv - fails for field next_kwargs is export contains a json object

If a task_instance export has a json object on next_kwargs field, the mwaa_export.py writes the object as a string with single quotes (i.e. "{'__var': {}, '__type': 'dict'}")

The mwaa_import.py COPY command, for json objects is expecting the field in the following format: {"__var": {}, "__type": "dict"}

as result, the import fails with the following error:

[2024-05-17T11:48:18.260+0000] {{taskinstance.py:1937}} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 192, in execute
    return_value = self.execute_callable()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 209, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/dags/mwaa_import.py", line 192, in load_data
    cursor.copy_expert(query, f)
psycopg2.errors.InvalidTextRepresentation: invalid input syntax for type json
DETAIL:  Token "'" is invalid.
CONTEXT:  JSON data, line 1: {'...
COPY task_instance, line 4811, column next_kwargs: "{'__var': {}, '__type': 'dict'}"

start-stop-mwaa-environment - npm build error

Hi team,

I strictly followed the instructions - having prerequisite prepared and exporting relevant env variables in shell, but I got this error when I ran npm build:

lib/lambda/dags-cli.test.ts:49:7 - error TS2322: Type 'AwsStub<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>' is not assignable to type 'never'.

49       mwaaMock = mockClient(MWAAClient);
         ~~~~~~~~

lib/lambda/dags-cli.test.ts:49:29 - error TS2345: Argument of type 'typeof MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.
  Type 'typeof MWAAClient' is not assignable to type 'ClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.
    The types of 'prototype.middlewareStack.concat' are incompatible between these types.
      Type '<InputType extends ServiceInputTypes, OutputType extends ServiceOutputTypes>(from: MiddlewareStack<InputType, OutputType>) => MiddlewareStack<...>' is not assignable to type '<InputType extends ServiceInputTypes, OutputType extends MetadataBearer>(from: MiddlewareStack<InputType, OutputType>) => MiddlewareStack<InputType, OutputType>'.
        Types of parameters 'from' and 'from' are incompatible.
          Type 'MiddlewareStack<InputType, OutputType>' is not assignable to type 'MiddlewareStack<InputType, ServiceOutputTypes>'.
            Types of property 'addRelativeTo' are incompatible.
              Type '(middleware: MiddlewareType<InputType, OutputType>, options: RelativeMiddlewareOptions) => void' is not assignable to type '(middleware: MiddlewareType<InputType, ServiceOutputTypes>, options: RelativeMiddlewareOptions) => void'.
                Types of parameters 'middleware' and 'middleware' are incompatible.
                  Type 'MiddlewareType<InputType, ServiceOutputTypes>' is not assignable to type 'MiddlewareType<InputType, OutputType>'.
                    Type 'InitializeMiddleware<InputType, ServiceOutputTypes>' is not assignable to type 'MiddlewareType<InputType, OutputType>'.
                      Type 'InitializeMiddleware<InputType, ServiceOutputTypes>' is not assignable to type 'InitializeMiddleware<InputType, OutputType>'.
                        Call signature return types 'InitializeHandler<InputType, ServiceOutputTypes>' and 'InitializeHandler<InputType, OutputType>' are incompatible.
                          Type 'Promise<InitializeHandlerOutput<ServiceOutputTypes>>' is not assignable to type 'Promise<InitializeHandlerOutput<OutputType>>'.
                            Type 'InitializeHandlerOutput<ServiceOutputTypes>' is not assignable to type 'InitializeHandlerOutput<OutputType>'.
                              Types of property 'output' are incompatible.
                                Type 'ServiceOutputTypes' is not assignable to type 'OutputType'.
                                  'ServiceOutputTypes' is assignable to the constraint of type 'OutputType', but 'OutputType' could be instantiated with a different subtype of constraint 'MetadataBearer'.
                                    Type 'CreateCliTokenCommandOutput' is not assignable to type 'OutputType'.
                                      'CreateCliTokenCommandOutput' is assignable to the constraint of type 'OutputType', but 'OutputType' could be instantiated with a different subtype of constraint 'MetadataBearer'.

49       mwaaMock = mockClient(MWAAClient);
                               ~~~~~~~~~~

lib/lambda/dags-cli.test.ts:50:16 - error TS2339: Property 'on' does not exist on type 'never'.

50       mwaaMock.on(CreateCliTokenCommand).resolves(token);
                  ~~

lib/lambda/dags-cli.test.ts:54:16 - error TS2339: Property 'restore' does not exist on type 'never'.

54       mwaaMock.restore();
                  ~~~~~~~

lib/lambda/dags-cli.test.ts:75:48 - error TS2345: Argument of type 'typeof CreateCliTokenCommand' is not assignable to parameter of type 'new (input: CreateCliTokenCommandInput) => AwsCommand<CreateCliTokenCommandInput, MetadataBearer>'.
  Construct signature return types 'CreateCliTokenCommand' and 'AwsCommand<CreateCliTokenCommandInput, MetadataBearer>' are incompatible.
    The types of 'middlewareStack.concat' are incompatible between these types.
      Type '<InputType extends CreateCliTokenCommandInput, OutputType extends CreateCliTokenCommandOutput>(from: MiddlewareStack<InputType, OutputType>) => MiddlewareStack<...>' is not assignable to type '<InputType extends CreateCliTokenCommandInput, OutputType extends MetadataBearer>(from: MiddlewareStack<InputType, OutputType>) => MiddlewareStack<...>'.
        Types of parameters 'from' and 'from' are incompatible.
          Type 'MiddlewareStack<InputType, OutputType>' is not assignable to type 'MiddlewareStack<InputType, CreateCliTokenCommandOutput>'.
            Types of property 'addRelativeTo' are incompatible.
              Type '(middleware: MiddlewareType<InputType, OutputType>, options: RelativeMiddlewareOptions) => void' is not assignable to type '(middleware: MiddlewareType<InputType, CreateCliTokenCommandOutput>, options: RelativeMiddlewareOptions) => void'.
                Types of parameters 'middleware' and 'middleware' are incompatible.
                  Type 'MiddlewareType<InputType, CreateCliTokenCommandOutput>' is not assignable to type 'MiddlewareType<InputType, OutputType>'.
                    Type 'InitializeMiddleware<InputType, CreateCliTokenCommandOutput>' is not assignable to type 'MiddlewareType<InputType, OutputType>'.
                      Type 'InitializeMiddleware<InputType, CreateCliTokenCommandOutput>' is not assignable to type 'InitializeMiddleware<InputType, OutputType>'.
                        Call signature return types 'InitializeHandler<InputType, CreateCliTokenCommandOutput>' and 'InitializeHandler<InputType, OutputType>' are incompatible.
                          Type 'Promise<InitializeHandlerOutput<CreateCliTokenCommandOutput>>' is not assignable to type 'Promise<InitializeHandlerOutput<OutputType>>'.
                            Type 'InitializeHandlerOutput<CreateCliTokenCommandOutput>' is not assignable to type 'InitializeHandlerOutput<OutputType>'.
                              Types of property 'output' are incompatible.
                                Type 'CreateCliTokenCommandOutput' is not assignable to type 'OutputType'.
                                  'CreateCliTokenCommandOutput' is assignable to the constraint of type 'OutputType', but 'OutputType' could be instantiated with a different subtype of constraint 'MetadataBearer'.

75         expect(mwaaMock).toHaveReceivedCommand(CreateCliTokenCommand);
                                                  ~~~~~~~~~~~~~~~~~~~~~

lib/lambda/dags-cli.test.ts:85:53 - error TS2345: Argument of type 'typeof CreateCliTokenCommand' is not assignable to parameter of type 'new (input: CreateCliTokenCommandInput) => AwsCommand<CreateCliTokenCommandInput, MetadataBearer>'.
  Type 'CreateCliTokenCommand' is not assignable to type 'AwsCommand<CreateCliTokenCommandInput, MetadataBearer>'.

85         expect(mwaaMock).toHaveReceivedCommandTimes(CreateCliTokenCommand, 1);
                                                       ~~~~~~~~~~~~~~~~~~~~~

lib/lambda/mwaa-new-environment-function.test.ts:27:5 - error TS2322: Type 'AwsStub<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>' is not assignable to type 'never'.

27     mwaaMock = mockClient(MWAAClient);
       ~~~~~~~~

lib/lambda/mwaa-new-environment-function.test.ts:27:27 - error TS2345: Argument of type 'typeof MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.

27     mwaaMock = mockClient(MWAAClient);
                             ~~~~~~~~~~

lib/lambda/mwaa-new-environment-function.test.ts:31:14 - error TS2339: Property 'restore' does not exist on type 'never'.

31     mwaaMock.restore();
                ~~~~~~~

lib/lambda/mwaa-new-environment-function.test.ts:35:14 - error TS2339: Property 'on' does not exist on type 'never'.

35     mwaaMock.on(CreateEnvironmentCommand).resolves({ Arn: 'my-env-arn' });
                ~~

lib/lambda/mwaa-new-environment-function.test.ts:48:44 - error TS2345: Argument of type 'typeof CreateEnvironmentCommand' is not assignable to parameter of type 'new (input: CreateEnvironmentCommandInput) => AwsCommand<CreateEnvironmentCommandInput, MetadataBearer>'.
  Construct signature return types 'CreateEnvironmentCommand' and 'AwsCommand<CreateEnvironmentCommandInput, MetadataBearer>' are incompatible.
    The types of 'middlewareStack.concat' are incompatible between these types.
      Type '<InputType extends CreateEnvironmentCommandInput, OutputType extends CreateEnvironmentCommandOutput>(from: MiddlewareStack<InputType, OutputType>) => MiddlewareStack<...>' is not assignable to type '<InputType extends CreateEnvironmentCommandInput, OutputType extends MetadataBearer>(from: MiddlewareStack<InputType, OutputType>) => MiddlewareStack<...>'.
        Types of parameters 'from' and 'from' are incompatible.
          Type 'MiddlewareStack<InputType, OutputType>' is not assignable to type 'MiddlewareStack<InputType, CreateEnvironmentCommandOutput>'.
            Types of property 'addRelativeTo' are incompatible.
              Type '(middleware: MiddlewareType<InputType, OutputType>, options: RelativeMiddlewareOptions) => void' is not assignable to type '(middleware: MiddlewareType<InputType, CreateEnvironmentCommandOutput>, options: RelativeMiddlewareOptions) => void'.
                Types of parameters 'middleware' and 'middleware' are incompatible.
                  Type 'MiddlewareType<InputType, CreateEnvironmentCommandOutput>' is not assignable to type 'MiddlewareType<InputType, OutputType>'.
                    Type 'InitializeMiddleware<InputType, CreateEnvironmentCommandOutput>' is not assignable to type 'MiddlewareType<InputType, OutputType>'.
                      Type 'InitializeMiddleware<InputType, CreateEnvironmentCommandOutput>' is not assignable to type 'InitializeMiddleware<InputType, OutputType>'.
                        Call signature return types 'InitializeHandler<InputType, CreateEnvironmentCommandOutput>' and 'InitializeHandler<InputType, OutputType>' are incompatible.
                          Type 'Promise<InitializeHandlerOutput<CreateEnvironmentCommandOutput>>' is not assignable to type 'Promise<InitializeHandlerOutput<OutputType>>'.
                            Type 'InitializeHandlerOutput<CreateEnvironmentCommandOutput>' is not assignable to type 'InitializeHandlerOutput<OutputType>'.
                              Types of property 'output' are incompatible.
                                Type 'CreateEnvironmentCommandOutput' is not assignable to type 'OutputType'.
                                  'CreateEnvironmentCommandOutput' is assignable to the constraint of type 'OutputType', but 'OutputType' could be instantiated with a different subtype of constraint 'MetadataBearer'.

48     expect(mwaaMock).toHaveReceivedCommand(CreateEnvironmentCommand);
                                              ~~~~~~~~~~~~~~~~~~~~~~~~

lib/lambda/mwaa-new-environment-function.test.ts:53:14 - error TS2339: Property 'on' does not exist on type 'never'.

53     mwaaMock.on(CreateEnvironmentCommand).resolves({ Arn: 'my-env-arn' });
                ~~

lib/lambda/mwaa-new-environment-function.test.ts:66:44 - error TS2345: Argument of type 'typeof CreateEnvironmentCommand' is not assignable to parameter of type 'new (input: CreateEnvironmentCommandInput) => AwsCommand<CreateEnvironmentCommandInput, MetadataBearer>'.

66     expect(mwaaMock).toHaveReceivedCommand(CreateEnvironmentCommand);
                                              ~~~~~~~~~~~~~~~~~~~~~~~~

lib/lambda/mwaa-new-environment-function.test.ts:71:14 - error TS2339: Property 'on' does not exist on type 'never'.

71     mwaaMock.on(CreateEnvironmentCommand).resolves({ Arn: 'my-env-arn' });
                ~~

lib/lambda/mwaa-new-environment-function.test.ts:83:44 - error TS2345: Argument of type 'typeof CreateEnvironmentCommand' is not assignable to parameter of type 'new (input: CreateEnvironmentCommandInput) => AwsCommand<CreateEnvironmentCommandInput, MetadataBearer>'.

83     expect(mwaaMock).toHaveReceivedCommand(CreateEnvironmentCommand);
                                              ~~~~~~~~~~~~~~~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:33:41 - error TS2345: Argument of type 'MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.
  Type 'MWAAClient' is not assignable to type 'Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>'.

33       const mwaaClientMock = mockClient(mwaaClient);
                                           ~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:35:9 - error TS2353: Object literal may only specify known properties, and 'Environment' does not exist in type 'CommandResponse<MetadataBearer>'.

35         Environment: {
           ~~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:48:41 - error TS2345: Argument of type 'MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.

48       const mwaaClientMock = mockClient(mwaaClient);
                                           ~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:50:9 - error TS2353: Object literal may only specify known properties, and 'Environment' does not exist in type 'CommandResponse<MetadataBearer>'.

50         Environment: {
           ~~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:63:41 - error TS2345: Argument of type 'MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.

63       const mwaaClientMock = mockClient(mwaaClient);
                                           ~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:74:41 - error TS2345: Argument of type 'MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.

74       const mwaaClientMock = mockClient(mwaaClient);
                                           ~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:87:41 - error TS2345: Argument of type 'MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.

87       const mwaaClientMock = mockClient(mwaaClient);
                                           ~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:102:41 - error TS2345: Argument of type 'MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.

102       const mwaaClientMock = mockClient(mwaaClient);
                                            ~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:104:9 - error TS2353: Object literal may only specify known properties, and 'Environment' does not exist in type 'CommandResponse<MetadataBearer>'.

104         Environment: {
            ~~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:117:41 - error TS2345: Argument of type 'MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.

117       const mwaaClientMock = mockClient(mwaaClient);
                                            ~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:119:9 - error TS2353: Object literal may only specify known properties, and 'Environment' does not exist in type 'CommandResponse<MetadataBearer>'.

119         Environment: {
            ~~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:132:41 - error TS2345: Argument of type 'MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.

132       const mwaaClientMock = mockClient(mwaaClient);
                                            ~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:143:41 - error TS2345: Argument of type 'MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.

143       const mwaaClientMock = mockClient(mwaaClient);
                                            ~~~~~~~~~~

lib/lambda/mwaa-status-poller.test.ts:156:41 - error TS2345: Argument of type 'MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.

156       const mwaaClientMock = mockClient(mwaaClient);
                                            ~~~~~~~~~~

lib/lambda/mwaa-update-environment-function.test.ts:27:5 - error TS2322: Type 'AwsStub<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>' is not assignable to type 'never'.

27     mwaaMock = mockClient(MWAAClient);
       ~~~~~~~~

lib/lambda/mwaa-update-environment-function.test.ts:27:27 - error TS2345: Argument of type 'typeof MWAAClient' is not assignable to parameter of type 'InstanceOrClassType<Client<ServiceInputTypes, MetadataBearer, SmithyResolvedConfiguration<HttpHandlerOptions>>>'.

27     mwaaMock = mockClient(MWAAClient);
                             ~~~~~~~~~~

lib/lambda/mwaa-update-environment-function.test.ts:31:14 - error TS2339: Property 'restore' does not exist on type 'never'.

31     mwaaMock.restore();
                ~~~~~~~

lib/lambda/mwaa-update-environment-function.test.ts:35:14 - error TS2339: Property 'on' does not exist on type 'never'.

35     mwaaMock.on(UpdateEnvironmentCommand).resolves({ Arn: 'my-env-arn' });
                ~~

lib/lambda/mwaa-update-environment-function.test.ts:39:44 - error TS2345: Argument of type 'typeof UpdateEnvironmentCommand' is not assignable to parameter of type 'new (input: UpdateEnvironmentCommandInput) => AwsCommand<UpdateEnvironmentCommandInput, MetadataBearer>'.
  Construct signature return types 'UpdateEnvironmentCommand' and 'AwsCommand<UpdateEnvironmentCommandInput, MetadataBearer>' are incompatible.
    The types of 'middlewareStack.concat' are incompatible between these types.
      Type '<InputType extends UpdateEnvironmentCommandInput, OutputType extends UpdateEnvironmentCommandOutput>(from: MiddlewareStack<InputType, OutputType>) => MiddlewareStack<...>' is not assignable to type '<InputType extends UpdateEnvironmentCommandInput, OutputType extends MetadataBearer>(from: MiddlewareStack<InputType, OutputType>) => MiddlewareStack<...>'.
        Types of parameters 'from' and 'from' are incompatible.
          Type 'MiddlewareStack<InputType, OutputType>' is not assignable to type 'MiddlewareStack<InputType, UpdateEnvironmentCommandOutput>'.
            Types of property 'addRelativeTo' are incompatible.
              Type '(middleware: MiddlewareType<InputType, OutputType>, options: RelativeMiddlewareOptions) => void' is not assignable to type '(middleware: MiddlewareType<InputType, UpdateEnvironmentCommandOutput>, options: RelativeMiddlewareOptions) => void'.
                Types of parameters 'middleware' and 'middleware' are incompatible.
                  Type 'MiddlewareType<InputType, UpdateEnvironmentCommandOutput>' is not assignable to type 'MiddlewareType<InputType, OutputType>'.
                    Type 'InitializeMiddleware<InputType, UpdateEnvironmentCommandOutput>' is not assignable to type 'MiddlewareType<InputType, OutputType>'.
                      Type 'InitializeMiddleware<InputType, UpdateEnvironmentCommandOutput>' is not assignable to type 'InitializeMiddleware<InputType, OutputType>'.
                        Call signature return types 'InitializeHandler<InputType, UpdateEnvironmentCommandOutput>' and 'InitializeHandler<InputType, OutputType>' are incompatible.
                          Type 'Promise<InitializeHandlerOutput<UpdateEnvironmentCommandOutput>>' is not assignable to type 'Promise<InitializeHandlerOutput<OutputType>>'.
                            Type 'InitializeHandlerOutput<UpdateEnvironmentCommandOutput>' is not assignable to type 'InitializeHandlerOutput<OutputType>'.
                              Types of property 'output' are incompatible.
                                Type 'UpdateEnvironmentCommandOutput' is not assignable to type 'OutputType'.
                                  'UpdateEnvironmentCommandOutput' is assignable to the constraint of type 'OutputType', but 'OutputType' could be instantiated with a different subtype of constraint 'MetadataBearer'.

39     expect(mwaaMock).toHaveReceivedCommand(UpdateEnvironmentCommand);
                                              ~~~~~~~~~~~~~~~~~~~~~~~~


Found 34 errors in 4 files.

Errors  Files
     6  lib/lambda/dags-cli.test.ts:49
     9  lib/lambda/mwaa-new-environment-function.test.ts:27
    14  lib/lambda/mwaa-status-poller.test.ts:33
     5  lib/lambda/mwaa-update-environment-function.test.ts:27

Could you please help to direct the right way to troubleshoot the issues above? Thank you!

execution_date no longer exists in task_instance table

In version 2.2.2 the code here should be updated from:
r.execution_date = ti.execution_date
to
r.run_id = ti.run_id

execution_date column update: apache/airflow#19593

Create variable for VPC Endpoint creation if already created

Hi Team,

We are encountering an issue while deploying the start-stop solution in an account that has multiple MWAA environments. When we try to deploy the start-stop solution for another environment, the CloudFormation stack creation is failing and going into a rollback state.

The error message we are receiving is:

"Resource handler returned message: "private-dns-enabled cannot be set because there is already a conflicting DNS domain for states.us-east-1.amazonaws.com in the VPC vpc-0d301afa40a5768ca (Service: Ec2, Status Code: 400, Request ID: ccc14144-39f2-4078-b7e4-8f8dfc0b6d26)" (RequestToken: 3dc89c65-c60e-9c5c-6bf8-b261ab1a9fb7, HandlerErrorCode: GeneralServiceException)"

Could you please help us create a variable for the VPC endpoint creation, in case it already exists? This would allow us to handle the conflict and successfully deploy the start-stop solution for the additional MWAA environment.

Please let me know if you need any further information or assistance from our end.

Thanks,
Shubham

export_data failed due to Negsignal.SIGKILL

I am exporting data from 2.0.2 to 2.4.3.

I run the DAG export_data and it failed to export

Task exited with return code Negsignal.SIGKILL

Adding logger to this DAG and it failed to export task_instance.

MWAA Metadata Dag Export: AccessDenied when calling the CreateMultipartUpload operation, Access Denied

While trying to migrate from my current MWAA environment, I am taking my ideas from your repo from this folder example:
usecases/metadata-migration/2.2.2-2.4.3/export_data.py

Obviously I have adapted to our correspondent situation which is migrating from MWAA version 2.6.3 to 2.8.1, when deploying the DAG I am getting an issue for export_data_t, export_active_dags_t and export_variable_t with the following error message

ValueError: the bucket XXX does not exist, or is forbidden for access (ClientError('An error occurred (AccessDenied) when calling the CreateMultipartUpload operation: Access Denied'))

add metadata-migration for new version 2.6.3

There is a new MWAA version 2.6.3 available as [announced on Aug 10th, see also [MWAA versions]
It would be helpful to have adapted migration DAGs, updated with the newer version of airflow 2.6.3.

add metadata-migration for new version 2.5.1

There is a new MWAA version 2.5.1 available as announced on April 11th, see also MWAA versions.

It would be helpful to have adapted migration DAGs, ideally

2.2.2 -> 2.5.1 (my team is still with v2.2.2)
2.4.3 -> 2.5.1

start-stop-mwaa-environment - mwaa_import_data.py - variable.csv - fails for field larger than field limit

Hello guys,

In our case the resume step fails because the mwaa_import_data dag fails while importing variable.csv
This is the error:

[2024-05-16, 08:01:16 UTC] {{taskinstance.py:1937}} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 192, in execute
    return_value = self.execute_callable()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 209, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/dags/mwaa_import_data.py", line 146, in importVariable
    for row in reader:
_csv.Error: field larger than field limit (131072)

Just FYI I've fixed it by adding these few lines of code to mwaa_import_data.py:

import sys
import csv
maxInt = sys.maxsize

while True:
    # decrease the maxInt value by factor 10 
    # as long as the OverflowError occurs.

    try:
        csv.field_size_limit(maxInt)
        break
    except OverflowError:
        maxInt = int(maxInt/10)

Coming from this: https://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072

I am not sure if this is the correct way to manage this, but for what I've seen it seems to be working fine for us.