Unable to access Glue tables with data.all dataset role due to KMS:decrypt permission not found about dataall HOT 4 OPEN

Hi @dlpzx , Thanks for taking a look at this issue. As a quick measure we have updated our docs.

For a long term solution, if we add the dataset admin role explicitly to the KMS policy ( like how it is done for created dataset where the KMS policy is created during the stack creation phase and then a SID "EnableDatasetIAMRoleKeyUsage" with permissions is added allowing dataset admin role to access KMS key ) then this should solve the problem.

As this cannot be done via CDK for imported KMS keys during the dataset_stack creation phase ( https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_kms/README.html#:~:text=alias/foo%22)-,Note,-that%20a%20call )

Another way would be to do the update by assuming the pivot Role and then adding a statement in KMS policy for dataset admin role to have access. This could be done during the post_deployment() step mentioned in the cdk_cli_wrapper.py file. Thus the dataset module will have a cli_wrapper.post_deployment implementation which will perform this after the dataset_stack creation is completed

Please let me knw your thoughts on this one

from dataall.

Hi @TejasRGitHub I understand the issue and it is not surprising for more strict KMS keys that remove the root permissions. What do you think is the best course of action?
As a first quick measurement, we can update the docs; but, I think that for the long term, we should look for automatic ways of ensuring access.

For example we could add the statement upon import and re-check during shares. Similar to generate_enable_pivot_role_permissions_policy_statement. What do you think?

from dataall.

Hi @TejasRGitHub I am picking up this issue, that as you pointed out, is trickier than it looks.

We should add an statement that grants permissions to the dataset_role and I imagine other roles environment_team_roles to the imported KMS key policy when the dataset is created. Because we do not control the IaC of the KMS key we need to perform these grants using SDK calls.

If we want to use the dataallPivotRole with SDK calls:
When we create/import a dataset 2 stacks get deployed: dataset and environment. The permission that grants access to this KMS key to the dataallPivotRole is added when the environment stack is updated. So what needs to happen:

1. user triggers dataset import
2. dataset stack and environment stacks triggered deployment
3. in environment stack wait for completion of dataallPivotRole update
4. update KMS policy with a custom resource/post_execution with dataallPivotRole

From a logical perspective, any action on the dataset KMS key should happen in the deployment of the dataset, not of the environment. But if we add a post_deployment to the dataset stack we cannot ensure that the pivot role is updated. We could have it as part of the environment stack, but it is a bit weird that there is a KMS dataset action after the environment is updated.

I am thinking of avoiding using the dataallpivotRole completely and using some sort of custom resource with the CDK exec role. I need to do some research

Let me think about it, because I am not convinced by the solutions proposed yet

from dataall.

Hi @dlpzx , thanks a lot for taking a look at this issue. I was of the view that post_deployment would start when the stacks are completely deployed. Thanks for clearing this out.

For now, we have found out an internal solution for mitigating this issue. Although, not needed rightnow we could certainly take a look at this in the future releases when we have a concrete solution. I think it will be helpful if data.all can work with super restrictive policies ( as the one mentioned in this GH issue ).

from dataall.