pbudzon / aws-maintenance Goto Github PK

Hello,

I'm getting the error below. Role has attached FullRDS rights.

Any idea?

13:08:02
START RequestId: a41934f6-9264-41f4-bd0c-cc2990538f3c Version: $LATEST

13:08:02
Latest snapshot found: 'rds:swat-rds-prd-2020-04-06-02-04' from 2020-04-06 02:04:32.297000+00:00

13:08:02
Checking if 'swat-rds-prd-None-rds-swat-rds-prd-2020-04-06-02-04' exists in target region

13:08:02
[ERROR] ClientError: An error occurred (AccessDenied) when calling the DescribeDBClusterSnapshots operation: Unknown Traceback (most recent call last): File "/var/task/lambda_function.py", line 237, in lambda_handler copy_latest_snapshot(account_id, cluster, True) File "/var/task/lambda_function.py", line 140, in copy_latest_snapshot print_encryption_info(source_snapshot_arn, is_aurora

13:08:02
END RequestId: a41934f6-9264-41f4-bd0c-cc2990538f3c

Hi @pbudzon

I just tested the updated Lambda function with the provided example event and it fails because no KMS key is provided for the cross-region RDS snapshot copy.

{
  "errorMessage": "An error occurred (InvalidParameterValue) when calling the CopyDBSnapshot operation: Must specify new KMS key for cross region encrypted snapshot copy.",
  "errorType": "ClientError",
  "stackTrace": [
    [
      "/var/task/index.py",
      103,
      "lambda_handler",
      "copy_latest_snapshot(account_id, message['Source ID'])"
    ],
    [
      "/var/task/index.py",
      48,
      "copy_latest_snapshot",
      "CopyTags=True"
    ],
    [
      "/var/runtime/botocore/client.py",
      317,
      "_api_call",
      "return self._make_api_call(operation_name, kwargs)"
    ],
    [
      "/var/runtime/botocore/client.py",
      615,
      "_make_api_call",
      "raise error_class(parsed_response, operation_name)"
    ]
  ]
}

Many thanks again.

After running the stack and executing a Lambda test I get the following: "errorMessage": "An error occurred (KMSKeyNotAccessibleFault) when calling the CopyDBSnapshot operation: The source snapshot KMS key does not exist, is not enabled or you do not have permissions to access it"

Is there something I need to add to the IAM role to get this working?

Hi @pbudzon

I followed your new instructions to enable the cross-region copy of encrypted RDS snapshots: I've created a new key in the destination region, downloaded the lambda python file, added it to a S3 bucket in the source region, downloaded the JSON file and changed the S3 bucket name in it, but when I try to create a new CF stack it fails.

Many thanks!

I am getting error, could you please guide. Please see the attached screenshot

Thank you for this Paulina, I have implemented via the template and it is working. However, the snapshots in the target region do not seem to be deleting and nothing in the cloudwatch logs suggest that the deletion is happening either. I ran into an AWS limit of not being able to have over 100 manual snapshots so am having to manually delete them. Have you seen this issue or maybe know a possible cause? Thank you @pbudzon

Can we copy 4 snapshots at a time to another region?

An error occurred (SnapshotQuotaExceeded) when calling the CopyDBSnapshot operation: Cannot copy more than 5 snapshots across regions: SnapshotQuotaExceededFault

AWS only allows 5 snapshots copy across regions. So, 'IN PROGRESS' status can't exceed more than 5.

File: backup-rds.py

Hi @pbudzon - and apologies if this is not the right place to discuss this error.

I've used your amazing code to deploy a CloudFormation stack to copy RDS snapshots from one region to another, following the readme instructions. First I tried to deploy it on the destination region and it failed, but then it was successfully deployed in the snapshot source region - the one that has the RDS instance that is generating the snapshots.

But the Lambda Function is failing to copy the snapshot. I don't know if it is related to the fact that our snapshots are encrypted or something else, but the log suggests that it is failing to identify that there's no copy of the snapshot in the destination region, like if the snapshot was already copied, and then the function quits without running the copy itself.

Can you advise on this please? Or is there a restriction on encrypted snapshots?

Many thanks.

I cloned your git repository to my local machine, uploaded the zipped python for RDS cross region backup copy to an s3 bucket. However when I upload the CFN template into the console it fails. Are you sure the provided template is valid?

When I am running the test function for aurora I am getting below error. Until now it was working fine but now i am seeing this error
An error occurred (SnapshotQuotaExceeded) when calling the CopyDBClusterSnapshot operation: Cannot create more than 100 manual snapshots: SnapshotQuotaExceededFault
Traceback (most recent call last):
File “/var/task/backup-rds.py”, line 259, in lambda_handler
copy_latest_snapshot(account_id, cluster, True)
File “/var/task/backup-rds.py”, line 172, in copy_latest_snapshot
SourceRegion=SOURCE_REGION
File “/var/runtime/botocore/client.py”, line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File “/var/runtime/botocore/client.py”, line 612, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.SnapshotQuotaExceededFault: An error occurred (SnapshotQuotaExceeded) when calling the CopyDBClusterSnapshot operation: Cannot create more than 100 manual snapshots

@pbudzon : The cross region backup for aurora instances removes snapshots which are one day older. If we want to implement some retention policy like keeping all the snapshots for last 30 days & 1 snapshot for day 1 of each of the previous months upto six months? I can keep the backup for 30 days by changing snapshots_to_remove = [i[0] for i in sorted_snapshots[29:]]. But for keeping the snapshot for each month of the first day (for last six months), do i need to change the name of the snapshot? Could you please suggest

Hello! I was wondering if it is possible to adapt the template so that, in addition to calling the lambda function with sns, it is possible to include or change the trigger for an event with Event Bridge that allows the lambda to be executed at certain times on certain days.

I have tried to do it "raw" by adding a trigger with event bride manually after the stack´s creation, but when executed it seems that it does not pass the parameters of the databases that I want to copy the snapshot and it fails.

Hello,

When I try to create the CloudFormation stack from the json template I got this error:

"The following resource types are not supported for resource import: AWS::Lambda::Permission,AWS::RDS::EventSubscription,AWS::Lambda::Permission"

Can you advice?

Thanks,
Carlos

I have multiple instances in RDS, one with AuroraMySQL engine & one with MySQL engine. In my source region snapshot has been created for both but in the target region only the snapshot related to MySQL engine has been created not for the AuroraMySQL engine.
As mentioned in the readme file i have left the DatabasesToUse as empty because i wanted the snapshots of all the RDS instances to be copied in the destination region.
Do i need to do something else in the code? Kindly suggest

Hi @pbudzon - when trying to deploy the CloudFormation stack for cross-region RDS snapshot copy to the destination region the stack creation fails. Our snapshots are encrypted.

The evidence is in the third post - the previous image on this post was from a different Lambda function.

Many thanks!