TooManyRequestsException: Too many requests about cognito-backup HOT 11 CLOSED

did your data (number of users) increase over the last days?
Are you running cognito-backup backup-users only?
I think this could be a bug in aws-sdk, because it's supposed to handle retry with back-off automatically

from cognito-backup.

A couple of users increased then, but the total number is only 20,000.

I'm running backup-users only, some parameters are specified tough:

backup-users $POOL_ID --region ap-northeast-1 --profile $PROFILE --file backup/$TODAY.users.json

According to CHANGELOG.md, aws-sdk is 2.1125.0, which was installed with cognito-backup.

# Changelog for AWS SDK for JavaScript
<!--LATEST=2.1125.0-->
<!--ENTRYINSERT-->

from cognito-backup.

AFAIK the AWS SDK should automatically retry on these kind of errors: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html

not sure why it's not doing it. you don't happen to have any minimal reproducible code you can share?

from cognito-backup.

Hi!
Through trial and error, I've found that half the limit worked fine.

c.f.
hykw@4f0b5db#diff-180b3d2af89416b99563fff8c37eb8ca1ef7591fc1ddd736b5dacf740ccd3760L78

from cognito-backup.

Cool! I have now replaced the bottleneck module with aws-sdk v3's built in throttling logic, so it should in theory not require any trial and error and should automatically back-off if it gets throttled. try out the new version will you! (3.0.1)

from cognito-backup.

@mifi
Hi!
Thank you for the update. Though the backup time took longer than before, 3.0.1 worked correctly!

FYI.
I have about 46000 users in a pool.

• The former version(3.0.0) took 10 minutes
• The trial/error version took 20 minutes
• This version(3.0.1): 39 minutes

from cognito-backup.

thanks for testing this! do you have groups also that gets backed up? are most users connected to at least one group?

from cognito-backup.

I have released a new version with a flag --concurrency. maybe you can try that when you have time (e.g. try set it to 2)

from cognito-backup.

@mifi
Hi!
Significantly improved performance!

I have a few groups, but most of the users are connected to one of the group.

from cognito-backup.

Ok, that's great. and no errors?

from cognito-backup.

at least so far(until concurrency=4), no errors at all!

from cognito-backup.