Wondering what version of Node this will be using going forward with the EOL coming up soon.

The first "cd ../source/services/limitreport" works but then the second "cd ../source/services/slacknotify" fails because it wants to change to: "source/services/source/services/slacknotify".

The Trust advisior only checks some aws service limits. Many other aws service limits are not included.
Is it possible with a custom template for limit monitor to check other aws service limits(not included in Trusted advisior)? Do you have an example template for this?

best regards

@aws-limit-monitor
Limit Monitor Documentation
AccountId
XXXXXXXXXXXXXX
Status
:warning:
TimeStamp
2020-01-21T20:00:15Z
Region
us-east-1
Service
RDS
LimitName
Max auths per security group
CurrentUsage
20
LimitAmount
20

It seems that even though the service limit has been excluded from the TA service limit check, the alert is still being sent as a warning. As you may know. RDS's "Max auths per security group" is a hard limit

Hi
I am planning to run the 'build-s3-dist.sh' from the deployment directory. I have noted that the 1st prerequisite is :
Important notes and prereq's:

1. The initialize-repo.sh script must have been run in order for this script to

function properly.

Can anyone please advise me where I could get this above mentioned 'initialize-repo.sh script?'

Dipankar Chakrabarti

Example key from latest version of Limit Monitor:

{
  "AccountId": "123456789012",
  "CurrentUsage": "Green", <-
  "ExpiryTime": 1560964756524,
  "LimitAmount": "0", <-
  "LimitName": "1000000", <-
  "MessageId": "02102f5b-01cf-4f88-86b2-3cda37bd1fe5",
  "Region": "ap-south-1",
  "Service": "EBS",
  "Status": "OK",
  "TimeStamp": "2019-06-04T17:16:04Z"
}

Culprit seems to be here, as Slack notifications are referencing proper attributes

I have deployed the stack, but dont believe it is working as expected.

I enabled DEBUG on all the lambdas and I see in the limitCheckStack only the following message:
START RequestId: b3f11639-74d7-439e-9751-896b78fa6c48 Version: $LATEST
18:55:17
2020-02-06T18:55:17.338Z b3f11639-74d7-439e-9751-896b78fa6c48 INFO [ERROR]UnknownEndpoint: Inaccessible host: servicequotas.eu-north-1.amazonaws.com'. This service may not be available in the eu-north-1' region.
18:56:09
END RequestId: b3f11639-74d7-439e-9751-896b78fa6c48

There is nothing in the dynamodb table and nothing being sent to either of the SQS. The Troubleshooting section suggests ensuring that the account ID is correct. I have verified the account is correct and "surrounded".

I see from the Lambda > Application > Monitoring > CW Events that all of the cw events have "Failed Invocations". No other monitoring tabs show any errors.

Please assist.

Thanks,
Ryan

New service limit checks were added in January 2019 for Route53 and DynamoDB: https://aws.amazon.com/about-aws/whats-new/2019/01/aws-trusted-advisor-expands-functionality/

A great feature would be to have a support for StackSet SERVICE_MANAGED using Organization OU.

Currently if i want to deploy spoke stacks through Stackset on a Organization OU, i still need to add the new account into the "Account list" master stack parameter.

Additionnaly, nested stacks are not supported in stackset SERVICE_MANAGED permission model.

With over 100+ accounts loaded in the accountlist, I am hitting the parameter value limit for this. Solution is to have 2-3 parameter lists and doing a join

We've deployed this solution and we're now getting notifications from best practices checks about the inline policy used by this solution:
https://github.com/awslabs/aws-limit-monitor/blob/1f365564eae725aeac994bbbd5b2895d76c30332/deployment/service-quotas-checks.template#L57-L75

Would it be possible to change this from an inline policy to a managed policy?

Supporting docs:

The different types of policies are for different use cases. In most cases, we recommend that you use managed policies instead of inline policies.
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html

For custom policies, we recommend that you use managed policies instead of inline policies.
https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#best-practice-managed-vs-inline

You have exceeded your maximum gp2 storage limit of 600 TiB in this region. Please contact AWS Support to request an Elastic Block Store service limit increase.

It would be nice to get messages when approaching our EBS limits like we do with other EC2 limits

Hi,

When deploying the AWS Limit Monitor to 30+ accounts, we are getting this error on the Lambda Function LimtrHelperFunction, that support the CFn Custom Resource.

{
    "CreateTrust": {
        "status": "ERROR",
        "account": "xxxxxxxxxxxx",
        "response": {
            "message": "Policy size would be larger than the maximum allowed.",
            "code": "PolicyLengthExceededException",
            "time": "2019-xx-xxTxx:xx:xx.263Z",
            "requestId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
            "statusCode": 400,
            "retryable": false,
            "retryDelay": 89.38078875757718
        }
    }
}

From what I could find out this is happening because we are reaching the limit of 10KB for the permission policy of the event bus.

From the python SDK [1] (I believe on nodejs is the same) I've read the following:

if all the accounts are members of the same AWS organization, you can run PutPermission once specifying Principal as "*" and specifying the AWS organization ID in Condition , to grant permissions to all accounts in that organization

So, is it possible to update the CFn templates and CFn Lambda Helper functions to support using the AWS Org ID in the createTrust function?

[1] - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/events.html#CloudWatchEvents.Client.put_permission

I am trying to work out the full policy to manually put in to avoid the issue "Amazon CloudWatch Events Bus Permissions Error". The instructions in the troubleshooter are no longer correct.

Firstly we can't add them anymore in cloudwatch console we have to add them in EventBridge, secondly adding the permissions fro the root/master account alone doesn't work fully, I think I have to do something in the sending account as well as the receiving account.

I am adding something similar in receiving/master account:

"Version": "2012-10-17",
"Statement": [{
"Sid": "limtr-458054464678",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<ID of master/receiver account with event bus in it>:root"
},
"Action": "events:PutEvents",
"Resource": "arn:aws:events:us-east-1:<ID of master/receiver account with event bus in it>:event-bus/default"
}, {
"Sid": "842046a0-70a3-11eb-95a7-93a72639c6a91613513316362",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<ID of secondary/sender account>:root"
},
"Action": "events:*",
"Resource": "arn:aws:events:us-east-1:<ID of master/receiver account with event bus in it>:event-bus/default"
}]
}

Hello, I have two problems with using this tool to monitor our enterprise AWS account.

1. The VPC Elastic IP addresses (EIPs) limit
   It was reported reaching the 80%(4/5) on 08/26, and the limit was increased to 6 on 08/27. But the monitoring still report below message during 08/28 to 08/29(today). Something wrong with this check. We had other two limits reported and they don't have same problem after the limits were increased.
   VPC Elastic IP addresses (EIPs)
   Region: us-east-1
   Resource Limit: 5
   Resource Usage: 4
2. Totally there are 92 limits info reported from trusted advisor under our AWS account, it looks only a few of them are covered by this aws-limit-monitor, and we do have 3 limits already alerted from Trusted Advisor but was not reported by aws-limit-monitor. Can I have idea about what limits this tool covers?

Thanks.

Got this error:
Cannot find module 'uuid/v4' when invoking the helper function.

Easy fix.
const uuidv4 = require('uuid').v4;

It seems that every once in a while (happens ~2 per day) we are getting a timeout from the Lambda.
Looking at the cloudwatch logs there's nothing there.
In addition I see that this function consumes ~100-120MB of RAM and the limit of that function is 128MB.

I've added the graphs of the invocation and timeout, and the cloudwatch logs of a timed out invocation:

This is running the latest version of your code, deployed through the link in the AWS Limit Monitor home page without any modifications.

Seems the limit-monitor-spoke template is now failing because it is creating a function called TARefresher instead of {stackname}-TARefresher-{randomcharacters}

This is preventing me from creating multiple stacks in the same account

Using the latest version of your template (5.3.3 at the time of writing) i've found a wrong "Condition" for the "TASNSRule" Resource (row 2145 in limit-monitor.template)

It's:
'Condition': 'SlackTrue'

It would be:
'Condition', 'SNSTrue'

If i don't set the Slack-related parameters (that was my use case) the EventBridge Rule for SNS will not be created.

Hello,

We received a warning from AWS that the following checks are now deprecated and scheduled to be removed on Nov 18th:

This notification will impact you if you are using one of the following checks in AWS Trusted Advisor:

1. EC2Config Service for EC2 Windows Instances in Fault Tolerance category with Check ID V77iOLlBqz
2. PV Driver Version for EC2 Windows Instances in Fault Tolerance category with Check ID Wnwm9Il5bG
3. NVMe Driver Version for EC2 Windows Instances in Fault Tolerance category with Check ID yHAGQJV9K5
4. ENA Driver Version for EC2 Windows Instances in Fault Tolerance category with Check ID TyfdMXG69d
5. EBS Active Volumes in Service Limits category with Check ID fH7LL0l7J9

What is the change?

AWS Trusted Advisor will remove 5 checks on November 18, 2020. On this date, 5 checks listed above will be removed from the list of checks in both Support API and the Trusted Advisor console. If you consume results from all checks in Trusted Advisor, these checks will not show up in check results. If you specifically consume any of the 5 checks listed above (such as by hard coding the Check ID in an API call), you will need to remove these checks from your list to avoid API call errors.

We see in the current iteration of aws-limit-monitor that the EBS Active Volumes check seems to be used: https://github.com/awslabs/aws-limit-monitor/blob/master/source/services/tarefresh/lib/ta-refresh.js#L38 - there may be others, this is just the first I spotted.

Are there any plans to update this tool so the above check deprecation doesn't cause breakage?

Thanks!

Hi,

When trying to execute a Test Event I receive the following error:

cannot concatenate 'str' and 'int' objects: TypeError
Traceback (most recent call last):
File "/var/task/limitMaster.py", line 10, in lambda_handler
print "do stuff here with AWS account "+id+"
"
TypeError: cannot concatenate 'str' and 'int' objects

Any help would be appreciated.

Cheers,
S

I ran the solution with 4 accounts and set up Slack configuration from the Stack. A parameter store for the slack was not created and the lambda fails when sending alerts:

2019-09-27T00:27:36.225Z 0008c69c-9c4f-4758-9f27-aec000e61988 [ERROR]AccessDeniedException: No access to reserved parameter name: awslimitschecker.
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:51:27)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)

Hello, I don't see a way to change the frequency of the alert.

We are getting an alert every 30 minutes. And it's becoming annoying. We place support tickets to request increases, but sometime its takes more than 24 hours for the limit to be increased and in the mean time we keep getting the alerts.

Can we limit the alert to once or twice a day?

It looks like this only supports the multiline email output.

Hello, I am getting

2019-11-05T02:21:27.912Z c58f1545-4b3d-457a-ab90-4ac3f5206fa3 Error: connect ECONNREFUSED 127.0.0.1:443
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)

error in lambda slack notification, I only find this error in the logs.

I get the email correctly, but slack notifications don't work.

I used the oficial documentation https://aws.amazon.com/solutions/limit-monitor/

someone experiencing the same problem?
thanks

Without reading through the code, there doesn't appear to be any list of what limits are checked here (and it's now doing more than just TrustedAdvisor checks, as you include DynamoDB).

By default we receive test notifications only when “WARN” or “ERROR” state is reached. So to get notifications I set the value to "OK" and still I was not able to get the notifications

I gave below parameters while creating stack
SlackChannel : slacknotifier
SlackHookURL :https://hooks.slack.com/services/xxxxxx/xxx/xxxx

CF created only one parameter in parameter store as - slacknotifier and value as slack_dummy.

Result : Lambda function logs shows - INFO [ERROR]ParameterNotFound: null

Let me know what we need to do to solve this issue ?
Let me know key, value pairs to be created in parameter store.

CloudWatch event(SQSRule) invocation failed when trying to send message to SQS. Seems SQS uses an AWS managed key alias/aws/sqs for encryption. https://github.com/awslabs/aws-limit-monitor/blob/master/deployment/limit-monitor.template#L225
However, this key is not accessible for CloudWatch event due to the following limit defined in the key policy.

"Condition": {
    "StringEquals": {
        "kms:CallerAccount": "xxxxxxxxxxxxxxx",
        "kms:ViaService": "sqs.us-east-1.amazonaws.com"
    }
}

I suppose that a customer managed key should be used here just like SNS part. https://github.com/awslabs/aws-limit-monitor/blob/master/deployment/limit-monitor.template#L471

It looks as though a Trusted Advisor limit that is labeled as "VPC" in the TA UI is instead named "Network Interfaces" in the SNS notification.

Email Notification Level
List of alert levels to send email alerts in response to. Leave blank if you do not wish to receive email notifications. Must be double-quoted and comma separated.

Email Address
(Required) The email address to subscribe for alert messages.

It doesn't make sense to me to have an option in email notification level to leave it blank if we don't want notifications and then to also require that an email address be present.

also if I do try to leave it blank for Email Notification Level I get the following error:

Template format error: Unresolved resource dependencies [SNSTopic] in the Resources block of the template

so basically it an Email Notification Level appears to be required.

Instructions Incorrect (https://docs.aws.amazon.com/solutions/latest/limit-monitor/deployment.html#step1)

In the section about Slack Notification Levels:

Choose the status event level(s) that will trigger Slack notifications. For example, “WARN”, “ERROR”. Note that the format is double quotation marks and comma separated (for multiple values).
Note
Leave this parameter blank if you do not want to receive Slack notifications. Note that the Slack notification components will not be deployed.

However if left Blank, the Cloudformation template fails validation, citing SlackRole Dependency check

For instance, the DynamoDB table has one tag defined, I would like to provide my own list of tags and values.

Error when using template in master and on S3.

Cannot render the template because of an error.: YAMLException: duplicated mapping key at line 334, column 30: Effect: Allow ^

Caused by:
3d71c27#diff-9b113a30da49f7487f0c44f9f6556492R334

It appears that when this lambda executes it "always" runs a trusted advisor update check, which can be problematic when you have an account that's very large. I recommend adding in some validation to help it to determine if it's been updated within X amount of time to make sure it's not just always hitting refresh.

The results of this are the following error:
Unable to retrieve Trusted Advisor results. Some data may be from the previous report.

Version

5.2

Issue

When a slack notification is enabled with the values:

SlackChannel: service-limit-alerts
SlackHookURL: https://hooks.slack.com/services/xxxxxx/xxx/xxx

The following error is logged in CloudWatch:

{
    "SSMParameter": {
        "channelKey": "service-limit-alerts",
        "hookKey": "https://hooks.slack.com/services/xxxxxx/xxx/xxx",
        "response": {
            "message": "Parameter name must be a fully qualified name.",
            "code": "ValidationException",
            "time": "2019-08-08T12:05:04.216Z",
            "requestId": "07cae9a5-1a44-4090-a361-7d070cf7f151",
            "statusCode": 400,
            "retryable": false,
            "retryDelay": 94.35200443904534
        }
    }
}

Investigation

LimtrHelperFunction -> lib/index.js revealed the SSM parameter value by default is SLACK_DUMMY:

await Promise.all(
      data.InvalidParameters.map(async ssmParam => {
        console.log(ssmParam);
        await ssm
          .putParameter({
            Name: ssmParam /* required */,
            Type: 'String' /* required */,
            Value: 'SLACK_DUMMY' /* required */,
          })
          .promise();
      })
    );

Also, there is this block of code in SlackNotifier -> lib/slack-notify.js:

self.ssm.getParameter(
            {
              Name: _self.slackHookURL,
              WithDecryption: true,
            },
            function(err, _hookData) {
              if (err) return cb(err, null);
                 LOGGER.log('ERROR', err.stack);
                 return cb(err, null); // an error occurred;
              } else {
                  let _slackURL = _hookData.Parameter.Value;

                   let _slackMssg = _self.slackMessageBuilder(event);
                   _slackMssg.channel = _channelData.Parameter.Value;
              ...
           }

This obviously means whatever value is retrieved from SlackChannel and SlackHookURL are what would be used to try to send a message to Slack; in both cases SLACK_DUMMY would be returned by SSM.

Potential Solution

I think what needs to be done is to allow SlackChannel, SlackHookURL and their corresponding SSM keys SlackChannelKey, SlackHookURLKey to be specified as SSMParameters -> Properties. And then stored as name and value pairs in SSM.

Happy to work on a fix, if you want!

Hello,

We have implemented the aws-limit-monitor and its spokes in a lot of our accounts. But the Lambda LimitMonitorFunction has some problems. It is throwing NoSuchResourceException and keeps running for a looooong time. I have set the logging level to DEBUG and these logs are repeated in CloudWatch over and over.

| 2020-08-10T16:17:56.785+02:00 | START RequestId: 306fe37b-5bf2-4915-8402-dd0f07c03234 Version: $LATEST
| 2020-08-10T16:17:56.789+02:00 | 2020-08-10T14:17:56.789Z 306fe37b-5bf2-4915-8402-dd0f07c03234 INFO [DEBUG]Received event: { "version": "0", "id": "61c3ab5c-ce1e-a92b-ca1c-b3071460314e", "detail-type": "Scheduled Event", "source": "aws.events", "account": "451413662958", "time": "2020-08-10T14:17:31Z", "region": "us-east-1", "resources": [ "arn:aws:events:us-east-1:451413662958:rule/aws-limit-monitor-spoke-limitCh-LimitCheckSchedule-1393J5ZZR9EQZ" ], "detail": {} }
| 2020-08-10T16:17:59.639+02:00 | 2020-08-10T14:17:59.639Z	306fe37b-5bf2-4915-8402-dd0f07c03234	INFO	[ERROR]NoSuchResourceException: The request failed because the specified service does not exist.
| 2020-08-10T16:18:45.604+02:00 | 2020-08-10T14:18:45.603Z 306fe37b-5bf2-4915-8402-dd0f07c03234 INFO [DEBUG]Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances
| 2020-08-10T16:18:53.823+02:00 | END RequestId: 306fe37b-5bf2-4915-8402-dd0f07c03234
| 2020-08-10T16:18:53.823+02:00 | REPORT RequestId: 306fe37b-5bf2-4915-8402-dd0f07c03234 Duration: 57034.52 ms Billed Duration: 57100 ms Memory Size: 128 MB Max Memory Used: 110 MB

We are getting notifications to Slack. But this is adding unnecessary costs having the Lambda run for 57 seconds multiple times per hour.

Sorry if this is a user error from my side. But (I think) I have followed the guide correctly while setting it up.

Team AWS / Limit Monitor Solution-

First and foremost - thanks for this helpful tool / solution. We have multiple internal partners in our enterprise who are keenly interested in this service (and more).

We struggled a bit with the deployment - and along the way, found some hard-coded references in the Spoke Template which assume that the centralized (main template) resources in the Primary account are deployed in us-east-1. At first, we simply updated these references to point to us-west-2, where we had Deployed the primary stack in a centralized account. When things didn't work, we cut bait, went back to the 'stock' templates provided in the Solution, and deployed everything in us-east-1.

Happy to submit a PR to address this hard-coded dependency - by either allowing the user to 'select' the region for the centralized / 'back-end' services -- or my changing the hard-coded references to intrinsic functions and just looking for those resources in the same region in which the spoke template is executed.

Thanks again for this Solution! We hope to use it as a foundation, incorporating additional limit monitoring not currently provided by Trusted Advisor, but feeding into the same collection / notification / archiving back-end - and will be sure to offer any enhancements developed back to the solution in the form of a PR.

First of all, Thanks so much for this! It has been a huge help and we are super on top of our service limits now. I did want to note that we are getting the same alert up to four or more times per day in the slack channel which is strange because the cloudwatch event is set to only run once per day. Another thing is that the DynamoDB table is filling up with old readings. I see some TTL column, but is there anything that lifecycles out those old dynamodb records after a month or some other period of time? I don't see much advantage in hanging on to those historical numbers. What purpose does the ExpiryTime( TTL) column serve in the table if I may ask?

Has anyone successfully deployed this solution to a different region?

"Serverless Framework" (https://serverless.com)
https://github.com/serverless/serverless - 37,000 ⭐ stars on github 😮

Basically, this request amounts to adding a serverless.yaml to the project so you can easily deploy with sls deploy instead of wrestling with these CloudFormation templates. Usually you don't need to rework any actual app/code, unless you want to.

As shown in the screenshot above the values under LimitName, CurrentUsage and LimitAmount is wrong.

There seems to be a missing command and some other improvements which could be made to improve the README.md file for new developers.

1) Remove newline characters

newline characters are not needed in bash:

cd ./deployment
chmod +x ./run-unit-tests.sh  \n
./run-unit-tests.sh \n

Should either be removed:

cd ./deployment
chmod +x ./run-unit-tests.sh
./run-unit-tests.sh

Or if intending to be a single multiline command:

cd ./deployment && \
  chmod +x ./run-unit-tests.sh && \
  ./run-unit-tests.sh

2) Include missing command

It is not clear which dist folder this refers to:

aws s3 cp ./dist/ s3://my-bucket-name/limit-monitor/latest/ --recursive --exclude "*" --include "*.template" --acl bucket-owner-full-control --profile aws-cred-profile-name

Would be better to include the cd command to make it clear:

cd ../source/lambda/services/limitreport
aws s3 cp ./dist/ s3://my-bucket-name/limit-monitor/latest/ --recursive --exclude "*" --include "*.template" --acl bucket-owner-full-control --profile aws-cred-profile-name

Or if we can pass through relative paths this would be clearer:

aws s3 cp ../source/lambda/services/limitreport/dist/ s3://my-bucket-name/limit-monitor/latest/ --recursive --exclude "*" --include "*.template" --acl bucket-owner-full-control --profile aws-cred-profile-name

Hi,
I setup the aws limit checker for my aws account but when our sns topic limit reached,
I don't get any email from this.
What is the issue here, I am missing something or there is some bug with the code

(ap-southeast-1)
If you need more details I can share wtih you

New spoke deploy.

INFO [ERROR]AccessDeniedException: User: arn:aws:sts::[account id]:assumed-role/limit-monitor-limitCheckStack-GQH-LimitMonitorRole-1HLZ084R28ABB/limit-monitor-limitCheckStack-LimitMonitorFunction-ENB26B3ETQB8 is not authorized to perform: servicequotas:GetAWSDefaultServiceQuota with an explicit deny

IAM role attached to Lambda function has servicequotas:GetAWSDefaultServiceQuota for all resources so I don't understand why it's getting an explicit deny.

There are situations where the limits are perfectly ok and there is no need to increase them. Example: EC2 on-demand instances

Would like to have an ack button in the Slack notification that will disable future alerts for that particular limit.