It would be nice if this utility could support the --region and --profile options like the aws-cli does.
This would enable us to use this tool in an environment with many accounts, regions and profiles.

E.g. I have several profiles for several accounts, and each account has one profile for readonly and admin access.

cwlog-minder --dry-run delete-empty-log-streams --profile my-admin would allow me to use this tool much more easily on the command line.

The current CloudFormation template does not work with AWS GovCloud.

It returns the following error message:

Partition "aws" is not valid for resource "arn:aws:lambda:us-gov-west-1:{acct-id}:function:aws-cloudwatch-delete-empty-log-streams". (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: xxxx)

If you're open to a pull request, I'd be happy to make the small alteration necessary to the CloudFormation template to support this deployment scenario.

It would be nice to have a command to delete empty log groups. This seems useful after running delete_empty_log_streams.

The below seems to work for me (need to clean up)

def _delete_empty_log_groups(
    group: dict, purge_non_empty: bool = False, dry_run: bool = False
):
    now = datetime.utcnow().replace(hour=0, minute=0, second=0, microsecond=0)
    log_group_name = group["logGroupName"]
    retention_in_days = group.get("retentionInDays", 0)
    if not retention_in_days:
        log.info(
            "skipping log group %s as it has no retention period set",
            log_group_name,
        )
        return

    kwargs = {
        "logGroupName": log_group_name,
        "orderBy": "LastEventTime",
        "descending": False,
        "PaginationConfig": {"PageSize": 50},
    }

    for response in cw_logs.get_paginator("describe_log_streams").paginate(
        **kwargs
    ):
        if len(response["logStreams"]) == 0:
            log.info(
                "Deleting group %s", log_group_name,
            )
            cw_logs.delete_log_group(logGroupName=log_group_name)

Thanks for such a great python script!

The lambda delete_empty_log_groups is not included in the Cloudformation and is not run as part of the emptying of the log streams (or on a separate schedule). After a while there is a buildup of empty log groups that causes the delete_empty_log_streams lambda to run for a long time and thousands of needless invocations.
Should the delete_empty_log_groups be included in the Cloudformation (with a schedule)?

Hi,

I am not receiving any output when running set-log-retention in dry-run mode. I wasn't brave enough to try it without the dry-run flag. I also have a lot of log groups with no retention set.

My guess is that this is related to https://stackoverflow.com/questions/43109355/logging-setlevel-is-being-ignored and pull request #1 would fix this.

This looks like a great utility, but I was noticing that delete-empty-log-groups didn't seem to actually delete any log groups. When looking at the code, there doesn't seem to be anything that actually performs the delete (it just logs that it would happen)

I was running log minder on a big collection of log steams. Since there was no log output I thought the tool was stuck or not working. When I turned on Debug logging I discovered it was doing a lot of log deletion. No log messages were shown because they were set to debug level.

Maybe a summary log message every 10 seconds can help show progress so that users don't think the script is not working.

See pull requesT #25 for the fix of this issue.

Hi,

Thanks for this great tool!

I have a few suggestions and issues:

1. It will be easier to use a single line to deploy with cloud formation:

aws cloudformation deploy --stack-name=aws-cloudwatch-log-minder --template-file=./cloudformation/aws-cloudwatch-log-minder.yaml --capabilities CAPABILITY_IAM --parameter-overrides LambdaS3Bucket=<my bucket> CFNCustomProviderZipFileName=<zipfile location>

2. Add cloudformation parameter to change the default retention period
3. List all the required permissions for the cloudformation stack creation
4. The public s3 bucket that contains the zipped version is not publicly accessible - I had to manually build it (using make) and deploy it to my own bucket
5. When trying to build manually, the make command failed because you pass the encoding param in setup.py to the open method (line 10), and apparently it is not supported. Removing this param solved the issue for me.

side note - this is the first time I dealt with python code :-)

@mvanholsteijn any thoughts?

From https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/logs.html#CloudWatchLogs.Client.describe_log_streams

storedBytes (integer) --
The number of bytes stored.
IMPORTANT: On June 17, 2019, this parameter was deprecated for log streams, and is always reported as zero. This change applies only to log streams. The storedBytes parameter for log groups is not affected.

The code uses this parameter in delete_empty_log_streams.py.
I am observing some storedBytes to have a value other than zero when the stream is empty.

Hi, I just want to inform that the link https://binxio-public-eu-central-1.s3.eu-central-1.amazonaws.com/lambdas/aws-cloudwatch-log-minder-0.4.1.zip is currently not accessible so CloudFormation stack creation fails.

Thank you.

Great job with the tool!

It would be great if the tool tags the log groups automatically with Name tag.

see guardian/cloudwatch-logs-management#19

I used this as a workaround:

aws logs describe-log-groups | jq -r '.logGroups[].logGroupName' | xargs -t -I% -n1 aws logs tag-log-group --tags Name=% --log-group-name=%

I have been running an application for more than 3 years without the log minder. There are >100k log streams in 1 log group. With the rate limits of AWS Cloudwatch and the default maximum log minder duration of 5 minutes it would take weeks to clean up all the log streams.

For now I've been running log-minder from my laptop and leaving it on throughout the night. This has cleaned my log group in 30 hours or so.

Maybe log-minder can run continuously when it notices there is more work to do. That would be easier than running the CLI manually.

It would be nice if we can filter out some Log Groups that do not have any retention and have a tag like - NoExpire= true as part of set-log-retention command.

We have some scenarios were we don't want to set retention for certain log groups, so filtering them out and setting default retention for other would be great.

cwlog-minder --dry-run set-log-retention --days 30 --filter tagKey:tagValue