lucagrulla / cw Goto Github PK

Has anyone thought to use this tool to view cloudwatch metrics combining with something like gnuplot or https://github.com/holman/spark ?

The app doesn't support Daylight saving properly

Current purge strategy for log id cache is naive.

Hit the panic scenario which Tail method was called multiple times, or you have a little more time delay so that the Ticker fired more signal to trigger the page handler, which then try to close the closed chanel.

Using the Ticker is a little problematic here as it has no exit but rely on the chanel being closed and also does not allow caller to reconsume the Tail method.

I have used a fix of using the removed Ticker, it just worked fine.

A feature request for this tool is the ability to tail multiple groups.

This would be extremely useful for Lambda, where an "app" is composed of many Lambda functions, and each function has its own log groups.

I'd like to be able to cw tail /aws/lambda/*:

Would you consider a PR with the feature?

Describe the bug
On ubuntu (18.04) server I have installed aws cli and it works without issue, but with cw I have region issue. When try to run cw ls groups getting this error: could not find region configuration. Also I tried run it like this cw ls groups --region="us-east-2", but in that case getting this error no valid providers in chain. Deprecated.

To Reproduce
Steps to reproduce the behavior:

1. install cw to ubuntu 18.04 server
2. cw ls groups --region="us-east-2"
3. cw ls groups

Expected behavior
I expect to get output like this, what works fine on my local Mac machine
/aws/rds/cluster/item-importer-sneki-test-cluster/error
/aws/rds/cluster/snekitest-cluster/error

Screenshots
If applicable, add screenshots to help explain your

• problem.
  

Desktop (please complete the following information):

• OS: Ubuntu 18.04

Configure goreleaser for scoop.sh based installation

Describe the bug
stdin pipe in currently won't work

To Reproduce
echo "mygroup:*" | cw tail

Expected behavior
shoul tail "mygroup".

Other utilities like https://github.com/jorgebastida/awslogs colorise the output, it would be nice to have the same feature here

Marks-MacBook-Pro[13:32:55] cw --profile=zjdevacct --region=us-east-2 ls streams "API-Gateway-Execution-Logs_72oi6rv36b/Alpha"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13d6c8b]

goroutine 8 [running]:
github.com/lucagrulla/cw/cloudwatch.(*CW).LsStreams.func1.1(0x1, 0x0, 0xc0000c3db8)
/Users/lucagrulla/Projects/cw/cloudwatch/lsstreams.go:24 +0x2b
sort.insertionSort_func(0xc0000c3e38, 0xc000487c00, 0x0, 0x14)
/usr/local/Cellar/go/1.11.5/libexec/src/sort/zfuncversion.go:12 +0x93
sort.stable_func(0xc0000c3e38, 0xc000487c00, 0x32)
/usr/local/Cellar/go/1.11.5/libexec/src/sort/zfuncversion.go:167 +0x51
sort.SliceStable(0x146ace0, 0xc000487be0, 0xc0000c3e38)
/usr/local/Cellar/go/1.11.5/libexec/src/sort/slice.go:31 +0x106
github.com/lucagrulla/cw/cloudwatch.(*CW).LsStreams.func1(0xc00000b460, 0xc000030a00, 0xc0000556b8)
/Users/lucagrulla/Projects/cw/cloudwatch/lsstreams.go:23 +0x81
github.com/aws/aws-sdk-go/service/cloudwatchlogs.(*CloudWatchLogs).DescribeLogStreamsPagesWithContext(0xc00000c270, 0x1a74188, 0xc0000260b0, 0xc000183320, 0xc000061820, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/service/cloudwatchlogs/api.go:1670 +0x116
github.com/aws/aws-sdk-go/service/cloudwatchlogs.(*CloudWatchLogs).DescribeLogStreamsPages(0xc00000c270, 0xc000183320, 0xc000061820, 0x0, 0x0)
/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/service/cloudwatchlogs/api.go:1642 +0x90
github.com/lucagrulla/cw/cloudwatch.(*CW).LsStreams.func2(0xc000061810, 0xc000183320, 0xc000061820)
/Users/lucagrulla/Projects/cw/cloudwatch/lsstreams.go:37 +0x49
created by github.com/lucagrulla/cw/cloudwatch.(*CW).LsStreams
/Users/lucagrulla/Projects/cw/cloudwatch/lsstreams.go:36 +0xf1

PS C:\Users\Narsi Nallamilli> Get-Host | Select-Object Version
Version

5.1.18362.752

PS C:\Users\Narsi Nallamilli> iwr -useb get.scoop.sh | iex
iex : At line:1 char:1

• #Requires -Version 5

This script contains malicious content and has been blocked by your antivirus software.
At line:1 char:26

• iwr -useb get.scoop.sh | iex

•                      ~~~
  

  • CategoryInfo : ParserError: (:) [Invoke-Expression], ParseException
  • FullyQualifiedErrorId : ScriptContainedMaliciousContent,Microsoft.PowerShell.Commands.InvokeExpressionCommand

I don't know if this is a mistake or not... I report you just in case.

lucagrulla/cw/cw -> lucagrulla/cw ?

$ brew info cw
lucagrulla/cw/cw: stable 2.1.1

$ brew upgrade cw
==> Upgrading 1 outdated package:
lucagrulla/cw/cw 2.1.0 -> 2.1.1
==> Upgrading lucagrulla/cw/cw
==> Downloading https://github.com/lucagrulla/cw/releases/download/v2.1.1/cw_2.1.1_Darwin_x86_64.tar.gz

Is your feature request related to a problem? Please describe.

One of the amazing features of CloudWatch is the ability to log to JSON, and then build intelligant searches based upon the JSON output.

This means logs often endup as long single lines of JSON output:

{"error": "this was an error", "level": "error", "data": {"key1": 1, "key2": 2}}

To make this better cloud watch takes the JSON and pretty prints it. cw outputs it as the line was put into cloudwatch.

It would be nice if cw detected if a line was JSON or not, and if JSON would pretty print it for easier reading. Possibly as a flag to enable/disable.

Describe alternatives you've considered
I'm not sure if there is an easy "fix". Normally I'd pipe the output to jq but that doesn't work since some automatic lambda and other input is not JSON.

Additionally pretty printing the JSON to cloudwatch doesn't work becasue then cloudwatch doesn't consider a JSON blob as a single JSON event but instead each line is a different text event.

Thanks for nice tool 😃
I want to use the --profile flag like below to specify other aws credential profile.

$ cw --profile second ls groups

[default]
aws_access_key_id = ****
aws_secret_access_key = ****

[second]
aws_access_key_id = ****
aws_secret_access_key = ****

Which package manager to target?
Look into snapcraft

the --following feature randomly repeats logs.

Actually it's not even installed properly

The help text for the 2 tail params is misleading in the identification of which parts or optional

Is your feature request related to a problem? Please describe.
see #45

Describe the solution you'd like
Support authentication with 2FA.

Additional context
Accounts with 2FA enabled won't work.

Brew recipe works only for Mac.

Currently depending on:
goreleaser/goreleaser#428

Snapcraft security sandbox blocks access to the config files.

Describe the bug
I'm running the latest cw release (3.2.2) and there is an superfluous linebreak after every log line.

To Reproduce
Happens to me with all tail commands

Expected behavior
There shouldn't be an extra newline between each log line

Screenshots

Desktop (please complete the following information):

• OS: macos 10.15.1
• Terminal: zsh 5.7.1 in iTerm2 3.3.7

Best regards,
Felix

• After downloading cw, I was not able to execute any of the command listed in the documentation, including cw ls groups. I ultimately want to run cw tail.

• I'm getting a The security token included in the request is invalid. error.

• I was allowed full access to Cloudwatch logs, and my aws creds are correct since I was able to use aws cli.

We provide our IAM permissions based on needs, and I can't figure out the permissions needed to use cw.

It'd be great if there was some guidance for the needed IAM roles somewhere. In the documentation somewhere like in the Readme, or as a cw -list-permissions or a cw -list-iam command.

I went through the codebase and found some mention of ec2 roles, but again I can't tell which ones.

Thank you!

Update: Rotating my access keys allowed me access. Why though

Describe the bug
Current linuxbrew manifest doesn't support ARM architecture.

Is your feature request related to a problem? Please describe.
I'm working on a script that starts an ECS task and then tails the log from Cloudwatch. I wait until the task is actually in RUNNING state before kicking off cw, but there's still a few seconds between that state transition and the creation of the log group, so cw
errors out with "The specified log group does not exist."

Describe the solution you'd like
tail has the following options:

-F     same as --follow=name --retry
...
--retry keep trying to open a file if it is inaccessible

Basically I'd like to see -F replicated in cw. If the log group doesn't exist, retry until it does. Emit STDERR messages if the initial log group check fails, if the log group appears, and if the log group disappears.
My script already kills cw when the task has completed, so there's no concern about the log group never appearing.

Describe alternatives you've considered
I can test if the log group exists myself, and wait to invoke cw until it does exist.

Additional context
As inspiration for the type of output I'd like to see here's tail's STDERR output when following a file that does not exist on invocation, appears while tail is running, and is then deleted while tail is running:

$ tail -F /tmp/logfile > /dev/null
tail: cannot open '/tmp/logfile' for reading: No such file or directory
tail: '/tmp/logfile' has appeared;  following new file
tail: '/tmp/logfile' has become inaccessible: No such file or directory

Describe the bug
the long format of the --start flag should be begin rather than start for consistency

Sometimes, when the application is running multiple times, the error "Rate exceeded" is thrown and the application exits. It would be a good idea to have a parameter to continue, or add a delay, when that error happens.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x80 pc=0x142790f]

goroutine 21 [running]:
main.fetchLatestVersion.func1(0xc0001f2cc0)
	/Users/lucagrulla/Projects/cw/main.go:113 +0x3f
created by main.fetchLatestVersion
	/Users/lucagrulla/Projects/cw/main.go:110 +0x58

This might be because the network request to check for a new version is failing?

Once installed it tells you to add to source this file
/usr/local/share/zsh/site-functions/_cw

instead of

/usr/local/share/zsh/site-functions/cw

==> Caveats

In order to get cw completion,
  [bash] you need to install `bash-completion` with brew.
  OR
  [zsh], add the following line to your ~/.zshrc:
    source /usr/local/share/zsh/site-functions/_cw

Bash completion has been installed to:
  /usr/local/etc/bash_completion.d

zsh functions have been installed to:
  /usr/local/share/zsh/site-functions

Describe the bug
My AWS profile has enabled MFA, When I ssue the following command, I get Authentication errors as below:

cw ls groups

panic: AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.
goroutine 1 [running]:
github.com/aws/aws-sdk-go/aws/session.Must(...)
/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/aws/session/session.go:281
github.com/lucagrulla/cw/cloudwatch.New(0xc00005d3a0, 0xc00005d3b0, 0xc0000946e0, 0x0)
/Users/lucagrulla/Projects/cw/cloudwatch/client.go:34 +0x284
main.main()
/Users/lucagrulla/Projects/cw/main.go:176 +0x251

To Reproduce
Steps to reproduce the behavior:

1. Which command
   cw ls groups
2. Which flags/arguments are used
   ls

Expected behavior
Expected to list groups

Screenshots
N/A

Desktop (please complete the following information):

• MacOS 10.14.2
• Terminal (bash)

Additional context
N/A

CW doesn't cap the number of tailed streams and hit the API limits of 50.

Describe the bug

The new AWS CLI v2 provides Single Sign On:

https://aws.amazon.com/blogs/developer/aws-cli-v2-is-now-generally-available/

But doesn't seem to play well with cw:

$ cw ls -p dev groups
no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

To Reproduce

$ aws configure sso
SSO start URL [https://yourcompany.awsapps.com/start]:
SSO Region [ap-southeast-2]:
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:

https://device.sso.ap-southeast-2.amazonaws.com/

Then enter the code:

CQCS-BFPZ
There are 8 AWS accounts available to you.
Using the account ID <ACCOUNT_ID>
There are 3 roles available to you.
Using the role name "default"
CLI default client Region [ap-southeast-2]:
CLI default output format [None]:
CLI profile name [default]: default

$ aws s3 ls
<LIST OF BUCKETS>

$ cw ls -p dev groups
no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Describe the bug
if you install cs by go get github.com/lucagrulla/cw or go get -u github.com/lucagrulla/cw you get version 1.7.2 and will be stuck at that version.

To Reproduce
just do go get -u github.com/lucagrulla/cw
I have "go version go1.13.4 darwin/amd64" installed , on a mac with latests macos, in zsh,

Expected behavior
able to get the latest version of cw which is 3.2.3

Screenshots
NA

Desktop (please complete the following information):

• OS: MacOS 10.15.2
• Terminal zsh, in iTerm

Is your feature request related to a problem? Please describe.
We develop locally with localstack. I would like use the same tooling locally as on aws. I.e. I would like point cw to the url of my localstack and tail logs via cw as if it where the real aws.

I currently see no way how to point cw to localstack.

Describe the solution you'd like
Add a command line flag "--endpoint-url" analogous to the flag of aws cli with same name. So the call to tail logs from localstack would be:

./cw --endpoint-url=http://localhost:4566 tail -f /aws/lambda/input-processor-lambda

Describe alternatives you've considered
No alternatives considered yet.

Additional context
No additional information yet.

Describe the bug
The package name that you're using for the debian package, cw conflicts with this package name in Ubuntu : https://launchpad.net/ubuntu/bionic/amd64/cw

As a result, if a user has both packages they replace each other depending on which one is installed last.

To Reproduce
Steps to reproduce the behavior:

1. On an Ubuntu system run sudo apt install cw
2. run wget https://github.com/lucagrulla/cw/releases/download/v3.3.0/cw_amd64.deb && sudo dpkg -i cw_amd64.deb

Expected behavior
The existing cw package wouldn't be replaced and the CloudWatch tool would have a non conflicting name

Desktop (please complete the following information):

• OS: Ubuntu 18.04
• Terminal : any

Thanks for 2.0.0 release 🎉

Log not found

$ date -u
Wed Oct 17 03:32:52 UTC 2018

It was correctly outputted by the way used until. (FYI: I am in the time zone of JST so some logs below are displayed at +0900)

$ cw -p my-profile tail -ft /ecs/dev-my-server
2018-10-17T12:32:42 - [2018-10-17 03:32:42.533]    0 ms -- /health/check [:GET]   200 [] "-" "-" ::ffff:10.0.1.137
2018-10-17T12:32:49 - [2018-10-17 03:32:49.065]    0 ms -- /health/check [:GET]   200 [] "-" "-" ::ffff:10.0.0.174

But it is not output as a new way.

$ cw -p my-profile tail -ft /ecs/dev-my-server \* 5m

It was not found even if full log stream name was specified.

$ cw -p my-profile tail -ft /ecs/dev-my-server ecs/my-server/bcc9fc24-7184-4064-aad7-921dd0bf32fc 5m
No such log stream(s).

$ cw -p my-profile ls streams /ecs/dev-my-server | grep bcc9fc24
ecs/my-server/bcc9fc24-7184-4064-aad7-921dd0bf32fc

panic: send on closed channel

About the case where wrong args is specified, I sometimes encounter panic error.

$ cw -p my-profile tail -ft /ecs/dev-my-server 2m
No such log stream(s).
panic: send on closed channel

goroutine 55 [running]:
github.com/lucagrulla/cw/cloudwatch.(*CW).Tail.func4(0xc000056680, 0xc000056501, 0x1012bd2)
	/Users/lucagrulla/Projects/cw/cloudwatch/tail.go:150 +0x175
github.com/aws/aws-sdk-go/service/cloudwatchlogs.(*CloudWatchLogs).FilterLogEventsPagesWithContext(0xc0001ae050, 0x1fb1128, 0xc0000240b0, 0xc0001000f0, 0xc000056580, 0x0, 0x0, 0x0, 0xc0001a8b01, 0xc0001000f0)
	/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/service/cloudwatchlogs/api.go:2288 +0x116
github.com/aws/aws-sdk-go/service/cloudwatchlogs.(*CloudWatchLogs).FilterLogEventsPages(0xc0001ae050, 0xc0001000f0, 0xc000056580, 0x0, 0x0)
	/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/service/cloudwatchlogs/api.go:2260 +0x90
github.com/lucagrulla/cw/cloudwatch.(*CW).Tail.func5(0xc0001cc2d0, 0xc00004d480, 0xc0001a8b40, 0xc0001c45d8, 0x0, 0xc00004d450, 0xc000024ad7, 0xc0001ae058, 0xc000056580)
	/Users/lucagrulla/Projects/cw/cloudwatch/tail.go:175 +0x235
created by github.com/lucagrulla/cw/cloudwatch.(*CW).Tail
	/Users/lucagrulla/Projects/cw/cloudwatch/tail.go:171 +0x40d

Is your feature request related to a problem? Please describe.
Not able to connect. Getting send request failed

Describe the solution you'd like
Add support for proxy

Describe the solution you'd like
In aws logs CLI get-log-events has a --limit flag to define how many logs you want to receive.

https://github.com/goreleaser/godownloader

From time to time we hit AWS req limit.

First of all, thanks for the project! It works amazingly well.

Today I realized it can be integrated with fzf quite easily too. Here is my little snippet that I am using:

cw tail -f "$(cw ls groups | fzf)"

Probably could be a useful doc addition. Or even a feature inside cw itself.

Cheers!

Describe the bug
when a Tail is completed the trigger channel should be removed from the round robin scheduler

I am satisfied with the features of cw already, but if you want to more better to improve, I propose the Time options feature implementation.

I used awslogs until recently, but the problem with this tool is that it depends on python 2.x version. (In that respect, cw implemented with golang is very smart!)

awslogs has a feature called Time options, which is useful for viewing the most recent log.

• https://github.com/jorgebastida/awslogs#time-options
  • --start=2m
  • --start=1h

The tail command of cw can specify UTC start time as the third argument, but in my environment the timezone is GMT+09:00, which is a bit confusing.

$ cw -p my_other_profile tail -f -t /ecs/my-server ecs/my-server/f276cdff-375d-4c65-a4d6-29afd025f221 2018-10-01T10:13:00
2018-10-01T19:13:13 - [2018-10-01 10:13:13.488]    0 ms -- /health/check [:GET]   200 [] "-" "-" ::ffff:10.0.0.174
2018-10-01T19:13:17 - [2018-10-01 10:13:17.282]    0 ms -- /health/check [:GET]   200 [] "-" "-" ::ffff:10.0.1.137
2018-10-01T19:14:13 - [2018-10-01 10:14:13.518]    0 ms -- /health/check [:GET]   200 [] "-" "-" ::ffff:10.0.0.174
2018-10-01T19:14:17 - [2018-10-01 10:14:17.312]    0 ms -- /health/check [:GET]   200 [] "-" "-" ::ffff:10.0.1.137

This is because JST(GMT+09:00) time is output with -t flag although I have specified time in UTC.
Therefore it is convenient to specify like Time option.

$ cw tail --follow --timestamp my-log-group my-log-stream-prefix 2m

or

$ cw tail --follow --timestamp --start 2m my-log-group my-log-stream-prefix

When running in -f follow mode tail.go polls cloudwatch every ~205ms for logs and checks against a cache to ignore seen events. Most of the time the cache is hit and the event correctly ignored, but when the cache is emptied (every ~70 secs) we get a repeat of any events that occurred at the same time as the last event (for events with timestamp diff > 1ms this manifests as logging the last event over and over again). This isn't ideal.

Hi. Nice project, but I've hit a bug.

These streams exist:

$ cw ls streams /var/log/stuff/uat
access/instance1
access/instance2
catalina/instance1
catalina/instance2
localhost/instance1
localhost/instance2

But I can't tail some of them:

$ cw tail /var/log/stuff/uat localhost/instance2
No such log stream(s).
panic: send on closed channel

goroutine 42 [running]:
github.com/lucagrulla/cw/cloudwatch.(*CW).Tail.func4(0xc00001c780, 0xc00001c701, 0x412da2)
	/Users/lucagrulla/Projects/cw/cloudwatch/tail.go:149 +0x175
github.com/aws/aws-sdk-go/service/cloudwatchlogs.(*CloudWatchLogs).FilterLogEventsPagesWithContext(0xc00000c2a8, 0x7f6299adb130, 0xc000026120, 0xc00007d590, 0xc00001c6c0, 0x0, 0x0, 0x0, 0xc00016f101, 0xc00007d590)
	/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/service/cloudwatchlogs/api.go:2288 +0x116
github.com/aws/aws-sdk-go/service/cloudwatchlogs.(*CloudWatchLogs).FilterLogEventsPages(0xc00000c2a8, 0xc00007d590, 0xc00001c6c0, 0x0, 0x0)
	/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/service/cloudwatchlogs/api.go:2260 +0x90
github.com/lucagrulla/cw/cloudwatch.(*CW).Tail.func5(0xc00007e910, 0xc0000593c0, 0xc00016f1d0, 0xc000026ff8, 0x0, 0xc000059390, 0xc000026b47, 0xc00000c2b0, 0xc00001c6c0)
	/Users/lucagrulla/Projects/cw/cloudwatch/tail.go:174 +0x235
created by github.com/lucagrulla/cw/cloudwatch.(*CW).Tail
	/Users/lucagrulla/Projects/cw/cloudwatch/tail.go:170 +0x3b3

Unless I specify the date.

$ cw tail /var/log/stuff/uat localhost/instance2 2018-11-09T14:52
09-Nov-2018 14:52:17.320 INFO [main] org.apache.catalina.core.ApplicationContext.log ContextListener: contextInitialized()
09-Nov-2018 14:52:17.320 INFO [main] org.apache.catalina.core.ApplicationContext.log SessionListener: contextInitialized()
09-Nov-2018 14:52:17.321 INFO [main] org.apache.catalina.core.ApplicationContext.log ContextListener: attributeAdded('StockTicker', 'async.Stockticker@62379589')
09-Nov-2018 14:52:20.921 INFO [main] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
09-Nov-2018 14:52:21.208 INFO [main] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

This stream hasn't had any activity for a few days, and that's the only difference I can think of between the broken streams and the working ones.

I'm using the latest release.

$ cw --version
2.1.1