heroku / log-shuttle Goto Github PK

HTTP log transport.

Makefile 1.33% Go 98.32% Dockerfile 0.30% Procfile 0.05%

log-shuttle's Introduction

Log Shuttle

Log-shuttle is an open source UNIX program that delivers messages from applications and daemons to log routers and processors via HTTPs.

One of the motivations behind log-shuttle is to provide a simpler form of encrypted & authenticated log delivery. Using HTTPs & Basic Authentication is simpler than the techniques described in RFC5425. TLS transport mapping for Syslog requires that you maintain both client & server certificates for authentication. In multi-tenant environments, the maintenance of certificate management can be quite burdensome.

Log-shuttle accepts input from stdin in a newline (\n) delimited format.

When using log-shuttle with logplex it is recommended that you spawn 1 log-shuttle per logplex token. This will isolate data between tokens and ensure a good QoS.

When using log-shuttle with either Amazon's Kinesis or Amazon's Cloud Watch Logs services all the details for the service are supplied in the -logs-url (or $LOGS_URL env variable). See the Amazon Endpoints documentation for supported regions and hostnames. See the Kinesis and Cloud Watch Logs sections of this document.

To block as little as possible, log-shuttle will drop outstanding batches if it accumulates > -back-buff amount.

Kinesis

log-shuttle sends data into Kinesis using the PutRecords API call. Each Kinesis record is encoded as length prefixed rfc5424 encoded logs as per rfc6587 (this is the same format logplex accepts). One record per log line.

Log-shuttle expects the following encoding of -logs-url when using Amazon Kinesis:

```
https://<AWS_KEY>:<AWS_SECRET>@kinesis.<AMAZON_REGION>.amazonaws.com/<STREAM NAME>
```

Kinesis Caveats

Things that should be handled better/things you should know:

AWS_SECRET, AWS_KEY, AMAZON_REGION & STREAM NAME need to be properly url encoded.
log-shuttle assumes a 200 response means everything is good. Kinesis can return a 200, meaning the http request was good, but include per record errors in the response body.
The maximum number of records in a PutRecords requests is 500, so set the batch size no higher than 498 (500 - 2 records for possible drops / lost).
Logplex max line length is 10k, Kinesis max record size is 50k of base64 encoded data. A -max-line-length of somewhere less than 37500 should work for Kinesis w/o causing errors.
Kinesis does not support the -gzip option as that option compresses the body of the request.
Even with -kinesis-shards, no guarantees can be made about writing to unique shards.

CloudWatch Logs

log-shuttle sends logs to CloudWatch Logs using the PutLogEvents API call. Each log line is a seperate event and delivered in the same order received by log-shuttle.

log-shuttle uses the aws-sdk-go library to determine the AWS credentials it will use, starting CloudWatch Logs sequence token and to sign requests, but does not otherwise use the aws-sdk-go's clients.

log-shuttle expects the following encoding of -logs-url to use Amazon CloudWatch Logs:

```
https://logs.<AMAZON_REGION>.amazonaws.com/<log group name>/<log stream name>
```

Cloudwatch Caveats

Things that should be handled better/things you should know:

log-shuttle doesn't do any special handling around service limits atm: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html
log-shuttle needs more testing against CloudWatch to ensure it handles errors and limits better
PutLogEvents has a hard upper limit of 5 requests per second. If log-shuttle's input / settings causes > 5 batches per second to be created this limit could be exceeded. Modulating batch size and wait duration would be needed to fix this on a case by case basis.
Really long lines are not split like they are with logplex

Install

go get -u github.com/heroku/log-shuttle/...

After that $GOPATH/bin/log-shuttle should be available.

Making Debs

Requires:

dpkg (see also brew install dpkg)
go 1.6+

make debs

Docker

There is a Makefile target named docker that can be used to build a docker image.

Hacking on log-shuttle

Fork the repo, hack, submit PRs.

Testing

go test -v ./...

Submitting Code

Open an issue on GitHub.
Keep changes in a feature branch
Submit PR

License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

log-shuttle's People

Contributors

Stargazers

Watchers

log-shuttle's Issues

Kinesis records only use a single partition key

This limits the scalability of Kinesis, as each partition key is mapped to only one shard.

http://docs.aws.amazon.com/kinesis/latest/dev/key-concepts.html

Update Docs

For instance: Purpose of the Makefile

Log-Shuttle 0.13.0/0.13.1 Release not hosted on Docker Hub

I am attempting to use log-shuttle with kinesis and want to make use multiple shards but 0.13.x has not been built into a container and hosted on Docker Hub.

I've built 0.13.1 and self-hosted for the time being but could we get a release pushed up to Docker Hub?

Thanks!
Gabe

Version Mismatch

The version shown in the repository is 0.1.5. The version shown with the pre-built log-shuttle is 0.2.

[root@localhost ~]# log-shuttle -version                                                                             
0.2

Retry Harder

If the length of batch queue is less than some watermark, retry a batch on io errors.

Can't build on OS X 10.9

This is the 2nd golang project I've ever tried to build, so I'm probably doing it wrong.

go1.3.3 installed via homebrew. I added the GOPATH bits because it complained when it didn't have one, otherwise these are the commands from the README.

seth@tiny:~/src/heroku $ go version
go version go1.3.3 darwin/amd64
seth@tiny:~/src/heroku $ mkdir $HOME/go
seth@tiny:~/src/heroku $ export GOPATH=$HOME/go
seth@tiny:~/src/heroku $ export PATH=$PATH:$GOPATH/bin
seth@tiny:~/src/heroku $ git clone https://github.com/heroku/log-shuttle.git
Cloning into 'log-shuttle'...
remote: Counting objects: 1725, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 1725 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1725/1725), 426.76 KiB | 372.00 KiB/s, done.
Resolving deltas: 100% (868/868), done.
Checking connectivity... done.
seth@tiny:~/src/heroku $ go get github.com/tools/godep
seth@tiny:~/src/heroku $ cd log-shuttle/
seth@tiny:~/src/heroku/log-shuttle [master] $ godep go build ./...
main.go:8:2: cannot find package "github.com/heroku/log-shuttle/shuttle" in any of:
    /usr/local/Cellar/go/1.3.3/libexec/src/pkg/github.com/heroku/log-shuttle/shuttle (from $GOROOT)
    /Users/seth/src/heroku/log-shuttle/Godeps/_workspace/src/github.com/heroku/log-shuttle/shuttle (from $GOPATH)
    /Users/seth/go/src/github.com/heroku/log-shuttle/shuttle
godep: go exit status 1

Libraries that emit stuff to stdout/stderr/syslog kinda suck.

The user of the lib should decide what to do with errors.

Add a kafka outlet

We have kinesis, let's add kafka.

Deliver messages from `logger` when reading from unix sockets

Moving from ryandotsmith/log-shuttle#20.

Timestamp later

record the timestamp early on (in reader), but do the formatting later in the pipeline, either in batcher or in outlet (which was the original place).

New reader/batcher doesn't update batch.fill metrics

This was missing when I rolled the batcher into the reader. :-(

GC Churn

I realized that any work I do on GC churn doesn't matter than much because each time we're reading from STDIN we're allocating a new buffer ([]byte or string). If we're serious about working around GC churn then was need to re-use these buffers instead, if possible.

Log-shuttle binary not copied into docker container for 0.15.0 and 0.16.0 releases

Hi!
I'm attempting to run the 0.16.0 release of the log-shuttle in the container tag 0.16.0 on docker hub.

When I run either the 0.15.0 or 0.16.0 release containers docker says the log-shuttle binary is not located at the entry point:

docker run -it --rm heroku/log-shuttle:0.16.0
docker: Error response from daemon: Container command '/bin/log-shuttle' not found or does not exist..

Looking at the Dockerfile there is a comment that mentions that the binary is copied in to the container after being pre-built. https://github.com/heroku/log-shuttle/blob/master/Dockerfile#L5

From the error it looks like the log-shuttle binary didn't make it into the container or the location changed and the entry point needs to be updated.

Thanks for the help!

an alltime.processed or similar would be nice

from @dpiddy

Use Logplex-Loss-Count instead of Logshuttle-Drops as drop header

/cc @archaelus

Option to report metrics to librato

Right now we only have a syslog reporter.

It would be nice to support another metrics destination and the one we use mostly is librato.

Probably a small patch with a set of command line options / switch.

github.com/pebbe/util.IsTerminal failing on Go 1.5rc1

spew | log-shuttle -verbose -skip-verify -logs-url=http://...snip... -logplex-token=`cat run.token`
panic: Not implemented

goroutine 1 [running]:
github.com/heroku/log-shuttle/Godeps/_workspace/src/github.com/pebbe/util.IsTerminal(0x820648008, 0x820622640)
    /Users/sratnakumar/go/src/github.com/heroku/log-shuttle/Godeps/_workspace/src/github.com/pebbe/util/isterminal.go:15 +0x65
main.useStdin(0x2710)
    /Users/sratnakumar/go/src/github.com/heroku/log-shuttle/cmd/log-shuttle/main.go:32 +0x23
main.main()
    /Users/sratnakumar/go/src/github.com/heroku/log-shuttle/cmd/log-shuttle/main.go:222 +0x167
^C2015/08/10 17:14:38 Received interrupt; sending annotation to librato...
2015/08/10 17:14:38 ERROR sending to librato: Librato config not set
make: *** [run-stream] Error 2

Disordered Logs

Hi. We're getting a rearrangement of our logs when we view them in Logplex. (Note the timestamps)

2015-01-23T21:58:24.342756+00:00 shinyapps[shuttle]: SEND {
2015-01-23T21:58:24.336726+00:00 shinyapps[shuttle]: "values": "\"domain\"\n293.052725\n1627.343375" 
2015-01-23T21:58:24.338810+00:00 shinyapps[shuttle]:  "name": "scale/y",
2015-01-23T21:58:24.343985+00:00 shinyapps[shuttle]: "inputMessages": [] 
2015-01-23T21:58:24.343986+00:00 shinyapps[shuttle]: }
2015-01-23T21:58:24.340089+00:00 shinyapps[shuttle]: "command": "update" 
2015-01-23T21:58:24.338812+00:00 shinyapps[shuttle]:  "type": "csv",
2015-01-23T21:58:24.340087+00:00 shinyapps[shuttle]:  "ggvis_command": {
2015-01-23T21:58:24.342760+00:00 shinyapps[shuttle]:  "plotId": "BarPlot",
2015-01-23T21:58:24.342758+00:00 shinyapps[shuttle]:  "custom": {
2015-01-23T21:58:24.343981+00:00 shinyapps[shuttle]:  "TSTitle": "<i>Accommodation</i> tourist spend in Auckland",
2015-01-23T21:58:24.338791+00:00 shinyapps[shuttle]:  "plotId": "BarPlot",
2015-01-23T21:58:24.338818+00:00 shinyapps[shuttle]: } 
2015-01-23T21:58:24.341426+00:00 shinyapps[shuttle]: "command": "update" 
2015-01-23T21:58:24.338820+00:00 shinyapps[shuttle]: }
2015-01-23T21:58:24.341428+00:00 shinyapps[shuttle]: } 
2015-01-23T21:58:24.343979+00:00 shinyapps[shuttle]:  "errors": [],
2015-01-23T21:58:24.343980+00:00 shinyapps[shuttle]: "values": {
2015-01-23T21:58:24.338817+00:00 shinyapps[shuttle]: ] 
2015-01-23T21:58:24.340088+00:00 shinyapps[shuttle]:  "plotId": "TSPlot",
2015-01-23T21:58:24.342762+00:00 shinyapps[shuttle]: } 
2015-01-23T21:58:24.341425+00:00 shinyapps[shuttle]:  "plotId": "Map",
2015-01-23T21:58:24.343977+00:00 shinyapps[shuttle]: SEND {
2015-01-23T21:58:24.340092+00:00 shinyapps[shuttle]: }
2015-01-23T21:58:24.341429+00:00 shinyapps[shuttle]: }
2015-01-23T21:58:24.342764+00:00 shinyapps[shuttle]: }
2015-01-23T21:58:24.342759+00:00 shinyapps[shuttle]:  "ggvis_command": {
2015-01-23T21:58:24.343984+00:00 shinyapps[shuttle]: },
2015-01-23T21:58:24.343983+00:00 shinyapps[shuttle]: "BarTitle": "Product distribution of tourism spend in Auckland"

I'm under the impression that log shuttle is merely sending unordered HTTP requests, so perhaps this is expected with a series of rapid-fire requests like this.

For our use case, however, we expect rapid bursts of logging, but not necessarily long periods of heavy log writing. Is it possible to tune the configuration (e.g. using only 1 "Batcher" or "Outlet") to increase the likelihood that logs get through in order?

Even if there's not a recommended best practice, I'm happy to try out whatever hypotheses you think might help.

Thanks!

Timestap log lines on input not output

Log-shuttle is responsible for setting a syslog timestamp on log entries. It currently does this when formatting output (time.Now(). etc) - but this means the stamp given may be way off from the time it received the log entry on stdin.

The timestamp should instead be taken as close to when the log entry is received as possible.

Race condition on LogLineReader.ReadLogLines and Shuttle.Land

Shuttle.Land closes shared channels while Readers may still be trying to write to them. This causes a panic at runtime.

Shuttle.Land needs to notify and/or synchronize with all its readers, stopping them before closing the channel.

/cc @sclasen

return error codes

Opening this to have a convo with @fabiokung

Based on a conversation elsewhere ...

I don't understand what you mean by replacing those "Fatal(f) uses with returning errors at some point." That is how log-shuttle returns errors to the caller by using log.Fatalf(msg), which calls os.Exit(1). This returns a non-zero exit code to the calling program.

Docker images from 0.16.3 onward cannot be run

Hey folks,

I pulled the latest set of images for log-shuttle today to get started on setting it up for some log consumption and noticed that running the image fails.

➜  log-shuttle git:(master) docker run --rm -ti heroku/log-shuttle:0.17.1 --help 
Unable to find image 'heroku/log-shuttle:0.17.1' locally
0.17.1: Pulling from heroku/log-shuttle
97dd1b22539e: Already exists 
cfebb3372630: Pull complete 
29c6e1666acb: Pull complete 
Digest: sha256:35fe8120e32e6d280da424c807a7d0617ab65fa5f0102b0b65c5926e53a1f465
Status: Downloaded newer image for heroku/log-shuttle:0.17.1
standard_init_linux.go:211: exec user process caused "no such file or directory"

Some digging showed that the binary is present in the image:

➜  log-shuttle git:(master) docker run --rm -ti --entrypoint "" heroku/log-shuttle:0.17.1 ls -l /bin/log-shuttle
-rwxrwxr-x    1 root     root       7769490 Mar  6  2018 /bin/log-shuttle

Rebuilding the binary and image with CGO_ENABLED=0 fixes this problem as it's most likely related to the binary being built on a non-alpine system and then moved into one. I can't inspect the values but there were changes to the env vars in the travis build from v0.16.2 to v0.16.3 so it's possible the ENV var was present and then got removed?

See below:

➜  log-shuttle git:(4476c45) CGO_ENABLED=0 make docker 
rm -rf .docker_build
GOOS=linux GOARCH=amd64 go build -v -o .docker_build/log-shuttle -ldflags "-X main.version=v0.16.3" ./cmd/log-shuttle
github.com/heroku/log-shuttle
github.com/heroku/log-shuttle/cmd/log-shuttle
docker build -t heroku/log-shuttle:0.16.3 ./
Sending build context to Docker daemon  10.93MB
Step 1/4 : FROM gliderlabs/alpine:3.3
 ---> 70d584ebde68
Step 2/4 : RUN apk-install ca-certificates
 ---> Using cache
 ---> a9e18b6f5b1b
Step 3/4 : COPY .docker_build/log-shuttle /bin/log-shuttle
 ---> 0dc0b5b13f15
Step 4/4 : ENTRYPOINT ["/bin/log-shuttle"]
 ---> Running in 7be4e4d41cc7
Removing intermediate container 7be4e4d41cc7
 ---> dd0fd8deb130
Successfully built dd0fd8deb130
Successfully tagged heroku/log-shuttle:0.16.3
make clean-docker-build
make[1]: Entering directory '/home/dewald.viljoen/Dev/go/src/github.com/heroku/log-shuttle'
rm -rf .docker_build
make[1]: Leaving directory '/home/dewald.viljoen/Dev/go/src/github.com/heroku/log-shuttle'
➜  log-shuttle git:(4476c45) docker run --rm -ti heroku/log-shuttle:0.16.3 --help  
Usage of /bin/log-shuttle:
  -appname string
        The app-name field for the syslog header. (default "token")
  -back-buff int
        Number of batches to buffer before dropping. (default 50)

There are two possible solutions for this.

We put CGO_ENABLED=0 in the build step for the docker-image version of the log-shuttle binary.
We build log-shuttle in an Alpine container and copy it over to the final container.

I'm happy to submit a PR for the former. The latter would be a much larger change to the build processes and so I'm not likely well placed to execute on that one.

Thoughts?

AWS Kinesis delivery requires static credentials

Hey folks,

Currently log-shuttle only support Kinesis streams using static credentials embedded in the Kinesis URL at startup. We have a requirement to run log-shuttle with more dynamic credential sources, specifically the EC2 instance profile endpoint.

To support this we would need to change KinesisFormatter to more closely resemble the CloudWatchLogsFormatter as that is already using the aws-sdk-go for its signing process.

The benefits would be log-shuttle supporting the regular credential provider chain so it would be able to reach keys from Env vars, config files, instance profiles etc.