Hi, the log doesn't stop generating after n lines when logging to file:

docker run -it -v ${PWD}/log:/log --rm mingrammer/flog -t log -w -o /log/apache.log -b 10485760 -p 1048576 -n 10 -d 1s

Is there a way to stop at n lines when logging to file?

Thanks.

As a flog user, I'd like to be able to use it to send test logs to a log management system like Splunk or Graylog. Presently, I have to output to a file, or try and netcat the output to a specific endpoint, which seems rather cumbersome. Adding the ability to directly output to a given endpoint (like an IP/hostname) would shorten the amount of steps I need to take to send flogs logs off.

I am try use a grok with your log generator, but the combined pattern diverge of the apache log format:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

My grok

HTTPD_COMMONLOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} \[%{TIMESTAMP_ISO8601:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} %{QS:referrer} %{QS:agent}

flog.go creates new file each time Generate is called. It'll be nice if option param can take more info about writing- to append to existing file or create new file

Thanks for great work!

RFC3164 was superseded by RFC5424, which is a slightly modified format as follow:

<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG

For example:
<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

Could you please add support for generating logs in this format?

Per RFC5425#section-4.3.1, RFC6587#section-3.4.1, messages can be prefixed with an octet count.

Other programs such as telegraf expect this to be present, so it would be a nice feature to optionally enable.

See also this fluentd reference to it as well.

just does not work
i typing "flog --help" and nothing happened

If you set the sleep time with decimal value (ex. -s 0.5), it does not sleep for 500 milliseconds, because of time.Duration is int64. So I should change the sleep unit to milliseconds from seconds.

Current Dockerfile:

FROM golang

ENV CGO_ENABLED=0
ENV GO111MODULE=on

WORKDIR /go/src/flog

COPY go.mod go.sum ./
RUN go mod download

COPY . ./
RUN go build -o /bin/flog

FROM scratch
COPY --from=0 /bin/flog /bin/flog
ENTRYPOINT ["flog"]

But we can build binary file directly like this:

FROM golang

ENV CGO_ENABLED=0
ENV GO111MODULE=on

WORKDIR /go/src/flog

COPY . ./
RUN go build -ldflags="-w -s" -o /bin/flog

FROM scratch
COPY --from=0 /bin/flog /bin/flog
ENTRYPOINT ["flog"]

Is there any problem to build binary file directly? I know it's not essential or critical point, but I want to make it simpler. Thanks for reading.

From Go 1.11, Go supports go module in its toolchain. So we can build a Docker image without dep installation.

Hi! Trying to install flog using Homebrew fails with the follwing error message:

Error: An exception occurred within a child process:
  NoMethodError: undefined method `path' for nil:NilClass
Did you mean?  paths

Thanks for this project. I get issue when try install with brew:

brunowego@ally [docker-monitoring] brew tap mingrammer/flog                                                                                                           18:33:54  ☁  feature/ansible ☂ ⚡ ✚ ✭
brunowego@ally [docker-monitoring] brew install flog                                                                                                                  18:34:02  ☁  feature/ansible ☂ ⚡ ✚ ✭
Error: No available formula with the name "flog"
==> Searching for a previously deleted formula (in the last month)...
Warning: homebrew/core is shallow clone. To get complete history run:
  git -C "$(brew --repo homebrew/core)" fetch --unshallow

Error: No previously deleted formula found.
==> Searching for similarly named formulae...
Error: No similarly named formulae found.
==> Searching taps...
==> Searching taps on GitHub...
Error: No formulae found in taps.

I am using latest within a Kubernetes set up,

          command: ["flog"]
          args: ["-t", "log", "-w", "-o", "/var/log/test-log.log", "-n", "1000", "-s", "10", "-l"]

In order to keep the container running, I am using loop so that flog keeps getting executed. I noticed when using loop, no matter what sleep number I specify it does not seem like it's being upheld when using loop. I see events coming in hundreds of thousands as the program is not pausing, ( not upholding the sleep flag).

If I remove loop, the number and sleep cycle I specify seems to be working, however the container dies and rebuilds and it is in a crash loop.

I'm hoping loop can be fixed to be properly aligned with sleep.

Hi, I have tried your repo in windows 10 it works fine but In Linux, the output command does not generate any file at the location.

Should change the split-by to rotate or rotating. And will support one more type, by datetime.

Occasionally a multi-word string will be generated for the APPLICATION field.

If this happens, the spaces should be replaced with a hypen or something, as the APPLICATION field only accepts a single word.

This is awesome tool. I really like this. Congrats.

What you think about a parameter for delay?

Hi, thanks for this great project!

It would be really nice if one could generate a flog with random stacktraces of the most common programming languages, such as Java, Python, Ruby, etc. This would be incredibly useful to test the configuraiton of data collection software such as Fluentd or Logstash.

Unfortunately I do not know and Go (yet), so I cannot provide a PR.

Just a question.
when? log forwarding can available. Or how can I log forwarding

My command

flog -f apache_common -n 1 -s 10 --loop

Logs are generated non stop, even if I use a very large number like 100, 1000.

edit: I am seeing this on Linux with the latest release, and on MacOS when I build master from source.

I should add change log section in the README.md

When running the "go get -u github.com/minigrammer/flog" command to install the program, I get these errors:

go/src/github.com/mingrammer/flog/log.go:38:3: undefined: gofakeit.StatusCode
go/src/github.com/mingrammer/flog/log.go:53:3: undefined: gofakeit.StatusCode
go/src/github.com/mingrammer/flog/log.go:114:3: undefined: gofakeit.StatusCode
go/src/github.com/mingrammer/flog/log.go:129:3: undefined: gofakeit.StatusCode

I am running this on a CentOS 7 server.

Hi.

I was testing your log generator, in order to quickly generate log messages for testing purpose. In particular, I am interested in messages formatted with the RFC 3164 format.

I don't want to sound pedantic but I believe the message generated by flog is in a wrong format.
According to the RFC 3164, section 5.4 Examples, the log format should be like the following:

<34>Oct 11 22:14:15 mymachine su: 'su root' failed for user1 on /dev/pts/8

Where <34> is the priority of the log message, followed by the timestamp in the format of
Mmm dd hh:mm:ss (see section 4.1.2 of the aforementioned RFC). The one you are
using is in the RFC 3339 format, which is fine for log messages which comply to the RFC 5424 (section 6.2.3).

A message generated by flog is instead presented in the following format (generated with
flog -f rfc3164 -n 10):

2018-10-05T11:23:56+02:00 Murazik5260 Decentralized[844]: Overriding the system won't do anything, we need to quantify the neural GB array!

The timestamp format is not correct and the priority is missing. As far as I can see from the source code, messages in the RFC3164 format are generated through the function NewRFC3164Log inside the log.go package.

Would it be possible to rewrite the timestamp in the following way?

t := time.Now()
str := fmt.Sprintf("%d %s %02d %d:%d:%d", t.Year(), t.Month().String()[:3], t.Day(), t.Hour(), t.Minute(), t.Second())
fmt.Println(str)

I can also submit a pull request, if you like.

I tried the following options and neither one are generating 1 line or 1 byte in the beginning, then wait 100 seconds before logging 1 line or byte again:
flog -t log -n 1 -s 100 -w -l

or

flog -t log -b 1 -s 100 -w -l

Am I using it right? I want to generate 1 line every 100 seconds.

Thanks!

want to support custom format logging

Good day;

I need to test some database query performance in my log management stack and in order to do so I need to generate bulks of log messages that are 6 months old ( fake timestamp ), is it possible to do so with this tool?

Sorry if I've asked this question in the wrong place.

Thanks in advance.

• https://docs.nginx.com/nginx/admin-guide/monitoring/logging/
• https://nginx.org/en/docs/http/ngx_http_log_module.html

Logs generated for RFC5424 only compliant about 40% of the time.

Sample set of 100 logs generated with flog: https://regex101.com/r/nMlGtT/1

Testing using the Fluent Bit syslog-rfc5424 parser.

[PARSER]
    Name        syslog-rfc5424
    Format      regex
    Regex       ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*?)\]|-)) (?<message>.+)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On

all logs print same time

$ docker run -it --rm mingrammer/flog -d 1 -l
93.189.152.55 - - [04/Mar/2020:10:04:37 +0000] "DELETE /sexy HTTP/2.0" 503 26738
63.112.22.61 - - [04/Mar/2020:10:04:37 +0000] "POST /e-markets/clicks-and-mortar/networks HTTP/2.0" 200 15059
94.57.233.71 - wisozk5474 [04/Mar/2020:10:04:37 +0000] "HEAD /cross-media/exploit/roi HTTP/1.0" 200 2615
196.41.177.239 - - [04/Mar/2020:10:04:37 +0000] "PATCH /visionary/enhance HTTP/1.1" 201 6863
6.196.104.137 - - [04/Mar/2020:10:04:37 +0000] "DELETE /seize/user-centric/roi HTTP/1.1" 302 21633
131.223.76.168 - lueilwitz6155 [04/Mar/2020:10:04:37 +0000] "DELETE /viral/e-enable HTTP/2.0" 405 15423
52.143.147.88 - cole7581 [04/Mar/2020:10:04:37 +0000] "PUT /rich/portals/visualize HTTP/2.0" 205 17807

sleep option can be used but that prints the future timestamp

$ docker run -it --rm mingrammer/flog -d 1 -l -s 1
140.10.32.50 - volkman7687 [04/Mar/2020:10:05:00 +0000] "PUT /virtual/eyeballs/mindshare/wireless HTTP/1.1" 404 818
131.118.198.64 - - [04/Mar/2020:10:05:01 +0000] "GET /transition/b2c/transition/e-business HTTP/1.1" 501 13686
17.242.209.250 - - [04/Mar/2020:10:05:02 +0000] "HEAD /architect/matrix/action-items HTTP/1.0" 302 20037
234.255.50.102 - berge2025 [04/Mar/2020:10:05:03 +0000] "GET /transform/dynamic/e-services HTTP/2.0" 403 682
197.65.196.232 - cummings8717 [04/Mar/2020:10:05:04 +0000] "PATCH /proactive/visionary/e-markets HTTP/2.0" 203 18018
148.146.11.174 - - [04/Mar/2020:10:05:05 +0000] "PATCH /clicks-and-mortar/cutting-edge HTTP/1.0" 201 660
207.148.236.65 - - [04/Mar/2020:10:05:06 +0000] "POST /niches HTTP/1.0" 504 22918

whenever log lines are printed it should re-read the time instead of keeping same.

Hello,

I was unable to get the sleep function to work using the provided example:
flog -t gz -o log.gz -n 3000 -s 10

The code would execute and generate the log.gz archive, but not update. Same outcome on both Mac and Win10.

Background

I would like to test how fluent bit's throughput varies depending on the size of the log line.. i have seen its throuput reduced as the line size of a log file increases..

If this feature is there in this tool the i would be able to do the same

I am trying to use Flog to generate dummy logs in OpenShift environment. But the Flog pod keeps going into CrashLoopBackOff state.

[core@master2 ~]$ oc get po |grep flog
flog-5c659f86b4-dsq5c      0/1     CrashLoopBackOff   1571       5d13h

I am unable to find any error message in the logs as well. Could someone please help me debug the issue here ?

NOTE: I'm not a 'go' programmer by trade and I love the diagrams python module you created...thanks for your contributions!

While using your tool to generate some logs while performance testing a log shipper setup, I noticed the timestamp in the emitted logs (while using the -l forever option), slowly became further and further behind the current time. This made it appear that the log shipper was lagging behind.

A basic test I performed was as follows:

Run the flog generator

./flog -f json -o /tmp/json.log -t log -d 1ms -l

In another terminal, every few seconds compare the last log line with the current date:

tail -1 /tmp/json.log && date

What you'll notice is the timestamp in the last log line slowly gets behind from the current time.

I was able to make a minor modification to flog.go which corrected the problem:

diff --git a/flog.go b/flog.go
index 896026b..728a1dc 100644
--- a/flog.go
+++ b/flog.go
@@ -38,7 +38,7 @@ func Generate(option *Option) error {
        if option.Forever {
                for {
                        time.Sleep(delay)
-                       log := NewLog(option.Format, created)
+                       log := NewLog(option.Format, time.Now())
                        _, _ = writer.Write([]byte(log + "\n"))
                        created = created.Add(interval)
                }

EDIT: Your implementation allows for better precision (using delays less than 1ms), it's just the slow skew that I'm not sure can be completely accounted for since the load generator requires some wall time to execute as well.

Hi!
I was going to try your interesting app, but following the instructions, it seems that it is not possible to install it via brew.
This is what I got after running brew tap mingrammer/flog:

==> Tapping mingrammer/flog
Cloning into '/usr/local/Homebrew/Library/Taps/mingrammer/homebrew-flog'...
remote: Enumerating objects: 39, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 39 (delta 0), reused 0 (delta 0), pack-reused 36
Unpacking objects: 100% (39/39), done.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/mingrammer/homebrew-flog/flog.rb
flog: Calling bottle :unneeded is disabled! There is no replacement.
Please report this issue to the mingrammer/flog tap (not Homebrew/brew or Homebrew/core):
  /usr/local/Homebrew/Library/Taps/mingrammer/homebrew-flog/flog.rb:6

Error: Cannot tap mingrammer/flog: invalid syntax in tap!

Do you know what we can do to install it in Mac using brew?

Best

Please add a feature to output in json format.

Think about this scenario, a common scenario, where the logging is added one by one with configurable frequency.

I see on time.go that apache format is like:
Apache = "02/Jan/2006:15:04:05 -0700"

However generated logs use numeric month:

$ flog
191.134.83.211 - emmerich3317 [10/12/2019:13:36:06 +0100] "PUT /revolutionary/optimize/supply-chains" 500 2483
252.148.56.22 - morissette4712 [10/12/2019:13:36:06 +0100] "PUT /mesh/granular" 205 28065
27.230.215.188 - shanahan1170 [10/12/2019:13:36:06 +0100] "GET /cutting-edge/integrated" 416 11029
144.8.217.152 - - [10/12/2019:13:36:06 +0100] "HEAD /synergies/redefine/functionalities" 404 4465
152.124.206.88 - - [10/12/2019:13:36:06 +0100] "POST /engage/unleash" 406 4335
175.154.38.72 - - [10/12/2019:13:36:06 +0100] "DELETE /interfaces/maximize" 204 11974
158.86.63.249 - - [10/12/2019:13:36:06 +0100] "PATCH /extend/innovate/optimize/content" 500 12895
102.218.94.253 - lehner1112 [10/12/2019:13:36:06 +0100] "HEAD /redefine/harness/metrics/enable" 500 2719

Using brew installed 0.3.1 on mac

Hi,
there is no Docker image with tag/version 0.4.0. on docker hub:

https://hub.docker.com/r/mingrammer/flog/tags

Can you please push it?

Hi @mingrammer, can you update the image version to the last? Thanks!

Hi, i have a proposal, that -d and -s parameter can support time string.
For example,

500ms for 500 millisceond
1s for one second
1m for one minute

And then, we can control the interval duration more precisely