satyrius / gonx Goto Github PK

I try figure out why

file := strings.NewReader(`37.219.240.1 [05/Nov/2014:02:15:54 +0200] test.fi "GET / HTTP/1.1"`)
format := `$remote_addr [$time_local] $host "$request" `

Ok.

file := strings.NewReader(`37.219.240.1 [05/Nov/2014:02:15:54 +0200] test.fi "GET / HTTP/1.1" 200`)
format := `$remote_addr [$time_local] $host "$request"  $status`

Don't work.

From other side

file := strings.NewReader(`37.219.240.1 [05/Nov/2014:02:15:54 +0200] test.fi "GET / HTTP/1.1" "200"`)
format := `$remote_addr [$time_local] $host "$request"  "$status"`

OK?

Hello,

(Love your project, great work !!)
I am not able to use the gonx.NewReader on a compress/gzip Reader.
It seems to not be able to read and return 0 lines.

Did you ever tried? Do you have any idea of the problem?
**Nevermind, it work flawlessly, you can delete this issue **

Thanks

I wan't to put the results to ElasticSearch. Is there any method that can be used to get all fields of an Entry?

Hi!
It would be great to add parameter "startReadFromLine" in NewNginxReader function. To start reading log file from some line. 10x

Hello,

I've noticed recently that running gonx-based applications on more CPUs doesn't improve the performance. So, I've created a benchmark processing 1 million records and measured the results on four multi-core VM instances on Google Cloud Platform — https://cloud.google.com/compute/docs/machine-types#highcpu

Also I've compared 'em with another benchmark results utilizing glow library.

Here are the resulting charts:

• I was trying to parse some logs with this beautiful package, but I get EOF.
• Even though I have debug inside package MapReduce it is reading lines.
• But Read() isn't returning it.

Any Help on that

Hey @satyrius,

• Currently, we are having multiple modules inside one repo which will not allow us to build in GitHub action.
• We can rewrite the benchmark as part of testing, and examples will be covered in the readme file.
• After that we can use the GitHub action for the test case.

Let me know your thoughts on this!

I would like it to take it further

Hi, there are multiple go vet errors in gonx/reader_bench_test.go:

https://goreportcard.com/report/github.com/satyrius/gonx#go_vet

The Sprintf doesn't do anything in these cases because it's not assigned to any variable. You might have meant just Printf, or maybe you don't need to print anything at all. Just wanted to let you know.

func NewChain(reducers ...Reducer) *Chain {
    chain := &Chain{
        reducers: reducers,
    }
    for _, r := range reducers {
        if f, ok := interface{}(r).(Filter); ok {
            chain.filters = append(chain.filters, f)
        } else {
            chain.reducers = append(chain.reducers, r)
        }
    }
    return chain
}

Is this your original mean?

Hi, I have a log format like this:

$remote_addr [$time_local] "$host$request_uri" $status

currently gonx will parse the format as regexp:

^(?P<remote_addr>[^ ]*) \[(?P<time_local>[^]]*)\] "(?P<host>[^$]*)\$request_uri" (?P<status>[^ ]*)

the $request_uri that after $host is not parsed as a regexp.

If there is a line in nginx log with empty user agent it fails with a message like this:

panic: Access log line 'xx.xx.xx.xx - [21/Apr/2014:06:25:10 +0000] "GET / HTTP/1.0" 200 15389 "-" ""' does not match given format '^(?P<remote_addr>[^ ]+) - [(?P<time_local>[^]]+)] "(?P[^ ]+) (?P[^ ]+) (?P[^"]+)" (?P[^ ]+) (?P[^ ]+) "(?P[^"]+)" "(?P[^"]+)"$'

Hello @satyrius, thanks for this amazing project. Hope you're doing well

We(as a team) have created a GitHub org called daftcreations for OSS projects and for educations content on YouTube. All members were colleagues once. The thought of creating org was from using day to day life tools, most of the themes were the open-source or some part of them. I, Myself, have contributed to many OSS projects and most of the members of org members as well. We got a team of awesome Designers, UI, UX, Frontend, Backend, ChainOps(blockchain DevOps like myself) and DevOps people. And that inspires us to contribute our expertise to the open-source world, and make it better. This is the way.

We jointly created our first project called gcps which is gcloud profile switcher, which was initiated by @naman2706 and other devs, we're just on the edge of releasing it.

So, We wanted you to donate(transfer) this awesome project to our org. Our amazing golang dev @aman00323 and @naman2706 will look into it and try to give it a new life with the latest golang knowledge.

Fin

Is there any way to get the row line ?

I was playing around, however I really need the row number, and currently thats not possible since the reader is no sync.

Hello,

I'm trying to process entries with fileds separated by either spaces (one or more) or tabs.

Is it possible to somehow specify such cases in the format string? If not, I could try to implement some FieldsParser (based on https://golang.org/pkg/strings/#Fields). Would you be interested in such pull request?

Thanks

My nginx.conf includes following definition of log format (sorry for bad formatting):

log_format log_with_cookie '[$time_local]\t'
         '$remote_addr\t'
        '$geoip_country_code\t'
        '$msec\t'
        '$remote_user\t'
        '"$request"\t'
        '$status\t'
         '$body_bytes_sent\t'
         '$cookie_uid\t'
         '$uid_got\t'
        '$uid_set\t'
        '$cookie_u\t'
        '"$http_accept_language"\t'
         '$cookie_c\t'
         '"$http_user_agent"\t'
         '"$http_referer"\t'
         '"$args"\t'
    '"$sent_http_set_cookie"';

Because of use of regexp.QuoteMeta in NewParser function, tab characters are not properly recognized during parsing.

Is there some particular reason why it is done this way?

Nginx's $http_x_forwarded_for can contain multiple IP addresses (in cases where a reverse proxy is used). In such cases the format used is [IP Address]([Comma][Space][IP Address])+. I guess since this value is gotten from the X-Forwarded-For header.

Gonx is able to parse [IP Address]([Comma][Space][IP Address])+ but not [IP Address]([Comma][IP Address])+.