satyrius / gonx Goto Github PK
View Code? Open in Web Editor NEWNginx access log parser for Golang
License: MIT License
Nginx access log parser for Golang
License: MIT License
func NewChain(reducers ...Reducer) *Chain {
chain := &Chain{
reducers: reducers,
}
for _, r := range reducers {
if f, ok := interface{}(r).(Filter); ok {
chain.filters = append(chain.filters, f)
} else {
chain.reducers = append(chain.reducers, r)
}
}
return chain
}
Is this your original mean?
I wan't to put the results to ElasticSearch. Is there any method that can be used to get all fields of an Entry?
Nginx's $http_x_forwarded_for
can contain multiple IP addresses (in cases where a reverse proxy is used). In such cases the format used is [IP Address]([Comma][Space][IP Address])+
. I guess since this value is gotten from the X-Forwarded-For header.
Gonx is able to parse [IP Address]([Comma][Space][IP Address])+
but not [IP Address]([Comma][IP Address])+
.
Hello,
I've noticed recently that running gonx-based applications on more CPUs doesn't improve the performance. So, I've created a benchmark processing 1 million records and measured the results on four multi-core VM instances on Google Cloud Platform โ https://cloud.google.com/compute/docs/machine-types#highcpu
Also I've compared 'em with another benchmark results utilizing glow library.
Hello,
I'm trying to process entries with fileds separated by either spaces (one or more) or tabs.
Is it possible to somehow specify such cases in the format string? If not, I could try to implement some FieldsParser (based on https://golang.org/pkg/strings/#Fields). Would you be interested in such pull request?
Thanks
Is there any way to get the row line ?
I was playing around, however I really need the row number, and currently thats not possible since the reader is no sync.
Hello @satyrius, thanks for this amazing project. Hope you're doing well
We(as a team) have created a GitHub org called daftcreations for OSS projects and for educations content on YouTube. All members were colleagues once. The thought of creating org was from using day to day life tools, most of the themes were the open-source or some part of them. I, Myself, have contributed to many OSS projects and most of the members of org members as well. We got a team of awesome Designers, UI, UX, Frontend, Backend, ChainOps(blockchain DevOps like myself) and DevOps people. And that inspires us to contribute our expertise to the open-source world, and make it better. This is the way.
We jointly created our first project called gcps which is gcloud profile switcher, which was initiated by @naman2706 and other devs, we're just on the edge of releasing it.
So, We wanted you to donate(transfer) this awesome project to our org. Our amazing golang dev @aman00323 and @naman2706 will look into it and try to give it a new life with the latest golang knowledge.
Fin
Hi!
It would be great to add parameter "startReadFromLine" in NewNginxReader function. To start reading log file from some line. 10x
My nginx.conf
includes following definition of log format (sorry for bad formatting):
log_format log_with_cookie '[$time_local]\t'
'$remote_addr\t'
'$geoip_country_code\t'
'$msec\t'
'$remote_user\t'
'"$request"\t'
'$status\t'
'$body_bytes_sent\t'
'$cookie_uid\t'
'$uid_got\t'
'$uid_set\t'
'$cookie_u\t'
'"$http_accept_language"\t'
'$cookie_c\t'
'"$http_user_agent"\t'
'"$http_referer"\t'
'"$args"\t'
'"$sent_http_set_cookie"';
Because of use of regexp.QuoteMeta
in NewParser
function, tab characters are not properly recognized during parsing.
Is there some particular reason why it is done this way?
MapReduce
it is reading lines.Read()
isn't returning it.Any Help on that
Hi, I have a log format like this:
$remote_addr [$time_local] "$host$request_uri" $status
currently gonx
will parse the format as regexp:
^(?P<remote_addr>[^ ]*) \[(?P<time_local>[^]]*)\] "(?P<host>[^$]*)\$request_uri" (?P<status>[^ ]*)
the $request_uri that after $host is not parsed as a regexp.
Hi, there are multiple go vet
errors in gonx/reader_bench_test.go
:
https://goreportcard.com/report/github.com/satyrius/gonx#go_vet
The Sprintf
doesn't do anything in these cases because it's not assigned to any variable. You might have meant just Printf
, or maybe you don't need to print anything at all. Just wanted to let you know.
Hello,
(Love your project, great work !!)
I am not able to use the gonx.NewReader on a compress/gzip Reader.
It seems to not be able to read and return 0 lines.
Did you ever tried? Do you have any idea of the problem?
**Nevermind, it work flawlessly, you can delete this issue **
Thanks
Hey @satyrius,
Let me know your thoughts on this!
I would like it to take it further
If there is a line in nginx log with empty user agent it fails with a message like this:
panic: Access log line 'xx.xx.xx.xx - [21/Apr/2014:06:25:10 +0000] "GET / HTTP/1.0" 200 15389 "-" ""' does not match given format '^(?P<remote_addr>[^ ]+) - [(?P<time_local>[^]]+)] "(?P[^ ]+) (?P[^ ]+) (?P[^"]+)" (?P[^ ]+) (?P[^ ]+) "(?P[^"]+)" "(?P[^"]+)"$'
I try figure out why
file := strings.NewReader(`37.219.240.1 [05/Nov/2014:02:15:54 +0200] test.fi "GET / HTTP/1.1"`)
format := `$remote_addr [$time_local] $host "$request" `
Ok.
file := strings.NewReader(`37.219.240.1 [05/Nov/2014:02:15:54 +0200] test.fi "GET / HTTP/1.1" 200`)
format := `$remote_addr [$time_local] $host "$request" $status`
Don't work.
From other side
file := strings.NewReader(`37.219.240.1 [05/Nov/2014:02:15:54 +0200] test.fi "GET / HTTP/1.1" "200"`)
format := `$remote_addr [$time_local] $host "$request" "$status"`
OK?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.