teler-sh / teler Goto Github PK

Describe the bug

The Apache log format example in the official documentation looks like this:

log_format: |
  $remote_addr - $remote_user [$time_local] "$request_method $request_uri $request_protocol" $status $body_bytes_sent

but it will not match because it misses the last two fields. Instead, it should look like this:

log_format: |
  $remote_addr - $remote_user [$time_local] "$request_method $request_uri $request_protocol" $status $body_bytes_sent "$http_referer" "$http_user_agent"

$request format log must be separated again into 3 parts:

• Method name,
• Request URI, and
• Protocol

For example:

NGINX Ingress

• 127.0.0.1 - [127.0.0.1] - - [22/Jul/2020:00:34:14 +0000] "GET /_next/static/images/logo_ktbs_word_white-e12c3b97d3137c13e35f664a66b03096.png HTTP/2.0" 200 45088 "https://kitabisa.com/service-worker.js" "Mozilla/5.0 (Linux; Android 6.0; CPH1609) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36" 83 0.324 [kanvas-kanvas-app-prod-http] 10.0.0.100:9001 45088 0.324 200 eb344f13d3e1a7b72ef8ea9b37f157ad

Parser format as:

• $remote_addr - [$remote_addr] - - [$time_local] "$method $request_uri $protocol" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id

Returns a record of type Entry (which is customized map[string][string]):

&{map[
	body_bytes_sent:45088
	http_referer:https://kitabisa.com/service-worker.js
	http_user_agent:Mozilla/5.0 (Linux; Android 6.0; CPH1609) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36
	method:GET
	protocol:HTTP/2.0
	proxy_upstream_name:kanvas-kanvas-app-prod-http
	remote_addr:127.0.0.1
	req_id:eb344f13d3e1a7b72ef8ea9b37f157ad
	request_length:83
	request_time:0.324
	request_uri:/_next/static/images/logo_ktbs_word_white-e12c3b97d3137c13e35f664a66b03096.png
	status:200
	time_local:22/Jul/2020:00:34:14 +0000
	upstream_addr:10.0.0.100:9001
	upstream_response_length:45088
	upstream_response_time:0.324
	upstream_status:200
]}

Is your feature request related to a problem? Please describe.
Added external resources to check $request_uri on Google Hack Database from Exploit-DB.

Describe the solution you'd like
N/A.

Describe alternatives you've considered
N/A.

Additional context
Of course this will slow down analysis and alerting because it's calling the API.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
e.g.

    customs:
      - name: Large File Upload
        condition: AND
        rules:
          - element: body_bytes_sent
            pattern: \d{6,}

          - element: request_method
            pattern: P(OST|UT)

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Describe the bug

maybe this is not a bug, but I can't find a suitable classification of this issue.

the Dockerfile add a useless instruction RUN mkdir -p /app
just as the docker documentation says: If the WORKDIR doesn’t exist, it will be created, so there is no need to add this insctruction before WORKDIR /app, but if you indeed want to add it, it works fine!
so leave it or not? it's up to you.

And for linux platform, the go build produced binary wasn't added to .gitignore

To Reproduce

nothing

Your teler config file...
// Please redacted your token and/ other sensitive informations

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: [e.g. mac, linux]
• OS version: [uname -a]
• teler Version [teler --version]

Additional context
Add any other context about the problem here. Full output log is probably a helpful thing to add here.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Describe the bug

I've trouble getting the below logformat to work. Am I missing something? What

    log_format  main_timed  '$remote_addr - $remote_user [$time_local] "$request" '
                            '$status $body_bytes_sent "$http_referer" '
                            '"$http_user_agent" "$http_x_forwarded_for" '
                            '$request_time $upstream_response_time $pipe $upstream_cache_status';

Example row:

2.55.123.55 - - [03/Nov/2021:14:12:54 +0100] "GET /api/v1/settings HTTP/1.1" 200 208 "-" "iOS" "-" 0.605 0.605 . -

config:

log_format: |
  $remote_addr - $remote_user - [$time_local]
  "$request_method $request_uri $request_protocol" $status $body_bytes_sent
  "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $request_time $upstream_response_time $pipe $upstream_cache_status

Output

[INF] Analyzing...
[WRN] No logs analyzed, did you write log format correctly?
[INF] Done!

Environment (please complete the following information):

• OS: Mac
• teler Version 1.2.2

Summary

In the REAMDE, Docker image name is reference as teler, which is not recognized by Docker.

Fully qualified image name is kitabisa/teler and should be used in docker run ... commands.

Motivation

Have ready-to-use examples in README for Docker users.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Please could you add mattermost integration, configuration would be similar to slack but a server name would be needed as well as a token (or just the entire webhook which is a URL containing server and token)

I believe the format would be the same as for slack so it is just a matter of changing the post address accordingly?

This link may be useful to see what needs doing:-

https://docs.mattermost.com/developer/webhooks-incoming.html?highlight=webhook%20slack

Describe the bug

Cannot run make build. Exits with following errors:

To Reproduce

Steps to reproduce the behavior:

On a fresh Ubuntu install, install golang, clone the repo and run make build.

Your teler config file...
The default one.

// Please redacted your token and/ other sensitive informations

Expected behavior

It builds without errors

Screenshots

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: Ubuntu Bionic (18.04)
• OS version: Linux vpn-machine 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• teler Version N/A

Additional context
Add any other context about the problem here. Full output log is probably a helpful thing to add here.

• See screenshot

Additional context
It supports cross-platforms.

Describe the bug

I can't use custom excludes to reduce false positives.

To Reproduce

# Lighttpd default log format
log_format: |
  $remote_addr $host $remote_user [$time_local] "$request_method $request_uri $request_protocol" $status $body_bytes_sent "$http_referer" "$http_user_agent"

# Rules
rules:
  cache: true
  threat:
    excludes:
      - "79\\.0\\.10\\.100"
      - "^/favicon\\.ico"

But neither of them prevents this alert:

$ tail -n1000 lighttpd-access.log | teler -c ~/teler.yaml -o foobar.log
[04/Mar/2021:00:46:10 +0100] [79.0.10.100] [Directory Bruteforce] /favicon.ico

Expected behavior

A clear description of what you expected to happen: No output is expected.

Environment (please complete the following information):

• OS: GNU/Linux
• OS version: x86_64
• teler 1.1.0

Is your feature request related to a problem? Please describe.
Add whitelists in teler configuration file; which supports all threat categories.

Describe the solution you'd like
N/A

Describe alternatives you've considered
N/A

Additional context
Reducing false-positive results.

Is your feature request related to a problem? Please describe.
Instead of developing your own alerting sub-system, I think it might be preferable to use something that already supports dozens of backends. FalcoSideKick might be a good option

Describe the solution you'd like
See above

Summary

I want to integrate the Envoy access log with Teler, a lot of service mesh tooling use Envoy as the main proxy. Do you know any variable that teler support from the access log itself? So I can make a PR with the correct Envoy format.

Some recommended topics to cover:
N/A

Motivation

• Envoy used in service mesh like Istio

Describe the bug

fatal error: concurrent map iteration and map write

To Reproduce

Steps to reproduce the behavior:

run this a couple of times on a logfile with no issues:

tail /var/log/httpd/domains/example.com.log | ./teler -c teler.yml

result

fatal error: concurrent map iteration and map write

goroutine 27 [running]:
runtime.throw(0x86a1c18, 0x26)
	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/panic.go:1116 +0x6a fp=0xa4b8c74 sp=0xa4b8c60 pc=0x807821a
runtime.mapiternext(0xa4b8edc)
	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/map.go:853 +0x481 fp=0xa4b8cbc sp=0xa4b8c74 pc=0x80550b1
runtime.mapiterinit(0x85dc6a0, 0xa5084a0, 0xa4b8edc)
	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/map.go:843 +0x189 fp=0xa4b8cc8 sp=0xa4b8cbc pc=0x8054b49
ktbs.dev/teler/pkg/teler.Analyze(0xa50f440, 0xa40e120, 0x0, 0x8050ebf)
	/home/runner/work/teler/teler/pkg/teler/teler.go:32 +0x2c0 fp=0xa4b8f10 sp=0xa4b8cc8 pc=0x8547fd0
ktbs.dev/teler/internal/runner.New.func3.1(0xa5800a0, 0xa50f440, 0x0, 0xa502158, 0xa40e120)
	/home/runner/work/teler/teler/internal/runner/runner.go:72 +0x72 fp=0xa4b8fd8 sp=0xa4b8f10 pc=0x8551652
runtime.goexit()
	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/asm_386.s:1337 +0x1 fp=0xa4b8fdc sp=0xa4b8fd8 pc=0x80a40b1
created by ktbs.dev/teler/internal/runner.New.func3
	/home/runner/work/teler/teler/internal/runner/runner.go:69 +0x93

goroutine 1 [chan receive]:
github.com/satyrius/gonx.(*Reader).Read(0xa48df84, 0xa48df40, 0x0, 0x0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/reader.go:41 +0x3f
ktbs.dev/teler/internal/runner.New(0xa50f440)
	/home/runner/work/teler/teler/internal/runner/runner.go:126 +0x3fc
main.main()
	/home/runner/work/teler/teler/cmd/teler/main.go:20 +0x1f

goroutine 35 [syscall]:
os/signal.signal_recv(0x0)
	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/sigqueue.go:147 +0x12f
os/signal.loop()
	/opt/hostedtoolcache/go/1.14.15/x64/src/os/signal/signal_unix.go:23 +0x1a
created by os/signal.Notify.func1
	/opt/hostedtoolcache/go/1.14.15/x64/src/os/signal/signal.go:127 +0x33

goroutine 22 [chan receive]:
ktbs.dev/teler/internal/runner.New.func2(0xa5ae080, 0xa50c200, 0xa49c01c)
	/home/runner/work/teler/teler/internal/runner/runner.go:57 +0x2d
created by ktbs.dev/teler/internal/runner.New
	/home/runner/work/teler/teler/internal/runner/runner.go:56 +0x246

goroutine 23 [chan receive]:
ktbs.dev/teler/internal/runner.New.func3(0xa50c200, 0xa5800a0, 0xa50f440, 0x0, 0xa502158)
	/home/runner/work/teler/teler/internal/runner/runner.go:67 +0xa7
created by ktbs.dev/teler/internal/runner.New
	/home/runner/work/teler/teler/internal/runner/runner.go:66 +0x312

goroutine 24 [semacquire]:
sync.runtime_Semacquire(0xa42a0a8)
	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/sema.go:56 +0x36
sync.(*WaitGroup).Wait(0xa42a0a0)
	/opt/hostedtoolcache/go/1.14.15/x64/src/sync/waitgroup.go:130 +0x7c
github.com/satyrius/gonx.MapReduce.func1(0xa432180, 0x8775d10, 0xa40c090, 0xa5055c0, 0xa)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:67 +0x104
created by github.com/satyrius/gonx.MapReduce
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:26 +0x96

goroutine 25 [chan receive]:
github.com/satyrius/gonx.(*ReadAll).Reduce(0x8bc038c, 0xa5055c0, 0xa4321c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/reducer.go:23 +0x57
created by github.com/satyrius/gonx.MapReduce
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:73 +0xe1

goroutine 38 [runnable]:
regexp/syntax.(*compiler).inst(...)
	/opt/hostedtoolcache/go/1.14.15/x64/src/regexp/syntax/compile.go:173
regexp/syntax.Compile(0xa5ae400, 0xa5ae400, 0x2, 0x2)
	/opt/hostedtoolcache/go/1.14.15/x64/src/regexp/syntax/compile.go:84 +0x186
regexp.compile(0x868de83, 0x9, 0x80000d4, 0x85dc7e0, 0xa5086e0, 0x868d034)
	/opt/hostedtoolcache/go/1.14.15/x64/src/regexp/regexp.go:178 +0x9f
regexp.Compile(...)
	/opt/hostedtoolcache/go/1.14.15/x64/src/regexp/regexp.go:133
regexp.MustCompile(0x868de83, 0x9, 0xa5120e0)
	/opt/hostedtoolcache/go/1.14.15/x64/src/regexp/regexp.go:309 +0x39
ktbs.dev/teler/pkg/matchers.IsMatch(0x868de83, 0x9, 0xa5fc2ae, 0x3, 0xa550200)
	/home/runner/work/teler/teler/pkg/matchers/regex.go:10 +0x4b
ktbs.dev/teler/pkg/teler.Analyze(0xa50f440, 0xa502180, 0x1, 0x8050ebf)
	/home/runner/work/teler/teler/pkg/teler/teler.go:212 +0x1ae1
ktbs.dev/teler/internal/runner.New.func3.1(0xa5800a0, 0xa50f440, 0x0, 0xa502158, 0xa502180)
	/home/runner/work/teler/teler/internal/runner/runner.go:72 +0x72
created by ktbs.dev/teler/internal/runner.New.func3
	/home/runner/work/teler/teler/internal/runner/runner.go:69 +0x93

goroutine 7 [runnable]:
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:46 +0x66
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 8 [runnable]:
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:46 +0x66
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 9 [runnable]:
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:46 +0x66
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 10 [runnable]:
github.com/satyrius/gonx.(*Entry).SetField(...)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/entry.go:50
github.com/satyrius/gonx.(*Parser).ParseString(0xa40c090, 0xa5fc6c0, 0x115, 0x0, 0x0, 0x0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/parser.go:46 +0xf7
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:53 +0x97
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 11 [runnable]:
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:46 +0x66
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 12 [runnable]:
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:46 +0x66
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 13 [runnable]:
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:46 +0x66
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 16 [runnable]:
reflect.(*structType).FieldByName(0x85fa060, 0x868ca05, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/opt/hostedtoolcache/go/1.14.15/x64/src/reflect/type.go:1343 +0x23c
reflect.(*rtype).FieldByName(0x85fa060, 0x868ca05, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/opt/hostedtoolcache/go/1.14.15/x64/src/reflect/type.go:936 +0x6f
reflect.Value.FieldByName(0x85fa060, 0xa47e000, 0x199, 0x868ca05, 0x6, 0x199, 0x8050d14, 0xa)
	/opt/hostedtoolcache/go/1.14.15/x64/src/reflect/value.go:888 +0x66
ktbs.dev/teler/pkg/teler.Analyze(0xa50f440, 0xa47e000, 0x0, 0x8050ebf)
	/home/runner/work/teler/teler/pkg/teler/teler.go:23 +0xdb
ktbs.dev/teler/internal/runner.New.func3.1(0xa5800a0, 0xa50f440, 0x0, 0xa502158, 0xa47e000)
	/home/runner/work/teler/teler/internal/runner/runner.go:72 +0x72
created by ktbs.dev/teler/internal/runner.New.func3
	/home/runner/work/teler/teler/internal/runner/runner.go:69 +0x93

Additional context
Scrapping from nuclei-templates/cves/.

is this feature already in progress?

I already create it on my repo, free to feel for testing it and maybe add some suggestions.

if everything is good let me create the PR.here some PoC

Is your feature request related to a problem? Please describe.
add exporters in teler, so we can get a statistical data

Describe the solution you'd like
N/A

Describe alternatives you've considered
N/A

Additional context
N/A

Describe the bug

when I run the command tail -f /var/log/apache2/access.log | teler -c /var/www/html/teler/teler.yaml -x 25 but the response from teler analyzing is not complete

To Reproduce

Steps to reproduce the behavior:

tail -f /var/log/apache2/access.log | teler -c /var/www/html/teler/teler.yaml -x 25

# To write log format, see https://github.com/kitabisa/teler#configuration
log_format: |
  $remote_addr - [$remote_addr] $remote_user - [$time_local] 
  "$request_method $request_uri $request_protocol" $status $body_bytes_sent 
  "$http_referer" "$http_user_agent" $request_length $request_time 
  [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id
rules:
  cache: true
  threat:
    excludes:
      - "Common Web Attack"
      - "CVE"
      - "Bad IP Address"
      - "Bad Referrer"
      - "Bad Crawler"
      - "Directory Bruteforce"

    # It can be user-agent, request path, HTTP referrer, IP address and/or request query values parsed in regExp
    whitelists:
      # - "(curl|Go-http-client|okhttp)/*"
      # - "^/wp-login\\.php"
      # - "https://www\\.facebook\\.com"
      # - "192\\.168\\.0\\.1"

# prometheus:
  # active: false
  # host: "localhost"
  # port: 9099
  # endpoint: "/metrics"

alert:
  active: true
  provider: "slack"

notifications:
  slack:
    token: "xxxxxxxxxx"
    color: "#ffd21a"
    channel: "teler"

 # telegram:
   # token: "123456:ABC-DEF1234...-..."
   # chat_id: "-111000"

 # discord:
   # token: "NkWkawkawkawkawka.X0xo.n-kmZwA8aWAA"
   # color: "16312092"
   # channel: "700000000000000..."

Screenshots

https://drive.google.com/file/d/1vvbFHk9e-AVJgzUhIeaLynskmG24jYeT/view?usp=sharing

Environment (please complete the following information):

• OS: linux
• OS version: Linux kali 5.7.0-kali1-amd54 #1 SMP Debian 5.7.6-1kali2 (2020-07-01) x86_64 GNU/Linux
• teler Version : teler 1.0.1

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.
Supporting https://github.com/prabhatsharma/zinc

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

• NCSA Combined Log Format

%h %^[%d:%t %^] "%r" %s %b "%R" "%u"" "%u"%R%^,"%R","%u"

• NCSA Combined Log Format with Virtual Host

%^:%^ %h %^[%d:%t %^] "%r" %s %b "%R" "%u"%R%^,"%R","%u"

• Common Log Format (CLF)

%h %^[%d:%t %^] "%r" %s %b %s %b "%R" "%u"%R%^,"%R","%u"

• Common Log Format (CLF) with Virtual Host

%^:%^ %h %^[%d:%t %^] "%r" %s %b"%u"" "%u"%R%^,"%R","%u"

• W3C

%d %t %h %^ %^ %^ %m %r %^ %s %b %^ %^ %u %R%^,"%R","%u"

• CloudFront (Download Distribution)

%d\t%t\t%^\t%b\t%h\t%m\t%^\t%r\t%s\t%R\t%u\t%^,"%R","%u"

• Google Cloud Storage

"%x","%h",%^,%^,"%m","%U","%s",%^,"%b","%D",%^,"%R","%u"

• AWS Elastic Load Balancing (HTTP/S)

%dT%t.%^ %^ %h:%^ %^ %T %^ %^ %^ %s %^ %b "%r" "%u","%u"

Hello, every time I run the tool I get the same error:

[WRN] No logs analyzed, did you write log format correctly?

I have the tool configured but the problem persists.

foo bar bar

Is your feature request related to a problem? Please describe.
I know teler was a IDS,but i think it's will be great if teler can fight back against the threat
like reporting the source of ip address threat to abuseipdb

Describe the solution you'd like
N/A

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

All in the title.

Currently, this handles text logs like a charm, but it could get a bit complicated to parse JSON output like traefik does.

Is your feature request related to a problem? Please describe.
In its current state, teler always consumes internet connection if it wants to use resources.

Describe the solution you'd like
Give options to downloading resources and storing them locally: in other words the user DOES NOT need to use an internet connection if he wants to analyze logs with persistent data.

Describe alternatives you've considered
N/A

Additional context
Like adding an -dl or --download-resources flag; stores all resources at $HOME, checks if all resources are in local storage; otherwise it will use the internet connection to download resources.

For companies with a certain scale, Nginx is deployed in clusters, and the access.log file is distributed in each node machine, and the production environment server is not allowed to run programs that occupy unstable resources

Therefore, I hope to support Kafka as an input data source

// Thanks for open source this project, this is great work :)

Describe the bug

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x57c702]

goroutine 72 [running]:
net/url.(*URL).Query(0x0, 0x2f)
        /snap/go/6274/src/net/url/url.go:1032 +0x22
ktbs.dev/teler/pkg/teler.Analyze(0xc00006a370, 0xc00039a038, 0xc000667f01, 0x1)
        /home/dw1/Tools/teler/pkg/teler/teler.go:46 +0x112b
ktbs.dev/teler/internal/runner.New.func1(0xc00001c6c0, 0xc00006a370, 0xc000018460)
        /home/dw1/Tools/teler/internal/runner/runner.go:35 +0x91
created by ktbs.dev/teler/internal/runner.New
        /home/dw1/Tools/teler/internal/runner/runner.go:33 +0xfd
tail: error writing 'standard output': Broken pipe

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Where can I find those files?

common-web-attacks.json
cves.json
bad-ip-addresses.txt
bad-referrers.txt
bad-crawlers.txt
directory-bruteforces.txt

Move the documentation on Wiki

command line option -v, the output is always 1.0.1

run bin/teler -v the output is always

$ bin/teler -v
teler 1.0.1

cause the version varibale defined in constants.go is not overwritten at build time in Makefile

To Reproduce
just clone the repo, and run make build to compile the binary, then run bin/teler -v
the output is always

$ bin/teler -v
teler 1.0.1

Expected behavior
at this time, the output version should be

 bin/teler -v
teler v1.0.1-29-g62623bc

if we tag a new commit, the -v option should produce a new version

Screenshots
sorry for my company network environment, I can just describe in text

Environment:
I think this bug should reproduce in all OS
I am using windows10 and git-bash

• OS: [windows10]
• OS version: [windows10]
• teler Version the problem is about version, so ........

Is your feature request related to a problem? Please describe.
So that resources are used properly.

Describe the solution you'd like
Parse RAW HTTP request.

Describe alternatives you've considered
Using net/http.ReadRequest.

Additional context
N/A.

Describe the bug

Parameters & queries ignore whitelists.

Describe the bug

The regex which validates Slack tokens expects the third part of the token to be exactly 12 digits. However, I generated a token via Slack which has 13 digits. I run my command like this:

tail -f -n300 /var/log/caddy.log | teler -c ~/teler.yml

I receive the following error.

[ERR] Error! Only validates token; please check your config file
[INF] Use "-h" flag for more info about command.
Terminated

To Reproduce

Steps to reproduce the behavior:
Go to Slack and create a new bot at (yourworkspace).slack.com/apps/manage/custom-integrations
Then try to use the new token in your yml configuration.

log_format: |
  $remote_addr - $remote_user [$time_local] "$request_method $request_uri $request_protocol" $status $body_bytes_sent

alert:
  active: true
  provider: "slack"

notifications:
  slack:
    token: "xoxb-nnnnnnnnnnn-nnnnnnnnnnnnn-XXXXXXXXXXXXXXXXXXXXXXXX"
    color: "#ffd21a"
    channel: "XXXXXXXXXXX"

Expected behavior

I expected the Slack token to be accepted as a valid token.

Environment (please complete the following information):

• OS: linux
• OS version: Linux Ubuntu-1604-xenial-64-minimal 4.15.0-96-generic #97~16.04.1-Ubuntu SMP Wed Apr 1 03:03:31 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• teler Version: teler 0.0.4-dev

Additional context

I believe the issue is the regex found here https://github.com/kitabisa/teler/blob/f0f8ed54399dd938f77778e89b70001d03aa9703/pkg/matchers/patterns.go#L4

I think the d{12} should be changed to d{12,13}.