luzilla / dnsbl_exporter Goto Github PK

View Code? Open in Web Editor NEW

30.0 3.0 8.0 635 KB

Prometheus compatible exporter to query DNSBLs/RBLs.

Home Page: https://www.luzilla-capital.com/

License: Other

Makefile 2.57% Go 90.86% Dockerfile 1.78% Smarty 4.79%

prometheus prometheus-exporter dnsbl dnsbl-checker rbl

dnsbl_exporter's People

Contributors

Stargazers

Watchers

Forkers

ralfyang drmurx supu2 zentavr s-diez xelite dragoangel vquie

dnsbl_exporter's Issues

segfaults caused by not checking return values of err

Thanks for dnsbl_exporter, it's fantastic and looks to be just the tool I'm looking for.

I've encountered the potential for it to segfault, though:

time="2019-12-15T23:06:46Z" level=info msg="We had an ip XXX.XXX.XXX.XXX"

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x836a8b]
goroutine 33 [running]:
github.com/luzilla/dnsbl_exporter/collector.(*Rbl).getARecords(0xc00006fe90, 0xc000092210, 0x23, 0x2, 0x2, 0xc000092210, 0x23, 0xc000090a00)
#011/Users/till/Documents/workspaces/luzilla/dnsbl_exporter/collector/rbl.go:65 +0x8b
github.com/luzilla/dnsbl_exporter/collector.(*Rbl).query(0xc00006fe90, 0xc000090a00, 0xc, 0xc000026a80, 0x16, 0xc0001f7ea8)
#011/Users/till/Documents/workspaces/luzilla/dnsbl_exporter/collector/rbl.go:104 +0x1f9
github.com/luzilla/dnsbl_exporter/collector.(*Rbl).lookup(0xc00006fe90, 0xc000026a80, 0x16, 0xc000022450, 0xc, 0x1, 0x1, 0x0)
#011/Users/till/Documents/workspaces/luzilla/dnsbl_exporter/collector/rbl.go:145 +0x2ca
github.com/luzilla/dnsbl_exporter/collector.(*Rbl).Update.func1(0xc0000909a0, 0xc00006fe90, 0xc000026a80, 0x16, 0xc000022450, 0xc)
#011/Users/till/Documents/workspaces/luzilla/dnsbl_exporter/collector/rbl.go:166 +0x133
created by github.com/luzilla/dnsbl_exporter/collector.(*Rbl).Update
#011/Users/till/Documents/workspaces/luzilla/dnsbl_exporter/collector/rbl.go:161 +0xf0

I'm guessing that this is because getARecords does not check the value of err on line 61, before it goes and tries to iterate through the response.

Would you like me to submit a PR to handle some of the missing checks?

systemd unit

The configuration is pretty straightforward

[Unit]
Description=DNSBL Exporter
StartLimitBurst=5


[Service]
User=root
ExecStart=/root/prometheus-monitoring/dnsbl_exporter/dnsbl_exporter --config.dns-resolver [REDACTED] --config.rbls /root/prometheus-monitoring/config-files/dnsbl_exporter/rbls.ini --config.targets /root/prometheus-monitoring/config-files/dnsbl_exporter/targets.ini
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=default.target

The version used is the latest release

./dnsbl_exporter --version
dnsbl-exporter version 0.4.3

Originally posted by @andbuitra in #64 (comment)

Chart not built

Hi,

the charts version was update with #229. Unfortunately it wasn't built yet.

Error: ghcr.io/luzilla/charts/dnsbl-exporter:0.1.1: not found

upload sample alert (alertmanager)

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

github.com/luzilla/dnsbl_exporter/collector: cannot find module providing package github.com/luzilla/dnsbl_exporter/collector
	github.com/luzilla/dnsbl_exporter/config: cannot find module providing package github.com/luzilla/dnsbl_exporter/config

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Support IPv6

Currently IPv6 isn't supported by this module.

For example IP 2003:e8:7f1d:f600:: is listed on zen.spamhaus.org and to check if it is the case we can resolve next record: 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.6.f.d.1.f.7.8.e.0.0.3.0.0.2.zen.spamhaus.org IN A which is expanded version of IPv6 requested in backward direction as any other IP RBL. as result we will get 127.0.0.11, but /prober?target=2003:e8:7f1d:f600:: will return luzilla_rbls_errors{rbl="zen.spamhaus.org"} 1, manually expanding IPv6 and querying exporter also doesn't works /prober?target=2003:00e8:7f1d:f600:0000:0000:0000:0000.

To support IPv6 we need:

check user provided value if it is a valid IPv6
if this is valid IPv6 we need expand IPv6 to it's full version
if this is not valid IPv6 we need check if this is valid IPv4, and if this also not the case set an error for this target
if this IPv6: remove all : as we don't need them, split IP to array by 1 char, invert order of array and join it back to string by .
if this IPv4: split IP by . to array, invert over of array and join it back to string by .
append . and rbl domain to the end of record and resolve it, profit.

Provide service files

systemd (Linux) #86
rc.d (FreeBSD)

Crash after running a few days

ERRO[357497]
ERRO[358096]
ERRO[358096]
ERRO[358096]
ERRO[358098]
ERRO[358696]
ERRO[359597]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x46f1d3]

goroutine 213683 [running]:
github.com/Luzilla/dnsbl_exporter/collector.(*Rbl).lookup(0xc0002c0030, {0xc000230420, 0xc}, {0xc000230550, 0xc})
        /home/lapo/dnsbl_exporter/collector/rbl.go:163 +0x4b9
github.com/Luzilla/dnsbl_exporter/collector.(*Rbl).Update.func1(0xc0000b2000, 0xc0002c0030, {0xc000230420, 0xc}, {0xc000230550, 0xc})
        /home/lapo/dnsbl_exporter/collector/rbl.go:181 +0x16d
created by github.com/Luzilla/dnsbl_exporter/collector.(*Rbl).Update
        /home/lapo/dnsbl_exporter/collector/rbl.go:176 +0x105

run as non-root

The exporter currently runs as a root user. But instead, it should be a regular account (daemon).

Hide build in metrics (Golang)

Turn them off by default.

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

github.com/luzilla/dnsbl_exporter/collector: cannot find module providing package github.com/luzilla/dnsbl_exporter/collector
github.com/luzilla/dnsbl_exporter imports
	github.com/luzilla/dnsbl_exporter/config: cannot find module providing package github.com/luzilla/dnsbl_exporter/config

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Implement --debug

Flag is there but not being used.

Support modules configuration

I think such exporter would be fit much better with /probe?module=rbl_ips&target=192.0.2.1 or /probe?module=rbl_domains&target=example.com, like it done in blackbox exporter, ssl exporter, etc. It will allow people who using stuff like Prometheus Operator control which things to monitor by simply proving CRDs in k8s to probe, instead of creating\adjusting configuration of exporter, this is much more dynamic flow. People without Prometheus Operator still can just edit prometheys.yaml without accessing and redeploying dnsbl exporter.

In this way dnsbl_exporter.conf configuration of module would have next scheme:

modules:
  <module_name_used_by_probe>: # meaning name of rbls to check with
    prober: dnsbl # name of modules that supported by exporter, if there no differnce between rbl resolving that should be treaded specially - this will be always dnsbl
    timeout: <duration> # module timeout
    dnsbl: # if not changed in prober, configures module behaviour
      # qeuery ips by default, f.e IPv4 192.0.2.1 would query 0.2.0.192.rbl-example.com
      # and IPv6 2001:db8:: - 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.rbl-example.com
      # if set to false query will be done as-is, f.e:
      # if target is mysite.com - then query is mysite.com.rbl-example.com.
      ips: true
      hashing: nil # defaults to no hashing, options: md5, sha1, sha256, etc.
      rbls:
        - <rbl.example.com>
        - <rbl2.example.com>
      resolvers: # if unset system resolvers will be used
        - <ns1.example.com>
        - <ns2.example.com>

resulting in:

modules:
  rbls_ips:
    prober: dnsbl
    timeout: 3s
    dnsbl:
      rbl_servers:
        - ix.dnsbl.manitu.net
        - zen.spamhaus.org
      resolvers:
        - 192.0.2.10
  rbl_domains:
    prober: dnsbl
    timeout: 3s
    dnsbl:
      ips: false
      rbl_servers:
        - dbl.spamhaus.org

Such workflow also as you see would allow using custom dns resolver per each module.

Here how would Prometheus probes would look like with probes flow:

apiVersion: monitoring.coreos.com/v1
kind: Probe
spec:
  interval: 15m
  jobName: myips
  module: rbl_ips
  prober:
    path: /probe
    scheme: http
    url: dnsbl-exporter.svc:9211
  scrapeTimeout: 5s
  targets:
    staticConfig:
      static:
        - 192.0.2.1
        - 192.0.2.2
        - 192.0.2.3
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
spec:
  interval: 15m
  jobName: mydomains
  module: rbl_domains
  prober:
    path: /probe
    scheme: http
    url: dnsbl-exporter.svc:9211
  scrapeTimeout: 5s
  targets:
    staticConfig:
      static:
        - example.com
        - example.org

Support multi-target exporter pattern

Feature proposal:

Support the multi-target exporter pattern so that the targets can be defined (or discovered) in the Prometheus config instead of target.ini.

helm repo needs authentication

Installing the helm chart does not work. The repo requires auth.

Please allow unauthenticated access to the repo.

STDERR:
  Error: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Aluzilla%2Fcharts%2Fdnsbl-exporter%2Fdnsbl-exporter%3Apull&service=ghcr.io: 403 Forbidden

Setup homebrew-tab

use goreleaser: https://goreleaser.com/homebrew/
provide services file (similar to #10)

Registry broken

@till Thanks, I might do so later. Right now busy ... / and I assume it's more of a docker-issue in my place.
btw your Dockerfile fails and there are no images in the mentioned registry ;-) / I assume you know this. No complaint, only feedback.

Originally posted by @stefangweichinger in #87 (comment)

Ability to use list with many resolvers in --config.dns-resolver

I would like to use multiple resolvers or even better - use defaults from OS when --config.dns-resolver is not provided.

Add an ability to use the certain black list to query only domain names

There are a certain black list which contain only the domains, but not IPs. I.E.: dbl.spamhaus.org. From DBL FAQ they mention:

The Spamhaus DBL is a list of domain names with poor reputations. It is published in a domain DNSBL format. These domain reputations are calculated from many factors, and maintained in a database which in turn feeds the DBL zone itself.
It ONLY lists domains. No IP addresses are listed by the DBL.
The DBL is a domain-only blocklist and does not include or support IP addresses.
An IP query against the DBL always returns a positive (listed) return code

It would be great to have a separate list file which would be used only for these kind of checks

remove logrus (for slog)

Exporter suddenly stop working.

I have installed latest release from git.

I set exporter to work as systemd service. This is systemd file:

[Unit]
Description=RBLExporter

[Service]
TimeoutStartSec=0
User=rbl_exporter
ExecStart=/usr/bin/dnsbl-exporter --web.listen-address=0.0.0.0:9211 --config.rbls "/etc/rbl-exporter/rbl_list.ini" --config.targets "/etc/rbl-exporter/rbl_targets.ini"

[Install]
WantedBy=multi-user.target

OS is: VERSION="22.04.2 LTS (Jammy Jellyfish)"

Service working some time, sometimes several hours, sometimes days or two and suddenly stop working with error:

<autogenerated>:1 +0x29 fp=0xc0001b0f78 sp=0xc0001b0f48 pc=0x5d5c49
rbl_exporter.service: Consumed 46.831s CPU time.
net/http.(*connReader).backgroundRead(0xc0002787e0)
/opt/hostedtoolcache/go/1.20.3/x64/src/net/http/server.go:674 +0x3f fp=0xc0001b0fc8 sp=0xc0001b0f78 pc=0x6a821f
net/http.(*connReader).startBackgroundRead.func2()
/opt/hostedtoolcache/go/1.20.3/x64/src/net/http/server.go:670 +0x26 fp=0xc0001b0fe0 sp=0xc0001b0fc8 pc=0x6a8146
runtime.goexit()
/opt/hostedtoolcache/go/1.20.3/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001b0fe8 sp=0xc0001b0fe0 pc=0x46a341
created by net/http.(*connReader).startBackgroundRead
/opt/hostedtoolcache/go/1.20.3/x64/src/net/http/server.go:670 +0xca

Do automated release

When we push a tag. It should work, but doesn't for some reason.

Move CI to Actions

Should copy over files from other go projects as I am tired of debugging Circle.

Release v0.6.0

Setup dockerfile

Is there any chance that we could get more information about the block reason?

Hello @till!

Thank you for the nice tool!
Very oftenly DNSBL providers provide the description of the block, when you query TXT record. For example:

$ dig -t TXT 212.40.245.168.dnsbl-3.uceprotect.net

;; ANSWER SECTION:
212.40.245.168.dnsbl-3.uceprotect.net. 1973 IN TXT "Your ISP SENDGRID, US/AS11377 is UCEPROTECT-Level3 listed because of a spamscore of 55.7. See: http://www.uceprotect.net/rblcheck.php?ipr=168.245.40.212"

I wonder if it is possible to put this, saying, into the label?

Helm Chart

Hello @till.

I'd developed a helm chart which installs dnsbl-exporter into Kubernetes clusters. Just a simple Daemonset, configmap (no Ingress). If you are interested, I can submit a PR of that.

P.S.: Unfortunately I cannot fetch the docker image from:

image:
  registry: ghcr.io
  repository: luzilla/dnsbl_exporter
  tag: v0.6.0

...so I'd build my own container using the binaries from the Release page of that repo.

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

github.com/Luzilla/dnsbl_exporter/collector: cannot find module providing package github.com/Luzilla/dnsbl_exporter/collector
	github.com/Luzilla/dnsbl_exporter/config: cannot find module providing package github.com/Luzilla/dnsbl_exporter/config

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

scraping metrics with a dockerized prometheus

systemd service

Downloaded current binary into:

ls -l /usr/local/sbin/dnsbl-exporter 
-rwxr-xr-x 1 root docker 10285056 May  6  2023 /usr/local/sbin/dnsbl-exporter

Set up service file as in #86 (comment)

modified paths to ini-files etc ->

# cat /etc/systemd/system/dnsbl-exporter.service 
[Unit]
Description=DNSBL Exporter
StartLimitBurst=5

[Service]
User=root
#ExecStart=/usr/local/sbin/dnsbl-exporter --config.dns-resolver 127.0.0.1 --config.rbls /etc/prometheus/dnsbl-exporter/rbls.ini --config.targets /etc/prometheus/dnsbl-exporter/targets.ini
ExecStart=/usr/local/sbin/dnsbl-exporter --config.rbls /etc/prometheus/dnsbl-exporter/rbls.ini --config.targets /etc/prometheus/dnsbl-exporter/targets.ini
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=default.target

#  ls -l /etc/prometheus/dnsbl-exporter/
total 8
-rw-r--r-- 1 root root 2764 Nov 12 09:22 rbls.ini
-rw-r--r-- 1 root root  182 Nov 12 09:24 targets.ini

Service runs, and returns metrics:

 curl http://localhost:9211/metrics
# HELP luzilla_rbls_duration The scrape's duration (in seconds)
# TYPE luzilla_rbls_duration gauge
luzilla_rbls_duration 0.002717003
# HELP luzilla_rbls_ips_blacklisted Blacklisted IPs
# TYPE luzilla_rbls_ips_blacklisted gauge

[..]

luzilla_rbls_ips_blacklisted{hostname="oc.oops.co.at",ip="45.84.138.128",rbl="ix.dnsbl.manitu.net"} 0

[..]

# HELP luzilla_rbls_listed The number of listings in RBLs (this is bad)
# TYPE luzilla_rbls_listed gauge
luzilla_rbls_listed{rbl="ix.dnsbl.manitu.net"} 0
luzilla_rbls_listed{rbl="pbl.spamhaus.org"} 0
luzilla_rbls_listed{rbl="sbl.spamhaus.org"} 0
luzilla_rbls_listed{rbl="xbl.spamhaus.org"} 0
luzilla_rbls_listed{rbl="zen.spamhaus.org"} 0
# HELP luzilla_rbls_targets The number of targets that are being probed (configured via targets.ini or ?target=)
# TYPE luzilla_rbls_targets gauge
luzilla_rbls_targets 4
# HELP luzilla_rbls_used The number of RBLs to check IPs against (configured via rbls.ini)
# TYPE luzilla_rbls_used gauge
luzilla_rbls_used 5
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0

prometheus

Prometheus runs in docker, so I have to enable it to access ports on the host level.
This is done by setting "extra_hosts":

services:

  prometheus:
    image: prom/prometheus:v2.47.2
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - 9090:9090
    networks:
      - back-tier
      - front-tier
    restart: always
    extra_hosts:
      - "host.docker.internal:host-gateway"

The scraping is configured in prometheus.yml:

  - job_name: 'dnsbl-exporter'

    static_configs:
      - targets: ['host.docker.internal:9211']

No errors are shown, but I don't see any metrics with "luzilla*" in Grafana/Prometheus.

BC: replace `blacklisted` with `blocklisted` or similar

Setup CI

Maybe Github Actions? Or CircleCI.

Anyway, CI should run, and also do releases.

Recurring SIGSEGV

Hello,

We deployed dnsbl_exporter on a CentOS 7 machine as a systemd service. It's currently going offline pretty often complaining about memory (either oom or sigsegv). This is the error:

feb 13 04:31:34 monitor2-co dnsbl_exporter[685]: panic: runtime error: invalid memory address or nil pointer dereference
feb 13 04:31:34 monitor2-co dnsbl_exporter[685]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4640e7]
feb 13 04:31:34 monitor2-co dnsbl_exporter[685]: goroutine 316739 [running]:
feb 13 04:31:34 monitor2-co dnsbl_exporter[685]: github.com/luzilla/dnsbl_exporter/collector.(*Rbl).lookup(0xc000330510, 0xc00019a120, 0x12, 0xc00019a1e0, 0x1d, 0x1, 0x1, 0x0)
feb 13 04:31:34 monitor2-co dnsbl_exporter[685]: /home/runner/work/dnsbl_exporter/dnsbl_exporter/collector/rbl.go:147 +0x3bd
feb 13 04:31:34 monitor2-co dnsbl_exporter[685]: github.com/luzilla/dnsbl_exporter/collector.(*Rbl).Update.func1(0xc0002da1b0, 0xc000330510, 0xc00019a120, 0x12, 0xc00019a1e0, 0x1d)
feb 13 04:31:34 monitor2-co dnsbl_exporter[685]: /home/runner/work/dnsbl_exporter/dnsbl_exporter/collector/rbl.go:166 +0x113
feb 13 04:31:34 monitor2-co dnsbl_exporter[685]: created by github.com/luzilla/dnsbl_exporter/collector.(*Rbl).Update
feb 13 04:31:34 monitor2-co dnsbl_exporter[685]: /home/runner/work/dnsbl_exporter/dnsbl_exporter/collector/rbl.go:161 +0xf0
feb 13 04:31:34 monitor2-co systemd[1]: dnsbl_exporter.service: main process exited, code=exited, status=2/INVALIDARGUMENT
feb 13 04:31:34 monitor2-co systemd[1]: Unit dnsbl_exporter.service entered failed state.
feb 13 04:31:34 monitor2-co systemd[1]: dnsbl_exporter.service failed.

There's plenty of memory available (more than 6 GB) so this shouldn't be an issue. So far I've resorted to configure auto restart for the systemd unit. If relevant, the log also shows plenty of these:

feb 13 04:17:35 monitor2-co dnsbl_exporter[685]: time="2021-02-13T04:17:35-05:00" level=error
feb 13 04:17:39 monitor2-co dnsbl_exporter[685]: time="2021-02-13T04:17:39-05:00" level=error
feb 13 04:19:35 monitor2-co dnsbl_exporter[685]: time="2021-02-13T04:19:35-05:00" level=error
feb 13 04:19:35 monitor2-co dnsbl_exporter[685]: time="2021-02-13T04:19:35-05:00" level=error
feb 13 04:25:36 monitor2-co dnsbl_exporter[685]: time="2021-02-13T04:25:36-05:00" level=error
feb 13 04:25:36 monitor2-co dnsbl_exporter[685]: time="2021-02-13T04:25:36-05:00" level=error
feb 13 04:29:36 monitor2-co dnsbl_exporter[685]: time="2021-02-13T04:29:36-05:00" level=error
feb 13 04:29:36 monitor2-co dnsbl_exporter[685]: time="2021-02-13T04:29:36-05:00" level=error

There's nothing too special about our config. The only thing is that we load the RBLs and targets (using the proper args with absolute paths) from a folder that is linked to a git repo.

Stray "foo"

I get this in /metrics/:

# HELP luzilla_rbls_listed The number of listings in RBLs (this is bad)
# TYPE luzilla_rbls_listed gauge
luzilla_rbls_listed{rbl="foo"} 1
# HELP luzilla_rbls_used The number of RBLs to check IPs against (configured via rbls.ini)
# TYPE luzilla_rbls_used gauge
luzilla_rbls_used 86

I know no Go, but I guess this might the problem.

PS: nice exporter, will be useful to me, thanks!

roadmap

#185
#167

#218
code: add golangci or something
docs: example config for prometheus
docs: example for alertmanager #88
docs: upload example grafana dashboard #87

bugfix: #159
bugfix: #178
bugfix: #193

luzilla / dnsbl_exporter Goto Github PK

dnsbl_exporter's People

Contributors

Stargazers

Watchers

Forkers

dnsbl_exporter's Issues

systemd service

prometheus

Recommend Projects

Recommend Topics

Recommend Org