Each time you start the custom-bouncer you get multiple time the same order.
On the restart this morning for me, I had 95 order for the exact same IP. (and you have a full list of IP with that).
Multiple time the exact same order, but with different ban time.

time="10-12-2020 07:40:04" level=info msg="custom [/usr/local/bin/ban.php] : del ban on 180.253.162.196/32 for -509410 sec (crowdsecurity/http-path-traversal-probing)"
time="10-12-2020 07:41:04" level=info msg="custom [/usr/local/bin/ban.php] : del ban on 180.253.162.196/32 for -508364 sec (crowdsecurity/http-path-traversal-probing)"
time="10-12-2020 07:41:07" level=info msg="custom [/usr/local/bin/ban.php] : del ban on 180.253.162.196/32 for -508015 sec (crowdsecurity/http-path-traversal-probing)"

time="10-12-2020 07:45:45" level=info msg="custom [/usr/local/bin/ban.php] : add ban on 180.253.162.196/32 for 14799 sec (crowdsecurity/http-path-traversal-probing)"
time="10-12-2020 07:45:53" level=info msg="custom [/usr/local/bin/ban.php] : add ban on 180.253.162.196/32 for 21999 sec (crowdsecurity/http-path-traversal-probing)"
time="10-12-2020 07:45:59" level=info msg="custom [/usr/local/bin/ban.php] : add ban on 180.253.162.196/32 for 27311 sec (crowdsecurity/http-path-traversal-probing)"

To finally remove that ban :
time="10-12-2020 07:40:04" level=info msg="custom [/usr/local/bin/ban.php] : del ban on 180.253.162.196/32 for -509410 sec (crowdsecurity/http-path-traversal-probing)"
time="10-12-2020 07:45:19" level=info msg="custom [/usr/local/bin/ban.php] : add ban on 180.253.162.196/32 for 399 sec (crowdsecurity/http-path-traversal-probing)"
time="10-12-2020 07:52:07" level=info msg="custom [/usr/local/bin/ban.php] : del ban on 180.253.162.196/32 for -323 sec (crowdsecurity/http-path-traversal-probing)"

Using a local cache to not execute the same order twice improve a lot the start time when you call some external API like cloudflare, but this should be done on the bouncer side.
This reduce the CPU usage too, but can probably be improved a lot if the bouncer don't send useless order from the begining.

Maybe make something like fail2ban.
On stop try to delete the actual active ban list.
On start only add new ban.
And maybe add some cscli decisions delete -time 48H or something like that, to force delete ban based on time to clear external ban list (like cloudflare).

log.txt

When the bouncer stops, the custom script does not know about it, preventing it from being able to clean up any resources it might have created.

We should:

• Add a shutdown message when invoking the script for each decision to allow it to clean anything up if it needs to
• Send SIGTERM instead of the default SIGKILL when feeding via stdin to allow the script to catch it and clean up.

In both cases, we also need a termination_timeout parameter to handle custom scripts that would take too long to clean up. After that, we would SIGKILL the custom script (to avoid the bouncer getting stuck on stop).

It seems recently the project was renamed from crowdsec-custom-bouncer to cs-custom-bouncer .

So, In readme :

• logo image point to the incorrect repo
• url to release point to the incorrect repo
• From source : url of the git repo is incorrect, with all the request (not sure if github do a link to the new name)
• start : command seems invalid (on my install, service name seems to be cs-custom-bouncer.service )
• Upgrade : bad link to last release
• Configuration : on my installation, config seems to be in /etc/crowdsec/cs-custom-bouncer/cs-custom-bouncer.yaml and not /etc/crowdsec/bouncers/crowdsec-custom-bouncer.yaml ( and so, not under /etc/crowdsec/bouncers/ )
• "You can then start the service:" : on my installation, service name seems to be cs-custom-bouncer.service

Problem

Currently, each decision being fed to the custom-bouncer leads to an execution of the script. With the ever-growing (yay) list of decisions in the community-blocklist, it is (and will) become an issue. We need to have something that scales best.

cf. https://discourse.crowdsec.net/t/custom-bouncer-firewalld-questions/565

Solution

Allowing the user script to be fed via stdin seems to be a good alternative. Named pipe or such might be too, but would add unnecessary complexity ?
Allowing configurable batching parameters at the bouncer level (frequency and/or number of items)

Question / Things to deal with

• Configurable batch windows size/time
• Configurable behavior on child process crash (restart/die)

Can we apply something like this? I thought this flag would be merged to all bouncers.
crowdsecurity/cs-firewall-bouncer@194c45f

To get only the version number is still not straight forward.
I use currently:
crowdsec-custom-bouncer -v 2>&1 | grep level=info | sed -E "s/^.*v([0-9]+(\.[0-9]+)+)(.*\S.*)/\1/g"

To close properly #11 and crowdsecurity/crowdsec#827 we should document the cache retention mechanism in the README

If you restart the LAPI (in my case on a central node) the bouncer just stop working.
You got this message in log :
time="10-12-2020 07:37:31" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: connect: connection refused"
After that the process just stop.

You need to restart the bouncer (and go through the whole process on add/del ban).

This is really a problem on my setup. If I need to change the config on the central node, I have to restart all the bouncer on every servers. And use a lot of time to add/delete again the full ban list.

Maybe if the LAPI doesn't respond wait a little time, and retry connecting, maybe wait 1min or 2 and then restart the whole bouncer. But then we need some improvement on the process start to not use too much cpu time.

Both crowdsec and crowdsec-firewall-bouncer support a .yaml.local override mechanism, which was determined to be the most appropriate way to deploy both an unchanged upstream config file (registered via conffiles) and a local configuration file with an API key after automatic registration, via the postinst script of the Debian packages.

Unfortunately, that doesn't seem to be supported yet by the custom bouncer.

The crowdsec-custom-bouncer packages is not available in the offical packagecloud.io repositories for Ubuntu unlike stated in the official documentation.

https://doc.crowdsec.net/docs/bouncers/custom/#installation-from-packages

root ~ # apt install crowdsec-custom-bouncer
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package crowdsec-custom-bouncer

root ~ # lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

root ~ # cat /etc/apt/sources.list.d/crowdsec_crowdsec.list
# this file was generated by packagecloud.io for
# the repository at https://packagecloud.io/crowdsec/crowdsec

deb [signed-by=/usr/share/keyrings/crowdsec_crowdsec-archive-keyring.gpg] https://packagecloud.io/crowdsec/crowdsec/ubuntu/ jammy main
deb-src [signed-by=/usr/share/keyrings/crowdsec_crowdsec-archive-keyring.gpg] https://packagecloud.io/crowdsec/crowdsec/ubuntu/ jammy main

This is much some improvement that issue but well, have to post that somewhere.

Make the "add xx decision" and the "custom ..." on 2 different log level.
and improve documentation about log level :)

time="21-01-2021 17:00:58" level=info msg="adding '3' decisions"
time="21-01-2021 17:00:58" level=info msg="custom [xxxx] : add ban on xxxx for 86364 sec (...)"

This idea is to only log "adding '3' decisions" in production, but being able to set "debug" (or whatever you want) for the "custom [xxxx] "

Provide vendor files to support:

• Freebsd
• Gentoo

Most (all?) bouncers seem to be available as a Docker image, but not this one. Arguably depending on what the custom script is doing, a custom image will still need to be built (to install dependencies), but having an official image to extend would make the process easier.

In the meantime if anyone having the same issue, here's a sample Dockerfile that works for me (the custom script just needs to be mounted to the bin_path value in the YAML config):

ARG VERSION=v0.0.17-rc6

FROM golang:1.21.4 as build-stage
ARG VERSION
WORKDIR /app

RUN git clone --branch $VERSION --depth 1 https://github.com/crowdsecurity/cs-custom-bouncer.git
WORKDIR /app/cs-custom-bouncer
RUN CGO_ENABLED=0 GOOS=linux make release

FROM ubuntu:22.04
ARG VERSION
WORKDIR /

RUN mkdir -p /etc/crowdsec/bouncers
COPY --from=build-stage /app/cs-custom-bouncer/crowdsec-custom-bouncer-$VERSION/crowdsec-custom-bouncer \
    /usr/bin/crowdsec-custom-bouncer
COPY --from=build-stage /app/cs-custom-bouncer/crowdsec-custom-bouncer-$VERSION/config/crowdsec-custom-bouncer.yaml \
    /etc/crowdsec/bouncers/crowdsec-custom-bouncer.yaml

CMD ["/usr/bin/crowdsec-custom-bouncer", "-c", "/etc/crowdsec/bouncers/crowdsec-custom-bouncer.yaml"]

Alternatively just providing that sample Dockerfile in the docs would be nice

Hello,

A significant change was made in go-cs-bouncer (commit : crowdsecurity/go-cs-bouncer@5aef3b3)

the .Run method of go-cs-bouncer doesn't Fatal upon initialization failure (this failure would namely happen if the LAPI cannot be reached or cannot authenticate to LAPI).

The reason for this change is to allow bouncers/IPS to take advantage of defer statements for cleanup.

You can see reflected change in firewall-bouncer that uses this new version :

• crowdsecurity/cs-firewall-bouncer#163