shizunge / endlessh-go Goto Github PK
View Code? Open in Web Editor NEWA golang implementation of endlessh exporting Prometheus metrics, visualized by a Grafana dashboard.
License: GNU General Public License v3.0
A golang implementation of endlessh exporting Prometheus metrics, visualized by a Grafana dashboard.
License: GNU General Public License v3.0
I see that endlessh-go is using glog for logging along with gflags
(I think?) to pass options as args. Some of endlessh-go's args are pulled directly from logging.cc
but when I try to use flags not listed, timestamp_in_logfile_name
and max_log_size
, I get an error.
flag provided but not defined: -max_log_size
Usage of /endlessh
-alsologtostderr
...
I don't see where in endlessh-go's code where the logging flags are defined... Would it be possible to use these additional flags? I'm try to configure basic log rotation. Thanks!
Hi. how can I use ./endlessh-go options ?
for example how to use -geoip_supplier option ?
thanks.
$> docker --version
Docker version 24.0.7, build afdd53b4e3
$> sudo docker pull ghcr.io/shizunge/endlessh-go:latest
latest: Pulling from shizunge/endlessh-go
manifest unknown
Pulling from Docker Hub works fine.
The Grafana locations panel don't show the client, if you use the MaxMind offline DB.
"Time","name","country","geohash","instance","ip","job","location","Value #Seen"
2022-12-19 11:39:00,endlessh_client_open_count,China,s000,192.168.10.214:2112,106.13.29.110,endlessh,China,3
2022-12-20 03:35:00,endlessh_client_open_count,China,s000,192.168.10.214:2112,111.67.202.249,endlessh,China,28
2022-12-19 15:52:00,endlessh_client_open_count,China,s000,192.168.10.214:2112,124.223.26.191,endlessh,China,37
2022-12-20 09:22:00,endlessh_client_open_count,China,s000,192.168.10.214:2112,163.53.90.10,endlessh,China,1
2022-12-19 16:50:00,endlessh_client_open_count,China,s000,192.168.10.214:2112,175.178.40.24,endlessh,China,36
i assume, geohash shouldn't be "S000".
Any idea, how to fix it?
I'm working on an enhanced crowdsec collection for endlessh-go as well as a basic notifier to discord. Neither of these can take advantage of prometheus exports directly and instead need to use the log file generated to parse events. It would be helpful if the geo information parsed for prometheus metrics was included in the log lines (on ACCEPT) so that I don't have to make another call to ip-api to get the same information again.
Add capability to use offline IP-GEO database (e.g. maxmind)
Hi,
any chance of getting an example on how to use max-mind-db on endlessh-go?
Because there are three different db on max-mind an each comes in two flavors.
The linked online alternatives IP-API.com and freegeoip.live are offline.
-geoip_supplier=max-mind-db -max_mind_db=/absolute/path/to/GeoLite2-Country.mmdb
-geoip_supplier=max-mind-db -max_mind_db=./GeoLite2-Country.mmdb
-geoip_supplier=max-mind-db -max_mind_db ./GeoLite2-Country.mmdb
I guess I am using the wrong database because none of them worked...
could you point me in the right direction? Thankx.
Clients data did not display correctly in Grafana 8.3.4. Any thing depends on clients also shows NaN.
It is working with Grafana 8.3.3
No problem for Sent Bytes, Unique IPs and Current Connections.
For example, to make it compatible with systemd's socket activation.
func geohashAndLocationFromFreegeoip(address string) (string, string, string, error) {
var geo freegeoip
response, err := http.Get("https://freegeoip.live/json/" + address)
As mentioned here https://freegeoip.io/ the api call should be different: freegeoip.io/{format}/{IP_or_hostname}.
hello, i'm using unraid 6.9.2 and i tried your endlessh-go docker. I run it like this:
sudo docker run -d -p 2222:2222 shizunge/endlessh-go -logtostderr -v=1 -enable_prometheus
your tool is running, i get "trapped" on port 2222 when trying to connect via ssh.
but i dont get managed to get the prometheus metrics from it and i dont know why.
looking at the log, all seems to be fine - it starts and logs my ssh connection attemp:
I0705 04:11:18.579285 1 main.go:102] Starting Prometheus on 0.0.0.0:2112, entry point is /metrics
I0705 04:11:18.580008 1 main.go:142] Listening on 0.0.0.0:2222
I0705 04:13:29.392797 1 client.go:69] ACCEPT host=192.168.2.2 port=1693 n=1/4096
I0705 04:13:34.395198 1 client.go:109] CLOSE host=192.168.2.2 port=1693 time=5.002307282 bytes=71
In the "Clients" Panel, i get an error saying:
"v is undefined".
Do you have any idea what's that all about?
I'm unable to enable logging to a file so my Crowdsec instance could parse those logs and issue bans.
Workaround is to use syslog logging driver of the Docker and make Crowdsec parse syslog, but IMO its not optimal even though it works.
This is my docker compose, basically its the maxmind example from here. I've tried multiple other variations of commands, that I could think of, but no log file appears on host system at /portainer/Files/AppData/Config/endlessh-go/logs/
-folder.
version: '3.5'
services:
endlessh:
container_name: endlessh
image: shizunge/endlessh-go:latest
restart: unless-stopped
user: root
command:
- "-log_dir=/log/"
- "-interval_ms=1000"
- "-v=1"
- "-enable_prometheus"
- "-geoip_supplier=max-mind-db"
- "-max_mind_db=/geo-data/GeoLite2-City.mmdb"
- "-alsologtostderr"
networks:
- default
hostname: endlessh
ports:
- 2222:2222 # SSH port
- 2112:2112 # Prometheus metrics port
volumes:
- /portainer/Files/AppData/Config/endlessh-go/logs/:/log:ro # logs
- /portainer/Files/AppData/Config/maxmind/geo-data/:/geo-data/:ro # geoip data
logging:
driver: "syslog"
...continues with maxmind portion
By default the info messages about ACCEPT
and CLOSE
of this service are listed as errors in my systemd journal.
Which results in spamming me with log reports since there are many attempts in my honeypot.
I thought these are level 1 and the default stderrthreshold
is 2.
Endlessh-go reports country, location, and geohash data as unknown; after looking at the systemd page, I see this error message: Failed to obatin the geohash of [ipv4 redacted]: open /GeoLite2-Country.mmdb: no such file or directory.
when running ls -lh /GeoLite2-Country.mmdb
I get: -rw-r--r-- 1 root root 6.2M Mar 5 13:41 /GeoLite2-Country.mmdb
, however even after running chmod 777 /GeoLite2-Country.mmdb
I still get the same error. I'm running in bare metal (not docker) so there shouldn't be anything preventing it from accessing the database file. Does it need to be owned by a specific user? Does it need to be in a specific directory/have a specific name?
If it helps with debugging, I'm running this on nixOS 22.11.3810.3d302c67ab8 (due to using nixos & endlessh-go not having a --version flag, I can only say that I'm running the latest version as of 2023/08/23). I have my service configuration as follows:
services.endlessh-go = {
enable = true;
port = 22;
openFirewall = true;
extraOptions = [
"-enable_prometheus"
"-prometheus_entry ''"
"-geoip_supplier 'max-mind-db'"
"-max_mind_db '/GeoLite2-Country.mmdb"
];
prometheus = {
enable = false;
listenAddress = "127.0.0.1";
port = 2112;
};
};
I am trying to get the logs so fail2ban can read them, but wondered where the logs to to so i can bind them to a volume/mount.
version: '3.5'
services:
endlessh:
container_name: tarpit-endlessh
image: shizunge/endlessh-go:latest
restart: unless-stopped
command:
- -interval_ms=1000
- -logtostderr
- -v=1
- -enable_prometheus
- -geoip_supplier=ip-api
- -log_dir=/example
ports:
- 22:2222
# - 9101:2112 #prometheus port (now handled via networks)
networks:
- default
- prometheus
volumes:
- /home/user/example:/example
networks:
prometheus:
external: true
Here i tried to put it into /example of the container and then ls /home/user/example, but nothing was logged there.
I am probably doing something wrong so if you see what mistake i made could you please point it out to me, thanks :)
Nice project. I converted to using this one from the linuxserver.io one (https://docs.linuxserver.io/images/docker-endlessh/) because you have prometheus metrics built in.. thanks for your work on this
I think that there is something a bit odd about the actual counting of connections / time spent.
From the prometheus output:
endlessh_client_trapped_time_seconds{ip="218.92.0.125",local_port="2222"} 5242.06399999999
but my container has only been running a short while
$ docker ps | grep end
3e32fbf10ef8 shizunge/endlessh-go "/endlessh -enable_p…" 30 minutes ago Up 30 minutes
30mins = 1800 seconds
I don't see how I can have 5242 seconds of 'trapped time'
Here is the full prometheus output
# HELP endlessh_client_closed_count_total Total number of clients that stopped connecting to this host.
# TYPE endlessh_client_closed_count_total counter
endlessh_client_closed_count_total{local_port="2222"} 29
# HELP endlessh_client_open_count Number of connections of clients.
# TYPE endlessh_client_open_count counter
endlessh_client_open_count{country="China",geohash="wtssb2djg",ip="218.92.0.125",local_port="2222",location="China, Jiangsu, Nanjing"} 33
endlessh_client_open_count{country="China",geohash="wx4g0kzf1",ip="54.222.238.164",local_port="2222",location="China, Beijing, Beijing"} 1
# HELP endlessh_client_open_count_total Total number of clients that tried to connect to this host.
# TYPE endlessh_client_open_count_total counter
endlessh_client_open_count_total{local_port="2222"} 34
# HELP endlessh_client_trapped_time_seconds Seconds a client spends on endlessh.
# TYPE endlessh_client_trapped_time_seconds counter
endlessh_client_trapped_time_seconds{ip="218.92.0.125",local_port="2222"} 5242.06399999999
endlessh_client_trapped_time_seconds{ip="54.222.238.164",local_port="2222"} 938.4460000000003
# HELP endlessh_sent_bytes_total Total bytes sent to clients that tried to connect to this host.
# TYPE endlessh_sent_bytes_total counter
endlessh_sent_bytes_total{local_port="2222"} 101505
# HELP endlessh_trapped_time_seconds_total Total seconds clients spent on endlessh.
# TYPE endlessh_trapped_time_seconds_total counter
endlessh_trapped_time_seconds_total{local_port="2222"} 6180.5099999999875
(so.. maybe there is a concurrent connection thing happening so that's what's up?).. but the Grafana dashboard (where I noticed this first) -- just seems way off?
Important to note - that the data was captured a few minutes apart - so there may be differences in the Prometheus data dump vs. the Grafana image capture -- but only a minute or two difference.
Add the ability to use a yaml/toml configuration file.
To my point of view cli_args should either define a configuration file either define args directly.
I tried to add example like parameters on hub.docker.com like e= but it dosnt work.
It should have maybe have a volume (v:) for config or enviroments (e:) directly in docker run . There should at least be a volume options for logs.
-alsologtostderr
log to standard error as well as files
-conn_type string
Connection type. Possible values are tcp, tcp4, tcp6 (default "tcp")
-enable_prometheus
Enable prometheus
-geoip_supplier string
Supplier to obtain Geohash of IPs. Possible values are "off", "ip-api", "max-mind-db" (default "off")
-host string
SSH listening address (default "0.0.0.0")
-interval_ms int
Message millisecond delay (default 1000)
-line_length int
Maximum banner line length (default 32)
-log_backtrace_at value
when logging hits line file:N, emit a stack trace
-log_dir string
If non-empty, write log files in this directory
-log_link string
If non-empty, add symbolic links in this directory to the log files
-logbuflevel int
Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; ...). Has limited applicability on non-prod platforms.
-logtostderr
log to standard error instead of files
-max_clients int
Maximum number of clients (default 4096)
-max_mind_db string
Path to the MaxMind DB file.
-port value
SSH listening port. You may provide multiple -port flags to listen to multiple ports. (default "2222")
-prometheus_clean_unseen_seconds int
Remove series if the IP is not seen for the given time. Set to 0 to disable. (default 0)
-prometheus_entry string
Entry point for prometheus (default "metrics")
-prometheus_host string
The address for prometheus (default "0.0.0.0")
-prometheus_port string
The port for prometheus (default "2112")
-stderrthreshold value
logs at or above this threshold go to stderr (default 2)
-v value
log level for V logs
-vmodule value
comma-separated list of pattern=N settings for file-filtered logging
The usage of CLI arguments works well for binary usage but with docker images it would be better to propose environments variables.
We could imagine something like:
cli_args if not undefined else anv_agrs if not undefined else default
I had the error as shown in the screenshot above and tried to completely rebuild and deploy all my docker containers and the Grafana dashboard. I also deleted all storage volumes and completely rebuilt the stack, but unfortunately the error did not disappear. Here is my docker stack which I run in my portainer:
version: '3.8'
services:
endlessh:
image: shizunge/endlessh-go:latest
container_name: endlessh
command:
- -interval_ms=1000
- -logtostderr
- -v=1
- -enable_prometheus
- -geoip_supplier=ip-api
ports:
# SSH port
- 2222:2222
# Prometheus metrics port
- 2112:2112
restart: always
stdin_open: true # docker run -i
tty: true # docker run -t
prometheus:
image: prom/prometheus:latest
container_name: prometheus
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=45d
- --web.console.libraries=/usr/share/prometheus/console_libraries
- --web.console.templates=/usr/share/prometheus/consoles
- --web.enable-admin-api
ports:
- 9090:9090
volumes:
- /var/lib/docker/volumes/endlessh_prometheus/_data/prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus:/prometheus
restart: always
stdin_open: true # docker run -i
tty: true # docker run -t
volumes:
prometheus:
Am I doing something wrong, or is there a bug somewhere in the containers?
We may use Prometheus increase
fuction to speed up loading the clients.
But there are clients missing due to how Grafana samples the data.
The increase
function generate a pulse whose width is half of the sample period.
e.g.
if Ganfana samples at time -4, -2, 0, 2, 4
Then the the pulse generated by increase will be last from 0-1, or 1-2. The pulse at 1-2 will not be captured by Grafana.
Because the number of countries is more bounded than IP.
We may still keep the IP field in the metric, but report countries in that field, to keep dashboard working for both cases.
Hi, love the project,
Can you provide an example docker-compose with everything predefined, so we can get the same mertic flowing into Prometheus for Grafana?
Hello,
First I want to say this is a very cool and interesting app, I love the statistics output feature. Recently, however, my deployment started giving me the following errors:
W0329 14:54:20.989396 1 metrics.go:128] Failed to obatin the geohash of 114.204.218.154: invalid character '<' looking for beginning of value.
I0329 14:54:33.933339 1 client.go:99] CLOSE host=114.204.218.154 port=36798 time=13.00694231 bytes=170
I0329 14:55:00.133239 1 client.go:58] ACCEPT host=152.32.210.193 port=32194 n=2/4096
W0329 14:55:00.205932 1 metrics.go:128] Failed to obatin the geohash of 152.32.210.193: invalid character '<' looking for beginning of value.
I0329 14:55:13.139388 1 client.go:99] CLOSE host=152.32.210.193 port=32194 time=13.006089047 bytes=252
I0329 14:55:48.516476 1 client.go:58] ACCEPT host=103.103.0.216 port=54992 n=2/4096
W0329 14:55:48.582872 1 metrics.go:128] Failed to obatin the geohash of 103.103.0.216: invalid character '<' looking for beginning of value.
I0329 14:56:01.521648 1 client.go:99] CLOSE host=103.103.0.216 port=54992 time=13.005138886 bytes=164
This came seemingly out of nowhere as the pod was working just fine when I went to bed last night, but I woke up to these errors. There isn't an error before the pod switches from successfully geohashing to these errors.
Below is my deployment.yaml file for K8s:
apiVersion: apps/v1
kind: Deployment
metadata:
name: endlessh
annotations:
keel.sh/policy: all
labels:
app: endlessh
spec:
replicas: 1
selector:
matchLabels:
app: endlessh
template:
metadata:
labels:
app: endlessh
spec:
hostNetwork: true
nodeSelector:
kubernetes.io/hostname: obsidiana
containers:
- name: endlessh
image: shizunge/endlessh-go
args:
- "-logtostderr"
- "-v=1"
- "-enable_prometheus"
- "-port=22420"
- "-geoip_supplier=ip-api"
The systemd service is not able to resolve the geohash API host. Oddly it's pointing to localhost to try and resolve it. resolveconf.service
is enabled and active. There are valid nameservers in /etc/resolv.conf
. I can dig ip-api.com
just fine. Here's the systemd status for endlessh-go:
● endlessh-go.service - SSH tarpit
Loaded: loaded (/etc/systemd/system/endlessh-go.service; enabled; preset: enabled)
Active: active (running) since Thu 2024-05-02 15:22:11 UTC; 54min ago
Main PID: 271204 (endlessh-go)
IP: 170.9K in, 533.3K out
IO: 6.3M read, 0B written
Tasks: 5 (limit: 1152)
Memory: 11.4M
CPU: 884ms
CGroup: /system.slice/endlessh-go.service
└─271204 /nix/store/i4kqp7h5n5j2avwj6gjjjihhg6axxxqz-endlessh-go-20230625-3/bin/endlessh-go -logtostderr -host=x.x.x.x -port=22 -enable_prometheus -prometheus_host=x.x.x.x -prometheus_port=2112 -geoip_supplier=ip-api
May 02 16:06:01 nixos endlessh-go[271204]: W0502 16:06:01.326615 271204 client.go:60] Failed to obatin the geohash of 218.92.0.97: Get "http://ip-api.com/json/218.92.0.97": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:34947->[::1]:53: read: connection refused.
May 02 16:11:30 nixos endlessh-go[271204]: W0502 16:11:30.645957 271204 client.go:60] Failed to obatin the geohash of 180.101.88.196: Get "http://ip-api.com/json/180.101.88.196": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:34406->[::1]:53: read: connection refused.
May 02 16:11:57 nixos endlessh-go[271204]: W0502 16:11:57.753493 271204 client.go:60] Failed to obatin the geohash of 218.92.0.96: Get "http://ip-api.com/json/218.92.0.96": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:53675->[::1]:53: read: connection refused.
May 02 16:12:19 nixos endlessh-go[271204]: W0502 16:12:19.649602 271204 client.go:60] Failed to obatin the geohash of 218.92.0.113: Get "http://ip-api.com/json/218.92.0.113": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:50887->[::1]:53: read: connection refused.
May 02 16:12:50 nixos endlessh-go[271204]: W0502 16:12:50.385315 271204 client.go:60] Failed to obatin the geohash of 94.203.171.157: Get "http://ip-api.com/json/94.203.171.157": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:46093->[::1]:53: read: connection refused.
May 02 16:13:07 nixos endlessh-go[271204]: W0502 16:13:07.635677 271204 client.go:60] Failed to obatin the geohash of 183.129.208.82: Get "http://ip-api.com/json/183.129.208.82": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:47188->[::1]:53: read: connection refused.
May 02 16:13:12 nixos endlessh-go[271204]: W0502 16:13:12.766532 271204 client.go:60] Failed to obatin the geohash of 121.135.254.129: Get "http://ip-api.com/json/121.135.254.129": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:39893->[::1]:53: read: connection refused.
May 02 16:13:22 nixos endlessh-go[271204]: W0502 16:13:22.202812 271204 client.go:60] Failed to obatin the geohash of 218.92.0.96: Get "http://ip-api.com/json/218.92.0.96": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:41506->[::1]:53: read: connection refused.
May 02 16:14:44 nixos endlessh-go[271204]: W0502 16:14:44.648870 271204 client.go:60] Failed to obatin the geohash of 218.92.0.96: Get "http://ip-api.com/json/218.92.0.96": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:45123->[::1]:53: read: connection refused.
May 02 16:16:06 nixos endlessh-go[271204]: W0502 16:16:06.965790 271204 client.go:60] Failed to obatin the geohash of 218.92.0.96: Get "http://ip-api.com/json/218.92.0.96": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:45047->[::1]:53: read: connection refused.
Running on NixOS 23.11.
Relevant config:
services.endlessh-go = {
enable = true;
listenAddress = "<PUBLIC IP>";
port = 22;
prometheus = {
enable = true;
listenAddress = "<VPN IP>";
};
extraOptions = [
"-geoip_supplier=ip-api"
];
};
Create integration tests and run via GitHub workflow.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.