shizunge / endlessh-go Goto Github PK

I see that endlessh-go is using glog for logging along with gflags (I think?) to pass options as args. Some of endlessh-go's args are pulled directly from logging.cc but when I try to use flags not listed, timestamp_in_logfile_name and max_log_size, I get an error.

flag provided but not defined: -max_log_size
Usage of /endlessh 
  -alsologtostderr
...

I don't see where in endlessh-go's code where the logging flags are defined... Would it be possible to use these additional flags? I'm try to configure basic log rotation. Thanks!

Hi. how can I use ./endlessh-go options ?
for example how to use -geoip_supplier option ?

thanks.

$> docker --version
Docker version 24.0.7, build afdd53b4e3
$> sudo docker pull ghcr.io/shizunge/endlessh-go:latest
latest: Pulling from shizunge/endlessh-go
manifest unknown

Pulling from Docker Hub works fine.

The Grafana locations panel don't show the client, if you use the MaxMind offline DB.
"Time","name","country","geohash","instance","ip","job","location","Value #Seen"
2022-12-19 11:39:00,endlessh_client_open_count,China,s000,192.168.10.214:2112,106.13.29.110,endlessh,China,3
2022-12-20 03:35:00,endlessh_client_open_count,China,s000,192.168.10.214:2112,111.67.202.249,endlessh,China,28
2022-12-19 15:52:00,endlessh_client_open_count,China,s000,192.168.10.214:2112,124.223.26.191,endlessh,China,37
2022-12-20 09:22:00,endlessh_client_open_count,China,s000,192.168.10.214:2112,163.53.90.10,endlessh,China,1
2022-12-19 16:50:00,endlessh_client_open_count,China,s000,192.168.10.214:2112,175.178.40.24,endlessh,China,36

i assume, geohash shouldn't be "S000".
Any idea, how to fix it?

I'm working on an enhanced crowdsec collection for endlessh-go as well as a basic notifier to discord. Neither of these can take advantage of prometheus exports directly and instead need to use the log file generated to parse events. It would be helpful if the geo information parsed for prometheus metrics was included in the log lines (on ACCEPT) so that I don't have to make another call to ip-api to get the same information again.

Add capability to use offline IP-GEO database (e.g. maxmind)

Hi,

any chance of getting an example on how to use max-mind-db on endlessh-go?

Because there are three different db on max-mind an each comes in two flavors.

The linked online alternatives IP-API.com and freegeoip.live are offline.

-geoip_supplier=max-mind-db -max_mind_db=/absolute/path/to/GeoLite2-Country.mmdb
-geoip_supplier=max-mind-db -max_mind_db=./GeoLite2-Country.mmdb
-geoip_supplier=max-mind-db -max_mind_db ./GeoLite2-Country.mmdb

I guess I am using the wrong database because none of them worked...

could you point me in the right direction? Thankx.

Clients data did not display correctly in Grafana 8.3.4. Any thing depends on clients also shows NaN.

It is working with Grafana 8.3.3

No problem for Sent Bytes, Unique IPs and Current Connections.

For example, to make it compatible with systemd's socket activation.

i get every time the fault

is this a fault on my side?
if yes, how can i fix the problem?

https://github.com/shizunge/endlessh-go/tree/main/examples

func geohashAndLocationFromFreegeoip(address string) (string, string, string, error) {
var geo freegeoip
response, err := http.Get("https://freegeoip.live/json/" + address)

As mentioned here https://freegeoip.io/ the api call should be different: freegeoip.io/{format}/{IP_or_hostname}.

hello, i'm using unraid 6.9.2 and i tried your endlessh-go docker. I run it like this:

sudo docker run -d -p 2222:2222 shizunge/endlessh-go -logtostderr -v=1 -enable_prometheus

your tool is running, i get "trapped" on port 2222 when trying to connect via ssh.
but i dont get managed to get the prometheus metrics from it and i dont know why.
looking at the log, all seems to be fine - it starts and logs my ssh connection attemp:

I0705 04:11:18.579285 1 main.go:102] Starting Prometheus on 0.0.0.0:2112, entry point is /metrics
I0705 04:11:18.580008 1 main.go:142] Listening on 0.0.0.0:2222
I0705 04:13:29.392797 1 client.go:69] ACCEPT host=192.168.2.2 port=1693 n=1/4096
I0705 04:13:34.395198 1 client.go:109] CLOSE host=192.168.2.2 port=1693 time=5.002307282 bytes=71

In the "Clients" Panel, i get an error saying:

"v is undefined".

Do you have any idea what's that all about?

I'm unable to enable logging to a file so my Crowdsec instance could parse those logs and issue bans.
Workaround is to use syslog logging driver of the Docker and make Crowdsec parse syslog, but IMO its not optimal even though it works.

This is my docker compose, basically its the maxmind example from here. I've tried multiple other variations of commands, that I could think of, but no log file appears on host system at /portainer/Files/AppData/Config/endlessh-go/logs/ -folder.

version: '3.5'
services:
  endlessh:
    container_name: endlessh
    image: shizunge/endlessh-go:latest
    restart: unless-stopped
    user: root
    command:
      - "-log_dir=/log/"
      - "-interval_ms=1000"
      - "-v=1"
      - "-enable_prometheus"
      - "-geoip_supplier=max-mind-db"
      - "-max_mind_db=/geo-data/GeoLite2-City.mmdb"
      - "-alsologtostderr"
    networks:
      - default
    hostname: endlessh
    ports:
      - 2222:2222 # SSH port
      - 2112:2112 # Prometheus metrics port
    volumes:
      - /portainer/Files/AppData/Config/endlessh-go/logs/:/log:ro # logs
      - /portainer/Files/AppData/Config/maxmind/geo-data/:/geo-data/:ro # geoip data
    logging:
      driver: "syslog"

...continues with maxmind portion

By default the info messages about ACCEPT and CLOSE of this service are listed as errors in my systemd journal.
Which results in spamming me with log reports since there are many attempts in my honeypot.

I thought these are level 1 and the default stderrthreshold is 2.

Endlessh-go reports country, location, and geohash data as unknown; after looking at the systemd page, I see this error message: Failed to obatin the geohash of [ipv4 redacted]: open /GeoLite2-Country.mmdb: no such file or directory.

when running ls -lh /GeoLite2-Country.mmdb I get: -rw-r--r-- 1 root root 6.2M Mar 5 13:41 /GeoLite2-Country.mmdb, however even after running chmod 777 /GeoLite2-Country.mmdb I still get the same error. I'm running in bare metal (not docker) so there shouldn't be anything preventing it from accessing the database file. Does it need to be owned by a specific user? Does it need to be in a specific directory/have a specific name?

If it helps with debugging, I'm running this on nixOS 22.11.3810.3d302c67ab8 (due to using nixos & endlessh-go not having a --version flag, I can only say that I'm running the latest version as of 2023/08/23). I have my service configuration as follows:

services.endlessh-go = {
	
	enable = true;
	port = 22;
	openFirewall = true;
	extraOptions = [
		"-enable_prometheus"
		"-prometheus_entry ''"
		"-geoip_supplier 'max-mind-db'"
		"-max_mind_db '/GeoLite2-Country.mmdb"
	];
	prometheus = {
		
		enable = false;
		listenAddress = "127.0.0.1";
		port = 2112;
	};
};

I am trying to get the logs so fail2ban can read them, but wondered where the logs to to so i can bind them to a volume/mount.

version: '3.5'
services:
  endlessh:
    container_name: tarpit-endlessh
    image: shizunge/endlessh-go:latest
    restart: unless-stopped
    command:
      - -interval_ms=1000
      - -logtostderr
      - -v=1
      - -enable_prometheus
      - -geoip_supplier=ip-api
      - -log_dir=/example
    ports:
      - 22:2222
#      - 9101:2112 #prometheus port (now handled via networks)
    networks:
    - default
    - prometheus
    volumes:
    - /home/user/example:/example

networks:
  prometheus:
    external: true

Here i tried to put it into /example of the container and then ls /home/user/example, but nothing was logged there.
I am probably doing something wrong so if you see what mistake i made could you please point it out to me, thanks :)

Hi, some metrics are missing from grafana dashboard.

Grafana 8.3.4
Docker: '-logtostderr' '-v=1' '-enable_prometheus' '-geoip_supplier=freegeoip' - Can confirm that IP's, country and trap times metrics are getting to prometheus:2112/metrics (image below).

Nice project. I converted to using this one from the linuxserver.io one (https://docs.linuxserver.io/images/docker-endlessh/) because you have prometheus metrics built in.. thanks for your work on this

I think that there is something a bit odd about the actual counting of connections / time spent.

From the prometheus output:

endlessh_client_trapped_time_seconds{ip="218.92.0.125",local_port="2222"} 5242.06399999999

but my container has only been running a short while

$ docker ps | grep end
3e32fbf10ef8   shizunge/endlessh-go                       "/endlessh -enable_p…"   30 minutes ago   Up 30 minutes

30mins = 1800 seconds
I don't see how I can have 5242 seconds of 'trapped time'

Here is the full prometheus output

# HELP endlessh_client_closed_count_total Total number of clients that stopped connecting to this host.
# TYPE endlessh_client_closed_count_total counter
endlessh_client_closed_count_total{local_port="2222"} 29
# HELP endlessh_client_open_count Number of connections of clients.
# TYPE endlessh_client_open_count counter
endlessh_client_open_count{country="China",geohash="wtssb2djg",ip="218.92.0.125",local_port="2222",location="China, Jiangsu, Nanjing"} 33
endlessh_client_open_count{country="China",geohash="wx4g0kzf1",ip="54.222.238.164",local_port="2222",location="China, Beijing, Beijing"} 1
# HELP endlessh_client_open_count_total Total number of clients that tried to connect to this host.
# TYPE endlessh_client_open_count_total counter
endlessh_client_open_count_total{local_port="2222"} 34
# HELP endlessh_client_trapped_time_seconds Seconds a client spends on endlessh.
# TYPE endlessh_client_trapped_time_seconds counter
endlessh_client_trapped_time_seconds{ip="218.92.0.125",local_port="2222"} 5242.06399999999
endlessh_client_trapped_time_seconds{ip="54.222.238.164",local_port="2222"} 938.4460000000003
# HELP endlessh_sent_bytes_total Total bytes sent to clients that tried to connect to this host.
# TYPE endlessh_sent_bytes_total counter
endlessh_sent_bytes_total{local_port="2222"} 101505
# HELP endlessh_trapped_time_seconds_total Total seconds clients spent on endlessh.
# TYPE endlessh_trapped_time_seconds_total counter
endlessh_trapped_time_seconds_total{local_port="2222"} 6180.5099999999875

(so.. maybe there is a concurrent connection thing happening so that's what's up?).. but the Grafana dashboard (where I noticed this first) -- just seems way off?

Important to note - that the data was captured a few minutes apart - so there may be differences in the Prometheus data dump vs. the Grafana image capture -- but only a minute or two difference.

Add the ability to use a yaml/toml configuration file.
To my point of view cli_args should either define a configuration file either define args directly.

I tried to add example like parameters on hub.docker.com like e= but it dosnt work.

It should have maybe have a volume (v:) for config or enviroments (e:) directly in docker run . There should at least be a volume options for logs.

-alsologtostderr
log to standard error as well as files
-conn_type string
Connection type. Possible values are tcp, tcp4, tcp6 (default "tcp")
-enable_prometheus
Enable prometheus
-geoip_supplier string
Supplier to obtain Geohash of IPs. Possible values are "off", "ip-api", "max-mind-db" (default "off")
-host string
SSH listening address (default "0.0.0.0")
-interval_ms int
Message millisecond delay (default 1000)
-line_length int
Maximum banner line length (default 32)
-log_backtrace_at value
when logging hits line file:N, emit a stack trace
-log_dir string
If non-empty, write log files in this directory
-log_link string
If non-empty, add symbolic links in this directory to the log files
-logbuflevel int
Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only; ...). Has limited applicability on non-prod platforms.
-logtostderr
log to standard error instead of files
-max_clients int
Maximum number of clients (default 4096)
-max_mind_db string
Path to the MaxMind DB file.
-port value
SSH listening port. You may provide multiple -port flags to listen to multiple ports. (default "2222")
-prometheus_clean_unseen_seconds int
Remove series if the IP is not seen for the given time. Set to 0 to disable. (default 0)
-prometheus_entry string
Entry point for prometheus (default "metrics")
-prometheus_host string
The address for prometheus (default "0.0.0.0")
-prometheus_port string
The port for prometheus (default "2112")
-stderrthreshold value
logs at or above this threshold go to stderr (default 2)
-v value
log level for V logs
-vmodule value
comma-separated list of pattern=N settings for file-filtered logging

The usage of CLI arguments works well for binary usage but with docker images it would be better to propose environments variables.
We could imagine something like:
cli_args if not undefined else anv_agrs if not undefined else default

I had the error as shown in the screenshot above and tried to completely rebuild and deploy all my docker containers and the Grafana dashboard. I also deleted all storage volumes and completely rebuilt the stack, but unfortunately the error did not disappear. Here is my docker stack which I run in my portainer:

version: '3.8'
services:
  endlessh:
    image: shizunge/endlessh-go:latest
    container_name: endlessh
    command:
      - -interval_ms=1000
      - -logtostderr
      - -v=1
      - -enable_prometheus
      - -geoip_supplier=ip-api
    ports:
      # SSH port
      - 2222:2222
      # Prometheus metrics port
      - 2112:2112
    restart: always
    stdin_open: true # docker run -i
    tty: true # docker run -t

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    command: 
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=45d
      - --web.console.libraries=/usr/share/prometheus/console_libraries
      - --web.console.templates=/usr/share/prometheus/consoles
      - --web.enable-admin-api
    ports:
      - 9090:9090
    volumes:
      - /var/lib/docker/volumes/endlessh_prometheus/_data/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus:/prometheus
    restart: always
    stdin_open: true # docker run -i
    tty: true # docker run -t
volumes:
  prometheus:

Am I doing something wrong, or is there a bug somewhere in the containers?

We may use Prometheus increase fuction to speed up loading the clients.

But there are clients missing due to how Grafana samples the data.

The increase function generate a pulse whose width is half of the sample period.
e.g.
if Ganfana samples at time -4, -2, 0, 2, 4
Then the the pulse generated by increase will be last from 0-1, or 1-2. The pulse at 1-2 will not be captured by Grafana.

Hi,

I installed the 3 containers and the router forwards external Port 22 to dockers POrt 2222.

After hours Grafana only shows the docker own IP 172.20.0.1
How can I prevent logging this internal IP?

Because the number of countries is more bounded than IP.

We may still keep the IP field in the metric, but report countries in that field, to keep dashboard working for both cases.

Hi, love the project,
Can you provide an example docker-compose with everything predefined, so we can get the same mertic flowing into Prometheus for Grafana?

Hello,

First I want to say this is a very cool and interesting app, I love the statistics output feature. Recently, however, my deployment started giving me the following errors:

W0329 14:54:20.989396       1 metrics.go:128] Failed to obatin the geohash of 114.204.218.154: invalid character '<' looking for beginning of value.
I0329 14:54:33.933339       1 client.go:99] CLOSE host=114.204.218.154 port=36798 time=13.00694231 bytes=170
I0329 14:55:00.133239       1 client.go:58] ACCEPT host=152.32.210.193 port=32194 n=2/4096
W0329 14:55:00.205932       1 metrics.go:128] Failed to obatin the geohash of 152.32.210.193: invalid character '<' looking for beginning of value.
I0329 14:55:13.139388       1 client.go:99] CLOSE host=152.32.210.193 port=32194 time=13.006089047 bytes=252
I0329 14:55:48.516476       1 client.go:58] ACCEPT host=103.103.0.216 port=54992 n=2/4096
W0329 14:55:48.582872       1 metrics.go:128] Failed to obatin the geohash of 103.103.0.216: invalid character '<' looking for beginning of value.
I0329 14:56:01.521648       1 client.go:99] CLOSE host=103.103.0.216 port=54992 time=13.005138886 bytes=164

This came seemingly out of nowhere as the pod was working just fine when I went to bed last night, but I woke up to these errors. There isn't an error before the pod switches from successfully geohashing to these errors.

Below is my deployment.yaml file for K8s:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: endlessh
  annotations:
    keel.sh/policy: all
  labels:
    app: endlessh
spec:
  replicas: 1
  selector:
    matchLabels:
      app: endlessh
  template:
    metadata:
      labels:
        app: endlessh
    spec:
      hostNetwork: true
      nodeSelector:
        kubernetes.io/hostname: obsidiana
      containers:
        - name: endlessh  
          image: shizunge/endlessh-go
          args: 
            - "-logtostderr"
            - "-v=1"
            - "-enable_prometheus"
            - "-port=22420"
            - "-geoip_supplier=ip-api"

The systemd service is not able to resolve the geohash API host. Oddly it's pointing to localhost to try and resolve it. resolveconf.service is enabled and active. There are valid nameservers in /etc/resolv.conf. I can dig ip-api.com just fine. Here's the systemd status for endlessh-go:

● endlessh-go.service - SSH tarpit
     Loaded: loaded (/etc/systemd/system/endlessh-go.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-05-02 15:22:11 UTC; 54min ago
   Main PID: 271204 (endlessh-go)
         IP: 170.9K in, 533.3K out
         IO: 6.3M read, 0B written
      Tasks: 5 (limit: 1152)
     Memory: 11.4M
        CPU: 884ms
     CGroup: /system.slice/endlessh-go.service
             └─271204 /nix/store/i4kqp7h5n5j2avwj6gjjjihhg6axxxqz-endlessh-go-20230625-3/bin/endlessh-go -logtostderr -host=x.x.x.x -port=22 -enable_prometheus -prometheus_host=x.x.x.x -prometheus_port=2112 -geoip_supplier=ip-api

May 02 16:06:01 nixos endlessh-go[271204]: W0502 16:06:01.326615  271204 client.go:60] Failed to obatin the geohash of 218.92.0.97: Get "http://ip-api.com/json/218.92.0.97": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:34947->[::1]:53: read: connection refused.
May 02 16:11:30 nixos endlessh-go[271204]: W0502 16:11:30.645957  271204 client.go:60] Failed to obatin the geohash of 180.101.88.196: Get "http://ip-api.com/json/180.101.88.196": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:34406->[::1]:53: read: connection refused.
May 02 16:11:57 nixos endlessh-go[271204]: W0502 16:11:57.753493  271204 client.go:60] Failed to obatin the geohash of 218.92.0.96: Get "http://ip-api.com/json/218.92.0.96": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:53675->[::1]:53: read: connection refused.
May 02 16:12:19 nixos endlessh-go[271204]: W0502 16:12:19.649602  271204 client.go:60] Failed to obatin the geohash of 218.92.0.113: Get "http://ip-api.com/json/218.92.0.113": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:50887->[::1]:53: read: connection refused.
May 02 16:12:50 nixos endlessh-go[271204]: W0502 16:12:50.385315  271204 client.go:60] Failed to obatin the geohash of 94.203.171.157: Get "http://ip-api.com/json/94.203.171.157": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:46093->[::1]:53: read: connection refused.
May 02 16:13:07 nixos endlessh-go[271204]: W0502 16:13:07.635677  271204 client.go:60] Failed to obatin the geohash of 183.129.208.82: Get "http://ip-api.com/json/183.129.208.82": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:47188->[::1]:53: read: connection refused.
May 02 16:13:12 nixos endlessh-go[271204]: W0502 16:13:12.766532  271204 client.go:60] Failed to obatin the geohash of 121.135.254.129: Get "http://ip-api.com/json/121.135.254.129": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:39893->[::1]:53: read: connection refused.
May 02 16:13:22 nixos endlessh-go[271204]: W0502 16:13:22.202812  271204 client.go:60] Failed to obatin the geohash of 218.92.0.96: Get "http://ip-api.com/json/218.92.0.96": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:41506->[::1]:53: read: connection refused.
May 02 16:14:44 nixos endlessh-go[271204]: W0502 16:14:44.648870  271204 client.go:60] Failed to obatin the geohash of 218.92.0.96: Get "http://ip-api.com/json/218.92.0.96": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:45123->[::1]:53: read: connection refused.
May 02 16:16:06 nixos endlessh-go[271204]: W0502 16:16:06.965790  271204 client.go:60] Failed to obatin the geohash of 218.92.0.96: Get "http://ip-api.com/json/218.92.0.96": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:45047->[::1]:53: read: connection refused.

Running on NixOS 23.11.
Relevant config:

services.endlessh-go = {
  enable = true;
  listenAddress = "<PUBLIC IP>";
  port = 22;
  prometheus = {
    enable = true;
    listenAddress = "<VPN IP>";
  };
  extraOptions = [
    "-geoip_supplier=ip-api"
  ];
};

Create integration tests and run via GitHub workflow.