Please add a client setting that allows it to keep retrying (forever or for a specified number of attempts) to connect to the server after a connect timeout (e.g. due to connectivity problem or server down).

Currently:

./client
panic: dial tcp [x.x.x.x]:8443: connect: connection timed out
goroutine 1 [running]:
main.main()
/home/runner/work/wsvpn/wsvpn/client/main.go:112 +0x950

I have some complicated network scenarios in which I need to run both wsvpn client and server on the same device. Since embedded devices may have very little flash storage, it would be great if we can have a single binary that can run as either a client or server. That will reduce the storage space.

binary --client or --server

Wsvpn server 1.5.1 keeps failing when wsvpn client 1.5.1 tries to connect to it. Here's the error on the server:

2022/06/14 13:11:49 [0] Error reading packet from WS: websocket: close 1006 (abnormal closure): unexpected EOF

However, if the wsvpn server is replaced with the version 1.4.0 while keeping the wsvpn client 1.5.1, it works.

CLIENT 1.5.1

wsvpn_client_arm_1.5.1 --auth-file $HTPASSWD --proxy http://$PROXY_SERVER:$PROXY_PORT
--tls-cipher-preference CHACHA --tls-min-version 1.3
--ping-interval $PING_INTERVAL --ping-timeout $PING_TIMEOUT
--ca-certificates $CA_CERT --connect wss://$WSVPN_SERVER_IP:$WSVPN_SERVER_PORT

2022/06/14 13:11:48 Connecting to wss://something:8443 as user bitso
2022/06/14 13:11:48 Using HTTP proxy http://somewhere:8080
2022/06/14 13:11:49 TLS 1.3 connection established with cipher=TLS_CHACHA20_POLY1305_SHA256
2022/06/14 13:11:49 [0] Ping enabled with interval 24h0m0s and timeout 10s
2022/06/14 13:11:49 [0] Error reading packet from WS: websocket: close 1006 (abnormal closure): unexpected EOF

SERVER 1.5.1

wsvpn_server_arm --authenticator htpasswd --listen 0.0.0.0:8443 --mtu 1460
--tls-cipher-preference CHACHA --tls-min-version 1.3
--ping-interval $PING_INTERVAL --ping-timeout $PING_TIMEOUT
--tls-cert $SERVER_CERT --tls-key $SERVER_KEY

2022/06/14 20:09:55 [S] VPN server online at 0.0.0.0:8443, mode TUN, serving subnet 192.168.3.0/24 (253 max clients) with MTU 1460

2022/06/14 20:11:49 http: panic serving somewhere:19955: unaligned 64-bit atomic operation
goroutine 6 [running]:
net/http.(*conn).serve.func1()
/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1825 +0xec
panic({0x29a588, 0x33484c})
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/panic.go:844 +0x2ac
runtime/internal/atomic.panicUnaligned()
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/internal/atomic/unaligned.go:8 +0x24
runtime/internal/atomic.Xadd64(0x20a6f2c, 0x1)
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/internal/atomic/atomic_arm.s:258 +0x14
github.com/Doridian/wsvpn/shared.(*Socket).SendCommand(0x20a6f00, {0x2ca85e, 0x4}, {0x20c3c8c, 0x3, 0x3})
/home/runner/work/wsvpn/wsvpn/shared/socketHandler.go:106 +0x34
main.serveWs({0x335fec, 0x20aa1e0}, 0x208e480)
/home/runner/work/wsvpn/wsvpn/server/main.go:252 +0x804
net/http.HandlerFunc.ServeHTTP(0x2e4adc, {0x335fec, 0x20aa1e0}, 0x208e480)
/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084 +0x34
net/http.(*ServeMux).ServeHTTP(0x4a7ec8, {0x335fec, 0x20aa1e0}, 0x208e480)
/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462 +0x164
net/http.serverHandler.ServeHTTP({0x20923f0}, {0x335fec, 0x20aa1e0}, 0x208e480)
/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916 +0x408
net/http.(*conn).serve(0x2182060, {0x336110, 0x20ca4b0})
/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966 +0x654
created by net/http.(*Server).Serve
/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:3071 +0x4e0

Won't work for TAP as TAP devices are shared between all clients, so cannot adjust per-client.
Could maybe adjust to lowest value between all clients, but that does not seem like a good idea.

Hi Mark,

Is there any additional MIPS arch that the go compiler can handle? I've tried all the mips images but none of them seems to work on the following platform (it's supposed to be 32-bits only, but I also ran 64-bits images for reference):

https://openwrt.org/docs/techref/instructionset/mipsel_74kc

system type : Qualcomm Atheros QCA9558 ver 1 rev 0
machine : TP-Link Archer C7 v2
processor : 0
cpu model : MIPS 74Kc V5.0
BogoMIPS : 359.42
wait instruction : yes
microsecond timers : yes
tlb_entries : 32
extra interrupt vector : yes
hardware watchpoint : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
isa : mips1 mips2 mips32r1 mips32r2
ASEs implemented : mips16 dsp dsp2
Options implemented : tlb 4kex 4k_cache prefetch mcheck ejtag llsc dc_aliases perf_cntr_intr_bit
cdmm contextconfig perf
shadow register sets : 1
kscratch registers : 0
package : 0
core : 0
VCED exceptions : not available
VCEI exceptions : not available

./server-linux-mips -h
Illegal instruction

./server-linux-mipsle -h
./server-linux-mipsle: line 1: syntax error: unexpected "("

./server-linux-mips64 -h
./server-linux-mips64: line 1: syntax error: unexpected ")"

./server-linux-mips64le -h
./server-linux-mips64le: line 1: syntax error: unterminated quoted string

When you have some time, please add the ability for wsvpn client and wsvpn server to invoke a config file with the command line settings in it, since there are lots of possible command line settings to set. Doing so would also help prevent other users from seeing the name/location of the htpasswd, certificate, private key files etc when doing "ps". Thanks a lot.

Specifically a problem in WebTransport if the MTU of the tunnel is too high

The existing release binaries don't have authentication support that you added later. Thanks a lot.

Hi Mark,

I've tested your new WSVPN 1.2.0.

If I specify the signing CA certificate "ca.crt" for the client's ca-certificates flag, it works as expected.

However, if I copy the same server identity certificate server.crt (used on the WSVPN server) to the WSVPN client and set it to the client's ca-certificates flag, the certificate validation also seems to work. It should only work if the matching ca.crt is specified.

CLIENT

sudo ./wsvpn_client_arm --auth-file htpasswd --proxy http://192.168.1.1:8080 --connect wss://192.168.100.1:8444
2022/06/11 14:12:18 Connecting to wss://192.168.100.1:8444 as user test
2022/06/11 14:12:18 Using HTTP proxy http://192.168.1.1:8080
panic: x509: certificate signed by unknown authority <---- No matching signing CA certificate in the local CA truststore.
goroutine 1 [running]:
main.main()
/home/runner/work/wsvpn/wsvpn/client/main.go:138 +0xc38

sudo ./wsvpn_client_arm --auth-file htpasswd --proxy http://192.168.1.1:8080 --ca-certificates ./server.crt --connect wss://192.168.100.1:8444
2022/06/11 13:58:13 Connecting to wss://192.168.100.1:8444 as user test
2022/06/11 13:58:13 Using HTTP proxy http://192.168.1.1:8080
2022/06/11 13:58:14 Network mode TUN, subnet 192.168.3.2/24, mtu 1280
2022/06/11 13:58:14 Opened tun0
2022/06/11 13:58:14 Configured interface, starting operations <-- Working even with server.crt

SERVER
./wsvpn_server_arm --authenticator htpasswd --listen 0.0.0.0:8444 --tls-cert ./server.crt -tls-key ./server.key

CERTIFICATE INFO

server.crt:
X509v3 Extended Key Usage:
TLS Web Server Authentication <===== Not a signing CA certificate
X509v3 Key Usage:
Digital Signature, Key Encipherment

ca.crt:
X509v3 Basic Constraints:
CA:TRUE
X509v3 Key Usage:
Certificate Sign, CRL Sign

I kill the server by pressing ctrl-c, then restart it. I expect the client to connect to the new instance after a while, but it fails to create a working VPN. Hopefully, the attached logs shine some light on the details. Also, I start both the client and the server on localhost. Ie., client.server is "webtransport://127.0.0.1:9000".

client.log
server.log

Hi Mark,

Different network environments (behind NAT, behind proxy, no NAT/no proxy etc.) may place different restrictions on Websocket connections. Based on my tests with the current WSVPN, it looks like the WSVPN server sends a PING frame to the client every 10 minutes. Can you please make the WSVPN server's PING frame interval configurable so it can be adjusted for different network environments.

For very reliable networks between WSVPN client and server with persistent VPN connection, we can configure the PING frame interval to be very long or disable it altogether. This will make the connection as silent as possible (kind of the same approach used by Wireguard). This will also help reduce battery power consumption for low power IoT devices running the WSVPN client etc.

This capability will make WSVPN the most incredibly versatile and flexible VPN solution in the whole world :-). Thanks so much!

Hi Mark,

Is there a way to force WSVPN to use only TLS 1.3 and use a particular TLS 1.3 only cipher such as TLS_CHACHA20_POLY1305_SHA256.

CHACHA20_POLY1305 is also used by Wireguard and is the fastest cipher on basic embedded general purpose CPU.

llvm-lipo looks good

Regarding 3.2.0:

1. What is component-id? It seems that there is a duplicate description for tun and tap interfaces in the client default config:

interface:
tap:
name: ""
persist: false
component-id: root\tap0901 # Defaults: root\tap0901 or tap0901 <====
tun:
name: ""
component-id: root\tap0901 # Defaults: root\tap0901 or tap0901 <====

2. Can I still use "auth-file" for the client yaml, e.g. Can you please add it to the example client config.

Thanks a lot.

./wsvpn_client_arm --print-default-config
tunnel:
set-default-gateway: false
ping:
interval: 30s
timeout: 5s

interface:
tap:
name: ""
persist: false
component-id: root\tap0901 # Defaults: root\tap0901 or tap0901
tun:
name: ""
component-id: root\tap0901 # Defaults: root\tap0901 or tap0901

scripts:

These scripts get run as "script operation subnet interface"

Example: "./handler.sh up 192.168.3.2/24 tun0"

up: ""
down: ""

client:
server: "" # Examples: ws://example.com:9000 wss://secure.example.com:9000
proxy: "" # Example: http://user:[email protected]:8080
tls:
ca: "" # Filename of CA bundle for verifying server cert
certificate: "" # Filename of certificate for mTLS
key: "" # Filename of private key for mTLS
config:
insecure: false
min-version: 1.2
max-version: 1.3
cipher-preference: "" # blank, AES or CHACHA

By itself, TLS client authentication is a secure authentication method for clients.

When used in combination with htpasswd authentication method, TLS client authentication can be treated as a form of 2-FA.

./server-linux-amd64 -h
WSVPN version 1.7.0
Usage of ./server-linux-amd64:
-authenticator string
Which authenticator to use (allow-all, htpasswd) (default "allow-all")
-authenticator-config string
Authebnticator config file (ex. htpasswd file for htpasswd authenticator, empty for default)

WebTransport is a new protocol using HTTP/3 aka QUIC, which uses UDP.
This would allow the VPN to use unrealiable API to send ethernet packets and rely on the underlying network to retry (such as the encapsulated TCP streams).
Once libraries emerge making this possible, will add option to server to enable WebTransport and option in client to enable using it.
Will be opt-in due to how many networks might not pass UDP traffic well.

I compiled your new 3.12.2 version by running build.sh. Besides the wsvpn client and server docker images, it created a bunch of nameless/tagless docker images as seen below.

docker --version
Docker version 20.10.17, build 100c70180f

docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
ghcr.io/doridian/wsvpn/client 3.12.2-1-g52b1f27 934e61782b4f 2 minutes ago 9.38MB
ghcr.io/doridian/wsvpn/server 3.12.2-1-g52b1f27 2bb7c38112de 2 minutes ago 10.4MB
196b0cf1ffaf 2 minutes ago 9.45MB
80f803ce84d3 2 minutes ago 10.4MB
69e435eb927f 2 minutes ago 9.45MB
c2efdf7b024d 2 minutes ago 10.4MB
640316a6c963 2 minutes ago 9.45MB
316116944507 2 minutes ago 10.5MB
f3d0ff975532 2 minutes ago 9.8MB
47fb1a1a04b1 2 minutes ago 10.8MB
98b70f008a64 2 minutes ago 9.55MB
1a34837575bc 2 minutes ago 10.5MB

Building for: windows / 386 / 386
Building for: windows / amd64 / amd64
Building for: windows / arm64 / arm64
Building for: linux / 386 / 386
Building for: linux / amd64 / amd64
Building for: linux / arm / arm
Building for: linux / arm / arm32v5
Building for: linux / arm / arm32v6
Building for: linux / arm / arm32v7
Building for: linux / arm64 / arm64
Building for: linux / mips / mips
Building for: linux / mips-softfloat / mips-softfloat
Building for: linux / mipsle / mipsle
Building for: linux / mipsle-softfloat / mipsle-softfloat
Building for: linux / mips64 / mips64
Building for: linux / mips64le / mips64le
Building for: darwin / amd64 / amd64
Building for: darwin / arm64 / arm64
[+] Building 0.5s (5/5) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile.server                                                                      0.1s
 => => transferring dockerfile: 253B                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                0.2s
 => => transferring context: 2B                                                                                                  0.0s
 => [internal] load build context                                                                                                0.2s
 => => transferring context: 10.54MB                                                                                             0.1s
 => CACHED [1/1] COPY --chown=0:0 --chmod=755 dist/server-linux-386 /server                                                      0.0s
 => exporting to image                                                                                                           0.0s
 => => exporting layers                                                                                                          0.0s
 => => writing image sha256:1a34837575bce739310a0d51eba19d80e71cd33efc16aae8b1d5c66ee25209c8                                     0.0s
 => => naming to ghcr.io/doridian/wsvpn/server:3.12.2-1-g52b1f27                                                                 0.0s
[+] Building 0.5s (5/5) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile.client                                                                      0.1s
 => => transferring dockerfile: 253B                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                0.1s
 => => transferring context: 2B                                                                                                  0.0s
 => [internal] load build context                                                                                                0.2s
 => => transferring context: 9.55MB                                                                                              0.1s
 => CACHED [1/1] COPY --chown=0:0 --chmod=755 dist/client-linux-386 /client                                                      0.0s
 => exporting to image                                                                                                           0.0s
 => => exporting layers                                                                                                          0.0s
 => => writing image sha256:98b70f008a64dd5e60c9bd7e4120cb9821f2cb627f76f1f340ff0fdaef260894                                     0.0s
 => => naming to ghcr.io/doridian/wsvpn/client:3.12.2-1-g52b1f27                                                                 0.0s
[+] Building 0.5s (5/5) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile.server                                                                      0.1s
 => => transferring dockerfile: 253B                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                0.1s
 => => transferring context: 2B                                                                                                  0.0s
 => [internal] load build context                                                                                                0.2s
 => => transferring context: 10.85MB                                                                                             0.1s
 => CACHED [1/1] COPY --chown=0:0 --chmod=755 dist/server-linux-amd64 /server                                                    0.0s
 => exporting to image                                                                                                           0.0s
 => => exporting layers                                                                                                          0.0s
 => => writing image sha256:47fb1a1a04b1677a96a232b5f9b2820b61dbd22c77271a2268c8a24bcf605603                                     0.0s
 => => naming to ghcr.io/doridian/wsvpn/server:3.12.2-1-g52b1f27                                                                 0.0s
[+] Building 0.5s (5/5) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile.client                                                                      0.1s
 => => transferring dockerfile: 253B                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                0.1s
 => => transferring context: 2B                                                                                                  0.0s
 => [internal] load build context                                                                                                0.2s
 => => transferring context: 9.81MB                                                                                              0.1s
 => CACHED [1/1] COPY --chown=0:0 --chmod=755 dist/client-linux-amd64 /client                                                    0.0s
 => exporting to image                                                                                                           0.0s
 => => exporting layers                                                                                                          0.0s
 => => writing image sha256:f3d0ff975532238108f7ba6d437041929b5dba31550fb7e09f6c40ce452ab9b7                                     0.0s
 => => naming to ghcr.io/doridian/wsvpn/client:3.12.2-1-g52b1f27                                                                 0.0s
[+] Building 0.5s (5/5) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile.server                                                                      0.1s
 => => transferring dockerfile: 253B                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                0.2s
 => => transferring context: 2B                                                                                                  0.0s
 => [internal] load build context                                                                                                0.2s
 => => transferring context: 10.47MB                                                                                             0.1s
 => CACHED [1/1] COPY --chown=0:0 --chmod=755 dist/server-linux-arm32v5 /server                                                  0.0s
 => exporting to image                                                                                                           0.0s
 => => exporting layers                                                                                                          0.0s
 => => writing image sha256:3161169445073fe7f2ec02048dd8449e0a79697cfd4cca9786eb73137280a4ba                                     0.0s
 => => naming to ghcr.io/doridian/wsvpn/server:3.12.2-1-g52b1f27                                                                 0.0s
[+] Building 0.5s (5/5) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile.client                                                                      0.1s
 => => transferring dockerfile: 253B                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                0.1s
 => => transferring context: 2B                                                                                                  0.0s
 => [internal] load build context                                                                                                0.2s
 => => transferring context: 9.45MB                                                                                              0.1s
 => CACHED [1/1] COPY --chown=0:0 --chmod=755 dist/client-linux-arm32v5 /client                                                  0.0s
 => exporting to image                                                                                                           0.0s
 => => exporting layers                                                                                                          0.0s
 => => writing image sha256:640316a6c96327231446222925a6fadf77a3a322dadc0919bed7c6fd10072c60                                     0.0s
 => => naming to ghcr.io/doridian/wsvpn/client:3.12.2-1-g52b1f27                                                                 0.0s
[+] Building 0.5s (5/5) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile.server                                                                      0.1s
 => => transferring dockerfile: 253B                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                0.1s
 => => transferring context: 2B                                                                                                  0.0s
 => [internal] load build context                                                                                                0.2s
 => => transferring context: 10.41MB                                                                                             0.1s
 => CACHED [1/1] COPY --chown=0:0 --chmod=755 dist/server-linux-arm32v6 /server                                                  0.0s
 => exporting to image                                                                                                           0.0s
 => => exporting layers                                                                                                          0.0s
 => => writing image sha256:c2efdf7b024d81f0d3e2deeda5600d2abd5bc62dfcc1b525443cc13ccd800846                                     0.0s
 => => naming to ghcr.io/doridian/wsvpn/server:3.12.2-1-g52b1f27                                                                 0.0s
[+] Building 0.8s (5/5) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile.client                                                                      0.1s
 => => transferring dockerfile: 253B                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                0.1s
 => => transferring context: 2B                                                                                                  0.0s
 => [internal] load build context                                                                                                0.2s
 => => transferring context: 9.45MB                                                                                              0.1s
 => CACHED [1/1] COPY --chown=0:0 --chmod=755 dist/client-linux-arm32v6 /client                                                  0.0s
 => exporting to image                                                                                                           0.2s
 => => exporting layers                                                                                                          0.0s
 => => writing image sha256:69e435eb927fbcc9316dae19be67f0ed0950c52731b936f6ee88fa638a068d4c                                     0.1s
 => => naming to ghcr.io/doridian/wsvpn/client:3.12.2-1-g52b1f27                                                                 0.1s
[+] Building 0.5s (5/5) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile.server                                                                      0.1s
 => => transferring dockerfile: 253B                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                0.2s
 => => transferring context: 2B                                                                                                  0.0s
 => [internal] load build context                                                                                                0.2s
 => => transferring context: 10.40MB                                                                                             0.1s
 => CACHED [1/1] COPY --chown=0:0 --chmod=755 dist/server-linux-arm32v7 /server                                                  0.0s
 => exporting to image                                                                                                           0.0s
 => => exporting layers                                                                                                          0.0s
 => => writing image sha256:80f803ce84d30af47dfa6feb206f0b9beb9f8a9179b7c7a084cdbb8ac2382f6c                                     0.0s
 => => naming to ghcr.io/doridian/wsvpn/server:3.12.2-1-g52b1f27                                                                 0.0s
[+] Building 0.4s (5/5) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile.client                                                                      0.1s
 => => transferring dockerfile: 253B                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                0.1s
 => => transferring context: 2B                                                                                                  0.0s
 => [internal] load build context                                                                                                0.1s
 => => transferring context: 9.45MB                                                                                              0.1s
 => CACHED [1/1] COPY --chown=0:0 --chmod=755 dist/client-linux-arm32v7 /client                                                  0.0s
 => exporting to image                                                                                                           0.0s
 => => exporting layers                                                                                                          0.0s
 => => writing image sha256:196b0cf1ffaf467d0cd17b31d5c1dac96c8bc87429a3d0c5a9c33b13e282ba5e                                     0.0s
 => => naming to ghcr.io/doridian/wsvpn/client:3.12.2-1-g52b1f27                                                                 0.0s
[+] Building 0.4s (5/5) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile.server                                                                      0.1s
 => => transferring dockerfile: 253B                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                0.1s
 => => transferring context: 2B                                                                                                  0.0s
 => [internal] load build context                                                                                                0.2s
 => => transferring context: 10.37MB                                                                                             0.1s
 => CACHED [1/1] COPY --chown=0:0 --chmod=755 dist/server-linux-arm64 /server                                                    0.0s
 => exporting to image                                                                                                           0.0s
 => => exporting layers                                                                                                          0.0s
 => => writing image sha256:2bb7c38112dedca5838291372ea4b627857112ec19099d5a2ece3bb5d2d6722a                                     0.0s
 => => naming to ghcr.io/doridian/wsvpn/server:3.12.2-1-g52b1f27                                                                 0.0s
[+] Building 0.5s (5/5) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile.client                                                                      0.1s
 => => transferring dockerfile: 253B                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                0.1s
 => => transferring context: 2B                                                                                                  0.0s
 => [internal] load build context                                                                                                0.2s
 => => transferring context: 9.39MB                                                                                              0.1s
 => CACHED [1/1] COPY --chown=0:0 --chmod=755 dist/client-linux-arm64 /client                                                    0.0s
 => exporting to image                                                                                                           0.0s
 => => exporting layers                                                                                                          0.0s
 => => writing image sha256:934e61782b4fed05caaff88be0c0394d8262b5e66ed3e5dadd6547e2667ee513                                     0.0s
 => => naming to ghcr.io/doridian/wsvpn/client:3.12.2-1-g52b1f27 

Is it possible to start the server/client so that it writes its secrets into a KeyLog file?

I've searched the project source files quickly and I'm guessing the answer is no. In that case, can you, please, add this functionality? I'm not really familiar with go and I'm guessing once again that this and this probably show how this can be done.

Thank you.

Hi @Doridian,

First of all, I really like the concept of "VPN over WebSocket", so I tried to run this project, but it doesn't seem to work properly. Is this project a complete project? Or is it an experiment?

I used this project as the VPN server, and then I wrote an iOS App myself as the VPN client.

The server program is stuck here, that is, all IP packets read from the client TUN disappeared after being written to the server TUN, and the server program unable to read any IP packets from the server TUN.

All IP packets read from the client TUN have the same source IP and are assigned here. This seems fine.

So, do you have any idea about the problem I encountered?

I'm not good at the VPN technique, so I really hope to get some suggestions from you, thank you! 😊

The power of WSVPN comes from its extreme flexibility and security. Forcing the server to use a single TUN interface for all clients takes away all the routing flexibility on the server itself. It also takes away the ability to quickly know how many clients are connected.

Please add the option to use single TUN or multiple TUN on the server. Thanks a lot.

We have arrived at a state where I am somewhat happy with the overall code structure.
Now is a good time to add unit/integration/E2E tests.

I would love help with this from any direction:

• Suggestions on what to test
• PRs implementing testing at any level
• etc

First target is probably to just run server and client binaries in two separate containers and run some packets (small and large) through the tunnel in various modes (TUN, TAP, WebSocket, WebTransport) and make sure they arrive intact.

Hello,

Please add HTTP proxy support for wsvpn client. It will make it work in virtually any network environment. Thanks a lot.

Typo errors: suffxed -> suffixed, ex. wstun -> e.g. wstun

-tun-naming-prefix string
Use specific naming prefix for TUN interfaces, suffxed with a number starting at 0 (ex. wstun)

==>

-tun-naming-prefix string
Use specific naming prefix for TUN interfaces (e.g. wstun), automatically suffixed with a number starting at 0

Hi Mark,

Is there any reason why you set the default MTU to 1280 bytes? During my testing, I can set it as high as 2000 bytes and such setting provides the best throughput.

When you plan to make any change or fix later, maybe the "ca" key under cilent tls should also be changed to "server-ca" to make it self-explanatory as that of the server tls.

server:
tls:
client-ca

client:
tls:
ca: <--- change it to "server-ca" ?

Hi Mark,
Can you give me some instructions on how to locally build WSVPN client and server binaries.

I cloned your git project and ran ./build.sh but got the following errors.

go version
go version go1.10.4 linux/amd64

$ ./build.sh
go: unknown subcommand "mod"
Run 'go help' for usage.
Building for: windows / 386
client/config.go:4:2: cannot find package "embed" in any of:
/usr/lib/go-1.10/src/embed (from $GOROOT)
/home/python/go/src/embed (from $GOPATH)
client/config.go:9:2: cannot find package "github.com/Doridian/wsvpn/client/clients" in any of:
/usr/lib/go-1.10/src/github.com/Doridian/wsvpn/client/clients (from $GOROOT)
/home/python/go/src/github.com/Doridian/wsvpn/client/clients (from $GOPATH)
client/config.go:10:2: cannot find package "github.com/Doridian/wsvpn/shared" in any of:
/usr/lib/go-1.10/src/github.com/Doridian/wsvpn/shared (from $GOROOT)
/home/python/go/src/github.com/Doridian/wsvpn/shared (from $GOPATH)
client/config.go:11:2: cannot find package "github.com/Doridian/wsvpn/shared/cli" in any of:
/usr/lib/go-1.10/src/github.com/Doridian/wsvpn/shared/cli (from $GOROOT)
/home/python/go/src/github.com/Doridian/wsvpn/shared/cli (from $GOPATH)
server/config.go:4:2: cannot find package "embed" in any of:
/usr/lib/go-1.10/src/embed (from $GOROOT)
/home/python/go/src/embed (from $GOPATH)
server/main.go:11:2: cannot find package "github.com/Doridian/wsvpn/server/authenticators" in any of:
/usr/lib/go-1.10/src/github.com/Doridian/wsvpn/server/authenticators (from $GOROOT)
/home/python/go/src/github.com/Doridian/wsvpn/server/authenticators (from $GOPATH)
server/main.go:12:2: cannot find package "github.com/Doridian/wsvpn/server/macswitch" in any of:
/usr/lib/go-1.10/src/github.com/Doridian/wsvpn/server/macswitch (from $GOROOT)
/home/python/go/src/github.com/Doridian/wsvpn/server/macswitch (from $GOPATH)
server/config.go:8:2: cannot find package "github.com/Doridian/wsvpn/server/servers" in any of:
/usr/lib/go-1.10/src/github.com/Doridian/wsvpn/server/servers (from $GOROOT)
/home/python/go/src/github.com/Doridian/wsvpn/server/servers (from $GOPATH)

Please add the option on the client to allow it to ignore the TLS SNI checking of the server certificate. The client may be in an environment that blocks DNS traffic (or sinks it to its own very restrictive DNS server). The server IP may be not static (but sequentially predictable in a certain range). I can't use a hostname/FQDN in such case.

In the server certificate, I added a few IP Addresses and DNS names in the its SAN field. Unfortunately, the actual IP address may keep changing, and I can't use the DNS name to connect the client to the server.

X509v3 Subject Alternative Name:
IP Address:192.168.1.1, IP Address:11.12.13.14, DNS:servername, DNS:servername.tld

I wondered about using webtransport in wsvpn as it seems the underlying quic protocol is capable to provide everything wsvpn requires from webtransport. However, http3 allows wsvpn to handle endpoints differently. For example, https://server.example.com:9000/ could return a webpage containing the current usage statistics and https://server.example.com:9000/vpn could be used to create a connection. Although, this feature might not be that useful. What do you think? Thank you.

I don't know golang but from a quick look at the code, it looks like wsvpn does not provide its own encryption and users are supposed to have their own application encryption (e.g ssh etc.) for the traffic flowing through it. Is it correct? Thanks a lot.

Please add the --version flag for both wsvpn client/server, or just print the version when showing the flag usage.

For example:

./wsvpn_client_amd64
Version 1.5.2 <===
Usage of ./wsvpn_client_amd64:
-auth-file string
...

Hi Mark,

Currently, the wsvpn client only validates the wsvpn server using its IP SAN attribute in its TLS certificate. The IP SAN attribute can always be fabricated in the server's TLS certificate.

I believe wsvpn client should definitely be enhanced to support the validation of the server's certificate using its signing ca certificate. That would render wsvpn much better from a security point of view. Without it, an adversary can always wage a MITM attack by setting the -authenticator allow-all to defeat the basic authentication.

Example of the new option:

wsvpn-client
-signing-ca-cert
Signing CA certificate of the server's TLS certificate

If auth is enabled as a new argument

I cannot run wsvpn in podman/docker on amd64. It turns out the binary is for arm64.

Pull:

podman pull --platform=linux/amd64 ghcr.io/doridian/wsvpn/server:3.13.0
Trying to pull ghcr.io/doridian/wsvpn/server:3.13.0...
Getting image source signatures
Copying blob 2f2410d3308a done  
Copying config 7b4af325aa done  
Writing manifest to image destination
Storing signatures
7b4af325aa0b755369156733b46943adb1ca0da00de78e50c79f2c0f0c5590b1

Run:

podman run --rm -v "$(pwd)"/config:/config ghcr.io/doridian/wsvpn/server:3.13.0
{"msg":"exec container process `/server`: Exec format error","level":"error","time":"2022-08-05T10:08:58.000664449Z"}

Similar to the flags -tls-cert, --tls-key for the server, or -auth-file, --ca-certificates for the client, users should be allowed to specify the path to the .htpasswd file on the server if the htpasswd authenticator is used. This helps with scripting without having to change to the path of wsvpn binary. Also, the server should expect a .htpasswd instead of htpasswd (to make it consistent with common practice).

For example:

./wsvpn_server_arm:
-authenticator string
Which authenticator to use (allow-all, htpasswd) <-- currently
Which authenticator to use (allow-all, path to .htpasswd file) (default "allow-all") <--- enhancement

AUTH_FILE=/root/secrets/.htpasswd
/root/wsvpn/wsvpn_server_arm --authenticator $AUTH_FILE

The desired outcome here is having two modes: "kill oldest connection" or "prevent new connections"

Hi there, I love the premise of this application. I have been trying to set it up and I've managed to get a secure connection between my NAS and Laptop (both Linux). Despite having the tunnel interfaces set up on both server and client, I'm having trouble getting all of my packets sent through the VPN.

If found this article (https://help.ubuntu.com/community/SSH_VPN - particularly the 'Plugging into the Network' bit) which talks about sending data over the tunnel interface. Despite this, it seems a bit old and outdated so I wondered if you had any advice on doing this - should I even need to do this or is it supposed to be automatic?

Many thanks,
Will

P.S. It took me quite a while to figure out how to set-up the mTLS authentication. I compiled a list of all the commands I used if you wish to review it and share it (below).

#!/bin/sh
# ============== Change these settings ============== #
# Certificate Authority settings
CA_NAME="lp-personal-ca"
COUNTRY_CODE="UK"
COUNTY="Hertfordshire" # Or region
LOCAL="Harpenden"
ORGANISATION="Lime Parallelogram"

# Client settings
CLIENT_NAME="will-ZenBook"

# Server Keys
SERVER_NAME="vpnserver" # Should not include spaces

# Edit [alt_names] section in the following block. Add optional IP.1 attribute.
cat > $SERVER_NAME.v3.ext << EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = example.com
EOF

# ============== No more settings to change ============== #
echo "You will now have to choose a password. This will be required every time you attemp to sign a new certificate using your authority. You will need to enter it 5 times duting this process."
# Generate the private key that is used to sign new keys
openssl genrsa -aes256 -out $CA_NAME.key 4096

# Generate a certificate for the new CA
openssl req -x509 -new -nodes -key $CA_NAME.key -sha256 -days 1826 -out $CA_NAME.crt -subj "/CN=$CA_NAME CA/C=$COUNTRY_CODE/ST=$COUNTY/L=$LOCAL/O=$ORGANISATION"

## - server - ##
# Generate key and CSR for server
openssl req -new -nodes -out $SERVER_NAME.csr -newkey rsa:4096 -keyout $SERVER_NAME.key -subj "/CN=$SERVER_NAME /C=$COUNTRY_CODE/ST=$COUNTY/L=$LOCAL/O=$ORGANISATION"

# Create signed certificate for the server using the CSR
openssl x509 -req -in $SERVER_NAME.csr -CA $CA_NAME.crt -CAkey $CA_NAME.key -CAcreateserial -out $SERVER_NAME.crt -days 730 -sha256 -extfile $SERVER_NAME.v3.ext

## - client - ##
# Generate key and CSR for client
openssl req -new -nodes -out $CLIENT_NAME.csr -newkey rsa:4096 -keyout $CLIENT_NAME.key -subj "/CN=$CLIENT_NAME /C=$COUNTRY_CODE/ST=$COUNTY/L=$LOCAL/O=$ORGANISATION"

# Create signed certificate for the client using the CSR
openssl x509 -req -in $CLIENT_NAME.csr -CA $CA_NAME.crt -CAkey $CA_NAME.key -CAcreateserial -out $CLIENT_NAME.crt -days 730 -sha256

# Clean up uneccessary files
rm *.csr
rm *.v3.ext

$ ./build.py
Starting: Go build client-linux-amd64
Starting: Go build client-linux-386
Starting: Go build client-linux-arm64
Starting: Go build client-linux-arm32v5
Done: Go build client-linux-arm64
Starting: Go build client-linux-arm32v6
Done: Go build client-linux-amd64
Starting: Go build client-linux-arm32v7
Done: Go build client-linux-386
Starting: Go build client-linux-mips
Done: Go build client-linux-arm32v5
Starting: Go build client-linux-mips-softfloat
Done: Go build client-linux-arm32v6
Starting: Go build client-linux-mipsle
Done: Go build client-linux-mips
Starting: Go build client-linux-mipsle-softfloat
Done: Go build client-linux-arm32v7
Starting: Go build client-linux-mips64
Done: Go build client-linux-mips-softfloat
Starting: Go build client-linux-mips64le
Done: Go build client-linux-mipsle-softfloat
Done: Go build client-linux-mipsle
Starting: Go build client-darwin-amd64
Done: Go build client-linux-mips64le
Starting: Go build client-darwin-arm64
Starting: Go build client-windows-amd64.exe
Done: Go build client-linux-mips64
Starting: Go build client-windows-386.exe
Done: Go build client-darwin-amd64
Starting: Go build client-windows-arm64.exe
Done: Go build client-windows-amd64.exe
Starting: Go build server-linux-amd64
Done: Go build client-windows-386.exe
Starting: Go build server-linux-386
Done: Go build client-darwin-arm64
Starting: Lipo
Done: Lipo
Starting: Go build server-linux-arm64
Done: Go build client-windows-arm64.exe
Starting: Go build server-linux-arm32v5
Done: Go build server-linux-amd64
Starting: Go build server-linux-arm32v6
Done: Go build server-linux-386
Starting: Go build server-linux-arm32v7
Done: Go build server-linux-arm64
Starting: Go build server-linux-mips
Done: Go build server-linux-arm32v5
Starting: Go build server-linux-mips-softfloat
Done: Go build server-linux-arm32v7
Starting: Go build server-linux-mipsle
Done: Go build server-linux-arm32v6
Starting: Go build server-linux-mipsle-softfloat
Done: Go build server-linux-mips
Starting: Go build server-linux-mips64
Done: Go build server-linux-mips-softfloat
Starting: Go build server-linux-mips64le
Done: Go build server-linux-mipsle
Starting: Go build server-darwin-amd64
Done: Go build server-linux-mipsle-softfloat
Starting: Go build server-darwin-arm64
Done: Go build server-linux-mips64
Starting: Go build server-windows-amd64.exe
Done: Go build server-linux-mips64le
Starting: Go build server-windows-386.exe
Done: Go build server-windows-amd64.exe
Starting: Go build server-windows-arm64.exe
Done: Go build server-darwin-amd64
Starting: Go build dual-linux-amd64
Done: Go build server-darwin-arm64
Starting: Lipo
Done: Lipo
Starting: Go build dual-linux-386
Done: Go build server-windows-386.exe
Starting: Go build dual-linux-arm64
Done: Go build server-windows-arm64.exe
Starting: Go build dual-linux-arm32v5
Done: Go build dual-linux-386
Starting: Go build dual-linux-arm32v6
Done: Go build dual-linux-arm64
Starting: Go build dual-linux-arm32v7
Done: Go build dual-linux-amd64
Starting: Go build dual-linux-mips
Done: Go build dual-linux-arm32v5
Starting: Go build dual-linux-mips-softfloat
Done: Go build dual-linux-arm32v6
Starting: Go build dual-linux-mipsle
Done: Go build dual-linux-arm32v7
Done: Go build dual-linux-mips
Starting: Go build dual-linux-mipsle-softfloat
Done: Go build dual-linux-mips-softfloat
Starting: Go build dual-linux-mips64
Starting: Go build dual-linux-mips64le
Done: Go build dual-linux-mipsle
Starting: Go build dual-darwin-amd64
Done: Go build dual-linux-mipsle-softfloat
Starting: Go build dual-darwin-arm64
Done: Go build dual-linux-mips64
Starting: Go build dual-windows-amd64.exe
Done: Go build dual-linux-mips64le
Starting: Go build dual-windows-386.exe
Done: Go build dual-darwin-amd64
Starting: Go build dual-windows-arm64.exe
Done: Go build dual-windows-amd64.exe
Done: Go build dual-windows-386.exe
Done: Go build dual-darwin-arm64
Starting: Lipo
Done: Lipo
Traceback (most recent call last):
File "/home/python/Venv/WSVPN/wsvpn/./build.py", line 337, in
main()
File "/home/python/Venv/WSVPN/wsvpn/./build.py", line 331, in main
task.join()
File "/home/python/Venv/WSVPN/wsvpn/./build.py", line 137, in join
raise self.exc
File "/home/python/Venv/WSVPN/wsvpn/./build.py", line 125, in run
self._run()
File "/home/python/Venv/WSVPN/wsvpn/./build.py", line 227, in _run
check_call(args)
File "/usr/lib/python3.10/subprocess.py", line 364, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib/python3.10/subprocess.py", line 345, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.10/subprocess.py", line 969, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.10/subprocess.py", line 1720, in _execute_child
and os.path.dirname(executable)
File "/usr/lib/python3.10/posixpath.py", line 152, in dirname
p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Done: Go build dual-windows-arm64.exe

Hi Mark,

Could you please also build wsvpn binary for armv7l. Lots of modern OpenWrt routers are armv7l, and RPI2/3/4 (with 32-bit Linux) is also armv7l. Thanks a ton!

Please provide the script execution setting for WSVPN server. Whenever any of the client tunnels is up or down, the server would automatically invoke the script, similar to the same capability on WSVPN client.

I wonder what is the indention behind creating the simple docker image as they only contain the go binaries. Should they be used by setting the scripts.up/down in the config and mounting the scripts themselves into the containers?

Based on the Dockerfile of the testcontainer and the output of ldd $(which ip) I was able to use the sever container with the following:

docker run --rm --cap-add=NET_ADMIN \
       -v `pwd`/config:/config \
       --device /dev/net/tun:/dev/net/tun \
       -v "/dev/net/tun:/dev/net/run" \
       -v /bin:/bin:ro \
       -v /lib64:/lib64:ro \
       -v /usr/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:ro \
       -v /lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu:ro \
       -p 9000:9000/udp \
       ghcr.io/doridian/wsvpn/server:4.7.0

But it kind of defeats the purpose of containerization.

Thanks.

Hi Mark,

If the htpasswd file contains clear-text password (e.g. wsvpn:user), basic authentication using htpasswd works and wsvpn tunnel is established.

If the htpasswd file contains secure password hash (created using htpasswd command), basic authentication using htpasswd always fails with the error "panic: websocket: bad handshake" shown on the client side.

the same htpasswd file is saved on server and client

cat htpasswd
wsvpn:$apr1$XU7CFUAq$qHS6n7DDMZxrx4I77FfKD0

SERVER
sudo ./server-linux-arm -authenticator htpasswd -listen 192.168.1.1:9000
2022/06/10 05:25:55 [S] VPN server online at 192.168.1.1:9000, mode TUN, serving subnet 192.168.3.0/24 (253 max clients) with MTU 1280

CLIENT
sudo ./client-linux-amd64 -auth-file htpasswd -proxy http://http:[email protected]:8080 -connect ws://192.168.1.1:9000
2022/06/09 22:27:07 Connecting to ws://192.168.1.1:9000 as user wsvpn
2022/06/09 22:27:07 Using HTTP proxy http://http:[email protected]:8080
panic: websocket: bad handshake

goroutine 1 [running]:
main.main()
/home/runner/work/wsvpn/wsvpn/client/main.go:110 +0x9de

For example:

2022/06/12 07:18:23 Connecting to wss://somewhere:8444 as user test
2022/06/12 07:18:23 TLS cipher for data channel: 'xxxxxxxxxxxxxxxxx' <===
2022/06/12 07:18:23 Network mode TUN, subnet 192.168.3.3/24, mtu 1460
2022/06/12 07:18:23 Opened tun0
2022/06/12 07:18:23 Configured interface, starting operations

Hi Mark,

Can you please enhance the proxy support for wsvpn client to include proxy authentication if needed. Thanks a ton. You rock!

For example:
-proxy http://10.10.10.10:8080 -proxy-auth user:password

CURRENT WSVPN CLIENT 1.1.0

Usage of ./client-linux-amd64:
-auth-file string
File to read authentication from in the format user:password
-connect string
Server address to connect to (default "ws://example.com")
-default-gateway
Route all traffic through VPN
-down-script string
Script to run when the VPN goes offline
-proxy string
HTTP proxy to use for connection (ex. http://10.10.10.10:8080)
-up-script string
Script to run once the VPN is online

I have several different tun interfaces on the server, e.g. tun0 for openvpn. For wsvpn, it does not allow me to specify a fix tun interface, and may create and bind to either tun1, or tun2, or tun3 etc. if the server is rebooted and the startup script starts wsvpn server. Since I use zone-based firewall, this change of tun interface affects the firewall zoning.

Mainly, the option to easily just build binaries for the current local machine, or any architecture.
Seems pointless/overkill for local dev otherwise.

I enable WebTransport support (enable-http3) on the wsvpn server. When the client connects to the wsvpn server, it does not show WebTransport connection. Can you please add an additional log message indicating whether the finally negotiated transport is Websocket or WebTransport. Thanks a lot.

2022/07/08 07:29:34 Local version is: 3.4.0 (protocol 7)
2022/07/08 07:29:34 base.go:76: [CLIENT-UNSET] Connecting to wss://x.x.x.x:xxxx with authentications: mTLS, HTTP
2022/07/08 07:29:36 connection.go:55: [CLIENT-UNSET] Command serialization: json
2022/07/08 07:29:36 connection.go:59: [CLIENT-UNSET] TLS 1.3 WebSocket connection established with cipher=TLS_CHACHA20_POLY1305_SHA256
2022/07/08 07:29:36 ping.go:48: [CLIENT-UNSET] Ping enabled with interval 1h0m0s and timeout 5s
2022/07/08 07:29:36 commands.go:60: [CLIENT-UNSET] Remote version is: 3.4.0 (protocol 7)
2022/07/08 07:29:36 commands.go:73: [CLIENT-UNSET] Got reply to command ID 9e4e7c09-21b8-4504-88ad-f35bcaf38030 (ok): OK
2022/07/08 07:29:36 commands.go:58: [CLIENT-1aa60733-dc8a-4113-9281-80e4c03414e6] Network mode TUN, Subnet 10.10.11.3/24, MTU 1472, IPConfig enabled
2022/07/08 07:29:36 commands.go:74: [CLIENT-1aa60733-dc8a-4113-9281-80e4c03414e6] Opened wsclient
2022/07/08 07:29:36 commands.go:87: [CLIENT-1aa60733-dc8a-4113-9281-80e4c03414e6] Configured interface, starting operations

This is just a cosmetic enhancement request to make WSVPN even more perfect :-). I've been so happy discovering your awesome program when wandering around on github.

POSIX guideline: https://www.gnu.org/prep/standards/html_node/Command_002dLine-Interfaces.html

Use the prefix "--" instead of "-" for all command line options that have more than 1 character (e.g. -P but --proxy).

Usage of ./wsvpn_server_arm:
--authenticator string
Which authenticator to use (allow-all, htpasswd) (default "allow-all")
--client-to-client
Allow client-to-client communication (in TAP)
--listen string
Listen address for the WebSocket interface (default "127.0.0.1:9000")

etc.