Comments (33)
Work for this underway. In addition to supporting WireGuard, we'll be supporting IPSec-based encrypted networks. We'll be packaging and deploying charon so it works out of the box.
from k3s.
@bduisenov the wireguard backend on flannel works flawlessly. I've been running it for a couple of months and it's awesome.
To make it work you only have to install wireguard and it will install the required kernel's modules. Once this is done, just start k3s with flannel and the backend as wireguard.
https://rancher.com/docs/k3s/latest/en/installation/network-options/#flannel-options
from k3s.
Try this:
https://github.com/squat/kilo
from k3s.
Encryption support has been added to v0.10.0-alpha1.
There is an experimental server --flannel-backend
flag to control what type of encryption is used. It currently defaults to vxlan
(no encryption), but we are planning to default it to ipsec
in a future release.
There are two reasons we are planning to default to ipsec
instead of wireguard
:
ipsec
only needs thecharon
user space process (provided with k3s), wherewireguard
needs kernel modules installed or the use of user-space wireguard implementations which may be buggy or have poor performanceipsec
is faster thanwireguard
kernel modules from initial testing. This contradicts claims of the wireguard community so probably needs extra investigation, my initial results are here: https://docs.google.com/spreadsheets/d/1CuaLtmGmSEzW1OUaedZJfiPnoPFxCoXFQVTT2X5cAPs/edit?usp=sharing. The performance results of wireguard & UDP seem unrealistic compared to the baseline results and probably needs extra attention.
from k3s.
v0.10.0-rc2
WireGuard and IPSEC support has been validated. With the upcoming v0.10.0 release encryption will be off by default as Erik has mentioned here: #50 (comment)
To enable encryption with WireGuard or IPSEC pass the --flannel-backend
flag on the server such as --flannel-backend=ipsec
or --flannel-backend=wireguard
for example. Properly installing WireGuard on each node is recommended so you have the necessary kernel modules if you wish to use wg. IPSEC does not require any kernel modules.
We tested primarily on Ubuntu 18.04. For WireGuard we installed via the recommended method https://www.wireguard.com/install/ using the wg PPA. In the future we plan to improve our test coverage by testing in other flavors of linux and other environments.
from k3s.
Wireguard is available, there is a type typo there. Instead of flannel
it should say wireguard
. :)
from k3s.
Does the wireguard option just use the preconfigured wg0 adapter? Since you can have multiple wg adapters (wg0, wg1 etc).
So I'm assuming, that if my nodes can see each other through wg0 interface and communicate through it, --flannel-backend=wireguard should just work out of the box?
Edit: Also, lets say that I had wg1 and I wanted to use that, would that be in any way possible?
from k3s.
Maybe this network plugin is easily usable? https://gravitational.com/blog/announcing_wormhole/
from k3s.
The result with my test to v0.10.0
with docker-compose:
I'm testing in debian9 with wireguard installed. that run with kilo over wireguard is ok.
1.1: --flannel-backend=ipsec
, master and node in same Lan boots up ok.
server_1 | 15[IKE] remote host is behind NAT
server_1 | 15[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
server_1 | 15[NET] sending packet: from 172.22.0.2[500] to 172.22.0.4[500] (728 bytes)
server_1 | 08[NET] received packet: from 172.22.0.4[4500] to 172.22.0.2[4500] (256 bytes)
server_1 | 08[ENC] parsed IKE_AUTH request 1 [ IDi AUTH SA TSi TSr N(MOBIKE_SUP) N(NO_ADD_ADDR) N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
server_1 | 08[CFG] looking for peer configs matching 172.22.0.2[%any]...172.22.0.4[172.22.0.4]
server_1 | 08[CFG] selected peer config '172.22.0.2-7.0.1.0/24-7.0.0.0/24-172.22.0.4'
server_1 | 08[IKE] authentication of '172.22.0.4' with pre-shared key successful
server_1 | 08[IKE] peer supports MOBIKE
server_1 | 08[CFG] no IDr configured, fall back on IP address
server_1 | 08[IKE] authentication of '172.22.0.2' (myself) with pre-shared key
server_1 | 08[IKE] IKE_SA 172.22.0.2-7.0.1.0/24-7.0.0.0/24-172.22.0.4[2] established between 172.22.0.2[172.22.0.2]...172.22.0.4[172.22.0.4]
server_1 | 08[CFG] selected proposal: ESP:AES_GCM_16_128/NO_EXT_SEQ
server_1 | 08[IKE] CHILD_SA 7.0.1.0/24-7.0.0.0/24{1} established with SPIs cf633ec7_i c999f08c_o and TS 7.0.1.0/24 === 7.0.0.0/24
node_1 | 06[IKE] authentication of '172.22.0.4' (myself) with pre-shared key
node_1 | 06[IKE] establishing CHILD_SA 7.0.0.0/24-7.0.1.0/24{1}
node_1 | 06[ENC] generating IKE_AUTH request 1 [ IDi AUTH SA TSi TSr N(MOBIKE_SUP) N(NO_ADD_ADDR) N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
node_1 | 06[NET] sending packet: from 172.22.0.4[4500] to 172.22.0.2[4500] (256 bytes)
node_1 | 07[NET] received packet: from 172.22.0.2[4500] to 172.22.0.4[4500] (224 bytes)
node_1 | 07[ENC] parsed IKE_AUTH response 1 [ IDr AUTH SA TSi TSr N(MOBIKE_SUP) N(NO_ADD_ADDR) ]
node_1 | 07[IKE] authentication of '172.22.0.2' with pre-shared key successful
node_1 | 07[IKE] IKE_SA 172.22.0.4-7.0.0.0/24-7.0.1.0/24-172.22.0.2[2] established between 172.22.0.4[172.22.0.4]...172.22.0.2[172.22.0.2]
node_1 | 07[CFG] selected proposal: ESP:AES_GCM_16_128/NO_EXT_SEQ
node_1 | 07[IKE] CHILD_SA 7.0.0.0/24-7.0.1.0/24{1} established with SPIs c999f08c_i cf633ec7_o and TS 7.0.0.0/24 === 7.0.1.0/24
[root@(⎈ |default:default) ~]$ kc get no -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node1 Ready <none> 18h v1.16.2-k3s.1 172.22.0.4 <none> Unknown 4.9.0-4-amd64 containerd://1.3.0-k3s.2
server Ready master 18h v1.16.2-k3s.1 172.22.0.2 <none> Unknown 4.9.0-4-amd64 containerd://1.3.0-k3s.2
1.2: --flannel-backend=ipsec
,When node in diff Lan, The LB-IP of master's can not be reached.
root @ debian in /opt/dcp-k3s/v0-10/t_node |09:57:20
$ dcp -f node2.yml up
Recreating t_node_node_1 ... done
Attaching to t_node_node_1
node_1 | time="2019-10-27T01:57:24.915012409Z" level=info msg="Starting k3s agent v0.10.0 (f9888ca3)"
node_1 | time="2019-10-27T01:57:24.916715615Z" level=info msg="Running load balancer 127.0.0.1:33369 -> [6.6.6.201:6443]"
node_1 | time="2019-10-27T01:57:25.264112941Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
node_1 | time="2019-10-27T01:57:25.264604596Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
node_1 | time="2019-10-27T01:57:25.265333548Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""
node_1 | time="2019-10-27T01:57:26.270894058Z" level=info msg="module br_netfilter was already loaded"
node_1 | time="2019-10-27T01:57:26.271032326Z" level=warning msg="failed to write value 1 at /proc/sys/net/bridge/bridge-nf-call-iptables: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory"
node_1 | time="2019-10-27T01:57:26.296940758Z" level=info msg="Updating load balancer server addresses -> [172.22.0.2:6443 6.6.6.201:6443]"
node_1 | time="2019-10-27T01:57:26.298428602Z" level=info msg="Connecting to proxy" url="wss://172.22.0.2:6443/v1-k3s/connect"
node_1 | time="2019-10-27T01:59:35.814752257Z" level=error msg="Failed to connect to proxy" error="dial tcp 172.22.0.2:6443: connect: connection timed out"
Can I infer that IPSEC just resolv the encrytion in Lan/Nat (Which master can be reached by node's network), but not the connection for node to master when the node startup(When master in Nat env). And I didn't see new tunnel-device in master/node: no flannel.x and no strongswan's tunnel, if something wrong with my env?
root @ debian in /opt/dcp-k3s/v0-10 |10:15:23
$ dcp exec server sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1415 qdisc noqueue state UP group default qlen 1000
link/ether 7a:05:6a:c5:6b:eb brd ff:ff:ff:ff:ff:ff
inet 7.0.1.1/24 scope global cni0
valid_lft forever preferred_lft forever
3: veth5067deaf: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1415 qdisc noqueue master cni0 state UP group default
link/ether 26:76:97:66:57:b6 brd ff:ff:ff:ff:ff:ff link-netns cni-31e6c753-d276-13dd-6920-91d22113e7c1
4: veth951b7871@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1415 qdisc noqueue master cni0 state UP group default
link/ether 82:78:eb:c7:f1:a7 brd ff:ff:ff:ff:ff:ff link-netns cni-3fa4ff39-8c64-2ce4-506a-257da77f922b
70: eth0@if71: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:16:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.22.0.2/16 brd 172.22.255.255 scope global eth0
valid_lft forever preferred_lft forever
2. --flannel-backend=wireguard
, need wg to be executed, but it not in the rancher/k3s:v0.10.0-amd64
container's image. that say we need to run k3s binary in the host with wireguard's installation. we can't use it in DinD mode in the current image.
server_1 | time="2019-10-24T10:50:54.439686673Z" level=fatal msg="flannel exited: failed to run command: wg genkey | tee privatekey | wg pubkey Err: exit status 127 Output: sh: wg: not found\nsh: wg: not found"
node_1 | time="2019-10-24T10:50:54.619537523Z" level=fatal msg="flannel exited: failed to run command: wg genkey | tee privatekey | wg pubkey Err: exit status 127 Output: sh: wg: not found\nsh: wg: not found"
Suggestion for wireguard usage in DinD:
-
Make
wg
into therancher/k3s
image. -
Can we just compile wg into k3s binary? when I use kilo it runs just without the
wg
binary.
from k3s.
I'll correct myself, the ipsec backend does not support NAT. I thought it work in my case, but id didn't. Wireguard, though, it does. Without anything special, only having the kernel modules.
It may require restarting the boxes, in my case there were a mix of problems with the routes in both workers and master.
from k3s.
wireguard
support which ships with k3s would be super super helpful. I love the simplicity of k3s
and how easy it is to setup a cluster. Having unencrypted traffic between the components of the cluster is actually a no-go for production-kind of setups though. Especially on cloud providers which don't support private networks (even then an encrypted communication would be required).
Do you have something like this on your roadmap? 🙂
from k3s.
Try this:
https://github.com/squat/kilo
I'm using this, ref:
from k3s.
That's great feature! We can see v0.10.0-alpha1 with size 54.6 MB
, which more than 10Mb than v0.9.1(43.5M
), I'll try this latter.
Can ipsec
be work in nat? (I'm not similar with it)
-
Wireguard sure needs the kernel support, I've used wg in debian9, centos7, barge-os that works.
-
Sure wireguard use udp for nat hole for connection which may cause unstable package trans.
1.In my env when my nodes both with a public ip (cloud vm with wan eip, but lan eth0) that works fine.
2.But when my node just in nat without wan-ip routes the wireguard's udp port and set the wan ip in kilo, then packages in connection lost or long-time return in period.
from k3s.
Yah the size of k3s is going up slowly but surely. charon
adds about 2.7M uncompressed and swanctl
(which we can probably drop) about 864K. Upgrading buildroot to 2019.2 LTS release and k8s update to 1.16 also helped to increase the binary size.
I am guessing ipsec
should work as well as vxlan
over NAT, but any testing that could be provided by the community is greatly appreciated.
If wireguard
is broken for this use case I feel like we should just remove it as an option. I am also somewhat concerned that wireguard does not require a PSK for setting up the network, that indicates to me it is configuring the network in an insecure way. edit: looks like flannel is storing and sharing certs through the kubernetes node resource, so psk not needed
from k3s.
Hi! Is ipsec still the better choice for performance? Thanks
from k3s.
So I installed a cluster with ipsec and specifying the private network interface for Flannel as well. How can I check that the traffic is going through the private network and is encrypted? Thanks a lot in advance 🙂
from k3s.
Is WireGuard support still not released?
#/usr/local/bin/k3s --version
k3s version v1.0.0 (18bd921c)
# /usr/local/bin/k3s server --help | grep flannel
--flannel-backend value (networking) One of 'none', 'vxlan', 'ipsec', or 'flannel' (default: "vxlan")
--flannel-iface value (agent/networking) Override default flannel interface
--flannel-conf value (agent/networking) Override default flannel config file
--no-flannel (deprecated) use --flannel-backend=none
from k3s.
Has anyone been able to get either IPSec or Wireguard to work through NAT traversal, i.e. a private arm SoC behind an ISP router “at home” and a “server” with a public IP on a cloud host.
from k3s.
Has anyone been able to get either IPSec or Wireguard to work through NAT traversal, i.e. a private arm SoC behind an ISP router “at home” and a “server” with a public IP on a cloud host.
I have the exact same setup, did you find a working solution ?
from k3s.
I do have it with that setup, it works out of the box if the master is in a remote server.
Also, It could work WHEN the master is at home if you do have a static IP, thought.
from k3s.
I do have it with that setup, it works out of the box if the master is in a remote server.
Also, It could work WHEN the master is at home if you do have a static IP, thought.
Hello,
So just using k3s with the wireguard flag make it work ? All node are able to talk to each other ? Even those behind NAT at home ?
from k3s.
@adi90x it should work. Starting with v1.17.0 there is a persistent-keepalive every 25s to keep the tunnel open https://github.com/rancher/k3s/pull/1223/files
from k3s.
Yes @adi90x. I only tried with ipsec, though.
from k3s.
from k3s.
WireGuard should work fine as long as it's set up on each node before K3s install. For example you probably need the kernel modules. Reference https://www.wireguard.com/install/
from k3s.
just specifying --flannel-iface=wg0 works w/o any backend specification... so what does that --flannel-backend=wireguard then? Very confusing documentation
from k3s.
You can usually establish the network and schedule pods, but I found that inter-pod communication did not work, especially behind NAT.
I.e. an API that talks to a DB with both running on separate nodes.
from k3s.
@alexellis I'm experiencing the same issue, were you able to find a solution to that?
from k3s.
@bduisenov you may try kilo as an alternative: https://github.com/squat/kilo
from k3s.
I think cillium also has an encrypted networking plugin. Not sure.
If anyone wants to write a guide for k3s, it'll be appreciated.
from k3s.
What about when the public machine peer is reported as the internal ip, see #2689 - where can I override the wireguard peer ip?
from k3s.
@matti use this annotation on the node: flannel.alpha.coreos.com/public-ip-overwrite=yourip
from k3s.
@unixfox thanks, just found this also: https://github.com/alekc-go/flannel-fixer
from k3s.
Related Issues (20)
- Use private ip with external cloud manager that doesn't know about that ip
- Allow setting `priorityClassName` on ServiceLB daemonset. HOT 5
- k3s cluster should use GPU without mentioning runtimeClassName: "nvidia" in pod specs HOT 3
- Fail to run static pod without master running HOT 2
- --node-external-ip can not add annotate flannel.alpha.coreos.com/public-ip-overwrite HOT 1
- Pods are going into pending state after upgrading from v1.26.12-k3s1 to v1.27.11-k3s1 and v1.28.5-k3s1 (Issue is quite random)
- leader-elected etcd controllers not consistently functional when leader election/lease mismatches occur HOT 4
- Update minio-go to `v7.0.70` or newer HOT 2
- Embedded helm controller does not track owner references properly
- Network policy in dual-stack cluster stops allowing ingress after pod restart HOT 14
- ingresses stop working every now and then HOT 8
- Create a new single node cluster with embedded registry mirror does not work HOT 5
- K3S startup stuck in a deadlock when a KMS provider is configured and the node is rebooted HOT 4
- Expose k3s metrics over http HOT 2
- Capturing K3s metrics over http/https from VM HOT 1
- Agent node IP not replaced with Tailscale VPN IP HOT 7
- Pods not always creating logs HOT 1
- Best practice of setting the cluster name at deployment level HOT 4
- k3s crashes if I do not start the cloud-provider
- sh: 0: cannot open –: No such file
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k3s.