I'd like to run this in my homelab, but I'm not sure how often my IP address changes. So I'd like to be able to use a domain name for my external endpoint.

If you have made a mistake in adding a peer and the invitation has not yet been used you may want to delete the peer in order to recreate it instead of modifying the sqlite database.

Hi,

While reading the "Windows support" issue, I saw a comment implying that wireguard-go must be used for a userspace implementation. I recalled that Cloudflare has their own userspace implementation, in Rust no less.

Is there interest in adopting boringtun? Either in pursuit of Windows support, or for its own merits?

Cheers, and thanks for building/releasing/maintaining innernet!

Linux users should be able to create an innernet user that has permissions to /etc/innernet and /var/lib/innernet, and the binaries should be runnable after giving it CAP_NET_ADMIN permission.

As far as I know, no such equivalent exists on macOS.

Any chance for this? I see shared/src/wg.rs only targets macos & linux... thanks!!

I really like the clever solution of the ephemeral keypair to do bootstrapping, it works well where a node has an intermediate state. What I'm struggling with though is when that peer is ephemeral. In that case, the peer will permanently occupy an IP in a CIDR and exhaust the CIDR eventually. There should probably be some kind of garbage collection mechanism that an administrator can run to garbage collect peers that haven't been seen in some configurable amount of time, or even just passing explicit peer names. Without this the entire innernet model kind of falls apart in an environment where machines are dynamically popping in and out of existence.

Hello! It would be great to have ipv6 support for the innernet-server.

I'd like to use this when traveling away from home, to have traffic routed through the network created via innernet, if that is an appropriate use case.

If we have established a home router as a peer 10.42.0.1, and laptop as 10.42.10.123, what would be needed for macOS / Linux to route all traffic through that interface?

Would there have to be some sort of interface bridging involved, if the router defaults to being a gateway at 192.168.0.1?

Can it be done if the IP doesn't change?

Hi, we just found innernet and so far like it a lot. We're wondering whether you already have any plans or setups to run this in Kubernetes environments? We are thinking about using innernet to hide our development and production clusters within a private company network.
We would go ahead and produce a dockerfile and a helm chart to try it, but just wanted to ask first, whether it has been done already somewhere.

You can reach me on Discord as Eeems#2244 for the webook if you'd like to set it up. If you'd rather not touch Discord yourself, just let me know what medium to get into contact with you on.

I added a matrix chat room for innernet #innernet:matrix.org

I tried several times to run this unsuccessfully until I tried having the server on a separate machine worked as intended.

According to the README, I'm registering a server in a machine with some peers. Peers that I install on other machines work as intended, but I can't install a peer on the server machine itself, so that it is registered on mesh with a name, as an admin.

Maybe I'm misunderstanding something.

This is similar to #43, but different somehow.

Here's the server log:

innernet-server[506]:  INFO  warp                       > IP:PORT "POST /v1/user/redeem HTTP/1.1" 410 "-" "ureq/2.1.0" 352.841µs
innernet-server[506]: rejection: Rejection(NotFound)

I am unable to get the innernet-server systemd service to successfully start. I have followed the install guide on Ubuntu 18.04.5 LTS, 20.04, Debian 10, and Kali Linux, and each time I get the same log messages:

Mar 30 18:20:30 ubuntu-test systemd[1]: Started innernet server for sprawl.
Mar 30 18:20:30 ubuntu-test innernet-server[2493]:  INFO  innernet_server > bringing up interface.
Mar 30 18:20:30 ubuntu-test innernet-server[2493]: Error: Os { code: 2, kind: NotFound, message: "No such file or directory" }
Mar 30 18:20:30 ubuntu-test systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Mar 30 18:20:30 ubuntu-test systemd[1]: [email protected]: Failed with result 'exit-code'.
Mar 30 18:20:30 ubuntu-test systemd[1]: [email protected]: Scheduled restart job, restart counter is at 4.

It tries to restart a few times then fails. I am using the latest 1.0.0 release .deb.

The CIDR selection menu when using inn add-{cidr,peer} should also ideally be in tree form, similar to inn list --tree.

Any plan for an Android client?

If so, what would be the alternative to iptables on non-rooted android clients?

Thanks, nice project!

Thank you for this wonderful project, this sounds exactly like what I wanted (something like Tailscale, but without a proprietary server part, in Rust, using the in-kernel WireGuard module! 🎉)

There's one thing stopping me from adopting Innernet immediately: from the glance at the client source code, it appears that Innernet edits /etc/hosts in order to enable .wg names to work.

In my book, that's very wrong. /etc/hosts is the configuration written and maintained by me, the system administrator, not by any software.

Off the top of my head, the two better ways to add your hostnames to the system are:

• make a glibc NSS module (like nss-mymachines)
• run a special DNS server on the WireGuard interface; tell the rest of the system (i.e. with resolvconf or systemd-resolved API) to use it for the wg domain.

If you add a CIDR and realize that you made a mistake it would be good to allow removing the entry without having to edit the sqlite database.

warning: could not execute `llvm-config` one or more times, if the LLVM_CONFIG_PATH environment variable is set to a full path to valid `llvm-config` executable
 it will be used to try to find an instance of `libclang` on your system: "couldn't execute `llvm-config --prefix` (error: No such file or directory (os error 2
))"

error: failed to run custom build command for `clang-sys v1.1.1`

Adding these to the PKGBUILD makes it work

makedepends=('git' 'cargo' 'clang' 'llvm')

I saw your blog post broadcasted on the Nebula Slack, and I must say, I am extremely interested in this project. Also, I'm a Rust fanboy, so this project gets bonus points there as well 😛 .

Now for my question: Is Windows support planned? I mostly use Linux/macOS, but there are instances where I want to connect with a Windows machine as well.

In the innernet blog-post there are some claims about authenticated and trust in IP-addresses.

However, innernet does not actually live up to these claims.

Test methodology:

With three devices:

• one innernet server (that lives on the internet) (10.42.0.0/16)
• one innernet client ( NAT-ed, on fex. a WiFi) (10.42.0.1/24)
• One rogue actor ( NAT-ed, on the same net as the client)

Assuming that the client LAN is 192.168.0.1/24.
Assuming that the innernet server is set up with the default cidr of 10.42.0.0/16, we hook up the client and enable the server.

Next, we put a super-seekrit service on the client IP of 10.42.0.1 ( telnet, with admin/admin as password!)

On the "rogue" actor, we then do the following (assuming that eth0 is the ip address):

ip address add 10.42.0.8 dev eth0
ip route add 10.42.0.0/16 dev eth0

Now the "rogue" client can freely access services that are supposed to only be accessible to internal clients on the innernet ip.

Problem

Linux by default responds to traffic directed to it's own IP addresses on any interface. This can be adjusted with a sysctl, but that sysctl should not be configured by software as a security mechanism, as it may break a fair few things. The proper(Er) solution is to add an interface-level filter for traffic.

This means that innernet client should also make sure to drop traffic directed to it's internal IP that arrives on any interface other than the wg interface.

(This continues the discussion from #26)

Temporary solution

We can have some basic CSRF protection by requiring custom headers with some unique information, as experimented with in #37.

This type of solution is far from ideal, and is instead just a means to stop the simplest of this class of attacks.

Idea A: Signature-based authentication

Curve25519 keys, those used by WireGuard, can be converted to be used in Ed25519 signatures.

Peers could use their private keys to sign API requests (kind of like what AWS does), and through that would prove that the initiator of the request is in possession of the peer's private key.

Advantages

• No extra keys or magic secrets
• Could be used as an authentication method not just for innernet-server, but eventually for other services on an innernet network.

Disadvantages

• Curve25519

Idea B: Token-based authentication

The innernet-server could give peers a token when peers first join (or re-join) the network, and require that they pass that token along with every request.

Advantages

• Simple, tried-and-true method

Disadvantages

• Not completely clear when exactly the token is generated and given to peers, and how/if it's rotated.
• Only really an applicable solution for innernet-server, every other service has to work their own stuff out.

Idea C: ???

I have a couple of arm64 systems (Pi 400, Pi 4, and a server-class Ampere Altra) that I'd be interested in adding to an innernet network. If the project built binaries I'd test them, ditto with make targets.

Trying to cross-compile innernet on my M1 Macbook (because my Arm machines don't have enough ram to even install rust via rustup) and I'm blocked by a missing bindings.rs

$ cargo build --release --bin innernet --target armv7-unknown-linux-gnueabihf
[...]
error: couldn't read /Users/rjp/git/_others/innernet/target/armv7-unknown-linux-gnueabihf/release/build/wgctrl-sys-9a837bdbbe74630f/out/bindings.rs: No such file or directory (os error 2)
 --> wgctrl-sys/src/lib.rs:6:1
  |
6 | include!(concat!(env!("OUT_DIR"), "/bindings.rs"));
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

error: could not compile `wgctrl-sys`

Works fine when native* compiling (x86_64-apple-darwin) or cross-compiling to M1 (aarch64-apple-darwin).

Hi,

I like the project ideas and goal but I'm wondering how things could be refactored to allow for a serverless setup.

My understanding is that the API/server logic is being wireguard when it could be hosted on a serverless platform (I'm thinking of Cloudflare Workers).
The initial invite logic could be substituted by oneshot API tokens and it would be relatively easy to leverage SSO/IdP and other features without having to do a lot of work.

Is this something that was explored during the design of innernet?

Cheers

I'm trying to compile innernet on RPi4 (Ubuntu, 20.04, 32bit ARMv7), but fails. I've been trying with these two toolchains:

armv7-unknown-linux-gnueabihf
armv7-unknown-linux-musleabihf

(I really need the latter one, because I need this to be statically linked as I'm going to use it on another ARMv7 install in the end).

armv7-unknown-linux-gnueabihf fails on:

   Compiling wgctrl v1.1.0 (/home/ubuntu/src/innernet/wgctrl-rs)
error[E0308]: mismatched types
   --> wgctrl-rs/src/device.rs:142:20
    |
142 |             *out = *b as i8;
    |                    ^^^^^^^^ expected `u8`, found `i8`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0308`.
error: could not compile `wgctrl`

armv7-unknown-linux-musleabihf fails on:

   Compiling wgctrl-sys v1.1.0 (/home/ubuntu/src/innernet/wgctrl-sys)
error: failed to run custom build command for `wgctrl-sys v1.1.0 (/home/ubuntu/src/innernet/wgctrl-sys)`

Caused by:
  process didn't exit successfully: `/home/ubuntu/src/innernet/target/release/build/wgctrl-sys-050e5c811584b48d/build-script-build` (exit co
de: 101)
  --- stderr
  /usr/include/sys/types.h:144:10: fatal error: 'stddef.h' file not found
  /usr/include/sys/types.h:144:10: fatal error: 'stddef.h' file not found, err: true
  thread 'main' panicked at 'Unable to generate bindings: ()', wgctrl-sys/build.rs:15:44
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Does innernet create any additional files besides the systemd ones? Want to cleanup some test networks I created and I didn't see a command for it

For those of us who prefer to cargo install and build from source rather than using a binary.

Hi, this looks like useful software, thank you for releasing it open source!

In the readme you say:

Enabling/Disabling Peers

For security reasons, IP addresses cannot be re-used by new peers, and therefore peers cannot be deleted. However, they can be disabled. Disabled peers will not show up in the list of peers when fetching the config for an interface.

Could you expand on the security properties here? Specifically:

1. When will other peers start to reject connections from a disabled peer?
2. How are other peers told to reject these connections?

Thanks!

I needed to modprobe wireguard (and apt install wireguard) in order for innernet-server serve to work.

Not sure what the most morally correct way is to do this persistently. I wound up doing echo wireguard > /etc/modules-load.d/wireguard.conf.

I'd like to use this to replace some openvpn site-to-site madness but two of the sites only have servers that are kubernetes nodes. I can see some cool ways to run this - like having an innernet server in each k8s cluster, having a sidecar that does some sort of on-start-get-priv-key thing from the local innernet server to join the network. However, it appears the way the client tries to run is not permitted inside a docker image.

From a windows host:
Using ubuntu:20.04, I get this error when trying to add a network:

root@f513c653fc32:~# inn install invite.toml
✔ Interface name · test
[*] bringing up the interface.

[ERROR] Operation not permitted (os error 1)

From a linux host:

root@074863c023a2:~# inn install /mydocker.toml
✔ Interface name · test
[*] bringing up the interface.

[ERROR] No such file or directory (os error 2)

thoughts on if this could be possible?

Related issue: toltec-dev/toltec#350

The current paths of e.g. /etc/innernet are hard coded and can be problematic for some of the more esoteric setups.

I am not sure how to address this, i believe, the behavior is sort of intentional, but it might be also b.a.d...

Problem:
Peers of the same cidr can connect each other without further actions. It makes a lot of sense to allow this configuration on site2site vpns and also on cidrs for servers, as any service will have its own authentication, but not on human users local endpoint systems (classic name: roadwarriors).

I think, this is problematic as most devs see their local machine also as server for testing services locally and these may run without any authentication as they are local to the dev.

These services are exposed to the vpn, if they are not bound to a dedicated interface. Most services are really bound to 0.0.0.0 as default, if they are not databases which usually bind to 127.0.0.1 as default.

Even if the developer does not portforward her innernet port to her cablemodem/router: It is sufficient, if she initiates a connection to another vpn and keep it running to expose this port as it is udp.

Example:
Alice and Bob are humans and in the cidr humans. Machine-A is a service in the cidr svc. Humans cidr is allowed to connect to cidr svc.

Alice connects to machine-a (vpn peer, not in her cidr, but allowed)
Bob does inn list --tree and nmaps Alices ip address:

$ nmap 10.42.4.1
Starting Nmap 7.80 ( https://nmap.org ) at 2021-04-23 08:56 UTC
Nmap scan report for alice.fcloud.wg (10.42.4.1)
Host is up (0.024s latency).
Not shown: 996 closed ports
PORT     STATE SERVICE
80/tcp   open  http
3000/tcp open  ppp
8086/tcp open  d-s-n
8888/tcp open  sun-answerbook

From Alices point of view (the nmap only catches tcp):

[0] % sudo netstat -tpaun |awk '{print $1,$4}' | grep -E '0.0.0.0|:::'
tcp 0.0.0.0:80
tcp6 :::8086
tcp6 :::8888
tcp6 :::3000
tcp6 :::80
udp 0.0.0.0:49029
udp 0.0.0.0:68
udp 0.0.0.0:5353
udp6 :::49029

In a classic (centralized) wireguard vpn network it is easy to prevent this by managing routing on the central server and sending human peer vpn config stencils maintaining the AllowedIPs netmask as /32.

To achive the same with innernet i would need to create a lot of /30 nets with one human user in each. And i would need to enable any single human peer instead of groups of users in the same cidr.

Please keep in mind that it will not help to setup local firewall rules against this on Alices machine as the traffic has already passed the external interface (is authenticated and decrypted) and than checked, if the source ip is allowed.
(https://www.wireguard.com/#cryptokey-routing)

As an admin i feel the need trying to protect my users as most as possible against unforeseen co-drivers on their personal hardware.
Kudos to the attendees in the discord discussion which made it possible to sharpen the request.

I propose to introduce a configuration flag (prevent inter-cidr communication or similar) to prevent that peers of a cidr can connect each other without further configuration.

At least we should add a remark to the security-recommendations chapter, but i feel that most classical users will not read it.

I've noticed when I run inn fetch <interface> that communication between the two associated peers doesn't work until I run inn fetch <interface> on both peers. That's a little cumbersome and would be nice to handle that automatically via the innernet-server.

Currently innernet does static routing using the OS's built in tools in shared/src/wg.rs's add_route function which is a good default functionality. However, if this functionality could be turned of somehow (not necessarily at runtime), that would enable users to choose other routing methods/protocols such as babel.

Making innernet do that configuration would probably be a bit much, so by just switching this behaviour off (e.g. with a feature flag at compile time), people could configure their other routing methods themselves.

If this sounds like a reasonable idea, I can prepare a pull request on this.

Hi,

i am using v1.0.1 on server (linux) and on client side (mac, linux).
I couldnt register to the server in the first try while a made a mistake with the portforwarding on the server side and got an error:

[1] % sudo inn install ./aleks.toml
✔ Interface name testvpn
[*] bringing up the interface.
[*] Generating new keypair.
[*] Registering keypair with server (at 10.42.0.1:51821).

[ERROR] http://10.42.0.1:51821/v1/user/redeem: Connection Failed: Connect error: connection timed out

I checked, if the portforward to the internal server ip really works with netcat on both sides and it worked.
I monitored the initial register try with tcpdump, but saw nothing on server side but bad udp checksum notes on client side.
Digging further around, i found a leftover virtual interface on the client system from a former try which used the same ip range and catched the connetion somehow.

I removed it (sudo ip link delete <vpnname>), and then it worked as expected and i could register my client.
Same same on mac, but it seems more difficult (or not possible?) to remove an utun type interface. i found it sufficient to take it down (sudo ifconfig utunX down).

It would be nice, if an unsuccessful register will remove all stuff which was setup by it.

Hey, innernet is reeeally promissing and looks great so far!
We missing some things (for example user management (decomission users, connection to ldap or oauth provider, ...) but this is v1, and it looks possible to adapt things with the api.

innernet-server runs ok as root, but gives error when run in systemd:

INFO  innernet_server > bringing up interface.
Error: Os { code: 2, kind: NotFound, message: "No such file or directory" }

Which file might that be?

The innernet-server (10.42.0.1/32) is not an eligible CIDR for creating peers in as its a /32.
Maybe remove it from the list ?

Hi!

Thank you so much for creating this tool, i really enjoy it thus far!

I'd like to suggest adding a feature to make this usable in some kind of automation. For that i think it would be really helpfull to be able to use innernet without answering to interactive prompts and use parameters instead.

Installation could then be done by calling something like

innernet install --iface kermpany --delete-invitation ./invitation.toml

Adding a peer:

innernet add-peer kermpany --cidr 10.42.128.0/17 --assign-ip 10.42.128.42 --name ryo --admin

and so on...

I've never written anything in rust up until yesterday but i might give it a shot if this is something you'd like to see in innernet.

As title says.

I would love to create a site to site network if I want to attach an entire site to the created network. The use case would for example be if you have a network where you have devices that cannot run wireguard (like a old NAS) and you still want resources from it available. Simply create a route on a router and/or server to the VPN gateway and you get the site connected.

This would also mean that the peers that are allowed to access the sites network should get that routing information.

Can the client run under macOS? If so, could it be possible to release a version of it, or building instructions? Maybe even a brew cask/formulae?

Update: Rats! Just saw the building instructions for macOS. Ugh, sorry, closing. Thanks!

I just set up a network with an IPv6 CIDR, some subnets inside it, and one peer. When I run sudo inn install ./peer.toml I get

[*] Registering keypair with server (at [fdxx:xxx:xxx::]:51820).
[*] bringing down the interface.
[!] Failed to redeem invite. Now's a good time to make sure the server is started and accessible!

but the .toml file has

external-endpoint = "12.34.56.78:51820"

Shouldn't inn install be using that address instead of the value of internal-endpoint?

While I like the ideas behind warp, it causes compilation/linker slowness because of its extensive use of generics, and it makes some middleware more difficult to handle than it needs to be.

At this point, I'm considering just using hyper directly, since our API is not particularly complicated.

Hi,

i'd like to make the suggestion to use the same udp port for all peers of a vpn as a default option (even it is nice to be able to configure it).

I think it is less effort to configure all firewalls same same, especially if you need to let do it by other people. It would be one documentation boilerplate for all peer admins (which might be also classic end users messing around with isp provided (cable)modem/routers and portforwarding).
It needs a test, if the port is unused, but i think thats easy to achive.

low prio :-D

I wonder what different openwrt archs would make sense to support. I guess armv7l would be quite common

Hi,

it would be nice and better to read if the order of listed cidrs in inn list --tree are in nummerical order, seems like today they are in the order of setup timestamp:

interface: fcloud (YOURMOM...)
  listening_port: 64180
  ip: 10.42.4.7
  10.42.0.0/16 fcloud
    10.42.0.1/32 innernet-server
    | 10.42.0.1 innernet-server
    10.42.4.0/22 humans
      10.42.4.0/25 fcloud-team
      | 10.42.4.1 aleks
      | 10.42.4.7 aleks-mac
      10.42.5.0/24 developers
      10.42.6.0/24 non-devs
    10.42.1.0/24 server
    | 10.42.1.1 teamspeak-srv

This might be a problem on bigger setups. Or sort it alphabetical with the names (has other drawbacks).

Prio: low

I will add more of these, even if they are not important.
I learned if i get used to new software i will accept a lot of little uncomfortable things as the right way to do it, even if they annoy new users. So it is better to note these little annoyances to track them for later fixing.

I've created an association between CIDRs: 192.168.1.0/24 and 192.168.250.1/32 which should allow any IP address in 192.168.1.0/24 to communicate with 192.168.250.1` however...that only works if the incoming IP is an active innernet peer.

I'd like to be able to have a local subnet (192.168.1.0/24) with devices on it and have a single innernet peer in that subnet say 192.168.1.254. Currently that peer can talk to 192.168.250.1 as expected.

If I change the default gateway on the subnet to be 192.168.1.254 and enable ip forwarding on that host then all traffic from other devices make it to the innernet peer but they can't get to 192.168.250.1 unless I add a MASQUERADE and nat the traffic.

That would be fine in some use cases, but I'd like to be able to specify specific device IP's on my local subnet to have associations with specific innernet-peers without them having to have innernet installed on them.

I'm trying to compile the client binary on CentOS 7 with cargo install --path client, but I'm running into below error when compiling wgctrl-sys.

error: linking with `cc` failed: exit code: 1

The full error can be seen here: https://gist.githubusercontent.com/Kerwood/623f446f03447659721cf97861ec56a1/raw/0fd33a7a2d038cb7562e9ec7f7ee3c6b59ac0432/gistfile1.txt

Besides Rust, I've installed clang-devel sqlite-devel and llvm-devel.

Am I missing something ?