nsntrace / nsntrace Goto Github PK

Perform network trace of a single process by using network namespaces.

License: GNU General Public License v2.0

Makefile 1.41% C 89.54% Shell 6.24% M4 2.80%

nsntrace's Introduction

nsntrace

Perform network trace of a single process by using network namespaces.

This application uses Linux network namespaces to perform a network trace of a single application. The trace is saved as a pcap file. And can later be analyzed by for instance Wireshark or tshark.

The nsntrace application is heavily inspired by the askbubuntu reply here. And uses the same approach only confined to a single C program.

What the application does is use the clone syscall to create a new network namespace (CLONE_NEWNET) and from that namespace launch the requested process as well as start a trace using libpcap. This will ensure that all the packets we trace come from the process.

The problem we are left with is that the process is isolated in the namespace and cannot reach any other network. We get around that by creating virtual network interfaces. We keep one of them in the root network namespace and but the other one in the newly created one where our tracing takes place. We set the root namespaced one as the default gateway of the trace namespaced virtual device.

And then to make sure we can reach our intended net, we use iptables and NAT to forward all traffic from the virtual device to our default network interface.

This will allow us to capture the packets from a single process while it is communicating with our default network. A limitation is that our ip address will be the NAT one of the virtual device.

Another limitation is, that since we are using iptables and since we are tracing raw sockets. This application needs to be run as root.

On many systems today the nameserver functionality is handled by an application such as systemd-resolved or dnsmasq and the nameserver address is a loopback address (like 127.0.0.53) where that application listens for incoming DNS queries.

This will not work for us in this network namespace environment, since we have our own namespaced loopback device. To work around this one can use the --use-public-dns option to override resolv.conf in the namespace. Then nsntrace will use nameservers from Quad9 (9.9.9.9), Cloudflare (1.1.1.1), Google (8.8.8.8) and OpenDNS (208.67.222.222) to perform DNS queries.

usage

$ nsntrace
usage: nsntrace [options] program [arguments]
Perform network trace of a single process by using network namespaces.

Options:
-o file          	send trace output to file (default nsntrace.pcap)
-d device        	the network device to trace
-f filter        	an optional capture filter
-u username      	run program as username/uid
--use-public-dns	override resolv.conf to use public nameservers from
                	Quad9, Cloudflare, Google and OpenDNS

example

$ sudo nsntrace -d eth1 wget www.google.com
Starting network trace of 'wget' on interface eth1.
Your IP address in this trace is 172.16.42.255.
Use ctrl-c to end at any time.

--2016-07-15 12:12:17--  http://www.google.com/
Location: http://www.google.se/?gfe_rd=cr&ei=AbeIV5zZHcaq8wfTlrjgCA [following]
--2016-07-15 12:12:17--  http://www.google.se/?gfe_rd=cr&ei=AbeIV5zZHcaq8wfTlrjgCA
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                                         [ <=>                                                                                                   ]  10.72K  --.-KB/s   in 0.001s

2016-07-15 12:12:17 (15.3 MB/s) - ‘index.html’ saved [10980]

Finished capturing 42 packets.

$ tshark -r nsntrace.pcap -Y 'http.response or http.request'
16   0.998839 172.16.42.255 -> 195.249.146.104    HTTP 229 GET http://www.google.com/ HTTP/1.1
20   1.010671    195.249.146.104 -> 172.16.42.255 HTTP 324 HTTP/1.1 302 Moved Temporarily  (text/html)
22   1.010898 172.16.42.255 -> 195.249.146.104    HTTP 263 GET http://www.google.se/?gfe_rd=cr&ei=AbeIV5zZHcaq8wfTlrjgCA HTTP/1.1
31   1.051006    195.249.146.104 -> 172.16.42.255 HTTP 71 HTTP/1.1 200 OK  (text/html)

live capture using tshark

$ sudo nsntrace -f tcp -o - wget www.google.com  2> /dev/null | tshark -r -
1   0.000000 172.16.42.255 → 142.250.74.36 TCP 74 51088 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=1362504482 TSecr=0 WS=128
2   0.014010 142.250.74.36 → 172.16.42.255 TCP 74 80 → 51088 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1430 SACK_PERM=1 TSval=2846449454 Secr=1362504482 WS=256
3   0.014078 172.16.42.255 → 142.250.74.36 TCP 66 51088 → 80 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=1362504496 TSecr=2846449454
4   0.014221 172.16.42.255 → 142.250.74.36 HTTP 207 GET / HTTP/1.1

5   0.033935 142.250.74.36 → 172.16.42.255 TCP 66 80 → 51088 [ACK] Seq=1 Ack=142 Win=66816 Len=0 TSval=2846449475 TSecr=1362504496
6   0.093989 142.250.74.36 → 172.16.42.255 TCP 1484 HTTP/1.1 200 OK  [TCP segment of a reassembled PDU]
7   0.094022 172.16.42.255 → 142.250.74.36 TCP 66 51088 → 80 [ACK] Seq=142 Ack=1419 Win=63360 Len=0 TSval=1362504576 TSecr=2846449532
8   0.096447 142.250.74.36 → 172.16.42.255 TCP 2902 HTTP/1.1 200 OK  [TCP segment of a reassembled PDU]
9   0.096478 172.16.42.255 → 142.250.74.36 TCP 66 51088 → 80 [ACK] Seq=142 Ack=4255 Win=62208 Len=0 TSval=1362504578 TSecr=2846449532
10   0.099871 142.250.74.36 → 172.16.42.255 HTTP 9626 Continuation[Packet size limited during capture]
11   0.099936 172.16.42.255 → 142.250.74.36 TCP 66 51088 → 80 [ACK] Seq=142 Ack=13815 Win=56320 Len=0 TSval=1362504582 TSecr=2846449532
12   0.100743 172.16.42.255 → 142.250.74.36 TCP 66 51088 → 80 [FIN, ACK] Seq=142 Ack=13815 Win=64128 Len=0 TSval=1362504583 TSecr=2846449532
13   0.113167 142.250.74.36 → 172.16.42.255 TCP 66 80 → 51088 [FIN, ACK] Seq=13815 Ack=143 Win=66816 Len=0 TSval=2846449554 TSecr=1362504583
14   0.113190 172.16.42.255 → 142.250.74.36 TCP 66 51088 → 80 [ACK] Seq=143 Ack=13816 Win=64128 Len=0 TSval=1362504595 TSecr=2846449554

building from source

To build nsntrace from source the following steps are needed:

$ ./autogen.sh
$ ./configure
$ make

And to install:

$ sudo make install

dependencies

The packages needed to build nsntrace are (Debian/Ubuntu style naming):

automake
docbook-xml
docbook-xsl
iptables
libnl-route-3-dev
lippcap-dev
pkg-config
xsltproc

nsntrace's People

Contributors

Stargazers

Watchers

nsntrace's Issues

Writing to stdout is done wrong

I am using nsntrace to dump and analyze a processes network traffic in real time. In order to do that, I need to get the network dump piped to another process, not written to a file.

I am currently using this command:
sudo stdbuf -i0 -e0 -o0 nsntrace -o /dev/fd/2 -u user bash -c "some-process 1>/dev/null 2>/dev/null" 3>&2 2>&1 1>&3

This way is necessary because when I just enter - as output file name, the program doesn't correctly behave according to the standard. What nsntrace would be supposed to do is to detect that the file descriptor is stdout, and should then cease writing text (like "Starting network trace ...") to stdout and write it to the terminal or to stderr instead. By printing to stdout, the textual output gets mixed with the binary dump file and the resulting dump is corrupted and useless.

This means one needs to use this unnecessarily large construct, making the tool write to fd 2 and then, using bash tricks, swap fd 1 and fd 2, just to parse the output in a pipe.

Ideally, it should behave like tcpdump - when a output file name of "-" is given, only the binary data is given to the next program in the pipe, not any text data. That can be done by checking if stdout (fd 1) is connected to a pipe or to a terminal (or to a file).

Installation instructions / help

Hi,

Thanks for sharing this tool! I am trying to build nsntrace on Ubuntu 14.04. After installing various required packages (libpcap, xsltproc, libnl-route-3) I got as far as make which fails with:

$ make
make  all-recursive
make[1]: Entering directory `/home/quux00/lang/c/nsntrace'
Making all in src
make[2]: Entering directory `/home/quux00/lang/c/nsntrace/src'
gcc -DHAVE_CONFIG_H -I. -I..  -I/usr/include/libnl3   -Wall -Wno-missing-braces   -g -O2 -MT nsntrace-net.o -MD -MP -MF .deps/nsntrace-net.Tpo -c -o nsntrace-net.o `test -f 'net.c' || echo './'`net.c
net.c:28:37: fatal error: netlink/route/link/veth.h: No such file or directory
 #include <netlink/route/link/veth.h>

The installation steps are not documented, but here's what I figured out:

$ ./autogen.sh
$ ./configure
(install any missing libs found)
$ make

Here's the output of my configure step:

$ ./configure 
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for pcap_create in -lpcap... yes
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking pcap/pcap.h usability... yes
checking pcap/pcap.h presence... yes
checking for pcap/pcap.h... yes
checking for xsltproc... /usr/bin/xsltproc
checking for iptables... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for LIBNL... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating man/Makefile
config.status: creating tests/Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands

I tried googling the make issue, but came up short. Thanks for any help/instructions you can provide

Run tests as non-root

Right now, since the tests need to be run as root, since they use nsntrace binary which listens to raw sockets and use iptables.

Can we avoid thiss? With namespace/container trickery?

print sysctl command to enable IP forwarding instead of cat

Forwarding https://bugs.debian.org/832268

When IP forwarding isn't enabled I get this output:

$ sudo nsntrace wget debian.org
[sudo] password for pabs: 
IP forward must be enabled to run this application
# cat /proc/sys/net/ipv4/ip_forward

I would suggest this output instead:

$ sudo nsntrace wget debian.org
[sudo] password for pabs: 
Please enable IP forwarding:
# sysctl net.ipv4.ip_forward=1

If the user has run nsntrace via sudo you could do this instead:

$ sudo nsntrace wget debian.org
[sudo] password for pabs: 
Please enable
IP forwarding:
$ sudo sysctl net.ipv4.ip_forward=1

Inside a Docker (debian:buster) Container?

I am trying to get this to work from inside a Docker container. This is what I have so far:

I can build the nsntrace app in the container;
These commands in Dockerfile switch from nftables to iptables:
RUN update-alternatives --set iptables /usr/sbin/iptables-legacy
RUN update-alternatives --set iptables-restore /usr/sbin/iptables-legacy-restore
RUN update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
Start the container using the --cap NET_ADMIN option;

When I try to run it with a command like this:

nsntrace ping 1.1.1.1

I get errors:

clone: Operation not permitted
iptables: No chain/target/match by that name.

To make sure the iptables config is "sane", I have tried using iptables-restore on a file with these contents:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited

COMMIT

But the result is the same. Any ideas?

UPDATE: Adding the "--privileged" flag to the Docker run command solved it! Thanks for writing this fantastic little tool!

Leads to getaddrinfo to fail

$> sudo nsntrace -d wlp82s0 --user $USER httpfs2 -f http://datasets.datalad.org/labs/churchland/najafi-2018-nwb/data/FN_dataSharing/nwb/mouse1_fni16_150819_001_ch2-PnevPanResults-170815-163235.nwb /mnt/httpfs
Starting network trace of 'httpfs2' on interface wlp82s0.
Your IP address in this trace is 172.16.42.255.
Use ctrl-c to end at any time.

file name: 	mouse1_fni16_150819_001_ch2-PnevPanResults-170815-163235.nwb
host name: 	datasets.datalad.org
port number: 	80
protocol: 	http
request path: 	/labs/churchland/najafi-2018-nwb/data/FN_dataSharing/nwb/mouse1_fni16_150819_001_ch2-PnevPanResults-170815-163235.nwb
auth data: 	(null)
httpfs2: getaddrinfo datasets.datalad.org - Name or service not known
Connection failed.
Finished capturing 44 packets.

Command works fine if I run without nsntrace. The error when running without --user

some details of network interfaces

$> ifconfig         
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:1aff:fe8b:ce25  prefixlen 64  scopeid 0x20<link>
        ether 02:42:1a:8b:ce:25  txqueuelen 0  (Ethernet)
        RX packets 1128  bytes 68306 (66.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1233  bytes 16254976 (15.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1374  bytes 145239 (141.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1374  bytes 145239 (141.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlp82s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.34  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::5435:e7b4:e4eb:2abd  prefixlen 64  scopeid 0x20<link>
        ether 94:e6:f7:72:7c:70  txqueuelen 1000  (Ethernet)
        RX packets 1802284  bytes 2072415424 (1.9 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 664687  bytes 153508175 (146.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


$> cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 192.168.1.1

$> route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         ESCAPE          0.0.0.0         UG    600    0        0 wlp82s0
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 wlp82s0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp82s0

Version is 0~20160806-1+b1, but I have not spotted any possibly relevant changes in the git history here.

Allow parameter -p to attach to running processes

Akin to strace :
-p pid trace process with process id PID

xsl:import : unable to load http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl

After

$ ./autogen.sh
$ ./configure

make fails with

$ make
make  all-recursive
make[1]: Entering directory '/home/simon/src/nsntrace'
Making all in src
make[2]: Entering directory '/home/simon/src/nsntrace/src'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/simon/src/nsntrace/src'
Making all in tests
make[2]: Entering directory '/home/simon/src/nsntrace/tests'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/simon/src/nsntrace/tests'
Making all in man
make[2]: Entering directory '/home/simon/src/nsntrace/man'
/usr/bin/xsltproc --nonet -o nsntrace.1 ./man.xsl nsntrace.xml
I/O error : Attempt to load network entity http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl
warning: failed to load external entity "http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl"
compilation error: file ./man.xsl line 8 element import
xsl:import : unable to load http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl
Makefile:484: recipe for target 'nsntrace.1' failed
make[2]: *** [nsntrace.1] Error 5
make[2]: Leaving directory '/home/simon/src/nsntrace/man'
Makefile:359: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/simon/src/nsntrace'
Makefile:300: recipe for target 'all' failed
make: *** [all] Error 2

As a workaround, is there a way to build this without building the manpages?

No network connectivity when running inside nsntrace

When I run :

sudo tcpdump -w my.pcap -i wlp2s0
wget www.google.com

I see network requests when opening my.pcap in wireshark.

However, when running

sudo ./nsntrace -d wlp2s0 wget www.google.com

I get :

sudo ./nsntrace -d wlp2s0 wget www.google.com
Starting network trace of 'wget' on interface wlp2s0.
Your IP address in this trace is 172.16.42.255.
Use ctrl-c to end at any time.

--2017-06-29 12:30:03--  http://www.google.com/
Resolving www.google.com (www.google.com)...


failed: Name or service not known.
wget: unable to resolve host address ‘www.google.com’

Any idea what is happening ?

Do not call out to the iptables binary

I would really like to stop calling out to the iptables binary to set the iptables rules. Could we replace this with C code in someway? How do other applications like network-manager do this?

.

...

Static selection of IPs

Right now we always use 172.16.42.[254|255] for our veth device. It is unlikely, but possible that this is already taken. Maybe we should check and select a different range in that case?

I do not want a too complicated solution for this though.

Connecting to DBus in namespace not possible

Example:

** (epiphany:18019): WARNING **: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-zVCTcw3GtQ: Connection refused

pcap_findalldevs memory is not freed

The switch to pcap_findalldevs without also adding a call to pcap_freealldevs means that the memory used by the interface list returned from pcap_findalldevs is not freed and leaks into the process.

Running multiple instances of nsntrace at one time

edit: doing research, probably gonna solve that in few upcoming days
edit2: research ended, wrote python script that does the job. I don't have time right now for that, maybe in future I will solve that problem

Hi,
Im trying to capture packets of few processes at the same time.
Currently, when I run nsntrace like that:
user@host:~/nsntrace$ sudo ./src/nsntrace -d eth0 ping google.com
It produces following output:

Starting network trace of 'ping' on interface eth0.
Your IP address in this trace is 172.16.42.255.
Use ctrl-c to end at any time.

PING google.com (216.58.209.206) 56(84) bytes of data.
64 bytes from bud02s22-in-f206.1e100.net (216.58.209.206): icmp_seq=1 ttl=55 time=14.4 ms
64 bytes from bud02s22-in-f206.1e100.net (216.58.209.206): icmp_seq=2 ttl=55 time=14.5 ms
[...]

When I execute same command in another terminal window (previous still running), it produces such output:
(lines starting with [X] are added to code manually by myself, for debug purposes)

[X] failed at _nsntrace_net_create_veth -> rtnl_link_veth_add
[X] failed at nsntrace_net_init -> _nsntrace_net_create_veth
Failed to setup networking environment
user@host:~/nsntrace$

And window that was properly pinging changes to:

[...]
64 bytes from bud02s22-in-f206.1e100.net (216.58.209.206): icmp_seq=4 ttl=55 time=14.7 ms
64 bytes from bud02s22-in-f206.1e100.net (216.58.209.206): icmp_seq=5 ttl=55 time=14.4 ms
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
ping: sendmsg: Network is unreachable
^C
--- google.com ping statistics ---
13 packets transmitted, 5 received, 61% packet loss, time 12060ms
Capture interrupted, cleaning up
Finished capturing 55 packets.
iptables: No chain/target/match by that name.
user@host:~/nsntrace$

That means that nsntrace was not designed to run in parallel. I want to modify it in such way, that it would work simultaneously.
As far, I changed defines:

#define IF_BASE "nsntrace"
#define NS_IF IF_BASE "-netns"
#define GW_IF IF_BASE

To variables, randomly generated at start of the program. Also replaced all occurances of GW_BASE in the code to IF_BASE, as they have the same value. Without much reading of a code, I thought that would be the way of approaching what I wanted.
Now, clues lead to that part of code (net.c, 259 line in my setup):

_nsntrace_net_create_veth(const char *gw_iface,
			  const char *ns_iface,
			  pid_t ns_pid)
{
	int ret;
	struct nl_sock *sock = _nsntrace_net_get_nl_socket();

	if ((ret = rtnl_link_veth_add(sock, gw_iface, ns_iface, ns_pid)) < 0) {
		printf("failed at _nsntrace_net_create_veth -> rtnl_link_veth_add\n");
		return ret;
	}

	return 0;
}

Why that fails?
Why both of the instances fails? It looks like they fight with each other. They have something in common. Socket?
_nsntrace_net_get_nl_socket()
Uses:
nl_socket_alloc()
https://www.infradead.org/~tgr/libnl/doc/api/group__socket.html#gaf9458e6b88324962065f93a13d53cf2c
If I'm thinking right... that should create new socket? So why it interferes with each other?

So, final question:
Dear community, dear author: what else should I do in order to run multiple instances of nsntrace?
P.s.: Also tried to change (because that pair of IPs should be unique for each session too. Right?)
#define GW_IP IP_BASE ".254"
to
#define GW_IP IP_BASE ".253"
But that doesn't work at all:
(Same result whe4n changed NS_IP.)

user@host:~/nsntrace$ sudo ./src/nsntrace -d eth0 ping google.com
[sudo] password for user: 
Starting network trace of 'ping' on interface eth0.
Your IP address in this trace is 172.16.42.255.
Use ctrl-c to end at any time.

ping: unknown host google.com
Finished capturing 28 packets.
user@host:~/nsntrace$