Once in a while the pod watcher closes down and we start receiving nil objects in the event in the following code at line# 406.

https://github.com/kinvolk/fanotify-experiments/blob/962516fe9ca06d14095e732d4a2949deb6bae804/main.go#L392-L411

Figure out a way to keep this watcher alive or reinitiate it again.

It would be easy to make the code more configurable. We could use https://github.com/spf13/cobra

Right now the whole of code is in the main.go, move it to individual packages for docker, containerd, fanotify, etc.

This application is supposed to run as Daemonset on Kubernetes. There already is a Dockerfile, now we need to make it run from a Kubernetes pod.

Things to consider:

• We will need access to the containerd socket: /run/containerd/containerd.sock
• I am not sure if we need access to the docker socket.
• Add support to switch between using Kubeconfig and incluster config, depending on what is available.
• This pod needs privileged permissions to run!
• On the k8s API side this app needs permission to list all the pods on the current node. We can reuse the kubelet kubeconfig, but that will be forgery! Also it won't work long-term because going forward when we add CRDs we will need extensive RBAC policies.

Right now instead of going into implementing full-fledged CRDs, rely on a configmap for the policy information. The app will read the policy from a configmap with name fanotify-policy in fanotify-mon namespace with key policy. This policy will be parsed by the application to decide what pods should be selected for policy enforcement.

Here is the document to flesh out the policy language.

Every time we start monitoring a container a new goroutine is spawned but that remains active even when the container has stopped and the fanotify FD is closed.

The goroutine handleEvent, which is doing a blocking call to binary.Read via fanotify lib method GetEvent. While making a call to this function an io.Reader is passed generated from the the file descriptor.

rd := bufio.NewReader(os.NewFile(uintptr(fd), ""))

But since this call is blocked on reading from this file descriptor that we received from the kernel, even if this file descriptor is closed by us this call never returns. This is currently causing goroutine leaks.

10 @ 0xcd4925 0xcd258d 0xd3a545 0xd3a52d 0xd3a2e5 0xd436fe 0xd436f6 0xdd1af4 0xcebd3a 0xd25e27 0xd25e28 0x1fa4e85 0x1fac26a 0x1fad049 0xcbeb41
#       0xcd4924        syscall.Syscall+0x4                                                     /usr/local/go/src/syscall/asm_linux_amd64.s:20
#       0xcd258c        syscall.read+0x4c                                                       /usr/local/go/src/syscall/zsyscall_linux_amd64.go:687
#       0xd3a544        syscall.Read+0x284                                                      /usr/local/go/src/syscall/syscall_unix.go:189
#       0xd3a52c        internal/poll.ignoringEINTRIO+0x26c                                     /usr/local/go/src/internal/poll/fd_unix.go:582
#       0xd3a2e4        internal/poll.(*FD).Read+0x24                                           /usr/local/go/src/internal/poll/fd_unix.go:163
#       0xd436fd        os.(*File).read+0x5d                                                    /usr/local/go/src/os/file_posix.go:32
#       0xd436f5        os.(*File).Read+0x55                                                    /usr/local/go/src/os/file.go:119
#       0xdd1af3        bufio.(*Reader).Read+0x1b3                                              /usr/local/go/src/bufio/bufio.go:227
#       0xcebd39        io.ReadAtLeast+0x99                                                     /usr/local/go/src/io/io.go:328
#       0xd25e26        io.ReadFull+0xc46                                                       /usr/local/go/src/io/io.go:347
#       0xd25e27        encoding/binary.Read+0xc47                                              /usr/local/go/src/encoding/binary/binary.go:256
#       0x1fa4e84       github.com/s3rj1k/go-fanotify/fanotify.(*NotifyFD).GetEvent+0x64        /home/user/go/pkg/mod/github.com/s3rj1k/go-fanotify/[email protected]/fanotify.go:194
#       0x1fac269       main.(*ContainerNotifier).handleEvent+0xe9                              /home/user/code/work/fanotify-experiments/main.go:254
#       0x1fad048       main.watchContainerFANotifyEvents+0x28                                  /home/user/code/work/fanotify-experiments/main.go:368

NOTE: Got this information by adding the following snippet to the main function:

// Debug go routines
go func() {
        log.Println(http.ListenAndServe("192.168.122.252:6060", nil))
}()

Then visit curl 192.168.122.252:6060/debug/pprof/goroutine?debug=1.