When tetragon is installed via helm as instructed, some fields are populated by null strings (example in data below) instead of the expected data:
{"process_exec":{"process":{"exec_id":"AAAAAAAAADoxNjcxODQ1Mjc0NTk0MTc6MTczMjQ4OA==","pid":1732488,"uid":0,"cwd":"/","binary":"/usr/bin/tetra","arguments":"status","flags":"execve rootcwd clone","start_time":"2022-06-24T01:27:13.470Z","auid":4294967295,"pod":{"namespace":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","container":{"id":"containerd://9001291f84d1fa67aeaafd7880621865e6b7b398a67294803716581e27c5d7a5","name":"tetragon","image":{"id":"quay.io/cilium/tetragon@sha256:bb81d915aafdefa1a7873de30791e5a4698322d463af51195b4c262060fcc703","name":"quay.io/cilium/tetragon:v0.8.0"},"start_time":"2022-06-23T18:52:01Z","pid":31562}},"docker":"9001291f84d1fa67aeaafd788062186","parent_exec_id":"AAAAAAAAADoxNjcxODQ0ODI1MDI1Nzg6MTczMjQ3OA==","refcnt":1},"parent":{"exec_id":"AAAAAAAAADoxNjcxODQ0ODI1MDI1Nzg6MTczMjQ3OA==","pid":1732478,"uid":0,"cwd":"/var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/462cbabc0ce9d0f9c11d21fbb6c8d62850f21eee7bd4289f04745b26cf1e6acd/","binary":"/snap/microk8s/3272/bin/runc","arguments":"--root /run/containerd/runc/k8s.io --log /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9001291f84d1fa67aeaaf 7880621865e6b7b398a67294803716581e27c5d7a5/log.json --log-format json exec --process /var/snap/microk8s/common/run/runc-process2712392004 --detach --pid-file /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9001291f84d1fa67aeaaf 7880621865e6b7b398a67294803716581e27c5d7a5/5a342fb48ed6670a76e757b94846ea207d1e73296b5fde67622d6ac3 09bc01f.pid 9001291f84d1fa67aeaafd7880621865e6b7b398a67294803716581e27c5d7a5","flags":"execve clone","start_time":"2022-06-24T01:27:13.425Z","auid":4294967295,"parent_exec_id":"AAAAAAAAADoxNjI1OTAwMDAwMDA6NTEwOQ==","refcnt":1}},"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-06-24T01:27:13.470Z"}
{"process_exit":{"process":{"exec_id":"AAAAAAAAADoxNjcxODQ1Mjc0NTk0MTc6MTczMjQ4OA==","pid":1732488,"uid":0,"cwd":"/","binary":"/usr/bin/tetra","arguments":"status","flags":"execve rootcwd clone","start_time":"2022-06-24T01:27:13.470Z","auid":4294967295,"pod":{"namespace":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","container":{"id":"containerd://9001291f84d1fa67aeaafd7880621865e6b7b398a67294803716581e27c5d7a5","name":"tetragon","image":{"id":"quay.io/cilium/tetragon@sha256:bb81d915aafdefa1a7873de30791e5a4698322d463af51195b4c262060fcc703","name":"quay.io/cilium/tetragon:v0.8.0"},"start_time":"2022-06-23T18:52:01Z","pid":31562}},"docker":"9001291f84d1fa67aeaafd788062186","parent_exec_id":"AAAAAAAAADoxNjcxODQ0ODI1MDI1Nzg6MTczMjQ3OA=="},"parent":{"exec_id":"AAAAAAAAADoxNjcxODQ0ODI1MDI1Nzg6MTczMjQ3OA==","pid":1732478,"uid":0,"cwd":"/var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/462cbabc0ce9d0f9c11d21fbb6c8d62850f21eee7bd4289f04745b26cf1e6acd/","binary":"/snap/microk8s/3272/bin/runc","arguments":"--root /run/containerd/runc/k8s.io --log /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9001291f84d1fa67aeaaf 7880621865e6b7b398a67294803716581e27c5d7a5/log.json --log-format json exec --process /var/snap/microk8s/common/run/runc-process2712392004 --detach --pid-file /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/9001291f84d1fa67aeaaf 7880621865e6b7b398a67294803716581e27c5d7a5/5a342fb48ed6670a76e757b94846ea207d1e73296b5fde67622d6ac3 09bc01f.pid 9001291f84d1fa67aeaafd7880621865e6b7b398a67294803716581e27c5d7a5","flags":"execve clone","start_time":"2022-06-24T01:27:13.425Z","auid":4294967295,"parent_exec_id":"AAAAAAAAADoxNjI1OTAwMDAwMDA6NTEwOQ==","refcnt":4294967295}},"node_name":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000","time":"2022-06-24T01:27:13.476Z"}
It isn't just tetragon container events that have these nulls like the example, other container's tetragon trace data has also been observed to be impacted.
This behavior hasn't been appearing immediately when tetragon is installed, it seems to take some time before it starts happening, perhaps a few hours or a day, then it continues to occur.
I have observed this behavior across multiple builds of k3s and microk8s clusters but only when tetragon is deployed. We can also see related data in the microk8s kubelite daemon log:
microk8s.daemon-kubelite.log:Jun 23 13:17:55 moon2 microk8s.daemon-kubelite[16056]: E0623 13:17:55.084224 16056 status.go:71] apiserver received an error that is not an metav1.Status: storage.InvalidError{Errs:field.ErrorList{(*field.Error)(0xc014330000)}}: resourceVersion: Invalid value: "\x00\x00\x00\x00\x00\x00": strconv.ParseUint: parsing "\x00\x00\x00\x00\x00\x00": invalid syntax
This behavior does not occur when tetragon is not running/installed into the clusters.
Tests for this I have done so far have been with both k3s and microk8s clusters.
Cluster nodes in the tests have been running OpenSUSE Leap 15.4, Ubuntu 22 Server, Ubuntu 22 Desktop, (kernels 5.15.0-39-generic and 5.14.21-150400.22-default at the moment).
Currently I have a microk8s cluster running with this happening.