sysdiglabs / kubectl-capture Goto Github PK
View Code? Open in Web Editor NEWA kubectl plugin which triggers a Sysdig capture
Home Page: https://sysdig.com/opensource/inspect/
License: Apache License 2.0
A kubectl plugin which triggers a Sysdig capture
Home Page: https://sysdig.com/opensource/inspect/
License: Apache License 2.0
Hi team,
I am executing : kubectl capture xxx-deployment-687bd87b86-p2264
but getting error during gunzip
$ gunzip capture-xxx-deployment-687bd87b86-p2264-1571295964.scap.gz
gzip: capture-xxx-deployment-687bd87b86-p2264-1571295964.scap.gz: unexpected end of file
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl logs capture-xxx-deployment-687bd87b86-p2264-1571295508 -f
* Setting up /usr/src links from host
* Unloading sysdig-probe, if present
* Running dkms install for sysdig
Kernel preparation unnecessary for this kernel. Skipping...
Building module:
cleaning build area.......
make -j24 KERNELRELEASE=4.4.0-161-generic -C /lib/modules/4.4.0-161-generic/build M=/var/lib/dkms/sysdig/0.26.4/build..................(bad exit status: 2)
Error! Bad return status for module build on kernel: 4.4.0-161-generic (x86_64)
Consult /var/lib/dkms/sysdig/0.26.4/build/make.log for more information.
* Running dkms build failed, dumping /var/lib/dkms/sysdig/0.26.4/build/make.log
DKMS make.log for sysdig-0.26.4 for kernel 4.4.0-161-generic (x86_64)
Thu Oct 17 06:58:45 UTC 2019
make: Entering directory '/host/usr/src/linux-headers-4.4.0-161-generic'
LD /var/lib/dkms/sysdig/0.26.4/build/built-in.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/main.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/dynamic_params_table.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/fillers_table.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/flags_table.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/ppm_events.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/ppm_fillers.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/event_table.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/syscall_table.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/ppm_cputime.o
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:285: /var/lib/dkms/sysdig/0.26.4/build/syscall_table.o] Error 4
make[1]: *** Waiting for unfinished jobs....
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:285: /var/lib/dkms/sysdig/0.26.4/build/main.o] Error 4
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:285: /var/lib/dkms/sysdig/0.26.4/build/ppm_fillers.o] Error 4
make: *** [Makefile:1454: _module_/var/lib/dkms/sysdig/0.26.4/build] Error 2
make: Leaving directory '/host/usr/src/linux-headers-4.4.0-161-generic'
* Trying to load a system sysdig-probe, if present
* Trying to find precompiled sysdig-probe for 4.4.0-161-generic
Found kernel config at /host/boot/config-4.4.0-161-generic
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.26.4-x86_64-4.4.0-161-generic-d4ee864423f81196c164a77a83c77e7a.ko
Download succeeded, loading module
* Capturing system calls
/bin/bash: line 1: 1085 Segmentation fault (core dumped) sysdig -S -M 120 -pk -z -w /capture-xxx-deployment-687bd87b86-p2264-1571295508.scap.gz
$ kubectl exec -it capture-xxx-deployment-687bd87b86-p2264-1571295964 bash
root@capture-xxx-deployment-687bd87b86-p2264-1571295964:/# tail -f /var/lib/dkms/sysdig/0.26.4/build/make.log
DKMS make.log for sysdig-0.26.4 for kernel 4.4.0-161-generic (x86_64)
Thu Oct 17 07:06:21 UTC 2019
make: Entering directory '/host/usr/src/linux-headers-4.4.0-161-generic'
LD /var/lib/dkms/sysdig/0.26.4/build/built-in.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/main.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/dynamic_params_table.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/fillers_table.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/flags_table.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/ppm_events.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/ppm_fillers.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/event_table.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/syscall_table.o
CC [M] /var/lib/dkms/sysdig/0.26.4/build/ppm_cputime.o
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:285: /var/lib/dkms/sysdig/0.26.4/build/ppm_fillers.o] Error 4
make[1]: *** Waiting for unfinished jobs....
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:285: /var/lib/dkms/sysdig/0.26.4/build/main.o] Error 4
make: *** [Makefile:1454: _module_/var/lib/dkms/sysdig/0.26.4/build] Error 2
make: Leaving directory '/host/usr/src/linux-headers-4.4.0-161-generic'
command terminated with exit code 137
$ kubectl capture app-586868cdc-8dkgm -ns mynamespace -M 30 --snaplen 256
Sysdig is starting to capture system calls:
Node: gke-cluster-default-pool-93caf4f1-6b2j
Pod: app-586868cdc-8dkgm
Duration: 30 seconds
Parameters for Sysdig: -S -M 30 -pk -z -w /capture-app-586868cdc-8dkgm-1592332339.scap.gz --snaplen 256
The capture has been downloaded to your hard disk at:
/Users/john/capture-app-586868cdc-8dkgm-1592332339.scap.gz
$ ls /Users/john/capture-app-586868cdc-8dkgm-1592332339.scap.gz
gls: cannot access '/Users/john/capture-app-586868cdc-8dkgm-1592332339.scap.gz': No such file or directory
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-13T11:51:44Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.9-gke.24", GitCommit:"39e41a8d6b7221b901a95d3af358dea6994b4a40", GitTreeState:"clean", BuildDate:"2020-02-29T01:24:35Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}
Not 100% sure why it isn't working, but the message that it wrote out the file makes it seem like kubectl-capture isn't validating the results of the sysdig capture.
Any chance of passing a -n for the namespace like the rest of kubectl and most kubectl plugins?
$ kubectl capture -ns coolnamespace utility
Sysdig is starting to capture system calls:
Node:
Pod: -ns
Duration: 120 seconds
Parameters for Sysdig: -S -M 120 -pk -z -w /capture--ns-1592326182.scap.gz coolnamespace utility
^^ confusing.
When I run the following command:
sudo kubectl capture test --ebpf -M 10 --snaplen 256
It says the capture has been saved to the working directory, this is however not the case. I can't find the file anywhere even if I copy paste the entire listed directory path.
Another thing that I noticed is that when I run the same command without --ebpf
the capture gzip file seems to be corrupted.
gzip: capture-test-1557830100.scap.gz: unexpected end of file
When I try to use archive manager to extract the files it says that an error occurred.
Trying to capture the systemcalls both with or without the --ebpf flag. No file is created on my pc after the capture as the tool says it has done:
`kubectl capture podName --namespace NameSpaceName --ebpf
Sysdig is starting to capture system calls:
Node: k8s-nod01
Pod: podName
Duration: 120 seconds
Parameters for Sysdig: -S -M 120 -pk -z -w /capture-podName.scap.gz
The capture has been downloaded to your hard disk at:
/home/USER/captures/capture-podName.scap.gz
`
if I go ls in my folder, theres nothing there.
I cannot start properly the capturer. Seems that sysdig pod doesn´t generate gz because seems that it is not able to start.
Is there any kind of incompatibilty?
Context
Running capture on CoreOS Container Linux by CoreOS 2023.4.0 (Rhyolite)
Nodes on AWS.
Kubernetes version v1.13.4
Command
kubectl capture kube-proxy-z4g6c -ns kube-system -M 10 --snaplen 256
* Setting up /usr/src links from host
ls: cannot access '/host/usr/src': No such file or directory
* Unloading sysdig-probe, if present
* Running dkms install for sysdig
Kernel preparation unnecessary for this kernel. Skipping...
Building module:
cleaning build area.....
make -j4 KERNELRELEASE=4.19.23-coreos-r1 -C /lib/modules/4.19.23-coreos-r1/build M=/var/lib/dkms/sysdig/0.25/build..........(bad exit status: 2)
Error! Bad return status for module build on kernel: 4.19.23-coreos-r1 (x86_64)
Consult /var/lib/dkms/sysdig/0.25/build/make.log for more information.
* Running dkms build failed, dumping /var/lib/dkms/sysdig/0.25/build/make.log
DKMS make.log for sysdig-0.25 for kernel 4.19.23-coreos-r1 (x86_64)
Fri Apr 5 09:40:27 UTC 2019
make: Entering directory '/host/lib/modules/4.19.23-coreos-r1/build'
CC [M] /var/lib/dkms/sysdig/0.25/build/main.o
CC [M] /var/lib/dkms/sysdig/0.25/build/dynamic_params_table.o
CC [M] /var/lib/dkms/sysdig/0.25/build/fillers_table.o
CC [M] /var/lib/dkms/sysdig/0.25/build/flags_table.o
CC [M] /var/lib/dkms/sysdig/0.25/build/ppm_events.o
CC [M] /var/lib/dkms/sysdig/0.25/build/ppm_fillers.o
CC [M] /var/lib/dkms/sysdig/0.25/build/event_table.o
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[3]: *** [../source/scripts/Makefile.build:303: /var/lib/dkms/sysdig/0.25/build/main.o] Error 4
make[3]: *** Deleting file '/var/lib/dkms/sysdig/0.25/build/main.o'
make[3]: *** Waiting for unfinished jobs....
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[3]: *** [../source/scripts/Makefile.build:303: /var/lib/dkms/sysdig/0.25/build/ppm_fillers.o] Error 4
make[3]: *** Deleting file '/var/lib/dkms/sysdig/0.25/build/ppm_fillers.o'
make[3]: *** [../source/scripts/Makefile.build:303: /var/lib/dkms/sysdig/0.25/build/ppm_events.o] Error 4
make[3]: *** Deleting file '/var/lib/dkms/sysdig/0.25/build/ppm_events.o'
make[2]: *** [/host/lib/modules/4.19.23-coreos-r1/source/Makefile:1521: _module_/var/lib/dkms/sysdig/0.25/build] Error 2
make[1]: *** [Makefile:146: sub-make] Error 2
make: *** [Makefile:24: __sub-make] Error 2
make: Leaving directory '/host/lib/modules/4.19.23-coreos-r1/build'
* Trying to load a system sysdig-probe, if present
* Trying to find precompiled sysdig-probe for 4.19.23-coreos-r1
Found kernel config at /proc/config.gz
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.25-x86_64-4.19.23-coreos-r1-03bf994bd8b87756106f34511fc1aadb.ko
Download failed, consider compiling your own sysdig-probe and loading it or getting in touch with the sysdig community
* Capturing system calls
Unable to load the driver
error opening device /host/dev/sysdig0. Make sure you have root credentials and that the sysdig-probe module is loaded.
----------------------
Event #Calls
----------------------
I came across a following issue. Once started the capture will never stop (until I ctrl+c it). Looking into the code I found out that the wait_until_finished function is checking if the sysdig is done by trying to copy .finished file from the container. So I checked what happens if I run the command manually. It fails because of following:
$ kubectl cp capture--ns-1586179517:/.finished .finished Error from server (Forbidden): pods "capture--ns-1586179517" is forbidden: cannot exec into or attach to a privileged container
Perhaps the function could be moved to a dedicated "watchdog" sidecar container (not privileged) sharing a mount with the main - sysdig container where the .finished and the capture file would be written to?
here is the capturer pod log
* Setting up /usr/src links from host
* Unloading sysdig-probe, if present
* Running dkms install for sysdig
Error! echo
Your kernel headers for kernel 4.19.12-1.el7.elrepo.x86_64 cannot be found at
/lib/modules/4.19.12-1.el7.elrepo.x86_64/build or /lib/modules/4.19.12-1.el7.elrepo.x86_64/source.
* Running dkms build failed, couldn't find /var/lib/dkms/sysdig/0.26.7/build/make.log
* Trying to load a system sysdig-probe, if present
* Trying to find precompiled sysdig-probe for 4.19.12-1.el7.elrepo.x86_64
Found kernel config at /host/boot/config-4.19.12-1.el7.elrepo.x86_64
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.26.7-x86_64-4.19.12-1.el7.elrepo.x86_64-6fcd60dfcaa30c08e9b5cc3ebeb7efe5.ko
Download of sysdig-probe for version 0.26.7 failed. This is because the probe for this particular version does not exist in the repo.
Consider compiling your own sysdig-probe and loading it or getting in touch with the sysdig community
* Capturing system calls
Unable to load the driver
error opening device /host/dev/sysdig0. Make sure you have root credentials and that the sysdig-probe module is loaded.
what should i do ?
Looks like this plugin would be a great fit for adding to the krew
package manager.
minikube
type: Control Plane
host: Running
kubelet: Stopped
apiserver: Stopped
kubeconfig: Configured
Successfully capture pod log and scrap.gz file download to local MacOs machine. But unable to extract .gz file and getting
gunzip - unexpected end of file
tar - Unrecognized archive format
adamantium:kubectl-capture jhayner$ /Applications/Sysdig\ Inspect.app/Contents/Resources/app/ember-electron/resources/sysdig/sysdig --version
sysdig version 0.26.1
adamantium:kubectl-capture jhayner$ file capture-1563982162.scap
capture-1563982162.scap: pcap-ng capture file - version 1.2
adamantium:kubectl-capture jhayner$ /Applications/Sysdig\ Inspect.app/Contents/Resources/app/ember-electron/resources/sysdig/sysdig -r capture-1563982162.scap
res = 1
event block length 17835928 greater than read buffer size 65536
I get the same results on a minimal-install of CentOS 7.6 and running the automatic installation.
uname -a
Linux prometheius@adamantium 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
sysdig --version
sysdig version 0.26.1
[jhayner@prometheius@adamantium ~]$ file capture-content-engineering-jenkins-worker-ephemeral-dedicated-07ccv-1564003390.scap
capture-content-engineering-jenkins-worker-ephemeral-dedicated-07ccv-1564003390.scap: pcap-ng capture file - version 1.2
[jhayner@prometheius@adamantium ~]$ sysdig -r capture-content-engineering-jenkins-worker-ephemeral-dedicated-07ccv-1564003390.scap
res = 1
event block length 17835928 greater than read buffer size 65536
Please also reference the following issues:
draios/sysdig#867
draios/sysdig-inspect#58 (comment)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.