Git Product home page Git Product logo

kubectl-capture's Introduction

Kubectl Sysdig Capture plugin

Sysdig is a powerful open source tool for container troubleshooting, performance tunning and security investigation.

This repository implements a kubectl plugin which triggers a capture in the underlying host which is running a pod. A capture file is created for a duration of time and is download locally in order to use it with Sysdig Inspect.

Installing

In order to use this plugin, just copy the kubectl-capture to your PATH, and ensure it has execution permissions.

You can verify its installation using kubectl:

$ kubectl plugin list
The following kubectl-compatible plugins are available:

/usr/local/bin/kubectl-capture

In this case is installed into /usr/local/bin, but will work with another location listed in PATH.

Getting started

Once you have the kubectl plugin installed, you can start taking captures:

$ kubectl capture nginx-78f5d695bd-bcbd8
Sysdig is starting to capture system calls:

Node: gke-sysdig-work-default-pool-e35da3a1-m8vp
Pod: nginx-78f5d695bd-bcbd8
Duration: 30 seconds
Parameters for Sysdig: -S -M 30 -pk -z -w /capture-nginx-78f5d695bd-bcbd8-1550246926.scap.gz

The capture has been downloaded to your hard disk at:
~/captures/capture-nginx-78f5d695bd-bcbd8-1550246926.scap.gz

And then, you can start troubleshooting with Sysdig Inspect.

Extra initialization time

When the capture container is being spinned, it takes some time to compile the Sysdig Kernel module and start to capture system calls. You can check the logs of the Sysdig Capture Pod if you need to know with accuracy when Sysdig starts to capture.

Parameters

There are a few parameters for this plugin:

Flag Description
-ns or --namespace The namespace scope of the target Pod
--ebpf Use eBPF probe instead of kernel module for capturing syscalls

Aditionally, all the flags for the sysdig cli tool are supported. You can check more of these parameters in its documentation.

Cleanup

You can uninstall this plugin from kubectl by simply removing it from your PATH:

$ rm /usr/local/bin/kubectl-capture

kubectl-capture's People

Contributors

nestorsalceda avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

kubectl-capture's Issues

Error during capturing

Hi team,
I am executing : kubectl capture xxx-deployment-687bd87b86-p2264 but getting error during gunzip

$ gunzip capture-xxx-deployment-687bd87b86-p2264-1571295964.scap.gz

gzip: capture-xxx-deployment-687bd87b86-p2264-1571295964.scap.gz: unexpected end of file
$  kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl logs capture-xxx-deployment-687bd87b86-p2264-1571295508 -f
* Setting up /usr/src links from host
* Unloading sysdig-probe, if present
* Running dkms install for sysdig

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area.......
make -j24 KERNELRELEASE=4.4.0-161-generic -C /lib/modules/4.4.0-161-generic/build M=/var/lib/dkms/sysdig/0.26.4/build..................(bad exit status: 2)
Error! Bad return status for module build on kernel: 4.4.0-161-generic (x86_64)
Consult /var/lib/dkms/sysdig/0.26.4/build/make.log for more information.
* Running dkms build failed, dumping /var/lib/dkms/sysdig/0.26.4/build/make.log
DKMS make.log for sysdig-0.26.4 for kernel 4.4.0-161-generic (x86_64)
Thu Oct 17 06:58:45 UTC 2019
make: Entering directory '/host/usr/src/linux-headers-4.4.0-161-generic'
  LD      /var/lib/dkms/sysdig/0.26.4/build/built-in.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/main.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/dynamic_params_table.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/fillers_table.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/flags_table.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/ppm_events.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/ppm_fillers.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/event_table.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/syscall_table.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/ppm_cputime.o
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:285: /var/lib/dkms/sysdig/0.26.4/build/syscall_table.o] Error 4
make[1]: *** Waiting for unfinished jobs....
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:285: /var/lib/dkms/sysdig/0.26.4/build/main.o] Error 4
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:285: /var/lib/dkms/sysdig/0.26.4/build/ppm_fillers.o] Error 4
make: *** [Makefile:1454: _module_/var/lib/dkms/sysdig/0.26.4/build] Error 2
make: Leaving directory '/host/usr/src/linux-headers-4.4.0-161-generic'
* Trying to load a system sysdig-probe, if present
* Trying to find precompiled sysdig-probe for 4.4.0-161-generic
Found kernel config at /host/boot/config-4.4.0-161-generic
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.26.4-x86_64-4.4.0-161-generic-d4ee864423f81196c164a77a83c77e7a.ko
Download succeeded, loading module
* Capturing system calls
/bin/bash: line 1:  1085 Segmentation fault      (core dumped) sysdig -S -M 120 -pk -z -w /capture-xxx-deployment-687bd87b86-p2264-1571295508.scap.gz
$  kubectl exec -it capture-xxx-deployment-687bd87b86-p2264-1571295964 bash
root@capture-xxx-deployment-687bd87b86-p2264-1571295964:/# tail -f /var/lib/dkms/sysdig/0.26.4/build/make.log
DKMS make.log for sysdig-0.26.4 for kernel 4.4.0-161-generic (x86_64)
Thu Oct 17 07:06:21 UTC 2019
make: Entering directory '/host/usr/src/linux-headers-4.4.0-161-generic'
  LD      /var/lib/dkms/sysdig/0.26.4/build/built-in.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/main.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/dynamic_params_table.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/fillers_table.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/flags_table.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/ppm_events.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/ppm_fillers.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/event_table.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/syscall_table.o
  CC [M]  /var/lib/dkms/sysdig/0.26.4/build/ppm_cputime.o
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:285: /var/lib/dkms/sysdig/0.26.4/build/ppm_fillers.o] Error 4
make[1]: *** Waiting for unfinished jobs....
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[1]: *** [scripts/Makefile.build:285: /var/lib/dkms/sysdig/0.26.4/build/main.o] Error 4
make: *** [Makefile:1454: _module_/var/lib/dkms/sysdig/0.26.4/build] Error 2
make: Leaving directory '/host/usr/src/linux-headers-4.4.0-161-generic'
command terminated with exit code 137

Cannot capture pod

here is the capturer pod log

* Setting up /usr/src links from host
* Unloading sysdig-probe, if present
* Running dkms install for sysdig
Error! echo
Your kernel headers for kernel 4.19.12-1.el7.elrepo.x86_64 cannot be found at
/lib/modules/4.19.12-1.el7.elrepo.x86_64/build or /lib/modules/4.19.12-1.el7.elrepo.x86_64/source.
* Running dkms build failed, couldn't find /var/lib/dkms/sysdig/0.26.7/build/make.log
* Trying to load a system sysdig-probe, if present
* Trying to find precompiled sysdig-probe for 4.19.12-1.el7.elrepo.x86_64
Found kernel config at /host/boot/config-4.19.12-1.el7.elrepo.x86_64
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.26.7-x86_64-4.19.12-1.el7.elrepo.x86_64-6fcd60dfcaa30c08e9b5cc3ebeb7efe5.ko
Download of sysdig-probe for version 0.26.7 failed. This is because the probe for this particular version does not exist in the repo.
Consider compiling your own sysdig-probe and loading it or getting in touch with the sysdig community
* Capturing system calls
Unable to load the driver
error opening device /host/dev/sysdig0. Make sure you have root credentials and that the sysdig-probe module is loaded.

what should i do ?

Weird cli syntax.

Any chance of passing a -n for the namespace like the rest of kubectl and most kubectl plugins?

$ kubectl capture -ns coolnamespace utility
Sysdig is starting to capture system calls:

Node:
Pod: -ns
Duration: 120 seconds
Parameters for Sysdig: -S -M 120 -pk -z -w /capture--ns-1592326182.scap.gz coolnamespace utility

^^ confusing.

Capture files not saved

$ kubectl capture app-586868cdc-8dkgm -ns mynamespace -M 30 --snaplen 256
Sysdig is starting to capture system calls:

Node: gke-cluster-default-pool-93caf4f1-6b2j
Pod: app-586868cdc-8dkgm
Duration: 30 seconds
Parameters for Sysdig: -S -M 30 -pk -z -w /capture-app-586868cdc-8dkgm-1592332339.scap.gz  --snaplen 256

The capture has been downloaded to your hard disk at:
/Users/john/capture-app-586868cdc-8dkgm-1592332339.scap.gz


$ ls /Users/john/capture-app-586868cdc-8dkgm-1592332339.scap.gz
gls: cannot access '/Users/john/capture-app-586868cdc-8dkgm-1592332339.scap.gz': No such file or directory
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-13T11:51:44Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.9-gke.24", GitCommit:"39e41a8d6b7221b901a95d3af358dea6994b4a40", GitTreeState:"clean", BuildDate:"2020-02-29T01:24:35Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

Not 100% sure why it isn't working, but the message that it wrote out the file makes it seem like kubectl-capture isn't validating the results of the sysdig capture.

kubectl capture dosen't create the captured file

Trying to capture the systemcalls both with or without the --ebpf flag. No file is created on my pc after the capture as the tool says it has done:

`kubectl capture podName --namespace NameSpaceName --ebpf
Sysdig is starting to capture system calls:

Node: k8s-nod01
Pod: podName
Duration: 120 seconds
Parameters for Sysdig: -S -M 120 -pk -z -w /capture-podName.scap.gz

The capture has been downloaded to your hard disk at:
/home/USER/captures/capture-podName.scap.gz
`
if I go ls in my folder, theres nothing there.

Block errors when opening scap files from the kubectl capture plugin

adamantium:kubectl-capture jhayner$ /Applications/Sysdig\ Inspect.app/Contents/Resources/app/ember-electron/resources/sysdig/sysdig --version

sysdig version 0.26.1

adamantium:kubectl-capture jhayner$ file capture-1563982162.scap

capture-1563982162.scap: pcap-ng capture file - version 1.2

adamantium:kubectl-capture jhayner$ /Applications/Sysdig\ Inspect.app/Contents/Resources/app/ember-electron/resources/sysdig/sysdig -r capture-1563982162.scap

res = 1
event block length 17835928 greater than read buffer size 65536

I get the same results on a minimal-install of CentOS 7.6 and running the automatic installation.

uname -a
Linux prometheius@adamantium 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

sysdig --version

sysdig version 0.26.1

[jhayner@prometheius@adamantium ~]$ file capture-content-engineering-jenkins-worker-ephemeral-dedicated-07ccv-1564003390.scap

capture-content-engineering-jenkins-worker-ephemeral-dedicated-07ccv-1564003390.scap: pcap-ng capture file - version 1.2
[jhayner@prometheius@adamantium ~]$ sysdig -r capture-content-engineering-jenkins-worker-ephemeral-dedicated-07ccv-1564003390.scap

res = 1
event block length 17835928 greater than read buffer size 65536

Please also reference the following issues:
draios/sysdig#867
draios/sysdig-inspect#58 (comment)

Capture pod doesn´t start properly

I cannot start properly the capturer. Seems that sysdig pod doesn´t generate gz because seems that it is not able to start.
Is there any kind of incompatibilty?

Context
Running capture on CoreOS Container Linux by CoreOS 2023.4.0 (Rhyolite)
Nodes on AWS.
Kubernetes version v1.13.4

Command
kubectl capture kube-proxy-z4g6c -ns kube-system -M 10 --snaplen 256

* Setting up /usr/src links from host
ls: cannot access '/host/usr/src': No such file or directory
* Unloading sysdig-probe, if present
* Running dkms install for sysdig

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area.....
make -j4 KERNELRELEASE=4.19.23-coreos-r1 -C /lib/modules/4.19.23-coreos-r1/build M=/var/lib/dkms/sysdig/0.25/build..........(bad exit status: 2)
Error! Bad return status for module build on kernel: 4.19.23-coreos-r1 (x86_64)
Consult /var/lib/dkms/sysdig/0.25/build/make.log for more information.
* Running dkms build failed, dumping /var/lib/dkms/sysdig/0.25/build/make.log
DKMS make.log for sysdig-0.25 for kernel 4.19.23-coreos-r1 (x86_64)
Fri Apr  5 09:40:27 UTC 2019
make: Entering directory '/host/lib/modules/4.19.23-coreos-r1/build'
  CC [M]  /var/lib/dkms/sysdig/0.25/build/main.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/dynamic_params_table.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/fillers_table.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/flags_table.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/ppm_events.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/ppm_fillers.o
  CC [M]  /var/lib/dkms/sysdig/0.25/build/event_table.o
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[3]: *** [../source/scripts/Makefile.build:303: /var/lib/dkms/sysdig/0.25/build/main.o] Error 4
make[3]: *** Deleting file '/var/lib/dkms/sysdig/0.25/build/main.o'
make[3]: *** Waiting for unfinished jobs....
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
make[3]: *** [../source/scripts/Makefile.build:303: /var/lib/dkms/sysdig/0.25/build/ppm_fillers.o] Error 4
make[3]: *** Deleting file '/var/lib/dkms/sysdig/0.25/build/ppm_fillers.o'
make[3]: *** [../source/scripts/Makefile.build:303: /var/lib/dkms/sysdig/0.25/build/ppm_events.o] Error 4
make[3]: *** Deleting file '/var/lib/dkms/sysdig/0.25/build/ppm_events.o'
make[2]: *** [/host/lib/modules/4.19.23-coreos-r1/source/Makefile:1521: _module_/var/lib/dkms/sysdig/0.25/build] Error 2
make[1]: *** [Makefile:146: sub-make] Error 2
make: *** [Makefile:24: __sub-make] Error 2
make: Leaving directory '/host/lib/modules/4.19.23-coreos-r1/build'
* Trying to load a system sysdig-probe, if present
* Trying to find precompiled sysdig-probe for 4.19.23-coreos-r1
Found kernel config at /proc/config.gz
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.25-x86_64-4.19.23-coreos-r1-03bf994bd8b87756106f34511fc1aadb.ko
Download failed, consider compiling your own sysdig-probe and loading it or getting in touch with the sysdig community
* Capturing system calls
Unable to load the driver
error opening device /host/dev/sysdig0. Make sure you have root credentials and that the sysdig-probe module is loaded.
----------------------
Event           #Calls
----------------------

Capture files not saved with ebpf and wrong format without it

When I run the following command:
sudo kubectl capture test --ebpf -M 10 --snaplen 256

It says the capture has been saved to the working directory, this is however not the case. I can't find the file anywhere even if I copy paste the entire listed directory path.

Another thing that I noticed is that when I run the same command without --ebpf the capture gzip file seems to be corrupted.

gzip: capture-test-1557830100.scap.gz: unexpected end of file

When I try to use archive manager to extract the files it says that an error occurred.

Cannot copy from privileged container

I came across a following issue. Once started the capture will never stop (until I ctrl+c it). Looking into the code I found out that the wait_until_finished function is checking if the sysdig is done by trying to copy .finished file from the container. So I checked what happens if I run the command manually. It fails because of following:

$ kubectl cp capture--ns-1586179517:/.finished .finished Error from server (Forbidden): pods "capture--ns-1586179517" is forbidden: cannot exec into or attach to a privileged container
Perhaps the function could be moved to a dedicated "watchdog" sidecar container (not privileged) sharing a mount with the main - sysdig container where the .finished and the capture file would be written to?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.