critest for container security context:

• NamespaceOption
  • hostPID
  • hostIPC #82
  • hostNetwork #75
• Priviledged #78
• Capabilities
• RunAsUser/RunAsUserName #41
• ReadOnlyRootfs

Currently if a container exit with exit code 0, when we do crictl inspect, there won't be exit code field, because that is omitted https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/v1alpha1/runtime/api.pb.go#L2119.

We should use github.com/golang/protobuf/jsonpb, so that we could configure EmitDefaults to fix this.

We introduced new test to validate "non exist host path is allowed": #148

But I think though it's consistent with Docker, it may not suitable to apply to all runtimes. For example, runV does not allow no-exist host path since user may mis-spelled the host path. (Not sure what's case for cri-o?)

So I would like to make sure if we really think validate no exist host path is necessary (then we will update runV), otherwise we may want to skip or remove this test.

cc @Random-Liu @feiskyer @runcom

Currently we are listing image by id, and only include the first repo tag.

We should organize the information better.

Docker lists all image tags even they have the same id.

Add CLI for crictl image, e.g.

crictl image list
crictl image pull [--auth ] NAME[:TAG|@DIGEST]
crictl image remove NAME[:TAG|@DIGEST]
crictl image status NAME[:TAG|@DIGEST]

We are about to release first version for kubernetes v1.7.

TODOs

• Bump tools version to 0.1.0
• Add release notes
• Add changelog

crictl is designed for CRI general debugging purpose, json format is less easy to read/write for human, is there any plan to support yaml format in run podsandbox or create container related command?

Umbrella issues for crictl, the CLI for kubelet CRI.

global options

--debug
--help
--timeout
--runtime-endpoint
--image-endpoint

Sub commands

• version
• status
• sandbox #40
  • sandbox run config.json|yaml
  • sandbox status sandbox-id
  • sandbox stop sandbox-id
  • sandbox remove sandbox-id
  • sandbox list
• container
  • container create sandbox-id config.json|yaml
  • container start container-id
  • container stop container-id
  • container remove container-id
  • container status container-id
  • container list
• image
  • image list
  • image pull [--auth ] NAME[:TAG|@digest]
  • image remove NAME[:TAG|@digest]
  • image status NAME[:TAG|@digest]
• exec [--sync]
• attach
• portforward

cc/ @mrunalp @yujuhong @Random-Liu @xlgao-zju Any comments on the crictl?

Add CLI for crictl sandbox, e.g.

crictl sandbox run config.json|yaml
crictl sandbox status sandbox-id
crictl sandbox stop sandbox-id
crictl sandbox remove sandbox-id
crictl sandbox list

Add CLI for crictl exec [--sync].

Umbrella issues for benchmark tests.

• P0: Parallel Sandbox & containers operation
  • Candidate Operation List (benchmark one operation in one test case):
    • sandbox run #85
    • sandbox status
    • remove sandbox
    • stop sandbox
    • list sandboxes
    • create container #90
    • container status
    • start container
    • stop container
    • remove container
    • list container
  • Metrics:
    • Node:
      • CPU usage (%)
      • MEM usage (%)
    • Runtime daemon perf:
      • pod list latency (ms, 50%ile 90%ile 99%ile)
    • Core:
      • operation latency (ns, 50%ile 90%ile 99%ile)
  • Test Case Variable:
    • concurrency of operation
• p0: Image write operation (stick to specific image)
  • Candidate Operation List (benchmark one operation in one test case):
    • image pull
    • image remove
  • Metrics:
    • Node:
      • CPU usage (%)
      • MEM usage (%)
      • DISK IO (%iowait)
      • NETWORK IO (bandwidth usage: bytes/sec)
  • Test Case Variable:
    • repeat number of operation
• p0: Image read operation (stick to specific image)
  • Candidate Operation List (benchmark one operation in one test case):
    - [ ] image list
    - [ ] image status
  • Metrics:
    • Node:
      • CPU usage (%)
      • MEM usage (%)
    • Core:
      • operation latency (ns, 50%ile 90%ile 99%ile)
  • Test Case Variable:
    • repeat number of operation
• p0: Parallel image operation (stick to specific image)
  • Candidate Operation List (benchmark one operation in one test case):
    • image pull
    • image remove
    • image list
  • Metrics:
    • Node:
      • CPU usage (%)
      • MEM usage (%)
      • DISK IO (%iowait)
      • NETWORK IO (bandwidth usage: bytes/sec)
  • Test Case Variable:
    • concurrency of operation
• P2: Streaming (maybe next release)
  • Metrics:
    • Node:
      • CPU usage (%)
      • MEM usage (%)
    • Core:
      • every Op latency (ms, 50%ile 90%ile 99%ile)
  • Operation:
    • Op = exec/attach/portforward
  • WorkFlow:
    • start -> run sandbox -> start container -> loop(parallel_do(exec -> attach -> portforward)) -> finish
  • Variable:
    • loop count
    • routine number
    • duration of every parrallel_do operation

See more details about each test case.

cc/ @yujuhong @Random-Liu @mrunalp @xlgao-zju

Would someone like to add the performance test framework?

Or indicate the user that the we cann't output logs because the container don't set logpath.

root@ubuntu:/home/cloud/test# crictl logs 331
FATA[0000] Get log path of container failed
root@ubuntu:/home/cloud/test#
root@ubuntu:/home/cloud/test#
root@ubuntu:/home/cloud/test# crictl inspect 331
DEBU[0000] ContainerStatusRequest: &ContainerStatusRequest{ContainerId:331,}
DEBU[0000] ContainerStatusResponse: &ContainerStatusResponse{Status:&ContainerStatus{Id:3317b118e40044e2ea8981e08bcece969f3b6533c087daeef0aa3f1bb9d2570c,Metadata:&ContainerMetadata{Name:busybox2,Attempt:0,},State:CONTAINER_RUNNING,CreatedAt:1509674580490484539,StartedAt:1509674623756901309,FinishedAt:0,ExitCode:0,Image:&ImageSpec{Image:busybox,},ImageRef:docker.io/library/busybox@sha256:0b408ce12dfae8721f5640f16f90f6592cda42a5bbc0a4ee2b31ee18c1b8985a,Reason:,Message:,Labels:map[string]string{},Annotations:map[string]string{},Mounts:[],LogPath:,},}

We don't have created timestamp for image in CRI, so this will be a difference with docker images.

We may not need it, but file an issue here to track this.

Related #131
We should update CRI version for supporting mount propagation.
I tried to update the dependency but always failed.

Maybe I'm not familiar with godep, could anyone help to update this please? Thanks!

/cc @Random-Liu @feiskyer

In trying to get criod (https://github.com/kubernetes-incubator/cri-o) to pass the cri-tests, I am seeing an issue where criod expects there to be SecurityContext information in the config.

@Random-Liu Is SecurityContext required to be in the config or is it optional? If the latter, i can fix our code. If the former, I'm happy to provide a PR if you have some default options you want defined.

@runcom fyi

I think we can use pkg github.com/urfave/cli which is used in cri-o too.

The cli can be used when we build CRI-cli.

And how about making the e2e test as a subcommand in CRI-cli? Like cri test or something? Or we will build two cli, one for e2e test, the other for CRI-cli?

cc @feiskyer @Random-Liu @yujuhong

Add CLI for crictl attach.

Here is the list of functionality test, based on #4 (comment) and #4 (comment):

• P0 Basic (Conformance):
  • sandbox: run/stop/remove/list/status #8
  • container: create/start/execSync/remove/status #13
  • image: pull/list/status/remove #9
  • version #8
  • status #11
  • logs @Helen-Xie #28
  • networking: DNSOptions/PortMapping/IP @xlgao-zju #30
  • volumes #27
• P0 SecurityContext (only subset of this function are testable):
  • container with security context:
    • NamespaceOption
      • hostPID #56
      • hostIPC #82
      • hostNetwork #75
    • Priviledged #78
    • Capabilities #80
    • RunAsUser/RunAsUserName #41
    • ReadOnlyRootfs #53
• P1 Streaming (Conformance):
  • exec #58
  • attach #64
  • portfoward #71
• P2 Resource Limits
• P2 updateRuntimeConfig
• P2 Experimental: apparmor/sysctls/seccomp
• P2 Devices
• P2 SELinuxOption

See more details about each test.

The list may not be sufficient enough. Please feel free to:

• Point out anything is missing or incorrect.
• Claim any test you'd like to help.

/cc @feiskyer @resouer @xlgao-zju @Helen-Xie @yujuhong @dchen1107 @mrunalp @lucab

We need CRI validation test for new CRI features:

• NoNewPrivilege (May be hard to test) #141
• Mount propagation (easy to test) #142

We should at least add test for 2).

@kubernetes-incubator/maintainers-cri-tools

Add CLI for crictl container, e.g.

crictl container create sandbox-id config.json|yaml
crictl container start container-id
crictl container stop container-id
crictl container remove container-id
crictl container status container-id
crictl container list

critest for experimental features, including

• apparmor #140
• sysctls #139
• seccomp #137

We should test sysctl, seccomp and apparmor. These tests may not be conformance test, but are required in CRI validation test suite.

From @Random-Liu: Probably just migrate test from node e2e https://github.com/kubernetes/kubernetes/blob/master/test/e2e/node/security_context.go.

note: sysctl/apparmor tests are also not included in node e2e tests.

We should update CRI version after kubernetes/kubernetes#47818 is merged.

Add CLI of crictl status.

We should test sysctl, seccomp and apparmor. These tests may not be conformance test, but are required in CRI validation test suite.

Probably just migrate test from node e2e https://github.com/kubernetes/kubernetes/blob/master/test/e2e/node/security_context.go.

TODO lists:

• Bump version to v0.2 #146
• Release with v0.2 tag with release notes
• Update change log

Draft release notes:

cri-tools v0.2 enhances validation testings, improves crictl UX and also fixes several bugs. It has also updates container runtime interface (CRI) to kubernetes v1.8.

CRI validation testing (critest)

• [#127] Adds validation tests for supplemental groups
• [#135] [#137] and [#144] Adds validation tests for seccomp
• [#139] Adds validation tests for sysctls
• [#140] Adds validation tests for AppArmor
• [#141] Adds validation tests for NoNewPrivs
• [#142] Adds validation tests for mount propagation
• [#115] Fixes image validation tests
• [#116] Fixes validation message
• [#126] Fixes sandbox leak in port forward validation tests

CRI CLI (crictl)

• [#122] Adds support for authenticated image pull
• [#123] Improves crictl UX
• [#124] Adds support for creating sandboxes and containers from yaml
• [#133] Adds timeout support for container stop

Add CLI for crictl portforward.

We added the verbose option in CRI to make it possible for container runtime to provide more information for debug. kubernetes/kubernetes#53965

crictl should properly consume these information in inspect, inspecti, inspects and info.

Add support for CRI stats in cri-tools. Just query the current container/image stats.

This won't work for now, because there is no container runtime implementing this. However, we're going to implement this in cri-containerd soon.

Currently, ps, sandboxes and images result changes everytime.

panic: runtime error: index out of range

I can fix it.

Log is below.

root@ubuntu:/home/cloud/go/src/github.com/kubernetes-incubator/cri-containerd# crictl pull docker.io/library/busybox@sha256:be3c11fdba7cfe299214e46edc642e09514dbb9bbefcd0d3836c05a1e0cd0642
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:docker.io/library/busybox@sha256:be3c11fdba7cfe299214e46edc642e09514dbb9bbefcd0d3836c05a1e0cd0642,},Auth:nil,SandboxConfig:nil,} 
DEBU[0002] PullImageResponse: &PullImageResponse{ImageRef:sha256:c30178c5239f2937c21c261b0365efcda25be4921ccb95acd63beeeb78786f27,} 
Image is update to date for sha256:c30178c5239f2937c21c261b0365efcda25be4921ccb95acd63beeeb78786f27
root@ubuntu:/home/cloud/go/src/github.com/kubernetes-incubator/cri-containerd# crictl images
DEBU[0000] ListImagesRequest: &ListImagesRequest{Filter:&ImageFilter{Image:&ImageSpec{Image:,},},} 
DEBU[0000] ListImagesResponse: &ListImagesResponse{Images:[&Image{Id:sha256:c30178c5239f2937c21c261b0365efcda25be4921ccb95acd63beeeb78786f27,RepoTags:[],RepoDigests:[docker.io/library/busybox@sha256:be3c11fdba7cfe299214e46edc642e09514dbb9bbefcd0d3836c05a1e0cd0642],Size_:701277,Uid:nil,Username:,}],} 
panic: runtime error: index out of range

goroutine 1 [running]:
main.glob..func10(0xc4200e4780, 0x0, 0xc4200e4780)
	/home/cloud/go/src/github.com/kubernetes-incubator/cri-tools/cmd/crictl/image.go:135 +0x5b1
github.com/kubernetes-incubator/cri-tools/vendor/github.com/urfave/cli.HandleAction(0x1375cc0, 0x15c3870, 0xc4200e4780, 0xc42006f400, 0x0)
	/home/cloud/go/src/github.com/kubernetes-incubator/cri-tools/vendor/github.com/urfave/cli/app.go:485 +0xd4
github.com/kubernetes-incubator/cri-tools/vendor/github.com/urfave/cli.Command.Run(0x155c234, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x156176e, 0xb, 0x0, ...)
	/home/cloud/go/src/github.com/kubernetes-incubator/cri-tools/vendor/github.com/urfave/cli/command.go:207 +0xb72
github.com/kubernetes-incubator/cri-tools/vendor/github.com/urfave/cli.(*App).Run(0xc420087ba0, 0xc42000c1e0, 0x2, 0x2, 0x0, 0x0)
	/home/cloud/go/src/github.com/kubernetes-incubator/cri-tools/vendor/github.com/urfave/cli/app.go:250 +0x7d0
main.main()
	/home/cloud/go/src/github.com/kubernetes-incubator/cri-tools/cmd/crictl/main.go:195 +0xaaf

critest for streaming interfaces

• exec
• attach
• portforward

Moving SELinuxOptions test to P2.

e.g. to run a single test:

critest --ginkgo-flags=-focus='runtime should support stopping container'

but critest actually running 21 tests.

While evaluating cri validation tests, it still hard to write a simple dedicated test quickly. So I wrote a python client of kubelet CRI. It could help when we drafting codes for cri-tools and other runtimes.

The python client could be easily installed by pip.

pip install cri

A full example of creating a new busybox pod:

import grpc
from cri import api_pb2

channel = grpc.insecure_channel('unix:///var/run/dockershim.sock')
runtime_stub = api_pb2.RuntimeServiceStub(channel)
image_stub = api_pb2.ImageServiceStub(channel)


sandboxConfig = api_pb2.PodSandboxConfig(
    metadata=api_pb2.PodSandboxMetadata(name="sandbox", namespace="test"),
    dns_config=api_pb2.DNSConfig(servers=["3.3.3.3"], searches=["google.com"])
)

sandbox_resp = runtime_stub.RunPodSandbox(
    api_pb2.RunPodSandboxRequest(config=sandboxConfig))

print image_stub.PullImage(api_pb2.PullImageRequest(image=api_pb2.ImageSpec(image="busybox")))

containerConfig = api_pb2.ContainerConfig(
    metadata=api_pb2.ContainerMetadata(name="busybox"),
    image=api_pb2.ImageSpec(image="busybox"),
    command=["sh", "-c", "top"],
)

container_resp = runtime_stub.CreateContainer(api_pb2.CreateContainerRequest(
    pod_sandbox_id=sandbox_resp.pod_sandbox_id,
    config=containerConfig,
    sandbox_config=sandboxConfig,
))

print runtime_stub.StartContainer(api_pb2.StartContainerRequest(
    container_id=container_resp.container_id))

See https://github.com/feiskyer/python-cri for more details.

cc/ @xlgao-zju @yujuhong @resouer @kubernetes-incubator/maintainers-cri-tools

• NamespaceOption
  • hostPID
  • hostIPC
  • hostNetwork

Currently, we copy all the code from kubelet.

We should make the code in kubelet to be a CRI log parsing library. And import the library in crictl.

https://docs.google.com/document/d/1BbohEUHwwodCSVltWXs3nvCazZ7OJ4MwjA_SFBT8GAQ/edit?usp=sharing

Feel free to comment.

See https://travis-ci.org/kubernetes-incubator/cri-tools/jobs/226291949.

Even when there are no items.

crictl should automate load config file from default path no matter wether it is exist in command line args.
Now even though I has writed config file in /etc/crictl.yaml , I must enter "crictl --config /etc/crictl.yaml".

@Random-Liu

transport: dial unix /var/run/dockershim.sock: connect: no such file or directory

any suggest..?

Assigned to @Helen-Xie