mattfenwick / cyclonus Goto Github PK

Add traffic cases where one side (either source or destination) is external to the cluster, to the synthetic probes.

https://github.com/actions/virtual-environments

See:

• https://github.com/kubernetes/kubernetes/pull/98639/files
• https://github.com/kubernetes/kubernetes/pull/98636/files

• egress: can ping some common ip
• add external traffic to simulated probe
• how to simulate ingress?

see https://github.com/kubernetes/enhancements/pull/2522/files for details (search for "yaml")

https://github.com/ovn-org/ovn-kubernetes/blob/master/docs/kind.md

https://github.com/ovn-org/ovn-kubernetes/blob/master/contrib/kind.sh

see https://github.com/docker/build-push-action

antrea-io/antrea#1765 (comment)

containing cyclonus binary

See:

• https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/2079-network-policy-port-range
• projectcalico/libcalico-go#1357
• kubernetes/kubernetes#97058

We're keen to integrate cyclonus into Cilium's CI (cilium/cilium#14722).

As a stretch goal, would it be possible to extend cyclonus to support CiliumNetworkPolicys? These include features beyond standard Kubernetes NetworkPolicys.

Including:

underlying OS:

• linux
• windows

kube version:

• 1.21
• 1.20
• 1.19

IP stack:

• IPV4
• dual-stack
• IPV6

CNI:

• Antrea
• Calico
• Cilium
• ovn-kubernetes
• flannel
• weave

Cluster:

• kind
• k3s
• GKE
• AKS
• EKS
• bare-metal

What else?

https://bykeshed.slack.com/archives/C01R5417TG8/p1615613820095400

The following policy is incorrectly indented, so fails to create in kubernetes.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-all-ingress-egress-by-label
  namespace: "y"
spec:
  policyTypes:
    - Egress
    - Ingress
  podSelector:
    matchExpressions:
      - key: pod
        operator: In
        values: [a, b, c]
  egress:
    - to:
      - podSelector:
        matchLabels:
          use: db
  ingress:
    - from:
      - ipBlock:
        cidr: 172.17.0.0/16
        except:
        - 172.17.1.0/24
      - namespaceSelector:
        matchLabels:
          project: myproject
      - podSelector:
        matchLabels:
          role: frontend
      ports:
        - protocol: TCP
          port: 6379

$ kubectl create -f policy.yaml
error: error validating "policy.yaml": error validating data: [ValidationError(NetworkPolicy.spec.egress[0].to[0]): unknown field "matchLabels" in io.k8s.api.networking.v1.NetworkPolicyPeer, ValidationError(NetworkPolicy.spec.ingress[0].from[0]): unknown field "cidr" in io.k8s.api.networking.v1.NetworkPolicyPeer, ValidationError(NetworkPolicy.spec.ingress[0].from[0]): unknown field "except" in io.k8s.api.networking.v1.NetworkPolicyPeer, ValidationError(NetworkPolicy.spec.ingress[0].from[1]): unknown field "matchLabels" in io.k8s.api.networking.v1.NetworkPolicyPeer, ValidationError(NetworkPolicy.spec.ingress[0].from[2]): unknown field "matchLabels" in io.k8s.api.networking.v1.NetworkPolicyPeer]; if you choose to ignore these errors, turn validation off with --validate=false

However, cyclonus doesn't notice these problems.

Cyclonus should notice these problems and report them.

Cyclonus assumes that the kubeconfig is at ~/.kube/config and does not recognize the KUBECONFIG env variable.

• service account + privileges to
  • CRUD network policies
  • CRUD namespaces
  • CRUD pods
  • CRUD services
  • execute remote commands
• k8s client instantation

Goal: make it easier, faster, and more useful for a user to understand cyclonus output.

Ideas:

• color
• formatting
• schema
• extra information

And tag appropriately!

See https://github.com/mattfenwick/cyclonus/tree/master/pkg/generator

Since this requires a kube cluster, it means the command will fail if you don't have one.

Instead, default to an empty slice of namespaces.

Procedure:

• spin up cluster with a CNI
• run cyclonus job
  • check out https://github.com/mattfenwick/cyclonus/tree/master/jobs -- notice the different arguments for each CNI, this is because of their differences in support for some features
• capture:
  • cyclonus invocation
  • cyclonus job logs
  • all cluster info (version, ip version, etc.)
  • all CNI info (config, version, etc.)

IPV4 Progress table:

KinD:

Azure:

GKE:

EKS:

Partial data:

• weave on KinD (pending verification from @dougsland @rikatz)

Network policies not supported, so don't need data:

• Flannel

IPV6 is not yet supported by Cyclonus

Bugs reported

• Calico bug: projectcalico/libcalico-go#1370
• Antrea bug: antrea-io/antrea#1764
• Cilium bug: cilium/cilium#14720
• Antrea CI antrea-io/antrea#1765
• Cilium CI cilium/cilium#14889
• ovn missing named port support: ovn-org/ovn-kubernetes#2117

Cyclonus currently assumes that pods are all in the same /24 subnet; if this assumption is violated, spurious failures will be reported.

Cyclonus should not make this assumption.

Example:

• good: 1.2.3.0/24
• bad: 1.2.3.4/24

See:

• antrea-io/antrea#1764

see: #74 (comment)

cyclonus currently has a fuzz command that is described as:

Generate network policies, install the policies one at a time in kubernetes, and compare actual measured connectivity to expected connectivity using a truth table.

Fuzzing is described by Wikipedia as:

Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program.

Since generate is a more accurate description of what the fuzz command currently does, would it make sense to rename the command from fuzz to generate?

see https://github.com/K8sbykeshed/k8s-local-dev

Check out the destination-type flag under the generate command:

cyclonus generate --destination-type=pod-ip

So that the user can get a pod-centric view of which network policies apply to each pod, and what those policies are doing.

follow up to #53

see: https://github.com/mattfenwick/cyclonus/runs/2371412592?check_suite_focus=true

TCP requests were nearly always timing, while analagous UDP and SCTP requests were not.

Cilium currently does not pass all of cyclonus's tests (cilium/cilium#14678), and as cyclonus's test suite expands Cilium might fail other tests due to missing features in Cilium's NetworkPolicy implementation.

So that cyclonus can be used in Cilium's CI (which requires all tests to pass before a PR can be merged), it would be nice if individual tests could be marked as "expected to fail". For these tests, cyclonus should still run them, but report success if they fail and failure if they succeed. This allows CI to pass for known failures, and will alert developers to update the cyclonus test configuration if/when Cilium fixes the bug.

Marking individual tests as "expected to fail" could either be done through a CLI option or a configuration file, depending on the number of tests expected to fail.

ovn-org/ovn-kubernetes#1782 (comment)

From yaml -- to verify that policies are indented correctly, parsed as intended, etc.

-A: default to false
can't have both set

This happens on newer kube versions, see: kubernetes/kubernetes#96968

Logs:
Untitled.txt

time="2021-04-09T09:52:57Z" level=fatal msg="for namespace y, expected labels map[ns:y] (found map[kubernetes.io/metadata.name:y ns:y])\n
github.com/mattfenwick/cyclonus/pkg/connectivity.(*TestCaseState).verifyClusterStateHelper
  /Users/mfenwick/go/src/github.com/mattfenwick/cyclonus/pkg/connectivity/testcasestate.go:238
github.com/mattfenwick/cyclonus/pkg/connectivity.(*TestCaseState).VerifyClusterState
  /Users/mfenwick/go/src/github.com/mattfenwick/cyclonus/pkg/connectivity/testcasestate.go:286
github.com/mattfenwick/cyclonus/pkg/connectivity.(*Interpreter).ExecuteTestCase
  /Users/mfenwick/go/src/github.com/mattfenwick/cyclonus/pkg/connectivity/interpreter.go:85
github.com/mattfenwick/cyclonus/pkg/cli.RunGenerateCommand\n\t/Users/mfenwick/go/src/github.com/mattfenwick/cyclonus/pkg/cli/generate.go:135\ngithub.com/mattfenwick/cyclonus/pkg/cli.SetupGenerateCommand.func1\n\t/Users/mfenwick/go/src/github.com/mattfenwick/cyclonus/pkg/cli/generate.go:44\ngithub.com/spf13/cobra.
(*Command).execute\n\t/Users/mfenwick/go/pkg/mod/github.com/spf13/[email protected]/command.go:846\ngithub.com/spf13/cobra.
(*Command).ExecuteC\n\t/Users/mfenwick/go/pkg/mod/github.com/spf13/[email protected]/command.go:950\ngithub.com/spf13/cobra.
(*Command).Execute\n\t/Users/mfenwick/go/pkg/mod/github.com/spf13/[email protected]/command.go:887\ngithub.com/mattfenwick/cyclonus/pkg/cli.RunRootCommand\n\t/Users/mfenwick/go/src/github.com/mattfenwick/cyclonus/pkg/cli/root.go:13\nmain.main\n\t/Users/mfenwick/go/src/github.com/mattfenwick/cyclonus/cmd/cyclonus/main.go:8\nruntime.main\n\t/usr/local/Cellar/go/1.15.5/libexec/src/runtime/proc.go:204\nruntime.goexit\n\t/usr/local/Cellar/go/1.15.5/libexec/src/runtime/asm_amd64.s:1374"

TODOs:

• get a cluster of 1.21 to repro this against
• come up with some code that works on both <= 1.20 and >= 1.21
• (maybe) just ignore extra labels
• (maybe) detect cluster version and do the right thing automatically
• (maybe) provide a CLI switch to turn on/off the labels

To help a user generate the right job yaml for their cluster (protocols, loopback, wait, include/exclude, etc.)

Connectivity matrices with multiple ports and protocols expand vertically and can quickly take more than a single screen of space.

It's really useful to see the whole thing at once!

track https://github.com/GoogleCloudPlatform/gke-fqdnnetworkpolicies-golang

We currently only run UDP and TCP tests on antrea/KinD.

Add flag to delete them, or delete them automatically (and add a flag to NOT delete them).

See:

• kubernetes/kubernetes#96968

• add scripts for setting up KinD clusters on IPV6 for major CNIs
  • antrea
  • calico
  • cilium: see example
  • ovn
• #112 handle traffic matching in IP matcher
• join ports to ips correctly
• #108 speed up TCP/service/IPV6 calls
• #110 support IPV6 in IPBlock test cases