onosproject / onos-test Goto Github PK

At this moment we have one big document which explains everything including how to run onit, how to test it, etc. Since onit its own repo now, it would be nice to organize it a little bit. We also need to go through the documents to make sure everything is up to date and correct after the changes that we made in the last few weeks.

To reproduce the issue:

onit create cluster
onit create network network-1
onit remove network network-1

and the output will be
onit remove network network-1

panic: runtime error: index out of range

goroutine 1 [running]:
github.com/onosproject/onos-test/pkg/onit/k8s.(*ClusterController).removeNetworkFromConfig(0xc00003d580, 0x7fff5fbffa74, 0x9, 0xc0001b6850, 0x0, 0xc00002d7a0)
	/Users/adibrastegarnia/go/src/github.com/onosproject/onos-test/pkg/onit/k8s/onos-topo.go:456 +0x348
github.com/onosproject/onos-test/pkg/onit/k8s.(*ClusterController).RemoveNetwork(0xc00003d580, 0x7fff5fbffa74, 0x9, 0x7fff5fbffa74, 0x9)
	/Users/adibrastegarnia/go/src/github.com/onosproject/onos-test/pkg/onit/k8s/cluster.go:511 +0x388
github.com/onosproject/onos-test/pkg/onit/cli.getRemoveNetworkCommand.func1(0xc000377400, 0xc0000e3e00, 0x1, 0x1)
	/Users/adibrastegarnia/go/src/github.com/onosproject/onos-test/pkg/onit/cli/remove.go:86 +0x181
github.com/spf13/cobra.(*Command).execute(0xc000377400, 0xc0000e3dd0, 0x1, 0x1, 0xc000377400, 0xc0000e3dd0)
	/Users/adibrastegarnia/go/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x2ae
github.com/spf13/cobra.(*Command).ExecuteC(0xc00010fb80, 0xc00010fb80, 0xc00035df88, 0x100777f)
	/Users/adibrastegarnia/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fc
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/adibrastegarnia/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
	/Users/adibrastegarnia/go/src/github.com/onosproject/onos-test/cmd/onit/main.go:31 +0x40

I'm running the following command to install onit:

go get github.com/onosproject/onos-test/cmd/onit

which results in the following output:

go: finding github.com/onosproject/onos-test latest
# github.com/onosproject/onos-test/test/env
../../../../pkg/mod/github.com/onosproject/[email protected]/test/env/env.go:191:42: undefined: "github.com/onosproject/onos-config/pkg/northbound/admin".ConfigAdminServiceClient
../../../../pkg/mod/github.com/onosproject/[email protected]/test/env/env.go:200:15: undefined: "github.com/onosproject/onos-config/pkg/northbound/admin".NewConfigAdminServiceClient

which indicates the install fails. Attempting to run the command shows this to be the case:

$ onit
zsh: command not found: onit

The message is slightly different for topo than config - with topo it mentions something about /bin/bash

scondon@Z420:~/go/src/github.com/onosproject/onos-test$ onit create cluster sean-1 --image-tags="config=latest,topo=latest"
 ✓ Creating cluster namespace
 ✓ Setting up RBAC
 ✓ Setting up Atomix controller 
 ✓ Starting Raft partitions 
 ✓ Adding secrets
 ✗ Bootstrapping onos-topo cluster          Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "b88786edb8a9d0599fe941217054b67312b7877a621706157d3bf2add63889e3": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "exec: \"/bin/bash\": stat /bin/bash: no such file or directory": unknown
Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "b88786edb8a9d0599fe941217054b67312b7877a621706157d3bf2add63889e3": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "exec: \"/bin/bash\": stat /bin/bash: no such file or directory": unknown
scondon@Z420:~/go/src/github.com/onosproject/onos-test$

When I leave out topo and use only config, I get:

scondon@Z420:~/go/src/github.com/onosproject/onos-test$ onit create cluster sean-1 --image-tags="config=latest"
 ✓ Creating cluster namespace
 ✓ Setting up RBAC 
 ✓ Setting up Atomix controller 
 ✓ Starting Raft partitions 
 ✓ Adding secrets
 ✓ Bootstrapping onos-topo cluster 
 ✗ Bootstrapping onos-config cluster        Internal error occurred: error executing command in container: failed to exec in container: failed to create exec "0bab0cfe7276d27e54d3d2bdeeb4e479742e1834c37694cf7d60c31e1409f5a1": cannot exec in a stopped state: unknown
Internal error occurred: error executing command in container: failed to exec in container: failed to create exec "0bab0cfe7276d27e54d3d2bdeeb4e479742e1834c37694cf7d60c31e1409f5a1": cannot exec in a stopped state: unknown
scondon@Z420:~/go/src/github.com/onosproject/onos-test$

To use onit in google cloud, we need to add the following import to the onit and all of test and benchamrk images

_ "k8s.io/client-go/plugin/pkg/client/auth/gcp"

I opened an issue that we address it sometime soon.

As a test engineer I would like the ability to delete a device via onit so that I can easily tear down a device with one command and not rely on issuing multiple k8s commands.

Only options at present are to delete the onit installation via onit delete cluster or to delete via k8s which means you first need to delete via onos topo remove device <device id> and then kubectl delete pod <pod id>, kubectl delete configmap <config id> etc

We need to update readme to add build, godoc, license badges.

This issue is opened to setup travis for onos-test repo.

Is your feature request related to a problem? Please describe.
When we make a change in an integration test, we need to load the image into the kind cluster, shut down the onit cluster, and run it again. That would be nice if we avoid shutting down the onit cluster.

Describe the solution you'd like
We should investigate the possibility of redeploying the onos-config pod when the onit cluster is running.

Like other subsystems (config, topo, gui), we should add onos-cli to the deployments.

Is your feature request related to a problem? Please describe.
Currently Testing are done based on the simulator.

Describe the solution you'd like
We would need the capability to add a Stratum device to onos-config cluster and run tests with one or more paths supported by stratum,

Is your feature request related to a problem? Please describe.
onit command help does not currently describe sufficiently the features of onit and their uses. Users should be able to learn the commands through the command help.

Describe the solution you'd like
Add thorough documentation with examples to all onit commands.

Try something like this. It will hang:

onit test --image onosproject/onos-ric-tests:latest --test no-such-test

Benchmarks often require that pods be configured to use a fixed amount of memory or CPUs. Onit commands should support resource requirement overrides for core subsystems with --set.

To reproduce the bug:

onit create cluster
onit get clusters // returns the list of all clusters
onit get cluster // returns the current cluster
onit delete cluster
onit get clusters // it is empty
onit get cluster // it still returns the current cluster

@kuujo @SeanCondon
If we would like to support ingress for local cluster (i.e. to avoid using port-forwarding), we can use k3d. But to do that, minor changes is needed to support traefik ingress controller as well as nginx when we create an ingress for gui and onos subsystems.

Onit now supports comma separated lists of suites or tests to run. But Cobra supports a string slice flag that can be more elegant. The CLI should use the string slice flag for suites and tests.

After GolangCI-Lint is used in onos-config successfully, we will make the changes in this repo as well.

This issue is opened to kick off moving onit code from onos-config to onos-test.
1- Move code and organize the code based on the go project layout.
2- Add a makefile and build the project.

Some bugs can be difficult to reproduce. In ONOS, the STW command was frequently useful for reproducing race conditions and other unpredictable bugs. Onit should support flags to run a fixed number of iterations or run until failure:

• --iterations/-n - run each suite a set number of times
• --untul-failure - run all suites until any suite fails

Since the onit help is going to be extended to included examples (#5, #25), it would be nice to split the cli.go into multiple files and each file include a command and its subcommands.

One of the useful features of stc was that it could detect when exceptions occurred inside a node even if the node didn’t crash. Onit should be able to detect exceptional events like these to point developers towards relevant errors. In stc this was done by simply using a regex to find exceptions in the log. But there may be a more elegant solution for detecting errors in Golang/k8s.

Needs investigation.

Currently, test namespaces are torn down immediately following test completion. When this happens, logs are lost and cannot be recovered. Onit should persist application logs inside the kube-test namespace as part of the recorded history of test runs. Additionally, onit should support commands for viewing/formatting/downloading logs for past test runs persisted in the kube-test namespace.

Is your feature request related to a problem? Please describe.
Currently, integration tests run on whatever infrastructure was provided by the user. This can be useful for manual testing, but to support reproducibility and running multiple tests with different network configurations, tests should also be able to prescribe the network.

Describe the solution you'd like
It’s not totally obvious what the solution is here yet. Tests could setup devices in an API, but that would mean every test always sets up its devices and could add significant time to running all tests. It may be more sensible for test suites to prescribe the network configuration, and for that configuration to be installed when no network has been created by the user.

logs for this bug

./onit onos-cli   
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/onosproject/onos-test/pkg/onit/cli.getOnosCliCommand.func1(0xc0003f1180, 0x6869388, 0x0, 0x0)
        /Users/yeya24/hub/gowork/onos-test/pkg/onit/cli/ssh.go:47 +0x2a1
github.com/spf13/cobra.(*Command).execute(0xc0003f1180, 0x6869388, 0x0, 0x0, 0xc0003f1180, 0x6869388)
        /Users/yeya24/go/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x2ae
github.com/spf13/cobra.(*Command).ExecuteC(0xc0003de000, 0xc0003de000, 0xc000127f88, 0x400769f)
        /Users/yeya24/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fc
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/yeya24/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
        /Users/yeya24/hub/gowork/onos-test/cmd/onit/main.go:33 +0x40

How to reproduce this bug:

1. ./onit create cluster test
2. Then use CTRL+D to stop the process
3. Use ./onit onos-cli

Actually you can reproduce this bug by manually delete the onos-cli pod in your current ns as well.

Since I stop the cluster creating process, the onos-cli pod is not created. Then the onos-cli cmd get panic. We should ensure this pod is in that namespace, otherwise return an err.

Test history is persisted in the kube-test namespace until deleted by an administrator. This includes test logs, exit codes, etc. Onit should provide commands to query the history of test runs and results and view test logs.

Integration tests required 2 simulators to be present in the cluster. They are usually created with

onit add simulator
onit add simulator

Simulators are usually named like device-2727455432 in this case

However adding with a given name like the following seems to cause unexplained failures in the tests

onit add simulator devicesim-1
onit add simulator devicesim-2

This issue was raised to keep track of it - it might be related to some hardcoded expectation of a name length or pattern. Needs investigation

To use the images from the private docker registry, we need to add an option (--docker-registery) to prepend address of the private docker registry to the image tag like the following to use that image instead for creating the onos-* deployments:

10.128.100.91:30500 + /onosproject/onos-config:debug = 10.128.100.91:30500/onosproject/onos-config:debug

It just requires an extra
"-modelPlugin=/usr/local/lib/stratum-debug.so.1.0.0"
in github.com/onosproject/onos-test/pkg/onit/onos-config.go

A common requirement when testing and debugging is connecting to nodes via SSH to debug. Because onit runs on Kubernetes, this can require a non-trivial command. But users should be abstracted from the details of Kubernetes to the extent possible, so onit should provide a command for connecting to nodes that does not require a custom kubectl exec command.

We need to provide a short description for each package (as a comment on top of package name) that they can appear in godcos.

After the latest onos-topo and onos-config have been deployed, this is what I get when adding a simulator

uonos@uonos-S210-X12RS-V2:~$ onit add simulator 
 ✓ Setting up simulator 
⢀⡱ Reconfiguring onos-config nodes Unable to parse device info from id: 'device-3492326417', address: 'device-3492326417:10161' version: '1.0.0' devicetype: 'Devicesim': line 1.77: unknown field name "devicetype" in admin.DeviceInfo
 ✗ Reconfiguring onos-config nodes          command terminated with exit code 1
command terminated with exit code 1
uonos@uonos-S210-X12RS-V2:~$

The device pod is created, just not configured in onos-config/onos-topo

uonos@uonos-S210-X12RS-V2:~$ kubectl get pods -n onit-1
NAME                                 READY   STATUS    RESTARTS   AGE
atomix-controller-6bfdfbf4bc-kpch9   1/1     Running   0          47m
device-3492326417                    1/1     Running   0          30m
onos-config-66f985864f-n867t         1/1     Running   0          46m
onos-config-envoy-799df9dcfb-2626c   1/1     Running   0          46m
onos-gui-6c78895d94-r8rwz            1/1     Running   0          46m
onos-topo-686c57874b-ssscj           1/1     Running   0          47m
onos-topo-envoy-764898476b-khfrs     1/1     Running   0          47m
raft-1-0                             1/1     Running   0          47m

After we merge #113, we need to document the API that we provide for dictating the network architecture in tests.

Documentation for the test API is lagging behind the master branch, and documentation for the benchmark API is largely non-existent. To promote development of new tests, documentation needs to be significantly improved for onit.

I already added this flag to disable tls but it looks like it is missing.

In order to use a shared Docker registry in development with onit, users have to be able to deploy images from the registry that do not override images from other developers working in parallel. To do so, users should be able to push images with custom tags - e.g. onosproject/onos-config:jordan - and deploy those images in onit. ONIT can support an --image-tag that overrides the default latest tag to deploy images with a development tag.

To ease the inspection of onos-config nodes, it would be convenient to add support for executing an onos command against any onos-config node running in onit.

onit add app command need to be enhanced to support auto completion.

To improve the development/testing cycle, it would be nice for onit to support local, in-process “clusters.” This is now feasible since Atomix now has a Golang implementation of its gRPC API.

To run a local cluster, we should add an onit create local command. I would suggest we make the ClusterController an interface, move the k8s implementation into a k8s package, and add a local package. In the local implementation, we need to:
• Start a local Atomix server
• Start the topo manager
• Start the config manager

This may require some refactoring of the topo/config mains to allow them to be called programmatically, and in atomix-go-node we’ll probably want to make the local Protocol currently used in tests public.

Describe the bug
The onit debug command only supports forwarding a single port to an onos-config node.

To Reproduce
Steps to reproduce the behavior:

1. Run onit create cluster --nodes 2 to create a 2 node cluster
2. Run onit debug for each node in the cluster
3. See error

Expected behavior
onit should not specify the local port to forward to the node being debugged. Instead, we should forward to :40000 and allow the k8s client to select the local port. We may even be able to forward multiple local ports to each pod in a multi-node cluster with the onit debug command when no node ID is provided.

Reminder - when running onit without a suite identifier, it either crashes or hangs.

Run:
onit get store-presets

The output:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4f93a12]

goroutine 1 [running]:
github.com/onosproject/onos-test/pkg/onit/cli.getStorePresets.func1(0xc0000dae40, 0x55, 0x0, 0x0, 0x54eca40, 0xc0000bd080, 0x10, 0x5167da0)
	/Users/adibrastegarnia/go/src/github.com/onosproject/onos-test/pkg/onit/cli/config.go:56 +0x32
path/filepath.Walk(0xc0000dae40, 0x55, 0xc0000ba7d0, 0xc0001c3d68, 0x41c8bb5)
	/usr/local/opt/go/libexec/src/path/filepath/path.go:402 +0x6a
github.com/onosproject/onos-test/pkg/onit/cli.getStorePresets(0xc0001c3d68, 0xc0001c3d68, 0x4156bf4)
	/Users/adibrastegarnia/go/src/github.com/onosproject/onos-test/pkg/onit/cli/config.go:55 +0xbe
github.com/onosproject/onos-test/pkg/onit/cli.getGetStorePresetsCommand.func1(0xc00033bb80, 0x63857a8, 0x0, 0x0)
	/Users/adibrastegarnia/go/src/github.com/onosproject/onos-test/pkg/onit/cli/get.go:222 +0x26
github.com/spf13/cobra.(*Command).execute(0xc00033bb80, 0x63857a8, 0x0, 0x0, 0xc00033bb80, 0x63857a8)
	/Users/adibrastegarnia/go/src/github.com/spf13/cobra/command.go:830 +0x2ae
github.com/spf13/cobra.(*Command).ExecuteC(0xc000332000, 0xc000332000, 0xc0001c3f88, 0x4007dcf)
	/Users/adibrastegarnia/go/src/github.com/spf13/cobra/command.go:914 +0x2fc
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/adibrastegarnia/go/src/github.com/spf13/cobra/command.go:864
main.main()
	/Users/adibrastegarnia/go/src/github.com/onosproject/onos-test/cmd/onit/main.go:29 +0x40

If we run onit test command with --verbose flag, then it does not enable verbose output.

Currently, there are separate flags for controlling number of onos-config and onos-topo nodes. It would be better to define a map (like imageTags) that we can control replicas for each of the susbsytems.

When I run onit run test foo, onit creates a test job and only fails inside the test. The onit run x command should fail and exit immediately when an invalid test is provided.

@kuujo

It looks like when we create a cluster in the new version of onit, we cannot give a name to onit create cluster command (the default name which is given to the cluster is onos). Given that, we cannot create the second cluster because of this bug. If we run the command for the second time:

onit create cluster
‣ 2019-11-19T16:17:07-05:00 onos Setup ONOS cluster
‣ 2019-11-19T16:17:07-05:00 onos Setup namespace
✓ 2019-11-19T16:17:07-05:00 onos Setup namespace
‣ 2019-11-19T16:17:07-05:00 onos Set up RBAC
✓ 2019-11-19T16:17:07-05:00 onos Set up RBAC
✓ 2019-11-19T16:17:07-05:00 onos Setup ONOS cluster
‣ 2019-11-19T16:17:07-05:00 onos Setup Atomix controller
✗ 2019-11-19T16:17:07-05:00 onos Setup Atomix controller
Error: deployments.apps "atomix-controller" already exists
deployments.apps "atomix-controller" already exists

https://github.com/onosproject/onos-test/blob/master/pkg/onit/k8s/onos-cli.go#L104

this method concatenates everything in a string for the it to be uses as a value in the atomix.yaml key. I'd prefer having these as separate k/v pairs in the configMap of K8s.

I ran
onit add simulator devicesim1
and a few minutes later I ran
onit add simulator devicesim2

but when I run
onos config get configs
I only see the first one

~ $ onos config get configs
devicesim1-1.0.0	(devicesim1)	1.0.0	Devicesim	2019-08-15T09:11:05Z
	wupuOO6KMFdE8Tq1elu/qmQ29a0=
~ $ 

I tried this with automatic name assignment and it did not seem to give the same problem

Is your feature request related to a problem? Please describe.
Once stores are distributed, integration tests will require a mechanism for testing fault tolerance. This requires that tests be able to execute k8s commands internally to e.g. delete pods, modify services, etc.

Describe the solution you'd like
Integration tests should provide an API for listing nodes, killing/restarting nodes, reconfiguring nodes, etc for HA and upgrade tests.

Additional context
This requires that special permissions be provided to integration test pods and the k8s Go client be used to access cluster information.