gardener / test-infra Goto Github PK

Test machinery for orchestration of integration/e2e/smoke style tests

License: Apache License 2.0

Shell 1.60% Python 1.53% Dockerfile 0.33% Makefile 0.62% Go 92.84% CSS 0.28% HTML 2.05% Smarty 0.41% Mustache 0.35%

test-infra's Introduction

Test Machinery

The Test Machinery is the test infrastructure for the gardener project. Gardener uses the Test Machinery to automatically qualify releases and run every sort of integration test.
These tests include periodically executed shoot integration tests and Kubernetes e2e tests (e2e runner) as well as Gardener Lifecycle tests (validate dev versions of gardener).

Gardener offers managed Kubernetes clusters in different flavors including cloud providers (Alicloud, AWS, GCP, Azure, Openstack), K8s versions, Operating Systems and more. Therefore, the Test Machinery is designed to test and validate this huge variety of clusters in a scalable and efficient way.

The executed tests are uploaded to the Kubernetes testgrid:

Read detailed docs here.

See here how new tests can be easily added.

Additional Tools in this Repository

Testrunner
Testrunner is an additional component of the Test Machinery that abstracts templating, deploying and watching of Testruns and provide additional functionality like storing test results or notifying test-owners.
Kubernetes e2e Testrunner
Executes the Kubernetes e2e/Conformance tests and uploads them for further analysis to testgrid or elasticsearch.
Kubernetes CNCF Pull Request Creator
Prepares a pull request for CNCF kubernetes certification
Host Scheduler
The hostscheduler selects available cluster from specific providers and locks the selected cluster so that a fresh gardener can be installed. When the cluster is not needed anymore, the host scheduler cleans and releases the cluster.
Shoot Telemetry
A telemetry controller to get granular insights of Shoot apiserver and etcd availability.
Tests
Testruns that are rendered by the testrunner or github app.
TM GitHub Bot
A GitHub bot to run tests on PullRequests.

test-infra's People

Stargazers

Watchers

test-infra's Issues

could not construct Shoot client: unsupported kubernetes version \"v1.19.4\"

Checking the logs from a conformance-testgrid-parallel step execution, I am able to find events from the Seed but I think the Shoot related events are not being fetched because of could not construct Shoot client: unsupported kubernetes version \"v1.19.4\":

time="2020-11-16T18:04:33Z" level=info msg="[SEED foo] [NAMESPACE shoot--it--tmau7-mxu] [EXTENSIONS] [OS]"
2020-11-16 20:04:33	
time="2020-11-16T18:04:33Z" level=info msg="[SEED foo] [NAMESPACE shoot--it--tmau7-mxu] [EXTENSIONS] [CONTROLPLANE]"
2020-11-16 20:04:33	
time="2020-11-16T18:04:33Z" level=info msg="[SEED foo] [NAMESPACE shoot--it--tmau7-mxu] [EXTENSIONS] [INFRASTRUCTURE]"
2020-11-16 20:04:33	
time="2020-11-16T18:04:33Z" level=info msg="[SEED foo]"
2020-11-16 20:04:33	
time="2020-11-16T18:04:33Z" level=error msg="retry failed with context deadline exceeded, last error: could not construct Shoot client: unsupported kubernetes version \"v1.19.4\""
2020-11-16 19:59:33	
time="2020-11-16T17:59:33Z" level=error msg="<nil>"
2020-11-16 19:59:33	
time="2020-11-16T17:59:33Z" level=info msg="END: dump kubetest stdout last 5000 bytes"
2020-11-16 19:59:33	
time="2020-11-16T17:59:33Z" level=info msg=" Test Suite Failed"
2020-11-16 19:59:33	
time="2020-11-16T17:59:33Z" level=info msg=" Ginkgo ran 1 suite in 1m40.92987864s"
time="2020-11-16T17:59:33Z" level=info msg=" "
2020-11-16 19:59:33	
time="2020-11-16T17:59:33Z" level=info msg=" FAIL! -- 4 Passed | 1 Failed | 0 Pending | 5229 Skipped"
2020-11-16 19:59:33	
time="2020-11-16T17:59:33Z" level=info msg=" Ran 5 of 5234 Specs in 89.982 seconds"

I believe the github.com/gardener/gardener dependency needs to be updated to include gardener/gardener@8076764.

Support more machine image version patterns in flavors

In flavors we can today specify either a fixed machine image version or latest.
For certain tests we want to test all non-deprecated versions of a certain machine image.
It would be helpful if the flavors would support such a version specification similar to how it was done for the k8sVersions (were we support latest and semver).

Example:

      machine:
        type: n1-standard-2
        image:
          name: gardenlinux
          version: "*"
          filterDeprecatedVersions: true

Use Telemetry Controller Under Test

What would you like to be added:
@dkistner has implemented a "telemetry controller" that keeps track of the control plane availability. It would make sense to have it observing the state of clusters under reconciliation/maintenance/test and report this metric to alert about poor shoot cluster control plane availability and eventually break the release/transport if KPIs are not met. Or shall this be part of the specific Gardener tests instead?

Why is this needed:
We sometimes miss issues here and lack repeating test results of this most important metric (it is the only metric relevant in our SLO).

Testrun was unable to run if a step finds no testrun

What happened:

Testmachinery silently errors in the workflow with the following error message
message: 'invalid spec: templates.testrun.tasks.delete-shoot-delete-testflow templates.delete-shoot-delete-testflow inputs.artifacts.repo was not supplied'

The problem with the testrun is that the step before delete-shoot does not match a test-def.

steps:
- name: create
    dependsOn: [ generate-provider ]
    definition:
      name: create-shoot
  - name: tests
    dependsOn: [ create ]
    definition:
      label: no-test
      continueOnError: true
  - name: delete
    dependsOn: [ tests ]
    definition:
      name: delete-shoot

What you expected to happen:

test should ignore the step if no tests can be found and only execute the found ones.
We may want to define min required 1 on labels.

Refactor Alerting

We need a proper alerting for failing testmachinery test steps.

Required information:

which test steps have failing
links to kibana dashboard, grafana logs, argo workflow, concourse step
mention of corresponding people in the alert (owner and change commiters)
environment context (provider, gardener version, k8s version, landscape)

The alert should be preferably be implemented as a Slack Alert. Alternative would be alerting via email.

Add support for global artifacts.

Issue

As a user I want to use global artifacts from serial steps as input source.

Description

Currently serial steps write their output to the global artifact registry.
To consume this kind of artifacts a special argo reference is needed.
Therefore another parameter (like globalInput) needs to be introduced that specifies the usage of a global artifact.

This is also needed to share files between the testflow and the onExit flow.

testflow:
- name: demo-step
  definition:
      name: demo
- name: label-step
  globalInput: true
  dependsOn: [ demo-step ]
  definition:
      label: default

Workflow failed with duplicate input parameter

Description

Workflow created by the TestMachinery failed with the error:

'invalid spec: templates.create-shoot-create-testflow.inputs.parameters[8].name
    ''env-r5h'' is not unique'

This issue can occur if the random generated parameter names contain duplicates.

fatal error: concurrent map read and map write

Find below the logs

Expand me

fatal error: concurrent map read and map write

goroutine 617 [running]:
runtime.throw(0x1e07ae8, 0x21)
	/usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc000a41b38 sp=0xc000a41b08 pc=0x42e5c2
runtime.mapaccess2_faststr(0x1ab3160, 0xc0007a8690, 0xc0065cf640, 0x18, 0x3, 0xc0065cf640)
	/usr/local/go/src/runtime/map_faststr.go:116 +0x48f fp=0xc000a41ba8 sp=0xc000a41b38 pc=0x412c6f
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00248de50, 0x7, 0xc00248de40, 0x10, 0xc002a40a50, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:63 +0x160 fp=0xc000a41d10 sp=0xc000a41ba8 pc=0x1690b00
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc0004dbb00, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420 fp=0xc000a41ea8 sp=0xc000a41d10 pc=0x1693c40
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff fp=0xc000a41f68 sp=0xc000a41ea8 pc=0x169587f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000a41f70 sp=0xc000a41f68 pc=0x45b9a1
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc00105cd78)
	/usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc00105cd70)
	/usr/local/go/src/sync/waitgroup.go:130 +0x64
github.com/gardener/test-infra/pkg/testrunner.RunList.Run(0xc00065a000, 0x2e, 0x30, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb, 0xc00013c1c0, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:103 +0x330
github.com/gardener/test-infra/pkg/testrunner.ExecuteTestruns(0x212a420, 0xc00105b9e0, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0x7ffdf61e7a5e, 0xb, 0xc00013c1c0, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/testrunner.go:40 +0x266
github.com/gardener/test-infra/cmd/testrunner/cmd/run_template.glob..func1(0x30e0ca0, 0xc000239e40, 0x0, 0x16)
	/go/src/github.com/gardener/test-infra/cmd/testrunner/cmd/run_template/run_template.go:117 +0x59c
github.com/spf13/cobra.(*Command).execute(0x30e0ca0, 0xc000239ce0, 0x16, 0x16, 0x30e0ca0, 0xc000239ce0)
	/go/src/github.com/gardener/test-infra/vendor/github.com/spf13/cobra/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0x30dfb20, 0x30e11a0, 0xc000793f20, 0x43c30a)
	/go/src/github.com/gardener/test-infra/vendor/github.com/spf13/cobra/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/gardener/test-infra/vendor/github.com/spf13/cobra/command.go:864
github.com/gardener/test-infra/cmd/testrunner/cmd.Execute()
	/go/src/github.com/gardener/test-infra/cmd/testrunner/cmd/cmd.go:50 +0x31
main.main()
	/go/src/github.com/gardener/test-infra/cmd/testrunner/main.go:22 +0x20

goroutine 6 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x30f43e0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/klog/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/klog/klog.go:411 +0xd6

goroutine 68 [syscall]:
os/signal.signal_recv(0x45b9a6)
	/usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 69 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0006c7c20)
	/go/src/github.com/gardener/test-infra/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
	/go/src/github.com/gardener/test-infra/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57

goroutine 639 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0025518c9, 0x7, 0xc0025518b0, 0x10, 0xc00293b440, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000d60900, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 633 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0027c4919, 0x7, 0xc0027c4900, 0x10, 0xc002553230, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000d61ac0, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 626 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002867809, 0x7, 0xc0028677f0, 0x10, 0xc002869a70, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000e61440, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 28 [IO wait]:
internal/poll.runtime_pollWait(0x7fc18ed90f08, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00041e498, 0x72, 0xd100, 0xd198, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00041e480, 0xc000d8c000, 0xd198, 0xd198, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc00041e480, 0xc000d8c000, 0xd198, 0xd198, 0x203000, 0x0, 0x62b6)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0002b2008, 0xc000d8c000, 0xd198, 0xd198, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x68
crypto/tls.(*atLeastReader).Read(0xc002b003c0, 0xc000d8c000, 0xd198, 0xd198, 0xefb, 0x20ca060, 0xc000edb970)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc0004b0958, 0x20c9e60, 0xc002b003c0, 0x409e05, 0x1b2ce80, 0x1d5cf00)
	/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc0004b0700, 0x20cc920, 0xc0002b2008, 0x5, 0xc0002b2008, 0x9)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc0004b0700, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc0004b0700, 0xc000835000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
bufio.(*Reader).Read(0xc0004c2f60, 0xc00080c1f8, 0x9, 0x9, 0xc000edbd10, 0x0, 0x87e875)
	/usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x20c9c00, 0xc0004c2f60, 0xc00080c1f8, 0x9, 0x9, 0x9, 0xc000094060, 0x0, 0x20ca060)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc00080c1f8, 0x9, 0x9, 0x20c9c00, 0xc0004c2f60, 0x0, 0x0, 0xc00293b020, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00080c1c0, 0xc00293b020, 0x0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000edbfb8, 0xc00046d7a8, 0x867c08)
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/transport.go:1732 +0x8c
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00025a300)
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/transport.go:1660 +0xa3
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/transport.go:673 +0x62f

goroutine 643 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0024c49d0, 0x7, 0xc0024c49c0, 0x10, 0xc00293a8d0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000fad140, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 279 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x20e3840, 0xc0007a8630, 0xc0007a8600)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/record/event.go:219 +0xc5
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/record/event.go:217 +0x6e

goroutine 96 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc000762188, 0x26)
	/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc000762178)
	/usr/local/go/src/sync/cond.go:56 +0x9d
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc000762160, 0xc000332210, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:423 +0xa6
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc00022ae00)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/controller.go:150 +0x40
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000797e00)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000797e00, 0x3b9aca00, 0x0, 0xc00053b601, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*controller).Run(0xc00022ae00, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/controller.go:124 +0x2fd
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0008394d0, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/shared_informer.go:227 +0x47f
created by sigs.k8s.io/controller-runtime/pkg/cache/internal.(*specificInformersMap).Start.func1
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126 +0x122

goroutine 625 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0025514e9, 0x7, 0xc0025514d0, 0x10, 0xc002b02990, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000373100, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 622 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00270e8c9, 0x7, 0xc00270e8b0, 0x10, 0xc00293b5f0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000a7bc00, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 685 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000d61f00, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 689 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000ef9600, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 693 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000a7ba80, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 764 [chan receive]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc000132540, 0xc0007f7b00)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/shared_informer.go:425 +0x49
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0024c4d50, 0xc0024df580)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x59
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

goroutine 620 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00255c490, 0x7, 0xc00255c480, 0x10, 0xc00293aa80, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000751fc0, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 698 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000ef8600, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 280 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x20e3840, 0xc0007a8660, 0xc0002f9fd0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/record/event.go:219 +0xc5
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/record/event.go:217 +0x6e

goroutine 660 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0026cb140, 0x7, 0xc0026cb130, 0x10, 0xc0024e52f0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc001088800, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 694 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000bd9bc0, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 683 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000373100, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 95 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*specificInformersMap).Start(0xc00080c2a0, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:132 +0x4c
created by sigs.k8s.io/controller-runtime/pkg/cache/internal.(*InformersMap).Start
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/deleg_map.go:61 +0x80

goroutine 378 [IO wait]:
internal/poll.runtime_pollWait(0x7fc18ed90e38, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00022b798, 0x72, 0xce00, 0xceb5, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00022b780, 0xc000b68000, 0xceb5, 0xceb5, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc00022b780, 0xc000b68000, 0xceb5, 0xceb5, 0x203000, 0x0, 0xb82c)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0002b20e0, 0xc000b68000, 0xceb5, 0xceb5, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x68
crypto/tls.(*atLeastReader).Read(0xc000822f80, 0xc000b68000, 0xceb5, 0xceb5, 0x2000, 0x400, 0xc000799970)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc0008f4258, 0x20c9e60, 0xc000822f80, 0x409e05, 0x1b2ce80, 0x1d5cf00)
	/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc0008f4000, 0x20cc920, 0xc0002b20e0, 0x5, 0xc0002b20e0, 0x9)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc0008f4000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc0008f4000, 0xc000941000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
bufio.(*Reader).Read(0xc000a394a0, 0xc00080c9d8, 0x9, 0x9, 0xc000799d10, 0x0, 0x87e875)
	/usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x20c9c00, 0xc000a394a0, 0xc00080c9d8, 0x9, 0x9, 0x9, 0xc000094060, 0x0, 0x20ca060)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc00080c9d8, 0x9, 0x9, 0x20c9c00, 0xc000a394a0, 0x0, 0x0, 0xc000bda180, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00080c9a0, 0xc000bda180, 0x0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000799fb8, 0xc0002a87a8, 0x867c08)
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/transport.go:1732 +0x8c
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000538d80)
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/transport.go:1660 +0xa3
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/transport.go:673 +0x62f

goroutine 646 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002866c90, 0x7, 0xc002866c80, 0x10, 0xc0024e5440, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc00040e300, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 618 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0026cad00, 0x7, 0xc0026cacf0, 0x10, 0xc00293b620, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000bd84c0, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 641 [runnable]:
k8s.io/apimachinery/pkg/types.NamespacedName.String(...)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/types/namespacedname.go:42
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00255dd30, 0x7, 0xc00255dd20, 0x10, 0xc00293b6b0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:69 +0x189
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000e60a80, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 585 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000750c80, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 94 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*specificInformersMap).Start(0xc00080c000, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:132 +0x4c
created by sigs.k8s.io/controller-runtime/pkg/cache/internal.(*InformersMap).Start
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/deleg_map.go:60 +0x50

goroutine 632 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0028de829, 0x7, 0xc0028de810, 0x10, 0xc00273f260, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc00040f940, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 687 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000ae9440, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 688 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000ef8ac0, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 658 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002867079, 0x7, 0xc002867060, 0x10, 0xc002869fb0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000372380, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 584 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000373e00, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 670 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000ef6840, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 729 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start(0xc0008ae000, 0xc00005a300, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:163 +0x3dc
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElectionRunnables.func1(0xc0001948c0, 0x20cce80, 0xc0008ae000)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:321 +0x42
created by sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElectionRunnables
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:320 +0x102

goroutine 634 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002550140, 0x7, 0xc002550130, 0x10, 0xc002b02510, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000ae9440, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 690 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc00040f5c0, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 642 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0028dec79, 0x7, 0xc0028dec60, 0x10, 0xc00293af90, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000efee80, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 728 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc000979380, 0xc00000000a)
	/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc000979370)
	/usr/local/go/src/sync/cond.go:56 +0x9d
golang.org/x/net/http2.(*pipe).Read(0xc000979368, 0xc002514001, 0x3dff, 0x3dff, 0x0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/pipe.go:65 +0xa6
golang.org/x/net/http2.transportResponseBody.Read(0xc000979340, 0xc002514001, 0x3dff, 0x3dff, 0x0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/transport.go:2005 +0xac
encoding/json.(*Decoder).refill(0xc000979760, 0xc002b2a30a, 0x9)
	/usr/local/go/src/encoding/json/stream.go:161 +0xeb
encoding/json.(*Decoder).readValue(0xc000979760, 0x0, 0x0, 0x1acf360)
	/usr/local/go/src/encoding/json/stream.go:136 +0x1dc
encoding/json.(*Decoder).Decode(0xc000979760, 0x1af5c00, 0xc002b2a300, 0x0, 0x0)
	/usr/local/go/src/encoding/json/stream.go:63 +0x79
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000a68540, 0xc00252a000, 0x4000, 0x4a80, 0x20dda80, 0x0, 0x38)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:150 +0x1a1
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc002342500, 0x0, 0x20e3240, 0xc0053d5e40, 0x0, 0x0, 0x0, 0xc0007f6f58, 0x455940)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc005942fc0, 0xc0006b9fa8, 0x5, 0x20dda80, 0xc002acfb00, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x7c
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000a68570)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93 +0x175
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:60 +0x9f

goroutine 627 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002ad7329, 0x7, 0xc002ad7310, 0x10, 0xc002ab0d80, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000d61f00, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 674 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000a7a680, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 667 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000efee80, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 681 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc00005ce00, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 288 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0004c3860)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xe0
created by k8s.io/client-go/util/workqueue.newQueue
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x132

goroutine 624 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002550539, 0x7, 0xc002550520, 0x10, 0xc002b02690, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000bd8e40, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 619 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00255c0b9, 0x7, 0xc00255c0a0, 0x10, 0xc00293aa50, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000d60140, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 477 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc000740810, 0xc000000024)
	/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc000740800)
	/usr/local/go/src/sync/cond.go:56 +0x9d
k8s.io/client-go/util/workqueue.(*Type).Get(0xc0004c3860, 0x0, 0x0, 0x1ba4200)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/util/workqueue/queue.go:145 +0xa0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0008ae000, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:178 +0x43
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc0008ae000)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:171 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc001a025a0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc001a025a0, 0x3b9aca00, 0x0, 0x201, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc001a025a0, 0x3b9aca00, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157 +0x32e

goroutine 640 [select]:
k8s.io/apimachinery/pkg/util/wait.WaitFor(0xc00639c560, 0xc000709968, 0xc00005a300, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:403 +0x12e
k8s.io/apimachinery/pkg/util/wait.PollUntil(0x5f5e100, 0xc000709968, 0xc00005a300, 0x8, 0xc000709938)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:356 +0x56
k8s.io/client-go/tools/cache.WaitForCacheSync(0xc00005a300, 0xc000709a08, 0x1, 0x1, 0x7)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/shared_informer.go:108 +0x7c
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*specificInformersMap).Get(0xc00080c000, 0x1de23dd, 0xa, 0x1ddeb70, 0x7, 0x18f1564, 0x7, 0x20e1b00, 0xc000554f00, 0x0, ...)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:166 +0x13b
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*InformersMap).Get(0xc0003238a0, 0x1de23dd, 0xa, 0x1ddeb70, 0x7, 0x18f1564, 0x7, 0x20e1b00, 0xc000554f00, 0x0, ...)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/deleg_map.go:85 +0x105
sigs.k8s.io/controller-runtime/pkg/cache.(*informerCache).Get(0xc0004a8020, 0x211aa80, 0xc00004e090, 0x1ddded5, 0x7, 0x1dddca5, 0x7, 0x20e1b00, 0xc000554f00, 0x0, ...)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/cache/informer_cache.go:53 +0xe4
sigs.k8s.io/controller-runtime/pkg/client.(*DelegatingReader).Get(0xc000323b80, 0x211aa80, 0xc00004e090, 0x1ddded5, 0x7, 0x1dddca5, 0x7, 0x20e1b00, 0xc000554f00, 0xc0003fc701, ...)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/client/split.go:51 +0x13e
github.com/gardener/test-infra/pkg/testrunner.GetHostURLFromIngress(0x2130720, 0xc000789860, 0x1ddded5, 0x7, 0x1dddca5, 0x7, 0x0, 0x0, 0x0, 0xc0019eae70)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/util.go:97 +0xc3
github.com/gardener/test-infra/pkg/testrunner.GetArgoHost(...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/util.go:47
github.com/gardener/test-infra/pkg/testrunner.GetArgoURL(0x2130720, 0xc000789860, 0xc0003e98c0, 0x0, 0x0, 0x0, 0x21)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/util.go:33 +0x63
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000740d00, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:56 +0x2e7
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 630 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0028de389, 0x7, 0xc0028de370, 0x10, 0xc0028697d0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000750c80, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 695 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc00005db00, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 638 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002a4a0b9, 0x7, 0xc002a4a0a0, 0x10, 0xc002a401e0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000bd9800, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 696 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000e617c0, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 684 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000e61440, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 586 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000bd9380, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 761 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0019eaf48, 0x1)
	/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0019eaf38)
	/usr/local/go/src/sync/cond.go:56 +0x9d
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0019eaf20, 0xc005502c50, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:423 +0xa6
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0024e8800)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/controller.go:150 +0x40
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000edce00)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000edce00, 0x3b9aca00, 0x0, 0xc00077fe01, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*controller).Run(0xc0024e8800, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/controller.go:124 +0x2fd
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc00241e510, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/shared_informer.go:227 +0x47f
created by sigs.k8s.io/controller-runtime/pkg/cache/internal.(*specificInformersMap).addInformerToMap
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:204 +0x3db

goroutine 628 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002867fb9, 0x7, 0xc002867fa0, 0x10, 0xc002869d10, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000ef7740, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 691 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000facb40, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 631 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002867bd9, 0x7, 0xc002867bc0, 0x10, 0xc002869bc0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000bd9380, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 686 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000ef7740, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 644 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00248da69, 0x7, 0xc00248da50, 0x10, 0xc002b026c0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc00005c580, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 645 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00265ed70, 0x7, 0xc00265ed60, 0x10, 0xc002a8e180, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000d61f80, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 386 [chan receive]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0006c5810, 0xc0007f6120)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/shared_informer.go:425 +0x49
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00027c390, 0xc00000c680)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x59
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

goroutine 679 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000ae8640, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 370 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0004c39e0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:209 +0x344
created by k8s.io/client-go/util/workqueue.newDelayingQueue
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:55 +0x1d4

goroutine 791 [runnable]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc00248e6c0, 0x5f5e100, 0x0, 0xc00248e660)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:450 +0x18e
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:433 +0x8c

goroutine 676 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000bd84c0, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 621 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00270e4f0, 0x7, 0xc00270e4e0, 0x10, 0xc00293b650, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000ae8640, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 371 [select]:
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).Start(0xc0001948c0, 0xc00005b0e0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:282 +0x170
github.com/gardener/test-infra/pkg/testrunner.StartWatchController.func1(0x2143f60, 0xc0001948c0, 0xc00005b0e0, 0x212a420, 0xc000138fc0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/testrunner.go:75 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.StartWatchController
	/go/src/github.com/gardener/test-infra/pkg/testrunner/testrunner.go:74 +0x490

goroutine 629 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00265e589, 0x7, 0xc00265e570, 0x10, 0xc002a40240, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000373e00, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 636 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00248d649, 0x7, 0xc00248d630, 0x10, 0xc00293a900, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000ef9600, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 223 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0009dbf00)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/shared_informer.go:517 +0x18d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0006c5870, 0xc0001ae400)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x59
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

goroutine 637 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002550929, 0x7, 0xc002550910, 0x10, 0xc002b026f0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc00040f5c0, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 692 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc00005c580, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 697 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000750080, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 623 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0026b0409, 0x7, 0xc0026b03f0, 0x10, 0xc00273f0e0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc00005ce00, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 659 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00265e990, 0x7, 0xc00265e980, 0x10, 0xc0024ebf80, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000ef6840, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 278 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc000740200)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/watch/mux.go:207 +0x66
created by k8s.io/apimachinery/pkg/watch.NewBroadcaster
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/watch/mux.go:75 +0xcc

goroutine 682 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000bd8e40, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 92 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*InformersMap).Start(0xc0003238a0, 0xc00005a300, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/cache/internal/deleg_map.go:62 +0x97
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).waitForCache.func1(0xc0001948c0)
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:338 +0x39
created by sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).waitForCache
	/go/src/github.com/gardener/test-infra/vendor/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:337 +0x5a

goroutine 222 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1.1(0xc0009e2dc8, 0x41742e, 0x7fc18ed92558)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/shared_informer.go:549 +0x54
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0x0, 0xc000794dd8, 0xc0009e2ee8, 0xc0009e2de8)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:265 +0x51
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/shared_informer.go:548 +0x79
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0009e2f40)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000794f40, 0xdf8475800, 0x0, 0x1, 0xc00057c000)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0009dbf00)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/shared_informer.go:546 +0x9b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0006c5870, 0xc0001ae3f0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x59
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

goroutine 635 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00270e110, 0x7, 0xc00270e100, 0x10, 0xc00293b5c0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000ef8ac0, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 387 [chan receive]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc00005a300, 0xc00022ae00)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/controller.go:103 +0x34
created by k8s.io/client-go/tools/cache.(*controller).Run
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0xae

goroutine 388 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc000836140, 0x20e3800, 0xc000a68570, 0xc000563c00, 0xc001056420, 0xc00005a300, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/reflector.go:316 +0x1ec
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000836140, 0xc00005a300, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/reflector.go:287 +0x833
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/reflector.go:125 +0x33
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00007def0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000563ef0, 0x3b9aca00, 0x0, 0x1, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000836140, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/reflector.go:124 +0x169
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc00027c7e0, 0xc00000c760)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x59
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

goroutine 726 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc000836140, 0xc00005a300, 0xc0007f6de0, 0xc001056420)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/reflector.go:227 +0x178
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x260

goroutine 657 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00255d969, 0x7, 0xc00255d950, 0x10, 0xc00293b680, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000facb40, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 590 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc00040e300, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 528 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0027ff0c0, 0xc000000000)
	/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0027ff0b0)
	/usr/local/go/src/sync/cond.go:56 +0x9d
golang.org/x/net/http2.(*pipe).Read(0xc0027ff0a8, 0xc002505200, 0x200, 0x200, 0x0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/pipe.go:65 +0xa6
golang.org/x/net/http2.transportResponseBody.Read(0xc0027ff080, 0xc002505200, 0x200, 0x200, 0x0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/golang.org/x/net/http2/transport.go:2005 +0xac
encoding/json.(*Decoder).refill(0xc0027286e0, 0xc0025e23c0, 0x40be26)
	/usr/local/go/src/encoding/json/stream.go:161 +0xeb
encoding/json.(*Decoder).readValue(0xc0027286e0, 0x0, 0x0, 0x1acf360)
	/usr/local/go/src/encoding/json/stream.go:136 +0x1dc
encoding/json.(*Decoder).Decode(0xc0027286e0, 0x1af5c00, 0xc0025e23c0, 0xc000a09500, 0x10d2a6a)
	/usr/local/go/src/encoding/json/stream.go:63 +0x79
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc002711e30, 0xc002770c00, 0x400, 0x400, 0x0, 0x203000, 0x38)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:150 +0x1a1
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc00272b310, 0x0, 0x20e3240, 0xc00190b340, 0x7fc190ffdd98, 0x0, 0xc000082700, 0xc000443440, 0x437a6e)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc002719240, 0xc00005db00, 0xc00057c3c0, 0x0, 0xc000a09758, 0x169587f, 0x872796)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x7c
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc002711e60)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93 +0x175
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:60 +0x9f

goroutine 705 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000ef8000, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 675 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc0004dbb00, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 654 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0027c4110, 0x7, 0xc0027c4100, 0x10, 0xc002a40900, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000e617c0, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 651 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00255cca9, 0x7, 0xc00255cc90, 0x10, 0xc002a40690, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000a7ba80, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 680 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000a7bc00, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 652 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002ad6f29, 0x7, 0xc002ad6f10, 0x10, 0xc002ab0ba0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000bd9bc0, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 647 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0024889a0, 0x7, 0xc002488990, 0x10, 0xc002711fb0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000ae8000, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 648 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002550d09, 0x7, 0xc002550cf0, 0x10, 0xc00293b470, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc0004db940, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 649 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc00255c8b9, 0x7, 0xc00255c8a0, 0x10, 0xc00293aab0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000ef8000, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 765 [chan receive]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc00005a300, 0xc0024e8800)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/controller.go:103 +0x34
created by k8s.io/client-go/tools/cache.(*controller).Run
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0xae

goroutine 655 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002867449, 0x7, 0xc002867430, 0x10, 0xc002869920, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000750080, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 656 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0028df069, 0x7, 0xc0028df050, 0x10, 0xc002b02660, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000ef8600, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 673 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000bd9800, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 583 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000d61ac0, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 677 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000d60140, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 587 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc00040f940, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 653 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002866870, 0x7, 0xc002866860, 0x10, 0xc002869e60, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc00005db00, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 589 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000d61f80, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 592 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc0004db940, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 678 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000751fc0, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 669 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000372380, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 661 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0025510f9, 0x7, 0xc0025510e0, 0x10, 0xc002b02840, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc0010598c0, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 671 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc001088800, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 662 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc0010598c0, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 666 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000e60a80, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 668 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000fad140, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 616 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc0026b0100, 0x7, 0xc0026b00f0, 0x10, 0xc00273e9f0, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000a7a680, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 650 [select]:
github.com/gardener/test-infra/pkg/testmachinery/controller/watch.(*watch).WatchUntil(0xc0007a86c0, 0x274a48a78000, 0xc002a4a4b9, 0x7, 0xc002a4a4a0, 0x10, 0xc002a40420, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/pkg/testmachinery/controller/watch/watch.go:83 +0x371
github.com/gardener/test-infra/pkg/testrunner.(*Run).Exec(0xc000efe380, 0x212a420, 0xc00105bc00, 0x30f2940, 0x7ffdf61e7a5e, 0xb)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/run.go:61 +0x420
github.com/gardener/test-infra/pkg/testrunner.RunList.Run.func1(0xc00105cd70, 0x30f2940, 0xc00065a000, 0x2e, 0x30, 0xc000ff5260, 0x24, 0xc00013c1c0, 0x1, 0x1, ...)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:81 +0x1ff
created by github.com/gardener/test-infra/pkg/testrunner.RunList.Run
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:75 +0x302

goroutine 591 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000ae8000, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 588 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000efe380, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 665 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000740d00, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 664 [chan send]:
github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent.func1(0xc000d60900, 0xc00057c3c0)
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:150 +0x3f
created by github.com/gardener/test-infra/pkg/testrunner.triggerRunEvent
	/go/src/github.com/gardener/test-infra/pkg/testrunner/list.go:149 +0x56

goroutine 766 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc00168f9a0, 0x20e3800, 0xc002711e60, 0xc0006bbc00, 0xc0004f9e00, 0xc00005a300, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/reflector.go:316 +0x1ec
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc00168f9a0, 0xc00005a300, 0x0, 0x0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/reflector.go:287 +0x833
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/reflector.go:125 +0x33
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0005f8ef0)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006bbef0, 0x3b9aca00, 0x0, 0x1, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc00168f9a0, 0xc00005a300)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/reflector.go:124 +0x169
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0024c4d90, 0xc0024df640)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x59
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

goroutine 718 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc00168f9a0, 0xc00005a300, 0xc002694ae0, 0xc0004f9e00)
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/reflector.go:227 +0x178
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
	/go/src/github.com/gardener/test-infra/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x260

GitHub Bot command to run testruns from repo

What would you like to be added:
We need a mechanism to run predefined testruns as PR tests with the bot.

In addition, a default test or location for the testruns should be customizable.

add direct grafana/loki links to argo workflow resource

Argo 2.7.0 introduces the possibility to directly link to external logging facitilies.
Add the respective grafana/loki links to the argo resources (workflow).
See argoproj/argo-workflows@14478bc

Log Collector and Log Evaluation (for excessive logs and errors) Under Test

What would you like to be added:
We don't really collect and look into pod logs regularly, but should do that actually automatically, so that we do not miss excessive (=useless) logs and errors that under normal circumstances should not appear in logs of a healthy system.

Why is this needed:
We sometimes miss excessive (=useless) logs and errors that have otherwise no side effects that the current tests would catch.

Add PDBs to test pods

Currently our test pods get terminated when cluster nodes are rolling (i.e. due to OS updates).
However we'd prefer to let the tests pass and roll the node only after the last test pod terminated on its own.

Hence some kind of PDBs are needed for the test pods to let mcm drain the nodes by waiting for PDBs to be fulfilled. Unfortunately we cannot specify .spec.maxUnavailable as the testpods are not supervised by a deployment/replicaset/... (see https://kubernetes.io/docs/tasks/run-application/configure-pdb/#arbitrary-controllers-and-selectors)
Hence we could only use .spec.minAvailable which is however pretty dynamic (as the number of currently active testpods changes all the time), so this would need to be applied continuously by some controller.

Deploy prerequisiteis at runtime

What would you like to be added:

The prerequisites chart (containing argo workflows, crds, minio, ..) should be automatically installed during runtime so it can be also automatically updated and checked if all needed components are available.

Therefore we should have a look at https://github.com/gobuffalo/packr to include the complete helmchart.
We could also think of reusing the gardener resource manager(https://github.com/gardener/gardener-resource-manager) to ensure everything is up and running and gets reconciled otherwise we would have to manage leaked resource on our own.

as a result I want to move all configuration to a dedicated configuration file that then also includes the helm values for the additional deployed charts.

WDYT @dguendisch @OlegLoewen

Adjust the deployment docs

What happened:
Following the docs I found some issues when I tried to deploy the testmachinery.

make install-controller was renamed to deploy-controller.
After executing make deploy-controller the testmachinery-controller runs into an ErrImageNeverPull error because the pull policy in the local-values.yaml is set to Never.
Now the container pod runs into an ImagePullBackOff because it can't find the version 0.192.0-dev. Do I have to push it by myself?
When I change the controller version manually to latest, the pod at least pulls the right image but then an error occurs in the controller:

"msg":"error during ensureDependencies","error":"[elasticsearchConfiguration: Required value: elastic search config is required if collector is enabled, s3Configuration.server.endpoint: Required value: endpoint or minio has to be defined]","errorCauses":[{"error":"elasticsearchConfiguration: Required value: elastic search config is required if collector is enabled"},{"error":"s3Configuration.server.endpoint: Required value: endpoint or minio has to be defined"}]

What you expected to happen:
The deployment setup is properly documented and works.

How to reproduce it (as minimally and precisely as possible):

Create Gardner AWS Cluster
Run make deploy-controller

Environment:
AWS k8s version: 1.18.14
test-infra repo: commit

Configuration Namespace

Problem

When writing testruns where we have a lot of nested workflows global config (spec.config) to share configuration is not enough as subtrees share a lot of configuration.

example:
When running a full gardener test configuration like shootname, secrets, provider, etc is shared across one sub shoot test but parallel subtree's need different configuration.

Description

To save duplicated configuration configuration of steps should be shared within its subtree.
Therefore config should be able to be inherited.

[Flaky Test] controller_test

How to categorize this issue?

/area testing
/kind flake
/priority 3

Which test(s)/suite(s) are flaking:
https://github.com/gardener/test-infra/blob/f494cd53f4571bc4e4d3d0bbc9f81dc379420668/pkg/testmachinery/controller/ttl/controller_test.go

CI link:
https://concourse.ci.gardener.cloud/teams/gardener/pipelines/test-infra-master/jobs/master-pull-request-job/builds/67

Reason for failure:
A test namespace is actually composed of a rand int from 0 to 1000, hence with some tests it collides here and there.
However upon checking I noticed that the AfterEach

test-infra/pkg/testmachinery/controller/ttl/controller_test.go

Lines 64 to 72 in f494cd5

 Eventually(func() error { 

 if err := fakeClient.Get(ctx, client.ObjectKeyFromObject(ns), &corev1.Namespace{}); err != nil { 

 if apierrors.IsNotFound(err) { 

 return nil 

 } 

 return err 

 } 

 return nil 

 }, 1*time.Minute, 10*time.Second).Should(Succeed())

doesn't really make sense: it seems this should wait until a namespace is deleted but it happily progresses when the namespace is simply existing and in Terminating state as this won't throw any error...
I think that the namespace will actually never delete as there is probably no gc controller running in the testenv that would also terminate the contained resources like TestRun.

@schrodit do I miss another intent? If not, either testenv can be setup with a gc controller or we remove the resp. wait for ns deletion and instead increase the randomness greatly.

Anything else we need to know:

Passing secret to test-def : Format is changed when I access inside the container

I have a use-case where I want to pass a yaml file as a environment variable inside the container. When I make it as a secret and access it inside the container. The yaml format getting changed.

rotate-kubeconfig integration tests breaks later tests

How to categorize this issue?
/area testing
/kind bug
/priority normal

What happened:
When promoting the integration test should rotate kubeconfig for a shoot cluster in gardener/gardener#3300 from beta to default, we missed to understand that this tests will "invalidate" the kubeconfigs used by subsequent tests.
The attempt to overcome this with gardener/gardener#3326 unfortunately doesn't work, as the kubeconfigs from TestMachinery are currently not meant to be "updated" via intermediate steps, hence all tests will only get the kubeconfigs from the create-shoot step, see e.g. this generated argo task for the e2e tests:

      - arguments:
          artifacts:
          - from: '{{tasks.create-shoot-create-testflow.outputs.artifacts.kubeconfigs}}'
            name: kubeconfigs
          ...
        dependencies:
        - shoot-default-serial-test-suite-tests-default-testflow
        name: e2e-fast-e2e-testflow

What you expected to happen:
Maybe kubeconfigs should be allowed to be overwritten by intermediate steps.

Binary files in TM configs

When TM file config is used with binary base64 data, the file is corrupted and the size of the file at stage input is larger than the original file encoded with the base64 utility.

I'm not a UTF-8 expert or read all the code paths, but I presume it's due to usage of string() typecasting and possibly due to protobufs internal to Argo. It seems likely there is some code point expansion going on as a result.

Support more version patterns in flavor configs

Today we support things like

kubernetes:
    pattern: "latest"

(see https://github.com/gardener/test-infra/blob/master/docs/testrunner/README.md#shoot-flavor-configuration)

Sometimes we want to express a flavor that should select one minor version older than the latest one, something like latest - 0.1.
If arithmetic is too complex, we could also simply introduce a few fixed named patterns like oneMinorBeforeLatest, twoMinorBeforeLatest, ... (as we usually anyway only support a limited set of minor K8s versions at a time).

replace "component-spec" language bindings w/ OCM-Lib

Context / Motivation

component-cli was deprecated (superseded by ocm-cli and removed from gardener-cicd (pipeline template and all pipelines) as part of cleanup and consolidation activity.

component-cli as well as testmachinery uses golang bindings from component-spec repository for retrieving and deserialising OCM-Component-Descriptors. Said language-bindings were also superseded by a re-implementation (in golang) in ocm-repository.

The re-implemented ocm-lib supports handling of different versions of component-descriptors w/ a stable API (which is promised to be kept stable also for future spec-versions of OCM; there are already two spec-versions: v2 and v3 (gardener-cicd currently uses v2).

Adopting the re-implemented ocm-lib in testmachinery will allow us to archive the (currently unmaintained) "component-spec"-language-bindings and (hopefully) reduce related maintenance efforts in the future.

Implementation Proposal

Remove dependencies from testmachinery towards https://github.com/gardener/component-spec, and port it to using ocm-lib instead.

Update integration step image

Currently the integration step fails to compile given test because it contains [email protected] features - ref https://concourse.ci.gardener.cloud/teams/gardener/pipelines/test-infra-master/jobs/master-head-update-job/builds/6.

Currently the used image is eu.gcr.io/gardener-project/cc/job-image-golang:0.10.0

$ docker run eu.gcr.io/gardener-project/cc/job-image-golang:0.10.0 go version
go version go1.12.6 linux/amd64

The image needs to be updated to eu.gcr.io/gardener-project/cc/job-image-golang:0.12.0

$ docker run eu.gcr.io/gardener-project/cc/job-image-golang:0.12.0 go version
go version go1.13.9 linux/amd64

PS: I am opening an issue as I don't have rights to push to refs/meta/ci or create a PR about it :(

Add support for `adminKubeconfig` subresource to tests

As of today, TestDefinitions and their respective code in this repo expect a kubeconfig to be presented at runtime (https://github.com/gardener/test-infra/blob/master/.test-defs/conformanceTestgrid.yaml#L56). While this is still valid for local test executions, we should also support requesting/using short-lived kubeconfigs.

Gardener issues short-lived admin kubeconfigs as an alternative to static tokens: https://gardener.cloud/docs/gardener/api-reference/authentication/#adminkubeconfigrequest

Ideally, the test framework used here can be adapted to:

accept an existing kubeconfig, if it is presented with such a file
request an adminkubeconfig, if a Gardener kubeconfig is presented

Allow filtering of testruns

What would you like to be added:
It would be nice if in the tm dashboard it is possible to filter test runs by their status, details (cloud provider, Kubernetes version, OS), time range, etc.

Why is this needed:
Currently it is hard to correlate failed integration test cluster with the respective testrun to check the actual failure reason.

Add some webhook support github <-> testmachinery internal

What would you like to be added:
Some repos like https://github.com/gardener/gardener-extension-provider-openstack want to run integration tests via the tm-bot that target non-public facing networks.
For tests targeting non-public networks we have a testmachinery installation residing in said non-public networks (let's call it tm-internal).
However the tm-bot just reacts to github webhook events, hence public github cannot push events to the tm-bot from tm-internal.

We should add some mechanism that allows a tm-bot from tm-internal to react on public github events.
Possible options include:

webhook queue (like gardener bot)
tm-public to store received webhook events targetting internal networks; tm-internal to watch these received webhook events in tm-public and act accordingly

Why is this needed:
Allow public github repos to trigger integration tests targetting a non-public network.

Automation of Synthetic Load Tests

What would you like to be added:
Measure CPU and memory footprint, measure disk and network throughput and latency, measure API server and ETCD throughput and latency of our shoot clusters automatically, so that we do not miss regressions.

Why is this needed:
We sometimes miss regressions that the current tests do not catch.

⚠️ Note: Avoid duplicating the Kubernetes performance tests: https://github.com/kubernetes/perf-tests

Enable parallel testing of garden-setup on multiple host-cloudproviders

Currently, the testmachinery can run garden-setup on one host cloudprovider. I'd like to be able to test whether creating a gardener works on multiple infrastructures. I can do that already, by starting one test for each infrastructure after the previous one is finished, but doing this sequentially takes a lot of time
Therefore I'd like to spin up a gardener on multiple infrastructures in parallel within one test.

Add fallback to e2e test execution

We should have some fallback when encountering a cluster with e.g. k8s 1.21 but no desc files are available yet (i.e. I just added some via #340).
The fallback could just take the most recent desc files for the given clusters k8s version.

@schrodit wdyt?

Support selecting the network type calico<->cilium

Testmachinery should support switching the network type via the shoot spec to allow tests to be created that don't target the default (calico) but can target cilium as well.

Testmachinery flavors need to support this
CreateShoot step in g/g needs to support this

Add semver pattern support for gardener extensions

Instead of using hard coded values here it may be a good idea to use semver ranges. Gardener/Gem offers great support for this use case.

Add support to specify semver ranges in the test configuration of the tm bot like it is implemeneted in the g/gems.

It should be possible to specify the following exmaple:

# configure default values for pull request tests
test:

  # gardener extensions to be deployed
  # will be merged with the default extensions defined in garden setup.
  gardener-extensions:
    value: |
      os-coreos: 
          repo: "https://github.com/gardener/gardener-extensions.git"
          version": "1.0.x"

Implementation
Reuse the extension gems to resolve th correct version.
We would need to replace the git tolling with GitHub api requests.

Reserve Excess capacity pods are monitored in the TM health check

What happened:
The reserve-excess-capacity deployment is monitored via the argo mr.
If the cluster currently scales up the health check fails which causes the tm to be unhealthy

What you expected to happen:
The access capacity pods are not considered for healthiness

PR check Pending for successful test run

/kind bug

What happened:

On gardener/gardener-extension-provider-aws#325, I triggered a test run via /test, which finished successfully, but the PR check is still Pending:

What you expected to happen:

The PR check status to reflect the correct test result?
Probably, there is some retry mechanism missing.

Wrong additionalLocations in flavors lead to rejected Testruns

A flavor snippet like this:

- provider: gcp
  additionalLocations:
  - type: git
  allowPrivilegedContainers: false

Leads to a rendered Testrun that contains a location without any repo and revision and is therefore consequently rejected by the admission webhook.

admission webhook \"validate-testrun.tm.garden.cloud\" denied the request: [spec.locationSets[0].locations[84].repo: Required value: repo has to be defined for git TestDefinition locations, spec.locationSets[0].locations[84].revision: Required value: revision has to be defined for git TestDefinition locations]"

The flavor parsing should instead immediately reject meaningless additionalLocations fields (that have no repo and revision but just a type).

Add support for artifactFrom

Problem

As a user a want to specify specific steps to get the input artifact.
This feature give the opportunity to create shoot --> update gardener to a new version --> test the same shoot again.

Description:

Add a artifactFromparameter to the step configuration so that specific steps can be referenced as input sources.
⚠️ Only steps with definition.name are allowed as definition.labels could include multiple steps.

testflow:
- name: step1
  definition:
    name: test
- name: step2
  definition:
    label: default
  artifactFrom: step1

Remove testgrid sections for gardener e2e tests

Remove testgrid sections for gardener e2e tests (presumably located in https://github.com/kubernetes/test-infra/tree/master/config/testgrids/gardener) as we only want to report/upload conformance tests.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

	Eventually(func() error {
	if err := fakeClient.Get(ctx, client.ObjectKeyFromObject(ns), &corev1.Namespace{}); err != nil {
	if apierrors.IsNotFound(err) {
	return nil
	}
	return err
	}
	return nil
	}, 1time.Minute, 10time.Second).Should(Succeed())

gardener / test-infra Goto Github PK

test-infra's Introduction

Test Machinery

Additional Tools in this Repository

test-infra's People

Stargazers

Watchers

Forkers

test-infra's Issues

Issue

Description

Description

Problem

Description

Context / Motivation

Implementation Proposal

Problem

Description:

Recommend Projects

Recommend Topics

Recommend Org