We need a tool that will:
Look at commits from master and create a PR on stable branch. This will trigger other tests. For each module we 'll mark some tests as required.
The tool will look for PRs on stable, and if required check are passing, it will create a tag on the commit, and fast forward stable to that commit, and close the PR.
I can case of failed checks, the PR will be closed.

Everything is documented in the Readme. We might want to create options for depoying kubernetes, persistent disk and hazelcast.

Currently the pkg_check tool reads per-package code coverage requirement from a local file. If a package does not have an entry in the file, the code coverage requirement is effectively 0 (a.k.a. no tests are needed at all).

It would be better if we can set code coverage requirement for the whole repo, and uses the file to lower the coverage for certain package (e.g. a package only containing a main method).

cc @yutongz @sebastienvas

The script attempted to delete master branch. We should only delete branch that we created.

The list of comparision should include:

• Kokoro
• Container Cloud Builder
• Drone.io
• Prow

and point out the pain points of our existing Jenkins bases infra.

According to @fejta, Kubernetes is using a fork of gazelle and have code coverage and linters done differently. @ixdy might be able to help us.

PRs might leave behind stale resources in the cluster due to accidents or developer error.
We should try to reset the cluster every few hours / daily.

Once PR have moved to master, there is no point to check code coverage and cla again. However we cannot disable them. We need to add a flag that list all the checks that we can ignore:

curl https://api.github.com/repos/istio/mixer/status/bb04cef3c86676529737166b7db7dd40e2670739

"statuses": [
{
"url": "https://api.github.com/repos/istio/mixer/statuses/bb04cef3c86676529737166b7db7dd40e2670739",
"id": 1129469319,
"state": "failure",
"description": "The Travis CI build failed",
"target_url": "https://travis-ci.org/istio/mixer/builds/218654453",
"context": "continuous-integration/travis-ci/push",
"created_at": "2017-04-04T22:18:07Z",
"updated_at": "2017-04-04T22:18:07Z"
},
{
"url": "https://api.github.com/repos/istio/mixer/statuses/bb04cef3c86676529737166b7db7dd40e2670739",
"id": 1129471994,
"state": "success",
"description": "Jenkins job istio/mixer-pr-stable passed",
"target_url": "https://testing.istio.io/job/istio/job/mixer-pr-stable/19/",
"context": "istio/mixer-pr-stable",
"created_at": "2017-04-04T22:19:12Z",
"updated_at": "2017-04-04T22:19:12Z"
},
{
"url": "https://api.github.com/repos/istio/mixer/statuses/bb04cef3c86676529737166b7db7dd40e2670739",
"id": 1129472448,
"state": "success",
"description": "All necessary CLAs are signed",
"target_url": null,
"context": "cla/google",
"created_at": "2017-04-04T22:19:22Z",
"updated_at": "2017-04-04T22:19:22Z"
}

We should delete all namespace that are not accounted for

Based on istio/istio#186
We want to create another cluster for E2E testing and setup different runs/

What is the testing policy for new features.

As part of istio release 0.2, we ll be publishing debian packages. Those package will be available for people to download and install manually, but we want to create a repository available to the public.

For short term we can use Google internal tools to do that, but this means that only Googler will be able to update the repo with release packages.

An alternative would be to create and maintain our own repo until we can push it to official debian repos. If creating our own repo, we need to be able to publish debian artifacts in a central location and have the repo make it available.

This will help us understand where we need to focus our efforts:
https://github.com/kubernetes/test-infra/tree/master/velodrome/

Bazel uses the SHA of the workspace to define where to store cache. Since we have multiple projects for the same module, we have a different cache for each. It would be good to share the cache to speed up builds.

We can overwrite when Jenkins checkout the code in Jenkins, as we do for goBuildNode.

Note that we may want to also speed up builds by separating workspaces when we do builds using different configs like asan, tsan, msan, release, etc...

prow: https://github.com/kubernetes/test-infra/tree/master/prow

Currently we are using Jenkins, because some of the workflow are not supported by prow:

• Fast Forwarding passing commit from master to stable
• Allow to run Jobs in module B when a PR is created on module A

Let's work with spxtr@ to see if this is something we might do.

This process is currently manual, and because of that never gets done which impact our release schedule

Use Github API for creating annotated tags
Use Github API to upload the tar(s) created by create_release_archives.sh

Here are some information about the release
https://github.com/istio/istio/blob/master/release/README.md

a couple of prow issues I noticed:

• it only shows "Build Still Running!" while in progress, we need to see the current log (and stages)
• once done, the raw build log seems to display as html instead of text https://storage.googleapis.com/istio-prow/pull/istio_istio/460/istio-presubmit/45/build-log.txt - strangely it's fine when using curl... so User-Agent magic played by googleapis ?
• it says "prow/istio-presubmit.sh — Skipped" on istio/istio#470 and I can't seem to tell why (probably because it's a branch of a branch? but jenkins does run fine)
• can't seem to find PR log or other overview (maybe a permission issue?) from https://k8s-gubernator.appspot.com/builds/istio-prow/pull/istio_istio/460/istio-presubmit/ ? tried https://k8s-gubernator.appspot.com/builds/istio-prow/pull/istio_istio/ but nothing really working there

This being said It's great already it builds with /ok-to-build without having the submitter in the org!

Currently, we have several metrics tracking issues, pull-request-related info on velodrome.istio.io In order to monitor developing process and support external contributors better, extra metrics are required like build duration, recent build failure rate, code coverage changing. Release support is also something we are interested.

Istio is going to be running in its own namespace. Same thing for istio-ca. It would be helpful to have a manager pool of cluster to do the testing such that each test does not need to wait for cluster creation.

let s use this for qualification at first, not for smoke testing.

to make layered builds

PR incoming

/assign ldemailly

Background:
In a previous system (proprietary, not OSS), we had a list of packages with their baseline coverage and perf numbers. Checkins were not allowed to enter the repo if these numbers regressed. Developers could change the baselines at will in the case a regression was inevitable. So the point was not to prevent forward progress, it was more to prevent accidental regressions. We also had a process that would regularly increase any coverage or perf numbers based on observed current levels. So the numbers would naturally tend to get better over time and never backslide.

So imagine just having a file at the top of the repo that lists the various packages and their coverage numbers. As part of the checking process, we run a little script that compares the output of the coverage test run with what's in the file, and then fails the commit if there's a regression.

In every repo we could have a codecov.info file which defines the threshold and we could have the script in the toolbox repo such that all modules can use it.

Our current release note generation is manual and therefore nonexistent. We can get assistance from @david-mcmahon and team on how to set this.

note: posting after the fact to have record of the issue

need to specify prowjob_namespace in config.yaml

currently seeing:

{"error":"error listing prow jobs: response has status \"404 Not Found\" and body \"{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"the server could not find the requested resource\",\"reason\":\"NotFound\",\"details\"{},\"code\":404}\n\"","level":"error","msg":"Error syncing periodic jobs.","time":"2017-07-25T22:43:39Z"}

Currently we have a PD that we use for Jenkins. We create backup of config file each time we make changes or update plugins. It would be nice to generate a new PD from the backup in case something bad happen to the PD.

Currently automated qualification uses master as its head branch. Any new commit pushed to master will trigger another round of tests. We should use a specific branch for head such that it does not change and delete that branch when PR is closed or merged.

I propose that we customize the link posted with different Prow job statuses.

I have the following use case in mind:

...job starts...
plank posts to Github with the prow/deck logs link
...the job completes...
plank posts to Github with the gubernator job link. 

This will speed up build time.

across the istio repos, all of the build status badges are broken. This appears to be because the link to jenkins requires "login" now.

Before any maintenance, we need to:

• Put Jenkins in shutdown mode to prevent new jobs from starting
• Create a snapshot of Jenkins persistent disk
• Run the backup command which gather the important configuration from Jenkins, stored them in source code, and get Jenkins credentials
• Push the commit created by the last command.

• Create a script to create a snapshot.

We need to define the testcases that will cover all features to be delivered in alpha and implement them.

In order to reduce code duplication in Jenkinsfile, we need to create libaries according to https://jenkins.io/doc/book/pipeline/shared-libraries/

Codecov.io is mostly working with Jenkins on istio.io/manager. We still need to generate coverage reports upon successful merge into the master branch so codecov.io can properly generate coverage % diff against master (see warning message below). I think we just need to re-run coverage report on post-submit/commit trigger.

No coverage uploaded for pull request base (master@f3fdddb). Click here to learn what that means.

Mungegithub is going to be used for:

• Submit Queue
• Release Note Generation

Due to the fact that credentials archive in cloud storage needs to be the same as the commit on the branch it forces us to push to the branch directly which might me difficult when multiple contributors are working on istio-testing. We should move backup to another repo and move backup and create pd scripts there was well.

Because we post message when a presubmit passed using a golang program we need to run maniFlow{} on a slave. It would be good to have it run on a master executor to free up slaves.

As discussed, let's rename this repo to test-infra and provide a valid description.

When starting prowbazel:0.2.1 without privilege the container dies with

• [[ 1000 -ne 0 ]]
• SUDO=sudo
• sudo service docker start

In theory we don't need docker to work when running without privilege so we can probably discard this issue, but I am not sure there is a way to check if we are running in privilege mode or not.

Currently Manager e2e test run directly on Jenkins Cluster. Another cluster should be created and plugged in to the tests.

Jenkins libraries resources are very limited. It would be good to have a submodules with all checks (linter, codecov, docker, etc...) checked in such that we can update all modules at once.

Change would be implemented on the sumodules and then a PR to each module would be necessary to enable it, which will also include potential change to make the checks pass.

That way we could also call just one method that does all the checks, and user would still be able to run the checks from the repo they are working on. (Which is impossible with Jenkins pipeline library resource)

We need to publish code coverage to cloud storage in order to get publish to velodrome.

Error logs from hook:

// ...
{"error":"error unmarshaling /etc/config/config: error converting YAML to JSON: yaml: line 202: did not find expected key","level":"error","msg":"Error loading config.","path":"/etc/config/config","time":"2017-07-24T16:26:00Z"}
// ...

Currently each bazel build will download workspace dependencies. Same thing for go deps.

Note that I tried using a NFS mount point on each slave, but bazel just hangs writing a lock.

My next try will be to use have a daily jobs that persist each module bazel cache and mount it using aufs on each slave. This would require changing the slave entrypoint and probably using a NFS in conjonction with aufs mount.

Artifacts link is defined here
https://github.com/istio/test-infra/blob/master/src/org/istio/testutils/GitUtilities.groovy#L152

It should be updated to list all docker images created and pushlished.