googlecloudplatform / runtimes-common Goto Github PK

e.g. using a requirements.txt file from the localhost in the test container

Currently iDiff requires the image hashes of two images in order to diff them. It might be useful if users could use the URLs of the images directly as at least one of the images they want to compare will likely be stored in a container registry.

Ref #6

This data is needed for adding a deploy latency metrics to the silver metrics dashboard.

containers describe should now be able to resolve digests, so we don't need to do that ourselves.

We can also add in more logging now of the actual vulnerabilities.

this could be accomplished either through cloning a project which contains the template inside of it, or a script which uses a template plus user-provided parameters (repo, image name, etc) to generate a cloudbuild YAML/JSON config with some standardized build steps (tag checker, structure tests, etc).

@nkubala

It would be great to be able to assert file permissions as part of fileExistenceChecks. Without this, it doesn't do much good to verify that a file exists if it's not readable/writable/executable/etc by the default docker user.

This would help debug bad config file formats

We're using a pretty old version of gcloud in that image, we should bump it and switch to the beta command group.

The image is outdated. It doesn't include the custom test code and includes a logging test which is not in the documentation.

We should support deploying sample applications for running integration tests to all environments. This will require use of different APIs and expecting slightly different behavior depending on the environment; for example, writing Stackdriver Logs is a different process in GAE standard vs flex. Test driver should account for that.

Currently all shell variables, and $() directives, in the commands are replaced by --vars substitutions. We should enable at least an escape path for this.

First, $() should remain unmodified.

Option 1: $VARIABLE should take the --vars substitution value if VARIABLE is defined, or remains unchanged. This confuses a variable intended to used as an env against a --vars substitution.
Option 2: --vars substitutions should begin with underscores to be separate from env variables, for example $_SUBSTITUTION. Unresolved substitutions should result in errors.
Option 3: Don't use shell variable syntax for substitutions.

The retag job itself takes a list of config files as command line arguments, but all of the other associated scripts (mostly for testing) have the directories hardcoded into them. We should change them to all take in a list of config files as arguments from the command line.

Our structure test is failing and we see
./ext_run.sh: line 150: IMAGE_TAR: unbound variable
I see it has "set -u" on so the check of IMAGE_TAR fails if we don't use the "-t" flag.

There is a requirement for runtimes to log with trace ID extracted from HTTP header and at a specific log level, neither of which is possible by just logging to stdout/stderr. So, I think the Logging test should be extended, or a new one created, to verify logging with trace ID and log level.

@nkubala

It's not obvious that it needs to be named "Dockerfile.in".

The idea is, the sample application can (optionally) return a list of {endpoint, timeout} tuples to the test driver, containing endpoints where custom tests are exposed. The test driver can then hit these as normal and verify output according to some spec.

What template variables are available to use in the Dockerfile? Looking at the code, I see that ${STAGING_IMAGE} is replaced with the built image name. Are there others?

We should try and remove the dependency on having a working Docker daemon on the host machine for the tool. It should be possible to use the Docker Registry API directly to do things like image push/pull and history.

I'm seeing this message intermittently. The image being tested, gcr.io/google-appengine/python:latest has /app as a symlink to /home/vmagent/app, so maybe that's a problem? But it's been that way for more than a year.

Error from Kokoro log:

Step #3: 6bcb0d4f1d1d: Pull complete
Step #3: Digest: sha256:eaa3d87858f06447663029e08b6d5166c73d40b9a7b7880e5eea9a57dd762faf
Step #3: Status: Downloaded newer image for gcr.io/gcp-runtimes/structure_test:latest
Starting Step #3
Step #3: container_linux.go:247: starting container process caused "chdir to cwd (\"/app\") set in config.json failed: no such file or directory"
Step #3: docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "chdir to cwd (\"/app\") set in config.json failed: no such file or directory".
Finished Step #3
ERROR
ERROR: build step "gcr.io/gcp-runtimes/structure_test:latest" failed: exit status 127

cloudbuild.yaml: https://github.com/GoogleCloudPlatform/python-runtime/blob/master/cloudbuild.yaml#L14

It also happened while building on my machine yesterday, but I can't remember the specifics, and I can't replicate it locally.

The flow is like this:

CONFIG_DIR=$(pwd)/.cfg
...
VOLUME_STR="--volumes-from st_container -v $CONFIG_DIR:/cfg"
...
VOLUME_STR=$VOLUME_STR" -v $FULLPATH:/workspace"
...
docker run --rm --entrypoint="$ENTRYPOINT" $VOLUME_STR "$IMAGE_NAME" $CMD_STRING

I will change the script from /bin/sh to /bin/bash and use a bash array to fix this, unless you have an objection to the bash dependency or better idea.

It happens when the test fails. Here is the log:

Step #3: Digest: sha256:54d3da62dd1d7b2a5a23c37ab23e113a08d4e1845985947198ec788f23920731
Step #3: Status: Downloaded newer image for gcr.io/gcp-runtimes/structure_test:latest
Starting Step #3
Step #3: === RUN   TestAll
Step #3: 2016/12/07 20:27:51 Running tests for file /workspace/php-nginx.yaml
DEBUG: Reading GCS logfile: 206 (read 2727 bytes)
Step #3: --- FAIL: TestAll (0.02s)
Step #3:        command_test_v1.go:62: COMMAND TEST: php-path
Step #3:        structure_test_v1.go:137: Executing: [which php]
Step #3:        structure_test_v1.go:160: Error running command: exit status 1. Continuing.
Step #3:        structure_test.go:56: Expected string '/opt/php/bin/php' not found in output!
Step #3:        structure_test_v1.go:219: Test php-path exited with incorrect error code! Expected: 0, Actual: 1
Step #3:        command_test_v1.go:62: COMMAND TEST: version
Step #3:        structure_test_v1.go:137: Executing: [php -v]
Step #3:        structure_test_v1.go:160: Error running command: exec: "php": executable file not found in $PATH. Continuing.
Step #3: panic: interface conversion: error is *exec.Error, not *exec.ExitError [recovered]
Step #3:        panic: interface conversion: error is *exec.Error, not *exec.ExitError
Step #3: 
Step #3: goroutine 5 [running]:
Step #3: panic(0x5580e0, 0xc42000ad80)
Step #3:        /usr/local/go/src/runtime/panic.go:500 +0x1a1
Step #3: testing.tRunner.func1(0xc420062180)
Step #3:        /usr/local/go/src/testing/testing.go:579 +0x25d
Step #3: panic(0x5580e0, 0xc42000ad80)
Step #3:        /usr/local/go/src/runtime/panic.go:458 +0x243
Step #3: github.com/GoogleCloudPlatform/runtimes-common/structure_tests.ProcessCommand(0xc420062180, 0x0, 0x0, 0x0, 0xc42000aa80, 0x2, 0x4, 0x1, 0x0, 0x0, ...)
Step #3:        /workspace/gopath/src/github.com/GoogleCloudPlatform/runtimes-common/structure_tests/structure_test_v1.go:164 +0x686
Step #3: github.com/GoogleCloudPlatform/runtimes-common/structure_tests.StructureTestv1.RunCommandTests(0x0, 0x0, 0x0, 0xc42004b880, 0x2, 0x4, 0x0, 0x0, 0x0, 0x0, ...)
Step #3:        /workspace/gopath/src/github.com/GoogleCloudPlatform/runtimes-common/structure_tests/structure_test_v1.go:52 +0x22f
Step #3: github.com/GoogleCloudPlatform/runtimes-common/structure_tests.StructureTestv1.RunAll(0x0, 0x0, 0x0, 0xc42004b880, 0x2, 0x4, 0x0, 0x0, 0x0, 0x0, ...)
Step #3:        /workspace/gopath/src/github.com/GoogleCloudPlatform/runtimes-common/structure_tests/structure_test_v1.go:38 +0xee
Step #3: github.com/GoogleCloudPlatform/runtimes-common/structure_tests.(*StructureTestv1).RunAll(0xc42000cfc0, 0xc420062180, 0xc42002ff28)
Step #3:        <autogenerated>:1 +0x79
Step #3: github.com/GoogleCloudPlatform/runtimes-common/structure_tests.TestAll(0xc420062180)
Step #3:        /workspace/gopath/src/github.com/GoogleCloudPlatform/runtimes-common/structure_tests/structure_test.go:40 +0x187
Step #3: testing.tRunner(0xc420062180, 0x593c48)
Step #3:        /usr/local/go/src/testing/testing.go:610 +0x81
Step #3: created by testing.(*T).Run
Step #3:        /usr/local/go/src/testing/testing.go:646 +0x2ec
Finished Step #3
ERROR
ERROR: build step "gcr.io/gcp-runtimes/structure_test" failed: exit status 2

This is a shared issue with https://github.com/GoogleCloudPlatform/debian-docker. Commands like envsubst are not standard on OS X, and require an installation of homebrew plus the required kegs to already be installed. This is probably fine to require, but we should at least document this somewhere. We could alternatively write a script that will install these for a Mac user.

The doc states that the POST data would:

• hit /logging_standard. Instead I see the /logging endpoint being hit.
• Have token and level request parameters. I see token and log_name.

This will make it easier to change the structure test internals without breaking every existing test case.

RIght now to test a very stripped down image, you have to do something like this:

# Define a tiny binary to run the test
go_binary(
  name = "mytestbinary",
  srcs = "my_test.go"
)

# Create a new image derived from the one you want to test
docker_build(
  name = "my_test_image",
  base = ":the_image_i_want_to_test",
  files = [":mytestbinary"],
)

# Then reference that one in your test
structure_test(
  name = "my_test",
  image = ":my_test_image",
  config = "myconfig.yaml",
)

It would be nicer to remove the need to make the intermediate image. Maybe something like:

structure_test(
  name = "my_test",
  image = ":the_image_i_want_to_test",
  files = ":mytestbinary",
  config = "myconfig.yaml",
)

This would handle either mounting in your test binary or building the intermediate image for the user.