google / shipshape Goto Github PK

We should hide the "Verbose mode", and "Docker socket" options behind an Advanced... button to make the Jenkins plugin configuration page simpler.

I tried to run with local docker images. I run the following cmds:
$ ./setup-bazel.sh
$ bazel build //...
$ bazel build //shipshape/docker:service
$ bazel build //shipshape/androidlint_analyzer/docker:android_lint
$ ./shipshape/test/end_to_end_test.sh --tag local
but end_to_end.log contains some errors.

I also run the following cmds:
$ ./bazel-bin/shipshape/cli/shipshape --categories="AndroidLint" --analyzer_images=android_lint:local --tag=local Directory
$ ./bazel-bin/shipshape/cli/shipshape --tag=local --analyzer_images=service:local --categories="go vet,JSHint,PyLint" Directory

Similar error occurs:

E0629 14:54:32.085070 51943 shipshape.go:426] stderr:
Error response from daemon: no such id: shipping_container
time="2015-06-29T14:54:32-07:00" level=fatal msg="Error: failed to stop one or more containers"
E0629 14:54:32.873040 51943 shipshape.go:426] stderr:
Unable to find image 'gcr.io/shipshape_releases/service:local' locally
Pulling repository gcr.io/shipshape_releases/service
time="2015-06-29T14:54:32-07:00" level=fatal msg="Tag local not found in repository gcr.io/shipshape_releases/service"
E0629 14:54:32.873106 51943 shipshape.go:218] HTTP client did not become healthy: exit status 1

However, $docker images shows the images are there:
REPOSITORY TAG
gcr.io/shipshape_releases/android_lint latest
android_lint local
gcr.io/shipshape_releases/service latest
service local
debian wheezy
gcr.io/_b_dev_containers/cloud-dev-java prod

Am I wrong with any of these the steps?

Synced to commit dcafb8b from origin/master and using the 0.1.0 release of Bazel, I get errors like:

$ bazel build //...
........
____Loading package: third_party/openjdk
ERROR: $HOME/git/shipshape/third_party/go-glog/BUILD:19:6: syntax error at '"//tools/build_rules/go"': First argument of load() is a path, not a label. It should start with a single slash if it is an absolute path..
ERROR: $HOME/git/shipshape/third_party/go-glog/BUILD:19:6: file '/tools/build_rules/go.bzl' was not correctly loaded. Make sure the 'load' statement appears in the global scope, in the BUILD file.
ERROR: $HOME/git/shipshape/third_party/go-glog/BUILD:21:1: name 'go_library' is not defined.
ERROR: package contains errors: third_party/go-glog.
ERROR: error loading package 'third_party/go-glog': Package 'third_party/go-glog' contains errors.
____Elapsed time: 0.740s

It looks like this check was recently added to Bazel:
bazelbuild/bazel@45deb33

I tried converting the load statements thusly:

find . -name BUILD -print0 | xargs -0 sed -i 's/"\/\/tools\/build_rules\//"\/tools\/build_rules\//'

but that leads to a whole different kind of error:

$ bazel build //...
____Loading package: tools
____Loading package: shipshape/util/httpencoding
____Loading package: shipshape/test/dind/docker
____Loading package: shipshape/java/com/google/shipshape/util/rpc
____Loading package: shipshape/jenkins_plugin/src/main/java/com/google/shipshape/jenkins
____Loading package: third_party
ERROR: $HOME/.bazel/base_workspace/third_party/BUILD:183:12: Label '//third_party:gson/gson-2.2.4.jar' crosses boundary of subpackage 'third_party/gson' (have you deleted third_party/gson/BUILD? If so, use the --deleted_packages=third_party/gson option).
ERROR: $HOME/.bazel/base_workspace/third_party/BUILD:231:12: Label '//third_party:guava/guava-18.0.jar' crosses boundary of subpackage 'third_party/guava' (have you deleted third_party/guava/BUILD? If so, use the --deleted_packages=third_party/guava option).
ERROR: $HOME/.bazel/base_workspace/third_party/BUILD:313:12: Label '//third_party:guava/guava-testlib.jar' crosses boundary of subpackage 'third_party/guava' (have you deleted third_party/guava/BUILD? If so, use the --deleted_packages=third_party/guava option).
ERROR: package contains errors: third_party.
ERROR: error loading package 'third_party': Package 'third_party' contains errors.

The linter binary should be at /android-sdk-linux/tools/lint, but it's disappearing during the Docker build:

The extractor image we want to use lives here:
https://github.com/google/kythe/tree/master/kythe/java/com/google/devtools/kythe/extractors/java/maven

The Shipshape CLI fails every other time due to this error:

0625 20:40:50.884702 1312 shipshape.go:227] Error making service call: received an error from calling run: service needs to be called with triggered categories and/or a repo root with a valid .shipshape file with the event manual

May just be that the health check needs to wait longer.

I followed the instructions for the Jenkins plugin and failed to run it in jenkins.

Jenkins version: https://github.com/junit-team/junit.git

Console output:

Started by user anonymous
Building in workspace /var/jenkins_home/jobs/JUnit/workspace
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/junit-team/junit.git # timeout=10
Fetching upstream changes from https://github.com/junit-team/junit.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress https://github.com/junit-team/junit.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision fe1753703f4f20718b3479dd71e01e9e4d96cf1f (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fe1753703f4f20718b3479dd71e01e9e4d96cf1f
 > git rev-list fe1753703f4f20718b3479dd71e01e9e4d96cf1f # timeout=10
[Shipshape] Verbose mode enabled
[Shipshape] Using workspace path: /var/jenkins_home/jobs/JUnit/workspace
[Shipshape] Using output path: /var/jenkins_home/jobs/JUnit/workspace
[Shipshape] Using categories: ExtendJ
[Shipshape] Using socket: unix:///var/run/docker.sock
[Shipshape] Creating Shipshape request ...
FATAL: com/google/protobuf/ProtocolStringList
java.lang.NoClassDefFoundError: com/google/protobuf/ProtocolStringList
    at com.google.jenkins.plugins.analysis.ShipshapeSlave.constructShipshapeRequest(ShipshapeSlave.java:179)
    at com.google.jenkins.plugins.analysis.ShipshapeSlave.call(ShipshapeSlave.java:154)
    at com.google.jenkins.plugins.analysis.ShipshapeSlave.call(ShipshapeSlave.java:47)
    at hudson.FilePath.act(FilePath.java:1076)
    at com.google.jenkins.plugins.analysis.AnalysisRunner.perform(AnalysisRunner.java:118)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:761)
    at hudson.model.Build$BuildExecution.build(Build.java:203)
    at hudson.model.Build$BuildExecution.doRun(Build.java:160)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
    at hudson.model.Run.execute(Run.java:1741)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:374)
Caused by: java.lang.ClassNotFoundException: com.google.protobuf.ProtocolStringList
    at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1375)
    at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1325)
    at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1078)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 14 more
Finished: FAILURE

Currently the Jenkins build halts if the Shipshape plugin has any problems, such as not being able to find the specified analyzers.

It would be nice to have an option to let a Jenkins build continue even if the Shipshape Jenkins plugin has problems. The option could be specified with a checkbox named for example "Stop build on error" or "Stop if analysis fails".

When I run just the Shipshape service using docker I get the following errors:

$ docker run --rm --link extend_0 --name=test gcr.io/shipshape_releases/service:local /bin/bash                                                                                                                      
/endpoint.sh: line 25: /shipshape-output/shipshape.shipping_container.log: No such file or directory
/endpoint.sh: line 18: /shipshape-output/shipshape.go_dispatcher.log: No such file or directory
/endpoint.sh: line 19: /shipshape-output/shipshape.java_dispatcher.log: No such file or directory
/endpoint.sh: line 20: /shipshape-output/shipshape.javac_dispatcher.log: No such file or directory

Is this a problem? Is everything still working as it should?

I have downloaded Docker, Shipshape, Blaze, and try to follow the instructions from "Building" and forward. I fail on the last step in "Run with Local Docker Images":

$ ./shipshape/test/end_to_end_test.sh --tag local
INFO: Detailed output will appear in end_to_end_test.log
INFO: Running with locally built containers
INFO: Building and deploying //shipshape/docker:service locally ...
INFO: Running command [bazel build //shipshape/docker:service]
Error response from daemon: could not find image: no such id: service:local

I tried to build and run the Jenkins plug-in. After building and uploading the plug-in I was able to configure a build step (PRE_BUILD stage), but forcing a build results in the following error:

Started by user anonymous
Building in workspace /var/lib/jenkins/jobs/Test JS/workspace
[Shipshape] Creating Shipshape request ...
FATAL: com/google/protobuf/ProtocolStringList
java.lang.NoClassDefFoundError: com/google/protobuf/ProtocolStringList
    at com.google.jenkins.plugins.analysis.ShipshapeSlave.constructShipshapeRequest(ShipshapeSlave.java:179)
    at com.google.jenkins.plugins.analysis.ShipshapeSlave.call(ShipshapeSlave.java:154)
    at com.google.jenkins.plugins.analysis.ShipshapeSlave.call(ShipshapeSlave.java:47)
    at hudson.FilePath.act(FilePath.java:1078)
    at com.google.jenkins.plugins.analysis.AnalysisRunner.perform(AnalysisRunner.java:118)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
    at hudson.model.Build$BuildExecution.build(Build.java:205)
    at hudson.model.Build$BuildExecution.doRun(Build.java:162)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
    at hudson.model.Run.execute(Run.java:1744)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:374)
Caused by: java.lang.ClassNotFoundException: com.google.protobuf.ProtocolStringList
    at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1376)
    at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1326)
    at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1079)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 14 more
Finished: FAILURE

This seems not to be specific to my setup

Add a CLI command to list the available analyzer categories. See issue #63 for additional use cases.

Right now, the CLI assumes it runs on a directory. Also allow a single file. Amin, do you need to operate on dirty buffers? That will be more difficult to support.

I ran some tests with a third-party analyzer for extendj using:

$ bazel-bin/shipshape/cli/shipshape --categories="ExtendJ"
--analyzer_images="extendj_service/extendj"
--tag=local

Then I ran the end-to-end-test.sh and got this error in the test log file:
...
Pulling complete
I0629 14:48:52.256346 13531 shipshape.go:382] Analyzers pulled
I0629 14:48:52.256384 13531 shipshape.go:413] Waiting for dockerized analyzers to start up...
I0629 14:48:52.302231 13531 shipshape.go:394] Found no analyzer container (android_lint_0) to reuse for gcr.io/shipshape_releases/android_lint:
I0629 14:48:52.323912 13531 shipshape.go:399] Failed to stop android_lint_0 (may not be running)
I0629 14:48:52.323982 13531 docker.go:131] Running 'docker [run -p=127.0.0.1:10010:10005 -v=/tmp/shipshape-tests:/shipshape-workspace -v=/tmp:/shipshape-output --name=android_lint_0 -d gcr.io/shipshape_releases/android_lint:]'
I0629 14:48:52.683436 13531 shipshape.go:403] Could not start gcr.io/shipshape_releases/android_lint: at localhost:10010: exit status 1, stderr: Error response from daemon: Cannot start container c1efd504d306bbe4ec09f19fbe0eaa7f36780d930a2275fa18e5fbebe1eaa675: Bind for 127.0.0.1:10010 failed: port is already allocated
...

The extendj run started a third-party analyzer container on 10010 and then the second run with a different third-party analyzer container fails because the port is already bound.

We should have a more dynamic way of picking ports. We could test until a port is free in an interval of ports.

At https://github.com/google/shipshape#downloading-and-running-the-cli , would it be possible to give an example of using a third-party analysis? It's not clear what to pass as the --analyzer_images argument nor how to determine the legal values for the --categories argument.

If docker is not installed, make an appropriate output line, rather than logging to stderr and failing to analyze.

running on a go file I get this:

WARNING: Analyzer go vet failed to run: go vet gave incorrectly formatted issue: "can't load package: "

We need to add an RPC that analyses can fill in about where to file bugs/complaints about results. Perhaps this just returns a URI for the issue tracker of the analyzer.

We might also want it to return an email for contacting the authors directly.

I followed the simple installation instructions at https://github.com/google/shipshape .

Then, when I ran Shipshape, I got

% shipshape --categories="go vet,JSHint,PyLint" /home/mernst/research/types/checker-framework
E0523 08:21:41.998751 35994 shipshape.go:327] Error from pull: exit status 1
[run -p=127.0.0.1:10007:10007 -v=/home/mernst/research/types/checker-framework:/shipshape-workspace -v=/tmp:/shipshape-output -e="START_SERVICE=true" -e="ANALYZERS=localhost:10005,localhost:10006" --name=shipping_container -d gcr.io/shipshape_releases/service:prod]E0523 08:21:42.108346 35994 shipshape.go:213] HTTP client did not become healthy: exit status 1

Unfortunately, I cannot interpret this output.

We should add Checkstyle so that we have a Java based linter.

I noted there were no tagged releases for ShipShape yet. I suggest using Semantic Versioning. Tagging releases is useful for people who want to depend on ShipShape.

Similarly to: #33

The long-lived service initializes the analyzers. If a new run requires a new analyzer, it will not actually be started up by the service. Instead, if an analyzer is specified that does not exist, we should tear down the service and restart it.

Our endpoint script doesn't handle SIGTERM properly. This means that when we run docker stop, we get a SIGTERM, disregard it, wait until the timeout, then receive the SIGKILL.

Workaround: we can run with docker stop -t 0, which will immediately SIGKILL. Fine for now since we don't have any cleanup, but we should probably do this correctly...

Lots of info about this:
The original discussion on github: moby/moby#3240
Docker and PID 1: http://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/
Exec v. shell form (we are using exec form): https://docs.docker.com/reference/builder/#entrypoint
Other projects that ran into this, and their solutions:
moby/moby#2436
https://gist.github.com/zba/27d9e54e7293c1bb2da4
http://veithen.github.io/2014/11/16/sigterm-propagation.html (<-----we may need this one)
moby/moby#3766

Most people seem to solve this problem by (1) using exec mode when starting their service (we already do that) and (2) if the entrypoint is a shell script, using exec when starting their service there. This however, will only work if you have a single service you are starting. We have several.

I suspect this means we need to actually trap the signal in our shell script.

I am trying to set up a Gradle build for a standalone analyzer. It's difficult to figure out which dependencies are needed just from looking at the shipshape build. Also, some of the dependencies do not seem to be available as Jars. Some things we can do to better support other build systems are:

• Document all dependencies - artifact names and versions
• Make sure dependencies are available on a Maven repository
• Provide an example for using e.g. Gradle

When running the end to end test, we are getting calls for ErrorProne, but not actually getting any results.

We should add go lint support.

./shipshape --categories="JSHint" shipshape-dir2
WARNING: Analyzer Driver setup failed to run: chdir /shipshape-workspace/2: no such file or directory
E0626 16:20:47.110433 28327 shipshape.go:227] Error making service call: received an error from calling run: chdir /shipshape-workspace/2: no such file or directory

It would be nice if the Shipshape Jars needed to build a separate analyzer were available on Maven. That should simplify building other analyzers with other build systems.

I got the following email from Yu Lin.

I can't build shipshape anymore (my plugin works fine before this upgrade), it gives the following error:

____Loading...
____Loading complete. Analyzing...
____Found 171 targets...
blaze: Entering directory /home/yulin/.cache/bazel/_bazel_yulin/5f4d978cf0eb54a79459fe5f5f58e11b/s2/' ____[0 / 38] BazelWorkspaceStatusAction stable-status.txt ____From ProtocJava shipshape/proto/shipshape_config_proto_java_src.srcjar: /bin/bash: line 4: jar: command not found ERROR: /home/yulin/Downloads/s2/tools/build_rules/genproto.bzl:127:5: error executing shell command: 'set -e rm -rf bazel-out/local_linux-fastbuild/genfiles/shipshape/proto/shipshape_config_proto_java_src.srcjar.srcs mkdir bazel-out/local_linux-fastbuild/genfiles/shipshape/proto/shipshape_config_pr...' failed: bash failed: error executing com\ mand (cd /home/yulin/.cache/bazel/_bazel_yulin/5f4d978cf0eb54a79459fe5f5f58e11b/s2 && \ exec env - \ /bin/bash -c 'set -e rm -rf bazel-out/local_linux-fastbuild/genfiles/shipshape/proto/shipshape_config_proto_java_src.srcjar.srcs mkdir bazel-out/local_linux-fastbuild/genfiles/shipshape/proto/shipshape_config_proto_java_src.srcjar.srcs bazel-out/local_linux-fastbuild/bin/third_party/proto/protoc --java_out=bazel-out/local_linux-fastbuild/genfiles/shipshape/prot\ o/shipshape_config_proto_java_src.srcjar.srcs shipshape/proto/shipshape_config.proto jar cMf bazel-out/local_linux-fastbuild/genfiles/shipshape/proto/shipshape_config_proto_java_src.srcjar -C bazel-out/local_linu\ x-fastbuild/genfiles/shipshape/proto/shipshape_config_proto_java_src.srcjar.srcs . rm -rf bazel-out/local_linux-fastbuild/genfiles/shipshape/proto/shipshape_config_proto_java_src.srcjar.srcs'): com.google.devto\ ols.build.lib.shell.BadExitStatusException: Process exited with status 127: bash failed: error executing command (cd /home/yulin/.cache/bazel/_bazel_yulin/5f4d978cf0eb54a79459fe5f5f58e11b/s2 && \ exec env - \ /bin/bash -c 'set -e rm -rf bazel-out/local_linux-fastbuild/genfiles/shipshape/proto/shipshape_config_proto_java_src.srcjar.srcs mkdir bazel-out/local_linux-fastbuild/genfiles/shipshape/proto/shipshape_config_proto_java_src.srcjar.srcs bazel-out/local_linux-fastbuild/bin/third_party/proto/protoc --java_out=bazel-out/local_linux-fastbuild/genfiles/shipshape/prot\ o/shipshape_config_proto_java_src.srcjar.srcs shipshape/proto/shipshape_config.proto jar cMf bazel-out/local_linux-fastbuild/genfiles/shipshape/proto/shipshape_config_proto_java_src.srcjar -C bazel-out/local_linu\ x-fastbuild/genfiles/shipshape/proto/shipshape_config_proto_java_src.srcjar.srcs . rm -rf bazel-out/local_linux-fastbuild/genfiles/shipshape/proto/shipshape_config_proto_java_src.srcjar.srcs'): com.google.devto\ ols.build.lib.shell.BadExitStatusException: Process exited with status 127. blaze: Leaving directory/home/yulin/.cache/bazel/_bazel_yulin/5f4d978cf0eb54a79459fe5f5f58e11b/s2/'
____Building complete.
____Elapsed time: 1.021s, Critical Path: 0.55s

It says "jar: command not found", but jar can be executed actually. Do you know why?

amshali got this warning:

shipshape --categories="JSHint" /tmp/test.js
WARNING: Analyzer Driver setup failed to run: chdir /shipshape-workspace/tmp: no such file or directory
E0611 17:59:09.468988 1095 shipshape.go:227] Error making service call: received an error from calling run: chdir /shipshape-workspace/tmp: no such file or directory

To run docker in docker we need to (at least) pass a --privileged flag. The suggested solution is to add a --dind flag to the shipshape cli.

Interestingly, when running the end to end test, AndroidLint does believe that it is giving back 9 notes and 0 errors.

LOG_FILE[/tmp/shipshape.android_lint.log]:
-- Starting server endpoint at ":10005"
2015/09/16 23:15:29 called with: shipshape_context: <
file_path: "TicTacToeLib/AndroidManifest.xml"
file_path: "TicTacToeLib/LICENSE"
file_path: "TicTacToeLib/README.google"
file_path: "TicTacToeLib/README.txt"
file_path: "TicTacToeLib/_index.html"
file_path: "TicTacToeLib/default.properties"
file_path: "TicTacToeLib/res/drawable/lib_bg.9.png"
file_path: "TicTacToeLib/res/drawable/lib_circle.png"
file_path: "TicTacToeLib/res/drawable/lib_cross.png"
file_path: "TicTacToeLib/res/layout/lib_game.xml"
file_path: "TicTacToeLib/res/layout-land/lib_game.xml"
file_path: "TicTacToeLib/res/values/strings.xml"
file_path: "TicTacToeLib/src/com/example/android/tictactoe/library/GameActivity.java"
file_path: "TicTacToeLib/src/com/example/android/tictactoe/library/GameView.java"
file_path: "pom.xml"
file_path: "src/main/java/com/google/shipshape/App.java"
file_path: "test.js"
repo_root: "/shipshape-workspace"

category: "AndroidLint"
2015/09/16 23:15:29 starting analyzing
2015/09/16 23:15:29 About to run analyzer: AndroidLint
2015/09/16 23:15:33 lint output is ""
2015/09/16 23:15:33 finished analyzing, sending back 9 notes and 0 errors

END[Logs from second CLI run for AndroidLint]
INFO: Checking analyzer results ...
ERROR: Wrong number of ErrorProne results, expected 2, found 0
ERROR: Wrong number of AndroidLint results, expected 8, found 0

Shipshape seems to be sending requests to analyze symlinks to my standalone analyzer. When I run shipshape on itself it sends requests to analyze bazel-bin (shouldn't this be ignored in .shipshape?), and tools/defaults. These are both symlinks in the shipshape directory, so maybe they did not get copied over to the docker volume.

Is this intended, or should Shipshape ignore symlinks instead?

As pulling the container and starting up the service is slow, we should have a CLI mode where we just pull and start the container (and any specified analyzers).

amshali reports about a 5 second latency to run, even when analyzing a file in the same directory. We should make sure this is mostly/entirely the analyzer and that there aren't obvious performance problems in shipshape.

The command names "lint", "jshint" and "pylint" and hard-coded in the analyzer code, which means they must be installed and on the PATH. This isn't a problem when we use the container images, but androidlint_test fails locally if the Android SDK isn't on the PATH. (jshint and pylint would have the same problem, if they had tests. ;-).)

The paths to the analyzers should be configurable somehow, either via a flag or a local config file.

We deal with PHP a lot. Can we please have this linter?
https://github.com/SublimeLinter/SublimeLinter-php

The fields in the Jenkins plugin configuration should have some help text, as in other plugins that have small question icons next to each configuration field.

We now have two tests of the CLI written in shell, which just gives me shivers.

Let's actually create a test harness of the CLI in go. We also should add a test for running on a single file, and for the streams mode (if we decide to keep it around).

We have an example with AndroidLint, but we should create tutorial notes for SPLASH that give a step-by-step process for adding a hello-world analyzer.

It would be nice if the Jenkins plugin configuration page had a list of the currently available analyzers. Each analyzer could be represented with a checkbox and the user could enable/disable an analyzer by toggling the checkbox.

Currently the shipping container needs to be started with links to containers it needs to talk to. We would like for it to become aware of new third-party containers without having to restart it. For instance, a new analyzer could be registered via the CLI or running AnalyzerServices could be scanned for on localhost.

One possible direction: have a third-party dispatcher running outside the container and let that tunnel requests to third-party containers.

Kythe has moved over so we should be able to do so as well.

Some notes from schroederc:

Context: http://kythe.io/phabricator/T29

Kythe has custom rules, unsupported by the Bazel team, to build proto and Go. They should be relatively easy to get working in the Shipshape repo since they are derived from the campfire rules. See https://github.com/google/kythe/tree/master/tools/build_rules/ for the Skylark code.

Before starting up, we need to check the volume that is mounted to the shipshape directory. If the files we need to analyzer are not contained within that volume, we need to tear it down and restart it.

We run the risk of always restarting if the user keeps changing which directory they analyze. We could possibly keep multiple copies of the container open in this case. I'm going to start by just restarting when it doesn't match, and we'll see how that works in practice.

We should have a markdown file, perhaps on the main page, that lists the built-in analyzers and shows the docker URI and categories for known third-party analyzers.

Cody attempted to run shipshape using the latest version. He ran it first on kythe/go/platforms and it worked. Then he ran it on kythe/go and it fails because it can't cd into the right directory.

Right now, we're only supporting maven builds. Kythe now also supports using bazel.

CONTRIBUTING.md talks about picking a reviewer from the Phabricator users list, but I can not find a list of Phabricator users.

Repro steps:
docker stop shipping_container
docker rm shipping_container
./shipshape/test/end_to_end_test.sh --tag local
./shipshape/test/end_to_end_test.sh --tag local

Notice the difference in output from the last two commands.

When I run the CLI now I get some docker errors, but they are only visible when I add the flag --stderrthreshold=INFO:

$ bazel-bin/shipshape/cli/shipshape --categories="ExtendJ" --analyzer_images=gcr.io/extendj_service/servic
e:latest --tag=local --stderrthreshold=INFO .                                                             
I0626 10:31:02.948521   89407 shipshape.go:150] Starting shipshape using gcr.io/shipshape_releases/service:local on /usr/local/google/home/joqvist/git/shipshape
I0626 10:31:02.948578   89407 shipshape.go:413] Waiting for dockerized analyzers to start up...
I0626 10:31:03.169525   89407 shipshape.go:392] Reusing analyzer gcr.io/extendj_service/service:latest started at localhost:10010
I0626 10:31:03.169611   89407 shipshape.go:415] Analyzers up
I0626 10:31:03.169631   89407 shipshape.go:281] Starting shipshape...
I0626 10:31:03.210703   89407 shipshape.go:292] Restarting container with gcr.io/shipshape_releases/service:local
I0626 10:31:03.210752   89407 shipshape.go:361] Stopping and removing shipping_container
I0626 10:31:03.234766   89407 shipshape.go:426] stderr:
Error response from daemon: no such id: shipping_container
Error: failed to stop containers: [shipping_container]
I0626 10:31:03.234826   89407 shipshape.go:365] Could not stop shipping_container: exit status 1
I0626 10:31:03.234909   89407 docker.go:166] Running 'docker [run -p=127.0.0.1:10007:10007 -v=/usr/local/google/home/joqvist/git/shipshape:/shipshape-workspace -v=/tmp:/shipshape-output -e="START_SERVICE=true" -e="ANALYZERS=localhost:10005,localhost:10006,localhost:10008" --name=shipping_container -d gcr.io/shipshape_releases/service:local]'
I0626 10:31:04.186082   89407 shipshape.go:426] stderr:
Unable to find image 'gcr.io/shipshape_releases/service:local' locally
Pulling repository gcr.io/shipshape_releases/service
Tag local not found in repository gcr.io/shipshape_releases/service
E0626 10:31:04.186159   89407 shipshape.go:218] HTTP client did not become healthy: exit status 1

When I omit the --stderrthreshold flag I only get the very undescriptive error message HTTP client did not become healthy: exit status 1