weaveworks / build-tools Goto Github PK
View Code? Open in Web Editor NEWCollection of build & test tools shared by various Weaveworks projects
License: Other
Collection of build & test tools shared by various Weaveworks projects
License: Other
Many of these files:
build-tools/build/golang/Dockerfile
Lines 28 to 38 in dedde7f
can be replaced by https://github.com/golangci/golangci-lint
Otherwise we never do it properly
Upon merging #129 the build failed:
...
# cd .; git clone https://go.googlesource.com/lint /go/src/golang.org/x/lint
Cloning into '/go/src/golang.org/x/lint'...
fatal: remote error: Git repository not found
package golang.org/x/lint: exit status 128
The command '/bin/sh -c go get -tags netgo github.com/FiloSottile/gvt github.com/client9/misspell/cmd/misspell github.com/fatih/hclfmt github.com/fzipp/gocyclo github.com/gogo/protobuf/gogoproto github.com/gogo/protobuf/protoc-gen-gogoslick github.com/golang/dep/... github.com/golang/lint/golint github.com/golang/protobuf/protoc-gen-go github.com/kisielk/errcheck github.com/mjibson/esc github.com/prometheus/prometheus/cmd/promtool && rm -rf /go/pkg /go/src' returned a non-zero code: 1
make: *** [golang/.uptodate] Error 1
....
As used in weaveworks/weave#2843
I expect some tuning would be required.
We often have the problem that we don't know where the source code for an image lives.
There's a more-or-less standard way of providing this information, using Docker labels. We should do that.
e.g. https://github.com/weaveworks/weave/blob/master/prog/weaver/Dockerfile.template#L7-L13
LABEL maintainer "Weaveworks Inc <[email protected]>"
LABEL \
org.label-schema.name="Weave Net" \
org.label-schema.description="Weave Net creates a virtual network that connects Docker containers across multiple hosts and enables their automatic discovery" \
org.label-schema.url="https://weave.works" \
org.label-schema.vcs-url="https://github.com/weaveworks/weave" \
org.label-schema.vendor="Weaveworks"
There are errors on the app engine dashboard that look like this:
RuntimeError: Failed to get running builds for repository "weaveworks/wks". URL: https://circleci.com/api/v1/project/weaveworks/wks?circle-token=, Status code: 404. Response: {"message":"Project not found"}
The integration test runs have been moved from wks
-> wksctl
My understanding is:
However, the tools in build-tools can depend on specific versions of things. Often, spellcheckers and linters introduce new checks in new versions, breaking existing code.
It would be ideal if the build-tools tools could be provided as standard in the build images. That way, we wouldn't need to subtree this repo in, and we could be sure that the dependencies were the right ones for the tools.
Take a look at a recent Weave CI test, in particular the output produced by the
$ cd $SRCDIR/test; eval $(./gce.sh hosts); export COVERAGE=true; ./run_all.sh (0)
test step.
It contains the entire output twice. Careful examination shows that it is literally copied, i.e. the tests have in fact only been run once - otherwise reported timing would differ.
The same happens in Scope CI.
I've gone back all the way to our first successful CI run, and the same problem is visible there.
So I do wonder whether this is in fact a recent Circle CI display breakage.
Example: there are 25 tests run, but only 23 of the "finished with success" messages.
>>> Running ./800_plugin_cross_hosts_2_test.sh on [host1-4339-0.us-central1-a.positive-cocoa-90213 host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Running ./190_no_fastdp_2_test.sh on [host3-4339-0.us-central1-a.positive-cocoa-90213 host4-4339-0.us-central1-a.positive-cocoa-90213]
>>> Running ./685_proxy_weave_run_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./685_proxy_weave_run_test.sh finished with success after 7.4 secs
>>> Running ./680_proxy_hostname_derivation_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./800_plugin_cross_hosts_2_test.sh finished with success after 11.6 secs
>>> Running ./165_reset_2_test.sh on [host1-4339-0.us-central1-a.positive-cocoa-90213 host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./190_no_fastdp_2_test.sh finished with success after 17.7 secs
>>> Running ./150_connect_forget_2_test.sh on [host3-4339-0.us-central1-a.positive-cocoa-90213 host4-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./165_reset_2_test.sh finished with success after 19.6 secs
>>> Running ./105_frag_2_test.sh on [host1-4339-0.us-central1-a.positive-cocoa-90213 host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./150_connect_forget_2_test.sh finished with success after 19.8 secs
>>> Running ./100_cross_hosts_2_test.sh on [host3-4339-0.us-central1-a.positive-cocoa-90213 host4-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./680_proxy_hostname_derivation_test.sh finished with success after 35.6 secs
>>> Running ./670_proxy_tls_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./670_proxy_tls_test.sh finished with success after 6.7 secs
>>> Running ./666_abuse_of_start_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./100_cross_hosts_2_test.sh finished with success after 22.3 secs
>>> Running ./655_proxy_large_http_chunks_test.sh on [host3-4339-0.us-central1-a.positive-cocoa-90213]
>>> Running ./640_proxy_restart_reattaches_test.sh on [host4-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./666_abuse_of_start_test.sh finished with success after 11.6 secs
>>> Running ./630_proxy_dns_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./655_proxy_large_http_chunks_test.sh finished with success after 6.1 secs
>>> Running ./620_proxy_entrypoint_command_test.sh on [host3-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./640_proxy_restart_reattaches_test.sh finished with success after 16.4 secs
>>> Running ./604_proxy_docker_py_test.sh on [host4-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./620_proxy_entrypoint_command_test.sh finished with success after 11.5 secs
>>> Running ./601_proxy_docker_py_test.sh on [host3-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./630_proxy_dns_test.sh finished with success after 18.2 secs
>>> Running ./500_weave_multi_cidr_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./105_frag_2_test.sh finished with success after 50.4 secs
>>> Running ./295_dns_large_response_test.sh on [host1-4339-0.us-central1-a.positive-cocoa-90213]
>>> Running ./280_with_or_without_dns_test.sh on [host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./295_dns_large_response_test.sh finished with success after 9.0 secs
>>> Running ./270_use_name_as_hostname_test.sh on [host1-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./280_with_or_without_dns_test.sh finished with success after 10.3 secs
>>> Running ./265_dns_reverse_test.sh on [host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./265_dns_reverse_test.sh finished with success after 13.1 secs
>>> Running ./250_dns_negative_test.sh on [host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./250_dns_negative_test.sh finished with success after 7.3 secs
>>> Running ./225_dns_subdomain_test.sh on [host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./270_use_name_as_hostname_test.sh finished with success after 21.7 secs
>>> Running ./195_no_multicast_route_1_test.sh on [host1-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./500_weave_multi_cidr_test.sh finished with success after 34.6 secs
>>> Running ./170_ip_reuse_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./225_dns_subdomain_test.sh finished with success after 11.0 secs
>>> Test ./170_ip_reuse_test.sh finished with success after 33.8 secs
>>> Test ./604_proxy_docker_py_test.sh finished with success after 81.8 secs
>>> Ran 25 tests, all succeeded
build-tools is a collection of shell, Python, Go, and Terraform scripts that represent build tooling common to Weaveworks projects, proprietary and open source.
It is currently embedded in each of our projects via git subtree or git submodules.
Both git subtree and git submodules are problematic. They are unfamiliar commands to most git users, which introduces a level of friction with both new & existing hires.
When we make an improvement to build-tools, we face the problem of manually identifying all repositories which use it and then filing a PR one by one. This becomes more and more difficult the more we expand.
Writing robust shell code is hard, even with shellcheck. Writing shell code that communicates intent is even harder.
wbuild
with subcommands that make all of the required build and lint functionality availableWhile projects should just fetch the latest build-tools, we should ensure that each master commit has a downloadable binary so that projects that need to pin to older versions can.
We have a collection of Terraform files for provisioning test machines.
This should be moved to Weave Net repository. If making it public is impractical, then it should move to its own private repository.
So CircleCI can parse them. I don't know how useful this would be, but it seems fairly straighforward.
I think this would involve extending the summary
function in runner/runner.go
Some guidance to the desired format here
flake8 and https://github.com/google/yapf
It was removed upstream in moby/moby#25354
We should probably use cobra instead, just like docker did.
... At some point (in another PR) we should also get rid of gvt since it has being obsoleted by go modules.
Similar to #108 but this time do it in the right place.
This should fix many of the test flakes on 'defunct' processes
See #42 and golang/lint#246
Related: #40
since I managed to waste a couple of hours not spotting I had an extra space, which showed up as a "did not find expected key" response from kubectl
./tools/lint .
ansible/ansible-helper: run shfmt -i 4 -w ansible/ansible-helper
ansible/ansible-playbook-helper: run shfmt -i 4 -w ansible/ansible-playbook-helper
ansible/playbooks/roles/bootstrap/files/ansible_prereqs.sh: run shfmt -i 4 -w ansible/playbooks/roles/bootstrap/files/ansible_prereqs.sh
ansible/update_inventory: run shfmt -i 4 -w ansible/update_inventory
ansible/update_kubernetes_certs: run shfmt -i 4 -w ansible/update_kubernetes_certs
ansible/update_master_unschedulable: run shfmt -i 4 -w ansible/update_master_unschedulable
infra/common.sh: run shfmt -i 4 -w infra/common.sh
infra/ssh-master: run shfmt -i 4 -w infra/ssh-master
infra/ssh-minion: run shfmt -i 4 -w infra/ssh-minion
Got a bunch of errors, but exit code was 0.
shfmt v1.0.0 changes =
to ==
:
# git diff
diff --git a/tools/lint b/tools/lint
index 15a97d4..566e329 100755
--- a/tools/lint
+++ b/tools/lint
@@ -173,7 +173,7 @@ lint() {
coverage.html) return ;;
esac
- if [[ "$(file --mime-type "${filename}" | awk '{print $2}')" = "text/x-shellscript" ]]; then
+ if [[ "$(file --mime-type "${filename}" | awk '{print $2}')" == "text/x-shellscript" ]]; then
ext="sh"
fi
# shfmt --version
v1.0.0
This breaks the weaveworks/service-conf build
If the tests have failed, the scheduler GC keeps deleting the instances, because the build status is listed as timedout
(or w/e).
To tell when the ssh-enabled build is cancelled/timedout, we should also check for "why": "ssh"
, and maybe something else.
API Call for reference: https://circleci.com/api/v1/project/weaveworks/weave
Since python2 is EOL, we should port ./scheduler to python3.
These lines:
build-tools/build/golang/Dockerfile
Lines 25 to 27 in dedde7f
are not necessary since Go added a per-target build cache, and probably don't achieve anything
since we typically bind-mount another directory for the build cache when compilations are run: https://github.com/weaveworks/prom-aggregation-gateway/blob/c4415bbe1411d77eb4132f3df51c87cf7fcca840/Makefile#L60
To prevent things like #38 from breaking builds again
These lines:
build-tools/build/golang/build.sh
Lines 12 to 20 in dedde7f
can be removed if we run the build as the desired user. For example see weaveworks/scope@833f947
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.