skroutz / mistry Goto Github PK
View Code? Open in Web Editor NEWGeneral-purpose build server
Home Page: https://engineering.skroutz.gr/blog/speeding-up-build-pipelines-with-mistry/
License: GNU General Public License v3.0
General-purpose build server
Home Page: https://engineering.skroutz.gr/blog/speeding-up-build-pipelines-with-mistry/
License: GNU General Public License v3.0
Right now, on $ mistryd -h
we only show the default filesystem adapter ("plain"). We should somehow display all the available ones.
It is hard to setup, not going to be used in production environments and overall rsync is a better solution. The only think we decided to add scp
was for use in tests and as a more easy-to-setup default than rsync. However, it seems that it's not so easy since it still requires setting up SSH keys and also required more configuration to work as we want it (as opposed to rsync).
If scp doesn't provide the benefits we chose it for, we should drop it completely. For this, #52 should be done first so that we have an alternative (ie. running tests with rsync without having to setup an rsync daemon).
This might be beneficial when documentation gets bigger and we need page hierarchies and stuff.
Handle non-existing ready and pending directories of jobs so that the index page doesn't break.
This can happen in cases where the BootstrapProject
hasn't finished the directories creation when the index handler polls for jobs.
https://github.com/skroutz/mistry/wiki/File-system-layout
We should mention latest symlinks (with or without groups) in the aforementioned wiki page.
We should add 3-5 use cases for mistry. Some ideas: rake assets:precompile
, bundle install
, yarn install
. We could be creative here π
There's an existing work in branch use-cases
.
There are already some test helpers like assertEq
, assert
, failIfError
, assertNotEq
.
This issue is about standardizing their usage such as removing duplicate behavior or extracting them to a proper package.
build_result.json
.docker build
logs (#37)agis:~/dev/go/src/github.com/skroutz/mistry [master] $ ./mistry-cli build --project simple --target nonexistingpath
agis:~/dev/go/src/github.com/skroutz/mistry [master] $ ls nonexistingpath
nonexistingpath
agis:~/dev/go/src/github.com/skroutz/mistry [master] $ file nonexistingpath
nonexistingpath: empty
Expected: nonexistingpath
should be created as a directory, and the build artifacts should be inside it.
Currently, if data_path
is a symlink, docker on Mac complains with:
$ ./mistry-cli build --host localhost --project simple
2018/04/12 13:21:47 Error creating job: Error building {project=simple params=map[] group= id=3dc923b}: work: could not start docker container; Error response from daemon: invalid mount config for type "bind": bind source path does not exist
, http code: 500
This is due to docker/for-mac#1298.
We should resolve symlinks after reading the configuration, and update it accordingly before booting the server.
I think that it makes more sense not to cache build failures, most of the time those would be transient errors that should be retried next time someone asks for that build.
In that case we might want to keep the build directory (in a failed
subdirectory?) for debugging purposes.
We already added the job params inside BuildResult{}
so it might make sense to add the whole job.
If jobs for a new project (that hasn't been bootstrapped yet) happen to arrive at the same time, one of the (2) calls to BootstrapProject()
might fall due to an "directory exists" error.
We could introduce some locking when bootstrapping a project.
This could be a --force
option. This needs to implemented in JobRequest{}
too.
As it stands the rsync module is hardcoded to be "mistry":
module := "mistry"
The client should be able to set this option.
We could report if the build was cloned from a previous build (ie. build cache was used) or not. This will be useful for debugging purposes.
BuildInfo.Err
is supposed to contain build error messages, if any. At the moment, we do not populate it. We should, when possible (eg. when a docker build or docker run fails).
Right now, we get a bunch of json lines. We could perhaps https://github.com/moby/moby/blob/master/pkg/streamformatter/streamformatter.go to parse them.
This should make it feasible to prune mistry-specific containers on boot etc.
Error message:
$ ./mistry build --host localhost --port 8462 --project simple_with_directory \
--group foo --target /tmp/foo --transport scp -- --x=$RANDOM
/tmp/mistry: Is a directory
The project used should produce artifacts with a directory structure depth equal or bigger than 3.
e.g.
files
βββmore_files
βββ foo.js
This layer (let's suppose it is a Storage
package) will expose functions like:
func (s *Storage) GetJobs(project string) ([]Job, error)
and
func (s *Storage) GetJob(project, id string) (Job, error)
This will abstract the storage solution away and will enable more intuitive interactions in regard with fetching saved jobs.
Right now, we remove containers in StartContainer()
after they run successfully. However, it might happen, for example in case of a mistry failure while a container was running (eg. SIGKILL
), that containers are left around with nobody to remove them.
We should decide on how to treat such cases.
This could be a configuration option in the json.
Document how to use incremental building (ie. groups). This should be mentioned briefly in the README but explained thoroughly in a new wiki page.
These would demonstrate the complete flow and would contain real flags. For example:
$ mistry-cli --project rails --group staging --target /tmp
# ... server output
# ... synced artifacts
It should document how one can install the package and how to use it.
Sometimes, it's necessary to just trigger a job and not wait for it to finish.
A common scenario is when someone wants to eagerly prepare a build that knows is going to be needed in the future (eg. triggering a build from a github webhook to precompile the assets of a Rails application).
The server should support this option first, and then it has to be implemented in the client.
Some times builds are timing out: https://travis-ci.org/skroutz/mistry/jobs/362986195
In some scenarios, you want the artifacts fetched from mistry to completely override the previous contents of the --target-path
. To support such use cases, we could add a --replace
(or replace-target
) boolea option that, if enabled will completely replace the given path.
In the case of rsync, it would just pass the --delete
option to rsync.
There should be a strategy for somehow pruning old builds. This could be in the form of a "retention period" configuration.
This is necessary, otherwise disk usage will grow indefinitely.
Right now, there are no limits enforced on how much time a job could take. This might lead to cases where clients wait on a never-ending job, that also consumes resources (ie concurrency
/backlog
configuration options).
For example, this happens on some of our projects when a network issue occurs between our internal network and rubygems (ie. fastly CDNs), resulting in CI builds getting stalled indefinitely.
There should be a hard limit as to how much time a job may take so that we safeguard against such situations. Ideally, it should be both a global configuration that applies across all projects AND a per-project configuration (not sure how this should be specified though).
Right now, the client will block indefinitely if mistry is down (connect timeout) or a job takes forever to complete (read timeout).
However, there are cases when this is not acceptable. We should have the option to set a read timeout and a connect timeout on the client. Perhaps we only should expose this as a single timeout (how much the whole command should run for).
$ ./mistry-cli build --host localhost --project bar -- --foo bar
[foo]
panic: runtime error: index out of range
This requires putting both of these main packages under a cmd
directory (which is a Go convention).
We should handle errors that the user cannot do something about and somehow segregate them from user-land errors.
For example, an container failed to build because another container with the same ID already exists, is not something the user fix.
This could look like this:
$ mistry build --project foo --stream-output
Building zxc...
Building abc...
When using the btrfs filesystem, the mod time of the subvolumes that have been snapshotted is the mod time of the first subvolume. This creates an ambiguity when running stat
or ls -al
.
We should update the mod time to reflect the actual mod time of the directory.
https://golang.org/pkg/os/#Chtimes may help mitigate this.
The following naming scheme seems more natural:
mistryd
mistry
This needs not affect the repository/project name, only the generated binaries.
The user should be able to specify for example flags passed to rsync and also the source directories.
This information exists in the BuildInfo struct as of #47.
The job marshalling should also be updated accordingly.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.