gaia-pipeline / gaia Goto Github PK
View Code? Open in Web Editor NEWBuild powerful pipelines in any programming language.
License: Apache License 2.0
Build powerful pipelines in any programming language.
License: Apache License 2.0
... or other VCSes? Gaia doesn't actually do any version control; does it need to do anything more than what Go is capable of in the core toolset, of downloading source trees from multiple VCS sources? And if Gaia does need more capabilities, why not use something more flexible than go-git, such as SourceGraph's go-vcs?
We need a secure way to store passwords, tokens, api keys and everything that should be kept secret.
For example when the user wants to access vault via a pipeline. Usually a token is required to get access to vault and this token should be stored in the secret store.
After a secret has been stored, it must be possible to automatically inject this password during runtime in the pipeline as a job argument. This is not a problem at all, we just have to decide how the user can setup the relationship between a secret and a pipeline. That means, which secret should be available in which pipeline.
First option: In the details page create a new view to configure the pipeline. Add a subview to manage secrets.
Second and preferred option: Create a new view below settings. This view is for managing secrets. When a view is created or modified it can be linked to multiple pipelines. This has the advantage that one secret can be shared with multiple pipelines.
What do you think?
We need tests for the whole plugin package.
Please use the standard test framework for this: https://golang.org/pkg/testing/
In a multi-user environment it would be nice to have user permissions.
i.e only the admin can add/remove users.
It would also be useful to have pipeline permissions so the viewing/modification/triggering of a pipeline can be limited to a specific user or group of users.
This would also allow easy integration into SSO (single sign on) systems like OpenID Connect.
Details:
2018-08-15T21:47:24.501+0200 [DEBUG] Gaia: error during job execution: error="job failed" job="{2296842551 Job1 [] running}"
panic: send on closed channel
goroutine 13 [running]:
github.com/gaia-pipeline/gaia/scheduler.(*Scheduler).resolveDependencies(0xc420020f00, 0x89e70aca, 0x11e6932, 0x4, 0x0, 0x0, 0xc42000e2d0, 0x1, 0x1, 0x11ea8ba, ...)
.../golang/src/github.com/gaia-pipeline/gaia/scheduler/scheduler.go:376 +0x2f0
github.com/gaia-pipeline/gaia/scheduler.(*Scheduler).resolveDependencies(0xc420020f00, 0x8ae70c5d, 0x11e6936, 0x4, 0x0, 0x0, 0xc42000e2d8, 0x1, 0x1, 0x11ea8ba, ...)
.../golang/src/github.com/gaia-pipeline/gaia/scheduler/scheduler.go:358 +0x223
created by github.com/gaia-pipeline/gaia/scheduler.(*Scheduler).executeScheduler
...golang/src/github.com/gaia-pipeline/gaia/scheduler/scheduler.go:429 +0x2c9
Circle CI link: https://circleci.com/gh/gaia-pipeline/gaia/457?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link
We need tests for the whole store package.
Please use the standard test framework for this: https://golang.org/pkg/testing/
Currently, we have one docker image with golang integrated. With the integraten of #15 we now have to support Java too. Soon, we will also support Python (#68).
My proposal is to create one big docker image with the "latest"-Tag where all languages are included. This means when someone runs docker run gaiapipeline/gaia
without a specific tag, the latest tag will be pulled which can be huge (probably gigabytes).
To provide a workaround, I would like to offer other docker images like "gaia:go", "gaia:java", "gaia:python" where only this specific language is included.
Would like to hear you opinions. 🤗
Restarting the gaia process invalidates all JWT tokens. It seems like the jwt key is auto generated on startup instead of being passed in via a flag.
Smells:
scheduler_test.go:144:146
// Wait some time to pickup work and finish.
// We have to wait at least 3 seconds for scheduler tick interval.
time.Sleep(5 * time.Second)
This just failed with the error that the job is in an invalid status. I think this should be fortified or it will cause problems in the future. The scheduler needs to be tested in a different way, or manually kicked off at this stage, or somehow signal on a channel that it's done with its round.
The only way to delete a pipeline is to manually remove the binary in the pipeline folder.
Then a manual restart of gaia is required since the pipeline is not automatically removed from the active global pipeline list.
What we have to do:
During the pipeline check we should check if a pipeline has been removed. If so, we should remove it from the active pipeline list.
Right now, the git module can only be tested via actually cloning stuff.
I would like to avoid that, but my current PR is convoluted enough. :) I'm going to do that refactoring with in the confines of this ticket.
The go example is actually broken. When executed, the following error will occur:
2018-08-22T19:39:12.930Z [WARN ] Gaia: using auto-generated key to sign jwt tokens, do not use in production
2018-08-22T19:39:13.327Z [INFO ] Gaia: vault file doesn't exist. creating...
⇨ http server started on [::]:8080
2018-08-22T19:42:03.714Z [DEBUG] plugin: starting plugin: path=/data/pipelines/TestPipeline_golang args=[]
2018-08-22T19:42:03.719Z [DEBUG] plugin: waiting for RPC address: path=/data/pipelines/TestPipeline_golang
2018-08-22T19:42:03.746Z [DEBUG] plugin.TestPipeline_golang: panic: job 'DoSomethingAwesome' has dependency '' which is not declared
2018-08-22T19:42:03.746Z [DEBUG] plugin.TestPipeline_golang:
2018-08-22T19:42:03.747Z [DEBUG] plugin.TestPipeline_golang: goroutine 1 [running]:
2018-08-22T19:42:03.747Z [DEBUG] plugin.TestPipeline_golang: main.main()
2018-08-22T19:42:03.747Z [DEBUG] plugin.TestPipeline_golang: /data/tmp/golang/src/6a8ecca2-cdf8-4d3a-81de-28f28072d195/test.go:31 +0x104
2018-08-22T19:42:03.747Z [DEBUG] plugin: plugin process exited: path=/data/pipelines/TestPipeline_golang
2018-08-22T19:42:03.747Z [DEBUG] Gaia: cannot connect to pipeline: error="plugin exited before we could connect" pipeline="&{1 TestPipeline {https://[email protected]/scm/doc/gaia-pipelines.git bneil nope { } refs/heads/master [] /data/tmp/golang/src/6a8ecca2-cdf8-4d3a-81de-28f28072d195} golang /data/pipelines/TestPipeline_golang [95 153 137 156 178 5 50 123 168 0 59 4 149 100 40 11 139 61 107 19 123 141 63 200 234 82 0 252 50 241 90 190] [] 2018-08-22 19:42:00.8241888 +0000 UTC 6a8ecca2-cdf8-4d3a-81de-28f28072d195}"
2018-08-22T19:42:03.748Z [DEBUG] Gaia: cannot get jobs from pipeline: error="plugin exited before we could connect" pipeline="&{1 TestPipeline {https://[email protected]/scm/doc/gaia-pipelines.git bneil nope { } refs/heads/master [] /data/tmp/golang/src/6a8ecca2-cdf8-4d3a-81de-28f28072d195} golang /data/pipelines/TestPipeline_golang [95 153 137 156 178 5 50 123 168 0 59 4 149 100 40 11 139 61 107 19 123 141 63 200 234 82 0 252 50 241 90 190] [] 2018-08-22 19:42:00.8241888 +0000 UTC 6a8ecca2-cdf8-4d3a-81de-28f28072d195}"
When a new pipeline has been created and detected by Gaia, it usually tries to start the pipeline and get all jobs for the detailed overview: https://github.com/gaia-pipeline/gaia/blob/master/pipeline/ticker.go#L114
Sometimes this can fail when:
We should make this more resilient and inform the user when something goes wrong.
When a pipeline is triggered, every job will execute it's own pipeline process. This brings several disadvantages with it:
We need documentation about how to use java pipelines.
I was wanting to know more about how Gaia runs, is there anything that explains where state is saved, or if it is HA or has plans to be. I didn't see any links to some more info on where credentials for checking out git repos are saved and such. I like all the readme links that do exist but if there is link to further operation instructions I guess I missed it.
Thanks, sounds like a promising project.
After all the feedback, Java was the most wished one.
Let's develop an SDK for it and support it in Gaia.
We already have a Settings
subpage. We need a new tab-pane for managing pipelines.
Maybe a table with all currently active pipelines and every entry has buttons/icons to modify (renaming) and removing this pipeline.
What needs to be done:
We need tests for the whole handlers package.
Please use the standard test framework for this: https://golang.org/pkg/testing/
We need tests for the whole scheduler package.
Please use the standard test framework for this: https://golang.org/pkg/testing/
When triggering a pipeline it would be nice to be able to specify input parameters. This would allow users to have dynamic configuration for their pipelines.
An example would be a pipeline that runs database migrations and it needs an input of the database name.
Not sure if this is considered or is part of the featureset that you want to support.
In my experience, in real systems, it's often unrealistic to except existing software to be modified or rewritten to be made compatible with a pipeline.
Perhaps such support can be added as a plugin or a pipeline wrapper.
Overall, it is just an idle query since it is still alpha software.
Test this.
Currently, Gaia uses unix sockets as communication channel for Gaia <-> Pipeline (Plugin) communication.
It has the following disadvantages:
It has the following advantages:
We need to change to real port communication. It also means that we have to generate certificates for the communication between Gaia and the Plugin.
Gaia Pipelines in python support.
Currently using the latest via
docker run -d -p 8080:8080 -v $PWD:/data gaiapipeline/gaia:latest
After generating a pipeline (no jobs yet run), i went to the dashboard and got the following error when trying to start pipeline
.
index.vue?258e:110 Uncaught TypeError: Cannot read property 'length' of undefined
at s.checkPipelineArgs (index.vue?258e:110)
at s.n [as checkPipelineArgs] (vue.runtime.esm.js:190)
at click (index.vue?7896:65)
at t (vue.runtime.esm.js:2000)
at HTMLAnchorElement.e._withTask.e._withTask (vue.runtime.esm.js:1798)
checkPipelineArgs () {
// check if this pipeline has args
for (let x = 0, y = this.pipeline.jobs.length; x < y; x++) { <--
if (this.pipeline.jobs[x].args && this.pipeline.jobs[x].args.type !== 'vault') {
// we found args. Redirect user to params view.
this.$router.push({path: '/pipeline/params', query: { pipelineid: this.pipeline.id }})
return
}
}
// No args. Just start pipeline.
this.startPipeline()
},
this.pipeline.jobs
- jobs
is currently null which is causing the issue
I can see that there is a circleci integration but consider adding that to the PR as a hook so it can't be merged before everything is passing.
This will help shorten git_test.go
by at least 5-7 seconds.
I understand there's not much to gain from this, but in the spirit of going native, I propose this project moves from Make to Mage.
Mage is a native Go-based replacement where you can actually write real code for your make functions rather than scripting a series of shell commands.
More at https://github.com/magefile/mage
I would be prepared to produce most (if not all) of this work if you're interested.
When the create pipeline process is triggered, there is no visual feedback for the user. He needs to wait until the create pipeline history
table has been updated.
We should add a notify message to inform the user like we always do. For example https://github.com/gaia-pipeline/gaia/blob/master/frontend/client/views/vault/index.vue#L298
At the moment the priority is just a ordering.
Other than that I really like this approach.
You could solve all this by just embeding NATS and using choreography over the NATS pub sub.
Then a job can be subscribed to another job ending.
You can also get sharing etc between components I side pipelines easily.
NATS over grpc is a well trodden path.
If you need some links to show what I mean just ask . I can really see me using this for iOS but also for applications
Anyway hope you don't mind me making suggestions
We need integration-tests to make sure the frontend is working but also to be sure that frontend and backend is behaving correctly.
Cypress (https://www.cypress.io/) looks promising but I'm also open for other options.
Don't use package level storeService var
If all of our constructors and functions take their dependencies explicitly, then we no longer have any use for globals. Instead, we can construct all of our database connections, our loggers, our resource pools, in our func main, so that future readers can very clearly map out a component graph.
https://peter.bourgon.org/blog/2017/06/09/theory-of-modern-go.html
instead we could use something like in a simpler manner:
// main.go
package main
import (
"os"
"github.com/boltdb/bolt"
"github.com/gorilla/mux"
)
type BoltDBInstance interface {
Open(path string, mode os.FileMode, options *bolt.Options) (*bolt.DB, error)
}
type server struct {
storeDB BoltDBInstance
router *mux.Router
}
func main() {
server := server{}
storeDB, err := bolt.Open("store.db", 0600, nil)
if err != nil {
// do something
}
server.storeDB = storeDB
server.routes()
}
// routes.go
func (s *server) routes() {
// define routes
}
if you @michelvocks argue with this I can make this happen then testing handlers would be much easier.
please offer java(mvn) example ,thanks
Do you have plans to use turn Gaia into a Master <-> Worker system where jobs can be run on remote servers?
Hello @michelvocks
Right now there is no way to edit the pipeline code without having to manually remove the binary and set things up all over again.
Are you planning on supporting something like this in the future?
Thanks!
Dep didn't remove my periodically added forks and added them to the vendor folder.
We should remove this folder: https://github.com/gaia-pipeline/gaia/tree/master/vendor/github.com/michelvocks
We need tests for the whole pipeline package.
Please use the standard test framework for this: https://golang.org/pkg/testing/
When deploying gaia it would be nice to be able to set the configuration flags via environment variables.
For ease of development it may be useful to use a cli/configuration system like https://github.com/spf13/viper or https://github.com/alecthomas/kingpin
Hi.
I'm trying to start contributing to this project and committing the vendor folder isn't a great way to start things off because they collide most of the time.
That's why the dep Lock file is there. When you start developing, you basically check out the project and go dep ensure
. The lock file will make sure that all the right versions of the packages are used.
I can understand the committing vendor seems like a good idea because then you'll have the right code always with you, but makes for a poor developer experience.
Feel free to ignore this if you think otherwise, it's fine. :)
When I run go test
a bunch of things are left under pipeline
that pollute the work tree.
~/golang/src/github.com/gaia-pipeline/gaia makefile_refactor* 10s
❯ git status
On branch makefile_refactor
Your branch is up-to-date with 'skarlso/makefile_refactor'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
pipeline/.circleci/
pipeline/.gitignore
pipeline/.goreleaser.yml
pipeline/Gopkg.lock
pipeline/Gopkg.toml
pipeline/LICENSE
pipeline/Makefile
pipeline/README.rst
pipeline/cmd/
pipeline/docker/
pipeline/frontend/
pipeline/gaia.go
pipeline/handlers/
pipeline/pipeline/
pipeline/plugin/
pipeline/scheduler/
pipeline/store/
pipeline/vendor/
The use of strings.Builder
requires Go 1.10+. Since this is only used in the test cases, it leaves an interesting situation where I'm able to run Gaia on Go 1.9, but while developing the tests fail. I think replacing the strings.Builder, which only serves as an io.Writer
for the logging would normalize the dev/test/prod environments.
Alternatively an update to the README under the Contributing section could clear this up.
Thoughts?
We use the tabs-view for vault but I don't see any reason to do that. WDYT @Skarlso ?
I would officially like to request support for PHP.
I understand this won't come for some time, but I want the maintainers to be aware of my interest in it.
Now that slack is avaliable we should put it into the readme.
~/golang/src/github.com/gaia-pipeline/gaia makefile_refactor 6s
❯ go test ./...
? github.com/gaia-pipeline/gaia [no test files]
? github.com/gaia-pipeline/gaia/cmd/gaia [no test files]
? github.com/gaia-pipeline/gaia/handlers [no test files]
ok github.com/gaia-pipeline/gaia/pipeline 8.290s
? github.com/gaia-pipeline/gaia/plugin [no test files]
ok github.com/gaia-pipeline/gaia/scheduler 0.016s
--- FAIL: TestUserAuth (0.01s)
store_test.go:153: crypto/bcrypt: hashedPassword is not the hash of the given password
FAIL
cannot remove data folder: remove data: directory not empty
FAIL github.com/gaia-pipeline/gaia/store 0.026s
This is a pin for later investigation. Osx, just doing go test ./...
.
I heard from a lot of people that the description (README file) is not very clear. Especially the term "pipeline" led to confusion. We should improve this soon to make clear what Gaia is and maybe how is it comparable to other projects (competitors).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.