bob-cd / bob Goto Github PK

Similar to #26, when a stop pipeline request is received in a multi node setup, its possible that the node is actually running the relevant container may not be the one that received the request.

Add descriptive paths to make values more readable.
eg for logs:

http://bob:7777/api/pipeline/logs/group/dev/name/test/run/1/offset/0/lines/50

• bootstrap bob to launch a container of the docker:dind image
• expose it to a random available port on the host
• use that connection to do all activities
• mount the db in that container too
• /gc routes will clear the container

Why hasn't this been done already? Duh! 🙄

Hammock for moving to a record per log format, easier to timestamp and stream logs.

How should builds which depend on specific OS features like MacOS/iOS builds which need OS specific build tools and features be done?
These are platforms which do not support Docker natively yet.

As a newbie and/or contributor to Bob, the user should be able to understand the rationale behind Bob, successfully navigate the code base and can easily contribute and improve the project.

Checklist:

• Prepare an architecture diagram
• Improve README to have more relevant info
• Prepare a gh-pages branch and have a proper web page
• Document code further

Use https://github.com/lispyclouds/clj-docker-client

• Create better messages
• Create better and relevant HTTP status codes

Currently due to the issue docker-java/docker-java#1245 in docker-java which is used by clj-docker-client which is the lib for Bob, its not possible to run Bob on JDK 13.

Release a new clj-docker-client when a fix is available and make Bob great again!

Since Postgres is the only source of state, Bobs can replicate. Figure out a way to create a cluster to load balance Bobs. Maybe needs a service discovery mechanism?

Implement Internal Resources as a way to signify the dependency of a pipeline to another.

Maybe implement this as an artifact fetch internally?

Also auto-trigger upstream pipeline(s)?

For example ~/.bob.conf so that Bob can be configured from outside before startup.

Think about the state management as the nodes won't have the pipeline in
question executing with them.

Check all sub systems on GET /api/can-we-build-it

Checks:

• Docker is reachable and anwsers positively on ping
• H2 is operational

this is to help possible contributors

Currently Bob executes the build on one of the many nodes in a build cluster. During a deployment this node will be taken down along with its potentially running build(s).

We need to come up with a way to continuously deploy new versions of Bob while not affecting the running builds.

• How would bob recover in case of one of the nodes going down during a build?
• What happens to the dangling build still in running state?
• If thats to be marked as failed, who should be doing that?
• More questions?

Since all of the core functionality of Bob is made as functions which are called on REST endpoints, they can be easily imported if Bob is to be used as an embedded engine in other Clojure programs.

Create a PUT endpoint to modify an already existing bob pipeline.

Most of Bob should have a decent test suite covering most of the cases (duh!)

This makes it fully data driven and will ease out the comms to docker and Java 13+ build issues.

Bob plugin requirements:

• Should expose a REST API
• Should have the following paths:
  • GET /bob_request: Params to be encoded as K-V pairs. Returns a
    resource (typically a zip file) with code 200 else 400.
  • GET /ping: Health check endpoint, should return 200 if fully functional.

Bob requirements

• Should expose a /plugin/register/name endpoint where the plugin
  registers itself with a POST request where name is a
  unique name without spaces and a JSON body having the following:

  {
    "url": "fully-recheable-url-with-protocol"
  }

  Bob should return 200 if successfully registered else 409 in case of conflict.
• Should expose a /plugin/unregister/name endpoint where the plugin
  registers itself with a POST request where name is a
  unique name without spaces. Bob should return 200 if successfully removed
  and this should be idempotent
• Should expose a /plugins endpoint answering a GET request which
  lists all registered plugins

• Annotate each step with a needs_resource key
• Fetch the resource by name when the step needs to be executed
• Rethink resource mount logic

Now that Bob core is stateless, the dangling images and containers can be safely removed to conserve (LOTS OF) space.

Implement password-only authentication for Bob. As an execution engine, Bob should not care about user management. But to provide one layer of security Bob may be configured with a list of passwords that can be used to access Bob's apis.

Currently Bob only supports a single Artifact Provider for simplicity reasons.

But in the case of federating Bobs, the need to multiple Artifact stores may arise as like in the case of the need to store artifacts in different places/regions/providers.

Learn some UI and/or ask someone for help in building a reference UI for Bob

Bob should be able to deliver code in a de-coupled way akin to fetching resource or uploading artifacts.

Specially for failure scenarios

Events like resource fetches and artifact pushes should be in the build logs for debugging both internal and external errors

Reitit offers data driven design, has much better performance and has better debugging experience. Check if Manifold works with it without Seppari? Alternatively use Yada, which should mostly work.

Parses

sort -t "\t" -k2 test > test-sorted

wrongly.

Should include

• steps
• environment vars
• current state denoted by number

• Add health checks for registered resource providers and artifact stores
• Maybe do an automatic heartbeat at an interval to all subsystems and log for failure?

Artifacts should be able to specify a relative path in the context of a resource.

• GET all pipelines
• Maybe filter based on names?
• Filter on status?
• Other filters?

This would make it easier for (new) users to quickly spin up a cluster on providers like AWS.

This addresses the issue of managing distributed state in a multi-node setup. Examples are the pausing/stopping pipelines in #26 and #59. This would allow Bobs to synchronize and notify one another for shared states and still keep the DB as the single source of shared state.

See https://gist.github.com/mikeball/ba04dd5479f51c00205f as an example to define a async handler for a trigger that notifies when a DB value changes.

Lay the foundation for the fully distributed peer network for Bob. Cassandra for inspiration??

• Perform health checks beforehand
• Handle errors including timeouts during resource fetch