jumppad-labs / jumppad Goto Github PK
View Code? Open in Web Editor NEWModern cloud native development environments
Home Page: https://jumppad.dev
License: Mozilla Public License 2.0
Modern cloud native development environments
Home Page: https://jumppad.dev
License: Mozilla Public License 2.0
Hi,
I wanted to check-in regarding any plans in the road map to support multi node k8s clusters using Shipyard. I am trying to test out a few things using Consul with k8s and would love to see how shipyard can help enable that setup.
The status
command currently returns JSON which is difficult to read and is not ordered. This is actually the raw working structure.
status
should have two modes
default: Human readable
json flag: JSON output
{
"blueprint": null,
"resources": [
{
"command": [
"consul",
"agent",
"-config-file=/config/dc1.hcl"
],
"depends_on": [
"network.dc1",
"network.wan"
],
"image": {
"Name": "consul:1.7.0",
"Password": "",
"Username": ""
},
"name": "consul_dc1",
"networks": [
{
"name": "network.dc1"
},
{
"name": "network.wan"
}
],
"status": "applied",
"type": "container",
"volumes": [
{
"destination": "/config/dc1.hcl",
"source": "/home/nicj/go/src/github.com/shipyard-run/blueprints/consul-gateways/consul_config/dc1.hcl"
}
]
},
{
"command": [
"consul",
"connect",
"envoy",
"-mesh-gateway",
"-register",
"-address",
"10.15.0.202:443",
"-wan-address",
"192.168.0.202:443",
"--",
"-l",
"debug"
],
Example new output:
Resource: Test
Type: Cluster
...options
``
Hey @nicholasjackson while the update message is useful for letting users know a new version of shipyard is available. The message takes a lot of real estate on stdout, especially if I am not supporting colors it looks jumbled.
Can you reduce this a bit?
A single line IMHO would suffice in letting me know a new version is available for me to download.
When shipyard pulls an image it should add it to a local image repo.
We can then run a command shipyard prune
which would clean up any images pulled by shipyard.
shipyard list: should show blueprints that are running
This would be a nice command to show all currently running blueprints. This should be backlogged, however, because at this time multiple running blueprints do not coexist well together.
Problem:
When working on a micro service developers often require the local version of the service to be connected to a full environment. While Shipyard provides some level of capability, spinning up large and complex applications is beyond the hardware capabilities of most development machines.
Generally the approach taken is that the service is deployed to the shared environment and tested externally.
While remote debuggers allow a connection from a local IDE like VSCode to a service running in a remote cluster, there are several problems:
Ideally it should be possible to route traffic to a locally running instance of a service from a remote cluster.
It should also be possible to route traffic based on HTTP metadata, etc specifically to the local service without impacting any other element of the shared environment.
Enabling testing
One core problem with testing is generally the inputs for the test cases. Test cases are often defined by known issues and functionality. Often bugs exist in application code due to missing functionality which was not defined. Since the test case is directly tied to functionality errors occur. To aid in the discoverability of bugs, a developer should be able to run the latest code with production inputs. This feature would enable the shadowing of traffic from a production environment to a local dev or test environment.
Proposed solution:
A new remote network resource which allows traffic to be routed to remote destinations
A new local service resource which allows local applications to be part of the shipyard stack
Functional overview
A central component (controller) within the remote network would enable tunnelling of connections between the remote dev machine and service traffic in the cluster.
The controller is API driven and runs as a single monolithic instance. When a dev environment connects to the controller it specifies which service it would like to masquerade as.
Specific to a Consul Service Mesh
For example we wish to send traffic destined for the remote products-api
service to the local dev instance.
The controller will register a "fake" service instance of the products-api
service with Consul, the advertise address for the service would be the address of the controller.
Since the Consul cluster would regard the "fake" instance as another service instance it will be able to take part in the service catalog and L7 routing.
The controller would automatically configure L7 routing for the service such as enabling traffic splitting or HTTP meta data based routing. This ensures that the fake service can be isolated from the normal network traffic.
The local client makes an outbound connection to the controller through the Consul Gateway using a valid mTLS certificate. Upon successful connection a persistent TCP connection is opened, this circumvents the need for the local machine to be remotely accessible (public IP, etc)
When traffic is sent controller and the controller will proxy this through the tunnel back to the dev environment.
To connect to the controller the dev environment will use a valid mesh mTLS certificate and use the remote gateway as an ingress.
Example Config
Below is an example of what the configuration might look like. It is assumed that the remote controller would be deployed separately.
// A remote_network defines a network which does not exist within the current machine
// this could be a service mesh such as Consul running on a remote cluster.
remote_network "consul-remote" {
endpoint = "192.179.231.1"
}
// make remote services accessible locally
// the service would be accessible using the FQDN `api-products-db.consul-remote.shipyard:5432`
// and the port
remote_ingress "product-api" {
target = "consul-remote" // remote network
service = "product-api"
port {
local = 4646
remote = 4646
host = 14646
}
}
// A local service defines an application component which is running on the local machine
// this can be used to proxy requests from the dev stack to local debug code
// if the network target is a remote_network then traffic from a remote cluster will be sent
// to the local instance
local_service "dev-products-http" {
name "api-products" //service name is the registered service name
network = "consul-remote"
local_port = 8080
// routing rules are optional however all elements have a combatinatination (invented a word) effect.
// i.e they are combined.
routing {
http_header {
key = "DEBUG"
value = "nicstest"
}
// send 10 percent of remote traffic to the local instance
traffic_split = 10
// a copy of the traffic is sent to the local machine, the original request
// also arrives at the normal destination, think TEE
shadow_traffic = true
}
}
Currently when you set the number of nodes to anything higher than 1, the health check for the cluster will fail because it is currently not possible to create multi-node clusters.
Defining a resource using:
foo.example.com
results in an error. I would expect I can use dot notation to define my resources. You can sanitize this during blueprint parse as you read in the resource names.
This was a major request for my team FYI.
Currently nomad runs as a single node, implement the capability to run multiple nodes for a cluster
Integrate shipyard with Jupiter Notebooks, this would be nice to integrate as a full resource.
jupyter "mynotebook" {
target = "container.mine"
}
Ref:
https://jupyter.org/
Ingress resources should be specific to the target i.e:
k8s_ingress "k8s-dashboard" {
target = "k3s"
service = "kubernetes-dashboard"
namespace = "kubernetes-dashboard"
port {
local = 8443
remote = 8443
host = 18443
}
}
container_ingress "nomad-http" {
target = "nomad"
port {
local = 4646
remote = 4646
host = 14646
}
}
nomad_ingress "nomad-products" {
target = "dev"
job = "product-api"
group = "product-api"
task = "product-api-service"
port {
local = 4646
remote = 4646
host = 14646
}
}
remote_ingress "product-api" {
target = "consul-remote" // remote network
service = "product-api"
port {
local = 4646
remote = 4646
host = 14646
}
}
Nomad allows a network namespace to be defined for a task group, this namespace does not allow ingress traffic from any other namespace. Traffic in the group can talk freely.
In order to enable Ingress when network namespaces are used we need a way to route from a host level to a particular application running in a network namespace.
Ideas:
Need to define a stanza to allow the running of Nomad jobs
nomad_job "something" {
cluster = "nomad.test"
}
We are on the fence about whether Shipyard should be a Terraform provider, after much deliveration we felt the workflow was not quite right.
There were things like wanting to be able to do shipyard ssh or having the capability to connect to remote stacks. This did not feel like it fit with the Terraform workflow and follow the Unix principles of a single tool for a single job.
That said, the architecture of the code was set out so that you could import Shipyard as a library and then create a provider from it.
At the moment our feeling is that you turn the Terraform provider on its head a little. We could see shipyard having a Terraform provider which allows you to execute Terraform config as part of the run workflow.
However we are totally open to this, I think the key thing is getting the sweet spot in the workflow, we would be lying if we said we were not heavily influenced by Vagrant and Terraform. We already have the need for a DAG and state, the provider model too should follow Terraform's practices of separate binaries, maybe.
I am following the guide at https://learn.hashicorp.com/consul/gs-consul-service-mesh/understand-consul-service-mesh#setup-a-kubernetes-environment-with-shipyard getting the following error. :(
curl https://shipyard.run/apply | \
bash -s github.com/shipyard-run/blueprints//learn-consul-service-mesh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 315 100 315 0 0 77 0 0:00:04 0:00:04 --:--:-- 77
Running configuration from: github.com/shipyard-run/blueprints//learn-consul-service-mesh
Error parsing README.md front: unknown delim
2020-06-05T12:52:27.182+0200 [DEBUG] Statefile does not exist
If a Blueprint file specifies one or more web browsers to be opened, the current logic is that on Darwin open
is used. On Linux
xdg-open
is used.
// After apply runs the following browser windows are opened (does not open them if run with --headless)
browser_windows = [
"http://localhost:18200",
"http://localhost:18080",
"http://localhost:18443",
]
Need to implement more robust logic round checking for these programs, also need to implement vanilla Windows version.
When using WSL 2 on windows xdg
open` needs to be installed and a browser such as chrome need to be aliased as a bash command.
Define an ingress resource which allows external access to a task running in Nomad
nomad_ingress "nomad-products" {
target = "nomad_cluster.dev"
job = "product-api"
group = "product-api"
task = "product-api-service"
port {
local = "http"
remote = "4646"
host = "14646"
}
}
Process:
Limitations:
I would like to see a blueprint validate so errors can be caught before run. Validate could also be used in CI as I upload blueprints to github
shipyard validate
Improve Unit test coverage by testing unhappy paths.
Make tests more generic and re-usable.
Refactor config testing.
This is currently underway, I am refactoring the Docker SDK into a more task oriented API. For example creating a container requires 4 calls to the Docker API. Testing the providers which create containers is incredibly verbose because of this.
By creating a Task Driven API the providers will depend on this rather than the Docker client API. When testing the Task Driven API can be mocked meaning way cleaner tests. There will be a full test suite separately for the Task Driven API.
The Task driven API will also abstract the implementation of Docker from the provider.
This way we can replace Docker with ContainerD or PodMan, etc.
Ensure all file paths are generalised across all operating systems to ensure that Windows (non WSL/2) and Unix based systems are supported.
For clarification, this relates to Shipyard running in windows cmd prompt with Docker desktop. Shipyard works well with WSL2.
Ability to install Shipyard from a curl command
curl http://shipyard.run/install | bash
All in one install and apply
curl http://shipyard.run/apply | bash http://github.com/blueprints/something
Let's try and trap and send the appropriate exit code.
For instance. On error, I should not see exit code zero and would expect 1 or another code depending on the exact issue that was encountered.
When creating a container, should only pull from the remote docker registry if the tagged version of the image does not exist in the local registry. Need to think of a way to override this behavior.
Hello again! I tried to shipyard run github.com/shipyard-run/blueprints//consul-docker
and got the following panic.
Running configuration from: github.com/shipyard-run/blueprints//consul-docker
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x19cd8d3]
goroutine 1 [running]:
github.com/shipyard-run/shipyard/pkg/clients.(*GetterImpl).Get(0xc00015a430, 0x7ffd524691a6, 0x31, 0xc0006cc0c0, 0x51, 0x1, 0x1)
/home/runner/work/shipyard/shipyard/pkg/clients/getter.go:80 +0xc3
github.com/shipyard-run/shipyard/cmd.newRunCmdFunc.func1(0xc0001cf180, 0xc00015a4f0, 0x1, 0x1, 0x0, 0x0)
/home/runner/work/shipyard/shipyard/cmd/run.go:93 +0x23ab
github.com/spf13/cobra.(*Command).execute(0xc0001cf180, 0xc00015a4c0, 0x1, 0x1, 0xc0001cf180, 0xc00015a4c0)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:826 +0x460
github.com/spf13/cobra.(*Command).ExecuteC(0x3806f60, 0x0, 0x1dd4f20, 0xc0000b4058)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
github.com/shipyard-run/shipyard/cmd.Execute(...)
/home/runner/work/shipyard/shipyard/cmd/root.go:92
main.main()
/home/runner/work/shipyard/shipyard/main.go:10 +0x52
shipyard version
Shipyard version: 0.0.33
Currently it is not possible to build this application using Go 1.14.0. This is due to an upstream problem with the Helm package v3.1.1.
The application will build, however when running a blueprint with a Helm chart, the run will fail.
E.g.
2020-03-01T10:57:43.033Z [ERROR] Unable to apply blueprint: error="Error running chart: template: consul/templates/client-daemonset.yaml:121:23: executing "consul/templates/client-daemonset.yaml" at <(.Values.client.join) and (gt (len .Values.client.join) 0)>: can't give argument to non-function .Values.client.join"
Go 1.13.8 does not have this problem.
While it's easy enough to view logs of the individual docker containers that are provisioned using shipyard. I am finding myself wishing for a log view similar to docker-compose.
shipyard logs - should show all logs for the stack like docker-compose up
shipyard logs <container_resource> - should show logs for that resource.
Hello! I use vscode with a custom dark theme, and when I open shipyard project for the first time it changed an Activity bar and a Status bar color. I think color settings for vscode can be safely rremoved from .vscode/settings.json.
Current Behavior
Currently it is possible to configure the blueprint by adding a browser_windows
section in the .yard
file.
browser_windows = [
"http://localhost:18443",
"http://localhost:18500",
]
This works fine in cases where the blueprint is created for a complete scenario (aka all applications that need to be exposed are started with the first shipyard run ...
).
Extended behavior
I'd like a way to configure the browser_windows
behavior inside the ingress
object:
ingress "consul-ui" {
target = "k8s_cluster.k8s"
service = "svc/hashicorp-consul-ui"
# Defines whether to honor the `browser_window` param or not
headless = true
# Defines URL to open in browser after ingress is created
browser_window = "http://localhost:18500"
network {
name = "network.local"
}
port {
local = 80
remote = 80
host = 18500
}
}
ingress "consul-ui" {
target = "k8s_cluster.k8s"
service = "svc/hashicorp-consul-ui"
# can be honored globally by `shipyard` config
# local_env could potentially be expanded for new feature that leverage desktop integration
local_env {
# Defines whether to honor the `browser_window` param or not
# headless = true # having this too here might be an overkill
# Defines URL to open in browser after ingress is created
browser_window = "http://localhost:18500"
}
network {
name = "network.local"
}
port {
local = 80
remote = 80
host = 18500
}
}
Shipyard should be able to reference other blueprints and configuration sources. This would allow a modular approach to building environments.
When shipyard run
is executed Shipyard would include the referenced module in the list of files it is currently parsing. It would then create those resources as normal. If the reference is a remote repository then Shipyard would download it to the users local cache before execution.
module "consul" {
source = "./subfolder"
}
module "consul" {
source = "github.com/shipyard-run/blueprints//consul-nomad"
}
shipyard run
should validate the configuration beyond HCL2 validation.
Is the current process for Ingress the best approach?
We have thought about differing approaches for Ingress to move this into a single container, however
finding a balance between UX and something which can deliver the requirements is not easy.
While the current process is not hugely optimised it satisfies the requirements allows simple UX and consumes few resources.
Requirements:
Shipyard should have an exec command which allows interactive shell to a container, pod or nomad job.
The UX around the shell should have the following options:
shipyard exec [target] [job | pod] [task | container] -- [command: default shell]
# container in a Nomad job, run `ls`
shipyard exec cluster.nomad product-api api-service -- ls
# container in Kubernetes pod, default shell
shipyard exec cluster.nomad product-api api-service
# docker container, default shell
shipyard exec container.consul
Notes:
$ shipyard exec
# Which target would you like to use?
[1] Consul Container
[2] Kuberentes Cluster
[3] Nomad (cluster)
$ 3
# Would you like to access the node or a job?
[1] Node
[2] Job (default)
$ 2
# What job in the Nomad cluster would you like to use?
[1] Products API
[2] Front end service
$ 1
# Which container would you like to attach to?
[1] Products Service
[2] Envoy Sidecar
$ 1
# What command would you like to execute (press return for default shell)?
$ ls -lha
Notes:
$ shipyard exec con[tab]
Consul Cluster Consul Container Confluence Container
If I supply a command in which arguments are required. I'd expect help to be displayed on stdout vs doing nothing and exit which is currently what is being experienced.
The bash version of shipyard used to have a tools
command which ran an interactive terminal containing common tools such as HashiCorp command line tools and KubeCtl.
We need to bring this feature to Shipyard. The tools container should be run as follows.
shipyard tools
If an existing tools container is running then a shell to this container will be attached
If the tools container is not running then it will be started.
In addition we need to improve the user experience for the tools container.
At present you define the tools container as follows:
container "tools" {
image {
name = "shipyardrun/tools:latest"
}
command = ["tail", "-f", "/dev/null"]
# Nomad files
volume {
source = "./nomad_config"
destination = "/files/nomad"
}
network = "network.cloud"
env {
key = "NOMAD_ADDR"
value = "http://nomad-http.cloud.shipyard:4646"
}
}
Tools should be rolled into a custom resource called tools
and many of the following options should be automatically set:
When using the nomad_cluser.image
configuration resource, debug log lines are weirdly formatted with blank lines and trailing characters. It would be nice to clean this up to improve the UX.
2020-05-26T16:12:38.564+0200 [DEBUG] oaded image: consul:1.7.2
2020-05-26T16:12:39.446+0200 [DEBUG]
2020-05-26T16:12:39.839+0200 [DEBUG] 7Loaded image: gcr.io/google_containers/pause-amd64:3.0
2020-05-26T16:12:40.457+0200 [DEBUG]
2020-05-26T16:12:41.161+0200 [DEBUG] 2Loaded image: nicholasjackson/fake-service:v0.9.0
2020-05-26T16:12:41.469+0200 [DEBUG]
2020-05-26T16:12:46.624+0200 [DEBUG] :Loaded image: nicholasjackson/consul-envoy:v1.7.2-v0.14.1
2020-05-26T16:12:47.550+0200 [DEBUG]
2020-05-26T16:12:57.494+0200 [DEBUG] %Loaded image: prom/prometheus:latest
2020-05-26T16:12:57.608+0200 [DEBUG]
2020-05-26T16:12:58.437+0200 [DEBUG] *Loaded image: prom/statsd-exporter:latest
The main blueprint file should be deprecated in favour of a README.md markdown file which contains front-matter. Browser windows currently defined in the blueprint should be moved to the individual resources as defined in #43
It should be possible to distribute shipyard using Apt, Chocolatey, Brew, Snap, Windows Installer.
No CLI should be without an applicable man page. I'd want to check here first vs having to go online to shipyard's documentation.
Figure out UX for apply
Options 1 - Apply with a url
Automatically clones the blueprint and copies into $HOME/.shipyard/blueprints before running
yard apply https://github.com/blueprints/consul-routing
Does not add to blueprints folder just applies
yard apply ./local/folder
Simple no args show blueprints in $HOME/.shipyard/blueprints
yard apply
Which blueprint would you like to run
[1] Consul Service Mesh (blueprint tag)
[2] Vault (#vault)
[3] Consul HTTP Routing example (#consulhttp)
[r] Remote repo
[l] Local repo
Run specific blueprint using the tag
yard apply #consulhttp
Menu of blueprints from an archive
yard apply https://github.com/shipyard-run/blueprints
Which blueprint would you like to run
[1] Consul Routing
[2] Vault Kubernetes
Currently, there is not a way to have multiple blueprints coexist sanely. Blueprints should be namespaced or ordinal numbers added to their resource names in order to segregate and list them.
Make it exist
At present when downloading a blueprint from GitHub only the last directory is used in the Blueprint cache at $HOME/blueprints
.
├── consul-gateways
│ ├── consul_config
│ │ ├── api.hcl
│ │ ├── dc1.hcl
│ │ ├── dc2.hcl
│ │ └── web.hcl
│ ├── dc1.hcl
│ ├── dc2.hcl
│ ├── gateways.yard
│ └── network.hcl
├── examples
│ └── container
│ └── minimal
│ └── config.hcl
└── minimal
└── config.hcl
The full path of the github repo should be stored similar to how GOPATH is used, this is to avoid accidentally overwriting a blueprint.
Currently the code is implemented in:
https://github.com/shipyard-run/shipyard/blob/a3ef03f5b2a9ce77912483ad16050babba41e2b5/cmd/get.go#L59
However this function should be moved to the engine as when modules are implemented it will need to be used.
Modify the behaviour of Shipyard so that networks are namespaced [network].shipyard
.
This resolves an interesting failure to apply a blueprint as the user had an existing network with the same name.
Currently state and dependencies in Shipyard are very rudimentary. Ideally we need to provide capability for incremental application of resources.
Implement a driver for Kubernetes clusters which uses the Kind library
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.