nuvolaris / nuvolaris Goto Github PK
View Code? Open in Web Editor NEWNuvolaris is as a distribution of the Apache OpenWhisk serverless engine that runs in every Kubernetes
License: Apache License 2.0
Nuvolaris is as a distribution of the Apache OpenWhisk serverless engine that runs in every Kubernetes
License: Apache License 2.0
Each runtime in OpenWhisk has a basic test suite written in scala like this in go.
Those tests start a runtime image, intialize it and then run a set of validation tests. We need to replace those basic tests with test written in Javascript with Jests as running them with Scala is awkward and slow (to say the least).
So contribute to the nuvolaris-testing repo a test suite written in jest that remplements the basic nodejs tests.
Every file with code requires a proper license header.
Furthermore we should add a test in CI that check all the files have a proper license header.
nuv wsk
is supposed to use ~/.nuvolaris/.wskprops
by setting the environment variable WSK_CONFIG_FILE
before executing the body of wsk
.
It does not look it works.
Furthermore in the ~/.nuvolaris/.wskprops there is APIHOST=https://
and should be http
Contribute an ansible script to the nuvolaris-testing repo that builds in a test server accessible with ssh an OKD Kubernetes cluster to be usable as a test environment. It should export the kubernetes cluster access port and make available the key in /etc/kube/config, so with an scp server:/etc/kube/config ~/.kube/config
is possible to retrieve it and access the cluster with kubectl
This is the issue number 1 and it is symbolic as usually you close it when you reach the goal of your project. It will stay open until we reach 1.0.
Requirements for Nuvolaris 1.0:
curl -sL get.nuvolaris.org | kubectl apply -f -
Contribute to the nuvolaris-controller a demo using CouchDB
Modify this demo.
In detail, copy the demo folder to demo-couchdb, add to deploy the scripts to launch couchdb from here, replace the configuration file in the image in /home/etc/standalone-kcf.conf
with a configmap so, to task deploy
create a cluster using couchdb.
Add a command version
that prints the value of CLI_VERSION
.
Currently the database initialization is executed by an ansible scripts run by a job. This is very slow to run and consumes memory.
We want a python module (nuvolaris/couchdb_init.py) in the nuvolaris-operator
that performs the same task in python.
In order to implement follow those instructions
task start-couchdb
Now you can execute task init-couchdb
that is currenly an example only showhing how to reuse templates and connect to couchdb
You have to re-implement in python (using requests and jinja2) the logic in the ansible script orig/initdb.yml
Note that all the templates used by those scripts are available in nuvolaris/templates
.
All the secrets should come from an YAML (that can be modified) so put them in tests/operator-obj.yaml
The script should include a simple command to launch and destroy, and a configuration file to create and destroy an AKS cluster in Kubernetes to test nuvolaris.
Please follow this guide
Every action need to receive environment variables to find his mongodb database.
This task requires implemeting the needed changes to OpenWhisk so a database configured by the configurator will be accessible by the corresponding actions.
Write a './setup.sh' that will install
and check prerequisites
Add the devcontainer support, build a kubernetes cluster and run a basic operator in it.
Document development procedure.
Contribute an ansible script to the nuvolaris-testing repo that builds in a test server accessible with ssh a Kubernetes witk MicroK8S cluster to be usable as a test environment. It should export the kubernetes cluster access port and make available the key in /etc/kube/config, so with an scp server:/etc/kube/config ~/.kube/config
is possible to retrieve it and access the cluster with kubectl
Fork apache/openwhisk then create a top level nuvolaris-controller that pull the repo and builds and publish in github docker registry. Builds and publish standalone OpenWhisk, to be used by the controller.
This issue should inagurate the in nuvolaris-controller repo.
In order to provide an environment that works for everyone build an image to be used as a development environment.
The image should work on Intel and ARM 64.
Furthermore, customize the environment of the various subprojects to work with Run in Container feature of VSCode.
If you try to use the subcommands nuv kind --name
or nuv wsk action invoke -r
you cannot because the first level wsk is eating the parameters so you have to use nuv wsk -- action -r
.
This should not be necessary and it should be possible to do nuv wsk ... -r
or nuv kind ... --name
witouth the --
Contribute to the nuvolaris-controller a demo using RedPanda
Modify this demo.
In detail, copy the demo folder to demo-redpanda, add to deploy the scripts to launch kafka from here, replace the configuration file in the image in /home/etc/standalone-kcf.conf
with a configmap so, to task deploy
create a cluster using redpanda.
It may be necessary to fix OpenWhisk as it could not be supported! But simply open the bugs in this case, do not fix them.
Contribute bash script to the [nuvolaris-testing]
(https://github.com/nuvolaris/nuvolaris-testing) folder cloud/k3s
a script that creates in AWS a multi-node cluster based on k3s.
The script should be unattended and assume keys for AWS are in environment variables.
Note there is the k3sup
tool that apparently creates a single node k3s cluster. We do now know what is the best way to setup a k3s based cluster.
Crete devcontainers, setup go and create a basic cli that will start the operator
The configurator currenty has an user interface that writes on mongodb, creating users and associating a number of namespaces to it
We need to:
change it to write in couchdb, creating proper namespaces and users in couchdb
there is not an user concept in openwhisk, only namespaces and keys, so we will have to add our user table
Implements the requirement
https://github.com/nuvolaris/nuvolaris-cli/blob/main/README.md#static-frontend
Nuv must scan the folders and create an yaml file the static contents.
Format of the YAML:
namespace: <namespace>
path: <path>
resources:
<file1>: <body1>
<file2>: <body2>
...
Where the <fileN>
is the relative path of under the various web
folders.
The <bodyN>
file is the base64 encoded of the corresponding file.
Those files will be crd to be deployed to Kubernetes.
Contribute to the nuvolaris-controller a demo using Kafka-Kraft
Modify this demo.
In detail, copy the demo folder to demo-kafka, add to deploy the scripts to launch kafka from here, replace the configuration file in the image in /home/etc/standalone-kcf.conf
with a configmap so, to task deploy
create a cluster using couchdb.
It may be necessary to fix OpenWhisk as it could not be supported! But simply open the bugs in this case, do not fix them.
Contribute to the nuvolaris-controller a demo using MongoDB
Modify this demo.
In detail, copy the demo-couchdb folder to demo-mongodb, add to deploy the scripts to launch also mongodb replace the configuration file in the image in /home/etc/standalone-kcf.conf
with a configmap so, to task deploy
create a cluster using couchdb.
It may be necessary to fix OpenWhisk as it could not be supported! But simply open the bugs in this case, do not fix them.
Currently .github actions just runs on every push, building an image for each push.
The image is named neo-latest.
We should improve to do that when releasing (tagging) and use the tag to build the image.
So far we found the following pain point that should be documented properly in the developer guide
Nice to have. nuv setup --devcluster
creates the devcluster with a single command and deploy it...
Add a basic subproject using svelte and tailwind and run it with devcontainers.
Change hello in ciao in nuvolaris-hello
Use nuvolaris-hello
Select and deploy an ingress controller that works in kind
(and others) to publish static resources and include to the operator.
Add it to the testing repo
Each runtime in OpenWhisk has a basic test suite written in scala like this for python.
Those tests start a runtime image, intialize it and then run a set of validation tests. We need to replace those basic tests with test written in Javascript with Jests as running them with Scala is awkward and slow (to say the least).
So contribute to the nuvolaris-testing repo a test suite written in jest that reimplements the basic nodejs tests.
The operator must deploy the database couchdb
It will receive a CRD with all the credentials from the CLI.
OpenWhisk has configuration file that includes database access credentials.
You need to deploy couchdb and openwhisk with the parameters to access the database.
So the operator must:
The outcome is a design document to be implemented:
The cli setup
currently only checks that the operator is up and running.
Howevr the operator starts the controller and in turns initializes some action.
We need to add a check that waits until openwhisk is ready to answer and a check that deploying functions works.
The check that waits until openwhisk is ready can be done executing wsk namespace
until it does not return an error.
Then it should deploy a simple action and execute it.
A simple action can be function main(args) { return { "body":"hello"} }
and the operation to execute are
wsk action create hello hello.js
until it succeedswsk action invoke hello
returns helloI was playing around with the Dockerfile and noticed a small oversight.
In the Dockerfile for the remote container, the path used with dpkg to install delta is missing the path "/tmp/".
line 65: sudo dpkg -i "$FILE" ; rm "/tmp/$FILE"
The installation fails because it's not finding the file (the error is also visible in the build logs in github).
The operator should receive CRDs of type Static and deploy them.
Deployment should happen creating a configmap from the crd, then an http server that will expose the content to the ingress controller.
Deployment should happen writing the content to s3-compatible buckets, since Kubernetes objects (hence CRDs) are limited to 1MB so they are not fit for deploying static content.
There will be an s3 bucket for each user.
There will be a nuv s3 put command able to write in that bucket.
All we need to do is to generate the taskfile that will invoke that command after buinding.
Each runtime in OpenWhisk has a basic test suite written in scala like this in node. They customize this
Those tests start a runtime image, intialize it and then run a set of validation tests. We need to replace those basic tests with test written in Javascript with Jests as running them with Scala is awkward and slow (to say the least). The protocol a runtime implements is described here
So contribute to the nuvolaris-testing repo a test suite written in jest that remplements the basic nodejs tests.
It would be nice (as requirement) for ML datapipelines having the support inside nuvolaris for durable functions using as
backend (https://docs.temporal.io/blog/workflow-engine-principles/) for stateful function. This will leverage nuvolaris not only as
serverless but also as ETL platform since already we've Red Panda and other tools as a platform.
Add to the operator the ability to deploy MongoDB.
You should:
deploy/mongodb
(you can use the operator)note that the kind in lastest developer image already exposes the right port
running nuv devcluster create
will create a kind cluster
note that
Also
nuv devcluster destroy
will destroy it
please also add unit tests for the various functions and use the "dryrun" method so you can output the various commands generated, unit testing should only test the commands and the files generated are correct
if we need integration tests, let's add separately, i plan to use jest for integration tests (I mean checking that it actually created a custer and it works).
This task should stop at cluster creation, but to be complete we will have also to deploy the operator that we will do once I complete it
Creating a basic operator using kopf
that starts standalone Openwhisk, the one built with #2
We need to start the configurator.
The idea is:
We need a frontend adding users and namespaces.
Each user has an username in the form of an email and a password and can have a number of namespaces associated to it (at least one, but it can be more than one)
You should be able to add, remove, edit users and associated namespaces.
You can mock the API as you like as it will implemented later based on the prototype.
You can decide for the UI, but please use tailwind.
Implement the nuv setup
command that
wskprops
file as in the task wsk hereso you can start using nuv wsk
to create actions as in the task test here
Some notes
kubectl
but talk directly with the kubernetes client so nuv does not have any dependencies on kubectl
nuvolaris
namespace. If this fails the write an error message suggesting to use nuv devcluster
to create a local cluster firstAlso note that the example uses Kustomize to specify the current image to run. Since we do not want to depend on kubectl
and we will use kube descriptors only once, as once deployed the operator we do not interact any more with kubernetes but only with the operator, just create a static kubectl files in ~/.nuvolaris and apply it.
Finally the wskprops file is by default in ~/.wskprops but to make things and not conflict with wsk
and others, create one in ~/.nuvolaris/.wskprops. Then when running nuv wsk
we set the WHISK_CONFIG_FILE
environment to point to the right one.
In nuvolaris-cli
write a test (using jest) that
add the github actions to execute a test against
nuv setup --devcluster
We can assume those clusters are up-and-running, parameters to access them are stored as GitHub Secrets
After deployment nuv wsk action list
fails.
Looks like the WSK_CONFIG_FILE is set only when doing the setup.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.