Git Product home page Git Product logo

apache / openwhisk-wskdeploy Goto Github PK

View Code? Open in Web Editor NEW
76.0 41.0 75.0 28.77 MB

Apache OpenWhisk utility for deploying and managing OpenWhisk projects and packages

Home Page: https://openwhisk.apache.org/

License: Apache License 2.0

Go 83.91% Makefile 0.20% JavaScript 11.86% Swift 0.41% Shell 1.52% Java 0.62% Python 0.52% PHP 0.27% Dockerfile 0.29% Ruby 0.23% Rust 0.18%
openwhisk apache continuous-delivery continuous-deployment deployment faas functions functions-as-a-service serverless serverless-functions

openwhisk-wskdeploy's Introduction

Whisk Deploy wskdeploy

License Build Status

wskdeploy is a utility to help you describe and deploy any part of the OpenWhisk programming model using a YAML manifest file. Use it to deploy all of your OpenWhisk project's Packages, Actions, Triggers, and Rules, together, using a single command!

Running wskdeploy standalone

You can use this utility separately from the OpenWhisk CLI as it uses the same OpenWhisk "Go" Client as the Openwhisk CLI does to create its HTTP REST calls for deploying and undeploying your Openwhisk packages and entities.

Running wskdeploy as part of the wsk CLI

Alternatively, you can use the wskdeploy functionality within the OpenWhisk CLI as it is now embedded as the deploy command. That is, you can invoke it as wsk deploy using all the same parameters documented for the standalone utility.

Using wskdeploy to manage OpenWhisk entities as projects

In addition to simple deployment, wskdeploy also has the powerful export command to manage sets of OpenWhisk entities that work together as a named project. The command:

wskdeploy export --projectname <managed_project_name>`

allows you to "export" a specified project into a local file system and manage it as a single entity.

In the above example, a <managed_project_name>.yml Manifest file would be created automatically which can be used with wskdeploy to redeploy the managed project on a different OpenWhisk instance. If the managed project contains dependencies on other managed projects, then these projects will be exported automatically into their respective manifests.

Getting started

Here are some quick links to help you get started:


Downloading released binaries

Executable binaries of wskdeploy are available for download on the project's GitHub releases page:

We currently provide binaries for the following Operating Systems (OS) and architecture combinations:

Operating System Architectures
Linux 386, AMD64, ARM, ARM64, PPC64 (Power), S/390 and IBM Z
Mac OS (Darwin) 3861, AMD64
Windows 386, AMD64
  1. Mac OS, 32-bit (386) released versions are not available for builds using Go lang version 1.15 and greater.

We also provide instructions on how to build your own binaries from source code. See Building the project.


Running wskdeploy

Start by verifying the utility can display the command line help:

$ ./wskdeploy --help

then try deploying an OpenWhisk Manifest and Deployment file:

$ ./wskdeploy -m tests/usecases/triggerrule/manifest.yml -d tests/usecases/triggerrule/deployment.yml

Building the project

GoLang setup

The wskdeploy utility is a GoLang program so you will first need to Download and install GoLang onto your local machine.

Note Go version 1.15 or higher is recommended

Make sure your $GOPATH is defined correctly in your environment. For detailed setup of your GoLang development environment, please read How to Write Go Code.

Download the source code from GitHub

As the code is managed using GitHub, it is easiest to retrieve the code using the git clone command.

if you just want to build the code and do not intend to be a Contributor, you can clone the latest code from the Apache repository:

git clone [email protected]:apache/openwhisk-wskdeploy

or you can specify a release (tag) if you do not want the latest code by using the --branch <tag> flag. For example, you can clone the source code for the tagged 1.1.0 release

git clone --branch 1.1.0 [email protected]:apache/openwhisk-wskdeploy

You can also pull the code from a fork of the repository. If you intend to become a Contributor to the project, read the section Contributing to the project below on how to setup a fork.

Build using go build

Use the Go utility to build the wskdeploy binary.

Change into the cloned project directory and use go build with the target output name for the binary:

$ go build -o wskdeploy

If successful, an executable named wskdeploy will be created in the project directory compatible with your current operating system and architecture.

Building for other Operating Systems (GOOS) and Architectures (GOARCH)

If you would like to build the binary for a specific operating system and processor architecture, you may add the arguments GOOS and GOARCH into the Go build command (as inline environment variables).

For example, run the following command to build the binary for 64-bit Linux:

$ GOOS=linux GOARCH=amd64 go build -o wskdeploy

Supported value combinations include:

GOOS GOARCH
linux 386 (32-bit), amd64 (64-bit), s390x (S/390, Z), ppc64le (Power), arm (32-bit), arm64 (64-bit)
darwin (Mac OS) amd64
windows 386 (32-bit), amd64 (64-bit)

Build using Gradle

The project includes its own packaged version of Gradle called Gradle Wrapper which is invoked using the gradlew command on Linux/Unix/Mac or gradlew.bat on Windows.

  1. Gradle requires requires you to install Java JDK version 8 or higher

  2. Clone the openwhisk-wskdeploy repo:

    git clone https://github.com/apache/openwhisk-wskdeploy

    and change into the project directory.

  3. Cross-compile binaries for all supported Operating Systems and Architectures:

    ./gradlew goBuild
  4. Upon a successful build, the wskdeploy binaries can be found under the corresponding build/<os>-<architecture>/ folder of your project:

    $ ls build
    darwin-amd64  linux-amd64   linux-arm64   linux-s390x   windows-amd64
    linux-386     linux-arm     linux-ppc64le windows-386

Compiling for a single OS/ARCH

  1. View gradle build tasks for supported Operating Systems and Architectures:

    ./gradlew tasks

    you will see build tasks for supported OS/ARCH combinations:

    Gogradle tasks
    --------------
    buildDarwinAmd64 - Custom go task.
    buildLinux386 - Custom go task.
    buildLinuxAmd64 - Custom go task.
    buildLinuxArm - Custom go task.
    buildLinuxArm64 - Custom go task.
    buildLinuxPpc64le - Custom go task.
    buildLinuxS390x - Custom go task.
    buildWindows386 - Custom go task.
    buildWindowsAmd64 - Custom go task.

    Note: The buildWindows386 option is only supported on Golang versions less than 1.15.

  2. Build using one of these tasks, for example:

    $ ./gradlew buildDarwinAmd64

Using your own local Gradle to build

Alternatively, you can choose to Install Gradle and use it instead of the project's Gradle Wrapper. If so, you would use the gradle command instead of gradlew. If you do elect to use your own Gradle, verify its version is 6.8.1 or higher:

gradle -version

Note If using your own local Gradle installation, use the gradle command instead of the ./gradlew command in the build instructions below.

Building for internationalization

Please follow this process for building any changes to translatable strings:


Running as a Go program

Since wskdeploy is a GoLang program, you may choose to run it using the Go utility. After building the wskdeploy binary, you can run it as follows:

$ go run main.go --help

and deploying using the Go utility would look like:

$ go run main.go -m tests/usecases/triggerrule/manifest.yml -d tests/usecases/triggerrule/deployment.yml

Contributing to the project

Git repository setup

  1. Fork the Apache repository

    If you intend to contribute code, you will want to fork the apache/openwhisk-wskdeploy repository into your github account and use that as the source for your clone.

  2. Clone the repository from your fork:

    git clone [email protected]:${GITHUB_ACCOUNT_USERNAME}/openwhisk-wskdeploy.git
  3. Add the Apache repository as a remote with the upstream alias:

    git remote add upstream [email protected]:apache/openwhisk-wskdeploy

    You can now use git push to push local commit changes to your origin repository and submit pull requests to the upstream project repository.

  4. Optionally, prevent accidental pushes to upstream using this command:

    git remote set-url --push upstream no_push

Be sure to Sync your fork before starting any contributions to keep it up-to-date with the upstream repository.

Running unit tests

You may use go test to test all unit tests within a package, for example:

go test ./deployers -tags=unit -v
go test ./parsers -tags=unit -v

or to run individual function tests, for example:

go test ./parsers -tags=unit -v -run TestParseManifestForSingleLineParams

Running integration tests

Integration tests are best left to the Travis CI build as they depend on a fully functional OpenWhisk environment to be deployed.

Adding new dependencies

Please use go get to add new dependencies to the go.mod file:

go get github.com/project/[email protected]

Please avoid using commit hashes for referencing non-OpenWhisk libraries.

Removing unused dependencies

Please us go tidy to remove any unused dependencies after any significant code changes:

go mod tidy

Updating dependency versions

Although you might be tempted to edit the go.mod file directly, please use the recommended method of using the go get command:

go get -u github.com/project/libname  # Using "latest" version
go get -u github.com/project/[email protected] # Using tagged version
go get -u github.com/project/libname@aee5cab1c  # Using a commit hash

Updating Go version

Although you could edit the version directly in the go.mod file, it is better to use the go edit command:

go mod edit -go=1.15

Creating Tagged Releases

Committers can find instructions on how to create tagged releases here:


Troubleshooting

Known issues

Git commands using HTTPS, not SSH

The "go get" command uses HTTPS with GitHub and when you attempt to "commit" code you might be prompted with your GitHub credentials. If you wish to use your SSH credentials, you may need to issue the following command to set the appropriate URL for your "origin" fork:

git remote set-url origin [email protected]:<username>/openwhisk-wskdeploy.git

or you can manually change the remote (origin) url within your .git/config file:

[remote "origin"]
    url = [email protected]:<username>/openwhisk-wskdeploy

while there, you can verify that your upstream repository is set correctly:

[remote "upstream"]
    url = [email protected]:apache/openwhisk-wskdeploy

Git clone RPC failed: HTTP 301

This sometimes occurs using "go get" the wskdeploy code (which indirectly invokes "git clone").

You might get this error when downloading openwhisk-wskdeploy:

Cloning into ''$GOAPTH/src/gopkg.in/yaml.v2'...
error: RPC failed; HTTP 301 curl 22 The requested URL returned error: 301
fatal: The remote end hung up unexpectedly

This is caused by newer git versions not forwarding requests anymore. One solution is to allow forwarding for gopkg.in

$ git config --global http.https://gopkg.in.followRedirects true

openwhisk-wskdeploy's People

Contributors

akrabat avatar allen-servedio avatar alvarolopez avatar brunogirin avatar bvennam avatar csantanapr avatar daisy-ycguo avatar dgrove-oss avatar huydoan2 avatar jiangpengcheng avatar juice10 avatar jzaccone avatar keonhee avatar lionelvillard avatar lornajane avatar lzbj avatar mdeuser avatar mptap avatar mrutkows avatar ningyougang avatar paulcastro avatar pritidesai avatar pwplusnick avatar rabbah avatar sciabarracom avatar shawnallen85 avatar srl295 avatar style95 avatar tpei avatar violagao avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

openwhisk-wskdeploy's Issues

Proposal: add a new element to spec to support for API Gateway mapping

There is currently an experimental API for an API Gateway. We should investigate support for this API inside a manifest and/or deployment.

The experimental API currently runs in the CLI and we can derive the calls from running the CLI in verbose mode.

An example of possible syntax:

api-experimental:
    newbot:
      basepath: /whiskbot
      path: /newbot
      verb: get
      action: newbot

where the action is newbot, and the constructed API is a GET call with path /whiskbot/newbot

Generate template package using "init" or "generate" command

It would be useful to have a wskdeploy generate <package-name> or wskdeploy init <package-name> , which could prompt the user for basic information about a package and generate a starter manifest.yaml as well as a starter directory structure (e.g. a src directory). It could also generate a starter deployment.yaml-template file. That way, a developer can use wskdeploy almost exclusively from the start of development to deploy their OpenWhisk assets.

build and install wsktool raises problem.

With current codebase, when I try make build, the go will complain "fatal: repository 'https://github.com/openwhisk/go-whisk/' not found", because currently the repo doesn't exists.
So the import will fail.

If I change the package import source code to github.com/openwhisk/openwhisk/tools/cli/go-whisk/whisk, the go will complain that
" Installing dependencies
$GOPATH/src/github.com/openwhisk/openwhisk/tools/cli/go-whisk/whisk/client.go:31:5: local import "../wski18n" in non-local package"

Develop 1st test case

Develop the first test case of Wsk Tool.
I recommend to use the "HelloWorld" use case.
I will submit the related files to "tests" folder.

Daisy

Remove dependency on i18n library

The i18n library:
"github.com/nicksnyder/go-i18n/i18n"

Has an ad-hoc license which may prevent us from exiting Apache incubator. Please find an alternative or attempt to remove. Otherwise we will need to submit it for legal review at the ASF.

Add testcase3

The third test case will support feed action. It uses the github hook as a sample. This use case will install a feed to fire a trigger when there is activity in a specified GitHub repository.

The package will deploy:
a feed action named as "webhook_feed"
a trigger named as "webhook_trigger"
a action named as "sendemail"

When the package is deployed successfully, it can be tested as following:
#Install package
$ wsktool deploy -p /tests/testcases/githubfeed
#When there is a "PUSH" event to the specified github, an email will be sent out.

[WIP] Create DEPENDENCIES.md to track external libs.

I would like to create a new top-level file (which I intent to make a convention for all OpenWhisk family of projects) that lists all external libraries (e.g., imports) we rely upon to build (or run). The list should for convenience list the type of license and provide a hyperlink to that project/license for convenience when we need to provide reviews for Apache.

The goal is to become aware that in order to exit Apache incubator we must assure that all code dependencies have Apache 2.0 compatible licenses.

The wskdeploy tool deploy rule failed.

We deploy rule with the sample file, the rule deployment failed. With the error message:
Got error creating rule with error message: {{.msg}} (code {{.code}}) and error code: 144.

Other action, trigger, package creation failed, it seems related to this issue.
apache/openwhisk#1577

We need to investigate and fix this.

rename project openwhisk-wskdeploy to follow ASF conventions

All openwhisk family projects should start with "openwhisk-".

Please note "master" and "development" or out of sync and should be have merges (from both sides) to sync them. Ideally, "master" should be "master" and reflect the working code for GA.

Add a travis file for the project.

Add a travis file, so make sure all the tests(unitary, integration) are passed and build success, for a minimum gate before any merge of pr.

[WIP]Add basic support of deployment.yml

We need to start supporting deployment file. According to the Specification, a Deployment file contains configuration information, e.g.

  • input parameters
  • authorization credentials
  • namespaces

and so on.

There is a simple Deployment file in tests/testcases/triggerrule, with namespaces of application and packages, authorization credential of OW, and input parameters to rule. We can start the support of deployment file from this basic case.

Refactor the yaml parser.

Currently there are two yaml file to help deploy the wsk objects, one is deployment yaml and one is manifest yaml, if we need those 2 files, we'd better parse them in separate go modules, currently they are in one module. It's better to put them in 2 diff module.

Add interactive flag

wskdeploy --interactive-mode yes|no can toggle on and off any prompting. Default is no (or false), we should allow interactive mode.

Move deploy code into root command

With the new name, wskdeploy deploy is redundant. We should remove the deploy command and move its function into the wskdeploy root command

Add support to manage dependencies

The current tool cannot reference other OpenWhisk packages as dependencies. At a minimum we should be able to pull in and reference entities in other packages w/o requiring the developer to copy the source code. This requires:

  • create a module to parse dependency graph and validate it
  • create feature to download/obtain dependencies into the current project path and be able to reference these from inside the manifest and deployment yaml files.

This is a large task and likely should be broken up into multiple issues.

As a starting step let's propose:

  • support for getting a Github-based OpenWhisk package
  • mange a semantic version applied to an entire deployment (as opposed to individual entities)

[WIP]Add deployment of feed

We need to:

  • update parser to support feed
  • update servicedeployer.go to support feed
  • update the webhook test case.

Add parser for manifest.yml and integrate with servicedeployer.

We want a parser to support the elements in the manifest files of testcase1.
I'm going to make the following changes:

  • utils/manifest.go will be added for manifest_full.yaml parser.
  • tests/manifest_test.go will be added accordingly.
  • remove deployment.go and test code according.
  • Integrate manifest.go with deployerService.
  • Add corresponding test cases for the integration.
  • Refactor and change existing code if necessary.

Add defaults flag to command

wskdeploy --no-defaults yes | no (or true | false) should affect if we use file system conventions and allow default values for manifest file. Default is yes (we should not allow defaults)

Is the action needs to be hard coded in the sample manifesto yaml?

Currently the manifest_full.yaml the action hello is hard coded, which makes parser a little bit confused, so the better action name would be action, and put the action name as a property?

Another issue, do we need to consider some web contents, I read the spec, it seems some action or whisk entities maybe on the web?

Create an apiClient that integrates with the go-whisk package

The OpenWhisk API has an existing Go package go-whisk that implements the OpenWhisk REST API. It is designed to be re-used by other applications. We should integrate our tool with this package to handle the network calls to an OpenWhisk instance.

go-whisk will provide us with:

  • A REST API implementation
  • A framework for adding i18n to our code

Rename repo. openwhisk-wskdeploy

We need to conform to the convention established for all repos. under the OpenWhisk family. This is something we discussed among the initial committers for OpenWhisk under Apache as the established naming convention going forward. This will align us also with existing Apache project GitHub repos. conventions and ease any potential movement under a different org.

Support "composition entity"

Use case "Data Processing with MessageHub" needs an action sequence. We should support composition entity in wskdeploy.

Sample:

my_action_sequence:
    type: sequence
    sequence:
        - action_1
        - action_2
        - action_3
    inputs:
        simple_input_string: string
    outputs:
        annotated_output_string: string

Add test case for bots

This test case includes proposal for declaring sequence, use api-experimental, add annotations

Add testcase2

testcase2 is based on testcase1 but it can be deployed and tested independently because the artifacts of testcase1 is included.

The testcase2 to deploy a basic trigger and rule based. The demo can be shown with following commands:

#deploy package, including
#Action "hello" is deployed
#Trigger "locationUpdate" is deployed
#Rule "myFirstRule" is deployed
$ wsktool deploy -p /tests/testcases/triggerrule
#Fire trigger "locationUpdate"
$ wsk trigger fire locationUpdate --param name Donald --param place "Washington, D.C."
#Verify "hello" is invoked
$ wsk activation list --limit 1 hello
$ wsk activation result ###############

[WIP] Add verify command.

Add a verify command which would read the manifest and deployment files, then check to see what is already installed in OpenWhisk.
This is a derivative of #93

Improve traces and hints.

Now if there is no default manifest/deployments yaml in current dir, the tool will panic with stack traces, which is not friendly to users, we need to improve this. We also need better description for basic commands currently already supported etc.

[WIP] Add deploy cmd.

Currently the deploy were a default behavior under root command, and the manifest and deployment flag are global flags, we need to separate deploy/undeploy and corresponding
flags under each command. Add this issue to separate the refactor progress.

How to integrate wskdeploy into wsk cli ?

We need to think about how to integrate wskdeploy as a part of wsk cli. We may consider to use wsk deploy to invoke the real go program wskdeploy. Shall we integrate them at the source code level ? Or shall we integrate them at build level ?

Support multiple packages deployment in a single command

We need to discuss how to support multiple packages deployment in a single command. There are two options:

  • add multiple packages in a single manifest file
  • use multiple manifest files, 1 package per a manifest

Let's discuss the solution here.

Add a simple UI through web browser then CLi.

We could add a simple UI through web browser if user don't want to bother with Cli, it's perhaps more intuitive, we could wrap up go cli with a web server, and some basic HTML template.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.