Pull Jobs from Anywhere

Supraworker is an abstraction layer around jobs, allowing one to pull any job from any API, callback your APIs, observe execution times and control concurrent execution.

TL;DR: Supraworker makes an API request for a job => API responds with a bash command => Supraworker executes a bash command. Easy.

It can pull any commands (jobs) from your APIs, run commands, and stream logs back to your API. It can also send state updates to a remote API.

Prerequisites:

1. An API service that can serve jobs:
   • You can check this simple implantation of an API service written in Python
2. Supraworker configuration:
   • You can check the configuration for the above API service written in Python
3. A server to run Supraworker on

Homebrew is a free and open-source package management system for Mac OS X.

 brew tap wix/tap

if you are using ssh then

brew tap wix/tap  [email protected]:wix/homebrew-brew.git

brew update
brew install wix/tap/supraworker

To update to the latest, run

brew upgrade wix/tap/supraworker

Find the version you wish to install on the GitHub Releases page and download either the darwin-amd64 binary for MacOS or the linux-amd64 binary for Linux. No pre-built binaries are provided for other operating systems or architectures at this time.

Running releases on MacOS:

You need to the download file, extract it, and remove attributes with the following command (where ~/Downloads/supraworker_darwin_amd64/supraworker is path to the file):

$ xattr -d com.apple.quarantine ~/Downloads/supraworker_darwin_amd64/supraworker
$ ~/Downloads/supraworker_darwin_amd64/supraworker

curl --silent -L  "https://api.github.com/repos/wix/supraworker/releases/latest"  \
| jq --arg PLATFORM_ARCH "$(echo `uname -s`_amd| tr '[:upper:]' '[:lower:]')" -r '.assets[] | select(.name | contains($PLATFORM_ARCH)).browser_download_url' \
| xargs -I % curl -sSL  % \
| sudo tar --strip-components=1  -xzf  -

1. install Go v1.13+

2. clone into your $GOPATH:

   mkdir -p $GOPATH/src/github.com/wix
   git clone https://github.com/wix/supraworker $GOPATH/src/github.com/wix/supraworker
   cd $GOPATH/src/github.com/wix/supraworker

3. install golangci-lint for linting + static analysis

   • Lint: docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:v1.24.0 golangci-lint run -v

Define the config file at $HOME/supraworker.yaml:

NOTE: Keys are not case-sensitive (spf13/viper#635 (comment)): Viper's default behaviour of lowercasing keys for key insensitivity is incompatible with these standards (like when using case-sensitive API credentials). For eg: MyApiKey=MySecret

# ClientId in case you need to identify the worker
clientId: "my_uniq_client_id"
# how often to call your API, in seconds
api_delay_sec: 10
# operations related to the jobs
jobs:
  get:
    url: "http://localhost:80/get/new/job"
    method: POST
    headers:
      "Content-type": "application/json"
      params:
        "clientid": "{{ .ClientId}}"

• Airflow sends a task to be executed on AWS EMR
• You're building your CI/CD system

• expires all test results

$ go clean -testcache

• runs all tests

$ go test -bench= -test.v  ./...