leopardslab / dunner Goto Github PK

Is your feature request related to a problem? Please describe.
when I want to see what are the tasks in a particular Dunner file, I have to go into the file and check. ( In most of the cases I can't remember the tasks I have in Dunner file )

Describe the solution you'd like
If I have a command, like dunner tasks which list down the tasks, that makes life easier

Describe alternatives you've considered
Peeking into the Dunner file seems the only other way

Currently only a single command exists: do. Let's say that someone wants to only check for a valid dunner file, then there is no option except to run all the commands using do.

Although it is easy to write the dunner file, it would be great if we could add a new command as syntax validator that can point some of the errors if present without actually running the dunner file.

An alternate solution could be to add support of flag --check-lint in the do command itself, but I recommend a separate command considering the future potential and complexity of this tool.

The users should be able to set ENV variables to the Docker containers from .dunner.yaml

- name: 'mesosphere/aws-cli'
  command: ['aws', 'elasticbeanstalk update-application --application-name myapp']
  envs: 
   - AWS_ACCESS_KEY_ID=8O7GO86F87GY
   - AWS_SECRET_ACCESS_KEY='767tg87to867TRI68ri76R87To867T8I6rbO87T&$@bo87Tbo87' 
   - AWS_DEFAULT_REGION=us-east1

dunner validate will validate the .dunner.yaml and if there is an issue, it will show warning m messages.

https://github.com/leopardslab/dunner/projects/3#card-21710200

I need some help from someone who is good with Golang, in defining the structure of the project well.

I need to extract out the logic in do.go into a service layer and do.go to call these services.
I'm my opinion, reading the yaml should be one service and interacting with the Docker should be another service.

Dunner should be packaged and distributed as a Snap Package.
Anyone should be able to install Dunner as snap install dunner

Describe the bug
INFO[2019-07-19 23:39:20] Pulling an image: 'golang' This message is misleading when the system already has the Docker image pulled.

To Reproduce
Steps to reproduce the behaviour:

1. Create a .dunner.yaml
2. Add a task
3. Run the task ( This will pull the image )
4. Run the task again

Expected behaviour
Dunner should not pull the image again nor show an INFO message saying pulling image.

Screenshots

Add a GoDoc badge to README.md

When the .dunner.yaml is as follows, users should be able to run Dunner status prod where prod is an argument to the task status

status:
- name: 'mesosphere/aws-cli'
  command: ['aws', 'elasticbeanstalk describe-events --environment-name $1'] 
  # This uses args passed to the task, `$1` means first arg

With latest master, try :

commands:
      - ["echo","$FOO_USER"]
      - ["echo", "$BUILD_DIR"]
envs:
      - FOO_USER=foo
      - BUILD_DIR=`$GOPATH`

Environment variables are not passed to container. The commands above do not print passed values.

We need to have unit tests to cover all the units of Dunner

If a command in a list of commands fail, the execution should halt and exit code should be non-zero.

Currently if there are two commands, first one fails, it still continues to second and gives exit code as zero, marking it success. This is invalid.

• Add commands in Makefile to build, lint, format, test the source files.
• A command to be run before each commit that can keep the code changes in standard format, say precommit.

Approach 1

Have a global envs and mounts section applicable for all tasks

Example:

envs:
  - FOO=bar
  - HELLO=world

mounts: 
  - /var:/foo

tasks:
  deploy:
  - image: 'emeraldsquad/sonar-scanner'
    commands:
    - ['sonar', 'scan']
  - image: 'golang'
    commands:
    - ['go', 'install']
  - image: 'mesosphere/aws-cli'
    commands:
    - ['aws', 'elasticbeanstalk update-application --application-name myapp']
    envs:
     - AWS_ACCESS_KEY_ID=`$AWS_KEY`
     - AWS_SECRET_ACCESS_KEY=`$AWS_SECRET`
     - AWS_DEFAULT_REGION=us-east1

Describe the bug

setup:
    image: golang
    commands:
     ....
build:
  - follow: setup
    image: golang
    commands:
    .....

Currently, it runs setup task first and not run build task at all after that.

Expected behavior
This is expected to run setup task first. If if fails, it should not continue. If it passes it should continue execution of build task.

dunner init

should initialize any project with default dunner task file. This can be edited by used based on needs. Filename can be overridden by -t flag.

Describe the bug
When I run a task which is not defined in.dunner.yaml it silently fails without notifying the user.

To Reproduce
Steps to reproduce the behaviour:

1. Create a .dunner.yaml
2. Run a task which is not defined in the .dunner.yaml

Expected behaviour
Dunner should notify me that the executed task is not defined and should exit with a non-zero exit code.

Screenshots

Dunner should be packaged and distributed as a RPM Package

Dunner should be packaged and distributed as an Arch User Repository (AUR) Package

We need to test and build the Dunner everytime a new change is merged to a branch and even the PRs created.

Describe the bug
Owner of the files created by Docker images via Dunner is root, thus can not be modified by a user without sudo or changing the file permissions or ownership.

To Reproduce
Steps to reproduce the behaviour:

1. Create a .dunner.yaml
2. Add a task with a file creation step ( eg: echo "Hi" > test.txt )
3. Try to edit the generated file on the host machine
4. This gives a permission denied error

Expected behaviour
A user should be able to edit and modify files. Ideally, the owner of the file should be the user who ran the Dunner task.

Screenshots

Ability to refer status task from deploy task as @status.

deploy:
- name: 'golang'
  command: ['go', 'install']
- name: 'mesosphere/aws-cli'
  command: ['aws', 'elasticbeanstalk update-application --application-name myapp']
- name: '@status' #This refers to another task and can pass args too
  args: 'prod'
status:
- name: 'mesosphere/aws-cli'
  command: ['aws', 'elasticbeanstalk describe-events --environment-name $1'] 

deploy:
- image: 'node'
  command: ['node', '--version']
- name: '@status'

makes no sense, let's rename it to

deploy:
- image: 'node'
  command: ['node', '--version']
- from: 'status'

I'm open for suggesions also.

Dunner is not the official name of this project, it is just a code name!
It might be the official name, it might not. Give us your suggestions for a name with a good relevant meaning.

We should be able to define multiple commands for one step. All these commands will run on the same container.

deploy:
- image: 'node'
  command: 
  - ['node', '--version']
  - ['npm', '--version']

Bug
Dunner dir is not correctly set when we define it in .dunner.yaml

To Reproduce
Steps to reproduce the behaviour:

1. Create a .dunner.yaml
2. Add a task which has a step to echo $PW
3. Define dir as /go/
4. See if the working directory is /go or /dunner/go

Expected behavior
It should be /go not /dunner/go

Screenshots

We need to run lint checks everytime a new change is merged into a branch and even the PRs created.

In the mounts config of dunner, we cannot use environment variables.
Eg:

    mounts:
      - "`$BUILD_DIR`:/root/pics:wr"

Above gives error in config.

Expected

Dunner config should allow use of env for source and dest dir in mount directory.

Dunner should be packaged and distributed as a Deb package

A single dunner file can contain multiple images whose containers will have many commands to run. Pulling images from a remote server (like Docker hub) one by one is quite slow. Concurrent pulling of images can be implemented using goroutines.

When someone tries to commit the code, it should run a linter and fail if lint errors are there.

Integrate and configure a good logging library for logging purposes.

https://github.com/op/go-logging seems good.

Describe the bug

test:
  - follow: build
  - image: golang
    commands:
    - ["go", "test", "./..."]

build:
  - image: golang
...

Above dunner task file fails validations saying:

Validation failed with following errors:
task 'test': image is a required field

Expected behavior
When follow is used, task image should be optional.

Bug
WARN[2019-07-19 23:23:09] open .env: no such file or directory this warning is kinda misleading. This makes a user wonder if having a .env is mandatory.
THis

To Reproduce
Steps to reproduce the behavior:

1. Run any dunner task

Expected behaviour
This ideally should be an INFO message. eg: INFO[2019-07-19 23:23:09] No .env file found

We need a Makefile for doing the following,

make = dep ensure
make install = go install -ldflags '-s'
make test = go test -v ./...

Currently, the Docker API version is defined in the settings module, which can be changed using environment variables, or project settings file. There should be some way to find the latest compatible version of Docker API according to the installed Docker on the host.

Bug
Dunner can not recognize the locally built Docker images

To Reproduce
Steps to reproduce the behaviour:

1. Create a .dunner.yaml
2. Ad a task which refers to a locally built Docker image rather a hosted one in a repo
3. Run the task

Expected behaviour
Dunner should pick the local Docker image and run the commands on it.

Screenshots

Dunner should be able to mount the current directory as a volume to the Docker working directory.

Say the project is a Java project and the user uses an MVN + Java Docker to compile it and create a jar file. Then the Dunner should mount the source code of the project to this MVN + Java Docker in order to compile it.

Add comments for the following,

• Packages, a very brief description of its purpose
• Exported functions and structs, detailed explanation on what it does and or represents.
• Examples to use some important functions/structs

Assigning this to myself.

Dunner do somecommand --dry-run will pull the images fro Docker hub but will not run the commands, just like dry run in other applications.

Current behaviour

Currently, output is printed on console only after all the commands execution is complete. This makes user wonder if the task is even getting executed or its stuck.

Expected

As each command is executed, its output should be shown on console. This gives user the progress of execution.

Alternates available

As of now, Only option is to wait till the complete execution is complete to see the ouput.

I'm adding a brew formula for Dunner, as it's a go-to package manager for Mac OS. I have started to create a PR on this, but brew needs a URL link to project source tarball. Usually, its referenced from GitHub releases. However, I see no releases made for Dunner so far.

Is it possible to make an initial release for this repository?

When the containers are running, outputs on stderr is not shown making the user unable to see if something went wrong.

Users should be able to mount some directories to the Docker containers. Like,

deploy:
- name: 'node'
  mount:
    - './secrets/.npmrc:~/.npmrc:r'
  command: ['npm', 'install']
- name: 'golang'
  mount:
    - '~/go/src/projectname:~/go/src/projectname:wr'
  command: ['go', 'build']

mount should be an array of strings, and a string has 3 parts as host-path:container-path:flags
flags can be optional

Users should be able to define the sub-directory of the source directory which should be mounted as a volume to the Docker as dir

deploy:
- name: 'node'
  dir: 'frontend'
  command: ['sonar', 'scan']
- name: 'mvn'
  dir: 'backend'
  command: ['mvn', 'build']

Issue

If the image mentioned in .dunner.yaml is not present on the system, ImagePull is not able pull the image according to current implementation.

Workaround

After referring to this thread, it was found that the reader returned by ImagePull needs to be handled.
Modifying the code in do.go from line no. L49 to

reader, err := cli.ImagePull(ctx, step.Image, types.ImagePullOptions{})  
if err != nil {  
   panic(err)  
}  
_, err = stdcopy.StdCopy(os.Stdout, os.Stderr, reader)  
if err != nil {  
   panic(err)
}

produces the following error,

panic: Unrecognized input header: 123

This error is produced by stdcopy.StdCopy method. However, this method produces no error at L81.

Using io instead of stdcopy for output works smoothly.

reader, err := cli.ImagePull(ctx, step.Image, types.ImagePullOptions{})  
if err != nil {  
   panic(err)  
}  
_, err = io.Copy(os.Stdout, reader)  
if err != nil {  
   panic(err)  
}

Further Problem

However the produced output logs every step of download/extract process in separate line, which becomes very untidy. Is there any way to silent the output and simultaneously handle the returned reader by ImagePull.

Platform Specifications

OS: x86_64 GNU/Linux
Docker:
* Client version: 18.09.0 
* Engine version: 18.09.0
Golang version: go1.11.2

Currently dunner can be run from project root directory only and config file can be passed from anywhere using flag/present working directory by default.

Ref: https://github.com/leopardslab/dunner/blob/v2.1.3/pkg/docker/docker.go#L62

		hostMountFilepath          = "./"

This has pwd as default host mount file path. This can be made configurable.

In cases like CI systems/complex project structure, having a flag to set working directory can be useful. This can be done using a flag.

With new changes in config, structure, following validations has to be added.

• Validate task exists when referenced via follow
• Validate Commands field of config

If there are any more validations to be added, it can be included.

Expected

• When dunner validate is run, above validations should be checked
• When dunner do <task> is run, same validations should work