Git Product home page Git Product logo

docker-exec / dexec Goto Github PK

View Code? Open in Web Editor NEW
329.0 21.0 18.0 1014 KB

:whale: Command line interface for running code in many languages via Docker.

Home Page: https://docker-exec.github.io/

License: MIT License

Go 79.74% Shell 8.67% C 0.96% Clojure 0.41% CoffeeScript 0.34% C++ 1.07% C# 1.22% D 0.72% Erlang 1.29% F# 0.19% Forth 0.21% Groovy 0.30% Haskell 0.44% Java 1.23% Common Lisp 0.36% JavaScript 0.43% Objective-C 0.99% OCaml 0.52% Perl 0.33% PHP 0.56%
polyglot go docker execution

dexec's Introduction

dexec Build Status Download

A command line utility for executing code in many different languages using the Docker Exec images, written in Go.

dexec demo animation

Installation

Using Bintray

Download the appropriate binary for your OS and architecture, then unzip or untar and move the dexec executable to where it can be found on your PATH.

OS 64-bit 32-bit
Linux 64-bit 32-bit
Mac 64-bit 32-bit
Windows 64-bit 32-bit

Binaries for other distributions are available on Bintray.

Using Go

Install with the go get command.

$ go get github.com/docker-exec/dexec

Using Homebrew

If you're on OSX you can install the latest release of dexec with brew.

$ brew install docker-exec/formula/dexec

Reference

These examples use a .cpp source file, but any of the supported languages can be used instead. Arguments can be passed in any order, using any style of the acceptable switch styles described.

The application provides help and version information as follows:

$ dexec --version
$ dexec --help

Pass source files to execute

Multiple source files can be passed to the compiler or interpreter as follows. The first source file's extension is used to pick the appropriate Docker Exec image, e.g. .cpp retrieves dexec/cpp from the Docker registry.

$ dexec foo.cpp
$ dexec foo.cpp bar.cpp

The sources are mounted individually using the default Docker mount permissions (rw) and can be specified by appending :ro or :rw to the source file.

Pass arguments for build

For compiled languages, arguments can be passed to the compiler.

$ dexec foo.cpp --build-arg=-std=c++11
$ dexec foo.cpp --build-arg -std=c++11
$ dexec foo.cpp -b -std=c++11

Pass arguments for execution

Arguments can be passed to the executing code. Enclose arguments with single quotes to preserve whitespace.

$ dexec foo.cpp --arg=hello --arg=world --arg='hello world'
$ dexec foo.cpp --arg hello --arg world --arg 'hello world'
$ dexec foo.cpp -a hello -a world -a 'hello world'

Specify location of source files

By default, dexec assumes the sources are in the directory from which it is being invoked from. It is possible to override the working directory by passing the -C flag.

$ dexec -C /path/to/sources foo.cpp bar.cpp

Read from STDIN

dexec will forward your terminal's STDIN to the executing code. You can redirect from a file or use pipe:

$ dexec foo.cpp <input.txt
$ curl http://input | dexec foo.cpp

Include files and folders

Individual files can be mounted without being passed to the compiler, for example header files in C & C++, or input files for program execution. These can be included in the following way.

$ dexec foo.cpp --include=bar.hpp
$ dexec foo.cpp --include bar.hpp
$ dexec foo.cpp -i bar.hpp

In addition, a program may require read and/or write access to several files on the host system. The most efficient way to achieve this is to include a directory.

$ dexec foo.cpp --include=.
$ dexec foo.cpp --include .
$ dexec foo.cpp -i .

Files and directories are relative to either the current working directory, or the directory specified with the -C flag.

As with sources, included files and directories are mounted using the default Docker mount permissions (rw) and can be specified by appending :ro or :rw to the source file.

Override the image used by dexec

dexec stores a map of file extensions to Docker images and uses this to look up the right image to run for a given source file. This can be overridden in the following ways:

Override image by name/tag

$ dexec foo.c --image=dexec/lang-cpp
$ dexec foo.c --image dexec/lang-cpp
$ dexec foo.c -m dexec/lang-cpp

This will cause dexec to attempt to use the supplied image. If no image version is specified, "latest" is used.

Override image by file extension

$ dexec foo.c --extension=cpp
$ dexec foo.c --extension cpp
$ dexec foo.c -e cpp

This will cause dexec to attempt to lookup the image for the supplied extension in its map.

Force dexec to pull latest version of image

Primarily for debugging purposes, the --update command triggers a docker pull of the target image before executing the code.

$ dexec foo.cpp -u
$ dexec foo.cpp --update

Force dexec to remove all dexec images

The --clean command removes all versions of images matching /^dexec/lang-([^:\s])$/. It can be combined with source files or STDIN input if you wish to remove all containers stored locally before executing.

$ dexec --clean

Executable source with shebang

dexec can be used to make source code executable by adding a shebang that invokes it at the top of a source file.

The shebang is stripped out at execution time but the original source containing the shebang is preserved.

#!/usr/bin/env dexec
#include <iostream>
int main() {
    std::cout << "hello world" << std::endl;
}

then

$ chmod +x foo.cpp
$ ./foo.cpp

Contributors

See also

dexec's People

Contributors

alixaxel avatar andystanton avatar dependabot-preview[bot] avatar docker-exec-bot avatar inthecloud247 avatar johnhany97 avatar kroton avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dexec's Issues

Organise dexec sources better

The sources look like they could be organised a bit better. I don't know what Go best practises for this are so a small investigation is required.

Add global configuration file to store common parameters

I almost always use the latest C++ with dexec, so rather than having to type this every time:

$ dexec -b -std=c++14 source.cpp

I'd rather just have a .dexecrc file in my user home which contained something like this:

---
c++:
    build-args:
        - -std=c++11

---

It doesn't have to be yaml, but yaml is quite succinct. Go has good out of the box support for json.

running python prog on windows 10 gives No connection could be made because the target machine actively refused it.

So I have python1.py:

#!usr/bin/python
print("hello from python")

I dexec and the python file in the following folder, this happens :

C:\Users\user\Downloads\dexec_1.0.7_windows_amd64>dexec.exe python1.py
2018/01/28 15:24:58 Get http://127.0.0.1:2375/_ping: dial tcp 127.0.0.1:2375: connectex: No connection could be made because the target machine actively refused it.

I allowed dexec.exe through my firewall.
Is there anything i'm missing.?

Homebrew requires --HEAD

Running the brew command as listed in the readme (brew install docker-exec/formula/dexec) results in an error message:

Error: docker-exec/formula/dexec is a head-only formula
Install with `brew install --HEAD docker-exec/formula/dexec`

With the --HEAD option, brewing succeeds.

Allow code to be read from stdin

It should be possible to read code from stdin - either piped in or read line by line and terminated by eof.

e.g.

$ echo "console.log('hello world')" | dexec --extension js

hello world

dexec should create a randomly generated filename, dump the contents in there and execute it as js.

same with

$ dexec --extension js
Type your code. Ctrl-D to exit
console.log('hello world')
<Ctrl-D>

hello world

It should work with this new extension override option -e or --extension as well as the existing -s or --specify-image, but only if no sources are specified.

Getting inputs from a file.

Trying to get file data this way:

dexec test.c < text.txt

Doesn't work.

What are the alternatives? Or how do we do this?

Add contributor credits

There were some PRs in the 1.0.1 release and the contributors weren't credited. This should be addressed.

Allow "caching" of dexec containers

Currently containers run as part of dexec are automatically removed. The tool should allow an option to "cache" containers, i.e. not remove them after running, and take an alias for the container. The cached version can then be run with the alias to save compilation time and provide a way to reuse containers.

This requires a bit more thought, but that's the basic idea.

get "Docker not found" when trying to use linux binary.

Tried two machines an Ubuntu and a Gentoo box. on both did the following:
curl -L https://bintray.com/artifact/download/dexec/release/dexec_1.0.3_linux_amd64.tar.gz | tar -xz && ./dexec_1.0.3_linux_amd64/dexec
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 988k 100 988k 0 0 977k 0 0:00:01 0:00:01 --:--:-- 977k
2016/04/14 16:44:05 Docker not found

however docker is in path (/usr/bin/docker) I can strace and see it does find it. I did try grepping the source for the "docker not found" error message but couldn't find it.

UnicodeEncodeError on Python3

helloworld.py

# -*- coding: utf-8 -*-
print('Hello, 世界!')

$ dexec helloworld.py
UnicodeEncodeError: 'ascii' codec can't encode characters in position 7-9: ordinal not in range(128)

Passing in non-existent source file causes an error

This happens:

$ dexec asdgas.js

sed: read error on asdgas.js: Is a directory

dexec should check the file exists on the local file system before telling docker to mount it - otherwise docker helpfully creates it as a directory in the container.

Support for SQL[ite]

I guess having at least one kind of SQL dialect implement would be a good addition.

I'm gonna try and submit a PR sometime soon for .sql > SQLite support.

Make Perl 5 default

A few people have mentioned that Perl6 isn't official yet:

I've merged docker-exec/perl#2 which will repurpose the perl image as perl 5. A separate perl6 image has been created and will be mapped from .p6 files.

Further to this #11 will allow the perl6 to be used for .pl files if so required.

Shebang script doesn't work with Java

File:

Shebang.java

Permissions:

$ ls -la Shebang.java
-rwxr-xr-x  1 andy  staff  140 20 Nov 17:36 Shebang.java

Shebang.java:

#!/usr/bin/env dexec
public class Shebang {
    public static void main(String[] args) {
        System.out.println("hello world");
    }
}

Command:

$ ./Shebang.java

Error:

Error: Could not find or load main class

Single character file names cause help message to be displayed instead of executing

Steps to reproduce:

$ echo "console.log(\"foo\")" >a.js
$ dexec a.js
Name:
    dexec - Execute code in many languages with Docker!

Usage:
    dexec [options] <source files...>

Options:
    -C <dir>                            Specify source directory
    --arg, -a <argument>                Pass <argument> to the executing code
    --build-arg, -b <build argument>    Pass <build argument> to compiler
    --include, -i <file|path>           Mount local <file|path> in dexec container
    --extension, -e <extension>         Override the image used by <extension>
    --image, -m <name>                  Override the image used by <name>
    --update, -u                        Force update of image
    --clean                             Remove all local dexec images
    --help, -h                          Show help
    --version, -v                       Display version info

starting container with non-empty request body was deprecated

Here's what I get when trying to run dexec:

dexec hcl.go
2017/01/31 11:44:18 API error (400): {"message":"starting container with non-empty request body was deprecated since v1.10 and removed in v1.12"}

Here is my docker version:

Client:
 Version:      1.13.0
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Wed Jan 18 16:20:26 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      1.13.0
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Wed Jan 18 16:20:26 2017
 OS/Arch:      linux/amd64
 Experimental: true

And dexec version:

dexec --version
dexec 1.0.4-SNAPSHOT

Use a single repo for all Dockerfiles

Check to see if the Docker hub tooling has improved and will only rebuild images in a given path. If so, switch to a single repo for all Dockerfiles so that the image-common git submodule can go away as it causes problems like docker-exec/groovy#1.

Temporary file can not be found?

I'm always getting the following error whenever I try to dexec any file (regardless of language):

/tmp/dexec/image-common/dexec-script.sh: line 28: /home/alix/foo.js_tmp: No such file or directory

Here's some info that could be useful:

alix@ThinkPad:~$ docker info
Containers: 0
Images: 15
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 15
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-28-generic
Operating System: Ubuntu 15.04
CPUs: 4
Total Memory: 7.496 GiB
Name: ThinkPad
ID: R7GC:HCG4:T7R2:3T62:DLYM:XBFR:MLQH:BD5M:VDUN:OP2Y:G4VA:GSYF
WARNING: No swap limit support

alix@ThinkPad:~$ docker version
Client:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0a8c2e3
 Built:        Thu Sep 10 19:21:21 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0a8c2e3
 Built:        Thu Sep 10 19:21:21 UTC 2015
 OS/Arch:      linux/amd64

alix@ThinkPad:~$ sudo dexec /home/alix/foo.js
Unable to find image 'dexec/lang-node:1.0.2' locally
1.0.2: Pulling from dexec/lang-node
a1589201fdd2: Pull complete 
059c05383383: Pull complete 
c99874c78a04: Pull complete 
5c979081e8a8: Pull complete 
0f81d39fb182: Pull complete 
1ea59cd801ad: Pull complete 
3f53c4f49f02: Already exists 
3cb35ae859e7: Already exists 
41b730702607: Already exists 
Digest: sha256:ef25b334cf651b8ba889e4eff5c80c9974a3fdaa2b46bab4e70e44fa612e3320
Status: Downloaded newer image for dexec/lang-node:1.0.2
/tmp/dexec/image-common/dexec-script.sh: line 28: /home/alix/foo.js_tmp: No such file or directory

I'm running dexec 1.0.2, I tried both the binary from Bintray and I also compiled the code locally.

Is there anything I'm missing? I tried the same in a Mac OS and everything worked perfectly.

Fix broken tests

While working on #22 I didn't realise I'd silently broken some of the tests.

I was running them with:

go test

but this only runs the ones in the root directory. Instead I should have run:

go test ./...

which causes go test to search for tests recursively.

.travis.yml should be updated with the script command explicitly set to go test ./... instead of using the default which is go test.

Change shelled out calls to Docker remote API

The approach I've taken to #22 involves more shelled out calls to the Docker CLI. This is getting ugly and there may be a better way: talking to the Docker remote API directly and parsing the json it returns instead of hacking around with the arbitrary text returned by the CLI.

Boot2docker example query piped into jq:

curl \
    --insecure \
    --cert ~/.boot2docker/certs/boot2docker-vm/cert.pem \
    --key ~/.boot2docker/certs/boot2docker-vm/key.pem \
    https://$(boot2docker ip):2376/images/json \
| jq .

Gives a (much longer) version of this response:

[
  {
    "Created": 1422379591,
    "Id": "c90d655b99b2ec5b7e94d38c87f92dce015c17a313caeaae0e980d9b9bed8444",
    "Labels": null,
    "ParentId": "30d39e59ffe287f29a41a3f8bd70734afc8728329e3289945cbdc5bbf07cd980",
    "RepoDigests": [],
    "RepoTags": [
      "debian:latest"
    ],
    "Size": 0,
    "VirtualSize": 85014569
  }
]

In this thread, it's noted that the version of curl that comes with osx doesn't work with the above command (I've confirmed on Yosemite which is curl 7.30.0).

Running the latest curl downloadable through brew (7.42.0 currently) allows the command to work which is something to bear in mind when developing this change as I'm not sure what http libraries are available in go.

Add 'clean' option to delete all local dexec images

The dexec images are quite big. Combined with the fact that there are now multiple versions, this means that they consume a lot of space.

This feature provides a command line option to clean up all dexec images. It will be analagous to running:

docker images \
    | grep dexec \
    | awk '{print $3}' \
    | xargs -I {} docker rmi {}

I think it should also be rmi -f i.e. - delete the image even if its being used by a running container. Maybe the error should be propagated to the user but if you've chosen to run this cleanup command it's because you don't want any containers to make use of the dexec images anyway. I'll see what the implementation looks like.

Output is not streamed

This program should print "sleeping for 5 seconds" immediately, then after 5 seconds print "done". When run with dexec, the whole output is returned only when the container has finished.

#include <stdio.h>
#include <unistd.h>

int main() {
    printf("sleeping for 5 seconds\n");
    sleep(5);
    printf("done\n");
    return 0;
}

Instead, output should be streamed in the case of long running or streaming applications.

Doesn't work under Windows 7 (docker toolbox)

Failed on sed command inside .sh script because mounted volume is not a file, as expected, but a directory.

As I can understand, directory mount is only possible option for windows.

Display progress while pulling images

Currently if dexec is pulling an image, no indication is displayed. As this can sometimes take a few minutes, it's bad user experience.

This issue has two phases:

  1. Print a message to stderr saying an image is being downloaded, specifying the name and tag of the image.
  2. Parse the information received from the Docker remote API and either output it in the same way as the Docker client does, or some other minimal way e.g. single percentage rather than per layer percentages.

Prefix Docker hub automated builds with "lang-"

The automated builds for each language image are currently of the form dexec/{{language abbreviation}}. Some languages (C, D, Go & R) are shorter than the required 3 character limit and have started breaking. This is a bit annoying because it let me create them in the first place.

The approach I want to take is to change the format of the build name to dexec/lang-{{language abbreviation}} and eventually apply this to the github repo names.

This change will not remove the existing automated builds - so the current images will still be available at dexec/{{language abbreviation}} but going forward dexec will make use of the dexec/lang-{{language-abbreviation}} form.

Images should reflect language versions

Currently the versions of Docker images reflect changes made to the Dockerfiles. This seems a pointless thing to keep track of when the really important thing is the version of the language the Dockerfile contains.

For example, rather than lang-cpp:1.0.2 it is more useful to have lang-cpp:11 which could be an alias for even more detailed information like lang-cpp:clang-3.4 or lang-cpp:gcc-4.9.3.

This requires some thinking.

Forward STDOUT/STDERR

With a simple PHP script that writes to both STDOUT and STDERR:

fwrite(STDOUT, "stdout\n");
fwrite(STDERR, "stderr\n");

The output streams work as expected when executed with PHP directly:

alix@ThinkPad:~/dexec_1.0.2_linux_386$ php streams.php 1>/dev/null
stderr
alix@ThinkPad:~/dexec_1.0.2_linux_386$ php streams.php 2>/dev/null
stdout

However, when running via dexec, STDERR is treated as if it was STDOUT:

alix@ThinkPad:~/dexec_1.0.2_linux_386$ dexec streams.php 1>/dev/null
alix@ThinkPad:~/dexec_1.0.2_linux_386$ dexec streams.php 2>/dev/null
stdout
stderr

The exact same thing happens with a Node.js script:

console.log("stdout");
console.error("stderr");
alix@ThinkPad:~/dexec_1.0.2_linux_386$ node streams.js 1>/dev/null
stderr
alix@ThinkPad:~/dexec_1.0.2_linux_386$ node streams.js 2>/dev/null
stdout
alix@ThinkPad:~/dexec_1.0.2_linux_386$ dexec streams.js 1>/dev/null
alix@ThinkPad:~/dexec_1.0.2_linux_386$ dexec streams.js 2>/dev/null
stdout
stderr

Is there any option to make dexec respect and forward STDOUT and STDERR?

Migrate dexec to Go Modules

dexec currently uses godep and vendoring for version management. The current solution is dep, but Go Modules are the future, so I would like to migrate dexec to this solution.

Replace custom Docker code with library calls

Ticket #35 details changing the shelled out calls to the Docker CLI to http calls to the Docker remote API. This is admirable and I got this working, but ultimately there is no point maintaining this code which someone else has already written better. In this case, the Docker developer documentation mentions two mature libraries that perform this function.

This ticket will involve researching which is a better fit for dexec and then switch dexec to the chosen library.

Remote execution

I think the next step would be enabling the use of an external Docker server (for Windows machines that have acces to a Linux box w/Docker).
I know that it may involve ssh commands or acces to the Docker remote API and that is a complete different beast.

Does the current architecture allows the use of an external docker daemon?
Amazing project.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.