sashaov / oneclick-hadoop-launcher Goto Github PK

I am using Cygwin.

The error I get while running mvn hadoop-launcher:run:

! The source and destination cannot both be remote.
! rsync error: syntax or usage error (code 1) at /home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c(1148) [Receiver=3.0.9]
! The source and destination cannot both be remote.

Added some debug logs, and found that the plugin tries to copy the jar file from D:\My Documents\GitHub\oneclick-hadoop-launcher\hadoop-word-count\target\hadoop-word-count-1.0.jar:

rsync -azv -e ssh -o D:\My Documents\GitHub\oneclick-hadoop-launcher\hadoop-word-count\target\hadoop-word-count-1.0.jar host:~/hadoop-word-count/lib/

The absolutePath of the source jar doesn't fit the cygwin path format.
It should be:

rsync -azv -e ssh -o /cygdrive/d/My\ Documents/GitHub/oneclick-hadoop-launcher/hadoop-word-count/target/hadoop-word-count-1.0.jar host:~/hadoop-word-count/lib/

The running of the plugin should be stopped if one of the shell processes failed.

Allow to run Azkaban jobs

Gradle is an up-and-coming build system which grows in usage (particularily, in LinkedIn). Create a gradle plugin like the Maven one

Use cases:

1. Add -K to the ssh command to support Kerberos authentication
2. Override "rsync" command by a custom script to allow running on CYGWIN

Here is an example configuration for customizing the commands:

      <plugin>
        <configuration>
          ...
          <commands>
            <ssh.options>-K</ssh.options>
          </commands>

If there is already a job running, uploading the same jar with different parameters will change the jar in the distributed cache which results in an Exception

It would be great to have a smart system where a version of a jar is bumped, the old jar is deleted from the gateway. Otherwise we will have version incompatibility issues.

When the build is interrupted, the Hadoop job remains orphaned. This(these) job(s) need to be killed.

This can be done in the wrapper that would remember the time it started and upon exit, find all jobs started by this user. This is potentially dangerous as the user could have started jobs at the same time so we will also need configuration parameter, true