An HTTP job processing service.

Here the definition of a job is a collection of tasks, where each task has a name and a shell command. Tasks may depend on other tasks and require that those are executed beforehand. This service takes care of sorting the tasks to create a proper execution order.

{
   "tasks":[
      {
         "name":"task-1",
         "command":"touch /tmp/file1"
      },
      {
         "name":"task-2",
         "command":"cat /tmp/file1",
         "requires":[
            "task-3"
         ]
      },
      {
         "name":"task-3",
         "command":"echo 'Hello World!' > /tmp/file1",
         "requires":[
            "task-1"
         ]
      },
      {
         "name":"task-4",
         "command":"rm /tmp/file1",
         "requires":[
            "task-2",
            "task-3"
         ]
      }
   ]
}

The server can respond in two format: JSON and bash script in the form of text. The content will be negotiated via Accept header:

• Accept: application/json will give out a json response.
• Accept: text/plain will respond with the bash script in plain text.

The above example will be processed to this response:

[
   {
      "name":"task-1",
      "command":"touch /tmp/file1"
   },
   {
      "name":"task-3",
      "command":"echo 'Hello World!' > /tmp/file1"
   },
   {
      "name":"task-2",
      "command":"cat /tmp/file1"
   },
   {
      "name":"task-4",
      "command":"rm /tmp/file1"
   }
]

The bash script response will look like this:

#!/usr/bin/env bash

touch /tmp/file1
echo "Hello World!" > /tmp/file1
cat /tmp/file1
rm /tmp/file1

This way, it's possible to run it directly like so:

$ curl -d @mytasks.json -X POST -H 'Accept: text/plain' -H "Content-Type:application/json" http://localhost:4000/process_job | bash

Requirements

• elixir 1.8
• erlang 21.2

To start your Phoenix server:

• Install dependencies with mix deps.get
• Create and migrate your database with mix ecto.setup
• Start Phoenix endpoint with mix phx.server

• You can open production Swagger API documentation at https://job-processor.gigalixirapp.com/api/swagger.
• You can re-generate or update it by running mix phx.swagger.generate
• locally will be available at http://localhost:4000/api/swagger once phoenix is running.

• Run tests with mix test
• Run coverage with mix coveralls
• Codecov is being used as a coverage webpage.

Credo is a static code analysis tool for the Elixir language with a focus on teaching and code consistency.

• Run credo with mix credo

Dialyzer can analyze code with typespec to find type inconsistencies and possible bugs.

• Run dialyzer with mix dialyzer

Be aware this takes a while to run for the first time. Runs after the first will be a lot faster.

Code documentation generated with ex_doc:

• mix docs && open doc/index.html

The project is being built and tested in Circle CI.

Using Distillery:

• mix release to generate release.

Currently being deployed to production at gigalixir:

• https://job-processor.gigalixirapp.com/process_job

I made the assumption that authentication, authorization and persistence are outside the scope of this test.

For functionality exposed via HTTP, Phoenix seemed like a good choice. It has great productivity, facilitators, code generators and much more at the cost of very little overhead. I started a new phoenix app without frontend and also without Ecto, because of the above assumptions.

The problem of ordering tasks seemed to me to be in the domain of Graphs. Since elixir doesn't have this data structure native, I used a library.

Then, the basic idea was to build a directed acyclic graph, and to do a breadth-first scan to find all necessary tasks in an order that makes sense.

It looks something like this: