Kelpie shepherds long-running jobs through to completion on interruptible hardware, coordinating with the Kelpie API

• 🐕 Kelpie (beta)
  • Who is it for?
  • Adding the kelpie Worker To Your Container Image
  • What it DOES NOT do
  • API Authorization
    • Base URL
  • Queueing a job
    • POST /jobs
  • Canceling a job
    • DELETE /jobs/:id
  • Checking on a job
    • GET /jobs/:id
  • Listing Your Jobs
    • GET /jobs
  • Job Lifecycle
  • Status Webhooks
    • Webhook Authorization

Kelpie is for anyone who wants to run long running compute-intensive jobs on Salad, whether that's LoRA training, Monte Carlo simulations, or anything else. You bring your own docker container that contains your script and dependencies, add the Kelpie binary to it, and deploy.

If you'd like to join the Kelpie beta, and are an existing Salad customer, just reach out to your point of contact via email, discord, or slack. If you're interested in Kelpie and are new to Salad, sign up for a demo, and mention you're interested in using Kelpie.

You can find a working example here

FROM yourimage:yourtag

# Install wget if it is not already present in your image
RUN apt-get update && install -y wget

# kelpie is a standalone x86-64 linux binary
RUN wget https://github.com/SaladTechnologies/kelpie/releases/download/0.4.2/kelpie -O /kelpie && chmod +x /kelpie

CMD ["/kelpie"]

When running the image, you will need additional configuration in the environment:

• AWS/Cloudflare Credentials: Provide AWS_ACCESS_KEY_ID, etc to enable the kelpie worker to upload and download from your bucket storage. We use the s3 compatability api, so any s3-compatible storage should work.
• KELPIE_API_URL: the root URL for the coordination API, e.g. kelpie.saladexamples.com
• KELPIE_API_KEY: Your api key for the coordination API, issued by Salad for use with kelpie. NOT your Salad API Key.

Additionally, your script must support the following things:

• Environment variables - If these are set by you in your container group configuration, they will be respected, otherwise they will be set by kelpie.
  • INPUT_DIR: Where to look for whatever data is needed as input. This will be downloaded from your bucket storage by kelpie prior to running the script.
  • CHECKPOINT_DIR: This is where to save progress checkpoints locally. kelpie will handle syncing the contents to your bucket storage, and will make sure any existing checkpoint is downloaded prior to running the script.
  • OUTPUT_DIR: This is where to save any output artifacts. kelpie will upload your artifacts to your bucket storage.
• Saving and Resuming From Checkpoints: Your script should periodically output progress checkpoints to CHECKPOINT_DIR, so that the job can be resumed if it gets interrupted. Similarly, when your script starts, it should check CHECKPOINT_DIR to see if there is anything to resume, and only start from the beginning if no checkpoint is present.
• It must exit "successfully" with an exit code of 0 upon completion.

1. kelpie does not store your data on our servers or in our storage buckets, beyond the job definition you submit. It merely facilitates syncing your data from local node storage to your preferred s3-compatible storage.
2. kelpie does not monitor the ongoing progress of your job, beyond ensuring it eventually exits successfully. You should integrate your own monitoring solution, e.g. Weights and Balances
3. kelpie does not containerize your job for you. It provides a binary that can be added to existing containerized jobs.
4. kelpie does not create or delete your container groups. If configured with scaling rules, kelpie can start, stop, and scale your container group in response to job volume.

There are live swagger docs that should be considered more accurate and up to date than this readme: https://kelpie.saladexamples.com/docs

Your kelpie api key is used by you to submit work, and also by kelpie workers to pull and process work.

All requests to the Kelpie API must include the header:

X-Kelpie-Key: myapikey

All API requests should use a base url of https://kelpie.saladexamples.com.

Queueing a job for processing is a simple post request to the Kelpie API

Request Body

{
  "command": "python",
  "arguments": [
    "/path/to/main.py",
    "--arg",
    "value"
  ],
  "environment": { "SOME_VAR": "string"},
  "input_bucket": "my-bucket",
  "input_prefix": "inputs/job1/",
  "checkpoint_bucket": "my-bucket",
  "checkpoint_prefix": "checkpoints/job1/",
  "output_bucket": "my-bucket",
  "output_prefix": "outputs/job1/",
  "webhook": "https://myapi.com/kelpie-webhooks",
  "container_group_id": "97f504e8-6de6-4322-b5d5-1777a59a7ad3"
}

Response Body

{
  "id": "8b9c902c-7da6-4af3-be0b-59cd4487895a",
  "user_id": "your-user-id",
  "status": "pending",
  "created": "2024-04-19T18:53:31.000Z",
  "started": null,
  "completed": null,
  "canceled": null,
  "failed": null,
  "command": "python",
  "arguments": [
    "/path/to/main.py",
    "--arg",
    "value"
  ],
  "environment": { "SOME_VAR": "string"},
  "input_bucket": "my-bucket",
  "input_prefix": "inputs/job1/",
  "checkpoint_bucket": "my-bucket",
  "checkpoint_prefix": "checkpoints/job1/",
  "output_bucket": "my-bucket",
  "output_prefix": "outputs/job1/",
  "webhook": "https://myapi.com/kelpie-webhooks",
  "heartbeat": null,
  "num_failures": 0,
  "container_group_id": "97f504e8-6de6-4322-b5d5-1777a59a7ad3",
  "machine_id": null
}

You can cancel a job using the job id

Response Body

{
  "message": "Job canceled"
}

As mentioned above, Kelpie does not monitor the progress of your job, but it does track the status (pending, running, canceled, completed, failed). You can get a job using the job id:

Response Body

{
  "id": "8b9c902c-7da6-4af3-be0b-59cd4487895a",
  "user_id": "your-user-id",
  "status": "pending",
  "created": "2024-04-19T18:53:31.000Z",
  "started": null,
  "completed": null,
  "canceled": null,
  "failed": null,
  "command": "python",
  "arguments": [
    "/path/to/main.py",
    "--arg",
    "value"
  ],
  "input_bucket": "my-bucket",
  "input_prefix": "inputs/job1/",
  "checkpoint_bucket": "my-bucket",
  "checkpoint_prefix": "checkpoints/job1/",
  "output_bucket": "my-bucket",
  "output_prefix": "outputs/job1/",
  "webhook": "https://myapi.com/kelpie-webhooks",
  "heartbeat": null,
  "num_failures": 0,
  "container_group_id": "97f504e8-6de6-4322-b5d5-1777a59a7ad3",
  "machine_id": null
}

Get your jobs in bulk.

Query Parameters

All query parameters for this endpoint are optional.

Response Body

{
  "_count": 1,
  "jobs": [
    {
      "id": "8b9c902c-7da6-4af3-be0b-59cd4487895a",
      "user_id": "your-user-id",
      "status": "pending",
      "created": "2024-04-19T18:53:31.000Z",
      "started": null,
      "completed": null,
      "canceled": null,
      "failed": null,
      "command": "python",
      "arguments": [
        "/path/to/main.py",
        "--arg",
        "value"
      ],
      "input_bucket": "my-bucket",
      "input_prefix": "inputs/job1/",
      "checkpoint_bucket": "my-bucket",
      "checkpoint_prefix": "checkpoints/job1/",
      "output_bucket": "my-bucket",
      "output_prefix": "outputs/job1/",
      "webhook": "https://myapi.com/kelpie-webhooks",
      "heartbeat": null,
      "num_failures": 0,
      "container_group_id": "97f504e8-6de6-4322-b5d5-1777a59a7ad3",
      "machine_id": null
    }
  ]
}

1. When kelpie starts on a new node, it starts polling for available work from /work. In these requests, it includes some information about what salad node you're on, including the machine id and container group id. This ensures we only hand out work to the correct container group, and that we do not hand out to a machine where that job has previously failed.
2. When a job is started, a webhook is sent, if configured.
3. Once it receives a job, kelpie downloads your inputs, and your checkpoint
4. Once required files are downloaded, kelpie executes your command with the provided arguments, adding environment variables as documented above.
5. Whenever files are added to the checkpoint directory, kelpie syncs the directory to the checkpoint bucket and prefix.
6. Whenever files are added to the output directory, kelpie syncs the directory to the output bucket and prefix.
7. When your command exits 0, the job is marked as complete, and a webhook is sent (if configured) to notify you about the job's completion.
   1. If your job fails, meaning exits non-0, it will be reported as a failure to the api. When this occurs, the number of failures for the job is incremented, up to 3. The machine id reporting the failure will be blocked from receiving that job again. If the job fails 3 times, it is marked failed, and a webhook is sent, if configured. If a machine id is blocked from 5 jobs, the container will be reallocated to a different machine, provided you have added the kelpie user to your salad org.
8. input, checkpoint, and output directories are purged, and the cycle begins again

If you provide a url in the webhook field, the Kelpie API will send status webhooks. It makes a POST request to the url provided, with a JSON request body:

{
  "status": "running",
  "job_id": "some-job-id",
  "machine_id": "some-machine-id",
  "container_group_id": "some-container-group-id"
}

Webhook status may be running, failed, or completed

Webhooks sent by the Kelpie API will be secured with your API token in the X-Kelpie-Key header.