aazuspan / taskee Goto Github PK

Mobile and native notifications for Earth Engine tasks

License: MIT License

Python 100.00%

google-earth-engine task-manager tasks gee pushbullet android windows linux macos cli

taskee's Introduction

taskee is a command-line tool for monitoring Google Earth Engine tasks that can send notifications to your phone or computer to let you know when your tasks finish.

Features

🔍 Monitor Google Earth Engine tasks created with the Python API and/or the Javascript Code Editor
💻 Native notifications for Linux, Mac, and Windows
💬 Mobile push notifications for Android
📺 Built-in CLI dashboard

Setup

pip install taskee

Run the setup steps below as needed.

Earth Engine (required)

Set up a Google Earth Engine account. If you haven't authenticated Earth Engine before, you'll be asked to do so the first time you run taskee.

Pushbullet (optional)

If you want to receive mobile notifications (Android only), you'll need to create or connect an account with Pushbullet, download the app on your device(s), and install the Pushbullet Python API using pip install pushbullet.py. Once Pushbullet is installed and you're logged in, go to your Account Settings, create an Access Token, and copy the API key. The first time you run taskee with a pushbullet notifier, you'll need to enter your API key. That key will be stored locally so you don't have to enter it again.

notify-send (Linux only)

Linux users may need to install notify-send to enable native notifications. If taskee is not working with the native notifier, run sudo apt install libnotify-bin.

Usage

taskee offers a few different commands which you can see by running taskee --help.

Usage: taskee [OPTIONS] COMMAND [ARGS]...

  Monitor Earth Engine tasks and send notifications when they change states.

  Examples
      $ taskee test
      $ taskee tasks
      $ taskee start log
      $ taskee start dashboard failed completed -n pushbullet -i 0.5

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  start  Start running the notification system.
  tasks  Display a table of current Earth Engine tasks.
  test   Send test notifications.

Starting taskee

The taskee start command initializes and runs the notification system. It will continue running until you cancel it, periodically checking your Earth Engine tasks and sending you notifications.

taskee start requires one argument: mode.

Choosing a Mode

There are two modes that taskee start can run in: dashboard and log.

dashboard shows a live-updating dashboard that makes it easy to visually keep track of your tasks and events.

taskee start dashboard

log is designed to be run in the background and prints minimal logs as events occur.

taskee start log

Filtering Events

There are a lot of possible events that can happen to Earth Engine tasks. The list below describes the events recognized by taskee.

Event	Description
created	🌱 A new task is submitted.
started	🌿 A task starts processing.
attempted	🍂 An attempt fails and the task is restarted.
completed	🌲 A task finished successfully.
failed	🔥 A task fails to complete.
cancelled	🪓 The user cancels the task.
error	❗ `taskee` crashes.

By default, taskee will notify you of errors and completed or failed tasks, but you can specify which events to watch for by listing them when you launch taskee. For example:

taskee start dashboard failed attempted cancelled error

You can also use all as a shortcut and taskee will notify you of all events.

taskee start dashboard all

Selecting Notifiers

By default, taskee will use the native notification system built into your computer's operating system.

If you want notifications on other devices, set up Pushbullet and then select it with the -n --notifier option.

taskee start dashboard --notifier pushbullet

Like with events, you can use all as a shortcut and taskee will send both native and pushbullet notifications.

taskee start dashboard -n all

Other Options

You can set how often tasks are re-checked (in minutes) using the -i --interval_mins option.

taskee start dashboard -i 10

Warning
taskee doesn't set a minimum interval, but if updates occur too frequently you may run into rate limits for Earth Engine or Pushbullet.

Service Credentials

By default, taskee uses the persistent credentials stored on your local machine (whichever account you authenticated last). To use a service account instead, pass the relative path to your private key file using the -k --private-key option.

taskee start dashboard -k path/to/private-key.json

Example

Using what we learned above, let's set up taskee to start running in log mode, check for cancelled or completed task events, send us notifications using pushbullet, and update every 30 minutes.

taskee start log cancelled completed -n pushbullet -i 30

Other Commands

Aside from running the notification system, taskee has a few more commands that may be helpful.

Task Summaries

The tasks command checks your Earth Engine tasks once, giving you a snapshot of your current tasks in a nice, readable table.

taskee tasks

Test Notifications

The test command sends a mock notification to any notifiers selected with the -n --notifier option. You can use this to make sure notifications are set up and working.

taskee test -n native

taskee's People

Contributors

Stargazers

Watchers

taskee's Issues

Native notifications for desktop

Desktop notifications can be accessed with the Pushbullet app, but for users who want a simpler setup it would be nice to enable native notifications. One option for a cross-platform notification package is notify-py, but I need to do more research/testing to see if there's a better option.

Once desktop notifications are added, I should add options to allow users to choose which notification engine they want to use, e.g. native, pushbullet, or both.

Switch from `getTaskList` to `listOperations`

getTaskList was deprecated in 0.1.178 in favor of listOperations. Switching over will be non-trivial since we'll have to rework the task parsing. This might be a good time to bring in pydantic?

Report completed task runtime instead of total time

Notifications for completed tasks list the time since creation, rather than the active runtime. The latter is more consistent with the Code Editor interface and probably more helpful overall.

Handle unknown `Operation.type`

Unknown operation types crash taskee because they fail pydantic validation:

  File "C:\Users\Aaron\miniforge3\envs\ee\Lib\site-packages\taskee\cli\cli.py", line 121, in start_command
    raise e
  File "C:\Users\Aaron\miniforge3\envs\ee\Lib\site-packages\taskee\cli\cli.py", line 116, in start_command
    mode_func(t, interval_minutes=interval_mins)
  File "C:\Users\Aaron\miniforge3\envs\ee\Lib\site-packages\taskee\cli\commands\log.py", line 41, in start
    new_events = t.update()
                 ^^^^^^^^^^
  File "C:\Users\Aaron\miniforge3\envs\ee\Lib\site-packages\taskee\taskee.py", line 69, in update
    new_events = self._get_events()
                 ^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aaron\miniforge3\envs\ee\Lib\site-packages\taskee\taskee.py", line 94, in _get_events
    ops = tuple(Operation(**op) for op in ee.data.listOperations())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Aaron\miniforge3\envs\ee\Lib\site-packages\taskee\taskee.py", line 94, in <genexpr>
    ops = tuple(Operation(**op) for op in ee.data.listOperations())
                ^^^^^^^^^^^^^^^
 __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Operation
metadata.type
  Input should be 'EXPORT_FEATURES', 'EXPORT_IMAGE', 'EXPORT_VIDEO' or 'UNKNOWN' [type=enum, input_value='INGEST_TABLE', input_type=str]

We should add as many types as possible, but also make sure we can handle unexpected types.

Report how many tasks remaining

I typically start up taskee when I run a batch of tasks, and it's hard to keep track of what's completed and what's still running. To solve that, I think we could just report how many tasks are still running whenever a task completes, e.g. Task "test_02" completed successfully after 2 hours! (3 tasks remaining).

Allow service accounts and custom credentials

I'm running Python via a Miniconda3 environment, and I need to query the batch operations that were run under specific service account credentials. e.g., this works:

import ee
ee.Initialize(ee.ServiceAccountCredentials('[email protected]', 'secret.json'))
ee.data.listOperations()

How can I set credentials for taskee to use, or run it from Python interactive after my own call to ee.Initialize()?

Add rich-click

Use rich-click to improve the CLI help formatting.

Crash on image ingestion completed

At a glance, it looks like the auto-generated description for ingestion tasks is None, causing the string formatting to fail. The failure also prevents an error message from emitting.

Need to make sure we can handle empty descriptions and figure out why this error isn't getting dispatched.

  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\*****\Scripts\taskee.exe\__main__.py", line 7, in <module>
  File "C:\*****\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\*****\Lib\site-packages\rich_click\rich_command.py", line 126, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\*****\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\*****\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\*****\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\*****\Lib\site-packages\taskee\cli\cli.py", line 121, in start_command
    raise e
  File "C:\*****\Lib\site-packages\taskee\cli\cli.py", line 116, in start_command
    mode_func(t, interval_minutes=interval_mins)
  File "C:\*****\Lib\site-packages\taskee\cli\commands\log.py", line 42, in start
    t.dispatch()
  File "C:\*****\Lib\site-packages\taskee\taskee.py", line 80, in dispatch
    message = event.message
              ^^^^^^^^^^^^^
  File "C:\*****\Lib\site-packages\taskee\events.py", line 70, in message
    f"Task '{self.task.metadata.description}' completed successfully! "
TypeError: unsupported format string passed to NoneType.__format__

Add TUI

This is a long-term goal to create an interactive terminal interface using the Textual framework, allowing users to navigate through tasks and events using a tree structure.

`Completed` event missed for short-running task

I had a task that was created and completed within a single interval and only registered the Created event. It should have registered a Created and Completed event and sent notifications for both (ignoring for now that multiple events for a single task should maybe be packaged into a single, compound notification).

Incorrect event time in dashboard

#14 introduced a display bug where the event time in the dashboard is time since epoch time, so it shows 54 years.

Add notify.run notifier

Consider adding notify.run as an open-source alternative to Pushbullet. This is currently still Android/desktop only due to lack of support for the Web Push API in iOS, so the primary advantage would be that it is open source, doesn't require a login or a separate app, and doesn't have any monthly limits.