imjoy-team / dev.imjoy.io Goto Github PK

ImJoy is a plugin powered hybrid computing platform for deploying deep learning applications such as advanced image analysis tools.

ImJoy runs on mobile and desktop environment cross different operating systems, plugins can run in the browser, localhost, remote and cloud servers.

With ImJoy, delivering Deep Learning tools to the end users is simple and easy thanks to its flexible plugin system and sharable plugin URL. Developer can easily add rich and interactive web interfaces to existing Python code.

Checkout the documentation for how to get started and more details for how to develop ImJoy plugins: ImJoy Docs

• Minimal and flexible plugin powered web application

• Server-less solution with offline support

• Support mobile devices

• Rich and interactive user interface powered by web technologies

  • use any existing web design libraries
  • Rendering multi-dimensional data in 3D with webGL, Three.js etc.

• Easy-to-use workflow composition

• Isolated workspaces for grouping plugins

• Self-contained plugin prototyping and development

  • Built-in code editor, no extra IDE is needed for development

• Powerful computational backend powered by the Python ecosystem

  • Support Javascript, native Python and web Python
  • Concurrent plugin execution through asynchronous programming
  • Run Python plugins in the browser with Webassembly
  • Browser plugins are isolated with secured sandboxes
  • Support async/await syntax for Python3 and Javascript
  • Support Conda virtual environments and pip packages for Python
  • Support libraries hosted on Github or CDNs for javascript
  • Easy plugin deployment and sharing through GitHub or Gist
  • Deploying your own plugin repository to Github

• Native support for n-dimensional arrays and tensors

  • Support ndarrays from Numpy for data exchange
  • Support Tensorflow.js and native Tensorflow for deep learning

The easiest way to try ImJoy is to start with a plugin running directly in the browser.

For example, you can install our Image Recognition plugin with this link. Clicking the link will open the ImJoy Web App and show a dialog to confirm the installation of the plugin. Once installed, you can launch the plugin by clicking the button Image Recognition in the plugin menu on the left. The demo plugin will run a trained deep neural network model (MobileNet) to perform image classification (e.g. identifying an elephant in an image).

This plugin uses your browser as its computational backend, so all computation is done locally, no data will be sent to a remote server.

Besides running plugins in the browser, ImJoy provides the flexibility to keep the GUI locally in your browser, and perform computational tasks with Python. These computations can be performed on your computer utilising the full power of the local GPU/CPU. The computational backend can also be launched on remote servers including cloud servers on Amazon, Google Compute, or an institutional computing cluster.

To learn more details about ImJoy, please go to ImJoy Docs.

By design, ImJoy itself provides very minimal task specific functionality. It provides a flexible plugin interface to support plugins to extend the user interface and perform computational tasks. Plugins can be designed to perform simple tasks such as reading tif file, or more complex tasks such as training a deep learning model for image segmentation.

Imjoy consists of three parts, each part can be extended with plugins:

1. Web User Interface. ImJoy is a web application. This means that using ImJoy is as easy as opening a web page from ImJoy.io. With ubiquitousness of the web, ImJoy provides unified user experience with browsers running on different operating systems, including mobile devices. The user can install plugins, and use them from the web interface. Besides a easy-to-use minimal interface, plugins can create their own window to create rich and interactive web interface by making use of the full power of the entire HTML/CSS/JS frameworks. For example, with D3.js or Three.js, one can easily provide powerful interactive charts or 3D visualisation plugins. For developers, the web interface is also used for writing and testing code. A stand-alone desktop application (desktopApp) is also provided.

2. Web Computational Backend. Computational tasks can be execution directly in the web browsers. Browsers are highly optimised and JavaScript engines such as Google Chrome V8 and Firefox Quantum can be used to perform computational tasks. The performance of JavaScript Engines are close or even better than native languages such as Java and Python (Benchmarks). With HTML5/WebGL, browser can access to GPUs, allowing transparent GPU computing with libraries such as Tensorflow.js. With a new web standard called WebAssembly, software/libraries written in high-level languages like C/C++/Rust can be ported to run in the browser. ImJoy uses "Pyodide" to run Python plugins directly in the browser. The advantage of using the web computational backend is it requires almost zero setup and can run on mobile devices. Importantly, browser provides a unified, secured and sandboxed environment with maximised security and stability.

3. Native Computational Backend. This is supported with the installation of an additional Python module called "ImJoy Plugin Engine". This allows to access the entire Python ecosystem which covers most of the scientific computing applications. With Conda, ImJoy plugin engine handles the requirements of plugins automatically and provide isolate processes and virtual environments for different Python plugins. It provides maximum flexibility and has full access to the file system, GPU and other local or remote resources. The plugin engine can be launched either on the local machine, or remotely on a cloud server or a cluster to perform computationally intensive tasks, e.g. with institutional computing cluster, Amazon Cloud, or Google Compute.

Currently, ImJoy consists of the following repositories:

• ImJoy Web App (core) (this repository)

• ImJoy Plugin Engine

• ImJoy Desktop App

• ImJoy Plugin Repository

• ImJoy Demo Plugins

• ImJoy Project Template

1. We provide a detailed ImJoy Docs. This includes a section with frequently asked questions.
2. For further help and questions for how to use ImJoy and process data with ImJoy, you can post on image.sc.
3. To report bugs or suggest enhancements for ImJoy itself, please file a GitHub issue.

This is a non-exhaustive list of the open-source tools and libraries we used in ImJoy:

• Joy.js (This is where the name ImJoy comes from!)
• Jailed (This helps ImJoy to isolate plugins)
• Vue.js (The main ImJoy App is written in Vue.js)
• vue-grid-layout (For window management)
• python-socketio (This how the plugin engine talk with the ImJoy app)
• pyodide (Enables web python mode with WebAssembly)
• conda (The plugin engine uses Conda for managing virtual env. and packages)
• docsify (The ImJoy documentation is created with docsify)

We thank all the authors for offering these excellent work to the open-source community.

Please star the ImJoy GitHub repository to support ImJoy.

We would like ImJoy to be a community driven framework, everyone is welcome to contribute your idea, feedback, plugins and code to the project.

We don't have a guideline for that yet, for now please feel free to use the issues and fork the project.

MIT License