phovea / phovea_processing_queue Goto Github PK

View Code? Open in Web Editor NEW

0.0 7.0 1.0 216 KB

Process long-running tasks in the background using Celery

Home Page: http://caleydo.org/

License: BSD 3-Clause "New" or "Revised" License

Python 61.21% JavaScript 15.82% TypeScript 19.88% Dockerfile 3.09%

phovea phovea-server-plugin python celery

phovea_processing_queue's Introduction

phovea_processing_queue

Define long-running tasks that should be processed in the background using Celery.

Installation

git clone https://github.com/phovea/phovea_processing_queue.git
cd phovea_processing_queue
npm install

Testing

npm test

Building

npm run build

This repository is part of Phovea, a platform for developing web-based visualization applications. For tutorials, API docs, and more information about the build and deployment process, see the documentation page.

phovea_processing_queue's People

Contributors

Watchers

Forkers

ai4labs

phovea_processing_queue's Issues

Update Celery to 4.1.1

Creating the Celery Docker container with v4.0.1, the aysnc dependency was not found/installed.

WOrked fter updating to 4.1.1, probably due to it's deprecation: https://pypi.org/project/async/

Update requirements and dependencies

We should use the latest versions both for requirements and dependencies.
Except for Typescript, tslint and d3 dependencies, which will be updated in a separate step.

Update template files and dependencies

We should update the template files with the new generator-phovea v3 to update the dependencies.

Enable gif and webp files

See https://github.com/datavisyn/visyn2020/issues/40

Summary

enable gif and webp files where necessary

how to authenticate within celery tasks

e.g. to access private data

Update Codeowner

The code owner of this repository has changed, therefore it is necessary to update the repo.

Update dependencies

See https://github.com/datavisyn/visyn2020/issues/3

We should update all dependencies to the latest accessible versions, except for:

all webpack dependencies
all test dependencies
d3 and bootstrap

It might be necessary to update jQuery and React in a separate step as well.

Update requirements

See https://github.com/datavisyn/visyn2020/issues/9

We should update all requirements to the latest accessible versions.

Update to Typescript 3.8

Observed behavior

Updating the typescript and typedoc versions according to the snippet below without using stricter linting makes it possible to use the current typescript version with a minimum amount of errors.

"typedoc": "~0.16.9",
"typescript": "~3.8.1-rc",

Expected behavior

Updating the dependencies in the package.json and adding the following line of code in tsconfig.json (as well as fixing appearing type errors) should be sufficient for the usage of Typescript 3.8 as a first step.

"downlevelIteration": true, // required as long as target is `es5`

Add json stringify to build id

Observed behaviour

Build ID in webpack config causes errors if it is used without JSON.stringify()

switch to Dask?

see http://dask.pydata.org/en/latest/

Migrate to Jest

We should migrate from Karma to Jest for testing due to compatibility issues.

After the update to Typescript 3.8, we can use the latest version.
For the usage of ts-jest it is important to use a pinned version without ^. The reason for that is that its Major version follows Jest, which means that minor changes can be breaking. See the docs.

Update Celery to v4.4.0

Currently we are using Celery 4.2.2 with Python 3.7. The most recent release (as of today) is Celery 4.4.0. At the moment we could not switch to that release, as it requires redis>=3.2.0 and is in conflict with our phovea_data_redis redis~=2.10.6. Hence, we must update the redis version in phovea_data_redis first, and then update the celery version here.

create simple base task to execute R scripts

e.g. based on
https://rpy2.bitbucket.io/