Git Product home page Git Product logo

dd_platform_docker's Introduction

DeepDetect Platform Docker

Installation

# Setup platform project name, directory and architecture
export PROJECT_NAME=dd_platform
export DD_PLATFORM=$HOME/deepdetect
export ARCH=cpu # gpu also available

# Install platform repository
git clone https://github.com/jolibrain/dd_platform_docker.git \
    ${DD_PLATFORM}

# Go to directory
cd ${DD_PLATFORM}/code/${ARCH}/

# Start platform docker containers
CURRENT_UID=$(id -u):$(id -g) MUID=$(id -u) \
    docker-compose -p $PROJECT_NAME up -d

# Go to http://localhost:1912

Connecting to an existing DeepDetect server

If you need to connect a new DeepDetect server to the platform, you must modify the nginx configuration to access to this server, and the platform-ui configuration to update the UI.

config/nginx/nginx.conf

Set a new upstream in the http section:

upstream UPSTREAM_NAME {
	least_conn;
	server DD_HOST:DD_PORT max_fails=3 fail_timeout=30s;
}

Set a new location in the server section:

location ~ /api/SERVER_NAME/(?<url>.*) {
	proxy_pass http://UPSTREAM_NAME/$url$is_args$args;
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection 'upgrade';
	proxy_set_header Host $host;
	proxy_cache_bypass $http_upgrade;
}

config/platform_ui/config.json

In deepdetect.servers array, add the following item:

{
  "name": "SERVER_NAME",
  "settings": {
    "path": "/api/SERVER_NAME",
    "isWritable": true
  }
}

The flag isWritable will allow you to create and delete services on this server from the platform_ui web interface.

Monitoring another GPU

In order to monitor a GPU, you'll need to modify the platform_ui config file to update the interface, and the nginx config to let the interface access it.

config/nginx/nginx.conf

Set a new upstream in the http section:

upstream gpu_stat_GPU_NAME {
  least_conn;
  server GPU_STATS_HOST:GPU_STATS_PORT max_fails=3 fail_timeout=30s;
}

Usually, GPU_STATS_PORT is 12345

Set a new location in the server section:

location ~ /gpu/ENDPOINT {
  proxy_pass http://gpu_stat_GPU_NAME/$url$is_args$args;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection 'upgrade';
  proxy_set_header Host $host;
  proxy_cache_bypass $http_upgrade;
}

config/platform_ui/config.json

In gpuInfo.servers array, add the following item:

{
  "name": "GPU_NAME",
  "url": "/gpu/ENDPOINT"
}

Configuration of docker image type and versions

Docker-compose uses the latest tag for all docker images that correspond to the latest stable version.

You can select the wanted version with these following environment variables:

DD_SERVER_IMAGE=gpu_torch
DD_SERVER_TAG=v0.14.0
DD_PLATFORM_UI_TAG=latest
DD_JUPYTER_TAG=latest
DD_FILEBROWSER_TAG=latest

Allow direct connection to deepdetect API

You can add to deepdetect container of code/gpu/docker-compose.yml:

    ports:
      - "127.0.0.0:12345:8080"

dd_platform_docker's People

Contributors

alx avatar beniz avatar bycob avatar dependabot[bot] avatar sileht avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

dd_platform_docker's Issues

Port forwarding issue

What should I change in the file when doing port forwarding?
In the file, only the nginx related content has the port setting.
When I modified the facade part with the port I want to configure, it doesn't work to be port forwarding.

And also, Jupiter errors also appear in the docker log contents. It is also restarting when nginx checks with docker ps.

[BUG] DDPlatform not starting

Hi,

I try to install and execute the DDPlatform on Debian 10
I have follow the instructions on the readme, but 1 container is not starting (dd_platform_platform_ui_1).

root@deepdetect:~# docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS                          PORTS               NAMES
20b35fb17edd        jolibrain/platform_ui:v0.16.0      "/docker-entrypoint.…"   44 hours ago        Restarting (1) 39 seconds ago                       dd_platform_platform_ui_1
a313b4d8d8fa        jolibrain/deepdetect_cpu:v0.16.0   "/bin/sh -c '/opt/de…"   44 hours ago        Up 44 hours                     8080/tcp            dd_platform_deepdetect_1
60d556b61334        jolibrain/filebrowser:latest       "/filebrowser -d /da…"   44 hours ago        Up 44 hours                     80/tcp              dd_platform_filebrowser_1
9a784633337b        jolibrain/ouroboros                "ouroboros"              44 hours ago        Up 44 hours                                         dd_platform_ouroboros_1
5d06e3cd4505        amir20/dozzle                      "/dozzle"                44 hours ago        Up 44 hours                     8080/tcp            dd_platform_dozzle_1

In the logs of the container :

docker logs -f dd_platform_platform_ui_1
2021/05/06 08:55:44 [emerg] 1#1: host not found in upstream "jupyter:8888" in /etc/nginx/nginx.conf:55
nginx: [emerg] host not found in upstream "jupyter:8888" in /etc/nginx/nginx.conf:55

Also the update.sh is not working :

root@deepdetect:~/deepdetect/code/cpu# sh update.sh
update.sh: 8: update.sh: function: not found
-e example with *another_cpu* project name: ./update.sh -p another_cpu

update.sh: 10: update.sh: Syntax error: "}" unexpected

Thanks.

Missing images in test+train "cars" examples

examples/cars/test.txt missing image files in img/

imgs//youtube_frames/toronto-main-street-000262.jpg
imgs//youtube_frames/vancouver-000854.jpg
imgs//youtube_frames/vancouver-000776.jpg
imgs//youtube_frames/panama_1482_000030.jpg
imgs//youtube_frames/vancouver-000537.jpg
imgs//youtube_frames/vancouver-000732.jpg
imgs//youtube_frames/vancouver-000840.jpg
imgs//youtube_frames/vancouver-000842.jpg
imgs//youtube_frames/vancouver-001067.jpg
imgs//youtube_frames/panama_1482_000730.jpg
imgs//youtube_frames/vancouver-001046.jpg
imgs//youtube_frames/vancouver-000751.jpg
imgs//youtube_frames/panama_1482_000501.jpg
imgs//youtube_frames/vancouver-000836.jpg
imgs//youtube_frames/toronto-000021.jpg
imgs//youtube_frames/vancouver-000499.jpg
imgs//youtube_frames/vancouver-000770.jpg
imgs//youtube_frames/toronto-000026.jpg
imgs//youtube_frames/vancouver-001069.jpg
imgs//youtube_frames/panama_1482_001100.jpg
imgs//youtube_frames/vancouver-000718.jpg
imgs//youtube_frames/vancouver-000247.jpg
imgs//youtube_frames/vancouver-000841.jpg
imgs//youtube_frames/vancouver-000681.jpg
imgs//youtube_frames/vancouver-001097.jpg
imgs//youtube_frames/vancouver-000263.jpg
imgs//youtube_frames/toronto-000011.jpg
imgs//youtube_frames/toronto-main-street-001219.jpg
imgs//youtube_frames/toronto-000033.jpg
imgs//youtube_frames/vancouver-001027.jpg
imgs//youtube_frames/toronto-000008.jpg
imgs//youtube_frames/toronto-main-street-000795.jpg
imgs//youtube_frames/vancouver-000755.jpg

missing in train.txt

imgs//youtube_frames/toronto-main-street-000147.jpg
imgs//youtube_frames/toronto-main-street-000051.jpg
imgs//youtube_frames/vancouver-000837.jpg
imgs//youtube_frames/vancouver-000630.jpg
imgs//youtube_frames/vancouver-000283.jpg
imgs//youtube_frames/toronto-main-street-000289.jpg
imgs//youtube_frames/toronto-main-street-000919.jpg
imgs//youtube_frames/toronto-main-street-000645.jpg
imgs//youtube_frames/vancouver-000632.jpg
imgs//youtube_frames/vancouver-000823.jpg
imgs//youtube_frames/vancouver-000825.jpg
imgs//youtube_frames/vancouver-000120.jpg
imgs//youtube_frames/toronto-main-street-000050.jpg
imgs//youtube_frames/vancouver-000534.jpg
imgs//youtube_frames/vancouver-001055.jpg
imgs//youtube_frames/vancouver-000258.jpg
imgs//youtube_frames/toronto-main-street-000601.jpg
imgs//youtube_frames/toronto-main-street-001046.jpg
imgs//youtube_frames/toronto-000013.jpg
imgs//youtube_frames/vancouver-000117.jpg
imgs//youtube_frames/vancouver-000762.jpg
imgs//youtube_frames/vancouver-000754.jpg
imgs//youtube_frames/out-000188.jpg
imgs//youtube_frames/toronto-main-street-001040.jpg
imgs//youtube_frames/vancouver-000782.jpg
imgs//youtube_frames/vancouver-000834.jpg
imgs//youtube_frames/vancouver-000800.jpg
imgs//youtube_frames/vancouver-000265.jpg
imgs//youtube_frames/vancouver-000009.jpg
imgs//youtube_frames/vancouver-001049.jpg
imgs//youtube_frames/toronto-000015.jpg
imgs//youtube_frames/vancouver-000331.jpg
imgs//youtube_frames/vancouver-000500.jpg
imgs//youtube_frames/vancouver-000759.jpg
imgs//youtube_frames/vancouver-000844.jpg
imgs//youtube_frames/vancouver-000011.jpg
imgs//youtube_frames/vancouver-000846.jpg
imgs//youtube_frames/vancouver-000096.jpg
imgs//youtube_frames/vancouver-000542.jpg
imgs//youtube_frames/toronto-main-street-001000.jpg
imgs//youtube_frames/vancouver-000597.jpg
imgs//youtube_frames/toronto-main-street-000269.jpg
imgs//youtube_frames/vancouver-001054.jpg
imgs//youtube_frames/toronto-main-street-000111.jpg
imgs//youtube_frames/vancouver-000529.jpg
imgs//youtube_frames/vancouver-000848.jpg
imgs//youtube_frames/vancouver-000824.jpg
imgs//youtube_frames/vancouver-000438.jpg
imgs//youtube_frames/panama_1482_000029.jpg
imgs//youtube_frames/vancouver-001050.jpg
imgs//youtube_frames/vancouver-001065.jpg
imgs//youtube_frames/panama_1482_000084.jpg
imgs//youtube_frames/toronto-main-street-000135.jpg
imgs//youtube_frames/vancouver-000731.jpg
imgs//youtube_frames/vancouver-000627.jpg
imgs//youtube_frames/vancouver-000778.jpg
imgs//youtube_frames/toronto-main-street-000520.jpg
imgs//youtube_frames/vancouver-000797.jpg
imgs//youtube_frames/vancouver-000659.jpg
imgs//youtube_frames/toronto-main-street-000893.jpg
imgs//youtube_frames/toronto-main-street-001215.jpg
imgs//youtube_frames/toronto-main-street-001216.jpg
imgs//youtube_frames/vancouver-000727.jpg
imgs//youtube_frames/panama_1482_000043.jpg
imgs//youtube_frames/toronto-main-street-001222.jpg
imgs//youtube_frames/panama_1482_000530.jpg
imgs//youtube_frames/toronto-main-street-000019.jpg
imgs//youtube_frames/panama_1482_000492.jpg
imgs//youtube_frames/vancouver-000783.jpg
imgs//youtube_frames/vancouver-000849.jpg
imgs//youtube_frames/vancouver-000524.jpg
imgs//youtube_frames/vancouver-000658.jpg
imgs//youtube_frames/vancouver-001056.jpg
imgs//youtube_frames/vancouver-000434.jpg
imgs//youtube_frames/panama_1482_000798.jpg
imgs//youtube_frames/toronto-main-street-000392.jpg
imgs//youtube_frames/vancouver-000488.jpg
imgs//youtube_frames/vancouver-000153.jpg
imgs//youtube_frames/vancouver-000002.jpg
imgs//youtube_frames/toronto-000009.jpg
imgs//youtube_frames/vancouver-000749.jpg
imgs//youtube_frames/vancouver-000750.jpg
imgs//youtube_frames/toronto-main-street-000270.jpg
imgs//youtube_frames/vancouver-000585.jpg
imgs//youtube_frames/vancouver-001052.jpg
imgs//youtube_frames/panama_1482_000085.jpg
imgs//youtube_frames/vancouver-000102.jpg
imgs//youtube_frames/vancouver-001059.jpg
imgs//youtube_frames/out-000161.jpg
imgs//youtube_frames/panama_1482_000092.jpg
imgs//youtube_frames/vancouver-000653.jpg
imgs//youtube_frames/toronto-main-street-000071.jpg
imgs//youtube_frames/vancouver-001044.jpg
imgs//youtube_frames/vancouver-000167.jpg
imgs//youtube_frames/vancouver-000839.jpg
imgs//youtube_frames/vancouver-000273.jpg
imgs//youtube_frames/vancouver-000125.jpg
imgs//youtube_frames/vancouver-000173.jpg
imgs//youtube_frames/vancouver-000631.jpg
imgs//youtube_frames/vancouver-000115.jpg
imgs//youtube_frames/vancouver-000827.jpg
imgs//youtube_frames/toronto-main-street-001128.jpg
imgs//youtube_frames/panama_1482_000057.jpg
imgs//youtube_frames/vancouver-000496.jpg
imgs//youtube_frames/vancouver-000543.jpg
imgs//youtube_frames/vancouver-001061.jpg
imgs//youtube_frames/vancouver-000523.jpg
imgs//youtube_frames/vancouver-000789.jpg
imgs//youtube_frames/vancouver-001053.jpg
imgs//youtube_frames/vancouver-000757.jpg
imgs//youtube_frames/vancouver-000758.jpg
imgs//youtube_frames/vancouver-001003.jpg
imgs//youtube_frames/vancouver-000753.jpg
imgs//youtube_frames/toronto-main-street-000024.jpg
imgs//youtube_frames/toronto-main-street-000268.jpg
imgs//youtube_frames/panama_1482_000477.jpg
imgs//youtube_frames/toronto-main-street-000107.jpg
imgs//youtube_frames/toronto-main-street-000007.jpg
imgs//youtube_frames/toronto-main-street-000220.jpg
imgs//youtube_frames/out-000178.jpg
imgs//youtube_frames/vancouver-000795.jpg
imgs//youtube_frames/vancouver-001071.jpg
imgs//youtube_frames/vancouver-000804.jpg
imgs//youtube_frames/panama_1482_000799.jpg
imgs//youtube_frames/toronto-main-street-001043.jpg
imgs//youtube_frames/toronto-main-street-000915.jpg
imgs//youtube_frames/vancouver-001068.jpg
imgs//youtube_frames/vancouver-000798.jpg
imgs//youtube_frames/toronto-main-street-000916.jpg
imgs//youtube_frames/vancouver-000678.jpg
imgs//youtube_frames/vancouver-000421.jpg
imgs//youtube_frames/out-000150.jpg
imgs//youtube_frames/panama_1482_000708.jpg
imgs//youtube_frames/vancouver-000482.jpg
imgs//youtube_frames/toronto-main-street-000001.jpg
imgs//youtube_frames/toronto-000234.jpg
imgs//youtube_frames/panama_1482_000042.jpg
imgs//youtube_frames/vancouver-001001.jpg
imgs//youtube_frames/vancouver-000533.jpg
imgs//youtube_frames/panama_1482_000489.jpg
imgs//youtube_frames/vancouver-001057.jpg
imgs//youtube_frames/toronto-main-street-000171.jpg
imgs//youtube_frames/panama_1482_000608.jpg
imgs//youtube_frames/vancouver-000832.jpg
imgs//youtube_frames/vancouver-000662.jpg
imgs//youtube_frames/toronto-main-street-000129.jpg
imgs//youtube_frames/panama_1482_000104.jpg
imgs//youtube_frames/vancouver-000330.jpg
imgs//youtube_frames/vancouver-000626.jpg
imgs//youtube_frames/vancouver-000664.jpg
imgs//youtube_frames/vancouver-000435.jpg
imgs//youtube_frames/vancouver-000012.jpg
imgs//youtube_frames/vancouver-000669.jpg
imgs//youtube_frames/panama_1482_000106.jpg
imgs//youtube_frames/toronto-main-street-000039.jpg
imgs//youtube_frames/vancouver-000249.jpg
imgs//youtube_frames/vancouver-000989.jpg
imgs//youtube_frames/vancouver-001045.jpg
imgs//youtube_frames/vancouver-000752.jpg
imgs//youtube_frames/vancouver-000682.jpg
imgs//youtube_frames/vancouver-000835.jpg
imgs//youtube_frames/toronto-main-street-001042.jpg
imgs//youtube_frames/vancouver-000190.jpg
imgs//youtube_frames/vancouver-000855.jpg
imgs//youtube_frames/toronto-main-street-000775.jpg
imgs//youtube_frames/panama_1482_000014.jpg
imgs//youtube_frames/vancouver-000527.jpg
imgs//youtube_frames/vancouver-000498.jpg
imgs//youtube_frames/vancouver-000095.jpg
imgs//youtube_frames/vancouver-001058.jpg
imgs//youtube_frames/panama_1482_001036.jpg
imgs//youtube_frames/panama_1482_000697.jpg
imgs//youtube_frames/vancouver-000544.jpg
imgs//youtube_frames/vancouver-001064.jpg
imgs//youtube_frames/vancouver-000853.jpg
imgs//youtube_frames/vancouver-001011.jpg
imgs//youtube_frames/vancouver-000432.jpg
imgs//youtube_frames/vancouver-000591.jpg
imgs//youtube_frames/vancouver-000535.jpg
imgs//youtube_frames/vancouver-000484.jpg
imgs//youtube_frames/vancouver-000773.jpg
imgs//youtube_frames/toronto-000012.jpg
imgs//youtube_frames/panama_1482_000586.jpg
imgs//youtube_frames/toronto-000014.jpg
imgs//youtube_frames/toronto-main-street-000964.jpg
imgs//youtube_frames/vancouver-000010.jpg
imgs//youtube_frames/vancouver-000766.jpg
imgs//youtube_frames/panama_1482_000628.jpg
imgs//youtube_frames/vancouver-001070.jpg
imgs//youtube_frames/toronto-000051.jpg
imgs//youtube_frames/panama_1482_000055.jpg
imgs//youtube_frames/vancouver-000519.jpg
imgs//youtube_frames/vancouver-000660.jpg
imgs//youtube_frames/toronto-main-street-000271.jpg
imgs//youtube_frames/vancouver-000761.jpg
imgs//youtube_frames/panama_1482_000010.jpg
imgs//youtube_frames/vancouver-000661.jpg
imgs//youtube_frames/panama_1482_000734.jpg
imgs//youtube_frames/vancouver-000629.jpg
imgs//youtube_frames/vancouver-000257.jpg
imgs//youtube_frames/toronto-main-street-001087.jpg
imgs//youtube_frames/vancouver-000833.jpg
imgs//youtube_frames/vancouver-000734.jpg
imgs//youtube_frames/panama_1482_000011.jpg
imgs//youtube_frames/panama_1482_000594.jpg
imgs//youtube_frames/vancouver-000433.jpg
imgs//youtube_frames/vancouver-000650.jpg
imgs//youtube_frames/toronto-main-street-000859.jpg
imgs//youtube_frames/vancouver-000801.jpg
imgs//youtube_frames/vancouver-000172.jpg
imgs//youtube_frames/vancouver-000477.jpg
imgs//youtube_frames/toronto-main-street-001214.jpg
imgs//youtube_frames/panama_1482_000832.jpg
imgs//youtube_frames/vancouver-000651.jpg
imgs//youtube_frames/out-000155.jpg
imgs//youtube_frames/panama_1482_000709.jpg
imgs//youtube_frames/vancouver-000746.jpg
imgs//youtube_frames/vancouver-000250.jpg
imgs//youtube_frames/panama_1482_000091.jpg
imgs//youtube_frames/vancouver-000414.jpg
imgs//youtube_frames/vancouver-000756.jpg
imgs//youtube_frames/toronto-main-street-000099.jpg
imgs//youtube_frames/panama_1482_000013.jpg
imgs//youtube_frames/toronto-main-street-000825.jpg
imgs//youtube_frames/vancouver-000480.jpg
imgs//youtube_frames/toronto-main-street-000346.jpg
imgs//youtube_frames/toronto-000019.jpg
imgs//youtube_frames/vancouver-001063.jpg
imgs//youtube_frames/vancouver-000774.jpg
imgs//youtube_frames/vancouver-000777.jpg
imgs//youtube_frames/panama_1482_000093.jpg
imgs//youtube_frames/toronto-000006.jpg
imgs//youtube_frames/toronto-main-street-001009.jpg
imgs//youtube_frames/vancouver-000100.jpg
imgs//youtube_frames/toronto-main-street-000032.jpg
imgs//youtube_frames/toronto-main-street-000162.jpg
imgs//youtube_frames/vancouver-000845.jpg
imgs//youtube_frames/vancouver-000014.jpg
imgs//youtube_frames/vancouver-000772.jpg
imgs//youtube_frames/vancouver-000847.jpg
imgs//youtube_frames/vancouver-001051.jpg
imgs//youtube_frames/vancouver-000437.jpg
imgs//youtube_frames/vancouver-000583.jpg
imgs//youtube_frames/out-000177.jpg
imgs//youtube_frames/vancouver-000779.jpg

Jupyter Python Kernel 404

The entire log of cpu_jupyter_1 (jolibrain/jupyter_dd_notebook) is variations on this 404.

[W 15:54:12.290 NotebookApp] 404 GET /api/kernels/da74ed0a-187c-40ce-9030-e126c549a155/channels?session_id=faeefe8c77a322255b5d384987f79eeb (172.20.0.8) 1.49ms referer=None

Cells never leave the busy [*] state when run.
Screen Shot 2020-09-13 at 12 00 18 PM

Manifest missing!?

$ sh update.sh
Already up to date.
Pulling platform_ui (jolibrain/platform_ui:v0.10.0)...
v0.10.0: Pulling from jolibrain/platform_ui
Digest: sha256:d10316d36b5e3215af71cc17f7f5e5660f49b126a734eae1c5ffc093f2ec6e32
Status: Image is up to date for jolibrain/platform_ui:v0.10.0
Pulling deepdetect (jolibrain/deepdetect_cpu:v0.11.0)...
manifest for jolibrain/deepdetect_cpu:v0.11.0 not found: manifest unknown: manifest unknown

Docker containers not updating

I ran sh update.sh as detailed here, but at least the platform_ui container is not updating.

Screen Shot 2020-09-21 at 9 44 05 PM

My git log indicates that the lastest version of the repo was pulled:
Screen Shot 2020-09-21 at 9 45 25 PM

I even removed the volumes entirely with docker-compose rm -f -s -v and re-installed to no effect.

CPU Platform UI constantly makes request for GPU States

I'm using the CPU container, which does not have a GPU. The Platform UI at localhost:1912 has a section that displays the GPU stats (figure 1). However, there are no GPU states which results in a constant stream of 404s (figure 2).
Screen Shot 2020-09-22 at 9 55 28 PM

Screen Shot 2020-09-22 at 9 55 19 PM

Disregard the 502s for this issue.

Launch with Docker Compose failed

Hi Jolibrain Team,

I'm looking into Deepdetect right now, but I have problems starting the Plattform via docker compose, as it is shown in your example on the frontpage.

I use Ubuntu 18.04 with the docker from ubuntus repos, and docker-compose from the official site.

But the nginx crashes all the time:

2019/05/17 12:46:37 [emerg] 1#1: host not found in upstream "jupyter:8888" in /etc/nginx/nginx.conf:50
nginx: [emerg] host not found in upstream "jupyter:8888" in /etc/nginx/nginx.conf:50
2019/05/17 12:46:38 [emerg] 1#1: host not found in upstream "jupyter:8888" in /etc/nginx/nginx.conf:50
nginx: [emerg] host not found in upstream "jupyter:8888" in /etc/nginx/nginx.conf:50
2019/05/17 12:46:40 [emerg] 1#1: host not found in upstream "jupyter:8888" in /etc/nginx/nginx.conf:50
nginx: [emerg] host not found in upstream "jupyter:8888" in /etc/nginx/nginx.conf:50

I can't find the jupyter running:

CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS                          PORTS               NAMES
0dc08ea76fd6        nginx:latest                      "nginx -g 'daemon of…"   3 minutes ago       Restarting (1) 58 seconds ago                       jolibrain_nginx
badbc8101e15        jolibrain/ouroboros               "ouroboros"              11 minutes ago      Up 3 minutes                                        badbc8101e15_jolibrain_ouroboros
1f258dd37c5f        jolibrain/platform_ui:latest      "nginx -g 'daemon of…"   11 minutes ago      Up 11 minutes                   80/tcp              jolibrain_platform_ui
0baf52f72da8        jolibrain/platform_data:latest    "rsync -avz /tmp/dat…"   11 minutes ago      Exited (0) 2 minutes ago                            0baf52f72da8_jolibrain_platform_data
34ed2b00548e        jolibrain/deepdetect_cpu          "/bin/sh -c './dede …"   11 minutes ago      Up 3 minutes                    8080/tcp            34ed2b00548e_jolibrain_deepdetect
c3630536dc82        jolibrain/jupyter_dd_notebook     "tini -g -- start-no…"   11 minutes ago      Exited (4) 3 minutes ago                            c3630536dc82_jolibrain_jupyter
f3aefa588c27        filebrowser/filebrowser:v1.10.0   "/filebrowser --conf…"   11 minutes ago      Up 3 minutes                    80/tcp              f3aefa588c27_jolibrain_filebrowser

Logs don't help much: docker logs c3630536dc82

/usr/local/bin/start-notebook.sh: ignoring /usr/local/bin/start-notebook.d/*

Set username to: jovyan
usermod: no changes
Set jovyan UID to: 0
usermod: UID '0' already exists
/usr/local/bin/start-notebook.sh: ignoring /usr/local/bin/start-notebook.d/*

Set username to: jovyan
usermod: no changes
Set jovyan UID to: 0
usermod: UID '0' already exists

I followed this guide for cpu:
https://www.deepdetect.com/quickstart-platform/

The system it runs on is a fresh VM.

unable to install the platform on Window

Hi,

I tried to install the dd_platform under Windows, because it is our development environment.

So, to do this:
1 - I set the DD_PLATFORM, CURRENT_UID, MUID env variables. I used '1000' for MUID by example.
2 - I use the following command for the install: docker-compose up -d

then, everything seems correct until I got the following error message:

"ERROR: write /var/lib/docker/tmp/GetImageBlob264633635: no space left on device"

Any idea?

Unable to start the service

$ docker-compose -version
docker-compose version 1.25.1, build a82fef0
$ CURRENT_UID=$(id -u):$(id -g) MUID=$(id -u) docker-compose up -d
Creating network "gpu_default" with the default driver
Creating jolibrain_gpustat_server ... done
Creating jolibrain_platform_ui    ... done
Creating jolibrain_deepdetect     ... done
Creating jolibrain_jupyter        ... done
Creating jolibrain_platform_data  ... done
Creating jolibrain_ouroboros      ... done
Creating jolibrain_filebrowser    ... done
Creating jolibrain_nginx          ... done
$ docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS                         PORTS               NAMES
e82fe5b475df        nginx:latest                    "nginx -g 'daemon of…"   11 seconds ago      Restarting (1) 2 seconds ago                       jolibrain_nginx
229cdf56f625        jolibrain/ouroboros             "ouroboros"              15 seconds ago      Up 12 seconds                                      jolibrain_ouroboros
fe33e721fa94        jolibrain/deepdetect_gpu        "/bin/sh -c './dede …"   15 seconds ago      Up 11 seconds                  8080/tcp            jolibrain_deepdetect
425293d06154        jolibrain/jupyter_dd_notebook   "tini -g -- start-no…"   15 seconds ago      Up 12 seconds                  8888/tcp            jolibrain_jupyter
13effd622627        jolibrain/platform_ui:latest    "nginx -g 'daemon of…"   15 seconds ago      Up 13 seconds                  80/tcp              jolibrain_platform_ui
f63136d11be3        jolibrain/gpustat_server        "python3 -m gpustat_…"   15 seconds ago      Up 11 seconds                  12345/tcp           jolibrain_gpustat_server
$ docker logs -f jolibrain_nginx
2020/01/10 17:18:20 [emerg] 1#1: host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
nginx: [emerg] host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
2020/01/10 17:18:21 [emerg] 1#1: host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
nginx: [emerg] host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
2020/01/10 17:18:22 [emerg] 1#1: host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
nginx: [emerg] host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
2020/01/10 17:18:23 [emerg] 1#1: host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
nginx: [emerg] host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
2020/01/10 17:18:25 [emerg] 1#1: host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
nginx: [emerg] host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
2020/01/10 17:18:27 [emerg] 1#1: host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
nginx: [emerg] host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
2020/01/10 17:18:31 [emerg] 1#1: host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
nginx: [emerg] host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
2020/01/10 17:18:38 [emerg] 1#1: host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
nginx: [emerg] host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
2020/01/10 17:18:52 [emerg] 1#1: host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
nginx: [emerg] host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
2020/01/10 17:19:18 [emerg] 1#1: host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40
nginx: [emerg] host not found in upstream "filebrowser:8888" in /etc/nginx/nginx.conf:40

Out of date dd_client.py installed by CPU image

The version of dd_client.py that is installed with the CPU image is out of date. Specifically, it still uses the async keyword as a function parameter:
def post_train(self, sname, data, parameters_input, parameters_mllib, parameters_output, async=True):

I noticed that the version of dd_client.py on the main repo has already been fixed.

Generate downloadable tgz on server for each model

When a new model is created after a training job, it should also generate a tgz archive that user could access/download.

Here is a solution using a new Docker container to periodically scan model folders, and create a tgz archive in these folders if it's missing.

FROM alpine:3.7
COPY createDownloadableService /etc/periodic/hourly/createDownloadableService
RUN chmod +x /etc/periodic/hourly/createDownloadableService
CMD crond -l 2 -f
  • createDownloadableService script:
#!/bin/sh
SERVICE_PATH=/opt/platform/models/public

for f in $SERVICE_PATH/*; do
    if [[ -d "$f" ]]; then
    # $f is a directory
        
        MODEL_NAME="$(basename $f)"
        MODEL_ARCHIVE="$f/$MODEL_NAME.tar.gz"

        if [[ ! -f "$MODEL_ARCHIVE" ]]; then
           # if $f/$MODEL_NAME.tar.gz doesn't exist
           # create it with content from $f
           tar -zcf $MODEL_ARCHIVE -C $f .
        fi
    fi
done

update.sh script should not restart deepdetect containers

DeepDetect containers might be running training jobs during the update process.

With the current update.sh script to update the platform containers, DeepDetect containers are restarted, and training jobs are killed.

A fix should be found to update the platform without restarting deepdetect container (they don't depend on the platform update)

502 Bad Gateway Error when sending more than one image with dd_client.post_predict

Everything behaves as expected when I send a single filename as shown below:

dd.post_predict(
        "text_detect_v2",
        data=[f],
        parameters_input={},
        parameters_mllib={},
        parameters_output={"confidence_threshold": 0.2, "bbox": True},
    )

But if I send an array of filenames, I get the following error:
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: http://localhost:1912/api/deepdetect/predict

Given that it's a 502 error, I suspect that there is an issue in nginx.config such that the proxy is not set up to recognize the shape of the response body for an array of images.

I'm running the cpu version.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.