What is standard dask-labextension behavior and usage about dask-labextension HOT 14 CLOSED

They would probably work today. You can switch the default away from LocalCluster by changing the values in the labextension.yaml file.

@mrocklin, I can confirm that this works like a charm with dask_jobqueue

from dask-labextension.

The most recent version of the python package (dask_labextension) bundles its own version of jupyter-server-proxy (at least for the time being). So it should be able to work without it. We are still having troubles with tornado 6 however, so I would keep things on tornado 5.1.1 if at all possible.

from dask-labextension.

As @ian-r-rose mentioned in #36 (comment):

this functionality is intended to work with clusters managed by the extension (rather than ones launched in your notebook). It currently works with LocalCluster, and (possibly) KubeCluster

I am wondering if there's a plan for supporting same functionality for Cluster objects such as SLURMCluster , PBSCluster created via dask_jobqueue? (Is this even feasible?)

from dask-labextension.

Start a cluster inside my notebook

This is being handled in #36

Indicate scheduler URL in dask-labextension left panel
Open views on my lab environment

You can handle this by including a workspace json file in the user's environment.
This is what we do for examples.dask.org and also Pangeo I think. https://github.com/dask/dask-examples/blob/master/binder/jupyterlab-workspace.json

I am wondering if there's a plan for supporting same functionality for Cluster objects such as SLURMCluster , PBSCluster created via dask_jobqueue? (Is this even feasible?)

They would probably work today. You can switch the default away from LocalCluster by changing the values in the labextension.yaml file.

However in order to make JupyterLab happy ideally we would be able to start and close a cluster asynchronously with

cluster = await SGECluster(...)

and

await cluster.close()

I've been working on this within KubeCluster (currently available in the dev branch). I suspect that it will be easier to accomplish within dask-jobqueue.

All of this could use better documentation.

from dask-labextension.

That's great @andersy005, thanks for checking! Would you be willing to document an example config file that works with dask_jobqueue? I'd like to start posting some of those in the README.md

from dask-labextension.

Would you be willing to document an example config file that works with dask_jobqueue?

@ian-r-rose, definitely! I will submit a PR for this.

from dask-labextension.

I'd also love to see this working in the NCAR JupyterHub :)

It would be cool to see users manage large clusters with the sidebar .

from dask-labextension.

Thanks for all the answers here. As per my original problem, there are some interrogations still open:

• Indicate scheduler URL in dask-labextension left panel: can the extension detect a new Cluster, without automatically starting it via #36?
• How must be configured the New button? Here is whta the readme says for reminder:

Options for how to launch these clusters are set via the dask configuration system.

from dask-labextension.

@andersy005 ping on this, do you happen to have an example configuration that works for dask_jobqueue that we can point to in the README.md?

from dask-labextension.

@guillaumeeb Sorry for the slow cycle on this. To your specific questions

• No, there is currently no way to detect a cluster that has not been launched by the cluster manager itself. Though it doesn't need to be connected to via #36, you can explicitly write out the client connection code yourself, or inject it via the side panel.
• The configuration lies in the labextension.yml file, or in a dask config file. That describes a module, a cluster class, and args+kwargs for creating the cluster. That is what you will want to define for your setup, which will then be used to create new clusters. Unfortunately, I don't have access to the breadth of clusters, nor the personal bandwidth to really run down all the different types of clusters that people are likely to want to use. So it would be really helpful if the community could try this out in a number of contexts, let me know what issues they find, and contribute back example configurations.

from dask-labextension.

@ian-r-rose,

I am sorry for not getting back to you sooner. My labextension.yml contains:

labextension:
   factory:
     module: 'dask_jobqueue'
     class: 'SLURMCluster'
     args: []
     kwargs: {}
   default:
    workers: null
    adapt: null
      # minimum: 0
      # maximum: 10
   initial: []

Since at NCAR we have two HPC systems that use different schedulers, in order to use the PBSCluster, I need to edit my labextension.yml

labextension:
   factory:
     module: 'dask_jobqueue'
     class: 'PBSCluster'
     args: []
     kwargs: {}
   default:
    workers: null
    adapt: null
      # minimum: 0
      # maximum: 10
   initial: []

P.S.:

The last time I played with this, I made a screencast just in case people are interested to see this feature in action: https://youtu.be/Z8rcPaKHQk0

from dask-labextension.

Thanks @andersy005! Would you mind if I include these and a link to your video in the README?

from dask-labextension.

Would you mind if I include these and a link to your video in the README?

Sure thing! I was about to take some screenshots that can be added to the README but for some reason things are broken despite my attempt at using the fix proposed in jupyterhub/jupyter-server-proxy#109

Traceback (most recent call last):
  File "/glade/work/abanihi/softwares/miniconda3/lib/python3.6/site-packages/tornado/web.py", line 1592, in _execute
    result = yield result
  File "/glade/work/abanihi/softwares/miniconda3/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/glade/work/abanihi/softwares/miniconda3/lib/python3.6/site-packages/jupyter_server_proxy/websocket.py", line 87, in get
    return await self.http_get(*args, **kwargs)
  File "/glade/work/abanihi/softwares/miniconda3/lib/python3.6/site-packages/jupyter_server_proxy/handlers.py", line 247, in http_get
    return await self.proxy(port, proxy_path)
TypeError: object NoneType can't be used in 'await' expression

from dask-labextension.

I've added some more documentation and examples to the README.md. Please let me know if it is still unclear, and we can reopen this and continue working on this.

Thanks for the feedback, all!

from dask-labextension.