Unable to view the status dashboard about dask-jobqueue HOT 12 CLOSED

The way I have got a similar setup to work in the past is to do ssh tunneling:

ssh -fN your-login@scheduler-machine -L 33023:localhost:33023

Then I can just open http://localhost:33023 on my local machine and see the status page. Slightly cumbersome but I am afraid I don't know of a more convenient way of doing it. This approach (which is not dask-specific at all) is mentioned in the distributed doc.

from dask-jobqueue.

@jhamman happy to do so. Give me a few hours (busy till later) and I'll be happy to PR in.

from dask-jobqueue.

from dask-jobqueue.

Originally, I did the following:

cluster = SGECluster(queue='default.q', walltime="1500000", processes=10, memory='20GB')

This gave the client address:

http://172.16.23.102:8787/status

After setting the interface to eth0:

cluster = SGECluster(queue='default.q', walltime="1500000", processes=10, memory='20GB', interface='eth0')

the client address is:

http://172.16.23.102:43906/status

This was not accessible (also timed-out).

I also tried setting the cluster interface to ib0. The client address then became:

http://10.145.71.204:8787/status

This also timed-out.

Setting the eth1 or ib1 also gave errors (likely because these interfaces are not 'enabled' somehow).

from dask-jobqueue.

Is there something that's blocking access that I'm missing?

Maybe. It might be worth checking with your IT staff.

You might consider seeing if machines on the SGE cluster can see that address, perhaps by using requests to download that page, both from the client machine, and possibly from one of the workers:

import requests
requests.get(addr)  # does this work?

client.run(requests.get, addr)  # does this work?

from dask-jobqueue.

@mrocklin thanks for the help! I tried the following:

import requests
addr = "http://cluster.server.ip.addr:port/status"  # with appropriate modifications.
requests.get(addr)  # times out

In the absence of any information from IT, my current hypothesis is that there is something blocking the opening of ports from node to node. I'll continue to keep tabs on this issue.

from dask-jobqueue.

@lesteve thank you for the excellent tip! I have added this to my TextExpander snippets 😄

from dask-jobqueue.

@ericmjl - would you be interested in adding a note in the dask-jobqueue documentation identifying this issue? I suspect other users will run into the port-forwarding issue.

from dask-jobqueue.

you may also be interested in this PR: pangeo-data/pangeo#317

from dask-jobqueue.

We also do the same thing as @lesteve has suggested.

At some point we may consider revitalizing the JupyterLab extension for Dask (especially as JupyterLab has become more stable), which would make adding the extra ssh tunnel unnecessary. AIUI JupyterLab developers are eager to have users that can give feedback on this sort of thing. So getting help shouldn't be too hard if someone has cycles/interest to pursue this.

ref: https://github.com/dask/dask-labextension

from dask-jobqueue.

from dask-jobqueue.

Issue ( dask/dask-labextension#15 ) had some discussion about how this might be done. Not sure if that advice is still current.

from dask-jobqueue.