Comments (18)
This does not appear to be behind my 504s, but might we consider changing the poll from every two seconds (for the dashboard URL) or five seconds (for cluster list) to something that, I dunno, maybe backs off exponentially up to like a minute each? The logs are very, very noisy at the default polling frequencies.
from dask-labextension.
I have an implementation against 3.0 that I think helps. I'll forward-port it to master soon. In the meantime: effectively, rather than setInterval, it uses setTimeout and just makes sure to call setTimeout exactly once before returning.
https://github.com/lsst-sqre/dask-labextension/tree/tickets/DM-18415c
is the feature against 3.0.
from dask-labextension.
Sorry to be so delinquent in responding here @athornton, you are right that we should be able to back this off. I had wanted the dashboards to feel responsive, but there should be a better way.
I had been planning to wait until jupyterlab/jupyterlab#6141 to land, which adds some functionality for smarter polling to the core of JupyterLab. This should be published in a new prerelease in a day or two, and I plan to take a crack at this shortly thereafter.
In particular, I think we may want be handle a 302 as a special case (as you suggest above). Can you think of other situations where that might come up? Might we want to just back off polling when any 302 is detected, rather than trying to parse the content?
from dask-labextension.
(Your solution looks like a good one, by the way, I had just hoped to offload the logic onto the functionality referenced above)
from dask-labextension.
I think that would be OK. I'm relying (in my environment) on the same trick we used to get Bokeh proxying in JupyterLab, but I think the TCP forwarder is literally copying the bytes from its target rather than doing any cute internal redirection tricks, so as far as I remember we don't internally see a redirect.
https://github.com/lsst-sqre/jupyterlabutils/blob/master/jupyterlabutils/notebook/clusterproxy.py
There's not a huge hurry on this; this week is the LSST Science Platform Review so I'm being unusually cautious in putting new functionality on any of our instances. Usually I feel like I can break our integration testing environment and that our GKE environment is entirely mine to abuse, but this week we want to make very sure we have fallbacks in case something breaks on our stable environment (because so much of our function is tied to our data and to our auth flow, testing locally is not very useful for me).
from dask-labextension.
....and I see the upstream PR was merged. I agree that using the JupyterLab Poll class configured with backoff is a much better idea than a one-off patch like this.
from dask-labextension.
Question regarding implementation of the redirect handling, @athornton: when a 302 gets sent, would it be safe to stop the polling all-together? That is, will the user always need to login and refresh the page? Or is there some way to re-authenticate while remaining in the same JS context?
Basically, should we stop polling when the login redirect happens, or should we try only exponential backoff and see if that is good enough?
from dask-labextension.
I've just published a prerelease of [email protected]
targeting JupyterLab 1.0a3. This uses the new polling functionality with exponential backoff (backing off to once per minute if the requests fail). @athornton can you check to see if this is good enough for your use-case? If not, we can go back and experiment with specific handling of 302s.
from dask-labextension.
I can check, but possibly not till next week...I'm on the road until then. I don't think it's safe to stop polling entirely. Certainly if a cluster is deleted and a new one created, the new one's worker monitor endpoints will change. The dashboard is probably constant?
from dask-labextension.
Okay, no rush. Safe travels!
from dask-labextension.
I am running the new version, but the behavior doesn't seem to be any different for the case where I have a running lab but not a dask dashboard (to be fair, this is the case where I am getting a 500 from the cluster dashboard rather than a 302).
[athornton@nb-athornton-exp-d-2019-04-24 ~]$ jupyter labextension list
JupyterLab v1.0.0a3
Known labextensions:
app dir: /usr/local/share/jupyter/lab
@jupyter-widgets/jupyterlab-manager v0.40.0 enabled OK
@jupyterlab/celltags v0.1.5 enabled OK
@jupyterlab/hub-extension v1.0.0-alpha.6 enabled OK*
@jupyterlab/toc v1.0.0-pre.1 enabled OK
@lsst-sqre/jupyterlab-lsstquery v1.0.0-alpha.6 enabled OK*
@lsst-sqre/jupyterlab-savequit v1.0.0-alpha.6 enabled OK*
@pyviz/jupyterlab_pyviz v1.0.0-alpha.6 enabled OK*
bqplot v0.4.5 enabled OK
dask-labextension v0.4.0-pre enabled OK
ipyaladin v0.1.5 enabled OK*
ipyevents v1.4.1 enabled OK
ipyvolume v0.5.1 enabled OK
jupyter-matplotlib v0.3.0 enabled OK
jupyter-threejs v2.0.3 enabled OK
jupyter_firefly_extensions v0.2.3 enabled OK
jupyterlab-datawidgets v6.1.0 enabled OK
jupyterlab-server-proxy v1.0.1-alpha.6 enabled OK*
jupyterlab_bokeh v1.0.0-alpha.6 enabled OK*
nbdime-jupyterlab v0.6.1 enabled OK*
local extensions:
@jupyterlab/hub-extension: /usr/share/git/jupyterlab-hub
@lsst-sqre/jupyterlab-lsstquery: /usr/share/git/jupyterlab-lsstquery
@lsst-sqre/jupyterlab-savequit: /usr/share/git/jupyterlab-savequit
@pyviz/jupyterlab_pyviz: /usr/share/git/pyviz_comms
ipyaladin: /usr/share/git/ipyaladin/js
jupyterlab-server-proxy: /usr/share/git/jupyter-server-proxy/jupyterlab-server-proxy
jupyterlab_bokeh: /usr/share/git/jupyterlab_bokeh
nbdime-jupyterlab: /usr/share/git/nbdime/packages/labextension
Lab logs look like:
[E 2019-04-24 17:19:38.250 SingleUserLabApp log:166] {
"Cookie": "jupyterhub-user-athornton=[secret]; usrkey=[secret]; _xsrf=[secret]; jupyterhub-session-id=[secret]",
"Dnt": "1",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Referer": "https://nublado.lsst.codes/nb/user/athornton/lab?",
"Accept": "*/*",
"Content-Type": "application/json",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36",
"X-Xsrftoken": "2|c12759b1|20a52c8b5f8485fe3e19a8e8289a4d4c|1553710802",
"Authorization": "token [secret]",
"X-Forwarded-Path": "/nb/",
"X-Scheme": "https",
"X-Original-Uri": "/nb/user/athornton/proxy/8787/individual-plots.json?1556126378205",
"X-Forwarded-Proto": "https, https,http",
"X-Forwarded-Port": "443, 443,80",
"X-Forwarded-Host": "nublado.lsst.codes",
"X-Forwarded-For": "140.252.33.60,10.24.24.215",
"X-Real-Ip": "140.252.33.60",
"X-Request-Id": "25adb4c4216525e56d40dc916486b0a9",
"Connection": "close",
"Host": "nublado.lsst.codes"
}
[E 2019-04-24 17:19:38.251 SingleUserLabApp log:174] 500 GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126378205 ([email protected]) 6.17ms
[D 2019-04-24 17:19:42.316 SingleUserLabApp auth:871] Allowing whitelisted Hub user athornton
[D 2019-04-24 17:19:42.316 SingleUserLabApp auth:744] Setting oauth cookie for 140.252.33.60: jupyterhub-user-athornton, {'path': '/nb/user/athornton/', 'httponly': True, 'secure': True}
[E 2019-04-24 17:19:42.321 SingleUserLabApp log:166] {
"Cookie": "jupyterhub-user-athornton=[secret]; usrkey=[secret]; _xsrf=[secret]; jupyterhub-session-id=[secret]",
"Dnt": "1",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Referer": "https://nublado.lsst.codes/nb/user/athornton/lab?",
"Accept": "*/*",
"Content-Type": "application/json",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36",
"X-Xsrftoken": "2|c12759b1|20a52c8b5f8485fe3e19a8e8289a4d4c|1553710802",
"Authorization": "token [secret]",
"X-Forwarded-Path": "/nb/",
"X-Scheme": "https",
"X-Original-Uri": "/nb/user/athornton/proxy/8787/individual-plots.json?1556126382277",
"X-Forwarded-Proto": "https, https,http",
"X-Forwarded-Port": "443, 443,80",
"X-Forwarded-Host": "nublado.lsst.codes",
"X-Forwarded-For": "140.252.33.60,10.24.24.215",
"X-Real-Ip": "140.252.33.60",
"X-Request-Id": "40f88cc168d1776e9a1c0361904eb21d",
"Connection": "close",
"Host": "nublado.lsst.codes"
}
[E 2019-04-24 17:19:42.321 SingleUserLabApp log:174] 500 GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126382277 ([email protected]) 5.48ms
And my ingress-controller logs look like....(and aren't slowing down)
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:15:05 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126105525 HTTP/2.0" 500 30 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 70 0.009 [sticky-nublado-proxy-8000] 10.24.24.191:8000 30 0.008 500 a5d2d970316fc177a431ad72387db2a5
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:15:09 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126109597 HTTP/2.0" 500 30 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 217 0.008 [sticky-nublado-proxy-8000] 10.24.24.191:8000 30 0.008 500 33b04afb4e5ddc44273123a6cf8233d3
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:15:10 +0000] "GET /nb/user/athornton/api/contents/notebooks/dask-experiments?content=1&1556126110075 HTTP/2.0" 200 349 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 232 0.009 [sticky-nublado-proxy-8000] 10.24.24.191:8000 1366 0.009 200 e47cad355a699cb5d711f95a8d2c2e1a
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:15:12 +0000] "GET /nb/user/athornton/dask/clusters?1556126112239 HTTP/2.0" 404 2381 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 201 0.007 [sticky-nublado-proxy-8000] 10.24.24.191:8000 7451 0.006 404 879c5a489b9e7abd5231731356d5c00d
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:15:13 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126113669 HTTP/2.0" 500 30 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 216 0.009 [sticky-nublado-proxy-8000] 10.24.24.191:8000 30 0.009 500 47b39580e85bdbb478
from dask-labextension.
Now testing what happens when the underlying Lab has died but the user window is still open....
Doesn't look like it's doing any better, alas: still a 3 or 4 second poll. Although I don't understand why some are going to /nb/user
and some to /nb/hub/user
for the clusters endpoint. But the clusters endpoint gets quiet pretty quickly, and it's the proxy poll (which is the dask dashboard endpoint) that doesn't seem to back off.
Is any of that useful diagnostic info?
adam@ixitxachitl:/tmp$ grep -v metrics dead-lab.txt | grep -v kernels | grep -v terminals | grep -v sessions | grep -v notebooks
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:21:32 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126492190 HTTP/2.0" 503 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 218 0.076 [sticky-nublado-proxy-8000] 10.24.24.191:8000 5 0.075 503 ebb46e155ce78088909e260261761138
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:21:57 +0000] "GET /nb/user/athornton/dask/clusters?1556126514803 HTTP/2.0" 302 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 56 2.376 [sticky-nublado-proxy-8000] 10.24.24.191:8000 0 2.378 302 b8fe73d673ebbb3337edf847c51e51da
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:00 +0000] "GET /nb/hub/user/athornton/dask/clusters?1556126514803 HTTP/2.0" 503 4666 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 58 3.271 [sticky-nublado-proxy-8000] 10.24.24.191:8000 4666 3.270 503 073753e43c4d9f287f3a19aa3eed978e
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:14 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126496325 HTTP/2.0" 503 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 75 37.743 [sticky-nublado-proxy-8000] 10.24.24.191:8000 5 37.743 503 d9d7185919936be8bf6fcdb0f819feb5
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:18 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126538127 HTTP/2.0" 302 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 70 0.008 [sticky-nublado-proxy-8000] 10.24.24.191:8000 0 0.008 302 1e7aa7197118510fc0a3669d698bbb34
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:21 +0000] "GET /nb/hub/user/athornton/proxy/8787/individual-plots.json?1556126538127 HTTP/2.0" 503 4687 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 72 3.269 [sticky-nublado-proxy-8000] 10.24.24.191:8000 4687 3.269 503 b2b5095e4529f5e08a8608435ca93229
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:25 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126545520 HTTP/2.0" 302 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 70 0.009 [sticky-nublado-proxy-8000] 10.24.24.191:8000 0 0.008 302 6a117cd76410473836d9484d41a99281
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:28 +0000] "GET /nb/hub/user/athornton/proxy/8787/individual-plots.json?1556126545520 HTTP/2.0" 503 4687 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 72 3.258 [sticky-nublado-proxy-8000] 10.24.24.191:8000 4687 3.257 503 cc7012395fe594e6ab0bd3a003018173
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:33 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126552903 HTTP/2.0" 302 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 70 0.973 [sticky-nublado-proxy-8000] 10.24.24.191:8000 0 0.973 302 84dccf70595f9bfafc46be001c759ee4
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:37 +0000] "GET /nb/hub/user/athornton/proxy/8787/individual-plots.json?1556126552903 HTTP/2.0" 503 4687 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 72 3.298 [sticky-nublado-proxy-8000] 10.24.24.191:8000 4687 3.297 503 785d9fd7de78091cc0c0558c617b8ee5
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:42 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126561288 HTTP/2.0" 302 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 70 0.745 [sticky-nublado-proxy-8000] 10.24.24.191:8000 0 0.746 302 a57fe475e343455cdf3ec73b75c46bea
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:45 +0000] "GET /nb/hub/user/athornton/proxy/8787/individual-plots.json?1556126561288 HTTP/2.0" 503 4687 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 72 3.361 [sticky-nublado-proxy-8000] 10.24.24.191:8000 4687 3.361 503 2c6a0bc8814064abb6c06bf7e0fd8052
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:50 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126569921 HTTP/2.0" 302 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 70 0.963 [sticky-nublado-proxy-8000] 10.24.24.191:8000 0 0.963 302 15afb38b36927142c33e214c8a522f45
from dask-labextension.
Hmm, odd. When I look it seems like it is backing off correctly. Can you make sure you are using the latest 0.4.0-pre.3
? It looks like you are a minor bump back.
from dask-labextension.
I'll take a look; right now we're fighting getting a FITS viewer to work but I should have time to try tomorrow.
from dask-labextension.
The cluster endpoint is definitely backing off. Individual-plots is still chatty. But, yeah, I think we can close this particular issue (until I diagnose individual-plots) as the backoff is doing its job.
from dask-labextension.
If individual plots is still not backing off, it seems there is still some work to do here, so feel free to leave open. I'm pretty puzzled, since my local install seems to be working as expected
from dask-labextension.
I'm going to bet it is my fault somehow. Individual-plots is connecting (for me) to an nbproxy address (that is, a local path .../proxy/8787/.... that doesn't actually exist if I haven't created a cluster yet, and I suspect that's what's muddying the waters.
from dask-labextension.
Just wanted to report that I have a 404 try to connect to individual-plots.json
every ~3s after logging in to JupyterLab with the dask-labextension enabled (but before starting a cluster)
from dask-labextension.
Related Issues (20)
- Dashboard doesn't show when scheduler does not listen on localhost HOT 1
- Update branding
- Open Dashboard programmatically HOT 3
- Specify default address to look for scheduler HOT 13
- Dashboard shows cluster of different user HOT 1
- No visibility (black background) with JupyterLab in Dark theme HOT 3
- Can't connect to local cluster using local IP and port HOT 1
- 5.3.0 does not create a schemas/dask-labextension/plugin.json file on install HOT 11
- Default layout HOT 4
- Need to press the new "Launch in JupyterLab" button twice HOT 1
- Help connecting to existing KubeCluster using the build-in Discovery Mechanism HOT 2
- Using +NEW freezes Jupyterlab completely HOT 3
- JLab 4.0 side-effect: Class 'RenderedCommon' incorrectly implements interface 'IRenderer'. HOT 12
- Logos no longer working well HOT 4
- Conflicting dependencies with jupyterlab=4.0.2 HOT 3
- Inject client code adds an empty cell. HOT 3
- Release HOT 3
- Button Launch Dashboard in Jupyterlab not available HOT 1
- JLab 4-compatible release HOT 5
- Dashboard does not display graphs with bokeh = 3.3.0 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dask-labextension.