Git Product home page Git Product logo

Comments (18)

athornton avatar athornton commented on August 13, 2024

This does not appear to be behind my 504s, but might we consider changing the poll from every two seconds (for the dashboard URL) or five seconds (for cluster list) to something that, I dunno, maybe backs off exponentially up to like a minute each? The logs are very, very noisy at the default polling frequencies.

from dask-labextension.

athornton avatar athornton commented on August 13, 2024

I have an implementation against 3.0 that I think helps. I'll forward-port it to master soon. In the meantime: effectively, rather than setInterval, it uses setTimeout and just makes sure to call setTimeout exactly once before returning.

https://github.com/lsst-sqre/dask-labextension/tree/tickets/DM-18415c

is the feature against 3.0.

from dask-labextension.

ian-r-rose avatar ian-r-rose commented on August 13, 2024

Sorry to be so delinquent in responding here @athornton, you are right that we should be able to back this off. I had wanted the dashboards to feel responsive, but there should be a better way.

I had been planning to wait until jupyterlab/jupyterlab#6141 to land, which adds some functionality for smarter polling to the core of JupyterLab. This should be published in a new prerelease in a day or two, and I plan to take a crack at this shortly thereafter.

In particular, I think we may want be handle a 302 as a special case (as you suggest above). Can you think of other situations where that might come up? Might we want to just back off polling when any 302 is detected, rather than trying to parse the content?

from dask-labextension.

ian-r-rose avatar ian-r-rose commented on August 13, 2024

(Your solution looks like a good one, by the way, I had just hoped to offload the logic onto the functionality referenced above)

from dask-labextension.

athornton avatar athornton commented on August 13, 2024

I think that would be OK. I'm relying (in my environment) on the same trick we used to get Bokeh proxying in JupyterLab, but I think the TCP forwarder is literally copying the bytes from its target rather than doing any cute internal redirection tricks, so as far as I remember we don't internally see a redirect.

https://github.com/lsst-sqre/jupyterlabutils/blob/master/jupyterlabutils/notebook/clusterproxy.py

There's not a huge hurry on this; this week is the LSST Science Platform Review so I'm being unusually cautious in putting new functionality on any of our instances. Usually I feel like I can break our integration testing environment and that our GKE environment is entirely mine to abuse, but this week we want to make very sure we have fallbacks in case something breaks on our stable environment (because so much of our function is tied to our data and to our auth flow, testing locally is not very useful for me).

from dask-labextension.

athornton avatar athornton commented on August 13, 2024

....and I see the upstream PR was merged. I agree that using the JupyterLab Poll class configured with backoff is a much better idea than a one-off patch like this.

from dask-labextension.

ian-r-rose avatar ian-r-rose commented on August 13, 2024

Question regarding implementation of the redirect handling, @athornton: when a 302 gets sent, would it be safe to stop the polling all-together? That is, will the user always need to login and refresh the page? Or is there some way to re-authenticate while remaining in the same JS context?

Basically, should we stop polling when the login redirect happens, or should we try only exponential backoff and see if that is good enough?

from dask-labextension.

ian-r-rose avatar ian-r-rose commented on August 13, 2024

I've just published a prerelease of [email protected] targeting JupyterLab 1.0a3. This uses the new polling functionality with exponential backoff (backing off to once per minute if the requests fail). @athornton can you check to see if this is good enough for your use-case? If not, we can go back and experiment with specific handling of 302s.

from dask-labextension.

athornton avatar athornton commented on August 13, 2024

I can check, but possibly not till next week...I'm on the road until then. I don't think it's safe to stop polling entirely. Certainly if a cluster is deleted and a new one created, the new one's worker monitor endpoints will change. The dashboard is probably constant?

from dask-labextension.

ian-r-rose avatar ian-r-rose commented on August 13, 2024

Okay, no rush. Safe travels!

from dask-labextension.

athornton avatar athornton commented on August 13, 2024

I am running the new version, but the behavior doesn't seem to be any different for the case where I have a running lab but not a dask dashboard (to be fair, this is the case where I am getting a 500 from the cluster dashboard rather than a 302).

[athornton@nb-athornton-exp-d-2019-04-24 ~]$ jupyter labextension list
JupyterLab v1.0.0a3
Known labextensions:
   app dir: /usr/local/share/jupyter/lab
        @jupyter-widgets/jupyterlab-manager v0.40.0  enabled  OK
        @jupyterlab/celltags v0.1.5  enabled  OK
        @jupyterlab/hub-extension v1.0.0-alpha.6  enabled  OK*
        @jupyterlab/toc v1.0.0-pre.1  enabled  OK
        @lsst-sqre/jupyterlab-lsstquery v1.0.0-alpha.6  enabled  OK*
        @lsst-sqre/jupyterlab-savequit v1.0.0-alpha.6  enabled  OK*
        @pyviz/jupyterlab_pyviz v1.0.0-alpha.6  enabled  OK*
        bqplot v0.4.5  enabled  OK
        dask-labextension v0.4.0-pre  enabled  OK
        ipyaladin v0.1.5  enabled  OK*
        ipyevents v1.4.1  enabled  OK
        ipyvolume v0.5.1  enabled  OK
        jupyter-matplotlib v0.3.0  enabled  OK
        jupyter-threejs v2.0.3  enabled  OK
        jupyter_firefly_extensions v0.2.3  enabled  OK
        jupyterlab-datawidgets v6.1.0  enabled  OK
        jupyterlab-server-proxy v1.0.1-alpha.6  enabled  OK*
        jupyterlab_bokeh v1.0.0-alpha.6  enabled  OK*
        nbdime-jupyterlab v0.6.1  enabled  OK*

   local extensions:
        @jupyterlab/hub-extension: /usr/share/git/jupyterlab-hub
        @lsst-sqre/jupyterlab-lsstquery: /usr/share/git/jupyterlab-lsstquery
        @lsst-sqre/jupyterlab-savequit: /usr/share/git/jupyterlab-savequit
        @pyviz/jupyterlab_pyviz: /usr/share/git/pyviz_comms
        ipyaladin: /usr/share/git/ipyaladin/js
        jupyterlab-server-proxy: /usr/share/git/jupyter-server-proxy/jupyterlab-server-proxy
        jupyterlab_bokeh: /usr/share/git/jupyterlab_bokeh
        nbdime-jupyterlab: /usr/share/git/nbdime/packages/labextension

Lab logs look like:

[E 2019-04-24 17:19:38.250 SingleUserLabApp log:166] {
      "Cookie": "jupyterhub-user-athornton=[secret]; usrkey=[secret]; _xsrf=[secret]; jupyterhub-session-id=[secret]",
      "Dnt": "1",
      "Accept-Language": "en-US,en;q=0.9",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "https://nublado.lsst.codes/nb/user/athornton/lab?",
      "Accept": "*/*",
      "Content-Type": "application/json",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36",
      "X-Xsrftoken": "2|c12759b1|20a52c8b5f8485fe3e19a8e8289a4d4c|1553710802",
      "Authorization": "token [secret]",
      "X-Forwarded-Path": "/nb/",
      "X-Scheme": "https",
      "X-Original-Uri": "/nb/user/athornton/proxy/8787/individual-plots.json?1556126378205",
      "X-Forwarded-Proto": "https, https,http",
      "X-Forwarded-Port": "443, 443,80",
      "X-Forwarded-Host": "nublado.lsst.codes",
      "X-Forwarded-For": "140.252.33.60,10.24.24.215",
      "X-Real-Ip": "140.252.33.60",
      "X-Request-Id": "25adb4c4216525e56d40dc916486b0a9",
      "Connection": "close",
      "Host": "nublado.lsst.codes"
    }
[E 2019-04-24 17:19:38.251 SingleUserLabApp log:174] 500 GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126378205 ([email protected]) 6.17ms
[D 2019-04-24 17:19:42.316 SingleUserLabApp auth:871] Allowing whitelisted Hub user athornton
[D 2019-04-24 17:19:42.316 SingleUserLabApp auth:744] Setting oauth cookie for 140.252.33.60: jupyterhub-user-athornton, {'path': '/nb/user/athornton/', 'httponly': True, 'secure': True}
[E 2019-04-24 17:19:42.321 SingleUserLabApp log:166] {
      "Cookie": "jupyterhub-user-athornton=[secret]; usrkey=[secret]; _xsrf=[secret]; jupyterhub-session-id=[secret]",
      "Dnt": "1",
      "Accept-Language": "en-US,en;q=0.9",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "https://nublado.lsst.codes/nb/user/athornton/lab?",
      "Accept": "*/*",
      "Content-Type": "application/json",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36",
      "X-Xsrftoken": "2|c12759b1|20a52c8b5f8485fe3e19a8e8289a4d4c|1553710802",
      "Authorization": "token [secret]",
      "X-Forwarded-Path": "/nb/",
      "X-Scheme": "https",
      "X-Original-Uri": "/nb/user/athornton/proxy/8787/individual-plots.json?1556126382277",
      "X-Forwarded-Proto": "https, https,http",
      "X-Forwarded-Port": "443, 443,80",
      "X-Forwarded-Host": "nublado.lsst.codes",
      "X-Forwarded-For": "140.252.33.60,10.24.24.215",
      "X-Real-Ip": "140.252.33.60",
      "X-Request-Id": "40f88cc168d1776e9a1c0361904eb21d",
      "Connection": "close",
      "Host": "nublado.lsst.codes"
    }
[E 2019-04-24 17:19:42.321 SingleUserLabApp log:174] 500 GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126382277 ([email protected]) 5.48ms

And my ingress-controller logs look like....(and aren't slowing down)

140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:15:05 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126105525 HTTP/2.0" 500 30 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 70 0.009 [sticky-nublado-proxy-8000] 10.24.24.191:8000 30 0.008 500 a5d2d970316fc177a431ad72387db2a5
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:15:09 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126109597 HTTP/2.0" 500 30 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 217 0.008 [sticky-nublado-proxy-8000] 10.24.24.191:8000 30 0.008 500 33b04afb4e5ddc44273123a6cf8233d3
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:15:10 +0000] "GET /nb/user/athornton/api/contents/notebooks/dask-experiments?content=1&1556126110075 HTTP/2.0" 200 349 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 232 0.009 [sticky-nublado-proxy-8000] 10.24.24.191:8000 1366 0.009 200 e47cad355a699cb5d711f95a8d2c2e1a
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:15:12 +0000] "GET /nb/user/athornton/dask/clusters?1556126112239 HTTP/2.0" 404 2381 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 201 0.007 [sticky-nublado-proxy-8000] 10.24.24.191:8000 7451 0.006 404 879c5a489b9e7abd5231731356d5c00d
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:15:13 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126113669 HTTP/2.0" 500 30 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 216 0.009 [sticky-nublado-proxy-8000] 10.24.24.191:8000 30 0.009 500 47b39580e85bdbb478

from dask-labextension.

athornton avatar athornton commented on August 13, 2024

Now testing what happens when the underlying Lab has died but the user window is still open....

Doesn't look like it's doing any better, alas: still a 3 or 4 second poll. Although I don't understand why some are going to /nb/user and some to /nb/hub/user for the clusters endpoint. But the clusters endpoint gets quiet pretty quickly, and it's the proxy poll (which is the dask dashboard endpoint) that doesn't seem to back off.

Is any of that useful diagnostic info?

adam@ixitxachitl:/tmp$ grep -v metrics dead-lab.txt | grep -v kernels | grep -v terminals | grep -v sessions | grep -v notebooks
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:21:32 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126492190 HTTP/2.0" 503 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 218 0.076 [sticky-nublado-proxy-8000] 10.24.24.191:8000 5 0.075 503 ebb46e155ce78088909e260261761138
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:21:57 +0000] "GET /nb/user/athornton/dask/clusters?1556126514803 HTTP/2.0" 302 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 56 2.376 [sticky-nublado-proxy-8000] 10.24.24.191:8000 0 2.378 302 b8fe73d673ebbb3337edf847c51e51da
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:00 +0000] "GET /nb/hub/user/athornton/dask/clusters?1556126514803 HTTP/2.0" 503 4666 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 58 3.271 [sticky-nublado-proxy-8000] 10.24.24.191:8000 4666 3.270 503 073753e43c4d9f287f3a19aa3eed978e
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:14 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126496325 HTTP/2.0" 503 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 75 37.743 [sticky-nublado-proxy-8000] 10.24.24.191:8000 5 37.743 503 d9d7185919936be8bf6fcdb0f819feb5
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:18 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126538127 HTTP/2.0" 302 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 70 0.008 [sticky-nublado-proxy-8000] 10.24.24.191:8000 0 0.008 302 1e7aa7197118510fc0a3669d698bbb34
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:21 +0000] "GET /nb/hub/user/athornton/proxy/8787/individual-plots.json?1556126538127 HTTP/2.0" 503 4687 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 72 3.269 [sticky-nublado-proxy-8000] 10.24.24.191:8000 4687 3.269 503 b2b5095e4529f5e08a8608435ca93229
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:25 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126545520 HTTP/2.0" 302 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 70 0.009 [sticky-nublado-proxy-8000] 10.24.24.191:8000 0 0.008 302 6a117cd76410473836d9484d41a99281
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:28 +0000] "GET /nb/hub/user/athornton/proxy/8787/individual-plots.json?1556126545520 HTTP/2.0" 503 4687 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 72 3.258 [sticky-nublado-proxy-8000] 10.24.24.191:8000 4687 3.257 503 cc7012395fe594e6ab0bd3a003018173
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:33 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126552903 HTTP/2.0" 302 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 70 0.973 [sticky-nublado-proxy-8000] 10.24.24.191:8000 0 0.973 302 84dccf70595f9bfafc46be001c759ee4
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:37 +0000] "GET /nb/hub/user/athornton/proxy/8787/individual-plots.json?1556126552903 HTTP/2.0" 503 4687 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 72 3.298 [sticky-nublado-proxy-8000] 10.24.24.191:8000 4687 3.297 503 785d9fd7de78091cc0c0558c617b8ee5
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:42 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126561288 HTTP/2.0" 302 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 70 0.745 [sticky-nublado-proxy-8000] 10.24.24.191:8000 0 0.746 302 a57fe475e343455cdf3ec73b75c46bea
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:45 +0000] "GET /nb/hub/user/athornton/proxy/8787/individual-plots.json?1556126561288 HTTP/2.0" 503 4687 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 72 3.361 [sticky-nublado-proxy-8000] 10.24.24.191:8000 4687 3.361 503 2c6a0bc8814064abb6c06bf7e0fd8052
140.252.33.60 - [140.252.33.60] - - [24/Apr/2019:17:22:50 +0000] "GET /nb/user/athornton/proxy/8787/individual-plots.json?1556126569921 HTTP/2.0" 302 0 "https://nublado.lsst.codes/nb/user/athornton/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" 70 0.963 [sticky-nublado-proxy-8000] 10.24.24.191:8000 0 0.963 302 15afb38b36927142c33e214c8a522f45

from dask-labextension.

ian-r-rose avatar ian-r-rose commented on August 13, 2024

Hmm, odd. When I look it seems like it is backing off correctly. Can you make sure you are using the latest 0.4.0-pre.3? It looks like you are a minor bump back.

from dask-labextension.

athornton avatar athornton commented on August 13, 2024

I'll take a look; right now we're fighting getting a FITS viewer to work but I should have time to try tomorrow.

from dask-labextension.

athornton avatar athornton commented on August 13, 2024

The cluster endpoint is definitely backing off. Individual-plots is still chatty. But, yeah, I think we can close this particular issue (until I diagnose individual-plots) as the backoff is doing its job.

from dask-labextension.

ian-r-rose avatar ian-r-rose commented on August 13, 2024

If individual plots is still not backing off, it seems there is still some work to do here, so feel free to leave open. I'm pretty puzzled, since my local install seems to be working as expected

from dask-labextension.

athornton avatar athornton commented on August 13, 2024

I'm going to bet it is my fault somehow. Individual-plots is connecting (for me) to an nbproxy address (that is, a local path .../proxy/8787/.... that doesn't actually exist if I haven't created a cluster yet, and I suspect that's what's muddying the waters.

from dask-labextension.

andreas-h avatar andreas-h commented on August 13, 2024

Just wanted to report that I have a 404 try to connect to individual-plots.json every ~3s after logging in to JupyterLab with the dask-labextension enabled (but before starting a cluster)

from dask-labextension.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.