aaronkollasch / jupyter-o2 Goto Github PK

Hi Aaron,
Fyi, sometimes starting jupyter-o2 results in the below error. Retrying right after resolves the issue. I remember other users have encountered this previously too.

vpn-10-1-5-125:~ davidd$ jupyter-o2
Jupyter subcommand not provided. Using default: notebook
Connect to O2 server for jupyter notebook
Enter your passphrase:
Connecting to [email protected]
Hostname: login05.o2.rc.hms.harvard.edu

Starting an interactive session.
srun: job 40261194 queued and waiting for resources
srun: job 40261194 has been allocated resources
Node: compute-a-17-95

Starting Jupyter notebook.
jupyter notebook --port=8887 --no-browser
Failed to launch jupyter. (timed out, 60)

Starting a second connection to the login node.
Connecting to the interactive node.

Jupyter is ready! Access at:
False
Opening in browser...
Traceback (most recent call last):
File "/Users/davidd/anaconda/bin/jupyter-o2", line 11, in
sys.exit(main())
File "/Users/davidd/anaconda/lib/python2.7/site-packages/jupyter_o2/jupyter_o2.py", line 649, in main
jupyter_o2_runner.run()
File "/Users/davidd/anaconda/lib/python2.7/site-packages/jupyter_o2/jupyter_o2.py", line 348, in run
if self.connect() or self.keep_alive:
File "/Users/davidd/anaconda/lib/python2.7/site-packages/jupyter_o2/jupyter_o2.py", line 420, in connect
if not self.open_in_browser(jp_site):
File "/Users/davidd/anaconda/lib/python2.7/site-packages/jupyter_o2/jupyter_o2.py", line 526, in open_in_browser
webbrowser.open(site, new=2)
File "/Users/davidd/anaconda/lib/python2.7/webbrowser.py", line 61, in open
if browser.open(url, new, autoraise):
File "/Users/davidd/anaconda/lib/python2.7/webbrowser.py", line 623, in open
script = 'open location "%s"' % url.replace('"', '%22') # opens in default browser
AttributeError: 'bool' object has no attribute 'replace'

Hi Aaron! Looks like this endeavor is still alive and well! We've pointed a number of users here already, and we haven't gotten any complaints so far, so I'd like to thank you again for setting this up.

Anyway, we actually encountered a ticket recently where a user was having trouble using this repo to make their connections, and it occurred to me that they were attempting to configure their stuff on the cluster directly, as opposed to locally. I'm not sure if this is a wider problem that's solving itself as users debug without our intervention, but perhaps it may be good to specify that you should be installing jupyter-o2 locally, and running the command to create the session locally? There's some weirdness that pops up when you try to run this from a login node, for example, but it's probably not worth investigating because that's an incorrect use case for this anyway. Anyway, just a thought. Thanks again!

-Alex

O2 will soon require two-factor authentication, which will break automated SSH logins with pexpect.pxssh.

However, I have plans to allow both interactive and automated communication with the 2FA prompt during SSH logins.

There seems to have been an update to the wiki page for Jupyter on O2 dated Jul 20, 2020.

Based on what I remember of the old instructions, the changes include:

1. --tunnel flag in the srun command to complete the tunnel (for x11 forwarding, presumably)
2. a second terminal ssh'ing into the compute node from (1) is no longer used

Hi,
so far its been working great, but lately i have issue with jupyter login, it shows; error :
SSHError: pxssh error: could not synchronize with original prompt

Any clue, what must be going on!

Running jupyter-o2 with --2fa produces the following error:

Connecting to <user>@o2.hms.harvard.edu
RUN: ssh <user>@o2.hms.harvard.edu
Traceback (most recent call last):
  File "anaconda/lib/python3.7/site-packages/pexpect/expect.py", line 111, in expect_loop
    incoming = spawn.read_nonblocking(spawn.maxread, timeout)
  File "anaconda/lib/python3.7/site-packages/pexpect/pty_spawn.py", line 485, in read_nonblocking
    return super(spawn, self).read_nonblocking(size)
  File "anaconda/lib/python3.7/site-packages/pexpect/spawnbase.py", line 176, in read_nonblocking
    raise EOF('End Of File (EOF). Empty string style platform.')
pexpect.exceptions.EOF: End Of File (EOF). Empty string style platform.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "anaconda/bin/jupyter-o2", line 10, in <module>
    sys.exit(main())
  File "anaconda/lib/python3.7/site-packages/jupyter_o2/jupyter_o2.py", line 649, in main
    jupyter_o2_runner.run()
  File "anaconda/lib/python3.7/site-packages/jupyter_o2/jupyter_o2.py", line 348, in run
    if self.connect() or self.keep_alive:
  File "anaconda/lib/python3.7/site-packages/jupyter_o2/jupyter_o2.py", line 374, in connect
    if not self._login_ssh.login_2fa(self.host, self.user, self.__pass, self.codes_2fa):
  File "anaconda/lib/python3.7/site-packages/jupyter_o2/jupyter_o2.py", line 84, in login_2fa
    self.set_unique_prompt()
  File "anaconda/lib/python3.7/site-packages/pexpect/pxssh.py", line 491, in set_unique_prompt
    i = self.expect ([TIMEOUT, self.PROMPT], timeout=10)
  File "anaconda/lib/python3.7/site-packages/pexpect/spawnbase.py", line 341, in expect
    timeout, searchwindowsize, async_)
  File "anaconda/lib/python3.7/site-packages/pexpect/spawnbase.py", line 369, in expect_list
    return exp.expect_loop(timeout)
  File "anaconda/lib/python3.7/site-packages/pexpect/expect.py", line 117, in expect_loop
    return self.eof(e)
  File "anaconda/lib/python3.7/site-packages/pexpect/expect.py", line 63, in eof
    raise EOF(msg)
pexpect.exceptions.EOF: End Of File (EOF). Empty string style platform.
<jupyter_o2.jupyter_o2.CustomSSH object at 0x10749acf8>
command: /usr/bin/ssh
args: ['/usr/bin/ssh', '-o', 'ForwardX11=yes', '-o', 'LocalForward=8887 127.0.0.1:8887', '-o', 'PubkeyAuthentication=no', '-q', '-oRSAAuthentication=no', '-o', 'PubkeyAuthentication=no', '-l', '<user>', 'o2.hms.harvard.edu']
buffer (last 100 chars): b''
before (last 100 chars): b'S passcodes to XXX-XXX-XXXX\r\n\r\nPasscode or option (1-3): \r\nIncorrect passcode. Please try again.\r\n\r\n'
after: <class 'pexpect.exceptions.EOF'>
match: None
match_index: None
exitstatus: None
flag_eof: True
pid: 66259
child_fd: 8
closed: False
timeout: 60
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_re:
    0: TIMEOUT
    1: re.compile(b'\\[PEXPECT\\][\\$\\#] ')
Cleaning up
Closing login_ssh

In verbose mode:

Connecting to [email protected]
RUN: ssh [email protected]
Connected.
Hostname output: b' \r\n'
JupyterO2Error: Could not get hostname. A communication error may have occured.
Please try again.
Cleaning up
Closing login_ssh

Workaround: Try using the option -Y for trusted X11 forwarding. If you have updated XQuartz, restart your computer.

Background:
This seems to stem from the SSH process hanging on X11 after updating XQuartz.

Test in a separate terminal if you can connect with a normal SSH connection with the options -o ForwardX11=yes -o ForwardX11Trusted=no.
Using ssh with options -o ForwardX11=yes -o ForwardX11Trusted=no -vvv causes a hang at debug2: client_x11_get_proto: xauth command: …, while -o ForwardX11=yes -o ForwardX11Trusted=yes -vvv does not hang. The -Y option tells Jupyter-O2 to use trusted X11 forwarding.

Sometimes when signing in, I see these messages (in verbose mode), and jupyter-o2 immediately exits.

Responding to Duo 2FA prompt
Found prompt #0
SSH connection did not reach command prompt.

This happens a couple of times in a row, but the same command usually works on the third try. Will need to investigate why this is happening, but in the mean time, a few retries works for me.