`triton ssh INST COMMAND` missing stdout with node >=0.12 and ControlMaster about node-triton HOT 20 OPEN

I have a similar problem on node.js 5.4.0 and triton ssh. I can see that the ssh command has executed successfully, but it doesn't appear to be connected or piped through to the invoking shell.

from node-triton.

Repro of the bug (at least for me):

$ node --version
v0.10.42
$ triton ssh vm1 hostname
20633805-9e11-4996-8b19-90424946db2d

But with node 4:

$ node --version
v4.3.0
$ ./bin/triton ssh vm1 hostname

No output

from node-triton.

With some help from @misterdjules: He suspects that node >=4.x changes things so that when you use spawn with {stdio: 'inherit'} that the node process won't wait around for the std handles to flush before exiting. Haven't verified this though.

To work around this we above 'inherit' and manually pass through stdout/stderr content.

from node-triton.

this is why i wish node had built in process.exec to just become a new process, and not have to fork and manage a child process

from node-triton.

ditto on the real exec option.

FWIW, we can repro the error when ssh'ing to an IP for which I have ssh ControlMaster enabled. If I disable that, then no issue (or at least get luckier on a possible race).

from node-triton.

@trentm we could check out https://github.com/jprichardson/node-kexec. I've never used it, but based on the docs it seems legit.

If ssh itself hangs because of a user configuration or option that's out of our jurisdiction.

from node-triton.

binary module, not going there

from node-triton.

ssh isn't hanging here, it is node stdio handle flushing

from node-triton.

@trentm that's my only reservation... based on the nature of this desire though that's the only way it'll be possible - native extension.

I made an issue with node core a couple years ago saying there should be process.exec... idk whatever happened with it

from node-triton.

ah my mistake then

from node-triton.

Re-opening because the commit above is a workaround. This isn't fully understood.

from node-triton.

@misterdjules's suspicion now is that it is node's console.log lazy mucking with standard handles interacting poorly with ssh's ControlMaster doing something with fd's

from node-triton.

FWIW, I do have ControlMaster auto in my ~/.ssh/config.

from node-triton.

see PUBAPI-1275 (apparently the change above can cause problems?)

from node-triton.

Problem is there is no TTY when we aren't using 'inherit' with node's spawn. So that breaks interactive ssh session. tl;dr: we need to back out the change above.

Then back to old situation: you don't get (all) the std output from triton ssh INST COMMAND when (a) using node >=4 and (b) using ControlMaster with ssh.

from node-triton.

I know it would be a hack to do so, but what about adding -o ControlMaster=none to the command line flags?

from node-triton.

Brian, that would potentially force re-auth, which could fail. But yes that is the only option right now

A quick kexec attempt didn't work for me either. Would need to look into exec options there

--Trent

On Mar 10, 2016, at 6:43 PM, Brian Bennett [email protected] wrote:

I know it would be a hack to do so, but what about adding -o ControlMaster=none to the command line flags?

—
Reply to this email directly or view it on GitHub.

from node-triton.

Notes from discussion with @misterdjules about what might be going on:

jgilli [4:53 PM]
so basically the problem seems to be that when /dev/tty is reopened by node when
process.stdout is initialized (when the console global getter is called), the
close-on-exec flag is set on that new fd that replaces fd 1, which means that
any forked process will have its stdout closed

[4:54]
and I tried a fix that seemed straightforward but it didn't fix the problem

jgilli [4:57 PM]
trentm: the part where it gets tricky is that on OSX it seems that async I/O
with kqueue doesn't work on /dev/tty, so node does something a bit complex where
it creates a thread to call select on that new fd. And that makes it a bit
trick to debug/trace what's going on 😞 (edited)

[5:02]
trentm: this is basically the code that reopens /dev/tty:
https://github.com/nodejs/node/blob/master/deps/uv/src/unix/tty.c#L64-L87

[5:02]
trentm: and this is the OSX workaround for /dev/tty not being pollable via
kqueue:
https://github.com/nodejs/node/blob/master/deps/uv/src/unix/tty.c#L96-L104

[5:03]
trentm: which is implemented like this:
https://github.com/nodejs/node/blob/master/deps/uv/src/unix/stream.c#L134-L386

[5:04]
trentm: not calling dup2 and not setting close on exec on the fd 1 here:
https://github.com/nodejs/node/blob/master/deps/uv/src/unix/tty.c#L76 "fixes"
the problem

[5:05]
trentm: but it's gross, at least that's what I thought at the time

jgilli [5:06 PM]
trentm: because you end up with two different fds for writing to the tty, and
process.stdout is not fd 1 but the next available fd when you first use
console.log or process.stdout

[5:08]
trentm: my original attempt at fixing it was to use the standard dup2 call,
which does not set close on exec, and not using uv__open_cloexec to open
/dev/tty but the standard open syscall, but that didn't fix the problem, and I
didn't spend more time than that

Note that it still isn't clear why disabling ControlMaster makes the difference here.

from node-triton.

Note that it still isn't clear why disabling ControlMaster makes the difference here.

Right, because it seems that if the issue was about the standard output/error fds' having their close-on-exec flag et to true, then it would be an issue even when not using ControlMaster. One of the differences between ssh running with and without ControlMaster is that when ControlMaster is used, the standard output/error fds are passed from the ssh client process to the control master process through a UNIX socket. I don't know why that would make a difference so.

In other words, I need to spend more time investigating that issue, probably on SmartOS so that I can DTrace it without getting the noise from the separate thread used on OSX to poll /dev/tty.

from node-triton.

triton 4.8.0 is published to npm with the workaround

from node-triton.