Bolt silently fails on all commands about bolt HOT 29 CLOSED

One likely possibility is that we don't correctly parse ~ in the private-key setting. Can you try changing that to an absolute path?

from bolt.

Also we may be hiding some net-ssh errors that we should be logging. Worth looking into.

from bolt.

No change passing it an absolute path for the private-key. Interestingly I can get it to run fine by removing bolt.yaml and passing in the username and password on the command line.

from bolt.

You can also pass --private-key on the command line. Does that work?

from bolt.

Private key on the command line does not work, same output. However putting the password into bolt.yaml works fine, so the issue seems to be with the private key. Using it normally with ssh works fine, however

from bolt.

Is it a password-protected private key?

Also, tomorrow I'll work on getting some better debugging options for SSH issues.

from bolt.

No password on the key, no. I had a few alias'es in place for the ssh binary, but removing them has made no difference. I just tested it in a clean /bin/sh shell, and it has the same result.

from bolt.

Not sure what's going on :( if you want to dig in, you should be able to set verbosity at https://github.com/puppetlabs/bolt/blob/master/lib/bolt/transport/ssh/connection.rb#L89-L92 according to https://net-ssh.github.io/ssh/v2/api/classes/Net/SSH.html, like verbose: :debug).

I won't be able to look at it until tomorrow (PDT).

from bolt.

Got some debug output by making the tweaks as suggested, but there doesn't seem to be much of interest in them, at least to me:

<snip>
Disabling use_agent in net-ssh: ssh-agent is not available
establishing connection to <hostname>:22
connection established
negotiating protocol version
local is `SSH-2.0-Ruby/Net::SSH_4.2.0 x86_64-linux'
remote is `SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u4'
sending KEXINIT
queueing packet nr 0 type 20 len 1156
sent 1160 bytes
read 1080 bytes
received packet nr 0 type 20 len 1076
got KEXINIT from server
negotiating algorithms
negotiated:
* kex: diffie-hellman-group14-sha1
* host_key: ecdsa-sha2-nistp256
* encryption_server: aes128-ctr
* encryption_client: aes128-ctr
* hmac_client: hmac-sha1
* hmac_server: hmac-sha1
* compression_client: none
* compression_server: none
* language_client: 
* language_server: 
exchanging keys
queueing packet nr 1 type 30 len 268
sent 272 bytes
read 504 bytes
received packet nr 1 type 31 len 484
queueing packet nr 2 type 21 len 20
sent 24 bytes
received packet nr 2 type 21 len 12
beginning authentication of `puppet-bolt'
queueing packet nr 3 type 5 len 28
sent 52 bytes
read 52 bytes
received packet nr 3 type 6 len 28
trying none
queueing packet nr 4 type 50 len 60
sent 84 bytes
read 68 bytes
received packet nr 4 type 51 len 44
allowed methods: publickey,password
none failed
trying publickey
trying publickey (1f:06:93:29:9f:b2:aa:b8:29:8e:b9:82:3b:93:fe:75)
queueing packet nr 5 type 50 len 348
sent 372 bytes
read 324 bytes
received packet nr 5 type 60 len 300
Finished: command 'echo $HOME' with 1 failure in 0.16 sec
Failed on 1 node: <hostname>
Ran on 1 node in 0.16 seconds

If there's anything else you can think of to shed some more light, let me know. Though it's 1am here, so I probably wont reply til tomorrow now!

from bolt.

That definitely looks like the public key exchange fails. A successful exchange continues with

queueing packet nr 6 type 50 len 620
sent 644 bytes
read 36 bytes
received packet nr 6 type 52 len 12
publickey succeeded (<fingerprint>)
Opened session
Executing: ...

Is there any special config on sshd you can share? Or your ssh config?

I've tried the same thing from a Debian 9.0 box (connecting to itself) and don't seem to have any issues: bolt command run hostname -n <hostname> --user root --no-host-key-check --private-key ~/.ssh/<private-key>

from bolt.

Are you using a gem install or package? What version of openssl and ruby?

from bolt.

I actually have a bit of an apology to make, I've been running bolt on Arch instead of Debian. A few too many development boxes, and it only really dawned on me after seeing the more recent OpenSSL version than what Debian had. I'm still trying to connect to a Debian 9 box however. That said, I'm running OpenSSL 1.1.1 11 Sep 2018, and ruby 2.5.1p57 (2018-03-29 revision 63029). Here is a gist of the sshd config from the host I am connecting to, although as far as I know its simply the one that ships with Debian.

from bolt.

Just tried arch (updated to today, same OpenSSL and ruby versions) against a Debian 9.0 box that I made sure had the same sshd config. Still no issues.

Some differences still to explore: my host key is ssh-rsa not ecdsa-sha2-nistp256, I'm logging in as root, and I was on Debian 9u1, not u4.

from bolt.

Ok, the ecdsa-sha2-nistp256 (presumably generated with ssh-keygen -t ecdsa) appears to be the problem.

from bolt.

This seems similar to https://tickets.puppetlabs.com/browse/BOLT-380, but not the same. In that ticket net-ssh warns about extra requirements for ed25519.

#418 also refers to a related problem with ECDSA keys.

net-ssh/net-ssh#568 refers to the issue but is not particularly helpful.

from bolt.

I'm going to file a new ticket for this. Not sure how to fix it yet, but something in net-ssh is probably broken.

from bolt.

Filed as https://tickets.puppetlabs.com/browse/BOLT-920.

from bolt.

I'll quickly try disabling ECDSA host keys on the host and see if that fixes it, but it looks promising.
For what it's worth, ECDSA host keys are actually preferred by default, according to the man page for sshd_config (line 249). Does net-ssh support overriding the host key algorithms in the same way the ssh binary does( eg. ssh -oHostKeyAlgorithms=ssh-rsa-cert-v01 [email protected])? Might be useful as a temporary fix til net-ssh supports ECDSA fully.

from bolt.

Bad news - forcing it to a normal ssh-rsa host key does not fix the issue.

bolt command run 'echo $HOME' --nodes <hostname> --debug --trace --verbose --private-key /home/adam/.ssh/puppet-bolt                    
Loaded inventory from /home/adam/.puppetlabs/bolt/inventory.yaml
Analytics opt-out is set, analytics will be disabled
Skipping submission of 'command_run' screenview because analytics is disabled
Started with 100 max thread(s)
Starting: command 'echo $HOME' on <hostname>
Authentication method 'gssapi-with-mic' is not available. Please install the kerberos gem with `gem install net-ssh-krb`
Skipping submission of 'Transport initialize' event because analytics is disabled
Running command 'echo $HOME' on ["<hostname>"]
Running command 'echo $HOME' on <hostname>
Started on <hostname>...
Disabling use_agent in net-ssh: ssh-agent is not available
establishing connection to <hostname>:22
connection established
negotiating protocol version
local is `SSH-2.0-Ruby/Net::SSH_4.2.0 x86_64-linux'
remote is `SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u4'
sending KEXINIT
queueing packet nr 0 type 20 len 1156
sent 1160 bytes
read 1048 bytes
received packet nr 0 type 20 len 1044
got KEXINIT from server
negotiating algorithms
negotiated:
* kex: diffie-hellman-group14-sha1
* host_key: ssh-rsa
* encryption_server: aes128-ctr
* encryption_client: aes128-ctr
* hmac_client: hmac-sha1
* hmac_server: hmac-sha1
* compression_client: none
* compression_server: none
* language_client: 
* language_server: 
exchanging keys
queueing packet nr 1 type 30 len 268
sent 272 bytes
read 848 bytes
received packet nr 1 type 31 len 828
queueing packet nr 2 type 21 len 20
sent 24 bytes
received packet nr 2 type 21 len 12
beginning authentication of `puppet-bolt'
queueing packet nr 3 type 5 len 28
sent 52 bytes
read 52 bytes
received packet nr 3 type 6 len 28
trying none
queueing packet nr 4 type 50 len 60
sent 84 bytes
read 68 bytes
received packet nr 4 type 51 len 44
allowed methods: publickey,password
none failed
trying publickey
trying publickey (1f:06:93:29:9f:b2:aa:b8:29:8e:b9:82:3b:93:fe:75)
queueing packet nr 5 type 50 len 348
sent 372 bytes
read 324 bytes
received packet nr 5 type 60 len 300
Finished: command 'echo $HOME' with 1 failure in 0.17 sec
Failed on 1 node: <hostname>
Ran on 1 node in 0.17 seconds

from bolt.

I have no idea what's going on there. How did you "force" it to ssh-rsa?

from bolt.

Adding these two lines to sshd_config, only allowing it to negotiate with ssh-rsa:

HostKey /etc/ssh/ssh_host_rsa_key
HostKeyAlgorithms [email protected],ssh-rsa

from bolt.

I think that's using the same ecdsa key but forcing ssh-rsa negotiation. I don't see HostKey as a supported config file option; I use IdentityFile to force a specific key.

from bolt.

This is the config file for the ssh daemon on the server bolt is connecting to. IdentityFile is used on the client to select the identity used (eg the key which the client holds the private key for), but the server can also require a specific host key algorithm to be used with HostKey. The file at that path is definitely a rsa key, and theres a corresponding key file there for ECDSA in the same directory, but is not used due to the HostKey directive not specifying it.

ssh-keygen -lf /etc/ssh/ssh_host_rsa_key
2048 SHA256:D3KhxsRIdYxBuYWip3T2mBPBW4P+jwR41QjPXwysYtA root@(none) (RSA)
ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key
256 SHA256:S3QfIRtF7vZusaHUCrUPqYJKRUfcaq+XXWdl/7/Ms0g root@(none) (ECDSA)

The relevant bit from the sshd_config man page:

from bolt.

I'm not sure what's going on with the sshd config. I am suspicious of the key generated via ssh-keygen on Arch though. When I try to parse it with openssl asn1parse -in <private-key>, I get an error. And when I add the ed25519 gem for Bolt to enable it in net-ssh, I get an error expected 64-byte String, got 65.

from bolt.

With https://github.com/crypto-rb/ed25519 installed and an ecdsa key generated using ssh-keygen on my mac (OpenSSH_7.6p1, LibreSSL 2.6.2) or Debian 9 (OpenSSH_7.4p1 Debian-10+deb9u1, OpenSSL 1.0.2l), things seem to work. Arch is running (OpenSSH_7.8p1, OpenSSL 1.1.1).

from bolt.

This seems to be a change in OpenSSH 7.8 (release notes). The file is now written out with the OpenSSH format instead of OpenSSL PEM format. It looks like net-ssh doesn't handle that well for ecdsa keys. You can write out using the old format with -m PEM.

from bolt.

I think net-ssh/net-ssh#633 is relevant. net-ssh is interpreting any OpenSSH format key as ed25519.

from bolt.

The next release of Bolt will give an actual error message (Bolt was hiding the error thrown by net-ssh), but because net-ssh is broken it'll be wrong and unhelpful 🌟. Fixing net-ssh with OpenSSH formatted private keys will take longer.

from bolt.

Is BOLT-920 the only thing left to do on this issue? If so can I close this?

from bolt.