Failed to unwrap provided token about goldfish HOT 18 CLOSED

If there is no error from vault between panic and goroutine then the wrapped token has been unwrapped successfully, but seems to not contain the required data (the secret ID). This suggests to me that goldfish connected to vault, but the wrapping token did not contain the secret ID (perhaps it wasn't wrapped?)

Have you tried manually generating the wrapped token and then unwrap it manually to see if it contains a secretID at all?

What is the output of this?

WRAPPINGTOKEN=$(vault write -address=http://127.0.0.1:8200 -f -wrap-ttl=20m -format=json auth/approle/role/goldfish/secret-id | jq -r .wrap_info.token)
vault unwrap $WRAPPINGTOKEN

It should look something like this:

Key               	Value
---               	-----
secret_id         	<UUID>
secret_id_accessor	<UUID>

If this works, then generate a WRAPPINGTOKEN again and give the wrapping token to goldfish via command line

from goldfish.

root@host:~# WRAPPINGTOKEN=$(vault write -f -wrap-ttl=20m -format=json auth/approle/role/goldfish/secret-id | jq -r .wrap_info.token)
root@host:~# echo $WRAPPINGTOKEN
<UUID>

The command has always worked but starting up goldfish gives me that error every single time. I don't really know how to unwrap the token unfortunately. I'll go have a look at vaults docs and see if I can divine it out.

from goldfish.

I specified unwrapping right below:

vault unwrap $WRAPPINGTOKEN

Just because I'm paranoid, try not to paste any non-dev UUIDs :)

To be more specific, the bash script I added in my previous post should generate exactly what the result looks like. Let me know if it does not.

from goldfish.

I just found it
root@host:~# vault unwrap $WRAPPINGTOKEN
Key Value

secret_id
secret_id_accessor <accessor_value>

And you beat me to the edit above :P

from goldfish.

And running the below results in error? wrappingtoken must be regenerated

WRAPPINGTOKEN=$(vault write -f -wrap-ttl=20m -format=json auth/approle/role/goldfish/secret-id | jq -r .wrap_info.token)
goldfish -config=config.hcl -token=$WRAPPINGTOKEN

from goldfish.

Yes it does.

root@host:/opt/goldfish# WRAPPINGTOKEN=$(vault write -f -wrap-ttl=20m -format=json auth/approle/role/goldfish/secret-id | jq -r .wrap_info.token)
root@host:/opt/goldfish# ./goldfish -config=config.hcl -token=$WRAPPINGTOKEN
panic: Failed to unwrap provided token, revoke it if possible

goroutine 1 [running]:
main.main()
	/Users/tony/work/src/github.com/caiyeon/goldfish/server.go:77 +0x1dd
root@host:/opt/goldfish#

from goldfish.

I would like to add some log prints to your binary. Are you compiling yourself? (I can attach a single go file) or, if you aren't compiling yourself, I can just attach a binary

from goldfish.

I'm using your pre-compiled binaries. I see no reason to hurt myself and compile it on my stuff ;)

from goldfish.

That's the preferred way. But I suppose it makes me giving you a unknown binary more dangerous. Are you willing to run an attached binary?

from goldfish.

I've got a dev system that I can run it on so I feel modestly safe

Stepping away from the keyboard for little bit, time to drive home but don't worry I have a vault server there that I was testing all the systemd stuff I was doing out on so I can still run your binary (it has the same problem)

from goldfish.

Sanity check: your $VAULT_ADDR and the vault's address in config file are the same, right?

If so:
In this zip is the modified go file and the compiled binary (for linux amd64)
wrappingtest.zip

Try

WRAPPINGTOKEN=$(vault write -f -wrap-ttl=20m -format=json auth/approle/role/goldfish/secret-id | jq -r .wrap_info.token)
./goldfish-wrappingtest -config=config.hcl -token=$WRAPPINGTOKEN

If this errors out, it will at least have a specific enough error to let us know where it's coming from

Thanks for your patience! I'm not sure why this doesn't work, since the vagrant build uses a completely separate vault instance (and would only differ from your build by https/certificates)

from goldfish.

I quadruple checked that $VAULT_ADDR and the address in the config file are the exact same.

root@host:/opt/goldfish# ./goldfish-wrappingtest -config=config.hcl -token=$WRAPPINGTOKEN
panic: Response from vault is nil

goroutine 1 [running]:
main.main()
	/Users/tony/work/src/github.com/caiyeon/goldfish/server.go:77 +0x1dd

The wrapping token is valid and unwraps properly using vault unwrap.

from goldfish.

Just tried going via IP instead of DNS name. No joy on that, same error. I started up a fresh vault instance on the system with goldfish on it just to see if it was related to the two items living in different locations this is what I got when starting it up. The wrapping token I got was indeed valid. I enabled the audit backend of syslog and I go other useful data out of it.

./goldfish-wrappingtest -config=config.hcl -token=$WRAPPINGTOKEN
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xdf9c03]

goroutine 1 [running]:
github.com/caiyeon/goldfish/vendor/github.com/hashicorp/vault/api.(*Client).NewRequest(0xc4201afda0, 0x1afd069, 0x3, 0x1b2421f, 0x17, 0x1b29fcb)
	/Users/tony/work/src/github.com/caiyeon/goldfish/vendor/github.com/hashicorp/vault/api/client.go:340 +0x43
github.com/caiyeon/goldfish/vendor/github.com/hashicorp/vault/api.(*Logical).Unwrap(0xc42040ddb0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/tony/work/src/github.com/caiyeon/goldfish/vendor/github.com/hashicorp/vault/api/logical.go:132 +0xe2
github.com/caiyeon/goldfish/vault.StartGoldfishWrapper(0x7ffd44a5688f, 0x24, 0x1b18d8f, 0x12, 0x1b03af3, 0x8, 0x4d6645, 0x27bd7a0)
	/Users/tony/work/src/github.com/caiyeon/goldfish/vault/vault.go:66 +0x158
main.main()
	/Users/tony/work/src/github.com/caiyeon/goldfish/server.go:76 +0x1ba

from goldfish.

In config.hcl, is vault's address prefixed with the protocol? E.g. http:// or https://?

from goldfish.

I missed that. When I put in https I get nil....changed to http and it came up this time. That's super weird to me since I have valid ssl certs in place...

from goldfish.

So https -> nil, http -> goldfish launches correctly?
Which one is your $VAULT_ADDR set to?

from goldfish.

inside config.hcl it's set to http since I just forced it that direction. $VAULT_ADDR set to http as well when I generate the wrapping token.

from goldfish.

Then it seems like goldfish is acting identical to the vault CLI. I would assume that if you got vault CLI to work with $VAULT_ADDR in https, goldfish would also mimic vault's behaviour.

However, this does raise the concern that a non scheme'd address should not have been accepted in the first place, just like vault CLI itself. So I'll be changing that, and closing this ticket once that's done.

Cheers, and thanks for the debugging!

from goldfish.