• Summary
• Motivation
• Pre-requirements
• Installation
• Available commands
  • List
  • Read
    • File output
    • JSON output
    • YAML output
• Filtering results
• Configuration
• Running in Docker
• Detecting KV version

RVault (standing for 'recursive vault') is a small cli utility aiming to perform some recursive actions over Hashicorp's Vault.

The supported actions are:

• Listing secrets from a kv engine
• Reading secrets from a kv engine

Read secrets can be saved as files (each key/value will be stored to a separate file), or written to stdout in json/yaml format.

On a scenario where I had to download multiple secrets from Vault, I realized this is currently not supported by vault cli. There is an open issue to request recursive key listing. According to the discussion, recursivity is complex as current list operations are not optimized for recursion. So this is just a workaround until a proper approach is implemented in Vault.

In order for this tool to work you will need:

• A Vault instance to read from (kind of useless without it). I have tested it with Vault 1.4.2 but I guess it should work with any Vault 1.x version.
• A Token with enough priviledges to list the secrets from the selected engine. Additionally, the tool relies on listing current mounts (through the v1/sys/mounts endpoint) to determine the kv version for the engine. If the token doesn't have enough priviledges the version value should be provided either via the configuration or as cli argument.

You can download the binary from the releases and uncompress it somewhere in your path.

For instance, to download the latest release to /usr/local/bin on Linux/MacOS:

version=$(curl -s https://api.github.com/repos/kir4h/rvault/releases/latest | jq -r .tag_name)
platform=$(uname | tr '[:upper:]' '[:lower:]')
curl -L -s https://github.com/kir4h/rvault/releases/latest/download/rvault-${version}-${platform}-amd64.tar.gz \
| sudo tar xz -C /usr/local/bin

You can also simply run the tool from the kir4h/rvault Docker image.

The list allows recursively listing secrets from a given path.

$ rvault list --help

Recursively list secrets for a given path

Usage:
  rvault list <engine> [flags]

Flags:
  -h, --help                help for list
  -k, --kv-version string   KV Version
  -p, --path string         Path to look for secrets (default "/")

Global Flags:
  -a, --address string          Vault address
      --alsologtostderr         log to standard error as well as files
  -c, --concurrency uint32      Maximum number of concurrent queries to Vault (default 20)
      --config string           config file (default is $HOME/.config/rvault/config.yaml)
  -e, --exclude-paths strings   KV paths to be excluded (Applied on 'include-paths' output
  -i, --include-paths strings   KV paths to be included (default [*])
      --insecure                Enables or disables SSL verification
      --log_dir string          If non-empty, write log files in this directory
      --log_file string         If non-empty, use this log file
      --logtostderr             log to standard error instead of files (default true)
  -t, --token string            Vault token
  -v, --v Level                 number for the log level verbosity

For instance, in order to list secrets under v2 secret engine (default one when Vault is launched in dev mode), for the spain subpath:

$ rvault list secret -k 2 -p spain -v=0
/spain/central/ssh.key
/spain/south/passwd.conf

The read command allows recursive read of secrets from a given path. Read secrets can be written to files or to stdout in json/yaml format.

$ rvault read --help

Recursively read secrets for a given path

Usage:
  rvault read <engine> [flags]

Flags:
      --file-permission uint32     Permissions for created secret files (file format only) (default 384)
      --folder-permission uint32   Permissions for newly created folders (file format only) (default 448)
  -f, --format string              Output format ('file', 'yaml', 'json') (default "file")
  -h, --help                       help for read
  -k, --kv-version string          KV Version
  -o, --output string              Output folder for 'file' format  (default ".")
  -w, --overwrite                  Overwrite existing files (file format only)
  -p, --path string                Path to look for secrets (default "/")

Global Flags:
  -a, --address string          Vault address
      --alsologtostderr         log to standard error as well as files
  -c, --concurrency uint32      Maximum number of concurrent queries to Vault (default 20)
      --config string           config file (default is $HOME/.config/rvault/config.yaml)
  -e, --exclude-paths strings   KV paths to be excluded (Applied on 'include-paths' output
  -i, --include-paths strings   KV paths to be included (default [*])
      --insecure                Enables or disables SSL verification
      --log_dir string          If non-empty, write log files in this directory
      --log_file string         If non-empty, use this log file
      --logtostderr             log to standard error instead of files (default true)
  -t, --token string            Vault token
  -v, --v Level                 number for the log level verbosity

• Reading all secrets under v2 secret engine for the spain subpath and store them as files under the /tmp/secret folder

  $ rvault read secret -f file -k 2 -o /tmp/secret -p spain
  I0712 19:04:03.791067   27410 root.go:84] Using config file: '/home/user/.config/rvault/config.toml'
  I0712 19:04:03.795243   27410 secrets.go:193] Secrets written: 2
  
  $ find /tmp/secret -type f
  /tmp/secret/spain/north/ssh.key/key
  /tmp/secret/spain/south/passwd.conf/key

  Please note that in above example, key is just the key name of the only key/value in the secret. If more than one key/value is found in the secret multiple files are created, each one having as name the corresponding key name.

• Reading all secrets under v2 secret engine for the spain subpath, returning them as json

  Running

  rvault read secret -f json -k 2 -p spain 2>/dev/null | jq .

  Will produce the output

  {
      "spain/north/ssh.key": {
          "key": "This is north's secret key"
      },
      "spain/south/passwd.conf": {
          "key": "This is south's secret key"
      }
  }

• Reading all secrets under v2 secret engine for the spain subpath, returning them as yaml

  Running

  rvault read secret -f json -k 2 -p spain 2>/dev/null | jq .

  Will produce the output

  spain/north/ssh.key:
      value: "This is north's secret key"
  spain/south/passwd.conf:
      value: "This is south's secret key"

The include-paths and exclude-paths flags can be used in both list and read commands to return only secrets matching a set of given patterns through wildcard patterns.

Let's say we have a secret engine with the following secrets:

$ rvault list secret -k 2 -v=0
/france/paris/id_rsa
/france/vault.kdbx
/spain/north/id_rsa
/spain/south/shadow
/uk/london/bigben.crypt

We could return only secrets with the id_rsa name:

$ rvault list secret -k 2 -v=0 -i */id_rsa
/france/paris/id_rsa
/spain/north/id_rsa

Or secrets in france and uk:

$ rvault list secret -k 2 -v=0 -i /france/*,/uk/*
/france/paris/id_rsa
/france/vault.kdbx
/uk/london/bigben.crypt

The include filter can also be expressed as a repeteable flag:

$ rvault list secret -k 2 -v=0 -i /france/* -i /uk/*
/france/paris/id_rsa
/france/vault.kdbx
/uk/london/bigben.crypt

We can use the exclude-paths to perform a second filter:

$ rvault list secret -k 2 -v=0 -i /france/* -i /uk/* -e **/*.kdbx
/france/paris/id_rsa
/uk/london/bigben.crypt

The pattern matching is done through the gobwas/go globbing library, so you can refer to its documentation for further reference and valid syntax.

Arguments can be provided as cli arguments but also through means of a configuration file. Viper is used for configuration management so configuration can be stored in any of the formats supported by it (JSON, TOML, YAML, HCL, envfile and Java properties config files).

The default path for the configuration file is $HOME/.config/rvault/config.<ext>, where <ext> represents the extension for the configuration format (json, toml, yaml, ...)

A sample of the configuration in TOML format:

[global]
# Vault address
address = "http://127.0.0.1:8200"
# Vault token
token = "devtoken"
# Log verbosity. 0 (less verbose) to 5 (more verbose)
verbosity = 2
# Maximum number of concurrent queries to Vault. '0' for unlimited (use with care)
concurrency = 20
# List of path patterns to return
include_paths = ["*"]
# List of paths to be excluded from the selected 'include_paths'
exclude_paths = []
# Default kv version to use
kv_version = ""
# Enables or disables SSL Verification
insecure = false

[list]
# Default path to use for listing
path = "/"

[read]
# Default path to use for reading
path = "/"
# Whether to overwrite existing secrets or not when using 'file' format
overwrite = false
# Permissions for newly created files when using 'file' format
file_permission = 0o0600
# Permissions for newly created folders when using 'file' format
folder_permission = 0o0700

#[engines]
## Properties for an engine named 'secret'
#[engines.secret]
## Set engine 'secret' as kv version '2'
## Please note that this takes precedence over 'global.kv_version'
#kv_version=2
## Properties for an engine named 'secret'
#[engines.secretkv1]
## Set engine 'secret' as kv version '1'
## Please note that this takes precedence over 'global.kv_version'
#kv_version=1

Additionally, environment variables VAULT_ADDR and VAULT_TOKEN are bound to global.address and global.token configuration variables for convenience, since they might be already in place under a shell using vault cli. As a last option, token can be retrieved from the ~/.vault-token helper file.

NOTE: If the token are specified in both the vault token helper and configuration files the latter has higher precedence.

A Docker image is built with this repository and pushed to Docker Hub (see rvault Docker image)

There is nothing special about running it in docker other than bind mounting your configuration if desired, or the output folder if using the read command with the file format.

Listing secrets for the secret engine with our custom configuration and the spain path:

$ docker run -it --rm -v ${HOME}/.config/rvault/:/config kir4h/rvault  --config /config/config.toml list secret -p spain
I0715 22:19:36.175957       1 root.go:85] Using config file: '/config/config.toml'
/spain/central/ssh.key
/spain/south/passwd.conf

Reading those secrets and dumping them into files:

$ docker run -it --rm -v ${HOME}/.config/rvault/:/config -v $(pwd)/out:/out kir4h/rvault --config /config/config.toml read secret -o /out
I0715 22:21:46.676216       1 root.go:85] Using config file: '/config/config.toml'
I0715 22:21:46.683996       1 output.go:77] Secrets written: 2

If KV version is not provided, rvault will try to list the mounts by using the v1/sys/mounts endpoint. In order to do so, the provided token must have enough priviledges for that query.

There is an alternative unauthenticated endpoint at /sys/internal/ui/mounts that could be used instead to retrieve KV version, but since it's an internal endpoint it's not exposed by vault library and its usage is discouraged as there is no guarantee on its backwards compatibility.