You cannot update a config file when deployed via Helm to Kubernetes. If you access vault pod with shell, you do not have user permissions to edit file. Unable to add plugin field, so therefore unable to add Venafi plugin. Please help.

BUSINESS PROBLEM
Today the integration does not support JWT as an authentication method for the plugin. Adding JWT support would bring the integration to parity with other Venafi integrations. VCert SDK and VCert CLI already support JWT, so it's a matter of making those variables available to be passed from the plugin. Today OAuth is the primary authentication method within the plugin.

PROPOSED SOLUTION
Allow JWT an an authentication method.

CURRENT ALTERNATIVES
None. Currently authenticate with Oauth.

PROBLEM SUMMARY
Chain in response to certificate requests differs from the native Vault PKI secrets engine

STEPS TO REPRODUCE

1. Initialize a Vault server and unseal it.
2. Register the vault-pki-backend-venafi plugin in the Vault system catalog.
3. Enable the native Vault PKI secrets engine on path called "pki-native" and create the required local CA and role.
4. Enable the Venafi PKI secrets engine on path called "pki-backend" and create the required Venafi secret and role.
5. Request a certificate from the native Vault PKI secrets engine both ways, "issue" and "sign".
   • vault write -format=json pki-native/issue/tls-server common_name=issue-me.venafi.example
   • vault write -format=json pki-native/sign/tls-server common_name=sign-me.venafi.example
6. Request a certificate from the Venafi PKI secrets engine both ways, "issue" and "sign".
   • vault write -format=json pki-backend/issue/tls-server common_name=issue-me.venafi.example
   • vault write -format=json pki-backend/sign/tls-server common_name=sign-me.venafi.example
7. Compare the responses.

EXPECTED RESULTS
The native Vault PKI secrets engine and Venafi PKI secrets engine responses mirror each other (same parameters for certificate, private key, chain certificates, etc.). The Venafi PKI secrets engine should be a "drop in place" substitute for the native Vault PKI secrets engine so that any consumer that works with the native Vault PKI secrets engine can successfully use this solution.

ACTUAL RESULTS
The native Vault PKI secrets engine responses include ca_chain (array of the CA certificates) and issuing_ca (just the CA certificate of the issuer) whereas the Venafi PKI secrets engine responses include certificate_chain (string comprised of all certificates, CA and end-entity, concatenated together).

Generate Certificate:

# vault write -format=json pki-native/issue/tls-server common_name=test.venafi.example
{
  "request_id": "0f18eca4-b9c6-ff7c-f84a-7aea2bbae7cf",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "ca_chain": [
      "-----BEGIN CERTIFICATE-----\n ... \n-----END CERTIFICATE-----"
    ],
    "certificate": "-----BEGIN CERTIFICATE-----\n ... \n-----END CERTIFICATE-----",
    "expiration": 1608492102,
    "issuing_ca": "-----BEGIN CERTIFICATE-----\n ... \n-----END CERTIFICATE-----",
    "private_key": "-----BEGIN RSA PRIVATE KEY-----\n ... \n-----END RSA PRIVATE KEY-----",
    "private_key_type": "rsa",
    "serial_number": "3c:c2:6e:d1:c6:2a:7b:22:c0:82:0f:0f:ef:55:d2:a5:da:0e:0c:5d"
  },
  "warnings": null
}
# vault write -format=json pki-backend/issue/tls-server common_name=test.venafi.example
{
  "request_id": "e5090a81-8e39-1a2c-0fb1-e862e1ed01cc",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "certificate": "-----BEGIN CERTIFICATE-----\n ... \n-----END CERTIFICATE-----\n",
    "certificate_chain": "-----BEGIN CERTIFICATE-----\n ... \n-----END CERTIFICATE-----\n",
    "common_name": "test.venafi.example",
    "private_key": "-----BEGIN RSA PRIVATE KEY-----\n ... \n-----END RSA PRIVATE KEY-----\n",
    "serial_number": "33:51:fd:7e:00:00:00:06:b5:54"
  },
  "warnings": [
    "Read access to this endpoint should be controlled via ACLs as it will return the connection private key as it is."
  ]
}

Sign Certificate:

# vault write -format=json pki-native/sign/tls-sign [email protected] ttl=24h
{
  "request_id": "bf9f5a6e-ed0e-9833-0fcd-6de6f5601e9a",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "ca_chain": [
      "-----BEGIN CERTIFICATE-----\n ... \n-----END CERTIFICATE-----"
    ],
    "certificate": "-----BEGIN CERTIFICATE-----\n ... \n-----END CERTIFICATE-----",
    "expiration": 1605985715,
    "issuing_ca": "-----BEGIN CERTIFICATE-----\n ... \n-----END CERTIFICATE-----",
    "serial_number": "20:3c:8e:81:27:7d:a8:fb:c6:1e:da:11:a6:dd:1c:8c:f0:98:72:21"
  },
  "warnings": null
} 
# vault write -format=json pki-backend/sign/tls-sign [email protected] ttl=24h
{
  "request_id": "77538bea-f792-b899-7c68-67219e1d9e4d",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "certificate": "-----BEGIN CERTIFICATE-----\n ... \n-----END CERTIFICATE-----\n",
    "certificate_chain": "-----BEGIN CERTIFICATE-----\n ... \n-----END CERTIFICATE-----\n",
    "common_name": "",
    "serial_number": "32:5e:be:b5:00:00:00:06:b5:52"
  },
  "warnings": null
}

ENVIRONMENT DETAILS
Venafi Trust Protection Platform 20.3.2 or Venafi Cloud
HashiCorp Vault 1.5.5

COMMENTS/WORKAROUNDS
The fix for this issue should add the ca_chain and issuing_ca to the response but should not remove the certificate_chain because that would be a breaking change. Also, it looks like when using the "sign" operation, the Venafi-only common_name parameter is not being populated with the CN of the issued certificate.

PROBLEM SUMMARY

We're trying to use this plugin with a command like:

vault write mount/venafi/pki/myrole common_name=mycert.mydomain.example ttl=288h

And there are several problems:

1. The max_ttl of the PKI role is not enforced
2. The ttl and max_ttl of the role seem to have no effect
3. The returned ttl seems to be based only on the Venafi "template" for the provider. Again, the client's specified ttl seems to be ignored.

STEPS TO REPRODUCE

vault write venafi/pki/roles/myrole \
generate_lease=true \
store_by_cn=true \
store_pkey=true \
tpp_user=xxx \
tpp_password=xxx \
zone='mypath\myzone' \
tpp_url='https://venafi-host:443/VEDSDK' \
ttl=720h \
max_ttl=720h

vault write venafi/pki/myrole common_name=mycert.mydomain.example ttl=900h

EXPECTED RESULTS

The ttl should be honored - the duration of the certificate should be equal to the ttl
when the requested ttl > max_ttl - request is rejected, no request made to Venafi

ACTUAL RESULTS

ttl and max_ttl in the role and the request seem to be ignored. This seems related to #33 .

ENVIRONMENT DETAILS
Vault 1.2.2, backend 0.5.2

COMMENTS/WORKAROUNDS

It looks like support is needed for the CA specific attributes CASpecificAttributes - but how can these vary based on the role / request?

Hi Team,

I have a customer that is running into some error messages while they are using the Vault-PKI-Backend-Venafi. They would like to know the reason for these errors so the devops team can create some error handling to better address these errors as they come up.

Here are the errors they are concerned about:

1. {"errors":["unable to retrieve: Unexpected status code on TPP Certificate Retrieval. Status: 500 Certificate \VED\Policy\Integrations\HashiCorp\Test\Standard\gposetup-rms-oytydev1.ose-dev39-red.aws-use1.cloud.marriott.com has encountered an error while processing, Status: This certificate cannot be processed while it is in an error state. Fix any errors, and then click Retry., Stage: 500."]}

NOTE: We believe this error is related to the CA not responding in time and Venafi places the cert in Error

2. ERROR: {"errors":["2 errors occurred:\n\t* errors from both primary and secondary; primary error was unable to retrieve: Post https://venafiintegration.marriott.com/vedsdk/certificates/retrieve: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers); secondary errors follow\n\t* unable to retrieve: Post https://venafiintegration.marriott.com/vedsdk/certificates/retrieve: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n\n"]}

NOTE: We believe this is due to Venafi not getting the cert within the 60 second time limit for the CSR to stay within HashiCorp Vaults memory, and the cert cannot be completed

3. [ERROR] core: failed to register lease: request_path=venafi-pki/issue/tpp-backend error="rpc error: code = Canceled desc = context canceled""}

NOTE: Not sure what caused this, but we had a lot of these at once, so maybe an issue of Vault reaching TPP?

Let me know if you need any additional information for this request.

Thanks,
Geoff

Error writing data to venafi-pki/sign/tpp-backend: Error making API request.

Code: 404. Errors:

• 1 error occurred:
  * unsupported path

PROBLEM SUMMARY
We are attempting to enable the Venafi plugin, but run into errors when setting up the role. We are following these instructions: https://www.vaultproject.io/docs/secrets/venafi

We are successfully able to get the Venafi engine enabled (we have tried at both the root and within a namespace), but the role fails with an error 500.

STEPS TO REPRODUCE

1. Follow steps 1-7 in https://www.vaultproject.io/docs/secrets/venafi:

wget https://github.com/Venafi/vault-pki-backend-venafi/releases/download/v0.6.2/venafi-pki-backend_v0.6.2+743_linux.zip
unzip venafi-pki-backend_v0.6.2+743_linux.zip
sudo mkdir -p /etc/vault/vault_plugins
sudo cp venafi-pki-backend /etc/vault/vault_plugins
sudo chown -R vault:vault /etc/vault/vault_plugins
sudo chmod -R go-rwX /etc/vault/vault_plugins

echo "plugin_directory = "/etc/vault/vault_plugins"" | sudo tee -a /etc/vault.d/vault.hcl > /dev/null
sudo systemctl restart vault

(Above performed on all Vault servers)

SHA256=$(sudo sha256sum /etc/vault/vault_plugins/venafi-pki-backend | cut -d' ' -f1)
vault write sys/plugins/catalog/secret/venafi-pki-backend sha_256="${SHA256}" command="venafi-pki-backend"
vault secrets enable -path=venafi-pki -plugin-name=venafi-pki-backend plugin

2. Step 8, create the role (command below - performed by on client with PowerShell and locally on server)

vault write -ns=techmod venafi-pki/roles/tpp tpp_url="https://<myserver>/vedsdk"
tpp_user="[email protected]" tpp_password='' zone="\\Automation\\InternalAutomation\\Vault"
generate_lease=true store_by_serial=true store_pkey=true allowed_domains=domain.org
allow_subdomains=true

EXPECTED RESULTS
Role successfully created

ACTUAL RESULTS
Receive the below error:
Error writing data to venafi-pki/roles/tpp: Error making API request.

URL: PUT https://:8200/v1/venafi-pki/roles/tpp
Code: 500. Errors:

• 1 error occurred:
• internal error

sudo tail -n1 /var/log/vault/audit.log
{"time":"2020-05-21T23:43:37.024959318Z","type":"response","auth":{"token_type":"default"},"request":{"id":"350ffd79-cf39-78bb-669d-4c6a54b84d82","operation":"update","client_token":"","client_token_accessor":"","namespace":{"id":"rMChS","path":"techmod/"},"path":"venafi-pki/roles/tpp","data":{"allow_subdomains":"","allowed_domains":"","generate_lease":"","store_by_serial":"","store_pkey":"","tpp_password":"","tpp_url":"","tpp_user":"","zone":""},"remote_address":""},"response":{},"error":"1 error occurred:\n\t* internal error\n\n"}

ENVIRONMENT DETAILS
OS: RHEL 7
Vault: v1.4.0
Venafi: TPP 19.2
Plugin: v0.6.2 (also tried v0.6.1)

COMMENTS/WORKAROUNDS
We have validated with our Venafi engineer that the service account can hit the vedsdk authentication endpoint API and return a token (manually via Postman), and Hashicorp support has validated we followed the correct steps for setup. We also did password rotations to ensure certain special characters were not causing issues, and changed the user/domain syntax - however, Venafi IIS logs never show an authentication attempt when setting up the role so we don't believe it is actually hitting the Venafi server. We can hit the Venafi instance via curl from the Vault servers without issue.

For reference, we are using CIS Hardening standards on the OS, but I have granted the vault service account full ownership of the plugin folder/file.

PROBLEM SUMMARY
<!- When I try to write to venafi-pki-backend/venafi/tpp, with all of the parameters needed, I get an error message. This issue only occurs with tokens that are not root.

STEPS TO REPRODUCE

COMMENTS/WORKAROUNDS

BUSINESS PROBLEM
There are customers that want to utilize this plugin to manage different certificates (for different servers) with the same subject and SANs.

Currently, when all those certificates are renewed all of them but one will be overriden. That is, they will all be mapped to the exact same object in TPP, so the certificate that was renewed last will be that object's current certificate and the rest will be put in the history.

PROPOSED SOLUTION
If the plugin allowed providing a DN or nickname in the request then only the corresponding object in TPP's policy tree will be affected, so for each server there could be a "certificate-X" object in TPP with X being a server-specific string that differentiates it from the rest.

CURRENT ALTERNATIVES
There is no good alternative.

Using an adatable log you could move somewhere else the generated certificate object right after first-issuance, so next issuance of a certificate with the same subject and SANs will not clash with that one. However, that means too that when all these certificates are renewed they will be created as individual certificates instead of being put inside the history of its corresponding certificate.

VENAFI EXPERIENCE
5 years.

I am using plugin's /issue endpoint for generating a cert from venafi. It worked fine until our cert admin team requires a CustomFields in the Venafi /vedsdk/certificates/Request API. But I don't find a way to add the name and value into either the Role or /issue request. Could you help to add this functionality? Thanks.

BTW, cert-manager didn't support CustomFields before but now it adds supports too.

BUSINESS PROBLEM

We have many thousands of services, each with different subdomains, that are reading their certificates from Vault through the venafi-pki plugin.

Currently, we limit access to issue new certificates with a policy path such as:

venafi-pki/issue/*.domain.com

However, because of the amount of services that could be restarting at any given time, we have a vault-agent config that checks for certificates existing first on the venafi-pki/cert/ path, and if they don't exist then the venafi-pki/issue/ path will be targeted for issuing a new one.

This is needed, because at any given time we could be generating thousands of new certificates otherwise, which we want to avoid because of the costs associated with issuing that many new publicly signed certificates.

In order for our services to read existing certs from venafi-pki/cert/ however, we are using a wildcard policy path:

venafi-pki/cert/*

This is a problem, because it means that any service with read access on that path can request any certificate that currently exists. In order to limit access using the current existing features in the venafi-pki plugin, we would have to have policies for each specific subdomain:

venafi-pki/cert/example-1.domain.com
venafi-pki/cert/example-2.domain.com
venafi-pki/cert/example-3.domain.com
etc...

However, because of the amount of domains that certificates will be issued for, managing thousands of subdomains in our policy list isn't feasible. We want to have policies such as:

venafi-pki/cert/*.domain.com
venafi-pki/cert/*.domain-2.com
venafi-pki/cert/*.domain-3.com

This would allow us to limit read access by domain, and allow us to have a much smaller more manageable policy set.

PROPOSED SOLUTION

The Vault PKI engine has a feature called allow_glob_domains, that allows for domains to contain glob patterns, such as ftp*.example.com or *.example.com.

The role in venafi should have the alow_glob_domains boolean flag added to pki/path_roles.go, and the ability to have wildcards in the paths added topki/path_venafi_cert_read.go based on this role flag.

The result should be that the following policy paths should be valid:

venafi-pki/cert/*.domain.com
venafi-pki/cert/*.example.domain.com
venafi-pki/cert/ftp*.domain.com
venafi-pki/cert/example.*.com

CURRENT ALTERNATIVES

In order for our services to read existing certs from venafi-pki/cert/ we have to use a policy path like:

venafi-pki/cert/*

This will, however, not pass our audits. In order for us to pass auditing, we would have to specify each subdomain explicitly:

venafi-pki/cert/example-1.domain.com
venafi-pki/cert/example-2.domain.com
venafi-pki/cert/example-3.domain.com
etc...

But, due to the amount of certs we have at any given time, scaling up our deployment won't be possible. This will soon become a blocker for us deploying more services onto this venafi+vault workflow.

VENAFI EXPERIENCE

👍

Readme.md contains as a first step to use the following commands to exact the plugin:

wget https://github.com/Venafi/vault-pki-backend-venafi/releases/download/v0.3-11.5-alpha.161/venafi-pki-backend_0.3-11.5.161_linux.zip
unzip venafi-pki-backend_0.3-11.5.161_linux.zip
mv vault-pki-backend-venafi /etc/vault/vault_plugins

The extracted filename in the archive is venafi-pki-backend, so the last command should be
mv venafi-pki-backend /etc/vault/vault_plugins

test
test
ryba

OS version:
CentOS Linux release 7.5.1804 (Core)

Docker version:
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:23:03 2018
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:25:29 2018
OS/Arch: linux/amd64
Experimental: false

Problem Description:
When using 'make prod' per the README to setup the integration, the process fails right after the Vault containers are created. I can recover by logging into the Vault container and executing the setup steps to initialize the vault manually, but the first use experience with the make commands fails. Here is relevant output:

[wgoulet@linuxdesktop vault-pki-backend-venafi]$ make prod
/bin/sh: vault: command not found
Using vault client version 
docker-compose down --remove-orphans
Removing vaultpkibackendvenafi_vault_1 ... done
Removing vaultpkibackendvenafi_consul_1 ... done
Removing network vaultpkibackendvenafi_default
docker-compose up -d
Creating network "vaultpkibackendvenafi_default" with the default driver
Creating vaultpkibackendvenafi_consul_1
Creating vaultpkibackendvenafi_vault_1
Run: docker-compose logs
to see the logs
Run: docker exec -it cault_vault_1 sh
to login into vault container
Waiting until server start
sleep 10
docker exec -it $(docker-compose ps |grep Up|grep vault_1|awk '{print $1}') vault operator init -key-shares=1 -key-threshold=1
Error: No such container: vault
make: *** [prod_server_init] Error 1

PROBLEM SUMMARY
Our Venafi instance is using Entrust CA GW and have a limit of 20 concurrent connections currently. To compensate for the 60 second TTL for the CSRs within Hashicorp Vault we have worked with the DevOps team to create retries after 5 or 10 seconds after receiving errors from Hashicorp vault. In our load tests with the retries we had some certificates returned where the public key did not match the private key. We dug into this issue and found some of the retries were happening while Venafi was in the middle of processing the first request and it had not errored out. Venafi ended up processing both certificates through the CA within seconds of each other. Unfortunately, the certificate that was returned to vault was the Frist requested certificate, but the private key was from the second request. It does not look like the backend tool does any validation to ensure that the Public key matches the Private Key, which should of returned an error to the pipeline.

STEPS TO REPRODUCE
Configure Entrust CA GW Template and configure it with 20 concurrent sessions. Create pipeline to generate 60 threads to request 25 individual certificates equaling around 1500 certs. If an error is returned from vault then retry certificate 5 seconds later.

EXPECTED RESULTS
vault-pki-backend-venafi tool would check the public key and private key to ensure they match, if no match discard certificate and return an error

ACTUAL RESULTS
Public Key and Private key do not match, it is stored in Vault and the pipeline receives a invalid cert.

ENVIRONMENT DETAILS
Versions used in environment:
Venafi: 20.3
vault-pki-backend-venafi: v0.9.0

COMMENTS/WORKAROUNDS
In this test we submitted enough certs that would force the retries to take action, which was 60 threads of 25 individual certs for a total of 1500 certificates. This is a extreme test, but the fact that the tool is not validating the keys it could cause an outage. With using a public CA there could be slow downs that this could happen with less requests, and want to avoid this in the future.

PROBLEM SUMMARY
A generic error message may result when trust_bundle_file is not specified during a vault write -address https://vault.amplisight.io:8200 venafi-pki/venafi/tpp url="https://tpp-amplisight-trial.venafidemo.com/vedsdk" access_token="XXX" refresh_token="XXX" zone="Vault-DevSecOps-Demo" command is run.

The resulting error is:

Error writing data to venafi-pki/venafi/tpp: Error making API request.
URL: PUT https://vault.amplisight.io:8200/v1/venafi-pki/venafi/tpp
Code: 500. Errors:
* 1 error occurred:
        * internal error

EXPECTED RESULTS
The expected result is Success! Data written to: venafi-pki/venafi/tpp

COMMENTS/WORKAROUNDS
A better error message might direct the user to specify the trust_bundle_file

PROBLEM SUMMARY
The default setting of a PKI role for the vault-pki-backend-venafi plugin is store_by=serial. However the certificate name becomes the serial and then searching for a particular certificate is not easy. If we switch to store_by=cn however, the plugin loses integrity and returns invalid certificate bundles. What I mean by this is the following:
When you request a certificate using the /issue endpoint you expect to get a cert A and private key A.
When you request the same cert (i.e. with the same CN and everything in it), you expect to get cert B and private key B.
The two certs are essentially the same but they should have different serials, valid from/to dates and private keys. Their modulus should also be different.
When you configure the role with store_by="cn" and store_pkey=true, however, when you issue the second key you get cert A and private key B. Since the integrity is broken the combo is invalid.

STEPS TO REPRODUCE
Set a PKI role with store_by=cn and issue two certs with the same CN and other settings one after another.

EXPECTED RESULTS
The certificate field in the secret should be updated with the latest certificate.

ACTUAL RESULTS
the certificate field is not updated with the latest certificate, only the private key field is update. This breaks the integrity of the generated certificate.

ENVIRONMENT DETAILS
I am using the latest stable version

COMMENTS/WORKAROUNDS
The options combinations which I found that do not break integrity are:

1. no_store=true
2. store_by=serial (store_pkey=false)
3. store_by=serial (store_pkey=true)

BUSINESS PROBLEM
Secrets Manager does not support oAuth.

PROPOSED SOLUTION
Please include support for oAuth.

CURRENT ALTERNATIVES
API keys.

VENAFI EXPERIENCE
Daily for 6+ years.

PROBLEM SUMMARY
The plugin isn't being listed in the Secrets list when the plugin has a version set to it when initialized.

STEPS TO REPRODUCE

vault write sys/plugins/catalog/secret/venafi-pki-backend sha_256="${SHA256}" command="venafi-pki-backend" version="0.12.2"
Success! Data written to: sys/plugins/catalog/secret/venafi-pki-backend
vault read --format=json /sys/plugins/catalog
{
      {
        "builtin": false,
        "name": "venafi-pki-backend",
        "sha256": "af545920446f0ff953123f88547aa14983bc227935cc2e767151f6746a659ceb",
        "type": "secret",
        "version": "v0.12.2"
      }
}
{
      "secret": [
      "ad",
      "alicloud",
      "aws",
      "azure",
      "consul",
      "gcp",
      "gcpkms",
      "kubernetes",
      "kv",
      "ldap",
      "mongodbatlas",
      "nomad",
      "openldap",
      "pki",
      "rabbitmq",
      "ssh",
      "terraform",
      "totp",
      "transit"
    ]

}

Notice how the venafi-vault plugin doesn't get listed under the secret list even though it returns type secret

EXPECTED RESULTS
The plugin when initialized shows up in the secret list.

ACTUAL RESULTS
The plugin does not show up on the secrets list.

ENVIRONMENT DETAILS
Vault 1.14.2. Plugin 0.12.2

COMMENTS/WORKAROUNDS
Use vault read --format=json /sys/plugins/catalog |jq .data.detailed to get the plugin information.

BUSINESS PROBLEM
When getting a certificate from Venafi Secrets Engine you get the certificate_chain, but in our case we need the ca_chain. I noticed that ca_chain is available on POST request and wondered if it could be added to GET also. This will allow us to just pull the chain instead of pulling the certificate chain and having to take extra engineering steps to remove the certificate.

PROPOSED SOLUTION
Add Property ca_chain to GET request

CURRENT ALTERNATIVES
Once you get the certificate chain you have to use regex or special tools to pull out the certificate.

PROBLEM SUMMARY
Receiving TTL errors while using the monitor plugin. Original issue was " we can issue the cert, sign it, but can't save it in Venafi. also getting some strange warnings logged: (TTL of "2159h59m59.500822605s" exceeded the effective max_ttl of "768h"; TTL value is capped accordingly) As if it is trying to cap on TTL of logged in identity.}}"

This seemed to be resolved by editing the settings on Vault and changing the ttl value. The error no longer showed. However the new value is throwing a different error.

Now, receiving "Now ttl=”100h” term is giving an error ("error":"1 error occurred:\n\t* permission denied\n\n"}) " when attempting certificate issuance.

STEPS TO REPRODUCE
Change the ttl on vault to 100h. This TPP environment is hosted in Azure. Exact reproduction steps are unknown, as we only know what they are receiving when setting the TTL to 100h.
TPP version 20.2
Vault plugin version: 0.7.1

EXPECTED RESULTS
Expect to see regular issuance of certificates and not be getting these errors within the plugin. We know that the Venafi plugin lacks the functionality to consider the TTL settings from vault anyway, but wondering why we are getting this new permission denied error.

ACTUAL RESULTS
Now ttl=”100h” term is giving an error ("error":"1 error occurred:\n\t* permission denied\n\n"})

ENVIRONMENT DETAILS
This is an Azure environment. TPP version is 20.2 Vault plugin version is 0.7.1.

COMMENTS/WORKAROUNDS

Hi,

A Venafi Cloud user recently attempted to use this plugin and received the following error:
`URL: PUT https://vault-test-example.com/v1/venafi-pki/issue/cloud-backend
Code: 400. Errors:

• failed to get Venafi issuer client: Get https://api.venafi.cloud/v1/useraccounts: x509: certificate signed by unknown authority
  `
  The user provided additional debug logs which indicate that a SSL interception device is in use for connections to api.venafi.cloud. As a result, the issuing chain for api.venafi.cloud from the plugin's perspective is not trusted. Can the trust_bundle option supported for connections to TPP be added for connections to the Cloud endpoint?

Thanks,
Walter

BUSINESS PROBLEM
Some cloud services require certificates in PKCS12 format. This format is not supported by Venafi Secrets Engine today.

PROPOSED SOLUTION
Add support to getting Certificates in PKCS12 format.

CURRENT ALTERNATIVES
No

VENAFI EXPERIENCE
5 years

Hello team,

We are planning to use the latest version of this plugin, but when running the package through our sec scan tool, we got vulnerability hits on the Golang version.
We are assuming it's a false positive but wanted to make sure & re-run the scan with a targeted Golang version.
We asked the question to Venafi support, but they directed us to post a GH issue to get the answer.

Could you specify which version of Golang was used for plugin version 0.12.1.

Also as a feature request, I think it will be beneficial to include the version of Golang used in the changelog,
an example would be the HashiCorp Vault changelog, which calls out the Golang version for each release.

Appreciate the help :)

Dependency line:

github.com/Venafi/vault-pki-backend-venafi --> github.com/hashicorp/vault --> gopkg.in/ory-am/dockertest.v2 --> github.com/dancannon/gorethink

gopkg.in/ory-am/dockertest.v2 v2.2.3 --> github.com/dancannon/gorethink v2.1.3
https://github.com/ory/dockertest/blob/v2.2.3/glide.yaml#L14

- package: github.com/dancannon/gorethink
  version: ~2.1.3

Background

1. The gorethink has already renamed it’s import path from "github.com/dancannon/gorethink" to "gopkg.in/gorethink/gorethink.v2", in the version v2.1.3 .
As README of gorethink v2.1.3 said, downstream repos should use "gopkg.in/dancannon/gorethink.v2" to get or import gorethink.

**Installation**
go get gopkg.in/dancannon/gorethink.v2

**Example**
package gorethink_test

import (
	"fmt"
	"log"

	r "gopkg.in/dancannon/gorethink.v2"
)
…

But gopkg.in/ory-am/dockertest.v2 still used the old path:
https://github.com/ory/dockertest/blob/v2.2.3/glide.yaml#L14

package: github.com/dancannon/gorethink
  version: ~2.1.3

So module pulled the last version which didn’t have go.mod, v4.0.0. From the Go Modules's point of view, path github.com/dancannon/gorethink equals to version v0/v1 or the latest version that didn’t use the module.

2. I find that gopkg.in/gorethink/gorethink.v4 and github.com/dancannon/gorethink coexist in this repo：
https://github.com/Venafi/vault-pki-backend-venafi/blob/master/go.mod （Line 34 & 116）

github.com/dancannon/gorethink v4.0.0+incompatible // indirect
gopkg.in/gorethink/gorethink.v4 v4.1.0 // indirect 

That’s because the gorethink has already renamed it’s import path from "github.com/dancannon/gorethink" to "gopkg.in/gorethink/gorethink.v4"，in the version v4.x.y . When you use the old path "github.com/dancannon/gorethink" to import the gorethink, will reintroduces gorethink through the import statements "import gopkg.in/gorethink/gorethink.v4" in the go source file of gorethink.

https://github.com/rethinkdb/rethinkdb-go/blob/v4.0.0/query_db.go#L4

package gorethink
import (
	p "gopkg.in/gorethink/gorethink.v4/ql2"
)

"gopkg.in/gorethink/gorethink.v4" and "github.com/dancannon/gorethink" are the same repos. This will work in isolation, bring about potential risks and problems. And actually the version that gopkg.in/ory-am/dockertest.v2 required is v2.1.3. All of these can bring potential problems

Solution

1. Add replace statement in the go.mod file:

replace github.com/dancannon/gorethink => gopkg.in/gorethink/gorethink.v2 v2.1.3

Or keep use the version v4.0.0. v2.1.3 is too old, might bring some old incompatible path, such as "github.com/Sirupsen/logrus".

replace github.com/dancannon/gorethink => gopkg.in/gorethink/gorethink.v4 v4.0.0

Then clean the dependencies.
2. Update the direct dependency github.com/hashicorp/vault. The latest version of github.com/hashicorp/vault is v1.5.0. This problem does not exist in the new version.

PROBLEM SUMMARY
When using the backend plugin with a Microsoft ADCS Certificate Authority the shortest validity that can be received by the CA is 24 h

STEPS TO REPRODUCE
Install and configure vault-pki-backend plugin, set issuer_hint=m

vault read myMSCA/roles/tpp
Key Value

chain_option last
generate_lease false
issuer_hint m
max_ttl 0s
no_store false
role_zone n/a
service_generated_cert false
store_by n/a
store_pkey false
ttl 0s
venafi_secret tpp

EXPECTED RESULTS

Request a certificate with 24h validity returns the correct duration

This is as expected - TTL=24h
vault write myMSCA/issue/tpp common_name="$CERT" alt_names="$CERT" ttl="24h" -format=json | jq -r '.data.certificate' |openssl x509 -noout -dates -serial
notBefore=Jun 22 21:12:27 2023 GMT
notAfter=Jun 23 21:22:26 2023 GMT
serial=25000014DCA3F9DCF86579676F0001000014DC

This is not - TTL=12h
vault write myMSCA/issue/tpp common_name="$CERT" alt_names="$CERT" ttl="12h" -format=json | jq -r '.data.certificate' |openssl x509 -noout -dates -serial
notBefore=Jun 22 21:17:02 2023 GMT
notAfter=Jun 23 21:27:02 2023 GMT
serial=25000014DDA353939076EF42FA0001000014DD
ACTUAL RESULTS

The validity period is 24h when 12h was requested

ENVIRONMENT DETAILS

Calling the Venafi API directly to obtain a cert with ~ 2h validity works

POST /vedsdk/certificates/request
...
"CASpecificAttributes": [
{
"Name": "Microsoft CA:Specific End Date",
"Value": "22-JUN-23 23:00:00"
}
]

Return a cert with the following validity

    "ValidFrom": "2023-06-22T21:19:39.0000000Z",
    "ValidTo": "2023-06-22T23:00:00.0000000Z"

COMMENTS/WORKAROUNDS

Please replace this link https://pki.venafi.com/venafi-cloud/ with this link:
https://www.venafi.com/platform/cloud/devops

in README.md

PROBLEM SUMMARY
When specifying allowed_domains parameter on the role, the role should only issue certificates within the provided allowed domains.

This may be one of many constraints not being enforced. I also tested allow_wildcard_certificates = false and was also able to create a wildcard certificate.

According to the documentation: https://developer.hashicorp.com/vault/docs/secrets/venafi
These constraints should be enforced.

STEPS TO REPRODUCE
Run terraform code provisioning the Venafi mount and role. Include the allowed_domains in the role creation. Attempt to issue a certificate using the role, with a domain name that is outside of the allowed domains.

resource "vault_mount" "venafi" {
  path        = "venafi"
  type        = "venafi-pki-backend"
  description = "Venafi PKI Secrets Engine"
}

resource "vault_generic_endpoint" "venafi" {
  path = "${vault_mount.venafi.path}/venafi/tpp"

  data_json = jsonencode({
    url          = <venafi_url>
    access_token = <access_token>
    zone         = <venafi_zone>
  })
}

resource "vault_generic_endpoint" "role" {
  path = "venafi/roles/tpp"

  data_json = jsonencode({
    venafi_secret    = "tpp"
    chain_option     = "last"
    generate_lease   = true
    store_by         = "cn"
    store_pkey       = true
    allowed_domains  = ["example.com"]
    allow_subdomains = true
  })

  depends_on = [vault_generic_endpoint.venafi]
}

resource "vault_pki_secret_backend_cert" "issue" {
  backend = vault_mount.venafi.path
  name    = "tpp"

  common_name = "test.otherdomain.com"

  depends_on = [vault_generic_endpoint.role]
}

EXPECTED RESULTS
We expect that the creation of this certificate would be denied because the common_name domain requested is not within the allowed_domains on the role.

ACTUAL RESULTS
The certificate gets created:

  # vault_generic_endpoint.role will be created
  + resource "vault_generic_endpoint" "role" {
      + data_json            = (sensitive value)
      + disable_delete       = false
      + disable_read         = false
      + id                   = (known after apply)
      + ignore_absent_fields = false
      + path                 = "venafi/roles/tpp"
      + write_data           = (known after apply)
      + write_data_json      = (known after apply)
    }

  # vault_generic_endpoint.venafi will be created
  + resource "vault_generic_endpoint" "venafi" {
      + data_json            = (sensitive value)
      + disable_delete       = false
      + disable_read         = false
      + id                   = (known after apply)
      + ignore_absent_fields = false
      + path                 = "venafi/venafi/tpp"
      + write_data           = (known after apply)
      + write_data_json      = (known after apply)
    }

  # vault_mount.venafi will be created
  + resource "vault_mount" "venafi" {
      + accessor                     = (known after apply)
      + audit_non_hmac_request_keys  = (known after apply)
      + audit_non_hmac_response_keys = (known after apply)
      + default_lease_ttl_seconds    = (known after apply)
      + description                  = "Venafi PKI Secrets Engine"
      + external_entropy_access      = false
      + id                           = (known after apply)
      + max_lease_ttl_seconds        = (known after apply)
      + path                         = "venafi"
      + seal_wrap                    = (known after apply)
      + type                         = "venafi-pki-backend"
    }

  # vault_pki_secret_backend_cert.issue will be created
  + resource "vault_pki_secret_backend_cert" "issue" {
      + auto_renew            = false
      + backend               = "venafi"
      + ca_chain              = (known after apply)
      + certificate           = (known after apply)
      + common_name           = "test.otherdomain.com"
      + expiration            = (known after apply)
      + format                = "pem"
      + id                    = (known after apply)
      + issuing_ca            = (known after apply)
      + min_seconds_remaining = 604800
      + name                  = "tpp"
      + private_key           = (sensitive value)
      + private_key_format    = "der"
      + private_key_type      = (known after apply)
      + renew_pending         = (known after apply)
      + revoke                = false
      + serial_number         = (known after apply)
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

vault_mount.venafi: Creating...
vault_mount.venafi: Creation complete after 1s [id=venafi]
vault_generic_endpoint.venafi: Creating...
vault_generic_endpoint.venafi: Creation complete after 0s [id=venafi/venafi/tpp]
vault_generic_endpoint.role: Creating...
vault_generic_endpoint.role: Creation complete after 0s [id=venafi/roles/tpp]
vault_pki_secret_backend_cert.issue: Creating...
vault_pki_secret_backend_cert.issue: Still creating... [10s elapsed]
vault_pki_secret_backend_cert.issue: Creation complete after 16s [id=venafi/tpp/test.otherdomain.com]

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

ENVIRONMENT DETAILS
HCP Vault v1.14.3
Terraform 1.6.1
Terraform Vault provider v3.21.0

COMMENTS/WORKAROUNDS

1. Does the plugin automatically store the private keys for certificates requested through this plugin?

2. Is there a way to enforce TTL on certificates issued via the plugin? We are seeing an issues where teams are requesting numerous public certificates that are consuming licenses. If no, is there a recommendation on how to address this scenario? I understand we can partially do this with this on the template side within Veanfi.

BUSINESS PROBLEM
When configuring the plugin against TLS Protect Datacenter, trust must be established between the Venafi and Vault servers. Establishing that trust is currently a manual process.

PROPOSED SOLUTION
After updating the go-plugin and Vault SDK, autoMTLS will be enabled in the plugin and manually establishing trust will no longer be necessary.

CURRENT ALTERNATIVES
Use a publicly trusted certificate for the Venafi Operational Certificate (uncommon), or manually establish trust by permanently adding the VOC CA Cert to the Vault trust store or referencing it with the trust_bundle_file parameter.

Wrong cert bundle and key returned in certain conditions
This is a continuation of #90

I was able to reproduce this bug. To be precise I found a scenario where I can trigger the bug 100% of the time.

I little bit of context and a tiny step back. How I discovered the scenario? Since this was not fixed I've put these two lines in our certificate provisioning system that massages the received cert and key from vault via venafi

KEY_MODULUS=openssl rsa -noout -modulus -in key.pem | openssl md5
CERT_MODULES=openssl x509 -noout -modulus -in cert.pem | openssl md5

If these two don't match this causes the cert renewal process to short-circuit. Consider this a safety fuse in this process. So we use the vault-pki-backend-venafi to fetch new certs for some of our services. We use it with combination of a client side tool that can talk to vault and this we use vault as a kind of proxy when we fetch certs from venafi. What I noticed is that on several of our machines, all part of the same cluster and thus having the same CN for the certificate they request from venafi (but it is different one for every machine) this fuse started firing. This meant that the modulus of the certificate and key was different on every machine part of that cluster. Moreover every time I tried to create a new cert, the cert was the same but the key was different. After some back and forth I found that the certificate was in a messed state in Venafi. What I mean is that the processing of the certificate got into some messy state and it was not in OK status. However, this is the interesting bit:

When we requested a cert via vault-pki-backend-venafi, Vault created the private key itself along with the CSR and sents it to venafi, Venafi returns an old (valid) certificate. It is the same certificate whole processing was in a messy state and it was NOT in an OK state. Vault happily returns both - key and cert bundle, although they do not match. Once we are in this situation there is a 100% reproducibility.

I believe we are hitting a cascade of bugs here.

On Venafi TPP side

• First Venafi should not return an old certificate when we request a new one.
• Second if the certicate state is in not OK, it should return any certificate but throw an error.

On the Vault backend plugin side

• Vault should be able to handle the error. Probably it has some generic error handling, but in this particular case the plugin should be able to understand that this is a server-side error and return it as it is to the client. This will help avoid any confusion.
• The plugin should be able to calculate the modulus of the cert vs the key it has generated as a very rudimentary healthcheck and do this when it receives the bundle. So it does not return an invalid cert and key back to the end user.

How to reproduce

• setup vault with the venafi plugin
• create a certificate via vault. Create another one with the same settings/CN. Manually (this is the tricky part) set the status of the certificate as NOT OK. This will involve some knowledge how to simulated failure on the Venafi side. Since the certificate creation involves several stages, we need to fail one of these stages. The important is that we need to have one of more valid certificates (previous versions) and the current one must be in some kind of failed state.
• try to fetch a new certificate via vault+the plugin. The certificate should be an old version but the key will be new (created by vault).

ENVIRONMENT DETAILS
Venafi TTP version - 22.3.0.2139
DB Schema version - 22.3.0.0.0.0
Vault - v1.9.2
vault-pki-backend-venafi - 0.10.1

COMMENTS/WORKAROUNDS
The only workaround is to fix the certificate in Venafi. Either click retry or delete it so new certs can be properly created.

I'm playing around with this engine and stumbled across a few items. If I generate a certificate and key with the Venafi Vault API I get a cert and key back. The key isn't wrapped or protected with a passphrase. Also, the key isn't stored in Venafi.

Questions:
1.) Is there a way to add a passphrase to this key so it's protected at all times?
2.) Can this key somehow be stored in Venafi?

PROBLEM SUMMARY
We upgraded to v0.10.02 to take advantage of the 'service_generated_cert = true'. When using this setting it requires us to include key_password in the JSON with a passphrase. When the plugin retrieves the certificate from Venafi it is storing the private key in the vault encrypted with the passphrase provided. This will create a issue with the pipeline to keep track of these passwords or have a hard set password in the pipeline, which is not desirable.

STEPS TO REPRODUCE
Enable configuration for 'service_generated_cert = true'. Request cert using key_password in JSON. When you retrieve the Certificate from vault the private key will be encrypted with passphrase provided

EXPECTED RESULTS
Since Venafi requires a password to download the private key, the plugin should generate a random password for service_generated_certs and use that to request the certificate from Venafi. Once it retrieves the certificate it should be stored in the Vault with an unencrypted private key. the key_password should not be required for the certificate request, this should only be needed if the pipeline is requesting a certificate w/encrypted private key.

ACTUAL RESULTS
Private key is encrypted in the vault, which requires the password it was originally requested with.

ENVIRONMENT DETAILS
plugin Version v0.10.2, Venafi Version 20.3.x

COMMENTS/WORKAROUNDS
Only work around would be to have a hard coded password in the pipeline or an extreme amount of code to handle tracking of randomly generated passwords, which would be hard to manage.

PROBLEM SUMMARY
We have successfully enabled Venafi engine and are trying to configure a venafi secret to authenticate for enrolling certificates using Venafi by passing access and refresh tokens.

When its the initial run, it was working fine for each update of the tokens.

However, we are facing issues with the refresh token when we re-run it. It throws this below error:
"Code: 400. Errors: * unexpected status code on TPP Authorize. Status: 400 Grant has been revoked, has expired, or the refresh token is invalid"

We haven't made any changes to the refresh or access tokens in between the first and the second run. How is this grant being revoked?

During the process, we observe that:

1. A new token is issued.
2. A token is successfully refreshed.
3. Token refresh failure. (Received an token refresh request from xx.xxx.xxx.xxx but the refresh token supplied could not be found; the lookup failure detail was: %Event.Text1 but the refresh token supplied could not be found; the lookup failure detail was: %Event.Text1$.
   Token 'xxxxxxxxxxx' not found)

STEPS TO REPRODUCE

1. Follow the steps from 1 to 8 in "https://github.com/Venafi/vault-pki-backend-venafi".
2. Repeat step 8 without any change.

EXPECTED RESULTS
Success! Data written to: venafi-pki/venafi/tpp

ACTUAL RESULTS
Initial run => Success! Data written to: pki/humana-external/venafi/vault1.app0001613
Second run =>
Error writing data to venafi-pki/venafi/tpp: Error making API request.

URL: PUT https://xxxx/xx/xxx/venafi-pki/venafi/tpp
Code: 400. Errors:

• unexpected status code on TPP Authorize. Status: 400 Grant has been revoked, has expired, or the refresh token is invalid

BUSINESS PROBLEM
Currently the Venafi PKI Secrets Engine is bootstrapped with both an Access Token and a Refresh Token, assuming security best practices are followed and the validity of the Access Token is limited to a few days. The Secrets Engine uses the Access Token until it expires and then it responds by using the Refresh Token to obtain a new pair of tokens and this process repeats until TPP no longer allows the tokens to be refreshed.

PROPOSED SOLUTION
Require only a Refresh Token to bootstrap the Venafi PKI Secrets Engine which would have a few advantages. One is that the provided Refresh Token would need to be used immediately by the Secrets Engine to obtain a new pair of tokens and that ensures the original Access Token cannot be used by any other application or person because the act of refreshing it immediately invalidates it.

Another advantage is that the Secrets Engine will have immediate access to the expiration date of the Access Token so it would be able to store and monitor that date to proactively refresh the token within day of its expiration, for example. A general objective is to minimize the amount of data a user is required to provide to get things working which is why the original expiration date wasn't considered to be specified along with the Access Token and Refresh Token in the current implementation.

Assuming this enhancement is accepted, the access_token parameter would then only apply for the case of a long-validity and/or non-refreshable grant.

CURRENT ALTERNATIVES
None other than to provide both the Access Token and Refresh Token as described in the BUSINESS PROBLEM section.

When we do this we should also revise our reactive token refresh to be triggered by any HTTP 401 response that includes "error" values of "expired_token" (TPP 20.1-20.3) or "invalid_token" (TPP 20.4). If we're doing proactive token refresh based on a stored access token expiration date, I think we'd only be doing reactive refresh if the secrets engine isn't used within whatever we define as the "refresh window" (1 day?).

PROBLEM SUMMARY
When the HashiCorp Vault plugin is starting up it can take up to 60 seconds to start. This is an issue as the Vault becomes locked during that time and no other vault requests can be handled.

STEPS TO REPRODUCE
I do not have specific steps at this time as I am opening this for a customer. I will get this information and relay when I have it or the customer can update if possible.

EXPECTED RESULTS
That the plugin will not prevent other vault requests while starting.

ACTUAL RESULTS
While plugin is starting up the HashCorp Vault is locked and not able to handle any further requests until the plugin startup completes ~60 seconds.

ENVIRONMENT DETAILS
Venafi TPP: 18.2.0
Venafi Vault Plugin: v0.6.4
HashiCorp Vault: unknown at this time

High Level architecture:
Vault HA cluster with a replicated HA cluster in a different location.

COMMENTS/WORKAROUNDS
no known workaround at this time.

BUSINESS PROBLEM
As of Vault version 1.13.x, the default pki now supports some health check endpoints. https://developer.hashicorp.com/vault/docs/commands/pki/health-check

This would provide a framework to enhance the backend plugin to support different health checks that may be more relevant to Venafi. i.e: Does the vault pki know if tpp or venafi cloud is available

PROPOSED SOLUTION
Add support for a health check api to allow Vault users or admins to check important information related to the Vault PKI. This may be related to connectivity, health of the service, etc.

CURRENT ALTERNATIVES
N/A

Currently using Vault v1.2.3 and tested on OSX and Linux. Configured like this:

cat /etc/vault.d/vault.hcl
ui = true
disable_mlock = true
plugin_directory = "/home/ubuntu/vault/plugins"
api_addr = "https://vault.venafidevopslab.com:8200"

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_enable = 1
  tls_cert_file = "/home/ubuntu/vault/fullchain.pem"
  tls_key_file  = "/home/ubuntu/vault/privkey.pem"
}

backend "file" {
  path = "/home/ubuntu/vault/backend"
}

PROBLEM SUMMARY
After successfully requesting certs from the vault-pki-backend-venafi using the following command:

vault write pki-venafi/issue/tpp common_name="madonna.venafi.example"

The vault list pki-venafi/certs command seems to no longer work and returns an empty list.

STEPS TO REPRODUCE

Successfully registered and configured the vault-pki-backend-venafi backend.

$vault write sys/plugins/catalog/venafi-pki-backend sha_256="${SHA256}" command="venafi-pki-backend"

Success! Data written to: sys/plugins/catalog/venafi-pki-backend

$vault write sys/plugins/catalog/venafi-pki-backend sha_256="${SHA256}" command="venafi-pki-backend"
Success! Data written to: sys/plugins/catalog/venafi-pki-backend

$vault secrets enable -path=pki-venafi -description="Venafi backend for certificate enrollment" -plugin-name=venafi-pki-backend plugin
Success! Enabled the venafi-pki-backend secrets engine at: pki-venafi/

Then creates a new role.

vault write pki-venafi/roles/tpp \
        generate_lease=true \
        max_ttl=2160h \
        tpp_url="https://EC2AMAZ-Q89P18M.acme.com" \
        zone="DevOps\\Vault\\Requests" \
        trust_bundle_file="trust-bundle.pem" \
        tpp_user="apiuser" \
        tpp_password="*********"
        store_by_cn="true" \
        store_by_serial="true" \
        store_pkey="true"

Requested a new cert:

$vault write pki-venafi/issue/tpp common_name="madonna.venafi.example"

WARNING! The following warnings were returned from Vault:

  * Read access to this endpoint should be controlled via ACLs as it will
  return the connection private key as it is.

  * TTL of "17519h49m57.029465s" exceeded the effective max_ttl of "768h0m0s";
  TTL value is capped accordingly

Key                  Value
---                  -----
lease_id             pki-venafi/issue/tpp/MHvdKw20v2JcoUh3VXZBvE3X
lease_duration       768h
lease_renewable      false
certificate          -----BEGIN CERTIFICATE-----
MIIFXzCCBEegAwIBAgITfgAAABNPmXq8Txpk1QAAAAAAEzANBgkqhkiG9w0BAQsF
ADBNMRMwEQYKCZImiZPyLGQBGRYDY29tMRQwEgYKCZImiZPyLGQBGRYEYWNtZTEg
MB4GA1UEAxMXYWNtZS1FQzJBTUFaLVE4OVAxOE0tQ0EwHhcNMTkxMDI5MTU1MTA0
WhcNMjExMDI4MTU1MTA0WjAuMQswCQYDVQQGEwJHQjEfMB0GA1UEAxMWbWFkb25u
YS52ZW5hZmkuZXhhbXBsZTCCASIwDQYJKoZIhvcNAQEBBQADggEP...........

Then tried to list certs:

$vault list pki-venafi/certs/                                             
No value found at pki-venafi/certs/