Git Product home page Git Product logo

hashi-helper's Issues

[parser] rework parser so file order for resources won't matter

Today partial resources (e.g. mount -> role without mount config) must be in files that are lexically sorted after the full mount config (hense the best practice of prefixing the mount config file with _)

Rewriting the parser to be two pass, one to extract all resources and one to merge them together so order of files won't matter

Missing vendor file?

I'm not very familier with govendor, on osx I get the following:

brannon:hashi-helper seph$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

brannon:hashi-helper seph$ git log | head -1
commit 969ca9ce0ef1560b279b83130d1e570b35aa18ad

brannon:hashi-helper seph$ make build
go get github.com/kardianos/govendor
govendor sync
Error: Vendor file at "vendor" not found.
govendor sync
	Ensures the contents of the vendor folder matches the vendor file.
	Options:
		-n           dry run, print out action only
		-insecure    allow downloading over insecure connection
		-v           verbose output
make: *** [install] Error 2

It sounds like I should have a vendor/vendor.json?

Any chance of tagging v3.0.1?

This is off the back of #34 and #35, a tag containing these fixes would really save me having to mess about with building from a fork or pinning to a git sha for our internal package building CI :)

Cannot get latest version: module contains a go.mod file, so module path should be github.com/seatgeek/hashi-helper/v3

Background

The github.com/seatgeek/hashi-helper uses Go modules and the current release version is v3. And it’s module path is "github.com/seatgeek/hashi-helper", instead of "github.com/seatgeek/hashi-helper/v3". It must comply with the specification of "Releasing Modules for v2 or higher" available in the Modules documentation. Quoting the specification:

A package that has opted in to modules must include the major version in the import path to import any v2+ modules
To preserve import compatibility, the go command requires that modules with major version v2 or later use a module path with that major version as the final element. For example, version v2.0.0 of example.com/m must instead use module path example.com/m/v2.
https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher

Steps to Reproduce

GO111MODULE=on, run go get targeting any version >= v3.2.1 of the seatgeek/hashi-helper:

$ go get github.com/seatgeek/[email protected]
go: finding github.com/seatgeek/hashi-helper v3.3.3
go: finding github.com/seatgeek/hashi-helper v3.3.3
go get github.com/seatgeek/[email protected]: github.com/seatgeek/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v3

run go get github.com/seatgeek/hashi-helper, the version will stuck in v3.2.0:

$go get github.com/seatgeek/hashi-helper
go: downloading github.com/seatgeek/hashi-helper v3.2.0+incompatible
go: github.com/seatgeek/hashi-helper upgrade => v3.2.0+incompatible

SO anyone using Go modules will not be able to easily use any newer version of seatgeek/hashi-helper.

Solution

1. Kill the go.mod files, rolling back to GOPATH.

This would push them back to not being managed by Go modules (instead of incorrectly using Go modules).
Ensure compatibility for downstream module-aware projects and module-unaware projects projects

2. Fix module path to strictly follow SIV rules.

Patch the go.mod file to declare the module path as github.com/seatgeek/hashi-helper/v3 as per the specs. And adjust all internal imports.
The downstream projects might be negatively affected in their building if they are module-unaware (Go versions older than 1.9.7 and 1.10.3; Or use third-party dependency management tools, such as: Dep, glide,govendor…).

If you don't want to break the above repos. This method can provides better backwards-compatibility.
Release a v2 or higher module through the major subdirectory strategy: Create a new v3 subdirectory (github.com/seatgeek/hashi-helper/v3) and place a new go.mod file in that subdirectory. The module path must end with /v3. Copy or move the code into the v3 subdirectory. Update import statements within the module to also use /v3 (import "github.com/seatgeek/hashi-helper/v3/…"). Tag the release with v3.x.y.

3. Suggest your downstream module users use hash instead of a version tag.

If the standard rule of go modules conflicts with your development mode. Or not intended to be used as a library and does not make any guarantees about the API. So you can’t comply with the specification of "Releasing Modules for v2 or higher" available in the Modules documentation.
Regardless, since it's against one of the design choices of Go, it'll be a bit of a hack. Instead of go get github.com/seatgeek/hashi-helper@version-tag, module users need to use this following way to get the seatgeek/hashi-helper:
(1) Search for the tag you want (in browser)
(2) Get the commit hash for the tag you want
(3) Run go get github.com/seatgeek/hashi-helper@commit-hash
(4) Edit the go.mod file to put a comment about which version you actually used
This will make it difficult for module users to get and upgrade seatgeek/hashi-helper.

Summary

You can make a choice to fix DM issues by balancing your own development schedules/mode against the affects on the downstream projects.

For this issue, Solution 2 can maximize your benefits and with minimal impacts to your downstream projects the ecosystem.

References

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:


If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:


If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Arrays resulting from lookups have unquoted members

I'm creating a PKI role that looks like:

environment "all" {
  secret "/user-ca/roles/user" {
    "allowed_domains" = [[ lookup "all_engineers" ]]
    ...
  }
}

where all_engineers is populated by passing in a variables file of the format:

{
  "all_engineers": [
    "joel",
    "jolene",
    ...
  ]
}

when I attempt to apply this configuration, I get the error Could not format HCL file user_ca.hcl: At 3:26: unexpected token while parsing list: IDENT

If I then (erroneously, but for debug purposes) put the templated lookup inside quotes, it's populated with a string literal of the result of the lookup (the array of usernames) whereby none of the members are quoted (which is consistent with the IDENT problem). This is counter to what I would expect to be the default behaviour to render as a list of quoted strings.

As a workaround I've had to hack together:

    "allowed_domains" = [ [[ range $i, $name := lookup "all_engineers" ]][[if $i]] ,[[end]]"[[$name]]"[[end]] ]

which templates out the list as I want it formatted - this isn't ideal!

I'm happy to try to fix this myself but I'd appreciate guidance on where to look and whether you'd accept this patch in the first place.

Missing VAULT_ADDR on vault-unseal-keybase without --consul-service-name causes SIGSEGV

To repro:

  • set CONSUL_HTTP_ADDR
  • set VAULT_UNSEAL_KEY
  • run hashi-helper vault-unseal-keybase
$ hashi-helper vault-unseal-keybase
INFO[0000] Decoding VAULT_UNSEAL_KEY
INFO[0000] Decrypting VAULT_UNSEAL_KEY
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x10ff5a6]

goroutine 1 [running]:
github.com/seatgeek/hashi-helper/vendor/github.com/Sirupsen/logrus.(*Entry).Info(0x0, 0xc420108f60, 0x1, 0x1)
    /Users/jose/.go/1.9.1/src/github.com/seatgeek/hashi-helper/vendor/github.com/Sirupsen/logrus/entry.go:139 +0x26
github.com/seatgeek/hashi-helper/command/vault.sendUnseal(0xc420186000, 0x42, 0x0, 0x0, 0x0)
    /Users/jose/.go/1.9.1/src/github.com/seatgeek/hashi-helper/command/vault/unseal_keybase.go:89 +0xa3
github.com/seatgeek/hashi-helper/command/vault.UnsealKeybase(0xc4200b6dc0, 0xc4201092a8, 0x137aec1)
    /Users/jose/.go/1.9.1/src/github.com/seatgeek/hashi-helper/command/vault/unseal_keybase.go:47 +0xc29
main.main.func4(0xc4200b6dc0, 0x0, 0xc4200b6dc0)
    /Users/jose/.go/1.9.1/src/github.com/seatgeek/hashi-helper/main.go:87 +0x2b
github.com/seatgeek/hashi-helper/vendor/gopkg.in/urfave/cli%2ev1.HandleAction(0x141fe60, 0x14cd150, 0xc4200b6dc0, 0xc42005c500, 0x0)
    /Users/jose/.go/1.9.1/src/github.com/seatgeek/hashi-helper/vendor/gopkg.in/urfave/cli.v1/app.go:485 +0xd2
github.com/seatgeek/hashi-helper/vendor/gopkg.in/urfave/cli%2ev1.Command.Run(0x14b7e9f, 0x14, 0x0, 0x0, 0x0, 0x0, 0x0, 0x14c45c8, 0x31, 0x0, ...)
    /Users/jose/.go/1.9.1/src/github.com/seatgeek/hashi-helper/vendor/gopkg.in/urfave/cli.v1/command.go:193 +0xa99
github.com/seatgeek/hashi-helper/vendor/gopkg.in/urfave/cli%2ev1.(*App).Run(0xc42007f040, 0xc42000a080, 0x2, 0x2, 0x0, 0x0)
    /Users/jose/.go/1.9.1/src/github.com/seatgeek/hashi-helper/vendor/gopkg.in/urfave/cli.v1/app.go:250 +0x758
main.main()
    /Users/jose/.go/1.9.1/src/github.com/seatgeek/hashi-helper/main.go:280 +0x1653

SIGSEGV when running vault-list-secrets --remote --detailed

Hi, when tries it, a Sevg happened as below, maybe some fix need to make that be more reliable.
thanks,

$  ./hashi-helper  vault-list-secrets --remote --detailed
INFO[0000] value = http://localhost:8181                 app=api-admin env=production secret=API_URL
INFO[0000]
panic: runtime error: invalid memory address or nil pointer dereference
**[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7ca144]**

goroutine 1 [running]:
github.com/seatgeek/hashi-helper/command/vault.secretListLocal(0xc4200c4b40, 0x8b3601, 0x6)
        **/Users/go_src/src/github.com/seatgeek/hashi-helper/command/vault/secrets_list.go:29 +0x184**
github.com/seatgeek/hashi-helper/command/vault.SecretsList(0xc4200c4b40, 0xc420049320, 0x778241)
        /Users/go_src/src/github.com/seatgeek/hashi-helper/command/vault/secrets_list.go:17 +0x87
main.main.func5(0xc4200c4b40, 0x100, 0xc4200c4b40)
        /Users/go_src/src/github.com/seatgeek/hashi-helper/main.go:101 +0x2b
gopkg.in/urfave/cli%2ev1.HandleAction(0x823ea0, 0x8cb2a8, 0xc4200c4b40, 0xc42005e700, 0x0)
        /Users/go_src/src/gopkg.in/urfave/cli.v1/app.go:485 +0xd4
gopkg.in/urfave/cli%2ev1.Command.Run(0x8b8623, 0x12, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8c2d0a, 0x27, 0x0, ...)
        /Users/go_src/src/gopkg.in/urfave/cli.v1/command.go:193 +0xb72
gopkg.in/urfave/cli%2ev1.(*App).Run(0xc4200d89c0, 0xc4200101c0, 0x4, 0x4, 0x0, 0x0)
        /Users/go_src/src/gopkg.in/urfave/cli.v1/app.go:250 +0x7d0
main.main()
        /Users/go_src/src/github.com/seatgeek/hashi-helper/main.go:255 +0x12d5

improve documentation around 'encrypted file' in workflow examples

As a potential user of hashi-helper, I would like for the documentation around encrypted file workflow to be more understandable.

In several places in the README.md, files are referred to as 'encrypted files', but there is little context given for how those files are encrypted - what is the tooling being used there? Is it that keybase encrypt is intended to be used, or something else? Or just that the vault-secret values stored in the files are somehow encrypted?

E.g., "The following can be stored in an encrypted file at production/apps/api-admin.hcl".

The code block following is in cleartext, and clearly the whole thing is meant to be crypto'ed, but no where else in the docs does it say how we should do that, or what the workflow is to make our locally-encrypted stuff usable with the vault-push-all or vault-push-secrets commands.

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:


If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Pull all secrets?

I would like to pull all secrets from a remote but I notice that this feature hasn't been implemented yet.

Any ETA on this feature?

[bug] Build failed: cannot use vc Client in argument to h.Auth

go/bin/go1.12.8 get github.com/seatgeek/hashi-helper
# github.com/seatgeek/hashi-helper/command/profile
go/src/github.com/seatgeek/hashi-helper/command/profile/use_profile.go:277:24: cannot use vc (type *"github.com/hashicorp/vault/api".Client) as type *"github.com/hashicorp/vault/vendor/github.com/hashicorp/vault/api".Client in argument to h.Auth

Unable compile: type *http.Client has no field or method CloseIdleConnections

$ go get github.com/seatgeek/hashi-helper# github.com/hashicorp/vault/vendor/github.com/hashicorp/go-retryablehttp
go/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/go-retryablehttp/client.go:447:17: c.HTTPClient.CloseIdleConnections undefined (type *http.Client has no field or method CloseIdleConnections)
go/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/go-retryablehttp/client.go:507:16: c.HTTPClient.CloseIdleConnections undefined (type *http.Client has no field or method CloseIdleConnections)
go/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/go-retryablehttp/client.go:538:16: c.HTTPClient.CloseIdleConnections undefined (type *http.Client has no field or method CloseIdleConnections)
go/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/go-retryablehttp/client.go:545:15: c.HTTPClient.CloseIdleConnections undefined (type *http.Client has no field or method CloseIdleConnections)
go/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/go-retryablehttp/client.go:554:14: c.HTTPClient.CloseIdleConnections undefined (type *http.Client has no field or method CloseIdleConnections)

I fix it by https://godoc.org/golang.org/dl/go1.12.8, so go version will be 1.22.8

is this possible to use @ in configuration file?

for example, like vault CLI, use the content of a file as the input?

environment "production" {

  # the name/path the mount will be mounted at,
  # must be unique for the environment
  auth "kubernetes" {

    # the mount backend type to use (see Vault docs)
    type = "kubernetes"

    # mount configuration, see Vault docs for details
    config ""{
        kubernetes_host = "..."
        kubernetes_ca_cert = "@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
    }

  }
}

seems it will be treated as a string, instead of read the file content.

thanks,

Configuring LDAP authentication engine & TTLs

Hi! hashi-helper is super helpful. I'm looking to move a lot of my vault configuration over that way and am running into a few things.

Configure LDAP Auth Engine

One thing that i'm not sure how to do as hashi-helper is right now is configure the LDAP authentication engine. I had tried something like:

environment "dev" {
    auth "ldap" {
        type = "ldap"
        config "" { # should write to /auth/ldap/config i imagine?
            url      = "ldap://blah"
            starttls = true
            #more ldap config bits here
        }
        duo "access" { # might? write to /auth/ldap/duo/access?
            host = "..."
            ikey = "..."
            skey = "..."
        }
        mfa_config "" { # /auth/ldap/mfa_config?
            type = "duo"
        }
        groups "vaultadmin" { # /auth/ldap/groups/vaultadmin?
            policies = "default,user-vaultadmin"
        }
    }
}

Is this something I'm going to have to wire up in type_vault_auth.go and parse_vault_auth.go? Or is there a generic "write key/value to place" resource thing i could use?

Auth engine TTLs

Let's say I want to set default lease/max lease TTLs for a particular auth engine...

environment "dev" {
    auth "aws-ec2" {
        type = "aws-ec2"
        role "testrole" {
            bound_account_id          = "1234"
            policies                  = "default,pki"
            disallow_reauthentication = true
            role_tag                  = "vault_role"
        }
        defaultleasettl = "24h"
        maxleasettl     = "8w"
    }
}

I get this when i run hashi-helper:

* invalid key 'defaultleasettl' in line 10:9
* invalid key 'maxleasettl' in line 11:9

I see the struct in type_vault_auth.go includes those TTL values, but i dont think it gets mapped into that in parse_vault_auth.go. Should I just tune this with the mount type instead?

    mount "auth/aws-ec2" {
        type              = "aws-ec2"
        default_lease_ttl = "1h"
        max_lease_ttl     = "1y"
    }

[bug] No default to all when no environment specified.

The documentation reads:
--environment / ENVIRONMENT: The environment to process for (optional; default: all)

But if the --environment is not explicitly defined it get's skipped.

Example config file to setup nomad vault roles/auth/policy
(this might not be a fully real working example)

environment "dev" {

    auth "nomad-cluster-aws" {
      type = "aws"

      role "nomad-cluster" { # :role
        policies                       = "nomad-server"
        max_ttl                        = "6h"
        period                         = "1h"
        allow_instance_migration       = false
        bound_iam_role_arn             = "arn:aws:iam::99999999999:role/nomad"
        }
    }


    auth "nomad-server-token" {
      type = "token"

      role "nomad-server" { # :role
        disallowed_policies = "nomad-server"
        explicit_max_ttl    = 0
        name                = "nomad-cluster"
        orphan              = false
        period              = 3600
        renewable           = true
      }
    }


    policy "nomad-server" {
          # Allow creating tokens under "nomad-cluster" token role. The token role name
          # should be updated if "nomad-cluster" is not used.
          path "auth/token/create/nomad-cluster" {
            capabilities = ["update"]
          }

          # Allow looking up "nomad-cluster" token role. The token role name should be
          # updated if "nomad-cluster" is not used.
          path "auth/token/roles/nomad-cluster" {
            capabilities = ["read"]
          }

          # Allow looking up incoming tokens to validate they have permissions to access
          # the tokens they are requesting. This is only required if
          # `allow_unauthenticated` is set to false.
          path "auth/token/lookup" {
            capabilities = ["update"]
          }

          # Allow revoking tokens that should no longer exist. This allows revoking
          # tokens for dead tasks.
          path "auth/token/revoke-accessor" {
            capabilities = ["update"]
          }

          # Allow checking the capabilities of our own token. This is used to validate the
          # token upon startup.
          path "/sys/capabilities-self" {
            capabilities = ["update"]
          }

          # Allow our own token to be renewed.
          path "auth/token/renew-self" {
            capabilities = ["update"]
          }
    }

}

No --environment specified.

$ hashi-helper --log-level=debug --config-file=dev.hcl  vault-push-all
DEBU[0000] Parsing file dev.hcl
DEBU[0000]   Found environment dev
DEBU[0000]   Scanning for applications
DEBU[0000]   Scanning for vault auth backends
DEBU[0000]   Scanning for vault secrets
DEBU[0000]   Scanning for vault policies
DEBU[0000]   Scanning for vault mounts
DEBU[0000]   Scanning for consul services
DEBU[0000]   Scanning for consul KV
$ 

With environment specified.

$ hashi-helper --log-level=debug --config-file=dev.hcl --environment=dev vault-push-all
DEBU[0000] Parsing file dev.hcl
DEBU[0000]   Found environment dev
DEBU[0000]   Scanning for applications
DEBU[0000]   Scanning for vault auth backends
DEBU[0000]   Scanning for vault secrets
DEBU[0000]   Scanning for vault policies
DEBU[0000]   Scanning for vault mounts
DEBU[0000]   Scanning for consul services
DEBU[0000]   Scanning for consul KV
INFO[0000] Auth backend nomad-cluster-aws already exist
INFO[0000]   Writing auth backend role: auth/nomad-cluster-aws/role/nomad-cluster
[...]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.