Currently, each tool is installed using go install command which will name the executable after the folder it is located in. For example, if main.go is in .../cli/main.go and is installed, the name of the binary will be cli. Therefore, there is a possibility for a clash if more than one tool is contained in a folder of the same name.

Solution

Allow for (optional) custom naming of each tool set in the tools.json

Say I have a tool shovel and a tool rake. Both of these tools depend on different versions of package metal_head. In the current implementation this would fail to build from a clean install simultaneously.

Proposal

This is due to retool being different from the traditional vendor problem: rather than building from a single package tree, retool can build multiple binaries from a collection of trees. In its current model these trees can share sub trees or leaves which leads to the problem described above.

I propose changing how retool collects and manipulates the GOPATH to be a one-to-one relationship between the tool and directory. To maintain ease of use there should continue to be a single GOBIN directory.

As a side benefit this proposal would also allow retool to not have to download all tools to sync with the manifest, as it could ask git for the current checked out sha.

Implementation

Change tools dir structure, and cache dir, to be

_tools
├── bin
│   ├── shovel
│   └── rake
├── manifest.json
├── shovel
│   ├── pkg
│   │  └── linux_amd64
│   └── src
│      └── github.com
└── rake
    ├── pkg
    │  └── linux_amd64
    └── src
       └── github.com

Change the install command to set GOBIN appropriately, and keep track of each tools dir inside the tool struct.

Currently, each tool is installed using go install command which is pretty limiting. Certain tools require arguments to be passed in when building them. In order to accommodate that, the go build command needs to be used.

Solution

Allow for passing extra parameters when building each tool which can be defined in the tools.json.

For example:

{
      "Repository": "github.com/golang-migrate/migrate/cli",
      "Commit": "4937cd088f7c7193ee59b319c1c2caa7482aae8e",
      "ExecutableName": "migrate",
      "PreParams": [
          "-tags", 
          "'postgres'"
      ]
    }

PreParams are the arguments passed in just after go build command and before the path and other arguments, which in this example will result in:

go build -tags 'postgres' -o <toolDirPath>/bin/migrate github.com/golang-migrate/migrate/cli

We're starting to use this at work, and it looks awesome. However, the tools.json file only tracks commit hashes, which are not human friendly. There's no easy way to look and see what version of a tool we're requiring.

My suggestion would be to just add another field to the json, describing the requested tag/branch.

It would also be useful to have a field with the timestamp of the given commit hash, so it's more obvious if it's way out of date.

These two fields would only really be informational, the tool wouldn't need to read them at all, so I think it would be pretty easy to add (and thus would also likely be 100% backwards compatible).

I'd be happy to work on a PR if this is something that seems like it would be ok.

If I have 2 go exes in the same vendored package, I have to refer to the folders containing the mains to get retool to vendor them. However when I add one tool then add the second (2 command line calls to retool) the second one will fail.

In order to integrate better with dep, retool could look for packages specified in the required section and import those into retool's configuration.

When using retool build with GOBIN env variable set the produced binaries will not land where retool do expects them to be. Instead, they will be installed to GOBIN directory which takes precedence over $GOPATH/bin.

To reproduce run GOBIN=/tmp/not-here retool build. The _tools/bin directory will not be populated like it is expected.

Proposed solution is to override both GOPATH and GOBIN in the install function. I'm happy to provide a PR but at a later date.

It'd be useful to hook into the script generated by godownloader for repositories that publish their dependencies via https://github.com/goreleaser/goreleaser and https://github.com/goreleaser/godownloader.

What do you reckon?

I can't easily tell what version of retool is installed. WOuld love a --version flag to show what version of retool I'm running.

As part of bootstrapping use of retool in our process, it would be great if there were binary releases of retool for popular platforms that we can just fetch with a curl against a versioned URL rather than needing to go get.

retool add github.com/golang/dep origin/master
retool do dep
retool: fatal err: failed on 'dep': exec: "dep": executable file not found in $PATH

Why is this happening?

I could install a dep release to get it on my $PATH, however isn't that what retool is trying to avoid?

Said another way, shouldn't retool add github.com/golang/dep origin/master install the binary local to my project such that retool do dep can find and execute it?

because _tools potentially contains Go code that isn't stored in vendor, tooling, especially those using triple dot syntax (...) detects vendored tools as sub-packages.

When I perform audits on code and use tree, even blacklisting vendor is not enough because retool replicates the source tree and my command explodes

Currently there's two sets of help, retool --help, which prints the flags, and retool help that prints help about the commands. At the base level, they need to be combined, so you can get a full picture of how to use the tool.

$ retool --help
Usage of retool:
  -base-dir string
        Path of project root.  If not specified and the working directory is within a git repository, the root of the repository is used.  If the working directory is not within a git repository, the working directory is used.
  -f string
        Use a fork of the repository rather than the default upstream
  -tool-dir string
        Path where tools are stored.  The default value is the subdirectory of -base-dir named '_tools'.
  -verbose
        Enable more detailed output that may be helpful for troubleshooting.

Currently retool doesn't work on windows. This is due to sync.go. The sync function uses exec.Command to delete a directory by using rm which is a linux specific command.

https://github.com/twitchtv/retool/blob/master/sync.go#L11

I have a fix+test, but am waiting on approval from my supervisor to open up the PR.

even when a tool is correctly up to date in _tools and you sync, retool goes and redownloads that tool. This is bad for several reasons:

• really sucks on slow network connections
• sucks when you have 1/2 of your packages in one private repo behind one corporate firewall and another 1/2 of your packages behind a different firewall
• slower

Hi,

I tried to use your tool and it works without any problems with some tools I use (govendor and goveralls) except gometalinter.

As you can see, they use a _linters directory containing the different linters they support. The problem is that directory is cleaned by the cleanup function called on tool sync, making gometalinter throwing gometalinter: error: could not find vendored linters in GOPATH.

The only solutions I can see are:

• to disable the vendored linters feature with --vendored-linters=false, but I didn't like because it implies gometalinter to fetch linters without controlling the version
• add a new flag like retool add --cleaning=false github.com/alecthomas/gometalinter origin/master which generate something like this

{
  "Tools": [
    {
      "Repository": "github.com/alecthomas/gometalinter",
      "Commit": "bae2f1293d092fd8167939d5108d1b025eaef9de",
      "Clean": false
    }
  ]
}

• remove the s.cleanup() line because who care to have useless files in the _tools directory ? It's not supposed to be in the VCS

Is there another solutions ? Can you do something about this ?
Thank you.

Using the default GOPATH (not explicitly setting it) results in an error saying that git is not in PATH (it is):

> retool do
retool: syncing
retool: downloading github.com/vektra/mockery/cmd/mockery
retool: fatal err: execution error on "go get -d github.com/vektra/mockery/cmd/mockery": go: missing Git command. See https://golang.org/s/gogetcmd
package github.com/vektra/mockery/cmd/mockery: exec: "git": executable file not found in $PATH
: failed to 'go get' tool: exit status 1

It would be nice if retool worked with the default GOPATH or at least warned if GOPATH isn't set.

> retool version
retool v1.0%

retool add github.com/golang/protobuf/protoc-gen-go master

retool: downloading github.com/golang/protobuf/protoc-gen-go
retool: setting version for github.com/golang/protobuf/protoc-gen-go
retool: fatal err: execution error on "git checkout ": chdir mydirectorystructure/_tools/src/github.com/golang/protobuf/protoc-gen-go: no such file or directory

but ignoring or disabling go modules temporarily seems to fix it
GO111MODULE=off retool add github.com/golang/protobuf/protoc-gen-go master

retool: downloading github.com/golang/protobuf/protoc-gen-go
retool: setting version for github.com/golang/protobuf/protoc-gen-go
retool: parsing revision "master"
retool: parsed as "d23c5127dc24889085f8ccea5c9d560a57a879d8"
retool: installing github.com/golang/protobuf/protoc-gen-go

also, creating a random directory and running retool works fine, only happens when a go.mod/sum file is present

Steps to reproduce:

mkdir test_retool
cd test_retool
retool add github.com/golang/protobuf/protoc-gen-go master

= error

Is it possible to have a -config-file option for retool?

My issue is that I would like to follow the standard of having a tools folder for tools generated by the project rather than dev tools. I can configure the _tools directory name, but it doesn't seem like I can configure the config file name, which is awkward when there is this different concept of project-generated "tools".

It seems that retool currently does not analyze the dependency on .h files, thus making necessary files missing. An example of compiling with .h files is https://github.com/mattn/go-sqlite3

And maybe there are other cases the cleanup function should take care of.

I see this project is not maintained anymore, is there an alternative tool that does the same job? Maybe even one that goes a step further and vendors tools written in any language, not just Go.

When I follow the instructions in the README, I can't seem to get retool to add anything successfully. I wonder if retool doesn't work with Go modules and/or Go 1.11.2+?

In the below, notice that I'm running all this outside of $GOPATH.

MacBook-Pro:myapp Jonathan$ go get github.com/twitchtv/retool
go: finding github.com/twitchtv/retool v1.3.7
go: downloading github.com/twitchtv/retool v1.3.7
go: finding github.com/Masterminds/semver v1.4.2
go: downloading github.com/Masterminds/semver v1.4.2
MacBook-Pro:myapp Jonathan$ retool add github.com/jteeuwen/go-bindata/go-bindata origin/master
retool: downloading github.com/jteeuwen/go-bindata/go-bindata
retool: setting version for github.com/jteeuwen/go-bindata/go-bindata
retool: fatal err: execution error on "git checkout ": chdir /private/tmp/tmpcode/myapp/_tools/src/github.com/jteeuwen/go-bindata/go-bindata: no such file or directory
MacBook-Pro:myapp Jonathan$ retool version
retool 1.3.7
MacBook-Pro:myapp Jonathan$ go version
go version go1.11.2 darwin/amd64
MacBook-Pro:myapp Jonathan$ pwd
/tmp/tmpcode/myapp
MacBook-Pro:myapp Jonathan$ echo $GOPATH
/Users/Jonathan/go

As you can see the directory it's looking for doesn't actually exist:

MacBook-Pro:myapp Jonathan$ ls -la /private/tmp/tmpcode/myapp/_tools/
total 8
drwxr-xr-x   4 Jonathan  wheel  128  5 Dec 10:18 .
drwxr-xr-x  18 Jonathan  wheel  576  5 Dec 10:18 ..
-rw-r--r--   1 Jonathan  wheel   27  5 Dec 10:20 .gitignore
drwxr-xr-x   3 Jonathan  wheel   96  5 Dec 10:18 pkg
MacBook-Pro:myapp Jonathan$ ls -la /tmp/tmpcode/myapp/_tools/
total 8
drwxr-xr-x   4 Jonathan  wheel  128  5 Dec 10:18 .
drwxr-xr-x  18 Jonathan  wheel  576  5 Dec 10:18 ..
-rw-r--r--   1 Jonathan  wheel   27  5 Dec 10:20 .gitignore
drwxr-xr-x   3 Jonathan  wheel   96  5 Dec 10:18 pkg

Does this seem like a bug or is it my issue?

eg.

https://github.com/volatiletech/sqlboiler

This command tool needs a set of template files to generate codes.

I know, things get much complicated if runtime dependencies are considered, so I am wondering is it possible to provide a way to indicate which kind of files/dir should not be dropped?

go version go1.16beta1 darwin/arm64

mkdir t
cd t
retool add github.com/golang/protobuf/protoc-gen-go master

retool: downloading github.com/golang/protobuf/protoc-gen-go
retool: setting version for github.com/golang/protobuf/protoc-gen-go
retool: fatal err: execution error on "git checkout ": chdir /Users/az/t/_tools/src/github.com/golang/protobuf/protoc-gen->go: no such file or directory

Recently hit a bug where if the child process doesn't exit cleanly retool will leave it orphaned leaving you to force close the child process yourself.

This was hit with Terraform and is definitely an edge case when terraform doesn't cleanly exit (you need to send shutdown signal multiple times).