Git Product home page Git Product logo

go-cmdtest's Introduction

Build Status godoc Go Report Card

Testing your CLI

The cmdtest package simplifies testing of command-line interfaces. It provides a simple, cross-platform, shell-like language to express command execution. It can compare actual output with the expected output, and can also update a file with new "golden" output that is deemed correct.

Test files

Start using cmdtest by writing a test file with the extension .ct. The test file will consist of commands prefixed by $ and expected output following each command. Lines starting with # are comments. Example:

# Testing for my-cli.

# The "help" command.
$ my-cli help
my-cli is a CLI, and this is its help.

# Verify that an invalid command fails and prints a useful error.
$ my-cli invalidcmd --> FAIL
Error: unknown command "invalidcmd".

You can leave the expected output out and let cmdtest fill it in for you using update mode (see below).

More details on test file format:

  • Before the first line starting with a $, empty lines and lines beginning with "#" are ignored.
  • A sequence of consecutive lines starting with $ begin a test case. These lines are commands to execute. See below for the valid commands.
  • Lines following the $ lines are command output (merged stdout and stderr). Output is always treated literally.
  • After the command output there should be a blank line. Between that blank line and the next $ line, empty lines and lines beginning with # are ignored. (Because of these rules, cmdtest cannot distinguish trailing blank lines in the output.)
  • Syntax of a line beginning with $:
    • A sequence of space-separated words (no quoting is supported). The first word is the command, the rest are its args. If the next-to-last word is <, the last word is interpreted as a file and becomes the standard input to the command. None of the built-in commands (see below) support input redirection, but commands defined with Program do.
  • By default, commands are expected to succeed, and the test will fail otherwise. However, commands that are expected to fail can be marked with a --> FAIL suffix.

All test files in the same directory make up a test suite. See the TestSuite documentation for the syntax of test files, and the testdata/ directory for examples.

Commands

cmdtest comes with the following built-in commands:

  • cd DIR
  • cat FILE
  • mkdir DIR
  • setenv VAR VALUE
  • echo ARG1 ARG2 ...
  • fecho FILE ARG1 ARG2 ...

These all have their usual Unix shell meaning, except for fecho, which writes its arguments to a file (output redirection is not supported). All file and directory arguments must refer to the current directory; that is, they cannot contain slashes.

You can add your own custom commands by adding them to the TestSuite.Commands map; keep reading for an example.

Variable substitution

cmdtest does its own environment variable substitution, using the syntax ${VAR}. Test execution inherits the full environment of the test binary caller (typically, your shell). The environment variable ROOTDIR is set to the temporary directory created to run the test file (except in parallel mode; see below).

Running the tests

To test, first read the suite:

ts, err := cmdtest.Read("testdata")

Next, configure the resulting TestSuite by adding a Setup function and/or adding commands to the Commands map. In particular, you will want to add a command for your CLI. There are two ways to do this: you can run your CLI binary directly from from inside the test binary process, or you can build the CLI binary and have the test binary run it as a sub-process.

Invoking your CLI in-process

To run your CLI from inside the test binary, you will have to prevent it from calling os.Exit. You may be able to refactor your main function like this:

func main() {
        os.Exit(run())
}

func run() int {
    // Your previous main here, returning 0 for success.
}

Then, add the command for your CLI to the TestSuite:

ts.Commands["my-cli"] = cmdtest.InProcessProgram("my-cli", run)

Invoking your CLI out-of-process

You can also run your CLI as an ordinary program, if you build it first. You can do this outside of your test, or inside with code like

if err := exec.Command("go", "build", ".").Run(); err != nil {
        t.Fatal(err)
}
defer os.Remove("my-cli")

Then add the command for your CLI to the TestSuite:

ts.Commands["my-cli"] = cmdtest.Program("my-cli")

Running the test

Finally, call TestSuite.Run with false to compare the expected output to the actual output, or true to update the expected output. Typically, this boolean will be the value of a flag. So, your final test code will look something like:

var update = flag.Bool("update", false, "update test files with results")

func TestCLI(t *testing.T) {
    ts, err := cmdtest.Read("testdata")
    if err != nil {
        t.Fatal(err)
    }
    ts.Commands["my-cli"] = cmdtest.InProcessProgram("my-cli", run)
    ts.Run(t, *update)
}

Parallel mode

If you call ts.RunParallel instead of ts.Run, each file in the suite is run in parallel with the others. (The cases in a single file are still run sequentially, however.) In this mode, no temporary directories are created and ROOTDIR is not set.

go-cmdtest's People

Contributors

gbrlsnchs avatar jba avatar neild avatar toshi0607 avatar vangent avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-cmdtest's Issues

Filtering output support

Hey ๐Ÿ‘‹

Been using your package here to try to write some cli integration tests for my project box and am running into a few issues that I think could be solved by implementing support for filtering output with either regular expressions or glob patterns.

See for example Mercurial's test runner and the section on Filtering output

Would you consider adding this support? ๐Ÿค”

build error on Plan 9: undefined: syscall.Errno

Hello,

I happened to notice that the github.com/google/go-cmdtest package has a compilation error when targeting the Plan 9 operating system, which has three Go ports:

$ go tool dist list | grep plan9        
plan9/386
plan9/amd64
plan9/arm

The build error can be reproduced on another platform via cross-compilation, for example:

$ GOOS=plan9 GOARCH=amd64 go build ./...
# github.com/google/go-cmdtest
./cmdtest.go:536:17: undefined: syscall.Errno
$ echo $?
1

cmdtest.go imports "syscall" but the API of that package is system-specific. Its documentation says:

On most systems, that error has type syscall.Errno.

But isn't the case on Plan 9.

I don't use Plan 9 myself, but I noticed this when checking that another project (golang.org/x/vuln) at minimum compiles on all Go ports, and that project has go-cmdtest as a dependency. That project doesn't necessarily need to work on Plan 9 and can work around it, so this isn't causing a serious problem for it.

I'm reporting it here in case you find this information useful. If supporting Plan 9 is completely out of scope of go-cmdtest, you can leave the build error on Plan 9 as is and close this issue as not planned.

Thanks.

Handle different mount points when generating output via update feature

The problem

Currently, I can't update my ct files' outputs.

The reason

My /tmp is mounted at a different mount point than where my *.ct files are. Therefore, when updating them (by passing true as second argument to (*TestSuite).Run), I get the following error:

rename /tmp/cmdtest196280009 <my-ct-file-at-different-mount-than-tmp>: invalid cross-device link

This is due to using os.Rename to do the copy, which can't handle different mount points.

This happens because os.Rename uses the rename syscall under the hood, which, in its manual page, states:

(...) Linux permits a filesystem to be mounted at multiple points, but rename() does not work across different mount points, even if the same filesystem is mounted on both.

The solution

A possible solution would be to create, copy and remove the file in separate steps, which will avoid calling the rename syscall.


Nonetheless, great work so far, thanks!

Previous commands preventing new commands from running correctly.

I have setup a basic cobra CLI and found if you run if you run cmd --help before cmd --version the command returns help rather than version output. Suggesting that some of the state is being carried over between commands.

CT that functions as expected

$ cmd --version

CT that doesn't function as expected

$ cmd --help
...
$ cmd --version
...

Is there any way to prevent this or is this a bug with cobra?

I have set up an example repo:
https://github.com/Brookke/go-cmdtest-repo/blob/main/testdata/version.ct

I have also tried splitting up the commands into their own ct files, but still seeing the same issue.

Handle global path's in output?

Hi, i have test suite with absolute path in my stdout:

$ go-arch-lint check --project-path ${PWD}/test/check/project --arch-file invalid_spec.yml --output-color=false --> FAIL
...
[Archfile] path '$.components.not_exist.in': not found directories for 'not_exist' in '/home/neo/go/src/github.com/fe3dback/go-arch-lint/test/check/project/not_exist':
  14 |   main:
  15 |     in: .
...

image

This test will fall on all another computers, except my )
this lib have any option, to substitute variables in stdout, before comparing with real output?

I tried using ${PWD} but it didn't help.

check exit codes

Allow tests to express that a command fails with a particular exit code.

I propose that the syntax be extended so that in addition to

cmd --> FAIL

one can instead write

cmd --> FAIL 2

(or any other integer) to specify the desired exit code.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.