Git Product home page Git Product logo

sweet's Introduction

sweet

Sweet is a pluggable test runner capable of hooking into standard Go tests. It attempts to provide access to the standard Go test tool as close as possible while adding support for test suites and plugins that can hook into test results to add additional functionality.

Using Sweet

To use Sweet as your test runner you need to add the TestMain function from the Go testing package. This will allow Sweet to run code before and after the actual tests run. Inside the TestMain function you'll want to call the sweet.Run function to both set up the Sweet configuration as well as run the tests:

package mypackage

import (
    "testing"
    "github.com/aphistic/sweet"
)


func TestMain(m *testing.M) {
    sweet.Run(m, func(s *sweet.S) {
        // Configuration goes here
    })
}

Defining a Suite

A test suite in Sweet is just a normal Go struct with methods named similar to standard go test names (beginning with Test). Creating a suite named FailSuite with a single test that always fails and then running it with Sweet looks like the following code:

package mypackage

import (
    "testing"
    "github.com/aphistic/sweet"
)


func TestMain(m *testing.M) {
    sweet.Run(m, func(s *sweet.S) {
        s.AddSuite(&FailSuite{})
        // Add any additional suites the same way
    })
}

type FailSuite struct {}

func (s *FailSuite) TestAlwaysFails(t sweet.T) {
    t.Fail()
}

Using a Plugin

Sweet supports plugins to add functionality that isn't typically available with the standard Go testing tools. One such example is sweet-junit, a plugin that generates a junit.xml file for each package it's used in. To add to the previous examples, this is how you'd add the sweet-junit plugin to your tests:

package mypackage

import (
    "testing"
    "github.com/aphistic/sweet"
    junit "github.com/aphistic/sweet-junit"
)


func TestMain(m *testing.M) {
    sweet.Run(m, func(s *sweet.S) {
        s.RegisterPlugin(junit.NewPlugin())

        s.AddSuite(&FailSuite{})
        // Add any additional suites the same way
    })
}

type FailSuite struct {}

func (s *FailSuite) TestAlwaysFails(t sweet.T) {
    t.Fail()
}

Using an External Matcher

Sweet was designed with the capability to use external matchers in mind. You can write standard Go unit tests but you can also hook a different matcher library in and use that.

So far the only matcher that Sweet has been tested with and has hooks for is the Gomega library.

To use Gomega in the above example, you would do:

package mypackage

import (
    "testing"

    . "github.com/onsi/gomega"

    "github.com/aphistic/sweet"
    junit "github.com/aphistic/sweet-junit"
)


func TestMain(m *testing.M) {
    RegisterFailHandler(sweet.GomegaFail)

    sweet.Run(m, func(s *sweet.S) {
        s.RegisterPlugin(junit.NewPlugin())

        s.AddSuite(&FailSuite{})
        // Add any additional suites the same way
    })
}

type FailSuite struct {}

func (s *FailSuite) TestAlwaysFails(t sweet.T) {
    t.Fail()
}

sweet's People

Contributors

acmanderson avatar aphistic avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar

Forkers

acmanderson

sweet's Issues

Catch assertions in goroutines

Currently, it seems that sweet panics when an assertion failure occurs in a goroutine.

go func() {
	defer close(sync)

	err := server.Subscribe(&proto.SubscribeRequest{}, subscribeServer)
	Expect(err).To(BeNil())
}()

An example error:

panic: (*sweet.testFailure) (0x8b0ec0,0xc420153b90)

goroutine 8 [running]:
panic(0x8b0ec0, 0xc420153b90)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
<anon-project>vendor/github.com/aphistic/sweet.GomegaFail(0xc420087100, 0x77, 0xc420155840, 0x1, 0x1)
	/home/efritz/go/src/<anon-project>vendor/github.com/aphistic/sweet/failures.go:31 +0xc4
<anon-project>vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).match(0xc420226640, 0xc446c0, 0xc87588, 0x1, 0x0, 0x0, 0x0, 0xc87588)
	/home/efritz/go/src/<anon-project>vendor/github.com/onsi/gomega/internal/assertion/assertion.go:69 +0x1ed
<anon-project>vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).To(0xc420226640, 0xc446c0, 0xc87588, 0x0, 0x0, 0x0, 0xc420226640)
	/home/efritz/go/src/<anon-project>vendor/github.com/onsi/gomega/internal/assertion/assertion.go:35 +0xae
<anon-project>server.(*HandlersSuite).TestOnlyOneConcurrentSubscriberAllowed.func1(0xc420248000, 0xc420156e70, 0xc42002e240)
	/home/efritz/go/src/<anon-project>server/handlers_test.go:60 +0x12d
created by <anon-project>server.(*HandlersSuite).TestOnlyOneConcurrentSubscriberAllowed
	/home/efritz/go/src/<anon-project>server/handlers_test.go:61 +0x324
exit status 2
FAIL	<anon-project>	0.007s

Allow naming of suites

Users should be able to implement an interface that provides a friendly name for their suites in test results instead of the actual name of the suite struct.

Subtest Parallel() support

Expected failing subtests pass when using t.Parallel

In the code provided the difference can be seen by uncommenting the t.Parallel() allowing the failed second test to pass.

func TestMain(m *testing.M) {
	RegisterFailHandler(sweet.GomegaFail)

	sweet.Run(m, func(s *sweet.S){
		s.AddSuite(&checkout{})
	})
}

func main() {
	fmt.Println("vim-go")
}

func shopCheckout(count int) int {
	count += (count * 2)
	return count
}

func (s *checkout) TestAppleShopCheckout(t sweet.T) {
	tcs := []struct {
		testName string
		userName string
		count	int
		expectedTo  types.GomegaMatcher
	}{
		{
			testName:   "valid-price",
			userName:   "Bob",
			count:      15,
			expectedTo: Equal(30),
		},
		{
			testName:   "valid-price",
			userName:   "Gina",
			count:      12,
			expectedTo: Equal(44),
		},
	}
	for _, tc := range tcs {
		tc := tc
		t.Run(tc.testName, func(t sweet.T) {
			//t.Parallel()
			got := shopCheckout(tc.count)
			Expect(got).To(tc.expectedTo)
		})
	}
}

dafuq?

This is the bug I was talking about the other day, not sure where who's marking it as a pass.

ok  	bitbucket.org/rw_grim/convey/ssh	0.007s
0
grim@cloak:~/g/src/bitbucket.org/rw_grim/convey$ go test -v bitbucket.org/rw_grim/convey/ssh ; echo $?
=== RUN   Test
=== RUN   Test/sshSuite/TestKeysAvailableEmpty
=== RUN   Test/sshSuite/TestKeysAvailableWildcardNoKeys
huh?
=== RUN   Test/sshSuite/TestKeysAvailableWildcardWithKeys
huh?
-------------------------------------------------
FAIL: sshSuite/TestKeysAvailableWildcardWithKeys

agent_test.go:79
Expected
    <bool>: false
to be true

sshSuite - Total: 3, Passed: 2, Failed: 1
--- PASS: Test (0.00s)
    --- PASS: Test/sshSuite/TestKeysAvailableEmpty (0.00s)
    --- PASS: Test/sshSuite/TestKeysAvailableWildcardNoKeys (0.00s)
    --- PASS: Test/sshSuite/TestKeysAvailableWildcardWithKeys (0.00s)
PASS
ok  	bitbucket.org/rw_grim/convey/ssh	0.005s
0

Move plugin options to strings

Plugin command line options should be moved to strings so go test doesn't break if a plugin isn't added to a subpackage.

The idea atm is something like: -sweet.opt "junit.output=junit.xml"

Allow filtering on invocation

Would be nice to invoke tests based on a whitelist or blacklist filter (particularly by suite name). This would allow some tests that require an external dependency to run to be run separately from "pure" tests.

Add color coded diffs to failed test output

It's really hard to see what's wrong in a big struct or set of bytes sometimes, so adding some red, green, or yellow to added/missing/different data would be a HUGE help.

Test failtests fails

Golang 1.12.2 on Fedora Rawhide:

github.com/aphistic/sweet/failtests
-------------------------------------------------
FAIL: FailSuite/TestFails

failtests_test.go:21
Expected
    <bool>: false
to equal
    <bool>: true

-------------------------------------------------
FAIL: FailSuite/TestUtilFunc

failtests_test.go:33
failtests_test.go:27
Expected
    <bool>: false
to be true

--- FAIL: Tests (0.00s)
    --- FAIL: Tests/FailSuite (0.00s)
        --- FAIL: Tests/FailSuite/TestFails (0.00s)
        --- FAIL: Tests/FailSuite/TestUtilFunc (0.00s)
FAIL

Suite Results:
--------------
FailSuite - Total: 2, Passed: 0, Failed: 2, Skipped: 0

exit status 1
FAIL	github.com/aphistic/sweet/failtests	0.011s

Go 1.16: panic: reflect: Call using sweet.testDeps as type testing.testDeps

On Fedora Rawhide with Golang 1.16 beta 1, I get the following error running go test:

Testing    in: /builddir/build/BUILD/sweet-0.3.0/_build/src
         PATH: /builddir/build/BUILD/sweet-0.3.0/_build/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
       GOPATH: /builddir/build/BUILD/sweet-0.3.0/_build:/usr/share/gocode
  GO111MODULE: off
      command: go test -buildmode pie -compiler gc -ldflags " -X github.com/aphistic/sweet/version=0.3.0 -extldflags '-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld  '"
      testing: github.com/aphistic/sweet
github.com/aphistic/sweet
panic: reflect: Call using sweet.testDeps as type testing.testDeps
goroutine 1 [running]:
reflect.Value.call(0x5590e8fe3fe0, 0x5590e9025b40, 0x13, 0x5590e8e4a783, 0x4, 0xc00006c6c0, 0x4, 0x4, 0xc0001b1830, 0x5590e8c154df, ...)
	/usr/lib/golang/src/reflect/value.go:406 +0x1317
reflect.Value.Call(0x5590e8fe3fe0, 0x5590e9025b40, 0x13, 0xc00006c6c0, 0x4, 0x4, 0x2, 0x4, 0x40)
	/usr/lib/golang/src/reflect/value.go:337 +0xb9
github.com/aphistic/sweet.mainStart(0xc00006ac00, 0x5590e902eac8, 0xc0000572f0, 0x5590e926bb00)
	/builddir/build/BUILD/sweet-0.3.0/_build/src/github.com/aphistic/sweet/gotest_integration.go:45 +0x617
github.com/aphistic/sweet.Run(0xc000198180, 0x5590e90252d0)
	/builddir/build/BUILD/sweet-0.3.0/_build/src/github.com/aphistic/sweet/runner.go:86 +0x1bf
github.com/aphistic/sweet.TestMain(0xc000198180)
	/builddir/build/BUILD/sweet-0.3.0/_build/src/github.com/aphistic/sweet/sweet_test.go:18 +0x85
main.main()
	_testmain.go:43 +0x165
exit status 2
FAIL	github.com/aphistic/sweet	0.008s

Enhance support for filtering tests

Right now it's possible to filter for a specific suite of tests but it would be helpful to be able to filter for a specific test. Maybe adding regex support to allow filtering against the suite name + the test name?

More stack context on assertion failure

Sometimes it's helpful to define a method that does some assertions, but that doesn't give any help when a test actually fails - especially since such methods are called frequently in a test with different arguments.

For example, one may want to do something like this:

checkCounts := func(connecting, starting, idle, busy, finishing int) {
	counts := <-ch1 // shared scope
	Expect(counts.connecting).To(Equal(connecting))
	Expect(counts.starting).To(Equal(starting))
	Expect(counts.idle).To(Equal(idle))
	Expect(counts.busy).To(Equal(busy))
	Expect(counts.finishing).To(Equal(finishing))
}

for i := 0; i < numTasks; i++ {
	chs[i] <- StateStarting
	checkCounts(numTasks-(i+1), i+1, 0, 0, 0)
}

But the line number the failure occurs on is where the assertion happens and not where the method is originally called (which would also be useful).

Perhaps sweet could (with a runtime flag) display additional stack context up to the test method body.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.