flyingmutant / rapid Goto Github PK
View Code? Open in Web Editor NEWRapid is a modern Go property-based testing library
Home Page: https://pkg.go.dev/pgregory.net/rapid
License: Mozilla Public License 2.0
Rapid is a modern Go property-based testing library
Home Page: https://pkg.go.dev/pgregory.net/rapid
License: Mozilla Public License 2.0
I have a generator + a filter and try to draw from this. And in some cases, I get this error:
[rapid] flaky test, can not reproduce a failure
...
Traceback (group did not use any data from bitstream):
vendor/pgregory.net/rapid/engine.go:77 in pgregory.net/rapid.assertf
vendor/pgregory.net/rapid/data.go:128 in pgregory.net/rapid.(*recordedBits).endGroup
vendor/pgregory.net/rapid/combinators.go:107 in pgregory.net/rapid.find[...]
vendor/pgregory.net/rapid/combinators.go:33 in pgregory.net/rapid.(*customGen[...]).value
vendor/pgregory.net/rapid/generator.go:72 in pgregory.net/rapid.(*Generator[...]).value
vendor/pgregory.net/rapid/collections.go:78 in pgregory.net/rapid.(*sliceGen[...]).value
vendor/pgregory.net/rapid/generator.go:72 in pgregory.net/rapid.(*Generator[...]).value
vendor/pgregory.net/rapid/generator.go:45 in pgregory.net/rapid.(*Generator[...]).Draw
vendor/pgregory.net/rapid/combinators.go:47 in pgregory.net/rapid.(*customGen[...]).maybeValue
vendor/pgregory.net/rapid/combinators.go:106 in pgregory.net/rapid.find[...]
vendor/pgregory.net/rapid/combinators.go:33 in pgregory.net/rapid.(*customGen[...]).value
vendor/pgregory.net/rapid/generator.go:72 in pgregory.net/rapid.(*Generator[...]).value
vendor/pgregory.net/rapid/combinators.go:94 in pgregory.net/rapid.(*filteredGen[...]).maybeValue
vendor/pgregory.net/rapid/combinators.go:106 in pgregory.net/rapid.find[...]
vendor/pgregory.net/rapid/combinators.go:90 in pgregory.net/rapid.(*filteredGen[...]).value
vendor/pgregory.net/rapid/generator.go:72 in pgregory.net/rapid.(*Generator[...]).value
vendor/pgregory.net/rapid/generator.go:45 in pgregory.net/rapid.(*Generator[...]).Draw
testing/integration/my_rapid_test.go:167 in testing/integration.(*myStateMachine).AddSiteMissing
vendor/pgregory.net/rapid/statemachine.go:174 in pgregory.net/rapid.runAction
vendor/pgregory.net/rapid/statemachine.go:149 in pgregory.net/rapid.(*stateMachine).executeAction
vendor/pgregory.net/rapid/statemachine.go:83 in pgregory.net/rapid.Run[...].func1
vendor/pgregory.net/rapid/engine.go:319 in pgregory.net/rapid.checkOnce
Original traceback:
vendor/pgregory.net/rapid/engine.go:596 in pgregory.net/rapid.(*T).fail
vendor/pgregory.net/rapid/engine.go:559 in pgregory.net/rapid.(*T).Fatalf
testing/integration/my_rapid_test.go:276 in testing/integration.(*myStateMachine).RemoveNonExisting
vendor/pgregory.net/rapid/statemachine.go:174 in pgregory.net/rapid.runAction
vendor/pgregory.net/rapid/statemachine.go:149 in pgregory.net/rapid.(*stateMachine).executeAction
vendor/pgregory.net/rapid/statemachine.go:83 in pgregory.net/rapid.Run[...].func1
vendor/pgregory.net/rapid/engine.go:319 in pgregory.net/rapid.checkOnce
...
I'm unsure how to fix this TBH. My guess is the filter filters everything and rapid gives up at some point but... that's just a guess.
Also, the test I have is pretty huge, I cannot reproduce this with a small example. Any idea what I can try?
Right now there is no
type StateMachine interface {
Init(*T)
Check(*T)
Cleanup()
}
I think these should really be mandatory (at least the first 2, but the implementation for no-op (*Model)Cleanup()
is a one liner).
var _ rapid.StateMachine = (*Model)(nil)
Cleanup()
could be split into its own single method interface and reflection usage replaced with a _, ok := x.(Cleanuper); ok
checkFirst of all, thanks for a great library! I just updated from v0.4.8 to v1.0.0, so I could use the fuzzing capabilities.
However, in my first attempt I've got a quite unreadable result, and would like some help on how to obtain the list of actions as they usually appear on failure.
--- FAIL: FuzzMultipleRunesProperty (61.81s)
--- FAIL: FuzzMultipleRunesProperty (0.00s)
value.go:586: [rapid] failed: can't find a valid action
Failing input written to testdata/fuzz/FuzzMultipleRunesProperty/51c85e8f6597f491
To re-run:
go test -run=FuzzMultipleRunesProperty/51c85e8f6597f491
The affected code is available at crdteam/causal-tree#12. I've created a rather complex state machine and a regular test passes successfully, but the fuzzed one fails:
func TestMultipleRunesProperty(t *testing.T) {
rapid.Check(t, func(t *rapid.T) {
t.Repeat(rapid.StateMachineActions(newMultipleRunesModel()))
})
}
func FuzzMultipleRunesProperty(f *testing.F) {
f.Fuzz(rapid.MakeFuzz(func(t *rapid.T) {
t.Repeat(rapid.StateMachineActions(newMultipleRunesModel()))
}))
}
The issue I'm facing is the need to create a generator based on the output of another generator.
To illustrate, consider a situation where we have four or six potential scenarios. Each scenario produces arbitrary data but with different combinations. Additionally, all subsequent generators depend on the first one or between each other.
Please note:
The rapid.Map
function isn't sufficient in this case because sometimes we need to transform a value into another value that is not fixed.
The rapid.Custom
function also isn't enough because we need to track original values without keeping them fixed.
By using rapid.FlatMap
, it's straightforward to create another combinator, rapid.Map2
. This combinator takes two generators and a function. The function combines the two generated values, producing another generator that depends on the first two.
var genBaz *rapid.Generator[Baz] = rapid.Map2[Foo, Bar, Baz](
genFoo,
genBar,
func(f Foo, b Bar) Baz { return Baz { f, b } },
)
Consider the following use case, which is similar to a project I'm currently working on:
var enumValueGen *rapid.Generator[SomeEnumType] = rapid.SampledFrom(allValuesInEnum())
var aSecondValueBasedOnEnum *rapid.Generator[SomeOtherType] = rapid.FlatMap[SomeEnumType, SomeOtherType](
enumValueGen,
func (e SomeEnumType) *rapid.Generator[SomeOtherType] {
switch e {
case SomeEnumValue1:
return generatorForScenario1
case SomeEnumValue2:
return generatorForScenario2
// ...
}
},
)
To address this issue, I propose an implementation using only exported symbols. This is the approach I'm currently using in a project:
// FlatMap is a function that takes a generator, and a function that takes the element
// produced by the generator, and produces another generator
// this function returns another generator that produces elements
// of a second type
func FlatMap[A, B any](genA *rapid.Generator[A], f func(a A) *rapid.Generator[B]) *rapid.Generator[B] {
return rapid.Custom[B](func(t *rapid.T) B {
return rapid.Map(genA, f).Draw(t, "flatMap").Draw(t, "B")
})
}
To have a simple way of writing tests which operate on shuffled data (with shuffling shrinking to no-shuffling).
In Hypothesis and other QuickCheck-like implementations, it is possible to calculate statistics, usually to validate that test data generation works as expected or is skewed somehow, as described here: https://hypothesis.readthedocs.io/en/latest/details.html#test-statistics
While it is easy to implement something like the event()
function of Hypothesis for rapid, generating reports is not. There are no means for decorating a property (that I am aware of) and calling a PrintStats()
function at the end of the property will be run every time (i.e. 100 times for a single property). What seems to work is the following:
func TestStackRapid(t *testing.T) {
defer stats.PrintStats(t)
rapid.Check(t, func(t *rapid.T) {
....
stats.Event(fmt.Sprintf("some event"))
...
}
}
Is that a hack or an intended way of decorating a property? And: are you interested in an implementation for such statistics?
Is it possible to compose predefined generators to create new ones. Something similar to the following example. Generator
type embeds an unexported type generatorImpl
right now. Is there or will there be a public interface to achieve something like the following. Thank you.
type vals struct { a int, b int, }
type customGen int
func (_ *customGen) Generate() interface{} {
return vals{ rapid.IntsRange(5, 20), rapid.IntsRange(30,32) }
}
Love this library! Thank you!
Right now if I want to map a function over a generator, I must use rapid.Custom
. Would a simpler Map
function be possible where it lets me simply define the translation between two types?
It would be great if we could assert that, at least, all conditional action got ran without error (which we do already) and without them calling Skip()
.
The reasoning is that some of those conditional action need a precondition to be fulfilled before being able to run. We would like to assert that the precondition got fulfilled at least once by asserting that the conditional action succeeded without calling Skip.
For a concrete example, let's think about database methods testing. Let's pick a Find function, for which we can either expect the function to return an item or an error if the item does not exist. Ideally, we'd like to test at least each case once. The best we could come up with currently is to have two conditional actions - FindExisting(*rapid.T)
and FindNotexisting(*rapid.T)
- which does what they advertise. Now, the FindExisting
conditional action has a pre-condition which is there must at least be one item in the database. If not, it calls Skip. And so in this example, we would want to be sure that the FindExisting
conditional action could succeed.
Of course, this implies the tests could get flaky, so this behavior should be opt-in. And we could be fancy too by passing in the names of the conditional actions you want to assert were called successfully. But even just asserting that all conditional actions were called would be nice already.
This is a contrived example: https://go.dev/play/p/PO2g_XAOjXF copied here for convenience:
type stateMachine struct {
precondition int
}
func (sm *stateMachine) WaitPrecond(t *rapid.T) {
if sm.precondition < 100 {
t.Skip()
}
panic("uh oh")
}
func (sm *stateMachine) SetPrecond(t *rapid.T) {
sm.precondition++
}
func (sm *stateMachine) Check(t *rapid.T) {
}
func TestStateMachine(t *testing.T) {
rapid.Check(t, rapid.Run[*stateMachine]())
}
See #54.
Like go test -fuzz
does, to have a semi-automatic regression test suite.
Generating proper URLs is tricky, and seems like something a library like rapid should provide. The implementation in hypothesis seems like a good starting point, though the URL generator is limited to http(s) schemes.
I suspect that currently we may (sometimes?) leave them hanging around, which is not great.
Expose in API the startSeed variable so that the test algorithm itself can set this parameters. Today this is possible only by a command line argument.
I tried to pass this parameter to my tests with "go test -args rapid.seed=999" and it works only when all tests needs rapid. When we have various tests mixed with tests that don't need "rapid", it fails because the test binary receives an "unsupported" argument (because flag on those binaries doesn't need this argument).
Currently it doesn't seem like shrinking is applied to regex-generated strings. This would be very useful in a lot of cases (though not so much in e.g. the IPv4 example). Or is shrinking done for those already? I couldn't find a shrunken output in my testing, only a very long example.
Old one: link. Maybe expand it and add a comparison to built-in fuzzing as well.
I've noticed that the concrete type *testing.T
is used as argument in many places, like rapid.Check
. But in the implementation, it almost immediately calls checkTB
which converts it to the interface tb
, which is a mirror of TB
.
We would like to be able to pass a rapid.TB
instead of *testing.T
so that we can wrap the testing structure with additional information and behavior. Would you be opposed to changing the signature of the functions to accept rapid.TB
instead of *testing.T
? I believe this would be a transparent change to the user given that *testing.T
satisfies rapid.TB
. We'd be willing to do the work if approved.
I'd like to generate one of a slice of constants to be able for instance to draw from a finite slice of hard-coded user IDs strings.
With generators of constants one would be able to use #2 wrapped in a call to OneOf
.
Adding a Weighted
taking a slice of int
-weighted pairs of generators would help in my testing too. (A call to Weighted
with generators of equal weights should be equivalent to a call to OneOf
). Weights must be > 0
.
EDIT: WRT state machine
Being able to weight actions based on which ones to run most would be nice.
func TestBla(t *testing.T) {
rapid.Check(t, rapid.StateMachine(&Model{
K: 1, // K is set
}))
}
type Model struct {
K int
}
func (m *Model) Init(t *rapid.T) {
// Do something with K but...
// K is zero
}
If this is documented then I missed it! (and sorry for the noise)
I understand cloning (shallow? deep?) that pre-Init
*Model
so it is available to every call of Init
can be a pain. The determinism guarantees of that may not even be possible to uphold... it's just that it seemed very natural to use like this.
Thoughts?
Hey Folks
In real world complex scenarios, visualizing the buckets in which generated input is falling helps debug/improve the generators.
I did not find this functionality in library.
Please see similar feature in cpp rapidcheck library document.
Example: User generator with name, sex, date of birth can add tags like male/female
for sex, child/adult/old
for age.
After running test, distribution will show the buckets and will help identify if there is any mistake in generator or if we are generating domain-interesting inputs or not. This is very useful for real world scenarios.
I am not sure how a flaky test is determined. It seems like a failed test is run again, and the stack tracebacks are compared. I have a very simple example without any non-determinism which is marked flaky, and the tracebacks look the same.
// property based testing used rapid
func TestAbs6(t *testing.T) {
rapid.Check(t, func(t *rapid.T) {
value := rapid.Int().Draw(t, "value").(int)
result := Abs(value)
if (value * value) != (result * result) {
t.Errorf("square of result not equal to square of input: %d, result %d\n", value, result)
}
})
}
// function to be tested
func Abs(x int) int {
if x < 0 {
return -x
}
if x > 15 {
return x - 1
}
return x - 1
}
abs_test.go:89: [rapid] flaky test, can not reproduce a failure
To try to reproduce, specify -run="TestAbs6" -rapid.seed=1579796179831259001
Traceback (square of result not equal to square of input: 25541, result 25540
):
/Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:188 in github.com/flyingmutant/rapid.checkOnce
/Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:142 in github.com/flyingmutant/rapid.doCheck
/Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:100 in github.com/flyingmutant/rapid.checkTB
/Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:71 in github.com/flyingmutant/rapid.Check
/Users/allangold/go/src/pbt1/abs_test.go:89 in pbt1.TestAbs6
/usr/local/go/src/testing/testing.go:827 in testing.tRunner
/usr/local/go/src/runtime/asm_amd64.s:1333 in runtime.goexit
Original traceback (square of result not equal to square of input: 25541, result 25540
):
/Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:188 in github.com/flyingmutant/rapid.checkOnce
/Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:163 in github.com/flyingmutant/rapid.findBug
/Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:134 in github.com/flyingmutant/rapid.doCheck
/Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:100 in github.com/flyingmutant/rapid.checkTB
/Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:71 in github.com/flyingmutant/rapid.Check
/Users/allangold/go/src/pbt1/abs_test.go:89 in pbt1.TestAbs6
/usr/local/go/src/testing/testing.go:827 in testing.tRunner
/usr/local/go/src/runtime/asm_amd64.s:1333 in runtime.goexit
Failed test output:
abs_test.go:90: [rapid] draw value: 25541
abs_test.go:93: square of result not equal to square of input: 25541, result 25540
Right now rapid.T
does not provide a way to differentiate what the previously run action was.
Because of this one has to resort to some hopeful (and maybe thread-safe) logic in their own (*Model).Check(*rapid.T)
.
How about creating a rapid.M
that embeds rapid.T
and additionally provides a (*rapid.M).LastAction()A
where A
represents the last action that was run or some empty value for the first step/check (I'm not clear on the step-check relationship just yet: checks have steps? or is it the other way around?).
A
could be string
but if possible it could be a pointer to *Model
's method.
$ GO111MODULE=on go get github.com/flyingmutant/rapid
go get: github.com/flyingmutant/[email protected]: parsing go.mod:
module declares its path as: pgregory.net/rapid
but was required as: github.com/flyingmutant/rapid
Goal: Show extensibility with custom generator and generator result filter.
Problem: User must read the source code to figure out not simple var generation. He wants the generation of slices, structs, ids.
type testStruct struct {
id, value int
name string
}
func TestExample(t *testing.T) {
generatedIds := map[int]bool{}
gen := rapid.Custom(func(rt *rapid.T) testStruct {
id := rapid.Int().Filter(func(v int) bool {
return !generatedIds[v]
}).Draw(rt, "id").(int)
generatedIds[id] = true
return testStruct{
id: id,
value: rapid.Int().Draw(rt, "value").(int),
name: rapid.String().Draw(rt, "name").(string),
}
})
rapid.Check(t, func(t *rapid.T) {
generatedIds = map[int]bool{}
testSlice1 := rapid.SliceOf(gen).Draw(t, "test_slice1").([]testSamer)
testSlice2 := rapid.SliceOf(gen).Draw(t, "test_slice2").([]testSamer)
mergeResult := Merge(testSlice1, testSlice2)
revertedResult := Remove(mergeResult, testSlice2)
assert.Equal(t, testSlice, revertedResult)
})
}
I will try to figure out a better sample for PR.
Is this a good idea?
May be a bit related to #36 (if we want to allow specifying the bitstream directly).
Main use case is making sure important corner cases are covered without duplicating the property-based test code.
Right now, statemachine's cleanup method is of type func()
instead of func(*testing.T)
.
It might be too late to change the function signature, but it might be worth to support both?
One use case is that we can use Init
to test the "constructors" of a data structure, and use Cleanup
to test the "destructors".
My use case is I have a very long test case and running it with 100 checks and 100 steps take more than 10 minutes. I'm fine running it every now and then for that long but I'd like to, by default, reduce the number of checks and steps for just this check and leave the default for the other checks.
I'm thinking the best behavior would be command-line flags take precedence over overrides which in turn take precedence over default values.
Is this something you thought about already? Would you be open to adding this feature?
After looking at the source code, we definitely need to make the flags
variable local and pass it to checkTB
Line 119 in 110d7a5
flags
variable in any way we want.
One way that could make sense to customize the flags
variable is to change the signature of Check
to take in a variadic argument. This would make the change backwards compatible:
type SetOption func(*cmdline)
func Check(t *testing.T, prop func(*T), options ...SetOption)
And have a few helpers:
func OverrideChecks(checks int) SetOption {
return func(c *cmdline) {
c.checks = checks
}
}
If you're okay to add this feature, I'll submit a PR with a possible implementation.
Otherwise, you have to work around by doing methods like Restart
, which Stop
+ call Init
internally.
Thank you for the great tool!
Could you help me? I'm trying to write a big.Int and big.Int range generator, what is the best way to do that? I'm trying it as an external code (if it worked fine, will push the PR).
I'd like to limit how many tests are run but only for one of my tests that is slower than the rest. This doesn't seem to be possible without hacks (like forcing the check function to return if some counter has reached a threshold).
Explain:
See #25
As there is code that can work with a := []int{}
, but fail with var a []int
.
Can we treat empty slices as having something like len = -1
during generation?
Howdy.
First of I'd just like to flag that I am not a Go developer and there may be good reasons for why this is but I found it to be quite cumbersome and unintuitive.
As it stands right now in v0.5.5, this fails:
import (
"fmt"
"pgregory.net/rapid"
)
type thing struct {
X myType
}
type myType int
func main() {
gen := rapid.Make[thing]()
for i := 0; i < 5; i++ {
fmt.Println(gen.Example(i))
}
}
With the panic:
panic: Make[main.thing]() failed to generate an example in 1000 tries: reflect.Set: value of type int is not assignable to type main.myType
The in genAnyStruct
this line fails:
Line 167 in b561020
x
is based on the kind of x
which is reflect.Int
and since the type of x
is not int
the generated value cannot be assigned.Line 166 in b561020
f := reflect.ValueOf(fieldGens[i].value(t)).Convert(s.Field(i).Type())
I I could try to address this in a PR but as i said, I am not a Go developer and I don't really get it's type system and there
are probably some very good reasons to keep things the way they are.
First of all thanks for this great library!
I'm trying to use testing.TB
as a common interface for both *testing.T
and *rapid.T
but I'm getting this error:
*rapid.T does not implement testing.TB (missing Skipped method)
I was hoping to avoid reinventing an interface for this.
It is more convenient to re-run the test from the seed locally than to grab a file from a failed CI run.
go mod
and godoc use the v0.1.0 tag properlygo mod
guidelines on what is required for a releasego mod
and godoc use the v0.2.0 tag properlyHi @flyingmutant, thanks for your excellent PBT library!
I am trying to write some demo with rapid
to show the power of PBT to my colleagues, but I encountered an error: [rapid] panic after 0 tests: group did not use any data from bitstream; this is likely a result of Custom generator not calling any of the built-in generators
Here is a minimal case:
package main
import (
"pgregory.net/rapid"
"testing"
)
type AddRequest struct {
A, B int
}
func Add(r AddRequest) int {
return r.A + r.B
}
func TestAdd(t *testing.T) {
// property test
comm := func(t *rapid.T) {
gen := rapid.Custom(func(t *rapid.T) AddRequest {
return AddRequest{
A: rapid.Int().Draw(t, "A"),
B: rapid.Int().Draw(t, "B"),
}
})
//req := rapid.Make[AddRequest]().Draw(t, "request")
req := gen.Draw(t, "request")
if Add(req) != req.A+req.B {
t.Errorf("wrong!")
}
}
rapid.Check(t, comm)
}
Both rapid.Custom
and rapid.Make
will return this error.
My environment:
go 1.19.6
rapid v1.0
arm64 macOS
To have an ability to customuze e.g. checks or steps per-test, without altering global environment.
Spin-off of #35.
This is either a bug or a feature. If you view a string
generator as SliceOf(Byte).Transform(string)
then it's correct and probably finds strange edge cases in functions that should be fixed. If instead you view the string
generator as SliceOf(Rune).Transform(string)
then it's producing string
s that are not UTF-8 compatible. I haven't dug too deeply into this. I just know that I had to start building my own string generators for this reason.
So that people know that an error was found and the tests did not hang.
Noticed by profiling the cosmos-sdk project which uses this code that has a RAM profile of
ROUTINE ======================== pgregory.net/rapid.expandRangeTable in /Users/emmanuelodeke/go/pkg/mod/pgregory.net/[email protected]/strings.go
6.08MB 6.08MB (flat, cum) 32.29% of Total
. . 357:func expandRangeTable(t *unicode.RangeTable, key any) []rune {
. . 358: cached, ok := expandedTables.Load(key)
. . 359: if ok {
. . 360: return cached.([]rune)
. . 361: }
. . 362:
. . 363: var ret []rune
. . 364: for _, r := range t.R16 {
. . 365: for i := uint32(r.Lo); i <= uint32(r.Hi); i += uint32(r.Stride) {
2.64MB 2.64MB 366: ret = append(ret, rune(i))
. . 367: }
. . 368: }
. . 369: for _, r := range t.R32 {
. . 370: for i := uint64(r.Lo); i <= uint64(r.Hi); i += uint64(r.Stride) {
3.44MB 3.44MB 371: ret = append(ret, rune(i))
. . 372: }
. . 373: }
. . 374: expandedTables.Store(key, ret)
. . 375:
. . 376: return ret
but we can modify expandRangeTable to preallocate ret by
make([]rune, 0, len(t.R16)+len(t.R32))
On re-profiling the same code, we now get this profile
ROUTINE ======================== pgregory.net/rapid.expandRangeTable in /Users/emmanuelodeke/go/pkg/mod/pgregory.net/[email protected]/strings.go
4.20MB 4.20MB (flat, cum) 28.40% of Total
. . 357:func expandRangeTable(t *unicode.RangeTable, key any) []rune {
. . 358: cached, ok := expandedTables.Load(key)
. . 359: if ok {
. . 360: return cached.([]rune)
. . 361: }
. . 362:
. . 363: ret := make([]rune, 0, len(t.R16)+len(t.R32))
. . 364: for _, r := range t.R16 {
. . 365: for i := uint32(r.Lo); i <= uint32(r.Hi); i += uint32(r.Stride) {
609.50kB 609.50kB 366: ret = append(ret, rune(i))
. . 367: }
. . 368: }
. . 369: for _, r := range t.R32 {
. . 370: for i := uint64(r.Lo); i <= uint64(r.Hi); i += uint64(r.Stride) {
3.61MB 3.61MB 371: ret = append(ret, rune(i))
. . 372: }
. . 373: }
. . 374: expandedTables.Store(key, ret)
. . 375:
. . 376: return ret
https://travis-ci.org/flyingmutant/rapid/jobs/553494317
$ go test -race
--- FAIL: TestGenFloat32Range (0.11s)
<autogenerated>:1: [rapid] failed after 6 tests: -5.87747245476067e-39 (0xb800000020000000) is not a float32
To reproduce, specify -run="TestGenFloat32Range" -rapid.seed=1562106454032796589
Failed test output:
<autogenerated>:1: [rapid] draw args: (-1.2037062e-35, 1)
<autogenerated>:1: -5.87747245476067e-39 (0xb800000020000000) is not a float32
FAIL
exit status 1
Include into the file metadata about exact version of rapid used -- so that we can bail out of reproducing when using a different version.
I am depending on libraries that require a concrete testing.T, and I want to use these libraries in the Init
method of a state machine test. Is it possible?
The signature of Init takes a * rapid.T
which isn't useable
Lines 108 to 118 in 24f65e3
type Model struct {
// .. stuff
}
func (m *Model) Init(t *rapid.T) {
// initialise state using *testing.T
// ..
}
func (m *Model) Cleanup() {
}
func (m *Model) Check(t *rapid.T) {
if 0 != 0 {
t.Fatalf("wrong!")
}
}
func (m *Model) Foo(t *rapid.T) {
// Action
}
func TestPBT(t *testing.T) {
rapid.Check(t, rapid.Run[*Model]())
}
Can I achieve what I want another way?
I stumbled upon your library completely by accident earlier this evening, and wanted to first off say - it's quite great. I've been using gopter
since before generics were in Go. I ported a few simple tests to see how the UX feels during writing tests - gopter v rapid. (I love it!)
Wanted to callout a small gap in the library as it stands: the lack of explicit control of number of tests run/seed/etc. I've relied on gopter.TestParameters
to override that stuff via code and find it quite useful. I think the -rapid.args
is incredibly useful for testing/development but I'd love the ability to override them via code to capture the required knobs for CI and so on.
How do you feel about exposing cmdline in some shape to library users?
I would like to test a program’s concurrent behavior by calling it from multiple goroutines, each imitating some workload. In each goroutine, I would make calls such as rapid.Int().Draw(t, ...)
, possibly also t.Helper()
, t.Errorf()
. But it’s not clear if such concurrent calls are safe on rapid.(*T)
. By contrast, testing.(*T).Helper
and Errorf
are documented as concurrent-safe.
If rapid.(*T)
is safe for concurrent use, this should be documented. If it is not safe, I think it should be made safe.
I see the problems with reproducibility though...
Thanks for this fine library.
I have the same problem as in #6: utility functions that I would like to call from both regular and property-based tests. They can’t take *rapid.T
because there is none in regular tests. They could take *testing.T
— the same one that I pass to rapid.Check
— and this is how #6 was settled — but, firstly, this makes testing functions slightly less readable (can’t just pass the same t
everywhere, need separate variables), and secondly, is it even correct? don’t the *rapid.T
methods do anything rapid-specific above and beyond the underlying *testing.T
?
It would be most convenient to make these utility functions take an interface — defined and exported from rapid
, e.g. rapid.CommonT
or something — that both *rapid.T
and *testing.T
implement.
But I’m not sure this is the right approach. Feel free to close this issue if you don’t like the idea.
From gomock:
// cleanuper is used to check if TestHelper also has the `Cleanup` method. A
// common pattern is to pass in a `*testing.T` to
// `NewController(t TestReporter)`. In Go 1.14+, `*testing.T` has a cleanup
// method. This can be utilized to call `Finish()` so the caller of this library
// does not have to.
type cleanuper interface {
Cleanup(func())
}
I am using rapid with gomock and found some errors in my tests when I noticed that Finish
was not being called on the mock controller after each Check
When a failed test output contains pointers, I'd expect to see the pointer value, not the address
Example code:
package exmaple
import (
"testing"
"github.com/flyingmutant/rapid"
)
type Example struct {
ExposeMe *string
}
func TestRapidLogOutputForPointers(t *testing.T) {
rapid.Check(t, func(t *rapid.T) {
rapid.Custom(genExample).Draw(t, "example")
t.Fail()
})
}
func genExample(t *rapid.T) Example {
return Example{
ExposeMe: rapid.Ptr(rapid.String(), false).Draw(t, "example field").(*string),
}
}
Expected output:
$ go test ./...
--- FAIL: TestRapidLogOutputForPointers (0.00s)
example_test.go:14: [rapid] failed after 0 tests: (*T).Fail() called
To reproduce, specify -run="TestRapidLogOutputForPointers" -rapid.seed=1575628109522093485
Failed test output:
example_test.go:15: [rapid] draw example: exmaple.Example{ExposeMe:&"some string here"}
FAIL
FAIL example 0.009s
FAIL
Actual output:
$ go test ./...
--- FAIL: TestRapidLogOutputForPointers (0.00s)
example_test.go:14: [rapid] failed after 0 tests: (*T).Fail() called
To reproduce, specify -run="TestRapidLogOutputForPointers" -rapid.seed=1575628109522093485
Failed test output:
example_test.go:15: [rapid] draw example: exmaple.Example{ExposeMe:(*string)(0xc000082fb0)}
FAIL
FAIL example 0.009s
FAIL
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.