Fix CI about quickcheck-state-machine HOT 15 CLOSED

(Like we do with postgres, etc. Because then anyone with docker installed can run our test suite, and we already skip the tests which require docker if docker isn't installed.)

from quickcheck-state-machine.

I thinks it's some tests which take too long. I can try a couple things to make them faster.

from quickcheck-state-machine.

Here's another problem that I've not been able to reproduce locally:

 NoRaceBug:                                                    FAIL
      Exception: QuickCheck.elements used with empty list
      CallStack (from HasCallStack):
        error, called at ./Test/QuickCheck/Gen.hs:203:15 in QuickCheck-2.13.1-1rO57X9FmbTEL9U6cBekOs:Test.QuickCheck.Gen

(Could't find any seed or --quickcheck-replay to reproduce this in the logs, shouldn't that always be there?)

from quickcheck-state-machine.

Is this the correct link you have above? I couldn't find the failure..

from quickcheck-state-machine.

I'm pretty sure (I doubled checked when I posted), could it be that because I reran the job (because it timed out) it overwrote the logs...?

from quickcheck-state-machine.

Hmm, https://travis-ci.org/advancedtelematic/quickcheck-state-machine/jobs/553776901:

...
Setting up build cache
$ export CASHER_DIR=${TRAVIS_HOME}/.casher
0.16s$ Installing caching utilities
0.00s1.18sattempting to download cache archive
fetching master/cache-linux-trusty-811a73447d636739ee8e0adc0d376798ed2e3294169c24b7738557e7b2e4b580.tgz
could not download cache
0.00s1.07sadding /home/travis/.ghc to cache
creating directory /home/travis/.ghc
adding /home/travis/.cabal to cache
creating directory /home/travis/.cabal
adding /home/travis/.stack to cache
creating directory /home/travis/.stack
adding /home/travis/build/advancedtelematic/quickcheck-state-machine/.stack-work to cache
creating directory /home/travis/build/advancedtelematic/quickcheck-state-machine/.stack-work

and then a bit later, stack says:

No old cache found, populating cache from scratch

from quickcheck-state-machine.

Caching of stack build seems to work again now (no change made).

from quickcheck-state-machine.

The Overflow 4 thread property sometime fails on CI:

https://travis-ci.org/advancedtelematic/quickcheck-state-machine/jobs/553916088

When I tried to reproduce it locally it first failed, but not for the right reason:

      *** Exception running callback: 
      dot: runInteractiveProcess: runInteractiveProcess: exec: does not exist (No such file or directory)

Again bitten by how expectFailure thinks exceptions are failures...

Anyway, after installing graphviz, renaming testProperty "Overflow 4 threads" to testProperty "4-threads" and running:

stack test --ta="-p Overflow.4-threads --quickcheck-replay=481701"

(The reason for renaming the property is that I don't know how to run properties with spaces in the name with the -p flag.)

I got:

Tests
  Overflow
    4-threads: FAIL (27.30s)
      *** Failed! Passed 100 tests (expected failure).
      Use --quickcheck-replay=481701 to reproduce.

1 out of 1 tests failed (27.30s)

I guess we need to fix three things:

• Rename properties so we can run them separately
• Higher maxSuccess for that property
• Check if dot is in PATH, if not prompt the user to install it and exit gracefully
• Install graphviz on Travis?

from quickcheck-state-machine.

Sorry for creating this issue with dot

Install graphviz on Travis?

Do you think this is possible?

from quickcheck-state-machine.

Yeah, should be an apt-get install graphviz somewhere in the Travis config.

But, we probably want to avoid crashing when dot is not available and we will probably never look at the .svg produced on Travis. So maybe we don't need to actually install graphviz? It should just say, "Found counterexample, but cannot visualise it because graphviz isn't installed" -- I think that would be good enough for CI?

from quickcheck-state-machine.

Yes, you're right. I'll write some handlers for the case where graphviz is not install.

from quickcheck-state-machine.

I'm currently writing a test for rqlite, which may require installing https://github.com/rqlite/rqlite/releases/tag/v4.5.0 if we want to run it on ci.
Do you think that's possible?

from quickcheck-state-machine.

It's possible to install anything on Travis, but perhaps using docker is easier?

from quickcheck-state-machine.

The following CI run, https://travis-ci.org/advancedtelematic/quickcheck-state-machine/jobs/560407777 , contains a new problem:

    Bookstore input validation bug:                                
FAIL (20.01s)
      *** Failed! Passed 100 tests (expected failure).
      Use --quickcheck-replay=361731 to reproduce.

Need to bump maxSuccess...

from quickcheck-state-machine.

I think this is fixed, for now...

from quickcheck-state-machine.