batterseapower / test-framework Goto Github PK

Hi,

I have 4000 test cases, and when one fails, everything is printed (including all the OK tests), and the output is generally difficult to navigate - I have to scroll far back to find which test failed. Is there a way to print only the tests that failed when a test fails (and the test group that it is in)?

This might already exist, so this is more of a question than an issue, but if it isn't possible, could it be added?

Thanks.

-Josh

QuickCheck allows to classify test cases using classify function. When running such property with quickCheck function the output is:

+++ OK, passed 100 tests (29% Short)

However, test-framework-quickcheck2 only outputs [OK, passed 100 tests], which forces me to run my tests manually to check whether I get the correct test distribution. It would be good if test-framework-quickcheck2 could display that information as well.

There are two issues when I run test suites with failing qc properties from ghci:

1. The counterexample is printed interleaved with normal test-framework output (sometimes it gets somewhere into the test label of the previous test, sometimes into an OK-bracket, etc.)
2. It sometimes deadlocks just after the first character of the counterexample has been printed (say the " for the example below)

Below is a minimal example (Just load it into ghci, and run main repetitively).

import Test.Framework
import Test.Framework.Providers.QuickCheck2 (testProperty)

main = defaultMain
  [ testProperty "reverse" prop_reverse
  , testProperty "foobar" (/= "")
  ]

prop_reverse :: [Int] -> Bool
prop_reverse xs = xs == (reverse . reverse) xs

Problem in 1 sentence: My tests fail when performed concurrently.

I have written an interface to a micro-controller which communicates over USB. Since USB connections are a scarce resource (and micro-controllers even more so) I can only open 1 connection at a time. My test suite consists of tests which each open a connection with the 1 micro-controller and then perform some test. The tests fail if they can't open a connection. Only 1 thread/process can have the connection at any single point in time. Therefore any test executed while another is running will fail.

I have "solved" the problem by compiling the test-suite without the "-N" RTS option, but that is not really a satisfactory solution.

A solution would be a special test group, or a parameter for test groups, in which tests are performed sequentially.

Supporting doctest-haskell can be an invaluable feature for the Haskell development experience, as well as for the development of test-framework itself.

This is because test-framework itself has a number of test reporters that output test results onto the consoles. However, currently there are no tests documenting what the console would actually output.

test-framework itself (core) has some tests, but doesn't declare the tests as a Test-Suite in the test-framework.cabal file.

This issue is concerned with widening test coverage of the codebase.

Tests would make it much easier for contributors-to-be to contribute, as it enables TDD and we can prevent regressions from happening when changes are made.

The test-suite can be made to depend on previous versions (e.g. v0.7) of test-framework itself for convenience of running tests.

I get this error:

Test/Framework/Providers/QuickCheck2.hs:16:31:
  Module `Test.QuickCheck.Test' does not export `run'

I played with the buildTestBracketed feature today and got the impression,
that the cleanup happens too early.

My guess is, that cleanup should only be called after taking all mvars:

diff --git a/core/Test/Framework/Runners/Core.hs b/core/Test/Framework/Runners/Core.hs
index 6e15b0a..42db5b7 100644
--- a/core/Test/Framework/Runners/Core.hs
+++ b/core/Test/Framework/Runners/Core.hs
@@ -105,7 +105,7 @@ instance TestRunner StdRunner where
                 --
                 -- FIXME: this is a bit of a hack because it uses one pool thread just waiting
                 -- for some other pool threads to complete! Switch to parallel-io?
-                return $ Just (run_test, actions' ++ [(cleanup >> mapM_ takeMVar mvars)])
+                return $ Just (run_test, actions' ++ [(mapM_ takeMVar mvars >> cleanup)])

 runTests' :: [StdRunner] -> IO ([RunningTest], [IO ()])
 runTests' = fmap (onRight concat . unzip . catMaybes) . mapM run

I have a package for adding support for diffing on failure for HUnit equality assertions, and it uses ansi-terminal to add coloring. I'd like to integrate it into test-framework so that the coloring is aware of the --plain option. Is there a way to access the test-framework options from a HUnit assertion?

while trying to install new packages on an old ppc target with ghc 6.12, i came across this:

[14 of 23] Compiling Test.Framework.Core ( Test/Framework/Core.hs, dist/build/Test/Framework/Core.o )
[15 of 23] Compiling Test.Framework.Providers.API ( Test/Framework/Providers/API.hs, dist/build/Test/Framework/Providers/API.o )
[16 of 23] Compiling Test.Framework.Runners.Core ( Test/Framework/Runners/Core.hs, dist/build/Test/Framework/Runners/Core.o )

Test/Framework/Runners/Core.hs:15:26:
    Module `Control.Exception' does not export `mask'
Failed to install test-framework-0.8

i haven't look if it's easy or not to fix.

would be nice to pass a flag to see test cases as they're generated with QC, a la verboseCheck - is it on the roadmap? have i missed a way to do it?

There is most likely a way to elegantly do this already, but I'm afraid I can't see it. All of my tests are integration tests that talk directly to a database, currently implemented by performing runTest for every testCase, which opens a database connection, runs the provided action in a ReaderT DatabaseConnection type monad, and finally yields IO () for testCase. However, I'd really like to only open a database connection once per entire test run and re-use it for every test.

One option is to pass an environment to the test cases, and have them runReaderT against that environment - but that still makes declaring my test suite a bit of a pain. Is there a better way we could do this?

Reported by Oren Ben-Kiki | August 15th, 2012 @ 09:58 AM

I added something like this:

    quickCheckWithNewArgs :: QC.Testable prop => (Args -> Args)-> prop-> IO ()
    quickCheckWithNewArgs modifyArgs test = do
      result <- quickCheckWithResult (modifyArgs $ stdArgs { chatty = False }) test
      case result of
        Success {} ->
          return ()
        Failure {} ->
          assertFailure $ "QuickCheck failed\n" ++ extra result
        GaveUp {} ->
          assertFailure $ "QuickCheck gave up after: " ++ (show $ numTests result) ++ " tests\n" ++ extra result
        NoExpectedFailure {} ->
          assertFailure $ "QuickCheck failed to trigger expected failure\n" ++ extra result
      where
        extra result =
          output result ++ (concatMap showLabelFrequency $ labels result)
        showLabelFrequency (label, frequency) =
          "\nSeen label: " ++ label ++ " with frequency: " ++ show frequency

    quickCheckRepeatTests :: QC.Testable prop => Int-> prop -> IO ()
    quickCheckRepeatTests number test = do
      quickCheckWithNewArgs setTestsNumber test
      where
        setTestsNumber number args =
          args { maxSuccess = number }

It seems like something along these lines should be a part of test-framework-quickcheck2.

My numerals package [1] has a test-suite with 3383 tests. This number will likely grow in the future. When running this many tests you quickly loose the overview. Typically less than 50 of my ~3000 tests will fail. I would like to see at a glance which tests failed. Those that succeeded are not interesting.

I would like an option to only display the results of tests that fail. This could be an option for defaultMain or an option for individual TestGroups.

[1] - http://code.haskell.org/numerals/

test-framework-quickcheck2 has an upperbound on QC2 that is out of date.

See HSpec.

I was going to write a quick patch for Example.lhs to move the "maximum generated tests" configuration into the code re:

http://www.haskell.org/pipermail/beginners/2012-June/010169.html

The example fails to compile, though. Using the command at the top of the file (runghc fails in the same way):

$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.4.1
$ ghc -package test-framework -package test-framework-quickcheck -package test-framework-hunit -threaded Example.lhs -o Example                               
[1 of 1] Compiling Main             ( Example.lhs, Example.o )

Example.lhs:64:17:
    No instance for (QuickCheck-1.2.0.1:Test.QuickCheck.Testable
                       (Gen Prop))
      arising from a use of `testProperty'
    Possible fix:
      add an instance declaration for
      (QuickCheck-1.2.0.1:Test.QuickCheck.Testable (Gen Prop))
    In the expression: testProperty "sort2" prop_sort2
    In the second argument of `testGroup', namely
      `[testProperty "sort1" prop_sort1, testProperty "sort2" prop_sort2,
        testProperty "sort3" prop_sort3]'
    In the expression:
      testGroup
        "Sorting Group 1"
        [testProperty "sort1" prop_sort1, testProperty "sort2" prop_sort2,
         testProperty "sort3" prop_sort3]

Originally reported by Gregory Collins | November 24th, 2009 @ 09:05 PM

If test-framework can take a flag to output junit XML, then continuous integration frameworks like Hudson can trivially parse the output.

Originally reported by Edward Z. Yang | August 26th, 2010 @ 06:31 PM

Run a test-framework driver.

Expected: When it's done, my terminal cursor is restored
Actual: Terminal cursor not restored.

This might be ansi-terminal/ansi-wl-pprint's fault, or we might not be calling the correct reset frob.

See Smallcheck.

The cabal version fails to build with quickcheck-2.1.0.3. Output:

jasper@alice$ cabal install test-framework-quickchek2
Resolving dependencies...
Configuring test-framework-quickcheck2-0.2.4...
Preprocessing library test-framework-quickcheck2-0.2.4...
Building test-framework-quickcheck2-0.2.4...
[1 of 1] Compiling Test.Framework.Providers.QuickCheck2 ( Test/Framework/Providers/QuickCheck2.hs, dist/build/Test/Framework/Providers/QuickCheck2.o )

Test/Framework/Providers/QuickCheck2.hs:16:30:
    Module `Test.QuickCheck.Test' does not export `run'
cabal: Error: some packages failed to install:
test-framework-quickcheck2-0.2.4 failed during the building phase. The
exception was:
ExitFailure 1

It works when I use cabal install test-framework-quickcheck2 --constraint='QuickCheck<2.1.0.3' though.

HUnit outputs error messages in a way, that makes it is easy to spot differences, e.g.:

expected: "1010110001001011"
 but got: "1010100001001011"

Here test-framework gives the following output instead:

Failed: expected: "1010110001001011"
 but got: "1010100001001011"

I suggest that we simply remove "Failed: " from the output.

I am using the test-framework to test a main program. That is, the main program has various flags, and I test that it properly responds to them - the bulk of the logic resides in pure libraries that are tested elsewhere.

To test the program, I use hDuplicate tricks to redirect stdout/stderr/etc. to files, invoke the main program, redirect them back and verify the results.

The problem is that the test framework is executing at a separate thread, and also writes to stdout/stderr. This (sometimes) corrupts the program's output - there is only one global stdout/stderr and when I redirect it, it applies to the tested program as well as for the test framework.

Initially I thought that using -j 1 would solve the problem. However, it doesn't; -j 1 only restricts the number of test worker threads, but the test framework thread always exists.

Possible solutions would be:

• Allow for -j 0 (or something like that) that insists on everything happening in one thread - this may impact some functionality (recovering from certain nasty faults in the tested code), but that would be OK.
• Avoiding doing stdout/stderr IO when a test is running in some worker thread (not certain how practical that would be).
• Something else?

Thanks,

Oren Ben-Kiki

Here is an example:

import Test.Framework
import Test.Framework.Providers.QuickCheck2 (testProperty)
import Test.QuickCheck

main :: IO ()
main = defaultMain [ testProperty "reverse" prop_reverse]

prop_reverse :: [Int] -> Property
prop_reverse xs = False ==> xs == (reverse . reverse) xs

quickCheck prop_reverse fails on this with

*** Gave up! Passed only 0 tests.

But test-framework marks it as passed.

Author of the Cabal TeamCity Plugin here.

As noted in the Known Issues section of the README, it would be great to have some native TeamCity reporting support from test-framework.

This would allow any test-framework-enabled project to have its tests reported painlessly with TeamCity, much in the same way that teamcity-messages does this for Python unittests. There is even support for parallel test run flows.

This issue may require an abstraction for pluggable test reporters that can be enabled by the user through command line flags. e.g. cabal test --test-reporter=teamcity.

The test reporter would spit out test execution details in the format of
TeamCity Service Messages, which TeamCity then picks up and provides all sorts of lovely metrics for.

I'm using iTerm on OSX.

Output looks like this in color mode:

pathStatus:

  [Empty,Empty,Empty] -> Unfinished: [OK]

  [Empty,Piece X,Empty] -> Unfinished: [OK]

  [Piece X,Piece O,Empty] -> Unfinished: [OK]

  [Piece X,Piece X,Piece X] -> Win X: [OK]

  [Piece X,T,Piece X] -> Win X: [OK]

  [T,T,Piece X] -> Win X: [OK]

  [T,T,T] -> Draw: [OK]

  [Piece X,Piece X,Piece O] -> Draw: [OK]
finalGameStatus:

  [Win X,Unfinished] -> Win X: [OK]

  [Win X,Win O] -> Win X: [OK]

  [Unfinished,Unfinished] -> Unfinished: [OK]

  [Win X,Draw] -> Win X: [OK]

         Test Cases   Total
 Passed  12           12
 Failed  0            0
 Total   12           12

But looks fine in plain mode:

pathStatus:
  [Empty,Empty,Empty] -> Unfinished: [OK]
  [Empty,Piece X,Empty] -> Unfinished: [OK]
  [Piece X,Piece O,Empty] -> Unfinished: [OK]
  [Piece X,Piece X,Piece X] -> Win X: [OK]
  [Piece X,T,Piece X] -> Win X: [OK]
  [T,T,Piece X] -> Win X: [OK]
  [T,T,T] -> Draw: [OK]
  [Piece X,Piece X,Piece O] -> Draw: [OK]
finalGameStatus:
  [Win X,Unfinished] -> Win X: [OK]
  [Win X,Win O] -> Win X: [OK]
  [Unfinished,Unfinished] -> Unfinished: [OK]
  [Win X,Draw] -> Win X: [OK]

         Test Cases   Total
 Passed  12           12
 Failed  0            0
 Total   12           12

And hide successes looks like this:

pathStatus:








finalGameStatus:





         Test Cases   Total
 Passed  12           12
 Failed  0            0
 Total   12           12

I've recently tried to dust off the test suite of one of my packages and, surprisingly (everything was OK in November), all the tests failed with the "Arguments exhausted after 0 tests" message. After some wiggling, I found that it only happens when the value of topt_maximum_generated_tests is greater than 1000 (I had it at 10000). I'm using test-framework-0.8, test-framework-quickcheck2-0.3.0.1 and QuickCheck-2.5.1.1. Downgrading to test-framework-0.6.2 and test-framework-quickcheck2-0.2.12.4 eliminates the issue.

any plans for a smallCheck provider?

@batterseapower I've ported over all remaining significant issues that were still open on Lighthouse over to GitHub issues.

More recent issues reported by other users are also in GitHub.

This renders the lighthouse bugtracker (i.e. http://bsp.lighthouseapp.com/projects/15661-hs-test-framework/overview) redundant.

I suggest that the link to it should be removed from the GitHub repo description as soon as possible.

Also, it may help to link to GitHub Issues in test-framework.cabal file to ensure the definitive issue tracker is unambiguous.

I'm trying to compile test-framework 0.4.2.2, but am running into multiple instances of the following error (all in Run.hs):

Test/Framework/Runners/Console/Run.hs:48:46:
    Ambiguous occurrence `<>'
    It could refer to either `Text.PrettyPrint.ANSI.Leijen.<>',
                             imported from `Text.PrettyPrint.ANSI.Leijen' at Test/Framework/Runners/Console/Run.hs:19:1-35
                          or `Data.Monoid.<>',
                             imported from `Data.Monoid' at Test/Framework/Runners/Console/Run.hs:21:1-18

I'm running GHC 7.4.1 and Text.PrettyPrint.ANSI.Leijen is from ansi-wl-pprint 0.6.4.

Configuring test-framework-quickcheck2-0.2.4...
Building test-framework-quickcheck2-0.2.4...
Preprocessing library test-framework-quickcheck2-0.2.4...
[1 of 1] Compiling Test.Framework.Providers.QuickCheck2 ( Test/Framework/Providers/QuickCheck2.hs, dist/dist-sandbox-3f4f1baf/build/Test/Framework/Providers/QuickCheck2.o )

Test/Framework/Providers/QuickCheck2.hs:16:31:
Module ‘Test.QuickCheck.Test’ does not export ‘run’

I currently have:

main :: IO ()
main = cleanState >> defaultMain tests

So I can make sure that I have a clean state before running integration tests. However, this runs even if I'm just running with --help. Ideally, this would only run if tests are actually going to run.

The default output format is great; but it's not suitable for consumption by build management, reporting, or CI systems.

To make test-framework become a viable part of a software development pipleine, it's useful to support other formats of test result outputs.

For instance, Mocha supports a vast range of variably useful reporters:

$ mocha --reporters

    dot - dot matrix
    doc - html documentation
    spec - hierarchical spec list
    json - single json object
    progress - progress bar
    list - spec-style listing
    tap - test-anything-protocol
    landing - unicode landing strip
    xunit - xunit reportert
    teamcity - teamcity ci support  ***
    html-cov - HTML test coverage
    json-cov - JSON test coverage
    min - minimal reporter (great with --watch)
    json-stream - newline delimited json events
    markdown - markdown documentation (github flavour)
    nyan - nyan cat!

For instance, Mocha's spec reporter looks like this:

This issue requires refactoring that would abstract from the way test progress and information is printed onto standard out at the moment into something more versatile.

Test reporters can have

• streaming output (like test-framework's current one, or progress bar): more suitable for development use in the terminal
• holistic (like the single JSON object that gets spit out right at the end): more suitable for consumption by other build processes.

Some part of this is necessary for #38 to be implemented. It's questionnable whether this issue can become a part of the #38 altogether.

There are a few minor issues with the module layout which have made it somewhat frustrating for me to develop using test-framework, one of which is that the MutuallyExcluded constructor is not exported from Test.Framework.Core. I am simply, it seems, not able to create my own sets of mutually exclusive tests, instead, I must mark everything in an entire tree mutually exclusive?

It seems this should be exported:

data MutuallyExcluded t = ME (MVar ()) t
    deriving Typeable

As well, let's suppose I would like to run tests in conjunction with other code. Perhaps I need to ensure a database, a script, or something else is initialized. That is, there is some resource my tests depend on and without it, the tests are pointless. I would greatly prefer to be able to use something very much like the defaultMainWithOpts, except it does this:

defaultMainWithOpts :: [Test] -> RunnerOptions -> IO ()
defaultMainWithOpts tests ropts = do
    let ropts' = completeRunnerOptions ropts

    when (unK$ ropt_list_only ropts') $ do 
      putStr $ listTests tests
      exitSuccess

    ...

    exitWith $ if ts_no_failures test_statistics'
               then ExitSuccess
               else ExitFailure 1

The exitWith guarantees I can run nothing after defaultMainWithOpts. I cannot implement defaultMainWithOpts myself because it and its dependencies in Test.Framework.Runners.Console references the following things that are hidden:

• completeRunnerOptions - not exported and depends on:
  • processorCount in Test.Framework.Processors - though I could perform the short-circuit myself of using numCapabilities
• listTests - not exported and depends on:
  • runTests in Test.Framework.Runners.Core not exported and depends on:
    • type RunningTest, data RunTest (..), and data SomeImproving (..) in Test.Framework.Runners.Core - cannot even reimplement because these are data types
    • testPatternMatches in Test.Framework.Runners.TestPattern - not exported and has many dependencies
    • executeOnPool in Test.Framework.Runners.ThreadPool - not exported and has many dependencies
• showRunTestsTop - not exported and depends on:
  • Types RunningTest, FinishedTest again
  • hideCursorDuring in Test.Framework.Runners.Console.Utilities - not exported
  • initialTestStatistics and totalRunTestsList in Test.Framework.Runners.Statistics
• gatherStatistics and ts_no_failures - not exported from Test.Framework.Runners.Statistics
• XML.produceReport - entire module not exported
• And so on and so forth.

In short, there is no simple or feasible way for the test-framework to be used by an end-user to create their own runner as a part of a program. I am, essentially, stuck with the kludge of using buildTestBracketed.

Now, I would like to propose a couple changes and I would like your feedback on them:

1. defaultMainWithOpts should be composed of several components, each of which ought to be exported.
2. The exported components should probably be something like:
   • A function for parsing command line arguments, possibly with a prefix added.
   • A function for running the tests, which would take a result consumer and return a data type describing the tests and their results.
   • A function for consuming the result stream of the tests and displaying live output to the console. This would be just one possible consumer - alternative consumers could render a web page and display the results by posting them as they are completed via Web Sockets or AJAX or what-not. As well, one possible consumer could be a database component to store results over time.
   • A function for post-rendering of the tests, in conjunction with the runner. The XML output renderer should be just one possible renderer. (A JSON renderer for example, should be possible to implement by end users.) The rendering output should be an appropriate data type, not just a ByteString, to allow interoperability with other APIs. The rendering API should expose a method of displaying it to the console - for example, to allow pretty-printing the JSON or XML as opposed to normalized / compact JSON or XML.
   • A function for determining if the tests have overall succeeded or failed according to the parameters specified.
3. MutuallyExcluded should be examined as a data type to determine if it's fit to be exported. Is there a reason why it is not?

I am willing to do some of the work above, but the test-framework internals are complex to the uninitiated and it would take me some time to understand all of the working parts.

A nice thing about test-framework is that it can be extended with new test types without modifying the framework itself, thanks to existential types.

One thing that is not yet extensible is options. Here's a sketch of a proposal to address this.

{-# LANGUAGE ScopedTypeVariables, DeriveDataTypeable #-}
module Options
  ( OptionSet
  , IsOption(..)
  , option
  , lookupOption
  , plusTestOption
  , plusTestOptions
  , SomeOptDescr(..)
  ) where

import Data.Typeable
import qualified Data.Map as Map
import Data.Map (Map)
import Data.Monoid
import System.Console.GetOpt
import Test.Framework hiding (plusTestOptions)

class Typeable v => IsOption v where
  defaultValue :: v

data SomeOptDescr = forall v. IsOption v => SomeOption (OptDescr v)

data OptionValue = forall v . IsOption v => OptionValue v

newtype OptionSet = OptionSet (Map TypeRep OptionValue)

instance Monoid OptionSet where
  mempty = OptionSet Map.empty
  mappend (OptionSet s1) (OptionSet s2) =
    OptionSet $ Map.unionWith (flip const) s1 s2 -- right-biased

option :: IsOption v => v -> OptionSet
option v = OptionSet $ Map.singleton (typeOf v) (OptionValue v)

lookupOption :: forall v . IsOption v => OptionSet -> v
lookupOption (OptionSet s) =
  case Map.lookup (typeOf (undefined :: v)) s of
    Just (OptionValue x) | Just v <- cast x -> v
    Just {} -> error "OptionSet: broken invariant (shouldn't happen)"
    Nothing -> defaultValue

{-
class TestResultlike i r => Testlike i r t | t -> i r, r -> i where
    ...
    testOptions :: [SomeOptDescr]
-}

plusTestOptions :: OptionSet -> Test -> Test
plusTestOptions = undefined

plusTestOption :: IsOption v => v -> Test -> Test
plusTestOption = plusTestOptions . option

-- Example option
newtype QCMaxTests = QCMaxTests Int
  deriving Typeable

instance IsOption QCMaxTests where
  defaultValue = QCMaxTests 100

{-
instance Testlike PropertyTestCount PropertyResult Property where
  ...
  testOptions =
    [
      SomeOption $ Option
        ['a']
        ["maximum-generated-tests"]
          (ReqArg (QCMaxTests . read) "NUMBER")
          "how many automated tests QuickCheck should try"
    , ...]
-}

Advantages

• Any test provider can register its own options
• ./mytest --help would automatically show only those options that are relevant for a particular test suite. E.g. if I only use SmallCheck and HUnit, I wouldn't see opitons related to QuickCheck.

Stuff to think about

• I'd like to see similar extensibility for runner options, to allow alternative runners and switching between them.
• There's some probability of clash for short option names. It can be solved this way: by default, only long option names are registered. A function is exposed which allows to add short options conveniently. Then providers may expose convenience functions like "addQuickCheckShortOptions". The user can opt in for those, or she can even define her own option names.