While writing some tests that need to run sequentially, I tried using sequential from 4.0.0-rc1, but it does not seem to actually run the tests in sequence. The code is at:

ford-prefect/purescript-redis@2c04b23

It would be nice to be able to write a test body with assertions for pending tests, that isn't run, but is only there for documenting the future test. Perhaps a pending' DSL function that just ignores the Aff:

describe "my library" do
  pending' "should do math" $
    1 + 1 `shouldEqual` 2

I feel the API of this package is solid now, and not breaking often. There are some more features we'd like to have, but I think they can be added without breaking compatibility with existing users' code. Thus, I think we should go for version 1.0.0 to signal the stability.

@felixSchl thoughts?

I noticed there's some FFI modules for doing ansi color stuff. There's a pure PS implementation that only depends on purescript-strings: https://pursuit.purescript.org/packages/purescript-ansi/4.0.0.

Are you up for a PR to use purescript-ansi instead of writing FFI modules in this library?

Something like http://hspec.github.io/hspec-discover.html.

It could probably be done in runtime by searching the NODE_PATH after *Spec modules, but that locks it down to NodeJS.

I can see this ��� strange symbols when travis is running tests for example here (it's not just in one place). but this is not happening in my terminal (I'm using osx). not sure what reason could be.

Is there something like mocha's only modifier (i.e. it.only) to only run a certain test?

Start by just outputting the results in HTML.

Maybe integrate with Mocha/Karma.

Frameworks like Mocha needs to declare it, describe and so on synchronously and then await the asynchronous tests. Currently the reporter type class receives all test results after the tests have run (possibly asynchronously) which makes it impossible to use with Mocha.

See purescript-spec/purescript-spec-mocha#1.

For parallelism with shouldEqual/shouldNotEqual?

I think a structure like this would provide a much better separation of concerns:

shouldEqual :: String -> String -> Either String String
shouldEqual actual expected =
  if (actual /= expected)
  then Left "\"" <> actual <> "\" /= \"" <> expected <> "\""
  else Right ""


shouldBeOk :: forall r. Either String String -> Aff r Unit
shouldBeOk value = case value of
  Left error -> fail error
  Right _ -> (pure unit)


it "should do something" do
  shouldBeOk do
    actual <- getSomeEither
    expected <- getAnotherEither
    actual `shouldEqual` expected

Then all assertions would simply be of type String -> String -> Either String String

After install purescript-spec in a different repo and copy/pasting the example, I can't seem to get it to compile. I get the following error message:

Error found:
in module Main
at src/Main.purs line 13, column 8 - line 27, column 51

  Could not match type
          
    Effect
          
  with type
       
    Aff
       

while trying to match type Effect Unit
  with type Aff t0
while checking that expression (run [ consoleReporter                                        
                                    ]                                                        
                               )                                                             
                               ((describe "purescript-spec") ((discard (...)) (\$__unused -> 
                                                                                 ...         
                                                                              )              
                                                             )                               
                               )                                                             
  has type Aff t0
in value declaration main

where t0 is an unknown type

See https://github.com/purescript/documentation/blob/master/errors/TypesDoNotUnify.md for more information,
or to contribute content related to this error.

The file I'm trying to run is https://github.com/purescript-spec/purescript-spec/blob/master/example/Main.purs.

The ReadMe links to this page: https://purescript-spec.github.io/purescript-spec/

However, the actual docs for things like Hooks (https://github.com/purescript-spec/purescript-spec/blob/master/docs/writing-specs.md) aren't included.

DotReporterConfig contains slow prop which is not used in dotReporter

The type signature for run demands that all runners use the same config type, which is not the case. @felixSchl ideas?

I'm running Pandoc 2.7.3 at the moment and finding that options --chapters and -S have changed since the Makefile was created. Therefore, the docs can't be compiled to HTML at the moment.

Unable to find a suitable version for purescript-console, please choose one by typing one of the numbers below:
    1) purescript-console#3.0.0 which resolved to 3.0.0 and is required by test
    2) purescript-console#^3.0.0 which resolved to 3.0.0 and is required by purescript-psci-support#3.0.0
    3) purescript-console#^4.1.0 which resolved to 4.2.0 and is required by purescript-spec#3.1.0

any option is not working.

For an unattended test execution on CI/CD pipeline I don't need console colors.
Pulp has a nice global option --monochrome Don't colourise log output. for it.
However, it doesn't seem to be taken into effect by purescript-spec:

What do you think of the idea to have itOnly and describeOnly cause compile warnings similar to how traceAnyA and friends does. I can't speak for others, but I have accidentally committed an itOnly block far too many times, not realizing.

What should be a pretty straightforward use case becomes pretty crazy (and ugly)

main ∷ Effect Unit
main = do
  specs <- discover "\\.*Spec"
  runSpecT config [ consoleReporter ] specs # un Identity # launchAff_
  where
  config =
    defaultConfig
      { slow = 5.0 # Seconds # fromDuration
      , timeout = Nothing
      }

How about runSpecWithConfig in a way that the un Identity can be avoided?

I found that having a large amount of purely synchronously running tests can cause stack overflows.
I saw that Aff has a MonadRec instance breaking the execution context after every 100 binds.

I am not sure how to run it using the MonadRec instance though, as runAff is implemented using the FFI.
I am currently recovering from this by breaking the execution context with a manual later call.

Maybe we could provide a BaseReporter without the CONSOLE effect, only doing the pipes machinery, not having default implementations that print to the console. If someone would like to build a reporter for web browsers, or something like the xUnit reporter, the console effect might not be what you want.

I've created a pull request on package-sets https://github.com/purescript/package-sets/pulls , after adding pipes and its' dependency mmorph.

I'd be happy to contribute my psc-package.json as well. psc-package only runs 'build' at the moment, there is preliminary support for using it with pulp as well. I haven't figured out how best to run tests with it, I thought at least getting 'spec' to compile would be a good start.

(added because I've been bitten by transitive dependency hell with bower, the last time bower won and this seemed like a worthwile thing to try. Also frees purescript-spec from worrying about synchronizing version numbers with other packages or not.).

psc-bundle (and pulp build with the --to flag) gives me the error The module could not be parsed. whenever Test.Spec.* modules have been built to the output directory. When I delete those directories, I am able to bundle. Any idea what's causing this?

I am opening this issue as a placeholder in case anyone wants to discuss remaining tasks or attach release-related commits.

But I'm mostly just wondering whether the current "v4.0.0-rc1" tag might change at all, or if it is safe to depend on the (exact) version in a dependent project. If the latter, then this will help me close purescript-trout#12.

Thanks for your great work on this library!

What are thoughts on adding something like this this?

forAll :: ∀ e a f. Foldable f => (a -> String) -> String -> f a -> (a -> Aff e Unit) -> Spec e Unit
forAll itTitle title arb f = describe title do
  for_ arb \a -> it (itTitle a) (f a)

...
  forAll _.str "format (unformat a) = a" arb \({ str }) -> do
    (format $ unformat str) `shouldEqual` (Right str) 
...

I have been using for_ in it block but if some test case fails then other items are not tested and you can't see them in log. using this function you can see each item in log and all of them will be executed.

Regardless of how the describes and its are nested this should not happen:

A » B » C
✓︎ Yay!

A » B » C
✓︎ Oh noes...

It should be:

A » B » C
✓︎ Yay!
✓︎ Oh noes...

Export run'' function, please. And I think it should be run', since there is no run function with single prime.

Good evening, greetings from Australia and thank you for your amazing work on purescript!

Occasionally I'll have a Spec so short and to-the-point that a textual description of it serves only to duplicate the literal interpretation of the expression.

it "accepts strings that contains substrings" $
  "foobar" `AS.shouldContain` "foo"

(From Test.Spec.AssertionSpec)

Ruby uses polymorphism to accept a spec that only provides a block.

  it { expect("foobar").to contain "foo" }

(From http://www.betterspecs.org/#short)

lit creates a spec for test cases where the code reads as a literal description of its behaviour.

This requires a degree of connoisseurship as the spec reporter emits empty lines as spec descriptions:

* Build successful.
* Running tests...
 Test
  Spec
   Runner
    ✓︎ collects "it" and "pending" in Describe groups
    ✓︎ collects "it" and "pending" with shared Describes
    ✓︎ filters using "only" modifier on "describe" block
    ✓︎ filters using "only" modifier on nested "describe" block
    ✓︎ filters using "only" modifier on "it" block
    ✓︎ supports async
 Test
  Spec
   Assertions
    String
     shouldContain
      ✓︎ accepts strings that contains substrings
      ✓︎ 
      ✓︎ rejects strings that does not contain substrings

However, I would argue that for smaller projects, the cardinality of the specs is sufficient to determine success:

* Build successful.
* Running tests...
 Test
  Spec
   Runner
    ✓︎ 
    ✓︎ 
    ✓︎ 
    ✓︎ 
    ✓︎ 
    ✓︎ 
 Test
  Spec
   Assertions
    String
     shouldContain
      ✓︎ 
      ✓︎ 
      ✓︎ 

And in the case of failure, the ordinality of the failure is sufficient to locate it:

* Building project in /Users/htmldrum/code/purescript/purescript-spec
Compiling Test.Spec.AssertionSpec
Compiling Test.Main
* Build successful.
* Running tests...
 Test
  Spec
   Runner
    ✓︎ collects "it" and "pending" in Describe groups
    ✓︎ collects "it" and "pending" with shared Describes
    ✓︎ filters using "only" modifier on "describe" block
    ✓︎ filters using "only" modifier on nested "describe" block
    ✓︎ filters using "only" modifier on "it" block
    ✓︎ supports async
 Test
  Spec
   Assertions
    String
     shouldContain
      ✓︎ accepts strings that contains substrings
      1) 
      ✓︎ rejects strings that does not contain substrings
...
57 passing
3 pending
1 failed

1) Test Spec Assertions String shouldContain 
  "foo" ∉ "fozbar"
* ERROR: Subcommand terminated with exit code 1

Thank you for your time.

This library is wonderful. I'd like to start using it with purerl. But the problem is, it has dependencies on specifics of JavaScript. There are still a couple of FFI modules here, and there's a dependency on purescript-js-timers. Are you interested in making this library able to be used by different backends? I don't want to suggest that you should maintain different backends though. That's kind of a big ask from me.

I have some ideas how this could be separated out, but I don't want you to have to maintain multiple libraries or have to jump through hoops to get things working. Assuming you're interested in allowing different backends, do you have any thoughts on how you'd like to make that possible?

If this is the first module you are updating, you might want to read the 0.7 migration guide.

I was thinking on adding describePar and describeParOnly. so Description is marked as parallel and during execution in _run we are run producers in parallel. it would be very useful in case ps-spec is used for running integration tests and some tests don't effect each other.

so if you have:

describe "delay" do
  it "can delay for 100ms" do
    delay $ Milliseconds $ 100.0 * 1.0
  it "can delay for 200ms" do
    delay $ Milliseconds $ 100.0 * 2.0
  it "can delay for 300ms" do
    delay $ Milliseconds $ 100.0 * 3.0
  it "can delay for 400ms" do
    delay $ Milliseconds $ 100.0 * 4.0
  it "can delay for 500ms" do
    delay $ Milliseconds $ 100.0 * 5.0

which takes 1500 ms, you can apply this change:

-describe "Foo" do
+describePar "Foo" do

and it will take 500ms.

Main issue would be logging from tests which are executing in parallel, as you would have no way to know from which one a logged message is coming from.

Version 0.11 introduced convenient re-exports in Test.Spec.Reporter that have consequently been removed (accidentally?). Personally, I would welcome these convenience exports as it means I can easily import bundled reporters downstream, i.e. import Test.Spec.Reporter (consoleReporter). I apologize for not letting you know about adding these changes when crafting #32, I simply forgot amidst the many other changes.

When there are failing tests suite should process.exit(1).

Hello,

Thanks for writing this library, it is very useful.
It would be very cool if it would be possible to get a diff between expected and tested values, like in PyTest (see the "Making use of context-sensitive comparisons" section)

If you have some state you want to test in multiple it blocks, instead of having to instantiate it multiple times, it would be nice to be able to do it once in the describe block for those specs. Ie:

describe "some state" do
  someRef <- readSTRef myRef
  someArray <- freeze mySTArray

  it "should have this property" do
    ...

  it "should have that property" do
    ...

Later is deprecated so the examples don't work anymore.
It needs to be replaced with

delay (Milliseconds 100.0)
let res = "Alligator"

I think. Maybe it should also be forked or something?

Add something like setUp and tearDown, something like hooks in hspec:
https://hspec.github.io/writing-specs.html

Should be much more beginner friendly. Start with the regular pulp test setup, step-by-step. Then introduce details and how to build using psc.

It should use the pure version to get results and translate to purescript-spec's [Group] which can be consumed by describe.

https://github.com/purescript/purescript-quickcheck/blob/master/src/Test/QuickCheck.purs#L66

It would be nice to have this documented on Pursuit if possible.

The Node.Process dependency is placing a hard dependency where you have to run the tests on nodejs. The issue with that is that you can't use something like phantomjs to run tests and test libraries that interact with the DOM.

How do you feel about going back to defining a custom PROCESS effect?

Sorry, but you forgot to actually export run' function. As you can see, there is only run function, that is exported:

module Test.Spec.Runner
       ( RunnerEffects
       , run
       , runSpec
       , runSpec'
       , defaultConfig
       , timeout
       , Config
       , TestEvents
       , Reporter
) where

I get the following build error with purs v0.13.0

Compiling Test.Spec.Reporter.Console
Error found:
in module Test.Spec.Reporter.Console
at bower_components/purescript-spec/src/Test/Spec/Reporter/Console.purs:49:28 - 49:39 (line 49, column 28 - line 49, column 39)

  Unknown value flushCrumbs

Would be nice if that worked instead 😀

Hi, I attempted to copy and paste the example from the purescript-spec documentation and run it under PureScript 0.12.5, but the code does not compile. Here is the error:

Error found:
in module Main
at src/Main.purs:35:8 - 49:51 (line 35, column 8 - line 49, column 51)

  Could not match type

    Effect

  with type

    Aff

while trying to match type Effect Unit
  with type Aff t0
while checking that expression (run [ consoleReporter
                                    ]
                               )
                               ((describe "purescript-spec") ((discard (...)) (\$__unused ->
                                                                                 ...
                                                                              )
                                                             )
                               )
  has type Aff t0
in value declaration main

where t0 is an unknown type

See https://github.com/purescript/documentation/blob/master/errors/TypesDoNotUnify.md for more information,
or to contribute content related to this error.

This is for the code here: https://purescript-spec.github.io/purescript-spec/#full-example, i.e.

module Main where

import Prelude

import Data.Time.Duration (Milliseconds(..))
import Effect (Effect)
import Effect.Aff (launchAff_, delay)
import Test.Spec (pending, describe, it)
import Test.Spec.Assertions (shouldEqual)
import Test.Spec.Reporter.Console (consoleReporter)
import Test.Spec.Runner (run)

main :: Effect Unit
main = launchAff_ $ run [consoleReporter] do
  describe "purescript-spec" do
    describe "Attributes" do
      it "awesome" do
        let isAwesome = true
        isAwesome `shouldEqual` true
      pending "feature complete"
    describe "Features" do
      it "runs in NodeJS" $ pure unit
      it "runs in the browser" $ pure unit
      it "supports streaming reporters" $ pure unit
      it "supports async specs" do
        res <- delay (Milliseconds 100.0) $> "Alligator"
        res `shouldEqual` "Alligator"
      it "is PureScript 0.12.x compatible" $ pure unit