Git Product home page Git Product logo

Comments (12)

nomeata avatar nomeata commented on August 19, 2024 1

Yes, thinking the same as @berberman . Maybe worth another try to convince Roman to make expected test failures a first-class feature.

Or, alternatively, at least expose control over catching timeouts

from tasty-expected-failure.

felixonmars avatar felixonmars commented on August 19, 2024 1

@felixonmars, If you have the ability to change this value to 10000 in your local copy and confirm that the tests pass and are stable, that would be helpful to confirm a PR to make this corresponding change.

I can confirm that it makes the test stable here on all my build machines.

from tasty-expected-failure.

kquick avatar kquick commented on August 19, 2024 1

I've gotten rid of the ugly timeout hack and used normal Tasty options management to convince Tasty to let this package do the timeout management rather than Tasty's core (#26). This is working for me, and I'd love to get confirmation from some of the other folks encountering issues (@felixonmars and @mcwitt, especially).

from tasty-expected-failure.

felixonmars avatar felixonmars commented on August 19, 2024 1

@kquick The tests pass here with your PR. Many thanks!

from tasty-expected-failure.

nomeata avatar nomeata commented on August 19, 2024

That’s a duplicate of #19, isn’t it? I thought that got fixed in #20. @felixonmars, what is your tasty version? @kquick, care to investigate?

from tasty-expected-failure.

felixonmars avatar felixonmars commented on August 19, 2024

It looks quite different than #19 though, and the patch in #20 has already been applied. My tasty is at 1.3.1, and this issue is actually flaky (it doesn't reproduce in a second try).

from tasty-expected-failure.

kquick avatar kquick commented on August 19, 2024

Sigh, no bad hack goes unpunished. :-/

This is different than the patch in #20. The problem is that the test timing (supporting tasty's mkTimeout) is implemented in core tasty in a location that cannot be affected by the expected failure wrappers. The hack introduced here is to add another timeout inside the wrapper, where this timeout is set to fire slightly faster than the tasty timeout to cause this wrapper's timeout to fire and be absorbed by the wrapper to conclude this test before the internal tasty timeout fires and marks the test as a failure. This is a very egregious hack and I hate it, but it's the only thing I could come up with given that this module isn't integrated into the core tasty package.

Something like this is needed to ensure that any test that is marked expectFail includes catching a timeout as a failure. Otherwise a test marked expectFail might be unstable in that it would succeed if it had a normal failure within the timeout period, but be reported as a failure if the timeout period was exceeded.

The "faster" timeout is done here: https://github.com/nomeata/tasty-expected-failure/blob/master/Test/Tasty/ExpectedFailure.hs#L44

I confirmed that if I run this test locally for multiple iterations that the original value of 2000 (= 2ms) does occasionally result in the failure above for me. Increasing this to 4000 (= 4ms) seems to be stable and avoid the issue. My current thinking is to actually bump this to 10000 (= 10ms) so that the user's test is 10ms faster than they had configured, but this is hopefully still much smaller than the user's resolution would typically be but longer than the normal scheduling delays of delivering timeouts to ensure that the local timeout fires before the tasty timeout.

@felixonmars, If you have the ability to change this value to 10000 in your local copy and confirm that the tests pass and are stable, that would be helpful to confirm a PR to make this corresponding change.

@nomeata , if you have any other ideas on how this could be done I would be glad to get rid of this hack.

from tasty-expected-failure.

berberman avatar berberman commented on August 19, 2024

Hmm I think the whole wrapper is fairly a hacky workaround, just because the support of known-broken tests should be a primitive functionality in tasty, coupling with the internal implementation. Maybe hacks exposed by this issue can make it more plausible that tasty should accept this part.

from tasty-expected-failure.

felixonmars avatar felixonmars commented on August 19, 2024

Not sure what changed in the environment (the last two updates are data-fix 0.3.1 and code-page 0.2.1), but now both timeout tests are failing here:

(Note that the 2000 -> 10000 change is already applied).

Running 2 test suites...
Test suite expected-fail-tests: RUNNING...
Expected Failures
  clearly good:                  OK
  clearly bad:                   FAIL (expected)
    tests/TestExpectedFails.hs:17:
    expected: 2
     but got: 3 (expected failure)
  two wrongs make a right:       OK (unexpected) (expected)
     (unexpected success) (expected failure)
  throws failure:                FAIL (expected)
    Exception: user error (bad) (expected failure)
  throws error:                  FAIL (expected)
    Exception: also bad
    CallStack (from HasCallStack):
      error, called at tests/TestExpectedFails.hs:25:44 in main:Main (expected failure)
  takes too long:                TIMEOUT (1.31s)
    Timed out after 1s
  hello:                         FAIL (expected)
    Exception: not golden
    CallStack (from HasCallStack):
      error, called at tests/TestExpectedFails.hs:29:64 in main:Main (expected failure)
  this is expected to fail:      FAIL (expected)
    tests/TestExpectedFails.hs:33:
    expected: ""
     but got: "Hello, world!" (expected failure)
  this is also expected to fail: FAIL (expected)
    Exception: Prelude.undefined
    CallStack (from HasCallStack):
      error, called at libraries/base/GHC/Err.hs:79:14 in base:GHC.Err
      undefined, called at tests/TestExpectedFails.hs:49:26 in main:Main (expected failure)

1 out of 9 tests failed (1.31s)
Test suite expected-fail-tests: FAIL
Test suite logged to:
dist/test/tasty-expected-failure-0.12.2-expected-fail-tests.log
Test suite expected-fail-hh-tests: RUNNING...
Expected Hedgehog Failures
  good:                                OK
      ✓ good passed 100 tests.
  rarely good:                         FAIL (expected)
      ✗ rarely good failed at tests/TestExpectedFailsHH.hs:22:7
        after 22 tests and 7 shrinks.

           ┏━━ tests/TestExpectedFailsHH.hs ━━━
        16 ┃ main = defaultMain $
        17 ┃   localOption (mkTimeout 1000000) $  -- 1s
        18 ┃   testGroup "Expected Hedgehog Failures" $
        19 ┃   [ testProperty "good" $ property $ success
        20 ┃   , expectFail $ testProperty "rarely good" $ property $ do
        21 ┃       xs <- forAll $ Gen.list (Range.linear 0 10) Gen.alpha
           ┃       │ "ab"
        22 ┃       reverse xs === xs
           ┃       ^^^^^^^^^^^^^^^^^
           ┃       │ ━━━ Failed (- lhs) (+ rhs) ━━━
           ┃       │ - "ba"
           ┃       │ + "ab"
        23 ┃
        24 ┃   -- n.b. uncomment this to observe the results of a test that was
        25 ┃   -- expected to fail but actually passes.
        26 ┃   -- , expectFail $ testProperty "surprisingly good" $ property $ success
        27 ┃
        28 ┃   , expectFail $ testProperty "giving up" $ property $ discard
        29 ┃
        30 ┃   , expectFail $ expectFail $ testProperty "the failure of a failure is my good" $
        31 ┃     property $ success
        32 ┃
        33 ┃   , expectFail $ testProperty "throws failure" $
        34 ┃     property $ fail "bad"
        35 ┃
        36 ┃   , expectFail $ testProperty "too slow" $
        37 ┃     property $ do
        38 ┃       liftIO $ threadDelay 2000000
        39 ┃       success
        40 ┃   ]

        This failure can be reproduced by running:
        > recheck (Size 21) (Seed 13860484526646270131 2110852822944260057) rarely good

    Use '--hedgehog-replay "Size 21 Seed 13860484526646270131 2110852822944260057"' to reproduce.
     (expected failure)
  giving up:                           FAIL (expected)
      ⚐ giving up gave up after 100 discards, passed 0 tests. (expected failure)
  the failure of a failure is my good: OK (unexpected) (expected)
      ✓ the failure of a failure is my good passed 100 tests. (unexpected success) (expected failure)
  throws failure:                      FAIL (expected)
      ✗ throws failure failed
        after 1 test.

        bad

        This failure can be reproduced by running:
        > recheck (Size 0) (Seed 420030084359648001 290599314569365167) throws failure

    Use '--hedgehog-replay "Size 0 Seed 420030084359648001 290599314569365167"' to reproduce.
     (expected failure)
  too slow:                            TIMEOUT (1.31s)
    Timed out after 1s

1 out of 6 tests failed (1.31s)
Test suite expected-fail-hh-tests: FAIL
Test suite logged to:
dist/test/tasty-expected-failure-0.12.2-expected-fail-hh-tests.log
0 of 2 test suites (0 of 2 test cases) passed.

from tasty-expected-failure.

kquick avatar kquick commented on August 19, 2024

I will take a look to see if there's a way that tasty could be modified in an acceptable manner that would make this less hacky, but I won't have time until perhaps this weekend.

from tasty-expected-failure.

nomeata avatar nomeata commented on August 19, 2024

Thanks a lot!

from tasty-expected-failure.

kquick avatar kquick commented on August 19, 2024

Fix was merged. Close this issue? (I can't)

from tasty-expected-failure.

Related Issues (12)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.