Comments (12)
Yes, thinking the same as @berberman . Maybe worth another try to convince Roman to make expected test failures a first-class feature.
Or, alternatively, at least expose control over catching timeouts
from tasty-expected-failure.
@felixonmars, If you have the ability to change this value to 10000 in your local copy and confirm that the tests pass and are stable, that would be helpful to confirm a PR to make this corresponding change.
I can confirm that it makes the test stable here on all my build machines.
from tasty-expected-failure.
I've gotten rid of the ugly timeout hack and used normal Tasty options management to convince Tasty to let this package do the timeout management rather than Tasty's core (#26). This is working for me, and I'd love to get confirmation from some of the other folks encountering issues (@felixonmars and @mcwitt, especially).
from tasty-expected-failure.
@kquick The tests pass here with your PR. Many thanks!
from tasty-expected-failure.
That’s a duplicate of #19, isn’t it? I thought that got fixed in #20. @felixonmars, what is your tasty
version? @kquick, care to investigate?
from tasty-expected-failure.
It looks quite different than #19 though, and the patch in #20 has already been applied. My tasty
is at 1.3.1
, and this issue is actually flaky (it doesn't reproduce in a second try).
from tasty-expected-failure.
Sigh, no bad hack goes unpunished. :-/
This is different than the patch in #20. The problem is that the test timing (supporting tasty's mkTimeout
) is implemented in core tasty in a location that cannot be affected by the expected failure wrappers. The hack introduced here is to add another timeout inside the wrapper, where this timeout is set to fire slightly faster than the tasty timeout to cause this wrapper's timeout to fire and be absorbed by the wrapper to conclude this test before the internal tasty timeout fires and marks the test as a failure. This is a very egregious hack and I hate it, but it's the only thing I could come up with given that this module isn't integrated into the core tasty package.
Something like this is needed to ensure that any test that is marked expectFail
includes catching a timeout as a failure. Otherwise a test marked expectFail
might be unstable in that it would succeed if it had a normal failure within the timeout period, but be reported as a failure if the timeout period was exceeded.
The "faster" timeout is done here: https://github.com/nomeata/tasty-expected-failure/blob/master/Test/Tasty/ExpectedFailure.hs#L44
I confirmed that if I run this test locally for multiple iterations that the original value of 2000 (= 2ms) does occasionally result in the failure above for me. Increasing this to 4000 (= 4ms) seems to be stable and avoid the issue. My current thinking is to actually bump this to 10000 (= 10ms) so that the user's test is 10ms faster than they had configured, but this is hopefully still much smaller than the user's resolution would typically be but longer than the normal scheduling delays of delivering timeouts to ensure that the local timeout fires before the tasty timeout.
@felixonmars, If you have the ability to change this value to 10000 in your local copy and confirm that the tests pass and are stable, that would be helpful to confirm a PR to make this corresponding change.
@nomeata , if you have any other ideas on how this could be done I would be glad to get rid of this hack.
from tasty-expected-failure.
Hmm I think the whole wrapper is fairly a hacky workaround, just because the support of known-broken tests should be a primitive functionality in tasty
, coupling with the internal implementation. Maybe hacks exposed by this issue can make it more plausible that tasty
should accept this part.
from tasty-expected-failure.
Not sure what changed in the environment (the last two updates are data-fix 0.3.1 and code-page 0.2.1), but now both timeout tests are failing here:
(Note that the 2000 -> 10000 change is already applied).
Running 2 test suites...
Test suite expected-fail-tests: RUNNING...
Expected Failures
clearly good: OK
clearly bad: FAIL (expected)
tests/TestExpectedFails.hs:17:
expected: 2
but got: 3 (expected failure)
two wrongs make a right: OK (unexpected) (expected)
(unexpected success) (expected failure)
throws failure: FAIL (expected)
Exception: user error (bad) (expected failure)
throws error: FAIL (expected)
Exception: also bad
CallStack (from HasCallStack):
error, called at tests/TestExpectedFails.hs:25:44 in main:Main (expected failure)
takes too long: TIMEOUT (1.31s)
Timed out after 1s
hello: FAIL (expected)
Exception: not golden
CallStack (from HasCallStack):
error, called at tests/TestExpectedFails.hs:29:64 in main:Main (expected failure)
this is expected to fail: FAIL (expected)
tests/TestExpectedFails.hs:33:
expected: ""
but got: "Hello, world!" (expected failure)
this is also expected to fail: FAIL (expected)
Exception: Prelude.undefined
CallStack (from HasCallStack):
error, called at libraries/base/GHC/Err.hs:79:14 in base:GHC.Err
undefined, called at tests/TestExpectedFails.hs:49:26 in main:Main (expected failure)
1 out of 9 tests failed (1.31s)
Test suite expected-fail-tests: FAIL
Test suite logged to:
dist/test/tasty-expected-failure-0.12.2-expected-fail-tests.log
Test suite expected-fail-hh-tests: RUNNING...
Expected Hedgehog Failures
good: OK
✓ good passed 100 tests.
rarely good: FAIL (expected)
✗ rarely good failed at tests/TestExpectedFailsHH.hs:22:7
after 22 tests and 7 shrinks.
┏━━ tests/TestExpectedFailsHH.hs ━━━
16 ┃ main = defaultMain $
17 ┃ localOption (mkTimeout 1000000) $ -- 1s
18 ┃ testGroup "Expected Hedgehog Failures" $
19 ┃ [ testProperty "good" $ property $ success
20 ┃ , expectFail $ testProperty "rarely good" $ property $ do
21 ┃ xs <- forAll $ Gen.list (Range.linear 0 10) Gen.alpha
┃ │ "ab"
22 ┃ reverse xs === xs
┃ ^^^^^^^^^^^^^^^^^
┃ │ ━━━ Failed (- lhs) (+ rhs) ━━━
┃ │ - "ba"
┃ │ + "ab"
23 ┃
24 ┃ -- n.b. uncomment this to observe the results of a test that was
25 ┃ -- expected to fail but actually passes.
26 ┃ -- , expectFail $ testProperty "surprisingly good" $ property $ success
27 ┃
28 ┃ , expectFail $ testProperty "giving up" $ property $ discard
29 ┃
30 ┃ , expectFail $ expectFail $ testProperty "the failure of a failure is my good" $
31 ┃ property $ success
32 ┃
33 ┃ , expectFail $ testProperty "throws failure" $
34 ┃ property $ fail "bad"
35 ┃
36 ┃ , expectFail $ testProperty "too slow" $
37 ┃ property $ do
38 ┃ liftIO $ threadDelay 2000000
39 ┃ success
40 ┃ ]
This failure can be reproduced by running:
> recheck (Size 21) (Seed 13860484526646270131 2110852822944260057) rarely good
Use '--hedgehog-replay "Size 21 Seed 13860484526646270131 2110852822944260057"' to reproduce.
(expected failure)
giving up: FAIL (expected)
⚐ giving up gave up after 100 discards, passed 0 tests. (expected failure)
the failure of a failure is my good: OK (unexpected) (expected)
✓ the failure of a failure is my good passed 100 tests. (unexpected success) (expected failure)
throws failure: FAIL (expected)
✗ throws failure failed
after 1 test.
bad
This failure can be reproduced by running:
> recheck (Size 0) (Seed 420030084359648001 290599314569365167) throws failure
Use '--hedgehog-replay "Size 0 Seed 420030084359648001 290599314569365167"' to reproduce.
(expected failure)
too slow: TIMEOUT (1.31s)
Timed out after 1s
1 out of 6 tests failed (1.31s)
Test suite expected-fail-hh-tests: FAIL
Test suite logged to:
dist/test/tasty-expected-failure-0.12.2-expected-fail-hh-tests.log
0 of 2 test suites (0 of 2 test cases) passed.
from tasty-expected-failure.
I will take a look to see if there's a way that tasty could be modified in an acceptable manner that would make this less hacky, but I won't have time until perhaps this weekend.
from tasty-expected-failure.
Thanks a lot!
from tasty-expected-failure.
Fix was merged. Close this issue? (I can't)
from tasty-expected-failure.
Related Issues (12)
- Add `expectAnyFail`
- Exception throwing tests still fail HOT 4
- `expectFail` and timed-out tests HOT 2
- setup/teardown is executed for ignored tests? HOT 5
- Exceptions aren't counted as expected failures with tasty-golden HOT 1
- base 4.11
- Omit coverage for `expectFail`ed tests HOT 1
- ignore at runtime HOT 1
- Test failure in 0.12 HOT 2
- expectFail inconsistent with exceptions HOT 6
- Incompatibility with tasty-1.4 HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tasty-expected-failure.