Comments (9)
@Minoru: Ha, I wish I was in Feynman's class! Your tracking down the specific tasty commit was the biggest clue, followed by some hunches and experimentation, and then a too-long period of confusion until I figured out that the d5286ee
commit was not actually in the tasty 1.4 branch. I'm glad this fixes your issues, and thanks again for the reproduction and initial research!
@nomeata: I'm glad to help!
from tasty-expected-failure.
Forgot to tag them; fixed.
from tasty-expected-failure.
I lost track of the things related to exceptions and expected-failure
. @kquick is the most recent expert, maybe he can advise?
@Minoru , did you see #21 – maybe that applies to you as well?
from tasty-expected-failure.
This appears to be a different issue. I changed the "timeout offset" to 100ms, and my reproducer still misbehaves as described above. Hoping @kquick has some more insight into this.
from tasty-expected-failure.
You are correct, this is not issue #21.
Thank you for the minimal reproduction and for tracking this down to a specific commit in tasty!
The error you are encountering is actually a new behavior in tasty
itself: it is trying to force evaluation of the error report within the context of its timeout in case the show
diverges (does not finish). If you implement this test in tasty-hspec
without using tasty-expected-failure
, the same Exception: Prelude.undefined
message will be shown there. The reason is that tasty
is trying to show the failure, which includes the values that caused the failure, but trying to show
an undefined
causes an exception.
The situation gets even more interesting at this point. Looking at issue 280 in tasty
, the issue was closed with commit d5286ee
, which has protection for the situation where the failure reporting would throw an exception (which would have avoided this issue 24). However a day later, the fcfccc8
commit was made, which has the forcing but does not have the protections, and the d5286ee
commit is reported as orphaned. It's not clear why feuerbach decided to remove the protections in the final version.
[It's hard to actually add any checking for this to tasty itself... since all of its tests must succeed. Too bad it can't expect failures to check them successfully. :-) ]
The normal operation for tasty-expected-failure
is:
- if there is (unexpected!) success of the test, it shows "(unexpected success)" and increments the failed tests count
- if the test fails, it uses tasty's normal failure result display to show FAIL, the values involved, then notes that this is expected and the failed tests count is not incremented.
The problem here is that when the test fails (as expected!) the value is shown, and if the show
operation throws an exception (e.g. by trying to show
an undefined
) the old tasty
would show Exception: Prelude.undefined
, but it would do that after checking the test results, so the failure count is not incremented; in tasty
1.4
or later, the show
is forced after the test itself (and after tasty-expected-failure
says it's OK) but before the test post-processing completes, so the exception is not only shown but it increments the failure counter down in tasty
itself.
An interesting way to see this is to update your minimal failure with tasty
1.3
to add testCase "oops" (3 @?= 4)
and run it with tasty
1.3
which will cause a failure and therefore the display: you will see the Exception: Prelude.undefined
message now because there's another failing test that causes all test results to be shown.
The fix for this is a little byzantine. Essentially the fix in #25 updates tasty-expected-failure
to also force the display of the failure reason, which causes the undefined
exception to occur within tasty-expected-failure
. The behavior I added for catching exceptions therefore catches this exception.. and reports back to tasty that it got an expected failure due to an exception. This latter failure is reported instead of the original failure, but it doesn't cause a failure in tasty
's new post-processing because the text of the failure report is not an attempt to show
the undefined
, but instead the text of the exception generated by that attempt inside tasty-expected-failure
.
Thus, tasty-expected-failure
correctly reports that the test failed. It reports the wrong reason for that failure, but since the value will only be attempted to be shown (causing the reported failure) if the underlying test actually failed, the outcome is consistent with the intent, even though it's mis-reported. I could have modified the fix so that it indicated that both the original test failed and the show
of the failure also failed, but I don't think it would have been much more helpful.
from tasty-expected-failure.
Wow, thanks for that analysis and fix! I'm grateful for your help here!
from tasty-expected-failure.
@kquick This is mind-blowing. Thank you very much for figuring it out! I verified and can confirm that #25 fixes the problem in my "real" test suite, not just the reproducer.
Out of curiosity: how did you track down the cause? Was it by Feynman's algorithm, or is there some easier ways to do it? Did you just wrap tasty's internals in a catch
block and then narrowed its scope to figure out where the exception comes from?
from tasty-expected-failure.
#25 fixed this issue, so I'm closing it. I also notice that the fix is already released as part of tasty-expected-failure 0.12.2 — awesome! Thank you both for the prompt fix and the release!
from tasty-expected-failure.
@nomeata, by the way, this repo lacks 0.12.1 and 0.12.2 Git tags — did you forget to push them, or simply stopped tagging releases in Git history?
from tasty-expected-failure.
Related Issues (12)
- Exceptions aren't counted as expected failures with tasty-golden HOT 1
- base 4.11
- Omit coverage for `expectFail`ed tests HOT 1
- ignore at runtime HOT 1
- Test failure in 0.12 HOT 2
- Add `expectAnyFail`
- expected-fail-hh-tests failed in 0.12.1 HOT 12
- expectFail inconsistent with exceptions HOT 6
- Exception throwing tests still fail HOT 4
- `expectFail` and timed-out tests HOT 2
- setup/teardown is executed for ignored tests? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tasty-expected-failure.