Incompatibility with tasty-1.4 about tasty-expected-failure HOT 9 CLOSED

@Minoru: Ha, I wish I was in Feynman's class! Your tracking down the specific tasty commit was the biggest clue, followed by some hunches and experimentation, and then a too-long period of confusion until I figured out that the d5286ee commit was not actually in the tasty 1.4 branch. I'm glad this fixes your issues, and thanks again for the reproduction and initial research!

@nomeata: I'm glad to help!

from tasty-expected-failure.

Forgot to tag them; fixed.

from tasty-expected-failure.

I lost track of the things related to exceptions and expected-failure. @kquick is the most recent expert, maybe he can advise?

@Minoru , did you see #21 – maybe that applies to you as well?

from tasty-expected-failure.

This appears to be a different issue. I changed the "timeout offset" to 100ms, and my reproducer still misbehaves as described above. Hoping @kquick has some more insight into this.

from tasty-expected-failure.

You are correct, this is not issue #21.

Thank you for the minimal reproduction and for tracking this down to a specific commit in tasty!

The error you are encountering is actually a new behavior in tasty itself: it is trying to force evaluation of the error report within the context of its timeout in case the show diverges (does not finish). If you implement this test in tasty-hspec without using tasty-expected-failure, the same Exception: Prelude.undefined message will be shown there. The reason is that tasty is trying to show the failure, which includes the values that caused the failure, but trying to show an undefined causes an exception.

The situation gets even more interesting at this point. Looking at issue 280 in tasty, the issue was closed with commit d5286ee, which has protection for the situation where the failure reporting would throw an exception (which would have avoided this issue 24). However a day later, the fcfccc8 commit was made, which has the forcing but does not have the protections, and the d5286ee commit is reported as orphaned. It's not clear why feuerbach decided to remove the protections in the final version.

[It's hard to actually add any checking for this to tasty itself... since all of its tests must succeed. Too bad it can't expect failures to check them successfully. :-) ]

The normal operation for tasty-expected-failure is:

• if there is (unexpected!) success of the test, it shows "(unexpected success)" and increments the failed tests count
• if the test fails, it uses tasty's normal failure result display to show FAIL, the values involved, then notes that this is expected and the failed tests count is not incremented.

The problem here is that when the test fails (as expected!) the value is shown, and if the show operation throws an exception (e.g. by trying to show an undefined) the old tasty would show Exception: Prelude.undefined, but it would do that after checking the test results, so the failure count is not incremented; in tasty 1.4 or later, the show is forced after the test itself (and after tasty-expected-failure says it's OK) but before the test post-processing completes, so the exception is not only shown but it increments the failure counter down in tasty itself.

An interesting way to see this is to update your minimal failure with tasty 1.3 to add testCase "oops" (3 @?= 4) and run it with tasty 1.3 which will cause a failure and therefore the display: you will see the Exception: Prelude.undefined message now because there's another failing test that causes all test results to be shown.

The fix for this is a little byzantine. Essentially the fix in #25 updates tasty-expected-failure to also force the display of the failure reason, which causes the undefined exception to occur within tasty-expected-failure. The behavior I added for catching exceptions therefore catches this exception.. and reports back to tasty that it got an expected failure due to an exception. This latter failure is reported instead of the original failure, but it doesn't cause a failure in tasty's new post-processing because the text of the failure report is not an attempt to show the undefined, but instead the text of the exception generated by that attempt inside tasty-expected-failure.

Thus, tasty-expected-failure correctly reports that the test failed. It reports the wrong reason for that failure, but since the value will only be attempted to be shown (causing the reported failure) if the underlying test actually failed, the outcome is consistent with the intent, even though it's mis-reported. I could have modified the fix so that it indicated that both the original test failed and the show of the failure also failed, but I don't think it would have been much more helpful.

from tasty-expected-failure.

Wow, thanks for that analysis and fix! I'm grateful for your help here!

from tasty-expected-failure.

@kquick This is mind-blowing. Thank you very much for figuring it out! I verified and can confirm that #25 fixes the problem in my "real" test suite, not just the reproducer.

Out of curiosity: how did you track down the cause? Was it by Feynman's algorithm, or is there some easier ways to do it? Did you just wrap tasty's internals in a catch block and then narrowed its scope to figure out where the exception comes from?

from tasty-expected-failure.

#25 fixed this issue, so I'm closing it. I also notice that the fix is already released as part of tasty-expected-failure 0.12.2 — awesome! Thank you both for the prompt fix and the release!

from tasty-expected-failure.

@nomeata, by the way, this repo lacks 0.12.1 and 0.12.2 Git tags — did you forget to push them, or simply stopped tagging releases in Git history?

from tasty-expected-failure.