test_expect_failure test 2 = 2 does not fail with prove about sharness HOT 4 CLOSED

I think it is indeed expected that prove will always consider tests marked with test_expect_failure as passing whatever the actual results are. It's only when there is an unexpected failure, that a test is considered as failed.

By the way, even without prove the test was considered as passing ("ok 4"), so it would be strange if prove disagreed.

from sharness.

I'm just watching https://www.youtube.com/watch?v=0Gf5eGspG4s btw. =)

Oh ok, ok 4, but your tool gives some extra context.

It's only when there is an unexpected failure, that a test is considered as failed.

So I'm understanding this as an expected failure is considered broken (not failing), an unexpected failure is considered a fail.

from sharness.

Nice that you are watching one of my presentations :-)

prove is not a tool developed by Sharness or Git. It's a Perl tool for the TAP (Test Anything Protocol) that existed before Git and that we are just taking advantage of.

So yeah, there is some extra context, but prove interprets it only as a "TODO" thing, displaying TODO passed: 4. That's what the Test Anything Protocol allows.

prove doesn't make the "broken" vs "failing" distinction, but with Sharness there is an aggregate-results.sh script that makes this distinction, and that can be run automatically using for example a Makefile such as the one provided in Sharness. So you can use this script (and maybe the provided Makefile too), if you really want the distinction and if the TODO passed: 4 mentions are not enough.

from sharness.

For example if you clone the Sharness repo and run make in the test directory, you should get something like:

$ make
rm -f -r test-results
make aggregate-results-and-cleanup
make[1]: Entering directory '/home/ubuntu/git/sharness/test'
*** sharness.t ***
ok 1 - sourcing sharness succeeds
ok 2 - success is reported like this
not ok 3 - pretend we have a known breakage # TODO known breakage
ok 4 - pretend we have a fully passing test suite
ok 5 - pretend we have a partially passing test suite
...
make aggregate-results
make[2]: Entering directory '/home/ubuntu/git/sharness/test'
for f in test-results/*.counts; do \
        echo "$f"; \
done | '/bin/sh' 'aggregate-results.sh'
fixed   0
success 31
failed  0
broken  2
total   37
...

from sharness.