Comments (4)
I think it is indeed expected that prove
will always consider tests marked with test_expect_failure as passing whatever the actual results are. It's only when there is an unexpected failure, that a test is considered as failed.
By the way, even without prove
the test was considered as passing ("ok 4"), so it would be strange if prove
disagreed.
from sharness.
I'm just watching https://www.youtube.com/watch?v=0Gf5eGspG4s btw. =)
Oh ok, ok 4
, but your tool gives some extra context.
It's only when there is an unexpected failure, that a test is considered as failed.
So I'm understanding this as an expected failure is considered broken (not failing), an unexpected failure is considered a fail.
from sharness.
Nice that you are watching one of my presentations :-)
prove
is not a tool developed by Sharness or Git. It's a Perl tool for the TAP (Test Anything Protocol) that existed before Git and that we are just taking advantage of.
So yeah, there is some extra context, but prove
interprets it only as a "TODO" thing, displaying TODO passed: 4
. That's what the Test Anything Protocol allows.
prove
doesn't make the "broken" vs "failing" distinction, but with Sharness there is an aggregate-results.sh
script that makes this distinction, and that can be run automatically using for example a Makefile such as the one provided in Sharness. So you can use this script (and maybe the provided Makefile too), if you really want the distinction and if the TODO passed: 4
mentions are not enough.
from sharness.
For example if you clone the Sharness repo and run make
in the test
directory, you should get something like:
$ make
rm -f -r test-results
make aggregate-results-and-cleanup
make[1]: Entering directory '/home/ubuntu/git/sharness/test'
*** sharness.t ***
ok 1 - sourcing sharness succeeds
ok 2 - success is reported like this
not ok 3 - pretend we have a known breakage # TODO known breakage
ok 4 - pretend we have a fully passing test suite
ok 5 - pretend we have a partially passing test suite
...
make aggregate-results
make[2]: Entering directory '/home/ubuntu/git/sharness/test'
for f in test-results/*.counts; do \
echo "$f"; \
done | '/bin/sh' 'aggregate-results.sh'
fixed 0
success 31
failed 0
broken 2
total 37
...
from sharness.
Related Issues (20)
- SHARNESS_TEST_SRCDIR and extensions HOT 3
- Sharness self-test fails HOT 2
- Can't locate object method "follower" via package "IO::Pty"
- trash dir as home dir when run on Windows HOT 1
- testing scripts with sourced functions HOT 1
- Make a release v1.2 HOT 1
- Usage with `prove` would need more documentation HOT 6
- Calling functions from Git submodule fails with v1.2.0 release (Regression) HOT 8
- make test fails with unliked SHELL_PATH HOT 9
- add --tee option from git test-lib.sh HOT 1
- sharness.sh is executable? HOT 3
- Release tarballs are not signed HOT 4
- Clarify purpose of "aggregate-results.sh" HOT 12
- Cleanup actions are only run if no tests failed HOT 8
- Additional test function for string comparison? HOT 5
- Document usage of "trash" directory HOT 2
- Document how to release a new Sharness version HOT 2
- Line feed regression HOT 3
- Lots of dangling branches HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sharness.