Comments (10)
I just found that Git has a verbose
command that could help:
https://github.com/git/git/blob/master/t/test-lib-functions.sh#L652
from sharness.
Here are some places where it is used:
https://github.com/git/git/blob/master/t/t0020-crlf.sh#L112
https://github.com/git/git/blob/master/t/t1301-shared-repo.sh#L92
https://github.com/git/git/blob/master/t/t4022-diff-rewrite.sh#L23
https://github.com/git/git/blob/master/t/t4202-log.sh#L116
from sharness.
Hi. I want to dig into this. (I can provide PR with some guide)
I use sharness in https://github.com/albfan/bash-ini-parser/
and told contributors to mark test as failure (to describe something known not to work, then fix it and mark test as success.
I use test_cmp to make comparations
Problem is, only failed test are shown with diff error in test-suite.log
I want something like:
make check VERBOSE=failure
so any fail test or expected fail test shows the problem, so I can see in CI what is the problem described
from sharness.
By now my trick is to define verbose=t
and cat test-suite.log after pass test suite
https://github.com/albfan/bash-ini-parser/blob/master/.travis.yml#L7
So I can see the faulting diff https://travis-ci.org/albfan/bash-ini-parser/jobs/384044678#L536
from sharness.
@albfan yeah, a solution to this would be very welcome. I like the idea of a verbose=failure
mode.
I think the simplest way to implement this might be to have the output of each test sent to a different file and just cat the file if the test failed.
For example in Git the performance testing part of the test framework outputs the results of p7821-grep-engines-fixed.sh
in test-results/build_17cb5f85d06e82b7837a59064fd5fe61b44f9d12.p7821-grep-engines-fixed.X.times
where X is the test number in the test script.
from sharness.
Diving into it I realize all stuff is autotools. So I'm rereading docs https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html#Parallel-Test-Harness, and looking where to touch to have this behaviour
Can you explain detail steps to reproduce the git test we can follow to get this done?
from sharness.
After this fantastic answer
https://stackoverflow.com/a/21033699/848072
I came out with an implementation
https://github.com/albfan/bash-ini-parser/tree/show-fail-and-expected-fail-details
At that time, t0006-duplication.sh was a test explaining a detected error. It is already resolved on master
- define verbose=t as env var .travis.yml#L7
- tweak test-driver to detect failed test and expected failures to cat the log albfan/bash-ini-parser@cc92395
- define a test expected to success (but failing) t0006-duplication.sh
- mark test as expected failure in test suite (
XFAIL_TESTS
) albfan/bash-ini-parser@eaf4802
Yay! https://travis-ci.org/albfan/bash-ini-parser/jobs/384330600#L534
For me this trick works, so I would close this as is totally autotools related
About step 1, as test-driver accepts
test-driver --test-name=NAME --log-file=PATH --trs-file=PATH
[--expect-failure={yes|no}] [--color-tests={yes|no}]
[--enable-hard-errors={yes|no}] [--]
TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS]
autootols should provide a way to pass those arguments
Did you ever push a patch on autotools?
from sharness.
@albfan no I didn't sent any patch to autotools. Sorry.
I am happy that you found a way to do it that works for you, but I think something could still be done in Sharness so I will keep this open.
from sharness.
If I can help just tell me, don't know how to reproduce the test you mention to get things into sharness
from sharness.
I don't think this belongs in sharness: this is something the TAP driver should deal with.
What we could do is make the output of --verbose
be TAP-friendly so the TAP driver can show the comments on failure.
But that's a different feature request.
from sharness.
Related Issues (20)
- SHARNESS_TEST_SRCDIR and extensions HOT 3
- Sharness self-test fails HOT 2
- Can't locate object method "follower" via package "IO::Pty"
- test_expect_failure test 2 = 2 does not fail with prove HOT 4
- trash dir as home dir when run on Windows HOT 1
- testing scripts with sourced functions HOT 1
- Make a release v1.2 HOT 1
- Usage with `prove` would need more documentation HOT 6
- Calling functions from Git submodule fails with v1.2.0 release (Regression) HOT 8
- Let --help output command line options
- add --tee option from git test-lib.sh HOT 1
- sharness.sh is executable? HOT 3
- Release tarballs are not signed HOT 4
- Clarify purpose of "aggregate-results.sh" HOT 12
- Cleanup actions are only run if no tests failed HOT 8
- Additional test function for string comparison? HOT 5
- Document usage of "trash" directory HOT 2
- Document how to release a new Sharness version HOT 2
- Line feed regression HOT 3
- Lots of dangling branches HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sharness.