semi-verbose about sharness HOT 10 CLOSED

I just found that Git has a verbose command that could help:

https://github.com/git/git/blob/master/t/test-lib-functions.sh#L652

from sharness.

Here are some places where it is used:

https://github.com/git/git/blob/master/t/t0020-crlf.sh#L112
https://github.com/git/git/blob/master/t/t1301-shared-repo.sh#L92
https://github.com/git/git/blob/master/t/t4022-diff-rewrite.sh#L23
https://github.com/git/git/blob/master/t/t4202-log.sh#L116

from sharness.

Hi. I want to dig into this. (I can provide PR with some guide)

I use sharness in https://github.com/albfan/bash-ini-parser/

and told contributors to mark test as failure (to describe something known not to work, then fix it and mark test as success.

I use test_cmp to make comparations

Problem is, only failed test are shown with diff error in test-suite.log

https://travis-ci.org/albfan/bash-ini-parser/builds/383820058?utm_source=github_status&utm_medium=notification

I want something like:

make check VERBOSE=failure

so any fail test or expected fail test shows the problem, so I can see in CI what is the problem described

from sharness.

By now my trick is to define verbose=t and cat test-suite.log after pass test suite

https://github.com/albfan/bash-ini-parser/blob/master/.travis.yml#L7

So I can see the faulting diff https://travis-ci.org/albfan/bash-ini-parser/jobs/384044678#L536

from sharness.

@albfan yeah, a solution to this would be very welcome. I like the idea of a verbose=failure mode.

I think the simplest way to implement this might be to have the output of each test sent to a different file and just cat the file if the test failed.

For example in Git the performance testing part of the test framework outputs the results of p7821-grep-engines-fixed.sh in test-results/build_17cb5f85d06e82b7837a59064fd5fe61b44f9d12.p7821-grep-engines-fixed.X.times where X is the test number in the test script.

from sharness.

Diving into it I realize all stuff is autotools. So I'm rereading docs https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html#Parallel-Test-Harness, and looking where to touch to have this behaviour

Can you explain detail steps to reproduce the git test we can follow to get this done?

from sharness.

After this fantastic answer

https://stackoverflow.com/a/21033699/848072

I came out with an implementation

https://github.com/albfan/bash-ini-parser/tree/show-fail-and-expected-fail-details

At that time, t0006-duplication.sh was a test explaining a detected error. It is already resolved on master

1. define verbose=t as env var .travis.yml#L7
2. tweak test-driver to detect failed test and expected failures to cat the log albfan/bash-ini-parser@cc92395
3. define a test expected to success (but failing) t0006-duplication.sh
4. mark test as expected failure in test suite (XFAIL_TESTS) albfan/bash-ini-parser@eaf4802

Yay! https://travis-ci.org/albfan/bash-ini-parser/jobs/384330600#L534

For me this trick works, so I would close this as is totally autotools related

About step 1, as test-driver accepts

  test-driver --test-name=NAME --log-file=PATH --trs-file=PATH
              [--expect-failure={yes|no}] [--color-tests={yes|no}]
              [--enable-hard-errors={yes|no}] [--]
              TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS]

autootols should provide a way to pass those arguments

Did you ever push a patch on autotools?

from sharness.

@albfan no I didn't sent any patch to autotools. Sorry.

I am happy that you found a way to do it that works for you, but I think something could still be done in Sharness so I will keep this open.

from sharness.

If I can help just tell me, don't know how to reproduce the test you mention to get things into sharness

from sharness.

I don't think this belongs in sharness: this is something the TAP driver should deal with.

What we could do is make the output of --verbose be TAP-friendly so the TAP driver can show the comments on failure.

But that's a different feature request.

from sharness.