PR #2 (duplicated from sstephenson/bats#231, which passes on Travis) didn't trigger a Travis build. Presumably Travis just needs to be pointed at this repo, since the config is already in place.

Do a quick grep of the repo and repoint any hardcoded paths from sstephenson/bats to bats-core/bats-core if applicable and approprioate.

Now that #8 is in, fixing a couple bugs and substantially improving performance—but not adding any new features—would it be cool to bump the version to v0.4.1? Or should we wait for a few more cleanups to go in first (e.g. #11, #12, #13)?

cc: @btamayo

These might make it easier for people to file high-quality issues and PRs, and to automatically request a code review from the bats-core organization.

• https://github.com/blog/2111-issue-and-pull-request-templates
• https://help.github.com/articles/about-codeowners/

We should integrate https://github.com/koalaman/shellcheck in our CI process.

In the meanwhile, we can encourage the use of tools like https://github.com/vim-syntastic/syntastic, which has saved my life as a Vim user.

Fun fact: I haven't yet implemented shellcheck for mbland/go-script-bash yet because, last time I checked, it didn't recognize . as a valid function name character. (I should check again, and submit a PR if that still isn't the case.)

This is a carry-over from sstephenson/bats#80, and also discussed in mbland/go-script-bash#226 and mbland/go-script-bash#227.

Bats duplicates standard output as file descriptor 3 so that its framework functions will produce TAP output that isn't captured into BATS_OUT, i.e. the file used to capture output from the code under test. When the code under test spawns a child process that blocks with this file descriptor open, Bats will also block (i.e. hang) until that process unblocks and exits.

I don't think there's a general fix within Bats itself, other than to very clearly document this behavior. I tried setting BATS_OUTPUT_FD=3 and using that to close the file descriptor in mbland/go-script-bash#227, but while it's possible to use {BATS_OUTPUT_FD}>&- in Bash 4.x, it doesn't appear possible in Bats 3.2.57(1)-release to use anything but a bare integer. Chapter and verse from https://tiswww.case.edu/php/chet/bash/CHANGES:

This document details the changes between this version, bash-4.1-alpha,
and the previous version, bash-4.0-release.

3.  New Features in Bash

p.  If the optional left-hand-side of a redirection is of the form {var}, the
    shell assigns the file descriptor used to $var or uses $var as the file
    descriptor to move or close, depending on the redirection operator.

We could still set and document BATS_OUTPUT_FD=3, and advise users that they must use 3>&- in code that must run on 3.2.57(1)-release, but can use {BATS_OUTPUT_FD}>&- if running only on Bash 4.1-alpha and higher.

Or..., we might consider porting the background-process utilities into bats-core, or some such extra module. Thoughts?

Oh, one more thing @jasonkarns: Adding a feature to enforce this mode is an interesting idea. Wanna file it for v1.0.0 (or v1.1.0)?

#23 (comment)

As the moment it's only possible to install bats from bats-core from source on linux.

We should create packages to install easily on Linux.
It could be:

• apt install bats-core for Debian/Ubuntu/Linux Mint
• yum install bats-core for Fedora/RedHat/CentOS
• a package on (Flathub](www.flathub.org)
• a snap on https://snapcraft.io/
  ...

Flathub is probably the easiest to support most distributions in one package.
Starting point: https://github.com/flathub/flathub/wiki/App-Submission

(transferring bugs list from sstephenson/bats#196)

• Empty output lines are missing from ${lines[@]}.
  sstephenson/bats#89
  sstephenson/bats#90
  sstephenson/bats#92

  sstephenson/bats#90 has already been merged to master

• Use POSIX compliant options with head for better compatibility (e.g. Busybox support).
  sstephenson/bats#163

  closed by #17

• Incorrect TAP format for skipped tests.
  sstephenson/bats#141
  sstephenson/bats#142

  will be closed by #19

• set -o nounset breaks bats-exec-test.
  sstephenson/bats#171

• bats-preprocess breaks when translating a @test line that uses tabs for white space, instead of plain spaces.
  sstephenson/bats#98
  sstephenson/bats#122

• Bats failing on OS X 10.11.2.
  sstephenson/bats#140

Hi,

I'd like to report an odd issue using BATS on recent Mac OS X versions. When Apple added System Integrity Protection (SIP), an odd "feature" is that child processes no longer receive environment variables related to shared libraries, e.g. DYLIB_LIBRARY_PATH. This apparently has been known for a few years - https://www.postgresql.org/message-id/[email protected].

This caused me a big headache this evening, because I swapped to building my suite of tools with shared libraries, and while developing I install them a subdirectory in $HOME. My environment has DYLIB_LIBRARY_PATH correctly set.

Sadly, with this set up BATS does not work, because the sub-shells spawned by BATS do not pick up the DYLIB_LIBRARY_PATH variable. I have resorted to hard-coding an export statement in my tests for the moments as a workaround. An alternative is to install my tools to /usr/local, which Mac OS will search for shared libraries by default.

If anyone has any suggestions for a better way round this, then I am all ears.

I am running Alpine Linux and when I clone the master branch of bats-core and run install.sh I get an error. Here is an example of the output I am seeing.

$ cat /etc/alpine-release 
3.7.0

$ ls
CODE_OF_CONDUCT.md  Dockerfile          README.md           install.sh          man                 test
CONTRIBUTING.md     LICENSE.md          bin                 libexec             package.json

$ sudo ./install.sh /usr/local/
readlink: unrecognized option: e
BusyBox v1.27.2 (2018-01-29 15:47:03 UTC) multi-call binary.

Usage: readlink [-fnv] FILE

Display the value of a symlink

        -f      Canonicalize by following all symlinks
        -n      Don't add newline
        -v      Verbose

$ ls -l $(which readlink)
lrwxrwxrwx    1 root     root            12 Feb 17 18:00 /usr/bin/readlink -> /bin/busybox

As I'd mentioned in sstephenson/bats#150 (comment), my mbland/bats optimization refactoring resulted in a very significant performance improvement, especially on Windows. Of course I'm happy to help with implementing the other roadmap features, but I'd encourage merging these changes ahead of the others, as it's a straight refactoring that adds zero new behavior.

bash version: GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)

similar to #72 where toggling set +/-e during the command seems to be a workaround

test file

#!/usr/bin/env bats

setup() {
  set -u
}

@test "example" {
  echo $unbound_variable
}

output

→ ./test.bats
 ✗ example

1 test, 1 failure

with workaround

#!/usr/bin/env bats

setup() {
  set -u
}

@test "example" {
  set +e
  echo $unbound_variable
  set -e
}

workaround output

→ ./test.bats
 ✗ example
   (in test file test.bats, line 9)
     `echo $unbound_variable' failed
   /var/folders/dm/2_68m_h13h5dxq2f05wsc47c0000gq/T/bats.76966.src: line 9: unbound_variable: unbound variable

1 test, 1 failure

Right now run merges stdout and stderr in $output this can be usefull for some use cases. But I can't assert that some error message was printend in stderr instead of stdout.

Can I check this ussing run?

consider following test case:

#test_fastfail.bats
1 setup() {
2   source fastfail.sh
3 }
4
5 @test "fast fail" {
6   run fastfail
7   [ $status -eq 0 ]
8 }
# fastfail.sh 
1 function fastfail {
2   ls notexist
3   echo done
4 }
$ bats fastfall.bats
 ✓ fast fail

1 test, 0 failures

the problem is in function run, fastfail should be enable for subshell like:

 50 run() {
......
 58   output="$(set -e; "$@" 2>&1)"

Hi,
please add a junit report option,
as a flag to the run:
should be like bats (test files/folder) --junit

Hello all,

Since one of the main goals of this move is to prevent lock-out by giving access to more than one person, I thought that that would be apt to prioritize.

My decision is to pick three additional people that will be able to have admin access to this repository based on the following criteria:

1. Knowledge/usage about bats and/or bash
2. Recent Activity on GitHub (if they've been online recently)
3. Expressed interest in the original Call For Maintainers thread

Responsibilities

The main responsibility here is to be active and prevent lock-out. It doesn't necessarily mean that we will change existing processes, but it does mean that if, at some point, none of us can maintain the repo or merge PRs, one of us will have the authority and access to pass on the ability to do so.

That being said, the first order(s) of business (to which everyone/anyone can contribute towards) will be:

1. Finish moving PRs and issues to this org
2. Organize docs/tests
3. Finish issues in #196

Why three people?

Too many chefs spoil the broth, etc etc. But people should have access to the broth. :)

Who?

The following folks will be asked to help but they need to express interest (if they're not active on GitHub at all there wouldn't be much of a point):

1. @ztombol
2. @mbland ✅
3. @jasonkarns ✅

Since things could have changed for them (no longer interested, time commitments, etc.) please let me know if you're interested in helping maintain through this thread :)

Right now there's not really a way to use output and pass it through some function that can parse it, beyond just using the [ "$output" = "blah blah" ] shell tests. This is quite limiting in particular when the thing being tested can return somewhat complicated, or a gamut of messages on failure, depending on how it failed. An ability to take "$output" and pass it through a function, which itself returns 0 or 1 would be quite useful I believe.

Right now I have a test file with some code in the setup function. If I remove the code outside of the setup function it still gets run and, output aside, the result is the same. Is there any essential difference between the two ways? I am asking because I want to print some text in the setup function but I cannot. Outside of the setup I can.

This is a feature request.

Same as running all .bats files in a directory, but when -r flag is provided, tests will be discovered in sub directories as well (recursively).

Hi,

Are you planning on doing a new release soon?

Thanks,

I've used https://github.com/SimonKagstrom/kcov to great effect to collect coverage on mbland/go-script-bash and publish it to Coveralls.

The only wrinkle is that I clone and build the kcov repo fresh every time, since the Ubuntu package is a bit old. For this project and that one, I should look into finding a way to install an updated binary.

Currently, brew info bats would print this:

$ brew info bats
bats: stable 0.4.0, HEAD
TAP-compliant test framework for Bash scripts
https://github.com/sstephenson/bats
Not installed
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/bats.rb

This needs to be updated in Homebrew's core tap, or a new one should be created.

Update information that needs to be updated on current README, including but not limited to:

• Installing and usage - partially covered in README (TODO: sufficient?)
• Platform support - fixed in #67
• Adding versioning info - fixed in #71 (unmerged)
• Changelog update - fixed in #66 (unmerged)
• Bash minimum version - fixed in #67

This is a feature request.

Currently, failure of one test does not have an effect on the others.

There are some use cases where running consecutive tests is irrelevant and might waste time and computing resources or even confuse the person reviewing the test results.

For example if one tests is compiling the program under test and compilation fails, that every other tests operating on that program will fail as well.

One way to deal with this is to mark tests as special tests who's failure will end the testing of the current bats file.

In fact other testing frameworks like JUnit have this behavior as default (inside each test case) and the other way requires some workarounds (https://stackoverflow.com/questions/10221891/continuing-test-execution-in-junit4-even-when-one-of-the-asserts-fails)

Placeholder - I intend to work on this in the coming weeks.

• Basic usage
• Docker-y gotchas
• Extending from the base image
• slim vs full images
• Examples of running in various CIs

Any further topics?

Per my analysis in #72, we should update load() to toggle set +e/-e around the source command in its implementation, along with an explicit error check. Then we should strongly recommend its use over the builtin source command.

The new bin/bats is a wrapper to call the bats frontend:

  exec "${0%/*}/../libexec/bats" "$@"

The $0 in a basher installation and in other installations may be a symlink.
Taking the directory part of the symlink gives the wrong path, so an additional step is needed:

  _0=$(realpath "$0")

which is why perhaps a symlink is better. I see the objective (#32) is to avoid issues with symlinks on windows.

I'd really like to use a packer manager to get basher. basherpm works well for small projects, as long as some common sense rules are followed. Maybe this isn't supported, or maybe there is something better. What is the preferred installation method?

Part of issue creation and project migration effort. From sstephenson/bats #196.

The markdown template for this issue can be found here: Template for issue #1

Related issues:

1. sstephenson/bats#69
2. sstephenson/bats#10

Related PRs:

1. sstephenson/bats#47

What's left to do?

Links in the PR template point to .github/ for contributing and code of conduct files. (I'm assuming they're relative URLs.)

Should those files go under .github/? Or at the very least, make the links absolute? (I'm partial to moving the files under .github if github recognizes those files in there.)

Originally reported by https://github.com/ORESoftware
(This issue brought over from sstephenson/bats#245)

this is what I get:

$ bats test/src/dev/bats/a.test.sh --tap-json
Bats 0.4.0
Usage: bats [-c] [-p | -t] <test> [<test> ...]

this is what I should get:

$ bats test/src/dev/bats/a.test.sh --tap-json
Bats 0.4.0
Bad command line option "--tap-json"
Usage: bats [-c] [-p | -t] <test> [<test> ...]

let's make it easy for people to use software eh :) not everything needs to be a struggle

Also move the install.sh file into libexec.

Summer has just arrived and most of us will be having a little more time to spare on our favorite projects. How would you feel about organizing an online Bats sprint this summer?

My suggestion is a 1-2 day event to meet online, get to know each other and work on remaining and new bugs and features and plan the future. There is not a ton of work to carry but it would be nice to work on what there is on the issue tracker and on our minds cooperatively and in a dedicated manner.

Do you think it would be worth it?

Now that we're stable and starting to ship packages via brew (#11), Alpine (#alpinelinux/aports/pull/3182), and RPM (#111), we may want to move the changelog out of the README into a CHANGELOG.md file, and find a way to include it in various packages, git version tags, etc. (Per the discussion in #111 in particular.)

The extent to which we can auto-generate such logs in the first place, the better, but this issue is focused on taking the already-written CHANGELOG.md contents and reusing them elsewhere.

A potential source of inspiration: https://keepachangelog.com/en/1.0.0/

The maintainers of sstephenson/bats made it clear that they wanted to use as much posix as possible.

While this makes sense in some cases (and would entirely be possible), I think that it doesn't fit the projects' scope.

1. BATS stands for bash automated testing system
2. The codebase uses features such as pipefail, which are not part of the POSIX sh spec (and likely won't be for the near future)

A negligible point is the cost of the additional calls to external programs, as modern machines don't have a problem with those ([ e.g. was an alias to test until recently in bash, instead of =~ grep or expr have to be used).

In general bash is more friendly to develop and read as a posix compliant test, e.g., does not allow for compounds like [ 1 -eq 0 || 0 -eq 0 ], which is a common construct to use.

I'd like a clear statement if this project intents to go full posix or full bash. I'd be fine with either way, but currently it is a mix.

CONTRIBUTING.md should provide an overview of (roughly):

• git cloning, branching, rebasing, pushing
• QA/shellcheck/CI and style
• Creating PRs

I recently came across bats-support and bats-assert. Maybe there are others too. It seems to me that there are useful features in these projects and there is the danger that they become obsolete due to lack of maintenance.

I would like to suggest merging such extensions directly into bats due to the following advantages:

• ensured maintenance side-by-side core bats
• same quality of code as core bats
• easy discovery by users
• better overview of project and its dynamics, instead of having to keep track of several git repos.

Has there been past discussion on this?

This is a feature request.

The use case is having some kind of precondition checking or setup before all tests are executed and a cleanup after all tests are complete.

This is different from setup/teardown as these functions run before/after each test execution.

According to the TAP spec, diagnostic lines should start with #. Otherwise, while not treated as an error, it is up to the harness what it will do with it. Right now in Bats, if the user needs to output some text from within a test function and still get proper TAP output, they need to:

• redirect to FD 3
• prepend text with #

Eg, echo "# $variable" >&3

This seems to me like an unnecessary burden. It would be nice to have a function, perhaps called bats_print_diagnostic() that will print whatever arguments are passed to it in the proper manner.

I'm not sure what the goal of expecting a helper script to be an extension of .bash is, but it makes absolutely zero sense.

Let me specify a path and load that file. Don't force the extension on me.

I'm writing some tests with bats to test my docker image.

All my tests depends on the same image so I have two options:

• Build the image before every test (using setup()). Docker use the cache and it is not that slow... but it's slower that do nothing. And the output is poluted.
• Remeber to run the docker build before I run bats... this is much faster but it force me to remeber to launch the docker build command EVERY time. At the end of the day this is even slower because I forgot to do it and I need to launch bats twice.

It'd be great to define two new functions and call them before and after all the tests run.

When doing TDD it would be advantageous to be able to run specific test(s). Perhaps this could be an option at the CLI.

I currently play games like removing all the other tests to be able to get a single to run in a reasonable amount of time.

curl -O https://github.com/bats-core/bats-core/archive/v0.4.0.tar.gz
tar -xzvf v0.4.0.tar.gz

Results in

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

This is also true for the original release available here: https://github.com/sstephenson/bats/releases

automake prefers .test as the extension for running tests

for whatever reason, if I run tests through their tap-driver and use any extension other than .test, it doesn't display the TAP output

example

#!/usr/bin/env bats

@test "hello world" {
  true
}

<snip>
PASS: tests/ut/list/api/create.test 1 hello world
<snip>

vs

<snip>
PASS: tests/ut/list/api/create.bats
<snip>

Notice how the 1 hello world output is only shown if the file ended in .test

As it stands, bats requires .bats extension if you want to run bats directly and point it at a directory.

This is a request to support additional file extensions, e.g. to allow me to run bats against a directory but tell it to look for *.test rather than *.bats

This may be my lack of understanding of bash, so forgive me if it is but I've come across the following situation which doesn't 'feel' right.

I've created a setup function which currently fails. All the setup function is configured to do is source a file which will contain constants but, at the moment, doesn't yet exist. Obviously this is a problem I will fix, but what concerned me was it resulted in the subsequent test failing but without displaying an error.

This can be replicated with the following test script:

function setup {
    source "/non-existant-file"
}

@test "some random test" {
    # This should always pass
    return 0
}

This will result in:

 ✗ some random test

which doesn't feel right. If I force the setup function to explicitly return 1:

function setup {
    source "/non-existant-file" || return 1
}

The result makes more sense:

 ✗ some random test
   (from function `setup' in test file test.bats, line 2)
     `source "/non-existant-file" || return 1' failed
   /tmp/bats.14184.src: line 2: /non-existant-file: No such file or directory

Is this my lack of understanding of bash or is something else at work here?

Hi!

When I have a test like this,

@test "integers: 0" {
   cat <<-PROGRAM >program.scm
		0
	PROGRAM
   run ocameel program.scm

   [ "$status" -eq 0 ]
   [ "$output" = "0" ]
}

… output like this seems surprisingly unhelpful:

 ✗ integers: 0
   (in test file test/tests-1.1.bats, line 25)
     `[ "$output" = "0" ]' failed

I would expect to see, at least, something like this:

 ✗ integers: 0
   (in test file test/tests-1.1.bats, line 25)
     `[ "$output" = "0" ]' failed;
       $status: 0
       $output:
         this
         was some output

Of course, there's some concerns with output-length here; as well as some UX determination. For instance, if a failed line's variables are short enough, I'd really love to see interpolated output (i.e., instead of `[ "$output" = "0" ]' failed, if $output is ≤50 characters or something else arbitrary, `[ "this\nwas some output" = "0" ]' failed.)

Hope this is a well-received Issue, and one of some interest to the new team! Thanks for your consideration!

Let say if one test is failed, how to display some information from output of that command line in order to troubleshoot / diagnostic?

@test "addition using bc" {
  result="$(echo 2+2 | bc)"
  [ "$result" -eq 41 ]
}

I want to output $result (which is 4). This doesn't work [ "$result" -eq 41 ] || echo "$result"

Currently install.sh changes permissions on everything in $PREFIX/{bin,libexec}, which is bad if installing to /usr/local.

The tests must be fast. This way the developers will run tests more often and the bugs will decrease.

Print the time of each tests and the time of the complete suite helps to find the slow tests and fix them.

Corresponds to sstephenson/bats#49. I believe we can work around this during the preprocessing step by matching lines against \]\][[:space:]]*$ and appending || false to them, at least when [[ "${BASH_VERSION:0:1}" -lt '4' ]]. But of course I'd need to test it!