circleci-public / bats-orb Goto Github PK

Orb Version
1.0.0

Describe the bug
CircleCI expects that test jobs report their tests to CircleCI so that the tests can be included in the CircleCI test report.
BATS is a test framework that runs tests.
...but the bats/run job doesn't give the tests to CircleCI, which makes debugging difficult.

To Reproduce

1. Follow the instructions in https://circleci.com/docs/testing-orbs/#unit-testing
2. Create a simple BATS file that runs some tests; make some of the pass, some fail, have all of them output some text or other.
3. Run the build in CircleCI
4. Go to the job bats/run
5. See that there are no test results in the CircleCI UI - as far as it's concerned, it ran a command and the command failed, but it provides no further details.

Expected behavior
Each BATS test case should appear in the CircleCI UI as its own test case.
The user should be able to use the CircleCI WebUI's "TESTS" tab to see whether each test passed or failed without having to inspect the console output of the bats/run job.
...and should be able to inspect the times each took, stdout, stderr etc just as for any other "proper" unit-test framework
...and, ideally, should be able to split the tests to support parallel execution.

Additional context

This orb is called out in the CircleCI documentation as being the technology of choice for testing bash scripts used in CircleCI orbs, which leads to an expectation that this orb would at least support "normal" test functionality with CircleCI.
The lack of any store_test_results functionality in this orb was a surprise - tests should appear in the tests tab.

TL;DR: bats/run should call store_test_results and upload the results.

Is your feature request related to a problem? Please describe.
Is there a reason why bats helpers such as bats-support, bats-assert and bats-files are not installed? They are useful for writing clean bats tests.

Describe the solution you'd like
Please install bats helpers

Describe alternatives you've considered
Installing bats helpers in own orb project under src/tests/test_helper folder.

Is your feature request related to a problem? Please describe.

Currently jobs don't expose a resource_class to use, this means they use the default. Currently this is Large, which is significantly more expensive than Small which would probably be sufficient. If we can expose this santza on the jobs, we can switch to these smaller instances and make our build cheaper.

Describe the solution you'd like

Add the resource_class to the parameters section for all jobs.

parameters:
  resource_class:
    type: enum
    enum:
      - small
      - medium
      - medium+
      - large
      - xlarge
      - 2xlarge
      - 2xlarge+
    description: Configure the executor resource class
    default: small

An example of when this is exposed can be seen here:
https://circleci.com/developer/orbs/orb/circleci/orb-tools#jobs-pack

Describe alternatives you've considered

N/A

Additional context

N/A

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
I'm running on a Linux machine and the output shows that Bats is already installed but it is on version 0.4. I would like to update to version 1.9

Describe the solution you'd like
A clear and concise description of what you want to happen.
Allow an extra parameter to the install command to upgrade or install a specific version.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context about the feature request here.

Orb Version
1.1.0

Describe the bug
Bats orb does not know the circleci env subst command which is recommended in orb template.

# Error: unknown command "env" for "circleci"
Run 'circleci --help' for usage.
time="2023-09-04T13:55:39Z" level=error msg="Exit with status 1 due to generic error" error="unknown command \"env\" for \"circleci\""
Hello World!
not ok 1 my test # in 21 ms
# (in test file src/tests/test.bats, line 4)
#   `[ "$output" = "Hello World!" ]' failed

To Reproduce

1. Create project from template orb (use commit a6c9b42 if you want perfect reproduction)
2. Add bats: circleci/[email protected] to the orbs key of .circleci/config.yml
3. Write a simple test in src/tests/test.bats (see below)
4. Make src/scripts/greet.sh executable
5. Run bats src/tests locally and see success (checked with bats 1.10.0)
6. Run circleci local execute bats/run and see above error message

Expected behavior

The command circleci env subst works in the bats orb

Additional context

Simple test for reproduction steps

@test "my test" {
  run "$BATS_TEST_DIRNAME/../scripts/greet.sh"
  [ "$output" = "Hello World!" ]
}