Parameterized testing is a staple of any unit-testing framework.

This issue is to track adding support for basing parameterized testing so that tests can operate with different inputs. In particular, the goal of this feature is:

• Inputs can be accepted as function arguments, but specified as a simple parameter to the neotest attribute,
• Multiple inputs produce the complete-graph of possible inputs. This means that a = [1, 2, 3] and b = [4, 5] will produce 6 total tests: (1, 4), (1, 5), (2, 4), (2, 5), (3, 4), (3, 5).
• Specifying parameter inputs does not require the neotest parameters to be in the same order as function parameters

The definition for neotest fixtures currently does not allow for
the argument to be of a different refness or mutability (e.g. &Fixture
is disallowed, but Fixture is okay).

This issue is to track fixing this omission so that the calling convention
is honored based on the way the developer chooses to define their
fixture parameter.

Having tests return TestResult (Result<(), Error> objects), enables having assertions that don't panic, and also makes it possible to return from tests early.

Ideally this would be transparent to the user, so this ticket is to track the following implementation:

• If the user manually specifies -> TestResult, they must manually call Ok(()) themselves,
• If the user omits -> TestResult, then the test implicitly adds Ok(()) at the end

As an extension to #7, it can be useful in testing to provide non-fatal assertions for tests, which can allow the code to progress after failure.

The current proposed implementation is:

• Extend __Context to support storing assertion errors
• Add a proc-macro that writes to the implicit __context for failures (this is necessary since declarative macros will fail due to __context not being visible in the current scope)
• Create declarative macros that call the proc-macro to trigger errors

There should also be considerable code-sharing between #7 and this, since anything we want asserts of, we should also offer expects.

As an extension to #7, it'd be nice to have "matcher" logic, which can help simplify testing that values match a given expectation in a simplified way. These matchers can be based off of the way other testing libraries are written, such as Catch2 or GTest.

This should also be usable in an assertion, which would ideally generate a custom message. These will also be useful for mocking when we want expectations on types.

This is a larger task that will make use of #11 in order to generate/define mock functionality.

The exact syntax/specifics are yet to be determined, but the idea is to support something like:

trait Turtle {
    fn pen_down(&mut self);
    fn pen_up(&mut self);
}

#[neotest::mock]
struct MockTurtle;

impl Turtle for MockTurtle {
    // Set this as a mock function
    #[neotest::mock_fn]
    fn pen_down(&mut self);
    #[neotest::mock_fn]
    fn pen_up(&mut self);
}

// ...

fn test_draw_circle() {
    let mut turtle = MockTurtle::new();
    let painter = Painter::new(&mut turtle);

    subtest!{ |calls_pen_down| 
        let expectation = neotest::expect_call!( turtle.pen_down() => Times(1) );

        painter.draw_circle(0, 0, 10);
        
        assert!(expectation.satisfied());
    }    
}

(Syntax/names are subject to change, but this is the general idea).

As an extension to #11 , it's sometimes necessary to provide an expectation that certain events happen in a certain order (e.g. A before B, not vice-versa). This task is to track defining a mechanism that allows expectations to be in a forced-sequence.

To be able to implement/provide mocking-like functionality, we need a way to test whether expected behavior has occurred.
The conventional way of doing this is with an Expectation-like object.

This task is to track adding expectations in some manner as a new module in neotest.

Adding support for returning TestResult objects in #6 introduced an unexpected bug: the #[should_panic] attribute no longer functions correctly. #[should_panic] apparently only works in functions returning (), which raises questions as to the future direction of testing in this library:

1. Should an assertion library be made that returns Error objects as #7 suggests? Doing this would mean that #[should_panic] tests cannot actually work with these assertions

2. A custom assert_panic!(...) macro can be added that is implemented in terms of std::panic::handle_unwind -- however this is documented to only work for panics that perform unwinding. It's unclear if this is also the behavior of #[should_panic], or whether #[should_panic] can actually catch cases that abort.

A decision needs to be made in order to progress here. It's possible that supporting attributes in subtests in #14 might be sufficient for handling this, in that subtests can return () whereas other tests return Error.

The current mechanism for subtests simply acts as a series of statements defined in a subtest! { ... } macro. Although this functionally works, it suffers the significant drawback that it is unable to have custom attributes specified -- such as #[should_panic]or#[ignore]`, which are needed for testing.

It's unclear what the best direction forward for this is. A couple thoughts come to mind:

• Extend the DSL to support [<attribute>] syntax, e.g. subtest! { |name| [should_panic][ignore] ... }, this is kind of gross -- but would work.
• Change the syntax entirely; perhaps using function definitions instead:

  #[neotest::test]
  fn test_vec_index() {
      let vec: Vec<u8> = Vec::default();
  
      #[neotest::subtest]
      #[should_panic]
      fn out_of_bounds_panics() {
          vec[20];
      }
  }

  The downside to this approach is that it's not clear that vec is within fn out_of_bounds_panics() scope, since this typically is not a true closure. The upside is that this has a clear mechanism for identifying attributes, the test ident, etc.

The exact approach is yet to be determined.

Blocked by #6

The builtin assertions are lackluster for testing in that they only cover assert_eq, assert_ne, assert, and assert_match (nightly). Ideally we should have the full collection of unit test assertion helpers like mature unit test libraries have, such as GTest.

This is to track adding assertions for eq, ne, true, false, regex, match, etc.
These should make use of the test-return-types, which also enables adding simple succeed!() and fail!() macros.

The current approach to inputs works, but is a little counter-intuitive due to the combinatoric nature of the arguments. This also suffers when combined with subsections, since cohesion may drop if some subtests don't require inputs while others do.

Thus, this provides a different proposal: rather than using parameters = , provide an inputs![ ... ] macro that expands into an array that selects the correct input at runtime from the current __Context.

This could be implemented in the following way:

• __Context now stores current input state (no longer just the section path).
  • This logic needs to be tied per-section, since sections may now start their own input fields
• Have a inputs![...] macro that expands, effectively, into [...].into_iter().nth(__context.input_index())
• Have each base-test context iterate through all possible inputs
• Inputs now need to be determined by examining the unit test for each inputs![...] macro being used. This is applied per subtest.

This will now require a significant restructure to tests. Having subtests be able to drive parameters means that we no longer have a fixed inputs -> subtests ordering, but rather an arbitrary re-nesting of this.