Git Product home page Git Product logo

polish's Issues

Reducing test output verbosity

The output when running tests, while nicely formatted, is very verbose. Is there a way to reduce the output to just the name of each test run and its status?

For example, I have two tests defined as follows:

use super::*;
use polish::test_case::{TestRunner, TestCaseStatus, TestCase};
use polish::logger::Logger;

#[test]
fn tests() {
    TestRunner::new().run_tests(vec![
        TestCase::new("BrakeAmt::new()",
                      "calling with no input succeeds",
                      Box::new(|_logger: &mut Logger| -> TestCaseStatus {

            // GIVEN the method under test
            let expected_result = FrictBrakeAmt(Unorm::default());
            let sut = FrictBrakeAmt::new;

            // WHEN a BrakeAmt is created
            let result = sut();

            // THEN the request should succeed, containing the expected value
            match result == expected_result {
                true  => TestCaseStatus::PASSED,
                false => TestCaseStatus::FAILED,
            }
        })),

        TestCase::new("BrakeAmt::from_unorm()",
                      "calling with a unorm value succeeds",
                      Box::new(|_logger: &mut Logger| -> TestCaseStatus {

            // GIVEN the method under test
            let test_value = 0.42;
            #[allow(result_unwrap_used)]
            let unorm = Unorm::from_f64(test_value).unwrap();
            let expected_result = FrictBrakeAmt(unorm);
            let sut = FrictBrakeAmt::from_unorm;

            // WHEN a BrakeAmt is created
            let result = sut(unorm);

            // THEN the request should succeed, containing the expected value
            match result == expected_result {
              true  => TestCaseStatus::PASSED,
              false => TestCaseStatus::FAILED,
            }
        })),
    ]);
}

They yield the following output:

running 1 test
Starting BrakeAmt::new() at 14:39:10 on 2017-12-21
Ended BrakeAmt::new() at 14:39:10 on 2017-12-21
calling with no input succeeds ... ✅
0 PASS  0 FAIL  0 WARN  0 INFO
Starting BrakeAmt::from_unorm() at 14:39:10 on 2017-12-21
Ended BrakeAmt::from_unorm() at 14:39:10 on 2017-12-21
calling with a unorm value succeeds ... ✅
0 PASS  0 FAIL  0 WARN  0 INFO

BrakeAmt::new() (calling with no input succeeds) ... 1ns
BrakeAmt::from_unorm() (calling with a unorm value succeeds) ... 1ns

Ran 2 test(s) in 2ns
2 Passed  0 Failed  0 Skipped
test types::unit_tests::tests ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

The verbosity obscures the test results. Ideally, I'd like to see just a simple namespaced list with a global summary at the bottom (across all modules and workspace crates):

✅ chal::types::unit_tests::tests::BrakeAmt::new() (calling with no input succeeds) ... 1ns
✅ chal::types::unit_tests::tests::BrakeAmt::from_unorm() (calling with a unorm value succeeds) ... 1ns

Ran 2 test(s) in 2ns...  ok
2 Passed  0 Failed  0 Skipped

Possible?

Running multiple THEN's per test case

I have a test suite set up as follows:

#[test]
fn tests() {
    TestRunner::new()
        .set_module_path(module_path!())
        .set_attributes(TestRunnerAttributes.disable_final_stats | TestRunnerAttributes.minimize_output)
        .set_time_unit(TestRunnerTimeUnits.microseconds)
        .run_tests(vec![
            TestCase::new("App::run()", "yields arch width", Box::new(|_logger: &mut Logger| -> TestCaseStatus {
                // GIVEN an app
                let mock_width = 42;
                let expected_result = Ok::<String, Error>(format!("Hello, {}-bit world!", mock_width));
                let mock = MockArch::new(mock_width);
                let sut = App::new(&mock);

                // WHEN the app is run
                let result = sut.run();

                // THEN the result should contain the expected architecture width
                match result == expected_result {
                    true => TestCaseStatus::PASSED,
                    false => TestCaseStatus::FAILED,
                }
            })),
            TestCase::new("App::run()", "calls Info::width() once", Box::new(|_logger: &mut Logger| -> TestCaseStatus {
                // GIVEN an app
                let mock_width = 42;
                let mock = MockArch::new(mock_width);
                let sut = App::new(&mock);

                // WHEN the app is run
                let _ = sut.run();

                // THEN the app should have called Info::width() exactly once
                match mock.width_times_called.get() == 1 {
                    true => TestCaseStatus::PASSED,
                    false => TestCaseStatus::FAILED,
                }
            })),
        ]);
}

Instead of repeating nearly the entire test case, I would prefer to simply add an additional THEN clause to the end of the first test case, like so:

fn tests() {
    TestRunner::new()
        .set_module_path(module_path!())
        .set_attributes(TestRunnerAttributes.disable_final_stats | TestRunnerAttributes.minimize_output)
        .set_time_unit(TestRunnerTimeUnits.microseconds)
        .run_tests(vec![
            TestCase::new("App::run()", "yields arch width", Box::new(|_logger: &mut Logger| -> TestCaseStatus {
                // GIVEN an app
                let mock_width = 42;
                let expected_result = Ok::<String, Error>(format!("Hello, {}-bit world!", mock_width));
                let mock = MockArch::new(mock_width);
                let sut = App::new(&mock);

                // WHEN the app is run
                let result = sut.run();

                // THEN the result should contain the expected architecture width
                match result == expected_result {
                    true => TestCaseStatus::PASSED,
                    false => TestCaseStatus::FAILED,
                }

                // AND_THEN the app should have called Info::width() exactly once
                match mock.width_times_called.get() == 1 {
                    true => TestCaseStatus::PASSED,
                    false => TestCaseStatus::FAILED,
                }
            })),
        ]);
}

Obviously, this won't compile given the current signature. The multi-test test case becomes more important as the complexity of the tests increase, by keeping the amount of duplicated code to a minimum.

Your version is at 0.9, so I thought I would bring this up before you stabilize your API at 1.0, just in case it ends up being a breaking change.

Trying not to break the API, here is one idea that might work:

fn tests() {
    TestRunner::new()
        .set_module_path(module_path!())
        .set_attributes(TestRunnerAttributes.disable_final_stats | TestRunnerAttributes.minimize_output)
        .set_time_unit(TestRunnerTimeUnits.microseconds)
        .run_tests(vec![
            ResultTestCase::new("App::run()", "yields arch width", Box::new(|_logger: &mut Logger| -> Result<(), TestCaseStatus::FAILED>  {
                // GIVEN an app
                let mock_width = 42;
                let expected_result = Ok::<String, Error>(format!("Hello, {}-bit world!", mock_width));
                let mock = MockArch::new(mock_width);
                let sut = App::new(&mock);

                // WHEN the app is run
                let result = sut.run();

                // THEN the result should contain the expected architecture width
                test_case_assert_eq(match result, expected_result)?

                // AND_THEN the app should have called Info::width() exactly once
                test_case_assert_eq(mock.width_times_called.get(), 1)?
            })),
        ]);
}

The benefits are that the test fails at the exact line where the test fails. This means that a developer can read the error message and will know the precise issue without having to enter debug.
Contrasted with:

...
                let test_case_status = TestCaseStatus::FAILED;
                // THEN the result should contain the expected architecture width
                test_case_status = match result == expected_result {
                    true => TestCaseStatus::PASSED,
                    false => TestCaseStatus::FAILED,
                }
                // AND_THEN the app should have called Info::width() exactly once
                test_case_status = match mock.width_times_called.get() == 1 {
                    true => TestCaseStatus::PASSED,
                    false => TestCaseStatus::FAILED,
                }
                // AND_THEN ...
                ...

                // AND_THEN ...
                ...

                test_case_status
            })),

where a) state is required to be maintained by the developer, and b) in the event of a failure, the specific sub-test which failed is lost, necessitating c) a debug session.

Anyway, this is not urgent or anything--this is just a thought I wanted to share with you. Please let me know if you have thoughts on other ways to achieve this.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.