Diffing tool is never prompted to open about approvaltests.cpp HOT 13 CLOSED

Thank you for the update and the extra information.

I'll close the ticket, as no action is required.

from approvaltests.cpp.

Hi, thanks for using Approval Tests.

Your setup

What platform are you running on?

Please check the list of supported reporters for your platform:
https://approvaltestscpp.readthedocs.io/en/latest/generated_docs/Reporters.html

Which of the supported reporters (diff tools) have you got installed on your machine?

And are they in the default location that Approval Tests searches for?

Or if you are using VSCode as the diff tool, is it in your path?
https://approvaltestscpp.readthedocs.io/en/latest/generated_docs/Reporters.html#cross-platform

For example, clicking on the 'Snippet Source' links in the above page, you might find that your Araxis Merge installation is not found - in which case, you can search the code for AraxisMergeReporter and see where it looks.

from approvaltests.cpp.

If you have installed a diff tool which is not recognised by this project, there are other links on that page for setting up your own custom reporter and making it the default.

from approvaltests.cpp.

I'm on a Mac OS 13.6.3. I've got both Visual Studio Code and Sublime Merge installed. They are both located on the same path that ApprovalTests should search for. Is there a specific argument that needs to be provided to the test executable?

Here's my cmake configuration for completeness sake:

FetchContent_Declare(
    Catch2
    GIT_REPOSITORY "https://github.com/catchorg/Catch2.git"
    GIT_TAG "v2.13.10"
)

FetchContent_MakeAvailable(Catch2)

FetchContent_Declare(
    ApprovalTests
    GIT_REPOSITORY "https://github.com/approvals/ApprovalTests.cpp.git"
    GIT_TAG "v.10.12.2"
)

FetchContent_MakeAvailable(ApprovalTests)

file(GLOB UNIT_TESTS CONFIGURE_DEPENDS tests/**/*_spec.cpp)

enable_testing()

foreach(UNIT_TEST ${UNIT_TESTS})
    get_filename_component(TEST_BASENAME ${UNIT_TEST} NAME_WE)
    add_executable(${TEST_BASENAME} ${CMAKE_CURRENT_SOURCE_DIR}/tests/main_config.cpp ${UNIT_TEST})

    add_test(NAME ${TEST_BASENAME} COMMAND ${TEST_BASENAME})

    target_compile_features(${TEST_BASENAME} PUBLIC cxx_std_17)
    target_link_libraries(${TEST_BASENAME} ApprovalTests::ApprovalTests Catch2::Catch2 playa)
endforeach()

from approvaltests.cpp.

I'm on a Mac OS 13.6.3. I've got both Visual Studio Code and Sublime Merge installed. They are both located on the same path that ApprovalTests should search for.

Thanks for checking. That's surprising. We haven't had any such report previously, so nothing springs to mind about what the cause could be.

Is there a specific argument that needs to be provided to the test executable?

Any required arguments are included in the source code for the relevant reporter/diff tool

Here's my cmake configuration for completeness sake:

If your executable builds and runs, it won't be a CMake problem.

It's unlikely, but perhaps it may turn out that a diff tool has changed its command line arguments, for example.

All I can think of is to help you debug it for yourself.

First, please run these commands and tell me what happens:

echo hello > a.txt
echo bye > b.txt
"/Applications/Sublime Merge.app/Contents/SharedSupport/bin/smerge" mergetool --no-wait "a.txt" "b.txt" -o "b.txt"
"/Applications/Visual Studio Code.app/Contents/Resources/app/bin/code" -d "a.txt" "b.txt" &

They should both open up the respective diff tool, and show hello on the left, and bye on the right.

from approvaltests.cpp.

Thanks for the ongoing help @claremacrae.

Both commands run fine and show the desired behavior. I was asking about the arguments for the text executable, because I don't see any information regarding any of the diffing tools. When is the approval happening, on build or executable run?

from approvaltests.cpp.

Both commands run fine and show the desired behavior.

I think that you are saying that both commands run fine in the console, but when you run your tests and call one of the verify...() functions in this project, no diff tool is found.

This is really strange.

I was asking about the arguments for the text executable, because I don't see any information regarding any of the diffing tools. When is the approval happening, on build or executable run?

In order to offer Convention over configuration, it's a series of steps, and they all happen at run time.

In order to make that process visible to developers of the library, our tests create a text file showing the commands and their command-line args generated for all the supported diff tools on all the platforms (plus some invalid combinations, that can be ignored)

I copied the command-lines I gave you to test, from that file - so I am confident that I gave you the command-lines to execute that Approval Tests would be executing, if there was nothing in the user's source code to override the way that Approval Tests searches for diff tools.

Please can you show the source code of the test that does not pop up a diff command? Many thanks.

from approvaltests.cpp.

I think that you are saying that both commands run fine in the console, but when you run your tests and call one of the verify...() functions in this project, no diff tool is found.

Yes exactly.

In order to make that process visible to developers of the library, our tests create a text file showing the commands and their command-line args generated for all the supported diff tools on all the platforms (plus some invalid combinations, that can be ignored)

Why is the MacOS command windows: start "" "/Applications/Visual Studio Code.app/Contents/Resources/app/bin/code" -d "a.txt" "b.txt" with a windows label? Could this be the issue?

here's the test file:

#include <catch2/catch.hpp>
#include <vector>
#include "playa/core/event.h"
#include "ApprovalTests/Approvals.h"

TEST_CASE("Event Test suite")
{
    using namespace playa;

    SECTION("should expand rhythm into events")
    {
        // when
        auto events = Event::expandDuration(Rhythm<4>{n4, n4, n4, n4}, 0);
        etl::vector expected{Event{0u, n4, n4}, Event{n4, n4, n2}, Event{n4 * 2u, n4, n2 + n4}, Event{n4 * 3u, n4, n1}};

        auto expected_str = std::vector(events.begin(), events.end());
        // then
        ApprovalTests::Approvals::verifyAll("Expand Rhythms simple", expected_str);
    }
}

Full error:

-------------------------------------------------------------------------------
Event Test suite
  should expand rhythm into events
-------------------------------------------------------------------------------
/X/playa-cpp/tests/core/event_spec.cpp:13
...............................................................................

/X/playa-cpp/tests/core/event_spec.cpp:13: FAILED:
due to a fatal error condition:
  event: 0 960 960
  event: 960 960 1920
  event: 1920 960 2880
  event: 2880 960 3840
  SIGABRT - Abort (abnormal termination) signal

===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed

from approvaltests.cpp.

In order to make that process visible to developers of the library, our tests create a text file showing the commands and their command-line args generated for all the supported diff tools on all the platforms (plus some invalid combinations, that can be ignored)

Why is the MacOS command windows: start "" "/Applications/Visual Studio Code.app/Contents/Resources/app/bin/code" -d "a.txt" "b.txt" with a windows label? Could this be the issue?

No, that is covered by the "(plus some invalid combinations, that can be ignored)" comment above.

One of the steps that the code does is to first check whether the given exe/script path exists on the machine running the tests, and if it doesn't, it goes on to the next candidate reporter in the list.

This is a well-tried pattern over many implementations of Approval Tests in many different languages.

here's the test file:

Thank you.

Full error:

-------------------------------------------------------------------------------
Event Test suite
  should expand rhythm into events
-------------------------------------------------------------------------------
/X/playa-cpp/tests/core/event_spec.cpp:13
...............................................................................

/X/playa-cpp/tests/core/event_spec.cpp:13: FAILED:
due to a fatal error condition:
  event: 0 960 960
  event: 960 960 1920
  event: 1920 960 2880
  event: 2880 960 3840
  SIGABRT - Abort (abnormal termination) signal

===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed

Yes, so that never reached the Approval Tests code. Approvals has a very specific message if it cannot find a reporter.

Also, it's telling you it crashed in line 13, which is this one:

auto events = Event::expandDuration(Rhythm<4>{n4, n4, n4, n4}, 0);

from approvaltests.cpp.

I'm still super confused though, as in your original report above you said:

When I run the tests the diffing tool never opens up and the test crashes due to the error ApprovalMissingException.

ApprovalMissingException is the normal mode of test failure when you run an approval test the first time, as there is not yet an 'approved' file. It expects you to use the diff tool to copy content from the 'received' file to the 'approved' file.

But that SIGABRT output in your preview message seems to show that execution never reached as far as the verifyAll() call.

from approvaltests.cpp.

Also, it's telling you it crashed in line 13, which is this one:

I don't know why the stack trace is wrong, but it is. Here's the screenshot of the error when I debug the test case:

This is the last call in the stack which points to c++ code, the later ones are in assembly.

The test case passes fine when I use catch2 and test it with:

REQUIRE(events.size() == 4);
REQUIRE_THAT(std::vector(events.begin(), events.end()),
       Catch::Matchers::Equals(std::vector(expected.begin(), expected.end())));

from approvaltests.cpp.

I'm using /opt/homebrew/bin/aarch64-apple-darwin22-g++-13 -> g++-13 (Homebrew GCC 13.2.0) 13.2.0. It looks like the problem is with the unwinding of the exception.

from approvaltests.cpp.

So it works fine when I switch to:
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin22.6.0

from approvaltests.cpp.