5.7.1 breaks Bazel sample about junit5-samples HOT 13 CLOSED

5.7.2 is now released and resolves this issue.

from junit5-samples.

@jdai8 Thanks for reporting! Do you have time to debug to see what's going on?

from junit5-samples.

@marcphilipp sure, although I could use some guidance. The sample essentially just invokes the console launcher with --select-package:

From the 5.7.1 changelog, junit-team/junit5#2500 seemed like the only possible culprit, but AFAIK this sample isn't using modules.

from junit5-samples.

@jdai8 I could not reproduce this locally:

$ git checkout r5.7.1
Switched to branch 'r5.7.1'
Your branch is up to date with 'origin/r5.7.1'

$ ./bazelisk.py test //... --test_output all
INFO: Analyzed 2 targets (0 packages loaded, 0 targets configured).
INFO: Found 1 target and 1 test target...
INFO: Elapsed time: 0.163s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
PASSED: //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test (see /private/var/tmp/_bazel_marc/543254cdda28f56bba4a66a0e7feffaa/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/src/test/java/com/example/project/junit5-jupiter-starter-bazel-test/test.log)
INFO: From Testing //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test
==================== Test output for //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test:

Thanks for using JUnit! Support its development at https://junit.org/sponsoring

╷
└─ JUnit Jupiter ✔
   └─ CalculatorTests ✔
      ├─ 1 + 1 = 2 ✔
      └─ add(int, int, int) ✔
         ├─ 0 + 1 = 1 ✔
         ├─ 1 + 2 = 3 ✔
         ├─ 49 + 51 = 100 ✔
         └─ 1 + 100 = 101 ✔

Test run finished after 136 ms
[         3 containers found      ]
[         0 containers skipped    ]
[         3 containers started    ]
[         0 containers aborted    ]
[         3 containers successful ]
[         0 containers failed     ]
[         5 tests found           ]
[         0 tests skipped         ]
[         5 tests started         ]
[         0 tests aborted         ]
[         5 tests successful      ]
[         0 tests failed          ]

================================================================================
//src/test/java/com/example/project:junit5-jupiter-starter-bazel-test (cached) PASSED in 1.1s

Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action

from junit5-samples.

I've enabled test output on CI now as well and it works:
https://github.com/junit-team/junit5-samples/runs/2146954961?check_suite_focus=true#step:6:1267

Since I don't know what else to check, I'm closing this issue for now.

@jdai8 If you have any additional info as to why this might be failing for you, please let us know.

from junit5-samples.

@marcphilipp After updating a project of mine to 5.7.1, I started experiencing the same issue as described here. I've downgraded to 5.7.0 for now. My test plan creation logic looks like:

    val testLauncher = LauncherFactory.create()
    val testPlan = testLauncher.discover(
      LauncherDiscoveryRequestBuilder
        .request()
        .selectors(DiscoverySelectors.selectPackage(testPackageName))
        .listeners(LauncherDiscoveryListeners.abortOnFailure(), LauncherDiscoveryListeners.logging())
        .filters(filters: _*)
        .configurationParameters(configParamsAdjusted.asJava)
        .build())

The filters are looking for a tag, but given that the diff between 5.7.0 and 5.7.1 are pretty small and PR #2531 seems to be the only discovery-related change, I am suspecting it is related to the package selector as opposed to the filters.

It seems the change was just to force a / at the end of the package name, and it seems that although I am using Scala -- which I initially suspected may be doing something unexpected to the names -- this should still be safe in my environment.

Let's assume testPackageName above is equal to my.package.prefix, I checked in my JAR file and it looks as expected:

± unzip -l my-jar-file.jar | grep "my.package.prefix" | grep "TestClass.class"
    XXXXX  02-01-1980 00:00   my/package/prefix/TestClass.class

(package/jar/test class names have been changed but are representative)

Do you have any suggestions on how to go about further investigating this? I would be happy to provide more information to understand if this is something I am doing wrong on my end or a legitimate bug in 5.7.1.

from junit5-samples.

@xkrogen Thanks for the info! Essentially we'd need a reproducer. Could you try changing junit5-jupiter-starter-bazel until it reliably reproduces the issue in a fork?

from junit5-samples.

Hi @marcphilipp thanks for the quick response! I don't really know where to start investigating such an issue, especially as I know nothing about Bazel, so I don't think that will be a fruitful path for me. If you have suggestions on how I can perform further investigation within the context of my own project, such as logging to enable or areas of the code to step through in a debugger, I will be happy to help. Otherwise, I understand if you can't take any action at this time with the limited information available.

from junit5-samples.

@jdai8 Any ideas?

from junit5-samples.

Unfortunately, I haven't had time to look into this more. Looks like we're covered by junit-team/junit5#2600 though. I guess we had trouble reproducing this because I'm on Java 8 and it looks like CI is running Java 11.

from junit5-samples.

Nice! Great to see we have a working, minimal reproduction sample over there.

from junit5-samples.

A fix was recently applied. It's shipping in 5.8.0-SNAPSHOT (and soon in 5.7.2).

Would it make sense to extend the Bazel sample with a SNAPSHOT-consuming configuration?

from junit5-samples.

Hmm, I'm not really a fan of SNAPSHOTs because they're not reproducible. For example, when trying to figure out which junit version broke this sample, it would have been impossible to bisect if we were depending on a snapshot.

from junit5-samples.