IndexOutOfBoundsException being dumped about maven-surefire-junit5-tree-reporter HOT 12 OPEN

Hey, thanks for all the effort with debugging!

I just discovered this issue with <exclude> 5 days ago, the ticket was raised exactly while I was developing some features in July. So using <exclude> is absolutely untested and will get problems with Parameterized and Nested tests (afaik).

Unfortunately, there's no easy fix and I will need some more time to analyze what can be done. Of course, you're very welcome to help if you want.

from maven-surefire-junit5-tree-reporter.

Since I am not sure of the conditions causing this issue, it is hard to make a minimal working reproduction. However, I seem to be able to reproduce this by in my project. Should be able to trigger it with:

$ git clone [email protected]:ascopes/java-compiler-testing
$ cd java-compiler-testing
$ git checkout 511f1e87d4ff6316ac8330bb800e16c707d83953
$ ./mvnw clean test -am -e -pl java-compiler-testing -T4

from maven-surefire-junit5-tree-reporter.

Since I am not sure of the conditions causing this issue, it is hard to make a minimal working reproduction. However, I seem to be able to reproduce this by in my project. Should be able to trigger it with:

$ git clone [email protected]:ascopes/java-compiler-testing
$ cd java-compiler-testing
$ git checkout 511f1e87d4ff6316ac8330bb800e16c707d83953
$ ./mvnw clean test -am -e -pl java-compiler-testing -T4

I followed these steps and I was not able to reproduce :/

Not so long ago I've opened an issue in the SUREFIRE Jira and for some reason it looks like that the <exclude> tag can change the behavior of how the tests are printed. (https://issues.apache.org/jira/browse/SUREFIRE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17652515#comment-17652515)

Could you do a favor, please? Could you try to run the same test, but without the <exclude> and see what happens?

from maven-surefire-junit5-tree-reporter.

The first 15 or so times I tried this tonight, it seems to have not had the issue. I spotted one run after this that did have it and it would appear to be to do with the ordering of tests.

Looks like passing -Dsurefire.runOrder.random.seed=718860759208 to the Maven invocation makes this occur each time (might be a red herring, but one to try perhaps?)

Just about to try without exclude.

from maven-surefire-junit5-tree-reporter.

Without the exclude, it doesn't seem to do it with that above seed. Just rerunning a few runs with random order again as unsetting the <exclude> in my project will cause several integration tests to run which may mess with test ordering. Will also try and attach a debugger to see if I can get any info as to what it is processing when it fails and get back to you.

from maven-surefire-junit5-tree-reporter.

Ok, so I attached a debugger and I can see that testResult.getSourceName() is returning io.github.ascopes.jct.tests.unit.compilers.AbstractJctCompilerTest$TargetTests. The getTreeLength is going to return 1 from this, I assume.

The class results list itself only has the one element though:

...so the .get(1) is going to fail there.

I managed to debug this by creating mvnwDebug scripts (./mvnw wrapper:wrapper -DincludeDebug=true), and then running ./mvnwDebug -Dsurefire.runOrder.random.seed=718860759208 -T4 -pl java-compiler-testing -am test on git ref 511f1e87d4ff6316ac8330bb800e16c707d83953 as above.

I attached a remote debugger on port 8000 from IntelliJ once I ran this, and added an exception break point on any IndexOutOfBoundsException getting raised. I then manually added the junit5-tree-reporter plugin to my POM while the debugger was running and instructed IntelliJ to refresh the project dependencies so I could get the source code for the plugin while the debugger stopped on it.

Looks like it definitely might be related to what you suggested. I am guessing this is making the assumption that the outer class runs tests before the inner class or something. The bit I am a bit confused on is that the data in classResults shows a completely different test class to the one in testResult.

Could it be a race condition somewhere perhaps? The call to printTests:111 in TestReportHandler show at least two different test classes in the list of tests being rendered.

get:4959, Collections$SingletonList (java.util)
printClass:162, TreePrinter$TestPrinter (org.apache.maven.plugin.surefire.report)
printTest:96, TreePrinter$TestPrinter (org.apache.maven.plugin.surefire.report)
access$500:85, TreePrinter$TestPrinter (org.apache.maven.plugin.surefire.report)
lambda$printTests$1:82, TreePrinter (org.apache.maven.plugin.surefire.report)
accept:-1, TreePrinter$$Lambda$247/0x000000080145e230 (org.apache.maven.plugin.surefire.report)
accept:183, ForEachOps$ForEachOp$OfRef (java.util.stream)
accept:197, ReferencePipeline$3$1 (java.util.stream)
forEachRemaining:1625, ArrayList$ArrayListSpliterator (java.util)
copyInto:509, AbstractPipeline (java.util.stream)
wrapAndCopyInto:499, AbstractPipeline (java.util.stream)
evaluateSequential:150, ForEachOps$ForEachOp (java.util.stream)
evaluateSequential:173, ForEachOps$ForEachOp$OfRef (java.util.stream)
evaluate:234, AbstractPipeline (java.util.stream)
forEach:596, ReferencePipeline (java.util.stream)
printTests:82, TreePrinter (org.apache.maven.plugin.surefire.report)
printTests:111, TestReportHandler (org.apache.maven.plugin.surefire.report)
print:44, TestReportHandler (org.apache.maven.plugin.surefire.report)
testSetCompleted:26, ConsoleTreeReporterBase (org.apache.maven.plugin.surefire.report)
testSetCompleted:32, ConsoleTreeReporterUnicode (org.apache.maven.plugin.surefire.report)
testSetCompleted:8, ConsoleTreeReporterBase (org.apache.maven.plugin.surefire.report)
testSetCompleted:213, TestSetRunListener (org.apache.maven.plugin.surefire.report)
handle:147, ForkClient$TestSetCompletedListener (org.apache.maven.plugin.surefire.booterclient.output)
handle:135, ForkClient$TestSetCompletedListener (org.apache.maven.plugin.surefire.booterclient.output)
notifyEvent:233, ForkedProcessEventNotifier (org.apache.maven.plugin.surefire.booterclient.output)
handleEvent:353, ForkClient (org.apache.maven.plugin.surefire.booterclient.output)
handleEvent:59, ForkClient (org.apache.maven.plugin.surefire.booterclient.output)
run:96, ThreadedStreamConsumer$Pumper (org.apache.maven.plugin.surefire.booterclient.output)
run:1589, Thread (java.lang)

(☝🏻 stacktrace within the IntelliJ debugger)

I'm off to bed for the night now, but if I have time tomorrow and there is any other info you need, I'll try and find it out and let you know.

Thanks for the quick response!

from maven-surefire-junit5-tree-reporter.

Cool, no worries. I may be able to look at some stuff at the weekend if I have time, so feel free to shout if there is anything in particular.

from maven-surefire-junit5-tree-reporter.