mockito / mockito-testng Goto Github PK

Because of Bintray, JCenter will be sunset we have to switch publishing from Bintray to OSSRH until February 28th.

https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

useful link:

• https://central.sonatype.org/pages/gradle.html

If someone has experience in this topic help will be welcome.

Hi,

I'm using the @Listeners(MockitoTestNGListener.class) (mockito-testng 0.2.10) in a project to avoid all those open/close mocks.

While is it is running fine with some of use cases, I found a strange behavior:
Here a simple test class:

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.entry;
import static org.mockito.Mockito.when;

import java.util.Map;

import org.assertj.core.data.MapEntry;
import org.mockito.Mock;
import org.mockito.testng.MockitoTestNGListener;
import org.testng.annotations.BeforeTest;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(MockitoTestNGListener.class)
public class MockitoTestNgListenerTest1 {

    @Mock
    private Map<String, Object> map;

    private MapEntry<String, String> entry;

    @SuppressWarnings("rawtypes")
    @BeforeTest
    public void setUp() {
        entry = entry("Boo", "Minsc");
        when(map.get(0)).thenReturn(entry);
    }

    @Test
    public void shouldWork() {
        assertThat(map.get(0)).isEqualTo(entry);
    }
}

No big deal. It runs ok with TestNG.

Now duplicate this class and rename it. Now launch the 2 classes and the exception is thrown. All unit tests fail.

org.mockito.exceptions.misusing.UnfinishedMockingSessionException: 
Unfinished mocking session detected.
Previous MockitoSession was not concluded with 'finishMocking()'.
For examples of correct usage see javadoc for MockitoSession class.

	at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1133)
	at org.testng.internal.invokers.InvokedMethodListenerInvoker.invokeListener(InvokedMethodListenerInvoker.java:56)
	at org.testng.internal.BaseInvoker.runInvokedMethodListeners(BaseInvoker.java:55)
	at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:351)
	at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:320)
	at org.testng.TestRunner.invokeTestConfigurations(TestRunner.java:617)
	at org.testng.TestRunner.beforeRun(TestRunner.java:607)
	at org.testng.TestRunner.run(TestRunner.java:578)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
	at org.testng.TestNG.runSuites(TestNG.java:1069)
	at org.testng.TestNG.run(TestNG.java:1037)
	at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
	at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

Note that those simple tests run fine with open/close mocks (or if the BeforeTest is changed to BeforeMethod).

Thanks.
------- Maven dependencies used:
JDK 11.0.6 / 11.0.8
Mockito-testng 0.2.10
TestNG version 7.3.0
Mockito 3.7.7

Release job hangs on bintrayPublish
During standard run no more useful logs are produced

2020-12-09T11:15:57.2854953Z > Task :bintrayPublish
2020-12-09T12:26:44.3516989Z ##[error]The operation was canceled.

Artifacts are deployed to bintray, but not synced to Maven Central.

So I suspect, that sync take too much time and during wait for status no output is produced what can be cause to kill some thread or process by github.

MockitoTestNGListener re-uses MockitoSessions between test methods. Instead of giving each test method its own independent mocks, it calls 'reset' on the mocks between test methods.

The effect of this is that mock instances are the same for multiple test methods, which can lead to confusing behavior, for example if the mock object was passed to a separate thread in one test, that separate thread can continue to make calls to the mock during later tests, causing spurious results.

• Mockito documentation states "Normally, you don't need to reset your mocks, just create new mocks for each test method."
  https://static.javadoc.io/org.mockito/mockito-core/2.27.0/org/mockito/Mockito.html#resetting_mocks
• Both MockitoRule and MockitoJUnitRunner documentation state "Mocks are initialized before each test method."

example:

@Listeners(MockitoTestNGListener.class)
public class InjectMocksWithConstructorAndFinalTest {

    interface Client {}

    static class Service {
        final Client client;

        public Service(Client client) {
            this.client = client;
        }
    }

    @Mock
    private Client client;

    @InjectMocks
    private Service service;

    @AfterMethod
    void cleanup() {
        service = null;
    }
   
  @Test(invocationCount = 4)
    void inject_mock_should_be_refreshed() {
      ....
    }
}

For each test method new @Mocks are created but service is initialized only once at the beginning.
And @InjectMocks cannot insert into final field, so we must clear service in @AfterMethod

In other case we will have in service reference to old mock client object.

mockito/mockito#1437

Mockito is converging on using strict stubbing mechanism so let's configure TestNGListener to also use strict stubs by default.

Strict stubbing in Mockito:

• default for JUnit5 Mockito
• default for MockitoSession API
• planned as default for Mockito 3 for JUnit 4 rules + runners

Since TestNG integration is not announced yet, and it has 0.* version, it is ok to update the defaults now.

Thoughts/feedback?

My reason for doing this is outlined in this issue comment. It's available in JUnitRule but not MockitoTestNGListener. At a minimum, strictness should be configurable as a suite parameter. Overriding the strictness at a lower level is out of scope.

Set in testng.xml

<?xml version="1.0" encoding="UTF-8"?>
<suite name="strictness">
  <parameter name="org.mockito.testng.strictness"  value="LENIENT"/>
  <!-- etc. -->
</suite>

Set in annotation

@BeforeSuite // or any TestNG annotation for that matter
public void beforeSuite(ITestContext iTestContext) {
  iTestContext.getSuite().getXmlSuite().setParameter("org.mockito.testng.strictness", Strictness.LENIENT);
}

The MockitoTestNGListener does not close the session of before methods in case an exception is thrown in the before method. All follow-up tests will fail. This is a problem especially in test pipelines when you run 100s of tests. Finding the cause takes longer than necessary. In these scenarios it also appears that getting the class name from ITestResult returns null. This causes an NPE when calling hasMockitoTestNGListener.

I would propose the following solutions in the listener:

protected boolean hasMockitoTestNGListener(ITestResult testResult) {
		return nonNull(testResult.getTestClass())
				&& findAnnotation(testResult, Listeners.class)
				.map(Listeners::value)
				.map(Arrays::stream)
				.orElseGet(Stream::empty)
				.anyMatch(listener -> listener == MockitoTestNGListener.class);
}

Important part is to return true in shouldBeRunAfterInvocation in case an exception was thrown in the before method.

private boolean shouldBeRunAfterInvocation(IInvokedMethod method, ITestResult testResult) {
	return (method.isTestMethod() && hasMockitoTestNGListener(testResult))
		|| (isBeforeMethod(method) && nonNull(testResult.getThrowable()));
}

Hope this makes sense. I actually ran into the situation and applying these changes fixed it for me. With the fix I only see the tests from the same test class failing and not other tests afterwards as well.

Similar to mockito/mockito#2089

I've just tried it and got:

Last Sync Errors: Invalid POM: /org/mockito/mockito-testng/0.1.0/mockito-testng-0.1.0.pom: Developer information missing, Invalid version for Dependency {groupId=org.mockito, artifactId=mockito-core, version=2.+, type=jar} - uses invalid dynamic version syntax. Replace with a fixed version or standard mathematical notation e.g., [1.5,) for version 1.5 and higher. Dropping existing partial staging repository.

We can annotate test with:

    @Mock
    private MyType mock;

    @Test(threadPoolSize = 3, invocationCount = 8)
    public void testParallel() {
     ....
    }

In this case MockitoTestNGListener will not work as we want.
In current implementation MockitoSession is bind to current test instance, but TestNG create one test instance and run test in multiple thread.

Issue need more investigate how to resolve problem.

We must to check and discover how mocked field should be distinguished on one test instance and multiple thread.

Please like / react on issue to we see how many people are interested this issue.

similar to mockito/mockito#2069

Many build from the same code should produce the same binary output