Git Product home page Git Product logo

phpunit-slow-test-detector's Introduction

phpunit-slow-test-detector

Integrate Merge Release Renew

Code Coverage Type Coverage

Latest Stable Version Total Downloads Monthly Downloads

This project provides a composer package and a Phar archive with an extension for detecting slow tests in phpunit/phpunit.

The extension is compatible with the following versions of phpunit/phpunit:

Installation

Installation with composer

Run

composer require --dev ergebnis/phpunit-slow-test-detector

to install ergebnis/phpunit-slow-test-detector as a composer package.

Installation as Phar

Download phpunit-slow-test-detector.phar from the latest release.

Usage

Bootstrapping the extension

Before the extension can detect slow tests in phpunit/phpunit, you need to bootstrap it. The bootstrapping mechanism depends on the version of phpunit/phpunit you are using.

Bootstrapping the extension as a composer package

To bootstrap the extension as a composer package when using

  • phpunit/phpunit:^6.5.0

adjust your phpunit.xml configuration file and configure the

 <phpunit
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
     bootstrap="vendor/autoload.php"
 >
+    <listeners>
+        <listener class="Ergebnis\PHPUnit\SlowTestDetector\Extension"/>
+    </listeners>
     <testsuites>
         <testsuite name="unit">
             <directory>test/Unit/</directory>
         </testsuite>
     </testsuites>
 </phpunit>

To bootstrap the extension as a composer package when using

  • phpunit/phpunit:^7.5.0
  • phpunit/phpunit:^8.5.19
  • phpunit/phpunit:^9.0.0

adjust your phpunit.xml configuration file and configure the

 <phpunit
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
     bootstrap="vendor/autoload.php"
 >
+    <extensions>
+        <extension class="Ergebnis\PHPUnit\SlowTestDetector\Extension"/>
+    </extensions>
     <testsuites>
         <testsuite name="unit">
             <directory>test/Unit/</directory>
         </testsuite>
     </testsuites>
 </phpunit>

To bootstrap the extension as a composer package when using

  • phpunit/phpunit:^10.0.0
  • phpunit/phpunit:^11.0.0

adjust your phpunit.xml configuration file and configure the

 <phpunit
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
     bootstrap="vendor/autoload.php"
 >
+    <extensions>
+        <bootstrap class="Ergebnis\PHPUnit\SlowTestDetector\Extension"/>
+    </extensions>
     <testsuites>
         <testsuite name="unit">
             <directory>test/Unit/</directory>
         </testsuite>
     </testsuites>
 </phpunit>

Bootstrapping the extension as a PHAR

To bootstrap the extension as a PHAR when using

  • phpunit/phpunit:^7.5.0
  • phpunit/phpunit:^8.5.19
  • phpunit/phpunit:^9.0.0

adjust your phpunit.xml configuration file and configure the

 <phpunit
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
     bootstrap="vendor/autoload.php"
+    extensionsDirectory="directory/where/you/saved/the/extension/phars"
 >
+    <extensions>
+        <bootstrap class="Ergebnis\PHPUnit\SlowTestDetector\Extension"/>
+    </extensions>
     <testsuites>
         <testsuite name="unit">
             <directory>test/Unit/</directory>
         </testsuite>
     </testsuites>
 </phpunit>

To bootstrap the extension as a PHAR when using

  • phpunit/phpunit:^10.0.0
  • phpunit/phpunit:^11.0.0

adjust your phpunit.xml configuration file and configure the

 <phpunit
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
     bootstrap="vendor/autoload.php"
+    extensionsDirectory="directory/where/you/saved/the/extension/phars"
 >
+    <extensions>
+        <extension class="Ergebnis\PHPUnit\SlowTestDetector\Extension"/>
+    </extensions>
     <testsuites>
         <testsuite name="unit">
             <directory>test/Unit/</directory>
         </testsuite>
     </testsuites>
 </phpunit>

Configuring the extension

You can configure the extension with the following options in your phpunit.xml configuration file:

  • maximum-count, an int, the maximum count of slow test that should be reported, defaults to 10
  • maximum-duration, an int, the maximum duration in milliseconds for a test before the extension considers it as a slow test, defaults to 500

The configuration mechanism depends on the version of phpunit/phpunit you are using.

Configuring the extension

To configure the extension when using

  • phpunit/phpunit:^6.5.0

adjust your phpunit.xml configuration file and configure the

The following example configures the maximum count of slow tests to three, and the maximum duration for all tests to 250 milliseconds:

 <phpunit
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
     bootstrap="vendor/autoload.php"
 >
     <listeners>
-        <listener class="Ergebnis\PHPUnit\SlowTestDetector\Extension"/>
+        <listener class="Ergebnis\PHPUnit\SlowTestDetector\Extension">
+            <arguments>
+                <array>
+                    <element key="maximum-count">
+                        <integer>3</integer>
+                    </element>
+                    <element key="maximum-duration">
+                        <integer>250</integer>
+                    </element>
+                </array>
+            </arguments>
+        </listener>
     </listeners>
     <testsuites>
         <testsuite name="unit">
             <directory>test/Unit/</directory>
        </testsuite>
     </testsuites>
 </phpunit>

To configure the extension when using

  • phpunit/phpunit:^7.5.0
  • phpunit/phpunit:^8.5.19
  • phpunit/phpunit:^9.0.0

adjust your phpunit.xml configuration file and configure the

The following example configures the maximum count of slow tests to three, and the maximum duration for all tests to 250 milliseconds:

 <phpunit
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
     bootstrap="vendor/autoload.php"
 >
     <extensions>
-        <extension class="Ergebnis\PHPUnit\SlowTestDetector\Extension"/>
+        <extension class="Ergebnis\PHPUnit\SlowTestDetector\Extension">
+            <arguments>
+                <array>
+                    <element key="maximum-count">
+                        <integer>3</integer>
+                    </element>
+                    <element key="maximum-duration">
+                        <integer>250</integer>
+                    </element>
+                </array>
+            </arguments>
+        </extension>
     </extensions>
     <testsuites>
         <testsuite name="unit">
             <directory>test/Unit/</directory>
        </testsuite>
     </testsuites>
 </phpunit>

To configure the extension when using

  • phpunit/phpunit:^10.0.0
  • phpunit/phpunit:^11.0.0

adjust your phpunit.xml configuration file and configure one or more

The following example configures the maximum count of slow tests to three, and the maximum duration for all tests to 250 milliseconds:

 <phpunit
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
     bootstrap="vendor/autoload.php"
 >
     <extensions>
-        <bootstrap class="Ergebnis\PHPUnit\SlowTestDetector\Extension"/>
+        <bootstrap class="Ergebnis\PHPUnit\SlowTestDetector\Extension">
+            <parameter name="maximum-count" value="3"/>
+            <parameter name="maximum-duration" value="250"/>
+        </bootstrap>
     </extensions>
     <testsuites>
         <testsuite name="unit">
             <directory>test/Unit/</directory>
        </testsuite>
     </testsuites>
 </phpunit>

Configuring the maximum duration per test case

You can configure the maximum duration for a single test case with

  • an Attribute\MaximumDuration attribute when using
    • phpunit/phpunit:^10.0.0
    • phpunit/phpunit:^11.0.0
  • a @maximumDuration annotation in the DocBlock when using
    • phpunit/phpunit:^6.5.0
    • phpunit/phpunit:^7.5.0
    • phpunit/phpunit:^8.5.19
    • phpunit/phpunit:^9.0.0
  • a @slowThreshold annotation in the DocBlock when using
    • phpunit/phpunit:^6.5.0
    • phpunit/phpunit:^7.5.0
    • phpunit/phpunit:^8.5.19
    • phpunit/phpunit:^9.0.0

The following example configures the maximum durations for single test cases to 5.000 ms, 4.000 ms, and 3.000 ms:

<?php

declare(strict_types=1);

use PHPUnit\Framework;
use Ergebnis\PHPUnit\SlowTestDetector;

final class ExtraSlowTest extends Framework\TestCase
{
    /**
     */
    #[SlowTestDetector\Attribute\MaximumDuration(5000)]
    public function testExtraExtraSlow(): void
    {
        // ...
    }

    /**
     * @maximumDuration 4000
     */
    public function testAlsoQuiteSlow(): void
    {
        // ...
    }

    /**
     * @slowThreshold 3000
     */
    public function testQuiteSlow(): void
    {
        // ...
    }
}

Note

Support for the @slowThreshold annotation exists only to help you move from johnkary/phpunit-speedtrap. It will be deprecated and removed in the near future.

Running tests

When you have bootstrapped the extension, you can run your tests as usually:

vendor/bin/phpunit

When the extension has detected slow tests, it will report them:

PHPUnit 10.0.0 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.1.0
Configuration: test/EndToEnd/Default/phpunit.xml
Random Seed:   1676103726

.............                                                                                                                                                                                                                                                                                                   13 / 13 (100%)

Detected 11 tests where the duration exceeded the maximum duration.

 1. 1.604 (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Default\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider#9
 2. 1.505 (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Default\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider#8
 3. 1.403 (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Default\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider#7
 4. 1.303 (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Default\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider#6
 5. 1.205 (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Default\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider#5
 6. 1.103 (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Default\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider#4
 7. 1.005 (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Default\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider#3
 8. 0.905 (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Default\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider#2
 9. 0.805 (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Default\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider#1
10. 0.705 (0.500) Ergebnis\PHPUnit\SlowTestDetector\Test\EndToEnd\Default\SleeperTest::testSleeperSleepsLongerThanDefaultMaximumDurationWithDataProvider#0

There is 1 additional slow test that is not listed here.

Time: 00:12.601, Memory: 8.00 MB

OK (13 tests, 13 assertions)

Understanding measured test durations

When using

  • phpunit/phpunit:^6.5.0

  • phpunit/phpunit:^7.5.0

  • phpunit/phpunit:^8.5.19

  • phpunit/phpunit:^9.0.0

  • the extension uses the hooks event system of phpunit/phpunit.

The hooks event system supports eleven hook methods that phpunit/phpunit invokes during the execution of tests.

When the extension uses the hooks event system, it uses the PHPUnit\Runner\AfterTestHook, which receives the duration of invoking PHPUnit\Framework\TestCase::runBare() and more.

When phpunit/phpunit invokes PHPUnit\Framework\TestCase::runBare(), it will invoke the following methods before the first test method in the class:

When phpunit/phpunit invokes PHPUnit\Framework\TestCase::runBare(), it will invoke the following methods before every test method in the class:

When phpunit/phpunit invokes PHPUnit\Framework\TestCase::runBare(), it will invoke the following methods after every test method in the class:

When phpunit/phpunit invokes PHPUnit\Framework\TestCase::runBare(), it will invoke the following methods after the last test method in the class:

Note

Because of this behavior, the measured test durations can and will vary depending on the order in which phpunit/phpunit executes tests.

When using

  • phpunit/phpunit:^10.0.0
  • phpunit/phpunit:^11.0.0

the extension uses the new event system of phpunit/phpunit.

The new event system supports a wide range of events that phpunit/phpunit emits during the execution of tests.

When the extension uses the new event system, it uses and subscribes to the PHPUnit\Event\Test\PreparationStarted and PHPUnit\Event\Test\Finished events and measures the duration between the points in time when phpunit/phpunit emits the former and the latter.

When phpunit/phpunit invokes PHPUnit\Framework\TestCase::runBare(), it will invoke the following methods before the first test method in the class:

When phpunit/phpunit invokes PHPUnit\Framework\TestCase::runBare(), it will invoke the following methods before every test method in the class:

When phpunit/phpunit invokes PHPUnit\Framework\TestCase::runBare(), it will invoke the following methods after every test method in the class:

When phpunit/phpunit invokes PHPUnit\Framework\TestCase::runBare(), it will invoke the following methods after the last test method in the class:

Note

Because of this behavior, the measured test durations can and will vary depending on the order in which phpunit/phpunit executes tests.

Changelog

The maintainers of this project record notable changes to this project in a changelog.

Contributing

The maintainers of this project suggest following the contribution guide.

Code of Conduct

The maintainers of this project ask contributors to follow the code of conduct.

General Support Policy

The maintainers of this project provide limited support.

You can support the maintenance of this project by sponsoring @localheinz or requesting an invoice for services related to this project.

PHP Version Support Policy

This project supports PHP versions with active and security support.

The maintainers of this project add support for a PHP version following its initial release and drop support for a PHP version when it has reached the end of security support.

Security Policy

This project has a security policy.

License

This project uses the MIT license.

Credits

This package is inspired by johnkary/phpunit-speedtrap, originally licensed under MIT by John Kary.

Social

Follow @localheinz and @ergebnis on Twitter.

phpunit-slow-test-detector's People

Contributors

dependabot[bot] avatar ergebnis-bot avatar hypemc avatar kukulich avatar localheinz avatar mvorisek avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

phpunit-slow-test-detector's Issues

PHAR dependencies are not scoped yet

Steps required to reproduce the problem

I prepared a small repo with GitHub Actions to reproduce the issue here.

Problematic code:

        $parser = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
        $stmts = $parser->parse('<?php var_dump("test");');
        foreach ((new NodeFinder())->findInstanceOf($stmts, FuncCall::class) as $funcCalls) {
            $funcCalls->name->getParts();
        }

image

The problem in this example is that PhpParser\Node\Name is not used from the project autoloader but from the one in the PHAR.

Expected Result

I would expect that dependencies of that PHAR are prefixed and do not collide with project dependencies.

Actual Result

Dependencies of that PHAR are not prefixed and collide with project dependencies.

DX: Add Docker dev setup

Hello everyone!
I want to thank you so much for the amazing work done on this extension.
I really appreciate your work and would like to suggest adding docker to improve the contributions from the whole community.

To give you an idea of ​​what I mean I leave you a link to PHP-CS-FIXER that have already implemented this PHP-CS-Fixer/PHP-CS-Fixer#5234

Best regards

P.s. If you think it's something useful, I'll be happy to send my PR to add this feature!

Compatibility with PHP 8.0 and lower

#341 added phpunit 9.x support but it was releases after the compatibility with older PHP versions was dropped in #87 and #171.

In order to fully benefit from #341 and to allow this dependency in projects with PHP 8.1- support, the PHP version requirement must be lowered.

The latest johnkary/phpunit-speedtrap, at the time of writing this, requires PHP 7.2+ support - https://github.com/johnkary/phpunit-speedtrap/blob/master/composer.json#L22 so reintroducing support since this version would be welcomed to allow this package to be a real replacement.

Once the support for older PHP versions is added and at least one version tagged, the support for older PHP versions can be then removed again.

How to reproduce:

  1. add ergebnis/phpunit-slow-test-detector dependency to a PHP 7.4 - 8.3 project
  2. run composer update on/using PHP 7.4 platform

Measured time must include test setup/teardown

Currently the measured time does not include test teardown.

The solution is to drop
https://github.com/ergebnis/phpunit-slow-test-detector/blob/2.4.0/src/Subscriber/TestPassedSubscriber.php
as it is called in
https://github.com/sebastianbergmann/phpunit/blob/10.4.2/src/Framework/TestCase.php#L723
and use
https://github.com/sebastianbergmann/phpunit/blob/10.4.2/src/Framework/TestRunner.php#L238
"test finished" hook solely instead.

In "test finished" hook TestCase->status()->isFailure() && TestCase->status()->isError() can be used to filter out non-passing tests, but probably even non-passing tests can/should be reported if slow.

inefficient test detection

I wonder whether this extension could also rank tests by there efficency, meaning not just a ranking by the runtime involved, but a separate ranking for e.g. memory usage.

wdyt?

Measure duration of setup (before) & teardown (after) test class

TestCase::setUpBeforeClass and TestCasetearDownAfterClass are not measured now and cannot be measured now, as currently the duration is measured per test.

What about measuring these too and reporting it as Cl::setUpBeforeClass/Cl::tearDownAfterClass items?

basic tests added in: #397

please note this must work with test process isolation correctly, where it might be counted into individual test duration

one example of (possibly) slow TestCase::setUpBeforeClass is https://github.com/atk4/ui/blob/5.0.0/tests/DemosHttpTest.php#L35, ie. things like starting (shared) webserver for a whole test class

related with #378, but not blocking

Values from data provider are not shown for slow test

Steps required to reproduce the problem

Basically the same as sebastianbergmann/phpunit#5616 but for this extension. Something like sebastianbergmann/phpunit@5d04989#diff-0185eeda9f2d10b7587e9a551c2658419c534b49fa2a6dfe0dc0a72d294557ceR522 is needed.

I would expected the data provider values to be shown the same way as phpunit list failed tests. This is actually working for PHPUnit 9.x and lower already.

Expected Result

(from phpunit 9.x)

Detected 4 tests where the duration exceeded the maximum duration.

1. 6.049 (0.500) Atk4\Data\Tests\ConditionSqlTest::testLikeCondition with data set #1 ('binary', true)
2. 6.041 (0.500) Atk4\Data\Tests\ConditionSqlTest::testLikeCondition with data set #2 ('text', false)
3. 5.897 (0.500) Atk4\Data\Tests\ConditionSqlTest::testLikeCondition with data set #3 ('blob', true)
4. 5.861 (0.500) Atk4\Data\Tests\ConditionSqlTest::testLikeCondition with data set #0 ('string', false)

Actual Result

(from phpunit 11.x, ignore different tests listed)

Detected 5 tests where the duration exceeded the maximum duration.

1. 3.128 (0.500) Atk4\Data\Tests\ScopeTest::testConditionOnReferencedRecords
2. 0.999 (0.500) Atk4\Data\Tests\ConditionSqlTest::testRegexpCondition#0
3. 0.968 (0.500) Atk4\Data\Tests\ConditionSqlTest::testRegexpCondition#3
4. 0.967 (0.500) Atk4\Data\Tests\ConditionSqlTest::testRegexpCondition#2
5. 0.953 (0.500) Atk4\Data\Tests\ConditionSqlTest::testRegexpCondition#1

Output is suppressed when using paratest

Steps required to reproduce the problem

  1. require phpunit/phpunit:^10.2.2
  2. require brianium/paratest:^7.1.0
  3. require ergebnis/phpunit-slow-test-detector:^2.3.0
  4. configure and bootstrap extension from ergebnis/phpunit-slow-test-detector
  5. use brianium/paratest to run tests

Expected Result

  • test results show output from extension

Actual Result

  • test results do not show output from extension

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.