not sure why the runner fails on 3.8 (and also 3.9) right now...

is it maybe related to the changes in sebastianbergmann/phpunit@13f77b7 ?

Do you have an idea? Just want to make sure that we can fix it before 3.8 gets released..

When running multiple tests in parallel the database of phpunit-clever-and-smart might locked for a short moment. Sometimes this leads to phpunit aborting execution.

Here's the error message with stack trace:

Warning:  SQLite3::query(): Unable to prepare statement: 5, database is locked in /myapp/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php on line 185
Stack trace:
   1. {main}() /myapp/vendor/phpunit/phpunit/phpunit:0
   2. PHPUnit_TextUI_Command::main() /myapp/vendor/phpunit/phpunit/phpunit:36
   3. PHPUnit_TextUI_Command->run() /myapp/vendor/phpunit/phpunit/src/TextUI/Command.php:105
   4. PHPUnit_TextUI_TestRunner->doRun() /myapp/vendor/phpunit/phpunit/src/TextUI/Command.php:153
   5. PHPUnit_TextUI_TestRunner->handleConfiguration() /myapp/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:153
   6. PHPUnit_Util_Configuration->getListenerConfiguration() /myapp/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:714
   7. PHPUnit_Util_XML::xmlToVariable() /myapp/vendor/phpunit/phpunit/src/Util/Configuration.php:342
   8. PHPUnit\Runner\CleverAndSmart\Storage\Sqlite3Storage->__construct() /myapp/vendor/phpunit/phpunit/src/Util/XML.php:245
   9. PHPUnit\Runner\CleverAndSmart\Storage\Sqlite3Storage->query() /myapp/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php:34
  10. PHPUnit\Runner\CleverAndSmart\Storage\Sqlite3Storage->doQuery() /myapp/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php:159
  11. SQLite3->query() /myapp/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php:185
 Fatal error:  Uncaught exception 'PHPUnit\Runner\CleverAndSmart\Exception\StorageException' with message 'database is locked (error code 5)' in /myapp/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/Exception/StorageException.php:8

Maybe we can change this behaviour to "wait a second and try again" or at least to ignore this exception (maybe print a warning to stderr).

would you accept a PR adding PHP 5.3 compat?

would be great to have a parameter in phpunit.xml which defines where the sql-lite db should be stored

It would be cool to use the last modification date of the file where the SUT is located for sorting.

I am not sure if it is at all possible, but I think we can discuss things here:

Atm I don't see a chance to use the testlistener in travis builds because all storage results are local in the build and cannot persist between ci builds.

One solution could use a remote db/webservice as a remote storage for travis builds. But maybe there is even a easier way to achieve the same thing?

• Use single table 'phpunit_0_3_0_test_results`
• Take failure timing into account
• Record test state next to timing
  -- Failure: 4
  -- Error: 3
  -- Skipped: 2
  -- Incomplete: 1
  -- Passed: 0

to measure the overhead between different phpunit versions, we should add benchmarks also for phpunit 3.8 & 3.9.

we expect a certain perf boost because of changes in sebastianbergmann/phpunit@af5610d

will be done after #21 landed.

Skipped and incomplete tests have a high chance to pass next time. Run them after failures. Necessary changes:

• Change TestListener
• Change PrioritySorter (and unit test)
• Add integration test

More like a feature request.
I once worked on a huge project where we couldn't run tests in isolation mode because of the amount of time that added on top of the already lengthy cycle. Without knowing we introduced a couple dependencies between tests that resulted in a lot of sh*t later. Having the option to randomise execution would've prevented this sort of behaviour.

Think of a clever way to keep the database clean

Ideas for strategies

• Multiples of average of tests count
• Result age
• SQLite file size
  -- Find out size when it becomes significantly slower

Investigate if pushing each insert/update to a queue and and execute it in batches in a shutdown function would reduce IO and speedup things.

Handle Ctrl+c through signal handling.

Add a testsuite of a stable external project (maybe just a released version) and compare suite runtime with and without clever&smart test listener to measure the actual overhead

Just for discussion...

There is a visualisation phpunit plugin which shows the slowest tests.
https://github.com/marmelab/phpunit-d3-report

Maybe we can use it to verify our sorting for runtime related tests is accurate

I think it would be cool to also use the amount of time a test takes for sorting. Longer tests should ideally be run later to create potentially failing tests faster.

Capture fatal errors through register_shutdown_function to rerun them first.

some classes define some crazy and un-used use statements. cleanup on next sweep.

e.g.
https://github.com/lstrojny/phpunit-clever-and-smart/blob/master/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php#L4-L9

Some tests are being marked as skipped because they depend on another test. The test in question passed but the others were skipped.

This is happenning in a private repo of which I can't provide the code, but i'll try to reproduce the issue in a public repo ASAP.

Hey there and thanks for CaS, it's a little tool I love (and believe it should be part of PHPUnit's core).

I just had some problems getting the listener to work with Travis CI. Usually paths like src/ or build/... resolve in PHPUnit's XML config.

When I configure CaS with a relative path I get the following problem:

Stack trace:
#0 /home/travis/build/mihaeu/movie-manager/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php(24): SQLite3->__construct('build/.phpunit-...')
#1 [internal function]: PHPUnit\Runner\CleverAndSmart\Storage\Sqlite3Storage->__construct('build/.phpunit-...')
#2 phar:///home/travis/.phpenv/versions/5.4.32/bin/phpunit/phpunit/Util/XML.php(276): ReflectionClass->newInstanceArgs(Array)
#3 phar:///home/travis/.phpenv/versions/5.4.32/bin/phpunit/phpunit/Util/Configuration.php(374): PHPUnit_Util_XML::xmlToVariable(Object(DOMElement))
#4 /home/travis/build/mihaeu/movie-manager/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(722): PHPUnit_Util_Configuration->getListenerConfiguration()
#5 /home/travis/build/mihaeu/mo in /home/travis/build/mihaeu/movie-manager/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php on line 24

https://travis-ci.org/mihaeu/movie-manager/jobs/37203163#L249

I played around a bit and solved it by extending Sqlite3Storage

<?php

namespace Mihaeu\MovieManager\Tests;

use PHPUnit\Runner\CleverAndSmart\Storage\Sqlite3Storage;

class Sqlite3TestStorage extends Sqlite3Storage
{
    public function __construct($fileName = '.phpunit-cas.db')
    {
        $fileName = realpath(__DIR__.'/../../..').'/'.$fileName;
        if (!file_exists(dirname($fileName))) {
            mkdir(dirname($fileName), 0777, true);
        }
        parent::__construct($fileName);
    }
}

Of course it's not a nice solution, but it works in my case (the path has to be adjusted of course if loaded from a vendor dir). In the case of Travis getcwd() could be used to find the base path as well. Of course using CaD on Travis doesn't make much sense, because there are no repeated test runs, but it would be nice to be able to use the same phpunit.xml

What do you think?

The following test will crash PHPUnit if phpunit-clever-and-smart is registered as a test listener:

/**
 * Class ExampleCrashTest
 *
 * @requires function i_dont_exist
 */
class ExampleCrashTest extends PHPUnit_Framework_TestCase
{

}

Apparently, if an entire class is skipped instead of a single test method, a PHPUnit_Framework_TestSuite object is passed to \PHPUnit\Runner\CleverAndSmart\TestListener::addSkippedTest. This object implements the \PHPUnit_Framework_Test interface. However in \PHPUnit\Runner\CleverAndSmart\Storage\Sqlite3Storage::record, suddenly a \PHPUnit_Framework_TestCase object is required. The typehint fails, crashing the phpunit process.

Fatal error: Uncaught exception 'ErrorException' with message 'Argument 2 passed to PHPUnit\Runner\CleverAndSmart\Storage\Sqlite3Storage::record() must be an instance of PHPUnit_Framework_TestCase, instance of PHPUnit_Framework_TestSuite given, called in /Users/willem/Sites/mollie/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/TestListener.php on line 107 and defined' in /Users/willem/Sites/mollie/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/Storage/Sqlite3Storage.php on line 73

Call stack:

Call Stack:
    0.0003     225232   1. {main}() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/phpunit:0
    0.0099     712344   2. PHPUnit_TextUI_Command::main() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/phpunit:47
    0.0099     712984   3. PHPUnit_TextUI_Command->run() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/TextUI/Command.php:109
    0.0099     715912   4. PHPUnit_TextUI_Command->handleArguments() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/TextUI/Command.php:120
    0.0160    1146544   5. PHPUnit_TextUI_Command->handleBootstrap() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/TextUI/Command.php:633
    0.0166    1154992   6. PHPUnit_Util_Fileloader::checkAndLoad() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/TextUI/Command.php:804
    0.0168    1155152   7. PHPUnit_Util_Fileloader::load() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/Util/Fileloader.php:38
    0.0170    1167880   8. include_once('/Users/willem/Sites/mollie/tests/bootstrap.php') /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/Util/Fileloader.php:56
    0.6078    5443552   9. require_once('/Users/willem/Sites/mollie/scripts/phpstorm/ide-phpunit-9.php') /Users/willem/Sites/mollie/tests/bootstrap.php:47
    0.6102    5496072  10. IDE_Base_PHPUnit_TextUI_Command::main() /Users/willem/Sites/mollie/scripts/phpstorm/ide-phpunit-9.php:553
    0.6102    5496696  11. PHPUnit_TextUI_Command->run() /Users/willem/Sites/mollie/scripts/phpstorm/ide-phpunit-9.php:296
    0.6202    6172160  12. PHPUnit_TextUI_TestRunner->doRun() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/TextUI/Command.php:158
    0.6437    6470952  13. PHPUnit_Framework_TestSuite->run() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:436
    0.6551    6810536  14. PHPUnit_Framework_TestResult->addFailure() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/Framework/TestSuite.php:705
    0.6740    8401768  15. PHPUnit\Runner\CleverAndSmart\TestListener->addSkippedTest() /Users/willem/Sites/mollie/vendor/phpunit/phpunit/src/Framework/TestResult.php:329
    0.6741    8401848  16. PHPUnit\Runner\CleverAndSmart\Storage\Sqlite3Storage->record() /Users/willem/Sites/mollie/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/TestListener.php:107
    0.6741    8403248  17. PHPUnit_Util_Fileloader::{closure:/Users/willem/Sites/mollie/core/tests/bootstrap.php:15-23}() /Users/willem/Sites/mollie/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/TestListener.php:73

• php version

PHP 7.2.34-1+ubuntu18.04.1+deb.sury.org+1 (cli) (built: Oct 6 2020 12:01:28) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.2.34-1+ubuntu18.04.1+deb.sury.org+1, Copyright (c) 1999-2018, by Zend Technologies

• phpunit version

PHPUnit 7.5.20 by Sebastian Bergmann and contributors.

• phpumit.xml

<?xml version="1.0" encoding="UTF-8"?>
<phpunit backupGlobals="false"
         backupStaticAttributes="false"
         bootstrap="vendor/autoload.php"
         convertErrorsToExceptions="true"
         convertNoticesToExceptions="true"
         convertWarningsToExceptions="true"
         processIsolation="false"
         stderr="true"
         stopOnFailure="false"
         verbose="true"
         colors="true">
   ...
    <listeners>
        <listener class="PHPUnit\Runner\CleverAndSmart\TestListener">
            <arguments>
                <object class="PHPUnit\Runner\CleverAndSmart\Storage\Sqlite3Storage"/>
            </arguments>
        </listener>
    </listeners>
    ...
</phpunit>

• error message

 PHP Fatal error:  Interface 'PHPUnit_Framework_TestListener' not found in /path/vendor/lstrojny/phpunit-clever-and-smart/src/PHPUnit/Runner/CleverAndSmart/TestListener.php on line 15

How can I solve the problem that PHPUnit_Framework_TestListener does not exist?

I think it would help debugging and configure the runner if it would output a message like Running testsuite using clever-and-smart before the actual test suite run so people see immediately that they use a different setup...

could you elaborate a bit, why you have chosen sqlite3 over sqlite-pdo?