The rush to produce features increased the technical debt (redundant imports, etc)

In particular we need to remove hashes of URLs to URIs for performance reasons

Some 3rd party tools will work better with PDSL if they have the sentences that were run in the test case.

If the TestCase is visible they can grab the Filtered Context off of the test case to get the phrases.

A tag in with a UUID in double quotes in parenthesis doesn't match tag expressions like it should

I am guessing you uploaded the uber/fat jar to the Maven central.
In that case the supplier of you framework/dependency is not able to exclude some dependency (transitive) that can came from your repo.
And regular maven mechanism to exclude dependencies does not work, and is required to implement some work around (re-pack your artifact and delete some classes)

This approach does not work with the uber/fat jar:

   <dependency>
      <groupId>com.google.pdsl</groupId>
      <artifactId>pdsl</artifactId>
      <version>${pdsl.version}</version>
      <exclusions>

        <exclusion>
          <groupId>commons-logging</groupId>
          <artifactId>commons-logging</artifactId>
        </exclusion>

        <exclusion>
          <groupId>ch.qos.logback</groupId>
          <artifactId>logback-classic</artifactId>
        </exclusion>

        <exclusion>
          <groupId>ch.qos.logback</groupId>
          <artifactId>logback-core</artifactId>
        </exclusion>

        <exclusion>
          <groupId>org.jacoco</groupId>
          <artifactId>jacoco-maven-plugin</artifactId>
        </exclusion>

        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-api</artifactId>
        </exclusion>

        <exclusion>
          <groupId>org.sonatype.sisu</groupId>
          <artifactId>sisu-guice</artifactId>
        </exclusion>

      </exclusions>
    </dependency>

Maven documentation:
Dependency management - this allows project authors to directly specify the versions of artifacts to be used when they are encountered in transitive dependencies or in dependencies where no version has been specified. In the example in the preceding section a dependency was directly added to A even though it is not directly used by A. Instead, A can include D as a dependency in its dependencyManagement section and directly control which version of D is used when, or if, it is ever referenced.

Establish a convention similar to polymorphicDslAllRules that can parse the grammar checking for specific violations before creating the test specification.

Shade breaks dependency management for other users who use our library.
We used shade because it didn't force users to import every dependency PDSL uses, but this is having unacceptable side effects for our users.

Build the library in a way that allows it to use transitive dependencies while simultaneously not requiring users to import every dependency we rely on ourselves.

For Gherkin-style tests ANTLR4 gives us two choices for the listener: an abstract class with empty implementations for everything or an interface.

The problem with the abstract class is it is too easy for the grammar to change and other clients will not realize they need to provide an implementation. This can lead to false positives.

The interface requires implementing a lot of methods that have no value (such as the exit methods).

It would be great to have a third option that provided an empty implementation for all exit methods as well as any rules that follow some convention (such as a '_' character at the end of the rule name. In this manner the user will only need to provide implementations for the enter methods of consequence and changes will allow other clients to see they need to add new functionality.

We show how to use PDSL using a test runner, but this doesn't explain how it works at the lowest levels. Create documentation that explains how the framework behaves, step by step.

We are observing the interruption in Maven surefire report generation stages.
If the TC pass, it all goes fine, but if the TC FAILED we see this stackTrace:

# Created at 2023-04-13T20:49:17.349
org.apache.maven.surefire.testset.TestSetFailedException: Test mechanism :: null
	at org.apache.maven.surefire.common.junit4.JUnit4RunListener.rethrowAnyTestMechanismFailures(JUnit4RunListener.java:223)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:167)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.NullPointerException
	at com.pdsl.reports.TestRunResults$FailedTestResults.getStackTrace(TestRunResults.java:49)
	at org.apache.maven.surefire.report.SmartStackTraceParser.<init>(SmartStackTraceParser.java:62)
	at org.apache.maven.surefire.common.junit4.JUnit4StackTraceWriter.smartTrimmedStackTrace(JUnit4StackTraceWriter.java:98)
	at org.apache.maven.surefire.booter.ForkingRunListener.encode(ForkingRunListener.java:409)
	at org.apache.maven.surefire.booter.ForkingRunListener.encode(ForkingRunListener.java:393)
	at org.apache.maven.surefire.booter.ForkingRunListener.toString(ForkingRunListener.java:348)
	at org.apache.maven.surefire.booter.ForkingRunListener.testError(ForkingRunListener.java:171)
	at org.apache.maven.surefire.common.junit4.JUnit4RunListener.testFailure(JUnit4RunListener.java:128)
	at org.junit.runner.notification.SynchronizedRunListener.testFailure(SynchronizedRunListener.java:94)
	at org.junit.runner.notification.RunNotifier$6.notifyListener(RunNotifier.java:177)
	at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
	at org.junit.runner.notification.RunNotifier.fireTestFailures(RunNotifier.java:173)
	at org.junit.runner.notification.RunNotifier.fireTestFailure(RunNotifier.java:167)
	at org.apache.maven.surefire.common.junit4.Notifier.fireTestFailure(Notifier.java:114)
	at com.pdsl.runners.junit.PdslGherkinJUnit4Runner.runChild(PdslGherkinJUnit4Runner.java:233)
	at com.pdsl.runners.junit.PdslGherkinJUnit4Runner.run(PdslGherkinJUnit4Runner.java:190)
	at com.pdsl.runners.PdslTestSuite.runChild(PdslTestSuite.java:172)
	at com.pdsl.runners.PdslTestSuite.runChild(PdslTestSuite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at com.pdsl.runners.PdslTestSuite.run(PdslTestSuite.java:148)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	... 4 more

Currently the PDSL framework produces poor, unreadable error messages when it encounters a sentence it doesn't recognize. It also produces errors and noise when behaving as expected.

We need to handle the following scenarios properly

1. We've added a new phrase to the grammar but the old test doesn't recognize it
   • Produce an error message explaining clearly where the lexer/parser error is
   • Halt execution
2. We've got garbage in our grammar
   • Produce an error message explaining clearly where the lexer/parser error is
   • Halt execution
3. The phrase is in the grammar BUT we want to filter it out
   • Don't produce any error messages for the sentence we're intentionally filtering out

I suspect we'll need to return to an earlier API design where we have a listener that is aware of all valid rules and is responsible for checking syntax and a 2nd listener that actually executes on a sub-dialect of the grammar.

System.out can be used by malicious attackers to gather information for a future attack.

We somewhat mitigated this by introducting the PdslThreadSafeOutputStream as the default printer, however this class has a vulnerability in its static constructor. Because we manually associate a handler with it we have a risk (albeit lower than previously) of a security exploit.

We might be able to get around this by writing our own logger or something else. We might just accept the risk as it is admittedly low.

JUnit 4 works for Google, but everyone else has moved on. Create a new runner using modern JUnit.

The rushed initial commits do not have sufficient code health. There is a general need for cleanup as the codebase exists today.

The current implementation requires phrases to be broken up individually and matched.Our TestSpecificationFactory essentially breaks up the input into sentences and then uses a rule that can match any of them and goes from there. This doesn't really constitute much of a grammar.

For example, the following nonsensical example is legal:

Given I shut the door
When I walk through the door
Then I can open the door

A grammar rule can be invoked that could require that the DSL we are testing be more formal. It could be empowered to complain when it encounters the above example to require a door to be opened before it is walked through, or open before it is shut.

The names are unique, but too long. Create a short display name.

We can perhaps cache the names of all tests and shorten them to the scenario name unless there is a conflict.

The PDSL framework is unique in that a test specification can apply to integration, API, component and unit tests. Because of that failures can be easily isolated to a specific tier of an application.

The TestRunResults should be capable of producing a user friendly report that is useful for root cause analysis

Currently all the stack traces print at the end of the test. It's difficult to map the failures to the particular test that failed.

Print the stack trace at the moment of failure.

Multiple failure causes the stack traces to be dumped at the end one after another. There is no demarcation between the issues and no association with the original tests that ran them. This makes it more difficult than needed to debug.

The terminal output looks fine, but in intellJ it is preceeded by a bunch of 0 characters: It appears to appear as whitespace in the terminal (and in this message), but if you highlight the text before "Given a new customer" you'll notice a bunch of characters. IntelliJ apparently renders these as '0'
Given a new customer

Then the customer receives an order confirmation

And the new customer has successfully created an account for future purchases

The JUnit runner essentially treats each annotated method as a test suite. The tests run and are displayed correctly, but there is an empty result posted for the test suite itself. This is difficult to interpret and may be mistaken for noise.

Some users of the framework ran into dependency injection trouble because when we create the PdslExecutable we call the suppliers eagerly to construct the test.

Other people wanted to have the JUnit5 hooks call before the visitor/listener suppliers were. We did a release for this, but we should lazily evaluate the executor and anything else we can sanely delay.

Some usages of the annotation have created weird uses of inheritance to accommodate the need for a default constructor.

We should modify the listener field to follow a provider pattern or something more mature.

@PdslTest(
      includesResources = "Some.feature",
      parser = SomeParser.class,
      lexer = SomeLexer.class,
      listener = PortalProspectListener.class
)

A python and typescript implementation of PDSL is going to be necessary. At a minimum the proto files and the gherkin parser need to be broken out into their own project.

In addition to this we should probably break the JUnit4 and JUnit5 implementations into their own repository as well.

This will produce breaking changes. While we're at it we should do other housekeeping tasks.

One would be to replace providers with suppliers instead. This will remove the need for a library and allow us to use a native (and more flexible) feature in the standard library instead.

It is possible to get a false positive if errorNodes are encountered. The default implementation is to do nothing. The user can implement an exception to be thrown manually, but I'm concerned that it will be easy to forget and lead to difficult to diagnose bugs.

I'm going to see if this can be built into the generated code ANTLR produces with its "tool" code.

The linux terminal is not colorized.

Pairwise testing becomes a necessity when testing large permutations of data.

The requirement is to create an extension to the gherkin protocol that allows for specifying equivalence classes, inclusions and exclusions.

The DefaultPolymorphicDslPhraseFilter (mostly) requires creating another grammar on top of the grammar we actually care about. It also has a verbose constructor.

If possible we could compile the grammars at run time and generate the "meta" parser to remove this overhead. This way we could just write visitors/listeners for the parser we care about at less cost.

Se currently have the beginning of this in the InterpreterBasedPhraseFilter, however we haven't solved the problem of how to get the new context objects to work with the listeners generated from the original grammar (they are incompatible)

This problem may be large enough that we may need to abandon the effort.

One potentials way to solve this would be by generating code that used some form of method forwarding to invoke the listeners properly. This would require the ability to convert the "meta" context to the original parser context

Another might be to generate code that just has a walker call the corresponding rule with the encountered parse tree.

In either case it seems will need to find a way to wrap the original parser in the meta-parser some way if it is ever going to work.

If the wrong type of listener is used it seems that ANTLR's parse tree walker just ignores it by default. We want to throw an exception if one of our users does this.

When we have something like this the exception thrown is not very helpful. Sometimes the column names in the examples table don't match the values needed for the steps.

Scenario:
  Given a <FOO>

Examples
| FO |
| test|

Some minor bugs exist in the proto file we created that prevent the objects from being created in the proper package

The JUnit4 methodBlock method associates @before and @after from parent classes with the test. PDSL creates many tests from one method, but the tests aren't getting the hooks.

Associate the hooks in runChild with each PDSL test.

Now that we have some reporting features it would be useful to integrate with JUnit, aggregate all test run information and create PDSL reports.

The method `com.pdsl.executors.DefaultPolymorphicDslTestExecutor#runTestsWithMetadata(java.util.Collection<com.pdsl.testcases.SharedTestCase>, java.lang.String)' should be able to process the metadata (@tags) in the feature file.

The method `com.pdsl.executors.DefaultPolymorphicDslTestExecutor#runTestsWithMetadata(java.util.Collection<com.pdsl.testcases.SharedTestCase>, java.lang.String)' should be able to handle the duplicate steps in the feature file.

The error says that it is required to use @PdslConfiguration, but this isn't true for the gherkin runner. It should be @PdslGherkinApplication instead.

Currently JUnit shows a run as a single pass/fail. We need to make it so each individual test case is reported.

It isn't clear how PDSL test suites work, how they differ from JUnit and what they accomplish. Document this and provide code examples.

Show how to use PDSL inside unsupported frameworks.

Many other BDD tools allow files to be discovered with a regular expression. Create a class that will locate files in a similar manner and produce a collection of URLs that can be used by a TestSpecificationFactory