antlr / antlr5 Goto Github PK

Beyond tooling issues, we also need to deal with paradigms that cannot work with WebAssembly.

In the current runtime, the Lexer is an abstract class, and the generated actual XXXLexer inherits from it.
This paradigm won't work with WebAssembly, especially not across language targets.

Looking at a generated XXXLexer, it doesn't provide behavior, rather it provides data that the runtime Lexer will use.
Therefore an idea that comes to mind is to evolve the design as follows:

• the generated XXXLexer becomes a standalone class that:
  • provides data to a runtime Lexer instance
  • forwards calls such as nextToken to that Lexer instance
• the data itself sits in a LexerData record (data class in Kotlin)
• the runtime Lexer becomes a concrete class that requires a LexerData record when instantiated

My plan is to first make the above work in Kotlin, then compile to Wasm.
Your comments on the proposed design are welcome.

There is a need to revisit the docs:

• cater for removed targets
• supported hosts (Chrome, Deno etc...)
• supported IDEs
  .../...

Grammar

grammar Test;

a: e;
e: a;

b: c;
c: d;
d: c;

A: A 'B';
C: D;
D: E;
E: D;

Actual

error(119): Test.g4::: The following sets of rules are mutually left-recursive [A] and [D, E]
error(119): Test.g4::: The following sets of rules are mutually left-recursive [a, e] and [c, d]

Expected

error(119): Test.g4:10:0: The following sets of rules are mutually left-recursive [A]
error(119): Test.g4:12:0: The following sets of rules are mutually left-recursive [D, E]
error(119): Test.g4:3:0: The following sets of rules are mutually left-recursive [a, e]
error(119): Test.g4:7:0: The following sets of rules are mutually left-recursive [c, d]

In the context of #40, some property and param types were changed to nullable. There may be an opportunity to improve performance slightly by making them non-nullable again, but that requires significant work on the java side (runtime, tool, runtime-test suite and tool-testsuite)

I am sorry, but I have no experience with Maven and I am having headaches in getting locally the same results that I get on the CI.
For example, at this time on my machine when running mvn test from tool-testuite I get errors I do not get on the CI.

What I am doing is to run:

mvn clean install -e -U -Dmaven.test.skip=true
cd tool-testsuite
mvn test

Am I missing something?

I think that other potential contributors could come from a Gradle background and be running into surprising challenges when dealing with Maven

Once #40 is merged, we will be aligned with our target architecture, which seems a good time to switch to Gradle so we can start generating WebAssembly using the Gradle plugin.

For example, in the README

• Update the references in the badges

Since we plan to have only 1 runtime, we should in principle remove all non-Kotlin runtimes.

That said it might be convenient during the initial phase to keep:

• the Java runtime, as our reference implementation
• The Typescript runtime, as our first target language, for later conversion to an add-on

Currently we generate this code for the Kotlin target:

object Modes {
  const val DEFAULT_MODE = 0
  const val FOO = 1 // made uppercase by #8 
  const val BAR = 2
}

This is inconsistent because only the default mode has the _MODE suffix. We should therefore rename it, by working on the template for the Kotlin target and updating references in the Kotlin runtime

ANTLR tool maven build relies on com.webguys:string-template-maven-plugin:1.1 plugin to generate org/antlr/v5/unicode/UnicodeData.java.
When switching to Gradle, we could not find an equivalent Gradle plugin so we temporarily opted for a tactical solution i.e. use the UnicodeData.java file generated by a previous maven build.
This is acceptable as a tactical solution because the underlying data (emanating from com.ibm.icu.text.UnicodeSet) rarely changes.
But on the long term we need a solution that generates the file dynamically.

Options that come to mind:

• create a Gradle plugin for string-template
• use MavenExecutor, wrapping the existing maven plugin
• use an alternative templating engine for which a Gradle plugin already exists
• move the code to a custom Gradle task
• ...

I am the author of a tool for converting the antlr parse tree into a denser AST, which can output a standalone JavaScript version to query/transform/print/visualize using CSS selectors. (The end goal is to use machine learning with rule induction for AST transformation/printing.) Demo

Since there is the possibility of changing things with a new version, it would be nice to have a unique antlr state for as many positions in the grammar as possible. This does not currently apply to recursive rules.

At the moment we have hundreds of files inherited from ANTLR Kotlin starting in this way:

// Copyright 2017-present Strumenta and contributors, licensed under Apache 2.0.
// Copyright 2024-present Strumenta and contributors, licensed under BSD 3-Clause.

package com.strumenta.antlrkotlin.runtime

We should align the Copyright message to the one we have in other files, and for those other files we should update the end year:

/*
 * Copyright (c) 2012-2017 The ANTLR Project. All rights reserved.
 * Use of this file is governed by the BSD 3-clause license that
 * can be found in the LICENSE.txt file in the project root.
 */

It could make sense to update all files to be in the form:

/*
* Copyright (c) 2012-present The ANTLR Project. All rights reserved.
* Use of this file is governed by the BSD 3-clause license that
* can be found in the LICENSE.txt file in the project root.
*/

Regarding the packages, we should rename com.strumenta.antlrkotlin.runtime into org.antlr.v5.wasmruntime. Would this make sense?

I think this class should be renamed

See PR #37

Target is java but folder is runtime/Java
See #36 for consequences

The java runtime package is named org.antlr.v5.runtime which is confusing since the actual runtime is in org.antlr.v5.runtime.core and the current name doesn't exhibit the target language (unlike the kotlin runtime).

Hi @KvanTTT @bashor

I'm experimenting with Kotlin/Wasm and facing an issue you might be able to help with...

I've downloaded kotlinc 2.0.0-Beta3, and kotlinc-js -version shows the correct version.

When I run:

kotlinc-js src/main/kotlin/Test.kt -Xwasm -Xwasm-target=wasm -ir-output-name test.js -ir-output-dir out/wasm -kotlin-home /Users/ericvergnaud/Development/kotlinc

I get:

exception: java.lang.IllegalStateException: Class not found: kotlin/Float

How do I tell kotlin-js where to find builtins (I thought -kotlin-home was about that) ?
Btw where are they actually located ?

Since we'll be evolving grammar syntax, there is a need to prevent false bugs in antlr4, caused by antlr5 syntax unsupported in antlr4. This is easy to achieve by changing the grammar file extension from g5 to g5.

We will be using Kotlin to generate the WebAssembly runtime.
There is a need to integrate it from Strumenta's repo.
There is also a need to ensure it supports the same level of testing as the existing Java runtime.
(we don't want to embark on WebAssembly without checking that the runtime is 100% compatible)

This line seems not necessary anymore as the Python3 target was dropped: