antlr / antlr4-intellij-adaptor Goto Github PK

I am new to antlr, intellij plugins and maven...so i figure i have done something wrong but i cant see it.

Environment :
Intellij IDEA 2016.1
JDK 1.8.0_73

I have cloned the jetbrains-plugin-sample
I have cloned, as a submodule, antlr/jetbrains

I have set the adapter/src directory as a source root in idea

I installed the latest antlr plugin and then set the antlr preferences for SampleLanguage.g4 to
Auto-generate parsers upon save ENABLED
package : org.antlr.jetbrains.sample.parser
generate parse tree listener ENABLED
generate parse tree visitor ENABLED

I then hit "ctrl-shift-G" on the SampleLanguage.g4 file

I then set the gen directory as a source root for idea

Now compile and I get a couple of errors

1. FunctionSubtree constructor isnt compatible with IdentifierDefSubtree (doesnt have IElementType)
2. VardefSubtree same error.

The IdentifierDefSubtree hides the ANTLRPsiNode c'tor so I just altered these 2 classes to inherit directly from ANTLRPsiNode, I know this probably breaks stuff down the line, in fact i know it does cause when i run the tests i get 7 failed, but all i want for my project is mylanguage syntax highlighting in intellij.

This issue was migrated from this jetbrains dev forum post; you can the whole discussion up to this point there. I kind of hijacked that thread -- sorry about that! Probably should've moved stuff here sooner :)

Here are some of the latest syntax error highlighting quirks I've stumbled across.

I have a uses/import list rule that looks like:

usesList
    :   'uses' ID (',' ID)* ';'
    ;

So I type

Precis X;
     uses y

end X;

doesn't point out the missing semicolon after y.
If I do something like uses x, y it'll complain -- so it's only for lists with a single element then I guess?

Another one too:

Precis X;

     //error, should be something (not semi colon) immediately after ':'
     Definition x : ;
end X;

The grammar for this lang can be read here. It has a fairly big expression hierarchy and just two types of declarations: definitions and theorems. Definition's can take on several signature styles such prefix, postfix, outfix. I've only tried the prefix right now, before I ran into the issue mentioned above.

The rules composing the definition construct mentioned above starts at mathStandardDefinitionDecl (the signature referenced will be mathPrefixDefinitionSig).

I'd like to propose a second "rule" processor (ANTLRParseTreeToPSIConverter) by the Rule Context's class name instead of the rule index in Antlr's generated Parser class. The goal is to provide enough information to the Parser Definition's createElement, though ASTNodes.elementType, to replicate the Context classes in the PSI Tree

Antlr has a generator option that includes subclassing rules by name. The grammar below generates a single Rule integer for simpleLiteral but three Context classes in the parser: SimpleLiteralContext, SimpleStringLiteralContext, SimpleNumberLiteralContext.

simpleLiteral
    : STRING           #simpleStringLiteral
    | NUMBER           #simpleNumberLiteral
    ;

SimpleStringLiteralContext, SimpleNumberLiteralContext extend SimpleLiteralContext

public static class SimpleLiteralContext extends ParserRuleContext {
    public SimpleLiteralContext(ParserRuleContext parent, int invokingState) {
        super(parent, invokingState);
    }
    @Override public int getRuleIndex() { return RULE_simpleLiteral; }
    //...
}
public static class SimpleNumberLiteralContext extends SimpleLiteralContext {
    public TerminalNode NUMBER() { return getToken(cqlParser.NUMBER, 0); }
    public SimpleNumberLiteralContext(SimpleLiteralContext ctx) { copyFrom(ctx); }
    //...
}
public static class SimpleStringLiteralContext extends SimpleLiteralContext {
    public TerminalNode STRING() { return getToken(cqlParser.STRING, 0); }
    public SimpleStringLiteralContext(SimpleLiteralContext ctx) { copyFrom(ctx); }
    //...
}

The change would need a RuleMap class as below:

// All Rule names, including subrules (subclasses)
val RULE_NAME_INDEX = antlrParser::class.nestedClasses.map { it.simpleName }.toTypedArray()

// List of Indexed Rule Types (required by IntelliJ)
private val RULES: List<RuleIElementType> = RULE_NAME_INDEX.mapIndexed { index, s ->
    RuleIElementType(index, s, CqlLanguage)
}

// Map between Context names and Rule Types
val RULE_MAP: Map<String, RuleIElementType> = RULES.associateBy { it.debugName }

And then a new ANTLRParseTreeToPSIConverter listener that would be very similar to the current one but with maker.done by Class name and not rule index.

class AntlrContextToPSIConverter(language: Language?, parser: Parser?, builder: PsiBuilder?) :
    ANTLRParseTreeToPSIConverter(language, parser, builder) {
    override fun exitEveryRule(ctx: ParserRuleContext) {
        ...
        marker.done(CqlRuleTypes.RULE_MAP[ctx.javaClass.simpleName]!!)
    }
}

The Parser Definition then create PSINodes for each subclass

   //...
   override fun createElement(node: ASTNode): PsiElement {
        val elType = node.elementType
        if (elType is TokenIElementType) {
            return ANTLRPsiNode(node)
        }
        if (elType !is RuleIElementType) {
            return ANTLRPsiNode(node)
        }

        return when(CqlRuleTypes.RULE_NAMES[elType.ruleIndex]) {
            "SimpleLiteral" -> SimpleLiteralPSI(node)
            "SimpleStringLiteral" -> SimpleStringLiteralPSI(node)
            "SimpleNumberLiteral" -> SimpleNumberLiteralPSI(node)
            //..
        }
    }

We implemented this variation in this project: https://github.com/Path-Check/intellij-cql

As you can see, we had to go around some of the interface design (Map of Rules instead of List) for the adaptor's converters to make it work.

Apologies for blending Java and Kotlin in the same text. :)

When building my plugin, there is a NullPointerException in the ANTLR adaptor. It seems benign because the build continues, is eventually successful, and the built plugin works correctly.

Steps to reproduce

1. Clean gradle cache
2. execute the gradle task buildPlugin (actually, it should suffice to execute the gradle task buildSearchableOptions)

Expected behavior

Build completes successfully and without any errors

Actual behavior

Build completes successfully but with one error due to the following NullPointerException:

> Task :myModule:buildSearchableOptions
CompileCommand: exclude com/intellij/openapi/vfs/impl/FilePartNodeRoot.trieDescend bool exclude = true
2023-04-05 14:00:17,534 [   4793] SEVERE - #c.i.u.u.EdtInvocationManager - null
java.lang.reflect.InvocationTargetException
        at java.desktop/java.awt.EventQueue.invokeAndWait(EventQueue.java:1379)
        at java.desktop/java.awt.EventQueue.invokeAndWait(EventQueue.java:1354)
        at com.intellij.util.ui.EdtInvocationManager$SwingEdtInvocationManager.invokeAndWait(EdtInvocationManager.java:103)
        at com.intellij.util.ui.EdtInvocationManager.invokeAndWaitIfNeeded(EdtInvocationManager.java:84)
        at com.intellij.ide.ui.search.TraverseUIStarter.startup(TraverseUIStarter.java:115)
        at com.intellij.ide.ui.search.TraverseUIStarter.main(TraverseUIStarter.java:93)
        at com.intellij.idea.ApplicationLoader.initApplicationImpl$lambda-3(ApplicationLoader.kt:174)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1796)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: java.lang.NullPointerException: Cannot invoke "java.util.List.get(int)" because "this.tokenElementTypes" is null
        at org.antlr.intellij.adaptor.lexer.ANTLRLexerAdaptor.getTokenType(ANTLRLexerAdaptor.java:188)
        at org.antlr.intellij.adaptor.lexer.ANTLRLexerAdaptor.getTokenType(ANTLRLexerAdaptor.java:178)
        at com.intellij.openapi.editor.ex.util.ValidatingLexerWrapper.getTokenType(ValidatingLexerWrapper.java:50)
        at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.doSetText(LexerEditorHighlighter.java:441)
        at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.setText(LexerEditorHighlighter.java:418)
        at com.intellij.openapi.editor.impl.EditorImpl.setHighlighter(EditorImpl.java:1359)
        at com.intellij.application.options.colors.SimpleEditorPreview.updateView(SimpleEditorPreview.java:192)
        at com.intellij.application.options.colors.NewColorAndFontPanel$3.schemeChanged(NewColorAndFontPanel.java:105)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.intellij.util.EventDispatcher.dispatchVoidMethod(EventDispatcher.java:120)
        at com.intellij.util.EventDispatcher.lambda$createMulticaster$1(EventDispatcher.java:85)
        at jdk.proxy2/jdk.proxy2.$Proxy80.schemeChanged(Unknown Source)
        at com.intellij.application.options.colors.SchemesPanel.resetSchemesCombo(SchemesPanel.java:56)
        at com.intellij.application.options.colors.NewColorAndFontPanel.resetSchemesCombo(NewColorAndFontPanel.java:156)
        at com.intellij.application.options.colors.NewColorAndFontPanel.reset(NewColorAndFontPanel.java:144)
        at com.intellij.application.options.colors.ColorAndFontOptions$InnerSearchableConfigurable.createPanel(ColorAndFontOptions.java:1333)
        at com.intellij.application.options.colors.ColorAndFontOptions$InnerSearchableConfigurable.createComponent(ColorAndFontOptions.java:1363)
        at com.intellij.ide.ui.search.SearchUtil.processConfigurables(SearchUtil.java:75)
        at com.intellij.ide.ui.search.TraverseUIStarter.lambda$startup$0(TraverseUIStarter.java:120)
        at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:308)
        at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:779)
        at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:730)
        at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:724)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
        at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
        at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:749)
        at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:909)
        at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:756)
        at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$5(IdeEventQueue.java:437)
        at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:774)
        at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$6(IdeEventQueue.java:436)
        at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:105)
        at com.intellij.ide.IdeEventQueue.performActivity(IdeEventQueue.java:615)
        at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$7(IdeEventQueue.java:434)
        at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:813)
        at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:480)
        at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:207)
        at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:128)
        at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:117)
        at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:113)
        at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:105)
        at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:92)

Suggested fix

I think the fix is rather straight forward. Here is the code in question:

If tokenElementTypes is null (on line 188), then I think this method should return null. I had a similar problem in my code, and returning null in this case seems to have fixed the problem.

Other

For the record, this happens with the 0.1 release, which is currently the only release.

Hello

I have difficulty to use the functions generated by ANTLR4 for Intellij.
I searched in the doc but It's not clear.

What is the name of the function to call to have the possible rules and tokens in function of offset in the file ? I see the error messages with the possible rules and tokens but I didn't found the good function to call.

I don't know if ANTLR4 for Intellij can generate the LL1 tools for my language. Have you examples of CompletionContributor class that use the basic completion tool and the LL1 tool ?

My code:

public class TestCompletionContributor extends CompletionContributor {
    public TestCompletionContributor() {
        extend(CompletionType.BASIC,
                PlatformPatterns.psiElement(),
                new CompletionProvider<CompletionParameters>() {
                    public void addCompletions(@NotNull CompletionParameters parameters,
                                               ProcessingContext context,
                                               @NotNull CompletionResultSet resultSet) {

                        //  How do the list of Rules and Tokens probable ? (with a LL1 tool ?)

                        resultSet.addElement(
                                        PrioritizedLookupElement.withPriority(
                                                LookupElementBuilder
                                                        .create("prevision1"),
                                                probability
                                        )
                                );
                    }
                }
        );
    }

    public static final String DUMMY_IDENTIFIER = "";

    @Override
    public void beforeCompletion(@NotNull CompletionInitializationContext context) {
        context.setDummyIdentifier(DUMMY_IDENTIFIER);
    }
}

Thanks

Grammar-Kit supports error recovery via pin/recoveryWhile. Could we have something similar?

Update: or even have automatic error recovery similar to recoveryWhile="#auto".

Hi,

I think it'd be useful if this library supported intellij's Stub feature. I'd like to be able to add some StubIndex's to my language plugin.

There seems to be a problem when processing emoji,

The input text is utf-8.

The lexer rule is: ValkyrieAntlrLexer.g4

UNICODE_ID: [_\p{XID_start}\p{Emoji_Presentation}] [\p{XID_continue}\p{Emoji}]*;

The input is: 🔥extract

Wrong behavior

Expected behavior

it seems that all surrogate pairs have problems.

Totally understand why we should use the PSI tree, for example, to get all the inbuilt features like renaming and so on. And the ANTLR like Xpath feature works well. In sum, no complaints.

However, I do have a number of ANTLR walkers (for example that do more fine grained work around identifier type safety and other analysis). I really can't be bothered to re-do for PSI trees (and also not sure how to make a walker in PSI - I guess I could learn!).

So I think I could quite easily pump the whole file through the ANTLR Lexer and Parser in an external annotator, and could get row and column numbers to sync, and hence add some warning annotations etc. Although I can't find an example of how to access the file's contents the virtual file system seems to be the answer.

So finally (!) the question is this - can I access either the actual ANTLR (not PSI) parse tree from an annotator, or perhaps the token stream? I obviously have looked but can't decide if there is a suitable extension point or not. For example somehow when the PSI tree is built.

I am thinking it is not possible at the moment and I need to reparse the file. Is this right?

I think it should be possible for the ANTLR adaptor to get the list of token literals that the user could be typing and allow a plugin to include these token literals as code completion suggestions.

Background

To be able to resolve a reference, we must implement the interface PsiReference, which has a method called getVariants that returns code completion suggestions:

  /**
   * Returns the array of String, {@link PsiElement} and/or {@link com.intellij.codeInsight.lookup.LookupElement}
   * instances representing all identifiers that are visible at the location of the reference. The contents
   * of the returned array are used to build the lookup list for basic code completion. (The list
   * of visible identifiers may not be filtered by the completion prefix string - the
   * filtering is performed later by the IDE.)
   * <p>
   * This method is default since 2018.3.
   *
   * @return the array of available identifiers.
   */
  default Object @NotNull [] getVariants()

Feature Request

I can implement PsiReference::getVariants to return suggested code completions for semantic references in my language. Sometimes I also return token literals that are possible at a certain location. However, I feel like I am duplicating information from my lexer's and parser's grammars.

Instead, I think it should be possible for (ANTLR's) parser to determine what token literals are possible at a specified location in the parse tree. Then I could give those token literals to IntelliJ as something the user might be trying to type.

For the ANTLR plugin, this functionality could suggest that the user completes

myRule r
--------^ cursor location

as

myRule returns

I just tested with the current version of ANTLR's plugin (version 1.20), and return is not a code completion suggestion in that case.

Hello everyone, I'm not sure if this question related to previous issues or not, but I have a difficulty.

In my verilog support plugin, I got a grammar from here and started to implement the plugin. I successfully implemented highlighting, referencing and completion, but I sometimes get strange behavior, when proposed variants in completion proposals are duplicated.

It happens because if I insert incomplete name it is not only considered as a bad character bud rebuild PsiTree.

For example, right tree

and tree with bad character

As you can see a parent of highlighted flop1 differs and lead to duplicates in completion proposals. It's because all elements of type IdentifierPsiNode are proposed now (if everything is correct they appear only in declarations so it's okay).

So my question is if there any ways to bypass this problem, for example, stopping tree rebuilding if bad characters are found. It's desirable to find decision which doesn't involve correcting grammar.

Thanks.

I think it makes sense to rename this repository and library to antlr-jetbrains-adapter (or something similar).

Hi,

Is there an ILazyParseableElementType implementation for the antlr4-intellij-adapter? I have a separate preprocessor grammar (incl. lexer + parser) that I want to recognize within the regular grammar. In the Intellij plugin development notes there is a chapter on this topic, but I don't see any way how I can make my preprocessor tokens lazy within the antlr-adapter. Do you have any ideas or recommendations?

Thanks in advance.

ANTLRPsiNode created via createElement seems unable to obtain ParserRuleContext.

ParserDefinition.createElement(node: ASTNode): PsiElement =  ANTLRPsiNode(node)

I want to get child nodes by judging ParserRuleContext's type.

But the child node is also a ParserRuleContext, I don’t know how to convert it to a PsiElement

When injecting my language into other file types such as XML or JSON, the IDE throws an error everytime I type:

class com.intellij.psi.impl.source.xml.XmlTextImpl cannot be cast to class org.antlr.intellij.adaptor.psi.ScopeNode
java.lang.ClassCastException: class com.intellij.psi.impl.source.xml.XmlTextImpl cannot be cast to class org.antlr.intellij.adaptor.psi.ScopeNode (com.intellij.psi.impl.source.xml.XmlTextImpl is in unnamed module of loader com.intellij.util.lang.PathClassLoader @67b64c45; org.antlr.intellij.adaptor.psi.ScopeNode is in unnamed module of loader com.intellij.ide.plugins.cl.PluginClassLoader @5466ba6)
	at org.antlr.intellij.adaptor.SymtabUtils.getContextFor(SymtabUtils.java:51)
	at org.antlr.intellij.adaptor.psi.ANTLRPsiNode.getContext(ANTLRPsiNode.java:53)
	at org.antlr.intellij.adaptor.psi.ANTLRPsiNode.getContext(ANTLRPsiNode.java:31)
	at org.antlr.intellij.adaptor.SymtabUtils.getContextFor(SymtabUtils.java:51)
	at org.antlr.intellij.adaptor.psi.ANTLRPsiNode.getContext(ANTLRPsiNode.java:53)
	at org.antlr.intellij.adaptor.psi.ANTLRPsiNode.getContext(ANTLRPsiNode.java:31)
	at org.antlr.intellij.adaptor.SymtabUtils.getContextFor(SymtabUtils.java:51)
	at org.antlr.intellij.adaptor.psi.ANTLRPsiNode.getContext(ANTLRPsiNode.java:53)
	at org.antlr.intellij.adaptor.psi.ANTLRPsiNode.getContext(ANTLRPsiNode.java:31)
	at com.intellij.psi.util.PsiTreeUtil.getContextOfType(PsiTreeUtil.java:570)
	at com.intellij.lang.javascript.intentions.JSIntroduceVariableIntention.findExpression(JSIntroduceVariableIntention.java:149)
	at com.intellij.lang.javascript.intentions.JSIntroduceVariableIntention.isAvailable(JSIntroduceVariableIntention.java:180)
	...

This seems to get triggered by JavaScript code insights.

I found that there is always a separate root node under PsiFile.

Is there a way to remove the grammar root node and let its child nodes belong to PsiFile?

And I found that the first comment does not belong to the first statement, but to its root node, causing problems when I query the PsiDocument.

In my language I have TAB and UNTAB tokens (it is important like in Python). It is impossible to process TAB and UNTAB in grammar.g4 correctly so I need special class (I guess it is BaseListener) for processing TAB/UNTAB when I enter some rules.
How to make my BaseListener work in background? This class now is highlighted with grey color, it is not used. Where should I create it?

The reviewers of jetbrain market asked me to resolve this compatibility issue before listing.

My configuration is as follows:

https://github.com/nyar-vm/valkyrie-intellij/blob/775990cd685586d675f6201ac441883a24266ffe/build.gradle.kts#L18-L22

dependencies {
    implementation(kotlin("stdlib-jdk8"))
    antlr("org.antlr:antlr4:${properties("antlrVersion")}")
    implementation("org.antlr:antlr4-intellij-adaptor:0.1")
}

How can I eliminate this error?

It happens for injected languages when startOffset is not zero and initialState is zero. In this case stateCache is empty and we get:

java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.util.Objects.checkIndex(Objects.java:372)
	at java.base/java.util.ArrayList.get(ArrayList.java:458)
	at org.antlr.intellij.adaptor.lexer.ANTLRLexerAdaptor.toLexerState(ANTLRLexerAdaptor.java:280)
	at org.antlr.intellij.adaptor.lexer.ANTLRLexerAdaptor.start(ANTLRLexerAdaptor.java:168)
	at com.intellij.lexer.HtmlHighlightingLexer$XmlEmbeddmentHandler.handleElement(HtmlHighlightingLexer.java:269)
	at com.intellij.lexer.BaseHtmlLexer.advance(BaseHtmlLexer.java:415)
	at com.intellij.lexer.HtmlHighlightingLexer.advance(HtmlHighlightingLexer.java:173)

I have a gamma that works fine standalone - but not with the antlr4-intellij-adaptor. Archaic language with a line continuation.

Simplified lexer snip

LINECONTIUNE
: '&' '\n' -> channel(HIDDEN) ;

This is fine standalone but with the adaptor the parser gets the LINECONTIUNE token ...

Note: Block comments also hidden and work fine, Whitespace also fine!?

Any ideas?

Well, not just block comments, but it's reproducible with antlr/jetbrains-plugin-sample: http://recordit.co/ODFaSxunOl

Same issue in my project using this adapter: http://recordit.co/dsL5dlHS9t