eclipse / tm4e Goto Github PK

Support Grammar Injection to support syntax coloration like:

• HTML & JS
• HTML & Freemarker

Usually syntax coloration is customized with Eclipse preferences. My idea is to provide this customization with CSS too. To do that I need a SAC parser. By default Eclipse provides a SAC Parser implementation with batik that I will use but we could use other SAC Parser.

@mickaelistria is it a problem to have this "optional" batic dependency?

My other idea is to customize grammar with TextMate theme. See #2

Write documentation in the wiki about:

• core
• eclipse

@mickaelistria I'm developping JUnit classes wich will execute tests from the vscode-textmate at https://github.com/Microsoft/vscode-textmate/tree/master/test-cases

Thoses files are JSON and grammar. I think it will be a problem when textmate.java will move to Eclipse.org, no?

For the moment I will add thoses tests inside org.eclipse.textmate4e.core and we will see after.

Using the version from the update site:

In order to prepare the move to Eclipse.org, it would be good to have bundle/feature versions be 0.1.0 as the project will benefit of Incubation state (include more IP tolerance)

In order to produce a minimal amount of different versions with the exact same content, we should set up reproducible qualifier and baseline replacement at build-time.
See https://wiki.eclipse.org/Tycho/Reproducible_Version_Qualifiers

Using the jacoco-maven-plugin, we should enable code coverage on build and report it in SonarQube.

When building via maven, the "org.eclipse.tm4e.core.grammar.test.VSCodeTextMateTest" does not complete.

Hangs on test "org.eclipse.tm4e.core.grammar.test.VSCodeTextMateTest":

mvn clean verify

Does not hang (workaround):

mvn clean verify -Dskiptests

Hi,

My plugin used to dynamically be able to change file associations for my given editor type. Since using the TMPresentationReconciler, this ability appears to have been lost (conjecture on why later).

Here is my editor definition (NOTE : "my' is the associated file type, NOT 'pie') :

Here is the method in MySourceViewerConfiguration class:

Here is the TM4E definition (NOTE : "my' is the associated file type, NOT 'pie') :

Here is the code that I used to dynamically modify file associations to choose my editor (this previously worked before adding in the TM4E syntax highlighting) :

public void resetFileAssociations(java.util.Set<java.lang.String> extensionToAddSet) {
      IWorkbench workbench = PlatformUI.getWorkbench();
      final org.eclipse.ui.internal.registry.EditorRegistry editorReg = (org.eclipse.ui.internal.registry.EditorRegistry) workbench.getEditorRegistry();

      org.eclipse.ui.internal.registry.EditorDescriptor editor = (org.eclipse.ui.internal.registry.EditorDescriptor) editorReg.findEditor("com.consoli.myrion.eclipse.ui.editor.MyEditorId3");

      IFileEditorMapping[] originalMappings = editorReg.getFileEditorMappings();

      java.util.ArrayList<org.eclipse.ui.internal.registry.FileEditorMapping> newMappings = new java.util.ArrayList<org.eclipse.ui.internal.registry.FileEditorMapping>();

      boolean firstTime = false;

      if (originalMappings2 == null) {
         originalMappings2 = new java.util.HashSet<String>();
         firstTime = true;
      }

      int numOriginalMappings = originalMappings.length;

      for (int i = 0; i < numOriginalMappings; i++) {

         org.eclipse.ui.internal.registry.FileEditorMapping m = (org.eclipse.ui.internal.registry.FileEditorMapping) originalMappings[i];

         String extension = m.getExtension();

         IEditorDescriptor[] editors = m.getEditors();

         for (IEditorDescriptor currentEditor : editors) {
            boolean isMyEditor = currentEditor == editor;
            if (isMyEditor) {

               if (firstTime) {
                  originalMappings2.add(extension);
               } else {
                  if (!originalMappings2.contains(extension)) {
                     m.removeEditor(editor);
                  }
               }
            }
         }

         if (editors.length > 0) {
            newMappings.add(m);
         }
      }

      for (String extensionToAdd : extensionToAddSet) {
         if (!originalMappings2.contains(extensionToAdd)) {
            org.eclipse.ui.internal.registry.FileEditorMapping mapping = new org.eclipse.ui.internal.registry.FileEditorMapping(extensionToAdd);
            System.out.println("Associating " + extensionToAdd);
            mapping.addEditor(editor);
            mapping.setDefaultEditor(editor);
            newMappings.add(mapping);
         }
      }

      final FileEditorMapping[] fileEditorMappings = newMappings.toArray(new org.eclipse.ui.internal.registry.FileEditorMapping[newMappings.size()]);
      final Display default1 = Display.getDefault();
      final Runnable runnable = new Runnable() {
         @Override
         public void run() {
            editorReg.setFileEditorMappings(fileEditorMappings);
         }
      };
      default1.syncExec(runnable);
}

Now we wish to dynamically associate the 'pie' file type / suffix with the same editor as 'my'.

So we call the resetFileAssociations() method with set containing ["pie", "my"]. This worked perfectly prior to tm4e integration. It also works now actually. It associates the editor type with the new suffix, but TM4E then uses the content type to grammar map and finds it empty for "pie". Same with themes.

As you would expect from the previous description, here is what happens when I attempt to access a ".pie" (previously it worked because the previous reconciler did not require configuration):

I believe that TM4E is using IGrammarRegistryManager to look up grammar / theme associations, and this cannot be modified at runtime (as far as I am aware) - therefore the .pie file suffix resolves to null. I would like to be able to alter grammar/theme associations on the fly same as is possible with the suffix to default editor associations.

The use case for this is that I have one file type that when saved, controls file associations for other file types. For example, lets say I have a file with suffix .mysettings, that file might contain a single line with a comma separated list of file suffixes such as "one, two, three". Now the builder sees when that file changes and calls the method detailed earlier. It reads the content of the file, and now dynamically associates the files with the editor (described earlier). That editor is associated with the TMPresentationReconciler. BUT, the presentation reconciler will not allow files without statically registered suffixes and themes to render. Eclipse has the functionality to change file associations at runtime, but TM4E does not (yet) (as far as I know) have the ability to render for a file suffix that has not been statically registered via the plugin.xml.

Today when editor changed, it colorize all lines from the line which have changed to last line of document. This issue should improve this behaviour to colorize only lines where tokens have changed. It will improve a lot the performances.

Trying to open non-workspace ruby file fails with:
java.lang.NullPointerException
at org.eclipse.tm4e.ui.internal.model.DocumentHelper.getContentTypes(DocumentHelper.java:101)
at org.eclipse.tm4e.ui.internal.model.DocumentHelper.getContentTypes(DocumentHelper.java:96)
at org.eclipse.tm4e.ui.text.TMPresentationReconciler$InternalListener.inputDocumentChanged(TMPresentationReconciler.java:160)
at org.eclipse.jface.text.TextViewer.fireInputDocumentChanged(TextViewer.java:2759)
at org.eclipse.jface.text.TextViewer.setDocument(TextViewer.java:2800)
at org.eclipse.jface.text.source.SourceViewer.setDocument(SourceViewer.java:634)
at org.eclipse.jface.text.source.projection.ProjectionViewer.setDocument(ProjectionViewer.java:365)
at org.eclipse.jface.text.source.SourceViewer.setDocument(SourceViewer.java:584)
at org.eclipse.ui.texteditor.AbstractTextEditor.initializeSourceViewer(AbstractTextEditor.java:3995)
at org.eclipse.ui.texteditor.AbstractTextEditor.createPartControl(AbstractTextEditor.java:3465)
at org.eclipse.ui.texteditor.StatusTextEditor.createPartControl(StatusTextEditor.java:55)
at org.eclipse.ui.texteditor.AbstractDecoratedTextEditor.createPartControl(AbstractDecoratedTextEditor.java:445)
at org.eclipse.dltk.internal.ui.editor.ScriptEditor.createPartControl(ScriptEditor.java:1690)
at org.eclipse.dltk.ruby.internal.ui.editor.RubyEditor.createPartControl(RubyEditor.java:111)
at org.eclipse.ui.internal.e4.compatibility.CompatibilityPart.createPartControl(CompatibilityPart.java:150)
at org.eclipse.ui.internal.e4.compatibility.CompatibilityEditor.createPartControl(CompatibilityEditor.java:99)
at org.eclipse.ui.internal.e4.compatibility.CompatibilityPart.create(CompatibilityPart.java:340)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.e4.core.internal.di.MethodRequestor.execute(MethodRequestor.java:55)
at org.eclipse.e4.core.internal.di.InjectorImpl.processAnnotated(InjectorImpl.java:966)
at org.eclipse.e4.core.internal.di.InjectorImpl.processAnnotated(InjectorImpl.java:931)
at org.eclipse.e4.core.internal.di.InjectorImpl.inject(InjectorImpl.java:151)
at org.eclipse.e4.core.internal.di.InjectorImpl.internalMake(InjectorImpl.java:375)
at org.eclipse.e4.core.internal.di.InjectorImpl.make(InjectorImpl.java:294)
at org.eclipse.e4.core.contexts.ContextInjectionFactory.make(ContextInjectionFactory.java:162)
at org.eclipse.e4.ui.internal.workbench.ReflectionContributionFactory.createFromBundle(ReflectionContributionFactory.java:105)
at org.eclipse.e4.ui.internal.workbench.ReflectionContributionFactory.doCreate(ReflectionContributionFactory.java:74)
at org.eclipse.e4.ui.internal.workbench.ReflectionContributionFactory.create(ReflectionContributionFactory.java:56)
at org.eclipse.e4.ui.workbench.renderers.swt.ContributedPartRenderer.createWidget(ContributedPartRenderer.java:129)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createWidget(PartRenderingEngine.java:975)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:651)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:757)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$0(PartRenderingEngine.java:728)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$2.run(PartRenderingEngine.java:722)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:706)
at org.eclipse.e4.ui.internal.workbench.PartServiceImpl$1.handleEvent(PartServiceImpl.java:104)
at org.eclipse.e4.ui.services.internal.events.UIEventHandler$1.run(UIEventHandler.java:40)
at org.eclipse.swt.widgets.Synchronizer.syncExec(Synchronizer.java:233)
at org.eclipse.ui.internal.UISynchronizer.syncExec(UISynchronizer.java:145)
at org.eclipse.swt.widgets.Display.syncExec(Display.java:5414)
at org.eclipse.e4.ui.internal.workbench.swt.E4Application$1.syncExec(E4Application.java:211)
at org.eclipse.e4.ui.services.internal.events.UIEventHandler.handleEvent(UIEventHandler.java:36)
at org.eclipse.equinox.internal.event.EventHandlerWrapper.handleEvent(EventHandlerWrapper.java:201)
at org.eclipse.equinox.internal.event.EventHandlerTracker.dispatchEvent(EventHandlerTracker.java:197)
at org.eclipse.equinox.internal.event.EventHandlerTracker.dispatchEvent(EventHandlerTracker.java:1)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
at org.eclipse.equinox.internal.event.EventAdminImpl.dispatchEvent(EventAdminImpl.java:135)
at org.eclipse.equinox.internal.event.EventAdminImpl.sendEvent(EventAdminImpl.java:78)
at org.eclipse.equinox.internal.event.EventComponent.sendEvent(EventComponent.java:39)
at org.eclipse.e4.ui.services.internal.events.EventBroker.send(EventBroker.java:94)
at org.eclipse.e4.ui.internal.workbench.UIEventPublisher.notifyChanged(UIEventPublisher.java:60)
at org.eclipse.emf.common.notify.impl.BasicNotifierImpl.eNotify(BasicNotifierImpl.java:374)
at org.eclipse.e4.ui.model.application.ui.impl.ElementContainerImpl.setSelectedElement(ElementContainerImpl.java:173)
at org.eclipse.e4.ui.internal.workbench.ModelServiceImpl.showElementInWindow(ModelServiceImpl.java:617)
at org.eclipse.e4.ui.internal.workbench.ModelServiceImpl.bringToTop(ModelServiceImpl.java:581)
at org.eclipse.e4.ui.internal.workbench.PartServiceImpl.delegateBringToTop(PartServiceImpl.java:770)
at org.eclipse.e4.ui.internal.workbench.PartServiceImpl.showPart(PartServiceImpl.java:1211)
at org.eclipse.ui.internal.WorkbenchPage.busyOpenEditor(WorkbenchPage.java:3261)
at org.eclipse.ui.internal.WorkbenchPage.access$25(WorkbenchPage.java:3176)
at org.eclipse.ui.internal.WorkbenchPage$10.run(WorkbenchPage.java:3158)
at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:70)
at org.eclipse.ui.internal.WorkbenchPage.openEditor(WorkbenchPage.java:3153)
at org.eclipse.ui.internal.WorkbenchPage.openEditor(WorkbenchPage.java:3117)
at org.eclipse.ui.internal.WorkbenchPage.openEditor(WorkbenchPage.java:3107)
at org.eclipse.dltk.internal.ui.editor.EditorUtility.openInEditor(EditorUtility.java:412)
at org.eclipse.dltk.internal.ui.editor.EditorUtility.openInEditor(EditorUtility.java:175)
at org.eclipse.dltk.internal.ui.actions.OpenActionUtil.open(OpenActionUtil.java:45)
at org.eclipse.dltk.ui.actions.OpenAction.run(OpenAction.java:266)
at org.eclipse.dltk.ruby.internal.ui.text.hyperlink.RubyRequireHyperlink.doOpen(RubyRequireHyperlink.java:89)
at org.eclipse.dltk.ruby.internal.ui.text.hyperlink.RubyRequireHyperlink.open(RubyRequireHyperlink.java:65)
at org.eclipse.jface.text.hyperlink.HyperlinkManager.mouseUp(HyperlinkManager.java:424)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:221)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:5227)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1340)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4561)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:4151)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$4.run(PartRenderingEngine.java:1121)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1022)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:150)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:693)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:610)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:148)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:138)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:673)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:610)
at org.eclipse.equinox.launcher.Main.run(Main.java:1519)
at org.eclipse.equinox.launcher.Main.main(Main.java:1492)

But it works just fine for ruby files inside the workspace.

Provide extension point to register theme (CSS, TextMate theme, eclipse prefs) in other words to register ITokenProvider.

• rename to org.eclipse.textmate4e package
• provide Rename to org.eclipse.textmate4e.core + org.eclipse.textmate4e.ui plugins

See microsoft/vscode#16206 (comment) and https://github.com/Microsoft/vscode-textmate/blob/master/src/tests/themedTokenizer.ts#L25

The tokenizeLine2 seems to provide:

• binary result (perhaps it will improve memory?)
• result is a merge between the grammar and the theme (perhaps it will improve performance?)

Provide an update site

I'm playing wiht PHP syntax coloring and I noticed Index out of bounds exception in some cases. I'm using latest master.

To reproduce:

• create file, its important to have many empty lines at the end (e.g. 4)

<?php

$aaa = new Test();
str(

• put cursor at the end of file
• remove all empty lines till '(' (with backspace)

Stacktrace:
java.lang.IllegalArgumentException: Index out of bounds
at org.eclipse.swt.SWT.error(SWT.java:4514)
at org.eclipse.swt.SWT.error(SWT.java:4448)
at org.eclipse.swt.SWT.error(SWT.java:4419)
at org.eclipse.swt.custom.StyledText.setStyleRanges(StyledText.java:9910)
at org.eclipse.swt.custom.StyledText.replaceStyleRanges(StyledText.java:7911)
at org.eclipse.jface.text.TextViewer.addPresentation(TextViewer.java:4692)
at org.eclipse.jface.text.TextViewer.changeTextPresentation(TextViewer.java:4769)
at org.eclipse.tm4e.ui.text.TMPresentationReconciler.applyTextRegionCollection(TMPresentationReconciler.java:493)
at org.eclipse.tm4e.ui.text.TMPresentationReconciler.colorize(TMPresentationReconciler.java:394)
at org.eclipse.tm4e.ui.text.TMPresentationReconciler.access$5(TMPresentationReconciler.java:326)
at org.eclipse.tm4e.ui.text.TMPresentationReconciler$InternalListener.textChanged(TMPresentationReconciler.java:191)
at org.eclipse.jface.text.TextViewer.updateTextListeners(TextViewer.java:2699)
at org.eclipse.jface.text.TextViewer.invalidateTextPresentation(TextViewer.java:3344)
at org.eclipse.jface.text.source.AnnotationPainter.invalidateTextPresentation(AnnotationPainter.java:970)
at org.eclipse.jface.text.source.AnnotationPainter.updatePainting(AnnotationPainter.java:952)
at org.eclipse.jface.text.source.AnnotationPainter.access$1(AnnotationPainter.java:946)
at org.eclipse.jface.text.source.AnnotationPainter$1.run(AnnotationPainter.java:1079)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:37)

Static analysis show multiple errors in the code that for sure will lead to bugs or under-efficient code. They are reported at https://dev.eclipse.org/sonar/dashboard/index/149544 . At least, the Blocker and Critical issues should be fixed.

When I try mvn clean verify on some machines (including my laptop), I get compilation errors because junit classes can't be resolved.
Is this something you happen to see too @pascalleclercq @angelozerr ?

Not strictly an issue but writing to ask if this library is available in a standalone "core" version without any dependencies on Eclipse. I really would like to see this integrated into a Swing or FX editor pane. I saw that there was an issue raised on the RSyntaxTextArea github issues page regarding this, but it appears that the core project still contains Eclipse PDT dependencies.

Chris

Do you know if anyone is working on a Swing, SWT (non Eclipse) or JavaFX component that can sit atop of this library?

I attempted to integrate TM4E with RSyntaxTextArea but my gut feeling is that they were incompatible.

Looking at RSyntaxTextArea's documentation is it the opinion of the developers that it is possible to hook up TM4E efficiently? If I know it is is possible, I'll go back and take a second look.

• do tests for for TMPresentationReconciler
• implement a test generator to generate tests when editor consumes TextMate grammar.

VSCode uses a JSON representation for TextMate like for C#. See https://github.com/OmniSharp/omnisharp-vscode/blob/master/syntaxes/csharp.json

Support this JSON Parser like XML PList.

TextMate grammar can define folding https://manual.macromates.com/en/navigation_overview

It should be cool if tm4e could provide the org.eclipse.tm4e.ui.folding.TMFoldingStrategy which supports this feature.

see $subject

tm4e really should be in EPP together with a library of as many textmate files as possible IMO.

Should not require users to install additional libraries or having to do another tycho plugin buil dsetup just to get started on syntax highlighting imo :)

The colorization is done with async mode to avoid having freeze (like VSCode does): it means that TextMate model is computed in a Thread and when it is available the colorization is done.

It works great (with a big file like https://raw.githubusercontent.com/Microsoft/TypeScript/master/lib/tsserver.js) the editor doens't freeze and colorization is done step by step (you can see colorization for the first lines as soon as editor is opened and the other lines are colorized when model si parsed).

But when you try to update th eeditor when colorization is done for big files, TextMate model and editor is not synch.

The AbstractTMModel is a complex class, with a lot of synchronized. We should find out ways to simplify it and better separate the thread and the model. I suggest we try to make the BackgroundThread static and we avoid to consume fields from AbstractTMModel directly, and try to avoid sharing any reference and pass arguments by value as much as possible. That should progressively remove all needs for synchronized and should hopefully improve performance.

I have java.lang.IllegalArgumentException: Argument not valid when I'm working with PHP file colored with textmate4e.

To reproduce:

• open/create file

<?php

$aaa = new Test();
collator_get_|error_code()|

• delete text from between '|'

Stacktrace:

java.lang.IllegalArgumentException: Argument not valid
	at org.eclipse.swt.SWT.error(SWT.java:4514)
	at org.eclipse.swt.SWT.error(SWT.java:4448)
	at org.eclipse.swt.SWT.error(SWT.java:4419)
	at org.eclipse.swt.custom.StyledText.setStyleRanges(StyledText.java:9928)
	at org.eclipse.swt.custom.StyledText.replaceStyleRanges(StyledText.java:7911)
	at org.eclipse.jface.text.TextViewer.addPresentation(TextViewer.java:4692)
	at org.eclipse.jface.text.TextViewer.changeTextPresentation(TextViewer.java:4769)
	at org.eclipse.tm4e.ui.text.TMPresentationReconciler.applyTextRegionCollection(TMPresentationReconciler.java:426)
	at org.eclipse.tm4e.ui.text.TMPresentationReconciler.colorize(TMPresentationReconciler.java:351)
	at org.eclipse.tm4e.ui.text.TMPresentationReconciler.access$4(TMPresentationReconciler.java:311)
	at org.eclipse.tm4e.ui.text.TMPresentationReconciler$InternalListener.textChanged(TMPresentationReconciler.java:185)
	at org.eclipse.jface.text.TextViewer.updateTextListeners(TextViewer.java:2699)
	at org.eclipse.jface.text.TextViewer$VisibleDocumentListener.documentChanged(TextViewer.java:401)
	at org.eclipse.jface.text.AbstractDocument.doFireDocumentChanged2(AbstractDocument.java:743)
	at org.eclipse.jface.text.AbstractDocument.doFireDocumentChanged(AbstractDocument.java:712)
	at org.eclipse.jface.text.AbstractDocument.doFireDocumentChanged(AbstractDocument.java:696)
	at org.eclipse.jface.text.AbstractDocument.fireDocumentChanged(AbstractDocument.java:770)
	at org.eclipse.jface.text.AbstractDocument.replace(AbstractDocument.java:1101)
	at org.eclipse.core.internal.filebuffers.SynchronizableDocument.replace(SynchronizableDocument.java:173)
	at org.eclipse.jface.text.AbstractDocument.replace(AbstractDocument.java:1119)
	at org.eclipse.core.internal.filebuffers.SynchronizableDocument.replace(SynchronizableDocument.java:161)
	at org.eclipse.jface.text.DefaultDocumentAdapter.replaceTextRange(DefaultDocumentAdapter.java:233)
	at org.eclipse.swt.custom.StyledText.modifyContent(StyledText.java:7350)

Version from master.

Provide an IInformationControl to render Markdown syntax with hover

See https://bugs.eclipse.org/bugs/show_bug.cgi?id=508560

Eclipse.org bundle should be pack200'd and signed using the jarsigner Maven plugin, so they would be faster to download and wouldn't show a pop-up on install. See https://wiki.eclipse.org/Tycho/Pack200#Pack200_and_Signing

I am trying to use this with the generic editor for YAML files, but it throws this exception:

java.lang.NullPointerException
	at org.eclipse.tm4e.core.model.Tokenizer.tokenize(Tokenizer.java:49)
	at org.eclipse.tm4e.core.model.AbstractTMModel._updateTokensUntilLine(AbstractTMModel.java:533)
	at org.eclipse.tm4e.core.model.AbstractTMModel.lambda$0(AbstractTMModel.java:494)
	at org.eclipse.tm4e.core.model.AbstractTMModel._withModelTokensChangedEventBuilder(AbstractTMModel.java:435)
	at org.eclipse.tm4e.core.model.AbstractTMModel._revalidateTokensNow(AbstractTMModel.java:448)
	at org.eclipse.tm4e.core.model.AbstractTMModel.access$3(AbstractTMModel.java:447)
	at org.eclipse.tm4e.core.model.AbstractTMModel$BackgroundThread.run(AbstractTMModel.java:200)

I am using this grammar file:
https://raw.githubusercontent.com/textmate/yaml.tmbundle/master/Syntaxes/YAML.tmLanguage

and configured the extension like this:

   <extension
         point="org.eclipse.tm4e.core.grammars">
      <grammar
      		scopeName="cf.manifest.yml"
            path="./syntaxes/YAML.tmLanguage" >
      </grammar>      
      <scopeNameContentTypeBinding
            contentTypeId="org.springframework.boot.ide.manifest.yml"
            scopeName="cf.manifest.yml">
      </scopeNameContentTypeBinding>
   </extension>

Any idea what is going wrong here?

When I type in a syntax highlighted region of a document, as I type, the text I type is always white, even if the region I am typing in (for example) is yellow. I can see that the syntax highlighting only executes when I stop typing for a few milliseconds, then the text I just typed becomes yellow as it should (in this particular example).

I think the better behaviour is to either adopt the style of the character to the left when typing, then do the full recalculate with corrections on the existing time OR even better, to calculate the style of the first character only in context, then use that for all the typed characters before fully re-rendering.

The second approach is better than the first in my opinion in the case of quoted text. In quoted text, the quotes might be of the punctuation style, and the text will be in the literal text style. If we take the first approach and adopt the character to the left (or the style of the last character on the previous line, then as the user types, it will be in the style of the punctuation. Taking the second approach it will be in the style of the quoted text.

The ideal approach is obviously to calculate the style per character instantly but I don't know if that is possible within the constraints of the implementation so calculating the first character then carrying that forward until a full revalidation of the dirty region is performed seems like a reasonable workaround.

It should be cleaner to remove Joni&JCoding JARS from org.eclipse.textmate4e.core and use import packages.

But there are 2 problems:

• Joni&JCoding are not OSGi. See jruby/joni#26
• Joni&JCoding should be in Eclipse orbit.

@mickaelistria have you an idea how to fix thoses 2 problems?

Today the TM default theme is everytime ligtht. When user changes the "Apparence Theme" to dark it should select the TM theme "dark".

When plugins which contains grammar, theme is not unzipped, the theme, grammar extension point cannot be loaded.

Improve that with http://blog.vogella.com/2010/07/06/reading-resources-from-plugin/

Since org.eclipse.tm4e.core is Eclipse-independent and can be used in plain Java context, it would be great to publish it to Nexus: https://bugs.eclipse.org/bugs/show_bug.cgi?id=514346

Once GenericEditor presentationReconcilers will support "org.eclipse.core.runtime.text" to bind IPresentationReconclier (see https://bugs.eclipse.org/bugs/show_bug.cgi?id=507772), the org.eclipse.textmate4e.ui could declare in the plugin.xml:

<extension
         point="org.eclipse.ui.genericeditor.presentationReconcilers">
      <presentationReconciler           class="org.eclipse.language.textmate.eclipse.text.TMPresentationReconciler"
            contentType="org.eclipse.core.runtime.text">

It will give the capability to register a TextMate grammar with UI preferences and when user will open a file with GenericEditor, the syntax coloration should be done.

See https://bugs.eclipse.org/bugs/show_bug.cgi?id=507753#c2

It should be cool to provide a global TextMate preferences with those preferences pages:

• TextMate
  • Editor: this page display a table which show links between content type and TextMate grammar. Those links comes from:
    • extension point
    • user preference (provide an Add/Remove buttons to add/remove links between content type and grammar)
  • Grammar: this page display a table which show regsitered grammars:
    • extension point
    • user preference (provide an Add/Remove buttons to add/remove a TextMate grammar.
  • Theme: this page display a table which show regsitered themes (CSS and TextMate themes):
    • extension point
    • user preference (provide an Add/Remove buttons to add/remove a theme.

@mickaelistria what do you think about this idea? Do you like preference page names?

As this project fits in the big theme of Eclipse Platform and IDE "Reduce cost of adding support for new languages" and as it completes the LSP4E project which is missing a good implementation of syntax highlighting, this Eclipse/TextMate integration would be a great project to contribute to Eclipse.org!

Current state:

• Project approved and provisioned https://projects.eclipse.org/projects/technology.tm4e
• Move to version 0.1.0 (issue #35)
• Get confirmation for copyright header and update if necessary, see #37 and #39
• Create CQ for initial contribution https://dev.eclipse.org/ipzilla/show_bug.cgi?id=12577

Blockers

• Create CQ for jcodings use in TM4E (piggy back Orbit CQ)
• Create CQ for joni use in TM4E (piggy back Orbit CQ)
• GitHub repository at Eclipse.org
• Push initial code to Eclipse GitHub repo

Other tasks (releng) are non-blockers and will be tracked in the "official" repo

In the case with:

• big file
• editor is plug with code folding, validation, etc which invalidates the text viewer

We have the following problem:

java.lang.NullPointerException
	at org.eclipse.tm4e.ui.text.TMPresentationReconciler.colorize(TMPresentationReconciler.java:366)
	at org.eclipse.tm4e.ui.text.TMPresentationReconciler.access$5(TMPresentationReconciler.java:346)
	at org.eclipse.tm4e.ui.text.TMPresentationReconciler$InternalListener.textChanged(TMPresentationReconciler.java:225)
	at org.eclipse.jface.text.TextViewer.updateTextListeners(TextViewer.java:2699)
	at org.eclipse.jface.text.TextViewer.invalidateTextPresentation(TextViewer.java:3344)
	at org.eclipse.jface.text.source.AnnotationPainter.invalidateTextPresentation(AnnotationPainter.java:970)
	at org.eclipse.jface.text.source.AnnotationPainter.updatePainting(AnnotationPainter.java:952)
	at org.eclipse.jface.text.source.AnnotationPainter.access$1(AnnotationPainter.java:946)
	at org.eclipse.jface.text.source.AnnotationPainter$1.run(AnnotationPainter.java:1079)
	at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
	at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:182)
	at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4211)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3827)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$4.run(PartRenderingEngine.java:1121)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1022)
	at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:150)
	at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:693)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
	at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:610)
	at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:148)
	at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:138)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:673)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:610)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1519)
	at org.eclipse.equinox.launcher.Main.main(Main.java:1492)

@mickaelistria I think it's a very important issue, tell me if you need more info.

As the infrastructure is moving to Eclipse.org, maybe there's an opportunity to drop the reference to org.sonatype.oss:oss-parent .
@pascalleclercq Which feature provided by the parent is tm4e using?

When importing textmate.java into the IDE, I see build issues because of missing src/*/resource folders.

I'm using the version from the update site, and it can't seem to find extension point 'org.eclipse.textmate4e.core.grammars' even though I have imported the correct bundles.

Require-Bundle: org.eclipse.ui,
 org.eclipse.core.runtime,
 org.eclipse.core.resources,
 org.eclipse.jdt.launching,
 org.eclipse.jdt.core,
 org.eclipse.debug.ui,
 org.eclipse.jdt.debug,
 org.eclipse.jdt.debug.ui,
 org.eclipse.jdt.ui,
 org.eclipse.jface.text,
 org.eclipse.ui,
 org.eclipse.core.resources,
 org.eclipse.core.filesystem,
 org.eclipse.ui.editors,
 org.eclipse.ui.views,
 org.eclipse.textmate4e.core,
 org.eclipse.textmate4e.ui

If I go to the "Add Extensions GUI in Eclipse, the extension point is missing there too:

Today, the CSS theme are hosted in the sample plugin https://github.com/angelozerr/textmate.java/tree/master/eclipse/org.eclipse.language.textmate.eclipse.samples/themes but I think it should be perhaps better to provide default theme, in other words I would like to move those CSS themes inside org.eclipse.language.textmate.eclipse.

@mickaelistria are you OK with that?

It'd be nice to have monokai theme support included.