Heredoc improperly highlighted. about language-ruby HOT 12 OPEN

I don't settle for bugs, so: no.

from language-ruby.

I tried playing around with this and learned a few things:

1. There are a few different rules that apply to heredocs:
   a. vanilla heredoc:

   language-ruby/grammars/ruby.cson

   Lines 1817 to 1838 in d88a4cf

   	{
   	'begin': '(?>=\\s*<<(\\w+))'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'end': '^\\1$'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}

   
   b. special heredocs that signal other languages (HTML, XML, SQL, GRAPHQL, CSS, CPP, C, JS, JQUERY, SH, LUA, and RUBY):

   language-ruby/grammars/ruby.cson

   Lines 1409 to 1816 in d88a4cf

   	{
   	'begin': '(?=(?><<[-~]("?)((?:[_\\w]+_|)HTML)\\b\\1))'
   	'comment': 'Heredoc with embedded html'
   	'end': '(?!\\G)'
   	'name': 'meta.embedded.block.html'
   	'patterns': [
   	{
   	'begin': '(?><<[-~]("?)((?:[_\\w]+_|)HTML)\\b\\1)'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'contentName': 'text.html'
   	'end': '^\\s*\\2$\\n?'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': 'text.html.basic'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}
   	]
   	}
   	{
   	'begin': '(?=(?><<[-~]("?)((?:[_\\w]+_|)XML)\\b\\1))'
   	'comment': 'Heredoc with embedded xml'
   	'end': '(?!\\G)'
   	'name': 'meta.embedded.block.xml'
   	'patterns': [
   	{
   	'begin': '(?><<[-~]("?)((?:[_\\w]+_|)XML)\\b\\1)'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'contentName': 'text.xml'
   	'end': '^\\s*\\2$\\n?'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': 'text.xml'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}
   	]
   	}
   	{
   	'begin': '(?=(?><<[-~]("?)((?:[_\\w]+_|)SQL)\\b\\1))'
   	'comment': 'Heredoc with embedded sql'
   	'end': '(?!\\G)'
   	'name': 'meta.embedded.block.sql'
   	'patterns': [
   	{
   	'begin': '(?><<[-~]("?)((?:[_\\w]+_|)SQL)\\b\\1)'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'contentName': 'source.sql'
   	'end': '^\\s*\\2$\\n?'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': 'source.sql'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}
   	]
   	}
   	{
   	'begin': '(?=(?><<[-~]("?)((?:[_\\w]+_|)GRAPHQL)\\b\\1))'
   	'comment': 'Heredoc with embedded GraphQL'
   	'end': '(?!\\G)'
   	'name': 'meta.embedded.block.graphql'
   	'patterns': [
   	{
   	'begin': '(?><<[-~]("?)((?:[_\\w]+_|)GRAPHQL)\\b\\1)'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'contentName': 'source.graphql'
   	'end': '^\\s*\\2$\\n?'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': 'source.graphql'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}
   	]
   	}
   	{
   	'begin': '(?=(?><<[-~]("?)((?:[_\\w]+_|)CSS)\\b\\1))'
   	'comment': 'Heredoc with embedded css'
   	'end': '(?!\\G)'
   	'name': 'meta.embedded.block.css'
   	'patterns': [
   	{
   	'begin': '(?><<[-~]("?)((?:[_\\w]+_|)CSS)\\b\\1)'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'contentName': 'source.css'
   	'end': '^\\s*\\2$\\n?'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': 'source.css'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}
   	]
   	}
   	{
   	'begin': '(?=(?><<[-~]("?)((?:[_\\w]+_|)CPP)\\b\\1))'
   	'comment': 'Heredoc with embedded c++'
   	'end': '(?!\\G)'
   	'name': 'meta.embedded.block.cpp'
   	'patterns': [
   	{
   	'begin': '(?><<[-~]("?)((?:[_\\w]+_|)CPP)\\b\\1)'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'contentName': 'source.cpp'
   	'end': '^\\s*\\2$\\n?'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': 'source.cpp'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}
   	]
   	}
   	{
   	'begin': '(?=(?><<[-~]("?)((?:[_\\w]+_|)C)\\b\\1))'
   	'comment': 'Heredoc with embedded c'
   	'end': '(?!\\G)'
   	'name': 'meta.embedded.block.c'
   	'patterns': [
   	{
   	'begin': '(?><<[-~]("?)((?:[_\\w]+_|)C)\\b\\1)'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'contentName': 'source.c'
   	'end': '^\\s*\\2$\\n?'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': 'source.c'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}
   	]
   	}
   	{
   	'begin': '(?=(?><<[-~]("?)((?:[_\\w]+_|)(?:JS|JAVASCRIPT))\\b\\1))'
   	'comment': 'Heredoc with embedded javascript'
   	'end': '(?!\\G)'
   	'name': 'meta.embedded.block.js'
   	'patterns': [
   	{
   	'begin': '(?><<[-~]("?)((?:[_\\w]+_|)(?:JS|JAVASCRIPT))\\b\\1)'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'contentName': 'source.js'
   	'end': '^\\s*\\2$\\n?'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': 'source.js'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}
   	]
   	}
   	{
   	'begin': '(?=(?><<[-~]("?)((?:[_\\w]+_|)JQUERY)\\b\\1))'
   	'comment': 'Heredoc with embedded jQuery javascript'
   	'end': '(?!\\G)'
   	'name': 'meta.embedded.block.js.jquery'
   	'patterns': [
   	{
   	'begin': '(?><<[-~]("?)((?:[_\\w]+_|)JQUERY)\\b\\1)'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'contentName': 'source.js.jquery'
   	'end': '^\\s*\\2$\\n?'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': 'source.js.jquery'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}
   	]
   	}
   	{
   	'begin': '(?=(?><<[-~]("?)((?:[_\\w]+_|)(?:SH|SHELL))\\b\\1))'
   	'comment': 'Heredoc with embedded shell'
   	'end': '(?!\\G)'
   	'name': 'meta.embedded.block.shell'
   	'patterns': [
   	{
   	'begin': '(?><<[-~]("?)((?:[_\\w]+_|)(?:SH|SHELL))\\b\\1)'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'contentName': 'source.shell'
   	'end': '^\\s*\\2$\\n?'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': 'source.shell'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}
   	]
   	}
   	{
   	'begin': '(?=(?><<[-~]("?)((?:[_\\w]+_|)LUA)\\b\\1))'
   	'comment': 'Heredoc with embedded lua'
   	'end': '(?!\\G)'
   	'name': 'meta.embedded.block.lua'
   	'patterns': [
   	{
   	'begin': '(?><<[-~]("?)((?:[_\\w]+_|)LUA)\\b\\1)'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'contentName': 'source.lua'
   	'end': '^\\s*\\2$\\n?'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': 'source.lua'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}
   	]
   	}
   	{
   	'begin': '(?=(?><<[-~]("?)((?:[_\\w]+_|)RUBY)\\b\\1))'
   	'comment': 'Heredoc with embedded ruby'
   	'end': '(?!\\G)'
   	'name': 'meta.embedded.block.ruby'
   	'patterns': [
   	{
   	'begin': '(?><<[-~]("?)((?:[_\\w]+_|)RUBY)\\b\\1)'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'contentName': 'source.ruby'
   	'end': '^\\s*\\2$\\n?'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': 'source.ruby'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}
   	]
   	}

   
   c. a more complicated vanilla pattern that covers opening multiple heredocs in the same line:

   language-ruby/grammars/ruby.cson

   Lines 1839 to 1861 in d88a4cf

   	{
   	'begin': '(?>((<<[-~](\\w+),\\s?)*<<[-~](\\w+)))'
   	'beginCaptures':
   	'0':
   	'name': 'punctuation.definition.string.begin.ruby'
   	'comment': 'heredoc with multiple inputs and indented terminator'
   	'end': '^\\s*\\4$'
   	'endCaptures':
   	'0':
   	'name': 'punctuation.definition.string.end.ruby'
   	'name': 'string.unquoted.heredoc.ruby'
   	'patterns': [
   	{
   	'include': '#heredoc'
   	}
   	{
   	'include': '#interpolated_ruby'
   	}
   	{
   	'include': '#escaped_char'
   	}
   	]
   	}

Though Atom's guide for creating a grammar is still WIP, the TextMate Manual is very helpful.

For most cases, it shouldn't be too hard to fix things. If we add a capture group for the remainder of the line, we can apply a beginCapture pattern to include source.ruby for that group.

This naive approach broke down for me when I tried to adjust the rule for multiple heredocs per line. For example, this bit of code:

<<-ONE + <<-TWO
  Hello,
ONE
  world!
TWO
# => "Hello, \nworld!\n"

I could sort of approximate the right behavior here, but ran into some hurdles. I think we'll have to settle for something less than perfection here, because:

the regular expressions are matched against only a single line of the document at a time.
http://manual.macromates.com/en/language_grammars#language_rules

from language-ruby.

Is there anyone looking at this? Is there anything I can do to help move the needle forward?

from language-ruby.

@envygeeks and @SampsonCrowley your comments were deleted as a violation of the Atom Code of Conduct as they were insulting or derogatory. You may both consider this an official warning.

from language-ruby.

Since this issue has been open for over three years now, I would welcome even a partial fix at this point. I agree, I don't want to settle for a partial fix indefinitely, but @SamGerber's proposed solution would solve the problem for a majority of cases.

from language-ruby.

@kevinsawicki also adding that this ends up in some funky highlighting of everything after the heredoc:

<<-SQL
"
SQL

It doesn't happen with STR but it does happen when invoking SQL, Liquid and others!

from language-ruby.

from language-ruby.

😞

I would expect line 3 to use proper highlighting at , locals: { params: params }. In Vim:

from language-ruby.

).where(blackout_dates: { id: nil }) is all highlighted as SQL.

from language-ruby.

I'm not sure this is obtainable using regex.

from language-ruby.

@envygeeks then you'll need to test #225, because as @SamGerber and I have both discovered you'll be waiting here forever if you want a bug-free regex fix. Though I should say that your attitude is pretty discouraging for someone who just wants to help out :/.

from language-ruby.

I didn't say it had to be a "bug-free regex fix", I didn't even mention regex at all; I said, I don't settle for unfixed bugs, or being told that bugs can't be fixed. And my "attitude" is not discouraging, stop trying to shame people because they dissent to an opinion expressed.

And I am happy to test #225, but that pull-request states no way to test it, without having to build my own package, and ship into Atom editor, manually, I think it there were a beta pipeline for such testing then my comment would have never of been made in the first place as I would have traced it back here and tried it already, and started commenting.

from language-ruby.