nathansobo / treetop Goto Github PK

the docs indicate that polyglot is optional. from http://treetop.rubyforge.org/using_in_ruby.html

In order to use Polyglot dynamic loading of .treetop or .tt files though, you need to require the Polyglot gem before you require the Treetop gem as Treetop will only create hooks into Polyglot for the treetop files if Polyglot is already loaded. So you need to use:

require 'polyglot'
require 'treetop'

in order to use Polyglot auto loading with Treetop in Ruby.

I for one would prefer the documented behavior. However, I can see that in the treetop gem on my system, Polyglot is required by default. Messing with Kernel#require by default smells like bad practice.

In my case, I need treetop as a dependency for json_select. I have no need for treetop otherwise, or dynamic loading of .tt files. But now when I'm debugging require issues in my project, I've got to consider if polyglot's require is behaving differently from Kernel's require.

In any case, at the very least, the docs should be updated to match the actual behavior.

Instead of:

grammar QueryGrammar
  rule attribute
    [a-zA-Z0-9_.]+ <Search::Parser::QueryGrammar::Attribute>
  end
end

It'd be nice if I could write:

grammar Search::Parser::QueryGrammar
  rule attribute
    [a-zA-Z0-9_.]+ <Attribute>
  end
end

Apologies if there is a way to do this, but just adding the colons in the grammar name gave me an error: RuntimeError: Expected one of [A-Za-z_], [0-9], [ \t\n\r], '#' at line 1, column 15 (byte 15) after grammar Search

ActiveSupport now includes its own String#indent method in Core Extensions:

https://github.com/rails/rails/blob/master/activesupport/lib/active_support/core_ext/string/indent.rb

Since this library depends on ActiveSupport, any project that depends on TreeTop will also have the ActiveSupport code available. Thus I suggest deprecating it here.

Line 17 checks for the existence of all the files, and File.exist?("-o") fails.

I can throw together a patch tonight.

In developing a parser, I came across this example at the bottom of the "Semantics" page in the rubyforge documentation:

rule labels
  first_letter:[a-z] rest_letters:(', ' letter:[a-z])* {
    def letters
      [first_letter] + rest_letters
    end

    def rest_letters
      super.map { |comma_and_letter| comma_and_letter.letter }
    end
  }
end

I was trying to get this to work, with a + node, but the call to "super.map" always failed. I believe it is supposed to be super.elements.map, as this is how it is done in the previous example on the page, and how I got it to work in my case.

I have my rules defined like this:

grammar Language

  rule integer
    [0-9]+
  end

  rule identifier
    [a-z]+
  end

end

And when I try to parse any number, it's fine. But, when I try to parse any identifier, it don't work. Then, I inverted the rules and tried again, guess what? I was able do parse identifiers and not integers. Somebody know what's happening here?

Treetop is not released from this repository, and issues may not be acknowledged. Please submit them to my repository instead.

Wish you could require files within treetop files and it would just pass them through.

Something like:

require File.dirname(__FILE__) + "/attributes/class_id"

module RedCloth
  module Parser
    grammar Attributes
      include ClassId

Thank you for the excellent parser generator library. It makes it straightforward to handle complex grammars in ruby.

I ran into a problem where a regular expression looking for a left curly brace was causing Treetop parsing to fail. Here is a patch with a test case that will trigger the issue:

diff --git a/spec/compiler/grammar_spec.rb b/spec/compiler/grammar_spec.rb
index 7b8e515..f16935d 100644
--- a/spec/compiler/grammar_spec.rb
+++ b/spec/compiler/grammar_spec.rb
@@ -38,4 +38,21 @@ module GrammarSpec
       self.class.const_get(:Foo).ancestors.should include(GrammarSpec::Bar)
     end
   end
+
+  describe "handling of curly braces" do
+    before do
+      @grammar = %{
+        grammar Test
+          rule foo
+            'foo' { def foo_method(str); str =~ /\{/ end }
+          end
+        end
+      }
+    end
+
+    it "does not raise for an unbalanced left curly brace within a block" do
+      expect { parse_with_metagrammar(@grammar.strip, :grammar) }.to_not raise_error
+    end
+  end
 end

The rule in metagrammar.treetop that is leading to the exception appears to be this one:

  rule inline_module
    '{' (inline_module / ![{}] .)* '}' <InlineModule>
  end

I think the logic needs to be a little more complex here. I would have attempted a fix, but I'm having difficulty getting the existing specs to pass.

Regards,
Eric

Hi.

I'm throwing together a toy language at https://github.com/jamesotron/CrimsonScript - so far it has only a simple description of nil and integer literals:

rule literal
   integer_literal / nil_literal <Literal>
end

rule integer_literal
  hex_integer_literal / octal_integer_literal / binary_integer_literal / decimal_integer_literal / zero_integer_literal <IntegerLiteral>
end

rule nil_literal
  "nil" <NilLiteral>
end

rule zero_integer_literal
  '-'? '0'
end

rule decimal_integer_literal
  '-'? [1-9] [0-9]*
end

rule binary_integer_literal
  '-'? '0b' [0-1]+
end

rule octal_integer_literal
  '-'? '0o' [0-7]+
end

rule hex_integer_literal
  '-'? '0x' [0-9a-fA-F]+
end

However when I try and parse 0 I get:

1.9.3-p0 :001 > Crimson::Parser.parse('0')
TypeError: wrong argument type Class (expected Module)
    from (eval):72:in `extend'
    from (eval):72:in `_nt_integer_literal'
    from (eval):24:in `_nt_literal'
    from /Users/jnh/.rvm/gems/ruby-1.9.3-p0@crimsonscript/gems/treetop-1.4.10/lib/treetop/runtime/compiled_parser.rb:18:in `parse'
    from /Users/jnh/Dev/Toys/CrimsonScript/lib/crimson/parser.rb:11:in `parse'
    from (irb):1
    from /Users/jnh/.rvm/rubies/ruby-1.9.3-p0/bin/irb:16:in `<main>'

I'm not sure what I'm doing wrong, but when I compile the grammar to ruby I can see that it's calling #extend on the result of _nt_zero_integer_literal. I'm not sure why that's happening. Is this a bug?

Hi! Thanks for the nice tool.

I am trying to implement a parse for Toki Pona language. and I faced an issue. The parser can't distinguish between sin and sina. Semantically it's 2 different words, but if parser sees sina it detects its as sin. I can't change the order, because sin has higher priority.

Thanks!

For rails projects, it might be nice to have a rake task that generates grammars and their respective tests

In Ruby, module A::B::C sets the scope to the module C inside A::B. Currently, treetop doesn’t support this syntax. It would be easy to add by changing alphanumeric_char* to (alphanumeric_char / '::')* in the module_declaration rule.

Nowhere in the site or in the docs can i find any mention of whether this is expected to run on 1.9.2

i tried to run the test suite but there is no Gemfile and i don't know what version of rspec this requires and as of rspec (2.3.0), when doing rake -T
no such file to load -- spec/rake/spectask

because this has changed to 'rspec/core/rake_task' at some point,

and when trying to run the test suite itself with "rspec ." i get similar results.

It's annoying that changes in rspec broke this, but this is of course what bundler is for. if you tell me the version on rspec i can probably make a Gemfile and give u a pull request.

Thanks!

I’m sorry that I didn’t think of it when I suggested issue #15, but would it be possible to support

grammar A::B::C
  …
end

to define grammar C under A::B?

Also, since this (and issue #15) adds to the public API, shouldn’t this be 1.5.0 so that one can use '~> 1.5.0' in one’s gemspec if one depends on this feature?

I'm a Gentoo Linux developer and I'm working on packaging of Ruby extensions. As it is the treetop gems released have no testsuite to speak of, which makes it very difficult to know whether the code is working correctly in our environment, in particular on Ruby 1.9 and JRuby.

We can use tarballs instead of gems, and GitHub tags in place of tarballs, but for treetop there seem to be no option for that. Could you please consider either releasing a new gem with the specs or tagging the previous releases?

Thanks!

Hi,

I believe the string parsing example in the website should be changed: the order of the 'escaped quote' and 'any character' choice should be swapped. In other words, the string parsing expression should be:

rule string
   '"' ('\"' / !'"' . )* '"'
end

The following test case illustrates this point:

assert @parser.parse('"He said, \"Hi!\""')

Hi!

I'm writing my grammar using treetop, for a new programming language (prototype for while), but I cannot found a way to ignore whitespace/comments assertively.

There's an feature in tool, or will be implemented soon?

Try this with Treetop 1.3.0 installed:

ruby -w -e "require 'rubygems';require 'treetop'"

As you can see, the metagrammar.rb file has a couple of methods that are redefined. Here is an extract:

module ModuleDeclaration0
  def space
    elements[1]
  end

  def space
    elements[4]
  end
end

This sort of duplicated methods also exist in the Cucumber parser

I'm not sure what causes this. In any case - I think Treetop should either avoid outputting methods that will be overwritten or issue an error if this is happening because the grammar is ambiguous or inconsistent in some way.

I'm test-driving my grammar, because I'm learning Treetop at the same time as designing my language. As an intermediate step, I try to write a grammar with a rule whose expression I want to label. When I do this, the grammar itself no longer parses correctly. It looks like what I've done somewhere else, so I can't figure out the problem. Finally, I conclude that labeling an expression requires having at least one more expression following it. I don't find that constraint helpful, so why have it? This spec describes the difference. Skip to the end, then read it backwards.

require 'treetop'

describe "Grammar with a simple rule that uses a label" do
  context "Labeled subexpression followed by another expression" do
    let(:subject) { Treetop.load_from_string(
<<GRAMMAR
grammar SimpleRuleWithLabel
  rule word
    letters:[A-Za-z]+ [A-Za-z]*
  end
end
GRAMMAR
    )}

    let (:parser) { subject.new }

    context "matching many letters, the match result" do
      let(:result) { parser.parse("aBcDeF") }

      it { result.should respond_to(:letters) }
      it { result.letters.text_value.should == "aBcDeF" }
    end
  end

  context "Labeled subexpression without another expression" do
    it "does not represent a valid grammar, even though I think it should" do
      lambda {
        Treetop.load_from_string(
<<GRAMMAR
grammar SimpleRuleWithLabel
  rule word
    letters:[A-Za-z]+
  end
end
GRAMMAR
      )}.should raise_error(RuntimeError, /Expected \#/)
    end
  end
end

Some issue makes the rule

rule a
  [a-z] {
    def b
      0
    end
  }
end

throw ArgumentError (Negative Argument) in Treetop.load.
while the same code indendated slightly altered is valid:

rule a
  [a-z] {
  def b
    0
  end
  }
end

This problem occures in rspecs when I require 'rails_helper', but not when I only require 'treetop' on its own. I unfortunately do not have the time for more testing right now, but I will leave you my Gemfile.lock for information.

https://gist.github.com/bonflintstone/b915b5ee52ccac73ebffa292bd229d5e

Hello, I am currently considering updating the treetop version I am using, which is 1.6.3, to the new version of the project. However, I could not find any release note about the most recent version. Is there a place where I can read it ?

Also, if that place does exist, maybe that file should also be placed on the project root directory as well.

There doesn't seem to be any place to get help with TreeTop.

Where can I ask questions about usage?

Thanks!
Josiah

Had to google to find the github page

Is there any way I could make treetop not use polyglot (which overwrites require),
I want to get rid of this hack

Hi,

I have used both Treetop and Parslet. The most feature in Parslet that I miss so much is that I can define shorthand method in derived Parslet::Parser class as:

class Foo < Parslet::Parser
    def spaced(thing)
        if thing.class == String
            atoms = space.maybe >> str(thing)
        elsif is_atom(thing)
            atoms = space.maybe >> thing
        end
        return atoms
    end
    ...
    rule(:comma) {
        breaked(spaced(','))
    }
end

Is there any way in Treetop to achieve that? That will definitely make grammar more readable.

With ruby 1.8.7 (2012-10-12 patchlevel 371) [x86_64-linux] I get the following failures when trying to run the specs.

  1) a character class with a negated POSIX bracket expression matches a character outside the negated class
     Failure/Error: parse('a').should_not be_nil
     RegexpError:
       invalid regular expression; [:^space:] is not a character class: /\G[[:^space:]]/
     # ./lib/treetop/runtime/compiled_parser.rb:97:in `initialize'
     # ./lib/treetop/runtime/compiled_parser.rb:97:in `new'
     # ./lib/treetop/runtime/compiled_parser.rb:97:in `has_terminal?'
     # (eval):19:in `_nt_expression_under_test'
     # ./lib/treetop/runtime/compiled_parser.rb:18:in `send'
     # ./lib/treetop/runtime/compiled_parser.rb:18:in `parse'
     # ./spec/compiler/character_class_spec.rb:91

  2) a character class with a negated POSIX bracket expression doesn't match a character within the negated class
     Failure/Error: parse(' ').should be_nil
     RegexpError:
       invalid regular expression; [:^space:] is not a character class: /\G[[:^space:]]/
     # ./lib/treetop/runtime/compiled_parser.rb:97:in `initialize'
     # ./lib/treetop/runtime/compiled_parser.rb:97:in `new'
     # ./lib/treetop/runtime/compiled_parser.rb:97:in `has_terminal?'
     # (eval):19:in `_nt_expression_under_test'
     # ./lib/treetop/runtime/compiled_parser.rb:18:in `send'
     # ./lib/treetop/runtime/compiled_parser.rb:18:in `parse'
     # ./spec/compiler/character_class_spec.rb:94

From http://treetop.rubyforge.org/syntactic_recognition.html,

"foo" &"bar" matches "foobar" but only consumes up to the end "foo". It will not match "foobaz"

However, given simple.treetop with contents:

grammar Simple
  rule positive_lookahead
    "foo" &"bar"
  end
end

I get the follwing results:

$ tt simple.treetop 
$ ruby -rubygems -e 'require "treetop"; require "simple"; puts SimpleParser.new.parse("foobar").inspect'
nil

Now, either I am crazy, or the simplest possible example in the documentation on positive lookahead doesn't work?