Obscure behavior when text is longer then grammar can consume about parglare HOT 7 CLOSED

Yes, it is intended behaviour although not yet documented as it seems.

parglare will not assume by default that the whole input should be consumed. If you want to do that use EOF rule at the end:

g = Grammar.from_string('S: T EOF; T: NL* L+ NL* | NL*; L: /.+/; NL: "\n";', re_flags=re.MULTILINE)

I'll close this issue when the docs get fixed.

from parglare.

Documented here

from parglare.

By the way, your solution above results into irritating None at the end of the ast with one more nesting level, i.e.:

g = Grammar.from_string('S: T EOF; T: NL* L+[NL] NL* | NL*; terminals L: /.+/; NL: "\n";', re_flags=re.MULTILINE)
print(Parser(g, ws=False).parse('\nL1\nL2\n\n'))
# [[['\n'], ['L1', 'L2'], ['\n', '\n']], None]

g = Grammar.from_string('S: L* EOF; terminals L: /.*/;')
print(Parser(g).parse('\nL1\nL2\n\n'))
# [['L1', 'L2'], None]

Is it intended? How to get rid of that None? Or has it some useful purpose?

from parglare.

Each rule, when reduced, will either call registered action or the default action. The default action for non-terminals is to create a list of elements where each element correspond to RHS element. I decided not to introduce any implicit removal, for the user to be sure how many elements will be in the list, so EOF also produces a result which is None. If you are using the default action, like in the example above, be sure to unpack your result:

result, _ = Parser(g).parse('...')

or

result = Parser(g).parse('...')[:-1]

If you prefer to get a clean result from the parse call than register your actions, at least for the root rule.
There is a built-in action pass_single which will work nice in your example:

g = Grammar.from_string('@pass_single S: L* EOF; terminals L: /.*/;')
print(Parser(g).parse('\nL1\nL2\n\n'))
# ['L1', 'L2']

from parglare.

As a side note, I think that a special built-in actions strip_first and strip_last could be added as that feels to be a frequent requirement. In that case strip_last would be a more general approach as it would work in the case when you have more than one rule before EOF. I'm registering issue #63 for tracking this.

from parglare.

By the way, I found only now, but the very first example which started this whole issue works as I had expected at the very beginning, i.e. parses till the very end. Isn't it regression comparing to what you described earlier and in the current docs?
http://www.igordejanovic.net/parglare/stable/grammar_language/#special-built-in-rules
I expected that same snippet will continue to work same way, but now it produces error as I wanted...

g = Grammar.from_string('T: NL* L+ NL* | NL*; L: /.+/; NL: "\n";', re_flags=re.MULTILINE)
print(Parser(g, ws=False).parse('\nL1\nL2\n\n'))
# ParseError: Error at 3:0:"\nL1\n*L2\n\n" => Expected: STOP but found <L(L2)>

I re-tried that example to be sure that using @pass_single won't discard EOF directive at the end and it will continue to produce errors instead of incomplete parsing. So, as I couldn't get partial parsing anymore, I hope you reassure me that @pass_single won't affect enforced parsing till the end and only affect resulting AST.

from parglare.

Seems that your are right. It looks like a regression. The old behavior should be the correct one. Will investigate.

Yes, actions are not affecting parsing in any way, they are affecting just the resulting output. In the future there might be a support for side-effects and interplay between actions, recognizers and dynamic disambiguation which will bring a possibility to affect the parsing course from different parts of user written code.

from parglare.