Seeking Guidance on Implementing Parser Autocomplete about parsy HOT 6 CLOSED

There are probably other ways you can do this in Python with introspection. In particular, with sys._getframe or by examining the traceback from a ParseError exception, you can extract all the information you need about how something failed, including local variables of functions and closures at any point in the stack.

I find the easiest way to explore this kind of things is with a REPL. Python's frame objects do have some documentation, but I find them much easier to explore interactively.

The difficulties of this are you effectively have to "pattern match" on what you expect to find e.g. you know that for the seq combinator, if you go back two frames and get the variable x you can get the info you need. It's highly tied to current implementation and fragile. However, it might be what you need!

from parsy.

parsy was not created with your use case in mind, and I cannot see an easy way to get this info out of Parsy objects. In general I suspect it isn't possible, because a Parsy parser can do anything it wants in the body of the parsing function.

I'm not actually an expert in parsing (I'm just the maintainer of this library which was created by someone else), but I think you may be able to get where you want from a Context Free Grammar definition - see this Stack Overflow answer for example: https://stackoverflow.com/questions/603687/how-do-i-generate-sentences-from-a-formal-grammar

from parsy.

@spookylukey Sure - this becomes possible by maintaining the grammar in multiple formats—something I would like to avoid if possible. Maybe people have done something similar with parsy already. But I fear you are right, and it is not the right tool for this job. Thanks for your answer!

from parsy.

This is a cool idea, I don't know whether it would work all the way to a full implementation, but here's a possible way to start: you can use the fact that when a parser fails, it raises a ParseError which contains information about which specific set of parsers were expected to match at the point of failure, in ParseError.expected:

from typing import List, Tuple

from parsy import ParseError, regex, seq, string

padding = regex(r"\s*")
# Use `desc` to define the string which appears in `ParseError.expected` when parsing fails
var_name = regex(r"\w+").desc("*variable name*")
parser = seq(
    string("is("),
    var_name,
    string(")") << padding,
    (string("and") | string("or")) << padding,
    var_name << padding,
    (string("==") | string("in") | string("=~") | string("!=")) << padding,
    var_name << padding,
)


def suggest(text: str) -> Tuple[int, str, List[str]]:
    """Return (index of last match, remaining text which didn't match, expected parsers)"""
    try:
        parser.parse(text)
    except ParseError as exception:
        remainder = text[exception.index :]
        return (exception.index, remainder, list(exception.expected))

    # Parser consumed the whole input - return the end index with no suggestions
    return (len(text), "", [])


# Examples

print(suggest("is(acco"))  # (7, '', [')'])
print(suggest("is(account)"))  # (11, '', ['and', 'or'])
print(suggest("is(account) and"))  # (15, '', ['*variable name*'])
print(suggest("is(account) and name"))  # (20, '', ['in', '==', '!=', '=~'])
print(suggest("is(account) and name == other"))  # (29, '', [])

# Can tell when there's text in the input which didn't match any parser - "bla"
print(suggest("is(account) bla"))  # (12, 'bla', ['and', 'or'])

It'd take more effort to give a good autocomplete suggestion for something like *variable name* above, where the possible strings aren't known to the parser/grammar if they're dynamic like variable names.

from parsy.

Re question 2:

Here's a possible way to keep track of contextual info, building on the example above - e.g. for a filter statement like <some_var> == "<literal value>", the parser for <literal value> can know the context it's in (e.g. the value which matched <some_var>):

from typing import List, Tuple

from parsy import ParseError, Parser, regex, string


def suggest(parser: Parser, text: str) -> Tuple[int, str, List[str]]:
    """Return (index of last match, remaining text which didn't match, expected parsers)"""
    try:
        parser.parse(text)
    except ParseError as exception:
        remainder = text[exception.index :]
        return (exception.index, remainder, list(exception.expected))

    # Parser consumed the whole input - return the end index with no suggestions
    return (len(text), "", [])


var_name = regex(r"\w+").desc("*variable name*")
literal_text = regex(r'"(\w*)"', group=1).desc("*variable name*")


def literal_for_variable(variable_name: str) -> Parser:
    return literal_text.desc(f"Value for `{variable_name}`")


filter_parser = (var_name << string(" == ")).bind(literal_for_variable)

# Examples

print(suggest(filter_parser, "my_var == "))  # (10, '', ['Value for `my_var`'])
print(suggest(filter_parser, "something == "))  # (13, '', ['Value for `something`'])

I'm not a big fan of having to store all the suggestions in the desc string, but the type of Result.expected is FrozenSet[str] so strings are all that's available with this approach. If it was possible to store that info in a dataclass it would be easier to use in any downstream logic - maybe there's a way to make that simpler but I can't see it

from parsy.

@robjhornby Thanks for your ideas! I wasn't aware of desc - need to think about your proposal.

from parsy.