async/await support in generate() about parsy HOT 6 CLOSED

I realized the as_tuple() is already provided with parsy.seq, but beyond the example the subject of await is still relevant

from parsy.

Are you able to provide a good motivating example, if seq covers your original use case?

from parsy.

As motivated in my initial post, yield syntax has issues in various places. I cannot list them by heart but that is even acknowledged in the PEP 492 introducing await syntax:

Support for asynchronous calls is limited to expressions where yield is allowed syntactically, limiting the usefulness of syntactic features, such as with and for statements.

https://peps.python.org/pep-0492/

Another example I remember: this is not valid python: f(yield x) but this works f(await x).

Given that both the yield syntax and await syntax can be supported with the same backing code with a thin layer of glue for yield and that they are almost identical for the user, it seemed like a good idea to be able to used the syntax that allows the most flexibility for the user.

from parsy.

Without a motivating example, it's difficult to evaluate the need for this.

In particular, I don't want to add anything that will unnecessarily make the code or the documentation more complex. One way you can do this is adding "more than one way to do it" without a good justification for why you would want both methods.

It sounds like that if you want to use the await syntax, you'll also have to add a decorator onto the parser function, which is a significant downside in terms of simplicity.

from parsy.

I created a pull request #67 with a an updated example showing:

• Both await and yield can be used. This change is fully backward compatible.
• The new internal bit inserts a single convertion:

        if inspect.iscoroutine(iterator):
            iterator = _coro_to_gen(iterator)

The rest of the code is entirely unaffected as the coroutine is basically a generator yielding "futures" (parsers in our case).

• The user-exposed API does not gain anything new apart from allowing coroutine functions in generate(). In particular, absolutely no new decorator needs to be applied to user function.
• It is not more complicated to work with async/await compare to yield. Apart from more/less parenthesis, it is exactly the same. Except the Python grammar of "await" is a fixed version of the one of "yield". There is basically no win in using "yield" compared to "await" these days for this sort of syntactic sugar, unless the code itself can be async in the more typical way (e.g. asyncio). Even then, interop between multiple "kinds" of async is 100% possible.

In terms of how "await" is better designed than "yield":

• https://bugs.python.org/issue10544. No such issue exists for "await" AFAIK, this just works with "await" instead of being dismissed as wontfix.
• As can be seen in the Python grammar, "yield" has been added sparingly in a bunch of rules. "await" OTH is deep rooted as the only consumer of "primary", which AFAICT means it is usable basically anywhere any kind of expression is expected, without any arbitrary limitation.
  https://docs.python.org/3/reference/grammar.html

from parsy.

Thanks @douglas-raillard-arm for the new PR and explanation, and all the work you put into this. I've thought about it some more.

I think about things like this from the point of view of someone reading code that uses Parsy. For such a person, what are they going to think when they come across the following?

@generate
async def foo():
   x = await parsy.string(...)  # etc

For many Python users, it may be the first time they've seen the async and await keywords. In this situation, it's pretty confusing, because the task at hand has got nothing to do with asynchronous execution as typically described as the whole point of async. They will also naturally ask "how is this different from using yield with @generate?", to which the answer is "it's not". The next question is "so why do we have await here instead of yield?" and there probably won't be a good answer to that question. So far it's been very difficult to come up with any examples at all where you would prefer await to yield - the syntactical advantages are real, they just very rarely come up in real world Parsy usage AFAICS.

The user will also have to understand that despite appearances, the library is doing something clever under the hood to convert async function to a normal generator.

So, from the point of view of comprehension, we've introduced a bunch more things that the user needs to understand to be able to maintain this code, including 2 new keywords and more knowledge of Parsy internals. "There should be one-- and preferably only one --obvious way to do it" is the principle I'm thinking of here. Obviously there are many times when we violate that, but only when there are compelling advantages. In this case, the potential for making life harder for the reader seems to far outstrip potential benefits for the writer of the code.

Thanks again!

from parsy.