How best to work with Mistletoe? about myst-parser HOT 12 CLOSED

yep for sure - I think that either way though we've gotta start by making a (probably small) PR...if we decide to escalate one way or another, then we can say that we've gone through the typical ways of engaging first

from myst-parser.

Issue opened miyuchina/mistletoe#92

from myst-parser.

Shall we open up a PR in mistletoe? We haven't heard anything back from the author since our original post...perhaps a minor improvement, or the bugfix mentioned in #102 ? Just wanna keep the heartbeat going there...

from myst-parser.

Sounds good - it does increasingly look like the author isn't going to be very responsive. At some point, if we decide there's nothing we can do about mistletoe maintenance etc, we should rename this parser to something else and release it as a separate project (e.g. something other than just mistletoe-ebp) in the hopes we can get more community buy-in on it.

from myst-parser.

I think the first step would be to try making a few small-ish PRs and see how responsive they are. We can slowly try to close the gap between this fork and mistletoe master, but if we make a giant PR all at once it'll probably be much less-likely they'll engage.

In an ideal world, we'd get the keys to mistletoe but it may be some time before that happens depending on what kinda maintainer they are. In an extreme case, if we decide it's not worth engaging w/ a fork, then we can hard-fork it and rename the project to something else.

WDYT?

from myst-parser.

I imagine making a PR will just hit a brick wall, like the other pending PRs. But can give it a go.

Yeh it depends on the reason why he is currently not engaging. If its just that he's super busy with other things, it might be good to send him a nicely worded email at some point, and explain our position and that we would (maybe) be happy to pick up some of the slack on maintenance.

from myst-parser.

Another note - Mistletoe was announced on Reddit 2 years ago here:

https://www.reddit.com/r/Python/comments/6sjz3x/mistletoe_a_fast_extensible_markdown_parser_in/

from myst-parser.

Shall we open up a PR in mistletoe?

See miyuchina/mistletoe#94

from myst-parser.

nice - thanks

from myst-parser.

FYI, here is a working list of 'up-streamable' changes that would be desired:

• Move testing from unittest to pytest: pytest is now the de facto testing architecture and vastly improves the usability/flexibility of testing. (already introduced in fork)
• Introduce pre-commit code linting and formatting: This standardizes the code style across the package, and ensures that new commits and Pull Requests also conform to it. (already introduced in fork)
• Introduce ReadTheDocs documentation
• Add a conda-forge distribution of the package
• Improve the AST API and documentation: I view panflute's implementation of the pandoc API in python, as the gold standard for how a pythonic AST API should be written and documented. Some tweaks to the current token class objects, and creating auto-generated RTD documentation, could achieve this.
• Storage of source line/column ranges: LSP and good rendering reporting of warnings/errors, requires access to the source line and column ranges of each parsed token.
• Asynchronous parsing: LSP requires documents to be parsed asynchronously. Currently, mistletoe contains a number of global state objects, which make parsing inherently not thread-safe. The simple solution to this is to store these items as threading.local objects. A related but slightly more complete solution is to introduce the idea of a 'scoped session', similar to that used by sqlalchemy for database access: Contextual/Thread-local Sessions
• Improve extensibility of block tokens: A Markdown parser is inherently a Finite State-Machine + Memory (a.k.a Push-down Automata (PDA)), with parsing tokens as states (for a good example of a python state-machine see pytransitions). The problem with extensibility, is that inherently states are interdependent; when introducing a new state/token you must provide logic to all the other tokens, w.r.t to when to transition to this new token. Currently, MyST Parser sub-classes nearly all the Mistletoe block tokens to implement the extensions it requires, but it would be ideal if there was a more systematic approach for this.
• Improve extensibility for span tokens: Mistletoe does allow for span token extensions to be added, at least in a simple way. However, as with block tokens above, there is often an interconnectivity to them, especially when considering nested span tokens. As of 7cc2c92, MyST-Parser now overrides some of Mistetoe's core logic to achieve correct parsing of Math tokens, but if possible this should be made more general.
• Improve rendering logic: Currently, there is no concept of recursive walk-throughs or 'visitor' patterns in the Misteltoe BaseRenderer, which is a better method for rendering tree like structures (as used by docutils/panflute). Also, the current token instantiating (within context managers) needs improvement (see miyuchina/mistletoe#56).

from myst-parser.

Nice - this is really helpful.

I think the other question we should all consider is: what happens if it just doesn't work out? I am curious what @chrisjsewell thinks about the maintainability and sustainability of caring for Mistletoe (or in a worst-case scenario, needing to care for a differently-named fork of it). After having worked with the codebase thus far, what are your thoughts on that?

from myst-parser.

As you've noticed, I got impatient lol. mistletoe-ebp now has CD on pypi and soon on conda-forge. Then I can propagate this to MyST-Parser.

I still think mistletoe is the right tool for the job, in terms of its base API, it just needs the improvements above. I don't think maintainability/sustainability should be too much of an issue, given that a robust test suite is in place, since the underpinning Markdown syntax isn't exactly going to be changing any time soon (if ever from what we've seen from the decade long forum conversations lol).

from myst-parser.