t3rn0 / ast-comments Goto Github PK

code = "a=1;b=1;print(a+b)"

no error ==> tree = ast.parse(code)

error from ==> tree = ast_comments.parse(code)
File "/home/.../ast_comments.py", line 39, in _enrich
nodes = sorted([(x.lineno, x) for x in ast.walk(tree) if isinstance(x, ast.stmt)])
TypeError: '<' not supported between instances of 'Assign' and 'Assign'

Original builtin ast.unparse collapses nested "ifs" to their equivalent "elifs".
ast_comments.unparse breaks this logic:

# example.py
source = """
if a: ...  
# elif
elif b: ... 
else: ...
"""
print(astcom.unparse(astcom.parse(source)))
print("*"*10)
print(ast.unparse(ast.parse(source)))

$ python example.py
if a:
    ...
else:
    # elif
    if b:
        ...
    else:
        ...
**********
if a:
    ...
elif b:
    ...
else:
    ...

"parse-unparse" produces valid code but it's badly formatted.

This behavior was previously mentioned here

Not an issue nor feature request. Just asking if there's an easy way to convert the tree back to code without using other libraries for parsing.
I tried:
ast_comments.unparse(tree)
ast.unparse(tree)
astor.to_source(tree)
All of them work but comments are missing in the final code.

Hi, the comments usually work great, but when there is comment directly above docstring, the docstring is broken, illustrated by this tests:

import ast_comments as ast
import pytest

comment_below = '''
"""
Multiline string
"""
# comment
'''.strip()


def test_this_one_is_fine():
    assert comment_below == ast.unparse(ast.parse(comment_below))


comment_above = '''
# comment
"""
Multiline string
"""
'''.strip()


@pytest.mark.xfail(reason="This one is failing")
def test_this_one_is_failing():
    assert comment_above == ast.unparse(ast.parse(comment_above))

Currently we don't identify inlined comments in code, all comments are just regular (not inlined) (https://github.com/t3rn0/ast-comments#notes).
Difference between them is lost after the code is parsed into the tree.
We should try distinguish inlined and regular comments when unparsing the tree back to the code.

It'd be great to cover following cases:

• simple expression:

  a = 1  # c1
  

• if-elif-else

  if a:  # c2
      ...
  elif b:  # c3
      ...
  else:  # c4
      ...
  

• try-except-else-finally
• for-else
• match

Hello,

I am experimenting with this module to auto-generate sync modules from async modules, see psycopg/psycopg@16c2276.

After a roundtrip from code to ast and back to code, the vertical blank line are lost.

Do you think it's possible to improve ast-comments to reproduce the same amount of blank lines in the code emitted?

Thank you very much!

Let's say we have code

>>> source = """a = 1; b = 2  # some comment"""
>>> ast.dump(ast.parse(source))
"Module(body=[Assign(targets=[Name(id='a', ctx=Store())], value=Constant(value=1, kind=None), type_comment=None), Assign(targets=[Name(id='b', ctx=Store())], value=Constant(value=2, kind=None), type_comment=None)], type_ignores=[])"
>>>
>>> astcom.dump(astcom.parse(source))
>>> "Module(body=[Assign(targets=[Name(id='a', ctx=Store())], value=Constant(value=1, kind=None), type_comment=None, comments=('some comment',)), Assign(targets=[Name(id='b', ctx=Store())], value=Constant(value=2, kind=None), type_comment=None, comments=())], type_ignores=[])"

Parsed tree has two nodes. Which node do we want to place the comment on? To "a"-part? To "b"-part? To both?

With the current version (0.1.2) the comment goes to "a"-part but I don't think it's the right choice

Hello,

This line:

Causes typing objects to eclypse the ast objects imported in from ast import *. As a consequence checking for objects to be Dict, List, Tuple fails to perform the expected check.

>>> import ast
>>> import ast_comments
>>> ast.Str is ast_comments.Str
True
>>> ast.Dict is ast_comments.Dict
False
>>> ast_comments.Str
<class 'ast.Str'>
>>> ast_comments.Dict
typing.Dict

If the list (tuple, dict, call, ...) is written across multiple lines there can be comments to certain elements of the list:

source = [
    # c1
    a,
    b,  # c2
]

If we parse-unpase it the comments will go out of the list. This may result in the loss of some potentially important info:

# c1
# c2
uparsed = [a, b]

Say here is the source code I want to do some transformation like replace codes between #### Your solution #### and "#### End of solution ####" with pass. Compared with the source code, I got the wrong indention structure.

foo = """
def foo():
    #### Your solution ####
    c = a + b
    #### End of solution ####
    if a:
        #### Your solution ####
        a += 1
        #### End of solution ####
        #### Your solution ####
        b += 1
        #### End of solution ####
"""
print(ast.unparse(ast.parse(foo)))

bar = """
def bar():
    #### Your solution ####
    c = a + b
    #### End of solution ####
    if a:
        pass
        #### Your solution ####
        #### End of solution ####
    """
print(ast.unparse(ast.parse(bar)))

Expected Behavior

The code generated will have the identical indentation format as the original source code.

Current Behavior

def foo():
    #### Your solution ####
    c = a + b
    #### End of solution ####
    if a:
        #### Your solution ####
        a += 1
        #### End of solution ####
        #### Your solution ####
        b += 1
#### End of solution ####

def bar():
    #### Your solution ####
    c = a + b
    #### End of solution ####
    if a:
        pass
#### Your solution ####
#### End of solution ####

Thanks for the project! Love it and really impressed with the clean code. I hope they pull this into main line functionality soon!

Unsurprisingly, you can't run compile() on the output of your extended parser. Specifically, compile will fail with a message like this:

>>> t = ast_comments.parse("def hi():\n\t# yup\n\t#    more comment\n\tprint('hi')\n")
>>> compile(t, "", "exec")
Traceback (most recent call last):
  File "/home/bill/.local/share/JetBrains/Toolbox/apps/pycharm-professional/plugins/python/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
           ^^^^^^
  File "<input>", line 1, in <module>
TypeError: expected some sort of stmt, but got <ast_comments.Comment object at 0x7fabe6efb940>

My app would like to be able to preserve comments, walk and modify the tree, and then compile my brand new code. The cleanest way to do this would be to have a modified version of compile() or some helper function in ast-comments which provides the necessary fixes.

I'm thinking for a fix: use the node update visitor to walk the tree and drop every node of type Comment.

Would you be open to having this fix in the ast-comments package? Do you have a solution already in mind? Would you like me to add one with a PR? if you'd like me to do this, please specify impl details if you are 'picky' :) and I'll try to get something up in the next day or two

Maybe this is a niche feature. I can just put it into my own app, if you don't want it in ast-comments.

I think it would be great to have an addition to the parse function or a pre-packaged helper that walks the tree and merges consecutive Comment nodes in cases where they fit some rule.

In my case (and I think this is not uncommon), I use this convention for multiline comments:

def my_func():
   """
   Standard function documentation .... not part of my example
   """
   function_doing_some_stuff = 1 + 2
   # TODO: Come back to this later this is really not a great
   #     solution. I think there's a better one here: https://.....
   my_dumb_solution()

In the above example, it is a multiline comment that is a long TODO item. The above convention happens to be the one followed in PyCharm, ie it will identify the above as a multiline TODO and keep track of it for you in its list of todos.

I'm proposing you add some merger predicate option which allows people to provide their own convention for merging one comment with another. Probably something like: Callable[[str, str], bool] meaning, we'll give you the body of the current comment and the body of the one that follows, you return True if these two should be considered part of the same comment. ie the second will get absorbed into the node above it.

As with my other issue ticket, I'm happy to impl this in a PR if you like.

Thanks!

Hello,

In 1.1.1 there is a regression w.r.t. 1.1.0 and an inline comment gets misplaced.

The file in question is this. In 1.1.1, the noqa comment on line 5 ends up associated to the if true at line 444. Using this script for processing:

# test-ast.py

import sys
import ast_comments as ast

tree = ast.parse(sys.stdin.read())
print(ast.unparse(tree))

and processing the file with the two versions:

$ wget -q https://raw.githubusercontent.com/psycopg/psycopg/d0bc9241881161d8aaf22597e3ad007f93d9c516/tests/pool/test_pool_null_async.py

$ pip install ast-comments==1.1.0
...
Successfully installed ast-comments-1.1.0

$ python test-ast.py < test_pool_null_async.py > ast-comments-1.1.0.py

$ pip install ast-comments==1.1.1
...
Successfully installed ast-comments-1.1.1

$ python test-ast.py < test_pool_null_async.py > ast-comments-1.1.1.py

$ diff -u ast-comments-1.1.0.py ast-comments-1.1.1.py

The difference between the files is:

--- ast-comments-1.1.0.py	2023-10-02 11:55:19.532576524 +0200
+++ ast-comments-1.1.1.py	2023-10-02 11:56:22.216400976 +0200
@@ -1,7 +1,7 @@
 import logging
 from typing import Any, Dict, List
 import pytest
-from packaging.version import parse as ver  # noqa: F401  # used in skipif
+from packaging.version import parse as ver
 import psycopg
 from psycopg.pq import TransactionStatus
 from psycopg.rows import class_row, Row, TupleRow
@@ -360,7 +360,8 @@
         assert stats.get('connections_errors', 0) == 0
         assert stats.get('connections_lost', 0) == 0
         assert 200 <= stats['connections_ms'] < 300
-if True:  # ASYNC
+if True:  # noqa: F401  # used in skipif
+    # ASYNC
 
     @pytest.mark.skipif('sys.version_info < (3, 8)', reason='asyncio bug')
     async def test_cancellation_in_queue(dsn):

Hey, just wanted to thank you for this very useful/interesting piece of software 🙂
It could be really useful for one of my projects, if I ever decide to support comments!
I'm surprised by how short the code is 😄 Well done!