google / pasta Goto Github PK

View Code? Open in Web Editor NEW

334.0 17.0 39.0 514 KB

Library to refactor python code through AST manipulation.

License: Apache License 2.0

Python 100.00%

pasta's Introduction

pasta: Python AST Augmentation

This is still a work-in-progress; there is much more to do. Existing functionality may not be perfect.

Mission

Enable python source code refactoring through AST modifications.

Sample use cases:

Facilitate moving or renaming python modules by rewriting import statements.
Refactor code to enforce a certain style, such as reordering function definitions.
Safely migrate code from one API to another.

Design Goals

Symmetry: Given any input source, it should hold that pasta.dump(pasta.parse(src)) == src.
Mutability: Any changes made in the AST are reflected in the code generated from it.
Standardization: The syntax tree parsed by pasta will not introduce new nodes or structure that the user must learn.

Python Version Support

Supports python 2.7 and up to 3.8.

Dependencies

pasta depends on six.

Basic Usage

import pasta
tree = pasta.parse(source_code)

# ... Augment contents of tree ...

source_code = pasta.dump(tree)

Built-in Augmentations

Pasta includes some common augmentations out-of-the-box. These can be used as building blocks for more complex refactoring actions.

There will be more of these basic augmentations added over time. Stay tuned!

Rename an imported name

Rewrites references to an imported name, module or package. For some more examples, see pasta/augment/rename_test.py.

# Rewrite references from one module to another
rename.rename_external(tree, 'pkg.subpkg.module', 'pkg.other_module')

# Rewrite references from one package to another
rename.rename_external(tree, 'pkg.subpkg', 'pkg.other_pkg')

# Rewrite references to an imported name in another module
rename.rename_external(tree, 'pkg.module.Query', 'pkg.module.ExecuteQuery')

Known issues and limitations

Changing the indentation level of a block of code is not supported. This is not an issue for renames, but would cause problems for refactors like extracting a method.
pasta works under the assumption that the python version that the code is written for and the version used to run pasta are the same. This is because pasta relies on ast.parse
Some python features are not fully supported, including global.

Developing

This project uses setuptools to facilitate testing and packaging.

# Run all tests
python setup.py test

# Run a single test suite
python setup.py test -s pasta.base.annotate_test.suite

Disclaimer

This is not an official Google product.

pasta's People

Stargazers

Watchers

pasta's Issues

Python 3.10 support

Golden data is missing for Python 3.10, which would be lovely to have.

Auto-detect indentation in a file and reuse on unformatted code

Follow-on for #62.

If we insert a node without formatting data into a tree with formatting data, the new node will be printed with default format, including 2 spaces for indentation. If the rest of the file uses tabs or four spaces or whatever, the new code should match this.

The suggestion here is to find the most prevalent indentation style in the file and apply it as the default when indenting unformatted nodes.

Support PEP 526 Variable Annotations

https://www.python.org/dev/peps/pep-0526/

Failed to recognize trailing comma on import statement with parenthese

Hi,

Trailing comma is not allowed in import, but when inside parentheses it's valid syntax
I know that this one can sound too specific, but still a valid syntax =)

file:

from foo.a import (do,)

do()

Script:

import pasta
from pasta.augment import rename

path = r'test.py'


with open(path) as file:
    tree = pasta.parse(file.read())

Full stack trace

Traceback (most recent call last):
  File "script.py", line 8, in <module>
    tree = pasta.parse(file.read())
  File "/opt/pasta/pasta/__init__.py", line 25, in parse
    annotator.visit(t)
  File "/opt/pasta/pasta/base/annotate.py", line 1055, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py27/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 168, in visit_Module
    self.generic_visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py27/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/opt/pasta/pasta/base/annotate.py", line 1055, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py27/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 520, in visit_Expr
    self.visit(node.value)
  File "/opt/pasta/pasta/base/annotate.py", line 1055, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py27/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 637, in visit_Call
    self.visit(node.func)
  File "/opt/pasta/pasta/base/annotate.py", line 1055, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py27/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 754, in visit_Name
    self.token(node.id)
  File "/opt/pasta/pasta/base/annotate.py", line 1118, in token
    token_val, token.src, token.start[0], token.line))
pasta.base.annotate.AnnotationError: Expected 'do' but found ','
line 1: from foo.a import (do,)

cc @nicoddemus

Missing support for relative imports over direct ancestor

✔ Works:

import pasta
tree = pasta.parse('from .uncle.package import foo')

✔ Works:

import pasta
tree = pasta.parse('from ..granduncle.package import foo')

❌ Fails:

import pasta
tree = pasta.parse('from ...greatgranduncle.package import foo')

with error AnnotationError: Expected '.' but found '...'

Support for with statement with multiple arguments

handle situation like:

with (a() as a0, b() as b0):
    ...

Lacking test covered in codegen_test.py

#74 should have been caught in tests; need to test codegen for JoinedStr. Based on the test inputs in testdata/codegen, there are likely other nodes that need coverage as well.

get_unused_import_aliases struggling with imports not at the top

I get an error when there's an unused import alias but its inside the code:

In [4]: import pasta

In [5]: src = """
   ...: def fun():
   ...:   import a_thing
   ...:   return 1
   ...: 
   ...: fun()
   ...: """

In [6]: tree = pasta.parse(src)

In [7]: from pasta.augment import import_utils

In [8]: import_utils.get_unused_import_aliases(tree)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-12-1eb7af7a6dac> in <module>()
----> 1 import_utils.get_unused_import_aliases(tree)

pasta/augment/import_utils.py in get_unused_import_aliases(tree, sc)
     76     if isinstance(node, ast.alias):
     77       name = sc.names[
---> 78           node.asname if node.asname is not None else node.name]
     79       if not name.reads:
     80         unused_aliases.add(node)

KeyError: 'a_thing'

Bad indentation when using try..finally

Hi! It's me again, this time with more pasta edge cases!

When I run the following Python 2 code:

import ast
import pasta

source_code = '''
try:
    if 1:
        a=1
finally:
    a=1
'''

class SampleTransformer(ast.NodeTransformer):
    def visit_If(self, node):
        node.body = [ast.Pass()]
        return node

tree = pasta.parse(source_code)
SampleTransformer().visit(tree)

print pasta.dump(tree)

The expected output of it should be

try:
    if 1:
        pass
finally:
    a=1

But the output is this instead:

try:
    if 1:
    pass
finally:
    a=1

This produces a SyntaxError in the rewritten code.

If I change the code of the visit_TryFinally function in pasta/base/annotate.py to use self.indented(node, 'body') instead of node.body, the previous example works ok. However, when I run the test suite, there are six failing tests so I suppose there has to be a better way to fix this.

Docstring with \n causes AnnotationError

repro.py

class C:
  def f1(self):
    """Doc.

    String\nwith\nnewlines.
    """
    pass

  def f2(self):
    pass

Python 2.7.13 (default, Nov 24 2017, 17:33:09) 
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pasta
>>> with open('repro.py', 'r') as f:
...   src = f.read()
... 
>>> t = pasta.parse(src)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pasta/__init__.py", line 25, in parse
    annotator.visit(t)
  File "pasta/base/annotate.py", line 1113, in visit
    super(AstAnnotator, self).visit(node)
  File "pasta/base/annotate.py", line 126, in visit
    super(BaseVisitor, self).visit(node)
  File "/usr/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "pasta/base/annotate.py", line 46, in wrapped
    f(self, node, *args, **kwargs)
  File "pasta/base/annotate.py", line 210, in visit_Module
    self.generic_visit(node)
  File "/usr/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "pasta/base/annotate.py", line 1113, in visit
    super(AstAnnotator, self).visit(node)
  File "pasta/base/annotate.py", line 126, in visit
    super(BaseVisitor, self).visit(node)
  File "/usr/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "pasta/base/annotate.py", line 89, in wrapped
    f(self, node, *args, **kwargs)
  File "pasta/base/annotate.py", line 371, in visit_ClassDef
    self.visit(stmt)
  File "pasta/base/annotate.py", line 1113, in visit
    super(AstAnnotator, self).visit(node)
  File "pasta/base/annotate.py", line 126, in visit
    super(BaseVisitor, self).visit(node)
  File "/usr/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "pasta/base/annotate.py", line 89, in wrapped
    f(self, node, *args, **kwargs)
  File "pasta/base/annotate.py", line 398, in visit_FunctionDef
    for stmt in self.indented(node, 'body'):
  File "pasta/base/annotate.py", line 1143, in indented
    'more than the outer indentation.' % lineno)
pasta.base.annotate.AnnotationError: Indent detection failed (line 1); inner indentation level is not more than the outer indentation.

Undefined name 'lineno' in ./pasta/base/annotate.py

flake8 testing of https://github.com/google/pasta on Python 3.7.1

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./pasta/base/annotate.py:1143:48: F821 undefined name 'lineno'
          'more than the outer indentation.' % lineno)
                                               ^
1     F821 undefined name 'lineno'
1

rename_external breaking with ValueError

Hi,

Running the rename_external on the following file test.py causes a ValueError: <_ast.Name object at 0x0000000005322F98> is not in list

File: test.py

from a.b import c

def ABC(oi=c):
    pass

Log:

(test_pasta) λ python s2.py
Traceback (most recent call last):
  File "s2.py", line 13, in <module>
    rename.rename_external(tree, 'a.b.c', 'a.b.d')
  File "w:\william\repo_pasta\pasta\augment\rename.py", line 92, in rename_external
    _rename_reads(sc, t, rename_old, rename_new)
  File "w:\william\repo_pasta\pasta\augment\rename.py", line 153, in _rename_reads
    ast.parse(new_name).body[0].value)
  File "w:\william\repo_pasta\pasta\base\ast_utils.py", line 240, in replace_child
    field_val[field_val.index(node)] = replace_with
ValueError: <_ast.Name object at 0x0000000005252F98> is not in list

Script:

with open('test.py', mode='r') as file:
    tree = pasta.parse(file.read())
    rename.rename_external(tree, 'a.b.c', 'a.b.d')

cc @nicoddemus

Support PEP 498 Literal String Interpolation (3.6+)

https://www.python.org/dev/peps/pep-0498/

There may be names referenced in fstrings that have to change as part of a refactoring. Pasta should be able to help with this, at least detecting where they are used.

Failed to recognize comma on exec function on python2.7

Hi,

The parser does not recognize the comma on exec function with python2.7

I did try to create a PR solving this issue, I was doing a function " in_or_comma() " but I realized that the problem would be deeper than this because the visit_exec is treating the exec as a statement instead of function. Am I right?

Here it's a small reproducible example of the error:

File:
exec('x = 10' , {} )

Script:

import pasta
from pasta.augment import rename

path = r'test.py'

with open(path) as file:
        tree = pasta.parse(file.read())

Error:

pasta.base.annotate.AnnotationError: Expected 'in' but found ','
line 1: exec ('x = 10', {})

cc @nicoddemus

Whitespace different for methods with tuples at end

In [25]: code = """\                                                                                                                                                                                                
class Foo():
  def a(self):
    return 1

  def b(self):
    return 2

  def c(self):
    return 3
"""

In [26]: tree = pasta.parse(code)                                                                                                                                                                                   

In [27]: tree.body[0].body.pop(1).name                                                                                                                                                                              
Out[27]: 'b'

In [28]: print pasta.dump(tree)                                                                                                                                                                                     
class Foo():
  def a(self):
    return 1

  def c(self):
    return 3


In [29]: code = """\                                                                                                                                                                                                
class Foo():
  def a(self):
    return 1

  def b(self):
    return 2,'a'

  def c(self):
    return 3
"""

In [30]: tree = pasta.parse(code)

In [31]: tree.body[0].body.pop(1).name
Out[31]: 'b'

In [32]: print pasta.dump(tree)
class Foo():
  def a(self):
    return 1
def c(self):
    return 3

The first snippet here acts as expected. I would expect the result of the second snippet to be the same, but in reality a newline seems to get lost.

Bad indentation when rewriting funcion body (new bug)

Hi! I found an issue similar to #62. When certain conditions are met and I rewrite the body of a function, it won't be properly indented, therefore it will produce incorrect or syntactically invalid code.

I found the bug while rewriting some parts of this HostImporter class. I shrinked the code to find the minimum code required to trigger the bug and ended up with this:

import ast
import pasta

source_code = '''
class K(object):
    def f1():
        if True:
            yield host_ip
            pass

    def f2():
        pass
'''

class SampleTransformer(ast.NodeTransformer):
    def visit_FunctionDef(self, node):
        node.body = [ast.copy_location(ast.Pass(), node.body[0])]
        return node

tree = pasta.parse(source_code)
SampleTransformer().visit(tree)

print pasta.dump(tree)

The expected output of running this script should be

class K(object):
    def f1():
        pass
    def f2():
        pass

But the output is this instead:

class K(object):
    def f1():
        pass
def f2():
        pass

This code is incorrect, but not yet syntactically invalid. If you add a third f3() method to the K class, then the output will be:

class K(object):
    def f1():
        pass
def f2():
        pass
    def f3():
        pass

This will raise a SyntaxError when executed.

Missing support for Python3 features

These AST nodes are not supported and can cause parse failures:

Will update this list as support is added.

Parsing for non-defaulted kwargs after `*args` fails

import pasta

aaa = '''
def meow(self, *args, blah):
    pass
'''

pasta.parse(aaa)

---------------------------------------------------------------------------
AnnotationError                           Traceback (most recent call last)
<ipython-input-12-cf34fed7cd55> in <module>
      6 '''
      7
----> 8 pasta.parse(aaa)

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/__init__.py in parse(src)
     23   t = ast_utils.parse(src)
     24   annotator = annotate.AstAnnotator(src)
---> 25   annotator.visit(t)
     26   return t
     27

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in visit(self, node)
   1170       fmt.set(node, 'indent', self._indent)
   1171       fmt.set(node, 'indent_diff', self._indent_diff)
-> 1172       super(AstAnnotator, self).visit(node)
   1173     except (TypeError, ValueError, IndexError, KeyError) as e:
   1174       raise AnnotationError(e)

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in visit(self, node)
    130   def visit(self, node):
    131     self._stack.append(node)
--> 132     super(BaseVisitor, self).visit(node)
    133     assert node is self._stack.pop()
    134

/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py in visit(self, node)
    260         method = 'visit_' + node.__class__.__name__
    261         visitor = getattr(self, method, self.generic_visit)
--> 262         return visitor(node)
    263
    264     def generic_visit(self, node):

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in wrapped(self, node, *args, **kwargs)
     45       if prefix:
     46         self.prefix(node, default=self._indent if statement else '')
---> 47       f(self, node, *args, **kwargs)
     48       if suffix:
     49         self.suffix(node, max_lines=max_suffix_lines, semicolon=semicolon,

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in visit_Module(self, node)
    218   @module
    219   def visit_Module(self, node):
--> 220     self.generic_visit(node)
    221
    222   @block_statement

/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py in generic_visit(self, node)
    268                 for item in value:
    269                     if isinstance(item, AST):
--> 270                         self.visit(item)
    271             elif isinstance(value, AST):
    272                 self.visit(value)

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in visit(self, node)
   1170       fmt.set(node, 'indent', self._indent)
   1171       fmt.set(node, 'indent_diff', self._indent_diff)
-> 1172       super(AstAnnotator, self).visit(node)
   1173     except (TypeError, ValueError, IndexError, KeyError) as e:
   1174       raise AnnotationError(e)

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in visit(self, node)
    130   def visit(self, node):
    131     self._stack.append(node)
--> 132     super(BaseVisitor, self).visit(node)
    133     assert node is self._stack.pop()
    134

/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py in visit(self, node)
    260         method = 'visit_' + node.__class__.__name__
    261         visitor = getattr(self, method, self.generic_visit)
--> 262         return visitor(node)
    263
    264     def generic_visit(self, node):

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in wrapped(self, node, *args, **kwargs)
     93   def wrapped(self, node, *args, **kwargs):
     94     self.prefix(node, default=self._indent)
---> 95     f(self, node, *args, **kwargs)
     96     if hasattr(self, 'block_suffix'):
     97       last_child = ast_utils.get_last_child(node)

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in visit_FunctionDef(self, node)
    399     with self.scope(node, 'args', trailing_comma=args_count > 0,
    400                     default_parens=True):
--> 401       self.visit(node.args)
    402
    403     if getattr(node, 'returns', None):

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in visit(self, node)
   1170       fmt.set(node, 'indent', self._indent)
   1171       fmt.set(node, 'indent_diff', self._indent_diff)
-> 1172       super(AstAnnotator, self).visit(node)
   1173     except (TypeError, ValueError, IndexError, KeyError) as e:
   1174       raise AnnotationError(e)

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in visit(self, node)
    130   def visit(self, node):
    131     self._stack.append(node)
--> 132     super(BaseVisitor, self).visit(node)
    133     assert node is self._stack.pop()
    134

/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py in visit(self, node)
    260         method = 'visit_' + node.__class__.__name__
    261         visitor = getattr(self, method, self.generic_visit)
--> 262         return visitor(node)
    263
    264     def generic_visit(self, node):

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in wrapped(self, node, *args, **kwargs)
     45       if prefix:
     46         self.prefix(node, default=self._indent if statement else '')
---> 47       f(self, node, *args, **kwargs)
     48       if suffix:
     49         self.suffix(node, max_lines=max_suffix_lines, semicolon=semicolon,

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in visit_arguments(self, node)
   1083       self.visit(arg)
   1084       self.attr(node, 'kw_default_%d' % i, [self.ws, '=', self.ws],
-> 1085                 default='=')
   1086       self.visit(default)
   1087       arg_i += 1

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in attr(***failed resolving arguments***)
   1387     for attr_val in attr_vals:
   1388       if isinstance(attr_val, six.string_types):
-> 1389         attr_parts.append(self.token(attr_val))
   1390       else:
   1391         attr_parts.append(attr_val())

/private/tmp/test2/venv/lib/python3.7/site-packages/pasta/base/annotate.py in token(self, token_val)
   1313     if token.src != token_val:
   1314       raise AnnotationError("Expected %r but found %r\nline %d: %s" % (
-> 1315           token_val, token.src, token.start[0], token.line))
   1316
   1317     # If the token opens or closes a parentheses scope, keep track of it

AnnotationError: Expected '=' but found ')'
line 2: def meow(self, *args, blah):

This is valid syntax in Python 3. IIRC, this wasn't always the case but later added in a 3.X version, although I'm not exactly sure when.

Comment at the end of the file is being removed after executing pasta.dump

Hi again =)

I found this issue while running the following script "script.py "on file "rename.py"
Script.py

import os
import pasta
from pasta.augment import rename

path = r'rename.py'
with open(path, mode='r') as file:
    tree = pasta.parse(file.read())
    for moved_classes in list_of_classes_to_move:
        rename.rename_external(tree, 'a.b.c', 'a.b.d')
    source_code = pasta.dump(tree)

with open(path, mode='w') as file:
    print(source_code)
    file.write(source_code)

File before the pasta.dump:

from a.b import c 

c

#Testing
#Test1

File after the pasta.dump:

from a.b import d 

d

I've noticed that this issue just occurs when the commentary is at the end of the file, without any more code after it.

So in this example, the pasta.dump works just fine:

from a.b import c 

c

#Testing
#Test1

print('Hi')

After dump:

from a.b import d 

d

#Testing
#Test1

print('Hi')

cc @nicoddemus

Doesn't respect comments between functions

When you try to remove a function it seems to count comments after it that are clearly not on the indentation level of the body of the function as part of the function and they seem to get removed.

In [4]: src = """\
   ...: x = 3
   ...: def f(y):
   ...:   return y + 2
   ...: # lol
   ...: def g(y):
   ...:   return y + 7
   ...: """

In [5]: n = pasta.parse(src)

In [6]: n
Out[6]: <_ast.Module at 0x3ac1750>

In [7]: n.body
Out[7]: 
[<_ast.Assign at 0x3ac1790>,
 <_ast.FunctionDef at 0x3ac1b10>,
 <_ast.FunctionDef at 0x3ac19d0>]

In [8]: n.body = n.body[n.body[0], n.body[2]]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-5d440e76f12a> in <module>()
----> 1 n.body = n.body[n.body[0], n.body[2]]

TypeError: list indices must be integers, not tuple

In [9]: n.body = [n.body[0], n.body[2]]

In [10]: n.body
Out[10]: [<_ast.Assign at 0x3ac1790>, <_ast.FunctionDef at 0x3ac19d0>]

In [11]: pasta.dump(n)
Out[11]: 'x = 3\ndef g(y):\n  return y + 7\n'

In [12]: print pasta.dump(n)
x = 3
def g(y):
  return y + 7

Support for 'nonlocal' keywords

Seems pasta does not support nonlocal at this time.

def a():
    f=1
    def b():
        nonlocal f

Problem to recognize syntax

Hi,

I have the following snippet that is valid in Python 2.7 and is correctly parsed.

numpy.array([list(range(10)), list(range(10))])[1, 3]
numpy.array([list(range(10)), list(range(10))])[1:2, 3]

When I move the end bracket to a new line, the library cannot parse the code with a slice.

What I mean is, this following code works correctly:

numpy.array([list(range(10)), list(range(10))])[1, 3
]

But this one doesn't work:

numpy.array([list(range(10)), list(range(10))])[1:2, 3
]

Executing:

(test_pasta) λ python test.py

(test_pasta) λ python script.py
Traceback (most recent call last):
  File "script.py", line 6, in <module>
    tree = pasta.parse(file.read())
  File "w:\william\repo_pasta\pasta\__init__.py", line 25, in parse
    annotator.visit(t)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 168, in visit_Module
    self.generic_visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 249, in generic_visit
    self.visit(item)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 513, in visit_Expr
    self.visit(node.value)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 779, in visit_Subscript
    self.visit(node.slice)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1013, in visit_ExtSlice
    self.token(']')
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1106, in token
    token_val, token.src, token.start[0], token.line))
pasta.base.annotate.AnnotationError: Expected ']' but found '\n'
line 1: numpy.array([list(range(10)), list(range(10))])[1:2, 3

cc @nicoddemus

Fails to parse usages of metaclass

I tried to use the tensorflow update script on a codebase that uses abstract classes with ABCMeta. Parsing failed with

AnnotationError: Expected ':' but found 'metaclass'

Minimal example:

pasta.parse('class A(metaclass=X): pass')

Support for async code

Hi! I noted that pasta isn't able to parse async Python code. For example, this script will raise an error instead of succeeding:

import pasta

source_code = """
async def f():
    return
"""

pasta.parse(source_code)

I got the following traceback:

Traceback (most recent call last):
  File "sscce.py", line 8, in <module>
    pasta.parse(source_code)
  File "/src/.venv/src/pasta/pasta/__init__.py", line 25, in parse
    annotator.visit(t)
  File "/src/.venv/src/pasta/pasta/base/annotate.py", line 1214, in visit
    super(AstAnnotator, self).visit(node)
  File "/src/.venv/src/pasta/pasta/base/annotate.py", line 133, in visit
    super(BaseVisitor, self).visit(node)
  File "/nix/store/hmcha8bv2ziwabla76hd612vgxr20pnd-python3-3.7.6/lib/python3.7/ast.py", line 271, in visit
    return visitor(node)
  File "/src/.venv/src/pasta/pasta/base/annotate.py", line 47, in wrapped
    f(self, node, *args, **kwargs)
  File "/src/.venv/src/pasta/pasta/base/annotate.py", line 225, in visit_Module
    self.generic_visit(node)
  File "/nix/store/hmcha8bv2ziwabla76hd612vgxr20pnd-python3-3.7.6/lib/python3.7/ast.py", line 279, in generic_visit
    self.visit(item)
  File "/src/.venv/src/pasta/pasta/base/annotate.py", line 1214, in visit
    super(AstAnnotator, self).visit(node)
  File "/src/.venv/src/pasta/pasta/base/annotate.py", line 133, in visit
    super(BaseVisitor, self).visit(node)
  File "/nix/store/hmcha8bv2ziwabla76hd612vgxr20pnd-python3-3.7.6/lib/python3.7/ast.py", line 271, in visit
    return visitor(node)
  File "/nix/store/hmcha8bv2ziwabla76hd612vgxr20pnd-python3-3.7.6/lib/python3.7/ast.py", line 279, in generic_visit
    self.visit(item)
  File "/src/.venv/src/pasta/pasta/base/annotate.py", line 1214, in visit
    super(AstAnnotator, self).visit(node)
  File "/src/.venv/src/pasta/pasta/base/annotate.py", line 133, in visit
    super(BaseVisitor, self).visit(node)
  File "/nix/store/hmcha8bv2ziwabla76hd612vgxr20pnd-python3-3.7.6/lib/python3.7/ast.py", line 271, in visit
    return visitor(node)
  File "/src/.venv/src/pasta/pasta/base/annotate.py", line 47, in wrapped
    f(self, node, *args, **kwargs)
  File "/src/.venv/src/pasta/pasta/base/annotate.py", line 680, in visit_Return
    self.token('return')
  File "/src/.venv/src/pasta/pasta/base/annotate.py", line 1353, in token
    token_val, token.src, token.start[0], token.line))
pasta.base.annotate.AnnotationError: Expected 'return' but found 'async'
line 2: async def f():

It would be nice if it could handle it properly.

Failed to recognize python2 syntax for exception

Hi,

Running google-pasta with python 2.7 against a file that has an try/except that follows the syntax from Python2 throws a pasta.base.annotate.AnnotationError saying that Expected 'as' but found ','

Below you can see a minimum code that reproduces the problem

Using Python 2.7

(test_pasta) λ python --version
Python 2.7.12

File test.py

def foo():
    try:
        pass
    except Exception, e:
        pass

Script:

import pasta
path = r'test.py'

with open(path) as file:
    tree = pasta.parse(file.read())

Throws the following error:

pasta.base.annotate.AnnotationError: Expected 'as' but found ','
line 4:     except Exception, e:

Full StackStrace

Traceback (most recent call last):
  File "past_script.py", line 15, in <module>
    tree = pasta.parse(file.read())
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\__init__.py", line 25, in parse
    annotator.visit(t)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 1033, in visit
    super(AstAnnotator, self).visit(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\william\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 158, in visit_Module
    self.generic_visit(node)
  File "W:\william\envs\test_pasta\lib\ast.py", line 249, in generic_visit
    self.visit(item)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 1033, in visit
    super(AstAnnotator, self).visit(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\william\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 347, in visit_FunctionDef
    self.visit(stmt)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 1033, in visit
    super(AstAnnotator, self).visit(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\william\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 371, in visit_TryExcept
    self.visit(handler)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 1033, in visit
    super(AstAnnotator, self).visit(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\william\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 407, in visit_ExceptHandler
    self.attr(node, 'as', [self.ws, 'as', self.ws], default=' as ')
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 1139, in attr
    attr_parts.append(self.token(attr_val))
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 1083, in token
    token_val, token.src, token.start[0], token.line))
pasta.base.annotate.AnnotationError: Expected 'as' but found ','
line 4:     except Exception, e:

cc: @nicoddemus

Failed to recognize empty init.py

Hello,

Running google-pasta with python 2.7 against a empty __init__.py file throws a pasta.base.annotate.AnnotationError list index out of range

Script:

import pasta
path = r'__init__.py'
with open(path) as file:
    tree = pasta.parse(file.read())

Stack:

Traceback (most recent call last):
  File "script.py", line 10, in <module>
    tree = pasta.parse(file.read())
  File "w:\william\repo_pasta\pasta\__init__.py", line 25, in parse
    annotator.visit(t)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1042, in visit
    raise AnnotationError(e)
pasta.base.annotate.AnnotationError: list index out of range

Error parsing [...]

I am running tf_upgrade_v2 tool in TensorFlow which is based on Pasta but hitting error parsing [...].

Steps to reproduce:
import pasta
tree = pasta.parse('foo[...] = 5')

This gives the following error:
pasta.base.annotate.AnnotationError: Expected '.' but found '['
line 1: foo[...] = 5

Some tests are broken

Latest build:
https://travis-ci.org/github/google/pasta/builds/755358468

Oddly, they pass locally. Perhaps a different minor version? Looking into it.

Plethora of 2.7 parse issues

Tried on our large and some-parts-somewhat-ancient 2.7-compatible codebase (the kind it'd be nice to have, say, automatic transformations for...). Most of the files worked but a few recurring issues - perhaps these come under "some features not supported" but I didn't see an explicit list anywhere. All come from real files that work.

Complex numbers don't work

1j
# AnnotationError: no ordering relation is defined for complex numbers

Various issues with tuple parameter unpacking:

def a((b,c), d):
  pass
# AnnotationError: Expected ',' but found ')'
def some((a,b,c)):
  pass
# AnnotationError: Expected ':' but found ')'

Slicing with steps:

some_sliceable[::]
# AnnotationError: Expected 'None' but found ']'

Calling functions with extra newlines:

def call_func(a, something_else): # valid function
  pass
call_func((1
,), something_else=True)

# AnnotationError: Expected 'something_else' but found ')'

<> operator:

True <> False
# AnnotationError: Expected '!=' but found '<>'

BOM at start of file:

pasta.parse("\xef\xbb\xbf\r\nimport sys")
# Expected 'import' but found '\xef'`

Old-style repr (for obvious reasons having trouble typing this one in markdown)

pasta.parse("\x60True\x60")
# Expected 'repr' but found '\x60'

Something with exec and lines?

exec("", None, None)
True
# Expected 'True' but found ')'

f-strings over multiple lines

Thanks for fixing issue #65 !
I still have one file in my project though that I can't upgrade to Tensorflow 2.0 because of a new fstring-related error. There is no Tensorflow in this file, so I could just upgrade the project without this file, but maybe there are others experiencing the same issue.

The problem arises in the following setting:
I'm writing html/css code to a file, using strings that are split over multiple lines.
In addition to this, some strings are formatted using f-strings.
So as a minimal example, this is what it could look like:

with open("test.out", "w") as f:
	a = 1
	f.write(f"{a}"
                 "{"
		 "}")

I believe the problem lies in the fact that the first string is an f-string. The second string is not, but is treated like one. Thus the opening bracket causes an issue.

Incorrect rename behavior

When running rename_external on code with both import x and from x import y, the behavior seems to be incorrect.

In the example below, the generated code will contain an NameError: name 'custom_request' is not defined

The expected output should be my_app.custom_request.get('https://google.com')

>>> import pasta
>>> from pasta.augment import rename
>>> code = """
... import requests
... from requests import Response
...
... requests.get('https://google.com')"""
>>> tree = pasta.parse(code)
>>> rename.rename_external(tree, 'requests', 'myapp.custom_request')
True
>>> print(pasta.dump(tree))

import myapp.custom_request
from myapp.custom_request import Response

custom_request.get('https://google.com')

Failed to recognize nested try:

Hi,

Running the parser against a file with "nested try" results in the following error:

pasta.base.annotate.AnnotationError: Expected 'error' but found 'try'
line 2:     try:

I found this issue with

Python 2.7.8 :: Continuum Analytics, Inc.
Python 2.7.9 :: Continuum Analytics, Inc.
Python 2.7.12

(In python 3.4.5 works just fine)

Test.py

try:
    try:
        error = 1/0
    except:
        print('1')
finally:
    print('2')

Script.py:

import os
import pasta

path = r'test.py'
with open(path) as file:
   tree = pasta.parse(file.read())

Full Log:

Traceback (most recent call last):
  File "script.py", line 10, in <module>
    tree = pasta.parse(file.read())
  File "w:\william\repo_pasta\pasta\__init__.py", line 25, in parse
    annotator.visit(t)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 168, in visit_Module
    self.generic_visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 249, in generic_visit
    self.visit(item)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 367, in visit_TryFinally
    self.visit(stmt)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 379, in visit_TryExcept
    self.visit(stmt)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 470, in visit_Assign
    self.visit(target)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 747, in visit_Name
    self.token(node.id)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1106, in token
    token_val, token.src, token.start[0], token.line))
pasta.base.annotate.AnnotationError: Expected 'error' but found 'try'
line 2:     try:

Bad indentation when rewriting funcion body

Hi! I'm using your library in a toy project to change the body of some functions. When I change it and dump the modified ast, the code isn't indented right, so executing the dumped code will cause a SyntaxError.

Here is an example script that describes the issue:

import ast
import pasta

source_code = '''
def test():
    return 5
'''

class SampleTransformer(ast.NodeTransformer):
    def visit_FunctionDef(self, node):
        node.body = [ast.copy_location(ast.Pass(), node.body[0])]
        return node

tree = pasta.parse(source_code)
SampleTransformer().visit(tree)

print pasta.dump(tree)

When I run it, I get the following output:

def test():
pass

Instead of what I expected, that has proper indentation:

def test():
    pass

I read that there is a known issue related to changing the indentation level of a block, but I think this is another (but maybe related) issue.

foo[:bar] doesn't work if [:bar] is on a new line

I am seeing an error when parsing foo[:bar] if [:bar] is on the next line.

The following code:

import pasta
pasta.parse('test(foo\n[:2])')

causes error:

pasta.base.annotate.AnnotationError: Expected '[' but found '\n'
line 1: test(foo

Failed to recognize import with parentheses

Hi Nick,

Running google-pasta with python 2.7 against a file that has an import with parentheses throws a pasta.base.annotate.AnnotationError:

I'm using Python 2.7

(test_pasta) λ python --version
Python 2.7.12

With the following script:

import pasta
from pasta.augment import rename

path = r'test.py'

with open(path) as file:
    tree = pasta.parse(file.read())

On the test.py file:

from test2 import (test2)
print(test2.ID)

Error:

pasta.base.annotate.AnnotationError: Expected 'test2' but found '('
line 1: from test2 import (test2)

Full stack trace:

Traceback (most recent call last):
  File "script.py", line 10, in <module>
    tree = pasta.parse(file.read())
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\__init__.py", line 25, in parse
    annotator.visit(t)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 1038, in visit
    super(AstAnnotator, self).visit(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\william\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 162, in visit_Module
    self.generic_visit(node)
  File "W:\william\envs\test_pasta\lib\ast.py", line 249, in generic_visit
    self.visit(item)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 1038, in visit
    super(AstAnnotator, self).visit(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\william\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 545, in visit_ImportFrom
    self.visit(alias)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 1038, in visit
    super(AstAnnotator, self).visit(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\william\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 910, in visit_alias
    default=node.name)
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 1148, in attr
    attr_parts.append(self.token(attr_val))
  File "W:\william\envs\test_pasta\lib\site-packages\pasta\base\annotate.py", line 1088, in token
    token_val, token.src, token.start[0], token.line))

cc @nicoddemus

Colgate

Incomplete handling of complex f-strings

Pasta, which is required for the tf_upgrade_v2 script of Tensorflow, throws an error on a specific f-string. I believe support for f-strings has been added recently, but this seems to be incomplete.

This concerns the following minimal code example

import os
path = ["path", "to", "file"]
c = f"/{(os.sep).join(path)}"

When running tf_upgrade_v2 (pip3 install tensorflow==tensorflow==2.0.0-alpha0)

Extend scope.analyze to support PEP 484 forward references

https://www.python.org/dev/peps/pep-0484/#forward-references

This style of type annotation isn't recognized by scope.analyze right now. The b imported in this module appears to be unused.

# File models/a.py
from models import b
class A(Model):
    def foo(self, b: 'b.B'): ...

Support formatting-preserving insertion into collections

Original report:

########## Example 1: insert the "'b':'b_s'" into dict D by alphabet order
D = {
    'a': 'a_s',
               # <-----------------------------------'b': 'b_s'
    'c': 'c_s',
}

# expected output:
D = {
    'a': 'a_s',
    'b': 'b_s',
    'c': 'c_s',
}

# output actually got:
D = {
    'a': 'a_s',
    'b': 'b_s', 'c': 'c_s',
}

########## Example 2: insert the "'y':'y_s'" into dict D by alphabet order
D = {
    'a': 'a_s',
    'c': 'c_s',
                   # <-----------------------------------'y': 'y_s'
}

# expected output:
D = {
    'a': 'a_s',
    'c': 'c_s',
    'y': 'y_s',
}

# output actually got:
D = {
    'a': 'a_s',
    'c': 'c_s', 'y': 'y_s',
}

Notes:
This should be supported by a collection_insert (or similar) function, taking the collection node (e.g. a Dict), the node (or nodes) to insert, and the index to insert at. This should result in consistent formatting as described above.

MIgrate from setuptools test to tox

setuptools test command is deprecated, tox is the recommended replacement.

Support PEP 448 - Additional Unpacking Generalizations

(extracted from #58)

For example merged = {**a, **b} fails with a cryptic error

  File "pasta/base/annotate.py", line 804, in visit_Dict
    self.visit(key)
  File "pasta/base/annotate.py", line 1189, in visit
    fmt.set(node, 'indent', self._indent)
  File "pasta/base/formatting.py", line 37, in set
    _formatting_dict(node)[name] = value
  File "pasta/base/formatting.py", line 53, in _formatting_dict
    return getattr(node, PASTA_DICT)
AttributeError: 'NoneType' object has no attribute '__pasta__'

because the node key in visit_Dict will be None

Default formatting Subscript slice is inconsistent in python3.9+

ast.dump(ast.parse('l[1:2, 3]'))

3.7:  Module(body=[Expr(value=Subscript(value=Name(id='l', ctx=Load()), slice=ExtSlice(dims=[Slice(lower=Num(n=1), upper=Num(n=2), step=None), Index(value=Num(n=3))]), ctx=Load()))])

pasta.dump(ast.parse('l[1:2, 3]')) --> 'l[1:2, 3]'

3.9:  Module(body=[Expr(value=Subscript(value=Name(id='l', ctx=Load()), slice=Tuple(elts=[Slice(lower=Constant(value=1), upper=Constant(value=2)), Constant(value=3)], ctx=Load()), ctx=Load()))], type_ignores=[])

pasta.dump(ast.parse('l[1:2, 3]')) --> 'l[(1:2, 3)]'

See: https://bugs.python.org/issue34822

Create release in pypi

Failed to recognize semicolons on python file

Hi,

The parser does not recognize the semicolons on python files.
I don't know how often people use semicolons as separator on python, but I think that it's a good idea to provide some support for such cases =)

Here it's a small reproducible example of the error:

File:
a=(1+1);print(a)

Script

import pasta
from pasta.augment import rename

path = r'test.py'

with open(path) as file:
        tree = pasta.parse(file.read())

Error:

Traceback (most recent call last):

  File "script.py", line 7, in <module>
    tree = pasta.parse(file.read())
  File "/opt/pasta/pasta/__init__.py", line 25, in parse
    annotator.visit(t)
  File "/opt/pasta/pasta/base/annotate.py", line 1044, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py34/lib/python3.4/ast.py", line 245, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 166, in visit_Module
    self.generic_visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py34/lib/python3.4/ast.py", line 253, in generic_visit
    self.visit(item)
  File "/opt/pasta/pasta/base/annotate.py", line 1044, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py34/lib/python3.4/ast.py", line 245, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 509, in visit_Expr
    self.visit(node.value)
  File "/opt/pasta/pasta/base/annotate.py", line 1044, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py34/lib/python3.4/ast.py", line 245, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 626, in visit_Call
    self.visit(node.func)
  File "/opt/pasta/pasta/base/annotate.py", line 1044, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py34/lib/python3.4/ast.py", line 245, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 743, in visit_Name
    self.token(node.id)
  File "/opt/pasta/pasta/base/annotate.py", line 1094, in token
    token_val, token.src, token.start[0], token.line))
pasta.base.annotate.AnnotationError: Expected 'print' but found ';'
line 1: a=(1+1);print(a)

Consider jedi for refactoring

The is already some work done in jedi for refactoring

CI doesn't works

Hello,

Looks like Travis has stopped working in the Google Cloud Platform organization. we should migrate to something else ex. github actions.

Best regards,
Kamil Bregula

Support mixing tabs and spaces in python2

Yuck... but python2 recognizes a tab as 8 spaces when the two are mixed. Currently this produces an IndentationError when parsing.

Some good explanation here:
https://stackoverflow.com/questions/2034517/pythons-interpretation-of-tabs-and-spaces-to-indent

There is probably a more graceful way to handle this situation, anyway, aside from failing outright.

IndentationError when appending new node to function body

Hi! I found a bug in pasta when I try to add a node to a function body (I think it's the same with every node class that has a body). I built the following script to reproduce the bug:

import ast
import pasta

source_code = """
if 1:  # This is required
    def f():
        a=1
"""

class SampleTransformer(ast.NodeTransformer):
    def visit_FunctionDef(self, node):
        return_node = ast.copy_location(
            ast.Return(value=None), node.body[0])
        # return_node = ast.Return(value=None)  # This produces the same error than above
        node.body.append(return_node)
        return node

tree = pasta.parse(source_code)
SampleTransformer().visit(tree)

print(pasta.dump(tree))

The expected output of it should be:

if 1:  # This is required
    def f():
        a=1
        return

But instead it produces a syntactically invalid code:

if 1:  # This is required
    def f():
        a=1
            return

google / pasta Goto Github PK

pasta's Introduction

pasta: Python AST Augmentation

Mission

Design Goals

Python Version Support

Dependencies

Basic Usage

Built-in Augmentations

Rename an imported name

Known issues and limitations

Developing

Disclaimer

pasta's People

Stargazers

Watchers

Forkers

pasta's Issues

Recommend Projects

Recommend Topics

Recommend Org