hchasestevens / astpath Goto Github PK

View Code? Open in Web Editor NEW

243.0 6.0 57.0 37 KB

A command-line search utility for Python ASTs using XPath syntax.

Home Page: https://pypi.python.org/pypi/astpath

Python 100.00%

python-ast xpath xpath-syntax python ast metaprogramming search command-line-tool

astpath's Introduction

astpath

[e ɛs ti pæθ] n.

Ⅰ. A command-line utility for querying Python ASTs using XPath syntax.

ⅠⅠ. A better way of searching through your codebase.

Example usage

Finding all usages of the eval builtin:

$ astpath ".//Call/func/Name[@id='eval']" | head -5
./rlcompleter.py:136    >            thisobject = eval(expr, self.namespace)
./warnings.py:176       >            cat = eval(category)
./rexec.py:328  >        return eval(code, m.__dict__)
./pdb.py:387    >                    func = eval(arg,
./pdb.py:760    >            return eval(arg, self.curframe.f_globals,

Finding all numbers:

$ astpath ".//Num" | head -5
./DocXMLRPCServer.py:31 >        here = 0
./DocXMLRPCServer.py:41 >        while 1:
./DocXMLRPCServer.py:57 >            elif text[end:end+1] == '(':
./DocXMLRPCServer.py:82 >                    args[1:],
./DocXMLRPCServer.py:96 >            argspec = object[0] or argspec

... that are never assigned to a variable:

$ astpath ".//Num[not(ancestor::Assign)]" | head -5
./DocXMLRPCServer.py:41 >        while 1:
./DocXMLRPCServer.py:57 >            elif text[end:end+1] == '(':
./DocXMLRPCServer.py:201        >                assert 0, "Could not find method in self.functions and no "\
./DocXMLRPCServer.py:237        >        self.send_response(200)
./DocXMLRPCServer.py:252        >                 logRequests=1, allow_none=False, encoding=None,

... and are greater than 1000:

$ astpath ".//Num[not(ancestor::Assign) and number(@n) > 1000]" | head -5
./decimal.py:959      >                    return 314159
./fractions.py:206    >    def limit_denominator(self, max_denominator=1000000):
./pty.py:138  >    return os.read(fd, 1024)
./whichdb.py:94       >    if magic in (0x13579ace, 0x13579acd, 0x13579acf):
./whichdb.py:94       >    if magic in (0x13579ace, 0x13579acd, 0x13579acf):

Finding names longer than 42 characters:

$ astpath "//Name[string-length(@id) > 42]"
./site-packages/setuptools/dist.py:59   >_patch_distribution_metadata_write_pkg_info()
./site-packages/setuptools/command/easy_install.py:1759 >        updater=clear_and_remove_cached_zip_archive_directory_data)
./test/test_reprlib.py:268      >        module = areallylongpackageandmodulenametotestreprtruncation
./test/test_argparse.py:2744    >    MEPBase, TestMutuallyExclusiveOptionalsAndPositionalsMixed):

Finding except clauses that raise a different exception class than they catch:

$ astpath "//ExceptHandler[body//Raise/exc//Name and not(contains(body//Raise/exc//Name/@id, type/Name/@id))]" | head -5
./hashlib.py:144        >except ImportError:
./plistlib.py:89        >        except KeyError:
./plistlib.py:103       >        except KeyError:
./nntplib.py:868        >        except ValueError:
./argparse.py:1116      >        except KeyError:

Finding beginnings of unreachable code blocks:

$ astpath "//body/*[preceding-sibling::Return or preceding-sibling::Raise][1]"
./unittest/test/testmock/testhelpers.py:381     >        class Foo(object):
./test/test_deque.py:16 >    yield 1
./test/test_posix.py:728        >            def _create_and_do_getcwd(dirname, current_path_length = 0):

Finding candidates for replacement with sum:

$ astpath -A 1 "//For/body[AugAssign/op/Add and count(child::*)=1]" | head -6
./functools.py:374      >        for item in sorted_items:
./functools.py:375                   key += item
./statistics.py:177     >    for d, n in sorted(partials.items()):
./statistics.py:178              total += Fraction(n, d)
./pstats.py:512 >    for calls in callers.values():
./pstats.py:513          nc += calls

Finding classes matching a regular expression:

$ astpath "//ClassDef[re:match('.*Var', @name)]" | head -5
./typing.py:452  >      class TypeVar(_TypingBase, _root=True):
./typing.py:1366 >      class _ClassVar(_FinalTypingBase, _root=True):
./tkinter/__init__.py:287  >    class Variable:
./tkinter/__init__.py:463  >    class StringVar(Variable):
./tkinter/__init__.py:485  >    class IntVar(Variable):

astpath can also be imported and used programmatically:

>>> from astpath import search
>>> len(search('.', '//Print', print_matches=False))  # number of print statements in the codebase
751

Installation

It is recommended that astpath be installed with the optional lxml dependency, to allow full use of the XPath query language. To do so,

pip install astpath[xpath]

Alternatively, a no-dependency version using Python's builtin XPath subset can be installed via

pip install astpath

astpath supports both Python 2.7 and 3.x.

Contacts

Name: H. Chase Stevens
Twitter: @hchasestevens

astpath's People

Contributors

Stargazers

Watchers

astpath's Issues

How to "find" function call with arguments?

Hi, I've just discovered astpath and it's really cool!

I can get the file and line at the start of the match, but how do I get the whole match?
Suppose I'm searching for logging.debug and my code looks as follows, thanks to black:

...
logging.debug(
  "A very %s long %r line %s with %s a lot of %r format specifiers",
  something.foo.bar,
  [some.expression > that - results - in - an < object for x in y],
  yet.another.blah,
  ...
)
...

I wish that astpath could tell me the exact boundaries of the function call including the call arguments.

I can simulate some of this with before_ and after_context but that's kinda hacky...

Error if directory doesn't exist

Currently, searching a directory that doesn't exist will silently fail.

$ astpath -d /tmp/this-dir-doesnt-exist './/Try'

If a directory doesn't exist because I misspelled it, I'd like to get an error.

Bunch of backwards incompatible changes - fork or PR?

Thanks for astpath!

I'm looking at patching it in a number of backwards incompatible ways:

output with 1-indexed line numbers (as per grep)
output that is more compatible with grep and other tools e.g. column numbers that work out of the box with emacs
colour output instead of >
internal functions might get re-organised in backwards incompatible ways.

Are you interested in PRs, or should I just go ahead and create my own fork? I'm also thinking of adding other modern things:

black formatting
test suite
GitHub Actions etc.

Thanks!

`TypeError: 'float' object is not iterable` when attempting to emit a count of elements

I'm currently using astpath to determine if we need a function's lone keyword to be optional, or if all of its callers pass it and therefore we can make it unconditionally required. I can get a list of callsites which use it:

$ astpath './/Call[func[Name[@id="report_diagnostic_event"]] and count(keywords) = 1]' -d cloudinit | wc -l
46

and then I thought I'd get clever:

$ astpath 'count(.//Call[func[Name[@id="report_diagnostic_event"]] and count(keywords) = 1])' -d cloudinit
Traceback (most recent call last):
  File "/home/daniel/.virtualenvs/cloud-init/bin/astpath", line 8, in <module>
    sys.exit(main())
  File "/home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/astpath/cli.py", line 37, in main
    search(
  File "/home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/astpath/search.py", line 149, in search
    file_matches = find_in_ast(
  File "/home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/astpath/search.py", line 42, in find_in_ast
    for result in results:
TypeError: 'float' object is not iterable

I don't really know if I'm specifying valid XPath, but if this should work then I'd like it to (and if it shouldn't then perhaps a friendlier error message?).

Return matching lines instead of printing them

search("core", ".//Call/func/Name[@id='AccountFrozenEmail']", recurse=True, print_matches=True)
[('core/services/account.py', 15)]

Instead of printing matches, would it be possible to return the matches as strings?

Allow the specification of multiple directories/files

Given #13, I would like to run astpath explicitly against my code directory (cloudinit/) and my legacy tests directory (tests/). If I invoke astpath with both directories:

astpath "//Str[@s='system']" -d cloudinit -d tests

then only tests/ gets searched; I have to invoke astpath separately for each directory I would like to search.

Ideally, I would be able to specify -d multiple times and have astpath search each one.

Element has no ancestor with line number.

I'd like to print an attribute, but I'm getting an error:

$ astpath '//Name/@id'
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/astpath/search.py", line 44, in find_in_ast
    linenos = query(
  File "/usr/local/lib/python3.9/site-packages/astpath/search.py", line 12, in lxml_query
    return element.xpath(expression)
AttributeError: 'lxml.etree._ElementUnicodeResult' object has no attribute 'xpath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/astpath", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/astpath/cli.py", line 37, in main
    search(
  File "/usr/local/lib/python3.9/site-packages/astpath/search.py", line 149, in search
    file_matches = find_in_ast(
  File "/usr/local/lib/python3.9/site-packages/astpath/search.py", line 49, in find_in_ast
    raise AttributeError(
AttributeError: Element has no ancestor with line number.

Maybe if it's an ElementUnicodeResult we should just return instead of recursing into element.expath(expression)? Thanks for your work on this project!

how to search in class basses for class hierarchy?

I want to search for subclasses of a class as below:

from testing_classes import Child, Parent

class test(Child):
    def __init__(self):
        pass

    def child_method(self):
        print('child')

    def child_class_method(self):
        print('child class method')

class test2(Child):
    def __init__(self):
        pass

    def child_method(self):
        print('child')

    def child_class_method(self):
        print('child class method')

Using the following xpath expression but no result
>>> astpath "//ClassDef/bases[contains(@id, 'Child')]"

Please guide me on how to achieve this, also its better if we have a list of more examples for the beginners

Error if invalid node given

Cool project.

Is it possible to raise an error if I search for a node type that doesn't really exist? For example if I typo the name //Listcomp instead of //ListComp, I'd like to get an error instead of concluding there aren't any list comprehensions.

Using ASTpath to access internal data structure key/value (dictionary)

a = None
b = 5
c = {'f': 6, 'g': {'i': {'l': 10}}}

In Python, I can just:

print(c['g']['i']['l'])

How do I do equivalent with ASTpath?

Thanks

$ python -V
Python 3.8.0b1+

$ echo "x = 3" > 1.py   

$ echo "(x := 3)" > 2.py

$ astpath ".//Num"

No output.

With Python 3.7 output for same files is

$ astpath ".//Num"
./1.py:0    >	x = 3

hchasestevens / astpath Goto Github PK

astpath's Introduction

astpath

Example usage

Installation

Links

Contacts

astpath's People

Contributors

Stargazers

Watchers

Forkers

astpath's Issues

Recommend Projects

Recommend Topics

Recommend Org