Git Product home page Git Product logo

executing's Introduction

executing

Build Status Coverage Status Supports Python versions 3.5+, including PyPy

This mini-package lets you get information about what a frame is currently doing, particularly the AST node being executed.

Usage

Getting the AST node

import executing

node = executing.Source.executing(frame).node

Then node will be an AST node (from the ast standard library module) or None if the node couldn't be identified (which may happen often and should always be checked).

node will always be the same instance for multiple calls with frames at the same point of execution.

If you have a traceback object, pass it directly to Source.executing() rather than the tb_frame attribute to get the correct node.

Getting the source code of the node

For this you will need to separately install the asttokens library, then obtain an ASTTokens object:

executing.Source.executing(frame).source.asttokens()

or:

executing.Source.for_frame(frame).asttokens()

or use one of the convenience methods:

executing.Source.executing(frame).text()
executing.Source.executing(frame).text_range()

Getting the __qualname__ of the current function

executing.Source.executing(frame).code_qualname()

or:

executing.Source.for_frame(frame).code_qualname(frame.f_code)

The Source class

Everything goes through the Source class. Only one instance of the class is created for each filename. Subclassing it to add more attributes on creation or methods is recommended. The classmethods such as executing will respect this. See the source code and docstrings for more detail.

Installation

pip install executing

If you don't like that you can just copy the file executing.py, there are no dependencies (but of course you won't get updates).

How does it work?

Suppose the frame is executing this line:

self.foo(bar.x)

and in particular it's currently obtaining the attribute self.foo. Looking at the bytecode, specifically frame.f_code.co_code[frame.f_lasti], we can tell that it's loading an attribute, but it's not obvious which one. We can narrow down the statement being executed using frame.f_lineno and find the two ast.Attribute nodes representing self.foo and bar.x. How do we find out which one it is, without recreating the entire compiler in Python?

The trick is to modify the AST slightly for each candidate expression and observe the changes in the bytecode instructions. We change the AST to this:

(self.foo ** 'longuniqueconstant')(bar.x)

and compile it, and the bytecode will be almost the same but there will be two new instructions:

LOAD_CONST 'longuniqueconstant'
BINARY_POWER

and just before that will be a LOAD_ATTR instruction corresponding to self.foo. Seeing that it's in the same position as the original instruction lets us know we've found our match.

Is it reliable?

Yes - if it identifies a node, you can trust that it's identified the correct one. The tests are very thorough - in addition to unit tests which check various situations directly, there are property tests against a large number of files (see the filenames printed in this build) with real code. Specifically, for each file, the tests:

  1. Identify as many nodes as possible from all the bytecode instructions in the file, and assert that they are all distinct
  2. Find all the nodes that should be identifiable, and assert that they were indeed identified somewhere

In other words, it shows that there is a one-to-one mapping between the nodes and the instructions that can be handled. This leaves very little room for a bug to creep in.

Furthermore, executing checks that the instructions compiled from the modified AST exactly match the original code save for a few small known exceptions. This accounts for all the quirks and optimisations in the interpreter.

Which nodes can it identify?

Currently it works in almost all cases for the following ast nodes:

  • Call, e.g. self.foo(bar)
  • Attribute, e.g. point.x
  • Subscript, e.g. lst[1]
  • BinOp, e.g. x + y (doesn't include and and or)
  • UnaryOp, e.g. -n (includes not but only works sometimes)
  • Compare e.g. a < b (not for chains such as 0 < p < 1)

The plan is to extend to more operations in the future.

Projects that use this

My Projects

  • stack_data: Extracts data from stack frames and tracebacks, particularly to display more useful tracebacks than the default. Also uses another related library of mine: pure_eval.
  • futurecoder: Highlights the executing node in tracebacks using executing via stack_data, and provides debugging with snoop.
  • snoop: A feature-rich and convenient debugging library. Uses executing to show the operation which caused an exception and to allow the pp function to display the source of its arguments.
  • heartrate: A simple real time visualisation of the execution of a Python program. Uses executing to highlight currently executing operations, particularly in each frame of the stack trace.
  • sorcery: Dark magic delights in Python. Uses executing to let special callables called spells know where they're being called from.

Projects I've contributed to

  • IPython: Highlights the executing node in tracebacks using executing via stack_data.
  • icecream: ๐Ÿฆ Sweet and creamy print debugging. Uses executing to identify where ic is called and print its arguments.
  • friendly_traceback: Uses stack_data and executing to pinpoint the cause of errors and provide helpful explanations.
  • python-devtools: Uses executing for print debugging similar to icecream.
  • sentry_sdk: Add the integration sentry_sdk.integrations.executingExecutingIntegration() to show the function __qualname__ in each frame in sentry events.
  • varname: Dark magics about variable names in python. Uses executing to find where its various magical functions like varname and nameof are called from.

executing's People

Contributors

15r10nk avatar alexmojaki avatar asafkahlon avatar frenzymadness avatar hugovk avatar inakimalerba avatar kolanich avatar matoro avatar palfrey avatar pwwang avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

executing's Issues

test_main.py::test_global_tester_calls fails with python 3.11.9

test_global_tester_calls fails with python 3.11.9 here, it works fine with 3.12.5. Not sure what's the difference to github's CI, where it seems to pass, too.

โฏ python3.11 -m pytest
=============================================================================== test session starts ===============================================================================
platform linux -- Python 3.11.9, pytest-8.1.1, pluggy-1.5.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/heiko/Quelltexte/git-sources/python/executing
configfile: pyproject.toml
plugins: benchmark-4.0.0, expect-1.1.0, dependency-0.5.1, services-2.2.1, forked-1.6.0, pytest_httpserver-1.0.8, cov-4.1.0, flake8-1.1.1, regressions-2.5.0, datadir-1.5.0, pytest_param_files-0.6.0, sphinx_pytest-0.2.0, timeout-2.3.1, flaky-3.8.1, django-4.8.0, jaraco.test-5.4.0, mock-3.14.0, mypy-plugins-3.1.2, xdist-3.6.1, typeguard-4.3.0, xprocess-1.0.2, requests-mock-1.12.1, socket-0.7.0, pyfakefs-5.6.0, anyio-4.4.0, asyncio-0.23.8, subtests-0.13.1, Faker-28.0.0, trio-0.8.0, respx-0.21.1, hypothesis-6.111.0
asyncio: mode=Mode.STRICT
collected 208 items                                                                                                                                                               

tests/test_ipython.py ..                                                                                                                                                    [  0%]
tests/test_main.py ..................ss.................................................................................................................................... [ 74%]
..........................sssssssssssssssF                                                                                                                                  [ 94%]
tests/test_pytest.py ............                                                                                                                                           [100%]

==================================================================================== FAILURES =====================================================================================
____________________________________________________________________________ test_global_tester_calls _____________________________________________________________________________

    def test_global_tester_calls():
        # tester calls should be tested at global scope
>       from . import global_tester_calls

tests/test_main.py:1547: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/usr/host/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/global_tester_calls.py:5: in <module>
    assert tester([1, 2, 3]) == [1, 2, 3]
tests/utils.py:54: in __call__
    ex = self.get_executing(inspect.currentframe().f_back)
tests/utils.py:42: in get_executing
    return Source.executing(frame)
executing/executing.py:273: in executing
    node_finder = NodeFinder(frame, stmts, tree, lasti, source)
executing/_position_node_finder.py:164: in __init__
    self.known_issues(self.result, instruction)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <executing._position_node_finder.PositionNodeFinder object at 0x7f50d9371010>, node = <ast.Assert object at 0x7f50d937b340>
instruction = Instruction(opname='CALL', opcode=171, arg=1, argval=1, argrepr='', offset=58, starts_line=None, is_jump_target=False, positions=Positions(lineno=5, end_lineno=5, col_offset=0, end_col_offset=37))

    def known_issues(self, node: EnhancedAST, instruction: dis.Instruction) -> None:
        if instruction.opname in ("COMPARE_OP", "IS_OP", "CONTAINS_OP") and isinstance(
            node, types_cmp_issue
        ):
            if isinstance(node, types_cmp_issue_fix):
                # this is a workaround for https://github.com/python/cpython/issues/95921
                # we can fix cases with only on comparison inside the test condition
                #
                # we can not fix cases like:
                # if a<b<c and d<e<f: pass
                # if (a<b<c)!=d!=e: pass
                # because we don't know which comparison caused the problem
    
                comparisons = [
                    n
                    for n in ast.walk(node.test) # type: ignore[attr-defined]
                    if isinstance(n, ast.Compare) and len(n.ops) > 1
                ]
    
                assert_(comparisons, "expected at least one comparison")
    
                if len(comparisons) == 1:
                    node = self.result = cast(EnhancedAST, comparisons[0])
                else:
                    raise KnownIssue(
                        "multiple chain comparison inside %s can not be fixed" % (node)
                    )
    
            else:
                # Comprehension and generators get not fixed for now.
                raise KnownIssue("chain comparison inside %s can not be fixed" % (node))
    
        if (
            sys.version_info[:3] == (3, 11, 1)
            and isinstance(node, ast.Compare)
            and instruction.opname == "CALL"
            and any(isinstance(n, ast.Assert) for n in node_and_parents(node))
        ):
            raise KnownIssue(
                "known bug in 3.11.1 https://github.com/python/cpython/issues/95921"
            )
    
        if isinstance(node, ast.Assert):
            # pytest assigns the position of the assertion to all expressions of the rewritten assertion.
            # All the rewritten expressions get mapped to ast.Assert, which is the wrong ast-node.
            # We don't report this wrong result.
>           raise KnownIssue("assert")
E           executing._exceptions.KnownIssue: assert

executing/_position_node_finder.py:293: KnownIssue
================================================================================ warnings summary =================================================================================
tests/test_main.py::test_small_samples[4851dc1b626a95e97dbe0c53f96099d165b755dd1bd552c6ca771f7bca6d30f5.py]
  /home/heiko/Quelltexte/git-sources/python/executing/tests/small_samples/4851dc1b626a95e97dbe0c53f96099d165b755dd1bd552c6ca771f7bca6d30f5.py:16: SyntaxWarning: "is" with a literal. Did you mean "=="?
    if 1 < 1 > 1 == 1 >= 1 <= 1 != 1 in 1 not in 1 is 1 is not 1:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================= short test summary info =============================================================================
FAILED tests/test_main.py::test_global_tester_calls - executing._exceptions.KnownIssue: assert
============================================================== 1 failed, 190 passed, 17 skipped, 1 warning in 6.09s ===============================================================

Failure with 3.12.5

[   22s] Traceback (most recent call last):
[   22s]   File "/home/abuild/rpmbuild/BUILD/executing-2.0.1/tests/test_main.py", line 604, in test_iter
[   22s]     for i in iter_test(ast.For):
[   22s]              ^^^^^^^^^^^^^^^^^^
[   22s]   File "/home/abuild/rpmbuild/BUILD/executing-2.0.1/tests/test_main.py", line 587, in __iter__
[   22s]     assert isinstance(calling_expression(), self.typ)
[   22s]                       ^^^^^^^^^^^^^^^^^^^^
[   22s]   File "/home/abuild/rpmbuild/BUILD/executing-2.0.1/tests/test_main.py", line 48, in calling_expression
[   22s]     return Source.executing(frame).node
[   22s]            ^^^^^^^^^^^^^^^^^^^^^^^
[   22s]   File "/home/abuild/rpmbuild/BUILD/executing-2.0.1/executing/executing.py", line 273, in executing
[   22s]     node_finder = NodeFinder(frame, stmts, tree, lasti, source)
[   22s]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   22s]   File "/home/abuild/rpmbuild/BUILD/executing-2.0.1/executing/_position_node_finder.py", line 165, in __init__
[   22s]     self.verify(self.result, self.instruction(lasti))
[   22s]   File "/home/abuild/rpmbuild/BUILD/executing-2.0.1/executing/_position_node_finder.py", line 772, in verify
[   22s]     raise VerifierFailure(title, node, instruction)
[   22s] executing._exceptions.VerifierFailure: ast.Call is not created from GET_ITER

The above test works fine with 3.12.4, but gives the following traceback under 3.12.5.

How safe is this to modify AST?

If I am understand README well, this module uses tricks and modifications in AST to detect correct attribute nodes. So how safe are these modifications for runtime and correct execution?
This is actual question since you use executing in icecream, which one looks like just logging tool, but in fact everything goes to AST modifications to inspect variable names, function calls and list/dict elements access.

UPD. Sorry for opening Issues in both executing and icecream, maybe this can be useful for issue/questions archive in both of your projects.

Bug: Incorrect node text when __slots__ and class variables are used (3.11)

Apologies for not providing a self-contained test with executing for the following, as I don't know how.
The following raises a ValueError exception:

class F:
    __slots__ = ["a", "b"]
    a = 1

With Python 3.10 (and prior), the text of the node identified causing the exception is:

class F:

With Python 3.11, we get this:

class F: __slots__ = ["a", "b"] a = 1

This is not a typo: the text of the node identified a single line, with no \n, which is clearly a SyntaxError.

This is the only bug I found when running the unit tests for friendly_traceback with executing 1.1.0 and Python 3.11. Everything still works for me with Python 3.6 to 3.10, although I found that the tests I run take about 2 to 3 times as long as compared with using executing 0.8.3. The slowdown seemed to be already present with executing 1.0.0.

Executing cannot find node inside Jupyter lab/notebooks

Using friendly, I found that excuting could not locate the correct location of a node causing a problem inside Jupyter notebooks (or Jupyter lab) but could do so with IPython. First, here's a screen capture showing the correct result using IPython, with some additional outputs from print statements inserted for this specific report.

image

Next, the same code run within a Jupyter notebook
image

Perhaps this is is caused by the new way that Jupyter runs code, using temporary files instead of using exec() on code.

With the "old" friendly-traceback (before it made use of stack_data), when Jupyter was not using temporary files, I see from screen captures on https://aroberge.github.io/friendly-traceback-docs/docs/html/jupyter.html that the location was correctly picked up.

Failed to detect nodes in ipython/jupyter since v1.1.0

Python 3.9.5 (default, Jun  4 2021, 12:28:51) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sys 
   ...: from executing import Source, __version__ 
   ...:  
   ...:  
   ...: def get_node(): 
   ...:     frame = sys._getframe(1) 
   ...:     return Source.executing(frame).node 
   ...:  
   ...:  
   ...: node = get_node() 
   ...: print(node) 
   ...:
None

In [2]: __version__
Out[2]: '1.1.0'

Used to work with v1.0.0

Python 3.9.5 (default, Jun  4 2021, 12:28:51) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sys 
   ...: from executing import Source, __version__ 
   ...:  
   ...:  
   ...: def get_node(): 
   ...:     frame = sys._getframe(1) 
   ...:     return Source.executing(frame).node 
   ...:  
   ...:  
   ...: node = get_node() 
   ...: print(node) 
   ...:
<ast.Call object at 0x7fb7de6748b0>

In [2]: __version__
Out[2]: '1.0.0'

no source distribution on pypi

Would it be possible to upload the sdist in addition to the wheel file on pypi? Currently the conda-forge distribution does not automatically pick up updates unless the sources are present.

Possible memory leak?

While debugging why icecream has issues with Qt, I discovered that executing.Source.executing keeps some sort of hidden reference to the local variables of the function called. In the code sample below, the issue is noticeable by the fact that method0 and method1 show only one window while method2 shows two windows (one being basically a black screen). This happens because in Qt widgets remain shown while someone is holding a reference to them.

#!/usr/bin/env python3
import sys
import PySide6
from PySide6 import QtWidgets
from PySide6.QtCore import QLibraryInfo, qVersion
from PySide6.QtWidgets import QMainWindow
from PySide6.QtMultimediaWidgets import QVideoWidget

import inspect
import executing

def my_print(*args):
    callFrame = inspect.currentframe().f_back
    executing.Source.executing(callFrame)
    return None

class MyWindow(QMainWindow):
    def __init__(self):
        super().__init__()

    def showEvent(self, evt):
        # If you replace method2 with method1 or method0, no window will appear for QVideoWidget
        self.method2()

    def method0(self):
        self.setWindowTitle('Method 0')
        camera_view = QVideoWidget()
        camera_view.show()

    def method1(self):
        self.setWindowTitle('Method 1')
        camera_view = QVideoWidget()
        print(camera_view)
        camera_view.show()

    def method2(self):
        self.setWindowTitle('Method 2')
        camera_view = QVideoWidget()
        my_print(camera_view)
        camera_view.show()

def main():
    print('Python {}.{}'.format(sys.version_info[0], sys.version_info[1]))
    print(QLibraryInfo.build())
    print(f"PySide6 version: {PySide6.__version__}")

    app = QtWidgets.QApplication(sys.argv)
    window = MyWindow()
    window.show()
    sys.exit(app.exec())

if __name__ == "__main__":
    main()

0.8.3: pytest is failing

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

  • python3 -sBm build -w --no-isolation
  • because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
  • install .whl file in </install/prefix>
  • run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-executing-0.8.3-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-executing-0.8.3-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.13, pytest-7.1.1, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/executing-0.8.3
collected 2 items / 1 error

================================================================================== ERRORS ==================================================================================
___________________________________________________________________ ERROR collecting tests/test_main.py ____________________________________________________________________
executing/executing.py:317: in executing
    args = executing_cache[key]
E   KeyError: (<code object <module> at 0x7f98855a0ea0, file "/home/tkloczko/rpmbuild/BUILD/executing-0.8.3/tests/test_main.py", line 2>, 140293049028256, 418)

During handling of the above exception, another exception occurred:
executing/executing.py:346: in find
    node_finder = NodeFinder(frame, stmts, tree, lasti)
executing/executing.py:636: in __init__
    matching = list(self.matching_nodes(exprs))
executing/executing.py:702: in matching_nodes
    original_index = only(
executing/executing.py:164: in only
    raise NotOneValueFound('Expected one value, found 0')
E   executing.executing.NotOneValueFound: Expected one value, found 0

During handling of the above exception, another exception occurred:
tests/test_main.py:682: in <module>
    assert tester([1, 2, 3]) == [1, 2, 3]
tests/utils.py:40: in __call__
    ex = self.get_executing(inspect.currentframe().f_back)
tests/utils.py:28: in get_executing
    return Source.executing(frame)
executing/executing.py:369: in executing
    args = find(source=cls.for_frame(frame), retry_cache=True)
executing/executing.py:355: in find
    return find(
executing/executing.py:346: in find
    node_finder = NodeFinder(frame, stmts, tree, lasti)
executing/executing.py:636: in __init__
    matching = list(self.matching_nodes(exprs))
executing/executing.py:702: in matching_nodes
    original_index = only(
executing/executing.py:164: in only
    raise NotOneValueFound('Expected one value, found 0')
E   executing.executing.NotOneValueFound: Expected one value, found 0
========================================================================= short test summary info ==========================================================================
ERROR tests/test_main.py - executing.executing.NotOneValueFound: Expected one value, found 0
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================= 1 error in 0.44s =============================================================================

3.11 decorator detection bug

import inspect

import executing

frame = inspect.currentframe()


class Tester:
    def __call__(self, f):
        deco(f)


tester = Tester()


def deco(f):
    assert f.__name__ == 'foo'
    ex = executing.Source.executing(frame)
    print(f"{ex.node = }\n{ex.decorator = }\n")


for d in [deco, tester]:
    @d
    def foo():
        pass

3.10 output:

ex.node = <ast.FunctionDef object at 0x7f9c7e65b1c0>
ex.decorator = <ast.Name object at 0x7f9c7e65a2f0>

ex.node = <ast.FunctionDef object at 0x7f9c7e65b1c0>
ex.decorator = <ast.Name object at 0x7f9c7e65a2f0>

3.11 output:

ex.node = <ast.Name object at 0x7fd9fe5a96f0>
ex.decorator = None

ex.node = <ast.FunctionDef object at 0x7fd9fe5a9780>
ex.decorator = <ast.Name object at 0x7fd9fe5a96f0>

Somehow, the decorator is only detected when it's an object with __call__. When it's just a function, executing sets decorator to None and node to the decorator instead of the function definition. This is REALLY weird.

Is it possible to know whether a frame is from a method call?

For example

def do_func(self):
    frame = sys._getframe()

class A:
    def do_method(self):
        frame = sys._getframe()

Is it possible to know whether frame is inside a regular function call or method call? From https://stackoverflow.com/q/2203424/2142577 it seems this is no way except relying on the convention that self is the first argument of method signature. Do we have a better way?

I also tried using gc.get_referrers(frame.f_code)), but it turns out the code is referrenced by the unbound function, no matter whether it is actually a method.

Improve tests samples

I would like to improve the quality of the test samples, because I had some issues with them during the implementation of #31. I would like to get some feedback on this Idea.

The problems

  • They take long to execute and are therefore only executed with EXECUTING_SLOW_TESTS=1 enabled.
  • They cover by far not all kinds of ast/bytecode python could generate
  • It is not useful to add more to them because they would run even longer
  • They are also a major slowdown for the build pipeline.

I tested the #31 branch with ~400000 python files which I collected in different ways from github/pypi and found a lot of issues there which where not covered by any existing test.
I got the idea, at the end of #31, that it might be useful to add minimized versions of this files which contain only the problematic code to the samples. However this idea came to late and I finished #31 without adding tests to the samples.

But there is maybe another way:

  • Lets say we use a new sample folder ... lets call it tests/issues/ for now which is initial empty.

  • It should be possible to use mutmut to generate a code change inside executing/_position_node_finder.py which produces no problem with an code sample in tests/issues/ (this the case for the first run and gets more interesting later).

  • We can run then over a much bigger set of files (400000 if it is necessary) to find a file which contains some code which raises an error for the change mutmut did in executing.

  • This file can be minimized to the exact code which leads to the problem. Adding this minimized file to tests/issues would show mutmut that this issue is covered. mutmut would search for another issue in the next run.

This process could be completely automated and generate a new set of small and diverse sample files.
Every file would contain some special case which should be handled by executing.

It should be almost impossible to change the implementation of executing in a bad way (this is what mutmut tries really hard all the time), because there is a new issue file for almost every case.

I know that this will probably not work perfect, but I think it is worth to try it.

Some questions ...

I have started to work on the minimizer for the python source, because I could not find any thing I could use, but does anyone knows a tool I could use?
It should take a python source file as input and remove code from it as long as some condition holds.
In our case as long as the source file produces the issue.

I would like to know about any ideas or questions to this approach.

Supporting `STORE_ATTR` and `STORE_SUBSCR`

Any plans to support those two operations? E.g. a.x = 1 and a['x'] = 1 that invoke __setattr__() and __setitem__().

Or are you accepting a PR? I can give it a try.

Here are some key points:

  1. Set typ = ast.Assign when op_name is either STORE_ATTR or STORE_SUBSCR in NodeFinder.__init__() (Is extra_filter needed?)
  2. In NodeFinder.matching_nodes(), if expr is an Assign, yield it immediately (don't insert the sentinel). The reason, I think, is that we can't have two assignments in one statement.

Had a prototype and it works, but need to confirm the details with you.

Related: pwwang/python-varname#78

2.0.0: pytest is failing in `tests/test_ipython.py::test_two_statement_lookups` unit

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

  • python3 -sBm build -w --no-isolation
  • because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
  • install .whl file in </install/prefix> using 'installer` module
  • run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
  • build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-executing-2.0.0-3.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-executing-2.0.0-3.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
============================= test session starts ==============================
platform linux -- Python 3.8.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/tkloczko/rpmbuild/BUILD/executing-2.0.0
collected 198 items

tests/test_ipython.py .F                                                 [  1%]
tests/test_main.py ..................................................x.. [ 27%]
..............x....x....x............................................... [ 64%]
........................................x..xsssssssssssssss.             [ 94%]
tests/test_pytest.py ...........                                         [100%]

=================================== FAILURES ===================================
__________________________ test_two_statement_lookups __________________________

    def test_two_statement_lookups():
        p = run(test_function_code + "test();test()")
>       assert "test failure" in p
E       AssertionError: assert 'test failure' in ''

tests/test_ipython.py:41: AssertionError
----------------------------- Captured stdout call -----------------------------

----------------------------- Captured stderr call -----------------------------
/usr/bin/python3: No module named IPython
=========================== short test summary info ============================
SKIPPED [14] tests/test_main.py:754: These tests are very slow, enable them explicitly
SKIPPED [1] tests/test_main.py:770: These tests are very slow, enable them explicitly
XFAIL tests/test_main.py::test_small_samples[206e0609ff0589a0a32422ee902f09156af91746e27157c32c9595d12072f92a.py] - reason: SentinelNodeFinder does not find some of the nodes (maybe a bug)
XFAIL tests/test_main.py::test_small_samples[42a37b8a823eb2e510b967332661afd679c82c60b7177b992a47c16d81117c8a.py] - reason: SentinelNodeFinder does not find some of the nodes (maybe a bug)
XFAIL tests/test_main.py::test_small_samples[4851dc1b626a95e97dbe0c53f96099d165b755dd1bd552c6ca771f7bca6d30f5.py] - reason: SentinelNodeFinder does not find some of the nodes (maybe a bug)
XFAIL tests/test_main.py::test_small_samples[508ccd0dcac13ecee6f0cea939b73ba5319c780ddbb6c496be96fe5614871d4a.py] - reason: SentinelNodeFinder does not find some of the nodes (maybe a bug)
XFAIL tests/test_main.py::test_small_samples[fc6eb521024986baa84af2634f638e40af090be4aa70ab3c22f3d022e8068228.py] - reason: SentinelNodeFinder does not find some of the nodes (maybe a bug)
XFAIL tests/test_main.py::test_small_samples[load_deref.py] - reason: SentinelNodeFinder does not find some of the nodes (maybe a bug)
FAILED tests/test_ipython.py::test_two_statement_lookups - AssertionError: as...
============ 1 failed, 176 passed, 15 skipped, 6 xfailed in 22.52s =============
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de886006-989d-4184-bc6d-b571bc244097/test_rmtree_errorhandler_rerai0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_rmtree_errorhandler_rerai0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de886006-989d-4184-bc6d-b571bc244097/test_rmtree_errorhandler_reado0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_rmtree_errorhandler_reado0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de886006-989d-4184-bc6d-b571bc244097/test_safe_delete_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_delete_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de886006-989d-4184-bc6d-b571bc244097/test_safe_set_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_set_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de886006-989d-4184-bc6d-b571bc244097/test_safe_get_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_get_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-de886006-989d-4184-bc6d-b571bc244097
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-tkloczko/garbage-de886006-989d-4184-bc6d-b571bc244097'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-07863f54-4301-46f4-b0f5-ae75827ff0eb/test_rmtree_errorhandler_rerai0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_rmtree_errorhandler_rerai0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-07863f54-4301-46f4-b0f5-ae75827ff0eb/test_rmtree_errorhandler_reado0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_rmtree_errorhandler_reado0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-07863f54-4301-46f4-b0f5-ae75827ff0eb/test_safe_delete_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_delete_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-07863f54-4301-46f4-b0f5-ae75827ff0eb/test_safe_set_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_set_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-07863f54-4301-46f4-b0f5-ae75827ff0eb/test_safe_get_no_perms0
<class 'OSError'>: [Errno 39] Directory not empty: 'test_safe_get_no_perms0'
  warnings.warn(
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:95: PytestWarning: (rm_rf) error removing /tmp/pytest-of-tkloczko/garbage-07863f54-4301-46f4-b0f5-ae75827ff0eb
<class 'OSError'>: [Errno 39] Directory not empty: '/tmp/pytest-of-tkloczko/garbage-07863f54-4301-46f4-b0f5-ae75827ff0eb'
  warnings.warn(

Here is list of installed modules in build env

Package                       Version
----------------------------- -------
alabaster                     0.7.13
asttokens                     2.2.1
Babel                         2.13.1
build                         1.0.0
charset-normalizer            3.3.1
cppclean                      0.13
distro                        1.8.0
docutils                      0.20.1
exceptiongroup                1.1.3
gpg                           1.22.0
idna                          3.4
imagesize                     1.4.1
importlib-metadata            6.8.0
iniconfig                     2.0.0
installer                     0.7.0
Jinja2                        3.1.2
MarkupSafe                    2.1.3
packaging                     23.2
pluggy                        1.3.0
Pygments                      2.16.1
pyproject_hooks               1.0.0
pytest                        7.4.3
python-dateutil               2.8.2
pytz                          2023.3
requests                      2.31.0
setuptools                    68.2.2
setuptools-scm                8.0.4
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        7.1.2
sphinxcontrib-applehelp       1.0.4
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.4
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.9
tomli                         2.0.1
typing_extensions             4.8.0
urllib3                       1.26.18
wheel                         0.41.2
zipp                          3.17.0

parent pointers in AST nodes hurt deepcopy performance

Obligatory "I'm a huge fan of your work".

Background: I maintain a library called ipyflow, which uses another library I maintain called pyccolo. I noticed that on ipython >= 8.0, which uses executing for better stack traces, ipyflow would have really bad performance regressions the first time after a cell throws some exception.

Eventually I traced it to pyccolo's use of copy.deepcopy in a few places -- performance was bad because the parent pointers that executing added to AST nodes were causing deepcopy to do a lot of extra unnecessary work.

I ended up working around it on the pyccolo side, but I figured you may be interested in this for other libraries that may want to use executing and get surprised when deepcopy has bad perf. The way we maintain deepcopy-ability in pyccolo is to maintain a mapping from id(node) to parent for parent pointers, which avoids setting an attribute on the AST node directly.

Thanks for this great library!

Release missing for 0.4.3

Dear Maintainer,

The package version number was bumped on 943ad1a but no release was tagged on GitHub.

Could you please create it?

Thanks !

Various tests fail with Python 3.12

Hello, in Fedora we are rebuilding all Python packages with the upcoming Python 3.12, currently 7th alpha.
I'd like to ask for help with the following test failures. Thank you.

=================================== FAILURES ===================================
___________________________ TestStuff.test_decorator ___________________________

self = <tests.test_main.TestStuff testMethod=test_decorator>

    def test_decorator(self):
        @empty_decorator  # 0
        @decorator_with_args(tester('123'), x=int())  # 1
        @tester(list(tuple([1, 2])))  # 2!
        @tester(  # 3!
            list(
                tuple(
                    [3, 4])),
            )
        @empty_decorator  # 4
        @decorator_with_args(  # 5
            str(),
            x=int())
        @tester(list(tuple([5, 6])))  # 6!
>       @tester(list(tuple([7, 8])))  # 7!

tests/test_main.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/utils.py:63: in __call__
    self.check(call.args[0], arg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tests.utils.Tester object at 0x7f702c74c050>
node = <ast.Call object at 0x7f702c7f22d0>
value = <function TestStuff.test_decorator.<locals>.foo at 0x7f702c62b740>

    def check(self, node, value):
        frame = inspect.currentframe().f_back.f_back
        result = eval(
            compile(ast.Expression(node), frame.f_code.co_filename, 'eval'),
            frame.f_globals,
            frame.f_locals,
        )
>       assert result == value, (result, value)
E       AssertionError: ([7, 8], <function TestStuff.test_decorator.<locals>.foo at 0x7f702c62b740>)

tests/utils.py:51: AssertionError
__________________ TestStuff.test_decorator_cache_instruction __________________

self = <tests.test_main.TestStuff testMethod=test_decorator_cache_instruction>

    def test_decorator_cache_instruction(self):
        frame = inspect.currentframe()
    
        def deco(f):
            assert f.__name__ == "foo"
            ex = Source.executing(frame)
            assert isinstance(ex.node, ast.FunctionDef)
            assert isinstance(ex.decorator, ast.Name)
    
>       @deco

tests/test_main.py:587: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_main.py:583: in deco
    ex = Source.executing(frame)
executing/executing.py:368: in executing
    node_finder = NodeFinder(frame, stmts, tree, lasti, source)
executing/_position_node_finder.py:158: in __init__
    self.verify(self.result, self.instruction(lasti))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <executing._position_node_finder.PositionNodeFinder object at 0x7f702c3f75c0>
node = <ast.Name object at 0x7f702c5875d0>
instruction = Instruction(opname='CALL', opcode=171, arg=0, argval=0, argrepr='', offset=62, starts_line=587, is_jump_target=False, positions=Positions(lineno=587, end_lineno=587, col_offset=9, end_col_offset=13))

    def verify(self, node: EnhancedAST, instruction: dis.Instruction) -> None:
        """
        checks if this node could gererate this instruction
        """
    
        op_name = instruction.opname
        extra_filter: Callable[[EnhancedAST], bool] = lambda e: True
        ctx: Type = type(None)
    
        def inst_match(opnames: Union[str, Sequence[str]], **kwargs: Any) -> bool:
            """
            match instruction
    
            Parameters:
                opnames: (str|Seq[str]): inst.opname has to be equal to or in `opname`
                **kwargs: every arg has to match inst.arg
    
            Returns:
                True if all conditions match the instruction
    
            """
    
            if isinstance(opnames, str):
                opnames = [opnames]
            return instruction.opname in opnames and kwargs == {
                k: getattr(instruction, k) for k in kwargs
            }
    
        def node_match(node_type: Union[Type, Tuple[Type, ...]], **kwargs: Any) -> bool:
            """
            match the ast-node
    
            Parameters:
                node_type: type of the node
                **kwargs: every `arg` has to be equal `node.arg`
                        or `node.arg` has to be an instance of `arg` if it is a type.
            """
            return isinstance(node, node_type) and all(
                isinstance(getattr(node, k), v)
                if isinstance(v, type)
                else getattr(node, k) == v
                for k, v in kwargs.items()
            )
    
        if op_name == "CACHE":
            return
    
        if inst_match("CALL") and node_match((ast.With, ast.AsyncWith)):
            # call to context.__exit__
            return
    
        if inst_match(("CALL", "LOAD_FAST")) and node_match(
            (ast.ListComp, ast.GeneratorExp, ast.SetComp, ast.DictComp)
        ):
            # call to the generator function
            return
    
        if inst_match(("CALL", "CALL_FUNCTION_EX")) and node_match(
            (ast.ClassDef, ast.Call)
        ):
            return
    
        if inst_match(("COMPARE_OP", "IS_OP", "CONTAINS_OP")) and node_match(
            ast.Compare
        ):
            return
    
        if inst_match("LOAD_NAME", argval="__annotations__") and node_match(
            ast.AnnAssign
        ):
            return
    
        if (
            (
                inst_match("LOAD_METHOD", argval="join")
                or inst_match(("CALL", "BUILD_STRING"))
            )
            and node_match(ast.BinOp, left=ast.Constant, op=ast.Mod)
            and isinstance(cast(ast.Constant, cast(ast.BinOp, node).left).value, str)
        ):
            # "..."%(...) uses "".join
            return
    
        if inst_match("STORE_SUBSCR") and node_match(ast.AnnAssign):
            # data: int
            return
    
        if self.is_except_cleanup(instruction, node):
            return
    
        if inst_match(("DELETE_NAME", "DELETE_FAST")) and node_match(
            ast.Name, id=instruction.argval, ctx=ast.Del
        ):
            return
    
        if inst_match("BUILD_STRING") and (
            node_match(ast.JoinedStr) or node_match(ast.BinOp, op=ast.Mod)
        ):
            return
    
        if inst_match(("BEFORE_WITH","WITH_EXCEPT_START")) and node_match(ast.With):
            return
    
        if inst_match(("STORE_NAME", "STORE_GLOBAL"), argval="__doc__") and node_match(
            ast.Constant
        ):
            # store docstrings
            return
    
        if (
            inst_match(("STORE_NAME", "STORE_FAST", "STORE_GLOBAL", "STORE_DEREF"))
            and node_match(ast.ExceptHandler)
            and instruction.argval == mangled_name(node)
        ):
            # store exception in variable
            return
    
        if (
            inst_match(("STORE_NAME", "STORE_FAST", "STORE_DEREF", "STORE_GLOBAL"))
            and node_match((ast.Import, ast.ImportFrom))
            and any(mangled_name(cast(EnhancedAST, alias)) == instruction.argval for alias in cast(ast.Import, node).names)
        ):
            # store imported module in variable
            return
    
        if (
            inst_match(("STORE_FAST", "STORE_DEREF", "STORE_NAME", "STORE_GLOBAL"))
            and (
                node_match((ast.FunctionDef, ast.ClassDef, ast.AsyncFunctionDef))
                or node_match(
                    ast.Name,
                    ctx=ast.Store,
                )
            )
            and instruction.argval == mangled_name(node)
        ):
            return
    
        if False:
            # TODO: match expressions are not supported for now
            if inst_match(("STORE_FAST", "STORE_NAME")) and node_match(
                ast.MatchAs, name=instruction.argval
            ):
                return
    
            if inst_match("COMPARE_OP", argval="==") and node_match(ast.MatchSequence):
                return
    
            if inst_match("COMPARE_OP", argval="==") and node_match(ast.MatchValue):
                return
    
        if inst_match("BINARY_OP") and node_match(
            ast.AugAssign, op=op_type_map[instruction.argrepr.removesuffix("=")]
        ):
            # a+=5
            return
    
        if node_match(ast.Attribute, ctx=ast.Del) and inst_match(
            "DELETE_ATTR", argval=mangled_name(node)
        ):
            return
    
        if inst_match(("JUMP_IF_TRUE_OR_POP", "JUMP_IF_FALSE_OR_POP")) and node_match(
            ast.BoolOp
        ):
            # and/or short circuit
            return
    
        if inst_match("DELETE_SUBSCR") and node_match(ast.Subscript, ctx=ast.Del):
            return
    
        if node_match(ast.Name, ctx=ast.Load) and inst_match(
            ("LOAD_NAME", "LOAD_FAST", "LOAD_GLOBAL"), argval=mangled_name(node)
        ):
            return
    
        if node_match(ast.Name, ctx=ast.Del) and inst_match(
            ("DELETE_NAME", "DELETE_GLOBAL"), argval=mangled_name(node)
        ):
            return
    
        # old verifier
    
        typ: Type = type(None)
        op_type: Type = type(None)
    
        if op_name.startswith(("BINARY_SUBSCR", "SLICE+")):
            typ = ast.Subscript
            ctx = ast.Load
        elif op_name.startswith("BINARY_"):
            typ = ast.BinOp
            op_type = op_type_map[instruction.argrepr]
            extra_filter = lambda e: isinstance(cast(ast.BinOp, e).op, op_type)
        elif op_name.startswith("UNARY_"):
            typ = ast.UnaryOp
            op_type = dict(
                UNARY_POSITIVE=ast.UAdd,
                UNARY_NEGATIVE=ast.USub,
                UNARY_NOT=ast.Not,
                UNARY_INVERT=ast.Invert,
            )[op_name]
            extra_filter = lambda e: isinstance(cast(ast.UnaryOp, e).op, op_type)
        elif op_name in ("LOAD_ATTR", "LOAD_METHOD", "LOOKUP_METHOD"):
            typ = ast.Attribute
            ctx = ast.Load
            extra_filter = lambda e: mangled_name(e) == instruction.argval
        elif op_name in (
            "LOAD_NAME",
            "LOAD_GLOBAL",
            "LOAD_FAST",
            "LOAD_DEREF",
            "LOAD_CLASSDEREF",
        ):
            typ = ast.Name
            ctx = ast.Load
            extra_filter = lambda e: cast(ast.Name, e).id == instruction.argval
        elif op_name in ("COMPARE_OP", "IS_OP", "CONTAINS_OP"):
            typ = ast.Compare
            extra_filter = lambda e: len(cast(ast.Compare, e).ops) == 1
        elif op_name.startswith(("STORE_SLICE", "STORE_SUBSCR")):
            ctx = ast.Store
            typ = ast.Subscript
        elif op_name.startswith("STORE_ATTR"):
            ctx = ast.Store
            typ = ast.Attribute
            extra_filter = lambda e: mangled_name(e) == instruction.argval
    
        node_ctx = getattr(node, "ctx", None)
    
        ctx_match = (
            ctx is not type(None)
            or not hasattr(node, "ctx")
            or isinstance(node_ctx, ctx)
        )
    
        # check for old verifier
        if isinstance(node, typ) and ctx_match and extra_filter(node):
            return
    
        # generate error
    
        title = "ast.%s is not created from %s" % (
            type(node).__name__,
            instruction.opname,
        )
    
>       raise VerifierFailure(title, node, instruction)
E       executing._exceptions.VerifierFailure: ast.Name is not created from CALL

executing/_position_node_finder.py:545: VerifierFailure
_____________________________ TestStuff.test_names _____________________________

self = <tests.test_main.TestStuff testMethod=test_names>

    @contextlib.contextmanager
    def assert_name_error(self):
        try:
>           yield

tests/test_main.py:536: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tests.test_main.TestStuff testMethod=test_names>

    def test_names(self):
        with self.assert_name_error():
            self, completely_nonexistent  # noqa
    
        with self.assert_name_error():
            self, global_never_defined  # noqa
    
        with self.assert_name_error():
>           self, local_not_defined_yet  # noqa
E           UnboundLocalError: cannot access local variable 'local_not_defined_yet' where it is not associated with a value

tests/test_main.py:554: UnboundLocalError

During handling of the above exception, another exception occurred:

self = <tests.test_main.TestStuff testMethod=test_names>

    def test_names(self):
        with self.assert_name_error():
            self, completely_nonexistent  # noqa
    
        with self.assert_name_error():
            self, global_never_defined  # noqa
    
>       with self.assert_name_error():

tests/test_main.py:553: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.12/contextlib.py:155: in __exit__
    self.gen.throw(value)
tests/test_main.py:539: in assert_name_error
    ex = Source.executing(tb.tb_next)
executing/executing.py:368: in executing
    node_finder = NodeFinder(frame, stmts, tree, lasti, source)
executing/_position_node_finder.py:158: in __init__
    self.verify(self.result, self.instruction(lasti))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <executing._position_node_finder.PositionNodeFinder object at 0x7f7029434e30>
node = <ast.Name object at 0x7f702c781450>
instruction = Instruction(opname='LOAD_FAST_CHECK', opcode=127, arg=1, argval='local_not_defined_yet', argrepr='local_not_defined_ye...rts_line=None, is_jump_target=False, positions=Positions(lineno=554, end_lineno=554, col_offset=18, end_col_offset=39))

    def verify(self, node: EnhancedAST, instruction: dis.Instruction) -> None:
        """
        checks if this node could gererate this instruction
        """
    
        op_name = instruction.opname
        extra_filter: Callable[[EnhancedAST], bool] = lambda e: True
        ctx: Type = type(None)
    
        def inst_match(opnames: Union[str, Sequence[str]], **kwargs: Any) -> bool:
            """
            match instruction
    
            Parameters:
                opnames: (str|Seq[str]): inst.opname has to be equal to or in `opname`
                **kwargs: every arg has to match inst.arg
    
            Returns:
                True if all conditions match the instruction
    
            """
    
            if isinstance(opnames, str):
                opnames = [opnames]
            return instruction.opname in opnames and kwargs == {
                k: getattr(instruction, k) for k in kwargs
            }
    
        def node_match(node_type: Union[Type, Tuple[Type, ...]], **kwargs: Any) -> bool:
            """
            match the ast-node
    
            Parameters:
                node_type: type of the node
                **kwargs: every `arg` has to be equal `node.arg`
                        or `node.arg` has to be an instance of `arg` if it is a type.
            """
            return isinstance(node, node_type) and all(
                isinstance(getattr(node, k), v)
                if isinstance(v, type)
                else getattr(node, k) == v
                for k, v in kwargs.items()
            )
    
        if op_name == "CACHE":
            return
    
        if inst_match("CALL") and node_match((ast.With, ast.AsyncWith)):
            # call to context.__exit__
            return
    
        if inst_match(("CALL", "LOAD_FAST")) and node_match(
            (ast.ListComp, ast.GeneratorExp, ast.SetComp, ast.DictComp)
        ):
            # call to the generator function
            return
    
        if inst_match(("CALL", "CALL_FUNCTION_EX")) and node_match(
            (ast.ClassDef, ast.Call)
        ):
            return
    
        if inst_match(("COMPARE_OP", "IS_OP", "CONTAINS_OP")) and node_match(
            ast.Compare
        ):
            return
    
        if inst_match("LOAD_NAME", argval="__annotations__") and node_match(
            ast.AnnAssign
        ):
            return
    
        if (
            (
                inst_match("LOAD_METHOD", argval="join")
                or inst_match(("CALL", "BUILD_STRING"))
            )
            and node_match(ast.BinOp, left=ast.Constant, op=ast.Mod)
            and isinstance(cast(ast.Constant, cast(ast.BinOp, node).left).value, str)
        ):
            # "..."%(...) uses "".join
            return
    
        if inst_match("STORE_SUBSCR") and node_match(ast.AnnAssign):
            # data: int
            return
    
        if self.is_except_cleanup(instruction, node):
            return
    
        if inst_match(("DELETE_NAME", "DELETE_FAST")) and node_match(
            ast.Name, id=instruction.argval, ctx=ast.Del
        ):
            return
    
        if inst_match("BUILD_STRING") and (
            node_match(ast.JoinedStr) or node_match(ast.BinOp, op=ast.Mod)
        ):
            return
    
        if inst_match(("BEFORE_WITH","WITH_EXCEPT_START")) and node_match(ast.With):
            return
    
        if inst_match(("STORE_NAME", "STORE_GLOBAL"), argval="__doc__") and node_match(
            ast.Constant
        ):
            # store docstrings
            return
    
        if (
            inst_match(("STORE_NAME", "STORE_FAST", "STORE_GLOBAL", "STORE_DEREF"))
            and node_match(ast.ExceptHandler)
            and instruction.argval == mangled_name(node)
        ):
            # store exception in variable
            return
    
        if (
            inst_match(("STORE_NAME", "STORE_FAST", "STORE_DEREF", "STORE_GLOBAL"))
            and node_match((ast.Import, ast.ImportFrom))
            and any(mangled_name(cast(EnhancedAST, alias)) == instruction.argval for alias in cast(ast.Import, node).names)
        ):
            # store imported module in variable
            return
    
        if (
            inst_match(("STORE_FAST", "STORE_DEREF", "STORE_NAME", "STORE_GLOBAL"))
            and (
                node_match((ast.FunctionDef, ast.ClassDef, ast.AsyncFunctionDef))
                or node_match(
                    ast.Name,
                    ctx=ast.Store,
                )
            )
            and instruction.argval == mangled_name(node)
        ):
            return
    
        if False:
            # TODO: match expressions are not supported for now
            if inst_match(("STORE_FAST", "STORE_NAME")) and node_match(
                ast.MatchAs, name=instruction.argval
            ):
                return
    
            if inst_match("COMPARE_OP", argval="==") and node_match(ast.MatchSequence):
                return
    
            if inst_match("COMPARE_OP", argval="==") and node_match(ast.MatchValue):
                return
    
        if inst_match("BINARY_OP") and node_match(
            ast.AugAssign, op=op_type_map[instruction.argrepr.removesuffix("=")]
        ):
            # a+=5
            return
    
        if node_match(ast.Attribute, ctx=ast.Del) and inst_match(
            "DELETE_ATTR", argval=mangled_name(node)
        ):
            return
    
        if inst_match(("JUMP_IF_TRUE_OR_POP", "JUMP_IF_FALSE_OR_POP")) and node_match(
            ast.BoolOp
        ):
            # and/or short circuit
            return
    
        if inst_match("DELETE_SUBSCR") and node_match(ast.Subscript, ctx=ast.Del):
            return
    
        if node_match(ast.Name, ctx=ast.Load) and inst_match(
            ("LOAD_NAME", "LOAD_FAST", "LOAD_GLOBAL"), argval=mangled_name(node)
        ):
            return
    
        if node_match(ast.Name, ctx=ast.Del) and inst_match(
            ("DELETE_NAME", "DELETE_GLOBAL"), argval=mangled_name(node)
        ):
            return
    
        # old verifier
    
        typ: Type = type(None)
        op_type: Type = type(None)
    
        if op_name.startswith(("BINARY_SUBSCR", "SLICE+")):
            typ = ast.Subscript
            ctx = ast.Load
        elif op_name.startswith("BINARY_"):
            typ = ast.BinOp
            op_type = op_type_map[instruction.argrepr]
            extra_filter = lambda e: isinstance(cast(ast.BinOp, e).op, op_type)
        elif op_name.startswith("UNARY_"):
            typ = ast.UnaryOp
            op_type = dict(
                UNARY_POSITIVE=ast.UAdd,
                UNARY_NEGATIVE=ast.USub,
                UNARY_NOT=ast.Not,
                UNARY_INVERT=ast.Invert,
            )[op_name]
            extra_filter = lambda e: isinstance(cast(ast.UnaryOp, e).op, op_type)
        elif op_name in ("LOAD_ATTR", "LOAD_METHOD", "LOOKUP_METHOD"):
            typ = ast.Attribute
            ctx = ast.Load
            extra_filter = lambda e: mangled_name(e) == instruction.argval
        elif op_name in (
            "LOAD_NAME",
            "LOAD_GLOBAL",
            "LOAD_FAST",
            "LOAD_DEREF",
            "LOAD_CLASSDEREF",
        ):
            typ = ast.Name
            ctx = ast.Load
            extra_filter = lambda e: cast(ast.Name, e).id == instruction.argval
        elif op_name in ("COMPARE_OP", "IS_OP", "CONTAINS_OP"):
            typ = ast.Compare
            extra_filter = lambda e: len(cast(ast.Compare, e).ops) == 1
        elif op_name.startswith(("STORE_SLICE", "STORE_SUBSCR")):
            ctx = ast.Store
            typ = ast.Subscript
        elif op_name.startswith("STORE_ATTR"):
            ctx = ast.Store
            typ = ast.Attribute
            extra_filter = lambda e: mangled_name(e) == instruction.argval
    
        node_ctx = getattr(node, "ctx", None)
    
        ctx_match = (
            ctx is not type(None)
            or not hasattr(node, "ctx")
            or isinstance(node_ctx, ctx)
        )
    
        # check for old verifier
        if isinstance(node, typ) and ctx_match and extra_filter(node):
            return
    
        # generate error
    
        title = "ast.%s is not created from %s" % (
            type(node).__name__,
            instruction.opname,
        )
    
>       raise VerifierFailure(title, node, instruction)
E       executing._exceptions.VerifierFailure: ast.Name is not created from LOAD_FAST_CHECK

executing/_position_node_finder.py:545: VerifierFailure
___________________________ test_global_tester_calls ___________________________

    def test_global_tester_calls():
        # tester calls should be tested at global scope
>       from . import global_tester_calls

tests/test_main.py:1397: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/global_tester_calls.py:13: in <module>
    assert -tester is +tester is ~tester is tester
tests/utils.py:131: in __invert__
    node = self.get_node(ast.UnaryOp)
tests/utils.py:35: in get_node
    ex = self.get_executing(inspect.currentframe().f_back.f_back)
tests/utils.py:42: in get_executing
    return Source.executing(frame)
executing/executing.py:368: in executing
    node_finder = NodeFinder(frame, stmts, tree, lasti, source)
executing/_position_node_finder.py:158: in __init__
    self.verify(self.result, self.instruction(lasti))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <executing._position_node_finder.PositionNodeFinder object at 0x7f702c84ede0>
node = <ast.UnaryOp object at 0x7f7029326590>
instruction = Instruction(opname='CALL_INTRINSIC_1', opcode=173, arg=5, argval=5, argrepr='', offset=178, starts_line=None, is_jump_target=False, positions=Positions(lineno=13, end_lineno=13, col_offset=18, end_col_offset=25))

    def verify(self, node: EnhancedAST, instruction: dis.Instruction) -> None:
        """
        checks if this node could gererate this instruction
        """
    
        op_name = instruction.opname
        extra_filter: Callable[[EnhancedAST], bool] = lambda e: True
        ctx: Type = type(None)
    
        def inst_match(opnames: Union[str, Sequence[str]], **kwargs: Any) -> bool:
            """
            match instruction
    
            Parameters:
                opnames: (str|Seq[str]): inst.opname has to be equal to or in `opname`
                **kwargs: every arg has to match inst.arg
    
            Returns:
                True if all conditions match the instruction
    
            """
    
            if isinstance(opnames, str):
                opnames = [opnames]
            return instruction.opname in opnames and kwargs == {
                k: getattr(instruction, k) for k in kwargs
            }
    
        def node_match(node_type: Union[Type, Tuple[Type, ...]], **kwargs: Any) -> bool:
            """
            match the ast-node
    
            Parameters:
                node_type: type of the node
                **kwargs: every `arg` has to be equal `node.arg`
                        or `node.arg` has to be an instance of `arg` if it is a type.
            """
            return isinstance(node, node_type) and all(
                isinstance(getattr(node, k), v)
                if isinstance(v, type)
                else getattr(node, k) == v
                for k, v in kwargs.items()
            )
    
        if op_name == "CACHE":
            return
    
        if inst_match("CALL") and node_match((ast.With, ast.AsyncWith)):
            # call to context.__exit__
            return
    
        if inst_match(("CALL", "LOAD_FAST")) and node_match(
            (ast.ListComp, ast.GeneratorExp, ast.SetComp, ast.DictComp)
        ):
            # call to the generator function
            return
    
        if inst_match(("CALL", "CALL_FUNCTION_EX")) and node_match(
            (ast.ClassDef, ast.Call)
        ):
            return
    
        if inst_match(("COMPARE_OP", "IS_OP", "CONTAINS_OP")) and node_match(
            ast.Compare
        ):
            return
    
        if inst_match("LOAD_NAME", argval="__annotations__") and node_match(
            ast.AnnAssign
        ):
            return
    
        if (
            (
                inst_match("LOAD_METHOD", argval="join")
                or inst_match(("CALL", "BUILD_STRING"))
            )
            and node_match(ast.BinOp, left=ast.Constant, op=ast.Mod)
            and isinstance(cast(ast.Constant, cast(ast.BinOp, node).left).value, str)
        ):
            # "..."%(...) uses "".join
            return
    
        if inst_match("STORE_SUBSCR") and node_match(ast.AnnAssign):
            # data: int
            return
    
        if self.is_except_cleanup(instruction, node):
            return
    
        if inst_match(("DELETE_NAME", "DELETE_FAST")) and node_match(
            ast.Name, id=instruction.argval, ctx=ast.Del
        ):
            return
    
        if inst_match("BUILD_STRING") and (
            node_match(ast.JoinedStr) or node_match(ast.BinOp, op=ast.Mod)
        ):
            return
    
        if inst_match(("BEFORE_WITH","WITH_EXCEPT_START")) and node_match(ast.With):
            return
    
        if inst_match(("STORE_NAME", "STORE_GLOBAL"), argval="__doc__") and node_match(
            ast.Constant
        ):
            # store docstrings
            return
    
        if (
            inst_match(("STORE_NAME", "STORE_FAST", "STORE_GLOBAL", "STORE_DEREF"))
            and node_match(ast.ExceptHandler)
            and instruction.argval == mangled_name(node)
        ):
            # store exception in variable
            return
    
        if (
            inst_match(("STORE_NAME", "STORE_FAST", "STORE_DEREF", "STORE_GLOBAL"))
            and node_match((ast.Import, ast.ImportFrom))
            and any(mangled_name(cast(EnhancedAST, alias)) == instruction.argval for alias in cast(ast.Import, node).names)
        ):
            # store imported module in variable
            return
    
        if (
            inst_match(("STORE_FAST", "STORE_DEREF", "STORE_NAME", "STORE_GLOBAL"))
            and (
                node_match((ast.FunctionDef, ast.ClassDef, ast.AsyncFunctionDef))
                or node_match(
                    ast.Name,
                    ctx=ast.Store,
                )
            )
            and instruction.argval == mangled_name(node)
        ):
            return
    
        if False:
            # TODO: match expressions are not supported for now
            if inst_match(("STORE_FAST", "STORE_NAME")) and node_match(
                ast.MatchAs, name=instruction.argval
            ):
                return
    
            if inst_match("COMPARE_OP", argval="==") and node_match(ast.MatchSequence):
                return
    
            if inst_match("COMPARE_OP", argval="==") and node_match(ast.MatchValue):
                return
    
        if inst_match("BINARY_OP") and node_match(
            ast.AugAssign, op=op_type_map[instruction.argrepr.removesuffix("=")]
        ):
            # a+=5
            return
    
        if node_match(ast.Attribute, ctx=ast.Del) and inst_match(
            "DELETE_ATTR", argval=mangled_name(node)
        ):
            return
    
        if inst_match(("JUMP_IF_TRUE_OR_POP", "JUMP_IF_FALSE_OR_POP")) and node_match(
            ast.BoolOp
        ):
            # and/or short circuit
            return
    
        if inst_match("DELETE_SUBSCR") and node_match(ast.Subscript, ctx=ast.Del):
            return
    
        if node_match(ast.Name, ctx=ast.Load) and inst_match(
            ("LOAD_NAME", "LOAD_FAST", "LOAD_GLOBAL"), argval=mangled_name(node)
        ):
            return
    
        if node_match(ast.Name, ctx=ast.Del) and inst_match(
            ("DELETE_NAME", "DELETE_GLOBAL"), argval=mangled_name(node)
        ):
            return
    
        # old verifier
    
        typ: Type = type(None)
        op_type: Type = type(None)
    
        if op_name.startswith(("BINARY_SUBSCR", "SLICE+")):
            typ = ast.Subscript
            ctx = ast.Load
        elif op_name.startswith("BINARY_"):
            typ = ast.BinOp
            op_type = op_type_map[instruction.argrepr]
            extra_filter = lambda e: isinstance(cast(ast.BinOp, e).op, op_type)
        elif op_name.startswith("UNARY_"):
            typ = ast.UnaryOp
            op_type = dict(
                UNARY_POSITIVE=ast.UAdd,
                UNARY_NEGATIVE=ast.USub,
                UNARY_NOT=ast.Not,
                UNARY_INVERT=ast.Invert,
            )[op_name]
            extra_filter = lambda e: isinstance(cast(ast.UnaryOp, e).op, op_type)
        elif op_name in ("LOAD_ATTR", "LOAD_METHOD", "LOOKUP_METHOD"):
            typ = ast.Attribute
            ctx = ast.Load
            extra_filter = lambda e: mangled_name(e) == instruction.argval
        elif op_name in (
            "LOAD_NAME",
            "LOAD_GLOBAL",
            "LOAD_FAST",
            "LOAD_DEREF",
            "LOAD_CLASSDEREF",
        ):
            typ = ast.Name
            ctx = ast.Load
            extra_filter = lambda e: cast(ast.Name, e).id == instruction.argval
        elif op_name in ("COMPARE_OP", "IS_OP", "CONTAINS_OP"):
            typ = ast.Compare
            extra_filter = lambda e: len(cast(ast.Compare, e).ops) == 1
        elif op_name.startswith(("STORE_SLICE", "STORE_SUBSCR")):
            ctx = ast.Store
            typ = ast.Subscript
        elif op_name.startswith("STORE_ATTR"):
            ctx = ast.Store
            typ = ast.Attribute
            extra_filter = lambda e: mangled_name(e) == instruction.argval
    
        node_ctx = getattr(node, "ctx", None)
    
        ctx_match = (
            ctx is not type(None)
            or not hasattr(node, "ctx")
            or isinstance(node_ctx, ctx)
        )
    
        # check for old verifier
        if isinstance(node, typ) and ctx_match and extra_filter(node):
            return
    
        # generate error
    
        title = "ast.%s is not created from %s" % (
            type(node).__name__,
            instruction.opname,
        )
    
>       raise VerifierFailure(title, node, instruction)
E       executing._exceptions.VerifierFailure: ast.UnaryOp is not created from CALL_INTRINSIC_1

executing/_position_node_finder.py:545: VerifierFailure
=========================== short test summary info ============================
FAILED tests/test_main.py::TestStuff::test_decorator - AssertionError: ([7, 8...
FAILED tests/test_main.py::TestStuff::test_decorator_cache_instruction - exec...
FAILED tests/test_main.py::TestStuff::test_names - executing._exceptions.Veri...
FAILED tests/test_main.py::test_global_tester_calls - executing._exceptions.V...
=================== 4 failed, 41 passed, 15 skipped in 3.02s ===================

Let GH index who's using this package

After experimenting a little bit, I found that for python packages, including a requirements.txt in the repo would enable GH to index who's using the package.

The list can be seen on both the main page of the repo or Insights -> Dependency graph -> Dependents

Request for help: a function that accepts (filename, qualname), finds that function and returns its source code

I am trying to make a caching system that must check a hash of the source of a function and all of the functions that it calls.

So far, I am able to use settrace and executing to get the filename and qualname of the function. What I haven't been able to figure out is how to retrieve the source given a filename and qualname.

I figure this would be right up the alley of a developer on this project. Help would be much appreciated ;)

Better ideas for implementing such a caching system are also appreciated.

Maybe an edge case that fails executing to find the node

Like the idea I mentioned in the latest issue at sorcery repo:
I am trying to do this:

data >> mean(X.a)
# trying to get the mean of the data attached to `data.a`
# This is actually doing `mean(data.a)`

To do that, I need to fetch X.a and turn it into lambda X: X.a.
So X is just a symbol that connects everything inside mean to make it a node that can be fetched later:

import sys
import ast

from executing import Source

class Symbolic:
    """A symbolic representation to make X.a and alike valid python syntaxes"""

    def __init__(self, exet=None):
        self.name = 'X'
        self.exet = exet

    def _any_args(self, *args, **kwargs):
        return Symbolic(Source.executing(sys._getframe(1)))

    def _single_arg(self, arg):
        return Symbolic(Source.executing(sys._getframe(1)))

    __call__ = _any_args
    __getattr__ = __contains__ = _single_arg

    def __rrshift__(self, data):
        return data

    @property
    def eval_(self):
        """Convert the symbolic representation into a lambda"""
        print('NODE:', self.exet.node)
        lambd_node = ast.Expression(
            body=ast.Lambda(
                ast.arguments(posonlyargs=[],
                              args=[ast.arg(arg=self.name)],
                              kwonlyargs=[],
                              kw_defaults=[],
                              defaults=[]),
                body=(Transformer(self.name).visit(self.exet.node)
                      if self.exet
                      else ast.Name(id=self.name, ctx=ast.Load()))
            )
        )
        ast.fix_missing_locations(lambd_node)
        code = compile(lambd_node, filename='<pipedy-ast>', mode='eval')
        if not self.exet:
            context = globals()
        else:
            context = self.exet.frame.f_globals.copy()
            context.update(self.exet.frame.f_locals)
        return eval(code, context)

Then the mean should be only executed at >>, to implement that, it can be an object of the following class:

class Piped:
    """A wrapper for the verbs"""
    def __init__(self, func, args):
        self.func = func
        self.args = args

    def eval_args(self, data):
        """Evaluate the args if there are Symbolic objects"""
        return (arg.eval_(data) if isinstance(arg, Symbolic) else arg
                for arg in self.args)

    def __rrshift__(self, data):
        """Evaluate the actual function"""
        return self.func(data, *self.eval_args(data))

Then I need a decorator to register the mean and alike functions:

def register_verb(func):
    """It simply turn the function to a Piped object"""
    def wrapper(*args):
        return Piped(func, args)

    return wrapper

So, I should be able to do this now:

@register_verb
def mean(data, arg):
  return sum(arg) / float(len(arg))

class Data:
  a = [1,2,3]

Data >> mean(X.a)  # gives us 2

Till now, everything works perfectly.

However, consider this, we want to get the mean of certain data attributes:

class Data:
  aa = [1,2,3]
  ab = [5,6,7]
  bc = [8,9,10]
# we only want to take the mean of `aa` and `ab`
Data >> mean(starts_with('a')) 

# then mean should probably be:
@register_verb
def mean(data, args):
  """Suppose args from starts_with('a') has the desired attribute names (aa and ab)"""
  s = 0.0
  for arg in args:
    s += sum(getattr(data, arg))
  return s / float(sum(len(arg) for arg in args))

Now we have to turn starts_with as s Symbolic object, too. So that it won't break the node. However, when we evaluate it, we still need it to return the real value. The trick is to save the actual function to mean.__pipedy__, and modify the node to call the real function:

def register_func(func):
    """Register a function used in Piped arguments"""
    def wrapper(*args):
        return Symbolic(Source.executing(sys._getframe(1)))
    wrapper.__pipedy__ = func
    return wrapper

# to register starts_with:
@register_func
def starts_with(data, prefix):
  return [attr for attr in data.__dict__ if attr.startswith(prefix)]

To modify the node, we wrote a NodeTransformer:

class Transformer(ast.NodeTransformer):
    """Transform a call into the real call"""
    def __init__(self, name):
        self.name = name

    def visit_Call(self, node):
        """Get the real calls"""
        # insert the data
        node.args.insert(0, ast.Name(id=self.name, ctx=ast.Load()))
        ret = ast.Call(
            func=ast.Attribute(value=node.func,
                               attr='__pipedy__', # get the original function
                               ctx=ast.Load()),
            args=node.args,
            keywords=node.keywords
        )
        return ret

Now the example works.

However, when we have 2 of these functions registered:

@register_func
def ends_with(data, suffix):
  return [attr for attr in data.__dict__ if attr.endswith(suffix)]

Data >> mean(starts_with('a')) # works fine 
Data >> mean(ends_with('c'))  # failed to fetch the node

After I dug into executing's source code, I found that for the second one:

for index_num, sentinel_index in enumerate(indices):
sentinel_index -= index_num * 2
new_index = sentinel_index - 1
if new_index != original_index:
continue

The new_index was shifted by 1.

Sorry for the super long post. I have tried to simplify the code and the scenario. However, this is the simplest one I could make.

A copy of the runnable code is attached.

test.zip

Thank you so much!

Misuse of list in _position_node_finder.get_instructions

The return type of this function is given as list[dis.Instruction], but it should be List[dis.Instruction], with List imported from typing. This causes a TypeError (at least with Python 3.7 and 3.8) when trying to import.

NameError: f-string node cannot be identified

Here's an example I found:

> python -m friendly_traceback

Friendly-traceback version 0.4.28. [Python version: 3.8.10]
Type 'Friendly' for help on special functions/methods.

[1]: print(f"Here is {xxx} again")

Traceback (most recent call last):
  Code block [1], line 1, in <module>
    print(f"Here is {xxx} again")
NameError: name 'xxx' is not defined

[2]: tb = Friendly._get_tb_data()

[3]: print(tb.node)
None

I would not be surprised if it would be impossible to extract the relevant information, due to the way that fstrings are evaluated. Still, just in case, I thought it was worth reporting.

What prompted me to try this example is a bug report on Python https://bugs.python.org/issue44885 for the planned 3.11 version which will include more information about the exact location of an error.

Feel free to close this issue as being "out of scope".

RFE: is it possible to restart making github releases?๐Ÿค”

On create github release entry is created email notification to those whom have set in your repo the web UI Watch->Releases.
gh release can contain additional comments (li changelog) or additional assets like release tar balls (by default it contains only assets from git tag) however all those part are not obligatory.
In simplest variant gh release can be empty because subiekt of the sent email contains git tag name.

I'm asking because my automation process uses those email notifications by trying to make preliminary automated upgrades of building packages, which allows saving some time on maintaining packaging procedures.
Probably other people may be interested to be instantly informed about release new version as well.

Documentation and examples of generate gh releases:
https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository
https://cli.github.com/manual/gh_release_upload/
https://github.com/marketplace/actions/github-release
https://pgjones.dev/blog/trusted-plublishing-2023/
jbms/sphinx-immaterial#281 (comment)
tox target to publish on pypi and make gh release https://github.com/jaraco/skeleton/blob/928e9a86d61d3a660948bcba7689f90216cc8243/tox.ini#L42-L58

Fetching `Attribute(..., ctx=Store())` failed for some python versions

Script to test:

import sys
import ast
import executing

print("")
print("-" * 80)
print(sys.version)

class Foo:

    def __setattr__(self, name, value) -> None:
        # print(argname("name", vars_only=True))
        node = executing.Source.executing(sys._getframe(1)).node
        print(ast.dump(node))

f = Foo()

f.x = 1
โฏ for i in py36 py37 py38 py3_9_10 py310; conda run -n $i --no-capture-output python test.py; end  
                             
--------------------------------------------------------------------------------
3.6.15 | packaged by conda-forge | (default, Dec  3 2021, 18:49:41) 
[GCC 9.4.0]
Attribute(value=Name(id='f', ctx=Load()), attr='x', ctx=Store())

--------------------------------------------------------------------------------
3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0]
Traceback (most recent call last):
  File "test.py", line 17, in <module>
    f.x = 1
  File "test.py", line 13, in __setattr__
    print(ast.dump(node))
  File "/home/pwwang/miniconda3/envs/py37/lib/python3.7/ast.py", line 120, in dump
    raise TypeError('expected AST, got %r' % node.__class__.__name__)
TypeError: expected AST, got 'NoneType'

--------------------------------------------------------------------------------
3.8.10 (default, Jun  4 2021, 15:09:15) 
[GCC 7.5.0]
Attribute(value=Name(id='f', ctx=Load()), attr='x', ctx=Store())

--------------------------------------------------------------------------------
3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:37) 
[GCC 9.4.0]
Traceback (most recent call last):
  File "/home/pwwang/github/python-varname/test.py", line 17, in <module>
    f.x = 1
  File "/home/pwwang/github/python-varname/test.py", line 13, in __setattr__
    print(ast.dump(node))
  File "/home/pwwang/miniconda3/envs/py3_9_10/lib/python3.9/ast.py", line 169, in dump
    raise TypeError('expected AST, got %r' % node.__class__.__name__)
TypeError: expected AST, got 'NoneType'

--------------------------------------------------------------------------------
3.10.0 (default, Mar  3 2022, 09:58:08) [GCC 7.5.0]
Traceback (most recent call last):
  File "/home/pwwang/github/python-varname/test.py", line 17, in <module>
    f.x = 1
  File "/home/pwwang/github/python-varname/test.py", line 13, in __setattr__
    print(ast.dump(node))
  File "/home/pwwang/miniconda3/envs/py310/lib/python3.10/ast.py", line 172, in dump
    raise TypeError('expected AST, got %r' % node.__class__.__name__)
TypeError: expected AST, got 'NoneType'

Looks like only python 3.6 and 3.8 worked.

__setitem__() behaved the same.

Include new library files in coverage measurement

Currently GHA has coverage run --include=executing/executing.py <run tests> which doesn't measure coverage in the recently added files in the executing folder. The original motivation was lines in __init__.py that would never be covered, but that should be handled by a # pragma: no cover comment or something.

Support for Python 3.11?

With today's release of Python 3.11.0a3, I suddenly have many failing tests for friendly-traceback that were still passing with 3.11.0a2. It looks like the nodes are no longer properly identified.

For example, if I have

a = (1, 2)(3, 4)

the node was previously identified as (1, 2)(3, 4) whereas now it is the entire line that is identified as the problematic node.

I can provide more examples if needed, but I suspect that your own unit tests would likely give you some results. It might be that the problem is upstream (with asttoken) instead of in executing.

Support pytest

pytest modifies the AST under the hood to show informative error messages when a plain assert fails. This affects the bytecode in ways executing doesn't expect. I'm not actually sure if it's possible to robustly deal with this, but it's worth taking a look at.

tests are failing for storing to two attributes with the same name

I'm currently working on #64 and have some problems with the tests for 3.7 (reproducible on master)

The pipeline which is failing is the following:
https://github.com/alexmojaki/executing/actions/runs/4362346591/jobs/7627135034

I minimized the problem down to the following line (file was stored in tests/small_samples):

__arg.__deprecated__ = __new__.__deprecated__ = __msg

part of the test output (python 3.7.15):

mapping failed
Expected one value, found 2
search bytecode Instruction(opname='STORE_ATTR', opcode=95, arg=2, argval='__deprecated__', argrepr='__deprecated__', offset=6, starts_line=None, is_jump_target=False)

I don't know if this issue could be fixed in the SentinelNodeFinder or if it should be handled as an known issue and be ignored.

I also don't know why it started to appear now.

executing failed to find the calling node on 3.10

As per this run of devtools CI, this test is failing.

The culprit is this, code:

        source = executing.Source.for_frame(call_frame)
        if not source.text:
            warning = 'no code context for debug call, code inspection impossible'
            arguments = list(self._args_inspection_failed(args, kwargs))
        else:
            ex = source.executing(call_frame)
            function = ex.code_qualname()
            if not ex.node:  <-- ex.node is falsey causing the error
                warning = "executing failed to find the calling node"

This is only happening on 3.10.5, it wasn't happening on 3.10.0-rc.1 which we were previously testing against. All other tests are passing, so I'm not sure what's causing it?

I'm going to xfail that test to get a new release out, but would be great to fix medium term.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.