google-research / python-graphs Goto Github PK

A static analysis library for computing graph representations of Python programs suitable for use with graph neural networks.

License: Apache License 2.0

Python 100.00%

python-graphs's Introduction

python_graphs

This package is for computing graph representations of Python programs for machine learning applications. It includes the following modules:

control_flow For computing control flow graphs statically from Python programs.
data_flow For computing data flow analyses of Python programs.
program_graph For computing graphs statically to represent arbitrary Python programs or functions.
cyclomatic_complexity For computing the cyclomatic complexity of a Python function.

Installation

To install python_graphs with pip, run: pip install python_graphs.

To install python_graphs from source, run: python setup.py develop.

Common Tasks

Generate a control flow graph from a function fn:

from python_graphs import control_flow
graph = control_flow.get_control_flow_graph(fn)

Generate a program graph from a function fn:

from python_graphs import program_graph
graph = program_graph.get_program_graph(fn)

Compute the cyclomatic complexity of a function fn:

from python_graphs import control_flow
from python_graphs import cyclomatic_complexity
graph = control_flow.get_control_flow_graph(fn)
value = cyclomatic_complexity.cyclomatic_complexity(graph)

This is not an officially supported Google product.

python-graphs's People

Contributors

Stargazers

Watchers

python-graphs's Issues

How do we solve the error when installing python-graphs?

Hello,

I encountered an error "fatal error: 'graphviz/cgraph.h' file not found" when trying to install python_graphs. How do I solve this issue, please? Thanks.

Clarify installation instructions so `pip install` works out of the box

pip3 install python_graphs didn't work for me on macOS 13.5 (Apple Silicon) with Python 3.10:

$ pip3 install python_graphs
Collecting python_graphs
  Using cached python_graphs-1.2.3-py3-none-any.whl
Requirement already satisfied: absl-py in ./venv/lib/python3.10/site-packages (from python_graphs) (1.4.0)
Collecting astunparse (from python_graphs)
  Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting gast (from python_graphs)
  Using cached gast-0.5.4-py3-none-any.whl (19 kB)
Requirement already satisfied: networkx in ./venv/lib/python3.10/site-packages (from python_graphs) (3.1)
Collecting pygraphviz (from python_graphs)
  Using cached pygraphviz-1.11.zip (120 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: six in ./venv/lib/python3.10/site-packages (from python_graphs) (1.16.0)
Requirement already satisfied: wheel<1.0,>=0.23.0 in ./venv/lib/python3.10/site-packages (from astunparse->python_graphs) (0.40.0)
Building wheels for collected packages: pygraphviz
  Building wheel for pygraphviz (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [55 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-13-arm64-cpython-310
      creating build/lib.macosx-13-arm64-cpython-310/pygraphviz
      copying pygraphviz/scraper.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz
      copying pygraphviz/graphviz.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz
      copying pygraphviz/__init__.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz
      copying pygraphviz/agraph.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz
      copying pygraphviz/testing.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz
      creating build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_unicode.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_scraper.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_readwrite.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_string.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/__init__.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_html.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_node_attributes.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_drawing.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_repr_mimebundle.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_subgraph.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_close.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_edge_attributes.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_clear.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_layout.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_attribute_defaults.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      copying pygraphviz/tests/test_graph.py -> build/lib.macosx-13-arm64-cpython-310/pygraphviz/tests
      running egg_info
      writing pygraphviz.egg-info/PKG-INFO
      writing dependency_links to pygraphviz.egg-info/dependency_links.txt
      writing top-level names to pygraphviz.egg-info/top_level.txt
      reading manifest file 'pygraphviz.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching '*.png' under directory 'doc'
      warning: no files found matching '*.txt' under directory 'doc'
      warning: no files found matching '*.css' under directory 'doc'
      warning: no previously-included files matching '*~' found anywhere in distribution
      warning: no previously-included files matching '*.pyc' found anywhere in distribution
      warning: no previously-included files matching '.svn' found anywhere in distribution
      no previously-included directories found matching 'doc/build'
      adding license file 'LICENSE'
      writing manifest file 'pygraphviz.egg-info/SOURCES.txt'
      copying pygraphviz/graphviz.i -> build/lib.macosx-13-arm64-cpython-310/pygraphviz
      copying pygraphviz/graphviz_wrap.c -> build/lib.macosx-13-arm64-cpython-310/pygraphviz
      running build_ext
      building 'pygraphviz._graphviz' extension
      creating build/temp.macosx-13-arm64-cpython-310
      creating build/temp.macosx-13-arm64-cpython-310/pygraphviz
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -DSWIG_PYTHON_STRICT_BYTE_CHAR -I/…/venv/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c pygraphviz/graphviz_wrap.c -o build/temp.macosx-13-arm64-cpython-310/pygraphviz/graphviz_wrap.o
      pygraphviz/graphviz_wrap.c:3020:10: fatal error: 'graphviz/cgraph.h' file not found
      #include "graphviz/cgraph.h"
               ^~~~~~~~~~~~~~~~~~~
      1 error generated.
      error: command '/opt/homebrew/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pygraphviz
  Running setup.py clean for pygraphviz
Failed to build pygraphviz
ERROR: Could not build wheels for pygraphviz, which is required to install pyproject.toml-based projects

The reason is pygraphviz has extra dependencies (graphviz) and special installation instructions.

_{https://pygraphviz.github.io/documentation/stable/install.html}

Figuring out the correct commands to install pygraphviz took quite a few minutes. Documenting this would be great.

Control Flow is incorrectly produced for code containing "with" statement

Hello! Trying to produce control flow graph for the following code:

def get_fun(fun):
	with _get_serv(ret=None, commit=True) as cur:
		sql = 'SELECT s.id, s.jid, s.full_ret FROM salt_returns s JOIN (SELECT MAX(`jid`) as jid from salt_returns GROUP BY fun, id) max ON s.jid = max.jid WHERE s.fun = %s'
		cur.execute(sql, (fun,))
		data = cur.fetchall()
		ret = {}
		if data:
			for (minion, _, full_ret) in data:
				ret[minion] = full_ret
		return ret

I get the following graph:

I believe this graph does not represent the actual control flow of the code

How to generate a Holistic Data Flow Graph for a given Function ?

@dbieber, Thanks for this awesome work.

Question

control_flow.get_control_flow_graph, returns a Control Flow Graph for a given Function Object. There is one data_flow class, Is there a way to generate a complete Data Flow Graph given a Function Object?

Thanks.

Set up GitHub Actions CI

Does this tool support python2？

Great jobs!
Is this work compatible with programs written in python2.7?

How do we solve the

KeyError when trying to get program_graph

When I try to create a program graph, I encounter a KeyError. If I remove all the and and or expressions from the python file (buggy.py) the error does not occur.

This is how I use the library:

graph = program_graph.get_program_graph(code)
program_graph_graphviz.render(graph, path='source.png')

where code is simply the code in the attached file buggy.py.txt.

I have also attached the log file log.txt.

buggy.py.txt

log.txt

More information:
python 3.9.5
commit head=44c15b92197f374c3550353ff827997ef1c1d857
gast 0.5.3

TypeError: can only concatenate str (not "Name") to str

Exception getting code graph in test_core -- can only concatenate str (not "Name") to str
Traceback (most recent call last):
  File "/home/panjie/code/CATLM/Evaluation/Test Generation/TestDataPrep/input_context_generation_python.py", line 125, in get_context_and_baseline_files
    test_file_graph, test_file_content = get_python_program_graph(test_filepath)
  File "/home/panjie/code/CATLM/Evaluation/Test Generation/TestDataPrep/ast_utils.py", line 10, in get_python_program_graph
    graph = program_graph.get_program_graph(file_content)
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/python_graphs/program_graph.py", line 641, in get_program_graph
    SyntaxNodeUnparser(program_node, program_graph)
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/python_graphs/program_graph.py", line 783, in __init__
    super(SyntaxNodeUnparser, self).__init__(ast_node, file=devnull)
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/astunparse/unparser.py", line 38, in __init__
    self.dispatch(tree)
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/python_graphs/program_graph.py", line 789, in dispatch
    super(SyntaxNodeUnparser, self).dispatch(ast_node)
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/astunparse/unparser.py", line 66, in dispatch
    meth(tree)
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/astunparse/unparser.py", line 78, in _Module
    self.dispatch(stmt)
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/python_graphs/program_graph.py", line 789, in dispatch
    super(SyntaxNodeUnparser, self).dispatch(ast_node)
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/astunparse/unparser.py", line 66, in dispatch
    meth(tree)
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/astunparse/unparser.py", line 347, in _FunctionDef
    self.__FunctionDef_helper(t, "def")
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/astunparse/unparser.py", line 359, in __FunctionDef_helper
    self.dispatch(t.args)
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/python_graphs/program_graph.py", line 789, in dispatch
    super(SyntaxNodeUnparser, self).dispatch(ast_node)
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/astunparse/unparser.py", line 66, in dispatch
    meth(tree)
  File "/home/panjie/miniconda3/envs/CAT-LM/lib/python3.9/site-packages/astunparse/unparser.py", line 832, in _arguments
    self.write("**"+t.kwarg)
TypeError: can only concatenate str (not "Name") to str

I run my code and I meet this error, so I change the line 832 in file python3.9/site-packages/astunparse/unparser.py to self.write("**"+str(t.kwarg)), and this bug never appeared, maybe it is a bug in the code.

Can you provide a quick start example?

Super cool project! Love the idea and think it has a lot of potential.

it would be awesome to have an examples/ directory containing some sample usages - maybe even just plotting the graphs with networkX and matplotlib.