PyCG: Practical Python Call Graphs

Vitalis Salis

Last update: Dec 29, 2022

Related tags

Overview

PyCG - Practical Python Call Graphs

PyCG generates call graphs for Python code using static analysis. It efficiently supports

Higher order functions
Twisted class inherritance schemes
Automatic discovery of imported modules for further analysis
Nested definitions

You can read the full methodology as well as a complete evaluation on the ICSE 2021 paper.

Abstract:

Call graphs play an important role in different contexts, such as profiling and vulnerability propagation analysis. Generating call graphs in an efficient manner can be a challenging task when it comes to high-level languages that are modular and incorporate dynamic features and higher-order functions. Despite the language's popularity, there have been very few tools aiming to generate call graphs for Python programs. Worse, these tools suffer from several effectiveness issues that limit their practicality in realistic programs. We propose a pragmatic, static approach for call graph generation in Python. We compute all assignment relations between program identifiers of functions, variables, classes, and modules through an inter-procedural analysis. Based on these assignment relations, we produce the resulting call graph by resolving all calls to potentially invoked functions. Notably, the underlying analysis is designed to be efficient and scalable, handling several Python features, such as modules, generators, function closures, and multiple inheritance. We have evaluated our prototype implementation, which we call PyCG, using two benchmarks: a micro-benchmark suite containing small Python programs and a set of macro-benchmarks with several popular real-world Python packages. Our results indicate that PyCG can efficiently handle thousands of lines of code in less than a second (0.34 seconds for 1k LoC on average). Further, it outperforms the state-of-the-art for Python in both precision and recall: PyCG achieves high rates of precision ~99, and adequate recall ~69.3. Finally, we demonstrate how PyCG can aid dependency impact analysis by showcasing a potential enhancement to GitHub's security advisory notification service using a real-world example.

Installation

PyCG is implemented in Python3 and has no dependencies. Simply:

pip install pycg

Usage

~ >>> pycg -h
usage: pycg [-h] [--package PACKAGE] [--fasten] [--product PRODUCT]
            [--forge FORGE] [--version VERSION] [--timestamp TIMESTAMP]
            [-o OUTPUT]
            [entry_point [entry_point ...]]

positional arguments:
  entry_point           Entry points to be processed

optional arguments:
  -h, --help            show this help message and exit
  --package PACKAGE     Package containing the code to be analyzed
  --fasten              Produce call graph using the FASTEN format
  --product PRODUCT     Package name
  --forge FORGE         Source the product was downloaded from
  --version VERSION     Version of the product
  --timestamp TIMESTAMP
                        Timestamp of the package's version
  -o OUTPUT, --output OUTPUT
                        Output path

where the command line arguments are:

entry_point: A list of paths to Python modules that PyCG will analyze. It is suggested that this list of paths contains only entry points since PyCG automatically discovers all other (local) imported modules.
--package: The unix path to the module's namespace (i.e. the path from which the module would be executed). This parameter is really important for the correct resolving of imports.
--fasten: Output the callgraph in FASTEN format.
-output: The unix path where the output call graph will be stored in JSON format.

The following command line arguments should used only when --fasten is provied:

--product: The name of the package.
--forge: Source the package was downloaded from.
--version: The version of the package.
--timestamp : The timestamp of the package's version.

Output

Simple JSON format

The call edges are in the form of an adjacency list where an edge (src, dst) is represented as an entry of dst in the list assigned to key src:

{
    "node1": ["node2", "node3"],
    "node2": ["node3"],
    "node3": []
}

FASTEN Format

For an up-to-date description of the FASTEN format refer to the FASTEN wiki.

Examples

All the entry points are known and we want the simple JSON format

~ >>> pycg --package pkg_root pkg_root/module1.py pkg_root/subpackage/module2.py -o cg.json

All entry points are not known and we want the simple JSON format

~ >>> pycg --package django $(find django -type f -name "*.py") -o django.json

We want the FASTEN format:

~ >>> pycg --package pypi_pkg --fasten --product "pypipkg" --forge "PyPI" \
        --version "0.1" --timestamp 42 \
        pypi_pkg/module1.py pkg_root/subpackage/module2.py -o cg.json

Running Tests

From the root directory:

make test

Comments

Error running pycg.

Hi there,

writing the Fasten-PyPI-plugin, we would need to run pycg locally to analyze missing call graphs on the FASTEN server.

Trying out the tool, I got with several packages this error.

michelescarlato@splinter:~/call-graphs-creation/scipy-1.8.1$ pycg --fasten  --package scipy $(find scipy -type f -name "*.py") --product "scipy" --version "1.8.1" --forge "PyPI" --max-iter -1 -o cg.json
Traceback (most recent call last):
  File "/home/michelescarlato/.local/bin/pycg", line 8, in <module>
    sys.exit(main())
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/pycg.py", line 155, in analyze
    self.do_pass(PreProcessor, True,
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/pycg.py", line 144, in do_pass
    self.import_manager.install_hooks()
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/machinery/imports.py", line 203, in install_hooks
    loader = get_custom_loader(self)
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/machinery/imports.py", line 34, in get_custom_loader
    class CustomLoader(importlib.abc.SourceLoader):
AttributeError: module 'importlib' has no attribute 'abc'. Did you mean: '_abc'?

Am I doing something wrong?

opened by michelescarlato 8

Require Python 3.4 version or latest
Description

Following the fix of https://github.com/vitsalis/PyCG/pull/31, we need to configure PyCG to require a Python version greater or equal than 3.4, in order maintain the changes introduced.

I also updated the project's readme in order to inform users regarding this additional requirement
opened by gdrosos 2

FIX: Avoid (Re)Importing Used Modules

Description

Currently, PyCG failed to produce a call graph of itself due to a bug. More specifically, when executed in the source code of itself, it yielded an AttributeError as it could not resolve the pycg.utils.constantsmodule.

The problem is located on line 153 on the following code block: https://github.com/vitsalis/PyCG/blob/5d2506f79c09f6925a1af4983109c200e2110d0e/pycg/machinery/imports.py#L148-L153 More precisely, when PyCG encounters an import, it checks whether the imported module is part of the standard library installed locally (by checking if it exists in the sys.modules dictionary). If this is not the case, it imports the module and returns the module object which is later processed.

The problem is that for modules which are already imported this command will statically import them, meaning that the info of the initial module used from pycg will be lost. Later, if PyCG tries to use any functions of the module which is lost from memory, an exception will be raised as it will not be able to locate any method.

Fix

To fix this, I added checking whether the module of the specific package exists. If this is the case, we return the existing module object in order to continue the import handling. If this is not the case, we import the module.

Testing

Tested on the source code of PyCG, and it succesfully produces the call graph. Also tested on 2 sample pypi packages and the json output is the same before and after this fix, meaning that no regression issues seem to be introduced.

Aditional Context

Minimal Example to Reproduce Issue

git clone https://github.com/vitsalis/PyCG.git
cd PyCG
pip3 install pycg
pycg  pycg/utils/constants.py  pycg/utils/__init__.py

Logs

pycg pycg/utils/constants.py  pycg/utils/__init__.py 
Traceback (most recent call last):
  File "/home/gdrosos/.local/bin/pycg", line 8, in <module>
    sys.exit(main())
  File "/home/gdrosos/.local/lib/python3.6/site-packages/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/home/gdrosos/.local/lib/python3.6/site-packages/pycg/pycg.py", line 157, in analyze
    self.class_manager, self.module_manager)
  File "/home/gdrosos/.local/lib/python3.6/site-packages/pycg/pycg.py", line 148, in do_pass
    processor.analyze()
  File "/home/gdrosos/.local/lib/python3.6/site-packages/pycg/processing/preprocessor.py", line 375, in analyze
    self.visit(ast.parse(self.contents, self.filename))
  File "/usr/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File "/home/gdrosos/.local/lib/python3.6/site-packages/pycg/processing/preprocessor.py", line 114, in visit_Module
    super().visit_Module(node)
  File "/home/gdrosos/.local/lib/python3.6/site-packages/pycg/processing/base.py", line 61, in visit_Module
    self.generic_visit(node)
  File "/usr/lib/python3.6/ast.py", line 261, in generic_visit
    self.visit(item)
  File "/usr/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File "/home/gdrosos/.local/lib/python3.6/site-packages/pycg/processing/preprocessor.py", line 209, in visit_ImportFrom
    self.visit_Import(node, prefix=node.module, level=node.level)
  File "/home/gdrosos/.local/lib/python3.6/site-packages/pycg/processing/preprocessor.py", line 181, in visit_Import
    add_external_def(src_name, tgt_name)
  File "/home/gdrosos/.local/lib/python3.6/site-packages/pycg/processing/preprocessor.py", line 164, in add_external_def
    defi = self.def_manager.create(name, utils.constants.EXT_DEF)
AttributeError: module 'pycg.utils.constants' has no attribute 'EXT_DEF'

opened by gdrosos 2

Produce Call Graphs in Fasten Version 2 Format

Description

This Pull Request aims to update the Version of the Call Graphs produced by PyCG on Fasten Format from Version 1 to Version 2, as described on the Fasten Wiki. The methodology used to make the convertion stems from the Fasten Call Graph Producer, which was previously used to make the convertion between the different formats, but we tried to develop the most optimal implementation.

Motivation

Since the whole Python Pipeline on Fasten Project depends on Call Graphs following the Version 2 format, it makes sense that PyCG directly produces them without having one extra tool as a mediator to make the convertion. Also, with this change the Call Graphs produced by PyCG can be directly stitched through PyCG Stitcher, since the Stitcher only supports Version 2 Call Graphs

Testing

Adapted the Fasten format unit tests to support the new format.

opened by gdrosos 2
Can pycg analyze a function call chain?
Thank you for making the interesting tool public! I am reading the ICSE paper and trying to use this tool.

I have one question about the behavior of pycg. I have the following code example.py:

class C: def __init__(self): pass def m(self): return "x({0})" obj = C() print(obj.m().format(1))

I executed pycg example.py.

I expected that pycg reports that the example file calls str.format (called in the bottom line) because the function body is included in the analysis.

However, pycg resulted in:

{"example": ["example.C.m", "example.C.__init__", "<builtin>.print"], "example.C.__init__": [], "example.C.m": [], "<builtin>.print": []}

The str.format function is not included in the result.

Do I miss any important options?
opened by takashi-ishio 2
Detect Pandas errors

shell: pycg --package pandas-1.2.3 $(find pandas-1.2.3 -type f -name "*.py") Got the following error： Traceback (most recent call last): File "/usr/local/bin/pycg", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/pycg/__main__.py", line 56, in main cg.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/pycg.py", line 99, in analyze self.class_manager, self.module_manager) File "/usr/local/lib/python3.7/site-packages/pycg/pycg.py", line 89, in do_pass processor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 202, in visit_Import self.analyze_submodule(modname) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 202, in visit_Import self.analyze_submodule(modname) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 292, in visit_FunctionDef super().visit_FunctionDef(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 70, in visit_FunctionDef self.visit(stmt) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 292, in visit_FunctionDef super().visit_FunctionDef(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 70, in visit_FunctionDef self.visit(stmt) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 308, in visit_Assign self._visit_assign(node.value, node.targets) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 192, in _visit_assign do_assign(decoded, target) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 181, in do_assign do_assign(decoded[pos], elt) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 180, in do_assign if pos < len(decoded): TypeError: object of type 'Definition' has no len()

opened by YZ-eng 2
What tool have you used to generate the figures from JSON/FASTEN?

The paper contains some figures with arrows and colors, very useful for human (rather than machine) understanding.

However it looks like this project generates output in either JSON or FASTEN format which are machine-friendly but not human friendly and certainly not graphics. How have you generated the figures in the paper and/or what would you recommend people to use to go from JSON (or FASTEN) to an output more suitable for human consumption?

opened by davide-q 1
Broken link

The link to the paper on the homepage is broken. If you don't have any other ways, you may want to link https://web.archive.org/web/20220331023325/https://vitsalis.com/papers/pycg.pdf instead

opened by davide-q 1
Help interpreting the outputs

Are there some examples that help explain the output produced by PyCG? My question is specifically about the two possible outputs from PyCG: (1) --as-graph-output AS_GRAPH_OUTPUT Output for the assignment graph (2) -o OUTPUT, --output OUTPUT Output path

What is the difference between these two? It would be great if there an example input with these two outputs along with an explanation on interpreting them.

opened by beejaya 1

Can't add edge to a non existing node

When I try to run a simple example on PyCG, it always raises exception pycg.machinery.imports.ImportManagerError.

My setting is as follow: python version: 3.6.9 pycg commit: 99c991

I setup test directory like

mypkg/
  __init__.py
  main.py

and main.py contains

def foo():
    bar()
def bar():
    print('bar')

Running pycg --package mypkg mypkg/main.py -o cg.json raises an exception as follow:

Traceback (most recent call last):
  File "/home/jwhur/.pyenv/versions/3.6.9/bin/pycg", line 11, in <module>
    sys.exit(main())
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/pycg.py", line 157, in analyze
    self.class_manager, self.module_manager)
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/pycg.py", line 147, in do_pass
    modules_analyzed=modules_analyzed, *args, **kwargs)
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/processing/preprocessor.py", line 33, in __init__
    super().__init__(filename, modname, modules_analyzed)
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/processing/base.py", line 36, in __init__
    with open(filename, "rt") as f:
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 894, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1129, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1273, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1229, in _get_spec
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/machinery/imports.py", line 39, in __init__
    ig_obj.create_edge(self.fullname)
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/machinery/imports.py", line 87, in create_edge
    raise ImportManagerError("Can't add edge to a non existing node")
pycg.machinery.imports.ImportManagerError: Can't add edge to a non existing node

opened by JaewonHur 1

`pycg` throws `pycg.machinery.imports.ImportManagerError: Can't add edge to a non existing node Command exited with non-zero status 1`

Hey @vitsalis and @gdrosos ,

I am trying to run the pypi-plugin on branch cg_producer-integration upon the pipup package, which results in the error messages written down in the attached output.txt file.

Strange thing is, that one Call Graph can be created (the one for zipp, line 89 in output.txt) while all the others can not. I have the feeling that maybe the tool is not able to "go deeper" into the folder structure: For zipp, the structure is /pycg-data/untar/zipp.py while for the others its e.g. pycg-data/untar/more_itertools. To find python files, it has to dive in one deeper (this is just a guess though).

output_verbose.txt is almost the same output, except that I added some print() statements here to see where the process failed.

I also added a print("Package Path to generate Call Graph") and print(package_path) here, which gives me the feeling, that the problem lays in this line as it has to go one folder deeper to find the python files.

All just guesses, let me know what you think and thx in advance!

output.txt output_verbose.txt

opened by amphioxis 1
list definition creation causes an infinite loop in the post processor

i will give and example since it happens vary rarely but maybe you will be able to see where the issue is:

example.py: list1 = [123]

def func1(): for func in list1: x = func([], []) list2 = [456] for func in list2: x = func([], [])

opened by shlakperson 0
PermissionError : Permission denied

This is a simple problem but I cant seem to solve it. Using the command "pycg folder" gives me a PermissionError in all cases.

I am using pycg on anaconda, windows 10, and tried to launch anaconda on administrator mode but it still results in this error, even for an empty folder.

Did anyone stumble across the same problem ?

opened by QuanticDisaster 1
`pycg` uses a lot of memory to create a Call Graph for specific packages

Hey @gdrosos and @vitsalis ,

running the pypi-plugin which uses pycg internally, for the packages click:7.0 and py:1.11.0, the process of creating a Call Graph uses all my memory (16 GB) and all my swap space (30 GB) without creating a Call Graph, while for other packages it only consumes a tiny fraction of that.

Not sure how to explain that better, so let me know which further information you need.

Thx for your help!

Additional packages: pyrsistent:0.18.1 Django:1.11.29 rjsmin:1.2.0 pyinotify:0.9.6 pytest:3.2.5 tqdm:4.64.0

opened by amphioxis 0
functions call stack with random order?
Hello,

very nice tool with plentry of potential IMHO!

I was just wondering whether the order of the function call could be preserved?

For instance for a function I would get the following:

"flow.runner.run":[ "<builtin>.len", "<builtin>.print", "flow.display.plot", "pandas.read_csv", "numpy.arange", "pandas.DataFrame" ],

the order of the functions called in the array is not the same order as the original python file. Also if I run pycg several times in a row, I would get a different order each time.

Would it be possible to keep the actual call order? That would definitely be valuable to ensure the sequence of functions being called. Thanks for your insight.
opened by theweaklink 1
pycg can‘t find the real function

repo：https://github.com/PaddlePaddle/PaddleClas cmd :pycg --package ppclas $(find ppclas -type f -name "*.py") -o ppclas.json error: funciton:ppcls/arch/backbone/base/theseus_layer.py is subclass paddle.nn.Layer the function train is from nn.Layer but the ppclas.json not show this message e.g.:

opened by Gzxl 3