Get information about what a Python frame is currently doing, particularly the AST node being executed

Related tags

Miscellaneous python
Overview

executing

Build Status Coverage Status Supports Python versions 2.7 and 3.4+, including PyPy

This mini-package lets you get information about what a frame is currently doing, particularly the AST node being executed.

Usage

Getting the AST node

import executing

node = executing.Source.executing(frame).node

Then node will be an AST node (from the ast standard library module) or None if the node couldn't be identified (which may happen often and should always be checked).

node will always be the same instance for multiple calls with frames at the same point of execution.

If you have a traceback object, pass it directly to Source.executing() rather than the tb_frame attribute to get the correct node.

Getting the source code of the node

For this you will need to separately install the asttokens library, then obtain an ASTTokens object:

executing.Source.executing(frame).source.asttokens()

or:

executing.Source.for_frame(frame).asttokens()

or use one of the convenience methods:

executing.Source.executing(frame).text()
executing.Source.executing(frame).text_range()

Getting the __qualname__ of the current function

executing.Source.executing(frame).code_qualname()

or:

executing.Source.for_frame(frame).code_qualname(frame.f_code)

The Source class

Everything goes through the Source class. Only one instance of the class is created for each filename. Subclassing it to add more attributes on creation or methods is recommended. The classmethods such as executing will respect this. See the source code and docstrings for more detail.

Installation

pip install executing

If you don't like that you can just copy the file executing.py, there are no dependencies (but of course you won't get updates).

How does it work?

Suppose the frame is executing this line:

self.foo(bar.x)

and in particular it's currently obtaining the attribute self.foo. Looking at the bytecode, specifically frame.f_code.co_code[frame.f_lasti], we can tell that it's loading an attribute, but it's not obvious which one. We can narrow down the statement being executed using frame.f_lineno and find the two ast.Attribute nodes representing self.foo and bar.x. How do we find out which one it is, without recreating the entire compiler in Python?

The trick is to modify the AST slightly for each candidate expression and observe the changes in the bytecode instructions. We change the AST to this:

(self.foo ** 'longuniqueconstant')(bar.x)

and compile it, and the bytecode will be almost the same but there will be two new instructions:

LOAD_CONST 'longuniqueconstant'
BINARY_POWER

and just before that will be a LOAD_ATTR instruction corresponding to self.foo. Seeing that it's in the same position as the original instruction lets us know we've found our match.

Is it reliable?

Yes - if it identifies a node, you can trust that it's identified the correct one. The tests are very thorough - in addition to unit tests which check various situations directly, there are property tests against a large number of files (see the filenames printed in this build) with real code. Specifically, for each file, the tests:

  1. Identify as many nodes as possible from all the bytecode instructions in the file, and assert that they are all distinct
  2. Find all the nodes that should be identifiable, and assert that they were indeed identified somewhere

In other words, it shows that there is a one-to-one mapping between the nodes and the instructions that can be handled. This leaves very little room for a bug to creep in.

Furthermore, executing checks that the instructions compiled from the modified AST exactly match the original code save for a few small known exceptions. This accounts for all the quirks and optimisations in the interpreter.

Which nodes can it identify?

Currently it works in almost all cases for the following ast nodes:

  • Call, e.g. self.foo(bar)
  • Attribute, e.g. point.x
  • Subscript, e.g. lst[1]
  • BinOp, e.g. x + y (doesn't include and and or)
  • UnaryOp, e.g. -n (includes not but only works sometimes)
  • Compare e.g. a < b (not for chains such as 0 < p < 1)

The plan is to extend to more operations in the future.

Libraries that use this

My libraries

  • stack_data: Extracts data from stack frames and tracebacks, particularly to display more useful tracebacks than the default. Also uses another related library of mine: pure_eval.
  • snoop: A feature-rich and convenient debugging library. Uses executing to show the operation which caused an exception and to allow the pp function to display the source of its arguments.
  • heartrate: A simple real time visualisation of the execution of a Python program. Uses executing to highlight currently executing operations, particularly in each frame of the stack trace.
  • sorcery: Dark magic delights in Python. Uses executing to let special callables called spells know where they're being called from.

Libraries I've contributed to

  • IPython: Highlights the executing node in tracebacks using executing via stack_data.
  • icecream: 🍦 Sweet and creamy print debugging. Uses executing to identify where ic is called and print its arguments.
  • python-devtools: Uses executing for print debugging similar to icecream.
  • sentry_sdk: Add the integration sentry_sdk.integrations.executingExecutingIntegration() to show the function __qualname__ in each frame in sentry events. Highlighting the executing node is hopefully coming soon.
  • varname: Dark magics about variable names in python. Uses executing to find where its various magical functions like varname and nameof are called from.
Comments
  • feat: support for python 3.11

    feat: support for python 3.11

    • This provides a new implementation, which uses co_positions() to lookup the ast node.
    • It has a simpler implementation and better performance.
    • Some limitations in the unit tests are removed for 3.11.
      • support for and and or
      • no ambiguities for generators
      • fixes #30
    opened by 15r10nk 80
  • Let GH index who's using this package

    Let GH index who's using this package

    After experimenting a little bit, I found that for python packages, including a requirements.txt in the repo would enable GH to index who's using the package.

    The list can be seen on both the main page of the repo or Insights -> Dependency graph -> Dependents

    opened by pwwang 20
  • Use only pytest

    Use only pytest

    I was able to execute all tests with pytest. However assertion rewriting is still disabled for test_main.py.

    But this has already several benefits:

    • tox -e py310 -- --sw works
    • no need to maintain two test environments

    I also converted the test_sample_files to an parametrized test which makes it more useful for pytest --sw

    I also don't know how I should handle the timeouts. I removed them for test_sample_files because they are not really useful for single file checks. I kept them for test_module_files because this takes already a bit longer. What's your opinion there?

    I wanted to get your feedback before I continue.

    opened by 15r10nk 12
  • Small samples

    Small samples

    generate small samples to improve test coverage.

    • The samples where generated with tox -e mutmut (which calls tests/mutmut_workflow.py, which class tests/generate_small_sample.py).
    • The samples are placed in tests/small_samples

    Every sample is the result of a code mutation detected by mutmut which was not covered by any other test (excluding the long running tests).

    generate_small_sample.py tries to find a source file with some code which fails with the mutated implementation and removes code from this file with pysource_minimize.

    I have also found some issues:

    • The ExceptHandler cleanup is now an KnownIssue. Before it was verified as an valid mapping, which is not ok.
    • DELETE/STORE_DEREF had some issues.

    I had to reimplement the Deadcode analysis. The problem was that I always tried to reimplement the deadcode analysis of the python interpreter, which did not scale very well. The new implementation replaces the node with an sentinel compiles the source and checks if the sentinel is in the bytecode. The downside is that I had to remove the inverse deadcode check, because the new implementation is a bit more complex ( compiles the whole file for every node to test).

    Let me know what you think.

    closes #55

    opened by 15r10nk 10
  • fix decorator detection bug

    fix decorator detection bug

    try to fix #49

    I also reported frame.f_lasti and the problem is that the instruction is a CACHE instruction in the deco case (f_lasti == 130).

    The Tester case gets the expected instruction and maps the bytecode correctly.

    You can find my source here https://github.com/15r10nk/executing/tree/decorator-detection-bug

    grafik

    I could ignore CACHE instructions until now. I will try to figure out what they are good for.

    opened by 15r10nk 9
  • Support for Python 3.11?

    Support for Python 3.11?

    With today's release of Python 3.11.0a3, I suddenly have many failing tests for friendly-traceback that were still passing with 3.11.0a2. It looks like the nodes are no longer properly identified.

    For example, if I have

    a = (1, 2)(3, 4)
    

    the node was previously identified as (1, 2)(3, 4) whereas now it is the entire line that is identified as the problematic node.

    I can provide more examples if needed, but I suspect that your own unit tests would likely give you some results. It might be that the problem is upstream (with asttoken) instead of in executing.

    opened by aroberge 9
  • Bug:  Incorrect node text when __slots__ and class variables are used (3.11)

    Bug: Incorrect node text when __slots__ and class variables are used (3.11)

    Apologies for not providing a self-contained test with executing for the following, as I don't know how. The following raises a ValueError exception:

    class F:
        __slots__ = ["a", "b"]
        a = 1
    

    With Python 3.10 (and prior), the text of the node identified causing the exception is:

    class F:
    

    With Python 3.11, we get this:

    class F: __slots__ = ["a", "b"] a = 1
    

    This is not a typo: the text of the node identified a single line, with no \n, which is clearly a SyntaxError.

    This is the only bug I found when running the unit tests for friendly_traceback with executing 1.1.0 and Python 3.11. Everything still works for me with Python 3.6 to 3.10, although I found that the tests I run take about 2 to 3 times as long as compared with using executing 0.8.3. The slowdown seemed to be already present with executing 1.0.0.

    opened by aroberge 7
  • Executing cannot find node inside Jupyter lab/notebooks

    Executing cannot find node inside Jupyter lab/notebooks

    Using friendly, I found that excuting could not locate the correct location of a node causing a problem inside Jupyter notebooks (or Jupyter lab) but could do so with IPython. First, here's a screen capture showing the correct result using IPython, with some additional outputs from print statements inserted for this specific report.

    image

    Next, the same code run within a Jupyter notebook image

    Perhaps this is is caused by the new way that Jupyter runs code, using temporary files instead of using exec() on code.

    With the "old" friendly-traceback (before it made use of stack_data), when Jupyter was not using temporary files, I see from screen captures on https://aroberge.github.io/friendly-traceback-docs/docs/html/jupyter.html that the location was correctly picked up.

    opened by aroberge 6
  • no source distribution on pypi

    no source distribution on pypi

    Would it be possible to upload the sdist in addition to the wheel file on pypi? Currently the conda-forge distribution does not automatically pick up updates unless the sources are present.

    opened by Anthchirp 6
  • 0.8.3: pytest is failing

    0.8.3: pytest is failing

    I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

    • python3 -sBm build -w --no-isolation
    • because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
    • install .whl file in </install/prefix>
    • run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

    Here is pytest output:

    + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-executing-0.8.3-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-executing-0.8.3-2.fc35.x86_64/usr/lib/python3.8/site-packages
    + /usr/bin/pytest -ra
    =========================================================================== test session starts ============================================================================
    platform linux -- Python 3.8.13, pytest-7.1.1, pluggy-1.0.0
    rootdir: /home/tkloczko/rpmbuild/BUILD/executing-0.8.3
    collected 2 items / 1 error
    
    ================================================================================== ERRORS ==================================================================================
    ___________________________________________________________________ ERROR collecting tests/test_main.py ____________________________________________________________________
    executing/executing.py:317: in executing
        args = executing_cache[key]
    E   KeyError: (<code object <module> at 0x7f98855a0ea0, file "/home/tkloczko/rpmbuild/BUILD/executing-0.8.3/tests/test_main.py", line 2>, 140293049028256, 418)
    
    During handling of the above exception, another exception occurred:
    executing/executing.py:346: in find
        node_finder = NodeFinder(frame, stmts, tree, lasti)
    executing/executing.py:636: in __init__
        matching = list(self.matching_nodes(exprs))
    executing/executing.py:702: in matching_nodes
        original_index = only(
    executing/executing.py:164: in only
        raise NotOneValueFound('Expected one value, found 0')
    E   executing.executing.NotOneValueFound: Expected one value, found 0
    
    During handling of the above exception, another exception occurred:
    tests/test_main.py:682: in <module>
        assert tester([1, 2, 3]) == [1, 2, 3]
    tests/utils.py:40: in __call__
        ex = self.get_executing(inspect.currentframe().f_back)
    tests/utils.py:28: in get_executing
        return Source.executing(frame)
    executing/executing.py:369: in executing
        args = find(source=cls.for_frame(frame), retry_cache=True)
    executing/executing.py:355: in find
        return find(
    executing/executing.py:346: in find
        node_finder = NodeFinder(frame, stmts, tree, lasti)
    executing/executing.py:636: in __init__
        matching = list(self.matching_nodes(exprs))
    executing/executing.py:702: in matching_nodes
        original_index = only(
    executing/executing.py:164: in only
        raise NotOneValueFound('Expected one value, found 0')
    E   executing.executing.NotOneValueFound: Expected one value, found 0
    ========================================================================= short test summary info ==========================================================================
    ERROR tests/test_main.py - executing.executing.NotOneValueFound: Expected one value, found 0
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    ============================================================================= 1 error in 0.44s =============================================================================
    
    opened by kloczek 5
  • Is it possible to know whether a frame is from a method call?

    Is it possible to know whether a frame is from a method call?

    For example

    def do_func(self):
        frame = sys._getframe()
    
    class A:
        def do_method(self):
            frame = sys._getframe()
    

    Is it possible to know whether frame is inside a regular function call or method call? From https://stackoverflow.com/q/2203424/2142577 it seems this is no way except relying on the convention that self is the first argument of method signature. Do we have a better way?

    I also tried using gc.get_referrers(frame.f_code)), but it turns out the code is referrenced by the unbound function, no matter whether it is actually a method.

    opened by laike9m 4
  • Index node finder

    Index node finder

    This is still work in progress.

    The goal is an NodeFinder with similar capabilities like PositionNodeFinder.

    todo:

    • [ ] workaround for https://github.com/python/cpython/issues/100537
    • [ ] integrate node verification ( create Verifier which can be used for python 3.11 and 3.10)
    • expand to:
      • [ ] 3.9
      • [ ] 3.8
      • [ ] 3.7
      • [ ] 3.6 (if possible)
    opened by 15r10nk 0
  • parent pointers in AST nodes hurt deepcopy performance

    parent pointers in AST nodes hurt deepcopy performance

    Obligatory "I'm a huge fan of your work".

    Background: I maintain a library called ipyflow, which uses another library I maintain called pyccolo. I noticed that on ipython >= 8.0, which uses executing for better stack traces, ipyflow would have really bad performance regressions the first time after a cell throws some exception.

    Eventually I traced it to pyccolo's use of copy.deepcopy in a few places -- performance was bad because the parent pointers that executing added to AST nodes were causing deepcopy to do a lot of extra unnecessary work.

    I ended up working around it on the pyccolo side, but I figured you may be interested in this for other libraries that may want to use executing and get surprised when deepcopy has bad perf. The way we maintain deepcopy-ability in pyccolo is to maintain a mapping from id(node) to parent for parent pointers, which avoids setting an attribute on the AST node directly.

    Thanks for this great library!

    opened by smacke 0
  • Include new library files in coverage measurement

    Include new library files in coverage measurement

    Currently GHA has coverage run --include=executing/executing.py <run tests> which doesn't measure coverage in the recently added files in the executing folder. The original motivation was lines in __init__.py that would never be covered, but that should be handled by a # pragma: no cover comment or something.

    opened by alexmojaki 3
Releases(v0.5.4)
Owner
Alex Hall
Python metaprogrammer
Alex Hall
Adds a Bake node to Blender's shader node system

Bake to Target This Blender Addon adds a new shader node type capable of reducing the texture-bake step to a single button press. Please note that thi

Thomas 8 Oct 4, 2022
Node editor view image node

A Blender addon to quickly view images from image nodes in Blender's image viewer.

null 5 Nov 27, 2022
Craxk is a SINGLE AND NON-REPLICABLE Hash that uses data from the hardware where it is executed to form a hash that can only be reproduced by a single machine.

What is Craxk ? Craxk is a UNIQUE AND NON-REPLICABLE Hash that uses data from the hardware where it is executed to form a hash that can only be reprod

null 5 Jun 19, 2021
An osu! cheat made in c++ rewritten in python and currently undetected.

megumi-python An osu! cheat made in c++ rewritten in python and currently undetected. Installation Guide Download python 3.9 from https://python.org C

Elaina 2 Nov 18, 2022
Pykeeb - A small Python script that prints out currently connected keyboards

pykeeb ?? ⌨️ A small Python script that detects and prints out currently connect

Jordan Duabe 1 May 8, 2022
Neogex is a human readable parser standard, being implemented in Python

Neogex (New Expressions) Parsing Standard Much like Regex, Neogex allows for string parsing and validation based on a set of requirements. Unlike Rege

Seamus Donnellan 1 Dec 17, 2021
New multi tool im making adding features currently

Emera Multi Tool New multi tool im making adding features currently Current List of Planned Features - Linkvertise Bypasser - Discord Auto Bump - Gith

Lamp 3 Dec 3, 2021
A feed generator. Currently supports generating RSS feeds from Google, Bing, and Yahoo news.

A feed generator. Currently supports generating RSS feeds from Google, Bing, and Yahoo news.

Josh Cardenzana 0 Dec 13, 2021
This an Anki add on that automatically converts Notion notes into Anki flash cards. Currently in development!

NotionFlash This is an Anki add on in development that will allow automatically convert your Notion study notes into Anki flash cards. The Anki deck c

Neeraj Patel 10 Oct 7, 2022
A guy with a lot of useful things to do when doing AtCoder in Python

atcoder_python_env Python で AtCoder をやるときに便利な諸々を用意したやつ コンテスト用フォルダの作成 セットアップ 自動テス

null 2 Dec 28, 2021
An open letter in support of Richard Matthew Stallman being reinstated by the Free Software Foundation

An open letter in support of RMS. To sign, click here and name the file <username>.yaml (replace <username> with your name) with the following content

null 2.4k Jan 7, 2023
Inspect the resources of your android projects and understand which ones are not being used and could potentially be removed.

Android Resources Checker What This program will inspect the resources of your app and help you understand which ones are not being used and could pot

Fábio Carballo 39 Feb 8, 2022
Aerial Ace is a helper bot for poketwo which provide various functionalities on top of being a pokedex.

Aerial Ace is a helper bot for poketwo which provide various functionalities on top of being a pokedex.

Devanshu Mishra 1 Dec 1, 2021
Users can read others' travel journeys in addition to being able to upload and delete posts detailing their own experiences

Users can read others' travel journeys in addition to being able to upload and delete posts detailing their own experiences! Posts are organized by country and destination within that country.

Christopher Zeas 1 Feb 3, 2022
Ikaros is a free financial library built in pure python that can be used to get information for single stocks, generate signals and build prortfolios

Ikaros is a free financial library built in pure python that can be used to get information for single stocks, generate signals and build prortfolios

Salma Saidane 64 Sep 28, 2022
🛠️ Learn a technology X by doing a project - Search engine of project-based learning

Learn X by doing Y ??️ Learn a technology X by doing a project Y Website You can contribute by adding projects to the CSV file.

William 408 Dec 20, 2022
Just some mtk tool for exploitation, reading/writing flash and doing crazy stuff

Just some mtk tool for exploitation, reading/writing flash and doing crazy stuff. For linux, a patched kernel is needed (see Setup folder) (except for read/write flash). For windows, you need to install zadig driver and replace pid 0003 / pid 2000 driver.

Bjoern Kerler 1.1k Dec 31, 2022
A example project's description is a high-level overview of why you’re doing a project.

A example project's description is a high-level overview of why you’re doing a project.

Nikita Matyukhin 12 Mar 23, 2022
Run python scripts and pass data between multiple python and node processes using this npm module

Run python scripts and pass data between multiple python and node processes using this npm module. process-communication has a event based architecture for interacting with python data and errors inside nodejs.

Tyler Laceby 2 Aug 6, 2021