NoPdb: Non-interactive Python Debugger

Overview

NoPdb: Non-interactive Python Debugger

PyPI Package Documentation Status Lint Status Lint Status

NoPdb is a programmatic (non-interactive) debugger for Python. This means it gives you access to debugger-like superpowers directly from your code. With NoPdb, you can:

  • capture function calls, including arguments, local variables, return values and stack traces
  • set "breakpoints" that trigger user-defined actions when hit, namely:
    • evaluate expressions to retrieve their values later
    • execute arbitrary code, including modifying local variables
    • enter an interactive debugger like pdb

NoPdb is also a convenient tool for inspecting machine learning model internals. For example, this notebook shows how to use it to visualize Transformer attention in PyTorch.

NoPdb should run at least under CPython and PyPy. Most features should work under any implementation as long as it has sys.settrace().

Note: This project is in its early development stage. Contributions and improvement ideas are welcome.

Capturing function calls

The functions capture_call() and capture_calls() allow capturing useful information about calls to a given function. They are typically used as context managers, e.g.:

with nopdb.capture_calls(fn) as calls:
    some_code_that_calls_fn()

    print(calls)  # see details about how fn() was called

The information we can retrieve includes the function's arguments, return value, local variables and stack trace. For example:

>>> with nopdb.capture_call(f) as call:
...     g(1)
>>> call
CallCapture(name='f', args=OrderedDict(x=1, y=1), return_value=4)
>>> call.print_stack()
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 2, in g
  File "<stdin>", line 1, in f
>>> call.args['x']
1
>>> call.return_value
4
>>> call.locals
{'y': 1, 'x': 1, 'z': 2}

Setting breakpoints

Like conventional debuggers, NoPdb can set breakpoints. However, because NoPdb is a non-interactive debugger, its breakpoints do not actually stop the execution of the program. Instead, they allow executing actions scheduled in advance, such as evaluating expressions.

To set a breakpoint, call the breakpoint() function. A breakpoint object is returned, allowing to schedule actions using its methods such as eval() and exec(). For example:

# Break at line 3 of the file or notebook cell where f is defined
with nopdb.breakpoint(function=f, line=3) as bp:
    x = bp.eval("x")             # Schedule an expression
    type_y = bp.eval("type(y)")  # Another one
    bp.exec("print(y)")          # Schedule a print statement

    some_code_that_calls_f()

print(x, type_y)  # Retrieve the captured values

There are other ways to specify the breakpoint location. For example:

# Break at any line with the given source code in the given file
with nopdb.breakpoint(file="pathlib.py", line="return obj") as bp:
    ...

# Break as soon as any function with the given name is called
with nopdb.breakpoint(function="myfunc") as bp:
    ...

Not only can we capture values, we can also modify them!

>>> with nopdb.breakpoint(function=f, line=3) as bp:
...     # Get the value of x, then increment it, then get the new value
...     x_before = bp.eval('x')
...     bp.exec('x += 1')
...     x_after = bp.eval('x')
...
...     some_code_that_calls_f()
>>> x_before
[2]
>>> x_after
[3]

Planned features

Functionalities that do not exist, but could be added in the future:

  • Breakpoint.callback() for calling a given callback function, passing information about the current frame as an argument.
  • Breakpoint.jump() for jumping to a different line in the same function.
  • A way to disable breakpoints.

Limitations

  • Like Pdb, NoPdb only works with pure-Python functions. Calls to built-ins and C extensions cannot be captured. This also applies to ML frameworks that compile models into static graphs; for NoPdb to work, this feature needs to be disabled, e.g. with tf.config.run_functions_eagerly(True) in TensorFlow and with the jax.disable_jit() context manager in JAX.
  • Local variable assignment in Breakpoint.exec() is only supported under CPython and PyPy.
You might also like...
PINCE is a front-end/reverse engineering tool for the GNU Project Debugger (GDB), focused on games.
PINCE is a front-end/reverse engineering tool for the GNU Project Debugger (GDB), focused on games.

PINCE is a front-end/reverse engineering tool for the GNU Project Debugger (GDB), focused on games. However, it can be used for any reverse-engi

Little helper to run Steam apps under Proton with a GDB debugger

protongdb A small little helper for running games with Proton and debugging with GDB Requirements At least Python 3.5 protontricks pip package and its

Full featured multi arch/os debugger built on top of PyQt5 and frida

Full featured multi arch/os debugger built on top of PyQt5 and frida

A simple rubber duck debugger

Rubber Duck Debugger I found myself many times asking a question on StackOverflow or to one of my colleagues just for finding the solution simply by d

Hdbg - Historical Debugger
Hdbg - Historical Debugger

hdbg - Historical Debugger This is in no way a finished product. Do not use this

Trashdbg - TrashDBG the world's worse debugger
Trashdbg - TrashDBG the world's worse debugger

The world's worse debugger Over the course of multiple OALABS Twitch streams we

Trace any Python program, anywhere!

lptrace lptrace is strace for Python programs. It lets you see in real-time what functions a Python program is running. It's particularly useful to de

Debugging manhole for python applications.

Overview docs tests package Manhole is in-process service that will accept unix domain socket connections and present the stacktraces for all threads

(OLD REPO) Line-by-line profiling for Python - Current repo -

line_profiler and kernprof line_profiler is a module for doing line-by-line profiling of functions. kernprof is a convenient script for running either

Comments
  • Functionality questions

    Functionality questions

    Thanks for making this project. A non-interactive python debugger is exactly what I'm looking for. I'd like to use your project to support the Python probe for Source++. I've been able to successfully implement some portions of the functionality, but I'm having some trouble understanding some Python debugging relating things. I think some of this trouble might be alleviated with new functionality for nopdb and I'm more than willing to implement that functionality. My background is with JVM languages so I'm just looking for a bit of guidance on how things work the Python way.

    Breakpoint doesn't hit on Flask app

    I don't think this issue is a nopdb issue as I'm unable to hit the breakpoint using import pdb; pdb.set_trace() in the main method either. I believe this issue is because of either compiled code within Flask or pdb is not hitting because of separate threads. Either way, I'm able to set breakpoints via PyCharm so it appears to be possible in some way. Hopefully, it's just a threading thing and breakpoints can be placed for every thread. Do you know how this functionality could be implemented with nopdb?

    Here is some code that shows a breakpoint that's never hit:

    import os
    
    from flask import Flask
    from nopdb import nopdb
    
    app = Flask(__name__)
    
    
    @app.route('/')
    def hello():
        provider = str(os.environ.get('PROVIDER', 'world'))
        return 'Hello ' + provider + '!'
    
    
    if __name__ == '__main__':
        with nopdb.breakpoint(file="FlaskTest.py", line=12) as bp:
            bp.exec("print(\"provider: \" + provider)")
            port = int(os.environ.get('PORT', 5000))
            app.run(host='0.0.0.0', port=port)
    
    

    Ability to add breakpoints without calling code via context manager

    This might be related to the issue above, but how do you add breakpoints to code that you don't want to call via the context manager? As in, code that is already running or that will run by itself sometime in the future? I have a small example here where I want to set a breakpoint before a function is called in the future via a delay. Can nopdb address this type of scenario?

    Code which shows a breakpoint that's never hit:

    import threading
    
    from nopdb import nopdb
    
    
    def fun():
        x = 5
        y = x
    
    
    if __name__ == '__main__':
        start_time = threading.Timer(10, fun)
        start_time.start()
    
       # I'd like to set the breakpoint ahead of time and just output x once it's hit
        with nopdb.breakpoint(file="SeparateThread.py", line=8) as bp:
            bp.exec("print(\"x: \" + str(x))")
    
    

    Does cpdb make sense to extend?

    This might be a bit off-topic, but while looking into Python debuggers I ran across https://pypi.org/project/CPdb/. Does this project make sense to extend for nopdb? The reason I ask is because of the claim cpdb makes about "long-running" applications running better with it. I would like to use nopdb in "long-running" applications so I'm curious if it makes sense to integrate the technologies.

    opened by BFergerson 0
Releases(v0.2.0)
Owner
Ondřej Cífka
PhD candidate @ Télécom Paris. Working on music style transfer
Ondřej Cífka
The official code of LM-Debugger, an interactive tool for inspection and intervention in transformer-based language models.

LM-Debugger is an open-source interactive tool for inspection and intervention in transformer-based language models. This repository includes the code

Mor Geva 96 Sep 19, 2022
pdb++, a drop-in replacement for pdb (the Python debugger)

pdb++, a drop-in replacement for pdb What is it? This module is an extension of the pdb module of the standard library. It is meant to be fully compat

null 988 Sep 29, 2022
Full-screen console debugger for Python

PuDB: a console-based visual debugger for Python Its goal is to provide all the niceties of modern GUI-based debuggers in a more lightweight and keybo

Andreas Klöckner 2.5k Sep 26, 2022
Debugger capable of attaching to and injecting code into python processes.

DISCLAIMER: This is not an official google project, this is just something I wrote while at Google. Pyringe What this is Pyringe is a python debugger

Google 1.6k Sep 23, 2022
pdb++, a drop-in replacement for pdb (the Python debugger)

pdb++, a drop-in replacement for pdb What is it? This module is an extension of the pdb module of the standard library. It is meant to be fully compat

null 987 Sep 23, 2022
Graphical Python debugger which lets you easily view the values of all evaluated expressions

birdseye birdseye is a Python debugger which records the values of expressions in a function call and lets you easily view them after the function exi

Alex Hall 1.5k Sep 19, 2022
Voltron is an extensible debugger UI toolkit written in Python.

Voltron is an extensible debugger UI toolkit written in Python. It aims to improve the user experience of various debuggers (LLDB, GDB, VDB an

snare 5.8k Sep 23, 2022
Tracing instruction in lldb debugger.Just a python-script for lldb.

lldb-trace Tracing instruction in lldb debugger. just a python-script for lldb. How to use it? Break at an address where you want to begin tracing. Im

null 147 Sep 4, 2022
Visual Interaction with Code - A portable visual debugger for python

VIC Visual Interaction with Code A simple tool for debugging and interacting with running python code. This tool is designed to make it easy to inspec

Nathan Blank 1 Nov 16, 2021
An improbable web debugger through WebSockets

wdb - Web Debugger Description wdb is a full featured web debugger based on a client-server architecture. The wdb server which is responsible of manag

Kozea 1.6k Sep 18, 2022