Trace any Python program, anywhere!

Overview

lptrace

lptrace is strace for Python programs. It lets you see in real-time what functions a Python program is running. It's particularly useful to debug weird issues on production.

For example, let's debug a non-trivial program, the Python SimpleHTTPServer. First, let's run the server:

[email protected]:/vagrant$ python -m SimpleHTTPServer 8080 &
[1] 1818
[email protected]:/vagrant$ Serving HTTP on 0.0.0.0 port 8080 ...

Now let's connect lptrace to it:

[email protected]:/vagrant$ sudo python lptrace -p 1818
...
fileno (/usr/lib/python2.7/SocketServer.py:438)
meth (/usr/lib/python2.7/socket.py:223)

fileno (/usr/lib/python2.7/SocketServer.py:438)
meth (/usr/lib/python2.7/socket.py:223)

_handle_request_noblock (/usr/lib/python2.7/SocketServer.py:271)
get_request (/usr/lib/python2.7/SocketServer.py:446)
accept (/usr/lib/python2.7/socket.py:201)
__init__ (/usr/lib/python2.7/socket.py:185)
verify_request (/usr/lib/python2.7/SocketServer.py:296)
process_request (/usr/lib/python2.7/SocketServer.py:304)
finish_request (/usr/lib/python2.7/SocketServer.py:321)
__init__ (/usr/lib/python2.7/SocketServer.py:632)
setup (/usr/lib/python2.7/SocketServer.py:681)
makefile (/usr/lib/python2.7/socket.py:212)
__init__ (/usr/lib/python2.7/socket.py:246)
makefile (/usr/lib/python2.7/socket.py:212)
__init__ (/usr/lib/python2.7/socket.py:246)
handle (/usr/lib/python2.7/BaseHTTPServer.py:336)
handle_one_request (/usr/lib/python2.7/BaseHTTPServer.py:301)
^CReceived Ctrl-C, quitting
[email protected]:/vagrant$

You can see that the server is handling the request in real time! After pressing Ctrl-C, the trace is removed and the program execution resumes normally.

How it works

gdb is an awesome debugger. It lets you attach to any running program, as long as you're root. It also lets you call any C function this program exposes.

What's interesting is that among the C functions the Python interpreter exposes, one function PyRun\_SimpleString, lets you run a single expression of Python code.

We use this function to ask the Python process to read a temporary file lptrace created. This file contains a hook to the sys.settrace function, which allows us to get notified whenever a function is called.

Finally, we need to output the tracing data somewhere. We could do this in the program we're tracing but that wouldn't be very useful. Instead, we write it to a FIFO so that lptrace can display it in its own window.

That's about it. I encourage you to read the source --- it's short and pretty simple!

Running lptrace

lptrace was written to be run on production servers. Because of this, you only need lptrace to run the whole program. You can get lptrace by installing the lptrace PyPI package or simply by downloading the main source file.

Usage

Tracing a Python program

sudo python lptrace -p <process_id>

Getting a pdb prompt inside a Python program

Sometimes it's useful to get a pdb prompt inside a Python program. Note that this requires that the Python program you're attaching to has access to stdin.

sudo python lptrace -p <process_id> -d

Requirements

lptrace requires Python 2.7.x and GDB 7.x. It has been tested on Linux successfully, and it should run on most recent Unices.

Issues

Please open a ticket here

Security

lptrace is a debugging tool. It uses temporary files, so it may be vulnerable to some race conditions. Caveat emptor!

Special Thanks

I'd like to thank the Pyrasite project for coming up with the idea to inject code into a running Python process.

Issues
  • seems not work well.

    seems not work well.

    in one terminal run the script: from time import sleep for i in range(10000): print "in loop" sleep(2)

    in another terminal run the: sudo lptrace -p PID_OF_ABOVE_SCRIPT

    no output in the second terminal, could you help to take a look why? Thanks in advance.

    opened by jinchenx 4
  • Use launcher script and keep main code in a .py file

    Use launcher script and keep main code in a .py file

    This somewhat trivial PR allows lptrace to be called from the command line while keeping the body of the code in lptrace.py. It provides a small launcher script as the file lptrace and moves the code to the .py file. No other changes have been made.

    opened by ajyoon 4
  • Python3 mode

    Python3 mode

    I haven't gotten the script to work with Python3 yet. Things I did so far:

    • print-statements (obviously)
    • octal literals with 0o666 instead of 0666
    • cmd = 'exec(open(\\"{}\\", "r").read())'.format(name) because Python3 doesn't have execfile
    • tmp.write(script.encode()) due to bytes/string conversion

    I'm guessing that I'm either screwing up with bytes/strings/file reads or the Python3 interface differs in a way that doesn't raise an exception explaining the problem. Anybody with an insight here?

    opened by rixx 4
  • Failed to run on macOS

    Failed to run on macOS

    First of all, thanks for providing this little cool script to help debugging Python programs.

    I've been using it successfully on ubuntu server, but when I tried this on my Mac, sudo python lptrace -p 96365 hanged there and the target program was failed by segmentation fault:

    $ python -m SimpleHTTPServer 10000
    Serving HTTP on 0.0.0.0 port 10000 ...
    [1]    96365 segmentation fault  python -m SimpleHTTPServer 10000
    

    I was using Python 2.7.12 installed by homebrew. My system is OS X 10.11.6.

    opened by reorx 2
  • Check if gdb exists

    Check if gdb exists

    Right now if you launch lptrace without having gdb on your system, it will just end up waiting doing nothing. I guess it would be better if an error was returned.

    opened by ramnes 1
  • Publish on PyPI

    Publish on PyPI

    As easy integration with production environments is a concern, this would be much easier to integrate into workflows if it was on PyPI and could be installed with pip. Even better would be to have pip install lptrace register it such that it can be called from the command line without providing an explicit path to the script (as pytest and many others do).

    opened by ajyoon 1
  • Python version should be explicit

    Python version should be explicit

    With more and more distributions switching to python3 by default and python2 being clearly deprecated putting a little shebang

    #!/bin/env python2
    

    sounds like a reasonable thing to do. Sure it's little trouble to write an alias to python2 lptrace.py but it shouldn't be necessary.

    opened by cym13 1
  • Add option to print stacktrace of all active threads

    Add option to print stacktrace of all active threads

    1. replase optparse with argparse because we introduced mutually exclusive options
    2. refactor runfile so we can make block FIFO works
    3. add -s option to print stacktrace for all active threads
    opened by krizex 0
  • detaching results in SIGABRT

    detaching results in SIGABRT

    I tried running this trivial Python program with lptrace:

    $ cat loop_forever.py
    #! /usr/bin/env python
    
    import os
    print os.getpid()
    
    while True:
        pass
    

    I ran it:

    $ ./loop_forever.py 
    25116
    

    then I ran lptrace in another terminal.

    $ sudo python lptrace -p 25116
    

    I had to use Ctrl-C twice to get it to quit.

    $ sudo python lptrace -p 25116
    ^CReceived Ctrl-C, quitting
    ^CReceived Ctrl-C, quitting
    

    After this, loop_forever.py exited with signal 6 (SIGABRT):

    $ ./loop_forever.py 
    25116
    Fatal Python error: This thread state must be current when releasing
    Aborted (core dumped)
    

    This may be the same bug as #11, but I wanted to demonstrate it is easy to reproduce on a trivial program, not just Guake.

    I am on Debian 9.3 (Stretch), running Python 2.7.13, GDB version 7.12-6, and the latest Github version of lptrace.

    $ lsb_release -a
    No LSB modules are available.
    Distributor ID:	Debian
    Description:	Debian GNU/Linux 9.3 (stretch)
    Release:	9.3
    Codename:	stretch
    $ python --version
    Python 2.7.13
    $ gdb --version | head -n 1
    GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
    $ git rev-parse HEAD
    537053c5fd6c3d84d0fb73d675fe926d470c82d5
    

    Full backtrace of dumped core file is below:

    Core was generated by `python ./loop_forever.py'.
    Program terminated with signal SIGABRT, Aborted.
    #0  __GI_raise ([email protected]=6) at ../sysdeps/unix/sysv/linux/raise.c:51
    51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
    set = {__val = {0, 140451948041416, 140726353628400, 4294967301, 140451948040560, 140726353628384, 93833867754558, 252833149, 4294967295, 1, 140451930217944, 140451947630592, 93833869879752, 93833869879750, 140451946566352, 8}}
    pid = <optimized out>
    tid = <optimized out>
    sig = 6
      Id   Target Id         Frame 
    * 1    Thread 0x7fbd84723700 (LWP 31965) __GI_raise ([email protected]=6) at ../sysdeps/unix/sysv/linux/raise.c:51
    
    Thread 1 (Thread 0x7fbd84723700 (LWP 31965)):
    #0  __GI_raise ([email protected]=6) at ../sysdeps/unix/sysv/linux/raise.c:51
            set = {__val = {0, 140451948041416, 140726353628400, 4294967301, 140451948040560, 140726353628384, 93833867754558, 252833149, 4294967295, 1, 140451930217944, 140451947630592, 93833869879752, 93833869879750, 140451946566352, 8}}
            pid = <optimized out>
            tid = <optimized out>
    #1  0x00007fbd836b73fa in __GI_abort () at abort.c:89
            save_stage = 2
            act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0, 93833891493728, 0, 1, 140451943638619, 93833891511584, 93833868582196, 93833891493728, 0, 0, 2880, 0, 140451933972304, 140451933967648, 140451933972304, 140451947624192}}, sa_flags = -2089902068, sa_restorer = 0x7fbd83a1d6e0 <stderr>}
            sigs = {__val = {32, 0 <repeats 15 times>}}
    #2  0x00005557665fb8b2 in Py_FatalError (msg=<optimized out>) at ../Python/pythonrun.c:1700
    No locals.
    #3  0x000055576664aff8 in PyGILState_Release (oldstate=PyGILState_LOCKED) at ../Python/pystate.c:635
            tcur = <optimized out>
    #4  0x00007ffd68518f0f in ?? ()
    No symbol table info available.
    #5  0x0000000000000300 in ?? ()
    No symbol table info available.
    #6  0xcc00000000000330 in ?? ()
    No symbol table info available.
    #7  0x000000000000000c in ?? ()
    No symbol table info available.
    #8  0x4fab7054cad2ce00 in ?? ()
    No symbol table info available.
    #9  0x0000000000000002 in ?? ()
    No symbol table info available.
    #10 0x00007fbd8465e5d0 in ?? ()
    No symbol table info available.
    #11 0x00007fbd845fbf80 in ?? ()
    No symbol table info available.
    #12 0x0000000000000000 in ?? ()
    No symbol table info available.
    
    opened by nbeaver 0
  • Lptrace just hang

    Lptrace just hang

    When I use lptrace, i just simply run python lptrace -p 21640. But this command just hang forever. When i add some print log in lptrace like this ` signal.signal(signal.SIGINT, sigint_handler)

    print("bbbbbbbbbb")
    with open(fifo_name) as fd:
        print("ssssssssssss")
        while True:
            print("aaaaaaaa")
            data = fd.read()
            print(111111111111)
            if data != '':
                print data
    
    print("mmmmmmmm")
    

    ` it will only print "bbbbbbb", but not any other logs. Did i miss anything

    opened by demonguy 2
  • Detaching from Guake terminal crashes Guake process.

    Detaching from Guake terminal crashes Guake process.

    I attached to Guake and then pressed Ctrl-C. Guake is closed.

    Expected result: guake continue to work

    (guake version - 0.8.8, lptrace - 1.0) https://dl.dropboxusercontent.com/1/view/k5kard6wqahrskb/Apps/Shutter/%C3%90%C2%92%C3%91%C2%8B%C3%90%C2%B4%C3%90%C2%B5%C3%90%C2%BB%C3%90%C2%B5%C3%90%C2%BD%C3%90%C2%B8%C3%90%C2%B5_003.png

    opened by dmitryTsatsarin 2
  • Funky Zero length field name error.

    Funky Zero length field name error.

    Traceback (most recent call last):
      File "lptrace", line 116, in <module>
        main()
      File "lptrace", line 112, in main
        strace(pid)
      File "lptrace", line 50, in strace
        runfile(pid, trace_code)
      File "lptrace", line 39, in runfile
        cmd = 'execfile(\\"{}\\")'.format(name)
    ValueError: zero length field name in forma
    

    I went through the instructions in your README and this error was returned. Any thoughts?

    opened by Ryanb58 2
VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.

VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.

null 2k May 26, 2022
Trace all method entries and exits, the exit also prints the return value, if it is of basic type

Trace all method entries and exits, the exit also prints the return value, if it is of basic type. The apk must have set the android:debuggable="true" flag.

Kurt Nistelberger 6 Mar 22, 2022
An x86 old-debug-like program.

An x86 old-debug-like program.

Pablo Niklas 1 Jan 10, 2022
pdb++, a drop-in replacement for pdb (the Python debugger)

pdb++, a drop-in replacement for pdb What is it? This module is an extension of the pdb module of the standard library. It is meant to be fully compat

null 899 May 20, 2022
Full-screen console debugger for Python

PuDB: a console-based visual debugger for Python Its goal is to provide all the niceties of modern GUI-based debuggers in a more lightweight and keybo

Andreas Klöckner 2.4k May 18, 2022
Debugging manhole for python applications.

Overview docs tests package Manhole is in-process service that will accept unix domain socket connections and present the stacktraces for all threads

Ionel Cristian Mărieș 327 May 5, 2022
Debugger capable of attaching to and injecting code into python processes.

DISCLAIMER: This is not an official google project, this is just something I wrote while at Google. Pyringe What this is Pyringe is a python debugger

Google 1.6k May 15, 2022
(OLD REPO) Line-by-line profiling for Python - Current repo ->

line_profiler and kernprof line_profiler is a module for doing line-by-line profiling of functions. kernprof is a convenient script for running either

Robert Kern 3.6k May 23, 2022
Monitor Memory usage of Python code

Memory Profiler This is a python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for pyth

Fabian Pedregosa 75 May 25, 2022
Sampling profiler for Python programs

py-spy: Sampling profiler for Python programs py-spy is a sampling profiler for Python programs. It lets you visualize what your Python program is spe

Ben Frederickson 8.5k May 19, 2022
🔥 Pyflame: A Ptracing Profiler For Python. This project is deprecated and not maintained.

Pyflame: A Ptracing Profiler For Python (This project is deprecated and not maintained.) Pyflame is a high performance profiling tool that generates f

Uber Archive 2.9k May 20, 2022
Visual profiler for Python

vprof vprof is a Python package providing rich and interactive visualizations for various Python program characteristics such as running time and memo

Nick Volynets 3.8k May 22, 2022
Parsing ELF and DWARF in Python

pyelftools pyelftools is a pure-Python library for parsing and analyzing ELF files and DWARF debugging information. See the User's guide for more deta

Eli Bendersky 1.4k May 25, 2022
pdb++, a drop-in replacement for pdb (the Python debugger)

pdb++, a drop-in replacement for pdb What is it? This module is an extension of the pdb module of the standard library. It is meant to be fully compat

null 901 May 25, 2022
Run-time type checker for Python

This library provides run-time type checking for functions defined with PEP 484 argument (and return) type annotations. Four principal ways to do type

Alex Grönholm 876 May 19, 2022
Graphical Python debugger which lets you easily view the values of all evaluated expressions

birdseye birdseye is a Python debugger which records the values of expressions in a function call and lets you easily view them after the function exi

Alex Hall 1.5k May 22, 2022
A powerful set of Python debugging tools, based on PySnooper

snoop snoop is a powerful set of Python debugging tools. It's primarily meant to be a more featureful and refined version of PySnooper. It also includ

Alex Hall 753 May 23, 2022
Inject code into running Python processes

pyrasite Tools for injecting arbitrary code into running Python processes. homepage: http://pyrasite.com documentation: http://pyrasite.rtfd.org downl

Luke Macken 2.6k May 20, 2022
Cyberbrain: Python debugging, redefined.

Cyberbrain1(电子脑) aims to free programmers from debugging.

laike9m 2.2k May 26, 2022