I think this is related to #53 (cc/ @jamespic) . I am consistently seeing Pyflame hang when profiling threads on my Chromebook. This is running Debian Jessie in a chroot, and the host kernel is 3.14.0. IIRC I did test this branch on a regular Jessie host, so I think it might be related to the ChromeOS kernel? I'll have to check.

What I am seeing is that it goes to pause both threads in a Python process running test_threaded_sleeper.py, and gets stuck on the second thread. By "stuck" I mean the call to wait() that is made after the PTRACE_ATTACH never completes. The full stack is like:

(gdb) bt
#0  0x00007f0358495ab2 in __libc_wait (stat_loc=0x0) at ../sysdeps/unix/sysv/linux/wait.c:30
#1  0x00000000004065eb in pyflame::PtraceAttach (pid=pid@entry=26554) at ptrace.cc:42
#2  0x00000000004079a1 in PauseChildThreads (pid=26553) at ptrace.cc:209
#3  pyflame::PtraceCallFunction (pid=26553, addr=4794543) at ptrace.cc:222
#4  0x000000000040a5f1 in pyflame::PyFrob::DetectPython (this=this@entry=0x7fffd588a130) at pyfrob.cc:135
#5  0x0000000000403845 in main (argc=<optimized out>, argv=<optimized out>) at pyflame.cc:268

This is a blocker to tag a new release.

Bleeding edge I know. Testing with Ubuntu 16.04 x64,

virtualenv -p python3.6 env
source env/bin/activate
pyflame -t python -c 'println(sum(i for i in range(0, 100000)))'

I only get (idle) 2 out of it. The same invocation works for python 3.5.2 and 2.7 on the same system. Attaching to a 3.6.1 process also only shows idle time. Sometimes, I instead get Failed to PTRACE_PEEKDATA at 0x10: Input/output error with different running processes, but I haven't figured out how to reproduce that.

This is how you do it:

• each interpreter has a field called tstate_head which is the head of a linked list of thread states
• each thread state has a field called next which is a pointer to the next thread state
• the last thread state has a null pointer for next

Likewise:

• each thread state has a back reference to the interpreter state in a field called interp
• there's a single linked list of interpreters whose head is the static symbol interp_head

In the single-threaded case there is one interpreter and one thread state.

In the generic case the way you enumerate the thread states is:

• locate _PyThreadState_Current
• follow interp up to the interpreter
• follow tstate_head to the first thread state (which will always be the same as _PyThreadState_Current for a single-threaded program, but could be different for a multi-threaded program)
• follow the next field until NULL is encountered

It should be fine to ignore the multiple interpreter thing, no one really uses that feature nowadays.

pyflame 1.5.0 (compiled from d1a76174e6b570c7e98af79a694f8769271f922a)
Python 2.7.12
$ uname -a
Linux test-VirtualBox 4.10.0-28-generic #32~16.04.2-Ubuntu SMP Thu Jul 20 10:19:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
(I had the same problem on 14 ubuntu, IIRC)

$ cat pyflametest.py
import time

def sleep300 ():
    time.sleep(.300)

def sleep700 ():
    time.sleep(.700)

def main ():
    t1 = time.time()
    while True:
        if time.time() - t1 >= 5:
            return
        sleep300()
        sleep700()

main()

works:
$ pyflame -t python pyflametest.py 
(idle) 3470
/usr/lib/python2.7/site.py:<module>:563;/usr/lib/python2.7/site.py:main:545;/usr/lib/python2.7/site.py:addusersitepackages:272;/usr/lib/python2.7/site.py:getusersitepackages:247;/usr/lib/python2.7/site.py:getuserbase:237;/usr/lib/python2.7/sysconfig.py:get_config_var:582;/usr/lib/python2.7/sysconfig.py:get_config_vars:509;/usr/lib/python2.7/re.py:<module>:105;/usr/lib/python2.7/sre_compile.py:<module>:15;/usr/lib/python2.7/sre_parse.py:<module>:706 1

crashes:
test@test-VirtualBox:~$ pyflame --threads -t python pyflametest.py
terminate called after throwing an instance of 'pyflame::PtraceException'
  what():  Failed to PTRACE_GETREGS: No such process
Aborted (core dumped)

$ make test
./runtests.sh
Running test suite against Python 2.7.12
.....s..............
19 passed, 1 skipped in 13.38 seconds
Running test suite against Python 3.5.2
Already using interpreter /usr/bin/python3
....................
20 passed in 14.47 seconds

Hi,

I'm getting "Target is Python 0, which is not supported by this pyflame build." error when running pyflame against uwsgi (despite what the issue #23 says, that it should be working).

OS: Ubuntu Xenial 16.04 Python: 2.7 uwsgi: 2.0.14 pyflame: 1.2.1

Steps to reproduce:

• the same as #23

Running uwsgi:

$ uwsgi --workers 1 --module app:app --http-socket=127.0.0.1:8011
...
Python version: 2.7.12 (default, Nov 19 2016, 06:48:10)  [GCC 5.4.0 20160609]
...

Running pyflame:

$ sudo ./pyflame/src/pyflame `pgrep uwsgi`
Target is Python 0, which is not supported by this pyflame build.
$

uwsgi really is loading libpython2.7

$ sudo lsof -p `pgrep uwsgi` | grep libpython
uwsgi   11149 ubuntu  mem       REG  253,1  3582904  19947 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
$

Looking at the src/pyfrob.cc, you seem to fallback to "libpython2.7.so", which does exist on my system:

$ ldconfig -p | grep libpython
	libpython3.5m.so.1.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0
	libpython2.7.so.1.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
	libpython2.7.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpython2.7.so
$

I have also tried uwsgi options --enable-threads, --single-interpreter, --master/workers with no luck.

Currently Pyflame works with Python 3 only as long as the file names are ASCII. If the file names include Unicode characters, Pyflame will fail to decode them.

Some Background

If you download the source code for Python, you can find the implementation of strings (actually unicode objects) in Objects/unicodeobject.c. There are a few other files, but this is the main one.

A unicode object in Python actually has one of three representations; it can be any of the following:

• PyASCIIObject
• PyCompactUnicodeObject
• PyUnicodeObject

Which one is actually used depends on what codepoints are actually in the string. If the string just contains ASCII characters a compact PyASCIIObject representation will be used. This representation is just like a Python 2 string: it has a size, and a pointer to bytes. Currently Pyflame assumes that string objects are really of type PyASCIIObject.

A Solution

A fix for this would include:

• logic to detect what underlying type the unicode object actually is
• logic for getting the raw bytes from the Unicode object
• logic for converting the raw bytes to UTF-8 for non-ASCII strings

You should be pretty familiar with the internals of unicodeobject.c before attempting to implement a fix here.

As of this writing, the code here should probably go into src/frob.cc which is where the existing string handling code is. Although depending on the complexity of the solution, it may make sense to break out into another file.

I'm using python 3.5 , when I execute the command "src/pyflame -s 60 -r 0.01 -p 25192 | /FlameGraph/flamegraph.pl > test_flame.svg" The error 'Failed to locate libpython within timeout period' was raised. Is that a compatible issue?

This adds a new target lib during compilation which includes the python 3.6 libraries. There were some symbols added which broke pyflame for python 3.6. See here

Tried it out against 3.6.1, 3.4, and 2.7.

When profiling a long lived multithreaded python process, sometimes the stack can't be read correctly.

This change simply handles such errors by logging the failure to stderr, and continues sampling.

Doesn't solve the underlying issue from #129, but it works around such failures (under the assumption it's ok to lose a handful of samples out of a few thousand)

The python version auto detection fails in some scenarios

Example: when trying to trace a uWSGI (python app server) process

This PR lets the user explicitly choose the version while the detection is fixed

How to use it:

# For python 2:
pyflame --python 2 ...

# For python 3:
pyflame --python 3 ...

# Or use the shorter form:
pyflame -p2 ...
pyflame -p3 ...

To be able to use -p2 or -p3 pyflame has to be compiled with ENABLE_PY2 and/or ENABLE_PY3 respectively

My application is a web application hosted by gunicorn, using gevent as async worker.

gunicorn pre-forked two processes, every one is a gevent worker (single thread). My attachment graph is profiling for one worker.

From the doc, since pyflame can't profile IO and c code, I think my left IDLE part should be related to it.

The middle part is my real business logic related code.

For the right long part, all in gevent.threadpool:_worker (task = task_queue.get()), I'm not familiar with gevent internal, is it normal? seems it's gevent main greenlet waiting for available tasks, My guess is because gevent is blocked by some CPU heavy code. But how can it be profiled with my real application code on the same level (If my assumption is right, the right part and middle part should be mixed together)

If only look at the middle part (my business logic code), does it only means CPU usage of my call stack? For example I have a very complex function, it will do both heavy CPU work and call a slow rpc service (waiting for IO). From pyflame, the waiting for IO part should be in IDLE, so my rpc part on flame graph will be narrow, is my understanding right?

Thanks for any advice.

I followed the official docs for the installation on ubuntu. First i installed all dependencies:

sudo apt-get install autoconf automake autotools-dev g++ pkg-config python-dev python3-dev libtool make

then i executed

./autogen.sh
./configure 

so far everything was fine, then i executed the make command

Making all in src
make[1]: Entering directory '/home/smartcow/PythonProfiler/pyflame/src'
make  all-am
make[2]: Entering directory '/home/smartcow/PythonProfiler/pyflame/src'
  CXX      libfrob26_la-frob26.lo
  CXXLD    libfrob26.la
ar: `u' modifier ignored since `D' is the default (see `U')
  CXX      libfrob36_la-frob36.lo
  CXXLD    libfrob36.la
ar: `u' modifier ignored since `D' is the default (see `U')
  CXX      aslr.o
  CXX      frame.o
  CXX      thread.o
  CXX      namespace.o
  CXX      posix.o
  CXX      prober.o
  CXX      ptrace.o
ptrace.cc: In function ‘user_regs_struct pyflame::PtraceGetRegs(pid_t)’:
ptrace.cc:127:14: error: ‘PTRACE_GETREGS’ was not declared in this scope
   if (ptrace(PTRACE_GETREGS, pid, 0, &regs)) {
              ^~~~~~~~~~~~~~
ptrace.cc:127:14: note: suggested alternative: ‘PTRACE_GETREGSET’
   if (ptrace(PTRACE_GETREGS, pid, 0, &regs)) {
              ^~~~~~~~~~~~~~
              PTRACE_GETREGSET
ptrace.cc: In function ‘void pyflame::PtraceSetRegs(pid_t, user_regs_struct)’:
ptrace.cc:136:14: error: ‘PTRACE_SETREGS’ was not declared in this scope
   if (ptrace(PTRACE_SETREGS, pid, 0, &regs)) {
              ^~~~~~~~~~~~~~
ptrace.cc:136:14: note: suggested alternative: ‘PTRACE_SETREGSET’
   if (ptrace(PTRACE_SETREGS, pid, 0, &regs)) {
              ^~~~~~~~~~~~~~
              PTRACE_SETREGSET
Makefile:512: recipe for target 'ptrace.o' failed
make[2]: *** [ptrace.o] Error 1
make[2]: Leaving directory '/home/smartcow/PythonProfiler/pyflame/src'
Makefile:369: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/home/smartcow/PythonProfiler/pyflame/src'
Makefile:475: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

ptrace might be causing any issues so i tried below command

sudo apt-get update
sudo apt-get install python-ptrace

Reading package lists... Done
Building dependency tree       
Reading state information... Done
python-ptrace is already the newest version (0.7-1).
0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded.

still the error is same.

1.pyflame -v pyflame 1.6.6 linux-gnu x86_64 (ABI list: 26) 2.python2.7 3.when execute pyflame -p PID, prompt "Failed to locate libpython within timeout period."

Pyflame causes segfaults in OSQP (https://osqp.org/) when solver is run with polish=1.

• [ X ] Include the output of pyflame -v pyflame 1.6.6 linux-gnu x86_64 (ABI list: 26)
• [ X ] Include the exact version of the Python interpreter you are profiling. Python 2.7.16
• [ X ] Include the exact text of any error messages. From core dump:

Program terminated with signal 11, Segmentation fault.
#0  0x00007f4db095b4c0 in QDLDL_factor () from /data/homes/raidb/home/msokolov/py2/lib/python2.7/site-packages/osqp/_osqp.so
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 glibc-2.17-292.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_7.2.x86_64 libcom_err-1.42.9-16.el7.x86_64 libffi-3.0.13-18.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 pcre-8.32-17.el7.x86_64 python27-python-libs-2.7.16-6.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt
#0  0x00007f4db095b4c0 in QDLDL_factor () from /data/homes/raidb/home/msokolov/py2/lib/python2.7/site-packages/osqp/_osqp.so
#1  0x00007f4db095ab35 in init_linsys_solver_qdldl () from /data/homes/raidb/home/msokolov/py2/lib/python2.7/site-packages/osqp/_osqp.so
#2  0x00007f4db09569e6 in polish () from /data/homes/raidb/home/msokolov/py2/lib/python2.7/site-packages/osqp/_osqp.so
#3  0x00007f4db095472f in osqp_solve () from /data/homes/raidb/home/msokolov/py2/lib/python2.7/site-packages/osqp/_osqp.so
#4  0x00007f4db094fb77 in ?? () from /data/homes/raidb/home/msokolov/py2/lib/python2.7/site-packages/osqp/_osqp.so
#5  0x00007f4dbc2b5e2f in PyEval_EvalFrameEx () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#6  0x00007f4dbc2b791d in PyEval_EvalCodeEx () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#7  0x00007f4dbc2b407b in PyEval_EvalFrameEx () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#8  0x00007f4dbc2b791d in PyEval_EvalCodeEx () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#9  0x00007f4dbc2b7a22 in PyEval_EvalCode () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#10 0x00007f4dbc2d13ff in ?? () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#11 0x00007f4dbc2d25de in PyRun_FileExFlags () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#12 0x00007f4dbc2d3849 in PyRun_SimpleFileExFlags () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#13 0x00007f4dbc2e4c4f in Py_Main () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#14 0x00007f4dbb4fd505 in __libc_start_main () from /lib64/libc.so.6
#15 0x000000000040066e in _start ()

Disassembly:

 0x00007f4db095b49c <+268>:   xor    %ebp,%ebp
   0x00007f4db095b49e <+270>:   mov    0x8(%rax,%r10,8),%rbx
   0x00007f4db095b4a3 <+275>:   mov    (%rax,%r10,8),%rcx
   0x00007f4db095b4a7 <+279>:   cmp    %rcx,%rbx
   0x00007f4db095b4aa <+282>:   jle    0x7f4db095b665 <QDLDL_factor+725>
   0x00007f4db095b4b0 <+288>:   mov    %r9,-0x28(%rsp)
   0x00007f4db095b4b5 <+293>:   mov    -0x18(%rsp),%r13
   0x00007f4db095b4ba <+298>:   mov    -0x8(%rsp),%r9
   0x00007f4db095b4bf <+303>:   nop
=> 0x00007f4db095b4c0 <+304>:   mov    (%r9,%rcx,8),%rax
   0x00007f4db095b4c4 <+308>:   movsd  0x0(%r13,%rcx,8),%xmm0
   0x00007f4db095b4cb <+315>:   cmp    %r10,%rax
   0x00007f4db095b4ce <+318>:   je     0x7f4db095b6c0 <QDLDL_factor+816>
   0x00007f4db095b4d4 <+324>:   lea    (%r11,%rax,1),%rdx
   0x00007f4db095b4d8 <+328>:   movsd  %xmm0,(%rsi,%rax,8)
   0x00007f4db095b4dd <+333>:   cmpb   $0x0,(%rdx)
   0x00007f4db095b4e0 <+336>:   jne    0x7f4db095b57d <QDLDL_factor+493>
   0x00007f4db095b4e6 <+342>:   movb   $0x1,(%rdx)
   0x00007f4db095b4e9 <+345>:   mov    %rax,(%r8)
   0x00007f4db095b4ec <+348>:   mov    (%r12,%rax,8),%rax

Registers:

(gdb) info registers
rax            0x3437050        54751312
rbx            0xb129   45353
rcx            0xae82   44674
rdx            0x3432b18        54733592
rsi            0x342e8b0        54716592
rdi            0x367a730        57124656
rbp            0x0      0x0
rsp            0x7ffebc323f68   0x7ffebc323f68
r8             0x3426118        54681880
r9             0x7f4dbc7cf002   139971851513858
r10            0x84f    2127
r11            0x32d9370        53318512
r12            0x34151d0        54612432
r13            0x34cc420        55362592
r14            0x3421d50        54664528
r15            0x340c8d8        54577368
rip            0x7f4db095b4c0   0x7f4db095b4c0 <QDLDL_factor+304>
eflags         0x10202  [ IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0

Operating system:

CentOS Linux release 7.7.1908 (Core)

Kernel:

Linux 3.10.0-957.12.2.el7.x86_64

Repro:

# portfolio.py
# python==2.7.16
import osqp  # osqp==0.4.1
import numpy as np  # numpy==1.14.3
import scipy as sp  # scipy==1.1.0
from scipy import sparse
import threading
import time

# Generate problem data
sp.random.seed(1)
n = 1000
k = 100
F = sparse.random(n, k, density=0.7, format='csc')
D = sparse.diags(np.random.rand(n) * np.sqrt(k), format='csc')
mu = np.random.randn(n)
gamma = 1

# OSQP data
P = sparse.block_diag([D, sparse.eye(k)], format='csc')
q = np.hstack([-mu / (2*gamma), np.zeros(k)])
A = sparse.vstack([
        sparse.hstack([F.T, -sparse.eye(k)]),
        sparse.hstack([sparse.csc_matrix(np.ones((1, n))), sparse.csc_matrix((1, k))]),
        sparse.hstack((sparse.eye(n), sparse.csc_matrix((n, k))))
    ], format='csc')
l = np.hstack([np.zeros(k), 1., np.zeros(n)])
u = np.hstack([np.zeros(k), 1., np.ones(n)])

# Create an OSQP object
prob = osqp.OSQP()
# Setup workspace
prob.setup(P, q, A, l, u,
    polish=1) #  polish=1 causes segmentation faults under pyflame

# Solve problem
for i in range(1000000):
    res = prob.solve()

Run portfolio.py:

$ ulimit -c unlimited 
$ python portfolio.py &> /dev/null

Run pyflame several times:

$ pyflame --pid=$(ps aux | grep python | grep portfolio | awk '{print $2}') -o pyflame.prof -s 5 --threads

Get a segmentation falut.

$ python portfolio.py &> /dev/null
[1]    1225 segmentation fault (core dumped)  python portfolio.py &> /dev/null

$ pyflame --pid=$(ps aux | grep python | grep portfolio | awk '{print $2}') -o pyflame.prof -s 5 --threads
Unexpected ptrace(2) exception: waitpid() indicated a WIFSTOPPED process, but got unexpected signal 11

Make sure that these boxes are checked before submitting your issue:

• [x] Include the output of pyflame -v
• pyflame 1.6.6 linux-gnu x86_64 (ABI list: 26 34 36)
• [x] Include the exact version of the Python interpreter you are profiling.
• Python 3.5.3
• [x] Include the exact text of any error messages.
• I don't have any error message really.

I'm using the following python app:

• https://github.com/odoo/odoo

It has a parameter called workers Using this feature many python process PID are started.

pyflame currently doesn't support -p PID1 -p PID2 -p PID3 since that I just tested and it is not collecting the full information of a particular process.

I need using a

pyflame ... -o test_1.flame -p PID1
pyflame ... -o test_2.flame -p PID2
pyflame ... -o test_3.flame -p PID3
pyflame ... -o test_4.flame -p PID4

After I just transform all output to svg using: flamegraph.pl test_*.flame >test.svg

Now It collects the full information of the particular process.

IMHO It could be a good feature if pyflame supports multiple PID with -p

The last commit is already ~1 year old. I wrote a mail to [email protected] and asked if somebody from uber still maintains pyflame. Let's see if there is an answer...

You can use docker to build pyflame

sudo docker build --tag pyflame .
sudo docker run -it -v $(pwd):/root/pyflame pyflame /bin/bash -c "cd /root/pyflame;./autogen.sh;./configure;make"

This will also produce the executable at src/pyflame, which support py2.6/2.7/3.4/3.5/3.6