Debugging manhole for python applications.

Overview

Overview

docs Documentation Status
tests
Travis-CI Build Status Requirements Status
Coverage Status Coverage Status
package

Manhole is in-process service that will accept unix domain socket connections and present the stacktraces for all threads and an interactive prompt. It can either work as a python daemon thread waiting for connections at all times or a signal handler (stopping your application and waiting for a connection).

Access to the socket is restricted to the application's effective user id or root.

This is just like Twisted's manhole. It's simpler (no dependencies), it only runs on Unix domain sockets (in contrast to Twisted's manhole which can run on telnet or ssh) and it integrates well with various types of applications.

Documentation: http://python-manhole.readthedocs.org/en/latest/

Usage

Install it:

pip install manhole

You can put this in your django settings, wsgi app file, some module that's always imported early etc:

import manhole
manhole.install() # this will start the daemon thread

# and now you start your app, eg: server.serve_forever()

Now in a shell you can do either of these:

netcat -U /tmp/manhole-1234
socat - unix-connect:/tmp/manhole-1234
socat readline unix-connect:/tmp/manhole-1234

Socat with readline is best (history, editing etc). If your socat doesn't have readme try this.

Sample output:

$ nc -U /tmp/manhole-1234

Python 2.7.3 (default, Apr 10 2013, 06:20:15)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> dir()
['__builtins__', 'dump_stacktraces', 'os', 'socket', 'sys', 'traceback']
>>> print 'foobar'
foobar

Alternative client

There's a new experimental manhole-cli bin since 1.1.0, that emulates socat:

usage: manhole-cli [-h] [-t TIMEOUT] [-1 | -2 | -s SIGNAL] PID

Connect to a manhole.

positional arguments:
  PID                   A numerical process id, or a path in the form:
                        /tmp/manhole-1234

optional arguments:
  -h, --help            show this help message and exit
  -t TIMEOUT, --timeout TIMEOUT
                        Timeout to use. Default: 1 seconds.
  -1, -USR1             Send USR1 (10) to the process before connecting.
  -2, -USR2             Send USR2 (12) to the process before connecting.
  -s SIGNAL, --signal SIGNAL
                        Send the given SIGNAL to the process before
                        connecting.

Features

  • Uses unix domain sockets, only root or same effective user can connect.

  • Can run the connection in a thread or in a signal handler (see oneshot_on option).

  • Can start the thread listening for connections from a signal handler (see activate_on option)

  • Compatible with apps that fork, reinstalls the Manhole thread after fork - had to monkeypatch os.fork/os.forkpty for this.

  • Compatible with gevent and eventlet with some limitations - you need to either:

    • Use oneshot_on, or
    • Disable thread monkeypatching (eg: gevent.monkey.patch_all(thread=False), eventlet.monkey_patch(thread=False)

    Note: on eventlet you might need to setup the hub first to prevent circular import problems:

    import eventlet
    eventlet.hubs.get_hub()  # do this first
    eventlet.monkey_patch(thread=False)
  • The thread is compatible with apps that use signalfd (will mask all signals for the Manhole threads).

Options

manhole.install(
    verbose=True,
    verbose_destination=2,
    patch_fork=True,
    activate_on=None,
    oneshot_on=None,
    sigmask=manhole.ALL_SIGNALS,
    socket_path=None,
    reinstall_delay=0.5,
    locals=None,
    strict=True,
)
  • verbose - Set it to False to squelch the logging.
  • verbose_destination - Destination for verbose messages. Set it to a file descriptor or handle. Default is unbuffered stderr (stderr 2 file descriptor).
  • patch_fork - Set it to False if you don't want your os.fork and os.forkpy monkeypatched
  • activate_on - Set to "USR1", "USR2" or some other signal name, or a number if you want the Manhole thread to start when this signal is sent. This is desirable in case you don't want the thread active all the time.
  • thread - Set to True to start the always-on ManholeThread. Default: True. Automatically switched to False if oneshot_on or activate_on are used.
  • oneshot_on - Set to "USR1", "USR2" or some other signal name, or a number if you want the Manhole to listen for connection in the signal handler. This is desireable in case you don't want threads at all.
  • sigmask - Will set the signal mask to the given list (using signalfd.sigprocmask). No action is done if signalfd is not importable. NOTE: This is done so that the Manhole thread doesn't steal any signals; Normally that is fine because Python will force all the signal handling to be run in the main thread but signalfd doesn't.
  • socket_path - Use a specific path for the unix domain socket (instead of /tmp/manhole-<pid>). This disables patch_fork as children cannot reuse the same path.
  • reinstall_delay - Delay the unix domain socket creation reinstall_delay seconds. This alleviates cleanup failures when using fork+exec patterns.
  • locals - Names to add to manhole interactive shell locals.
  • daemon_connection - The connection thread is daemonic (dies on app exit). Default: False.
  • redirect_stderr - Redirect output from stderr to manhole console. Default: True.
  • strict - If True then AlreadyInstalled will be raised when attempting to install manhole twice. Default: True.

Environment variable installation

Manhole can be installed via the PYTHONMANHOLE environment variable.

This:

PYTHONMANHOLE='' python yourapp.py

Is equivalent to having this in yourapp.py:

import manhole
manhole.install()

Any extra text in the environment variable is passed to manhole.install(). Example:

PYTHONMANHOLE='onshot_on="USR2"' python yourapp.py

What happens when you actually connect to the socket

  1. Credentials are checked (if it's same user or root)
  2. sys.__std*__/sys.std* are redirected to the UDS
  3. Stacktraces for each thread are written to the UDS
  4. REPL is started so you can fiddle with the process

Known issues

  • Using threads and file handle (not raw file descriptor) verbose_destination can cause deadlocks. See bug reports: PyPy and Python 3.4.

SIGTERM and socket cleanup

By default Python doesn't call the atexit callbacks with the default SIGTERM handling. This makes manhole leave stray socket files around. If this is undesirable you should install a custom SIGTERM handler so atexit is properly invoked.

Example:

import signal
import sys

def handle_sigterm(signo, frame):
    sys.exit(128 + signo)  # this will raise SystemExit and cause atexit to be called

signal.signal(signal.SIGTERM, handle_sigterm)

Using Manhole with uWSGI

Because uWSGI overrides signal handling Manhole is a bit more tricky to setup. One way is to use "uWSGI signals" (not the POSIX signals) and have the workers check a file for the pid you want to open the Manhole in.

Stick something this in your WSGI application file:

from __future__ import print_function
import sys
import os
import manhole

stack_dump_file = '/tmp/manhole-pid'
uwsgi_signal_number = 17

try:
    import uwsgi

    if not os.path.exists(stack_dump_file):
        open(stack_dump_file, 'w')

    def open_manhole(dummy_signum):
        with open(stack_dump_file, 'r') as fh:
            pid = fh.read().strip()
            if pid == str(os.getpid()):
                inst = manhole.install(strict=False, thread=False)
                inst.handle_oneshot(dummy_signum, dummy_signum)

    uwsgi.register_signal(uwsgi_signal_number, 'workers', open_manhole)
    uwsgi.add_file_monitor(uwsgi_signal_number, stack_dump_file)

    print("Listening for stack mahole requests via %r" % (stack_dump_file,), file=sys.stderr)
except ImportError:
    print("Not running under uwsgi; unable to configure manhole trigger", file=sys.stderr)
except IOError:
    print("IOError creating manhole trigger %r" % (stack_dump_file,), file=sys.stderr)


# somewhere bellow you'd have something like
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
# or
def application(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain'), ('Content-Length', '2')])
    yield b'OK'

To open the Manhole just run echo 1234 > /tmp/manhole-pid and then manhole-cli 1234.

Requirements

OS: Linux, OS X
Runtime: Python 2.7, 3.4, 3.5, 3.6 or PyPy

Similar projects

Issues
  • Do not modify anything during fork+exec flow

    Do not modify anything during fork+exec flow

    Recently a reinstall_bind_delay was added to avoid binding of unix socket during fork+exec flow. However the socket is not the only issue in this flow. Changing process signal mask should not be done, as such changes may be inherited by the new process. Generally, since we don't know yet if the manhole is needed, we should do nothing.

    This patch changes the semantics of the reinstall_bind_delay so we do not make any modification to child process (except starting a thread) until the delay is finished.

    The user visible parameter was shortened to "reinstall_delay", as the user should not be conceded with the internals of the manhole. Internally the Manhole class call it "setup_delay", as this is the delay before setting up the manhole.

    opened by nirs 23
  • Use socket-bounded file wrapper

    Use socket-bounded file wrapper

    Hello,

    I faced with strange behavior when used this tool — random files/connections were closed after I inspecting session. After debug I found the root cause.

    os.fdopen-created file objects only bounded to the fd value. And they do call self.close() on garbage collection. So if such instance is GCed after closing of original socket — it may close another file/connection that reused same fd.

    Demo:

    import os, socket
    
    local_socket, remote_socket = socket.socketpair()
    print('Sockets FDs:', local_socket.fileno(), remote_socket.fileno())
    
    a_file_wrapper = os.fdopen(local_socket.fileno())
    
    local_socket.send(b'x')    # Successful send
    
    # Close original connection, open new
    local_socket.close(); remote_socket.close()
    
    new_local_socket, new_remote_socket = socket.socketpair()
    print('Sockets FDs:', new_local_socket.fileno(), new_remote_socket.fileno())
    
    new_local_socket.send(b'x')    # Successful send
    
    # Deletion of the file wrapper…
    del a_file_wrapper
    
    # … closes unrelated connection
    new_local_socket.send(b'x')    # OSError: [Errno 9] Bad file descriptor
    

    socket.makefile() creates fileobject that is bounded to exact socket instance, so it never closes unrelated fds.

    opened by anton-ryzhov 22
  • Fix failures with pypy on slow or overloaded build machines

    Fix failures with pypy on slow or overloaded build machines

    We used to do 500 iterations in the fork_exec test, which is not really needed. With no reinstall_delay, we need only 10-20 iterations to expose this issue, so running 50 iterations is just fine.

    opened by nirs 22
  • Changes for OS X (tested with OS X 10.9/Mavericks only)

    Changes for OS X (tested with OS X 10.9/Mavericks only)

    • use LOCAL_PEERCRED on OS X for checking credentials;
    • don't do setsockopt(SOL_SOCKET, SO_RCVBUF | SO_SNDBUF, 0) on OS X;– things are working properly still anyway, no apparent buffering is present
    opened by razzmatazz 22
  • installing manhole causes sys.getfilesystemencoding() return None

    installing manhole causes sys.getfilesystemencoding() return None

    Not sure how this could happen.

    If I start python with manhole env set:

    [email protected]:/# python
    Python 2.7.14 (default, Jun  5 2018, 08:26:18)
    [GCC 4.8.4] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> Manhole[41:1528398043.5660]: Manhole UDS path: /tmp/manhole-41
    Manhole[41:1528398043.5662]: Waiting for new connection (in pid:41) ...
    import sys
    >>> sys.getfilesystemencoding()
    >>>
    

    If I unset the env first:

    [email protected]:/# unset PYTHONMANHOLE
    [email protected]:/# python
    Python 2.7.14 (default, Jun  5 2018, 08:26:18)
    [GCC 4.8.4] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> sys.getfilesystemencoding()
    'ANSI_X3.4-1968'
    >>>
    
    opened by honnix 16
  • eventlet tests fail with

    eventlet tests fail with "RuntimeError: no suitable implementation for this system"

    File "/home/travis/build/ionelmc/python-manhole/.tox/2.7-signalfd-eventlet-nocov-patchthread/lib/python2.7/site-packages/eventlet/support/monotonic.py", line 167, in <module>
        raise RuntimeError('no suitable implementation for this system')
    RuntimeError: no suitable implementation for this system
    

    https://travis-ci.org/ionelmc/python-manhole/builds/319810334

    opened by nirs 6
  • Avoid deadlock after fork triggered by manhole.cry()

    Avoid deadlock after fork triggered by manhole.cry()

    On python 3 and pypy, manhole may trigger a deadlock by writing to stderr while fork is invoked by user code. With correct timing, the child finds stderr lock in locked state, and deadlock when tyring to write to stderr in cry().

    This issue can be reproduced when running the fork_exec tests using pypy:

    manhole.install()           # cry() in manhole thread
    subprocess.call(['true'])   # cry() after fork
    

    It can also be reproduced by this code, with both python3 and pypy:

    import os
    import sys
    import threading
    
    def run():
        sys.stderr.write("in parent thread\n")
    
    threading.Thread(target=run).start()
    pid = os.fork()
    if pid:
        os.waitpid(pid, 0)
    else:
        sys.stderr.write("in child\n")
    

    This patch adds a _CRY_LOCK lock, and ensures that it is held while writing to stderr or forking, and that _CRY_LOCK is released in the child after forking.

    If forking functions are not patched, this change does not have any effect. We should consider always patching fork functions, even when we don't want to reinstall manhole, only to avoid this possible deadlock.

    opened by nirs 6
  • Move all gloabls to top of the module

    Move all gloabls to top of the module

    We have too many globals, and they are speard all over the module, making it hard to see the whole picture. Now they are all in the top of the module.

    Some gloabls were defined using "a = b = ... = None" syntax. This syntax is hard to read, so these globals are defined now one global per line.

    opened by nirs 5
  • Graceful connection shutdown in CLI

    Graceful connection shutdown in CLI

    Simplify and improve the CLI tool — close the connection gracefully, handle disconnects.

    Ctrl-C, Ctrl-D or exit() ways of exit are handled.

    Remote end disconnection causes CLI to exit immediately

    opened by anton-ryzhov 4
  • Remove last exception's references if any

    Remove last exception's references if any

    code module used underneath stores exception details in sys module.

    That creates a memory leak in case of exception in manhole session. This exception encapsulates entire session with all stack frames with their locals and manhole internals (such as ManholeConsole instance and connection file wrapper).

    Simple test. Call this via manhole and target process will be 100Mb bigger even after disconnect:

    >>> padding = ' ' * 100 * 1024**2
    >>> assert 0 
    Traceback (most recent call last):
      File "<console>", line 1, in <module>
    AssertionError
    >>> ^D
    now exiting ManholeConsole...
    
    opened by anton-ryzhov 4
  • User defined locals in manhole shell

    User defined locals in manhole shell

    Typically, when you add a manhole to existing application, the application was not designed for this, so finding stuff from the manhole shell is hard or even impossible.

    This patch adds a new locals optional argument, allowing a user to add application specific objects to manhole shell locals.

    Example usage:

    manhole.install(locals={'server', my_server})
    

    From the manhole shell, you can now use "server":

    >>> server.status()
    

    Installed locals are not inherited by child processes, since they may be invalid in the child process.

    Issue: #16

    opened by nirs 4
  • 1.8.0: pytest is failing and deprecacion warnings

    1.8.0: pytest is failing and deprecacion warnings

    I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

    • python3 -sBm build -w
    • install .whl file in </install/prefix>
    • run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

    Here is pytest output with some deprecation warnings as well:

    + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
    + /usr/bin/pytest -ra
    =========================================================================== test session starts ============================================================================
    platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
    rootdir: /home/tkloczko/rpmbuild/BUILD/python-manhole-1.8.0, configfile: setup.cfg, testpaths: tests
    plugins: asyncio-0.16.0, steps-1.8.0, harvest-1.10.3
    collected 45 items
    
    tests/test_manhole.py ....FF..FF...............ss.....Fs.F                                                                                                           [ 80%]
    tests/test_manhole_cli.py .........                                                                                                                                  [100%]
    
    ================================================================================= FAILURES =================================================================================
    ____________________________________________________________________ test_connection_handler_exec[str] _____________________________________________________________________
    tests/test_manhole.py:105: in test_connection_handler_exec
        wait_for_strings(proc.read, TIMEOUT, 'TETE')
    /usr/lib/python3.8/site-packages/process_tests.py:247: in wait_for_strings
        raise AssertionError("Waited %0.2fsecs but %s did not appear in output in the given order !" % (
    E   AssertionError: Waited 10.00secs but ['TETE'] did not appear in output in the given order !
    --------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
    *********** OUTPUT ***********
    
    ******************************
    *********** OUTPUT ***********
    Manhole[2975552:1641271765.2248]: Started ManholeConnectionThread thread. Checking credentials ...
    Manhole[2975552:1641271765.2248]: Accepted connection on fd:4 from PID:2975520 UID:1001 GID:1001
    Manhole[2975552:1641271765.2748]: Running: "print('FOOBAR')\n".
    Manhole[2975552:1641271765.3249]: Running: 'tete()\n'.
    
    ******************************
    ____________________________________________________________________ test_connection_handler_exec[func] ____________________________________________________________________
    tests/test_manhole.py:105: in test_connection_handler_exec
        wait_for_strings(proc.read, TIMEOUT, 'TETE')
    /usr/lib/python3.8/site-packages/process_tests.py:247: in wait_for_strings
        raise AssertionError("Waited %0.2fsecs but %s did not appear in output in the given order !" % (
    E   AssertionError: Waited 10.00secs but ['TETE'] did not appear in output in the given order !
    --------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
    *********** OUTPUT ***********
    
    ******************************
    *********** OUTPUT ***********
    Manhole[2975588:1641271776.7484]: Started ManholeConnectionThread thread. Checking credentials ...
    Manhole[2975588:1641271776.7485]: Accepted connection on fd:4 from PID:2975520 UID:1001 GID:1001
    Manhole[2975588:1641271776.7987]: Running: "print('FOOBAR')\n".
    Manhole[2975588:1641271776.8489]: Running: 'tete()\n'.
    
    ******************************
    __________________________________________________________________________ test_daemon_connection __________________________________________________________________________
    tests/test_manhole.py:143: in test_daemon_connection
        pytest.raises((socket.error, OSError), assert_manhole_running, proc, uds_path, extra=terminate_and_read)
    tests/test_manhole.py:56: in assert_manhole_running
        extra(client)
    tests/test_manhole.py:137: in terminate_and_read
        wait_for_strings(proc.read, TIMEOUT, 'Died with KeyboardInterrupt', 'DIED.')
    /usr/lib/python3.8/site-packages/process_tests.py:247: in wait_for_strings
        raise AssertionError("Waited %0.2fsecs but %s did not appear in output in the given order !" % (
    E   AssertionError: Waited 10.00secs but ['DIED.', 'Died with KeyboardInterrupt'] did not appear in output in the given order !
    --------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
    *********** OUTPUT ***********
    
    ######### ProcessID=2975625, ThreadID=140147246274112 #########
    File: "/usr/lib64/python3.8/threading.py", line 890, in _bootstrap
      self._bootstrap_inner()
    File: "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
      self.run()
    File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 238, in run
      self.connection_handler(self.client)
    File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 305, in handle_connection_repl
      handle_repl(_MANHOLE.locals)
    File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 354, in handle_repl
      dump_stacktraces()
    File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 633, in dump_stacktraces
      for filename, lineno, name, line in traceback.extract_stack(stack):
    
    ######### ProcessID=2975625, ThreadID=140147254666816 #########
    File: "/usr/lib64/python3.8/threading.py", line 890, in _bootstrap
      self._bootstrap_inner()
    File: "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
      self.run()
    File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 207, in run
      client.join()
    File: "/usr/lib64/python3.8/threading.py", line 1011, in join
      self._wait_for_tstate_lock()
    File: "/usr/lib64/python3.8/threading.py", line 1027, in _wait_for_tstate_lock
      elif lock.acquire(block, timeout):
    
    ######### ProcessID=2975625, ThreadID=140147484260160 #########
    File: "/home/tkloczko/rpmbuild/BUILD/python-manhole-1.8.0/tests/helper.py", line 215, in <module>
      time.sleep(TIMEOUT)
    #############################################
    
    
    Python 3.8.12 (default, Dec 17 2021, 08:35:49)
    [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    (ManholeConsole)
    >>> FOOBAR
    >>>
    ******************************
    *********** OUTPUT ***********
    Manhole[2975625:1641271789.7462]: Patched <built-in function fork> and <built-in function forkpty>.
    Manhole[2975625:1641271789.7463]: Manhole UDS path: /tmp/manhole-2975625
    Manhole[2975625:1641271789.7463]: Waiting for new connection (in pid:2975625) ...
    Manhole[2975625:1641271789.7832]: Started ManholeConnectionThread thread. Checking credentials ...
    Manhole[2975625:1641271789.7832]: Accepted connection on fd:4 from PID:2975520 UID:1001 GID:1001
    Fatal Python error: could not acquire lock for <_io.BufferedReader name=4> at interpreter shutdown, possibly due to daemon threads
    Python runtime state: finalizing (tstate=0x55e9be27f720)
    
    Thread 0x00007f7692d49640 (most recent call first):
      File "/usr/lib64/python3.8/socket.py", line 669 in readinto
      File "/usr/lib64/python3.8/code.py", line 274 in raw_input
      File "/usr/lib64/python3.8/code.py", line 227 in interact
      File "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 365 in handle_repl
      File "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 305 in handle_connection_repl
      File "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 238 in run
      File "/usr/lib64/python3.8/threading.py", line 932 in _bootstrap_inner
      File "/usr/lib64/python3.8/threading.py", line 890 in _bootstrap
    
    Thread 0x00007f769354a640 (most recent call first):
      File "/usr/lib64/python3.8/threading.py", line 1027 in _wait_for_tstate_lock
      File "/usr/lib64/python3.8/threading.py", line 1011 in join
      File "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 207 in run
      File "/usr/lib64/python3.8/threading.py", line 932 in _bootstrap_inner
      File "/usr/lib64/python3.8/threading.py", line 890 in _bootstrap
    
    Current thread 0x00007f76a103f740 (most recent call first):
    <no Python frame>
    
    ******************************
    ________________________________________________________________________ test_non_daemon_connection ________________________________________________________________________
    tests/test_manhole.py:164: in test_non_daemon_connection
        assert_manhole_running(proc, uds_path, extra=terminate_and_read, oneshot=True)
    tests/test_manhole.py:56: in assert_manhole_running
        extra(client)
    tests/test_manhole.py:158: in terminate_and_read
        wait_for_strings(proc.read, TIMEOUT, 'Died with KeyboardInterrupt')
    /usr/lib/python3.8/site-packages/process_tests.py:247: in wait_for_strings
        raise AssertionError("Waited %0.2fsecs but %s did not appear in output in the given order !" % (
    E   AssertionError: Waited 10.00secs but ['Died with KeyboardInterrupt'] did not appear in output in the given order !
    --------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
    *********** OUTPUT ***********
    
    ######### ProcessID=2975649, ThreadID=140492733548096 #########
    File: "/usr/lib64/python3.8/threading.py", line 890, in _bootstrap
      self._bootstrap_inner()
    File: "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
      self.run()
    File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 238, in run
      self.connection_handler(self.client)
    File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 305, in handle_connection_repl
      handle_repl(_MANHOLE.locals)
    File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 354, in handle_repl
      dump_stacktraces()
    File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 633, in dump_stacktraces
      for filename, lineno, name, line in traceback.extract_stack(stack):
    
    ######### ProcessID=2975649, ThreadID=140492741940800 #########
    File: "/usr/lib64/python3.8/threading.py", line 890, in _bootstrap
      self._bootstrap_inner()
    File: "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
      self.run()
    File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 207, in run
      client.join()
    File: "/usr/lib64/python3.8/threading.py", line 857, in start
      self._started.wait()
    File: "/usr/lib64/python3.8/threading.py", line 559, in wait
      return signaled
    File: "/usr/lib64/python3.8/threading.py", line 309, in wait
      return gotit
    
    ######### ProcessID=2975649, ThreadID=140492971534144 #########
    File: "/home/tkloczko/rpmbuild/BUILD/python-manhole-1.8.0/tests/helper.py", line 244, in <module>
      time.sleep(0.3)  # give the manhole a bit enough time to start
    #############################################
    
    
    Python 3.8.12 (default, Dec 17 2021, 08:35:49)
    [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    (ManholeConsole)
    >>> FOOBAR
    >>>
    ******************************
    *********** OUTPUT ***********
    Manhole[2975649:1641271800.0385]: Patched <built-in function fork> and <built-in function forkpty>.
    Manhole[2975649:1641271800.0385]: Manhole UDS path: /tmp/manhole-2975649
    Manhole[2975649:1641271800.0386]: Waiting for new connection (in pid:2975649) ...
    Manhole[2975649:1641271800.0752]: Started ManholeConnectionThread thread. Checking credentials ...
    Manhole[2975649:1641271800.0752]: Accepted connection on fd:4 from PID:2975520 UID:1001 GID:1001
    
    ******************************
    _____________________________________________________________________ test_environ_variable_activation _____________________________________________________________________
    tests/test_manhole.py:493: in test_environ_variable_activation
        wait_for_strings(proc.read, TIMEOUT,
    /usr/lib/python3.8/site-packages/process_tests.py:247: in wait_for_strings
        raise AssertionError("Waited %0.2fsecs but %s did not appear in output in the given order !" % (
    E   AssertionError: Waited 10.00secs but ['Not patching os.fork and os.forkpty. Oneshot activation is done by signal'] did not appear in output in the given order !
    --------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
    *********** OUTPUT ***********
    
    ******************************
    ________________________________________________________________________________ test_uwsgi ________________________________________________________________________________
    tests/test_manhole.py:529: in test_uwsgi
        with TestProcess(
    /usr/lib/python3.8/site-packages/process_tests.py:127: in __init__
        self.proc = subprocess.Popen(
    /usr/lib64/python3.8/subprocess.py:858: in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
    /usr/lib64/python3.8/subprocess.py:1704: in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    E   FileNotFoundError: [Errno 2] No such file or directory: 'uwsgi'
    ============================================================================= warnings summary =============================================================================
    ../../../../../usr/lib/python3.8/site-packages/_pytest/config/__init__.py:1183
      /usr/lib/python3.8/site-packages/_pytest/config/__init__.py:1183: PytestDeprecationWarning: The --strict option is deprecated, use --strict-markers instead.
        self.issue_config_time_warning(
    
    tests/test_manhole.py:3
      /home/tkloczko/rpmbuild/BUILD/python-manhole-1.8.0/tests/test_manhole.py:3: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
        import imp
    
    -- Docs: https://docs.pytest.org/en/stable/warnings.html
    ========================================================================= short test summary info ==========================================================================
    SKIPPED [1] tests/test_manhole.py:402: condition: not is_module_available("signalfd")
    SKIPPED [1] tests/test_manhole.py:413: condition: not is_module_available("signalfd")
    SKIPPED [1] tests/test_manhole.py:502: condition: not is_module_available("signalfd")
    FAILED tests/test_manhole.py::test_connection_handler_exec[str] - AssertionError: Waited 10.00secs but ['TETE'] did not appear in output in the given order !
    FAILED tests/test_manhole.py::test_connection_handler_exec[func] - AssertionError: Waited 10.00secs but ['TETE'] did not appear in output in the given order !
    FAILED tests/test_manhole.py::test_daemon_connection - AssertionError: Waited 10.00secs but ['DIED.', 'Died with KeyboardInterrupt'] did not appear in output in the give...
    FAILED tests/test_manhole.py::test_non_daemon_connection - AssertionError: Waited 10.00secs but ['Died with KeyboardInterrupt'] did not appear in output in the given ord...
    FAILED tests/test_manhole.py::test_environ_variable_activation - AssertionError: Waited 10.00secs but ['Not patching os.fork and os.forkpty. Oneshot activation is done b...
    FAILED tests/test_manhole.py::test_uwsgi - FileNotFoundError: [Errno 2] No such file or directory: 'uwsgi'
    ===================================================== 6 failed, 36 passed, 3 skipped, 2 warnings in 157.19s (0:02:37) ======================================================
    
    opened by kloczek 1
  • "manhole-cli" can NOT connect when we use "socket_path" in the server

    "manhole-cli" requires a PID to connect. This doesn't work when the manhole server uses a specified name. "manhole-cli" REQUIRES the parameter to be a number or "/tmp/manhole-NUMBER". Also, "NUMBER" is used to send signals.

    So, in order to use "manhole-cli":

    1. We can not use arbitrary names in the unix socket name.
    2. The unix socket MUST be in "/tmp/".
    3. If we relax the regexp in "manhole-cli" to be able to use arbitrary unix socket names, we can not send signals to the process.
    opened by jcea 2
  • Tests too slow, need much faster feedback

    Tests too slow, need much faster feedback

    Even after #13, tests are too slow.

    $ tox -e 2.7 -- py.test -v --durations=10 2.7 develop-inst-nodeps: /home/nsoffer/src/python-manhole 2.7 runtests: PYTHONHASHSEED='4080427460' 2.7 runtests: commands[0] | py.test -v --durations=10 ============================================================================= test session starts ============================================================================== platform linux2 -- Python 2.7.5 -- py-1.4.23 -- pytest-2.6.1 -- /home/nsoffer/src/python-manhole/.tox/2.7/bin/python2.7 plugins: cov, capturelog collected 14 items

    tests/test_manhole.py::test_simple PASSED tests/test_manhole.py::test_fork_exec PASSED tests/test_manhole.py::test_socket_path PASSED tests/test_manhole.py::test_socket_path_with_fork PASSED tests/test_manhole.py::test_exit_with_grace PASSED tests/test_manhole.py::test_with_fork PASSED tests/test_manhole.py::test_with_forkpty PASSED tests/test_manhole.py::test_auth_fail PASSED tests/test_manhole.py::test_activate_on_usr2 PASSED tests/test_manhole.py::test_activate_on_with_oneshot_on PASSED tests/test_manhole.py::test_oneshot_on_usr2 PASSED tests/test_manhole.py::test_fail_to_cry PASSED tests/test_manhole.py::test_oneshot_on_usr2_error PASSED tests/test_manhole.py::test_interrupt_on_accept PASSED

    ========================================================================== slowest 10 test durations =========================================================================== 11.11s call tests/test_manhole.py::test_oneshot_on_usr2_error 10.81s call tests/test_manhole.py::test_oneshot_on_usr2 10.70s call tests/test_manhole.py::test_activate_on_usr2 4.39s call tests/test_manhole.py::test_simple 2.62s call tests/test_manhole.py::test_with_fork 2.57s call tests/test_manhole.py::test_with_forkpty 2.56s call tests/test_manhole.py::test_socket_path_with_fork 2.12s call tests/test_manhole.py::test_interrupt_on_accept 1.16s call tests/test_manhole.py::test_fork_exec 0.61s call tests/test_manhole.py::test_exit_with_grace ==================================================================== 14 passed, 2 warnings in 49.98 seconds ====================================================================

    opened by nirs 5
  • manhole socket not removed when terminating a process

    manhole socket not removed when terminating a process

    Although manhole register the removal function using atexit module, the removal function is not run when sending the process SIGTERM.

    To reproduce:

    1. Create a script::

      import manhole import time manhole.install(socket_path='exit.manhole') time.sleep(30)

    2. Run the script in the backround

      python exit.py &

    3. And terminate it

      kill

    The removal function is not called, and manhole socket is not removed.

    Tested on Fedora 19.

    This seems faulty implementation of atexit module.

    Workarounds:

    • terminate the process using SIGINT
    • manually remove the socket in the application - easy when using socket_path option
    opened by nirs 6
Releases(v1.7.0)
  • v1.7.0(Mar 24, 2021)

    • Fixed memory leak via sys.last_type, sys.last_value, sys.last_traceback. Contributed by Anton Ryzhov in #59.
    • Fixed a bunch of double-close bugs and simplified stream handler code. Contributed by Anton Ryzhov in #58.
    • Loosen up pid argument parsing in manhole-cli to allow using paths with any prefix (not just /tmp).
    Source code(tar.gz)
    Source code(zip)
  • v1.6.0(Mar 24, 2021)

    • Testing improvements (changed some skips to xfail, added osx in Travis).
    • Fixed long standing Python 2.7 bug where sys.getfilesystemencoding() would be broken after installing a threaded manhole. See #51.
    • Dropped support for Python 2.6, 3.3 and 3.4.
    • Fixed handling when socket.setdefaulttimeout() is used. Contributed by "honnix" in #53.
    • Fixed some typos. Contributed by Jesús Cea in #43.
    • Fixed handling in manhole-cli so that timeout is actually seconds and not milliseconds. Contributed by Nir Soffer in #45.
    • Cleaned up useless polling options in manhole-cli. Contributed by Nir Soffer in #46.
    • Documented and implemented a solution for using Manhole with Eventlet. See #49.
    Source code(tar.gz)
    Source code(zip)
  • v1.5.0(Mar 24, 2021)

    • Added two string aliases for connection_handler option. Now you can conveniently use connection_handler="exec".
    • Improved handle_connection_exec. It now has a clean way to exit (exit()) and properly closes the socket.
    Source code(tar.gz)
    Source code(zip)
  • v1.4.0(Mar 24, 2021)

    • Added the connection_handler install option. Default value is manhole.handle_connection_repl, and alternate manhole.handle_connection_exec is provided (very simple: no output redirection, no stacktrace dumping).
    • Dropped Python 3.2 from the test grid. It may work but it's a huge pain to support (pip/pytest don't support it anymore).
    • Added Python 3.5 and 3.6 in the test grid.
    • Fixed issues with piping to manhole-cli. Now echo foobar | manhole-cli will wait 1 second for output from manhole (you can customize this with the --timeout option).
    • Fixed issues with newer PyPy (caused by gevent/eventlet socket unwrapping).
    Source code(tar.gz)
    Source code(zip)
  • v1.3.0(Mar 24, 2021)

    • Allowed Manhole to be configured without any thread or activation (in case you want to manually activate).
    • Added an example and tests for using Manhole with uWSGi.
    • Fixed error handling in manhole-cli on Python 3 (exc vars don't leak anymore).
    • Fixed support for running in gevent/eventlet-using apps on Python 3 (now that they support Python 3).
    • Allowed reinstalling the manhole (in non-strict mode). Previous install is undone.
    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Mar 24, 2021)

    • Changed manhole-cli:
      • Won't spam the terminal with errors if socket file doesn't exist.
      • Allowed sending any signal (new --signal argument).
      • Fixed some validation issues for the PID argument.
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Mar 24, 2021)

    • Added support for installing the manhole via the PYTHONMANHOLE environment variable.
    • Added a strict install option. Set it to false to avoid getting the AlreadyInstalled exception.
    • Added a manhole-cli script that emulates socat readline unix-connect:/tmp/manhole-1234.
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jun 5, 2015)

  • v0.6.2(Jun 5, 2015)

  • v0.6.1(Jun 5, 2015)

  • v0.5.1(Aug 19, 2013)

  • v0.5.0(Aug 17, 2013)

    • Python 3.2, 3.3 and PyPy support !
    • Fixed a bug with closing the connection socket (it was causing crashes in some rare scenarios)
    • Created a test matrix in Travis
    Source code(tar.gz)
    Source code(zip)
A powerful set of Python debugging tools, based on PySnooper

snoop snoop is a powerful set of Python debugging tools. It's primarily meant to be a more featureful and refined version of PySnooper. It also includ

Alex Hall 753 May 23, 2022
Cyberbrain: Python debugging, redefined.

Cyberbrain1(电子脑) aims to free programmers from debugging.

laike9m 2.2k May 26, 2022
VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.

VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.

null 2k May 26, 2022
A package containing a lot of useful utilities for Python developing and debugging.

Vpack A package containing a lot of useful utilities for Python developing and debugging. Features Sigview: press Ctrl+C to print the current stack in

volltin 14 Dec 3, 2021
Never use print for debugging again

PySnooper - Never use print for debugging again PySnooper is a poor man's debugger. If you've used Bash, it's like set -x for Python, except it's fanc

Ram Rachum 15k May 27, 2022
A web-based visualization and debugging platform for NuPIC

Cerebro 2 A web-based visualization and debugging platform for NuPIC. Usage Set up cerebro2.server to export your model state. Then, run: cd static py

Numenta 24 Oct 13, 2021
Hypothesis debugging with vscode

Hypothesis debugging with vscode

Oliver Mannion 0 Feb 9, 2022
pdb++, a drop-in replacement for pdb (the Python debugger)

pdb++, a drop-in replacement for pdb What is it? This module is an extension of the pdb module of the standard library. It is meant to be fully compat

null 899 May 20, 2022
Full-screen console debugger for Python

PuDB: a console-based visual debugger for Python Its goal is to provide all the niceties of modern GUI-based debuggers in a more lightweight and keybo

Andreas Klöckner 2.4k May 18, 2022
Trace any Python program, anywhere!

lptrace lptrace is strace for Python programs. It lets you see in real-time what functions a Python program is running. It's particularly useful to de

Karim Hamidou 683 May 26, 2022
Debugger capable of attaching to and injecting code into python processes.

DISCLAIMER: This is not an official google project, this is just something I wrote while at Google. Pyringe What this is Pyringe is a python debugger

Google 1.6k May 15, 2022
(OLD REPO) Line-by-line profiling for Python - Current repo ->

line_profiler and kernprof line_profiler is a module for doing line-by-line profiling of functions. kernprof is a convenient script for running either

Robert Kern 3.6k May 23, 2022
Monitor Memory usage of Python code

Memory Profiler This is a python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for pyth

Fabian Pedregosa 75 May 25, 2022
Sampling profiler for Python programs

py-spy: Sampling profiler for Python programs py-spy is a sampling profiler for Python programs. It lets you visualize what your Python program is spe

Ben Frederickson 8.5k May 19, 2022
🔥 Pyflame: A Ptracing Profiler For Python. This project is deprecated and not maintained.

Pyflame: A Ptracing Profiler For Python (This project is deprecated and not maintained.) Pyflame is a high performance profiling tool that generates f

Uber Archive 2.9k May 20, 2022
Visual profiler for Python

vprof vprof is a Python package providing rich and interactive visualizations for various Python program characteristics such as running time and memo

Nick Volynets 3.8k May 22, 2022
Parsing ELF and DWARF in Python

pyelftools pyelftools is a pure-Python library for parsing and analyzing ELF files and DWARF debugging information. See the User's guide for more deta

Eli Bendersky 1.4k May 25, 2022
pdb++, a drop-in replacement for pdb (the Python debugger)

pdb++, a drop-in replacement for pdb What is it? This module is an extension of the pdb module of the standard library. It is meant to be fully compat

null 901 May 25, 2022
Run-time type checker for Python

This library provides run-time type checking for functions defined with PEP 484 argument (and return) type annotations. Four principal ways to do type

Alex Grönholm 876 May 19, 2022