Debugging manhole for python applications.

Ionel Cristian Mărieș

Last update: Dec 7, 2022

Related tags

Overview

docs
tests
package

Manhole is in-process service that will accept unix domain socket connections and present the stacktraces for all threads and an interactive prompt. It can either work as a python daemon thread waiting for connections at all times or a signal handler (stopping your application and waiting for a connection).

Access to the socket is restricted to the application's effective user id or root.

This is just like Twisted's manhole. It's simpler (no dependencies), it only runs on Unix domain sockets (in contrast to Twisted's manhole which can run on telnet or ssh) and it integrates well with various types of applications.

Documentation:	http://python-manhole.readthedocs.org/en/latest/

Usage

Install it:

pip install manhole

You can put this in your django settings, wsgi app file, some module that's always imported early etc:

import manhole
manhole.install() # this will start the daemon thread

# and now you start your app, eg: server.serve_forever()

Now in a shell you can do either of these:

netcat -U /tmp/manhole-1234
socat - unix-connect:/tmp/manhole-1234
socat readline unix-connect:/tmp/manhole-1234

Socat with readline is best (history, editing etc). If your socat doesn't have readme try this.

Sample output:

$ nc -U /tmp/manhole-1234

Python 2.7.3 (default, Apr 10 2013, 06:20:15)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> dir()
['__builtins__', 'dump_stacktraces', 'os', 'socket', 'sys', 'traceback']
>>> print 'foobar'
foobar

Alternative client

There's a new experimental manhole-cli bin since 1.1.0, that emulates socat:

usage: manhole-cli [-h] [-t TIMEOUT] [-1 | -2 | -s SIGNAL] PID

Connect to a manhole.

positional arguments:
  PID                   A numerical process id, or a path in the form:
                        /tmp/manhole-1234

optional arguments:
  -h, --help            show this help message and exit
  -t TIMEOUT, --timeout TIMEOUT
                        Timeout to use. Default: 1 seconds.
  -1, -USR1             Send USR1 (10) to the process before connecting.
  -2, -USR2             Send USR2 (12) to the process before connecting.
  -s SIGNAL, --signal SIGNAL
                        Send the given SIGNAL to the process before
                        connecting.

Features

Uses unix domain sockets, only root or same effective user can connect.
Can run the connection in a thread or in a signal handler (see oneshot_on option).
Can start the thread listening for connections from a signal handler (see activate_on option)
Compatible with apps that fork, reinstalls the Manhole thread after fork - had to monkeypatch os.fork/os.forkpty for this.
Compatible with gevent and eventlet with some limitations - you need to either:
- Use oneshot_on, or
- Disable thread monkeypatching (eg: gevent.monkey.patch_all(thread=False), eventlet.monkey_patch(thread=False)
Note: on eventlet you might need to setup the hub first to prevent circular import problems:
```
import eventlet
eventlet.hubs.get_hub()  # do this first
eventlet.monkey_patch(thread=False)
```
The thread is compatible with apps that use signalfd (will mask all signals for the Manhole threads).

Options

manhole.install(
    verbose=True,
    verbose_destination=2,
    patch_fork=True,
    activate_on=None,
    oneshot_on=None,
    sigmask=manhole.ALL_SIGNALS,
    socket_path=None,
    reinstall_delay=0.5,
    locals=None,
    strict=True,
)

verbose - Set it to False to squelch the logging.
verbose_destination - Destination for verbose messages. Set it to a file descriptor or handle. Default is unbuffered stderr (stderr 2 file descriptor).
patch_fork - Set it to False if you don't want your os.fork and os.forkpy monkeypatched
activate_on - Set to "USR1", "USR2" or some other signal name, or a number if you want the Manhole thread to start when this signal is sent. This is desirable in case you don't want the thread active all the time.
thread - Set to True to start the always-on ManholeThread. Default: True. Automatically switched to False if oneshot_on or activate_on are used.
oneshot_on - Set to "USR1", "USR2" or some other signal name, or a number if you want the Manhole to listen for connection in the signal handler. This is desireable in case you don't want threads at all.
sigmask - Will set the signal mask to the given list (using signalfd.sigprocmask). No action is done if signalfd is not importable. NOTE: This is done so that the Manhole thread doesn't steal any signals; Normally that is fine because Python will force all the signal handling to be run in the main thread but signalfd doesn't.
socket_path - Use a specific path for the unix domain socket (instead of /tmp/manhole-<pid>). This disables patch_fork as children cannot reuse the same path.
reinstall_delay - Delay the unix domain socket creation reinstall_delay seconds. This alleviates cleanup failures when using fork+exec patterns.
locals - Names to add to manhole interactive shell locals.
daemon_connection - The connection thread is daemonic (dies on app exit). Default: False.
redirect_stderr - Redirect output from stderr to manhole console. Default: True.
strict - If True then AlreadyInstalled will be raised when attempting to install manhole twice. Default: True.

Environment variable installation

Manhole can be installed via the PYTHONMANHOLE environment variable.

This:

PYTHONMANHOLE='' python yourapp.py

Is equivalent to having this in yourapp.py:

import manhole
manhole.install()

Any extra text in the environment variable is passed to manhole.install(). Example:

PYTHONMANHOLE='onshot_on="USR2"' python yourapp.py

What happens when you actually connect to the socket

Credentials are checked (if it's same user or root)
sys.__std*__/sys.std* are redirected to the UDS
Stacktraces for each thread are written to the UDS
REPL is started so you can fiddle with the process

Known issues

Using threads and file handle (not raw file descriptor) verbose_destination can cause deadlocks. See bug reports: PyPy and Python 3.4.

SIGTERM and socket cleanup

By default Python doesn't call the atexit callbacks with the default SIGTERM handling. This makes manhole leave stray socket files around. If this is undesirable you should install a custom SIGTERM handler so atexit is properly invoked.

Example:

import signal
import sys

def handle_sigterm(signo, frame):
    sys.exit(128 + signo)  # this will raise SystemExit and cause atexit to be called

signal.signal(signal.SIGTERM, handle_sigterm)

Using Manhole with uWSGI

Because uWSGI overrides signal handling Manhole is a bit more tricky to setup. One way is to use "uWSGI signals" (not the POSIX signals) and have the workers check a file for the pid you want to open the Manhole in.

Stick something this in your WSGI application file:

from __future__ import print_function
import sys
import os
import manhole

stack_dump_file = '/tmp/manhole-pid'
uwsgi_signal_number = 17

try:
    import uwsgi

    if not os.path.exists(stack_dump_file):
        open(stack_dump_file, 'w')

    def open_manhole(dummy_signum):
        with open(stack_dump_file, 'r') as fh:
            pid = fh.read().strip()
            if pid == str(os.getpid()):
                inst = manhole.install(strict=False, thread=False)
                inst.handle_oneshot(dummy_signum, dummy_signum)

    uwsgi.register_signal(uwsgi_signal_number, 'workers', open_manhole)
    uwsgi.add_file_monitor(uwsgi_signal_number, stack_dump_file)

    print("Listening for stack mahole requests via %r" % (stack_dump_file,), file=sys.stderr)
except ImportError:
    print("Not running under uwsgi; unable to configure manhole trigger", file=sys.stderr)
except IOError:
    print("IOError creating manhole trigger %r" % (stack_dump_file,), file=sys.stderr)


# somewhere bellow you'd have something like
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
# or
def application(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain'), ('Content-Length', '2')])
    yield b'OK'

To open the Manhole just run echo 1234 > /tmp/manhole-pid and then manhole-cli 1234.

Requirements

OS:	Linux, OS X
Runtime:	Python 2.7, 3.4, 3.5, 3.6 or PyPy

Similar projects

Twisted's manhole - it has colors and server-side history.
wsgi-shell - spawns a thread.
pyrasite - uses gdb to inject code.
pydbattach - uses gdb to inject code.
pystuck - very similar, uses rpyc for communication.
pyringe - uses gdb to inject code, more reliable, but relies on dbg python builds unfortunatelly.
pdb-clone - uses gdb to inject code, with a different strategy.

Comments

Do not modify anything during fork+exec flow

Recently a reinstall_bind_delay was added to avoid binding of unix socket during fork+exec flow. However the socket is not the only issue in this flow. Changing process signal mask should not be done, as such changes may be inherited by the new process. Generally, since we don't know yet if the manhole is needed, we should do nothing.

This patch changes the semantics of the reinstall_bind_delay so we do not make any modification to child process (except starting a thread) until the delay is finished.

The user visible parameter was shortened to "reinstall_delay", as the user should not be conceded with the internals of the manhole. Internally the Manhole class call it "setup_delay", as this is the delay before setting up the manhole.

opened by nirs 23

Use socket-bounded file wrapper

Hello,

I faced with strange behavior when used this tool — random files/connections were closed after I inspecting session. After debug I found the root cause.

os.fdopen-created file objects only bounded to the fd value. And they do call self.close() on garbage collection. So if such instance is GCed after closing of original socket — it may close another file/connection that reused same fd.

Demo:

import os, socket

local_socket, remote_socket = socket.socketpair()
print('Sockets FDs:', local_socket.fileno(), remote_socket.fileno())

a_file_wrapper = os.fdopen(local_socket.fileno())

local_socket.send(b'x')    # Successful send

# Close original connection, open new
local_socket.close(); remote_socket.close()

new_local_socket, new_remote_socket = socket.socketpair()
print('Sockets FDs:', new_local_socket.fileno(), new_remote_socket.fileno())

new_local_socket.send(b'x')    # Successful send

# Deletion of the file wrapper…
del a_file_wrapper

# … closes unrelated connection
new_local_socket.send(b'x')    # OSError: [Errno 9] Bad file descriptor

socket.makefile() creates fileobject that is bounded to exact socket instance, so it never closes unrelated fds.

opened by anton-ryzhov 22

Fix failures with pypy on slow or overloaded build machines

We used to do 500 iterations in the fork_exec test, which is not really needed. With no reinstall_delay, we need only 10-20 iterations to expose this issue, so running 50 iterations is just fine.

opened by nirs 22
Changes for OS X (tested with OS X 10.9/Mavericks only)
use LOCAL_PEERCRED on OS X for checking credentials;

don't do setsockopt(SOL_SOCKET, SO_RCVBUF | SO_SNDBUF, 0) on OS X;– things are working properly still anyway, no apparent buffering is present
opened by razzmatazz 22

installing manhole causes sys.getfilesystemencoding() return None

Not sure how this could happen.

If I start python with manhole env set:

root@39d6880c52eb:/# python
Python 2.7.14 (default, Jun  5 2018, 08:26:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> Manhole[41:1528398043.5660]: Manhole UDS path: /tmp/manhole-41
Manhole[41:1528398043.5662]: Waiting for new connection (in pid:41) ...
import sys
>>> sys.getfilesystemencoding()
>>>

If I unset the env first:

root@39d6880c52eb:/# unset PYTHONMANHOLE
root@39d6880c52eb:/# python
Python 2.7.14 (default, Jun  5 2018, 08:26:18)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getfilesystemencoding()
'ANSI_X3.4-1968'
>>>

opened by honnix 16

eventlet tests fail with "RuntimeError: no suitable implementation for this system"

File "/home/travis/build/ionelmc/python-manhole/.tox/2.7-signalfd-eventlet-nocov-patchthread/lib/python2.7/site-packages/eventlet/support/monotonic.py", line 167, in <module>
    raise RuntimeError('no suitable implementation for this system')
RuntimeError: no suitable implementation for this system

https://travis-ci.org/ionelmc/python-manhole/builds/319810334

opened by nirs 6

Avoid deadlock after fork triggered by manhole.cry()
On python 3 and pypy, manhole may trigger a deadlock by writing to stderr while fork is invoked by user code. With correct timing, the child finds stderr lock in locked state, and deadlock when tyring to write to stderr in cry().

This issue can be reproduced when running the fork_exec tests using pypy:

manhole.install() # cry() in manhole thread subprocess.call(['true']) # cry() after fork

It can also be reproduced by this code, with both python3 and pypy:

import os import sys import threading def run(): sys.stderr.write("in parent thread\n") threading.Thread(target=run).start() pid = os.fork() if pid: os.waitpid(pid, 0) else: sys.stderr.write("in child\n")

This patch adds a _CRY_LOCK lock, and ensures that it is held while writing to stderr or forking, and that _CRY_LOCK is released in the child after forking.

If forking functions are not patched, this change does not have any effect. We should consider always patching fork functions, even when we don't want to reinstall manhole, only to avoid this possible deadlock.
opened by nirs 6
Move all gloabls to top of the module

We have too many globals, and they are speard all over the module, making it hard to see the whole picture. Now they are all in the top of the module.

Some gloabls were defined using "a = b = ... = None" syntax. This syntax is hard to read, so these globals are defined now one global per line.

opened by nirs 5
Graceful connection shutdown in CLI

Simplify and improve the CLI tool — close the connection gracefully, handle disconnects.

Ctrl-C, Ctrl-D or exit() ways of exit are handled.

Remote end disconnection causes CLI to exit immediately

opened by anton-ryzhov 4
Remove last exception's references if any
code module used underneath stores exception details in sys module.

That creates a memory leak in case of exception in manhole session. This exception encapsulates entire session with all stack frames with their locals and manhole internals (such as ManholeConsole instance and connection file wrapper).

Simple test. Call this via manhole and target process will be 100Mb bigger even after disconnect:

>>> padding = ' ' * 100 * 1024**2 >>> assert 0 Traceback (most recent call last): File "<console>", line 1, in <module> AssertionError >>> ^D now exiting ManholeConsole...
opened by anton-ryzhov 4
User defined locals in manhole shell
Typically, when you add a manhole to existing application, the application was not designed for this, so finding stuff from the manhole shell is hard or even impossible.

This patch adds a new locals optional argument, allowing a user to add application specific objects to manhole shell locals.

Example usage:

manhole.install(locals={'server', my_server})

From the manhole shell, you can now use "server":

>>> server.status()

Installed locals are not inherited by child processes, since they may be invalid in the child process.

Issue: #16
opened by nirs 4

1.8.0: pytest is failing and deprecacion warnings

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

python3 -sBm build -w
install .whl file in </install/prefix>
run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output with some deprecation warnings as well:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/python-manhole-1.8.0, configfile: setup.cfg, testpaths: tests
plugins: asyncio-0.16.0, steps-1.8.0, harvest-1.10.3
collected 45 items

tests/test_manhole.py ....FF..FF...............ss.....Fs.F                                                                                                           [ 80%]
tests/test_manhole_cli.py .........                                                                                                                                  [100%]

================================================================================= FAILURES =================================================================================
____________________________________________________________________ test_connection_handler_exec[str] _____________________________________________________________________
tests/test_manhole.py:105: in test_connection_handler_exec
    wait_for_strings(proc.read, TIMEOUT, 'TETE')
/usr/lib/python3.8/site-packages/process_tests.py:247: in wait_for_strings
    raise AssertionError("Waited %0.2fsecs but %s did not appear in output in the given order !" % (
E   AssertionError: Waited 10.00secs but ['TETE'] did not appear in output in the given order !
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
*********** OUTPUT ***********

******************************
*********** OUTPUT ***********
Manhole[2975552:1641271765.2248]: Started ManholeConnectionThread thread. Checking credentials ...
Manhole[2975552:1641271765.2248]: Accepted connection on fd:4 from PID:2975520 UID:1001 GID:1001
Manhole[2975552:1641271765.2748]: Running: "print('FOOBAR')\n".
Manhole[2975552:1641271765.3249]: Running: 'tete()\n'.

******************************
____________________________________________________________________ test_connection_handler_exec[func] ____________________________________________________________________
tests/test_manhole.py:105: in test_connection_handler_exec
    wait_for_strings(proc.read, TIMEOUT, 'TETE')
/usr/lib/python3.8/site-packages/process_tests.py:247: in wait_for_strings
    raise AssertionError("Waited %0.2fsecs but %s did not appear in output in the given order !" % (
E   AssertionError: Waited 10.00secs but ['TETE'] did not appear in output in the given order !
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
*********** OUTPUT ***********

******************************
*********** OUTPUT ***********
Manhole[2975588:1641271776.7484]: Started ManholeConnectionThread thread. Checking credentials ...
Manhole[2975588:1641271776.7485]: Accepted connection on fd:4 from PID:2975520 UID:1001 GID:1001
Manhole[2975588:1641271776.7987]: Running: "print('FOOBAR')\n".
Manhole[2975588:1641271776.8489]: Running: 'tete()\n'.

******************************
__________________________________________________________________________ test_daemon_connection __________________________________________________________________________
tests/test_manhole.py:143: in test_daemon_connection
    pytest.raises((socket.error, OSError), assert_manhole_running, proc, uds_path, extra=terminate_and_read)
tests/test_manhole.py:56: in assert_manhole_running
    extra(client)
tests/test_manhole.py:137: in terminate_and_read
    wait_for_strings(proc.read, TIMEOUT, 'Died with KeyboardInterrupt', 'DIED.')
/usr/lib/python3.8/site-packages/process_tests.py:247: in wait_for_strings
    raise AssertionError("Waited %0.2fsecs but %s did not appear in output in the given order !" % (
E   AssertionError: Waited 10.00secs but ['DIED.', 'Died with KeyboardInterrupt'] did not appear in output in the given order !
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
*********** OUTPUT ***********

######### ProcessID=2975625, ThreadID=140147246274112 #########
File: "/usr/lib64/python3.8/threading.py", line 890, in _bootstrap
  self._bootstrap_inner()
File: "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
  self.run()
File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 238, in run
  self.connection_handler(self.client)
File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 305, in handle_connection_repl
  handle_repl(_MANHOLE.locals)
File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 354, in handle_repl
  dump_stacktraces()
File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 633, in dump_stacktraces
  for filename, lineno, name, line in traceback.extract_stack(stack):

######### ProcessID=2975625, ThreadID=140147254666816 #########
File: "/usr/lib64/python3.8/threading.py", line 890, in _bootstrap
  self._bootstrap_inner()
File: "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
  self.run()
File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 207, in run
  client.join()
File: "/usr/lib64/python3.8/threading.py", line 1011, in join
  self._wait_for_tstate_lock()
File: "/usr/lib64/python3.8/threading.py", line 1027, in _wait_for_tstate_lock
  elif lock.acquire(block, timeout):

######### ProcessID=2975625, ThreadID=140147484260160 #########
File: "/home/tkloczko/rpmbuild/BUILD/python-manhole-1.8.0/tests/helper.py", line 215, in <module>
  time.sleep(TIMEOUT)
#############################################


Python 3.8.12 (default, Dec 17 2021, 08:35:49)
[GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
(ManholeConsole)
>>> FOOBAR
>>>
******************************
*********** OUTPUT ***********
Manhole[2975625:1641271789.7462]: Patched <built-in function fork> and <built-in function forkpty>.
Manhole[2975625:1641271789.7463]: Manhole UDS path: /tmp/manhole-2975625
Manhole[2975625:1641271789.7463]: Waiting for new connection (in pid:2975625) ...
Manhole[2975625:1641271789.7832]: Started ManholeConnectionThread thread. Checking credentials ...
Manhole[2975625:1641271789.7832]: Accepted connection on fd:4 from PID:2975520 UID:1001 GID:1001
Fatal Python error: could not acquire lock for <_io.BufferedReader name=4> at interpreter shutdown, possibly due to daemon threads
Python runtime state: finalizing (tstate=0x55e9be27f720)

Thread 0x00007f7692d49640 (most recent call first):
  File "/usr/lib64/python3.8/socket.py", line 669 in readinto
  File "/usr/lib64/python3.8/code.py", line 274 in raw_input
  File "/usr/lib64/python3.8/code.py", line 227 in interact
  File "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 365 in handle_repl
  File "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 305 in handle_connection_repl
  File "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 238 in run
  File "/usr/lib64/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/lib64/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007f769354a640 (most recent call first):
  File "/usr/lib64/python3.8/threading.py", line 1027 in _wait_for_tstate_lock
  File "/usr/lib64/python3.8/threading.py", line 1011 in join
  File "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 207 in run
  File "/usr/lib64/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/lib64/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x00007f76a103f740 (most recent call first):
<no Python frame>

******************************
________________________________________________________________________ test_non_daemon_connection ________________________________________________________________________
tests/test_manhole.py:164: in test_non_daemon_connection
    assert_manhole_running(proc, uds_path, extra=terminate_and_read, oneshot=True)
tests/test_manhole.py:56: in assert_manhole_running
    extra(client)
tests/test_manhole.py:158: in terminate_and_read
    wait_for_strings(proc.read, TIMEOUT, 'Died with KeyboardInterrupt')
/usr/lib/python3.8/site-packages/process_tests.py:247: in wait_for_strings
    raise AssertionError("Waited %0.2fsecs but %s did not appear in output in the given order !" % (
E   AssertionError: Waited 10.00secs but ['Died with KeyboardInterrupt'] did not appear in output in the given order !
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
*********** OUTPUT ***********

######### ProcessID=2975649, ThreadID=140492733548096 #########
File: "/usr/lib64/python3.8/threading.py", line 890, in _bootstrap
  self._bootstrap_inner()
File: "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
  self.run()
File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 238, in run
  self.connection_handler(self.client)
File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 305, in handle_connection_repl
  handle_repl(_MANHOLE.locals)
File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 354, in handle_repl
  dump_stacktraces()
File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 633, in dump_stacktraces
  for filename, lineno, name, line in traceback.extract_stack(stack):

######### ProcessID=2975649, ThreadID=140492741940800 #########
File: "/usr/lib64/python3.8/threading.py", line 890, in _bootstrap
  self._bootstrap_inner()
File: "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
  self.run()
File: "/home/tkloczko/rpmbuild/BUILDROOT/python-manhole-1.8.0-2.fc35.x86_64/usr/lib/python3.8/site-packages/manhole/__init__.py", line 207, in run
  client.join()
File: "/usr/lib64/python3.8/threading.py", line 857, in start
  self._started.wait()
File: "/usr/lib64/python3.8/threading.py", line 559, in wait
  return signaled
File: "/usr/lib64/python3.8/threading.py", line 309, in wait
  return gotit

######### ProcessID=2975649, ThreadID=140492971534144 #########
File: "/home/tkloczko/rpmbuild/BUILD/python-manhole-1.8.0/tests/helper.py", line 244, in <module>
  time.sleep(0.3)  # give the manhole a bit enough time to start
#############################################


Python 3.8.12 (default, Dec 17 2021, 08:35:49)
[GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
(ManholeConsole)
>>> FOOBAR
>>>
******************************
*********** OUTPUT ***********
Manhole[2975649:1641271800.0385]: Patched <built-in function fork> and <built-in function forkpty>.
Manhole[2975649:1641271800.0385]: Manhole UDS path: /tmp/manhole-2975649
Manhole[2975649:1641271800.0386]: Waiting for new connection (in pid:2975649) ...
Manhole[2975649:1641271800.0752]: Started ManholeConnectionThread thread. Checking credentials ...
Manhole[2975649:1641271800.0752]: Accepted connection on fd:4 from PID:2975520 UID:1001 GID:1001

******************************
_____________________________________________________________________ test_environ_variable_activation _____________________________________________________________________
tests/test_manhole.py:493: in test_environ_variable_activation
    wait_for_strings(proc.read, TIMEOUT,
/usr/lib/python3.8/site-packages/process_tests.py:247: in wait_for_strings
    raise AssertionError("Waited %0.2fsecs but %s did not appear in output in the given order !" % (
E   AssertionError: Waited 10.00secs but ['Not patching os.fork and os.forkpty. Oneshot activation is done by signal'] did not appear in output in the given order !
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
*********** OUTPUT ***********

******************************
________________________________________________________________________________ test_uwsgi ________________________________________________________________________________
tests/test_manhole.py:529: in test_uwsgi
    with TestProcess(
/usr/lib/python3.8/site-packages/process_tests.py:127: in __init__
    self.proc = subprocess.Popen(
/usr/lib64/python3.8/subprocess.py:858: in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
/usr/lib64/python3.8/subprocess.py:1704: in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
E   FileNotFoundError: [Errno 2] No such file or directory: 'uwsgi'
============================================================================= warnings summary =============================================================================
../../../../../usr/lib/python3.8/site-packages/_pytest/config/__init__.py:1183
  /usr/lib/python3.8/site-packages/_pytest/config/__init__.py:1183: PytestDeprecationWarning: The --strict option is deprecated, use --strict-markers instead.
    self.issue_config_time_warning(

tests/test_manhole.py:3
  /home/tkloczko/rpmbuild/BUILD/python-manhole-1.8.0/tests/test_manhole.py:3: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================= short test summary info ==========================================================================
SKIPPED [1] tests/test_manhole.py:402: condition: not is_module_available("signalfd")
SKIPPED [1] tests/test_manhole.py:413: condition: not is_module_available("signalfd")
SKIPPED [1] tests/test_manhole.py:502: condition: not is_module_available("signalfd")
FAILED tests/test_manhole.py::test_connection_handler_exec[str] - AssertionError: Waited 10.00secs but ['TETE'] did not appear in output in the given order !
FAILED tests/test_manhole.py::test_connection_handler_exec[func] - AssertionError: Waited 10.00secs but ['TETE'] did not appear in output in the given order !
FAILED tests/test_manhole.py::test_daemon_connection - AssertionError: Waited 10.00secs but ['DIED.', 'Died with KeyboardInterrupt'] did not appear in output in the give...
FAILED tests/test_manhole.py::test_non_daemon_connection - AssertionError: Waited 10.00secs but ['Died with KeyboardInterrupt'] did not appear in output in the given ord...
FAILED tests/test_manhole.py::test_environ_variable_activation - AssertionError: Waited 10.00secs but ['Not patching os.fork and os.forkpty. Oneshot activation is done b...
FAILED tests/test_manhole.py::test_uwsgi - FileNotFoundError: [Errno 2] No such file or directory: 'uwsgi'
===================================================== 6 failed, 36 passed, 3 skipped, 2 warnings in 157.19s (0:02:37) ======================================================

opened by kloczek 1

"manhole-cli" can NOT connect when we use "socket_path" in the server
"manhole-cli" requires a PID to connect. This doesn't work when the manhole server uses a specified name. "manhole-cli" REQUIRES the parameter to be a number or "/tmp/manhole-NUMBER". Also, "NUMBER" is used to send signals.

So, in order to use "manhole-cli":

We can not use arbitrary names in the unix socket name.

The unix socket MUST be in "/tmp/".

If we relax the regexp in "manhole-cli" to be able to use arbitrary unix socket names, we can not send signals to the process.
opened by jcea 2
Tests too slow, need much faster feedback

Even after #13, tests are too slow.

$ tox -e 2.7 -- py.test -v --durations=10 2.7 develop-inst-nodeps: /home/nsoffer/src/python-manhole 2.7 runtests: PYTHONHASHSEED='4080427460' 2.7 runtests: commands[0] | py.test -v --durations=10 ============================================================================= test session starts ============================================================================== platform linux2 -- Python 2.7.5 -- py-1.4.23 -- pytest-2.6.1 -- /home/nsoffer/src/python-manhole/.tox/2.7/bin/python2.7 plugins: cov, capturelog collected 14 items

tests/test_manhole.py::test_simple PASSED tests/test_manhole.py::test_fork_exec PASSED tests/test_manhole.py::test_socket_path PASSED tests/test_manhole.py::test_socket_path_with_fork PASSED tests/test_manhole.py::test_exit_with_grace PASSED tests/test_manhole.py::test_with_fork PASSED tests/test_manhole.py::test_with_forkpty PASSED tests/test_manhole.py::test_auth_fail PASSED tests/test_manhole.py::test_activate_on_usr2 PASSED tests/test_manhole.py::test_activate_on_with_oneshot_on PASSED tests/test_manhole.py::test_oneshot_on_usr2 PASSED tests/test_manhole.py::test_fail_to_cry PASSED tests/test_manhole.py::test_oneshot_on_usr2_error PASSED tests/test_manhole.py::test_interrupt_on_accept PASSED

========================================================================== slowest 10 test durations =========================================================================== 11.11s call tests/test_manhole.py::test_oneshot_on_usr2_error 10.81s call tests/test_manhole.py::test_oneshot_on_usr2 10.70s call tests/test_manhole.py::test_activate_on_usr2 4.39s call tests/test_manhole.py::test_simple 2.62s call tests/test_manhole.py::test_with_fork 2.57s call tests/test_manhole.py::test_with_forkpty 2.56s call tests/test_manhole.py::test_socket_path_with_fork 2.12s call tests/test_manhole.py::test_interrupt_on_accept 1.16s call tests/test_manhole.py::test_fork_exec 0.61s call tests/test_manhole.py::test_exit_with_grace ==================================================================== 14 passed, 2 warnings in 49.98 seconds ====================================================================

opened by nirs 5
manhole socket not removed when terminating a process
Although manhole register the removal function using atexit module, the removal function is not run when sending the process SIGTERM.

To reproduce:

Create a script::

import manhole import time manhole.install(socket_path='exit.manhole') time.sleep(30)

Run the script in the backround

python exit.py &

And terminate it

kill

The removal function is not called, and manhole socket is not removed.

Tested on Fedora 19.

This seems faulty implementation of atexit module.

Workarounds:

terminate the process using SIGINT

manually remove the socket in the application - easy when using socket_path option
opened by nirs 6

Releases(v1.7.0)

v1.7.0(Mar 24, 2021)
Fixed memory leak via sys.last_type, sys.last_value, sys.last_traceback. Contributed by Anton Ryzhov in #59.

Fixed a bunch of double-close bugs and simplified stream handler code. Contributed by Anton Ryzhov in #58.

Loosen up pid argument parsing in manhole-cli to allow using paths with any prefix (not just /tmp).

Source code(tar.gz)
Source code(zip)
v1.6.0(Mar 24, 2021)
Testing improvements (changed some skips to xfail, added osx in Travis).

Fixed long standing Python 2.7 bug where sys.getfilesystemencoding() would be broken after installing a threaded manhole. See #51.

Dropped support for Python 2.6, 3.3 and 3.4.

Fixed handling when socket.setdefaulttimeout() is used. Contributed by "honnix" in #53.

Fixed some typos. Contributed by Jesús Cea in #43.

Fixed handling in manhole-cli so that timeout is actually seconds and not milliseconds. Contributed by Nir Soffer in #45.

Cleaned up useless polling options in manhole-cli. Contributed by Nir Soffer in #46.

Documented and implemented a solution for using Manhole with Eventlet. See #49.

Source code(tar.gz)
Source code(zip)
v1.5.0(Mar 24, 2021)
Added two string aliases for connection_handler option. Now you can conveniently use connection_handler="exec".

Improved handle_connection_exec. It now has a clean way to exit (exit()) and properly closes the socket.

Source code(tar.gz)
Source code(zip)
v1.4.0(Mar 24, 2021)
Added the connection_handler install option. Default value is manhole.handle_connection_repl, and alternate manhole.handle_connection_exec is provided (very simple: no output redirection, no stacktrace dumping).

Dropped Python 3.2 from the test grid. It may work but it's a huge pain to support (pip/pytest don't support it anymore).

Added Python 3.5 and 3.6 in the test grid.

Fixed issues with piping to manhole-cli. Now echo foobar | manhole-cli will wait 1 second for output from manhole (you can customize this with the --timeout option).

Fixed issues with newer PyPy (caused by gevent/eventlet socket unwrapping).

Source code(tar.gz)
Source code(zip)
v1.3.0(Mar 24, 2021)
Allowed Manhole to be configured without any thread or activation (in case you want to manually activate).

Added an example and tests for using Manhole with uWSGi.

Fixed error handling in manhole-cli on Python 3 (exc vars don't leak anymore).

Fixed support for running in gevent/eventlet-using apps on Python 3 (now that they support Python 3).

Allowed reinstalling the manhole (in non-strict mode). Previous install is undone.

Source code(tar.gz)
Source code(zip)
v1.2.0(Mar 24, 2021)
Changed manhole-cli:

Won't spam the terminal with errors if socket file doesn't exist.

Allowed sending any signal (new --signal argument).

Fixed some validation issues for the PID argument.

Source code(tar.gz)
Source code(zip)
v1.1.0(Mar 24, 2021)
Added support for installing the manhole via the PYTHONMANHOLE environment variable.

Added a strict install option. Set it to false to avoid getting the AlreadyInstalled exception.

Added a manhole-cli script that emulates socat readline unix-connect:/tmp/manhole-1234.

Source code(tar.gz)
Source code(zip)
v1.0.0(Jun 5, 2015)
Added socket_path install option (contributed by Nir Soffer).

Added reinstall_delay install option.

Added locals install option (contributed by Nir Soffer).

Added redirect_stderr install option (contributed by Nir Soffer).

Lots of internals cleanup (contributed by Nir Soffer).

Source code(tar.gz)
Source code(zip)
v0.6.2(Jun 5, 2015)
Fix OS X regression.

Source code(tar.gz)
Source code(zip)
v0.6.1(Jun 5, 2015)
Support for OS X (contributed by Saulius Menkevičius).

Source code(tar.gz)
Source code(zip)
v0.5.2(Oct 30, 2013)

Source code(tar.gz)
Source code(zip)
v0.5.1(Aug 19, 2013)
fixed issue in cleaning after closing manhole connection

Source code(tar.gz)
Source code(zip)
v0.5.0(Aug 17, 2013)
Python 3.2, 3.3 and PyPy support !

Fixed a bug with closing the connection socket (it was causing crashes in some rare scenarios)

Created a test matrix in Travis

Source code(tar.gz)
Source code(zip)
v0.4.0(Aug 13, 2013)

Source code(tar.gz)
Source code(zip)