Raise asynchronous exceptions in other thread, control the timeout of blocks or callables with a context manager or a decorator

Gilles Lenfant

Last update: Oct 12, 2022

Related tags

Concurrency and Parallelism stopit

Overview

stopit

Raise asynchronous exceptions in other threads, control the timeout of blocks or callables with two context managers and two decorators.

Attention!

API Changes

Users of 1.0.0 should upgrade their source code:

stopit.Timeout is renamed stopit.ThreadingTimeout
stopit.timeoutable is renamed stopit.threading_timeoutable

Explications follow below...

Contents

Overview
Installation
- Using stopit in your application
- Developing stopit
Public API
Known issues
- Timeout accuracy
- gevent support
Tests and demos
Links
Credits
License

Overview

This module provides:

a function that raises an exception in another thread, including the main thread.
two context managers that may stop its inner block activity on timeout.
two decorators that may stop its decorated callables on timeout.

Developed and tested with CPython 2.6, 2.7, 3.3 and 3.4 on MacOSX. Should work on any OS (xBSD, Linux, Windows) except when explicitly mentioned.

Note

Signal based timeout controls, namely SignalTimeout context manager and signal_timeoutable decorator won't work in Windows that has no support for signal.SIGALRM. Any help to work around this is welcome.

Installation

Using `stopit` in your application

Both work identically:

easy_install stopit
pip install stopit

Developing `stopit`

# You should prefer forking if you have a Github account
git clone https://github.com/glenfant/stopit.git
cd stopit
python setup.py develop

# Does it work for you ?
python setup.py test

Public API

Exception

`stopit.TimeoutException`

A stopit.TimeoutException may be raised in a timeout context manager controlled block.

This exception may be propagated in your application at the end of execution of the context manager controlled block, see the swallow_ex parameter of the context managers.

Note that the stopit.TimeoutException is always swallowed after the execution of functions decorated with xxx_timeoutable(...). Anyway, you may catch this exception within the decorated function.

Threading based resources

Warning

Threading based resources will only work with CPython implementations since we use CPython specific low level API. This excludes Iron Python, Jython, Pypy, ...

Will not stop the execution of blocking Python atomic instructions that acquire the GIL. In example, if the destination thread is actually executing a time.sleep(20), the asynchronous exception is effective after its execution.

`stopit.async_raise`

A function that raises an arbitrary exception in another thread

async_raise(tid, exception)

tid is the thread identifier as provided by the ident attribute of a thread object. See the documentation of the threading module for further information.
exception is the exception class or object to raise in the thread.

`stopit.ThreadingTimeout`

A context manager that "kills" its inner block execution that exceeds the provided time.

ThreadingTimeout(seconds, swallow_exc=True)

seconds is the number of seconds allowed to the execution of the context managed block.
swallow_exc : if False, the possible stopit.TimeoutException will be re-raised when quitting the context managed block. Attention: a True value does not swallow other potential exceptions.

Methods and attributes

of a stopit.ThreadingTimeout context manager.

Method / Attribute	Description
`.cancel()`	Cancels the timeout control. This method is intended for use within the block that's under timeout control, specifically to cancel the timeout control. Means that all code executed after this call may be executed till the end.
`.state`	This attribute indicated the actual status of the timeout control. It may take the value of the `EXECUTED`, `EXECUTING`, `TIMED_OUT`, `INTERRUPTED` or `CANCELED` attributes. See below.
`.EXECUTING`	The timeout control is under execution. We are typically executing within the code under control of the context manager.
`.EXECUTED`	Good news: the code under timeout control completed normally within the assigned time frame.
`.TIMED_OUT`	Bad news: the code under timeout control has been sleeping too long. The objects supposed to be created or changed within the timeout controlled block should be considered as non existing or corrupted. Don't play with them otherwise informed.
`.INTERRUPTED`	The code under timeout control may itself raise explicit `stopit.TimeoutException` for any application logic reason that may occur. This intentional exit can be spotted from outside the timeout controlled block with this state value.
`.CANCELED`	The timeout control has been intentionally canceled and the code running under timeout control did complete normally. But perhaps after the assigned time frame.

A typical usage:

import stopit
# ...
with stopit.ThreadingTimeout(10) as to_ctx_mgr:
    assert to_ctx_mgr.state == to_ctx_mgr.EXECUTING
    # Something potentially very long but which
    # ...

# OK, let's check what happened
if to_ctx_mgr.state == to_ctx_mgr.EXECUTED:
    # All's fine, everything was executed within 10 seconds
elif to_ctx_mgr.state == to_ctx_mgr.EXECUTING:
    # Hmm, that's not possible outside the block
elif to_ctx_mgr.state == to_ctx_mgr.TIMED_OUT:
    # Eeek the 10 seconds timeout occurred while executing the block
elif to_ctx_mgr.state == to_ctx_mgr.INTERRUPTED:
    # Oh you raised specifically the TimeoutException in the block
elif to_ctx_mgr.state == to_ctx_mgr.CANCELED:
    # Oh you called to_ctx_mgr.cancel() method within the block but it
    # executed till the end
else:
    # That's not possible

Notice that the context manager object may be considered as a boolean indicating (if True) that the block executed normally:

if to_ctx_mgr:
    # Yes, the code under timeout control completed
    # Objects it created or changed may be considered consistent

`stopit.threading_timeoutable`

A decorator that kills the function or method it decorates, if it does not return within a given time frame.

stopit.threading_timeoutable([default [, timeout_param]])

default is the value to be returned by the decorated function or method of when its execution timed out, to notify the caller code that the function did not complete within the assigned time frame.

If this parameter is not provided, the decorated function or method will return a None value when its execution times out.
```
@stopit.threading_timeoutable(default='not finished')
def infinite_loop():
    # As its name says...

result = infinite_loop(timeout=5)
assert result == 'not finished'
```
timeout_param: The function or method you have decorated may require a timeout named parameter for whatever reason. This empowers you to change the name of the timeout parameter in the decorated function signature to whatever suits, and prevent a potential naming conflict.
```
@stopit.threading_timeoutable(timeout_param='my_timeout')
def some_slow_function(a, b, timeout='whatever'):
    # As its name says...

result = some_slow_function(1, 2, timeout="something", my_timeout=2)
```

About the decorated function

or method...

As you noticed above, you just need to add the timeout parameter when calling the function or method. Or whatever other name for this you chose with the timeout_param of the decorator. When calling the real inner function or method, this parameter is removed.

Signaling based resources

Warning

Using signaling based resources will not work under Windows or any OS that's not based on Unix.

stopit.SignalTimeout and stopit.signal_timeoutable have exactly the same API as their respective threading based resources, namely stopit.ThreadingTimeout and stopit.threading_timeoutable.

See the comparison chart that warns on the more or less subtle differences between the Threading based resources and the Signaling based resources.

Logging

The stopit named logger emits a warning each time a block of code execution exceeds the associated timeout. To turn logging off, just:

import logging
stopit_logger = logging.getLogger('stopit')
stopit_logger.setLevel(logging.ERROR)

Comparing thread based and signal based timeout control

Feature	Threading based resources	Signaling based resources
GIL	Can't interrupt a long Python atomic instruction. e.g. if `time.sleep(20.0)` is actually executing, the timeout will take effect at the end of the execution of this line.	Don't care of it
Thread safety	Yes : Thread safe as long as each thread uses its own `ThreadingTimeout` context manager or `threading_timeoutable` decorator.	Not thread safe. Could yield unpredictable results in a multithreads application.
Nestable context managers	Yes : you can nest threading based context managers	No : never nest a signaling based context manager in another one. The innermost context manager will automatically cancel the timeout control of outer ones.
Accuracy	Any positive floating value is accepted as timeout value. The accuracy depends on the GIL interval checking of your platform. See the doc on `sys.getcheckinterval` and `sys.setcheckinterval` for your Python version.	Due to the use of `signal.SIGALRM`, we need provide an integer number of seconds. So a timeout of `0.6` seconds will ve automatically converted into a timeout of zero second!
Supported platforms	Any CPython 2.6, 2.7 or 3.3 on any OS with threading support.	Any Python 2.6, 2.7 or 3.3 with `signal.SIGALRM` support. This excludes Windows boxes

Known issues

Timeout accuracy

Important: the way CPython supports threading and asynchronous features has impacts on the accuracy of the timeout. In other words, if you assign a 2.0 seconds timeout to a context managed block or a decorated callable, the effective code block / callable execution interruption may occur some fractions of seconds after this assigned timeout.

For more background about this issue - that cannot be fixed - please read Python gurus thoughts about Python threading, the GIL and context switching like these ones:

This is the reason why I am more "tolerant" on timeout accuracy in the tests you can read thereafter than I should be for a critical real-time application (that's not in the scope of Python).

It is anyway possible to improve this accuracy at the expense of the global performances decreasing the check interval which defaults to 100. See:

If this is a real issue for users (want a precise timeout and not an approximative one), a future release will add the optional check_interval parameter to the context managers and decorators. This parameter will enable to lower temporarily the threads switching check interval, having a more accurate timeout at the expense of the overall performances while the context managed block or decorated functions are executing.

`gevent` support

Threading timeout control as mentioned in Threading based resources does not work as expected when used in the context of a gevent worker.

See the discussion in Issue 13 for more details.

Tests and demos

>>> import threading
>>> from stopit import async_raise, TimeoutException

In a real application, you should either use threading based timeout resources:

>>> from stopit import ThreadingTimeout as Timeout, threading_timeoutable as timeoutable  #doctest: +SKIP

Or the POSIX signal based resources:

>>> from stopit import SignalTimeout as Timeout, signal_timeoutable as timeoutable  #doctest: +SKIP

Let's define some utilities:

>>> import time
>>> def fast_func():
...     return 0
>>> def variable_duration_func(duration):
...     t0 = time.time()
...     while True:
...         dummy = 0
...         if time.time() - t0 > duration:
...             break
>>> exc_traces = []
>>> def variable_duration_func_handling_exc(duration, exc_traces):
...     try:
...         t0 = time.time()
...         while True:
...             dummy = 0
...             if time.time() - t0 > duration:
...                 break
...     except Exception as exc:
...         exc_traces.append(exc)
>>> def func_with_exception():
...     raise LookupError()

`async_raise` function raises an exception in another thread

Testing async_raise() with a thread of 5 seconds:

>>> five_seconds_threads = threading.Thread(
...     target=variable_duration_func_handling_exc, args=(5.0, exc_traces))
>>> start_time = time.time()
>>> five_seconds_threads.start()
>>> thread_ident = five_seconds_threads.ident
>>> five_seconds_threads.is_alive()
True

We raise a LookupError in that thread:

>>> async_raise(thread_ident, LookupError)

Okay but we must wait few milliseconds the thread death since the exception is asynchronous:

>>> while five_seconds_threads.is_alive():
...     pass

And we can notice that we stopped the thread before it stopped by itself:

>>> time.time() - start_time < 0.5
True
>>> len(exc_traces)
1
>>> exc_traces[-1].__class__.__name__
'LookupError'

`Timeout` context manager

The context manager stops the execution of its inner block after a given time. You may manage the way the timeout occurs using a try: ... except: ... construct or by inspecting the context manager state attribute after the block.

Swallowing Timeout exceptions

We check that the fast functions return as outside our context manager:

>>> with Timeout(5.0) as timeout_ctx:
...     result = fast_func()
>>> result
0
>>> timeout_ctx.state == timeout_ctx.EXECUTED
True

And the context manager is considered as True (the block executed its last line):

>>> bool(timeout_ctx)
True

We check that slow functions are interrupted:

>>> start_time = time.time()
>>> with Timeout(2.0) as timeout_ctx:
...     variable_duration_func(5.0)
>>> time.time() - start_time < 2.2
True
>>> timeout_ctx.state == timeout_ctx.TIMED_OUT
True

And the context manager is considered as False since the block did timeout.

>>> bool(timeout_ctx)
False

Other exceptions are propagated and must be treated as usual:

>>> try:
...     with Timeout(5.0) as timeout_ctx:
...         result = func_with_exception()
... except LookupError:
...     result = 'exception_seen'
>>> timeout_ctx.state == timeout_ctx.EXECUTING
True
>>> result
'exception_seen'

Propagating `TimeoutException`

We can choose to propagate the TimeoutException too. Potential exceptions have to be handled:

>>> result = None
>>> start_time = time.time()
>>> try:
...     with Timeout(2.0, swallow_exc=False) as timeout_ctx:
...         variable_duration_func(5.0)
... except TimeoutException:
...     result = 'exception_seen'
>>> time.time() - start_time < 2.2
True
>>> result
'exception_seen'
>>> timeout_ctx.state == timeout_ctx.TIMED_OUT
True

Other exceptions must be handled too:

>>> result = None
>>> start_time = time.time()
>>> try:
...     with Timeout(2.0, swallow_exc=False) as timeout_ctx:
...         func_with_exception()
... except Exception:
...     result = 'exception_seen'
>>> time.time() - start_time < 0.1
True
>>> result
'exception_seen'
>>> timeout_ctx.state == timeout_ctx.EXECUTING
True

`timeoutable` callable decorator

This decorator stops the execution of any callable that should not last a certain amount of time.

You may use a decorated callable without timeout control if you don't provide the timeout optional argument:

>>> @timeoutable()
... def fast_double(value):
...     return value * 2
>>> fast_double(3)
6

You may specify that timeout with the timeout optional argument. Interrupted callables return None:

>>> @timeoutable()
... def infinite():
...     while True:
...         pass
...     return 'whatever'
>>> infinite(timeout=1) is None
True

Or any other value provided to the timeoutable decorator parameter:

>>> @timeoutable('unexpected')
... def infinite():
...     while True:
...         pass
...     return 'whatever'
>>> infinite(timeout=1)
'unexpected'

If the timeout parameter name may clash with your callable signature, you may change it using timeout_param:

>>> @timeoutable('unexpected', timeout_param='my_timeout')
... def infinite():
...     while True:
...         pass
...     return 'whatever'
>>> infinite(my_timeout=1)
'unexpected'

It works on instance methods too:

>>> class Anything(object):
...     @timeoutable('unexpected')
...     def infinite(self, value):
...         assert type(value) is int
...         while True:
...             pass
>>> obj = Anything()
>>> obj.infinite(2, timeout=1)
'unexpected'

Credits

This is a NIH package which is mainly a theft of Gabriel Ahtune's recipe with tests, minor improvements and refactorings, documentation and setuptools awareness I made since I'm somehow tired to copy/paste this recipe among projects that need timeout control.
Gilles Lenfant: package creator and maintainer.

License

This software is open source delivered under the terms of the MIT license. See the LICENSE file of this repository.

Comments

threading locks and stopit

I've added stopit to the w3af framework with the objective of killing document parsers (w3af is a crawler with security features) which take more than N seconds to parse an HTTP response body.

While running some test scans I've noticed (but not yet confirmed 100%) that in some cases all the threads are dead-locked waiting for a threading.RLock to be released:

w3af-10287-2015-05-01-05_26.threads:        "  File \"parser_cache.py\", line 84, in get_document_parser_for\n    with self._LRULock:\n", 
w3af-10287-2015-05-01-05_26.threads:        "  File \"parser_cache.py\", line 84, in get_document_parser_for\n    with self._LRULock:\n", 
...
w3af-10287-2015-05-01-05_26.threads:        "  File \"parser_cache.py\", line 84, in get_document_parser_for\n    with self._LRULock:\n",

This output is generated by this thread traceback dump script which hasn't failed for me in the past. When the framework parser cache is working as expected the output looks like:

w3af-10287-2015-05-01-05_19.threads:        "  File \"parser_cache.py\", line 84, in get_document_parser_for\n    with self._LRULock:\n", 
w3af-10287-2015-05-01-05_19.threads:        "  File \"parser_cache.py\", line 90, in get_document_parser_for\n    res = DocumentParser.DocumentParser(http_response)\n", 
w3af-10287-2015-05-01-05_19.threads:        "  File \"parser_cache.py\", line 84, in get_document_parser_for\n    with self._LRULock:\n",

The pseudo-code for the code I'm having problems with is:

with self._LRULock:
    try:
        with ThreadingTimeout(self.PARSER_TIMEOUT, swallow_exc=False):
            self._parser = parser(http_resp)
    except TimeoutException:
        logging.debug(...)

I know that the timeout is being triggered because the debug message appears in the logs.

The question is... is there any known bug with stopit , PyThreadState_SetAsyncExc and threading locks?

I've tried to reproduce this issue in this commit but all seems to be working as expected.

opened by andresriancho 9

Inconsistent failing on testing on Python 2

I don't know how stopit does behind the scene, but just want to give some feedback.

The following command doesn't give same test results every time I ran it:

$ python2 setup.py test
[snip]
Doctest: README.rst ... FAIL
======================================================================
FAIL: /home/livibetter/p/stopit/README.rst
Doctest: README.rst
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/doctest.py", line 2201, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for README.rst
  File "/home/livibetter/p/stopit/README.rst", line 0

----------------------------------------------------------------------
File "/home/livibetter/p/stopit/README.rst", line 174, in README.rst
Failed example:
    time.time() - start_time < 2.1
Expected:
    True
Got:
    False
----------------------------------------------------------------------
File "/home/livibetter/p/stopit/README.rst", line 204, in README.rst
Failed example:
    time.time() - start_time < 2.1
Expected:
    True
Got:
    False


----------------------------------------------------------------------
Ran 1 test in 8.948s

FAILED (failures=1)

The result above has two failures, but sometimes both pass, sometimes only one of them fails. The testing should be have same results every time, maybe there is something with Python 2?

I tried with Python 3 a few times, it seems all passed, which is consistent.

 ~/p/stopit [git:master] $ python2 --version
Python 2.7.5
 ~/p/stopit [git:master] $ python3 --version
Python 3.3.3
 ~/p/stopit [git:master] $ uname -a
Linux dfed 3.10.25-gentoo-1 #1 SMP PREEMPT Tue Jan 21 03:19:58 CST 2014 x86_64 Intel(R) Core(TM)2 CPU T5600 @ 1.83GHz GenuineIntel GNU/Linux

As I said in beginning, I don't know anything about stopit, but I can help with information from my system if that helps.

opened by livibetter 6

TimeoutException is not raised if using subprocess
I'm using Tesseract as OCR engine to process some documents. However, I would like to restrict the time of execution for tesseract process max to 5 mins.

The problem is that the code below sometimes kills the process properly after 5 mins, but sometimes I see a slowdown in the system due to tesseract process running for much longer. For example, today I've noticed in logs that it took 37 minutes between Will run OCR... and Tesseract timed out. Killing the ocr process ... (Code block execution exceeded 300 seconds timeout internal stopit log message).

try: with stopit.ThreadingTimeout(300, swallow_exc=False): logger.info('Will run OCR...') command = 'tesseract {} {} --psm 1 -l eng hocr'.format(image_path, hocr_file_path).split() ocr_process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) # Wait for the tesseract process to finish and fetch stdout/stderr stdout, stderr = self.ocr_process.communicate() # ... # Do some processing of results except stopit.utils.TimeoutException: logger.exception('Tesseract timed out. Killing the ocr process ...') ocr_process.kill() # ...

Can you help me please to understand if I'm using stopit for the wrong scenario or what can be the problem?
opened by zastupailo 5

Why does this stopit code fail?

I am facing a strange issue with the stopit module. The below code works fine:

import stopit

def check(num):
    with stopit.ThreadingTimeout(3) as to_ctx_mrg:
        count = 0
        while(True):

            if False:
                continue    

    if to_ctx_mrg.state == to_ctx_mrg.TIMED_OUT:
            print(to_ctx_mrg.state)
            print("Time out")
            raise TimeoutError

try:
    check(1)
except TimeoutError:
    print("Caught the timeout")

But, when I change the if condition slightly, stopit is not able to recognize this infinite loop and the program does not stop. Take a look at the modified code:

import stopit

def check(num):
    with stopit.ThreadingTimeout(3) as to_ctx_mrg:
        count = 0
        while(True):

            if num>31:
                continue    

    if to_ctx_mrg.state == to_ctx_mrg.TIMED_OUT:
            print(to_ctx_mrg.state)
            print("Time out")
            raise TimeoutError

try:
    check(1)
except TimeoutError:
    print("Caught the timeout")

opened by VeereshElango 5

Not working with gunicorn + gevent workers

I'm trying to use the threading_timeoutable decorator on a function. It works if I run the script directly, but doesn't work when running via gunicorn using gevent workers.

Any ideas on making it work?

opened by anshulk 4
pybuild is failing

I: pybuild base:184: /usr/bin/python3 setup.py build running build running build_py creating /tmp/b/a/stopit/.pybuild/pythonX.Y_3.5/build/stopit copying src/stopit/init.py -> /tmp/b/a/stopit/.pybuild/pythonX.Y_3.5/build/stopit copying src/stopit/signalstop.py -> /tmp/b/a/stopit/.pybuild/pythonX.Y_3.5/build/stopit copying src/stopit/utils.py -> /tmp/b/a/stopit/.pybuild/pythonX.Y_3.5/build/stopit copying src/stopit/threadstop.py -> /tmp/b/a/stopit/.pybuild/pythonX.Y_3.5/build/stopit dh_auto_test -O--buildsystem=pybuild I: pybuild base:184: cd /tmp/b/a/stopit/.pybuild/pythonX.Y_2.7/build; python2.7 -m pytest ============================= test session starts ============================== platform linux2 -- Python 2.7.11+, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 rootdir: /tmp/b/a/stopit, inifile: collected 0 items

========================= no tests ran in 0.00 seconds ========================= E: pybuild pybuild:274: test: plugin distutils failed with: exit code=5: cd /tmp/b/a/stopit/.pybuild/pythonX.Y_2.7/build; python2.7 -m pytest dh_auto_test: pybuild --test --test-pytest -i python{version} -p 2.7 --dir . returned exit code 13 debian/rules:4: recipe for target 'build' failed make: *** [build] Error 25 dpkg-buildpackage: error: debian/rules build gave error exit status 2 debuild: fatal error at line 1376:

opened by alvesadrian 3

stopit not timing out while opening file

Hi,

On windows 10 stopit doesn't seem to timeout at all when doing file reads with pyarrow.Parquet. This works fine for me:

import stopit
import pyarrow.parquet as pq

def infiniteloop():
    with stopit.ThreadingTimeout(5) as to_ctx_mgr:
        assert to_ctx_mgr.state == to_ctx_mgr.EXECUTING
        while True:
            pass
    print(str(to_ctx_mgr.state))
    return False

infiniteloop()

however this:

import stopit
import pyarrow.parquet as pq

def ReadMetaFromDataset(filepath):
    with stopit.ThreadingTimeout(5) as to_ctx_mgr:
        assert to_ctx_mgr.state == to_ctx_mgr.EXECUTING
        schema = pq.ParquetDataset(filepath).schema
        return schema
    print(str(to_ctx_mgr.state))
    return False

ReadMetaFromDataset(r"\\lcdevnas01\development\ENav2")

and this:

import stopit.threading_timeoutable
import pyarrow.parquet as pq

@threading_timeoutable(default='not finished')
def ReadMetaFromDataset(filepath):
    with stopit.ThreadingTimeout(10) as to_ctx_mgr:
        assert to_ctx_mgr.state == to_ctx_mgr.EXECUTING
        schema = pq.ParquetDataset(filepath).schema
        return schema
    print(str(to_ctx_mgr.state))
    return False

print(ReadMetaFromDataset(r"\\lcdevnas01\development\ENav2", timeout=5))

runs to completion after 90 minutes. Anything I can do with stopit to force timeouts or do I have to look into threading and kill from a separate process?

Thanks Asbjorn

opened by asbjornb 1

Block does not stop after timeout
I am using version 1.1.2 with Python 3.7.

My goal is to use this module to wrap a call to the Python Docker SDK's wait function so that I can determine whether a container finishes executing within a timeout, and kill the container if it continues executing beyond that timeout.

I have code that looks like this:

container = # ... a detached docker container with stopit.ThreadingTimeout(10) as timeout_ctx: assert timeout_ctx.state == timeout_ctx.EXECUTING container.wait() # blocking call if timeout_ctx.state == timeout_ctx.EXECUTED: print("Executed within timeout") elif timeout_ctx.state == timeout_ctx.TIMED_OUT: print("Executed beyond timeout")

For containers that run beyond the (10 second) timeout above, I see Executed beyond timeout as I'd expect. However, the wait() call continues blocking until the container finishes (which could be much more time than 10 seconds).

Is this expected behavior?
opened by nmagerko 1
Question: stopit uses user time or wall time?

I am wondering if the ThreadingTimeout a proper CPU resource cost measure. I want to control the timeout of a numerical simulation which should be regarded as wrong if it costs too much time. Normally it should be completed in less than one second. If the running time exceed 1 second, the parameter set might be wrong and it will never stop.

However, AFAIK user time is used in thread rather than wall time. I am not sure what stopit is using to measure the timeout.

Additionally, part of CPU consuming algorithm of my simulation is written in cython. Will it be a problem if it do not release GIL?

opened by ghost 1
Typographical errors in example

Hello, on https://pypi.python.org/pypi/stopit the example shows "status" but I believe "state" is desired. Also, "mgr" is sometimes spelled "mrg". The example that worked for me:

import stopit import time

...

with stopit.ThreadingTimeout(3) as to_ctx_mgr: assert to_ctx_mgr.state == to_ctx_mgr.EXECUTING # Something potentially very long but which time.sleep(2)

OK, let's check what happened

if to_ctx_mgr.state == to_ctx_mgr.EXECUTED: # All's fine, everything was executed within 10 seconds print("good") elif to_ctx_mgr.state == to_ctx_mgr.EXECUTING: # Hmm, that's not possible outside the block print("not possible") elif to_ctx_mgr.state == to_ctx_mgr.TIMED_OUT: # Eeek the 10 seconds timeout occurred while executing the block print("timeout") elif to_ctx_mgr.state == to_ctx_mgr.INTERRUPTED: # Oh you raised specifically the TimeoutException in the block print("timeout exception raised") elif to_ctx_mgr.state == to_ctx_mgr.CANCELED: # Oh you called to_ctx_mgr.cancel() method within the block but it # executed till the end print("canceled") else: # That's not possible pass

Thank you for providing this software to the world! I appreciate it.

opened by jsf80238 1
Add a check_interval parameter to the decorators and context managers

Read https://github.com/glenfant/stopit/issues/2#issuecomment-42029222 for more background about this future change.

Add a check_interval=None parameter to the context manager constructors and the decorators to give a chance to the programmer to improve timeout accuracy at the expense of the performances of the decorated function or context manager inner block.

See https://docs.python.org/2.7/library/sys.html#sys.setcheckinterval about the check interval.
enhancement

opened by glenfant 1
make swallow_exc a paramater with default value

Hi, thanks for the great library!

Currently, swallow_exc here is hardcoded to True, which means that using the decorator @stopit.threading_timeoutable() will never raise a stopit.utils.TimeoutException

This change allows the user to decide whether they want the exception to be swallowed when using a decorator, and includes logging to say whether the exception will be swallowed

opened by jsmith2021Brandeis 0
Whl file

Hi, I want to use this on an offline computer. I am struggling to install it as I cannot find a wheel file for it, and pip keeps failing to create one when I try installing it through the folder. Error: Directory C:.... is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

opened by thelawati 0

Your code doesn't work

import time
from functools import wraps

import stopit

class TimeoutError(Exception):
  pass

def timeouter(timeout):
  def decorator(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
      with stopit.ThreadingTimeout(timeout) as to_ctx_mgr:
        result = func(*args, **kwargs) # <- Function is working 100 seconds but "stopit" doesn't stop it.
        print(func.__name__+" ok")
        return result
      if to_ctx_mgr.state == to_ctx_mgr.TIMED_OUT:
        print(func.__name__+" timed out")
        raise TimeoutError
    return wrapper
  return decorator

@timeouter(2)
def sleepy():
  time.sleep(100)
  return "return"

sleepy()

opened by crazyfrogggg 0

Problem with very short timeouts, including zero

I have a tool for which it makes sense to allow a timeout to be specified on the command line. It also makes sense for a user to try a timeout of zero to see what the timeout failure looks like. But this doesn't work: despite not setting swallow_exc to False, the TimeoutExceptiongets raised. Digging into threading.Timer a bit, this appears to be the result of a 'feature', in that a timeout of 0 is treated specially, causing the stop function to be called from inside the call to Timer.start.

This in turn appears to trigger a race condition wrt the ___exit__ clause on BaseTimeout, based on evidence from the following experiment.

The attached code is a trivial use of stopit, with timeout supplied from the command line.

TL;DR Illustrates the problem, reproduced on various systems. Also discovers a related problem on Windows 10, for very small values of the timeout

How to reproduce

Here are four example input/output pairs, two good to start, then two that illustrate the two problems: Runs w/o timing out

>: while :; do python3 test_stopit.py 1; done
done 0
done 0
done 0
done 0
done 0
[repeats]

Times out

>: while :; do python3 test_stopit.py 0.01; done
done 2
done 2
done 2
done 2
done 2
[repeats]

Timeout of 0: always raises TimeoutException

>: while :; do python3 test_stopit.py 0; done
Traceback (most recent call last):
  File "test_stopit.py", line 3, in <module>
    with stopit.ThreadingTimeout(float(sys.argv[1])) as to_ctx_mgr:
  File "/usr/local/lib/python3.8/site-packages/stopit/utils.py", line 73, in __enter__
    self.setup_interrupt()
  File "/usr/local/lib/python3.8/site-packages/stopit/threadstop.py", line 60, in setup_interrupt
    self.timer.start()
  File "/usr/lib/python3.8/threading.py", line 857, in start
    self._started.wait()
  File "/usr/lib/python3.8/threading.py", line 558, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.8/threading.py", line 302, in wait
    waiter.acquire()
stopit.utils.TimeoutException
[repeats]

Very small timeout, two different results

>: while :; do python3 test_stopit.py 0.0000000001; done
done 2
done 2
[repeats 5 times]
done 2
Traceback (most recent call last):
  File "test_stopit.py", line 3, in <module>
    with stopit.ThreadingTimeout(float(sys.argv[1])) as to_ctx_mgr:
  File "/usr/local/lib/python3.8/site-packages/stopit/utils.py", line 73, in __enter__
    self.setup_interrupt()
  File "/usr/local/lib/python3.8/site-packages/stopit/threadstop.py", line 60, in setup_interrupt
    self.timer.start()
  File "/usr/lib/python3.8/threading.py", line 857, in start
    self._started.wait()
  File "/usr/lib/python3.8/threading.py", line 558, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.8/threading.py", line 302, in wait
    waiter.acquire()
stopit.utils.TimeoutException
done 2
done 2
Traceback (most recent call last):
 [as above]
stopit.utils.TimeoutException
Traceback (most recent call last):
  [as above]
stopit.utils.TimeoutException
done 2
[etc., _ad nauseum]

The above were run under Cygwin, Python 3.8.10. Under Ubuntu focal, Python 3.8.10, the zero case fails as above, but the very small timeout always successfully gives done 2. Likewise with Debian buster, Python 3.7.3 and several other Linux distros. Under Windows 10, Python 3.8.10 however, the zero case sometimes fails as above (TimeoutExceptionunder waitout.acquire), and sometimes trips the assertion error!

AssertionError: state is 2, not 1

For the very small timeout value, we get some successes, i.e. 'done 2', and also some failures, but the failure surfaces at a different point:

>python test_stopit.py 0.0000000001
done 0
Traceback (most recent call last):
  File "D:\work\jar\doi\pdf1\test_stopit.py", line 9, in <module>
    print('done',to_ctx_mgr.state)
stopit.utils.TimeoutException

Conclusions It would be nice to

Handle a timeout of zero in stopit itself, or at the very least include a warning in the documentation;
Debug the race condition on Windows/Cygwin (I'm assuming the underlying problem is the same), but this probably needs to involve someone familiar with the internals of the threading module...

Test code

import stopit, sys

with stopit.ThreadingTimeout(float(sys.argv[1])) as to_ctx_mgr:
    assert to_ctx_mgr.state == to_ctx_mgr.EXECUTING, \
            "state is %s, not %s"%(to_ctx_mgr.state, to_ctx_mgr.EXECUTING)
    for i in range(1000000):
      if i>0:
        pass

print('done',to_ctx_mgr.state)

opened by htInEdin 0

Python 2.7.15 doesn't respect swallow_exc

Running the following simple code with python 2.7.15:

from stopit import ThreadingTimeout                                                                                     
                                                                                                                        
def func():                                                                                                             
    with ThreadingTimeout(1, swallow_exc=True):                                                                         
        import time; time.sleep(2)                                                                                      
                                                                                                                         
if __name__ == '__main__':                                                                                               
    func()

I get the following exception being thrown:

Traceback (most recent call last):
  File "main.py", line 8, in <module>
    func()
  File "main.py", line 5, in func
    import time; time.sleep(2)
  File "py_envs/py2/lib/python2.7/site-packages/stopit/utils.py", line 87, in __exit__
    self.suppress_interrupt()
  File "py_envs/py2/lib/python2.7/site-packages/stopit/threadstop.py", line 65, in suppress_interrupt
    self.timer.cancel()
  File "homebrew/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 1066, in cancel
    def cancel(self):
stopit.utils.TimeoutException

When I run with python 3 I don't get any output as expected, because I've swallowed the exception. Any ideas?

opened by ferdinandvanwyk 1