Retrying is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything.

Overview

Retrying

Retrying is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything.

The simplest use case is retrying a flaky function whenever an Exception occurs until a value is returned.

import random
from retrying import retry

@retry
def do_something_unreliable():
    if random.randint(0, 10) > 1:
        raise IOError("Broken sauce, everything is hosed!!!111one")
    else:
        return "Awesome sauce!"

print do_something_unreliable()

Features

  • Generic Decorator API
  • Specify stop condition (i.e. limit by number of attempts)
  • Specify wait condition (i.e. exponential backoff sleeping between attempts)
  • Customize retrying on Exceptions
  • Customize retrying on expected returned result

Installation

To install retrying, simply:

$ pip install retrying

Or, if you absolutely must:

$ easy_install retrying

But, you might regret that later.

Examples

As you saw above, the default behavior is to retry forever without waiting.

@retry
def never_give_up_never_surrender():
    print "Retry forever ignoring Exceptions, don't wait between retries"

Let's be a little less persistent and set some boundaries, such as the number of attempts before giving up.

@retry(stop_max_attempt_number=7)
def stop_after_7_attempts():
    print "Stopping after 7 attempts"

We don't have all day, so let's set a boundary for how long we should be retrying stuff.

@retry(stop_max_delay=10000)
def stop_after_10_s():
    print "Stopping after 10 seconds"

Most things don't like to be polled as fast as possible, so let's just wait 2 seconds between retries.

@retry(wait_fixed=2000)
def wait_2_s():
    print "Wait 2 second between retries"

Some things perform best with a bit of randomness injected.

@retry(wait_random_min=1000, wait_random_max=2000)
def wait_random_1_to_2_s():
    print "Randomly wait 1 to 2 seconds between retries"

Then again, it's hard to beat exponential backoff when retrying distributed services and other remote endpoints.

@retry(wait_exponential_multiplier=1000, wait_exponential_max=10000)
def wait_exponential_1000():
    print "Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards"

We have a few options for dealing with retries that raise specific or general exceptions, as in the cases here.

def retry_if_io_error(exception):
    """Return True if we should retry (in this case when it's an IOError), False otherwise"""
    return isinstance(exception, IOError)

@retry(retry_on_exception=retry_if_io_error)
def might_io_error():
    print "Retry forever with no wait if an IOError occurs, raise any other errors"

@retry(retry_on_exception=retry_if_io_error, wrap_exception=True)
def only_raise_retry_error_when_not_io_error():
    print "Retry forever with no wait if an IOError occurs, raise any other errors wrapped in RetryError"

We can also use the result of the function to alter the behavior of retrying.

def retry_if_result_none(result):
    """Return True if we should retry (in this case when result is None), False otherwise"""
    return result is None

@retry(retry_on_result=retry_if_result_none)
def might_return_none():
    print "Retry forever ignoring Exceptions with no wait if return value is None"

Any combination of stop, wait, etc. is also supported to give you the freedom to mix and match.

Contribute

  1. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.
  2. Fork the repository on GitHub to start making your changes to the master branch (or branch off of it).
  3. Write a test which shows that the bug was fixed or that the feature works as expected.
  4. Send a pull request and bug the maintainer until it gets merged and published. :) Make sure to add yourself to AUTHORS.
Issues
  • Just depend on six

    Just depend on six

    Instead of copying six functions into retrying codebase why not just require the six module itself? This was if six updates then you automatically get this update as well...

    question 
    opened by harlowja 13
  • Support for marked wait

    Support for marked wait

    Often times retrying is done in a stepping manner whereupon the first N attempts wait X ms, the next M attempts wait for Y ms, etc. This patch implements support for such waiting by introducing the notion of 'wait markers'. Wait markers are simply tuples that define a set of attempts and their corresponding waits to go into effect.

    A handful of unit tests are also included.

    opened by bodenr 10
  • Retry on coroutines

    Retry on coroutines

    Hi,

    I'm a happy user of retrying, and i would like to use its features to retry coroutines. I wrote a PoC at https://github.com/novafloss/jenkins-epo/blob/master/jenkins_epo/utils.py#L34 , but i would like to contribute a more sane and integrated support.

    So here is my contribution :) It comes with one unit test which i hope is enough for code coverage.

    What do you think of this ? I would like to make it as consistent as possible with existing code and minimal.

    Regards, Étienne

    opened by bersace 9
  • Support retrying as context manager

    Support retrying as context manager

    Would be very helpful to support retrying as a context manager:

    Something along the lines of:

    def do_something_unreliable():
        if random.randint(0, 10) > 1:
            raise IOError("Broken sauce, everything is hosed!!!111one")
        else:
            return "Awesome sauce!"
    
    with retry(wait_fixed=2000):
        do_something_unreliable()
    
    opened by odedfos 7
  • Extend API to allow specifying the exception class

    Extend API to allow specifying the exception class

    Adds an on_exception argument to specify the class of exception to catch. This provides a cleaner API to catching only specific exception classes.

    Opening this in part as a POC and to start discussion. I think it is a much nicer API for this use case and also solves the issue of catching BaseException by default.

    See #17

    opened by d0ugal 7
  • Adding a back-off algorithm: Exponential Backoff And Jitter

    Adding a back-off algorithm: Exponential Backoff And Jitter

    As described in https://www.awsarchitectureblog.com/2015/03/backoff.html, this allows better distribution of competing clients across time during the retries.

    opened by RobinNil 4
  • Callbacks for before/after attempts

    Callbacks for before/after attempts

    Adds two additional parameters to the Retrying() constructor:

    _before_attempts
    _after_attempts
    

    Each of these should be a reference to a function that takes a single argument, attempt_number.

    _before_attempts will be called before each attempt is run; _after_attempts is run after any unsuccessful attempt.

    Handy for logging status messages, etc in between attempts.

    opened by cyrusd 3
  • Raising a `RetryError` on failure.

    Raising a `RetryError` on failure.

    Hi, I just found your library and it looks to do exactly what I was looking for. However when launching the code below, I am getting a RetryError. Is is the intended behavior?

    from retrying import retry, RetryError
    
    @retry(wait_fixed=100,
           stop_max_attempt_number=10,
           wrap_exception=False)
    def foo():
        raise ValueError()
    
    try:
        foo()
    except ValueError:
        print "ValueError"
    except RetryError as e:
        print "RetryError"
    

    Intuitively, I was expecting a ValueError. That would let me use the retry decorator without modifying the rest of the code.

    bug 
    opened by simondolle 3
  • require six >= 1.7

    require six >= 1.7

    retrying uses six.wraps, which requires at least version 1.7: https://bitbucket.org/gutworth/six/src/35f6d89f826f63486214e4746af028ccaa088901/six.py?at=1.7.0#cl-689

    opened by jobevers 2
  • python 2.6 / AttributeError: 'module' object has no attribute 'wraps'

    python 2.6 / AttributeError: 'module' object has no attribute 'wraps'

    hi, does someone have a clue on this error ? thanks

    $ python -V Python 2.6.6 $$ >>> six.version '1.6.1'

    $ python setup.py test running test running egg_info writing requirements to retrying.egg-info/requires.txt writing retrying.egg-info/PKG-INFO writing top-level names to retrying.egg-info/top_level.txt writing dependency_links to retrying.egg-info/dependency_links.txt reading manifest file 'retrying.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'retrying.egg-info/SOURCES.txt' running build_ext Traceback (most recent call last): File "setup.py", line 57, in setup(**settings) File "/usr/lib/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/usr/lib/python2.6/dist-packages/setuptools/command/test.py", line 137, in run self.with_project_on_sys_path(self.run_tests) File "/usr/lib/python2.6/dist-packages/setuptools/command/test.py", line 117, in with_project_on_sys_path func() File "/usr/lib/python2.6/dist-packages/setuptools/command/test.py", line 146, in run_tests testLoader = loader_class() File "/usr/lib/python2.6/unittest.py", line 816, in init self.parseArgs(argv) File "/usr/lib/python2.6/unittest.py", line 843, in parseArgs self.createTests() File "/usr/lib/python2.6/unittest.py", line 849, in createTests self.module) File "/usr/lib/python2.6/unittest.py", line 613, in loadTestsFromNames suites = [self.loadTestsFromName(name, module) for name in names] File "/usr/lib/python2.6/unittest.py", line 576, in loadTestsFromName module = import('.'.join(parts_copy)) File "/root/retrying-master/test_retrying.py", line 245, in @retry(wait_fixed=50, retry_on_result=retry_if_result_none) File "/root/retrying-master/retrying.py", line 47, in wrap @six.wraps(f) AttributeError: 'module' object has no attribute 'wraps'

    opened by nesies 2
  • Use functools/six.wraps (or other).

    Use functools/six.wraps (or other).

    Instead of having a decorator which loses the parents docstrings and name and other the decorator should be using functools.wraps (or six.wraps). Or even better it seems like you could just use https://pypi.python.org/pypi/wrapt which is likely better than the wraps() methods.

    enhancement 
    opened by harlowja 2
  • Update retrying.py

    Update retrying.py

    Add print_exception attribute for exception printing for every attempt. For example

    ####Define functions (before_attempts, after_attempts)#### def print_attemt(attempt): print(f"Start Attempt # {attempt}") return attempt

    def print_attemt_after(attempt): print(f"Attempt # {attempt} was failed") return attempt

    Then call a func function with retry (we also can use retry as a decorator with func definition)

    retry(stop_max_attempt_number=3, print_exception=True, before_attempts=print_attemt, after_attempts=print_attemt_after)(func)(par)

    After func call we get the below

    Start Attempt # 1 [Errno 2] No such file or directory: 'F:\TEST\test.txt' Attempt # 1 was failed Start Attempt # 2 [Errno 2] No such file or directory: 'F:\TEST\test.txt' Attempt # 2 was failed Start Attempt # 3 [Errno 2] No such file or directory: 'F:\TEST\test.txt' Attempt # 3 was failed

    opened by Sergfalt 0
  • Creating a new release from master

    Creating a new release from master

    Hey guys, could you please create a new release (1.3.4?) from master? I really need the latests commits that are found in master.

    Thanks in advance :)

    opened by chenvsy 0
  • Method retry

    Method retry

    I want to decorate methods of a class with @retry and use a specific method as the "retry_on_exception" function. Issue is of course, that you can not do

    @retry(retry_on_exception=self.retry_handler)
    

    since self is not defined in that case. This code snippet is an attempt to work around this:

    def method_retry(*dargs, **dkw):
        def decorator_retry(func):
            @functools.wraps(func)
            def retry_func(self, *args, **kwargs):
                fixed_dkw = dkw.copy()
                if retry_on_ex := fixed_dkw.get("retry_on_exception"):
                    fixed_dkw["retry_on_exception"] = lambda ex: retry_on_ex(self, ex)
                
                if retry_on_res := fixed_dkw.get("wait_on_result"):
                    fixed_dkw["retry_on_result"] = lambda res: retry_on_res(self, res)
    
                return Retrying(*dargs, **fixed_dkw).call(func, self, *args, **kwargs)
            return retry_func
        return decorator_retry
    
    

    *dargs would have to be modified similarly to kwargs. There might be an even more elegant solution though. But this is the best I could come up with so far.

    This seems to work, although you can not do

    class MyClass:
        def my_exception_handler(self):
            pass
    
        @method_retry(retry_on_exception=MyClass.my_exception_handler)
        def my_method(self):
            pass
    

    since MyClass is not defined at that point yet.

    so you have to do

    def my_exception_handler(self):
         pass
    
    class MyClass:
        @method_retry(retry_on_exception=my_exception_handler)
        def my_method(self):
            pass
    

    which is quite annoying.

    Would be cool if anyone had a better idea.

    opened by FelixBenning 0
  • Wheel support for linux aarch64/x86

    Wheel support for linux aarch64/x86

    Summary Installing retrying on aarch64/x86 via pip using command "pip3 install retrying" tries to build wheel from source code

    Problem description retrying doesn't have wheel for aarch64/x86 on PyPI repository. So, while installing retrying via pip on aarch64/x86 machine, pip builds the source code resulting in it takes more time to install retrying. Making wheel available for aarch64/x86 will benefit aarch64/x86 users by minimizing retrying installation time. We can release the wheel using CI/CD tools such as travis-ci, circle-ci, azure-pipelne etc..

    Expected Output Pip should be able to download retrying wheel from PyPI repository rather than building it from source code.

    @retrying-taem, please let me know if I can help you building wheel/uploading to PyPI repository. I am curious to make retrying wheel available for aarch64. It will be a great opportunity for me to work with you.

    opened by odidev 0
  • retrying

    retrying

    using the retrying package. How do I use it for retrying, and if the retries ends without a successful match, the code block leaves?

    import random
    from retrying import retry
    
    @retry(wait_fixed=500, stop_max_delay=5000 )
    def do_something_unreliable():
        result = random.randint(0, 10)
        print result
        if result != 11:
            raise IOError("Broken")
        else:
            return True
    
    print do_something_unreliable()
    

    I want the the raise IOError to be replaced with an option of returning, say, False

    In other words, I want to use this decorator, in case of actual values is different than expected, while the actual is created frequently from other function

    opened by itaybz 3
Owner
Ray Holder
Ray Holder
A general purpose low level programming language written in Python.

A general purpose low level programming language written in Python. Basal is an easy mid level programming language compiling to C. It has an easy syntax, similar to Python, Rust etc.

Snm Logic 6 Mar 30, 2022
Repo created for the purpose of adding any kind of programs and projects

Programs and Project Repository A repository for adding programs and projects of any kind starting from beginners level to expert ones Contributing to

Unicorn Dev Community 2 Oct 30, 2021
Repo created for the purpose of adding any kind of programs and projects

Programs and Project Repository A repository for adding programs and projects of any kind starting from beginners level to expert ones Contributing to

Unicorn Dev Community 2 Oct 30, 2021
General Purpose Python Library by Techman

General Purpose Python Library by Techman

Jack Hubbard 0 Feb 9, 2022
Nimbus - Open Source Cloud Computing Software - 100% Apache2 licensed

⚠️ The Nimbus infrastructure project is no longer under development. ⚠️ For more information, please read the news announcement. If you are interested

Nimbus 195 Jun 14, 2022
Sodium is a general purpose programming language which is instruction-oriented

Sodium is a general purpose programming language which is instruction-oriented (a new programming concept that we are developing and devising)

Satin Wuker 22 Jan 11, 2022
A general-purpose wallet generator, for supported coins only

2gen A general-purpose generator for keys. Designed for all cryptocurrencies supporting the Bitcoin format of keys and addresses. Functions To enable

Vlad Usatii 1 Jan 12, 2022
A python script to simplify recompiling, signing and installing reverse engineered android apps.

urszi.py A python script to simplify the Uninstall Recompile Sign Zipalign Install cycle when reverse engineering Android applications. It checks if d

Ahmed Harmouche 3 May 29, 2021
Python client SDK designed to simplify integrations by automating key generation and certificate enrollment using Venafi machine identity services.

This open source project is community-supported. To report a problem or share an idea, use Issues; and if you have a suggestion for fixing the issue,

Venafi, Inc. 11 Mar 22, 2022
solsim is the Solana complex systems simulator. It simulates behavior of dynamical systems—DeFi protocols, DAO governance, cryptocurrencies, and more—built on the Solana blockchain

solsim is the Solana complex systems simulator. It simulates behavior of dynamical systems—DeFi protocols, DAO governance, cryptocurrencies, and more—built on the Solana blockchain

William Wolf 11 May 26, 2022
Project aims to map out common user behavior on the computer

User-Behavior-Mapping-Tool Project aims to map out common user behavior on the computer. Most of the code is based on the research by kacos2000 found

trustedsec 123 Jun 26, 2022
Fabric mod where anyone can PR anything, concerning or not. I'll merge everything as soon as it works.

Guess What Will Happen In This Fabric mod where anyone can PR anything, concerning or not (Unless it's too concerning). I'll merge everything as soon

anatom 60 Feb 16, 2022
Always fill your package requirements without the user having to do anything! Simple and easy!

WSL Should now work always-fill-reqs-python3 Always fill your package requirements without the user having to do anything! Simple and easy! Supported

Hashm 7 Jan 19, 2022
Would upload anything I do with/related to brainfuck

My Brainfu*k Repo Basically wanted to create something with Brainfu*k but realized that with the smol brain I have, I need to see the cell values real

Rafeed 1 Mar 22, 2022
YunoHost is an operating system aiming to simplify as much as possible the administration of a server.

YunoHost is an operating system aiming to simplify as much as possible the administration of a server. This repository corresponds to the core code, written mostly in Python and Bash.

YunoHost 1.3k Jun 22, 2022
fetchmesh is a tool to simplify working with Atlas anchoring mesh measurements

A Python library for working with the RIPE Atlas anchoring mesh. fetchmesh is a tool to simplify working with Atlas anchoring mesh measurements. It ca

null 2 Apr 11, 2022
New multi tool im making adding features currently

Emera Multi Tool New multi tool im making adding features currently Current List of Planned Features - Linkvertise Bypasser - Discord Auto Bump - Gith

Lamp 3 Dec 3, 2021
Starscape is a Blender add-on for adding stars to the background of a scene.

Starscape Starscape is a Blender add-on for adding stars to the background of a scene. Features The add-on provides the following features: Procedural

Marco Rossini 4 Nov 13, 2021
A C-like hardware description language (HDL) adding high level synthesis(HLS)-like automatic pipelining as a language construct/compiler feature.

██████╗ ██╗██████╗ ███████╗██╗ ██╗███╗ ██╗███████╗ ██████╗ ██╔══██╗██║██╔══██╗██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝ ██████╔╝██║██████╔╝█

Julian Kemmerer 269 Jun 22, 2022