RQ (Redis Queue) integration for Flask applications

Related tags

Task Queues flask-rq
Overview

Flask-RQ

https://travis-ci.org/mattupstate/flask-rq.svg?branch=master

RQ (Redis Queue) integration for Flask applications

Resources

Installation

$ pip install flask-rq

Getting started

To quickly start using rq, simply create an RQ instance:

from flask import Flask
from flask.ext.rq import RQ


app = Flask(__name__)

RQ(app)

@job decorator

Provides a way to quickly set a function as an rq job:

from flask.ext.rq import job


@job
def process(i):
    #  Long stuff to process


process.delay(3)

A specific queue name can also be passed as argument:

@job('low')
def process(i):
    #  Long stuff to process


process.delay(2)

get_queue function

Returns default queue or specific queue for name given as argument:

from flask.ext.rq import get_queue


job = get_queue().enqueue(stuff)  # Creates a job on ``default`` queue
job = get_queue('low').enqueue(stuff)  # Creates a job on ``low`` queue

get_worker function

Returns a worker for default queue or specific queues for names given as arguments:

from flask.ext.rq import get_worker


# Creates a worker that handle jobs in ``default`` queue.
get_worker().work(True)
# Creates a worker that handle jobs in both ``default`` and ``low`` queues.
get_worker('default', 'low').work(True)
# Note: These queues have to share the same connection

Configuration

By default Flask-RQ will connect to the default, locally running Redis server. One can change the connection settings for the default server like so:

app.config['RQ_DEFAULT_HOST'] = 'somewhere.com'
app.config['RQ_DEFAULT_PORT'] = 6479
app.config['RQ_DEFAULT_PASSWORD'] = 'password'
app.config['RQ_DEFAULT_DB'] = 1

Queue connection can also be set using a DSN:

app.config['RQ_LOW_URL'] = 'redis://localhost:6379/2'
Comments
  • More usuable stuff

    More usuable stuff

    • removes the circular dependency between app and RQ instance
    • helpers are now module to level member
    • adds DSN configuration for queues
    • doc reflect changes
    • tests
    • travisci integration
    opened by jeanphix 15
  • Flask context inside of delayed job.

    Flask context inside of delayed job.

    I found useful to be able to access flask configuration inside of delayed job. So I implemented additional method that allow to run delayed job inside of flask context. It is already useful for me.

    opened by xen 9
  • Access job_id?

    Access job_id?

    Hi thanks for the great lib.

    Quick question is there currently a way to access the current_job? to acquire the job.id al la: http://python-rq.org/docs/jobs/

    Thanks in advance

    opened by rosscdh 4
  • Added optional timeout parameter to the job decorator.

    Added optional timeout parameter to the job decorator.

    Added support for an optional timeout parameter to the job decorator so one may override the redis default of 180 seconds. Does not affect past implementations that use flask-rq. Added an example and entry to the readme accordingly.

    opened by samzscott 4
  • Update Travis

    Update Travis

    • Currently all PR builds are failing because rq has started supporting wheel packages which don't include the deps for python 2.6. This should fix that. Also created an issue there (nvie/rq#540)
    • Added redis requirement
    • Updated the image
    opened by joostdevries 2
  • AttributeError: 'RQ' object has no attribute 'job'

    AttributeError: 'RQ' object has no attribute 'job'

    I've been getting these errors in my code trying to get this plugin to work. Then I tried on your examples and I'm running into the same issue.

    (env)nuke@arch (~/flask-rq/example)[master*] $: python run.py 
Traceback (most recent call last):
  File "run.py", line 6, in <module>
    from app import app
  File "/home/nuke/flask-rq/example/app.py", line 21, in <module>
    @rq.job
AttributeError: 'RQ' object has no attribute 'job'

    (env)nuke@arch (~/flask-rq/example)[master*] $: python app.py
Traceback (most recent call last):
  File "app.py", line 21, in <module>
    @rq.job
AttributeError: 'RQ' object has no attribute 'job'
    opened by thatarchguy 2
  • Is this project still alive?

    Is this project still alive?

    @mattupstate friendly ping whether there this project is still active. I've seen several issues without response and the last commit is from > 1yr ago. Discussing the best implementation of RQ with flask in nvie/rq#538.

    opened by joostdevries 1
  • docs: fix simple typo, shat -> that

    docs: fix simple typo, shat -> that

    There is a small typo in docs/conf.py.

    Should read that rather than shat.

    Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

    opened by timgates42 0
  • Can't let worker to handle with the job in queue

    Can't let worker to handle with the job in queue

    Hello, Senior engineer.I got a problem that I could not handle my task when I put the put the job into the queue.

    The code is as following:

    import time
from flask import Flask
from flask_rq2 import RQ
app = Flask(__name__)
app.debug = True

app.config['RQ_OTHER_HOST'] = 'localhost'
app.config['RQ_OTHER_PORT'] = 6379
app.config['RQ_OTHER_PASSWORD'] = None
app.config['RQ_OTHER_DB'] = 0

rq = RQ(app)

@app.route('/doit2')
def doit2():
    echo = rq.get_queue().enqueue(takes_a_while, 'do it 2',
                                  name="low", connection="other")
    return 'Success2'


@app.route('/doit7')
def doit7():
    echo = rq.get_worker('low').work(True)
    print(echo)
    return 'Success7'

    Web Server got a wrong when I request the http://127.0.0.1:5000/doit7:

    ERRORS：

      File "E:\envs\lingang-newapi\lib\site-packages\flask\app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "E:\envs\lingang-newapi\lib\site-packages\flask\_compat.py", line 39, in reraise
    raise value
  File "E:\envs\lingang-newapi\lib\site-packages\flask\app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "E:\envs\lingang-newapi\lib\site-packages\flask\app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "E:\pythonProjects\DeCheng\module_exercise\flask-rq\my_example\example\app.py", line 101, in doit7
    echo = rq.get_worker().work(True)
  File "E:\envs\lingang-newapi\lib\site-packages\rq\worker.py", line 526, in work
    self._install_signal_handlers()
  File "E:\envs\lingang-newapi\lib\site-packages\rq\worker.py", line 389, in _install_signal_handlers
    signal.signal(signal.SIGINT, self.request_stop)
  File "e:\programdata\anaconda3\lib\signal.py", line 47, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread

    Just like the example code,but I couldn't run it successfully. Thank you very much.

    opened by xiangzhaocheng12 1
  • Using flask-rq with flask-sqlalchemy occasionally raise ResourceClosedError

    Using flask-rq with flask-sqlalchemy occasionally raise ResourceClosedError

    Recently, I run the flask rq-worker with the flask-sqlalchemy in order to do some model operations by async-task in the worker. While it occasionally raised the Error

    ResourceClosedError: This result object does not return rows. It has been closed automatically.

    I don't know what happen to the worker cause the issue. I could not reproduce the error and have searched for a long time on the problem. While get nothing solution on it.

    I wonder if any examples on flask-rq worked with flask-sqlalchemy? Here are the code snippets my own.

    if __name__ == '__main__':
    with app.app_context():
        QUEUE = get_queue()
        worker = get_worker()
        worker.push_exc_handler(retry_handler)
        worker.work()

    And the async task

    @job
def async_restart_task(task_id):
    creating_info = {"status": TaskStatus.Creating}  # NOTE: Reset created_at ?
    task = TaskModel.update_task(task_id, creating_info)

    task = TaskModel.get_task_by_id(task_id)
    service = get_service(task.service)

    for jobs in iter_group(10, service.gen_jobs(task)):
        JobModel.add_jobs(jobs)
    opened by jkryanchou 0
  • Note python3 support in setup.py

    Note python3 support in setup.py

    See #28.

    Right now, the caniusepython3 script fails on this project.

    $ pip install caniusepython3
$ caniusepython3 -r requirements.txt

You need 2 projects to transition to Python 3.
Of those 2 projects, 2 have no direct dependencies blocking their transition:

  flask-rq
  flask-sslify

    From the documentation

    How do you tell if a project has been ported to Python 3?

    On PyPI each project can specify various trove classifiers (typically in a project's setup.py through a classifier argument to setup()). There are various classifiers related to what version of Python a project can run on. E.g.:

    Programming Language :: Python :: 3 Programming Language :: Python :: 3.0 Programming Language :: Python :: 3.1 Programming Language :: Python :: 3.2 Programming Language :: Python :: 3.3 Programming Language :: Python :: 3.4 As long as a trove classifier for some version of Python 3 is specified then the project is considered to support Python 3 (project owners: it is preferred you at least specify Programming Language :: Python :: 3 as that is how you end up listed on the Python 3 Packages list on PyPI; you can represent Python 2 support with Programming Language :: Python).

    opened by sandlerben 0
  • Use app context when performing job

    Use app context when performing job

    Based on #26, fixes #10, related to #3

    This PR should make sure we use the Flask app's app_context when performing a job. This allows you to access stuff like flask.url_for etc.

    Changes:

    • Register the app on module global _flask_app
    • Use that variable instead of current_app so you can access the config without app context present.
    • Subclass rq.Worker to use the app context in FlaskRQWorker.perform_job
    opened by joostdevries 0
Owner
Matt Wright
Matt Wright
GitHub
A simple app that provides django integration for RQ (Redis Queue)

Django-RQ Django integration with RQ, a Redis based Python queuing library. Django-RQ is a simple app that allows you to configure your queues in djan

RQ 1.6k Dec 28, 2022
Redis-backed message queue implementation that can hook into a discord bot written with hikari-lightbulb.

Redis-backed FIFO message queue implementation that can hook into a discord bot written with hikari-lightbulb. This is eventually intended to be the backend communication between a bot and a web dashboard.

thomm.o 7 Dec 5, 2022
a little task queue for python

a lightweight alternative. huey is: a task queue (2019-04-01: version 2.0 released) written in python (2.7+, 3.4+) clean and simple API redis, sqlite,

Charles Leifer 4.3k Jan 8, 2023
Distributed Task Queue (development branch)

Version: 5.0.5 (singularity) Web: http://celeryproject.org/ Download: https://pypi.org/project/celery/ Source: https://github.com/celery/celery/ Keywo

Celery 20.7k Jan 2, 2023
A multiprocessing distributed task queue for Django

A multiprocessing distributed task queue for Django Features Multiprocessing worker pool Asynchronous tasks Scheduled, cron and repeated tasks Signed

Ilan Steemers 1.7k Jan 3, 2023
Distributed Task Queue (development branch)

Version: 5.1.0b1 (singularity) Web: https://docs.celeryproject.org/en/stable/index.html Download: https://pypi.org/project/celery/ Source: https://git

Celery 20.7k Jan 1, 2023
Accept queue automatically on League of Legends.

Accept queue automatically on League of Legends. I was inspired by the lucassmonn code accept-queue-lol-telegram, and I modify it according to my need

null 2 Sep 6, 2022
Sync Laravel queue with Python. Provides an interface for communication between Laravel and Python.

Python Laravel Queue Queue sync between Python and Laravel using Redis driver. You can process jobs dispatched from Laravel in Python. NOTE: This pack

Sinan Bekar 3 Oct 1, 2022
Full featured redis cache backend for Django.

Redis cache backend for Django This is a Jazzband project. By contributing you agree to abide by the Contributor Code of Conduct and follow the guidel

Jazzband 2.5k Jan 3, 2023
Asynchronous tasks in Python with Celery + RabbitMQ + Redis

python-asynchronous-tasks Setup & Installation Create a virtual environment and install the dependencies: $ python -m venv venv $ source env/bin/activ

Valon Januzaj 40 Dec 3, 2022
Mr. Queue - A distributed worker task queue in Python using Redis & gevent

MRQ MRQ is a distributed task queue for python built on top of mongo, redis and gevent. Full documentation is available on readthedocs Why? MRQ is an

Pricing Assistant 871 Dec 25, 2022
A django integration for huey task queue that supports multi queue management

django-huey This package is an extension of huey contrib djhuey package that allows users to manage multiple queues. Installation Using pip package ma

GAIA Software 32 Nov 26, 2022
A simple app that provides django integration for RQ (Redis Queue)

Django-RQ Django integration with RQ, a Redis based Python queuing library. Django-RQ is a simple app that allows you to configure your queues in djan

RQ 1.6k Jan 6, 2023
A simple app that provides django integration for RQ (Redis Queue)

Django-RQ Django integration with RQ, a Redis based Python queuing library. Django-RQ is a simple app that allows you to configure your queues in djan

RQ 1.6k Dec 28, 2022
Minimal example utilizing fastapi and celery with RabbitMQ for task queue, Redis for celery backend and flower for monitoring the celery tasks.

FastAPI with Celery Minimal example utilizing FastAPI and Celery with RabbitMQ for task queue, Redis for Celery backend and flower for monitoring the

Grega Vrbančič 371 Jan 1, 2023
A simple docker-compose app for orchestrating a fastapi application, a celery queue with rabbitmq(broker) and redis(backend)

fastapi - celery - rabbitmq - redis -> Docker A simple docker-compose app for orchestrating a fastapi application, a celery queue with rabbitmq(broker

Kartheekasasanka Kaipa 83 Dec 19, 2022
Aiorq is a distributed task queue with asyncio and redis

Aiorq is a distributed task queue with asyncio and redis, which rewrite from arq to make improvement and include web interface.

PY-GZKY 5 Mar 18, 2022
Redis-backed message queue implementation that can hook into a discord bot written with hikari-lightbulb.

Redis-backed FIFO message queue implementation that can hook into a discord bot written with hikari-lightbulb. This is eventually intended to be the backend communication between a bot and a web dashboard.

thomm.o 7 Dec 5, 2022
Interactive Redis: A Terminal Client for Redis with AutoCompletion and Syntax Highlighting.

Interactive Redis: A Cli for Redis with AutoCompletion and Syntax Highlighting. IRedis is a terminal client for redis with auto-completion and syntax

null 2.2k Dec 29, 2022
Python cluster client for the official redis cluster. Redis 3.0+.

redis-py-cluster This client provides a client for redis cluster that was added in redis 3.0. This project is a port of redis-rb-cluster by antirez, w

Grokzen 1.1k Jan 5, 2023