Django database backed celery periodic task scheduler with support for task dependency graph

Overview

Djag Scheduler

(Dj)ango Task D(AG) (Scheduler)

Overview

  • Djag scheduler associates scheduling information with celery tasks
  • The task schedule is persisted in the database using Django ORM
  • Djag scheduler allows for defining task dependencies
  • Djag scheduler can handle misfire events
  • The schedules can be managed through the Django admin interface

Quick Setup

  1. Install djag-scheduler from pypi

    pip install djag-scheduler
  2. Add djag_scheduler, timezone_field, django_json_widget (installed by default) to the Django INSTALLED_APPS

    INSTALLED_APPS = [
         ...,
         
         # Djag Scheduler Apps
         'timezone_field',
         'django_json_widget',
         'djag_scheduler'
    ]
  3. Setup Django CACHE. You can choose any cache backend. But it is important to configure 'djag_scheduler' cache. For example:

    CACHES = {
        'default': {
            'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
            'LOCATION': '127.0.0.1:11211',
        },
         
        # Djag Scheduler cache
        'djag_scheduler': {
            'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
            'LOCATION': '127.0.0.1:11211',
            'TIMEOUT': None
        }
    }

    Caution: Local-memory caching will not work. See Case for Django Cache

  4. It is highly recommended to configure Django Timezone since djag-scheduler relies on Django ORM.

    USE_TZ = True
  5. Optional Configurations (In Django project settings).

    ...
    
    # Djag Scheduler Configuration
    [CELERY_NAMESPACE]_TIMEZONE = 'UTC'
    DJAG_DEFAULT_INTERVAL = 60
    DJAG_SCHEDULE_CHECK_INTERVAL = 300
    DJAG_TASK_ESTIMATED_RUN_TIME = 60
    DJAG_RESILIENT_SYNC_INTERVAL = 600
  6. Run migrations

    python manage.py migrate

Now you can run the server, navigate to the Django admin interface and start creating schedules.

Note: This is the configuration for djag-scheduler, and this alone will not run the tasks. See Running Celery Services for task execution

Running Celery Services

Djag Scheduler provides a custom scheduler which can be used with celery services.

  1. Configure celery in your Django project and add celery tasks

  2. Run celery worker service from the Django project root

    celery -A [project_name] worker --loglevel=info
  3. Run celery beat service

    celery -A [project_name] beat -l info --scheduler djag_scheduler.scheduler:DjagScheduler 

Djag Scheduler will fetch the schedule from the database and starts executing the tasks. Both worker and beat services can be started in one process. Services can be daemonized as well

Optional Configurations

  • [CELERY_NAMESPACE]_TIMEZONE: Djag uses the same default timezone the celery is configured to use (Timezone can be configured per crontab). Default: UTC

  • DJAG_DEFAULT_INTERVAL: Djag scheduler's default max loop interval. Default (sec): 60

  • DJAG_SCHEDULE_CHECK_INTERVAL: The interval at which djag checks for schedule changes. In the interim schedule is considered unchanged. Default (sec): 300

  • DJAG_TASK_ESTIMATED_RUN_TIME: Hint djag-scheduler about task execution period. Default (sec): 60

  • DJAG_RESILIENT_SYNC_INTERVAL: The scheduling stats of a task are synced to the database at each task activation and deactivation. This option is there to build resiliency if the past syncs fail. Default (sec): 600

Note: Almost everyone should configure these options according to their needs instead of relying on defaults. Let's consider you have a task with Skip Misfires, and the following configuration: Estimated Run Time > Default Interval > Task's Grace Period. Djag scheduler might end up skipping the task just because you hinted at it with a higher estimate. In general, increase your grace periods with higher runtime estimates.

Dependency Resolution

Djag Scheduler at the core builds Task DAG for dependency management. But due to the addition of crontab information, djag-scheduler resolves dependencies slightly in a different way. The following conditions are to be met for the task (depender) to execute:

  • There should be a task (depender) crontab event pending execution (on-time, delayed, coalesced, ...).

  • For each dependency, there should be at least one new executed dependee event (on-time, delayed, coalesced, ...) since the depender's last event execution start time.

Sometimes we might need to hold for the depender's execution (Ex: consume whole output) to execute the next version of the dependee. Such dependencies can be defined by checking the Future Depends flag. The future dependencies differ in the first time execution. For the first time, dependence is assumed to be resolved irrespective of the dependee's status, and the subsequent versions follow the same rules as described above.

Admin Interface

Most options are straightforward. The following listed options might require some attention:

Crontabs

Periodic Tasks

  • Grace Period: Number of seconds from the actual cron value for the task to be categorized as a misfire

  • Skip Misfires: Skip all the misfire events

  • Coalesce Misfires: Run one event (latest among misfires) for all the misfires

  • Cron Base: Initial time for evaluating crontab. Time displayed will be in Django's TIME_ZONE. Cron base is set to current time by default.

  • Task's *args or **kwargs can be set under Arguments section. The rendered JSON widget comes from django-json-widget

Task Dependencies

  • Future Depends: The first version is independent, but the future versions of the depender depends on the dependee. See Dependency Resolution

User Actions

  • The idea is to provide a mechanism for creating actions, which serves as control signals for running tasks. For generic purposes, only Unclassified Action is added to the drop-down. If you are extending this project, you can add actions and create forms per action (See user_action_forms).

  • Some internal actions are created for djag's functioning.

User Action Audits

Case for Django Cache

Djag Scheduler requires the task status for proper dependency resolution. This can be communicated from the worker service either by configuring the celery result backend or by using celery signals with Django cache as a bridge between worker and beat services. For now, the Django cache solution is adopted. For the same reason, cache backends like Local-memory caching whose cache store is unique to Django instance will not work.

Initial Credits

djag-scheduler started as an extended version of django-celery-beat, providing support for task dependencies and resiliency. django-celery-beat served as a reference in writing a database-backed custom Scheduler for celery-beat service. For more information see, LICENSE

You might also like...
Mr. Queue - A distributed worker task queue in Python using Redis & gevent
Mr. Queue - A distributed worker task queue in Python using Redis & gevent

MRQ MRQ is a distributed task queue for python built on top of mongo, redis and gevent. Full documentation is available on readthedocs Why? MRQ is an

Distributed Task Queue (development branch)
Distributed Task Queue (development branch)

Version: 5.0.5 (singularity) Web: http://celeryproject.org/ Download: https://pypi.org/project/celery/ Source: https://github.com/celery/celery/ Keywo

Distributed Task Queue (development branch)
Distributed Task Queue (development branch)

Version: 5.1.0b1 (singularity) Web: https://docs.celeryproject.org/en/stable/index.html Download: https://pypi.org/project/celery/ Source: https://git

A simple app that provides django integration for RQ (Redis Queue)
A simple app that provides django integration for RQ (Redis Queue)

Django-RQ Django integration with RQ, a Redis based Python queuing library. Django-RQ is a simple app that allows you to configure your queues in djan

Full featured redis cache backend for Django.

Redis cache backend for Django This is a Jazzband project. By contributing you agree to abide by the Contributor Code of Conduct and follow the guidel

A Django app that integrates with Dramatiq.

django_dramatiq django_dramatiq is a Django app that integrates with Dramatiq. Requirements Django 1.11+ Dramatiq 0.18+ Example You can find an exampl

A fully-featured e-commerce application powered by Django
A fully-featured e-commerce application powered by Django

kobbyshop - Django Ecommerce App A fully featured e-commerce application powered by Django. Sections Project Description Features Technology Setup Scr

Minimal example utilizing fastapi and celery with RabbitMQ for task queue, Redis for celery backend and flower for monitoring the celery tasks.

FastAPI with Celery Minimal example utilizing FastAPI and Celery with RabbitMQ for task queue, Redis for Celery backend and flower for monitoring the

Beatserver, a periodic task scheduler for Django 🎡
Beatserver, a periodic task scheduler for Django 🎡

Beat Server Beatserver, a periodic task scheduler for django channels | beta software How to install Prerequirements: Follow django channels documenta

A Django app that allows you to send email asynchronously in Django. Supports HTML email, database backed templates and logging.

Django Post Office Django Post Office is a simple app to send and manage your emails in Django. Some awesome features are: Allows you to send email as

Fully Automated YouTube Channel ▢️with Added Extra Features.

Fully Automated Youtube Channel β–’β–ˆβ–€β–€β–ˆ β–ˆβ–€β–€β–ˆ β–€β–€β–ˆβ–€β–€ β–€β–€β–ˆβ–€β–€ β–ˆβ–‘β–‘β–ˆ β–ˆβ–€β–€β–„ β–ˆβ–€β–€ β–ˆβ–€β–€β–ˆ β–’β–ˆβ–€β–€β–„ β–ˆβ–‘β–‘β–ˆ β–‘β–‘β–ˆβ–‘β–‘ β–‘β–’β–ˆβ–‘β–‘ β–ˆβ–‘β–‘β–ˆ β–ˆβ–€β–€β–„ β–ˆβ–€β–€ β–ˆβ–„β–„β–€ β–’β–ˆβ–„β–„β–ˆ β–€β–€β–€β–€ β–‘β–‘β–€β–‘β–‘ β–‘β–’β–ˆβ–‘β–‘ β–‘β–€β–€β–€ β–€β–€β–€β–‘

A task scheduler with task scheduling, timing and task completion time tracking functions

A task scheduler with task scheduling, timing and task completion time tracking functions. Could be helpful for time management in daily life.

A Django email backend that uses a celery task for sending the email.

django-celery-email - A Celery-backed Django Email Backend A Django email backend that uses a Celery queue for out-of-band sending of the messages. Wa

Dependency Injector is a dependency injection framework for Python.
Dependency Injector is a dependency injection framework for Python.

What is Dependency Injector? Dependency Injector is a dependency injection framework for Python. It helps implementing the dependency injection princi

Dependency Combobulator is an Open-Source, modular and extensible framework to detect and prevent dependency confusion leakage and potential attacks.

Dependency Combobulator Dependency Combobulator is an Open-Source, modular and extensible framework to detect and prevent dependency confusion leakage

DepFine Is a tool to find the unregistered dependency based on dependency confusion valunerablility and lead to RCE
DepFine Is a tool to find the unregistered dependency based on dependency confusion valunerablility and lead to RCE

DepFine DepFine Is a tool to find the unregistered dependency based on dependency confusion valunerablility and lead to RCE Installation: You Can inst

A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, celery and redis.

Django Channels Websocket Chatbot A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, c

Python disk-backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python.

DiskCache is an Apache2 licensed disk and file backed cache library, written in pure-Python, and compatible with Django.

🏭 An easy-to-use implementation of Creation Methods for Django, backed by Faker.

Django-fakery An easy-to-use implementation of Creation Methods (aka Object Factory) for Django, backed by Faker. django_fakery will try to guess the

Releases(v2.0.1)
Owner
Mohith Reddy
B.Tech Grad from IIT Varanasi | SDE at Visa
Mohith Reddy
Queuing with django celery and rabbitmq

queuing-with-django-celery-and-rabbitmq Install Python 3.6 or above sudo apt-get install python3.6 Install RabbitMQ sudo apt-get install rabbitmq-ser

null 1 Dec 22, 2021
Redis-backed message queue implementation that can hook into a discord bot written with hikari-lightbulb.

Redis-backed FIFO message queue implementation that can hook into a discord bot written with hikari-lightbulb. This is eventually intended to be the backend communication between a bot and a web dashboard.

thomm.o 7 Dec 5, 2022
Flower is a web based tool for monitoring and administrating Celery clusters.

Real-time monitor and web admin for Celery distributed task queue

Mher Movsisyan 5.5k Jan 2, 2023
FastAPI with Celery

Minimal example utilizing fastapi and celery with RabbitMQ for task queue, Redis for celery backend and flower for monitoring the celery tasks.

Grega Vrbančič 371 Jan 1, 2023
Asynchronous tasks in Python with Celery + RabbitMQ +Β Redis

python-asynchronous-tasks Setup & Installation Create a virtual environment and install the dependencies: $ python -m venv venv $ source env/bin/activ

Valon Januzaj 40 Dec 3, 2022
Clearly see and debug your celery cluster in real time!

Clearly see and debug your celery cluster in real time! Do you use celery, and monitor your tasks with flower? You'll probably like Clearly! ?? Clearl

RogΓ©rio Sampaio de Almeida 364 Jan 2, 2023
Dagon - An Asynchronous Task Graph Execution Engine

Dagon - An Asynchronous Task Graph Execution Engine Dagon is a job execution sys

null 8 Nov 17, 2022
A multiprocessing distributed task queue for Django

A multiprocessing distributed task queue for Django Features Multiprocessing worker pool Asynchronous tasks Scheduled, cron and repeated tasks Signed

Ilan Steemers 1.7k Jan 3, 2023
A fast and reliable background task processing library for Python 3.

dramatiq A fast and reliable distributed task processing library for Python 3. Changelog: https://dramatiq.io/changelog.html Community: https://groups

Bogdan Popa 3.4k Jan 1, 2023
a little task queue for python

a lightweight alternative. huey is: a task queue (2019-04-01: version 2.0 released) written in python (2.7+, 3.4+) clean and simple API redis, sqlite,

Charles Leifer 4.3k Jan 8, 2023