A flexible python library for building your own cron-like system, with REST APIs and a Web UI.

Overview

Nextdoor Scheduler

Apache Build Status

ndscheduler is a flexible python library for building your own cron-like system to schedule jobs, which is to run a tornado process to serve REST APIs and a web ui.

Check out our blog post - We Don't Run Cron Jobs at Nextdoor

ndscheduler currently supports Python 2 & 3 on Mac OS X / Linux.

Table of contents

Key Abstractions

  • CoreScheduler: encapsulates all core scheduling functionality, and consists of:
    • Datastore: manages database connections and makes queries; could support Postgres, MySQL, and sqlite.
      • Job: represents a schedule job and decides how to run a paricular job.
      • Execution: represents an instance of job execution.
      • AuditLog: logs when and who runs what job.
    • ScheduleManager: access Datastore to manage jobs, i.e., schedule/modify/delete/pause/resume a job.
  • Server: a tornado server that runs ScheduleManager and provides REST APIs and serves UI.
  • Web UI: a single page HTML app; this is a default implementation.

Note: corescheduler can also be used independently within your own service if you use a different Tornado server / Web UI.

Try it NOW

From source code:

git clone https://github.com/Nextdoor/ndscheduler.git
cd ndscheduler
make simple

Or use docker:

docker run -it -p 8888:8888 wenbinf/ndscheduler

Open your browser and go to localhost:8888.

Demo (Click for fullscreen play) ndscheduler demo

How to build Your own cron-replacement

Install ndscheduler

Using pip (from GitHub repo)

#
# Put this in requirements.txt, then run
#    pip install -r requirements.txt
#

# If you want the latest build
git+https://github.com/Nextdoor/ndscheduler.git#egg=ndscheduler

# Or put this if you want a specific commit
git+https://github.com/Nextdoor/ndscheduler.git@5843322ebb440d324ca5a66ba55fea1fd00dabe8

# Or put this if you want a specific tag version
git+https://github.com/Nextdoor/[email protected]#egg=ndscheduler

#
# Run from command line
#

pip install -e git+https://github.com/Nextdoor/ndscheduler.git#egg=ndscheduler

(We'll upload the package to PyPI soon.)

Three things

You have to implement three things for your scheduler, i.e., Settings, Server, and Jobs.

Settings

In your implementation, you need to provide a settings file to override default settings (e.g., settings in simple_scheduler). You need to specify the python import path in the environment variable NDSCHEDULER_SETTINGS_MODULE before running the server.

All available settings can be found in default_settings.py file.

Server

You need to have a server file to import and run ndscheduler.server.server.SchedulerServer.

Jobs

Each job should be a standalone class that is a subclass of ndscheduler.job.JobBase and put the main logic of the job in run() function.

After you set up Settings, Server and Jobs, you can run the whole thing like this:

NDSCHEDULER_SETTINGS_MODULE=simple_scheduler.settings \
PYTHONPATH=.:$(PYTHONPATH) \
	    python simple_scheduler/scheduler.py

Upgrading

It is best practice to backup your database before doing any upgrade. ndscheduler relies on apscheduler to serialize jobs to the database, and while it is usually backwards-compatible (i.e. jobs created with an older version of apscheduler will continue to work after upgrading apscheduler) this is not guaranteed, and it is known that downgrading apscheduler can cause issues. See this PR comment for more details.

Reference Implementation

See code in the simple_scheduler/ directory for inspiration :)

Run it

make simple

Access the web ui via localhost:8888

The reference implementation also comes with several sample jobs.

  • AwesomeJob: it just prints out 2 arguments you pass in.
  • SlackJob: it sends a slack message periodically, for example, team standup reminder.
  • ShellJob: it runs an executable command, for example, run curl to crawl web pages.
  • CurlJob: it's like running curl periodically.

And it's dockerized.

Contribute code to ndscheduler

Install dependencies

# Each time we introduce a new dependency in setup.py, you have to run this
make install

Run unit tests

make test

Clean everything and start from scratch

make clean

Finally, send pull request. Please make sure the CI passes for your PR.

REST APIs

Please see README.md in ndscheduler/server/handlers.

Web UI

We provide a default implementation of web ui. You can replace the default web ui by overwriting these settings

STATIC_DIR_PATH = :static asset directory paths:
TEMPLATE_DIR_PATH = :template directory path:
APP_INDEX_PAGE = :the file name of the single page app's html:

The default web ui

List of jobs

List of jobs

List of executions

List of executions

Audit Logs

Audit logs

Modify a job

Modify a job

Comments
  • Wrong time

    Wrong time

    When I create a new job in ndscheduler with 10 mins, its not scheduled to run on 10 mins.

    Ex:

    1. Consider current system time is 19:21 IST
    2. Create a new job with 30 mins interval
    3. The next time to run should be 19:51 IST. But ndscheduler shows 20:00 IST

    Ex2:

    1. Consider current system time is 19:32 IST
    2. create a new job with 9 mins interval
    3. The next time to run should be 19:41 IST. But ndscheduler shows 19:39 IST

    My parameters are, Mins hour (UTC) day month day of week */30 * * * *

    Please help me to run the scheduler at the desired time. Thanks in advance Bala

    opened by balaCode63 6
  • Status of this project

    Status of this project

    Hello,

    I see that no commits have been made recently and am wondering about the status of this project. This looks like the tool I would need to use but I'm wondering if that's smart seeing the activity. Is this not in used anymore at nextdoor ? What are the alternatives if any ? Thanks !

    opened by EmFl 4
  • Installation fails

    Installation fails

    Collecting ndscheduler from git+https://github.com/Nextdoor/ndscheduler.git#egg=ndscheduler Cloning https://github.com/Nextdoor/ndscheduler.git to /tmp/pip-build-mm359amx/ndscheduler Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/tmp/pip-build-mm359amx/ndscheduler/setup.py", line 65, in long_description=open('README.md').read(), File "/home/thomas/production/env/lib/python3.5/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 495: ordinal not in range(128)

    ----------------------------------------
    

    Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-mm359amx/ndscheduler/

    opened by tschm 4
  • Not able to override default settings

    Not able to override default settings

    We're trying to alter the setting JOB_MAX_INSTANCES, but not having any luck.

    Dockerfile

    FROM python:2-alpine3.7
    
    RUN apk update && apk add git postgresql-dev gcc python-dev musl-dev
    
    COPY requirements.txt requirements.txt
    RUN pip install -r requirements.txt
    
    ENV NDSCHEDULER_SETTINGS_MODULE=settings
    
    ADD . .
    
    CMD ["python", "scheduler.py"]
    

    settings.py

    JOB_MAX_INSTANCES = 10
    

    scheduler.py

    """Run the scheduler process."""
    from ndscheduler.server import server
    
    class Tassadar(server.SchedulerServer):
      pass
    
    if __name__ == "__main__":
        Tassadar.run()
    

    And we boot up the container and try to run more than 3 instances:

    ndscheduler.core.scheduler_manager - WARNING - Execution of job "REDACTED (trigger: cron[month='*', day='*', day_of_week='*', hour='*', minute='0'], next run at: 2018-07-13 04:00:00 UTC)" skipped: maximum number of running instances reached (3)

    Any ideas? I would expect it to not limit out at 3 any more, but rather 10.

    Thanks!

    opened by noqcks 3
  • Info; ndscheduler make simple fail

    Info; ndscheduler make simple fail

    From external [redacted]:

    make simple times out:

      Running command git clone --filter=blob:none --quiet git://github.com/djacobs/PyAPNs.git /private/var/folders/t_/rc213wmx66g3r6zwqfvrh3cc0000gn/T/pip-install-o58t0_l9/pyapns_3e89f11475f9455fa0fea75da18a3993
      fatal: unable to connect to [github.com](http://github.com/):
      [github.com](http://github.com/)[0: 140.82.113.4]: errno=Operation timed out
    

    I am able to clone from the command line with:

    git clone --filter=blob:none --quiet https://github.com/djacobs/PyAPNs.git
    

    But, changing requirements.txt

    < git+git://github.com/djacobs/PyAPNs.git#egg=PyAPNs
    ---
    > git+https://github.com/djacobs/PyAPNs.git
    

    Leads to issue with ‘tornado not installed’.

    Any help.?

    opened by LaikaN57 2
  • Is it possible to backup ndscheduler jobs?

    Is it possible to backup ndscheduler jobs?

    I have a over 50 jobs running in ndscheduler running and want to run them on another instance with a new hard drive.

    How do I retain my job schedule? Is it possible?

    opened by noqcks 2
  • [Question] Scalability

    [Question] Scalability

    I am interested in using ndscheduler in a solution that I am developing but I have the following requirement:

    1. My app will use the ndscheduler API to generate about 30,000 schedules per day within 2 hours time frame.

    2. These schedules will run only once and their purpose will be to send an attendance confirmation email to a customer.

    3. Schedules will have their execution start date and time distributed over a time period of 6 hours during the day.

    4. After the schedule is successfully executed it can be deleted from the ndscheduler.

    Does the ndscheduler support this workload?

    Thank you.

    opened by abnerjacobsen 2
  • Error report

    Error report

    I test error on running batch/shell file. but job execution status is "Success" and Result is: { "returncode": 1 } Can I print the error on shell script to result trough a Job argument?

    And what best practice argument for job shell script?

    opened by ibnbay00 2
  • Not compatible with Python 3.5

    Not compatible with Python 3.5

    Traceback (most recent call last): File "/usr/local/lib/python3.5/site-packages/tornado/web.py", line 1443, in _execute result = method(*self.path_args, **self.path_kwargs) File "/src/ndscheduler/ndscheduler/server/handlers/index.py", line 16, in get meta_info = utils.get_all_available_jobs() File "/src/ndscheduler/ndscheduler/utils.py", line 155, in get_all_available_jobs job_module = importlib.import_module('%s.%s' % (job_class_package, module_name)) File "/usr/local/lib/python3.5/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 986, in _gcd_import File "", line 969, in _find_and_load File "", line 958, in _find_and_load_unlocked File "", line 673, in _load_unlocked File "", line 673, in exec_module File "", line 222, in _call_with_frames_removed File "/src/ndscheduler/simple_scheduler/jobs/apns_job.py", line 7, in from apns import APNs, Payload File "/usr/local/lib/python3.5/site-packages/apns.py", line 218 except ssl.SSLError, err: ^ SyntaxError: invalid syntax 2017-05-04 21:05:03,639 - tornado.access - ERROR - 500 GET / (172.17.0.1) 21.06ms

    opened by rhaarm 2
  • Make script fails on Windows

    Make script fails on Windows

    Running make simple in a Windows environment, using Cygwinyields an error when it attempts to execute the .venv/bin/activate command in the init target.

    . .venv/bin/activate && .venv/bin/pip install flake8
    /bin/bash: .venv/bin/activate: No such file or directory
    Makefile:10: recipe for target 'init' failed
    make[2]: *** [init] Error 1`
    make[2]: Leaving directory '/cygdrive/c/Nextdoor/ndscheduler'
    Makefile:24: recipe for target 'install' failed
    make[1]: *** [install] Error 2
    make[1]: Leaving directory '/cygdrive/c/Nextdoor/ndscheduler'
    Makefile:36: recipe for target 'simple' failed
    

    The initial difficulty is that in a Windows environment, the virtualenv command doesn't create a .venv/bin directory. Instead, the executables are placed in: .venv/Scripts. (As of Python 2.7.2.)

    The PYTHON, PIP and SOURCE_VENV variables must be changed to reflect the .venv/Scripts location. e.g.

    PYTHON=.venv/Scripts/python
    PIP=.venv/Scripts/pip
    SOURCE_VENV=.venv/Scripts/activate
    

    This change is not sufficient to resolve the problem. The virtualenv command installs activate as a .bat file instead of a .exe. The bash shell doesn't know to look for .bat files, and the command fails. Changing the SOURCE_VENV variable to reference activate.bat (with the extension) allows activate to be found, however bash doesn't know how to interpret the file, and the script still fails.

    opened by thatblairguy 2
  • Bulk update

    Bulk update

    Many changes added form fork:

    • Option for YMAL config
    • tornado >=6 support
    • authenticatio feature
    • basic user and password management
    • https support
    • several small improvements
    • email notification on failed jobs
    • handler for SIGTREM
    • clean-up interrupted executions
    • formatting compliant with black
    • example for systemd service
    opened by palto42 1
  • make simple fails possibly due to problem with virtualenv version

    make simple fails possibly due to problem with virtualenv version

    I am having problems trying to build according to the readme. Here is my log when trying make simple

    sudo make simple

    if [ ! -d ".venv" ]; then make install; fi

    make[1]: Entering directory '/home/ubuntu/new/ndscheduler'

    make init

    make[2]: Entering directory '/home/ubuntu/new/ndscheduler'

    Initialize dev environment for ndscheduler ...

    Install pre-commit hook for git.

    Setup python virtual environment.

    if [ ! -d ".venv" ]; then virtualenv .venv; fi

    Using base prefix '/usr'

    New python executable in /home/ubuntu/new/ndscheduler/.venv/bin/python3

    Also creating executable in /home/ubuntu/new/ndscheduler/.venv/bin/python

    Traceback (most recent call last): File "/usr/local/bin/virtualenv", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.10/dist-packages/virtualenv.py", line 860, in main create_environment( File "/usr/local/lib/python3.10/dist-packages/virtualenv.py", line 1162, in create_environment install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages=site_packages, clear=clear, symlink=symlink) File "/usr/local/lib/python3.10/dist-packages/virtualenv.py", line 1721, in install_python fix_local_scheme(home_dir, symlink) File "/usr/local/lib/python3.10/dist-packages/virtualenv.py", line 1807, in fix_local_scheme if sysconfig._get_default_scheme() == "posix_local":

    AttributeError: module 'sysconfig' has no attribute '_get_default_scheme'. Did you mean: 'get_default_scheme'?

    make[2]: *** [Makefile:14: init] Error 1

    make[2]: Leaving directory '/home/ubuntu/new/ndscheduler'

    make[1]: *** [Makefile:25: install] Error 2

    make[1]: Leaving directory '/home/ubuntu/new/ndscheduler'

    make: *** [Makefile:37: simple] Error 2

    ubuntu@ip-172-31-4-204://home/ubuntu/new/ndscheduler$

    If I run the same command again, I get a different output now:

    sudo make simple if [ ! -d ".venv" ]; then make install; fi # Install dependencies .venv/bin/pip install -r simple_scheduler/requirements.txt; /bin/bash: line 1: .venv/bin/pip: No such file or directory make: *** [Makefile:39: simple] Error 127

    This stackoverflow response seems to indicate that it is a problem with the virtualenv version.

    from stackoverflow:

    Using python 3.10.2 and virtualenv 16.7.5 gives me the same error. Looks like virtualenv 16.7.5 is too old for 3.10.2. Upgrade you package with this command and everything will work out.

    pip install --upgrade virtualenv

    ...However I do not have the luxury of running this update command because this is all running inside make. I suppose I could downgrade my python version, but that seems less than ideal. Any help here is greatly appreciated.

    opened by findbl0k 1
  • Conflict with Notebook package in the same environment

    Conflict with Notebook package in the same environment

    Hello,

    When I try to install latest notebook package in the same environment as this package, there is a conflict on tornado. ndscheduler is pinned at tornado<6 but latest notebook package requires a higher version of it. Can you please help with this issue?

    Regards, Akshita

    opened by Akshita07 0
  • Unable to run manual settings rather than default settings

    Unable to run manual settings rather than default settings

    Hi, can someone tell me how to pass the ENV variable "NDSCHEDULER_SETTINGS_MODULE" or how I can run manual settings rather than the default settings?

    opened by pratikmore96 1
  • Confusing licence badge on README.md

    Confusing licence badge on README.md

    The README.md shows an Apache-2.0 license badge, but on the LICENSE.txt page it states 'BSD 2-Clause "Simplified" License'.

    Can you please clarify which license is correct and update either the README.md or the LICENSE.txt file?

    opened by palto42 1
  • Not able to use MySQL

    Not able to use MySQL

    When I override DB setting, I got error. See below

    settings.py DATABASE_CLASS = 'ndscheduler.core.datastore.providers.mysql.DatastoreMySQL' DATABASE_CONFIG_DICT = { 'user': 'username', 'password': '', 'hostname': 'localhost', 'port': 3306, 'database': 'scheduler' }

    Error app_1 | Traceback (most recent call last): app_1 | File "scheduler.py", line 46, in <module> app_1 | SampleJobServer.run() app_1 | File "/usr/local/lib/python3.6/site-packages/ndscheduler/server/server.py", line 83, in run app_1 | cls.singleton = cls(scheduler_manager.SchedulerManager.get_instance()) app_1 | File "/usr/local/lib/python3.6/site-packages/ndscheduler/core/scheduler_manager.py", line 22, in get_instance app_1 | cls.instance = cls() app_1 | File "/usr/local/lib/python3.6/site-packages/ndscheduler/core/scheduler_manager.py", line 27, in __init__ app_1 | 'default': utils.get_datastore_instance() app_1 | File "/usr/local/lib/python3.6/site-packages/ndscheduler/utils.py", line 106, in get_datastore_instance app_1 | database_class = import_from_path(settings.DATABASE_CLASS) app_1 | File "/usr/local/lib/python3.6/site-packages/ndscheduler/utils.py", line 31, in import_from_path app_1 | return getattr(module, components[-1]) app_1 | AttributeError: module 'ndscheduler.core.datastore.providers.mysql' has no attribute 'DatastoreMySQL'

    opened by SatoshiMD 1
Releases(v0.3.0)
Owner
Nextdoor is the free private social network for your neighborhood community.
null
The easiest way to automate your data

Hello, world! ?? We've rebuilt data engineering for the data science era. Prefect is a new workflow management system, designed for modern infrastruct

Prefect 10.9k Jan 4, 2023
generate HPC scheduler systems jobs input scripts and submit these scripts to HPC systems and poke until they finish

DPDispatcher DPDispatcher is a python package used to generate HPC(High Performance Computing) scheduler systems (Slurm/PBS/LSF/dpcloudserver) jobs in

DeepModeling 23 Nov 30, 2022
Crontab jobs management in Python

Plan Plan is a Python package for writing and deploying cron jobs. Plan will convert Python code to cron syntax. You can easily manage you

Shipeng Feng 1.2k Dec 28, 2022
Python job scheduling for humans.

schedule Python job scheduling for humans. Run Python functions (or any other callable) periodically using a friendly syntax. A simple to use API for

Dan Bader 10.4k Jan 2, 2023
A powerful workflow engine implemented in pure Python

Spiff Workflow Summary Spiff Workflow is a workflow engine implemented in pure Python. It is based on the excellent work of the Workflow Patterns init

Samuel 1.3k Jan 8, 2023
Python-Repeated-Timer is an open-source & highly performing timer using only standard-libraries.

Python Repeated Timer Python-Repeated-Timer is an open-source & highly performing timer using only standard-libraries.

TACKHYUN JUNG 3 Oct 9, 2022
Automate SQL Jobs Monitoring with python

Automate_SQLJobsMonitoring_python Using python 3rd party modules we can automate

Aejaz Ayaz 1 Dec 27, 2021
A simple scheduler tool that provides desktop notifications about classes and opens their meet links in the browser automatically at the start of the class.

This application provides desktop notifications about classes and opens their meet links in browser automatically at the start of the class.

Anshit 14 Jun 29, 2022
Here is the live demonstration of endpoints and celery worker along with RabbitMQ

whelp-task Here is the live demonstration of endpoints and celery worker along with RabbitMQ Before running the application make sure that you have yo

Yalchin403 0 Nov 14, 2021
Aiorq is a distributed task queue with asyncio and redis

Aiorq is a distributed task queue with asyncio and redis, which rewrite from arq to make improvement and include web interface.

PY-GZKY 5 Mar 18, 2022
A task scheduler with task scheduling, timing and task completion time tracking functions

A task scheduler with task scheduling, timing and task completion time tracking functions. Could be helpful for time management in daily life.

ArthurLCW 0 Jan 15, 2022
Fully Automated YouTube Channel ▶️with Added Extra Features.

Fully Automated Youtube Channel ▒█▀▀█ █▀▀█ ▀▀█▀▀ ▀▀█▀▀ █░░█ █▀▀▄ █▀▀ █▀▀█ ▒█▀▀▄ █░░█ ░░█░░ ░▒█░░ █░░█ █▀▀▄ █▀▀ █▄▄▀ ▒█▄▄█ ▀▀▀▀ ░░▀░░ ░▒█░░ ░▀▀▀ ▀▀▀░

sam-sepiol 249 Jan 2, 2023
Croniter provides iteration for the datetime object with a cron like format

Introduction Contents Introduction Travis badge Usage About DST About second repeats Testing if a date matches a crontab Gaps between date matches Ite

kiorky 152 Dec 30, 2022
Industry ready custom API payload with an easy format for building Python APIs (Django/Django Rest Framework)

Industry ready custom API payload with an easy format for building Python APIs (Django/Django Rest Framework) Yosh! If you are a django backend develo

Abram (^o^) 7 Sep 30, 2022
This is a passport scanning web service to help you scan, identify and validate your passport created with a simple and flexible design and ready to be integrated right into your system!

Passport-Recogniton-System This is a passport scanning web service to help you scan, identify and validate your passport created with a simple and fle

Mo'men Ashraf Muhamed 7 Jan 4, 2023
Sanic-RESTPlus is an extension for Sanic that adds support for quickly building REST APIs.

Sanic RestPlus Sanic-RESTPlus is an extension for Sanic that adds support for quickly building REST APIs. Sanic-RESTPlus encourages best practices wit

Ashley Sommer 106 Oct 14, 2022
RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.

Microsoft 1.8k Jan 4, 2023
Atualizando o projeto APIs REST Django REST 2.0

APIs REST Django REST 3.0-KevinSoffa Atualização do projeto APIs REST Django REST 2.0-Kevin Soffa Melhorando e adicionando funcionalidades O que já fo

Kevin Soffa 2 Dec 13, 2022
A command-line tool and Python library and Pytest plugin for automated testing of RESTful APIs, with a simple, concise and flexible YAML-based syntax

1.0 Release See here for details about breaking changes with the upcoming 1.0 release: https://github.com/taverntesting/tavern/issues/495 Easier API t

null 909 Dec 15, 2022
scaffold django rest apis like a champion 🚀

dr_scaffold Scaffold django rest apis like a champion ⚡ . said no one before Overview This library will help you to scaffold full Restful API Resource

Abdenasser Elidrissi 133 Jan 5, 2023