Rich Python data types for Redis

Related tags

hot-redis
Overview
https://secure.travis-ci.org/stephenmcd/hot-redis.png?branch=master

Created by Stephen McDonald

Introduction

HOT Redis is a wrapper library for the redis-py client. Rather than calling the Redis commands directly from a client library, HOT Redis provides a wide range of data types that mimic many of the built-in data types provided by Python, such as lists, dicts, sets, and more, as well as many of the classes found throughout the standard library, such as those found in the Queue, threading, and collections modules.

These types are then backed by Redis, allowing objects to be manipulated atomically over the network - the atomic nature of the methods implemented on objects in HOT Redis is one of its core features, and many of these are backed by Lua code executed within Redis, which ensures atomic operations where applicable.

The name HOT Redis originally stood for "Higher Order Types for Redis", but since the implementation doesn't strictly fit the definition, the recursive acronym "HOT Object Toolkit for Redis" should appease the most luscious of bearded necks.

HOT Redis was drawn from the infrastructure behind the Kouio RSS reader, a popular alternative to Google Reader.

Installation

The easiest way to install hot-redis is directly from PyPi using pip by running the following command:

$ pip install -U hot-redis

Otherwise you can download and install it directly from source:

$ python setup.py install

Usage

Each of the types provided by HOT Redis strive to implement the same method signatures and return values as their Python built-in and standard library counterparts. The main difference is each type's __init__ method. Every HOT Redis type's __init__ method will optionally accept initial and key keyword arguments, which are used for defining an initial value to be stored in Redis for the object, and the key that should be used, respectively. If no key is provided, a key will be generated, which can then be accessed via the key attribute:

>>> from hot_redis import List
>>> my_list = List()
>>> my_list.key
'93366bdb-90b2-4226-a52a-556f678af40e'
>>> my_list_with_key = List(key="foo")
>>> my_list_with_key.key
'foo'

Once you've determined a strategy for naming keys, you can then create HOT Redis objects and interact with them over the network, for example here is a List created on a computer we'll refer to as computer A:

>>> list_on_computer_a = List(key="foo", initial=["a", "b", "c"])

then on another computer we'll creatively refer to as computer B:

>>> list_on_computer_b = List(key="foo")
>>> list_on_computer_b[:]  # Performs: LRANGE foo 0 -1
['a', 'b', 'c']
>>> list_on_computer_b += ['d', 'e', 'f']  # Performs: RPUSH foo d e f

and back to computer A:

>>> list_on_computer_a[:]  # Performs: LRANGE foo 0 -1
['a', 'b', 'c', 'd', 'e', 'f']
>>> 'c' in list_on_computer_a  # Works like Python lists where expected
True
>>> list_on_computer_a.reverse()
>>> list_on_computer_a[:]
['f', 'e', 'd', 'c', 'b', 'a']

The last interaction here is an interesting one. Python's list.reverse() is an in-place reversal of the list, that is, it modifies the existing list, rather than returning a reversed copy. If we were to implement this naively, we would first read the list from Redis, reverse it locally, then store the reversed list back in Redis again. But what if another client were to modify the list at approximately the same time? One computer's modification to the list would certainly overwrite the other's. In this scenario, and many others, HOT Redis provides its own Lua routine specifically for reversing the list in-place, within Redis atomically. I wrote in more detail about this in a blog post, Bitwise Lua Operations in Redis.

Configuration

By default, HOT Redis attempts to connect to a Redis instance running locally on the default port 6379. You can configure the default client by calling the hot_redis.configure function, prior to instantiating any HOT Redis objects. The arguments given to configure are passed onto the underlying redis-py client:

>>> from hot_redis import configure
configure(host='myremotehost', port=6380)

Alternatively, if you wish to use a different client per object, you can explicitly create a HotClient instance, and pass it to each object:

>>> from hot_redis import HotClient, Queue
>>> client = HotClient(host="myremotehost", port=6380)
>>> my_queue = Queue(client=client)

Transactions

Basic support for thread-safe transactions are provided using the Redis MULTI and EXEC commands:

>>> from hot_redis import List, Queue, transaction
>>> my_list = List(key="foo")
>>> my_queue = Queue(key="bar")
>>> with transaction():
...     for i in range(20):
...         my_list.append(i)
...         my_queue.put(i)

In the above example, all of the append and put calls are batched together into a single transaction, that is executed once the transaction() context is exited.

Data Types

The following table is the complete list of types provided by HOT Redis, mapped to their Python counterparts and underlying Redis types, along with any special considerations worth noting.

HOT Redis Python Redis Notes
List list list  
Set set set  
Dict dict hash  
String string string Mutable - string methods that normally create a new string object in Python will mutate the string stored in Redis
ImmutableString string string Immutable - behaves like a regular Python string
Int int int  
Float float float  
Queue Queue.Queue list  
LifoQueue Queue.LifoQueue list  
SetQueue N/A list + set Extension of Queue with unique members
LifoSetQueue N/A list + set Extension of LifoQueue with unique members
BoundedSemaphore threading.BoundedSemaphore list Extension of Queue leveraging Redis' blocking list pop operations with timeouts, while using Queue's maxsize arg to provide BoundedSemaphore's value arg
Semaphore threading.Semaphore list Extension of BoundedSemaphore without a queue size
Lock threading.Lock list Extension of BoundedSemaphore with a queue size of 1
RLock threading.RLock list Extension of Lock allowing multiple acquire calls
DefaultDict collections.DefaultDict hash  
MultiSet collections.Counter hash  
Issues
  • Working with arbitrary clients

    Working with arbitrary clients

    HotClient and Base now supports taking an arbitrary client, for example from a separate connection pool or a legacy system. Removed transactions since they will not work properly in current form.

    opened by jmizgajski 8
  • Added support for using remote Redis instances

    Added support for using remote Redis instances

    The previous version worked fine with a local Redis instance running on the default port/db, but there was no obvious way to specify a remote Redis instance of use non-default parameters. This patch is fully backwards-compatible, but allows users to pass a HotClient instance (with any Redis connection parameters) to HOT Redis types using the "client" keyword. A .gitignore file has been added to mirror the .hgignore. Documentation has also been updated for the patch.

    opened by brendanwood 6
  • Add transaction (or multi) support

    Add transaction (or multi) support

    It would be very useful to have transaction support in form of

    with hot_redis.transations.Transaction() as t:
        #do some stuff
    
    

    not all operations have to be supported at first.

    This would greatly improve interoperability with other transactional dbs, which is a common usecase.

    Kudos for a great library btw:)

    opened by jmizgajski 6
  • Multikey lua support, rank_lists_by_length, rank_sets_by_cardinality

    Multikey lua support, rank_lists_by_length, rank_sets_by_cardinality

    Suport for complex (multikey) lua scripts and ranking scripts for ranking lists according to lenght and sets according to cardinality. Minor refactor of script handling by HotClient

    opened by jmizgajski 6
  • feature/collections_using_redis_sorted_set - MultiSet on Redis SortedSet

    feature/collections_using_redis_sorted_set - MultiSet on Redis SortedSet

    Adds implementation of Counter on Redis SortedSets with better performance on 'most_common'. Also updated documentation. I also would like to know what others feel about inheriting from collections.MutableMapping abstract classes, as i think it might be a bit clearer in terms what should objects provide than simply writing methods. Lua implemention of in place operations will be added soon

    opened by glowskir 5
  • Integers in Dicts are not preserved

    Integers in Dicts are not preserved

    Adding the following test case, the tests fail.

    class DictTests(BaseTestCase):
    
        def test_integer(self):
            a = {"wagwaan": 5, "flute": "don"}
            self.assertEqual(hot_redis.Dict(a), a)
    

    with:

    python tests.py
    ...................F.....................................................................................
    ======================================================================
    FAIL: test_integer (__main__.DictTests)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "tests.py", line 397, in test_integer
        self.assertEqual(hot_redis.Dict(a), a)
    AssertionError: Dict({u'wagwaan': u'5', u'flute': u'don'}, '5f74dc31-1bbb-4304-8ce8-a80a6337514a') != {'wagwaan': 5, 'flute': 'don'}
    
    ----------------------------------------------------------------------
    Ran 105 tests in 4.337s
    
    FAILED (failures=1)
    

    Any idea why the 'wagwaan' int is transformed to a string?

    opened by stratosgear 2
  • List type is broken

    List type is broken

    If it's important, the below behavior was seen under Python 3.4.

    It's not possible to overwrite your List object once it's created. In normal Python list type, if you set a list equal to a new list then it will just overwrite the existing list. However, in your List type if you try to overwrite the value it will call extend every time.

    I even tried closing my application and re-starting. Since the object in Redis is actually still storing the old value, the problem persists even through application restart. This is especially fun when you use the "initial" parameter on startup. You basically end up with a list that gets extended over and over again every time the application restarts.

    Also, you should respect the types of objects passed inside the list itself. In my example below, I'm passing INT types inside my list; but your library gives the list items back as STRING types. In normal python lists, if the list contains INT you get back INT.

    PYTHON EXAMPLE:

    >>> pythonList = [1,2,3]
    >>> print(pythonList)
    [1, 2, 3]
    >>> pythonList = [4,5,6]
    >>> print(pythonList)
    [4, 5, 6]
    

    HOT_REDIS EXAMPLE:

    >>> from hot_redis import List
    >>> theList = List(key='abc', initial=[1,2,3])
    >>> print(theList)
    List(['1', '2', '3'], 'abc')
    >>> print(theList.value)
    ['1', '2', '3']
    >>> theList = List(key='abc', initial=[4,5,6])
    >>> print(theList)
    List(['1', '2', '3', '4', '5', '6'], 'abc')
    >>> print(theList.value)
    ['1', '2', '3', '4', '5', '6']
    >>> theList.value = [1,2,3]
    >>> print(theList)
    List(['1', '2', '3', '4', '5', '6', '1', '2', '3'], 'abc')
    >>> print(theList.value)
    ['1', '2', '3', '4', '5', '6', '1', '2', '3']
    
    opened by CompPhy 2
  • Fix BoundedSemaphore.release method and Add Tests for Lock, BoundedSemaphore, Semaphore.

    Fix BoundedSemaphore.release method and Add Tests for Lock, BoundedSemaphore, Semaphore.

    Hi! BoundedSemaphore.release method always fails when is it called more than once. Here's an example.

    In [1]: import hot_redis
    In [2]: sem = hot_redis.BoundedSemaphore(value=2)
    In [3]: sem.acquire()
    Out[3]: True
    In [4]: sem.acquire()
    Out[4]: True
    In [5]: sem.release()
    In [6]: sem.release() # the second calling release method failed.
    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-6-33a6461ef954> in <module>()
    ----> 1 sem.release()
    
    /Users/bluele/.virtualenvs/hot_redis_master/lib/python2.7/site-packages/hot_redis/types.pyc in release(self)
        704     def release(self):
        705         if not getattr(self, "acquired", False):
    --> 706             raise RuntimeError("Cannot release unacquired lock")
        707         self.acquired = False
        708       
    

    This pull-request fixed the above problem and added test code for Lock, BoundedSemaphore, Semaphore.

    opened by bluele 1
  • Python 3 has no module

    Python 3 has no module "Queue"

    It switched the naming convention to be lowercase, eg. "queue", so installing or using hot-redis causes an import error in Python 3

    opened by dralley 1
  • After modifying data, the data is not updated in a timely manner!

    After modifying data, the data is not updated in a timely manner!

    image Hello, I met such a problem, I want to the inside of the django data import to redis, my code is as follows

    image

    opened by 8hours 1
  • Is this project dead?

    Is this project dead?

    I was looking for something like it and was excited when I found this project. But then I noticed that it seems to be dead. Did anyone find other alternatives?

    opened by fernandocamargoai 0
  • def join(self):

    def join(self):

    def join(self): while not self.empty(): time.sleep(.1) what's this for

    opened by forgeries 0
  • hot_redis in pytest

    hot_redis in pytest

    How would I use hot_redis within a pytest environment ?

    I would like to use a mock redis client connection, as say with pytest-redis.

    New to redis testing, so not sure if i'm even asking the right question.

    opened by gauss345 0
  • Fix simple typo: repectively -> respectively

    Fix simple typo: repectively -> respectively

    Closes #16

    opened by timgates42 0
  • Fix simple typo: repectively -> respectively

    Fix simple typo: repectively -> respectively

    There is a small typo in hot_redis/types.py. Should read respectively rather than repectively.

    opened by timgates42 0
  • Fixing collections.abc DeprecationWarning

    Fixing collections.abc DeprecationWarning

    That will help hot-redis support python 3.9

    /hot-redis/hot_redis/types.py:919: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
        collections.MutableMapping.register(MultiSet)
    
    opened by ErhoSen 0
  • Which versions of Python are supported?

    Which versions of Python are supported?

    Could you add that to the docs and/or setup.py?

    opened by dangayle 0
Owner
Stephen McDonald
Stephen McDonald
A curated list of awesome tools for SQLAlchemy

Awesome SQLAlchemy A curated list of awesome extra libraries and resources for SQLAlchemy. Inspired by awesome-python. (See also other awesome lists!)

Hong Minhee (洪 民憙) 2.3k Oct 21, 2021
The ormar package is an async mini ORM for Python, with support for Postgres, MySQL, and SQLite.

python async mini orm with fastapi in mind and pydantic validation

null 674 Oct 23, 2021
An async ORM. 🗃

ORM The orm package is an async ORM for Python, with support for Postgres, MySQL, and SQLite. ORM is built with: SQLAlchemy core for query building. d

Encode 1.4k Oct 11, 2021
A pure Python Database Abstraction Layer

pyDAL pyDAL is a pure Python Database Abstraction Layer. It dynamically generates the SQL/noSQL in realtime using the specified dialect for the databa

null 397 Oct 23, 2021
The Orator ORM provides a simple yet beautiful ActiveRecord implementation.

Orator The Orator ORM provides a simple yet beautiful ActiveRecord implementation. It is inspired by the database part of the Laravel framework, but l

Sébastien Eustace 1.3k Oct 17, 2021
SQLModel is a library for interacting with SQL databases from Python code, with Python objects.

SQLModel is a library for interacting with SQL databases from Python code, with Python objects. It is designed to be intuitive, easy to use, highly compatible, and robust.

Sebastián Ramírez 5.5k Oct 24, 2021
A pythonic interface to Amazon's DynamoDB

PynamoDB A Pythonic interface for Amazon's DynamoDB. DynamoDB is a great NoSQL service provided by Amazon, but the API is verbose. PynamoDB presents y

null 1.7k Oct 23, 2021
Tortoise ORM is an easy-to-use asyncio ORM inspired by Django.

Tortoise ORM was build with relations in mind and admiration for the excellent and popular Django ORM. It's engraved in it's design that you are working not with just tables, you work with relational data.

Tortoise 2.4k Oct 20, 2021
Piccolo - A fast, user friendly ORM and query builder which supports asyncio.

A fast, user friendly ORM and query builder which supports asyncio.

null 566 Oct 20, 2021
a small, expressive orm -- supports postgresql, mysql and sqlite

peewee Peewee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use. a small, expressive ORM p

Charles Leifer 8.7k Oct 25, 2021
A Python Object-Document-Mapper for working with MongoDB

MongoEngine Info: MongoEngine is an ORM-like layer on top of PyMongo. Repository: https://github.com/MongoEngine/mongoengine Author: Harry Marr (http:

MongoEngine 3.6k Oct 24, 2021
A very simple CRUD class for SQLModel! ✨

Base SQLModel A very simple CRUD class for SQLModel! ✨ Inspired on: Full Stack FastAPI and PostgreSQL - Base Project Generator FastAPI Microservices I

Marcelo Trylesinski 12 Oct 22, 2021
Pydantic model support for Django ORM

Pydantic model support for Django ORM

Jordan Eremieff 189 Oct 23, 2021
Pony Object Relational Mapper

Downloads Pony Object-Relational Mapper Pony is an advanced object-relational mapper. The most interesting feature of Pony is its ability to write que

null 2.7k Oct 25, 2021
Easy-to-use data handling for SQL data stores with support for implicit table creation, bulk loading, and transactions.

dataset: databases for lazy people In short, dataset makes reading and writing data in databases as simple as reading and writing JSON files. Read the

Friedrich Lindenberg 4.1k Oct 18, 2021
Beanie - is an Asynchronous Python object-document mapper (ODM) for MongoDB

Beanie - is an Asynchronous Python object-document mapper (ODM) for MongoDB, based on Motor and Pydantic.

Roman 277 Oct 18, 2021
MongoEngine flask extension with WTF model forms support

Flask-MongoEngine Info: MongoEngine for Flask web applications. Repository: https://github.com/MongoEngine/flask-mongoengine About Flask-MongoEngine i

MongoEngine 788 Oct 14, 2021
Adds SQLAlchemy support to Flask

Flask-SQLAlchemy Flask-SQLAlchemy is an extension for Flask that adds support for SQLAlchemy to your application. It aims to simplify using SQLAlchemy

The Pallets Projects 3.6k Oct 22, 2021