A Python Library for Simple Models and Containers Persisted in Redis

Related tags

ORM redisco
Overview

Redisco

Python Containers and Simple Models for Redis

Description

Redisco allows you to store objects in Redis. It is inspired by the Ruby library Ohm and its design and code are loosely based on Ohm and the Django ORM. It is built on top of redis-py. It includes container classes that allow easier access to Redis sets, lists, and sorted sets.

Installation

Redisco requires redis-py 2.0.0 so get it first.

pip install redis

Then install redisco.

pip install redisco

Documentation

The documentation is available at : https://redisco.readthedocs.org

Which version should I consider ?

  • v0.1

https://secure.travis-ci.org/kiddouk/redisco.png?branch=0.1

If you want something that is compatible with the original project developed by you should consider v0.1. It works.

  • v0.2

https://secure.travis-ci.org/kiddouk/redisco.png?branch=0.2

If you are adventurous and want to try a version that is going closer to Ohm project you should consider v0.2. Warning, your indexing keys will be broken (if you are planning to migrate).

  • master

https://secure.travis-ci.org/kiddouk/redisco.png?branch=master

Well, expect things to be broken. Really broken.

Models

from redisco import models
class Person(models.Model):
    name = models.Attribute(required=True)
    created_at = models.DateTimeField(auto_now_add=True)
    fave_colors = models.ListField(str)

>>> person = Person(name="Conchita")
>>> person.is_valid()
True
>>> person.save()
True
>>> conchita = Person.objects.filter(name='Conchita')[0]
>>> conchita.name
'Conchita'
>>> conchita.created_at
datetime.datetime(2010, 5, 24, 16, 0, 31, 954704)

Model Attributes

Attribute
Stores unicode strings. If used for large bodies of text, turn indexing of this field off by setting indexed=False.
IntegerField
Stores an int. Ints are stringified using unicode() before saving to Redis.
Counter
An IntegerField that can only be accessed via Model.incr and Model.decr.
DateTimeField
Can store a DateTime object. Saved in the Redis store as a float.
DateField
Can store a Date object. Saved in Redis as a float.
TimeDeltaField
Can store a TimeDelta object. Saved in Redis as a float.
FloatField
Can store floats.
BooleanField
Can store bools. Saved in Redis as 1's and 0's.
ReferenceField
Can reference other redisco model.
ListField
Can store a list of unicode, int, float, as well as other redisco models.

Attribute Options

required
If True, the attirbute cannot be None or empty. Strings are stripped to check if they are empty. Default is False.
default
Sets the default value of the attribute. Default is None.
indexed
If True, redisco will create index entries for the attribute. Indexes are used in filtering and ordering results of queries. For large bodies of strings, this should be set to False. Default is True.
validator
Set this to a callable that accepts two arguments -- the field name and the value of the attribute. The callable should return a list of tuples with the first item is the field name, and the second item is the error.
unique
The field must be unique. Default is False.

DateField and DateTimeField Options

auto_now_add
Automatically set the datetime/date field to now/today when the object is first created. Default is False.
auto_now
Automatically set the datetime/date field to now/today everytime the object is saved. Default is False.

Class options

You can specify some options in your Model to control the behaviour of the back scene.

class User(models.Model):
    firstname = models.Attribute()
    lastname = models.Attribute()

    @property
    def fullname(self):
        return "%s %s" % (self.firstname, self.lastname)

    class Meta:
        indices = ['fullname']
        db = redis.Redis(host="localhost", db="6666")
        key = 'Account'

indices is used to add extra indices that will be saved in the model. db object will be used instead of the global redisco redis_client key will be used as the main key in the redis Hash (and sub objects) instead of the class name.

Custom Managers

Managers are attached to Model attributes by looking for a __attr_name__ class attribute. If not present, then it defaults to the lowercase attribute name in the Model.

class User(models.Model):
    firstname = models.Attribute()
    lastname = models.Attribute()
    active = models.BooleanField(default=True)

    class History(models.managers.Manager):
        pass

    class ObjectsManager(models.managers.Manager):
        __attr_name__ = "objects"
        def get_model_set(self):
            return super(User.ObjectsManager, self).\
                get_model_set().filter(active=True)

Saving and Validating

To save an object, call its save method. This returns True on success (i.e. when the object is valid) and False otherwise.

Calling Model.is_valid will validate the attributes and lists. Model.is_valid is called when the instance is being saved. When there are invalid fields, Model.errors will hold the list of tuples containing the invalid fields and the reason for its invalidity. E.g. [('name', 'required'),('name', 'too short')]

Fields can be validated using the validator argument of the attribute. Just pass a callable that accepts two arguments -- the field name and the value of the attribute. The callable should return a list of errors.

Model.validate will also be called before saving the instance. Override it to validate instances not related to attributes.

def not_me(field_name, value):
    if value == 'Me':
        return ((field_name, 'it is me'),)

class Person(models.Model):
    name = models.Attribute(required=True, validator=not_me)
    age = models.IntegerField()

    def validate(self):
        if self.age and self.age < 21:
            self._errors.append(('age', 'below 21'))

>>> person = Person(name='Me')
>>> person.is_valid()
False
>>> person.errors
[('name', 'it is me')]

Queries

Queries are executed using a manager, accessed via the objects class attribute.

Person.objects.all()
Person.objects.filter(name='Conchita')
Person.objects.filter(name='Conchita').first()
Person.objects.all().order('name')
Person.objects.filter(fave_colors='Red')

Ranged Queries

Redisco has a limited support for queries involving ranges -- it can only filter fields that are numeric, i.e. DateField, DateTimeField, IntegerField, and FloatField. The zfilter method of the manager is used for these queries.

Person.objects.zfilter(created_at__lt=datetime(2010, 4, 20, 5, 2, 0))
Person.objects.zfilter(created_at__gte=datetime(2010, 4, 20, 5, 2, 0))
Person.objects.zfilter(created_at__in=(datetime(2010, 4, 20, 5, 2, 0), datetime(2010, 5, 1)))

Containers

Redisco has three containers that roughly match Redis's supported data structures: lists, sets, sorted set. Anything done to the container is persisted to Redis.

Sets
>>> from redisco.containers import Set
>>> s = Set('myset')
>>> s.add('apple')
>>> s.add('orange')
>>> s.add('bananas', 'tomatoes')
>>> s.add(['blackberries', 'strawberries'])
>>> s.members
set(['apple', 'blackberries', 'strawberries', 'orange', 'tomatoes', 'bananas'])
>>> s.remove('apple', 'orange')
True
set(['strawberries', 'bananas', 'tomatoes', 'blackberries'])
>>> s.remove(['bananas', 'blackberries'])
True
>> s.members
set(['strawberries', 'bananas', 'tomatoes'])
>>> t = Set('nset')
>>> t.add('kiwi')
>>> t.add('guava')
>>> t.members
set(['kiwi', 'guava'])
>>> s.update(t)
>>> s.members
set(['kiwi', 'orange', 'guava', 'apple'])
Lists
>>> from redisco.containers import List
>>> l = List('alpha')
>>> l.append('a')
>>> l.append(['b', 'c'])
>>> l.append('d', 'e', 'f')
>>> 'a' in l
True
>>> 'd' in l
False
>>> len(l)
6
>>> l.index('b')
1
>>> l.members
['a', 'b', 'c', 'd', 'e', 'f']
Sorted Sets
>>> zset = SortedSet('zset')
>>> zset.members
['d', 'a', 'b', 'c']
>>> 'e' in zset
False
>>> 'a' in zset
True
>>> zset.rank('d')
0
>>> zset.rank('b')
2
>>> zset[1]
'a'
>>> zset.add({'f' : 200, 'e' : 201})
>>> zset.members
['d', 'a', 'b', 'c', 'f', 'e']
>>> zset.add('d', 99)
>>> zset.members
['a', 'b', 'c', 'd', 'f', 'e']
Dicts/Hashes
>>> h = cont.Hash('hkey')
>>> len(h)
0
>>> h['name'] = "Richard Cypher"
>>> h['real_name'] = "Richard Rahl"
>>> h
<Hash 'hkey' {'name': 'Richard Cypher', 'real_name': 'Richard Rahl'}>
>>> h.dict
{'name': 'Richard Cypher', 'real_name': 'Richard Rahl'}

Additional Info on Containers

Some methods of the Redis client that require the key as the first argument can be accessed from the container itself.

>>> l = List('mylist')
>>> l.lrange(0, -1)
0
>>> l.rpush('b')
>>> l.rpush('c')
>>> l.lpush('a')
>>> l.lrange(0, -1)
['a', 'b', 'c']
>>> h = Hash('hkey')
>>> h.hset('name', 'Richard Rahl')
>>> h
<Hash 'hkey' {'name': 'Richard Rahl'}>

Connecting to Redis

All models and containers use a global Redis client object to interact with the key-value storage. By default, it connects to localhost:6379, selecting db 0. If you wish to specify settings:

import redisco
redisco.connection_setup(host='localhost', port=6380, db=10)

The arguments to connect are simply passed to the redis.Redis init method.

For the containers, you can specify a second argument as the Redis client. That client object will be used instead of the default.

>>> import redis
>>> r = redis.Redis(host='localhost', port=6381)
>>> Set('someset', r)

Unit tests

Redisco uses nose for testing.

Install nosetests:

$ pip install nose

And test:

$ nosetests

Credits

Most of the concepts are taken from Soveran's Redis related Ruby libraries. cyx for sharing his expertise in indexing in Redis. Django, of course, for the popular model API.

Comments
  • Setting None to ReferenceField raises AttributeError

    Setting None to ReferenceField raises AttributeError

    Having the following models:

    class Video(Model):
        title = Attribute()
    
    class Playlist(Model):
        now_playing = ReferenceField(Video)
    
    

    I can save the playlist model without having to assign the now_playing attribute, but if I assign None to the attribute, I get

    line 346, in __set__
        setattr(instance, self.attname, value.id)
    AttributeError: 'NoneType' object has no attribute 'id'
    

    I'm currently using redisco 0.1.4.

    I think I know where the problem is, should i send a pull request?

    opened by cinemascop89 4
  • Adding support for multiple managers

    Adding support for multiple managers

    Hi, I'm working on a project where we use multiple managers (like the django orm) to keep models and logic separated from the rest of our code. We patched Redisco to add such functionality, and we though you might be interested on it.

    opened by pcostesi 3
  • Using Mixins to add common fields to a model doesn't work

    Using Mixins to add common fields to a model doesn't work

    I tried using a mixin with some common fields, but it doesn't work:

    class CeleryTaskMixin:
        celery_status = models.CharField()  # last known status, call update_status
        celery_id = models.CharField()      # celery uuid
        f_name = models.CharField(indexed=True)
    
    
    class RunningTask(CeleryTaskMixin, models.Model):
        def __init__(self, t=None, *args, **kwargs):
            CeleryTaskMixin.__init__(self)
            models.Model.__init__(self)
    
        task_type = models.CharField()
        start_time = models.DateTimeField(auto_now=True)
        end_time = models.DateTimeField()
    

    Then when I try and do

    models.RunningTask.objects.filter(f_name="blah")
    

    It says

      File "/home/stu/.virtualenvs/ng/src/redisco/redisco/models/modelset.py", line 52, in __iter__
        for id in self._set:
      File "/home/stu/.virtualenvs/ng/src/redisco/redisco/models/modelset.py", line 294, in _set
        s = self._add_set_filter(s)
      File "/home/stu/.virtualenvs/ng/src/redisco/redisco/models/modelset.py", line 317, in _add_set_filter
        (k, self.model_class.__name__))
    redisco.models.exceptions.AttributeNotIndexed: Attribute f_name is not indexed in RunningTask class.
    

    This seems like something that should work... (?)

    opened by stuaxo 2
  • Travis tests all failing with connection refused

    Travis tests all failing with connection refused

    Will see if there is anything obvious:

    ======================================================================
    
    ERROR: test_CharField (redisco.tests.BooleanFieldTestCase)
    
    ----------------------------------------------------------------------
    
    Traceback (most recent call last):
    
      File "/home/travis/build/kiddouk/redisco/redisco/models/basetests.py", line 26, in setUp
    
        self.client.flushdb()
    
      File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/redis/client.py", line 652, in flushdb
    
        return self.execute_command('FLUSHDB')
    
      File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
    
        connection.send_command(*args)
    
      File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
    
        self.send_packed_command(self.pack_command(*args))
    
      File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
    
        self.connect()
    
      File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
    
        raise ConnectionError(self._error_message(e))
    
    ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
    
    opened by stuaxo 2
  • How to run rediscos tests?

    How to run rediscos tests?

    Hi, How do I run rediscos tests, on python 3.5 ?

    I tried python setup.py test but that didn't work ?

    I want to add a feature, but need to make sure I don't break anything ?

    Cheers S

    opened by stuaxo 2
  • add __getitem__ to ModelBase for [] call syntax

    add __getitem__ to ModelBase for [] call syntax

    Hi, i have add \_\_getitem\_\_ method for ModelBase class, so i can to write Model[id] instead of Model.objects.get_by_id(), Although the modification is small, but I think it is useful. Already test passed.

    Thank you for create this repo, it's really nice.

    from redisco import models
    
    class Person(models.Model):
        name = models.Attribute(required=True, unique=True, indexed=True)
    
    Person(name="debugger").save()
    p = Person.objects.filter(name="debugger").first()
    
    print p.id
    print Person.objects.get_by_id(p.id)
    # short code for above
    print Person[p.id]
    
    opened by boostbob 2
  • ge method of SortedSet only returns greater than, not greater than or equal to

    ge method of SortedSet only returns greater than, not greater than or equal to

    In the ge method of SortedSet in containers.py, the score for the min should not have an open parenthesis. With the open parenthesis it becomes an open interval, as described in the documentation for redis: http://redis.io/commands/zrangebyscore.

    opened by luipugs 2
  • Memory Leak - Redisco doesn't delete filter keys properly

    Memory Leak - Redisco doesn't delete filter keys properly

    Considering the following code :

    class A(models.Model):
       att = models.Attribute()
    
    class B(models.Model):
       a = models.ReferenceField()
    
    a = A()
    a.att = "test"
    a.save()
    
    b = B()
    b.a = a
    b.save()
    
    B.objects.filter(a_id=a.id)
    # Here redis doesn't delete the filter keys as "_expire_or_delete" should. Leaking 2 more keys in the DB.
    

    Redisco end up leaking 2 filter keys (best case) and n filtering keys in case of multiple filtering.

    Suggestion

    give a timeout to each temporary keys to let say 60 seconds. It will give time to the operation to succeed and let redis handle the clean up for us.

    urgent 
    opened by kiddouk 2
  • Apply type validation to falsy values as well

    Apply type validation to falsy values as well

    For example, this instance would previously get saved:

    class Person(models.Model):
        age = models.IntegerField()
    
    >>> p = Person(age=[])
    >>> p.save()
    True
    

    …and fail with a value error when loaded:

    >>> Person.objects.get_by_id(p.id)
    Traceback (most recent call last):
      File "<ipython-input-5-623349df4911>", line 1, in <module>
        Person.objects.get_by_id(p.id)
      File "redisco/models/managers.py", line 40, in get_by_id
        return self.get_model_set().get_by_id(id)
      File "redisco/models/modelset.py", line 63, in get_by_id
        return self._get_item_with_id(id)
      File "redisco/models/modelset.py", line 276, in _get_item_with_id
        instance.id = str(id)
      File "redisco/models/base.py", line 328, in id
        att.__set__(self, att.typecast_for_read(stored_attrs[att.name]))
      File "redisco/models/attributes.py", line 153, in typecast_for_read
        return int(value)
    ValueError: invalid literal for int() with base 10: '[]'
    

    With this patch:

    >>> p = Person(age=[])
    >>> p.save()
    [('age', 'bad type')]
    
    opened by jakubroztocil 2
  • Using transactions in Mutex.

    Using transactions in Mutex.

    I think that this is much more clearly defined and visible to the eyes, Redis is good avoiding race conditions, using transactions looks more natural in this context.

    opened by jeloou 2
  • A better way to add data to containers.

    A better way to add data to containers.

    This allows to add multiple values to containers in a easier way.

    Sets:

    from redisco.containers import Set
    s = Set('s')
    s.add('bananas', 'tomatoes')
    s.add(['blackberries', 'strawberries'])
    print s.members # -> set(['blackberries', 'strawberries', 'tomatoes', 'bananas'])
    s.remove('bananas', 'blackberries') # -> True
    print s.members # -> set(['strawberries',  'tomatoes'])
    

    Lists:

    from redisco.containers import List
    l = List('l')
    l.append('a')
    l.append(['b', 'c'])
    l.append('d', 'e', 'f')
    print l # -> ['a', 'b', 'c', 'd', 'e', 'f']
    

    Sorted set:

    from redisco.containers import SortedSet 
    z = SortedSet('z')
    z.add('a', 1)
    z.add({'f' : 200, 'e' : 201})
    print z.members # -> ['a', 'f', 'e'] 
    
    opened by jeloou 2
  • Merge python 3.5 compatability fork

    Merge python 3.5 compatability fork

    https://github.com/dongweiming/redisco/commit/aa93f31f81bf3f163b29f2783a70748ec4479f22

    Without it one gets an Import Error can not import * from base.

    opened by vanpelt 1
  • Modelset filter doesn't work in other DB except 0

    Modelset filter doesn't work in other DB except 0

    I have multiple dbs to store data. If I specify the db 1 in model class as below: class Param(models.Model): Name = models.Attribute(required=True, unique=True) class Meta: indices = ['Name'] db = redis.Redis(host="127.0.0.1", port=6379, db='1')

    The object creation is successful, the object will be created and stored in db 1. But with Param.objects.filter(Name='xxxx'), the resultset is empty. If I don't have this db specification in model class meta, the creation and query both work.

    Anyone aware of this issue?

    opened by XinfuZheng 0
Owner
sebastien requiem
Myeah.
sebastien requiem
SQLModel is a library for interacting with SQL databases from Python code, with Python objects.

SQLModel is a library for interacting with SQL databases from Python code, with Python objects. It is designed to be intuitive, easy to use, highly compatible, and robust.

Sebastián Ramírez 9.1k Dec 31, 2022
The Orator ORM provides a simple yet beautiful ActiveRecord implementation.

Orator The Orator ORM provides a simple yet beautiful ActiveRecord implementation. It is inspired by the database part of the Laravel framework, but l

Sébastien Eustace 1.4k Jan 1, 2023
A very simple CRUD class for SQLModel! ✨

Base SQLModel A very simple CRUD class for SQLModel! ✨ Inspired on: Full Stack FastAPI and PostgreSQL - Base Project Generator FastAPI Microservices I

Marcelo Trylesinski 40 Dec 14, 2022
A simple project to explore the number of GCs when doing basic ORM work.

Question: Does Python do extremely too many GCs for ORMs? YES, OMG YES. Check this out Python Default GC Settings: SQLAlchemy - 20,000 records in one

Michael Kennedy 26 Jun 5, 2022
The ormar package is an async mini ORM for Python, with support for Postgres, MySQL, and SQLite.

python async mini orm with fastapi in mind and pydantic validation

null 1.2k Jan 5, 2023
Python 3.6+ Asyncio PostgreSQL query builder and model

windyquery - A non-blocking Python PostgreSQL query builder Windyquery is a non-blocking PostgreSQL query builder with Asyncio. Installation $ pip ins

null 67 Sep 1, 2022
A pure Python Database Abstraction Layer

pyDAL pyDAL is a pure Python Database Abstraction Layer. It dynamically generates the SQL/noSQL in realtime using the specified dialect for the databa

null 440 Nov 13, 2022
A Python Object-Document-Mapper for working with MongoDB

MongoEngine Info: MongoEngine is an ORM-like layer on top of PyMongo. Repository: https://github.com/MongoEngine/mongoengine Author: Harry Marr (http:

MongoEngine 3.9k Dec 30, 2022
Beanie - is an Asynchronous Python object-document mapper (ODM) for MongoDB

Beanie - is an Asynchronous Python object-document mapper (ODM) for MongoDB, based on Motor and Pydantic.

Roman 993 Jan 3, 2023
Solrorm : A sort-of solr ORM for python

solrorm : A sort-of solr ORM for python solrpy - deprecated solrorm - currently in dev Usage Cores The first step to interact with solr using solrorm

Aj 1 Nov 21, 2021
A PostgreSQL or SQLite orm for Python

Prom An opinionated lightweight orm for PostgreSQL or SQLite. Prom has been used in both single threaded and multi-threaded environments, including en

Jay Marcyes 18 Dec 1, 2022
ORM for Python for PostgreSQL.

New generation (or genius) ORM for Python for PostgreSQL. Fully-typed for any query with Pydantic and auto-model generation, compatible with any sync or async driver

Yan Kurbatov 3 Apr 13, 2022
A new ORM for Python specially for PostgreSQL

A new ORM for Python specially for PostgreSQL. Fully-typed for any query with Pydantic and auto-model generation, compatible with any sync or async driver

Yan Kurbatov 3 Apr 13, 2022
Easy-to-use data handling for SQL data stores with support for implicit table creation, bulk loading, and transactions.

dataset: databases for lazy people In short, dataset makes reading and writing data in databases as simple as reading and writing JSON files. Read the

Friedrich Lindenberg 4.2k Dec 26, 2022
a small, expressive orm -- supports postgresql, mysql and sqlite

peewee Peewee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use. a small, expressive ORM p

Charles Leifer 9.7k Jan 8, 2023
Piccolo - A fast, user friendly ORM and query builder which supports asyncio.

A fast, user friendly ORM and query builder which supports asyncio.

null 919 Jan 4, 2023
Support for Apollo's Automatic Persisted Queries in Strawberry GraphQL 🍓

strawberry-apollo-apq Supporting Apollo's automatic persisted queries in Strawberry GraphQL ?? Notes Don't use this for production yet, unless you kno

Bas 3 May 17, 2022
Cache-house - Caching tool for python, working with Redis single instance and Redis cluster mode

Caching tool for python, working with Redis single instance and Redis cluster mo

Tural 14 Jan 6, 2022
Python cluster client for the official redis cluster. Redis 3.0+.

redis-py-cluster This client provides a client for redis cluster that was added in redis 3.0. This project is a port of redis-rb-cluster by antirez, w

Grokzen 1.1k Jan 5, 2023