Object-data mapper and advanced query manager for non relational databases

Overview

Object data mapper and advanced query manager for non relational databases.

The data is owned by different, configurable back-end databases and it is accessed using a light-weight Object Data Mapper (ODM). The ODM presents a method of associating user-defined Python classes with database collections, and instances of those classes with items in their corresponding collections. Collections and items are different for different backend databases but are treated in the same way in the python language domain.

Master CI: master-build coverage
Dev CI: dev-build coverage-dev
Documentation: http://pythonhosted.org/python-stdnet/
Dowloads: http://pypi.python.org/pypi/python-stdnet/
Source: https://github.com/lsbardel/python-stdnet
Platforms: Linux, OS X, Windows. Python 2.6, 2.7, 3.2, 3.3, pypy
Mailing List: https://groups.google.com/group/python-stdnet
Keywords: server, database, redis, odm

Contents

Features

  • Models with scalar and multi-value fields.
  • Rich query API including unions, intersections, exclusions, ranges and more.
  • Minimal server round-trips via backend scripting (lua for redis).
  • Full text search.
  • Signals handling to allow decoupled applications to get notified on changes.
  • Synchronous and asynchronous database connection.
  • Multi-variate numeric timeseries application.
  • Asynchronous Publish/Subscribe application.
  • 90% Test coverage.
  • Fully documented.

Requirements

  • Python 2.6, 2.7, 3.2, 3.3 and pypy. Single code-base.
  • redis-py for redis backend.
  • Optional pulsar when using the asynchronous connections or the test suite.
  • You need access to a Redis server version 2.6 or above.

Philosophy

Key-valued pairs databases, also know as key-value stores, have many differences from traditional relational databases, most important being they do not use SQL as their query language, storage does not require a fixed table schemas and usually they do not support complex queries.

Stdnet aims to accommodate a flexible schema and join type operations via a lightweight object data mapper. Importantly, it is designed with large data sets in mind. You pull data you need, nothing more, nothing less. Bandwidth and server round-trips can be reduced to the bare minimum so that your application is fast and memory efficient.

Installing

To install, download, uncompress and type:

python setup.py install

otherwise use easy_install:

easy_install python-stdnet

or pip:

pip install python-stdnet

Version Check

To know which version you have installed:

>>> import stdnet
>>> stdnet.__version__
'0.8.0'
>>> stdnet.VERSION
stdnet_version(major=0, minor=8, micro=0, releaselevel='final', serial=1)

Backends

Backend data-stores are the backbone of the library. Currently the list is limited to

Object Data Mapper

The stdnet.odm module is the ODM, it maps python objects into database data and vice-versa. It is design to be fast and safe to use:

from stdnet import odm

class Base(odm.StdModel):
    '''An abstract model. This won't have any data in the database.'''
    name = odm.SymbolField(unique = True)
    ccy  = odm.SymbolField()

    def __unicode__(self):
        return self.name

    class Meta:
        abstract = True


class Instrument(Base):
    itype = odm.SymbolField()


class Fund(Base):
    description = odm.CharField()


class PositionDescriptor(odm.StdModel):
    dt    = odm.DateField()
    size  = odm.FloatField()
    price = odm.FloatField()
    position = odm.ForeignKey("Position", index=False)


class Position(odm.StdModel):
    instrument = odm.ForeignKey(Instrument, related_name='positions')
    fund       = odm.ForeignKey(Fund)
    history    = odm.ListField(model=PositionDescriptor)

    def __unicode__(self):
        return '%s: %s @ %s' % (self.fund,self.instrument,self.dt)

Register models with backend:

models = orm.Router('redis://localhost?db=1')
models.register(Instrument)
models.register(Fund)
models.register(PositionDescriptor,'redis://localhost?db=2')
models.register(Position,'redis://localhost?db=2')

And play with the API:

>>> f = models.fund.new(name="pluto, description="The pluto fund", ccy="EUR")
>>> f
Fund: pluto

Running Tests

At the moment, only redis back-end is available and therefore to run tests you need to install Redis. If you are using linux, it can be achieved simply by downloading, uncompressing and running make, if you are using windows you can find sources from MSOpenTech.

Requirements for running tests:

  • python-stdnet project directory.
  • pulsar.

To run tests open a shell and launch Redis. On another shell, from within the python-stdnet package directory, type:

python runtests.py

Tests are run against a local redis server on port 6379 and database 7 by default. To change the server and database where to run tests pass the --server option as follow:

python runtests.py --server redis://myserver.com:6450?db=12&password=bla

For more information type:

python runtests.py -h

Kudos

  • Redis simply because this library uses its awesome features.
  • SQLAlchemy and Django for ideas and API design.

Contributing

Development of stdnet happens at Github: http://github.com/lsbardel/python-stdnet

We very much welcome your contribution of course. To do so, simply follow these guidelines:

  1. Fork python-stdnet on github
  2. Create a topic branch git checkout -b my_branch
  3. Push to your branch git push origin my_branch
  4. Create an issue at https://github.com/lsbardel/python-stdnet/issues with a link to your patch

License

This software is licensed under the New BSD License. See the LICENSE file in the top distribution directory for the full license text.

Comments
  • exclude and 'ResponseError: unknown command 'ZDIFFSTORE' '

    exclude and 'ResponseError: unknown command 'ZDIFFSTORE' '

    I am on redis 2.4.8, This error message occurs when I try to use exclude method to filter some objects out.

    Here is the fulll traceback:

    In [18]: len(ActiveUser.objects.exclude(id__in=[1, 2, 4]).filter(id_in=[7, 8, 9]))

    QuerySetError Traceback (most recent call last) /home/kuno/utopia/sogoke/sogoke/ in () ----> 1 len(ActiveUser.objects.exclude(id__in=[1, 2, 4]).filter(id_in=[7, 8, 9]))

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in len(self) 167 168 def len(self): -->169 return self.count() 170 171 # PRIVATE METHODS

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in count(self) 149 receive any data from the server. It construct the queries and count the 150 objects on the server side.''' -->151 self._buildquery() 152 return self.qset.count() 153

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in _buildquery(self) 178 if self.fargs: 179 self.simple = not self.filter_sets -->180 fargs = self.aggregate(self.fargs) 181 else: 182 fargs = None

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in aggregate(self, kwargs) 192 193 def aggregate(self, kwargs): -->194 return sorted(self._aggregate(kwargs), key = lambda x : x.name) 195 196 def _aggregate(self, kwargs):

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in _aggregate(self, kwargs) 206 if name not in fields: 207 raise QuerySetError('Could not filter on model "{0}".
    -->208 Field "{1}" does not exist.'.format(meta,name)) 209 field = fields[name] 210 value = (field.serialize(value),)

    QuerySetError: Could not filter on model "streams.activeuser". Field "id_in" does not exist.

    In [19]: len(ActiveUser.objects.exclude(id__in=[1, 2, 4]).filter(id__in=[7, 8, 9])) ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (77, 0))


    ResponseError Traceback (most recent call last) /home/kuno/utopia/sogoke/sogoke/ in () ----> 1 len(ActiveUser.objects.exclude(id__in=[1, 2, 4]).filter(id__in=[7, 8, 9]))

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in len(self) 167 168 def len(self): -->169 return self.count() 170 171 # PRIVATE METHODS

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in count(self) 149 receive any data from the server. It construct the queries and count the 150 objects on the server side.''' -->151 self._buildquery() 152 return self.qset.count() 153

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in _buildquery(self) 189 self.simple = False 190 self.qset = self._meta.cursor.Query(self,fargs,eargs, -->191 queries = self.queries) 192 193 def aggregate(self, kwargs):

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/backends/base.pyc in init(self, qs, fargs, eargs, timeout, queries) 35 code = self._sha.getvalue() 36 self._sha = code if not code else sha1(code).hexdigest() --->37 self.execute() 38 39 @property

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/backends/redisb.pyc in execute(self) 236 self.query_set = key 237 else: -->238 self.result = self.pipe.execute() 239 240 def order(self):

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/lib/redis.pyc in execute(self, with_callbacks) 1091 conn = self.connection_pool.get_connection('MULTI', self.shard_hint) 1092 try: ->1093 return execute(conn, stack, with_callbacks) 1094 except ConnectionError: 1095 conn.disconnect()

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/lib/redis.pyc in execute_transaction(self, connection, commands, with_callbacks) 1050 parse_response = self.parse_response 1051 for i in range(len(commands)+1): ->1052 parse_response(connection, '') 1053 # parse the EXEC.

    1054 response = parse_response(connection, '_')

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/lib/redis.pyc in parse_response(self, connection, command_name, *_options) 233 def parse_response(self, connection, command_name, *_options): 234 "Parses a response from the Redis server" -->235 response = connection.read_response() 236 if command_name in self.response_callbacks: 237 return self.response_callbacks[command_name](response, **options)

    /home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/lib/connection.pyc in read_response(self) 181 response = self._parser.read_response() 182 if response.class == ResponseError: -->183 raise response 184 return response 185

    opened by kuno 13
  • MemoryError on large transactions

    MemoryError on large transactions

    Hi Luca,

    I did some testing with biggish transactions. When adding 25000 records, I get a MemoryError:

    <type 'exceptions.MemoryError'>         <traceback object at 0x1153d1b28>'
    

    (I can't get a traceback somehow, it's running from inside C code through pythons C api on unix and that might complicate things)

    10000 records per transaction goes okay, so I split it up in chunks.

    Is there a way to check how much memory I got left, or even better: use a streaming interface that streams to the backend and doesn't have a memory limitation?

    Kind regards, tw

    opened by tw-bert 9
  • Running tests - not enough info in the documentation for me, help appreciated

    Running tests - not enough info in the documentation for me, help appreciated

    A warning up front: I'm not very experienced in python, so please forgive me if I ask something trivial.

    I'm trying to follow the test instructions at http://lsbardel.github.io/python-stdnet/overview.html#running-tests .

    I run Python 2.7.4 32-bit on windows (and 64-bit on IBM AIX and linux, but no stdnet there yet, first I'd like to do some tests).

    I installed the following packages: easy_install (by distribute_setup.py) easy_install pip Cython (win32 python 2.7 installer) pip install mock pip install coverage pip install pulsar pip install stdnet

    The documentation refers to a "runtests.py", but that file is not installed by the pip stdnet install. Then I downloaded a snapshot of the latest stdnet (just to try it) from github, grabbed runtests.py. Got a message: Traceback (most recent call last): File "runtests.py", line 30, in from stdnet.utils import test ImportError: cannot import name test So likewise, I put test.py from the snapshot in the site-packages/stdnet.

    I issued the following command (note: my test redis server is on another machine, linux, and is up and running, and reachable from my windows machine):

    D:\Program Files\Program_Progs\Python\2.7\Lib\site-packages\stdnet-tests>python runtests.py --server redis://srv-redis-dev-01:14111?db=12

    Which gives me: 2.7.4 (default, Apr 6 2013, 19:54:46) [MSC v.1500 32 bit (Intel)] Error occurred before starting tests Traceback (most recent call last): File "D:\Program Files\Program_Progs\Python\2.7\lib\site-packages\pulsar\apps\test__init__.py", line 267, in monitor_start self.runner.on_start() File "D:\Program Files\Program_Progs\Python\2.7\lib\site-packages\pulsar\apps\test\result.py", line 352, in _ c = getattr(p, name)(*args) File "D:\Program Files\Program_Progs\Python\2.7\lib\site-packages\stdnet\utils\test.py", line 179, in on_start raise pulsar.HaltServer('No server available. BAILING OUT') HaltServer: No server available. BAILING OUT

    Now I'm stuck. Any ideas? Did I miss something in the documentation?

    Kind regards + TIA, tw

    opened by tw-bert 7
  • Wrong number of response items

    Wrong number of response items

    I can not make this (nice looking!) library work, even with a very basic example:

    from stdnet import odm
    
    class Author(odm.StdModel):
        name = odm.SymbolField()
    
    odm.register(Author, 'redis://localhost:6379?db=2')
    a = Author(name='coucou')
    a.save()
    

    throws the exception:

    Traceback (most recent call last):
      File "test.py", line 31, in <module>
        a.save()
      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/odm/base.py", line 522, in save
        t.add(self)
      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/odm/session.py", line 384, in __exit__
        result = self.commit()
      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/odm/session.py", line 407, in commit
        return self.backend.execute_session(self.session, self.post_commit)
      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/backends/redisb.py", line 831, in execute_session
        command, result = redis_execution(pipe, session_result)
      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/backends/redisb.py", line 139, in redis_execution
        results = pipe.execute(load_script=True)
      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/lib/redis/client.py", line 1304, in execute
        return self.request(load_script).execute()
      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/lib/redis/connection.py", line 219, in execute
        return self.read_response()
      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/lib/redis/connection.py", line 237, in read_response
        response = self.parse(stream)
      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/lib/redis/connection.py", line 180, in parse
        return self.close()
      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/lib/redis/connection.py", line 158, in close
        response = self.client.parse_response(self)
      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/lib/redis/client.py", line 1278, in parse_response
        raise ResponseError("Wrong number of response items from "
    NameError: global name 'ResponseError' is not defined
    

    What am I doing wrong?

    bug 
    opened by abulte 6
  • Only an issue in 0.8.1 beta version , only as information

    Only an issue in 0.8.1 beta version , only as information

    Hi Luca,

    Just wanted to share my experience with you with installing/running this version on windows:

    Revision: 2790d9d5fbd93d412999b4c25f2a05d8bc818e5e
    Author: quantmind <[email protected]>
    Date: 18-6-2013 19:48:30
    Message:
    pep 8 minor changes
    

    Cython is properly recognized during setup:

    Cythonizing E:\work\eclipse\11.1A\python-stdnet\extensions\src/cparser.pyx
    running install
    [...]
    creating build\temp.win32-2.7\Release\work\eclipse\11.1A\python-stdnet\extensions\src
    D:\Program Files\Program_Progs\Microsoft VisualStudio Express 2008\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IE:\work\eclipse\11.1A\python-stdnet\extensions\src "-ID:\Program Files\Program_Progs\Python\2.7\include" "-ID:\Program Files\Program_Progs\Python\2.7\PC" /TpE:\work\eclipse\11.1A\python-stdnet\extensions\src/cparser.cpp /Fobuild\temp.win32-2.7\Release\work\eclipse\11.1A\python-stdnet\extensions\src/cparser.obj
    cparser.cpp
    D:\Program Files\Program_Progs\Microsoft VisualStudio Express 2008\VC\INCLUDE\xlocale(342) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc
    e:\work\eclipse\11.1a\python-stdnet\extensions\src\parser.h(129) : warning C4244: 'argument' : conversion from 'integer' to 'unsigned int', possible loss of data
    e:\work\eclipse\11.1a\python-stdnet\extensions\src\parser.h(130) : warning C4244: 'argument' : conversion from 'integer' to 'unsigned int', possible loss of data
    e:\work\eclipse\11.1a\python-stdnet\extensions\src\parser.h(162) : warning C4244: 'argument' : conversion from 'integer' to 'unsigned int', possible loss of data
    e:\work\eclipse\11.1a\python-stdnet\extensions\src\parser.h(163) : warning C4244: 'argument' : conversion from 'integer' to 'unsigned int', possible loss of data
    e:\work\eclipse\11.1a\python-stdnet\extensions\src\parser.h(221) : warning C4244: 'argument' : conversion from 'integer' to 'unsigned int', possible loss of data
    e:\work\eclipse\11.1a\python-stdnet\extensions\src\parser.h(222) : warning C4244: 'argument' : conversion from 'integer' to 'Py_ssize_t', possible loss of data
    e:\work\eclipse\11.1a\python-stdnet\extensions\src\parser.h(224) : warning C4244: 'argument' : conversion from 'integer' to 'unsigned int', possible loss of data
    e:\work\eclipse\11.1a\python-stdnet\extensions\src\parser.h(226) : warning C4244: 'argument' : conversion from 'integer' to 'unsigned int', possible loss of data
    D:\Program Files\Program_Progs\Microsoft VisualStudio Express 2008\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO "/LIBPATH:D:\Program Files\Program_Progs\Python\2.7\libs" "/LIBPATH:D:\Program Files\Program_Progs\Python\2.7\PCbuild" /EXPORT:initcparser build\temp.win32-2.7\Release\work\eclipse\11.1A\python-stdnet\extensions\src/cparser.obj /OUT:build\lib.win32-2.7\stdnet\backends\redisb\cparser.pyd /IMPLIB:build\temp.win32-2.7\Release\work\eclipse\11.1A\python-stdnet\extensions\src\cparser.lib /MANIFESTFILE:build\temp.win32-2.7\Release\work\eclipse\11.1A\python-stdnet\extensions\src\cparser.pyd.manifest
       Creating library build\temp.win32-2.7\Release\work\eclipse\11.1A\python-stdnet\extensions\src\cparser.lib and object build\temp.win32-2.7\Release\work\eclipse\11.1A\python-stdnet\extensions\src\cparser.exp
    
    

    At runtime, python crashes at the statement response = self._parser.get() in client.py line 68. Unknown exception.

    Just to let you know, kind regards, tw

    Note: 0.8, which does not use Cython (and is thus less optimized) installs and runs fine on windows.

    opened by tw-bert 5
  • Fix test broken

    Fix test broken

    test broken

    There was lua script loading problem with PrefixedRedis.

    This is re-produce code.

    from stdnet.backends import getdb
    
    backend = getdb('redis://127.0.0.1:6379?db=1')
    
    prefixed_client = backend.client.prefixed('foo')
    # hide the problem when un-comment this.
    #print backend.client.execute_script('keyinfo', (), '*')
    print prefixed_client.execute_script('keyinfo', (), '*')
    

    tox

    python-stdnet supports multi python platform. so tests should also be run in them.

    opened by jbking 4
  • Probably obvious but can't make it work: foreign keys in custom transaction scope

    Probably obvious but can't make it work: foreign keys in custom transaction scope

    What's wrong with the following example code? It's probably obvious, but i can't get it to work. I can't use "with" syntax, because the actual code is part of an incoming data stream, so I have to begin and commit the transactions myself. I also need (not shown in example code) to put multiple parents and their childs in one transaction.

    
    import redis
    from stdnet import odm
    
    def testTrans():
    
      oRedis = redis.StrictRedis(host="srv-redis-dev-01", port=14111, db=1)
      oRedis.flushdb()
    
      models = odm.Router('redis://srv-redis-dev-01:14111?db=1')
      models.register(TestParent)  
      models.register(TestChild)
      session = models.session()
    
      session.begin()
      oCurrParent = models[TestParent](
        cName           = "SunflowerParent01",
        cTestParentAttr = "Seeds01")  
      session.add(oCurrParent)
      oCurrChild = models[TestChild](
        oParent        = oCurrParent,
        cName          = "SunflowerChild01",
        cTestChildAttr = "Petals01")             
      session.add(oCurrChild)           
      oCurrChild = models[TestChild](
        oParent        = oCurrParent,
        cName          = "SunflowerChild02",
        cTestChildAttr = "Petals02")             
      session.add(oCurrChild)
      print oCurrParent,oCurrChild,oCurrParent.children.all()
      session.commit()
    
      session.begin()
      oCurrParent = models[TestParent](
        cName           = "SunflowerParent02",
        cTestParentAttr = "Seeds02")  
      session.add(oCurrParent)           
      oCurrChild = models[TestChild](
        oParent        = oCurrParent,
        cName          = "SunflowerChild03",
        cTestChildAttr = "Petals03")             
      session.add(oCurrChild)           
      print oCurrParent,oCurrChild
      session.commit()
    
    class TestBase(odm.StdModel):
        cName    = odm.SymbolField(unique = True)
        def __unicode__(self):
            return self.cName
        class Meta:
            abstract = True
    
    class TestParent(TestBase):
        cTestParentAttr = odm.CharField()
    
    class TestChild(TestBase):
        oParent        = odm.ForeignKey(TestParent, related_name='children')
        cTestChildAttr = odm.CharField()
    
    testTrans()
    
    

    The following exception is thrown:

    
    Traceback (most recent call last):
      File "E:\work\eclipse\11.1A\amber_python\src\amber\mxroot\mxflux\db_r\sc_mxflux_dfexport_r_idetest.py", line 66, in <module>
        testTrans()
      File "E:\work\eclipse\11.1A\amber_python\src\amber\mxroot\mxflux\db_r\sc_mxflux_dfexport_r_idetest.py", line 37, in testTrans
        session.commit()
      File "D:\Program Files\Program_Progs\Python\2.7\lib\site-packages\python_stdnet-0.8.1b2-py2.7-win32.egg\stdnet\odm\session.py", line 608, in commit
        return self.transaction.commit()
      File "D:\Program Files\Program_Progs\Python\2.7\lib\site-packages\python_stdnet-0.8.1b2-py2.7-win32.egg\stdnet\odm\session.py", line 453, in commit
        self.on_result = self._commit()
      File "D:\Program Files\Program_Progs\Python\2.7\lib\site-packages\python_stdnet-0.8.1b2-py2.7-win32.egg\stdnet\utils\async.py", line 76, in _
        return execute_generator(f(*args, **kwargs))
      File "D:\Program Files\Program_Progs\Python\2.7\lib\site-packages\python_stdnet-0.8.1b2-py2.7-win32.egg\stdnet\utils\async.py", line 40, in execute_generator
        result = gen.send(result)
      File "D:\Program Files\Program_Progs\Python\2.7\lib\site-packages\python_stdnet-0.8.1b2-py2.7-win32.egg\stdnet\odm\session.py", line 470, in _commit
        multi.append(backend.execute_session(data))
      File "D:\Program Files\Program_Progs\Python\2.7\lib\site-packages\python_stdnet-0.8.1b2-py2.7-win32.egg\stdnet\backends\redisb\__init__.py", line 820, in execute_session
        json.dumps(instance._dbdata['errors']))
    stdnet.utils.exceptions.FieldValueError: {"oParent_id": "Field 'oParent_id' is required for '__main__.testchild'."}
    

    Different issue: what also surprises me, is that the transaction failed but isn't rolled back in the backend:

    redis 127.0.0.1:14111[1]> keys *
    1) "__main__.testparent:obj:1"
    2) "__main__.testparent:ids"
    3) "__main__.testparent:id"
    4) "__main__.testparent:uni:cName"
    

    Kind regards, tw

    opened by tw-bert 4
  • Global Lua variables problem with new Redis

    Global Lua variables problem with new Redis

    Hi.

    I was trying to use the latest Stdnet version (0.7.c2). I have problems with all my Redis versions: 2.2 from Ubuntu 11.10 2.6.0-rc3 from redis.io and a newer build from Redis repository

    2.2 fails with pipeline command error and tests fail with "unknown SCRIPT command". I assume Stdnet 0.7+ requires Lua support in Redis. With 0.6 I have had no problems, but I'd like to use the new features...

    Alas, the newer Redis fails with "Script attempted to create global variable" where variable depends upon the script used.

    It seems Redis did allow global variables at some point: https://github.com/antirez/redis/issues/484

    So my question is which Redis version is supported at the moment?

    Thank you.

    bug lua 
    opened by jarpineh 4
  • Range filtering

    Range filtering

    Indexing in redis backend is obtained via sets or sorted sets, depending on the meta ordering attribute. These indices allow the filtering with respect specific index values. For example

    MyModel.objects.filter(field1='bla', field2='foo')
    

    But if we need to filter for a set of values we are stuck.

    For example, lets consider an integer field, we would like to be able to do

    MyModel.objects.filter(field3__ge=3)
    

    To retrieve all instances with field3 greater or equal 3.

    This can be achieved in redis, by adding a sorted set for each indexed field.

    new feature redis 
    opened by lsbardel 4
  • WATCH transactions

    WATCH transactions

    Hi.

    Stdnet looks like a very powerful, yet simple ORM. I read the docs and tried looking through the code for transaction support.

    I have an use case almost exactly like in the Redis docs: http://redis.io/topics/transactions -> section on optimistic locking and check-and-set

    Does transactions with Stdnet support WATCH command?

    My thanks.

    Jussi

    transaction lua 
    opened by jarpineh 4
  • Password connection string

    Password connection string

    Hi,

    I'm having trouble configuring a stdnet/redis connection with a login/password.

    I tried this:

    odm.register(Build, 'redis://root:redispasswd@localhost:6379?db=1')
    

    Which gives me:

      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/lib/redis/connection.py", line 363, in connect
        raise RedisConnectionError(self._error_message(e))
    RedisConnectionError: Error 2 connecting root:redispasswd@localhost:6379. No such file or directory.
    

    By the way, there is a problem in doc at http://lsbardel.github.com/python-stdnet/api/models.html#register. If you put a slash after the host name, you will get:

      File "/home/vagrant/pyenv/local/lib/python2.7/site-packages/stdnet/backends/main.py", line 38, in parse_backend
        Found {0}".format(path))
    stdnet.exceptions.ImproperlyConfigured: Backend URI must not have a path. Found /?db=1
    
    opened by abulte 3
  • docs: Fix a few typos

    docs: Fix a few typos

    There are small typos in:

    • docs/source/examples/related.rst
    • docs/source/examples/serialise.rst
    • examples/permissions.py
    • stdnet/backends/redisb/init.py
    • stdnet/odm/fields.py
    • stdnet/odm/mapper.py
    • stdnet/odm/search.py
    • stdnet/odm/session.py
    • stdnet/utils/test.py

    Fixes:

    • Should read shortcut rather than shurtcut.
    • Should read false rather than falue.
    • Should read through rather than throgh.
    • Should read temporary rather than temorary.
    • Should read straightforward rather than strightforward.
    • Should read responsibility rather than responsability.
    • Should read relationships rather than relationaships.
    • Should read receiving rather than receving.

    Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

    opened by timgates42 0
  • TypeError trying to run simple example from the documentation

    TypeError trying to run simple example from the documentation

    Hi there.

    I was trying to run a simple example from the docs to see if stdnet was usefull for a project but I keep getting a TypeError: <lambda>() takes no arguments (1 given).

    The example is this:

    from stdnet import odm
    
    class Fund(odm.StdModel):
        name = odm.SymbolField(unique=True)
        ccy = odm.SymbolField()
        description = odm.CharField()
    
        def __unicode__(self):
            return self.name
    
    if __name__ == "__main__":
        models = odm.Router('redis://localhost:6379')
    
        models.register(Fund)
        models.register(Instrument)
        models.register(Position)
    
        with models.session().begin() as transaction:
            transaction.add(models.fund(name='Markowitz', ccy='EUR'))
    

    And the traceback is this:

    (recsys)[/tmp]$ python fund.py
    Traceback (most recent call last):
      File "fund.py", line 42, in <module>
        transaction.add(models.fund(name='Markowitz', ccy='EUR'))
      File "/home/calsaverini/.virtualenvs/recsys/local/lib/python2.7/site-packages/stdnet/odm/session.py", line 434, in __exit__
        return self.commit()
      File "/home/calsaverini/.virtualenvs/recsys/local/lib/python2.7/site-packages/stdnet/odm/session.py", line 454, in commit
        self.on_result = self._commit()
      File "/home/calsaverini/.virtualenvs/recsys/local/lib/python2.7/site-packages/stdnet/utils/async.py", line 76, in _
        return execute_generator(f(*args, **kwargs))
      File "/home/calsaverini/.virtualenvs/recsys/local/lib/python2.7/site-packages/stdnet/utils/async.py", line 40, in execute_generator
        result = gen.send(result)
      File "/home/calsaverini/.virtualenvs/recsys/local/lib/python2.7/site-packages/stdnet/odm/session.py", line 471, in _commit
        multi.append(backend.execute_session(data))
      File "/home/calsaverini/.virtualenvs/recsys/local/lib/python2.7/site-packages/stdnet/backends/redisb/__init__.py", line 839, in execute_session
        return pipe.execute()
      File "/home/calsaverini/.virtualenvs/recsys/local/lib/python2.7/site-packages/stdnet/backends/redisb/client.py", line 350, in execute
        raise_on_error=raise_on_error)
      File "/home/calsaverini/.virtualenvs/recsys/local/lib/python2.7/site-packages/stdnet/backends/redisb/client.py", line 152, in request_pipeline
        conn = self.get_connection('MULTI', pipeline.shard_hint)
      File "/home/calsaverini/.virtualenvs/recsys/local/lib/python2.7/site-packages/redis/connection.py", line 874, in get_connection
        connection = self.make_connection()
      File "/home/calsaverini/.virtualenvs/recsys/local/lib/python2.7/site-packages/redis/connection.py", line 883, in make_connection
        return self.connection_class(**self.connection_kwargs)
      File "/home/calsaverini/.virtualenvs/recsys/local/lib/python2.7/site-packages/redis/connection.py", line 405, in __init__
        self._parser = parser_class(socket_read_size=socket_read_size)
    TypeError: <lambda>() takes no arguments (1 given)
    

    I'm pretty sure it must be something dumb I did, but I couldn't find the problem. The version of stdnet is python-stdnet==0.8.2 installed via pip.

    opened by rcalsaverini 2
  • How to develop?

    How to develop?

    With the master (d362bff9) branch seems passed at https://travis-ci.org/lsbardel/python-stdnet/jobs/16918109 But for now, testing with pulsar is failed even (0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.9.0) released and the rev (8e682b6d7)

    This is with pulsar 0.9.0 released.

    % python -m covrun
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
        "__main__", fname, loader, pkg_name)
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
        exec code in run_globals
      File "/Users/yusuke/src/python-stdnet/covrun.py", line 11, in <module>
        run()
      File "runtests.py", line 41, in run
        runtests(**params)
      File "runtests.py", line 56, in runtests
        suite.bind_event('tests', test.create_tests)
    AttributeError: 'TestSuite' object has no attribute 'bind_event'
    

    Which version of pulsar is usable? or needs something more for development? I'm using Python 2.7.6 (homebrew) on Mac OSX Marvericks

    opened by jbking 1
  • Interface mismatch with Redis backend

    Interface mismatch with Redis backend

    The only implementation of backend for Redis doesn't has second parameter, callback, but interface has. Usage also drops the callback. So I assume the implementation is correct.

    opened by jbking 0
  • TypeError in  models.session().add( object ) ?

    TypeError in models.session().add( object ) ?

    Hi guys, I am getting the following error when i try to add a model instance to the session.

    Error: TypeError: raise_first_error() takes exactly 3 arguments (2 given)

    Raised when: models = odm.Router('redis://localhost:6379?db=0') models.session().add( my_object)

    Stdnet Version: 0.8.2

    Python: 2.7

    Redis: 2.9.0

    Thanks

    opened by razaHamdani 2
Owner
Luca Sbardella
quantitative wizard at @quantmind
Luca Sbardella
Advanced Variable Manager {AVM} [0.8.0]

Advanced Variable Manager {AVM} [0.8.0] By Grosse pastèque#6705 WARNING : This modules need some typing modifications ! If you try to run it without t

Big watermelon 1 Dec 11, 2021
Installer, package manager, build wrapper and version manager for Piccolo

Piccl Installer, package manager, build wrapper and version manager for Piccolo

null 1 Dec 19, 2021
A simple and easy to use Python's PIP configuration manager, similar to the Arch Linux's Java manager.

PIPCONF - The PIP configuration manager If you need to manage multiple configurations containing indexes and trusted hosts for PIP, this project was m

João Paulo Carvalho 11 Nov 30, 2022
ArinjoyTheDev 1 Jul 17, 2022
Expense-manager - Expense manager with python

Expense_manager TO-DO Source extractor: Credit Card, Wallet Destination extracto

null 1 Feb 13, 2022
BloodCheck enables Red and Blue Teams to manage multiple Neo4j databases and run Cypher queries against a BloodHound dataset.

BloodCheck BloodCheck enables Red and Blue Teams to manage multiple Neo4j databases and run Cypher queries against a BloodHound dataset. Installation

Mr B0b 16 Nov 5, 2021
A project to work with databases in 4 worksheets, insert, update, select, delete using Python and MySqI

A project to work with databases in 4 worksheets, insert, update, select, delete using Python and MySqI As a small project for school or college hope it is useful

Sina Org 1 Jan 11, 2022
Hashcrack - A non-object oriented open source, Software for Windows/Linux made in Python 3

Multi Force This project is a non-object oriented open source, Software for Wind

Radiationbolt 3 Jan 2, 2023
Introduction to Databases Coursework 2 (SQL) - dataset generator

Introduction to Databases Coursework 2 (SQL) - dataset generator This is python script generates a text file with insert queries for the schema.sql fi

Javier Bosch 1 Nov 8, 2021
A simple flashcard app built as a final project for a databases class.

CS2300 Final Project - Flashcard app 'FlashStudy' Tech stack Backend Python (Language) Django (Web framework) SQLite (Database) Frontend HTML/CSS/Java

Christopher Spencer 2 Feb 3, 2022
Craxk is a SINGLE AND NON-REPLICABLE Hash that uses data from the hardware where it is executed to form a hash that can only be reproduced by a single machine.

What is Craxk ? Craxk is a UNIQUE AND NON-REPLICABLE Hash that uses data from the hardware where it is executed to form a hash that can only be reprod

null 5 Jun 19, 2021
Data Poisoning based on Adversarial Attacks using Non-Robust Features

Data Poisoning based on Adversarial Attacks using Non-Robust Features Usage python main.py [-h] [--gpu | -g GPU] [--eps |-e EPSILON] [--pert | -p PER

Jonathan E. 1 Nov 2, 2021
This is a far more in-depth and advanced version of "Write user interface to a file API Sample"

Fusion360-Write-UserInterface This is a far more in-depth and advanced version of "Write user interface to a file API Sample" from https://help.autode

null 4 Mar 18, 2022
JD-backup is an advanced Python script, that will extract all links from a jDownloader 2 file list and export them to a text file.

JD-backup is an advanced Python script, that will extract all links from a jDownloader 2 file list and export them to a text file.

Kraken.snv 3 Jun 7, 2022
An Advanced Wordlist Library Written In Python For Acm114

RBAPG ->RBAPG is the abbreviation of "Rule Based Attack Password Generator". ->This module is a wordlist generator module. ->You can generate randomly

Aziz Kaplan 11 Aug 28, 2022
An advanced pencil sketch generator

Pencilate An advanced pencil sketch generator About : An advanced pencil sketch maker made in just 12 lines of code. Yes you read it right, JUST 12 LI

MAINAK CHAUDHURI 23 Dec 17, 2022
Advanced Developing of Python Apps Final Exercise

Advanced-Developing-of-Python-Apps-Final-Exercise This is an exercise that I did for a python advanced learning course. The exercise is divided into t

Alejandro Méndez Fernández 1 Dec 4, 2021
Video Stream is an Advanced Telegram Bot that's allow you to play Video & Music on Telegram Group Video Chat

Video Stream is an Advanced Telegram Bot that's allow you to play Video & Music on Telegram Group Video Chat ?? Stats ?? Get SESSION_NAME from below:

dark phoenix 12 May 8, 2022
An advanced NFT Generator

NFT Generator An advanced NFT Generator Free software: GNU General Public License v3 Documentation: https://nft-generator.readthedocs.io. Features TOD

NFT Generator 5 Apr 21, 2022