Redis Python Client - The Python interface to the Redis key-value store.

Overview

redis-py

The Python interface to the Redis key-value store.

CI docs MIT licensed pypi codecov Total alerts

Installation | Contributing | Getting Started | Connecting To Redis


Installation

redis-py requires a running Redis server. See Redis's quickstart for installation instructions.

redis-py can be installed using pip similar to other Python packages. Do not use sudo with pip. It is usually good to work in a virtualenv or venv to avoid conflicts with other package managers and Python projects. For a quick introduction see Python Virtual Environments in Five Minutes.

To install redis-py, simply:

$ pip install redis

or from source:

$ python setup.py install

Contributing

Want to contribute a feature, bug fix, or report an issue? Check out our guide to contributing.

Getting Started

redis-py supports Python 3.6+.

>>> import redis
>>> r = redis.Redis(host='localhost', port=6379, db=0)
>>> r.set('foo', 'bar')
True
>>> r.get('foo')
b'bar'

By default, all responses are returned as bytes in Python 3.

If all string responses from a client should be decoded, the user can specify decode_responses=True in Redis.__init__. In this case, any Redis command that returns a string type will be decoded with the encoding specified.

The default encoding is utf-8, but this can be customized by specifiying the encoding argument for the redis.Redis class. The encoding will be used to automatically encode any strings passed to commands, such as key names and values.


MSET, MSETNX and ZADD

These commands all accept a mapping of key/value pairs. In redis-py 2.X this mapping could be specified as *args or as **kwargs. Both of these styles caused issues when Redis introduced optional flags to ZADD. Relying on *args caused issues with the optional argument order, especially in Python 2.7. Relying on **kwargs caused potential collision issues of user keys with the argument names in the method signature.

To resolve this, redis-py 3.0 has changed these three commands to all accept a single positional argument named mapping that is expected to be a dict. For MSET and MSETNX, the dict is a mapping of key-names -> values. For ZADD, the dict is a mapping of element-names -> score.

MSET, MSETNX and ZADD now look like:

def mset(self, mapping):
def msetnx(self, mapping):
def zadd(self, name, mapping, nx=False, xx=False, ch=False, incr=False):

All 2.X users that use these commands must modify their code to supply keys and values as a dict to these commands.

ZINCRBY

redis-py 2.X accidentally modified the argument order of ZINCRBY, swapping the order of value and amount. ZINCRBY now looks like:

def zincrby(self, name, amount, value):

All 2.X users that rely on ZINCRBY must swap the order of amount and value for the command to continue to work as intended.

Encoding of User Input

redis-py 3.0 only accepts user data as bytes, strings or numbers (ints, longs and floats). Attempting to specify a key or a value as any other type will raise a DataError exception.

redis-py 2.X attempted to coerce any type of input into a string. While occasionally convenient, this caused all sorts of hidden errors when users passed boolean values (which were coerced to 'True' or 'False'), a None value (which was coerced to 'None') or other values, such as user defined types.

All 2.X users should make sure that the keys and values they pass into redis-py are either bytes, strings or numbers.

Locks

redis-py 3.0 drops support for the pipeline-based Lock and now only supports the Lua-based lock. In doing so, LuaLock has been renamed to Lock. This also means that redis-py Lock objects require Redis server 2.6 or greater.

2.X users that were explicitly referring to LuaLock will have to now refer to Lock instead.

Locks as Context Managers

redis-py 3.0 now raises a LockError when using a lock as a context manager and the lock cannot be acquired within the specified timeout. This is more of a bug fix than a backwards incompatible change. However, given an error is now raised where none was before, this might alarm some users.

2.X users should make sure they're wrapping their lock code in a try/catch like this:

try:
    with r.lock('my-lock-key', blocking_timeout=5) as lock:
        # code you want executed only after the lock has been acquired
except LockError:
    # the lock wasn't acquired

API Reference

The official Redis command documentation does a great job of explaining each command in detail. redis-py attempts to adhere to the official command syntax. There are a few exceptions:

  • SELECT: Not implemented. See the explanation in the Thread Safety section below.
  • DEL: del is a reserved keyword in the Python syntax. Therefore redis-py uses delete instead.
  • MULTI/EXEC: These are implemented as part of the Pipeline class. The pipeline is wrapped with the MULTI and EXEC statements by default when it is executed, which can be disabled by specifying transaction=False. See more about Pipelines below.
  • SUBSCRIBE/LISTEN: Similar to pipelines, PubSub is implemented as a separate class as it places the underlying connection in a state where it can't execute non-pubsub commands. Calling the pubsub method from the Redis client will return a PubSub instance where you can subscribe to channels and listen for messages. You can only call PUBLISH from the Redis client (see this comment on issue #151 for details).
  • SCAN/SSCAN/HSCAN/ZSCAN: The *SCAN commands are implemented as they exist in the Redis documentation. In addition, each command has an equivalent iterator method. These are purely for convenience so the user doesn't have to keep track of the cursor while iterating. Use the scan_iter/sscan_iter/hscan_iter/zscan_iter methods for this behavior.

Connecting to Redis

Client Classes: Redis and StrictRedis

redis-py 3.0 drops support for the legacy Redis client class. StrictRedis has been renamed to Redis and an alias named StrictRedis is provided so that users previously using StrictRedis can continue to run unchanged.

The 2.X Redis class provided alternative implementations of a few commands. This confused users (rightfully so) and caused a number of support issues. To make things easier going forward, it was decided to drop support for these alternate implementations and instead focus on a single client class.

2.X users that are already using StrictRedis don't have to change the class name. StrictRedis will continue to work for the foreseeable future.

2.X users that are using the Redis class will have to make changes if they use any of the following commands:

  • SETEX: The argument order has changed. The new order is (name, time, value).
  • LREM: The argument order has changed. The new order is (name, num, value).
  • TTL and PTTL: The return value is now always an int and matches the official Redis command (>0 indicates the timeout, -1 indicates that the key exists but that it has no expire time set, -2 indicates that the key does not exist)

Connection Pools

Behind the scenes, redis-py uses a connection pool to manage connections to a Redis server. By default, each Redis instance you create will in turn create its own connection pool. You can override this behavior and use an existing connection pool by passing an already created connection pool instance to the connection_pool argument of the Redis class. You may choose to do this in order to implement client side sharding or have fine-grain control of how connections are managed.

>>> pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
>>> r = redis.Redis(connection_pool=pool)

Connections

ConnectionPools manage a set of Connection instances. redis-py ships with two types of Connections. The default, Connection, is a normal TCP socket based connection. The UnixDomainSocketConnection allows for clients running on the same device as the server to connect via a unix domain socket. To use a UnixDomainSocketConnection connection, simply pass the unix_socket_path argument, which is a string to the unix domain socket file. Additionally, make sure the unixsocket parameter is defined in your redis.conf file. It's commented out by default.

>>> r = redis.Redis(unix_socket_path='/tmp/redis.sock')

You can create your own Connection subclasses as well. This may be useful if you want to control the socket behavior within an async framework. To instantiate a client class using your own connection, you need to create a connection pool, passing your class to the connection_class argument. Other keyword parameters you pass to the pool will be passed to the class specified during initialization.

>>> pool = redis.ConnectionPool(connection_class=YourConnectionClass,
                                your_arg='...', ...)

Connections maintain an open socket to the Redis server. Sometimes these sockets are interrupted or disconnected for a variety of reasons. For example, network appliances, load balancers and other services that sit between clients and servers are often configured to kill connections that remain idle for a given threshold.

When a connection becomes disconnected, the next command issued on that connection will fail and redis-py will raise a ConnectionError to the caller. This allows each application that uses redis-py to handle errors in a way that's fitting for that specific application. However, constant error handling can be verbose and cumbersome, especially when socket disconnections happen frequently in many production environments.

To combat this, redis-py can issue regular health checks to assess the liveliness of a connection just before issuing a command. Users can pass health_check_interval=N to the Redis or ConnectionPool classes or as a query argument within a Redis URL. The value of health_check_interval must be an integer. A value of 0, the default, disables health checks. Any positive integer will enable health checks. Health checks are performed just before a command is executed if the underlying connection has been idle for more than health_check_interval seconds. For example, health_check_interval=30 will ensure that a health check is run on any connection that has been idle for 30 or more seconds just before a command is executed on that connection.

If your application is running in an environment that disconnects idle connections after 30 seconds you should set the health_check_interval option to a value less than 30.

This option also works on any PubSub connection that is created from a client with health_check_interval enabled. PubSub users need to ensure that get_message() or listen() are called more frequently than health_check_interval seconds. It is assumed that most workloads already do this.

If your PubSub use case doesn't call get_message() or listen() frequently, you should call pubsub.check_health() explicitly on a regularly basis.

SSL Connections

redis-py 3.0 changes the default value of the ssl_cert_reqs option from None to 'required'. See Issue 1016. This change enforces hostname validation when accepting a cert from a remote SSL terminator. If the terminator doesn't properly set the hostname on the cert this will cause redis-py 3.0 to raise a ConnectionError.

This check can be disabled by setting ssl_cert_reqs to None. Note that doing so removes the security check. Do so at your own risk.

Example with hostname verification using a local certificate bundle (linux):

>>> import redis
>>> r = redis.Redis(host='xxxxxx.cache.amazonaws.com', port=6379, db=0,
                    ssl=True,
                    ssl_ca_certs='/etc/ssl/certs/ca-certificates.crt')
>>> r.set('foo', 'bar')
True
>>> r.get('foo')
b'bar'

Example with hostname verification using certifi:

>>> import redis, certifi
>>> r = redis.Redis(host='xxxxxx.cache.amazonaws.com', port=6379, db=0,
                    ssl=True, ssl_ca_certs=certifi.where())
>>> r.set('foo', 'bar')
True
>>> r.get('foo')
b'bar'

Example turning off hostname verification (not recommended):

>>> import redis
>>> r = redis.Redis(host='xxxxxx.cache.amazonaws.com', port=6379, db=0,
                    ssl=True, ssl_cert_reqs=None)
>>> r.set('foo', 'bar')
True
>>> r.get('foo')
b'bar'

Sentinel support

redis-py can be used together with Redis Sentinel to discover Redis nodes. You need to have at least one Sentinel daemon running in order to use redis-py's Sentinel support.

Connecting redis-py to the Sentinel instance(s) is easy. You can use a Sentinel connection to discover the master and slaves network addresses:

>>> from redis import Sentinel
>>> sentinel = Sentinel([('localhost', 26379)], socket_timeout=0.1)
>>> sentinel.discover_master('mymaster')
('127.0.0.1', 6379)
>>> sentinel.discover_slaves('mymaster')
[('127.0.0.1', 6380)]

To connect to a sentinel which uses SSL (see SSL connections for more examples of SSL configurations):

>>> from redis import Sentinel
>>> sentinel = Sentinel([('localhost', 26379)],
                        ssl=True,
                        ssl_ca_certs='/etc/ssl/certs/ca-certificates.crt')
>>> sentinel.discover_master('mymaster')
('127.0.0.1', 6379)

You can also create Redis client connections from a Sentinel instance. You can connect to either the master (for write operations) or a slave (for read-only operations).

>>> master = sentinel.master_for('mymaster', socket_timeout=0.1)
>>> slave = sentinel.slave_for('mymaster', socket_timeout=0.1)
>>> master.set('foo', 'bar')
>>> slave.get('foo')
b'bar'

The master and slave objects are normal Redis instances with their connection pool bound to the Sentinel instance. When a Sentinel backed client attempts to establish a connection, it first queries the Sentinel servers to determine an appropriate host to connect to. If no server is found, a MasterNotFoundError or SlaveNotFoundError is raised. Both exceptions are subclasses of ConnectionError.

When trying to connect to a slave client, the Sentinel connection pool will iterate over the list of slaves until it finds one that can be connected to. If no slaves can be connected to, a connection will be established with the master.

See Guidelines for Redis clients with support for Redis Sentinel to learn more about Redis Sentinel.


Parsers

Parser classes provide a way to control how responses from the Redis server are parsed. redis-py ships with two parser classes, the PythonParser and the HiredisParser. By default, redis-py will attempt to use the HiredisParser if you have the hiredis module installed and will fallback to the PythonParser otherwise.

Hiredis is a C library maintained by the core Redis team. Pieter Noordhuis was kind enough to create Python bindings. Using Hiredis can provide up to a 10x speed improvement in parsing responses from the Redis server. The performance increase is most noticeable when retrieving many pieces of data, such as from LRANGE or SMEMBERS operations.

Hiredis is available on PyPI, and can be installed via pip just like redis-py.

$ pip install hiredis

Response Callbacks

The client class uses a set of callbacks to cast Redis responses to the appropriate Python type. There are a number of these callbacks defined on the Redis client class in a dictionary called RESPONSE_CALLBACKS.

Custom callbacks can be added on a per-instance basis using the set_response_callback method. This method accepts two arguments: a command name and the callback. Callbacks added in this manner are only valid on the instance the callback is added to. If you want to define or override a callback globally, you should make a subclass of the Redis client and add your callback to its RESPONSE_CALLBACKS class dictionary.

Response callbacks take at least one parameter: the response from the Redis server. Keyword arguments may also be accepted in order to further control how to interpret the response. These keyword arguments are specified during the command's call to execute_command. The ZRANGE implementation demonstrates the use of response callback keyword arguments with its "withscores" argument.

Thread Safety

Redis client instances can safely be shared between threads. Internally, connection instances are only retrieved from the connection pool during command execution, and returned to the pool directly after. Command execution never modifies state on the client instance.

However, there is one caveat: the Redis SELECT command. The SELECT command allows you to switch the database currently in use by the connection. That database remains selected until another is selected or until the connection is closed. This creates an issue in that connections could be returned to the pool that are connected to a different database.

As a result, redis-py does not implement the SELECT command on client instances. If you use multiple Redis databases within the same application, you should create a separate client instance (and possibly a separate connection pool) for each database.

It is not safe to pass PubSub or Pipeline objects between threads.

Pipelines

Pipelines are a subclass of the base Redis class that provide support for buffering multiple commands to the server in a single request. They can be used to dramatically increase the performance of groups of commands by reducing the number of back-and-forth TCP packets between the client and server.

Pipelines are quite simple to use:

>>> r = redis.Redis(...)
>>> r.set('bing', 'baz')
>>> # Use the pipeline() method to create a pipeline instance
>>> pipe = r.pipeline()
>>> # The following SET commands are buffered
>>> pipe.set('foo', 'bar')
>>> pipe.get('bing')
>>> # the EXECUTE call sends all buffered commands to the server, returning
>>> # a list of responses, one for each command.
>>> pipe.execute()
[True, b'baz']

For ease of use, all commands being buffered into the pipeline return the pipeline object itself. Therefore calls can be chained like:

>>> pipe.set('foo', 'bar').sadd('faz', 'baz').incr('auto_number').execute()
[True, True, 6]

In addition, pipelines can also ensure the buffered commands are executed atomically as a group. This happens by default. If you want to disable the atomic nature of a pipeline but still want to buffer commands, you can turn off transactions.

>>> pipe = r.pipeline(transaction=False)

A common issue occurs when requiring atomic transactions but needing to retrieve values in Redis prior for use within the transaction. For instance, let's assume that the INCR command didn't exist and we need to build an atomic version of INCR in Python.

The completely naive implementation could GET the value, increment it in Python, and SET the new value back. However, this is not atomic because multiple clients could be doing this at the same time, each getting the same value from GET.

Enter the WATCH command. WATCH provides the ability to monitor one or more keys prior to starting a transaction. If any of those keys change prior the execution of that transaction, the entire transaction will be canceled and a WatchError will be raised. To implement our own client-side INCR command, we could do something like this:

>>> with r.pipeline() as pipe:
...     while True:
...         try:
...             # put a WATCH on the key that holds our sequence value
...             pipe.watch('OUR-SEQUENCE-KEY')
...             # after WATCHing, the pipeline is put into immediate execution
...             # mode until we tell it to start buffering commands again.
...             # this allows us to get the current value of our sequence
...             current_value = pipe.get('OUR-SEQUENCE-KEY')
...             next_value = int(current_value) + 1
...             # now we can put the pipeline back into buffered mode with MULTI
...             pipe.multi()
...             pipe.set('OUR-SEQUENCE-KEY', next_value)
...             # and finally, execute the pipeline (the set command)
...             pipe.execute()
...             # if a WatchError wasn't raised during execution, everything
...             # we just did happened atomically.
...             break
...        except WatchError:
...             # another client must have changed 'OUR-SEQUENCE-KEY' between
...             # the time we started WATCHing it and the pipeline's execution.
...             # our best bet is to just retry.
...             continue

Note that, because the Pipeline must bind to a single connection for the duration of a WATCH, care must be taken to ensure that the connection is returned to the connection pool by calling the reset() method. If the Pipeline is used as a context manager (as in the example above) reset() will be called automatically. Of course you can do this the manual way by explicitly calling reset():

>>> pipe = r.pipeline()
>>> while True:
...     try:
...         pipe.watch('OUR-SEQUENCE-KEY')
...         ...
...         pipe.execute()
...         break
...     except WatchError:
...         continue
...     finally:
...         pipe.reset()

A convenience method named "transaction" exists for handling all the boilerplate of handling and retrying watch errors. It takes a callable that should expect a single parameter, a pipeline object, and any number of keys to be WATCHed. Our client-side INCR command above can be written like this, which is much easier to read:

>>> def client_side_incr(pipe):
...     current_value = pipe.get('OUR-SEQUENCE-KEY')
...     next_value = int(current_value) + 1
...     pipe.multi()
...     pipe.set('OUR-SEQUENCE-KEY', next_value)
>>>
>>> r.transaction(client_side_incr, 'OUR-SEQUENCE-KEY')
[True]

Be sure to call pipe.multi() in the callable passed to Redis.transaction prior to any write commands.

Publish / Subscribe

redis-py includes a PubSub object that subscribes to channels and listens for new messages. Creating a PubSub object is easy.

>>> r = redis.Redis(...)
>>> p = r.pubsub()

Once a PubSub instance is created, channels and patterns can be subscribed to.

>>> p.subscribe('my-first-channel', 'my-second-channel', ...)
>>> p.psubscribe('my-*', ...)

The PubSub instance is now subscribed to those channels/patterns. The subscription confirmations can be seen by reading messages from the PubSub instance.

>>> p.get_message()
{'pattern': None, 'type': 'subscribe', 'channel': b'my-second-channel', 'data': 1}
>>> p.get_message()
{'pattern': None, 'type': 'subscribe', 'channel': b'my-first-channel', 'data': 2}
>>> p.get_message()
{'pattern': None, 'type': 'psubscribe', 'channel': b'my-*', 'data': 3}

Every message read from a PubSub instance will be a dictionary with the following keys.

  • type: One of the following: 'subscribe', 'unsubscribe', 'psubscribe', 'punsubscribe', 'message', 'pmessage'
  • channel: The channel [un]subscribed to or the channel a message was published to
  • pattern: The pattern that matched a published message's channel. Will be None in all cases except for 'pmessage' types.
  • data: The message data. With [un]subscribe messages, this value will be the number of channels and patterns the connection is currently subscribed to. With [p]message messages, this value will be the actual published message.

Let's send a message now.

# the publish method returns the number matching channel and pattern
# subscriptions. 'my-first-channel' matches both the 'my-first-channel'
# subscription and the 'my-*' pattern subscription, so this message will
# be delivered to 2 channels/patterns
>>> r.publish('my-first-channel', 'some data')
2
>>> p.get_message()
{'channel': b'my-first-channel', 'data': b'some data', 'pattern': None, 'type': 'message'}
>>> p.get_message()
{'channel': b'my-first-channel', 'data': b'some data', 'pattern': b'my-*', 'type': 'pmessage'}

Unsubscribing works just like subscribing. If no arguments are passed to [p]unsubscribe, all channels or patterns will be unsubscribed from.

>>> p.unsubscribe()
>>> p.punsubscribe('my-*')
>>> p.get_message()
{'channel': b'my-second-channel', 'data': 2, 'pattern': None, 'type': 'unsubscribe'}
>>> p.get_message()
{'channel': b'my-first-channel', 'data': 1, 'pattern': None, 'type': 'unsubscribe'}
>>> p.get_message()
{'channel': b'my-*', 'data': 0, 'pattern': None, 'type': 'punsubscribe'}

redis-py also allows you to register callback functions to handle published messages. Message handlers take a single argument, the message, which is a dictionary just like the examples above. To subscribe to a channel or pattern with a message handler, pass the channel or pattern name as a keyword argument with its value being the callback function.

When a message is read on a channel or pattern with a message handler, the message dictionary is created and passed to the message handler. In this case, a None value is returned from get_message() since the message was already handled.

>>> def my_handler(message):
...     print('MY HANDLER: ', message['data'])
>>> p.subscribe(**{'my-channel': my_handler})
# read the subscribe confirmation message
>>> p.get_message()
{'pattern': None, 'type': 'subscribe', 'channel': b'my-channel', 'data': 1}
>>> r.publish('my-channel', 'awesome data')
1
# for the message handler to work, we need tell the instance to read data.
# this can be done in several ways (read more below). we'll just use
# the familiar get_message() function for now
>>> message = p.get_message()
MY HANDLER:  awesome data
# note here that the my_handler callback printed the string above.
# `message` is None because the message was handled by our handler.
>>> print(message)
None

If your application is not interested in the (sometimes noisy) subscribe/unsubscribe confirmation messages, you can ignore them by passing ignore_subscribe_messages=True to r.pubsub(). This will cause all subscribe/unsubscribe messages to be read, but they won't bubble up to your application.

>>> p = r.pubsub(ignore_subscribe_messages=True)
>>> p.subscribe('my-channel')
>>> p.get_message()  # hides the subscribe message and returns None
>>> r.publish('my-channel', 'my data')
1
>>> p.get_message()
{'channel': b'my-channel', 'data': b'my data', 'pattern': None, 'type': 'message'}

There are three different strategies for reading messages.

The examples above have been using pubsub.get_message(). Behind the scenes, get_message() uses the system's 'select' module to quickly poll the connection's socket. If there's data available to be read, get_message() will read it, format the message and return it or pass it to a message handler. If there's no data to be read, get_message() will immediately return None. This makes it trivial to integrate into an existing event loop inside your application.

>>> while True:
>>>     message = p.get_message()
>>>     if message:
>>>         # do something with the message
>>>     time.sleep(0.001)  # be nice to the system :)

Older versions of redis-py only read messages with pubsub.listen(). listen() is a generator that blocks until a message is available. If your application doesn't need to do anything else but receive and act on messages received from redis, listen() is an easy way to get up an running.

>>> for message in p.listen():
...     # do something with the message

The third option runs an event loop in a separate thread. pubsub.run_in_thread() creates a new thread and starts the event loop. The thread object is returned to the caller of [un_in_thread(). The caller can use the thread.stop() method to shut down the event loop and thread. Behind the scenes, this is simply a wrapper around get_message() that runs in a separate thread, essentially creating a tiny non-blocking event loop for you. run_in_thread() takes an optional sleep_time argument. If specified, the event loop will call time.sleep() with the value in each iteration of the loop.

Note: Since we're running in a separate thread, there's no way to handle messages that aren't automatically handled with registered message handlers. Therefore, redis-py prevents you from calling run_in_thread() if you're subscribed to patterns or channels that don't have message handlers attached.

>>> p.subscribe(**{'my-channel': my_handler})
>>> thread = p.run_in_thread(sleep_time=0.001)
# the event loop is now running in the background processing messages
# when it's time to shut it down...
>>> thread.stop()

run_in_thread also supports an optional exception handler, which lets you catch exceptions that occur within the worker thread and handle them appropriately. The exception handler will take as arguments the exception itself, the pubsub object, and the worker thread returned by run_in_thread.

>>> p.subscribe(**{'my-channel': my_handler})
>>> def exception_handler(ex, pubsub, thread):
>>>     print(ex)
>>>     thread.stop()
>>>     thread.join(timeout=1.0)
>>>     pubsub.close()
>>> thread = p.run_in_thread(exception_handler=exception_handler)

A PubSub object adheres to the same encoding semantics as the client instance it was created from. Any channel or pattern that's unicode will be encoded using the charset specified on the client before being sent to Redis. If the client's decode_responses flag is set the False (the default), the 'channel', 'pattern' and 'data' values in message dictionaries will be byte strings (str on Python 2, bytes on Python 3). If the client's decode_responses is True, then the 'channel', 'pattern' and 'data' values will be automatically decoded to unicode strings using the client's charset.

PubSub objects remember what channels and patterns they are subscribed to. In the event of a disconnection such as a network error or timeout, the PubSub object will re-subscribe to all prior channels and patterns when reconnecting. Messages that were published while the client was disconnected cannot be delivered. When you're finished with a PubSub object, call its .close() method to shutdown the connection.

>>> p = r.pubsub()
>>> ...
>>> p.close()

The PUBSUB set of subcommands CHANNELS, NUMSUB and NUMPAT are also supported:

>>> r.pubsub_channels()
[b'foo', b'bar']
>>> r.pubsub_numsub('foo', 'bar')
[(b'foo', 9001), (b'bar', 42)]
>>> r.pubsub_numsub('baz')
[(b'baz', 0)]
>>> r.pubsub_numpat()
1204

Monitor

redis-py includes a Monitor object that streams every command processed by the Redis server. Use listen() on the Monitor object to block until a command is received.

>>> r = redis.Redis(...)
>>> with r.monitor() as m:
>>>     for command in m.listen():
>>>         print(command)

Lua Scripting

redis-py supports the EVAL, EVALSHA, and SCRIPT commands. However, there are a number of edge cases that make these commands tedious to use in real world scenarios. Therefore, redis-py exposes a Script object that makes scripting much easier to use.

To create a Script instance, use the register_script function on a client instance passing the Lua code as the first argument. register_script returns a Script instance that you can use throughout your code.

The following trivial Lua script accepts two parameters: the name of a key and a multiplier value. The script fetches the value stored in the key, multiplies it with the multiplier value and returns the result.

>> multiply = r.register_script(lua) ">
>>> r = redis.Redis()
>>> lua = """
... local value = redis.call('GET', KEYS[1])
... value = tonumber(value)
... return value * ARGV[1]"""
>>> multiply = r.register_script(lua)

multiply is now a Script instance that is invoked by calling it like a function. Script instances accept the following optional arguments:

  • keys: A list of key names that the script will access. This becomes the KEYS list in Lua.
  • args: A list of argument values. This becomes the ARGV list in Lua.
  • client: A redis-py Client or Pipeline instance that will invoke the script. If client isn't specified, the client that initially created the Script instance (the one that register_script was invoked from) will be used.

Continuing the example from above:

>>> r.set('foo', 2)
>>> multiply(keys=['foo'], args=[5])
10

The value of key 'foo' is set to 2. When multiply is invoked, the 'foo' key is passed to the script along with the multiplier value of 5. Lua executes the script and returns the result, 10.

Script instances can be executed using a different client instance, even one that points to a completely different Redis server.

>>> r2 = redis.Redis('redis2.example.com')
>>> r2.set('foo', 3)
>>> multiply(keys=['foo'], args=[5], client=r2)
15

The Script object ensures that the Lua script is loaded into Redis's script cache. In the event of a NOSCRIPT error, it will load the script and retry executing it.

Script objects can also be used in pipelines. The pipeline instance should be passed as the client argument when calling the script. Care is taken to ensure that the script is registered in Redis's script cache just prior to pipeline execution.

>>> pipe = r.pipeline()
>>> pipe.set('foo', 5)
>>> multiply(keys=['foo'], args=[5], client=pipe)
>>> pipe.execute()
[True, 25]

Scan Iterators

The *SCAN commands introduced in Redis 2.8 can be cumbersome to use. While these commands are fully supported, redis-py also exposes the following methods that return Python iterators for convenience: scan_iter, hscan_iter, sscan_iter and zscan_iter.

>>> for key, value in (('A', '1'), ('B', '2'), ('C', '3')):
...     r.set(key, value)
>>> for key in r.scan_iter():
...     print(key, r.get(key))
A 1
B 2
C 3

Cluster Mode

redis-py is now supports cluster mode and provides a client for Redis Cluster.

The cluster client is based on Grokzen's redis-py-cluster, has added bug fixes, and now supersedes that library. Support for these changes is thanks to his contributions.

Create RedisCluster:

Connecting redis-py to a Redis Cluster instance(s) requires at a minimum a single node for cluster discovery. There are multiple ways in which a cluster instance can be created:

  • Using 'host' and 'port' arguments:
    >>> from redis.cluster import RedisCluster as Redis
    >>> rc = Redis(host='localhost', port=6379)
    >>> print(rc.get_nodes())
    [[host=127.0.0.1,port=6379,name=127.0.0.1:6379,server_type=primary,redis_connection=Redis
   
    
     >>], [host=127.0.0.1,port=6378,name=127.0.0.1:6378,server_type=primary,redis_connection=Redis
     
      
       
        >>], [host=127.0.0.1,port=6377,name=127.0.0.1:6377,server_type=replica,redis_connection=Redis
        
         
          
           >>]]
          
         
        
       
      
     
    
   
  
  • Using the Redis URL specification:
    >>> from redis.cluster import RedisCluster as Redis
    >>> rc = Redis.from_url("redis://localhost:6379/0")
  • Directly, via the ClusterNode class:
    >>> from redis.cluster import RedisCluster as Redis
    >>> from redis.cluster import ClusterNode
    >>> nodes = [ClusterNode('localhost', 6379), ClusterNode('localhost', 6378)]
    >>> rc = Redis(startup_nodes=nodes)

When a RedisCluster instance is being created it first attempts to establish a connection to one of the provided startup nodes. If none of the startup nodes are reachable, a 'RedisClusterException' will be thrown. After a connection to the one of the cluster's nodes is established, the RedisCluster instance will be initialized with 3 caches: a slots cache which maps each of the 16384 slots to the node/s handling them, a nodes cache that contains ClusterNode objects (name, host, port, redis connection) for all of the cluster's nodes, and a commands cache contains all the server supported commands that were retrieved using the Redis 'COMMAND' output.

RedisCluster instance can be directly used to execute Redis commands. When a command is being executed through the cluster instance, the target node(s) will be internally determined. When using a key-based command, the target node will be the node that holds the key's slot. Cluster management commands and other commands that are not key-based have a parameter called 'target_nodes' where you can specify which nodes to execute the command on. In the absence of target_nodes, the command will be executed on the default cluster node. As part of cluster instance initialization, the cluster's default node is randomly selected from the cluster's primaries, and will be updated upon reinitialization. Using r.get_default_node(), you can get the cluster's default node, or you can change it using the 'set_default_node' method.

The 'target_nodes' parameter is explained in the following section, 'Specifying Target Nodes'.

    >>> # target-nodes: the node that holds 'foo1's key slot
    >>> rc.set('foo1', 'bar1')
    >>> # target-nodes: the node that holds 'foo2's key slot
    >>> rc.set('foo2', 'bar2')
    >>> # target-nodes: the node that holds 'foo1's key slot
    >>> print(rc.get('foo1'))
    b'bar'
    >>> # target-node: default-node
    >>> print(rc.keys())
    [b'foo1']
    >>> # target-node: default-node
    >>> rc.ping()

Specifying Target Nodes:

As mentioned above, all non key-based RedisCluster commands accept the kwarg parameter 'target_nodes' that specifies the node/nodes that the command should be executed on. The best practice is to specify target nodes using RedisCluster class's node flags: PRIMARIES, REPLICAS, ALL_NODES, RANDOM. When a nodes flag is passed along with a command, it will be internally resolved to the relevant node/s. If the nodes topology of the cluster changes during the execution of a command, the client will be able to resolve the nodes flag again with the new topology and attempt to retry executing the command.

    >>> from redis.cluster import RedisCluster as Redis
    >>> # run cluster-meet command on all of the cluster's nodes
    >>> rc.cluster_meet('127.0.0.1', 6379, target_nodes=Redis.ALL_NODES)
    >>> # ping all replicas
    >>> rc.ping(target_nodes=Redis.REPLICAS)
    >>> # ping a random node
    >>> rc.ping(target_nodes=Redis.RANDOM)
    >>> # get the keys from all cluster nodes
    >>> rc.keys(target_nodes=Redis.ALL_NODES)
    [b'foo1', b'foo2']
    >>> # execute bgsave in all primaries
    >>> rc.bgsave(Redis.PRIMARIES)

You could also pass ClusterNodes directly if you want to execute a command on a specific node / node group that isn't addressed by the nodes flag. However, if the command execution fails due to cluster topology changes, a retry attempt will not be made, since the passed target node/s may no longer be valid, and the relevant cluster or connection error will be returned.

    >>> node = rc.get_node('localhost', 6379)
    >>> # Get the keys only for that specific node
    >>> rc.keys(target_nodes=node)
    >>> # get Redis info from a subset of primaries
    >>> subset_primaries = [node for node in rc.get_primaries() if node.port > 6378]
    >>> rc.info(target_nodes=subset_primaries)

In addition, the RedisCluster instance can query the Redis instance of a specific node and execute commands on that node directly. The Redis client, however, does not handle cluster failures and retries.

    >>> cluster_node = rc.get_node(host='localhost', port=6379)
    >>> print(cluster_node)
    [host=127.0.0.1,port=6379,name=127.0.0.1:6379,server_type=primary,redis_connection=Redis
   
    
     >>]
    >>> r = cluster_node.redis_connection
    >>> r.client_list()
    [{'id': '276', 'addr': '127.0.0.1:64108', 'fd': '16', 'name': '', 'age': '0', 'idle': '0', 'flags': 'N', 'db': '0', 'sub': '0', 'psub': '0', 'multi': '-1', 'qbuf': '26', 'qbuf-free': '32742', 'argv-mem': '10', 'obl': '0', 'oll': '0', 'omem': '0', 'tot-mem': '54298', 'events': 'r', 'cmd': 'client', 'user': 'default'}]
    >>> # Get the keys only for that specific node
    >>> r.keys()
    [b'foo1']
    
   
  

Multi-key commands:

Redis supports multi-key commands in Cluster Mode, such as Set type unions or intersections, mset and mget, as long as the keys all hash to the same slot. By using RedisCluster client, you can use the known functions (e.g. mget, mset) to perform an atomic multi-key operation. However, you must ensure all keys are mapped to the same slot, otherwise a RedisClusterException will be thrown. Redis Cluster implements a concept called hash tags that can be used in order to force certain keys to be stored in the same hash slot, see Keys hash tag. You can also use nonatomic for some of the multikey operations, and pass keys that aren't mapped to the same slot. The client will then map the keys to the relevant slots, sending the commands to the slots' node owners. Non-atomic operations batch the keys according to their hash value, and then each batch is sent separately to the slot's owner.

    #  Atomic operations can be used when all keys are mapped to the same slot
    >>> rc.mset({'{foo}1': 'bar1', '{foo}2': 'bar2'})
    >>> rc.mget('{foo}1', '{foo}2')
    [b'bar1', b'bar2']
    # Non-atomic multi-key operations splits the keys into different slots
    >>> rc.mset_nonatomic({'foo': 'value1', 'bar': 'value2', 'zzz': 'value3')
    >>> rc.mget_nonatomic('foo', 'bar', 'zzz')
    [b'value1', b'value2', b'value3']

Cluster PubSub:

When a ClusterPubSub instance is created without specifying a node, a single node will be transparently chosen for the pubsub connection on the first command execution. The node will be determined by:

  1. Hashing the channel name in the request to find its keyslot
  2. Selecting a node that handles the keyslot: If read_from_replicas is set to true, a replica can be selected.

Known limitations with pubsub:

Pattern subscribe and publish do not currently work properly due to key slots. If we hash a pattern like fo* we will receive a keyslot for that string but there are endless possibilities for channel names based on this pattern - unknowable in advance. This feature is not disabled but the commands are not currently recommended for use. See redis-py-cluster documentation for more.

    >>> p1 = rc.pubsub()
    # p1 connection will be set to the node that holds 'foo' keyslot
    >>> p1.subscribe('foo')
    # p2 connection will be set to node 'localhost:6379'
    >>> p2 = rc.pubsub(rc.get_node('localhost', 6379))

Read Only Mode

By default, Redis Cluster always returns MOVE redirection response on accessing a replica node. You can overcome this limitation and scale read commands by triggering READONLY mode.

To enable READONLY mode pass read_from_replicas=True to RedisCluster constructor. When set to true, read commands will be assigned between the primary and its replications in a Round-Robin manner.

READONLY mode can be set at runtime by calling the readonly() method with target_nodes='replicas', and read-write access can be restored by calling the readwrite() method.

    >>> from cluster import RedisCluster as Redis
    # Use 'debug' log level to print the node that the command is executed on
    >>> rc_readonly = Redis(startup_nodes=startup_nodes,
                    read_from_replicas=True)
    >>> rc_readonly.set('{foo}1', 'bar1')
    >>> for i in range(0, 4):
            # Assigns read command to the slot's hosts in a Round-Robin manner
    >>>     rc_readonly.get('{foo}1')
    # set command would be directed only to the slot's primary node
    >>> rc_readonly.set('{foo}2', 'bar2')
    # reset READONLY flag
    >>> rc_readonly.readwrite(target_nodes='replicas')
    # now the get command would be directed only to the slot's primary node
    >>> rc_readonly.get('{foo}1')

Cluster Pipeline

ClusterPipeline is a subclass of RedisCluster that provides support for Redis pipelines in cluster mode. When calling the execute() command, all the commands are grouped by the node on which they will be executed, and are then executed by the respective nodes in parallel. The pipeline instance will wait for all the nodes to respond before returning the result to the caller. Command responses are returned as a list sorted in the same order in which they were sent. Pipelines can be used to dramatically increase the throughput of Redis Cluster by significantly reducing the the number of network round trips between the client and the server.

    >>> with rc.pipeline() as pipe:
    >>>     pipe.set('foo', 'value1')
    >>>     pipe.set('bar', 'value2')
    >>>     pipe.get('foo')
    >>>     pipe.get('bar')
    >>>     print(pipe.execute())
    [True, True, b'value1', b'value2']
    >>>     pipe.set('foo1', 'bar1').get('foo1').execute()
    [True, b'bar1']

Please note:

  • RedisCluster pipelines currently only support key-based commands.
  • The pipeline gets its 'read_from_replicas' value from the cluster's parameter. Thus, if read from replications is enabled in the cluster instance, the pipeline will also direct read commands to replicas.
  • The 'transcation' option is NOT supported in cluster-mode. In non-cluster mode, the 'transaction' option is available when executing pipelines. This wraps the pipeline commands with MULTI/EXEC commands, and effectively turns the pipeline commands into a single transaction block. This means that all commands are executed sequentially without any interruptions from other clients. However, in cluster-mode this is not possible, because commands are partitioned according to their respective destination nodes. This means that we can not turn the pipeline commands into one transaction block, because in most cases they are split up into several smaller pipelines.

See Redis Cluster tutorial and Redis Cluster specifications to learn more about Redis Cluster.

Author

redis-py is developed and maintained by Redis Inc. It can be found here, or downloaded from pypi.

Special thanks to:

  • Andy McCurdy ([email protected]) the original author of redis-py.
  • Ludovico Magnocavallo, author of the original Python Redis client, from which some of the socket code is still used.
  • Alexander Solovyov for ideas on the generic response callback system.
  • Paul Hubbard for initial packaging support.

Redis

Comments
  • 3.2.0 Error while reading from socket: ('Connection closed by server.',)

    3.2.0 Error while reading from socket: ('Connection closed by server.',)

    Version: Python: 3.6.7 Redis: 3.2.7 (Azure Redis) Redis-py: 3.2.0 Django: 2.1.1

    Description: Hi Experts,

    Our service met a similar issue with the issue #1127 3.1.0 causing intermittent connection closed by server error. By reviewing the whole discussion of the issue #1127, we upgraded redis-py to the version 3.2.0 and the issue has been mitigated but still happening. Due to the Azure Redis server will close the connections which are idle more than 10 mins, and the default redis-py behavior is to not close connections, recycling them when possible, could you please suggest how to avoid the exception "Redis ConnectionError: Error while reading from socket: ('Connection closed by server.',)" on our product?

    Configuration settings: {'default': {'BACKEND': 'django_redis.cache.RedisCache', 'LOCATION': 'redis://xx.x.x.xxx:6379/0', 'TIMEOUT': 60, 'OPTIONS': {'DB': 0, 'SOCKET_TIMEOUT': 120, 'SOCKET_CONNECT_TIMEOUT': 30, 'COMPRESSOR': 'django_redis.compressors.zlib.ZlibCompressor', 'IGNORE_EXCEPTIONS': True, 'REDIS_CLIENT_KWARGS': {'socket_keepalive': True}, 'PASSWORD': 'xxxxxxxxxxxxxxx='}}, 'cachalot': {'BACKEND': 'django_redis.cache.RedisCache', 'LOCATION': 'redis://xx.x.x.xxx:6379/1', 'TIMEOUT': 60, 'OPTIONS': {'DB': 1, 'SOCKET_TIMEOUT': 120, 'SOCKET_CONNECT_TIMEOUT': 30, 'COMPRESSOR': 'django_redis.compressors.zlib.ZlibCompressor', 'IGNORE_EXCEPTIONS': True, 'REDIS_CLIENT_KWARGS': {'socket_keepalive': True}, 'PASSWORD': 'xxxxxxxxxxxxxxxxxxxxxx='}}, }

    Error Message: Redis ConnectionError: Error while reading from socket: ('Connection closed by server.',)

    Thanks a lot.

    opened by LucyWengCSS 55
  • redis-py client side hang; is there a stale or timeout re-connect paramter?

    redis-py client side hang; is there a stale or timeout re-connect paramter?

    can reproduce this from a simple python shell, make a redis connection and let it idle for some time (some hours or I don't know what's the actual server side timeout), try a get all keys call, or any other call, it takes 930 seconds to return, or 15 minutes, this is a simple test server only have a few small keys, and get all keys by normal is just 0.003 second, so I suppose it's the idling has caused either side closed the connection, but why can't the client side detect it earlier, raise an exception, or do something earlier? I tried normal connection or ConnectionPool doesn't help;

    In [18]: r = redis.Redis(connection_pool=redis.ConnectionPool(host='redis-server-...', port=6379, db='0'))
    
    ## idle for some hours
    
    In [37]: start = datetime.now(); print r.keys(); end = datetime.now(); print "{:.3f} seconds".format((end - start).total_seconds())
    # printed keys...
    930.306 seconds
    

    I'm reading the document https://pypi.python.org/pypi/redis don't see an stale option or client side timeout option?

    https://github.com/andymccurdy/redis-py/issues?q=hang I have searched this there are many hang problems, however I am not seeing a duplicate

    opened by c0b 39
  • Random ConnectionErrors (104, Connection reset by peer)

    Random ConnectionErrors (104, Connection reset by peer)

    Version: redis-py: 3.2.1 redis server: 5.0.5

    Platform: Debian

    Description: I have a connection to the Server that is idle for a few hours, and when I try to perform an operation, I get a redis.exceptions.ConnectionError: Error while reading from socket: (104, 'Connection reset by peer'). This happens not always but often enough to be annoying. I can wrap it in a try-except and just reconnect and it will work, however i don't want to wrap every command to redis like this.

    Some Background: The program is getting tasks from a Queue implemented in Redis, computing for a few hours, getting the next task, and so on.

    The question is, is this a bug or expected? Is there an option to prevent this? I don't have any timeouts set on the client or server (everything is default). The current workaround would be to emulate a pre-ping, by sending a ping wrapped in a try-except-and-reconnect-if-needed, but it's not very pretty.

    opened by venthur 35
  • 3.1.0 causing intermittent Connection closed by server error

    3.1.0 causing intermittent Connection closed by server error

    Version: redis-py: 3.1.0 redis: 3.2.4 django-redis: 4.10.0

    Platform: Python 2.7 on Alpine-Linux inside Docker

    Description: After upgrading from redis-py 3.0.1, our service becomes very unstable talking to the existing redis server. It generates around 30 'Connection closed by server.' errors in 10 minutes while the server is under ~20 QPS. The error is intermittent and I am not able to reproduce what exactly caused the errors. I tried restarting the redis server, rebuild our Docker images without any cache, and none of them worked.

    After rolling back to redis-py==3.0.1, all errors are gone.

    I understand that I don't really provide enough information to fix the problem, but I hope to at least highlight this problem and others might provide more.

    Errors

    File "lib/last_seen/models.py" in user_seen
      96.     seen = cache.get(cache_key)
    
    File "/usr/lib/python2.7/site-packages/django_redis/cache.py" in _decorator
      39.             raise e.parent
    
    Exception Type: ConnectionError at /helper/listing/list_591.5917864/
    Exception Value: Error while reading from socket: (u'Connection closed by server.',)
    
    opened by andyfoundi 35
  • redis.pipeline = no real pipeline to redis. Thoughts?

    redis.pipeline = no real pipeline to redis. Thoughts?

    According to my knowledge (which might be wrong or incomplete), the redis.pipeline object doesn't pipe/stream to redis. It cumulates the data internally in python's private memory, and this queue as a whole is only communicated to redis after calling pipeline.execute().

    This causes unwanted spikes in network traffic, as well as CPU spikes in Redis. Both with possible concurrent performance issues.

    Another issue is, having to flush manually (using execute) every N records, otherwise I run out of Python's stack space.

    Yet another issue is, that if the client crashes, all queued data is lost. I prefer it streamed to redis.

    When I look at (for example) C# ServiceStack.Redis, pipelined data does seem to be streamed continuously (back and forth).

    What are your thoughts? Is it hard to implement this as a Thread or greenlet? Should it be an attribute of the redis.pipeline object itself?

    Kind regards, TW

    Stale 
    opened by tw-bert 34
  • Race condition in connection disconnect

    Race condition in connection disconnect

    It seems like there is a race condition in the code for Connection.disconnect(). I run redis as a celery broker, so I seem to be running into a situation where I get a TypeError: NoneType has no attribute _close(). It would appear that in one process disconnect() is being called and setting the self._sock to None while another process has already executed the if statement that checks if self._sock is None.

        def disconnect(self):
            "Disconnects from the Redis server"
            self._parser.on_disconnect()
            if self._sock is None:
                return
            try:
                self._sock.shutdown(socket.SHUT_RDWR)
                self._sock.close()
            except socket.error:
                pass
            self._sock = None
    
    opened by jpulec 32
  • python redis error

    python redis error

    import redis r = redis.StrictRedis(host='localhost', port=6379, db=0)

    as instructed in the readme file, I get this error:

    File "", line 1, in
    AttributeError: 'module' object has no attribute 'StrictRedis'.
    just begining redis, i have this error, do have please any idea?
    
    opened by elastic10 32
  • hold connection for duration of WATCH

    hold connection for duration of WATCH

    The first commit causes watch() to make the connection become 'sticky' until pipeline() or unwatch() is executed. It preserves existing calls, only adding an optional keyword argument for constructing the PubSub and Pipeline classes. It is still safe to share the Redis client for normal operation, but using watch() will mean taking care to not share blindly. After looking at the code for a long time I think I'm happiest with this compromise.

    Second commit modifies the test slightly to make sure that starting a pipeline() after watch() frees the Redis object for immediate re-use.

    opened by tilgovi 32
  • Compatibility For Python3

    Compatibility For Python3

    I've completed the changes required to have redis-py work with python 2 and python 3. The only issue outstanding is that hiredis-py will not work with python 3. I have begun that port and made some good progress, but there are still a few unfinished items. I'm not sure its a blocker for us.

    opened by dcolish 31
  • SPOP Count?

    SPOP Count?

    Hey andy/community,

    Redis supports a count field when calling SPOP (http://redis.io/commands/spop), but I don't think this framework has support for that.

    Not sure if this was intentional (for some odd reason of which I'm unaware) or not, but it'd be useful to have.

    I copy and pasted my patch below if you want to use it. I basically just duplicated the code for srandmember since even Redis docs mention they work the same. Although, it would've been nice to have the arg be count and not number.

    diff --git a/redis/client.py b/redis/client.py
    index 8b5a3fa..0f5ad93 100755
    --- a/redis/client.py
    +++ b/redis/client.py
    @@ -1532,9 +1532,17 @@ class StrictRedis(object):
             "Move ``value`` from set ``src`` to set ``dst`` atomically"
             return self.execute_command('SMOVE', src, dst, value)
    
    -    def spop(self, name):
    -        "Remove and return a random member of set ``name``"
    -        return self.execute_command('SPOP', name)
    +    def spop(self, name, number=None):
    +        """
    +        Remove and return a random member of set ``name``"
    +
    +        If ``number`` is None, pops a random member of set ``name``.
    +
    +        If ``number`` is supplied, pops a list of ``number`` random
    +        members of set ``name``.
    +        """
    +        args = number and [number] or []
    +        return self.execute_command('SPOP', name, *args)
    
         def srandmember(self, name, number=None):
             """
    diff --git a/tests/test_commands.py b/tests/test_commands.py
    index 286ea04..5c43feb 100644
    --- a/tests/test_commands.py
    +++ b/tests/test_commands.py
    @@ -844,6 +844,13 @@ class TestRedisCommands(object):
             assert value in s
             assert r.smembers('a') == set(s) - set([value])
    
    +    def test_spop_multi_value(self, r):
    +        s = [b('1'), b('2'), b('3')]
    +        r.sadd('a', *s)
    +        values = r.spop('a', 2)
    +        assert values in s
    +        assert r.smembers('a') == set(values) < set(s)
    +
         def test_srandmember(self, r):
             s = [b('1'), b('2'), b('3')]
             r.sadd('a', *s)
    

    [Edited: Added patch to tests in test_commands.py] Ryan

    opened by seaux 28
  • Add Async RedisCluster

    Add Async RedisCluster

    Pull Request check-list

    • [x] Does $ tox pass with this change (including linting)?
    • [x] Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
    • [x] Is the new or changed code fully tested?
    • [x] Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
    • [ ] Is there an example added to the examples folder (if applicable)?
    • [ ] Was the change added to CHANGES file?

    Description of change

    Add Async Redis Cluster Client

    feature 
    opened by utkarshgupta137 27
  • RuntimeError: readuntil() called while another coroutine is already waiting for incoming data

    RuntimeError: readuntil() called while another coroutine is already waiting for incoming data

    Version: redis==4.4.0, Redis: redis:6.2-alpine (docker)

    Platform: Python 3.9.16, docker python:3.9

    Description: Redis client raises RuntimeError while doing multiple concurrency operations.

    Reproduced on 4.4.0, ok on 4.3.5

    Example code:

    import asyncio
    import uuid
    
    from redis.asyncio import Redis
    
    
    async def main():
    
        client = Redis(
            host="redis",
            port=6379,
            db=0,
            username=None,
            password=None,
            retry_on_timeout=True,
            single_connection_client=True,
            client_name='test_client',
        )
    
        # At my PC stable reproduces with n=10, but let set n=100 for better guarantees
        n = 100
    
        async def problem():
            await client.delete(uuid.uuid4().hex)
            await client.delete(uuid.uuid4().hex)  # <- problem is at this line
    
        operations = [problem() for i in range(n)]
    
        await asyncio.gather(*operations)
    
        print('success!')
    

    Stacktrace:

    Traceback (most recent call last):
      File "/app/x.py", line 36, in <module>
        asyncio.get_event_loop().run_until_complete(main())
      File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
        return future.result()
      File "/app/x.py", line 33, in main
        await asyncio.gather(*operations)
      File "/app/x.py", line 29, in problem
        await client.delete(uuid.uuid4().hex)  # <- problem is at this line
      File "/tmp/venv/lib/python3.9/site-packages/redis/asyncio/client.py", line 505, in execute_command
        return await conn.retry.call_with_retry(
      File "/tmp/venv/lib/python3.9/site-packages/redis/asyncio/retry.py", line 59, in call_with_retry
        return await do()
      File "/tmp/venv/lib/python3.9/site-packages/redis/asyncio/client.py", line 481, in _send_command_parse_response
        return await self.parse_response(conn, command_name, **options)
      File "/tmp/venv/lib/python3.9/site-packages/redis/asyncio/client.py", line 524, in parse_response
        response = await connection.read_response()
      File "/tmp/venv/lib/python3.9/site-packages/redis/asyncio/connection.py", line 800, in read_response
        response = await self._parser.read_response(
      File "/tmp/venv/lib/python3.9/site-packages/redis/asyncio/connection.py", line 245, in read_response
        raw = await self._readline()
      File "/tmp/venv/lib/python3.9/site-packages/redis/asyncio/connection.py", line 311, in _readline
        data = await self._stream.readline()
      File "/usr/local/lib/python3.9/asyncio/streams.py", line 540, in readline
        line = await self.readuntil(sep)
      File "/usr/local/lib/python3.9/asyncio/streams.py", line 632, in readuntil
        await self._wait_for_data('readuntil')
      File "/usr/local/lib/python3.9/asyncio/streams.py", line 503, in _wait_for_data
        raise RuntimeError(
    RuntimeError: readuntil() called while another coroutine is already waiting for incoming data
    
    opened by Tiendil 0
  • Update __init__ for issue 2215

    Update __init__ for issue 2215

    Fallback for issue https://github.com/redis/redis-py/issues/2215

    Pull Request check-list

    Please make sure to review and check all of these items:

    • [ ] Does $ tox pass with this change (including linting)?
    • [ ] Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
    • [ ] Is the new or changed code fully tested?
    • [ ] Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
    • [ ] Is there an example added to the examples folder (if applicable)?
    • [ ] Was the change added to CHANGES file?

    NOTE: these things are not required to open a PR and can be done afterwards / while the PR is open.

    Description of change

    Please provide a description of the change here.

    Fallback for splitting version NoneType

    opened by SessionIssue 1
  • add support for GRAPH.CONSTRAINT

    add support for GRAPH.CONSTRAINT

    Pull Request check-list

    Please make sure to review and check all of these items:

    • [x] Does $ tox pass with this change (including linting)?
    • [x] Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
    • [x] Is the new or changed code fully tested?
    • [x] Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
    • [x] Is there an example added to the examples folder (if applicable)?
    • [x] Was the change added to CHANGES file?

    NOTE: these things are not required to open a PR and can be done afterwards / while the PR is open.

    Description of change

    Please provide a description of the change here.

    opened by OfirMos 1
  • Fix incorrect _disconnect_raise docstring

    Fix incorrect _disconnect_raise docstring

    Pull Request check-list

    Please make sure to review and check all of these items:

    • [ ] Does $ tox pass with this change (including linting)?
    • [ ] Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
    • [ ] Is the new or changed code fully tested?
    • [x] Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
    • [ ] Is there an example added to the examples folder (if applicable)?
    • [ ] Was the change added to CHANGES file?

    NOTE: these things are not required to open a PR and can be done afterwards / while the PR is open.

    Description of change

    Updates docstring for _disconnect_raise to correctly reflect code

    opened by mohsinhaider 1
  • Request for supporting xread returning a list instead of dict

    Request for supporting xread returning a list instead of dict

    version: redis 4.4.0 platform: Python 3.8 on Ubuntu 20.04

    For reasons of (memory and time) efficiency we are streaming values to redis as raw numbers instead of key-value pairs, using xadd. Unfortunately, python redis only supports returning the list of values as a dictionary of key-value pairs. For our use case we would need an option to have xread return just a list of values/strings.

    I checked out the code and found the location where the mapping to key-value pairs is performed, but I'm wondering what the best generic method would be to pass this option to xread (or perhaps a config option of the Redis object). Any ideas are appreciated.

    Example to reproduce the issue is given below:

    import os
    import redis
    
    REDIS_URL = os.environ.get("REDIS_URL", "127.0.0.1:6379")
    rds = redis.StrictRedis.from_url(f'redis://{REDIS_URL}')
    
    """
    redis-cli xadd 99:8:1:40 1672345126328633-0 0.950178 8.667000 8.668000 8.662473 8.662473 154.934304 -202.148236 8.665024 8.662473 8.662473 8.662473 0.000000
    redis-cli xread count 1 streams 99:8:1:40 0-0
    1) 1) "99:8:1:40"
       2) 1) 1) "1672345126328633-0"
             2)  1) "0.950178"
                 2) "8.667000"
                 3) "8.668000"
                 4) "8.662473"
                 5) "8.662473"
                 6) "154.934304"
                 7) "-202.148236"
                 8) "8.665024"
                 9) "8.662473"
                10) "8.662473"
                11) "8.662473"
                12) "0.000000"
    
    """
    channel = "99:8:1:40"
    data = [ 0.950178, 8.667000, 8.668000, 8.662473, 8.662473, 154.934304, -202.148236, 8.665024, 8.662473, 8.662473, 8.662473, 0.000000 ]
    
    events = rds.xread(streams={channel: '0-0'}, count=1)
    print(events)
    
    # Returns: [[b'99:8:1:40', [(b'1672345126328633-0', {b'0.950178': b'8.667000', b'8.668000': b'8.662473', b'8.662473': b'0.000000', b'-202.148236': b'8.665024'})]]]
    # instead of 
    # [[b'99:8:1:40', [(b'1672345126328633-0', [b'0.950178', b'8.667000', b'8.668000', b'8.662473', b'8.662473', b'154.934304', b'-202.148236', b'8.665024', b'8.662473', b'8.662473', b'8.662473', b'0.000000' ]] )]]]
    # or
    # [[b'99:8:1:40', [(b'1672345126328633-0', [(b'0.950178', b'8.667000)', b'(8.668000', b'8.662473)', b'(8.662473', b'154.934304)', b'(-202.148236', b'8.665024)', b'(8.662473', b'8.662473)', b'(8.662473', b'0.000000') ]] )]]]
    
    opened by chill1n 0
Releases(v4.4.0)
  • v4.4.0(Dec 4, 2022)

    Changes

    4.4.0rc4 release notes 4.4.0rc3 release notes 4.4.0rc2 release notes 4.4.0rc1 release notes

    πŸš€ New Features (since 4.4.0rc4)

    • Async clusters: Support creating locks inside async functions (#2471)

    πŸ› Bug Fixes (since 4.4.0rc4)

    • Async: added 'blocking' argument to call lock method (#2454)
    • Added a replacement for the default cluster node in the event of failure. (#2463)
    • Fixed geosearch: Wrong number of arguments for geosearch command (#2464)

    🧰 Maintenance (since 4.4.0rc4)

    • Updating dev dependencies (#2475)
    • Removing deprecated LGTM (#2473)
    • Added an explicit index name in RediSearch example (#2466)
    • Adding connection step to bloom filter examples (#2478)

    Contributors (since 4.4.0rc4)

    We'd like to thank all the contributors who worked on this release!

    @Sibuken, @barshaul, @chayim, @dvora-h, @nermiller, @uglide and @utkarshgupta137

    Source code(tar.gz)
    Source code(zip)
  • v4.3.5(Nov 22, 2022)

    Changes

    This is a maintenance release of redis-py, prior to the release of 4.4.0. This release contains both bug fixes, and features, keeping pace with the release of redis-stack capabilities.

    πŸš€ New Features

    • Add support for TIMESERIES 1.8 (#2296)
    • Graph - add counters for removed labels and properties (#2292)
    • Add support for TDIGEST.QUANTILE extensions (#2317)
    • Add TDIGEST.TRIMMED_MEAN (#2300)
    • Add support for async GRAPH module (#2273)
    • Support TDIGEST.MERGESTORE and make compression optional on TDIGEST.CREATE (#2319)
    • Adding reserve as an alias for create, so that we have BF.RESERVE and CF.RESERVE accuratenly supported (#2331)

    πŸ› Bug Fixes

    • Fix async connection.is_connected to return a boolean value (#2278)
    • Fix: workaround asyncio bug on connection reset by peer (#2259)
    • Fix crash: key expire while search (#2270)
    • Async cluster: fix concurrent pipeline (#2280)
    • Fix async SEARCH pipeline (#2316)
    • Fix KeyError in async cluster - initialize before execute multi key commands (#2439)

    🧰 Maintenance

    • Supply chain risk reduction: remove dependency on library named deprecated (#2386)
    • Search test - Ignore order of the items in the response (#2322)
    • Fix GRAPH.LIST & TDIGEST.QUANTILE tests (#2335)
    • Fix TimeSeries range aggregation (twa) tests (#2358)
    • Mark TOPK.COUNT as deprecated (#2363)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @sileht, @utkarshgupta137, @dvora-h, @akx, @bodevone, @chayim, @DvirDukhan

    Source code(tar.gz)
    Source code(zip)
  • v4.4.0rc4(Nov 10, 2022)

    Changes

    πŸš€ New Features

    • CredentialsProvider class added to support password rotation (#2261)
    • Enable AsyncIO cluster mode lock (#2446)

    πŸ› Bug Fixes

    • Failover handling improvements for RedisCluster and Async RedisCluster (#2377)
    • Improved response parsing options handler for special cases (#2302)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @KMilhan, @barshaul, @dvora-h and @fadida

    Source code(tar.gz)
    Source code(zip)
  • v4.4.0rc3(Nov 8, 2022)

    Changes

    πŸ§ͺ Experimental Features

    • Remove weight parameter from TDIGEST.ADD and fixed tests (#2408)

    πŸš€ New Features

    • Add to_string method for GeoValue (#2404)

    πŸ› Bug Fixes

    • Remove default None value from LMPOP direction argument (#2438)
    • Fix KeyError in async cluster - initialize before execute multi key commands (#2439)
    • Fix catch OSError on asyncio (#2412)

    🧰 Maintenance

    • Remove the superflous SocketBuffer from asyncio PythonParser (#2418)
    • Fix docs for password protected socket access (#2378)
    • Supply chain risk reduction: remove dependency on library named deprecated (#2386)
    • Tidy up asyncio examples. (#2431)
    • Docs: fix redismodules.rst (#2429)
    • Fix test test_get_moveable_keys (#2411)
    • Fixing RedisBloom tests for TDIGEST.RANK results (#2430)
    • Deprecate add_doccument (FT.ADD) (#2425)
    • Updating the docker image for the module cluster (#2426)
    • Release drafter formatting improvement: titles (#2422)
    • Fix TIMESERIES (round floats) and JSON tests (#2421)
    • Fixing invalid vulnerability report (#2424)
    • Support M1 development with updated dockers (#2409)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @AnneYang720, @Dreamsorcerer, @aandrukhovich, @akx, @ashikshezan, @chayim, @dvora-h, @hofrob, @hyeongguen-song, @kristjanvalur and @sobolevn

    Source code(tar.gz)
    Source code(zip)
  • v4.4.0rc2(Sep 29, 2022)

    Changes

    Breaking Changes

    • Catch Exception and not BaseException in the Connection (#2104)

    πŸ§ͺ Experimental Features

    • Add support for new TDIGEST (RedisBloom 2.4) features (#2392)
    • Enhanced TDIGEST.CREATE support (#2348)

    πŸš€ New Features

    • Simplify async timeouts and allowing timeout=None in PubSub.get_message() to wait forever (https://github.com/redis/redis-py/pull/2295)
    • Add nowait flag to asyncio.Connection.disconnect() (#2356)
    • Dev/no lock (#2308)
    • Adding reserve as an alias for create, so that BF.RESERVE and CF.RESERVE accurately supported (#2331)
    • Add support for BITFIELD_RO (#2340)

    πŸ› Bug Fixes

    • Catch Exception and not BaseException in the Connection (#2104)
    • FT.SEARCH: Fixing aggregation request argument for LIMIT (#2393)
    • Change return type of parse_boolean from int to bool (#2364)
    • WRONGPASS response doesn't raise AuthenticationError exception (#2329)
    • Fix type hint annotations of expire time (#2361)
    • Increased strictness of redis url scheme parsing (#2343)

    🧰 Maintenance

    • Simplify async timeouts and allowing timeout=None in PubSub.get_message() to wait forever (https://github.com/redis/redis-py/pull/2295)
    • Dev/no can read (#2360)
    • Update VecSim search example to highlight dialect (#2402)
    • Update GitHub Actions versions (#2389)
    • Support Hiredis >= 1.x only (remove packaging dependency) (#2385)
    • Reinstate Codecov in CI (#2394)
    • Adding support for readable test results in CI (#2381)
    • Fix readme links (#2368)
    • Fixing async test for module list (#2367)
    • Mark TOPK.COUNT as deprecated (#2363)
    • Document Redis(single_connection_client) (#2327)
    • Update LICENSE (#2362)
    • Fix TimeSeries range aggregation (twa) tests (#2358)
    • Fix documentation typos (#2341)
    • Adding dependency vulnerability scanning to the CI process (#2332)
    • Fix GRAPH.LIST & TDIGEST.QUANTILE tests (#2335)
    • GitHub workflow update" trailofbits/gh-action-pip-audit is now pypa/gh-action-pip-audit (#2380)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @akx, @bodevone, @chayim, @dvora-h, @gkorland, @graeme44, @kristjanvalur, @leven-cn, @lucacillario, @n8wb, @povilasb, @vanschelven and @woodruffw

    Source code(tar.gz)
    Source code(zip)
  • v4.4.0rc1(Aug 4, 2022)

    Changes

    Breaking Changes

    • Change dropindex() function to use FT.DROPINDEX (instead FT.DROP) (#2315)
    • Drop python 3.6 support (#2306)
    • Async cluster: fix max_connections/ssl & improve args (#2217)
    • Remove verbose logging from redis-py/redis/cluster.py (#2238)
    • Drop python 3.6 support (#2306)

    πŸ§ͺ Experimental Features

    • Add support for WITHSUFFIXTRIE to FT.CREATE (#2324)

    πŸš€ New Features

    • Add TDIGEST.TRIMMED_MEAN (#2300)
    • Support TDIGEST.MERGESTORE and make compression optional on TDIGEST.CREATE (#2319)
    • Add support for TDIGEST.QUANTILE extensions (#2317)
    • Add support for async GRAPH module (#2273)
    • Add support for TIMESERIES 1.8 (#2296)
    • Use retry mechanism in async version of Connection objects (#2271)
    • Graph - add counters for removed labels and properties (#2292)

    πŸ› Bug Fixes

    • Fix async SEARCH pipeline (#2316)
    • Change dropindex() to use FT.DROPINDEX (instead FT.DROP) (#2315)
    • Automatically reconnect pubsub when reading messages in blocking mode (#2281)
    • Async cluster: fix max_connections/ssl & improve args (#2217)
    • Async cluster: fix concurrent pipeline (#2280)
    • Fix crash: key expire while search (#2270)
    • Fix: workaround asyncio bug on connection reset by peer (#2259)
    • Fix async connection.is_connected to return a boolean value (#2278)
    • ClusterPipeline Doesn't Handle ConnectionError for Dead Hosts (#2225)

    🧰 Maintenance

    • Adding experimental features, as a release option (#2323)
    • Search test - Ignore order of the items in the response (#2322)
    • Replace ensure_future with create_task (#2311)
    • Fix typo in README.md (#2314)
    • Remove python2-only __nonzero__ method from cluster.py (#2313)
    • Fix typing on smembers command (#2312)
    • drop invocation of redismod_cluster docker (#2307)
    • Cleaning up the readme and moving docs into readthedocs (#2291)
    • Fix warnings and resource usage problems in asyncio unittests (#2258)
    • Doc add timeseries example (#2267)
    • Fix: start_id type for XAUTOCLAIM (#2257)
    • redis stream example (#2269)
    • Remove verbose logging from redis-py/redis/cluster.py (#2238)
    • Docs: Fix a few typos (#2274)
    • Pypy 3.8 CI support (#2266)
    • Fix timezone handling for datetime to unixtime conversions (#2213)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @DvirDukhan, @GaMeRaM, @Iglesys347, @aandrukhovich, @agusdmb, @bodevone, @chayim, @dvora-h, @grippy, @joekohlsdorf, @kristjanvalur, @nialdaly, @pedrofrazao, @sileht, @sobolevn, @szumka, @timgates42 and @utkarshgupta137

    Source code(tar.gz)
    Source code(zip)
  • v4.3.4(Jun 27, 2022)

    Changes

    πŸ”₯ Breaking Changes

    • Fix backward compatibility from 4.3.2 in Lock.acquire() (#2254)
    • Fix XAUTOCLAIM to return the full response, instead of only keys 2+ (#2252)

    πŸš€ New Features

    • Added dynamic_startup_nodes configuration to RedisCluster. (#2244, #2251)

    πŸ› Bug Fixes

    • Fix retries in async mode (#2180)
    • Async cluster: fix simultaneous initialize (#2231)
    • Uppercased commands in CommandsParser.get_keys (#2236)

    🧰 Maintenance

    • Late eval of the skip condition in async tests(#2248, #2253)
    • Reuse the old nodes' connections when a cluster topology refresh is being done (#2235)
    • Docs: add pipeline examples (#2240)
    • Correct retention_msecs value (#2232)
    • Cluster: use pipeline to execute split commands (#2230)
    • Docs: Add a note about client_setname and client_name difference (#2247)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @Iglesys347, @barshaul, @dvora-h, @earthgecko, @elemoine, @falk-h, @kristjanvalur, @psrok1 and @utkarshgupta137

    Source code(tar.gz)
    Source code(zip)
  • v4.3.3(Jun 2, 2022)

    Changes

    πŸ› Bug Fixes

    • Fix Lock crash, and versioning 4.3.3 (#2210)

    🧰 Maintenance

    • Async cluster: improve docs (#2208)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @dvora-h and @utkarshgupta137

    Source code(tar.gz)
    Source code(zip)
  • v4.3.2(Jun 1, 2022)

    Changes

    πŸš€ New Features

    • SHUTDOWN - add support for the new NOW, FORCE and ABORT modifiers (#2150)
    • Adding pipeline support for async cluster (#2199)
    • Support CF.MEXISTS + Clean bf/commands.py (#2184)
    • Extending query_params for FT.PROFILE (#2198)
    • Implementing ClusterPipeline Lock (#2190)

    πŸ› Bug Fixes

    • Set default response_callbacks to redis.asyncio.cluster.ClusterNode (#2201)
    • Add default None for maxlen at xtrim command (#2188)

    🧰 Maintenance

    • Async cluster: add/update typing (#2195)
    • Changed list type to single element type (#2203)
    • Made sync lock consistent and added types to it (#2137)
    • Async cluster: optimisations (#2205)
    • Fix typos in README (#2206)
    • Fix modules links to https://redis.io/commands/ (#2185)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @Avital-Fine, @Olegt0rr, @WisdomPill, @dvora-h, @grippy, @mfgnik, @rapidia, @ryanrussell and @utkarshgupta137

    Source code(tar.gz)
    Source code(zip)
  • v4.3.1(May 9, 2022)

    Changes

    πŸ› Bug Fixes

    • Fix asyncio Search crash (#2175)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @dvora-h

    Source code(tar.gz)
    Source code(zip)
  • v4.3.0(May 8, 2022)

    Changes

    πŸ”₯ Breaking Changes

    • Replace OSError exceptions from can_read with redis.ConnectionError (#2140)
    • Updated FUNCTION LOAD changes (from release 7.0 rc3 to support redis 7.0 final) (#2139)

    πŸš€ New Features

    • Get command keys for subcommands (#2170)
    • Add support for CLUSTER SHARDS (#2151)
    • Add support for COMMAND LIST (#2149)
    • Add Async RedisCluster (#2099)
    • ACL SETUSER - add selectors and key based permissions (#2161)
    • Support for redis 7 streams features (#2157)
    • Async Connection: Allow PubSub.run() without previous subscribe() (#2148)
    • Implemented LATENCY HISTOGRAM by always throwing NotImplementedError (#2147)
    • Add async supoort for SEARCH commands (#2096)
    • Retry(): Support negative retries value (#2110)
    • Add support for MODULE LOADEX (#2146)
    • INFO - add support for taking multiple section arguments (#2145)
    • CONFIG SET - add the ability to set multiple parameters in one call (#2143)
    • CONFIG GET - add the ability to pass multiple pattern parameters in one call (#2142)
    • Add support for COMMAND GETKEYSANDFLAGS (#2141)
    • Support CASESENSITIVE for TAG fields (#2112)

    πŸ› Bug Fixes

    • Rename 'update_supported_erros' to 'update_supported_errors' in Retry module (#2144)
    • Fix execute_command() determining nodes error when no key command (#2097)
    • Fix incorrect return statement in auth (#2086) (#2092)

    🧰 Maintenance

    • Add unittest for PubSub.connect() (#2167)
    • Fix incorrect return annotation in asyncio.lock (#2155)
    • Minor cleanups in commands/cluster.py (#2094)
    • Update xtrim type annotation (#2093)
    • Async tests for redis commands, json, bloom, timeseries (#2087)
    • Fixed typing in getex command (#2088)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @Andrew-Chen-Wang, @Ankhas, @Avital-Fine, @JelleZijlstra, @chayim, @dvora-h, @enjoy-binbin, @kamyabzad, @kristjanvalur, @richli, @suxb201 and @utkarshgupta137

    Source code(tar.gz)
    Source code(zip)
  • v4.2.2(Apr 4, 2022)

    Changes

    πŸš€ New Features

    • Extended "CLUSTER NODES" parser to support special slot entries (#2080)
      • πŸ”₯ NOTE: This change is potentially breaking depending on your use of specialized slot entries. Support for bytes was added
    • Add support for BIT|BYTE option available in redis 7 (#2068)

    πŸ› Bug Fixes

    • Fix imports in for async (#2079)
    • Always clear reference to closed reader/writer (#2077)
    • Fix disable decode for dump command in async (#2070)

    🧰 Maintenance

    • Cluster commands linkdocs (#2069)
    • Clarify bit type in setbit/getbit documentation (#2078)
    • Clean up test supoort enterprise environments (#2082)
    • Vector similarity search example (#2083)
    • Fix search query with params tests (#2076)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @barshaul, @beasteers, @dsoprea, @dvora-h, @enjoy-binbin, @nosammai and @wwqgtxx

    Source code(tar.gz)
    Source code(zip)
  • v4.2.1(Mar 31, 2022)

    Changes

    πŸš€ New Features

    • Add support for CLUSTER MYID (#2063)
    • Add dialect support for RediSearch queries (#2071)

    🧰 Maintenance

    • Fix black (#2075)
    • Make typing_extensions conditional to Python < 3.8 (#2062)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @dvora-h, @enjoy-binbin and @mgorny

    Source code(tar.gz)
    Source code(zip)
  • v4.2.0(Mar 23, 2022)

    Changes

    4.2.0-rc1 release notes 4.2.0-rc2 release notes 4.2.0-rc3 release notes

    πŸš€ New Features

    • Support for Vector Fields for Vector Similarity Search (#2041)

    πŸ› Bug Fixes

    • Fix cluster scan command cursors & scan_iter (#2054)

    🧰 Maintenance

    • Remove verbose logging from cluster initializers (#2055)
    • Mark tests for redis-stack (#2052)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @Avital-Fine, @dvora-h, @joni- and @utkarshgupta137

    Source code(tar.gz)
    Source code(zip)
  • v4.2.0rc3(Mar 14, 2022)

    Changes

    πŸš€ New Features

    • Add support for SORT_RO available in redis 7(#1858)
    • Added support for redisearch module commands in cluster (#2042)
    • Add support for PEXPIREAT command options available in redis 7 (#2027)
    • Add support for PEXPIRE command option available in redis 7 (#2026)
    • Add support for EXPIREAT command options available in redis 7 (#2024)
    • Add support for EXPIRE command's options available in redis 7 (#2002)
    • SSL connections can now load certificates from strings instead of just files (#2048)
    • Add support for CLUSTER LINKS available in redis 7 (#2019)
    • Add support for CLUSTER DELSLOTSRANGE available in redis 7 (#2018)
    • Add support for CLUSTER ADDSLOTSRANGE available in redis 7 (#2017)

    πŸ› Bug Fixes

    • Fix TypeError in graph commands (#2050)
    • Async - ensure redis_connect_func is set on uds connection (#2046)

    🧰 Maintenance

    • Fix: avoiding issue with PytestUnraisableExceptionWarning (#1458)
    • Documentation typos (#2045, #2043)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @advance512, @dogukanteber, @dvora-h, @fuho, @lazToum and @razzeee

    Source code(tar.gz)
    Source code(zip)
  • v4.2.0rc2(Mar 8, 2022)

    Changes

    πŸš€ New Features

    • Add pipeline support for Search (#2038)
    • Add support for PEXPIRETIME (#1861)
    • Add support for EXPIRETIME (#1860)
    • Implemented COMMAND DOCS by always throwing NotImplementedError (#2020)
    • Add cluster support for functions (#2016)
    • Add support for HSET items (#2006)
    • Add support for JSON, TIMESERIES, BLOOM & GRAPH commands in cluster (#2032)
    • Add support for AUTH (#1929)
    • Implement locks for RedisCluster (#2013)

    🧰 Maintenance

    • Add async typing support to commands' returns (#2010)
    • Fix asyncio retry.py typing (#2011)
    • Fix typo in comment (#2025)
    • Fixing TimesSeries tests (#2029)
    • Add AsyncFunctionCommands (#2009)
    • Renaming chore as maintenance (#2015)
    • Add Python 3.6 back to hash verify CI (#2008)
    • Add redismod cluster docker (#2022)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @Andrew-Chen-Wang, @OfirMos, @chayim, @dogukanteber, @dvora-h, @jakebarnwell, @mdczaplicki and @spagh-eddie

    Source code(tar.gz)
    Source code(zip)
  • v4.2.0rc1(Feb 22, 2022)

    Changes

    πŸ”₯ Breaking Changes

    • Functions within RedisGraph support, were renamed to match Python naming conventions. (#1872)
    • Modified rootPath argument in JSON.SET to root_path as part of type hint addition. (#1921)

    πŸš€ New Features

    • Added ASYNC support, merging with aioredis (#1899)
    • Add cluster support for scripting (#1937)
    • Add support for Redis 7 functions (#1998)
    • Add support for ACL DRYRUN (#1992)
    • Add support for BZMPOP (#1851)
    • Add support for CLIENT NO-EVICT (#1856)
    • Add support for ZINTERCARD (#1857)
    • Add support for EVAL_RO (#1862)
    • Add support for EVAL_RO (#1863)
    • Add support for SINTERCARD (#1859)
    • Add support for LMPOP (#1843)
    • Add support for BLMPOP (#1849)
    • Add support for ZMPOP (#1923)
    • Add support for LCS (#1924)
    • Implemented AUTH by always throwing NotImplementedError (#1910)
    • Implemented FAILOVER by always throwing NotImplementedError (#1911)
    • Implemented HELLO by always throwing NotImplementedError (#1912)

    πŸ› Bug Fixes

    • Catch OSError in BlockingConnectionPool.get_connection (#2001)
    • Fixing Elevated CPU utilization on one node when using RedisCluster pipeline (#1985)

    🧰 Maintenance

    • Fixing type annotation for EXPLAIN (#1997)
    • Add type hints for list commands (#1917)
    • Add type hints for set commands (#1918)
    • Add type hints for hash commands (#1919)
    • Add type hints for JSON commands (#1921)
    • Add set and get examples (#1916)
    • Docs: fix the wrong link to watch command (#1984)
    • Fix typo in readme (#1936)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @Andrew-Chen-Wang, @chayim, @dogukanteber, @dvora-h, @elemoine, @jakebarnwell, @shchurov, @ashtul, @Pagliacii, @joseotoro and @srittau

    Source code(tar.gz)
    Source code(zip)
  • v4.1.4(Feb 16, 2022)

  • v4.1.3(Feb 8, 2022)

    πŸ› Bug Fixes

    • Fix flushdb and flushall (#1926)

    🧰 Maintenance

    • Add redis5 and redis4 dockers (#1871)
    • Change json.clear test multi to be up to date with redisjson (#1922)
    • Fixing volume for unstable_cluster docker (#1914)
    • Update changes file with changes since 4.0.0-beta2 (#1915)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @MeirShpilraien, @dvora-h

    Source code(tar.gz)
    Source code(zip)
  • v4.1.2(Jan 27, 2022)

    Changes

    πŸš€ New Features

    • Invalid OCSP certificates should raise ConnectionError on failed validation (#1907)
    • Added retry mechanism on socket timeouts when connecting to the server (#1895)

    πŸ› Bug Fixes

    • LMOVE, BLMOVE return incorrect responses (#1906)
    • Fixing AttributeError in UnixDomainSocketConnection (#1903)
    • Fixing TypeError in GraphCommands.explain (#1901)

    🧰 Maintenance

    • For tests, increasing wait time for the cluster (#1908)
    • Increased pubsub's wait_for_messages timeout to prevent flaky tests (#1893)
    • README code snippets formatted to highlight properly (#1888)
    • Fix link in the main page (#1897)
    • Documentation fixes: JSON Example, SSL Connection Examples, RTD version (#1887)
    • Direct link to readthedocs (#1885)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @barshaul, @chayim, @dvora-h, @soumendrak and @varunchopra

    Source code(tar.gz)
    Source code(zip)
  • v4.1.1(Jan 17, 2022)

    Changes

    πŸš€ New Features

    • Add retries to connections in Sentinel Pools (#1879)
    • OCSP Stapling Support (#1873)
    • Define incr/decr as aliases of incrby/decrby (#1874)
    • FT.CREATE - support MAXTEXTFIELDS, TEMPORARY, NOHL, NOFREQS, SKIPINITIALSCAN (#1847)

    πŸ› Bug Fixes

    • Timeseries docs fix (#1877)
    • get_connection: catch OSError too (#1832)
    • Set keys var otherwise variable not created (#1853)
    • Clusters should optionally require full slot coverage (#1845)

    🧰 Maintenance

    • Triple quote docstrings in client.py PEP 257 (#1876)
    • syncing requirements (#1870)
    • Typo and typing in GraphCommands documentation (#1855)
    • Allowing poetry and redis-py to install together (#1854)
    • setup.py: Add project_urls for PyPI (#1867)
    • Support test with redis unstable docker (#1850)
    • Connection examples (#1835)
    • Documentation cleanup (#1841)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @Andrew-Chen-Wang, @AvitalFineRedis, @Enchufa2, @barshaul, @chayim, @dvora-h, @jdieter, @salty-horse, @sbv-csis and @tony

    Source code(tar.gz)
    Source code(zip)
  • v4.1.0(Dec 26, 2021)

    Changes

    4.1.0-rc2 release notes 4.1.0-rc1 release notes

    πŸš€ New Features

    • OCSP stapling support (#1820)
    • Support for SELECT (#1825)
    • Support for specifying error types with retry (#1817)
    • Support for RESET command since Redis 6.2.0 (#1824)
    • Support CLIENT TRACKING (#1612)
    • Support WRITE in CLIENT PAUSE (#1549)
    • JSON set_file and set_path support (#1818)
    • Allow ssl_ca_path with rediss:// urls (#1814)
    • Support for password-encrypted SSL private keys (#1782)
    • Support SYNC and PSYNC (#1741)

    πŸ› Bug Fixes

    • Retry on error exception and timeout fixes (#1821)
    • Fixing read race condition during pubsub (#1737)
    • Fixing exception in listen (#1823)
    • Fixed MovedError, and stopped iterating through startup nodes when slots are fully covered (#1819)
    • Socket not closing after server disconnect (#1797)
    • Single sourcing the package version (#1791)
    • Ensure redis_connect_func is set on uds connection (#1794)

    🧰 Maintenance

    • SRTALGO - Skip for redis versions greater than 7.0.0 (#1831)
    • Documentation updates (#1822)
    • Add CI action to install package from repository commit hash (#1781) (#1790)
    • Fix link in lmove docstring (#1793)
    • Disabling JSON.DEBUG tests (#1787)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @Akuli, @AvitalFineRedis, @alisaifee, @ashwani99, @barshaul, @chayim, @dvora-h, @pawl and @yanivhershkovich

    Source code(tar.gz)
    Source code(zip)
  • v4.1.0rc2(Dec 9, 2021)

    Changes

    πŸ”₯ Breaking Changes (from 4.1.0 rc1)

    • Migrated targeted nodes to kwargs in Cluster Mode (#1762)

    πŸš€ New Features

    • Added support for MONITOR in clusters (#1756)
    • Adding ROLE Command (#1610)
    • Integrate RedisBloom support (#1683)
    • Adding RedisGraph support (#1556)
    • Allow overriding connection class via keyword arguments (#1752)
    • Aggregation LOAD * support for RediSearch (#1735)

    🧰 Maintenance

    • Adding cluster, bloom, and graph docs (#1779)
    • Add packaging to setup_requires, and use >= to play nice to setup.py (fixes #1625) (#1780)
    • Fixing the license link in the readme (#1778)
    • Removing distutils from tests (#1773)
    • Fix cluster ACL tests (#1774)
    • Improved RedisCluster's reinitialize_steps and documentation (#1765)
    • Added black and isort (#1734)
    • Link Documents for all module commands (#1711)
    • Pyupgrade + flynt + f-strings (#1759)
    • Remove unused aggregation subclasses in RediSearch (#1754)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @AvitalFineRedis, @WisdomPill, @akx, @barshaul, @asheryerm, @chayim, @hartwork and @m-novikov

    Source code(tar.gz)
    Source code(zip)
  • v4.1.0rc1(Nov 25, 2021)

    Changes

    πŸš€ New Features

    • Adding RedisCluster client to support Redis Cluster Mode (#1660)
    • Support RediSearch FT.PROFILE command (#1727)
    • Adding support for non-decodable commands (#1731)
    • COMMAND GETKEYS support (#1738)
    • RedisJSON 2.0.4 behaviour support (#1747)

    πŸ› Bug Fixes

    • Removing deprecating distutils (PEP 632) (#1730)

    🧰 Maintenance

    • Updating PR template (#1745)
    • Removing duplication of Script class (#1751)
    • Splitting documentation for read the docs (#1743)
    • Improve code coverage for aggregation tests (#1713)
    • Fixing COMMAND GETKEYS tests (#1750)
    • GitHub release improvements (#1684)
    • 4.1.0rc1 (#1742)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @AvitalFineRedis, @barshaul and @chayim

    Source code(tar.gz)
    Source code(zip)
  • v4.0.2(Nov 22, 2021)

    Changes

    πŸ› Bug Fixes

    • Restoring Sentinel commands to redis client (#1723)
    • Better removal of hiredis warning (#1726)

    🧰 Maintenance

    • Adding links to redis documents in function calls (#1719)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @Carlosbogo, @sculley and @wuisawesome

    Source code(tar.gz)
    Source code(zip)
  • v4.0.1(Nov 17, 2021)

    πŸ› Bug Fixes

    • Removing command on initial connections (#1722)
    • Removing hiredis warning when not installed (#1721)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @chayim

    Source code(tar.gz)
    Source code(zip)
  • v4.0.0(Nov 15, 2021)

    Changes Since 3.5.3

    4.0.0-rc2 release notes 4.0.0-rc1 release notes 4.0.0-beta3 release notes 4.0.0-beta2 release notes 4.0.0-beta1 ​release notes

    πŸš€ New Features

    • FT.EXPLAINCLI intentionally raising NotImplementedError (#1705)

    πŸ› Bug Fixes

    • Restoring ZRANGE desc for Redis < 6.2.0 (#1697)
    • Response parsing occasionally fails to parse floats (#1692)
    • Re-enabling read-the-docs (#1707)

    🧰 Maintenance

    • Call HSET after FT.CREATE to avoid keyspace scan (#1706)
    • Unit tests fixes for compatibility (#1703)
    • Improve documentation about Locks (#1701)
    • Fixes to allow --redis-url to pass through all tests (#1700)
    • Fix unit tests running against Redis 4.0.0 (#1699)
    • Search alias test fix (#1695)
    • Adding RediSearch/RedisJSON tests (#1691)
    • Updating codecov rules (#1689)
    • Tests to validate custom JSON decoders (#1681)
    • Added breaking icon to release drafter (#1702)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @AvitalFineRedis, @ashtul, @chayim and @jerr0328

    Source code(tar.gz)
    Source code(zip)
  • v4.0.0rc2(Nov 9, 2021)

    πŸ› Bug Fixes

    • Removing dependency on six (#1676)
    • Re-enable pipeline support for JSON and TimeSeries (#1674)
    • Export Sentinel, and SSL like other classes (#1671)
    • Restore zrange functionality for older versions of Redis (#1670)
    • Fixed garbage collection deadlock (#1578)

    🧰 Maintenance

    • Tests to validate built python packages (#1678)
    • Sleep for flaky search test (#1680)
    • Test function renames, to match standards (#1679)
    • Docstring improvements for Redis class (#1675)
    • Fix georadius tests (#1672)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @AvitalFineRedis, @chayim, @emorozov and @terencehonles

    Source code(tar.gz)
    Source code(zip)
  • v4.0.0rc1(Nov 4, 2021)

    Breaking Changes

    • Improvements to JSON coverage (#1666) Impact: JSON commands available since 4.0.0beta3 now better match RedisJSON
    • Add python_requires setuptools check for python > 3.6 (#1656)

    πŸš€ New Features

    • SMISMEMBER support (#1667)
    • Exposing the module version in loaded_modules (#1648)
    • RedisTimeSeries support (#1652)
    • Support for json multipath ($) (#1663)

    πŸ› Bug Fixes

    • Added boolean parsing to PEXPIRE and PEXPIREAT (#1665)
    • Add python_requires setuptools check for python > 3.6 (#1656)

    🧰 Maintenance

    • Adding vulture for static analysis (#1655)
    • Starting to clean the docs (#1657)
    • Update README.md (#1654)
    • Adding description format for package (#1651)
    • Publish to pypi as releases are generated with the release drafter (#1647)
    • Restore actions to prs (#1653)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @Nicusor97, @WisdomPill, @chayim and @gkorland

    Source code(tar.gz)
    Source code(zip)
  • v4.0.0b3(Oct 26, 2021)

    Changes

    πŸ› Bug Fixes

    • Fixing the package to include commands (#1649)

    🧰 Maintenance

    • Re-enabling codecov as part of CI process (#1646)

    Contributors

    We'd like to thank all the contributors who worked on this release!

    @chayim

    Source code(tar.gz)
    Source code(zip)
Owner
Redis
Redis
Redis client for Python asyncio (PEP 3156)

Redis client for Python asyncio. Redis client for the PEP 3156 Python event loop. This Redis library is a completely asynchronous, non-blocking client

Jonathan Slenders 554 Dec 4, 2022
A tiny python web application based on Flask to set, get, expire, delete keys of Redis database easily with direct link at the browser.

First Redis Python (CRUD) A tiny python web application based on Flask to set, get, expire, delete keys of Redis database easily with direct link at t

Max Base 9 Dec 24, 2022
An asyncio compatible Redis driver, written purely in Python. This is really just a pet-project for me.

asyncredis An asyncio compatible Redis driver. Just a pet-project. Information asyncredis is, like I've said above, just a pet-project for me. I reall

Vish M 1 Dec 25, 2021
asyncio (PEP 3156) Redis support

aioredis asyncio (PEP 3156) Redis client library. Features hiredis parser Yes Pure-python parser Yes Low-level & High-level APIs Yes Connections Pool

aio-libs 2.2k Jan 4, 2023
Python interface to Oracle Database conforming to the Python DB API 2.0 specification.

cx_Oracle version 8.2 (Development) cx_Oracle is a Python extension module that enables access to Oracle Database. It conforms to the Python database

Oracle 841 Dec 21, 2022
ClickHouse Python Driver with native interface support

ClickHouse Python Driver ClickHouse Python Driver with native (TCP) interface support. Asynchronous wrapper is available here: https://github.com/myma

Marilyn System 957 Dec 30, 2022
TileDB-Py is a Python interface to the TileDB Storage Engine.

TileDB-Py TileDB-Py is a Python interface to the TileDB Storage Engine. Quick Links Installation Build Instructions TileDB Documentation Python API re

TileDB, Inc. 149 Nov 28, 2022
Asynchronous interface for peewee ORM powered by asyncio

peewee-async Asynchronous interface for peewee ORM powered by asyncio. Important notes Since version 0.6.0a only peewee 3.5+ is supported If you still

05Bit 666 Dec 30, 2022
A pythonic interface to Amazon's DynamoDB

PynamoDB A Pythonic interface for Amazon's DynamoDB. DynamoDB is a great NoSQL service provided by Amazon, but the API is verbose. PynamoDB presents y

null 2.1k Dec 30, 2022
A Pythonic, object-oriented interface for working with MongoDB.

PyMODM MongoDB has paused the development of PyMODM. If there are any users who want to take over and maintain this project, or if you just have quest

mongodb 345 Dec 25, 2022
Pure Python MySQL Client

PyMySQL Table of Contents Requirements Installation Documentation Example Resources License This package contains a pure-Python MySQL client library,

PyMySQL 7.2k Jan 9, 2023
Python client for Apache Kafka

Kafka Python client Python client for the Apache Kafka distributed stream processing system. kafka-python is designed to function much like the offici

Dana Powers 5.1k Jan 8, 2023
A fast PostgreSQL Database Client Library for Python/asyncio.

asyncpg -- A fast PostgreSQL Database Client Library for Python/asyncio asyncpg is a database interface library designed specifically for PostgreSQL a

magicstack 5.8k Dec 31, 2022
Asynchronous Python client for InfluxDB

aioinflux Asynchronous Python client for InfluxDB. Built on top of aiohttp and asyncio. Aioinflux is an alternative to the official InfluxDB Python cl

Gustavo Bezerra 159 Dec 27, 2022
Google Cloud Client Library for Python

Google Cloud Python Client Python idiomatic clients for Google Cloud Platform services. Stability levels The development status classifier on PyPI ind

Google APIs 4.1k Jan 1, 2023
Official Python low-level client for Elasticsearch

Python Elasticsearch Client Official low-level client for Elasticsearch. Its goal is to provide common ground for all Elasticsearch-related code in Py

elastic 3.8k Jan 1, 2023
python-bigquery Apache-2python-bigquery (πŸ₯ˆ34 Β· ⭐ 3.5K Β· πŸ“ˆ) - Google BigQuery API client library. Apache-2

Python Client for Google BigQuery Querying massive datasets can be time consuming and expensive without the right hardware and infrastructure. Google

Google APIs 550 Jan 1, 2023
High level Python client for Elasticsearch

Elasticsearch DSL Elasticsearch DSL is a high-level library whose aim is to help with writing and running queries against Elasticsearch. It is built o

elastic 3.6k Jan 3, 2023
Python client for InfluxDB

InfluxDB-Python InfluxDB-Python is a client for interacting with InfluxDB. Development of this library is maintained by: Github ID URL @aviau (https:/

InfluxData 1.6k Dec 24, 2022