Asynchronous, fast, pythonic DynamoDB Client

Overview

AsyncIO DynamoDB

CircleCI Code style: black Documentation Status

Asynchronous pythonic DynamoDB client; 2x faster than aiobotocore/boto3/botocore.

Quick start

With httpx

Install this library

pip install "aiodynamo[httpx]" or, for poetry users poetry add aiodynamo -E httpx

Connect to DynamoDB

from aiodynamo.client import Client
from aiodynamo.credentials import Credentials
from aiodynamo.http.httpx import HTTPX
from httpx import AsyncClient

    async with AsyncClient() as h:
        client = Client(HTTPX(h), Credentials.auto(), "us-east-1")

With aiohttp

Install this library

pip install "aiodynamo[aiohttp]" or, for poetry users poetry add aiodynamo -E aiohttp

Connect to DynamoDB

from aiodynamo.client import Client
from aiodynamo.credentials import Credentials
from aiodynamo.http.aiohttp import AIOHTTP
from aiohttp import ClientSession

    async with ClientSession() as session:
        client = Client(AIOHTTP(session), Credentials.auto(), "us-east-1")

API use

        table = client.table("my-table")

        # Create table if it doesn't exist
        if not await table.exists():
            await table.create(
                Throughput(read=10, write=10),
                KeySchema(hash_key=KeySpec("key", KeyType.string)),
            )

        # Create or override an item
        await table.put_item({"key": "my-item", "value": 1})
        # Get an item
        item = await table.get_item({"key": "my-item"})
        print(item)
        # Update an item, if it exists.
        await table.update_item(
            {"key": "my-item"}, F("value").add(1), condition=F("key").exists()
        )

Why aiodynamo

  • boto3 and botocore are synchronous. aiodynamo is built for asynchronous apps.
  • aiodynamo is fast. Two times faster than aiobotocore, botocore or boto3 for operations such as query or scan.
  • aiobotocore is very low level. aiodynamo provides a pythonic API, using modern Python features. For example, paginated APIs are automatically depaginated using asynchronous iterators.
  • Legible source code. botocore and derived libraries generate their interface at runtime, so it cannot be inspected and isn't typed. aiodynamo is hand written code you can read, inspect and understand.
  • Pluggable HTTP client. If you're already using an asynchronous HTTP client in your project, you can use it with aiodynamo and don't need to add extra dependencies or run into dependency resolution issues.

Complete documentation is here

Comments
  • question: Any chance for transaction support?

    question: Any chance for transaction support?

    I'm wondering if there are any plans to support TransactWriteItems or just Transactions in general.

    I can probably work on this if/when I have more time 🤔

    opened by stupoid 16
  • add AWS_PROFILE env var to credentials chain

    add AWS_PROFILE env var to credentials chain

    Adds support for using AWS_PROFILE env var in the credentials chain.
    This behavior is more in line with the behavior expected by users coming from boto3 or aiobotocore. Added a unit test to cover this feature.

    Relates to: Issue #139

    opened by BTripp1986 8
  • Http wrapping issues

    Http wrapping issues

    When I was adding credentials handler to assume role with web identity which we need for our EKS setup I came across few issues why I am for now implementing it as custom code in our own project instead of providing it for the library.

    Http wrappers assume response is json, https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py but as endpoint response is xml (and at least I can't find any mention of possibility to request it as json, but you know AWS docs 💩 ). This is blocker and thought that perhaps there needs to be some refactoring around http handling which takes too long for us.

    Some additional things I noticed

    1. in MetadataCredentials there is fetch_with_retry which couldn't use because it assumes GET but endpoint for assume role is POST. Was thinking should/could this retry option be in base http classes?

    2. Missing timeout in POST, would prefer to have timeout also for this credentials call. As mentioned in https://github.com/HENNGE/aiodynamo/issues/45

    3. http.post(..) requires body as bytes even when it can in aiohttp and httpx also be None. https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py#L28-L30

    opened by jarikujansuu 6
  • Add support for pay_per_request billing mode during table creation

    Add support for pay_per_request billing mode during table creation

    Currently aiodynamo creates tables using the provisioned billing mode. DynamoDB also supports pay_per_request where no throughput config is provided. This PR is backwards compatible, but also adds the support to provide a billing_mode parameter to create_table. The throughput parameter has been changed to optional as the pay_per_request billing mode doesn't use it.

    PS. I'd be very grateful if you could also add the hacktoberfest-accepted label to this PR once it is accepted so that it can be counted as a hacktoberfest PR. Thanks :)

    hacktoberfest-accepted 
    opened by TheEdgeOfRage 6
  • Handle ClientConnectorError in connector

    Handle ClientConnectorError in connector

    Exception text: ClientConnectorError: Cannot connect to host dynamodb.<snip>.amazonaws.com:443 ssl:default [None]

    Exception type: aiohttp.client_exceptions.ClientConnectorError

    Stack:

    aiohttp/connector.py in _wrap_create_connection at line 943
    aiohttp/connector.py in _create_direct_connection at line 980
    aiohttp/connector.py in _create_direct_connection at line 1004
    aiohttp/connector.py in _create_connection at line 858
    aiohttp/connector.py in connect at line 523
    aiohttp/client.py in _request at line 480
    aiohttp/client.py in __aenter__ at line 1012
    aiodynamo/http/aiohttp.py in post at line 31
    aiodynamo/client.py in send_request at line 651
    aiodynamo/client.py in get_item at line 454
    aiodynamo/client.py in get_item at line 124
    
    opened by dimaqq 6
  • Raise asyncio.TimeoutError on httpx.TimeoutException

    Raise asyncio.TimeoutError on httpx.TimeoutException

    Now asyncio.TimeoutError is handled in the client logic. But httpx wrapper raises httpx.TimeoutException. This pull request convert the exception and make the client be able to handle timeouts in httpx.

    ref.) httpx timeout exceptions https://github.com/encode/httpx/blob/master/httpx/_exceptions.py#L7-L11

    ref.) Custom HTTP Client Adaptor document https://github.com/HENNGE/aiodynamo/blob/2e6c4716a3ac9fe5669bbdcaa73e6bbe0f73cfbb/docs/advanced.rst

    opened by gunyarakun 5
  • Count limits

    Count limits

    This was motivated by our real-life use case. We sometimes need to limit counting to preserve latency and Dynamo resources. I tried using query_single_page but it doesn't support count queries, so I just added limit to count.

    opened by Tinche 5
  • Use 3rd-party ddbcereal library for deserialize/serialize.

    Use 3rd-party ddbcereal library for deserialize/serialize.

    Figured I'd put this out here as an option. I've been working on a serializer/deserializer lib to supplement my use of aiobotocore, but it might make sense for this project too. It should be as fast or faster than this one.

    Overall, it reduces/simplifies the code, but the initialization of Client is a tad uglier due to ddbcereal needing a pre-constructed Deserializer and Client being frozen. It could be made pretty by unfreezing it or alternatively, making a seperate client-spawning function that passes the Deserializer to the Client constructor.

    opened by JustinTArthur 5
  • Don't send `StreamSpecification: null`

    Don't send `StreamSpecification: null`

    https://github.com/HENNGE/aiodynamo/blob/81b6f98111ce4bed41aeb8d07747695af836c76a/src/aiodynamo/client.py#L41-L49

    Sends StreamSpecification=None when none is supplied by the caller. Perhaps aiodynamo should not inject this key if the value is None.

    It seems that's OK for AWS SaaS dynamo, dynamodb-local and dynalite, but not ScyllaDB:

    https://github.com/scylladb/scylla/issues/5796

    opened by dimaqq 5
  • How to update map element using a pattern?

    How to update map element using a pattern?

    Described in https://dev.to/matthewvielkind/updating-values-in-dyanmodb-map-attributes-can

    UpdateExpression="SET players.#player_id.score = :score_val",  
    ExpressionAttributeNames={"#player_id": player_id},  
    

    I'm not sure what the API is in this case... Would I need to overload F?

    opened by dimaqq 4
  • Support consistent read

    Support consistent read

    Purpose

    Support consistent read in get_item, query, scan, count and batch get operations.

    Note: Tests do not actually check anything except that DynamoDb calls do not explode if using consistent_read option. Local DynamoDb is always strongly consistent. And might want to reduce copied code in them.

    opened by jarikujansuu 4
  • Empty-set safety

    Empty-set safety

    aiodynamo used to be empty-string-safe when DynamoDB didn't support empty string values. This mercifully got fixed in DynamoDB, but DynamoDB still doesn't accept empty sets (NS, SS, BS). There's an argument to be made that aiodynamo should handle this automatically, converting F("some_set").set(set()) to F("some_set").remove(), possibly with a warning.

    opened by ojii 3
  • Shall we try goodfirstissue.dev ?

    Shall we try goodfirstissue.dev ?

    links: https://goodfirstissue.dev/language/python https://github.com/deepsourcelabs/good-first-issue#adding-a-new-project

    I think this project may qualify, and would get some exposure and helping hands

    opened by dimaqq 0
  • Support AWS Web Identity Token File authentication

    Support AWS Web Identity Token File authentication

    aiodynamo ~~doesn't support AWS Web Identity Token File authentication~~ doesn't support AWS Web Identity Token File authentication out of the box.

    It looks like a typical case. For my team, we would like to use that authentication in our services in k8s by using IAM Roles for Service Accounts(IRSA)

    Related issue: https://github.com/HENNGE/aiodynamo/issues/128 Related PR: https://github.com/HENNGE/aiodynamo/pull/127

    There was a similar issue, but I decided to create another one because I think we can describe our goal and issue more directly.

    So, as discovered in a related PR, we should implement the AssumeRoleWithWebIdentity API for this authentication.

    opened by mrkovalchuk 2
  • Auth Error With No Default Credentials

    Auth Error With No Default Credentials

    I have access to several AWS accounts via SSO and no default profile for AWS. When I sign in to AWS via SSO the .aws/credentials file is updated with a fresh Access Key, Secret Key, and Security Token.

    I specify which credentials I will use with the AWS_PROFILE env var, which is checked as part of the credentials chain lookup. If using boto3 or the aws-cli this method works. With this method I can sign in via SSO to dev and prod accounts. Then if I AWS_PROFILE=dev aws s3 ls(or similar call in boto3) it will run the command against the dev account. Then if I AWS_PROFILE=prod aws s3 ls it will run against prod.

    This doesn't seem to work with aiodynamo. In order to get it to work I need to create a default profile and then copy/paste the credentials from the desired profile into the default profile. Aiodynamo should respect the AWS_PROFILE env var.

    opened by BTripp1986 1
  • Surprising behaviour in `Client.table_exists`

    Surprising behaviour in `Client.table_exists`

    We use Table.exists as part of a health check for the containers in our application, which in turn calls Client.table_exists. Today we had a short outage when all health checks began failing. It turned out the Dynamo table had entered UPDATING status.

    The implementation does the following:

    async def table_exists(self, name: TableName) -> bool:
            try:
                description = await self.describe_table(name)
            except TableNotFound:
                return False
    
            return description.status is TableStatus.active
    

    This feels like a bug to us: UPDATING is a valid state while a variety of operations are being applied to the table. So the code should do return description.status in [TableStatus.active, TableStatus.updating].

    Thoughts?

    opened by darylweir 1
Releases(21.9)
  • 21.9(Sep 1, 2021)

  • 21.8(Aug 24, 2021)

  • 21.7(Aug 24, 2021)

  • 21.6(Aug 24, 2021)

  • 21.5(Aug 24, 2021)

  • 20.4.2(Apr 21, 2020)

    Release Date: April 15th, 2020

    • Fix comparison conditions (equals, not_equals, gt, gte, lt, lte on aiodynamo.expressions.F and aiodynamo.expressions.Size via aiodynamo.expressions.F.size() to support referencing other fields (using aiodynamo.expressions.F)
    • Fix timeout handling in aiohttp based client.
    Source code(tar.gz)
    Source code(zip)
  • 20.4.1(Apr 21, 2020)

    Release Date: April 13th, 2020

    • Fixed put_item and delete_item with a condition which does not carry any values.
    • Wrap underlying HTTP client errors, such as connection issues, so networking issues during requests are retried.
    Source code(tar.gz)
    Source code(zip)
  • 20.4(Apr 21, 2020)

    Release Date: April 3rd, 2020

    • Fixed scan with a projection but no filter_expression.
    • Fixed logs leaking session tokens (request sending) and keys (metadata fetch).
    Source code(tar.gz)
    Source code(zip)
  • 20.3(Apr 21, 2020)

    Release Date: March 31st, 2020

    • Added TTL support
    • Added support for pluggable HTTP clients. Built in support for httpx and aiohttp.
    • Added custom client implementation.
    • Added custom credentials loaders, with support for custom credential loaders.
    • Fixed a typo in delete_item
    • Improved item deserialization performance
    • Improved overall client performance, especially for query, scan and count, which are now up to twice as fast.
    • Changed condition, key condition and filter expression APIs to not rely on boto3.
    • Moved aiodynamo.models.F to aiodynamo.expressions.F.
    • Removed boto3 dependency
    • Removed botocore dependency
    • Removed aiobotocore dependency
    Source code(tar.gz)
    Source code(zip)
  • 19.9(Apr 21, 2020)

  • 19.3(Mar 6, 2019)

  • 19.1(Jan 22, 2019)

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

AWS Data Wrangler Pandas on AWS Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretMana

Amazon Web Services - Labs 3.3k Dec 31, 2022
Dinamopy is a python helper library for dynamodb

Dinamopy is a python helper library for dynamodb. You can define your access patterns in a json file and can use dynamic method names to make operations.

Rasim Andıran 2 Jul 18, 2022
Asynchronous Python client for InfluxDB

aioinflux Asynchronous Python client for InfluxDB. Built on top of aiohttp and asyncio. Aioinflux is an alternative to the official InfluxDB Python cl

Gustavo Bezerra 159 Dec 27, 2022
A Pythonic, object-oriented interface for working with MongoDB.

PyMODM MongoDB has paused the development of PyMODM. If there are any users who want to take over and maintain this project, or if you just have quest

mongodb 345 Dec 25, 2022
A library for python made by me,to make the use of MySQL easier and more pythonic

my_ezql A library for python made by me,to make the use of MySQL easier and more pythonic This library was made by Tony Hasson , a 25 year old student

null 3 Nov 19, 2021
A fast PostgreSQL Database Client Library for Python/asyncio.

asyncpg -- A fast PostgreSQL Database Client Library for Python/asyncio asyncpg is a database interface library designed specifically for PostgreSQL a

magicstack 5.8k Dec 31, 2022
Asynchronous interface for peewee ORM powered by asyncio

peewee-async Asynchronous interface for peewee ORM powered by asyncio. Important notes Since version 0.6.0a only peewee 3.5+ is supported If you still

05Bit 666 Dec 30, 2022
Pure Python MySQL Client

PyMySQL Table of Contents Requirements Installation Documentation Example Resources License This package contains a pure-Python MySQL client library,

PyMySQL 7.2k Jan 9, 2023
Python client for Apache Kafka

Kafka Python client Python client for the Apache Kafka distributed stream processing system. kafka-python is designed to function much like the offici

Dana Powers 5.1k Jan 8, 2023
Redis Python Client

redis-py The Python interface to the Redis key-value store. Python 2 Compatibility Note redis-py 3.5.x will be the last version of redis-py that suppo

Andy McCurdy 11k Dec 29, 2022
Redis client for Python asyncio (PEP 3156)

Redis client for Python asyncio. Redis client for the PEP 3156 Python event loop. This Redis library is a completely asynchronous, non-blocking client

Jonathan Slenders 554 Dec 4, 2022
CouchDB client built on top of aiohttp (asyncio)

aiocouchdb source: https://github.com/aio-libs/aiocouchdb documentation: http://aiocouchdb.readthedocs.org/en/latest/ license: BSD CouchDB client buil

aio-libs 53 Apr 5, 2022
Google Cloud Client Library for Python

Google Cloud Python Client Python idiomatic clients for Google Cloud Platform services. Stability levels The development status classifier on PyPI ind

Google APIs 4.1k Jan 1, 2023
Official Python low-level client for Elasticsearch

Python Elasticsearch Client Official low-level client for Elasticsearch. Its goal is to provide common ground for all Elasticsearch-related code in Py

elastic 3.8k Jan 1, 2023
python-bigquery Apache-2python-bigquery (🥈34 · ⭐ 3.5K · 📈) - Google BigQuery API client library. Apache-2

Python Client for Google BigQuery Querying massive datasets can be time consuming and expensive without the right hardware and infrastructure. Google

Google APIs 550 Jan 1, 2023
google-cloud-bigtable Apache-2google-cloud-bigtable (🥈31 · ⭐ 3.5K) - Google Cloud Bigtable API client library. Apache-2

Python Client for Google Cloud Bigtable Google Cloud Bigtable is Google's NoSQL Big Data database service. It's the same database that powers many cor

Google APIs 39 Dec 3, 2022
High level Python client for Elasticsearch

Elasticsearch DSL Elasticsearch DSL is a high-level library whose aim is to help with writing and running queries against Elasticsearch. It is built o

elastic 3.6k Jan 3, 2023
Python client for InfluxDB

InfluxDB-Python InfluxDB-Python is a client for interacting with InfluxDB. Development of this library is maintained by: Github ID URL @aviau (https:/

InfluxData 1.6k Dec 24, 2022
Confluent's Kafka Python Client

Confluent's Python Client for Apache KafkaTM confluent-kafka-python provides a high-level Producer, Consumer and AdminClient compatible with all Apach

Confluent Inc. 3.1k Jan 5, 2023