Bringing Async Capabilities to django ORM

Overview

Disclaimer: Don't use this module in production it's still in active development.

Django Async Orm

Django module that brings async to django ORM.

Installing

python -m pip install django-async-orm

then add django_async_orm to your INSTALLED_APPS list:

INSTALLED_APPS = [
    ...,
    'django_async_orm'
]

Usage

Django Async Orm will patch all your existing models to add async_* prefixed methods. To be

example:

class MyModel(models.Model):
    name = models.CharField(max_length=250)

you can use it as follow:

async def get_model():
    return await  MyModel.objects.async_get(name="something")

you can also iterate over a query set with async for:

async def all_models():
    all_result_set = await MyModel.objects.async_all()
    async for obj in all_result_set:
        print(obj)

Some wrappers are also available for template rendering, form validation and login/logout

Async login

from django_async_orm.wrappers import async_login

async def my_async_view(request):
    await async_login(request)
    ...

Form validation

from django_async_orm.wrappers import async_form_is_valid
async def a_view(request):
    form = MyForm(request.POST)
    is_valid_form = await async_form_is_valid(form)
    if is_valid_form:
        ...
    

Django ORM support:

This is an on going projects, not all model methods are ported.

Manager:

methods supported comments
Model.objects.async_get
Model.objects.async_create
Model.objects.async_bulk_create
Model.objects.async_bulk_update
Model.objects.async_get_or_create
Model.objects.async_update_or_create
Model.objects.async_earliest
Model.objects.async_latest
Model.objects.async_first
Model.objects.async_last
Model.objects.async_in_bulk
Model.objects.async_delete
Model.objects.async_update
Model.objects.async_exists
Model.objects.async_explain
Model.objects.async_raw
Model.objects.async_all
Model.objects.async_filter
Model.objects.async_exclude
Model.objects.async_complex_filter
Model.objects.async_union
Model.objects.async_intersection
Model.objects.async_difference
Model.objects.async_select_for_update
Model.objects.async_prefetch_related
Model.objects.async_annotate
Model.objects.async_order_by
Model.objects.async_distinct
Model.objects.async_difference
Model.objects.async_extra
Model.objects.async_reverse
Model.objects.async_defer
Model.objects.async_only
Model.objects.async_using
Model.objects.async_resolve_expression
Model.objects.async_ordered
__aiter__
__repr__
__len__
__getitem__
Model.objects.async_iterator

Model:

methods supported comments
Model.async_save
Model.async_update
Model.async_delete
...

User Model / Manager

methods supported comments
UserModel.is_authenticated
UserModel.is_super_user
UserModel.objects.async_create_user
...

Foreign object lazy loading:

Not supported

Wrappers:

methods supported comments
wrappers.async_render
wrappers.async_login
wrappers.async_logout
Comments
  • add none query feature and test for the same

    add none query feature and test for the same

    Added none query feature:

    in Django ORM, calling none() will create a queryset that never returns any objects and no query will be executed when accessing the results. A qs.none() queryset is an instance of EmptyQuerySet.

    example:

    Entry.objects.none() <QuerySet []> from django.db.models.query import EmptyQuerySet isinstance(Entry.objects.none(), EmptyQuerySet) True

    opened by rkisdp 4
  • replace the default ThreadPoolExecutor with gevent.threadpool.ThreadPoolExecutor

    replace the default ThreadPoolExecutor with gevent.threadpool.ThreadPoolExecutor

    I have no idea if this will work, but maybe it will?

    Here's a bit of my research on what I was thinking: https://github.com/rednaks/django-async-orm/discussions/9

    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabce2e9d0>
    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabf109bb0>
    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabf109250>
    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabc5654f0>
    web_1         | DEBUG:root:concurrent.futures.Future is expected, got <gevent.threadpool._FutureProxy object at 0x7fcabd7846d0>
    ^CGracefully stopping... (press Ctrl+C again to force)
    Stopping benchmark-django-fastapi_web_1      ... done
    Stopping benchmark-django-fastapi_postgres_1 ... done
    
    

    looks kind of close ... although something is expecting concurrent.futures.Future instead of also allowing gevent.threadpool._FutureProxy

    update

    a recursion error of some sort logged from here : https://github.com/rednaks/django-async-orm/pull/7/files#diff-142ac986547c5b13f77ae025cb60609eed65082cc41ac6e81f4b778422d50017R116

    recursion.error.txt

    opened by allen-munsch 4
  • TypeError: 'NoneType' object is not iterable

    TypeError: 'NoneType' object is not iterable

    Thanks for sharing this library. Very interesting!

    Any suggestions on why NoneType would be returned from async_all?

    web              |   File "/venv/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
    web              |     await super().__call__(scope, receive, send)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
    web              |     await self.middleware_stack(scope, receive, send)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    web              |     raise exc from None
    web              |   File "/venv/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    web              |     raise exc from None
    web              |   File "/venv/lib/python3.8/site-packages/starlette/exceptions.py", line 93, in __call__
    web              |     await response(scope, receive, sender)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/exceptions.py", line 93, in __call__
    web              |     await response(scope, receive, sender)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/routing.py", line 580, in __call__
    web              |     await route.handle(scope, receive, send)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/routing.py", line 241, in handle
    web              |     await self.app(scope, receive, send)
    web              |   File "/venv/lib/python3.8/site-packages/starlette/routing.py", line 52, in app
    web              |     response = await func(request)
    web              |   File "/venv/lib/python3.8/site-packages/fastapi/routing.py", line 201, in app
    web              |     raw_response = await run_endpoint_function(
    web              |   File "/venv/lib/python3.8/site-packages/fastapi/routing.py", line 148, in run_endpoint_function
    web              |     return await dependant.call(**values)
    web              |   File "/app/api/generated_app/main.py", line 100, in get_buildings
    web              |     data = await get_data()
    web              |   File "/app/api/generated_app/main.py", line 88, in get_data
    web              |     async for building in data:
    web              |   File "/venv/lib/python3.8/site-packages/django_async_orm/query.py", line 71, in __aiter__
    web              |     return AsyncIter(self._result_cache)
    web              |   File "/venv/lib/python3.8/site-packages/django_async_orm/iter.py", line 6, in __init__
    web              |     self._iter = iter(iterable)
    web              | TypeError: 'NoneType' object is not iterable
    

    py3.8.10 django3.2

    async def get_data():
        data = await Building.objects.async_all()
        result = []
        async for building in data:
            print(data, dir(data))
            print(building.__dict__)
            result.append({"id": building.id, "number": building.number})
        return result
    
    @app.get("/buildings", response_model=List[BuildingSummary])
    async def get_buildings(limit: conint(le=500), skip: conint(le=500)) -> List[BuildingSummary]:
        """
        Building List
        """
        # this is a custom thing
        data = await get_data()
        return data
    
    opened by allen-munsch 3
  • PR for issue #2

    PR for issue #2

    async def get_data():
        return await Building.objects.async_all()
    
    @app.get("/buildings", response_model=List[BuildingSummary])
    async def get_buildings(
        limit: conint(le=500),
        skip: conint(le=500),
        x_pm_org: Optional[List[int]] = Header(None, alias="x-pm-org"),
        x_pm_org_id: Optional[List[int]] = Header(None, alias="x-pm-org-id"),
        x_pm_user_type: Optional[str] = Header(None, alias="x-pm-user-type"),
    ) -> List[BuildingSummary]:
        """
        Building List
        [{'ApiKeyAuth': []}, {'PmOrg': []}]
        """
    
        org.set(x_pm_org)
        pydant, item_all = paths["get_buildings"]
    
    
    
        # this works
        data = await get_data()
    
        # this also works
        data2 = await Buildings.objects.async_all()
    
    
    
        result = []
        async for item in data[skip:limit]:
            print(item.__dict__)
            result.append(pydant.from_orm(item).dict())
        return result
    

    Related to: #2

    I just tested it locally and it appears to work.

    Thanks for the quick response.

    Not sure where to ask this, possibly could enable the "discussions" section on the repo?

    But how does the ThreadPoolExecutor work with something like psycogreen/eventlet?

    opened by allen-munsch 1
  • comply with official django async orm syntax

    comply with official django async orm syntax

    when we patch models, we prefix the methods with async_ but the new official django api prefixes the methods with a ex:

    patched api : sync objects.get => async objects.async_get official api: sync objects.get => async objects.aget

    so to make it forward compatible we need to comply to django's new api.

    be aware to keep our api so we don't break the internet

    opened by rednaks 0
  • Problem with default update_or_create()

    Problem with default update_or_create()

    Hello! After installing django_async_orm facing such problem with default update_or_create() function

    image

    Do you have any ideas by a chance why is it going like this?

    python 3.8 django 3.2.9 django-async-orm latest

    opened by max4nik 0
  • Count feature, related test and import optimisation

    Count feature, related test and import optimisation

    Hi, @rednaks,

    I have added the count feature and test for the same, and with that, I have also changed some other tests accordingly and removed unused imports from the file. Please have a look and give your feedback.

    opened by rkisdp 8
  • Perhaps it is better to use channels?

    Perhaps it is better to use channels?

    Perhaps it's worth starting django-channels library for asynchronous db calls?

    There is a function there. And, as I understand it, it will be more effective than it is now. image

    opened by daveusa31 0
Releases(0.1.4)
An async ORM. 🗃

ORM The orm package is an async ORM for Python, with support for Postgres, MySQL, and SQLite. ORM is built with: SQLAlchemy core for query building. d

Encode 1.7k Dec 28, 2022
The ormar package is an async mini ORM for Python, with support for Postgres, MySQL, and SQLite.

python async mini orm with fastapi in mind and pydantic validation

null 1.2k Jan 5, 2023
Pydantic model support for Django ORM

Pydantic model support for Django ORM

Jordan Eremieff 318 Jan 3, 2023
The Orator ORM provides a simple yet beautiful ActiveRecord implementation.

Orator The Orator ORM provides a simple yet beautiful ActiveRecord implementation. It is inspired by the database part of the Laravel framework, but l

Sébastien Eustace 1.4k Jan 1, 2023
a small, expressive orm -- supports postgresql, mysql and sqlite

peewee Peewee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use. a small, expressive ORM p

Charles Leifer 9.7k Jan 8, 2023
Piccolo - A fast, user friendly ORM and query builder which supports asyncio.

A fast, user friendly ORM and query builder which supports asyncio.

null 919 Jan 4, 2023
Solrorm : A sort-of solr ORM for python

solrorm : A sort-of solr ORM for python solrpy - deprecated solrorm - currently in dev Usage Cores The first step to interact with solr using solrorm

Aj 1 Nov 21, 2021
A PostgreSQL or SQLite orm for Python

Prom An opinionated lightweight orm for PostgreSQL or SQLite. Prom has been used in both single threaded and multi-threaded environments, including en

Jay Marcyes 18 Dec 1, 2022
A simple project to explore the number of GCs when doing basic ORM work.

Question: Does Python do extremely too many GCs for ORMs? YES, OMG YES. Check this out Python Default GC Settings: SQLAlchemy - 20,000 records in one

Michael Kennedy 26 Jun 5, 2022
A dataclasses-based ORM framework

dcorm A dataclasses-based ORM framework. [WIP] - Work in progress This framework is currently under development. A first release will be announced in

HOMEINFO - Digitale Informationssysteme GmbH 1 Dec 24, 2021
ORM for Python for PostgreSQL.

New generation (or genius) ORM for Python for PostgreSQL. Fully-typed for any query with Pydantic and auto-model generation, compatible with any sync or async driver

Yan Kurbatov 3 Apr 13, 2022
A new ORM for Python specially for PostgreSQL

A new ORM for Python specially for PostgreSQL. Fully-typed for any query with Pydantic and auto-model generation, compatible with any sync or async driver

Yan Kurbatov 3 Apr 13, 2022
Tortoise ORM is an easy-to-use asyncio ORM inspired by Django.

Tortoise ORM was build with relations in mind and admiration for the excellent and popular Django ORM. It's engraved in it's design that you are working not with just tables, you work with relational data.

Tortoise 3.3k Jan 7, 2023
GINO Is Not ORM - a Python asyncio ORM on SQLAlchemy core.

GINO - GINO Is Not ORM - is a lightweight asynchronous ORM built on top of SQLAlchemy core for Python asyncio. GINO 1.0 supports only PostgreSQL with

GINO Community 2.5k Dec 27, 2022
Bringing together django, django rest framework, and htmx

This is Just an Idea There is no code, this README just represents an idea for a minimal library that, as of now, does not exist. django-htmx-rest A l

Jack DeVries 5 Nov 24, 2022
An async ORM. 🗃

ORM The orm package is an async ORM for Python, with support for Postgres, MySQL, and SQLite. ORM is built with: SQLAlchemy core for query building. d

Encode 1.7k Dec 28, 2022
The ormar package is an async mini ORM for Python, with support for Postgres, MySQL, and SQLite.

python async mini orm with fastapi in mind and pydantic validation

null 1.2k Jan 5, 2023
Async ORM based on PyPika

PyPika-ORM - ORM for PyPika SQL Query Builder The package gives you ORM for PyPika with asycio support for a range of databases (SQLite, PostgreSQL, M

Kirill Klenov 7 Jun 4, 2022
Async timeit - Async version of python's timeit

Async Timeit Replica of default python timeit module with small changes to allow

Raghava G Dhanya 3 Apr 13, 2022
Automatic caching and invalidation for Django models through the ORM.

Cache Machine Cache Machine provides automatic caching and invalidation for Django models through the ORM. For full docs, see https://cache-machine.re

null 846 Nov 26, 2022