The ormar package is an async mini ORM for Python, with support for Postgres, MySQL, and SQLite.

Overview

ormar

Pypi version Pypi version Build Status Coverage CodeFactor

Overview

The ormar package is an async mini ORM for Python, with support for Postgres, MySQL, and SQLite.

The main benefit of using ormar are:

  • getting an async ORM that can be used with async frameworks (fastapi, starlette etc.)
  • getting just one model to maintain - you don't have to maintain pydantic and other orm model (sqlalchemy, peewee, gino etc.)

The goal was to create a simple ORM that can be used directly (as request and response models) with fastapi that bases it's data validation on pydantic.

Ormar - apart form obvious ORM in name - get it's name from ormar in swedish which means snakes, and ormar(e) in italian which means cabinet.

And what's a better name for python ORM than snakes cabinet :)

Documentation

Check out the documentation for details.

Note that for brevity most of the documentation snippets omit the creation of the database and scheduling the execution of functions for asynchronous run.

If you want more real life examples than in the documentation you can see tests folder, since they actually have to create and connect to database in most of the tests.

Yet remember that those are - well - tests and not all solutions are suitable to be used in real life applications.

Part of the fastapi ecosystem

As part of the fastapi ecosystem ormar is supported in libraries that somehow work with databases.

As of now ormar is supported by:

If you maintain or use different library and would like it to support ormar let us know how we can help.

Dependencies

Ormar is built with:

Migrating from sqlalchemy

If you currently use sqlalchemy and would like to switch to ormar check out the auto-translation tool that can help you with translating existing sqlalchemy orm models so you do not have to do it manually.

Beta versions available at github: sqlalchemy-to-ormar or simply pip install sqlalchemy-to-ormar

Migrations & Database creation

Because ormar is built on SQLAlchemy core, you can use alembic to provide database migrations (and you really should for production code).

For tests and basic applications the sqlalchemy is more than enough:

# note this is just a partial snippet full working example below
# 1. Imports
import sqlalchemy
import databases

# 2. Initialization
DATABASE_URL = "sqlite:///db.sqlite"
database = databases.Database(DATABASE_URL)
metadata = sqlalchemy.MetaData()

# Define models here

# 3. Database creation and tables creation
engine = sqlalchemy.create_engine(DATABASE_URL)
metadata.create_all(engine)

For a sample configuration of alembic and more information regarding migrations and database creation visit migrations documentation section.

Package versions

ormar is still under development: We recommend pinning any dependencies (with i.e. ormar~=0.9.1)

ormar also follows the release numeration that breaking changes bump the major number, while other changes and fixes bump minor number, so with the latter you should be safe to update, yet always read the releases docs before. example: (0.5.2 -> 0.6.0 - breaking, 0.5.2 -> 0.5.3 - non breaking).

Asynchronous Python

Note that ormar is an asynchronous ORM, which means that you have to await the calls to the methods, that are scheduled for execution in an event loop. Python has a builtin module asyncio that allows you to do just that.

Note that most of "normal" python interpreters do not allow execution of await outside of a function (cause you actually schedule this function for delayed execution and don't get the result immediately).

In a modern web frameworks (like fastapi), the framework will handle this for you, but if you plan to do this on your own you need to perform this manually like described in a quick start below.

Quick Start

Note that you can find the same script in examples folder on github.

from typing import Optional

import databases
import pydantic

import ormar
import sqlalchemy

DATABASE_URL = "sqlite:///db.sqlite"
database = databases.Database(DATABASE_URL)
metadata = sqlalchemy.MetaData()


# note that this step is optional -> all ormar cares is a internal
# class with name Meta and proper parameters, but this way you do not
# have to repeat the same parameters if you use only one database
class BaseMeta(ormar.ModelMeta):
    metadata = metadata
    database = database


# Note that all type hints are optional
# below is a perfectly valid model declaration
# class Author(ormar.Model):
#     class Meta(BaseMeta):
#         tablename = "authors"
#
#     id = ormar.Integer(primary_key=True) # <= notice no field types
#     name = ormar.String(max_length=100)

class Author(ormar.Model):
    class Meta(BaseMeta):
        tablename = "authors"

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)


class Book(ormar.Model):
    class Meta(BaseMeta):
        tablename = "books"

    id: int = ormar.Integer(primary_key=True)
    author: Optional[Author] = ormar.ForeignKey(Author)
    title: str = ormar.String(max_length=100)
    year: int = ormar.Integer(nullable=True)


# create the database
# note that in production you should use migrations
# note that this is not required if you connect to existing database
engine = sqlalchemy.create_engine(DATABASE_URL)
# just to be sure we clear the db before
metadata.drop_all(engine)
metadata.create_all(engine)


# all functions below are divided into functionality categories
# note how all functions are defined with async - hence can use await AND needs to
# be awaited on their own
async def create():
    # Create some records to work with through QuerySet.create method.
    # Note that queryset is exposed on each Model's class as objects
    tolkien = await Author.objects.create(name="J.R.R. Tolkien")
    await Book.objects.create(author=tolkien,
                              title="The Hobbit",
                              year=1937)
    await Book.objects.create(author=tolkien,
                              title="The Lord of the Rings",
                              year=1955)
    await Book.objects.create(author=tolkien,
                              title="The Silmarillion",
                              year=1977)

    # alternative creation of object divided into 2 steps
    sapkowski = Author(name="Andrzej Sapkowski")
    # do some stuff
    await sapkowski.save()

    # or save() after initialization
    await Book(author=sapkowski, title="The Witcher", year=1990).save()
    await Book(author=sapkowski, title="The Tower of Fools", year=2002).save()

    # to read more about inserting data into the database
    # visit: https://collerek.github.io/ormar/queries/create/


async def read():
    # Fetch an instance, without loading a foreign key relationship on it.
    # Django style
    book = await Book.objects.get(title="The Hobbit")
    # or python style
    book = await Book.objects.get(Book.title == "The Hobbit")
    book2 = await Book.objects.first()

    # first() fetch the instance with lower primary key value
    assert book == book2

    # you can access all fields on loaded model
    assert book.title == "The Hobbit"
    assert book.year == 1937

    # when no condition is passed to get()
    # it behaves as last() based on primary key column
    book3 = await Book.objects.get()
    assert book3.title == "The Tower of Fools"

    # When you have a relation, ormar always defines a related model for you
    # even when all you loaded is a foreign key value like in this example
    assert isinstance(book.author, Author)
    # primary key is populated from foreign key stored in books table
    assert book.author.pk == 1
    # since the related model was not loaded all other fields are None
    assert book.author.name is None

    # Load the relationship from the database when you already have the related model
    # alternatively see joins section below
    await book.author.load()
    assert book.author.name == "J.R.R. Tolkien"

    # get all rows for given model
    authors = await Author.objects.all()
    assert len(authors) == 2

    # to read more about reading data from the database
    # visit: https://collerek.github.io/ormar/queries/read/


async def update():
    # read existing row from db
    tolkien = await Author.objects.get(name="J.R.R. Tolkien")
    assert tolkien.name == "J.R.R. Tolkien"
    tolkien_id = tolkien.id

    # change the selected property
    tolkien.name = "John Ronald Reuel Tolkien"
    # call update on a model instance
    await tolkien.update()

    # confirm that object was updated
    tolkien = await Author.objects.get(name="John Ronald Reuel Tolkien")
    assert tolkien.name == "John Ronald Reuel Tolkien"
    assert tolkien.id == tolkien_id

    # alternatively update data without loading
    await Author.objects.filter(name__contains="Tolkien").update(name="J.R.R. Tolkien")

    # to read more about updating data in the database
    # visit: https://collerek.github.io/ormar/queries/update/


async def delete():
    silmarillion = await Book.objects.get(year=1977)
    # call delete() on instance
    await silmarillion.delete()

    # alternatively delete without loading
    await Book.objects.delete(title="The Tower of Fools")

    # note that when there is no record ormar raises NoMatch exception
    try:
        await Book.objects.get(year=1977)
    except ormar.NoMatch:
        print("No book from 1977!")

    # to read more about deleting data from the database
    # visit: https://collerek.github.io/ormar/queries/delete/

    # note that despite the fact that record no longer exists in database
    # the object above is still accessible and you can use it (and i.e. save()) again.
    tolkien = silmarillion.author
    await Book.objects.create(author=tolkien,
                              title="The Silmarillion",
                              year=1977)


async def joins():
    # Tho join two models use select_related
    book = await Book.objects.select_related("author").get(title="The Hobbit")
    # now the author is already prefetched
    assert book.author.name == "J.R.R. Tolkien"

    # By default you also get a second side of the relation
    # constructed as lowercase source model name +'s' (books in this case)
    # you can also provide custom name with parameter related_name
    author = await Author.objects.select_related("books").all(name="J.R.R. Tolkien")
    assert len(author[0].books) == 3

    # for reverse and many to many relations you can also prefetch_related
    # that executes a separate query for each of related models

    author = await Author.objects.prefetch_related("books").get(name="J.R.R. Tolkien")
    assert len(author.books) == 3

    # to read more about relations
    # visit: https://collerek.github.io/ormar/relations/

    # to read more about joins and subqueries
    # visit: https://collerek.github.io/ormar/queries/joins-and-subqueries/


async def filter_and_sort():
    # to filter the query you can use filter() or pass key-value pars to
    # get(), all() etc.
    # to use special methods or access related model fields use double
    # underscore like to filter by the name of the author use author__name
    # Django style
    books = await Book.objects.all(author__name="J.R.R. Tolkien")
    # python style
    books = await Book.objects.all(Book.author.name == "J.R.R. Tolkien")
    assert len(books) == 3

    # filter can accept special methods also separated with double underscore
    # to issue sql query ` where authors.name like "%tolkien%"` that is not
    # case sensitive (hence small t in Tolkien)
    # Django style
    books = await Book.objects.filter(author__name__icontains="tolkien").all()
    # python style
    books = await Book.objects.filter(Book.author.name.icontains("tolkien")).all()
    assert len(books) == 3

    # to sort use order_by() function of queryset
    # to sort decreasing use hyphen before the field name
    # same as with filter you can use double underscores to access related fields
    # Django style
    books = await Book.objects.filter(author__name__icontains="tolkien").order_by(
        "-year").all()
    # python style
    books = await Book.objects.filter(Book.author.name.icontains("tolkien")).order_by(
        Book.year.desc()).all()
    assert len(books) == 3
    assert books[0].title == "The Silmarillion"
    assert books[2].title == "The Hobbit"

    # to read more about filtering and ordering
    # visit: https://collerek.github.io/ormar/queries/filter-and-sort/


async def subset_of_columns():
    # to exclude some columns from loading when querying the database
    # you can use fileds() method
    hobbit = await Book.objects.fields(["title"]).get(title="The Hobbit")
    # note that fields not included in fields are empty (set to None)
    assert hobbit.year is None
    assert hobbit.author is None

    # selected field is there
    assert hobbit.title == "The Hobbit"

    # alternatively you can provide columns you want to exclude
    hobbit = await Book.objects.exclude_fields(["year"]).get(title="The Hobbit")
    # year is still not set
    assert hobbit.year is None
    # but author is back
    assert hobbit.author is not None

    # also you cannot exclude primary key column - it's always there
    # even if you EXPLICITLY exclude it it will be there

    # note that each model have a shortcut for primary_key column which is pk
    # and you can filter/access/set the values by this alias like below
    assert hobbit.pk is not None

    # note that you cannot exclude fields that are not nullable
    # (required) in model definition
    try:
        await Book.objects.exclude_fields(["title"]).get(title="The Hobbit")
    except pydantic.ValidationError:
        print("Cannot exclude non nullable field title")

    # to read more about selecting subset of columns
    # visit: https://collerek.github.io/ormar/queries/select-columns/


async def pagination():
    # to limit number of returned rows use limit()
    books = await Book.objects.limit(1).all()
    assert len(books) == 1
    assert books[0].title == "The Hobbit"

    # to offset number of returned rows use offset()
    books = await Book.objects.limit(1).offset(1).all()
    assert len(books) == 1
    assert books[0].title == "The Lord of the Rings"

    # alternatively use paginate that combines both
    books = await Book.objects.paginate(page=2, page_size=2).all()
    assert len(books) == 2
    # note that we removed one book of Sapkowski in delete()
    # and recreated The Silmarillion - by default when no order_by is set
    # ordering sorts by primary_key column
    assert books[0].title == "The Witcher"
    assert books[1].title == "The Silmarillion"

    # to read more about pagination and number of rows
    # visit: https://collerek.github.io/ormar/queries/pagination-and-rows-number/


async def aggregations():
    # count:
    assert 2 == await Author.objects.count()

    # exists:
    assert await Book.objects.filter(title="The Hobbit").exists()

    # max:
    assert 1990 == await Book.objects.max(columns=["year"])

    # min:
    assert 1937 == await Book.objects.min(columns=["year"])

    # avg:
    assert 1964.75 == await Book.objects.avg(columns=["year"])

    # sum:
    assert 7859 == await Book.objects.sum(columns=["year"])

    # to read more about aggregated functions
    # visit: https://collerek.github.io/ormar/queries/aggregations/

    
async def with_connect(function):
    # note that for any other backend than sqlite you actually need to
    # connect to the database to perform db operations
    async with database:
        await function()

    # note that if you use framework like `fastapi` you shouldn't connect
    # in your endpoints but have a global connection pool
    # check https://collerek.github.io/ormar/fastapi/ and section with db connection

# gather and execute all functions
# note - normally import should be at the beginning of the file
import asyncio

# note that normally you use gather() function to run several functions
# concurrently but we actually modify the data and we rely on the order of functions
for func in [create, read, update, delete, joins,
             filter_and_sort, subset_of_columns,
             pagination, aggregations]:
    print(f"Executing: {func.__name__}")
    asyncio.run(with_connect(func))

# drop the database tables
metadata.drop_all(engine)

Ormar Specification

QuerySet methods

  • create(**kwargs): -> Model
  • get(*args, **kwargs): -> Model
  • get_or_none(*args, **kwargs): -> Optional[Model]
  • get_or_create(*args, **kwargs) -> Model
  • first(*args, **kwargs): -> Model
  • update(each: bool = False, **kwargs) -> int
  • update_or_create(**kwargs) -> Model
  • bulk_create(objects: List[Model]) -> None
  • bulk_update(objects: List[Model], columns: List[str] = None) -> None
  • delete(*args, each: bool = False, **kwargs) -> int
  • all(*args, **kwargs) -> List[Optional[Model]]
  • filter(*args, **kwargs) -> QuerySet
  • exclude(*args, **kwargs) -> QuerySet
  • select_related(related: Union[List, str]) -> QuerySet
  • prefetch_related(related: Union[List, str]) -> QuerySet
  • limit(limit_count: int) -> QuerySet
  • offset(offset: int) -> QuerySet
  • count() -> int
  • exists() -> bool
  • max(columns: List[str]) -> Any
  • min(columns: List[str]) -> Any
  • avg(columns: List[str]) -> Any
  • sum(columns: List[str]) -> Any
  • fields(columns: Union[List, str, set, dict]) -> QuerySet
  • exclude_fields(columns: Union[List, str, set, dict]) -> QuerySet
  • order_by(columns:Union[List, str]) -> QuerySet

Relation types

  • One to many - with ForeignKey(to: Model)
  • Many to many - with ManyToMany(to: Model, Optional[through]: Model)

Model fields types

Available Model Fields (with required args - optional ones in docs):

  • String(max_length)
  • Text()
  • Boolean()
  • Integer()
  • Float()
  • Date()
  • Time()
  • DateTime()
  • JSON()
  • BigInteger()
  • Decimal(scale, precision)
  • UUID()
  • LargeBinary(max_length)
  • EnumField - by passing choices to any other Field type
  • EncryptedString - by passing encrypt_secret and encrypt_backend
  • ForeignKey(to)
  • ManyToMany(to, through)

Available fields options

The following keyword arguments are supported on all field types.

  • primary_key: bool
  • nullable: bool
  • default: Any
  • server_default: Any
  • index: bool
  • unique: bool
  • choices: typing.Sequence
  • name: str
  • pydantic_only: bool

All fields are required unless one of the following is set:

  • nullable - Creates a nullable column. Sets the default to None.
  • default - Set a default value for the field. Not available for relation fields
  • server_default - Set a default value for the field on server side (like sqlalchemy's func.now()). Not available for relation fields
  • primary key with autoincrement - When a column is set to primary key and autoincrement is set on this column. Autoincrement is set by default on int primary keys.
  • pydantic_only - Field is available only as normal pydantic field, not stored in the database.

Available signals

Signals allow to trigger your function for a given event on a given Model.

  • pre_save
  • post_save
  • pre_update
  • post_update
  • pre_delete
  • post_delete
Issues
  • Model.dict() is taking to much time to respond

    Model.dict() is taking to much time to respond

    Describe the bug Im using a model with 1 many to many rel and 1 foreign key, with ormar, but the seralization is taking too much to do it. After disabling dict() the overloaded method, the response model does not have any related model, but it takes almost a 1/100 (I didn't make the exact calculations) of the time to do it. This problem also affects to make the json response, so we should expect a slow response time. Is there any way to disable this override method or is there any plan/idea to optimize it?

    With ormar dict(): Captura de Pantalla 2021-09-06 a la(s) 12 29 23

    With only-pydantic dict(): Captura de Pantalla 2021-09-06 a la(s) 12 28 13

    The list of models that I´m loading doesn't have any related model loaded, neither items on the many to many relation

    bug 
    opened by luisjimenez6245 20
  • Support Base64 Encoding LargeBinary Field

    Support Base64 Encoding LargeBinary Field

    Is your feature request related to a problem? Please describe.

    In a Model such as:

    class BaseMessageModel(Model):
        class Meta:
            metadata = metadata
            database = database
    
        data: bytes = LargeBinary(max_length=1000)
    

    One of the challenges that comes up is actually serializing this binary data effectively. If there are any UTF-8 incompatible bytes:

    Traceback (most recent call last):
      File "venv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi
        result = await app(self.scope, self.receive, self.send)
      File "venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
        return await self.app(scope, receive, send)
      File "venv/lib/python3.9/site-packages/fastapi/applications.py", line 199, in __call__
        await super().__call__(scope, receive, send)
      File "venv/lib/python3.9/site-packages/starlette/applications.py", line 111, in __call__
        await self.middleware_stack(scope, receive, send)
      File "venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
        raise exc from None
      File "venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
        await self.app(scope, receive, _send)
      File "venv/lib/python3.9/site-packages/starlette/middleware/cors.py", line 78, in __call__
        await self.app(scope, receive, send)
      File "venv/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
        raise exc from None
      File "venv/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
        await self.app(scope, receive, sender)
      File "venv/lib/python3.9/site-packages/starlette/routing.py", line 566, in __call__
        await route.handle(scope, receive, send)
      File "venv/lib/python3.9/site-packages/starlette/routing.py", line 376, in handle
        await self.app(scope, receive, send)
      File "venv/lib/python3.9/site-packages/fastapi/applications.py", line 199, in __call__
        await super().__call__(scope, receive, send)
      File "venv/lib/python3.9/site-packages/starlette/applications.py", line 111, in __call__
        await self.middleware_stack(scope, receive, send)
      File "venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
        raise exc from None
      File "venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
        await self.app(scope, receive, _send)
      File "venv/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
        raise exc from None
      File "venv/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
        await self.app(scope, receive, sender)
      File "venv/lib/python3.9/site-packages/starlette/routing.py", line 566, in __call__
        await route.handle(scope, receive, send)
      File "venv/lib/python3.9/site-packages/starlette/routing.py", line 227, in handle
        await self.app(scope, receive, send)
      File "venv/lib/python3.9/site-packages/starlette/routing.py", line 41, in app
        response = await func(request)
      File "venv/lib/python3.9/site-packages/fastapi/routing.py", line 209, in app
        response_data = await serialize_response(
      File "venv/lib/python3.9/site-packages/fastapi/routing.py", line 127, in serialize_response
        return jsonable_encoder(
      File "venv/lib/python3.9/site-packages/fastapi/encoders.py", line 104, in jsonable_encoder
        jsonable_encoder(
      File "venv/lib/python3.9/site-packages/fastapi/encoders.py", line 57, in jsonable_encoder
        return jsonable_encoder(
      File "venv/lib/python3.9/site-packages/fastapi/encoders.py", line 90, in jsonable_encoder
        encoded_value = jsonable_encoder(
      File "venv/lib/python3.9/site-packages/fastapi/encoders.py", line 127, in jsonable_encoder
        return ENCODERS_BY_TYPE[type(obj)](obj)
      File "pydantic/json.py", line 43, in pydantic.json.lambda
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 1: invalid start byte
    

    This makes sense since you can't send arbitrary bytes through a fundamentally text-based (UTF-8) JSON representation.

    Describe the solution you'd like

    class BaseMessageModel(Model):
        class Meta:
            metadata = metadata
            database = database
    
        data: str = LargeBinary(max_length=1000, represent_as_base64=True)
    

    And internally this would call something like:

        @validator("data")
        def convert_to_base64_str(cls, v: bytes) -> str:
            return base64.b64encode(v).decode()
    

    Edit: It looks like FastAPI doesn't use OpenAPI 3.1 yet (it is newly released), so probably the older 3.0 format should be used:

    https://github.com/OAI/OpenAPI-Specification/blob/fbe62006211838a8bb7bf2433a1d15f1a5838a03/versions/3.0.1.md#considerations-for-file-uploads

    # content transferred with base64 encoding
    schema:
      type: string
      format: base64
    

    Future Considerations

    In OpenAPI 3.1 (not yet supported by FastAPI): https://github.com/OAI/OpenAPI-Specification/issues/1547#issuecomment-595497718

    type: string
    contentEncoding: base64
    

    Describe alternatives you've considered

    Ideally Pydantic would support this serialization upstream: https://github.com/samuelcolvin/pydantic/issues/692 but it's been that issue has been open for 2 years.

    enhancement 
    opened by johnthagen 20
  • Add python_version marker constraint in setup.py

    Add python_version marker constraint in setup.py

    Describe the bug I've found a dependency resolution problem when installing ormar with black using poetry.

      Because no versions of black match >21.9b0,<22.0
       and black (21.9b0) depends on typing-extensions (>=3.10.0.0), black (>=21.9b0,<22.0) requires typing-extensions (>=3.10.0.0).
      And because ormar (0.10.20) depends on typing-extensions (>=3.7,<3.10.0.3)
       and no versions of ormar match >0.10.20,<0.11.0, black (>=21.9b0,<22.0) is incompatible with ormar (>=0.10.20,<0.11.0).
      So, because ormartest depends on both ormar (^0.10.20) and black (^21.9b0), version solving failed.
    

    To Reproduce Steps to reproduce the behavior:

    1. poetry new test
    2. cd test
    3. poetry add black --dev
    4. poetry add ormar
    5. You found the error.

    Expected behavior A clear and concise description of what you expected to happen.

    Versions (please complete the following information):

    • Database backend used (mysql/sqlite/postgress)
    • Python version
    • poetry ^1.1.11
    • ormar ^0.10.20
    • black ^21.9b0

    Possible solution

    Since you use typing_extensions only for providing Protocol and Literal, you may add constraint to avoid this issue. The solution is to specify the typing_extension version in your setup.py as following:

    install_requires=[
            ...
            "typing_extensions>=3.7,<3.10.0.3; python_version < '3.8'",
        ],
    

    Additional context

    This issue is crucial for me, since I want to integrate your super cool library in my project generator. It uses black by default, but I can't get it work with your library because of this issue.

    Thanks in advance.

    bug 
    opened by s3rius 14
  • Bug in request generated

    Bug in request generated

    Hi, I have a bug with a request generated when trying to fetch data from a many 2 many relation:

    • the request generated reference a user.id field as a primary key, but the primary key in the model is user.registrationnumber
    • I also had some errors when using field name containing capitalized letter ou containing '_' You will find all the details in the sample below.

    OS: Centos 7.9 (docker) python version: 3.8.3 ormar version: 0.7.3 database backend: postgresql

    To reproduce the error i made the script below:

    import asyncio
    import uuid
    from datetime import date, datetime
    from os import major
    from typing import List, Optional, Union
    
    import databases
    import ormar
    import sqlalchemy
    from fastapi.encoders import jsonable_encoder
    from sqlalchemy import func, text
    import jsonpickle
    
    
    DATABASE_URL="postgresql://postgres:[email protected]:5432/test"
    database = databases.Database(DATABASE_URL)
    metadata = sqlalchemy.MetaData()
    
    
    class MainMeta(ormar.ModelMeta):
        metadata = metadata
        database = database
    
    
    class Role(ormar.Model):
        class Meta(MainMeta):
            pass
        name                : str = ormar.Text(primary_key=True)
        order               : int = ormar.Integer(default=0)
        description         : str = ormar.Text()
    
    
    class Company(ormar.Model):
        class Meta(MainMeta):
            pass
        name                : str = ormar.Text(primary_key=True)
    
    
    class UserRoleCompany(ormar.Model):
        class Meta(MainMeta):
            pass
    
    
    class User(ormar.Model):
        class Meta(MainMeta):
            pass
        registrationnumber  : str = ormar.Text(primary_key=True)
        company             : Company = ormar.ForeignKey(Company)
        name                : str = ormar.Text()
        role                : Optional[Role] = ormar.ForeignKey(Role)
        roleforcompanies    : Optional[Union[Company, List[Company]]] = ormar.ManyToMany(Company, through=UserRoleCompany)
        lastupdate          : date = ormar.DateTime(server_default=sqlalchemy.func.now())
    
    
    async def main():
        if not database.is_connected:
            print("connection to db {}.".format(DATABASE_URL))
            await database.connect()
        ##########################################################################################
        try:
            print("adding role")
            role_0 = await Role.objects.create(name="user", order=0, description = "no administration right")
            role_1 = await Role.objects.create(name="admin", order=1, description = "standard administration right")
            role_2 = await Role.objects.create(name="super_admin", order=2, description = "super administration right")
            assert await Role.objects.count() == 3
    
            print("adding company")
            company_0 = await Company.objects.create(name="Company")
            company_1 = await Company.objects.create(name="Subsidiary Company 1")
            company_2 = await Company.objects.create(name="Subsidiary Company 2")
            company_3 = await Company.objects.create(name="Subsidiary Company 3")
            assert await Company.objects.count() == 4
    
            print("adding user")
            user = await User.objects.create(registrationnumber="00-00000", company=company_0, name="admin", role=role_1)
            assert await User.objects.count() == 1
    
            print("removing user")
            await user.delete()
            assert await User.objects.count() == 0
    
            print("adding user with company-role")
            companies: List[Company] = [company_1, company_2]
            # user = await User.objects.create(registrationnumber="00-00000", company=company_0, name="admin", role=role_1, roleforcompanies=companies)
            user = await User.objects.create(registrationnumber="00-00000", company=company_0, name="admin", role=role_1)
            # print(User.__fields__)
            await user.roleforcompanies.add(company_1)
            await user.roleforcompanies.add(company_2)
    
            users = await User.objects.select_related("roleforcompanies").all()
            print(jsonpickle.encode(jsonable_encoder(users), unpicklable=False, keys=True ))
    
        except Exception as error:
            print(error)
    
    
        """
    
        This is the request generated:
        'SELECT
        users.registrationnumber as registrationnumber,
        users.company as company,
        users.name as name, users.role as role,
        users.lastupdate as lastupdate,
        cy24b4_userrolecompanys.id as cy24b4_id,
        cy24b4_userrolecompanys.company as cy24b4_company,
        cy24b4_userrolecompanys.user as cy24b4_user,
        jn50a4_companys.name as jn50a4_name \n
        FROM users
        LEFT OUTER JOIN userrolecompanys cy24b4_userrolecompanys ON cy24b4_userrolecompanys.user=users.id
        LEFT OUTER JOIN companys jn50a4_companys ON jn50a4_companys.name=cy24b4_userrolecompanys.company
        ORDER BY users.registrationnumber, jn50a4_companys.name'
    
        There is an error in the First LEFT OUTER JOIN generated:
        ... companys.user=users.id
        should be:
       ... companys.user=users.registrationnumber
    
        There is also a \n in the midle of the string...
    
        The execution produce the error: column users.id does not exist
        """
    
        ##########################################################################################
        if database.is_connected:
            await database.disconnect()
            print("db closed.")
    
    
    
    
    if __name__ == '__main__':
        asyncio.run(main())
    
    

    I'm new to python, sqlalchemy, fastapi anr ormar.... maybe i made some mistakes... Thanks for this great project.

    bug 
    opened by schaulet 14
  • FastAPI JSON weird

    FastAPI JSON weird

    So, I'm not sure which library bears responsibility here, but I'll start with ormar and you can forward me somewhere else if the problem lies elsewhere. Consider the following code:

    ...
    class Thing(ormar.Model):
        class Meta(BaseMeta):
            tablename = "things"
        id: UUID = ormar.UUID(primary_key=True, default=uuid4)
        name: str = ormar.Text(default="")
        js: pydantic.Json = ormar.JSON()
    ...
    @app.get("/things")
    async def read_things():
        return await Thing.objects.all()
    ...
    

    What I get when I call this endpoint is e.g.

    [
      {
        "id": "1932caad-1157-4224-9688-e280f9623e67",
        "name": "",
        "js": "[\"asdf\", \"asdf\", \"bobby\", \"nigel\"]"
      },
      {
        "id": "3e6a15b2-2cd5-456b-a4dc-24e3cd76d96e",
        "name": "test",
        "js": "[\"lemon\", \"raspberry\", \"lime\", \"pumice\"]"
      }
    ]
    

    Note how rather than being JSON, the js field is a plain string containing JSON. Is this on purpose? Does it HAVE to be that way? It seems to me like it would make more sense for a JSON field, when its container is serialized to JSON, to just...be JSON. (I note that thing.json() preserves the "convert json to string" behavior.) Is there an easy way around this behavior, perhaps a flag or setting? Is this actually a result of a different library?

    bug 
    opened by Erhannis 13
  • QuerySet operations returns Sequence instead of List

    QuerySet operations returns Sequence instead of List

    Describe the bug The QuerySet operations return value is Sequence["Model"] even though on the DocString it is stated as List["Model"] It is causing for lint errors as mismatch types when trying to retrieve the returned object as List.

    for example:

    rows: List[User] = await User.objects.all()
    

    Will cause a lint error on mismatch type.

    Looking into Ormar source code it seems the first time where this is starting to get confused is at: ormar.models.mixins.merge_mixin.py ==> merge_instances_list class method

    from source-code:

    @classmethod
        def merge_instances_list(cls, result_rows: Sequence["Model"]) -> Sequence["Model"]:
            """
            Merges a list of models into list of unique models.
    
            Models can duplicate during joins when parent model has multiple child rows,
            in the end all parent (main) models should be unique.
    
            :param result_rows: list of already initialized Models with child models
            populated, each instance is one row in db and some models can duplicate
            :type result_rows: List["Model"]
            :return: list of merged models where each main model is unique
            :rtype: List["Model"]
            """
    

    as you can see rtype on DocString is List["Model"] while the annotated return type is Sequence["Model"].

    If there is any specific reason and I can assist with fixing - I will be very happy to do so! The package is very helpful and it is much appreciated!

    bug 
    opened by dudil 12
  • KeyError when building openapi file

    KeyError when building openapi file

    Describe the bug I deployed a service using FastAPI and Ormar on two different servers. On the first one it works, and on the second one I get this error when accessing /docs or /openapi.json (I believe most of the traceback is not useful):

    Summary: (click to unfold the whole traceback)
      File "/api/pkgs/fastapi/utils.py", line 28, in get_model_definitions
        model_name = model_name_map[model]
                     │              └ <class 'abc.Team_PJX'>
                     └ {<class 'abc.Team_TLK'>: 'Team_TLK', <enum 'TicketStatus'>: 'TicketStatus', <class 'abc.Project_RLS'>: 'Project_RLS', <class ...
    
    KeyError: <class 'abc.Team_PJX'>
    
    Exception in ASGI application
    
    Traceback (most recent call last):
    
      File "/api/run.py", line 70, in <module>
        server.run()
        │      └ <function Server.run at 0x7fbf689e23a0>
        └ <uvicorn.main.Server object at 0x7fbf6ea4bd00>
    
      File "/api/pkgs/uvicorn/main.py", line 419, in run
        loop.run_until_complete(self.serve(sockets=sockets))
        │    │                  │    │             └ None
        │    │                  │    └ <function Server.serve at 0x7fbf689e2430>
        │    │                  └ <uvicorn.main.Server object at 0x7fbf6ea4bd00>
        │    └ <function BaseEventLoop.run_until_complete at 0x7fbf6b0f3700>
        └ <_UnixSelectorEventLoop running=True closed=False debug=False>
    
      File "/usr/lib64/python3.8/asyncio/base_events.py", line 603, in run_until_complete
        self.run_forever()
        │    └ <function BaseEventLoop.run_forever at 0x7fbf6b0f3670>
        └ <_UnixSelectorEventLoop running=True closed=False debug=False>
      File "/usr/lib64/python3.8/asyncio/base_events.py", line 570, in run_forever
        self._run_once()
        │    └ <function BaseEventLoop._run_once at 0x7fbf6b0f61f0>
        └ <_UnixSelectorEventLoop running=True closed=False debug=False>
      File "/usr/lib64/python3.8/asyncio/base_events.py", line 1859, in _run_once
        handle._run()
        │      └ <function Handle._run at 0x7fbf6b574f70>
        └ <Handle <TaskWakeupMethWrapper object at 0x7fbf5c5aaee0>(<Future finished result=None>)>
      File "/usr/lib64/python3.8/asyncio/events.py", line 81, in _run
        self._context.run(self._callback, *self._args)
        │    │            │    │           │    └ <member '_args' of 'Handle' objects>
        │    │            │    │           └ <Handle <TaskWakeupMethWrapper object at 0x7fbf5c5aaee0>(<Future finished result=None>)>
        │    │            │    └ <member '_callback' of 'Handle' objects>
        │    │            └ <Handle <TaskWakeupMethWrapper object at 0x7fbf5c5aaee0>(<Future finished result=None>)>
        │    └ <member '_context' of 'Handle' objects>
        └ <Handle <TaskWakeupMethWrapper object at 0x7fbf5c5aaee0>(<Future finished result=None>)>
    
      File "/api/pkgs/uvicorn/protocols/http/h11_impl.py", line 389, in run_asgi
        result = await app(self.scope, self.receive, self.send)
                       │   │    │      │    │        │    └ <function RequestResponseCycle.send at 0x7fbf68768790>
                       │   │    │      │    │        └ <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>
                       │   │    │      │    └ <function RequestResponseCycle.receive at 0x7fbf68768820>
                       │   │    │      └ <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>
                       │   │    └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.42.56.164', 8080),...
                       │   └ <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>
                       └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x7fbf687941f0>
    
      File "/api/pkgs/uvicorn/middleware/proxy_headers.py", line 45, in __call__
        return await self.app(scope, receive, send)
                     │    │   │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>>
                     │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>>
                     │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.42.56.164', 8080),...
                     │    └ <fastapi.applications.FastAPI object at 0x7fbf66e39850>
                     └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x7fbf687941f0>
    
      File "/api/pkgs/fastapi/applications.py", line 199, in __call__
        await super().__call__(scope, receive, send)
                               │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>>
                               │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>>
                               └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.42.56.164', 8080),...
    
      File "/api/pkgs/starlette/applications.py", line 112, in __call__
        await self.middleware_stack(scope, receive, send)
              │    │                │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>>
              │    │                │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>>
              │    │                └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.42.56.164', 8080),...
              │    └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x7fbf64fdd9d0>
              └ <fastapi.applications.FastAPI object at 0x7fbf66e39850>
    
      File "/api/pkgs/starlette/middleware/errors.py", line 181, in __call__
        raise exc from None
    
      File "/api/pkgs/starlette/middleware/errors.py", line 159, in __call__
        await self.app(scope, receive, _send)
              │    │   │      │        └ <function ServerErrorMiddleware.__call__.<locals>._send at 0x7fbf64fdeee0>
              │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>>
              │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.42.56.164', 8080),...
              │    └ <starlette.middleware.base.BaseHTTPMiddleware object at 0x7fbf64fdd9a0>
              └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x7fbf64fdd9d0>
    
      File "/api/pkgs/starlette/middleware/base.py", line 25, in __call__
        response = await self.dispatch_func(request, self.call_next)
                         │    │             │        │    └ <function BaseHTTPMiddleware.call_next at 0x7fbf66a819d0>
                         │    │             │        └ <starlette.middleware.base.BaseHTTPMiddleware object at 0x7fbf64fdd9a0>
                         │    │             └ <starlette.requests.Request object at 0x7fbf640dd340>
                         │    └ <function PrometheusFastApiInstrumentator.instrument.<locals>.dispatch_middleware at 0x7fbf65196ee0>
                         └ <starlette.middleware.base.BaseHTTPMiddleware object at 0x7fbf64fdd9a0>
    
      File "/api/pkgs/prometheus_fastapi_instrumentator/instrumentation.py", line 150, in dispatch_middleware
        raise e from None
    
      File "/api/pkgs/prometheus_fastapi_instrumentator/instrumentation.py", line 145, in dispatch_middleware
        response = await call_next(request)
                         │         └ <starlette.requests.Request object at 0x7fbf640dd340>
                         └ <bound method BaseHTTPMiddleware.call_next of <starlette.middleware.base.BaseHTTPMiddleware object at 0x7fbf64fdd9a0>>
    
      File "/api/pkgs/starlette/middleware/base.py", line 45, in call_next
        task.result()
        │    └ <method 'result' of '_asyncio.Task' objects>
        └ <Task finished name='Task-5469' coro=<BaseHTTPMiddleware.call_next.<locals>.coro() done, defined at /api/pkgs/starlette/middl...
    
      File "/api/pkgs/starlette/middleware/base.py", line 38, in coro
        await self.app(scope, receive, send)
              │    │   │      │        └ <bound method Queue.put of <Queue at 0x7fbf64e0ea30 maxsize=0 tasks=1>>
              │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>>
              │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.42.56.164', 8080),...
              │    └ <starlette.middleware.cors.CORSMiddleware object at 0x7fbf64fdd970>
              └ <starlette.middleware.base.BaseHTTPMiddleware object at 0x7fbf64fdd9a0>
    
      File "/api/pkgs/starlette/middleware/cors.py", line 78, in __call__
        await self.app(scope, receive, send)
              │    │   │      │        └ <bound method Queue.put of <Queue at 0x7fbf64e0ea30 maxsize=0 tasks=1>>
              │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>>
              │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.42.56.164', 8080),...
              │    └ <starlette.exceptions.ExceptionMiddleware object at 0x7fbf64fdda60>
              └ <starlette.middleware.cors.CORSMiddleware object at 0x7fbf64fdd970>
    
      File "/api/pkgs/starlette/exceptions.py", line 82, in __call__
        raise exc from None
    
      File "/api/pkgs/starlette/exceptions.py", line 71, in __call__
        await self.app(scope, receive, sender)
              │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x7fbf64a939d0>
              │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>>
              │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.42.56.164', 8080),...
              │    └ <fastapi.routing.APIRouter object at 0x7fbf657a9250>
              └ <starlette.exceptions.ExceptionMiddleware object at 0x7fbf64fdda60>
    
      File "/api/pkgs/starlette/routing.py", line 580, in __call__
        await route.handle(scope, receive, send)
              │     │      │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x7fbf64a939d0>
              │     │      │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>>
              │     │      └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.42.56.164', 8080),...
              │     └ <function Route.handle at 0x7fbf66ad3b80>
              └ <starlette.routing.Route object at 0x7fbf653fb5b0>
    
      File "/api/pkgs/starlette/routing.py", line 241, in handle
        await self.app(scope, receive, send)
              │    │   │      │        └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x7fbf64a939d0>
              │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x7fbf5c5aad90>>
              │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.42.56.164', 8080),...
              │    └ <function request_response.<locals>.app at 0x7fbf653fdee0>
              └ <starlette.routing.Route object at 0x7fbf653fb5b0>
    
      File "/api/pkgs/starlette/routing.py", line 52, in app
        response = await func(request)
                         │    └ <starlette.requests.Request object at 0x7fbf5c5aa130>
                         └ <function FastAPI.setup.<locals>.openapi at 0x7fbf653fdd30>
    
      File "/api/pkgs/fastapi/applications.py", line 152, in openapi
        return JSONResponse(self.openapi())
               │            │    └ <function FastAPI.openapi at 0x7fbf66a91af0>
               │            └ <fastapi.applications.FastAPI object at 0x7fbf66e39850>
               └ <class 'starlette.responses.JSONResponse'>
    
      File "/api/pkgs/fastapi/applications.py", line 130, in openapi
        self.openapi_schema = get_openapi(
        │    │                └ <function get_openapi at 0x7fbf66a81430>
        │    └ None
        └ <fastapi.applications.FastAPI object at 0x7fbf66e39850>
    
      File "/api/pkgs/fastapi/openapi/utils.py", line 363, in get_openapi
        definitions = get_model_definitions(
                      └ <function get_model_definitions at 0x7fbf66b27af0>
    
      File "/api/pkgs/fastapi/utils.py", line 28, in get_model_definitions
        model_name = model_name_map[model]
                     │              └ <class 'abc.Team_PJX'>
                     └ {<class 'abc.Team_TLK'>: 'Team_TLK', <enum 'TicketStatus'>: 'TicketStatus', <class 'abc.Project_RLS'>: 'Project_RLS', <class ...
    
    KeyError: <class 'abc.Team_PJX'>
    

    To Reproduce

    I'm not sure how to reproduce this. As stated above, only one of two identical deployments fails with this error. I'll see what happens when I redeploy the two services (if the failing one is fixed by the update) and report back. In the meanwhile maybe you'll have a hunch 🙂

    Expected behavior No errors, openapi.json can be generated.

    Versions (please complete the following information):

    • Database backend used: sqlite
    • Python version: 3.8
    • ormar version: 0.10.15
    • pydantic version: 1.8.2
    • if applicable fastapi version: 0.65.2
    bug 
    opened by pawamoy 11
  • UUIDs cannot be queried using get as primary key

    UUIDs cannot be queried using get as primary key

    Recently, I started using ormar instead of tortoise-orm in my projects, but have encountered a strange problem, if I use uuid as the primary key of a model, I can't match the data when using get query.

    example:

    class MainMeta(ormar.ModelMeta):
        metadata = metadata
        database = database
    
    
    class UserModel(ormar.Model):
    
        class Meta(MainMeta):
            tablename = "usermodel"
    
        id: ormar.UUID(primary_key=True)
        username = ormar.String(index=True, unique=True, null=False, max_length=255)
        email = ormar.String(index=True, unique=True, nullable=False, max_length=255)
        hashed_password = ormar.String(null=False, max_length=255)
        is_active = ormar.Boolean(default=True, nullable=False)
        is_superuser = ormar.Boolean(default=False, nullable=False)
    
    user = await UserModel.objects.first()
    await UserModel.objects.get(id=user.id) # raise NoMatch
    await UserModel.objects.get(username=user.username) # Match user
    
    enhancement 
    opened by hidaris 10
  • `ValidationError` is not thrown out correctly with `get_pydantic` method

    `ValidationError` is not thrown out correctly with `get_pydantic` method

    Description

    According to the documentation I noticed that the ValidationError is not thrown out correctly if models generated with get_pydantic are used in FastAPI requests.

    Example:

    class EnumExample(str, enum.Enum):
        A = 'A'
        B = 'B'
        C = 'C'
    
    
    class ModelExample(ormar.Model):
        class Meta(ormar.ModelMeta):
            database = database
            metadata = metadata
            tablename = "examples"
        
        id: int = ormar.Integer(primary_key=True)
        str_field: str = ormar.String(min_length=5, max_length=10, nullable=False)
        enum_field: str = ormar.String(max_length=1, nullable=False, choices=list(EnumExample))
    
        @pydantic.validator('str_field')
        def validate_str_field(cls, v):
            if ' ' not in v:
                raise ValueError('must contain a space')
            return v
    
    
    ModelExampleCreate = ModelExample.get_pydantic(exclude={'id'})
    
    @app.post("/examples/", response_model=ModelExample)
    async def create_example(example: ModelExampleCreate):
        return await ModelExample(**example.dict()).save()
    

    Result:

    Client receives an Internal Server Error, the ValidationError is only output in the error log.

      File "/home/vscode/.local/lib/python3.9/site-packages/ormar/models/newbasemodel.py", line 143, in __init__
        raise validation_error
    pydantic.error_wrappers.ValidationError: 1 validation error for ModelExample
    __root__
      enum_field: 'D' not in allowed choices set: ['A', 'B', 'C'] (type=value_error)
    
      File "/home/vscode/.local/lib/python3.9/site-packages/ormar/models/newbasemodel.py", line 143, in __init__
        raise validation_error
    pydantic.error_wrappers.ValidationError: 1 validation error for ModelExample
    str_field
      must contain a space (type=value_error)
    

    Expected result:

    Client receives the ValidationError.

    Note:

    Everything goes as expected with the original model:

    @app.post("/examples/", response_model=ModelExample)
    async def create_example(example: ModelExample):
        return await example.save()
    

    Versions:

    • ormar 0.10.20
    • pydantic 1.8.2
    • fastapi 0.68.2
    bug 
    opened by derzinn 10
  • Proper annotations

    Proper annotations

    Right now mypy, pyright and other static type analyzers are screaming from annotations like this: id: Integer(primary_key=True)

    mypy error:

    lmat/db/models.py:33: error: Invalid type comment or annotation  [valid-type]
    lmat/db/models.py:33: note: Suggestion: use Integer[...] instead of Integer(...)
    

    pyright error:

    Expected class type but received "Integer" Pyright (reportGeneralTypeIssues)
    

    Why id = Integer(primary_key=True) syntax is not used (like in orm)? I guess it is because of the way you extract annotations, do you consider changing the approach? I would like to use this library in one project, but it is unusable in this state (I cannot just put # type: ignore[valid-type] on every line in models)

    Can help with this when I have some spare time, unfortunately I am forced to go back to orm.

    opened by HarrySky 9
  • `get_or_create` to return an additional Boolean field

    `get_or_create` to return an additional Boolean field

    Is your feature request related to a problem? Please describe.

    The get_or_create currently returns a Model. Add a boolean([Model, bool]) to specify whether the item was created or not. Similar to django get_or_create

    Describe the solution you'd like

    By going through the ormar get_or_create code, the function can implement it by returning a bool. Modified body:

        ...
        try:
            return await self.get(*args, **kwargs), False
        except NoMatch:
            return await self.create(**kwargs), True
    

    Any plans to add this feature? Would you accept a PR?

    enhancement 
    opened by amalshaji 0
  • Allow nesting ormar models in pydantic models

    Allow nesting ormar models in pydantic models

    Describe the bug Getting AttributeError: _orm while building pydantic response model

    Traceback
    Traceback (most recent call last):
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 394, in run_asgi
        result = await app(self.scope, self.receive, self.send)
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 45, in __call__
        return await self.app(scope, receive, send)
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\fastapi\applications.py", line 199, in __call__
        await super().__call__(scope, receive, send)
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\starlette\applications.py", line 111, in __call__
        await self.middleware_stack(scope, receive, send)
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\starlette\middleware\errors.py", line 181, in __call__
        raise exc from None
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\starlette\middleware\errors.py", line 159, in __call__
        await self.app(scope, receive, _send)
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\starlette\exceptions.py", line 82, in __call__
        raise exc from None
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\starlette\exceptions.py", line 71, in __call__
        await self.app(scope, receive, sender)
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\starlette\routing.py", line 566, in __call__
        await route.handle(scope, receive, send)
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\starlette\routing.py", line 227, in handle
        await self.app(scope, receive, send)
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\starlette\routing.py", line 41, in app
        response = await func(request)
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\fastapi\routing.py", line 218, in app
        is_coroutine=is_coroutine,
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\fastapi\routing.py", line 113, in serialize_response
        exclude_none=exclude_none,
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\fastapi\routing.py", line 70, in _prepare_response_content
        exclude_none=exclude_none,
      File "pydantic\main.py", line 459, in pydantic.main.BaseModel.dict
      File "pydantic\main.py", line 802, in _iter
      File "pydantic\main.py", line 697, in pydantic.main.BaseModel._get_value
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\ormar\models\newbasemodel.py", line 637, in dict
        exclude=exclude,  # type: ignore
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\ormar\models\newbasemodel.py", line 561, in _extract_nested_models
        nested_model = getattr(self, field)
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\ormar\models\newbasemodel.py", line 269, in __getattribute__
        )(item)
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\ormar\models\newbasemodel.py", line 306, in _extract_related_model_instead_of_field
        if item in self._orm:
      File "C:\Users\usr\AppData\Local\Programs\Python\Python37\lib\site-packages\ormar\models\newbasemodel.py", line 263, in __getattribute__
        return object.__getattribute__(self, item)
    AttributeError: _orm
    

    To Reproduce

    Application code
    import databases
    import fastapi
    import ormar
    import pydantic
    import sqlalchemy
    
    from uvicorn import run
    from random import randint
    
    
    uri: str = 'postgresql://xxx:[email protected]/xxx'
    metadata = sqlalchemy.MetaData()
    engine = sqlalchemy.create_engine(uri)
    
    database = databases.Database(uri)
    
    app = fastapi.FastAPI()
    app.state.database = database
    
    
    @app.on_event('startup')
    async def startup() -> None:
      database_ = app.state.database
      if not database_.is_connected:
          await database_.connect()
    
    
    @app.on_event('shutdown')
    async def shutdown() -> None:
      database_ = app.state.database
      if database_.is_connected:
          await database_.disconnect()
    
    
    class UserMeta(ormar.Model):
      class Meta:
          metadata = metadata
          database = database
          tablename = 'user_metas'
    
      id: int = ormar.Integer(primary_key=True, autoincrement=True)
    
      username: str = ormar.String(max_length=64, nullable=True, default=None)
      title: str = ormar.String(max_length=64, nullable=True, default=None)
    
    
    class User(ormar.Model):
      class Meta:
          metadata = metadata
          database = database
          tablename = 'users'  # noqa
    
      id: int = ormar.Integer(primary_key=True, autoincrement=True)
    
      email: str = ormar.String(max_length=120, unique=True)
    
      meta: UserMeta = ormar.ForeignKey(UserMeta)
    
    
    class SomeResponse(pydantic.BaseModel):
      authorized: bool = True
      me: User
    
    
    @app.get('/test', response_model=SomeResponse)
    async def get():
      meta: UserMeta = UserMeta(username='es3n1n')
      await meta.save()
      u: User = User(
          email=f'me+{randint(555, 999)}@es3n.in',
          meta=meta
      )
      await u.save()
      return SomeResponse(me=u)
    
    
    if __name__ == '__main__':
      run(app, host='0.0.0.0', port=2525)
    

    Expected behavior Response should be {"authorized": true, "me": {...}}

    Versions (please complete the following information):

    • Database backend used (mysql/sqlite/postgress): psql (PostgreSQL) 10.16 (Ubuntu 10.16-0ubuntu0.18.04.1)
    • Python version: Python 3.7.0
    • ormar version: 0.10.18
    • pydantic version: 1.8.2
    • if applicable fastapi version: 0.68.1

    Additional context Works fine only if i'm returning just u, but if i'm wrapping it in another pydantic model, then getting this error.

    enhancement 
    opened by es3n1n 3
  • Add 'force_insert' flag to .save_related() method

    Add 'force_insert' flag to .save_related() method

    Is your feature request related to a problem? Please describe. When using the .save_related() method with kwargs 'follow=true' and 'save_all=true' to insert nested objects with parts split across multiple tables, Ormar will either insert the object with a new primary key if none was provided (by either autoincrementing or using the 'server_default') or try to update the relevant table and row for the object using the existing primary key that is provided.

    Describe the solution you'd like Please add a 'force_insert' flag that allows the user to force Ormar to try to perform inserts for provided keys on the nested objects (rather than falling back to updates), under the assumption that the user is willing to deal with any potentially resulting integrity errors.

    enhancement 
    opened by franzwollang 0
  • ForeignKey reverse relations do not take into account configured schema (postgres).

    ForeignKey reverse relations do not take into account configured schema (postgres).

    Describe the bug When using postgres with multiple schemas, while the direct relation with ForeignKey works, the reverse relation fails with an error:

    asyncpg.exceptions.UndefinedTableError: invalid reference to FROM-clause entry for table "al0994_tablename"
    

    To Reproduce Define a relation between schemas. That is, configure in the MetaData class for one entity to be of one schema, and the related entity of antoher.

    class AlphaMeta(ModelMeta):
        metadata = MetaData(schema="alpha")
        database = db
    
    class BetaMeta(ModelMeta):
        metadata = MetaData(schema="beta")
        database = db
    
    class Alpaca(Model):
        class Meta(AlphaMeta):
            tablename = "alpaca"
    
        id: str = UUID(primary_key=True)
    
    class Bear(Model):
        class Meta(BetaMeta):
            tablename = "bear"
    
        id: str = UUID(primary_key=True)
        alpaca: Alpaca = ForeignKey(Alpaca)
    

    Under this scenario, bear.alpaca succeeds, alpaca.bears.all() will fail.

    Expected behavior That the reverse relation executes a query taking the related tables schema into account.

    Screenshots N/A

    Versions (please complete the following information):

    • Database backend used (mysql/sqlite/postgress) postgres
    • Python version 3.9.6
    • ormar version 0.10.5
    • pydantic version 1.8.2
    • if applicable fastapi version N/A

    Additional context N/A

    bug 
    opened by naturalethic 3
  • PostgreSQL Array Columns

    PostgreSQL Array Columns

    I really like the Postgres feature of having array type columns. sqlAlchemy supports the PostgreSQL dialect refer: https://docs.sqlalchemy.org/en/14/dialects/postgresql.html?highlight=array#sqlalchemy.dialects.postgresql.ARRAY

    It would be nice to use them with ormar like: characteristics:list[str] = ormar.Array(element_type='String') or something like that.

    Until that is possible I will use JSON() but it seems to be an overkill

    Considering the DRY Principle, this package is the way to go when using a Database for an API. Thank you for that

    enhancement 
    opened by Lester1989 0
  • GROUP BY, HAVING and aggregation functions support

    GROUP BY, HAVING and aggregation functions support

    Describe the solution you'd like GROUP BY, HAVING and aggregation functions (such as count, avg, max and so on) support in queries. It could be very useful for making queries like this:

    # Select all users with votes and count votes for each user
    select user.id, count(vote.id) as votes_count
    from user
             left join vote on user.id = vote.user
    group by user.id
    having votes_count > 0;
    

    Before ormar I used to use peewee and I really like a way how it implemented there.

    I think it could be easily implemented for queries returning raw values. But it could be a problem with dynamic assigning of Pydantic model fields.

    enhancement 
    opened by MIREX 1
  • There is a problem with Pycharm autocompletion (Pycharm bug - reported)

    There is a problem with Pycharm autocompletion (Pycharm bug - reported)

    I'm aware of the fact that there are some problems with pycharm autocompletion in the latest versions. It's not a bug in ormar, but a bug in pycharm, which causes no autocompletion for Model.objects.

    We will see how quickly they gonna act on this. If it's gonna take them long I will try to change the internals to workaround this before they fix that.

    Works as expected in vscode: image

    I created a bug report for JetBrains team, it's reproduced and should be fixed soon: https://youtrack.jetbrains.com/issue/PY-49306

    Originally posted by @collerek in https://github.com/collerek/ormar/discussions/244#discussioncomment-878551

    bug 
    opened by collerek 0
  • Add OneToOne relation type

    Add OneToOne relation type

    apart from OneToMany (by ForeignKey) and ManyToMany add also support for OneToOne relations.

    enhancement 
    opened by collerek 6
  • Create mypy plugin to handle common pitfalls

    Create mypy plugin to handle common pitfalls

    Since a lot of things in ormar are generated dynamically mypy does not play well with some of them.

    That's the idea behind custom plugin enhancing how mypy handles ormar, similar to the one used by pydantic.

    enhancement help wanted 
    opened by collerek 0
  • Add support for generated columns

    Add support for generated columns

    Is your feature request related to a problem? Please describe.

    I would like to efficiently query my model on a field which is calculated from another field.

    I expect the following behaviour of this field:

    • indices: i should be able to create an index on the calculated field
    • db side filtering and aggregations: I should be able to filter and group by that field like any other DB column
    • read-only: I should not be able to update the field
    • follow updates from parent field: when the parent has changed, the calculated field changes too

    Describe the solution you'd like

    I would like to be able to define a field as generated.

    import datetime
    import ormar
    from .base import BaseMeta
    
    
    class Foo(ormar.Model):
        class Meta(BaseMeta):
            pass
    
        id: int = ormar.Integer(primary_key=True)
        dt: datetime.date = ormar.Date()
        year: int = ormar.Integer(generated_as="CAST(strftime('%Y', dt) AS INTEGER)", stored=False, index=True)
    

    Describe alternatives you've considered

    Some of this behaviour can be achieved by defining signals for pre-update and pre-save, which would ensure the calculated column is always stored correctly.

    Anyways, this does not prevent me of changing the field value the model or in the DB.

    I've also considered ormar.property_field, but this would require all filtering and grouping being implemented on application side.

    Additional context

    enhancement 
    opened by m1racoli 1
Releases(0.10.22)
  • 0.10.22(Oct 15, 2021)

  • 0.10.21(Oct 13, 2021)

    0.10.21

    🐛 Fixes

    • Add ormar implementation of construct classmethod that allows to build Model instances without validating the input to speed up the whole flow, if your data is already validated #318
    • Fix for "inheriting" field validators from ormar model when newly created pydanic model is generated with get_pydantic #365
    Source code(tar.gz)
    Source code(zip)
  • 0.10.20(Sep 26, 2021)

    0.10.20

    ✨ Features

    • Add extra parameter in Model.Meta that accepts Extra.ignore and Extra.forbid (default) and either ignores the extra fields passed to ormar model or raises an exception if one is encountered #358

    🐛 Fixes

    • Allow None if field is nullable and have choices set #354
    • Always set primary_key to not null regardless of autoincrement and explicit nullable setting to avoid problems with migrations #348
    Source code(tar.gz)
    Source code(zip)
  • 0.10.19(Sep 13, 2021)

    0.10.19

    ✨ Features

    • Add support for multi-column non-unique IndexColumns in Meta.constraints #307
    • Add sql_nullable field attribute that allows to set different nullable setting for pydantic model and for underlying sql column #308

    🐛 Fixes

    • Enable caching of relation map to increase performance #337
    • Clarify and fix documentation in regard of nullable fields #339

    💬 Other

    • Bump supported databases version to <=5.2.
    Source code(tar.gz)
    Source code(zip)
  • 0.10.18(Sep 1, 2021)

  • 0.10.17(Aug 30, 2021)

  • 0.10.16(Aug 6, 2021)

    0.10.16

    ✨ Features

    • Allow passing your own pydantic Config to ormar.Model that will be merged with the default one by @naturalethic (thanks!) #285
    • Add SmallInteger field type by @ProgrammerPlus1998 (thanks!) #297

    🐛 Fixes

    • Fix generating openapi schema by removing obsolete pydantic field parameters that were directly exposed in schema #291
    • Fix unnecessary warning for auto generated through models #295
    Source code(tar.gz)
    Source code(zip)
  • 0.10.15(Jul 21, 2021)

    0.10.15

    🐛 Fixes

    • Fix generating pydantic models tree with nested models (by @pawamoy - thanks!) #278
    • Fix missing f-string in warning about missing primary key field #274
    • Fix passing foreign key value as relation (additional guard, fixed already in the latest release) #270
    Source code(tar.gz)
    Source code(zip)
  • 0.10.14(Jul 6, 2021)

    0.10.14

    ✨ Features

    • Allow passing timezone:bool = False parameter to DateTime and Time fields for timezone aware database columns #264
    • Allow passing datetime, date and time for filter on DateTime, Time and Date fields to allow filtering by datetimes instead of converting the filter value to string #79

    🐛 Fixes

    • Fix dependencies from psycopg2 to psycopg2-binary #255
    Source code(tar.gz)
    Source code(zip)
  • 0.10.13(Jun 25, 2021)

    0.10.13

    ✨ Features

    • Allow passing field accessors in select_related and prefetch_related aka. python style select_related #225.
      • Previously:
        await Post.objects.select_related(["author", "categories"]).get()
        await Author.objects.prefetch_related("posts__categories").get()
      
      • Now also:
        await Post.objects.select_related([Post.author, Post.categories]).get()
        await Author.objects.prefetch_related(Author.posts.categories).get()
      

    🐛 Fixes

    • Fix overwriting default value for inherited primary key #253
    Source code(tar.gz)
    Source code(zip)
  • 0.10.12(Jun 22, 2021)

  • 0.10.11(Jun 8, 2021)

    0.10.11

    ✨ Features

    • Add values and values_list to QuerySet and QuerysetProxy that allows to return raw data from query #223.
      • Allow returning list of tuples or list of dictionaries from a query
      • Skips parsing the data to ormar model so skips also the validation
      • Allow excluding models in between in chain of relations, so you can extract only needed columns
      • values_list allows you to flatten the result if you extract only one column.

    🐛 Fixes

    • Fix creation of auto through model for m2m relation with ForwardRef #226
    Source code(tar.gz)
    Source code(zip)
  • 0.10.10(Jun 2, 2021)

    0.10.10

    ✨ Features

    • Add get_pydantic function that allows you to auto generate equivalent pydantic models tree from ormar.Model. This newly generated model tree can be used in requests and responses to exclude fields you do not want to include in the data.
    • Add exclude_parent_fields parameter to model Meta that allows you to exclude fields from parent models during inheritance. Note that best practice is to combine models and mixins but if you have many similar models and just one that differs it might be useful tool to achieve that.

    🐛 Fixes

    • Fix is null filter with pagination and relations (by @erichaydel) #214
    • Fix not saving child object on reverse side of the relation if not saved before #216

    💬 Other

    • Expand fastapi part of the documentation to show samples of using ormar in requests and responses in fastapi.
    • Improve the docs in regard of default, ForeignKey.add etc.
    Source code(tar.gz)
    Source code(zip)
  • 0.10.9(May 30, 2021)

    0.10.9

    Important security fix

    • Update pin for pydantic to fix security vulnerability CVE-2021-29510

    You are advised to update to version of pydantic that was patched. In 0.10.9 ormar excludes versions with vulnerability in pinned dependencies.

    🐛 Fixes

    • Fix OpenAPi schema for LargeBinary #204
    Source code(tar.gz)
    Source code(zip)
  • 0.10.8(May 18, 2021)

    0.10.8

    🐛 Fixes

    • Fix populating default values in pk_only child models #202
    • Fix mypy for LargeBinary fields with base64 str representation #199
    • Fix OpenAPI schema format for LargeBinary fields with base64 str representation #199
    • Fix OpenAPI choices encoding for LargeBinary fields with base64 str representation
    Source code(tar.gz)
    Source code(zip)
  • 0.10.7(May 17, 2021)

    0.10.7

    ✨ Features

    • Add exclude_primary_keys: bool = False flag to dict() method that allows to exclude all primary key columns in the resulting dictionaru. #164
    • Add exclude_through_models: bool = False flag to dict() that allows excluding all through models from ManyToMany relations #164
    • Add represent_as_base64_str: bool = False parameter that allows conversion of bytes LargeBinary field to base64 encoded string. String is returned in dict(), on access to attribute and string is converted to bytes on setting. Data in database is stored as bytes. #187
    • Add pk alias to allow field access by Model.pk in filters and order by clauses (python style)

    🐛 Fixes

    • Remove default None option for max_length for LargeBinary field #186
    • Remove default None option for max_length for String field

    💬 Other

    • Provide a guide and samples of dict() parameters in the docs
    • Major refactor of getting/setting attributes from magic methods into descriptors -> noticeable performance improvement
    Source code(tar.gz)
    Source code(zip)
  • 0.10.6(May 2, 2021)

    0.10.6

    ✨ Features

    • Add LargeBinary(max_length) field type #166

    • Add support for normal pydantic fields (including Models) instead of pydantic_only attribute which is now deprecated #160. Pydantic fields should be declared normally as in pydantic model next to ormar fields, note that (obviously) ormar does not save and load the value for this field in database that mean that ONE of the following has to be true:

      • pydantic field declared on ormar model has to be Optional (defaults to None)
      • pydantic field has to have a default value set
      • pydantic field has default_factory function set
      • ormar.Model with pydantic field has to overwrite __init__() and provide the value there

      If none of the above ormar (or rather pydantic) will fail during loading data from the database, with missing required value for declared pydantic field.

    • Ormar provides now a meaningful examples in openapi schema, including nested models. The same algorithm is used to iterate related models without looks as with dict() and select/load_all. Examples appear also in fastapi. #157

    🐛 Fixes

    • By default pydantic is not validating fields during assignment, which is not a desirable setting for an ORM, now all ormar.Models have validation turned-on during assignment (like model.column = 'value')

    💬 Other

    • Add connecting to the database in QuickStart in readme #180
    • OpenAPI schema does no longer include ormar.Model docstring as description, instead just model name is provided if you do not provide your own docstring.
    • Some performance improvements.
    Source code(tar.gz)
    Source code(zip)
  • 0.10.5(Apr 23, 2021)

    0.10.5

    🐛 Fixes

    • Fix bug in fastapi-pagination #73
    • Remove unnecessary Optional in List[Optional[T]] in return value for QuerySet.all() and Querysetproxy.all() return values #174
    • Run tests coverage publish only on internal prs instead of all in github action.
    Source code(tar.gz)
    Source code(zip)
  • 0.10.4(Apr 21, 2021)

    0.10.4

    ✨ Features

    • Add Python style to filter and order_by with field access instead of dunder separated strings. #51
      • Accessing a field with attribute access (chain of dot notation) can be used to construct FilterGroups (ormar.and_ and ormar.or_)
      • Field access overloads set of python operators and provide a set of functions to allow same functionality as with dunder separated param names in **kwargs, that means that querying from sample model Track related to model Album now you have more options:
        • exact - exact match to value, sql column = <VALUE>
          • OLD: album__name__exact='Malibu'
          • NEW: can be also written as Track.album.name == 'Malibu
        • iexact - exact match sql column = <VALUE> (case insensitive)
          • OLD: album__name__iexact='malibu'
          • NEW: can be also written as Track.album.name.iexact('malibu')
        • contains - sql column LIKE '%<VALUE>%'
          • OLD: album__name__contains='Mal'
          • NEW: can be also written as Track.album.name % 'Mal')
          • NEW: can be also written as Track.album.name.contains('Mal')
        • icontains - sql column LIKE '%<VALUE>%' (case insensitive)
          • OLD: album__name__icontains='mal'
          • NEW: can be also written as Track.album.name.icontains('mal')
        • in - sql column IN (<VALUE1>, <VALUE2>, ...)
          • OLD: album__name__in=['Malibu', 'Barclay']
          • NEW: can be also written as Track.album.name << ['Malibu', 'Barclay']
          • NEW: can be also written as Track.album.name.in_(['Malibu', 'Barclay'])
        • isnull - sql column IS NULL (and sql column IS NOT NULL)
          • OLD: album__name__isnull=True (isnotnull album__name__isnull=False)
          • NEW: can be also written as Track.album.name >> None
          • NEW: can be also written as Track.album.name.is_null(True)
          • NEW: not null can be also written as Track.album.name.is_null(False)
          • NEW: not null can be also written as ~(Track.album.name >> None)
          • NEW: not null can be also written as ~(Track.album.name.is_null(True))
        • gt - sql column > <VALUE> (greater than)
          • OLD: position__gt=3
          • NEW: can be also written as Track.album.name > 3
        • gte - sql column >= <VALUE> (greater or equal than)
          • OLD: position__gte=3
          • NEW: can be also written as Track.album.name >= 3
        • lt - sql column < <VALUE> (lower than)
          • OLD: position__lt=3
          • NEW: can be also written as Track.album.name < 3
        • lte - sql column <= <VALUE> (lower equal than)
          • OLD: position__lte=3
          • NEW: can be also written as Track.album.name <= 3
        • startswith - sql column LIKE '<VALUE>%' (exact start match)
          • OLD: album__name__startswith='Mal'
          • NEW: can be also written as Track.album.name.startswith('Mal')
        • istartswith - sql column LIKE '<VALUE>%' (case insensitive)
          • OLD: album__name__istartswith='mal'
          • NEW: can be also written as Track.album.name.istartswith('mal')
        • endswith - sql column LIKE '%<VALUE>' (exact end match)
          • OLD: album__name__endswith='ibu'
          • NEW: can be also written as Track.album.name.endswith('ibu')
        • iendswith - sql column LIKE '%<VALUE>' (case insensitive)
          • OLD: album__name__iendswith='IBU'
          • NEW: can be also written as Track.album.name.iendswith('IBU')
    • You can provide FilterGroups not only in filter() and exclude() but also in:
      • get()
      • get_or_none()
      • get_or_create()
      • first()
      • all()
      • delete()
    • With FilterGroups (ormar.and_ and ormar.or_) you can now use:
      • & - as and_ instead of next level of nesting
      • | - as `or_' instead of next level of nesting
      • ~ - as negation of the filter group
    • To combine groups of filters into one set of conditions use & (sql AND) and | (sql OR)
      # Following queries are equivalent:
      # sql: ( product.name = 'Test'  AND  product.rating >= 3.0 ) 
      
      # ormar OPTION 1 - OLD one
      Product.objects.filter(name='Test', rating__gte=3.0).get()
      
      # ormar OPTION 2 - OLD one
      Product.objects.filter(ormar.and_(name='Test', rating__gte=3.0)).get()
      
      # ormar OPTION 3 - NEW one (field access)
      Product.objects.filter((Product.name == 'Test') & (Product.rating >=3.0)).get()
      
    • Same applies to nested complicated filters
      # Following queries are equivalent:
      # sql: ( product.name = 'Test' AND product.rating >= 3.0 ) 
      #       OR (categories.name IN ('Toys', 'Books'))
      
      # ormar OPTION 1 - OLD one
      Product.objects.filter(ormar.or_(
                                ormar.and_(name='Test', rating__gte=3.0), 
                                categories__name__in=['Toys', 'Books'])
                            ).get()
      
      # ormar OPTION 2 - NEW one (instead of nested or use `|`)
      Product.objects.filter(
                            ormar.and_(name='Test', rating__gte=3.0) | 
                            ormar.and_(categories__name__in=['Toys', 'Books'])
                            ).get()
      
      # ormar OPTION 3 - NEW one (field access)
      Product.objects.filter(
                            ((Product.name='Test') & (Product.rating >= 3.0)) | 
                            (Product.categories.name << ['Toys', 'Books'])
                            ).get()
      
    • Now you can also use field access to provide OrderActions to order_by()
      • Order ascending:
        • OLD: Product.objects.order_by("name").all()
        • NEW: Product.objects.order_by(Product.name.asc()).all()
      • Order descending:
        • OLD: Product.objects.order_by("-name").all()
        • NEW: Product.objects.order_by(Product.name.desc()).all()
      • You can of course also combine different models and many order_bys: Product.objects.order_by([Product.category.name.asc(), Product.name.desc()]).all()

    🐛 Fixes

    • Not really a bug but rather inconsistency. Providing a filter with nested model i.e. album__category__name = 'AA' is checking if album and category models are included in select_related() and if not it's auto-adding them there. The same functionality was not working for FilterGroups (and_ and or_), now it works (also for python style filters which return FilterGroups).
    Source code(tar.gz)
    Source code(zip)
  • 0.10.3(Apr 16, 2021)

    0.10.3

    ✨ Features

    • ForeignKey and ManyToMany now support skip_reverse: bool = False flag #118. If you set skip_reverse flag internally the field is still registered on the other side of the relationship so you can:

      • filter by related models fields from reverse model
      • order_by by related models fields from reverse model

      But you cannot:

      • access the related field from reverse model with related_name
      • even if you select_related from reverse side of the model the returned models won't be populated in reversed instance (the join is not prevented so you still can filter and order_by)
      • the relation won't be populated in dict() and json()
      • you cannot pass the nested related objects when populating from dict() or json() (also through fastapi). It will be either ignored or raise error depending on extra setting in pydantic Config.
    • Model.save_related() now can save whole data tree in once #148 meaning:

      • it knows if it should save main Model or related Model first to preserve the relation

      • it saves main Model if

        • it's not saved,
        • has no pk value
        • or save_all=True flag is set

        in those cases you don't have to split save into two calls (save() and save_related())

      • it supports also ManyToMany relations

      • it supports also optional Through model values for m2m relations

    • Add possibility to customize Through model relation field names.

    • By default Through model relation names default to related model name in lowercase. So in example like this:

      ... # course declaration ommited
      class Student(ormar.Model):
          class Meta:
              database = database
              metadata = metadata
      
          id: int = ormar.Integer(primary_key=True)
          name: str = ormar.String(max_length=100)
          courses = ormar.ManyToMany(Course)
      
      # will produce default Through model like follows (example simplified)
      class StudentCourse(ormar.Model):
          class Meta:
              database = database
              metadata = metadata
              tablename = "students_courses"
      
          id: int = ormar.Integer(primary_key=True)
          student = ormar.ForeignKey(Student) # default name
          course = ormar.ForeignKey(Course)  # default name
      
    • To customize the names of fields/relation in Through model now you can use new parameters to ManyToMany:

      • through_relation_name - name of the field leading to the model in which ManyToMany is declared
      • through_reverse_relation_name - name of the field leading to the model to which ManyToMany leads to

      Example:

      ... # course declaration ommited
      class Student(ormar.Model):
          class Meta:
              database = database
              metadata = metadata
      
          id: int = ormar.Integer(primary_key=True)
          name: str = ormar.String(max_length=100)
          courses = ormar.ManyToMany(Course,
                                     through_relation_name="student_id",
                                     through_reverse_relation_name="course_id")
      
      # will produce default Through model like follows (example simplified)
      class StudentCourse(ormar.Model):
          class Meta:
              database = database
              metadata = metadata
              tablename = "students_courses"
      
          id: int = ormar.Integer(primary_key=True)
          student_id = ormar.ForeignKey(Student) # set by through_relation_name
          course_id = ormar.ForeignKey(Course)  # set by through_reverse_relation_name
      
    Source code(tar.gz)
    Source code(zip)
  • 0.10.2(Apr 6, 2021)

    0.10.2

    ✨ Features

    • Model.save_related(follow=False) now accept also two additional arguments: Model.save_related(follow=False, save_all=False, exclude=None).
      • save_all:bool -> By default (so with save_all=False) ormar only upserts models that are not saved (so new or updated ones), with save_all=True all related models are saved, regardless of saved status, which might be useful if updated models comes from api call, so are not changed in the backend.
      • exclude: Union[Set, Dict, None] -> set/dict of relations to exclude from save, those relation won't be saved even with follow=True and save_all=True. To exclude nested relations pass a nested dictionary like: exclude={"child":{"sub_child": {"exclude_sub_child_realtion"}}}. The allowed values follow the fields/exclude_fields (from QuerySet) methods schema so when in doubt you can refer to docs in queries -> selecting subset of fields -> fields.
    • Model.update() method now accepts _columns: List[str] = None parameter, that accepts list of column names to update. If passed only those columns will be updated in database. Note that update() does not refresh the instance of the Model, so if you change more columns than you pass in _columns list your Model instance will have different values than the database!
    • Model.dict() method previously included only directly related models or nested models if they were not nullable and not virtual, now all related models not previously visited without loops are included in dict(). This should be not breaking as just more data will be dumped to dict, but it should not be missing.
    • QuerySet.delete(each=False, **kwargs) previously required that you either pass a filter (by **kwargs or as a separate filter() call) or set each=True now also accepts exclude() calls that generates NOT filter. So either each=True needs to be set to delete whole table or at least one of filter/exclude clauses.
    • Same thing applies to QuerySet.update(each=False, **kwargs) which also previously required that you either pass a filter (by **kwargs or as a separate filter() call) or set each=True now also accepts exclude() calls that generates NOT filter. So either each=True needs to be set to update whole table or at least one of filter/exclude clauses.
    • Same thing applies to QuerysetProxy.update(each=False, **kwargs) which also previously required that you either pass a filter (by **kwargs or as a separate filter() call) or set each=True now also accepts exclude() calls that generates NOT filter. So either each=True needs to be set to update whole table or at least one of filter/exclude clauses.

    🐛 Fixes

    • Fix improper relation field resolution in QuerysetProxy if fk column has different database alias.
    • Fix hitting recursion error with very complicated models structure with loops when calling dict().
    • Fix bug when two non-relation fields were merged (appended) in query result when they were not relation fields (i.e. JSON)
    • Fix bug when during translation to dict from list the same relation name is used in chain but leads to different models
    • Fix bug when bulk_create would try to save also property_field decorated methods and pydantic fields
    • Fix wrong merging of deeply nested chain of reversed relations

    💬 Other

    • Performance optimizations
    • Split tests into packages based on tested area
    Source code(tar.gz)
    Source code(zip)
  • 0.10.1(Mar 23, 2021)

    0.10.1

    Features

    • add get_or_none(**kwargs) method to QuerySet and QuerysetProxy. It is exact equivalent of get(**kwargs) but instead of raising ormar.NoMatch exception if there is no db record matching the criteria, get_or_none simply returns None.

    Fixes

    • Fix dialect dependent quoting of column and table names in order_by clauses not working properly in postgres.
    Source code(tar.gz)
    Source code(zip)
  • 0.10.0(Mar 23, 2021)

    0.10.0

    Breaking

    • Dropped supported for long deprecated notation of field definition in which you use ormar fields as type hints i.e. test_field: ormar.Integger() = None
    • Improved type hints -> mypy can properly resolve related models fields (ForeignKey and ManyToMany) as well as return types of QuerySet methods. Those mentioned are now returning proper model (i.e. Book) instead or ormar.Model type. There is still problem with reverse sides of relation and QuerysetProxy methods, to ease type hints now those return Any. Partially fixes #112.

    Features

    • add select_all(follow: bool = False) method to QuerySet and QuerysetProxy. It is kind of equivalent of the Model's load_all() method but can be used directly in a query. By default select_all() adds only directly related models, with follow=True also related models of related models are added without loops in relations. Note that it's not and end async model so you still have to issue get(), all() etc. as select_all() returns a QuerySet (or proxy) like fields() or order_by(). #131

    Internals

    • ormar fields are no longer stored as classes in Meta.model_fields dictionary but instead they are stored as instances.
    Source code(tar.gz)
    Source code(zip)
  • 0.9.9(Mar 15, 2021)

    0.9.9

    Features

    • Add possibility to change default ordering of relations and models.
      • To change model sorting pass orders_by = [columns] where columns: List[str] to model Meta class
      • To change relation order_by pass orders_by = [columns] where columns: List[str]
      • To change reverse relation order_by pass related_orders_by = [columns] where columns: List[str]
      • Arguments can be column names or -{col_name} to sort descending
      • In relations you can sort only by directly related model columns or for ManyToMany columns also Through model columns "{through_field_name}__{column_name}"
      • Order in which order_by clauses are applied is as follows:
        • Explicitly passed order_by() calls in query
        • Relation passed orders_by if exists
        • Model Meta class orders_by
        • Model primary key column asc (fallback, used if none of above provided)
    • Add 4 new aggregated functions -> min, max, sum and avg that are their corresponding sql equivalents.
      • You can pass one or many column names including related columns.
      • As of now each column passed is aggregated separately (so sum(col1+col2) is not possible, you can have sum(col1, col2) and later add 2 returned sums in python)
      • You cannot sum and avg non numeric columns
      • If you aggregate on one column, the single value is directly returned as a result
      • If you aggregate on multiple columns a dictionary with column: result pairs is returned
    • Add 4 new signals -> pre_relation_add, post_relation_add, pre_relation_remove and post_relation_remove
      • The newly added signals are emitted for ManyToMany relations (both sides) and reverse side of ForeignKey relation (same as QuerysetProxy is exposed).
      • Signals recieve following args: sender: Type[Model] - sender class, instance: Model - instance to which related model is added, child: Model - model being added, relation_name: str - name of the relation to which child is added, for add signals also passed_kwargs: Dict - dict of kwargs passed to add()

    Changes

    • Through models for ManyToMany relations are now instantiated on creation, deletion and update, so you can provide not only autoincrement int as a primary key but any column type with default function provided.
    • Since Through models are now instantiated you can also subscribe to Through model pre/post save/update/delete signals
    • pre_update signals receivers now get also passed_args argument which is a dict of values passed to update function if any (else empty dict)

    Fixes

    • pre_update signal now is sent before the extraction of values so you can modify the passed instance in place and modified fields values will be reflected in database
    • bulk_update now works correctly also with UUID primary key column type
    Source code(tar.gz)
    Source code(zip)
  • 0.9.8(Mar 10, 2021)

    0.9.8

    Features

    • Add possibility to encrypt the selected field(s) in the database
      • As minimum you need to provide encrypt_secret and encrypt_backend
      • encrypt_backend can be one of the ormar.EncryptBackends enum (NONE, FERNET, HASH, CUSTOM) - default: NONE
      • When custom backend is selected you need to provide your backend class that subclasses ormar.fields.EncryptBackend
      • You cannot encrypt primary_key column and relation columns (FK and M2M).
      • Provided are 2 backends: HASH and FERNET
        • HASH is a one-way hash (like for password), never decrypted on retrieval
        • FERNET is a two-way encrypt/decrypt backend
      • Note that in FERNET backend you loose filtering possibility altogether as part of the encrypted value is a timestamp.
      • Note that in HASH backend you can filter by full value but filters like contain will not work as comparison is make on encrypted values
      • Note that adding encrypt_backend changes the database column type to TEXT, which needs to be reflected in db either by migration or manual change

    Fixes

    • (Advanced/ Internal) Restore custom sqlalchemy types (by types.TypeDecorator subclass) functionality that ceased to working so process_result_value was never called
    Source code(tar.gz)
    Source code(zip)
  • 0.9.7(Mar 9, 2021)

    0.9.7

    Features

    • Add isnull operator to filter and exclude methods.
      album__name__isnull=True #(sql: album.name is null)
      album__name__isnull=False #(sql: album.name is not null))
      
    • Add ormar.or_ and ormar.and_ functions that can be used to compose complex queries with nested conditions. Sample query:
      books = (
          await Book.objects.select_related("author")
          .filter(
              ormar.and_(
                  ormar.or_(year__gt=1960, year__lt=1940),
                  author__name="J.R.R. Tolkien",
              )
          )
          .all()
      )
      

      Check the updated docs in Queries -> Filtering and sorting -> Complex filters

    Other

    • Setting default on ForeignKey or ManyToMany raises and ModelDefinition exception as it is (and was) not supported
    Source code(tar.gz)
    Source code(zip)
  • 0.9.6(Mar 5, 2021)

    0.9.6

    Important

    • Through model for ManyToMany relations now becomes optional. It's not a breaking change since if you provide it everything works just fine as it used to. So if you don't want or need any additional fields on Through model you can skip it. Note that it's going to be created for you automatically and still has to be included in example in alembic migrations. If you want to delete existing one check the default naming convention to adjust your existing database structure.

      Note that you still need to provide it if you want to customize the Through model name or the database table name.

    Features

    • Add update method to QuerysetProxy so now it's possible to update related models directly from parent model in ManyToMany relations and in reverse ForeignKey relations. Note that update like in QuerySet update returns number of updated models and does not update related models in place on parent model. To get the refreshed data on parent model you need to refresh the related models (i.e. await model_instance.related.all())
    • Add load_all(follow=False, exclude=None) model method that allows to load current instance of the model with all related models in one call. By default it loads only directly related models but setting follow=True causes traversing the tree (avoiding loops). You can also pass exclude parameter that works the same as QuerySet.exclude_fields() method.
    • Added possibility to add more fields on Through model for ManyToMany relationships:
      • name of the through model field is the lowercase name of the Through class
      • you can pass additional fields when calling add(child, **kwargs) on relation (on QuerysetProxy)
      • you can pass additional fields when calling create(**kwargs) on relation (on QuerysetProxy) when one of the keyword arguments should be the through model name with a dict of values
      • you can order by on through model fields
      • you can filter on through model fields
      • you can include and exclude fields on through models
      • through models are attached only to related models (i.e. if you query from A to B -> only on B)
      • note that through models are explicitly loaded without relations -> relation is already populated in ManyToMany field.
      • note that just like before you cannot declare the relation fields on through model, they will be populated for you by ormar, but now if you try to do so ModelDefinitionError will be thrown
      • check the updated ManyToMany relation docs for more information

    Other

    • Updated docs and api docs
    • Refactors and optimisations mainly related to filters, exclusions and order bys
    Source code(tar.gz)
    Source code(zip)
  • 0.9.5(Feb 28, 2021)

    0.9.5

    Fixes

    • Fix creation of pydantic FieldInfo after update of pydantic to version >=1.8
    • Pin required dependency versions to avoid such situations in the future
    Source code(tar.gz)
    Source code(zip)
  • 0.9.4(Feb 17, 2021)

  • 0.9.3(Feb 11, 2021)

    Fixes

    • Fix JSON field being double escaped when setting value after initialization
    • Fix JSON field not respecting nullable field setting due to pydantic internals
    • Fix choices verification for JSON field
    • Fix choices not being verified when setting the attribute after initialization
    • Fix choices not being verified during update call from QuerySet
    Source code(tar.gz)
    Source code(zip)
An async ORM. 🗃

ORM The orm package is an async ORM for Python, with support for Postgres, MySQL, and SQLite. ORM is built with: SQLAlchemy core for query building. d

Encode 1.4k Oct 11, 2021
Piccolo - A fast, user friendly ORM and query builder which supports asyncio.

A fast, user friendly ORM and query builder which supports asyncio.

null 566 Oct 20, 2021
a small, expressive orm -- supports postgresql, mysql and sqlite

peewee Peewee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use. a small, expressive ORM p

Charles Leifer 8.7k Oct 25, 2021
The Orator ORM provides a simple yet beautiful ActiveRecord implementation.

Orator The Orator ORM provides a simple yet beautiful ActiveRecord implementation. It is inspired by the database part of the Laravel framework, but l

Sébastien Eustace 1.3k Oct 17, 2021
A curated list of awesome tools for SQLAlchemy

Awesome SQLAlchemy A curated list of awesome extra libraries and resources for SQLAlchemy. Inspired by awesome-python. (See also other awesome lists!)

Hong Minhee (洪 民憙) 2.3k Oct 21, 2021
Pony Object Relational Mapper

Downloads Pony Object-Relational Mapper Pony is an advanced object-relational mapper. The most interesting feature of Pony is its ability to write que

null 2.7k Oct 25, 2021
A very simple CRUD class for SQLModel! ✨

Base SQLModel A very simple CRUD class for SQLModel! ✨ Inspired on: Full Stack FastAPI and PostgreSQL - Base Project Generator FastAPI Microservices I

Marcelo Trylesinski 12 Oct 22, 2021
SQLModel is a library for interacting with SQL databases from Python code, with Python objects.

SQLModel is a library for interacting with SQL databases from Python code, with Python objects. It is designed to be intuitive, easy to use, highly compatible, and robust.

Sebastián Ramírez 5.5k Oct 24, 2021
A Python Object-Document-Mapper for working with MongoDB

MongoEngine Info: MongoEngine is an ORM-like layer on top of PyMongo. Repository: https://github.com/MongoEngine/mongoengine Author: Harry Marr (http:

MongoEngine 3.6k Oct 24, 2021
A pure Python Database Abstraction Layer

pyDAL pyDAL is a pure Python Database Abstraction Layer. It dynamically generates the SQL/noSQL in realtime using the specified dialect for the databa

null 397 Oct 23, 2021
Pydantic model support for Django ORM

Pydantic model support for Django ORM

Jordan Eremieff 189 Oct 23, 2021
Adds SQLAlchemy support to Flask

Flask-SQLAlchemy Flask-SQLAlchemy is an extension for Flask that adds support for SQLAlchemy to your application. It aims to simplify using SQLAlchemy

The Pallets Projects 3.6k Oct 22, 2021
A pythonic interface to Amazon's DynamoDB

PynamoDB A Pythonic interface for Amazon's DynamoDB. DynamoDB is a great NoSQL service provided by Amazon, but the API is verbose. PynamoDB presents y

null 1.7k Oct 23, 2021
A Python Library for Simple Models and Containers Persisted in Redis

Redisco Python Containers and Simple Models for Redis Description Redisco allows you to store objects in Redis. It is inspired by the Ruby library Ohm

sebastien requiem 434 Oct 1, 2021
Beanie - is an Asynchronous Python object-document mapper (ODM) for MongoDB

Beanie - is an Asynchronous Python object-document mapper (ODM) for MongoDB, based on Motor and Pydantic.

Roman 277 Oct 18, 2021
MongoEngine flask extension with WTF model forms support

Flask-MongoEngine Info: MongoEngine for Flask web applications. Repository: https://github.com/MongoEngine/flask-mongoengine About Flask-MongoEngine i

MongoEngine 788 Oct 14, 2021
Easy-to-use data handling for SQL data stores with support for implicit table creation, bulk loading, and transactions.

dataset: databases for lazy people In short, dataset makes reading and writing data in databases as simple as reading and writing JSON files. Read the

Friedrich Lindenberg 4.1k Oct 18, 2021
Rich Python data types for Redis

Created by Stephen McDonald Introduction HOT Redis is a wrapper library for the redis-py client. Rather than calling the Redis commands directly from

Stephen McDonald 273 Sep 2, 2021