Tortoise ORM is an easy-to-use asyncio ORM inspired by Django.

Overview

Tortoise ORM

https://badges.gitter.im/tortoise/community.png https://img.shields.io/pypi/v/tortoise-orm.svg?style=flat https://readthedocs.org/projects/tortoise-orm/badge/?version=latest https://pepy.tech/badge/tortoise-orm/month https://api.codacy.com/project/badge/Grade/b5b77021ba284e4a9e0c033a4611b046

Introduction

Tortoise ORM is an easy-to-use asyncio ORM (Object Relational Mapper) inspired by Django.

Tortoise ORM was build with relations in mind and admiration for the excellent and popular Django ORM. It's engraved in it's design that you are working not with just tables, you work with relational data.

You can find docs at ReadTheDocs

Note

Tortoise ORM is young project and breaking changes are to be expected. We keep a Changelog and it will have possible breakage clearly documented.

Tortoise ORM is supported on CPython >= 3.7 for SQLite, MySQL and PostgreSQL.

Why was Tortoise ORM built?

Python has many existing and mature ORMs, unfortunately they are designed with an opposing paradigm of how I/O gets processed. asyncio is relatively new technology that has a very different concurrency model, and the largest change is regarding how I/O is handled.

However, Tortoise ORM is not first attempt of building asyncio ORM, there are many cases of developers attempting to map synchronous python ORMs to the async world, initial attempts did not have a clean API.

Hence we started Tortoise ORM.

Tortoise ORM is designed to be functional, yet familiar, to ease the migration of developers wishing to switch to asyncio.

It also performs well when compared to other Python ORMs, trading places with Pony ORM:

https://raw.githubusercontent.com/tortoise/tortoise-orm/develop/docs/ORM_Perf.png

How is an ORM useful?

When you build an application or service that uses a relational database, there is a point when you can't just get away with just using parameterized queries or even query builder, you just keep repeating yourself, writing slightly different code for each entity. Code has no idea about relations between data, so you end up concatenating your data almost manually. It is also easy to make a mistake in how you access your database, making it easy for SQL-injection attacks to occur. Your data rules are also distributed, increasing the complexity of managing your data, and even worse, is applied inconsistently.

An ORM (Object Relational Mapper) is designed to address these issues, by centralising your data model and data rules, ensuring that your data is managed safely (providing immunity to SQL-injection) and keeps track of relationships so you don't have to.

Getting Started

Installation

First you have to install tortoise like this:

pip install tortoise-orm

You can also install with your db driver (aiosqlite is builtin):

pip install tortoise-orm[asyncpg]

Or for MySQL:

pip install tortoise-orm[aiomysql]

Or another asyncio MySQL driver asyncmy:

pip install tortoise-orm[asyncmy]

Quick Tutorial

Primary entity of tortoise is tortoise.models.Model. You can start writing models like this:

from tortoise.models import Model
from tortoise import fields

class Tournament(Model):
    id = fields.IntField(pk=True)
    name = fields.TextField()

    def __str__(self):
        return self.name


class Event(Model):
    id = fields.IntField(pk=True)
    name = fields.TextField()
    tournament = fields.ForeignKeyField('models.Tournament', related_name='events')
    participants = fields.ManyToManyField('models.Team', related_name='events', through='event_team')

    def __str__(self):
        return self.name


class Team(Model):
    id = fields.IntField(pk=True)
    name = fields.TextField()

    def __str__(self):
        return self.name

After you defined all your models, tortoise needs you to init them, in order to create backward relations between models and match your db client with appropriate models.

You can do it like this:

from tortoise import Tortoise

async def init():
    # Here we connect to a SQLite DB file.
    # also specify the app name of "models"
    # which contain models from "app.models"
    await Tortoise.init(
        db_url='sqlite://db.sqlite3',
        modules={'models': ['app.models']}
    )
    # Generate the schema
    await Tortoise.generate_schemas()

Here we create connection to SQLite database in the local directory called db.sqlite3, and then we discover & initialise models.

Tortoise ORM currently supports the following databases:

  • SQLite (requires aiosqlite)
  • PostgreSQL (requires asyncpg)
  • MySQL (requires aiomysql)

generate_schema generates the schema on an empty database. Tortoise generates schemas in safe mode by default which includes the IF NOT EXISTS clause, so you may include it in your main code.

After that you can start using your models:

# Create instance by save
tournament = Tournament(name='New Tournament')
await tournament.save()

# Or by .create()
await Event.create(name='Without participants', tournament=tournament)
event = await Event.create(name='Test', tournament=tournament)
participants = []
for i in range(2):
    team = await Team.create(name='Team {}'.format(i + 1))
    participants.append(team)

# M2M Relationship management is quite straightforward
# (also look for methods .remove(...) and .clear())
await event.participants.add(*participants)

# You can query related entity just with async for
async for team in event.participants:
    pass

# After making related query you can iterate with regular for,
# which can be extremely convenient for using with other packages,
# for example some kind of serializers with nested support
for team in event.participants:
    pass


# Or you can make preemptive call to fetch related objects
selected_events = await Event.filter(
    participants=participants[0].id
).prefetch_related('participants', 'tournament')

# Tortoise supports variable depth of prefetching related entities
# This will fetch all events for team and in those events tournaments will be prefetched
await Team.all().prefetch_related('events__tournament')

# You can filter and order by related models too
await Tournament.filter(
    events__name__in=['Test', 'Prod']
).order_by('-events__participants__name').distinct()

Migration

Tortoise ORM use Aerich as database migrations tool, see more detail at it's docs.

Contributing

Please have a look at the Contribution Guide

License

This project is licensed under the Apache License - see the LICENSE.txt file for details

Issues
  • Migrations

    Migrations

    (edited by @grigi ) Migrations is currently planned as a post-v1.0 feature.

    Some useful resources for getting this working right now:

    • https://github.com/tortoise/tortoise-orm/issues/8#issuecomment-534946871 (Native python, MySQL)
    • https://github.com/tortoise/tortoise-orm/issues/8#issuecomment-575982472 (Go, MySQL/PostgreSQL/SQLite)

    Forward migrations

    Best guess at this time for a complete solution, not a "quick" solution:

    • [ ] Make the describe_model() contain all the data one needs to generate DDL from.
    • [ ] Generate a sequence of high-level DDL instructions from describe_model()
    • [ ] Port generate_schema to use this high-level DDL instructions to generate a schema in the requested dialect. (This would require some decoupling from the driver instance, look at #72)
    • [ ] Build a persistance model (read/write) for high-level DDL insctuctions much like Django Migrations
    • [ ] And need a way to run "management" commands. So possibly a standard command-line utility.
    • [ ] Build a full model (e.g. same format as describe_model())from a series of high-level DDL instructions (as in read the migration persistance model)
    • [ ] Diff models, and generate a diff DDL instruction set.
    • [ ] If a Diff requires a default not pre-specified, we need to ask the user for something.
    • [ ] Have some way of determining the "version" of the models persisted in the DB.
    • [ ] Put it all together and make it work

    I'm not particularily happy about many migration systems storing state in the DB itself, as for some environments that won't work, but it is a very good place from a consistency PoV. We should have a backup for when we generate DDL that one would pass on to the DB team in cases where DDL changes have an established process (e.g. enterprise/consulting)

    Data migrations

    Proper data migration management makes this very useful:

    • [ ] Allow custom user scripts as migrations
    • [ ] Allow data-migrations to get a version of a model as it was at that time. So ability to build a model class from the intermediate describe_model() data
    • [ ] Handle DB edge cases, like Postgres being able to do DDL changes in a transaction OR data changes in a transaction, but not both.

    Backward migrations

    There is some cases where backwards migrations cannot be done as a forward migration clobbered/destroyed some data. In cases where we detect that we need mising data, and request a sane default. What do we do about data migrations? Is it safe to just emit a warning?

    discussion Future Release 
    opened by abondar 31
  • pymysql.err.InternalError: Packet sequence number wrong - got X expected 1

    pymysql.err.InternalError: Packet sequence number wrong - got X expected 1

    Describe the bug I use aiohttp+tortoise-orm+mysql for simple API, which can only insert a row to a single mysql database table called "TableName" and read the rows from it. If my API is idle for some time, say, 3 hours, and I do API request which should return me table rows, it shows "pymysql.err.InternalError: Packet sequence number wrong - got 0 expected 1" (not necessarily 0, it could be other number) error and status 500 instead of returning rows.

    It shows this error when run this piece of code: rows = await TableName.filter(some_field=some_field)

    To Reproduce

    1. Create models.py file:
    from tortoise import Model, fields
    
    
    class TableName(Model):
        id = fields.IntField(pk=True)
        some_field = fields.CharField(250)
    
    1. Create main.py file:
    from aiohttp import web
    from tortoise.contrib.aiohttp import register_tortoise
    
    from models import TableName
    
    
    async def list_rows(request):
        data = await request.json()
        try:
            some_field = data['some_field']
        except KeyError:
            return web.json_response({'message': f'Not all fields were specified'}, status=400)
    
        rows = await TableName.filter(some_field=some_field)
    
        rows_json = [row.__dict__ for row in rows]
        return web.json_response(rows_json)
    
    
    app = web.Application()
    app.add_routes([
        web.post("/rows", list_rows)
    ])
    
    register_tortoise(
        app, db_url="mysql://user:[email protected]:3306/any-database-name", modules={"models": ["models"]},
        generate_schemas=True
    )
    
    if __name__ == "__main__":
        web.run_app(app, port=6000)
    
    1. pack it into docker container with such Dockerfile:
    FROM python:3.9.5
    COPY requirements.txt . 
    RUN pip install -r requirements.txt
    COPY . .
    CMD ["python", "main.py"]
    

    docker build -t image-name:tag-name .

    1. Run API with command:

    docker run -d -p 6000:6000 image-name:tag-name

    1. After 3 hours of working, run curl query:

    curl -i -X POST --data '{"some_field": "blabla"}' http://localhost:6000/rows

    1. curl will return 500 error, inside container's logs you will see error:
    pymysql.err.InternalError: Packet sequence number wrong - got 35 expected 1
    

    Expected behavior curl query should return json with rows' values in it.

    Additional context Full trace:

    ERROR:aiohttp.server:Error handling request
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/tortoise/backends/mysql/client.py", line 44, in translate_exceptions_
        return await func(self, *args)
      File "/usr/local/lib/python3.9/site-packages/tortoise/backends/mysql/client.py", line 199, in execute_query
        await cursor.execute(query, values)
      File "/usr/local/lib/python3.9/site-packages/aiomysql/cursors.py", line 239, in execute
        await self._query(query)
      File "/usr/local/lib/python3.9/site-packages/aiomysql/cursors.py", line 457, in _query
        await conn.query(q)
      File "/usr/local/lib/python3.9/site-packages/aiomysql/connection.py", line 428, in query
        await self._read_query_result(unbuffered=unbuffered)
      File "/usr/local/lib/python3.9/site-packages/aiomysql/connection.py", line 622, in _read_query_result
        await result.read()
      File "/usr/local/lib/python3.9/site-packages/aiomysql/connection.py", line 1105, in read
        first_packet = await self.connection._read_packet()
      File "/usr/local/lib/python3.9/site-packages/aiomysql/connection.py", line 574, in _read_packet
        raise InternalError(
    pymysql.err.InternalError: Packet sequence number wrong - got 0 expected 1
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
        resp = await self._request_handler(request)
      File "/usr/local/lib/python3.9/site-packages/aiohttp/web_app.py", line 499, in _handle
        resp = await handler(request)
      File "//main.py", line 37, in list_rowa
        rows = await TableName.filter(some_field=some_field)
      File "/usr/local/lib/python3.9/site-packages/tortoise/queryset.py", line 879, in _execute
        instance_list = await self._db.executor_class(
      File "/usr/local/lib/python3.9/site-packages/tortoise/backends/base/executor.py", line 124, in execute_select
        _, raw_results = await self.db.execute_query(query.get_sql())
      File "/usr/local/lib/python3.9/site-packages/tortoise/backends/mysql/client.py", line 52, in translate_exceptions_
        raise OperationalError(exc)
    tortoise.exceptions.OperationalError: Packet sequence number wrong - got 0 expected 1
    
    opened by osintegrator 27
  • Add contains and contained_by filter to JSONField

    Add contains and contained_by filter to JSONField

    Description

    JSON is a native type in postgresql (since 9.2) and you can apply the filter on json object. The tortoise-orm not supported some filters such as contains , contained_by, filter by key and index in JSONField.

    Motivation and Context

    For example when we have an array of objects in json , we need to filter jsonfield if contains some objects or vice versa. Also to search inside the object by keys and index with equal , is_null, not_is_null, not options.

    How Has This Been Tested?

    I have added some methods in TestJSONFields class for testing.

    Checklist:

    • [x] My code follows the code style of this project.
    • [x] My change requires a change to the documentation.
    • [x] I have updated the documentation accordingly.
    • [x] I have read the CONTRIBUTING document.
    • [x] I have added tests to cover my changes.
    • [x] All new and existing tests passed.
    opened by ahmadgh74 22
  • fix bug with select_related not yielding null #up

    fix bug with select_related not yielding null #up

    Fixed (a bug ???) when select_related yielded unfilled instances of related objects instead of just nulls.

    Description

    Ran into this unexpected behaviour with something like this:

    class B(Model):
          id = fields.UUIDField(pk=True)
    class A(Model):
        the_b = fields.OneToOneField(
            "models.B", related_name="the_a", on_delete=fields.SET_NULL, null=True
        )
    

    then i got it like:

    a1 = await A.create(the_b=await B.create())
    a2 = await A.create(the_b=None)
    a = A.all().select_related('the_b').get(id=a2.id)
    # and for some reason the next checks failed
    assert a.the_b is None
    assert a.the_b == a2.the_b
    

    Motivation and Context

    Looks like this packages follows the ideas of django-orm and duplicates its best features in most cases, so i thought, like at least select_related behaviour should be same as it in django-orm. linked to this issue https://github.com/tortoise/tortoise-orm/issues/825.

    How Has This Been Tested?

    tested this with code like the one above. + as import dep in my pet proj, just assured that it don't gives me issues anymore.

    Checklist:

    • [x] My code follows the code style of this project.
    • [ ] My change requires a change to the documentation.
    • [ ] I have updated the documentation accordingly.
    • [x] I have read the CONTRIBUTING document.
    • [x] I have added tests to cover my changes.
    • [x] All new and existing tests passed.
    opened by urm8 19
  • Q-objects

    Q-objects

    after working withe the Q objects i have some questions, remarks and things that don't work as intended/expected

    • a blank Q object in AND mode works with just 1 argument to filter on, however OR mode requires a minimum of 2, quite inconvienient when your Q object is assembled based on client data
    • you can't combine keywords and nested Q objects into the same Q object but you got to wrap them into seperate Q objects to then combine into one master Q object to handle both, throws an explicit exception so seems to be intended behaviour but makes things more complex
    • are there any performance penalties to using nested Q objects several layers deep (other then ofc the overhead of python constructing objects)
    bug enhancement Next Release Waiting for feedback 
    opened by AEnterprise 18
  • Duplicate model instance

    Duplicate model instance

    Is your feature request related to a problem? Please describe. Usually in django to duplicate a model and to not copy every field there is a work around that is pretty straightforward and easy to do. This can be done via setting the pk to None. But with tortoise is not that easy, setting also the attribute ._saved_in_db is necessary.

    Describe the solution you'd like Not to access a private variables for this trick to work.

    Describe alternatives you've considered Maybe a copy method as well, that would be much more intuitive.

    Additional context Example of the code needed as of now

    book = Book(title='1984', author='George Orwell')
    
    print(book.pk)
    
    await book.save()
    
    print(book.pk)
    
    book.pk = None
    book._saved_in_db = False
    book.title = 'Animal farm'
    await book.save()
    
    print(book.pk)
    
    print(await Book.all().count())
    

    Example of the code that I would like to use instead

    book = Book(title='1984', author='George Orwell')
    
    print(book.pk)
    
    await book.save()
    
    print(book.pk)
    
    book.pk = None
    book.title = 'Animal farm'
    await book.save()
    
    print(book.pk)
    
    print(await Book.all().count())
    
    enhancement Next Release 
    opened by WisdomPill 18
  • Ability to SUM multiple columns

    Ability to SUM multiple columns

    Is your feature request related to a problem? Please describe. I'd like to be able to use the Sum function over multiple columns.

    Example for a model

    class Stats(Model):
        one_column = fields.IntField(default=0)
        two_column = fields.IntField(default=0)
    

    I'd like to be able to do run a query like select sum(one_column+two_column) from stats

    Describe the solution you'd like Stats.annotate(total_sum=Sum('one_column+two_column')) (or something similar)

    Additional context

    Pypika can already resolve this as seen in this test:

    https://github.com/kayak/pypika/blob/16339f85ed871d937609e9dbe26487b382ca5211/pypika/tests/test_formats.py

    fn.Sum(self.table_abc.fizz + self.table_abc.buzz)

    enhancement 
    opened by bbedward 18
  • New pool (WIP)

    New pool (WIP)

    Now that code base is simpler, and test runner should be more sane, attempt to add connection pooling for the third time.

    The plan is to change to connection pooling ONLY, as we currently implement persistent connections, but only one persistent connection. A connection pool should:

    • add robustness (if connection dies, then reconnect)
    • Allow multiple DB clients to operate at the same time (up to maxsize)
    • Allow more conflicts, so we need to handle rollback/retries explicitly.

    Things done:

    • [x] Change to a connection pooling system for MySQL.
    • [ ] Add tests for concurrency
    • [ ] Add tests for robustness (hackery allowed)
    • [ ] Add tests for handling conflicts.

    Concerns:

    • Can SQLite be concurrent at all? If difficult, should we limit it?
    • We need to add concurrency to the benchmarks, to manage performance
    opened by grigi 18
  • Differentiate between timezone aware and timezone naive datetimes

    Differentiate between timezone aware and timezone naive datetimes

    SQLite supports any valid ISO8601 datetime string when defining dates (amongst others) which may or may not include the timezone. This means that a field may have a mixture of timezone aware and timezone naive values. Postgres differentiates between timezone aware and timezone naive fields and so code that is valid when using the ORM with sqlite may cause errors in other engines.

    Ideally, the DatetimeField, since it is not timezone aware, would catch any timezone aware datetimes and provide a descriptive error instead trying (and failing) to insert them. Another option is to provide the desired timezone that the datetime should be stored in, so that the field can convert a timezone-aware datetime into the timezone-naive datetime for the supplied timezone.

    class Event(Model):
        ...
        start_time = fields.DatetimeField(in_timezone=timezone.utc)
    

    In the example above, any timezone-aware datetime is converted to a timezone-naive utc time before it is saved. If the timezone is supplied, datetimes retrieved from the model are timezone aware.

    Is this reasonable? Or should we remove the in_timezone flag and create a separate timezone aware field? For some context, I am receiving strings that may or may not be timezone-aware and it is a pain having to manually guard and convert them over. So if I were able to define on the model that all datetimes once they hit the database are guaranteed UTC then it would vastly simplify the logic.

    opened by arlyon 17
  • Model Signals

    Model Signals

    Does tortoise saves the old values before a save? So I can compare what has been changed.

    The context here is I'm trying to create a "application trigger" module, for example :

    #When a task is done, all its child tasks should be automatically done too
    @Task.bind('after-save')
    async def update_child_tasks(self, old_values):
      if old_values.status != self.status:
        if self.status == 'done'
          await Task.filter(mother_task=self).update(status='done')
    

    This is only a small example, there are many other cases where I need triggers like this

    Hope it is understood what I'm trying to do, emphasis on trying, I'm accepting ideas on different ways to approach this.

    Also it wouldn't work on batch QuerySet methods, like update or batch_create, if anyone has any idea on how to overcome this that'd be great

    enhancement discussion 
    opened by mojimi 17
  • Automatic conversion of validators in pydantic_model_creator

    Automatic conversion of validators in pydantic_model_creator

    Is your feature request related to a problem? Please describe. Consider the following example of a route using FastAPI:

    class Person(Model):
      name = fields.CharField(50)
      age = fields.IntField(validators=[MinValueValidator(1)]
    
    PersonSchema = pydantic_model_creator(Person)
    
    @fast_api_app.post("/", response_model=PersonSchema)
    async def create_person(data: PersonSchema):
      p = Person.create(**data.dict())
      return p
    

    The PersonSchema does not contain the MinValueValidator. So the PersonSchema can hold an invalid age value, only throwing ValidationError when calling Person.create.

    Describe the solution you'd like I would like that pydantic_model_creator would create a model with the same validators as specified on the Tortoise Model.

    Describe alternatives you've considered I tried this:

    def create_pydantic_model(model: Model, **kwargs):
        schema = pydantic_model_creator(model, **kwargs)
        for name, field in model._meta.fields_map.items():
            validators = [Validator(v) for v in field.validators]
            [setattr(v.func, "__name__", name + "_validator") for v in validators]
            schema.__validators__[name] = validators
        return schema
    

    But Tortoise validators have this signature function(value: T) -> None and Pydantic validators are function(value: T) -> T, so they are kinda incompatible.

    The final alternative was implementing an exception handler for Tortoise ValidationErrors.

    opened by jordi-reinsma 0
  • How to achieve SELECT DISTINCT field .....

    How to achieve SELECT DISTINCT field .....

    Model class Transactions(Model): id = fields.CharField(65, pk=True, null=False) confirmation = fields.IntField(pk=False, max_length=1, default=0, null=False) confirmed_hash = fields.CharField(65, null=False)

    Dataset: id - confirmation - confirmed_hash 1 - 0 - aaaaaaaaaaaaaaaaaaaa 2 - 1 - aaaaaaaaaaaaaaaaaaaa 3 - 1 - bbbbbbbbbbbbbbbbbb 4 - 0 - aaaaaaaaaaaaaaaaaaaa 5 - 0 - cccccccccccccccccccc 6 - 0 - cccccccccccccccccccc 7 - 1 - aaaaaaaaaaaaaaaaaaaa

    From this dataset I want to return only the DISTINCT 'confirmed_hash' rows. I would expect to do this like this:

    Transactions.all().distinct('confirmed_hash')

    to produce SQL along the lines of

    "SELECT DISTINCT confirmed_hash FROM Transactions....."

    Which would return:

    [ 'aaaaaaaaaaaaaaaaaaaa', bbbbbbbbbbbbbbbbbb', 'cccccccccccccccccccc' ]

    Can anyone clarify how to achieve this with Tortoise? Thanks

    opened by rstmsn 1
  • Is there a way to pass extra parameters to a signal?

    Is there a way to pass extra parameters to a signal?

    When using signals when updating/creating/deleting a model, is there a way to pass extra parameters to the signals?

    Let's say I have an app state of loaded dataframes that I would to be visible for the signals, how can I achieve this?

    opened by angel-langdon 0
  • Add __like and __ilike filters (#421)

    Add __like and __ilike filters (#421)

    Adds support for __like and __ilike filters (#421)

    Description

    Adds support for filtering by arbitrary LIKE patterns. For example:

    await User.filter(name__like='J_hn%')

    I made three potentially contentious implementation decisions:

    1. The caller needs to escape % and _ with backslashes in the pattern string if they desire to match those characters literally. If they are not escaped, they are treated as SQL wildcards. The caller does not need to escape \ to match it literally. As far as I can see, this is consistent with how __contains works, for example. It also escapes \ on behalf of the caller.
    2. I implemented __ilike using UPPER + LIKE. Therefore, it works with flavors of SQL that do not support the ILIKE operator, such as MySQL.
    3. The LIKE operator, and therefore __like, at least in the current implementation, is case-sensitive or case-insensitive depending on the flavor of SQL that is used. If desired, this might be changed, so that __like is guaranteed to be case-sensitive independently of the underlying database. I did not try to enforce case-sensitivity because LIKE queries on the database and __like filters in Tortoise would yield different results, which might be confusing.

    Motivation and Context

    Fixes #421 (The issue is assigned to someone else but has not been worked on for over a year. I hope it's fine that I opened a PR addressing it.)

    How Has This Been Tested?

    I added tests in test_filters.py. Someone with a good understanding of SQL injections should take a look at the tests and let me know if there are further tests I should add.

    Checklist:

    • [x] My code follows the code style of this project.
    • [x] My change requires a change to the documentation.
    • [x] I have updated the documentation accordingly.
    • [x] I have added the changelog accordingly.
    • [x] I have read the CONTRIBUTING document.
    • [x] I have added tests to cover my changes.
    • [ ] All new and existing tests passed.

    The two tests tests/test_default.py::TestDefault::test_default and tests/fields/test_time.py::TestDatetimeFields::test_update fail when running make test_postgres locally on my machine. However, that is the case on the develop branch as well and seems unrelated to the changes I made.

    This is my first contribution to Tortoise ORM. Please let me know if there are more elegant ways to implement LIKE filters using existing functionality in the code base that I might not be familiar with.

    opened by SimonSimonB 0
  • Allow adding and removing m2m fields using id

    Allow adding and removing m2m fields using id

    The add() and remove() functions on the m2m field currently only accept instances of the related model.

    It would be useful if it also accepted ids of the related model. This would avoid having to query the database to fetch the instances just to add or remove them.

    opened by bhch 2
  • Close to Django alike feature

    Close to Django alike feature

    I want more django alike feature. How it can be possible? Creating object: Event.objects.create(name='Test') instead of Event.create(name='Test')

    Getting object: Event.objects.get(id=1) instead of Event.get(id=1)

    Filtering object: Event.objects.filter(name='Test') instead of Event.filter(name='Test')

    Deleting object: Event.objects.get(id=1).delete() instead of Event.get(id=1).delete()

    opened by nikolas310 3
  • Update base.py

    Update base.py

    Added blank=True/False for required field

    Description

    Motivation and Context

    How Has This Been Tested?

    Checklist:

    • [ ] My code follows the code style of this project.
    • [ ] My change requires a change to the documentation.
    • [ ] I have updated the documentation accordingly.
    • [ ] I have added the changelog accordingly.
    • [ ] I have read the CONTRIBUTING document.
    • [ ] I have added tests to cover my changes.
    • [ ] All new and existing tests passed.
    opened by nikolas310 3
  • Little improvements on queryset.py

    Little improvements on queryset.py

    Description

    • Destruct variable in for loops where it uses [0] and [1] in for loop body. (Provides more readable variables names)
    • Use generator instead of list comprehension in ValuesQuery._execute() since we just filter it to new list. (0.000001% speed boost)
    • Specify arguments in long super().__init__() without kwargs (Helps understand what arguments exactly we use)

    Motivation and Context

    This changes improves readability of code (and uses best practises such as generators over lists)

    How Has This Been Tested?

    This changes doesn't affect on any TORM logic. Was tested on SQLite only

    Checklist:

    • [x] My code follows the code style of this project.
    • [ ] My change requires a change to the documentation.
    • [ ] I have updated the documentation accordingly.
    • [ ] I have added the changelog accordingly. (should it be updated?)
    • [x] I have read the CONTRIBUTING document.
    • [ ] I have added tests to cover my changes.
    • [x] All new and existing tests passed.
    opened by Masynchin 1
  • im nt able to use async functions in PydanticMeta computed array

    im nt able to use async functions in PydanticMeta computed array

    Describe the bug A clear and concise description of what the bug is.

    To Reproduce Steps to reproduce the behavior, preferaby a small code snippet.

    Expected behavior A clear and concise description of what you expected to happen.

    Additional context Add any other context about the problem here.

    opened by sakthiRathinam 1
  • Support real enums

    Support real enums

    Both MySQL and PostgreSQL support real type-safe enums, while TortoiseORM only provides some sugar to access a numerical or text field as if it was enum.

    opened by numberZero 0
Releases(0.17.8)
  • 0.17.8(Oct 6, 2021)

    Added

    • Add Model.raw method to support the raw sql query.
    • Add QuerySet.bulk_update method. (#924)
    • Add QuerySet.in_bulk method.
    • Add MaxValueValidator and MinValueValidator (#927)

    Fixed

    • Fix QuerySet subclass being lost when _clone is run on the instance.
    • Fix bug in .values with source_field. (#844)
    • Fix contrib.blacksheep exception handlers, use builtin json response. (#914)
    • Fix Indexes defined in Meta class do not make use of exists parameter in their template (#928)

    Changed

    • Allow negative values with IntEnumField. (#889)
    • Make .values() and .values_list() awaited return more consistent. (#899)
    Source code(tar.gz)
    Source code(zip)
  • 0.17.7(Aug 31, 2021)

    • Fix select_related behaviour for forward relation. (#825)
    • Fix bug in nested QuerySet and Manager. (#864)
    • Add Concat function for MySQL/PostgreSQL. (#873)
    • Patch for use_index/force_index mutable problem when making query. (#888)
    • Lift annotation field's priority in make query. (#883)
    • Make use/force index available in select type Query. (#893)
    • Fix all logging to use Tortoise's logger instead of root logger. (#879)
    • Rename db_client logger to tortoise.db_client.
    • Add indexes to Model.describe.
    Source code(tar.gz)
    Source code(zip)
  • 0.17.6(Jul 26, 2021)

    • Add RawSQL expression.
    • Fix columns count with annotations in _make_query. (#776)
    • Make functions nested. (#828)
    • Add db_constraint in field describe.
    Source code(tar.gz)
    Source code(zip)
  • 0.17.5(Jul 7, 2021)

    • Set field_type of fk and o2o same to which relation field type. (#443)
    • Fix error sql for .sql() call more than once. (#796)
    • Fix incorrect splitting of the import route when using Router (#798)
    • Fix filter error after annotate with F. (#806)
    • Fix select_related for reverse relation. (#808)
    Source code(tar.gz)
    Source code(zip)
  • 0.17.4(Jun 3, 2021)

  • 0.17.3(May 22, 2021)

    • Fix duplicates when using custom through association class on M2M relations
    • Fix update_or_create and get_or_create. (#721)
    • Fix refresh_from_db without fields pass. (#734)
    • Make update query work with limit and order_by. (#748)
    • Add Subquery expression. (#756) (#9) (#337)
    • Use JSON in JSONField.
    Source code(tar.gz)
    Source code(zip)
  • 0.17.2(Apr 9, 2021)

    • Add more index types.
    • Add force_index, use_index to queryset.
    • Fix F in update error with update_fields.
    • Make delete query work with limit and order_by. (#697)
    • Filter backward FK fields with IS NULL and NOT IS NULL filters (#700)
    • Add select_for_update in update_or_create. (#702)
    • Add Model.select_for_update.
    • Add __search full text search to queryset.
    Source code(tar.gz)
    Source code(zip)
  • 0.17.1(Mar 27, 2021)

    • Fix type for modules.
    • Fix select_related when related model specified more than once. (#679)
    • Add __iter__ to model, now can just return model/models in fastapi response.
    • Fix in_transaction bug caused by router. (#677) (#678)
    Source code(tar.gz)
    Source code(zip)
  • 0.17.0(Mar 20, 2021)

  • 0.16.21(Feb 4, 2021)

    • Fixed validating JSON before decoding. (#623)
    • Add model method update_or_create.
    • Add batch_size parameter for bulk_create method.
    • Fix save with F expression and field with source_field.
    Source code(tar.gz)
    Source code(zip)
  • 0.16.20(Jan 23, 2021)

  • 0.16.19(Dec 23, 2020)

    • Replace set TZ environment variable to TIMEZONE to avoid affecting global timezone.
    • Allow passing module objects to models_paths param of Tortoise.init_models(). (#561)
    • Implement PydanticMeta.backward_relations. (#536)
    • Allow overriding PydanticMeta in PydanticModelCreator. (#536)
    • Fixed make_native typo to make_naive in timezone module
    Source code(tar.gz)
    Source code(zip)
  • 0.16.18(Nov 16, 2020)

    • Support custom function in update. (#537)
    • Add Model.refresh_from_db. (#549)
    • Add timezone support, be careful to upgrade to this version, see docs for details. (#335)
    • Remove aerich in case of cyclic dependency. (#558)
    Source code(tar.gz)
    Source code(zip)
  • 0.16.17(Oct 23, 2020)

    • Add on_delete in ManyToManyField. (#508)
    • Support F expression in annotate. (#475)
    • Fix QuerySet.select_related in case of join same table twice. (#525)
    • Integrate Aerich into the install. (#530)
    Source code(tar.gz)
    Source code(zip)
  • 0.16.16(Sep 24, 2020)

  • 0.16.15(Sep 16, 2020)

    • Make DateField accept valid date str.
    • Add QuerySet.select_for_update().
    • check default for not None on pydantic model creation
    • propagate default to pydantic model
    • Add QuerySet.select_related().
    • Add custom attribute name for Prefetch instruction.
    • Add db_constraint for RelationalField family.
    Source code(tar.gz)
    Source code(zip)
  • 0.16.14(Jul 25, 2020)

    • We now do CI runs on a Windows VM as well, to try and prevent Windows specific regressions.
    • Make F expression work with QuerySet.filter().
    • Include py.typed in source distribution.
    • Added datetime parsing from int for fields.DatetimeField.
    • get_or_create passes the using_db= on if provided.
    • Allow custom loop and connection_class parameters to be passed on to asyncpg.
    Source code(tar.gz)
    Source code(zip)
  • 0.16.13(Jun 2, 2020)

    • Default install of tortoise-orm now installs with no C-dependencies, if you want to use the C accelerators, please do a pip install tortoise-orm[accel] instead.
    • Added <instance>.clone() method that will create a cloned instance in memory. To persist it you still need to call .save()
    • .clone() will raise a ParamsError if tortoise can't generate a primary key. In that case do a .clone(pk=<newval>)
    • If manually setting the primary key value to None and the primary key can be automatically generated, this will create a new record. We however still recommend the .clone() method instead.
    • .save() can be forced to do a create by setting force_create=True
    • .save() can be forced to do an update by setting force_update=True
    • Setting update_fields for a .save() operation will strongly prefer to do an update if possible
    Source code(tar.gz)
    Source code(zip)
  • 0.16.12(May 22, 2020)

    • Make Field.default effect on db level when generate table
    • Add converters instead of importing from pymysql
    • Fix PostgreSQL BooleanField default value convertion
    • Fix JSONField typed in pydantic_model_creator
    • Add .sql() method on QuerySet
    Source code(tar.gz)
    Source code(zip)
  • 0.16.11(May 14, 2020)

    • fix: sqlite://:memory: in Windows thrown OSError: [WinError 123]
    • Support bulk_create() insertion of records with overridden primary key when the primary key is DB-generated
    • Add queryset.exists() and Model.exists().
    • Add model subscription lookup, Model[<pkval>] that will return the object or raise KeyError
    Source code(tar.gz)
    Source code(zip)
  • 0.16.10(Apr 30, 2020)

    • Fix bad import of basestring
    • Better handling of NULL characters in strings. Fixes SQLite, raises better error for PostgreSQL.
    • Support .group_by() with join now
    Source code(tar.gz)
    Source code(zip)
  • 0.16.9(Apr 26, 2020)

    • Support F expression in .save() now
    • IntEnumField accept valid int value and CharEnumField accept valid str value
    • Pydantic models get created with globally unique identifier
    • Leaf-detection to minimize duplicate Pydantic model creation
    • Pydantic models with a Primary Key that is also a raw field of a relation is now not hidden when exclude_raw_fields=True as it is a critically important field
    • Raise an informative error when a field is set as nullable and primary key at the same time
    • Foreign key id's are now described to have the positive-integer range of the field it is related to
    • Fixed prefetching over OneToOne relations
    • Fixed __contains for non-text fields (e.g. JSONB)
    Source code(tar.gz)
    Source code(zip)
  • 0.16.8(Apr 22, 2020)

    • Allow Q expression to function with _filter parameter on aggregations
    • Add manual .group_by() support
    • Fixed regression where GROUP BY class is missing for an aggregate with a specified order.
    Source code(tar.gz)
    Source code(zip)
  • 0.15.24(Apr 22, 2020)

  • 0.16.7(Apr 19, 2020)

    • Added preliminary support for Python 3.9
    • TruncationTestCase now properly quotes table names when it clears them out.
    • Add model signals support
    • Added app_label to test initializer(...) and TORTOISE_TEST_APP as test environment variable.
    Source code(tar.gz)
    Source code(zip)
  • 0.16.6(Apr 18, 2020)

    This is a security fix release. We strongly recommend people upgrade.

    Security fixes:

    • Fixed SQL injection issue in MySQL
    • Fixed SQL injection issues in MySQL when using contains, starts_with or ends_with filters (and their case-insensitive counterparts)
    • Fixed malformed SQL for PostgreSQL and SQLite when using contains, starts_with or ends_with filters (and their case-insensitive counterparts)

    Other changes:

    • Added support for partial models:

      To create a partial model, one can do a .only(<fieldnames-as-strings>) as part of the QuerySet. This will create model instances that only have those values fetched.

      Persisting changes on the model is allowed only when:

      • All the fields you want to update is specified in <model>.save(update_fields=[...])
      • You included the Model primary key in the `.only(...)``

      To protect against common mistakes we ensure that errors get raised:

      • If you access a field that is not specified, you will get an AttributeError.
      • If you do a <model>.save() a IncompleteInstanceError will be raised as the model is, as requested, incomplete.
      • If you do a <model>.save(update_fields=[...]) and you didn't include the primary key in the .only(...), then IncompleteInstanceError will be raised indicating that updates can't be done without the primary key being known.
      • If you do a <model>.save(update_fields=[...]) and one of the fields in update_fields was not in the .only(...), then IncompleteInstanceError as that field is not available to be updated.
    • Fixed bad SQL generation when doing a .values() query over a Foreign Key
    • Added <model>.update_from_dict({...}) that will mass update values safely from a dictionary
    • Fixed processing URL encoded password in connection string
    Source code(tar.gz)
    Source code(zip)
  • 0.15.23(Apr 18, 2020)

    This is a security fix release. We strongly recommend everyone upgrade.

    • Fixed SQL injection issue in MySQL
    • Fixed SQL injection issues in MySQL when using contains, starts_with or ends_with filters (and their case-insensitive counterparts)
    • Fixed malformed SQL for PostgreSQL and SQLite when using contains, starts_with or ends_with filters (and their case-insensitive counterparts)
    Source code(tar.gz)
    Source code(zip)
  • 0.16.5(Apr 10, 2020)

    Bugfixes

    • Fix for generate_schemas param being ignored in tortoise.contrib.quart.register_tortoise
    • Fix join query with source_field param

    Changed

    • Moved Tortoise.describe_model(<MODEL>, ...) to <MODEL>.describe(...)
    • Deprecated Tortoise.describe_model()
    Source code(tar.gz)
    Source code(zip)
  • 0.15.22(Apr 10, 2020)

    • Fix the aggregates using the wrong side of the join when doing a self-referential aggregation.
    • Fix for generate_schemas param being ignored in tortoise.contrib.quart.register_tortoise
    Source code(tar.gz)
    Source code(zip)
  • 0.16.4(Apr 1, 2020)

    • More consistent escaping of db columns, fixes using SQL reserved keywords as field names with a function.
    • Fix the aggregates using the wrong side of the join when doing a self-referential aggregation.
    • Fix F funtions wrapped forgetting about distinct=True
    Source code(tar.gz)
    Source code(zip)
Owner
Tortoise
Familiar asyncio ORM for python, built with relations in mind
Tortoise
Piccolo - A fast, user friendly ORM and query builder which supports asyncio.

A fast, user friendly ORM and query builder which supports asyncio.

null 566 Oct 20, 2021
The ormar package is an async mini ORM for Python, with support for Postgres, MySQL, and SQLite.

python async mini orm with fastapi in mind and pydantic validation

null 674 Oct 23, 2021
A curated list of awesome tools for SQLAlchemy

Awesome SQLAlchemy A curated list of awesome extra libraries and resources for SQLAlchemy. Inspired by awesome-python. (See also other awesome lists!)

Hong Minhee (洪 民憙) 2.3k Oct 21, 2021
a small, expressive orm -- supports postgresql, mysql and sqlite

peewee Peewee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use. a small, expressive ORM p

Charles Leifer 8.7k Oct 25, 2021
The Orator ORM provides a simple yet beautiful ActiveRecord implementation.

Orator The Orator ORM provides a simple yet beautiful ActiveRecord implementation. It is inspired by the database part of the Laravel framework, but l

Sébastien Eustace 1.3k Oct 17, 2021
Pony Object Relational Mapper

Downloads Pony Object-Relational Mapper Pony is an advanced object-relational mapper. The most interesting feature of Pony is its ability to write que

null 2.7k Oct 25, 2021
A very simple CRUD class for SQLModel! ✨

Base SQLModel A very simple CRUD class for SQLModel! ✨ Inspired on: Full Stack FastAPI and PostgreSQL - Base Project Generator FastAPI Microservices I

Marcelo Trylesinski 12 Oct 22, 2021
Pydantic model support for Django ORM

Pydantic model support for Django ORM

Jordan Eremieff 189 Oct 23, 2021
A Python Object-Document-Mapper for working with MongoDB

MongoEngine Info: MongoEngine is an ORM-like layer on top of PyMongo. Repository: https://github.com/MongoEngine/mongoengine Author: Harry Marr (http:

MongoEngine 3.6k Oct 24, 2021
A Python Library for Simple Models and Containers Persisted in Redis

Redisco Python Containers and Simple Models for Redis Description Redisco allows you to store objects in Redis. It is inspired by the Ruby library Ohm

sebastien requiem 434 Oct 1, 2021
Beanie - is an Asynchronous Python object-document mapper (ODM) for MongoDB

Beanie - is an Asynchronous Python object-document mapper (ODM) for MongoDB, based on Motor and Pydantic.

Roman 277 Oct 18, 2021
A pythonic interface to Amazon's DynamoDB

PynamoDB A Pythonic interface for Amazon's DynamoDB. DynamoDB is a great NoSQL service provided by Amazon, but the API is verbose. PynamoDB presents y

null 1.7k Oct 23, 2021
SQLModel is a library for interacting with SQL databases from Python code, with Python objects.

SQLModel is a library for interacting with SQL databases from Python code, with Python objects. It is designed to be intuitive, easy to use, highly compatible, and robust.

Sebastián Ramírez 5.5k Oct 24, 2021
Easy-to-use data handling for SQL data stores with support for implicit table creation, bulk loading, and transactions.

dataset: databases for lazy people In short, dataset makes reading and writing data in databases as simple as reading and writing JSON files. Read the

Friedrich Lindenberg 4.1k Oct 18, 2021
MongoEngine flask extension with WTF model forms support

Flask-MongoEngine Info: MongoEngine for Flask web applications. Repository: https://github.com/MongoEngine/flask-mongoengine About Flask-MongoEngine i

MongoEngine 788 Oct 14, 2021
Rich Python data types for Redis

Created by Stephen McDonald Introduction HOT Redis is a wrapper library for the redis-py client. Rather than calling the Redis commands directly from

Stephen McDonald 273 Sep 2, 2021