The Python SQL Toolkit and Object Relational Mapper

Related tags

ORM sqlalchemy
Overview

SQLAlchemy

PyPI PyPI - Python Version PyPI - Downloads

The Python SQL Toolkit and Object Relational Mapper

Introduction

SQLAlchemy is the Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL. SQLAlchemy provides a full suite of well known enterprise-level persistence patterns, designed for efficient and high-performing database access, adapted into a simple and Pythonic domain language.

Major SQLAlchemy features include:

  • An industrial strength ORM, built from the core on the identity map, unit of work, and data mapper patterns. These patterns allow transparent persistence of objects using a declarative configuration system. Domain models can be constructed and manipulated naturally, and changes are synchronized with the current transaction automatically.
  • A relationally-oriented query system, exposing the full range of SQL's capabilities explicitly, including joins, subqueries, correlation, and most everything else, in terms of the object model. Writing queries with the ORM uses the same techniques of relational composition you use when writing SQL. While you can drop into literal SQL at any time, it's virtually never needed.
  • A comprehensive and flexible system of eager loading for related collections and objects. Collections are cached within a session, and can be loaded on individual access, all at once using joins, or by query per collection across the full result set.
  • A Core SQL construction system and DBAPI interaction layer. The SQLAlchemy Core is separate from the ORM and is a full database abstraction layer in its own right, and includes an extensible Python-based SQL expression language, schema metadata, connection pooling, type coercion, and custom types.
  • All primary and foreign key constraints are assumed to be composite and natural. Surrogate integer primary keys are of course still the norm, but SQLAlchemy never assumes or hardcodes to this model.
  • Database introspection and generation. Database schemas can be "reflected" in one step into Python structures representing database metadata; those same structures can then generate CREATE statements right back out - all within the Core, independent of the ORM.

SQLAlchemy's philosophy:

  • SQL databases behave less and less like object collections the more size and performance start to matter; object collections behave less and less like tables and rows the more abstraction starts to matter. SQLAlchemy aims to accommodate both of these principles.
  • An ORM doesn't need to hide the "R". A relational database provides rich, set-based functionality that should be fully exposed. SQLAlchemy's ORM provides an open-ended set of patterns that allow a developer to construct a custom mediation layer between a domain model and a relational schema, turning the so-called "object relational impedance" issue into a distant memory.
  • The developer, in all cases, makes all decisions regarding the design, structure, and naming conventions of both the object model as well as the relational schema. SQLAlchemy only provides the means to automate the execution of these decisions.
  • With SQLAlchemy, there's no such thing as "the ORM generated a bad query" - you retain full control over the structure of queries, including how joins are organized, how subqueries and correlation is used, what columns are requested. Everything SQLAlchemy does is ultimately the result of a developer- initiated decision.
  • Don't use an ORM if the problem doesn't need one. SQLAlchemy consists of a Core and separate ORM component. The Core offers a full SQL expression language that allows Pythonic construction of SQL constructs that render directly to SQL strings for a target database, returning result sets that are essentially enhanced DBAPI cursors.
  • Transactions should be the norm. With SQLAlchemy's ORM, nothing goes to permanent storage until commit() is called. SQLAlchemy encourages applications to create a consistent means of delineating the start and end of a series of operations.
  • Never render a literal value in a SQL statement. Bound parameters are used to the greatest degree possible, allowing query optimizers to cache query plans effectively and making SQL injection attacks a non-issue.

Documentation

Latest documentation is at:

https://www.sqlalchemy.org/docs/

Installation / Requirements

Full documentation for installation is at Installation.

Getting Help / Development / Bug reporting

Please refer to the SQLAlchemy Community Guide.

Code of Conduct

Above all, SQLAlchemy places great emphasis on polite, thoughtful, and constructive communication between users and developers. Please see our current Code of Conduct at Code of Conduct.

License

SQLAlchemy is distributed under the MIT license.

Comments
  • Add 'FOR NO KEY UPDATE' support

    Add 'FOR NO KEY UPDATE' support

    My PR add support for using Postgresql lockmode 'FOR NO KEY UPDATE', which gives good possibility for no locks/dead locks, where we have many tables with foreign keys and lock rows in tables, which is master of foreign keys, and then trying to insert data into linked tables.

    opened by grizlik 26
  • Feature/postgres relkind

    Feature/postgres relkind

    First run at fixing https://bitbucket.org/zzzeek/sqlalchemy/issue/2891/support-materialized-views-in-postgresql#comment-11769903

    • Tables now take the postresql_relkind parameter in their constructor. This reflection option gets passed to get_table_oid() to filter its search for table oids. Can be full names (e.g. "materialized", "foreign", "view") or short-hand ("m", "f", "v", "r").
    • get_table_oid() also always searches through tables with 'v' for backwards compatibility... Any better ideas on how to handle this?
    • Added some basic tests for 'm' and 'f' relkinds. Added the postgres_test_db_link configuration for testing.

    Another question: All calls to get_table_oid() need to come with table.dialect_kwargs to pass postgresql_relkind. Unfortunately, I couldn't find a way to reference table.dialect_kwargs from the get_unique_constraints method of the Inspector class. Any ideas on how to solve this?

    https://github.com/rclmenezes/sqlalchemy/blob/feature/postgres-relkind/lib/sqlalchemy/engine/reflection.py#L385

    opened by rclmenezes 25
  • Oracle stores INTEGER as NULL precision with scale == 0

    Oracle stores INTEGER as NULL precision with scale == 0

    SqlAlchemy should render NUMBER of NULL precision and scale == 0 as INTEGER instead of NUMBER (Otherwise roundtrip reflection re-creation changes original datatype.)

    See RoundTripNumericTypeTest for failure case.

    opened by kbower 21
  • Use a callable to determine persisted enum values

    Use a callable to determine persisted enum values

    This intends to resolve https://bitbucket.org/zzzeek/sqlalchemy/issues/3906

    The idea is to use values_callable as a new kwarg to Enum. It returns the values to be used, allowing for a variety of uses, including the one mentioned in the above issue.

    Tests have been added as well as updates to the documentation. This is my first PR to sqlalchemy so apologies in advance for things that are out of place. Particularly I wonder if this needs to be supported in postgres as the work and tests I did were for mysql.

    opened by razor-1 21
  • rst and docstrings code snippets syntax

    rst and docstrings code snippets syntax

    Hello, i'am return with new shiny tool flake8-rst. As you proposed in old PR. Tool inspired with blacked-docs.

    Idea is simple, crop with regex code snippets and put it in flake8.

    You can try it youself, just add this to tox.ini:

    [flake8-rst]
    filename = *.rst *.py
    show-source = True
    ignore = E711,E712,E721,N806,D,F811,F821,F401
    exclude=.venv,.git,.tox,dist,doc,*egg,build
    bootstrap = from sqlalchemy import Table, Column, Sequence, DateTime, Integer, String, Index, Base, ForeignKey, create_engine
                from sqlalchemy import event, trigger, listen
                from sqlalchemy import select, func
                from sqlalchemy.ext.declarative import declared_attr
                from sqlalchemy.orm.util import polymorphic_union
                from sqlalchemy.orm import relationship
                from fake_import import engine, my_engine, meta, mytable
                from fake_core import employees, users, addresses, conn
    
    $ pip install flake8-rst
    $ flake8-rst lib doc
    

    This first part of changes:

    1. add sourcecode directive;
    2. fix syntax error.

    After your approving of this approach, i am can fix other pep8 issues.

    opened by kataev 20
  • Add AppVeyor build with autoupload when commit is tagged with 'rel_*'

    Add AppVeyor build with autoupload when commit is tagged with 'rel_*'

    As discussed in #3747 (BitBucket)

    1. Create an AppVeyor account and enable the SQLAlchemy repository (presumably the bitbucket one).
    2. Encrypt PyPI credentials on AppVeyor by clicking your name at the top right and clicking "Encrypt data" under "Tools".

    image 3. Put those values in appveyor.yml.

    I've included the appveyor/getwheels.py script, which may be useful if you want to download the latest wheels for a build. I can remove it if you don't want it in the repo.

    opened by RazerM 20
  • Proposing LIFO for connection pooling

    Proposing LIFO for connection pooling

    Why

    • SQLAlchemy's connection pool is using the Queue data structure to provide the connection pools to threads round-robin basis. Which leads to maintain all connections even though the server's load is not that high enough to maintain all connections.
    • Using the Stack data structure could solve this problem as connections that are not working will be disconnected from the database.

    Changes

    • Added StackPool in lib/sqlalchemy/pool/impl.py.
    • Added Stack in lib/sqlalchemy/util/queue.py.
    opened by TaeMinPark 18
  • Update URL for pymssql

    Update URL for pymssql

    At some point it would also be nice to review the limitations that are listed in this file (mostly FreeTDS limitations I think as you point out) and delete the ones that don't apply anymore with reasonably recent versions of FreeTDS and pymssql.

    Cc: @rsyring, @ramiro, @damoxc, @sontek, @aconrad

    opened by msabramo 18
  • Raise error on empty in_

    Raise error on empty in_

    I think it would be good idea to add error on empy in_ usage in version 1.1.* Because current implementation is not fully correct: Firstly, it's not optimised, because of sequential scan Secondly, in current implementation null in () will return null, when in sql it should return False, try this: select null in (select 1 from some_table where false); -> false but not null

    I understant that this can break some legacy code, but current implementation is obscure and quite expensive

    opened by erebus1 17
  • Added new dialect.paramstyle option to support PostgreSQL positional arguments

    Added new dialect.paramstyle option to support PostgreSQL positional arguments

    This is specifically needed to properly integrate SQLAlchemy with asyncpg library. This library uses positional arguments and PostgreSQL requires $1-like syntax to accept such positional params.

    Maybe you (@zzzeek) know how to make this possible in a simpler way? Current state looks terrible: asyncpgsa/connection.py - people are using weird regexp/replace stuff just to change paramstyle :(

    Here I tried to make things better, but it still is not perfect. Using even one re.sub doesn't feels right.

    And one more question. Currently it is not possible to compile final query statement with final params, which can be sent to the driver/db. Libraries like aiopg and asyncpgsa have to partially copy DefaultExecutionContext._init_compiled and other methods, accessing non-public _bind_processors property and so on. So is it possible to made such low-level usage possible and public to help integrate SQAlchemy with other libraries? You can see how this looks right now: aiopg/sa/connection.py :(

    opened by vmagamedov 16
  • proposal for using service_name in cx_Oracle.makedsn

    proposal for using service_name in cx_Oracle.makedsn

    The problem:

    Currently, in Oracle dialect, there is no way to tell sqlalchemy to connect to a database via service_name (a very common situation in Oracle RAC instances). The exisiting solutions are:

    • use a full DSN in the host part of the connection string (very verbose)
    • configure tnsnames.ora on server side and use that configured TNS name in your connection string (not very transferable)

    Proposal

    This PR introduces a new bool argument use_service_name (currently only via query string arg). It would control how to create a DSN from the cx_Oracle.makedsn function i.e. it would treat database part of the connection string as a service_name rather than sid.

    Further development

    I would be more than happy to also update OracleDialect_cx_oracle.__init__ method, write a test case and update documentation after this proposal gets an initial acceptance.

    opened by slafs 15
  • Bump pypa/cibuildwheel from 2.11.2 to 2.11.4

    Bump pypa/cibuildwheel from 2.11.2 to 2.11.4

    Bumps pypa/cibuildwheel from 2.11.2 to 2.11.4.

    Release notes

    Sourced from pypa/cibuildwheel's releases.

    v2.11.4

    • 🐛 Fix a bug that caused missing wheels on Windows when a test was skipped using CIBW_TEST_SKIP (#1377)
    • 🛠 Updates CPython 3.11 to 3.11.1 (#1371)
    • 🛠 Updates PyPy 3.7 to 3.7.10, except on macOS which remains on 7.3.9 due to a bug. (#1371)
    • 📚 Added a reference to abi3audit to the docs (#1347)

    v2.11.3

    • ✨ Improves the 'build options' log output that's printed at the start of each run (#1352)
    • ✨ Added a friendly error message to a common misconfiguration of the CIBW_TEST_COMMAND option - not specifying path using the {project} placeholder (#1336)
    • 🛠 The GitHub Action now uses Powershell on Windows to avoid occasional incompabilities with bash (#1346)
    Changelog

    Sourced from pypa/cibuildwheel's changelog.

    v2.11.4

    24 Dec 2022

    • 🐛 Fix a bug that caused missing wheels on Windows when a test was skipped using CIBW_TEST_SKIP (#1377)
    • 🛠 Updates CPython 3.11 to 3.11.1 (#1371)
    • 🛠 Updates PyPy to 7.3.10, except on macOS which remains on 7.3.9 due to a bug on that platform. (#1371)
    • 📚 Added a reference to abi3audit to the docs (#1347)

    v2.11.3

    5 Dec 2022

    • ✨ Improves the 'build options' log output that's printed at the start of each run (#1352)
    • ✨ Added a friendly error message to a common misconfiguration of the CIBW_TEST_COMMAND option - not specifying path using the {project} placeholder (#1336)
    • 🛠 The GitHub Action now uses Powershell on Windows to avoid occasional incompabilities with bash (#1346)
    Commits
    • 27fc88e Bump version: v2.11.4
    • a7e9ece Merge pull request #1371 from pypa/update-dependencies-pr
    • b9a3ed8 Update cibuildwheel/resources/build-platforms.toml
    • 3dcc2ff fix: not skipping the tests stops the copy (Windows ARM) (#1377)
    • 1c9ec76 Merge pull request #1378 from pypa/henryiii-patch-3
    • 22b433d Merge pull request #1379 from pypa/pre-commit-ci-update-config
    • 98fdf8c [pre-commit.ci] pre-commit autoupdate
    • cefc5a5 Update dependencies
    • e53253d ci: move to ubuntu 20
    • e9ecc65 [pre-commit.ci] pre-commit autoupdate (#1374)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
Owner
mike bayer
SQLAlchemy, Alembic, Mako, dogpile.cache
mike bayer
Beanie - is an Asynchronous Python object-document mapper (ODM) for MongoDB

Beanie - is an Asynchronous Python object-document mapper (ODM) for MongoDB, based on Motor and Pydantic.

Roman 993 Jan 3, 2023
SQLModel is a library for interacting with SQL databases from Python code, with Python objects.

SQLModel is a library for interacting with SQL databases from Python code, with Python objects. It is designed to be intuitive, easy to use, highly compatible, and robust.

Sebastián Ramírez 9.1k Dec 31, 2022
Easy-to-use data handling for SQL data stores with support for implicit table creation, bulk loading, and transactions.

dataset: databases for lazy people In short, dataset makes reading and writing data in databases as simple as reading and writing JSON files. Read the

Friedrich Lindenberg 4.2k Dec 26, 2022
A Python Library for Simple Models and Containers Persisted in Redis

Redisco Python Containers and Simple Models for Redis Description Redisco allows you to store objects in Redis. It is inspired by the Ruby library Ohm

sebastien requiem 436 Nov 10, 2022
The ormar package is an async mini ORM for Python, with support for Postgres, MySQL, and SQLite.

python async mini orm with fastapi in mind and pydantic validation

null 1.2k Jan 5, 2023
Python 3.6+ Asyncio PostgreSQL query builder and model

windyquery - A non-blocking Python PostgreSQL query builder Windyquery is a non-blocking PostgreSQL query builder with Asyncio. Installation $ pip ins

null 67 Sep 1, 2022
A pure Python Database Abstraction Layer

pyDAL pyDAL is a pure Python Database Abstraction Layer. It dynamically generates the SQL/noSQL in realtime using the specified dialect for the databa

null 440 Nov 13, 2022
Rich Python data types for Redis

Created by Stephen McDonald Introduction HOT Redis is a wrapper library for the redis-py client. Rather than calling the Redis commands directly from

Stephen McDonald 281 Nov 10, 2022
Solrorm : A sort-of solr ORM for python

solrorm : A sort-of solr ORM for python solrpy - deprecated solrorm - currently in dev Usage Cores The first step to interact with solr using solrorm

Aj 1 Nov 21, 2021
A PostgreSQL or SQLite orm for Python

Prom An opinionated lightweight orm for PostgreSQL or SQLite. Prom has been used in both single threaded and multi-threaded environments, including en

Jay Marcyes 18 Dec 1, 2022
ORM for Python for PostgreSQL.

New generation (or genius) ORM for Python for PostgreSQL. Fully-typed for any query with Pydantic and auto-model generation, compatible with any sync or async driver

Yan Kurbatov 3 Apr 13, 2022
A new ORM for Python specially for PostgreSQL

A new ORM for Python specially for PostgreSQL. Fully-typed for any query with Pydantic and auto-model generation, compatible with any sync or async driver

Yan Kurbatov 3 Apr 13, 2022
a small, expressive orm -- supports postgresql, mysql and sqlite

peewee Peewee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use. a small, expressive ORM p

Charles Leifer 9.7k Jan 8, 2023
Piccolo - A fast, user friendly ORM and query builder which supports asyncio.

A fast, user friendly ORM and query builder which supports asyncio.

null 919 Jan 4, 2023
Pony Object Relational Mapper

Downloads Pony Object-Relational Mapper Pony is an advanced object-relational mapper. The most interesting feature of Pony is its ability to write que

null 3.1k Jan 4, 2023
PyPika is a python SQL query builder that exposes the full richness of the SQL language using a syntax that reflects the resulting query. PyPika excels at all sorts of SQL queries but is especially useful for data analysis.

PyPika - Python Query Builder Abstract What is PyPika? PyPika is a Python API for building SQL queries. The motivation behind PyPika is to provide a s

KAYAK 1.9k Jan 4, 2023
Simplest SQL mapper in Python, probably

SQL MAPPER Basically what it does is: it executes some SQL thru a database connector you fed it, maps it to some model and gives to u. Also it can cre

null 2 Nov 7, 2022
Pytorch implementation of "A simple neural network module for relational reasoning" (Relational Networks)

Pytorch implementation of Relational Networks - A simple neural network module for relational reasoning Implemented & tested on Sort-of-CLEVR task. So

Kim Heecheol 800 Dec 5, 2022
PubMed Mapper: A Python library that map PubMed XML to Python object

pubmed-mapper: A Python Library that map PubMed XML to Python object 中文文档 1. Philosophy view UML Programmatically access PubMed article is a common ta

灵魂工具人 33 Dec 8, 2022
A Python Object-Document-Mapper for working with MongoDB

MongoEngine Info: MongoEngine is an ORM-like layer on top of PyMongo. Repository: https://github.com/MongoEngine/mongoengine Author: Harry Marr (http:

MongoEngine 3.9k Dec 30, 2022