A Django backed for PostgreSQL using Psycopg 3

Related tags

Django django-psycopg3-backend
Overview

A Django backend for PostgreSQL using Psycopg > 2

The backend passes the entire Django test suite, but it needs a few modifications to Django and to its test suite. These changes will be proposed to the Django project in a series of merge requests.

The modifications required iare available in the psycopg3-support Django branch. This is the list of changes.

Approximative instructions

Create and activate a virtualenv any way you like:

python3 -m venv .venv
source .venv/bin/activate

Install django, from a branch supporting Psycopg 3. Clone the repos to get the test suite too. Example:

git clone -b psycopg3-support https://github.com/dvarrazzo/django.git
pip install ./django

Install Psycopg 3 from the master branch:

git clone https://github.com/psycopg/psycopg.git
pip install -e ./psycopg/psycopg

Install the backend (this project):

pip install -e .

Customise the test config module if necessary:

vim configs/test_psycopg3.py
# hack hack
# :wq

Run the django test suite:

python django/tests/runtests.py --settings=configs.test_psycopg3 -v2 --parallel=1 --noinput

A configs.test_psycopg2 module is also available to run the same tests with psycopg2 and check for regressions.

Comments
  • Landing in Django 4.1?

    Landing in Django 4.1?

    Thank you for psycopg3!!

    I'm looking forward to using it in Django; the feature freeze for Django 4.1 is May 17. Were you planning to get ticket 33308 completed / closed by then?

    opened by thinkwelltwd 5
  • Fix Django ticket #33279 (PR #15078)

    Fix Django ticket #33279 (PR #15078)

    This fixes two Django test suite errors:

    ======================================================================
ERROR: test_extract_func_with_timezone_minus_no_offset (db_functions.datetime.test_extract_trunc.DateFunctionWithTimeZoneTests) [backports.zoneinfo.ZoneInfo(key='Asia/Ust-Nera')]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/home/runner/.local/lib/python3.8/site-packages/psycopg/cursor.py", line 568, in execute
    raise ex.with_traceback(None)
psycopg.errors.InvalidParameterValue: time zone "Asia/Ust+Nera" not recognized

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/tests/db_functions/datetime/test_extract_trunc.py", line 1226, in test_extract_func_with_timezone_minus_no_offset
    utc_model = qs.get()
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/models/query.py", line 437, in get
    num = len(clone)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/models/query.py", line 1358, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/models/sql/compiler.py", line 1234, in execute_sql
    cursor.execute(sql, params)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/home/runner/.local/lib/python3.8/site-packages/psycopg/cursor.py", line 568, in execute
    raise ex.with_traceback(None)
django.db.utils.DataError: time zone "Asia/Ust+Nera" not recognized

======================================================================
ERROR: test_extract_func_with_timezone_minus_no_offset (db_functions.datetime.test_extract_trunc.DateFunctionWithTimeZoneTests) [<DstTzInfo 'Asia/Ust-Nera' LMT+9:33:00 STD>]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/home/runner/.local/lib/python3.8/site-packages/psycopg/cursor.py", line 568, in execute
    raise ex.with_traceback(None)
psycopg.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/tests/db_functions/datetime/test_extract_trunc.py", line 1226, in test_extract_func_with_timezone_minus_no_offset
    utc_model = qs.get()
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/models/query.py", line 437, in get
    num = len(clone)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/models/query.py", line 1358, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/models/sql/compiler.py", line 1234, in execute_sql
    cursor.execute(sql, params)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/home/runner/.local/lib/python3.8/site-packages/psycopg/cursor.py", line 568, in execute
    raise ex.with_traceback(None)
django.db.utils.InternalError: current transaction is aborted, commands ignored until end of transaction block
    opened by ramiro 3
  • Implement `operations.on_conflict_suffix_sql()`

    Implement `operations.on_conflict_suffix_sql()`

    See https://github.com/django/django/commit/0f6946495a8ec955b471ca1baaf408ceb53d4796

    This should fix 12 occurrencs of error:

    ERROR: test_add (many_to_many.tests.ManyToManyTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "django_repo/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/home/runner/.local/lib/python3.8/site-packages/psycopg/cursor.py", line 555, in execute
    raise ex.with_traceback(None)
psycopg.errors.UniqueViolation: duplicate key value violates unique constraint "many_to_many_article_pub_article_id_publication_i_caad31f7_uniq"
DETAIL:  Key (article_id, publication_id)=(6, 3) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "django_repo/tests/many_to_many/tests.py", line 53, in test_add
    a6.publications.add(self.p3)
  File "django_repo/django/db/models/fields/related_descriptors.py", line 1069, in add
    self._add_items(
  File "django_repo/django/db/models/fields/related_descriptors.py", line 1290, in _add_items
    self.through._default_manager.using(db).bulk_create(
  File "django_repo/django/db/models/query.py", line 682, in bulk_create
    returned_columns = self._batched_insert(
  File "django_repo/django/db/models/query.py", line 1586, in _batched_insert
    self._insert(
  File "django_repo/django/db/models/query.py", line 1552, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "django_repo/django/db/models/sql/compiler.py", line 1667, in execute_sql
    cursor.execute(sql, params)
  File "django_repo/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "django_repo/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "django_repo/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "django_repo/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "django_repo/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/home/runner/.local/lib/python3.8/site-packages/psycopg/cursor.py", line 555, in execute
    raise ex.with_traceback(None)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "many_to_many_article_pub_article_id_publication_i_caad31f7_uniq"
DETAIL:  Key (article_id, publication_id)=(6, 3) already exists.
    opened by ramiro 1
  • Fix Django ticket #33160 (PR #14918)

    Fix Django ticket #33160 (PR #14918)

    This fixes one Django test suite failure:

    ======================================================================
ERROR: test_nodb_cursor_reraise_exceptions (backends.postgresql.tests.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/.local/lib/python3.8/site-packages/django_psycopg3/base.py", line 335, in _nodb_cursor
    yield cursor
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/tests/backends/postgresql/tests.py", line 111, in test_nodb_cursor_reraise_exceptions
    raise DatabaseError('exception')
django.db.utils.DatabaseError: exception

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/tests/backends/postgresql/tests.py", line 111, in test_nodb_cursor_reraise_exceptions
    raise DatabaseError('exception')
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/runner/.local/lib/python3.8/site-packages/django_psycopg3/base.py", line 337, in _nodb_cursor
    warnings.warn(
RuntimeWarning: Normally Django will use a connection to the 'postgres' database to avoid running initialization queries against the production database when it's not needed (for example, when running tests). Django was unable to create a connection to the 'postgres' database and will use the first PostgreSQL database instead.
    opened by ramiro 1
  • Implement Django commit django/django@9ac3ef59

    Implement Django commit django/django@9ac3ef59

    https://github.com/django/django/commit/9ac3ef59

    Also, update minimum Postgres version supported to 12 in sync with Django main branch / 4.2 release cycle changes.

    Fixes error

    ======================================================================
ERROR: test_get_database_version (backends.postgresql.tests.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/tests/backends/postgresql/tests.py", line 346, in test_get_database_version
    self.assertEqual(new_connection.get_database_version(), (11, 9))
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/db/backends/base/base.py", line 191, in get_database_version
    raise NotImplementedError(
NotImplementedError: subclasses of BaseDatabaseWrapper may require a get_database_version() method.

    and failure

    ======================================================================
FAIL: test_check_database_version_supported (backends.postgresql.tests.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/mock.py", line 1325, in patched
    return func(*newargs, **newkeywargs)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/tests/backends/postgresql/tests.py", line 352, in test_check_database_version_supported
    connection.check_database_version_supported()
  File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/home/runner/work/django-psycopg3-backend/django-psycopg3-backend/django_repo/django/test/testcases.py", line 799, in _assert_raises_or_warns_cm
    yield cm
AssertionError: NotSupportedError not raised

    when running Django test suite.

    opened by ramiro 0
  • FIX exception handling

    FIX exception handling

    Works with /dvarrazzo/django/tree/psycopg3-4.1 well

    Test improvement From: FAILED (failures=1, errors=14, skipped=763, expected failures=7) To: FAILED (errors=15, skipped=695, expected failures=7)

    opened by tomasix 0
Owner
Daniele Varrazzo
Free software hacker, hacking on PostgreSQL and Python, maintainer of psycopg and other projects.
Daniele Varrazzo
GitHub
A starter template for building a backend with Django and django-rest-framework using docker with PostgreSQL as the primary DB.

Django-Rest-Template! This is a basic starter template for a backend project with Django as the server and PostgreSQL as the database. About the templ

Akshat Sharma 11 Dec 6, 2022
A ToDO Rest API using Django, PostgreSQL and Docker

This Rest API uses PostgreSQL, Docker and Django to implements a ToDo application.

Brenno Lima dos Santos 2 Jan 5, 2022
Fast / fuzzy PostgreSQL counts for Django

Created by Stephen McDonald Introduction Up until PostgreSQL 9.2, COUNT queries generally required scanning every row in a database table. With millio

stephenmcd 85 Oct 25, 2021
PostgreSQL with Docker + Portainer + pgAdmin + Django local

django-postgresql-docker Running PostgreSQL with Docker + Portainer + pgAdmin + Django local for development. This project was done with: Python 3.9.8

Regis Santos 4 Jun 12, 2022
Simpliest django(uvicorn)+postgresql+nginx docker-compose (ready for production and dev)

simpliest django(uvicorn)+postgresql+nginx docker-compose (ready for production and dev) To run in production: docker-compose up -d Site available on

Artyom Lisovskii 1 Dec 16, 2021
Django URL Shortener is a Django app to to include URL Shortening feature in your Django Project

Django URL Shortener Django URL Shortener is a Django app to to include URL Shortening feature in your Django Project Install this package to your Dja

Rishav Sinha 4 Nov 18, 2021
A beginner django project and also my first Django project which involves shortening of a longer URL into a short one using a unique id.

Django-URL-Shortener A beginner django project and also my first Django project which involves shortening of a longer URL into a short one using a uni

Rohini Rao 3 Aug 8, 2021
DRF_commands is a Django package that helps you to create django rest framework endpoints faster using manage.py.

DRF_commands is a Django package that helps you to create django rest framework endpoints faster using manage.py.

Mokrani Yacine 2 Sep 28, 2022
Django-discord-bot - Framework for creating Discord bots using Django

django-discord-bot Framework for creating Discord bots using Django Uses ASGI fo

Jamie Bliss 1 Mar 4, 2022
Django-Text-to-HTML-converter - The simple Text to HTML Converter using Django framework

Django-Text-to-HTML-converter This is the simple Text to HTML Converter using Dj

Nikit Singh Kanyal 6 Oct 9, 2022
Meta package to combine turbo-django and stimulus-django

Hotwire + Django This repository aims to help you integrate Hotwire with Django ?? Inspiration might be taken from @hotwired/hotwire-rails. We are sti

Hotwire for Django 31 Aug 9, 2022
django-reversion is an extension to the Django web framework that provides version control for model instances.

django-reversion django-reversion is an extension to the Django web framework that provides version control for model instances. Requirements Python 3

Dave Hall 2.8k Jan 2, 2023
Django-environ allows you to utilize 12factor inspired environment variables to configure your Django application.

Django-environ django-environ allows you to use Twelve-factor methodology to configure your Django application with environment variables. import envi

Daniele Faraglia 2.7k Jan 7, 2023
Rosetta is a Django application that eases the translation process of your Django projects

Rosetta Rosetta is a Django application that facilitates the translation process of your Django projects. Because it doesn't export any models, Rosett

Marco Bonetti 909 Dec 26, 2022
Cookiecutter Django is a framework for jumpstarting production-ready Django projects quickly.

Cookiecutter Django Powered by Cookiecutter, Cookiecutter Django is a framework for jumpstarting production-ready Django projects quickly. Documentati

Daniel Feldroy 10k Dec 31, 2022
Django project starter on steroids: quickly create a Django app AND generate source code for data models + REST/GraphQL APIs (the generated code is auto-linted and has 100% test coverage).

Create Django App ?? We're a Django project starter on steroids! One-line command to create a Django app with all the dependencies auto-installed AND

imagine.ai 68 Oct 19, 2022
django-quill-editor makes Quill.js easy to use on Django Forms and admin sites

django-quill-editor django-quill-editor makes Quill.js easy to use on Django Forms and admin sites No configuration required for static files! The ent

lhy 139 Dec 5, 2022
A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, celery and redis.

Django Channels Websocket Chatbot A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, c

Yunbo Shi 8 Oct 28, 2022
A handy tool for generating Django-based backend projects without coding. On the other hand, it is a code generator of the Django framework.

Django Sage Painless The django-sage-painless is a valuable package based on Django Web Framework & Django Rest Framework for high-level and rapid web

sageteam 51 Sep 15, 2022