Django query profiler - one profiler to rule them all. Shows queries, detects N+1 and gives recommendations on how to resolve them

Overview

Django Query Profiler

https://travis-ci.com/django-query-profiler/django-query-profiler.svg?branch=master https://codecov.io/gh/django-query-profiler/django-query-profiler/branch/master/graph/badge.svg?token=1Cv7WsOi2W https://readthedocs.org/projects/django-query-profiler/badge/?version=latest https://img.shields.io/pypi/djversions/django-query-profiler

This is a query profiler for Django applications, for helping developers answer the question "My Django code/page/API is slow, How do I find out why?"

Below are some of the features of the profiler:

  1. Shows code paths making N+1 sql calls: Shows the sql with stack_trace which is making N+1 calls, along with sql count
  2. Shows the proposed solution: If the solution to reduce sql is to simply apply a select_related or a prefetch_related, this is highlighted as a suggestion
  3. Shows exact sql duplicates: Count of the queries where (sql, parameters) is exactly the same. This is the kind of sql where implementing a query cache would help
  4. Flame Graph visualisation: Collects all the stack traces together to allow quickly identifying which area(s) of code is causing the load to the database
  5. Command line or chrome plugin: The profiler can be called from command line via context manager, or can be invoked via a middleware, and output shown in a chrome plugin
  6. Super easy to configure in any application: The only changes are in settings.py file and in urls.py file

This is the repo for the chrome plugin

Requirements

This works with any version of django >= 2.0, and running on python >= 3.6

Profiler in Action

as a chrome plugin

This image shows how the chrome plugin would display profiled data, once it is configured & installed

https://raw.githubusercontent.com/django-query-profiler/django-query-profiler/master/docs/_static/django_query_profiler_in_action.gif

on command line

See this file in the PR to see how to use the context manager, and how easy it is to find performance issues :-)

The output of Django query profiler is same for the command line or the chrome plugin. In fact, chrome plugin displays the output set by the middleware - which is just a plain wrapper around context manager.

Getting Started

installation

The simplest way to getting started is to install the django query profiler from pip, and get the chrome plugin from chrome web store.

Python package:

pip install django-query-profiler

Chrome Plugin:

Download from chrome webstore

This is covered in detail in the installation section in the docs

configuration:

This configuration is when we want to use the profiler along with the chrome plugin. If we want to just use it on the command line, the configuration is much more simpler (two lines of change to settings.py file) - that is covered in the docs

settings.py:

from django_query_profiler.settings import *

INSTALLED_APPS = (
    ...
    'django_query_profiler',
    ...
)

MIDDLEWARE = (
    ...
     # Request and all middleware that come after our middleware, would be profiled
    'django_query_profiler.client.middleware.QueryProfilerMiddleware',
    ...
)

DATABASES = (
    ...
    # Adding django_query_profiler as a prefix to your ENGINE setting
    # Assuming old ENGINE was "django.db.backends.sqlite3", this would be the new one
    "ENGINE": "django_query_profiler.django.db.backends.sqlite3",
)

urls.py:

# Add this line to existing urls.py
path('django_query_profiler/', include('django_query_profiler.client.urls'))

See this PR on how to configure this in your application, and how the plugin is going to look like after your configuration

https://raw.githubusercontent.com/django-query-profiler/django-query-profiler/master/docs/_static/chrome_plugin.png

This is covered in detail in the configuration instructions section in the docs

How the profiler works

This is also covered in detail in the documentation at how the profiler works section in the docs, along with how the code is organized.

The docs also contain references to various links which helped us to lear about internals of Django, and to various projects which helped us to learn on how to add hooks when Django executes a query

Choosing Profiler levels

We have two levels of profiler, and each of them have a different overhead. The two levels are:

  1. QUERY_SIGNATURE: This is the mode where we capture the query as well as the stack-trace. This mode figures out the N+1 code paths and also tells us the proposed solution
  2. QUERY: This is the mode where we just capture queries, and not the stack-trace

On an average, QUERY_SIGNATURE level adds an overhead of 1 millisecond per 7 queries, and QUERY_SIGNATURE adds an overhead of 1 millisecond per 25 queries.

It is simple to change the profiler level for all the requests, or can be configured per request. This is covered in the choosing profiler level section of the docs

Customizing the profiler

We have tried to make the profiler customizable by providing hooks at various points. Some of the use cases are covered here in the customizing the defaults section in docs.

We plan to add more hooks for customizing the profiler as we gather more feedback from real world use cases.

For contributors

https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square

The django query profiler is released under the BSD license, like Django itself.

If you like it, please consider contributing! The docs cover everything from how to setup locally, to how the code is organized to running tests.

Documentation

Full documentation is available at readthedocs

Comments
  • AttributeError: 'Connection' object has no attribute 'errno'

    AttributeError: 'Connection' object has no attribute 'errno'

    I am getting this error after installation.

    It seems to be based on this line:

    "ENGINE": "django_query_profiler.django.db.backends.mysql",

    When I remove the django_query_profiler, no issue, but when it is in the engine string, it always throws:

      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django_query_profiler/django/db/backends/mysql/base.py", line 12, in db_row_count
        return cursor.rowcount if not cursor.connection.errno() else -1
    AttributeError: 'Connection' object has no attribute 'errno'
    
    opened by haveamission 10
  • AttributeError: 'Settings' object has no attribute 'DJANGO_QUERY_PROFILER_REDIS_HOST'

    AttributeError: 'Settings' object has no attribute 'DJANGO_QUERY_PROFILER_REDIS_HOST'

    Hi Django Query Profiler authors,

    I've come across this project while looking for a solution for the current state of limbo at Django-Silk (https://github.com/jazzband/django-silk/issues/465). :)

    I've installed the package and configured settings.py and urls.py per https://django-query-profiler.readthedocs.io/en/latest/configuration_instructions.html. However, I hit this error AttributeError: 'Settings' object has no attribute 'DJANGO_QUERY_PROFILER_REDIS_HOST' when running the server.

    Could you let me know how I can overcome this problem?

    Thank you very much!

    Vinh

    opened by TheVinhLuong102 4
  • RecursionError at query-signature

    RecursionError at query-signature

    I seem to be running into a bug but I'm not really sure on why this fails at the point where it fails.

    RecursionError at /django_query_profiler/7c1ed3533d2a4c91aa3736812dffe345/QUERY_SIGNATURE maximum recursion depth exceeded while calling a Python object

    image

    This is in the file: django_query_profiler/templates/django_query_profiler_level_query_signature.html line 177. If I expand the if statement with {% if query_signature and query_signature.analysis.visible_in_ui %}, this works fine, just does not show the analysis name.

    opened by sajoku 3
  • Some Postgresql specific queries cause the QUERY_SIGNATURE view to crash

    Some Postgresql specific queries cause the QUERY_SIGNATURE view to crash

    Moz parser can't handle certain Postgresql specific queries and returns a ParseException (e.g. SELECT DISTINCT ON https://www.geekytidbits.com/postgres-distinct-on/).

    This results into the QUERY_SIGNATURE view to crash when trying to render the recommendation for those queries.

    ParseException at /django_query_profiler/2caa74b2f946492cade911200a7f7d8a/QUERY_SIGNATURE
    Expecting one of (binary_and, binary_or, concat, in, neq, offset, and, left join, inner join, group by, not_between, collate nocase, full join, full outer join, order by, between, is, or, left outer join, right join, nlike, nin, right outer join, union, like, limit, having, join) (at char 134), (line:1, col:135)
    
    opened by jur-clerkx 3
  • It relies on moz-sql-parser, which has already been archived and deprecated.

    It relies on moz-sql-parser, which has already been archived and deprecated.

    It relies on moz-sql-parser, which has already been archived and deprecated. https://github.com/mozilla/moz-sql-parser Is there any attempt to make this the successor mo-sql-parsing? https://github.com/klahnakoski/mo-sql-parsing

    I looked at the source code for an hour trying to do this, but I wasn't sure. sorry.

    opened by Taikono-Himazin 2
  • Changed

    Changed "ms" to "μs" for reporting query time

    The abbreviation "ms" is milliseconds while the abbreviation "μs" is microseconds. There are 1,000 microseconds in a millesecond. The value of the variable it is referencing is also called total_query_execution_time_in_micros. I did a cursory check of the logic behind this variable to verify it is reporting in microseconds:

    query_execution_time_in_micros = int((end_time - start_time) * 1000 * 1000)

    end_time - start_time is in seconds, multiply by 1,000 is in milleseconds and multiplying again by 1,000 is in microseconds. 👍

    While I was in these files, I also fixed some casing of words/acronyms that I saw.

    opened by dbudwin 2
  • Flamegraph Display

    Flamegraph Display

    127 0 0 1_8000_django_query_profiler_1b3e059ddedb40278aeb29816769ddc6_QUERY_SIGNATURE_name=food_order

    Uses d3 flame-graph, you can click to "zoom" etc.

    (Note ported from same feature added to Zenefits internal tool, there's some small handling here for "multiple roots" which maybe can't happen with the open-source version (i.e. middleware vs view for the same request UUID.)

    opened by glynn-zenefits 2
  • Fix for issue 19 and issue 21

    Fix for issue 19 and issue 21

    • Adding Django 3.1 and 3.2 in classifiers
    • Adding Python 3.9 in classifiers
    • Adding Django 3.2 in tox for testing on CI
    • Removing Django tip for now ; Going to add it back later in a separate PR
    opened by yashmaheshwari 1
  • Can't get details:

    Can't get details: "redis_or_urls.py_not_setup"

    I'm getting the following error when using the Chrome extension.

    image

    The error seems to be coming from here: https://github.com/django-query-profiler/django-query-profiler/blob/04483ccc7dd3ef93f51c35f938e77347b050bd21/django_query_profiler/client/middleware.py#L57

    There is a comment that says:

    The exception can happen because of two reasons:

    1. redis throws exception
    2. detailed_view_url not setup in urls.py

    I am not using Redis and I have triple-checked that I have added path('django_query_profiler/', include('django_query_profiler.client.urls')) to my urls.py. However, the comment does reference detailed_view_url and I am unsure what that is referring to. I went over the setup instructions multiple times to confirm I have configured my app correctly. I am using the SQLite version of the database engine. I have also messed with the order of the middleware too since I know that can be problematic sometimes.

    I am using Python 3.8, Django 3.1.5, DRF 3.12.1.

    opened by fgs-dbudwin 1
  • Fixing master with latest changes for dependent packages

    Fixing master with latest changes for dependent packages

    • Was getting some package incompatibility error for mo-future
    • We are not using the package directly
    • Maybe it means we should have a requirements.txt file ?
    • Figured out the version used by running pip freeze
    • Added fix for djangotip
    • Fixed for flake8
    opened by yashmaheshwari 1
  • Showing proposed solution only when it is actionable

    Showing proposed solution only when it is actionable

    1. In the detailed view, we should show recommendation for decreasing queries only when it is one of select_related or prefetch_related
    2. Adding a constructor param to enum for visible_in_ui to see if we should show the recommendation in chrome plugin
    opened by yashmaheshwari 1
  • Python 3.10 support

    Python 3.10 support

    Hi, this currently fails to install on Python 3.10, but only because there is no 3.10 compatible wheel available for the mmh3 dependency.

    Is it essential that mmh3 be used as the hash function here? Something more standard like the standard library's hashlib might prevent this issue from occurring in the future.

    opened by dmartin 1
Owner
Django Query Profiler
Contact us at [email protected]
Django Query Profiler
Django app for building dashboards using raw SQL queries

django-sql-dashboard Django app for building dashboards using raw SQL queries Brings a useful subset of Datasette to Django. Currently only works with

Simon Willison 383 Jan 6, 2023
A beginner django project and also my first Django project which involves shortening of a longer URL into a short one using a unique id.

Django-URL-Shortener A beginner django project and also my first Django project which involves shortening of a longer URL into a short one using a uni

Rohini Rao 3 Aug 8, 2021
Sampling profiler for Python programs

py-spy: Sampling profiler for Python programs py-spy is a sampling profiler for Python programs. It lets you visualize what your Python program is spe

Ben Frederickson 9.5k Jan 1, 2023
Django URL Shortener is a Django app to to include URL Shortening feature in your Django Project

Django URL Shortener Django URL Shortener is a Django app to to include URL Shortening feature in your Django Project Install this package to your Dja

Rishav Sinha 4 Nov 18, 2021
Easily share data across your company via SQL queries. From Grove Collab.

SQL Explorer SQL Explorer aims to make the flow of data between people fast, simple, and confusion-free. It is a Django-based application that you can

Grove Collaborative 2.1k Dec 30, 2022
Auto-detecting the n+1 queries problem in Python

nplusone nplusone is a library for detecting the n+1 queries problem in Python ORMs, including SQLAlchemy, Peewee, and the Django ORM. The Problem Man

Joshua Carp 837 Dec 29, 2022
A drop-in replacement for django's ImageField that provides a flexible, intuitive and easily-extensible interface for quickly creating new images from the one assigned to the field.

django-versatileimagefield A drop-in replacement for django's ImageField that provides a flexible, intuitive and easily-extensible interface for creat

Jonathan Ellenberger 490 Dec 13, 2022
A web app which allows user to query the weather info of any place in the world

weather-app This is a web app which allows user to get the weather info of any place in the world as soon as possible. It makes use of OpenWeatherMap

Oladipo Adesiyan 3 Sep 20, 2021
Yet another Django audit log app, hopefully the simplest one.

django-easy-audit Yet another Django audit log app, hopefully the easiest one. This app allows you to keep track of every action taken by your users.

Natán 510 Jan 2, 2023
Django's class-based generic views are awesome, let's have more of them.

Django Extra Views - The missing class-based generic views for Django Django-extra-views is a Django package which introduces additional class-based v

Andy Ingram 1.3k Jan 4, 2023
Get inside your stronghold and make all your Django views default login_required

Stronghold Get inside your stronghold and make all your Django views default login_required Stronghold is a very small and easy to use django app that

Mike Grouchy 384 Nov 23, 2022
Django project starter on steroids: quickly create a Django app AND generate source code for data models + REST/GraphQL APIs (the generated code is auto-linted and has 100% test coverage).

Create Django App ?? We're a Django project starter on steroids! One-line command to create a Django app with all the dependencies auto-installed AND

imagine.ai 68 Oct 19, 2022
A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, celery and redis.

Django Channels Websocket Chatbot A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, c

Yunbo Shi 8 Oct 28, 2022
Blog focused on skills enhancement and knowledge sharing. Tech Stack's: Vue.js, Django and Django-Ninja

Blog focused on skills enhancement and knowledge sharing. Tech Stack's: Vue.js, Django and Django-Ninja

Wanderson Fontes 2 Sep 21, 2022
Django GUID attaches a unique correlation ID/request ID to all your log outputs for every request.

Django GUID Now with ASGI support! Django GUID attaches a unique correlation ID/request ID to all your log outputs for every request. In other words,

snok 300 Dec 29, 2022
Meta package to combine turbo-django and stimulus-django

Hotwire + Django This repository aims to help you integrate Hotwire with Django ?? Inspiration might be taken from @hotwired/hotwire-rails. We are sti

Hotwire for Django 31 Aug 9, 2022
django-quill-editor makes Quill.js easy to use on Django Forms and admin sites

django-quill-editor django-quill-editor makes Quill.js easy to use on Django Forms and admin sites No configuration required for static files! The ent

lhy 139 Dec 5, 2022
Dockerizing Django with Postgres, Gunicorn, Nginx and Certbot. A fully Django starter project.

Dockerizing Django with Postgres, Gunicorn, Nginx and Certbot ?? Features A Django stater project with fully basic requirements for a production-ready

null 8 Jun 27, 2022
APIs for a Chat app. Written with Django Rest framework and Django channels.

ChatAPI APIs for a Chat app. Written with Django Rest framework and Django channels. The documentation for the http end points can be found here This

Victor Aderibigbe 18 Sep 9, 2022