Django query profiler - one profiler to rule them all. Shows queries, detects N+1 and gives recommendations on how to resolve them

Overview

Django Query Profiler

https://travis-ci.com/django-query-profiler/django-query-profiler.svg?branch=master https://codecov.io/gh/django-query-profiler/django-query-profiler/branch/master/graph/badge.svg?token=1Cv7WsOi2W https://readthedocs.org/projects/django-query-profiler/badge/?version=latest https://img.shields.io/pypi/djversions/django-query-profiler

This is a query profiler for Django applications, for helping developers answer the question "My Django code/page/API is slow, How do I find out why?"

Below are some of the features of the profiler:

  1. Shows code paths making N+1 sql calls: Shows the sql with stack_trace which is making N+1 calls, along with sql count
  2. Shows the proposed solution: If the solution to reduce sql is to simply apply a select_related or a prefetch_related, this is highlighted as a suggestion
  3. Shows exact sql duplicates: Count of the queries where (sql, parameters) is exactly the same. This is the kind of sql where implementing a query cache would help
  4. Flame Graph visualisation: Collects all the stack traces together to allow quickly identifying which area(s) of code is causing the load to the database
  5. Command line or chrome plugin: The profiler can be called from command line via context manager, or can be invoked via a middleware, and output shown in a chrome plugin
  6. Super easy to configure in any application: The only changes are in settings.py file and in urls.py file

This is the repo for the chrome plugin

Requirements

This works with any version of django >= 2.0, and running on python >= 3.6

Profiler in Action

as a chrome plugin

This image shows how the chrome plugin would display profiled data, once it is configured & installed

https://raw.githubusercontent.com/django-query-profiler/django-query-profiler/master/docs/_static/django_query_profiler_in_action.gif

on command line

See this file in the PR to see how to use the context manager, and how easy it is to find performance issues :-)

The output of Django query profiler is same for the command line or the chrome plugin. In fact, chrome plugin displays the output set by the middleware - which is just a plain wrapper around context manager.

Getting Started

installation

The simplest way to getting started is to install the django query profiler from pip, and get the chrome plugin from chrome web store.

Python package:

pip install django-query-profiler

Chrome Plugin:

Download from chrome webstore

This is covered in detail in the installation section in the docs

configuration:

This configuration is when we want to use the profiler along with the chrome plugin. If we want to just use it on the command line, the configuration is much more simpler (two lines of change to settings.py file) - that is covered in the docs

settings.py:

from django_query_profiler.settings import *

INSTALLED_APPS = (
    ...
    'django_query_profiler',
    ...
)

MIDDLEWARE = (
    ...
     # Request and all middleware that come after our middleware, would be profiled
    'django_query_profiler.client.middleware.QueryProfilerMiddleware',
    ...
)

DATABASES = (
    ...
    # Adding django_query_profiler as a prefix to your ENGINE setting
    # Assuming old ENGINE was "django.db.backends.sqlite3", this would be the new one
    "ENGINE": "django_query_profiler.django.db.backends.sqlite3",
)

urls.py:

# Add this line to existing urls.py
path('django_query_profiler/', include('django_query_profiler.client.urls'))

See this PR on how to configure this in your application, and how the plugin is going to look like after your configuration

https://raw.githubusercontent.com/django-query-profiler/django-query-profiler/master/docs/_static/chrome_plugin.png

This is covered in detail in the configuration instructions section in the docs

How the profiler works

This is also covered in detail in the documentation at how the profiler works section in the docs, along with how the code is organized.

The docs also contain references to various links which helped us to lear about internals of Django, and to various projects which helped us to learn on how to add hooks when Django executes a query

Choosing Profiler levels

We have two levels of profiler, and each of them have a different overhead. The two levels are:

  1. QUERY_SIGNATURE: This is the mode where we capture the query as well as the stack-trace. This mode figures out the N+1 code paths and also tells us the proposed solution
  2. QUERY: This is the mode where we just capture queries, and not the stack-trace

On an average, QUERY_SIGNATURE level adds an overhead of 1 millisecond per 7 queries, and QUERY_SIGNATURE adds an overhead of 1 millisecond per 25 queries.

It is simple to change the profiler level for all the requests, or can be configured per request. This is covered in the choosing profiler level section of the docs

Customizing the profiler

We have tried to make the profiler customizable by providing hooks at various points. Some of the use cases are covered here in the customizing the defaults section in docs.

We plan to add more hooks for customizing the profiler as we gather more feedback from real world use cases.

For contributors

https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square

The django query profiler is released under the BSD license, like Django itself.

If you like it, please consider contributing! The docs cover everything from how to setup locally, to how the code is organized to running tests.

Documentation

Full documentation is available at readthedocs

Comments
  • AttributeError: 'Connection' object has no attribute 'errno'

    AttributeError: 'Connection' object has no attribute 'errno'

    I am getting this error after installation.

    It seems to be based on this line:

    "ENGINE": "django_query_profiler.django.db.backends.mysql",

    When I remove the django_query_profiler, no issue, but when it is in the engine string, it always throws:

      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/django_query_profiler/django/db/backends/mysql/base.py", line 12, in db_row_count
        return cursor.rowcount if not cursor.connection.errno() else -1
    AttributeError: 'Connection' object has no attribute 'errno'
    
    opened by haveamission 10
  • AttributeError: 'Settings' object has no attribute 'DJANGO_QUERY_PROFILER_REDIS_HOST'

    AttributeError: 'Settings' object has no attribute 'DJANGO_QUERY_PROFILER_REDIS_HOST'

    Hi Django Query Profiler authors,

    I've come across this project while looking for a solution for the current state of limbo at Django-Silk (https://github.com/jazzband/django-silk/issues/465). :)

    I've installed the package and configured settings.py and urls.py per https://django-query-profiler.readthedocs.io/en/latest/configuration_instructions.html. However, I hit this error AttributeError: 'Settings' object has no attribute 'DJANGO_QUERY_PROFILER_REDIS_HOST' when running the server.

    Could you let me know how I can overcome this problem?

    Thank you very much!

    Vinh

    opened by TheVinhLuong102 4
  • RecursionError at query-signature

    RecursionError at query-signature

    I seem to be running into a bug but I'm not really sure on why this fails at the point where it fails.

    RecursionError at /django_query_profiler/7c1ed3533d2a4c91aa3736812dffe345/QUERY_SIGNATURE maximum recursion depth exceeded while calling a Python object

    image

    This is in the file: django_query_profiler/templates/django_query_profiler_level_query_signature.html line 177. If I expand the if statement with {% if query_signature and query_signature.analysis.visible_in_ui %}, this works fine, just does not show the analysis name.

    opened by sajoku 3
  • Some Postgresql specific queries cause the QUERY_SIGNATURE view to crash

    Some Postgresql specific queries cause the QUERY_SIGNATURE view to crash

    Moz parser can't handle certain Postgresql specific queries and returns a ParseException (e.g. SELECT DISTINCT ON https://www.geekytidbits.com/postgres-distinct-on/).

    This results into the QUERY_SIGNATURE view to crash when trying to render the recommendation for those queries.

    ParseException at /django_query_profiler/2caa74b2f946492cade911200a7f7d8a/QUERY_SIGNATURE
    Expecting one of (binary_and, binary_or, concat, in, neq, offset, and, left join, inner join, group by, not_between, collate nocase, full join, full outer join, order by, between, is, or, left outer join, right join, nlike, nin, right outer join, union, like, limit, having, join) (at char 134), (line:1, col:135)
    
    opened by jur-clerkx 3
  • It relies on moz-sql-parser, which has already been archived and deprecated.

    It relies on moz-sql-parser, which has already been archived and deprecated.

    It relies on moz-sql-parser, which has already been archived and deprecated. https://github.com/mozilla/moz-sql-parser Is there any attempt to make this the successor mo-sql-parsing? https://github.com/klahnakoski/mo-sql-parsing

    I looked at the source code for an hour trying to do this, but I wasn't sure. sorry.

    opened by Taikono-Himazin 2
  • Changed

    Changed "ms" to "μs" for reporting query time

    The abbreviation "ms" is milliseconds while the abbreviation "μs" is microseconds. There are 1,000 microseconds in a millesecond. The value of the variable it is referencing is also called total_query_execution_time_in_micros. I did a cursory check of the logic behind this variable to verify it is reporting in microseconds:

    query_execution_time_in_micros = int((end_time - start_time) * 1000 * 1000)

    end_time - start_time is in seconds, multiply by 1,000 is in milleseconds and multiplying again by 1,000 is in microseconds. 👍

    While I was in these files, I also fixed some casing of words/acronyms that I saw.

    opened by dbudwin 2
  • Flamegraph Display

    Flamegraph Display

    127 0 0 1_8000_django_query_profiler_1b3e059ddedb40278aeb29816769ddc6_QUERY_SIGNATURE_name=food_order

    Uses d3 flame-graph, you can click to "zoom" etc.

    (Note ported from same feature added to Zenefits internal tool, there's some small handling here for "multiple roots" which maybe can't happen with the open-source version (i.e. middleware vs view for the same request UUID.)

    opened by glynn-zenefits 2
  • Fix for issue 19 and issue 21

    Fix for issue 19 and issue 21

    • Adding Django 3.1 and 3.2 in classifiers
    • Adding Python 3.9 in classifiers
    • Adding Django 3.2 in tox for testing on CI
    • Removing Django tip for now ; Going to add it back later in a separate PR
    opened by yashmaheshwari 1
  • Can't get details:

    Can't get details: "redis_or_urls.py_not_setup"

    I'm getting the following error when using the Chrome extension.

    image

    The error seems to be coming from here: https://github.com/django-query-profiler/django-query-profiler/blob/04483ccc7dd3ef93f51c35f938e77347b050bd21/django_query_profiler/client/middleware.py#L57

    There is a comment that says:

    The exception can happen because of two reasons:

    1. redis throws exception
    2. detailed_view_url not setup in urls.py

    I am not using Redis and I have triple-checked that I have added path('django_query_profiler/', include('django_query_profiler.client.urls')) to my urls.py. However, the comment does reference detailed_view_url and I am unsure what that is referring to. I went over the setup instructions multiple times to confirm I have configured my app correctly. I am using the SQLite version of the database engine. I have also messed with the order of the middleware too since I know that can be problematic sometimes.

    I am using Python 3.8, Django 3.1.5, DRF 3.12.1.

    opened by fgs-dbudwin 1
  • Fixing master with latest changes for dependent packages

    Fixing master with latest changes for dependent packages

    • Was getting some package incompatibility error for mo-future
    • We are not using the package directly
    • Maybe it means we should have a requirements.txt file ?
    • Figured out the version used by running pip freeze
    • Added fix for djangotip
    • Fixed for flake8
    opened by yashmaheshwari 1
  • Showing proposed solution only when it is actionable

    Showing proposed solution only when it is actionable

    1. In the detailed view, we should show recommendation for decreasing queries only when it is one of select_related or prefetch_related
    2. Adding a constructor param to enum for visible_in_ui to see if we should show the recommendation in chrome plugin
    opened by yashmaheshwari 1
  • Python 3.10 support

    Python 3.10 support

    Hi, this currently fails to install on Python 3.10, but only because there is no 3.10 compatible wheel available for the mmh3 dependency.

    Is it essential that mmh3 be used as the hash function here? Something more standard like the standard library's hashlib might prevent this issue from occurring in the future.

    opened by dmartin 1
Owner
Django Query Profiler
Contact us at [email protected]
Django Query Profiler
Silky smooth profiling for Django

Silk Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before prese

Jazzband 3.7k Jan 1, 2023
guapow is an on-demand and auto performance optimizer for Linux applications.

guapow is an on-demand and auto performance optimizer for Linux applications. This project's name is an abbreviation for Guarana powder (Guaraná is a fruit from the Amazon rainforest with a highly caffeinated seed).

Vinícius Moreira 19 Nov 18, 2022
Pearpy - a Python package for writing multithreaded code and parallelizing tasks across CPU threads.

Pearpy The Python package for (pear)allelizing your tasks across multiple CPU threads. Installation The latest version of Pearpy can be installed with

MLH Fellowship 5 Nov 1, 2021
Django query profiler - one profiler to rule them all. Shows queries, detects N+1 and gives recommendations on how to resolve them

Django Query Profiler This is a query profiler for Django applications, for helping developers answer the question "My Django code/page/API is slow, H

Django Query Profiler 116 Dec 15, 2022
One version package to rule them all, One version package to find them, One version package to bring them all, and in the darkness bind them.

AwesomeVersion One version package to rule them all, One version package to find them, One version package to bring them all, and in the darkness bind

Joakim Sørensen 39 Dec 31, 2022
PyPika is a python SQL query builder that exposes the full richness of the SQL language using a syntax that reflects the resulting query. PyPika excels at all sorts of SQL queries but is especially useful for data analysis.

PyPika - Python Query Builder Abstract What is PyPika? PyPika is a Python API for building SQL queries. The motivation behind PyPika is to provide a s

KAYAK 1.9k Jan 4, 2023
One Metrics Library to Rule Them All!

onemetric Installation Install onemetric from PyPI (recommended): pip install onemetric Install onemetric from the GitHub source: git clone https://gi

Piotr Skalski 49 Jan 3, 2023
Turn SELECT queries returned by a query into links to execute them

datasette-query-links Turn SELECT queries returned by a query into links to execute them Installation Install this plugin in the same environment as D

Simon Willison 5 Apr 27, 2022
Bot simply search for the files from provided channel according to given query and gives link to those files as buttons!

Auto Filter Bot ㅤㅤㅤㅤㅤㅤㅤ ㅤㅤㅤㅤㅤㅤㅤ You can call this as an Auto Filter Bot if you like :D Bot simply search for the files from provided channel according

TroJanzHEX 89 Nov 23, 2022
🚴 Call stack profiler for Python. Shows you why your code is slow!

pyinstrument Pyinstrument is a Python profiler. A profiler is a tool to help you 'optimize' your code - make it faster. It sounds obvious, but to get

Joe Rickerby 5k Jan 1, 2023
Automatic Video Library Manager for TV Shows. It watches for new episodes of your favorite shows, and when they are posted it does its magic.

Automatic Video Library Manager for TV Shows. It watches for new episodes of your favorite shows, and when they are posted it does its magic. Exclusiv

pyMedusa 1.5k Dec 30, 2022
A CLI for streaming, downloading anime shows. The shows data is indexed through GogoAnime.

Anime-cli A CLI for streaming, downloading anime shows. The shows data is indexed through GogoAnime. Please install mpv video-player for better experi

Chirag Singla 31 Oct 23, 2022
Ahmed Hossam 12 Oct 17, 2022
tfquery: Run SQL queries on your Terraform infrastructure. Query resources and analyze its configuration using a SQL-powered framework.

??️ tfquery ??️ Run SQL queries on your Terraform infrastructure. Ask questions that are hard to answer ?? What is tfquery? tfquery is a framework tha

Mazin Ahmed 311 Dec 21, 2022
A python framework to transform natural language questions to queries in a database query language.

__ _ _ _ ___ _ __ _ _ / _` | | | |/ _ \ '_ \| | | | | (_| | |_| | __/ |_) | |_| | \__, |\__,_|\___| .__/ \__, | |_| |_| |___/

Machinalis 1.2k Dec 18, 2022
A Project to resolve hostname and receive IP

hostname-resolver A Project to resolve hostname and receive IP Installation git clone https://github.com/ihapiw/hostname-resolver.git Head into the ho

iHapiW 5 Sep 12, 2022
:package: :fire: Python project management. Manage packages: convert between formats, lock, install, resolve, isolate, test, build graph, show outdated, audit. Manage venvs, build package, bump version.

THE PROJECT IS ARCHIVED Forks: https://github.com/orsinium/forks DepHell -- project management for Python. Why it is better than all other tools: Form

DepHell 1.7k Dec 30, 2022
Automatically resolve RidderMaster based on TensorFlow & OpenCV

AutoRiddleMaster Automatically resolve RidderMaster based on TensorFlow & OpenCV 基于 TensorFlow 和 OpenCV 实现的全自动化解御迷士小马谜题 Demo How to use Deploy the ser

神龙章轩 5 Nov 19, 2021
A raw implementation of the nearest insertion algorithm to resolve TSP problems in a TXT format.

TSP-Nearest-Insertion A raw implementation of the nearest insertion algorithm to resolve TSP problems in a TXT format. Instructions Load a txt file wi

sjas_Phantom 1 Dec 2, 2021