Middleware that Prints the number of DB queries to the runserver console.

Overview

Django Querycount

Current Release License

Inspired by this post by David Szotten, this project gives you a middleware that prints DB query counts in Django's runserver console output.

django-querycount in action

Installation

pip install django-querycount

Just add querycount.middleware.QueryCountMiddleware to your MIDDLEWARE.

Notice that django-querycount is hard coded to work only in DEBUG mode set to true

Settings

There are two possible settings for this app: The first defines threshold values used to color output, while the second allows you customize requests that will be ignored by the middleware. The default settings are:

QUERYCOUNT = {
    'THRESHOLDS': {
        'MEDIUM': 50,
        'HIGH': 200,
        'MIN_TIME_TO_LOG':0,
        'MIN_QUERY_COUNT_TO_LOG':0
    },
    'IGNORE_REQUEST_PATTERNS': [],
    'IGNORE_SQL_PATTERNS': [],
    'DISPLAY_DUPLICATES': None,
    'RESPONSE_HEADER': 'X-DjangoQueryCount-Count'
}

The QUERYCOUNT['THRESHOLDS'] settings will determine how many queries are interpreted as high or medium (and the color-coded output). In previous versions of this app, this settings was called QUERYCOUNT_THRESHOLDS and that setting is still supported.

The QUERYCOUNT['IGNORE_REQUEST_PATTERNS'] setting allows you to define a list of regexp patterns that get applied to each request's path. If there is a match, the middleware will not be applied to that request. For example, the following setting would bypass the querycount middleware for all requests to the admin:

QUERYCOUNT = {
    'IGNORE_REQUEST_PATTERNS': [r'^/admin/']
}

The QUERYCOUNT['IGNORE_SQL_PATTERNS'] setting allows you to define a list of regexp patterns that ignored to statistic sql query count. For example, the following setting would bypass the querycount middleware for django-silk sql query:

QUERYCOUNT = {
    'IGNORE_SQL_PATTERNS': [r'silk_']
}

The QUERYCOUNT['RESPONSE_HEADER'] setting allows you to define a custom response header that contains the total number of queries executed. To disable this header, the supply None as the value:

QUERYCOUNT = {
    'RESPONSE_HEADER': None
}

New in 0.4.0. The QUERYCOUNT['DISPLAY_DUPLICATES'] setting allows you to control how the most common duplicate queries are displayed. If the setting is None (the default), duplicate queries are not displayed. Otherwise, this should be an integer. For example, the following setting would always print the 5 most duplicated queries:

QUERYCOUNT = {
    'DISPLAY_DUPLICATES': 5,
}

License

This code is distributed under the terms of the MIT license.

Testing

Run python manage.py test querycount to run the tests. Note that this will modify your settings so that your project is in DEBUG mode for the duration of the querycount tests.

(side-note: this project needs better tests; for the moment, there are only smoke tests that set up the middleware and call two simple test views).

Contributing

Bug fixes and new features are welcome! Fork this project and send a Pull Request to have your work included. Be sure to add yourself to AUTHORS.rst.

Comments
  • Usage without request and response

    Usage without request and response

    Hi, thanks for this awesome package. I really enjoy using it.

    But I have a question about how to use this package without a request instance? Here are some examples where I need this:

    1. celery tasks, in DEBUG I want them to print the query count in the same manner middleware does
    2. telegram bots, we do develop some telegram bots on top of django. telegram bots serve as clients which does not have any django's request instance. But I still want this app to work

    Thanks!

    opened by sobolevn 4
  • Feature/response header count and setting reload

    Feature/response header count and setting reload

    This PR adds functionality to return a custom response header with total request & response query count. We plan to use this functionality in our integration test suite, to ensure all endpoints are under the allowed count.

    Additionally, when attempting to override the settings to ensure the header was being removed, I found the need to ensure the settings are reloaded, so I've cleaned up the settings application and wired them up to the Django setting changed signal.

    opened by LifeCoder45 3
  • Added an

    Added an "elapsed" and "query count" log threshold optional setting and documented the settings in readme

    This is a very useful tool to debug django db performance.. but data can get cluttered between tons of small and meaningless requests. this allows the middleware user to selected the query elapsed time and the minimal query count that he wants the middleware to log. Enabling the user to focus on bottlenecks and problems.

    Without explicitly changing the settings the current behavior isn't changed in any way

    opened by alonisser 3
  • AttributeError: 'QueryCountMiddleware' object has no attribute 'get_response'

    AttributeError: 'QueryCountMiddleware' object has no attribute 'get_response'

    Test client throws above error when I try to test this snippet

    class IndexPageTest(TestCase):
        def setUp(self):
            self.client = Client()
    
        def test_uses_login_template(self):
            response = self.client.get('/')
            self.assertTemplateUsed(response, 'index.html')
    

    Although this code is perfectly working if I disable the middleware from settings.py .

    Here is full output

    Creating test database for alias 'default'...
    E
    ======================================================================
    ERROR: test_uses_login_template (....tests.IndexPageTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File ".../tests.py", line 11, in test_uses_login_template
        response = self.client.get('/')
      File "....env/lib/python3.5/site-packages/django/test/client.py", line 529, in get
        **extra)
      File "....env/lib/python3.5/site-packages/django/test/client.py", line 333, in get
        return self.generic('GET', path, secure=secure, **r)
      File ".../.env/lib/python3.5/site-packages/django/test/client.py", line 409, in generic
        return self.request(**r)
      File ".../.env/lib/python3.5/site-packages/django/test/client.py", line 494, in request
        six.reraise(*exc_info)
      File "....env/lib/python3.5/site-packages/django/utils/six.py", line 686, in reraise
        raise value
      File ".../.env/lib/python3.5/site-packages/django/core/handlers/exception.py", line 42, in inner
        response = get_response(request)
      File "....env/lib/python3.5/site-packages/django/utils/deprecation.py", line 136, in __call__
        response = self.get_response(request)
    AttributeError: 'QueryCountMiddleware' object has no attribute 'get_response'
    
    ----------------------------------------------------------------------
    Ran 1 test in 2.621s
    
    FAILED (errors=1)
    
    
    opened by pyprism 1
  • Support Django 1.10 middleware

    Support Django 1.10 middleware

    This adds support for Django 1.10-style middleware: https://docs.djangoproject.com/en/1.10/topics/http/middleware/#upgrading-pre-django-1-10-style-middleware

    opened by ryankask 1
  • Disable colors when not supported (or explicitely disabled)

    Disable colors when not supported (or explicitely disabled)

    On Windows, using powershell, the output looks bad as colors are not supported :

    http://127.0.0.1:8000/admin/jsi18n/
    ?[0m|------|-----------|----------|----------|----------|------------|
    | Type | Database  |   Reads  |  Writes  |  Totals  | Duplicates |
    |------|-----------|----------|----------|----------|------------|
    ?[32m| RESP |  default  |    2     |    0     |    2     |     1      |
    ?[0m|------|-----------|----------|----------|----------|------------|
    ?[32mTotal queries: 2 in 0.0160s
    

    Colors should be disabled when not supported or disabled. I believe Django has all the logic needed in django\core\management\color.py

    opened by olivierdalang 0
  • "Repeated 1 times"

    It prints this:

    web_1            | Repeated 1 times.
    web_1            | SELECT `django_session`.`session_key`,
    web_1            | `django_session`.`session_data`, `django_session`.`expire_date` FROM
    web_1            | `django_session` WHERE (`django_session`.`session_key` =
    web_1            | 'cln2uzh2qk21qqb6sfg7nvsrqwg2crbm' AND `django_session`.`expire_date`
    web_1            | > '2019-09-20 12:53:39.352699')
    

    But I think it actually ever happens 1 time for that request. So, the logic for this inclusion should be something like [query for queries if query.times > 1].

    opened by peterbe 5
  • How to view the total time of repeated sql queries

    How to view the total time of repeated sql queries

    Can we show the total time take for the repeated calls individually

    For example: presently it shows Repeated 2 times.

    Can we have Repeated 2 times. [0.023]

    opened by sant527 0
  • Using sql parse and pygments to show sql formatted nicely

    Using sql parse and pygments to show sql formatted nicely

    I wanted the sql to be formatted so i have tried the below code for that. Can you incorporate sqlparse also if possible

    def _duplicate_queries(self, output):
            """Appends the most common duplicate queries to the given output."""
            if QC_SETTINGS['DISPLAY_DUPLICATES']:
                for query, count in self.queries.most_common(QC_SETTINGS['DISPLAY_DUPLICATES']):
                    lines = '\nRepeated {0} times.'.format(count)
                    #lines += wrap(query)
                    #lines = "\n".join(lines) + "\n"
                    lines += "\n" + self._str_query(query) + "\n"
                    output += self._colorize(lines, count)
            return output
    
        def _str_query(self,sql):
    
            # Check if Pygments is available for coloring
            try:
                import pygments
                from pygments.lexers import SqlLexer
                from pygments.formatters import TerminalTrueColorFormatter
            except ImportError:
                pygments = None
            # Check if sqlparse is available for indentation
            try:
                import sqlparse
            except ImportError:
                sqlparse = None
            # Remove leading and trailing whitespaces
            if sqlparse:
                # Indent the SQL query
                sql = sqlparse.format(sql, reindent=True)
            if pygments:
                # Highlight the SQL query
                sql = pygments.highlight(
                    sql,
                    SqlLexer(),
                    #TerminalTrueColorFormatter(style='monokai')
                    TerminalTrueColorFormatter()
                )
    
            return sql
    
    opened by sant527 1
  • Whitelist request patterns instead of blacklisting them

    Whitelist request patterns instead of blacklisting them

    I often debug a single endpoint with a unique request pattern. To be able to only see that in the console I have to blacklist all other request patterns that my application may produce during debugging. That can be quite annoying. It would be helpful to have a separate whitelist setting like INCLUDE_REQUEST_PATTERNS where I just state what I want to be printed to the console and everything else is silenced. Of course the User would have to decide whether to blacklist or whitelist, as using both does not make much sense.

    I can do a PR if you want, let me know if it's something worth putting time into.

    opened by cb109 1
Owner
Brad Montgomery
Pythonista & Djangonaut. I write other languages too. Sometimes English.
Brad Montgomery
A drop-in replacement for Django's runserver.

About A drop in replacement for Django's built-in runserver command. Features include: An extendable interface for handling things such as real-time l

David Cramer 1.3k Dec 15, 2022
A drop-in replacement for Django's runserver.

About A drop in replacement for Django's built-in runserver command. Features include: An extendable interface for handling things such as real-time l

David Cramer 1.3k Dec 15, 2022
Trace all method entries and exits, the exit also prints the return value, if it is of basic type

Trace all method entries and exits, the exit also prints the return value, if it is of basic type. The apk must have set the android:debuggable="true" flag.

Kurt Nistelberger 7 Aug 10, 2022
Full-screen console debugger for Python

PuDB: a console-based visual debugger for Python Its goal is to provide all the niceties of modern GUI-based debuggers in a more lightweight and keybo

Andreas Klöckner 2.6k Jan 1, 2023
Auto-detecting the n+1 queries problem in Python

nplusone nplusone is a library for detecting the n+1 queries problem in Python ORMs, including SQLAlchemy, Peewee, and the Django ORM. The Problem Man

Joshua Carp 837 Dec 29, 2022
Django package to log request values such as device, IP address, user CPU time, system CPU time, No of queries, SQL time, no of cache calls, missing, setting data cache calls for a particular URL with a basic UI.

django-web-profiler's documentation: Introduction: django-web-profiler is a django profiling tool which logs, stores debug toolbar statistics and also

MicroPyramid 77 Oct 29, 2022
Code Specialist 27 Oct 16, 2022
A listener for RF >= 4.0 that prints a Stack Trace to console to faster find the code section where the failure appears.

robotframework-stacktrace A listener for RF >= 4.0 that prints a Stack Trace to console to faster find the code section where the failure appears. Ins

marketsquare 16 Nov 24, 2022
Tool for quickly gathering information from Shodan.io about the number of IPs which satisfy large number of different queries

TriOp Tool for quickly gathering information from Shodan.io about the number of IPs which satisfy large number of different queries For furt

Jan Kopriva 27 Nov 3, 2022
Dahua Console, access internal debug console and/or other researched functions in Dahua devices.

Dahua Console, access internal debug console and/or other researched functions in Dahua devices.

bashis 156 Dec 28, 2022
A drop-in replacement for Django's runserver.

About A drop in replacement for Django's built-in runserver command. Features include: An extendable interface for handling things such as real-time l

David Cramer 1.3k Dec 15, 2022
A drop-in replacement for Django's runserver.

About A drop in replacement for Django's built-in runserver command. Features include: An extendable interface for handling things such as real-time l

David Cramer 1.3k Dec 15, 2022
A simple plugin to attach a debugger in Django on runserver command.

django-debugger A simple plugin to attach a debugger in Django during runserver Installation pip install django-debugger Usage Prepend django_debugger

Sajal Shrestha 11 Nov 15, 2021
A drop-in replacement for Django's runserver.

About A drop in replacement for Django's built-in runserver command. Features include: An extendable interface for handling things such as real-time l

David Cramer 1.3k Dec 15, 2022
Convert any-bit number to decimal number and vise versa.

2deci Convert any-bit number to decimal number and vise versa. --bit n to set bit to n --exp xxx to set expression to xxx --r to run reversely (from d

null 3 Sep 15, 2021
Get-Phone-Number-Details-using-Python - To get the details of any number, we can use an amazing Python module known as phonenumbers.

Get-Phone-Number-Details-using-Python To get the details of any number, we can use an amazing Python module known as phonenumbers. We can use the amaz

Coding Taggers 1 Jan 1, 2022
PyLaboratory 0 Feb 7, 2022
Guess The Random Number - A sample Random Number Guessing Game Python Program

Guess_The_Random_Number This repo contains a simple "Random Number Guessing Game

Pramod Kumar 3 Feb 9, 2022
VG-Scraper is a python program using the module called BeautifulSoup which allows anyone to scrape something off an website. This program lets you put in a number trough an input and a number is 1 news article.

VG-Scraper VG-Scraper is a convinient program where you can find all the news articles instead of finding one yourself. Installing [Linux] Open a term

null 3 Feb 13, 2022
Prints values and types during compilation!

Compile-Time Printer Compile-Time Printer prints values and types at compile-time in C++. Teaser test.cpp compile-time-printer

null 43 Dec 26, 2022