Real-time metrics for nginx server

Overview

ngxtop - real-time metrics for nginx server (and others)

ngxtop parses your nginx access log and outputs useful, top-like, metrics of your nginx server. So you can tell what is happening with your server in real-time.

ngxtop is designed to run in a short-period time just like the top command for troubleshooting and monitoring your Nginx server at the moment. If you need a long running monitoring process or storing your webserver stats in external monitoring / graphing system, you can try Luameter.

ngxtop tries to determine the correct location and format of nginx access log file by default, so you can just run ngxtop and having a close look at all requests coming to your nginx server. But it does not limit you to nginx and the default top view. ngxtop is flexible enough for you to configure and change most of its behaviours. You can query for different things, specify your log and format, even parse remote Apache common access log with ease. See sample usages below for some ideas about what you can do with it.

Installation

pip install ngxtop

Note: ngxtop is primarily developed and tested with python2 but also supports python3.

Usage

Usage:
    ngxtop [options]
    ngxtop [options] (print|top|avg|sum) <var>
    ngxtop info

Options:
    -l <file>, --access-log <file>  access log file to parse.
    -f <format>, --log-format <format>  log format as specify in log_format directive.
    --no-follow  ngxtop default behavior is to ignore current lines in log
                     and only watch for new lines as they are written to the access log.
                     Use this flag to tell ngxtop to process the current content of the access log instead.
    -t <seconds>, --interval <seconds>  report interval when running in follow mode [default: 2.0]

    -g <var>, --group-by <var>  group by variable [default: request_path]
    -w <var>, --having <expr>  having clause [default: 1]
    -o <var>, --order-by <var>  order of output for default query [default: count]
    -n <number>, --limit <number>  limit the number of records included in report for top command [default: 10]
    -a <exp> ..., --a <exp> ...  add exp (must be aggregation exp: sum, avg, min, max, etc.) into output

    -v, --verbose  more verbose output
    -d, --debug  print every line and parsed record
    -h, --help  print this help message.
    --version  print version information.

    Advanced / experimental options:
    -c <file>, --config <file>  allow ngxtop to parse nginx config file for log format and location.
    -i <filter-expression>, --filter <filter-expression>  filter in, records satisfied given expression are processed.
    -p <filter-expression>, --pre-filter <filter-expression> in-filter expression to check in pre-parsing phase.

Samples

Default output

$ ngxtop
running for 411 seconds, 64332 records processed: 156.60 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|   64332 |         2775.251 | 61262 |  2994 |    71 |     5 |

Detailed:
| request_path                             |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|------------------------------------------+---------+------------------+-------+-------+-------+-------|
| /abc/xyz/xxxx                            |   20946 |          434.693 | 20935 |     0 |    11 |     0 |
| /xxxxx.json                              |    5633 |         1483.723 |  5633 |     0 |     0 |     0 |
| /xxxxx/xxx/xxxxxxxxxxxxx                 |    3629 |         6835.499 |  3626 |     0 |     3 |     0 |
| /xxxxx/xxx/xxxxxxxx                      |    3627 |        15971.885 |  3623 |     0 |     4 |     0 |
| /xxxxx/xxx/xxxxxxx                       |    3624 |         7830.236 |  3621 |     0 |     3 |     0 |
| /static/js/minified/utils.min.js         |    3031 |         1781.155 |  2104 |   927 |     0 |     0 |
| /static/js/minified/xxxxxxx.min.v1.js    |    2889 |         2210.235 |  2068 |   821 |     0 |     0 |
| /static/tracking/js/xxxxxxxx.js          |    2594 |         1325.681 |  1927 |   667 |     0 |     0 |
| /xxxxx/xxx.html                          |    2521 |          573.597 |  2520 |     0 |     1 |     0 |
| /xxxxx/xxxx.json                         |    1840 |          800.542 |  1839 |     0 |     1 |     0 |

View top source IPs of clients

$ ngxtop top remote_addr
running for 20 seconds, 3215 records processed: 159.62 req/sec

top remote_addr
| remote_addr     |   count |
|-----------------+---------|
| 118.173.177.161 |      20 |
| 110.78.145.3    |      16 |
| 171.7.153.7     |      16 |
| 180.183.67.155  |      16 |
| 183.89.65.9     |      16 |
| 202.28.182.5    |      16 |
| 1.47.170.12     |      15 |
| 119.46.184.2    |      15 |
| 125.26.135.219  |      15 |
| 125.26.213.203  |      15 |

List 4xx or 5xx responses together with HTTP referer

$ ngxtop -i 'status >= 400' print request status http_referer
running for 2 seconds, 28 records processed: 13.95 req/sec

request, status, http_referer:
| request   |   status | http_referer   |
|-----------+----------+----------------|
| -         |      400 | -              |

Parse apache log from remote server with common format

$ ssh user@remote_server tail -f /var/log/apache2/access.log | ngxtop -f common
running for 20 seconds, 1068 records processed: 53.01 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|    1068 |        28026.763 |  1029 |    20 |    19 |     0 |

Detailed:
| request_path                             |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|------------------------------------------+---------+------------------+-------+-------+-------+-------|
| /xxxxxxxxxx                              |     199 |        55150.402 |   199 |     0 |     0 |     0 |
| /xxxxxxxx/xxxxx                          |     167 |        47591.826 |   167 |     0 |     0 |     0 |
| /xxxxxxxxxxxxx/xxxxxx                    |      25 |         7432.200 |    25 |     0 |     0 |     0 |
| /xxxx/xxxxx/x/xxxxxxxxxxxxx/xxxxxxx      |      22 |          698.727 |    22 |     0 |     0 |     0 |
| /xxxx/xxxxx/x/xxxxxxxxxxxxx/xxxxxx       |      19 |         7431.632 |    19 |     0 |     0 |     0 |
| /xxxxx/xxxxx/                            |      18 |         7840.889 |    18 |     0 |     0 |     0 |
| /xxxxxxxx/xxxxxxxxxxxxxxxxx              |      15 |         7356.000 |    15 |     0 |     0 |     0 |
| /xxxxxxxxxxx/xxxxxxxx                    |      15 |         9978.800 |    15 |     0 |     0 |     0 |
| /xxxxx/                                  |      14 |            0.000 |     0 |    14 |     0 |     0 |
| /xxxxxxxxxx/xxxxxxxx/xxxxx               |      13 |        20530.154 |    13 |     0 |     0 |     0 |
Comments
  • ImportError: No module named 'config_parser'

    ImportError: No module named 'config_parser'

    After installing when I am trying to run ngxtop its giving me error

    Traceback (most recent call last):
      File "/usr/bin/ngxtop", line 7, in <module>
        from ngxtop.ngxtop import main
      File "/usr/lib64/python3.4/site-packages/ngxtop/ngxtop.py", line 77, in <module>
        from config_parser import detect_log_config, detect_config_path, extract_variables, build_pattern
    ImportError: No module named 'config_parser'
    
    opened by wonderer007 8
  • Support for reading from stdin and dumping db to a file

    Support for reading from stdin and dumping db to a file

    If the only access you have to the log is via cat or tail from ssh connection, the script will need reading from stdin. And maybe you want backup in a file the data you're printing.

    opened by mcortizo 8
  • ngxtop no data gathering from access log

    ngxtop no data gathering from access log

    Can you help me what problem is? no data summary from access log

    ngxtop

    running for 42 seconds, 0 records processed: 0.00 req/sec

    Summary: | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx | |---------+------------------+-------+-------+-------+-------| | 0 || 0 | 0 | 0 | 0 |

    Detailed: | request_path | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx | |----------------+---------+------------------+-------+-------+-------+-------|246

    opened by jk28189 3
  • --access-log doesn't seem to be working

    --access-log doesn't seem to be working

    ngxtop -l /var/log/nginx/access.log
    Traceback (most recent call last):
      File "/home/adys/bin/ngxtop", line 437, in <module>
        main(args)
      File "/home/adys/bin/ngxtop", line 420, in main
        total = process_log(source, pattern, processor, arguments)
      File "/home/adys/bin/ngxtop", line 325, in process_log
        total = processor.process(records)
      File "/home/adys/bin/ngxtop", line 272, in process
        for r in records:
      File "/home/adys/bin/ngxtop", line 206, in add_field
        for item in dict_sequence:
      File "/home/adys/bin/ngxtop", line 196, in map_field
        for item in dict_sequence:
      File "/home/adys/bin/ngxtop", line 196, in map_field
        for item in dict_sequence:
      File "/home/adys/bin/ngxtop", line 206, in add_field
        for item in dict_sequence:
      File "/home/adys/bin/ngxtop", line 206, in add_field
        for item in dict_sequence:
      File "/home/adys/bin/ngxtop", line 196, in map_field
        for item in dict_sequence:
      File "/home/adys/bin/ngxtop", line 245, in <genexpr>
        records = (m.groupdict() for m in matches if m is not None)
      File "/home/adys/bin/ngxtop", line 244, in <genexpr>
        matches = (pattern.match(l) for l in lines)
      File "/home/adys/bin/ngxtop", line 181, in follow
        with open(the_file) as f:
    IOError: [Errno 2] No such file or directory: 'logs/access.log'
    
    opened by jleclanche 3
  • Error

    Error

    Hi

    $ ngxtop info Traceback (most recent call last): File "/usr/local/bin/ngxtop", line 6, in from ngxtop.ngxtop import main File "/usr/local/lib/python3.6/dist-packages/ngxtop/ngxtop.py", line 77, in from config_parser import detect_log_config, detect_config_path, extract_variables, build_pattern ModuleNotFoundError: No module named 'config_parser'

    What's wrong/missing?

    opened by matt-matt1 2
  • Any way to filter the request_path ?

    Any way to filter the request_path ?

    Hello

    Is possible that any way to filter the request_path? For example I want to see analyze the just ".css" files (hit count /4xx 3xx 2xx errors)

    Are there any parameter ?

    opened by maxiceptr 2
  • No matching distribution found for sqlite-devel

    No matching distribution found for sqlite-devel

    I run the that command via ssh yum -y install python-pip; pip install ngxtop sqlite-devel

    And it says:

    Installed: python-pip.noarch 0:7.1.0-1.el7
    Complete! /usr/lib/python2.7/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning You are using pip version 7.1.0, however version 7.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Collecting ngxtop /usr/lib/python2.7/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning Downloading ngxtop-0.0.2-py2.py3-none-any.whl Collecting sqlite-devel Could not find a version that satisfies the requirement sqlite-devel (from versions: ) No matching distribution found for sqlite-devel

    I think, a sqlite-devel problem.

    P.S: Operation system is centos 7 64bit

    opened by maxiceptr 2
  • ngxtop work with buffered access_log ?

    ngxtop work with buffered access_log ?

    Hi I am playing around with ngxtop for my Centmin Mod Nginx project https://gist.github.com/centminmod/d613128cdb9eed86c9ec and was wondering do ngxtop live stats work with buffered access_log files ?

    I can not see any live stats but when i use --no-follow i do see previously logged data

    thanks

    opened by centminmod 2
  • Support multiple fields in --group-by

    Support multiple fields in --group-by

    This allows --group-by to take a comma-separated list of field names, and processes the field list before passing it to the SQLProcessor. When displaying the results, it displays as two separate fields.

    opened by amyreese 2
  • Multiple group by columns

    Multiple group by columns

    I'd like to show requests by hostname and request path, so I tried:

    ngxtop -g http_host,request_path
    

    which gives me

    running for 8 seconds, 0 records processed: 0.00 req/sec
    
    Summary:
    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |---------+------------------+-------+-------+-------+-------|
    |       0 |                  |     0 |     0 |     0 |     0 |
    
    Detailed:
    
    Traceback (most recent call last):
      File "/usr/bin/ngxtop", line 9, in <module>
        load_entry_point('ngxtop==0.0.1', 'console_scripts', 'ngxtop')()
      File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 446, in main
        process(args)
      File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 429, in process
        total = process_log(source, pattern, processor, arguments)
      File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 334, in process_log
        total = processor.process(records)
      File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 282, in process
        cursor.execute(insert, r)
    sqlite3.OperationalError: no such column: request_path
    

    So then I tried nginxtop -g http_host -g request_path, which just gives me back the usage information. Grouping by either http_host or request_path works fine, it's just the combination together that I can't seem to get working.

    opened by alanpearce 2
  • Got error messages in centos 6.5

    Got error messages in centos 6.5

    Centos 6.5

    Python 2.6.6 (r266:84292, Nov 22 2013, 12:16:22)

    ./ngxtop.py
    Traceback (most recent call last):
      File "./ngxtop.py", line 440, in <module>
        main(args)
      File "./ngxtop.py", line 402, in main
        config = get_nginx_conf_path()
      File "./ngxtop.py", line 117, in get_nginx_conf_path
        proc = subprocess.Popen(['nginx', '-V'], stderr=subprocess.PIPE)
      File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
        errread, errwrite)
      File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
        raise child_exception
    OSError: [Errno 2] No such file or directory
    
    opened by jiajie999 2
  • [Looking for Help] Show all IPs that have accesses in the last 5 minutes

    [Looking for Help] Show all IPs that have accesses in the last 5 minutes

    Looking for some help to use ngxtop to show just the connections in the last 5 minutes and not all time.

    We're using this to gauge a live listener count for HLS streams. Using the following command works great, but can we make IPs drop off the list if there have been no requests in X amount of time?

    ngxtop -l /var/log/nginx/stream-HLS-access.log --group-by remote_addr

    opened by CFMCoromandel 0
  • go forward to ngxtop version 0.1.0.

    go forward to ngxtop version 0.1.0.

    1. add traff speed summery info
    2. add avg_request_time in each pannel
    3. modify: -g , --group-by group by variable [default: request_path,http_host,remote_addr]
    4. set version 0.1.0
    opened by pythonwood 1
  • Buffed logs

    Buffed logs

    Please add warning about buffered nginx access logs to readme. Because it's not obvious that buffered logs can make ngxtop displaying incorrect values.

    Thank you very much for such a good tool.

    opened by AlexanderMatveev 0
  • support inheritance log_format for child configs

    support inheritance log_format for child configs

    In case main nginx.conf uses include directive for child sites but log_format specified in nginx.conf tool couldn't capture log strings. It stops with Error: Incorrect format name set in config for access log file or produces nothing.

    This patch looks for log_format in main nginx.conf and applies it if needed.

    Known limitation: works only when site config is specified, not log.

    opened by IgorPyN 1
Releases(v0.0.3)
  • v0.0.3(Dec 21, 2020)

  • 0.0.2(Apr 9, 2014)

    Improvements:

    • Proper parsing of nginx configuration file with PyParsing: better detection of access log file and format, should work out of the box in most cases.
    • Get rid of reporting thread, use timer signal instead.
    • Use curses to display metrics instead of repeating clear screen.
    • Allow parsing of log from stdin and include 'combined' log format of Apache.
    Source code(tar.gz)
    Source code(zip)
Owner
Binh Le
Curious programmer, brave sysadmin. SRE @google, previously @cloudflare and @adatao.
Binh Le
ASGI middleware to record and emit timing metrics (to something like statsd)

timing-asgi This is a timing middleware for ASGI, useful for automatic instrumentation of ASGI endpoints. This was developed at GRID for use with our

Steinn Eldjárn Sigurðarson 99 Nov 21, 2022
Exports osu! user stats to prometheus metrics for a specified set of users

osu! to prometheus exporter This tool exports osu! user statistics into prometheus metrics for a specified set of user ids. Just copy the config.json.

Peter Oettig 1 Feb 24, 2022
GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.

GoAccess What is it? GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal on *nix systems or through y

Gerardo O. 15.6k Jan 2, 2023
System monitor - A python-based real-time system monitoring tool

System monitor A python-based real-time system monitoring tool Screenshots Installation Run My project with these commands pip install -r requiremen

Sachit Yadav 4 Feb 11, 2022
Yet Another Python Profiler, but this time thread&coroutine&greenlet aware.

Yappi Yet Another Python Profiler, but this time thread&coroutine&greenlet aware. Highlights Fast: Yappi is fast. It is completely written in C and lo

Sümer Cip 1k Jan 1, 2023
Nginx UI allows you to access and modify the nginx configurations files without cli.

nginx ui Table of Contents nginx ui Introduction Setup Example Docker UI Authentication Configure the auth file Configure nginx Introduction We use ng

David Schenk 4.3k Dec 31, 2022
Pytorch Lightning 1.2k Jan 6, 2023
Metrics-advisor - Analyze reshaped metrics from TiDB cluster Prometheus and give some advice about anomalies and correlation.

metrics-advisor Analyze reshaped metrics from TiDB cluster Prometheus and give some advice about anomalies and correlation. Team freedeaths mashenjun

null 3 Jan 7, 2022
FAIR Enough Metrics is an API for various FAIR Metrics Tests, written in python

☑️ FAIR Enough metrics for research FAIR Enough Metrics is an API for various FAIR Metrics Tests, written in python, conforming to the specifications

Maastricht University IDS 3 Jul 6, 2022
A server shell for you to play with Powered by Django + Nginx + Postgres + Bootstrap + Celery.

A server shell for you to play with Powered by Django + Nginx + Postgres + Bootstrap + Celery.

Mengting Song 1 Jan 10, 2022
Real-Time-Student-Attendence-System - Real Time Student Attendence System

Real-Time-Student-Attendence-System The Student Attendance Management System Pro

Rounak Das 1 Feb 15, 2022
Flask + Docker + Nginx + Gunicorn + MySQL + Factory Method Pattern

This Flask project is reusable and also an example of how to merge Flask, Docker, Nginx, Gunicorn, MySQL, new: Flask-RESTX, Factory Method design pattern, and other optional dependencies such as Dynaconf, Marshmallow, SQLAlchemy, Faker, PyMySQL, Pytest, etc... which are installed inside the virtual environment "env_flask".

Facundo Padilla 19 Jul 23, 2022
Ubuntu env build; Nginx build; DB build;

Deploy 介绍 Deploy related scripts bitnami Dependencies Ubuntu openssl envsubst docker v18.06.3 docker-compose init base env upload https://gitlab-runn

Colin(liuji) 10 Dec 1, 2021
Dockerizing Django with Postgres, Gunicorn, Nginx and Certbot. A fully Django starter project.

Dockerizing Django with Postgres, Gunicorn, Nginx and Certbot ?? Features A Django stater project with fully basic requirements for a production-ready

null 8 Jun 27, 2022
基于Pytorch的脚手架项目,Celery+FastAPI+Gunicorn+Nginx+Supervisor实现服务部署,支持Docker发布

cookiecutter-pytorch-fastapi 基于Pytorch的 脚手架项目 按规范添加推理函数即可实现Celery+FastAPI+Gunicorn+Nginx+Supervisor+Docker的快速部署 Requirements Python >= 3.6 with pip in

null 17 Dec 23, 2022
Command line parser for common log format (Nginx default).

Command line parser for common log format (Nginx default).

Lucian Marin 138 Dec 19, 2022
docker-compose uWSGI nginx flask

docker-compose uWSGI nginx flask Note that this was tested on CentOS 7 Usage sudo yum install docker

Abdolkarim Saeedi 3 Sep 11, 2022
Log processor for nginx or apache that extracts user and user sessions and calculates other types of useful data for bot detection or traffic analysis

Log processor for nginx or apache that extracts user and user sessions and calculates other types of useful data for bot detection or traffic analysis

David Puerta Martín 1 Nov 11, 2021