Json Formatter for the standard python logger

Related tags

Logging python logging
Overview

Build Status License Version

Overview

This library is provided to allow standard python logging to output log data as json objects. With JSON we can make our logs more readable by machines and we can stop writing custom parsers for syslog type records.

News

Hi, I see this package is quiet alive and I am sorry for ignoring it so long. I will be stepping up my maintenance of this package so please allow me a week to get things back in order (and most likely a new minor version) and i'll post and update here once I am caught up.

Installing

Pip:

pip install python-json-logger

Pypi:

https://pypi.python.org/pypi/python-json-logger

Manual:

python setup.py install

Usage

Integrating with Python's logging framework

Json outputs are provided by the JsonFormatter logging formatter. You can add the custom formatter like below:

Please note: version 0.1.0 has changed the import structure, please update to the following example for proper importing

    import logging
    from pythonjsonlogger import jsonlogger

    logger = logging.getLogger()

    logHandler = logging.StreamHandler()
    formatter = jsonlogger.JsonFormatter()
    logHandler.setFormatter(formatter)
    logger.addHandler(logHandler)

Customizing fields

The fmt parser can also be overidden if you want to have required fields that differ from the default of just message.

These two invocations are equivalent:

class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def parse(self):
        return self._fmt.split(';')

formatter = CustomJsonFormatter('one;two')

# is equivalent to:

formatter = jsonlogger.JsonFormatter('%(one)s %(two)s')

You can also add extra fields to your json output by specifying a dict in place of message, as well as by specifying an extra={} argument.

Contents of these dictionaries will be added at the root level of the entry and may override basic fields.

You can also use the add_fields method to add to or generally normalize the set of default set of fields, it is called for every log event. For example, to unify default fields with those provided by structlog you could do something like this:

class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, log_record, record, message_dict):
        super(CustomJsonFormatter, self).add_fields(log_record, record, message_dict)
        if not log_record.get('timestamp'):
            # this doesn't use record.created, so it is slightly off
            now = datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S.%fZ')
            log_record['timestamp'] = now
        if log_record.get('level'):
            log_record['level'] = log_record['level'].upper()
        else:
            log_record['level'] = record.levelname

formatter = CustomJsonFormatter('%(timestamp)s %(level)s %(name)s %(message)s')

Items added to the log record will be included in every log message, no matter what the format requires.

Adding custom object serialization

For custom handling of object serialization you can specify default json object translator or provide a custom encoder

def json_translate(obj):
    if isinstance(obj, MyClass):
        return {"special": obj.special}

formatter = jsonlogger.JsonFormatter(json_default=json_translate,
                                     json_encoder=json.JSONEncoder)
logHandler.setFormatter(formatter)

logger.info({"special": "value", "run": 12})
logger.info("classic message", extra={"special": "value", "run": 12})

Using a Config File

To use the module with a config file using the fileConfig function, use the class pythonjsonlogger.jsonlogger.JsonFormatter. Here is a sample config file.

[loggers]
keys = root,custom

[logger_root]
handlers =

[logger_custom]
level = INFO
handlers = custom
qualname = custom

[handlers]
keys = custom

[handler_custom]
class = StreamHandler
level = INFO
formatter = json
args = (sys.stdout,)

[formatters]
keys = json

[formatter_json]
format = %(message)s
class = pythonjsonlogger.jsonlogger.JsonFormatter

Example Output

Sample JSON with a full formatter (basically the log message from the unit test). Every log message will appear on 1 line like a typical logger.

{
    "threadName": "MainThread",
    "name": "root",
    "thread": 140735202359648,
    "created": 1336281068.506248,
    "process": 41937,
    "processName": "MainProcess",
    "relativeCreated": 9.100914001464844,
    "module": "tests",
    "funcName": "testFormatKeys",
    "levelno": 20,
    "msecs": 506.24799728393555,
    "pathname": "tests/tests.py",
    "lineno": 60,
    "asctime": ["12-05-05 22:11:08,506248"],
    "message": "testing logging format",
    "filename": "tests.py",
    "levelname": "INFO",
    "special": "value",
    "run": 12
}

External Examples

Comments
  • Add PEP 561 marker

    Add PEP 561 marker

    • Add a PEP 561 marker
    • Add basic mypy configuration
    • Run mypy from tox
    • Minor code modifications to make mypy happy

    This is intended to be a relatively minimal change just to enable basic type checking. Future changes should fill out the type annotations more and enable stricter mypy configuration (set all mypy options to 'true').

    This change is a small step towards https://github.com/madzak/python-json-logger/issues/118

    Note that the marker is intentionally not shipped via setup.py -- the annotations need more cleanup first (turn on a few key mypy config options, e.g. no-untyped-defs).

    opened by bringhurst 11
  • Jsonformatter formats the newline character

    Jsonformatter formats the newline character

    Jsonformatter will not format the newline character that is inside the message fields too.

    Ex: without this commit { "rid": "6c0320a4-ed87-4d0a-a668-b7ceef94eecf", "asctime": "2021-03-23 14:13:50,151", "filename": "", "module": "", "funcName": "", "lineno": 16, "levelname": "INFO", "message": "log line 1\nlog line 2" }

    After this commit { "rid": "6c0320a4-ed87-4d0a-a668-b7ceef94eecf", "asctime": "2021-03-23 14:13:50,151", "filename": "", "module": "", "funcName": "", "lineno": 16, "levelname": "INFO", "message": "log line 1 log line 2" }

    opened by anandtripathi5 7
  • Python 2.6 test fix and travis ci

    Python 2.6 test fix and travis ci

    This encompasses #39 and #40 with the "addition" of removing Python 2.6 from the list of allowed failures. Merging this in will close those too.

    opened by nelsonjchen 7
  • No StackTrace info on the log

    No StackTrace info on the log

    Hello.

    I'm trying to use this module to format django's log. The problem I'm having is that the full message of the stack trace when there's an exception, including the actual Exception name and message, is not showing up in my log files.

    Any idea for how I might fix this?

    opened by hitokiri82 5
  • Drop Python 3.5 support and pypy 3.6 support

    Drop Python 3.5 support and pypy 3.6 support

    To allow for typing annotations and enforcement of typing annotations, we need to:

    • Drop Python 3.5 support since it doesn't support the syntax
    • Drop pypy 3.6 support since it doesn't support mypy

    See https://github.com/madzak/python-json-logger/pull/129#issuecomment-1030708627 for additional background.

    opened by bringhurst 4
  • Update examples and tests for Python 3.8 compatibility

    Update examples and tests for Python 3.8 compatibility

    See #86 See #74

    This PR fixes up a couple of small issues encountered when upgrading to Python 3.8:

    • updates examples in README.md to be compatible with Python 3.8
    • updates test code to match the formatting example provided in the readme
    • adds Python 3.8 testing to Travis and tox
    • marks the package as Python 3.8 compatible in setup.py
    opened by tommilligan 4
  • Invalid format '(levelname) (name) (message)' for '%' style

    Invalid format '(levelname) (name) (message)' for '%' style

    It seems like with Python 3.8 the format string needs to be specified differently.

    I have this small test setup:

    This is test.py:

    import os
    import logging
    from pythonjsonlogger import jsonlogger
    
    logger = logging.getLogger()
    logHandler = logging.StreamHandler()
    formatter = jsonlogger.JsonFormatter("(levelname) (name) (message)")
    logHandler.setFormatter(formatter)
    logger.handlers = [logHandler]
    logger.setLevel(logging.INFO)
    
    logger.info("Hello world")
    

    and this Dockerfile:

    FROM python:3.7
    RUN pip install python-json-logger
    COPY test.py .
    CMD ["python", "test.py"]
    

    If I run docker build -t test . && docker run -it test it works as expected, the output is {"levelname": "INFO", "name": "root", "message": "Hello world"}.

    Under Python 3.8 this results in ValueError: Invalid format '(levelname) (name) (message)' for '%' style. I have to change the format line to read formatter = jsonlogger.JsonFormatter("%(levelname) %(name) %(message)"). I'm not sure if this is expected, but it might (at least) warrant an update of the documentation.

    opened by stchris 4
  • How to append logs in the current dict ?

    How to append logs in the current dict ?

    Thanks for the nice library ! I tried your example which is given in the readme section. It's working fine.

    My problem is I am using logging.FileHandler('reuben.json') to write the dump in a file. What happening is every time a log message is entering as a separate entry instead of an item of a list or dictionary.

    Is it possible to create the json in the given below format :

    log.error({"host": "host1", "message": "Failed to install"})
    log.info({"host": "host2", "message": "successfully installed"})
    

    json :

    [{"host": "host1", "message": "Failed to install"}, {"host": "host2", "message": "successfully installed"}]
    
    opened by reubenur-rahman 4
  • Allow editing of log_record in two possible ways

    Allow editing of log_record in two possible ways

    One more thing related to https://github.com/madzak/python-json-logger/issues/16: I just noticed that while process_log_record returns the log_record, that returned value isn't actually stored in a variable. By making the change in this commit, the user can use two approaches in process_log_record in a subclass to modify the log_record:

    • modify the given log_record in process_log_record (in-place) and return that,
    • or, use the given log_record to compute an entirely new dictionary instance and return that instead.
    opened by svisser 4
  • Release 2.0

    Release 2.0

    Hello @madzak.

    The readme says:

    NOTICE: Python2.7 is only supported up to v0.1.11. The next release, v2.0, will support Python3 only

    Python 2 is unsupported since 1st Jan 2020. Is there any roadmap for 2.0 or could you just make a new release supporting only Python3?

    Thanks!

    opened by PabloCastellano 3
  • Fix/setuptools

    Fix/setuptools

    opened by orsinium 3
  • 2.0.4: pytest is not able to find units

    2.0.4: pytest is not able to find units

    + /usr/bin/pytest -ra
    =========================================================================== test session starts ============================================================================
    platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
    rootdir: /home/tkloczko/rpmbuild/BUILD/python-json-logger-2.0.4
    collected 0 items
    
    ========================================================================== no tests ran in 0.01s ===========================================================================
    

    However

    + /usr/bin/pytest -ra tests/tests.py
    =========================================================================== test session starts ============================================================================
    platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
    rootdir: /home/tkloczko/rpmbuild/BUILD/python-json-logger-2.0.4
    collected 16 items
    
    tests/tests.py ................                                                                                                                                      [100%]
    
    ============================================================================ 16 passed in 0.03s ============================================================================
    

    pytest automacically scans tests_*.py files so tests/tests.py should be renamed to fix this.

    opened by kloczek 0
  • Change format of message to Message

    Change format of message to Message

    Hello, this isn't an issue, but wanted to know if there was an easier way to call Message instead of message or do we have to go the CustomJsonFormatter class route? For example, how to change the message here

    formatter = jsonlogger.JsonFormatter("%(asctime)s %(filename)s %(module)s %(funcName)s %(lineno)s %(message)s")
    

    to Message, because changing it directly in this line of code wouldn't work?

    Thank you.

    opened by sharathsridhar 0
  • dict as object of message

    dict as object of message

    With this patch we can do something like this:

        formatter = jsonlogger.JsonFormatter(dict_as_object=True)
        # ...
        my_dict = {
            "result": 0,
            "text": "some text",
        }
        logger.info("Just text")
        logger.info(my_dict)
    

    Output:

    {
      "message": "Just text"
    }
    {
      "message": {
        "result": 0,
        "text": "some text",
      }
    }
    
    opened by DmitruMoth 0
  • Drop support of old python versions

    Drop support of old python versions

    While browsing the code, I realized that some code is written with python2 in mind (especially the test). python2 is not supported since January 2020 (https://www.python.org/doc/sunset-python-2/).

    I'd also suggest changing the packaging of this library to drop the support of python3.5 (EOL 2020-09-13) and python3.6 (EOL 2021-12-23) (source: https://endoflife.date/python)

    Some guidance can be found here: https://packaging.python.org/en/latest/guides/dropping-older-python-versions/

    opened by lopagela 1
  • How to include all available fields in a log record + some custom fields?

    How to include all available fields in a log record + some custom fields?

    Currently, I am writing a custom formatter to include all fields with the intention of adding some of my own later.

    I have two sets of questions

    Q1 : Is this the correct way? i.e. subclass the formatter and than copying field by field over ? or am I going about it the wrong way ?

    Ultimately, I will want to include most of the default fields plus I am going to add some custom in-house fields.

    class CustomJsonFormatter(jsonlogger.JsonFormatter):
        
        def add_fields(self, log_record, record, message_dict):
            import datetime
            super(CustomJsonFormatter, self).add_fields(log_record, record, message_dict)
    
            log_record['args'] = record.args
            # log_record['asctime'] = record.asctime
            log_record['created'] = record.created
            log_record['exc_info'] = record.exc_info
            log_record['exc_text'] = record.exc_text
            log_record['filename'] = record.filename
            log_record['funcName'] = record.funcName
            log_record['levelname'] = record.levelname
            log_record['levelno'] = record.levelno
            log_record['lineno'] = record.lineno
            log_record['module'] = record.module
            log_record['msecs'] = record.msecs
            log_record['message'] = record.message
            log_record['msg'] = record.msg
            log_record['name'] = record.name
            log_record['pathname'] = record.pathname
            log_record['process'] = record.process
            log_record['processName'] = record.processName
            log_record['relativeCreated'] = record.relativeCreated
            log_record['stack_info'] = record.stack_info
            log_record['thread'] = record.thread
            log_record['threadName'] = record.threadName
    

    Q2 : What if I want most but not all the default fields, is there some pythonic way to do that ?

    opened by nyue 0
Releases(v2.0.4)
  • v2.0.4(Jul 11, 2022)

  • v2.0.3(Jul 8, 2022)

    Added

    • Add PEP 561 marker/basic mypy configuration. - @bringhurst
    • Workaround logging.LogRecord.msg type of string. - @bringhurst

    Changed

    • Changed a link archive of the reference page in case it's down. - @ahonnecke
    • Removed unnecessary try-except around OrderedDict usage - @sozofaan
    • Update documentation link to json module + use https - @deronnax
    • Dropped 3.5 support. - @bringhurst
    Source code(tar.gz)
    Source code(zip)
  • v2.0.2(Jul 27, 2021)

    Added

    • Officially supporting 3.9 - @felixonmars.
    • You can now add static fields to log objects - @cosimomeli.

    Changed

    • Dropped 3.4 support.
    • Dropped Travis CI for Github Actions.
    • Wheel should build for python 3 instead of just 3.4 now.
    Source code(tar.gz)
    Source code(zip)
  • v2.0.1(Oct 12, 2020)

  • v0.1.5(Jun 12, 2016)

Owner
Zakaria Zajac
Zakaria Zajac
Key Logger - Key Logger using Python

Key_Logger Key Logger using Python This is the basic Keylogger that i have made

Mudit Sinha 2 Jan 15, 2022
Discord-Image-Logger - Discord Image Logger With Python

Discord-Image-Logger A exploit I found in discord. Working as of now. Explanatio

null 111 Dec 31, 2022
A Python library that tees the standard output & standard error from the current process to files on disk, while preserving terminal semantics

A Python library that tees the standard output & standard error from the current process to files on disk, while preserving terminal semantics (so breakpoint(), etc work as normal)

Greg Brockman 7 Nov 30, 2022
A colored formatter for the python logging module

Log formatting with colors! colorlog.ColoredFormatter is a formatter for use with Python's logging module that outputs records using terminal colors.

Sam Clements 778 Dec 26, 2022
A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics.

A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics, originally intended for keyboard layout optimization.

Ga68 56 Jan 3, 2023
This is a key logger based in python which when executed records all the keystrokes of the system it has been executed on .

This is a key logger based in python which when executed records all the keystrokes of the system it has been executed on

Purbayan Majumder 0 Mar 28, 2022
Fancy console logger and wise assistant within your python projects

Fancy console logger and wise assistant within your python projects. Made to save tons of hours for common routines.

BoB 5 Apr 1, 2022
Ultimate Logger - A Discord bot that logs lots of events in a channel written in python

Ultimate Logger - A Discord bot that logs lots of events in a channel written in python

Luca 2 Mar 27, 2022
Multi-processing capable print-like logger for Python

MPLogger Multi-processing capable print-like logger for Python Requirements and Installation Python 3.8+ is required Pip pip install mplogger Manual P

Eötvös Loránd University Department of Digital Humanities 1 Jan 28, 2022
Token Logger with python

Oxy Token Stealer Features Grabs discord tokens Grabs chrome passwords Grabs edge passwords Nothing else, I don't feel like releasing full on malware

oxy 1 Feb 12, 2022
This open-source python3 script is a builder to the very popular token logger that is on my github that many people use.

Discord-Logger-Builder This open-source python3 script is a builder to the very popular token logger that is on my github that many people use. This i

Local 4 Nov 17, 2021
giving — the reactive logger

giving is a simple, magical library that lets you log or "give" arbitrary data throughout a program and then process it as an event stream.

Olivier Breuleux 0 May 24, 2022
Fuzzy-logger - Fuzzy project is here Log all your pc's actions Simple and free to use Security of datas !

Fuzzy-logger - ➡️⭐ Fuzzy ⭐ project is here ! ➡️ Log all your pc's actions ! ➡️ Simple and free to use ➡️ Security of datas !

natrix_dev 2 Oct 2, 2022
A watchdog and logger to Discord for hosting ScPrime servers.

ScpDog A watchdog and logger to Discord for hosting ScPrime servers. Designed to work on Linux servers. This is only capable of sending the logs from

Keagan Landfried 3 Jan 10, 2022
Translating symbolicated Apple JSON format crash log into our old friends :)

CrashTranslation Translating symbolicated Apple JSON format crash log into our old friends :) Usage python3 translation.py -i {input_sybolicated_json_

Kam-To 11 May 16, 2022
Pretty-print tabular data in Python, a library and a command-line utility. Repository migrated from bitbucket.org/astanin/python-tabulate.

python-tabulate Pretty-print tabular data in Python, a library and a command-line utility. The main use cases of the library are: printing small table

Sergey Astanin 1.5k Jan 6, 2023
Progressbar 2 - A progress bar for Python 2 and Python 3 - "pip install progressbar2"

Text progress bar library for Python. Travis status: Coverage: Install The package can be installed through pip (this is the recommended method): pip

Rick van Hattem 795 Dec 18, 2022
Python logging made (stupidly) simple

Loguru is a library which aims to bring enjoyable logging in Python. Did you ever feel lazy about configuring a logger and used print() instead?... I

null 13.7k Jan 2, 2023
The new Python SDK for Sentry.io

sentry-python - Sentry SDK for Python This is the next line of the Python SDK for Sentry, intended to replace the raven package on PyPI. from sentry_s

Sentry 1.4k Dec 31, 2022