Small toolkit for python multiprocessing logging to file

Qishuai

Last update: Nov 10, 2021

Related tags

Logging mp-socket-logging

Overview

Small Toolkit for Python Multiprocessing Logging

This is a small toolkit for solving unsafe python mutliprocess logging (file logging and rotation)

Install

Only for Python 3.5+

pip install mp-socket-logging

Features

                            (child process)
                           +---------------+
             +<------------+     client    |
 +-------+-+-+             +---------------+
 |           |              (child process)
 +  Socket   |             +---------------+
 |           +<------------+     client    |
 +---------+-+             +---------------+
       |
       |                 (main process)
       |               +---------------+          +--------------+
       +-------------->+     server    +--------->+   log file   +
                       +---------------+          +--------------+

The package uses unix socket for communication between processes.
I/O operations will all be done by server in main process.
Client Class itself is a logging.Handler which will be added to user's logger inside child process. Client handler will do the formatting and sending the logs to server side through socket.
Inside child process, user can do logging as normal with the logger.

Quick Start

start a server before in the main process

server_handler = ServerHandler(LOG_FILE, maxBytes=MAX_BYTES)
server_handler.setLevel(logging.INFO)
server = Server(server_handler, batch_size=BATCH_SIZE)
server.start()

register the client to your logger inside multiprocessing worker for sending logs to server through socket
you can define your own formatter to add to the handler (otherwise a default formatter will be used)

client_logger = logging.getLogger("client_logger")
formatter = logging.Formatter(
    "%(asctime)s - %(name)s - %(filename)s- [%(levelname)s]: %(message)s"
)
register_handler(client_logger, logging_level=logging.INFO, formatter=formatter)
client_logger.setLevel(logging.INFO)

A Python package which supports global logfmt formatted logging.

Python Logfmter A Python package which supports global logfmt formatted logging. Install $ pip install logfmter Usage Before integrating this library,

15 Dec 29, 2022

Stand-alone parser for User Access Logging from Server 2012 and newer systems

KStrike Stand-alone parser for User Access Logging from Server 2012 and newer systems BriMor Labs KStrike This script will parse data from the User Ac

69 Nov 1, 2022

Logging system for the TPC software.

tpc_logger Logging system for the TPC software. The TPC Logger class provides a singleton for logging information within C++ code or in the python API

1 Jan 10, 2022

Outlog it's a library to make logging a simple task

outlog Outlog it's a library to make logging a simple task!. I'm a lazy python user, the times that i do logging on my apps it's hard to do, a lot of

2 Mar 5, 2022

metovlogs is a very simple logging library

metovlogs is a very simple logging library. Setup is one line, then you can use it as a drop-in print replacement. Sane and useful log format out of the box. Best for small or early projects.

1 Mar 1, 2022

A small utility to pretty-print Python tracebacks. ⛺

TBVaccine TBVaccine is a utility that pretty-prints Python tracebacks. It automatically highlights lines you care about and deemphasizes lines you don

365 Nov 11, 2022

Track Nano accounts and notify via log file or email

nano-address-notifier Track accounts and notify via log file or email Required python libs

4 Nov 8, 2021

Pretty-print tabular data in Python, a library and a command-line utility. Repository migrated from bitbucket.org/astanin/python-tabulate.

python-tabulate Pretty-print tabular data in Python, a library and a command-line utility. The main use cases of the library are: printing small table

1.5k Jan 6, 2023

Progressbar 2 - A progress bar for Python 2 and Python 3 - "pip install progressbar2"

Text progress bar library for Python. Travis status: Coverage: Install The package can be installed through pip (this is the recommended method): pip

795 Dec 18, 2022

Releases(v0.1.1)

v0.1.1(Nov 10, 2021)
What's Changed

add verbose and update version by @FerdinandZhong in https://github.com/FerdinandZhong/mp-socket-logging/pull/6

Full Changelog: https://github.com/FerdinandZhong/mp-socket-logging/compare/v0.1.0...v0.1.1
Source code(tar.gz)
Source code(zip)
v0.1.0(Oct 19, 2021)
What's Changed

Develop the package

Add auto-testing workflow

Add brief introduction

New Contributors

@FerdinandZhong

Full Changelog: https://github.com/FerdinandZhong/mp-socket-logging/commits/v0.1.0
Source code(tar.gz)
Source code(zip)