HiQ - A Modern Observability System

Overview

🦉 A Modern Observability System

Documentation Status CodeCov Github release lic

HiQ is a declarative, non-intrusive, dynamic and transparent tracking system for both monolithic application and distributed system. It brings the runtime information tracking and optimization to a new level without compromising with speed and system performance, or hiding any tracking overhead information. HiQ applies for both I/O bound and CPU bound applications.

To explain the four features, declarative means you can declare the things you want to track in a text file, which could be a json, yaml or even csv,and no need to change program code. Non-intrusive means HiQ doesn't requires to modify original python code. Dynamic means HiQ supports tracing metrics featuring at run time, which can be used for adaptive tracing. Transparent means HiQ provides the tracing overhead and doesn't hide it no matter it is huge or tiny.

In addition to latency tracking, HiQ provides memory, disk I/O and Network I/O tracking out of the box. The output can be saved in form of normal line by line log file, or HiQ tree, or span graph.

HiQ's philosophy is to decouple observability logic from business logic. We don't have to enter the black hole to observe it. Do you like the idea? Leave a ⭐ if you enjoy the project and welcome to say Hi to us on Slack 👋

Observability of Blackhole, NASA 2019

Installation

pip install hiq-python

Get Started

Let start with a simplest example by running HiQ against a simple monolithic python code 📄 main.py:

# this is the main.py python source code
import time

def func1():
    time.sleep(1.5)
    print("func1")
    func2()

def func2():
    time.sleep(2.5)
    print("func2")

def main():
    func1()

if __name__ == "__main__":
    main()

In this code, there is a simple chain of function calls: main() -> func1 -> func2.

Now we want to trace the functions without modifying its code. Let's run the following:

git clone https://github.com/oracle-samples/hiq.git
cd hiq/examples/quick_start
python main_driver.py

If everything is fine, you should be able to see the output like this:

HiQ Simplest Example

From the screenshot we can see the timestamp and the latency of each function:

main func1 func2 tracing overhead
latency(second) 4.0045 4.0044 2.5026 0.0000163

HiQ just traced the main.py file running without touching one line of its code.

Documentation

HTML: 🔗 HiQ Online Documents | PDF: Please check 🔗 HiQ User Guide.


Logging: https://hiq.readthedocs.io/en/latest/4_o_advanced.html#log-monkey-king
Tracing: https://hiq.readthedocs.io/en/latest/5_distributed.html

Metrics:

Streaming:

Jupyter NoteBook

Add Observability to PaddlePaddle (PaddleOCR)

Add Observability to Onnxruntime (AlexNet)

Examples

Please check 🔗 examples for usage examples.

Contributing

HiQ welcomes contributions from the community. Before submitting a pull request, please review our 🔗 contribution guide.

Security

Please consult the 🔗 security guide for our responsible security vulnerability disclosure process.

Change Log

dev

  • add non-intrusive auto instrumentation for flask: HiQFlaskLatencyOtel ( 🔗 example)

v1.0.2

  • add non-intrusive auto instrumentation for Onnxruntime, Paddlepaddle, PaddleOCR

License

Copyright (c) 2022 Oracle and/or its affiliates. Released under the Universal Permissive License v1.0 as shown at https://oss.oracle.com/licenses/upl/.

Presentation and Demos

You might also like...
Waydroid is a container-based approach to boot a full Android system on a regular GNU/Linux system like Ubuntu.

Waydroid is a container-based approach to boot a full Android system on a regular GNU/Linux system like Ubuntu.

System Design Assignments as part of Arpit's System Design Masterclass
System Design Assignments as part of Arpit's System Design Masterclass

System Design Assignments The repository contains a set of problem statements around Software Architecture and System Design as conducted by Arpit's S

A fast Python in-process signal/event dispatching system.

Blinker Blinker provides a fast dispatching system that allows any number of interested parties to subscribe to events, or "signals". Signal receivers

A simple but flexible plugin system for Python.

PluginBase PluginBase is a module for Python that enables the development of flexible plugin systems in Python. Step 1: from pluginbase import PluginB

Python based scripts for obtaining system information from Linux.

sysinfo Python based scripts for obtaining system information from Linux. Python2 and Python3 compatible Output in JSON format Simple scripts and exte

MiniJVM is simple java virtual machine written by python language, it can load class file from file system and run it.

MiniJVM MiniJVM是一款使用python编写的简易JVM,能够从本地加载class文件并且执行绝大多数指令。 支持的功能 1.从本地磁盘加载class并解析 2.支持绝大多数指令集的执行 3.支持虚拟机内存分区以及对象的创建 4.支持方法的调用和参数传递 5.支持静态代码块的初始化 不支

A minimalist production ready plugin system

pluggy - A minimalist production ready plugin system This is the core framework used by the pytest, tox, and devpi projects. Please read the docs to l

:art: Diagram as Code for prototyping cloud system architectures
:art: Diagram as Code for prototyping cloud system architectures

Diagrams Diagram as Code. Diagrams lets you draw the cloud system architecture in Python code. It was born for prototyping a new system architecture d

A simple but flexible plugin system for Python.

PluginBase PluginBase is a module for Python that enables the development of flexible plugin systems in Python. Step 1: from pluginbase import PluginB

Comments
  • Bump numpy from 1.19.5 to 1.21.0 in /hiq

    Bump numpy from 1.19.5 to 1.21.0 in /hiq

    Bumps numpy from 1.19.5 to 1.21.0.

    Release notes

    Sourced from numpy's releases.

    v1.21.0

    NumPy 1.21.0 Release Notes

    The NumPy 1.21.0 release highlights are

    • continued SIMD work covering more functions and platforms,
    • initial work on the new dtype infrastructure and casting,
    • universal2 wheels for Python 3.8 and Python 3.9 on Mac,
    • improved documentation,
    • improved annotations,
    • new PCG64DXSM bitgenerator for random numbers.

    In addition there are the usual large number of bug fixes and other improvements.

    The Python versions supported for this release are 3.7-3.9. Official support for Python 3.10 will be added when it is released.

    :warning: Warning: there are unresolved problems compiling NumPy 1.21.0 with gcc-11.1 .

    • Optimization level -O3 results in many wrong warnings when running the tests.
    • On some hardware NumPy will hang in an infinite loop.

    New functions

    Add PCG64DXSM BitGenerator

    Uses of the PCG64 BitGenerator in a massively-parallel context have been shown to have statistical weaknesses that were not apparent at the first release in numpy 1.17. Most users will never observe this weakness and are safe to continue to use PCG64. We have introduced a new PCG64DXSM BitGenerator that will eventually become the new default BitGenerator implementation used by default_rng in future releases. PCG64DXSM solves the statistical weakness while preserving the performance and the features of PCG64.

    See upgrading-pcg64 for more details.

    (gh-18906)

    Expired deprecations

    • The shape argument numpy.unravel_index cannot be passed as dims keyword argument anymore. (Was deprecated in NumPy 1.16.)

    ... (truncated)

    Commits
    • b235f9e Merge pull request #19283 from charris/prepare-1.21.0-release
    • 34aebc2 MAINT: Update 1.21.0-notes.rst
    • 493b64b MAINT: Update 1.21.0-changelog.rst
    • 07d7e72 MAINT: Remove accidentally created directory.
    • 032fca5 Merge pull request #19280 from charris/backport-19277
    • 7d25b81 BUG: Fix refcount leak in ResultType
    • fa5754e BUG: Add missing DECREF in new path
    • 61127bb Merge pull request #19268 from charris/backport-19264
    • 143d45f Merge pull request #19269 from charris/backport-19228
    • d80e473 BUG: Removed typing for == and != in dtypes
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
Owner
Oracle Sample Code
Oracle Sample Code
Oracle Sample Code
Tracing and Observability with OpenFaaS

Tracing and Observability with OpenFaaS Today we will walk through how to add OpenTracing or OpenTelemetry with Grafana's Tempo. For this walk-through

Lucas Roesler 8 Nov 17, 2022
A collection of modern themes for Tkinter TTK

ttkbootstrap A collection of modern flat themes inspired by Bootstrap. Also includes TTK Creator which allows you to easily create and use your own th

Israel Dryer 827 Jan 4, 2023
A modern Python build backend

trampolim A modern Python build backend. Features Task system, allowing to run arbitrary Python code during the build process (Planned) Easy to use CL

Filipe Laíns 39 Nov 8, 2022
A python program with an Objective-C GUI for building and booting OpenCore on both legacy and modern Macs

A python program with an Objective-C GUI for building and booting OpenCore on both legacy and modern Macs, see our in-depth Guide for more information.

dortania 4.7k Jan 2, 2023
A Modern Fetch Tool for Linux!

Ufetch A Modern Fetch Tool for Linux! Programming Language: Python IDE: Visual Studio Code Developed by Avishek Dutta If you get any kind of problem,

Avishek Dutta 7 Dec 12, 2021
Lightweight and Modern kernel for VK Bots

This is the kernel for creating VK Bots written in Python 3.9

Yrvijo 4 Nov 21, 2021
Convert Roman numerals to modern numerals and vice-versa

Roman Numeral Conversion Utilities This is a utility module for converting from and to Roman numerals. It supports numbers upto 3,999,999, using the v

Fictive Kin 1 Dec 17, 2021
A modern python module including many useful features that make discord bot programming extremely easy.

discord-super-utils Documentation Secondary Documentation A modern python module including many useful features that make discord bot programming extr

null 106 Dec 19, 2022
This Program Automates The Procces Of Adding Camos On Guns And Saving Them On Modern Warfare Guns

This Program Automates The Procces Of Adding Camos On Guns And Saving Them On Modern Warfare Guns

Flex Tools 6 May 26, 2022
A free and powerful system for awareness and research of the American judicial system.

CourtListener Started in 2009, CourtListener.com is the main initiative of Free Law Project. The goal of CourtListener.com is to provide high quality

Free Law Project 332 Dec 25, 2022