A Prometheus Python client library for asyncio-based applications

Overview
https://github.com/claws/aioprometheus/workflows/Python%20Package%20Workflow/badge.svg?branch=master https://readthedocs.org/projects/aioprometheus/badge/?version=latest

aioprometheus

aioprometheus is a Prometheus Python client library for asyncio-based applications. It provides metrics collection and serving capabilities, supports multiple data formats and pushing metrics to a gateway.

The project documentation can be found on ReadTheDocs.

Install

$ pip install aioprometheus

A Prometheus Push Gateway client and ASGI service are also included, but their dependencies are not installed by default. You can install them alongside aioprometheus by running:

$ pip install aioprometheus[aiohttp]

Prometheus 2.0 removed support for the binary protocol, so in version 20.0.0 the dependency on prometheus-metrics-proto, which provides binary support, is now optional. If you want binary response support, for use with an older Prometheus, you will need to specify the 'binary' optional extra:

$ pip install aioprometheus[binary]

Multiple optional dependencies can be listed at once, such as:

$ pip install aioprometheus[aiohttp,binary]

Example

The example below shows a single Counter metric collector being created and exposed via the optional aiohttp service endpoint.

#!/usr/bin/env python
"""
This example demonstrates how a single Counter metric collector can be created
and exposed via a HTTP endpoint.
"""
import asyncio
import socket
from aioprometheus import Counter, Service


if __name__ == "__main__":

    async def main(svr: Service) -> None:

        events_counter = Counter(
            "events", "Number of events.", const_labels={"host": socket.gethostname()}
        )
        svr.register(events_counter)
        await svr.start(addr="127.0.0.1", port=5000)
        print(f"Serving prometheus metrics on: {svr.metrics_url}")

        # Now start another coroutine to periodically update a metric to
        # simulate the application making some progress.
        async def updater(c: Counter):
            while True:
                c.inc({"kind": "timer_expiry"})
                await asyncio.sleep(1.0)

        await updater(events_counter)

    loop = asyncio.get_event_loop()
    svr = Service()
    try:
        loop.run_until_complete(main(svr))
    except KeyboardInterrupt:
        pass
    finally:
        loop.run_until_complete(svr.stop())
    loop.close()

In this simple example the counter metric is tracking the number of while loop iterations executed by the updater coroutine. In a realistic application a metric might track the number of requests, etc.

Following typical asyncio usage, an event loop is instantiated first then a metrics service is instantiated. The metrics service is responsible for managing metric collectors and responding to metrics requests.

The service accepts various arguments such as the interface and port to bind to. A collector registry is used within the service to hold metrics collectors that will be exposed by the service. The service will create a new collector registry if one is not passed in.

A counter metric is created and registered with the service. The service is started and then a coroutine is started to periodically update the metric to simulate progress.

This example and demonstration requires some optional extra to be installed.

$ pip install aioprometheus[aiohttp,binary]

The example script can then be run using:

(venv) $ cd examples
(venv) $ python simple-example.py
Serving prometheus metrics on: http://127.0.0.1:5000/metrics

In another terminal fetch the metrics using the curl command line tool to verify they can be retrieved by Prometheus server.

By default metrics will be returned in plan text format.

$ curl http://127.0.0.1:5000/metrics
# HELP events Number of events.
# TYPE events counter
events{host="alpha",kind="timer_expiry"} 33

Similarly, you can request metrics in binary format, though the output will be hard to read on the command line.

$ curl http://127.0.0.1:5000/metrics -H "ACCEPT: application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily; encoding=delimited"

The metrics service also responds to requests sent to its / route. The response is simple HTML. This route can be useful as a Kubernetes /healthz style health indicator as it does not incur any overhead within the service to serialize a full metrics response.

$ curl http://127.0.0.1:5000/
<html><body><a href='/metrics'>metrics</a></body></html>

The aioprometheus package provides a number of convenience decorator functions that can assist with updating metrics.

The examples directory contains many examples showing how to use the aioprometheus package. The app-example.py file will likely be of interest as it provides a more representative application example than the simple example shown above.

Examples in the examples/frameworks directory show how aioprometheus can be used within various web application frameworks without needing to create a separate aioprometheus.Service endpoint to handle metrics. The FastAPI example is shown below.

#!/usr/bin/env python
"""
Sometimes you may not want to expose Prometheus metrics from a dedicated
Prometheus metrics server but instead want to use an existing web framework.

This example uses the registry from the aioprometheus package to add
Prometheus instrumentation to a FastAPI application. In this example a registry
and a counter metric is instantiated and gets updated whenever the "/" route
is accessed. A '/metrics' route is added to the application using the standard
web framework method. The metrics route renders Prometheus metrics into the
appropriate format.

Run:

  $ pip install fastapi uvicorn
  $ uvicorn fastapi_example:app

"""

from aioprometheus import render, Counter, Registry
from fastapi import FastAPI, Header, Response
from typing import List


app = FastAPI()
app.registry = Registry()
app.events_counter = Counter("events", "Number of events.")
app.registry.register(app.events_counter)


@app.get("/")
async def hello():
    app.events_counter.inc({"path": "/"})
    return "hello"


@app.get("/metrics")
async def handle_metrics(response: Response, accept: List[str] = Header(None)):
    content, http_headers = render(app.registry, accept)
    return Response(content=content, media_type=http_headers["Content-Type"])

License

aioprometheus is released under the MIT license.

aioprometheus originates from the (now deprecated) prometheus python package which was released under the MIT license. aioprometheus continues to use the MIT license and contains a copy of the original MIT license from the prometheus-python project as instructed by the original license.

Issues
  • Aiohttp 3+ start/shutdown changes

    Aiohttp 3+ start/shutdown changes

    Hey, aiohttp3 introduced advanced site runners https://docs.aiohttp.org/en/stable/web_advanced.html#application-runners

    I am going to make PR to adjust aioprometheus to use it, but it will force to support aiohttp3+. Is it acceptable for You @claws ?

    opened by hellysmile 5
  • metrics handler to existing server

    metrics handler to existing server

    hello, would be nice to include some documentation on how to do something like:

    from aiohttp import web
    from aioprometheus import Service
    
    prometheus_service = Service()
    
    async def handle(request):
        text = "Hello aiohttp"
        return web.Response(text=text)
    
    app = web.Application()
    app.router.add_get('/', handle)
    app.router.add_get('/metrics', prometheus_service.handle_metrics)
    
    web.run_app(app)
    
    opened by benitogf 4
  • Make aiohttp dependency optional

    Make aiohttp dependency optional

    Includes:

    • Documentation update calling out the aiohttp extra for both the Service and Pusher features
    • Import guards in pusher and service
    • Runtime checks in the Pusher and Service initializers, to retain consistency in the top-level API
    • Quoted type annotations where aiohttp.web.* might not be available
    • Version bump to 19.11.0

    @claws let me know if there is anything I missed or if you have any concerns, thanks! :)

    opened by pirogoeth 3
  • Optional protoc

    Optional protoc

    As discussed https://github.com/claws/aioprometheus/issues/41

    opened by hellysmile 3
  • Adding a /healthz method

    Adding a /healthz method

    There are cases where aioprometheus will be the only http-serving port an application has, and in these instances it's helpful to use it for an orchestrator (AWS ECS, Kubernetes etc) to check the application's health.

    Now it tolerates the lack of an Accept: header it's possible to use /metrics for this. But generating the textual metrics list only to throw the output away is somewhat wasteful.

    To help out in this case would you be open to a PR adding a /healthz method that does nothing but return a 200 code?

    opened by alexmbird 3
  • prometheus_metrics_proto not present in requirements.txt

    prometheus_metrics_proto not present in requirements.txt

    Hi,

    aioprometheus imports prometheus_metrics_proto yet the package is not present in requirements.txt. A colleague of mine has spent some time finding out what is wrong, so it would be cool to fix this. Not sure what version to put there, so just opening an issue, not sending a pull request...

    Cheers!

    opened by tchap 3
  • pypi release 20.0.0 with optional binary support

    pypi release 20.0.0 with optional binary support

    Can You please release latest master to pypi?

    opened by hellysmile 2
  • [BUG] Multiple decorator bug

    [BUG] Multiple decorator bug

    While using inprogress and count_exceptions inprogress doesn't reach the metric.dec call Suggested fix (for inprogress):

    @wraps(func) async def func_wrapper(*args, **kwds): try: metric.inc(labels) rv = func(*args, **kwds) if isinstance(rv, asyncio.Future) or asyncio.iscoroutine(rv): rv = await rv finally: metric.dec(labels) return rv

    bug 
    opened by grootg 2
  • Add Default for Accept: Header

    Add Default for Accept: Header

    At present aioprometheus blows up if it receives a request without an Accept: header set. Witness:

    $ curl -v --header "Accept:" http://my.prom.metrics:29137/metrics
    ...
    < HTTP/1.1 500 Internal Server Error
    < Date: Tue, 24 Apr 2018 16:17:29 GMT
    < Content-Type: text/html; charset=utf-8
    < Content-Length: 141
    < Connection: keep-alive
    < Server: Python/3.6 aiohttp/3.1.2
    <
    * Connection #0 to host my.prom.metrics left intact
    <html><head><title>500 Internal Server Error</title></head><body><h1>500 Internal Server Error</h1>Server got itself in trouble</body></html>
    

    It happens because the call to request.headers.getall() is made without a default to return if the Accept: header is missing. Not a problem if you're Prometheus, but it is if you happen to be using this as an AWS ALB health check for your service.

    I've added a default, [], which allows negotiate() to do its thing and fall back to text mode. I've tested it with a local deploy of the new code but make venv is broken by a known problem with pip==10.0.1.

    opened by alexmbird 2
  • Documentation seems to be outdated

    Documentation seems to be outdated

    Dear all,

    the example is showing an outdated example due to the recent changes to the API, that introduced automatic registeration of metrics when they are created. Could you adjust the documentation accordingly?

    Thanks, Manuel

    opened by giffels 2
  • Deprecate support for binary format metrics

    Deprecate support for binary format metrics

    Prometheus stopped supporting the binary metric format a while ago. To simplify maintaining this library it is recommended that the support for binary format metrics be deprecated and eventually removed.

    opened by claws 0
  • ProcessCollector support

    ProcessCollector support

    Hi. It would be nice to have something like ProcessCollector. Using ProcessCollector from prometheus_client raises an error:

    Traceback (most recent call last):
      File "prom.py", line 5, in <module>
        prometheus_service.register(ProcessCollector(namespace='mydaemon'))
      File "/usr/local/lib/python3.7/site-packages/aioprometheus/service.py", line 177, in register
        self.registry.register(collector)
      File "/usr/local/lib/python3.7/site-packages/aioprometheus/registry.py", line 32, in register
        raise TypeError("Invalid collector type: {}".format(collector))
    TypeError: Invalid collector type: <prometheus_client.process_collector.ProcessCollector object at 0x7f2887ca0c10>
    

    Code:

    from prometheus_client import ProcessCollector
    from aioprometheus import Service
    
    prometheus_service = Service()
    prometheus_service.register(ProcessCollector(namespace='mydaemon'))
    
    enhancement 
    opened by easysugar 0
  • Add an auth parameter to Pusher class

    Add an auth parameter to Pusher class

    The push gateway is often used with an authentication. It would be nice to have an auth parameter that would be passed to the aiohttp.ClientSession.

    enhancement 
    opened by matclab 2
  • Summary doesn`t work correctly

    Summary doesn`t work correctly

    HI, I`ve just launched example from readme and after some time I saw:

    # HELP request_processing_seconds Time spent processing request
    # TYPE request_processing_seconds summary
    request_processing_seconds_count 142
    request_processing_seconds_sum 74.18954712400955
    request_processing_seconds{quantile="0.5"} 0.45743998500256566
    request_processing_seconds{quantile="0.9"} 0.8106792879989371
    request_processing_seconds{quantile="0.99"} 0.6946089910015871
    

    In results 99 quantile is lower than 90. It`s wrong.

    opened by mkabischev 0
python template private service

Template for private python service This is a cookiecutter template for an internal REST API service, written in Python, inspired by layout-golang. Th

UrvanovCompany 12 Nov 24, 2020
Monitor Python applications using Spring Boot Admin

Pyctuator Monitor Python web apps using Spring Boot Admin. Pyctuator supports Flask, FastAPI, aiohttp and Tornado. Django support is planned as well.

SolarEdge Technologies 97 Oct 22, 2021
asgi-server-timing-middleware

ASGI Server-Timing middleware An ASGI middleware that wraps the excellent yappi profiler to let you measure the execution time of any function or coro

null 19 Oct 5, 2021
FastAPI Boilerplate

FastAPI Boilerplate Features SQlAlchemy session Custom user class Top-level dependency Dependencies for specific permissions Celery SQLAlchemy for asy

Hide 62 Oct 15, 2021
FastAPI framework plugins

Plugins for FastAPI framework, high performance, easy to learn, fast to code, ready for production fastapi-plugins FastAPI framework plugins Cache Mem

RES 163 Oct 14, 2021
FastAPI构建的API服务

使用FastAPI 构建的商城项目API 学习FastAPI 构建项目目录 构建项目接口: 对应博客:https://www.charmcode.cn/article/2020-06-08_vue_mall_api 声明 此项目已经不再维护, 可以参考我另外一个项目https://github.co

王小右 47 Sep 7, 2021
Publish Xarray Datasets via a REST API.

Xpublish Publish Xarray Datasets via a REST API. Serverside: Publish a Xarray Dataset through a rest API ds.rest.serve(host="0.0.0.0", port=9000) Clie

xarray-contrib 70 Oct 18, 2021
FastAPI Admin Dashboard based on FastAPI and Tortoise ORM.

FastAPI ADMIN 中文文档 Introduction FastAPI-Admin is a admin dashboard based on fastapi and tortoise-orm. FastAPI-Admin provide crud feature out-of-the-bo

long2ice 929 Oct 23, 2021
The template for building scalable web APIs based on FastAPI, Tortoise ORM and other.

FastAPI and Tortoise ORM. Powerful but simple template for web APIs w/ FastAPI (as web framework) and Tortoise-ORM (for working via database without h

prostomarkeloff 72 Oct 15, 2021
Backend logic implementation for realworld with awesome FastAPI

Backend logic implementation for realworld with awesome FastAPI

Nik 1.4k Oct 23, 2021
Ready-to-use and customizable users management for FastAPI

FastAPI Users Ready-to-use and customizable users management for FastAPI Documentation: https://frankie567.github.io/fastapi-users/ Source Code: https

François Voron 1.3k Oct 23, 2021
ASGI middleware for authentication, rate limiting, and building CRUD endpoints.

Piccolo API Utilities for easily exposing Piccolo models as REST endpoints in ASGI apps, such as Starlette and FastAPI. Includes a bunch of useful ASG

null 37 Oct 13, 2021
api versioning for fastapi web applications

fastapi-versioning api versioning for fastapi web applications Installation pip install fastapi-versioning Examples from fastapi import FastAPI from f

Dean Way 303 Oct 21, 2021
Social Distancing Detector using deep learning and capable to run on edge AI devices such as NVIDIA Jetson, Google Coral, and more.

Smart Social Distancing Smart Social Distancing Introduction Getting Started Prerequisites Usage Processor Optional Parameters Configuring AWS credent

Neuralet 99 Oct 21, 2021
row level security for FastAPI framework

Row Level Permissions for FastAPI While trying out the excellent FastApi framework there was one peace missing for me: an easy, declarative way to def

Holger Frey 187 Oct 17, 2021
ReST based network device broker

The Open API Platform for Network Devices netpalm makes it easy to push and pull state from your apps to your network by providing multiple southbound

null 316 Sep 28, 2021
基于Pytorch的脚手架项目,Celery+FastAPI+Gunicorn+Nginx+Supervisor实现服务部署,支持Docker发布

cookiecutter-pytorch-fastapi 基于Pytorch的 脚手架项目 按规范添加推理函数即可实现Celery+FastAPI+Gunicorn+Nginx+Supervisor+Docker的快速部署 Requirements Python >= 3.6 with pip in

null 12 Oct 15, 2021
Backend, modern REST API for obtaining match and odds data crawled from multiple sites. Using FastAPI, MongoDB as database, Motor as async MongoDB client, Scrapy as crawler and Docker.

Introduction Apiestas is a project composed of a backend powered by the awesome framework FastAPI and a crawler powered by Scrapy. This project has fo

Fran Lozano 43 Oct 9, 2021
An extension for GINO to support Starlette server.

gino-starlette Introduction An extension for GINO to support starlette server. Usage The common usage looks like this: from starlette.applications imp

GINO Community 64 Oct 18, 2021