The Python agent for Apache SkyWalking

Last update: Dec 12, 2022

Overview

SkyWalking Python Agent

SkyWalking-Python: The Python Agent for Apache SkyWalking, which provides the native tracing abilities for Python project.

SkyWalking: an APM(application performance monitor) system, especially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures.

Documentation

Official documentation
Blog about the Python Agent Profiling Feature

Installation Requirements

SkyWalking Python Agent requires SkyWalking 8.0+ and Python 3.6+.

If you would like to try out the latest features that are not released yet, please refer to this guide to build from sources.

Contributing

Before submitting a pull request or pushing a commit, please read our contributing and developer guide.

Contact Us

Submit an GitHub Issue by using [Python] as title prefix.
Mail list: [email protected]. Mail to [email protected], follow the reply to subscribe the mail list.
Join skywalking channel at Apache Slack. If the link is not working, find the latest one at Apache INFRA WIKI.
Twitter, ASFSkyWalking

License

Apache 2.0

Comments

feat: Add support for MeterReportService

Add support for MeterReportService

Behavior

This service runs along with the agent like other services and reports roughly every 20 seconds.

Syntactic sugar

Counter

c = Counter('c2', CounterMode.INCREMENT)
c.build()

# increase Counter c by the time the with-wrapped codes consumed
with c.create_timer():
    # some codes may consume a certain time

c = Counter('c3', CounterMode.INCREMENT)
c.build()

# increase Counter c by num once counter_decorator_test gets called
@Counter.increase(name='c3', num=2)
def counter_decorator_test():
    # some codes

c = Counter('c4', CounterMode.INCREMENT)
c.build()

# increase Counter c by the time counter_decorator_test consumed
@Counter.timer(name='c4')
def counter_decorator_test(s):
    # some codes may consume a certain time

Histogram

h = Histogram('h3', [i / 10 for i in range(10)])
h.build()

# Histogram h will record the time the with-wrapped codes consumed
with h.create_timer():
    # some codes may consume a certain time

h = Histogram('h2', [i / 10 for i in range(10)])
h.build()

# Histogram h will record the time histogram_decorator_test consumed
@Histogram.timer(name='h2')
def histogram_decorator_test(s):
    time.sleep(s)

Simple test

insert following codes below this line agent/__init__.py#L151

from skywalking.meter.gauge import Gauge
import gc

def generator_for_gc_g0():
    while(True):
        yield gc.get_stats()[0]['collected']

def generator_for_gc_g1():
    while(True):
        yield gc.get_stats()[1]['collected']

def generator_for_gc_g2():
    while(True):
        yield gc.get_stats()[2]['collected']


gc_g0_meter = Gauge("gc_g0", generator_for_gc_g0())
gc_g0_meter.build()

gc_g1_meter = Gauge("gc_g1", generator_for_gc_g1())
gc_g1_meter.build()

gc_g2_meter = Gauge("gc_g2", generator_for_gc_g2())
gc_g2_meter.build()

feature core

opened by jiang1997 19

fix: grpc timeout segment data loss

Please review this thoroughly because all I wanted to do was resend segments which failed to send due to a grpc timeout. But unexpectedly the behavior of grpc timeout changed, it doesn't time out anymore, the heartbeat seems to keep it alive now. I don't understand why this behavior changed so that is why I am requesting a thorough looking at this.
enhancement

opened by tom-pytel 18
Add http ignore by method
Added SW_HTTP_IGNORE_METHOD which will allow ignoring http operations by method (GET, POST, HEAD, OPTIONS, etc...)

Set default SW_AGENT_MAX_BUFFER_SIZE back to 10000 from 1000, which turns out to be too small for python (many unnecessarily dropped spans).

feature
opened by tom-pytel 14
feat: report metrics related to pvm

Metrics

CPU

total_cpu_utilization

process_cpu_utilization

cpu utilization of the current process

thread_active_count

number of threads invoked by the current process and its children

Memory

total_mem_utilization

process_mem_utilization

memory utilization of the current process

GC

gc_g0

number of objects of generation 0

gc_g1

number of objects of generation 1

gc_g2

number of objects of generation 2

gc_time

For now, MeterReportService is available with gRPC only. So does the e2e test. I will implement MeterReportService and enable e2e test for Kafka later.

Welcome any advice about naming.
feature core

opened by jiang1997 12
[Fix][Plugin] sw_flask general exceptions handled
sw_flask fix will handle errors like returning the wrong type from a handler or other internal errors.

Updated StackedSpan to track depth, and made depth variable instance instead of class level (this was a bug).

Changed how SpanContext decides when all spans finished to write Segment data, now counts span start / stops which should work better across different async scenarios.

Changed new_exit_span() with span.inject() to work simpler like the NodeJS agent, now plugins inject directly themselves if they need to.

Removed carrier from plugins which didn't actually use it.

bug plugin
opened by tom-pytel 12
Add plugin for bottle
[x] Add a test case for the new plugin

[x] Add a CHANGELOG entry for the new plugin

[x] Add a component id in the main repo

[x] Add a logo in the UI repo

[x] Rebuild the requirements.txt by running tools/env/build_requirements_(linux|windows).sh

[x] Rebuild the Plugins.md documentation by running make doc-gen

plugin
opened by jiang1997 11
Enable HTTP log reporting

This is a work in progress.

Now reporting logs in JSON through HTTP to http://oap/v3/logs does work.

But, I'm not sure if the oap/v3/logs endpoint is for such usage(It seems designed for fluent-bit batch reporting?). Reporting logs one by one through HTTP may not be ideal in terms of performance. I'm not sure whether the Java agent only implements gRPC reporter intentionally out of this reason.

Please advise.

Signed-off-by: YihaoChen [email protected]
feature

opened by Superskyyy 11
Feature: collect and report logs
This is a OSPP Summer 2021 project supervised by @Humbertzhang | apache/skywalking#7118

The feature implements optional log reporter functionalities in alignment with the SkyWalking Java agent.

Intercepts logs from Python logging module.

Reports logs via a new temporary gRPC channel(to be removed in the future).

Supports unformatted/ formatted logs with custom layout.

Supports custom logging level threshold.

Bumps up submodule to support log collection protocols.

Bumps up skywalking-eyes and adds a config entry to ignore .gitignore during license checks.

feature core
opened by Superskyyy 11
Introduce better coding style

This PR migrates existing usage of old-style string manipulation (%, .format and +) to f-strings for better clarity and some performance boost.

Adds stricter coding style.

Optimizes debug messages for performance.

Signed-off-by: Superskyyy [email protected]
enhancement chore

opened by Superskyyy 10
Add E2E coverage for Trace and Logging
Change Fastapi to FastAPI component name (sync with OAP change)

Add E2E coverage for Python agent (Trace and Logging), profiling is not tested for now.

Renamed a build variable to remove possible mix-ups with CLI environment Var.

Enhance plugin test to reuse images.

Combined CI to pass if changes are non-essential to agent and workflows.

Note: There's a flaky test due to a possible {{- contains }} issue in the Infra-e2e verifier, the logs seem to be queried in an unstable order by the CLI where later log often placed first in log list, combining with the contains bug, it becomes flaky.

A time.sleep() workaround makes sure the log arrives in clear order which passes the e2e for now.
test
opened by Superskyyy 9
[Enhancement] Async tasks should work 100%

These changes should address all remaining async-related problems not handled in #88. The spans list has been moved from an instance variable in SpanContext to a global task-local variable which allows "forking" it for new subtasks so that multiple concurrent async subtasks don't interfere with one another. spans is now a top-level module var because Python stipulates that ContextVar should be such and only one context exists at any given time so it is essentially a singleton anyway.
enhancement core

opened by tom-pytel 9
Enhance redis plugin to adapt Virtual Cache

Resolves: #10212
enhancement

opened by Jedore 2

Releases(v0.8.0)

v0.8.0(Jul 10, 2022)
What's Changed

spans correctly reference finished parents by @tom-pytel in https://github.com/apache/skywalking-python/pull/161

Refactor current Python agent docs to serve on SkyWalking official website by @Superskyyy in https://github.com/apache/skywalking-python/pull/162

fix broken url by @JaredTan95 in https://github.com/apache/skywalking-python/pull/163

Remove docs from main README.md - Add website doc links. by @Superskyyy in https://github.com/apache/skywalking-python/pull/164

Refactor SkyWalking Python to use the CLI for CI instead of legacy setup by @Superskyyy in https://github.com/apache/skywalking-python/pull/165

Remove places that mentions Python 3.5 support due to EOL by @Superskyyy in https://github.com/apache/skywalking-python/pull/166

Cleanup and Python 3.10 support by @Superskyyy in https://github.com/apache/skywalking-python/pull/167

filled out rest of psycopg 3 plugin by @tom-pytel in https://github.com/apache/skywalking-python/pull/168

Move flake configs all together by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/169

Enable faster CI by categorical parallelism by @Superskyyy in https://github.com/apache/skywalking-python/pull/170

Introduce better coding style by @Superskyyy in https://github.com/apache/skywalking-python/pull/171

Minimize exclude in lint by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/173

Introduce another set of flake8 extensions by @Superskyyy in https://github.com/apache/skywalking-python/pull/174

fix aiohttp outgoing request url by @tom-pytel in https://github.com/apache/skywalking-python/pull/175

bugfix: flask + nginx got KeyError: 'REMOTE_PORT' in sw_flask.py plugin by @VxCoder in https://github.com/apache/skywalking-python/pull/176

Add missing ending quote by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/177

updated deprecated configuration by @doddi in https://github.com/apache/skywalking-python/pull/179

Add plugin for mysqlclient by @katelei6 in https://github.com/apache/skywalking-python/pull/178

Fix sw-rabbitmq TypeError when there are no headers by @dcryans in https://github.com/apache/skywalking-python/pull/182

Fix agent bootup traceback not shown in sw-python CLI by @Superskyyy in https://github.com/apache/skywalking-python/pull/183

Add plugin for FastAPI by @katelei6 in https://github.com/apache/skywalking-python/pull/181

Support CI container logs by @Superskyyy in https://github.com/apache/skywalking-python/pull/185

Update mySQL plugin to support two different parameter keys. by @katelei6 in https://github.com/apache/skywalking-python/pull/186

add the code > 400 is error by @katelei6 in https://github.com/apache/skywalking-python/pull/187

Doc: add how to use with uwsgi by @arcosx in https://github.com/apache/skywalking-python/pull/188

Fix: right doc link for How To Use With uWSGI by @arcosx in https://github.com/apache/skywalking-python/pull/189

The local log stack depth is not truncated by @wzy960520 in https://github.com/apache/skywalking-python/pull/190

Revert "The local log stack depth is not truncated (#190)" by @Superskyyy in https://github.com/apache/skywalking-python/pull/191

Fix sw_formatter wrongly set cache that impacts user handlers by @Superskyyy in https://github.com/apache/skywalking-python/pull/192

Fix typo that cause failure in loading user sitecustomize.py by @Superskyyy in https://github.com/apache/skywalking-python/pull/193

Drop support for Flask 1.x due to EOL & Jinja2 issue by @Superskyyy in https://github.com/apache/skywalking-python/pull/195

Update agent docs, changelogs and PR template. by @Superskyyy in https://github.com/apache/skywalking-python/pull/197

Fix Python shown as UNKNOWN language in OAP by @Superskyyy in https://github.com/apache/skywalking-python/pull/194

Fix logging level not properly set according to config by @Superskyyy in https://github.com/apache/skywalking-python/pull/196

Fix the properties are not set correctly by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/198

Add E2E coverage for Trace and Logging by @Superskyyy in https://github.com/apache/skywalking-python/pull/199

Support log reporter safe mode by @Superskyyy in https://github.com/apache/skywalking-python/pull/200

Fix scheduled event fails changes CI by @Superskyyy in https://github.com/apache/skywalking-python/pull/201

Fix deadlink and CI on schedule by @Superskyyy in https://github.com/apache/skywalking-python/pull/203

Remove namespace to instance properties and add pid by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/205

Enhance Traceback depth default to 10 by @Superskyyy in https://github.com/apache/skywalking-python/pull/206

Unify the tag name with other agents by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/208

Improved ignore path regex by @tom-pytel in https://github.com/apache/skywalking-python/pull/210

Fix sw_psycopg2 register_type() by @tom-pytel in https://github.com/apache/skywalking-python/pull/211

Fix psycopg2 register_type() second arg default by @tom-pytel in https://github.com/apache/skywalking-python/pull/212

Add plugin doc check in CI by @JarvisG495 in https://github.com/apache/skywalking-python/pull/213

Fix typo in docker/Makefile by @jiang1997 in https://github.com/apache/skywalking-python/pull/216

Update UI repository link in PR template by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/217

Add plugin for bottle by @jiang1997 in https://github.com/apache/skywalking-python/pull/214

Draft release 0.8.0 by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/219

Migrate license checker to license-eye and adjust the release script by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/220

New Contributors

@JaredTan95 made their first contribution in https://github.com/apache/skywalking-python/pull/163

@VxCoder made their first contribution in https://github.com/apache/skywalking-python/pull/176

@doddi made their first contribution in https://github.com/apache/skywalking-python/pull/179

@katelei6 made their first contribution in https://github.com/apache/skywalking-python/pull/178

@dcryans made their first contribution in https://github.com/apache/skywalking-python/pull/182

@arcosx made their first contribution in https://github.com/apache/skywalking-python/pull/188

@wzy960520 made their first contribution in https://github.com/apache/skywalking-python/pull/190

@JarvisG495 made their first contribution in https://github.com/apache/skywalking-python/pull/213

@jiang1997 made their first contribution in https://github.com/apache/skywalking-python/pull/216

Full Changelog: https://github.com/apache/skywalking-python/compare/v0.7.0...v0.8.0
Source code(tar.gz)
Source code(zip)
v0.7.0(Sep 13, 2021)
Feature:

Support collecting and reporting logs to backend (#147)

Support profiling Python method level performance (#127

Add a new sw-python CLI that enables agent non-intrusive integration (#156)

Add exponential reconnection backoff strategy when OAP is down (#157)

Support ignoring traces by http method (#143)

NoopSpan on queue full, propagation downstream (#141)

Support agent namespace. (#126)

Support secure connection option for GRPC and HTTP (#134)

Plugins:

Add Falcon Plugin (#146)

Update sw_pymongo.py to be compatible with cluster mode (#150)

Add Python celery plugin (#125)

Support tornado5+ and tornado6+ (#119)

Fixes:

Remove HTTP basic auth credentials from log, stacktrace, segment (#152)

Fix @trace decorator not work (#136)

Fix grpc disconnect, add SW_AGENT_MAX_BUFFER_SIZE to control buffer queue size (#138)

Others:

Chore: bump up requests version to avoid license issue (#142)

Fix module wrapt as normal install dependency (#123)

Explicit component inheritance (#132)

Provide dockerfile & images for easy integration in containerized scenarios (#159)

Source code(tar.gz)
Source code(zip)
v0.6.0(Mar 31, 2021)
Fixes:

Segment data loss when gRPC timing out. (#116)

sw_tornado plugin async handler status set correctly. (#115)

sw_pymysql error when connection haven't db. (#113)

Source code(tar.gz)
Source code(zip)
v0.5.0(Dec 31, 2020)
New plugins

Pyramid Plugin (#102)

AioHttp Plugin (#101)

Sanic Plugin (#91)

API and enhancements

@trace decorator supports async functions

Supports async task context

Optimized path trace ignore

Moved exception check to Span.__exit__

Moved Method & Url tags before requests

Fixes:

BaseExceptions not recorded as errors

Allow pending data to send before exit

sw_flask general exceptions handled

Make skywalking logging Non-global

Chores and tests

Make tests really run on specified Python version

Deprecate 3.5 as it's EOL

Source code(tar.gz)
Source code(zip)
v0.4.0(Nov 24, 2020)
Feature: Support Kafka reporter protocol (#74)

BugFix: Move generated packages into skywalking namespace to avoid conflicts (#72)

BugFix: Agent cannot reconnect after server is down (#79)

Test: Mitigate unsafe yaml loading (#76)

Source code(tar.gz)
Source code(zip)
v0.3.0(Aug 28, 2020)
New plugins

Urllib3 Plugin (#69)

Elasticsearch Plugin (#64)

PyMongo Plugin (#60)

Rabbitmq Plugin (#53)

Make plugin compatible with Django (#52)

API

Add process propagation (#67)

Add tags to decorators (#65)

Add Check version of packages when install plugins (#63)

Add thread propagation (#62)

Add trace ignore (#59)

Support snapshot context (#56)

Support correlation context (#55)

Chores and tests

Test: run multiple versions of supported libraries (#66)

Chore: add pull request template for plugin (#61)

Chore: add dev doc and reorganize the structure (#58)

Test: update test health check (#57)

Chore: add make goal to package release tar ball (#54)

Source code(tar.gz)
Source code(zip)
v0.2.0(Jul 28, 2020)
New plugins

Kafka Plugin (#50)

Tornado Plugin (#48)

Redis Plugin (#44)

Django Plugin (#37)

PyMsql Plugin (#35)

Flask plugin (#31)

API

Add ignore_suffix Config (#40)

Add missing log method and simplify test codes (#34)

Add content equality of SegmentRef (#30)

Validate carrier before using it (#29)

Chores and tests

Test: print the diff list when validation failed (#46)

Created venv builders for linux/windows and req flashers + use documentation (#38)

Source code(tar.gz)
Source code(zip)
v0.1.0(Jun 28, 2020)
API: agent core APIs, check the APIs and the examples

Plugin: built-in libraries http, urllib.request and third-party library requests are supported.

Test: agent test framework is setup, and the corresponding tests of aforementioned plugins are also added.

Source code(tar.gz)
Source code(zip)

Owner

The Apache Software Foundation

GitHub https://skywalking.apache.org/

Python-geoarrow - Storing geometry data in Apache Arrow format

geoarrow Storing geometry data in Apache Arrow format Installation $ pip install

11 Mar 3, 2022

Project repository of Apache Airflow, deployed on Docker in Amazon EC2 via GitLab.

Airflow on Docker in EC2 + GitLab's CI/CD Personal project for simple data pipeline using Airflow. Airflow will be installed inside Docker container,

13 Nov 29, 2022

A tutorial presents several practical examples of how to build DAGs in Apache Airflow

Apache Airflow - Python Brasil 2021 Este tutorial apresenta vários exemplos práticos de como construir DAGs no Apache Airflow. Background Apache Airfl

14 Jun 3, 2022

Apache Superset out of box version(Windows 64-bit)

superset_app Apache Superset out of box version (Windows 64bit) prepare job download 3 files python-3.8.10-embed-amd64.zip get-pip.py python_geohash‑0

9 Oct 2, 2022

Make dbt docs and Apache Superset talk to one another

dbt-superset-lineage Make dbt docs and Apache Superset talk to one another Why do I need something like this? Odds are rather high that you use dbt to

81 Jan 6, 2023

A software dedicated to automaticaly select the agent of your desire in Valorant

AUTOPICKER A software dedicated to automaticaly select the agent of your desire in Valorant GUIDE Before stariting to use this program check if you ha

0 Sep 24, 2022

The learning agent learns firstly approaching to the football and then kicking the football to the target position

Football Court This project utilized Pytorch and Tensorflow so that the learning agent learns firstly approaching to the football and then kicking the

1 Nov 19, 2021

An evolutionary multi-agent platform based on mesa and NEAT

6 Dec 4, 2022

PressurePlate is a multi-agent environment that requires agents to cooperate during the traversal of a gridworld.

PressurePlate is a multi-agent environment that requires agents to cooperate during the traversal of a gridworld. The grid is partitioned into several rooms, and each room contains a plate and a closed doorway.

Autonomous Agents Research Group (University of Edinburgh)

6 Dec 3, 2022

Todos os exercícios do Curso de Python, do canal Curso em Vídeo, resolvidos em Python, Javascript, Java, C++, C# e mais...

Exercícios - CeV Oferecido por Linguagens utilizadas atualmente O que vai encontrar aqui? ?? Esse repositório é dedicado a armazenar todos os enunciad

43 Nov 10, 2022

PyDy, short for Python Dynamics, is a tool kit written in the Python

PyDy, short for Python Dynamics, is a tool kit written in the Python programming language that utilizes an array of scientific programs to enable the study of multibody dynamics. The goal is to have a modular framework and eventually a physics abstraction layer which utilizes a variety of backends that can provide the user with their desired workflow

307 Jan 1, 2023

The Python agent for Apache SkyWalking

Related tags

Overview

SkyWalking Python Agent

Documentation

Installation Requirements

Contributing

Contact Us

License

Comments

Behavior

Syntactic sugar

Counter

Histogram

Simple test

Metrics

CPU

total_cpu_utilization

process_cpu_utilization

thread_active_count

Memory

total_mem_utilization

process_mem_utilization

GC

gc_g0

gc_g1

gc_g2

gc_time

Releases(v0.8.0)

v0.8.0(Jul 10, 2022)

What's Changed

New Contributors

v0.7.0(Sep 13, 2021)

v0.6.0(Mar 31, 2021)

v0.5.0(Dec 31, 2020)

v0.4.0(Nov 24, 2020)

v0.3.0(Aug 28, 2020)

v0.2.0(Jul 28, 2020)

v0.1.0(Jun 28, 2020)

Owner

The Apache Software Foundation

Python-geoarrow - Storing geometry data in Apache Arrow format

Project repository of Apache Airflow, deployed on Docker in Amazon EC2 via GitLab.

A tutorial presents several practical examples of how to build DAGs in Apache Airflow

Apache Superset out of box version(Windows 64-bit)

Make dbt docs and Apache Superset talk to one another

A software dedicated to automaticaly select the agent of your desire in Valorant

The learning agent learns firstly approaching to the football and then kicking the football to the target position

An evolutionary multi-agent platform based on mesa and NEAT

PressurePlate is a multi-agent environment that requires agents to cooperate during the traversal of a gridworld.

Todos os exercícios do Curso de Python, do canal Curso em Vídeo, resolvidos em Python, Javascript, Java, C++, C# e mais...

PyDy, short for Python Dynamics, is a tool kit written in the Python

A Python script made for the Python Discord Pixels event.

this is a basic python project that I made using python

Analisador de strings feito em Python // String parser made in Python

Python with braces. Because Python is awesome, but whitespace is awful.

PSP (Python Starter Package) is meant for those who want to start coding in python but are new to the coding scene.

Py-Parser est un parser de code python en python encore en plien dévlopement.

Simple, high-school-leveled sequence library written in Python / 간단한 고등학교 수준 수열 라이브러리 (Python)

A community based economy bot with python works only with python 3.7.8 as web3 requires cytoolz