Instrument asyncio Python for distributed tracing with AWS X-Ray.

Overview

xraysink (aka xray-asyncio)

Package version Python versions Monthly downloads

Extra AWS X-Ray instrumentation to use distributed tracing with asyncio Python libraries that are not (yet) supported by the official aws_xray_sdk library.

Integrations Supported

  • Generic ASGI-compatible tracing middleware for any ASGI-compliant web framework. This has been tested with:
  • asyncio Task's
  • Background jobs/tasks

Installation

xraysink is distributed as a standard python package through pypi, so you can install it with your favourite Python package manager. For example:

pip install xraysink

How to use

xraysink augments the functionality provided by aws_xray_sdk. Before using the tools in xraysink, you first need to configure aws_xray_sdk - this will probably involve calling xray_recorder.configure() when your process starts, and optionally aws_xray_sdk.core.patch().

Extra instrumentation provided by xraysink is described below.

FastAPI

Instrument incoming requests in your FastAPI web server by adding the xray_middleware to your app. For example, you could do:

from starlette.middleware.base import BaseHTTPMiddleware
from xraysink.asgi.middleware import xray_middleware

# Standard asyncio X-Ray configuration, customise as you choose
xray_recorder.configure(context=AsyncContext(), service="my-cute-little-service")

# Create a FastAPI app with various middleware
app = FastAPI()
app.add_middleware(MyTracingDependentMiddleware)  # Any middleware that is added earlier will have the X-Ray tracing context available to it
app.add_middleware(BaseHTTPMiddleware, dispatch=xray_middleware)

Asyncio Tasks

If you start asyncio Task's from a standard request handler, then the AWS X-Ray SDK will not correctly instrument any outgoing requests made inside those Tasks.

Use the fixed AsyncContext from xraysink as a drop-in replacement, like so:

from aws_xray_sdk.core import xray_recorder
from xraysink.context import AsyncContext  # NB: Use the AsyncContext from xraysink

# Use the fixed AsyncContext when configuring X-Ray,
# and customise other configuration as you choose.
xray_recorder.configure(context=AsyncContext(use_task_factory=True))

Background Jobs/Tasks

If your process starts background tasks that make network calls (eg. to the database or an API in another service), then each execution of one of those tasks should be treated as a new X-Ray trace. Indeed, if you don't do so then you will likely get context_missing errors.

An async function that implements a background task can be easily instrumented using the @xray_task_async() decorator, like so:

from aws_xray_sdk.core import xray_recorder
from xraysink.tasks import xray_task_async

# Standard asyncio X-Ray configuration, customise as you choose
xray_recorder.configure(context=AsyncContext(), service="my-cute-little-service")

# Any call to this function will start a new X-Ray trace
@xray_task_async()
async def cleanup_stale_tokens():
    await database.get_table("tokens").delete(age__gt=1)

# Start your background task using your scheduling system of choice :)
schedule_recurring_task(cleanup_stale_tokens)

If your background task functions are called from a function that is already instrumented (eg. send an email immediately after handling a request), then the background task will appear as a child segment of that trace. In this case, you must ensure you use the fixed AsyncContext when configuring the recorder (ie. from xraysink.context import AsyncContext)

Process-Level Configuration

You can link your X-Ray traces to your CloudWatch Logs log records, which enhances the integration with AWS CLoudWatch ServiceLens. Take the following steps:

  1. Put the X-Ray trace ID into every log message. There is no convention for how to do this (it just has to appear verbatim in the log message somewhere), but if you are using structured logging then the convention is to use a field called traceId. Here's an example

    trace_id = xray_recorder.get_trace_entity().trace_id
    logging.getLogger("example").info("Hello World!", extra={"traceId": trace_id})
    
  2. Explicitly set the name of the CloudWatch Logs log group associated with your process. There is no general way to detect the Log Group from inside the process, hence it requires manual configuration as part of your process initialisation (eg. in the same place where you call xray_recorder.configure).

    set_xray_log_group("/example/service-name")
    

Note that this feature relies on undocumented functionality, and is not yet supported by the official Python SDK.

Licence

This project uses the Apache 2.0 licence, to make it compatible with aws_xray_sdk, the primary library for integrating with AWS X-Ray.

Comments
  • General Usage Question #2

    General Usage Question #2

    Following on https://github.com/garyd203/xraysink/issues/22

    I noticed the use of AsyncContext for FastApi, is this using the xraysink or aws sdk? Is it needed?

    I've attempted an implementation with either and have found that my async request handlers are unable to retrieve the context. This leads to AttributeErrors or just general errors where the segment cannot be found

    customer-support 
    opened by SLinAppliedBiomath 8
  • Question AWS- Background TASK

    Question AWS- Background TASK

    My question is, how can I make xray trace my calls to the database, this would be a background task. Currently you can see that the task is running but it does not give me information about the host and also the task is repeated, what is this due to?

    AWS-RAY conf image

    This is aws-trace image

    This is my async task image

    customer-support usage-question 
    opened by Arwiim 2
  • tracing exclusion rules for FastAPI endpoints

    tracing exclusion rules for FastAPI endpoints

    Very cool - thanks for the FastAPI middleware. Since the middleware means we don't have to decorate each function, i have the opposite problem - a situation where i'd like to exclude tracing for a particular endpoint. Is there a pattern for that which the AWS X-Ray SDK uses in cases like that? A function decorator? A path config used by the middleware?

    customer-support 
    opened by smcoll 2
  • General use questions

    General use questions

    I'm trying to get this going, but feeling like the documentation is confusing.

    xray_recorder.configure(context=AsyncContext(), service="my-cute-little-service")
    
    # Create a FastAPI app with various middleware
    app = FastAPI()
    app.add_middleware(MyTracingDependentMiddleware)  # Any middleware that is added earlier will have the X-Ray tracing context available to it
    app.add_middleware(BaseHTTPMiddleware, dispatch=xray_middleware)
    
    • which BastHTTPMiddleware is this referring to?
    • what is xray_middlewhere
    • do we import xraysink anywhere?
    • I'm using decorators on my endpoints @xray_recorder.capture_async("status") should I continue using these?
    customer-support 
    opened by brickfungus 2
  • fallback for missing host header

    fallback for missing host header

    In some scenarios (e.g. health check of lambda function behind ALB/multi target group), there is no host headed provided. It seems that with the fastapi integration, this breaks execution of this library. I added a fallback, which in the best case would be configurable later.

    relates to https://github.com/garyd203/xraysink/issues/65

    bug 
    opened by mroswald 1
  • auto-build `master` commits

    auto-build `master` commits

    Whenever a new commit is pushed to master (TBD: Or maybe some other trigger like setting a tag???), automatically do a minor version release, with associated version bumping and whatnot.

    desired actions:

    • validate we are on master
    • run tests (again)
    • bump package version (in multiple places) from 1.2.3-rc to 1.2.3
    • ~check git tag matches in-code version~
    • do poetry build, smoke test on artifact, then poetry publish
    • add git tag (and implied github release)
    • bump package version to 1.2.4-rc for continued development

    Ideas for a trigger:

    1. X create an issue by me, with a specifically formatted subject, or special label
    2. X use a deployment ?? Not sure that makes conceptual sense
    3. do a release. That's back to front though, I feel.
    4. X also rely on the workflow_run:completed event for a dependent workflow
    5. create a tag. this is identical to creating a release, really.
    6. -->> manually run the workflow https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow

    one method that might work:

    • use a script to bump version strings and add tag, and push upstream
    • have a workflow that listens for the tag and does the rest
    opened by garyd203 1
  • Fix instrumentation on dependent (gathered) asyncio tasks

    Fix instrumentation on dependent (gathered) asyncio tasks

    subsegment parents should work correctly. Use case is a request handler that spawns multiple asyncio tasks, and then gathers them all together to compose the response (eg. tartiflette)

    My working theory is that aws-xray-sdk has a bug where it copies a reference to the context dictionary onto a new task, rather than deep/shallow -copying the values. This menas that subsegments do push/pop on the same list, so the push/pop could be interleaved in an incorrect order.

    This bugfix should definitely be moved into the upstream aws-xray-sdk-python repo as a PR.

    opened by garyd203 1
  • do something with coverage

    do something with coverage

    • [x] Use the standard coverage module
    • [x] integrate with a normal pytest run using pytest-cov
    • [x] read doc for plugin and main module to find interesting features
    • [x] configure
    • [ ] https://github.com/wemake-services/coverage-conditional-plugin if necessary
    • [ ] set static contexts
    • [x] set up to show "something" in a PR. don't worry about 3rd party tools (boo!). Options:
      • https://github.com/orgoro/coverage
      • https://github.com/marketplace/actions/python-coverage-comment <- probably better, but more complex to setup
      • https://github.com/marketplace/actions/code-coverage-summary mebbe
    • [x] configure badge in readme
      • functionality in python-coverage-comment via shields.io
      • https://github.com/marketplace/actions/coverage-py-badge
    • [x] check plugin works for a real code change
    opened by garyd203 0
  • Documentation to help users self-diagnose usage problems

    Documentation to help users self-diagnose usage problems

    Add a default bug+usage checklist to help users find problems on their own. Perhaps also add to doco. See some ideas for content in the discussion in #88

    opened by garyd203 0
  • Support async+thread context propagation in FastAPI

    Support async+thread context propagation in FastAPI

    When FastAPI runs a non-async dependency or handler, it puts it into an internal thread pool so that the async execution doesn't get blocked. I'm pretty sure that the trace context doesn't get propagated into that thread pool (see Sean's problem in #88), which is obviously undesirable. It'd be nice if this worked seamlessly instead.

    We probably need a new context (or just extend the existing no-bugs AsyncContext in xraysink) which can hook into FastAPI's specific non-async dispatch in order to add the trace context to those calls. Size could be fairly large!

    bug enhancement 
    opened by garyd203 0
  • FastAPI implementation throwing missing segment

    FastAPI implementation throwing missing segment

    Hi, I am trying to utilize this library to implement X-Ray into my FastAPI api.

    I have the following code:

        xray_recorder.configure(context_missing="LOG_ERROR")
        xray_recorder.configure(service="API DEV")
        xray_recorder.configure(plugins=("ECSPlugin",))
        xray_patch_all()
        app.add_middleware(BaseHTTPMiddleware, dispatch=xray_middleware)
    

    But on each request I send, I see the daemon only receives 1 segment. Usually with a response time of 0.04ms, which definitely isn't the case. I see that this segment is sent before the response is even sent to the client.

    Along with this, the following is logged in my application: cannot find the current segment/subsegment, please make sure you have a segment open

    Is there a working example of this working with FastAPI? Am I missing something here? Thanks!

    usage-question 
    opened by sean-a-grant 8
  • Create convenience function to set up everything, with defaults

    Create convenience function to set up everything, with defaults

    The documentation assumes you know what you are doing, and that you only want to fix some of the known problems. As an alternative use case ,provide a convenience function that patches everything global and returns the recorder object

    documentation enhancement 
    opened by garyd203 0
Owner
Gary Donovan
Two-Thirds-Stack developer across Python, Java and AWS. Infrastructure, server-side coding, and serverless.
Gary Donovan
Automated AWS account hardening with AWS Control Tower and AWS Step Functions

Automate activities in Control Tower provisioned AWS accounts Table of contents Introduction Architecture Prerequisites Tools and services Usage Clean

AWS Samples 20 Dec 7, 2022
Implement backup and recovery with AWS Backup across your AWS Organizations using a CI/CD pipeline (AWS CodePipeline).

Backup and Recovery with AWS Backup This repository provides you with a management and deployment solution for implementing Backup and Recovery with A

AWS Samples 8 Nov 22, 2022
Python + AWS Lambda Hands OnPython + AWS Lambda Hands On

Python + AWS Lambda Hands On Python Criada em 1990, por Guido Van Rossum. "Bala de prata" (quase). Muito utilizado em: Automatizações - Selenium, Beau

Marcelo Ortiz de Santana 8 Sep 9, 2022
Aws-cidr-finder - A Python CLI tool for finding unused CIDR blocks in AWS VPCs

aws-cidr-finder Overview An Example Installation Configuration Contributing Over

Cooper Walbrun 18 Jul 31, 2022
Automatically compile an AWS Service Control Policy that ONLY allows AWS services that are compliant with your preferred compliance frameworks.

aws-allowlister Automatically compile an AWS Service Control Policy that ONLY allows AWS services that are compliant with your preferred compliance fr

Salesforce 189 Dec 8, 2022
SSH-Restricted deploys an SSH compliance rule (AWS Config) with auto-remediation via AWS Lambda if SSH access is public.

SSH-Restricted SSH-Restricted deploys an SSH compliance rule with auto-remediation via AWS Lambda if SSH access is public. SSH-Auto-Restricted checks

Adrian Hornsby 30 Nov 8, 2022
AWS Auto Inventory allows you to quickly and easily generate inventory reports of your AWS resources.

Photo by Denny Müller on Unsplash AWS Automated Inventory ( aws-auto-inventory ) Automates creation of detailed inventories from AWS resources. Table

AWS Samples 123 Dec 26, 2022
aws-lambda-scheduler lets you call any existing AWS Lambda Function you have in a future time.

aws-lambda-scheduler aws-lambda-scheduler lets you call any existing AWS Lambda Function you have in the future. This functionality is achieved by dyn

Oğuzhan Yılmaz 57 Dec 17, 2022
Project template for using aws-cdk, Chalice and React in concert, including RDS Postgresql and AWS Cognito

What is This? This repository is an opinonated project template for using aws-cdk, Chalice and React in concert. Where aws-cdk and Chalice are in Pyth

Rasmus Jones 4 Nov 7, 2022
POC de uma AWS lambda que executa a consulta de preços de criptomoedas, e é implantada na AWS usando Github actions.

Cryptocurrency Prices Overview Instalação Repositório Configuração CI/CD Roadmap Testes Overview A ideia deste projeto é aplicar o conteúdo estudado s

Gustavo Santos 3 Aug 31, 2022
Unauthenticated enumeration of services, roles, and users in an AWS account or in every AWS account in existence.

Quiet Riot ?? C'mon, Feel The Noise ?? An enumeration tool for scalable, unauthenticated validation of AWS principals; including AWS Acccount IDs, roo

Wes Ladd 89 Jan 5, 2023
AWS Blog post code for running feature-extraction on images using AWS Batch and Cloud Development Kit (CDK).

Batch processing with AWS Batch and CDK Welcome This repository demostrates provisioning the necessary infrastructure for running a job on AWS Batch u

AWS Samples 7 Oct 18, 2022
Aws-lambda-requests-wrapper - Request/Response wrapper for AWS Lambda with API Gateway

AWS Lambda Requests Wrapper Request/Response wrapper for AWS Lambda with API Gat

null 1 May 20, 2022
AWS-serverless-starter - AWS Lambda serverless stack via Serverless framework

Serverless app via AWS Lambda, ApiGateway and Serverless framework Configuration

 Bəxtiyar 3 Feb 2, 2022
AWS CloudSaga - Simulate security events in AWS

AWS CloudSaga - Simulate security events in AWS AWS CloudSaga is for customers to test security controls and alerts within their Amazon Web Services (

Amazon Web Services - Labs 325 Dec 1, 2022
A Python script for rendering glTF files with V-Ray App SDK

V-Ray glTF viewer Overview The V-Ray glTF viewer is a set of Python scripts for the V-Ray App SDK that allow the parsing and rendering of glTF (.gltf

Chaos 24 Dec 5, 2022
An asyncio Python wrapper around the Discord API, forked off of Rapptz's Discord.py.

Novus A modern, easy to use, feature-rich, and async ready API wrapper for Discord written in Python. A full fork of Rapptz's Discord.py library, with

Voxel Fox 60 Jan 3, 2023
Asynchronous and also synchronous non-official QvaPay client for asyncio and Python language.

Asynchronous and also synchronous non-official QvaPay client for asyncio and Python language. This library is still under development, the interface could be changed.

Leynier Gutiérrez González 8 Sep 18, 2021