Official repository for the BPF Performance Tools book

Related tags

Miscellaneous bpf-perf-tools-book
Overview

BPF Performance Tools

This is the official repository of BPF (eBPF) tools from the book BPF Performance Tools: Linux and Application Observability. The directories are:

  • originals: The original published version of the tools.
  • updated: Updated versions of the tools.
  • exercises: Exercise solutions.

These tools are documented in the book.

Tools:

In red are many of the new tools developed for the book, which are all in this repository. In black are the prior tools from BCC and bpftrace, many of which are also covered in the book.

Updated tools

You may contribute updated versions of the tools to the updated/ directory as you find they need fixes for newer kernels or other distributions (which can change CONFIG options or library paths, changing the probes). Please avoid ifdef's where possible, and create separate tools as they can be tested and maintained independently. Filename examples:

  • opensnoop_5.4.bt: opensnoop.bt for Linux 5.4 onwards
  • opensnoop_redfrog.bt: opensnoop for the RedFrog Linux distribution (I made that up).
  • opensnoop_redfrog11.bt: opensnoop for RedFrog release 11 onwards.
  • opensnoop_redfrog11_5.4.bt: opensnoop for RedFrog release 11 onwards, with Linux 5.4 onwards.

By contributing updates to this repository, you agree that the publisher has the necessary permissions to include your updates in possible later editions of the book. Attribution will be given: make it clear in the PR what your name is. Note that the tools are deliberately short to serve as textbook examples and to simplify maintenance, and updates should not add functionality. For that, see the next section.

Enhanced tools

A small selection of tools from this book has been added to the bpftrace and BCC repositories. It is expected that the BCC versions will be enhanced with various command line options. We are not planning to add any more of these tools to those repositories: they are limited to ensure that they can be well supported by the repository community.

Comments
  • Variable scoping inside if statements

    Variable scoping inside if statements

    Related to https://github.com/iovisor/bpftrace/issues/686

    There's only one script which defines variables inside an if-block and then uses them outside the block: https://github.com/brendangregg/bpf-perf-tools-book/blob/d928209118dd4ab8b3727bd7cc1c9c83b4f695b2/Ch10_Networking/tcpretrans.bt#L53

    Ideally what this script does should be fine, since $daddr and $saddr and defined in both branches of the if/else, but I think it'd be very hard (impossible?) to confirm this is true in the general case at compile time, so we should require variables to be declared in the scope they are used in, or a parent scope, like C does.

    To be compatible with a future bpftrace which has variable scoping, this bit of code:

        if ($inet_family == AF_INET) {
      $daddr = ntop($sk->__sk_common.skc_daddr);
      $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
    } else {
      $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
      $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
    }

    should be changed to:

        $daddr = 0;
    $saddr = 0;
    if ($inet_family == AF_INET) {
      $daddr = ntop($sk->__sk_common.skc_daddr);
      $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
    } else {
      $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
      $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
    }
    opened by ajor 4
  • tcplife: ICmp Assertion abort

    tcplife: ICmp Assertion abort

    This tool worked not that long ago, but now hits an Assertion:

    # tcplife.bt 
bpftrace: /usr/include/llvm-7/llvm/IR/Instructions.h:1117: void llvm::ICmpInst::AssertOK(): Assertion `getOperand(0)->getType() == getOperand(1)->getType() && "Both operands to ICmp instruction are not of the same type!"' failed.
Aborted (core dumped)

    (gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fffef922801 in __GI_abort () at abort.c:79
#2  0x00007fffef91239a in __assert_fail_base (fmt=0x7fffefa997d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x5555557ebbc8 "getOperand(0)->getType() == getOperand(1)->getType() && \"Both operands to ICmp instruction are not of the same type!\"", 
    file=file@entry=0x5555557eba78 "/usr/include/llvm-7/llvm/IR/Instructions.h", line=line@entry=1117, function=function@entry=0x5555557ed820 <llvm::ICmpInst::AssertOK()::__PRETTY_FUNCTION__> "void llvm::ICmpInst::AssertOK()")
    at assert.c:92
#3  0x00007fffef912412 in __GI___assert_fail (assertion=0x5555557ebbc8 "getOperand(0)->getType() == getOperand(1)->getType() && \"Both operands to ICmp instruction are not of the same type!\"", 
    file=0x5555557eba78 "/usr/include/llvm-7/llvm/IR/Instructions.h", line=1117, function=0x5555557ed820 <llvm::ICmpInst::AssertOK()::__PRETTY_FUNCTION__> "void llvm::ICmpInst::AssertOK()") at assert.c:101
#4  0x000055555578d2a6 in llvm::ICmpInst::AssertOK (this=0x5555562f1810) at /usr/include/llvm-7/llvm/IR/Instructions.h:1116
#5  0x000055555578d3a2 in llvm::ICmpInst::ICmpInst (this=0x5555562f1810, pred=llvm::CmpInst::ICMP_NE, LHS=0x555556157e10, RHS=0x55555604b8c0, NameStr=...) at /usr/include/llvm-7/llvm/IR/Instructions.h:1171
#6  0x00005555557922d1 in llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>::CreateICmp (this=0x7fffffffe618, P=llvm::CmpInst::ICMP_NE, LHS=0x555556157e10, RHS=0x55555604b8c0, Name=...)
    at /usr/include/llvm-7/llvm/IR/IRBuilder.h:1851
#7  0x00005555557924b2 in llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>::CreateICmpNE (this=0x7fffffffe618, LHS=0x555556157e10, RHS=0x55555604b8c0, Name=...) at /usr/include/llvm-7/llvm/IR/IRBuilder.h:1741
#8  0x0000555555782956 in bpftrace::ast::CodegenLLVM::createLogicalAnd (this=0x7fffffffe5f0, binop=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1480
#9  0x000055555577d09c in bpftrace::ast::CodegenLLVM::visit (this=0x7fffffffe5f0, binop=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:793
#10 0x0000555555771bae in bpftrace::ast::Binop::accept (this=0x7fffe851c180, v=...) at /home/bgregg/Build/bpftrace/src/ast/ast.cpp:45
#11 0x000055555577fc19 in bpftrace::ast::CodegenLLVM::visit (this=0x7fffffffe5f0, if_block=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1159
#12 0x0000555555771de0 in bpftrace::ast::If::accept (this=0x7fffe8a35eb0, v=...) at /home/bgregg/Build/bpftrace/src/ast/ast.cpp:89
#13 0x0000555555780be4 in bpftrace::ast::CodegenLLVM::visit (this=0x7fffffffe5f0, probe=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1290
#14 0x0000555555771e48 in bpftrace::ast::Probe::accept (this=0x555556384480, v=...) at /home/bgregg/Build/bpftrace/src/ast/ast.cpp:97
#15 0x00005555557817e5 in bpftrace::ast::CodegenLLVM::visit (this=0x7fffffffe5f0, program=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1377
#16 0x0000555555771e7c in bpftrace::ast::Program::accept (this=0x7fffe8011da0, v=...) at /home/bgregg/Build/bpftrace/src/ast/ast.cpp:101
#17 0x0000555555785037 in bpftrace::ast::CodegenLLVM::compile (this=0x7fffffffe5f0, debug=<incomplete type>, out=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1678
#18 0x000055555574ccf1 in main (argc=2, argv=0x7fffffffea38) at /home/bgregg/Build/bpftrace/src/main.cpp:417
    opened by brendangregg 2
  • iosched.bt: Linux 5.0

    iosched.bt: Linux 5.0

    bpftrace v0.9-236-g012e

    # bpftrace iosched.bt 
Attaching 5 probes...
cannot attach kprobe, probe entry may not exist
Warning: could not attach probe kprobe:blk_start_request, skipping.
cannot attach kprobe, probe entry may not exist
Error attaching probe: 'kprobe:__elv_add_request'
    opened by brendangregg 1
  • blostacks.bt kprobe:blk_start_request

    blostacks.bt kprobe:blk_start_request

    This function was removed in 5.0 (as 5.0 went multi-queue only).

    Warning: could not attach probe kprobe:blk_start_request, skipping.

    The tool works anyway, since it traces both old and new functions. It just prints a warning on 5.0+, which is ok. I'm going to assume that this behavior -- printing a warning -- is not going to change, and document it as such.

    opened by brendangregg 1
  • Slow startup of skblife (and possible others)

    Slow startup of skblife (and possible others)

    skblife is taking >3 seconds to start up. Using perf it shows that most of the time is spent on resolve_kname. I opened a PR to fix that (https://github.com/iovisor/bpftrace/pull/765).

    opened by mmarchini 1
  • kvmexits.bt

    kvmexits.bt

    When running kvmexits.bt on Ubuntu 20.04/5.4.0-31-generic, I'm not getting correct exit reason info from kvm_exit calls. Any idea what could be going wrong?

    Example output below:

    $ sudo bpftrace kvmexits.bt
Attaching 4 probes...
Tracing KVM exits. Ctrl-C to end

@exit_ns[97, ]:
[256, 512)             1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|

@exit_ns[119, ]:
[512, 1K)              1 |@@@@@@@@@@@@@@@@@@@@@@@@@@                          |
[1K, 2K)               2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2K, 4K)               1 |@@@@@@@@@@@@@@@@@@@@@@@@@@                          |

@exit_ns[4, ]:
[256, 512)            28 |@@@@@@@                                             |
[512, 1K)            188 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|

@exit_ns[114, ]:
[256, 512)           205 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)             37 |@@@@@@@@@                                           |
[1K, 2K)               6 |@                                                   |

@exit_ns[20, VMLAUNCH]:
[512, 1K)             33 |@@@@@@@@@                                           |
[1K, 2K)              54 |@@@@@@@@@@@@@@@                                     |
[2K, 4K)             116 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                    |
[4K, 8K)             187 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8K, 16K)             42 |@@@@@@@@@@@                                         |

@exit_ns[100, ]:
[256, 512)           178 |@@@@@@@@@                                           |
[512, 1K)            986 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1K, 2K)              10 |                                                    |

@exit_ns[1024, ]:
[512, 1K)              1 |                                                    |
[1K, 2K)              35 |@                                                   |
[2K, 4K)             139 |@@@@@                                               |
[4K, 8K)            1234 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8K, 16K)            297 |@@@@@@@@@@@@                                        |
[16K, 32K)            27 |@                                                   |
[32K, 64K)             1 |                                                    |

@exit_ns[96, ]:
[256, 512)           932 |@@@@@@@@@@@@@@                                      |
[512, 1K)           3458 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1K, 2K)            1866 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@                        |
[2K, 4K)             150 |@@                                                  |
[4K, 8K)               5 |                                                    |
[8K, 16K)              1 |                                                    |

@exit_ns[24, VMRESUME]:
[256, 512)         13683 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)           1612 |@@@@@@                                              |
[1K, 2K)              68 |                                                    |
[2K, 4K)               1 |                                                    |

@exit_ns[124, ]:
[256, 512)          6768 |@@@@@                                               |
[512, 1K)          59247 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1K, 2K)           36592 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                    |
[2K, 4K)           11950 |@@@@@@@@@@                                          |
[4K, 8K)            1000 |                                                    |
[8K, 16K)           5791 |@@@@@                                               |
[16K, 32K)          6892 |@@@@@@                                              |
[32K, 64K)             3 |                                                    |

@exit_ns[123, ]:
[256, 512)        111656 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)          10590 |@@@@                                                |
[1K, 2K)             243 |                                                    |
[2K, 4K)           85229 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@             |
[4K, 8K)           35191 |@@@@@@@@@@@@@@@@                                    |
[8K, 16K)           2102 |                                                    |
[16K, 32K)             1 |                                                    |

@exit_ns[120, ]:
[256, 512)            89 |                                                    |
[512, 1K)           3561 |                                                    |
[1K, 2K)            7425 |@                                                   |
[2K, 4K)            9633 |@@                                                  |
[4K, 8K)           13830 |@@@                                                 |
[8K, 16K)           9828 |@@                                                  |
[16K, 32K)         14671 |@@@                                                 |
[32K, 64K)          9344 |@@                                                  |
[64K, 128K)         7860 |@@                                                  |
[128K, 256K)        9456 |@@                                                  |
[256K, 512K)       45928 |@@@@@@@@@@@@                                        |
[512K, 1M)        193559 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1M, 2M)              53 |                                                    |
    opened by zrait 0
  • fix minor typos in nvmelatency.bt

    fix minor typos in nvmelatency.bt

    1. In nvme_complete_rq, we need to delete @start and @cmd entries for arg0 as key instead of tid
    2. In nvme_setup_cmd, the if condition for rq_disk will error out on CentOS 8, unless explicitly compared with 0

    Signed-off-by: Venkat Ramesh [email protected]

    opened by venkatrag1 0
  • 5.9.2 bpftrace -l 'usdt:*' doesnt run on new version.

    5.9.2 bpftrace -l 'usdt:*' doesnt run on new version.

    book section 5.9.2, page 150, quote"""

    bpftrace -l 'usdt:/usr/local/cpython/python'

    usdt:/usr/local/cpython/python:line usdt:/usr/local/cpython/python:function__entry usdt:/usr/local/cpython/python:function__return ...... """

    on my env: Oracle Linux 7.9 kernel: 5.4.17-2136.304.1.el7uek.x86_64

    bpftrace-0.13.0-2.el7.x86_64 $bpftrace --version bpftrace v0.13.0

    [root@EL7 ~]$bpftrace -l 'usdt:/usr/bin/python' stdin:1:1-21: ERROR: usdt probe type requires 2 or 3 arguments usdt:/usr/bin/python

    
seems bpftrace -l parameter changes in newer versions.
    opened by asmca 0
  • refactor mmapsnoop: retrieve mmaped filename from current_task

    refactor mmapsnoop: retrieve mmaped filename from current_task

    Addresses the TODO in the mmapsnoop code:

    • traverse the current_task struct to retrieve the filename from the open fds
    • remove the kprobes used to track fd2file association
    opened by valer-cara 0
  • jnistacks.bt

    jnistacks.bt

    In jnistacks.bt, the call to jmaps throws "Error occurred during initialization of VM Could not allocate metaspace: 1073741824 bytes". The jmaps by itself works fine. Tested out for multiple jvms. Ubuntu 16.04, kernel 4.15 and bpftrace-0.9.4

    opened by slowbreathing 0
  • 6.3.10 syscount

    6.3.10 syscount

    The -P option can be used to count by process ID instead:

    should be lower case p, i.e.

    The -p option can be used to count by process ID instead:

    Also includes the command line sample:

    #syscount -Pi 1

    opened by chrisgibson01 0
  • tcpconnect-tp.bt doesn't work in Ubuntu 18.04 w/ 4.19 kernel

    tcpconnect-tp.bt doesn't work in Ubuntu 18.04 w/ 4.19 kernel 

    ubuntu@ip-172-31-18-109:~$ sudo bpftrace tcpconnect-tp.bt 
tcpconnect-tp.bt:30:38-69: ERROR: ntop() expects an integer or array argument, got ctx
        printf("%-15s %-15s %-5d\n", ntop(args->family, args->saddr),
                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tcpconnect-tp.bt:31:13-44: ERROR: ntop() expects an integer or array argument, got ctx
            ntop(args->family, args->daddr), args->dport)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    environment

    ubuntu@ip-172-31-18-109:~$ uname -sr
Linux 4.19.0-041900-generic

    ubuntu@ip-172-31-18-109:~$ bpftrace --version
bpftrace v0.9.4-97-gef39
    opened by Obsecurus 1
Owner
Brendan Gregg
Cloud computing performance architect and engineer.
Brendan Gregg
GitHub
This program tries to book a tennis court slot in either Southwark Park or Tanner Street Park in Southwark, London.

Book tennis courts in London This program tries to book a tennis court slot in either Southwark Park or Tanner Street Park in Southwark, London. Note:

Daniele 1 Jul 25, 2022
That is a example of a Book app on Python, made with support of all JS libraries on React framework

React+Python Books App You can use this repository whenever you want Used for a video Create the database: python -m dbutils Start the web server: pyt

Koma Human 1 Apr 20, 2022
Open source book about making Python packages.

Python packages Tomas Beuzen & Tiffany Timbers Python packages are a core element of the Python programming language and are how you create organized,

Python Packages 169 Jan 6, 2023
x-tools is a collection of tools developed in Python

x-tools X-tools is a collection of tools developed in Python Commands\

null 5 Jan 24, 2022
This repository contains various tools useful for offensive operations (reversing, etc) regarding the PE (Portable Executable) format

PE-Tools This repository contains various tools useful for offensive operations (reversing, etc) regarding the PE (Portable Executable) format Install

stark0de 4 Oct 13, 2022
DownTime-Score is a Small project aimed to Monitor the performance and the availabillity of a variety of the Vital and Critical Moroccan Web Portals

DownTime-Score DownTime-Score is a Small project aimed to Monitor the performance and the availabillity of a variety of the Vital and Critical Morocca

adnane-tebbaa 5 Apr 30, 2022
Performance data for WASM SIMD instructions.

WASM SIMD Data This repository contains code and data which can be used to generate a JSON file containing information about the WASM SIMD proposal. F

Evan Nemerson 5 Jul 24, 2022
This is an online course where you can learn and master the skill of low-level performance analysis and tuning.

Performance Ninja Class This is an online course where you can learn to find and fix low-level performance issues, for example CPU cache misses and br

Denis Bakhvalov 1.2k Dec 30, 2022
EasyBuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way.

EasyBuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way.

EasyBuild community 87 Dec 27, 2022
Developed a website to analyze and generate report of students based on the curriculum that represents student’s academic performance.

Developed a website to analyze and generate report of students based on the curriculum that represents student’s academic performance. We have developed the system such that, it will automatically parse data onto the database from excel file, which will in return reduce time consumption of analysis of data.

VIJETA CHAVHAN 3 Nov 8, 2022
An interactive tool with which to explore the possible imaging performance of candidate ngEHT architectures.

ngEHTexplorer An interactive tool with which to explore the possible imaging performance of candidate ngEHT architectures. Welcome! ngEHTexplorer is a

Avery Broderick 7 Jan 28, 2022
Terrible sudoku solver with spaghetti code and performance issues

SudokuSolver Terrible sudoku solver with spaghetti code and performance issues - if it's unable to figure out next step it will stop working, it never

Kamil Bizoń 1 Dec 5, 2021
Performance monitoring and testing of OpenStack

Browbeat Browbeat is a performance tuning and analysis tool for OpenStack. Browbeat is free, Open Source software. Analyze and tune your Cloud for opt

cloud-bulldozer 83 Dec 14, 2022
Defichain maxi - Scripts to optimize performance on defichain rewards

defichain_maxi This script is made to optimize your defichain vault rewards by m

kuegi 75 Dec 31, 2022
Collie is for uncovering RDMA NIC performance anomalies

Collie is for uncovering RDMA NIC performance anomalies. Overview Prerequ

Bytedance Inc. 34 Dec 11, 2022
The Official Jaseci Code Repository

Jaseci Release Notes Version 1.2.2 Updates Added new built-ins for nodes and edges (context, info, and details) Fixed dot output Added reset command t

null 136 Dec 20, 2022
Script to change official Kali repository to mirrors

Script to change official Kali repository to mirrors. This helps increase packages update and downloading for some user.

Vineet Bhavsar 2 Nov 29, 2021
🏆 A ranked list of awesome Python open-source libraries and tools. Updated weekly.

Best-of Python ?? A ranked list of awesome Python open-source libraries & tools. Updated weekly. This curated list contains 230 awesome open-source pr

Machine Learning Tooling 2.7k Jan 3, 2023
A fancy and practical functional tools

Funcy A collection of fancy functional tools focused on practicality. Inspired by clojure, underscore and my own abstractions. Keep reading to get an

Alexander Schepanovski 2.9k Dec 29, 2022