Related to https://github.com/iovisor/bpftrace/issues/686

There's only one script which defines variables inside an if-block and then uses them outside the block: https://github.com/brendangregg/bpf-perf-tools-book/blob/d928209118dd4ab8b3727bd7cc1c9c83b4f695b2/Ch10_Networking/tcpretrans.bt#L53

Ideally what this script does should be fine, since $daddr and $saddr and defined in both branches of the if/else, but I think it'd be very hard (impossible?) to confirm this is true in the general case at compile time, so we should require variables to be declared in the scope they are used in, or a parent scope, like C does.

To be compatible with a future bpftrace which has variable scoping, this bit of code:

    if ($inet_family == AF_INET) {
      $daddr = ntop($sk->__sk_common.skc_daddr);
      $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
    } else {
      $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
      $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
    }

should be changed to:

    $daddr = 0;
    $saddr = 0;
    if ($inet_family == AF_INET) {
      $daddr = ntop($sk->__sk_common.skc_daddr);
      $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
    } else {
      $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
      $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
    }

This tool worked not that long ago, but now hits an Assertion:

# tcplife.bt 
bpftrace: /usr/include/llvm-7/llvm/IR/Instructions.h:1117: void llvm::ICmpInst::AssertOK(): Assertion `getOperand(0)->getType() == getOperand(1)->getType() && "Both operands to ICmp instruction are not of the same type!"' failed.
Aborted (core dumped)

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fffef922801 in __GI_abort () at abort.c:79
#2  0x00007fffef91239a in __assert_fail_base (fmt=0x7fffefa997d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x5555557ebbc8 "getOperand(0)->getType() == getOperand(1)->getType() && \"Both operands to ICmp instruction are not of the same type!\"", 
    file=file@entry=0x5555557eba78 "/usr/include/llvm-7/llvm/IR/Instructions.h", line=line@entry=1117, function=function@entry=0x5555557ed820 <llvm::ICmpInst::AssertOK()::__PRETTY_FUNCTION__> "void llvm::ICmpInst::AssertOK()")
    at assert.c:92
#3  0x00007fffef912412 in __GI___assert_fail (assertion=0x5555557ebbc8 "getOperand(0)->getType() == getOperand(1)->getType() && \"Both operands to ICmp instruction are not of the same type!\"", 
    file=0x5555557eba78 "/usr/include/llvm-7/llvm/IR/Instructions.h", line=1117, function=0x5555557ed820 <llvm::ICmpInst::AssertOK()::__PRETTY_FUNCTION__> "void llvm::ICmpInst::AssertOK()") at assert.c:101
#4  0x000055555578d2a6 in llvm::ICmpInst::AssertOK (this=0x5555562f1810) at /usr/include/llvm-7/llvm/IR/Instructions.h:1116
#5  0x000055555578d3a2 in llvm::ICmpInst::ICmpInst (this=0x5555562f1810, pred=llvm::CmpInst::ICMP_NE, LHS=0x555556157e10, RHS=0x55555604b8c0, NameStr=...) at /usr/include/llvm-7/llvm/IR/Instructions.h:1171
#6  0x00005555557922d1 in llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>::CreateICmp (this=0x7fffffffe618, P=llvm::CmpInst::ICMP_NE, LHS=0x555556157e10, RHS=0x55555604b8c0, Name=...)
    at /usr/include/llvm-7/llvm/IR/IRBuilder.h:1851
#7  0x00005555557924b2 in llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>::CreateICmpNE (this=0x7fffffffe618, LHS=0x555556157e10, RHS=0x55555604b8c0, Name=...) at /usr/include/llvm-7/llvm/IR/IRBuilder.h:1741
#8  0x0000555555782956 in bpftrace::ast::CodegenLLVM::createLogicalAnd (this=0x7fffffffe5f0, binop=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1480
#9  0x000055555577d09c in bpftrace::ast::CodegenLLVM::visit (this=0x7fffffffe5f0, binop=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:793
#10 0x0000555555771bae in bpftrace::ast::Binop::accept (this=0x7fffe851c180, v=...) at /home/bgregg/Build/bpftrace/src/ast/ast.cpp:45
#11 0x000055555577fc19 in bpftrace::ast::CodegenLLVM::visit (this=0x7fffffffe5f0, if_block=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1159
#12 0x0000555555771de0 in bpftrace::ast::If::accept (this=0x7fffe8a35eb0, v=...) at /home/bgregg/Build/bpftrace/src/ast/ast.cpp:89
#13 0x0000555555780be4 in bpftrace::ast::CodegenLLVM::visit (this=0x7fffffffe5f0, probe=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1290
#14 0x0000555555771e48 in bpftrace::ast::Probe::accept (this=0x555556384480, v=...) at /home/bgregg/Build/bpftrace/src/ast/ast.cpp:97
#15 0x00005555557817e5 in bpftrace::ast::CodegenLLVM::visit (this=0x7fffffffe5f0, program=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1377
#16 0x0000555555771e7c in bpftrace::ast::Program::accept (this=0x7fffe8011da0, v=...) at /home/bgregg/Build/bpftrace/src/ast/ast.cpp:101
#17 0x0000555555785037 in bpftrace::ast::CodegenLLVM::compile (this=0x7fffffffe5f0, debug=<incomplete type>, out=...) at /home/bgregg/Build/bpftrace/src/ast/codegen_llvm.cpp:1678
#18 0x000055555574ccf1 in main (argc=2, argv=0x7fffffffea38) at /home/bgregg/Build/bpftrace/src/main.cpp:417

bpftrace v0.9-236-g012e

# bpftrace iosched.bt 
Attaching 5 probes...
cannot attach kprobe, probe entry may not exist
Warning: could not attach probe kprobe:blk_start_request, skipping.
cannot attach kprobe, probe entry may not exist
Error attaching probe: 'kprobe:__elv_add_request'

This function was removed in 5.0 (as 5.0 went multi-queue only).

Warning: could not attach probe kprobe:blk_start_request, skipping.

The tool works anyway, since it traces both old and new functions. It just prints a warning on 5.0+, which is ok. I'm going to assume that this behavior -- printing a warning -- is not going to change, and document it as such.

skblife is taking >3 seconds to start up. Using perf it shows that most of the time is spent on resolve_kname. I opened a PR to fix that (https://github.com/iovisor/bpftrace/pull/765).

When running kvmexits.bt on Ubuntu 20.04/5.4.0-31-generic, I'm not getting correct exit reason info from kvm_exit calls. Any idea what could be going wrong?

Example output below:

$ sudo bpftrace kvmexits.bt
Attaching 4 probes...
Tracing KVM exits. Ctrl-C to end

@exit_ns[97, ]:
[256, 512)             1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|

@exit_ns[119, ]:
[512, 1K)              1 |@@@@@@@@@@@@@@@@@@@@@@@@@@                          |
[1K, 2K)               2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2K, 4K)               1 |@@@@@@@@@@@@@@@@@@@@@@@@@@                          |

@exit_ns[4, ]:
[256, 512)            28 |@@@@@@@                                             |
[512, 1K)            188 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|

@exit_ns[114, ]:
[256, 512)           205 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)             37 |@@@@@@@@@                                           |
[1K, 2K)               6 |@                                                   |

@exit_ns[20, VMLAUNCH]:
[512, 1K)             33 |@@@@@@@@@                                           |
[1K, 2K)              54 |@@@@@@@@@@@@@@@                                     |
[2K, 4K)             116 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                    |
[4K, 8K)             187 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8K, 16K)             42 |@@@@@@@@@@@                                         |

@exit_ns[100, ]:
[256, 512)           178 |@@@@@@@@@                                           |
[512, 1K)            986 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1K, 2K)              10 |                                                    |

@exit_ns[1024, ]:
[512, 1K)              1 |                                                    |
[1K, 2K)              35 |@                                                   |
[2K, 4K)             139 |@@@@@                                               |
[4K, 8K)            1234 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8K, 16K)            297 |@@@@@@@@@@@@                                        |
[16K, 32K)            27 |@                                                   |
[32K, 64K)             1 |                                                    |

@exit_ns[96, ]:
[256, 512)           932 |@@@@@@@@@@@@@@                                      |
[512, 1K)           3458 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1K, 2K)            1866 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@                        |
[2K, 4K)             150 |@@                                                  |
[4K, 8K)               5 |                                                    |
[8K, 16K)              1 |                                                    |

@exit_ns[24, VMRESUME]:
[256, 512)         13683 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)           1612 |@@@@@@                                              |
[1K, 2K)              68 |                                                    |
[2K, 4K)               1 |                                                    |

@exit_ns[124, ]:
[256, 512)          6768 |@@@@@                                               |
[512, 1K)          59247 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1K, 2K)           36592 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                    |
[2K, 4K)           11950 |@@@@@@@@@@                                          |
[4K, 8K)            1000 |                                                    |
[8K, 16K)           5791 |@@@@@                                               |
[16K, 32K)          6892 |@@@@@@                                              |
[32K, 64K)             3 |                                                    |

@exit_ns[123, ]:
[256, 512)        111656 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)          10590 |@@@@                                                |
[1K, 2K)             243 |                                                    |
[2K, 4K)           85229 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@             |
[4K, 8K)           35191 |@@@@@@@@@@@@@@@@                                    |
[8K, 16K)           2102 |                                                    |
[16K, 32K)             1 |                                                    |

@exit_ns[120, ]:
[256, 512)            89 |                                                    |
[512, 1K)           3561 |                                                    |
[1K, 2K)            7425 |@                                                   |
[2K, 4K)            9633 |@@                                                  |
[4K, 8K)           13830 |@@@                                                 |
[8K, 16K)           9828 |@@                                                  |
[16K, 32K)         14671 |@@@                                                 |
[32K, 64K)          9344 |@@                                                  |
[64K, 128K)         7860 |@@                                                  |
[128K, 256K)        9456 |@@                                                  |
[256K, 512K)       45928 |@@@@@@@@@@@@                                        |
[512K, 1M)        193559 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1M, 2M)              53 |                                                    |

1. In nvme_complete_rq, we need to delete @start and @cmd entries for arg0 as key instead of tid
2. In nvme_setup_cmd, the if condition for rq_disk will error out on CentOS 8, unless explicitly compared with 0

Signed-off-by: Venkat Ramesh [email protected]

book section 5.9.2, page 150, quote"""

bpftrace -l 'usdt:/usr/local/cpython/python'

usdt:/usr/local/cpython/python:line usdt:/usr/local/cpython/python:function__entry usdt:/usr/local/cpython/python:function__return ...... """

on my env: Oracle Linux 7.9 kernel: 5.4.17-2136.304.1.el7uek.x86_64

bpftrace-0.13.0-2.el7.x86_64 $bpftrace --version bpftrace v0.13.0

[root@EL7 ~]$bpftrace -l 'usdt:/usr/bin/python' stdin:1:1-21: ERROR: usdt probe type requires 2 or 3 arguments usdt:/usr/bin/python

seems bpftrace -l parameter changes in newer versions.

Addresses the TODO in the mmapsnoop code:

• traverse the current_task struct to retrieve the filename from the open fds
• remove the kprobes used to track fd2file association

In jnistacks.bt, the call to jmaps throws "Error occurred during initialization of VM Could not allocate metaspace: 1073741824 bytes". The jmaps by itself works fine. Tested out for multiple jvms. Ubuntu 16.04, kernel 4.15 and bpftrace-0.9.4

The -P option can be used to count by process ID instead:

should be lower case p, i.e.

The -p option can be used to count by process ID instead:

Also includes the command line sample:

#syscount -Pi 1

ubuntu@ip-172-31-18-109:~$ sudo bpftrace tcpconnect-tp.bt 
tcpconnect-tp.bt:30:38-69: ERROR: ntop() expects an integer or array argument, got ctx
        printf("%-15s %-15s %-5d\n", ntop(args->family, args->saddr),
                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tcpconnect-tp.bt:31:13-44: ERROR: ntop() expects an integer or array argument, got ctx
            ntop(args->family, args->daddr), args->dport)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

environment

ubuntu@ip-172-31-18-109:~$ uname -sr
Linux 4.19.0-041900-generic

ubuntu@ip-172-31-18-109:~$ bpftrace --version
bpftrace v0.9.4-97-gef39