Linux BCC (BPF Compiler Collection) — Complete Learning Notes & Output Guide

BCC (BPF Compiler Collection) is a toolkit and framework for creating efficient kernel tracing and manipulation programs using extended Berkeley Packet Filters (eBPF). It packages front-ends in Python, C++, and Lua on top of an LLVM/Clang compiler backend to load verified bytecode directly into the running Linux kernel.

---

1. What is BCC?

Historically, tracing Linux kernel internals required either writing and compiling out-of-tree kernel modules (which could crash the system) or using static tracers with limited programmability.

BCC solved this by allowing developers to write C code for the in-kernel probe logic and wrap it in a Python user-space harness that handles program loading, map reading, and output formatting.

+-------------------------------------------------------------------------+
|                              USER SPACE                                 |
|                                                                         |
|   +-----------------------------------------------------------------+   |
|   |         Python Script / BCC Tool (CLI Interface)                |   |
|   |   - Defines C probe source code as a multiline string           |   |
|   |   - Polls BPF ring buffers / maps                               |   |
|   |   - Formats human-readable output, histograms, and summaries    |   |
|   +-----------------------------------------------------------------+   |
|                                    |                                    |
|                                    | Invokes libbpf / Clang JIT         |
|                                    v                                    |
|   +-----------------------------------------------------------------+   |
|   |         Clang / LLVM JIT Compiler Infrastructure                |   |
|   |   - Parses and compiles in-line C code to eBPF bytecode         |   |
|   +-----------------------------------------------------------------+   |
|                                    |                                    |
|                                    | bpf(BPF_PROG_LOAD) system call     |
+------------------------------------|------------------------------------+
                                     v
+-------------------------------------------------------------------------+
|                         LINUX KERNEL SPACE                              |
|                                                                         |
|   +-----------------------------------------------------------------+   |
|   |                       eBPF In-Kernel Verifier                   |   |
|   |   - Validates memory safety, prevents unbounded loops/crashes   |   |
|   +-----------------------------------------------------------------+   |
|                                    |                                    |
|                                    | Hooks into execution point         |
|                                    v                                    |
|   +-----------------------------------------------------------------+   |
|   |            Instrumentation Points (Kprobes, Tracepoints)         |   |
|   |   - Executes BPF bytecode safely at native CPU speed            |   |
|   +-----------------------------------------------------------------+   |
|                                    |                                    |
|                                    | Aggregates data into               |
|                                    v                                    |
|   +-----------------------------------------------------------------+   |
|   |                 BPF Maps (Hash, Array, Perf Ring Buffer)        |   |
|   +-----------------------------------------------------------------+   |
+------------------------------------|------------------------------------+
                                     |
                                     +---> Read by Python user-space script

It answers production-grade diagnostic questions:

• Which system functions are taking longer than $10\,\text{ms}$ to return?
• What is the latency distribution (histogram) of storage I/O or runqueue scheduling?
• Which specific queries, sockets, or TCP connections are experiencing retransmissions?
• Can we aggregate metrics inside the kernel so user space only reads summary tables instead of millions of individual event streams?

---

2. Installation

BCC requires root privileges (sudo) or CAP_BPF / CAP_SYS_ADMIN, along with matching kernel header files.

Debian / Ubuntu

sudo apt update
sudo apt install bpfcc-tools python3-bpfcc linux-headers-$(uname -r)

(Binaries install with the -bpfcc suffix in /usr/sbin/, e.g., biolatency-bpfcc).

RHEL / Rocky / AlmaLinux / CentOS

sudo dnf install bcc-tools kernel-devel-$(uname -r)

(Executables reside in /usr/share/bcc/tools/).

Arch Linux

sudo pacman -S bcc bcc-tools python-bcc linux-headers

Verify your environment:

sudo /usr/share/bcc/tools/execsnoop -h 2>/dev/null || sudo execsnoop-bpfcc -h

---

3. Categorized Overview of Core BCC Tools

BCC ships with over 100 specialized performance-analysis tools. The most critical tools span several operating system subsystems:

Subsystem	Core BCC Tools	Diagnostic Role
CPU & Scheduler	execsnoop, runqlat, cpudist, profile	Tracks process lifecycles, CPU runqueue scheduling latency, and CPU cycles.
Block I/O & Storage	biolatency, biosnoop, bitesize, ext4slower	Measures block device latency histograms, I/O sizes, and VFS file operation stalls.
Networking	tcplife, tcpconnect, tcpretrans, tcpdrop	Tracks TCP connection durations, connection attempts, retransmits, and dropped packets.
Memory	memleak, oomkill, shmsnoop	Detects user/kernel memory leaks and tracks kernel Out-Of-Memory events.
Filesystem & VFS	opensnoop, filelife, filetop, syncsnoop	Traces file opens, short-lived file churn, and sync operations.

---

4. Deep-Dive: Output Analysis of Key BCC Tools

---

4.1 Block Device Latency: biolatency

biolatency calculates the time it takes for block I/O requests to be serviced by storage hardware and prints an in-kernel log2 histogram.

Command

sudo biolatency -m 5

(-m outputs timestamps in milliseconds instead of microseconds; 5 runs for a 5-second interval).

Raw Output

Tracing block device I/O... Hit Ctrl-C to end or wait 5 seconds.

     msecs               : count     distribution
         0 -> 1          : 45210    |****************************************|
         2 -> 3          : 1204     |*                                       |
         4 -> 7          : 342      |                                        |
         8 -> 15         : 89       |                                        |
        16 -> 31         : 14       |                                        |
        32 -> 63         : 3        |                                        |
        64 -> 127        : 1        |                                        |

Breakdown of Headings & Output Fields

Heading / Bucket	Example	Technical Meaning
Interval Range (msecs)	0 -> 1, 16 -> 31	The logarithmic latency bin (lower bound $\to$ upper bound) representing request completion time in milliseconds.
count	45210, 1	The total number of disk I/O requests that fell into that exact latency window.
distribution	`	****...

Troubleshooting Analysis

• Healthy Fast Storage: More than 95% of events sit in the 0 -> 1 ms bucket.
• Storage Bottleneck: Notice the single request in 64 -> 127. If counts begin clustering in the $>32\,\text{ms}$ ranges, physical disks are experiencing queue stall delays or NVMe drive controller stalls.

---

4.2 Tracking Process Lifecycles: execsnoop

execsnoop records newly spawned processes by tracing the execve() system call. It captures transient, short-lived commands that exit too fast to show up in top or ps.

Command

sudo execsnoop -T

Raw Output

TIME(s)   PCOMM            PID    PPID   RET ARGS
15:02:10  git              14201  14190    0 /usr/bin/git status --porcelain
15:02:10  sed              14205  14201    0 /usr/bin/sed -e s/pattern/replace/
15:02:11  sh               14210   1201    0 /bin/sh -c ./deploy.sh
15:02:11  deploy.sh        14210   1201    0 ./deploy.sh
15:02:11  curl             14212  14210   -2 /usr/bin/curl https://invalid.internal.local

Breakdown of Headings & Output Fields

Heading	Example	Technical Meaning
TIME(s)	15:02:10	Wall-clock timestamp (Hours:Minutes:Seconds) when the process attempted execution.
PCOMM	git, sed	Process command name executed via execve.
PID	14201	The Process ID assigned to the new task.
PPID	14190	The Parent Process ID that issued the fork/exec.
RET	0, -2	The return code of the execve() syscall. 0 indicates success; negative numbers represent kernel errors (e.g., -2 for -ENOENT if the executable binary does not exist).
ARGS	/usr/bin/git status ...	The full CLI parameter arguments string passed into the execution vector.

---

4.3 Network Connection Lifespans: tcplife

tcplife measures the duration of established TCP sessions from initial connection establishment to socket close, along with byte throughput.

Command

sudo tcplife

Raw Output

PID   COMM       LADDR           LPORT RADDR           RPORT  TX_KB  RX_KB MS
4512  python3    192.168.1.50    54210 10.0.0.5        5432      12    450 142.12
8910  nginx      192.168.1.50    443   172.16.0.12     61200    850     10  12.05
1201  curl       192.168.1.50    48912 93.184.216.34   80         1      5 5002.10

Breakdown of Headings & Output Fields

Heading	Example	Technical Meaning
PID	4512	Operating system Process ID managing the network socket.
COMM	python3	Executable name associated with the connection.
**LADDR / LPORT**	192.168.1.50:54210	Local IP address and source port.
**RADDR / RPORT**	10.0.0.5:5432	Remote destination IP address and service port.
TX_KB	12	Total Kilobytes transmitted (sent) over the lifetime of the connection.
RX_KB	450	Total Kilobytes received over the lifetime of the connection.
MS	142.12	Total connection lifespan in milliseconds (from completed handshake to FIN/RST close).

Troubleshooting Analysis

• Slow Database Calls: Notice the python3 row connecting to 5432 (PostgreSQL). It transferred only 12 KB out and 450 KB in, but took 142.12 ms—highlighting either a slow query or network transit delay.
• Connection Timeouts: The curl connection to port 80 lasted 5002.10 ms (~5 seconds), indicating an application hit a network timeout.

---

4.4 Tracing Slow Filesystem I/O: ext4slower / xfs_slower

Filesystem tracers hook into VFS and concrete filesystem drivers (ext4, xfs, btrfs, zfs) to log individual operations that exceed a configurable latency threshold.

Command

# Log any ext4 read, write, open, or sync operation taking longer than 10 milliseconds
sudo ext4slower 10

Raw Output

TIME     COMM           PID    T S OFF_KB   BYTES  FAST  MS     FILE
15:10:02 postgres       5410   W S 1048576  8192      0  14.25  base/16384/2610
15:10:05 rsyslogd       1120   S S 0        0         0  42.10  syslog
15:10:08 python3        8912   R S 4096     4096      0  11.20  large_dataset.arrow

Breakdown of Headings & Output Fields

Heading	Example	Technical Meaning
TIME	15:10:02	Timestamp when the slow operation completed.
COMM	postgres	Process command name.
PID	5410	Process ID executing the filesystem call.
T (Type)	W, R, S, O	Operation type: R (Read), W (Write), O (Open), S (Sync/fsync).
S (Status)	S	Status flag indicating operation completion.
OFF_KB	1048576	Target file offset where the read/write began, in Kilobytes.
BYTES	8192	Size of the I/O operation in bytes.
FAST	0	Indicates whether the request was fulfilled via page cache/fast path (1) or hit disk (0).
MS	14.25	Execution latency of the operation in milliseconds.
FILE	base/16384/2610	The relative or absolute path of the affected file.

---

5. Writing a Custom BCC Tool (Python + In-Kernel C)

One of BCC's primary advantages is the ability to write custom, ad-hoc kernel tracers using Python.

Below is an executable script that attaches to the kernel's sys_clone (process creation) entry point, increments an in-kernel hash map, and prints the calling process name:

#!/usr/bin/env python3
from bcc import BPF
from time import sleep

# 1. In-kernel C program executed inside the Linux kernel sandbox
bpf_source = """
#include <uapi/linux/ptrace.h>

// Define a BPF hash map: key=char[16] (comm), value=u64 (count)
BPF_HASH(clone_counts, char[16], u64);

int trace_clone_entry(struct pt_regs *ctx) {
    char comm[16];
    u64 zero = 0, *val;

    // Read current process executable name into comm buffer
    bpf_get_current_comm(&comm, sizeof(comm));

    // Lookup or initialize counter in map
    val = clone_counts.lookup_or_try_init(comm, &zero);
    if (val) {
        (*val)++;
    }
    return 0;
}
"""

# 2. Compile and load BPF program using in-memory Clang/LLVM
b = BPF(text=bpf_source)

# 3. Attach probe to the clone system call kernel routine
clone_fn = b.get_syscall_fnname("clone")
b.attach_kprobe(event=clone_fn, fn_name="trace_clone_entry")

print("Tracing process forks/clones... Hit Ctrl-C to end.")

# 4. Read data from kernel map in user space
try:
    sleep(5)
except KeyboardInterrupt:
    pass

print(f"\n{'COMMAND':<20} {'FORK_COUNT'}")
print("-" * 32)
counts = b.get_table("clone_counts")
for key, value in sorted(counts.items(), key=lambda kv: kv[1].value, reverse=True):
    print(f"{key.value.decode('utf-8'):<20} {value.value}")

Execution Output

COMMAND              FORK_COUNT
--------------------------------
bash                 12
worker_pool          8
python3              2

---

6. BCC vs. bpftrace vs. libbpf / CO-RE

Modern Linux tracing encompasses three major eBPF paradigms:

+-------------------------------------------------------------------------+
|                               bpftrace                                  |
|  * High-level domain-specific scripting language (awk-like syntax).     |
|  * Best for: Fast, one-line ad-hoc terminal diagnostics.               |
|  * Example: bpftrace -e 'kprobe:vfs_read { @[comm] = count(); }'       |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
|                        BCC (BPF Compiler Collection)                    |
|  * Python/C++ integration with dynamic runtime LLVM compilation.        |
|  * Best for: Complex multi-map tools, GUI integrations, custom CLIs.   |
|  * Trade-off: High memory footprint (requires Clang compiler on node).  |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
|                       libbpf + CO-RE (Modern Standard)                  |
|  * Compile Once, Run Everywhere (CO-RE) using BTF (BPF Type Format).    |
|  * Standalone, small binary footprint. Zero runtime compiler overhead. |
|  * Best for: Production monitoring agents (Datadog, Cilium, Katran).   |
+-------------------------------------------------------------------------+

Dimension	BCC	bpftrace	libbpf + CO-RE
Primary Interface	Python / C++	Custom awk-like DSL	Native C / Rust / Go
Compilation Model	Runtime Clang/LLVM JIT	Runtime JIT	Ahead-Of-Time (AOT) via clang
System Overhead	Heavy memory usage during compile (~80-150 MB)	Lightweight (~10-20 MB)	Tiny footprint (~1-5 MB)
Development Speed	Moderate	Very Fast	Slower (requires strict structs)
Target Audience	Custom tooling & complex scripts	Interactive troubleshooting	Low-overhead production daemons

---

7. Important Interview Questions & Answers

Q: Why does running a BCC tool consume 100+ MB of RAM when starting up?

Answer: BCC embeds the complete Clang and LLVM compiler toolchain into its Python bindings. When you execute a BCC script, user space compiles the inline C source code into eBPF bytecode at runtime, pulling in Linux kernel C headers for dynamic struct offsets. This JIT compilation phase consumes noticeable CPU cycles and memory. Once compiled and loaded into the kernel, however, the running eBPF bytecode executes with minimal overhead.

Q: What is BPF Type Format (BTF) and why is BCC transitioning toward CO-RE?

Answer: Traditionally, BCC required kernel headers (linux-headers-$(uname -r)) to be installed on every target production server so Clang could resolve struct definitions (like task_struct) for that specific kernel. BTF (BPF Type Format) embeds compact kernel metadata and struct layouts directly into the kernel image (/sys/kernel/btf/vmlinux). Combined with CO-RE (Compile Once – Run Everywhere), developers can compile eBPF programs once on a development machine; the kernel relocates field offsets at load time, eliminating the need to deploy LLVM compilers and header files to production servers.

Q: How do BCC in-kernel histograms (like in biolatency and runqlat) avoid dropping data?

Answer: Traditional profilers emit a trace stream for every event to user space, which can easily overwhelm user-kernel ring buffers during high-throughput workloads, causing dropped events. BCC tools like biolatency use in-kernel eBPF array/hash maps to increment logarithmic latency counter bins directly inside the kernel execution context. User space only reads the final aggregated bucket counts once per interval, keeping user-kernel communication and context switching near zero.

---