eBPF Linux Monitoring Tutorial: A Beginner’s Guide to System Observability

Welcome to this comprehensive eBPF Linux Monitoring Tutorial designed specifically for beginners. Extended Berkeley Packet Filter (eBPF) has revolutionized Linux system observability, allowing you to run sandboxed programs in the kernel without changing kernel source code or loading kernel modules. This guide will take you from zero knowledge to confidently monitoring your Linux systems with eBPF-powered tools.

What is eBPF and Why Should You Care?

eBPF is a revolutionary technology that enables safe, dynamic programmability of the Linux kernel. Originally designed for packet filtering, eBPF has evolved into a general-purpose execution engine that allows developers and system administrators to hook into kernel functions, trace system calls, monitor network traffic, and analyze performance with minimal overhead.

Traditional monitoring approaches often require kernel modules or patches, creating stability and security risks. eBPF Linux Monitoring Tutorial resources emphasize that eBPF programs run in a sandboxed virtual machine within the kernel, with automatic verification ensuring they cannot crash the system or run indefinitely.

The key benefits of eBPF monitoring include:

  • Low overhead: eBPF programs execute directly in kernel space, avoiding expensive context switches
  • Real-time visibility: Monitor system events as they happen without polling delays
  • Safe execution: The eBPF verifier ensures programs terminate and don’t access invalid memory
  • Dynamic loading: Attach and detach monitoring without restarting services or rebooting

Prerequisites and System Requirements

Before diving into eBPF monitoring, ensure your system meets these requirements:

Kernel Version Requirements

# Check your kernel version
uname -r

# Minimum versions for eBPF features:
# 5.3+  - Basic eBPF data plane support
# 5.8+  - TCP timestamp support
# 5.10+ - LRU conntrack table, improved performance
# 5.14+ - Full feature set including advanced tracing

Most modern distributions shipping kernel 5.10 or later provide excellent eBPF support. Ubuntu 22.04 LTS, RHEL 9, and Debian 12 all include capable kernels.

Verify BPF Filesystem

# Check if BPF filesystem is mounted
mount | grep bpf

# Expected output:
# bpffs on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)

# If not mounted, mount it manually:
sudo mount -t bpf none /sys/fs/bpf

Install Required Packages

# Ubuntu/Debian
sudo apt update
sudo apt install -y linux-headers-$(uname -r) bpftrace bcc-tools libbpfcc-dev

# RHEL/CentOS/Rocky Linux
sudo dnf install -y kernel-headers-$(uname -r) bpftrace bcc-tools

Getting Started with bpftrace

bpftrace is a high-level tracing language for eBPF, perfect for beginners. It provides a simple syntax similar to awk and C for writing powerful one-liners and scripts.

Your First eBPF Program

Let’s start with a simple program that traces all system calls:

# Trace all system calls with their process name
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { printf("%s: %s\n", comm, probe); }'

This one-liner attaches to the raw_syscalls:sys_enter tracepoint and prints the process name (comm) for every system call. Press Ctrl+C to stop tracing.

Monitoring Process Execution

Track which programs are being executed on your system:

# Monitor process execution with arguments
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve {
    printf("PID %d: %s executed %s\n", pid, comm, str(args->filename));
}'

This script shows the power of eBPF—capturing every program execution across the entire system in real-time with minimal overhead.

File Access Monitoring

# Monitor file opens
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
    printf("%s[%d]: open(%s)\n", comm, pid, str(args->filename));
}'

This traces all file open operations, helping you understand which files applications access.

Practical eBPF Monitoring Scenarios

1. Network Connection Monitoring

Monitor all TCP connections in real-time:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/socket.h>

kprobe:tcp_v4_connect {
    printf("TCP connect: %s[%d] to ", comm, pid);
}

kprobe:tcp_v4_connect /comm == "curl"/ {
    $sk = (struct sock *)arg0;
    $daddr = $sk->__sk_common.skc_daddr;
    printf("%d.%d.%d.%d:%d\n",
        ($daddr >> 0) & 0xff,
        ($daddr >> 8) & 0xff,
        ($daddr >> 16) & 0xff,
        ($daddr >> 24) & 0xff,
        $sk->__sk_common.skc_dport >> 8);
}'

This advanced example demonstrates how eBPF can access kernel data structures to extract meaningful information.

2. Disk I/O Performance Analysis

# Monitor disk I/O latency
sudo bpftrace -e '
kprobe:blk_account_io_start {
    @start[arg0] = nsecs;
}

kprobe:blk_account_io_done /@start[arg0]/ {
    $latency = nsecs - @start[arg0];
    @latency = hist($latency / 1000);
    delete(@start[arg0]);
}

END {
    printf("\nDisk I/O Latency Distribution (microseconds):\n");
    print(@latency);
}'

This script tracks block I/O operations and creates a histogram of latency values, helping identify storage performance issues.

3. CPU Usage by Process

# Sample CPU usage
sudo bpftrace -e '
profile:hz:99 {
    @[comm] = count();
}

END {
    printf("\nProcess CPU Samples (99Hz):\n");
    print(@);
}'

This uses eBPF profiling to sample which processes are consuming CPU time, similar to perf but with eBPF’s flexibility.

Using BCC Tools

BCC (BPF Compiler Collection) provides pre-built tools for common monitoring tasks. These are excellent starting points before writing custom programs.

Installation and Basic Usage

# List available BCC tools
ls /usr/share/bcc/tools/

# Some commonly used tools:
# - execsnoop: Trace new process execution
# - opensnoop: Trace open() syscalls
# - biosnoop: Trace block device I/O
# - tcpconnect: Trace TCP active connections
# - gethostlatency: Show latency of name resolution

Process Monitoring with execsnoop

# Trace all new process execution
sudo /usr/share/bcc/tools/execsnoop

# Sample output:
# PCOMM            PID    PPID   RET ARGS
# ls               12345  1000     0 /bin/ls -la
# curl             12346  1000     0 /usr/bin/curl https://example.com

execsnoop is invaluable for understanding what commands are being run on your system, especially useful for debugging startup scripts or identifying unexpected process execution.

File Access with opensnoop

# Monitor all file open operations
sudo /usr/share/bcc/tools/opensnoop

# Filter by specific process
sudo /usr/share/bcc/tools/opensnoop -n nginx

Network Monitoring with tcpconnect

# Trace all TCP connections
sudo /usr/share/bcc/tools/tcpconnect

# Include DNS resolution
sudo /usr/share/bcc/tools/tcpconnect -d

Advanced eBPF with Tetragon

Tetragon is an eBPF-based security observability tool from the Cilium project, ideal for production monitoring.

Installing Tetragon

# Add the Cilium repository
cat <<'REPO' | sudo tee /etc/yum.repos.d/cilium-tetragon.repo
[cilium-tetragon]
name=Cilium Tetragon
baseurl=https://download.cilium.io/tetragon/rpm/stable/
enabled=1
gpgcheck=1
gpgkey=https://download.cilium.io/tetragon/rpm/stable/GPG-KEY-cilium
REPO

# Install Tetragon
sudo dnf install -y tetragon tetragon-cli

# Start the service
sudo systemctl enable --now tetragon

Real-time Event Monitoring

# View events in compact format
sudo tetra getevents -o compact

# Detailed JSON output for processing
sudo tetra getevents | jq '.process_exec.process | {binary, arguments, uid}'

Creating Tracing Policies

Create a policy to monitor sensitive file access:

# /etc/tetragon/tetragon.tp.d/file-monitoring.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: sensitive-file-access
spec:
  kprobes:
  - call: "fd_install"
    syscall: false
    args:
    - index: 0
      type: int
    - index: 1
      type: "file"
    selectors:
    - matchArgs:
      - index: 1
        operator: "Prefix"
        values:
        - "/etc/shadow"
        - "/etc/passwd"
        - "/root/.ssh/"
      matchActions:
      - action: Post

Apply the policy and test:

sudo systemctl restart tetragon
cat /etc/shadow  # This will trigger an event
sudo tetra getevents -o compact | grep shadow

eBPF Performance Considerations

While eBPF is designed for low overhead, poorly written programs can impact system performance.

Best Practices for Production

  1. Start with sampling: Use periodic sampling instead of tracing every event for high-frequency operations
  2. Filter early: Apply filters in kernel space to reduce data transfer to userspace
  3. Limit map sizes: Unbounded maps can consume excessive memory
  4. Test in staging: Always validate eBPF programs in non-production environments first
  5. Monitor overhead: Use tools like bpftool to check program resource usage
# Check eBPF program resource usage
sudo bpftool prog list
sudo bpftool prog show id <ID> --json | jq

Writing Custom eBPF Programs

Once comfortable with bpftrace and BCC tools, you can write custom eBPF programs using C and libbpf.

Simple Custom Program Structure

// example.bpf.c
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

SEC("tracepoint/syscalls/sys_enter_openat")
int trace_openat(struct trace_event_raw_sys_enter *ctx)
{
    char filename[256];
    bpf_probe_read_user_str(filename, sizeof(filename), (char *)ctx->args[1]);
    bpf_printk("Open: %s\n", filename);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

Compile and load with:

clang -O2 -target bpf -c example.bpf.c -o example.bpf.o
sudo bpftool prog load example.bpf.o /sys/fs/bpf/example
sudo bpftool prog attach /sys/fs/bpf/example tracepoint syscalls sys_enter_openat

Troubleshooting Common Issues

Permission Denied Errors

eBPF requires CAP_SYS_ADMIN capability or root privileges:

# Run with sudo or as root
sudo bpftrace ...

# Or add capabilities to specific users
sudo setcap cap_sys_admin,cap_net_admin+eip /usr/bin/bpftrace

Kernel Headers Missing

# Install kernel headers matching your running kernel
sudo apt install linux-headers-$(uname -r)

Verifier Errors

eBPF programs must pass kernel verification. Common issues include:

  • Unbounded loops (not allowed in eBPF)
  • Invalid memory access
  • Excessive program complexity
# Enable verbose verifier output for debugging
sudo bpftrace -d -e 'your_program' 2>&1 | less

Real-World Use Cases

Security Monitoring

Use eBPF to detect anomalous behavior like privilege escalation, unauthorized file access, or unexpected network connections. To establish the baseline protection layers on your host, review our complete Linux server security hardening guide.

Performance Debugging

Identify bottlenecks in applications by tracing function calls, measuring latency, and analyzing resource usage patterns.

Compliance Auditing

Implement continuous compliance monitoring by tracking access to sensitive files and systems.

Cloud-Native Observability

Tools like Cilium, Pixie, and Falco use eBPF to provide Kubernetes-native observability and security.

Next Steps and Resources

This eBPF Linux Monitoring Tutorial covered the fundamentals, but there's much more to explore:

  1. Experiment with BCC tools in your environment
  2. Write custom bpftrace scripts for your specific needs
  3. Study production eBPF tools like Cilium and Falco
  4. Join the eBPF community on Slack and contribute to open-source projects
  5. Consider eBPF certifications from the Linux Foundation

eBPF is transforming Linux system observability. By mastering these tools, you gain unprecedented visibility into system behavior, enabling better security, performance, and debugging capabilities.

Related guides: Linux Performance Monitoring, Network Traffic Analysis Tools, Container Security Monitoring

Related Posts