How to Deploy Cilium eBPF on K3s for High-Performance Mesh Networking and Observability

K3s is the de facto standard lightweight Kubernetes distribution for edge deployments, IoT gateways, and modern homelabs. By default, K3s provisions Flannel as its Container Network Interface (CNI) alongside kube-proxy operating in legacy iptables mode. While this default configuration suffices for small-scale development workloads, it rapidly exposes severe architectural bottlenecks as cluster complexity grows. Every Kubernetes Service creates a linear chain of sequential iptables evaluation rules. In high-density clusters, packet traversal through thousands of sequential netfilter rules introduces latency spikes, exhausts Linux kernel connection tracking (conntrack) tables, and consumes significant CPU cycles.

Cilium revolutionizes Kubernetes networking by replacing traditional packet filtering with Extended Berkeley Packet Filter (eBPF). Running sandboxed bytecode directly inside the Linux kernel, Cilium bypasses the entire iptables datapath, eliminates kube-proxy completely, accelerates inter-pod communication with BPF host routing, and provides cryptographic transparent encryption. Furthermore, Cilium integrates Hubble, providing deep Layer 7 packet observability and real-time service dependency mapping without modifying application code. In this comprehensive technical guide, you will bootstrap a K3s cluster stripped of Flannel and kube-proxy, deploy Cilium via Helm with complete kube-proxy replacement, enable Hubble UI telemetry, and enforce zero-trust L7 security policies.

eBPF Networking Architecture vs. Legacy Iptables

To understand why Cilium is an indispensable upgrade for Kubernetes operators, consider the mechanical differences between traditional Linux networking and the eBPF datapath:

• O(1) Hash Map Lookups vs. O(N) Iptables Traversal: Traditional kube-proxy inspects every network packet sequentially across thousands of iptables chains. Cilium compiles Service VIP endpoints into BPF hash maps, performing instantaneous constant-time O(1) lookups regardless of whether your cluster has 10 or 10,000 Services.
• Socket-Layer Redirection (sockops): When two pods reside on the same worker node, Cilium bypasses the TCP/IP stack entirely. It intercepts payload buffers at the socket layer and redirects packets directly into the target pod’s socket buffer, cutting latency in half.
• Layer 7 Protocol Awareness: Unlike standard Kubernetes NetworkPolicies that operate strictly on Layer 3 (IP) and Layer 4 (Port), Cilium parses HTTP headers, gRPC methods, and DNS queries directly in the kernel, enabling granular zero-trust security.

+-----------------------------------------------------------------------+
|                          K3S KUBERNETES NODE                          |
|         Pod A (Frontend)                    Pod B (Backend API)       |
|      [veth / socket layer]                [veth / socket layer]       |
+-------------------+-----------------------------------+---------------+
                    |                                   |
                    v                                   v
+-----------------------------------------------------------------------+
|                     LINUX KERNEL eBPF DATAPLANE                       |
|   +---------------------------------------------------------------+   |
|   | Socket Redirection Layer (sockops: Fast Socket-to-Socket)     |   |
|   +---------------------------------------------------------------+   |
|   | eBPF Host Routing: O(1) BPF Map Lookups (No iptables / kube-proxy)| |
|   +---------------------------------------------------------------+   |
|   | XDP (eXpress Data Path) & TC (Traffic Control) Packet Filters |   |
|   +---------------------------------------------------------------+   |
|   | Hubble Ring Buffer Flow Exporter (L3/L4/L7 Packet Events)     |   |
|   +---------------------------------------------------------------+   |
+-----------------------------------+-----------------------------------+
                                    |
                    gRPC Telemetry Stream (Port 4245)
                                    v
+-----------------------------------------------------------------------+
|                         HUBBLE OBSERVABILITY                          |
|   - Hubble Relay: Distributed Telemetry Aggregation                   |
|   - Hubble UI: Graphical Service Dependency Maps & HTTP Metrics       |
|   - Hubble CLI: Realtime Flow Monitoring & DNS Drops                  |
+-----------------------------------------------------------------------+

Host Prerequisites & Kernel Verification

Because Cilium compiles and loads bytecode directly into the Linux kernel, your host operating system must satisfy these kernel requirements:

• A Linux distribution with a modern kernel (version 5.15 LTS, 6.1 LTS, or 6.8 LTS, such as Ubuntu 24.04 LTS, Debian 12, or Rocky Linux 9).
• At least 2 CPU cores and 4 GB of RAM per node.
• The BPF filesystem (/sys/fs/bpf) mounted and persistent.
• Helm v3 and kubectl installed on your administrative workstation.

Verify that your Linux kernel has BPF support enabled and mount the BPF filesystem:

# Verify kernel version
uname -r

# Verify that the BPF filesystem is mounted
mount | grep /sys/fs/bpf

# If not mounted, mount it manually and persist in /etc/fstab:
sudo mount bpffs -t bpf /sys/fs/bpf
echo "bpffs /sys/fs/bpf bpf defaults 0 0" | sudo tee -a /etc/fstab

Bootstrapping K3s without Flannel and Kube-Proxy

To let Cilium take over cluster networking completely, K3s must be instructed during initialization to omit its default networking components. We achieve this by writing a declarative K3s configuration file at /etc/rancher/k3s/config.yaml:

sudo mkdir -p /etc/rancher/k3s

cat << 'EOF' | sudo tee /etc/rancher/k3s/config.yaml
# Disable default networking to allow full Cilium eBPF takeover
flannel-backend: "none"
disable-network-policy: true
disable-kube-proxy: true

# Cluster configuration
write-kubeconfig-mode: "0644"
cluster-cidr: "10.42.0.0/16"
service-cidr: "10.43.0.0/16"
EOF

Install K3s using the official installation script:

curl -sfL https://get.k3s.io | sh -

# Verify installation status
sudo kubectl get nodes

Expected Observation: The node will initially report NotReady status. This is normal and expected because no CNI is active yet to allocate pod IP addresses.

Deploying Cilium via Helm with Kube-Proxy Replacement

We deploy Cilium using its official Helm chart. Because we disabled kube-proxy, we configure Cilium with kubeProxyReplacement=true and explicitly specify the K3s API server IP address and port (6443):

# Add Cilium Helm repository
helm repo add cilium https://helm.cilium.io/
helm repo update

# Identify your server's primary LAN IP (e.g. 192.168.1.100)
NODE_IP=$(ip -4 addr show eth0 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')

# Install Cilium with eBPF host routing, kube-proxy replacement, and Hubble
helm install cilium cilium/cilium --version 1.16.2 \
  --namespace kube-system \
  --set k8sServiceHost=${NODE_IP} \
  --set k8sServicePort=6443 \
  --set kubeProxyReplacement=true \
  --set bpf.masquerade=true \
  --set bpf.hostRouting=true \
  --set autoDirectNodeRoutes=true \
  --set routingMode=native \
  --set ipv4NativeRoutingCIDR="10.42.0.0/16" \
  --set ipam.mode=kubernetes \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set operator.replicas=1

Monitor the rollout of Cilium agent daemonsets and the operator pod:

kubectl -n kube-system rollout status daemonset/cilium
kubectl -n kube-system rollout status deployment/cilium-operator
kubectl get nodes

Once Cilium pods initialize, CoreDNS and local pods obtain IP allocations, and your K3s node transitions immediately to Ready.

Validating Cluster eBPF Datapath Status

Install the official Cilium CLI binary to run diagnostics against the running eBPF datapath:

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz

# Inspect Cilium operational status
cilium status

Verify that the output displays KubeProxyReplacement: Strict [native], confirming that iptables has been entirely superseded by eBPF in-kernel routing.

Observability with Hubble: CLI & Service Map UI

Hubble leverages eBPF to observe all network packets at the kernel level without adding sidecar proxies. Access the Hubble CLI to inspect real-time cluster traffic:

# Install Hubble CLI
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-amd64.tar.gz
sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin
rm hubble-linux-amd64.tar.gz

# Port-forward Hubble Relay to inspect live packet streams
cilium hubble port-forward &
hubble observe --follow

To view the graphical Service Dependency Map in your browser, expose the Hubble UI dashboard via port-forwarding or an Ingress:

cilium hubble ui

Opening http://localhost:12000 displays an interactive real-time visual graph of all communicating pods, throughput rates, TCP drop reasons, and HTTP request statuses across namespaces.

Enforcing Zero-Trust Layer 7 NetworkPolicies

Unlike standard Kubernetes NetworkPolicies that are limited to IP and port matching, Cilium’s eBPF filters understand Layer 7 protocols. Below is a production CiliumNetworkPolicy that restricts traffic to a backend service, allowing only HTTP GET requests on the /api/v1/metrics endpoint while dropping all other requests:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "secure-backend-l7"
  namespace: "default"
spec:
  endpointSelector:
    matchLabels:
      app: "backend-service"
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: "frontend-service"
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/api/v1/metrics"

Any attempt by the frontend to issue a POST request or access unauthorized paths (e.g. /admin) is intercepted and rejected with an HTTP 403 Forbidden status directly inside the Linux kernel, without the packet ever reaching the backend application container.

Automated End-to-End Connectivity Testing

Before deploying production workloads, execute Cilium’s automated connectivity validation suite. This suite spawns ephemeral pods across namespaces to test pod-to-pod latency, multi-node routing, DNS resolution, and egress filtering:

cilium connectivity test

All test suites (including pod-to-pod, pod-to-service, and dns-only) should report clean green passes, confirming that your eBPF mesh datapath is fully functional.

Production Hardening & Operational Best Practices

Maintain peak performance and security on Cilium-enabled K3s clusters by adhering to these guidelines:

• Enable Transparent WireGuard Encryption: To encrypt all inter-node pod communication transparently without CPU overhead, add --set encryption.enabled=true --set encryption.type=wireguard during Helm installation. Cilium manages keys automatically in the kernel.
• Configure Bandwidth Management with BPF: Replace conventional Linux traffic shaping (tc) with Cilium’s eBPF Earliest Departure Time (EDT) rate limiter: --set bandwidthManager.enabled=true.
• Prune Stale BPF Maps on Upgrade: When upgrading Cilium across minor versions, verify that host BPF maps are synchronized by running cilium-dbg bpf tunnel list or cilium-dbg bpf endpoint list.
• External Ingress Isolation: Route external internet traffic to Cilium Ingress endpoints using Cloudflare Tunnels or an authenticated Tailscale mesh exit node.

Troubleshooting Common Deployment Issues

Below are three frequently encountered issues when deploying Cilium on K3s, along with diagnostic commands and fixes:

1. Node Remains “NotReady” with CoreDNS in CrashLoopBackOff

Symptom: After bootstrapping K3s and Helm, coredns pods repeatedly crash, and node status remains stuck in NotReady.

Root Cause: The BPF filesystem was not mounted prior to K3s startup, or the Cilium agent pod failed to start due to an incorrect k8sServiceHost IP address.

Resolution: Verify the BPF mount using mount | grep /sys/fs/bpf. Confirm that k8sServiceHost in your Helm parameters matches your physical node IP and not 127.0.0.1.

2. “kube-proxy replacement: incompatible mode” Error

Symptom: The cilium agent pod logs report Error: kube-proxy replacement is enabled in strict mode but kube-proxy is still running.

Root Cause: K3s was started without the --disable-kube-proxy flag in /etc/rancher/k3s/config.yaml.

Resolution: Add disable-kube-proxy: true to /etc/rancher/k3s/config.yaml, restart K3s with sudo systemctl restart k3s, and delete existing Cilium pods to force re-initialization.

3. Hubble CLI Reports “Failed to connect to Hubble Relay: connection refused”

Symptom: Running hubble observe fails with gRPC connection errors.

Root Cause: Port-forwarding to the Hubble Relay pod has terminated or the relay service is not running.

Resolution: Verify that hubble-relay pods are healthy in kube-system, and re-establish port-forwarding with cilium hubble port-forward &.

Summary & Key Takeaways

By replacing Flannel and kube-proxy with Cilium eBPF on K3s, you unlock enterprise-grade networking performance, eliminate iptables scaling bottlenecks, and gain deep Layer 7 observability with Hubble. With O(1) BPF map lookups, socket-layer redirection, and declarative L7 security policies, your K3s cluster transitions from a lightweight sandbox into a hardened, high-throughput cloud-native platform.