Kubernetes the Hard Way, Part 4: Pod Networking by Hand, Without a CNI Plugin

In Part 1 you gave pods their network with a small config file for the bridge plugin and a few routes. It worked, and you could trust it, but you did not see what it did. This part opens it up.

You will build what that plugin builds with plain Linux commands, which means a network namespace for the pod, a cable into it, a switch on the host, a route to the other worker and a rule for the internet. Then you will hand the same job to the plugin yourself, exactly the way the container runtime does, and read what it answers. After that you will find out why a packet that is too big vanishes on some clouds, and how a Service address that belongs to no machine at all still reaches a pod.

We ran every command in this post, top to bottom, on fresh machines before publishing it, and the outputs on this page are the ones we got. Our machines were on Hetzner Cloud, whose private network is routed, so you will see its gateway in a few places.

What you need

The cluster from Part 1, with its jumpbox and both workers. Everything in this part happens on the workers, plus a few kubectl commands from the jumpbox. Part 1 already installed conntrack on the workers. The only new package is traceroute.

The pods you build by hand get addresses from 10.100.0.0/16, a range the cluster does not use. node-0 gets 10.100.0.0/24, node-1 gets 10.100.1.0/24.

Step 1. Look inside a real pod

Start one pod on each worker, so you know where they are.

for N in 0 1; do
  kubectl run net-${N} --image=busybox:1.37 --restart=Never \
    --overrides="{\"spec\":{\"nodeName\":\"node-${N}\"}}" -- sleep 7200
done
kubectl wait --for=condition=Ready pod/net-0 pod/net-1 --timeout=120s
kubectl get pods net-0 net-1 -o wide

NAME    READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
net-0   1/1     Running   0          3s    10.200.0.4   node-0   <none>           <none>
net-1   1/1     Running   0          3s    10.200.1.4   node-1   <none>           <none>

A pod’s network is a Linux network namespace, a private copy of the network stack with its own interfaces, addresses and routes. containerd creates one per pod and keeps it under /var/run/netns. Find the one for net-0 on node-0 and look inside.

POD=$(crictl pods --name net-0 -q)
NETNS=$(crictl inspectp -o go-template \
  --template '{{range .info.runtimeSpec.linux.namespaces}}{{if eq .type "network"}}{{.path}}{{end}}{{end}}' ${POD})
NS=$(basename ${NETNS})
echo ${NS}
ip -n ${NS} -br addr
ip -n ${NS} route

# eth0@ifN in the pod means the other end of its cable is interface N on the host
PEER_INDEX=$(ip -n ${NS} -o link show eth0 | grep -o '@if[0-9]*' | cut -c4-)
ip -o link show | grep "^${PEER_INDEX}:" | cut -d' ' -f1-9
ip -br link show master cni0
ls /var/lib/cni/networks/bridge

cni-4c6a91cf-d49d-13b2-7bff-68211852cbc0
lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0@if7         UP             10.200.0.4/24 fe80::3095:d9ff:fec4:a3aa/64
default via 10.200.0.1 dev eth0
10.200.0.0/24 dev eth0 proto kernel scope link src 10.200.0.4
7: vethcbcd0d19@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cni0
veth6a96ea1b@if2 UP             6e:f4:e5:8b:a2:ef <BROADCAST,MULTICAST,UP,LOWER_UP>
veth50e795cd@if2 UP             92:1a:5e:0f:2d:0d <BROADCAST,MULTICAST,UP,LOWER_UP>
vethcbcd0d19@if2 UP             d2:e5:77:26:5a:22 <BROADCAST,MULTICAST,UP,LOWER_UP>
10.200.0.2
10.200.0.3
10.200.0.4
last_reserved_ip.0
lock

That is the whole network of a pod.

• Inside the namespace there is a loopback interface and one eth0 with the pod’s address. Its routing table has two lines. Addresses in the worker’s own slice are reachable directly, and everything else goes to 10.200.0.1, which is cni0 on the host.
• The cable is a veth pair. eth0@if7 means its other end is interface 7 on the host, and interface 7 is vethcbcd0d19, plugged into cni0. Every pod on the worker has one such host end on the bridge, three in our case.
• The address book is a directory. The host-local plugin keeps one file per address it has handed out, which is how it never gives the same address to two pods.

Step 2. Build a pod by hand

Now build the same thing yourself, on both workers. A namespace called lab plays the pod. A bridge called lab0 plays cni0. A veth pair is the cable, a pair of virtual interfaces where whatever goes in one end comes out of the other. One end goes into the namespace and becomes its eth0, the other end is plugged into the bridge.

apt-get install -y -qq traceroute >/dev/null
N=$(hostname | cut -d- -f2)

ip netns add lab
ip link add lab0 type bridge
ip addr add 10.100.${N}.1/24 dev lab0
ip link set lab0 up

ip link add veth-lab type veth peer name eth0 netns lab
ip link set veth-lab master lab0 up

ip -n lab addr add 10.100.${N}.10/24 dev eth0
ip -n lab link set eth0 up
ip -n lab link set lo up
ip -n lab route add default via 10.100.${N}.1

ip -n lab -br addr
ping -c 2 -W 2 10.100.${N}.10 | tail -2

lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0@if9         UP             10.100.0.10/24 fe80::e0aa:29ff:fe0a:2ce0/64
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 0.142/0.207/0.272/0.065 ms

That is node-0. node-1 printed the same with 10.100.1.10.

The bridge has the address 10.100.N.1, and the namespace sends everything it cannot reach directly to that address. That makes the host the namespace’s router, which is exactly what "isGateway": true did in Part 1’s bridge config.

Step 3. Route between the workers

A packet from 10.100.0.10 to 10.100.1.10 leaves the namespace, reaches node-0, and node-0 has to know where to send it. Part 1 solved this for pods, and the same choice applies here.

If your workers share one network segment, each worker routes the other’s range straight to it. On a routed network such as Hetzner Cloud, every packet goes through the network’s gateway, so the network itself has to learn where each range lives. Add those routes with the hcloud tool, using your network’s name. Skip this block on a shared segment.

NETWORK=kthw
hcloud network add-route ${NETWORK} --destination 10.100.0.0/24 --gateway 10.240.0.20
hcloud network add-route ${NETWORK} --destination 10.100.1.0/24 --gateway 10.240.0.21

Then add the route on each worker. As in Part 1, set POD_GATEWAY to your network’s gateway on a routed network, or leave it empty on a shared segment.

N=$(hostname | cut -d- -f2)
PEER=$((1 - N))
PEER_IP=$(getent hosts node-${PEER} | awk '{print $1}')

# Empty when the workers share one network segment.
# On a routed network such as Hetzner Cloud, the private network's gateway.
POD_GATEWAY=""

if [ -n "${POD_GATEWAY}" ]; then
  ip route add 10.100.0.0/16 via ${POD_GATEWAY}
else
  ip route add 10.100.${PEER}.0/24 via ${PEER_IP}
fi
ip route get 10.100.${PEER}.10

10.100.1.10 via 10.240.0.1 dev enp7s0 src 10.240.0.20 uid 0
    cache

ip route get asks the kernel which route it would use for one address. On node-0 the answer is the network’s gateway, out of the private interface. node-1 gives the same answer for 10.100.0.10.

Now test across the workers from node-0, and see the path the packet takes.

ip netns exec lab ping -c 3 -W 2 10.100.1.10 | tail -2
ip netns exec lab traceroute -n -w 2 -q 1 10.100.1.10

3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.676/2.959/7.261/3.043 ms
traceroute to 10.100.1.10 (10.100.1.10), 30 hops max, 60 byte packets
 1  10.100.0.1  0.131 ms
 2  10.240.0.1  2.675 ms
 3  10.240.0.21  2.610 ms
 4  10.100.1.10  1.667 ms

Read the path one hop at a time. The packet goes from the namespace to its gateway on the bridge (10.100.0.1), from node-0 to the private network’s gateway (10.240.0.1), from there to node-1 (10.240.0.21), because of the route you gave the network, and from node-1 over its own bridge into the other namespace. On a shared segment the second hop is missing, because node-0 sends straight to node-1. No address was translated anywhere on the way.

Step 4. Let the pod reach the internet

Try to reach the internet from the namespace first.

ip netns exec lab ping -c 2 -W 2 1.1.1.1 | tail -2 || true

2 packets transmitted, 0 received, 100% packet loss, time 1026ms

The ping leaves, but no answer comes back. The packet carries the source address 10.100.0.10, which only means something inside your network, and the reply has nowhere to go. Traffic leaving the cluster needs its source swapped for the worker’s own address. That is masquerading, the same rule Part 1 added for pods. Note the exception. Traffic to other pods, yours or Kubernetes’, must keep its real source.

nft -f - <<'EOF'
table ip lab-nat {
  chain postrouting {
    type nat hook postrouting priority srcnat; policy accept;
    ip saddr 10.100.0.0/16 ip daddr != { 10.100.0.0/16, 10.200.0.0/16 } masquerade
  }
}
EOF

ip netns exec lab ping -c 2 -W 2 1.1.1.1 | tail -2
conntrack -L -p icmp -d 1.1.1.1 2>/dev/null | grep -v UNREPLIED | tail -1

2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.386/3.117/4.848/1.731 ms
icmp     1 29 src=10.100.0.10 dst=1.1.1.1 type=8 code=0 id=2519 src=1.1.1.1 dst=37.27.184.211 type=0 code=0 id=2519 mark=0 use=1

The conntrack line shows the translation in both directions. The ping went out from 10.100.0.10, and the reply came back to 37.27.184.211, node-0’s public address. The kernel remembers the pairing and hands the reply back to the namespace. The grep -v UNREPLIED skips the entries left over from the pings that failed a moment ago.

Step 5. Be the container runtime

Everything you just typed is what a CNI plugin does when containerd asks it to. The CNI contract is small. The runtime runs the plugin binary, passes the details in environment variables and the network config on standard input, and reads a JSON answer from standard output. You can be the runtime.

ip netns add lab2

cat > /root/lab-cni.conf <<'EOF'
{
  "cniVersion": "1.0.0",
  "name": "lab",
  "type": "bridge",
  "bridge": "lab0",
  "isGateway": true,
  "ipam": {
    "type": "host-local",
    "ranges": [[{ "subnet": "10.100.0.0/24", "rangeStart": "10.100.0.100" }]],
    "routes": [{ "dst": "0.0.0.0/0" }],
    "dataDir": "/run/lab-ipam"
  }
}
EOF

CNI_COMMAND=ADD CNI_CONTAINERID=lab2 CNI_NETNS=/var/run/netns/lab2 CNI_IFNAME=eth0 \
  CNI_PATH=/opt/cni/bin /opt/cni/bin/bridge < /root/lab-cni.conf

{
    "cniVersion": "1.0.0",
    "interfaces": [
        {
            "mac": "1e:96:dd:b6:6d:9f",
            "name": "lab0"
        },
        {
            "mac": "7e:56:4a:45:3f:6e",
            "name": "veth8a694dd9"
        },
        {
            "mac": "d6:b8:f5:f0:6c:77",
            "name": "eth0",
            "sandbox": "/var/run/netns/lab2"
        }
    ],
    "ips": [
        {
            "address": "10.100.0.100/24",
            "gateway": "10.100.0.1",
            "interface": 2
        }
    ],
    "routes": [
        {
            "dst": "0.0.0.0/0"
        }
    ]
}

That JSON is the plugin’s answer, and it is what containerd reads for every pod you start. The plugin found the existing lab0 bridge and reused it, created a new veth pair, put one end into lab2 as eth0, and asked its IPAM plugin, host-local, for an address. It started at rangeStart, so it could not collide with the .10 you assigned by hand. The interface: 2 under ips points at the third entry in interfaces, the eth0 inside the namespace.

ip -n lab2 -br addr
ip netns exec lab2 ping -c 2 -W 2 10.100.1.10 | tail -1

CNI_COMMAND=DEL CNI_CONTAINERID=lab2 CNI_NETNS=/var/run/netns/lab2 CNI_IFNAME=eth0 \
  CNI_PATH=/opt/cni/bin /opt/cni/bin/bridge < /root/lab-cni.conf
ip -n lab2 -br addr
ls /run/lab-ipam/lab
ip netns delete lab2

lo               DOWN
eth0@if10        UP             10.100.0.100/24 fe80::d4b8:f5ff:fef0:6c77/64
rtt min/avg/max/mdev = 0.907/0.940/0.973/0.033 ms
lo               DOWN
last_reserved_ip.0
lock

The plugin-made pod reaches the namespace on node-1 at once, because it sits on the same bridge and uses the same route. Then DEL, which containerd sends when a pod stops. The plugin removed eth0 and its veth pair, and gave the address back, so the IPAM directory no longer holds a file for 10.100.0.100. Notice the loopback is down. Bringing it up is the runtime’s job, not the plugin’s, so in a real cluster containerd does it.

Step 6. The packet that is too big

Every network link has a maximum transmission unit, the MTU, the biggest packet it carries in one piece. Ethernet’s is usually 1500 bytes. Look at the private network interface on a worker and at the pod bridge.

ip -br link show | awk '{print $1}' | grep -v -E '^(lo|veth)' | while read IF; do
  echo "${IF%@*} $(cat /sys/class/net/${IF%@*}/mtu)"
done

eth0 1500
enp7s0 1450
cni0 1500
lab0 1500

Here is a mismatch. Everything on the worker says 1500, except the interface to the private network, enp7s0, which carries at most 1450 bytes. That is what Hetzner Cloud sets for its private networks. Other clouds and overlay networks have their own limits, many of them below 1500 too. Now send packets of different sizes across and forbid the network to split them.

# 1472 bytes of data + 28 bytes of headers = 1500, with "do not fragment" set
ip netns exec lab ping -c 1 -W 2 -M do -s 1472 10.100.1.10 || true
# 1422 + 28 = 1450, the most the private link carries
ip netns exec lab ping -c 1 -W 2 -M do -s 1422 10.100.1.10 | tail -2
# One byte more fails at once, because the namespace now remembers the limit
ip netns exec lab ping -c 1 -W 2 -M do -s 1423 10.100.1.10 || true

PING 10.100.1.10 (10.100.1.10) 1472(1500) bytes of data.
From 10.100.0.1 icmp_seq=1 Frag needed and DF set (mtu = 1450)

--- 10.100.1.10 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.602/0.602/0.602/0.000 ms
PING 10.100.1.10 (10.100.1.10) 1423(1451) bytes of data.
ping: sendmsg: Message too long

--- 10.100.1.10 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

The 1500-byte packet left the namespace, reached node-0, and node-0 could not send it on. It answered with an ICMP message, Frag needed and DF set (mtu = 1450). The namespace learned the limit from that answer, which is called path MTU discovery. The 1450-byte packet went through, and the 1451-byte one never even left, because the namespace already knew it would not fit.

That answer is the weak point. Some networks and firewalls drop ICMP. Then the big packet disappears without a word, small requests work, and large responses hang. It is one of the classic ways a cluster that passes every quick test still breaks in production.

Step 7. Fix the MTU for real pods

The pods from Part 1 have the same mismatch. Their side of the cable says 1500.

kubectl exec net-0 -- cat /sys/class/net/eth0/mtu

1500

The bridge plugin has an mtu setting. Add it on both workers. containerd reads the config for every new pod, so nothing has to restart.

# The MTU of the interface that carries traffic to the other machines
DEV=$(ip -o route get $(getent hosts server | awk '{print $1}') | grep -o 'dev [^ ]*' | cut -d' ' -f2)
MTU=$(cat /sys/class/net/${DEV}/mtu)
sed -i "s/\"isGateway\": true,/\"isGateway\": true,\n      \"mtu\": ${MTU},/" /etc/cni/net.d/10-bridge.conflist
grep -n '"mtu"' /etc/cni/net.d/10-bridge.conflist

Existing pods keep their old cable until they are replaced. Replace one and look again.

kubectl delete pod net-0 --wait=true
kubectl run net-0 --image=busybox:1.37 --restart=Never \
  --overrides='{"spec":{"nodeName":"node-0"}}' -- sleep 7200
kubectl wait --for=condition=Ready pod/net-0 --timeout=120s
kubectl exec net-0 -- cat /sys/class/net/eth0/mtu
kubectl exec net-0 -- ping -c 2 -W 2 $(kubectl get pod net-1 -o jsonpath='{.status.podIP}') | tail -2
kubectl rollout restart deployment/nginx
kubectl rollout status deployment/nginx --timeout=180s

pod "net-0" deleted from default namespace
pod/net-0 created
pod/net-0 condition met
1450
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 1.027/3.778/6.530 ms
deployment.apps/nginx restarted
deployment "nginx" successfully rolled out

The new net-0 has an MTU of 1450 and still reaches net-1 on the other worker. Its TCP connections now start with a segment size that fits the private link, so they never depend on that ICMP answer. Restarting the nginx deployment gives its pods the new setting too.

Step 8. How a Service reaches a pod

A Service address belongs to no machine. Nothing answers ARP for it, and no interface carries it. Look at the nginx Service from Part 1 and the pods behind it.

kubectl get service nginx
kubectl get endpointslices -l kubernetes.io/service-name=nginx

NAME    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
nginx   ClusterIP   10.32.0.18   <none>        80/TCP    85s
NAME          ADDRESSTYPE   PORTS   ENDPOINTS               AGE
nginx-g6htr   IPv4          80      10.200.0.6,10.200.1.5   85s

kube-proxy on every worker turns that address into rules. In nftables mode it keeps a map from each Service address and port to a chain, and each chain picks a pod.

nft list map ip kube-proxy service-ips | grep 'default/nginx/tcp/'
CHAIN=$(nft list map ip kube-proxy service-ips | grep -o 'service-[A-Z0-9]*-default/nginx/tcp/')
nft list chain ip kube-proxy "${CHAIN}"

           10.32.0.18 . tcp . 80 : goto service-ECO34HEO-default/nginx/tcp/,
table ip kube-proxy {
  chain service-ECO34HEO-default/nginx/tcp/ {
    meta l4proto tcp dnat ip to numgen random mod 2 map { 0 : 10.200.0.6 . 80, 1 : 10.200.1.5 . 80 }
  }
}

Two rules are the whole Service. The map sends any packet for 10.32.0.18, TCP, port 80, to the Service’s own chain. That chain picks a random number, 0 or 1, and rewrites the destination to the matching pod. When a pod is added or removed, kube-proxy rewrites that one line.

Call the Service from net-0, then ask the kernel on node-0 what it did with the connection. conntrack lists the connections the kernel tracks, with the address each packet was sent to and the address the reply really came from.

kubectl exec net-0 -- wget -qO- -T 5 http://nginx | grep -m1 -o 'Welcome to nginx!'

CLUSTER_IP=$(nft list map ip kube-proxy service-ips | grep 'default/nginx/tcp/' | grep -o '10\.32\.[0-9.]*')
conntrack -L -p tcp -d ${CLUSTER_IP} --dport 80 2>/dev/null | tail -1

tcp      6 119 TIME_WAIT src=10.200.0.5 dst=10.32.0.18 sport=41900 dport=80 src=10.200.0.6 dst=10.200.0.5 sport=80 dport=41900 [ASSURED] mark=0 use=1

The first half is what net-0 (10.200.0.5 now) sent, a connection to the Service address. The second half is where the reply really came from, the nginx pod at 10.200.0.6. The pod never saw the Service address, and net-0 never saw the pod’s. The kernel translated both ways, and that is all a ClusterIP is.

Step 9. Clean up

The lab served its purpose. Remove it from both workers, and remove the network routes if you added them. The MTU change from step 7 stays, because it is a real fix.

nft delete table ip lab-nat
ip route del 10.100.0.0/16 2>/dev/null || ip route del 10.100.$((1 - $(hostname | cut -d- -f2))).0/24
ip netns delete lab
ip link delete lab0
rm -rf /run/lab-ipam /root/lab-cni.conf
ip -br link | grep -c lab || true

NETWORK=kthw
hcloud network remove-route ${NETWORK} --destination 10.100.0.0/24 --gateway 10.240.0.20
hcloud network remove-route ${NETWORK} --destination 10.100.1.0/24 --gateway 10.240.0.21

kubectl delete pod net-0 net-1

When something does not work

• Frag needed and DF set (mtu = 1450) or Message too long. A packet is bigger than a link on its path. Set the bridge plugin’s mtu to the smallest link between your machines, as in step 7.
• Pings to the internet get no answer. The masquerade rule is missing or does not match the source range. conntrack -L shows the reply addressed to the private address instead of the worker’s public one.
• conntrack -L shows [UNREPLIED] entries. Those are attempts that got no answer, often from before you fixed something. They expire on their own after half a minute for ICMP.
• nft list prints table ip filter is managed by iptables-nft, do not touch!. That table belongs to iptables, which some programs still use through its nftables backend. Read it, but change it with iptables, not nft.

What real network plugins add

The bridge plugin and a few routes are a complete pod network for a small cluster, and production plugins do the same jobs. What they add is mostly scale and safety.

• No hand-made routes. An overlay such as VXLAN wraps each pod packet inside a packet between nodes, so the network in between never has to learn pod addresses. The wrapping costs bytes, 50 for VXLAN over IPv4, which is why overlay pods use a smaller MTU than their nodes.
• Network policies. Rules that say which pods may talk to which. The bridge plugin cannot enforce them. Calico and Cilium can.
• Faster Services. Cilium can replace kube-proxy with eBPF programs in the kernel that do the same address rewriting you read in step 8.
• Encryption between nodes, with WireGuard or IPsec.

The easy way, once you know the hard way

Clusters Ankra builds with its default kubeadm distribution run Cilium, so the pod network, the routes between nodes and the policies come with the cluster. You choose the private network’s range when you create it.

ankra cluster hetzner create --name my-cluster --location hel1 \
  --credential-id <your-hetzner-credential> \
  --network-ip-range 10.0.0.0/16 \
  --control-plane-count 3 --worker-count 2

What you learned here is what you need when something in that network misbehaves anyway. A pod that cannot reach another is a namespace, a cable, a route and a rule, and now you know where to look for each one.

This is Part 4 of Kubernetes the Hard Way, a series of guides we test end to end before publishing. Part 5 covers certificates, expiry and renewal.

Get started: Create a free account on Ankra.

Join our community: Slack

Follow us on: LinkedIn | GitHub

Contact us: [email protected]