Kubernetes the Hard Way, Part 2: A Highly Available Control Plane

October 25, 2026

13 min read

On this page

In Part 1 you built a working Kubernetes cluster by hand. It had one weakness that every real cluster has to fix, which is a single control plane machine. Lose that one server and the cluster can no longer schedule a pod, scale a deployment or answer kubectl.

This part fixes that. You will build a control plane from three servers, run etcd as a three-member cluster with TLS on every connection, and put a load balancer in front of the API servers. Then you will break it on purpose. You will switch off one server and watch nothing happen, switch off a second and watch the control plane stop while your applications keep serving, and bring both back.

Like Part 1, we ran every command in this post, top to bottom, on fresh machines before publishing it, and the failure drills on this page show what really happened. If you have not done Part 1, start there. This part moves faster through the steps it already explained.

What you will build

Seven Linux machines, each with one job.

Machine	Role	What runs there
jumpbox	Your workbench	openssl, kubectl, all downloads and generated files
api	Load balancer	HAProxy, the single address everything uses to reach the API
server-0, server-1, server-2	Control plane	etcd, kube-apiserver, kube-controller-manager, kube-scheduler
node-0, node-1	Workers	containerd, kubelet, kube-proxy, your pods

These are the versions we tested with.

Component	Version
Kubernetes	v1.37.1
etcd	v3.7.3
containerd	v2.4.2
runc	v1.5.2
CNI plugins	v1.9.1
crictl	v1.37.0
CoreDNS	v1.14.6
HAProxy	Debian 13 package

Why three, and not two

High availability in Kubernetes is really high availability of etcd. Every other control plane component is stateless and can simply run more copies. etcd is different, because it stores the whole cluster and must never disagree with itself.

etcd uses the Raft consensus protocol. A change counts only when a majority of members have written it, and a majority of three is two. So a three-member cluster keeps working with one member down. A two-member cluster cannot, because its majority is still two, so losing either member stops it. That makes two members strictly worse than one. Five members survive two failures, which is why large clusters run five.

The other components follow simpler rules.

kube-apiserver runs on all three servers at once. Each one is a full copy, and the load balancer spreads requests across them.
kube-controller-manager and kube-scheduler also run on all three, but only one of each acts at a time. They use leader election, a lease object in the API, and a standby takes over when the leader’s lease stops being renewed.

Before you start

You need seven machines running Debian 13 on one private network, all with the same CPU architecture. The jumpbox and api can be small (1 vCPU, 1 GB of memory). Give the three servers and both workers at least 2 vCPU and 2 GB of memory each. As in Part 1, root on the jumpbox needs key-based SSH access as root to the other six. At Hetzner, for example, seven CX23 servers with IPv4 addresses cost about €0.07 an hour together at October 2026 list prices, before VAT.

The pod and service ranges are the same as in Part 1, 10.200.0.0/16 and 10.32.0.0/24, so your machine network must not overlap them. On Hetzner Cloud, put all seven servers on one private network and see the routed-network notes in step 9.

Step 1. Prepare the jumpbox

apt-get update
apt-get install -y curl openssl openssh-client
mkdir -p ~/kthw-ha

Describe the machines. Workers carry their pod slice as before. Replace the example addresses with your own.

cd ~/kthw-ha
cat > machines.txt <<'EOF'
10.240.0.9 api
10.240.0.10 server-0
10.240.0.11 server-1
10.240.0.12 server-2
10.240.0.20 node-0 10.200.0.0/24
10.240.0.21 node-1 10.200.1.0/24
EOF

Name every machine and share the host names, exactly as in Part 1.

cd ~/kthw-ha
awk '{print $1, $2}' machines.txt > hosts
cat hosts >> /etc/hosts

while read IP HOST SUBNET; do
  ssh -n -o StrictHostKeyChecking=accept-new root@${IP} "hostnamectl set-hostname ${HOST}"
  scp hosts root@${IP}:/tmp/hosts
  ssh -n root@${IP} "cat /tmp/hosts >> /etc/hosts && rm /tmp/hosts"
done < machines.txt

for HOST in $(awk '{print $2}' machines.txt); do
  ssh -n -o StrictHostKeyChecking=accept-new root@${HOST} hostname
done

Step 2. Download the binaries

The same downloads as Part 1, with a newer etcd patch release.

cd ~/kthw-ha
K8S_VERSION=v1.37.1
ETCD_VERSION=v3.7.3
CONTAINERD_VERSION=2.4.2
RUNC_VERSION=v1.5.2
CNI_VERSION=v1.9.1
CRICTL_VERSION=v1.37.0
ARCH=$(dpkg --print-architecture)
GH=https://github.com

mkdir -p downloads && cd downloads

for BIN in kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy kubectl; do
  curl -fsSLO "https://dl.k8s.io/${K8S_VERSION}/bin/linux/${ARCH}/${BIN}"
  echo "$(curl -fsSL "https://dl.k8s.io/${K8S_VERSION}/bin/linux/${ARCH}/${BIN}.sha256")  ${BIN}" | sha256sum --check
done

ETCD_TGZ=etcd-${ETCD_VERSION}-linux-${ARCH}.tar.gz
curl -fsSLO "${GH}/etcd-io/etcd/releases/download/${ETCD_VERSION}/${ETCD_TGZ}"
curl -fsSL "${GH}/etcd-io/etcd/releases/download/${ETCD_VERSION}/SHA256SUMS" | grep " ${ETCD_TGZ}$" | sha256sum --check

CONTAINERD_TGZ=containerd-${CONTAINERD_VERSION}-linux-${ARCH}.tar.gz
curl -fsSLO "${GH}/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/${CONTAINERD_TGZ}"
curl -fsSL "${GH}/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/${CONTAINERD_TGZ}.sha256sum" | sha256sum --check
curl -fsSL -o containerd.service "https://raw.githubusercontent.com/containerd/containerd/v${CONTAINERD_VERSION}/containerd.service"

curl -fsSLO "${GH}/opencontainers/runc/releases/download/${RUNC_VERSION}/runc.${ARCH}"
curl -fsSL "${GH}/opencontainers/runc/releases/download/${RUNC_VERSION}/runc.sha256sum" | grep " runc.${ARCH}$" | sha256sum --check
mv runc.${ARCH} runc

CNI_TGZ=cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz
curl -fsSLO "${GH}/containernetworking/plugins/releases/download/${CNI_VERSION}/${CNI_TGZ}"
curl -fsSL "${GH}/containernetworking/plugins/releases/download/${CNI_VERSION}/${CNI_TGZ}.sha256" | sha256sum --check

CRICTL_TGZ=crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz
curl -fsSLO "${GH}/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/${CRICTL_TGZ}"
echo "$(curl -fsSL "${GH}/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/${CRICTL_TGZ}.sha256")  ${CRICTL_TGZ}" | sha256sum --check

tar -xzf ${ETCD_TGZ} --strip-components=1 etcd-${ETCD_VERSION}-linux-${ARCH}/etcd etcd-${ETCD_VERSION}-linux-${ARCH}/etcdctl
chmod +x kube-* kubelet kubectl runc
install -m 0755 kubectl /usr/local/bin/kubectl

Step 3. Create the certificates

Part 1 kept etcd on the loopback address and skipped TLS for it. That shortcut ends here. The three etcd members talk to each other over the network, and the API servers reach etcd on other machines, so every one of those connections gets mutual TLS.

etcd also gets its own certificate authority. Anything signed by the Kubernetes CA, a kubelet certificate for example, must not be able to talk to the database. With a separate CA, etcd only trusts certificates you made for it. kubeadm does the same.

cd ~/kthw-ha
mkdir -p pki && cd pki

# make_ca NAME COMMON_NAME
make_ca() {
  openssl genpkey -quiet -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out "$1.key"
  openssl req -x509 -new -key "$1.key" -sha256 -days 3650 -subj "/CN=$2" \
    -addext "basicConstraints=critical,CA:TRUE" \
    -addext "keyUsage=critical,keyCertSign,cRLSign" \
    -out "$1.crt"
}

# make_cert NAME SUBJECT USAGE [SUBJECT_ALT_NAMES] [CA]
make_cert() {
  local name=$1 subject=$2 usage=$3 sans=$4 ca=${5:-ca}
  openssl genpkey -quiet -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out "${name}.key"
  openssl req -new -key "${name}.key" -subj "${subject}" -out "${name}.csr"
  {
    echo "basicConstraints=critical,CA:FALSE"
    echo "keyUsage=critical,digitalSignature,keyEncipherment"
    echo "extendedKeyUsage=${usage}"
    if [ -n "${sans}" ]; then echo "subjectAltName=${sans}"; fi
  } > "${name}.ext"
  openssl x509 -req -in "${name}.csr" -CA "${ca}.crt" -CAkey "${ca}.key" \
    -days 365 -sha256 -extfile "${name}.ext" -out "${name}.crt"
  rm "${name}.csr" "${name}.ext"
}

make_ca ca kubernetes-ca
make_ca etcd-ca etcd-ca

API_IP=$(awk '$2 == "api" {print $1}' ../machines.txt)
SERVER_SANS=$(awk '$2 ~ /^server-/ {printf ",IP:%s,DNS:%s", $1, $2}' ../machines.txt)

# Clients of the API server, as in Part 1
make_cert admin "/CN=admin/O=system:masters" clientAuth
make_cert kube-controller-manager "/CN=system:kube-controller-manager" clientAuth
make_cert kube-scheduler "/CN=system:kube-scheduler" clientAuth
make_cert kube-proxy "/CN=system:kube-proxy" clientAuth
make_cert apiserver-kubelet-client "/CN=kube-apiserver-kubelet-client" clientAuth

# One API server certificate, valid for every address a client might use
make_cert kube-apiserver "/CN=kube-apiserver" serverAuth \
  "IP:10.32.0.1,IP:127.0.0.1,IP:${API_IP},DNS:api${SERVER_SANS},DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster.local"

# etcd: a serving and a peer certificate per member, and one client certificate for the API servers
while read IP HOST SUBNET; do
  case ${HOST} in server-*) ;; *) continue ;; esac
  make_cert etcd-${HOST} "/CN=${HOST}" serverAuth,clientAuth "IP:${IP},IP:127.0.0.1,DNS:${HOST},DNS:localhost" etcd-ca
  make_cert etcd-peer-${HOST} "/CN=${HOST}" serverAuth,clientAuth "IP:${IP},DNS:${HOST}" etcd-ca
done < ../machines.txt
make_cert apiserver-etcd-client "/CN=kube-apiserver-etcd-client" clientAuth "" etcd-ca

# Workers, as in Part 1
while read IP HOST SUBNET; do
  if [ -z "${SUBNET}" ]; then continue; fi
  make_cert "${HOST}" "/CN=system:node:${HOST}/O=system:nodes" serverAuth,clientAuth "DNS:${HOST},IP:${IP}"
done < ../machines.txt

# One service account key pair, shared by all three API servers
openssl genpkey -quiet -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out service-account.key
openssl pkey -in service-account.key -pubout -out service-account.pub

ls

Two details in that block matter more than they look.

The API server certificate lists the load balancer’s address and all three servers. A client may reach any API server directly or through api, and the certificate has to match whichever name it used.

There is exactly one service account key pair. An API server signs service account tokens with its private key, and the next request from that pod may land on a different API server. If each server had its own key, tokens would randomly fail depending on which server answered. The same goes for the encryption key in step 5.

Step 4. Create the kubeconfig files

The same helper as Part 1. The difference is the address. Workers, kube-proxy and you reach the API through api, the load balancer. The controller manager and scheduler talk to the API server on their own machine.

cd ~/kthw-ha
mkdir -p configs && cd configs

# make_kubeconfig NAME USER SERVER_URL
make_kubeconfig() {
  local name=$1 user=$2 server=$3
  kubectl config set-cluster kthw-ha --certificate-authority=../pki/ca.crt \
    --embed-certs=true --server="${server}" --kubeconfig="${name}.kubeconfig"
  kubectl config set-credentials "${user}" --client-certificate="../pki/${name}.crt" \
    --client-key="../pki/${name}.key" --embed-certs=true --kubeconfig="${name}.kubeconfig"
  kubectl config set-context default --cluster=kthw-ha --user="${user}" --kubeconfig="${name}.kubeconfig"
  kubectl config use-context default --kubeconfig="${name}.kubeconfig"
}

make_kubeconfig node-0 system:node:node-0 https://api:6443
make_kubeconfig node-1 system:node:node-1 https://api:6443
make_kubeconfig kube-proxy system:kube-proxy https://api:6443
make_kubeconfig kube-controller-manager system:kube-controller-manager https://127.0.0.1:6443
make_kubeconfig kube-scheduler system:kube-scheduler https://127.0.0.1:6443
make_kubeconfig admin admin https://127.0.0.1:6443

Step 5. One encryption key for all three servers

The same encryption config as Part 1, generated once and copied to every server. If each server encrypted Secrets with its own key, a Secret written through one API server would be unreadable through the other two.

cd ~/kthw-ha/configs
ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)

cat > encryption-config.yaml <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - secretbox:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

Step 6. Write the control plane services

etcd first. Each member needs to know its own name and address, plus the full list of members to form the cluster with. The loop at the end writes one file per server from a template.

mkdir -p ~/kthw-ha/control-plane && cd ~/kthw-ha/control-plane
INITIAL_CLUSTER=$(awk '$2 ~ /^server-/ {printf "%s%s=https://%s:2380", (n++ ? "," : ""), $2, $1}' ../machines.txt)
ETCD_SERVERS=$(awk '$2 ~ /^server-/ {printf "%shttps://%s:2379", (n++ ? "," : ""), $1}' ../machines.txt)
echo "${INITIAL_CLUSTER}"
echo "${ETCD_SERVERS}"

cat > etcd.service.template <<'EOF'
[Unit]
Description=etcd
Documentation=https://etcd.io/docs/

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name=NAME \
  --data-dir=/var/lib/etcd \
  --listen-client-urls=https://IP:2379,https://127.0.0.1:2379 \
  --advertise-client-urls=https://IP:2379 \
  --listen-peer-urls=https://IP:2380 \
  --initial-advertise-peer-urls=https://IP:2380 \
  --initial-cluster=INITIAL_CLUSTER \
  --initial-cluster-state=new \
  --initial-cluster-token=kthw-ha \
  --client-cert-auth \
  --trusted-ca-file=/etc/etcd/etcd-ca.crt \
  --cert-file=/etc/etcd/server.crt \
  --key-file=/etc/etcd/server.key \
  --peer-client-cert-auth \
  --peer-trusted-ca-file=/etc/etcd/etcd-ca.crt \
  --peer-cert-file=/etc/etcd/peer.crt \
  --peer-key-file=/etc/etcd/peer.key
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

while read IP HOST SUBNET; do
  case ${HOST} in server-*) ;; *) continue ;; esac
  sed -e "s|NAME|${HOST}|" -e "s|IP|${IP}|g" -e "s|INITIAL_CLUSTER|${INITIAL_CLUSTER}|" \
    etcd.service.template > etcd-${HOST}.service
done < ../machines.txt

server-0=https://10.240.0.10:2380,server-1=https://10.240.0.11:2380,server-2=https://10.240.0.12:2380
https://10.240.0.10:2379,https://10.240.0.11:2379,https://10.240.0.12:2379

--client-cert-auth and --peer-client-cert-auth make etcd refuse any connection that does not present a certificate signed by the etcd CA. Without them, TLS would encrypt the traffic but still let anyone on the network read and write your cluster.

The API server is the same as Part 1, with three changes. It advertises its own address, it reaches all three etcd members over TLS with the client certificate from step 3, and its service account issuer names the load balancer, the one address every client uses.

cd ~/kthw-ha/control-plane
ETCD_SERVERS=$(awk '$2 ~ /^server-/ {printf "%shttps://%s:2379", (n++ ? "," : ""), $1}' ../machines.txt)

cat > kube-apiserver.service.template <<'EOF'
[Unit]
Description=Kubernetes API Server
Documentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/
After=etcd.service
Wants=etcd.service

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
  --advertise-address=IP \
  --allow-privileged=true \
  --authorization-mode=Node,RBAC \
  --client-ca-file=/var/lib/kubernetes/ca.crt \
  --enable-admission-plugins=NodeRestriction \
  --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \
  --etcd-servers=ETCD_SERVERS \
  --etcd-cafile=/var/lib/kubernetes/etcd-ca.crt \
  --etcd-certfile=/var/lib/kubernetes/apiserver-etcd-client.crt \
  --etcd-keyfile=/var/lib/kubernetes/apiserver-etcd-client.key \
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.crt \
  --kubelet-client-certificate=/var/lib/kubernetes/apiserver-kubelet-client.crt \
  --kubelet-client-key=/var/lib/kubernetes/apiserver-kubelet-client.key \
  --service-account-issuer=https://api:6443 \
  --service-account-key-file=/var/lib/kubernetes/service-account.pub \
  --service-account-signing-key-file=/var/lib/kubernetes/service-account.key \
  --service-cluster-ip-range=10.32.0.0/24 \
  --tls-cert-file=/var/lib/kubernetes/kube-apiserver.crt \
  --tls-private-key-file=/var/lib/kubernetes/kube-apiserver.key \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

while read IP HOST SUBNET; do
  case ${HOST} in server-*) ;; *) continue ;; esac
  sed -e "s|IP|${IP}|" -e "s|ETCD_SERVERS|${ETCD_SERVERS}|" \
    kube-apiserver.service.template > kube-apiserver-${HOST}.service
done < ../machines.txt

The controller manager and scheduler are identical on all three servers and identical to Part 1. Leader election, which Part 1 turned on without needing it, now does real work.

cd ~/kthw-ha/control-plane

cat > kube-controller-manager.service <<'EOF'
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/
After=kube-apiserver.service

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --bind-address=127.0.0.1 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.crt \
  --cluster-signing-key-file=/var/lib/kubernetes/ca.key \
  --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \
  --leader-elect=true \
  --root-ca-file=/var/lib/kubernetes/ca.crt \
  --service-account-private-key-file=/var/lib/kubernetes/service-account.key \
  --service-cluster-ip-range=10.32.0.0/24 \
  --use-service-account-credentials=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

cat > kube-scheduler.yaml <<'EOF'
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /var/lib/kubernetes/kube-scheduler.kubeconfig
leaderElection:
  leaderElect: true
EOF

cat > kube-scheduler.service <<'EOF'
[Unit]
Description=Kubernetes Scheduler
Documentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/
After=kube-apiserver.service

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --config=/var/lib/kubernetes/kube-scheduler.yaml \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

cat > etcdctl.sh <<'EOF'
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/etc/etcd/etcd-ca.crt
export ETCDCTL_CERT=/etc/etcd/apiserver-etcd-client.crt
export ETCDCTL_KEY=/etc/etcd/apiserver-etcd-client.key

# The full status table has 16 columns. Print the four this guide reads.
etcd_status() {
  etcdctl endpoint status --cluster | awk -F', ' '{printf "%-26s leader=%-5s term=%s index=%s\n", $1, $9, $11, $12}'
}
EOF

The last file saves typing. etcdctl now needs a certificate for every call, and these variables give it one on each server. etcd_status shows each member’s address, whether it is the Raft leader, the Raft term and the Raft index.

Step 7. Start etcd and the control plane

Copy each server its files, including its own etcd certificates and service files.

cd ~/kthw-ha
for HOST in server-0 server-1 server-2; do
  scp downloads/etcd downloads/etcdctl downloads/kube-apiserver \
    downloads/kube-controller-manager downloads/kube-scheduler downloads/kubectl \
    pki/ca.crt pki/ca.key pki/etcd-ca.crt pki/kube-apiserver.crt pki/kube-apiserver.key \
    pki/apiserver-kubelet-client.crt pki/apiserver-kubelet-client.key \
    pki/apiserver-etcd-client.crt pki/apiserver-etcd-client.key \
    pki/service-account.key pki/service-account.pub \
    pki/etcd-${HOST}.crt pki/etcd-${HOST}.key pki/etcd-peer-${HOST}.crt pki/etcd-peer-${HOST}.key \
    configs/kube-controller-manager.kubeconfig configs/kube-scheduler.kubeconfig \
    configs/admin.kubeconfig configs/encryption-config.yaml \
    control-plane/etcd-${HOST}.service control-plane/kube-apiserver-${HOST}.service \
    control-plane/kube-controller-manager.service control-plane/kube-scheduler.service \
    control-plane/kube-scheduler.yaml control-plane/etcdctl.sh \
    root@${HOST}:~/
done

Run the next block on each of the three servers. It is the same on all three, because it reads the machine’s own name.

cd ~
H=$(hostname)
install -m 0755 etcd etcdctl kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/
rm etcd etcdctl kube-apiserver kube-controller-manager kube-scheduler kubectl

mkdir -p /etc/etcd /var/lib/etcd /var/lib/kubernetes
chmod 700 /var/lib/etcd
cp etcd-ca.crt apiserver-etcd-client.crt apiserver-etcd-client.key /etc/etcd/
mv etcd-${H}.crt /etc/etcd/server.crt
mv etcd-${H}.key /etc/etcd/server.key
mv etcd-peer-${H}.crt /etc/etcd/peer.crt
mv etcd-peer-${H}.key /etc/etcd/peer.key
mv ca.crt ca.key etcd-ca.crt kube-apiserver.crt kube-apiserver.key \
  apiserver-kubelet-client.crt apiserver-kubelet-client.key \
  apiserver-etcd-client.crt apiserver-etcd-client.key \
  service-account.key service-account.pub \
  kube-controller-manager.kubeconfig kube-scheduler.kubeconfig \
  encryption-config.yaml kube-scheduler.yaml \
  /var/lib/kubernetes/
chmod 600 /etc/etcd/*.key /var/lib/kubernetes/*.key /var/lib/kubernetes/*.kubeconfig /var/lib/kubernetes/encryption-config.yaml

mv etcd-${H}.service /etc/systemd/system/etcd.service
mv kube-apiserver-${H}.service /etc/systemd/system/kube-apiserver.service
mv kube-controller-manager.service kube-scheduler.service /etc/systemd/system/
mv etcdctl.sh /etc/profile.d/etcdctl.sh

systemctl daemon-reload
systemctl enable --now --no-block etcd

Notice --no-block. A brand-new etcd member cannot finish starting until a majority of the cluster is up, so the first member waits for the second. Without the flag, the command on server-0 would hang until you started etcd on another server. That is quorum in action before the cluster even exists.

Once all three have run the block, check the cluster from any server.

source /etc/profile.d/etcdctl.sh
until etcdctl endpoint health --cluster >/dev/null 2>&1; do sleep 2; done
etcdctl member list -w table
etcd_status

┌──────────────────┬─────────┬──────────┬──────────────────────────┬──────────────────────────┬────────────┐
│        ID        │ STATUS  │   NAME   │        PEER ADDRS        │       CLIENT ADDRS       │ IS LEARNER │
├──────────────────┼─────────┼──────────┼──────────────────────────┼──────────────────────────┼────────────┤
│ 1edeaefba0dfa2a1 │ started │ server-1 │ https://10.240.0.11:2380 │ https://10.240.0.11:2379 │      false │
│ 2c8697548d85d416 │ started │ server-2 │ https://10.240.0.12:2380 │ https://10.240.0.12:2379 │      false │
│ e1bbd6834e3595f5 │ started │ server-0 │ https://10.240.0.10:2380 │ https://10.240.0.10:2379 │      false │
└──────────────────┴─────────┴──────────┴──────────────────────────┴──────────────────────────┴────────────┘
https://10.240.0.11:2379   leader=false term=2 index=11
https://10.240.0.12:2379   leader=false term=2 index=11
https://10.240.0.10:2379   leader=true  term=2 index=11

Exactly one member is the Raft leader. All writes go through it, and the others follow. The term counts elections, and the index counts the changes written to the log, so three members on the same term and index hold the same data.

Now start the rest of the control plane on all three servers.

systemctl enable --now kube-apiserver kube-controller-manager kube-scheduler
until kubectl --kubeconfig ~/admin.kubeconfig get --raw='/readyz' >/dev/null 2>&1; do sleep 2; done
echo "$(hostname): API server ready"

Step 8. Put a load balancer in front

The api machine runs HAProxy. It passes TCP connections straight through to the API servers without decrypting them, so the certificates from step 3 stay end to end. Every two seconds it asks each API server /readyz, and it stops sending traffic to one that fails twice in a row.

apt-get update
apt-get install -y haproxy

cat > /etc/haproxy/haproxy.cfg <<'EOF'
global
  log /dev/log local0
  maxconn 4096

defaults
  mode tcp
  log global
  option tcplog
  timeout connect 5s
  timeout client 1h
  timeout server 1h

frontend kube-apiserver
  bind *:6443
  default_backend kube-apiserver

backend kube-apiserver
  option httpchk GET /readyz
  http-check expect status 200
  default-server inter 2s fall 2 rise 2 check check-ssl verify none
  server server-0 server-0:6443
  server server-1 server-1:6443
  server server-2 server-2:6443
EOF

haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl enable haproxy
systemctl restart haproxy

The long client and server timeouts matter. kubectl logs -f, kubectl exec and every controller’s watch hold one connection open for minutes, and a short timeout would cut them off. The health check skips certificate verification because it only asks whether the server is ready. The real traffic is never decrypted by the load balancer at all.

Point kubectl on the jumpbox at the load balancer, and grant the API servers access to the kubelets as in Part 1.

cd ~/kthw-ha
kubectl config set-cluster kthw-ha --certificate-authority=pki/ca.crt \
  --embed-certs=true --server=https://api:6443
kubectl config set-credentials admin-ha --client-certificate=pki/admin.crt \
  --client-key=pki/admin.key --embed-certs=true
kubectl config set-context kthw-ha --cluster=kthw-ha --user=admin-ha
kubectl config use-context kthw-ha

kubectl get --raw='/readyz' && echo
kubectl get endpointslices -l kubernetes.io/service-name=kubernetes

kubectl create clusterrolebinding kube-apiserver-kubelet-client \
  --clusterrole=system:kubelet-api-admin \
  --user=kube-apiserver-kubelet-client

ok
NAME         ADDRESSTYPE   PORTS   ENDPOINTS                             AGE
kubernetes   IPv4          6443    10.240.0.10,10.240.0.11,10.240.0.12   25s

The kubernetes service, the address pods use to reach the API, now lists all three API servers. Each API server registers itself and keeps a lease, and one that stops renewing is dropped from the list.

Step 9. Prepare and start the workers

Workers are set up exactly as in Part 1. The only difference is already baked in. Their kubeconfigs point at api, so a worker never depends on a single server. The commands below are Part 1’s steps 9 to 12, condensed. Read Part 1 for what each setting does.

mkdir -p ~/kthw-ha/workers && cd ~/kthw-ha/workers

cat > containerd-config.toml <<'EOF'
version = 4

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
  runtime_type = 'io.containerd.runc.v2'

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins.'io.containerd.cri.v1.runtime'.cni]
  bin_dirs = ['/opt/cni/bin']
  conf_dir = '/etc/cni/net.d'
EOF

cat > crictl.yaml <<'EOF'
runtime-endpoint: unix:///run/containerd/containerd.sock
EOF

cat > kubelet-config.yaml <<'EOF'
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: /var/lib/kubelet/ca.crt
authorization:
  mode: Webhook
cgroupDriver: systemd
clusterDomain: cluster.local
clusterDNS:
  - 10.32.0.10
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
resolvConf: /etc/resolv.conf
tlsCertFile: /var/lib/kubelet/kubelet.crt
tlsPrivateKeyFile: /var/lib/kubelet/kubelet.key
EOF

cat > kubelet.service <<'EOF'
[Unit]
Description=Kubernetes Kubelet
Documentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \
  --config=/var/lib/kubelet/kubelet-config.yaml \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --node-ip=NODE_IP \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

cat > kube-proxy-config.yaml <<'EOF'
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clientConnection:
  kubeconfig: /var/lib/kube-proxy/kubeconfig
mode: nftables
clusterCIDR: 10.200.0.0/16
EOF

cat > kube-proxy.service <<'EOF'
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --config=/var/lib/kube-proxy/kube-proxy-config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

cat > 10-bridge.conflist <<'EOF'
{
  "cniVersion": "1.0.0",
  "name": "bridge",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": false,
      "hairpinMode": true,
      "ipam": {
        "type": "host-local",
        "ranges": [[{ "subnet": "POD_SUBNET" }]],
        "routes": [{ "dst": "0.0.0.0/0" }]
      }
    },
    {
      "type": "portmap",
      "capabilities": { "portMappings": true }
    }
  ]
}
EOF

cat > pod-nat.nft <<'EOF'
# Translate pod traffic that leaves the cluster, and nothing else.
table ip pod-nat
delete table ip pod-nat
table ip pod-nat {
  chain postrouting {
    type nat hook postrouting priority srcnat; policy accept;
    ip saddr 10.200.0.0/16 ip daddr != 10.200.0.0/16 masquerade
  }
}
EOF

The pod network services come next, as in Part 1, for the workers and the three servers. On a routed network such as Hetzner Cloud, add the network routes first and set POD_GATEWAY, as Part 1 explains.

NETWORK=kthw
hcloud network add-route ${NETWORK} --destination 10.200.0.0/24 --gateway 10.240.0.20
hcloud network add-route ${NETWORK} --destination 10.200.1.0/24 --gateway 10.240.0.21

cd ~/kthw-ha/workers

# Empty when the machines share one network segment.
# On a routed network such as Hetzner Cloud, the private network's gateway.
POD_GATEWAY=""

while read IP HOST SUBNET; do
  if [ "${HOST}" = "api" ]; then continue; fi
  {
    cat <<'EOF'
[Unit]
Description=Pod network routes and NAT
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
Restart=on-failure
RestartSec=5
EOF
    if [ -n "${POD_GATEWAY}" ]; then
      echo "ExecStart=/usr/sbin/ip route replace 10.200.0.0/16 via ${POD_GATEWAY}"
    else
      while read PEER_IP PEER_HOST PEER_SUBNET; do
        if [ -n "${PEER_SUBNET}" ] && [ "${PEER_HOST}" != "${HOST}" ]; then
          echo "ExecStart=/usr/sbin/ip route replace ${PEER_SUBNET} via ${PEER_IP}"
        fi
      done < ../machines.txt
    fi
    if [ -n "${SUBNET}" ]; then
      echo "ExecStart=/usr/sbin/nft -f /etc/kubernetes/pod-nat.nft"
    fi
    cat <<'EOF'

[Install]
WantedBy=multi-user.target
EOF
  } > pod-network-${HOST}.service
done < ../machines.txt

Copy each worker its files and each server its pod network service.

cd ~/kthw-ha

while read IP HOST SUBNET; do
  if [ -z "${SUBNET}" ]; then continue; fi
  mkdir -p workers/${HOST}
  sed "s|POD_SUBNET|${SUBNET}|" workers/10-bridge.conflist > workers/${HOST}/10-bridge.conflist
  sed "s|NODE_IP|${IP}|" workers/kubelet.service > workers/${HOST}/kubelet.service

  scp downloads/kubelet downloads/kube-proxy downloads/runc \
    downloads/containerd-*.tar.gz downloads/cni-plugins-*.tgz downloads/crictl-*.tar.gz \
    downloads/containerd.service \
    pki/ca.crt pki/${HOST}.crt pki/${HOST}.key \
    configs/${HOST}.kubeconfig configs/kube-proxy.kubeconfig \
    workers/containerd-config.toml workers/crictl.yaml workers/kubelet-config.yaml \
    workers/kube-proxy-config.yaml workers/kube-proxy.service workers/pod-nat.nft \
    workers/${HOST}/10-bridge.conflist workers/${HOST}/kubelet.service \
    workers/pod-network-${HOST}.service \
    root@${HOST}:~/
done < machines.txt

for HOST in server-0 server-1 server-2; do
  scp workers/pod-network-${HOST}.service root@${HOST}:~/
done

mv ~/pod-network-$(hostname).service /etc/systemd/system/pod-network.service
systemctl daemon-reload
systemctl enable --now pod-network
ip route | grep 10.200

Then on each worker, the same three steps as Part 1, in one block.

apt-get update
apt-get install -y conntrack nftables

swapoff -a
sed -i '/\sswap\s/ s/^/#/' /etc/fstab

cat > /etc/modules-load.d/kubernetes.conf <<'EOF'
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter

cat > /etc/sysctl.d/99-kubernetes.conf <<'EOF'
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sysctl --system >/dev/null

cd ~
tar -xzf containerd-*.tar.gz -C /usr/local
install -m 0755 runc /usr/local/sbin/runc
mkdir -p /opt/cni/bin /etc/cni/net.d /etc/containerd /etc/kubernetes
tar -xzf cni-plugins-*.tgz -C /opt/cni/bin
tar -xzf crictl-*.tar.gz -C /usr/local/bin
rm containerd-*.tar.gz cni-plugins-*.tgz crictl-*.tar.gz runc

mv containerd-config.toml /etc/containerd/config.toml
mv crictl.yaml /etc/crictl.yaml
mv 10-bridge.conflist /etc/cni/net.d/
mv pod-nat.nft /etc/kubernetes/pod-nat.nft
mv containerd.service /etc/systemd/system/containerd.service
mv pod-network-$(hostname).service /etc/systemd/system/pod-network.service

install -m 0755 kubelet kube-proxy /usr/local/bin/
rm kubelet kube-proxy
mkdir -p /var/lib/kubelet /var/lib/kube-proxy
mv ca.crt /var/lib/kubelet/ca.crt
mv $(hostname).crt /var/lib/kubelet/kubelet.crt
mv $(hostname).key /var/lib/kubelet/kubelet.key
mv $(hostname).kubeconfig /var/lib/kubelet/kubeconfig
mv kubelet-config.yaml /var/lib/kubelet/kubelet-config.yaml
mv kube-proxy.kubeconfig /var/lib/kube-proxy/kubeconfig
mv kube-proxy-config.yaml /var/lib/kube-proxy/kube-proxy-config.yaml
chmod 600 /var/lib/kubelet/kubelet.key /var/lib/kubelet/kubeconfig /var/lib/kube-proxy/kubeconfig

if systemctl is-active --quiet systemd-resolved; then
  sed -i 's|^resolvConf: .*|resolvConf: /run/systemd/resolve/resolv.conf|' /var/lib/kubelet/kubelet-config.yaml
fi

mv kubelet.service kube-proxy.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now containerd pod-network kubelet kube-proxy

Step 10. Add DNS and something to keep running

CoreDNS as in Part 1, then an nginx deployment with a NodePort service. It is what you will watch keep working while the control plane breaks.

cd ~/kthw-ha
kubectl wait --for=create node/node-0 node/node-1 --timeout=120s
kubectl wait --for=condition=Ready node/node-0 node/node-1 --timeout=180s

curl -fsSL https://raw.githubusercontent.com/kubernetes/kubernetes/v1.37.1/cluster/addons/dns/coredns/coredns.yaml.base \
  | sed -e 's/__DNS__DOMAIN__/cluster.local/g' \
        -e 's/__DNS__SERVER__/10.32.0.10/g' \
        -e 's/__DNS__MEMORY__LIMIT__/170Mi/g' \
  > coredns.yaml
kubectl apply -f coredns.yaml
kubectl -n kube-system rollout status deployment/coredns --timeout=180s

kubectl create deployment nginx --image=nginx:1.30 --replicas=2
kubectl rollout status deployment/nginx --timeout=180s
kubectl expose deployment nginx --name=nginx-nodeport --type=NodePort --port=80
kubectl get service nginx-nodeport -o jsonpath='{.spec.ports[0].nodePort}' > nodeport.txt
curl -s --retry 5 --retry-connrefused http://node-0:$(cat nodeport.txt) | grep -m1 -o 'Welcome to nginx!'

The NodePort is saved to a file on purpose. In the second drill you will want it at a moment when kubectl cannot answer.

Step 11. Break it on purpose

Who is in charge

Before breaking anything, look at who leads. The etcd leader, the active controller manager and the active scheduler can each sit on a different server. In our run, server-0 held all three.

ssh root@server-0 'source /etc/profile.d/etcdctl.sh && etcd_status'
kubectl -n kube-system get leases kube-controller-manager kube-scheduler \
  -o custom-columns=NAME:.metadata.name,HOLDER:.spec.holderIdentity

https://10.240.0.11:2379   leader=false term=2 index=692
https://10.240.0.12:2379   leader=false term=2 index=692
https://10.240.0.10:2379   leader=true  term=2 index=692
NAME                      HOLDER
kube-controller-manager   server-0_2083cf01-10c2-4e6d-98ac-caf23a90f184
kube-scheduler            server-0_fe1834af-6f07-45ce-93e9-76e0f83cd619

Drill 1. Lose one server

Stop every control plane component on server-0. A crashed or powered-off machine looks exactly the same to the rest of the cluster.

ssh root@server-0 systemctl stop kube-apiserver kube-controller-manager kube-scheduler etcd
sleep 20

kubectl get nodes
kubectl scale deployment nginx --replicas=3
kubectl rollout status deployment/nginx --timeout=120s

# This exits non-zero while a member is down, which is the point
ssh root@server-1 'source /etc/profile.d/etcdctl.sh && etcdctl endpoint health --cluster' || true
kubectl -n kube-system get leases kube-controller-manager kube-scheduler \
  -o custom-columns=NAME:.metadata.name,HOLDER:.spec.holderIdentity

The end of the output, with one long etcd client warning left out, looked like this in our run.

deployment.apps/nginx scaled
Waiting for deployment "nginx" rollout to finish: 2 of 3 updated replicas are available...
deployment "nginx" successfully rolled out
https://10.240.0.11:2379 is healthy: successfully committed proposal: took = 8.303587ms
https://10.240.0.12:2379 is healthy: successfully committed proposal: took = 13.511406ms
https://10.240.0.10:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
NAME                      HOLDER
kube-controller-manager   server-1_b209aebf-dec7-46cd-b404-f2b663dea352
kube-scheduler            server-2_266131f0-88ef-4786-901d-fec161b5f51f

Nothing you can see broke. kubectl still answers, because HAProxy stopped sending traffic to server-0 within a few seconds. The deployment scaled up, which needs the scheduler and the controller manager. etcd reports server-0 unhealthy, and the remaining two still form a majority, so they elected a new leader between them. The two leases moved too, here to two different servers. A standby takes over a lease once it has gone unrenewed for 15 seconds, the default lease duration.

Drill 2. Lose a second server

Now stop server-1 too. Only one of three etcd members is left, and one is not a majority.

cd ~/kthw-ha
ssh root@server-1 systemctl stop kube-apiserver kube-controller-manager kube-scheduler etcd
sleep 15

kubectl get nodes --request-timeout=10s || true
curl -s http://node-0:$(cat nodeport.txt) | grep -m1 -o 'Welcome to nginx!'
curl -s http://node-1:$(cat nodeport.txt) | grep -m1 -o 'Welcome to nginx!'

context deadline exceeded - error from a previous attempt: EOF
Welcome to nginx!
Welcome to nginx!

This is the most important lesson in this guide. The control plane is down. kubectl gets no answer, because server-2’s API server cannot reach a majority of etcd and fails its readiness check, so HAProxy has nowhere to send you. On api, journalctl -u haproxy logs each attempt as kube-apiserver/<NOSRV>, no server available. Nothing can be scheduled, scaled or changed.

And your application keeps serving. The pods were already running on the workers, kube-proxy already had its rules, and the kubelets keep the containers alive on their own. A control plane outage freezes the cluster’s state. It does not take down what is already running.

Drill 3. Bring it back

Start server-1 again. Two of three members are back, which is a majority, and everything resumes.

ssh root@server-1 systemctl start etcd kube-apiserver kube-controller-manager kube-scheduler
until kubectl get --raw='/readyz' >/dev/null 2>&1; do sleep 2; done
kubectl get nodes

ssh root@server-0 systemctl start etcd kube-apiserver kube-controller-manager kube-scheduler
sleep 15
ssh root@server-2 'source /etc/profile.d/etcdctl.sh && etcd_status'

NAME     STATUS   ROLES    AGE     VERSION
node-0   Ready    <none>   3m48s   v1.37.1
node-1   Ready    <none>   3m32s   v1.37.1
https://10.240.0.11:2379   leader=false term=5 index=1184
https://10.240.0.12:2379   leader=true  term=5 index=1184
https://10.240.0.10:2379   leader=false term=5 index=1184

server-0 rejoins and catches up on everything it missed. All three members report the same term and the same index again, so every member holds the same data. The term went up because the cluster held new elections while you broke it.

When something does not work

etcd never becomes healthy. Check journalctl -u etcd on each server. x509: certificate is valid for ..., not ... means a certificate is missing an address from step 3. connection refused on port 2380 usually means etcd is not running on the peer yet, or a firewall blocks ports 2379 and 2380 between the servers.
systemctl enable --now etcd hangs. You left out --no-block and the other members are not up yet. Start etcd on a second server and the first one finishes.
kubectl through api fails but each server answers locally. Run journalctl -u haproxy on api. HAProxy logs every server it marks down or up, with the reason from the health check.
etcdctl exits with conflicting environment variable is shadowed by corresponding command-line flag. You passed --endpoints while ETCDCTL_ENDPOINTS is set from etcdctl.sh. etcdctl refuses to pick one. Run unset ETCDCTL_ENDPOINTS first, or leave out the flag.
Pods fail with Unauthorized after a while. The API servers are not using the same service-account.key. Every server must have an identical copy.
A Secret reads fine through one API server and fails through another. The encryption config differs between servers. Copy the same file everywhere and restart the API servers.

What this control plane still does not have

The load balancer is now the single point of failure. In production you run two load balancers sharing a floating address with keepalived, or use your cloud’s managed load balancer.
No backups. Three etcd members protect you from losing a machine, not from deleting the wrong namespace. Part 3 of this series takes a snapshot and restores the cluster after deleting its control plane.
Certificates still expire in a year, now on more machines. Part 5 of this series renews them, mostly without a restart.

The easy way, once you know the hard way

Everything in this part is one flag on an Ankra cluster.

ankra cluster hetzner create --name my-cluster --location hel1 \
  --credential-id <your-hetzner-credential> \
  --control-plane-count 3 --worker-count 2

That builds three control planes with etcd running on them, the same stacked layout you built here. If you would rather keep etcd on its own machines, --etcd-topology external gives it three or five dedicated servers. The same works on the other providers Ankra builds on, and the first part explains what you get on top.

When a control plane machine does fail, you now know exactly what Ankra is telling you, and why your applications kept running while it was gone.

This is Part 2 of Kubernetes the Hard Way, a series of guides we test end to end before publishing. Part 3 covers backing up etcd and restoring a lost control plane.

Get started: Create a free account on Ankra.

Join our community: Slack

ShareX LinkedIn HN

Get the next post in your inbox

Kubernetes Engineering Homelab

Kubernetes the Hard Way, Part 3: Back Up etcd and Restore a Lost Control Plane

Back up a Kubernetes 1.37 cluster by hand with etcd snapshots, then delete the control plane and restore it. Includes the encryption key trap, all tested.

Mattias Åsell

October 25, 2026

12 min read

Kubernetes Engineering Homelab

Kubernetes the Hard Way, Part 5: Certificates, Expiry and Renewal Without Downtime

Read every Kubernetes certificate, watch one expire, and renew them all by hand. Most components reload a renewed certificate with no restart at all. Tested.

Mattias Åsell

October 25, 2026

9 min read

Back to Blog

Kubernetes the Hard Way, Part 2: A Highly Available Control Plane

What you will build

Why three, and not two

Before you start

Step 1. Prepare the jumpbox

Step 2. Download the binaries

Step 3. Create the certificates

Step 4. Create the kubeconfig files

Step 5. One encryption key for all three servers

Step 6. Write the control plane services

Step 7. Start etcd and the control plane

Step 8. Put a load balancer in front

Step 9. Prepare and start the workers

Step 10. Add DNS and something to keep running

Step 11. Break it on purpose

Who is in charge

Drill 1. Lose one server

Drill 2. Lose a second server

Drill 3. Bring it back

When something does not work

What this control plane still does not have

The easy way, once you know the hard way

Get the next post in your inbox

Related Posts

Kubernetes the Hard Way, Part 3: Back Up etcd and Restore a Lost Control Plane

Kubernetes the Hard Way, Part 5: Certificates, Expiry and Renewal Without Downtime

Cookie Settings

Necessary Cookies

Analytics and Ad Measurement