Kubernetes the Hard Way, Part 3: Back Up etcd and Restore a Lost Control Plane

A backup you have never restored is a hope, not a backup. This part of the series makes it real. You will back up the cluster from Part 1, delete everything its control plane stored, and bring the cluster back from that backup.

On the way you will hit the trap that catches most first restores. The etcd snapshot holds every object in the cluster, but the Secrets inside it are encrypted with a key that is not in the snapshot. Restore the data without that key and no Secret can be read. You will make that mistake on purpose, see exactly what it looks like, and fix it.

We ran every command in this post, top to bottom, on fresh machines before publishing it, and the outputs on this page are the ones we got.

What you need

The cluster from Part 1, with its jumpbox, its server and both workers, and Part 1’s files still in ~/kthw on the jumpbox. The commands use Part 1’s names and paths. The only new tool is already inside the etcd archive you downloaded in Part 1.

What a backup has to contain

Everything you create in Kubernetes, every Deployment, Secret, ConfigMap and node, is stored in etcd and nowhere else. The workers run containers, but they get their instructions from the API server, which reads them from etcd. So the heart of a backup is a copy of etcd.

That copy is not enough on its own. Some files on the server are not in etcd at all, and the cluster cannot run without them.

• The encryption config holds the key that encrypts Secrets before they reach etcd. Without it, the Secrets in your backup cannot be read.
• The certificate authority and its key signed every certificate in the cluster. A new CA means new certificates for every component and every kubeconfig.
• The service account key pair signs the tokens pods use to talk to the API. A new key makes every existing token invalid.
• The kubeconfigs and unit files are easy to write again, but copying them is easier still.

In Part 1 all of these live in /var/lib/kubernetes on the server, with a second copy in ~/kthw on the jumpbox. A real cluster often has no such second copy, so this part backs them up from the server itself.

Step 1. Put something worth saving in the cluster

Create a namespace with a Secret, a ConfigMap and a small Deployment. You will check each of them after the restore. Note the NodePort of Part 1’s nginx service too, for a test later.

kubectl create namespace shop
kubectl -n shop create secret generic db-password --from-literal=password=correct-horse-battery-staple
kubectl -n shop create configmap settings --from-literal=currency=EUR
kubectl -n shop create deployment web --image=nginx:1.30 --replicas=2
kubectl -n shop rollout status deployment/web --timeout=180s

mkdir -p ~/kthw/backups
kubectl get service nginx-nodeport -o jsonpath='{.spec.ports[0].nodePort}' > ~/kthw/backups/nodeport.txt

Step 2. Take a snapshot

etcd ships three tools. etcd is the database, etcdctl talks to a running etcd, and etcdutl works on etcd’s files directly, which is what a restore needs. Part 1 installed the first two. The third is in the same archive.

cd ~/kthw/downloads
ETCD_VERSION=v3.7.2
ARCH=$(dpkg --print-architecture)
tar -xzf etcd-${ETCD_VERSION}-linux-${ARCH}.tar.gz --strip-components=1 \
  etcd-${ETCD_VERSION}-linux-${ARCH}/etcdutl
scp etcdutl root@server:/usr/local/bin/

Now take a snapshot on the server and look at it.

mkdir -p /var/backups/kubernetes
etcdctl snapshot save /var/backups/kubernetes/etcd-snapshot.db
etcdutl snapshot status /var/backups/kubernetes/etcd-snapshot.db -w table

etcdctl prints a few log lines in JSON first. The rest of the output looked like this.

Snapshot saved at /var/backups/kubernetes/etcd-snapshot.db
Server version 3.7.0
┌──────────┬──────────┬────────────┬────────────┬─────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ VERSION │
├──────────┼──────────┼────────────┼────────────┼─────────┤
│ 1fd5d925 │      596 │        346 │     823 kB │   3.7.0 │
└──────────┴──────────┴────────────┴────────────┴─────────┘

etcdctl snapshot save asks the running etcd for a consistent copy of its data at one moment, so it is safe while the cluster works. Copying the files under /var/lib/etcd with cp is not. etcd writes to them constantly, and a copy taken in the middle of a write can be corrupt.

The status table tells you what you caught. The revision counts every change etcd has stored so far, and the total keys counts the objects in the snapshot. The version column is the format of the data, which can be older than the etcd binary that wrote it.

Step 3. Take snapshots on a schedule

A snapshot from last month restores last month’s cluster. A small script and a systemd timer take one every hour and keep the newest 24.

cat > /usr/local/bin/etcd-backup <<'EOF'
#!/bin/bash
set -euo pipefail
DIR=/var/backups/kubernetes/hourly
mkdir -p "${DIR}"
etcdctl snapshot save "${DIR}/etcd-$(date -u +%Y%m%dT%H%M%SZ).db" >/dev/null
ls -1t "${DIR}"/etcd-*.db | tail -n +25 | xargs -r rm --
EOF
chmod 0755 /usr/local/bin/etcd-backup

cat > /etc/systemd/system/etcd-backup.service <<'EOF'
[Unit]
Description=Take an etcd snapshot
After=etcd.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/etcd-backup
EOF

cat > /etc/systemd/system/etcd-backup.timer <<'EOF'
[Unit]
Description=Take an etcd snapshot every hour

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target
EOF

systemctl daemon-reload
systemctl enable --now etcd-backup.timer
systemctl start etcd-backup.service
ls -l /var/backups/kubernetes/hourly
systemctl list-timers etcd-backup.timer --no-pager

total 808
-rw------- 1 root root 823328 Oct 25 15:31 etcd-20261025T153147Z.db
NEXT                         LEFT LAST PASSED UNIT              ACTIVATES
Sun 2026-10-25 16:00:00 UTC 28min -         - etcd-backup.timer etcd-backup.service

1 timers listed.
Pass --all to see loaded but inactive timers, too.

The first snapshot is there, because you started the service by hand. The timer has not fired yet, which is why its last run is empty, and its next run is at the top of the hour.

Persistent=true makes up for runs the machine missed while it was switched off. The ls -1t ... | tail -n +25 line deletes all but the 24 newest snapshots, so the disk never fills up.

These snapshots protect you from mistakes, such as a deleted namespace. They do not protect you from losing the server, because they live on it. Step 5 copies a backup off the machine, and step 7 shows what happens to everything that stayed behind.

Step 4. Back up the files etcd does not hold

Pack the control plane’s files into one archive next to the snapshot, including the backup job you just wrote.

tar -czf /var/backups/kubernetes/control-plane-files.tar.gz \
  /var/lib/kubernetes \
  /etc/systemd/system/etcd.service /etc/systemd/system/kube-apiserver.service \
  /etc/systemd/system/kube-controller-manager.service /etc/systemd/system/kube-scheduler.service \
  /etc/systemd/system/etcd-backup.service /etc/systemd/system/etcd-backup.timer \
  /usr/local/bin/etcd-backup
chmod 600 /var/backups/kubernetes/*.db /var/backups/kubernetes/*.tar.gz
tar -tzf /var/backups/kubernetes/control-plane-files.tar.gz | grep -E 'encryption|ca.key|service-account'

tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
var/lib/kubernetes/service-account.key
var/lib/kubernetes/service-account.pub
var/lib/kubernetes/encryption-config.yaml
var/lib/kubernetes/ca.key

The two tar notes are harmless. They mean the paths inside the archive are relative, so you choose where they land when you unpack it.

This archive is the most sensitive file you own. It holds the key that decrypts your Secrets and the key that can sign a certificate for any user, including a cluster administrator. Anyone with both the archive and a snapshot can read every Secret in the cluster. Keep it encrypted, and in production store it apart from the snapshots.

Step 5. Get the backup off the server

A backup on the machine it protects is lost together with that machine. Copy both files to the jumpbox and check the copy of the snapshot.

cd ~/kthw/backups
scp root@server:/var/backups/kubernetes/etcd-snapshot.db root@server:/var/backups/kubernetes/control-plane-files.tar.gz .
chmod 600 etcd-snapshot.db control-plane-files.tar.gz
../downloads/etcdutl snapshot status etcd-snapshot.db -w table

┌──────────┬──────────┬────────────┬────────────┬─────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ VERSION │
├──────────┼──────────┼────────────┼────────────┼─────────┤
│ 1fd5d925 │      596 │        346 │     823 kB │   3.7.0 │
└──────────┴──────────┴────────────┴────────────┴─────────┘

The hash matches the one on the server, so the copy is complete. etcdutl snapshot status is the cheapest restore test there is, because a truncated or damaged file fails it.

The hourly snapshots from step 3 still sit on the server. In production, a second job copies each one somewhere else, such as object storage in another location, and checks that the newest copy passes etcdutl snapshot status.

Step 6. Change something after the backup

Anything you do after a snapshot is not in it. Make two changes now, so you can watch them disappear.

kubectl -n shop create configmap after-backup --from-literal=note=made-after-the-snapshot
kubectl -n shop scale deployment web --replicas=4
kubectl -n shop rollout status deployment/web --timeout=180s
kubectl -n shop get pods -o wide

NAME                  READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
web-9cf49b6d6-8rhzl   1/1     Running   0          16s   10.200.1.4   node-1   <none>           <none>
web-9cf49b6d6-hj4fj   1/1     Running   0          1s    10.200.1.5   node-1   <none>           <none>
web-9cf49b6d6-kxn88   1/1     Running   0          1s    10.200.0.5   node-0   <none>           <none>
web-9cf49b6d6-lcv5x   1/1     Running   0          16s   10.200.0.4   node-0   <none>           <none>

Two of these pods existed when you took the snapshot, and two did not. Remember their names.

Step 7. Lose the control plane

Now destroy the control plane. Stop every component and delete everything it stored. That is the etcd data, the certificates and keys, the unit files, the binaries, and the backups that were on the server. What is left is the same as a server whose disk died and was replaced.

systemctl disable --now etcd-backup.timer
systemctl stop kube-scheduler kube-controller-manager kube-apiserver etcd
rm -rf /var/lib/etcd /var/lib/kubernetes /var/backups/kubernetes
rm -f /etc/systemd/system/etcd.service /etc/systemd/system/kube-apiserver.service \
  /etc/systemd/system/kube-controller-manager.service /etc/systemd/system/kube-scheduler.service \
  /etc/systemd/system/etcd-backup.service /etc/systemd/system/etcd-backup.timer
rm -f /usr/local/bin/etcd /usr/local/bin/etcdctl /usr/local/bin/etcdutl /usr/local/bin/etcd-backup \
  /usr/local/bin/kube-apiserver /usr/local/bin/kube-controller-manager /usr/local/bin/kube-scheduler \
  /usr/local/bin/kubectl
systemctl daemon-reload

Look at the cluster from the jumpbox.

kubectl get nodes --request-timeout=10s || true
NODE_PORT=$(cat ~/kthw/backups/nodeport.txt)
curl -s -m 5 http://node-0:${NODE_PORT} | grep -m1 -o 'Welcome to nginx!'
curl -s -m 5 http://node-1:${NODE_PORT} | grep -m1 -o 'Welcome to nginx!'
ssh root@node-0 'crictl ps --name nginx -q | wc -l'
ssh root@node-1 'crictl ps --name nginx -q | wc -l'

The connection to the server server:6443 was refused - did you specify the right host or port?
Welcome to nginx!
Welcome to nginx!
3
3

The control plane is gone, so kubectl has nothing to talk to. The applications on the workers keep serving. Each worker still runs three nginx containers, its share of Part 1’s two and of the four web pods. A kubelet keeps its containers running on its own, and kube-proxy keeps the rules it already wrote, so the outage takes down nothing that was already running. It does freeze the cluster, because nothing can be scheduled, scaled or repaired until the control plane is back.

Step 8. Restore, the wrong way first

Put the binaries and the two backup files back on the server.

cd ~/kthw
scp downloads/etcd downloads/etcdctl downloads/etcdutl downloads/kube-apiserver \
  downloads/kube-controller-manager downloads/kube-scheduler downloads/kubectl \
  backups/etcd-snapshot.db backups/control-plane-files.tar.gz \
  root@server:~/

cd ~
install -m 0755 etcd etcdctl etcdutl kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/
rm etcd etcdctl etcdutl kube-apiserver kube-controller-manager kube-scheduler kubectl
tar -xzf control-plane-files.tar.gz -C /
ls /var/lib/kubernetes

apiserver-kubelet-client.crt
apiserver-kubelet-client.key
ca.crt
ca.key
encryption-config.yaml
kube-apiserver.crt
kube-apiserver.key
kube-controller-manager.kubeconfig
kube-scheduler.kubeconfig
kube-scheduler.yaml
service-account.key
service-account.pub

Now the restore itself. etcdutl snapshot restore builds a new etcd data directory from the snapshot. It needs the member name, peer address and cluster token the cluster was created with, all of which you can read in etcd.service.

etcdutl snapshot restore ~/etcd-snapshot.db \
  --name server \
  --data-dir /var/lib/etcd \
  --initial-cluster server=http://127.0.0.1:2380 \
  --initial-advertise-peer-urls http://127.0.0.1:2380 \
  --initial-cluster-token kthw \
  --bump-revision 1000000000 \
  --mark-compacted
chmod 700 /var/lib/etcd

etcdutl logs every step. Two of its lines show the flags below at work.

2026-10-25T15:33:19Z  info  snapshot/v3_snapshot.go:397  bumping latest revision  {"latest-revision": 596, "bump-amount": 1000000000, "new-latest-revision": 1000000596}
2026-10-25T15:33:19Z  info  snapshot/v3_snapshot.go:414  marking revision compacted  {"revision": 1000000596}

The last two flags matter for Kubernetes. Every kubelet and kube-proxy remembers the last revision it saw, and that revision is newer than your snapshot, because the cluster kept changing after it. --bump-revision moves the restored database’s revision far ahead of anything a client remembers (a billion covers a week-old snapshot on a busy cluster, by the etcd documentation’s own estimate), and --mark-compacted tells those clients that the revisions they remember no longer exist. Together they make every component throw away what it cached and read the cluster again from scratch, instead of waiting for changes that will never arrive.

Here is the wrong turn. Say the encryption config was not in your backup, so you write a new one with a fresh key, exactly as Part 1 did.

NEW_KEY=$(head -c 32 /dev/urandom | base64)
sed -i "s|secret: .*|secret: ${NEW_KEY}|" /var/lib/kubernetes/encryption-config.yaml

systemctl daemon-reload
systemctl enable --now etcd
systemctl enable --now kube-apiserver kube-controller-manager kube-scheduler

until kubectl -n shop get configmap settings >/dev/null 2>&1; do sleep 2; done
kubectl -n shop get configmap settings -o jsonpath='{.data.currency}'; echo
kubectl -n shop get secret db-password -o jsonpath='{.data.password}' || true
kubectl get secrets -A || true

# Give the API server a minute and a half to finish starting, then ask what is still not ready
sleep 90
curl -sk https://server:6443/readyz?verbose | grep -v '^\[+\]'

EUR
Error from server (InternalError): Internal error occurred: StorageError: corrupt object, Code: 7, Key: /secrets/shop/db-password, ResourceVersion: 0, AdditionalErrorMsg: data from the storage is not transformable revision=0: output array was not large enough for encryption
Error from server (StorageReadError): failed to read one or more secrets from the storage: StorageError: corrupt object, Code: 7, Key: /registry/secrets/default/kthw, ResourceVersion: 0, AdditionalErrorMsg: data from the storage is not transformable revision=0: output array was not large enough for encryption (and 1 more)
[-]informer-sync failed: reason withheld
readyz check failed

The ConfigMap reads fine, because ConfigMaps are not encrypted. Every Secret fails. The API server calls the data corrupt, but nothing is damaged. It holds a key that did not encrypt this data, so it cannot turn it back into a Secret. Listing all Secrets fails too, and it names both of the cluster’s Secrets, Part 1’s kthw and your db-password.

The last two lines are the worst part. The API server never becomes ready. Right after a start, a few more checks fail while it finishes booting, which is why the block waits. After that only one is left. It waits until the API server has loaded every Secret, and that can never happen. Anything that waits for a ready API server, such as the load balancer from Part 2, sends no traffic to it at all.

If the original key is truly lost, so are these Secrets. Every password, token and certificate stored in them has to be created again from its source. That is why the encryption config belongs in the backup.

Step 9. Fix it with the real key

The right key is in the archive from step 4. Put the original encryption config back and restart the API server.

tar -xzf ~/control-plane-files.tar.gz -C / var/lib/kubernetes/encryption-config.yaml
systemctl restart kube-apiserver

until kubectl get --raw='/readyz' >/dev/null 2>&1; do sleep 2; done
kubectl -n shop get secret db-password -o jsonpath='{.data.password}' | base64 -d; echo

correct-horse-battery-staple

Same snapshot, right key, and the Secret is back. The API server reports ready again.

Step 10. Check what came back

kubectl get nodes
kubectl -n shop get configmaps
kubectl -n shop get deployment web
kubectl -n shop get pods -o wide

NAME     STATUS   ROLES    AGE     VERSION
node-0   Ready    <none>   9m35s   v1.37.1
node-1   Ready    <none>   9m34s   v1.37.1
NAME               DATA   AGE
kube-root-ca.crt   1      8m45s
settings           1      8m44s
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    2/2     2            2           8m44s
NAME                  READY   STATUS    RESTARTS   AGE     IP           NODE     NOMINATED NODE   READINESS GATES
web-9cf49b6d6-8rhzl   1/1     Running   0          8m44s   10.200.1.4   node-1   <none>           <none>
web-9cf49b6d6-lcv5x   1/1     Running   0          8m44s   10.200.0.4   node-0   <none>           <none>

The cluster is back as it was at the moment of the snapshot. Both nodes are Ready without being touched, and the settings ConfigMap is there.

Everything after the snapshot is gone. The after-backup ConfigMap does not exist, and the deployment wants two replicas again, not four. The two pods that are left are the two that existed at snapshot time. The other two were still running on the workers, but the restored cluster had no record of them, so each kubelet stopped its extra pod as soon as the API server answered again. The kubelet log on node-0 shows it.

"SyncLoop REMOVE" source="api" pods=["shop/web-9cf49b6d6-kxn88"]
"Killing container with a grace period" pod="shop/web-9cf49b6d6-kxn88"

That is the price of a backup, and it has a name. The recovery point objective is how much work you accept to lose. With hourly snapshots it is up to an hour. If that is too much, take snapshots more often.

Last, turn the hourly snapshots back on. The timer came back with the archive, but it was disabled when the server was wiped.

systemctl daemon-reload
systemctl enable --now etcd-backup.timer
systemctl start etcd-backup.service
ls /var/backups/kubernetes/hourly

etcd-20261025T154027Z.db

After a restore, take a fresh backup before anything else. Your last good one is now the snapshot you restored from, and everything since then exists only in the cluster.

When something does not work

• Every Secret fails with corrupt object and data from the storage is not transformable, and the API server never becomes ready. The encryption config does not hold the key that encrypted the data. Put back the original encryption-config.yaml and restart the API server.
• etcdutl snapshot restore fails with data-dir "/var/lib/etcd" not empty or could not be read. The restore only writes into a directory that does not exist yet. Stop etcd and move the old directory out of the way first.
• etcdutl snapshot status crashes with panic: assertion failed: Page expected to be.... The snapshot file is damaged or incomplete, usually from a copy that did not finish. Copy it again, or take the next older snapshot.
• CoreDNS logs one forbidden error right after the API server restarts. The API server answers before it has loaded all permissions. One error at restart is harmless. If they keep coming, the RBAC objects were not restored.

Restoring the three-member cluster from Part 2

The three-member etcd from Part 2 is backed up the same way, with one snapshot from any member, and its etcdctl talks TLS through the variables in /etc/profile.d/etcdctl.sh. A restore is different. Every member is restored from the same snapshot file, each with its own --name and --initial-advertise-peer-urls and with the full three-member --initial-cluster list, and then all three are started. The etcd disaster recovery guide walks through it. Never restore only one member of a running cluster. A restored member is a new cluster and cannot join the old one.

The easy way, once you know the hard way

Everything in this part protects the cluster’s objects. It does not protect what your applications store on their volumes, such as the rows in a database, and that is usually the data people miss most.

Ankra covers that layer. You give it a backup vault, an S3-compatible bucket it can also create for you on Hetzner, UpCloud, DigitalOcean or Scaleway, and protect each stack that holds data.

ankra backup vaults provision
ankra cluster stacks protect shop --vault <your-vault> \
  --schedule daily --retention daily=7,weekly=4,monthly=6 \
  --backup-now --wait

Each run seals a restore point, a copy of that stack’s databases and the volumes you name, readable without the cluster it came from. ankra cluster backups status shows every stack and whether it is protected, and each restore point lists what it does not carry, next to what it does. You can restore a stack in place, or clone it with its data onto another cluster to rehearse the restore you practised here by hand.

The habit from this part still applies. A restore point you have never restored is a hope. Clone a stack onto a test cluster now and then, and look at what comes back.

This is Part 3 of Kubernetes the Hard Way, a series of guides we test end to end before publishing. Part 4 covers pod networking by hand.

Get started: Create a free account on Ankra.

Join our community: Slack

Follow us on: LinkedIn | GitHub

Contact us: [email protected]