# Kubernetes the Hard Way, Part 3: Back Up etcd and Restore a Lost Control Plane | Ankra Canonical: https://ankra.ai/blog/kubernetes-the-hard-way-backup-restore Back up a Kubernetes 1.37 cluster by hand with etcd snapshots, then delete the control plane and restore it. Includes the encryption key trap, all tested. [Back to Blog](https://ankra.ai/blog) [Kubernetes](https://ankra.ai/blog/tag/kubernetes) [Engineering](https://ankra.ai/blog/tag/engineering) [Homelab](https://ankra.ai/blog/tag/homelab) # Kubernetes the Hard Way, Part 3: Back Up etcd and Restore a Lost Control Plane [Mattias Åsell](https://ankra.ai/blog/author/mattias-asell) October 25, 2026 12 min read ### On this page A backup you have never restored is a hope, not a backup. This part of the series makes it real. You will back up the cluster from [Part 1](https://ankra.ai/blog/kubernetes-the-hard-way) , delete everything its control plane stored, and bring the cluster back from that backup. On the way you will hit the trap that catches most first restores. The etcd snapshot holds every object in the cluster, but the Secrets inside it are encrypted with a key that is not in the snapshot. Restore the data without that key and no Secret can be read. You will make that mistake on purpose, see exactly what it looks like, and fix it. We ran every command in this post, top to bottom, on fresh machines before publishing it, and the outputs on this page are the ones we got. ## What you need The cluster from [Part 1](https://ankra.ai/blog/kubernetes-the-hard-way) , with its jumpbox, its server and both workers, and Part 1’s files still in `~/kthw` on the jumpbox. The commands use Part 1’s names and paths. The only new tool is already inside the etcd archive you downloaded in Part 1. | Component | Version | | --- | --- | | Kubernetes | v1.37.1 | | etcd, etcdctl and etcdutl | v3.7.2 | ## What a backup has to contain Everything you create in Kubernetes, every Deployment, Secret, ConfigMap and node, is stored in etcd and nowhere else. The workers run containers, but they get their instructions from the API server, which reads them from etcd. So the heart of a backup is a copy of etcd. That copy is not enough on its own. Some files on the server are not in etcd at all, and the cluster cannot run without them. - **The encryption config** holds the key that encrypts Secrets before they reach etcd. Without it, the Secrets in your backup cannot be read. - **The certificate authority and its key** signed every certificate in the cluster. A new CA means new certificates for every component and every kubeconfig. - **The service account key pair** signs the tokens pods use to talk to the API. A new key makes every existing token invalid. - **The kubeconfigs and unit files** are easy to write again, but copying them is easier still. In Part 1 all of these live in `/var/lib/kubernetes` on the server, with a second copy in `~/kthw` on the jumpbox. A real cluster often has no such second copy, so this part backs them up from the server itself. ## Step 1. Put something worth saving in the cluster Create a namespace with a Secret, a ConfigMap and a small Deployment. You will check each of them after the restore. Note the NodePort of Part 1’s nginx service too, for a test later. jumpbox ``` kubectl create namespace shopkubectl -n shop create secret generic db-password --from-literal=password=correct-horse-battery-staplekubectl -n shop create configmap settings --from-literal=currency=EURkubectl -n shop create deployment web --image=nginx:1.30 --replicas=2kubectl -n shop rollout status deployment/web --timeout=180s mkdir -p ~/kthw/backupskubectl get service nginx-nodeport -o jsonpath='{.spec.ports[0].nodePort}' > ~/kthw/backups/nodeport.txt ``` ## Step 2. Take a snapshot etcd ships three tools. `etcd` is the database, `etcdctl` talks to a running etcd, and `etcdutl` works on etcd’s files directly, which is what a restore needs. Part 1 installed the first two. The third is in the same archive. jumpbox ``` cd ~/kthw/downloadsETCD_VERSION=v3.7.2ARCH=$(dpkg --print-architecture)tar -xzf etcd-${ETCD_VERSION}-linux-${ARCH}.tar.gz --strip-components=1 \ etcd-${ETCD_VERSION}-linux-${ARCH}/etcdutlscp etcdutl root@server:/usr/local/bin/ ``` Now take a snapshot on the server and look at it. server ``` mkdir -p /var/backups/kubernetesetcdctl snapshot save /var/backups/kubernetes/etcd-snapshot.dbetcdutl snapshot status /var/backups/kubernetes/etcd-snapshot.db -w table ``` etcdctl prints a few log lines in JSON first. The rest of the output looked like this. ``` Snapshot saved at /var/backups/kubernetes/etcd-snapshot.dbServer version 3.7.0┌──────────┬──────────┬────────────┬────────────┬─────────┐│ HASH │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ VERSION │├──────────┼──────────┼────────────┼────────────┼─────────┤│ 1fd5d925 │ 596 │ 346 │ 823 kB │ 3.7.0 │└──────────┴──────────┴────────────┴────────────┴─────────┘ ``` `etcdctl snapshot save` asks the running etcd for a consistent copy of its data at one moment, so it is safe while the cluster works. Copying the files under `/var/lib/etcd` with `cp` is not. etcd writes to them constantly, and a copy taken in the middle of a write can be corrupt. The status table tells you what you caught. The revision counts every change etcd has stored so far, and the total keys counts the objects in the snapshot. The version column is the format of the data, which can be older than the etcd binary that wrote it. ## Step 3. Take snapshots on a schedule A snapshot from last month restores last month’s cluster. A small script and a systemd timer take one every hour and keep the newest 24. server ``` cat > /usr/local/bin/etcd-backup <<'EOF'#!/bin/bashset -euo pipefailDIR=/var/backups/kubernetes/hourlymkdir -p "${DIR}"etcdctl snapshot save "${DIR}/etcd-$(date -u +%Y%m%dT%H%M%SZ).db" >/dev/nullls -1t "${DIR}"/etcd-*.db | tail -n +25 | xargs -r rm --EOFchmod 0755 /usr/local/bin/etcd-backup cat > /etc/systemd/system/etcd-backup.service <<'EOF'[Unit]Description=Take an etcd snapshotAfter=etcd.service [Service]Type=oneshotExecStart=/usr/local/bin/etcd-backupEOF cat > /etc/systemd/system/etcd-backup.timer <<'EOF'[Unit]Description=Take an etcd snapshot every hour [Timer]OnCalendar=hourlyPersistent=true [Install]WantedBy=timers.targetEOF systemctl daemon-reloadsystemctl enable --now etcd-backup.timersystemctl start etcd-backup.servicels -l /var/backups/kubernetes/hourlysystemctl list-timers etcd-backup.timer --no-pager ``` ``` total 808-rw------- 1 root root 823328 Oct 25 15:31 etcd-20261025T153147Z.dbNEXT LEFT LAST PASSED UNIT ACTIVATESSun 2026-10-25 16:00:00 UTC 28min - - etcd-backup.timer etcd-backup.service 1 timers listed.Pass --all to see loaded but inactive timers, too. ``` The first snapshot is there, because you started the service by hand. The timer has not fired yet, which is why its last run is empty, and its next run is at the top of the hour. `Persistent=true` makes up for runs the machine missed while it was switched off. The `ls -1t ... | tail -n +25` line deletes all but the 24 newest snapshots, so the disk never fills up. These snapshots protect you from mistakes, such as a deleted namespace. They do not protect you from losing the server, because they live on it. Step 5 copies a backup off the machine, and step 7 shows what happens to everything that stayed behind. ## Step 4. Back up the files etcd does not hold Pack the control plane’s files into one archive next to the snapshot, including the backup job you just wrote. server ``` tar -czf /var/backups/kubernetes/control-plane-files.tar.gz \ /var/lib/kubernetes \ /etc/systemd/system/etcd.service /etc/systemd/system/kube-apiserver.service \ /etc/systemd/system/kube-controller-manager.service /etc/systemd/system/kube-scheduler.service \ /etc/systemd/system/etcd-backup.service /etc/systemd/system/etcd-backup.timer \ /usr/local/bin/etcd-backupchmod 600 /var/backups/kubernetes/*.db /var/backups/kubernetes/*.tar.gztar -tzf /var/backups/kubernetes/control-plane-files.tar.gz | grep -E 'encryption|ca.key|service-account' ``` ``` tar: Removing leading `/' from member namestar: Removing leading `/' from hard link targetsvar/lib/kubernetes/service-account.keyvar/lib/kubernetes/service-account.pubvar/lib/kubernetes/encryption-config.yamlvar/lib/kubernetes/ca.key ``` The two `tar` notes are harmless. They mean the paths inside the archive are relative, so you choose where they land when you unpack it. This archive is the most sensitive file you own. It holds the key that decrypts your Secrets and the key that can sign a certificate for any user, including a cluster administrator. Anyone with both the archive and a snapshot can read every Secret in the cluster. Keep it encrypted, and in production store it apart from the snapshots. ## Step 5. Get the backup off the server A backup on the machine it protects is lost together with that machine. Copy both files to the jumpbox and check the copy of the snapshot. jumpbox ``` cd ~/kthw/backupsscp root@server:/var/backups/kubernetes/etcd-snapshot.db root@server:/var/backups/kubernetes/control-plane-files.tar.gz .chmod 600 etcd-snapshot.db control-plane-files.tar.gz../downloads/etcdutl snapshot status etcd-snapshot.db -w table ``` ``` ┌──────────┬──────────┬────────────┬────────────┬─────────┐│ HASH │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ VERSION │├──────────┼──────────┼────────────┼────────────┼─────────┤│ 1fd5d925 │ 596 │ 346 │ 823 kB │ 3.7.0 │└──────────┴──────────┴────────────┴────────────┴─────────┘ ``` The hash matches the one on the server, so the copy is complete. `etcdutl snapshot status` is the cheapest restore test there is, because a truncated or damaged file fails it. The hourly snapshots from step 3 still sit on the server. In production, a second job copies each one somewhere else, such as object storage in another location, and checks that the newest copy passes `etcdutl snapshot status`. ## Step 6. Change something after the backup Anything you do after a snapshot is not in it. Make two changes now, so you can watch them disappear. jumpbox ``` kubectl -n shop create configmap after-backup --from-literal=note=made-after-the-snapshotkubectl -n shop scale deployment web --replicas=4kubectl -n shop rollout status deployment/web --timeout=180skubectl -n shop get pods -o wide ``` ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESweb-9cf49b6d6-8rhzl 1/1 Running 0 16s 10.200.1.4 node-1 web-9cf49b6d6-hj4fj 1/1 Running 0 1s 10.200.1.5 node-1 web-9cf49b6d6-kxn88 1/1 Running 0 1s 10.200.0.5 node-0 web-9cf49b6d6-lcv5x 1/1 Running 0 16s 10.200.0.4 node-0 ``` Two of these pods existed when you took the snapshot, and two did not. Remember their names. ## Step 7. Lose the control plane Now destroy the control plane. Stop every component and delete everything it stored. That is the etcd data, the certificates and keys, the unit files, the binaries, and the backups that were on the server. What is left is the same as a server whose disk died and was replaced. server ``` systemctl disable --now etcd-backup.timersystemctl stop kube-scheduler kube-controller-manager kube-apiserver etcdrm -rf /var/lib/etcd /var/lib/kubernetes /var/backups/kubernetesrm -f /etc/systemd/system/etcd.service /etc/systemd/system/kube-apiserver.service \ /etc/systemd/system/kube-controller-manager.service /etc/systemd/system/kube-scheduler.service \ /etc/systemd/system/etcd-backup.service /etc/systemd/system/etcd-backup.timerrm -f /usr/local/bin/etcd /usr/local/bin/etcdctl /usr/local/bin/etcdutl /usr/local/bin/etcd-backup \ /usr/local/bin/kube-apiserver /usr/local/bin/kube-controller-manager /usr/local/bin/kube-scheduler \ /usr/local/bin/kubectlsystemctl daemon-reload ``` Look at the cluster from the jumpbox. jumpbox ``` kubectl get nodes --request-timeout=10s || trueNODE_PORT=$(cat ~/kthw/backups/nodeport.txt)curl -s -m 5 http://node-0:${NODE_PORT} | grep -m1 -o 'Welcome to nginx!'curl -s -m 5 http://node-1:${NODE_PORT} | grep -m1 -o 'Welcome to nginx!'ssh root@node-0 'crictl ps --name nginx -q | wc -l'ssh root@node-1 'crictl ps --name nginx -q | wc -l' ``` ``` The connection to the server server:6443 was refused - did you specify the right host or port?Welcome to nginx!Welcome to nginx!33 ``` The control plane is gone, so `kubectl` has nothing to talk to. The applications on the workers keep serving. Each worker still runs three nginx containers, its share of Part 1’s two and of the four `web` pods. A kubelet keeps its containers running on its own, and kube-proxy keeps the rules it already wrote, so the outage takes down nothing that was already running. It does freeze the cluster, because nothing can be scheduled, scaled or repaired until the control plane is back. ## Step 8. Restore, the wrong way first Put the binaries and the two backup files back on the server. jumpbox ``` cd ~/kthwscp downloads/etcd downloads/etcdctl downloads/etcdutl downloads/kube-apiserver \ downloads/kube-controller-manager downloads/kube-scheduler downloads/kubectl \ backups/etcd-snapshot.db backups/control-plane-files.tar.gz \ root@server:~/ ``` server ``` cd ~install -m 0755 etcd etcdctl etcdutl kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/rm etcd etcdctl etcdutl kube-apiserver kube-controller-manager kube-scheduler kubectltar -xzf control-plane-files.tar.gz -C /ls /var/lib/kubernetes ``` ``` apiserver-kubelet-client.crtapiserver-kubelet-client.keyca.crtca.keyencryption-config.yamlkube-apiserver.crtkube-apiserver.keykube-controller-manager.kubeconfigkube-scheduler.kubeconfigkube-scheduler.yamlservice-account.keyservice-account.pub ``` Now the restore itself. `etcdutl snapshot restore` builds a new etcd data directory from the snapshot. It needs the member name, peer address and cluster token the cluster was created with, all of which you can read in `etcd.service`. server ``` etcdutl snapshot restore ~/etcd-snapshot.db \ --name server \ --data-dir /var/lib/etcd \ --initial-cluster server=http://127.0.0.1:2380 \ --initial-advertise-peer-urls http://127.0.0.1:2380 \ --initial-cluster-token kthw \ --bump-revision 1000000000 \ --mark-compactedchmod 700 /var/lib/etcd ``` etcdutl logs every step. Two of its lines show the flags below at work. ``` 2026-10-25T15:33:19Z info snapshot/v3_snapshot.go:397 bumping latest revision {"latest-revision": 596, "bump-amount": 1000000000, "new-latest-revision": 1000000596}2026-10-25T15:33:19Z info snapshot/v3_snapshot.go:414 marking revision compacted {"revision": 1000000596} ``` The last two flags matter for Kubernetes. Every kubelet and kube-proxy remembers the last revision it saw, and that revision is newer than your snapshot, because the cluster kept changing after it. `--bump-revision` moves the restored database’s revision far ahead of anything a client remembers (a billion covers a week-old snapshot on a busy cluster, by the etcd documentation’s own estimate), and `--mark-compacted` tells those clients that the revisions they remember no longer exist. Together they make every component throw away what it cached and read the cluster again from scratch, instead of waiting for changes that will never arrive. Here is the wrong turn. Say the encryption config was not in your backup, so you write a new one with a fresh key, exactly as Part 1 did. server ``` NEW_KEY=$(head -c 32 /dev/urandom | base64)sed -i "s|secret: .*|secret: ${NEW_KEY}|" /var/lib/kubernetes/encryption-config.yaml systemctl daemon-reloadsystemctl enable --now etcdsystemctl enable --now kube-apiserver kube-controller-manager kube-scheduler ``` jumpbox ``` until kubectl -n shop get configmap settings >/dev/null 2>&1; do sleep 2; donekubectl -n shop get configmap settings -o jsonpath='{.data.currency}'; echokubectl -n shop get secret db-password -o jsonpath='{.data.password}' || truekubectl get secrets -A || true # Give the API server a minute and a half to finish starting, then ask what is still not readysleep 90curl -sk https://server:6443/readyz?verbose | grep -v '^\[+\]' ``` ``` EURError from server (InternalError): Internal error occurred: StorageError: corrupt object, Code: 7, Key: /secrets/shop/db-password, ResourceVersion: 0, AdditionalErrorMsg: data from the storage is not transformable revision=0: output array was not large enough for encryptionError from server (StorageReadError): failed to read one or more secrets from the storage: StorageError: corrupt object, Code: 7, Key: /registry/secrets/default/kthw, ResourceVersion: 0, AdditionalErrorMsg: data from the storage is not transformable revision=0: output array was not large enough for encryption (and 1 more)[-]informer-sync failed: reason withheldreadyz check failed ``` The ConfigMap reads fine, because ConfigMaps are not encrypted. Every Secret fails. The API server calls the data `corrupt`, but nothing is damaged. It holds a key that did not encrypt this data, so it cannot turn it back into a Secret. Listing all Secrets fails too, and it names both of the cluster’s Secrets, Part 1’s `kthw` and your `db-password`. The last two lines are the worst part. The API server never becomes ready. Right after a start, a few more checks fail while it finishes booting, which is why the block waits. After that only one is left. It waits until the API server has loaded every Secret, and that can never happen. Anything that waits for a ready API server, such as the load balancer from [Part 2](https://ankra.ai/blog/kubernetes-the-hard-way-high-availability) , sends no traffic to it at all. If the original key is truly lost, so are these Secrets. Every password, token and certificate stored in them has to be created again from its source. That is why the encryption config belongs in the backup. ## Step 9. Fix it with the real key The right key is in the archive from step 4. Put the original encryption config back and restart the API server. server ``` tar -xzf ~/control-plane-files.tar.gz -C / var/lib/kubernetes/encryption-config.yamlsystemctl restart kube-apiserver ``` jumpbox ``` until kubectl get --raw='/readyz' >/dev/null 2>&1; do sleep 2; donekubectl -n shop get secret db-password -o jsonpath='{.data.password}' | base64 -d; echo ``` ``` correct-horse-battery-staple ``` Same snapshot, right key, and the Secret is back. The API server reports ready again. ## Step 10. Check what came back jumpbox ``` kubectl get nodeskubectl -n shop get configmapskubectl -n shop get deployment webkubectl -n shop get pods -o wide ``` ``` NAME STATUS ROLES AGE VERSIONnode-0 Ready 9m35s v1.37.1node-1 Ready 9m34s v1.37.1NAME DATA AGEkube-root-ca.crt 1 8m45ssettings 1 8m44sNAME READY UP-TO-DATE AVAILABLE AGEweb 2/2 2 2 8m44sNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESweb-9cf49b6d6-8rhzl 1/1 Running 0 8m44s 10.200.1.4 node-1 web-9cf49b6d6-lcv5x 1/1 Running 0 8m44s 10.200.0.4 node-0 ``` The cluster is back as it was at the moment of the snapshot. Both nodes are `Ready` without being touched, and the `settings` ConfigMap is there. Everything after the snapshot is gone. The `after-backup` ConfigMap does not exist, and the deployment wants two replicas again, not four. The two pods that are left are the two that existed at snapshot time. The other two were still running on the workers, but the restored cluster had no record of them, so each kubelet stopped its extra pod as soon as the API server answered again. The kubelet log on node-0 shows it. ``` "SyncLoop REMOVE" source="api" pods=["shop/web-9cf49b6d6-kxn88"]"Killing container with a grace period" pod="shop/web-9cf49b6d6-kxn88" ``` That is the price of a backup, and it has a name. The recovery point objective is how much work you accept to lose. With hourly snapshots it is up to an hour. If that is too much, take snapshots more often. Last, turn the hourly snapshots back on. The timer came back with the archive, but it was disabled when the server was wiped. server ``` systemctl daemon-reloadsystemctl enable --now etcd-backup.timersystemctl start etcd-backup.servicels /var/backups/kubernetes/hourly ``` ``` etcd-20261025T154027Z.db ``` After a restore, take a fresh backup before anything else. Your last good one is now the snapshot you restored from, and everything since then exists only in the cluster. ## When something does not work - **Every Secret fails with `corrupt object` and `data from the storage is not transformable`, and the API server never becomes ready.** The encryption config does not hold the key that encrypted the data. Put back the original `encryption-config.yaml` and restart the API server. - **`etcdutl snapshot restore` fails with `data-dir "/var/lib/etcd" not empty or could not be read`.** The restore only writes into a directory that does not exist yet. Stop etcd and move the old directory out of the way first. - **`etcdutl snapshot status` crashes with `panic: assertion failed: Page expected to be...`.** The snapshot file is damaged or incomplete, usually from a copy that did not finish. Copy it again, or take the next older snapshot. - **CoreDNS logs one `forbidden` error right after the API server restarts.** The API server answers before it has loaded all permissions. One error at restart is harmless. If they keep coming, the RBAC objects were not restored. ## Restoring the three-member cluster from Part 2 The three-member etcd from [Part 2](https://ankra.ai/blog/kubernetes-the-hard-way-high-availability) is backed up the same way, with one snapshot from any member, and its etcdctl talks TLS through the variables in `/etc/profile.d/etcdctl.sh`. A restore is different. Every member is restored from the same snapshot file, each with its own `--name` and `--initial-advertise-peer-urls` and with the full three-member `--initial-cluster` list, and then all three are started. The [etcd disaster recovery guide](https://etcd.io/docs/v3.7/op-guide/recovery/) walks through it. Never restore only one member of a running cluster. A restored member is a new cluster and cannot join the old one. ## The easy way, once you know the hard way Everything in this part protects the cluster’s objects. It does not protect what your applications store on their volumes, such as the rows in a database, and that is usually the data people miss most. Ankra covers that layer. You give it a backup vault, an S3-compatible bucket it can also create for you on Hetzner, UpCloud, DigitalOcean or Scaleway, and protect each stack that holds data. Terminal window ``` ankra backup vaults provisionankra cluster stacks protect shop --vault \ --schedule daily --retention daily=7,weekly=4,monthly=6 \ --backup-now --wait ``` Each run seals a restore point, a copy of that stack’s databases and the volumes you name, readable without the cluster it came from. `ankra cluster backups status` shows every stack and whether it is protected, and each restore point lists what it does not carry, next to what it does. You can restore a stack in place, or clone it with its data onto another cluster to rehearse the restore you practised here by hand. The habit from this part still applies. A restore point you have never restored is a hope. Clone a stack onto a test cluster now and then, and look at what comes back. This is Part 3 of [Kubernetes the Hard Way](https://ankra.ai/kubernetes-the-hard-way) , a series of guides we test end to end before publishing. [Part 4](https://ankra.ai/blog/kubernetes-the-hard-way-networking) covers pod networking by hand. --- Get started: [Create a free account on Ankra](https://platform.ankra.app/auth/signup) . Join our community: [**Slack**](https://join.slack.com/t/ankra-community/shared_invite/zt-3a5rem8f8-cUho4epX2MoLT83bFf~VSA) Follow us on: [LinkedIn](https://www.linkedin.com/company/ankra) | [GitHub](https://github.com/ankraio) Contact us: [hello@ankra.ai](mailto:hello@ankra.ai) Share [X](https://x.com/intent/post?url=https%3A%2F%2Fankra.ai%2Fblog%2Fkubernetes-the-hard-way-backup-restore&text=Kubernetes%20the%20Hard%20Way%2C%20Part%203%3A%20Back%20Up%20etcd%20and%20Restore%20a%20Lost%20Control%20Plane) [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fankra.ai%2Fblog%2Fkubernetes-the-hard-way-backup-restore) [HN](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fankra.ai%2Fblog%2Fkubernetes-the-hard-way-backup-restore&t=Kubernetes%20the%20Hard%20Way%2C%20Part%203%3A%20Back%20Up%20etcd%20and%20Restore%20a%20Lost%20Control%20Plane) ### Get the next post in your inbox Written by [Mattias Åsell](https://ankra.ai/blog/author/mattias-asell) Mattias writes about GitOps, security, and platform architecture - from draft-first AI workflows to the engineering behind Ankra's native deployment engine. [All posts](https://ankra.ai/blog/author/mattias-asell) [LinkedIn](https://www.linkedin.com/in/mattiasasell/) ## Related Posts [Kubernetes](https://ankra.ai/blog/tag/kubernetes) [Engineering](https://ankra.ai/blog/tag/engineering) [Homelab](https://ankra.ai/blog/tag/homelab) [## Kubernetes the Hard Way, Part 5: Certificates, Expiry and Renewal Without Downtime](https://ankra.ai/blog/kubernetes-the-hard-way-certificates) Read every Kubernetes certificate, watch one expire, and renew them all by hand. Most components reload a renewed certificate with no restart at all. Tested. [Mattias Åsell](https://ankra.ai/blog/author/mattias-asell) October 25, 2026 9 min read [Kubernetes](https://ankra.ai/blog/tag/kubernetes) [Engineering](https://ankra.ai/blog/tag/engineering) [Homelab](https://ankra.ai/blog/tag/homelab) [## Kubernetes the Hard Way, Part 2: A Highly Available Control Plane](https://ankra.ai/blog/kubernetes-the-hard-way-high-availability) Build a Kubernetes 1.37 control plane that survives losing a server. Three control planes, etcd with TLS, a load balancer, and failure drills, all tested. [Mattias Åsell](https://ankra.ai/blog/author/mattias-asell) October 25, 2026 13 min read [Back to Blog](https://ankra.ai/blog)