A tested, hands-on series
Kubernetes the Hard Way, from the first certificate to GitOps.
Six guides that build Kubernetes 1.37 by hand from the official release binaries, then make it production-shaped one piece at a time. We ran every command, top to bottom, on fresh machines before publishing each part.
Kubernetes the Hard Way means building a cluster yourself, without an installer. You create the certificates, write the config files and start each component by hand, so you understand every piece. The idea comes from Kelsey Hightower's original guide.
This series is our own walkthrough, written for Kubernetes 1.37 on Debian 13. Part 1. Kubernetes 1.37 from scratch builds a working cluster on four machines. The parts after it each take one thing a real cluster needs, build it by hand, and break it on purpose to show what failure looks like.
Six parts, one cluster
Part 1 builds the cluster. Every later part builds on it, and you can take them in any order once Part 1 works.
Part 1. Kubernetes 1.37 from scratch
Certificates, kubeconfigs, etcd, the control plane, containerd, kubelets, pod routes, CoreDNS and a full set of tests, on a jumpbox, a server and two workers. Read Part 1
Part 2. A highly available control plane
Three servers, a three-member etcd with its own CA and mutual TLS, HAProxy in front, and drills that lose one server and then quorum. Read Part 2
Part 3. Back up etcd and restore
Snapshots on a timer, the files etcd does not hold, then a deleted control plane restored with etcdutl, including the encryption key trap. Read Part 3
Part 4. Pod networking by hand
Network namespaces, veth pairs, a bridge, routes and NAT with plain Linux commands, a CNI plugin called by hand, MTU and how a Service reaches a pod. Read Part 4
Part 5. Certificates, expiry and renewal
Every certificate and where it lives, a daily expiry check, two certificates expired on purpose, and renewal that mostly needs no restart. Read Part 5
Part 6. GitOps by hand
A Git repository, a reconciler with its own narrow identity and a 30-second loop with server-side apply and pruning, then drift, rollback and a broken commit. Read Part 6
Every command runs
before the post does.
Each part is written as a sequence of code blocks, each labelled with the machine it runs on. Before a part is published, a script pulls those blocks out of the post and runs them in order on freshly installed Debian 13 servers, stopping at the first command that fails.
The sample outputs in the posts are the ones those runs printed, and the troubleshooting sections list errors we actually hit along the way. When testing showed that a step was wrong or fragile, the step changed before the post went out.
# Part 1, the moment it works
kubectl get nodes
kubectl exec client -- wget -qO- -T 5 http://nginx
# Part 3, after deleting the control plane and restoring it
kubectl -n shop get secret db-password
# Part 6, a change that only goes through Git
git commit -am "Scale web to 3 replicas" && git pushCertificates are the identity system
A certificate's common name is a user and its organisation is a group. Parts 1 and 5 show what that means for logins, expiry and the one group that skips every permission check.
etcd is the cluster
Every object lives in etcd and nowhere else. Part 2 shows why it needs three members, and Part 3 what a snapshot does and does not carry.
A control plane outage freezes, it does not stop
Running pods keep serving while the control plane is gone, in Part 2's lost quorum and in Part 3's deleted control plane. What stops is every change.
A pod is a network namespace
A pod's network is a namespace, a veth cable and a route, and a Service is a destination rewrite in nftables. Part 4 builds and traces both.
Most renewals need no restart
The API server and the kubelets reload a renewed certificate from disk on their own. Only certificates embedded in kubeconfigs need a restart, and it leaves running containers alone.
GitOps is a loop
Fetch, reset, apply with server-side apply, prune what left the repository, repeat. Part 6 builds that loop in about twenty lines.
How to take the series
Rent a few small servers, keep one terminal on the jumpbox, and work through Part 1 before anything else.
- 1
Prepare four machines
A jumpbox, a server and two workers running Debian 13, on one private network. Part 1. Kubernetes 1.37 from scratch lists what each needs. At Hetzner, four small servers cost a few cents an hour.
- 2
Build the cluster in Part 1
Fifteen steps, from the first certificate to a cluster that passes every test. Keep the files in ~/kthw on the jumpbox, because every later part uses them.
mkdir -p ~/kthw - 3
Add high availability in Part 2, or keep one server
Part 2. A highly available control plane builds a separate seven-machine cluster with three control plane servers. Parts 3 to 6 work on the single-server cluster from Part 1.
- 4
Take Parts 3 to 6 in any order
Part 3. Back up etcd and restore a lost control plane, Part 4. Pod networking by hand, Part 5. Certificates, expiry and renewal and Part 6. GitOps by hand each start from a working Part 1 cluster.
- 5
Build it again without the guide
The second time, from memory and with the errors you hit the first time, is when it sticks.
The series at a glance
What each part builds, what it runs on, and what it breaks on purpose.
| What you build | Machines | What you break on purpose | |
|---|---|---|---|
| Part 1 | A complete cluster from the release binaries | Jumpbox, one server, two workers | Nothing, it proves everything works |
| Part 2 | Three control planes, three-member etcd with TLS, a load balancer | Seven, including the load balancer | One server, then a second, then quorum comes back |
| Part 3 | etcd snapshots, a backup of the key files, a restore | Part 1's cluster | The whole control plane, and a restore with the wrong encryption key |
| Part 4 | A pod network by hand, a CNI plugin call, an MTU fix | Part 1's cluster | A packet that is too big for the link |
| Part 5 | A certificate inventory, an expiry check, renewal | Part 1's cluster | An expired client certificate and an expired API server certificate |
| Part 6 | A Git repository and a reconcile loop with pruning | Part 1's cluster | Manual drift and a broken commit |
What the series does not cover yet
The cluster you build is a learning cluster. These are the gaps we know about.
- Upgrading Kubernetes by hand, in the right order, is planned for the Kubernetes 1.38 release, so it can be tested on a real upgrade.
- There is no ingress controller and no persistent storage. Applications are reached through NodePort Services, and pods keep no data.
- The pod network is the basic bridge plugin with static routes. It has no network policies and no encryption between nodes.
- Part 2's load balancer is a single machine, so it is the new single point of failure.
- Everything runs as root, and the admin certificate is in the system:masters group, which no role can restrict.
Where the idea comes from
Kubernetes The Hard Way by Kelsey Hightower has taught this lesson to a generation of engineers. This series is not a copy of it. It follows the same journey with its own steps, for Kubernetes 1.37 and Debian 13, and it goes further, into high availability, backups, networking, certificates and GitOps. Part 1. Kubernetes 1.37 from scratch lists what it does differently.
Each part ends with how Ankra handles the same job on clusters it builds or manages. The point of the series is that you know what sits underneath when you use a platform, so you can tell what it is telling you.
Guides and comparisons
Questions teams actually ask
What is Kubernetes the Hard Way?+
Building a Kubernetes cluster by hand, without an installer. You create the certificates, write the configuration and start each component yourself, so you understand how the pieces fit. Kelsey Hightower's original guide made the idea famous, and this series is a tested walkthrough of it for Kubernetes 1.37 and Debian 13.
Is Kubernetes the Hard Way still worth doing in 2026?+
Yes, if you operate or debug Kubernetes. Managed services and installers hide the same components you build here, and when something breaks they are what you end up reading logs from. It is not the way to run production clusters.
How long does each part take?+
Part 1 is the longest, typically an afternoon if you read along. The later parts are shorter and can each be done in a sitting, because they build on the cluster you already have.
What do I need to follow the series?+
Four Debian 13 machines on one private network for Part 1. A small jumpbox, plus a server and two workers with at least 2 vCPU and 2 GB of memory each. Part 2 needs seven. Cloud servers, local virtual machines and old laptops all work.
Which versions does the series use?+
Kubernetes v1.37.1 on Debian 13, with etcd 3.7, containerd 2.4, runc 1.5, the CNI plugins 1.9 and CoreDNS 1.14.6. Each part lists the exact versions it was tested with.
Do I need Ankra to follow it?+
No. Everything runs on your own machines with open source tools. Each part ends with how Ankra handles the same job, and Part 1 shows how to connect the cluster you built to Ankra if you want to see it through a platform.
Know the hard way, then use the easy one
Ankra builds and operates upstream Kubernetes on the cloud you choose, with high availability, backups into a vault you choose and a GitOps repository you own. Every organisation gets a free Trial Playground, and the free plan includes 30 vCPU of worker capacity.
Kubernetes is a registered trademark of The Linux Foundation. Kubernetes The Hard Way is a guide by Kelsey Hightower, who is not affiliated with Ankra. Every part was tested end to end on Debian 13 in October 2026.