# Kubernetes the Hard Way in 2026: Kubernetes 1.37 From Scratch | Ankra Canonical: https://ankra.ai/blog/kubernetes-the-hard-way Kubernetes the Hard Way, updated for Kubernetes 1.37 and Debian 13. Every command tested end to end, on local machines and on Hetzner Cloud. [Back to Blog](https://ankra.ai/blog) [Kubernetes](https://ankra.ai/blog/tag/kubernetes) [Engineering](https://ankra.ai/blog/tag/engineering) [Homelab](https://ankra.ai/blog/tag/homelab) # Kubernetes the Hard Way in 2026: Kubernetes 1.37 From Scratch [Mattias Åsell](https://ankra.ai/blog/author/mattias-asell) October 23, 2026 30 min read ### On this page Most people meet Kubernetes through a managed service or an installer. You run one command, wait a few minutes, and a cluster appears. That is great for getting work done. It is not great for learning, because everything interesting happened somewhere you could not see. This guide does the opposite. You will build a small cluster by hand from the official release binaries. You will create every certificate, write every config file and start every component yourself. When something breaks later in your career, you will know which piece to look at, because you will have installed each one. The idea is not ours. Kelsey Hightower’s [Kubernetes The Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way) has taught this lesson to a generation of engineers. This is our own walkthrough of the same journey, written for Kubernetes 1.37 and Debian 13. We ran every command in this post, top to bottom, on fresh machines before publishing it. If you have done the original, [what this guide does differently](https://ankra.ai/blog/kubernetes-the-hard-way#what-this-guide-does-differently) lists what changed and why. Set aside two to three hours. Copy and paste is fine, but read the text between the commands. That is where the learning is. ## What you will build Four Linux machines, each with one job. | Machine | Role | What runs there | | --- | --- | --- | | jumpbox | Your workbench | openssl, kubectl, all downloads and generated files | | server | Control plane | etcd, kube-apiserver, kube-controller-manager, kube-scheduler | | node-0 | Worker | containerd, kubelet, kube-proxy, your pods | | node-1 | Worker | containerd, kubelet, kube-proxy, your pods | The jumpbox is where you prepare everything. The other three only receive finished files and run them. Keeping the two apart makes it obvious what each machine actually needs. This is how the finished cluster fits together. Come back to it whenever a step mentions a certificate or a connection. Everything talks to the API server, and every solid arrow is a TLS connection where each side proves who it is with its own certificate. The API server also calls back into each kubelet for logs and exec. The one plain-text link is etcd, which listens only on the server’s loopback address. These are the versions we tested with. | Component | Version | | --- | --- | | Kubernetes | v1.37.1 | | etcd | v3.7.2 | | containerd | v2.4.2 | | runc | v1.5.2 | | CNI plugins | v1.9.1 | | crictl | v1.37.0 | | CoreDNS | v1.14.6 | CoreDNS and the CNI plugins are the exact versions the Kubernetes 1.37 release pins, and etcd is a newer patch release of the 3.7 line it pins. containerd, runc and crictl are the current releases of each project. ## Before you start You need four machines running Debian 13 (trixie). Cloud servers, local virtual machines or old laptops all work. A few rules matter. - All four must reach each other on a private network. - All four must use the same CPU architecture, either amd64 or arm64. The commands below detect it. - The jumpbox can be small (1 vCPU, 512 MB of memory). Give the server and both workers at least 2 vCPU and 2 GB of memory each, and 20 GB of disk. - On Hetzner Cloud, put all four servers on one private network. Step 10 shows the two extra settings that network needs. - The machine network must not overlap with the two ranges this cluster uses inside itself. Pods get addresses from `10.200.0.0/16` and services from `10.32.0.0/24`. - You can log in to the jumpbox as root, and root on the jumpbox can SSH into the other three machines as root with a key. If the last point is not set up yet, create a key on the jumpbox with `ssh-keygen -t ed25519`, then add the contents of `/root/.ssh/id_ed25519.pub` to `/root/.ssh/authorized_keys` on the server and both workers. Debian’s SSH server allows root to log in with a key by default, though some cloud images add their own default user and expect you to use sudo. In that case, run `sudo -i` first and set up root’s key the same way. Renting the machines costs very little. At Hetzner, for example, four CX23 servers (2 vCPU, 4 GB) with public IPv4 addresses cost about €0.04 an hour together at October 2026 list prices, before VAT. A three-hour session comes to around 12 cents. Delete them when you are done. Every command in this guide runs as root. Each code block is labelled with the machine it runs on. ## Step 1. Prepare the jumpbox Install the few tools you need and create a working directory. jumpbox ``` apt-get updateapt-get install -y curl openssl openssh-clientmkdir -p ~/kthw ``` Now describe your machines in one small file. Each line holds the private IP address, the hostname and, for workers, the pod subnet that machine will hand out. Replace the example addresses with your own. jumpbox ``` cd ~/kthwcat > machines.txt <<'EOF'10.240.0.10 server10.240.0.20 node-0 10.200.0.0/2410.240.0.21 node-1 10.200.1.0/24EOF ``` Every later step reads this file, so you never have to type an IP address again. Next, give each machine its hostname and teach all four machines each other’s names. Kubernetes identifies nodes by hostname, and the certificates you create later contain these names, so they have to match exactly. jumpbox ``` cd ~/kthwawk '{print $1, $2}' machines.txt > hostscat hosts >> /etc/hosts while read IP HOST SUBNET; do ssh -n -o StrictHostKeyChecking=accept-new root@${IP} "hostnamectl set-hostname ${HOST}" scp hosts root@${IP}:/tmp/hosts ssh -n root@${IP} "cat /tmp/hosts >> /etc/hosts && rm /tmp/hosts"done < machines.txt for HOST in server node-0 node-1; do ssh -n -o StrictHostKeyChecking=accept-new root@${HOST} hostnamedone ``` The `-n` flag matters. Without it, `ssh` reads the rest of `machines.txt` as its own input and the loop quietly stops after the first machine. The last loop should print `server`, `node-0` and `node-1`. If it does, name resolution and SSH both work. ## Step 2. Download the binaries Kubernetes is a handful of programs. You download each one once on the jumpbox, check it against the checksum its project publishes, and copy it out later. jumpbox ``` cd ~/kthwK8S_VERSION=v1.37.1ETCD_VERSION=v3.7.2CONTAINERD_VERSION=2.4.2RUNC_VERSION=v1.5.2CNI_VERSION=v1.9.1CRICTL_VERSION=v1.37.0ARCH=$(dpkg --print-architecture)GH=https://github.com mkdir -p downloads && cd downloads for BIN in kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy kubectl; do curl -fsSLO "https://dl.k8s.io/${K8S_VERSION}/bin/linux/${ARCH}/${BIN}" echo "$(curl -fsSL "https://dl.k8s.io/${K8S_VERSION}/bin/linux/${ARCH}/${BIN}.sha256") ${BIN}" | sha256sum --checkdone ETCD_TGZ=etcd-${ETCD_VERSION}-linux-${ARCH}.tar.gzcurl -fsSLO "${GH}/etcd-io/etcd/releases/download/${ETCD_VERSION}/${ETCD_TGZ}"curl -fsSL "${GH}/etcd-io/etcd/releases/download/${ETCD_VERSION}/SHA256SUMS" | grep " ${ETCD_TGZ}$" | sha256sum --check CONTAINERD_TGZ=containerd-${CONTAINERD_VERSION}-linux-${ARCH}.tar.gzcurl -fsSLO "${GH}/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/${CONTAINERD_TGZ}"curl -fsSL "${GH}/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/${CONTAINERD_TGZ}.sha256sum" | sha256sum --checkcurl -fsSL -o containerd.service "https://raw.githubusercontent.com/containerd/containerd/v${CONTAINERD_VERSION}/containerd.service" curl -fsSLO "${GH}/opencontainers/runc/releases/download/${RUNC_VERSION}/runc.${ARCH}"curl -fsSL "${GH}/opencontainers/runc/releases/download/${RUNC_VERSION}/runc.sha256sum" | grep " runc.${ARCH}$" | sha256sum --checkmv runc.${ARCH} runc CNI_TGZ=cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgzcurl -fsSLO "${GH}/containernetworking/plugins/releases/download/${CNI_VERSION}/${CNI_TGZ}"curl -fsSL "${GH}/containernetworking/plugins/releases/download/${CNI_VERSION}/${CNI_TGZ}.sha256" | sha256sum --check CRICTL_TGZ=crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gzcurl -fsSLO "${GH}/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/${CRICTL_TGZ}"echo "$(curl -fsSL "${GH}/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/${CRICTL_TGZ}.sha256") ${CRICTL_TGZ}" | sha256sum --check tar -xzf ${ETCD_TGZ} --strip-components=1 etcd-${ETCD_VERSION}-linux-${ARCH}/etcd etcd-${ETCD_VERSION}-linux-${ARCH}/etcdctlchmod +x kube-* kubelet kubectl runcinstall -m 0755 kubectl /usr/local/bin/kubectl ``` Every check should print `OK`. If one prints `FAILED`, delete that file and download it again. Never install a binary that failed its checksum. Notice what you just downloaded. Five Kubernetes programs, a database, a container runtime, a low-level container launcher, some network plugins and the command-line tools that talk to them. That is the whole system. Everything else in Kubernetes runs as pods on top of these. ## Step 3. Create the certificates Almost every connection inside Kubernetes uses TLS (the one exception in this guide is etcd, which only listens on the server’s loopback address), and almost every component proves who it is with a client certificate. The API server reads the certificate’s common name as the user name and the organisation field as the group. So the certificates are not only encryption. They are the identity system. You start with a certificate authority (CA), a key pair that signs everything else. Then you create one certificate per identity. This whole block runs in a single shell session, because it defines a small helper function first. jumpbox ``` cd ~/kthwmkdir -p pki && cd pki # The certificate authority that signs everything elseopenssl genpkey -quiet -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out ca.keyopenssl req -x509 -new -key ca.key -sha256 -days 3650 \ -subj "/CN=kubernetes-ca" \ -addext "basicConstraints=critical,CA:TRUE" \ -addext "keyUsage=critical,keyCertSign,cRLSign" \ -out ca.crt # make_cert NAME SUBJECT USAGE [SUBJECT_ALT_NAMES]make_cert() { local name=$1 subject=$2 usage=$3 sans=$4 openssl genpkey -quiet -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out "${name}.key" openssl req -new -key "${name}.key" -subj "${subject}" -out "${name}.csr" { echo "basicConstraints=critical,CA:FALSE" echo "keyUsage=critical,digitalSignature,keyEncipherment" echo "extendedKeyUsage=${usage}" if [ -n "${sans}" ]; then echo "subjectAltName=${sans}"; fi } > "${name}.ext" openssl x509 -req -in "${name}.csr" -CA ca.crt -CAkey ca.key \ -days 365 -sha256 -extfile "${name}.ext" -out "${name}.crt" rm "${name}.csr" "${name}.ext"} SERVER_IP=$(awk '$2 == "server" {print $1}' ../machines.txt) # Clients of the API servermake_cert admin "/CN=admin/O=system:masters" clientAuthmake_cert kube-controller-manager "/CN=system:kube-controller-manager" clientAuthmake_cert kube-scheduler "/CN=system:kube-scheduler" clientAuthmake_cert kube-proxy "/CN=system:kube-proxy" clientAuth # The API server's own certificate, and the one it uses to call kubeletsmake_cert kube-apiserver "/CN=kube-apiserver" serverAuth \ "IP:10.32.0.1,IP:127.0.0.1,IP:${SERVER_IP},DNS:server,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster.local"make_cert apiserver-kubelet-client "/CN=kube-apiserver-kubelet-client" clientAuth # One certificate per worker, used both as a client and as the kubelet's serverwhile read IP HOST SUBNET; do if [ "${HOST}" = "server" ]; then continue; fi make_cert "${HOST}" "/CN=system:node:${HOST}/O=system:nodes" serverAuth,clientAuth "DNS:${HOST},IP:${IP}"done < ../machines.txt # The key pair that signs and verifies service account tokensopenssl genpkey -quiet -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out service-account.keyopenssl pkey -in service-account.key -pubout -out service-account.pub ls ``` The names in those subjects are not decoration. Each one maps to a permission that already exists in every Kubernetes cluster. - `system:masters` is a group with full access to everything. It is how `admin` becomes a cluster administrator. - `system:kube-controller-manager`, `system:kube-scheduler` and `system:kube-proxy` are users that Kubernetes ships ready-made roles for. - `system:node:node-0` in the group `system:nodes` is how a kubelet identifies itself. The Node authorizer uses that exact pattern to limit each kubelet to its own node and to the Secrets and ConfigMaps of the pods scheduled on it. The API server certificate lists every name and address a client might use to reach it. `10.32.0.1` is the first address in the service range, which becomes the `kubernetes` service that pods use to talk to the API. If a name is missing from this list, clients that use it get a TLS error. Take a look at one of the worker certificates to see the identity inside it. jumpbox ``` cd ~/kthw/pkiopenssl x509 -in node-0.crt -noout -subject -ext subjectAltName ``` ``` subject=CN=system:node:node-0, O=system:nodesX509v3 Subject Alternative Name: DNS:node-0, IP Address:10.240.0.20 ``` One more thing about `system:masters`. That group skips every permission check, so you cannot take its access away by changing roles. The only ways out are waiting for the certificate to expire or replacing the CA. That is fine for a lab. Real clusters keep this certificate locked away and give people narrower access. ## Step 4. Create the kubeconfig files A kubeconfig file bundles three things. Where the API server is, which CA to trust, and which certificate to log in with. Each component gets its own. jumpbox ``` cd ~/kthwmkdir -p configs && cd configs # make_kubeconfig NAME USER SERVER_URLmake_kubeconfig() { local name=$1 user=$2 server=$3 kubectl config set-cluster kthw --certificate-authority=../pki/ca.crt \ --embed-certs=true --server="${server}" --kubeconfig="${name}.kubeconfig" kubectl config set-credentials "${user}" --client-certificate="../pki/${name}.crt" \ --client-key="../pki/${name}.key" --embed-certs=true --kubeconfig="${name}.kubeconfig" kubectl config set-context default --cluster=kthw --user="${user}" --kubeconfig="${name}.kubeconfig" kubectl config use-context default --kubeconfig="${name}.kubeconfig"} make_kubeconfig node-0 system:node:node-0 https://server:6443make_kubeconfig node-1 system:node:node-1 https://server:6443make_kubeconfig kube-proxy system:kube-proxy https://server:6443make_kubeconfig kube-controller-manager system:kube-controller-manager https://127.0.0.1:6443make_kubeconfig kube-scheduler system:kube-scheduler https://127.0.0.1:6443make_kubeconfig admin admin https://127.0.0.1:6443 ``` The workers reach the API server by its hostname. The controller manager and scheduler live on the server itself, so they use the loopback address. The user name inside a kubeconfig is only a local label. The identity the API server sees always comes from the certificate. ## Step 5. Encrypt secrets at rest By default, Kubernetes stores Secrets in etcd without any encryption. Anyone who copies the etcd data directory can read every password in the cluster. An encryption config tells the API server to encrypt Secrets before it writes them. jumpbox ``` cd ~/kthw/configsENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64) cat > encryption-config.yaml < etcd.service <<'EOF'[Unit]Description=etcdDocumentation=https://etcd.io/docs/ [Service]Type=notifyExecStart=/usr/local/bin/etcd \ --name=server \ --data-dir=/var/lib/etcd \ --listen-client-urls=http://127.0.0.1:2379 \ --advertise-client-urls=http://127.0.0.1:2379 \ --listen-peer-urls=http://127.0.0.1:2380 \ --initial-advertise-peer-urls=http://127.0.0.1:2380 \ --initial-cluster=server=http://127.0.0.1:2380 \ --initial-cluster-state=new \ --initial-cluster-token=kthwRestart=on-failureRestartSec=5 [Install]WantedBy=multi-user.targetEOF ``` This etcd listens only on the loopback address, so nothing outside the server can reach it, and that is why plain HTTP is acceptable here. A production control plane runs three or five etcd members across machines, and then every connection between them, and from the API server, needs TLS. The API server is the front door. Every other component, and you, talk to the cluster only through it. jumpbox ``` cd ~/kthw/control-planeSERVER_IP=$(awk '$2 == "server" {print $1}' ../machines.txt) cat > kube-apiserver.service <<'EOF'[Unit]Description=Kubernetes API ServerDocumentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/After=etcd.serviceWants=etcd.service [Service]ExecStart=/usr/local/bin/kube-apiserver \ --advertise-address=SERVER_IP \ --allow-privileged=true \ --authorization-mode=Node,RBAC \ --client-ca-file=/var/lib/kubernetes/ca.crt \ --enable-admission-plugins=NodeRestriction \ --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \ --etcd-servers=http://127.0.0.1:2379 \ --kubelet-certificate-authority=/var/lib/kubernetes/ca.crt \ --kubelet-client-certificate=/var/lib/kubernetes/apiserver-kubelet-client.crt \ --kubelet-client-key=/var/lib/kubernetes/apiserver-kubelet-client.key \ --service-account-issuer=https://server:6443 \ --service-account-key-file=/var/lib/kubernetes/service-account.pub \ --service-account-signing-key-file=/var/lib/kubernetes/service-account.key \ --service-cluster-ip-range=10.32.0.0/24 \ --tls-cert-file=/var/lib/kubernetes/kube-apiserver.crt \ --tls-private-key-file=/var/lib/kubernetes/kube-apiserver.key \ --v=2Restart=on-failureRestartSec=5 [Install]WantedBy=multi-user.targetEOF sed -i "s/SERVER_IP/${SERVER_IP}/" kube-apiserver.service ``` The heredoc is quoted so the shell leaves the line-ending backslashes alone, and `sed` fills in the one value that differs per cluster. A few flags are worth understanding. - `--authorization-mode=Node,RBAC` turns on the two permission systems. Node limits what each kubelet can read to what its own node and pods need. RBAC handles everyone else. - `--enable-admission-plugins=NodeRestriction` limits each kubelet to changing its own node and its own pods, so a stolen kubelet credential cannot be used to rewrite other nodes or their pods. - `--client-ca-file` is what makes the client certificates from step 3 work as logins. - The `--kubelet-*` flags let the API server open verified connections to kubelets. That is how `kubectl logs` and `kubectl exec` reach a container. The controller manager runs the control loops. It notices that a Deployment wants three pods and only two exist, and creates the third. jumpbox ``` cd ~/kthw/control-plane cat > kube-controller-manager.service <<'EOF'[Unit]Description=Kubernetes Controller ManagerDocumentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/After=kube-apiserver.service [Service]ExecStart=/usr/local/bin/kube-controller-manager \ --bind-address=127.0.0.1 \ --cluster-name=kubernetes \ --cluster-signing-cert-file=/var/lib/kubernetes/ca.crt \ --cluster-signing-key-file=/var/lib/kubernetes/ca.key \ --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \ --leader-elect=true \ --root-ca-file=/var/lib/kubernetes/ca.crt \ --service-account-private-key-file=/var/lib/kubernetes/service-account.key \ --service-cluster-ip-range=10.32.0.0/24 \ --use-service-account-credentials=true \ --v=2Restart=on-failureRestartSec=5 [Install]WantedBy=multi-user.targetEOF ``` `--use-service-account-credentials=true` gives each control loop its own identity instead of sharing one powerful one. When you read audit logs later, you can see which loop did what. The scheduler decides which node each new pod runs on. It reads its settings from a small config file. jumpbox ``` cd ~/kthw/control-plane cat > kube-scheduler.yaml <<'EOF'apiVersion: kubescheduler.config.k8s.io/v1kind: KubeSchedulerConfigurationclientConnection: kubeconfig: /var/lib/kubernetes/kube-scheduler.kubeconfigleaderElection: leaderElect: trueEOF cat > kube-scheduler.service <<'EOF'[Unit]Description=Kubernetes SchedulerDocumentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/After=kube-apiserver.service [Service]ExecStart=/usr/local/bin/kube-scheduler \ --config=/var/lib/kubernetes/kube-scheduler.yaml \ --v=2Restart=on-failureRestartSec=5 [Install]WantedBy=multi-user.targetEOF ``` Leader election matters once you run several control plane machines. Only one scheduler and one controller manager act at a time, and the others wait to take over. With one server it simply always wins. ## Step 7. Start the control plane Copy the binaries, certificates and configs to the server. jumpbox ``` cd ~/kthwscp downloads/etcd downloads/etcdctl downloads/kube-apiserver \ downloads/kube-controller-manager downloads/kube-scheduler downloads/kubectl \ pki/ca.crt pki/ca.key pki/kube-apiserver.crt pki/kube-apiserver.key \ pki/apiserver-kubelet-client.crt pki/apiserver-kubelet-client.key \ pki/service-account.key pki/service-account.pub \ configs/kube-controller-manager.kubeconfig configs/kube-scheduler.kubeconfig \ configs/admin.kubeconfig configs/encryption-config.yaml \ control-plane/* \ root@server:~/ ``` Now log in to the server and put each file where its unit expects it. server ``` cd ~install -m 0755 etcd etcdctl kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/rm etcd etcdctl kube-apiserver kube-controller-manager kube-scheduler kubectl mkdir -p /var/lib/kubernetes /var/lib/etcdchmod 700 /var/lib/etcdmv ca.crt ca.key kube-apiserver.crt kube-apiserver.key \ apiserver-kubelet-client.crt apiserver-kubelet-client.key \ service-account.key service-account.pub \ kube-controller-manager.kubeconfig kube-scheduler.kubeconfig \ encryption-config.yaml kube-scheduler.yaml \ /var/lib/kubernetes/chmod 600 /var/lib/kubernetes/*.key /var/lib/kubernetes/*.kubeconfig /var/lib/kubernetes/encryption-config.yamlmv etcd.service kube-apiserver.service kube-controller-manager.service kube-scheduler.service /etc/systemd/system/ systemctl daemon-reloadsystemctl enable --now etcdetcdctl endpoint health ``` ``` 127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.2ms ``` etcd is running. Start the rest of the control plane and wait until the API server reports ready. server ``` systemctl enable --now kube-apiserver kube-controller-manager kube-scheduler until kubectl --kubeconfig ~/admin.kubeconfig get --raw='/readyz' >/dev/null 2>&1; do sleep 2; donekubectl --kubeconfig ~/admin.kubeconfig get --raw='/readyz?verbose' | tail -n 3 ``` ``` [+]poststarthook/apiservice-openapiv3-controller ok[+]shutdown okreadyz check passed ``` The controller manager and scheduler each hold a leader-election lease, a small object the active instance renews every few seconds. It can take up to half a minute after startup before both appear. server ``` kubectl --kubeconfig ~/admin.kubeconfig -n kube-system get leases ``` ``` NAME HOLDER AGEapiserver-qiczmurnxxy4duoopog2qwr2ru apiserver-qiczmurnxxy4duoopog2qwr2ru_519c726f-6242-4354-b963-784da5a40654 30skube-controller-manager server_c2503a89-09b7-4144-812a-243655820899 24skube-scheduler server_d6b322cf-bd29-4d1e-861b-9d2ade038d2b 26s ``` If a component is missing, `journalctl -u kube-scheduler` (or the matching unit name) shows why. Do not worry about a burst of “forbidden” errors in the first seconds of the scheduler’s log. The API server creates its built-in roles just after it starts, and the errors stop once it has. ## Step 8. Connect kubectl from the jumpbox From here on you drive the cluster from the jumpbox, like you would from your own laptop. jumpbox ``` cd ~/kthwkubectl config set-cluster kthw --certificate-authority=pki/ca.crt \ --embed-certs=true --server=https://server:6443kubectl config set-credentials admin --client-certificate=pki/admin.crt \ --client-key=pki/admin.key --embed-certs=truekubectl config set-context kthw --cluster=kthw --user=adminkubectl config use-context kthw kubectl version ``` ``` Client Version: v1.37.1Kustomize Version: v5.8.1Server Version: v1.37.1 ``` There is one permission left to grant. The API server calls each kubelet as the user `kube-apiserver-kubelet-client`, and by default that user may do nothing. Kubernetes ships a role for exactly this job, so you only bind it. jumpbox ``` kubectl create clusterrolebinding kube-apiserver-kubelet-client \ --clusterrole=system:kubelet-api-admin \ --user=kube-apiserver-kubelet-client ``` Without this binding, `kubectl logs` and `kubectl exec` fail with a forbidden error later on, even though everything else works. ## Step 9. Prepare the worker configs A worker needs four things. A container runtime, a network for its pods, the kubelet and kube-proxy. You write their configs on the jumpbox first. containerd is the runtime. It pulls images and asks runc to start containers. jumpbox ``` mkdir -p ~/kthw/workers && cd ~/kthw/workers cat > containerd-config.toml <<'EOF'version = 4 [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc] runtime_type = 'io.containerd.runc.v2' [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options] SystemdCgroup = true [plugins.'io.containerd.cri.v1.runtime'.cni] bin_dirs = ['/opt/cni/bin'] conf_dir = '/etc/cni/net.d'EOF cat > crictl.yaml <<'EOF'runtime-endpoint: unix:///run/containerd/containerd.sockEOF ``` `SystemdCgroup = true` tells containerd to let systemd manage the resource limits of containers. On a machine where systemd is the init system, that is what you want. Otherwise the machine ends up with two managers and two views of what is using its resources, and the node can become unstable under pressure. containerd’s own default is `false`, so this line matters. Since Kubernetes 1.34 the kubelet asks containerd which driver it uses, so this is the setting that counts. The kubelet’s own `cgroupDriver` below is only a fallback for older runtimes. The kubelet is the agent on every worker. It watches the API server for pods assigned to its node and makes them run. jumpbox ``` cd ~/kthw/workers cat > kubelet-config.yaml <<'EOF'apiVersion: kubelet.config.k8s.io/v1beta1kind: KubeletConfigurationauthentication: anonymous: enabled: false webhook: enabled: true x509: clientCAFile: /var/lib/kubelet/ca.crtauthorization: mode: WebhookcgroupDriver: systemdclusterDomain: cluster.localclusterDNS: - 10.32.0.10containerRuntimeEndpoint: unix:///run/containerd/containerd.sockresolvConf: /etc/resolv.conftlsCertFile: /var/lib/kubelet/kubelet.crttlsPrivateKeyFile: /var/lib/kubelet/kubelet.keyEOF cat > kubelet.service <<'EOF'[Unit]Description=Kubernetes KubeletDocumentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/After=containerd.serviceRequires=containerd.service [Service]ExecStart=/usr/local/bin/kubelet \ --config=/var/lib/kubelet/kubelet-config.yaml \ --kubeconfig=/var/lib/kubelet/kubeconfig \ --node-ip=NODE_IP \ --v=2Restart=on-failureRestartSec=5 [Install]WantedBy=multi-user.targetEOF ``` The `authentication` and `authorization` blocks lock the kubelet down. Anonymous requests are refused, and every request is checked with the API server first. Those are already the defaults when the kubelet reads a config file, but writing them out makes the security model visible. The kubelet’s old command-line flags default to the opposite, with anonymous access on and every request allowed. `clientCAFile` is what lets the API server’s client certificate in, which `kubectl logs` and `kubectl exec` depend on. `--node-ip` pins the address the kubelet publishes for its node. Without it, the kubelet works the address out from the hostname and the default route, and on a machine with both a public and a private network interface it can pick the public one. kube-proxy turns Services into packet rules. When a pod connects to a service address, those rules send the packet to one of the pods behind it. jumpbox ``` cd ~/kthw/workers cat > kube-proxy-config.yaml <<'EOF'apiVersion: kubeproxy.config.k8s.io/v1alpha1kind: KubeProxyConfigurationclientConnection: kubeconfig: /var/lib/kube-proxy/kubeconfigmode: nftablesclusterCIDR: 10.200.0.0/16EOF cat > kube-proxy.service <<'EOF'[Unit]Description=Kubernetes Kube ProxyDocumentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/Wants=network-online.targetAfter=network-online.target [Service]ExecStart=/usr/local/bin/kube-proxy \ --config=/var/lib/kube-proxy/kube-proxy-config.yamlRestart=on-failureRestartSec=5 [Install]WantedBy=multi-user.targetEOF ``` We use the `nftables` mode. It is the modern replacement for the older iptables mode and matches the firewall framework Debian 13 uses. ## Step 10. Plan the pod network This is the part most installers hide completely, and the part people find hardest when it breaks. Kubernetes networking rests on one core rule. Every pod gets its own IP address, and every pod can reach every other pod at that address without any address translation in between. Kubernetes itself does not implement this. It hands the job to a CNI plugin. Real clusters use plugins like Cilium or Calico. Here you will do it by hand with the basic `bridge` plugin, so you can see what those plugins automate. [Part 4 of this series](https://ankra.ai/blog/kubernetes-the-hard-way-networking) takes that plugin apart and rebuilds its work with plain Linux commands. Each worker owns one slice of the pod range from `machines.txt`. The bridge plugin creates a virtual switch called `cni0` on the worker and gives each pod an address from that worker’s slice. jumpbox ``` cd ~/kthw/workers cat > 10-bridge.conflist <<'EOF'{ "cniVersion": "1.0.0", "name": "bridge", "plugins": [ { "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": false, "hairpinMode": true, "ipam": { "type": "host-local", "ranges": [[{ "subnet": "POD_SUBNET" }]], "routes": [{ "dst": "0.0.0.0/0" }] } }, { "type": "portmap", "capabilities": { "portMappings": true } } ]}EOF ``` That covers pods on the same worker. Pods on different workers need two more things. First, each machine needs a route to the other workers’ pod slices. A packet for `10.200.1.5` has to go to node-1, because only node-1 knows that pod. Second, pods need to reach the internet, for example to resolve outside DNS names. Pod addresses are private, so traffic leaving the cluster must be translated to the worker’s own address. Traffic between pods must not be translated, or the receiving pod sees the worker’s address instead of the real sender. That is why `ipMasq` is off in the bridge config above. Its built-in rule would also translate traffic going to pods on other workers. Instead, you add one precise rule. The solid path is a packet from a pod on node-0 to a pod on node-1. A plain route carries it across, and it arrives with the sender’s real pod address. Only traffic leaving the cluster takes the dashed path through the pod-nat rule, where the source is swapped for node-0’s address so replies can find their way back. jumpbox ``` cd ~/kthw/workers cat > pod-nat.nft <<'EOF'# Translate pod traffic that leaves the cluster, and nothing else.table ip pod-natdelete table ip pod-nattable ip pod-nat { chain postrouting { type nat hook postrouting priority srcnat; policy accept; ip saddr 10.200.0.0/16 ip daddr != 10.200.0.0/16 masquerade }}EOF ``` The first two lines make the file safe to load twice. They create the table if it is missing and then delete it, so the definition below always starts from scratch. Routes added by hand disappear on reboot, so you wrap the routes and the rule in a small systemd service per machine. One question decides what those routes look like. Can your machines reach each other directly? Run `ip -br -4 addr` on any of them and look at the private address. If it ends in something like `/24`, the machines share one network segment, and each one can send pod traffic straight to the worker that owns it. That is typical for local virtual machines. If it ends in `/32`, as on Hetzner Cloud, the network is routed. Every packet goes through the network’s gateway, so the network itself has to learn where each pod slice lives. ### On Hetzner Cloud and other routed networks On Hetzner Cloud you add those routes to the private network, from wherever you use the `hcloud` command-line tool. Use your network’s name. Other routed clouds have the same idea under another name, such as routes in a Google Cloud VPC. Skip this block if your machines share one segment. hcloud CLI ``` NETWORK=kthwhcloud network add-route ${NETWORK} --destination 10.200.0.0/24 --gateway 10.240.0.20hcloud network add-route ${NETWORK} --destination 10.200.1.0/24 --gateway 10.240.0.21 ``` Now generate the services. The loop reads `machines.txt` and writes one route per other worker. On a routed network, set `POD_GATEWAY` to the gateway that `ip route` shows for your private range (`10.240.0.1` on Hetzner Cloud in this example), and every machine sends the whole pod range there instead. jumpbox ``` cd ~/kthw/workers # Empty when the machines share one network segment.# On a routed network such as Hetzner Cloud, the private network's gateway.POD_GATEWAY="" while read IP HOST SUBNET; do { cat <<'EOF'[Unit]Description=Pod network routes and NATWants=network-online.targetAfter=network-online.target [Service]Type=oneshotRemainAfterExit=yesRestart=on-failureRestartSec=5EOF if [ -n "${POD_GATEWAY}" ]; then echo "ExecStart=/usr/sbin/ip route replace 10.200.0.0/16 via ${POD_GATEWAY}" else while read PEER_IP PEER_HOST PEER_SUBNET; do if [ -n "${PEER_SUBNET}" ] && [ "${PEER_HOST}" != "${HOST}" ]; then echo "ExecStart=/usr/sbin/ip route replace ${PEER_SUBNET} via ${PEER_IP}" fi done < ../machines.txt fi if [ -n "${SUBNET}" ]; then echo "ExecStart=/usr/sbin/nft -f /etc/kubernetes/pod-nat.nft" fi cat <<'EOF' [Install]WantedBy=multi-user.targetEOF } > pod-network-${HOST}.servicedone < ../machines.txt cat pod-network-node-0.service ``` ``` [Unit]Description=Pod network routes and NATWants=network-online.targetAfter=network-online.target [Service]Type=oneshotRemainAfterExit=yesRestart=on-failureRestartSec=5ExecStart=/usr/sbin/ip route replace 10.200.1.0/24 via 10.240.0.21ExecStart=/usr/sbin/nft -f /etc/kubernetes/pod-nat.nft [Install]WantedBy=multi-user.target ``` `Restart=on-failure` matters more than it looks. On some clouds the private network comes up a moment after the rest of the system, and a route to a gateway that is not reachable yet is refused. Without the retry, a reboot leaves pods on different workers unable to reach each other or the API server. We hit exactly that on Hetzner Cloud. Every line in the service is safe to run twice, so retrying is harmless. With `POD_GATEWAY` set, the route line reads `ExecStart=/usr/sbin/ip route replace 10.200.0.0/16 via 10.240.0.1` instead. A worker’s own slice still stays local, because the more specific route to its `cni0` bridge wins. The server gets routes too but no translation rule, because no pods run there. Either way, these routes only work because every machine sits on one private network. Across networks or clouds, this is exactly the problem an overlay plugin like Cilium or Flannel solves. ## Step 11. Copy everything to the workers Fill in the per-worker values and copy each worker its files. jumpbox ``` cd ~/kthw while read IP HOST SUBNET; do if [ "${HOST}" = "server" ]; then continue; fi mkdir -p workers/${HOST} sed "s|POD_SUBNET|${SUBNET}|" workers/10-bridge.conflist > workers/${HOST}/10-bridge.conflist sed "s|NODE_IP|${IP}|" workers/kubelet.service > workers/${HOST}/kubelet.service scp downloads/kubelet downloads/kube-proxy downloads/runc \ downloads/containerd-*.tar.gz downloads/cni-plugins-*.tgz downloads/crictl-*.tar.gz \ downloads/containerd.service \ pki/ca.crt pki/${HOST}.crt pki/${HOST}.key \ configs/${HOST}.kubeconfig configs/kube-proxy.kubeconfig \ workers/containerd-config.toml workers/crictl.yaml workers/kubelet-config.yaml \ workers/kube-proxy-config.yaml workers/kube-proxy.service workers/pod-nat.nft \ workers/${HOST}/10-bridge.conflist workers/${HOST}/kubelet.service \ workers/pod-network-${HOST}.service \ root@${HOST}:~/done < machines.txt scp workers/pod-network-server.service root@server:~/ ``` Turn on the server’s routes now. server ``` mv ~/pod-network-server.service /etc/systemd/system/pod-network.servicesystemctl daemon-reloadsystemctl enable --now pod-networkip route | grep 10.200 ``` ``` 10.200.0.0/24 via 10.240.0.20 dev eth010.200.1.0/24 via 10.240.0.21 dev eth0 ``` Your interface name may differ from `eth0`, for example `ens3` or `enp7s0`. With `POD_GATEWAY` set, you see a single line instead, `10.200.0.0/16 via 10.240.0.1`. ## Step 12. Bring up the workers Run the next three blocks on node-0, then on node-1. They are identical on both. First the operating system settings. The kubelet refuses to start while swap is on, containerd stores image layers with the `overlay` filesystem module, and pod networking needs the `br_netfilter` module and packet forwarding. node-0 and node-1 ``` apt-get updateapt-get install -y conntrack nftables swapoff -ased -i '/\sswap\s/ s/^/#/' /etc/fstab cat > /etc/modules-load.d/kubernetes.conf <<'EOF'overlaybr_netfilterEOFmodprobe overlaymodprobe br_netfilter cat > /etc/sysctl.d/99-kubernetes.conf <<'EOF'net.ipv4.ip_forward = 1net.bridge.bridge-nf-call-iptables = 1net.bridge.bridge-nf-call-ip6tables = 1EOFsysctl --system ``` `br_netfilter` is subtle. Pods on the same worker talk through the `cni0` bridge, and by default bridged traffic skips the host’s packet rules entirely. The module and the two `bridge-nf-call` settings make it pass through them, which kube-proxy needs when a pod talks to a service backed by a pod on the same worker. Next the container runtime and the pod network. node-0 and node-1 ``` cd ~tar -xzf containerd-*.tar.gz -C /usr/localinstall -m 0755 runc /usr/local/sbin/runcmkdir -p /opt/cni/bin /etc/cni/net.d /etc/containerd /etc/kubernetestar -xzf cni-plugins-*.tgz -C /opt/cni/bintar -xzf crictl-*.tar.gz -C /usr/local/binrm containerd-*.tar.gz cni-plugins-*.tgz crictl-*.tar.gz runc mv containerd-config.toml /etc/containerd/config.tomlmv crictl.yaml /etc/crictl.yamlmv 10-bridge.conflist /etc/cni/net.d/mv pod-nat.nft /etc/kubernetes/pod-nat.nftmv containerd.service /etc/systemd/system/containerd.servicemv pod-network-$(hostname).service /etc/systemd/system/pod-network.service systemctl daemon-reloadsystemctl enable --now containerd pod-networkcrictl info >/dev/null && echo "containerd is ready" ``` Finally the kubelet and kube-proxy. node-0 and node-1 ``` cd ~install -m 0755 kubelet kube-proxy /usr/local/bin/rm kubelet kube-proxymkdir -p /var/lib/kubelet /var/lib/kube-proxy mv ca.crt /var/lib/kubelet/ca.crtmv $(hostname).crt /var/lib/kubelet/kubelet.crtmv $(hostname).key /var/lib/kubelet/kubelet.keymv $(hostname).kubeconfig /var/lib/kubelet/kubeconfigmv kubelet-config.yaml /var/lib/kubelet/kubelet-config.yamlmv kube-proxy.kubeconfig /var/lib/kube-proxy/kubeconfigmv kube-proxy-config.yaml /var/lib/kube-proxy/kube-proxy-config.yamlchmod 600 /var/lib/kubelet/kubelet.key /var/lib/kubelet/kubeconfig /var/lib/kube-proxy/kubeconfig # When systemd-resolved runs, /etc/resolv.conf points at a local stub that pods cannot useif systemctl is-active --quiet systemd-resolved; then sed -i 's|^resolvConf: .*|resolvConf: /run/systemd/resolve/resolv.conf|' /var/lib/kubelet/kubelet-config.yamlfi mv kubelet.service kube-proxy.service /etc/systemd/system/systemctl daemon-reloadsystemctl enable --now kubelet kube-proxy ``` The `resolv.conf` check guards against a classic failure. When systemd-resolved is running, `/etc/resolv.conf` sends queries to `127.0.0.53`. Inside a pod that address is the pod itself, so CoreDNS would forward queries to itself, detect the loop and crash. Pointing the kubelet at the real upstream servers avoids it. kubeadm makes exactly the same check. Only test whether the service is running, not whether its file exists. A leftover file can vanish at the next reboot, and then no pod on that worker can start. Back on the jumpbox, wait for both workers to register and report ready. jumpbox ``` kubectl wait --for=create node/node-0 node/node-1 --timeout=120skubectl wait --for=condition=Ready node/node-0 node/node-1 --timeout=180skubectl get nodes ``` ``` NAME STATUS ROLES AGE VERSIONnode-0 Ready 30s v1.37.1node-1 Ready 30s v1.37.1 ``` The first wait is needed because a node object only exists once its kubelet has registered it, and `kubectl wait` cannot wait for the condition of an object that does not exist yet. Add `-o wide` to the last command to see each node’s address, operating system and container runtime. Notice that the server is not in this list. It runs no kubelet, so it is not a node. Its control plane programs run as plain systemd services, outside Kubernetes. ## Step 13. Add cluster DNS Pods find services by name, like `nginx.default.svc.cluster.local`. CoreDNS answers those questions. It is the first thing you install the normal way, as pods, using the cluster you just built. The Kubernetes repository ships a CoreDNS manifest template. You fill in the cluster domain, the DNS service address you gave the kubelets and a memory limit. jumpbox ``` cd ~/kthwcurl -fsSL https://raw.githubusercontent.com/kubernetes/kubernetes/v1.37.1/cluster/addons/dns/coredns/coredns.yaml.base \ | sed -e 's/__DNS__DOMAIN__/cluster.local/g' \ -e 's/__DNS__SERVER__/10.32.0.10/g' \ -e 's/__DNS__MEMORY__LIMIT__/170Mi/g' \ > coredns.yaml kubectl apply -f coredns.yamlkubectl -n kube-system rollout status deployment/coredns --timeout=180skubectl -n kube-system get pods -o wide ``` ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATEScoredns-xxxxxxxxxx-xxxxx 1/1 Running 0 20s 10.200.1.2 node-1 ``` That pod is the first proof your cluster works end to end. The scheduler picked a node, the kubelet asked containerd to pull the image and start it, the bridge plugin gave it an address, and CoreDNS used its service account token to read services from the API server through the `kubernetes` service. ## Step 14. Prove that it works Run each test from the jumpbox. Each one exercises a different part of what you built. **Secrets are encrypted in etcd.** Create a Secret, then read it straight from the database on the server. jumpbox ``` kubectl create secret generic kthw --from-literal=mykey=mydatassh root@server "etcdctl get /registry/secrets/default/kthw --print-value-only | head -c 26"; echo ``` ``` k8s:enc:secretbox:v1:key1: ``` The stored value starts with the provider and key name, followed by encrypted bytes. Your password is not in there in readable form. **Deployments schedule onto both workers.** jumpbox ``` kubectl create deployment nginx --image=nginx:1.30 --replicas=2kubectl rollout status deployment/nginx --timeout=180skubectl get pods -l app=nginx -o wide ``` **Pods reach pods on other workers, without translation.** Start a small client pod and call each nginx pod by its address. jumpbox ``` kubectl run client --image=busybox:1.37 --restart=Never -- sleep 3600kubectl wait --for=condition=Ready pod/client --timeout=120s for POD_IP in $(kubectl get pods -l app=nginx -o jsonpath='{.items[*].status.podIP}'); do kubectl exec client -- wget -qO- -T 5 http://${POD_IP} | grep -m1 -o 'Welcome to nginx!'done kubectl get pod client -o jsonpath='{.status.podIP}{"\n"}'kubectl logs -l app=nginx --tail=1 ``` The nginx access logs show the client pod’s own address as the caller. That is the networking rule from step 10, working. **Services and DNS work.** jumpbox ``` kubectl expose deployment nginx --port=80kubectl exec client -- nslookup nginx.default.svc.cluster.localkubectl exec client -- wget -qO- -T 5 http://nginx | grep -m1 -o 'Welcome to nginx!'kubectl exec client -- nslookup kubernetes.io ``` The first lookup is answered by CoreDNS from cluster data. The short name `nginx` works because the kubelet gave the pod a search list. The last lookup goes out to the internet through the translation rule. **Port forwarding, logs and exec work.** All three travel from kubectl to the API server and on to the kubelet. jumpbox ``` kubectl port-forward deployment/nginx 8080:80 >/dev/null &PF_PID=$!sleep 3curl -s http://127.0.0.1:8080 | grep -m1 -o 'Welcome to nginx!'kill ${PF_PID} kubectl logs deployment/nginx --tail=2kubectl exec deployment/nginx -- nginx -v ``` **NodePort services are reachable from outside the cluster.** jumpbox ``` kubectl expose deployment nginx --name=nginx-nodeport --type=NodePort --port=80NODE_PORT=$(kubectl get service nginx-nodeport -o jsonpath='{.spec.ports[0].nodePort}')curl -s --retry 5 --retry-connrefused http://node-0:${NODE_PORT} | grep -m1 -o 'Welcome to nginx!'curl -s --retry 5 --retry-connrefused http://node-1:${NODE_PORT} | grep -m1 -o 'Welcome to nginx!' ``` kube-proxy needs a moment to write the rules for a new service, which is why `curl` retries if the first attempt is refused. Both workers answer, even one that runs no nginx pod at all, because kube-proxy on every worker knows where the pods are. If all of these printed `Welcome to nginx!` and real addresses, you have built a working Kubernetes cluster from scratch. ## Step 15. Connect your cluster to Ankra Your cluster works, and you know every piece of it. Now see the same cluster through a platform. Ankra connects to any Kubernetes cluster, including one built by hand, and trying it is free. Ankra installs a small agent with Helm. Install Helm on the jumpbox first, checked against its published checksum like everything else. jumpbox ``` cd ~/kthw/downloadsHELM_VERSION=v4.3.1ARCH=$(dpkg --print-architecture)curl -fsSLO "https://get.helm.sh/helm-${HELM_VERSION}-linux-${ARCH}.tar.gz"curl -fsSL "https://get.helm.sh/helm-${HELM_VERSION}-linux-${ARCH}.tar.gz.sha256sum" | sha256sum --checktar -xzf helm-${HELM_VERSION}-linux-${ARCH}.tar.gz --strip-components=1 linux-${ARCH}/helminstall -m 0755 helm /usr/local/bin/helmhelm version --short ``` Then [create a free Ankra account](https://platform.ankra.app/auth/signup) , open **Clusters**, click **Import**, give the cluster a name and click **Import cluster**. Ankra shows a command that carries a token for your cluster. Copy it and run it on the jumpbox. It looks like this. Terminal window ``` helm upgrade --install ankra-agent oci://registry.ankra.cloud/ankra/ankra-agent \ --version \ --set config.ankra_url=https://platform.ankra.app \ --set config.token= \ --namespace=ankra --create-namespace ``` The agent only makes outbound connections to Ankra, so nothing on your cluster has to be opened to the internet. Wait for it to start. jumpbox ``` kubectl -n ankra rollout status deployment/ankra-agent --timeout=180s ``` Within seconds the cluster shows as online in Ankra, with your two nodes, every pod and their logs, all read through the API server you set up in step 7. Ankra also gives every connected cluster its own hostname under `ankra.cc` and installs external-dns to publish it, so do not be surprised by a new `external-dns` pod. You can ask Ankra’s AI about the cluster too. We asked how pod networking and DNS work on ours, and it recognised the hard-way design straight away. It explained that each node gets its own pod range, routed directly with no overlay plugin such as Calico or Flannel, and that pods resolve names through CoreDNS at `10.32.0.10`. ## When something does not work Most problems show up in a component’s logs. On the machine that runs it, `journalctl -u -e` shows the latest lines. A few failures are common enough to name. - **A node never becomes Ready.** Check `journalctl -u kubelet` on that worker. A certificate whose name does not match the hostname is the usual cause. - **Pods stay in `ContainerCreating`.** Run `kubectl describe pod` and read the events. Network setup errors point at `/etc/cni/net.d` or `/opt/cni/bin`. - **Cross-node traffic times out.** Run `ip route` on each worker and check that the routes from step 10 exist. Some cloud providers also drop traffic for addresses they did not assign to the machine. AWS calls the setting the source and destination check, and Google Cloud calls it IP forwarding. - **`journalctl -u pod-network` shows `Nexthop has invalid gateway`.** Your private network is routed, not shared. Set `POD_GATEWAY` and add the network routes as shown in step 10. - **`kubectl logs` fails with `Forbidden (user=kube-apiserver-kubelet-client, ...)`.** The role binding from step 8 is missing. Kubelets remember a refusal for 30 seconds, so give a fixed binding half a minute before trying again. - **CoreDNS keeps restarting.** Its logs usually mention a loop. Recheck the `resolvConf` step on the workers. ## What this guide does differently The original guide is still the best introduction to the idea, and its last update was in April 2025. Here is what we changed and why. - **Current, stable releases.** Kubernetes 1.37 with stable etcd and containerd releases. The original targets Kubernetes 1.32 with release candidates of etcd and containerd. - **Every download is checked.** Each binary is verified against the checksum its project publishes before anything runs it. - **No translation between pods.** The original turns on the bridge plugin’s `ipMasq`, which also rewrites traffic between pods on different workers. We translate only traffic leaving the cluster, so pods see each other’s real addresses. - **The network survives a reboot.** Pod routes and the translation rule live in a systemd service instead of one-off `ip route add` commands. We rebooted the whole cluster to prove it. - **kube-proxy in nftables mode** instead of iptables. - **`secretbox` for Secrets** instead of `aescbc`, which the Kubernetes documentation marks as not recommended. - **Cluster DNS.** CoreDNS is installed and tested, so pods find services by name. - **A dedicated identity for the API server’s calls to kubelets**, bound to the role Kubernetes ships for that job, instead of a hand-written one. ## What this cluster does not have You now understand the moving parts better than most people who run Kubernetes every day. It is also worth being honest about what a production cluster needs on top of this. - **No high availability.** One server and one etcd member means one machine failure takes the control plane down. Production runs three control plane machines behind a load balancer. [Part 2 of this series](https://ankra.ai/blog/kubernetes-the-hard-way-high-availability) builds exactly that and then breaks it on purpose. - **No backups.** Nothing here snapshots etcd. Lose the server’s disk and the cluster’s state is gone. [Part 3 of this series](https://ankra.ai/blog/kubernetes-the-hard-way-backup-restore) adds snapshots and restores the cluster after deleting its control plane. - **Certificates expire in a year.** Every certificate except the CA must be renewed and redistributed before then, without downtime. [Part 5 of this series](https://ankra.ai/blog/kubernetes-the-hard-way-certificates) does exactly that. - **Upgrades are manual.** Kubernetes ships three minor releases a year, and each is supported for about fourteen months. Every component must be upgraded in the right order, every few months, [forever](https://ankra.ai/blog/kubernetes-upgrade-treadmill) . - **No network policies, ingress, storage or monitoring.** Each is another component to choose, install, configure and keep up to date. None of this is out of reach. You could add each piece by hand, the same way you built the rest. That is a good way to learn, and a hard way to run a business. ## The easy way, once you know the hard way Step 15 showed Ankra looking at a cluster you built. It can also build the cluster. On Hetzner, one command covers what you did by hand in steps 1 to 14, plus the production pieces this guide left out. Terminal window ``` ankra cluster hetzner create --name my-cluster --location hel1 \ --credential-id \ --control-plane-count 3 --worker-count 2 ``` That builds upstream Kubernetes with kubeadm and Cilium (or k3s, if you prefer), with three control plane machines for high availability. Traefik and cert-manager for ingress and certificates come with it, and so do Hetzner’s drivers for load balancers and volumes. The same works on OVHcloud, UpCloud, DigitalOcean, Scaleway, AWS and your own Proxmox hardware. From there the day-2 work in the list above becomes routine. A Kubernetes version upgrade is a platform operation, not a weekend of SSH sessions. Add-ons like ingress, certificates, monitoring and databases are [pinned, versioned parts of a stack](https://ankra.ai/blog/delivery-simplified-with-stacks) , stored in your own Git repository. Secrets stay [encrypted in Git](https://ankra.ai/blog/gitops-secrets-sops-age) . And when something breaks, Ankra’s AI reads the same logs, events and metrics you learned to read in this guide, and tells you what it found. The knowledge from today still pays off. When Ankra flags a node that is not ready, or a pod stuck in `ContainerCreating`, you know which pieces sit underneath and where to look. If you want a cluster to keep practising on without four machines, every Ankra organisation gets a free [Trial Playground](https://ankra.ai/pricing#playground) , and the free plan includes 30 vCPU of worker capacity for real clusters. When you are done with your hard-way cluster, disconnect it from Ankra if you connected it (`helm uninstall ankra-agent -n ankra`, then `kubectl delete namespace ankra`), and delete the four machines. Then build it again in a week without looking at this guide. That second time is when it really sticks. This is Part 1 of [Kubernetes the Hard Way](https://ankra.ai/kubernetes-the-hard-way) , a series of guides we test end to end before publishing. [Part 2](https://ankra.ai/blog/kubernetes-the-hard-way-high-availability) covers a highly available control plane. --- Get started: [Create a free account on Ankra](https://platform.ankra.app/auth/signup) . Join our community: [**Slack**](https://join.slack.com/t/ankra-community/shared_invite/zt-3a5rem8f8-cUho4epX2MoLT83bFf~VSA) Follow us on: [LinkedIn](https://www.linkedin.com/company/ankra) | [GitHub](https://github.com/ankraio) Contact us: [hello@ankra.ai](mailto:hello@ankra.ai) Share [X](https://x.com/intent/post?url=https%3A%2F%2Fankra.ai%2Fblog%2Fkubernetes-the-hard-way&text=Kubernetes%20the%20Hard%20Way%20in%202026%3A%20Kubernetes%201.37%20From%20Scratch) [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fankra.ai%2Fblog%2Fkubernetes-the-hard-way) [HN](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fankra.ai%2Fblog%2Fkubernetes-the-hard-way&t=Kubernetes%20the%20Hard%20Way%20in%202026%3A%20Kubernetes%201.37%20From%20Scratch) ### Get the next post in your inbox Written by [Mattias Åsell](https://ankra.ai/blog/author/mattias-asell) Mattias writes about GitOps, security, and platform architecture - from draft-first AI workflows to the engineering behind Ankra's native deployment engine. [All posts](https://ankra.ai/blog/author/mattias-asell) [LinkedIn](https://www.linkedin.com/in/mattiasasell/) ## Related Posts [Kubernetes](https://ankra.ai/blog/tag/kubernetes) [Engineering](https://ankra.ai/blog/tag/engineering) [Homelab](https://ankra.ai/blog/tag/homelab) [## Kubernetes the Hard Way, Part 3: Back Up etcd and Restore a Lost Control Plane](https://ankra.ai/blog/kubernetes-the-hard-way-backup-restore) Back up a Kubernetes 1.37 cluster by hand with etcd snapshots, then delete the control plane and restore it. Includes the encryption key trap, all tested. [Mattias Åsell](https://ankra.ai/blog/author/mattias-asell) October 25, 2026 12 min read [Kubernetes](https://ankra.ai/blog/tag/kubernetes) [Engineering](https://ankra.ai/blog/tag/engineering) [Homelab](https://ankra.ai/blog/tag/homelab) [## Kubernetes the Hard Way, Part 5: Certificates, Expiry and Renewal Without Downtime](https://ankra.ai/blog/kubernetes-the-hard-way-certificates) Read every Kubernetes certificate, watch one expire, and renew them all by hand. Most components reload a renewed certificate with no restart at all. Tested. [Mattias Åsell](https://ankra.ai/blog/author/mattias-asell) October 25, 2026 9 min read [Back to Blog](https://ankra.ai/blog)