Kubernetes the Hard Way, Part 5: Certificates, Expiry and Renewal Without Downtime

October 25, 2026

9 min read

On this page

The cluster you built in Part 1 has a date on it. Its certificates are valid for one year, and on the day they run out the cluster stops answering. Nothing warns you. Every connection that starts after that moment fails, and the error messages do not always say why.

This part makes that date visible and moves it. You will read every certificate and find out where each one lives, write a check you can run every day, let a certificate expire on purpose to see exactly what breaks, and then renew all of them. Most of the cluster picks up a renewed certificate on its own, without a restart, and you will see which parts do and which do not.

We ran every command in this post, top to bottom, on fresh machines before publishing it, and the outputs on this page are the ones we got.

What you need

The cluster from Part 1, with its certificates still in ~/kthw/pki on the jumpbox. The commands use two newer OpenSSL options, -copy_extensions and -not_after, which the OpenSSL 3.5 in Debian 13 supports.

Component	Version
Kubernetes	v1.37.1
OpenSSL	3.5 (Debian 13 package)

Step 1. Every certificate, and when it ends

Part 1 made one certificate authority and eight certificates signed by it. List them all with their subject and end date.

cd ~/kthw/pki
for CRT in *.crt; do
  printf '%-30s %-45s %s\n' "${CRT}" \
    "$(openssl x509 -in ${CRT} -noout -subject | sed 's/^subject=//')" \
    "$(openssl x509 -in ${CRT} -noout -enddate | sed 's/^notAfter=//')"
done

admin.crt                      CN=admin, O=system:masters                    Oct 25 17:24:06 2027 GMT
apiserver-kubelet-client.crt   CN=kube-apiserver-kubelet-client              Oct 25 17:24:06 2027 GMT
ca.crt                         CN=kubernetes-ca                              Oct 22 17:24:05 2036 GMT
kube-apiserver.crt             CN=kube-apiserver                             Oct 25 17:24:06 2027 GMT
kube-controller-manager.crt    CN=system:kube-controller-manager             Oct 25 17:24:06 2027 GMT
kube-proxy.crt                 CN=system:kube-proxy                          Oct 25 17:24:06 2027 GMT
kube-scheduler.crt             CN=system:kube-scheduler                      Oct 25 17:24:06 2027 GMT
node-0.crt                     CN=system:node:node-0, O=system:nodes         Oct 25 17:24:06 2027 GMT
node-1.crt                     CN=system:node:node-1, O=system:nodes         Oct 25 17:24:07 2027 GMT

Two dates matter. The CA runs for ten years, and everything it signed runs for one. All eight end within a second of each other, because Part 1 created them in one go. That is convenient for renewal and dangerous for an outage, because the whole cluster expires at once.

Step 2. Where each certificate lives

A certificate on the jumpbox does nothing. What counts is the copy each component reads, and Part 1 gave them out in two different ways.

As files. The API server reads its serving certificate, its client certificate for the kubelets and the CA from files in /var/lib/kubernetes. Each kubelet reads its serving certificate from /var/lib/kubelet/kubelet.crt.
Inside kubeconfig files. The controller manager, the scheduler, kube-proxy, each kubelet’s login to the API server, and your own kubectl carry their certificate embedded in their kubeconfig, base64 encoded, because Part 1 built them with --embed-certs=true.

That difference decides how renewal works, as you will see in step 6. First, check what the servers really present. A TLS handshake shows the certificate a server is using right now, whatever is on disk.

for TARGET in server:6443 node-0:10250 node-1:10250; do
  echo "${TARGET}"
  echo | openssl s_client -connect ${TARGET} 2>/dev/null | openssl x509 -noout -subject -enddate
done

server:6443
subject=CN=kube-apiserver
notAfter=Oct 25 17:24:06 2027 GMT
node-0:10250
subject=CN=system:node:node-0, O=system:nodes
notAfter=Oct 25 17:24:06 2027 GMT
node-1:10250
subject=CN=system:node:node-1, O=system:nodes
notAfter=Oct 25 17:24:07 2027 GMT

Port 6443 is the API server and port 10250 is each kubelet’s own API, the one kubectl logs and kubectl exec go through. Notice that a kubelet uses its node certificate for both jobs, to log in to the API server and to serve its own port.

Step 3. A check you can run every day

Write one script that reads every certificate a machine uses, from files and from kubeconfigs alike, and prints the days left. It exits with an error when any certificate has fewer than 30 days left, so a cron job or a monitoring agent can act on it.

cd ~/kthw
cat > cert-check <<'EOF'
#!/bin/bash
# Days left for every certificate Kubernetes uses on this machine.
# Exits 1 when any of them has fewer than WARN_DAYS left.
WARN_DAYS=${WARN_DAYS:-30}
NOW=$(date +%s)
STATUS=0

report() {
  local end days
  end=$(openssl x509 -noout -enddate | sed 's/^notAfter=//')
  days=$(( ($(date -d "${end}" +%s) - NOW) / 86400 ))
  printf '%5s days  %-24s  %s\n' "${days}" "${end}" "$1"
  if [ "${days}" -lt "${WARN_DAYS}" ]; then STATUS=1; fi
}

echo "$(hostname)"

for CRT in /var/lib/kubernetes/*.crt /var/lib/kubelet/*.crt; do
  [ -f "${CRT}" ] && report "${CRT}" < "${CRT}"
done
for KC in /var/lib/kubernetes/*.kubeconfig /var/lib/kubelet/kubeconfig \
          /var/lib/kube-proxy/kubeconfig /root/admin.kubeconfig /root/.kube/config; do
  [ -f "${KC}" ] || continue
  # A kubeconfig can hold several users, or none with a certificate
  while read -r DATA; do
    echo "${DATA}" | base64 -d | report "${KC}"
  done < <(grep 'client-certificate-data' "${KC}" | awk '{print $2}')
done
exit ${STATUS}
EOF
chmod 0755 cert-check

for HOST in server node-0 node-1; do
  scp -q cert-check root@${HOST}:/usr/local/bin/cert-check
done
./cert-check
ssh root@server cert-check
ssh root@node-0 cert-check

kthws-jumpbox
  364 days  Oct 25 17:24:06 2027 GMT  /root/.kube/config
server
  364 days  Oct 25 17:24:06 2027 GMT  /var/lib/kubernetes/apiserver-kubelet-client.crt
 3649 days  Oct 22 17:24:05 2036 GMT  /var/lib/kubernetes/ca.crt
  364 days  Oct 25 17:24:06 2027 GMT  /var/lib/kubernetes/kube-apiserver.crt
  364 days  Oct 25 17:24:06 2027 GMT  /var/lib/kubernetes/kube-controller-manager.kubeconfig
  364 days  Oct 25 17:24:06 2027 GMT  /var/lib/kubernetes/kube-scheduler.kubeconfig
  364 days  Oct 25 17:24:06 2027 GMT  /root/admin.kubeconfig
node-0
 3649 days  Oct 22 17:24:05 2036 GMT  /var/lib/kubelet/ca.crt
  364 days  Oct 25 17:24:06 2027 GMT  /var/lib/kubelet/kubelet.crt
  364 days  Oct 25 17:24:06 2027 GMT  /var/lib/kubelet/kubeconfig
  364 days  Oct 25 17:24:06 2027 GMT  /var/lib/kube-proxy/kubeconfig

The first line is the jumpbox, whatever your cloud named it. Every certificate the cluster depends on is now on one list, including the ones hidden inside kubeconfigs. Run cert-check from cron or your monitoring every day, and the expiry date stops being a surprise.

Step 4. Let a certificate expire

Make two certificates that have already expired, with the same subject, names and key as the real ones. -copy_extensions copyall carries the names and usages over from the original, and -not_before and -not_after set dates in the past.

cd ~/kthw/pki

# expire_copy NAME OUT: the same certificate, but valid only in October 2025
expire_copy() {
  openssl x509 -x509toreq -in $1.crt -signkey $1.key -copy_extensions copyall -out /tmp/$1.csr
  openssl x509 -req -in /tmp/$1.csr -CA ca.crt -CAkey ca.key -copy_extensions copyall \
    -not_before 20251001000000Z -not_after 20251024000000Z -sha256 -out $2
  rm /tmp/$1.csr
}

expire_copy admin /tmp/admin-expired.crt
expire_copy kube-apiserver /tmp/kube-apiserver-expired.crt
openssl x509 -in /tmp/kube-apiserver-expired.crt -noout -subject -dates

subject=CN=admin, O=system:masters
subject=CN=kube-apiserver
subject=CN=kube-apiserver
notBefore=Oct  1 00:00:00 2025 GMT
notAfter=Oct 24 00:00:00 2025 GMT

First an expired client. Build a kubeconfig around the expired admin certificate and use it.

cd ~/kthw/pki
kubectl config view --raw > /tmp/expired-admin.kubeconfig
kubectl --kubeconfig /tmp/expired-admin.kubeconfig config set-credentials admin \
  --client-certificate=/tmp/admin-expired.crt --client-key=admin.key --embed-certs=true
kubectl --kubeconfig /tmp/expired-admin.kubeconfig get nodes || true

User "admin" set.
error: You must be logged in to the server (Unauthorized)

That is all kubectl tells you. The API server knows why, and keeps the reason in its log.

journalctl -u kube-apiserver --no-pager -o cat --since -2min \
  | grep -m1 'Unable to authenticate' | grep -o 'failed: x509.*'

failed: x509: certificate has expired or is not yet valid: current time 2026-10-25T17:26:03Z is after 2025-10-24T00:00:00Z"

Now the other side. Put the expired certificate in place of the API server’s own and try again with your normal, valid credentials.

scp /tmp/kube-apiserver-expired.crt root@server:/var/lib/kubernetes/kube-apiserver.crt
sleep 10
kubectl get nodes || true

Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-10-25T17:26:15Z is after 2025-10-24T00:00:00Z

This time kubectl refuses to talk at all, because it cannot trust the server. It also says so clearly. Remember the two messages, because on the day a real cluster expires you will see both. Unauthorized from a client whose own certificate ran out, and failed to verify certificate from everything that calls a server whose certificate ran out.

Notice also what you did not do. You copied a file, and ten seconds later the API server was using it, without a restart. That is the key to the next step.

Step 5. Renew without a restart

Renewing a certificate means signing the same request again with a new end date. You keep the key, the subject and every name, so nothing that trusts the CA can tell the difference, apart from the dates and the serial number. This small script does it for any certificate from Part 1. It is a file rather than a shell function, so it still works in the next SSH session.

cat > ~/kthw/renew-cert <<'EOF'
#!/bin/bash
# renew-cert NAME, run in ~/kthw/pki.
# A new certificate for the same key, subject and extensions, valid for a year.
set -euo pipefail
openssl x509 -x509toreq -in "$1.crt" -signkey "$1.key" -copy_extensions copyall -out "$1.csr"
openssl x509 -req -in "$1.csr" -CA ca.crt -CAkey ca.key -copy_extensions copyall \
  -days 365 -sha256 -out "$1.crt"
rm "$1.csr"
EOF
chmod 0755 ~/kthw/renew-cert

cd ~/kthw/pki
openssl x509 -in kube-apiserver.crt -noout -serial
../renew-cert kube-apiserver
openssl x509 -in kube-apiserver.crt -noout -serial -enddate
scp kube-apiserver.crt root@server:/var/lib/kubernetes/kube-apiserver.crt

sleep 10
kubectl get nodes
echo | openssl s_client -connect server:6443 2>/dev/null | openssl x509 -noout -serial

serial=32F65042B19DAD85A5FE44D43D60C578B2C46ABE
subject=CN=kube-apiserver
serial=1BEBC6B8BD0B83E322171CA2A65F6CA63CA73042
notAfter=Oct 25 17:26:16 2027 GMT
NAME     STATUS   ROLES    AGE   VERSION
node-0   Ready    <none>   64s   v1.37.1
node-1   Ready    <none>   63s   v1.37.1
serial=1BEBC6B8BD0B83E322171CA2A65F6CA63CA73042

The renewed certificate has a new serial and a new end date, and the API server is presenting it.

systemctl show kube-apiserver -p NRestarts
journalctl -u kube-apiserver --no-pager -o cat --since -2min | grep 'Loaded a new cert/key pair' | tail -1

NRestarts=0
I1025 17:26:16.749600    1291 dynamic_serving_content.go:116] "Loaded a new cert/key pair" name="serving-cert::/var/lib/kubernetes/kube-apiserver.crt::/var/lib/kubernetes/kube-apiserver.key"

The process never restarted. The API server watches its certificate files and loads a new pair as soon as the files change. Connections that are already open keep their old session, and new ones get the new certificate. That is renewal without downtime, for the most important certificate in the cluster.

The kubelets and the API server’s kubelet client certificate work the same way. Renew them, copy them out, and watch the kubelets start presenting their new certificates.

cd ~/kthw/pki
for NAME in node-0 node-1 apiserver-kubelet-client; do
  ../renew-cert ${NAME}
done
for HOST in node-0 node-1; do
  scp ${HOST}.crt root@${HOST}:/var/lib/kubelet/kubelet.crt
done
scp apiserver-kubelet-client.crt root@server:/var/lib/kubernetes/apiserver-kubelet-client.crt

sleep 10
for HOST in node-0 node-1; do
  echo "${HOST} on disk   $(openssl x509 -in ${HOST}.crt -noout -serial)"
  echo "${HOST} presented $(echo | openssl s_client -connect ${HOST}:10250 2>/dev/null | openssl x509 -noout -serial)"
done
kubectl -n kube-system logs deployment/coredns --tail=1 || true
kubectl -n kube-system logs deployment/coredns --tail=1

subject=CN=system:node:node-0, O=system:nodes
subject=CN=system:node:node-1, O=system:nodes
subject=CN=kube-apiserver-kubelet-client
node-0 on disk   serial=7DFC9C5F91BD897A3C6F2B4D67793F60869CBB23
node-0 presented serial=7DFC9C5F91BD897A3C6F2B4D67793F60869CBB23
node-1 on disk   serial=207489FFEFC20C0649A7816208E944B454BCA2FD
node-1 presented serial=207489FFEFC20C0649A7816208E944B454BCA2FD
linux/amd64, go1.26.5, 424d125
linux/amd64, go1.26.5, 424d125

Both kubelets present their new certificate, again without a restart. The last two lines are CoreDNS’s log, read through the API server and a kubelet, which proves the API server’s renewed client certificate works too. When the API server notices that file change, it closes its open connections to the kubelets so the next ones use the new certificate. In one of our test runs, the first kubectl logs right after the change failed with use of closed network connection, and the next one worked. That is why the block asks twice.

Step 6. Renew what needs a restart

The rest of the certificates live inside kubeconfig files, and a running program does not read its kubeconfig again. Renew the certificates, build fresh kubeconfigs with the same helper as Part 1, copy them out, and restart each program that uses one.

cd ~/kthw/pki
for NAME in admin kube-controller-manager kube-scheduler kube-proxy; do
  ../renew-cert ${NAME}
done

cd ~/kthw/configs
make_kubeconfig() {
  local name=$1 user=$2 server=$3
  kubectl config set-cluster kthw --certificate-authority=../pki/ca.crt \
    --embed-certs=true --server="${server}" --kubeconfig="${name}.kubeconfig"
  kubectl config set-credentials "${user}" --client-certificate="../pki/${name}.crt" \
    --client-key="../pki/${name}.key" --embed-certs=true --kubeconfig="${name}.kubeconfig"
  kubectl config set-context default --cluster=kthw --user="${user}" --kubeconfig="${name}.kubeconfig"
  kubectl config use-context default --kubeconfig="${name}.kubeconfig"
}

make_kubeconfig node-0 system:node:node-0 https://server:6443
make_kubeconfig node-1 system:node:node-1 https://server:6443
make_kubeconfig kube-proxy system:kube-proxy https://server:6443
make_kubeconfig kube-controller-manager system:kube-controller-manager https://127.0.0.1:6443
make_kubeconfig kube-scheduler system:kube-scheduler https://127.0.0.1:6443
make_kubeconfig admin admin https://127.0.0.1:6443

Copy them out and restart. On the workers, note which containers run before the restart, so you can prove afterwards that the restart did not touch them.

cd ~/kthw/configs
for HOST in node-0 node-1; do
  scp -q ${HOST}.kubeconfig root@${HOST}:/var/lib/kubelet/kubeconfig
  scp -q kube-proxy.kubeconfig root@${HOST}:/var/lib/kube-proxy/kubeconfig
  ssh root@${HOST} 'crictl ps -q | sort > /tmp/containers-before && systemctl restart kubelet kube-proxy'
done

scp -q kube-controller-manager.kubeconfig kube-scheduler.kubeconfig root@server:/var/lib/kubernetes/
scp -q admin.kubeconfig root@server:~/admin.kubeconfig
ssh root@server 'systemctl restart kube-controller-manager kube-scheduler'

cd ~/kthw
kubectl config set-credentials admin --client-certificate=pki/admin.crt \
  --client-key=pki/admin.key --embed-certs=true

sleep 20
kubectl get nodes
for HOST in node-0 node-1; do
  ssh root@${HOST} 'crictl ps -q | sort | diff -q /tmp/containers-before - && echo "$(hostname): the same $(crictl ps -q | wc -l) containers are still running"'
done

User "admin" set.
NAME     STATUS   ROLES    AGE    VERSION
node-0   Ready    <none>   105s   v1.37.1
node-1   Ready    <none>   104s   v1.37.1
node-0: the same 2 containers are still running
node-1: the same 2 containers are still running

Both nodes are Ready, and the restarts did not touch a single container. A kubelet that restarts finds its containers still running and takes them over again. Restarting the controller manager and scheduler pauses their control loops for a moment, but the API server keeps answering the whole time.

Step 7. Check everything again

Run the check from step 3 on every machine.

cd ~/kthw
FAILED=0
./cert-check || FAILED=1
for HOST in server node-0 node-1; do
  ssh root@${HOST} cert-check || FAILED=1
done
[ ${FAILED} = 0 ] && echo "all certificates have at least 30 days left"

kthws-jumpbox
  364 days  Oct 25 17:26:41 2027 GMT  /root/.kube/config
server
  364 days  Oct 25 17:26:28 2027 GMT  /var/lib/kubernetes/apiserver-kubelet-client.crt
 3649 days  Oct 22 17:24:05 2036 GMT  /var/lib/kubernetes/ca.crt
  364 days  Oct 25 17:26:16 2027 GMT  /var/lib/kubernetes/kube-apiserver.crt
  364 days  Oct 25 17:26:41 2027 GMT  /var/lib/kubernetes/kube-controller-manager.kubeconfig
  364 days  Oct 25 17:26:41 2027 GMT  /var/lib/kubernetes/kube-scheduler.kubeconfig
  364 days  Oct 25 17:26:41 2027 GMT  /root/admin.kubeconfig
node-0
 3649 days  Oct 22 17:24:05 2036 GMT  /var/lib/kubelet/ca.crt
  364 days  Oct 25 17:26:28 2027 GMT  /var/lib/kubelet/kubelet.crt
  364 days  Oct 25 17:26:28 2027 GMT  /var/lib/kubelet/kubeconfig
  364 days  Oct 25 17:26:41 2027 GMT  /var/lib/kube-proxy/kubeconfig
node-1
 3649 days  Oct 22 17:24:05 2036 GMT  /var/lib/kubelet/ca.crt
  364 days  Oct 25 17:26:28 2027 GMT  /var/lib/kubelet/kubelet.crt
  364 days  Oct 25 17:26:28 2027 GMT  /var/lib/kubelet/kubeconfig
  364 days  Oct 25 17:26:41 2027 GMT  /var/lib/kube-proxy/kubeconfig
all certificates have at least 30 days left

Compare the end dates with step 3. Every certificate except the CA now ends a few minutes later than before, which on a cluster you built today is the whole difference. On a cluster that is a year old, the same steps move every date forward by a year. In practice, renew a month or two before the end date, and set WARN_DAYS to give yourself that margin.

When something does not work

error: You must be logged in to the server (Unauthorized). The client’s own certificate expired, or was signed by a different CA. The reason is in the API server’s log, behind Unable to authenticate the request.
tls: failed to verify certificate: x509: certificate has expired or is not yet valid. The server’s certificate expired. Renew it and copy it in place, as in step 5. The API server and the kubelets pick it up without a restart.
use of closed network connection from kubectl logs or exec, once. The API server just reloaded its kubelet client certificate and closed its old connections. Run the command again.
renew-cert: command not found, or a helper function that is gone. Shell functions live only in the session that defined them, which is why renew-cert is a file in this guide. Run it as ../renew-cert from ~/kthw/pki.

The certificate authority

The CA’s own certificate expires too, in ten years here. There are two very different ways to deal with that.

Renewing the CA certificate with its existing key is like renewing any other certificate. You sign a new CA certificate with the same key and the same subject, and every certificate it already signed stays valid. We checked with openssl verify that all of Part 1’s certificates verify against such a renewed CA. Then you copy the new ca.crt to every place that holds the old one. That is the API server, the controller manager, the kubelets and every kubeconfig.

Replacing the CA key is a real migration. For a while, every component has to trust both the old and the new CA, while every certificate is reissued from the new one. The Kubernetes documentation has a step-by-step guide to rotating CA certificates by hand. It is the reason long CA lifetimes are normal. And it is the only way to take away an admin certificate from Part 1 before it expires, because the system:masters group skips every permission check.

The easy way, once you know the hard way

Two lessons from this part carry over to any cluster. Certificates fail all at once and without warning, and a long-lived administrator certificate is a credential you cannot take back.

Ankra covers both. In clusters Ankra builds, cert-manager comes with the default ingress add-ons and issues and renews the certificates for your applications’ hostnames. And people do not need a system:masters certificate. They can reach the cluster through Ankra’s gateway with short-lived tokens, and an administrator grants each person a role for a limited time, with a reason that lands in the audit log.

ankra cluster access grant [email protected] --cluster my-cluster \
  --role edit --namespace shop --expires 7d --reason "Onboarding to the shop team"

When the grant expires, the access is gone. No certificate to track down, and no CA to replace.

This is Part 5 of Kubernetes the Hard Way, a series of guides we test end to end before publishing. Part 6 covers GitOps by hand.

Get started: Create a free account on Ankra.

Join our community: Slack

ShareX LinkedIn HN

Get the next post in your inbox

Kubernetes Engineering Homelab

Kubernetes the Hard Way, Part 3: Back Up etcd and Restore a Lost Control Plane

Back up a Kubernetes 1.37 cluster by hand with etcd snapshots, then delete the control plane and restore it. Includes the encryption key trap, all tested.

Mattias Åsell

October 25, 2026

12 min read

Kubernetes Engineering Homelab

Kubernetes the Hard Way, Part 2: A Highly Available Control Plane

Build a Kubernetes 1.37 control plane that survives losing a server. Three control planes, etcd with TLS, a load balancer, and failure drills, all tested.

Mattias Åsell

October 25, 2026

13 min read

Back to Blog

Kubernetes the Hard Way, Part 5: Certificates, Expiry and Renewal Without Downtime

What you need

Step 1. Every certificate, and when it ends

Step 2. Where each certificate lives

Step 3. A check you can run every day

Step 4. Let a certificate expire

Step 5. Renew without a restart

Step 6. Renew what needs a restart

Step 7. Check everything again

When something does not work

The certificate authority

The easy way, once you know the hard way

Get the next post in your inbox

Related Posts

Kubernetes the Hard Way, Part 3: Back Up etcd and Restore a Lost Control Plane

Kubernetes the Hard Way, Part 2: A Highly Available Control Plane

Cookie Settings

Necessary Cookies

Analytics and Ad Measurement