Kubernetes the Hard Way, Part 6: GitOps by Hand, a Reconcile Loop With Git and kubectl

GitOps sounds like a product you have to install. It is really one idea. The state you want lives in a Git repository, and a loop keeps comparing the cluster with that repository and fixing every difference. Flux and Argo CD are polished, battle-tested versions of that loop.

In this part you build the loop yourself, on the cluster from Part 1. You will set up a Git repository, give a reconciler its own narrow identity, and run it every 30 seconds with systemd. Then you will use it the way a team does. You will change the cluster through a commit, watch it undo a manual change, delete something by deleting a file, roll back with git revert, and push a broken commit to see what the loop does with it.

We ran every command in this post, top to bottom, on fresh machines before publishing it, and the outputs on this page are the ones we got.

What you need

The cluster from Part 1, with its jumpbox and server and the files in ~/kthw. Git is the only new package.

One kubectl feature in this part is still marked alpha. --applyset, which decides what to prune, only works with KUBECTL_APPLYSET=true set. You will see why it is worth it in step 6.

Step 1. A repository for the cluster

The repository lives on the server, as a bare repository with no working copy, the same kind GitHub keeps for you. You work in a clone on the jumpbox and push to it over SSH, which root on the jumpbox can already do.

apt-get update -qq
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git >/dev/null
git init --quiet --bare --initial-branch=main /srv/git/store.git
ls /srv/git/store.git

branches
config
description
HEAD
hooks
info
objects
refs

apt-get update -qq
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git >/dev/null
git config --global user.name "Hard Way"
git config --global user.email "[email protected]"
git config --global init.defaultBranch main

git clone --quiet root@server:/srv/git/store.git ~/store
cd ~/store
mkdir manifests

cat > manifests/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.30
          ports:
            - containerPort: 80
EOF

cat > manifests/service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
EOF

cat > manifests/settings.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: settings
data:
  currency: EUR
EOF

git add manifests
git commit --quiet -m "Add the web app"
git push --quiet origin main
git log --oneline

warning: You appear to have cloned an empty repository.
653ec58 Add the web app

The manifests name no namespace. The reconciler decides that, so the same files could be applied to any namespace you point it at.

Step 2. An identity for the reconciler

The reconciler should be able to change the store namespace and nothing else. Give it a client certificate from the cluster’s CA, exactly like the identities in Part 1, and a role binding that grants the built-in edit role in that one namespace.

cd ~/kthw/pki
openssl genpkey -quiet -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out gitops.key
openssl req -new -key gitops.key -subj "/CN=gitops-reconciler" -out gitops.csr
printf 'basicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature,keyEncipherment\nextendedKeyUsage=clientAuth\n' > gitops.ext
openssl x509 -req -in gitops.csr -CA ca.crt -CAkey ca.key -days 365 -sha256 \
  -extfile gitops.ext -out gitops.crt 2>/dev/null
rm gitops.csr gitops.ext

cd ~/kthw/configs
kubectl config set-cluster kthw --certificate-authority=../pki/ca.crt \
  --embed-certs=true --server=https://127.0.0.1:6443 --kubeconfig=gitops.kubeconfig
kubectl config set-credentials gitops-reconciler --client-certificate=../pki/gitops.crt \
  --client-key=../pki/gitops.key --embed-certs=true --kubeconfig=gitops.kubeconfig
kubectl config set-context default --cluster=kthw --user=gitops-reconciler --kubeconfig=gitops.kubeconfig
kubectl config use-context default --kubeconfig=gitops.kubeconfig

kubectl create namespace store
kubectl -n store create rolebinding gitops-reconciler --clusterrole=edit --user=gitops-reconciler

kubectl auth can-i --as=gitops-reconciler -n store create deployments
kubectl auth can-i --as=gitops-reconciler -n default create deployments || true
kubectl auth can-i --as=gitops-reconciler -n store get secrets
kubectl auth can-i --as=gitops-reconciler create namespaces || true

ssh root@server mkdir -p /var/lib/gitops
scp -q gitops.kubeconfig root@server:/var/lib/gitops/gitops.kubeconfig
ssh root@server chmod 600 /var/lib/gitops/gitops.kubeconfig

Cluster "kthw" set.
User "gitops-reconciler" set.
Context "default" created.
Switched to context "default".
namespace/store created
rolebinding.rbac.authorization.k8s.io/gitops-reconciler created
yes
no
yes
Warning: resource 'namespaces' is not namespace scoped

no

The four can-i answers are the reconciler’s whole world. It may create Deployments in store, but not in default, and it may not create namespaces. The warning on the last one only says that a namespace belongs to no namespace, so kubectl asked about the cluster as a whole.

The third answer is worth a pause. The edit role can read Secrets in its namespace, so anyone who can push to this repository can, in effect, read those Secrets too. That is true of every GitOps tool, and it is why write access to a GitOps repository deserves the same care as access to the cluster.

Step 3. The reconcile loop

The loop is a short script and a systemd timer. Each run fetches the repository, moves its working copy to exactly what main says, and applies every manifest.

cat > /usr/local/bin/gitops-reconcile <<'EOF'
#!/bin/bash
# Make the store namespace match the main branch of the repository, once.
set -euo pipefail
REPO=/srv/git/store.git
WORK=/var/lib/gitops/store
export KUBECONFIG=/var/lib/gitops/gitops.kubeconfig
export KUBECTL_APPLYSET=true

[ -d "${WORK}" ] || git clone --quiet "${REPO}" "${WORK}"
git -C "${WORK}" fetch --quiet origin
git -C "${WORK}" reset --quiet --hard origin/main
REVISION=$(git -C "${WORK}" log -1 --format='%h %s')

kubectl apply --namespace store --server-side --field-manager=gitops --force-conflicts \
  --prune --applyset=configmap/gitops-applyset -f "${WORK}/manifests/"
echo "in sync with ${REVISION}"
EOF
chmod 0755 /usr/local/bin/gitops-reconcile

cat > /etc/systemd/system/gitops-reconcile.service <<'EOF'
[Unit]
Description=Apply the store repository to the cluster

[Service]
Type=oneshot
ExecStart=/usr/local/bin/gitops-reconcile
EOF

cat > /etc/systemd/system/gitops-reconcile.timer <<'EOF'
[Unit]
Description=Reconcile the store repository every 30 seconds

[Timer]
OnBootSec=30s
OnUnitInactiveSec=30s
AccuracySec=1s

[Install]
WantedBy=timers.target
EOF

systemctl daemon-reload
systemctl enable --now gitops-reconcile.timer
systemctl start gitops-reconcile.service
journalctl -u gitops-reconcile --no-pager -o cat | grep -v -E '^(Starting|Finished|gitops-reconcile.service)'

Created symlink '/etc/systemd/system/timers.target.wants/gitops-reconcile.timer' → '/etc/systemd/system/gitops-reconcile.timer'.
deployment.apps/web serverside-applied
service/web serverside-applied
configmap/settings serverside-applied
in sync with 653ec58 Add the web app

Three choices in that script do most of the work.

• git reset --hard origin/main makes the working copy exactly what the branch says, whatever happened to it before. The loop never merges and never keeps local changes.
• --server-side --field-manager=gitops --force-conflicts applies with server-side apply. The API server records which fields each manager set, and --force-conflicts lets the reconciler take back a field that someone else changed. That is what turns apply into a repair.
• --prune --applyset=configmap/gitops-applyset removes objects that disappeared from the repository. kubectl labels everything it applies as part of the set and keeps the list on that ConfigMap, so it prunes only what it created and never touches anything else in the namespace.

OnUnitInactiveSec=30s starts the next run 30 seconds after the previous one ended, so two runs never overlap.

Step 4. Change the cluster through Git

From now on, nobody changes the store namespace directly. You change the repository. Scale the deployment to three replicas with a commit.

cd ~/store
sed -i 's/replicas: 2/replicas: 3/' manifests/deployment.yaml
git commit --quiet -am "Scale web to 3 replicas"
git push --quiet origin main

until [ "$(kubectl -n store get deployment web -o jsonpath='{.spec.replicas}')" = 3 ]; do sleep 5; done
kubectl -n store rollout status deployment/web --timeout=120s
ssh root@server "journalctl -u gitops-reconcile --no-pager -o cat --since -2min | grep 'in sync with' | tail -1"

deployment "web" successfully rolled out
in sync with e8c9fd6 Scale web to 3 replicas

You never ran a kubectl command that changes anything. The cluster did what the commit said, and the reconciler’s log names the commit it applied. Every change to the namespace now has an author, a time and a message, and can be reviewed before it lands.

Step 5. Watch the loop undo a manual change

Now be the colleague who fixes something by hand at two in the morning. Scale the deployment and delete the Service directly with kubectl, then wait for one loop.

kubectl -n store scale deployment web --replicas=6
kubectl -n store delete service web
kubectl -n store create configmap made-by-hand --from-literal=note=not-in-git

sleep 45
kubectl -n store get deployment web
kubectl -n store get service web
kubectl -n store get configmaps

deployment.apps/web scaled
service "web" deleted from store namespace
configmap/made-by-hand created
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    3/3     3            3           79s
NAME   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
web    ClusterIP   10.32.0.180   <none>        80/TCP    17s
NAME               DATA   AGE
gitops-applyset    0      79s
kube-root-ca.crt   1      82s
made-by-hand       1      45s
settings           1      79s

Within one loop, the deployment is back to the three replicas the repository asks for, and the Service exists again, 17 seconds old, with a new address. The hand-made ConfigMap is still there. The reconciler repairs what it manages and leaves everything else alone, because only objects in its apply set are its business.

That is GitOps’ answer to the two-in-the-morning fix. It does not survive. If the fix was right, it belongs in a commit.

Step 6. Delete something by deleting a file

Remove the ConfigMap from the repository.

cd ~/store
git rm --quiet manifests/settings.yaml
git commit --quiet -m "Remove the settings ConfigMap"
git push --quiet origin main

until ! kubectl -n store get configmap settings >/dev/null 2>&1; do sleep 5; done
kubectl -n store get configmaps
ssh root@server "journalctl -u gitops-reconcile --no-pager -o cat --since -2min | grep -E 'pruned|in sync with' | tail -2"

NAME               DATA   AGE
gitops-applyset    0      96s
kube-root-ca.crt   1      99s
made-by-hand       1      62s
configmap/settings pruned
in sync with dbaf88c Remove the settings ConfigMap

The file is gone, so the object is gone. Without pruning, a plain kubectl apply would only ever add and change things, and a cluster managed that way slowly fills up with objects nobody remembers. Notice that made-by-hand survived this too.

Step 7. Roll back with git revert

Undoing a change is a commit too. git revert writes a new commit that reverses an old one, so the history keeps both.

cd ~/store
git revert --no-edit HEAD >/dev/null
git push --quiet origin main
git log --oneline

until kubectl -n store get configmap settings >/dev/null 2>&1; do sleep 5; done
kubectl -n store get configmap settings -o jsonpath='{.data.currency}'; echo

a0f6766 Revert "Remove the settings ConfigMap"
dbaf88c Remove the settings ConfigMap
e8c9fd6 Scale web to 3 replicas
653ec58 Add the web app
EUR

The ConfigMap is back with its data, and the log tells the whole story. A rollback in GitOps is not a special command. It is a commit that happens to restore an older state, reviewed and recorded like any other.

Step 8. Push a broken commit

A typo in a manifest is the most common GitOps incident. Push one on purpose.

cd ~/store
sed -i 's/replicas: 3/replicas: three/' manifests/deployment.yaml
git commit --quiet -am "Scale web, with a typo"
git push --quiet origin main

sleep 45
ssh root@server "systemctl is-failed gitops-reconcile.service; journalctl -u gitops-reconcile --no-pager -o cat --since -40s | grep -E 'applied|pruned|Error from server|Failed with result' | tail -4" || true
kubectl -n store get deployment web

failed
service/web serverside-applied
configmap/settings serverside-applied
Error from server: failed to create typed patch object (store/web; apps/v1, Kind=Deployment): .spec.replicas: expected numeric (int or float), got string
gitops-reconcile.service: Failed with result 'exit-code'.
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    3/3     3            3           2m56s

The API server rejected the Deployment, because three is not a number. The service is marked failed, and the deployment keeps running its three replicas, untouched. Read the lines above the error carefully, though. The Service and the ConfigMap were applied in the same run. kubectl applies each object on its own, so one bad object does not stop the others. It did stop the prune, because kubectl does not prune after an error.

The loop keeps retrying every 30 seconds and failing the same way until the repository is fixed. A real setup alerts on that failed state. Here, systemctl is-failed is the alarm.

Fix it the GitOps way, with another commit.

cd ~/store
git revert --no-edit HEAD >/dev/null
git push --quiet origin main
sleep 45
ssh root@server "systemctl is-failed gitops-reconcile.service || true; journalctl -u gitops-reconcile --no-pager -o cat --since -1min | grep 'in sync with' | tail -1"

inactive
in sync with d09e364 Revert "Scale web, with a typo"

inactive means the last run succeeded and the next one is waiting for its timer.

When something does not work

• E: Unable to locate package git. The machine’s package lists are empty, which is normal on a fresh cloud image. Run apt-get update first, as the blocks above do.
• Error from server: failed to create typed patch object ... expected numeric, and the service is failed. A manifest in the repository is invalid. Find it in the error, fix it with a commit, and the next run recovers.
• A manual change keeps coming back, or keeps disappearing. That is the loop working. Change the repository instead, or take the object out of the repository if a person should own it.
• warning: You appear to have cloned an empty repository. That is expected for the very first clone, before anything was pushed.

What a real GitOps tool adds

The loop you built is real GitOps, and it is about 20 lines. Flux and Argo CD do the same thing, plus what a team needs once more than one person relies on it.

• They run inside the cluster as controllers, pull from a hosted Git service and need no SSH access to a machine.
• They know about health. A sync is not done when apply returns, but when the new pods are ready, and a sync whose pods never become ready is reported as failed.
• They order things, such as a namespace and its CRDs before the objects that need them.
• They handle Secrets, which do not belong in Git in plain text. Encrypting them with SOPS is one answer, and this post on SOPS and age shows how.
• They report and alert, with a status per application, the commit it runs, and a notification when a sync fails.

The easy way, once you know the hard way

Ankra puts this loop under every cluster it manages, without a script on any machine. A cluster’s stacks, its add-ons, manifests and settings, are stored in a Git repository you own, on GitHub or Bitbucket Cloud, one directory per cluster, and Ankra’s engine keeps cluster and repository in sync in both directions. A change in the portal becomes a commit, and a merged pull request becomes a change in the cluster. Wire the repository when Ankra builds the cluster.

ankra cluster hetzner create --name my-cluster --location hel1 \
  --credential-id <your-hetzner-credential> \
  --gitops-repository my-org/platform-gitops \
  --gitops-credential-name my-github-credential --gitops-branch main
ankra cluster gitops status my-cluster

ankra cluster gitops status shows what your in sync with line showed here, and more. It names the repository and branch, the last commit applied, and any sync error. When the same object is changed in Git and in the portal at once, Ankra pauses the sync and shows you the conflict instead of letting one side win silently.

This is Part 6 of Kubernetes the Hard Way, a series of guides we test end to end before publishing. A part on upgrading Kubernetes by hand follows when Kubernetes 1.38 is released.

Get started: Create a free account on Ankra.

Join our community: Slack

Follow us on: LinkedIn | GitHub

Contact us: [email protected]