The Spec Sheet Is Not a Benchmark: Testing Kubernetes Clusters in 15 Minutes

We recently provisioned three Kubernetes clusters on three different providers, all in the same size class: one control plane, two workers, 2 vCPU and 4 GB each. Same Kubernetes distribution, same CNI, same everything we could control. Then we ran the same benchmark suite against all three.

On paper the clusters were interchangeable. In practice, one provider’s two “identical” worker nodes measured 2.8x apart on CPU throughput. Another pair differed 6.6x on sequential disk writes. One node’s fsync latency breached the bar etcd sets for a healthy control plane, which means the cluster passed every health check while carrying a datastore that would degrade under real write load. None of this is visible on a pricing page, and none of it showed up in the provider dashboard either.

This post is about the suite we built to find these things: three phases, about fifteen minutes end to end, run entirely in-cluster with a mechanical pass/fail verdict at the end. The design decisions matter more than the tools, so that is where we will spend most of the time.

What the spec sheet cannot tell you

A vCPU is a scheduling abstraction, not a promise. On shared tiers it is a slice of a physical core that other tenants are also slicing, and the size of your slice varies by the hour and by which physical host you landed on. Network is typically quoted as “up to” some figure that may be enforced by a cap, by the hypervisor, or not at all. Block storage sits behind QoS limits that the spec sheet rounds off to a marketing number.

Kubernetes then adds its own layers: an overlay network between nodes, kube-proxy in the service path, a scheduler that decides which of your not-actually-identical nodes a pod lands on. The only number that predicts how your workload behaves is one measured from inside a pod, through the CNI, against the disks and CPUs your pods will actually use. That is why the suite runs entirely in-cluster: three Kubernetes Jobs and a target deployment, no load generator on a laptop, no benchmark VM sitting outside the cluster measuring a path your traffic will never take.

The three phases map to the three ways clusters disappoint you in production: application latency collapses under load, the network between nodes becomes the bottleneck, or the node underneath (disk, fsync, CPU) turns out to be weaker than its twin.

Phase 1: load the cluster the way traffic arrives

The HTTP phase uses Grafana k6 running as a Job inside the cluster, aimed at a small target service (podinfo). The single most important choice here is the executor model, and it is the one most home-grown load tests get wrong.

A closed-loop test, the default in most tools, runs N virtual users that each wait for a response before sending the next request. When the target slows down, the test politely slows down with it, so the arrival rate drops exactly when the system is struggling. Your report shows modest latency at a modest rate and everyone concludes the cluster is fine. This is the coordinated omission problem, and it systematically flatters slow systems.

An open-model test holds the arrival rate constant no matter how slowly responses come back. If the target degrades, requests queue, latency percentiles blow out, and the test tells you the truth. In k6 that is the constant-arrival-rate and ramping-arrival-rate executors:

scenarios: {
  smoke:    { executor: 'constant-vus', vus: 2, duration: '30s' },
  baseline: { executor: 'constant-arrival-rate', rate: 50,
              timeUnit: '1s', duration: '3m', preAllocatedVUs: 30 },
  stress:   { executor: 'ramping-arrival-rate', preAllocatedVUs: 50,
              stages: [ { target: 100, duration: '1m' },
                        { target: 200, duration: '1m' },
                        { target: 300, duration: '1m' } ] },
  spike:    { executor: 'ramping-arrival-rate', preAllocatedVUs: 50,
              stages: [ { target: 400, duration: '20s' },
                        { target: 400, duration: '40s' } ] },
},
thresholds: {
  'http_req_duration{scenario:baseline}': ['p(95)<400', 'p(99)<800'],
  'http_req_duration{scenario:spike}':    ['p(99)<2000'],
  'dropped_iterations{scenario:baseline}': ['count<10'],
},

The thresholds are the second load-bearing choice. Every scenario carries SLO thresholds on latency percentiles and error rate, and a breach fails the k6 process, which fails the Job, which fails the runner script. The verdict is mechanical: exit code zero or not. Nobody squints at a latency graph and declares it “probably fine”, and the same property makes the suite usable as a CI gate for cluster changes: run it after a node pool resize or a CNI upgrade and let the pipeline decide.

One war story about validating the harness itself. Our first run produced median latencies around 0.1 ms together with a 10% error rate, which briefly looked like a fast cluster with a flaky app. It was neither: podinfo’s delay endpoint takes integer seconds, our script requested /delay/0.2, and every one of those requests 404ed instantly. Fast failures made latency look impossibly good while poisoning the error SLO. The lesson generalises: when a benchmark result is surprisingly excellent in one dimension and surprisingly bad in another, suspect the harness before the cluster.

Phase 2: measure the network between nodes, not the loopback

The network phase is iperf3, a server Deployment and a client Job, thirty seconds forward and thirty seconds reverse with four parallel streams. The whole phase hinges on five lines of YAML:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - topologyKey: kubernetes.io/hostname
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: iperf3-server

Without required anti-affinity, the scheduler will happily place client and server on the same node, and you will spend an afternoon admiring the throughput of the loopback interface. With it, the traffic crosses the actual node-to-node path: the overlay network, the virtual NIC, the provider’s fabric.

The throughput number alone is less interesting than the limiting factor, which the same run reveals if you look at CPU and retransmits. On one provider we measured 5.4 Gbit/s with the receiving pod’s CPU pinned at 95%: the network had headroom and the small node was the bottleneck, so a bigger instance would go faster. On another we saw 1.7 Gbit/s with CPU loafing at 55%: a provider-side cap, and no instance resize will change it. Same benchmark, opposite capacity-planning conclusions.

Phase 3: the node under the node

The third phase benchmarks each worker individually: fio for disk (sequential read/write at 1M block size, random read/write at 4k with direct IO) and sysbench for CPU and memory. Getting one pod onto every worker without hand-picking nodes is an Indexed Job whose pods carry required anti-affinity against each other, so the scheduler is forced to spread them one per node.

The measurement that has predicted the most real-world pain for us is none of the headline numbers. It is fsync:

fio --name=fsyncwrite --rw=write --bs=4k --size=64M \
    --runtime=30 --time_based --fdatasync=1 \
    --ioengine=psync --iodepth=1

Every write followed by fdatasync, queue depth one. This is the IO pattern of etcd’s write-ahead log and of any database that cares about durability, and it punishes storage that posts great parallel IOPS but stalls on synchronous flushes. etcd’s own guidance wants the 99th percentile of fdatasync under 10 ms. In our three-provider run, one node came in at 11.9 ms, on a cluster that passed every HTTP SLO in phase 1. That cluster would run a web tier beautifully and make a miserable home for a control plane or PostgreSQL. You want to learn that from a three-minute Job, not from an incident review.

The other thing this phase exposes is variance between supposedly identical nodes. Same instance type, same region, same hour: 2.8x apart on sysbench CPU events per second on one provider’s shared tier, 2.4x apart on memory bandwidth on another’s standard tier. Noisy neighbours are real, and they are invisible until you measure each node separately, which is exactly why the phase refuses to sample just one worker and call it representative.

Sequential phases, greppable results

Two operational rules hold the suite together. The first: phases run strictly one after another, never in parallel. k6 latency percentiles measured while fio is saturating the disks of a 2-vCPU node are not pessimistic, they are meaningless, and the contamination runs both ways. The full sequence is about fifteen minutes; resist the urge to shave five of them by overlapping.

The second: every phase emits its results as tagged lines in the pod logs (K6_SUMMARY_JSON, IPERF_FWD_JSON, FIO_RESULT), and the Jobs carry a 24-hour TTL after which the cluster cleans them up. No artifact storage, no volumes to mount, nothing to install on the cluster beforehand and nothing to scrub afterwards. Logs are the one output channel every Kubernetes setup already has, whatever is collecting them.

What three providers looked like on the same day

Condensed from one day’s run, same suite, same cluster shape everywhere; the full per-node numbers and the verdicts are in the companion results post. Not a buyer’s guide: one day, two workers per provider, one size class, and entry tiers differ in kind (DigitalOcean’s entry tier is shared vCPU by design, where a dedicated tier at roughly twice the price would be the fairer fight).

Every cluster passed all eight HTTP SLOs. The differences live underneath: one provider’s block storage holds a flat 100k IOPS while another’s caps at 20k; the best fsync latency and the best IOPS belong to different providers; and the shared tier’s twin workers might as well be different instance types. Which provider “wins” depends entirely on what you are placing: an etcd or database node weights fsync and node consistency, a stateless web tier barely notices any of it, and at these prices the cost conversation shifts from list price to price per delivered IOPS or Gbit.

That is the real argument for owning a benchmark suite rather than reading someone else’s numbers, including ours: the answer changes by provider, by tier, by region, and by which physical host you happened to land on this morning. The only benchmark that reflects your cluster is the one you ran on it.

Where a platform earns its keep

The unglamorous prerequisite for all of this is being able to stand up and tear down real clusters cheaply. The three-provider comparison existed because provisioning each cluster through Ankra was a ten-minute, repeatable operation against a declared cluster shape, not an afternoon of console clicking per provider. The benchmark workloads themselves ship as a stack: the target service and iperf3 server deploy as one unit onto any cluster Ankra manages, the three Job phases run in sequence, and the whole exercise costs whatever three small VMs cost for an hour. When the numbers are in, the clusters are deleted the same way they were created.

That loop, provision, measure, decide, delete, is the difference between benchmarking as a one-off blog post and benchmarking as a habit. Run the suite when you evaluate a provider, after you resize a node pool, before you promote a cluster to production, and any time two nodes that should be identical start behaving like they are not. The spec sheet tells you what you are paying for. Fifteen minutes of Jobs tells you what you got.

Get started: Create a free account on Ankra.

Join our community: Slack

Follow us on: LinkedIn | GitHub

Contact us: [email protected]