Prometheus vs VictoriaMetrics: When the Default Stops Being the Answer

Every Kubernetes cluster we see starts its monitoring life the same way: kube-prometheus-stack, Grafana, a dashboard someone found on the internet, done. That is the correct starting move. It is also why, a year or two later, we keep having the same conversation with teams whose Prometheus has started eating 40 GB of RAM, restarting into half-hour WAL replays, or getting OOM-killed the day a deploy added one innocent-looking label.

That conversation is usually titled “should we move to VictoriaMetrics?” and the answer is a genuine “it depends,” which is the most annoying answer in engineering unless someone tells you exactly what it depends on. That is this post. Ankra has no horse in this race: the platform reads metrics from any Prometheus-compatible endpoint, and both systems deploy as ordinary stacks. What follows is the comparison we walk customers through, with the numbers and the failure stories that inform it.

What Prometheus gets right, still

Let us start by being fair to the incumbent, because it earned its position.

Prometheus is the second project ever to graduate from the CNCF, after Kubernetes itself. Its governance is multi-vendor, its ecosystem is the industry’s shared vocabulary, and its integration with Kubernetes is not “good,” it is the reference: the Prometheus Operator’s ServiceMonitor, PodMonitor, and PrometheusRule CRDs are the most battle-tested way to wire scraping that exists. Every exporter targets it. Every alerting example assumes it. PromQL is the lingua franca that every competing backend, VictoriaMetrics included, must speak to be taken seriously.

And it has not stood still. The 3.x line (3.0 shipped in late 2024; 3.13 landed this month) brought a native OTLP receiver so OpenTelemetry metrics land without a collector in between, full UTF-8 metric and label names so OTel-style names no longer get mangled with underscores, and native histograms, stable since 3.8, which replace dozens of bucket series with a single efficient exponential histogram. The old benchmark gaps you find in blog posts from 2020 have narrowed with every release since.

For a single cluster with tens or a few hundreds of thousands of active series, Prometheus with 15 to 30 days of retention is boring, well-understood, and correct. If that describes you, you can stop reading and keep your stack. There is no prize for migrating a healthy system.

The wall, and why it is always cardinality

Prometheus keeps every active time series in an in-memory head block at a cost of roughly 3 to 4 KB of RAM per series. That is the single number that explains most Prometheus incidents: a million active series is several gigabytes of RAM before anyone runs a query. Every unique combination of label values is its own series. Attach a label with unbounded values, a user ID, a request path, a pod name in a high-churn deployment, and the series count multiplies silently.

The failure mode is not graceful degradation. It is the OOM killer, followed by a write-ahead-log replay on restart that can take tens of minutes while you have no monitoring at all. This is not a small-team skill issue. Cloudflare maintains custom patches on their 900-plus Prometheus fleet because upstream has no hard cap on series count. The UK Ministry of Justice’s cloud platform lost monitoring for over three hours in 2024 when a WAL replay outran the startup probe and put Prometheus into a restart loop. Teams running Prometheus on machines with hundreds of gigabytes of RAM have still hit the ceiling.

You can hold the wall back with discipline: recording rules, relabeling to drop offending labels, sample_limit per target, federation to shard the load. Plenty of teams do. But notice what you are doing: operating around the storage engine’s design, forever. At some scale, the question stops being “how do we tune Prometheus” and becomes “should the thing we tune be something else.”

What VictoriaMetrics actually changes

VictoriaMetrics is an Apache 2.0 time-series database built to be a drop-in Prometheus replacement: it scrapes the same targets with the same configs, serves the same HTTP query API so Grafana’s Prometheus datasource works unchanged, and accepts Prometheus remote_write. The differences are under the floor.

The storage engine. Instead of a WAL plus in-memory head block plus two-hour compacted blocks, VictoriaMetrics uses a MergeTree-inspired design closer to ClickHouse. The practical consequences: no WAL replay on restart, no compaction I/O spikes, and much tighter compression. When series churn exceeds what fits in memory, ingestion slows down (visible in a metric you can alert on) rather than the process dying. Running out of headroom becomes a dial you watch instead of a page you get.

The resource profile. Here honesty is required, because most published benchmarks come from the vendor. VictoriaMetrics’ own 2024 comparison showed 2.5x less disk and 1.7x less RAM than Prometheus on identical workloads; the older “7x less disk” figures trace to a 2020 test against a long-obsolete Prometheus and should be retired. The credible independent evidence points the same direction with workload-dependent magnitude: Criteo, running around a billion active series, reported roughly one byte per datapoint in production and cut their metrics backend from 382 nodes to 61 while storing 15 times more data. Prezi went on record with a 70 percent storage reduction, 60 percent less memory, and heavy queries dropping from over 30 seconds to single digits after moving 5 million series off Prometheus. The rule of thumb that survives contact with reality: budget about 1 GB of RAM per million active series for VictoriaMetrics against several for Prometheus.

The scaling story. A single VictoriaMetrics binary comfortably handles millions of active series, and past that there is a cluster version with horizontal scaling and replication set by one flag. Which brings us to the comparison people actually get wrong.

The real comparison is not Prometheus vs VictoriaMetrics

For anything beyond one cluster and a few weeks of retention, plain Prometheus is not the alternative, because plain Prometheus does not do high availability, global views, or years of retention at all. The honest comparison is Prometheus plus Thanos or Mimir versus VictoriaMetrics, and that changes the calculus entirely.

The Prometheus path to durability means running replica pairs per cluster, sidecars or remote writes shipping blocks to object storage, compactors, store gateways, and query frontends that deduplicate across replicas. Thanos and Mimir are both capable, proven systems. They are also each a distributed system you now operate on top of your monitoring, with their own failure modes, upgrade cycles, and 2 AM personalities.

VictoriaMetrics answers the same requirements inside one system: retention is a flag, replication is a flag, deduplication is a flag, and multi-year storage rides on the same compressed local disks. Criteo’s numbers above came from evaluating exactly this three-way choice. When teams tell us they want “Prometheus but durable,” a large part of the answer is deciding whether they want to assemble that from parts or get it in one box.

PromQL vs MetricsQL, the fine print

VictoriaMetrics queries with MetricsQL, a deliberate superset of PromQL. Nearly all existing queries and Grafana dashboards run unchanged, which is why migrations rarely break dashboards. It adds genuinely useful things: WITH expressions for reusable query fragments, extra rollup functions, the keep_metric_names modifier.

But “superset” hides deliberate divergence. In the independent PromLabs compliance suite, VictoriaMetrics scored 74 percent, against 100 for Thanos and Mimir. Most failures are intentional choices the VictoriaMetrics authors consider more correct: rate() and increase() consider the sample just before the window, NaN handling differs, metric names survive some functions that strip them in Prometheus. In day-to-day dashboarding you will likely never notice. Where it bites is precision work: a large library of recording rules and alerts tuned to Prometheus’ exact extrapolation behavior may fire differently after a migration, and alert thresholds deserve a review pass. If strict PromQL portability across backends is a hard requirement for you, that is a legitimate reason to stay in the Prometheus family.

Governance, because it matters now

The last two years taught everyone to read licenses before adopting infrastructure. Prometheus is CNCF-owned, multi-vendor, and structurally impossible for any one company to capture. VictoriaMetrics is one bootstrapped company of about 50 people, with an enterprise tier alongside the open core.

That asymmetry is real but smaller than it looks. VictoriaMetrics’ open source is Apache 2.0 and the project takes no contributor license agreement, meaning contributors keep their copyright and the existing code cannot be relicensed out from under you the way HashiCorp’s and Redis’s licenses were changed. The realistic risk is not a rug-pull; it is future features landing enterprise-first. Adopt with eyes open, as with any single-vendor dependency.

The decision, compressed

Stay on (or start with) Prometheus when:

• One or a few clusters, series counts in the low hundreds of thousands, retention needs measured in weeks
• Your organisation requires CNCF-governed, vendor-neutral tooling
• You maintain recording rules and alerts whose exact PromQL semantics are load-bearing
• Your stack is healthy today; migrating a working system is a cost with no benefit

Move to (or start with) VictoriaMetrics when:

• Prometheus RAM or OOM incidents are a recurring agenda item
• You need one queryable view over many clusters, the exact fleet problem we described in the edge post
• You want months or years of retention without operating Thanos or Mimir alongside
• You are paying for monitoring nodes and the 1 GB versus several GB per million series difference is a real invoice line, which after June’s Hetzner repricing it is

And remember the third option, which is where most successful migrations start: both. Add VictoriaMetrics as a remote_write target from your existing Prometheus, point long-range dashboards at it, and let the two run side by side until the cutover is a formality instead of a bet:

remote_write:
  - url: http://victoria-metrics.monitoring.svc.cluster.local:8428/api/v1/write

Running either on Ankra

Whichever way you decide, the mechanics on Ankra are the same, because both are just addons in a stack.

The classic stack is kube-prometheus-stack plus Loki and Promtail, with the namespace deployed first and dependencies wired through parents:

stacks:
  - name: monitoring
    manifests:
      - name: monitoring-ns
        from_file: "manifests/monitoring-namespace.yaml"
    addons:
      - name: kube-prometheus-stack
        chart_name: kube-prometheus-stack
        chart_version: 65.1.1
        repository_url: https://prometheus-community.github.io/helm-charts
        namespace: monitoring
        parents:
          - manifest: monitoring-ns
        configuration:
          from_file: "values/kube-prometheus-stack.yaml"
          encrypted_paths:
            - adminPassword

Swap the addon for victoria-metrics-single (or the cluster chart) from the VictoriaMetrics Helm repository and the shape of the stack does not change. Pin the chart version, keep the Grafana admin password SOPS-encrypted, and clone the same stack to every cluster that needs it.

Then point the platform at whichever backend you chose. Ankra’s metrics integration accepts any Prometheus-compatible query endpoint, and that compatibility is exactly why this choice is low-stakes at the platform layer:

That one setting feeds the resource graphs on every pod and deployment, the cluster dashboards, and, most importantly, the AI. When an incident report lands in Slack or the assistant explains why a pod is throttling, the metrics behind that analysis come from this data source. The AI does not care whether container_memory_working_set_bytes came out of Prometheus or VictoriaMetrics. It cares that the query answers quickly, which, if you have just moved a multi-million-series fleet to VictoriaMetrics, it now does.

Pick the engine for your scale, not for fashion. Prometheus by default, VictoriaMetrics when the default starts costing you sleep or servers, and either way the platform on top does not change.

Get started: Create a free account on Ankra.

Join our community: Slack

Follow us on: LinkedIn | GitHub

Contact us: [email protected]