A Managed AI Stack for Developers: Private, Under Your Control, Costed to the Token

Somewhere in your company this week, a developer pasted proprietary code into a public AI chatbot. Maybe it was a stack trace with internal hostnames in it. Maybe it was a customer document they wanted summarised, one covered by a confidentiality clause that stopped meaning anything the moment they hit enter. Nobody did it maliciously. They did it because the tool is genuinely useful, and because nobody gave them a better one.

Banning AI tools does not work. Developers route around bans the same way they route around slow procurement: personal accounts, personal laptops, a browser tab you will never see in an audit. The teams that have actually contained the problem did the opposite. They made the private path the easy path.

This post is about what that private path looks like in practice: a managed AI stack that runs on a GPU in your own cloud account, serves your team a chat UI and an OpenAI-compatible API, and gives you the three things a public chatbot never will. Your data stays on your infrastructure. You decide who has access and can prove it. And the cost of a million tokens is a number you have measured, not a bill you discover.

The prompt is the disclosure

When the conversation about AI risk happens at the contract level, it usually fixates on training: “will the provider train on our data?” That is the wrong first question. For anything under NDA, the breach happens at send time. The moment source code, a customer list, or a draft agreement crosses into a third party’s infrastructure, you have disclosed it. Whether it later ends up in a training run only decides how bad the story gets.

Every mitigation on offer from public services is a policy, not a property. Retention windows, enterprise tiers, “we do not train on business data”: all of it is somebody else’s promise about somebody else’s servers, revisable in the next terms update. Policies are what you fall back on when the architecture cannot give you a guarantee. An inference server running on hardware in your own cloud account is an architectural guarantee. There is no third party to trust because there is no third party in the request path.

Three requirements, not one

Talk to the engineering leaders wrestling with this and the same three requirements come up every time.

Privacy is the obvious one: prompts, uploaded documents, chat history, and embeddings should live and die on infrastructure you control.

Control is the one people discover second, usually after the first internal AI tool sprawls. Who has access? Can you give a contractor a key that expires with their contract? Can you cut one person off without rotating a shared secret for everyone else? Can you see who is actually using it?

Cost predictability is the one that decides whether the tool survives the budget review. Metered API pricing means every enthusiastic new user makes the bill worse, so success and cost anxiety arrive together. What finance wants is a ceiling. What developers want is to not think about the meter at all. A fixed-price GPU gives you both.

Why nobody builds this themselves

If self-hosting an LLM were easy, everyone under NDA constraints would already do it. The honest version of the shopping list is why they do not: GPU drivers and a device plugin for Kubernetes, an inference server tuned for your card, an OpenAI-compatible front door, DNS records and TLS certificates, authentication in front of all of it, per-user key management, rate limiting, usage metering, dashboards, and persistent storage so you are not re-downloading nine gigabytes of model weights on every restart.

Each of those is a solved problem. The composition of them is a quarter’s worth of platform engineering, plus the ongoing tax of keeping it patched. That gap between “solved in principle” and “running in production” is exactly what a managed stack has to close.

One stack, about thirty-five minutes

On Ankra the whole platform above is a single stack profile called gpu-chat. Deploying it takes two moves: create a cluster with a GPU node through the Ankra wizard (an NVIDIA L4 on UpCloud in the builds we measured), then instantiate the profile on it. The profile carries everything, wired together with dependency ordering so it comes up in the right sequence:

• The NVIDIA GPU Operator, which turns the raw GPU node into schedulable capacity
• vLLM serving a quantised Qwen3-8B, with model weights cached on a persistent volume so restarts reload in about two minutes
• An inference router that does prefix-cache-aware scheduling, so multi-turn chat traffic lands on the replica that already holds the conversation’s KV cache
• A CPU embeddings server (bge-m3) and a pgvector Postgres, which give the chat UI document upload and retrieval: drop a PDF in, ask questions about it
• Open WebUI as the chat front end, and a token broker in front of the API
• Prometheus and Grafana with GPU, inference, and token-usage dashboards
• Automatic DNS and Let’s Encrypt certificates on a generated domain, so you end with chat.<your-domain> and api.<your-domain> serving real TLS

Instantiating it is one form. The profile ships working defaults for everything else, so out of its 24 inputs only three need a value from you: your domain, and two secrets you generate. Pick the model size that fits your GPU and the form sets context length, memory ceiling, model store, and tool-call parser to match.

From clicking Create Cluster to the first answer streaming in the chat UI is about 35 minutes on a live build, and most of that is waiting on machines: cluster provisioning, the one-time model download, certificate issuance. The stack itself is 33 resources deploying in five to eight minutes.

The end state matters more than the speed. The API speaks the OpenAI protocol, so every SDK, editor plugin, and internal tool that takes a base URL works against it unchanged. Your developers do not adopt a worse workflow to get privacy. They change one URL.

Private means your account, your servers, your bill

Ankra provisions through an API token you create in your own cloud provider account. That detail carries the whole privacy story. The GPU node, the control plane, the volumes holding your model weights and your chat database: all of it appears in your provider’s console, on your invoice, under your organisation’s terms. Ankra holds the configuration and runs the operations. It does not sit between your users and the model, and it does not own the hardware.

So when a developer uploads a design document to the chat and asks about it, here is the complete journey: the file is chunked and embedded on a CPU pod in your cluster, the vectors land in a Postgres running in your cluster, the prompt and the retrieved context hit vLLM on your GPU, and the answer comes back through your ingress. The only thing that ever left your infrastructure was the one-time model weight download.

You do not have to take that on faith, which is the point. Open your cloud console and count the servers.

Control means keys you mint and revoke

The stack puts a token broker in front of the API, and every /v1 route requires a bearer token it minted. Behind the broker, vLLM independently checks an internal key, so there is no unauthenticated path to the model even from inside the cluster. This is not an aspiration in a README: on the live build, no key gets a 401, a wrong key gets a 401, and only a minted token gets a 200.

Minting a key is one API call, and it takes a budget:

curl -s -X POST https://api.<your-domain>/key/generate \
  -H "Authorization: Bearer <master-key>" \
  -d '{"key_alias": "alice", "max_budget": 5}'

That gives Alice her own token with a five euro ceiling. Her spend accrues to her key, she can query her own balance, and if she blows through the budget she starts getting 429s within a minute while everyone else carries on. When her contract ends, revoking her is one call or one click, and no shared secret rotates. The ingress adds per-IP rate limiting on top, and Grafana ships in the stack with a per-key spend and budget dashboard next to the GPU telemetry.

Compare that to the control surface of a public chatbot subscription: a seat list and hope.

Cost you can put in a spreadsheet

Here is where owning the GPU stops being a compliance decision and starts being a financial one. The cluster in the walkthrough build costs about €419 a month, and the wizard shows that figure before you create anything. That is the whole ceiling. No usage surprise lives inside a flat server bill.

What you get for it is measured, not modelled. On the live build, Qwen3-8B-FP8 on a single L4:

The gap between those rows is the economics of batching, and it is the most underappreciated number in self-hosted inference: vLLM serves concurrent requests nearly for free until the GPU saturates, so the busier your endpoint gets, the cheaper every token becomes. A tool your whole team adopts gets cheaper per use. On a metered public API, the same adoption curve is what makes the bill frightening. Embeddings ride the same keys at around €0.02 per million tokens, which rounds to free.

And because every key meters its own spend against those measured rates, “what does the AI tool cost per team?” becomes a dashboard you read instead of an allocation you argue about. When you outgrow one GPU, you raise the replica count, add a node, and the router spreads load across both caches. The cost curve stays linear and public.

What “managed” actually buys you

The division of labour is clean. You own the hardware, the data, the keys, and the bill. Ankra owns the operations: provisioning the cluster, deploying 33 resources in dependency order, wiring DNS and certificates, and tearing everything down cleanly when you are done, volumes included, so nothing lingers and bills quietly.

That is what a managed AI stack means to us. Not a hosted service wearing your logo, but your infrastructure with the platform engineering already done.

The IP leak problem was never really about careless developers. It was about the gap between the tools they were given and the tools they needed. Close the gap and the leak closes with it: same chat window, same API shape, except the machine answering is one you can point to on your own invoice. If you want the full walkthrough with every step, wait time, and failure mode measured on live builds, the GPU Chat Stack guide covers it end to end in twelve pages.

Get started: Create a free account on Ankra and provision your first cluster inside the free 30 vCPU allowance.

Join our community: Slack

Follow us on: LinkedIn | GitHub

Contact us: [email protected]