Menu
Image

NEWSLETTER

One email a week on what’s actually working in AI: tools, agents, infrastructure and funding. Plus first-hand lessons from my own founder journey, so you can build and scale AI products that actually work.

Follow Us

Add more Kubernetes logos

The GPU Is the New Server: How AI Startups Are Using Kubernetes to Stop Wasting $1M+ of Compute

Mélony Qin Published on November 22, 2026 0

For a decade, the server was the unit every startup counted. In AI startups, that unit is now the GPU. It is the line on the cloud bill that grows fastest, and the first one founders look at when the runway gets shorter.

Here is the uncomfortable part. Across tens of thousands of Kubernetes clusters, GPUs are busy only 5% of the time on average, according to the 2026 State of Kubernetes Optimization Report from Cast AI. CPU sits at 8% and memory at 20%. Most of what teams pay for simply waits.

That gap is where the $1M+ in the title comes from. And Kubernetes, the platform many AI startups already run on, now has the tools to close a large part of it.

The 5% problem

Cast AI’s figure comes from its own telemetry, not a survey, so it describes the clusters it can see. Surveys point the same way. In VentureBeat’s Pulse survey, 79% of the 218 companies that reported their GPU use said their GPUs are busy half the time or less. And 70% of those companies still plan a new or additional AI compute platform within 12 months.

So the instinct is to buy more. The better first move is usually to use what is already paid for.

Kubernetes clusters leave most capacity unusedAverage use across tens of thousands of clusters, bars drawn on a 0 to 100% scaleMemory20%CPU8%GPU5%Source: Cast AI, 2026 State of Kubernetes Optimization Reportcvisiona.com
Diagram: Kubernetes clusters leave most capacity unused, by CVisiona.

How idle GPUs turn into a $1M+ bill

Take a modest setup for training and inference: four AWS p5.48xlarge instances, each with eight NVIDIA H100 GPUs, so 32 GPUs in all. The on demand price in us-east-1 is $55.04 an hour per instance, as listed on DoiT’s live pricing page.

Run them around the clock for a year and the bill is $1,928,601.60. At the 5% average use Cast AI measured, $96,430.08 of that pays for real GPU work. The other $1,832,171.52 pays for GPUs that are switched on and waiting.

Your numbers will differ. Prices change by region, and a team with good habits will sit well above 5%. But the order of magnitude explains why compute waste has become a board level topic for AI startups, just like unchecked inference bills.

Where a year of GPU spend goes at 5% use32 H100 GPUs (4 p5.48xlarge on demand at $55.04 an hour) running all year.Purple to blue bars are the bill and its idle part, the light blue bar is real work.Billed for a year$1,928,601.60Paid for idle time$1,832,171.52Paid for GPU work$96,430.08Sources: AWS price via DoiT (us-east-1), utilization from Cast AI. Calculation by CVisiona.cvisiona.com
Diagram: Where a year of GPU spend goes at 5% use, by CVisiona.

Why GPUs sit idle on Kubernetes

Kubernetes was built for CPU and memory, which can be sliced finely. GPUs arrived as whole devices. For years, a pod could only ask for a number of GPUs, and the scheduler handed over entire cards. That is how the waste builds up:

  • A notebook or a small inference service takes a whole H100 and uses a fraction of it.
  • A distributed training job reserves GPUs and holds them while it waits for the rest of its pods.
  • Teams keep GPUs reserved just in case, and nobody else can borrow them.
  • Leftover capacity is spread across nodes in pieces too small for the next job.

None of this shows up as an error. The cluster looks healthy, the jobs finish, and the bill keeps growing.

The four layers AI startups use to fix it

The teams that raise their GPU use tend to add the same four layers, roughly in this order.

From idle GPUs to busy onesFour Kubernetes layers AI startups use to raise GPU use, in the order most teams add themSTEP 1Share each GPUTime-slicing or MIGSTEP 2Ask by attributesDynamic ResourceAllocation (DRA)STEP 3Queue and packKAI SchedulerSTEP 4Mix capacitySpot plus on demandSources: NVIDIA GPU Operator docs, Kubernetes docs, NVIDIA, AWScvisiona.com
Diagram: From idle GPUs to busy ones, by CVisiona.

1. Share each GPU

The NVIDIA GPU Operator can split one physical GPU in two ways. Time-slicing lets several pods take turns on the same card, which suits notebooks, tests and light inference. NVIDIA is clear about the trade off: unlike Multi-Instance GPU (MIG), “there is no memory or fault-isolation between replicas.” MIG instead cuts a supported card into hardware partitions, each with its own memory, which is the safer choice for production inference.

2. Ask for the GPU you need, not a count

Dynamic Resource Allocation (DRA) changes how workloads request devices. Its core APIs reached general availability in Kubernetes v1.34, and the Kubernetes documentation lists DRA as stable since v1.35. Instead of asking for one GPU, a pod references a resource claim that can filter devices by their attributes and can be shared by several pods. One limit to know: the Kubernetes scheduler does not support preemption for DRA resources.

3. Queue jobs and pack them tightly

NVIDIA open sourced the KAI Scheduler, the scheduler behind Run:ai, under the Apache 2.0 license in April 2025. It brings what GPU clouds do internally to any Kubernetes cluster: gang scheduling, so a training job starts only when all its pods can run, queues with a guaranteed share for each team, bin packing that fills partly used GPUs first, and consolidation that moves pods to free whole cards for bigger jobs. GPUs a team has reserved but is not using can be handed to others.

4. Mix capacity

Once jobs can wait in a queue, they no longer need the most expensive capacity. On the same DoiT page, the lowest Spot price for a p5.48xlarge in us-east-1 was $21.3246 an hour when we checked in November 2026, against $55.04 on demand. Spot capacity can be taken back, so it suits jobs that save checkpoints and restart, while serving stays on demand.

Where to start

For a startup, the order matters more than the tooling.

  1. Measure first. Track GPU use per team and per workload with NVIDIA’s DCGM metrics. DCGM-Exporter cannot tie metrics to individual containers when time-slicing is on, so measure before you share.
  2. Share the easy wins. Move notebooks, tests and small models onto time-sliced or MIG GPUs.
  3. Put training behind a queue. A scheduler such as KAI stops jobs from holding GPUs they cannot use yet.
  4. Then buy. Add capacity when the queue, not the habit, says you need it.

If Kubernetes still feels like a lot for a small team, our take on whether Kubernetes is too heavy for AI startups is a good place to start.

The takeaway

The GPU is the new server, and it is the most expensive one a startup has ever bought. The difference between 5% and 50% use is not a new model or a bigger round. It is scheduling. The startups that treat GPUs as a shared, queued and measured resource will stretch the same budget much further than the ones that keep buying cards to sit idle.

Sources & Credits

Text credit & news reporting source: Cast AI / VentureBeat / Kubernetes / NVIDIA / DoiT.

Enjoyed this breakdown? Stay ahead of the curve with our technical analyses and news. Join our community by subscribing to the CVisiona Newsletter for weekly deep dives, and subscribe to our CloudMelon Vis YouTube Channel for video breakdowns of Cloud-Native, AI, and Tech Enterprise news.

Question for you: what is the average GPU use in your cluster today, and which of the four layers would you add first? Tell us in the comments.

Written By

I'm an entrepreneur and creator, also a published author with 4 tech books on cloud computing and Kubernetes. I help tech entrepreneurs build and scale their AI business with cloud-native tech | Sub2 my newsletter : https://newsletter.cvisiona.com

Leave a Reply

Leave a Reply

error: Protect the unique creation from the owner !