For a decade, the server was the unit every startup counted. In AI startups, that unit is now the GPU. It is the line on the cloud bill that grows fastest, and the first one founders look at when the runway gets shorter.
Here is the uncomfortable part. Across tens of thousands of Kubernetes clusters, GPUs are busy only 5% of the time on average, according to the 2026 State of Kubernetes Optimization Report from Cast AI. CPU sits at 8% and memory at 20%. Most of what teams pay for simply waits.
That gap is where the $1M+ in the title comes from. And Kubernetes, the platform many AI startups already run on, now has the tools to close a large part of it.
Table of Contents
The 5% problem
Cast AI’s figure comes from its own telemetry, not a survey, so it describes the clusters it can see. Surveys point the same way. In VentureBeat’s Pulse survey, 79% of the 218 companies that reported their GPU use said their GPUs are busy half the time or less. And 70% of those companies still plan a new or additional AI compute platform within 12 months.
So the instinct is to buy more. The better first move is usually to use what is already paid for.
How idle GPUs turn into a $1M+ bill
Take a modest setup for training and inference: four AWS p5.48xlarge instances, each with eight NVIDIA H100 GPUs, so 32 GPUs in all. The on demand price in us-east-1 is $55.04 an hour per instance, as listed on DoiT’s live pricing page.
Run them around the clock for a year and the bill is $1,928,601.60. At the 5% average use Cast AI measured, $96,430.08 of that pays for real GPU work. The other $1,832,171.52 pays for GPUs that are switched on and waiting.
Your numbers will differ. Prices change by region, and a team with good habits will sit well above 5%. But the order of magnitude explains why compute waste has become a board level topic for AI startups, just like unchecked inference bills.
Why GPUs sit idle on Kubernetes
Kubernetes was built for CPU and memory, which can be sliced finely. GPUs arrived as whole devices. For years, a pod could only ask for a number of GPUs, and the scheduler handed over entire cards. That is how the waste builds up:
- A notebook or a small inference service takes a whole H100 and uses a fraction of it.
- A distributed training job reserves GPUs and holds them while it waits for the rest of its pods.
- Teams keep GPUs reserved just in case, and nobody else can borrow them.
- Leftover capacity is spread across nodes in pieces too small for the next job.
None of this shows up as an error. The cluster looks healthy, the jobs finish, and the bill keeps growing.
The four layers AI startups use to fix it
The teams that raise their GPU use tend to add the same four layers, roughly in this order.
1. Share each GPU
The NVIDIA GPU Operator can split one physical GPU in two ways. Time-slicing lets several pods take turns on the same card, which suits notebooks, tests and light inference. NVIDIA is clear about the trade off: unlike Multi-Instance GPU (MIG), “there is no memory or fault-isolation between replicas.” MIG instead cuts a supported card into hardware partitions, each with its own memory, which is the safer choice for production inference.
2. Ask for the GPU you need, not a count
Dynamic Resource Allocation (DRA) changes how workloads request devices. Its core APIs reached general availability in Kubernetes v1.34, and the Kubernetes documentation lists DRA as stable since v1.35. Instead of asking for one GPU, a pod references a resource claim that can filter devices by their attributes and can be shared by several pods. One limit to know: the Kubernetes scheduler does not support preemption for DRA resources.
3. Queue jobs and pack them tightly
NVIDIA open sourced the KAI Scheduler, the scheduler behind Run:ai, under the Apache 2.0 license in April 2025. It brings what GPU clouds do internally to any Kubernetes cluster: gang scheduling, so a training job starts only when all its pods can run, queues with a guaranteed share for each team, bin packing that fills partly used GPUs first, and consolidation that moves pods to free whole cards for bigger jobs. GPUs a team has reserved but is not using can be handed to others.
4. Mix capacity
Once jobs can wait in a queue, they no longer need the most expensive capacity. On the same DoiT page, the lowest Spot price for a p5.48xlarge in us-east-1 was $21.3246 an hour when we checked in November 2026, against $55.04 on demand. Spot capacity can be taken back, so it suits jobs that save checkpoints and restart, while serving stays on demand.
Where to start
For a startup, the order matters more than the tooling.
- Measure first. Track GPU use per team and per workload with NVIDIA’s DCGM metrics. DCGM-Exporter cannot tie metrics to individual containers when time-slicing is on, so measure before you share.
- Share the easy wins. Move notebooks, tests and small models onto time-sliced or MIG GPUs.
- Put training behind a queue. A scheduler such as KAI stops jobs from holding GPUs they cannot use yet.
- Then buy. Add capacity when the queue, not the habit, says you need it.
If Kubernetes still feels like a lot for a small team, our take on whether Kubernetes is too heavy for AI startups is a good place to start.
The takeaway
The GPU is the new server, and it is the most expensive one a startup has ever bought. The difference between 5% and 50% use is not a new model or a bigger round. It is scheduling. The startups that treat GPUs as a shared, queued and measured resource will stretch the same budget much further than the ones that keep buying cards to sit idle.
Sources & Credits
Text credit & news reporting source: Cast AI / VentureBeat / Kubernetes / NVIDIA / DoiT.
- Cast AI: 2026 State of Kubernetes Optimization Report
- VentureBeat: AI bubble update: 79% of companies say their GPUs sit idle at least half the time, yet most plan new AI platforms
- Kubernetes: Kubernetes v1.34: DRA has graduated to GA
- Kubernetes documentation: Dynamic Resource Allocation
- NVIDIA: NVIDIA Open Sources Run:ai Scheduler to Foster Community Collaboration
- NVIDIA documentation: Time-Slicing GPUs in Kubernetes
- DoiT: AWS EC2 p5.48xlarge Spot, On-Demand and Savings Plan Pricing in us-east-1
Enjoyed this breakdown? Stay ahead of the curve with our technical analyses and news. Join our community by subscribing to the CVisiona Newsletter for weekly deep dives, and subscribe to our CloudMelon Vis YouTube Channel for video breakdowns of Cloud-Native, AI, and Tech Enterprise news.
Question for you: what is the average GPU use in your cluster today, and which of the four layers would you add first? Tell us in the comments.
















The $48 Billion Bet That Every Engineer Will Manage AI Agents
Anthropic’s Record IPO Raised $81 Billion. Its Suppliers Own a Big Slice
Meta Wants $95 Billion From Investors. Its Own Chip Suppliers May Buy In
Your AI Agent Has the Keys. Who Decides What It Can Touch?
OpenAI Reportedly Wants Google’s AI Chips. Is Nvidia’s Biggest Customer Hedging?
Anthropic Reported an $80.7 Billion Loss. The Real Story Is a Single Share