Kubernetes Cost Optimization: Lessons from Real Production Incidents

Most Kubernetes cost-optimization advice reads like a checklist someone wrote without ever being paged. "Right-size your requests. Use HPA. Enable cluster autoscaler." Sure. The interesting failures all happen after you've done those things. I'll share two incidents from a large-scale financial services environment that shaped how I think about this now.

The silent over-provisioning that ran for months

A platform team set CPU requests for a fleet of stateless workers at 2000m based on a load test from when the service was first launched. The service was rewritten 14 months later — same name, completely different hot path. Actual P95 CPU usage in production: roughly 180m. The requests never moved.

The cluster autoscaler dutifully kept provisioning nodes large enough to satisfy the request, not the usage. Multiply by ~600 pods across three regions and we were paying for an entire node pool that did nothing. Nobody noticed because every dashboard the team looked at — utilization, throughput, error rate — looked healthy. Of course they looked healthy. The pods had 11x the CPU they needed.

The cost tool eventually surfaced it, but the cost tool was a lagging indicator measured in months. The actually-useful query was sitting in Prometheus the whole time:

(
  sum by (namespace, container) (
    rate(container_cpu_usage_seconds_total{container!="",container!="POD"}[7d])
  )
)
/
(
  sum by (namespace, container) (
    kube_pod_container_resource_requests{resource="cpu"}
  )
)

Any container whose 7-day ratio sits below 0.2 for more than a couple of weeks is over-provisioned by definition. Alert on it. Or at minimum, dashboard it next to your cost view so the two stories line up.

[IMAGE: Diagram of resource requests vs actual usage, showing the "waste gap"]

The tradeoff nobody wants to talk about

Once you find the waste, the temptation is to crush requests down to P95 usage and let bin-packing do its job. This works beautifully in steady state and breaks during traffic spikes. Aggressive bin-packing means there's no slack on a node when a noisy neighbor suddenly needs more CPU. The kernel scheduler does its best, P99 latency spikes, the upstream load balancer's circuit breaker trips, and now you have an availability incident caused by a cost optimization.

The honest framing is that requests are a reliability margin, not a capacity number. You're paying for the right to be wrong about peak. How much you pay should be a deliberate decision per service tier, not a side effect of whoever last edited the Helm chart.

What I do now: critical-path services keep requests at roughly P99 + 30%. Batch and async workers go to P95 + 10%. Dev and ephemeral environments go to P50. Three numbers, three tiers, written down somewhere the team can point to during code review.

[IMAGE: Simple chart showing cost vs reliability tradeoff curve]

The autoscaler misconfiguration

The second incident: a node group had scale-down-utilization-threshold set aggressively low (0.3) and scale-down-unneeded-time set to 5 minutes. Over a weekend, traffic dropped, the autoscaler scaled the node group from 22 nodes to 4. Monday morning's traffic ramp arrived faster than nodes could be provisioned — image pulls alone took 90 seconds — and we served degraded responses for about 12 minutes while the cluster caught up.

The fix wasn't to disable scale-down. It was to set scale-down-unneeded-time to 30 minutes and add a minimum node floor based on weekday-morning traffic, not weekend lows. Cost went up maybe 4%. The Monday morning surprise stopped happening.

Where I'd push back on the standard advice

"Use a Kubernetes cost tool" is necessary but nowhere near sufficient. Cost tools tell you what is expensive. They don't tell you whether the expense bought you reliability margin you actually need. That judgment lives with the engineers, not the tool, and it should be revisited per service every quarter — not chased as a continuous metric to drive down to zero.

Aggressive cost optimization done blind is one of the fastest ways I've seen to manufacture an availability incident. Which, in money terms, costs more than the savings.

Related reading: The Hidden Cost of Observability Tool Sprawl and more DevOps posts on DevNfra. The Kubernetes resource management docs are the canonical reference for the requests/limits semantics this whole post depends on.