What a Cluster-Wide Overcommit Taught Me About Multi-Tenant Kubernetes

A multi-tenant Kubernetes cluster can report healthy at the aggregate level while individual namespaces are already over-allocated. That’s the failure mode that showed up on a platform running 500+ Kubernetes deployments across 3 availability zones: cluster-wide metrics looked fine, but capacity allocation had drifted from real usage as the namespace count grew past 100.

Aggregate health metrics hide per-tenant drift

The instinct when cluster metrics look fine is to assume capacity planning is fine too. It isn’t necessarily. Resource requests and limits accumulate per namespace over time, and a cluster-wide view averages that drift away. The overcommit only became visible once usage was audited namespace by namespace.

A blanket fix would have masked the problem, not solved it

The tempting fix is a cluster-wide capacity bump — it’s fast and it makes the symptom go away immediately. It also doesn’t touch the actual cause: allocation that no longer matches usage at the tenant level. That version of the fix would have resurfaced as the platform kept growing. Right-sizing per namespace took longer, but it addressed the source.

Autoscaling and capacity planning are different problems

KEDA handled event-driven autoscaling for workload demand throughout this, and it kept working the whole time — autoscaling wasn’t the issue. Capacity allocation was. It’s easy to assume a good autoscaler means capacity planning takes care of itself; it doesn’t, because autoscaling responds to load, not to whether the baseline requests and limits were ever correct.

What mattered more than the tooling

Prometheus, Grafana, and CloudWatch gave visibility into the cluster, but the fix itself came from tracing the numbers back to individual namespaces, not from a new monitoring tool. The tooling made the investigation possible; the per-namespace audit is what made the fix hold.