Resource limits and requests, on every workload
A cluster with no resource requests/limits set is one noisy workload away from starving everything else on the node. This is the single most common cause of "random" production incidents in Kubernetes environments we've audited.
- •Every deployment has CPU/memory requests and limits set, based on real usage, not defaults copied from a tutorial.
- •Pod Disruption Budgets are defined for anything that can't tolerate simultaneous restarts.
- •Horizontal Pod Autoscaler is configured for workloads with variable load, tested under actual traffic patterns.
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"Security baseline
Kubernetes' default posture is permissive. Hardening isn't optional for anything handling real traffic or data.
- •RBAC scoped to least privilege, no workloads running with cluster-admin by default.
- •Network policies restrict pod-to-pod traffic to what's actually required.
- •Pod security standards enforced (no privileged containers, no host namespace access unless explicitly required).
- •Container images scanned for known vulnerabilities before deployment, not after an incident.
Operational readiness
The gap between "it works in staging" and "it survives production" is almost always operational, not architectural.
- •Liveness and readiness probes configured correctly: a missing readiness probe routes traffic to pods that aren't ready yet.
- •Centralized logging and metrics wired up before launch, not added after the first incident.
- •A tested rollback path for deployments. You should know how long a rollback takes before you need one under pressure.
- •Node autoscaling limits set deliberately, so a runaway workload can't scale your bill along with your pod count.
Autoscaling that doesn't fight itself
HPA, VPA, and the cluster autoscaler are often configured independently by whoever set each one up, and they can work against each other: HPA scaling pods out while VPA tries to resize them, or the cluster autoscaler removing a node HPA just scaled onto. Treat autoscaling as one system, not three separate settings.
- •Don't run HPA and VPA on CPU/memory for the same workload simultaneously. They'll fight over the same signal. Use VPA in recommendation-only mode if you also have HPA active.
- •Set HPA target utilization with headroom for scale-up latency. If pods take 30 seconds to become ready, scaling at 90% utilization is already too late.
- •Cluster autoscaler node group sizing should account for pod scheduling constraints (affinity, taints): a technically "available" node that nothing can actually schedule onto doesn't help.
- •Test scale-down behavior deliberately, not just scale-up. Premature node removal under a PDB violation is a common source of brief outages during low-traffic windows.
Multi-tenancy and namespace isolation
Most production incidents in shared clusters come from one team's workload affecting another's, not from external attacks. If multiple teams or environments share a cluster, isolation has to be deliberate.
- •ResourceQuotas set per namespace so one team can't consume the whole cluster's capacity, intentionally or by accident.
- •NetworkPolicies default-deny between namespaces, with explicit allow rules for the traffic that's actually required.
- •LimitRanges set per namespace as a backstop for workloads that ship without their own resource requests/limits.
- •Separate node pools (or taints/tolerations) for workloads with meaningfully different risk profiles. A shared batch-processing namespace probably shouldn't share nodes with customer-facing services.
Cost-aware scheduling
Kubernetes will happily schedule workloads in the most expensive way possible if nothing tells it not to. Cost and scheduling aren't separate concerns once the cluster is running real workloads.
- •Fault-tolerant, interruptible workloads (batch jobs, CI runners, non-critical background processing) run on spot/preemptible node pools, not on-demand.
- •PriorityClasses set so critical workloads preempt lower-priority ones under resource pressure, instead of everything competing equally.
- •Bin-packing considered explicitly: several small underutilized nodes cost more than fewer, well-packed ones, but over-packing removes headroom for spikes.
- •Idle namespaces and dev/staging clusters scaled to zero or torn down outside working hours, the same discipline applied to any other cloud resource.
- Do these apply the same way to managed Kubernetes (EKS/AKS/GKE)?
- Mostly yes. RBAC, resource limits, and probes are workload-level concerns regardless of who runs the control plane. A managed service handles control-plane HA for you, but hardening the workloads running on it is still on you.
- Is a Pod Disruption Budget necessary for a single-replica deployment?
- Less critical when there's only one replica, since there's nothing to protect during a voluntary disruption. Add it once you scale to multiple replicas.
Related service
Managed Kubernetes
Production-grade Kubernetes, hardened and operated, without hiring a platform team.
View service →