Kubernetes readiness gates and rolling deploys without downtime

You ship a new version on a quiet Tuesday afternoon. kubectl rollout status reports success, every pod is Running and Ready, and the dashboards look fine. Then someone notices a small spike of 502s in the load balancer logs, lined up exactly with the deploy. Not enough to page anyone. Enough that a few hundred users got an error page for no reason. This happens on almost every cluster that has never been tuned for it. Kubernetes rolling updates are safe by default in the sense that they will not take your whole service down. They are not zero downtime by default. The gaps are at the edges of a pod's life: the moment a new pod starts taking traffic, and the moment an old one stops. I covered readiness and liveness probes themselves in an earlier article. This one is about everything around them that decides whether a rollout actually drops requests. A Deployment with the default RollingUpdate strategy replaces pods in steps, controlled by two settings: maxSurge: how many extra pods can exist above the desired count during the rollout maxUnavailable: how many pods can be missing from the desired count Both default to 25%. On a 4 replica Deployment that means Kubernetes can kill one old pod before its replacement is ready. If your service is sized to need all 4 pods under load, you just lost a quarter of your capacity mid deploy. For services that need full capacity the whole time, set it explicitly: spec: replicas: 4 minReadySeconds: 10 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 maxUnavailable: 0 means an old pod is only removed after a new one is available. minReadySeconds adds a short wait after a pod becomes Ready before it counts as available, which catches pods that pass their first readiness check and then crash a few seconds later. A new pod is added to the Service endpoints as soon as its readiness probe passes. Inside the cluster, that is usually good enough. Behind a cloud load balancer that sends traffic directly to pod IPs, it often is not. The load balancer runs its own health checks on its own schedule. The pod can be Ready in Kubernetes while the load balancer has not registered it as a healthy target yet. If the rollout moves fast, Kubernetes can terminate old pods that the load balancer is still relying on, while the new ones are not yet receiving traffic from it. For a short window, the load balancer has fewer healthy targets than you think, sometimes none. A readiness gate is an extra condition a pod must meet before Kubernetes considers it Ready. Something outside the pod sets that condition. You declare it in the pod spec: spec: readinessGates: - conditionType: "target-health.elbv2.k8s.aws/my-target-group" With a readiness gate in place, the pod is not Ready until both its readiness probe passes and the external controller marks the condition true. For load balancers, that controller marks it true only when the load balancer itself reports the pod as a healthy target. Combined with maxUnavailable: 0, the rollout cannot remove an old pod until its replacement is actually receiving traffic from the load balancer. You rarely write that YAML by hand. On AWS, the Load Balancer Controller injects the gate automatically for pods in a namespace labeled elbv2.k8s.aws/pod-readiness-gate-inject: enabled. On GKE, container native load balancing adds its own readiness gate for pods behind a network endpoint group. Check your load balancer controller's docs for the equivalent, then confirm with kubectl get pod -o yaml that the gate is actually there. When Kubernetes terminates a pod, two things start at roughly the same time. The kubelet sends SIGTERM to your container, and the control plane removes the pod from the Service endpoints. Those happen in parallel, not in order. Endpoint removal then has to propagate to kube-proxy on every node, to your ingress controller, and to any external load balancer. That takes time. Meanwhile your app received SIGTERM and may already have stopped accepting connections. Requests still being routed to it fail. That is where most rollout 502s come from. The fix is to delay shutdown slightly, so routing updates finish before the app stops listening: spec: terminationGracePeriodSeconds: 45 containers: - name: api lifecycle: preStop: exec: command: ["sh", "-c", "sleep 10"] The preStop hook runs before SIGTERM is sent. During those 10 seconds the pod keeps serving while the rest of the cluster stops sending it new requests. Then SIGTERM arrives, and your app drains in flight requests and exits. Two details matter here. The grace period includes the preStop time, so if you sleep 10 seconds and your app needs 20 to drain, the grace period has to be comfortably above 30. And exec needs a shell in the image. If you use distroless or scratch images, newer Kubernetes versions support a built in sleep action for preStop, so check what your cluster version offers instead of adding a shell just for this. The app side still has to handle SIGTERM properly: stop accepting new connections, finish what is in flight, close database pools, then exit. If the process ignores SIGTERM, Kubernetes waits out the whole grace period and then kills it, mid request. A PodDisruptionBudget limits how many pods can be taken down at once by voluntary disruptions: apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: api spec: minAvailable: 3 selector: matchLabels: app: api This matters during node drains, cluster upgrades, and autoscaler scale downs. Without it, draining a node can evict every replica of your service at the same moment if they happen to live together. The common misunderstanding is that a PDB also protects rollouts. It does not. A Deployment rollout follows maxSurge and maxUnavailable and ignores the PDB. You need both: rollout settings for deploys, a PDB for everything else that moves pods around. Watch every deploy until it finishes: kubectl rollout status deployment/api --timeout=5m If new pods never become Ready, the Deployment stops making progress and, after progressDeadlineSeconds (600 by default), reports ProgressDeadlineExceeded. It does not roll back on its own. Your pipeline should treat that as a failure and run kubectl rollout undo deployment/api, or your deploy tool's equivalent, instead of leaving a half finished rollout in place. maxUnavailable: 0 and a small maxSurge for services that need full capacity minReadySeconds so a pod that crashes right after becoming Ready does not count A readiness gate when traffic comes from an external load balancer straight to pods A preStop delay so routing updates finish before shutdown starts A grace period longer than the delay plus your real drain time An app that actually handles SIGTERM and drains A PodDisruptionBudget for drains and upgrades rollout status with a timeout in CI, and an automatic undo on failure Rolling updates keep your service up. They do not, on their own, keep every request alive. The 502s come from the edges of a pod's life, and every one of those edges has a setting. Fix the start with readiness gates and maxUnavailable: 0, fix the end with a preStop delay and real SIGTERM handling, and your next Tuesday deploy will be as quiet in the logs as it is on the dashboard. Kubernetes readiness and liveness probes: what they actually protect you from Structured logging: why print statements do not scale past one service I'm Amit Shukla, a software engineer working across DevOps, cloud and applied AI. Behind the Stack is where I write up the things I learn the hard way, with working code and honest tradeoffs. Instagram: a short, practical build every day at @amitshuklabag. Comment the keyword on any Reel and I'll send you the full guide. LinkedIn: connect with me for longer discussions. X: @amitshuklabag If this saved you some time, a reaction or a comment helps more people find it.