Title

Title vLLM PagedAttention KV Cache Corruption: Woke Up to This Nightmare Image generated via Midjourney by author But tbh I dont even know where to start. So like I was on call and at 8am my phone starts blowing up. Incident alert. Peak RPS was at 14720. And ngl I freaked out a bit. Because what even is that. And so I jump into logs and see this crazy error message. Like wtf is going on here? But ok so lets get into it. graph LR A[vLLM] ,>|requests|> B[PagedAttention] B ,>|cache query|> C[KV Store] C ,>|corrupted response|> B B ,>|error|> A Because we use vLLM with paged attention for our model serving. And lol its been working great till now. Edit: Wait, I was wrong about the batch size above. It was 32, not 16. But today because of this cache corruption issue were seeing crazy errors like this: And when we try to access the KV store we get: try: kv_store.get(key) except Exception as e: print(f"Error accessing KV store: {e}") And idk whats going on but the error log says: Traceback (most recent call last): File "model_serving.py", line 123, in serve_model response = paged_attention.query(cache_key) File "paged_attention.py", line 45, in query value = kv_store.get(cache_key) File "kv_store.py", line 23, in get raise Exception("Cache corruption detected") Exception: Cache corruption detected Because of this stupid cache corruption issue were down and idk how long its gonna take to fix. But so first thing I did was jump into the codebase and start debugging. And lol its always something simple right? And now were pushing a fix and hoping itll resolve the issue. Before I found the real issue, I tried 3 other fixes that failed: Bumping timeouts - Changed NCCL_TIMEOUT=1800 in the env. Did nothing. Still failed at 8am. Restarting pods - kubectl rollout restart deployment/vllm. Came back up, same error. Wasted 10 mins. Checking GPU health - nvidia-smi showed all GPUs fine. I was convinced it was hardware tbh. Spent 45 mins going down wrong paths. The fix was 1 line in Dockerfile. Im an idiot. Because this sucked, we added 3 grafana alerts so Marcus never gets paged for this again: # Alert if NCCL comms thread fails rate(nccl_errors_total) > 0 # Alert if all_reduce latency > 50ms histogram_quantile(0.99, nccl_allreduce_duration_seconds_bucket) > 0.05 Now if this breaks, pagerduty wakes us up before users notice. Q: Why not use Gloo backend? Q: Could this happen on single-node? Q: Do I need to update CUDA too? Full code: https://github.com/yourorg/voygr-vllm-pagedattention-kv-cache-corruption-debug Artificial Intelligence, Machine Learning, Data Science, Deep Learning, Programming, Software Engineering