AI21 traced a 1-in-1,000 gibberish bug to one wrong data type

A rare, silent failure in vLLM serving AI21's Jamba model produced confident gibberish with no crash or error. Two engineers found it wasn't the kernels lying, it was the scheduler and a 32-bit index quietly overflowing — and the fixes were a scheduler flag and a single type change.

• The symptom — Roughly once in a thousand prompts, Jamba served on vLLM returned high-confidence gibberish with no crash, no warning, and no error, occurring only under load and only in vLLM, not other inference frameworks.
• Forcing reproduction — Unable to reproduce it by sending prompts normally, Asaf Gardin and Yuval Belfer dropped vLLM's GPU memory utilization flag from 90% to 20%, ran many simultaneous requests at temperature zero, and got request 8,854 to deterministically return gibberish.
• Baseline check — They used Hugging Face Transformers' plain Mamba kernel implementation as a reference, comparing log probs from vLLM's generation against a prefill-only forward pass through softmax to measure token-distribution divergence.
• First false lead — They inspected the CUDA prefill kernel for Mamba and ran Nvidia's compute sanitizer for out-of-bounds memory; both looked clean, so they isolated decode kernels by forcing everything through prefill, and the gibberish vanished — pointing wrongly at decode kernels.
• Adding identity — Because tensors inside the forward pass have no identity, they threaded the request ID into a class called forward context down to Mamba's prefill and decode kernel calls, then set a breakpoint on the offending request ID.
• The real bug — The breakpoint showed the scheduler running decode before prefill for that request; attention tolerates this because it writes KV before reading it, but Mamba reads its state first, so it computed over stale data left by prior requests.
• The fix — The fix made the scheduler mark any request whose tokens were never computed as prefill rather than decode or chunked, and the change was merged into vLLM.
• Second bug appears — During GRPO-style RL post-training on the same hybrid Jamba architecture, logprob spikes appeared between rollout and the FSDP step, before any weight update, recurring deterministically every 12 steps.
• Scaling to reproduce — Increasing rollouts per prompt from the default 8 to 16, 32, 64, and 128 moved the failure earlier each time; at 128 rollouts the spike hit immediately at step one instead of waiting for step 12 or 24.
• Wrong lever — Reducing GPU memory utilization from 0.9 to 0.2, the same trick that worked for bug one, made this issue disappear instead of reproducing it — a misleading signal.
• Root cause — The actual cause was a 32-bit unsigned integer index in the Mamba kernel's cache pointer that silently wrapped around after passing roughly 4 billion, rather than throwing an error; shrinking memory allocated a smaller state buffer that never reached that offset.
• The one-line fix — The fix changed a single variable's data type from uint32 to size_t, which on most modern architectures becomes a 64-bit unsigned integer large enough that the overflow never occurs.
• Takeaways — Both bugs shared the same signature — silent failures around the Mamba state cache, surfaced by memory pressure, diagnosed via logprob comparisons against a baseline framework — leading to advice to reproduce under memory pressure, vary knobs to see what shifts failure timing, and thread identity through stateless tensors.

In their words

we reduced it from 90% to 20%. And once we did that and then we started uh running a lot of requests uh simultaneously um all of a sudden request number let's say 8 854 suddenly returned gibberish4:34

So the kernels weren't doing the wrong thing, they were called at the wrong time for the wrong requests.10:13

we were like okay it's got to be the decode kernels. But you know how it is in software you get excited too quickly and then you figure out it's not what happened.8:01

we noticed that Mamba kernels um used um in 32 unsigned in 32 index pattern uh pointer. So, once the offset went past some you know 4 billion um uh numbers, it wrapped around instead of throwing an error.14:22

Disclosure · The speakers work at AI21 and are describing debugging work on their own Jamba model, and point viewers to AI21's blog writeup, which functions as promotion of their engineering work.

One thing to add — One thing to add — the striking pattern here is that the same debugging lever (shrinking GPU memory) exposed one bug and masked the other, a useful caution against treating any single reproduction trick as universally reliable. The talk is light on how long each investigation actually took, so the clean narrative arc likely compresses a messier real timeline.

One thing to try tonight
Pull up vLLM's GPU memory utilization flag in your own deployment and sweep it from 0.9 down to 0.2 while running a batch at temperature zero to see if any output changes deterministically.