
TL;DR: We rethink the role of pipeline parallelism (PP) in agentic serving. PP is usually considered less efficient than tensor parallelism (TP) and data-expert parallelism (DEP) due to its bubbles. We find that PP, in fact, is a good fit for agentic prefill, with long contexts and TTFT targets of a few seconds.
By shrinking bubbles with a copy-engine stage hand-off and async scheduling, PP outperforms TP and DEP, which synchronize their GPUs in every layer. Serving MiniMax-M3.1-Flash on two NVIDIA B300 GPUs with vLLM, PP2 sustains 23% more requests per GPU than DEP2 and 17% more than TP2 under a 2 s p50 TTFT target.
This post shares our findings on Pipeline Parallelism:
- PP sustains the highest rate for p50 TTFT targets of a few seconds; TP leads only under a low latency regime.
- Under those targets, PP is more efficient than TP and DEP because TP pays a fixed communication cost in every layer and DEP pays for rank imbalance below saturation.
- PP2 ties DEP2 within 1% at saturation and has lower p50 TTFT at every rate below it, because the copy-engine hand-off and async scheduling shrink PP’s bubbles and the remaining bubbles shrink with load.
Background: prefill is usually served with TP or DEP
Figure 1 shows the three common parallelisms for serving an LLM on N GPUs, here N = 2:
- Tensor parallelism (TP) splits every weight matrix across the GPUs. All GPUs work on every token and all-reduce their partial results twice per layer.
- Data-expert parallelism (DEP), for MoE models, runs one engine, called a rank, on each GPU. Each rank serves its own requests with a full copy of the attention and dense weights and 1/N of each MoE layer’s experts, and exchanges tokens with the other ranks in every MoE layer.
- Pipeline parallelism (PP) gives each GPU 1/N of the layers, called a stage. Hidden states move to the next stage once per batch, and N batches in flight keep all N GPUs busy.
Figure 1: What each GPU holds in one prefill instance on two GPUs. TP2 is drawn as plain TP. Its best config, TEP2, instead gives each GPU 64 whole experts per MoE layer and exchanges tokens inside MegaMoE (day-0 post), then all-reduces the two halves.
For prefill, the usual choices are TP and DEP, as recommended by the vLLM recipe for MiniMax-M3.1-Flash. PP is rarely used outside models that do not fit in one node, because it does not cut a request’s latency and because its stages sit idle, in pipeline bubbles, when few batches are in flight.
Our thesis: PP fits agentic prefill
For agentic prefill, a TTFT of a few seconds is enough
Agentic prefill weakens both objections. Each agent turn adds several thousand tokens to a long, mostly cached context and then spends seconds decoding and calling tools, so a TTFT target of a few seconds is enough. PP’s extra latency fits easily within it: 0.45 s against TP2’s 0.31 s for a 110K-token prompt at light load. What remains is throughput: how many requests per GPU each parallelism sustains under the target, and how small PP’s bubbles can be made.
PP synchronizes once per batch, versus TP and DEP in every layer
Figure 2: One prefill step over time in each parallelism. TP2 is drawn as plain TP, as in Figure 1.
A large prefill step holds thousands of tokens, so it reads every expert of every MoE layer. All three parallelisms therefore read the same weights per step. They differ in how often their GPUs synchronize, that is, wait for each other, and in what makes each wait long (Figure 2).
TP synchronizes most often, in every layer and at every load, in exchange for the lowest latency. DEP waits whenever one rank has fewer tokens than the other. PP synchronizes once per batch at each stage boundary, and with an asynchronous copy-engine hand-off, a stage waits only when the pipeline is not full. Under a TTFT target of a few seconds, PP therefore keeps its GPUs busiest.
Results: PP2 serves the most requests per GPU on MiniMax-M3.1-Flash
To test whether fewer synchronization points turn into more requests per GPU, we serve MiniMax-M3.1-Flash with vLLM on two B300 GPUs, the smallest instance that holds its 250 GB checkpoint. The workload follows agentic prefill: every request shares a 100K-token cached prefix, adds 7.5K–12.5K new tokens, and arrives as a Poisson process. For each p50 TTFT target, Figure 3 and the table give the highest request rate per GPU that each parallelism sustains.
Figure 3: p50 TTFT against offered requests per second per GPU, the median of three runs per rate. Whiskers span the runs, and a toggle shows p90.
Rates are in requests per second per GPU, and — means p50 TTFT exceeded 1 s even at the lowest rate tested, 1.75. In the 3 s and 5 s rows, TP2 and PP2 exceed their saturation rates, so we compare the parallelisms only at 1.5–2.5 s.
- Only the tightest target favors TP2. TP2 has the lowest TTFT at light and moderate load, so it leads at 1 s. PP2 leads from 1.5 s to 2.5 s: by one rate step at 1.5 s, and at 2 s by 23% over DEP2 and 17% over TP2.
- At saturation, PP2 and DEP2 tie at 3.60 and 3.57 requests per second per GPU. No service can run there for long, because the queue and TTFT grow without bound.
- At light load, TP2 is fastest, likely because both of its GPUs work on every request. We did not trace light load to confirm this.
Why PP2 Leads: where TP2 and DEP2 lose time
We traced each parallelism with nsys to see where its GPU time goes (Figure 4). The traces show what holds TP2 and DEP2 back. PP2’s pipeline bubbles are the subject of the next section.
TP2 pays for communication
Figure 4: GPU time per computed token, summed over both GPUs, by kernel category, in full 32,768-token steps.
In full steps, TP2 used 34.0 µs of GPU time per computed token, against 27.8 µs for DEP2 and 27.2 µs for PP2. The extra time is communication: the all-reduces after attention and after every MoE layer, with the copies that stage them, took 6.3 µs per token, or 18% of TP2’s GPU time. Each all-reduce carries the whole step’s hidden states and sits on the critical path, so in this setup it does not overlap compute.
Sequence parallelism could remove part of this cost; the recipe offers it for prefill only on MI355X, and we did not test it.
DEP2 waits when its ranks get uneven work
Each DEP2 rank builds its steps from its own requests, but the ranks synchronize in every MoE layer, so each step lasts as long as the heavier rank. MegaMoE overlaps most of the token exchange with its expert GEMMs, so for DEP2 the waiting matters more than the bytes. A rank’s work is its new tokens plus the cached context they attend to. Cached context adds little: MiniMax Sparse Attention reads at most 16 blocks per query, and in our workload every request shares the same 100K prefix, so it adds no imbalance. New tokens are where the ranks differ, for three reasons:
- The router counts requests. Our benchmarks used vLLM’s built-in data-parallel load balancer, which sends each new request to the rank with the fewest queued and running requests. Serving routers also count requests: round-robin is NVIDIA Dynamo’s default router mode, and Dynamo and Shepherd Model Gateway (SMG) offer policies that pick by in-flight requests.
- A request cannot be split across ranks. With few requests in flight, one rank often has none. It then runs an empty batch so that its peer can proceed, and no router can give it work.
- Each rank chunks its own requests. Each rank applies the 32,768-token step budget and the 8,192-token per-request cap to its own queue, so the two ranks cut their steps at different points.
Below saturation, uneven steps are the common case. At 2.20 requests per second per GPU, a load every parallelism kept up with, only 8% of DEP2’s steps had the two ranks within 1.2× of each other, and in 38% one rank had no tokens. Waiting for the other rank, and MegaMoE running slowly beside an idle rank, took 37% of DEP2’s busy GPU time. This single capture may overstate the cost, because the profiler raised p50 TTFT to 1.70 s from 1.3–1.5 s. At 3.0, fuller steps cut the steps with an idle rank to 14% and the cost to 20%. To isolate the effect, we kept six requests in flight and pinned how many went to each DEP2 rank:
With an even split, DEP2 beats PP2 at this load, but a 4:2 split already puts it 8% behind, and the loss grows with the split. TP2 delivered the most, in line with its low light-load TTFT.
Each split ran once, on random-token prompts. This is our reading of why DEP2’s p50 TTFT was above PP2’s at every rate we tested, from 0.62 s against 0.45 s at light load, although on paper DEP2 should be faster: its attention runs on one GPU and its MoE on both, while PP2 runs each request through one stage after the other. PP2 has one scheduler for both stages, so there is no split between ranks to get wrong.
DEP can be balanced better than by counting requests. DeepSeek’s DeepInfer deployment notes describe a prefill balancer that routes each request to the rank with the least estimated work, and serving routers such as Dynamo and SMG offer token-aware policies for vLLM’s data-parallel ranks. Such policies even out ranks that both have work, but cannot help a rank with no request. We tested none of them, and PP2 needs none.
Where PP2 loses time, and how we shrink it
Pipeline bubbles shrink with load
A pipeline bubble is time a stage waits for the other, idle or blocked in a send or receive. vLLM runs one scheduler per PP instance and, with asynchronous scheduling (the default here), keeps up to three batches in flight for PP2, so the next chunk of a long prompt enters stage 0 while the previous chunk is still in stage 1.
Bubbles appear when the queue is shallow: too few batches to fill both stages, consecutive batches of different sizes, or stage 0 blocked until stage 1 receives the previous send. At 2.20 requests per second per GPU, each stage sat idle about 29% of the time while the other computed, and at 3.0 about 15%. At 2.20, both stages computed at once only 33% of the time, and for 19% of it only one batch was in flight.
The copy-engine stage hand-off
We moved the stage hand-off from NCCL to CUDA copy engines, synchronized on the GPU, and are upstreaming the change (Figure 5). It helps in two ways:
- A one-sided push. An NCCL send is a rendezvous: stage 0’s next forward waits until stage 1 has received the previous batch, so a large batch on stage 1 also stalls stage 0. With copy engines, stage 0 writes into stage 1's receive buffer as soon as a GPU-side flag shows it is free, moves on, and waits only when stage 1 falls more than about a batch behind.
- No SMs. NCCL's send runs as a kernel beside the next batch’s compute. Many persistent kernels size their grid to the full SM count, which is wrong while another kernel holds some SMs. A copy engine leaves every SM to compute.
Figure 5: The same four batches through PP2’s stages with each hand-off. Stage 1’s initial fill, and its idle time when too few batches are in flight, stay the same.
We measured the change alone on the day-0 image vllm/vllm-openai:minimax-m31-flash, running both variants at once on separate GPU pairs and rotating them over the pairs in three rounds (seeds 42, 43 and 44), with the same accuracy checks. The hand-off cuts waiting, not compute: p50 TTFT falls at every load, and the saturated rate does not change.
At saturation, PP2 and DEP2 tie within 1%
At saturation, DEP2 and PP2 both run full steps and rarely wait. DEP2’s ranks waited for each other for 1.1% of its GPU time (a lower bound). With the even 30,30 split, PP2’s stage 0 took 3.8% longer per batch than stage 1, leaving stage 1 idle 4% of its time, or 2% of PP2’s GPU time. Moving one layer, about 3% of a stage, would gain under 0.5%. We did not test another split. The 1.05 ms hand-off per full batch ran on the copy engines alongside stage 0’s next batch. DEP2 and PP2 therefore spend about the same GPU time per token, and their saturated rates are within 1%.
Limits: what PP costs and what we have not measured
PP can add latency at light load and complicates P/D disaggregation
- One request may take longer at light load. A request runs through stage 0 and then stage 1, which is likely why TP2, with both GPUs on every request, had the lowest light-load TTFT.
- P/D disaggregation needs more work. A PP prefill instance splits each request’s KV cache across its stages, so the decode side must read KV from every stage. We are upstreaming P/D with a PP prefill for MiniMax-M3.1.
With more GPUs per instance, we expect the same trade-off
We measured only two-GPU instances, where each parallelism is in its plain form. With N GPUs per instance, we expect:
- TP: each GPU’s compute shrinks as 1/N, but every all-reduce still carries the step’s full hidden states, so communication takes a larger share. TP should keep its light-load lead.
- DEP: each MoE layer waits for the slowest of N ranks, so uneven work costs more as N grows unless something balances it.
- PP: filling N stages takes N batches in flight, so bubbles grow with N below saturation, and a request’s latency does not fall.
Hybrids, such as TP inside each PP stage, need four or more GPUs and should fall in between.
We expect the trade-off to keep its shape, with TP for the lowest latency and PP or a well-balanced DEP for throughput under a TTFT target. Next, we will measure where the crossover falls at four and eight GPUs.
Practical recommendations for serving
- Tightest TTFT targets and light load: TP. It had the lowest TTFT at light and moderate load, but sustained the highest rate only at a 1 s p50 target.
- TTFT targets of a few seconds: PP. It synchronizes once per batch instead of in every layer, and one scheduler builds every batch. At p50 targets of 1.5–2.5 s, it sustained up to 23% more requests per GPU.
- Throughput with no TTFT target: DEP or PP. At saturation, DEP2 with MegaMoE matched PP2 within 1%, but no service can run there for long.
- If you serve with DEP, balance its ranks by work. Counting requests leaves the ranks uneven below saturation.
- More GPUs per instance: not measured yet. “Limits and next steps” lists what we expect to change.
- Other models: measure, with PP included. Attention cost, expert sizes and context lengths all shift the balance between the parallelisms.
Acknowledgements
We thank the MiniMax team for open-sourcing MiniMax-M3.1-Flash; the FlashInfer, DeepGEMM and NCCL teams for the kernels and libraries used here; and the vLLM contributors who built pipeline parallelism and MiniMax-M3 support in vLLM.
We do this kind of research in the open and contribute the results to vLLM. Each improvement makes vLLM faster for everyone who serves with it, and because the Inferact platform runs on vLLM, the same improvements reach our production serving.


