Editorial note: Most of the literature on inference batching sees the balance between latency and throughput as a technical challenge. However, the reality is that this tradeoff is all about prioritizing certain customers and certain workloads. In addition, the market is starting to price the tradeoff in commercial terms. Companies that use default scheduling policies are actually making commercial decisions without realizing this.
Every big language model service operates in its own mini-auction. All the time, a scheduling system determines who gets the next bit of GPU time and who needs to wait. Engineers designing those systems talk in the terms of performance: throughput, utilization, time to first token. However, the experience of that process is quite different: it is about getting a response from a support chatbot instantly or not, getting a document pipeline processed at night or by lunchtime, serving one customer before the others.
Or in other words, batching is product policy. And this is how the industry starts to understand it.
Why the GPU cannot simply process the incoming requests one by one
First of all, it’s the nature of the real requests. Demonstrations usually make it look like everything is neat: requests arrive periodically, prompts have comparable lengths, replies also have comparable lengths. In reality, everything is chaotic from all perspectives at once: somebody sends in a query which needs only a few tokens to answer, while somebody else sends in a lengthy document to be processed. Some of the requests come from the human being who waits for the reply, while others come from the batch job which will not be checked until tomorrow.
Processing such requests equally means losing processing power, and this loss grows because of the way the language model generates tokens. The generation process is autoregressive: every query generates its tokens sequentially and ends when it produces an end-of-sequence token.
The convoy effect created by static batching
Static batching is an artifact inherited from traditional machine learning workloads, where a set of requests is executed altogether, and the next batch starts after completing all requests in the first batch. Static batching works well for cases where the cost of processing each input is approximately the same. When applied to autoregressive generation, static batching leads to the convoy effect: the GPU is bound to the slowest request and cannot release their resources of shorter requests until the slowest one finishes executing. The hardware appears to be busy; however, an increasing portion of computing cycles becomes wasted on waiting. Increased variability leads to decreased throughput, increased latency, and underutilization of expensive hardware.
The obvious solution – increase the size of the batches – reveals the tradeoff that lies at the core of the field. Increasing the size of batches leads to higher utilization; however, each request should wait until the number of tasks sufficient to form a batch is reached, and starting the execution of a bigger batch may lead to increasing the time per output token (TPOT). The configuration that satisfies latency SLO at the smaller batch size may violate it at the larger batch size.
From batches to iterations
Modern serving engines have eliminated the "convoy effect" by changing how the scheduler manages units of data. The "continuous batching" method - introduced in the research system Orca as iteration-level scheduling, and now used in vLLM, Hugging Face’s Text Generation Inference, and SGLang - allows for scheduling decisions to be made at every step of the generation (decoding) process, rather than just once for an entire batch of requests. When a request is completed, it immediately leaves the system, and a waiting request can take its place without having to wait for the other requests in the batch to finish. Consequently, resource utilization is limited by the length of the queue and the amount of available GPU memory - required to store the state of each request (the so-called KV cache) - rather than by the execution time of the slowest request. The goal is no longer to create the largest possible batches, but rather to ensure that the GPU is constantly occupied with productive work.
The second optimisation concerns two distinct stages of inference. Pre-fill processes the entire prompt in a single forward pass, so it is compute-bound, while the decode stage generates one token after another and thus is memory-bandwidth-bound. Because prefill needs so much compute, a single long prompt can monopolise compute and stall users who are already receiving responses. Chunked prefill splits a long prompt into smaller pieces and processes them alongside ongoing decode work in the same iteration. This lets operators balance time to first token (TTFT) for new requests against time per output token (TPOT) for requests already being served.
Observe the trend here. Every new development made a static characteristic adjustable – and adjustability poses a question about who gets to adjust it.
A scheduling policy engine
With scheduling being done on a token-by-token basis, rather than a per-request basis, the GPU resources can become a shared resource that can be managed according to specific rules, instead of being distributed on a first-come, first-served basis. Interactively used sessions, batch jobs, highly prioritised accounts and less important workloads can all be treated differently, and should be. A business process that allows scheduling should be more interested in maximising throughput than in response times.
This aspect has been factored into commercial pricing models. Large model hosting providers are offering a discounted version of their asynchronous batch APIs that offer results within a window of up to 24 hours rather than in seconds, and several of them have separate tiers for latency-sensitive and cost-sensitive traffic. This is a pricing model based on scheduling policies.
An enterprise running their own inference services faces the same dilemma internally, whether or not anyone calls it out explicitly: when every workload goes into the same queue, the scheduler has quietly decided that the overnight reports and the customer-facing assistant should carry the same importance.
Admission control is when the policy becomes quite explicit. Extra GPU capacity doesn't equate to extra capacity in terms of latency. Allowing every incoming request to enter the system may lead to an increase in utilisation rates, as well as to exceeding every user's SLOs. A well-designed scheduler will sometimes say "not now" – will postpone some work in order to keep the latency under control for the admitted users.
The architecture is splitting along the same line
The hardware is doing the same thing. Systems like DistServe and Splitwise suggested running prefill and decode on their respective pools of GPUs because each task is limited by a different bottleneck, with one being compute intensive and the other memory bandwidth intensive. This concept has gone from theory to practice; since March 2026, NVIDIA’s Dynamo framework uses separate pools of GPUs for the two tasks, and serving engines like vLLM and SGLang now also support this split. The reason behind this is the main premise of this article: different jobs require different handling.
Define your goals
There is no one-size-fits-all solution here. Appropriate batch sizes, scheduling algorithms and admission thresholds vary based on the task at hand and its latency commitments. There is no reason why a nightly data pipeline should be engineered according to the same principles as a coding assistant and vice versa. Or for that matter that a chatbot for support should share the same scheduler as an offline document processor if both of them use the same foundation model. The model might stay the same. The way it is served, however, should not.
The ultimate goal is to achieve the maximum throughput that is possible within SLOs of individual applications. This is an engineering task. What these SLOs actually are – is a question of business priorities. Engineering teams who consciously set these priorities and reflect them in their schedulers will get more out of the same hardware than those that rely on default parameters only. But even the latter make the decision too. They just fail to realize what it is.
