The case for the argument
The accelerator is the most fully described piece of hardware in the system and the least descriptive about the behavior of the system. Datasheets will tell you the peak tensor throughput and HBM bandwidths. They will not tell you what proportion of the GPU hours spent yielded a gradient step and an output generated within its latency goal. The proportion is set by five layers around the accelerator. These include the scheduler, the interconnect, the data path, the serving stack, and the failure domain.
My hypothesis is that the failure domain is the least underestimated among the five, and the other four layers can best be analyzed through the failure domain. Synchronous training converts component reliability to the reliability of jobs by multiplication; hence, the other layers are best described by the following question: How much work gets lost when something fails, and for how long? Schedulers, fabrics and storage systems are normally measured in terms of their performance in steady state. In terms of the scale where GPU usage peaks, steady state is not the determining factor.
Scheduling: placement is destiny
Training of both data-parallel and model-parallel jobs requires gang scheduling: either all ranks are scheduled at once or no ranks at all, since partially placed job consumes accelerators without doing anything useful. Generic container schedulers are not meant to do this kind of work. Upstream kube-scheduler schedules pods individually. Only recently, Kubernetes got all-or-nothing placement via the PodGroup and Workload APIs, with gang scheduling described as beta feature in v1.37. It is also off-by-default. In reality, the gap is being filled by batch scheduling layers: topology-aware scheduling of Kueue accepts a workload if its pods fit the required or preferred topology domain, whereas Slurm has offered а hierarchical view on switches through its topology plugin for a long time.
The importance of topology stems from the fact that the bandwidth is not the same everywhere. The Megatron-LM scaling study leaves tensor parallelism within a single NVLink-connected machine since its all-reduces are too frequent and too latency sensitive to happen on the scale-out network. Alibaba’s HPN publication describes the same disparity as intra-host bandwidth in hundreds of gigabytes per second versus inter-host links in hundreds of gigabits. Schedulers that ignore the NVLink domain, PCIe/NUMA affinity between GPU and NIC, or the leaf switch boundary transform fast collectives to slow.
Fragmentation is the cost of getting this right. Microsoft’s analysis of its Philly cluster found that insisting on locality for gang-scheduled jobs trades directly against queueing delay. Meta’s research-cluster study shows why the tension never goes away: more than 40% of jobs used a single GPU, yet jobs of 256 or more GPUs consumed over 66% of GPU time on one cluster. Small jobs backfill the gaps and then block the contiguous blocks that large jobs need. Preemption and priority classes resolve the conflict only by destroying someone’s work.
Sharing mechanisms reduce fragmentation at a price. The MIG user guide documents hardware partitions of supported GPUs into up to seven instances with separate memory and fault isolation. NVIDIA’s GPU Operator documentation is explicit that time-slicing offers neither memory nor fault isolation between replicas. Time-slicing looks like a utilization feature. It is also a decision to merge failure domains.
Utilization: the number that lies
The most widely reported GPU metric measures almost nothing. NVML defines utilization as the percentage of the sample period during which one or more kernels was executing. A single small kernel keeps it at 100%. DCGM’s profiling metrics are more honest because they are more specific. SM_ACTIVE is the fraction of time at least one warp is resident per multiprocessor. SM_OCCUPANCY is resident warps relative to the maximum. PIPE_TENSOR_ACTIVE tracks tensor-core cycles and DRAM_ACTIVE tracks memory traffic, which the documentation notes rarely exceeds roughly 80% in practice. A decode-heavy inference server shows high DRAM and low tensor activity, which is its correct steady state.
None of these metrics captures the waste that matters at scale. In synchronous data parallelism, the step completes when the slowest rank finishes, so one thermally throttled GPU or one degraded link sets the pace for thousands. The Llama 3 report describes Meta building tooling specifically to find such stragglers. Nor do device counters see idle-but-allocated capacity: a gang waiting for its last rank, a job paused while a checkpoint loads, or a hot spare held in reserve.
A defensible metric has to start from useful work and divide by allocated time. For training, the natural composite is Model FLOPs Utilization, as defined in the PaLM paper, multiplied by Meta’s effective training time ratio (ETTR): productive runtime over wall-clock time, net of queueing, restart and lost progress, as formalised in the research-cluster study. For serving, the equivalent is goodput in the sense DistServe uses: requests completed within both time-to-first-token and time-per-output-token targets, per allocated GPU-hour. The Llama 3 team reports 38–43% MFU and effective training time above 90%. ByteDance’s MegaScale reports 55.2% MFU on 12,288 GPUs. Neither is a utilization number in the dashboard sense, and both are more meaningful than one.
Networking: when the fabric sets the ceiling
Training traffic is a handful of collective patterns. Data-parallel gradient synchronisation is all-reduce, and in ring form each rank moves close to twice the buffer size regardless of rank count, as the nccl-tests performance notes derive for bus bandwidth. Tensor parallelism generates frequent small all-reduces, pipeline parallelism point-to-point sends between stages, and mixture-of-experts routing all-to-all exchanges that approach worst-case incast. NCCL selects among ring, tree and NVLink SHARP algorithms by message size and topology.
What distinguishes this traffic from cloud traffic is entropy. HPN describes LLM training as producing very few flows, each periodically bursting to full NIC line rate. ECMP hashing assumes many small flows and fails badly with a few large ones. Meta reports in its RoCE engineering write-up that it moved to enhanced ECMP with queue-pair scaling for this reason, citing up to 40% improvement on all-reduce. Topologies have adapted too. Rail-optimized designs, documented in NVIDIA’s DGX SuperPOD reference architecture, connect the same-indexed GPU in every server to the same leaf, so most collective traffic stays one hop away. HPN extends this with a two-tier, dual-plane design that it reports holds 15,000 GPUs in one pod.
Congestion control is where operators diverge and publish least. RoCE fabrics typically rely on PFC plus DCQCN, but Meta reports disabling DCQCN at 400G after firmware problems, relying instead on receiver-driven admission coordinated between the collective library and the transport. InfiniBand operators rely on credit-based flow control and adaptive routing. There is no public, controlled comparison of these approaches at ten-thousand-GPU scale. Operator reports describe what worked for them, not why alternatives failed.
Oversubscription is where the network visibly sets the ceiling. The Llama 3 cluster’s three-layer Clos has a 1:7 oversubscription ratio between pods, according to Meta’s report, and its scheduler places jobs to minimise cross-pod traffic. Meta’s RoCE paper describes cross-zone links as oversubscribed by design. The consequence (inference) is that scaling efficiency is bounded not by accelerator count but by how much of a job’s collective traffic the scheduler can keep below the oversubscribed tier. Beyond that boundary, adding GPUs adds communication faster than it adds compute.
Storage and the data path
Storage is where fleets discover that their workload is bursty. The Llama 3 report describes a 240 PB Tectonic deployment on 7,500 servers sustaining 2 TB/s and peaking at 7 TB/s, and calls checkpoint writes highly bursty. Every rank writes its shard at the same instant and then the system goes quiet. That is write amplification in time rather than space: the storage system must be sized for a peak it rarely sees. CheckFreq showed the standard mitigation: snapshot to host memory, persist asynchronously, and tune frequency online, keeping overhead within its reported 3.5% bound.
Input pipelines fail differently. Mohan et al.’s data-stalls study found that training time is often dominated by waiting for data to be fetched and preprocessed, not by computation. Random-access sample reads defeat page caches once a dataset exceeds memory. Parallel file systems tolerate this; object stores need a caching tier or sharded sequential formats, and because caches are local while jobs move, the choice ends up constraining the scheduler.
At serving time the data path becomes a latency problem. A replica cannot serve until tens or hundreds of gigabytes of weights reach HBM. ServerlessLLM (OSDI ’24) addresses this with a loading-optimised checkpoint format, multi-tier loading across SSD, DRAM and GPU, and locality-aware placement, reporting 10–200× latency reductions against the serverless baselines it tested. Its lesson is general: cold-start latency is set by where weights already sit, which makes model placement a scheduling input.
Serving: a different machine on the same hardware
Inference has two phases with opposite profiles. Prefill processes the whole prompt in parallel and is compute-bound. Decode generates one token per step per sequence and is bound by memory bandwidth. Microsoft’s Splitwise characterises the asymmetry, and DistServe shows that colocating the phases causes interference, reporting up to 7.4× more requests served within latency targets when they run on separate GPUs.
Memory, not arithmetic, is the binding constraint. Every in-flight sequence holds a key-value cache that grows with context length, and the cache bounds batch size. Orca introduced iteration-level scheduling so sequences can join and leave a batch between steps. PagedAttention made the cache allocatable in blocks, reporting near-zero fragmentation waste and 2–4× throughput over prior systems. Sarathi-Serve splits long prefills into chunks so they stop stalling decodes. Each is a fix for head-of-line blocking: one long prompt or one memory-hungry tenant delaying everyone batched with it.
Serving and training want opposite things from the same fleet. Training wants large, contiguous, topology-local gangs held for weeks and tolerates latency. Serving wants many small replicas spread across failure domains, scaled in minutes against bursty demand, and is judged on tail latency. Autoscaling is hard because demand arrives in token counts nobody knows in advance, and a new replica pays the weight-loading cost above. A shared fleet (inference) must either reserve headroom that training cannot use or preempt training, which converts a serving spike into lost training progress.
Failure domains: the underestimated layer
Meta’s research-cluster study, covering two A100 clusters and more than 150 million GPU-hours, gives the clearest public statement of the scaling law: mean time to failure falls in proportion to GPU count. The paper reports 47.7 days for 8-GPU jobs, 7.9 hours for 1,024-GPU jobs, and projects about 1.8 hours at 16,384 GPUs. The Llama 3 report is consistent with that projection. Across 54 days on up to 16,384 H100s it records 466 interruptions, 419 unexpected, with faulty GPUs and HBM3 memory the two largest categories and roughly 78% attributed to confirmed or suspected hardware.
The rest of the stack sets the blast radius. HPN reports that a single ToR failure can crash an entire training job, and that more than 40% of critical failures in its earlier data centres came from stacked dual-ToR designs. Llama 3 notes that synchronised GPU activity swings data-centre power by tens of megawatts, and that throughput varied 1–2% with time of day as temperature affected GPU clocks. Correlation is the norm: host, rack, switch, power feed and cooling loop each define a domain that can take out many ranks at once.
Then there are failures that do not announce themselves. Meta’s Silent Data Corruptions at Scale and Google’s Cores that don’t count established that defective processors compute wrong answers without raising errors. Google’s Gemini report states that at its training scale it expected SDC to affect training every week or two, and describes deterministic replay and proactive scanning to isolate it. An AWS study of SDC in LLM training found that unhealthy nodes can leave pretraining loss looking normal while steering models toward different weights, and can trigger loss spikes during fine-tuning. The gap here is large. Operators rarely publish per-device SDC rates, and there is no public accelerator-fleet equivalent of the CPU studies.
Checkpoint economics follow from all this. Young’s approximation sets the optimal checkpoint interval near the square root of twice the checkpoint cost times the mean time between failures. As MTTF falls with job width, the optimal interval shrinks, and checkpoint cost becomes a first-order term in throughput. The Gemini report shows the payoff of attacking recovery rather than failure: keeping redundant in-memory model replicas and restoring from them raised goodput for its largest job from 85% to 97%. No faster GPU would have bought twelve points.
Health checking is the other half. Meta runs checks every five minutes for XID errors, mount failures and service health, and reports that detecting 40 “lemon” nodes cut large-job failures from 14% to 4%. An intermittently failing node is worse than a dead one, because the scheduler keeps placing new gangs on it. Draining and quarantine are reliability mechanisms that look, on a utilization dashboard, like idle capacity.
Two interactions for which nobody plans.
A storage choice turns into a network challenge. Checkpointing is synchronous at each rank level, and thus each checkpoint is an incast burst from thousands of hosts to the storage layer. In Meta’s RoCE checkpointing paper, a separate network for front-end tasks like checkpoints and logging is kept away from the back-end training network, and the rack bandwidth is sized such that the storage traffic won’t impact training. Now mix this up with Young. With wider jobs, MTTF goes down, leading to an optimal shorter checkpointing period and more bursts. Thus, my takeaway is that the storage bandwidth required by a wider job will scale with the width of the job itself despite a fixed model size, and in case of converged network, those bursts end up occupying the same switch buffers that carry gradients for all-reduce operation. The first sign of such inefficiency would be unexplained slowdowns of the entire set of jobs.
A scheduling decision turns into a reliability problem. Topology-aware packing reduces the cross-pod communication cost, and the placement algorithm by Meta explicitly optimizes for this. However, with topology-aware packing, the job is concentrated on fewer ToRs, racks and power feeds, and each becomes a failure domain in itself. On the other hand, fragmented placement increases the number of switches the job is running on. As per HPN’s statistics, one of the ToRs crashing could mean the death of the job. There is no “safe” way out – maintaining hot spares for quick job recovery becomes yet another scheduling decision, visible on the dashboard as unused capacity.
The steel-man: just buy more GPUs
The strongest version of the opposing case is serious. Accelerators are the scarce input, and engineering time is scarcer. Each hardware generation improves performance per dollar faster than a systems team improves utilization. Headroom absorbs fragmentation, straggler replacement and failures without clever scheduling. For serving fleets built from independent replicas, it is simply correct: a failed replica costs one replica, and overprovisioning buys tail latency directly.
It fails for synchronous training, because adding GPUs to a job also adds failures to it. Meta’s data makes MTTF inversely proportional to width, so a job twice as wide fails roughly twice as often, needs checkpoints more frequently, and pushes more traffic across the oversubscribed tier. Spare GPUs help only if the scheduler can assemble them into a gang in the right topology, and they burn power and floor space that operators increasingly describe as the binding constraint (inference). More accelerators are necessary; they are not a substitute for a recovery path. Gemini’s twelve goodput points came from recovery engineering, not hardware.
Judgment
Buy the GPUs, then spend the engineering on what happens when they fail. For training fleets, the metric that belongs on the executive dashboard is MFU multiplied by ETTR, not device utilization. The work that moves it is mostly outside the accelerator: health checks that catch lemons before a gang lands on them, placement that treats switches and power feeds as failure domains as well as bandwidth domains, in-memory or peer-replicated checkpoints that make recovery cheap, and a frontend fabric sized for synchronous bursts. Serving fleets need the inverse discipline: spread, isolate, and treat KV-cache memory as the capacity unit.
What the field has not settled is silent corruption at accelerator scale. Loud failures are now measured, modelled and increasingly well handled. Silent ones are not: we lack public per-device rates, a cheap way to verify training arithmetic continuously, and any agreement on how much undetected corruption a model can absorb before its weights are wrong in ways no evaluation will catch. As jobs widen, that is the failure mode whose blast radius nobody can yet measure.
