Datacenter
AI storage tier architecture for GPU clusters: feeding the GPUs and the checkpoint burst
The tiered storage that feeds a GPU cluster: the hot parallel flash tier, capacity and cold tiers, the checkpoint write burst, GPUDirect, RDMA, and sizing to bandwidth and latency instead of capacity.
Direct answer
AI storage tier architecture is the layered storage that feeds a GPU cluster: a high-performance parallel flash tier near the GPUs to stream training data and absorb bursty checkpoint writes, with capacity and archive tiers behind it. Size it to bandwidth and latency, not capacity alone, and confirm the numbers against the workload and the design.
Key takeaways
- Size AI storage to bandwidth, IOPS, and latency first and capacity last; a system full of terabytes but short on sustained throughput starves the GPUs.
- Tier the storage hot (NVMe parallel flash for active data and checkpoints), warm (capacity flash or disk for staged datasets), and cold (object or archive for the data lake).
- Checkpoints run tens to hundreds of gigabytes (terabytes for the largest models) in a synchronized burst; complete the write in a small fraction of the checkpoint interval so the cluster does not stall.
- Absorb the checkpoint burst on the fastest tier (often local NVMe), then drain it to shared storage in the background while the GPUs resume.
- Commission under load with the real I/O profile: an aggregate multi-client read test, a checkpoint burst test, and a node-failure restart from the last checkpoint.
AI storage tier architecture, and why idle GPUs are the real cost
AI storage tier architecture is the layered storage that keeps a GPU cluster fed. The GPUs are the most expensive thing in the building, and they only earn their cost when they are computing. The job of the storage is to stream training data to them fast enough that they never wait, and to swallow the huge, synchronized checkpoint writes the training job throws at it without pausing the run. Miss either one and thousands of GPUs sit idle while storage catches up.
That is the whole problem in one sentence. Feed the GPUs, absorb the bursts, and never be the reason the cluster stalls. The architecture that does it is tiered. A fast parallel flash tier sits close to the GPUs and carries the active training data and the checkpoints. Capacity tiers sit behind it for the datasets that are not in play right now. A cheap object or archive tier holds the cold bulk. Data moves between them by need.
The network that carries the data and the power and space it occupies are covered in the sibling guides. This one is about the tiers themselves: what each does, how fast each has to be, and where the design goes wrong. The numbers here move with the workload, the platform, and the model size, so treat every figure as a starting point to size against the actual job, not a constant.
Why the most expensive asset waits on storage
A modern GPU node is the most expensive asset on the floor, and an idle one is pure waste. Rent or amortize a large cluster and every hour the GPUs spend waiting on data is money spent on nothing. The metric the operators watch is utilization, the fraction of wall-clock time the GPUs are actually computing, and slow storage is one of the few things that can quietly drag it down across the whole cluster at once.
The failure is easy to picture. The GPUs finish a step and need the next batch of training data. If the storage cannot deliver it in time, the GPUs stall, and because the job is synchronized, one slow reader holds up the rest. The same thing happens on the write side when a checkpoint lands and the cluster pauses to save state. Either way the most expensive hardware in the room waits on the cheapest, and you paid for the wait.
This is why storage for AI is sized differently than storage for almost anything else. You are not buying capacity for its own sake. You are buying the bandwidth and the low latency it takes to keep a very expensive consumer busy. Get that backwards and the cluster runs at a fraction of what it cost to build.
The data pipeline: load, compute, checkpoint, repeat
The training loop is a repeating cycle, and storage sits at two points in it. The job loads a batch of training data from storage into GPU memory. The GPUs compute on it, running the forward and backward pass and updating the model weights. Periodically the job writes a checkpoint, a snapshot of the model state, back to storage. Then it loops, for days or weeks.
Each phase puts a different demand on storage. The load phase is a sustained, heavy read, often the same dataset streamed over and over across many training epochs. The checkpoint phase is a sharp, bursty write that hits all at once and then goes quiet. Between them the storage is comparatively idle, which is why average throughput numbers mislead. The storage has to be sized for the peaks of each phase, not the average across the run.
Knowing where storage sits in the loop tells you what to optimize. Read bandwidth keeps the load phase from starving the compute. Write absorption keeps the checkpoint from stalling the cluster. Everything in the architecture serves one of those two moments.
How do you keep the GPUs fed?
You keep the GPUs fed by giving the storage enough read bandwidth to stream the training data faster than the GPUs can consume it, with latency low enough that the next batch is always ready. The figure that matters is aggregate read throughput to the cluster during the data-loading phase, and it scales with the GPU count.
The published reference designs give a sense of scale. A few hundred GPUs reading a dataset can demand hundreds of gigabytes per second of sustained read, and large reference architectures specify read tiers that climb well into the hundreds of GB/s as the cluster grows. The useful way to think about it is bytes per GPU per second. Work out how fast one GPU consumes data for your model and data type, multiply by the GPU count, and that aggregate is what the storage has to sustain. Not peak. Sustain.
The hedge matters here. The bandwidth a workload needs depends heavily on the model and the data. A large-image or video pipeline is data-hungry and pulls hard on storage. A compute-bound language model on tokenized text can run for long stretches on relatively little read bandwidth once the data is cached. Size to the actual workload's read pattern, confirm against the platform's reference design, and do not copy a number from a different job.
The checkpoint write burst
The checkpoint write is the storage problem unique to AI training. At intervals the job saves the entire model state, the weights and the optimizer state, from GPU memory to persistent storage. For a large model that is tens to hundreds of gigabytes, into the terabytes for the largest, and on a big job it is written by many ranks at once in a single synchronized burst. The storage either absorbs it fast or the whole cluster waits while it lands.
The cost of a slow checkpoint is brutal because of the multiplier. The stall might last a couple of minutes, but it is a couple of minutes times every GPU in the job, all idle, all paid for. That is why the operators care about checkpoint write bandwidth as much as read. Multi-node jobs can drive checkpoint writes into the tens of GB/s, and the target is to complete the write in a small fraction of the checkpoint interval so the pause barely registers.
The common fix is to absorb the burst on the fastest tier you have, often local NVMe or a flash tier sized for write bandwidth, and then drain it to slower storage in the background while the GPUs go back to work. How big the burst is and how often it lands depends on the model size and the checkpoint strategy, so size the write path to the real checkpoint, not a guess.
Why checkpoint at all
Checkpoints exist because large clusters fail often. This is the part newcomers underestimate. Individual GPUs are reliable, but put sixteen thousand of them in one synchronized job and the math turns against you. A widely cited account of a 16,384-GPU training run logged a failure roughly every three hours over its run, about half of them traced to the GPUs or their high-bandwidth memory. The bigger the cluster, the shorter the time between failures.
When any worker dies, the synchronized job stops, and without a checkpoint it would restart from zero and throw away days of compute. The checkpoint is the save point. The job restarts from the last checkpoint, not the beginning, so the loss is the work since the last save plus the time to reload. That same run reportedly held effective training time above 90 percent, and frequent checkpointing is a large part of how.
This sets the checkpoint interval against a tradeoff. Checkpoint too rarely and a failure costs you more lost work. Checkpoint too often and the write bursts and stalls eat into training time on their own. The interval that balances the two depends on the cluster's failure rate and the cost of a checkpoint, which is exactly why the write path and the commissioning that proves it under load matter. Validating that the cluster survives a failure and restarts cleanly is a commissioning step, not an afterthought.
What are the storage tiers?
The storage tiers are layers of storage ranked by speed and cost, with the fast, expensive media close to the GPUs and the slow, cheap media behind it. Data lives on the tier that matches how it is being used right now, and it moves as that changes. The standard split is hot, warm, and cold, with the node-local flash as a tier of its own.
The hot tier is all-flash, usually NVMe, and it carries the active training data and the checkpoints. It is the fastest and the most expensive per terabyte, and it is sized for bandwidth, not bulk. The warm tier is capacity flash or fast disk holding datasets that are staged but not in active use. The cold tier is object or archive storage, the cheap bulk where the full data lake and old checkpoints sit until something needs them.
The art is putting the right data on the right tier. Park a hot dataset on cold storage and the GPUs starve waiting for it. Keep everything on flash and the cost runs away from you. The tiering, manual or automated, is what holds both the performance and the budget in line.
| Tier | Typical media | What it holds | Sized for |
|---|---|---|---|
| Hot | NVMe all-flash, parallel | Active training data, checkpoints, scratch | Bandwidth and latency |
| Warm | Capacity flash or fast disk | Staged datasets, recent data and checkpoints | Cost per TB with usable speed |
| Cold / archive | Object storage, high-capacity disk, tape | The data lake, cold datasets, old checkpoints | Cheapest bulk capacity |
| Local (node) | NVMe inside the GPU server | Scratch, cache, fast checkpoint landing | Lowest latency, write absorption |
The hot tier: parallel NVMe flash
The hot tier is the high-performance flash that does the work during training. It is built from NVMe SSDs, almost always served by a parallel or scale-out file system so many nodes deliver throughput at once, and it sits as close to the GPUs on the network as the design allows. Two jobs land here: streaming the active training data to the GPUs, and catching the checkpoint write bursts.
This tier is sized by bandwidth, not capacity. You do not buy it to hold the whole dataset. You buy enough fast flash to hold the working set, the data currently in play, plus headroom for checkpoints, and you provision it to deliver the aggregate read and write the cluster demands. It is common for the hot tier to be a small fraction of total stored capacity while carrying nearly all of the performance.
Scratch space lives here too, the temporary working area for preprocessing, intermediate files, and shuffled data. Treat the hot tier as the one place where you pay for speed and accept the cost per terabyte, because this is the tier that decides whether the GPUs wait. Size it to the workload's working set and peak bandwidth, and confirm the numbers against the platform reference design.
The parallel file system
A single storage box cannot feed a large GPU cluster, so the hot tier runs on a parallel or distributed file system. The idea is striping. A file is split across many storage targets, and many storage nodes serve pieces of it at the same time, so the throughput is the sum of all of them rather than the limit of any one. Metadata servers track where everything lives and coordinate access. This is how a cluster reaches hundreds of gigabytes per second instead of the few any single server could give.
The established parallel file systems each have a following. Lustre and BeeGFS come out of HPC, IBM Storage Scale (formerly GPFS) is long-standing, and a wave of scale-out flash platforms target AI specifically. Some designs use a high-performance object store instead of a POSIX file system, which can simplify scaling at the cost of changing how the application reads data. Which one fits depends on the workload, the operations team's experience, and what the platform vendor supports.
The reason this layer matters is throughput scaling. A parallel file system lets the storage grow with the GPU count, adding nodes to add bandwidth. Size the number of storage nodes to the aggregate bandwidth target, not the capacity, because on AI workloads you usually hit the bandwidth ceiling long before the capacity one.
The warm capacity tier behind the flash
Behind the hot tier sits the capacity, or warm, tier. It holds the datasets that are staged and likely to be used soon but are not being read by the GPUs at this moment, along with recent checkpoints that have aged off the hot tier. It is built for a better cost per terabyte than the hot tier, using denser flash or fast disk, and it accepts more latency in exchange.
The reason to have a distinct warm tier is economics. Training datasets are large and most of the data is not in the active working set at any given time. Keeping all of it on the fastest flash is money spent to make idle data fast. The warm tier holds the bulk that needs to be reachable quickly when staged to hot, without paying hot-tier prices for data that is sitting still.
The line between hot and warm is a design choice, not a law. On a workload that cycles through a dataset that fits on flash, the warm tier can be thin. On one with a huge corpus and a small working set, the warm tier carries most of the data and the staging between warm and hot becomes the thing you tune. Size it to the dataset and the staging pattern.
The cold and archive tier
The cold tier is the cheap bulk, where the full data lake and the data you are not training on right now live. It is almost always object storage, sometimes on high-capacity disk, sometimes with tape behind it for the truly cold archive. It is the cheapest per terabyte by a wide margin and the slowest, and that tradeoff is the point.
This is where raw datasets land before they are cleaned and curated, where old model checkpoints go once you are past the point of needing to restart from them, and where regulatory or provenance copies sit. None of it needs to be fast. It needs to be cheap, durable, and large, because the data lake behind a serious training operation runs to petabytes and grows constantly.
The discipline is not letting cold data creep onto warm or hot storage because someone might need it. Cold data on flash is the most common way a storage budget quietly blows up. When a dataset is going to be trained on, it gets staged forward to warm and then hot on a schedule. Until then it stays cold, and the staging time gets planned into the run.
Size to bandwidth, IOPS, and latency, not capacity
What do you size AI storage for? Bandwidth, IOPS, and latency, in that order for most training workloads, and capacity last. This is the inversion that trips up teams coming from general enterprise storage, where capacity leads. On a GPU cluster the storage exists to move data fast, and a system with plenty of terabytes but not enough sustained throughput will starve the GPUs no matter how full or empty it is.
Bandwidth, the gigabytes per second the storage can sustain, governs the data-loading and checkpoint phases. IOPS, the count of operations per second, matters most when the workload reads many small files, which is the image and small-record case. Latency, the time to first byte, decides whether the next batch is ready when the GPU asks for it. A workload can be limited by any of the three, and which one binds depends on the data.
So size to the workload. A large-file streaming read is a bandwidth problem. Millions of small image files is an IOPS and metadata problem. A latency-sensitive pipeline needs low time-to-first-byte more than raw throughput. Capacity still has to be enough to hold the data, but it is the constraint you check last, not first. Pull the real I/O profile of the job and size to it, then confirm against the platform reference design.
The storage fabric: RDMA, shared or separate
The storage has to reach the GPUs over the network, and on a serious cluster that path uses remote direct memory access. RDMA, over InfiniBand or RoCE on Ethernet, lets the storage move data into a node's memory without going through the CPU on every transfer, which is what makes the high throughput and low latency possible. NVMe over Fabrics extends the local NVMe protocol across the network so remote flash behaves close to local.
A design decision worth getting right early is whether storage rides the same fabric as the GPU-to-GPU traffic or a separate one. The back-end compute fabric carries the all-to-all training traffic and is the most latency-sensitive network in the building. Putting heavy storage traffic, especially a checkpoint burst, on the same links risks colliding with that traffic and stalling training. Many designs give storage its own fabric or a dedicated set of ports for that reason, while some converge them deliberately and manage the contention. The optics, the speeds, and the fabric topology are covered in the GPU network and cabling guide.
The point for storage is that the fabric is part of the storage system, not a detail outside it. A fast flash tier behind a congested or undersized network delivers congested, undersized performance. Size the storage network to the same bandwidth target as the storage, and keep the checkpoint burst from fighting the training traffic.
Local NVMe in the GPU node
Every GPU server has NVMe inside it, and on AI workloads that local flash is a tier in its own right. It is the lowest-latency storage a GPU can reach, and it serves three jobs: scratch space for the running job, a cache for hot training data so repeated reads do not go back to the network, and a fast landing zone for checkpoints.
The checkpoint use is the one that has changed the design. A common pattern writes the checkpoint first to local NVMe, which can absorb the burst at full local speed, and then drains it to the shared parallel tier in the background while the GPUs resume. The cluster sees a short, fast local write instead of a long wait on the network, and the durable copy still lands on shared storage a moment later. Tiered checkpointing schemes lean on exactly this, writing frequently to node-local flash and less often to the shared system.
The limit of local NVMe is that it is local. It does not survive the node, it is not shared across the cluster, and it does not replace the parallel tier for data that has to be seen by everyone. Treat it as the fastest, most temporary tier, and make sure anything that has to be durable gets to shared storage before you count on it.
What is GPUDirect Storage?
GPUDirect Storage is a direct data path between storage and GPU memory that skips the CPU. Normally data read from storage lands in a buffer in CPU system memory first, and then the CPU copies it into GPU memory, the so-called bounce buffer. GPUDirect Storage uses DMA to move data straight from the NVMe device or the network into GPU memory, cutting the CPU and the extra copy out of the path.
The payoff is bandwidth and efficiency. Removing the bounce buffer reduces latency and can lift effective bandwidth into GPU memory severalfold, and it frees the CPU and the PCIe bus from shuttling data they did not need to touch. On a data-hungry workload that is the difference between the GPUs being fed and the host becoming the bottleneck. It is part of NVIDIA's Magnum IO software stack and depends on support across the storage system, the file system, and the drivers.
It is not automatic and not universal. The storage platform, the file system, and the application all have to support the path, and the benefit is largest on workloads that move a lot of data per unit of compute. Whether it is worth enabling depends on the workload and whether the rest of the stack supports it, so treat it as an optimization to validate, not a default to assume.
The small-file and metadata problem
The small-file problem is the one that quietly wrecks throughput. A storage system that streams a large sequential file at hundreds of gigabytes per second can crawl when the same workload reads millions of tiny files, because each file is a metadata operation, an open, a lookup, a close, and metadata operations do not stripe the way bulk data does. Computer-vision training on millions of small image files is the classic case.
The bottleneck moves from bandwidth to metadata. The metadata servers, the part of the parallel file system that tracks where files live, become the limit, and the GPUs starve waiting on file opens rather than on data. A system that benchmarked beautifully on large-file throughput can fall on its face on the real workload if the real workload is small files.
The fixes are workload-side and storage-side. On the workload side, packing many small samples into larger container files, a common format choice in training pipelines, turns a metadata problem back into a bandwidth one. On the storage side, a file system with strong metadata performance or a dedicated flash metadata tier helps. Either way, if the workload is small files, test the storage on small files, because the headline bandwidth number tells you nothing about how it will actually run.
Staging the dataset and the data lifecycle
Getting data onto the right tier at the right time is its own discipline. Before a job runs, the dataset it needs has to be staged from cold object storage up to the warm and hot tiers so the GPUs are not waiting on slow storage when the run starts. After the job, checkpoints and outputs age back down from hot to warm to cold as they stop being needed. This movement is the data lifecycle, and on a busy cluster it never stops.
Some of it is automated. Tiering software can watch access patterns and promote or demote data on its own, and that works well for steady patterns. Much of it is still scheduled around the training calendar, because the team knows which dataset the next run needs before any access pattern would reveal it. Staging a multi-petabyte dataset up from cold takes real time, and a run that starts before its data is staged is a run that starts slow.
The thing to plan is the staging time. Treat moving a dataset to the hot tier as a step with a duration, the same way you would treat any long-lead task, and start it early enough that the GPUs are fed the moment the job launches.
Storage power and space
Storage competes for the same power and space as everything else in the room. It is easy to plan the GPU racks and the cooling and forget that the storage tier has its own footprint, draws its own power, and adds its own heat. On a dense build that storage power comes out of the same budget as the compute, and the capacity-planning guide covers how that budget is tracked.
Flash helps more than people expect. An NVMe SSD draws a fraction of the power of a spinning disk at idle, and flash packs far more capacity into a rack unit, so an all-flash tier can hold a lot of data in little space with modest power compared to the disk array it replaces. Vendors are pushing flash density hard, with high-capacity QLC drives putting tens of petabytes in a single rack. The cold tier, on the other hand, is where high-capacity disk and tape still earn their place, because for data that is rarely touched the cost and power per terabyte beat flash by a wide margin.
The planning point is to count storage in the power and space model from the start. A storage tier sized for bandwidth can carry more flash than its capacity alone would suggest, and that flash draws power and needs cooling. Roll it into the same capacity plan as the racks, and confirm against the design rather than assuming storage is a rounding error.
Redundancy, erasure coding, and durability
The data has to survive, and the checkpoint above all. Lose a checkpoint to a storage failure and a job has nothing to restart from. Storage at this scale protects data with redundancy, most often erasure coding, which spreads data and parity across many drives so the system can lose drives without losing data and without the capacity overhead of full mirroring. Larger writes in particular suit wide erasure codes.
The durability target is a design decision tied to what the data is worth. The active training data can often be regenerated or re-staged from the cold copy, so it does not all need the highest protection. The cold data lake is the system of record and gets durable, redundant storage because re-acquiring it may be impossible. The checkpoint is the interesting middle. It is transient, but losing the current one at the wrong moment costs real time, which is why the durable copy needs to land on protected shared storage and not live only on a node's local flash.
Match the protection to the consequence. Full redundancy on data you can regenerate is wasted money, and thin protection on the data lake is a disaster waiting for a bad drive. Decide tier by tier what a loss actually costs, and protect accordingly.
The tier economics: flash, disk, and object
Storage cost on a GPU cluster is a tiering decision more than a media decision. Flash, disk, and object storage sit at very different price points per terabyte, and the way to keep the total in line is to hold each class of data on the cheapest tier that still meets its performance need. That is the entire economic argument for tiering.
The temptation is to buy flash for everything because flash is fast and the GPUs are expensive. On the hot working set that is correct. Across the full data lake it is ruinous, because most of that data is cold most of the time and paying flash prices to store idle petabytes is money lit on fire. The opposite mistake, trying to run training off cheap bulk storage to save money, starves the GPUs and wastes far more in idle compute than it saves in storage.
The balance is workload-specific. A small dataset that fits on flash changes the math entirely from a multi-petabyte corpus with a small working set. Price the tiers against the data's real access pattern, put the hot working set on flash and the cold bulk on object, and let the warm tier and the staging carry the middle. The right data on the right tier is the whole game.
The parallel-storage ecosystem: build versus buy
The storage under an AI cluster comes from a crowded field, and there is no single right answer. On one side are the HPC-rooted parallel file systems with long track records. On another are scale-out flash platforms built and marketed for AI, some delivering file and object together. On a third are object-first designs and the cloud providers' managed storage. The GPU and storage vendors publish reference architectures that pair specific storage with specific GPU platforms at named bandwidth tiers, and those references are the sanest place to start because someone has already validated the combination.
Build versus buy is the real fork. A large operator with deep storage expertise can build on open parallel file systems and tune them hard. Most teams are better served buying a validated platform that the GPU vendor's reference design already blesses, because the integration work and the support matter more than the per-terabyte price when the GPUs are the expensive part. The right choice depends on the team, the scale, and how much storage engineering you want to own.
Do not pick storage on a benchmark sheet alone. The number that matters is sustained performance on your workload, under your checkpoint pattern, integrated with your fabric. Match the storage to the GPU platform's reference design, validate it on the real job, and weight support and operability above the headline GB/s.
Commissioning: validate the bandwidth and the checkpoint
Storage gets commissioned like any other system, by proving it does the job under load before the cluster goes to work. The acceptance test is not a single-stream benchmark. It is the aggregate read bandwidth the design promised, measured across many clients at once, the way the training job will actually read. If the reference design says a number, the room has to hit it with the real client count, not in a lab.
The checkpoint test is the AI-specific one. Drive a synchronized write burst that matches the model's real checkpoint size and rank count, and confirm the storage absorbs it inside the target window without stalling. Then go further and prove the recovery path. Kill a node mid-run and confirm the job restarts cleanly from the last checkpoint. A cluster that has never been tested against a failure will find out the hard way that the restart does not work, three hours into a run, at scale.
Commission under load, with the real I/O profile, before the GPUs are earning. Storage that passed a clean-room benchmark and was never tested against the real read pattern and the real checkpoint burst is storage that will surprise you at bring-up. Tie the acceptance to numbers, write them down, and do not accept the tier until it meets them on the actual workload.
What to document
The storage architecture is only as good as the record of what was built and why. Six months in, when a job runs slow or a tier fills, the questions are which tier holds what, what bandwidth each was sized for, and what the acceptance test actually measured. If that lives only in someone's head, the next person re-derives it under pressure.
Capture the tier layout, the media and file system on each tier, the sized bandwidth and IOPS targets, the measured acceptance numbers, the checkpoint size and interval the write path was built for, the staging plan between tiers, and the fabric the storage rides. We track this kind of as-built and the acceptance results in FieldOS so the tier design, the bandwidth test, and the sign-off stay attached to the room instead of scattered across spreadsheets and email.
| Tier | Role | What to note |
|---|---|---|
| Hot (NVMe flash) | Active training data, checkpoints, scratch | Sized bandwidth and IOPS, file system, measured read and write |
| Warm (capacity) | Staged datasets, recent checkpoints | Capacity, staging plan to hot, cost per TB |
| Cold (object/archive) | Data lake, cold datasets, old checkpoints | Capacity, durability scheme, retrieval time |
| Local NVMe | Scratch, cache, checkpoint landing | Per-node capacity, drain path to shared |
| Storage fabric | Carries the storage to the GPUs | RDMA type, shared or separate from the compute fabric |
Common mistakes
- Sizing the storage by capacity instead of by sustained bandwidth and latency.
- A checkpoint write path that cannot absorb the burst, so the whole cluster stalls to save state.
- A slow tier in the read path that starves the GPUs and drags utilization down across the job.
- Benchmarking on large files when the workload is millions of small files, hiding a metadata bottleneck.
- Leaving storage out of the power, space, and cooling plan until the racks are already committed.
- An inefficient CPU-in-the-path data flow where a bounce buffer caps bandwidth that GPUDirect could lift.
- Parking cold data on flash, or running training off cold bulk, so the budget or the GPUs pay for it.
- Accepting a tier on a clean-room benchmark and never testing it against the real read pattern and checkpoint burst.
Field checklist
Want this checklist to run itself on every job — with photo proof and a signed record crews can hand the customer? That's FieldOS.
Standards and references
There is no single code that governs AI storage the way the NEC governs wiring. The practice comes from HPC storage architecture, the parallel-file-system world, and the reference designs the GPU and storage vendors publish. NVIDIA's cluster reference architectures, the SuperPOD-class designs, specify storage bandwidth tiers matched to GPU counts, and they are the closest thing to an authoritative starting point for sizing. Treat them as the baseline the workload adjusts, not a guarantee.
The technologies have their own specifications. NVMe and NVMe over Fabrics are defined by the NVM Express organization. RDMA transports run over InfiniBand or RoCE on Ethernet, with the relevant IEEE and IBTA work behind them, and the GPU network guide covers that layer. GPUDirect Storage is NVIDIA's, documented in their design and benchmarking guides. The MLPerf Storage benchmark from MLCommons is a useful neutral reference for how storage performs under AI training I/O, though a benchmark is a starting point, not your workload.
Hedge every number to the workload, the vendor platform, and the design. The bandwidth per GPU, the tier ratios, and the checkpoint sizing all move with the model, the data type, and the platform. Three principles hold across all of it: size storage to bandwidth and latency, not capacity alone; absorb the checkpoint burst and keep the GPUs fed; and tier the data hot, warm, and cold so the right data sits on the right media. Confirm the specifics against the platform reference design and the project requirements before you commit.
Units, terms, and conversions
AI storage borrows terms from HPC, from enterprise storage, and from the GPU vendors, so the same idea shows up under different names across a reference design, a spec sheet, and a benchmark report.
Bandwidth is gigabytes per second, sometimes written GB/s or GBps, and occasionally gigabits, Gb/s, which is eight times smaller for the same number, so watch the unit. Capacity runs from terabytes to petabytes and now exabytes for the largest lakes. Throughput, bandwidth, and GB/s usually mean the same thing here, while IOPS counts operations and latency is measured in microseconds or milliseconds.
- Storage tier
- A layer of storage at a given speed and cost, holding data that matches how it is being used right now
- Hot / warm / cold
- The tiers from fastest and most expensive (hot flash) to cheapest and slowest (cold object or archive)
- Parallel file system
- Storage that stripes files across many nodes so they serve throughput in parallel, scaling bandwidth with node count
- Checkpoint
- A saved snapshot of the model and optimizer state, written periodically so a failed job restarts from it instead of zero
- Bandwidth / IOPS / latency
- Throughput in GB/s, operation count per second, and time-to-first-byte; the three numbers AI storage is sized to
- GPUDirect Storage
- A direct DMA path from storage into GPU memory that skips the CPU bounce buffer
- Scratch
- Temporary high-speed working space for a running job, often on local or hot-tier flash, not meant to persist
- RDMA / NVMe-oF
- Remote direct memory access and NVMe over Fabrics, the network protocols that let remote flash feed GPUs at near-local speed
FAQ
Why does AI need fast storage?
AI needs fast storage because GPUs are the most expensive asset and idle ones waste money. If storage cannot stream training data fast enough, the GPUs stall waiting for the next batch, and a synchronized job stalls all of them at once. Storage exists to keep utilization high, not to hold capacity.
What is a checkpoint in AI training?
A checkpoint is a saved snapshot of the model weights and optimizer state, written to storage periodically during training. If a worker fails, the job restarts from the last checkpoint instead of from zero. Checkpoints are tens to hundreds of gigabytes, into the terabytes for the largest models and write in a synchronized burst the storage must absorb fast.
What are storage tiers?
Storage tiers are layers ranked by speed and cost. The hot tier is NVMe flash close to the GPUs for active data and checkpoints. The warm tier is capacity flash or disk for staged datasets. The cold tier is cheap object or archive storage for the data lake. Data moves between them by need.
What is GPUDirect Storage?
GPUDirect Storage is a direct data path that moves data straight from storage into GPU memory using DMA, skipping the CPU bounce buffer. Cutting the extra copy lowers latency and can raise effective bandwidth severalfold on data-heavy workloads. It needs support across the storage platform, the file system, and the application.
How much storage bandwidth does a GPU cluster need?
It depends on the model and data type, but a cluster of a few hundred GPUs can demand hundreds of gigabytes per second of sustained read during data loading. Reference designs scale the bandwidth tier with GPU count. Size to bytes-per-GPU for your workload, then confirm against the platform reference design.
Why do large GPU clusters fail so often?
At scale the failure math turns against you. One reliable GPU rarely fails, but tens of thousands in one synchronized job push the time between failures down to hours. A widely cited 16,384-GPU run logged a failure roughly every three hours, which is why frequent checkpointing is standard practice.
Should AI storage be sized for capacity or bandwidth?
Bandwidth and latency first, capacity last. A system with plenty of terabytes but not enough sustained throughput starves the GPUs no matter how full it is. Size the hot tier to the aggregate read and checkpoint write the cluster demands, and check that capacity is merely enough to hold the working set.
Should storage use the same network as the GPU fabric?
Often no. The back-end GPU fabric carries latency-sensitive training traffic, and a checkpoint burst on the same links can collide with it and stall the job. Many designs give storage its own fabric or dedicated ports. Some converge them deliberately and manage the contention. The design and workload decide.
What is a parallel file system and why does AI use one?
A parallel file system stripes files across many storage nodes so they serve data at once, summing their throughput. A single box cannot feed a large cluster, so AI uses parallel systems like Lustre, BeeGFS, or scale-out flash platforms to reach hundreds of GB/s and to scale bandwidth by adding nodes.
How do you keep checkpoint writes from stalling the cluster?
Absorb the burst on the fastest tier you have, often local NVMe or write-optimized flash, then drain it to shared storage in the background while the GPUs resume. Size the write path to the real checkpoint size and rank count, and target completing the write in a small fraction of the checkpoint interval.