Data center network architecture: the spine-leaf field guide

What the data center network does, and why the shape changed

The data center network connects the servers and storage to each other and to the outside, and the modern shape of it is spine-leaf: a flat, two-tier fabric instead of the old hierarchical pyramid. Every leaf switch sits at the top of a rack and connects the servers in that rack. Every leaf connects up to every spine, the spines tie the leaves together, and that is the whole structure. Two layers, one job, which is to move traffic between any two ports with a path length that does not change depending on which ports you picked.

It did not always look like this. For two decades the standard was three tiers, core over aggregation over access, built when most traffic went in and out of the building. Then virtualization, distributed applications, and now AI training flipped the traffic pattern. The bulk of it became server-to-server, bouncing sideways across the room, and the pyramid was the wrong shape for that. Spine-leaf is the answer the industry settled on.

This guide is about the network itself, the switches and how they connect. The physical media that carries it, the copper and fiber plant, is covered in the structured cabling guide, and the frames the switches and servers bolt into are covered in the rack and cabinet guide. Read those alongside this one, because a fabric design is only as good as the cabling and the racks that carry it.

What is east-west traffic?

East-west traffic is data moving server-to-server inside the data center, sideways across the room, as opposed to north-south traffic, which moves in and out of the building between the data center and the users or the internet. The names come from the way the old network diagrams were drawn, with the users at the top and the servers at the bottom, so client-to-server ran vertically and server-to-server ran horizontally.

The ratio between them is what drove the whole redesign. In hyperscale and cloud environments, studies put east-west at roughly 75 to 80 percent of all data center traffic, and the exact figure depends on the workload. A web request that looks like one north-south transaction to the user fans out into dozens of east-west calls behind it: the app server hits a database, a cache, an authentication service, a storage tier, each on a different rack. Virtual machines migrate between hosts. AI training shuffles enormous gradients between GPUs every step.

The three-tier network was built for the reverse, back when most traffic was a user pulling a file off a server. Server-to-server traffic in that design had to climb up to an aggregation or core switch and come back down, the long way around, every time. The flat fabric exists because the traffic went sideways and the network had to follow it.

The three-tier network and why it aged out

The legacy data center network had three layers. Access switches connected the servers. Aggregation switches, sometimes called distribution, gathered the access layer. Core switches tied the aggregation layer together and handed off to the outside. It was a clean hierarchy and it worked well for the traffic it was designed for, which mostly went up and out.

Two problems killed it for the modern load. The first is the path. Two servers on different access switches could only talk by going up to aggregation, maybe across to another aggregation switch, and back down, a long detour the industry calls tromboning. The latency was unpredictable because it depended on where the two servers happened to sit. The second is Spanning Tree Protocol. To prevent loops in a layered Layer 2 design, Spanning Tree deliberately blocks redundant links, so you paid for two uplinks and used one. The blocked link sat idle until the active one failed.

Stack those together and you get a network that oversubscribes badly under east-west load, wastes half its links, and gives you latency that swings with the layout. None of that mattered when traffic went north-south. All of it mattered the moment the traffic went sideways. The fabric flattened in response, and Spanning Tree gave way to equal-cost multipath routing that uses every link at once.

What is a spine-leaf network?

A spine-leaf network is a two-tier fabric built on the Clos topology, where every leaf switch connects to every spine switch, no leaf connects to another leaf, and no spine connects to another spine. The leaves are the access layer at the top of the racks. The spines are the layer that ties all the leaves together. There is no third tier above them in a standard fabric.

The design traces to Charles Clos, a Bell Labs engineer who showed in 1953 that a multistage arrangement of smaller switches could move traffic without blocking, the way one giant crossbar switch would, at a fraction of the cost. A leaf-spine fabric is a folded Clos network applied to Ethernet. The mathematics is old. The application to data centers is what changed.

The property that makes it work is uniformity. Because every leaf reaches every other leaf through exactly one spine in the middle, any server is the same distance from any other server: up to a spine, back down, done. Latency is predictable because the path length never changes. Bandwidth scales because traffic spreads across all the spines at once using equal-cost multipath, so adding a spine adds capacity to the whole fabric. There is no idle link waiting for a failure. Every uplink carries traffic all the time.

What is a leaf switch?

A leaf switch is the access switch that connects the servers and storage in a rack to the rest of the fabric, and in most builds it is the top-of-rack switch, the ToR, physically sitting in the top one or two rack units of the cabinet. The servers in that rack plug into the leaf. The leaf's uplink ports go to the spines. That is the leaf's whole role: down to the servers, up to the spines, nothing sideways.

The leaf has two distinct sides with different jobs. The downlink side faces the servers and runs at whatever the servers need, commonly 10, 25, or now 100 gigabit per port, on copper twinax or fiber over short distances inside the rack. The uplink side faces the spines and runs faster, commonly 100, 200, 400, or 800 gigabit, on fiber, because those few uplinks have to carry the combined traffic of every server below.

Leaves are deployed in pairs in any serious build. Two leaves per rack, each server dual-homed to both, so a leaf failure or a leaf reboot for a firmware update does not take the rack offline. That redundancy is a design assumption, not a luxury, and it doubles the leaf count and the uplink fiber you have to plan for. The rack guide covers how the leaves mount and where the cable management has to go.

The spine switch

The spine is the high-speed layer that connects all the leaves to each other, and its only job is to forward traffic between leaves as fast as possible. A spine switch does not connect to servers and it does not connect to other spines. Every one of its ports goes down to a leaf. A spine is, in effect, a big fast crossbar that any leaf can reach to get to any other leaf.

Because the spine's job is pure forwarding, the configuration on it is simple and almost identical from one spine to the next. The complexity in a modern fabric lives at the leaf, where the servers attach and where the overlay terminates. The spine just moves packets. That simplicity is deliberate. It keeps the layer that carries all the cross-rack traffic boring and predictable.

The number of spines is a capacity and resilience decision. More spines means more parallel paths between any two leaves, more total bandwidth, and a smaller hit when one spine drops. A fabric with four spines loses a quarter of its leaf-to-leaf capacity when a spine fails, which is usually acceptable. The port count on the spine sets the ceiling on how many leaves, and therefore how many racks, the fabric can hold before you need a larger spine or a second tier above it.

Should the switch go top-of-rack or end-of-row?

Top-of-rack puts the leaf in each cabinet so servers connect with short cables inside the rack, while end-of-row and middle-of-row put the switches in a cabinet at the end or center of the row and run structured cabling back to each server cabinet. The choice changes the cable plant more than it changes the logic, and it is one to settle with the cabling designer early.

ToR is the common modern default for server fabrics. Server connections stay inside the cabinet on short twinax or fiber, often under a couple of meters, which is cheap and keeps the high-speed copper runs short. ANSI/TIA-942 holds those point-to-point ToR runs to no more than about 10 meters. The cost is switch sprawl: every rack carries its own pair of switches, and a half-empty rack still pays for two ports' worth of leaf.

End-of-row and middle-of-row trade that for switch consolidation. Fewer, larger switches sit in the row and serve many cabinets through a structured copper and fiber plant with patch panels, so any server in the row reaches the same switch and you fill switch ports more efficiently. The cost is a much larger horizontal cable count and longer runs. In TIA-942 terms, ToR switches live in the equipment distribution area with the servers, while end-of-row switches live in the horizontal distribution area. The structured cabling guide covers those spaces and the media that connects them.

What oversubscription ratio should the fabric run?

Oversubscription is the ratio of a leaf's downlink capacity facing the servers to its uplink capacity facing the spines, and it tells you how much the rack can be congested if every server transmits at once. A 3:1 leaf has three times more server-facing bandwidth than uplink bandwidth, which is a common general-purpose figure. A 1:1 leaf, where downlink equals uplink, is a non-blocking fabric: no congestion in the leaf no matter how hard the servers push.

The right ratio depends entirely on the workload, and this is where designs go wrong by copying a number that fit a different room. For general enterprise compute, 3:1 is a widely used target because most servers rarely transmit at full line rate at the same instant, so the uplinks are not the limit in practice. For storage traffic and especially for AI training, where every node really does blast at line rate together, the target moves toward 1:1, fully non-blocking, because anything less throttles the workload and you have paid for GPUs that sit waiting on the network.

Work the math on real port speeds, not on a rule of thumb. A leaf with 48 server ports at 25 gigabit has 1,200 gigabit of downlink. Six 100-gigabit uplinks give 600 gigabit up, which is 2:1. The ratio is the design, so set it against the workload and confirm the leaf model can carry the uplink count you need. Vendor port counts and speeds control what is actually buildable.

The physical cabling the fabric needs

Spine-leaf trades a small core for a large mesh of leaf-to-spine links, and the fiber count climbs with it. In a fully connected fabric, every leaf runs a separate uplink to every spine, so the count is leaves times spines, doubled if the leaves are paired. A modest hall with 40 leaf pairs and 8 spines is already hundreds of uplink fibers before a single server is plugged in. Under-plan that fiber and the fabric cannot grow without a second pull.

The links split into two populations. Server-to-leaf runs are short and live inside or near the rack, so they are usually copper twinax or short-reach fiber, cheap and dense. Leaf-to-spine runs are longer, faster, and almost always fiber, and they cross the room from every rack to the spine location. That second population is what drives the fiber trunk count, the pathway fill, and the structured cabling design.

Plan the leaf-to-spine fiber as structured trunks with patch panels at both ends, not as a tangle of jumpers run point-to-point across the room. A pre-terminated trunk plant lets you add a spine or a leaf by patching, not by pulling new fiber through a full tray. The structured cabling guide covers the trunk and connector choices; the point here is that the fabric topology decides the fiber count, and the cabling design has to be sized for the full build, not the first row.

Fiber, speeds, and the optics that carry them

Fabric speeds have climbed fast: 10 and 25 gigabit at the server, 40 then 100, and now 200, 400, and 800 gigabit on the uplinks and spines, with 1.6 terabit optics entering early deployment. The migration is uneven across a fleet, so a real room often runs 25-gigabit servers into 100-gigabit uplinks into 400 or 800-gigabit spines at the same time. Pick the speed per layer against the workload and the switch generation, because the optics and the fiber plant have to match it.

Two fiber types carry it. Multimode is cheaper per port over short distances and is common for in-row and shorter uplink runs. Singlemode costs more at the optic but reaches far longer and has more headroom for future speed bumps, so many builds standardize on singlemode for the leaf-to-spine plant to avoid re-cabling at the next speed step. The reach you need and the speed roadmap decide it, not the day-one optic price alone.

Speeds, reaches, and module support move every cycle and depend on the switch and optic vendor, so treat any specific figure here as the design's to confirm. What does not change is the principle: the optic, the fiber type, and the connector have to agree end to end. A 400-gigabit module on one end of a fiber that was certified for a slower, different optic is a link that trains down or does not come up, and you find it at turnover, not in the plan.

DAC, AOC, and transceivers

Short links inside the rack do not need a transceiver and a fiber. A direct-attach copper cable, DAC or twinax, is a fixed copper assembly with the connectors built on both ends, cheap and low-latency, and it is the standard for server-to-leaf and for a leaf to a nearby spine within a few meters. The limit is distance. Copper at high speed runs out of reach quickly, so DAC is an in-rack and adjacent-rack solution, not a cross-room one.

When the run is too long for copper but you want a single assembly, an active optical cable, AOC, has the optics built into both ends and fiber in between, so it reaches farther than DAC while plugging in like a cable. Beyond that, the general case is a pluggable transceiver on each end with structured fiber between them, which is what the leaf-to-spine plant uses because it patches and re-routes through panels.

The transceiver form factors are worth knowing by name. QSFP and its denser successor QSFP-DD carry the common 100 and 400-gigabit links. OSFP, the octal form factor, runs eight electrical lanes and is the common choice for 800 gigabit because it has the power and thermal budget the higher speeds need. The form factor, the speed, and the cage on the switch all have to agree. The exact module support is vendor-specific, so confirm the optic against the switch's compatibility list before you order a reel of them.

The AI and GPU fabric

An AI training cluster needs its own network, separate from the general fabric, because the traffic between GPUs is unlike anything else in the room. During training, every GPU exchanges gradients with the others on every step, all at once, at line rate, and the whole job runs at the speed of the slowest exchange. A few microseconds of extra latency or a dropped packet that forces a retransmit stalls thousands of GPUs together. This network is built for lossless, low-latency, full-bandwidth transport and nothing less.

That is why the GPU fabric runs non-blocking, 1:1, and why it uses RDMA, remote direct memory access, which lets one GPU read another's memory across the network without going through the operating system kernel on either end. RDMA rides one of two transports. InfiniBand is a separate lossless network technology long used in high-performance computing and the reference design for large NVIDIA clusters. RoCE, RDMA over Converged Ethernet, carries the same RDMA traffic over Ethernet so the cluster can use Ethernet switching. Which one a build uses depends on the platform and the vendor, and both are in heavy use as of 2026.

The topology is often rail-optimized, a specific Clos arrangement for GPUs. Each GPU server has several network ports, commonly four to eight, and each port connects to a different leaf, called a rail, so the GPUs at the same position across many servers share a rail and reach each other with the least hops. It is still spine-leaf underneath, sized for the worst case rather than the average, because in AI the worst case is the normal case.

What is the difference between the front-end and back-end network?

The front-end network is the general-purpose fabric that handles north-south traffic and ordinary server connectivity, while the back-end network is the dedicated high-speed, low-latency fabric that connects the GPUs to each other for training. An AI build runs both, physically separate, and they have almost nothing in common except that both are spine-leaf.

The front-end is the network you already know: servers, storage front-ends, management access, the path out to users and the internet. It carries the data going into and coming out of the cluster, the storage reads that feed training, the job scheduling, the results. It runs at normal fabric speeds and a normal oversubscription ratio, because its traffic is bursty and tolerant.

The back-end is the GPU-to-GPU fabric described above: non-blocking, RDMA, often rail-optimized, running the fastest optics in the building. Keeping the two separate is a design rule, not a preference. Put GPU gradient traffic on the same fabric as storage and management and the bursts collide, the training stalls, and you have wasted the most expensive hardware in the room. The two networks are sized differently, often run different transports, and are cabled as two distinct plants. Mixing them is one of the more expensive mistakes in an AI build.

Underlay, overlay, and EVPN-VXLAN

Modern fabrics run in two layers of logic on the same physical switches: the underlay is the physical spine-leaf network that moves packets, and the overlay is a virtual network built on top of it that carries the tenants and the services. The split lets the physical network stay simple and stable while the logical network changes constantly above it.

The underlay is a plain routed IP network. Every leaf-to-spine link is a Layer 3 connection, usually running BGP, and equal-cost multipath spreads traffic across all the spines. That is all the underlay does: get a packet from any leaf to any leaf, fast and loop-free, with no Spanning Tree anywhere in it. It is deliberately dumb.

The overlay is where EVPN-VXLAN comes in. VXLAN wraps each tenant's traffic in a tunnel between leaves, so a virtual machine can sit on any rack and still appear to be on the same Layer 2 segment as another machine across the room. The leaves act as the tunnel endpoints, called VTEPs. EVPN is the control plane that tells every leaf where every endpoint lives so the tunnels know where to send traffic. The result is workload mobility and multi-tenant isolation over a stable routed underlay. The spine stays simple and just forwards; the intelligence lives at the leaf.

The edge: meet-me room and border leaves

Traffic that leaves the fabric goes out through dedicated leaves, commonly called border leaves, that connect to the wide-area network and the carriers, and in a colocation or large facility those carrier connections land in a meet-me room. The fabric handles everything inside the building. The border leaves and the meet-me room handle the handoff to everything outside it.

A border leaf is an ordinary leaf with a special role. Instead of facing servers, it faces routers, firewalls, and the WAN circuits, and it carries the north-south traffic in and out of the fabric. Keeping that function on dedicated leaves means the edge routing and security policy lives in one known place rather than scattered across the fabric, and you scale the edge by adding border leaves the same way you scale compute by adding server leaves.

The meet-me room is the neutral space where the carriers and the building's network cross-connect, usually a separate, secured room with its own cabling and redundancy. It is where the diverse circuits from different providers terminate so a single carrier outage or a single conduit cut does not take the facility off the network. The cabling and the spaces around the edge follow the same structured discipline as the rest of the room, covered in the cabling guide.

Redundancy and dual-homing

Resilience in a spine-leaf fabric comes from having many parallel paths, not from a single backup that waits for a failure. There are multiple spines, so losing one drops a fraction of the capacity and none of the connectivity. There are paired leaves in each rack, so a leaf failure or a planned reboot does not isolate the servers. The fabric degrades a little under a failure instead of going dark.

Servers reach that resilience by being dual-homed: each server connects to both leaves in its rack, so either leaf can carry it alone. The two leaves present as one logical switch to the server, or the server runs its two links as an active pair, depending on the design. Either way, a single cable, optic, port, or leaf can fail without taking the server offline. That is why the leaf count and the uplink fiber double, and why the cabling plan has to carry both legs on diverse paths.

The thing that undoes all of it is shared fate you did not notice. Two leaves in the same rack on the same power feed, or both server links through the same conduit, and a single event takes out the pair you built to be redundant. Diversity has to be real all the way down to the power and the pathway, not just present on the network diagram.

The out-of-band management network

The out-of-band management network is a separate, small network that reaches the console and management ports of every switch and server independently of the production fabric, so you can still get to a device when the fabric it lives on is broken. It is not optional in a serious build. The day you need it most is the day the main network is down, and an in-band-only management path is gone exactly when the data fabric is.

It is usually a simple, cheap Ethernet network, a 1-gigabit switch per row or per zone, wired to the management port, the baseboard management controller, and the serial console of every device. Through it you can reload a switch that dropped off the fabric, push firmware, watch a device boot, or recover a misconfiguration that cut the production path. It runs on its own gear and its own cabling, kept deliberately separate.

Skip it to save a few switches and you will pay for it the first time a fabric upgrade goes wrong at two in the morning and the only way to the switch is a path that no longer exists. The out-of-band network is cheap insurance, and the time to install it is during the build, not after the first outage.

The storage network

Storage traffic in a data center either rides the same Ethernet fabric as everything else or runs on a dedicated storage network, and which one depends on the storage type and how sensitive the workload is to a stalled read. The trend is toward carrying storage on the converged Ethernet fabric, but high-performance and latency-sensitive storage still often gets its own path.

A traditional storage area network, a SAN, connects servers to block storage, classically over Fibre Channel on its own dedicated switches and cabling, isolated from the data network entirely. Network-attached storage, NAS, serves files over the regular IP network. Many modern builds run storage over the same Ethernet fabric using protocols built for it, which simplifies the cabling but means the storage traffic now shares the fabric and the oversubscription budget with everything else.

The point that matters for the fabric design is that storage traffic is heavy and often latency-sensitive, much like AI traffic. If it shares the production fabric, the oversubscription ratio has to account for it, and a fabric sized for bursty server traffic can choke when storage replication or an AI training job pulls a dataset at full rate. Size for the storage load, or give it its own path.

Cable management and the fabric pathways

Spine-leaf moves the cable problem from a few thick core links to a dense mesh of leaf-to-spine fiber, and the pathways have to be sized for the full mesh, not the first row of racks. Hundreds of uplink fibers cross the room from every rack to the spines, and if the trays and the under-floor or overhead pathways were sized for the day-one count, the fabric cannot grow without re-working the cable plant.

Bad cable management is not just ugly. A matted bundle of fiber blocks airflow, hides which link is which, and turns a single optic swap into a half-hour of tracing. Fiber has a minimum bend radius, and a sharp bend or a cable crushed under a bundle attenuates the signal enough to bring the link down or, worse, make it intermittent. Intermittent links are the ones that cost days to find. Patch the fabric through structured panels and label every link to the standard, so a tech finds the right cable by reading it, not by pulling on it.

This is where the network design and the cabling design have to meet. The fabric topology sets the fiber count, the rack layout sets the run lengths, and the pathway has to carry both at full build with room to add. The structured cabling guide covers the labeling, the trunks, and the pathway design; the network engineer's job is to hand the cabling designer the real, full-build link count, not the phase-one number.

How do you scale a spine-leaf fabric?

You scale a spine-leaf fabric two ways: add leaves to connect more racks, and add spines to add bandwidth and resilience between them. Adding a leaf adds a rack of capacity and consumes one port on every spine. Adding a spine adds a parallel path to every leaf and more total cross-fabric bandwidth. The fabric grows in both directions until it runs into the port count on the spine switches.

That port count is the ceiling. A single tier of spines can only connect as many leaves as the spines have ports, so a fabric built on 32-port spines tops out at 32 leaves on a fully connected design. When you need more racks than one tier of spines can hold, you go to a three-stage Clos: add a layer of super-spines above the spines and group the spine-leaf blocks into pods, where each pod is its own spine-leaf fabric and the super-spines tie the pods together. The pattern repeats; it is Clos all the way up.

The design move that saves you later is to plan the spine port count for the building you will have, not the room you are opening with. Spines are the constraint, so a fabric that starts on small spines and fills them forces a forklift of the whole spine layer to grow, while one that starts on larger spines with empty ports grows by patching in leaves. Size the spine for the full hall.

The field install: switches, fiber, labels, and test

On the install side, the fabric is switches racked and grounded, fiber and DAC run and dressed, every link labeled, and every link tested before a server lands. The design is a diagram. The install is whether the diagram is true in the room, and the gap between them is found at test, not at turnover, if the testing is done.

Rack and ground the leaves and spines to the rack and bonding standard, then run the cabling as two efforts: the short server-to-leaf copper or fiber inside the racks, and the structured leaf-to-spine fiber trunks across the room. Label both ends of every link to the labeling standard as you go, because a fabric with hundreds of near-identical fibers is unworkable without it. Dress the cable to keep the bend radius and the airflow, and leave the service loops the cabling design calls for.

Then test, and keep the results. Certify the fiber and copper to the cabling standard, confirm every optic trains to its rated speed, and check the link error counters under load before you call it done, not the day after the servers move in. A fiber that passes a continuity check can still fail at speed because of a bend, a dirty connector, or a mismatched optic. The far more common field failure than a dead link is a dirty fiber endface, so clean and inspect every connector before you mate it. The cabling guide covers the certification and the turnover package.

Data center network vs enterprise network

A data center fabric and an enterprise campus network solve different problems, and a design that fits one is wrong for the other. The enterprise network connects people and their devices, so it is built around access for users, wireless, and the traffic going out to the internet, mostly north-south, and it tolerates a layered three-tier design because that traffic pattern still fits a pyramid.

The data center network connects machines to each other, so it is built around server-to-server bandwidth, predictable low latency, and east-west traffic at a scale a campus never sees. That is what pushes it to flat spine-leaf, non-blocking ratios, RDMA for the AI fabric, and fiber counts an office would never run. The campus optimizes for reach to many scattered users. The data center optimizes for bandwidth and latency between dense, co-located machines.

The mistake is carrying habits across. Build a data hall with a three-tier campus design and Spanning Tree and you get the tromboning and the idle links that the fabric was invented to kill. The east-west load that defines a data center is exactly what the campus design handles worst.

Standards and the physical layer

Spine-leaf itself is an architecture, not a single published standard, so the rules that govern a real fabric come from several places. The Clos topology is the mathematical model from Charles Clos's 1953 work. The Ethernet speeds and the optics are IEEE 802.3 standards, which define what 100, 400, and 800-gigabit Ethernet and their modules actually are. The transport for the AI fabric, InfiniBand, has its own specification, and RoCE rides standard Ethernet.

The physical layer is where the published cabling standards bite. ANSI/TIA-942 defines the data center spaces, the main and horizontal distribution areas, the equipment distribution area, and the reach limits like the roughly 10-meter cap on ToR point-to-point runs. ANSI/TIA-568 governs the structured copper and fiber. BICSI publishes design and installation practice for the cabling and the spaces. These are the standards the cabling design and the install are held to, and they are covered in the structured cabling guide.

Specific speeds, oversubscription ratios, optic support, and reach numbers depend on the switch and optic vendor and on the project design, and they change every cycle. Treat the figures in this guide as the common shape of current practice, and confirm the exact numbers against the switch vendor's documentation, the optic compatibility list, and the adopted edition of the cabling standards before you build to them.

What to document

At three in the morning, an undocumented fabric turns a mystery outage into hours of guesswork, while a real as-built record drops it to a ten-minute fix. Capture the topology and the per-link detail, not just a tidy diagram, because the diagram is the design and the record has to be what was actually built.

Record the fabric layout, the leaf and spine inventory with models and firmware, the oversubscription ratio per leaf and the math behind it, every link with both endpoints and its speed and media, the optic type on each end, the underlay addressing and routing, the overlay and tenant design, the out-of-band management addressing, and the test and certification results for the cable plant. If the AI back-end and front-end are separate, document them as two networks, because someone will need to know which is which when one of them breaks.

Common mistakes

• Building a three-tier, Spanning-Tree design for a modern east-west load and inheriting the tromboning and idle links it causes.
• Setting the wrong oversubscription ratio: a 3:1 fabric under a storage or AI load that needs 1:1, so the workload throttles on the network.
• Under-cabling the leaf-to-spine fiber for the first row instead of the full build, so the fabric cannot grow without a second pull.
• Putting GPU back-end traffic on the same fabric as front-end and storage, so the bursts collide and stall the training.
• Single-homing servers or running paired leaves on shared power or a shared conduit, so the redundancy is only on the diagram.
• Leaving out the out-of-band management network and losing the only path to a switch the moment the production fabric breaks.
• Poor cable management and unlabeled fiber, so a bent or dirty link is intermittent and an optic swap takes a half-hour of tracing.
• Mismatching optics and fiber across a link, so it trains down or never comes up and you find it at turnover.

Field checklist

Units, terms, and conversions

The fabric vocabulary overlaps with the cabling and the AI worlds, so the same idea shows up under different names across a switch datasheet, a cabling drawing, and a GPU reference design.

Speeds are given in gigabit per second, GbE or G, and now terabit, with 1.6 terabit written 1.6T or 1600G. Leaf is the access or top-of-rack switch; ToR is the same thing named for its location. Spine is the aggregation or fabric layer. Fabric and Clos and folded-Clos all name the same two-tier mesh. East-west is server-to-server; north-south is in and out of the building. Oversubscription is written as a ratio of downlink to uplink, like 3:1, where 1:1 means non-blocking.

Leaf / ToR

The access switch connecting servers in a rack, usually the top-of-rack switch, uplinking to every spine

Spine

The fabric layer that forwards traffic between leaves; connects to no servers and no other spine

Clos / fabric

The non-blocking multistage topology, after Charles Clos, that spine-leaf applies to Ethernet

East-west / north-south

Server-to-server traffic inside the data center versus traffic in and out of the building

Oversubscription

Ratio of a leaf's server-facing bandwidth to its uplink bandwidth; 1:1 is non-blocking

RDMA / RoCE / InfiniBand

Remote direct memory access for GPU fabrics, carried over Ethernet (RoCE) or InfiniBand

EVPN-VXLAN

The overlay that tunnels tenant traffic between leaves over the routed underlay

Underlay / overlay

The physical routed spine-leaf network and the virtual tenant network built on top of it

Related tools

Calculators and readiness checks for this work

Compare your options

FAQ

People also ask

• What is data center structured cabling?
• What is a rack unit (U)?
• What is network observability on an AI fabric?
• What is the GPU back-end fabric?
• Cast-in vs post-installed anchors: which should I use?