Data center network observability and monitoring field guide

What network observability is on an AI fabric

Network observability is seeing what the network is actually doing in detail, not just whether it is up. On an AI cluster that means the telemetry, the congestion, the queue depths, the errors, the latency, and the packet loss across every link in the fabric, at a resolution fine enough to catch a problem that lasts milliseconds. The point is to answer the hard question, which is not is the link up but what is this link doing and why is the job slow.

It matters more here than on a normal enterprise network, and the reason is the workload. A GPU training run is a synchronized job. The GPUs all hit a collective operation, an all-reduce or an all-gather, where every rank has to exchange data before any of them can take the next step. The whole job moves at the speed of the slowest path. One congested link, one optic that is fading, one port quietly dropping a fraction of a percent of its packets, and thousands of GPUs sit idle waiting on the straggler. Studies of large training runs put a substantial share of job completion time in network communication, roughly a fifth to a third on modern high-bandwidth fabrics and higher on slower ones, so the link that slows the collective slows the entire run.

This guide is about the network itself, the telemetry and the signals that find the slow link. The shape of that fabric, the spine and leaf switches and the separate AI back-end, is covered in the spine-leaf architecture guide. The facility side, the power and cooling and asset tracking, is the DCIM guide. Read those alongside this one. A monitoring program is only as good as the fabric design it watches and the facility data it sits next to.

The slow-link problem: the whole job goes as fast as the worst path

This is the one idea that changes everything about monitoring an AI fabric, so it goes first. A synchronized collective finishes when the last rank finishes. Not the average rank, the last one. If 4,095 GPUs complete their part of an all-reduce in the budgeted time and one is stuck behind a degraded link, all 4,096 wait for that one. The fast paths buy you nothing. The job runs at the speed of the worst link on the critical path.

Operators call the slow node the straggler, and on an AI cluster the straggler is often not a slow GPU at all. It is the network underneath one. A link running a half step below its rate, a buffer that overflows under load, a connection dropping packets and forcing retransmits, any of these turns a healthy GPU into a straggler because the data it needs shows up late. The compute looks fine on the dashboard. The job is still slow.

So the monitoring goal is blunt. Find the worst link before it costs you a training run, and find it fast, because every minute a 4,096-GPU job is bottlenecked is a minute of very expensive hardware producing nothing. Average-based, slow-polling monitoring is built to tell you the network is broadly healthy. It is the wrong tool for finding the one path in tens of thousands that is dragging the job. That gap is what high-resolution observability exists to close.

Why an AI fabric is different from a normal network

A normal enterprise or cloud network carries many small, independent, bursty flows that tolerate loss and jitter without anyone noticing. TCP retransmits a dropped packet, a web page loads a few milliseconds later, and no human cares. Average utilization is a fine health metric because no single flow holds the whole system hostage. An AI back-end fabric behaves nothing like that.

The traffic is synchronized, not independent. Thousands of NICs at 200, 400, or 800 Gbps fire at the same instant when a collective starts, all heading for the same set of peers, and nothing proceeds until the last byte of the step arrives. The flows are huge and long-lived, called elephant flows, instead of small and bursty. And the tolerance for loss and delay is far lower, because the cost of one slow path is multiplied across every GPU waiting on it.

That combination makes two things matter that a normal network can ignore. Tail latency, the worst case rather than the average, is what sets job completion time, because the collective waits on the tail. And quiet packet loss, the kind that hides as a rounding error in a utilization graph, kills throughput, because every lost packet forces a retransmit that stalls the pipeline the whole job is waiting on. On an AI fabric you monitor the tail and the loss, not the average.

What is the difference between monitoring and observability?

Monitoring answers is it up. Observability answers what is it doing and why. The distinction is not marketing. It is the difference between a tool that watches a fixed set of known signals and tells you when one crosses a line, and a tool that gives you enough detail to ask a question you did not predefine, such as which link slowed the all-reduce at 02:14.

Classic monitoring tracks interface status, average utilization, and a handful of counters, then alarms when a port goes down or a threshold trips. That is necessary and it is not enough for an AI fabric. A port that is up, lightly loaded on the five-minute average, and degrading the job through microbursts and quiet loss passes every up-or-down check while it strangles the collective. The monitor says green. The training run says slow.

Observability adds depth. High-resolution streaming telemetry, per-queue visibility, flow-level attribution, optics health, and the ability to correlate all of it back to the job that suffered. The practical test is whether you can sit down after a slow training run, with no alarm having fired, and find the link that caused it. If the data is too coarse or too shallow to answer that, you have monitoring, not observability.

Streaming telemetry versus SNMP polling

Streaming telemetry is the switch pushing its counters out continuously at sub-second intervals, instead of a server polling the switch every minute or five for one value. On an AI fabric the push model is not a preference. It is the only approach with enough resolution to see the events that slow the job, so treat slow polling as a known blind spot.

SNMP is pull-based and built for a slower era. The collector asks each device for its counters on a schedule, commonly every 30 seconds to 5 minutes, and the result is one averaged number for the whole window. That is fine for capacity trending and it is useless for catching a queue that overflowed for 50 milliseconds. A spike that saturated a buffer and dropped packets for a fraction of a second gets averaged into a flat, healthy-looking line. The event is real and the graph hides it.

Streaming telemetry, commonly over gNMI and gRPC against an OpenConfig or vendor model, pushes structured data at second or sub-second resolution so the congestion, the queue depth, and the drop show up while they are happening. The exact transport, the data models, and the achievable resolution depend on the switch platform and the NIC vendor, so confirm what your gear actually streams and at what rate before you design the collection around it. Keep SNMP if you like for the slow, broad inventory view. Do not rely on it to find the link that stalled a GPU job.

Congestion, queues, and buffer depth

Congestion on an AI fabric is not a slow pipe. It is a moment when more traffic arrives at a switch port than the port can send, so it piles up in the buffer, and when the buffer fills the switch either pauses the sender or drops the overflow. Both outcomes slow the collective. The signal you want is queue depth, how full the buffers are, watched over time at high resolution.

It builds in predictable places. Incast is the classic one: many senders hit the same receiver at once, which is exactly what a collective does, and the buffer at the last switch before the receiver takes the hit. Oversubscribed uplinks are another, where more rack traffic wants to go up than the uplinks can carry. The visibility that matters is per-queue occupancy and the count of times a queue crossed its threshold, not just the port's average utilization, because a port can average 40 percent and still overflow a queue in a burst.

Watch the buffers, not just the link rate. A fabric can look uncongested on utilization and be dropping or pausing in the queues every time a collective fires. The thresholds where a queue becomes a problem depend on the switch buffer architecture and the congestion-control design, so set them against the vendor's guidance and your own measured baseline rather than a generic number.

What is a microburst, and why slow polling misses it

A microburst is a spike of traffic that saturates a link or overflows a buffer for a tiny slice of time, often well under a second, then is gone. It is the signature failure on an AI fabric because synchronized collectives produce exactly this shape: every NIC fires at once, the buffer at a chokepoint fills in microseconds, packets get paused or dropped, and the average utilization barely moves.

Slow polling cannot see it, and this is the heart of why SNMP is the wrong tool here. A counter read once a minute gives one rate for the whole minute. A burst that lasted 50 milliseconds, even one that filled a queue and dropped packets, is a rounding error against 60 seconds of averaging and disappears from the graph. You get a flat green line over a real outage. Operators chase a slow job for days because the monitoring insists the network is fine.

Catching microbursts takes high-resolution telemetry, the queue-depth and drop counters streamed at sub-second rates, and on some platforms a dedicated microburst or buffer-histogram feature in the switch ASIC that records the peak between polls. What resolution you actually need, and what the hardware can give you, depends on the switch and the workload, so size the telemetry rate to the bursts you are trying to see and confirm the platform can sustain it.

Packet loss is the enemy of AI throughput

On an AI fabric, packet loss is not a minor degradation you ride through. It is the thing that quietly destroys throughput, and a loss rate that would be invisible on a normal network is a real problem here. The reason is the retransmit. When a packet is lost the data has to be sent again, and the collective that was waiting on that data stalls until it arrives. Multiply that stall across every GPU synchronized on the step and a tiny loss rate turns into a large hit to job completion time.

It is worse on lossless Ethernet running RDMA. RoCEv2's recovery, in its classic form, is go-back-N: lose one packet and the sender retransmits that packet and every packet after it in the window, not just the missing one. A single drop becomes a burst of resends. Newer transports and NIC features add selective retransmission to soften this, but the lesson holds. On this fabric, loss is expensive far out of proportion to its rate, so you hunt the source, the specific port and queue dropping packets, and you fix it instead of tolerating it.

So loss monitoring is not about a percentage threshold you accept. It is about finding any link that drops at all under load and treating it as a fault. Watch the per-port drop and discard counters at high resolution, tie them to the queue that overflowed, and remember that the loss which hurts the job is often the loss that never showed up on a five-minute average.

Latency, tail latency, and jitter

Latency is how long a packet takes to cross the fabric, and on an AI cluster the number that decides job speed is not the average. It is the tail, the worst case, often quoted as the 99th or 99.9th percentile. The collective waits on the slowest path, so the tail latency is the long pole that sets how long the step takes. A fabric with a beautiful average and a bad tail is a slow fabric for AI.

Jitter, the variation in latency from packet to packet, matters for the same reason. Synchronized communication is sensitive to variance, because a step that finishes late once stalls the whole pipeline once. Steady and slightly higher can beat fast and erratic when the job is a tight collective. So you measure the distribution, not a single mean, and you watch how the tail moves over time.

Measuring it well is its own discipline. Interface counters do not give you path latency, so operators lean on active probes, in-band telemetry that timestamps packets through the fabric, and NIC-level measurements, with the right method depending on the switch and NIC features available. Whatever the method, watch the percentiles and the jitter, not just the average, because the average is the one number that hides the problem the job actually feels.

Monitoring lossless Ethernet (RoCE) versus InfiniBand

AI back-end fabrics run RDMA, where the NIC moves data directly into remote memory and bypasses the CPU, and RDMA assumes the network does not drop packets. Two ways to get there dominate. RoCEv2 runs RDMA over Ethernet and makes the Ethernet behave losslessly using flow control and congestion notification. InfiniBand is a separate fabric with credit-based flow control built in, where a sender only transmits when the receiver has advertised buffer space, so it is lossless by construction. The choice between them is a design decision covered in the spine-leaf guide; what changes for monitoring is what you have to watch.

On InfiniBand the fabric manager and the subnet give you a coherent, purpose-built view of port counters, congestion, and link health, and the credit-based mechanism means there is no PFC behavior to tune and watch. On RoCE you are making general-purpose Ethernet act lossless, and that lossless behavior is a set of mechanisms you have to monitor, because when they misbehave the fabric either drops packets or pauses itself into a stall.

Either way, do not assume lossless means you can stop watching for loss. A misconfigured or congested RoCE fabric drops packets, and even InfiniBand has degrading links and failing optics. The hedge that holds across both: the right counters, thresholds, and tuning depend on the fabric type, the switch and NIC vendor, and the design, so monitor against the vendor's lossless reference and your measured baseline, not a generic Ethernet playbook.

Watching PFC and ECN

PFC and ECN are the two mechanisms that make RoCE lossless, and on a RoCE fabric you watch both because each one fails in its own way. PFC, priority flow control, is a link-layer pause: when a switch's buffer for a traffic class fills, it sends a pause frame upstream telling the sender to stop for that class, which prevents the drop but freezes traffic. ECN, explicit congestion notification, is the gentler, end-to-end signal: a congested switch marks packets instead of dropping them, the receiver tells the sender it saw congestion, and the sender slows down before the buffer overflows.

You need both, and you need to watch what each is doing. ECN alone and a sudden burst still overflows and drops. PFC alone and you get head-of-line blocking, where one congested queue pauses a whole link and the pause propagates back through the fabric, stalling senders that were not even part of the congestion. The worst case is PFC pause storms that spread and seize up large parts of the fabric, which is why pause-frame counts are a signal you watch closely.

Practically, monitor PFC pause frames sent and received per port and per class, ECN marked-packet counts, and the congestion-notification responses, and watch whether PFC is firing constantly, which means the ECN tuning is too slow and the fabric is leaning on pause as a crutch. The thresholds and the tuning, the ECN marking points and the congestion-control algorithm, are vendor-specific and design-specific, so set them from the switch and NIC reference and confirm against your baseline rather than copying numbers between fabrics.

Optics and transceiver health

The optics are where a slow degradation hides before it becomes a hard failure, and on a fabric with tens of thousands of transceivers running at 400 and 800 Gbps, watching them is some of the highest-value monitoring you can do. Every modern optical module reports its own health through digital diagnostics, commonly called DDM or DOM, and that data is your early warning for a link that is about to start hurting the job.

The parameters to stream are transmit power, receive power, laser bias current, module temperature, and voltage. The pattern that predicts trouble is a trend, not a single reading: receive power drifting down toward the sensitivity floor, transmit power sagging, bias current climbing as the laser ages and works harder to hold its output, temperature creeping up. A module showing those trends will eventually start producing errors and flapping, and a flapping link in an AI fabric drops the job's throughput every time it bounces.

The win is catching it before it flaps. Trend the DDM values per module across the fleet, alert on the drift and on readings approaching the vendor's warning and alarm thresholds, and pull a suspect optic on a planned maintenance window instead of during a training run. The exact thresholds and what counts as a failing trend depend on the module type, the reach, and the vendor's diagnostic specification, so trend against the module's own limits and your fleet baseline. This ties to the structured-cabling and optics work in the fabric build, where the physical link and its budget are set.

Link errors: CRC, symbol errors, and the degrading link

A link rarely fails clean. It degrades first, and the early sign is errors: CRC errors on received frames, symbol errors on the physical lane, and the corrected and uncorrected counts from forward error correction on the high-speed lanes. These are the fingerprints of a link that is still up and passing traffic but corrupting some of it, which on an AI fabric means retransmits and a stalled collective.

The signal is the trend and the rate, not the raw total. Every link accumulates some corrected errors over time and that is normal. What matters is a rate that is climbing, or uncorrected errors that should be near zero starting to appear, because that is the link sliding toward the point where it drops packets or flaps. A dirty connector, a marginal optic, a cable bent past its radius, or a seating problem all show up here first.

Watch the error counters at high resolution and alert on the slope, the change over time, against a baseline of what each link normally does, rather than a single fixed number. Pair the error trend with the optics DDM trend on the same link and you can usually tell whether you are chasing a transceiver, a connector, or a cable before the link ever takes the job down.

Flow visibility, elephant flows, and ECMP imbalance

Per-link counters tell you a port is busy. Flow-level visibility tells you which conversation is making it busy, and on an AI fabric that distinction matters because the traffic is dominated by elephant flows, the huge long-lived transfers between GPU pairs during a collective. A handful of those flows can saturate a link while the port count looks like ordinary load.

The specific failure to watch for is ECMP imbalance. The fabric spreads traffic across equal-cost paths by hashing each flow onto one path, which works beautifully for many small flows and badly for a few enormous ones. When two elephant flows hash onto the same path, that link is overwhelmed while parallel links sit idle, and the unlucky flow's tail latency blows out. Measurements of elephant traffic under plain ECMP have shown throughput collapsing toward roughly 60 percent of the link because of these hash collisions. The collective then waits on the congested path.

So flow visibility is how you see the imbalance: which flows landed on which paths, where collisions are happening, and which links are hot while their peers are cold. That visibility is also what justifies the load-balancing schemes that fight this, adaptive routing and packet-level spraying among them, whose behavior you then have to monitor too. The available flow telemetry, sFlow, IPFIX, in-band methods, and vendor flow features, depends on the platform, so confirm what your fabric exports before you build the analysis on it.

Network observability versus facility DCIM

There are two monitoring worlds on a data center floor and they answer different questions, so keep them straight. Network observability, this guide, watches the fabric: the links, the queues, the loss, the latency, the optics, the things that decide whether the GPU job runs fast. Facility monitoring, DCIM, watches the building: the power chain, the cooling, the space, the assets, the things that decide whether the room stays powered and within temperature. One is the traffic on the wires. The other is the watts and the degrees.

They are separate by design and they overlap at the edges. A cooling problem that lets a row run hot will show up in the network world as optics temperature climbing and error rates rising, because a transceiver baking in a hot aisle degrades. A power event that drops a rack takes its switches and NICs with it. The fabric does not exist apart from the facility, and the most useful operators can look across both, which is why these guides cross-reference each other.

Do not try to make one tool be both. DCIM is built for the power-and-cooling problem and is covered in its own guide. Network observability is built for the fabric problem and is this one. Run both, connect them where the signals genuinely cross, such as a thermal event that explains a cluster of optics alarms, and keep the boundary clear so neither view gets diluted into a dashboard that watches everything and finds nothing.

The tooling: collectors, time-series, and analytics

The shape of an observability stack is consistent even though the products vary, so think in layers rather than brands. At the bottom is collection: the agents and collectors that take the streaming telemetry, the flow records, and the device data off the fabric. Above that is storage, usually a time-series database tuned to ingest a very high rate of metrics and hold enough history to compare today against last week. On top is the analytics and the visualization, the dashboards, the queries, and the alerting that turn the raw streams into something an on-call engineer can act on.

The newer layer, and the one to weigh carefully, is analytics that apply machine learning to the telemetry, sometimes pitched as AI for network operations. Used well it spots an anomaly against a learned baseline faster than a human watching graphs and surfaces a correlation a person would miss. Used badly it is a black box that fires confident alerts nobody can trace. Treat it as an aid to the engineer, not a replacement, and judge it by whether it points at the actual slow link.

There is no single right product, and the fit depends on the fabric vendor, the scale, and what your switches and NICs actually stream, so match the stack to the gear rather than to a vendor's pitch. Plenty of strong stacks combine an open collector, an open time-series store, and a vendor or open analytics layer. The questions that matter are whether it ingests at the resolution your fabric needs, holds the history to baseline against, and lets you correlate the network to the job, not whose logo is on the dashboard.

Baselines and anomaly detection

You cannot tell that a link is misbehaving if you never learned what it looks like behaving. The baseline is the recorded picture of normal: what the queue depths, the error rates, the latencies, the pause counts, and the optics readings look like across the fabric over a representative span of real workloads. Without it, every alert is a guess and every threshold is a number somebody copied from a slide.

Anomaly detection is then the deviation from that baseline. Sometimes the right tool is a simple static threshold, a hard line for a counter that should always be near zero, like uncorrected FEC errors. Often it is a comparison: this link is dropping while its identical peers are clean, or this port's latency tail just stepped up from where it has sat for a month. The deviation between similar things is frequently a stronger signal than any absolute number, because the fabric is full of near-identical links that should behave alike.

Build the baseline from your own fabric under your own workloads, refresh it as the cluster and the jobs change, and let the anomaly logic lean on peer comparison and trend as much as on fixed limits. What normal is depends on the design, the vendor, and the traffic, so the baseline is yours to measure, not a constant to look up.

Correlating the network to the GPU job

The hardest and most valuable thing observability does on an AI fabric is connect a network event to the job it slowed. A training run reports that step times jumped at 02:14. The network has to be able to answer which link, which queue, which optic, which flow, in that window, was the cause. Without that link, the network team and the ML team blame each other and the slow job stays slow.

The way you get there is shared time and shared identity. Stamp the network telemetry and the job telemetry to the same clock so a GPU stall and a queue overflow can be lined up to the same instant. Tie flows to the workload that owns them so a congested elephant flow points back to a specific job and rank. Then a stall shows its fingerprints: a GPU's data supply dried up and its power draw dropped at the same moment a particular link started pausing or dropping. Recent work on this lines up GPU latency against host and network signals by cross-correlating the spikes to find the temporally related cause.

This is also where the up-or-down monitor fails hardest. It never fired, because nothing went down. The job was slow and the network looked green, and only correlation across the two worlds finds the link that was quietly stalling the collective. Build for that question from the start, because it is the question you will actually be asked after a slow run.

Automated remediation and draining a bad link

The emerging end of fabric observability is closing the loop, where the system that detects the bad link also acts on it without waiting for a human. The clearest case is draining a degrading link: the telemetry flags a port whose errors are climbing or whose optic is fading, and the fabric automatically shifts traffic off it and routes around it, so the job stops paying for the straggler while a tech is dispatched to swap the part on a planned window.

This is real and it is early, so calibrate the trust accordingly. Automatically draining a link is safe and increasingly common because routing around one port rarely makes things worse. More aggressive auto-remediation, retuning congestion control or reconfiguring transport on the fly, is promising and not something to hand the keys to blindly on a production cluster. The blast radius of a wrong automated action on a fabric carrying a multi-week training run is large.

The sane path is a closed loop with a short leash: automate the detections and the low-risk responses like draining a clearly bad link, keep a human in the loop for the changes that could ripple, and log every automated action so you can audit what it did and why. What can be safely automated depends on the fabric vendor's tooling and your own confidence in the baseline, so grow the loop as the detection earns trust, not before.

The telemetry scale problem

High-resolution telemetry across a large fabric is itself a large data problem, and ignoring that is how a monitoring project drowns. A fabric with tens of thousands of ports, each streaming dozens of counters and queue stats several times a second, plus per-optic diagnostics and flow records, produces a firehose of metrics. The telemetry you collect to watch the fabric needs its own capacity to collect, transport, and store, and that capacity is not free.

The tension is real: you want resolution fine enough to catch a 50-millisecond microburst, and you cannot keep every counter at that rate forever for every port without a storage bill that rivals the thing you are monitoring. The usual answer is tiered. Stream the highest-value signals, the queue depths, drops, pause frames, and optics, at high resolution, keep recent history hot for troubleshooting, and roll older data down to coarser aggregates for trending. Some platforms also do the first level of summarizing in the switch, like buffer histograms, so the collector is not asked to carry every sample.

Size the collection and storage deliberately against the fabric, because the right resolution, retention, and aggregation depend on the scale, the vendor's telemetry behavior, and the design. The failure mode to avoid is turning everything to maximum, overwhelming the collectors, and ending up with gaps in exactly the moments you needed the data.

Actionable alerting and the on-call

Telemetry that nobody can act on is just storage. The alerting layer is where observability either earns the on-call engineer's trust or loses it, and the failure mode is noise. A fabric this large, with thresholds set too tight or with no baseline behind them, generates a flood of alerts, and a flood teaches the on-call to ignore the channel. Then the one alert that mattered scrolls past with the rest.

Good alerting is tied to impact, not to raw counters. The signal that should page someone is the one that means the job is being hurt or is about to be: a link dropping under load, PFC pausing hard enough to risk a stall, an optic crossing its alarm threshold, a latency tail stepping up on a critical path. Lower-value deviations belong on a dashboard and a daily review, not on the pager. Each alert that does fire should carry enough context to start the work, which link, which queue, which job, so the responder is not starting from a blank graph at 03:00.

Back the alerts with runbooks. The value of a runbook on a fabric is that the steps to confirm and contain a microburst or a flapping optic are knowable in advance, so the on-call follows a tested path instead of improvising under pressure. Tune the thresholds against the baseline so the noise comes down, and protect the signal, because an ignored alert channel is worse than no alerting at all.

Keeping the records: baselines, incidents, and link history

Live dashboards tell you what the fabric is doing now. The record tells you what it has done over time, and on a fabric you run for years that history is what turns a recurring mystery into a solved problem. The baseline you measured, the incidents you worked, and the per-link history of errors, optics readings, and replacements are the institutional memory of the fabric, and they are worth keeping somewhere durable rather than in a chat log and a few people's heads.

The link history is the one that pays off most often. A port that has flapped three times, eaten two optics, and trended errors upward for a month is telling you something a single live reading cannot. Tracking each link's incidents, the optic serial numbers and their DDM trends, and the maintenance actions taken lets you catch the bad slot or the marginal cable run that keeps killing modules, instead of swapping parts one at a time forever.

Capturing that in the field, the link that flapped, the optic pulled and its readings, the action taken and who did it, is exactly the kind of structured record FieldOS is built to hold. Tie the telemetry-driven incident to a logged action against a specific link and asset, and the next person who works that port inherits the history instead of starting over. The observability stack shows the live signal. The field record keeps the history that explains it.

Common monitoring failures on an AI fabric

The same handful of gaps turn up across fabrics that look slow for reasons nobody can find, and they are worth naming as a group because they reinforce each other. Slow SNMP polling that averages away the microbursts. Monitoring that watches up-or-down and broad utilization but not loss and latency. Optics degradation ignored until the link flaps and takes the job with it. No baseline, so no way to tell normal from a problem. Alert noise that buries the one alert that mattered. And the big one, no correlation between a network event and the GPU stall, so a slow job and a green network sit side by side and nobody connects them.

Each gap shares a root cause, which is monitoring designed for a normal network and pointed at an AI fabric. The normal-network instincts, average utilization, five-minute polling, up-or-down alarms, are not wrong in their world. They are simply blind to the failures that matter when one slow link drags a synchronized job. Fix them as a set, because closing one while leaving the others open still leaves you chasing slow jobs you cannot explain.

What to monitor and record

The table below is the working list of signals that earn their place on an AI fabric, why each one matters, and what to keep in mind when you set it up. The thresholds are deliberately not numbers here, because the right value depends on the switch, the NIC, the optic, and the design, so set them from the vendor reference and your own baseline.

Field checklist

Common mistakes

• Relying on slow SNMP polling that averages microbursts and transient loss into a flat, healthy-looking line.
• Monitoring only up-or-down and utilization instead of the loss and latency that actually slow the collective.
• Ignoring optics degradation until the link flaps, instead of trending DDM and pulling the module on a planned window.
• Running with no baseline, so every threshold is a guess and normal cannot be told from a fault.
• Letting alert noise bury the real signal until the on-call ignores the channel.
• Failing to correlate a network event to the GPU stall, leaving a slow job and a green network side by side.
• Assuming lossless means loss-free and turning off drop monitoring on a RoCE fabric.
• Maxing every counter to full resolution, overwhelming the collectors, and losing data in the moments that mattered.

Standards and references

Network telemetry practice has settled around streaming over gNMI and gRPC with structured data models, commonly OpenConfig and vendor YANG models, alongside flow-export standards like sFlow and IPFIX for flow-level visibility. SNMP still exists and still has a place for slow inventory and capacity trending. Which models, transports, and resolutions a given fabric actually supports depends on the switch and NIC platform, so confirm the device's telemetry capabilities against its documentation before you design the collection.

The lossless-fabric mechanisms come from the relevant IEEE 802.1 work for priority flow control and from the IETF for explicit congestion notification, with RoCEv2 layering RDMA over them and congestion-control algorithms such as DCQCN coordinating PFC and ECN. InfiniBand follows its own specification with credit-based flow control and a fabric manager. The exact counters, marking points, and tuning are defined by the fabric type and the vendor's implementation, so monitor against the vendor's lossless reference design rather than a generic playbook.

Optical module diagnostics, the DDM or DOM data, come through the standardized transceiver management interfaces for the relevant form factors, and the warning and alarm thresholds are set per module by the manufacturer. The switch and NIC vendor telemetry, the optics specifications, and the fabric design together define what good looks like on your network. Three things hold across all of it: one slow or lossy link drags the whole synchronized AI job, high-resolution telemetry is what catches the loss, the microbursts, and the failing optics that slow polling misses, and correlating the network to the GPU stall is what turns a slow job into a fixed link. Treat the specific protocols and thresholds as the vendor, the fabric, and the design define them, and verify against your own measured baseline.

Units and terms

The same ideas show up under several names across switch documentation, NIC references, and monitoring tools, so the short glossary below keeps them straight.

Observability is seeing what the fabric is doing in detail and being able to ask why; monitoring is the narrower up-or-down view. Streaming telemetry is the device pushing structured metrics continuously at sub-second rates, the opposite of SNMP's slow polling. The rest of the terms below define the signals you watch and the mechanisms that produce them.

Observability

Seeing what the network is actually doing in detail, enough to answer why a job was slow, not just whether links are up

Streaming telemetry

The device pushing structured counters and stats continuously at sub-second rates, commonly over gNMI/gRPC, versus SNMP polling

Microburst

A traffic spike that saturates a link or overflows a buffer for well under a second, invisible to slow polling but enough to drop packets

Packet loss

Packets dropped in the fabric, which forces a retransmit and stalls the synchronized collective the GPUs are waiting on

Latency / tail latency

Time for a packet to cross the fabric; the tail (99th or 99.9th percentile) sets job completion time because the collective waits on the worst path

RoCE

RDMA over Converged Ethernet, running RDMA on Ethernet made lossless with PFC and ECN, versus InfiniBand's credit-based lossless fabric

PFC

Priority flow control, a link-layer pause that stops a traffic class to prevent drops but can cause head-of-line blocking and stalls

ECN

Explicit congestion notification, an end-to-end mark that tells a sender to slow down before a buffer overflows

DDM / DOM

Digital diagnostic monitoring in an optical module, reporting Tx and Rx power, laser bias, temperature, and voltage for failure prediction

ECMP

Equal-cost multi-path routing, which hashes flows across parallel paths and can overload one link when large flows collide on the same path

Related tools

Calculators and readiness checks for this work

Compare your options

FAQ

People also ask

• What is a spine-leaf network?
• What does DCIM stand for and what does it manage?
• What is a battery monitoring system?
• What is the GPU back-end fabric?
• Cast-in vs post-installed anchors: which should I use?