AI GPU cluster commissioning and burn-in field guide

What AI cluster commissioning is

AI cluster commissioning is the bring-up that proves a GPU cluster actually works, done after the building and its power and cooling have already passed facility commissioning. A cluster that powers on is not a cluster that works. Lights come up, fans spin, the operating system boots, and none of that tells you the fabric is wired to the design, that every GPU is healthy, or that the whole thing holds together under a real training run.

The work breaks into a sequence. You verify the physical build, that the racks are placed and the cables run the way the design says. You power on in stages and confirm the firmware and configuration match across the fleet. You validate the high-speed network fabric link by link. You check every GPU for health and inventory. Then you load the whole cluster, prove the cooling and power hold under that load, and run a sustained burn-in to find the weak parts before the workload does.

This is the IT and compute layer, not the building. Where facility commissioning proves the power train, the chillers, and the room, cluster commissioning proves the machines that fill it. The two are separate disciplines with separate runbooks, and the order matters: the facility is proven first, then the cluster lands on top of a plant that is already known good. The facility side is covered in the data center commissioning levels guide, and getting a space ready for these racks in the first place is covered in the AI and GPU rack readiness guide.

Facility commissioning vs cluster commissioning

Facility commissioning and cluster commissioning are two different jobs, and confusing them is how clusters get brought up on a plant that was never proven. Facility commissioning is the building: the utility feed, switchgear, UPS, generators, the chilled water and direct-to-chip cooling, and the integrated systems test (IST) that drops the utility at load and watches the whole plant ride through. That work runs as staged levels and ends before any meaningful IT load is on the floor. The levels and the IST are walked through in the data center commissioning levels guide.

Cluster commissioning is the compute. It starts after the facility is signed off and covers the GPUs, the servers, the high-speed network fabric, and the cluster software, brought up and proven as a working machine. The facility IST proves the room survives a fault with load banks standing in for the racks. Cluster commissioning proves the real racks work and then hands a known-good cluster to operations.

The line is worth holding because the failure modes are different. A facility finding is a breaker that did not transfer or a chiller that did not restart. A cluster finding is a flapping optic, a GPU throwing memory errors, or a fabric miswired so collective traffic takes the long way around. Different tools, different specialists, different acceptance documents. The integrator or OEM commissioning runbook scopes the cluster side, and the owner's commissioning plan scopes the facility side.

Why a powered-on cluster is not a working cluster

A powered-on cluster is not a working cluster because almost everything that makes a cluster usable is invisible at power-on. The node boots whether or not its NIC is linked at full speed. The GPU shows up in the inventory whether or not its memory is throwing correctable errors. The fabric forwards packets whether or not a third of the links are running degraded and silently retransmitting. None of that surfaces until you put real load on the machine and look at the right counters.

The cost of skipping the proof is paid later, at the worst time. A flaky GPU or a marginal optic that would have shown up in a day of burn-in instead shows up three weeks into a training run, kills the job, and forces a restart from the last checkpoint. At cluster scale that is real money in lost GPU-hours, and the fault is harder to isolate live than it ever was on a quiet commissioning afternoon.

Schedule pressure works against all of this. The GPUs are expensive, they are late, and everyone wants them earning. The discipline is to find the failures on a scheduled day with the cluster idle, not at 2 a.m. with a paying workload on the floor. Commissioning and burn-in is what moves the moment of failure earlier, into a test, where it is cheap to fix.

The scale problem: small failure rates, many faults

At AI scale the math turns small per-component failure rates into a large number of faults, and that is the reason commissioning a cluster is its own discipline. A training cluster is tens of thousands of GPUs, hundreds of switches, and far more cables and optical transceivers than GPUs, because every GPU needs several high-speed links and every link has connectors on both ends. Multiply a 0.1 percent defect rate across hundreds of thousands of optics and you are looking at hundreds of bad parts on day one.

Public numbers make the point. Meta reported that its 16,384 H100 training cluster for Llama 3 hit roughly one failure every three hours over a 54-day run, with about 419 unexpected interruptions. Faulty GPUs drove around 30 percent of those, HBM memory another 17 percent, and network switches and cables about 8 percent. That is a mature, well-run cluster in production. The infant-mortality rate at first power-on is higher, which is exactly what burn-in is there to absorb.

The takeaway for bring-up is statistical, not anecdotal. You are not hunting for the one bad part. You are running a process that assumes a predictable fraction of the fleet is dead or dying on arrival, finds them systematically, and replaces them before the cluster goes to work. Plan the schedule and the spares around that fraction, and confirm the expected yield with the integrator and the hardware vendor for the specific generation.

What has to be done before you commission the cluster

Before cluster commissioning starts, the facility has to be commissioned and stable. That means the power train is proven through its integrated systems test, the cooling plant holds its setpoints, and the room is delivering the temperature, the supply water, and the power quality the racks were specified for. Do not commission a cluster on an un-commissioned or shaky facility. Every cluster fault you chase will be contaminated by a facility variable you cannot rule out, and a cooling or power event during burn-in can damage hardware.

The cooling point is blunt and worth its own line. Do not power GPUs into load before the cooling loops that serve them are commissioned and proven. On direct-to-chip liquid systems a bad manifold connection or an unbled loop can cook hardware worth a fortune in minutes, and the rack does not warn you politely first. The facility side proves the loop; the cluster side confirms it is connected and flowing to the rack before any GPU sees a stress test.

The rest of the prerequisite list is logistics that decide whether the bring-up goes fast or stalls: the floor weight rating confirmed, the power feeds landed and metered, the management network up so you can reach every node, and the fabric cable plant installed and labeled. Readying the space to that standard is the subject of the AI and GPU rack readiness guide. Cluster commissioning assumes that gate is already cleared.

Verifying the rack build and the cabling against the design

The physical verification confirms the cluster was built the way it was drawn, because the fabric only works if the cables land where the topology says they should. You check that racks are in their grid positions, that compute trays and switches are in their assigned rack units, and that every cable runs from the right port on the right device to the right port on the right device. On a fat-tree or rail-optimized fabric the wiring is not arbitrary. A cable in the wrong port does not fail loudly. It quietly forces traffic across the wrong path and shows up later as a collective that runs slow for no obvious reason.

Labeling is part of the deliverable, not a nicety. Every cable, both ends, tagged to match the cable schedule, so that when a link errors at 3 a.m. six months out, someone can find both ends without tracing it by hand through a hundred racks. Miswires and mislabels are among the most common findings at this stage, and they are cheapest to fix now, with the cluster cold and the schedule open.

On blind-mate and high-density assemblies, also confirm the mechanical seating before power: connectors fully home, power whips landed on the right phases, and liquid connections made and checked. The verification is tedious and it is where disciplined integrators separate from the rest. The cable schedule, the rack elevations, and the fabric topology in the cluster design are the reference, and the OEM build guide governs the mechanical seating.

Staged power-on and sequencing

Power-on is staged, not a single switch, so that inrush is controlled and a fault is caught at the smallest possible scope. You bring up the rack power distribution first and confirm the feeds, the phases, and the metering before energizing the trays. PDUs come up while the compute stays in standby, so you can confirm the rack is taking power correctly before the load behind it draws. Then trays are powered in a controlled order rather than all at once.

Inrush is the reason for the staging. A floor of dense racks slamming on together presents a current surge the upstream breakers were never asked to see as a step, and a breaker that trips on energization is a finding you want at one rack, not across a row. Bring load on in blocks, watch the upstream protection, and confirm the rack draw against what the design predicted as you go.

Sequencing also exposes the obvious early failures cheaply: a tray that will not power, a power supply that is dead on arrival, a phase wired wrong. Catch those at the rack, with nothing else energized to confuse the picture. The power feed and density behind all of this is covered in the AI and GPU rack readiness guide. The energization order and the breaker coordination should follow the OEM power-on procedure and the electrical design, not a habit from a smaller deployment.

Firmware, BIOS, and configuration across the fleet

Firmware and configuration have to match across the whole fleet, because a cluster is only as consistent as its least-updated node. Every layer carries firmware: the system BIOS, the BMC, the NICs and the fabric switches, the GPU baseboard, and the components on it. When versions drift from node to node, you get failures that move around, performance that varies for no visible reason, and bugs that one vendor already fixed in a release half the fleet never got.

The fix is a golden image and a known baseline. A defined firmware package and a defined software stack, the GPU driver, the fabric drivers such as MOFED or DOCA, CUDA, the diagnostics like DCGM, and the fabric manager, all at versions that are validated to work together, applied to every node the same way. The driver stack has to match the firmware baseline, not float independently, or you reintroduce the drift you just removed.

Config drift is the quieter twin of firmware drift. BIOS settings that affect performance and stability, things like the power profile and the PCIe configuration, have to be identical across nodes, and they are easy to get wrong one tray at a time. Verify the firmware levels and the BIOS settings as a fleet, record the baseline, and check it as part of acceptance. The validated combination comes from the OEM and the integrator for the specific platform, not from chasing the newest version of each part.

Why you validate the network fabric link by link

The high-speed network fabric is the single largest source of cluster commissioning problems, so it gets validated link by link, every link confirmed up at its rated speed with clean error counters. The fabric is what lets thousands of GPUs act as one machine, whether it is InfiniBand or an Ethernet RoCE fabric such as Spectrum-X. When it is healthy, training scales. When one link in a thousand is degraded, collective operations stall on the slowest path and the whole job crawls while every counter at the node looks fine.

Validation starts with discovery and topology. The fabric manager has to find every endpoint and assign it an address, and the discovered topology has to match the design. On InfiniBand the subnet manager and a tool such as ibdiagnet sweep the fabric, confirm every link is up at the expected width and speed, and compare the wiring to the topology file so a miswire shows up as a discrepancy rather than a slow job later. RoCE fabrics use their own discovery and the switch counters do the same work.

Then you read the error counters, link by link, with the fabric under traffic. A link that trained up at full speed can still be quietly retransmitting because the optic is marginal or the cable is seated wrong. The fabric is where most of the day-one pain lives, and finding it now, with a clean topology file and the cluster idle, is far cheaper than finding it as a performance mystery in production. The link diagnostics, the counters, and the pass thresholds come from the fabric vendor's validation procedure and the cluster design.

Chasing link flaps, errors, and bad optics

Link errors on the fabric usually trace to the physical layer, so the first move is clean and reseat before you suspect anything deeper. A link that flaps, retrains repeatedly, or accumulates symbol errors is most often a transceiver seated poorly, a contaminated fiber end face, or a cable strained or bent past spec. On long heavy cables, weight and vibration over time let connectors creep, which is why reseating a cable or card clears a large share of these.

The number that anchors the conversation is bit error rate. The InfiniBand specification targets a BER of 1e-12, and the diagnostics flag links whose error rate runs worse than the threshold the fabric vendor sets. Symbol errors, link recoveries, receive errors, and link integrity errors are the counters that point at a bad cable or port, and the large majority of symbol errors are hardware, not software. The procedure is standard: clean and reseat first, then replace the cable or the optic if the errors persist, then suspect the port if a known-good cable still errors there.

Discipline matters more than cleverness here. Work one link at a time, clear the counters, run traffic, and re-read, so you know whether the fix took. A link that comes back clean after a reseat is fixed. A link that comes back dirty needs the next part swapped. The acceptable error thresholds and the exact counters to watch are set by the fabric vendor diagnostics and the cluster design, not by a feel for what looks bad.

Checking every GPU for health and inventory

Per-GPU health checking confirms that every accelerator the design called for is present, correctly placed, and free of the early defects that kill training jobs. Inventory comes first: the right count of GPUs on every node, each at the right link width on its bus and able to talk to its neighbors over the GPU-to-GPU interconnect such as NVLink. A GPU that fell off the bus or trained its interconnect at the wrong width is a node you do not want scheduled.

Health is read from the diagnostics and the driver. Tools like NVIDIA DCGM run staged diagnostic levels that exercise memory bandwidth, the PCIe path, the interconnect, and thermal behavior under load, and they surface the GPUs that fail. Underneath, the driver emits Xid codes to the system log that name specific failure modes: a double-bit ECC error, a row-remap failure on the memory, an interconnect error, or a GPU that has dropped off the bus. Correctable ECC errors and row remapping that pile up on one GPU are an early warning that the part is degrading, even before it fails outright.

The goal at commissioning is to find the dead and the sick ones and pull them now. Operators at scale lean on this continuously: large fleets run GPU diagnostics nightly across the whole estate to catch degrading parts before they take a job down. At bring-up you are doing the first and most thorough pass of that same check. The diagnostic levels, the pass criteria, and which Xid codes are fatal versus tolerable come from the GPU vendor's documentation and the OEM runbook.

Validating thermal performance under load

Thermal validation proves the cooling holds when the GPUs are actually working, not when the room is quiet. A rack that sits cool at idle can run hot the moment a full load lands, because an AI rack rejects most of its power as heat and the dense ones reject more than a hundred kilowatts from a single cabinet. The test is to load the GPUs and watch the temperatures: the die temperatures, the memory temperatures, and the coolant or air path, against the limits the design set.

Throttling is the symptom to hunt. When a GPU gets too hot it reduces its clocks to protect itself, which silently drops performance, and a cluster that throttles under load will never hit its benchmark no matter how clean the fabric is. You are confirming that under sustained full load nothing throttles on temperature, that hotspots are not forming on particular racks or particular spots in a rack, and that the cooling delivered to the rack matches what the facility was commissioned to provide.

This is where the facility and the cluster meet, and it has to be tested as one. The chilled water plant and the direct-to-chip loops were proven on the facility side, but only the real GPU load proves the heat actually moves from die to coolant to the plant without a bottleneck in between. The rack-level cooling design is covered in the AI and GPU rack readiness guide. The temperature limits and the acceptable thermal margin under load come from the hardware vendor and the cluster design, with the facility cooling design as the other half of the picture.

Validating power draw and spikes under load

Power validation confirms the real draw under load against what the design assumed, because nameplate and reality are not the same number. A rack's actual draw under a heavy training step can sit well below or, in bursts, above the steady figure people plan around, and AI workloads are spiky in a way legacy compute was not. Synchronized GPUs ramping together produce power steps and transients that the rack power and the upstream protection have to absorb without tripping.

The test is to load the cluster and measure: the per-rack draw, the per-phase balance, and the transient behavior as load steps on and off, compared to the design and to the breaker ratings. The headroom you think you have on a breaker is the first thing to disappear when several racks ramp together. A feed that looks fine at average draw can trip on a synchronized spike, and that is a finding you want under a controlled burn-in, not under a production job.

Tie the measured numbers back to the facility power that was commissioned. The plant was proven to deliver and protect a certain load profile; the cluster validation confirms the racks present a profile that fits inside it, transients included. The rack power feed and density are covered in the AI and GPU rack readiness guide. The acceptable draw, the phase balance, and the transient limits are set by the electrical design and the OEM power specification for the platform.

Burn-in: stress the cluster to flush infant mortality

Burn-in is sustained stress testing at full load over a set period, run to force the weak parts to fail now instead of in production. After every node is individually healthy and the fabric is clean, you load the whole cluster, or large blocks of it, and keep it loaded for a fixed duration that the runbook specifies, watching for anything that drops, errors, or throttles. The point is not to confirm the cluster runs once. It is to make the parts that are going to fail early fail under your watch.

The tools are stress generators and the diagnostics that watch them. GPU stress loads such as gpu-burn and the diagnostic suites push the accelerators to full power and temperature while DCGM and the driver log watch for ECC errors, Xid events, thermal violations, and any GPU that falls out. The same run exercises the fabric and the power and cooling at the same time, which is the point: real load on everything at once is the only condition that reproduces how the cluster will actually run.

A node or a link that errors during burn-in gets pulled, replaced, and the block re-run. Burn-in without that replace-and-retest loop is just a long idle. The duration, the load pattern, and the pass criteria are set by the OEM and integrator runbook and the cluster design, and they vary by platform and by how much risk the owner wants to retire before production. Longer burn-in catches more early failures and costs more idle GPU time, and that trade is the owner's to set.

The bathtub curve and why burn-in works

The bathtub curve is the reason burn-in exists. Plotted over a part's life, the failure rate starts high, drops to a low flat plateau for the useful life, then climbs again as the part wears out. The early high-failure region is infant mortality: defects from manufacturing and handling that surface in the first hours and days of real operation. The plateau is normal life. The far end is old age.

Burn-in deliberately runs the cluster through that early steep part of the curve before it goes to work. By holding the hardware at full load, and at the high temperature and power that comes with it, you accelerate the failure mechanisms that would have shown up in the first weeks of production and force them into the commissioning window instead. The parts that survive burn-in are the ones that have settled onto the flat part of the curve, where they belong before a workload depends on them.

This is borrowed straight from semiconductor practice, where burn-in at high stress to weed out infant mortality is a standard step, and it scales up cleanly to a cluster of tens of thousands of parts. The cluster version just runs it at system level, with real workloads as the stress. How hard and how long to push is the part the OEM runbook and the cluster design specify, since over-stressing healthy parts has its own cost.

Collective communication and NCCL bandwidth tests

Collective-communication tests prove the GPUs actually move data between each other at the speed the fabric promises, which is what training depends on. Distributed training does not just compute on each GPU. It constantly shares gradients across all of them with collective operations, the all-reduce being the one that dominates, and the cluster is only as fast as those collectives. A fabric that passes link checks can still deliver poor collective bandwidth if the topology is wrong, the routing is unbalanced, or one slow path drags the group.

The standard measure is a collective benchmark such as the NCCL tests, run as all-reduce and other patterns across growing scopes. You start at two nodes, confirm the number, then expand toward the whole cluster, because problems that hide at small scale appear when more paths are in play. Practitioner guides report well-tuned fabrics reaching on the order of 90 to 95 percent of the fabric's theoretical bandwidth on all-reduce, for example around 370 GB/s on a node with roughly 400 GB/s of aggregate fabric bandwidth, eight 400 Gb/s rails, and a result well below that points at a topology, routing, or link problem to chase.

Run collectives at every scope where the design has a structure: within a rail, within a rack, across racks, and across the whole cluster, so you isolate where bandwidth falls off. A clean per-link check plus a strong all-reduce across the full cluster is the pair that says the fabric is genuinely working, not just lit. The exact patterns, scopes, and the acceptable fraction of theoretical bandwidth are set by the cluster design and the OEM or integrator acceptance procedure.

Acceptance benchmarks and the baseline

Acceptance benchmarks confirm the assembled cluster hits the performance the design and the contract called for, end to end, not just component by component. Individual GPUs can pass diagnostics and the fabric can pass link checks while the cluster still falls short of its target on a real workload, because performance at scale depends on everything working together. The benchmark is the proof that it does, and it becomes the baseline every later health check measures against.

What you run depends on the cluster's job. A standardized suite such as MLPerf gives a recognized reference point for training and inference throughput, and many acceptance plans pair that with the actual workload the owner intends to run, which is the most honest test of all. The numbers get captured as the commissioning baseline: the collective bandwidth, the throughput, the scaling efficiency as you add nodes, recorded against the configuration that produced them.

That baseline is the asset, more than any single pass or fail. Six months out, when someone asks whether the cluster has slowed down, the only way to answer is to compare against a number captured when it was known good and fully documented. Run the acceptance benchmark, record it with the firmware and configuration it ran on, and hand it over. The specific benchmarks, the target numbers, and the pass thresholds are set by the contract, the cluster design, and the OEM or integrator, and any service-level commitment to the owner rides on them.

How do you isolate a fault in a cluster this size?

Fault isolation in a cluster works by narrowing the scope until the bad part is alone, then replacing it and re-running to confirm. With tens of thousands of GPUs and far more links, you cannot eyeball the bad one. You let the tests point at it: a node that fails diagnostics, a link that errors, a collective that runs slow across a specific path. The skill is reading the signal back to one part rather than rebooting the whole block and hoping.

The method is bisection and substitution. If a collective across a rack is slow, test each rail and each node within it until the slow path is cornered. If a node fails burn-in, swap the suspect part, the GPU, the optic, the cable, and re-run the same test on the same node to prove the fix took. A fault that moves with the part is a hardware fault. A fault that stays with the slot is the slot or the cable into it. That distinction is most of the battle.

Triage keeps the bring-up moving. Pull failing nodes out of the working set so the rest of commissioning continues, log every fault with its location and symptom, and batch the repairs rather than stopping the whole cluster for each one. Replace-and-retest is the loop that actually clears the list. The diagnostic flow and the decision of when to replace versus reseat versus reflash come from the OEM runbook and the integrator's experience with the platform.

What are the acceptance criteria for a commissioned cluster?

Acceptance is the point where the cluster is proven good enough to take a workload, and it is a defined set of criteria, not a feeling that things look fine. The common shape is this: every node present and healthy in the inventory, the fabric clean with all links up at speed and error counters inside threshold, GPU diagnostics passing across the fleet, thermal and power validated under load with no throttling, burn-in completed for the specified duration with failures replaced and re-run, and the acceptance benchmark meeting its target.

Each of those is a documented result, signed off, not a verbal assurance. Acceptance is also where the failing-but-tolerable gets decided honestly: a cluster may be accepted with a small, named set of nodes still in repair, on the agreement that they are tracked and the spare pool covers them. What is not acceptable is unknown state, a fabric never validated end to end, or a benchmark never run.

The criteria themselves belong to the contract and the cluster design, and the OEM or integrator acceptance procedure defines the tests and the thresholds. Treat acceptance as a gate, the same way facility commissioning treats its integrated systems test as the gate to go-live. The whole point of the bring-up is to arrive at this checkpoint with evidence, hand a known-good cluster to operations, and have the record to prove it.

What to document at handover

The commissioning record is what turns a working cluster into a supportable one, because the day-two team inherits whatever you wrote down and nothing you did not. Capture the as-built: the rack elevations, the cable schedule as actually wired, and the fabric topology that was validated. Capture the firmware and software baseline, every layer and version, so drift can be detected later against a known starting point. Capture the test results: the per-link fabric checks, the GPU diagnostics, the thermal and power validation under load, the burn-in outcome, and the acceptance benchmark numbers.

Capture the fault log too, not just the passes. Every part that failed during bring-up, where it was, what the symptom was, and what fixed it, because that history is the first thing the operations team will want when a similar fault appears in production. The baseline numbers are the reference for every later health check; without them, nobody can say whether the cluster has degraded or always ran this way.

This is the kind of record a field tool earns its keep on. Capturing per-rack and per-link results, photos of the as-built, firmware levels, and the fault log against each location in something like FieldOS, rather than a spreadsheet that lives on one laptop, is what lets operations actually use the handover. The owner's commissioning plan and the integrator's runbook set the required deliverables; the discipline is recording them as you go, not reconstructing them at the end.

Handover to operations and ongoing health

Handover gives operations a known-good cluster plus everything they need to keep it that way. The package is the as-built and the baseline, a runbook for the routine faults the bring-up already taught you about, and the monitoring set up to watch the same signals commissioning watched: the fabric error counters, the GPU diagnostics and Xid stream, and the thermal and power telemetry. Operations should inherit the thresholds, not invent them, because the commissioning numbers are the reference for what normal looks like.

Ongoing health is the same checks commissioning ran, now run continuously. Large operators run GPU diagnostics across the whole fleet on a schedule, watch the fabric counters for links starting to degrade, and track ECC and row-remap trends to catch a GPU going bad before it takes a job down. Commissioning sets the baseline; day-two operations is the discipline of measuring against it and acting on drift.

The clean handover is the difference between a cluster that stays healthy and one that erodes quietly. A fault that took a careful afternoon to isolate during commissioning takes minutes in production if the monitoring and the runbook are in place, and takes a dead training job if they are not. The monitoring stack, the alert thresholds, and the maintenance cadence are set by the operator with the OEM's guidance, building on the baseline the bring-up established.

Common mistakes

• Commissioning the cluster on an un-commissioned or unstable facility, so every cluster fault is contaminated by a power or cooling variable.
• Skipping link-by-link fabric validation and trusting that links which trained up are links that are clean.
• Running no real burn-in, so infant-mortality faults surface weeks into a production job instead of on a commissioning afternoon.
• Ignoring thermal and power under full load, then chasing throttling and tripped breakers once the workload is on.
• Letting firmware and configuration drift across the fleet instead of holding every node to a golden baseline.
• Powering GPUs into load before the cooling loops are proven, which can destroy hardware on a direct-to-chip system.
• Handing over with no baseline or documentation, leaving operations unable to tell degradation from how the cluster always ran.

Field checklist

Standards and references

There is no single code that governs cluster commissioning the way the NEC governs wiring, so the controlling document is the OEM and integrator commissioning runbook for the specific platform, read alongside the cluster design. The hardware and fabric vendors publish the diagnostics and the pass thresholds: the GPU vendor's diagnostic tooling and error-code documentation for GPU health, and the fabric vendor's validation procedure and diagnostic utilities for the network. Those define what passes, and they change by generation, so use the version that matches the hardware in front of you.

On the facility side, the relevant practice is the data center commissioning discipline and the integrated systems test, framed by the Uptime Institute Tier program and ANSI/TIA-942, with the thermal envelope from ASHRAE TC 9.9. That work is the prerequisite, not part of the cluster bring-up, and it is covered in the data center commissioning levels guide. Independent rating efforts for GPU cloud quality, such as the ClusterMAX framework, also describe what a well-commissioned cluster looks like in practice.

Hedge every test, threshold, and acceptance number to those sources rather than to a figure carried from another job. The principle that holds across all of them is the order and the rigor: commission the facility first and the cluster second, validate the fabric and GPU health link by link rather than in bulk, and burn the cluster in under real load to flush infant mortality before the workload arrives. The numbers move by platform and generation. That sequence does not.

Units and terms

Cluster commissioning carries its own vocabulary, and a few terms get used loosely in ways that cause confusion between the facility side and the IT side.

Cluster commissioning is the bring-up and proof of the IT cluster itself, distinct from facility commissioning, which proves the building's power and cooling. Burn-in is sustained stress at full load to force early failures out before production. Infant mortality is the high early-life failure rate, the steep front of the bathtub curve, which burn-in is designed to catch. Fabric validation is the link-by-link confirmation that the high-speed network is up at speed and clean. A collective is a group communication across GPUs, such as the all-reduce, and NCCL is the common library that runs it. Acceptance criteria are the documented set of results that gate the cluster to production.

Cluster commissioning

Bring-up and proof of the GPU cluster: rack, fabric, GPU, and software, after the facility is commissioned

Facility vs cluster Cx

Facility commissioning proves the building's power and cooling; cluster commissioning proves the compute on top of it

Burn-in

Sustained stress testing at full load over a set period to force early failures out before production

Infant mortality / bathtub curve

The high early-life failure rate at the front of the bathtub curve, the failures burn-in is meant to catch

Fabric validation

Link-by-link confirmation that the high-speed network is up at rated speed with error counters inside threshold

Collective / NCCL

Group communication across GPUs such as all-reduce; NCCL is the common library whose bandwidth tests prove it

Acceptance criteria

The documented results, healthy nodes, clean fabric, passing benchmark, that gate the cluster to production

Related tools

Calculators and readiness checks for this work

Compare your options

FAQ

People also ask

• What is data center commissioning?
• How much power does an AI rack use?
• What is the difference between cold and hot commissioning of a BESS?
• What is BMS commissioning?
• What is a CDU in a data center?