ANVILFIELD Try FieldOS

Datacenter

Data center interconnect (DCI) and optical transport field guide

How to connect data centers over campus, metro, and long-haul distances: coherent DWDM, 400G and 800G ZR pluggables, dark fiber versus leased waves, the optical link budget, latency, and diverse redundant paths.

Data Center InterconnectDWDM Coherent Optics400ZR 800ZRDark FiberOptical Transport

Direct answer

Data center interconnect (DCI) carries traffic between data centers across a campus, a metro, or a region, a different problem than the fabric inside one building because distance brings optical physics, carrier services, and huge bandwidth into play. It runs on coherent DWDM, increasingly 400G and 800G ZR pluggables, over dark fiber or leased waves.

Key takeaways

  • DCI carries traffic between data centers across campus, metro, or regional distances, running coherent DWDM on dark fiber or leased waves.
  • 400ZR and 800ZR coherent pluggables target roughly 80 to 120 km single-span metro links; ZR+ variants reach 600 to 1000 km and beyond.
  • Light in single-mode fiber adds about 5 microseconds per kilometer one way, near 10 microseconds round trip; a 100 km link adds about a millisecond round trip.
  • Two wavelengths in one conduit are not redundant; diversity requires physically separate routes and separate building entrances verified by route maps.
  • For coherent links OSNR governs reach, not raw power; build the link budget from manufacturer and fiber data and carry margin.

What data center interconnect is

Data center interconnect, DCI, is the transport that carries traffic between data centers rather than inside one. The sites might be two buildings on the same campus, two facilities across a metro, or two regions hundreds of miles apart. The distance is the whole point. Once the link leaves the building, the problem stops being switch ports and ribbon fiber and starts being optical physics, carrier territory, and bandwidth measured in terabits.

Inside a hall, light travels a few hundred meters at most and the loss budget is generous. Between data centers the light has to survive tens or hundreds of kilometers of glass, so it loses power, smears in time from dispersion, and needs amplification to reach the far end. That is a different engineering discipline than wiring a fabric, and it leans on coherent DWDM optics, a carefully built link budget, and a service model for the fiber itself.

The work breaks into three decisions that drive everything else. Match the coherent optics and the link budget to the distance. Choose the fiber and service model, dark fiber you light yourself or leased waves a carrier delivers. Build diverse, redundant paths and watch the latency the distance adds. Get those three right and the rest is provisioning. Get one wrong and the site is either dark, slow, or stranded on a single fiber that one backhoe can cut.

DCI versus the network inside the building

The fabric inside a data center and the transport between data centers solve different problems, and conflating them is the first mistake. The intra-DC network is the spine-leaf fabric covered in the companion guide on data center network architecture: leaf and spine switches, short-reach optics, and east-west traffic between servers in the same hall. Distances are meters to a few hundred meters, the optics are gray and unamplified, and the constraint is port count and oversubscription.

DCI starts where that fabric ends, at the building edge, and it answers a different question: how do you get the aggregated traffic of one site to another site that is kilometers away. Now the constraints are optical loss over distance, the carrier or fiber owner who controls the right-of-way, amplification, dispersion, and a link budget that decides whether the light arrives clean enough to recover. A spine switch does not care about chromatic dispersion. A 600 km coherent line absolutely does.

The practical tell is who you are negotiating with. Inside the building you spec switches and patch panels. For DCI you are talking to a fiber provider or a carrier, signing for dark fiber or a wavelength service, and engineering a link that lives partly outside your fence. Treat DCI as an extension of the internal fabric and you will undersize the optics, miss the latency, and forget that one conduit out of the building is one cut from isolation.

Why DCI matters more now

DCI used to be a back-office concern for replication and the occasional failover. The drivers have changed, and three of them now push real money and real bandwidth onto the link between sites.

AI training is the loudest. A training cluster has outgrown the single building. When the GPU count exceeds what one data hall can power and cool, the cluster spreads across buildings or across a campus, and the traffic between those buildings is enormous and sensitive to delay. That is a DCI problem at a scale the field has not seen before, and it is closely tied to the GPU optics covered in the companion guide on GPU network optics and cabling.

Disaster recovery and replication are the steady drivers. Synchronous replication holds a second copy in lockstep, which only works over short, low-latency links because every write waits for the far side to acknowledge. Asynchronous replication tolerates distance but trades off how much data you can lose. Both ride DCI, and both live or die on the latency and the path diversity. Cloud on-ramps are the third driver: a private, high-capacity link from your data center into a cloud provider's edge, so the traffic never touches the public internet. Underneath all of it is data gravity. Large data sets pull compute toward them, and once data lives in several places, the links between those places carry the weight of keeping them coherent.

Campus, metro, and long-haul: the distance sets the technology

Distance is the first thing you nail down, because it picks the optics, the amplification, and the service model before anything else. The industry sorts DCI into three rough tiers, and the boundaries are soft, so treat the kilometer figures as planning ranges, not hard lines.

Campus is the same site or adjacent buildings, commonly under 2 to 10 km, often on fiber you own or control, frequently without amplification. Metro is across a city, commonly 10 to 80 km, where single-span amplified DWDM and 400G or 800G ZR pluggables fit cleanly. Long-haul and regional run 80 km and well beyond, into the hundreds and thousands of kilometers, where you need multi-span amplification, dispersion handling, and the extended-reach coherent optics. The longer the link, the more the optical physics and the carrier relationship dominate the design. Confirm the actual route distance with the fiber provider, because the optical path follows the fiber, not the map, and it is routinely longer than the straight line between buildings.

TierTypical distanceTech and serviceWhat it enables
CampusUnder 2 to 10 kmOften unamplified, gray optics or short coherent, owned or dark fiberBuilding-to-building on one site, AI cluster scale-across
Metro10 to 80 kmSingle-span amplified DWDM, 400G/800G ZR pluggablesCity-wide DR, cloud on-ramp, active-active sites
Long-haul / regional80 km and beyondMulti-span amplified DWDM, coherent ZR+ or transponders, EDFA and RamanRegional replication, national transport

What is DWDM, and why it carries the capacity

Dense wavelength division multiplexing, DWDM, puts many independent channels of light on one fiber at the same time, each on its own wavelength, or color. Instead of one signal per fiber, you run dozens, and each can carry 400G or 800G of its own. That is the capacity multiplier that makes DCI economically possible, because trenching new fiber costs far more than lighting more colors on fiber you already have.

The wavelengths sit on a standardized grid so equipment from different vendors can agree on where each channel lives. The ITU-T defines the DWDM grid in the C-band, commonly at 50 GHz or 100 GHz channel spacing, with newer flexible-grid systems allocating variable-width slots so a wider 800G channel can take the room it needs. A multiplexer combines the colors onto the fiber at one end and a demultiplexer splits them out at the other.

The payoff is that capacity becomes a provisioning step instead of a construction project. Need more bandwidth between two sites lit with DWDM? You light another wavelength, up to the fill limit of the system and the fiber. That is why DWDM sits under almost every serious DCI link, whether you own the gear or buy the wavelengths from a carrier.

Coherent optics and why they win over distance

Coherent optics are the technology that made high-speed DCI practical over real distances. A direct-detect optic encodes data by switching the light on and off and reading intensity at the far end, which works fine across a data hall but falls apart over tens of kilometers once dispersion and noise smear the signal. Coherent optics encode data in the light's amplitude and phase across two polarizations, then use a digital signal processor at the receiver to recover it.

That DSP is the difference. It compensates for chromatic dispersion and polarization-mode dispersion electronically, so the link no longer needs the dispersion-compensating modules that older systems strung along the route. It cleans up impairments that would have killed a direct-detect signal at distance, and it lets the optic pack far more bits into each symbol through advanced modulation. The result is more capacity and more reach from the same fiber.

For the field, the takeaway is simple. Past a campus, you are running coherent, and the optic's modulation and DSP, not just its rated speed, decide how far it reaches and how clean it arrives. The specific reach and modulation behavior come from the optic manufacturer's data, so size the link against that sheet, not a rule of thumb.

400G and 800G ZR and ZR+ pluggables

The shift that reshaped DCI is coherent optics shrinking into a pluggable the size of a switch transceiver. The OIF, the Optical Internetworking Forum, defined 400ZR as a standardized 400G coherent interface for amplified single-span DWDM links, with a reach target around 80 to 120 km aimed squarely at metro DCI. It became the most widely deployed coherent technology because it put metro reach in a QSFP-DD or OSFP pluggable that drops into a router or switch.

The OIF finalized 800ZR in October 2024, doubling the capacity to 800G while holding a similar single-span 80 to 120 km target. For longer links, the OpenZR+ and ZR+ family extends reach with stronger modulation and higher transmit power: 400G OpenZR+ reaches several hundred kilometers, and 800G ZR+ pushes into the 600 to 1000 km range and beyond using probabilistic constellation shaping. Some vendor modules advertise metro links near 1000 km at 800G and regional links to roughly 2000 km at reduced rates. Treat those numbers as the manufacturer's claim for that module and confirm against the data sheet and your link budget.

The decision the pluggable forces is reach versus rate. A baseline ZR optic gives you full speed at metro distance. To go farther you either step down the rate, move to a ZR+ variant, or move to a dedicated transponder. The honest version is that reach, rate, and the optic are coupled, so you pick the optic to the distance and verify it survives the actual fiber, not the brochure span.

OpticCapacityReach targetWhere it fits
400ZR (OIF)400G~80 to 120 km, single span amplifiedMetro DCI, the volume coherent optic
400G OpenZR+100G to 400G~300 to 600 km, 1000+ km at lower ratesRegional, extended-reach metro
800ZR (OIF, 2024)800G~80 to 120 km, single spanMetro DCI at double the capacity
800G ZR+~600G to 800G~600 to 1000+ km with PCS, higher powerRegional and long-haul DCI

Pluggable in the router (IPoDWDM) versus an external transponder

Once coherent optics fit in a pluggable, you can put the DWDM optic directly in the router or switch and skip the separate transport box. That architecture is IP over DWDM, IPoDWDM. The router's line card carries the coherent pluggable, the colored light goes straight onto the DWDM line system, and a whole layer of equipment disappears. The wins are lower cost, lower power, lower latency, and fewer boxes to manage and upgrade.

The alternative is a dedicated external transponder or muxponder, the traditional model where the router hands a gray client signal to a transport box that does the coherent conversion and grooming. You give up some integration savings, but you get back things the pure-pluggable path drops: OTN framing, tandem connection monitoring across multiple carriers, management channels, and client adaptation with timing transparency. A muxponder also aggregates several lower-rate clients onto one high-rate wavelength, which fills a wave more efficiently.

The rule of thumb is that IPoDWDM fits IP-centric metro DCI where you control both ends and want the cost and power down. A transponder or muxponder fits longer, multi-carrier, or multi-service links where the OTN features and the operational separation between the IP layer and the transport layer earn their keep. Both are valid. The choice is an architecture decision, so make it with the carrier and the equipment vendor, not by default.

Will the light make it? The optical link budget

The optical link budget answers the one question every DCI link comes down to: does enough clean light reach the far end to recover the data. You add up the loss along the path, fiber attenuation per kilometer plus the loss of every connector, splice, and patch, and you compare it against the optic's power budget. If the loss exceeds the budget, the link does not close, full stop.

For coherent links, raw power loss is only half the story. The figure that governs reach is OSNR, optical signal-to-noise ratio, the ratio of signal power to the noise the amplifiers add along the way. Each amplifier boosts the signal but also injects noise, so OSNR degrades span by span, and a coherent receiver needs a minimum OSNR to hit its target bit error rate. A link can have plenty of power and still fail because the OSNR fell below what the optic needs at that modulation.

Always build in margin. A link engineered to close at exactly its limit will fail the first time a connector gets dirty, a splice is added on a repair, or the fiber ages and loss creeps up. The specific loss values, the amplifier noise figures, and the required OSNR come from the fiber provider and the optic manufacturer, so build the budget from their numbers and carry headroom on top. A link with no margin is a link waiting for the first bad day.

Amplification: EDFA and Raman over the long spans

Past a certain distance the light is too weak to recover, and you amplify it rather than regenerate it. The workhorse is the EDFA, the erbium-doped fiber amplifier, which boosts the whole band of DWDM wavelengths at once without converting back to electronics. Spans between EDFAs commonly run on the order of 80 to 100 km, with an amplifier site at each hop along a long-haul route.

Raman amplification adds reach and cleaner signal by pumping the transmission fiber itself so it amplifies along its length, lifting the signal power before it hits the next EDFA and improving OSNR. Hybrid EDFA-plus-Raman designs are common on the longest links because the OSNR improvement, which can be substantial, is exactly what coherent signals need to reach beyond a couple thousand kilometers. Each amplifier you add buys distance and costs OSNR in noise, so the amplifier plan is its own engineering problem.

On a campus or short metro link you may have no amplifiers at all. On a regional or national link the amplifier sites, their spacing, and their gain are central to whether the link closes. That design belongs to the transport vendor and the carrier who own the line system, and it should be reviewed as part of the link budget, not assumed.

Dispersion over distance, and how coherent handles it

Dispersion is the spreading of a light pulse as it travels, and over distance it smears symbols into each other until the receiver can no longer tell them apart. Chromatic dispersion comes from different wavelengths traveling at slightly different speeds in the glass, and it accumulates with distance and with data rate. Polarization-mode dispersion, PMD, comes from the two polarizations traveling at slightly different speeds, and it varies with the fiber and even the temperature.

Older systems fought chromatic dispersion with physical dispersion-compensating modules spliced into the route, spools of fiber that undid the spreading at the cost of loss and complexity. Coherent optics changed that. The receiver's DSP compensates chromatic dispersion and PMD electronically, so a modern coherent link does not need the compensating modules and tolerates dispersion that would have stopped a direct-detect signal cold.

The limit still exists. There is a ceiling on how much accumulated dispersion the DSP can correct, and it ties back to reach and modulation, which is one more reason the optic's reach figure is the number that governs the link. On older fiber with high PMD, the link may fall short of the optic's nominal reach, so the fiber's measured PMD belongs in the link planning. When in doubt, get the route's dispersion and PMD figures from the fiber provider and check them against the optic's tolerance.

Dark fiber, leased waves, or carrier service

How you get the fiber between sites is a business decision with real engineering consequences, and there are three common models. The split is control and capacity versus cost and effort, and the right answer depends on your scale and how much optical engineering you want to own.

Dark fiber means you lease or own the unlit glass and put your own optics on both ends. You control the capacity, the optics, and the upgrade path, and the marginal cost of more bandwidth is just lighting another wavelength. The price is a large upfront commitment and the staff to engineer and run the optical layer. A leased wavelength, a lambda, means a carrier delivers a lit channel of a given speed and runs the optics for you under a service-level agreement. Carrier Ethernet or a fully managed service hands you an Ethernet port and lets the provider own everything behind it.

The lean for a hyperscaler or a large enterprise with optical talent is dark fiber, because at scale the control and the per-bit economics win. For a site that needs a link turned up quickly, or an organization without optical engineers, leased waves or a managed service is the better trade even at a higher recurring cost. The mismatch to avoid is buying dark fiber you have no one to light and run, or paying recurring wavelength fees for years at a scale where dark fiber would have paid for itself. Run that math against the actual capacity growth, not the day-one need.

ModelWho owns the opticsCost shapeControl and when to pick
Dark fiberYou light and run both endsHigh upfront or IRU, low marginal capacity costMaximum control and capacity; pick at scale with optical staff
Leased wavelength (lambda)Carrier runs the optics, delivers a lit channelRecurring per-wave fee, SLACarrier owns the physics; pick for faster turn-up, fewer staff
Carrier Ethernet / managedCarrier owns everything to an Ethernet portRecurring, bandwidth-tieredEasiest, least control; pick for smaller sites or burst needs

Dark fiber: lease the glass, own the optics

Dark fiber is fiber that is installed but not lit, and taking it on means you own the transport end to end. You light it with your own DWDM gear, you set the capacity, and you decide when to upgrade. That is the maximum-control option, and it is why hyperscalers and large enterprises favor it where they can get it.

The commercial form is usually a lease or an IRU, an indefeasible right of use, which is a long-term right to specific fibers, often 15 to 20 years, paid largely up front. An IRU behaves like ownership for the term without the cost of building the route yourself. The trade is that you now carry the optical engineering, the amplifier plan on long routes, the sparing, and the operations. Dark fiber with nobody to run it is a liability, not an asset.

Where dark fiber shines is capacity growth. Once the glass is yours, adding bandwidth is lighting another wavelength rather than renegotiating a contract, so the per-bit cost falls as you fill the fiber. That is the economic case, and it only closes if your traffic actually grows into the fiber over the term.

Leased waves and managed transport: the carrier does the optics

A leased wavelength is a lit channel a carrier sells you between two points, at a fixed speed, with the optics, amplification, and dispersion handling all on the carrier's side of the demarcation. You hand off a client signal and the carrier delivers it to the far site under an SLA that defines availability and, often, latency. It is the faster, lower-effort path onto a DCI link.

The appeal is that you skip the optical engineering entirely. No amplifier plan, no link budget on your side of the handoff, no optical sparing. You buy a 400G wave between two cities and the carrier makes the light arrive. For a site that needs connectivity in weeks rather than a fiber build, or an organization without a transport team, that is the right trade.

The cost is recurring and the control is limited. You get the speed and the diversity the carrier offers, not an arbitrary capacity you can scale by lighting more colors yourself, and growth means buying more waves at the carrier's price. Read the SLA for what actually matters to your application: not just uptime, but the latency commitment and whether the protected path is genuinely diverse. A wave with a great uptime number on a single physical route is still one cut from down.

Latency is distance, and distance is physics

Latency on a DCI link is set first by the speed of light in glass, and you cannot engineer your way under it. Light in single-mode fiber travels at roughly two-thirds of its speed in vacuum, which works out to about 5 microseconds of one-way delay per kilometer, near 10 microseconds round trip per kilometer. A 100 km metro link adds about a millisecond round trip from propagation alone, before any equipment.

That number decides what the link can do. Synchronous replication, where every write waits for the remote site to confirm, only works over short, low-latency links, because the round-trip delay is added to every transaction. Push synchronous replication too far and the application slows to a crawl while it waits on the fiber. Beyond that distance you move to asynchronous replication, which decouples the write from the acknowledgment and trades latency tolerance for a recovery-point window, the amount of data you can lose if the primary fails.

For AI training across buildings, the same physics bites differently. Distributed training synchronizes gradients across the cluster, and added latency between the buildings stretches every synchronization step, which drags down the utilization of very expensive GPUs. This is the hard constraint behind every DCI decision involving replication or training: the optical path follows the fiber, the fiber is longer than the map, and the latency is whatever that real distance dictates. Measure latency on the actual provisioned path, because the route you got may be longer than the route you asked for.

Diverse paths so one cut cannot isolate a site

The failure that takes a data center off the network is almost never the optics. It is a fiber cut, and the way you survive it is genuine path diversity. Two wavelengths in the same conduit are not redundant. One backhoe, one cut, and both are gone together. Diversity means physically separate routes that do not share a conduit, a bridge crossing, or a building entrance.

The strong design gives a site two diverse paths and two separate entrances into the building, so no single fiber cut and no single entrance failure can isolate it. On a ring topology, traffic protects by wrapping the other way around the ring when a span fails. On point-to-point links you run two diverse routes and let the network reroute. Either way, the question to ask the carrier is not whether they offer protection but whether the protect path is physically diverse from the working path, end to end, including the last mile into each building.

This is where carrier claims need verifying. Two services sold as diverse can ride the same physical fiber in a shared segment, often near the building entrance or across a single river crossing, and you only find out when both drop at once. Ask for the route maps, confirm separate building entrances, and treat a single point of convergence anywhere on the path as a single point of failure, because that is exactly what it is.

The demarc, the meet-me, and the handoff

Every DCI link crosses a boundary where your responsibility ends and the carrier's begins, and that point is the demarcation, the demarc. It is where you hand off your signal and where you stop owning the physics. Knowing exactly where the demarc sits, and what the interface is, prevents the finger-pointing that follows every outage on a link nobody fully owns.

In a carrier-neutral facility the handoff usually happens in a meet-me room, the MMR, a shared space where carriers and tenants cross-connect to each other. You run a cross-connect from your cage to the carrier's equipment in the MMR, and that cross-connect is part of your link budget and part of your failure analysis. The same connector cleanliness and loss accounting that govern the optics inside the building, covered in the GPU optics and cabling guide, apply to that cross-connect too.

Treat the demarc as a documented, tested interface, not an afterthought. Record where it is, what type it is, the expected loss and power at that point, and who owns each side. When a link degrades, the first diagnostic is whether the problem is on your side or the carrier's, and a clear demarc with measured reference values is what lets you answer that in minutes instead of a day of conference calls.

Encryption in flight between sites

Traffic between data centers leaves your physical control, often crossing a carrier's network and public right-of-way, so anything sensitive should be encrypted in flight. The exposure is real: fiber can be tapped, and a leased service traverses equipment you do not own. Encrypting on the DCI link closes that gap regardless of who carries the light.

Two layers do this in practice. MACsec, IEEE 802.1AE, encrypts at the Ethernet layer and provides confidentiality, integrity, and authenticity on the link, and it is widely supported on the routers and switches that terminate DCI. Optical-layer or Layer 1 encryption sits in the transport gear and encrypts the payload of the line signal itself, which keeps full rate and low latency because it works below the packet layer. Both are good choices for DCI and offer similar protection; the pick depends on where you want the encryption to live and what your equipment supports.

The mistake is leaving DCI in the clear because it feels private. A dark-fiber link you own still runs through manholes and conduits you do not monitor, and a leased wave runs through a carrier's gear entirely. Encrypt it, manage the keys properly, and confirm the throughput and latency cost is acceptable for the application, which for hardware-based MACsec and Layer 1 encryption is generally small.

Sizing the link and planning for growth

Size DCI to where the traffic is going, not where it is, because the lead time to add capacity is long and AI traffic between sites grows fast. The capacity question is how many wavelengths you need now, how fast the demand climbs, and how much of the fiber's or system's fill you want to leave as headroom.

On dark fiber with your own DWDM, the system has a maximum number of wavelengths and a per-wavelength rate, and your usable capacity is the product, minus what you hold in reserve. Filling the last wavelengths is where it gets tight, because the OSNR budget tightens as the system loads up, so the practical capacity is usually below the theoretical channel count. On leased waves, capacity is simply how many waves you buy, and growth is a procurement and lead-time problem.

The growth mistake is sizing the link to the day-one need with no headroom, then scrambling when an AI cluster or a replication workload doubles the demand. Leave wavelengths spare on a system you own, or contract for a clear, fast path to more waves on a service you lease. The traffic between data centers is the part of the network growing fastest, and a link sized exactly to today is undersized by the time it turns up.

AI training across buildings and campuses

The driver reshaping DCI design is AI training spilling out of a single building. A large training cluster needs more power and cooling than one data hall can supply, so the GPUs spread across multiple halls, multiple buildings, or a whole campus, and the network that ties them together is now a DCI problem at a scale and sensitivity the field has not dealt with before.

Inside the cluster, the GPU back-end fabric is the lossless, high-radix network covered in the GPU network optics and cabling guide and built on the spine-leaf principles in the architecture guide. When that fabric has to reach across buildings, DCI carries the east-west training traffic between them, and the demand is brutal: enormous bandwidth, because gradient synchronization moves huge volumes, and tight latency, because every microsecond of added delay between buildings stretches each synchronization step and idles GPUs that cost a fortune to sit still.

That changes the DCI brief. The scale-across links want the highest-capacity coherent optics over the shortest, most diverse campus or metro paths you can engineer, with latency held down because the training job is synchronous in effect. This is the meeting point of the three datacenter guides: the fabric inside the hall, the optics that wire the GPUs, and the transport that joins the buildings into one cluster. Design them as one system, because the training job does not care which guide owns which link.

Commissioning and operating the link

A DCI link is not done when it is connected. It is done when it is tested, baselined, and proven against the link budget, and the turn-up is where you catch the problems that would otherwise show up as intermittent errors months later. Test the fiber first, then the optical layer, then the service.

Start with the fiber itself. An OTDR, an optical time-domain reflectometer, shoots the route and shows the loss and the location of every connector, splice, and event, so you find a dirty connector or a bad splice before it is buried in a live link. Measure end-to-end optical loss with a light source and power meter and compare it to the budget. On coherent links, confirm OSNR and the received power, then verify the bit error rate is at or below target with margin, ideally with a soak test that runs long enough to catch marginal behavior. The connector cleanliness and loss-budget discipline from the GPU optics and cabling guide applies to every connector on the DCI path, including the carrier cross-connect.

Then run it day to day. Monitor the optical layer continuously, the received power and OSNR per wavelength, the pre-forward-error-correction error rate as an early warning, and the path status so you know the moment a protect path is no longer protecting. Watch the spare-wavelength count against growth. Record a baseline at turn-up so that when a number drifts, you can see how far it has moved from a known-good state. A link you measured once and never watch is a link that will surprise you.

What to document

A DCI link is partly outside your building and partly inside someone else's contract, which makes the record the thing that lets you operate it and defend it. When a link degrades at 2 a.m., the record is what tells you the demarc location, the baseline power, and which path is supposed to be carrying traffic. We keep the link budget, the optics, the fiber, the paths, and the demarc in a field tool such as FieldOS so the next person inherits the link instead of reverse-engineering it.

Capture the elements below, and capture the baseline values at turn-up, because a measured reference is what turns a vague slow link into a specific drifted number you can act on.

ElementRequirement to recordNote
Distance and routeActual fiber path length, not map distanceSets latency and the link budget
OpticsOptic type, rate, modulation, reach rating, wavelengthThe reach figure governs the link
Link budgetLoss budget, measured loss, OSNR, marginProves the link closes with headroom
Fiber and service modelDark fiber or leased wave, IRU or SLA termsWho owns the optics and the path
Diverse pathsWorking and protect route maps, building entrancesConfirms one cut cannot isolate the site
Demarc and handoffLocation, interface type, reference power and lossSplits responsibility on an outage
EncryptionLayer and key management for in-flight encryptionConfirms data between sites is protected
Baseline and growthTurn-up baseline values, spare wavelength countDrift detection and capacity headroom

Common mistakes

  • Picking optics and a link budget that do not match the distance, so the link will not close or has no margin.
  • Running both services in one conduit or one building entrance, so a single fiber cut isolates the site.
  • Ignoring latency for AI training or synchronous replication, then finding the application stalls on the fiber.
  • Choosing the wrong dark-versus-leased model: buying fiber with nobody to light it, or paying recurring wave fees where dark fiber would have paid off.
  • Leaving DCI traffic unencrypted because the link feels private when it crosses manholes and carrier gear.
  • Sizing the link to the day-one need with no wavelength headroom for the growth that is coming.
  • Trusting a carrier's protected service without confirming the protect path is physically diverse end to end.

Field checklist

0 of 9 complete

Want this checklist to run itself on every job — with photo proof and a signed record crews can hand the customer? That's FieldOS.

Standards and references

The coherent pluggable interfaces come from the OIF, the Optical Internetworking Forum, which defines 400ZR and the 800ZR implementation agreement finalized in October 2024, with the OpenZR+ MSA covering the extended-reach variants. The DWDM wavelength grid and channel spacing come from the ITU-T, which specifies the C-band grid commonly at 50 GHz and 100 GHz spacing along with flexible-grid allocation. In-flight encryption on the Ethernet layer is IEEE 802.1AE, MACsec, while optical-layer encryption follows the transport vendor's implementation.

Reach, modulation, OSNR requirements, and the link budget for any specific optic come from the optic manufacturer's data sheet, and they vary by module, so size every link against that sheet, not a generic figure. The fiber's loss, dispersion, and PMD, the amplifier plan, and the route diversity come from the carrier or fiber provider, and they control what the link can actually do. The network design, how the paths are protected and how capacity grows, is yours to own with the carrier and the equipment vendor.

The figures in this guide, the 80 to 120 km ZR target, the roughly 5 microseconds per kilometer, the 80 to 100 km EDFA spans, are common planning numbers, not guarantees for your route. The three calls that decide a DCI link stand on their own: match the coherent optics and the link budget to the distance, choose dark fiber versus leased waves by control and cost, and build diverse paths while watching the latency. Verify the specifics with the manufacturer, the carrier, and the design before you commit.

Units and terms

DCI mixes optical, transport, and carrier vocabulary, so the same idea shows up under different names across a vendor sheet, an ITU document, and a carrier contract.

Distance drives reach and latency, capacity is counted in wavelengths times per-wave rate, and the service model decides who owns the physics. The terms below are the ones that recur across every DCI conversation.

DCI
Data center interconnect, the transport that carries traffic between data centers across campus, metro, or long-haul distance
DWDM
Dense wavelength division multiplexing, many independent channels of light on one fiber, each on its own wavelength on the ITU grid
Coherent optics
Optics that encode data in the light's amplitude and phase and use receiver DSP to recover it and compensate dispersion over distance
400G / 800G ZR
OIF-defined coherent pluggables for amplified single-span DWDM, targeting roughly 80 to 120 km metro DCI; ZR+ variants extend the reach
Link budget / OSNR
The accounting of optical loss against the optic's power budget, and the optical signal-to-noise ratio the coherent receiver needs to recover the signal
Dark fiber vs leased wave
Dark fiber is unlit glass you light and run yourself; a leased wave is a lit channel a carrier runs for you under an SLA
Latency
Delay set first by the speed of light in fiber, about 5 microseconds per kilometer one way, the hard limit for replication and training
Diverse path
A physically separate route, conduit, and building entrance from the working path, so a single fiber cut cannot isolate the site

Related tools

Calculators and readiness checks for this work

Compare your options

FAQ

What is data center interconnect (DCI)?

Data center interconnect, DCI, is the transport that carries traffic between data centers rather than inside one, across a campus, a metro, or a region. Unlike the fabric inside a building, it deals with optical loss over distance, carrier services, and terabit bandwidth, and it runs on coherent DWDM optics over dark fiber or leased waves.

How is DCI different from the network inside a data center?

The network inside a data center is the spine-leaf fabric of switches and short-reach optics carrying server-to-server traffic over meters. DCI connects separate data centers over kilometers, where optical physics, amplification, dispersion, and the carrier or fiber owner control the design. One is a switching problem, the other an optical transport problem.

What is DWDM?

Dense wavelength division multiplexing, DWDM, puts many independent channels of light on one fiber at the same time, each on its own wavelength on the ITU-T grid. Each channel can carry 400G or 800G, so one fiber carries dozens of terabits. It is the capacity multiplier that makes DCI affordable without trenching new fiber.

What is 400ZR?

400ZR is an OIF-standardized 400G coherent optical interface in a pluggable, built for amplified single-span DWDM links with a reach target around 80 to 120 km, aimed at metro DCI. It put metro-distance coherent optics into a router or switch pluggable, which made it the most widely deployed coherent technology.

Dark fiber vs leased waves: which should I choose?

Choose dark fiber when you have scale and optical staff, because you control capacity and the per-bit cost falls as you light more wavelengths. Choose leased waves when you need a link turned up fast or lack a transport team, since the carrier runs the optics under an SLA. The trade is control and capacity versus cost and effort.

How much latency does distance add on a DCI link?

Light in single-mode fiber adds about 5 microseconds of one-way delay per kilometer, near 10 microseconds round trip. A 100 km metro link adds roughly a millisecond round trip from propagation alone. This sets what the link can do: synchronous replication needs short links, and longer distance forces asynchronous replication.

Why does DCI matter for AI training?

A large AI training cluster outgrows one building's power and cooling, so the GPUs spread across buildings or a campus. DCI then carries the east-west training traffic between them, which demands enormous bandwidth and tight latency, because every microsecond of added delay stretches gradient synchronization and idles very expensive GPUs.

Do I need to encrypt traffic between data centers?

Yes, anything sensitive crossing a DCI link should be encrypted in flight, because the path runs through manholes, conduits, and carrier gear you do not control. MACsec encrypts at the Ethernet layer and Layer 1 encryption protects the optical payload. Both offer similar protection with low overhead on hardware that supports them.

What is IPoDWDM, and when do I use it instead of a transponder?

IPoDWDM puts the coherent DWDM optic directly in the router or switch, removing a separate transport box for lower cost, power, and latency. Use it for IP-centric metro DCI you control end to end. Use an external transponder or muxponder for longer, multi-carrier, or multi-service links that need OTN framing and operational separation.

How do I make a DCI link survive a fiber cut?

Build two physically diverse paths with separate conduits and separate building entrances, so no single cut isolates the site. Two wavelengths in one conduit are not redundant. Verify diversity with the carrier's route maps end to end, because services sold as diverse often share a segment near a building entrance or a single crossing.

People also ask

Codes cited in this guide

This guide is written and reviewed against the published standards below. Always confirm the current adopted edition with the authority having jurisdiction.

IEEE 802