Datacenter
Data center incident management and outage response field guide
What incident management is, why the minutes after something goes wrong decide whether it is a blip or a headline, and how severity levels, a named incident commander, a restore-first response, disciplined communication, and a blameless root-cause turn an incident into a near-miss.
Direct answer
Data center incident management is the structured way a facility detects an event, responds to it, and learns from it, from a single failed component to a full outage. It runs on severity levels, a named incident commander, a restore-first response, disciplined communication, and a blameless root-cause afterward. The operator's procedures and the contract govern the specifics.
Key takeaways
- Restore service first, then redundancy, then root cause, in that order, every time; never diagnose root cause while the load is down.
- Severity, commonly Sev1 through Sev4, is set by impact and scope and drives paging, update cadence, and contractual clocks; the operator defines the scale.
- The incident commander coordinates and never touches equipment; name them out loud so there is no ambiguity about who is in charge.
- Uptime Institute outage analyses put roughly 70 percent of outages on human error, mostly procedures not followed or flawed; fix the system, not the operator.
- Run a blameless post-mortem and track specific, owned, dated corrective actions to verified closure; an incident not learned from is one scheduled to repeat.
What incident management is, and why the first minutes decide it
Data center incident management is the structured way a facility detects an event, responds to it, and learns from it. The event can be anything from a single failed power supply to a full outage that takes the floor dark. The structure is the same: spot it, classify how bad it is, put one person in charge, stop the bleeding, tell the people who need to know, and afterward find the real cause without hunting for someone to blame.
The minutes after something goes wrong are where a blip becomes a headline or stays a blip. The equipment failure is often the small part. The damage that ends up in the outage report usually comes from the response: a wrong breaker thrown in a panic, a second fault created while chasing the first, an hour lost because nobody was running the room and three people were each doing their own thing. A practiced process, rehearsed roles, and a no-blame look afterward are what turn an incident into a near-miss instead of a disaster.
This guide is the incident process specifically. The broader day-2 program, the NOC, the shifts, the rounds, and escalation, lives in the data center operations and NOC runbooks guide. The written procedures the response follows, the MOPs, SOPs, and EOPs, live in the MOP, SOP, and EOP procedures guide. This one covers what happens from the moment the alarm fires to the moment the corrective action closes.
Why the response, not the fault, decides the outcome
A single component fails in a redundant facility every week somewhere, and almost none of those make the news. The design is built to ride through a fault on one path while the load runs on another. What turns a survivable fault into an outage is usually what people do next.
Three things make a bad event worse. Panic, where someone acts before they understand the state of the plant. Coordination failure, where nobody is in charge and the room fills with well-meaning people working at cross purposes. And the diagnostic detour, where the team stops to figure out why instead of getting the service back. Each of those is a response problem, not an equipment problem, and each is preventable with a process people have actually practiced.
The flip side is the payoff. A team that declares fast, puts one commander in charge, restores first, communicates on a cadence, and learns without blame will take the same fault and close it as a footnote. The learning is what keeps it a footnote the second time too. An incident you fix but do not learn from is an incident you have scheduled to happen again.
What counts as an incident?
An incident is any unplanned event that degrades or threatens the service the facility is contracted to deliver, or that breaks the redundancy protecting it. That covers a wide range: a UPS module that drops offline, a chiller that faults on a hot afternoon, a utility blip the generators caught, a cooling loop losing pressure, a breaker that trips for no obvious reason. Loss of redundancy counts even when the load never noticed, because the next fault now has nowhere to go.
The threshold to declare is lower than most people think, and that is on purpose. You declare an incident the moment something is wrong and the normal rounds-and-runbook response is not enough on its own. You do not wait for the load to drop. A redundant facility running on a single path after a fault is already in an incident, even with every server still up, because the protection is gone and the clock is running on the repair.
Distinguish an incident from a problem. The incident is the live event you are responding to right now. The problem is the underlying cause you chase afterward so it stops recurring. Mixing the two is how teams end up diagnosing root cause during the outage instead of restoring the service, which is the detour that costs the most time.
What are the incident severity levels?
Severity is how bad it is and how much response it gets, set in the first minutes so the reaction matches the event. Most programs run a scale of roughly four levels, often written Sev1 through Sev4, where Sev1 is a critical outage or imminent loss of the load and Sev4 is a minor issue with no service impact. Some operators add a Sev0 for a true site-down catastrophe or extend to Sev5. The number of levels and the exact wording are the operator's to define, so use your own program's scale, not a generic one.
What sets the level is impact and scope, not how stressful the event feels. The questions are how much of the service is affected, how many customers or racks are hit, whether redundancy is lost or the load is actually down, and how fast it is spreading. A fault touching one customer might be a Sev3 while the identical fault across the hall touching half the floor is a Sev1.
The severity drives everything downstream: who gets paged, how fast, whether an incident commander stands up, how often updates go out, and which contractual clocks start. Set it deliberately and revise it as the picture changes. A Sev3 that turns out to be eating redundancy gets promoted to Sev1 the moment you see it, and a Sev1 that proves contained gets demoted when the facts support it.
| Level (typical) | Impact | Response posture |
|---|---|---|
| Sev1 | Outage or imminent load loss, broad scope | All-in, incident commander, full bridge, fastest cadence |
| Sev2 | Major degradation or redundancy lost, partial scope | Paged response, commander, frequent updates |
| Sev3 | Limited impact, workaround exists, contained | On-call handles, lighter updates, tracked |
| Sev4 | Minor issue, no service impact | Logged and worked in normal flow |
Declare it early and escalate hard
The most common timing mistake is under-calling: treating a real incident as a routine ticket because admitting it is an incident feels like an overreaction. It is the opposite. The cost of declaring an incident that turns out minor is a few people briefly pulled in and stood back down. The cost of not declaring one that turns out major is the half hour you lost before anyone was actually running the response.
Declare early. Escalate hard. If you are debating whether it crosses the line, it crosses the line, and you can always downgrade once you know more. Standing a response down is cheap and nobody remembers it. Standing one up late is the line in the outage report that starts with the words the operator never wants to read.
Build the bias for early declaration into the culture, not just the runbook. An operator who gets second-guessed for calling an incident that fizzled will hesitate next time, and the hesitation is exactly what costs you on the one that does not fizzle. Reward the call, not the outcome.
What is an incident commander?
The incident commander is the one person who runs the response. Not the most senior person in the room, not the best engineer, the person holding the role. They own the decisions, the priorities, and the tempo, and everyone in the response answers to them for the duration of the event. When the incident is declared, the IC is named out loud so there is no ambiguity about who is in charge.
The defining rule of the role is that the commander coordinates, they do not do. The instant the IC is heads-down inside a panel or buried in a log, nobody is steering, and the response loses the one person whose job is to keep the whole picture. The IC delegates the hands-on work, holds the timeline, decides restore-versus-diagnose, controls who touches what, and keeps communication flowing. Their hands stay off the equipment so their head stays on the room.
Under pressure the role is mostly about staying calm and asking the right questions. What is the current state. What have we tried. What is the fastest path to restore. Who is doing it and who is verifying. What do the customers need to hear and when. A good IC makes the room quieter, slower, and more deliberate while the clock says hurry. That is the skill, and it is learned by drilling, not by reading.
The roles during the event
A real response is a small org chart that stands up in minutes and stands down when it is over. The incident commander runs it. Underneath, a few defined roles keep one person from trying to do everything at once, which is the failure mode the structure exists to prevent.
The technical lead or operations lead owns the actual hands-on work, directing the operators making moves on the plant while reporting state back to the IC. The communications lead handles the message: stakeholder updates, customer notifications, and fielding the inbound so the technical people are not answering phones while they work. The scribe keeps the timeline, logging what happened and when. And the bridge, a call or chat channel, is where the response coordinates so everyone is working from the same picture.
On a small overnight shift one person may wear two hats, and that is fine as long as it is deliberate and the IC role is never the one folded into hands-on work. The structure scales down. What does not scale down is the principle that someone is coordinating and someone is acting, and they are not the same person on a serious event.
| Role | Owns | Stays out of |
|---|---|---|
| Incident commander | Decisions, priorities, tempo, the whole picture | Hands-on repair |
| Technical / operations lead | The hands-on work and plant state | Customer comms |
| Communications lead | Stakeholder and customer updates, inbound | Touching equipment |
| Scribe | The timeline and the record | Making moves |
Restore the service first
The first job in an outage is to get the service back, not to understand why it broke. Restore first. Stabilize the load, recover redundancy, and make the facility safe before anyone opens an investigation into root cause. The why can wait. The load cannot.
This is the discipline that separates a practiced operation from a chaotic one, and it is the one most people get backwards under pressure. The engineer's instinct is to understand the failure, and understanding feels like progress. In an outage it is a detour. Every minute spent diagnosing why the UPS dropped is a minute the load is exposed, when the right move might be to transfer to the alternate path now and learn why afterward. The incident commander's job is to hold that line and stop the room from drifting into diagnosis while the service is down.
Restore-first does not mean act blindly. It means the goal is recovery, and the actions are the pre-written stabilizing steps in the emergency operating procedure, not improvisation. The EOP and the broader procedure set are covered in the MOP, SOP, and EOP guide. The point here is the priority order: service back, then redundancy back, then root cause, in that sequence, every time.
The EOP is the response you act on
Under pressure people do not rise to the occasion, they fall to their level of training, and the emergency operating procedure is that training written down. The EOP is the pre-built response to a specific failure: utility loss, generator or UPS failure, cooling loss, a leak, a fire. It leads with the immediate actions to protect the load and people, then the steps to restore redundancy and isolate the trouble, with the escalation path and contacts.
The reason it works is that the thinking happened earlier, when there was time to think. Mid-outage, with alarms going and a customer on the phone, is the worst possible moment to reason from first principles about a power topology. The EOP lets the team act correctly and fast because someone already worked out the right sequence and tested it. The MOP, SOP, and EOP guide covers how those procedures are written, graded, and kept current. In the response, the EOP is the script the incident commander runs the room from.
Do no harm: the panicked action that makes it worse
The fastest way to turn an incident into a disaster is a confident, wrong action taken in a hurry. The breaker thrown to the wrong bus. The transfer made onto a path that was not actually healthy. The reset that drops a load that was still running. A large share of the worst outages on record were not caused by the original fault but by what someone did to it in the first ten minutes.
Slow is smooth and smooth is fast. The IC controls who touches what, and on a serious event no consequential action goes to the equipment without being called out, confirmed against the procedure, and ideally verified by a second person before the hand moves. The pressure to act is real and it is exactly when the discipline matters most. A wrong move on live critical plant does not undo cleanly.
If the team is not sure an action is safe, the default is to not take it and to escalate for a second opinion. Doing nothing while you confirm is almost always recoverable. Doing the wrong thing fast often is not. The procedure exists so that the right action is the obvious one, and the IC exists so that the wrong one does not happen because three people were shouting at once.
How do you communicate during an outage?
Communication during an incident is a managed channel, not a reflex. The communications lead runs it so the technical people can work, and the message goes out on a deliberate cadence to the people who actually need it: internal leadership, affected customers, and any party the contract names. The bridge call or channel keeps the response itself coordinated, separate from the outbound updates to stakeholders.
The hardest discipline is sending updates when there is nothing new to report. Silence reads as loss of control, and a customer who hears nothing assumes the worst and starts calling. An update that says we are still working it, here is what we know, here is when we will update next, holds the relationship even when the news is bad. Set the next-update time and hit it, every time, even if the only content is that the previous update still stands.
Be accurate and be careful. During the event you report impact and status, not unconfirmed cause. Speculating about root cause in a customer update, or naming a vendor or a person before the facts are in, creates a second problem on top of the outage. State what is affected, what you are doing, and when the next update comes. Save the why for the root-cause report, where it belongs and where it can be said carefully. The exact notification timing and audiences are usually set by the operator's policy and the contract, so manage to those.
Status updates and the cadence
The cadence is the rhythm of updates, and it scales with severity. A Sev1 might update every 15 to 30 minutes; a contained Sev3 might update hourly or at milestones. The exact intervals belong to the operator's policy and, for customers, the contract, so set the cadence to those and hold it. The number matters less than the reliability of hitting it.
A good update has a shape: what is happening, what the current impact is, what is being done, and when the next update will come. Keep it short and keep it honest. The next-update commitment is the part that buys patience, because it tells the reader they do not need to chase you. Miss the committed time and you have spent the trust the cadence was protecting.
Many SLAs put a clock on the first notification, commonly somewhere in the range of 15 to 60 minutes from the moment service is determined unavailable, with the exact figure and audience set by the contract. Know your numbers before the event, because the first notification clock is running while the team is still figuring out what broke, and a late notice can be a contractual breach on its own regardless of how the recovery goes.
The scribe and the timeline
The scribe logs the timeline as the incident unfolds: when the alarm fired, when the incident was declared, when the IC was named, what was tried, what worked, when service came back. The log is written in the moment because nobody reconstructs an accurate timeline from memory afterward, and the timeline is the raw material the root-cause analysis runs on.
Capture facts with timestamps, not interpretations. The breaker tripped at 02:14. The transfer to the alternate feed completed at 02:21. The load was restored at 02:23. What it means comes later, in the post-mortem, with the pressure off. During the event the scribe's job is to get the sequence down accurately so the analysis afterward is arguing about a real record instead of three people's conflicting memories.
The honesty of the timeline depends on the blameless culture. If people know the log will be used to find fault, the record gets sanitized and the real sequence is lost. The timeline is only as truthful as the culture around it makes safe.
Stabilize and watch before you call it over
Service back is not incident over. After the restore comes the watch: confirm the load is stable, confirm redundancy is actually back and not just apparently back, and monitor for the fault returning before anyone declares victory. The early all-clear that turns into a second outage twenty minutes later is a familiar and avoidable embarrassment.
Hold the response open through a deliberate stabilization period sized to the event. A power transfer that restored the load might be running on a single path until the failed gear is repaired, which means you are recovered but not redundant, and that state needs to be named clearly so nobody relaxes while the facility is still exposed. The IC closes the incident when the facility is genuinely stable, not when the first alarm clears.
What is root cause analysis?
Root cause analysis is the structured look, after the service is back, for what actually caused the incident rather than the symptom that showed first. The symptom was the breaker tripped. The cause was the setting that was changed during a job last month and never verified. RCA is the work of getting from the first to the second, so the fix lands on the thing that will stop it happening again.
The 5-whys is the common method and it earns its keep when used honestly. You ask why, then why that, peeling back layers until you reach a cause you can actually fix rather than another symptom. The breaker tripped. Why. The bus was overloaded. Why. A load was added without an updated calculation. Why. The change process did not require one. That last answer is a systemic fix; the first was just a description.
Real incidents rarely have one clean cause. They have contributing factors, a chain where a technical fault met a procedural gap met a missed alarm, and any one of them broken would have stopped the outage. Look for the chain, not the single villain, and weigh the human and the technical together. The human action that triggered it almost always sits on top of a system that allowed it, and the system is the part you can actually change.
What is a blameless post-mortem?
A blameless post-mortem is the review of an incident that focuses entirely on what in the system let it happen, never on who to punish for it. The premise is that everyone involved acted reasonably given what they knew at the time, so the question is never why was this person careless, it is what about the tools, the procedures, the training, or the conditions made the wrong action the easy or obvious one. This is the most important part of the whole process, and the one most often done badly.
It works because of what it protects: the truth. People only tell you what actually happened, including their own mistakes, when telling the truth does not get them fired. Punish the operator who pushed the wrong button and you will never again hear the real sequence of events from anyone, the timelines turn defensive and vague, and you lose the ability to find the cause at all. The blameless rule is not about being soft. It is the only way to get accurate information out of the people who were there.
This sits inside what reliability and safety-critical fields call a just culture, an idea that came up through aviation and healthcare, where a buried mistake gets someone killed. The line it draws is between honest error inside the system, which the system owns and fixes, and reckless or malicious behavior, which is a different conversation. Honest error is the overwhelming majority, and treating it as a learning input rather than a firing offense is what makes the whole incident process work. The owner sets the program; hold the post-mortem to it.
The human-error reality
Most data center outages involve human action, and the data has been consistent on this for years. Uptime Institute's outage analyses have long put roughly 70 percent of outages down to human error rather than a fault in the design, and their more recent work ties the large majority of those to staff not following procedures or to the procedures themselves being flawed. The number moves a little year to year, and the exact figure is theirs to publish, but the direction has not changed: the people and the process, not the iron, are where outages come from.
The right reading of that statistic is not that operators are careless. It is that the systems around them let an ordinary mistake become an outage. A label that was wrong, a procedure that had a gap, an alarm that was buried in noise, a training that never covered the scenario. These are latent conditions, the holes that sit quietly in the system until an operator's action lines up with them on a bad day. Blaming the operator fixes nothing because the next operator faces the same holes.
So the fix is systemic, not personal. Fix the label, close the procedure gap, tune the alarm, add the training, build the interlock that makes the wrong action impossible instead of merely discouraged. An incident process that ends with someone disciplined and the system unchanged has learned nothing and will see the same outage again with a different name on it.
Corrective actions, tracked to closure
The post-mortem is worthless if its findings die in a document. Corrective and preventive actions are the concrete changes that come out of the root-cause analysis, each one assigned to an owner with a due date and tracked until it is actually done. This is the part that closes the loop, and it is the part that quietly fails most often.
Make the actions specific and verifiable. Update the EOP to add the verification step is an action; improve communication is not. Each action should trace to a cause the analysis found, so you can see that the fix addresses the real hole and not a symptom. The mix usually spans procedure changes, training, monitoring or alarm tuning, and physical or design fixes, and each needs an owner who is accountable for getting it across the line.
Track them like open work, because they are. An action with no owner and no date is a wish. Review the open corrective actions on a cadence, escalate the ones that stall, and do not consider the incident truly closed until they are done and verified. The discipline of closing the loop is the difference between an operation that gets steadily more reliable and one that keeps relearning the same lesson.
Preventing the repeat
The whole point of the learning is the incident that does not happen again. A program that is working shows it in the trend: the same cause does not show up twice, the corrective actions from past events are visibly done, and the lessons are folded back into the procedures and the drills so the next shift inherits them.
Watch the patterns across incidents, not just the single event. Three separate incidents that each traced to a stale drawing are not three problems, they are one problem about drawing control wearing three disguises. The repeat incident is the clearest signal that a corrective action was either never finished or never addressed the real cause. When you see the same thing twice, the failure is in the loop-closing, not the bad luck.
The SLA, the outage report, and credits
For a colocation or service customer, the incident has a contractual life that runs alongside the technical one. The service level agreement typically sets a notification clock, an uptime commitment, and a remedy when the commitment is missed, usually service credits against the bill. The numbers vary widely by contract, and the contract is the authority, so read yours rather than assuming an industry default.
The formal deliverable to the customer is the outage report, often called the RCA, a written account of what happened, the impact, the cause, and the corrective actions, due within a window the contract defines, commonly a handful of business days. It is a customer-facing document and it gets read by people deciding whether to renew, so it has to be accurate, honest, and free of the speculation that has no place in a customer update. The internal post-mortem feeds it, but the report is written for the contract, not for the engineers.
Know the contractual mechanics before the event, not during it. Credits often require the customer to claim within a defined window, the notification clock may carry its own separate penalty, and the uptime math has definitions that matter. Treat the SLA terms as part of the incident response, because the financial outcome of an outage is decided as much by how the contract is handled as by how fast the load came back. The specific figures, clocks, and definitions are set by the operator and the contract.
Drill the response before you need it
An incident process that lives only in a binder does not work when the alarm goes off. The response has to be drilled until the roles, the sequence, and the decisions are muscle memory, because the real event is the worst time to be reading a procedure for the first time. Practice is what makes the difference between a team that falls apart and a team that falls back on its training.
Run two kinds. The tabletop is a discussion-based walkthrough where the team talks through a scenario, surfaces gaps in the plan, and clarifies who does what, usually a couple of hours and run once or twice a year as a baseline. The live drill is the harder, more valuable version, where the team physically executes an EOP on the plant in a controlled window, proving the procedure works and the people can do it. The tabletop finds the gaps cheaply; the live drill finds the ones the tabletop missed.
Drill the parts that fail under pressure: the handoff to the incident commander, the communication cadence, the restore-first decision when the instinct is to diagnose. Inject surprises so the team practices adapting, not reciting. And debrief every drill the same blameless way you would a real incident, because the gaps a drill exposes are free lessons you got without an outage to pay for them.
Incident metrics and the trend
You measure the response so you can improve it, and the core metrics are time-based. MTTD, mean time to detect, is how long the fault existed before anyone knew. MTTR is the one to be careful with, because it gets used for mean time to restore service, mean time to repair the gear, and mean time to resolve the whole incident including the fix, and those are different numbers. Be explicit about which you mean, because a team can have a fast time to restore and a slow time to permanently resolve, and the gap tells you something about the post-mortem process.
Read the metrics as a trend across many incidents, not as a verdict on a single one. One bad MTTR on a genuinely novel failure says little; a creeping MTTD across a quarter says your detection is degrading and it is worth a look. Mean time to acknowledge is also worth watching, because a slow acknowledge points at paging or staffing, not at the repair itself.
Hold the metrics loosely as targets, not as a stick. The moment time-to-restore becomes a number people are punished for, the pressure pushes against the restore-first and do-no-harm discipline, and you get fast, reckless action instead of fast, correct action. What the metric should be tied to is the operator's reliability goals, not an individual's review. Set the targets to your own program.
The incident record
Every incident leaves a record, and the record is what the program runs on after the event is over. The timeline, the severity, the actions taken, the root-cause analysis, and the corrective actions with their owners and due dates all live in one place so they can be tracked, trended, and audited. An incident with no durable record is an incident the operation cannot learn from, because there is nothing to review and nothing to close.
Whatever tool holds it, the record has to capture the sequence as it happened and stay accessible afterward. A field platform such as FieldOS can hold the incident log, the timeline, the photos and readings captured during the response, and the corrective-action list, so the record built in the heat of the event survives the shift change and the open actions stay visible until they close. The discipline matters more than the tool: an incident is not closed when the load comes back, it is closed when the record is complete and the corrective actions are done.
Field checklist
Want this checklist to run itself on every job — with photo proof and a signed record crews can hand the customer? That's FieldOS.
What to document
The record is built in two passes. During the event the scribe captures the timeline and the actions as they happen. After the event the post-mortem adds the root cause and the corrective actions. Both passes feed the customer-facing outage report and the internal trend.
Capture the phase, the action, and a note for each entry, with a timestamp, so the sequence reconstructs cleanly and the analysis afterward argues from a real record. The table below is the shape of an incident log entry.
| Phase | Action | Note |
|---|---|---|
| Detect | Alarm received, fault identified | Timestamp, what fired, who saw it |
| Declare | Incident declared, severity set | Severity and the basis for it |
| Command | Incident commander named, roles assigned | Who holds IC, tech lead, comms, scribe |
| Restore | Stabilizing actions, service recovered | Steps taken, EOP used, restore time |
| Communicate | Stakeholder and customer updates sent | Times, audiences, next-update commitment |
| Stabilize | Redundancy confirmed, incident closed | Stabilization period, true close time |
| Learn | Root cause and corrective actions | Cause chain, owners, due dates, closure |
Common mistakes
- Under-declaring the severity, so the response is too small and starts too late.
- No incident commander, so the room fills with people working at cross purposes and nobody is steering.
- Stopping to diagnose root cause while the service is still down, instead of restoring first.
- A panicked, confident, wrong action on live plant that turns one fault into two.
- A blame culture that makes people sanitize the timeline, so the real cause is never found.
- No follow-through on corrective actions, so the same incident returns under a new name.
Standards and references
Incident and outage management in data centers borrows its discipline from a few places, and naming them correctly matters more than citing a single rulebook. The Uptime Institute frames it through operational sustainability and its M&O practices, and its annual outage analyses are the widely cited source for the human-error and procedure-failure data; treat the specific percentages as theirs and verify against the current report. ITIL gives the service-management vocabulary, the distinction between an incident and a problem, severity and priority, and incident versus problem management. The SRE practice from the software reliability world contributes the incident command model, the IC-coordinates-not-does rule, the named response roles, and the blameless post-mortem, which itself traces back to just-culture thinking out of aviation and healthcare.
The procedures the response runs on, the EOPs, sit in the operator's own work-control program and are covered in the MOP, SOP, and EOP guide. The severity scale, the notification cadence, the metric targets, and the SLA remedies are not set by any external standard; they are defined by the operator's policy and by the contract, and those are the authority. Where a number here is given as a range, read it as a starting point and set your own against your program and your agreements.
Cite the framework that actually governs the point. Use ITIL for the incident-versus-problem language, the SRE incident-command model for the roles and the blameless review, and the Uptime Institute for the operational practice and the outage data. Let the operator's program and the contract control severity, cadence, metrics, and credits, because those genuinely vary and no external document overrides them.
Terms and definitions
Incident response carries its own vocabulary, and the same word can mean slightly different things across ITIL, SRE, and a given operator's program, so confirm against your own definitions.
The terms below are the ones that show up in almost every incident, from the declaration to the post-mortem.
- Incident
- An unplanned event that degrades or threatens the contracted service or breaks the redundancy protecting it, from one failed component to a full outage
- Severity (Sev1 to Sev4)
- The classification of how bad an incident is, by impact and scope, that drives the size and speed of the response; the scale is the operator's to define
- Incident commander (IC)
- The one named person who runs the response, owns the decisions and tempo, and coordinates rather than performing the hands-on repair
- Root cause analysis (RCA)
- The structured search after restore for the real cause and contributing factors, not the first symptom, often using the 5-whys
- Blameless post-mortem
- The review that focuses on the systemic conditions that allowed the incident, never on punishing an individual, so people tell the truth
- MTTD / MTTR
- Mean time to detect a fault; mean time to restore, repair, or resolve, which are different measures, so state which one you mean
- Corrective action
- A specific, owned, dated change that fixes a cause found in the RCA, tracked until it is done and verified
FAQ
Why does a data center need a formal incident process?
Because the response, not the fault, usually decides the outcome. A single component failing is survivable in a redundant facility; what turns it into an outage is a panicked or uncoordinated reaction. A practiced process with a named commander, a restore-first order, and blameless learning turns the same fault into a near-miss instead of a headline.
What is an incident commander?
An incident commander is the one named person who runs an incident response. They own the decisions, priorities, and tempo, and everyone answers to them for the duration. The defining rule is that they coordinate rather than do: their hands stay off the equipment so they can keep the whole picture and steer the room.
What is a blameless post-mortem?
A blameless post-mortem is the review of an incident that focuses on the systemic conditions that allowed it, never on punishing a person. It assumes everyone acted reasonably with the information they had. The point is to protect the truth, because people only report their own mistakes honestly when doing so does not get them fired.
What is the difference between restoring service and root-causing an incident?
Restoring service is getting the load and redundancy back, which is the first priority during the outage. Root-causing is finding why it broke, which happens after the service is back. Mixing them is the classic mistake: stopping to diagnose while the service is down costs time the restore-first discipline is built to protect.
How quickly should you declare a data center incident?
Declare early, the moment something is wrong and the normal rounds-and-runbook response is not enough on its own, including when redundancy is lost but the load is still up. Under-calling costs far more than over-calling, since standing a response down is cheap and standing one up late is what fills the outage report.
What causes most data center outages?
Human action and process, not failed equipment. Uptime Institute outage analyses have long put roughly 70 percent of outages down to human error, with most of those tied to staff not following procedures or to flawed procedures. The fix is systemic, fixing the procedures, training, and alarms, not blaming the operator.
What are incident severity levels?
Severity levels classify how bad an incident is by impact and scope, commonly Sev1 through Sev4, where Sev1 is an outage or imminent load loss and Sev4 is a minor issue with no service impact. The level drives who is paged, the update cadence, and which contractual clocks start. The operator defines the scale.
What do I do if a data center outage hits an SLA?
Start the contractual clock alongside the technical response: notify within the SLA window, hold the update cadence, and prepare the outage report, the RCA deliverable, due within the contract's window. Service credits and notification penalties follow the contract's own numbers and claim rules, so manage to the agreement, not to an industry default.
What is MTTR in incident management?
MTTR is a time-based incident metric, but it ambiguously means time to restore service, time to repair the gear, or time to resolve the whole incident including the fix. State which you mean. Read it as a trend across many incidents, set targets to the operator's reliability goals, and never weaponize it against the restore-first discipline.
How often should a data center drill its incident response?
Run tabletop walkthroughs at least once or twice a year as a baseline and after any major change, plus live EOP drills in controlled windows to prove the procedures and the people. Drill the parts that fail under pressure, the IC handoff, the comms cadence, the restore-first call, and debrief each one blamelessly like a real incident.
People also ask
Codes cited in this guide
This guide is written and reviewed against the published standards below. Always confirm the current adopted edition with the authority having jurisdiction.