MTTR · 24Observe
MTTR explained: how to measure it and reduce incident recovery time
Swipe sideways to see the whole diagram.
What is MTTR? Repair, recovery, respond and resolve explained

MTTR is an average duration associated with repairing a failure, recovering service, responding to an incident or resolving it. Its meaning depends on where you start and stop the clock.
Choose the expansion based on the operational question. Time spent waiting for service to return differs from time spent making repairs or time an incident sits unattended. Each needs its own clock, even when dashboards label all of them “MTTR.”
Document the measurement boundary before setting a target. Specify which event starts the clock, which event stops it and what evidence verifies each. 24Observe is built by our group, so weigh our view accordingly.
For service-impact reporting, measure from impact onset to verified restoration. Measure active repair and substantive response separately, and record administrative closure on its own. Service can be restored before every permanent corrective action is complete.
Four meanings, four different clocks
The examples below propose a local measurement policy. Definitions vary, so agree on these boundaries with your NOC, SOC and service owners before comparing results.
| Expansion | What it measures | Example start | Example finish | Interpretation risk |
|---|---|---|---|---|
| Mean time to repair | Active work repairing the failed component | Hands-on repair begins | Repair and component checks finish | Excludes waiting and earlier interruption; elapsed job time may also include pauses unless explicitly excluded. |
| Mean time to recovery or restore | Elapsed service-impact duration | Impact begins | Service restoration is verified | Starting at detection misses earlier impact; a healthy component alone does not prove service recovery. |
| Mean time to respond | Delay before meaningful response begins | An actionable incident is opened | A responder begins documented triage or containment | Acknowledging a page proves receipt, not substantive response. |
| Mean time to resolve | Duration until the agreed resolution endpoint | Incident is opened | Verified restoration under this policy | “Resolved” could instead mean mitigated impact or administrative closure; those endpoints produce different results. |
For response, define the first substantive response as an action supported by evidence. That could mean examining relevant logs to test an incident hypothesis or beginning containment. Clicking “acknowledge” alone should leave the clock running. Clear on-call responsibilities establish who must act. Addressing alert fatigue remains a separate operational task.
What recovery MTTR means
LaunchDarkly’s guide describes restoration MTTR as the average interval required for recovery after an outage or system failure. The calculation divides accumulated downtime within a reporting window by the number of failure incidents in that window:
Recovery MTTR = total downtime ÷ failure-incident count
Under the proposed policy above, each incident contributes the interval from impact onset through verified restoration. Evidence from uptime monitoring and broader observability can help establish those boundaries. The team still needs to specify what constitutes restored service.
A workaround can restore service while corrective engineering remains open. Moving traffic away from a failed component, for example, may end user impact before engineers repair that component.
Google’s SRE Workbook describes resolution in terms of reducing impact, returning service to its prior state, or both. It gives ending impact priority over root-cause investigation unless the cause becomes clear early.
Record restoration and closure separately in your incident record so the recovery metric stays tied to service impact. Document the verification steps in the runbook, and keep permanent corrective work visible and assigned.
How do MTTD, MTTA and MTTR fit the incident timeline?
MTTD, MTTA and MTTR measure different parts of an incident. Before comparing them, define the event that starts each clock and the evidence that stops it.
- Mean time to detect (MTTD) is the average interval from incident onset, defined here as the start of service impact, to detection. In this example, detection occurs when the first actionable alert is generated. A failed uptime check could provide that signal.
- Mean time to acknowledge (MTTA) is the average interval from a defined alert milestone to recorded acknowledgment. Here, the clock starts at alert generation, rather than delivery or incident creation. Document that choice in your on-call procedure.
- MTTR is a recovery average whose meaning depends on its boundaries. Specify whether your recovery measurement starts at impact or detection, and what counts as recovered.
Acknowledgment records receipt or ownership according to the team’s policy. It provides no evidence that diagnosis or mitigation has started. Google’s incident-management guidance also distinguishes responsibilities and calls for explicit confirmation when command changes hands. Track ownership changes separately from technical progress.
One fictional incident, several valid clocks
This example is entirely fictional, with every timestamp falling on the same day.
| Event | Timestamp | Meaning for measurement |
|---|---|---|
| Service impact begins | 09:00 | Impact-based clocks start |
| First actionable alert generated | 09:04 | Detection recorded; example MTTA starts |
| Incident record created | Not recorded | Separate from alert generation |
| Alert delivered | Not recorded | Separate from generation and creation |
| Responder acknowledges | 09:07 | Receipt or ownership recorded |
| Diagnosis reaches a mitigation decision | 09:19 | A mitigation approach is selected |
| Mitigation applied | 09:27 | Intervention completed; recovery remains unverified |
| Service health verified | 09:32 | Recovery endpoint reached |
| Administrative closure | 09:40 | Incident record closed |
The creation and delivery timestamps are unknown. Leave them that way rather than substituting the alert-generation timestamp. Generation records when the alert existed; delivery records when it reached its destination.
The incident produces these measured durations:
| Measurement | Timestamp boundaries | Duration |
|---|---|---|
| Detection | 09:00 to 09:04 | 4 minutes |
| Acknowledgment | 09:04 to 09:07 | 3 minutes |
| Impact to verified recovery | 09:00 to 09:32 | 32 minutes |
| Alert to verified recovery | 09:04 to 09:32 | 28 minutes |
| Impact to administrative closure | 09:00 to 09:40 | 40 minutes |
Each duration belongs to a single incident. To calculate means, average the corresponding durations across the incidents included in your reporting period.
Keep recovery, acknowledgment and closure distinct
The 32-minute duration measures how long service impact lasted. Measured from the actionable alert, recovery took 28 minutes. Both figures are useful when their labels make the starting points clear.
Adding the four-minute detection interval to an MTTR measure that starts at impact would count that interval twice. Applying mitigation at 09:27 also leaves recovery unconfirmed until verification at 09:32. Keep both milestones in the response record.
24Observe is built by our group, so weigh our view accordingly. It has separate webhook events for incident opening, acknowledgment and resolution. Its delivery-testing documentation also explains that successful incident creation can be followed by failed notification delivery. Keep those events separate when defining your measurements.
For security incidents, track containment, service recovery and final case closure on separate clocks. Even where NOC and SOC responsibilities overlap, these endpoints remain distinct: restoring availability alone provides no confirmation that the threat is contained.
How do you calculate MTTR correctly?
Before calculating MTTR, define which incidents qualify and which timestamps start and stop the clock.
MTTR = sum of qualifying incident durations / number of qualifying incidents
For this worksheet, recovery duration = verified restoration − impact start, measured in elapsed minutes. Use the same clock definition, unit, incident scope and reporting policy for every included duration. Mixing impact-to-restoration measurements with acknowledgment-to-closure measurements produces an inconsistent calculation.
Start with one service and specify the incident types and severities to include. DORA’s measurement guidance recommends analysing individual services and warns against grouping dissimilar operational contexts. When no incidents qualify, report “not applicable” rather than zero.
Worked example: mean and median
These recovery durations are entirely hypothetical. They represent five qualifying incidents for one service.
| Incident | Recovery duration |
|---|---|
| INC-01 | 10 minutes |
| INC-02 | 20 minutes |
| INC-03 | 30 minutes |
| INC-04 | 40 minutes |
| INC-05 | 200 minutes |
| Total | 300 minutes |
Mean MTTR = 300 / 5 = 60 minutes. The median is 30 minutes, the middle duration when the five values are sorted.
The mean accounts for all recorded recovery time, including the lengthy incident. The median identifies the middle observation, so that extreme value has less influence on it. Report both with the incident count; neither measure replaces the other.
Keep summed incident durations separate from elapsed service unavailability. In a hypothetical example, two incidents affecting the same service from 10:00 to 10:30 contribute 60 incident-minutes, although the service was unavailable for 30 elapsed minutes. Merge overlapping outage intervals when calculating service unavailability. For incident MTTR, keep each qualifying incident’s duration.
Keep a timestamp worksheet
Create one row for each incident. In this worksheet, D identifies duration inputs, while S identifies fields used for selection or record control. The remaining timestamps help with investigation; they are excluded from this formula.
| Worksheet field | Required use |
|---|---|
| Incident identifier | S: unique record; prevent duplicate counting |
| Service | S: service being measured |
| Incident type | S: qualifying category |
| Severity | S: qualifying severity |
| Impact start | D: beginning of evidenced service impact |
| Detection | Supporting timeline timestamp |
| Acknowledgment | Supporting timeline timestamp |
| Mitigation | Supporting timeline timestamp |
| Verified restoration | D, S: recovery endpoint and period selection |
| Closure | Administrative timestamp, separate from restoration |
Agree in advance on the evidence needed to confirm restoration, such as successful checks against the service’s agreed recovery criteria. Completing a mitigation action alone does not establish that service has been restored.
Document the reporting rules
- Leave impact onset unknown unless there is evidence to support an estimate. For any estimate, record its basis and uncertainty. Exclude records without a defensible start time from this calculation and disclose how many were excluded. Substituting detection time without explanation changes the clock being measured.
- Store timestamps with full dates and explicit time-zone offsets. Normalise calculations to UTC, keeping the original timestamps available for investigation. Local clock times alone can be ambiguous when daylight-saving time changes.
- When correcting a timestamp, preserve the original and corrected values along with the reason, editor and revision time. Version the report or state when historical results were recalculated.
- Document how incidents are selected at reporting boundaries. For example, include incidents whose verified restoration occurs within the reporting month, with an inclusive start and exclusive end. Count each incident’s full duration even if its impact began in the previous month. Do not clip durations at midnight.
- Exclude incidents that lack verified restoration from completed-recovery MTTR. At the reporting cutoff, publish their count and a separate view of open-incident age. Never give an unfinished incident a duration of zero. A ticket that remains administratively open can still qualify under this endpoint-based policy if restoration has been verified.
For implementation references, 24Observe is built by our group, so weigh our view accordingly. Its incident records, event webhooks and API documentation offer references for the workflow. When planning restoration evidence, consult the documentation for logs, metrics and uptime monitoring, together with the playbook documentation and general documentation. These references do not mean that every worksheet field is captured automatically.
To test reproducibility, give another engineer your selection policy and source records, including timestamp revisions and the calculation. They should be able to reproduce the incident selection and durations, then arrive at the same result.
Which MTTR mistakes make incident reports misleading?
Before celebrating a lower MTTR, check whether recovery improved or the reporting rules changed.
Statistical errors that hide the distribution
-
Relying on the mean alone. Show individual incident durations and the median alongside the mean, and include the sample size. Add a suitable tail measure only when there are enough observations to support it. Percentiles from tiny samples give an unstable picture of recovery performance. Keep definitions consistent and calculations visible so readers can assess the results. Your MTTR reporting should explain these choices clearly enough for another reader to check the calculation.
-
Silently deleting outliers. Removing a long-running incident can reduce the mean even when incident handling stays unchanged. Treat that as a reporting change; it provides no evidence of better recovery. Record an eligibility reason for each exclusion and support it with incident evidence. Duplicate records and events outside the declared scope may qualify. An incident being inconvenient does not. Preserve the original result and disclose what you excluded.
-
Averaging team averages without weighting. Divide the total duration of eligible incidents by the total count of eligible incidents. You can also weight each team’s mean by its incident count. Equal weighting across teams answers a different question.
-
Mixing severities or services. Before comparing periods, separate incident populations that differ materially. More minor incidents in the mix can bring down the overall mean while major-outage recovery remains unchanged. Show the service context and use trends to guide improvements rather than rank teams.
Clock and counting errors
-
Changing clock boundaries or hiding detection time. State which events start and stop the clock, and version changes to those definitions. When the clock starts at detection, disclose that it leaves out pre-detection impact. Report that interval separately when it is known. Uptime monitoring and responder timestamps cover different portions of the incident timeline.
-
Counting alerts, cases and failures interchangeably. Multiple alerts for a single failure should not increase the denominator. Overlapping failures still need to be counted as distinct failures. Define your rules for grouping related incidents into cases and distinguish that grouping from evidence of shared-root context. That grouping does not automatically make a case the right reporting unit. Specify whether you count failure events or episodes of service impact. When reporting service availability, take care not to count overlapping downtime twice.
-
Treating closure as restoration. Require evidence that the affected user journeys have recovered, and record restoration separately from administrative closure. Keep manual and automatic closure distinct: document how resolution events are generated and whether your process requires operator closure for log-alert incidents. Closure alone cannot establish that service has recovered, regardless of the mechanism. When an incident reopens, preserve its original timeline and document whether the impact continued or a separate failure occurred.
Incentives that distort capture
- Rewarding a lower number without checking completeness. Targets should not encourage teams to close incidents prematurely, downgrade severity or leave incidents unrecorded. Assess recovery duration alongside incident frequency, customer impact and unresolved work. Recovery can get faster while failures remain frequent. Check that a fall in recorded failures reflects better operations rather than incomplete incident capture.
MTTR vs MTBF: are you recovering faster or failing less often?
Mean time between failures (MTBF) measures reliability for a repairable system: the operating time it accumulates per failure. MTTR measures how long recovery takes after a failure. Here, MTTR runs from service disruption to confirmed restoration.
Calculate MTBF = total operating time ÷ qualifying failures within a defined observation window. Make the measurement policy explicit. Use one service boundary, exclude outage time and planned maintenance from operating time, and count each distinct unplanned service interruption once. Duplicate alerts do not count as additional failures. Decide whether degraded service qualifies, then apply that definition consistently. If no failures occur, report the observation time and failure count rather than claiming infinite MTBF.
| Metric | Question answered | Required records | Interpretation |
|---|---|---|---|
| MTBF | How long does the service operate between failures? | Operating intervals and qualifying failure count | Higher suggests failures are less frequent under the same measurement policy. |
| MTTR | How long does recovery take? | Disruption and restoration timestamps for each incident | Lower indicates shorter average recovery, not fewer failures. |
| MTTD | How long before a failure is detected? | Failure-onset and detection timestamps | Helps identify detection delays. |
| MTTA | How long before someone acknowledges the alert? | Alert-delivery and acknowledgment timestamps | Helps identify response pickup delays. |
Use MTTD and MTTA to diagnose delays in the incident workflow. Neither replaces a measure of failure frequency or recovery duration.
Two illustrative service patterns
The following examples are hypothetical. They do not represent observed customer results:
- A service has 12 failures, with restoration taking five minutes each time. Its five-minute MTTR shows short recovery times, while users still face frequent interruptions.
- Another service has one failure during the same observation window and takes six hours to restore. It fails less often, but recovery remains difficult.
A recovery average alone cannot establish overall reliability. Assess it alongside failure frequency and user impact. Track the same service over time rather than ranking unrelated teams. Avoid converting MTBF and MTTR into a universal availability percentage: simplified formulas depend on assumptions that need separate treatment.
24Observe is built by our group, so weigh our view accordingly when using its guides. Supporting reading covers uptime, observability, monitoring distinctions and on-call practice. When investigating these patterns, consider recovery measurement and log management as well as alert fatigue and API monitoring. The documentation and blog index offer additional reading. Assess any guidance against your service boundary, incident definitions and observation window before drawing conclusions about reliability or recovery.
Is MTTR still a DORA metric?
DORA now calls its recovery measure failed deployment recovery time. Historically, the measure was associated with MTTR and the time needed to restore service. The current definition is narrower: it covers recovery when a failed deployment requires immediate intervention.
An urgent rollback after a deployment falls within this scope. An unrelated network outage or security investigation does not qualify simply because responders recorded how long recovery took. MTTR across all incidents remains useful, but it covers a different set of incidents.
What are the five current DORA measures?
DORA’s current framework distinguishes delivery throughput from deployment instability.
| Group | Measure | What it measures |
|---|---|---|
| Throughput | Deployment frequency | How often deployments occur, expressed as a count within a period or the interval separating deployments. |
| Throughput | Change lead time | Elapsed time between a version-control commit and that change reaching production. |
| Throughput | Failed deployment recovery time | Recovery duration when a deployment failure demands urgent intervention. |
| Instability | Change fail rate | The share of deployments needing urgent corrective action after release. |
| Instability | Deployment rework rate | The share of deployments made outside the plan in response to a production incident. |
Recovery time belongs under throughput here. Before reusing a dashboard built around the older four metrics, check the metric names and definitions, along with the filters that determine which incidents count.
Keep deployment failures identifiable
The dataset needs to separate recovery from deployment failures from other incident work. A combined average can shift when the mix of incidents changes, even if deployment recovery has not improved.
For each qualifying record, we recommend keeping the application or service, a deployment reference, evidence linking the failure to that deployment, the intervention taken, and the timestamps behind the recovery calculation. Write down the inclusion rule so another engineer can reproduce your selection.
For example, 24Observe is built by our group, so weigh our view accordingly. Its incident pages bring together logs, metrics and the analyst’s verdict. The context graph links operational entities with supporting evidence, while resolution events distinguish a manual closure from one driven by a monitor. You still need a deployment-specific rule for including those records: general MTTR reporting, operational monitoring and security incident analysis each cover more than this DORA measure.
Measure one service and improve within its context
Begin with a specific application or service. DORA advises against combining unlike teams or comparing applications whose operating contexts differ substantially. An average across the organization can hide a service that needs attention.
Review the five measures together with colleagues responsible for development, operations and releases. A shorter recovery time, on its own, does not show whether deployment failures or incident-driven rework are increasing. Track the service’s progress over time and avoid using the dashboard to rank teams.
Smaller change batches can help with recovery. DORA describes reducing batch size as a common improvement approach because smaller changes are easier to recover from when they fail. Try that approach within the chosen service and review its effect across the measures.
DORA Quick Check offers a starting baseline. Pair it with team discussions and retrospectives when assessing progress. An elaborate integration project is unnecessary at the outset: DORA explicitly warns that the initial work needed to obtain highly precise data from multiple systems may not be justified.
Where does time actually go during an incident?
A recovery interval tells you how long impact lasted. A stage-by-stage timeline helps explain why. Record detection, acknowledgment, diagnosis, mitigation and verification using incident evidence rather than assumed durations. Measure the overall interval from impact to verified recovery, and retain the timestamps that establish each stage boundary.
Communication runs alongside these stages, so do not automatically add its duration to the total.
Separate technical work from waiting
Use the table to guide your investigation. The possible blockers will vary by organization. Record waiting intervals separately from active work; stage totals alone cannot tell you how much time went into each.
| Stage | Interval to establish from evidence | Observable waiting time | Possible blocker | Responsible role | Evidence to collect |
|---|---|---|---|---|---|
| Detection | First evidence of impact to alert firing | Impact begins to alert firing | Missing signal or unsuitable threshold | Monitoring owner | First impact evidence; alert timestamp |
| Acknowledgment | Alert delivery to responder accepting ownership | Page delivery to ownership acceptance | Finding the right owner | On-call responder | Delivery, escalation and acknowledgment timestamps |
| Diagnosis | Investigation begins to a mitigation decision | Specialist requested to available; change history requested to retrieved | Specialist dependence; reconstructing recent changes | Operations lead | Queries, hypotheses, change records and requests for assistance |
| Mitigation | Mitigation decision to completion of the selected action | Access or approval requested to granted | Locating credentials; authorization delay | Operations lead and change approver | Access requests, approvals and executed commands |
| Verification | Mitigation completion to evidence confirming recovery | Change completion to sufficient recovery evidence | Unclear recovery criteria | Operations team | User-path checks, error trends and recovery timestamp |
Reading logs, testing a hypothesis and executing a rollback are active technical work. Waiting for access or an available specialist is queue time. Both consume elapsed time and require different interventions. Counting the entire diagnosis interval as hands-on troubleshooting obscures that distinction.
Keep coordination distinct from system changes
Google’s published incident-management model separates responsibilities:
- The incident commander tracks the overall situation and organizes responders.
- The operations lead directs technical response using operational tools. System changes are reserved for the operations team.
- The communications role sends recurring updates to responders and stakeholders.
Separating these responsibilities can reduce conflicting changes and repeated interruptions. The person coordinating the response does not have to run commands while answering every status request.
Google emphasizes a shared, continuously updated document that supports simultaneous editing. Start with a template that puts impact and current priorities first, then record hypotheses, actions and owners. At command handoff, require explicit acceptance before the outgoing commander leaves and tell responders who now leads. Keep alternative communication arrangements available so coordination does not depend entirely on the service being repaired.
24Observe is built by the Networkers Home group, so weigh our view accordingly. If your team uses it, reconstruct the timeline using incident events, logs, metrics, context evidence, analyst findings, action records and published updates. These records support review; you still need to record why someone was waiting.
The GKE review in Google’s SRE Workbook identified inadequate diagnostic logs, reliance on specialists and missing early mitigation options. Each finding points to a specific gap in investigation or readiness that a generic “slow response” label would obscure. Before choosing an intervention, use your own incident evidence to identify the largest delay and establish what responders were waiting for.
How to reduce MTTR: 10 concrete changes to test
Start with the existing incident timeline and find where avoidable waiting accumulates. If responders spend most of an incident finding the owner, faster diagnostics will leave that bottleneck untouched. If acknowledgment is prompt but mitigation stalls, investigate access, decision-making and recovery procedures.
Choose one or two changes, record the relevant stage timings, then compare similar incidents after the change. Keep severity, affected service and recovery criteria visible. Changing every stage simultaneously makes it difficult to identify what helped.
Disclosure: the background links include resources from 24Observe, built by the Networkers Home group, so weigh our view accordingly. For product-specific evaluation, consult its documentation and product information, and verify details relevant to your requirements. The recommendations below are tool-independent.
1. Improve detection around meaningful service symptoms
Choose a critical service and identify observable signs that users cannot complete important tasks. Review recent incidents for missed symptoms, late alerts and noisy conditions. Adjust checks, thresholds and evaluation windows against those examples to shorten the interval between impact and detection. A service-availability monitoring guide provides background, but your failure history should determine what gets tested.
Record impact-to-detection time, including incidents first reported by users. Where impact onset is uncertain, preserve that uncertainty rather than substituting the alert timestamp.
Fewer alerts do not establish better detection. Suppressing noise can hide genuine failures, so check whether the revised conditions would still detect previously missed incidents.
2. Route alerts to an accountable owner
For each service, maintain a primary owner, backup responder and escalation path. Include shared dependencies and clarify who coordinates when several teams are affected. Test delivery and handoffs after ownership or rota changes. Your on-call arrangements should resolve responsibility without requiring someone to search chat history.
Focus on the notification-to-engagement stage, particularly the gap between successful delivery and reaching someone able to act. Track delivery failures, incorrect assignments, escalations caused by unavailable owners and minutes spent locating the correct responder.
A delivery receipt does not prove that responsibility transferred. Require explicit acceptance during handoffs. Google’s guidance similarly calls for confirmation from the incoming commander and notification to the response team.
3. Make acknowledgment easy, but make ownership explicit
Review the responder’s path from receiving an alert to accepting responsibility. Remove unnecessary navigation and provide the service name, severity, incident reference and next step together. Define acknowledgment operationally: the accepting responder owns initial assessment or explicitly transfers that responsibility.
The aim is to reduce acknowledgment delay and attention lost to avoidable interaction steps. Addressing alert fatigue also requires examining whether responders receive actionable information. Track notification-to-acknowledgment time alongside acknowledgments that subsequently require reassignment or escalation.
A quick acknowledgment does not establish recovery. Avoid rewarding immediate acceptance if nobody starts work. Preserve separate timestamps for acknowledgment, responder engagement, mitigation and verified restoration so an easier interaction cannot conceal a slower response.
4. Prepare runbooks and generic mitigations before emergencies
Write a runbook for a recurring failure. Include prerequisites, required access, evidence to preserve, execution steps, rollback conditions, recovery checks and escalation criteria. Ask someone other than its author to follow it, looking for delays between identifying a plausible response and safely applying it.
Google’s incident-response guidance recommends preparing general mitigation mechanisms ahead of emergencies. Its examples include reversing a release and adjusting regional traffic distribution before the full cause is understood.
Measure time until mitigation starts, procedural blockers and attempts that require reversal.
Mitigation can create fresh disruption. Document its possible effects, authorization requirements and stop conditions. Verify customer-facing recovery rather than assuming successful execution means the service is healthy.
5. Correlate symptoms and collect investigation context
Create a shared investigation record containing affected services, known dependencies, recent changes and relevant evidence. Give each hypothesis a supporting observation, an owner and a next test. Keep confirmed facts separate from assumptions so responders can investigate without repeating work.
Sound log-management practices help responders retain evidence, while the distinction between monitoring and observability helps frame questions beyond whether a threshold fired.
Track time spent locating evidence, duplicated investigations and the interval to an evidence-supported mitigation decision.
Simultaneous symptoms do not establish a common cause. Keep contrary evidence visible and periodically reassess the theory. Reducing notification count is insufficient if responders still cannot explain what is affected.
6. Use AI-assisted investigation with human review
Test AI assistance on a bounded task, such as summarizing an incident timeline or proposing hypotheses from approved evidence. Require references to underlying records, distinguish observations from inference and ask what evidence would contradict the proposed explanation.
The aim is to reduce evidence-assembly and hypothesis-testing delays. This recommendation concerns the investigation workflow and makes no claims about a particular product. Compare time to a reviewer-accepted hypothesis, checking effort and the frequency of unsupported or incorrect statements.
Plausible wording is not proof. A reviewer should inspect evidence, consider alternatives and verify timestamps. Keep consequential actions under an explicit authorization policy that identifies permitted changes, approvers, rollback requirements and required recovery checks. Do not grant production authority merely because a generated explanation sounds confident.
7. Clear stale incidents without rewriting recovery history
Classify open incidents by whether they have a trustworthy recovery signal. Where appropriate, define condition-based closure with an observation period and a retained closure reason. For events without a recovery signal, require a person to record the disposition and supporting evidence.
This improves queue hygiene and reduces investigation effort wasted rediscovering already-understood events. It should also protect the integrity of the incident timeline. Track stale-incident age, reopened incidents, closure reasons and the gap between verified restoration and administrative closure.
Closure does not prove remediation. An expired signal, unavailable data source or manually dismissed event must not become an invented recovery timestamp. Preserve distinctions between service restoration, false-positive disposition, duplicate records and unresolved conditions requiring follow-up.
8. Separate stakeholder communication from technical mitigation
Assign a communications owner when update demands begin interrupting technical work. Prepare templates covering confirmed impact, current response, known limitations and the next update time. Maintain an internal record with the most important facts first; publish only information suitable for the intended audience.
To protect mitigation time while maintaining predictable communication, Google’s incident-management model separates responsibilities and assigns recurring stakeholder updates to a communications role. Consider external updates within the wider website-availability workflow.
Measure update cadence, missed commitments and interruptions to technical responders separately from recovery duration.
Avoid publishing sensitive evidence, unverified causes or speculative restoration deadlines. Keep response documents and communications accessible when the affected service is unavailable.
9. Turn post-incident review into assigned work
Reconstruct the timeline from evidence, then identify specific sources of delay: unclear ownership, missing access, repeated tests, unavailable diagnostics or unsafe recovery procedures. Convert each selected improvement into work with an owner, due date and acceptance test.
These changes target future response performance. Keep MTTR measurement tied to the stage the work is intended to improve. Check whether actions were completed and whether later incidents show less delay at that stage. Review recurrence separately.
Restoring service and preventing another failure are different objectives. A workaround may improve recovery without removing the cause. Avoid closing review actions because documentation exists; test whether responders can actually use the changed procedure.
10. Practice response, handoffs and recovery verification
Rehearse a known failure using realistic roles, evidence and escalation paths. Include a shift change, unavailable specialist or ambiguous recovery signal. Require the incoming lead to accept responsibility explicitly, then ask the team to demonstrate restoration against defined checks. The exercise should expose coordination and verification gaps before an emergency.
Google recommends including incident management in disaster-recovery tests and rehearsing previously resolved on-call problems. The Google SRE Workbook describes PagerDuty’s internal weekly Failure Friday drills, designated commanders and time-limited simulations. These describe organizational practices and make no claims about product capabilities.
Record ownership gaps, handoff delays, missed verification steps and procedural blockers.
Exercises cannot fully reproduce a major incident’s urgency. Use findings to select the next one or two changes, then check their effect in subsequent drills and real incidents.
How can a shared NOC and SOC incident workflow reduce delays?
A shared workflow can reduce handoff delays when an operational failure and a security signal concern the same service. Neither team should have to reconstruct the other’s investigation before acting. Both need an agreed owner, access to evidence and clear recovery criteria.
24Observe is built by our group at Networkers Home, so weigh our view accordingly. We use it here to show how a shared workflow works: monitoring, observability and SIEM use the same data store and incident records. This may reduce duplicate investigation and repeated explanations between teams. It does not establish a measured MTTR improvement or guarantee a recovery time.
Follow the incident through each stage
| Stage | Supported capability | Practical use | Limitation |
|---|---|---|---|
| Detection | Failed monitors, breached metric thresholds and matching detections enter a common incident workflow. The metric-alert documentation explains metric conditions and incident creation. | Give NOC and SOC responders a shared starting point, avoiding separate queues that need reconciliation. | Coverage depends on configured checks and rules. A metric series must already be ingested and visible before an alert can monitor it. |
| Routing and acknowledgment | Alerting supports PagerDuty and Opsgenie channels. Incident emails include Ack and Resolve actions. | Route the incident to responders and acknowledge ownership without opening another interface. | Incident creation and notification delivery are separate. An incident can exist even if its notification fails. Acknowledgment does not mean impact has ended. |
| Diagnosis | The context graph connects operational and security entities, adds incident context and groups symptoms that share a root. With organization-level LLM opt-in, the AI analyst investigates and provides reasoning with cited evidence. | Examine affected services, hosts, identities and preceding changes together. Grouping symptoms by root can avoid parallel investigations of downstream effects. | The analyst uses read-only investigation tools. Its recommendation does not prove that a remedy was executed or that the proposed remedy is correct. |
| Mitigation | The starting playbooks cover situations including service outages, credential attacks and resource exhaustion. Event subscriptions can pass work to automation selected by the user. | Agree before an outage which actions require approval and which may be automated. | Recommendations still require execution. Automation depends on the response the user has configured; suggested remedies do not all run automatically. |
| Verification and closure | Monitor recovery can automatically close its incident. Manual closure is also supported. The PagerDuty integration can receive a resolve event following monitor recovery. | Use recovery evidence to confirm the operational condition has cleared, then check whether related security work remains. | Log-alert incidents require an operator to close them. Automatic closure and PagerDuty resolve events do not apply to every incident type. |
| Communication | Status pages let teams group monitors into components and email subscribers when incidents open, change or are resolved. | Publish service-level information while responders continue working from detailed internal evidence. | Teams choose the visible components and wording. Internal logs, topology and analyst findings remain distinct from the information selected for publication. |
| Measurement | Webhook subscriptions provide signed events for opening, acknowledgment and resolution. Resolution payloads distinguish manual closure from closure triggered by monitor recovery. | Build a timeline that exposes delays between incident creation, acknowledgment and closure. | Workflow timestamps do not establish when customer impact actually began or ended. Preserve separate impact evidence. |
Make ownership explicit across NOC and SOC
A shared record helps only when responsibility is clear. Assign one response owner and specify who may change production systems. Designate someone to manage updates. At handoff, require the incoming owner to confirm acceptance and tell the other responders who now leads.
Consider a hypothetical failed service check alongside a suspicious authentication signal. The teams should test whether the signals are connected before treating them as separate emergencies. The graph supplies relationships and supporting context, but proximity alone does not prove that an event caused the outage. Traces feed context into that graph, including service relationships inferred from caller and callee information.
The analyst can offer an initial hypothesis with citations. Responders still need to inspect the evidence and record the proposed action separately from its approval, execution and observed result. Keeping those records distinct prevents a recommendation from being mistaken for a completed fix during a shift change.
Test delivery, mitigation and recovery separately
Before depending on a channel, use the channel-testing runbook. Tests report whether configured channels succeed or fail and include the specific error. If delivery fails while an incident opens, correct the configuration and retrigger the alert. An incident appearing in the queue does not establish that anyone received the page.
Exercise the complete path with NOC and SOC participants, from detection and delivery through acknowledgment, evidence review, an authorized response and verification. Check public messaging separately to keep sensitive investigative detail out of customer updates.
When measuring the workflow, distinguish incident creation from acknowledgment, attempted mitigation, verified restoration and administrative closure. A log alert that requires manual closure may remain open after service restoration. A recovered monitor may say nothing about an unfinished security investigation. Using either closure timestamp as universal recovery evidence can distort MTTR.
For existing Opsgenie users, its alerting and on-call functions are now included in Jira Service Management and Compass. The supplied migration deadline is before 5 April 2027. An automated migration tool is available for existing data and configurations. Include channel and handoff testing in that transition plan.
How do you build an MTTR measurement and improvement plan?
Start with one service and measurement rules its responders can explain. Group records by service, severity and incident type before aggregating them. Use the results to improve that service over time. Avoid ranking teams that work under different operating conditions.
The implementation example below uses 24Observe, built by our group, so weigh our view accordingly.
Use a worksheet that exposes missing evidence
Keep an incident-level register, with a summary for each reporting window.
| Worksheet field | What to record or calculate |
|---|---|
| Scope and eligibility | Service, severity, incident type, accountable owner, inclusion rules and reporting window. Document exclusions. |
| Clock definitions and evidence | Impact start, detection, acknowledgment, verified recovery and administrative closure. Name each timestamp’s source, including supporting logs or metrics, and mark unknowns. |
| Recovery summary | Mean and median recovery duration, eligible incident count and recovered incident count. State the denominator and treatment of missing timestamps. |
| Unresolved work | Open-incident count and each incident’s age at the reporting cutoff. Keep these visible beside completed-incident statistics. |
| Detection and acknowledgment | Detection minus impact start; acknowledgment minus detection. Label any alternative clock explicitly. |
| Tail view | Show individual durations or duration bands for small samples. Add p90 when sample size supports a useful interpretation; always show the count. |
| Delay categories | Time spent awaiting detection, ownership, diagnosis, access, approval, mitigation or recovery verification. Separate measured intervals from estimates. |
| Impact and recurrence | Affected customer journeys, impact scope, repeat-failure links and follow-up owner. Record uncertainty rather than inventing precision. |
| Deployment failures | Flag failures caused by deployments requiring immediate intervention, retaining deployment references and recovery timestamps for separate DORA reporting. |
When a timestamp is unavailable, record it as a data-quality finding. Do not silently substitute ticket creation for impact start, or closure for verified recovery.
Follow an illustrative four-week schedule
This schedule is an editorial recommendation. It carries no product promise and is not an industry benchmark.
| Week | Work and accountable owner |
|---|---|
| One: establish the contract | The service owner and incident lead agree scope, clock boundaries, eligible incidents, timestamp sources and field owners. Define what evidence demonstrates recovery. |
| Two: check the baseline | The reporting owner examines a historical window using those definitions. Audit sample records against telemetry and responder notes, including long-running, reopened and incomplete incidents. |
| Three: test one change | The response lead targets the largest observed delay. For example, test an ownership-routing change if handoffs dominate. Document the hypothesis, owner and review criteria; avoid changing several variables together. |
| Four: review the evidence | Review the duration distribution, unresolved incidents, customer impact, recurrence and data quality. Discuss whether apparent improvement reflects the intervention, different incident types or missing records. Decide whether to retain, revise or continue testing. |
Extend the observation window if incident volume is low, and validate the numbers with responders. A quieter month or a lower mean alone does not establish significance.
Build an event timeline, then verify its meaning
For implementation details, read the incident-event reference alongside the API documentation. 24Observe sends signed webhook events for opening, acknowledgment and resolution. Its response records retain evidence, actions and resolution timing.
Use those events to reconstruct operational activity. You still need evidence of when customers first experienced impact and when service health returned.
In your collector and audit process:
- Deduplicate repeated deliveries using
X-24Observe-Delivery-Id. - Audit manual closure separately from automatic closure, using the resolution payload’s distinguishing flag.
- Preserve reopening history and subsequent recovery intervals.
- Check alert-delivery evidence separately from incident creation.
- Attach independent recovery evidence before accepting closure as the recovery endpoint.
Check the implementation against the documentation and keep your measurement definitions and incident evidence available for review. For practical log-to-disposition work, examine Networkers Home’s Cybersecurity course. Its documented 49-scenario lab block has learners process device logs, perform detection and AI triage, and submit an incident disposition.
MTTR frequently asked questions
What does MTTR stand for?
MTTR can mean mean time to repair, recover, respond or resolve. State which meaning you use before interpreting a result. For service-recovery MTTR, measure from the onset of service impact to verified recovery. Acknowledgment, technical repair and administrative closure mark separate milestones. None should silently replace the recovery interval’s endpoints.
How do you calculate MTTR?
Add the recovery durations for the reporting set, then divide by the incident count. Measure each incident from impact onset to verified service recovery, keeping units consistent. Specify the service and reporting period, along with the inclusion rules. Report ongoing incidents separately. Do not assign them zero duration or omit them from the report. Include the incident count so readers can assess the sample.
What is a good MTTR?
No single recovery target suits every service. Base an improvement goal on the service’s baseline, business impact and recovery constraints. Compare incidents with similar severities and failure types. Inspect the median and slower incidents alongside the mean, and verify recovery with monitoring evidence. A falling average is unconvincing when customer impact worsens or the incident mix changes.
How are MTTR, MTTD and MTTA different?
Each measures a different interval during an incident. Here, MTTD covers impact onset to detection, and MTTA covers detection to acknowledgment. Service-recovery MTTR covers the full interval from impact onset to verified recovery. Adding all three would double-count the earlier stages. If acknowledgment is the bottleneck, examine on-call ownership and handoffs before changing diagnostic tooling.
Is MTTR still a DORA metric?
DORA now uses failed deployment recovery time in place of broad MTTR. The current definition applies to recovery after a failed deployment that requires urgent intervention. It does not cover every operational incident. Keep this deployment-specific measure separate from all-cause service recovery. DORA recommends assessing an individual application or service using several metrics; comparing very different applications can mislead.
Can MTTR improve while reliability gets worse?
Yes. Recovery can get faster while failures become more frequent. MTTR measures recovery duration; MTBF measures the interval between failures. Track incident frequency and cumulative impact alongside recovery duration. Uptime monitoring can inform the availability picture, but recovery averages alone cannot show whether a service is becoming more dependable. Investigate recurring incidents even when recovery is quick.
Must you identify the root cause before stopping the recovery clock?
No. You can verify service recovery before completing the root-cause explanation. Google’s response guidance prioritizes limiting damage and restoring service while retaining evidence for investigation. Follow a documented mitigation procedure and check its effect against the agreed recovery criteria. Continue the root-cause investigation and preventive work separately. Temporary relief does not establish that the fault will stay resolved.
Does an automatically resolved alert prove recovery or close the security case?
No. Alert resolution is separate from verified service recovery, and neither establishes security-case closure. 24Observe is built by our group, so weigh our view accordingly. Its incident emails include actions for acknowledgment and resolution. The PagerDuty integration emits a resolve event after monitor recovery, whereas log-alert incidents need operator closure. The incident.resolved payload differentiates manual closure from closure triggered by monitor recovery.
Treat these events as evidence for their respective milestones. They cannot serve as interchangeable endpoints. End the service-recovery interval once recovery is verified. Keep the security case open until its investigation and disposition criteria are met, even when service has already returned.
Frequently asked questions
What does MTTR stand for?
MTTR can mean mean time to repair, recovery, respond or resolve. These readings measure different parts of incident handling, so the abbreviation alone is insufficient. For service recovery, define when impact starts and what evidence confirms restoration. For response or resolution, state the acknowledgment, action or closure boundary you use. Publish that definition beside the number so readers can interpret it correctly.
What is the MTTR formula?
Add the durations of all qualifying incidents, then divide by the number of those incidents. Use consistent start and finish definitions throughout the calculation. For example, recovery durations of 10, 20, 30, 40 and 200 minutes total 300 minutes across five incidents, producing an MTTR of 60 minutes. Report the incident count and median as well, and disclose missing timestamps or exclusions.
What is a good MTTR?
There is no useful universal target without service context, incident scope and a defined clock. A recovery target should reflect business impact, service commitments and realistic response arrangements. Compare the same service over time, and examine incident frequency and severity alongside recovery duration. DORA also cautions against comparisons between very different applications and recommends using several measures rather than relying on one number.
What is the difference between MTTR and MTBF?
MTTR describes the average duration of a defined repair or recovery process. MTBF, or mean time between failures, describes operating time between failures in a repairable system. They answer different questions: how quickly you recover and how often failure interrupts operation. A service can recover quickly but fail repeatedly, so a low MTTR does not establish strong reliability. Track both with clearly documented measurement rules.
How are MTTD, MTTA and MTTR related?
MTTD measures the interval from failure onset to detection under your stated policy. MTTA measures the interval from a defined alert timestamp to acknowledgment. MTTR measures the recovery or resolution interval you select. These periods can sit within a larger incident timeline. If your recovery clock already starts at failure onset, adding MTTD again would double-count detection time. Always show the boundaries before combining measures.
Does DORA still use MTTR?
DORA's current software delivery model uses failed deployment recovery time in place of MTTR. This measure concerns recovery from a deployment failure requiring immediate intervention, so it has a narrower scope than a general outage average. DORA now describes five delivery measures and recommends applying them to a particular application or service. Keep unrelated infrastructure and security incidents distinguishable when producing deployment-recovery reporting.
How can teams reduce MTTR without rushing root-cause analysis?
Separate immediate impact reduction from permanent corrective work. Prepare mitigations, assign response roles, maintain a working incident record and verify restored service before closing the recovery clock. Google's incident guidance prioritizes stopping damage and recovering service while retaining evidence for later investigation. A rollback or other mitigation may restore operation before the underlying cause is fully understood, but its risks and verification steps still need attention.
Does automatic incident resolution mean the underlying problem is fixed?
No. Automatic closure can indicate that a monitored condition has recovered, while recurrence prevention still requires investigation. In 24Observe, monitor recovery can close an incident automatically, and the resolution event distinguishes automatic from manual closure. Log-alert incidents require manual closure. Teams should verify service health, retain the timeline and record any remaining corrective work rather than treating a closed incident as proof of permanent remediation.