> ## Documentation Index
> Fetch the complete documentation index at: https://docs.weaveos.com/llms.txt
> Use this file to discover all available pages before exploring further.

# CI deploy MTTR

> Mean time to recovery - average time from a failed deployment to the next successful one

## How it works

CI deploy MTTR (Mean Time To Recovery) measures how quickly your team recovers from failed production deployments. This is one of the four key [DORA metrics](https://dora.dev/guides/) for measuring software delivery performance.

Lower MTTR indicates:

* Fast incident response
* Effective rollback procedures
* Strong monitoring and alerting
* Experienced on-call teams

## Calculation

MTTR is calculated as the average time from the first failed deployment in a failure streak to the next successful deployment of the same workflow and project.

**Formula:**

```
MTTR = Average(Time from the first failed deployment in a streak to the next successful deployment)
```

### How recovery is measured

1. A deployment fails in production after the workflow's last successful run
2. The timer starts at that failure's completion timestamp; later failures before a recovery don't restart it
3. The timer stops when the next successful deployment completes
4. This duration is recorded as one recovery time
5. MTTR is the average of all recovery times in the period

### What counts as recovery

Recovery is the first successful deployment of the same workflow and project after the failure. Weave reads only the run's own status; it doesn't check health checks or whether the environment is back to normal.

Both rollbacks and forward fixes count as recovery when they run through the same workflow.

## Industry benchmarks

Based on DORA research:

| Performance Level | MTTR |
| - | - |
| **Elite** | Less than 1 hour |
| **High** | Less than 1 day |
| **Medium** | 1 day to 1 week |
| **Low** | More than 1 week |

<Warning>
  MTTR above 24 hours indicates significant gaps in incident response capabilities or deployment automation.
</Warning>

## Why it matters

Fast recovery time is critical for:

* **Service reliability:** Minimize customer impact from deployment issues
* **Business continuity:** Reduce revenue loss from downtime
* **Team stress:** Lower pressure on on-call engineers
* **Customer trust:** Demonstrate reliability and responsiveness

Long recovery times often indicate:

* Manual deployment processes
* Inadequate monitoring and alerting
* Complex rollback procedures
* Unclear incident response playbooks

## Recovery strategies

Teams typically use two approaches:

### 1. Rollback (fastest)

Revert to the last known good deployment:

* **Pros:** Fast recovery, well-tested code
* **Cons:** May require re-deploying the fix later
* **Best for:** Critical production issues

### 2. Forward fix (most complete)

Deploy a fix for the issue:

* **Pros:** Resolves the root cause immediately
* **Cons:** Takes longer, may introduce new issues
* **Best for:** Non-critical issues with known fixes

<Note>
  Elite teams can do forward fixes in under an hour because they have:

  * Small, focused deployments
  * Comprehensive automated testing
  * Fast CI/CD pipelines
  * Strong monitoring to quickly identify root causes
</Note>

## Improving MTTR

To reduce mean time to recovery:

1. **Automate rollbacks:**
   * Implement one-click rollback procedures
   * Use blue-green or canary deployments
   * Maintain deployment version history

2. **Improve monitoring:**
   * Set up real-time deployment health monitoring
   * Configure alerts for deployment failures
   * Track key performance indicators (KPIs) during deployments

3. **Streamline incident response:**
   * Create clear runbooks for common issues
   * Define on-call rotation and escalation paths
   * Practice incident response through game days

4. **Reduce deployment complexity:**
   * Deploy smaller, more frequent changes
   * Minimize deployment dependencies
   * Use feature flags to decouple deployment from release

5. **Learn from incidents:**
   * Conduct blameless postmortems
   * Track trends in deployment failures
   * Update processes based on lessons learned

## Technical details

<AccordionGroup>
  <Accordion icon="stopwatch" title="How is recovery time calculated?">
    Recovery time is measured from the **completion timestamp** of the first failed deployment in a streak to the **completion timestamp** of the next successful deployment:

    ```
    Failed Deployment (2024-01-15 10:30:00 UTC)
    ↓
    [Recovery Period]
    ↓
    Successful Deployment (2024-01-15 11:45:00 UTC)

    Recovery Time = 1 hour 15 minutes
    ```

    Only later runs of the same workflow and project are considered for recovery.
  </Accordion>

  <Accordion icon="filter" title="What deployments are included?">
    **Included:**

    * Failed production deployments
    * Subsequent successful runs of the same workflow and project
    * Both rollbacks and forward fixes

    **Excluded:**

    * Non-production deployments (staging, development)
    * Cancelled deployments
    * Runs of other workflows or projects
  </Accordion>

  <Accordion icon="calendar" title="How is MTTR aggregated over time?">
    MTTR is calculated as the **mean** (average) of all recovery times within your selected time period:

    * **Daily view:** Average recovery time for failure streaks whose recovering deployment was created that day
    * **Weekly view:** Average recovery time for failure streaks whose recovering deployment was created that week
    * **Monthly view:** Average recovery time for failure streaks whose recovering deployment was created that month

    If no recovery happened in a time period, MTTR has no value for it (rather than 0). A failure streak that hasn't recovered yet isn't counted.
  </Accordion>

  <Accordion icon="chart-line" title="What data sources are used?">
    CI deploy MTTR uses data from:

    * **GitHub Actions:** Workflow run timestamps and conclusions
    * **GitLab CI/CD:** Pipeline completion times and statuses
    * **Azure DevOps:** Build and classic release deployment completion times and results
    * **CircleCI:** Workflow completion times and statuses
    * **Jenkins:** Build start/end times and results
    * **Argo CD:** Application sync completion times and results

    Recovery time is calculated by correlating failed and successful deployments in temporal order.
  </Accordion>
</AccordionGroup>

## Public API

Available from [`GET /metrics`](/api-reference/metrics/get-metrics) as `ci_deploy_mttr`: average recovery time in **seconds** from the first failed production deploy in a streak to the next successful deploy of the same workflow and project. Requires a CI/CD connection and production-classified workflows. `group_by` is not supported; `teams` and `accounts` filters still apply.

## Related metrics

* **[CI deploy frequency](/metrics/ci-deploy-frequency):** How often you deploy
* **[CI deploy success rate](/metrics/ci-deploy-success-rate):** Percentage of successful deployments
* **[PR deploy lead time](/metrics/pr-deploy-lead-time):** Time from code merge to production


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.