Skip to main content

​
How it works

CI deploy MTTR (Mean Time To Recovery) measures how quickly your team recovers from failed production deployments. This is one of the four key DORA metrics for measuring software delivery performance. Lower MTTR indicates:
  • Fast incident response
  • Effective rollback procedures
  • Strong monitoring and alerting
  • Experienced on-call teams

​
Calculation

MTTR is calculated as the average time from the first failed deployment in a failure streak to the next successful deployment of the same workflow and project. Formula:

​
How recovery is measured

  1. A deployment fails in production after the workflow’s last successful run
  2. The timer starts at that failure’s completion timestamp; later failures before a recovery don’t restart it
  3. The timer stops when the next successful deployment completes
  4. This duration is recorded as one recovery time
  5. MTTR is the average of all recovery times in the period

​
What counts as recovery

Recovery is the first successful deployment of the same workflow and project after the failure. Weave reads only the run’s own status; it doesn’t check health checks or whether the environment is back to normal. Both rollbacks and forward fixes count as recovery when they run through the same workflow.

​
Industry benchmarks

Based on DORA research:
MTTR above 24 hours indicates significant gaps in incident response capabilities or deployment automation.

​
Why it matters

Fast recovery time is critical for:
  • Service reliability: Minimize customer impact from deployment issues
  • Business continuity: Reduce revenue loss from downtime
  • Team stress: Lower pressure on on-call engineers
  • Customer trust: Demonstrate reliability and responsiveness
Long recovery times often indicate:
  • Manual deployment processes
  • Inadequate monitoring and alerting
  • Complex rollback procedures
  • Unclear incident response playbooks

​
Recovery strategies

Teams typically use two approaches:

​
1. Rollback (fastest)

Revert to the last known good deployment:
  • Pros: Fast recovery, well-tested code
  • Cons: May require re-deploying the fix later
  • Best for: Critical production issues

​
2. Forward fix (most complete)

Deploy a fix for the issue:
  • Pros: Resolves the root cause immediately
  • Cons: Takes longer, may introduce new issues
  • Best for: Non-critical issues with known fixes
Elite teams can do forward fixes in under an hour because they have:
  • Small, focused deployments
  • Comprehensive automated testing
  • Fast CI/CD pipelines
  • Strong monitoring to quickly identify root causes

​
Improving MTTR

To reduce mean time to recovery:
  1. Automate rollbacks:
    • Implement one-click rollback procedures
    • Use blue-green or canary deployments
    • Maintain deployment version history
  2. Improve monitoring:
    • Set up real-time deployment health monitoring
    • Configure alerts for deployment failures
    • Track key performance indicators (KPIs) during deployments
  3. Streamline incident response:
    • Create clear runbooks for common issues
    • Define on-call rotation and escalation paths
    • Practice incident response through game days
  4. Reduce deployment complexity:
    • Deploy smaller, more frequent changes
    • Minimize deployment dependencies
    • Use feature flags to decouple deployment from release
  5. Learn from incidents:
    • Conduct blameless postmortems
    • Track trends in deployment failures
    • Update processes based on lessons learned

​
Technical details

Recovery time is measured from the completion timestamp of the first failed deployment in a streak to the completion timestamp of the next successful deployment:
Only later runs of the same workflow and project are considered for recovery.
Included:
  • Failed production deployments
  • Subsequent successful runs of the same workflow and project
  • Both rollbacks and forward fixes
Excluded:
  • Non-production deployments (staging, development)
  • Cancelled deployments
  • Runs of other workflows or projects
MTTR is calculated as the mean (average) of all recovery times within your selected time period:
  • Daily view: Average recovery time for failure streaks whose recovering deployment was created that day
  • Weekly view: Average recovery time for failure streaks whose recovering deployment was created that week
  • Monthly view: Average recovery time for failure streaks whose recovering deployment was created that month
If no recovery happened in a time period, MTTR has no value for it (rather than 0). A failure streak that hasn’t recovered yet isn’t counted.
CI deploy MTTR uses data from:
  • GitHub Actions: Workflow run timestamps and conclusions
  • GitLab CI/CD: Pipeline completion times and statuses
  • Azure DevOps: Build and classic release deployment completion times and results
  • CircleCI: Workflow completion times and statuses
  • Jenkins: Build start/end times and results
  • Argo CD: Application sync completion times and results
Recovery time is calculated by correlating failed and successful deployments in temporal order.

​
Public API

Available from GET /metrics as ci_deploy_mttr: average recovery time in seconds from the first failed production deploy in a streak to the next successful deploy of the same workflow and project. Requires a CI/CD connection and production-classified workflows. group_by is not supported; teams and accounts filters still apply.
Last modified on October 17, 2026