Most teams monitor infrastructure metrics: CPU, memory, disk, latency. These are necessary but not sufficient. What matters is whether users can actually use your service. An SLI (Service Level Indicator) measures that. An SLO (Service Level Objective) sets a target for the SLI. An error budget is what you get when you miss the SLO — time you can spend shipping new features or taking risks with deploys before violating your commitment to users.

The hard part isn’t understanding these concepts; it’s choosing the right SLIs, setting SLOs that balance reliability with velocity, and then actually using the error budget to make faster release decisions. This guide walks you through concrete examples, shows you the PromQL for multi-window burn-rate alerting (which prevents alert fatigue), and explains how error budgets should change your team’s behavior.

SLI: What Matters to Users

An SLI is a metric that measures user-facing behavior. It’s not “CPU usage” or “memory”; it’s “what percentage of requests succeeded” or “what percentage of page loads finished in under 3 seconds.”

Good SLIs are:

  • User-centric: measure what users experience, not internal infrastructure
  • Actionable: when the SLI degrades, you know what to do
  • Normalized: percentages or rates, not absolute counts (so dashboards are comparable across services)

Common SLIs:

Request success rate (for APIs):

sum(rate(http_requests_total{status=~"2.."}[5m]))
/
sum(rate(http_requests_total[5m]))

What it measures: what fraction of API calls succeeded? Failures include 4xx (client errors, don’t count) and 5xx (server errors, do count). A typical target is 99.9% success.

Request latency (for user experience):

histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

What it measures: 99% of requests complete in how many seconds? A typical target is 1 second or less. Note: this is the 99th percentile, not the average. Users experience the slowest requests, not the average.

Error budget: if your SLO is 99.9% success, you have a 0.1% error budget per day. On a service with 10 million requests daily, you can afford 10,000 failures. Spend it on a risky deploy, or save it for reliability.

Page load time (for web services):

histogram_quantile(0.95, sum(rate(page_load_time_seconds_bucket[5m])) by (le))

What it measures: 95% of page loads finish in how many seconds? Target: 3 seconds.

Availability (for long-running services):

(
  SAMPLE_COUNT(up{service="myapp"} == 1)
  /
  SAMPLE_COUNT(up{service="myapp"})
) * 100

What it measures: what percentage of time was the service reachable? Target: 99.95%.

Freshness (for data pipelines, caches):

(
  SAMPLE_COUNT(data_age_seconds < 3600)
  /
  SAMPLE_COUNT(data_age_seconds)
) * 100

What it measures: what percentage of data is fresh (less than 1 hour old)? Target: 99%.

Choose 1-3 SLIs per service. More than that, and you lose focus. For most APIs, success rate and P99 latency are enough.

SLO: Setting a Target You Can Defend

An SLO is your promise to users: “99.9% of requests will succeed” or “95% of page loads will finish in under 3 seconds.”

The goal is to balance reliability with velocity. A 99.99% SLO is safer but requires more investment in reliability. A 95% SLO is easier to achieve but risks user frustration.

Industry benchmarks:

  • Consumer services (social media, news): 99.9%
  • Financial services (banking, trading): 99.95% to 99.99%
  • Internal tools: 95% to 99%
  • Non-critical infrastructure: 90%

For a small team running on a single VPS, 99.9% is reasonable. You’re not committed to zero downtime, but you’re committed to high availability most of the time.

Error Budget: Spend It Wisely

If your SLO is 99.9%, you have a 0.1% error budget. Over 30 days:

30 days = 43,200 minutes
0.1% of 43,200 = 43.2 minutes

So you can afford 43 minutes of downtime (or equivalent error rate) per month.

With a 10-request-per-second service:

86,400 seconds/day * 10 RPS = 864,000 requests/day
0.1% of 864,000 = 864 failures/day

You can afford 864 failures per day. Spend this budget on:

  • Risky deploys (new algorithm, database migration)
  • Aggressive performance optimization (remove safety checks, batch requests aggressively)
  • Feature experiments (A/B tests that might break for some users)

When the error budget is exhausted, freeze risky deploys until the window rolls over (monthly or quarterly).

Tracking Error Budget in Prometheus

# Error budget (as a percentage)
100 * (
  1 -
  (
    sum(rate(http_requests_total{status=~"5.."}[30d]))
    /
    sum(rate(http_requests_total[30d]))
  )
)

This tracks your error budget over the last 30 days. If it’s below 0.1%, you’re in the red.

Create a Grafana gauge panel:

  • Query: the above PromQL
  • Thresholds: Green (> 0.05%), Yellow (0.01% - 0.05%), Red (< 0.01%)

When the gauge is red, it’s a signal to the team: no risky deploys this month.

Multi-Window Multi-Burn-Rate Alerting

Naive alerting on SLI violations causes alert fatigue. If your SLO is 99.9%, and you alert on any failure, you’ll be paged constantly. The solution is multi-burn-rate alerting.

A burn rate is how fast you’re consuming your error budget. A 10x burn rate means you’re violating your SLO 10 times faster than allowed.

The math:

SLO = 99.9% (0.1% error budget)
Error budget per second = 0.1% / (30 days * 86,400 sec) = 0.0000000039
10x burn rate = 0.000000039 per second = 0.00000004 per 5 minutes

This is abstract. The key insight: multiple windows at different speeds catch real problems while avoiding false positives.

Fast-burn window (short, high threshold): catch immediate failures (outage happening now) Slow-burn window (long, low threshold): catch chronic issues (things are slowly degrading)

# Fast burn: 30 minutes window, 6x burn rate
# If you're burning budget 6x faster than allowed, page immediately
alert: SLOBurnRateFast
expr: |
  (
    sum(rate(http_requests_total{status=~"5.."}[5m]))
    /
    sum(rate(http_requests_total[5m]))
  ) > (6 * (1 - 0.999) / (30*60))
for: 5m

# Slow burn: 6 hour window, 3x burn rate
# If you're burning budget 3x for hours, page someone
alert: SLOBurnRateSlow
expr: |
  (
    sum(rate(http_requests_total{status=~"5.."}[6h]))
    /
    sum(rate(http_requests_total[6h]))
  ) > (3 * (1 - 0.999) / (30*24*60*60))
for: 15m

Practical thresholds:

  • 1-hour window, 10x burn: something is very wrong right now (page immediately)
  • 6-hour window, 3x burn: sustained issues over hours (page, but not urgent)

Firing both alerts together means: ongoing issue, page someone to investigate. Firing only the fast alert means: brief spike, probably transient. Firing only the slow alert means: slow degradation, schedule an incident review.

SLO Calculation

For a service with multiple components, you can compose SLOs:

SLO(API) = 99.9%
SLO(Database) = 99.95%
SLO(Cache) = 99%

Composite SLO = 99.9% * 99.95% * 99% = 98.85%

This is pessimistic: if any component fails, the service fails. In practice, add redundancy to critical components.

When to Adjust Your SLO

Review your SLO quarterly:

  • Consistently exceeding it? Increase it (users expect higher reliability)
  • Constantly violating it? Decrease it (be honest about what you can sustain) or invest in reliability
  • New service? Start at 95%, increase as you learn how to run it

Document why you chose your SLO. It’s a contract with your users and your team.

Example: All-In-One Prometheus Config

# prometheus-slo.yml
groups:
  - name: api_slo
    rules:
      # SLI: request success rate
      - record: api:http_success_rate:5m
        expr: |
          sum(rate(http_requests_total{status=~"2.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service)

      # SLI: request latency (p99)
      - record: api:http_latency_p99:5m
        expr: |
          histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (service, le))

      # Error rate (inverse of success)
      - record: api:http_error_rate:5m
        expr: |
          1 - (api:http_success_rate:5m)

      # Fast burn (1h window, 10x burn)
      - alert: SLOBurnRateFast
        expr: |
          api:http_error_rate:5m > (10 * 0.001 / (3600))
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.service }} SLO burning too fast"
          description: "Error rate {{ $value | humanizePercentage }} exceeds 10x SLO threshold"

      # Slow burn (6h window, 3x burn)
      - alert: SLOBurnRateSlow
        expr: |
          api:http_error_rate:5m > (3 * 0.001 / (86400*30))
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.service }} SLO burning over 6h window"
          description: "Sustained error rate detected"

      # Error budget exhaustion
      - alert: ErrorBudgetExhausted
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[30d])) by (service)
            /
            sum(rate(http_requests_total[30d])) by (service)
          ) > 0.001
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.service }} error budget exhausted for month"

How Error Budgets Drive Decisions

With error budget visibility:

High budget (> 50% remaining):

  • Ship that risky database migration next week
  • Deploy without extra testing on Wednesday

Medium budget (20-50%):

  • Deploy new features, but defer experiments
  • Defer any breaking API changes

Low budget (< 20%):

  • Only bug fixes and security patches
  • Defer performance optimization (might introduce regressions)
  • Pause experiments

Exhausted:

  • Emergency reliability sprint
  • No new features until budget is rebuilt

This prevents the common failure mode where teams ship constantly and wonder why they’re always firefighting reliability issues.

Key Takeaways

  1. SLIs measure user experience — success rate, latency percentiles, availability, not CPU or memory
  2. SLOs are targets, not guarantees — 99.9% means you can afford 43 minutes of downtime per month
  3. Error budget is real — spend it on risky deployments, pause risky changes when exhausted
  4. Multi-window alerts prevent fatigue — fast-burn (immediate problems) and slow-burn (chronic issues) together
  5. Pick 1-3 SLIs per service — success rate and P99 latency are usually enough
  6. SLO should reflect business needs — not just “5 nines”, but what users actually expect
  7. Review SLOs quarterly — increase if consistently exceeded, decrease if unachievable
  8. Track error budget in dashboards — visibility changes behavior
  9. Compose SLOs for multi-tier systems — but add redundancy to critical components
  10. Error budgets replace “go faster / be safer” arguments — math-based decisions instead of gut feel