Blackbox Monitoring: Synthetic Checks That Catch What Metrics Miss

Every internal dashboard can be green while users see an error page. The application reports healthy, CPU is flat, the error counter is zero, and yet the DNS record points nowhere, the certificate expired, or the CDN is serving a stale redirect. Whitebox metrics describe what a system thinks about itself. Blackbox monitoring asks the question users ask: can I reach it, and does it work?

This article covers what blackbox and synthetic checks are good at, how to run them with the Prometheus blackbox exporter, how to alert on them without drowning in noise, and where they stop being enough.

Whitebox Versus Blackbox

Whitebox monitoring uses instrumentation inside the system: request counters, latency histograms, queue depths, memory use. It is rich and precise, and it explains why something is slow.

Blackbox monitoring probes from the outside, like a user: it sends a request and checks the answer. It knows very little about internals, but it detects symptoms that internal metrics cannot see:

• DNS misconfiguration or propagation problems.
• Expired or misconfigured TLS certificates.
• A reverse proxy or CDN returning errors while the application is healthy.
• A firewall or routing change that blocks traffic.
• A deploy that returns HTTP 200 with an error body.
• A dead process that stopped emitting metrics, so there is no data to alert on.

That last case matters. Absence of data is not the same as a healthy value. A service that crashed emits no error counter. Only an external probe notices that nothing answers. The Google SRE book makes the same point by recommending that symptoms users feel be monitored directly, with causes used mainly for diagnosis. The ideas tie in closely with SLOs, SLIs and error budgets: a probe success ratio is a simple, honest availability SLI.

The Prometheus Blackbox Exporter

The blackbox exporter is a small service that Prometheus asks to probe a target. It supports HTTP and HTTPS, TCP, ICMP, DNS and gRPC probes. Prometheus calls the exporter with the target and a module name, and the exporter returns metrics about that one probe.

Run it next to Prometheus:

services:
  blackbox:
    image: prom/blackbox-exporter:latest
    command: ["--config.file=/etc/blackbox/blackbox.yml"]
    volumes:
      - ./blackbox.yml:/etc/blackbox/blackbox.yml:ro
    restart: unless-stopped

Pin the image tag to a specific release in your own setup rather than using latest. Modules define how a probe behaves:

# blackbox.yml
modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      method: GET
      valid_status_codes: []        # empty means any 2xx
      follow_redirects: true
      fail_if_not_ssl: true
      preferred_ip_protocol: ip4

  http_content:
    prober: http
    timeout: 10s
    http:
      method: GET
      fail_if_body_not_matches_regexp:
        - "Sign in"

  tcp_connect:
    prober: tcp
    timeout: 5s

  dns_a:
    prober: dns
    timeout: 5s
    dns:
      query_name: "example.com"
      query_type: "A"
      valid_rcodes: ["NOERROR"]

Prometheus scrapes it with the relabeling pattern that turns a list of targets into probe parameters:

# prometheus.yml
scrape_configs:
  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://www.example.com/
          - https://api.example.com/healthz
          - https://app.example.org/login
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

The three relabel rules are the part people copy without understanding. The first copies the target into the target URL parameter, the second keeps it as the instance label so graphs show the real URL, and the third rewrites the scrape address to the exporter itself. For larger lists, use file-based service discovery so adding a site is a one-line change to a file:

    file_sd_configs:
      - files: ["/etc/prometheus/targets/http/*.yml"]

The Metrics You Get

Every probe returns a handful of useful series:

Metric	Meaning
probe_success	1 if the probe passed, 0 otherwise. The core signal.
probe_duration_seconds	Total probe time.
probe_http_status_code	Final HTTP status.
probe_http_duration_seconds	Per-phase timing (resolve, connect, tls, processing, transfer).
probe_ssl_earliest_cert_expiry	Unix time of the soonest certificate expiry in the chain.
probe_dns_lookup_time_seconds	Time spent resolving.
probe_http_redirects	Number of redirects followed.

The per-phase timings are valuable. When latency rises, they tell you whether DNS, TCP connect, TLS handshake or the server itself is responsible, without any application instrumentation.

Alert Rules That Behave

A single failed probe is not an outage. Networks drop packets. Alert on sustained failure, and alert on user-facing symptoms.

groups:
  - name: blackbox
    rules:
      - alert: EndpointDown
        expr: probe_success{job="blackbox-http"} == 0
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is failing its probe"

      - alert: EndpointSlow
        expr: probe_duration_seconds{job="blackbox-http"} > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} probe taking over 2 seconds"

      - alert: CertificateExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry{job="blackbox-http"} - time()) / 86400 < 14
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Certificate on {{ $labels.instance }} expires in under 14 days"

      - alert: UnexpectedStatus
        expr: probe_http_status_code{job="blackbox-http"} >= 500
        for: 3m
        labels:
          severity: critical

Design notes:

• Use for: generously. Three consecutive minutes at a one-minute scrape interval means three failed probes. This filters transient blips while still paging within minutes of a real outage.
• Set the scrape interval deliberately. A 15-second interval on hundreds of targets generates real load on the targets and the exporter. One minute is a reasonable default for most sites.
• Avoid slow-threshold alerts that fire constantly. If a threshold is breached every evening, either the threshold is wrong or the service needs work. Alerts that are routinely ignored teach people to ignore the rest. More on writing good rules in Prometheus Alerting Rules.
• Alert on the probe’s own health. If the exporter is down, probe_success series disappear, and == 0 never fires. Add an alert on absent(probe_success) or on the exporter’s up metric.

      - alert: BlackboxExporterDown
        expr: up{job="blackbox-http"} == 0 or absent(probe_success{job="blackbox-http"})
        for: 5m
        labels:
          severity: critical

Where You Probe From Matters

A probe launched from the same server, network or container host as the service tests very little. If the host loses its internet connection, both the service and the monitor are affected, and the alert never leaves the building. Principles:

• Probe the public path. Use the public hostname that users use, through the CDN or proxy, not the internal container address. This is what catches certificate, DNS and proxy problems.
• Probe from outside your failure domain. Run at least one probe location in a different network or provider from the thing being monitored. A small instance elsewhere, or a hosted uptime service, gives you an independent view.
• Keep the watcher independent of the stack. If your monitoring and alert routing run on the same host as production, a host failure silences the very system that should report it. An external heartbeat or a separately hosted checker covers that gap. A “dead man’s switch” pattern, where the monitoring system sends a regular heartbeat to an external service that alerts when it stops, solves the “who watches the watchers” problem.
• Probe from several locations when you serve a global audience, so a regional routing problem is distinguishable from a full outage. Alert when a majority of locations fail rather than any one location.

If your sites sit behind a CDN, probing through it matters, but also consider a second probe that tests the origin directly to separate edge failures from origin failures. Cloudflare in Front of Your Origin discusses that layering.

Checking Content, Not Just Status

Status code 200 is a weak assertion. A misconfigured application can return a 200 login page for every path, or a cached error page. Stronger checks:

• Match the body. fail_if_body_not_matches_regexp verifies the page contains an expected string. Use something stable and specific, not a string that appears in an error page too.
• Check headers. Assert on Content-Type or on the absence of an unexpected redirect target.
• Validate redirects. no_follow_redirects with valid_status_codes: [301] lets you assert that the apex domain redirects to www, or that HTTP redirects to HTTPS, rather than silently following the chain.
• Probe a dedicated health endpoint that exercises critical dependencies for a meaningful “is it up” answer, but keep a separate content check on a real user-facing page.

  redirect_to_https:
    prober: http
    timeout: 10s
    http:
      method: GET
      no_follow_redirects: true
      valid_status_codes: [301, 308]
      fail_if_header_not_matches:
        - header: Location
          regexp: "^https://"

Authenticated and Multi-Step Journeys

The blackbox exporter makes single requests. It can send headers, including an authorization token, and basic authentication, which covers a number of API checks:

  api_authenticated:
    prober: http
    timeout: 10s
    http:
      method: GET
      headers:
        Authorization: "Bearer REPLACE_WITH_PROBE_TOKEN"
      fail_if_body_not_matches_regexp:
        - '"status":"ok"'

Use a dedicated, read-only probe credential with minimal scope, and keep it out of version control. The exporter supports reading secrets from files in recent releases, which avoids embedding them in the module config.

For real user journeys (log in, add to cart, check out), a single request is not enough. That is synthetic monitoring in the stricter sense: a scripted browser or API sequence run on a schedule. Options include browser automation tools such as Playwright wrapped in a scheduled job that pushes a result metric, hosted synthetic services, or the k6 tool running a scripted scenario on a timer. Keep these journeys short and few. They are expensive to maintain, they break when the UI changes, and they use test accounts that must be isolated from real data. Pick the two or three flows that earn money or lose users, and leave the rest to simpler probes.

Other Probe Types Worth Having

• TCP. Verify that a database, SMTP or custom port accepts connections from where clients live. It will not tell you the service is working, only that something is listening.
• DNS. Check that your records resolve to the expected answers from public resolvers. Pair it with expected-answer validation when a wrong answer would be as bad as none.
• ICMP. Useful for network reachability to infrastructure. Many environments block ICMP, and it needs extra container privileges, so prefer TCP or HTTP where possible.
• gRPC. The exporter supports the standard gRPC health checking protocol for services that expose it.

Common Mistakes

• Probing the internal address and never the public path users take.
• Running the only monitor on the same host as the services.
• Alerting on a single failed probe, producing constant false pages.
• Treating HTTP 200 as success when the body is an error page.
• Forgetting to alert when the exporter itself or the whole monitoring stack stops.
• Putting probe credentials with broad permissions into the module config.
• Building a large browser-journey suite that nobody has time to keep green.
• Ignoring certificate expiry, which is cheap to monitor and a frequent cause of outages.

When Blackbox Is Not Enough

Blackbox probes tell you that something is wrong, not why, and they sample rather than measure. A probe every minute can miss a thirty-second failure and cannot report how many real users were affected. It also sees one path, from one place, with synthetic traffic. Combine it with:

• Whitebox metrics for causes, saturation and real-traffic error rates.
• Logs and traces for diagnosis once an alert fires. See distributed tracing.
• Real user monitoring where client-side experience matters, since synthetic checks cannot reproduce every device, browser and network.

The two approaches complement each other. Whitebox tells you why, blackbox tells you whether users are affected right now.

Key Takeaways

• Blackbox monitoring tests what users experience and catches DNS, TLS, proxy and routing failures that internal metrics cannot see.
• The Prometheus blackbox exporter turns a list of targets into probe_success, timing and certificate-expiry metrics with a standard relabeling pattern.
• Alert on sustained failure with a for: window, and add alerts for the exporter’s own absence so silence never looks like health.
• Probe the public path from outside your failure domain, and keep at least one watcher independent of the stack it monitors.
• Assert on content, redirects and headers rather than status code alone, and monitor certificate expiry on every public host.
• Use a small number of scripted journeys for critical flows only, and pair blackbox checks with whitebox metrics for diagnosis.