A certificate that renews itself for two years is the most dangerous kind. Nobody remembers how it works, nobody watches it, and when the renewal job quietly stops running, the first alert is a customer reporting a browser warning. The failure is rarely dramatic. A scheduler stops ticking, a DNS API token gets rotated, a rate limit is hit, and the logs say nothing because nothing was attempted.

ACME automation is a solved problem at the protocol level. What remains unsolved in most setups is observability. This article covers how the protocol works, which challenge type to choose, how to configure the common clients, and how to detect the failures that never produce an error message.

How ACME Works in Practice

ACME (RFC 8555) is the protocol Let’s Encrypt and other public certificate authorities use to issue certificates without a human. The flow is short:

  1. The client registers an account key with the CA.
  2. The client places an order for one or more identifiers (hostnames).
  3. For each identifier the CA hands out a challenge. The client must prove control of the name.
  4. Once every challenge validates, the client submits a certificate signing request (CSR) and downloads the signed certificate.

The account key identifies you to the CA. The certificate key is separate and is generated per certificate (or reused, depending on client settings). Losing the account key is annoying but recoverable. Losing the certificate private key means reissuing and redeploying.

Certificates from Let’s Encrypt have short lifetimes by design, and clients are expected to renew well before expiry, typically when roughly a third of the lifetime remains. Short lifetimes limit the damage of a leaked key and force you to keep automation honest. They also mean a broken renewal pipeline becomes an outage within weeks rather than years.

Choosing a Challenge Type

There are three challenge types you will meet.

HTTP-01. The CA fetches a token from http://<name>/.well-known/acme-challenge/<token> on port 80. It is the simplest option and needs no DNS credentials. It cannot issue wildcard certificates, requires port 80 reachable from the public internet, and validates only what that one server answers.

TLS-ALPN-01. Validation happens on port 443 using a special ALPN protocol. It is useful when port 80 is blocked, and some clients support it natively. It also cannot issue wildcards.

DNS-01. The client publishes a TXT record at _acme-challenge.<name> and the CA looks it up. It supports wildcard certificates, works for hosts that are not reachable from the internet, and does not care about your HTTP routing. The cost is that your automation needs credentials that can modify DNS.

A rough decision rule:

SituationPick
Single public site, port 80 open, no wildcardHTTP-01
Wildcard certificate neededDNS-01
Internal service, no public ingressDNS-01
Site behind a CDN or proxy that intercepts requestsDNS-01 (or confirm the proxy passes the challenge path)
Port 80 blocked, port 443 openTLS-ALPN-01 or DNS-01

If your origin sits behind a proxy such as Cloudflare, HTTP-01 can break in subtle ways when redirects or access rules intercept the challenge path. DNS-01 sidesteps that, which is why it pairs well with the setup described in Cloudflare in Front of Your Origin.

The Security Cost of DNS-01

DNS-01 trades network requirements for a powerful credential. An API token that can edit TXT records on your zone can also, depending on scope, repoint your A records. Treat it accordingly.

  • Scope the token narrowly. Most DNS providers support tokens limited to a single zone and to DNS edit permissions only. Do not use a global account key.
  • Delegate the challenge with a CNAME. Create _acme-challenge.example.com as a CNAME to a record in a separate zone you dedicate to validation (for example _acme-challenge.acme.example.org). The ACME client then only needs write access to that small zone, and a compromise cannot touch your production records. Several clients support this pattern directly.
  • Keep the token out of images and repositories. Pass it as a file-based secret or an environment variable injected at runtime.
  • Rotate with a plan. A rotated token that is not updated in the client is one of the classic silent failures, covered below.

Configuration Examples

Traefik with DNS-01

Traefik has a built-in ACME client. A static configuration with a DNS challenge looks like this:

# traefik.yml
entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

certificatesResolvers:
  dns:
    acme:
      email: [email protected]
      storage: /letsencrypt/acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"

The provider reads its credentials from environment variables (for Cloudflare, an API token variable). A service opts in with labels:

services:
  app:
    image: example/app:1.0
    labels:
      - "traefik.http.routers.app.rule=Host(`app.example.com`)"
      - "traefik.http.routers.app.entrypoints=websecure"
      - "traefik.http.routers.app.tls.certresolver=dns"

Make sure acme.json lives on a persistent volume with mode 600, otherwise Traefik refuses to use it, and a container recreation without the volume means a fresh set of issuances. Reissuing many certificates at once is the quickest way to run into CA rate limits. For a broader comparison of how proxies handle this, see Traefik vs Nginx vs Caddy.

Caddy

Caddy obtains and renews certificates by default with HTTP-01 or TLS-ALPN-01. For a wildcard you enable a DNS provider module (compiled into the binary with xcaddy):

*.example.com {
	tls {
		dns cloudflare {env.CF_API_TOKEN}
	}

	@app host app.example.com
	handle @app {
		reverse_proxy app:8080
	}

	handle {
		abort
	}
}

certbot with a DNS plugin

For servers that are not running an ACME-aware proxy, certbot with a DNS plugin works well:

certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
  --dns-cloudflare-propagation-seconds 30 \
  -d example.com -d '*.example.com' \
  --deploy-hook "systemctl reload nginx"

The credentials file must be readable only by root (chmod 600). The --deploy-hook is important: certbot renews files on disk, but a long-running daemon keeps serving the old certificate until it reloads. A renewed file that was never loaded is another silent failure.

Always test the whole path with a dry run:

certbot renew --dry-run

A dry run uses the CA’s staging environment and exercises authentication, the hook and the plugin. Put it in your runbook and run it after any change to credentials or DNS.

Rate Limits and the Staging Environment

Public CAs enforce rate limits to protect their infrastructure. The limits you are most likely to hit are on failed validations per account and per hostname, on duplicate certificates for the same set of names, and on the number of new orders in a time window. The exact numbers change over time, so read the CA’s current documentation rather than relying on a remembered figure.

Practical consequences:

  • Develop against staging. Both Let’s Encrypt and most other CAs provide a staging directory with far looser limits. Staging certificates are not trusted by browsers, which is the point.
  • Do not loop on failures. A misconfigured client retrying every minute can trip failed-validation limits and lock you out of issuance for a while, exactly when you need it.
  • Persist state. Containers that lose their certificate storage on every redeploy will reissue on every deploy.
  • Prefer one certificate with several names over many single-name certificates when the names share a lifecycle, but remember that one broken name then blocks the whole order.

The Failures Nobody Monitors

Most ACME outages share a pattern: the issuance path worked at setup time, then something changed and the failure produced no alert. The usual suspects:

  1. The renewal scheduler stops. The process is up and serving traffic, but its internal renewal loop is no longer running. There is no error because nothing is being attempted. Certificates then expire on schedule. Long-running proxies are particularly prone to this after restarts that left state in an odd condition, and the symptom is a proxy or CDN returning errors about an expired origin certificate.
  2. The DNS credential changed. The token was rotated, expired, or lost permission on the zone. The client logs an authentication error, but only when it tries, and it only tries near renewal time. That can be weeks after the change.
  3. The deploy hook fails or is missing. Files are renewed but the server never reloads them.
  4. DNS propagation is too slow. The CA queries the authoritative servers before your provider has published the TXT record. Increase the propagation wait or point the client’s resolvers at the authoritative servers.
  5. A CAA record blocks the CA. A CAA record that does not list the CA you use causes issuance to fail. This tends to appear after someone tightens DNS policy.
  6. Account or storage corruption. A truncated or unwritable storage file leaves the client unable to read its state.
  7. Rate limiting after a mass reissue. A redeploy without persistent storage requests everything again.

None of these will page you through application health checks, because the site keeps working with the certificate it already has until the day it does not.

Monitoring Certificate Expiry

The correct signal is the one users experience: the certificate actually presented on the wire. Measure that, not the files on disk and not the client’s logs.

With the Prometheus blackbox exporter, probe each hostname over TLS and alert on remaining validity:

# prometheus rule
groups:
  - name: certificates
    rules:
      - alert: CertificateExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Certificate for {{ $labels.instance }} expires in under 14 days"

      - alert: CertificateExpiryCritical
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 5
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Certificate for {{ $labels.instance }} expires in under 5 days"

Pick the warning threshold so that it fires well before the client would normally renew. If the client renews with a third of the lifetime remaining, a certificate that is still unrenewed far past that point means automation is broken, and you have weeks to fix it calmly. Alert rule design is covered in Prometheus Alerting Rules.

Two additional checks are worth having:

  • Probe through every layer. If a CDN terminates TLS for visitors and your origin has its own certificate, probe both. A valid edge certificate hides an expired origin certificate until the CDN’s SSL mode requires validation.
  • Check the origin directly. For a host you can reach, this one-liner prints validity dates straight from the server:
echo | openssl s_client -servername app.example.com -connect 203.0.113.10:443 2>/dev/null \
  | openssl x509 -noout -dates

Also consider watching Certificate Transparency logs for your domains. Unexpected issuances there are a security signal, separate from the expiry question.

Hardening the Pipeline

  • Test the renewal path on a schedule, not only at setup. A weekly dry run in CI or cron catches rotated credentials early.
  • Keep an emergency manual path. Document the exact command to force an issuance, and know which volume or directory holds the state.
  • Stagger and persist. Keep certificate storage on durable volumes and include it in backups. Losing it is recoverable, but reissuing under pressure risks rate limits.
  • Alert on the absence of attempts, where your client exposes metrics. Some clients and exporters report last renewal time, which lets you detect a scheduler that stopped.
  • Use a second CA as a fallback if your client supports it, so a CA-side outage or a rate limit does not leave you without a path to a valid certificate.

When Not to Use DNS-01

DNS-01 is not always the right tool. If you have one public site, port 80 reachable, and no wildcard requirement, HTTP-01 has fewer moving parts and no high-privilege token to protect. If your DNS provider has no API or a slow, unreliable one, propagation delays will cause intermittent failures, and TLS-ALPN-01 or HTTP-01 may serve you better. And if a platform already manages certificates for you (a managed load balancer, a CDN edge certificate), adding your own ACME client for the same hostname adds complexity without benefit.

Key Takeaways

  • Choose DNS-01 for wildcards and for hosts that are not publicly reachable; choose HTTP-01 when a single public site is all you need.
  • Scope DNS credentials to one zone, and consider delegating _acme-challenge by CNAME to a dedicated validation zone.
  • Develop against the staging CA, persist certificate storage, and never retry failed issuances in a tight loop.
  • The dangerous failures are silent: a stopped scheduler, a rotated token, a missing reload hook. None of them log anything until renewal time.
  • Monitor the certificate presented on the wire, at every layer, and alert well before the client’s normal renewal window closes.
  • Run a renewal dry run on a schedule so credential and DNS changes are caught while you still have weeks of validity left.