Kubernetes ships with a fixed set of built-in objects: Deployments, Services, ConfigMaps. Most of the time that vocabulary is enough. Then you hit a case where it isn’t — you need to provision a Postgres cluster with a specific failover topology, rotate a certificate on a schedule tied to an external CA, or reconcile state against a third-party API every time someone edits a YAML file. That’s the gap operators fill: they teach the Kubernetes API server a new noun, and teach the cluster a controller that knows how to make that noun real.

This is one of the most over-applied patterns in the ecosystem. Plenty of teams reach for an operator when a Helm chart and a cron job would have done the job in a tenth of the code. Before writing a single line of Go, it’s worth understanding exactly what an operator buys you, what it costs, and where the line sits.

CRDs: Teaching the API Server New Nouns

A Custom Resource Definition (CRD) registers a new resource type with the API server. Once registered, kubectl get and kubectl apply work on it exactly like they do on a Pod.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: postgresclusters.db.furybee.io
spec:
  group: db.furybee.io
  names:
    kind: PostgresCluster
    plural: postgresclusters
    singular: postgrescluster
    shortNames: ["pgc"]
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      storage: true
      subresources:
        status: {}
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["replicas", "version"]
              properties:
                replicas:
                  type: integer
                  minimum: 1
                version:
                  type: string
                storageSize:
                  type: string
                  default: "10Gi"
            status:
              type: object
              properties:
                phase:
                  type: string
                readyReplicas:
                  type: integer
                conditions:
                  type: array
                  items:
                    type: object
                    properties:
                      type:
                        type: string
                      status:
                        type: string
                      lastTransitionTime:
                        type: string

By itself, this CRD does nothing. kubectl apply -f cluster.yaml will happily create a PostgresCluster object and store it in etcd, but no Postgres instance appears anywhere. A CRD without a controller is just structured data sitting inert in the cluster. The controller is the part that actually watches for these objects and does something about them.

The Reconcile Loop

Every controller, whether it manages a built-in resource or a CRD, follows the same shape: watch for changes, compare desired state to observed state, and take action to close the gap. This is the reconcile loop, and it’s deliberately level-triggered rather than edge-triggered.

That distinction matters more than it sounds. An edge-triggered system reacts to events — “a field changed from A to B.” A level-triggered system reacts to state — “the object currently says B; make reality match B.” If your controller crashes mid-update, misses an event because of a network blip, or the cluster falls behind on its watch, level-triggering means the next reconcile call still converges correctly, because it’s not relying on having seen every intermediate event. Kubernetes controllers get called on a resync interval anyway (typically every 10 hours by default, plus on every relevant watch event), specifically to paper over missed events and cumulative drift.

func (r *PostgresClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var cluster dbv1alpha1.PostgresCluster
    if err := r.Get(ctx, req.NamespacedName, &cluster); err != nil {
        // Object was deleted; nothing to reconcile
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Desired state: a StatefulSet matching spec.replicas
    desired := buildStatefulSet(&cluster)

    var existing appsv1.StatefulSet
    err := r.Get(ctx, client.ObjectKeyFromObject(desired), &existing)
    switch {
    case apierrors.IsNotFound(err):
        if err := ctrl.SetControllerReference(&cluster, desired, r.Scheme); err != nil {
            return ctrl.Result{}, err
        }
        if err := r.Create(ctx, desired); err != nil {
            return ctrl.Result{}, err
        }
    case err != nil:
        return ctrl.Result{}, err
    case *existing.Spec.Replicas != *desired.Spec.Replicas:
        existing.Spec.Replicas = desired.Spec.Replicas
        if err := r.Update(ctx, &existing); err != nil {
            return ctrl.Result{}, err
        }
    }

    cluster.Status.ReadyReplicas = existing.Status.ReadyReplicas
    cluster.Status.Phase = phaseFor(&existing)
    if err := r.Status().Update(ctx, &cluster); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

Note the shape: read current state, compute the diff, apply only what’s needed, update status, return. There’s no “on create” or “on update” branching based on the watch event type — the function is called the same way regardless of what triggered it, and it must be safe to call repeatedly with no changes at all.

Idempotency Is Not Optional

The reconcile function above will be called far more often than the object actually changes: on every resync tick, on any change to an owned resource, and any time the controller restarts. If Reconcile isn’t idempotent, you get duplicate resources, thrashing updates, or worse — a controller that fights itself. The rule of thumb: reconcile should always compute “what should exist” from the spec and only touch the API server when the diff is non-zero. Never increment a counter, never append to a list, never assume you’re the only writer.

This is also why controllers use server-side apply or careful Get-then-Update patterns instead of blind Create/Patch calls — two reconcile loops (say, during a rolling restart of the controller itself) racing to create the same StatefulSet will otherwise produce a conflict or, worse, silently clobber each other’s changes.

The Status Subresource

Splitting spec (desired state, written by users) from status (observed state, written by the controller) into separate subresources isn’t cosmetic. It changes RBAC and concurrency semantics: a user with write access to the resource can be denied write access to status, and updates to status don’t bump metadata.generation, so they don’t spuriously re-trigger reconciliation loops that watch generation to detect real spec changes.

status:
  phase: Provisioning
  readyReplicas: 2
  conditions:
    - type: Available
      status: "False"
      reason: WaitingForReplicas
      message: "2/3 replicas ready"
      lastTransitionTime: "2026-07-10T09:14:00Z"

Modeling status as a list of conditions (as the built-in Pod and Deployment types do) rather than a single opaque string gives consumers — humans and other controllers — something machine-readable to poll instead of parsing free text.

Finalizers: Cleaning Up Before Deletion

Deleting a Kubernetes object is normally instantaneous from etcd’s perspective, but sometimes a controller needs to do work before the object actually disappears — deprovisioning a cloud load balancer, revoking a certificate, or draining a connection pool. Finalizers are the mechanism: a string in metadata.finalizers that blocks actual deletion until every finalizer is removed.

const postgresFinalizer = "db.furybee.io/cleanup"

func (r *PostgresClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var cluster dbv1alpha1.PostgresCluster
    if err := r.Get(ctx, req.NamespacedName, &cluster); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    if cluster.DeletionTimestamp != nil {
        if controllerutil.ContainsFinalizer(&cluster, postgresFinalizer) {
            if err := r.deprovisionBackupBucket(ctx, &cluster); err != nil {
                return ctrl.Result{}, err // retry later, finalizer stays
            }
            controllerutil.RemoveFinalizer(&cluster, postgresFinalizer)
            if err := r.Update(ctx, &cluster); err != nil {
                return ctrl.Result{}, err
            }
        }
        return ctrl.Result{}, nil
    }

    if !controllerutil.ContainsFinalizer(&cluster, postgresFinalizer) {
        controllerutil.AddFinalizer(&cluster, postgresFinalizer)
        if err := r.Update(ctx, &cluster); err != nil {
            return ctrl.Result{}, err
        }
    }

    // ... normal reconcile logic below
    return ctrl.Result{}, nil
}

Get this wrong and objects get stuck permanently in Terminating state because a finalizer never clears — a common support headache when the external cleanup call starts failing and nobody notices until someone tries to delete the namespace.

A Minimal kubebuilder Skeleton

kubebuilder scaffolds the boilerplate around the API types, the controller registration, and the RBAC markers that controller-gen turns into manifests.

kubebuilder init --domain furybee.io --repo github.com/furybee/postgres-operator
kubebuilder create api --group db --version v1alpha1 --kind PostgresCluster

That generates api/v1alpha1/postgrescluster_types.go (the Go struct behind the CRD schema) and controllers/postgrescluster_controller.go (the Reconcile skeleton). The controller registers its watches in SetupWithManager:

func (r *PostgresClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&dbv1alpha1.PostgresCluster{}).
        Owns(&appsv1.StatefulSet{}). // re-reconcile if an owned StatefulSet changes
        Complete(r)
}

Owns is what makes the controller react to someone manually editing (or a node evicting) the StatefulSet it created — the owner reference lets the manager map that change back to the parent PostgresCluster and requeue it, closing the loop even for drift the controller didn’t cause itself.

When NOT to Write an Operator

This is the section most operator tutorials skip, and it’s the one that saves the most engineering time.

SignalBetter fit than an operator
Config only changes at deploy time, no ongoing reconciliation neededHelm chart or plain manifests
Logic is “run this script on a schedule”CronJob
You need to react to one-off events, not maintain continuous stateAdmission webhook or a simple controller using client-go without a CRD
The “custom resource” has no distinct lifecycle from a DeploymentJust use a Deployment with a ConfigMap
Team has no Go (or bandwidth to maintain a long-running Go service)Consider Ansible Operator or a shell-script-based Kubernetes Job triggered externally
The state machine is genuinely simple (create once, rarely updated)A one-shot Job or init container

An operator is a long-running, privileged, stateful piece of software that now has to be monitored, upgraded, and debugged alongside everything it manages — and every reconcile bug has blast radius across every instance of the CRD in the cluster. Writing one is the right call when you have genuine ongoing reconciliation work (day-2 operations: backups, failover, scaling policies reacting to external signals) that a human or a cron job would otherwise have to babysit. It’s the wrong call when you’re really just trying to template a Deployment.

Testing Controllers

envtest (part of controller-runtime) spins up a real API server and etcd without kubelet or a scheduler, which is enough to exercise CRUD and reconcile logic without a full cluster:

var _ = Describe("PostgresCluster controller", func() {
    It("creates a StatefulSet matching replicas", func() {
        cluster := &dbv1alpha1.PostgresCluster{
            ObjectMeta: metav1.ObjectMeta{Name: "test-pg", Namespace: "default"},
            Spec:       dbv1alpha1.PostgresClusterSpec{Replicas: 3, Version: "16"},
        }
        Expect(k8sClient.Create(ctx, cluster)).To(Succeed())

        sts := &appsv1.StatefulSet{}
        Eventually(func() error {
            return k8sClient.Get(ctx, types.NamespacedName{Name: "test-pg", Namespace: "default"}, sts)
        }, "10s", "250ms").Should(Succeed())

        Expect(*sts.Spec.Replicas).To(Equal(int32(3)))
    })
})

This is the layer worth investing in before shipping any operator to production — it catches the class of bug where a reconcile function works fine on the happy path but leaves the cluster in a bad state on the second or third pass.

Key Takeaways

  • CRDs are just schema; nothing happens until a controller watches and reconciles them.
  • Reconcile loops must be level-triggered and idempotent — assume you’ll be called far more often than the spec actually changes, including on a periodic resync with no real change at all.
  • Split status into its own subresource so RBAC and generation-tracking work the way built-in resources do.
  • Finalizers block deletion until cleanup finishes; a failing finalizer is the most common cause of objects stuck in Terminating.
  • kubebuilder scaffolds the API types, RBAC markers, and manager wiring; Owns() is what lets your controller react to drift in resources it created, not just its own CRD.
  • Reach for an operator only when there’s genuine ongoing reconciliation work; a CronJob, Helm chart, or admission webhook covers most “I need custom behavior” needs more cheaply.
  • Test with envtest before production — most operator bugs show up on the second reconcile pass, not the first.