Felix Seifert

Article

Kubernetes Operators: The Reconciliation Loop

7 min read Felix Seifert

This is part 3 of a four-part series on building a Kubernetes operator: CRDs , Custom Resources , Operators (this post) and Kubebuilder .

Pattern and Not Kubernetes Built-In

An “operator” is not a Kubernetes resource type or an API, I would consider it an architectural pattern: It comprises of a Custom Resource Definition (CRD) that describes the desired state (covered in part one), and a controller that watches Custom Resource (CR) instances (covered in part two) and drives them towards this desired state. Something can be considered an operator rather than “just a controller” when it encodes real operational knowledge, which would otherwise live in a runbook, and runs continuously instead of being triggered by a human. You can think of a Kubernetes operator as the software representation of a human operator.

This post is specifically about the reconciliation loop that continuously tries to move the CRs’ actual state to the desired state; the post is about the reconciliaton loop’s structure. The post after this covers the tool kubebuilder , which scaffolds all of this for you. Even though knowing this post’s content about the reconciliation loop might not be strictly necessary, it is worth understanding what some automated tooling actually scaffolds.

Level-Triggered, Not Edge-Triggered

Operators have at least one reconciler, like in our example the BackupCheckReconciler. The most important thing to internalise about a reconciler: It is told that something changed, never what changed. A Reconcile call receives a namespace/name, not a diff. The reconciler’s job is to look at the current state, compare it to the desired state, and act on it, every single time, regardless of what triggered the call.

This is a deliberate trade-off:

  • An edge-triggered system reacts to specific diffs. It is generally more efficient, but fragile, because if it misses one event due to the controller being down, state can drift forever without anyone reacting to it.
  • Level-triggered reconciliation just know that something has changed and figures out the current diff itself. It gives up some of that efficiency in exchange for a strong guarantee: any reconcile can be repeated, delayed, run against stale information, or run twice in a row for the same change; eventually, it should still converge to the right answer. That guarantee is only true if your reconcile logic is idempotent, which is a central discipline of writing the reconcile logic well.

Informers: Watches and Local Cache Updates

A naive controller would call the API server directly inside every reconcile. In practice however, the controller-runtime (and every serious controller framework) interposes a cache between controller and API server: An informer does an initial List and then keeps a Watch open to update an in-memory store when changes arrive. The reconcile logic does fast reads from this cache without hammering the API server, while writes still go straight through to the real API.

Multiple controllers registered with the same manager share the same informers per object type: Watching BackupCheck objects from two different controllers does not mean that we have two separate watch connections.

On top of a watch, there is also a periodic full resync, a deliberate re-list of everything. This is specifically helpful as a backstop against missed or dropped watch events. This shows that any reconcile might be redundant, and that is both in theory and practice fine.

Events Calling Reconcile Indirectly Through Workqueue

Event handlers of an nformer (object added, updated, deleted) do not invoke Reconcile directly themselves. They instead enqueue a key (namespace/name) onto a workqueue. A pool of worker goroutines then pulls keys off it and calls Reconcile for each.

That indirection buys three things:

  1. Deduplication: Five rapid updates to the same object collapse into one queued key. This is okay since the queue only needs to that this object needs attention and not what changed.
  2. Rate-limited retry with backoff when a reconcile returns an error.
  3. Clean separation between how fast events arrive and how fast you can safely process them.

Anatomy of Reconcile(ctx, req)

The shape is purposefully small:

 1func (r *BackupCheckReconciler) Reconcile(
 2    ctx context.Context, req ctrl.Request,
 3) (ctrl.Result, error) {
 4    var check backupsv1alpha1.BackupCheck
 5    if err := r.Get(ctx, req.NamespacedName, &check); err != nil {
 6        return ctrl.Result{}, client.IgnoreNotFound(err)
 7    }
 8
 9    if !check.DeletionTimestamp.IsZero() {
10        return r.reconcileDelete(ctx, &check)
11    }
12
13    if err := r.ensureRestoreJob(ctx, &check); err != nil {
14        return ctrl.Result{}, err
15    }
16
17    if !jobFinished(&check) {
18        return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
19    }
20
21    return ctrl.Result{}, r.updateStatus(ctx, &check)
22}

A few things worth calling out about the reconciliation logic above:

  • NotFound is not an error: If the object you look is gone (deleted, or the event was stale), there is nothing to do. The reconciliation can return cleanly rather than treating it as a failure.
  • Deletion is explicitly checked: This ties directly back to the finaliser pattern from part two; the reconcile should the cleanup branch instead of the normal one if DeletionTimestamp is set.
  • The return value controls what happens next:
    • An empty Result{} with no error means that the reconciliation is done and should wait for the next real event.
    • RequeueAfter schedules another look without creating a fresh event. This is the mechanism for polling something like a running Job that does not properly trigger a watch event on BackupCheck upon completion.
    • Returning a non-nil error requeues automatically, with rate-limited backoff. This is usually the right response to anything transient, like a failed API call or a not-yet-ready dependency, rather than hand-rolling your own retry logic.

A reconciler needs to react quite often to changes in objects it does not own directly. This is covered with two mechanisms:

  • Owns() for objects the controller created, like the restore Job from part two’s owner-reference example. Changing an owned object triggers a reconcile of its owner, so the controller notices a Job finishing without needing to poll it on a fixed timer.
  • Watches() for arbitrary related objects, with a mapping function that translates “this unrelated object changed” into “here is which BackupCheck that affects”.

In practice, for something like polling a long-running restore job, RequeueAfter is often simpler than wiring up Owns() and is completely legitimate. The two approaches are not mutually exclusive, and picking the less clever one is usually fine.

Pre-Defined Status Conditions

Most operators, and most of core Kubernetes, use the same metav1.Condition structure for reporting the status: Type, Status (True/False/Unknown), Reason, Message, LastTransitionTime, and ObservedGeneration.

1status:
2  conditions:
3    - type: Verified
4      status: "True"
5      reason: RestoreSucceeded
6      message: "Backup restored and checksum verified"
7      lastTransitionTime: "2026-09-01T03:04:12Z"
8      observedGeneration: 1

The benefit of sticking to this convention is not purely aesthetic; tooling like kubectl wait --for=condition=Verified and generic status dashboards already know how to consume these status conditions, for free.

High Availability: Leader Election

Running more than one replica of a controller for resilience introduces an obvious problem: If more than one controller is actively reconciling, the controller can race. In our example of BackupCheck, duplicate Job creation is the obvious failure mode.

The standard fix is leader election, backed by a Lease object in the cluster. Only the elected leader actively reconciles, and the standby replicas sit idle until they acquire the lease. The leader election is just a couple of lines of configuration on the manager, not something you have to implement yourself.

What Comes Next

Everything described in this article — the manager, the cache, the workqueue, the RBAC the controller needs to actually read and write these objects — is a fair amount of plumbing to create by hand. The next article in this series about Kubebuilder describes the tool that scaffolds it, driven by the same marker comments that generated the CRD schema back in part one.