Custom Resources: What Happens After Defining the Schema

This is part 2 of a four-part series on building a Kubernetes operator: CRDs , Custom Resources (this post), Operators and Kubebuilder .

More Than YAML That Passes Validation

Last time, I defined BackupCheck as a Custom Resource Definition (CRD) and applied a Custom Resource against it:

1apiVersion: backups.example.io/v1alpha1
2kind: BackupCheck
3metadata:
4  name: nightly-verification
5  namespace: data-platform
6spec:
7  targetCluster: postgres-prod
8  schedule: "0 3 * * *"
9  retentionDays: 14

In that last post, I created a CRD focussed on the schema, describing what shape the object, the resulting Custom Resource (CR), is supposed to take. What I have not mentioned is everything that happens to the object after it is created: how the API server tracks changes to it, how a controller safely coordinates concurrent writes, how deletion actually works, and how child objects it creates get cleaned up automatically. This is going to be functionally more interesting.

Objects Carry More Than spec and status

Looking at the full metadata of any object applied, there is more there than you probably wrote:

1metadata:
2  name: nightly-verification
3  namespace: data-platform
4  uid: 3fa1b2c4-9e21-4a7f-b8e0-1234567890ab
5  resourceVersion: "48213"
6  generation: 2
7  creationTimestamp: "2026-08-30T03:00:00Z"

• uid is assigned once and never changes, the object’s true identity, independent of the name; this is what owner references actually point at under the hood.
• resourceVersion is an opaque string that is bumped on every write to the object that actually changes the object (spec, status, labels, anything). You should not attach meaning to it as a traditional counter, it purely exists for concurrency control.
• generation increments only when .spec or deletionTimestamp changes; changing status, label or annotation does not touch it. That distinction is what makes it useful.

Optimistic Concurrency, No Locking

Kubernetes does not lock objects for editing. Instead, writes usually include the respective resourceVersion. If someone else has written to the object in the meantime, the write is rejected with a conflict rather than silently overwriting the previous change.

For a controller, this shows up constantly: You fetch an object, do some work, and try to update its status. If anything else touched the object in between (a user editing spec, or your own controller processing a stale cache entry), the update fails and the object needs to be re-fetched and the edit retried. client-go’s retry.RetryOnConflict exists specifically for this loop, and it is worth using it rather than hand-rolling the same functionality.

The newer alternative is server-side apply: Instead of read-modify-write against the whole object, fields are tagged with the field manager which last set it (visible in metadata.managedFields), and the merge gets applied only at the field level. kubectl apply --server-side uses this path, and this is increasingly how controllers manage the specific fields they own without needing to know or preserve anything else of the object.

generation vs. observedGeneration

generation only moves when spec changes. It is therefore the basis for a very common status convention: a controller sets status.observedGeneration to the metadata.generation it has actually finished reconciling.

1status:
2  observedGeneration: 2
3  phase: Running
4  lastVerifiedTime: "2026-08-30T03:04:12Z"

If metadata.generation is ahead of status.observedGeneration, anything looking at the object (e.g. a dashboard, kubectl or another controller) can directly tell that a spec change is still waiting to be picked up, without needing to inspect timestamps or diff anything.

Finalisers: Pausing Deletion Until Cleanup Is Done

Deleting a Kubernetes object with kubectl delete removes it by default more or less immediately. This immediate removal can be prevented with a finaliser. This is a string in metadata.finalizers: When you delete an object with a finaliser present, the API server sets metadata.deletionTimestamp but keeps the object in etcd (K3s defauls to SQLite and other distributions might have another database) until every finaliser of this object has been removed.

1metadata:
2  finalizers:
3    - backups.example.io/cleanup
4  deletionTimestamp: "2026-08-15T09:00:00Z"

The pattern of the controller is to check on every reconcile whether the deletionTimestamp is set. If it is, it runs the cleanup logic instead of normal reconciliation. In case of our BackupCheck, that could mean tearing down the scratch restore instance and its volume; the finaliser should only be removed once that teardown is confirmed. Removing it too early results in leaking resources; forgetting to remove it at all results in the object being stuck in Terminating forever, which is an early footgun with this pattern.

Owner References and Garbage Collection

When the controller creates a child object, like a Job that performs the actual restore-and-verify for a BackupCheck, it is worth setting an owner reference on it:

 1apiVersion: batch/v1
 2kind: Job
 3metadata:
 4  name: nightly-verification-restore
 5  namespace: data-platform
 6  ownerReferences:
 7    - apiVersion: backups.example.io/v1alpha1
 8      kind: BackupCheck
 9      name: nightly-verification
10      uid: 3fa1b2c4-9e21-4a7f-b8e0-1234567890ab
11      controller: true
12      blockOwnerDeletion: true

This links the two objects, the Job and the BackupCheck, by UID. Kubernetes’ built-in garbage collector then uses it to cascade deletes: When the BackupCheck is deleted, the Job gets cleaned up automatically and no extra bookkeeping in your controller is required. It is a small thing, but it means the controller does not need to remember every child it ever created and you do not have to take care of this, the ownership graph does that for you.

Foreground Propagation != Background Propagation

blockOwnerDeletion: true deserves a note, because it does nothing in the previously described sequence; it only takes effect under foreground cascading deletion, where the garbage collector puts a foregroundDeletion finaliser on the owner and keeps it in place until every dependent blocker is gone.

kubectl delete uses background cascading deletion by default, so the BackupCheck disappears first and the Job is automatically collected after it. It is in the example because controllerutil.SetControllerReference sets it for you, and because the guarantee costs nothing if someone does delete with --cascade=foreground. There is one string attached: setting it requires update on the owner’s finalizers subresource, which is why kubebuilder scaffolds an RBAC marker for backupchecks/finalizers.

Status Writes Go Through Different Endpoint

Because the CRD from the last post enables the status subresource, there is a gotcha worth flagging: Once that subresource exists, writes to .status through the main resource endpoint are silently ignored. You have to write to it through /status specifically.

For controller-runtime’s client, this means calling r.Status().Update(ctx, obj), not r.Update(ctx, obj). Calling the regular update after setting obj.Status.Phase and wondering why it does not stick might be confusing; it did however exactly what it was supposed to do, just not to the field you were watching.

A Day in the Life of a BackupCheck Object

Putting it together can roughly be described as follows:

1. Object is created: generation starts at 1 and resourceVersion receives some number from the data store.
2. Controller reconciles, creates Job with owner reference back to the BackupCheck, and writes status.phase = Running via the status subresource: resourceVersion gets bumped, generation does not, because spec did not change.
3. Job finishes: controller updates status.phase = Succeeded and status.observedGeneration = 1.
4. User edits spec.retentionDays: generation becomes 2, observedGeneration is now stale and the controller picks that up on its next reconcile.
5. Someone runs kubectl delete backupcheck nightly-verification: deletionTimestamp is set and the object stays visible but Terminating, generation is incremented.
6. Controller notices: tears down whatever scratch resources need explicit cleanup, and removes its finaliser.
7. Object disappears from datastore: garbage collector cascades deletion to the owned Job.

None of this happens on its own, every step from 2 onwards is something a controller has to actually implement. That will be the subject of the next post: the reconciliation loop itself, and why it is built the way it is.