Felix Seifert

Article

Writing My First Kubernetes CRD: What I Actually Needed to Know

7 min read Felix Seifert

This is part 1 of a four-part series on building a Kubernetes operator: CRDs (this post), Custom Resources , Operators and Kubebuilder .

Why I Went Looking Into CRDs

A while back, I started operating an own bare metal Kubernetes cluster at home, aka a home lab. To further this adventure, I have been building a small operator to automate something I used to do by hand: verifying that PostgreSQL backups can actually be restored, not just that they exist. The idea is simple:

  1. Describe “check this backup” as a Kubernetes object
  2. Let a controller do the restoring and checking
  3. Let kubectl get display the result

That is a suitable problem for a Custom Resource Definition (CRD), but I had not written one from scratch before. So before touching any Go code, I spent a few evenings figuring out what a CRD actually is. This is the condensed version of what I wish I had read first.

What a CRD Actually Is

A CustomResourceDefinition is a Kubernetes object. When you apply one, you are not creating a resource but you are teaching the API server about a new kind of resource. The API server registers a new REST endpoint, and then, kubectl, RBAC, audit logging, and every relevant client library treat your custom object exactly like a built-in object, such as a Pod or a Deployment.

Two objects are involved which have to be differentiated properly:

  • CustomResourceDefinition (CRD): the schema. Defines the shape, validation rules, and the API endpoint for a new kind of object; it basically defines a Custom Resource. The developer has to write this once.
  • Custom Resource (CR): an instance of that kind. Users create these, the same way they would create a Pod or a Deployment.

Nothing runs when you apply a CRD by itself, it just enhances the cluster’s vocabulary by a word. The actual behavior comes from somewhere else: a controller that watches for CRs and reconciles the reality towards what the CRs describe. Writing the CRD is the “define the API” half of building an operator; the controller is the other half.

Vocabulary Worth Knowing

A handful of terms show up in every CRD, and it is worth being comfortable with them before you write your first one.

Group, Version, Kind (GVK): Every API object in Kubernetes is identified by its group (a DNS-style namespace, e.g. backups.example.io), a version (v1alpha1, v1beta1, v1), and a kind (BackupCheck). Together, they are known as the apiVersion/kind pair at the top of a manifest.

Scope: A CRD scope is either Namespaced or Cluster. Most application-level resources are namespaced; things that describe cluster-wide policy or infrastructure are commonly cluster-scoped.

Structural schema: Every CRD must define an OpenAPI v3 schema that fully describes its fields, so types, required fields, defaults, constraints. The days of preserveUnknownFields: true and schema-less CRDs are over; the API server rejects anything that is not a proper structural schema.

Subresources: status and scale can be split out as separate subresources with their own endpoints and RBAC rules. Splitting out status is close to mandatory in practice: it lets your controller update status without needing write access to spec, and it means users editing spec cannot accidentally stomp on a status your controller just wrote, or vice versa.

Versioning: CRDs can serve multiple versions simultaneously, with one marked as the storage version for actually getting persisted in etcd/whichever database you use. A genuinely breaking schema change needs a conversion webhook; anything additive can usually just be a new version with sane defaults.

A Basic CRD, Piece by Piece

Here is a trimmed but complete example, modeled on the backup-verification idea above:

 1apiVersion: apiextensions.k8s.io/v1
 2kind: CustomResourceDefinition
 3metadata:
 4  name: backupchecks.backups.example.io
 5spec:
 6  group: backups.example.io
 7  scope: Namespaced
 8  names:
 9    kind: BackupCheck
10    plural: backupchecks
11    singular: backupcheck
12    shortNames: ["bc"]
13  versions:
14    - name: v1alpha1
15      served: true
16      storage: true
17      subresources:
18        status: {}
19      additionalPrinterColumns:
20        - name: Phase
21          type: string
22          jsonPath: .status.phase
23        - name: Last Verified
24          type: string
25          jsonPath: .status.lastVerifiedTime
26      schema:
27        openAPIV3Schema:
28          type: object
29          properties:
30            spec:
31              type: object
32              required: ["targetCluster", "schedule"]
33              properties:
34                targetCluster:
35                  type: string
36                schedule:
37                  type: string
38                retentionDays:
39                  type: integer
40                  minimum: 1
41                  default: 30
42            status:
43              type: object
44              properties:
45                phase:
46                  type: string
47                  enum: ["Pending", "Running", "Succeeded", "Failed"]
48                lastVerifiedTime:
49                  type: string
50                  format: date-time

Apply this CRD and the cluster will understand BackupCheck objects:

1apiVersion: backups.example.io/v1alpha1
2kind: BackupCheck
3metadata:
4  name: nightly-verification
5spec:
6  targetCluster: postgres-prod
7  schedule: "0 3 * * *"
8  retentionDays: 14

1kubectl apply -f backupcheck-crd.yaml
2kubectl apply -f nightly-verification.yaml
3kubectl get backupchecks

Up until here, no Go code is needed: kubectl get bc already works, kubectl explain backupcheck.spec already documents your fields, and the object already shows up in kubectl get events and audit logs like anything else. That is what makes CRDs feel like a small superpower the first time you see and use it.

Do Not Hand-Write This YAML

In practice, if you are building an operator with kubebuilder or controller-runtime, the CRD manifest above is something you do not maintain by hand. You write a Go struct with marker comments: +kubebuilder:validation:Required, +kubebuilder:validation:Minimum=1, and so on. The recipe controller-gen compiles those markers then into an exact schema as shown above, status subresource and printer columns included. It is a meaningfully better workflow than keeping Go types and YAML (or the same information at any distint place) in sync manually, and it is going to be the whole subject of the last post in this series.

Validation Without Writing a Webhook

Another thing worth knowing about is the Kubernetes Common Expression Language (CEL) validation rules (x-kubernetes-validations), which is available in the more moder distributions (since 1.29). They let you express constraints which cannot be expressed in a plain OpenAPI schema: cross-field checks, immutability, conditional requirements. These CEL validation rules can be directly written in the CRD, without the tedious work of creating a validating admission webhook:

1x-kubernetes-validations:
2  - rule: "self.retentionDays >= 1"
3    message: "retentionDays must be at least 1"
4  - rule: "oldSelf.targetCluster == self.targetCluster"
5    message: "targetCluster is immutable after creation"

When using kubebuilder markers, the same rules look as follows:

1// +kubebuilder:validation:XValidation:rule="oldSelf.targetCluster == self.targetCluster",message="targetCluster is immutable after creation"

Things That Were Not Obvious at First

A few gotchas worth knowing before you hit them yourself:

  • Kind is singular and written in PascalCase; plural and singular names are separate fields. If you get the plural wrong, kubectl get yourkind will just not resolve, there is no fuzzy matching.
  • The CRD’s own name must be <plural>.<group>, exactly. Doing such a typo can be done easily, and the resulting error is not obviously related to your schema changes.
  • Split out the status subresource early. I suggest splitting it out as described above, retrofitting it once you have real users means revisiting every manifest and every RBAC rule that assumed a single write surface.
  • New versions should be additive. Removing or renaming a field is a breaking change that needs a conversion strategy, not just a version bump, try to avoid this hasle.
  • CEL rules run on every write. Keep them cheap, because the API server enforces a computed cost budget and rejects expressions it estimates as too expensive.
  • additionalPrinterColumns is free UX. A couple of jsonPath entries and kubectl get becomes more readable instead of an unhelpful wall of Age.

Why CRD Instead of Reaching for ConfigMap and Script

With a CRD, you get everything for free Kubernetes already built for its own resources: RBAC scoped down to the field level, kubectl and kubectl explain support, audit logs, optimistic concurrency, generated typed clients, and a GitOps-friendly declarative model that tools like ArgoCD or Flux can already reconcile. With a ConfigMap plus a cron job, you get none of that and it quietly breaks the moment someone edits the YAML by hand.

You also have to keep in mind the trade-off: You are taking on an API you own, and you have to version it responsibly; anyone using your cluster now depends on a schema you maintain. For a one-off script, that is an overkill. For anything you expect other people, or your future self, to operate against repeatedly, it tends to be worth it.

What Comes Next

Defining the CRD is a static description of intent with no behavior behind it yet; it is a schema which adapts some existing Kubernetes functionality to your use case. What I conveniently ignored over here is what happens to an actual BackupCheck object during its lifetime once it exists: resourceVersion, finalisers, owner references, and the difference between editing spec and editing status. That comes next in this series, before I come to the controller that gives these objects any behavior at all.