Building Kubernetes Operators part 2: Design and implementation details

  • March 21, 2019

Kubernetes operators were introduced as an implementation of the Infrastructure as software concept. Using them you can abstract the deployment of applications and services in a Kubernetes cluster. This is the second of a series of articles explaining how operators work, and how they can be implemented in different languages.


In the previous article we saw that operators function to extend Kubernetes API, easing some tasks, orchestrating and managing objects, for example.

Remember that operators combine custom resources and custom controllers. Custom resources are used to define new object kinds and collections of objects, accessible to the users via Kubernetes API. Reading and writing to these collections, users specify the desired state and query the actual one.

Custom controllers are programs that try to transform the actual state into the desired one, using Kubernetes API to watch changes and execute actions.

We are about to see how operators should be designed, why Declarative APIs are a great choice for this use case, and how they can be implemented in Kubernetes using custom resources and custom controllers.

Declarative APIs

As you’ve probably noticed, almost all Kubernetes APIs are declarative. This means that users designate a desired state rather than specific actions to be executed. This is the key to declarative APIs; users specify what they want, not how to achieve it.

For example, in Kubernetes users have no way to kill a pod. Instead, they delete the pod.  Kubernetes handles the details of its termination and frees its resources.

For certain use cases such as infrastructure and orchestration, declarative APIs are simpler and easier to use than imperative APIs. They abstract implementation details, hiding from the API user the complex mechanisms needed to deal with internal factors, like node failures and communication problems.

While declarative APIs are not always the best solution, they probably are the best for our use case: implementation of operators that manage objects on a cluster. Since our operator will extend Kubernetes API, it’s also a good idea to follow Kubernetes practices.

Unfortunately, implementing a declarative API is harder than using it. The developer must define an interface through which users will read and write the desired and actual states. He must also create a mechanism to drive the system from the current state to the desired one (states reconciliation).
We are going to use custom resources and custom controllers to implement our operators, building declarative APIs that extend the Kubernetes API. The following sections explain in detail how to use custom resources and custom controllers.

Custom Resource

Operators use custom resources to expose objects and collections of objects to the users via Kubernetes API. This way, users can access the desired and actual states as structured data stored inside those objects. To achieve this, operators add new object kinds to Kubernetes — each with its own fields and validations — and endpoints that extend the Kubernetes API, providing users with a REST API to access those objects.

We are going to use Custom Resource Definitions  (CRDs) to define our object kinds and API endpoints. CRD is a Kubernetes feature that enables users to create new types of resources and automatically exposes them through Kubernetes’ REST API. From a user perspective our custom resources behave exactly as any built-in Kubernetes resources (pods, for example) and can be managed using any client, such as kubectl.

There are no hard standards or limitations about how we should organize the structure of our objects, but there are some common Kubernetes best practices and conventions we should follow to make our API easier to understand. One key practice is to divide each object’s fields into two sections: “spec”, specifying the desired status, and “status”, specifying the actual status.

For example, let’s say that we are building an operator to provide users a way to create “immortal containers” — that is, users will specify the image to run, and the operator will create a pod with a container in which to run the image and then recreate the pod whenever it terminates or is deleted. Suppose that we also want the operator to expose the name of the created pod and the number of times it has been created. To achieve this we must create a new object kind, ImmortalContainer, with the following structure:

    - Spec
        - Image
    - Status
        - CurrentPod
        - StartTimes

The following yaml file is the CRD needed to create such an object kind.

kind: CustomResourceDefinition
    kind: ImmortalContainer
    listKind: ImmortalContainerList
    plural: immortalcontainers
    singular: immortalcontainer
  scope: Namespaced
    status: {}
          type: string
          type: string
          type: object
              minLength: 1
              type: string
          - image
          type: object
              type: string
              format: int64
              type: integer
          type: object
  version: v1alpha1
  - name: v1alpha1
    served: true
    storage: true
  • The group field corresponds to the operator name: to avoid conflicts it should end with a domain name.
  • The kind field defines the object kind name.
  • The validations section allows specification of data type validations and constraints.
  • The version field specifies the version of this API, making it easier to evolve it.

Note how in the crd.yaml example we divided the fields into two groups, spec and status. This distinction will be really useful when implementing the operator’s controller, since it separates desired and actual status.

To install the resource on a cluster we just need to run `kubectl apply -f crd.yaml`. After that, we will be able to read and write to the resource using kubectl.

Example: create an object 
$ kubectl apply -f example.yaml
kind: ImmortalContainer
  name: example-immortalcontainer
  image: nginx:latest
Example: list objects
$ kubectl get ImmortalContainer

Now we have provided users with a way to set the desired state and read the actual one, but we have not yet provided the logic needed to transform the actual state into the desired one. The next section details how we can build a custom controller to do that.

How Controllers Work ?

The main task of a controller is to reconcile the desired and actual states — that’s to say, transform the actual state into the desired one. In order to do that, the controller keeps watching for changes in both states and triggers the reconcile loop when a change is detected. The reconcile loop executes the actions needed to mutate the current state in order to match the desired state one.

Kubernetes API already provides a set of methods that can be used to watch for events on any resource. Controllers use the API to watch their main custom resources and any other relevant resource(s).

For example, to implement the immortal containers operator, the controller must not only watch for changes in ImmortalContainer objects but also respond to pod events. Note that not every pod is relevant to this controller, just the ones that are associated with an ImmortalContainer object.

To keep track of the associations between objects, Kubernetes has a mechanism called “owner references”. This allows controllers to set on any object a link to its parent or creator. Using this, on an incoming pod event the controller can tell which ImmortalContainer object might have been affected, and respond by checking and reconciling (if necessary) its desired and actual states.

The following diagram shows the controller’s main components for this example operator:

This controller sets up two watchers, one for ImmortalContainer objects and another for Pod objects. Upon the arrival of an event, the state of the concerned ImmortalContainer object is reconciled. The controller uses Kubernetes API to watch events, fetch objects, create pods, and update the status section of ImmortalContainer objects.

In Conclusion

We have just seen how using Custom Resource Definitions allows us to extend Kubernetes API with custom object kinds; and we have learned how to divide the object fields into spec and status sections in order to distinguish between fields related to desired and actual states.

We have also reviewed how controllers, using Kubernetes API, synchronize the desired and actual states, and how they keep track of relationships between objects by using owner references.

In future articles we are going to use what we have just learned to implement the example operator, immortal containers, in Go and in Python.