Building Custom Kubernetes Operators Part 6: Building Operators using Metacontroller

  • July 16, 2019

Kubernetes operators were introduced as an implementation of the Infrastructure as software concept. Using them you can abstract the deployment of applications and services in a Kubernetes cluster. This is the sixth and last of a series of articles explaining how operators work and how they can be implemented in different languages.



In the previous articles in this series we saw how operators can be implemented in different programming languages. We also saw that, regardless of the chosen language, all operator controllers work the same way, executing the following actions:

  1. Watch for events about relevant resources.
  2. Run the reconcile loop to transform the current state into the desired one.
    1. Query current state and desired state information stored in objects, using Kubernetes API.
    2. Calculate what the state should be.
    3. Execute actions, using Kubernetes API, to correct the state.

  Metacontroller is an add-on for Kubernetes that handles the tasks common to every operator controller, (events watching, objects’ states querying, and actions execution). The decoupling of these tasks from the functionality specific to each controller allows the developer to focus on the latter: calculation of the correct state.


In the next sections of this article we are going to see how Metacontroller works and how to implement a simple operator using it.


This article assumes you have Python (version at least 3.6) installed in your computer. You will also need access to a Kubernetes cluster to try the operator. You can use minikube to create a development cluster.


The complete source code for the operator described in this article can be found here


How Metacontroller works

A controller implementation using the Metacontroller has two components: the Metacontroller and one or more lambda controllers.


The Metacontroller is a server that extends Kubernetes API and encapsulates the common parts of writing custom controllers: events watching, states querying and actions execution.


Each lambda controller contains the business logic specific to a custom controller, that is, the functionality determining what the state should be.


The Metacontroller watches multiple resources in order to detect changes in the actual or desired state. Detection of a change invokes the relevant lambda controllers. Using the information from the lambda controller about the desired state, the Metacontroller applies the necessary changes, executing actions calling the Kubernetes API.


The Metacontroller communicates with lambda controllers using webhooks. This greatly simplifies the design and implementation of the lamda controllers, eliminating the need for direct use of Kubernetes API and allowing implementation of controllers in any language that can understand HTTP and JSON.



With Metacontroller, developers have only to create and register their lamda controllers. Metacontroller provides :

  • Label selectors (for defining flexible collections of objects)
  • Orphan/adopt semantics (controller reference)
  • Garbage collection (owner references for automatic cleanup)
  • Watches (for low latency)
  • Caching (shared informers/reflectors/listers)
  • Work queues (deduplicated parallelism)
  • Optimistic concurrency (resource version)
  • Retries with exponential backoff
  • Periodic relist/resync


Currently, Metacontroller supports the implementation of two kind of controllers:

  • CompositeController: We use this when the controller manages objects composed of other objects. This arrangement implies a parent-child relationship between objects, commonly various objects (children) are populated from user-provided information in a parent object that belongs to a custom resource.
  • DecoratorController: Deals with attaching new behavior to existing objects, such as pods, for example.


We are focusing on CompositeControllers in this article. When registering a composite controller, the developer specifies the parent and child resources and the URL of the lambda controller webhook. Metacontroller watches the specified resources and calls the webhook when a change occurs.


When invoked, the webhook receives the state of the parent and its children. It must return the desired state of the parent and a list of all the children that should exist. The Metacontroller then takes care of the actions necessary to move from the current state to the desired one.


In the next sections we are going to use Metacontroller to implement the example operator, immortal containers.


The immortal containers operator

As we said in previous articles, the purpose of the immortal containers operator is to enable users to define containers that should run forever — that is, whenever such containers terminate for any reason, they will be restarted.


Keep in mind that the operator demonstrated in this article is just a toy example which serves only to illustrate the steps involved in the implementation of an operator. The functionality it provides can be achieved with already existing Kubernetes features, such as deployments.


This operator defines a new object kind named ImmortalContainer. Users create objects of this kind to specify containers that must run forever. In each object the user specifies the image he wants to run.


For each ImmortalContainer object the operator’s controller creates a pod to run the container and then recreates the pod whenever it terminates or is deleted. In the same object the operator exposes the name of the created pod.  — In previous implementations we added a field to count the number of times the pod has been created; we will omit that step in the current implementation to make it simpler.


Each ImmortalContainer object has the following structure:


    - Spec
        - Image
    - Status
        - CurrentPod



Let’s say the operator has been installed and the user wants to create an immortal container to run the image nginx:latest. To do so, he can use kubectl to create an ImmortalContainer object.

# example.yaml
kind: ImmortalContainer
  name: example-immortalcontainer
  image: nginx:latest


$ kubectl apply -f example.yaml

The controller will detect the new immortal container and respond by creating a pod to run the image nginx:latest. The user can then view the running pod using the following command:


$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
example-immortalcontainer-immortalpod   1/1    unning 0    25m



If someone deletes the pod, it will be recreated.


$ kubectl delete pods example-immortalcontainer-immortalpod
pod "example-immortalcontainer-immortalpod" deleted
$ kubectl get pods                                         
NAME                          READY   STATUS            RESTARTS   AGE
example-immortalcontainer-immortalpod   0/1   ContainerCreating 0   3s



Finally, the user can edit the ImmortalContainer object he has created to see the CurrentPod field.


$ kubectl edit immortalcontainer example-immortalcontainer

kind: ImmortalContainer
 image: nginx:latest
 currentPod: example-immortalcontainer-immortalpod


Implementing an operator using Metacontroller


Custom Resource

The custom resources are used to expose the desired and actual states. They define endpoints that give access to collections of objects.


The operator we implemented uses a custom resource to expose a collection of objects belonging to the ImmortalContainer object kind. Users create objects of this kind to specify containers that need to run forever. 

As we said previously, each ImmortalContainer object has the following structure:


    - Spec
        - Image
    - Status
        - CurrentPod



We used a Custom Resource Definition to create the operator’s custom resource. Again, as in the previous article, we had to write the CRD yaml file. This file defines the new object kind, ImmortalContainer, with its fields and validations.



kind: CustomResourceDefinition
   kind: ImmortalContainer
   listKind: ImmortalContainerList
   plural: immortalcontainers
   singular: immortalcontainer
 scope: Namespaced
   status: {}
         type: string
         type: string
         type: object
             minLength: 1
             type: string
         - image
         type: object
             type: string
         type: object
 version: v1alpha1


Note that we’ve indicated that our API group is, the API version is v1alpha1, and the name of the new object kindis ImmortalContainer.


Lambda controller

Our operator’s lambda controller provides a webhook. This webhook receives a JSON object containing the current state of an ImmortalContainer object and its Pod child (if any), and returns information specifying the desired state.

The following block shows the complete source code of this lambda controller:

import json
from http.server import BaseHTTPRequestHandler, HTTPServer
def sync(parent, children):
 # Compute status based on observed state.
 desired_status = {}
  if len(children['Pod.v1']) == 1:
   desired_status['currentPod'] = list(children['Pod.v1'])[0]
   desired_status['currentPod'] = ''
 # Generate the desired child object(s).
 desired_pods = [
     'apiVersion': 'v1',
     'kind': 'Pod',
     'metadata': {
       'name': parent['metadata']['name']+'-immortalpod',
       'namespace': parent['metadata']['namespace'],
     'spec': {
       'restartPolicy': 'OnFailure',
       'containers': [
           'name': 'acontainer',
           'image': parent['spec']['image'],
 return {'status': desired_status, 'children': desired_pods}
class Controller(BaseHTTPRequestHandler):
 def do_POST(self):
   # Serve the sync() function as a JSON webhook.
   observed = json.loads('content-length'))))
   desired = sync(observed['parent'], observed['children'])
   self.send_header('Content-type', 'application/json')
if __name__ == '__main__':
 print("server starting...")
 HTTPServer(('', 80), Controller).serve_forever()


You might have noticed that the code is much shorter and simpler than the ones from previous implementations in Go and in Python without Metacontroller. This is one of the main advantages of working with Metacontroller.


The code above uses the following rules to compute desired state for an ImmortalContainer object.

  • If there is currently one child, then the CurrentPod field of the ImmortalContainer object must be equal to the name of the pod. Else, the CurrentPod field must be blank.
  • There must be one and only one child and it must be a Pod with one container running the specified image.

The lambda controller must be registered with the Metacontroller. Registration involves creation of a CompositeController object specifying parent and child resources and the webhook URL. We will see how to do this later.


Operator deployment

In order to use the operator we must deploy it to the cluster using the following steps:

  1. Install Metacontroller in the cluster.
  2. Deploy the lambda controller code to the cluster.
  3. Create a service to publish the webhook URL.
  4. Register the lambda controller.


Installing Metacontroller

Metacontroller can be installed in any Kubernetes cluster following its installation guide. Here we are reproducing the instructions:


Check the cluster availability running


$ kubectl get nodes


Execute these commands to install Metacontroller to the cluster.


# Create metacontroller namespace.
$ kubectl create namespace metacontroller
# Create metacontroller service account and role/binding.
$ kubectl apply -f
# Create CRDs for Metacontroller APIs, and the Metacontroller StatefulSet.
$ kubectl apply -f



Note that, in order for Metacontroller to function properly, the Kubernetes DNS Service must be enabled in the cluster. This is the default on most clusters, but on some development clusters it’s disabled.


After installing Metacontroller you are ready to deploy any existing lambda controller or create your own.


Deploying the lambda controller

Since the code of our controller is just one file with no extra dependencies, we run it using the default python image instead of building a custom image. To do this we created a ConfigMap to store the code, using kubectl.


$ kubectl -n immortalcontainers create configmap immortalcontainers-controller --dry-run -o yaml | kubectl apply -f - is the file containing the webhook code. Note that we created the ConfigMap inside the immortalcontainers namespace (using the -n option).

Then, we created a deployment that runs using the official Python image. It mounts the code from the ConfigMap using a volume.


apiVersion: apps/v1beta1
kind: Deployment
 name: immortalcontainers-controller
 replicas: 1
     app: immortalcontainers-controller
       app: immortalcontainers-controller
     - name: controller
       image: python:3.7
       command: ["python", "-u", "/hooks/"]
       - name: hooks
         mountPath: /hooks
     - name: hooks
         name: immortalcontainers-controller


Publishing the webhook using a service

Since the Metacontroller must be able to resolve and connect to the webhook URL, we created a service to make the webhook callable from outside its pod.


apiVersion: v1
kind: Service
 name: immortalcontainers-controller
   app: immortalcontainers-controller
 - port: 80


Lambda controller registration

The lambda controller must be registered with the Metacontroller. Registration requires creation of a CompositeController object containing the names of parent and child resources and the webhook URL.


kind: CompositeController
 name: immortalcontainers-controller
 generateSelector: true
   resource: immortalcontainers
 - apiVersion: v1
   resource: pods
     method: Recreate
       url: http://immortalcontainers-controller.immortalcontainers/sync

Full reference about CompositeController objects can be found here.


this information tells the Metacontroller what resources to watch and what webhook to invoke when a change occurs.


Trying the operator

Now that the operator is installed in the cluster, we can try it. We are going to create an ImmortalContainer object to run the nginx:latest image. To do this we need to edit the file config/example-use.yaml to make it look like this:


kind: ImmortalContainer
 name: example-immortal-container
 image: nginx:latest


We then use kubectl to create the ImmortalContainer object in the cluster.


$ kubectl apply -f config/example-use.yam



The controller will detect the new immortal container and create a pod to run its image. Let’s verify this.


$ kubectl get pods
NAME                          READY    STATUS     RESTARTS    AGE
example-immortal-container-immortalpod   1/1     Running 0    2m



Next, let’s see that the pod is recreated if we delete it.


$ kubectl delete pods example-immortal-container-immortalpod
pod "example-immortal-container-immortalpod" deleted
$ kubectl get pods                                         
NAME                           READY   STATUS              RESTARTS     AGE
example-immortal-container-immortalpod   0/1     ContainerCreating  0      3s



Finally, we can edit the ImmortalContainer object to see its status.



$ kubectl edit immortalcontainer example-immortalcontainer


kind: ImmortalContainer
 image: nginx:latest
 currentPod: example-immortal-container-immortalpod


As you can see, the operator works as expected.


Clean up

Finally, if you’ve been following and reproducing these steps in your cluster, you can use these commands to remove all the changes that have been made.


kubectl delete -f config/crds.yaml
kubectl delete -f
kubectl delete -f
kubectl delete namespace immortalcontainers
kubectl delete namespace metacontroller



In Conclusion

In this article we have seen how, by using Metacontroller, developers can avoid much of the complexity related to operators development. Metacontroller decouples and abstracts the tasks of events watching, state querying, and API calls, allowing developers to focus on the operators’ logic.


Another advantage of Metacontroller, as we have already mentioned, is that its architecture enables developers to use almost any programming language. Also, since lambda controllers don’t depend on any service, they are easy to test.


Finally, we think it’s worth noting that we have used only a subset of the Metacontroller features. You can learn more about the Metacontroller in the official documentation, which also provides examples in various programming languages.