Building Custom Kubernetes Operators Part 6: Building Operators using Metacontroller

Posted on 16 Jul 2019, By Miguel García under DevOps, Infrastructure as Code, Kubernetes

Get Started

Kubernetes operators were introduced as an implementation of the Infrastructure as software concept. Using them you can abstract the deployment of applications and services in a Kubernetes cluster. This is the sixth and last of a series of articles explaining how operators work and how they can be implemented in different languages.

 

Introduction

In the previous articles in this series we saw how operators can be implemented in different programming languages. We also saw that, regardless of the chosen language, all operator controllers work the same way, executing the following actions:

  1. Watch for events about relevant resources.
  2. Run the reconcile loop to transform the current state into the desired one.
    1. Query current state and desired state information stored in objects, using Kubernetes API.
    2. Calculate what the state should be.
    3. Execute actions, using Kubernetes API, to correct the state.

  Metacontroller is an add-on for Kubernetes that handles the tasks common to every operator controller, (events watching, objects’ states querying, and actions execution). The decoupling of these tasks from the functionality specific to each controller allows the developer to focus on the latter: calculation of the correct state.

 

In the next sections of this article we are going to see how Metacontroller works and how to implement a simple operator using it.

 

This article assumes you have Python (version at least 3.6) installed in your computer. You will also need access to a Kubernetes cluster to try the operator. You can use minikube to create a development cluster.

 

The complete source code for the operator described in this article can be found here  https://github.com/flugel-it/k8s-metacontroller-operator

 

How Metacontroller works

A controller implementation using the Metacontroller has two components: the Metacontroller and one or more lambda controllers.

 

The Metacontroller is a server that extends Kubernetes API and encapsulates the common parts of writing custom controllers: events watching, states querying and actions execution.

 

Each lambda controller contains the business logic specific to a custom controller, that is, the functionality determining what the state should be.

 

The Metacontroller watches multiple resources in order to detect changes in the actual or desired state. Detection of a change invokes the relevant lambda controllers. Using the information from the lambda controller about the desired state, the Metacontroller applies the necessary changes, executing actions calling the Kubernetes API.

 

The Metacontroller communicates with lambda controllers using webhooks. This greatly simplifies the design and implementation of the lamda controllers, eliminating the need for direct use of Kubernetes API and allowing implementation of controllers in any language that can understand HTTP and JSON.

 

 

With Metacontroller, developers have only to create and register their lamda controllers. Metacontroller provides :

 

Currently, Metacontroller supports the implementation of two kind of controllers:

 

We are focusing on CompositeControllers in this article. When registering a composite controller, the developer specifies the parent and child resources and the URL of the lambda controller webhook. Metacontroller watches the specified resources and calls the webhook when a change occurs.

 

When invoked, the webhook receives the state of the parent and its children. It must return the desired state of the parent and a list of all the children that should exist. The Metacontroller then takes care of the actions necessary to move from the current state to the desired one.

 

In the next sections we are going to use Metacontroller to implement the example operator, immortal containers.

 

The immortal containers operator

As we said in previous articles, the purpose of the immortal containers operator is to enable users to define containers that should run forever — that is, whenever such containers terminate for any reason, they will be restarted.

 

Keep in mind that the operator demonstrated in this article is just a toy example which serves only to illustrate the steps involved in the implementation of an operator. The functionality it provides can be achieved with already existing Kubernetes features, such as deployments.

 

This operator defines a new object kind named ImmortalContainer. Users create objects of this kind to specify containers that must run forever. In each object the user specifies the image he wants to run.

 

For each ImmortalContainer object the operator’s controller creates a pod to run the container and then recreates the pod whenever it terminates or is deleted. In the same object the operator exposes the name of the created pod.  — In previous implementations we added a field to count the number of times the pod has been created; we will omit that step in the current implementation to make it simpler.

 

Each ImmortalContainer object has the following structure:

ImmortalContainer
    - Spec
        - Image
    - Status
        - CurrentPod

 

Let’s say the operator has been installed and the user wants to create an immortal container to run the image nginx:latest. To do so, he can use kubectl to create an ImmortalContainer object.


# example.yaml
apiVersion: immortalcontainer.flugel.it/v1alpha1
kind: ImmortalContainer
metadata:
  name: example-immortalcontainer
spec:
  image: nginx:latest

 

$ kubectl apply -f example.yaml


The controller will detect the new immortal container and respond by creating a pod to run the image nginx:latest. The user can then view the running pod using the following command:

$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
example-immortalcontainer-immortalpod   1/1    unning 0    25m

 

If someone deletes the pod, it will be recreated.

$ kubectl delete pods example-immortalcontainer-immortalpod
pod "example-immortalcontainer-immortalpod" deleted
 
$ kubectl get pods                                         
NAME                          READY   STATUS            RESTARTS   AGE
example-immortalcontainer-immortalpod   0/1   ContainerCreating 0   3s

 

Finally, the user can edit the ImmortalContainer object he has created to see the CurrentPod field.

 

$ kubectl edit immortalcontainer example-immortalcontainer


apiVersion: immortalcontainer.flugel.it/v1alpha1
kind: ImmortalContainer
metadata:
…
spec:
 image: nginx:latest
status:
 currentPod: example-immortalcontainer-immortalpod

 

Implementing an operator using Metacontroller

 

Custom Resource

The custom resources are used to expose the desired and actual states. They define endpoints that give access to collections of objects.

 

The operator we implemented uses a custom resource to expose a collection of objects belonging to the ImmortalContainer object kind. Users create objects of this kind to specify containers that need to run forever. 


As we said previously, each ImmortalContainer object has the following structure:

ImmortalContainer
    - Spec
        - Image
    - Status
        - CurrentPod

 

We used a Custom Resource Definition to create the operator’s custom resource. Again, as in the previous article, we had to write the CRD yaml file. This file defines the new object kind, ImmortalContainer, with its fields and validations.

 

config/crds.yaml:


apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
 name: immortalcontainers.immortalcontainers.flugel.it
spec:
 group: immortalcontainers.flugel.it
 names:
   kind: ImmortalContainer
   listKind: ImmortalContainerList
   plural: immortalcontainers
   singular: immortalcontainer
 scope: Namespaced
 subresources:
   status: {}
 validation:
   openAPIV3Schema:
     properties:
       apiVersion:
         type: string
       kind:
         type: string
       metadata:
         type: object
       spec:
         properties:
           image:
             minLength: 1
             type: string
         required:
         - image
         type: object
       status:
         properties:
           currentPod:
             type: string
         type: object
 version: v1alpha1

 

Note that we’ve indicated that our API group is immortalcontainers.flugel.it, the API version is v1alpha1, and the name of the new object kindis ImmortalContainer.

 

Lambda controller

Our operator’s lambda controller provides a webhook. This webhook receives a JSON object containing the current state of an ImmortalContainer object and its Pod child (if any), and returns information specifying the desired state.


The following block shows the complete source code of this lambda controller:


# sync.py
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
 
def sync(parent, children):
 # Compute status based on observed state.
 desired_status = {}
  if len(children['Pod.v1']) == 1:
   desired_status['currentPod'] = list(children['Pod.v1'])[0]
 else:
   desired_status['currentPod'] = ''
 
 # Generate the desired child object(s).
 desired_pods = [
   {
     'apiVersion': 'v1',
     'kind': 'Pod',
     'metadata': {
       'name': parent['metadata']['name']+'-immortalpod',
       'namespace': parent['metadata']['namespace'],
     },
     'spec': {
       'restartPolicy': 'OnFailure',
       'containers': [
         {
           'name': 'acontainer',
           'image': parent['spec']['image'],
         }
       ]
     }
   }
 ]
 
 return {'status': desired_status, 'children': desired_pods}
 
 
class Controller(BaseHTTPRequestHandler):
 def do_POST(self):
   # Serve the sync() function as a JSON webhook.
   self.headers
   observed = json.loads(self.rfile.read(int(self.headers.get('content-length'))))
   desired = sync(observed['parent'], observed['children'])
 
   self.send_response(200)
   self.send_header('Content-type', 'application/json')
   self.end_headers()
   self.wfile.write(json.dumps(desired).encode())
 
if __name__ == '__main__':
 print("server starting...")
 HTTPServer(('', 80), Controller).serve_forever()

 

You might have noticed that the code is much shorter and simpler than the ones from previous implementations in Go and in Python without Metacontroller. This is one of the main advantages of working with Metacontroller.

 

The code above uses the following rules to compute desired state for an ImmortalContainer object.


The lambda controller must be registered with the Metacontroller. Registration involves creation of a CompositeController object specifying parent and child resources and the webhook URL. We will see how to do this later.

 

Operator deployment

In order to use the operator we must deploy it to the cluster using the following steps:

  1. Install Metacontroller in the cluster.
  2. Deploy the lambda controller code to the cluster.
  3. Create a service to publish the webhook URL.
  4. Register the lambda controller.

 

Installing Metacontroller

Metacontroller can be installed in any Kubernetes cluster following its installation guide. Here we are reproducing the instructions:

 

Check the cluster availability running

 

$ kubectl get nodes

 

Execute these commands to install Metacontroller to the cluster.

# Create metacontroller namespace.
$ kubectl create namespace metacontroller
 
# Create metacontroller service account and role/binding.
$ kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/metacontroller/master/manifests/metacontroller-rbac.yaml
 
# Create CRDs for Metacontroller APIs, and the Metacontroller StatefulSet.
$ kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/metacontroller/master/manifests/metacontroller.yaml

 

Note that, in order for Metacontroller to function properly, the Kubernetes DNS Service must be enabled in the cluster. This is the default on most clusters, but on some development clusters it’s disabled.

 

After installing Metacontroller you are ready to deploy any existing lambda controller or create your own.

 

Deploying the lambda controller

Since the code of our controller is just one file with no extra dependencies, we run it using the default python image instead of building a custom image. To do this we created a ConfigMap to store the code, using kubectl.

$ kubectl -n immortalcontainers create configmap immortalcontainers-controller --from-file=sync.py --dry-run -o yaml | kubectl apply -f -

 

sync.py is the file containing the webhook code. Note that we created the ConfigMap inside the immortalcontainers namespace (using the -n option).


Then, we created a deployment that runs sync.py using the official Python image. It mounts the code from the ConfigMap using a volume.

 


apiVersion: apps/v1beta1
kind: Deployment
metadata:
 name: immortalcontainers-controller
spec:
 replicas: 1
 selector:
   matchLabels:
     app: immortalcontainers-controller
 template:
   metadata:
     labels:
       app: immortalcontainers-controller
   spec:
     containers:
     - name: controller
       image: python:3.7
       command: ["python", "-u", "/hooks/sync.py"]
       volumeMounts:
       - name: hooks
         mountPath: /hooks
     volumes:
     - name: hooks
       configMap:
         name: immortalcontainers-controller

 

Publishing the webhook using a service

Since the Metacontroller must be able to resolve and connect to the webhook URL, we created a service to make the webhook callable from outside its pod.

 


apiVersion: v1
kind: Service
metadata:
 name: immortalcontainers-controller
spec:
 selector:
   app: immortalcontainers-controller
 ports:
 - port: 80

 

Lambda controller registration

The lambda controller must be registered with the Metacontroller. Registration requires creation of a CompositeController object containing the names of parent and child resources and the webhook URL.

 


apiVersion: metacontroller.k8s.io/v1alpha1
kind: CompositeController
metadata:
 name: immortalcontainers-controller
spec:
 generateSelector: true
 parentResource:
   apiVersion: immortalcontainers.flugel.it/v1alpha1
   resource: immortalcontainers
 childResources:
 - apiVersion: v1
   resource: pods
   updateStrategy:
     method: Recreate
 hooks:
   sync:
     webhook:
       url: http://immortalcontainers-controller.immortalcontainers/sync

Full reference about CompositeController objects can be found here.

 

this information tells the Metacontroller what resources to watch and what webhook to invoke when a change occurs.

 

Trying the operator

Now that the operator is installed in the cluster, we can try it. We are going to create an ImmortalContainer object to run the nginx:latest image. To do this we need to edit the file config/example-use.yaml to make it look like this:

 


apiVersion: immortalcontainer.flugel.it/v1alpha1
kind: ImmortalContainer
metadata:
 name: example-immortal-container
spec:
 image: nginx:latest

 

We then use kubectl to create the ImmortalContainer object in the cluster.

$ kubectl apply -f config/example-use.yam

 

The controller will detect the new immortal container and create a pod to run its image. Let’s verify this.

$ kubectl get pods
NAME                          READY    STATUS     RESTARTS    AGE
example-immortal-container-immortalpod   1/1     Running 0    2m

 

Next, let’s see that the pod is recreated if we delete it.

$ kubectl delete pods example-immortal-container-immortalpod
pod "example-immortal-container-immortalpod" deleted
 
$ kubectl get pods                                         
NAME                           READY   STATUS              RESTARTS     AGE
example-immortal-container-immortalpod   0/1     ContainerCreating  0      3s

 

Finally, we can edit the ImmortalContainer object to see its status.

 

$ kubectl edit immortalcontainer example-immortalcontainer

apiVersion: immortalcontainer.flugel.it/v1alpha1
kind: ImmortalContainer
metadata:
…
spec:
 image: nginx:latest
status:
 currentPod: example-immortal-container-immortalpod

 

As you can see, the operator works as expected.

 

Clean up

Finally, if you’ve been following and reproducing these steps in your cluster, you can use these commands to remove all the changes that have been made.

kubectl delete -f config/crds.yaml
kubectl delete -f https://raw.githubusercontent.com/GoogleCloudPlatform/metacontroller/master/manifests/metacontroller.yaml
kubectl delete -f https://raw.githubusercontent.com/GoogleCloudPlatform/metacontroller/master/manifests/metacontroller-rbac.yaml
kubectl delete namespace immortalcontainers
kubectl delete namespace metacontroller

 

In Conclusion

In this article we have seen how, by using Metacontroller, developers can avoid much of the complexity related to operators development. Metacontroller decouples and abstracts the tasks of events watching, state querying, and API calls, allowing developers to focus on the operators’ logic.

 

Another advantage of Metacontroller, as we have already mentioned, is that its architecture enables developers to use almost any programming language. Also, since lambda controllers don’t depend on any service, they are easy to test.

 

Finally, we think it’s worth noting that we have used only a subset of the Metacontroller features. You can learn more about the Metacontroller in the official documentation, which also provides examples in various programming languages.

 

References

https://metacontroller.app

https://github.com/flugel-it/k8s-python-operator

Prepare Your Infrastructure for the Challenges Ahead

Your clients, investors and business partners will notice the difference, from now on, you can progress with confidence.

Get Started