Kubernetes operators were introduced as an implementation of the Infrastructure as Software concept. Using them you can abstract the deployment of applications and services in a Kubernetes cluster. This is the fourth in a series of articles explaining how operators work, and how they can be implemented in different languages.
In the previous article we built a simple Kubernetes operator, named immortal containers, using Operator SDK. As you might remember, this operator provides the users with a way to define containers that should run forever — that is, whenever a container terminates for any reason, it will be restarted.
We also deployed the operator to a cluster and verified its correct function. Until now each of our tests has required manual intervention for the management of Kubernetes objects and verification of operator function.
In this article we will verify function through the implementation of automatic unit and end-to-end tests. Since information about the advantages of automatic testing is plentiful, we will not discuss that topic in detail.
As in the last issue, we are going to use the Go programming language and Operator SDK. So, we assume you have Go (version at least 1.11) installed in your computer. You will also need access to a Kubernetes cluster to try the operator; you can use minikube to create a development cluster.
The complete source code for the operator being built through this article can be found at: https://github.com/flugel-it/k8s-go-operator-sdk-operator
Unit testing is about testing units, or components, individually, in the absence of other components. A unit can be a function, method, or module. The goal of unit test is to assess that an individual component works as expected.
Unit testing allows detection of problems early, even before the remainder of the program is complete. Discovery of errors during unit test, before other components are introduced, also allows easier diagnosis and resolution of those problems.
To show how unit testing can be used in the development of operators, we are going to study a unit test for the Reconcile function. We chose that function because it’s the heart of the operator’s controller, so its integrity is crucial to the proper function of our operator.
In order to verify the Reconcile function, our test case must verify that, when an ImmortalContainer object must be (re)created, the function successfully creates the associated pod and correctly sets the status fields, CurrentPod and StartTimes. The code for the test relies on go test and Client-go’s Fake Client package. Fake Client allows developers to simulate the existence of Kubernetes objects and the calls to Kubernetes API without having to use a real cluster.
The proposed test case executes these steps:
To implement the test we created the file pkg/controller/immortalcontainer/immortalcontainer_controller_test.go. The following block shows the source code of the test. We added comments to make it easier to match each code section with the steps described above.
package immortalcontainer
import (
"context"
"testing"
immv1alpha1 "github.com/flugel-it/k8s-go-operator-sdk-operator/pkg/apis/immortalcontainer/v1alpha1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/kubernetes/scheme"
"sigs.k8s.io/controller-runtime/pkg/client/fake"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
)
func TestImmortalContainerControllerPodCreate(t *testing.T) {
var (
name = "example"
image = "nginx:latest"
namespace = "testnamespace"
)
// --------------------------------------------------
// 1- Create an ImmortalContainer object
immortalContainer := &immv1alpha1.ImmortalContainer{
ObjectMeta: metav1.ObjectMeta{
Name: name,
Namespace: namespace,
},
Spec: immv1alpha1.ImmortalContainerSpec{
Image: image,
},
}
// Register the object in the fake client.
objs := []runtime.Object{
immortalContainer,
}
// --------------------------------------------------
// --------------------------------------------------
// 2- Create a fake Kubernetes API client
// Register operator types with the runtime scheme.
s := scheme.Scheme
s.AddKnownTypes(immv1alpha1.SchemeGroupVersion, immortalContainer)
// Create a fake client to mock API calls.
cl := fake.NewFakeClient(objs...)
// Create a ReconcileImmortalContainer object with the scheme and fake client.
// --------------------------------------------------
// --------------------------------------------------
// 3- Create the context for the controller
r := &ReconcileImmortalContainer{client: cl, scheme: s}
// --------------------------------------------------
// --------------------------------------------------
// 4- Mock the event processing
// Create a request, that should be in the work queue.
req := reconcile.Request{
NamespacedName: types.NamespacedName{
Name: name,
Namespace: namespace,
},
}
// Process the request
res, err := r.Reconcile(req)
if err != nil {
t.Fatalf("reconcile: (%v)", err)
}
if res != (reconcile.Result{}) {
t.Error("reconcile did not return an empty Result")
}
// --------------------------------------------------
// --------------------------------------------------
// 5- Calculate how the expected ImmortalContainer’s pod should look like
expectedPod := newPodForImmortalContainer(immortalContainer)
// --------------------------------------------------
// --------------------------------------------------
// 6- Check that a pod matching the expected one was created
pod := &corev1.Pod{}
err = cl.Get(context.TODO(), types.NamespacedName{Name: expectedPod.Name, Namespace: expectedPod.Namespace}, pod)
if err != nil {
t.Fatalf("get pod: (%v)", err)
}
// --------------------------------------------------
// --------------------------------------------------
// 7- Check status is correctly updated
updatedImmortalContainer := &immv1alpha1.ImmortalContainer{}
err = cl.Get(context.TODO(), types.NamespacedName{Name: immortalContainer.Name, Namespace: immortalContainer.Namespace}, updatedImmortalContainer)
if err != nil {
t.Fatalf("get immortal container: (%v)", err)
}
if updatedImmortalContainer.Status.StartTimes != 1 {
t.Errorf("incorrect immortal container startTimes: (%v)", updatedImmortalContainer.Status.StartTimes)
}
if updatedImmortalContainer.Status.CurrentPod != expectedPod.Name {
t.Errorf("incorrect immortal container currentPod: (%v)", updatedImmortalContainer.Status.CurrentPod)
}
// --------------------------------------------------
}
Before running the test, it’s necessary to run the “dep ensure” command to install any missing dependencies. Then the test can be run using “go test”.
$ go test ./pkg/controller/immortalcontainer
ok github.com/flugel-it/k8s-go-operator-sdk-operator/pkg/controller/immortalcontainer 0.051sRunning “go test” reports back the status of the test and how long it took to run it.
Tests like this one help to catch problems early, even before trying the operator on a real cluster. You should also run these tests after making changes to the operator’s controller to avoid introducing errors.
Besides being sure each component functions properly, we want to assure correct function of the operator as a whole. Finally, we want also to be sure that the operator runs correctly when working on a real Kubernetes cluster.
This is why we need end-to-end (e2e for short) tests. This kind of test works by instantiating the operator on a cluster, performing some actions on cluster objects, and then validating real outcomes against the expected outcomes.
For example, we implemented a test for the immortal containers operator that works as follows:
Note that, unlike our unit test, which was implemented using the fake client, this test operates on a real cluster.
To create the test we followed the writing-e2e-tests guide from Operator SDK and based the implementation on a test for the Memcached operator, memcached_test.go. The core of our implementation can be found in test/e2e/immortalcontainer_test.go. Here you can see an excerpt from the source code. As before on our unit test, we have added comments to make it easier to match each code section with the steps described above.
// This is the main function for this test case. It initializes the framework we use later to access Kubernetes API from the test.
func TestImmortalContainer(t *testing.T) {
immortalContainerList := &operator.ImmortalContainerList{
TypeMeta: metav1.TypeMeta{
Kind: "ImmortalContainer",
APIVersion: "immortalcontainer.flugel.it/v1alpha1",
},
}
err := framework.AddToFrameworkScheme(apis.AddToScheme, immortalContainerList)
if err != nil {
t.Fatalf("failed to add custom resource scheme to framework: %v", err)
}
// run subtests
t.Run("immortalcontainer-group", func(t *testing.T) {
t.Run("Cluster", ImmortalContainerCluster)
})
}
// --------------------------------------------------
// 1- Deploy the operator to the cluster, wait until it ready and then
// call immortalContainerCreateTest, the real test function.
func ImmortalContainerCluster(t *testing.T) {
t.Parallel()
ctx := framework.NewTestCtx(t)
defer ctx.Cleanup()
err := ctx.InitializeClusterResources(&framework.CleanupOptions{TestContext: ctx, Timeout: cleanupTimeout, RetryInterval: cleanupRetryInterval})
if err != nil {
t.Fatalf("failed to initialize cluster resources: %v", err)
}
t.Log("Initialized cluster resources")
namespace, err := ctx.GetNamespace()
if err != nil {
t.Fatal(err)
}
// get global framework variables
f := framework.Global
// wait for operator to be ready
err = e2eutil.WaitForDeployment(t, f.KubeClient, namespace, "immortalcontainer-operator", 1, retryInterval, timeout)
if err != nil {
t.Fatal(err)
}
if err = immortalContainerCreateTest(t, f, ctx); err != nil {
t.Fatal(err)
}
}
// --------------------------------------------------
func immortalContainerCreateTest(t *testing.T, f *framework.Framework, ctx *framework.TestCtx) error {
namespace, err := ctx.GetNamespace()
if err != nil {
return fmt.Errorf("could not get namespace: %v", err)
}
// --------------------------------------------------
// 2- Create an ImmortalContainer object
immortalContainer := &operator.ImmortalContainer{
TypeMeta: metav1.TypeMeta{
Kind: "ImmortalContainer",
APIVersion: "immortalcontainer.flugel.it/v1alpha1",
},
ObjectMeta: metav1.ObjectMeta{
Name: "example",
Namespace: namespace,
},
Spec: operator.ImmortalContainerSpec{
Image: "nginx:latest",
},
}
// use TestCtx's create helper to create the object and add a cleanup function for the new object
err = f.Client.Create(goctx.TODO(), immortalContainer, &framework.CleanupOptions{TestContext: ctx, Timeout: cleanupTimeout, RetryInterval: cleanupRetryInterval})
if err != nil {
return err
}
// --------------------------------------------------
// --------------------------------------------------
// 3- Check the pod is created
err = WaitForPod(t, f.Client, namespace, "example-immortalpod", nil, retryInterval, timeout)
if err != nil {
return err
}
// --------------------------------------------------
// --------------------------------------------------
// 4- Delete the pod
pod := &corev1.Pod{}
err = f.Client.Get(goctx.TODO(), types.NamespacedName{Name: "example-immortalpod", Namespace: namespace}, pod)
if err != nil {
return err
}
pod1UID := &pod.UID
err = f.Client.Delete(goctx.TODO(), pod)
if err != nil {
return err
}
// --------------------------------------------------
// --------------------------------------------------
// 5- wait for pod recreation
// We wait for a pod with the same name, but different UID than the
// previously created one.
err = WaitForPod(t, f.Client, namespace, "example-immortalpod", pod1UID, retryInterval, timeout)
if err != nil {
return err
}
// --------------------------------------------------
return nil
}
The e2e tests require a Kubernetes cluster They require, too, that an image for your operator be built and made accessible to your cluster.
The e2e test’s main program runs in the developer’s computer (It’s also possible to run the test’s main program inside the cluster. This and other options for running the tests are fully described in the Operator SDK test guides). It deploys the operator’s CRDs and controller to the cluster. Then it reads, writes and deletes objects to and from cluster resources. All communications between the test process and the cluster is done via Kubernetes API.
With e2e as for the unit test, it’s necessary to run the dep ensure command to install any missing dependencies.
We use the following command to run our e2e test:
$ operator-sdk test local ./test/e2e --image flugelit/immortalcontainer-operator:dev
INFO[0000] Testing operator locally.
ok github.com/flugel-it/k8s-go-operator-sdk-operator/test/e2e 31.443s
INFO[0037] Local operator test successfully completed.
Most of the common principles and practices of software testing can be applied also to the testing of operators. In this article we’ve gone through specific ways of testing operators developed using Operator SDK.
Keep in mind that unit and end-to-end tests complement each other. Unit tests are useful for individual testing of isolated components, while e2e tests assure proper operator function in real-world scenarios.
We hope that these last two articles have helped you understand better how to create and test Kubernetes operators in Golang using Operator SDK. In future issues we will see how to implement operators in Python and other programming languages.
https://github.com/operator-framework/operator-sdk https://github.com/flugel-it/k8s-go-operator-sdk-operator
https://github.com/operator-framework/operator-sdk/blob/master/doc/user/unit-testing.md
2018, Cryptoland Theme by Artureanec - Ninetheme