Implementing Kubernetes integration tests in Travis CI

JuanJo Ciarlante

JuanJo Ciarlante


Open Source and Cloud Computing passionate, Sr SRE @bitnami


Share this article

Travis CI is a very popular online continuous integration (CI) service used by free and open-source software (FOSS) projects to exercise specific testing suites on code commits. Additionally, integrating Travis CI with Github is useful for pull request validation, testing status, badges and so on.

Adding integration and/or end-to-end tests to your Kubernetes projects is very helpful but can be challenging to do in Travis CI. This is because you need an available Kubernetes cluster to run your tests.

In this article we explore the feasible ways we found to create such a cluster in Travis CI, and discuss a complete integration-tests example.

Creating your own ephemeral Kubernetes cluster in Travis CI

When developing Kubernetes applications and/or testing manifests locally, it's handy and fairly easy to run a Minikube virtual machine in your own box as your Kubernetes "cluster". It would be convenient to do the same in Travis CI, but as of October 2017, Travis CI doesn’t support creating VMs (VirtualBox, KVM), and doesn't seem likely to support it in the future.

Additionally, the virtual environment provided by Travis CI is Ubuntu 14.04 Trusty, which further complicates the availability of more recent tools required by Minikube. On the other hand, Travis CI does provide a solid Docker-ready infrastructure, which we can use to build a docker-in-docker Kubernetes cluster.

While adding integration tests to Kubeless, we found two feasible ways for deploying an ephemeral Kubernetes cluster under Travis:

  • #1 Minikube without VMs
    • PROs:
      • Well-known behavior
      • Supports StorageClass out of the box, i.e. can use Kubernetes' PVs
  • #2 Mirantis kubeadm-dind-cluster
    • PROs:
      • As the Kubernetes cluster is created with multiple "nodes" (implemented as Docker containers in the running host), it's possible to stage affinity setups and run operations like node drain, cordon, etc.

Minikube without VMs

Minikube supports running its localkube monolithic control-plane process locally (i.e. as a process in the hosting node), but doing so is not possible in Travis CI because of missing tools, specifically, the nsenter tool is needed by the locally running Minikube to "enter" into Docker containers' namespace, but it's missing (it's provided by the more recent util-linux package).

Fortunately, we can build nsenter in Travis CI using the following code from cluster-up-minikube.sh:

check_or_build_nsenter() {
    which nsenter >/dev/null && return 0
    echo "INFO: Building 'nsenter' ..."
cat <<-EOF | docker run -i --rm -v "$(pwd):/build" ubuntu:14.04 >& nsenter.build.log
        apt-get update
        apt-get install -qy git bison build-essential autopoint libtool automake autoconf gettext pkg-config
        git clone --depth 1 git://git.kernel.org/pub/scm/utils/util-linux/util-linux.git /tmp/util-linux
        cd /tmp/util-linux
        ./autogen.sh
        ./configure --without-python --disable-all-programs --enable-nsenter
        make nsenter
        cp -pfv nsenter /build
EOF
    if [ ! -f ./nsenter ]; then
        echo "ERROR: nsenter build failed, log:"
        cat nsenter.build.log
        return 1
    fi
    echo "INFO: nsenter build OK"
}

Once you have nsenter ready inside your running Travis CI environment, bringing up Minikube is achieved by running the following:

# Start minikube locally
sudo -E ${MINIKUBE_BIN} start --vm-driver=none --extra-config=apiserver.Authorization.Mode=RBAC

Note that in this particular case we enable Kubernetes RBAC mode as way to force the testing suite to include proper RBAC roles and bindings.

Mirantis kubeadm-dind-cluster

The second option we found to bring up an ephemeral Kubernetes cluster is using Mirantis' kubeadm-dind-cluster implementation (DIND stands for Docker-IN-Docker), which runs smoothly in Travis CI by running the following:

wget https://cdn.rawgit.com/Mirantis/kubeadm-dind-cluster/master/fixed/dind-cluster-v1.7.sh
chmod +x dind-cluster-v1.7.sh
./dind-cluster-v1.7.sh up

Conveniently, kubeadm-dind-cluster created Docker containers are Debian based, which, if running it locally (instead of Minikube), allows you to easily tinker inside them. For example, to debug Kubernetes services, you can apt-get install netcat inside the Docker "nodes".

Putting it all together

We've prepared the https://github.com/bitnami/kubernetes-travis/ repository with ready to use git refs you can branch from:

Full integration-tests example

The asynchronous nature of Kubernetes API (especially if used via kubectl CLI) makes it harder to wait for "settled" conditions, for example, waiting for a Pod to be in Running state.

During our improvements on Kubeless integration tests, to improve our tests' predictability and repeatability, we also created a basic shell library with some helper functions such as, k8s_wait_for_pod_ready and k8s_wait_for_pod_gone.

As a first approach to implement the integration tests in bash, we used bats: Bash Automated Testing System.

The full example at https://github.com/bitnami/kubernetes-travis/ master branch verifies a set of Kubernetes manifests that integrate the following Kubernetes objects:

  • Traefik ingress controller to map HTML content at a specific externally accesible path via ${INGRESS_IP} (which is set by the testing scripts).
  • A simple nginx Service and Deployment.
  • HTML content served by nginx, loaded via a ConfigMap.

The result of implementing this service stack using kubeadm-dind-cluster is shown in the figure below:

Full integration-tests example

To see an example of the expected output see this Travis CI log. You can also see the same example result by running the following:

$ make integration-tests
./scripts/integration-tests dind
 ✓ Verify needed kubernetes tools installed
 ✓ Deploy stack

2 tests, 0 failures
 ✓ Verify expected content via ingress
 ✓ Verify expected content via service
 ✓ Launch ab-ing pod via ingress
 ✓ Launch ab-svc pod via service
 ✓ Verify ab-ing completed requests
 ✓ Verify ab-svc completed requests

6 tests, 0 failures
INFO: requests per seconds:
* ingress: Requests per second:    3982.73 [#/sec] (mean)
* service: Requests per second:    6590.78 [#/sec] (mean)
TESTS: PASS

Starting your own project based on this repository

Try it yourself:

  • Take a look into the repository: https://github.com/bitnami/kubernetes-travis, e.g.:
    • ./scripts/cluster-up-minikube.sh
    • ./scripts/cluster-up-dind.sh
    • ./scripts/integration-tests, search for __main__
    • ./bats/integration-tests.bats, search for __main__
  • Fork it either at https://github.com UI or via CLI (as explained by github howtos).
  • Visit https://travis-ci.org/, be sure to have your account available, and your kubernetes-travis repository enabled for testing.
  • Clone it locally, make some simple edits, push it back to github.com:
git clone https://github.com/YOUR_USER/kubernetes-travis
cd kubernetes-travis
# E.g. change "index.html" content at:
vim manifests/my-nginx-configmap.yaml
git commit -m 'test some changes' -a
git push origin
  • You should now be able to see Travis CI creating an ephemeral Kubernetes cluster for you, and running your (hopefully) successful tests.

Having integration tests added to our projects has been tremendously useful - particularly in the case of Kubeless, as it allowed us to easily add new function runtimes with confidence.

Now's the time for … your «Happy and confident k8s hacking» :) !

Want to reach the next level in Kubernetes?

Contact us for a Kubernetes Training