Practicing what we preach, benefiting from a move to Kubernetes

During 2018, Bitnami has been changing how we approach and utilize infrastructure to deliver our services, moving from a "single server per project" approach to one where we use Kubernetes. Doing so not only makes management easier, but also allows us to respond more quickly to infrastructure vulnerabilities. It also gives us the ability to utilize Bitnami Kubernetes Production Runtime (BKPR) and the production features such as logging, HTTPS ingress, and monitoring.

This change also shows how Bitnami, known for thought leadership and open source projects around Kubernetes and containerized applications, is utilizing the technology we contribute to and promote to improve our own internal operations.

In this blog post, I will explain in detail why and how Bitnami adopted the Kubernetes technology, and what you should consider before moving your projects to Kubernetes.

From dedicated servers to a centralized one

Issues of having all projects spread over several machines

As many companies probably do, at Bitnami we used a dedicated server for each project. Our Engineering Portal, for example, was running in a single VM.

But we realized that this approach had some drawbacks including:

  • Having different servers for every services increased the complexity of their management.
  • When a security issue came up, we had to apply the patches and upgrade all the services on each server.
  • The logs of the different servers were not centralized. To have an idea of the events happened on each server, we had to access each one individually.
  • We have some projects that are static sites, such as the Engineering Portal and our internal handbook, and having a dedicated server for these kinds of projects didn't make much sense.

It begged the question: since we already had Kubernetes clusters running on our servers and consuming resources, why not unify our infrastructure under a single technology?

The solution: unify the infrastructure under the most advanced technology

In addition to the obvious advantage of needing only to control a single infrastructure instead of having our projects spread over multiple machines, our Kubernetes clusters had already installed the collection of services included in BKPR.

This allowed us to reduce the time spent in obtaining all the site certificates for our products because cert-manager, included in BKPR, provides them. It also get the benefit of indexing, visualization, and having the possibility to query cluster logs thanks to the integration of Fluentd and Kibana with the cluster. (Say goodbye to backup for every single server!).

And Prometheus and Grafana, also part of BKPR, allowed us to monitor all the data and network cluster usage from a visual interface. To have more visibility, we are using the Bitnami NGINX container to expose more statistics using the built in VTS module.

Here is a summary of the benefits we obtained by moving our projects to Kubernetes:

  • Simplified the management of the different projects / servers we had.
  • Reduced the complexity of managing security issues: our projects use the Bitnami containers, so now we only need to update the base image when we upgrade a service.
  • Avoid the need to backup every server. All the backups of the logs are now managed by the cluster.
  • Reused already allocated resources.
  • Gained a lot of new features thanks to BKPR.

What should you consider before moving your projects to Kubernetes?

If you are thinking about moving your projects and resources to Kubernetes, you should consider three essential things: how Kubernetes handles services, the network requirements, and server resource consumption.

  • How Kubernetes handles services: In Kubernetes, it is very important to imagine a pod as a single service. Different services get managed as separate entities. So if you have several services running on your server today, with Kubernetes you will need to manage them separately and migrate them individually as single entities.
  • The network requirements: Your Kubernetes cluster must be able to handle the resource and network requirements of the service. This is not a trivial consideration. For example, if you want to move a website that receives a traffic of 500 req/s, your cluster must be able to manage this number of connections. Otherwise, your site will be vulnerable to down-time.
  • Server resource consumption: Having all projects in a single server implies the possibility of having an unresponsive service consuming all the resources (CPU, RAM, Networking), which can affect all projects you have running in the same node. To avoid this, you should add limits to the resources that each pod ("project server") can allocate in the cluster. You can also add limits per namespace.

From multiple servers to services in a Kubernetes cluster

In this section, I will show how Bitnami moved all its projects from multiple servers to services in a single Kubernetes cluster. These are the steps we followed, feel free to use them as a checklist for adopting Kubernetes as the main infrastructure for your projects!

  1. Select the service to move (based on the scope of your project, see the previous section).
  2. Make a list of the requirements for the chosen service:
    • How many different processes are needed to run: we used one container per process.
    • What are the networking requirements: Does the service need a public URL? Learn about Ingress in Kubernetes.
    • What are the storage and data requirements (if this applies to the service):
      • Define the required disk size.
      • Create the databases in external providers.
  3. Select the tools you will use. We selected Kubecfg and Jsonnet as the way to define the manifests of our services. However, there are more options you can use such as Helm.
    • Create the manifests and test them first on Minikube and later in a development cluster.
    • Prepare the deployment automation with Jenkins.
    • Test your changes in "Staging".
    • Deploy to production.
    • Switch the DNS to point to the Kubernetes cluster.
    • Enjoy your newly created Kubernetes cluster!!

Further Interesting readings

If you want to learn more about Kubernetes and how to start moving your servers to a Kubernetes cluster, the following articles might be of interest: