An Example of Real Kubernetes: Bitnami

Ok, you’ve set up your Kubernetes cluster and run the "hello world" example. Now what?

When looking around the Internet recently, we at Bitnami realised there are many great Kubernetes examples in documentation and blog posts, but all of these are deliberately "simple" in order to illustrate their respective points. Once you graduate from the beginner documentation however, there are suddenly few "real" operational examples or advice to draw from. So, github.com/bitnami/kube-manifests is a sample of the real production configuration, tools and workflow that we use internally at Bitnami to manage our own Kubernetes clusters.

Our Situation

The globally distributed SRE team at Bitnami operates 3 internal Kubernetes clusters, all running on private AWS infrastructure:

  • dev: developer experiments or alpha-stage software
  • int: internal production services, eg: our bugtracker and CI/CD machinery
  • web: externally-visible production services, eg: our various web properties

The Kubernetes clusters themselves were all created and are maintained using kops, and we keep the Kubernetes versions roughly synchronised. The 3 clusters are used for slightly different purposes and the AWS firewall and Kubernetes RBAC rules vary accordingly - but otherwise the cluster infrastructure is broadly similar.

Managing the configuration of these services poses a few high-level challenges:

  • How do we coordinate changes across multiple people and timezones?
  • How do we ensure common deployment patterns/policies are used across services?
  • How do we ensure a common service is deployed consistently across clusters?

(Words like service and deployment have specific meanings in Kubernetes. I’m going to use "Service" (capitalised) to refer to the Kubernetes Service resource, and "service" (lowercase) to refer to the usual generic English/computing term.)

A Brief Tour of Our Cluster Features

The included jsonnet config files are taken straight from our current live repository, with deliberately very little cleanup. You'll find comments from past legacy and in-progress experiments. We removed the RBAC rules and configs for Bitnami production software.

A lot of the workflow automation is driven by Jenkins running in the cluster.

All container logs are collected in a classic fluentd/elasticsearch/kibana stack.

Our Ingress resources are implemented with nginx-ingress and internal AWS ELBs. We automatically generate SSL certificates using letsencrypt via kube-cert-manager using DNS challenges because our ELBs are not available for external requests. We also have a DNS wildcard entry *.k.dev.bitnami.net pointing to the Ingress ELBs. All this allows our developers to just create a suitable Ingress rule pointing at their service, and then they will automatically get a hostname, SSL certificate, and HTTP->HTTPS redirect.

Our container and legacy VM services are monitored using prometheus, and can be self-configured by adding appropriate annotations or tags. Importantly jenkins, elasticsearch, and nginx mentioned above are already configured to export prometheus metrics, so services get HTTP-level request/status-code/etc statistics without having to do any additional steps.

The Big Picture

Drawing from previous experience, we chose to follow a declarative Infrastructure as Code approach. This involves capturing as much as possible about the configuration of the clusters as files in version control (git) and using our usual familiar "code" workflow of reviews and unittests to also manage our infrastructure.

Workflow

Putting our "desired state" in git is great because now our team can talk about versions of our infrastructure and follow a version through a develop, review, test, upgrade, rollback change cycle. All this reduces our team coordination complexity from O(n!) interactions between individual team members, to O(n) interactions between team and the central repository-of-truth.

Configuration: Patterns Everywhere!

Kubernetes natively describes everything in JSON (or the YAML equivalent).

Here's an example in YAML. (It's ok, I'm trying to make a high-level point - you can just skip over the detail):

apiVersion: v1
kind: Service
metadata:
 labels:
  name: proxy
 name: proxy
 namespace: webcache
spec:
 ports:
 - port: 80
   targetPort: proxy
 selector:
  name: proxy
 type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
  name: proxy
 name: proxy
 namespace: webcache
spec:
 minReadySeconds: 30
 replicas: 1
 revisionHistoryLimit: 10
 strategy:
   rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0
   type: RollingUpdate
 template:
   metadata:
     labels:
      name: proxy
   spec:
     containers:
     - name: squid
       # skipped, for clarity

Kubernetes resource definitions have a lot of repetitive boilerplate, and the same values appear repeatedly within and across closely related resources. We wanted a tool that allowed us to express the patterns inherent in Kubernetes resource definitions, and also in our own use of resources across our infrastructure. The tool has to expand that into JSON/YAML for Kubernetes. After looking at various options, we quickly settled on jsonnet for a number of reasons:

  • Declarative, side-effect-free language that reduces surprises when composed in complex setups.
  • Real language that natively generates JSON, not a limited domain-specific-language that pieces together fragments of opaque text.
  • Supports multiple input files and has a strong "merge" operation, allowing us to build up libraries of common templates.
  • Looks like a similar language we had experience with when working with a similar problem elsewhere (Google).
  • Downside: not widely known, and the tool is not already packaged for major distros.

For reference, using our kube.libsonnet library, the jsonnet syntax for the above looks like the following. Note that the "boring" parts are handled automatically by the library, highlighting the high-level intent, the relationship between Service and Deployment, and any exceptions to the usual base template (eg: explicitly setting the namespace).

{
  namespace: "webcache",

  squid_service: kube.Service("proxy") {
    metadata+: { namespace: $.namespace },
    target_pod: $.squid.spec.template,
    port: 80,
  },

  squid: kube.Deployment("proxy") {
    metadata+: { namespace: $.namespace },
    spec+: {
      template+: {
        spec+: {
          containers_+: {
            squid: kube.Container("squid") {
              // skipped, for clarity
            },
          },
        },
      },
    },
  },  // Kubernetes structures are deep :)
}

I won't re-explain jsonnet syntax here, go read the jsonnet tutorial when you're ready. The only thing I want to highlight is that jsonnet has a merge operation on JSON objects, which is a bit like python's dict.update except it happens before other expressions are expanded. This merge operation is super useful and uses "+", which is why you see it a lot above.

It is also important to realise that there are always exceptions - every configuration option was added to Kubernetes to satisfy a legitimate use case. Consequently we chose to pass around the full Kubernetes resource objects (with a few optional helpers to make it more "jsonnet native") rather than some "simpler" intermediate structure. This means we can just point to the standard Kubernetes documentation rather than write our own, and it allows any Kubernetes option to be overridden at any point in the template stack. Win!

Expressing Similarity

Our major "axes of similarity" are:

  • Common components running in multiple clusters
  • Common deployment patterns used across components
Common patterns

Consequently, most of our templates are 3 layers: generic base -> specific software -> specific deployment of software. Our directory structure matches this too:

  • lib/ - Base libraries that express common deployment patterns
  • common/ - Build on the base libraries and describe tightly-coupled resource declarations for individual software stacks
  • $cluster_name/ - Per-namespace files that assemble stacks from common/ and apply any final cluster-unique overrides

As a side note, unlike many simpler workflows we use explicit Kubernetes namespace declarations in all our resource definitions. This is because our team is managing many different software stacks, and we can't rely on some magic external environment to set the namespace correctly. Capture as much as possible in your version control!

Tools and Workflow

The simplicity of the tools referred to below is a deliberate design choice and demonstrates the power of the existing Kubernetes architecture. When infrastructure breaks, it's good to be able to piece it back together again from first principles.

  1. Humans edit jsonnet files in their favourite editor, as above. In practice, there's usually a tight "edit, run jsonnet and push to dev cluster, see result" loop for changes that involve some experimentation. The kubecfg.sh script makes this easy by expanding jsonnet on the fly. Our contract with the dev cluster is basically "be nice to others, and any changes may be reset to what's in git at any point."

  2. The jsonnet files get expanded into a tree of regular JSON files using a simple jsonnet-in-docker script invoked by a Makefile. We found the JSON files were very useful for the author and reviewers to know exactly what their change affects, particularly when modifying base templates.

  3. Various unittests are run against the jsonnet and JSON files. Our jsonnet code includes a number of assert statements that are verified during JSON generation. We additionally check for conformance against a jsonnet code style, the generated JSON files are actually accurate, the JSON meets the Kubernetes jsonschema, and that all resources declare an explicit namespace. Importantly, all these checks are safe to perform at any time, so we run these against all our github pull-requests before merging from jenkins.

  4. Change is reviewed by another team member using usual github code review. At this point they know that the various automated tests pass so they can focus on high-level correctness rather than syntax. When happy with the proposed change, they hit "approve" and merge the change.

  5. After merging, jenkins automatically deploys the change by running deploy.sh against each cluster. The existing Deployment readiness checks and rollout strategies prevent disastrous changes from making it through and we have continuous monitoring (via prometheus) to tell us about anything that is breaking. Importantly, the rollouts are slow enough that our monitoring can give us time to react and freeze the broken rollout via the regular kubectl rollout pause and undo commands. We have full history so can revert by rolling back the offending git change.

Lessons Learned and Future Work

It works well, but it's not all ponies and ice-cream. Jsonnet is a real (but small) programming language, and should be approached as such. You really do need to skim a jsonnet tutorial before just jumping in, and casual viewers tend to be put off by all the pluses and close-braces.

Our base templates currently express the resulting resource names directly, which makes it hard to have more than one instance of the same stack in the same Kubernetes namespace. It's easy enough to address that in the jsonnet libraries, but we don't do that yet

Overall, this is a toolbox and a process, not a shrink-wrap product. The upside is that you can tailor all this to fit your situation; the downside is that I can't just give you two generic command lines to cut and paste and be done.

The good news is that there is an active discussion going on within the Kubernetes sig-apps group and we're building on our collective experiences. I'm excited for the future improvements in tools and configuration now that our Kubernetes community is clearly passing into the post-demo maturity stage.

Want to reach the next level in Kubernetes?