Sealed Secrets: Protecting your passwords before they reach Kubernetes

We like "Infrastructure as Code" at Bitnami. We manage the configuration of all our Kubernetes clusters in a shared git repository, run automated tests, perform peer code reviews, automatically deploy changes to the cluster from main branch, and life is good.

Except for Secrets.

Sensitive material like database passwords or slack tokens can't be added to a public repository, and so we have an entirely separate process for managing Kubernetes Secrets. This means that secrets require special-cased cluster access to update, are not part of the same staged coherent change process, and can't be rolled back.

Until now!

Introducing "Sealed Secrets"

Sealed Secrets are a "one-way" encrypted Secret that can be created by anyone, but can only be decrypted by the controller running in the target cluster. The Sealed Secret is safe to share publicly, upload to git repositories, post to twitter, etc. Once the SealedSecret is safely uploaded to the target Kubernetes cluster, the sealed secrets controller will decrypt it and recover the original Secret.

Life of a SealedSecret

The SealedSecrets implementation consists of two components:

  • A controller that runs in-cluster, and implements a new SealedSecret Kubernetes API object via the "third party resource" mechanism.
  • A kubeseal command line tool that encrypts a regular Kubernetes Secret object (as YAML or JSON) into a SealedSecret.

Once decrypted by the controller, the enclosed Secret can be used exactly like a regular K8s Secret (it is a regular K8s Secret at this point!). If the SealedSecret object is deleted, the controller will garbage collect the generated Secret.

Usage

See the website for controller and kubeseal installation instructions.

The kubeseal tool reads the JSON/YAML representation of a Secret on stdin, and produces the equivalent (encrypted) SealedSecret on stdout. A Secret can be created in many ways, but one of the easiest is using kubectl create secret --dry-run, as shown in the following example. Note again that the kubectl --dry-run just creates a local file and doesn't upload anything to the cluster.

# Creates an example secret and encrypt it immediately:
kubectl create secret generic --dry-run -output json \
  mysecret  --from-literal=password=supersekret |
  kubeseal > mysealedsecret.json

# Safe to upload mysealedsecret.json to git, etc.

# Eventually upload mysealedsecret to cluster:
kubectl create -f mysealedsecret.json

# The original secret now exists in the cluster, like magic!
kubectl get secret mysecret

An important point to be aware of is that the SealedSecret and Secret must have the same namespace/name, and the controller will refuse to decrypt a SealedSecret where this is not the case. This is deliberate, to prevent another user of your cluster being able to upload your SealedSecret under their namespace and getting the controller to decrypt it for them.

Note that the user is now creating a SealedSecret object and any RBAC policies around Secrets might need to be adjusted accordingly.

Any labels, annotations, etc in the original Secret will be restored in the generated Secret but are not automatically reflected in the SealedSecret.

Automation Friendly

SealedSecrets and the kubeseal tool are designed to easily fit into automated workflows. Once converted into a SealedSecret, not even the original user will be able to retrieve the original Secret. kubeseal can also be run offline, without access to the cluster - it just needs a copy of the public key available on disk somewhere.

These features allow low-privileged automated tasks to generate new random secrets and feed them into a change management workflow without requiring any elevated access to existing secrets. Using this it is possible to set up workflows that encourage frequent rotation and replacement of secrets rather than managing additional backup copies of plain-text secrets.

It is important to be aware that SealedSecrets do not authenticate the user, by design. This means that any user with a copy of the right public key is able to create an offline SealedSecret for your cluster, but it is up to your regular change management processes, cluster RBAC rules, etc to make sure that only the intended SealedSecret makes it to your cluster. SealedSecrets are intentionally no different to all your other cluster configuration in this regard.

Frequently Asked Questions

How does this compare to the K8s 1.7 encryption-at-rest feature?

Kubernetes 1.7 introduces a new alpha feature that will encrypt secrets stored in etcd. This protects your secrets once they are inside K8s, but doesn't help during your workflow leading up to the K8s API boundary.

SealedSecrets are the exact complement to this - a SealedSecret is encrypted all the way up to (and inside) the K8s cluster boundary. Once the controller decrypts it and produces the original Secret, then the Secret becomes the responsibility of Kubernetes to protect. Without the Kubernetes 1.7 feature, the Secret was exposed to attackers with physical access to the cluster after decryption.

TL;DR:

  • The k8s 1.7 feature protects you if someone steals the hard disk from one of your etcd servers.
  • SealedSecrets protects you during all the workflow stages before that.

So How Does It Work?

SealedSecrets are a straightforward application of asymmetric (public key) cryptography. Public key cryptography involves a tightly-linked pair of keys (called "public" and "private"), and anything encrypted with one can only be decrypted by the other.

On first startup, the controller generates a public/private RSA key pair and persists it in a regular Kubernetes Secret. The controller publishes the public key to its log and over HTTP at /v1/cert.pem. The public key is not secret, although you do need to ensure you are using the correct public key.

RSA can only encrypt small amounts of data, so SealedSecrets use a typical two-stage combination of symmetric and asymmetric encryption. To encrypt, kubeseal JSON-encodes the Secret, and symmetrically encrypts it using AES-GCM with a randomly-generated single-use session key. The session key is then asymmetrically encrypted with the controller's public key using RSA-OAEP, and the original Secret's namespace/name as the OAEP input parameter (aka label). The final output is: 2 byte encrypted session key length || encrypted session key || encrypted Secret.

During decryption, the SealedSecret controller uses the namespace/name from the SealedSecret as the OAEP input parameter, ensuring that the SealedSecret and Secret are tied to the same namespace and name. It then performs the reverse steps: Use the controller's private key to decrypt the single-use session key, then use the session key to decrypt the JSON-encoded Secret. Finally, the Secret is created under the namespace/name of the SealedSecret, again enforcing this fixed relationship.

Want to reach the next level in Kubernetes?