How to create a Jenkins X workflow with BKPR

Published on April 26, 2019
jenkins
CI/CD
BKPR
Lets Encrypt
Sameer Naik

Sameer Naik


Share this article

Jenkins X is an open source continuous integration and continuous delivery platform that utilizes the power of Kubernetes to run Jenkins build agents. It enables application developers to harness the power of Kubernetes without having to dive deep into the intricacies of Kubernetes. Jenkins X provides the complete infrastructure for continuous delivery and for promoting applications to staging and production environments running in Kubernetes.

Bitnami Kubernetes Production Runtime (BKPR) is a curated collection of the services needed to deploy on top of your Kubernetes cluster to enable logging, monitoring, certificate management, automatic discovery of Kubernetes resources via public DNS servers, and other common infrastructure needs. The services are ready-to-run and pre-integrated with each other, so they work out of the box.

The post is targeted at application developers and it is split in two parts. In the first part you will be walked through the process of installing Jenkins X on a BKPR enabled Kubernetes cluster, followed by securing access to the Jenkins X services with TLS certificates.

In the second part a sample application will be created and you will learn about securing the application with TLS certificates and using OAuth based authentication to control access to the preview and staging environment.

Why BKPR?

The services installed in the Kubernetes cluster by Jenkins X are publicly accessible by external users over the plain HTTP protocol making them vulnerable to attackers. You will want to secure access to these services using TLS certificates.

Additionally you may want users to authenticate themselves to gain access to applications deployed in the Jenkins X preview and staging environments.

The pre-integrated BKPR components will enable you to easily provision LetsEncrypt TLS certificates to secure these services while using OAuth to make the preview and staging environments accessible to only users of the OAuth domain.

Assumptions

The post does not go into the specific details of Kubernetes or Jenkins X and assumes you have a basic understanding of these platforms. Additionally, as Jenkins X utilizes Helm charts for deploying workloads on Kubernetes, a basic understanding of Helm is assumed.

Before you install Jenkins X to your Kubernetes cluster, make sure you have already installed the Bitnami Kubernetes Production Runtime (BKPR) to the cluster. Please follow the BKPR installation guide if you already haven’t done so.

Prerequisites

Installing Jenkins X

Create a GitHub access token

Before deploying Jenkins X to your BKPR-enabled Kubernetes cluster, let's create a GitHub personal access token for the Jenkins X CLI. The access token will allow the CLI to manage your GitHub repositories and setup the CI/CD integration, among other things.

To create a new personal access token with the required permissions: follow this link, give the token a description, and click the Generate token button.

On the next screen your secret access token will be displayed, copy it to a safe place before continuing:

GitHub Personal Access Token

Deploy Jenkins X

You can now install Jenkins X to the Kubernetes cluster using the jx install command as shown below. Please update the values of the --provider, --domain, --git-username and --git-api-token flags before executing the command.

TIP: Get the complete list of available command line flags by executing jx install --help
# jx install \
  --provider="gke" \
  --domain="my-domain.com" \
  --git-username="xxbn" \
  --git-api-token="ccfe0561b8d1a0aa57f540eca4eff3ad3a8d51ef" \
  --skip-ingress="true" \
  --ingress-deployment="nginx-ingress-controller" \
  --ingress-namespace="kubeprod" \
  --ingress-service="nginx-ingress"

You will be prompted to select the Jenkins installation type and default build pack, choose Static Jenkins Server and Jenkinsfile and Kubernetes Workloads: Automated CI+CD with GitOps Promotion respectively as shown in the response snippet below:

? Do you wish to use xxsbn as the local Git user for GitHub server: Yes
? Do you wish to use GitHub as the pipelines Git server: Yes
? Select Jenkins installation type: Static Jenkins Server and Jenkinsfiles
? Pick default workload build pack: Kubernetes Workloads: Automated CI+CD with GitOps Promotion
? Select the organization where you want to create the environment repository: xxbn

Upon successful execution, the URL of the Jenkins instance, password for the Jenkins admin user, GitHub repositories created, etc. will be displayed in the output. Make a note of these details and ensure you copy the password for the admin user to a safe place before continuing.

Configuring TLS support

The jx install command installs externally visible services in your cluster. By default, these services are configured to be accessible over the plain HTTP protocol and are therefore vulnerable to network snooping attacks.

# jx open
Name                      URL
jenkins                   http://jenkins.jx.my-domain.com
jenkins-x-chartmuseum     http://chartmuseum.jx.my-domain.com
jenkins-x-docker-registry http://docker-registry.jx.my-domain.com
jenkins-x-monocular-api   http://monocular.jx.my-domain.com
jenkins-x-monocular-ui    http://monocular.jx.my-domain.com
nexus                     http://nexus.jx.my-domain.com

As BKPR already provides the infrastructure to perform automatic LetsEncrypt TLS certificate provisioning and management, we can use this infrastructure to secure these services with TLS.

The Ingress rules in the jx namespace should be upgraded to enable TLS support. During the upgrade you will be prompted to provide a Jenkins API access token. Generate one using the steps below:

Add API Token to Jenkins

  1. Execute jx open jenkins to launch the Jenkins UI in your web browser
  2. Login with the admin user credentials from the previous step
  3. Click on the "Administration" link displayed in the top-right corner of the page
  4. On the "Administration" page, click the "admin" link displayed in the top-right corner
  5. Select the "Configure" link from the left navigation menu
  6. Click the "Add new Token" button and then click on "Generate"
  7. Save the generated token and click the "Save" button

Upgrade the Ingress rules in the jx namespace using the following command:

jx upgrade ingress --namespaces=jx --skip-certmanager=true --wait-for-certs=false

The command will ask you to provide your input for configuring the TLS certificate request. Make sure you should select Ingress for the Expose type and "{{.Service}}.{{.Namespace}}.{{.Domain}}" for the UrlTemplate as shown in the response snippet below. The Jenkins API access token generated in the previous step should also be provided here.

? Existing ingress rules found in namespaces [jx] namespace. Confirm to delete and recreate them Yes
? Expose type Ingress
? Domain: my-domain.com
? If your network is publicly available would you like to enable cluster wide TLS? Yes
? Use LetsEncrypt staging or production? production
? Email address to register with LetsEncrypt: admin@my-domain.com
? UrlTemplate (press <Enter> to keep the current value): "{{.Service}}.{{.Namespace}}.{{.Domain}}"
? Using config values {}, ok? Yes
? Jenkins user name: admin
? API Token: **********************************
? Do you want to update all existing webhooks? (Y/n) Yes

Note that it generally takes a few minutes for the TLS certificates to be provisioned. You can execute the following command to check the status of the TLS certificate requests in the jx namespace:

# kubectl --namespace jx get certificates
NAME                  READY   SECRET                AGE
tls-chartmuseum       True    tls-chartmuseum       1m
tls-docker-registry   True    tls-docker-registry   1m
tls-jenkins           True    tls-jenkins           1m
tls-monocular         True    tls-monocular         1m
tls-nexus             True    tls-nexus             1m

After the certificates have been successfully provisioned, all the Jenkins X services will now be secured and accessible over the HTTPS protocol as can be seen below.

# jx open
Name                      URL
jenkins                   https://jenkins.jx.sameer-jxapp.tests.fuloi.org
jenkins-x-chartmuseum     https://chartmuseum.jx.sameer-jxapp.tests.fuloi.org
jenkins-x-docker-registry https://docker-registry.jx.sameer-jxapp.tests.fuloi.org
jenkins-x-monocular-api   https://monocular.jx.sameer-jxapp.tests.fuloi.org
jenkins-x-monocular-ui    https://monocular.jx.sameer-jxapp.tests.fuloi.org
nexus                     https://nexus.jx.sameer-jxapp.tests.fuloi.org
IMPORTANT: At the time of writing, the GitHub webhooks for the Jenkins X environment repositories are not updated to use the HTTPS protocol. You need to manually update the webhook URL’s for the staging and production environment repositories in GitHub before continuing.

Copy webhook URL

This concludes the installation of Jenkins X to the cluster and setting up HTTPS access to the services installed by Jenkins X. In the next section we will create a sample node application.

Developing with Jenkins X

In this section of the post, you will create a new NodeJS quickstart application using the Jenkins X CLI and learn about using the BKPR infrastructure to:

  • Secure the application with a TLS certificate
  • Enable OAuth based authentication for accessing the application

Creating a NodeJS application

Jenkins X quickstarts are pre-made applications you can use to start a new project. The jx create quickstart command is used to create the application and import the generated code into Git and Jenkins for CI/CD.

# jx create quickstart
? Do you wish to use xxsbn as the Git user name? Yes
? Which organisation do you want to use? xxsbn
? Enter the new repository name: jx-node
? Would you like to initialise git now? Yes
? Commit message:  Initial import

In essence, this command creates a new application from the quickstart in a subdirectory, adds its source code to a Git repository, pushes the repository to GitHub and registers webhooks for integration with Jenkins CI/CD.

Jenkins and GitHub CI integration

After the command is successfully executed, Jenkins X will automatically trigger a build of the project and you can query the status of the CI/CD pipeline using the jx get pipelines command.

# jx get pipelines jx-node
Name                 URL                                                                               LAST_BUILD STATUS   DURATION
xxsbn/jx-node/master https://jenkins.jx.my-domain.com/job/xxsbn/job/jx-node/job/master/ #1             Building -1ns(est.)

You can also access the pipeline build logs using the jx get build logs command:

# jx get build logs xxsbn/jx-node/master
view the log at: https://jenkins.jx.sameer-jxapp.tests.fuloi.org/job/xxsbn/job/jx-node/job/master/1/console 
tailing the log of xxsbn/jx-node/master #1 
…
…
GitHub has been notified of this commit’s build result
Finished: SUCCESS

By default, the application is deployed in the staging environment in Kubernetes. To get the list of applications deployed use the jx get applications command.

# jx get applications
APPLICATION STAGING PODS URL
jx-jx-node  0.0.1   1/1  http://jx-node.jx-staging.my-domain.com
NOTE: You can promote your application to the production environment using the jx promote command. Refer to the official documentation to learn more.

Configuring TLS support

To secure your application deployed in the staging environment with a LetsEncrypt TLS certificate, the Ingress rules of the jx-node service in the jx-staging namespace need to be upgraded.

# jx upgrade ingress --namespaces=jx-staging --services=jx-node --skip-certmanager=true --wait-for-certs=false
? Existing ingress rules found in namespaces [jx-staging] namespace.  Confirm to delete and recreate them Yes
? Expose type Ingress
? Domain: my-domain.com
? If your network is publicly available would you like to enable cluster wide TLS? Yes
? Use LetsEncrypt staging or production? production
? Email address to register with LetsEncrypt: admin@my-domain.com
? UrlTemplate (press <Enter> to keep the current value): "{{.Service}}.{{.Namespace}}.{{.Domain}}"
? Using config values {}, ok? Yes
? Do you want to update all existing webhooks? Yes

Check the status of the TLS certificate request using the following command:

kubectl --namespace jx-staging get certificates
NAME          READY   SECRET        AGE
tls-jx-node   True    tls-jx-node   4m

After the TLS certificate has been successfully provisioned you will be able to securely access the application over the HTTPS protocol.

# jx get applications --env=staging
APPLICATION STAGING PODS URL
jx-jx-node  0.0.1   1/1  https://jx-node.jx-staging.my-domain.com
NOTE: You can use the same steps to enable TLS support for services deployed in the production environment.

Enabling OAuth

It’s a common practice to control access to applications in the staging environment. BKPR makes it really easy for you to apply this restriction such that your application is only accessible by users of your organisation. This is achieved by utilizing the OAuth infrastructure provided by BKPR.

Login to GitHub and edit the env/values.yaml file present in the repository for staging environment.

Edit values.yaml

Append the following YAML snippet to the file and commit the changes to the master branch.

Remember to update all instances of my-domain.com in the snippet with the DNS zone of your BKPR cluster.

jx-node:
  service:
    annotations:
      fabric8.io/ingress.annotations: |-
        nginx.ingress.kubernetes.io/auth-signin: https://auth.my-domain.com/oauth2/start?rd=%2F$server_name$escaped_request_uri
        nginx.ingress.kubernetes.io/auth-url: https://auth.my-domain.com/oauth2/auth

After these changes are applied to the master branch of the repository on GitHub, you need to re-create the Ingress resources using the jx upgrade ingress command.

# jx upgrade ingress --namespaces=jx-staging --services=jx-node --skip-certmanager=true --wait-for-certs=false
? Existing ingress rules found in namespaces [jx-staging] namespace.  Confirm to delete and recreate them Yes
? Expose type Ingress
? Domain: my-domain.com
? If your network is publicly available would you like to enable cluster wide TLS? Yes
? Use LetsEncrypt staging or production? production
? Email address to register with LetsEncrypt: admin@my-domain.com
? UrlTemplate (press <Enter> to keep the current value): "{{.Service}}.{{.Namespace}}.{{.Domain}}"
? Using config values {}, ok? Yes
? Do you want to update all existing webhooks? Yes

With this update, the application in the staging environment will be protected by OAuth and users will now be prompted to authenticate themselves to access your application.

Preview Environment

The Jenkins X preview environment enables developers to get early feedback on changes to applications before the changes are merged into master. Jenkins X automatically creates preview environments inside the pipelines.

Like in the staging environment, you may want to secure access to the previews with TLS certificates as well as enforce OAuth based access controls.

To achieve this, you should edit the charts/preview/values.yaml file present in your application repository and apply the following changes:

diff --git a/charts/preview/values.yaml b/charts/preview/values.yaml
index b53ceaa..cfde4e7 100755
--- a/charts/preview/values.yaml
+++ b/charts/preview/values.yaml
@@ -5,8 +5,8 @@ expose:
     helm.sh/hook-delete-policy: hook-succeeded
   config:
     exposer: Ingress
-    http: true
-    tlsacme: false
+    http: false
+    tlsacme: true
 
 cleanup:
   Args:
@@ -16,6 +16,12 @@ cleanup:
     helm.sh/hook-delete-policy: hook-succeeded
 
 preview:
+  service:
+    annotations:
+      fabric8.io/ingress.annotations: |-
+        certmanager.k8s.io/cluster-issuer: letsencrypt-staging
+        nginx.ingress.kubernetes.io/auth-signin: https://auth.my-domain.com/oauth2/start?rd=%2F$server_name$escaped_request_uri
+        nginx.ingress.kubernetes.io/auth-url: https://auth.my-domain.com/oauth2/auth
   image:
     repository:
     tag:

The tlsacme: true configuration configures the TLS certificate provisioning while the certmanager.k8s.io/cluster-issuer: letsencrypt-staging annotation instructs BKPR to provision a TLS certificate from the LetsEncrypt staging environment.

Since the preview environments are short lived, it’s recommended that you utilize the LetsEncrypt staging environment to avoid encountering LetsEncrypt rate-limiting issues.

The nginx.ingress.kubernetes.io/auth-signin and nginx.ingress.kubernetes.io/auth-url annotations configure OAuth based access control for the previews. Remember to update all instances of my-domain.com with the value of the DNS zone configured with BKPR.

An improved Jenkins X workflow

The Bitnami Kubernetes Production Runtime (BKPR) provides a solid base for deploying the Jenkins X CD/CI platform. Using the pre-integrated BKPR components you were able to easily secure access to Jenkins X services and more importantly secure access to your application with TLS certificates.

Furthermore, by utilizing the OAuth infrastructure you were able to apply access control to services in the preview and staging environments. BKPR also provides infrastructure for logging and monitoring the performance of your application running in Kubernetes. Refer to the BKPR documentation to learn more .

If you’re using Jenkins X for your development pipeline, take BKPR for a spin.