In this post, I will demonstrate how to install Jenkins into a Kubernetes cluster in AWS and go on to explain how to configure a build pipeline for Java applications such that the built application can be automatically deployed to the Kubernetes cluster.

Prerequisites

Kubernetes cluster

First of all, you will need a Kubernetes cluster running on AWS. If you don’t yet have this then please see my other post Creating a Kubernetes cluster using KOPS in AWS.

Please refer to the Prerequisites section of that post, as you will need to have kubectl installed and configured to point at your cluster.

Before proceeding, please verify your cluster is ready using

$ kops validate cluster

You should see a message indicating that your cluster is ready.

Installing Jenkins

Create Jenkins Docker image

Basic image

Creating a Docker image is pretty simple as there is already an official Jenkins Docker image on DockerHub available at https://hub.docker.com/r/jenkins/jenkins/.

However, this will only give us a clean fresh install of Jenkins. It will need configuring in terms of plugins, global tools and credentials.

As a Java developer, I want a Jenkins instance to support Java applications. I host my applications on GitHub. I may want to build and publish a Docker image to DockerHub so I want Jenkins to support this. I want to deploy my built applications to a Kubernetes cluster.

Ideally, I want to make this immutable. I should be able to destroy the Jenkins instance and recreate it with minimal effort.

To create our own custom Jenkins Docker image, we can start with the official image in our Dockerfile:

1
FROM jenkins/jenkins:lts

So, first thing we need is Java support. We can simply use apt-get to install this:

RUN apt-get update \
 && apt-get install -y openjdk-8-jdk \
 && apt-get clean

Next let’s add Docker support:

RUN apt-get update \
 && apt-get install -y openjdk-8-jdk \
 && apt-get install -y curl \
 && curl -fsSL https://get.docker.com | sh \
 && usermod -aG docker jenkins \
 && apt-get clean

Finally let’s add kubectl, the Kubernetes command line tool, so we can deploy our application to a Kubernetes cluster:

 WORKDIR /usr/local/bin
 RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
 RUN chmod +x kubectl
Add Jenkins plugins

Now we have our basic Jenkins image. To automatically install the plugins at the Docker image build stage, we simply need to supply a text file containing a list of the plugins we require and a simple update to the Dockerfile:

COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt
Add tooling

To compile and build Java applications, we need some tools installing in Jenkins; Maven and Java.

As part of the standard Jenkins Docker image, any groovy scripts we copy to /usr/share/jenkins/ref/init.groovy.d/ will be executed on the first startup.

I, therefore, have two such groovy files for tooling:

java.groovy

import hudson.model.JDK
import jenkins.model.Jenkins

Jenkins.getInstance().getJDKs().add(new JDK("jdk8", "/usr/lib/jvm/java-8-openjdk-amd64"))

maven.groovy

import hudson.tasks.Maven
import hudson.tasks.Maven.MavenInstallation;
import hudson.tools.InstallSourceProperty;
import hudson.tools.ToolProperty;
import hudson.tools.ToolPropertyDescriptor
import hudson.tools.ZipExtractionInstaller;
import hudson.util.DescribableList
import jenkins.model.Jenkins;

def extensions = Jenkins.instance.getExtensionList(Maven.DescriptorImpl.class)[0]

List<MavenInstallation> installations = []

mavenToool = ['name': 'Maven', 'url': 'http://www.mirrorservice.org/sites/ftp.apache.org/maven/maven-3/3.5.2/binaries/apache-maven-3.5.2-bin.tar.gz', 'subdir': 'apache-maven-3.5.2']

println("Setting up tool: ${mavenToool.name} ")

def describableList = new DescribableList<ToolProperty<?>, ToolPropertyDescriptor>()
def installer = new ZipExtractionInstaller(mavenToool.label as String, mavenToool.url as String, mavenToool.subdir as String);

describableList.add(new InstallSourceProperty([installer]))

installations.add(new MavenInstallation(mavenToool.name as String, "", describableList))

extensions.setInstallations(installations.toArray(new MavenInstallation[installations.size()]))
extensions.save()
Add credentials

As part of our build pipeline, we need to be able to push to GitHub and DockerHub. We, therefore, can add credential configurations as part of the Docker image.

Naturally, it would be silly to include account details in clear text in your Docker images even if you have an internal Docker Registry. I, therefore, have a further groovy script to create blank credentials for GitHub and DockerHub that will require updating once Jenkins is deployed.

credentials.groovy

import com.cloudbees.plugins.credentials.CredentialsScope
import com.cloudbees.plugins.credentials.SystemCredentialsProvider
import com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl

Closure setCredsIfMissing = { String id, String descr, String user, String pass ->
  boolean credsMissing = SystemCredentialsProvider.getInstance().getCredentials().findAll {
    it.getDescriptor().getId() == id
  }.empty
  if (credsMissing) {
    println "Credential [${id}] is missing - will create it"
    SystemCredentialsProvider.getInstance().getCredentials().add(
      new UsernamePasswordCredentialsImpl(CredentialsScope.GLOBAL, id,
        descr, user, pass))
    SystemCredentialsProvider.getInstance().save()
  }
}

setCredsIfMissing("GitHub", "GitHub credential", "username", "password")
setCredsIfMissing("DockerHub", "DockerHub credential", "username", "password")
Final Dockerfile

The final complete Dockerfile is below.

 1 2 3 4 5 6 7 8 9101112131415161718192021222324
FROM jenkins/jenkins:lts

USER root

# Install dependencies
RUN apt-get update \
 && apt-get install -y curl \
 && apt-get install -y openjdk-8-jdk \
 && curl -fsSL https://get.docker.com | sh \
 && apt-get clean \
 && usermod -aG docker jenkins

# Add kubectl
WORKDIR /usr/local/bin
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
RUN chmod +x kubectl

# Back to Jenkins home
USER jenkins
WORKDIR $JENKINS_HOME

COPY *.groovy /usr/share/jenkins/ref/init.groovy.d/
COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt

The image can be built with the following command:

$ docker build -t iancollington/jenkins .

I have pushed this image to DockerHub but you may wish to build your own custom image and push to your own DockerHub account. The code for my Jenkins image can be found at https://github.com/iancollington/docker-images/tree/master/jenkins.

Deploy Jenkins image to Kubernetes

To deploy Jenkins to the Kubernetes cluster we need to create a deployment configuration file. Firstly, we create a PersistentVolume and PersistentVolumeClaim to provide the file storage that Jenkins requires.

kind: PersistentVolume
apiVersion: v1
metadata:
  name: jenkins
  labels:
    type: local
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/jenkins/"

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

Now we can define the service:

apiVersion: v1
kind: Service
metadata:
  name: jenkins
  labels:
    app: jenkins
spec:
  ports:
    - port: 8080
      targetPort: 8080
  selector:
    app: jenkins
    tier: jenkins
  type: NodePort

Finally we can define the deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
  labels:
    app: jenkins
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: jenkins
        tier: jenkins
    spec:
      containers:
      - image: iancollington/jenkins:latest
        name: jenkins
        env:
          - name: DOCKER_HOST
            value: tcp://localhost:2375
        ports:
        - containerPort: 8080
          name: jenkins
        volumeMounts:
        - name: jenkins-persistent-storage
          mountPath: /root/.jenkins
      volumes:
      - name: jenkins-persistent-storage
        persistentVolumeClaim:
          claimName: jenkins-claim

The above configuration will work fine for normal Jenkins jobs until we want to use Docker commands. Jenkins is already running in Docker so we want to run Docker-in-Docker!

This used to be a pain but the nice folks at Docker have provided a nice easy to use Docker image for this purpose. We can update our deployment configuration to include this Docker-in-Docker container in the same pod. We also need to set the DOCKER_HOST environment variable on the Jenkins container so that the Docker client knows where the Docker daemon is.

    spec:
      containers:
      - image: iancollington/jenkins:latest
        name: jenkins
        env:
          - name: DOCKER_HOST
            value: tcp://localhost:2375
        :
      - name: dind-daemon 
        image: docker:1.12.6-dind 
        resources: 
            requests: 
                cpu: 20m 
                memory: 512Mi 
        securityContext: 
            privileged: true 
        volumeMounts: 
          - name: docker-graph-storage 
            mountPath: /var/lib/docker
      volumes:
      - name: docker-graph-storage 
        emptyDir: {}

I want to access my Jenkins instance using the domain jenkins.iancollington.com. The simplest solution is to make the NodePort a LoadBalancer. This will create an AWS Load Balancer specifically for the jenkins-service. However, for each service you publically expose, another load balancer will be created which will increase costs.

To reduce costs, we can use an Ingress. An Ingress sits on the public boundary edge, proxying traffic to internal private services.

An Nginx based ingress can be installed as follows:

$ kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/ingress-nginx/v1.6.0.yaml

Once deployed, a new load balancer in AWS will be created. This is the only one that is required. We can configure the Ingress to route traffic based on the hostname to the appropriate service.

Let’s see what that looks like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: jenkins.iancollington.com
    http:
      paths:
      - backend:
          serviceName: jenkins
          servicePort: 8080

We are creating an Ingress service that will route traffic to the jenkins service on port 8080 when traffic is received on the jenkins.iancollington.com host.

Adding that block to the YAML file means it is ready to deploy. The full configuration file is available as a Gist and can be applied to your cluster with the following command:

$ kubectl apply -f https://gist.githubusercontent.com/iancollington/c2dd478f6186f9ceac2c2a2009730b46/raw/61bd99201664f84708fb37f915457624496976b8/k8s-jenkins.yaml

However, we need to set up the jenkins.iancollington.com subdomain and point it at the load balancer for this to work. This is as simple as going into Route 53, adding the subdomain as a CNAME and entering the public DNS name of the load balancer.

Route 53 Jenkins subdomain

It may take a few minutes for DNS to update but eventually, Jenkins should appear on that domain:

Jenkins New Install

Setup Jenkins

Initial setup

With Jenkins deployed, you’ll need to unlock Jenkins with the administrator password. This password is randomly generated at first deploy and will be output to the logs. To view the log, you need to find the name of the pod:

$ kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
jenkins-3117850509-5qmlv   2/2       Running   0          2m

Once you have this, you view the log:

$ kubectl logs jenkins-3117850509-5qmlv -c jenkins

Note here that we have added -c jenkins. The reason for this is that we have two containers running in this pod. One is the Jenkins container itself and the other one is the docker-in-docker container. Therefore we need to specify which containers logs we want.

Within the output, you should see a section similar to the following:

*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

5acbfc8aa40647199c9818331efc462a

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************

Copy and paste this password into the textbox and hit Continue.

You will then be prompted to customise Jenkins with plugins. The plugins we require are already installed as part of the Docker image so clicking the ‘Install suggested plugins’ will be really quick.

Once that is done you can create the first admin user, hit Save and Finished and you’ll have a new empty Jenkins instance ready to go.

Credentials

As part of the Docker image, I added credentials configuration for Git Hub and DockerHub. It would be unwise to include the real credentials in the Docker image so these will need to be set up each time.

Simply click the Credentials menu item on the left-hand side and you’ll see the two credentials listed.

For each credential, click the link in the Name column, then click the Update menu item on the left-hand side. You can then enter the correct username and password.

Create a Jenkins pipeline

Now we have a working Jenkins installation, we can do something useful with it and set up a build pipeline.

Jenkinsfile

A Jenkins Pipeline is defined in a file named Jenkinsfile. For this example, we will use the declarative pipeline which provides more flexibility.

We will use an existing GitHub repository for this at https://github.com/iancollington/simple-spring-boot-application. It is a simple Spring Boot application that exposes a /hello endpoint.

If you take a look at the Jenkinsfile in that GitHub repository, you’ll see we have five stages; Init, Version, Compile, Test, Build and Deploy.

Init stage

The init stage is all about setting up properties and variables. We also make sure that Maven is on the PATH and invoke Git to download the code. The final line retrieves the current POM version. This version is used to tag the Docker image when it is pushed to Docker Hub.

Version

This stage simply inserts the POM version into the application.properties file. This could be read by your application if you required.

Compile

This is the first stage that actually does something worthwhile. We simply check that the code compiles.

Test

The test stage is where, not surprisingly, we execute all the tests. This command is surrounded by a try/catch block to catch a test failure. Groovy being a JVM based language, the finally block is the same as Java here. We always get the Surefire and Failsafe plugin reports and publish them to Jenkins.

Package

Once we know the code is working, we can package it up. The first stage simple invokes Maven to create the JAR file. We skip the tests here as there is no point running them twice.

If the Maven packaging worked successfully then we know we have a releasable artefact. We, therefore, tag the branch with the POM version number and push back to Github. In a CI environment, the same snapshot will be in progress across multiple commits, so we force the tag creation and push each time.

Finally, we build the Docker image which uses the Dockerfile in the root of the repository. It is then pushed to Docker Hub, tagged as the latest version but also tagged with the Maven POM version.

Deploy

At this stage, we have an artefact in Docker Hub that can be deployed.

The src/main/k8s directory contains three files that describe the Kubernetes deployment.

The first, k8s-deployment.yaml, contains the Docker image and version that is to be deployed. The first line of this stage, therefore, updates this file to use the version that was just built and pushed to Docker Hub.

 sh '/usr/local/bin/kubectl set image -f src/main/k8s/k8s-deployment.yaml \
    $module=$dockerImagePrefix$module:$version --local -o yaml | \
    sed "s/BUILD_DATE_PLACEHOLDER/$(date)/g" - | \
    /usr/local/bin/kubectl apply --force -f -'

The kubectl set image command updates the Docker image and version in the k8s-deployment.yaml file and it sends the output to standard out which is piped to awk. The output of awk is then piped to the kubectl apply command which applies the change to the cluster.

The awk command here is crucial. If you take a look at the k8s-deployment.yaml file, you’ll see a section for environment variables:

        env:
          - name: BUILD_DATE
            value: BUILD_DATE_PLACEHOLDER

The awk command replaces the text BUILD_DATE_PLACEHOLDER with the current date. This ensures that the yaml file is actually different from the previous kubectl apply command. If the descriptor has not changed then Kubernetes will not take any action even if we use the --force flag.

The second line of this stage applies any changes to the service descriptor defined in the k8s-service.yaml file.

 sh "/usr/local/bin/kubectl apply -f src/main/k8s/k8s-service.yaml"

The third line of this stage applies any changes to the ingress descriptor defined in the k8s-ingress.yaml file. It is in this file that describes how we access the deployed application from the public internet.

 sh "/usr/local/bin/kubectl apply -f src/main/k8s/k8s-ingress.yaml" 

Like we did with the Jenkins ingress descriptor, here we direct any traffic to the hello.iancollington.com subdomain to the simple-spring-boot-application service on port 8080.

Create a new job

Creating a new pipeline job in Jenkins is easy. We just simply point it at the Git repository that contains a Jenkinsfile.

Click the Create new jobs link, enter simple-spring-boot-application for the job name and select Pipeline as the job type then OK. Scroll down to the bottom, and under Pipeline, Definition, Pipeline script from SCM. In the SCM drop down, select Git and enter https://github.com/iancollington/simple-spring-boot-application as the repository URL. Click Save and then the Build Now link in Jenkins.

Once the Jenkins job has completed it should have deployed the application to your cluster. You can verify this by checking the Kubernetes Dashboard:

Dashboard after Jenkins job

Before we can verify that the application is available, we need to update Route53 to point the hello.iancollington.com subdomain at the ingress load balancer. Again, this can be done within the AWS console by using the public DNS address used for the jenkins.iancollington.com subdomain.

Once DNS has had time to update, visiting http://hello.iancollington.com should result in the hello message being displayed:

Dashboard after Jenkins job

Next steps

This post demonstrates how to get a custom Jenkins image deployed as part of your Kubernetes cluster on AWS.

In a corporate environment, you may want to further customise this image to enable LDAP or even import a list of jobs from a Git repository.

If you really want a persistent state, then you could configure the PersistentVolume and PersistentVolumeClaim with an EFS (Elastic File System) such that the state can persist between instances.