Introduction
I always wanted to try and setup Jenkins in a Minikube instance. While not necessary, it is important to get DevOps type tools running in a cluster.
Prerequisites
These tools are recommended to get Minikube running:
- Homebrew - self described "Missing Package Manager for macOS"
- Minikube - tool to easily create a single node Kubernetes cluster
- Docker Desktop for M1 - container runtime
- kubectl - Kubernetes command-line tool (CLI)
- kubectx/kubens - Convenience tool for changing contexts and namespaces
- helm - defacto Kubernetes package manager
Homebrew
Homebrew is arguably the best package manager for the Mac to install terminal applications. It has a very simple :
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Minikube
Minikube is an easy to deploy single node Kubernetes cluster. It used Homebrew to install:
Docker Desktop
Docker Desktop is required to run Minikube on M1 Macs. Here is the direct link to the installer disk image for for M1 Macs:
https://desktop.docker.com/mac/main/arm64/Docker.dmg
Note that as of the time of this writing. You can't run Minikube in Virtual Box on M1 Macs. You will get this error:
$ minikube start --vm-driver=virtualbox
😄 minikube v1.24.0 on Darwin 12.0.1 (arm64)
✨ Using the virtualbox driver based on user configuration
❌ Exiting due to DRV_UNSUPPORTED_OS: The driver 'virtualbox' is not supported on darwin/arm64
You must use Docker.
kubectl
kubectl is the command-line tool for interacting with your Kubernetes API. It is easily installable via homebrew:
kubectx / kubens
The next two tools are not required per se. In my mind they are the easy shortcuts that should have been included with any install of kubectl
. Every docker image that I build that
- kubectx - Change the kubernetes context from one cluster to another
- kubens - Easily change the default kubernetes namespace
You can do both using kubectl config ...
commands, but these
They are both easily installable via homebrew using 1 command
More details about kubectx/kubens tools at the Github repository.
Helm
In the same way that Homebrew is the defacto package manager for macOS, helm is for Kubernetes.
Helm will be used to install Jenkins onto the Kubernetes cluster.
Docker Preferences
Kubernetes needs a bunch of resources to run in Docker. For example, if you run this command you will get the following error:
$ minikube start --memory 8192 --cpus 4 --vm-driver=docker
😄 minikube v1.24.0 on Darwin 12.0.1 (arm64)
✨ Using the docker driver based on user configuration
❌ Exiting due to MK_USAGE: Docker Desktop has only 1988MB memory but you specified 8192MB
To fix:
- Launch Docker Desktop
- Under the Preferences cog in the upper right choose "Resources"
- Set CPUs to something suitable for Kubernetes. I chose 4 CPUs and 10 GB of RAM.
- "Apply and Restart"
Minikube Start
It should now be possible to start minikube:
$ minikube start --memory 8192 --cpus 4 --vm-driver=docker
😄 minikube v1.24.0 on Darwin 12.0.1 (arm64)
✨ Using the docker driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
> gcr.io/k8s-minikube/kicbase: 321.58 MiB / 321.58 MiB 100.00% 2.38 MiB p/
🔥 Creating docker container (CPUs=4, Memory=8192MB) ...
> kubeadm.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
> kubelet.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
> kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
> kubectl: 41.44 MiB / 41.44 MiB [---------------] 100.00% 3.80 MiB p/s 11s
> kubeadm: 40.50 MiB / 40.50 MiB [---------------] 100.00% 3.14 MiB p/s 13s
> kubelet: 107.26 MiB / 107.26 MiB [-------------] 100.00% 5.41 MiB p/s 20s
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Minikube Dashboard / k9s
At this point you probably want to decide which tool you want to use to help you with troubleshooting issues with the cluster you created. Most minikube documentation points to using the Minikube Dashboard, which is easily invoked:
The UI will launch in your default web browser and is very similar to the Kubernetes Dashboard in design.
However, being a Terminal person, I tend to use k9s. It is easy to install, lightweight, and most reminds me of using something like top to get a handle of what is happening in your cluster. In fact I often run it in another pane of my Terminal. Here is how you install it:
Launching k9s
from the Terminal will give you a curses view where you can bounce around. Here is a more complete Medium post on how to use it:
K9s — the powerful terminal UI for Kubernetes
Installing Jenkins into Minikube
Now that all of the pre-requisites are out of the way, it is time to install Jenkins using Helm. The first step is to add the location of the helm repo to your helm install:
helm repo add jenkins https://charts.jenkins.io
helm repo update
You then can search for the latest helm chart by doing the following:
helm search repo jenkins
NAME CHART VERSION APP VERSION DESCRIPTION
jenkins/jenkins 3.9.0 2.303.3 Jenkins - Build great things at any scale! The ..
Pull the chart from the repo:
helm pull jenkins/jenkins
This should create a helm chart in the local folder. In this case it is name jenkins-3.9.0.tgz
Helm works by overiding the values from the chart to set your own values. Simply create the values for the chart by doing the following:
helm show values jenkins/jenkins > jenkins-values.yaml
It should have a bunch of stuff that is disabled by default. My values.yaml file was almost 900 lines long with a lot of comments. It is fine to remove most of this. For minikube I mainly want to override the namespace and the persistent volume (PV) that we want to use in the cluster, but before we do that we have to create them. I want jenkins to be installed in the jenkins
namespace and I want the PV to be installed locally in my home folder.
First create the namespace and a suitable definition for the PV:
$ kubectl apply -f jenkins-namespace.yaml
namespace/jenkins created
$ kubectl apply -f jenkins-volume.yaml
persistentvolume/jenkins-volume created
Minikube does not come with a LoadBalancer by default so you also have to change the service to a NodePort.
$ helm install jenkins ./jenkins-3.9.0.tgz -n jenkins -f jenkins-values.yaml
NAME: jenkins
LAST DEPLOYED: Sat Nov 27 17:29:40 2021
NAMESPACE: jenkins
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:
kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/chart-admin-password && echo
...
3. Login with the password from step 1 and the username: admin
4. Configure security realm and authorization strategy
5. Use Jenkins Configuration as Code by specifying configScripts in your values.yaml file, see documentation: http:///configuration-as-code and examples: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos
For more information on running Jenkins on Kubernetes, visit:
https://cloud.google.com/solutions/jenkins-on-container-engine
For more information about Jenkins Configuration as Code, visit:
https://jenkins.io/projects/jcasc/
NOTE: Consider using a custom image with pre-installed plugins
As it mentions in the info above, you have to get the default admin password that is auto-generated. In my case:
$ kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/chart-admin-password && echo
5HrilnfS7eHAVfwDfyKv9B
Due to Kubernetes being launched in Docker, you need to use the minikube service
command to tunnel in to launch the Jenkins UI in your default browser:
$ minikube service jenkins -n jenkins
|-----------|---------|-------------|---------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|---------|-------------|---------------------------|
| jenkins | jenkins | http/8080 | http://192.168.49.2:30897 |
|-----------|---------|-------------|---------------------------|
🏃 Starting tunnel for service jenkins.
|-----------|---------|-------------|------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|---------|-------------|------------------------|
| jenkins | jenkins | | http://127.0.0.1:52330 |
|-----------|---------|-------------|------------------------|
🎉 Opening service jenkins/jenkins in default browser...
❗ Because you are using a Docker driver on darwin, the terminal needs to be open to run it.
Use the admin
user and the password above to login to the UI.
You can now create a Freestyle or Pipeline Job that will launch agent images within the cluster that execute the build.
You can see the Jenkins Kubernetes cluster configuration under Manage Jenkins > Manage Nodes and Clouds > Configure Clouds > Kubernetes Cloud Details
Uninstall Jenkins and Minikube
If at any point you need to uninstall the K8s, kill the minikube service
command with a Ctrl-C and then uninstall the jenkins:
helm uninstall jenkins -n jenkins
And then kill minikube detritus:
minikube stop
minikube delete