gcr.io/k8s-minikube/kicbase: 321.58 MiB / 321.58 MiB 100.00% 2.38 MiB p/ 🔥 Creating docker container (CPUs=4, Memory=8192MB) ... > kubeadm.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s > kubelet.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s > kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s > kubectl: 41.44 MiB / 41.44 MiB [---------------] 100.00% 3.80 MiB p/s 11s > kubeadm: 40.50 MiB / 40.50 MiB [---------------] 100.00% 3.14 MiB p/s 13s > kubelet: 107.26 MiB / 107.26 MiB [-------------] 100.00% 5.41 MiB p/s 20s ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default Minikube Dashboard / k9s At this point you probably want to decide which tool you want to use to help you with troubleshooting issues with the cluster you created. Most minikube documentation points to using the Minikube Dashboard, which is easily invoked: minikube dashboard The UI will launch in your default web browser and is very similar to the Kubernetes Dashboard in design. However, being a Terminal person, I tend to use k9s. It is easy to install, lightweight, and most reminds me of using something like top to get a handle of what is happening in your cluster. In fact I often run it in another pane of my Terminal. Here is how you install it: brew install k9s Launching k9s from the Terminal will give you a curses view where you can bounce around. Here is a more complete Medium post on how to use it: K9s — the powerful terminal UI for Kubernetes Installing Jenkins into Minikube Now that all of the pre-requisites are out of the way, it is time to install Jenkins using Helm. The first step is to add the location of the helm repo to your helm install: helm repo add jenkins https://charts.jenkins.io helm repo update You then can search for the latest helm chart by doing the following: helm search repo jenkins NAME CHART VERSION APP VERSION DESCRIPTION jenkins/jenkins 3.9.0 2.303.3 Jenkins - Build great things at any scale! The .. Pull the chart from the repo: helm pull jenkins/jenkins This should create a helm chart in the local folder. In this case it is name jenkins-3.9.0.tgz Helm works by overiding the values from the chart to set your own values. Simply create the values for the chart by doing the following: helm show values jenkins/jenkins > jenkins-values.yaml It should have a bunch of stuff that is disabled by default. My values.yaml file was almost 900 lines long with a lot of comments. It is fine to remove most of this. For minikube I mainly want to override the namespace and the persistent volume (PV) that we want to use in the cluster, but before we do that we have to create them. I want jenkins to be installed in the jenkins namespace and I want the PV to be installed locally in my home folder. First create the namespace and a suitable definition for the PV: $ kubectl apply -f jenkins-namespace.yaml namespace/jenkins created $ kubectl apply -f jenkins-volume.yaml persistentvolume/jenkins-volume created Minikube does not come with a LoadBalancer by default so you also have to change the service to a NodePort. $ helm install jenkins ./jenkins-3.9.0.tgz -n jenkins -f jenkins-values.yaml NAME: jenkins LAST DEPLOYED: Sat Nov 27 17:29:40 2021 NAMESPACE: jenkins STATUS: deployed REVISION: 1 NOTES: 1. Get your 'admin' user password by running: kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/chart-admin-password && echo ... 3. Login with the password from step 1 and the username: admin 4. Configure security realm and authorization strategy 5. Use Jenkins Configuration as Code by specifying configScripts in your values.yaml file, see documentation: http:///configuration-as-code and examples: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos For more information on running Jenkins on Kubernetes, visit: https://cloud.google.com/solutions/jenkins-on-container-engine For more information about Jenkins Configuration as Code, visit: https://jenkins.io/projects/jcasc/ NOTE: Consider using a custom image with pre-installed plugins As it mentions in the info above, you have to get the default admin password that is auto-generated. In my case: $ kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/chart-admin-password && echo 5HrilnfS7eHAVfwDfyKv9B Due to Kubernetes being launched in Docker, you need to use the minikube service command to tunnel in to launch the Jenkins UI in your default browser: $ minikube service jenkins -n jenkins |-----------|---------|-------------|---------------------------| | NAMESPACE | NAME | TARGET PORT | URL | |-----------|---------|-------------|---------------------------| | jenkins | jenkins | http/8080 | http://192.168.49.2:30897 | |-----------|---------|-------------|---------------------------| 🏃 Starting tunnel for service jenkins. |-----------|---------|-------------|------------------------| | NAMESPACE | NAME | TARGET PORT | URL | |-----------|---------|-------------|------------------------| | jenkins | jenkins | | http://127.0.0.1:52330 | |-----------|---------|-------------|------------------------| 🎉 Opening service jenkins/jenkins in default browser... ❗ Because you are using a Docker driver on darwin, the terminal needs to be open to run it. Use the admin user and the password above to login to the UI. You can now create a Freestyle or Pipeline Job that will launch agent images within the cluster that execute the build. You can see the Jenkins Kubernetes cluster configuration under Manage Jenkins > Manage Nodes and Clouds > Configure Clouds > Kubernetes Cloud Details Uninstall Jenkins and Minikube If at any point you need to uninstall the K8s, kill the minikube service command with a Ctrl-C and then uninstall the jenkins: helm uninstall jenkins -n jenkins And then kill minikube detritus: minikube stop minikube delete" />

Erik's Thoughts and Musings

Apple, DevOps, Technology, and Reviews

Installing Jenkins in Minikube (M1 Mac)

Introduction

I always wanted to try and setup Jenkins in a Minikube instance. While not necessary, it is important to get DevOps type tools running in a cluster.

Prerequisites

These tools are recommended to get Minikube running:

  • Homebrew - self described "Missing Package Manager for macOS"
  • Minikube - tool to easily create a single node Kubernetes cluster
  • Docker Desktop for M1 - container runtime
  • kubectl - Kubernetes command-line tool (CLI)
  • kubectx/kubens - Convenience tool for changing contexts and namespaces
  • helm - defacto Kubernetes package manager

Homebrew

Homebrew is arguably the best package manager for the Mac to install terminal applications. It has a very simple : 

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Minikube

Minikube is an easy to deploy single node Kubernetes cluster. It used Homebrew to install:

brew install minikube

Docker Desktop

Docker Desktop is required to run Minikube on M1 Macs. Here is the direct link to the installer disk image for for M1 Macs:

https://desktop.docker.com/mac/main/arm64/Docker.dmg

Note that as of the time of this writing. You can't run Minikube in Virtual Box on M1 Macs. You will get this error:

$ minikube start --vm-driver=virtualbox
😄  minikube v1.24.0 on Darwin 12.0.1 (arm64)
✨  Using the virtualbox driver based on user configuration

❌  Exiting due to DRV_UNSUPPORTED_OS: The driver 'virtualbox' is not supported on darwin/arm64

You must use Docker.

kubectl

kubectl is the command-line tool for interacting with your Kubernetes API. It is easily installable via homebrew:

brew install kubectl

kubectx / kubens

The next two tools are not required per se. In my mind they are the easy shortcuts that should have been included with any install of kubectl. Every docker image that I build that

  • kubectx - Change the kubernetes context from one cluster to another
  • kubens - Easily change the default kubernetes namespace

You can do both using kubectl config ... commands, but these

They are both easily installable via homebrew using 1 command

brew install kubectx

More details about kubectx/kubens tools at the Github repository.

Helm

In the same way that Homebrew is the defacto package manager for macOS, helm is for Kubernetes.

brew install helm

Helm will be used to install Jenkins onto the Kubernetes cluster.

Docker Preferences

Kubernetes needs a bunch of resources to run in Docker. For example, if you run this command you will get the following error:

$ minikube start --memory 8192 --cpus 4 --vm-driver=docker
😄  minikube v1.24.0 on Darwin 12.0.1 (arm64)
✨  Using the docker driver based on user configuration

❌  Exiting due to MK_USAGE: Docker Desktop has only 1988MB memory but you specified 8192MB

To fix:

  • Launch Docker Desktop
  • Under the Preferences cog in the upper right choose "Resources"
  • Set CPUs to something suitable for Kubernetes. I chose 4 CPUs and 10 GB of RAM.
  • "Apply and Restart"

Minikube Start

It should now be possible to start minikube:

$ minikube start --memory 8192 --cpus 4 --vm-driver=docker
😄  minikube v1.24.0 on Darwin 12.0.1 (arm64)
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
    > gcr.io/k8s-minikube/kicbase: 321.58 MiB / 321.58 MiB  100.00% 2.38 MiB p/
🔥  Creating docker container (CPUs=4, Memory=8192MB) ...
    > kubeadm.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubelet.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubectl: 41.44 MiB / 41.44 MiB [---------------] 100.00% 3.80 MiB p/s 11s
    > kubeadm: 40.50 MiB / 40.50 MiB [---------------] 100.00% 3.14 MiB p/s 13s
    > kubelet: 107.26 MiB / 107.26 MiB [-------------] 100.00% 5.41 MiB p/s 20s

    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Minikube Dashboard / k9s

At this point you probably want to decide which tool you want to use to help you with troubleshooting issues with the cluster you created. Most minikube documentation points to using the Minikube Dashboard, which is easily invoked:

minikube dashboard

The UI will launch in your default web browser and is very similar to the Kubernetes Dashboard in design.

However, being a Terminal person, I tend to use k9s. It is easy to install, lightweight, and most reminds me of using something like top to get a handle of what is happening in your cluster. In fact I often run it in another pane of my Terminal. Here is how you install it:

brew install k9s

Launching k9s from the Terminal will give you a curses view where you can bounce around. Here is a more complete Medium post on how to use it:

K9s — the powerful terminal UI for Kubernetes

Installing Jenkins into Minikube

Now that all of the pre-requisites are out of the way, it is time to install Jenkins using Helm. The first step is to add the location of the helm repo to your helm install:

helm repo add jenkins https://charts.jenkins.io
helm repo update

You then can search for the latest helm chart by doing the following:

helm search repo jenkins
NAME            CHART VERSION   APP VERSION DESCRIPTION
jenkins/jenkins 3.9.0           2.303.3     Jenkins - Build great things at any scale! The ..

Pull the chart from the repo:

helm pull jenkins/jenkins

This should create a helm chart in the local folder. In this case it is name jenkins-3.9.0.tgz

Helm works by overiding the values from the chart to set your own values. Simply create the values for the chart by doing the following:

helm show values jenkins/jenkins > jenkins-values.yaml

It should have a bunch of stuff that is disabled by default. My values.yaml file was almost 900 lines long with a lot of comments. It is fine to remove most of this. For minikube I mainly want to override the namespace and the persistent volume (PV) that we want to use in the cluster, but before we do that we have to create them. I want jenkins to be installed in the jenkins namespace and I want the PV to be installed locally in my home folder.

First create the namespace and a suitable definition for the PV:

$ kubectl apply -f jenkins-namespace.yaml
namespace/jenkins created
$ kubectl apply -f jenkins-volume.yaml
persistentvolume/jenkins-volume created

Minikube does not come with a LoadBalancer by default so you also have to change the service to a NodePort.

$ helm install jenkins ./jenkins-3.9.0.tgz -n jenkins -f jenkins-values.yaml
NAME: jenkins
LAST DEPLOYED: Sat Nov 27 17:29:40 2021
NAMESPACE: jenkins
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:
  kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/chart-admin-password && echo
...

3. Login with the password from step 1 and the username: admin
4. Configure security realm and authorization strategy
5. Use Jenkins Configuration as Code by specifying configScripts in your values.yaml file, see documentation: http:///configuration-as-code and examples: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos

For more information on running Jenkins on Kubernetes, visit:
https://cloud.google.com/solutions/jenkins-on-container-engine

For more information about Jenkins Configuration as Code, visit:
https://jenkins.io/projects/jcasc/


NOTE: Consider using a custom image with pre-installed plugins

As it mentions in the info above, you have to get the default admin password that is auto-generated. In my case:

$ kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/chart-admin-password && echo
5HrilnfS7eHAVfwDfyKv9B

Due to Kubernetes being launched in Docker, you need to use the minikube service command to tunnel in to launch the Jenkins UI in your default browser:

$ minikube service jenkins -n jenkins
|-----------|---------|-------------|---------------------------|
| NAMESPACE |  NAME   | TARGET PORT |            URL            |
|-----------|---------|-------------|---------------------------|
| jenkins   | jenkins | http/8080   | http://192.168.49.2:30897 |
|-----------|---------|-------------|---------------------------|
🏃  Starting tunnel for service jenkins.
|-----------|---------|-------------|------------------------|
| NAMESPACE |  NAME   | TARGET PORT |          URL           |
|-----------|---------|-------------|------------------------|
| jenkins   | jenkins |             | http://127.0.0.1:52330 |
|-----------|---------|-------------|------------------------|
🎉  Opening service jenkins/jenkins in default browser...
❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.

Use the admin user and the password above to login to the UI.

You can now create a Freestyle or Pipeline Job that will launch agent images within the cluster that execute the build.

You can see the Jenkins Kubernetes cluster configuration under Manage Jenkins > Manage Nodes and Clouds > Configure Clouds > Kubernetes Cloud Details

Uninstall Jenkins and Minikube

If at any point you need to uninstall the K8s, kill the minikube service command with a Ctrl-C and then uninstall the jenkins:

helm uninstall jenkins -n jenkins

And then kill minikube detritus:

minikube stop
minikube delete