Erik's Thoughts and Musings

Apple, DevOps, Technology, and Reviews

Using kubectl with Velero

Velero is an open source tool to Many of the commands in the velero CLI can also be invoked straight from using kubectl. The reason this works is that Velero installs Custom Resource Definitions (CRDs) that extend the Kubernetes cluster. You can query items right from the cluster or via a tool like k9s.

Here are some examples:

# Get all schedules in the cluster
kubectl get schedule -A

# Get schedules in the velero namespace
kubectl get schedule -n velero

# Get the yaml configuration of the `my-aks-cluster` schedule
kubectl get schedule my-aks-cluster -n velero -o yaml

# Get all backups
kubectl get backup -n velero

# Describe an individual backup
kubectl describe backup my-aks-cluster-20231218120035 -n velero

# Get all restores
kubectl get restore -n velero

# Describe an individual restore
kubectl get restore test-backup-20231020134940 -n velero -o yaml

Using the Velero CLI

Velero comes with a pretty handy command-line interface (CLI) for pretty much anything you want to do regarding backups and restores:

  • Scheduling / Creating backups
  • Backup information
  • Deleting backups
  • Full restores from backups
  • Selective restores from backups

To manually schedule, backup or restore, the CLI tool is mandatory. If you simply want to review the state of backups and restores in the cluster, you can use kubectl via the installed Velero. I'll do a followup post about how to use Velero with kubectl

To install Velero CLI, follow the instructions on the Basic Install page:

https://velero.io/docs/v1.8/basic-install/

All of the following examples use velero to invoke actions with the Velero agent. If you are going to do a lot of interaction with the backups, I recommend you set the default namespace via either of the two commands, otherwise you need to add -n velero to all commands below. To set the default namespace:

# Third party tool via https://github.com/ahmetb/kubectx
kubens velero

# More verbose namespace 
kubectl config set-context --current --namespace velero

Scheduling / Creating Backups

The first thing that you will want to do with Velero is to create a scheduled backup. To create a backup schedule named my-aks-cluster that runs every 4 hours and will expire a backup after 30 days (720 hours)

velero create schedule my-aks-cluster -n velero --schedule="0 */6 * * *" --ttl 720h0m0s

This will backup all namespaces and all disk volumes in the cluster. At the present time, we don't have any persistent disk volumes in our development or production clusters.

If you want list all of the schedules that have been configured:

$ velero schedule get
NAME             STATUS    CREATED                         SCHEDULE      BACKUP TTL   LAST BACKUP   SELECTOR   PAUSED
my-aks-cluster   Enabled   2023-11-24 15:47:00 -0500 EST   0 */6 * * *   720h0m0s     1h ago        <none>     false

If you want to do a one off ad-hoc backup named ad-hoc-backup, you can use the scheduled one as a template:

velero backup create ad-hoc-backup --from-schedule my-aks-cluster

Or you can simply create one from scratch by specifying what to include or exclude:

# Create a backup including only the nginx namespace.
velero backup create nginx-backup --include-namespaces nginx

# Create a backup excluding the velero and default namespaces.
velero backup create selective-backup --exclude-namespaces velero,default

Backup Information

To list all backups that have been done on the cluster:

$ velero backup get
NAME                            STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
my-aks-cluster-20231218180035   Completed   0        0          2023-12-18 13:00:35 -0500 EST   29d       default            <none>
my-aks-cluster-20231218120035   Completed   0        0          2023-12-18 07:00:35 -0500 EST   29d       default            <none>
my-aks-cluster-20231218060035   Completed   0        0          2023-12-18 01:00:35 -0500 EST   29d       default            <none>
my-aks-cluster-20231218000034   Completed   0        0          2023-12-17 19:00:34 -0500 EST   29d       default            <none>
...
my-aks-cluster-20231120180030   Completed   0        0          2023-11-20 13:00:30 -0500 EST   1d        default            <none>
my-aks-cluster-20231120120029   Completed   0        0          2023-11-20 07:00:29 -0500 EST   1d        default            <none>
my-aks-cluster-20231120060029   Completed   0        0          2023-11-20 01:00:29 -0500 EST   1d        default            <none>
my-aks-cluster-20231120000029   Completed   0        0          2023-11-19 19:00:29 -0500 EST   1d        default            <none>
my-aks-cluster-20231119180029   Completed   0        0          2023-11-19 13:00:29 -0500 EST   22h       default            <none>
my-aks-cluster-20231119120028   Completed   0        0          2023-11-19 07:00:28 -0500 EST   16h       default            <none>
my-aks-cluster-20231119060028   Completed   0        0          2023-11-19 01:00:28 -0500 EST   10h       default            <none>
my-aks-cluster-20231119000028   Completed   0        0          2023-11-18 19:00:28 -0500 EST   4h        default            <none>
my-aks-cluster-20231023180022   Completed   0        0          2023-10-23 14:00:22 -0400 EDT   33d       default            <none>

You can describe an individual backup by using the describe command and choosing the name of the backup:

$ velero backup describe my-aks-cluster-20231218180035
Name:         my-aks-cluster-20231218180035
Namespace:    velero
...

Phase:  Completed


Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

...

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

...

Started:    2023-12-18 13:00:35 -0500 EST
Completed:  2023-12-18 13:00:52 -0500 EST

Expiration:  2024-01-17 13:00:35 -0500 EST

Total items to be backed up:  1262
Items backed up:              1262

Velero-Native Snapshots: <none included>

You can get the logs of an individual backup by using the logs command:

$ velero backup logs my-aks-cluster-20231218180035
time="2023-12-18T18:00:35Z" level=info msg="Setting up backup temp file" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/controller/backup_controller.go:617"
time="2023-12-18T18:00:35Z" level=info msg="Setting up plugin manager" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/controller/backup_controller.go:624"
time="2023-12-18T18:00:35Z" level=info msg="Getting backup item actions" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/controller/backup_controller.go:628"
time="2023-12-18T18:00:35Z" level=info msg="Setting up backup store to check for backup existence" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/controller/backup_controller.go:633"
time="2023-12-18T18:00:36Z" level=info msg="Writing backup version file" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/backup/backup.go:197"
time="2023-12-18T18:00:36Z" level=info msg="Including namespaces: *" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/backup/backup.go:203"
time="2023-12-18T18:00:36Z" level=info msg="Excluding namespaces: <none>" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/backup/backup.go:204"
time="2023-12-18T18:00:36Z" level=info msg="Including resources: *" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/util/collections/includes_excludes.go:506"
time="2023-12-18T18:00:36Z" level=info msg="Excluding resources: <none>" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/util/collections/includes_excludes.go:507"
time="2023-12-18T18:00:36Z" level=info msg="Backing up all volumes using pod volume backup: false" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/backup/backup.go:222"
...

Deleting Backups

Deleting a backup should not be necessary with the TTL set to 30 days, but here is the mechanism to delete it.

$ velero backup delete my-aks-cluster-20231023180022
Are you sure you want to continue (Y/N)? y
Request to delete backup "my-aks-cluster-20231023180022" submitted successfully.
The backup will be fully deleted after all associated data (disk snapshots, backup files, restores) are removed.

You can also simply delete the backup from the storage account manually.

Full restores

To fully restore all items from a backup into a cluster, supply the backup name to --from-backup

velero restore create --from-backup my-aks-cluster-20231023180022

All restores can be listed in the same way backups can be listed:

$ velero restore get
NAME                               BACKUP              STATUS      STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
nginx-test-backup-20231218173505   nginx-test-backup   Completed   2023-12-18 17:35:06 -0500 EST   2023-12-18 17:35:09 -0500 EST   0        1          2023-12-18 17:35:06 -0500 EST   <none>

An individual restore can also be described:

$ velero restore describe nginx-test-backup-20231218173505
Name:         nginx-test-backup-20231218173505
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                       Completed
Total items to be restored:  10
Items restored:              10

Started:    2023-12-18 17:35:06 -0500 EST
Completed:  2023-12-18 17:35:09 -0500 EST
...

As well as logs retrieved:

$ velero restore logs nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:523" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Starting restore of backup velero/nginx-test-backup" logSource="pkg/restore/restore.go:423" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Resource 'serviceaccounts' will be restored into namespace 'nginx-test'" logSource="pkg/restore/restore.go:2293" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Resource 'configmaps' will be restored into namespace 'nginx-test'" logSource="pkg/restore/restore.go:2293" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Resource 'pods' will be restored into namespace 'nginx-test'" logSource="pkg/restore/restore.go:2293" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Resource 'replicasets.apps' will be restored into namespace 'nginx-test'" logSource="pkg/restore/restore.go:2293" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:2206" resource=clusterclasses.cluster.x-k8s.io restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Resource 'endpoints' will be restored into namespace 'nginx-test'" logSource="pkg/restore/restore.go:2293" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Resource 'services' will be restored into namespace 'nginx-test'" logSource="pkg/restore/restore.go:2293" restore=velero/nginx-test-backup-202312181735

Selective restore

Often you will want to just restore a namespace or a set of namespaces from a backup. In this example it will only restore the test namespace:

velero restore create --from-backup test-backup-20231020134940 --include-namespaces test --include-resources "*"

With the --include-resources you can also choose what Kubernetes resources to restore. This example only restores pods in the test namespace:

velero restore create --from-backup test-backup-20231020134940 --include-namespaces test --include-resources "pods"

Installing Velero in AKS

Velero is an open source Kubernetes backup service. The Velero service runs within the cluster in the velero namespace. It can backup all of the Kubernetes configuration manifests (including Custom Resource Definitions - CRDs) as well as any persistent volumes (PVs) that are attached to pods.

Velero Setup

The Velero setup, install, and configuration is completed using a helm chart. To download the chart clone it from Github:

git clone https://github.com/vmware-tanzu/helm-charts.git
cd helm-charts/charts/velero/

In Azure Entra ID, create an App Registration that will be used as the service account for the backups. I created "AKS Velero Backup" that works across the tenant so I could potentially back up clusters in any of my subscriptions. This service account will backup the configuration and any persistent volumes (PVs) to a storage account.

$ az ad sp list --display-name "AKS Velero Backup" -o table
DisplayName           Id                                    AppId                                 CreatedDateTime
--------------------  ------------------------------------  ------------------------------------  --------------------
AKS Velero Backup     <redacted>                            <redacted>                            2023-12-28T16:22:09Z

Add Contributor IAM privileges to the app ID of "AKS Velero Backup":

az role assignment create --assignee <App ID> --role "Contributor" --scope /subscriptions/<Subscription ID>

The helm chart also requires a credentials file that is used to be able to backup to the Azure storage account:

cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID=<Azure Subscription ID>
AZURE_TENANT_ID=<Azure Tenant ID>
AZURE_CLIENT_ID=<App ID>
AZURE_CLIENT_SECRET=<App ID Secret>
AZURE_RESOURCE_GROUP=<Resource Group of the AKS nodes>
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

With the credentials created, it is just a matter of setting the Helm chart variables using the cloud credential file as a parameter.

  • ${SUBSCRIPTION_ID} is the Azure subscription ID of where the storage account (Azure bucket) lives.
  • savelerobackups is the storage account name to save the backups
  • rg-velero-backups is the resource group for the storage account
  • --set-file credentials.secretContents.cloud is where you set the credentials for the Azure subscription
helm upgrade --install velero velero \
     --repo https://vmware-tanzu.github.io/helm-charts \
     --create-namespace --namespace velero \
     --set configuration.backupStorageLocation[0].name=velero.io/azure \
     --set configuration.backupStorageLocation[0].bucket="my-aks-cluster" \
     --set configuration.backupStorageLocation[0].config.subscriptionId=${SUBSCRIPTION_ID} \
     --set configuration.backupStorageLocation[0].config.storageAccount=savelerobackups \
     --set configuration.backupStorageLocation[0].config.resourceGroup=rg-velero-backups \
     --set configuration.volumeStorageLocation[0].name=velero.io/azure \
     --set configuration.volumeSnapshotLocation[0].config.resourceGroup=rg-velero-backups \
     --set configuration.volumeSnapshotLocation[0].config.subscriptionId=${SUBSCRIPTION_ID} \
     --set initContainers[0].name=velero-plugin-for-microsoft-azure \
     --set initContainers[0].image=velero/velero-plugin-for-microsoft-azure:master \
     --set initContainers[0].volumeMounts[0].mountPath=/target \
     --set initContainers[0].volumeMounts[0].name=plugins \
     --set image.repository=velero/velero \
     --set image.pullPolicy=Always \
     --set backupsEnabled=true \
     --set snapshotsEnabled=true \
     --set-file credentials.secretContents.cloud=./credentials-velero

This should install the chart deploying the Kubernetes Deployment, CRDs, and any other dependencies needed by Velero. The end result is you should have the velero deployment and service in the velero namespace:

$ kubectl get all -n velero
NAME                         READY   STATUS    RESTARTS   AGE
pod/velero-79b6f59d6-hv46x   1/1     Running   0          4d15h

NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/velero   ClusterIP   10.0.13.58   <none>        8085/TCP   23d

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/velero   1/1     1            1           23d

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/velero-69b6f59d6   1         1         1       4d15h

Now that you have the backup agent installed, the next step is to create a backup schedule. Velero uses the Cron syntax for scheduled backups. Using the velero CLI tool, here is how you create a schedule that runs every 6 hours and the backup lives for 30 days (720 hours):

velero create schedule my-aks-cluster --schedule="0 */6 * * *" --ttl 720h0m0s -n velero

A more Kubernetes approach is the create a Kubernetes manifest configuration file (using Velero's Schedule CRD):

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: my-aks-cluster
  namespace: velero
spec:
  schedule: 0 */6 * * *
  template:
    csiSnapshotTimeout: 0s
    hooks: {}
    includedNamespaces:
    - '*'
    itemOperationTimeout: 0s
    metadata: {}
    ttl: 720h0m0s
  useOwnerReferencesInBackup: false

And then apply the configuration with kubectl. Within the next 6 hours, the cluster should be backed up to the service account in the Helm chart configuration variable configuration.backupStorageLocation[0].config.storageAccount.

Next blog post would be how to user Velero's CLI to backup and restore.

References

Windows Container Image - RabbitMQ

After years of using Docker, today was my first day of debugging a Docker container build for Windows. In fact I was so shocked it wasn't a Linux based container. I was like a deer in headlights of how to debug it when it was causing problems. It was a Rabbit MQ image. Fortunately the container has both cmd.exe and powershell.exe. It took a little web searching how to do certain things like cat and tail -f in PowerShell, but before long I was looking at the logs:

docker exec -it rabbitmq powershell.exe
cd \Users\ContainerAdministrator\AppData\Roaming\RabbitMQ\log
Get-Content .\rabbit@localhost.log -tail 100 -Wait

The log:

=WARNING REPORT==== 28-Dec-2023::06:00:39 ===
closing AMQP connection <0.25045.3> (10.0.83.5:60070 -> 172.25.205.23:5672, vhost: '/', user: 'admin'):
client unexpectedly closed TCP connection
=WARNING REPORT==== 28-Dec-2023::06:00:40 ===
closing AMQP connection <0.15059.3> (10.0.83.5:63367 -> 172.25.205.23:5672, vhost: '/', user: 'admin'):
client unexpectedly closed TCP connection
=INFO REPORT==== 28-Dec-2023::06:01:51 ===
accepting AMQP connection <0.25498.3> (10.0.83.5:60211 -> 172.25.205.23:5672)
=INFO REPORT==== 28-Dec-2023::06:01:51 ===
connection <0.25498.3> (10.0.83.5:60211 -> 172.25.205.23:5672): user 'admin' authenticated and granted access to vhost '/'

And then a little more searching how to get the command-line history when wanted to re-run commands:

doskey /history

Azure Workload Identity Federation

I started working on switching out our Azure DevOps service connections to used federated workload identities. There is a good page and Video in Azure about how Workload Identity Federation works:

https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation

Basically the way it works is there is a trust that is setup between Azure DevOps and our Azure subscriptions by using these parameters and their examples:

  • Issuer URL: https://vstoken.dev.azure.com/abcdefc4-ffff-fff-...
  • Subject: sc://org/product/test-emartin-federated
  • Audience: api://AzureADTokenExchange (always)

You then tie that to a service principal in AD that will be used as the identity for doing actions in the subscription. Here is another resource about how to set it up using Terraform:

https://techcommunity.microsoft.com/t5/azure-devops-blog/introduction-to-azure-devops-workload-identity-federation-oidc/ba-p/3908687

Why use Workload identity federation? Up until now the only way to avoid storing service principal secrets for Azure DevOps pipelines was to use a self-hosted Azure DevOps agents with managed identities. Now with Workload identity federation we remove that limitation and enable you to use short-lived tokens for authenticating to Azure. This significantly improves your security posture and removes the need to figure out how to share and rotate secrets. Workload identity federation works with many Azure DevOps tasks, not just the Terraform ones we are focussing on in this article, so you can use it for deploying code and other configuration tasks. I encourage you to learn more about the supported tasks here.

What is Workload identity federation and how does it work Workload identity federation is an OpenID Connect implementation for Azure DevOps that allow you to use short-lived credential free authentication to Azure without the need to provision self-hosted agents with managed identity. You configure a trust between your Azure DevOps organisation and an Azure service principal. Azure DevOps then provides a token that can be used to authenticate to the Azure API.

Here is the terraform:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">=3.0.0"
    }
    azuredevops = {
      source = "microsoft/azuredevops"
      version = ">= 0.9.0"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azuredevops_project" "example" {
  name               = "Example Project"
  visibility         = "private"
  version_control    = "Git"
  work_item_template = "Agile"
  description        = "Managed by Terraform"
}

resource "azurerm_resource_group" "identity" {
  name     = "identity"
  location = "UK South"
}

resource "azurerm_user_assigned_identity" "example" {
  location            = azurerm_resource_group.identity.location
  name                = "example-identity"
  resource_group_name = azurerm_resource_group.identity.name
}

resource "azuredevops_serviceendpoint_azurerm" "example" {
  project_id                             = azuredevops_project.example.id
  service_endpoint_name                  = "example-federated-sc"
  description                            = "Managed by Terraform"
  service_endpoint_authentication_scheme = "WorkloadIdentityFederation"
  credentials {
    serviceprincipalid = azurerm_user_assigned_identity.example.client_id
  }
  azurerm_spn_tenantid      = "00000000-0000-0000-0000-000000000000"
  azurerm_subscription_id   = "00000000-0000-0000-0000-000000000000"
  azurerm_subscription_name = "Example Subscription Name"
}

resource "azurerm_federated_identity_credential" "example" {
  name                = "example-federated-credential"
  resource_group_name = azurerm_resource_group.identity.name
  parent_id           = azurerm_user_assigned_identity.example.id
  audience            = ["api://AzureADTokenExchange"]
  issuer              = azuredevops_serviceendpoint_azurerm.example.workload_identity_federation_issuer
  subject             = azuredevops_serviceendpoint_azurerm.example.workload_identity_federation_subject
}

AWS - Reserved Instances

I have been using AWS at work for over 3 years to varying degrees. While I feel comfortable using and administering most things, I realized it is time for me to get serious and fill the gaps in my knowledge. AWS has so many bells and whistles that it is daunting to think you can learn everything and keep that knowledge relevant when day-to-day you probably only use 5% of the features.

To fix these gaps, last month I started taking an AWS course on Udemy that will prepare me for one of the lower level AWS DevOps certifications. Due to distractions with kids and life happening, I am still at the beginning 10% of the course still going over the basics. I am using my main AWS account as a sandbox for trying things out in the course. Today my class got to the EC2 section, I realized that I have not been smart when it comes to saving money on my own AWS workloads. For the last 9 months, this blog has been running in AWS. I have been using On-Demand Pricing not a Reserved Instance. I can save 30% of the cost of the server by getting a reserved instance for a year and roughly 60% for 3 years. I plunked down the money to pay for 1 year.

AWS - Modifying EC2 DeleteOnTermination

Delete on Termination

I created my web server late last year on an EC2 instance on AWS. While I built the instance with terraform, I didn't set the EBS for the EC2 instance's "Delete on Termination" flag to false. That would mean if I would terminate the instance instead of stop it, that my main EBS volume would just disappear. While that's not that big of a deal because I built the webserver with automation and could easily regenerate it quicky. I didn't necessarily want to lose things like server logs.

I started poking around the console looking for how to switch the flag and I was perplexed how to set it after the fact. I went poking around the web and found there was no way to do it! You have to use the aws ec2 modify-instance-attribute CLI command to change it

Parameters for the CLI

You need two things to be able to use the AWS CLI command

  • EC2 instance ID
  • Storage device name

The instance ID was easy to get either by using the console or in a roughshod way using the AWS CLI:

$ aws ec2 describe-instances --output yaml | grep Instance
  Instances:
...
    InstanceId: i-04753
    InstanceType: t2.micro
...

The device name is also easy to find in the console by going to the Storage tab, but can also be found via the CLI:

$ aws ec2 describe-instances --output yaml | grep -A 6 BlockDeviceMappings
    BlockDeviceMappings:
    - DeviceName: /dev/xvda
      Ebs:
        AttachTime: '2021-11-28T03:03:28+00:00'
        DeleteOnTermination: true
        Status: attached
        VolumeId: vol-0e40

That would mean our two parameters would be:

  • EC2 instance ID: i-04753
  • Storage device name: /dev/xvda

Running the CLI

First you need to create a json file that specifies the device name and the DeleteOnTermination flag:

[
  {
    "DeviceName": "/dev/xvda",
    "Ebs": {
      "DeleteOnTermination": false
      }
  }
]

And then you invoke the comand:

aws ec2 modify-instance-attribute --instance-id i-04753 --block-device-mappings file://storage.json

There is no output on a successful change, but you can confirm that the change was made with the same command as above:

$ aws ec2 describe-instances --output yaml | grep -A 6 BlockDeviceMappings
    BlockDeviceMappings:
    - DeviceName: /dev/xvda
      Ebs:
        AttachTime: '2021-11-28T03:03:28+00:00'
        DeleteOnTermination: false
        Status: attached
        VolumeId: vol-0e40

Notice DeleteOnTermination is now set to false.

(HT to Pete Wilcock)

Installing Jenkins in Minikube (M1 Mac)

Introduction

I always wanted to try and setup Jenkins in a Minikube instance. While not necessary, it is important to get DevOps type tools running in a cluster.

Prerequisites

These tools are recommended to get Minikube running:

  • Homebrew - self described "Missing Package Manager for macOS"
  • Minikube - tool to easily create a single node Kubernetes cluster
  • Docker Desktop for M1 - container runtime
  • kubectl - Kubernetes command-line tool (CLI)
  • kubectx/kubens - Convenience tool for changing contexts and namespaces
  • helm - defacto Kubernetes package manager

Homebrew

Homebrew is arguably the best package manager for the Mac to install terminal applications. It has a very simple : 

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Minikube

Minikube is an easy to deploy single node Kubernetes cluster. It used Homebrew to install:

brew install minikube

Docker Desktop

Docker Desktop is required to run Minikube on M1 Macs. Here is the direct link to the installer disk image for for M1 Macs:

https://desktop.docker.com/mac/main/arm64/Docker.dmg

Note that as of the time of this writing. You can't run Minikube in Virtual Box on M1 Macs. You will get this error:

$ minikube start --vm-driver=virtualbox
😄  minikube v1.24.0 on Darwin 12.0.1 (arm64)
✨  Using the virtualbox driver based on user configuration

❌  Exiting due to DRV_UNSUPPORTED_OS: The driver 'virtualbox' is not supported on darwin/arm64

You must use Docker.

kubectl

kubectl is the command-line tool for interacting with your Kubernetes API. It is easily installable via homebrew:

brew install kubectl

kubectx / kubens

The next two tools are not required per se. In my mind they are the easy shortcuts that should have been included with any install of kubectl. Every docker image that I build that

  • kubectx - Change the kubernetes context from one cluster to another
  • kubens - Easily change the default kubernetes namespace

You can do both using kubectl config ... commands, but these

They are both easily installable via homebrew using 1 command

brew install kubectx

More details about kubectx/kubens tools at the Github repository.

Helm

In the same way that Homebrew is the defacto package manager for macOS, helm is for Kubernetes.

brew install helm

Helm will be used to install Jenkins onto the Kubernetes cluster.

Docker Preferences

Kubernetes needs a bunch of resources to run in Docker. For example, if you run this command you will get the following error:

$ minikube start --memory 8192 --cpus 4 --vm-driver=docker
😄  minikube v1.24.0 on Darwin 12.0.1 (arm64)
✨  Using the docker driver based on user configuration

❌  Exiting due to MK_USAGE: Docker Desktop has only 1988MB memory but you specified 8192MB

To fix:

  • Launch Docker Desktop
  • Under the Preferences cog in the upper right choose "Resources"
  • Set CPUs to something suitable for Kubernetes. I chose 4 CPUs and 10 GB of RAM.
  • "Apply and Restart"

Minikube Start

It should now be possible to start minikube:

$ minikube start --memory 8192 --cpus 4 --vm-driver=docker
😄  minikube v1.24.0 on Darwin 12.0.1 (arm64)
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
    > gcr.io/k8s-minikube/kicbase: 321.58 MiB / 321.58 MiB  100.00% 2.38 MiB p/
🔥  Creating docker container (CPUs=4, Memory=8192MB) ...
    > kubeadm.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubelet.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubectl: 41.44 MiB / 41.44 MiB [---------------] 100.00% 3.80 MiB p/s 11s
    > kubeadm: 40.50 MiB / 40.50 MiB [---------------] 100.00% 3.14 MiB p/s 13s
    > kubelet: 107.26 MiB / 107.26 MiB [-------------] 100.00% 5.41 MiB p/s 20s

    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Minikube Dashboard / k9s

At this point you probably want to decide which tool you want to use to help you with troubleshooting issues with the cluster you created. Most minikube documentation points to using the Minikube Dashboard, which is easily invoked:

minikube dashboard

The UI will launch in your default web browser and is very similar to the Kubernetes Dashboard in design.

However, being a Terminal person, I tend to use k9s. It is easy to install, lightweight, and most reminds me of using something like top to get a handle of what is happening in your cluster. In fact I often run it in another pane of my Terminal. Here is how you install it:

brew install k9s

Launching k9s from the Terminal will give you a curses view where you can bounce around. Here is a more complete Medium post on how to use it:

K9s — the powerful terminal UI for Kubernetes

Installing Jenkins into Minikube

Now that all of the pre-requisites are out of the way, it is time to install Jenkins using Helm. The first step is to add the location of the helm repo to your helm install:

helm repo add jenkins https://charts.jenkins.io
helm repo update

You then can search for the latest helm chart by doing the following:

helm search repo jenkins
NAME            CHART VERSION   APP VERSION DESCRIPTION
jenkins/jenkins 3.9.0           2.303.3     Jenkins - Build great things at any scale! The ..

Pull the chart from the repo:

helm pull jenkins/jenkins

This should create a helm chart in the local folder. In this case it is name jenkins-3.9.0.tgz

Helm works by overiding the values from the chart to set your own values. Simply create the values for the chart by doing the following:

helm show values jenkins/jenkins > jenkins-values.yaml

It should have a bunch of stuff that is disabled by default. My values.yaml file was almost 900 lines long with a lot of comments. It is fine to remove most of this. For minikube I mainly want to override the namespace and the persistent volume (PV) that we want to use in the cluster, but before we do that we have to create them. I want jenkins to be installed in the jenkins namespace and I want the PV to be installed locally in my home folder.

First create the namespace and a suitable definition for the PV:

$ kubectl apply -f jenkins-namespace.yaml
namespace/jenkins created
$ kubectl apply -f jenkins-volume.yaml
persistentvolume/jenkins-volume created

Minikube does not come with a LoadBalancer by default so you also have to change the service to a NodePort.

$ helm install jenkins ./jenkins-3.9.0.tgz -n jenkins -f jenkins-values.yaml
NAME: jenkins
LAST DEPLOYED: Sat Nov 27 17:29:40 2021
NAMESPACE: jenkins
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:
  kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/chart-admin-password && echo
...

3. Login with the password from step 1 and the username: admin
4. Configure security realm and authorization strategy
5. Use Jenkins Configuration as Code by specifying configScripts in your values.yaml file, see documentation: http:///configuration-as-code and examples: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos

For more information on running Jenkins on Kubernetes, visit:
https://cloud.google.com/solutions/jenkins-on-container-engine

For more information about Jenkins Configuration as Code, visit:
https://jenkins.io/projects/jcasc/


NOTE: Consider using a custom image with pre-installed plugins

As it mentions in the info above, you have to get the default admin password that is auto-generated. In my case:

$ kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/chart-admin-password && echo
5HrilnfS7eHAVfwDfyKv9B

Due to Kubernetes being launched in Docker, you need to use the minikube service command to tunnel in to launch the Jenkins UI in your default browser:

$ minikube service jenkins -n jenkins
|-----------|---------|-------------|---------------------------|
| NAMESPACE |  NAME   | TARGET PORT |            URL            |
|-----------|---------|-------------|---------------------------|
| jenkins   | jenkins | http/8080   | http://192.168.49.2:30897 |
|-----------|---------|-------------|---------------------------|
🏃  Starting tunnel for service jenkins.
|-----------|---------|-------------|------------------------|
| NAMESPACE |  NAME   | TARGET PORT |          URL           |
|-----------|---------|-------------|------------------------|
| jenkins   | jenkins |             | http://127.0.0.1:52330 |
|-----------|---------|-------------|------------------------|
🎉  Opening service jenkins/jenkins in default browser...
❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.

Use the admin user and the password above to login to the UI.

You can now create a Freestyle or Pipeline Job that will launch agent images within the cluster that execute the build.

You can see the Jenkins Kubernetes cluster configuration under Manage Jenkins > Manage Nodes and Clouds > Configure Clouds > Kubernetes Cloud Details

Uninstall Jenkins and Minikube

If at any point you need to uninstall the K8s, kill the minikube service command with a Ctrl-C and then uninstall the jenkins:

helm uninstall jenkins -n jenkins

And then kill minikube detritus:

minikube stop
minikube delete

Create an AWS VPC From Scratch

Today I went through the process of doing something I have never done before. Using some videos I found on Udemy, I created an AWS VPC from scratch. It is not that I am new to AWS networking, it is just I have always based my instances off existing VPCs, subnets, and network security. To be able to do it from scratch feels like a minor accomplishment. Here is the rough workflow:

  1. Created a VPC

  2. Create the subnets:

    • 3 public subnets in Availability Zone 1a, 1b, and 1c.
    • 3 private subnets in Availability Zone 1d, 1e, 1f

  3. Don't forget that the public subnets have to autoassign IPs (Actions > Modify Auto-assign IPs > Enable auto-assign public IPv4 address)

  4. Create Internet Gateway and attach to VPC (Actions > Attach to VPC)

  5. Edit the default routing table for the public subnets and make sure it can route out the Internet Gateway

  6. Create a routing table for the private subnets that can't go out the Internet Gateway. Associate the private subnets

  7. Create a public security group that allows inbound rules for SSH from my personal IP.

  8. Create a private security group that allows inbound rules for SSH from my personal IP.

After all of that I was able to spin up a quick and dirty terraform file that build a t2.micro instance in the VPC and suprisingly it worked on the first time.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  profile = "default"
  region  = "us-east-1"
}

data "aws_subnet" "public_subnet_1" {
  id = "subnet-XYZ-public"
}

resource "aws_instance" "webserver" {
  ami             = var.ami_id
  instance_type   = var.instance_type
  subnet_id       = data.aws_subnet.public_subnet_1.id
  security_groups = ["sg-public"]
  key_name        = var.key_name

  tags = {
    Name        = "webserver"
    Environment = "prod"
  }
}

xargs

I always wanted to learn how to pass a value to xargs somewhere in the middle of the command. The -I option can do it where the {} is just a token replacer. For example, here is a way to search for an IP prefix in a bunch of text files:

echo 192.168.1. | xargs -I{} grep {} *.txt

Here is how to pass -l to the middle of an ls {} -F command:

$ echo "-l" | xargs -I{} ls {} -F
total 0
drwxr-xr-x@ 7 emartin  staff  224 Nov  6 22:58 Folder1/
drwxr-xr-x@ 7 emartin  staff  224 Nov  6 23:58 Folder2/

I am really going to find this handy to do things like doing a find and then copying the item to a base folder.

You can actually use almost any token for the -I option.

(HT: Stack Exchange)