Velero is an open source Kubernetes backup service. The Velero service runs within the cluster in the velero namespace. It can backup all of the Kubernetes configuration manifests (including Custom Resource Definitions - CRDs) as well as any persistent volumes (PVs) that are attached to pods.

Velero Setup

The Velero setup, install, and configuration is completed using a helm chart. To download the chart clone it from Github:

git clone https://github.com/vmware-tanzu/helm-charts.git
cd helm-charts/charts/velero/

In Azure Entra ID, create an App Registration that will be used as the service account for the backups. I created "AKS Velero Backup" that works across the tenant so I could potentially back up clusters in any of my subscriptions. This service account will backup the configuration and any persistent volumes (PVs) to a storage account.

$ az ad sp list --display-name "AKS Velero Backup" -o table
DisplayName           Id                                    AppId                                 CreatedDateTime
--------------------  ------------------------------------  ------------------------------------  --------------------
AKS Velero Backup     <redacted>                            <redacted>                            2023-12-28T16:22:09Z

Add Contributor IAM privileges to the app ID of "AKS Velero Backup":

az role assignment create --assignee <App ID> --role "Contributor" --scope /subscriptions/<Subscription ID>

The helm chart also requires a credentials file that is used to be able to backup to the Azure storage account:

cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID=<Azure Subscription ID>
AZURE_TENANT_ID=<Azure Tenant ID>
AZURE_CLIENT_ID=<App ID>
AZURE_CLIENT_SECRET=<App ID Secret>
AZURE_RESOURCE_GROUP=<Resource Group of the AKS nodes>
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

With the credentials created, it is just a matter of setting the Helm chart variables using the cloud credential file as a parameter.

${SUBSCRIPTION_ID} is the Azure subscription ID of where the storage account (Azure bucket) lives.
savelerobackups is the storage account name to save the backups
rg-velero-backups is the resource group for the storage account
--set-file credentials.secretContents.cloud is where you set the credentials for the Azure subscription

helm upgrade --install velero velero \
     --repo https://vmware-tanzu.github.io/helm-charts \
     --create-namespace --namespace velero \
     --set configuration.backupStorageLocation[0].name=velero.io/azure \
     --set configuration.backupStorageLocation[0].bucket="my-aks-cluster" \
     --set configuration.backupStorageLocation[0].config.subscriptionId=${SUBSCRIPTION_ID} \
     --set configuration.backupStorageLocation[0].config.storageAccount=savelerobackups \
     --set configuration.backupStorageLocation[0].config.resourceGroup=rg-velero-backups \
     --set configuration.volumeStorageLocation[0].name=velero.io/azure \
     --set configuration.volumeSnapshotLocation[0].config.resourceGroup=rg-velero-backups \
     --set configuration.volumeSnapshotLocation[0].config.subscriptionId=${SUBSCRIPTION_ID} \
     --set initContainers[0].name=velero-plugin-for-microsoft-azure \
     --set initContainers[0].image=velero/velero-plugin-for-microsoft-azure:master \
     --set initContainers[0].volumeMounts[0].mountPath=/target \
     --set initContainers[0].volumeMounts[0].name=plugins \
     --set image.repository=velero/velero \
     --set image.pullPolicy=Always \
     --set backupsEnabled=true \
     --set snapshotsEnabled=true \
     --set-file credentials.secretContents.cloud=./credentials-velero

This should install the chart deploying the Kubernetes Deployment, CRDs, and any other dependencies needed by Velero. The end result is you should have the velero deployment and service in the velero namespace:

$ kubectl get all -n velero
NAME                         READY   STATUS    RESTARTS   AGE
pod/velero-79b6f59d6-hv46x   1/1     Running   0          4d15h

NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/velero   ClusterIP   10.0.13.58   <none>        8085/TCP   23d

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/velero   1/1     1            1           23d

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/velero-69b6f59d6   1         1         1       4d15h

Now that you have the backup agent installed, the next step is to create a backup schedule. Velero uses the Cron syntax for scheduled backups. Using the velero CLI tool, here is how you create a schedule that runs every 6 hours and the backup lives for 30 days (720 hours):

velero create schedule my-aks-cluster --schedule="0 */6 * * *" --ttl 720h0m0s -n velero

A more Kubernetes approach is the create a Kubernetes manifest configuration file (using Velero's Schedule CRD):

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: my-aks-cluster
  namespace: velero
spec:
  schedule: 0 */6 * * *
  template:
    csiSnapshotTimeout: 0s
    hooks: {}
    includedNamespaces:
    - '*'
    itemOperationTimeout: 0s
    metadata: {}
    ttl: 720h0m0s
  useOwnerReferencesInBackup: false

And then apply the configuration with kubectl. Within the next 6 hours, the cluster should be backed up to the service account in the Helm chart configuration variable configuration.backupStorageLocation[0].config.storageAccount.

Next blog post would be how to user Velero's CLI to backup and restore.

References

Helpful Blog: https://bramvandenklinkenberg.com/2022/01/05/azure-aks-quickstart-to-backups-restore-with-velero/
Microsoft-based Blog: https://learn.microsoft.com/en-us/azure/aks/hybrid/backup-workload-cluster

Erik's Thoughts and Musings

Apple, DevOps, Technology, and Reviews

Installing Velero in AKS

Velero Setup

References