Velero comes with a pretty handy command-line interface (CLI) for pretty much anything you want to do regarding backups and restores:
- Scheduling / Creating backups
- Backup information
- Deleting backups
- Full restores from backups
- Selective restores from backups
To manually schedule, backup or restore, the CLI tool is mandatory. If you simply want to review the state of backups and restores in the cluster, you can use kubectl via the installed Velero. I'll do a followup post about how to use Velero with kubectl
To install Velero CLI, follow the instructions on the Basic Install page:
https://velero.io/docs/v1.8/basic-install/
All of the following examples use velero to invoke actions with the Velero agent. If you are going to do a lot of interaction with the backups, I recommend you set the default namespace via either of the two commands, otherwise you need to add -n velero to all commands below. To set the default namespace:
# Third party tool via https://github.com/ahmetb/kubectx
kubens velero
# More verbose namespace
kubectl config set-context --current --namespace velero
Scheduling / Creating Backups
The first thing that you will want to do with Velero is to create a scheduled backup. To create a backup schedule named my-aks-cluster that runs every 4 hours and will expire a backup after 30 days (720 hours)
velero create schedule my-aks-cluster -n velero --schedule="0 */6 * * *" --ttl 720h0m0s
This will backup all namespaces and all disk volumes in the cluster. At the present time, we don't have any persistent disk volumes in our development or production clusters.
If you want list all of the schedules that have been configured:
$ velero schedule get
NAME STATUS CREATED SCHEDULE BACKUP TTL LAST BACKUP SELECTOR PAUSED
my-aks-cluster Enabled 2023-11-24 15:47:00 -0500 EST 0 */6 * * * 720h0m0s 1h ago <none> false
If you want to do a one off ad-hoc backup named ad-hoc-backup, you can use the scheduled one as a template:
velero backup create ad-hoc-backup --from-schedule my-aks-cluster
Or you can simply create one from scratch by specifying what to include or exclude:
# Create a backup including only the nginx namespace.
velero backup create nginx-backup --include-namespaces nginx
# Create a backup excluding the velero and default namespaces.
velero backup create selective-backup --exclude-namespaces velero,default
Backup Information
To list all backups that have been done on the cluster:
$ velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
my-aks-cluster-20231218180035 Completed 0 0 2023-12-18 13:00:35 -0500 EST 29d default <none>
my-aks-cluster-20231218120035 Completed 0 0 2023-12-18 07:00:35 -0500 EST 29d default <none>
my-aks-cluster-20231218060035 Completed 0 0 2023-12-18 01:00:35 -0500 EST 29d default <none>
my-aks-cluster-20231218000034 Completed 0 0 2023-12-17 19:00:34 -0500 EST 29d default <none>
...
my-aks-cluster-20231120180030 Completed 0 0 2023-11-20 13:00:30 -0500 EST 1d default <none>
my-aks-cluster-20231120120029 Completed 0 0 2023-11-20 07:00:29 -0500 EST 1d default <none>
my-aks-cluster-20231120060029 Completed 0 0 2023-11-20 01:00:29 -0500 EST 1d default <none>
my-aks-cluster-20231120000029 Completed 0 0 2023-11-19 19:00:29 -0500 EST 1d default <none>
my-aks-cluster-20231119180029 Completed 0 0 2023-11-19 13:00:29 -0500 EST 22h default <none>
my-aks-cluster-20231119120028 Completed 0 0 2023-11-19 07:00:28 -0500 EST 16h default <none>
my-aks-cluster-20231119060028 Completed 0 0 2023-11-19 01:00:28 -0500 EST 10h default <none>
my-aks-cluster-20231119000028 Completed 0 0 2023-11-18 19:00:28 -0500 EST 4h default <none>
my-aks-cluster-20231023180022 Completed 0 0 2023-10-23 14:00:22 -0400 EDT 33d default <none>
You can describe an individual backup by using the describe command and choosing the name of the backup:
$ velero backup describe my-aks-cluster-20231218180035
Name: my-aks-cluster-20231218180035
Namespace: velero
...
Phase: Completed
Namespaces:
Included: *
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
...
TTL: 720h0m0s
CSISnapshotTimeout: 10m0s
ItemOperationTimeout: 4h0m0s
...
Started: 2023-12-18 13:00:35 -0500 EST
Completed: 2023-12-18 13:00:52 -0500 EST
Expiration: 2024-01-17 13:00:35 -0500 EST
Total items to be backed up: 1262
Items backed up: 1262
Velero-Native Snapshots: <none included>
You can get the logs of an individual backup by using the logs command:
$ velero backup logs my-aks-cluster-20231218180035
time="2023-12-18T18:00:35Z" level=info msg="Setting up backup temp file" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/controller/backup_controller.go:617"
time="2023-12-18T18:00:35Z" level=info msg="Setting up plugin manager" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/controller/backup_controller.go:624"
time="2023-12-18T18:00:35Z" level=info msg="Getting backup item actions" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/controller/backup_controller.go:628"
time="2023-12-18T18:00:35Z" level=info msg="Setting up backup store to check for backup existence" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/controller/backup_controller.go:633"
time="2023-12-18T18:00:36Z" level=info msg="Writing backup version file" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/backup/backup.go:197"
time="2023-12-18T18:00:36Z" level=info msg="Including namespaces: *" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/backup/backup.go:203"
time="2023-12-18T18:00:36Z" level=info msg="Excluding namespaces: <none>" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/backup/backup.go:204"
time="2023-12-18T18:00:36Z" level=info msg="Including resources: *" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/util/collections/includes_excludes.go:506"
time="2023-12-18T18:00:36Z" level=info msg="Excluding resources: <none>" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/util/collections/includes_excludes.go:507"
time="2023-12-18T18:00:36Z" level=info msg="Backing up all volumes using pod volume backup: false" backup=velero/my-aks-cluster-20231218180035 logSource="pkg/backup/backup.go:222"
...
Deleting Backups
Deleting a backup should not be necessary with the TTL set to 30 days, but here is the mechanism to delete it.
$ velero backup delete my-aks-cluster-20231023180022
Are you sure you want to continue (Y/N)? y
Request to delete backup "my-aks-cluster-20231023180022" submitted successfully.
The backup will be fully deleted after all associated data (disk snapshots, backup files, restores) are removed.
You can also simply delete the backup from the storage account manually.
Full restores
To fully restore all items from a backup into a cluster, supply the backup name to --from-backup
velero restore create --from-backup my-aks-cluster-20231023180022
All restores can be listed in the same way backups can be listed:
$ velero restore get
NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR
nginx-test-backup-20231218173505 nginx-test-backup Completed 2023-12-18 17:35:06 -0500 EST 2023-12-18 17:35:09 -0500 EST 0 1 2023-12-18 17:35:06 -0500 EST <none>
An individual restore can also be described:
$ velero restore describe nginx-test-backup-20231218173505
Name: nginx-test-backup-20231218173505
Namespace: velero
Labels: <none>
Annotations: <none>
Phase: Completed
Total items to be restored: 10
Items restored: 10
Started: 2023-12-18 17:35:06 -0500 EST
Completed: 2023-12-18 17:35:09 -0500 EST
...
As well as logs retrieved:
$ velero restore logs nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:523" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Starting restore of backup velero/nginx-test-backup" logSource="pkg/restore/restore.go:423" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Resource 'serviceaccounts' will be restored into namespace 'nginx-test'" logSource="pkg/restore/restore.go:2293" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Resource 'configmaps' will be restored into namespace 'nginx-test'" logSource="pkg/restore/restore.go:2293" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Resource 'pods' will be restored into namespace 'nginx-test'" logSource="pkg/restore/restore.go:2293" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Resource 'replicasets.apps' will be restored into namespace 'nginx-test'" logSource="pkg/restore/restore.go:2293" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:2206" resource=clusterclasses.cluster.x-k8s.io restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Resource 'endpoints' will be restored into namespace 'nginx-test'" logSource="pkg/restore/restore.go:2293" restore=velero/nginx-test-backup-20231218173505
time="2023-12-18T22:35:07Z" level=info msg="Resource 'services' will be restored into namespace 'nginx-test'" logSource="pkg/restore/restore.go:2293" restore=velero/nginx-test-backup-202312181735
Selective restore
Often you will want to just restore a namespace or a set of namespaces from a backup. In this example it will only restore the test namespace:
velero restore create --from-backup test-backup-20231020134940 --include-namespaces test --include-resources "*"
With the --include-resources you can also choose what Kubernetes resources to restore. This example only restores pods in the test namespace:
velero restore create --from-backup test-backup-20231020134940 --include-namespaces test --include-resources "pods"