Backup SingleStore using KubeStash Auto-Backup
KubeStash can automatically be configured to backup any SingleStore databases in your cluster. KubeStash enables cluster administrators to deploy backup blueprints ahead of time so database owners can easily backup any SingleStore database with a few annotations.
In this tutorial, we are going to show how you can configure a backup blueprint for SingleStore databases in your cluster and backup them with a few annotations.
Before You Begin
- At first, you need to have a Kubernetes cluster, and the
kubectlcommand-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by usingMinikubeorKind. - Install
KubeDBin your cluster following the steps here. - Install
KubeStashin your cluster following the steps here. - Install KubeStash
kubectlplugin following the steps here. - If you are not familiar with how KubeStash backup and restore SingleStore databases, please check the following guide here.
You should be familiar with the following KubeStash concepts:
To keep everything isolated, we are going to use a separate namespace called demo throughout this tutorial.
$ kubectl create ns demo
namespace/demo created
Prepare Backend
We are going to store our backed up data into a GCS bucket. We have to create a Secret with necessary credentials and a BackupStorage CR to use this backend. If you want to use a different backend, please read the respective backend configuration doc from here.
Create Secret:
Let’s create a secret called gcs-secret with access credentials to our desired GCS bucket,
$ echo -n '<your-project-id>' > GOOGLE_PROJECT_ID
$ cat /path/to/downloaded-sa-key.json > GOOGLE_SERVICE_ACCOUNT_JSON_KEY
$ kubectl create secret generic -n demo gcs-secret \
--from-file=./GOOGLE_PROJECT_ID \
--from-file=./GOOGLE_SERVICE_ACCOUNT_JSON_KEY
secret/gcs-secret created
Create BackupStorage:
Now, create a BackupStorage using this secret. Below is the YAML of BackupStorage CR we are going to create,
apiVersion: storage.kubestash.com/v1alpha1
kind: BackupStorage
metadata:
name: gcs-storage
namespace: demo
spec:
storage:
provider: gcs
gcs:
bucket: kubestash-qa
prefix: blueprint
secretName: gcs-secret
usagePolicy:
allowedNamespaces:
from: All
default: true
deletionPolicy: Delete
Let’s create the BackupStorage we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.9.30/docs/guides/singlestore/backup/kubestash/auto-backup/examples/backupstorage.yaml
backupstorage.storage.kubestash.com/gcs-storage created
Create RetentionPolicy:
Now, let’s create a RetentionPolicy to specify how the old Snapshots should be cleaned up.
Below is the YAML of the RetentionPolicy object that we are going to create,
apiVersion: storage.kubestash.com/v1alpha1
kind: RetentionPolicy
metadata:
name: demo-retention
namespace: demo
spec:
default: true
failedSnapshots:
last: 2
maxRetentionPeriod: 2mo
successfulSnapshots:
last: 5
usagePolicy:
allowedNamespaces:
from: All
Let’s create the above RetentionPolicy,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.9.30/docs/guides/singlestore/backup/kubestash/auto-backup/examples/retentionpolicy.yaml
retentionpolicy.storage.kubestash.com/demo-retention created
Create Secret:
We also need to create a secret with a Restic password for backup data encryption.
Let’s create a secret called encrypt-secret with the Restic password,
$ echo -n 'changeit' > RESTIC_PASSWORD
$ kubectl create secret generic -n demo encrypt-secret \
--from-file=./RESTIC_PASSWORD
secret "encrypt-secret" created
Auto-backup with default configurations
In this section, we are going to backup a SingleStore database of demo namespace. We are going to use the default configurations which will be specified in the Backup Blueprint CR.
Prepare Backup Blueprint
A BackupBlueprint allows you to specify a template for the Repository,Session or Variables of BackupConfiguration in a Kubernetes native way.
Now, we have to create a BackupBlueprint CR with a blueprint for BackupConfiguration object.
apiVersion: core.kubestash.com/v1alpha1
kind: BackupBlueprint
metadata:
name: singlestore-default-backup-blueprint
namespace: demo
spec:
usagePolicy:
allowedNamespaces:
from: All
backupConfigurationTemplate:
deletionPolicy: OnDelete
backends:
- name: gcs-backend
storageRef:
namespace: demo
name: gcs-storage
retentionPolicy:
name: demo-retention
namespace: demo
sessions:
- name: frequent-backup
sessionHistoryLimit: 3
scheduler:
schedule: "*/5 * * * *"
jobTemplate:
backoffLimit: 1
repositories:
- name: default-blueprint
backend: gcs-backend
directory: /default-blueprint
encryptionSecret:
name: encrypt-secret
namespace: demo
addon:
name: singlestore-addon
tasks:
- name: logical-backup
Here,
.spec.backupConfigurationTemplate.backends[*].storageRefrefers our earlier createdgcs-storagebackupStorage..spec.backupConfigurationTemplate.sessions[*].schedulespecifies that we want to backup the database at5 minutesinterval.
Let’s create the BackupBlueprint we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.9.30/docs/guides/singlestore/backup/kubestash/auto-backup/examples/default-backupblueprint.yaml
backupblueprint.core.kubestash.com/singlestore-default-backup-blueprint created
Now, we are ready to backup our SingleStore databases using few annotations.
Create SingleStore License Secret
We need SingleStore License to create SingleStore Database. So, Ensure that you have acquired a license and then simply pass the license by secret.
$ kubectl create secret generic -n demo license-secret \
--from-literal=username=license \
--from-literal=password='your-license-set-here'
secret/license-secret created
Create Database
Now, we are going to create an SingleStore CR in demo namespace. Below is the YAML of the SingleStore object that we are going to create,
apiVersion: kubedb.com/v1alpha2
kind: Singlestore
metadata:
name: sample-singlestore
namespace: demo
annotations:
blueprint.kubestash.com/name: singlestore-default-backup-blueprint
blueprint.kubestash.com/namespace: demo
spec:
version: "8.7.10"
topology:
aggregator:
replicas: 2
podTemplate:
spec:
containers:
- name: singlestore
resources:
limits:
memory: "2Gi"
cpu: "600m"
requests:
memory: "2Gi"
cpu: "600m"
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
leaf:
replicas: 3
podTemplate:
spec:
containers:
- name: singlestore
resources:
limits:
memory: "2Gi"
cpu: "600m"
requests:
memory: "2Gi"
cpu: "600m"
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
licenseSecret:
name: license-secret
storageType: Durable
deletionPolicy: WipeOut
Here,
.spec.annotations.blueprint.kubestash.com/name: singlestore-default-backup-blueprintspecifies the name of theBackupBlueprintthat will use in backup..spec.annotations.blueprint.kubestash.com/namespace: demospecifies the name of thenamespacewhere theBackupBlueprintresides.
Let’s create the SingleStore we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.9.30/docs/guides/singlestore/backup/kubestash/auto-backup/examples/sample-singlestore.yaml
singlestore.kubedb.com/sample-singlestore created
Verify BackupConfiguration
If everything is set up correctly, KubeStash will create a BackupConfiguration for our SingleStore instance in the demo namespace. The phase of this BackupConfiguration should be Ready. You can verify the BackupConfiguration object by running the following command,
$ kubectl get backupconfiguration -n demo
NAME PHASE PAUSED AGE
appbinding-sample-singlestore Ready 2m50m
Now, let’s check the YAML of the BackupConfiguration.
$ kubectl get backupconfiguration -n demo appbinding-sample-singlestore -o yaml
apiVersion: core.kubestash.com/v1alpha1
kind: BackupConfiguration
metadata:
creationTimestamp: "2024-08-21T09:47:34Z"
finalizers:
- kubestash.com/cleanup
generation: 1
labels:
app.kubernetes.io/managed-by: kubestash.com
kubestash.com/invoker-name: singlestore-default-backup-blueprint
kubestash.com/invoker-namespace: demo
name: appbinding-sample-singlestore
namespace: demo
resourceVersion: "113911"
uid: eef4c853-4df6-4b5e-b462-977c9b2188c0
spec:
backends:
- name: gcs-backend
retentionPolicy:
name: demo-retention
namespace: demo
storageRef:
name: gcs-storage
namespace: demo
sessions:
- addon:
name: singlestore-addon
tasks:
- name: logical-backup
name: frequent-backup
repositories:
- backend: gcs-backend
directory: /default-blueprint
encryptionSecret:
name: encrypt-secret
namespace: demo
name: default-blueprint
scheduler:
jobTemplate:
backoffLimit: 1
template:
controller: {}
metadata: {}
spec:
resources: {}
schedule: '*/5 * * * *'
sessionHistoryLimit: 3
target:
apiGroup: kubedb.com
kind: Singlestore
name: sample-singlestore
namespace: demo
Notice the spec.backends, spec.sessions and spec.target sections, KubeStash automatically resolved those info from the BackupBluePrint and created above BackupConfiguration.
Verify BackupSession:
KubeStash triggers an instant backup as soon as the BackupConfiguration is ready. After that, backups are scheduled according to the specified schedule.
$ kubectl get backupsession -n demo -w
NAME INVOKER-TYPE INVOKER-NAME PHASE DURATION AGE
appbinding-sample-singlestore-frequent-backup-1724236500 BackupConfiguration appbinding-sample-singlestore Succeeded 7m22s
We can see from the above output that the backup session has succeeded. Now, we are going to verify whether the backed up data has been stored in the backend.
Verify Backup:
Once a backup is complete, KubeStash will update the respective Repository CR to reflect the backup. Check that the repository default-blueprint has been updated by the following command,
$ kubectl get repository -n demo default-blueprint
NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE
default-blueprint true 1 806 B Ready 8m27s 9m18s
At this moment we have one Snapshot. Run the following command to check the respective Snapshot which represents the state of a backup run for an application.
$ kubectl get snapshots -n demo -l=kubestash.com/repo-name=default-blueprint
NAME REPOSITORY SESSION SNAPSHOT-TIME DELETION-POLICY PHASE AGE
default-blueprint-appbinding-sample-singlestore-frequent-backup-1724236500 default-blueprint frequent-backup 2024-01-23T13:10:54Z Delete Succeeded 16h
Note: KubeStash creates a
Snapshotwith the following labels:
kubedb.com/db-version: <db-version>kubestash.com/app-ref-kind: <target-kind>kubestash.com/app-ref-name: <target-name>kubestash.com/app-ref-namespace: <target-namespace>kubestash.com/repo-name: <repository-name>These labels can be used to watch only the
Snapshots related to our target Database orRepository.
If we check the YAML of the Snapshot, we can find the information about the backed up components of the Database.
$ kubectl get snapshots -n demo default-blueprint-appbinding-sample-singlestore-frequent-backup-1724236500 -oyaml
apiVersion: storage.kubestash.com/v1alpha1
kind: Snapshot
metadata:
creationTimestamp: "2024-08-21T10:35:00Z"
finalizers:
- kubestash.com/cleanup
generation: 1
labels:
kubedb.com/db-version: 8.7.10
kubestash.com/app-ref-kind: Singlestore
kubestash.com/app-ref-name: sample-singlestore
kubestash.com/app-ref-namespace: demo
kubestash.com/repo-name: default-blueprint
name: default-blueprint-appbinding-sample-singlestore-frequent-backup-1724236500
namespace: demo
ownerReferences:
- apiVersion: storage.kubestash.com/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Repository
name: default-blueprint
uid: 61e771fc-8262-480c-a9e7-3c5c11c8fd77
resourceVersion: "118423"
uid: 27e2235a-22c1-449a-be92-c53506fe1fe4
spec:
appRef:
apiGroup: kubedb.com
kind: Singlestore
name: sample-singlestore
namespace: demo
backupSession: appbinding-sample-singlestore-frequent-backup-1724236500
deletionPolicy: Delete
repository: default-blueprint
session: frequent-backup
snapshotID: 01J6V48XS6QM489WPKX1MDD4W9
type: FullBackup
version: v1
status:
components:
dump:
driver: Restic
duration: 6.692337543s
integrity: true
path: repository/v1/frequent-backup/dump
phase: Succeeded
resticStats:
- hostPath: dumpfile.sql
id: b83d7a5577940d1c8f5bcda0630592c7d5a04168c272c0e7560bf7dacfe35ea8
size: 3.657 MiB
uploaded: 121.343 KiB
size: 772.958 KiB
integrity: true
phase: Succeeded
size: 772.957 KiB
snapshotTime: "2024-08-21T10:35:00Z"
totalComponents: 1
KubeStash uses the
mysqldumpcommand to take backups of target SingleStore databases. Therefore, the component name forlogical backupsis set asdump.
Now, if we navigate to the GCS bucket, we will see the backed up data stored in the /blueprint/default-blueprint/repository/v1/frequent-backup/dump directory. KubeStash also keeps the backup for Snapshot YAMLs, which can be found in the blueprint/default-blueprintrepository/snapshots directory.
Note: KubeStash stores all dumped data encrypted in the backup directory, meaning it remains unreadable until decrypted.
Auto-backup with custom configurations
In this section, we are going to backup a SingleStore database of demo namespace. We are going to use the custom configurations which will be specified in the BackupBlueprint CR.
Prepare Backup Blueprint
A BackupBlueprint allows you to specify a template for the Repository,Session or Variables of BackupConfiguration in a Kubernetes native way.
Now, we have to create a BackupBlueprint CR with a blueprint for BackupConfiguration object.
apiVersion: core.kubestash.com/v1alpha1
kind: BackupBlueprint
metadata:
name: singlestore-customize-backup-blueprint
namespace: demo
spec:
usagePolicy:
allowedNamespaces:
from: All
backupConfigurationTemplate:
deletionPolicy: OnDelete
# ============== Blueprint for Backends of BackupConfiguration =================
backends:
- name: gcs-backend
storageRef:
namespace: demo
name: gcs-storage
retentionPolicy:
name: demo-retention
namespace: demo
# ============== Blueprint for Sessions of BackupConfiguration =================
sessions:
- name: frequent-backup
sessionHistoryLimit: 3
scheduler:
schedule: ${schedule}
jobTemplate:
backoffLimit: 1
repositories:
- name: ${repoName}
backend: gcs-backend
directory: ${namespace}/${targetName}
encryptionSecret:
name: encrypt-secret
namespace: demo
addon:
name: singlestore-addon
tasks:
- name: logical-backup
params:
databases: ${targetedDatabases}
Note that we have used some variables (format: ${<variable name>}) in different fields. KubeStash will substitute these variables with values from the respective target’s annotations. You’re free to use any variables you like.
Here,
.spec.backupConfigurationTemplate.backends[*].storageRefrefers our earlier createdgcs-storagebackupStorage..spec.backupConfigurationTemplate.sessions[*]:.scheduledefines${schedule}variable, which determines the time interval for the backup..repositories[*].namedefines the${repoName}variable, which specifies the name of the backupRepository..repositories[*].directorydefines two variables,${namespace}and${targetName}, which are used to determine the path where the backup will be stored..addon.tasks[*]databasesdefines${targetedDatabases}variable, which identifies list of databases to backup.
Let’s create the BackupBlueprint we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.9.30/docs/guides/singlestore/backup/kubestash/auto-backup/examples/customize-backupblueprint.yaml
backupblueprint.core.kubestash.com/singlestore-customize-backup-blueprint created
Now, we are ready to backup our SingleStore databases using few annotations. You can check available auto-backup annotations for a databases from here.
Create Database
Now, we are going to create an SingleStore CR in demo namespace. Below is the YAML of the SingleStore object that we are going to create,
apiVersion: kubedb.com/v1alpha2
kind: Singlestore
metadata:
name: sample-singlestore-2
namespace: demo
annotations:
blueprint.kubestash.com/name: singlestore-customize-backup-blueprint
blueprint.kubestash.com/namespace: demo
variables.kubestash.com/schedule: "*/10 * * * *"
variables.kubestash.com/repoName: customize-blueprint
variables.kubestash.com/namespace: demo
variables.kubestash.com/targetName: sample-singlestore-2
spec:
version: "8.7.10"
topology:
aggregator:
replicas: 2
podTemplate:
spec:
containers:
- name: singlestore
resources:
limits:
memory: "2Gi"
cpu: "600m"
requests:
memory: "2Gi"
cpu: "600m"
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
leaf:
replicas: 3
podTemplate:
spec:
containers:
- name: singlestore
resources:
limits:
memory: "2Gi"
cpu: "600m"
requests:
memory: "2Gi"
cpu: "600m"
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
licenseSecret:
name: license-secret
storageType: Durable
deletionPolicy: WipeOut
Notice the metadata.annotations field, where we have defined the annotations related to the automatic backup configuration. Specifically, we’ve set the BackupBlueprint name as singlestore-customize-backup-blueprint and the namespace as demo. We have also provided values for the blueprint template variables, such as the backup schedule, repositoryName, namespace, and targetName. These annotations will be used to create a BackupConfiguration for this SingleStore database.
Let’s create the SingleStore we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.9.30/docs/guides/singlestore/backup/kubestash/auto-backup/examples/sample-singlestore-2.yaml
singlestore.kubedb.com/sample-singlestore-2 created
Verify BackupConfiguration
If everything goes well, KubeStash should create a BackupConfiguration for our SingleStore in demo namespace and the phase of that BackupConfiguration should be Ready. Verify the BackupConfiguration object by the following command,
$ kubectl get backupconfiguration -n demo
NAME PHASE PAUSED AGE
appbinding-sample-singlestore-2 Ready 2m50m
Now, let’s check the YAML of the BackupConfiguration.
$ kubectl get backupconfiguration -n demo appbinding-sample-singlestore-2 -o yaml
apiVersion: core.kubestash.com/v1alpha1
kind: BackupConfiguration
metadata:
creationTimestamp: "2024-08-21T12:55:38Z"
finalizers:
- kubestash.com/cleanup
generation: 1
labels:
app.kubernetes.io/managed-by: kubestash.com
kubestash.com/invoker-name: singlestore-customize-backup-blueprint
kubestash.com/invoker-namespace: demo
name: appbinding-sample-singlestore-2
namespace: demo
resourceVersion: "129124"
uid: eb42b736-c9c9-4280-8379-bbb581790185
spec:
backends:
- name: gcs-backend
retentionPolicy:
name: demo-retention
namespace: demo
storageRef:
name: gcs-storage
namespace: demo
sessions:
- addon:
name: singlestore-addon
tasks:
- name: logical-backup
name: frequent-backup
repositories:
- backend: gcs-backend
directory: demo/sample-singlestore-2
encryptionSecret:
name: encrypt-secret
namespace: demo
name: customize-blueprint
scheduler:
jobTemplate:
backoffLimit: 1
template:
controller: {}
metadata: {}
spec:
resources: {}
schedule: '*/10 * * * *'
sessionHistoryLimit: 3
target:
apiGroup: kubedb.com
kind: Singlestore
name: sample-singlestore-2
namespace: demo
Notice the spec.backends, spec.sessions and spec.target sections, KubeStash automatically resolved those info from the BackupBluePrint and created above BackupConfiguration.
Verify BackupSession:
KubeStash triggers an instant backup as soon as the BackupConfiguration is ready. After that, backups are scheduled according to the specified schedule.
$ kubectl get backupsession -n demo -w
NAME INVOKER-TYPE INVOKER-NAME PHASE DURATION AGE
appbinding-sample-singlestore-2-frequent-backup-1725007200 BackupConfiguration appbinding-sample-singlestore-2 Succeeded 7m22s
We can see from the above output that the backup session has succeeded. Now, we are going to verify whether the backed up data has been stored in the backend.
Verify Backup:
Once a backup is complete, KubeStash will update the respective Repository CR to reflect the backup. Check that the repository customize-blueprint has been updated by the following command,
$ kubectl get repository -n demo customize-blueprint
NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE
customize-blueprint true 1 806 B Ready 8m27s 9m18s
At this moment we have one Snapshot. Run the following command to check the respective Snapshot which represents the state of a backup run for an application.
$ kubectl get snapshots -n demo -l=kubestash.com/repo-name=customize-blueprint
NAME REPOSITORY SESSION SNAPSHOT-TIME DELETION-POLICY PHASE AGE
customize-blueprint-appbinding-sample-singlestore-2-frequent-backup-1725007200 customize-blueprint frequent-backup 2024-01-23T13:10:54Z Delete Succeeded 16h
Note: KubeStash creates a
Snapshotwith the following labels:
kubedb.com/db-version: <db-version>kubestash.com/app-ref-kind: <target-kind>kubestash.com/app-ref-name: <target-name>kubestash.com/app-ref-namespace: <target-namespace>kubestash.com/repo-name: <repository-name>These labels can be used to watch only the
Snapshots related to our target Database orRepository.
If we check the YAML of the Snapshot, we can find the information about the backed up components of the Database.
$ kubectl get snapshots -n demo customize-blueprint-appbinding-sample-singlestore-2-frequent-backup-1725007200 -oyaml
apiVersion: storage.kubestash.com/v1alpha1
kind: Snapshot
metadata:
creationTimestamp: "2024-08-21T10:35:00Z"
finalizers:
- kubestash.com/cleanup
generation: 1
labels:
kubedb.com/db-version: 8.7.10
kubestash.com/app-ref-kind: Singlestore
kubestash.com/app-ref-name: sample-singlestore
kubestash.com/app-ref-namespace: demo
kubestash.com/repo-name: customize-blueprint
name: customize-blueprint-appbinding-sample-singlestore-2-frequent-backup-1725007200
namespace: demo
ownerReferences:
- apiVersion: storage.kubestash.com/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Repository
name: customize-blueprint
uid: 61e771fc-8262-480c-a9e7-3c5c11c8fd77
resourceVersion: "118423"
uid: 27e2235a-22c1-449a-be92-c53506fe1fe4
spec:
appRef:
apiGroup: kubedb.com
kind: Singlestore
name: sample-singlestore-2
namespace: demo
backupSession: appbinding-sample-singlestore-2-frequent-backup-1725007200
deletionPolicy: Delete
repository: customize-blueprint
session: frequent-backup
snapshotID: 01J6V48XZW2BJ02GSS4YBW3TWX
type: FullBackup
version: v1
status:
components:
dump:
driver: Restic
duration: 6.692337543s
integrity: true
path: repository/v1/frequent-backup/dump
phase: Succeeded
resticStats:
- hostPath: dumpfile.sql
id: b83d7a5577940d1c8f5bcda0630592c7d5a04168c272c0e7560bf7dacfe35ea8
size: 3.657 MiB
uploaded: 121.343 KiB
size: 772.958 KiB
integrity: true
phase: Succeeded
size: 772.957 KiB
snapshotTime: "2024-08-21T10:35:00Z"
totalComponents: 1
KubeStash uses the
mysqldumpcommand to take backups of target SingleStore databases. Therefore, the component name forlogical backupsis set asdump.
Now, if we navigate to the GCS bucket, we will see the backed up data stored in the /blueprint/custom-blueprint/repository/v1/frequent-backup/dump directory. KubeStash also keeps the backup for Snapshot YAMLs, which can be found in the blueprint/custom-blueprint/snapshots directory.
Note: KubeStash stores all dumped data encrypted in the backup directory, meaning it remains unreadable until decrypted.
Cleanup
To cleanup the resources crated by this tutorial, run the following commands,
kubectl delete backupblueprints.core.kubestash.com -n demo singlestore-default-backup-blueprint
kubectl delete backupblueprints.core.kubestash.com -n demo singlestore-customize-backup-blueprint
kubectl delete backupstorage -n demo gcs-storage
kubectl delete secret -n demo gcs-secret
kubectl delete secret -n demo encrypt-secret
kubectl delete retentionpolicies.storage.kubestash.com -n demo demo-retention
kubectl delete my -n demo sample-singlestore
kubectl delete my -n demo sample-singlestore-2