You are looking at the documentation of a prior release. To read the documentation of the latest release, please
visit here.
Backup SingleStore using KubeStash Auto-Backup
KubeStash can automatically be configured to backup any SingleStore
databases in your cluster. KubeStash enables cluster administrators to deploy backup blueprints
ahead of time so database owners can easily backup any SingleStore
database with a few annotations.
In this tutorial, we are going to show how you can configure a backup blueprint for SingleStore
databases in your cluster and backup them with a few annotations.
Before You Begin
- At first, you need to have a Kubernetes cluster, and the
kubectl
command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by usingMinikube
orKind
. - Install
KubeDB
in your cluster following the steps here. - Install
KubeStash
in your cluster following the steps here. - Install KubeStash
kubectl
plugin following the steps here. - If you are not familiar with how KubeStash backup and restore SingleStore databases, please check the following guide here.
You should be familiar with the following KubeStash
concepts:
To keep everything isolated, we are going to use a separate namespace called demo
throughout this tutorial.
$ kubectl create ns demo
namespace/demo created
Prepare Backend
We are going to store our backed up data into a GCS bucket. We have to create a Secret with necessary credentials and a BackupStorage
CR to use this backend. If you want to use a different backend, please read the respective backend configuration doc from here.
Create Secret:
Let’s create a secret called gcs-secret
with access credentials to our desired GCS bucket,
$ echo -n '<your-project-id>' > GOOGLE_PROJECT_ID
$ cat /path/to/downloaded-sa-key.json > GOOGLE_SERVICE_ACCOUNT_JSON_KEY
$ kubectl create secret generic -n demo gcs-secret \
--from-file=./GOOGLE_PROJECT_ID \
--from-file=./GOOGLE_SERVICE_ACCOUNT_JSON_KEY
secret/gcs-secret created
Create BackupStorage:
Now, create a BackupStorage
using this secret. Below is the YAML of BackupStorage
CR we are going to create,
apiVersion: storage.kubestash.com/v1alpha1
kind: BackupStorage
metadata:
name: gcs-storage
namespace: demo
spec:
storage:
provider: gcs
gcs:
bucket: kubestash-qa
prefix: blueprint
secretName: gcs-secret
usagePolicy:
allowedNamespaces:
from: All
default: true
deletionPolicy: Delete
Let’s create the BackupStorage we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.8-rc.0/docs/guides/singlestore/backup/kubestash/auto-backup/examples/backupstorage.yaml
backupstorage.storage.kubestash.com/gcs-storage created
Create RetentionPolicy:
Now, let’s create a RetentionPolicy
to specify how the old Snapshots should be cleaned up.
Below is the YAML of the RetentionPolicy
object that we are going to create,
apiVersion: storage.kubestash.com/v1alpha1
kind: RetentionPolicy
metadata:
name: demo-retention
namespace: demo
spec:
default: true
failedSnapshots:
last: 2
maxRetentionPeriod: 2mo
successfulSnapshots:
last: 5
usagePolicy:
allowedNamespaces:
from: All
Let’s create the above RetentionPolicy
,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.8-rc.0/docs/guides/singlestore/backup/kubestash/auto-backup/examples/retentionpolicy.yaml
retentionpolicy.storage.kubestash.com/demo-retention created
Create Secret:
We also need to create a secret with a Restic
password for backup data encryption.
Let’s create a secret called encrypt-secret
with the Restic password,
$ echo -n 'changeit' > RESTIC_PASSWORD
$ kubectl create secret generic -n demo encrypt-secret \
--from-file=./RESTIC_PASSWORD
secret "encrypt-secret" created
Auto-backup with default configurations
In this section, we are going to backup a SingleStore
database of demo
namespace. We are going to use the default configurations which will be specified in the Backup Blueprint
CR.
Prepare Backup Blueprint
A BackupBlueprint
allows you to specify a template for the Repository
,Session
or Variables
of BackupConfiguration
in a Kubernetes native way.
Now, we have to create a BackupBlueprint
CR with a blueprint for BackupConfiguration
object.
apiVersion: core.kubestash.com/v1alpha1
kind: BackupBlueprint
metadata:
name: singlestore-default-backup-blueprint
namespace: demo
spec:
usagePolicy:
allowedNamespaces:
from: All
backupConfigurationTemplate:
deletionPolicy: OnDelete
backends:
- name: gcs-backend
storageRef:
namespace: demo
name: gcs-storage
retentionPolicy:
name: demo-retention
namespace: demo
sessions:
- name: frequent-backup
sessionHistoryLimit: 3
scheduler:
schedule: "*/5 * * * *"
jobTemplate:
backoffLimit: 1
repositories:
- name: default-blueprint
backend: gcs-backend
directory: /default-blueprint
encryptionSecret:
name: encrypt-secret
namespace: demo
addon:
name: singlestore-addon
tasks:
- name: logical-backup
Here,
.spec.backupConfigurationTemplate.backends[*].storageRef
refers our earlier createdgcs-storage
backupStorage..spec.backupConfigurationTemplate.sessions[*].schedule
specifies that we want to backup the database at5 minutes
interval.
Let’s create the BackupBlueprint
we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.8-rc.0/docs/guides/singlestore/backup/kubestash/auto-backup/examples/default-backupblueprint.yaml
backupblueprint.core.kubestash.com/singlestore-default-backup-blueprint created
Now, we are ready to backup our SingleStore
databases using few annotations.
Create SingleStore License Secret
We need SingleStore License to create SingleStore Database. So, Ensure that you have acquired a license and then simply pass the license by secret.
$ kubectl create secret generic -n demo license-secret \
--from-literal=username=license \
--from-literal=password='your-license-set-here'
secret/license-secret created
Create Database
Now, we are going to create an SingleStore
CR in demo namespace. Below is the YAML of the SingleStore object that we are going to create,
apiVersion: kubedb.com/v1alpha2
kind: Singlestore
metadata:
name: sample-singlestore
namespace: demo
annotations:
blueprint.kubestash.com/name: singlestore-default-backup-blueprint
blueprint.kubestash.com/namespace: demo
spec:
version: "8.7.10"
topology:
aggregator:
replicas: 2
podTemplate:
spec:
containers:
- name: singlestore
resources:
limits:
memory: "2Gi"
cpu: "600m"
requests:
memory: "2Gi"
cpu: "600m"
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
leaf:
replicas: 3
podTemplate:
spec:
containers:
- name: singlestore
resources:
limits:
memory: "2Gi"
cpu: "600m"
requests:
memory: "2Gi"
cpu: "600m"
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
licenseSecret:
name: license-secret
storageType: Durable
deletionPolicy: WipeOut
Here,
.spec.annotations.blueprint.kubestash.com/name: singlestore-default-backup-blueprint
specifies the name of theBackupBlueprint
that will use in backup..spec.annotations.blueprint.kubestash.com/namespace: demo
specifies the name of thenamespace
where theBackupBlueprint
resides.
Let’s create the SingleStore
we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.8-rc.0/docs/guides/singlestore/backup/kubestash/auto-backup/examples/sample-singlestore.yaml
singlestore.kubedb.com/sample-singlestore created
Verify BackupConfiguration
If everything is set up correctly, KubeStash will create a BackupConfiguration
for our SingleStore instance in the demo namespace. The phase of this BackupConfiguration
should be Ready. You can verify the BackupConfiguration
object by running the following command,
$ kubectl get backupconfiguration -n demo
NAME PHASE PAUSED AGE
appbinding-sample-singlestore Ready 2m50m
Now, let’s check the YAML of the BackupConfiguration
.
$ kubectl get backupconfiguration -n demo appbinding-sample-singlestore -o yaml
apiVersion: core.kubestash.com/v1alpha1
kind: BackupConfiguration
metadata:
creationTimestamp: "2024-08-21T09:47:34Z"
finalizers:
- kubestash.com/cleanup
generation: 1
labels:
app.kubernetes.io/managed-by: kubestash.com
kubestash.com/invoker-name: singlestore-default-backup-blueprint
kubestash.com/invoker-namespace: demo
name: appbinding-sample-singlestore
namespace: demo
resourceVersion: "113911"
uid: eef4c853-4df6-4b5e-b462-977c9b2188c0
spec:
backends:
- name: gcs-backend
retentionPolicy:
name: demo-retention
namespace: demo
storageRef:
name: gcs-storage
namespace: demo
sessions:
- addon:
name: singlestore-addon
tasks:
- name: logical-backup
name: frequent-backup
repositories:
- backend: gcs-backend
directory: /default-blueprint
encryptionSecret:
name: encrypt-secret
namespace: demo
name: default-blueprint
scheduler:
jobTemplate:
backoffLimit: 1
template:
controller: {}
metadata: {}
spec:
resources: {}
schedule: '*/5 * * * *'
sessionHistoryLimit: 3
target:
apiGroup: kubedb.com
kind: Singlestore
name: sample-singlestore
namespace: demo
Notice the spec.backends
, spec.sessions
and spec.target
sections, KubeStash automatically resolved those info from the BackupBluePrint
and created above BackupConfiguration
.
Verify BackupSession:
KubeStash triggers an instant backup as soon as the BackupConfiguration
is ready. After that, backups are scheduled according to the specified schedule.
$ kubectl get backupsession -n demo -w
NAME INVOKER-TYPE INVOKER-NAME PHASE DURATION AGE
appbinding-sample-singlestore-frequent-backup-1724236500 BackupConfiguration appbinding-sample-singlestore Succeeded 7m22s
We can see from the above output that the backup session has succeeded. Now, we are going to verify whether the backed up data has been stored in the backend.
Verify Backup:
Once a backup is complete, KubeStash will update the respective Repository
CR to reflect the backup. Check that the repository default-blueprint
has been updated by the following command,
$ kubectl get repository -n demo default-blueprint
NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE
default-blueprint true 1 806 B Ready 8m27s 9m18s
At this moment we have one Snapshot
. Run the following command to check the respective Snapshot
which represents the state of a backup run for an application.
$ kubectl get snapshots -n demo -l=kubestash.com/repo-name=default-blueprint
NAME REPOSITORY SESSION SNAPSHOT-TIME DELETION-POLICY PHASE AGE
default-blueprint-appbinding-sample-singlestore-frequent-backup-1724236500 default-blueprint frequent-backup 2024-01-23T13:10:54Z Delete Succeeded 16h
Note: KubeStash creates a
Snapshot
with the following labels:
kubedb.com/db-version: <db-version>
kubestash.com/app-ref-kind: <target-kind>
kubestash.com/app-ref-name: <target-name>
kubestash.com/app-ref-namespace: <target-namespace>
kubestash.com/repo-name: <repository-name>
These labels can be used to watch only the
Snapshot
s related to our target Database orRepository
.
If we check the YAML of the Snapshot
, we can find the information about the backed up components of the Database.
$ kubectl get snapshots -n demo default-blueprint-appbinding-sample-singlestore-frequent-backup-1724236500 -oyaml
apiVersion: storage.kubestash.com/v1alpha1
kind: Snapshot
metadata:
creationTimestamp: "2024-08-21T10:35:00Z"
finalizers:
- kubestash.com/cleanup
generation: 1
labels:
kubedb.com/db-version: 8.7.10
kubestash.com/app-ref-kind: Singlestore
kubestash.com/app-ref-name: sample-singlestore
kubestash.com/app-ref-namespace: demo
kubestash.com/repo-name: default-blueprint
name: default-blueprint-appbinding-sample-singlestore-frequent-backup-1724236500
namespace: demo
ownerReferences:
- apiVersion: storage.kubestash.com/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Repository
name: default-blueprint
uid: 61e771fc-8262-480c-a9e7-3c5c11c8fd77
resourceVersion: "118423"
uid: 27e2235a-22c1-449a-be92-c53506fe1fe4
spec:
appRef:
apiGroup: kubedb.com
kind: Singlestore
name: sample-singlestore
namespace: demo
backupSession: appbinding-sample-singlestore-frequent-backup-1724236500
deletionPolicy: Delete
repository: default-blueprint
session: frequent-backup
snapshotID: 01J6V48XS6QM489WPKX1MDD4W9
type: FullBackup
version: v1
status:
components:
dump:
driver: Restic
duration: 6.692337543s
integrity: true
path: repository/v1/frequent-backup/dump
phase: Succeeded
resticStats:
- hostPath: dumpfile.sql
id: b83d7a5577940d1c8f5bcda0630592c7d5a04168c272c0e7560bf7dacfe35ea8
size: 3.657 MiB
uploaded: 121.343 KiB
size: 772.958 KiB
integrity: true
phase: Succeeded
size: 772.957 KiB
snapshotTime: "2024-08-21T10:35:00Z"
totalComponents: 1
KubeStash uses the
mysqldump
command to take backups of target SingleStore databases. Therefore, the component name forlogical backups
is set asdump
.
Now, if we navigate to the GCS bucket, we will see the backed up data stored in the /blueprint/default-blueprint/repository/v1/frequent-backup/dump
directory. KubeStash also keeps the backup for Snapshot
YAMLs, which can be found in the blueprint/default-blueprintrepository/snapshots
directory.
Note: KubeStash stores all dumped data encrypted in the backup directory, meaning it remains unreadable until decrypted.
Auto-backup with custom configurations
In this section, we are going to backup a SingleStore
database of demo
namespace. We are going to use the custom configurations which will be specified in the BackupBlueprint
CR.
Prepare Backup Blueprint
A BackupBlueprint
allows you to specify a template for the Repository
,Session
or Variables
of BackupConfiguration
in a Kubernetes native way.
Now, we have to create a BackupBlueprint
CR with a blueprint for BackupConfiguration
object.
apiVersion: core.kubestash.com/v1alpha1
kind: BackupBlueprint
metadata:
name: singlestore-customize-backup-blueprint
namespace: demo
spec:
usagePolicy:
allowedNamespaces:
from: All
backupConfigurationTemplate:
deletionPolicy: OnDelete
# ============== Blueprint for Backends of BackupConfiguration =================
backends:
- name: gcs-backend
storageRef:
namespace: demo
name: gcs-storage
retentionPolicy:
name: demo-retention
namespace: demo
# ============== Blueprint for Sessions of BackupConfiguration =================
sessions:
- name: frequent-backup
sessionHistoryLimit: 3
scheduler:
schedule: ${schedule}
jobTemplate:
backoffLimit: 1
repositories:
- name: ${repoName}
backend: gcs-backend
directory: ${namespace}/${targetName}
encryptionSecret:
name: encrypt-secret
namespace: demo
addon:
name: singlestore-addon
tasks:
- name: logical-backup
params:
databases: ${targetedDatabases}
Note that we have used some variables (format: ${<variable name>}
) in different fields. KubeStash will substitute these variables with values from the respective target’s annotations. You’re free to use any variables you like.
Here,
.spec.backupConfigurationTemplate.backends[*].storageRef
refers our earlier createdgcs-storage
backupStorage..spec.backupConfigurationTemplate.sessions[*]
:.schedule
defines${schedule}
variable, which determines the time interval for the backup..repositories[*].name
defines the${repoName}
variable, which specifies the name of the backupRepository
..repositories[*].directory
defines two variables,${namespace}
and${targetName}
, which are used to determine the path where the backup will be stored..addon.tasks[*]databases
defines${targetedDatabases}
variable, which identifies list of databases to backup.
Let’s create the BackupBlueprint
we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.8-rc.0/docs/guides/singlestore/backup/kubestash/auto-backup/examples/customize-backupblueprint.yaml
backupblueprint.core.kubestash.com/singlestore-customize-backup-blueprint created
Now, we are ready to backup our SingleStore
databases using few annotations. You can check available auto-backup annotations for a databases from here.
Create Database
Now, we are going to create an SingleStore
CR in demo namespace. Below is the YAML of the SingleStore object that we are going to create,
apiVersion: kubedb.com/v1alpha2
kind: Singlestore
metadata:
name: sample-singlestore-2
namespace: demo
annotations:
blueprint.kubestash.com/name: singlestore-customize-backup-blueprint
blueprint.kubestash.com/namespace: demo
variables.kubestash.com/schedule: "*/10 * * * *"
variables.kubestash.com/repoName: customize-blueprint
variables.kubestash.com/namespace: demo
variables.kubestash.com/targetName: sample-singlestore-2
spec:
version: "8.7.10"
topology:
aggregator:
replicas: 2
podTemplate:
spec:
containers:
- name: singlestore
resources:
limits:
memory: "2Gi"
cpu: "600m"
requests:
memory: "2Gi"
cpu: "600m"
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
leaf:
replicas: 3
podTemplate:
spec:
containers:
- name: singlestore
resources:
limits:
memory: "2Gi"
cpu: "600m"
requests:
memory: "2Gi"
cpu: "600m"
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
licenseSecret:
name: license-secret
storageType: Durable
deletionPolicy: WipeOut
Notice the metadata.annotations
field, where we have defined the annotations related to the automatic backup configuration. Specifically, we’ve set the BackupBlueprint
name as singlestore-customize-backup-blueprint
and the namespace as demo
. We have also provided values for the blueprint template variables, such as the backup schedule
, repositoryName
, namespace
, and targetName
. These annotations will be used to create a BackupConfiguration
for this SingleStore
database.
Let’s create the SingleStore
we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.8-rc.0/docs/guides/singlestore/backup/kubestash/auto-backup/examples/sample-singlestore-2.yaml
singlestore.kubedb.com/sample-singlestore-2 created
Verify BackupConfiguration
If everything goes well, KubeStash should create a BackupConfiguration
for our SingleStore in demo namespace and the phase of that BackupConfiguration
should be Ready
. Verify the BackupConfiguration
object by the following command,
$ kubectl get backupconfiguration -n demo
NAME PHASE PAUSED AGE
appbinding-sample-singlestore-2 Ready 2m50m
Now, let’s check the YAML of the BackupConfiguration
.
$ kubectl get backupconfiguration -n demo appbinding-sample-singlestore-2 -o yaml
apiVersion: core.kubestash.com/v1alpha1
kind: BackupConfiguration
metadata:
creationTimestamp: "2024-08-21T12:55:38Z"
finalizers:
- kubestash.com/cleanup
generation: 1
labels:
app.kubernetes.io/managed-by: kubestash.com
kubestash.com/invoker-name: singlestore-customize-backup-blueprint
kubestash.com/invoker-namespace: demo
name: appbinding-sample-singlestore-2
namespace: demo
resourceVersion: "129124"
uid: eb42b736-c9c9-4280-8379-bbb581790185
spec:
backends:
- name: gcs-backend
retentionPolicy:
name: demo-retention
namespace: demo
storageRef:
name: gcs-storage
namespace: demo
sessions:
- addon:
name: singlestore-addon
tasks:
- name: logical-backup
name: frequent-backup
repositories:
- backend: gcs-backend
directory: demo/sample-singlestore-2
encryptionSecret:
name: encrypt-secret
namespace: demo
name: customize-blueprint
scheduler:
jobTemplate:
backoffLimit: 1
template:
controller: {}
metadata: {}
spec:
resources: {}
schedule: '*/10 * * * *'
sessionHistoryLimit: 3
target:
apiGroup: kubedb.com
kind: Singlestore
name: sample-singlestore-2
namespace: demo
Notice the spec.backends
, spec.sessions
and spec.target
sections, KubeStash automatically resolved those info from the BackupBluePrint
and created above BackupConfiguration
.
Verify BackupSession:
KubeStash triggers an instant backup as soon as the BackupConfiguration
is ready. After that, backups are scheduled according to the specified schedule.
$ kubectl get backupsession -n demo -w
NAME INVOKER-TYPE INVOKER-NAME PHASE DURATION AGE
appbinding-sample-singlestore-2-frequent-backup-1725007200 BackupConfiguration appbinding-sample-singlestore-2 Succeeded 7m22s
We can see from the above output that the backup session has succeeded. Now, we are going to verify whether the backed up data has been stored in the backend.
Verify Backup:
Once a backup is complete, KubeStash will update the respective Repository
CR to reflect the backup. Check that the repository customize-blueprint
has been updated by the following command,
$ kubectl get repository -n demo customize-blueprint
NAME INTEGRITY SNAPSHOT-COUNT SIZE PHASE LAST-SUCCESSFUL-BACKUP AGE
customize-blueprint true 1 806 B Ready 8m27s 9m18s
At this moment we have one Snapshot
. Run the following command to check the respective Snapshot
which represents the state of a backup run for an application.
$ kubectl get snapshots -n demo -l=kubestash.com/repo-name=customize-blueprint
NAME REPOSITORY SESSION SNAPSHOT-TIME DELETION-POLICY PHASE AGE
customize-blueprint-appbinding-sample-singlestore-2-frequent-backup-1725007200 customize-blueprint frequent-backup 2024-01-23T13:10:54Z Delete Succeeded 16h
Note: KubeStash creates a
Snapshot
with the following labels:
kubedb.com/db-version: <db-version>
kubestash.com/app-ref-kind: <target-kind>
kubestash.com/app-ref-name: <target-name>
kubestash.com/app-ref-namespace: <target-namespace>
kubestash.com/repo-name: <repository-name>
These labels can be used to watch only the
Snapshot
s related to our target Database orRepository
.
If we check the YAML of the Snapshot
, we can find the information about the backed up components of the Database.
$ kubectl get snapshots -n demo customize-blueprint-appbinding-sample-singlestore-2-frequent-backup-1725007200 -oyaml
apiVersion: storage.kubestash.com/v1alpha1
kind: Snapshot
metadata:
creationTimestamp: "2024-08-21T10:35:00Z"
finalizers:
- kubestash.com/cleanup
generation: 1
labels:
kubedb.com/db-version: 8.7.10
kubestash.com/app-ref-kind: Singlestore
kubestash.com/app-ref-name: sample-singlestore
kubestash.com/app-ref-namespace: demo
kubestash.com/repo-name: customize-blueprint
name: customize-blueprint-appbinding-sample-singlestore-2-frequent-backup-1725007200
namespace: demo
ownerReferences:
- apiVersion: storage.kubestash.com/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Repository
name: customize-blueprint
uid: 61e771fc-8262-480c-a9e7-3c5c11c8fd77
resourceVersion: "118423"
uid: 27e2235a-22c1-449a-be92-c53506fe1fe4
spec:
appRef:
apiGroup: kubedb.com
kind: Singlestore
name: sample-singlestore-2
namespace: demo
backupSession: appbinding-sample-singlestore-2-frequent-backup-1725007200
deletionPolicy: Delete
repository: customize-blueprint
session: frequent-backup
snapshotID: 01J6V48XZW2BJ02GSS4YBW3TWX
type: FullBackup
version: v1
status:
components:
dump:
driver: Restic
duration: 6.692337543s
integrity: true
path: repository/v1/frequent-backup/dump
phase: Succeeded
resticStats:
- hostPath: dumpfile.sql
id: b83d7a5577940d1c8f5bcda0630592c7d5a04168c272c0e7560bf7dacfe35ea8
size: 3.657 MiB
uploaded: 121.343 KiB
size: 772.958 KiB
integrity: true
phase: Succeeded
size: 772.957 KiB
snapshotTime: "2024-08-21T10:35:00Z"
totalComponents: 1
KubeStash uses the
mysqldump
command to take backups of target SingleStore databases. Therefore, the component name forlogical backups
is set asdump
.
Now, if we navigate to the GCS bucket, we will see the backed up data stored in the /blueprint/custom-blueprint/repository/v1/frequent-backup/dump
directory. KubeStash also keeps the backup for Snapshot
YAMLs, which can be found in the blueprint/custom-blueprint/snapshots
directory.
Note: KubeStash stores all dumped data encrypted in the backup directory, meaning it remains unreadable until decrypted.
Cleanup
To cleanup the resources crated by this tutorial, run the following commands,
kubectl delete backupblueprints.core.kubestash.com -n demo singlestore-default-backup-blueprint
kubectl delete backupblueprints.core.kubestash.com -n demo singlestore-customize-backup-blueprint
kubectl delete backupstorage -n demo gcs-storage
kubectl delete secret -n demo gcs-secret
kubectl delete secret -n demo encrypt-secret
kubectl delete retentionpolicies.storage.kubestash.com -n demo demo-retention
kubectl delete my -n demo sample-singlestore
kubectl delete my -n demo sample-singlestore-2