You are looking at the documentation of a prior release. To read the documentation of the latest release, please visit here.

Backup MySQL using KubeStash Auto-Backup

KubeStash can automatically be configured to backup any MySQL databases in your cluster. KubeStash enables cluster administrators to deploy backup blueprints ahead of time so database owners can easily backup any MySQL database with a few annotations.

In this tutorial, we are going to show how you can configure a backup blueprint for MySQL databases in your cluster and backup them with a few annotations.

Before You Begin

  • At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using Minikube or Kind.
  • Install KubeDB in your cluster following the steps here.
  • Install KubeStash in your cluster following the steps here.
  • Install KubeStash kubectl plugin following the steps here.
  • If you are not familiar with how KubeStash backup and restore MySQL databases, please check the following guide here.

You should be familiar with the following KubeStash concepts:

To keep everything isolated, we are going to use a separate namespace called demo throughout this tutorial.

$ kubectl create ns demo
namespace/demo created

Prepare Backend

We are going to store our backed up data into a GCS bucket. We have to create a Secret with necessary credentials and a BackupStorage CR to use this backend. If you want to use a different backend, please read the respective backend configuration doc from here.

Create Secret:

Let’s create a secret called gcs-secret with access credentials to our desired GCS bucket,

$ echo -n '<your-project-id>' > GOOGLE_PROJECT_ID
$ cat /path/to/downloaded-sa-key.json > GOOGLE_SERVICE_ACCOUNT_JSON_KEY
$ kubectl create secret generic -n demo gcs-secret \
    --from-file=./GOOGLE_PROJECT_ID \
    --from-file=./GOOGLE_SERVICE_ACCOUNT_JSON_KEY
secret/gcs-secret created

Create BackupStorage:

Now, create a BackupStorage using this secret. Below is the YAML of BackupStorage CR we are going to create,

apiVersion: storage.kubestash.com/v1alpha1
kind: BackupStorage
metadata:
  name: gcs-storage
  namespace: demo
spec:
  storage:
    provider: gcs
    gcs:
      bucket: kubestash-qa
      prefix: blueprint
      secretName: gcs-secret
  usagePolicy:
    allowedNamespaces:
      from: All
  default: true
  deletionPolicy: Delete

Let’s create the BackupStorage we have shown above,

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.8-rc.0/docs/guides/mysql/backup/kubestash/auto-backup/examples/backupstorage.yaml
backupstorage.storage.kubestash.com/gcs-storage created

Create RetentionPolicy:

Now, let’s create a RetentionPolicy to specify how the old Snapshots should be cleaned up.

Below is the YAML of the RetentionPolicy object that we are going to create,

apiVersion: storage.kubestash.com/v1alpha1
kind: RetentionPolicy
metadata:
  name: demo-retention
  namespace: demo
spec:
  default: true
  failedSnapshots:
    last: 2
  maxRetentionPeriod: 2mo
  successfulSnapshots:
    last: 5
  usagePolicy:
    allowedNamespaces:
      from: All

Let’s create the above RetentionPolicy,

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.8-rc.0/docs/guides/mysql/backup/kubestash/auto-backup/examples/retentionpolicy.yaml
retentionpolicy.storage.kubestash.com/demo-retention created

Create Secret:

We also need to create a secret with a Restic password for backup data encryption.

Let’s create a secret called encrypt-secret with the Restic password,

$ echo -n 'changeit' > RESTIC_PASSWORD
$ kubectl create secret generic -n demo encrypt-secret \
    --from-file=./RESTIC_PASSWORD
secret "encrypt-secret" created

Auto-backup with default configurations

In this section, we are going to backup a MySQL database of demo namespace. We are going to use the default configurations which will be specified in the Backup Blueprint CR.

Prepare Backup Blueprint

A BackupBlueprint allows you to specify a template for the Repository,Session or Variables of BackupConfiguration in a Kubernetes native way.

Now, we have to create a BackupBlueprint CR with a blueprint for BackupConfiguration object.

apiVersion: core.kubestash.com/v1alpha1
kind: BackupBlueprint
metadata:
  name: mysql-default-backup-blueprint
  namespace: demo
spec:
  usagePolicy:
    allowedNamespaces:
      from: All
  backupConfigurationTemplate:
    deletionPolicy: OnDelete
    backends:
      - name: gcs-backend
        storageRef:
          namespace: demo
          name: gcs-storage
        retentionPolicy:
          name: demo-retention
          namespace: demo
    sessions:
      - name: frequent-backup
        sessionHistoryLimit: 3
        scheduler:
          schedule: "*/5 * * * *"
          jobTemplate:
            backoffLimit: 1
        repositories:
          - name: default-blueprint
            backend: gcs-backend
            directory: /default-blueprint
            encryptionSecret:
              name: encrypt-secret
              namespace: demo
        addon:
          name: mysql-addon
          tasks:
            - name: logical-backup

Here,

  • .spec.backupConfigurationTemplate.backends[*].storageRef refers our earlier created gcs-storage backupStorage.
  • .spec.backupConfigurationTemplate.sessions[*].schedule specifies that we want to backup the database at 5 minutes interval.

Let’s create the BackupBlueprint we have shown above,

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.8-rc.0/docs/guides/mysql/backup/kubestash/auto-backup/examples/default-backupblueprint.yaml
backupblueprint.core.kubestash.com/mysql-default-backup-blueprint created

Now, we are ready to backup our MySQL databases using few annotations.

Create Database

Now, we are going to create an MySQL CR in demo namespace. Below is the YAML of the MySQL object that we are going to create,

apiVersion: kubedb.com/v1alpha2
kind: MySQL
metadata:
  name: sample-mysql
  namespace: demo
  annotations:
    blueprint.kubestash.com/name: mysql-default-backup-blueprint
    blueprint.kubestash.com/namespace: demo
spec:
  version: "8.2.0"
  replicas: 1
  storageType: Durable
  storage:
    storageClassName: "standard"
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 50Mi
  terminationPolicy: WipeOut

Here,

  • .spec.annotations.blueprint.kubestash.com/name: mysql-default-backup-blueprint specifies the name of the BackupBlueprint that will use in backup.
  • .spec.annotations.blueprint.kubestash.com/namespace: demo specifies the name of the namespace where the BackupBlueprint resides.

Let’s create the MySQL we have shown above,

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.8-rc.0/docs/guides/mysql/backup/kubestash/auto-backup/examples/sample-mysql.yaml
mysql.kubedb.com/sample-mysql created

Verify BackupConfiguration

If everything goes well, KubeStash should create a BackupConfiguration for our MySQL in demo namespace and the phase of that BackupConfiguration should be Ready. Verify the BackupConfiguration object by the following command,

$ kubectl get backupconfiguration -n demo
NAME                      PHASE   PAUSED   AGE
appbinding-sample-mysql   Ready            2m50m

Now, let’s check the YAML of the BackupConfiguration.

$ kubectl get backupconfiguration -n demo appbinding-sample-mysql  -o yaml
apiVersion: core.kubestash.com/v1alpha1
kind: BackupConfiguration
metadata:
  creationTimestamp: "2024-08-21T09:47:34Z"
  finalizers:
  - kubestash.com/cleanup
  generation: 1
  labels:
    app.kubernetes.io/managed-by: kubestash.com
    kubestash.com/invoker-name: mysql-default-backup-blueprint
    kubestash.com/invoker-namespace: demo
  name: appbinding-sample-mysql
  namespace: demo
  resourceVersion: "113911"
  uid: eef4c853-4df6-4b5e-b462-977c9b2188c0
spec:
  backends:
  - name: gcs-backend
    retentionPolicy:
      name: demo-retention
      namespace: demo
    storageRef:
      name: gcs-storage
      namespace: demo
  sessions:
  - addon:
      name: mysql-addon
      tasks:
      - name: logical-backup
    name: frequent-backup
    repositories:
    - backend: gcs-backend
      directory: /default-blueprint
      encryptionSecret:
        name: encrypt-secret
        namespace: demo
      name: default-blueprint
    scheduler:
      jobTemplate:
        backoffLimit: 1
        template:
          controller: {}
          metadata: {}
          spec:
            resources: {}
      schedule: '*/5 * * * *'
    sessionHistoryLimit: 3
  target:
    apiGroup: kubedb.com
    kind: MySQL
    name: sample-mysql
    namespace: demo

Notice the spec.backends, spec.sessions and spec.target sections, KubeStash automatically resolved those info from the BackupBluePrint and created above BackupConfiguration.

Verify BackupSession:

KubeStash triggers an instant backup as soon as the BackupConfiguration is ready. After that, backups are scheduled according to the specified schedule.

$ kubectl get backupsession -n demo -w

NAME                                                 INVOKER-TYPE          INVOKER-NAME               PHASE       DURATION   AGE
appbinding-sample-mysql-frequent-backup-1724236500   BackupConfiguration   appbinding-sample-mysql    Succeeded              7m22s

We can see from the above output that the backup session has succeeded. Now, we are going to verify whether the backed up data has been stored in the backend.

Verify Backup:

Once a backup is complete, KubeStash will update the respective Repository CR to reflect the backup. Check that the repository default-blueprint has been updated by the following command,

$ kubectl get repository -n demo default-blueprint
NAME                    INTEGRITY   SNAPSHOT-COUNT   SIZE    PHASE   LAST-SUCCESSFUL-BACKUP   AGE
default-blueprint          true        1                806 B   Ready   8m27s                    9m18s

At this moment we have one Snapshot. Run the following command to check the respective Snapshot which represents the state of a backup run for an application.

$ kubectl get snapshots -n demo -l=kubestash.com/repo-name=default-blueprint
NAME                                                                 REPOSITORY          SESSION           SNAPSHOT-TIME          DELETION-POLICY   PHASE       AGE
default-blueprint-appbinding-sampleysql-frequent-backup-1724236500   default-blueprint   frequent-backup   2024-01-23T13:10:54Z   Delete            Succeeded   16h

Note: KubeStash creates a Snapshot with the following labels:

  • kubedb.com/db-version: <db-version>
  • kubestash.com/app-ref-kind: <target-kind>
  • kubestash.com/app-ref-name: <target-name>
  • kubestash.com/app-ref-namespace: <target-namespace>
  • kubestash.com/repo-name: <repository-name>

These labels can be used to watch only the Snapshots related to our target Database or Repository.

If we check the YAML of the Snapshot, we can find the information about the backed up components of the Database.

$ kubectl get snapshots -n demo default-blueprint-appbinding-sampleysql-frequent-backup-1724236500 -oyaml
apiVersion: storage.kubestash.com/v1alpha1
kind: Snapshot
metadata:
  creationTimestamp: "2024-08-21T10:35:00Z"
  finalizers:
    - kubestash.com/cleanup
  generation: 1
  labels:
    kubedb.com/db-version: 8.2.0
    kubestash.com/app-ref-kind: MySQL
    kubestash.com/app-ref-name: sample-mysql
    kubestash.com/app-ref-namespace: demo
    kubestash.com/repo-name: default-blueprint
  name: default-blueprint-appbinding-sampleysql-frequent-backup-1724236500
  namespace: demo
  ownerReferences:
    - apiVersion: storage.kubestash.com/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Repository
      name: default-blueprint
      uid: 61e771fc-8262-480c-a9e7-3c5c11c8fd77
  resourceVersion: "118423"
  uid: 27e2235a-22c1-449a-be92-c53506fe1fe4
spec:
  appRef:
    apiGroup: kubedb.com
    kind: MySQL
    name: sample-mysql
    namespace: demo
  backupSession: appbinding-sample-mysql-frequent-backup-1724236500
  deletionPolicy: Delete
  repository: default-blueprint
  session: frequent-backup
  snapshotID: 01J6V48XS6QM489WPKX1MDD4W9
  type: FullBackup
  version: v1
status:
  components:
    dump:
      driver: Restic
      duration: 6.692337543s
      integrity: true
      path: repository/v1/frequent-backup/dump
      phase: Succeeded
      resticStats:
        - hostPath: dumpfile.sql
          id: b83d7a5577940d1c8f5bcda0630592c7d5a04168c272c0e7560bf7dacfe35ea8
          size: 3.657 MiB
          uploaded: 121.343 KiB
      size: 772.958 KiB
  integrity: true
  phase: Succeeded
  size: 772.957 KiB
  snapshotTime: "2024-08-21T10:35:00Z"
  totalComponents: 1

KubeStash uses the mysqldump command to take backups of target MySQL databases. Therefore, the component name for logical backups is set as dump.

Now, if we navigate to the GCS bucket, we will see the backed up data stored in the /blueprint/default-blueprint/repository/v1/frequent-backup/dump directory. KubeStash also keeps the backup for Snapshot YAMLs, which can be found in the blueprint/default-blueprintrepository/snapshots directory.

Note: KubeStash stores all dumped data encrypted in the backup directory, meaning it remains unreadable until decrypted.

Auto-backup with custom configurations

In this section, we are going to backup a MySQL database of demo namespace. We are going to use the custom configurations which will be specified in the BackupBlueprint CR.

Prepare Backup Blueprint

A BackupBlueprint allows you to specify a template for the Repository,Session or Variables of BackupConfiguration in a Kubernetes native way.

Now, we have to create a BackupBlueprint CR with a blueprint for BackupConfiguration object.

apiVersion: core.kubestash.com/v1alpha1
kind: BackupBlueprint
metadata:
  name: mysql-customize-backup-blueprint
  namespace: demo
spec:
  usagePolicy:
    allowedNamespaces:
      from: All
  backupConfigurationTemplate:
    deletionPolicy: OnDelete
    # ============== Blueprint for Backends of BackupConfiguration  =================
    backends:
      - name: gcs-backend
        storageRef:
          namespace: demo
          name: gcs-storage
        retentionPolicy:
          name: demo-retention
          namespace: demo
    # ============== Blueprint for Sessions of BackupConfiguration  =================
    sessions:
      - name: frequent-backup
        sessionHistoryLimit: 3
        scheduler:
          schedule: ${schedule}
          jobTemplate:
            backoffLimit: 1
        repositories:
          - name: ${repoName}
            backend: gcs-backend
            directory: ${namespace}/${targetName}
            encryptionSecret:
              name: encrypt-secret
              namespace: demo
        addon:
          name: mysql-addon
          tasks:
            - name: logical-backup
              params:
                databases: ${targetedDatabases}

Note that we have used some variables (format: ${<variable name>}) in different fields. KubeStash will substitute these variables with values from the respective target’s annotations. You’re free to use any variables you like.

Here,

  • .spec.backupConfigurationTemplate.backends[*].storageRef refers our earlier created gcs-storage backupStorage.
  • .spec.backupConfigurationTemplate.sessions[*]:
    • .schedule defines ${schedule} variable, which determines the time interval for the backup.
    • .repositories[*].name defines the ${repoName} variable, which specifies the name of the backup Repository.
    • .repositories[*].directory defines two variables, ${namespace} and ${targetName}, which are used to determine the path where the backup will be stored.
    • .addon.tasks[*]databases defines ${targetedDatabases} variable, which identifies list of databases to backup.

Let’s create the BackupBlueprint we have shown above,

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.8-rc.0/docs/guides/mysql/backup/kubestash/auto-backup/examples/customize-backupblueprint.yaml
backupblueprint.core.kubestash.com/mysql-customize-backup-blueprint created

Now, we are ready to backup our MySQL databases using few annotations. You can check available auto-backup annotations for a databases from here.

Create Database

Now, we are going to create an MySQL CR in demo namespace. Below is the YAML of the MySQL object that we are going to create,

apiVersion: kubedb.com/v1alpha2
kind: MySQL
metadata:
  name: sample-mysql-2
  namespace: demo
  annotations:
    blueprint.kubestash.com/name: mysql-customize-backup-blueprint
    blueprint.kubestash.com/namespace: demo
    variables.kubestash.com/schedule: "*/10 * * * *"
    variables.kubestash.com/repoName: customize-blueprint
    variables.kubestash.com/namespace: demo
    variables.kubestash.com/targetName: sample-mysql-2
    variables.kubestash.com/targetedDatabases: mysql
spec:
  version: "8.2.0"
  replicas: 1
  storageType: Durable
  storage:
    storageClassName: "standard"
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 50Mi
  terminationPolicy: WipeOut

Notice the metadata.annotations field, where we have defined the annotations related to the automatic backup configuration. Specifically, we’ve set the BackupBlueprint name as mysql-customize-backup-blueprint and the namespace as demo. We have also provided values for the blueprint template variables, such as the backup schedule, repositoryName, namespace, targetName, and targetedDatabases. These annotations will be used to create a BackupConfiguration for this MySQL database.

Let’s create the MySQL we have shown above,

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.8-rc.0/docs/guides/mysql/backup/kubestash/auto-backup/examples/sample-mysql-2.yaml
mysql.kubedb.com/sample-mysql-2 created

Verify BackupConfiguration

If everything goes well, KubeStash should create a BackupConfiguration for our MySQL in demo namespace and the phase of that BackupConfiguration should be Ready. Verify the BackupConfiguration object by the following command,

$ kubectl get backupconfiguration -n demo
NAME                        PHASE   PAUSED   AGE
appbinding-sample-mysql-2   Ready            2m50m

Now, let’s check the YAML of the BackupConfiguration.

$ kubectl get backupconfiguration -n demo appbinding-sample-mysql-2  -o yaml
apiVersion: core.kubestash.com/v1alpha1
kind: BackupConfiguration
metadata:
  creationTimestamp: "2024-08-21T12:55:38Z"
  finalizers:
    - kubestash.com/cleanup
  generation: 1
  labels:
    app.kubernetes.io/managed-by: kubestash.com
    kubestash.com/invoker-name: mysql-customize-backup-blueprint
    kubestash.com/invoker-namespace: demo
  name: appbinding-sample-mysql-2
  namespace: demo
  resourceVersion: "129124"
  uid: eb42b736-c9c9-4280-8379-bbb581790185
spec:
  backends:
    - name: gcs-backend
      retentionPolicy:
        name: demo-retention
        namespace: demo
      storageRef:
        name: gcs-storage
        namespace: demo
  sessions:
    - addon:
        name: mysql-addon
        tasks:
          - name: logical-backup
            params:
              databases: mysql
      name: frequent-backup
      repositories:
        - backend: gcs-backend
          directory: demo/sample-mysql-2
          encryptionSecret:
            name: encrypt-secret
            namespace: demo
          name: customize-blueprint
      scheduler:
        jobTemplate:
          backoffLimit: 1
          template:
            controller: {}
            metadata: {}
            spec:
              resources: {}
        schedule: '*/10 * * * *'
      sessionHistoryLimit: 3
  target:
    apiGroup: kubedb.com
    kind: MySQL
    name: sample-mysql-2
    namespace: demo

Notice the spec.backends, spec.sessions and spec.target sections, KubeStash automatically resolved those info from the BackupBluePrint and created above BackupConfiguration.

Verify BackupSession:

KubeStash triggers an instant backup as soon as the BackupConfiguration is ready. After that, backups are scheduled according to the specified schedule.

$ kubectl get backupsession -n demo -w

NAME                                                   INVOKER-TYPE          INVOKER-NAME                 PHASE       DURATION   AGE
appbinding-sample-mysql-2-frequent-backup-1725007200   BackupConfiguration   appbinding-sample-mysql-2    Succeeded              7m22s

We can see from the above output that the backup session has succeeded. Now, we are going to verify whether the backed up data has been stored in the backend.

Verify Backup:

Once a backup is complete, KubeStash will update the respective Repository CR to reflect the backup. Check that the repository customize-blueprint has been updated by the following command,

$ kubectl get repository -n demo customize-blueprint
NAME                         INTEGRITY   SNAPSHOT-COUNT   SIZE    PHASE   LAST-SUCCESSFUL-BACKUP   AGE
customize-blueprint          true        1                806 B   Ready   8m27s                    9m18s

At this moment we have one Snapshot. Run the following command to check the respective Snapshot which represents the state of a backup run for an application.

$ kubectl get snapshots -n demo -l=kubestash.com/repo-name=customize-blueprint
NAME                                                               REPOSITORY            SESSION           SNAPSHOT-TIME          DELETION-POLICY   PHASE       AGE
customize-blueprint-appbinding-sql-2-frequent-backup-1725007200    customize-blueprint   frequent-backup   2024-01-23T13:10:54Z   Delete            Succeeded   16h

Note: KubeStash creates a Snapshot with the following labels:

  • kubedb.com/db-version: <db-version>
  • kubestash.com/app-ref-kind: <target-kind>
  • kubestash.com/app-ref-name: <target-name>
  • kubestash.com/app-ref-namespace: <target-namespace>
  • kubestash.com/repo-name: <repository-name>

These labels can be used to watch only the Snapshots related to our target Database or Repository.

If we check the YAML of the Snapshot, we can find the information about the backed up components of the Database.

$ kubectl get snapshots -n demo customize-blueprint-appbinding-sql-2-frequent-backup-1725007200 -oyaml
apiVersion: storage.kubestash.com/v1alpha1
kind: Snapshot
metadata:
  creationTimestamp: "2024-08-21T10:35:00Z"
  finalizers:
    - kubestash.com/cleanup
  generation: 1
  labels:
    kubedb.com/db-version: 8.2.0
    kubestash.com/app-ref-kind: MySQL
    kubestash.com/app-ref-name: sample-mysql
    kubestash.com/app-ref-namespace: demo
    kubestash.com/repo-name: customize-blueprint
  name: customize-blueprint-appbinding-sql-2-frequent-backup-1725007200
  namespace: demo
  ownerReferences:
    - apiVersion: storage.kubestash.com/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Repository
      name: customize-blueprint
      uid: 61e771fc-8262-480c-a9e7-3c5c11c8fd77
  resourceVersion: "118423"
  uid: 27e2235a-22c1-449a-be92-c53506fe1fe4
spec:
  appRef:
    apiGroup: kubedb.com
    kind: MySQL
    name: sample-mysql-2
    namespace: demo
  backupSession: appbinding-sample-mysql-2-frequent-backup-1725007200
  deletionPolicy: Delete
  repository: customize-blueprint
  session: frequent-backup
  snapshotID: 01J6V48XZW2BJ02GSS4YBW3TWX
  type: FullBackup
  version: v1
status:
  components:
    dump:
      driver: Restic
      duration: 6.692337543s
      integrity: true
      path: repository/v1/frequent-backup/dump
      phase: Succeeded
      resticStats:
        - hostPath: dumpfile.sql
          id: b83d7a5577940d1c8f5bcda0630592c7d5a04168c272c0e7560bf7dacfe35ea8
          size: 3.657 MiB
          uploaded: 121.343 KiB
      size: 772.958 KiB
  integrity: true
  phase: Succeeded
  size: 772.957 KiB
  snapshotTime: "2024-08-21T10:35:00Z"
  totalComponents: 1

KubeStash uses the mysqldump command to take backups of target MySQL databases. Therefore, the component name for logical backups is set as dump.

Now, if we navigate to the GCS bucket, we will see the backed up data stored in the /blueprint/custom-blueprint/repository/v1/frequent-backup/dump directory. KubeStash also keeps the backup for Snapshot YAMLs, which can be found in the blueprint/custom-blueprint/snapshots directory.

Note: KubeStash stores all dumped data encrypted in the backup directory, meaning it remains unreadable until decrypted.

Cleanup

To cleanup the resources crated by this tutorial, run the following commands,

kubectl delete backupblueprints.core.kubestash.com  -n demo mysql-default-backup-blueprint
kubectl delete backupblueprints.core.kubestash.com  -n demo mysql-customize-backup-blueprint
kubectl delete backupstorage -n demo gcs-storage
kubectl delete secret -n demo gcs-secret
kubectl delete secret -n demo encrypt-secret
kubectl delete retentionpolicies.storage.kubestash.com -n demo demo-retention
kubectl delete my -n demo sample-mysql
kubectl delete my -n demo sample-mysql-2