New to KubeDB? Please start here.
MongoDB Sharding
This tutorial will show you how to use KubeDB to run a sharded MongoDB cluster with arbiter.
Before You Begin
Before proceeding:
Read mongodb arbiter concept to get the concept about MongoDB Replica Set Arbiter.
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using kind.
Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps here.
To keep things isolated, this tutorial uses a separate namespace called demo throughout this tutorial. Run the following command to prepare your cluster for this tutorial:
$ kubectl create ns demo
namespace/demo created
Note: The yaml files used in this tutorial are stored in docs/examples/mongodb folder in GitHub repository kubedb/docs.
Deploy Sharded MongoDB Cluster
To deploy a MongoDB Sharding, user have to specify spec.shardTopology option in Mongodb CRD.
The following is an example of a Mongodb object which creates MongoDB Sharding of three type of members.
apiVersion: kubedb.com/v1alpha2
kind: MongoDB
metadata:
name: mongo-sh-arb
namespace: demo
spec:
version: "4.4.6"
shardTopology:
configServer:
replicas: 3
storage:
resources:
requests:
storage: 500Mi
storageClassName: standard
mongos:
replicas: 2
shard:
replicas: 2
podTemplate:
spec:
resources:
requests:
cpu: "400m"
memory: "300Mi"
shards: 2
storage:
resources:
requests:
storage: 500Mi
storageClassName: standard
arbiter:
podTemplate:
spec:
resources:
requests:
cpu: "200m"
memory: "200Mi"
terminationPolicy: WipeOut
$ kubectl create -f https://github.com/kubedb/docs/raw/v2023.02.28/docs/examples/mongodb/arbiter/sharding.yaml
mongodb.kubedb.com/mongo-sh-arb created
Here,
spec.shardTopology represents the topology configuration for sharding.shard represents configuration for Shard component of mongodb.shards represents number of shards for a mongodb deployment. Each shard is deployed as a replicaset.replicas represents number of replicas of each shard replicaset.prefix represents the prefix of each shard node.configSecret is an optional field to provide custom configuration file for shards (i.e mongod.cnf). If specified, this file will be used as configuration file otherwise a default configuration file will be used.podTemplate is an optional configuration for pods.storage to specify pvc spec for each node of sharding. You can specify any StorageClass available in your cluster with appropriate resource requests.
configServer represents configuration for ConfigServer component of mongodb.replicas represents number of replicas for configServer replicaset. Here, configServer is deployed as a replicaset of mongodb.prefix represents the prefix of configServer nodes.configSecret is an optional field to provide custom configuration file for configSource (i.e mongod.cnf). If specified, this file will be used as configuration file otherwise a default configuration file will be used.podTemplate is an optional configuration for pods.storage to specify pvc spec for each node of configServer. You can specify any StorageClass available in your cluster with appropriate resource requests.
mongos represents configuration for Mongos component of mongodb. Mongos instances run as stateless components (deployment).replicas represents number of replicas of Mongos instance. Here, Mongos is not deployed as replicaset.prefix represents the prefix of mongos nodes.configSecret is an optional field to provide custom configuration file for mongos (i.e mongod.cnf). If specified, this file will be used as configuration file otherwise a default configuration file will be used.podTemplate is an optional configuration for pods.
spec.keyFileSecret (optional) is a secret name that contains keyfile (a random string)against key.txt key. Each mongod instances in the replica set and shardTopology uses the contents of the keyfile as the shared password for authenticating other members in the replicaset. Only mongod instances with the correct keyfile can join the replica set. User can provide the keyFileSecret by creating a secret with key key.txt. See here to create the string for keyFileSecret. If keyFileSecret is not given, KubeDB operator will generate a keyFileSecret itself.spec.arbiter denotes arbiter spec of the deployed MongoDB CRD. There are two fields under it : configSecret & podTemplate. spec.arbiter.configSecret is an optional field to provide custom configuration file for database (i.e mongod.cnf). If specified, this file will be used as configuration file otherwise default configuration file will be used. spec.arbiter.podTemplate holds the arbiter-podSpec. null value of it, instructs kubedb operator to use the default arbiter podTemplate.
KubeDB operator watches for MongoDB objects using Kubernetes api. When a MongoDB object is created, KubeDB operator will create some new StatefulSets : 1 for mongos, 1 for configServer, and 1 for each of the shard & arbiter. It creates a primary Service with the matching MongoDB object name. KubeDB operator will also create governing services for StatefulSets with the name <mongodb-name>-<node-type>-pods.
MongoDB mongo-sh-arb state,
$ kubectl get mg -n demo
NAME VERSION STATUS AGE
mongodb.kubedb.com/mongo-sh-arb 4.4.6 Ready 97s
All the types of nodes Shard, ConfigServer & Mongos are deployed as statefulset.
$ kubectl get statefulset -n demo
NAME READY AGE
statefulset.apps/mongo-sh-arb-configsvr 3/3 97s
statefulset.apps/mongo-sh-arb-mongos 2/2 29s
statefulset.apps/mongo-sh-arb-shard0 2/2 97s
statefulset.apps/mongo-sh-arb-shard0-arbiter 1/1 53s
statefulset.apps/mongo-sh-arb-shard1 2/2 97s
statefulset.apps/mongo-sh-arb-shard1-arbiter 1/1 52s
All PVCs and PVs for MongoDB mongo-sh-arb,
$ kubectl get pvc -n demo
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/datadir-mongo-sh-arb-configsvr-0 Bound pvc-a9589ccb-24c2-4d17-8174-1e552d63d943 500Mi RWO standard 97s
persistentvolumeclaim/datadir-mongo-sh-arb-configsvr-1 Bound pvc-697aa035-6ff2-45c4-8e00-0787b520159b 500Mi RWO standard 75s
persistentvolumeclaim/datadir-mongo-sh-arb-configsvr-2 Bound pvc-2548ee7e-5416-4ddc-960b-33d17bd53b43 500Mi RWO standard 52s
persistentvolumeclaim/datadir-mongo-sh-arb-shard0-0 Bound pvc-a5cdb597-ad01-4362-b56e-c5d6226a38bb 500Mi RWO standard 97s
persistentvolumeclaim/datadir-mongo-sh-arb-shard0-1 Bound pvc-ae9e594a-7370-4339-9f51-6ec07588c8e0 500Mi RWO standard 75s
persistentvolumeclaim/datadir-mongo-sh-arb-shard0-arbiter-0 Bound pvc-8296c2bc-dfc0-47f4-b651-01fb802bf751 500Mi RWO standard 53s
persistentvolumeclaim/datadir-mongo-sh-arb-shard1-0 Bound pvc-33cde211-4ed5-49a9-b7a8-48e94690e12d 500Mi RWO standard 97s
persistentvolumeclaim/datadir-mongo-sh-arb-shard1-1 Bound pvc-569cedf8-b16e-4616-ae1d-74168aacc227 500Mi RWO standard 74s
persistentvolumeclaim/datadir-mongo-sh-arb-shard1-arbiter-0 Bound pvc-c65c7054-a9de-40c4-9797-4d0a730e9c5b 500Mi RWO standard 52s
$ kubectl get pv -n demo
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-2548ee7e-5416-4ddc-960b-33d17bd53b43 500Mi RWO Delete Bound demo/datadir-mongo-sh-arb-configsvr-2 standard 50s
persistentvolume/pvc-33cde211-4ed5-49a9-b7a8-48e94690e12d 500Mi RWO Delete Bound demo/datadir-mongo-sh-arb-shard1-0 standard 93s
persistentvolume/pvc-569cedf8-b16e-4616-ae1d-74168aacc227 500Mi RWO Delete Bound demo/datadir-mongo-sh-arb-shard1-1 standard 71s
persistentvolume/pvc-697aa035-6ff2-45c4-8e00-0787b520159b 500Mi RWO Delete Bound demo/datadir-mongo-sh-arb-configsvr-1 standard 73s
persistentvolume/pvc-8296c2bc-dfc0-47f4-b651-01fb802bf751 500Mi RWO Delete Bound demo/datadir-mongo-sh-arb-shard0-arbiter-0 standard 52s
persistentvolume/pvc-a5cdb597-ad01-4362-b56e-c5d6226a38bb 500Mi RWO Delete Bound demo/datadir-mongo-sh-arb-shard0-0 standard 94s
persistentvolume/pvc-a9589ccb-24c2-4d17-8174-1e552d63d943 500Mi RWO Delete Bound demo/datadir-mongo-sh-arb-configsvr-0 standard 94s
persistentvolume/pvc-ae9e594a-7370-4339-9f51-6ec07588c8e0 500Mi RWO Delete Bound demo/datadir-mongo-sh-arb-shard0-1 standard 73s
persistentvolume/pvc-c65c7054-a9de-40c4-9797-4d0a730e9c5b 500Mi RWO Delete Bound demo/datadir-mongo-sh-arb-shard1-arbiter-0 standard 49s
Services created for MongoDB mongo-sh-arb
$ kubectl get svc -n demo
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mongo-sh-arb ClusterIP 10.96.34.129 <none> 27017/TCP 97s
service/mongo-sh-arb-configsvr-pods ClusterIP None <none> 27017/TCP 97s
service/mongo-sh-arb-mongos-pods ClusterIP None <none> 27017/TCP 97s
service/mongo-sh-arb-shard0-pods ClusterIP None <none> 27017/TCP 97s
service/mongo-sh-arb-shard1-pods ClusterIP None <none> 27017/TCP 97s
KubeDB operator sets the status.phase to Ready once the database is successfully created. It has also defaulted some field of crd object. Run the following command to see the modified MongoDB object:
$ kubectl get mg -n demo mongo-sh-arb -o yaml
apiVersion: kubedb.com/v1alpha2
kind: MongoDB
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"kubedb.com/v1alpha2","kind":"MongoDB","metadata":{"annotations":{},"name":"mongo-sh-arb","namespace":"demo"},"spec":{"arbiter":{"podTemplate":{"spec":{"requests":{"cpu":"200m","memory":"200Mi"},"resources":null}}},"shardTopology":{"configServer":{"replicas":3,"storage":{"resources":{"requests":{"storage":"500Mi"}},"storageClassName":"standard"}},"mongos":{"replicas":2},"shard":{"podTemplate":{"spec":{"resources":{"requests":{"cpu":"400m","memory":"300Mi"}}}},"replicas":2,"shards":2,"storage":{"resources":{"requests":{"storage":"500Mi"}},"storageClassName":"standard"}}},"terminationPolicy":"WipeOut","version":"4.4.6"}}
creationTimestamp: "2022-04-21T09:29:07Z"
finalizers:
- kubedb.com
generation: 3
name: mongo-sh-arb
namespace: demo
resourceVersion: "31916"
uid: 0a31ab30-0002-400e-a312-f7e343ec6894
spec:
allowedSchemas:
namespaces:
from: Same
arbiter:
podTemplate:
controller: {}
metadata: {}
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: mongo-sh-arb
app.kubernetes.io/managed-by: kubedb.com
app.kubernetes.io/name: mongodbs.kubedb.com
mongodb.kubedb.com/node.shard: mongo-sh-arb-shard${SHARD_INDEX}
namespaces:
- demo
topologyKey: kubernetes.io/hostname
weight: 100
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: mongo-sh-arb
app.kubernetes.io/managed-by: kubedb.com
app.kubernetes.io/name: mongodbs.kubedb.com
mongodb.kubedb.com/node.shard: mongo-sh-arb-shard${SHARD_INDEX}
namespaces:
- demo
topologyKey: failure-domain.beta.kubernetes.io/zone
weight: 50
livenessProbe:
exec:
command:
- bash
- -c
- "set -x; if [[ $(mongo admin --host=localhost --quiet --eval \"db.adminCommand('ping').ok\"
) -eq \"1\" ]]; then \n exit 0\n fi\n exit 1"
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
exec:
command:
- bash
- -c
- "set -x; if [[ $(mongo admin --host=localhost --quiet --eval \"db.adminCommand('ping').ok\"
) -eq \"1\" ]]; then \n exit 0\n fi\n exit 1"
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
memory: 1Gi
requests:
cpu: 500m
memory: 1Gi
authSecret:
name: mongo-sh-arb-auth
clusterAuthMode: keyFile
coordinator:
resources: {}
keyFileSecret:
name: mongo-sh-arb-key
shardTopology:
configServer:
podTemplate:
controller: {}
metadata: {}
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: mongo-sh-arb
app.kubernetes.io/managed-by: kubedb.com
app.kubernetes.io/name: mongodbs.kubedb.com
mongodb.kubedb.com/node.config: mongo-sh-arb-configsvr
namespaces:
- demo
topologyKey: kubernetes.io/hostname
weight: 100
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: mongo-sh-arb
app.kubernetes.io/managed-by: kubedb.com
app.kubernetes.io/name: mongodbs.kubedb.com
mongodb.kubedb.com/node.config: mongo-sh-arb-configsvr
namespaces:
- demo
topologyKey: failure-domain.beta.kubernetes.io/zone
weight: 50
livenessProbe:
exec:
command:
- bash
- -c
- "set -x; if [[ $(mongo admin --host=localhost --username=$MONGO_INITDB_ROOT_USERNAME
--password=$MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase=admin
--quiet --eval \"db.adminCommand('ping').ok\" ) -eq \"1\" ]]; then
\n exit 0\n fi\n exit 1"
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
exec:
command:
- bash
- -c
- "set -x; if [[ $(mongo admin --host=localhost --username=$MONGO_INITDB_ROOT_USERNAME
--password=$MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase=admin
--quiet --eval \"db.adminCommand('ping').ok\" ) -eq \"1\" ]]; then
\n exit 0\n fi\n exit 1"
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
memory: 1Gi
requests:
cpu: 500m
memory: 1Gi
serviceAccountName: mongo-sh-arb
replicas: 3
storage:
resources:
requests:
storage: 500Mi
storageClassName: standard
mongos:
podTemplate:
controller: {}
metadata: {}
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: mongo-sh-arb
app.kubernetes.io/managed-by: kubedb.com
app.kubernetes.io/name: mongodbs.kubedb.com
mongodb.kubedb.com/node.mongos: mongo-sh-arb-mongos
namespaces:
- demo
topologyKey: kubernetes.io/hostname
weight: 100
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: mongo-sh-arb
app.kubernetes.io/managed-by: kubedb.com
app.kubernetes.io/name: mongodbs.kubedb.com
mongodb.kubedb.com/node.mongos: mongo-sh-arb-mongos
namespaces:
- demo
topologyKey: failure-domain.beta.kubernetes.io/zone
weight: 50
lifecycle:
preStop:
exec:
command:
- bash
- -c
- 'mongo admin --username=$MONGO_INITDB_ROOT_USERNAME --password=$MONGO_INITDB_ROOT_PASSWORD
--quiet --eval "db.adminCommand({ shutdown: 1 })" || true'
livenessProbe:
exec:
command:
- bash
- -c
- "set -x; if [[ $(mongo admin --host=localhost --username=$MONGO_INITDB_ROOT_USERNAME
--password=$MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase=admin
--quiet --eval \"db.adminCommand('ping').ok\" ) -eq \"1\" ]]; then
\n exit 0\n fi\n exit 1"
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
exec:
command:
- bash
- -c
- "set -x; if [[ $(mongo admin --host=localhost --username=$MONGO_INITDB_ROOT_USERNAME
--password=$MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase=admin
--quiet --eval \"db.adminCommand('ping').ok\" ) -eq \"1\" ]]; then
\n exit 0\n fi\n exit 1"
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
memory: 1Gi
requests:
cpu: 500m
memory: 1Gi
serviceAccountName: mongo-sh-arb
replicas: 2
shard:
podTemplate:
controller: {}
metadata: {}
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: mongo-sh-arb
app.kubernetes.io/managed-by: kubedb.com
app.kubernetes.io/name: mongodbs.kubedb.com
mongodb.kubedb.com/node.shard: mongo-sh-arb-shard${SHARD_INDEX}
namespaces:
- demo
topologyKey: kubernetes.io/hostname
weight: 100
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: mongo-sh-arb
app.kubernetes.io/managed-by: kubedb.com
app.kubernetes.io/name: mongodbs.kubedb.com
mongodb.kubedb.com/node.shard: mongo-sh-arb-shard${SHARD_INDEX}
namespaces:
- demo
topologyKey: failure-domain.beta.kubernetes.io/zone
weight: 50
livenessProbe:
exec:
command:
- bash
- -c
- "set -x; if [[ $(mongo admin --host=localhost --username=$MONGO_INITDB_ROOT_USERNAME
--password=$MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase=admin
--quiet --eval \"db.adminCommand('ping').ok\" ) -eq \"1\" ]]; then
\n exit 0\n fi\n exit 1"
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
exec:
command:
- bash
- -c
- "set -x; if [[ $(mongo admin --host=localhost --username=$MONGO_INITDB_ROOT_USERNAME
--password=$MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase=admin
--quiet --eval \"db.adminCommand('ping').ok\" ) -eq \"1\" ]]; then
\n exit 0\n fi\n exit 1"
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
memory: 300Mi
requests:
cpu: 400m
memory: 300Mi
serviceAccountName: mongo-sh-arb
replicas: 2
shards: 2
storage:
resources:
requests:
storage: 500Mi
storageClassName: standard
sslMode: disabled
storageEngine: wiredTiger
storageType: Durable
terminationPolicy: WipeOut
version: 4.4.6
status:
conditions:
- lastTransitionTime: "2022-04-21T09:29:07Z"
message: 'The KubeDB operator has started the provisioning of MongoDB: demo/mongo-sh-arb'
reason: DatabaseProvisioningStartedSuccessfully
status: "True"
type: ProvisioningStarted
- lastTransitionTime: "2022-04-21T09:30:39Z"
message: All desired replicas are ready.
reason: AllReplicasReady
status: "True"
type: ReplicaReady
- lastTransitionTime: "2022-04-21T09:30:37Z"
message: 'The MongoDB: demo/mongo-sh-arb is accepting client requests.'
observedGeneration: 3
reason: DatabaseAcceptingConnectionRequest
status: "True"
type: AcceptingConnection
- lastTransitionTime: "2022-04-21T09:30:37Z"
message: 'The MongoDB: demo/mongo-sh-arb is ready.'
observedGeneration: 3
reason: ReadinessCheckSucceeded
status: "True"
type: Ready
- lastTransitionTime: "2022-04-21T09:30:39Z"
message: 'The MongoDB: demo/mongo-sh-arb is successfully provisioned.'
observedGeneration: 3
reason: DatabaseSuccessfullyProvisioned
status: "True"
type: Provisioned
observedGeneration: 3
phase: Ready
Please note that KubeDB operator has created a new Secret called mongo-sh-arb-auth (format: {mongodb-object-name}-auth) for storing the password for mongodb superuser. This secret contains a username key which contains the username for MongoDB superuser and a password key which contains the password for MongoDB superuser.
If you want to use custom or existing secret please specify that when creating the MongoDB object using spec.authSecret.name. While creating this secret manually, make sure the secret contains these two keys containing data username and password. For more details, please see here.
Hostname/address: you can use any of these
- Service:
mongo-sh-arb.demo - Pod IP: (
$ kubectl get po -n demo -l mongodb.kubedb.com/node.mongos=mongo-sh-arb-mongos -o yaml | grep podIP)
Port: 27017
Username: Run following command to get username,
$ kubectl get secrets -n demo mongo-sh-arb-auth -o jsonpath='{.data.\username}' | base64 -d
root
Password: Run the following command to get password,
$ kubectl get secrets -n demo mongo-sh-arb-auth -o jsonpath='{.data.\password}' | base64 -d
6&UiN5;qq)Tnai=7
Now, you can connect to this database through mongo-shell.
Sharded Data
In this tutorial, we will insert sharded and unsharded document, and we will see if the data actually sharded across cluster or not.
$ kubectl get po -n demo -l mongodb.kubedb.com/node.mongos=mongo-sh-arb-mongos
NAME READY STATUS RESTARTS AGE
mongo-sh-arb-mongos-0 1/1 Running 0 6m34s
mongo-sh-arb-mongos-1 1/1 Running 0 6m20s
$ kubectl exec -it mongo-sh-arb-mongos-0 -n demo bash
mongodb@mongo-sh-mongos-0:/$ mongo admin -u root -p '6&UiN5;qq)Tnai=7'
MongoDB shell version v4.4.6
connecting to: mongodb://127.0.0.1:27017/admin?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("bf87addd-4245-45b1-a470-fabb3dcc19ab") }
MongoDB server version: 4.4.6
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
https://docs.mongodb.com/
Questions? Try the MongoDB Developer Community Forums
https://community.mongodb.com
---
The server generated these startup warnings when booting:
2022-04-21T09:30:28.259+00:00: You are running this process as the root user, which is not recommended
---
mongos>
To detect if the MongoDB instance that your client is connected to is mongos, use the isMaster command. When a client connects to a mongos, isMaster returns a document with a msg field that holds the string isdbgrid.
mongos> rs.isMaster()
{
"ismaster" : true,
"msg" : "isdbgrid",
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 100000,
"localTime" : ISODate("2022-04-21T09:38:52.370Z"),
"logicalSessionTimeoutMinutes" : 30,
"connectionId" : 253,
"maxWireVersion" : 9,
"minWireVersion" : 0,
"topologyVersion" : {
"processId" : ObjectId("62612434ea3cf5a7339dd36d"),
"counter" : NumberLong(0)
},
"ok" : 1,
"operationTime" : Timestamp(1650533931, 30),
"$clusterTime" : {
"clusterTime" : Timestamp(1650533931, 30),
"signature" : {
"hash" : BinData(0,"QhqwrAXFhPjlpvfTOPwNAESUR8c="),
"keyId" : NumberLong("7088986810746929174")
}
}
}
mongo-sh-arb Shard status,
mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("626123f2f1e4f6821ec73945")
}
shards:
{ "_id" : "shard0", "host" : "shard0/mongo-sh-arb-shard0-0.mongo-sh-arb-shard0-pods.demo.svc.cluster.local:27017,mongo-sh-arb-shard0-1.mongo-sh-arb-shard0-pods.demo.svc.cluster.local:27017", "state" : 1 }
{ "_id" : "shard1", "host" : "shard1/mongo-sh-arb-shard1-0.mongo-sh-arb-shard1-pods.demo.svc.cluster.local:27017,mongo-sh-arb-shard1-1.mongo-sh-arb-shard1-pods.demo.svc.cluster.local:27017", "state" : 1 }
active mongoses:
"4.4.6" : 2
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: yes
Collections with active migrations:
config.system.sessions started at Thu Apr 21 2022 09:39:13 GMT+0000 (UTC)
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
279 : Success
databases:
{ "_id" : "config", "primary" : "config", "partitioned" : true }
config.system.sessions
shard key: { "_id" : 1 }
unique: false
balancing: true
chunks:
shard0 745
shard1 279
too many chunks to print, use verbose if you want to force print
{ "_id" : "kubedb-system", "primary" : "shard0", "partitioned" : true, "version" : { "uuid" : UUID("79db6e4a-dcb1-4f1a-86c2-dcd86a944893"), "lastMod" : 1 } }
kubedb-system.health-check
shard key: { "id" : 1 }
unique: false
balancing: true
chunks:
shard0 1
{ "id" : { "$minKey" : 1 } } -->> { "id" : { "$maxKey" : 1 } } on : shard0 Timestamp(1, 0)
As sh.status() command only shows the data bearing members, if we want to assure that arbiter has been added correctly we need to exec into any shard-pod & run rs.status() command against the admin database. Open another terminal :
kubectl exec -it pod/mongo-sh-arb-shard0-1 -n demo bash
root@mongo-sh-arb-shard0-1:/ mongo admin -u root -p '6&UiN5;qq)Tnai=7'
MongoDB shell version v4.4.6
connecting to: mongodb://127.0.0.1:27017/admin
MongoDB server version: 4.4.6
Welcome to the MongoDB shell.
shard0:PRIMARY> rs.status().members
[
{
"_id" : 0,
"name" : "mongo-sh-arb-shard0-0.mongo-sh-arb-shard0-pods.demo.svc.cluster.local:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 350,
"optime" : {
"ts" : Timestamp(1650535338, 18),
"t" : NumberLong(3)
},
"optimeDurable" : {
"ts" : Timestamp(1650535338, 18),
"t" : NumberLong(3)
},
"optimeDate" : ISODate("2022-04-21T10:02:18Z"),
"optimeDurableDate" : ISODate("2022-04-21T10:02:18Z"),
"lastHeartbeat" : ISODate("2022-04-21T10:02:35.951Z"),
"lastHeartbeatRecv" : ISODate("2022-04-21T10:02:34.999Z"),
"pingMs" : NumberLong(0),
"lastHeartbeatMessage" : "",
"syncSourceHost" : "mongo-sh-arb-shard0-1.mongo-sh-arb-shard0-pods.demo.svc.cluster.local:27017",
"syncSourceId" : 1,
"infoMessage" : "",
"configVersion" : 4,
"configTerm" : 3
},
{
"_id" : 1,
"name" : "mongo-sh-arb-shard0-1.mongo-sh-arb-shard0-pods.demo.svc.cluster.local:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 352,
"optime" : {
"ts" : Timestamp(1650535338, 18),
"t" : NumberLong(3)
},
"optimeDate" : ISODate("2022-04-21T10:02:18Z"),
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"electionTime" : Timestamp(1650535017, 1),
"electionDate" : ISODate("2022-04-21T09:56:57Z"),
"configVersion" : 4,
"configTerm" : 3,
"self" : true,
"lastHeartbeatMessage" : ""
},
{
"_id" : 2,
"name" : "mongo-sh-arb-shard0-arbiter-0.mongo-sh-arb-shard0-pods.demo.svc.cluster.local:27017",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 328,
"lastHeartbeat" : ISODate("2022-04-21T10:02:35.950Z"),
"lastHeartbeatRecv" : ISODate("2022-04-21T10:02:35.585Z"),
"pingMs" : NumberLong(0),
"lastHeartbeatMessage" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"configVersion" : 4,
"configTerm" : 3
}
]
Enable sharding to collection songs.list and insert document. See sh.shardCollection(namespace, key, unique, options) for details about shardCollection command.
mongos> sh.enableSharding("songs");
{
"ok" : 1,
"operationTime" : Timestamp(1650534119, 40),
"$clusterTime" : {
"clusterTime" : Timestamp(1650534119, 40),
"signature" : {
"hash" : BinData(0,"vtfzghRf+pGMDwsY/W3y/irgF1s="),
"keyId" : NumberLong("7088986810746929174")
}
}
}
mongos> sh.shardCollection("songs.list", {"myfield": 1});
{
"collectionsharded" : "songs.list",
"collectionUUID" : UUID("320eccb3-1987-4ac9-affb-61fe2b9284a7"),
"ok" : 1,
"operationTime" : Timestamp(1650534144, 45),
"$clusterTime" : {
"clusterTime" : Timestamp(1650534144, 45),
"signature" : {
"hash" : BinData(0,"F6KJ8uibwEmuoAi4YPvLYFR71eg="),
"keyId" : NumberLong("7088986810746929174")
}
}
}
mongos> use songs
switched to db songs
mongos> db.list.insert({"led zeppelin": "stairway to heaven", "slipknot": "psychosocial"});
WriteResult({ "nInserted" : 1 })
mongos> db.list.insert({"pink floyd": "us and them", "nirvana": "smells like teen spirit", "john lennon" : "imagine" });
WriteResult({ "nInserted" : 1 })
mongos> db.list.find()
{ "_id" : ObjectId("6261275c18807d1843328e08"), "led zeppelin" : "stairway to heaven", "slipknot" : "psychosocial" }
{ "_id" : ObjectId("6261281c18807d1843328e09"), "pink floyd" : "us and them", "nirvana" : "smells like teen spirit", "john lennon" : "imagine" }
Run sh.status() to see whether the songs database has sharding enabled, and the primary shard for the songs database.
The Sharded Collection section sh.status.databases.<collection> provides information on the sharding details for sharded collection(s) (E.g. songs.list). For each sharded collection, the section displays the shard key, the number of chunks per shard(s), the distribution of documents across chunks, and the tag information, if any, for shard key range(s).
mongos> sh.status();
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("626123f2f1e4f6821ec73945")
}
shards:
{ "_id" : "shard0", "host" : "shard0/mongo-sh-arb-shard0-0.mongo-sh-arb-shard0-pods.demo.svc.cluster.local:27017,mongo-sh-arb-shard0-1.mongo-sh-arb-shard0-pods.demo.svc.cluster.local:27017", "state" : 1 }
{ "_id" : "shard1", "host" : "shard1/mongo-sh-arb-shard1-0.mongo-sh-arb-shard1-pods.demo.svc.cluster.local:27017,mongo-sh-arb-shard1-1.mongo-sh-arb-shard1-pods.demo.svc.cluster.local:27017", "state" : 1 }
active mongoses:
"4.4.6" : 2
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
512 : Success
databases:
{ "_id" : "config", "primary" : "config", "partitioned" : true }
config.system.sessions
shard key: { "_id" : 1 }
unique: false
balancing: true
chunks:
shard0 512
shard1 512
too many chunks to print, use verbose if you want to force print
{ "_id" : "kubedb-system", "primary" : "shard0", "partitioned" : true, "version" : { "uuid" : UUID("79db6e4a-dcb1-4f1a-86c2-dcd86a944893"), "lastMod" : 1 } }
kubedb-system.health-check
shard key: { "id" : 1 }
unique: false
balancing: true
chunks:
shard0 1
{ "id" : { "$minKey" : 1 } } -->> { "id" : { "$maxKey" : 1 } } on : shard0 Timestamp(1, 0)
{ "_id" : "songs", "primary" : "shard1", "partitioned" : true, "version" : { "uuid" : UUID("5a61681f-e427-463f-85ca-c1f0d8854a3b"), "lastMod" : 1 } }
songs.list
shard key: { "myfield" : 1 }
unique: false
balancing: true
chunks:
shard1 1
{ "myfield" : { "$minKey" : 1 } } -->> { "myfield" : { "$maxKey" : 1 } } on : shard1 Timestamp(1, 0)
Now create another database where partiotioned is not applied and see how the data is stored.
mongos> use demo
switched to db demo
mongos> db.anothercollection.insert({"myfield": "ccc", "otherfield": "this is non sharded", "kube" : "db" });
WriteResult({ "nInserted" : 1 })
mongos> db.anothercollection.insert({"myfield": "aaa", "more": "field" });
WriteResult({ "nInserted" : 1 })
mongos> db.anothercollection.find()
{ "_id" : ObjectId("626128f618807d1843328e0a"), "myfield" : "ccc", "otherfield" : "this is non sharded", "kube" : "db" }
{ "_id" : ObjectId("6261293c18807d1843328e0b"), "myfield" : "aaa", "more" : "field" }
Now, eventually sh.status()
mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("626123f2f1e4f6821ec73945")
}
shards:
{ "_id" : "shard0", "host" : "shard0/mongo-sh-arb-shard0-0.mongo-sh-arb-shard0-pods.demo.svc.cluster.local:27017,mongo-sh-arb-shard0-1.mongo-sh-arb-shard0-pods.demo.svc.cluster.local:27017", "state" : 1 }
{ "_id" : "shard1", "host" : "shard1/mongo-sh-arb-shard1-0.mongo-sh-arb-shard1-pods.demo.svc.cluster.local:27017,mongo-sh-arb-shard1-1.mongo-sh-arb-shard1-pods.demo.svc.cluster.local:27017", "state" : 1 }
active mongoses:
"4.4.6" : 2
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
512 : Success
databases:
{ "_id" : "config", "primary" : "config", "partitioned" : true }
config.system.sessions
shard key: { "_id" : 1 }
unique: false
balancing: true
chunks:
shard0 512
shard1 512
too many chunks to print, use verbose if you want to force print
{ "_id" : "demo", "primary" : "shard1", "partitioned" : false, "version" : { "uuid" : UUID("8af87f8c-b4ae-4d04-854f-d2ede7465acd"), "lastMod" : 1 } }
{ "_id" : "kubedb-system", "primary" : "shard0", "partitioned" : true, "version" : { "uuid" : UUID("79db6e4a-dcb1-4f1a-86c2-dcd86a944893"), "lastMod" : 1 } }
kubedb-system.health-check
shard key: { "id" : 1 }
unique: false
balancing: true
chunks:
shard0 1
{ "id" : { "$minKey" : 1 } } -->> { "id" : { "$maxKey" : 1 } } on : shard0 Timestamp(1, 0)
{ "_id" : "songs", "primary" : "shard1", "partitioned" : true, "version" : { "uuid" : UUID("5a61681f-e427-463f-85ca-c1f0d8854a3b"), "lastMod" : 1 } }
songs.list
shard key: { "myfield" : 1 }
unique: false
balancing: true
chunks:
shard1 1
{ "myfield" : { "$minKey" : 1 } } -->> { "myfield" : { "$maxKey" : 1 } } on : shard1 Timestamp(1, 0)
Here, demo database is not partitioned and all collections under demo database are stored in it’s primary shard, which is shard0.
Halt Database
When TerminationPolicy is set to halt, and you delete the mongodb object, the KubeDB operator will delete the StatefulSet and its pods but leaves the PVCs, secrets and database backup (snapshots) intact. Learn details of all TerminationPolicy here.
You can also keep the mongodb object and halt the database to resume it again later. If you halt the database, the kubedb will delete the statefulsets and services but will keep the mongodb object, pvcs, secrets and backup (snapshots).
To halt the database, first you have to set the terminationPolicy to Halt in existing database. You can use the below command to set the terminationPolicy to Halt, if it is not already set.
$ kubectl patch -n demo mg/mongo-sh-arb -p '{"spec":{"terminationPolicy":"Halt"}}' --type="merge"
mongodb.kubedb.com/mongo-sh-arb patched
Then, you have to set the spec.halted as true to set the database in a Halted state. You can use the below command.
$ kubectl patch -n demo mg/mongo-sh-arb -p '{"spec":{"halted":true}}' --type="merge"
mongodb.kubedb.com/mongo-sh-arb patched
After that, kubedb will delete the statefulsets and services and you can see the database Phase as Halted.
Now, you can run the following command to get all mongodb resources in demo namespaces,
$ kubectl get mg,sts,svc,secret,pvc -n demo
NAME VERSION STATUS AGE
mongodb.kubedb.com/mongo-sh-arb 4.4.6 Halted 26m
NAME TYPE DATA AGE
secret/default-token-bg2wb kubernetes.io/service-account-token 3 26m
secret/mongo-sh-arb-auth Opaque 2 26m
secret/mongo-sh-arb-key Opaque 1 26m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/datadir-mongo-sh-arb-configsvr-0 Bound pvc-a9589ccb-24c2-4d17-8174-1e552d63d943 500Mi RWO standard 26m
persistentvolumeclaim/datadir-mongo-sh-arb-configsvr-1 Bound pvc-697aa035-6ff2-45c4-8e00-0787b520159b 500Mi RWO standard 26m
persistentvolumeclaim/datadir-mongo-sh-arb-configsvr-2 Bound pvc-2548ee7e-5416-4ddc-960b-33d17bd53b43 500Mi RWO standard 25m
persistentvolumeclaim/datadir-mongo-sh-arb-shard0-0 Bound pvc-a5cdb597-ad01-4362-b56e-c5d6226a38bb 500Mi RWO standard 26m
persistentvolumeclaim/datadir-mongo-sh-arb-shard0-1 Bound pvc-ae9e594a-7370-4339-9f51-6ec07588c8e0 500Mi RWO standard 26m
persistentvolumeclaim/datadir-mongo-sh-arb-shard0-arbiter-0 Bound pvc-8296c2bc-dfc0-47f4-b651-01fb802bf751 500Mi RWO standard 25m
persistentvolumeclaim/datadir-mongo-sh-arb-shard1-0 Bound pvc-33cde211-4ed5-49a9-b7a8-48e94690e12d 500Mi RWO standard 26m
persistentvolumeclaim/datadir-mongo-sh-arb-shard1-1 Bound pvc-569cedf8-b16e-4616-ae1d-74168aacc227 500Mi RWO standard 26m
persistentvolumeclaim/datadir-mongo-sh-arb-shard1-arbiter-0 Bound pvc-c65c7054-a9de-40c4-9797-4d0a730e9c5b 500Mi RWO standard 25m
From the above output, you can see that MongoDB object, PVCs, Secret are still there.
Resume Halted Database
Now, to resume the database, i.e. to get the same database setup back again, you have to set the the spec.halted as false. You can use the below command.
$ kubectl patch -n demo mg/mongo-sh-arb -p '{"spec":{"halted":false}}' --type="merge"
mongodb.kubedb.com/mongo-sh-arb patched
When the database is resumed successfully, you can see the database Status is set to Ready.
$ kubectl get mg -n demo
NAME VERSION STATUS AGE
mongodb.kubedb.com/mongo-sh-arb 4.4.6 Ready 28m
Now, If you again exec into pod and look for previous data, you will see that, all the data persists.
$ kubectl get po -n demo -l mongodb.kubedb.com/node.mongos=mongo-sh-arb-mongos
NAME READY STATUS RESTARTS AGE
mongo-sh-arb-mongos-0 1/1 Running 0 89s
mongo-sh-arb-mongos-1 1/1 Running 0 29s
$ kubectl exec -it mongo-sh-arb-mongos-0 -n demo bash
mongodb@mongo-sh-mongos-0:/$ mongo admin -u root -p '6&UiN5;qq)Tnai=7'
mongos> use songs
switched to db songs
mongos> db.list.find()
{ "_id" : ObjectId("6261275c18807d1843328e08"), "led zeppelin" : "stairway to heaven", "slipknot" : "psychosocial" }
{ "_id" : ObjectId("6261281c18807d1843328e09"), "pink floyd" : "us and them", "nirvana" : "smells like teen spirit", "john lennon" : "imagine" }
mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("626123f2f1e4f6821ec73945")
}
shards:
{ "_id" : "shard0", "host" : "shard0/mongo-sh-arb-shard0-0.mongo-sh-arb-shard0-pods.demo.svc.cluster.local:27017,mongo-sh-arb-shard0-1.mongo-sh-arb-shard0-pods.demo.svc.cluster.local:27017", "state" : 1 }
{ "_id" : "shard1", "host" : "shard1/mongo-sh-arb-shard1-0.mongo-sh-arb-shard1-pods.demo.svc.cluster.local:27017,mongo-sh-arb-shard1-1.mongo-sh-arb-shard1-pods.demo.svc.cluster.local:27017", "state" : 1 }
active mongoses:
"4.4.6" : 2
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 1
Last reported error: Could not find host matching read preference { mode: "primary" } for set shard0
Time of Reported error: Thu Apr 21 2022 09:57:04 GMT+0000 (UTC)
Migration Results for the last 24 hours:
512 : Success
databases:
{ "_id" : "config", "primary" : "config", "partitioned" : true }
config.system.sessions
shard key: { "_id" : 1 }
unique: false
balancing: true
chunks:
shard0 512
shard1 512
too many chunks to print, use verbose if you want to force print
{ "_id" : "demo", "primary" : "shard1", "partitioned" : false, "version" : { "uuid" : UUID("8af87f8c-b4ae-4d04-854f-d2ede7465acd"), "lastMod" : 1 } }
{ "_id" : "kubedb-system", "primary" : "shard0", "partitioned" : true, "version" : { "uuid" : UUID("79db6e4a-dcb1-4f1a-86c2-dcd86a944893"), "lastMod" : 1 } }
kubedb-system.health-check
shard key: { "id" : 1 }
unique: false
balancing: true
chunks:
shard0 1
{ "id" : { "$minKey" : 1 } } -->> { "id" : { "$maxKey" : 1 } } on : shard0 Timestamp(1, 0)
{ "_id" : "songs", "primary" : "shard1", "partitioned" : true, "version" : { "uuid" : UUID("5a61681f-e427-463f-85ca-c1f0d8854a3b"), "lastMod" : 1 } }
songs.list
shard key: { "myfield" : 1 }
unique: false
balancing: true
chunks:
shard1 1
{ "myfield" : { "$minKey" : 1 } } -->> { "myfield" : { "$maxKey" : 1 } } on : shard1 Timestamp(1, 0)
Cleaning up
To cleanup the Kubernetes resources created by this tutorial, run:
kubectl patch -n demo mg/mongo-sh-arb -p '{"spec":{"terminationPolicy":"WipeOut"}}' --type="merge"
kubectl delete -n demo mg/mongo-sh-arb
kubectl delete ns demo
Next Steps
