Sunday, March 18, 2018

replica sets, Replication with mongoDB

Before we get into details of mongoDB replica sets, let's understand what is replication and why is it needed? Replication is a configuration that creates multiple copies of data by "replicating"(transferring) data from one site to one or more sites, provides redundancy and maximum availability for databases and its applications. Database is synchronized to remote sites in real time as soon as changes are loaded in to source database. These remote sites are called Disaster Recovery sites and they are strictly recommended to be spread across different geographical regions, different cities, countries, and even different continents. Nevertheless, some IT configurations do also have deployments called "Near DRs", a DR site that could be located in the same city.

Today's Disaster Recovery deployments
Is DR only used in a real disaster scenario? Well, Let me share my thoughts of Disaster Recovery in today's context. A DR site in today's super fast growing world is a necessity for all models of businesses and data, there's no escape of this expense for Enterprises. What about ROI, does one have to wait for a real disaster to strike to consume it, is that only built for sleeping but otherwise only to wait for a disaster and show its real deal? Well, long gone are those days. Most of the database platforms today support read replicas, while "The read feature" comes at an additional expense for some (like Oracle's active data guard), and others are free by default (mongoDB read replicas, Amazon RDS Read Replicas, and so on). Therefore, a DR site is daily driver of the business, power house for reporting, contributes to the growth of the business significantly. Alright, let's get into the topic.

Replication with mongoDB
The deployment of replication configuration in mongoDB is called, a replica set. Each replica set has below nodes.
Primary node: This is the source of everything, source of input, is always in READ WRITE mode.
Secondary node: This is the copy of the Primary node, a replica copy, is always in RECOVERY mode and also serves READ operations when requested. It is always recommended to have similar resource profile, hardware capacity, as Primary node. Multiple secondary copies could be maintained based on different requirements.
Arbiter node: This is only to help the failover in the event of unforeseen primary node outage, this is optional. Arbiters come at cheap price, they do not require a dedicated hardware and they do not hold copy of primary but helps in election of new optimal Primary by responding to heartbeat and election requests by other replica set members in replication.

                                      (Image by mongoDB Inc.)
Each replica set can have up to 50 members but only maximum of 7 members can participate in election.

How it works?
"mongod", a core background daemon process, handles synchronization of changes between all replica set instances. "mongod" on Primary records all types of database changes in a fixed size collection called oplog (operations log), "oplog.rs". For people with background of Oracle, this could be compared to redo log file, and transaction log file in SQL Server. The "mongod" daemon of every secondary consumes entries from Primary's oplog and applies them to respective secondary. In addition to this, all replica set members also send heartbeat signal to every other replica set instance and get oplog entries. This replication process is asynchronous.

Setup
The replica set configuration is pretty straight forward. The replica setup initialization consists of below two taks, they are triggered automatically by mongod process, no manual intervention is required.
-Initial synchronization
-Ongoing replication

The demo example of the setup consists of 1 primary, 3 secondaries, and 1 arbiter node. For mongod to replicate oplog, each mongod instance should run on different port but IP could remain same. Using this advantage, for demonstration and testing purposes, we will configure replica set on same host. A Production deployment must be set up across different geographical regions for maximum availability and fault tolerance.

I assume you already have a working mongo installation setup of version 3.6. If it is not too late, Please go ahead and download here, and follow these instructions for installation. Be it installation of software, or, setup of replica sets, it does not make much difference in regards to steps that are followed. Here's my demo environment, FYR.
OS and version : Oracle Linux Server release 7.4
mongoDB version : mongoDB 3.6.3
Software install location : /mongo/mongohome/bin
Database location : /mongo/data/db

Let's get into setup.
###########################################################
#Color coding:                                                                                          #
#Italic Blue => syntaxes and commands                                                   #
#Italic bold green => shell prompt                                                            #
#Italic black => syntax output                                                                    #
###########################################################

Step(1) : Create directories.
[mongo@cloud2 ~]$ mkdir -p /mongo/data/db/rep1 /mongo/data/db/rep2 /mongo/data/db/rep3 /mongo/data/db/rep4 /mongo/data/db/arb
Verify if directories are created.
[mongo@cloud2 ~]$ ls  /mongo/data/db/
arb  rep1  rep2  rep3  rep4
"rep1, this is the Primary database node. This could already exist in a Production deployment case. "rep2,rep3,rep4"  these are secondary database nodes in replica set. "arb" is the arbitrary node.
Note : For production deployments, /mongo/data/db must be a dedicated file system on each node with necessary storage configuration such as RAID, Disk type, and so on.

Step(2) : Create replica sets. 
 "--replSet" is the mongod comman line option that commands the mongod process that this instance is part of replica set configuration and specifies name. "rs" is the name of the replica set configuration in this demo.
Primary node
[mongo@cloud2 ~]$ mongod --fork  --replSet rs --bind_ip 192.168.1.80 --port 27017 --dbpath /mongo/data/db/rep1 --logpath /mongo/data/db/rep1/mongo.log
about to fork child process, waiting until server is ready for connections.
forked process: 5940
child process started successfully, parent exiting
Secondary nodes
[mongo@cloud2 ~]$ mongod --fork  --replSet rs --bind_ip 192.168.1.80 --port 27018 --dbpath /mongo/data/db/rep2 --logpath /mongo/data/db/rep2/mongo.log
about to fork child process, waiting until server is ready for connections.
forked process: 7054
child process started successfully, parent exiting
[mongo@cloud2 ~]$ mongod --fork  --replSet rs --bind_ip 192.168.1.80 --port 27019 --dbpath /mongo/data/db/rep3 --logpath /mongo/data/db/rep3/mongo.log
about to fork child process, waiting until server is ready for connections.
forked process: 7094
child process started successfully, parent exiting
[mongo@cloud2 ~]$ mongod --fork  --replSet rs --bind_ip 192.168.1.80 --port 27020 --dbpath /mongo/data/db/rep4 --logpath /mongo/data/db/rep4/mongo.log
about to fork child process, waiting until server is ready for connections.
forked process: 7146
child process started successfully, parent exiting
Arbitrary node
[mongo@cloud2 ~]$ mongod --fork  --port 30000 --bind_ip 192.168.1.80 --dbpath /mongo/data/db/arb --logpath /mongo/data/db/arb/mongo.log --replSet rs
about to fork child process, waiting until server is ready for connections.
forked process: 7184
child process started successfully, parent exiting

Step(3) : Initiate replication.
Login to the Primary instance.
[mongo@cloud2 ~]$ mongo --host 192.168.1.80 --port 27017
MongoDB shell version v3.6.3
connecting to: mongodb://192.168.1.80:27017/
MongoDB server version: 3.6.3
Server has startup warnings:
2018-03-18T10:48:08.364-0500 I CONTROL  [initandlisten]
2018-03-18T10:48:08.364-0500 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-03-18T10:48:08.364-0500 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-03-18T10:48:08.364-0500 I CONTROL  [initandlisten]
MongoDB Enterprise >
Initiate replication.
MongoDB Enterprise > rs.initiate({ _id: 'rs', version: 1, members: [{ _id: 1, host: '192.168.1.80:27017', priority: 10 }] })
{
"ok" : 1,
"operationTime" : Timestamp(1521390022, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1521390022, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
This is what happens as soon as replica set is initiated. As you see, It creates the replication oplog with 1013MB in size and creates various collections for its maintenance.



And the initialization syntax returns following shell prompt.
MongoDB Enterprise rs:OTHER>
Hit enter and then prompt changes to.
MongoDB Enterprise rs:PRIMARY>
It goes through so many transitions during this time, you can see that in the snippet below, from STARTUP to SECONDARY to PRIMARY.



And, this is configuration by default except the priority parameter.
Let us add other nodes into replica set configuration.
rep2
MongoDB Enterprise rs:PRIMARY> rs.add("192.168.1.80:27018")
{
"ok" : 1,
"operationTime" : Timestamp(1521392092, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1521392092, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}

}
rep3
MongoDB Enterprise rs:PRIMARY> rs.add("192.168.1.80:27019")
{
"ok" : 1,
"operationTime" : Timestamp(1521392112, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1521392112, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}

}
rep4
MongoDB Enterprise rs:PRIMARY> rs.add("192.168.1.80:27020")
{
"ok" : 1,
"operationTime" : Timestamp(1521392132, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1521392132, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}

}
arbiter 
MongoDB Enterprise rs:PRIMARY> rs.addArb("192.168.1.80:30000")
{
"ok" : 1,
"operationTime" : Timestamp(1521392150, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1521392150, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}

}
At this point, replica set configuration is ready. Let's login to one of the secondaries and check the prompt. The prompt confirms the status as expected, SECONDARY.



Check the configuration.
MongoDB Enterprise rs:SECONDARY> rs.conf()
{
"_id" : "rs",
"version" : 5,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 1,
"host" : "192.168.1.80:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 10,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 2,
"host" : "192.168.1.80:27018",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 3,
"host" : "192.168.1.80:27019",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 4,
"host" : "192.168.1.80:27020",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 5,
"host" : "192.168.1.80:30000",
"arbiterOnly" : true,
"buildIndexes" : true,
"hidden" : false,
"priority" : 0,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"catchUpTimeoutMillis" : -1,
"catchUpTakeoverDelayMillis" : 30000,
"getLastErrorModes" : {

},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
},
"replicaSetId" : ObjectId("5aae99c8ae99750324417362")
}
}

Testing replication
Step(3) : Load some test data into Primary and test the replication.
At this time, we have replica sets configured and also initialized. Let's quickly load some data into Primary replica set instance, rep1. For this purpose, I am using restaurants data available on github, download and save it as restaurants.json. I have it downloaded in my downloads directory on my laptop.
Nagas-MBP:Downloads nagaappani$ ls restaurants.json
restaurants.json
Import the collection, below syntax creates a database reptest and creates a collection called restaurants.
Nagas-MBP:Downloads nagaappani$ mongoimport --db reptest --collection restaurants --file restaurants.json --host 192.168.1.80 --port 27017

Let's validate if this collection is imported into Primary and is also replicated to all secondaries.
On Primary.
MongoDB Enterprise rs:PRIMARY> db.restaurants.count()
887565
On Secondaries.
Secondaries can not be queried unless we set a read preference option in the mongo session as shown below with "rs.slaveOk()".
On rep2
[mongo@cloud2 ~]$ mongo --host 192.168.1.80 --port 27018
MongoDB shell version v3.6.3
connecting to: mongodb://192.168.1.80:27018/
MongoDB server version: 3.6.3
Server has startup warnings:
2018-03-18T11:54:14.697-0500 I CONTROL  [initandlisten]
2018-03-18T11:54:14.697-0500 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-03-18T11:54:14.697-0500 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-03-18T11:54:14.697-0500 I CONTROL  [initandlisten]
MongoDB Enterprise rs:SECONDARY> rs.slaveOk()
MongoDB Enterprise rs:SECONDARY> use reptest
switched to db reptest
MongoDB Enterprise rs:SECONDARY> db.restaurants.count()
887565
On rep3
[mongo@cloud2 ~]$ mongo --host 192.168.1.80 --port 27019
MongoDB shell version v3.6.3
connecting to: mongodb://192.168.1.80:27019/
MongoDB server version: 3.6.3
Server has startup warnings:
2018-03-18T11:54:15.282-0500 I CONTROL  [initandlisten]
2018-03-18T11:54:15.282-0500 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-03-18T11:54:15.282-0500 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-03-18T11:54:15.282-0500 I CONTROL  [initandlisten]
MongoDB Enterprise rs:SECONDARY> rs.slaveOk()
MongoDB Enterprise rs:SECONDARY> use reptest
switched to db reptest
MongoDB Enterprise rs:SECONDARY> db.restaurants.count()
887565
On rep4
[mongo@cloud2 ~]$ mongo --host 192.168.1.80 --port 27020
MongoDB shell version v3.6.3
connecting to: mongodb://192.168.1.80:27020/
MongoDB server version: 3.6.3
Server has startup warnings:
2018-03-18T11:54:15.859-0500 I CONTROL  [initandlisten]
2018-03-18T11:54:15.859-0500 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-03-18T11:54:15.859-0500 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-03-18T11:54:15.859-0500 I CONTROL  [initandlisten]
MongoDB Enterprise rs:SECONDARY> rs.slaveOk()
MongoDB Enterprise rs:SECONDARY> use reptest
switched to db reptest
MongoDB Enterprise rs:SECONDARY> db.restaurants.count()
887565
As we could see, the count of number of documents in restaurants collection matches across the replica set configuration. This confirms the successful replication.

Important commands
db.isMaster() confirms the role of the mongod instance. It also helps finding IP and Ports information of the replica set and so much more.
To find all primary and secondary hosts.
MongoDB Enterprise rs:SECONDARY> db.isMaster().hosts
[
"192.168.1.80:27017",
"192.168.1.80:27018",
"192.168.1.80:27019",
"192.168.1.80:27020"

]
To find arbitrary hosts.
MongoDB Enterprise rs:SECONDARY> db.isMaster().arbiters
[ "192.168.1.80:30000" ]
To find primary.
MongoDB Enterprise rs:SECONDARY> db.isMaster().primary
192.168.1.80:27017

rs.conf() lists the core configuration of the replica set and configuration settings, and so much more.
To find settings of the replica set configuration.
MongoDB Enterprise rs:SECONDARY> rs.config().settings
{
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"catchUpTimeoutMillis" : -1,
"catchUpTakeoverDelayMillis" : 30000,
"getLastErrorModes" : {

},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
},
"replicaSetId" : ObjectId("5aae99c8ae99750324417362")
rs.status() lists the status of the replication health such as state, uptime, heartbeat, election time and date, and much more.
To find the status of the replication.
MongoDB Enterprise rs:SECONDARY>  rs.status().ok
1




replica sets, Replication with mongoDB

Before we get into details of mongoDB replica sets, let's understand what is replication and why is it needed? Replication is a config...