Sunday, March 18, 2018

replica sets, Replication with mongoDB

Before we get into details of mongoDB replica sets, let's understand what is replication and why is it needed? Replication is a configuration that creates multiple copies of data by "replicating"(transferring) data from one site to one or more sites, provides redundancy and maximum availability for databases and its applications. Database is synchronized to remote sites in real time as soon as changes are loaded in to source database. These remote sites are called Disaster Recovery sites and they are strictly recommended to be spread across different geographical regions, different cities, countries, and even different continents. Nevertheless, some IT configurations do also have deployments called "Near DRs", a DR site that could be located in the same city.

Today's Disaster Recovery deployments
Is DR only used in a real disaster scenario? Well, Let me share my thoughts of Disaster Recovery in today's context. A DR site in today's super fast growing world is a necessity for all models of businesses and data, there's no escape of this expense for Enterprises. What about ROI, does one have to wait for a real disaster to strike to consume it, is that only built for sleeping but otherwise only to wait for a disaster and show its real deal? Well, long gone are those days. Most of the database platforms today support read replicas, while "The read feature" comes at an additional expense for some (like Oracle's active data guard), and others are free by default (mongoDB read replicas, Amazon RDS Read Replicas, and so on). Therefore, a DR site is daily driver of the business, power house for reporting, contributes to the growth of the business significantly. Alright, let's get into the topic.

Replication with mongoDB
The deployment of replication configuration in mongoDB is called, a replica set. Each replica set has below nodes.
Primary node: This is the source of everything, source of input, is always in READ WRITE mode.
Secondary node: This is the copy of the Primary node, a replica copy, is always in RECOVERY mode and also serves READ operations when requested. It is always recommended to have similar resource profile, hardware capacity, as Primary node. Multiple secondary copies could be maintained based on different requirements.
Arbiter node: This is only to help the failover in the event of unforeseen primary node outage, this is optional. Arbiters come at cheap price, they do not require a dedicated hardware and they do not hold copy of primary but helps in election of new optimal Primary by responding to heartbeat and election requests by other replica set members in replication.

                                      (Image by mongoDB Inc.)
Each replica set can have up to 50 members but only maximum of 7 members can participate in election.

How it works?
"mongod", a core background daemon process, handles synchronization of changes between all replica set instances. "mongod" on Primary records all types of database changes in a fixed size collection called oplog (operations log), "oplog.rs". For people with background of Oracle, this could be compared to redo log file, and transaction log file in SQL Server. The "mongod" daemon of every secondary consumes entries from Primary's oplog and applies them to respective secondary. In addition to this, all replica set members also send heartbeat signal to every other replica set instance and get oplog entries. This replication process is asynchronous.

Setup
The replica set configuration is pretty straight forward. The replica setup initialization consists of below two taks, they are triggered automatically by mongod process, no manual intervention is required.
-Initial synchronization
-Ongoing replication

The demo example of the setup consists of 1 primary, 3 secondaries, and 1 arbiter node. For mongod to replicate oplog, each mongod instance should run on different port but IP could remain same. Using this advantage, for demonstration and testing purposes, we will configure replica set on same host. A Production deployment must be set up across different geographical regions for maximum availability and fault tolerance.

I assume you already have a working mongo installation setup of version 3.6. If it is not too late, Please go ahead and download here, and follow these instructions for installation. Be it installation of software, or, setup of replica sets, it does not make much difference in regards to steps that are followed. Here's my demo environment, FYR.
OS and version : Oracle Linux Server release 7.4
mongoDB version : mongoDB 3.6.3
Software install location : /mongo/mongohome/bin
Database location : /mongo/data/db

Let's get into setup.
###########################################################
#Color coding:                                                                                          #
#Italic Blue => syntaxes and commands                                                   #
#Italic bold green => shell prompt                                                            #
#Italic black => syntax output                                                                    #
###########################################################

Step(1) : Create directories.
[mongo@cloud2 ~]$ mkdir -p /mongo/data/db/rep1 /mongo/data/db/rep2 /mongo/data/db/rep3 /mongo/data/db/rep4 /mongo/data/db/arb
Verify if directories are created.
[mongo@cloud2 ~]$ ls  /mongo/data/db/
arb  rep1  rep2  rep3  rep4
"rep1, this is the Primary database node. This could already exist in a Production deployment case. "rep2,rep3,rep4"  these are secondary database nodes in replica set. "arb" is the arbitrary node.
Note : For production deployments, /mongo/data/db must be a dedicated file system on each node with necessary storage configuration such as RAID, Disk type, and so on.

Step(2) : Create replica sets. 
 "--replSet" is the mongod comman line option that commands the mongod process that this instance is part of replica set configuration and specifies name. "rs" is the name of the replica set configuration in this demo.
Primary node
[mongo@cloud2 ~]$ mongod --fork  --replSet rs --bind_ip 192.168.1.80 --port 27017 --dbpath /mongo/data/db/rep1 --logpath /mongo/data/db/rep1/mongo.log
about to fork child process, waiting until server is ready for connections.
forked process: 5940
child process started successfully, parent exiting
Secondary nodes
[mongo@cloud2 ~]$ mongod --fork  --replSet rs --bind_ip 192.168.1.80 --port 27018 --dbpath /mongo/data/db/rep2 --logpath /mongo/data/db/rep2/mongo.log
about to fork child process, waiting until server is ready for connections.
forked process: 7054
child process started successfully, parent exiting
[mongo@cloud2 ~]$ mongod --fork  --replSet rs --bind_ip 192.168.1.80 --port 27019 --dbpath /mongo/data/db/rep3 --logpath /mongo/data/db/rep3/mongo.log
about to fork child process, waiting until server is ready for connections.
forked process: 7094
child process started successfully, parent exiting
[mongo@cloud2 ~]$ mongod --fork  --replSet rs --bind_ip 192.168.1.80 --port 27020 --dbpath /mongo/data/db/rep4 --logpath /mongo/data/db/rep4/mongo.log
about to fork child process, waiting until server is ready for connections.
forked process: 7146
child process started successfully, parent exiting
Arbitrary node
[mongo@cloud2 ~]$ mongod --fork  --port 30000 --bind_ip 192.168.1.80 --dbpath /mongo/data/db/arb --logpath /mongo/data/db/arb/mongo.log --replSet rs
about to fork child process, waiting until server is ready for connections.
forked process: 7184
child process started successfully, parent exiting

Step(3) : Initiate replication.
Login to the Primary instance.
[mongo@cloud2 ~]$ mongo --host 192.168.1.80 --port 27017
MongoDB shell version v3.6.3
connecting to: mongodb://192.168.1.80:27017/
MongoDB server version: 3.6.3
Server has startup warnings:
2018-03-18T10:48:08.364-0500 I CONTROL  [initandlisten]
2018-03-18T10:48:08.364-0500 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-03-18T10:48:08.364-0500 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-03-18T10:48:08.364-0500 I CONTROL  [initandlisten]
MongoDB Enterprise >
Initiate replication.
MongoDB Enterprise > rs.initiate({ _id: 'rs', version: 1, members: [{ _id: 1, host: '192.168.1.80:27017', priority: 10 }] })
{
"ok" : 1,
"operationTime" : Timestamp(1521390022, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1521390022, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
This is what happens as soon as replica set is initiated. As you see, It creates the replication oplog with 1013MB in size and creates various collections for its maintenance.



And the initialization syntax returns following shell prompt.
MongoDB Enterprise rs:OTHER>
Hit enter and then prompt changes to.
MongoDB Enterprise rs:PRIMARY>
It goes through so many transitions during this time, you can see that in the snippet below, from STARTUP to SECONDARY to PRIMARY.



And, this is configuration by default except the priority parameter.
Let us add other nodes into replica set configuration.
rep2
MongoDB Enterprise rs:PRIMARY> rs.add("192.168.1.80:27018")
{
"ok" : 1,
"operationTime" : Timestamp(1521392092, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1521392092, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}

}
rep3
MongoDB Enterprise rs:PRIMARY> rs.add("192.168.1.80:27019")
{
"ok" : 1,
"operationTime" : Timestamp(1521392112, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1521392112, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}

}
rep4
MongoDB Enterprise rs:PRIMARY> rs.add("192.168.1.80:27020")
{
"ok" : 1,
"operationTime" : Timestamp(1521392132, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1521392132, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}

}
arbiter 
MongoDB Enterprise rs:PRIMARY> rs.addArb("192.168.1.80:30000")
{
"ok" : 1,
"operationTime" : Timestamp(1521392150, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1521392150, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}

}
At this point, replica set configuration is ready. Let's login to one of the secondaries and check the prompt. The prompt confirms the status as expected, SECONDARY.



Check the configuration.
MongoDB Enterprise rs:SECONDARY> rs.conf()
{
"_id" : "rs",
"version" : 5,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 1,
"host" : "192.168.1.80:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 10,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 2,
"host" : "192.168.1.80:27018",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 3,
"host" : "192.168.1.80:27019",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 4,
"host" : "192.168.1.80:27020",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 5,
"host" : "192.168.1.80:30000",
"arbiterOnly" : true,
"buildIndexes" : true,
"hidden" : false,
"priority" : 0,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"catchUpTimeoutMillis" : -1,
"catchUpTakeoverDelayMillis" : 30000,
"getLastErrorModes" : {

},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
},
"replicaSetId" : ObjectId("5aae99c8ae99750324417362")
}
}

Testing replication
Step(3) : Load some test data into Primary and test the replication.
At this time, we have replica sets configured and also initialized. Let's quickly load some data into Primary replica set instance, rep1. For this purpose, I am using restaurants data available on github, download and save it as restaurants.json. I have it downloaded in my downloads directory on my laptop.
Nagas-MBP:Downloads nagaappani$ ls restaurants.json
restaurants.json
Import the collection, below syntax creates a database reptest and creates a collection called restaurants.
Nagas-MBP:Downloads nagaappani$ mongoimport --db reptest --collection restaurants --file restaurants.json --host 192.168.1.80 --port 27017

Let's validate if this collection is imported into Primary and is also replicated to all secondaries.
On Primary.
MongoDB Enterprise rs:PRIMARY> db.restaurants.count()
887565
On Secondaries.
Secondaries can not be queried unless we set a read preference option in the mongo session as shown below with "rs.slaveOk()".
On rep2
[mongo@cloud2 ~]$ mongo --host 192.168.1.80 --port 27018
MongoDB shell version v3.6.3
connecting to: mongodb://192.168.1.80:27018/
MongoDB server version: 3.6.3
Server has startup warnings:
2018-03-18T11:54:14.697-0500 I CONTROL  [initandlisten]
2018-03-18T11:54:14.697-0500 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-03-18T11:54:14.697-0500 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-03-18T11:54:14.697-0500 I CONTROL  [initandlisten]
MongoDB Enterprise rs:SECONDARY> rs.slaveOk()
MongoDB Enterprise rs:SECONDARY> use reptest
switched to db reptest
MongoDB Enterprise rs:SECONDARY> db.restaurants.count()
887565
On rep3
[mongo@cloud2 ~]$ mongo --host 192.168.1.80 --port 27019
MongoDB shell version v3.6.3
connecting to: mongodb://192.168.1.80:27019/
MongoDB server version: 3.6.3
Server has startup warnings:
2018-03-18T11:54:15.282-0500 I CONTROL  [initandlisten]
2018-03-18T11:54:15.282-0500 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-03-18T11:54:15.282-0500 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-03-18T11:54:15.282-0500 I CONTROL  [initandlisten]
MongoDB Enterprise rs:SECONDARY> rs.slaveOk()
MongoDB Enterprise rs:SECONDARY> use reptest
switched to db reptest
MongoDB Enterprise rs:SECONDARY> db.restaurants.count()
887565
On rep4
[mongo@cloud2 ~]$ mongo --host 192.168.1.80 --port 27020
MongoDB shell version v3.6.3
connecting to: mongodb://192.168.1.80:27020/
MongoDB server version: 3.6.3
Server has startup warnings:
2018-03-18T11:54:15.859-0500 I CONTROL  [initandlisten]
2018-03-18T11:54:15.859-0500 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-03-18T11:54:15.859-0500 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-03-18T11:54:15.859-0500 I CONTROL  [initandlisten]
MongoDB Enterprise rs:SECONDARY> rs.slaveOk()
MongoDB Enterprise rs:SECONDARY> use reptest
switched to db reptest
MongoDB Enterprise rs:SECONDARY> db.restaurants.count()
887565
As we could see, the count of number of documents in restaurants collection matches across the replica set configuration. This confirms the successful replication.

Important commands
db.isMaster() confirms the role of the mongod instance. It also helps finding IP and Ports information of the replica set and so much more.
To find all primary and secondary hosts.
MongoDB Enterprise rs:SECONDARY> db.isMaster().hosts
[
"192.168.1.80:27017",
"192.168.1.80:27018",
"192.168.1.80:27019",
"192.168.1.80:27020"

]
To find arbitrary hosts.
MongoDB Enterprise rs:SECONDARY> db.isMaster().arbiters
[ "192.168.1.80:30000" ]
To find primary.
MongoDB Enterprise rs:SECONDARY> db.isMaster().primary
192.168.1.80:27017

rs.conf() lists the core configuration of the replica set and configuration settings, and so much more.
To find settings of the replica set configuration.
MongoDB Enterprise rs:SECONDARY> rs.config().settings
{
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"catchUpTimeoutMillis" : -1,
"catchUpTakeoverDelayMillis" : 30000,
"getLastErrorModes" : {

},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
},
"replicaSetId" : ObjectId("5aae99c8ae99750324417362")
rs.status() lists the status of the replication health such as state, uptime, heartbeat, election time and date, and much more.
To find the status of the replication.
MongoDB Enterprise rs:SECONDARY>  rs.status().ok
1




Monday, August 17, 2015

IPC Send timeout detected - The Story of a Database

Some errors make us panic, nervous, and beyond belief when we positively attempt to resolve a panic situation. Specific to case of Clusterware where it attempts to resolve a conflict by removing dead resources from the cluster but it suffers because of lack of additional resources to complete the task, and makes the whole situation even worst so everything that has in contact with the key resource also suffers. The key resource what we are talking is a RAC Database and the Clusterware under a panic situation is Oracle Clusterware.

Here's the environment
  1. Oracle RDBMS 11.2.0.4.4
  2. Oracle GI 11.2.0.4.4
  3. 3 Node RAC Database with vault function enabled
  4. NetApp NFS Filer based Storage


What was the issue?


On Instance 3
I have noticed "IPC Send timeout".
Fri Aug 14 04:46:05 2015
IPC Send timeout detected. Receiver ospid 20545 [
Fri Aug 14 04:49:55 2015
And then, Oracle has started suspending MMON actions one by one as we noticed below messages in the log
Fri Aug 14 10:49:07 2015
Suspending MMON action 'AWR Auto CPU USAGE Task' for 82800 seconds
Fri Aug 14 07:22:11 2015
Suspending MMON action 'tablespace alert monitor' for 82800 seconds

On Instance 1
Suspending MMON slave action kttetslv_ for 82800 seconds
Fri Aug 14 07:57:53 2015
IPC Send timeout detected. Sender: ospid 4812 [oracle@phyeimrac01-tst (J000)]
Receiver: inst 3 binc 4 ospid 20545
Fri Aug 14 08:03:11 2015

This usually occurs when there is high resource contention (Locks/CPU/Memory/Storage/Network), where processes queue increases while the demand for resource increases, and global enqueue and cache services stop responding to each other in a RAC environment. In the end, it gets to a situation where it can't make any actions except for being hung, and keep trying to resolve itself. Hence, we see "Inter Process Communication" Send timeout detected.

I came to know it was because of massive load on CPUs, found from AWR report. Here's the screenshot from AWR report.



How was it resolved?

Well, during the issue, I was completely helpless to see what's going on. Nothing worked period. srvctl/crsctl, well, nothing worked. Fortunately, a connection to SQLPLUS prompt worked on unaffected instances in the cluster i.e., instance 1 and instance 2 but again, GV$ views did not work. And, strange enough, Instance eviction was not happening at all while everything was just hanging. The real culprits, Application people keep complaining "Man, performance is terrible, please do something, it's not moving at all" :). How many times we hear this, well, quite often during benchmark phase of a development project. They keep changing the code left right center and think what they do will work. Eventually, it creates a nightmare for DBA.

As we have Golden Gate configured on node 1 (its VIP runs on node1), I first brought down all replicats and then brought down instance 1 and instance 2 using "SQLPLUS prompt by "SHUT ABORT" becuase srvctl does not respond at all. It makes sense because "srvctl" in clusterware environment is driven by clusterware for mutual benefit of clusterware resources. At this stage, Two down, one to go. Well, to bring down the instance 3, the only solution to reboot the node 3. So, communicated to the server team to reboot the node 3.

Here comes the real devil - "file 1 needs to be either taken out of backup mode or media recovered"

Node 3 was rebooted, you think we are all set now? Eh? Well, here came the devil, Database was not coming up, all instances were failing to open the Database with the error below.

Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/teim/teim1/trace/teim1_ora_11744.trc:
ORA-10873: file 1 needs to be either taken out of backup mode or media recovered
ORA-01110: data file 1: '/oracle/oradata/teim/data1/system01.dbf'
ORA-10873 signalled during: ALTER DATABASE OPEN /* db agent *//* {2:30594:28372} */... 

Because I still did not know what else has happened at this time, For a moment, I thought, what the heck I did, Technically, I have just aborted all instances and expected SMON to perform Instance Crash Recovery by replaying what's in redo (roll forward committed changes/rollback uncommitted changes, and make everything consistent), and it should be all set for operations again. But, why the heck did it ask me to perform a media recovery. Not even in my dreams could I have ever imagined this.

Ok, this had to be troubleshooted now. Panic over panic, eh? Don't be panic. Come on, Oracle!!

While digging into the issue
We use NetApp snapshots for Database backup, it is basically a traditional Oracle hot backup. Yes, the one that makes "redolog" life miserable :).  The script we have in place performs like this

  1. First put the Database in BEGIN BACKUP mode - Done 
  2. Make a log switch to archive - Done 
  3. "grep" for "oradata" from "df -h" and get the list of "Database file system" volumes - Hanging
  4. Create snapshots of each volume with the date - Forget it
  5. Make a log switch again to archive - Forget it
  6. End backup with "ALTER DATABASE END BACKUP" - Forget it

Well, the step 3 was hanging because there were some stale NFS mounts. This was the devil. So, the script did not manage to END the backup, and left all Database files in "ACTIVE BACKUP" mode.
We were not aware of this while resolving the other issue "IPC Send timeout detected". Of course, why did we think of the backup issue? Nope, we do not.

Alright, no need to be panic. follow the below to bring the Database up again

Bring the Database to mount state
$srvctl start database -d <db_unique_name> -o mount
End the backup
SQL>ALTER DATABASE END BACKUP;
Make sure none of the files in BACKUP mode
SQL>select * from gv$backup;
Bring the Database down
$srvctl stop database -d <db_unique_name>
Bring the Database up
$srvctl start database -d <db_unique_name>

Database is up, take a deep breath :)

I hope you have enjoyed reading this post as much as I have enjoyed resolving the issue. Please feel free to comment and share.

Thanks,
Naga


Monday, December 15, 2014

Rebuilding Standby Database Manually (11g and up)

One of my remote customers has been facing severe network outages between Production and DR sites to reasons unknown and challenges with certain geographic limitations, the bandwidth between sites is limited to 2MBPS and the replication between sites is aching slow. So, Customer has had tentative requirements to rebuild Standby more often between sites until the network issue is fully resolved.

This paper outlines the procedure to rebuild the STANDBY manually provided we have following things in place already between/on sites. And this procedure has a special case for One node RAC, Oracle dynamically applies SID naming convention in case of an online relocation. Due to this fact, I have to mention which SID to export before logging to the Database via SQLPLUS/RMAN/ASMCMD
(PR refers to - Production/Primary and PR1/PR2 are two nodes in RAC,  DR refers to - Disaster Recovery/Standby and DR1/DR2 are two nodes in RAC at DR Site)

  • A working Dataguard Configuration between Sites (All Parameters on both PR and DR)
  • A working Oracle Networking Configuration between sites (including Listener  and TNS)
  • A working RMAN backup
  • A RAC/One Node RAC/Single Node Oracle setup in place
  • ASM
  • Level 1 Skillset Oracle DBA Resource

In case if you would like to know how to configure Dataguard, Please refer to my post


On Production/Primary/PR1/PR2 (oracle os user prompt)

Find which SID is running
$ps -ef|grep pmon

Get the SID name and export it
$export ORACLE_SID=PRDB_1/PRDB_2

Create a directory (create this on DR too and create this on all nodes PR2/DR1/DR2)
$mkdir -p /backup/rman/4standby

Login to RMAN and take the copy of controlfile for STANDBY
$rman target /

Do log switches
RMAN>sql 'alter system switch logfile';
RMAN> sql 'alter system switch logfile';

Take the backup of controlfile for standby and full Database
RMAN>configure device type disk parallelism 10 backup type to backupset;
RMAN>configure channel device type disk format '/backup/rman/4standby/%d_% s_%p_%t';
RMAN>copy current controlfile for standby to '/backup/rman/4standby/standby_PR.ctl';
the above command dumps the controlfile to /backup/rman/4standby, SCP it to DR site to the same location '/backup/rman/4standby/'
RMAN>backup section size 500m as compressed backupset database;
RMAN>exit;

Disable the destination if it's running
$sqlplus / as sysdba
SQL>ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=DEFER SID='*';

Now take a copy of this backup to DR and keep it in the same location '/backup/rman/4standby'


Now we are on DR (DR/Standby/DR1/DR2)(oracle os user prompt)

Start the DR instance (on DR1/DR2)
$srvctl start database -d PRDB -o nomount

Find which SID is running
$ps -ef|grep pmon

Get the SID name and export it
$export ORACLE_SID=PRDBDR_1/PRDBDR_2

Login to RMAN and restore the controlfile and backup
$rman target /
RMAN>restore controlfile from '/backup/rman/4standby/standby_PR.ctl';
RMAN>sql 'alter database mount standby database';
RMAN>catalog start with '/backup/rman/4standby' noprompt';
RMAN>restore database;
RMAN>exit;


On PR at this time(oracle os user prompt)

Enable the destination to ship archivelogs
$sqlplus / as sysdba
SQL>ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE SID='*';


On DR at this time (grid os user prompt)

Remove existing STANDBY Logfiles
Login as grid user
Find which ASM instance is running
$ps -ef|grep pmon

Get the ASM instance name and export it
export ORACLE_SID=+ASM1/+ASM2

Login to asmcmd” prompt
asmcmd>cd +PRREDO1/PRDB
asmcmd>rm sr* (Choose “y” on prompt)
asmcmd>cd +PRREDO2/PRDB
asmcmd>rm sr* (Choose “y” on prompt)
asmcmd>exit;

Back to oracle user prompt(oracle os user prompt)
Login to SQLPLUS prompt as SYSDBA
$sqlplus / as sysdba

Recreate Standby Logfiles
SQL>ALTER DATABASE ADD STANDBY LOGFILE '+PRREDO1/PRDB/srl1.f' SIZE 536870912;
SQL>ALTER DATABASE ADD STANDBY LOGFILE '+PRREDO2/PRDB/srl2.f' SIZE 536870912;
SQL>ALTER DATABASE ADD STANDBY LOGFILE '+PRREDO1/PRDB/srl3.f' SIZE 536870912;
SQL>ALTER DATABASE ADD STANDBY LOGFILE '+PRREDO2/PRDB/srl4.f' SIZE 536870912;
SQL>ALTER DATABASE ADD STANDBY LOGFILE '+PRREDO1/PRDB/srl5.f' SIZE 536870912;
SQL>ALTER DATABASE ADD STANDBY LOGFILE '+PRREDO2/PRDB/srl6.f' SIZE 536870912;
SQL>ALTER DATABASE ADD STANDBY LOGFILE '+PRREDO1/PRDB/srl7.f' SIZE 536870912;
SQL>ALTER DATABASE ADD STANDBY LOGFILE '+PRREDO2/PRDB/srl8.f' SIZE 536870912;

Open the Database in Managed Recovery mode
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;


 Monitor the shipping and logs apply status
SQL>SELECT SEQUENCE#,BLOCK#,PROCESS,STATUS FROM V$MANAGED_STANDBY;


On PR at this time(oracle os user prompt)

Keep monitoring the logs apply status until the the previous sequence to current sequence updated with status "YES" under applied column from the below query
$sqlplus / as sysdba
SQL>SELECT SEQUENCE#,APPLIED FROM V$ARCHIVED_LOG WHERE SEQUENCE#=(SELECT MAX(SEQUENCE#) FROM V$ARCHIVED_LOG);

The updated status "YES" for the current log sequence at PR confirms the synchronisation with its DR peer.


On DR at the time(oracle os user prompt)

Enable Active Dataguard mode
$sqlplus / as sysdba
SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
SQL>ALTER DATABASE OPEN READ ONLY;
SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;
SQL><Query any application table to monitor the Active transactional SYNC which should match with PR transaction output>

If this has helped you, please do not forget share it and help others.

Good Luck!

Tuesday, February 11, 2014

Oracle Database 12c - Far Sync Instance



A New Standby destination type is allowed in 12c Active Dataguard environment that ensures zero data loss failover and zero performance impact on Primary Database by a new concept called "Far Sync Instance". 


What is Far Sync Instance?
As we knew so far (until 11g), A SYNCHRONOUS redo transport Dataguard configuration impacts the availability and performance of Primary Database in order to protect data loss that happens due to network outages with its Standby Site. To avoid such issues and give benefit of the same zero Data loss to customers at additional cost, Oracle has introduced a new type of Instance called Far Sync Instance.  A Far Sync Instance usually is very close in distance to its Primary Database Network receives the redo synchronously from its Primary partner and then pushes it to its remote Standby Partner, plays a dummy physical standby role that actually fakes its Primary Partner to forget network outages as it takes the responsibility of managing redo to actual Standby targets. As a result of it, It minimises the unplanned downtime and performance issues due to Wider Area Network latencies or simply Network latencies. 


What does it contain?
This destination contains parameterfile, controlfile, and standby redo logs (SRLs), it receives the redo from Primary Database, and archives it to local destination through Standby Redo Logs. There are no Datafiles here. As a result of it, it consumes very little disk and processing resources, yet provides the ability to failover to a terminal destination with zero data loss, as well as offload the primary database of other types of overhead (for example, redo transport).

How to configure?
(Same as configuring Dataguard except backup/restore operations of the Primary Database)
1)Create pfile from spfile of the Primary, and copy to to Far Sync Server, and start the Instance
2)Add the Far Sync Destination, EX: "CDBFS"(Net Service Name/DB_UNIQUE_NAME), to Dataguard configuration (under LOG_ARCHVE_CONFIG=DG_CONFIG)
3)Set a new destination parameter to enable Log Transport Services, EX: LOG_ARCHIVE_DEST_3 & LOG_ARCHIVE_DEST_STATE_3
4)Create the Far Sync Instance Control file, copy it to FSI Server, and mount
SYS@CDB AS SYSDBA 28-OCT-13> ALTER DATABASE CREATE FAR SYNC INSTANCE CONTROLFILE AS '/fra/fsi_control01.ctl';
Database altered.

5)Enable the destination to FSI on Primary

FSI now receives the redo by establishing RFS connections to its primary

Hope it helps, your feedback is appreciated, and thanks for reading..

replica sets, Replication with mongoDB

Before we get into details of mongoDB replica sets, let's understand what is replication and why is it needed? Replication is a config...