Data Doman – manipulate interface used for replication

When we deployed our second Data Domain i wanted to start migrating some shares from the old DD880 to the new DD890.  We have multiple virtual interfaces on DD880 and DD890  and i wanted to make sure that replication traffic was using specific interfaces that had layer 2 connectivity. I have documented the steps how to manipulate replication interfaces.

Configuration:

Source – DD880 – OS 5.1.1.0-291218

DNS Name – DD880PROD  – virtual interface name used for host backups
DNS Name – DD880PRIV – virtual interface i want to use for replication

DD880PROD# net show settings 
port         enabled   DHCP   IP address       netmask           type   additional setting   
----------   -------   ----   --------------   ---------------   ----   ---------------------
eth0a        yes       no     10.12.24.51      255.255.255.0     n/a                         
eth0b        yes       no     192.168.1.3      255.255.255.0     n/a                         
eth5a        yes       n/a    n/a              n/a               n/a    bonded to veth0      
eth5b        yes       n/a    n/a              n/a               n/a    bonded to veth0      
veth0        yes       no     0                255.255.252.0     n/a    failover: eth5a,eth5b
veth0.2694   yes       no     128.140.38.136   255.255.255.128   n/a    << --- Production (DNS Name DD880PROD)           
veth0.3442   yes       no     10.140.32.25     255.255.255.128   n/a    << -- Private      (DNS Name DD880PRIV)

Target – DD890 – OS 5.1.1.0-291218

DNS Name – DD890PROD  – virtual interface name used for host backups
DNS Name – DD890PRIV – virtual interface i want to use for replication

DD890PROD# net show settings
port         enabled   DHCP   IP address       netmask         type   additional setting             
----------   -------   ----   --------------   -------------   ----   -------------------------------
eth0a        no        n/a    n/a              n/a             n/a                                   
eth0b        no        n/a    n/a              n/a             n/a                                   
eth4a        yes       n/a    n/a              n/a             n/a    bonded to veth0                
eth4b        yes       n/a    n/a              n/a             n/a    bonded to veth0                
veth0        yes       no     0                255.255.255.0   n/a    lacp hash xor-L3L4: eth4a,eth4b
veth0.2997   yes       no     128.199.145.250  255.255.255.0   n/a    << --- Production (DNS Name DD890PROD)                            
veth0.3442   yes       no     10.140.32.26     255.255.255.128 n/a    << -- Private      (DNS Name DD890PRIV)

When you configure Data Domain replication session you have to specify host name assigned to Data Domain, you can get it by typing “hostname” and the prompt (it is case sensitive).

  • Let’s go ahead and create the session first, run this command on both source and target DD
replication add source dir://DD880PROD.mycompany.local/backup/oracle destination dir://DD890PROD.mycompany.local/backup/oracle
  • Verify session got created, also notice Connection Host. This is my production interface that i do not want to use for replication.
DD880PROD# replication show config
CTX   Source                                                       Destination                      Connection Host and Port                Low-bw-optim   Enabled

---   -------------------------------------------    ---------------------------------------------  -------------------------------------   ------------   -------

1    dir://DD880PROD.mycompany.local/backup/oracle   dir://DD890PROD.mycompany.local/backup/oracle  DD880PROD.mycompany.local   (default)     disabled       yes
  • Now let’s modify this session on each Data Domain so it uses private interfaces

On source (DD880)

replication modify rctx://1 connection-host DD890PRIV.mycompany.local

On target (DD890)

replication modify rctx://1 connection-host DD880PRIV.mycompany.local
  • Let’s see what it looks like (on source), notice it’s now displaying correct Connection Host name
DD880PROD# replication show config
CTX   Source                                                       Destination                      Connection Host and Port                Low-bw-optim   Enabled

---   -------------------------------------------    ---------------------------------------------  -------------------------------------   ------------   -------

1    dir://DD880PROD.mycompany.local/backup/oracle   dir://DD890PROD.mycompany.local/backup/oracle  DD880PRIV.mycompany.local   (default)     disabled       yes
  • At this point we are ready to start replication session, if you have issues starting the session make sure you don’t need to add any static routes
replication initialize dir://DD890PROD.mycompany.local/backup/oracle
Advertisements

Configuring DDBoost for RMAN and Oracle 10g

DDBoost for RMAN is an amazing product, we ran a couple of test backups and here are our results:

Server – HP Proliant G7, 1G NIC
Operating System – RedHat 4.7
Oracle 10g database, 1TB in size
Data Domain 890, 10G interface configured in LACP,  OS 5.1.1.0-291218
DDBoost – RMAN 1.0.1.4 for Linux

Test 1:

RMAN backup to Data Domain nfs export ~= 5 hours

Test 2:

DDBoost RMAN backup to Data Domain ~=36 minutes !!!

The process of configuring Data Domain and Oracle is very straightforward, you can also reference the official document from EMC titled “EMC Data Domain Boost for Oracle Recovery Manager Administration Guide”

  • Enable DD Boost feature (assumes you already own and installed the license)

# ddboost enable

  • Configure user account that will be used for DD Boost. This will be the only account that can utilize DD Boost and will have access to all storage units

# user add <username> password <password>

# ddboost set user-name <username>

  • Configure storage unit, it’s a logical container where you are going to send your backups to. When you create storage unit, it automatically creates an mtree and that will allow you to track storage unit utilization at a later time.  When i talked to Data Domain support they recommended not to exceed 16 mtrees per Data Domain unit as having more could have negative impact on performance

# ddboost storage-unit create <storageunitname>

  • Now we need to grant our Oracle host access to storage units, as of DDOS 5.1  i don’t see a way to grant access to specific storage unit, it’s global access

# ddboost access add clients <hostname>

We are done with Data Domain configuration, now let’s configure our Oracle host

  • Download RMAN Plug-in from Data Domain website and install it on database server, it’s just a couple of binary files in a tar file for *nix platforms
  • Next step is to register storage unit name, ddboost username and password.  I don’t know exactly what happens here but i know it’s not talking to Data Domain because you can enter incorrect password or username and it will still complete successfully

RUN
{
CONFIGURE DEFAULT DEVICE TYPE TO SBT_TAPE; # default
CONFIGURE DEVICE TYPE SBT_TAPE Backup TYPE to BACKUPSET;
CONFIGURE CHANNEL DEVICE TYPE ‘SBT_TAPE’
PARMS ‘SBT_LIBRARY=/u01/app/oracle/product/10.2.0/dbhome/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=/u01/app/oracle/product/10.2.0/dbhome)’;

ALLOCATE CHANNEL c1 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=/u01/app/oracle/product/10.2.0/dbhome/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=/u01/app/oracle/product/10.2.0/dbhome)’;
send ‘set username ddboostusername password XXXX servername datadomain890.mycompany.local’;

RELEASE CHANNEL c1;
}

  • Let’s run a backup, in this example we are using 6 channels. In our environment this gave us the most speed before we started seeing high spikes in CPU utilization

RUN {
CONFIGURE DEFAULT DEVICE TYPE TO SBT_TAPE; # default
CONFIGURE DEVICE TYPE SBT_TAPE Backup TYPE to BACKUPSET;
CONFIGURE CHANNEL DEVICE TYPE ‘SBT_TAPE’
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

ALLOCATE CHANNEL c1 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

ALLOCATE CHANNEL c2 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

ALLOCATE CHANNEL c3 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

ALLOCATE CHANNEL c4 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

ALLOCATE CHANNEL c5 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

ALLOCATE CHANNEL c6 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

backup filesperset 1 blocks all database format ‘%d_DATABASE_%T_%t_s%s_p%p’ tag ‘${ORACLE_DB} database backup’;
backup current controlfile format ‘%d_controlfile_%T_%t_s%s_p%p’ tag ‘${ORACLE_DB} Controlfile backup’;
backup spfile format ‘%d_spfile_%T_%t_s%s_p%p’ tag ‘${ORACLE_DB} Spfile backup’;

release channel c1;
release channel c2;
release channel c3;
release channel c4;
release channel c5;
release channel c6;
}

  • As this backup is running and you want to monitor its performance from Data Domain side, you can use this command:

# ddboost show stats interval 5 count 100

  • If you want to look at storage-unit statistics (# of files, compress/dedupe rate)

# ddboost storage-unit show compression

A couple of of issues that you may encounter with your implementation

  • We got this error message below when we  specified incorrect password during ddboost rman client registration (first RMAN script above)

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on c6 channel at 02/11/2013 17:59:10
ORA-19506: failed to create sequential file, name=”DB3_DATABASE_20130211_p1″, parms=””
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
sbtbackup: dd_rman_connect_to_backup_host failed

  • We got this error message when we exported storage unit via NFS and change ownership of some files.

RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on c3 channel at 02/11/2013 14:55:31
ORA-19506: failed to create sequential file, name=”DB3_DATABASE_20130211_p1″, parms=””
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
sbtbackup: Could not create file DB3_DATABASE_20130211_p1 on host datadomain890.mycompany.local, error 5034

Welcome!

Welcome, content coming soon. Don’t forget to follow this blog and receive notifications of my latest posts.

 

~Dynamox