Configuring DDBoost for RMAN and Oracle 10g

DDBoost for RMAN is an amazing product, we ran a couple of test backups and here are our results:

Server – HP Proliant G7, 1G NIC
Operating System – RedHat 4.7
Oracle 10g database, 1TB in size
Data Domain 890, 10G interface configured in LACP,  OS 5.1.1.0-291218
DDBoost – RMAN 1.0.1.4 for Linux

Test 1:

RMAN backup to Data Domain nfs export ~= 5 hours

Test 2:

DDBoost RMAN backup to Data Domain ~=36 minutes !!!

The process of configuring Data Domain and Oracle is very straightforward, you can also reference the official document from EMC titled “EMC Data Domain Boost for Oracle Recovery Manager Administration Guide”

  • Enable DD Boost feature (assumes you already own and installed the license)

# ddboost enable

  • Configure user account that will be used for DD Boost. This will be the only account that can utilize DD Boost and will have access to all storage units

# user add <username> password <password>

# ddboost set user-name <username>

  • Configure storage unit, it’s a logical container where you are going to send your backups to. When you create storage unit, it automatically creates an mtree and that will allow you to track storage unit utilization at a later time.  When i talked to Data Domain support they recommended not to exceed 16 mtrees per Data Domain unit as having more could have negative impact on performance

# ddboost storage-unit create <storageunitname>

  • Now we need to grant our Oracle host access to storage units, as of DDOS 5.1  i don’t see a way to grant access to specific storage unit, it’s global access

# ddboost access add clients <hostname>

We are done with Data Domain configuration, now let’s configure our Oracle host

  • Download RMAN Plug-in from Data Domain website and install it on database server, it’s just a couple of binary files in a tar file for *nix platforms
  • Next step is to register storage unit name, ddboost username and password.  I don’t know exactly what happens here but i know it’s not talking to Data Domain because you can enter incorrect password or username and it will still complete successfully

RUN
{
CONFIGURE DEFAULT DEVICE TYPE TO SBT_TAPE; # default
CONFIGURE DEVICE TYPE SBT_TAPE Backup TYPE to BACKUPSET;
CONFIGURE CHANNEL DEVICE TYPE ‘SBT_TAPE’
PARMS ‘SBT_LIBRARY=/u01/app/oracle/product/10.2.0/dbhome/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=/u01/app/oracle/product/10.2.0/dbhome)’;

ALLOCATE CHANNEL c1 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=/u01/app/oracle/product/10.2.0/dbhome/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=/u01/app/oracle/product/10.2.0/dbhome)’;
send ‘set username ddboostusername password XXXX servername datadomain890.mycompany.local’;

RELEASE CHANNEL c1;
}

  • Let’s run a backup, in this example we are using 6 channels. In our environment this gave us the most speed before we started seeing high spikes in CPU utilization

RUN {
CONFIGURE DEFAULT DEVICE TYPE TO SBT_TAPE; # default
CONFIGURE DEVICE TYPE SBT_TAPE Backup TYPE to BACKUPSET;
CONFIGURE CHANNEL DEVICE TYPE ‘SBT_TAPE’
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

ALLOCATE CHANNEL c1 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

ALLOCATE CHANNEL c2 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

ALLOCATE CHANNEL c3 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

ALLOCATE CHANNEL c4 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

ALLOCATE CHANNEL c5 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

ALLOCATE CHANNEL c6 TYPE SBT_TAPE
PARMS ‘SBT_LIBRARY=${ORACLE_HOME}/lib/libddobk.so, ENV=(STORAGE_UNIT=storageunitname,BACKUP_HOST=datadomain890.mycompany.local,ORACLE_HOME=${ORACLE_HOME})’;

backup filesperset 1 blocks all database format ‘%d_DATABASE_%T_%t_s%s_p%p’ tag ‘${ORACLE_DB} database backup’;
backup current controlfile format ‘%d_controlfile_%T_%t_s%s_p%p’ tag ‘${ORACLE_DB} Controlfile backup’;
backup spfile format ‘%d_spfile_%T_%t_s%s_p%p’ tag ‘${ORACLE_DB} Spfile backup’;

release channel c1;
release channel c2;
release channel c3;
release channel c4;
release channel c5;
release channel c6;
}

  • As this backup is running and you want to monitor its performance from Data Domain side, you can use this command:

# ddboost show stats interval 5 count 100

  • If you want to look at storage-unit statistics (# of files, compress/dedupe rate)

# ddboost storage-unit show compression

A couple of of issues that you may encounter with your implementation

  • We got this error message below when we  specified incorrect password during ddboost rman client registration (first RMAN script above)

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on c6 channel at 02/11/2013 17:59:10
ORA-19506: failed to create sequential file, name=”DB3_DATABASE_20130211_p1″, parms=””
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
sbtbackup: dd_rman_connect_to_backup_host failed

  • We got this error message when we exported storage unit via NFS and change ownership of some files.

RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on c3 channel at 02/11/2013 14:55:31
ORA-19506: failed to create sequential file, name=”DB3_DATABASE_20130211_p1″, parms=””
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
sbtbackup: Could not create file DB3_DATABASE_20130211_p1 on host datadomain890.mycompany.local, error 5034

Deploying virtual Isilon on ESXi

I’ve documented the steps on how to deploy virtual Isilon appliance on ESXi platform. I believe at the moment only existing EMC customers can get their hands on this appliance.  Isilon virtual appliance is to be used for testing purposes only. This appliance comes pre-built for VMware Workstation/Player, which is nice, but I wanted to deploy it on my ESXi server (free edition).

Current ESX server configuration:

Two vSwitches, vSwitch0 is my public network and vSwitch1 is my private network (I used 192.168.1.0/24 subnet, this will be used by Isilon cluster for intra-cluster connectivity, on real hardware Isilon uses InfiniBand switches).

DNS records:

You don’t have to create A records for internal interfaces, i am listing them here for documentation purposes only.

isilonintpoc1.local – 192.168.1.50
isilonintpoc2.local – 192.168.1.51
isilonintpoc3.local – 192.168.1.52

A record for each node of the cluster (external interfaces)

isilonpoc1.local – 10.144.4.11
isilonpoc2.local – 10.144.4.12
isilonpoc3.local – 10.144.4.13

A record for SmartConnect Service IP

isilonpoc0.local – 10.144.4.10

NS record for SmartConnect zone name, this record should point to the A record of SmartConnect Service IP

isilonpoc.local –> isilonpoc0.local

Let’s get started …

  • Extract content of zip file to your hard drive
  • Download and install VMware vCenter Converter Standalone
  • Open Converter, select “Convert Machine”. For Source type select “VMware Workstation or other VMware virtual machine”.  For “Virtual machine file browse to the extracted folder and select vmx file that is in the root of the directory.

vmxselect

  • Enter ESX server information and hit Next. Enter node name that you want to assign to this VM, select datastore, I left virtual machine version at 7.
  • The only thing that i modify on the next page is Networking, i change NIC1 to vSwitch1 (private) and NIC2 to vSwitch0 (public). It will take 5-10 minutes to convert the appliance. In my virtual cluster i will have three Isilon nodes so i repeat the same steps and convert two more node.
  • Let’s setup our first node, connect to ESX, open console of the first node and turn it on. You will be prompted to format ifs partition for all drives, select yes
  • Next we get to the wizard that will walk us through configuring the cluster (Note: if you make a typo and need to get back, simply tack “back” at the prompt)

2-11-2013 11-15-22 AM

  • Since this is the first node of our cluster we are going to select 1 to Create a new cluster, create password for root and admin account, select No for Enabling SupportIQ, this is a virtual appliance we don’t need to enable email-home support, enter cluster name, select enter for default encoding (utf-8)
  • Next we are going to configure intra-network settings, this is the network that Isilon nodes use to communicate to each other. I am using my private network (vSwitch1, 192.168.1.0/24) network.

2-11-2013 11-15-39 AM

  • Select 1 to configure netmask, select 3 to configure intra-cluster ip range. On my private network i will use range 192.168.1.50-53 where 192.168.1.50 is my low ip and 192.168.1.53 is my high ip.
  • Now that we are done with our internal network we are going to configure our external network, select 1 to configure external interface

2-11-2013 11-15-49 AM

  • Enter subnet mask information and then configure ext-1 ip range information. Next configure default gateway.
  • Now we are ready to configure SmartConnect settings of the cluster. SmartConnect is built-in load-balancer, you can read more about it on support.emc.com document titled “SmartConnect – Optimized Scale-out storage performance and availability”. You can also get a lot of tips on how to configure in production by visiting this blog http://www.jasemccarty.com/blog/

2-11-2013 11-16-02 AM

  • Select 1 to configure zone name, it should match NS record you created in DNS (delegation), in this example i will enter isilonpoc.local. Next configure SmartConnect service IP.
  • Configure DNS Servers and Search Domains (separate multiple entries with a comma)
  • Configure TimeZone and Date/Time
  • Cluster Join mode keep at Manual
  • Commit settings and at this point you should be able to connect to the SmartConnect zone name  (isilonpoc.local) and login to the cluster

Now that we have our cluster going let’s join another node to the cluster,  connect to console of another node and turn it on.

  • You will be prompted to format ifs partition for all drives, select yes
  • Now at the wizard select Join an existing cluster

2-11-2013 11-16-12 AM

  • Isilon node will automatically search local subnet and look for an existing cluster(s), if it finds it will present this option. Select your cluster index # and hit Enter

2-11-2013 11-16-21 AM

  • At this point you should see the new node show up in Isilon WebUI
  • When you login to WebUI you will see a couple of messages about “one or more drives area ready to be replaced”. This is normal since this a virtual appliance and does not have all the drives physical nodes have. Ssh into the cluster and clear  all alerts/events but running these two commands: “isi events cancel all” and “isi events quiet all”

Thank you @ChristopherImes for some pointers