Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Backup Performance and Fibre Channel Zoning

Posted on 2013-01-30
15
Medium Priority
?
1,629 Views
Last Modified: 2013-02-04
Hi,

We recently acquired an EMC VNX5300 and a FC Tape Library.
The original zoning configuration was not following EMC’s best practice for single HBA zoning so I went ahead and created new zones.
Before the change, one of my backup jobs which used to take 11+ hours went down to 5+ but after creating the new zones my backup jobs is taking again 11+ hours to complete.

To accomplish the backup, I am using a windows 2003 server with Arcserve R16 on it.  This server, BackupServer1 has a dual 8Gb HBA and connects to both FC switches.
The backup job backs up a LUN that it is attached to a Solaris server.  The Solaris server uses a single HBA 4Gb connected to SANSwitch1.

The VNX_SPA, VNX_SPB are the storage processors for the VNX 5300.

The tape library is an (Overland) IBM 3573 FC with dual LTO6 drives.  I have one drive attached to SANSwitch1 and the other attached to SANSwitch2.

The switches are EMC (Brocade) DS-300B 8Gb FC.

I am obviously not a storage person and I just started learning about Fibre Fabric and Zoning.

 Does my new zoning configuration make sense?  
Could it be improve?
Is anything on the way is configured causing my backup issue?

I know there are other things on the variable such as the backup server itself, backup software agent and the library that could be the culprit, but I need to start somewhere.

Thanks in advance for your help.

BEFORE

SANSwitch1 (3 Zones)

VNX_ESX
Members:  ESXi1, ESXi2, ESXi3, VNX_SPA_P2, VNX_SPB_P2
VNX_SOLARIS
Members: BackupServer1, TLTape1, VNX_SPA_P2, VNX_SPB_P2, SolarisServer
VNX_WIN
Members: BackupServer1, TLTape1, VNX_SPA_P2, VNX_SPB_P2

SANSwitch2 (3 Zones)

VNX_ESX
Members:  ESXi1, ESXi2, ESXi3, VNX_SPA_P3, VNX_SPB_P3
BACKUP_ZONE
Members: BackupServer1, DocImagingServer, VNX_SPA_P3, VNX_SPB_P3
VNX_WIN
Members: BackupServer1, TLTape2, VNX_SPA_P3, VNX_SPB_P3

AFTER

SANSwitch1

VNX_ESXi1
Members: ESXi1, VNX_SPA_P2, VNX_SPB_P2
VNX_ESXi2
Members: ESXi2, VNX_SPA_P2, VNX_SPB_P2
VNX_ESXi3
Members: ESXi3, VNX_SPA_P2, VNX_SPB_P2
VNX_SOLARIS
Members: SolarisServer, VNX_SPA_P2, VNX_SPA_P2
VRC_BACKUP
Members: VRCData, BackupServer1_1
BACKUP_TL
Members: BackupServer1_1, TLTape1
VNX_BACKUPSERVER
Members:  BackupServer1_1, VNX_SPA_P2, VNX_SPB_P2
            
SANSwitch2

VNX_ESXi1
Members: ESXi1, VNX_SPA_P3, VNX_SPB_P3
VNX_ESXi2
Members: ESXi2, VNX_SPA_P3, VNX_SPB_P3
VNX_ESXi3
Members: ESXi3, VNX_SPA_P3, VNX_SPB_P3
BACKUP_TL
Members: BackupServer1_2, TLTape2
VNX_BACKUPSERVER
Members:  BackupServer1, VNX_SPA_P3, VNX_SPB_P3
VNX_DOCIMAGINGSRV
Members: Poseidon, VNX_SPA_P3, VNX_SPB_P3
DOCIMGSRV_BACKUPSERVER
Members: DocImagingServer, BackupServer1_2
0
Comment
Question by:cartereverett
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 6
15 Comments
 
LVL 30

Accepted Solution

by:
Duncan Meyers earned 2000 total points
ID: 38838168
Yes - your 'after' zoning looks good, however there's a couple of things that need your attention. First is that the switches should be cross-connected to the SPs slightly differently. For SANSwitch1, you should have SPA Port 2 and SPB Port 3 connected.
For SANSwitch2, you should have SPA Port 3 and SPB Port 2 connected.

I'd recommend going slightly farther with your zoning and configure single initiator, single target zoninig, so:
VNX_ESXi1
Members: ESXi1, VNX_SPA_P2, VNX_SPB_P2

Becomes

VNX_ESXi1-SPA_P2
Members: ESXi1, VNX_SPA_P2 and

VNX_ESXi1-SPB_P3
Members: ESXi1, VNX_SPB_P3

The reason you do this is to prevent an issue down the track where the SPs try and log into themselves if you configure SANCopy of MirrorView for replication. Just saves you a bit of time later. If you're sure you won't be using either of those options, then stick with your proposed configuration.
0
 
LVL 17

Expert Comment

by:Gerald Connolly
ID: 38839057
Before the change, one of my backup jobs which used to take 11+ hours went down to 5+ but after creating the new zones my backup jobs is taking again 11+ hours to complete.

What happened to make it go down from 11+ to 5+ hours BEFORE you made the change
0
 

Author Comment

by:cartereverett
ID: 38839598
When it used to take 11+ hours the LUN that I was backing up was on an old iSCSI SAN.  When I moved the data to the new SAN and I did the backup, it went from 11 to 6 hours.  
So the change was the new SAN and on my initial tests zoning was configured no following single HBA zoning best practices.

Something very strange happened last night.  The AC unit in our server room stopped cooling and since the ETA for the technician was 30, to avoid damage to the unit I completely shutdown the VNX5300 including the additional DAEs.  The unit was off for over an hour.
I did my regular backups last night and the backup went back to 6 hours.
Now I'm really confused...
0
 [eBook] Windows Nano Server

Download this FREE eBook and learn all you need to get started with Windows Nano Server, including deployment options, remote management
and troubleshooting tips and tricks

 

Author Comment

by:cartereverett
ID: 38839689
Meyersd,

Thanks for answer.  Not to question your suggestion, but more to try to understand it...why should the switches be cross connected different?

I attached a diagram of how they are connected.
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38841192
It's EMC best practice. It's to protect against a very rare fibre loop failure condition. Port 2 on both SPs are logically on the same fibre loop.

I don't think the diagram came through.

Can you post more information about the backup job? How big is it? How is the SAN disk configured that the backup job is using?  Is the backup going straght to tape or are you doing disk to disk to tape?
0
 

Author Comment

by:cartereverett
ID: 38841628
Thanks for the explanation.  I created the new zones and now I have 1:1 single hba zoning configuration.

The backup job is about 1.3TB.  I created a 2TB LUN on Pool1.  Pool1 has two tiers (Performance (Raid 5 (4+1)) and Capacity (Raid 6 (6+2)).  
There was an initial full backup created.  Now a daily differential backup gets written to the LUN.  The differential happens on the Solaris box.  That portion takes about an hour to complete.  I then do a full backup of everything on the LUN to tape.

One thing I forgot to mention is that I mapped a LUN to a Windows server and it is showing twice (see attached picture) and some other DGC LUNZ are showing with red X’s.  
On the windows server I have a dual HBA connected to each switch.
Duplicate-LUNs.jpg
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38841761
Download PowerPath from Powerlink (https://powerlink.emc.com) and install it to resolve that problem. PowerPath handles path balancing and multi-path I/O. You can run it in unlicensed mode, but I'd recommend purchasing a license if this server is likely to need good storage performance.

You can also use Symantec Storage Foundation Basic as a freebie, but it is limited to two processors and 4 volumes under management. You can download it from the Symantec website.
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38841770
Try writing the backup directly to tape. If  you've got LTO-4 or LTO-5 you should be able to get over 400GB/hour
0
 

Author Comment

by:cartereverett
ID: 38841825
It is kind of cumbersome the way it works.  The Solaris server has an application which is vendor supported.  The application uses Oracle for the DB.
The vendor via the application provides a way to do the backup.  I can either use to tape or to disk.  
Either way will take the Oracle DB offline during the backup and bring them back online after the backup.  If I do a backup to tape the application does a full backup to tape which takes the Oracle DB offline during the 6+ or 11+ hours.  We can’t have the DB offline for more than two hours so the only way to accomplish this is by using the “To disk” option which does a differential backup to disk and it only take an hour to complete.  I then do my full backup to tape from a Windows server.

So I can’t really do the backup directly to tape.

Hopefully I’m making sense…
0
 

Author Comment

by:cartereverett
ID: 38841828
by the way, I have LTO6 TL
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38841843
Just a thought: you can do a Oracle RMAN backup direct to a Data Domain box with partial source-based dedupe - that would really fly.

For the pool that you're writing to: how much RAID 5 and how much RAID 6 disk do you have?
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38841844
And what tiering policy are you using?
0
 

Author Comment

by:cartereverett
ID: 38841854
3.5TB (raid 5)
32TB (raid 6)

"Start high then Auto-tier" policy
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38850066
If you zone the tape library to the Solaris server, can you have the Solaris server dump its disk-based data direct to tape?
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38853026
Thanks! Glad I could help.
0

Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article aims to explain the working of CircularLogArchiver. This tool was designed to solve the buildup of log file in cases where systems do not support circular logging or where circular logging is not enabled
A look at what happened in the Verizon cloud breach.
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
This tutorial will show how to configure a new Backup Exec 2012 server and move an existing database to that server with the use of the BEUtility. Install Backup Exec 2012 on the new server and apply all of the latest hotfixes and service packs. The…

715 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question