Solved

Backup Performance and Fibre Channel Zoning

Posted on 2013-01-30
15
1,544 Views
Last Modified: 2013-02-04
Hi,

We recently acquired an EMC VNX5300 and a FC Tape Library.
The original zoning configuration was not following EMC’s best practice for single HBA zoning so I went ahead and created new zones.
Before the change, one of my backup jobs which used to take 11+ hours went down to 5+ but after creating the new zones my backup jobs is taking again 11+ hours to complete.

To accomplish the backup, I am using a windows 2003 server with Arcserve R16 on it.  This server, BackupServer1 has a dual 8Gb HBA and connects to both FC switches.
The backup job backs up a LUN that it is attached to a Solaris server.  The Solaris server uses a single HBA 4Gb connected to SANSwitch1.

The VNX_SPA, VNX_SPB are the storage processors for the VNX 5300.

The tape library is an (Overland) IBM 3573 FC with dual LTO6 drives.  I have one drive attached to SANSwitch1 and the other attached to SANSwitch2.

The switches are EMC (Brocade) DS-300B 8Gb FC.

I am obviously not a storage person and I just started learning about Fibre Fabric and Zoning.

 Does my new zoning configuration make sense?  
Could it be improve?
Is anything on the way is configured causing my backup issue?

I know there are other things on the variable such as the backup server itself, backup software agent and the library that could be the culprit, but I need to start somewhere.

Thanks in advance for your help.

BEFORE

SANSwitch1 (3 Zones)

VNX_ESX
Members:  ESXi1, ESXi2, ESXi3, VNX_SPA_P2, VNX_SPB_P2
VNX_SOLARIS
Members: BackupServer1, TLTape1, VNX_SPA_P2, VNX_SPB_P2, SolarisServer
VNX_WIN
Members: BackupServer1, TLTape1, VNX_SPA_P2, VNX_SPB_P2

SANSwitch2 (3 Zones)

VNX_ESX
Members:  ESXi1, ESXi2, ESXi3, VNX_SPA_P3, VNX_SPB_P3
BACKUP_ZONE
Members: BackupServer1, DocImagingServer, VNX_SPA_P3, VNX_SPB_P3
VNX_WIN
Members: BackupServer1, TLTape2, VNX_SPA_P3, VNX_SPB_P3

AFTER

SANSwitch1

VNX_ESXi1
Members: ESXi1, VNX_SPA_P2, VNX_SPB_P2
VNX_ESXi2
Members: ESXi2, VNX_SPA_P2, VNX_SPB_P2
VNX_ESXi3
Members: ESXi3, VNX_SPA_P2, VNX_SPB_P2
VNX_SOLARIS
Members: SolarisServer, VNX_SPA_P2, VNX_SPA_P2
VRC_BACKUP
Members: VRCData, BackupServer1_1
BACKUP_TL
Members: BackupServer1_1, TLTape1
VNX_BACKUPSERVER
Members:  BackupServer1_1, VNX_SPA_P2, VNX_SPB_P2
            
SANSwitch2

VNX_ESXi1
Members: ESXi1, VNX_SPA_P3, VNX_SPB_P3
VNX_ESXi2
Members: ESXi2, VNX_SPA_P3, VNX_SPB_P3
VNX_ESXi3
Members: ESXi3, VNX_SPA_P3, VNX_SPB_P3
BACKUP_TL
Members: BackupServer1_2, TLTape2
VNX_BACKUPSERVER
Members:  BackupServer1, VNX_SPA_P3, VNX_SPB_P3
VNX_DOCIMAGINGSRV
Members: Poseidon, VNX_SPA_P3, VNX_SPB_P3
DOCIMGSRV_BACKUPSERVER
Members: DocImagingServer, BackupServer1_2
0
Comment
Question by:cartereverett
  • 8
  • 6
15 Comments
 
LVL 30

Accepted Solution

by:
Duncan Meyers earned 500 total points
ID: 38838168
Yes - your 'after' zoning looks good, however there's a couple of things that need your attention. First is that the switches should be cross-connected to the SPs slightly differently. For SANSwitch1, you should have SPA Port 2 and SPB Port 3 connected.
For SANSwitch2, you should have SPA Port 3 and SPB Port 2 connected.

I'd recommend going slightly farther with your zoning and configure single initiator, single target zoninig, so:
VNX_ESXi1
Members: ESXi1, VNX_SPA_P2, VNX_SPB_P2

Becomes

VNX_ESXi1-SPA_P2
Members: ESXi1, VNX_SPA_P2 and

VNX_ESXi1-SPB_P3
Members: ESXi1, VNX_SPB_P3

The reason you do this is to prevent an issue down the track where the SPs try and log into themselves if you configure SANCopy of MirrorView for replication. Just saves you a bit of time later. If you're sure you won't be using either of those options, then stick with your proposed configuration.
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 38839057
Before the change, one of my backup jobs which used to take 11+ hours went down to 5+ but after creating the new zones my backup jobs is taking again 11+ hours to complete.

What happened to make it go down from 11+ to 5+ hours BEFORE you made the change
0
 

Author Comment

by:cartereverett
ID: 38839598
When it used to take 11+ hours the LUN that I was backing up was on an old iSCSI SAN.  When I moved the data to the new SAN and I did the backup, it went from 11 to 6 hours.  
So the change was the new SAN and on my initial tests zoning was configured no following single HBA zoning best practices.

Something very strange happened last night.  The AC unit in our server room stopped cooling and since the ETA for the technician was 30, to avoid damage to the unit I completely shutdown the VNX5300 including the additional DAEs.  The unit was off for over an hour.
I did my regular backups last night and the backup went back to 6 hours.
Now I'm really confused...
0
 

Author Comment

by:cartereverett
ID: 38839689
Meyersd,

Thanks for answer.  Not to question your suggestion, but more to try to understand it...why should the switches be cross connected different?

I attached a diagram of how they are connected.
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38841192
It's EMC best practice. It's to protect against a very rare fibre loop failure condition. Port 2 on both SPs are logically on the same fibre loop.

I don't think the diagram came through.

Can you post more information about the backup job? How big is it? How is the SAN disk configured that the backup job is using?  Is the backup going straght to tape or are you doing disk to disk to tape?
0
 

Author Comment

by:cartereverett
ID: 38841628
Thanks for the explanation.  I created the new zones and now I have 1:1 single hba zoning configuration.

The backup job is about 1.3TB.  I created a 2TB LUN on Pool1.  Pool1 has two tiers (Performance (Raid 5 (4+1)) and Capacity (Raid 6 (6+2)).  
There was an initial full backup created.  Now a daily differential backup gets written to the LUN.  The differential happens on the Solaris box.  That portion takes about an hour to complete.  I then do a full backup of everything on the LUN to tape.

One thing I forgot to mention is that I mapped a LUN to a Windows server and it is showing twice (see attached picture) and some other DGC LUNZ are showing with red X’s.  
On the windows server I have a dual HBA connected to each switch.
Duplicate-LUNs.jpg
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38841761
Download PowerPath from Powerlink (https://powerlink.emc.com) and install it to resolve that problem. PowerPath handles path balancing and multi-path I/O. You can run it in unlicensed mode, but I'd recommend purchasing a license if this server is likely to need good storage performance.

You can also use Symantec Storage Foundation Basic as a freebie, but it is limited to two processors and 4 volumes under management. You can download it from the Symantec website.
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38841770
Try writing the backup directly to tape. If  you've got LTO-4 or LTO-5 you should be able to get over 400GB/hour
0
 

Author Comment

by:cartereverett
ID: 38841825
It is kind of cumbersome the way it works.  The Solaris server has an application which is vendor supported.  The application uses Oracle for the DB.
The vendor via the application provides a way to do the backup.  I can either use to tape or to disk.  
Either way will take the Oracle DB offline during the backup and bring them back online after the backup.  If I do a backup to tape the application does a full backup to tape which takes the Oracle DB offline during the 6+ or 11+ hours.  We can’t have the DB offline for more than two hours so the only way to accomplish this is by using the “To disk” option which does a differential backup to disk and it only take an hour to complete.  I then do my full backup to tape from a Windows server.

So I can’t really do the backup directly to tape.

Hopefully I’m making sense…
0
 

Author Comment

by:cartereverett
ID: 38841828
by the way, I have LTO6 TL
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38841843
Just a thought: you can do a Oracle RMAN backup direct to a Data Domain box with partial source-based dedupe - that would really fly.

For the pool that you're writing to: how much RAID 5 and how much RAID 6 disk do you have?
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38841844
And what tiering policy are you using?
0
 

Author Comment

by:cartereverett
ID: 38841854
3.5TB (raid 5)
32TB (raid 6)

"Start high then Auto-tier" policy
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38850066
If you zone the tape library to the Solaris server, can you have the Solaris server dump its disk-based data direct to tape?
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 38853026
Thanks! Glad I could help.
0

Featured Post

Network it in WD Red

There's an industry-leading WD Red drive for every compatible NAS system to help fulfill your data storage needs. With drives up to 8TB, WD Red offers a wide array of solutions for customers looking to build the biggest, best-performing NAS storage solution.  

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Cisco RSTP portfast 3 53
Disk provisioning with EMC Unity flash storage? 2 85
esx multi vlans 3 63
Can't create RAID on a HP Z220 CMT, preparation for VMWare. 6 81
AWS Glacier is Amazons cheapest storage option and is their answer to a ‘Cold’ storage service.  Customers primarily use this service for archival purposes and storage of infrastructure backups.  Its unlimited storage potential and low storage cost …
I previously wrote an article addressing the use of UBCD4WIN and SARDU. All are great, but I have always been an advocate of SARDU. Recently it was suggested that I go back and take a look at Easy2Boot in comparison.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This tutorial will walk an individual through the steps necessary to install and configure the Windows Server Backup Utility. Directly connect an external storage device such as a USB drive, or CD\DVD burner: If the device is a USB drive, ensure i…

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now