ISCSI or FIber

They're both right. Converged networking is now here and 10Gb Ethernet replaces them both with Fibre Channel over Ethernet and ordinary IP. It's still a little expensive, but it is the future. It's also referred to as Data Centre Ethernet. Having said all that, DCE leverages the Fibre Channel Protocol to overcome a lot of Ethernet's limitations, an dthe underlying storage protocol remains SCSI, whether it you use FC, iSCSI or FCoE.

>both solutions meet the following needs
>- 16TB of RAW space
>- Primary box at main datacenter all 15K 450GB drives
>- Second box across campus all SATA drives
>- Both solutions provide snapshots/volume copy and replication.

Some things to note: relying on raw space for sizing is extremely dangerous - you need to know how many IOPS your environment is generating. For example, your raw space requirements could potentially be fulfilled with 8 2TB SATA drives, but this will give you a maximum performance of 640 IOPS - 1000 IOPS. 16TB of 450GB drives will give you 7000 IOPS. A VMware (or any virtualisation environment for that matter) generates highly random I/O patterns, so you need to size the storage array with that in mind.  Also bear in mind that IOPS has an inverse relationship with bandwidth. When IOPS are high (that is; I/O is highly random), bandwidth will be low (it's common to see VMware environments generating 3000 IOPS and 20 MB/sec - no, that's not a typo). Conversely, low high bandwidth indicates a sequential I/O load and therefore throughput will be low. Sequential I/O loads include media serving and backup to disk.

>The IBM solution is fiber and is obviously going to be faster on the network side.  It has dual controllers that have 4 4GB ports each.  So that's a total of 32GB's of controller thruput.  Each server will have dual 4GB NICS.  It also replicates to the remote box via direct fiber connections.

The throughput of the front-end ports is almost 100% completely irrelevant. What matters (again... :-) ) is IOPS.

>- Primary box at main datacenter all 15K 450GB drives
>- Second box across campus all SATA drives
Be very careful performing synchronous replication from FC to SATA drives as you can easily overwhelm the available SATA performance at DR and affect production hosts. Ugly.

Until FCoE/DCE becomes ubiquitous and heaps cheaper, I recommend going with the FC option. The big advantage is low latency - important for any database application. Fibre Channel switching infrastructure is not expensive, especially when compared with implementing iSCSI properly.

To get a quick look at how much I/O your environment is generating, take a look at perfmon on Windows servers. Open perfmon, go to PhysicalDisk, select Current Disk Queue Length, % Disk Time and Disk Transfers/sec. You can then calculate Disk Service time = % Disk Time / Disk transfers/sec.

Disk Transfers/sec will give you the IOPS the host is generating - this is the important metric.
Disk Queue length shows if the host is waiting on disk access, and gives you a good idea of how hard the system is hitting disk.
Disk Service time is the disk response time which shoulkd be under 20mS

Add the IOPS of your hosts together at peak load and you have the total host load. You can then calculate how many disks you need:

Assume 50% read, 50% write. For the purposes of this example, assume you've measured 4,000 IOPS peak:
50% of 4,000 = 2,000 read IOPS
50% of 4,000 = 2,000 write IOPS

Calculate the RAID 5 write penalty (4 disk operations for each host I/O):
2,000 x 4 = 8,000 write IOPS at the disks
so total workload is 2000 read IOPS + 8000 write IOPS = 10,000 IOPS

Each 15K FC disk can handle 200 IOPS, each 10K FC disk can handle 140 IOPS and each SATA drive can handle 80 IOPS, so the number of disks you'll need are:

15K FC: 10,000 IOPS / 200  = 50 disks
10K FC: 10,000 IOPS / 140 = 72 disks
SATA: 10,000 / 80 = 125 disks

So you can see where the problem with sizing on space requirements lies. If you use SATA drives, you'll need 125 for the workload - or 125TB raw! Even 450GB disks will give you 22.5TB. If you size on space, you can see that you won't have anywhere near enough disk drives to handle the workload. Imagine trying to place the workload of 125 disk drives on only 16!

I don't have time to involve myself in the purely technical/numbers side of this deal, but here's what I CAN tell you.  Real-world.


At my previous company, I bought two IBM DS4300 Turbo's.  Every feature activated, 64 storage partitions, flash, close, snap, everything.  Dual controllers, each controller with two 2GB/s LC connections.  8 drive trays, all FC drives, combination of 146GB and 300GB.  I forget the exact number, but the raw storage was in the upper-20TB range on each.  The SANs connected to a pair of Cisco MDS9506 SAN directors, and VSANs were configured.  I back-ended three Exchange boxes, and an HP-UX box with a 2TB production database.  Also ran an ESX Enterprise cluster that supported nearly 60 VMs.


New job:  iSCSI from LeftHand (owned by HP now).  1GB ethernet.  SATA drives.  CRAP.  GARBAGE.  I want to set it on fire.  It will be replaced next year by FC.


Yes, yes, what you say about FC is partially true - the interface doesn't have as many "bells and whistles" as the iSCSI interface does.  Why?  iSCSI is a software-based solution, and it's usually found in entry-level shops, where you have people with entry-level skillsets.  FC is in mid-size and up shops, people that have a little more skill, experience, bigger budgets (in terms of both equipment and staff).  FC requires a lot more know-how.  You need to understand zoning, plan your oversubscription, etc etc.  Is it going away?  No.  Is iSCSI going away?  No.  They both have their market segment.  DCE, yes, there's been lots of noise about it, but I'm not so sure.  Reports?  Trending?  Again, that's your software-based iSCSI crap doing that.  You don't eversubscribe an IBM array.  There are no reports.  You have this much storage, that's what you get, use it wisely.  You don't need to know how fast you're filling things up and expanding volumes, because it doesn't happen.  Build it correctly the first time, and it'll perform like a CHAMP.

& if you're THAT concerned about the speed, FC is already up to 8GB/s.  Why are you buying a 4GB/s SAN?

No matter how you slice it, at the end of the day, IOPs go like this:

SATA
SAS
FC
SSD


Cost goes up as you go.  If you can get FC for only $10k more than iSCSI, personally, I think it's a COMPLETE no-brainer.  Just be prepared for banging your head on the wall while you learn a new technology.  It's not just plug-n-play, as some vendors may have you believe.  The interface might be uglier, but there are WAY more things you can do in there, right down to controlling the block size on a per physical partition basis.  As for automagically expanding LUNs?  Not a fan.  The whole storage over-subscription thing is just a bad idea IMHO.  Storage should not be a "black box", especially if you're centralizing where all of that storage is.  I want to know where my data is, be able to configure different RAID levels for each different partition, carve up the LUNs how I see fit.  If I need a boot LUN, I just RAID1 a pair of 146's and go to town.  If I need SPEED, I can RAID10 16 drives and get fast.  If I need big, RAID5 8 300GB drives and go.  Exchange log partition?  4k blocks.  SQL database partition?  64k blocks.  Most of your iSCSI stuff doesn't allow for that.  You just tell it how big you want, and it "handles" the rest.

Check that IBM solution and make sure it includes SAN switches, though.  They are pricey, and the licensing is weird.

One last thing - FC doesn't use "NICs".  It uses HBAs (Host Bus Adapter).  And they are MUCH more expensive than iSCSI TOE adapters.  FC cabling is more $$ too, but I'm sure you knew that also.  :-)


HTH,
exx