The fastest storage solution but low cost?

PaperTiger
PaperTiger used Ask the Experts™
on
I am a little confused. Please correct me if I am wrong.

For Directly Attached Storage (DAS):
The limitation is ether on the hard drive or the controller.
On SATA drives, the top speed is about 250MB/s
On SAS drives, the speed is limited on the PCIe controller, somewhere 300-400MB/s

For SAN/NAS
The limitation is on the network. If it's 1Gbps and the NIC is PCIe, there's no way for data transfer rate to go over 100MB/s

I have also looked at both Gluster and OpenFiler, but couldn't figure out how to use them to increase my throughput.

It seems the best option is to use DAS with RAID 0 or 1+0 SATA drives with a good PCIe controller.

Is this right?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
DavidPresident
Top Expert 2010
Commented:
For Directly Attached Storage (DAS):

Q. The limitation is ether on the hard drive or the controller.
 - No.  Bottlenecks can be anywhere, like CPU, or just inefficient I/O requests.  Imagine if your O/S or app asked for 1MB of data when you only wanted 512 bytes, or vise-versa.

Q. On SATA drives, the top speed is about 250MB/s
 - on a single mechanical disk, when data is in cache, and request is a large I/O.  Change some parameters, and the same disk could very well deliver under 40MB/sec
Q. On SAS drives, the speed is limited on the PCIe controller, somewhere 300-400MB/s
 See above for generalization, but you could very easily get several times that with the right configuration, or get to the 40MB/sec or less with wrong one.

Q. For SAN/NAS The limitation is on the network. If it's 1Gbps and the NIC is PCIe, there's no way for data transfer rate to go over 100MB/s
No. Limitations can be anywhere.  In addition a crappy dumb ethernet card on a saturated switch or one that doesn't sync up, and you can find yourself measuring in KB/sec instead of MB/sec.   You can bind ports and use a quad card and get 400MB/sec.
I have also looked at both Gluster and OpenFiler, but couldn't figure out how to use them to increase my throughput.

Q. It seems the best option is to use DAS with RAID 0 or 1+0 SATA drives with a good PCIe controller.
A gross generalization is yes.  Get an ethernet switch and a dual or quad PCIe controller and you can combine ports to get 400MB/sec



Author

Commented:
Would you suggest a few ways to combine NICs? I am really interested in seeing how high speed, 400MB for example, can be achieved relatively cheaply.

The answer depends on your io usage pattern :
-IOPS based io usage (random io) : Database data access is mainly random io; most io usages are very sensitive about this random access time
-Sequential io usage : WORM, Backup, Archive, Logs, etc
==> The fastest IOPS storage is SSD based where Sequential storage should stay on HDD (for price reason)

IOPS performance relies on the access time of your physical media (SSD < SAS 15k < HDD 10k < HDD 7200 < HDD 5400 < DVD...Network...Tapes...) and the HBA capability to use its large cache (backed by a battery) to order random io in a more sequential way
==> The best 2009 SSD may deliver almost 40MB/s throughput (rw 4KB pages), the best SAS 15k HDD may reach a superb 6MB/s ...and I would not expect any other bottleneck on the interfaces

Sequential throughput is to be mesured as a sustained throughput (the cache hit ratio is good at the application side...and stay poor below) : SSD will soon reach a sustained 270MB/s (Max out by the SATA 3Gbs and SAS 3Gbs protocol) where HDD stay between 80 (inner part of the platters) and 140MB/s (outer part)
==> Of course this will easily saturate the io chain...but do you need that ?
11/26 Forrester Webinar: Savings for Enterprise

How can your organization benefit from savings just by replacing your legacy backup solutions with Acronis' #CyberProtection? Join Forrester's Joe Branca and Ryan Davis from Acronis live as they explain how you can too.

DavidPresident
Top Expert 2010

Commented:
Just purchase NICs that support port bonding for whatever operating system(s) you are running, and make sure your ethernet switch is compatible with them.  Bonding is relatively common with premium equipment like HP & DELL.  Read the specs or google the term.

Author

Commented:
I think mostly my application would be random IOPS. The purpose is to create virtual desktop machines on top of this server. So most users would use them for web browsing, MS Office stuff and file sharing downloads.

Should I go with directly attached or network storage solution?
Port bonding/bridging is not that easy...because this is a BI-directionnal P2P interface.
It requires the SAN/iSCSI server to use port bonding as well...which is a specific feature that is usually relying on some mandatory prerequisites (Ex: Switch)

Regarding your random io usage, you may rely on SAN iSCSI or DAS without problem...may be NAS will add some overhead latency delay (unsure).

Anyway, the "fastest storage solution but low cost" depends on your IOPS and storage size needs. I would choose between:
-Good 2.5" SSD (in RAID 5/6/50/60) can sustained a lot more IOPS than many SAS 15k (cf. http://www.anandtech.com/IT/showdoc.aspx?i=3532&p=11 )...so it costs LESS money than their equivalent with SAS 15k performance...but offer LESS available storage size
-Bunch of low cost 7200 rpm 2.5" 500GB SATA (24x in 2U) in RAID 10
-Bunch of cheap 7200 rpm 3.5" 1TB+ SATA (12x in 2U) in RAID 10

In summary, I would start with 2U storage servers with a mix of 2.5" SATA HDD and SSD
Top Expert 2014

Commented:
Port bonding would certainly require a different initiator than the Microsoft one since that inly supports MPIO, StarWind supports teaming (bonding) but you could just run 10gig ethernet. You're better off with direct attached anyway, what's the sense in seperating the disks from the server with a bit of ethernet or fibre?

A bunch or virtual desktops isn't likely to need much disk space assuming the users files are stored elsewhere on the network but it is going to create very random I/O so you should steer well clear of slow disks. 15K SAS or SSD as mentioned above is the way to go.

Author

Commented:
Andy, I guess I don't quite understand here. In a virtual environment, how do I separate random I/O from network storage? I assume the random I/O would go where the virtual machine files are stored, no? If the vm files are stored on network storage, isn't the random I/O between the server and the network storage over the Ethernet?
Top Expert 2014

Commented:
Why do you need network storage (apart from user files which would probably be on a seperate fileserver anyway connected on LAN ising CIFS)? Unless you're sharing SAN storage between several servers DAS is faster and cheaper than adding the unnecessary complexity of SAN.

Author

Commented:
Andy, so you are suggesting a combo:

For each vm, I would have two drives:
Drive C: OS and applications, this sits on DAS
Drive D: mapped network drive for storage only, this uses NAS
Top Expert 2014
Commented:
Yes, that's what I'd do just like a normal office environment with PCs. After all you have to back up the users data but it's hardly worth backing up all those VMs. Don't know that I'd use D for the mount point of the shares though, H: for home and S: for shared maybe.

Author

Commented:
How do you re-configure all the "Documents and Settings" folders such as "my documents" stuff to S drive then?
Top Expert 2014

Commented:
With group policy - http://technet.microsoft.com/en-us/library/cc781907(WS.10).aspx or you can let the users do it themselves if you aren't using group ppolicy - r.click my documents, properties, target, move.
Top Expert 2014

Commented:
Thanks for points, but I think you should have split them, dlethe's first comment about speed and BigSchmuh's one about teaming were very useful.

Author

Commented:
realized that afterwards. how can I split now?
Top Expert 2014

Commented:
Click "request attention" and CS will re-open.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial