Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Hardware RAID

Posted on 2014-03-31
22
Medium Priority
?
419 Views
Last Modified: 2016-12-08
If using a hardware raid solution what is the host computer doing? For example, If I am using older hardware with a hardware raid solution should I be less concerned with main processing power and memory?

All the box will be doing is acting as a SAN and using NIC teaming with 2, 4Gb PCIe Fibre Channel Host Bus Adapter.  The processor is a dual core 2.0 GHZ AMD with 2 GB of ram. Do I really need to upgrade this since I will be adding a HARDWARE raid controller? Will it be able to process the network speeds required with a teaming set up?
0
Comment
Question by:KonfigurationKing
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 5
  • 5
  • +2
22 Comments
 
LVL 20

Expert Comment

by:carlmd
ID: 39968887
If you mean add a raid card to a system, it will make a difference in that you have added something to the system, but the "work" is done on the card to distribute the data across the RAID array. You are still writing data to a "disk". If the system is on the edge, it will matter. If the system had adequate memory, cpu etc,, it won't.

If you are referring to a totally external RAID array just connected to the bus (USB for example), it won't matter any more then servicing just an external disk. In this case the work is done externally and all you are doing if funneling the data.

Bottom line, if the system has the reserver capacity to receive and write the data you should be good in either case.
0
 
LVL 56

Expert Comment

by:andyalder
ID: 39969539
Depends what SAN software you are using as to how much RAM it needs and depends on what controller you are using and how many disks you have to know how fast it will be.
0
 
LVL 56

Expert Comment

by:andyalder
ID: 39969547
Depends what SAN software/OS you are using to know how much RAM/CPU resource you need and depends on what controller and disks you have to know how fast it is but you'll need a lot of hard disks to get anywhere near gigabit Ethernet speeds.
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 

Author Comment

by:KonfigurationKing
ID: 39970872
How do you know that it will take alot of harddisks to reach gigabit speeds? What if I take 6 harddrives with a maximum of 6GB a sec in a Raid 0 configuration? You are saying I would not need more than a 1gb ethernet?
0
 
LVL 27

Assisted Solution

by:skullnobrains
skullnobrains earned 1336 total points
ID: 39981272
even software raid would work if you stick to raid 0 or raid 1 without important performance impact on the processor

i concur that you will need LOTS of disks. a regular SATA disk can do about 100-120 Mo/s. SAS drives will reach about 200. some high level SSDs can do much more. the 6GB is the maximum burst speed of the SATA3 interface,  but no disk currently produce such a throughput. on linux and windows you most likely should divide these figures by 2.

then you may want to note that

- most raid1 implementations (including hardware) actually don't show a better performance throughput than a single disk

- the bus that you will plug the raid card in probably cannot achieve such speeds anyway. you'll have to split between several controllers which excludes hardware raid (or use a software raid of several hardware raids)

- the board you are using probably won't be able to cope with such a throughput even with DMA

- 2Gb of RAM is far from what would be reasonable if you expect such IO performance
0
 
LVL 17

Assisted Solution

by:Gerald Connolly
Gerald Connolly earned 444 total points
ID: 39982702
@Brains, you use a lot of mixed nomenclature! Mo/s - ?, 6GB - 6Gbit/s, 2Gb - 2GBytes - Note its usual to use "B" for bytes and "b" for bits.

I agrre with you that although the current SAS standard is 6Gbits/s this is just the size of the pipe, the best Enterprise class disk only do 200ish MB/s and also 200 IOPS, but a SATA disk is typically down less than 100 for both of these numbers, so 3 * Enterprise Disks might stress a single SAS pipe you are looking at maybe 8 * SATA disks to do the same thing.

Back to the original question. Yes you are right if you are using a server to act as a SAN gateway it doesnt have to be particularly high spec, but it probably needs a decent amount of RAM and 2GB is not enough, and if it is going to be running multiple GbE links it will need some horsepower to drive them at any kind of speed.
0
 
LVL 27

Assisted Solution

by:skullnobrains
skullnobrains earned 1336 total points
ID: 39983020
my bad 6GB should be read 6Gb/s

i'm unsure about the 200 IOPS. from experiments with database servers, i usually ended up around 250 on freeBSD and for some reason, more like over a thousand on linux. I assume some caches were in the way (iostat probably hooked above hardware level) and/or the transactions may have been smaller since overall, equivalent db setups perform better on bsd. the disks will actually handle more transactions per unit of time but the throughput drops dramatically when the threshold is reached.

i'll maintain proper quality SATA disks usually do 100-120Mo/s (or MB ;). lower figures usually are produced by wrong partition alignment (which is about always the case on windows, and the case on most linux boxes i came across)

from my experience raid 0, and 10 achieve about theorical speed increases when properly configured, RAID 1 seldom does because of frequent crazily poor implementations. above 4-5 SATA disks, it becomes necessary to split the disks on multiple controllers. also note that most SATA disks achieve better throughput on the beginning of the drive, so partitioning only part of the drive is one way to achieve overall better speed.

some SAS controllers do better than SATA III. 12Gb/s is not uncommon.

---

if you expect 8Gb throughput, you'll need something like an I5, a PCI-X bus doing at least x8 but x16 would be better, 2-4 raid cards (possibly 1 or 2 builtin SATA controllers can replace a number of raid cards), 16Gb of RAM, and about 10-12 SATA disks.

obviously this would be foolish because a RAID 0 of 10 disks is a no-go, so you'll have to do RAID 10 (or a 0 1 0 or 1 0 1 mix), use 4 RAID cards ( can be more or less depending on the raid cards ), and about 20 SATA disks

i'd also add an SSD for the filesystem's log and possibly a read cache, and tweak parts of the OS because it's log files (for example) may well become a bottleneck

note that this is achievable for much less $$ than an equivalent SAN (i'd say at least 10x less), but it requires quite some work to setup. distributed filesystems (or data) on lots of machines may be an alternative worthy to consider depending on your application

btw, why the hell do you need such tremendous speeds ?
0
 
LVL 56

Expert Comment

by:andyalder
ID: 39984012
You're doing a heck of a lot of work considering the questioner hasn't even said what SAN software they are using nor what RAID controller they have.
0
 

Author Comment

by:KonfigurationKing
ID: 39984751
Here is the device being used for the RAID controller:   LSISAS 2108 8-port PCI-E SAS-2 controller

  8-port (8 internal) 6Gb/s per port
    Supports RAID 0, 1, 5, 6, 10, 50, 60
    On board hardware IO Processor
    512MB 800MHz DDR2 on-card cache
    Advanced array configuration and management utilities
    Automatically negotiates PCI-E link widths
    Power management support
    Supports 1.5, 3.0 and 6.0 Gb/s SAS and SATA data transfer rates
    Port independent auto-negotiation
    Supports SSP, SMP, STP and SATA protocols
    Optional Battery Backup Unit for cached data protection (BTR-0018L-0000-LSI) - operating temperature (Ambient) from 10C° - 45C°
    OS Support : Windows XP/2003/2008/Vista, RedHat Enterprise and SUSE Linux
0
 

Author Comment

by:KonfigurationKing
ID: 39984774
@ skullnobrains

The reason I am setting up this lab is for experience. Why you ask so much speed? I have money to waste and frankly...BECAUSE ITS COOL!  :)

Right now I am supporting windows desktop only. Can I support enterprise environments if I have no experience ?  I hope to advance my career and become  like all of you experts and be the one answering NOT asking.

Scouring the internet trying to find information is challenging, because most articles assume the reader at a certain level of understanding.  A lot of the people here on EE have exceptional knowledge and get to work with server side IT systems. I want the benefit of that same experience, and be able to obtain mastery level at various open source technologies. I know of no way to gain this expertise other than to implement an enterprise like environment.

I am starting to realize the power of the free software movement. Open Source technology is a powerful tool that liberates a company or person from vendor lock in.
0
 
LVL 17

Expert Comment

by:Gerald Connolly
ID: 39985237
Your original post talked about Fibre Channnel HBA's are they now not in the picture, and is it only SAS that is important now?
0
 
LVL 27

Accepted Solution

by:
skullnobrains earned 1336 total points
ID: 39985358
 8-port (8 internal) 6Gb/s per port

you definitely need to test this for yourself : you'll most likely find out that transfer rates drop way before

start in JBOD (no raid)
saturate one drive with a single read thread and mesure the speed
saturate a second drive
... a third

at some point speeds are going to drop, either because you hit the bus's limit or the card's

it is interesting if you have time to experiment with multiple threads as well

obviously repeat with write threads. if you see a difference between read and write, change the write-back property

Why you ask so much speed? I have money to waste and frankly...BECAUSE ITS COOL!  :)

honestly, you should start trying to saturate a 1GB link. already hard enough, and much better than what most commercial SANs do regardless what they say. this is feasible with a single machine for about 4000 bucks and will already bring you the satisfaction to do much better than things HP or adaptec sell above 50K

you'll have a hard time to find an application to even test it anyway ;)

Right now I am supporting windows desktop only. Can I support enterprise environments if I have no experience ?

if you expect to run such a setup on windows, i don't think you have a chance, but i'd be more than happy to be proven wrong

if you want to support other things, it's all about learning and opensource software is well documented. given enough time, you will definitely be able to do lots of things

for this specific project, i'd recommend using zfs (filesystem and raid management) running on solaris of freebsd

I know of no way to gain this expertise other than to implement an enterprise like environment.

yes. books help, but they will never replace experience. then you'll probably need to actually provide services at some point rather than just a test lab

I am starting to realize the power of the free software movement. Open Source technology is a powerful tool that liberates a company or person from vendor lock in.

hmmm, correct but not completely. some opensource companies actually sell and license their software (redhat, novell...). GPL based opensource software is far from being "freely redistributable" as the licence says because they are quite a few conditions to meet. some companies/individuals have policies that free their customers, some don't, even in the opensource world.

if you want to be free, rather think BSD, Apache, CDDL (previously used by SUN), ...

note that not everybody agrees on such matters even in the opensource community. read about the openSSL exception of the GPL for example if you wan't to grab an idea. (wear a flame-proof suit ;)
0
 
LVL 17

Expert Comment

by:Gerald Connolly
ID: 39985416
Its not that hard to saturate a 1GbE link (100MB/s), but quite a lot harder to saturate a 10GbE link (1GB/s)
0
 
LVL 27

Assisted Solution

by:skullnobrains
skullnobrains earned 1336 total points
ID: 39985591
1GB is what the author wanted. i misread the question the second time when i stated "to start with", thanks for pointing out.

actually fairly easy using a bunch of 5-6 SAS drives with no redundancy providing the card can cope with that (which i hardly believe). much harder with redundancy, much harder with SATA drives (see above)... but feasible on reasonable hardware nevertheless

note that for a production system, 4 SAS drives in a raid 10 array would be a good match for that network speed : it won't be able to saturate it on the long term but will provide bursts that will.

i think it is reasonable to start with that motherboard and card. most likely at least additional RAM will be necessary. additional horsepower would depend on the actual use but that proc may be enough for at least a high performance san using iscsi. with no redundancy and a bunch of 10-12 SATA drives, the author should be able to achieve his goal and learn about quite a few pitfalls/limitations/tricks along the way.

adding redundancy should produce about half the write throughput and possibly the same read throughput which will already saturate the network most of the time and produce a production-level SAN
0
 
LVL 56

Assisted Solution

by:andyalder
andyalder earned 220 total points
ID: 39985842
The card has a dual-core RoC so it's not going to use any server CPU/RAM except for the drivers, still don't know what SAN software is going to be used.

The card can do 400,000 IOPS using SSDs but if the author is using 7.2K SATAs with a 4K random workload they'll only get about 400KB per disk so that would need 25 of them using RAID0 to get 100MB/s giving gigabit speed. If they're streaming it's a different kettle of fish since there aren't the random seeks.
0
 
LVL 27

Assisted Solution

by:skullnobrains
skullnobrains earned 1336 total points
ID: 39986166
where do the 400KB per disk figure comes from ?

as far as SAN software, i assume the author has not elected one yet. i'd suggest zfs. any other ideas ?

as far as CPU goes, i'm concerned with global throughput (disk-to-network) and not RAID operations. any modern computer can deal with RAID 0 1 and combinations easily without the help of the card. from my experience software raids are often better than hardware.

note : i think it is quite premature to worry about random workload before you get optimal performance with linear operations, and i also think it is better to start benchmarking locally before doing any bench over a network
0
 
LVL 56

Expert Comment

by:andyalder
ID: 39986192
4K * 100 IOPS (actually a bit generous). Since they haven't said what they are going to use this "SAN" for it's safest to suspect the worst which is OLTP database.
0
 
LVL 27

Expert Comment

by:skullnobrains
ID: 39986426
that is less than what i'm used to observe on a single SATA drive including with an sql database. the IO transfer size does not have to match the sector size. additionally you hardly ever use 4K blocks for rdbms
0
 

Author Comment

by:KonfigurationKing
ID: 39988042
Question by Gerald Connolly: Your original post talked about Fibre Channnel HBA's are they now not in the picture, and is it only SAS that is important now?

Answer: I was able to acquire 2 Dell FX775 Poweredge R805 Dual Port 10GBe PCIE. I was able to find the name of the chipset manufacturer and will have access to the proper drivers to load in GNU/LINUX (http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/SearchByProduct.aspx?ProductCategory=325&Product=1094&Os=65&ProductName=NXB-10GCX4&OSname=Linux%20Red%20Hat%20%2864-bit%29). I will post another question regarding network capabilities of the card.  I will only be using SATA DISKS so, it seems from the comments that the bottle neck will defiantly be on the machine bus.


as far as SAN software, i assume the author has not elected one yet. i'd suggest zfs. any other ideas ?

@skullnobrains and @andyalder /  andyalder too
I thought ZFS was just a filesystem like ext3/ext4 or NTFS!  Do you have a link to the recommendation you made?
I was thinking about Glusterfs (http://www.gluster.org/) on CENTOS but I am open to your suggestions and others. I have a little Fedora Linux experience and I use it as my primary desktop. I have never used Unix before and I don't know if my NIC card drivers can be ported to other *nix based systems. .My SAN storage will be used with XEN running CENTOS or other nix based system recommended...


....With no redundancy and a bunch of 10-12 SATA drives, the author should be able to achieve his goal and learn about quite a few pitfalls/limitations/tricks along the way.....

I will eventually purchase more drives for the array but for now it will only comprise of 4 SATA drives. I have more drives but they are a different make and size and I don't know if you can mix and match manufacturer types and sizes.


...from my experience software raids are often better than hardware....

Software raids better than hardware raids!?!!?! WOW that is impressive! Makes me want to test that out in the lab. It would also make me wonder how that would be possible since the controller should have optimized components/software?


...optimal performance with linear operations, and i also think it is better to start benchmarking locally before doing any bench over a network...

 *your great idea* I will also do as you recommend and get performance results locally before network bench. What kind of software should I use to get the needed test results?


if you expect to run such a setup on windows, i don't think you have a chance, but i'd be more than happy to be proven wrong


Running this setup on windows ? I think that would kill me a little on the inside :) Though I understand it is not entirely windows fault but at work because of the massive security (mcafee) / group policies pushed from the enterprise the machines are slow and not very efficient . I want to advance my career by gaining experience supporting heterogeneous environments with mostly *nix based tech.


GPL based opensource software is far from being "freely redistributable"

Darn you ruined it! I was just about to put up a statue of Richard Stallman in my back yard! Seriously though, that really surprised me I did not know that. I always assumed that GPL based software had protections built in to protect the user. Of coarse, I have not read the various GPLs and even if did I probably would not understand them. So, instead of hiring a lawyer to interpret the license I will take your word for it :)

Honestly, I really don't know what I don't know! Like skull was saying this is going to be an experience to learn more. As always I appreciate all the help...
0
 
LVL 27

Assisted Solution

by:skullnobrains
skullnobrains earned 1336 total points
ID: 39988163
I will post another question regarding network capabilities of the card.  I will only be using SATA DISKS so, it seems from the comments that the bottle neck will defiantly be on the machine bus.

don't : you need to test this yourself. this is even more important if you're doing this for educational reasons. use dd to generate reads and writes on the raw disks to start with. then play with options in the card (write-back will change many things). play changing the number of disks, play changing the number of threads, then play with a filesystem, and random reads/writes. what you'll learn is invaluable.

note that cards capabilities are usually far from what the vendor says

also note that RAID setups which are not first tested in this way, and configured to reach about theorical speeds (5% tolerancy tops) with a simple dd tests are most of the time heavily misconfigured. this (unfortunately) applies to most setups you'll see in real life.

I thought ZFS was just a filesystem like ext3/ext4 or NTFS!  Do you have a link to the recommendation you made?

no : zfs combines a volume manager and a filesystem. it supports raid levels similar to the usual ones with a few notable differences : raid0 is actually not partitionned so it is soft load sharing, and raidz is similar to raid 3 or 5 without the write hole.

adminitration is easy, and disks can be added or replaced on-the-fly without ever waiting for the raid to be built, rebuilt...

i have no specific link to provide because you'll find lots of docs on the internet, but i'll be happy to answer any question.

on linux, it was only supported through fuse last time i checked. if the kernel module is mature, it may be worth wile, if not i'm unsure about using it on linux

I was thinking about Glusterfs

don't : glusterfs is a distributed filesystem, suitable for hosting a single fs distributed on many computers. it is unrelated with what you are doing. btrfs might be an idea as it also includes a volume manager. otherwise, use raid hard or soft and any regular filesystem (ext, xfs, reiser) on top of that.

I don't know if my NIC card drivers can be ported to other *nix based systems

a general answer would be a linux driver will work on any linux distribution using a similar kernel version, and porting drivers will otherwise require lots of work. then it is easier to port a freebsd driver to dragonfly than to port an AIX driver to netbsd (examples pulled quite randomly from the top of my head)

My SAN storage will be used with XEN running CENTOS

very usual setup, but if you build your own san, you'll achieve much better performance and stability by directly plugging the disks in the xen host

I have more drives but they are a different make and size and I don't know if you can mix and match manufacturer types and sizes.

with zfs, you can fairly easily. i'm unsure about btrfs. with anything else, it will quickly bring you to the smallest disk size on all disk and the smallest performance as well.

It would also make me wonder how that would be possible since the controller should have optimized components/software?

raid0 and raid1 have little to no computational overhead.

then this is not ALWAYS true : for example writing to soft raid1 will put 2x more strain on the bus than writing to a hardware one because you'll have to send the data to both disks.  if the bus is a bottleneck, this is of concern.

in many cases, the card's caches find a better use by caching read-aheads and yet-to-be-committed-to-hardware disk writes than handling the raid.

zfs also does smart balancing between disks which raid cards do not.

What kind of software should I use to get the needed test results

dd is pretty good to start with
dd if=/dev/sdX of=/dev/null bs=16K count=1000000
will for example perform a read test
use iostat/sysstat (or better gstat if available on your system) to have a better view of what happens in terms of number of IOPs, transaction sizes and the likes

---

you're up to quite a great experience. feel free to post about it ; i think we'll all have things to gain as most of us don't have time to run such tests frequently or on that many setups.

---

completely off-topic but GPL does not protect the user and actually none of those licences do. GPL somehow protects the author and the opensource community by forcing any derivative work to be published as opensource under the GPL as well. (very quick resumee)

all opensource licenses i know of protect the author by making sure that if sombody looses their, data or break their brand new hardware by using/misusing the soft, they're not going to get sued (this software is not fit for any particular purpose...)
0
 
LVL 17

Assisted Solution

by:Gerald Connolly
Gerald Connolly earned 444 total points
ID: 39988200
As a Linux virgin, you might find things like Gluster a bridge too far at the moment.

Solaris with ZFS is meant to be a good combination, havent tried it myself

Yes a Software RAID-1 will be faster for Read at least than using Hardware RAID-1 on a Fake-RAID controller
0
 

Author Comment

by:KonfigurationKing
ID: 39990355
I have enough information to proceed with installation when all my parts arrive.I will proceed with the recommendations for ZFS initially and work from there.   Many thanks to all who participated in this thread.
0

Featured Post

Veeam Disaster Recovery in Microsoft Azure

Veeam PN for Microsoft Azure is a FREE solution designed to simplify and automate the setup of a DR site in Microsoft Azure using lightweight software-defined networking. It reduces the complexity of VPN deployments and is designed for businesses of ALL sizes.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article aims to explain the working of CircularLogArchiver. This tool was designed to solve the buildup of log file in cases where systems do not support circular logging or where circular logging is not enabled
When we purchase storage, we typically are advertised storage of 500GB, 1TB, 2TB and so on. However, when you actually install it into your computer, your 500GB HDD will actually show up as 465GB. Why? It has to do with the way people and computers…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
In this video, Percona Solutions Engineer Barrett Chambers discusses some of the basic syntax differences between MySQL and MongoDB. To learn more check out our webinar on MongoDB administration for MySQL DBA: https://www.percona.com/resources/we…
Suggested Courses
Course of the Month11 days, 15 hours left to enroll

636 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question