Solved

Hardware RAID

Posted on 2014-03-31
22
398 Views
Last Modified: 2016-02-12
If using a hardware raid solution what is the host computer doing? For example, If I am using older hardware with a hardware raid solution should I be less concerned with main processing power and memory?

All the box will be doing is acting as a SAN and using NIC teaming with 2, 4Gb PCIe Fibre Channel Host Bus Adapter.  The processor is a dual core 2.0 GHZ AMD with 2 GB of ram. Do I really need to upgrade this since I will be adding a HARDWARE raid controller? Will it be able to process the network speeds required with a teaming set up?
0
Comment
Question by:KonfigurationKing
  • 7
  • 5
  • 5
  • +2
22 Comments
 
LVL 20

Expert Comment

by:carlmd
Comment Utility
If you mean add a raid card to a system, it will make a difference in that you have added something to the system, but the "work" is done on the card to distribute the data across the RAID array. You are still writing data to a "disk". If the system is on the edge, it will matter. If the system had adequate memory, cpu etc,, it won't.

If you are referring to a totally external RAID array just connected to the bus (USB for example), it won't matter any more then servicing just an external disk. In this case the work is done externally and all you are doing if funneling the data.

Bottom line, if the system has the reserver capacity to receive and write the data you should be good in either case.
0
 
LVL 55

Expert Comment

by:andyalder
Comment Utility
Depends what SAN software you are using as to how much RAM it needs and depends on what controller you are using and how many disks you have to know how fast it will be.
0
 
LVL 55

Expert Comment

by:andyalder
Comment Utility
Depends what SAN software/OS you are using to know how much RAM/CPU resource you need and depends on what controller and disks you have to know how fast it is but you'll need a lot of hard disks to get anywhere near gigabit Ethernet speeds.
0
 

Author Comment

by:KonfigurationKing
Comment Utility
How do you know that it will take alot of harddisks to reach gigabit speeds? What if I take 6 harddrives with a maximum of 6GB a sec in a Raid 0 configuration? You are saying I would not need more than a 1gb ethernet?
0
 
LVL 26

Assisted Solution

by:skullnobrains
skullnobrains earned 334 total points
Comment Utility
even software raid would work if you stick to raid 0 or raid 1 without important performance impact on the processor

i concur that you will need LOTS of disks. a regular SATA disk can do about 100-120 Mo/s. SAS drives will reach about 200. some high level SSDs can do much more. the 6GB is the maximum burst speed of the SATA3 interface,  but no disk currently produce such a throughput. on linux and windows you most likely should divide these figures by 2.

then you may want to note that

- most raid1 implementations (including hardware) actually don't show a better performance throughput than a single disk

- the bus that you will plug the raid card in probably cannot achieve such speeds anyway. you'll have to split between several controllers which excludes hardware raid (or use a software raid of several hardware raids)

- the board you are using probably won't be able to cope with such a throughput even with DMA

- 2Gb of RAM is far from what would be reasonable if you expect such IO performance
0
 
LVL 16

Assisted Solution

by:Gerald Connolly
Gerald Connolly earned 111 total points
Comment Utility
@Brains, you use a lot of mixed nomenclature! Mo/s - ?, 6GB - 6Gbit/s, 2Gb - 2GBytes - Note its usual to use "B" for bytes and "b" for bits.

I agrre with you that although the current SAS standard is 6Gbits/s this is just the size of the pipe, the best Enterprise class disk only do 200ish MB/s and also 200 IOPS, but a SATA disk is typically down less than 100 for both of these numbers, so 3 * Enterprise Disks might stress a single SAS pipe you are looking at maybe 8 * SATA disks to do the same thing.

Back to the original question. Yes you are right if you are using a server to act as a SAN gateway it doesnt have to be particularly high spec, but it probably needs a decent amount of RAM and 2GB is not enough, and if it is going to be running multiple GbE links it will need some horsepower to drive them at any kind of speed.
0
 
LVL 26

Assisted Solution

by:skullnobrains
skullnobrains earned 334 total points
Comment Utility
my bad 6GB should be read 6Gb/s

i'm unsure about the 200 IOPS. from experiments with database servers, i usually ended up around 250 on freeBSD and for some reason, more like over a thousand on linux. I assume some caches were in the way (iostat probably hooked above hardware level) and/or the transactions may have been smaller since overall, equivalent db setups perform better on bsd. the disks will actually handle more transactions per unit of time but the throughput drops dramatically when the threshold is reached.

i'll maintain proper quality SATA disks usually do 100-120Mo/s (or MB ;). lower figures usually are produced by wrong partition alignment (which is about always the case on windows, and the case on most linux boxes i came across)

from my experience raid 0, and 10 achieve about theorical speed increases when properly configured, RAID 1 seldom does because of frequent crazily poor implementations. above 4-5 SATA disks, it becomes necessary to split the disks on multiple controllers. also note that most SATA disks achieve better throughput on the beginning of the drive, so partitioning only part of the drive is one way to achieve overall better speed.

some SAS controllers do better than SATA III. 12Gb/s is not uncommon.

---

if you expect 8Gb throughput, you'll need something like an I5, a PCI-X bus doing at least x8 but x16 would be better, 2-4 raid cards (possibly 1 or 2 builtin SATA controllers can replace a number of raid cards), 16Gb of RAM, and about 10-12 SATA disks.

obviously this would be foolish because a RAID 0 of 10 disks is a no-go, so you'll have to do RAID 10 (or a 0 1 0 or 1 0 1 mix), use 4 RAID cards ( can be more or less depending on the raid cards ), and about 20 SATA disks

i'd also add an SSD for the filesystem's log and possibly a read cache, and tweak parts of the OS because it's log files (for example) may well become a bottleneck

note that this is achievable for much less $$ than an equivalent SAN (i'd say at least 10x less), but it requires quite some work to setup. distributed filesystems (or data) on lots of machines may be an alternative worthy to consider depending on your application

btw, why the hell do you need such tremendous speeds ?
0
 
LVL 55

Expert Comment

by:andyalder
Comment Utility
You're doing a heck of a lot of work considering the questioner hasn't even said what SAN software they are using nor what RAID controller they have.
0
 

Author Comment

by:KonfigurationKing
Comment Utility
Here is the device being used for the RAID controller:   LSISAS 2108 8-port PCI-E SAS-2 controller

  8-port (8 internal) 6Gb/s per port
    Supports RAID 0, 1, 5, 6, 10, 50, 60
    On board hardware IO Processor
    512MB 800MHz DDR2 on-card cache
    Advanced array configuration and management utilities
    Automatically negotiates PCI-E link widths
    Power management support
    Supports 1.5, 3.0 and 6.0 Gb/s SAS and SATA data transfer rates
    Port independent auto-negotiation
    Supports SSP, SMP, STP and SATA protocols
    Optional Battery Backup Unit for cached data protection (BTR-0018L-0000-LSI) - operating temperature (Ambient) from 10C° - 45C°
    OS Support : Windows XP/2003/2008/Vista, RedHat Enterprise and SUSE Linux
0
 

Author Comment

by:KonfigurationKing
Comment Utility
@ skullnobrains

The reason I am setting up this lab is for experience. Why you ask so much speed? I have money to waste and frankly...BECAUSE ITS COOL!  :)

Right now I am supporting windows desktop only. Can I support enterprise environments if I have no experience ?  I hope to advance my career and become  like all of you experts and be the one answering NOT asking.

Scouring the internet trying to find information is challenging, because most articles assume the reader at a certain level of understanding.  A lot of the people here on EE have exceptional knowledge and get to work with server side IT systems. I want the benefit of that same experience, and be able to obtain mastery level at various open source technologies. I know of no way to gain this expertise other than to implement an enterprise like environment.

I am starting to realize the power of the free software movement. Open Source technology is a powerful tool that liberates a company or person from vendor lock in.
0
 
LVL 16

Expert Comment

by:Gerald Connolly
Comment Utility
Your original post talked about Fibre Channnel HBA's are they now not in the picture, and is it only SAS that is important now?
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 26

Accepted Solution

by:
skullnobrains earned 334 total points
Comment Utility
 8-port (8 internal) 6Gb/s per port

you definitely need to test this for yourself : you'll most likely find out that transfer rates drop way before

start in JBOD (no raid)
saturate one drive with a single read thread and mesure the speed
saturate a second drive
... a third

at some point speeds are going to drop, either because you hit the bus's limit or the card's

it is interesting if you have time to experiment with multiple threads as well

obviously repeat with write threads. if you see a difference between read and write, change the write-back property

Why you ask so much speed? I have money to waste and frankly...BECAUSE ITS COOL!  :)

honestly, you should start trying to saturate a 1GB link. already hard enough, and much better than what most commercial SANs do regardless what they say. this is feasible with a single machine for about 4000 bucks and will already bring you the satisfaction to do much better than things HP or adaptec sell above 50K

you'll have a hard time to find an application to even test it anyway ;)

Right now I am supporting windows desktop only. Can I support enterprise environments if I have no experience ?

if you expect to run such a setup on windows, i don't think you have a chance, but i'd be more than happy to be proven wrong

if you want to support other things, it's all about learning and opensource software is well documented. given enough time, you will definitely be able to do lots of things

for this specific project, i'd recommend using zfs (filesystem and raid management) running on solaris of freebsd

I know of no way to gain this expertise other than to implement an enterprise like environment.

yes. books help, but they will never replace experience. then you'll probably need to actually provide services at some point rather than just a test lab

I am starting to realize the power of the free software movement. Open Source technology is a powerful tool that liberates a company or person from vendor lock in.

hmmm, correct but not completely. some opensource companies actually sell and license their software (redhat, novell...). GPL based opensource software is far from being "freely redistributable" as the licence says because they are quite a few conditions to meet. some companies/individuals have policies that free their customers, some don't, even in the opensource world.

if you want to be free, rather think BSD, Apache, CDDL (previously used by SUN), ...

note that not everybody agrees on such matters even in the opensource community. read about the openSSL exception of the GPL for example if you wan't to grab an idea. (wear a flame-proof suit ;)
0
 
LVL 16

Expert Comment

by:Gerald Connolly
Comment Utility
Its not that hard to saturate a 1GbE link (100MB/s), but quite a lot harder to saturate a 10GbE link (1GB/s)
0
 
LVL 26

Assisted Solution

by:skullnobrains
skullnobrains earned 334 total points
Comment Utility
1GB is what the author wanted. i misread the question the second time when i stated "to start with", thanks for pointing out.

actually fairly easy using a bunch of 5-6 SAS drives with no redundancy providing the card can cope with that (which i hardly believe). much harder with redundancy, much harder with SATA drives (see above)... but feasible on reasonable hardware nevertheless

note that for a production system, 4 SAS drives in a raid 10 array would be a good match for that network speed : it won't be able to saturate it on the long term but will provide bursts that will.

i think it is reasonable to start with that motherboard and card. most likely at least additional RAM will be necessary. additional horsepower would depend on the actual use but that proc may be enough for at least a high performance san using iscsi. with no redundancy and a bunch of 10-12 SATA drives, the author should be able to achieve his goal and learn about quite a few pitfalls/limitations/tricks along the way.

adding redundancy should produce about half the write throughput and possibly the same read throughput which will already saturate the network most of the time and produce a production-level SAN
0
 
LVL 55

Assisted Solution

by:andyalder
andyalder earned 55 total points
Comment Utility
The card has a dual-core RoC so it's not going to use any server CPU/RAM except for the drivers, still don't know what SAN software is going to be used.

The card can do 400,000 IOPS using SSDs but if the author is using 7.2K SATAs with a 4K random workload they'll only get about 400KB per disk so that would need 25 of them using RAID0 to get 100MB/s giving gigabit speed. If they're streaming it's a different kettle of fish since there aren't the random seeks.
0
 
LVL 26

Assisted Solution

by:skullnobrains
skullnobrains earned 334 total points
Comment Utility
where do the 400KB per disk figure comes from ?

as far as SAN software, i assume the author has not elected one yet. i'd suggest zfs. any other ideas ?

as far as CPU goes, i'm concerned with global throughput (disk-to-network) and not RAID operations. any modern computer can deal with RAID 0 1 and combinations easily without the help of the card. from my experience software raids are often better than hardware.

note : i think it is quite premature to worry about random workload before you get optimal performance with linear operations, and i also think it is better to start benchmarking locally before doing any bench over a network
0
 
LVL 55

Expert Comment

by:andyalder
Comment Utility
4K * 100 IOPS (actually a bit generous). Since they haven't said what they are going to use this "SAN" for it's safest to suspect the worst which is OLTP database.
0
 
LVL 26

Expert Comment

by:skullnobrains
Comment Utility
that is less than what i'm used to observe on a single SATA drive including with an sql database. the IO transfer size does not have to match the sector size. additionally you hardly ever use 4K blocks for rdbms
0
 

Author Comment

by:KonfigurationKing
Comment Utility
Question by Gerald Connolly: Your original post talked about Fibre Channnel HBA's are they now not in the picture, and is it only SAS that is important now?

Answer: I was able to acquire 2 Dell FX775 Poweredge R805 Dual Port 10GBe PCIE. I was able to find the name of the chipset manufacturer and will have access to the proper drivers to load in GNU/LINUX (http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/SearchByProduct.aspx?ProductCategory=325&Product=1094&Os=65&ProductName=NXB-10GCX4&OSname=Linux%20Red%20Hat%20%2864-bit%29). I will post another question regarding network capabilities of the card.  I will only be using SATA DISKS so, it seems from the comments that the bottle neck will defiantly be on the machine bus.


as far as SAN software, i assume the author has not elected one yet. i'd suggest zfs. any other ideas ?

@skullnobrains and @andyalder /  andyalder too
I thought ZFS was just a filesystem like ext3/ext4 or NTFS!  Do you have a link to the recommendation you made?
I was thinking about Glusterfs (http://www.gluster.org/) on CENTOS but I am open to your suggestions and others. I have a little Fedora Linux experience and I use it as my primary desktop. I have never used Unix before and I don't know if my NIC card drivers can be ported to other *nix based systems. .My SAN storage will be used with XEN running CENTOS or other nix based system recommended...


....With no redundancy and a bunch of 10-12 SATA drives, the author should be able to achieve his goal and learn about quite a few pitfalls/limitations/tricks along the way.....

I will eventually purchase more drives for the array but for now it will only comprise of 4 SATA drives. I have more drives but they are a different make and size and I don't know if you can mix and match manufacturer types and sizes.


...from my experience software raids are often better than hardware....

Software raids better than hardware raids!?!!?! WOW that is impressive! Makes me want to test that out in the lab. It would also make me wonder how that would be possible since the controller should have optimized components/software?


...optimal performance with linear operations, and i also think it is better to start benchmarking locally before doing any bench over a network...

 *your great idea* I will also do as you recommend and get performance results locally before network bench. What kind of software should I use to get the needed test results?


if you expect to run such a setup on windows, i don't think you have a chance, but i'd be more than happy to be proven wrong


Running this setup on windows ? I think that would kill me a little on the inside :) Though I understand it is not entirely windows fault but at work because of the massive security (mcafee) / group policies pushed from the enterprise the machines are slow and not very efficient . I want to advance my career by gaining experience supporting heterogeneous environments with mostly *nix based tech.


GPL based opensource software is far from being "freely redistributable"

Darn you ruined it! I was just about to put up a statue of Richard Stallman in my back yard! Seriously though, that really surprised me I did not know that. I always assumed that GPL based software had protections built in to protect the user. Of coarse, I have not read the various GPLs and even if did I probably would not understand them. So, instead of hiring a lawyer to interpret the license I will take your word for it :)

Honestly, I really don't know what I don't know! Like skull was saying this is going to be an experience to learn more. As always I appreciate all the help...
0
 
LVL 26

Assisted Solution

by:skullnobrains
skullnobrains earned 334 total points
Comment Utility
I will post another question regarding network capabilities of the card.  I will only be using SATA DISKS so, it seems from the comments that the bottle neck will defiantly be on the machine bus.

don't : you need to test this yourself. this is even more important if you're doing this for educational reasons. use dd to generate reads and writes on the raw disks to start with. then play with options in the card (write-back will change many things). play changing the number of disks, play changing the number of threads, then play with a filesystem, and random reads/writes. what you'll learn is invaluable.

note that cards capabilities are usually far from what the vendor says

also note that RAID setups which are not first tested in this way, and configured to reach about theorical speeds (5% tolerancy tops) with a simple dd tests are most of the time heavily misconfigured. this (unfortunately) applies to most setups you'll see in real life.

I thought ZFS was just a filesystem like ext3/ext4 or NTFS!  Do you have a link to the recommendation you made?

no : zfs combines a volume manager and a filesystem. it supports raid levels similar to the usual ones with a few notable differences : raid0 is actually not partitionned so it is soft load sharing, and raidz is similar to raid 3 or 5 without the write hole.

adminitration is easy, and disks can be added or replaced on-the-fly without ever waiting for the raid to be built, rebuilt...

i have no specific link to provide because you'll find lots of docs on the internet, but i'll be happy to answer any question.

on linux, it was only supported through fuse last time i checked. if the kernel module is mature, it may be worth wile, if not i'm unsure about using it on linux

I was thinking about Glusterfs

don't : glusterfs is a distributed filesystem, suitable for hosting a single fs distributed on many computers. it is unrelated with what you are doing. btrfs might be an idea as it also includes a volume manager. otherwise, use raid hard or soft and any regular filesystem (ext, xfs, reiser) on top of that.

I don't know if my NIC card drivers can be ported to other *nix based systems

a general answer would be a linux driver will work on any linux distribution using a similar kernel version, and porting drivers will otherwise require lots of work. then it is easier to port a freebsd driver to dragonfly than to port an AIX driver to netbsd (examples pulled quite randomly from the top of my head)

My SAN storage will be used with XEN running CENTOS

very usual setup, but if you build your own san, you'll achieve much better performance and stability by directly plugging the disks in the xen host

I have more drives but they are a different make and size and I don't know if you can mix and match manufacturer types and sizes.

with zfs, you can fairly easily. i'm unsure about btrfs. with anything else, it will quickly bring you to the smallest disk size on all disk and the smallest performance as well.

It would also make me wonder how that would be possible since the controller should have optimized components/software?

raid0 and raid1 have little to no computational overhead.

then this is not ALWAYS true : for example writing to soft raid1 will put 2x more strain on the bus than writing to a hardware one because you'll have to send the data to both disks.  if the bus is a bottleneck, this is of concern.

in many cases, the card's caches find a better use by caching read-aheads and yet-to-be-committed-to-hardware disk writes than handling the raid.

zfs also does smart balancing between disks which raid cards do not.

What kind of software should I use to get the needed test results

dd is pretty good to start with
dd if=/dev/sdX of=/dev/null bs=16K count=1000000
will for example perform a read test
use iostat/sysstat (or better gstat if available on your system) to have a better view of what happens in terms of number of IOPs, transaction sizes and the likes

---

you're up to quite a great experience. feel free to post about it ; i think we'll all have things to gain as most of us don't have time to run such tests frequently or on that many setups.

---

completely off-topic but GPL does not protect the user and actually none of those licences do. GPL somehow protects the author and the opensource community by forcing any derivative work to be published as opensource under the GPL as well. (very quick resumee)

all opensource licenses i know of protect the author by making sure that if sombody looses their, data or break their brand new hardware by using/misusing the soft, they're not going to get sued (this software is not fit for any particular purpose...)
0
 
LVL 16

Assisted Solution

by:Gerald Connolly
Gerald Connolly earned 111 total points
Comment Utility
As a Linux virgin, you might find things like Gluster a bridge too far at the moment.

Solaris with ZFS is meant to be a good combination, havent tried it myself

Yes a Software RAID-1 will be faster for Read at least than using Hardware RAID-1 on a Fake-RAID controller
0
 

Author Comment

by:KonfigurationKing
Comment Utility
I have enough information to proceed with installation when all my parts arrive.I will proceed with the recommendations for ZFS initially and work from there.   Many thanks to all who participated in this thread.
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

this article is a guided solution for most of the common server issues in server hardware tasks we are facing in our routine job works. the topics in the following article covered are, 1) dell hardware raidlevel (Perc) 2) adding HDD 3) how t…
Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

7 Experts available now in Live!

Get 1:1 Help Now