Question

Building a iSCSI-SAN?

Asked by: alexciric

Hello experts,

we are about to getting deeper into virtualization with Xenserver.
It is our goal to virtualize windows terminalservers, win64 sql-file-servers and also exchange and scalix servers, which are at the moment already partly virtualized with for example vmware server 2.x.
At the moment we are talking about ~30 - 50 Users wich are organized in about 12 VMs in 4 seperated subnets.

We want to do the following:
- migrate the existing VMs to Xenserver for better performance and management.
- use xenservers livemotion to get better utilization of the vmhosts.
- take away the storage from the individual VMserver to a centralized storagehost

For this we think we need a ultrafast and reliable SAN to get the storage away from local storage into the network. Unfortunately we dont have the money to buy fancy expensive *enterprise* SAN-Solutions.... :-(. So we have to do it on ourselfs.

My plan is to setup a big server with multiple bondaged gbit-interfaces and to export the storage via iscsi to the xenservers (a couple of Quadcore Xeons).

The question is:
- is this the right way to achieve very high storage performance for all VMs?
- in the storage server: should i use SAS-drives or can i have similar performance with SATA-Raids? (already have 12 x73gb Savvio SAS drives for local storage. Move them into the SAN???)
- which raid-level would you prefer? 5?6?10? else?
- can i build this config easy with centos 5.4?
- software raid or is it a must to buy HW-raid-controllers?
- how many Disks do i have to install to get a really impressing drive performance for all the VMs together?
- how many gbit interfaces (or even 10gbit) should i bondage together to achieve the requested performance level?

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2009-11-03 at 00:13:07ID24866357
Tags

SAN

,

ISCSI

,

Linux

,

Virtualization

,

Xenserver

,

performance

Topics

Red Hat Linux

,

XenServer

,

Storage Technology

Participating Experts
2
Points
500
Comments
10

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. iSCSI on VM?
    I have a Dell iSCSI SAN that is hosted by a physical server. The physical server is having hardware problems and I was thinking of replacing it with a ESX VM (hosted on another SAN - NOT the iSCSI SAN). Is this possible? In other words, the VM will host the iSCSI SAN with ...
  2. VMware iSCSI SAN and File Sharing
    Hello, I am in the process of putting together a proposal for VMWare Infrastructure using an ISCSI SAN for storage. The SAN will consist of AIO1200r SAS (NAS/iSCSI) or Dell MD3000i (iSCSI) or similar product. My problem is this I (will) have multiple Windows VM's running lo...
  3. VMWare VCB from iscsi san very slow
    Having an issue backing up full vms in the following environment: ESX 3.5,VC2.5 Backup Exec 11d Proxy Server- Dell workstation with W2k3 iSCSI San- Dell CX3-10 Full vms copy very slowly to the proxy when using the 'san' method- a 10 GB vm taking over 2 hours. When I run the ...
  4. iSCSI SAN Recommendation For Hyper-V Server
    Hello all. My organization has a couple of Hyper-V servers, one of which is going to be replaced relatively soon (hardware refresh cycle). Instead of having all these independent servers with their own isolated RAID arrays (RAID 10), we were looking into putting in an iSCSI ...
  5. iSCSI SAN on a Budget
    We have been looking at implementing a iSCSI SAN for our VMWARE production environment to support HA/FT and Vmotion etc. I have been down the road of looking at Netapp, Lefthand and Equallogic all of which have their pros and cons but also fairly heafty price tags per GB of ...
  6. iSCSI SAN options and VMWare
    I am looking to consolidate several physical servers into a virtualized environment. I have serveral questions relating to the virtual machines and the storage solution. I do not have a problem with a solution for now that is not as flexible and may need some upkeep on my end...

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: BigSchmuhPosted on 2009-11-03 at 03:27:05ID: 25727813

Savvio 2,5" HDD comes in 2 flavors :

  • The 10K delivers up an avg of 140 iops 
  • The 15K delivers up an avg of 180 iops 
  • A SATA 7200 rpm would deliver about an avg of 90 iops 
  • ...and one single Intel X25-E SSD may deliver 2300 iops 

Your usage looks like relying more on IOPS than on sequential throughput:

  • RAID 10 delivers more iops than RAID 5 (which delivers more than RAID 6) 
  • Hardware raid HBA used to deliver more iops than software raid...and are mandatory for parity raid (a write back cache requires a battery to be reliable) 
  • A 32 io queue length per HDD would reach 400 iops per 15K HDD, 400x 32KB (stripe size) x 12 HDD = 150MB/s ... That is 30% more than what a 1 Gbe NIC can deliver... 

I would buy 2U "generic storage" $11k white boxes covered by good onsite and online services

  • 2U boxes using 24x 2,5" hot swap (like the Supermicro SC216) or a classic 12x 3,5" hot swap 
  • dual cpu  
  • 16-18 DIMM of 4GB (8GB is still too expensive) each allowing for a >60GB read cache 
  • on site next day hardware warranty / online 24-7 software support (OpenFiler has it) 
  • 2x 12 ports hardware raid HBA offering up to 2x 4GB write back cache backed by battery and remote management (only 1x 12 ports raid HBA for the 3,5" 2U rack) 
  • iSCSI target software (OpenFiler or IET or tgt) : OpenFiler offers a 24/7 online support ($2500 a year per node) where IET/tgt relies on less reliable opensource forums but already has some CentOs user 
  • 4 ports 1Gbe or 2 ports 10Gbe or Infiniband 40Gb/s 
  • shipping 

and some drives between the below list

  • 12x SATA 3,5" (only 12x 3,5" case) 2TB in RAID 10 for 12TB (or RAID 60 for 16TB) and sequential io usage pattern (Backups...) (+$3.5 = $1.2 per r10 GB or $0.9 per r60 GB) 
  • 24x SATA 500GB in RAID 10 for 6TB capacity and 2200 iops (+$5k = $2.7 per GB
  • 24x OCZ Vertex 250GB SSD in RAID 60 for 5TB capacity and 12000 iops (+$17k = $5.6 per GB
  • 24x SSD Intel X25-M 160GB in RAID 60 for 3.2TB capacity and 32000 iops (+$11k = $6.9 per GB) 

Please note that whatever solutions you choose (including SAN), you have to buy dedicated interconnection switches...this is where you have to take care as 1Gbe / 10Gbe / InfiniBand all differ in features and prices.

I would start with the 2U 12x SATA 2TB for live backups and 2U 12x SATA 500GB + your 12x Savvio to confirm that the additional 3TB ($2.7 per r10 GB) used with 60GB read cache and 8GB write cache are more than enough for your usage...It would end in very impressive performance by getting 12x Vertex (in raid 6) instead of the 12x SATA 500GB (in raid 10).

 

by: andyalderPosted on 2009-11-03 at 09:08:21ID: 25731086

How many servers in total? If 4 or less you might be better off using SAS attached rather than iSCSI although that may cause a migration problem.

I would recommend against using big SATA disks since the chance of unrecoverable read errors is too high. I'd also recommend against using a white box and building it yourself since you're not going to get any support. Can you really afford a single server handling your storage? What if the motherboard fails? All your hosts die plus the VMs on them. Far safer to have a dedicated highly available storage box with dual controllers. Dell, IBM and HP all do low end storage arrays and they're not that much more than a storage server, in fact they're cheaper if you want high availability because for that you would need two storage servers meaning twice the disk cost.

 

by: BigSchmuhPosted on 2009-11-03 at 11:41:50ID: 25732676

Regarding "big SATA disks", I recommend "Enterprise class" drives that are still way cheaper than all SAS HDD but can benefit of a UBE/BER of 1 per 10E15 (Means a 0.88% probability of 1 unreadable sector per 1TB read)
As an example, one can check that the Seagate Constellation 500GB 2.5 HDD exists in both SAS 2.0 and SATA 2.0 interfaces and has a 1 per 10E15 UBE.

Regarding "white boxes" sorry my french, I was not thinking of building it yourself (cf "on site next day warranty" above)...but to look for other less known brands.
An example at icc-usa (using the SuperMicro SC216 24x 2.5" hot swap 2U rack)
==> If you have the money to pay for the brand : GO for Dell, HP, IBM or EMC of course...

My current experiences regarding high availability is :

  • Too complex configurations have more unplanned time than others and the unavailability takes longuer to get fixed 
  • The key points are monitoring tools and production guys competencies 
  • Spare servers and storage means every production guy nows how to efficiently execute the disaster recovery plan 
  • Backup on disk, backup offsite 

@andyalder : if your "dedicated highly available storage server" has its "motherboards fails", you are having the same problem that you were complaining about above...but I have a spare ready, do you ? The point is that dual controllers means dual path drives whose are way too expensive for their low availability benefit. Most of the time, some RAIDed drives or a redunded PSU fails, with no serious impact...this is a matter of risk vs investment and not one of my "rules of thumb"

 

by: andyalderPosted on 2009-11-03 at 14:31:57ID: 25734563

I did not mention a "dedicated highly available storage server" since that implies something that has a motherboard. I said "dedicated highly available storage box".

There is no such thing as a motherboard in dual controller SAN boxes, just two controllers. I suppose you could say they have *two* motherboards in them if you want to call the controller a "motherboard with in-built interfaces and interconnect for heartbeat and cache mirroring" There's no SPOF in such boxes unless you count the passive interconnect between controllers.

Why all the bold rich text stuff? It just makes what you say unreadable.

 

by: alexciricPosted on 2009-11-04 at 01:50:45ID: 25737738

Hello again.
Thank you very much for you really very interesting information.
As i also read a lot in the last days i learned a lot about the howtos of SANs and the interaction with hypervisors...
With all this info i came to the conclusion, that AT THE MOMENT a full virtualized storage solution is in every case way beyond our financial possibilities (also with the fact in mind, that such a storage has to be fully redundant).
In the meantime we will stick with Xenserver and local Savvio SAS raid 10 installations.
With this config, we are able to collect the right number of drives in advance and then make a switch to SAN somewhere in the future with these disks.
THEN we will build two Quad-Core Servers with each 96gig of RAM, a 12port Raid60 Controller and 10gbase T Ethernetcontrollers. 1 Conroller leading to the Switch for the VMservers and one controller leading to the second stroage for redundancy.
With this config it should to keep the whole storage mirrored in realtime and to have the right performance for the VMservers.
At last there is only the question how many machines and users can run with such a storage and how much users and machines can run parallel on one vmserver?
Do you have any clue on this? (for simplicity we assume that 10 users have one sql/fileserver one Terminalserver and one Scalix-Server)

I thank you two guys again very much as you helped me to get my understanding of "enterprise" virtualization to a higher level :-D

 

by: alexciricPosted on 2009-11-04 at 02:27:45ID: 25737923

I give a little addition of my hardware plans:

Storage:
Quadcore Xeon with 96Gig ram, 12 SAS Savvio with 300Gig, Adaptec 51245 SAS Controller, 2x 10GbaseT Controller

mirrored to a second one with the same config.

three VMservers :
dual Quadcore Xeons with 96Gig RAM (expandable up to 144gig) and Xenserver, no local storage. 1 10Gbase T Controller

Each FileSQLserver is either Win2008X64 with 8Gig RAM or (a few old installations) Win2003 std. 32bit with 4Gig RAM.
Terminalservers are mostly Win2008 X64 with 6 Gig RAM (10 -20Users) or Win2003 32bit with 4Gig (max 10 users)
The Scalix Servers are Centos VMs with 4 (<15 Users) or 6-8gig RAM (>15/20 Users)

All users are commonly using applications for financial- tax- and payrollaccounting (MS SQL Backend), Word, Excel, Outlook.

 

by: BigSchmuhPosted on 2009-11-04 at 05:56:01ID: 25739298

@andyalder: English is not my native language...I felt agressed reading some of your arguments...I am sorry about that

Dual controllers with embedded interconnection interfaces, drives and redunded PSU : I agree that those should render a reliable storage system with no SPOF....but can you ask DELL why they only offer their MD3000 (non iSCSI) to their PowerEdge servers only...
Those "do-all-the-storage-task" controllers are more complex than a classic raid hba and not in a high volume market so they may not be as reliable as marketed. Any serious study about this reliability ? If not, I can tell you the story of a large french retailer that was using a SAN and had TWO major (1h to 3h) intraday failures last year due to the SAN FC switch firmware.

Regarding the SAS Savvio 10K 300GB and your other hw choices, I would recommend :

  • to mix some SATA HDD and SSD to get more capacity and more IOPS at a same or lower price - for example, getting 4x Ocz Vertex 250GB + 8x Seagate Constellation 500GB should offer both capacity (500GB r10 + 2TB r10) and IOPS (1600iops + 700iops)...definitely more than 12x SAS 10K 300GB (1.8TB r10 1400iops) capability 
  • to expect not being able to reuse your SAS drives in a future SAN as they may not comply with it - for example, many SAN requires drives with dual port SAS or FC interfaces where $500 per a 300GB 10K drive is a "very good" price 
  • to buy TWO raid hba instead of one ... and please, no raid 5/6/50/60 without a write back cache backed by a battery 

 

by: andyalderPosted on 2009-11-04 at 12:06:00ID: 25743348

@BigSchmuh, sorrt if I came over as agressive, I just don't think a mix of SATA and SSD disks will work in many environments unless there is some automatic tiering software involved, It's just too much work to administer if not.

I can see how a firmware problem could knock out one fibre channel fabric, strange that it would knock both out at the same time though, presumably they did have redundant fabric just like alexciric is presumably going to buy two switches for his iSCSI SAN to run on and not connect them together.

 

by: alexciricPosted on 2009-11-05 at 04:52:25ID: 25748860

Thank you guys for your input.
Now i am thinking of the following config:

SAN Hardware: (2x )
12x SAS drives (SSD i think, i dont want to mix drive models for higher compatibilit/reliability/managementcapabilities...) in a Supermicro Sc216 Case with an Adaptec 51205 Controller. Then 96gig ram, Quadcore CPU, Openfiler. 2x Dualport 10Gbase T Controller. (1 to the VMs, the 2nd to the second SAN). The 10gbt ports will be trunked to have 20GbaseT.

Also we will get two 10Gbase T Switches to avoid the redundancy traffic on the connection to the VMservers.

For the VMservers working with Xenserver i think about dual Quadcore machines with plenty of ram and also 10gbaset interfaces. (dual? trunked???)

Two of these devices should be able to  mirror the data in realtime and tto have enough performance to handle around 150 Users in several VMs.  (We are thinking of ~20-30 VMs on 4-6  dual Quadcore machines)

With this config it should be possible to reuse existing drives for the inital phase with low utilization and to scale to larger dimensions in the future.

What do you think?

Also i am a bit confused as an acquaintance gave said that virtual Terminalservers lose around 40% performance compared to native machines??? I cannot believe this. What is the point with this statement?

 

by: BigSchmuhPosted on 2009-11-05 at 06:46:06ID: 25749938

Some notes :

  • For price reason, I would go with 18x 4GB = 72GB instead of a 8GB based server - I priced (at icc-usa) the 72GB server at $7k (without drives), the same one with 96GB is $14k ! 
  • For price AND performance reasons, I would go with a mix of 500GB 7200rpm (SAS2 $270 or SATA2 $230) HDD and some SSD Intel X25-M G2 160GB ($500) instead of SAS2 300GB $330 10krpm drives 
  • dual E5502 +72GB +dual 10Gbe +1 raid hba +6x X25-M 160GB +6x 500GB SATA2 = $11k 
  • Using a faster greener cpu as dual L5520 would add $1k 
  • All drives above have a UBE/BER of 1 per 10E15 

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...