[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now


What are the Pros and Cons to one large iSCSI Target vs. smaller multiple iSCSI Targets?

Posted on 2011-03-09
Medium Priority
Last Modified: 2013-11-14
Very new to setting up a SAN and would like to know given the environment below. What are the Pros and Cons to one large iSCSI Target vs. smaller multiple iSCSI Targets?

We have 4 servers boxes [10 Gbe network]:
- (2) Windows Server 2008 R2 Standard with StarWind iSCSI (5TB each in RAID 10 config)
- (2) Windows Server 2008 R2 Datacenter with Hyper-V

(8) Hyper-V servers:
- (2) SQL 2005
- (1) Exchange Server
- (3) Web servers
- (2) Development servers
- (1) File Server

Question by:dinfopdfc
  • 4
  • 3
  • 3
  • +2
LVL 47

Expert Comment

ID: 35091991
Not all operating systems can deal with an iSCSI target > 2.09TB
you can "tune" RAID controllers to have different performance characteristics by manipulating the RAID & filesystem parameters, so you can make some targets faster for various I/O loads (like read vs. write and  sequential vs random I/O)

Multiple targets let you configure the RAID so I/O to any single target won't affect every disk you have.
LVL 20

Expert Comment

by:Svet Paperov
ID: 35095179
It depends.
•      One iSCSI Target can be attached to only one server, so you need at least 2 targets for your Hyper-V servers
•      If you are planning to use pass-through disks in the virtual machines – that are another targets (1 target = 1 pass-through disk)
•      Beside that there is no real advantages of having multiple targets

On the other side, if you decide on multiple iSCSI targets, 1 for each VM, this could make the space management more difficult. Say, for example, you have a single VHD on a 100GB iSCSI target. If, at some point you need to extend that VHD, you will have to go through long process of shutting down the VM, exporting the VHD, destroying the iSCSI target and creating a new one, bigger to host the extended VHD.

However, we should not forget that StarWind targets are actually image files on the host (again, I think, depending on the license of StarWind you are purchased). So having a huge image file of several TB makes me uncomfortable (this is my opinion: I love StarWind for the fact that I could turn an old WS 2003 server into a iSCSI target but I would prefer a physical iSCSI for my production data, something as Dell MD3000i – there are too much layers of virtualization with StarWind).  

So, it is up to the type and the size of data on each particular VM and up to your level of confidence in StarWind iSCSI technology and implementation.

With this particular setup I would go with pass-through disks for Exchange and File servers (data only), fixed size VHD for SQL and dynamic VHDs for all system drives and the rest of the servers. So, 5x targets of 1TB per StarWind image would be a balanced solution.

I am not quite sure but I think there is a StarWind license that allows you to attach physical disks as iSCSI target. If this is still the case, and you didn’t cut your RAID array yet I would go with 4 targets: 2 for pass-through disks and 2 for the rest of VHD.

Finally, I think, the deployment engineers in StarWind can give the best advice.

Conclusion: it depends.
LVL 42

Expert Comment

by:Paul Solovyovsky
ID: 35095395
The biggest issue with 1 target vs many is multipathing.  If you have lets say a Cisco switch and you configure etherchannel or link aggregation on the SAN and give it 2 network adapters in a virtual interface.  From that point you configure a single target with a single IP..if that is done that you have one IP communicating to another IP and thus a single session is formed.  Because of this the session can't go over a single physical nic on a multiport link aggregated interface.

Now..if you have multipe targets creating multiple sessions you can then use all the ports in a link aggregated link since sessions will redistrubute on different physical nics connecting to the SAN.

I think this is biggest PRO for multiple IPs.  The downside is management.
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.


Author Comment

ID: 35099895
Are there any reliability issues with Starwind? just ask because of your comments.

Also outside of speed what other pros are there on Pass-Through disk vs. Fixed?
LVL 42

Expert Comment

ID: 35101114
I don't think that volume size > 2 TB is a problem for you because you are using Microsoft Hyper-V. The CSV volume will need to be a GPT disk, but that isn't a problem.

Since you are using StarWind on top of a single RAID 10, you can't make some LUNs faster than others. Also, since every LUN is striped across all of your physical disks, activity on any LUN will affect all LUNs.

You should be using CSV with Hyper-V, so you can do failover and Live Migration. You can put all VMs on a single volume.

With Hyper-V R2, the performance between dynamic and fixed VHD pretty much went away, and I actually use dynamic VHD for everything other than dedicated VHD for the pagefile.

For SQL and file server data, you can use passthrough, dynamic VHD, fixed VHD, or create a separate LUN and use the iSCSI initiator inside the VMs to connect. I actually use the iSCSI initiator inside the VM to connect. The advantage is that you get all of the features that StarWind provides such as snapshots, clones, online volume expansion, etc. (I am not a StarWind user, so I don't actually know everything it does). This is some additional overhead to doing it this way, but it is the most flexible, and if wanted the best performance you wouldn't be running StarWind anyway.

I would make your initial LUN for your Hyper-V CSV as big you you need it to be initially, with some extra space but don't allocate everything to it. You will need space to be able to create additional LUNs going forward. You should be able to grow a LUN over time and without downtime, so there is no reason to assign something like 2 TB to your first volume. Maybe you start at 100-200 GB and go from there. There are days when I have expanded a LUN on my EqualLogic SAN 5 times in a day. You don't want to have to be doing that, but you should be able to.
LVL 20

Expert Comment

by:Svet Paperov
ID: 35102294
Regarding my comment on StarWind:
I am not aware of any reliability issue with StarWind but I don’t have very much experience with it as I am using the last free version that was available in 2010 on an old Windows Server 2003 to host some virtual machines for testing purposes (only to see how StarWind performs).  The only thing that bothers me is that its iSCSI targets are actually huge image files (2x 200GB in my case) and if something goes wrong with the StarWind host or with those files I will be stuck with a very long restore. Again, if you have the enterprise license of StarWind you could attach a physical volume instead of using img files. I doubt it, but may be in that case you could have similar output as with a hardware iSCSI solution.

Regarding pass-through disks:
kevinhsied already gave a good answer on the question fixed or dynamically expanding disks.
Just to add something on pass-through vs. virtual disks: again, for me, it’s the same reason – I don’t like using huge VHD files, instead I prefer the pass-through (650GB in my case). Actually, there are some drawbacks with the pass-through disks: you cannot back them up from the Hyper-V host since it doesn’t see them (you need a backup solution within the guest), and you cannot take a snapshot (but this is not a problem since the snapshots are not good idea in production environment, except for development purposes).

The only good advice that I can give you is: do a really good planning because you will be stuck with your decisions for quite long.
LVL 42

Expert Comment

ID: 35116384
Actually, if you are using smaller volumes such that you have free space to be able to create more volumes and to expand existing ones, I say that you will have the flexibility to change what you are doing over time. You may be doing some data migration, but you won't be stuck with what you have.
LVL 47

Expert Comment

ID: 35116635
"Since you are using StarWind on top of a single RAID 10, you can't make some LUNs faster than others. Also, since every LUN is striped across all of your physical disks, activity on any LUN will affect all LUNs."

True, but only to a limited extent.    Throughput is mutually exclusive with I/Os per second. If you are using databases, then 2 x RAID1s will provide TWICE the I/Os per second then 1 x RAID10, provided the arrays are built for 64KB chunk sizes, which is native to SQL Server.

Not only that, but iSCSI over ethernet still uses CSMA/CD.   The larger the packet size, the greater the probability of collisions & retries that kill performance.  Jumbo frames become an absolute necessity, and unless you are using fibre optic cabling, point-to-point, or have unbelievable quality CAT6, then you will not get full efficiency, at least not in this universe.

So first lesson -- throughput is mutually exclusive with IOPS.  RAID levels all have inherent strengths and weaknesses depending on the type of I/O.

If this was a direct-attach interface then it would be a completely different thing.    Also  what has not been mentioned is the need for a proper ethernet controller, one which offloads processing.  You don't want every single byte causing an interrupt.  If you look at block diagrams of motherboards, a fair number of them run the networking through the same bus as the primary disk interface.  You could very well get to a point where they compete for bandwidth.

Lesson 2 -- Consider the dynamics of the protocol.   You are not doing block-oriented I/O, so don't use their performance characteristics beyond the machine that is physically connected to the disks.

Lesson 3 -- Ethernet is inefficient, controllers, switches, and cabling is a real-world factor.  Choose unwisely and you could have SSDs running on the server, but you could end up with performance as if you had a USB-attached disk drive by the time the data gets to the client computers.
LVL 20

Expert Comment

by:Svet Paperov
ID: 35118224
kevinhsieh:"Actually, if you are using smaller volumes such that you have free space to be able to create more volumes and to expand existing ones, I say that you will have the flexibility to change what you are doing over time. You may be doing some data migration, but you won't be stuck with what you have."

True, except for … how small? With the size of a single VHD – not very practical – we are loosing the benefits of dynamically expanding VHD.  

To clarify a little bit the discussion I have some questions for the author:
•      Which version of StarWind do you have?
•      Are your using mirroring with StarWind (you could do it with the Enterprise version)?
•      Did you already partition your physical RAID 10 array?
•      Does StarWind allow you to attach physical disks (SCSI volumes) as iSCSI target?
•      Are you going to implement clustering on your Datacenter servers?
•      You said nothing about how many switches do you have. You will need two, dedicated to iSCSI traffic only, for a really good design    

In the perfect world scenario, I would have my StarWind Enterprise servers mirrored and the Database servers clustered. And, I would have two 10Gbps switches in order to implement iSCSI MPIO. Since I already have the number and the type of servers I would do careful planning and create several medium sized LUNs using all 5TB disk space. Then I could manage my virtual drives within Hyper-V keeping the free space there instead of on StarWind level.

Author Comment

ID: 35118275
Great info:  Here is more on our current config:
- Latest version of Starwind
- Planning on running in HA mode across two iSCSI boxes
- Dual 10GbE Switches with redundant paths for App Servers and iSCSI servers for data traffic only
- RAID partioning has already been done, but could be redone if necessary
- RAID:  8 x 300GB 15k SAS in RAID 10 => 1.1 TB  fast transactional storage
- RAID:  4 x 2TB SATA in RAID 10 => 4TB longer term non-transactional storage
- Point of this (amazingly complicated) exercise is to use Live Migration / Clustering for HyperV


LVL 42

Expert Comment

ID: 35119278
When I say smaller LUNs, I mean that ideally the free space on the RAID set should be larger than the size of any LUN on that RAID set. This gives you the option of cloning that LUN. So, for the 1.1 TB LUN I say that it would be preferable to two 300 GB LUNs instead of a single 600 GB LUN, for example. I would certainly not allocate all space to LUNs. Since you want to get started with CSV and live migration, create a small LUN on the SATA RAID for your Hyper-V cluster quorum, like 100 MB. I haven't figured out how small you can make it, but it only seems to get 15 MB written to it. Create another 100 GB LUN and add it to your Hyper-V cluster and then make it a CSV. Put a VM on it, configure your virtual networks, and try things out. You can always grow your LUN and move your VM to a new one. There is no better learning than to try. As long as you have space to be able to move things to a new LUN, you aren't locked into anything.

I haven't found clear documentation on it, but I have found that formatting the CSV volume to 64K seems to improve performance for my VMs, particularly for copying over VHD files for a new VM. My SAN also uses 64K IOs natively, so it makes sense that make all of the IOs the same size as they translated through the various layers.

LVL 47

Expert Comment

ID: 35119444
Formatting to 64K would absolutely improve performance.  Formatting to something larger would too, but the sweet spot is a function of several variables.  NO matter what, formatting to less than 64K would always be less efficient then 64KB
LVL 20

Accepted Solution

Svet Paperov earned 2000 total points
ID: 35121841
Wow! I am really impressed with your gear.

First, I really think you should ask the StarWind engineers for an advice about the setup of your StarWind boxes. That is the most important part and you would want to do it right. You got some great advices here from all experts about iSCSI in general but I don’t see enough discussion related to StarWind and my experience doesn’t go so far in this (as I sad earlier I am using the Standard version on a single Windows Server 2003 box for testing purposes only).

Second, if you are willing to repartition your RAID, I would go with a small number of huge LUNs/targets each one representing a single RAID partition attached as an external drive to StarWind manager (I am pretty sure that StarWind can convert a physical drive, RAID volume in your case, in an iSCSI target without first formatting it in Windows, but I could confirm that with the StarWind engineers). In that way you will eliminate one level of virtualization (image files in Windows OS) and your StarWind iSCSI boxes will be similar to a physical iSCSI box. And, may be you will have some gains on the performance bypassing Windows OS.
If you decide not to reparation one or both RAIDs, having smaller LUNs/targets (represented by images files on your StarWind boxes) may be is better idea – just to make them more manageable.

Finally, I think that you should solve your StarWind issue first; I mean, decide on size of and create the targets, setup the mirroring in HA mode and make sure that the iSCSI part is fully redundant. Then, if you need it, you could start a new discussion here regarding the setup your Hyper-V clustering and all other components of Hyper-V as the allocation unit size of the volumes, jumbo frames, network offloading, etc.

What you are trying to accomplish is really great and complicated in the same time. So, I will love to follow you and help you with Hyper-v, if I could, but, seriously, call StarWind. I am sure they will be willing to help you, especially with so high-end setup. May be they won’t charge you for that since you are advertising their solution here in the community. Personally, I would love to know what they have to say since, in my opinion, StarWind iSCSI is a very good alternative to all expensive hardware iSCSI solutions.            

Please, let us know of your final decisions and why you took them before closing the question. Thanks.

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The article will include the best Data Recovery Tools along with their Features, Capabilities, and their Download Links. Hope you’ll enjoy it and will choose the one as required by you.
The question appears often enough, how do I transfer my data from my old server to the new server while preserving file shares, share permissions, and NTFS permisions.  Here are my tips for handling such a transfer.
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…
Suggested Courses
Course of the Month19 days, 5 hours left to enroll

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question