Link to home
Start Free TrialLog in
Avatar of dinfopdfc

asked on

What are the Pros and Cons to one large iSCSI Target vs. smaller multiple iSCSI Targets?

Very new to setting up a SAN and would like to know given the environment below. What are the Pros and Cons to one large iSCSI Target vs. smaller multiple iSCSI Targets?

We have 4 servers boxes [10 Gbe network]:
- (2) Windows Server 2008 R2 Standard with StarWind iSCSI (5TB each in RAID 10 config)
- (2) Windows Server 2008 R2 Datacenter with Hyper-V

(8) Hyper-V servers:
- (2) SQL 2005
- (1) Exchange Server
- (3) Web servers
- (2) Development servers
- (1) File Server

Avatar of David
Flag of United States of America image

Not all operating systems can deal with an iSCSI target > 2.09TB
you can "tune" RAID controllers to have different performance characteristics by manipulating the RAID & filesystem parameters, so you can make some targets faster for various I/O loads (like read vs. write and  sequential vs random I/O)

Multiple targets let you configure the RAID so I/O to any single target won't affect every disk you have.
It depends.
•      One iSCSI Target can be attached to only one server, so you need at least 2 targets for your Hyper-V servers
•      If you are planning to use pass-through disks in the virtual machines – that are another targets (1 target = 1 pass-through disk)
•      Beside that there is no real advantages of having multiple targets

On the other side, if you decide on multiple iSCSI targets, 1 for each VM, this could make the space management more difficult. Say, for example, you have a single VHD on a 100GB iSCSI target. If, at some point you need to extend that VHD, you will have to go through long process of shutting down the VM, exporting the VHD, destroying the iSCSI target and creating a new one, bigger to host the extended VHD.

However, we should not forget that StarWind targets are actually image files on the host (again, I think, depending on the license of StarWind you are purchased). So having a huge image file of several TB makes me uncomfortable (this is my opinion: I love StarWind for the fact that I could turn an old WS 2003 server into a iSCSI target but I would prefer a physical iSCSI for my production data, something as Dell MD3000i – there are too much layers of virtualization with StarWind).  

So, it is up to the type and the size of data on each particular VM and up to your level of confidence in StarWind iSCSI technology and implementation.

With this particular setup I would go with pass-through disks for Exchange and File servers (data only), fixed size VHD for SQL and dynamic VHDs for all system drives and the rest of the servers. So, 5x targets of 1TB per StarWind image would be a balanced solution.

I am not quite sure but I think there is a StarWind license that allows you to attach physical disks as iSCSI target. If this is still the case, and you didn’t cut your RAID array yet I would go with 4 targets: 2 for pass-through disks and 2 for the rest of VHD.

Finally, I think, the deployment engineers in StarWind can give the best advice.

Conclusion: it depends.
The biggest issue with 1 target vs many is multipathing.  If you have lets say a Cisco switch and you configure etherchannel or link aggregation on the SAN and give it 2 network adapters in a virtual interface.  From that point you configure a single target with a single IP..if that is done that you have one IP communicating to another IP and thus a single session is formed.  Because of this the session can't go over a single physical nic on a multiport link aggregated interface.

Now..if you have multipe targets creating multiple sessions you can then use all the ports in a link aggregated link since sessions will redistrubute on different physical nics connecting to the SAN.

I think this is biggest PRO for multiple IPs.  The downside is management.
Avatar of dinfopdfc


Are there any reliability issues with Starwind? just ask because of your comments.

Also outside of speed what other pros are there on Pass-Through disk vs. Fixed?
I don't think that volume size > 2 TB is a problem for you because you are using Microsoft Hyper-V. The CSV volume will need to be a GPT disk, but that isn't a problem.

Since you are using StarWind on top of a single RAID 10, you can't make some LUNs faster than others. Also, since every LUN is striped across all of your physical disks, activity on any LUN will affect all LUNs.

You should be using CSV with Hyper-V, so you can do failover and Live Migration. You can put all VMs on a single volume.

With Hyper-V R2, the performance between dynamic and fixed VHD pretty much went away, and I actually use dynamic VHD for everything other than dedicated VHD for the pagefile.

For SQL and file server data, you can use passthrough, dynamic VHD, fixed VHD, or create a separate LUN and use the iSCSI initiator inside the VMs to connect. I actually use the iSCSI initiator inside the VM to connect. The advantage is that you get all of the features that StarWind provides such as snapshots, clones, online volume expansion, etc. (I am not a StarWind user, so I don't actually know everything it does). This is some additional overhead to doing it this way, but it is the most flexible, and if wanted the best performance you wouldn't be running StarWind anyway.

I would make your initial LUN for your Hyper-V CSV as big you you need it to be initially, with some extra space but don't allocate everything to it. You will need space to be able to create additional LUNs going forward. You should be able to grow a LUN over time and without downtime, so there is no reason to assign something like 2 TB to your first volume. Maybe you start at 100-200 GB and go from there. There are days when I have expanded a LUN on my EqualLogic SAN 5 times in a day. You don't want to have to be doing that, but you should be able to.
Regarding my comment on StarWind:
I am not aware of any reliability issue with StarWind but I don’t have very much experience with it as I am using the last free version that was available in 2010 on an old Windows Server 2003 to host some virtual machines for testing purposes (only to see how StarWind performs).  The only thing that bothers me is that its iSCSI targets are actually huge image files (2x 200GB in my case) and if something goes wrong with the StarWind host or with those files I will be stuck with a very long restore. Again, if you have the enterprise license of StarWind you could attach a physical volume instead of using img files. I doubt it, but may be in that case you could have similar output as with a hardware iSCSI solution.

Regarding pass-through disks:
kevinhsied already gave a good answer on the question fixed or dynamically expanding disks.
Just to add something on pass-through vs. virtual disks: again, for me, it’s the same reason – I don’t like using huge VHD files, instead I prefer the pass-through (650GB in my case). Actually, there are some drawbacks with the pass-through disks: you cannot back them up from the Hyper-V host since it doesn’t see them (you need a backup solution within the guest), and you cannot take a snapshot (but this is not a problem since the snapshots are not good idea in production environment, except for development purposes).

The only good advice that I can give you is: do a really good planning because you will be stuck with your decisions for quite long.
Actually, if you are using smaller volumes such that you have free space to be able to create more volumes and to expand existing ones, I say that you will have the flexibility to change what you are doing over time. You may be doing some data migration, but you won't be stuck with what you have.
"Since you are using StarWind on top of a single RAID 10, you can't make some LUNs faster than others. Also, since every LUN is striped across all of your physical disks, activity on any LUN will affect all LUNs."

True, but only to a limited extent.    Throughput is mutually exclusive with I/Os per second. If you are using databases, then 2 x RAID1s will provide TWICE the I/Os per second then 1 x RAID10, provided the arrays are built for 64KB chunk sizes, which is native to SQL Server.

Not only that, but iSCSI over ethernet still uses CSMA/CD.   The larger the packet size, the greater the probability of collisions & retries that kill performance.  Jumbo frames become an absolute necessity, and unless you are using fibre optic cabling, point-to-point, or have unbelievable quality CAT6, then you will not get full efficiency, at least not in this universe.

So first lesson -- throughput is mutually exclusive with IOPS.  RAID levels all have inherent strengths and weaknesses depending on the type of I/O.

If this was a direct-attach interface then it would be a completely different thing.    Also  what has not been mentioned is the need for a proper ethernet controller, one which offloads processing.  You don't want every single byte causing an interrupt.  If you look at block diagrams of motherboards, a fair number of them run the networking through the same bus as the primary disk interface.  You could very well get to a point where they compete for bandwidth.

Lesson 2 -- Consider the dynamics of the protocol.   You are not doing block-oriented I/O, so don't use their performance characteristics beyond the machine that is physically connected to the disks.

Lesson 3 -- Ethernet is inefficient, controllers, switches, and cabling is a real-world factor.  Choose unwisely and you could have SSDs running on the server, but you could end up with performance as if you had a USB-attached disk drive by the time the data gets to the client computers.
kevinhsieh:"Actually, if you are using smaller volumes such that you have free space to be able to create more volumes and to expand existing ones, I say that you will have the flexibility to change what you are doing over time. You may be doing some data migration, but you won't be stuck with what you have."

True, except for … how small? With the size of a single VHD – not very practical – we are loosing the benefits of dynamically expanding VHD.  

To clarify a little bit the discussion I have some questions for the author:
•      Which version of StarWind do you have?
•      Are your using mirroring with StarWind (you could do it with the Enterprise version)?
•      Did you already partition your physical RAID 10 array?
•      Does StarWind allow you to attach physical disks (SCSI volumes) as iSCSI target?
•      Are you going to implement clustering on your Datacenter servers?
•      You said nothing about how many switches do you have. You will need two, dedicated to iSCSI traffic only, for a really good design    

In the perfect world scenario, I would have my StarWind Enterprise servers mirrored and the Database servers clustered. And, I would have two 10Gbps switches in order to implement iSCSI MPIO. Since I already have the number and the type of servers I would do careful planning and create several medium sized LUNs using all 5TB disk space. Then I could manage my virtual drives within Hyper-V keeping the free space there instead of on StarWind level.
Great info:  Here is more on our current config:
- Latest version of Starwind
- Planning on running in HA mode across two iSCSI boxes
- Dual 10GbE Switches with redundant paths for App Servers and iSCSI servers for data traffic only
- RAID partioning has already been done, but could be redone if necessary
- RAID:  8 x 300GB 15k SAS in RAID 10 => 1.1 TB  fast transactional storage
- RAID:  4 x 2TB SATA in RAID 10 => 4TB longer term non-transactional storage
- Point of this (amazingly complicated) exercise is to use Live Migration / Clustering for HyperV


When I say smaller LUNs, I mean that ideally the free space on the RAID set should be larger than the size of any LUN on that RAID set. This gives you the option of cloning that LUN. So, for the 1.1 TB LUN I say that it would be preferable to two 300 GB LUNs instead of a single 600 GB LUN, for example. I would certainly not allocate all space to LUNs. Since you want to get started with CSV and live migration, create a small LUN on the SATA RAID for your Hyper-V cluster quorum, like 100 MB. I haven't figured out how small you can make it, but it only seems to get 15 MB written to it. Create another 100 GB LUN and add it to your Hyper-V cluster and then make it a CSV. Put a VM on it, configure your virtual networks, and try things out. You can always grow your LUN and move your VM to a new one. There is no better learning than to try. As long as you have space to be able to move things to a new LUN, you aren't locked into anything.

I haven't found clear documentation on it, but I have found that formatting the CSV volume to 64K seems to improve performance for my VMs, particularly for copying over VHD files for a new VM. My SAN also uses 64K IOs natively, so it makes sense that make all of the IOs the same size as they translated through the various layers.

Formatting to 64K would absolutely improve performance.  Formatting to something larger would too, but the sweet spot is a function of several variables.  NO matter what, formatting to less than 64K would always be less efficient then 64KB
Avatar of Svet Paperov
Svet Paperov
Flag of Canada image

Link to home
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial