HOWTO: iSCSI with Multi-Path in a Windows Server Environment

Introduction

iSCSI has fast become a good, cheap alternative to expensive Fibre Channel storage, and in most cases does not require any additional infrastructure changes.  You can use the same network cable and switches as you do for most of your standard network connections - since it is based on IP over Ethernet.

The thing is, reliability plays an important role in server environments, and while hardware is generally good and stable, it is still up to us to make it as reliable as possible - by adding redundancy.

Consider this:  You have a 12 Drive SAN providing iSCSI to your Windows Servers, similar to the drawing shown:

The 12 drives are naturally in a 2 x 6 drive RAID 5 configuration, providing redundancy at the drive level.  The SAN itself has two power supplies, thus providing redundancy at the power front, however, there is only 1 controller, and one network cable, providing no redundancy at the network front.

Easily resolved! Just add another controller, add another network cable, and Bob's your Uncle.  Unfortunately, it's not that simple.

This article is to help you set it up properly, so that you have fully redundant, multi-path IO to your SAN for your high availability needs.
 

Configuring your Hardware

In our sample configuration, I'm going to include two servers and an iSCSI SAN in a Clustered environment, and we can build it up from there.  Naturally, you don't need two servers, but since we are talking high availability here, you might be wanting to think about it.

In addition to the two servers, we're going to use an iSCSI SAN which has dual controllers, thus providing redundancy if a controller fails.  In addition to this, we're going to use both ports of each controller, so that we have redundancy if a switch fails.  This is what our redundancy now looks like.

The idea is, that if any one single device fails, there is no interruption to the service.

Try it!  Pretend that the left most of these two switches blows up. I've never heard of a Cisco switch blow up, but try to use your imagination.  Server A would have lost its connection to the IBM SAN over iSCSI VLAN A, but that's not a problem!  It has a connection to the survived switch, which in turn has a connection back to the IBM SAN.

NOTE: It is not necessary to have two separate VLAN's for your iSCSI multi-pathing, but it's highly recommended, because if you use the same VLAN (or more importantly, the same Subnet), you cannot guarantee the paths it's going to use!

As far as pathing is concerned, did you know that each server has 4 paths to the storage?  It's true!  Despite the fact that there are only two network cables dedicated to the iSCSI solution, there are in fact 4 paths.  Port A (the left port) on Server A can in fact talk to Channel 3 of Controller A (left controller) of the SAN, and it can talk to Channel 3 of Controller B (right controller) of the SAN.  This is achieved by means of the 10 Gigabit trunk between the two switches.  This is therefore, two paths to the storage - one source interface, and two target interfaces.  Similarly, Port B of the server can talk to Channel 4 of Controller A and Controller B - thus providing 4 channels to the storage.
 

IP Addressing

Knowing that iSCSI VLAN A and iSCSI VLAN B should be on different subnets, we can now set up our IP address scheme.  You can pretty much use any of the private IP Schemes available.  As 172.16-31 are rarely used, I'm going to use these.

Server A: iSCSI VLAN A - 172.31.1.100
Server A: iSCSI VLAN B - 172.31.2.100
Server B: iSCSI VLAN A - 172.31.1.101
Server B: iSCSI VLAN B - 172.31.2.101
SAN: Controller A - Channel 3 (VLAN A) - 172.31.1.1
SAN: Controller A - Channel 4 (VLAN B) - 172.31.2.1
SAN: Controller B - Channel 3 (VLAN A) - 172.31.1.2
SAN: Controller B - Channel 3 (VLAN B) - 172.31.2.2

Once I've documented my IP Addressing, I'm going to configure my SAN interfaces, and my Server interfaces to have these addresses.  Be sure that you can ping all the SANs interfaces
 

Configuring your Software

Connecting to your Servers to connect to your storage has to be done in stages, and may require multiple reboots.  Ensure that you do this only if you can get downtime to do it.
 

1. Installing MPIO

The first thing you have to do is install MPIO.  A reboot is required in order to complete the installation, so do this.  Installing MPIO can be done in powershell using the following commands:

 

Import-Module ServerManager
                      Add-WindowsFeature Multipath-IO

Select all Open in new window

Reboot the server.  NOTE: With Windows 2012, you do not need to reboot the server after adding MPIO.

2. Running the iSCSI Initiator

iSCSI is automatically a part of Windows 2008 R2 and Windows 2012, so all we have to do is run the iSCSI Initiator Control Panel.  The iSCSI service however, is installed but not configured to start automatically.  Running the iSCSI Initiator for the first time will ask you if it should start the service, and configure it to automatically start.  You say Yes to this.

To launch the iSCSI Initiator on Windows Core, log in to the server and run "iscsicpl" from the command prompt.

You will be presented with a screen that looks like this.

3. Add all the IP Targets

Most people would just enter in an IP address in the Target field and click "Quick Connect".  We're not going to do this.  What we should do is go to the "Discovery" tab and then click "Discover Portal..." and enter in the IP address of the iSCSI port on the SAN.  

NOTE: Your SAN will most likely have management interfaces in addition to the iSCSI port interfaces - do not specify the management interfaces here!

During this process, the server will communicate with the target you are adding, and will find its name and add it to the "Targets" tab.  This may take a short while and iSCSI Initiator may appear to stop responding - this is normal.

We add all IP's here so that if in the event of hardware failures, and the server is rebooted, it still has knowledge of all the target IP's it can communicate with in order to find its targets.

Repeat until all your iSCSI SANs IP's have been added

4. Define the Host in the SAN

As previously mentioned, adding the IP's will make iSCSI communicate with the SAN, so that it can discover what it's name is.  During this process, the SAN also learns about the host, but doesn't quite do anything with it yet.  What you usually have to do is add the host - and add it to a host group - so that we can define LUNs to the host (or group of hosts).

Since everybody's SAN is different, the process may be different for you, but I will show you what it looks like to define a host on the IBM DS3500.

NOTE: You only have to add each host once - not multiple times like you did with the iSCSI Target IP's.

5. Provision your LUN(s)

Still at the SAN, you should now provision some LUNs to your host or host group.  Doing this will make disks become available to the host when it establishes a connection through iSCSI.  Again, this may look different to you but below you can see that I have connected to Logical Drives with LUN ID's 1 and 2 to my Host Group "ClusterServers", which contains the host I have just added.

6. Connect your first path

Going back to your iSCSI Initiator on your server, select the "Targets" tab.  If all has gone well, you should see a single target (i.e. your SAN), and the status is Inactive.

Highlight the target, then click on "Connect".  A new dialog box will appear.  Ensure you check "Enable multi-path" here.  DO NOT CLICK OK YET!

Click "Advanced", and in the Advanced box, enter in the Initiator IP, which will be one of your iSCSI interface IP's, and a Target IP which should be on the same subnet as your local interface.

This becomes your first "path".  Before we can actually add more paths though, we have to identify the disks as Multipath.  The fact that you've enabled multipath on the iSCSI Initiator, and Multipath on the server, does not automatically make the disks multipath.  We do this in the next section.

Once you click OK, you will see that the target is now connected.

7. Enable iSCSI MPIO

Launch the MPIO Control Panel.  You will find this in your Administrative Tools in Windows 2008 R2, or in the Tools menu of Windows 2012, or by running "mpiocpl"

This tool will list all known hardware devices to which MPIO is supported.  Fibre Channel attached SAN's generally need software known as DSM's (Device Specific Module) to support its hardware, but because iSCSI is a non-vendor specific standard, we can use Microsoft's DSM.  The only thing is, this DSM may not exist on your machine yet, so we need to install it.  N.B. If your vendor does provide a DSM, install this instead of following this step!  It's better to use a Vendor Provided DSM rather than Microsoft's generic DSM, as Microsoft's DSM may offer you MPIO Policies which are incompatible with your SAN.

Go to the "Discover Multi-Paths" window.  This window will list all SP3-compliant devices (SCSI Primary Commands - version 3) that the system can find which does NOT have any DSM loaded.  This will not however show any iSCSI devices, but you still need to check "Add support for iSCSI devices" before clicking "Add".  

After a few minutes, it will prompt you to reboot.  This will be your final reboot.

8. Verify MPIO is available for your disks

So far, we've only created one path to our disk, but it is important to check that the disk has been successfully configured to support Multipathing.  If the disk does not support multipathing, then the disk will not fail over when the primary path fails, causing some horrible, horrible downtime.

Using Computer Manager (if on Windows Server Core, then use Computer Manager from another server or an administrators station and connect to the target server.  DISKPART does not list the detail we wish to see), find the volumes which you provisioned from your SAN earlier, select the properties of the disk and ensure that there is an MPIO tab.  If the tab is there, the disk supports it.  If the tab is missing, you need to trace your steps as you have probably missed something - or the iSCSI SAN you are communicating with does not support multipathing.

9. Connect the other paths

The last step is to add the other paths to the same connection we already have established.

Start up the iSCSI Initiator again - where you will note that your target(s) is/are already connected.  Select the one you wish to configure, then click "Properties"

This page shows you the sessions it currently has to the target.  Right now, it gives you an arbitrary Identifier, which contains the configuration for our first path.  Click on "Add session", and you will see a familiar dialog.

Again, check "Enable multi-path", and click Advanced, but this time choose another path.  In my scenario above, I know that I have a path to each of the two controllers, from each of my two network interfaces - this gives me 4 possible combinations, as shown in the table below.

As per my initial connection earlier, I have already used up the first path.  Now, I should configure the second path, so I configure using the second paths properties.

Once you click OK and OK again, you will see that there are now two identifiers in your Session.

Repeat this step until you have all 4 paths.

If at any point you need to verify the information inside these connections, you will find each one listed under "Favorite Targets".  The name will be the same for each, but the Details will be that of the IP configuration you had set.  You cannot make changes here unfortunately, but you can remove a session here by clicking Remove, and then use the "Add Session" step to create a new session.

10. Configuring Failover

Now that we have added all the paths, we can configure our Failover policies.  Each disk that you have provisioned has its own failover policy.  You configure these from iSCSI Initiator, using the "Devices" button.

You may notice that I've got a lot of disks here when in fact I only have 2.  Don't be alarmed.  It's showing both disks 4 times - one for each path.  This is perfectly normal.  To configure MPIO, you just have to do it once per disk.  Select a disk and then click MPIO

The default mode is "Round Robin", which means that the Operating system will select a path based on the number of paths.  For example, Disk 13 in my screenshot will likely use Path 1 for Operation A, then Path 2 for Operation B, then Path 3 for Operation C, et cetera.  The only thing is, my particular SAN doesn't enjoy operating in this mode (in fact, it won't work), so in my environment, I want to set mine to "Fail Over Only".

Choose a Load balance policy based on your hardware requirements and situation.  Check your SAN documentation for guidelines.  For more information about MPIO Policies, there is a link below.

11. Double-checking & Testing

It's always good to do sanity checking to ensure you've done this all properly, and the best way to verify that it's working is to check the following:

 




 

12. Repeat

Repeat these steps with the other server, and now you have two servers with shared storage so you can start your clustering fun!  A link to my Clustering Article is given below.
 

Other Considerations

Now that you've seen how to configure Failover iSCSI for Windows Servers, you may have some other questions that you would like to ask.  Unfortunately, I don't have all the answers, but luckily - Experts Exchange does!  I will however go over some common thoughts from system administrators and provide food for thought.
 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


 


 

Further Reading

Some of the following links may be useful for you.

Wikipedia: iSCSI
Technet: MPIO Policies
EE Article: Storage Explored, Explained, and Exampled
EE Article: Creating a Hyper-V Cluster on Windows 2012
 

Any Quesions?

If you have general questions regarding iSCSI, you're at the right place.  Experts Exchange is here to help all.  Please submit a question rather than adding a comment to this tutorial in the hopes that I (or somebody else) will reply.

Take care, and Happy iSCSIing!