We help IT Professionals succeed at work.

VMWare iSCSI problems

Steven Vona
Steven Vona asked
on
Medium Priority
472 Views
Last Modified: 2012-06-27
We have a few ESXI hosts and a dell equallogic iscsi that connect to a 3750 switch.  

Last night the network folks took down the switch temporarily to do an IOS upgrade.  That seems to have screwed up our linux virtual machines because they all had read-only filesystems when I came in this morning.

I ran fsck on the machines and it seems to have corrected all the problems except one server.

My question is, how can we stop our machines from having corrupted file systems if we lose a switch in the future?




Comment
Watch Question

Paul SolovyovskySenior IT Advisor
CERTIFIED EXPERT
Top Expert 2008

Commented:
Are you using iSCSI to connect to shared storage?
CERTIFIED EXPERT

Author

Commented:
Yes, we have 3 datastores for our vm guests, all housed on the iscsi array.

Paul SolovyovskySenior IT Advisor
CERTIFIED EXPERT
Top Expert 2008

Commented:
What most likely happened is that you didn't shutdown the VMs during the iOS pgrade and the switch lost connectivity during the upgrading leaving the VMs unstable.  Most of the time they will come back up but I have seen this corrupt VMs.

Did you shutdown the VMs? Anytime you perform maintenance such as this it is a pre-requisite
CERTIFIED EXPERT

Author

Commented:
No we did not.  

So if we lose a network switch chances are we can lose all out VMs too?  This seems like a major drawback to using VM.

Paul SolovyovskySenior IT Advisor
CERTIFIED EXPERT
Top Expert 2008

Commented:
It is not..keep in mind that you are writing to the OS..the OS uses iscsi which emulates the scsi protocol..you take that away and what would happen?

Think about taking a raid controller out of a live server and then try to find out what your OS system is not booting up when you put it back in.

You can avoid this by using FibreChannel and the only piece you would lose is your network connection to the VMs as you would with physical servers if you upgrade the cisco switches next time.

If all your infrastructure is flowing through the switches wouldn't it make sense to shutdown the infrastructure that will be affected by the upgrade?  
CERTIFIED EXPERT

Author

Commented:
Yes I understand your point.

So my question is, there is no way to have redundant connections to iSCSI?

Paul SolovyovskySenior IT Advisor
CERTIFIED EXPERT
Top Expert 2008

Commented:
How are you running your switches?  
How many of them are in the environment?
Are you using the stack cables? If so are you trunking between the switches?
CERTIFIED EXPERT

Author

Commented:
Right now we have two switches, both are trunked to the core switch, but not to each other.

Keep in mind I am not a network guy so I hope I am explaining this correctly.

I have plenty of switches at my disposal, I have about 15 cisco 3750 switches I can use for this if need be.


What do you suggest?  We would like to have an environment that if one switch goes down we can have our vm guests continue on the other switch (without file systems problems).


Senior IT Advisor
CERTIFIED EXPERT
Top Expert 2008
Commented:
If you have two nics going to each vswitch you could potentially take down one switch, disconnect from network and upgrade one at a time.  
CERTIFIED EXPERT

Author

Commented:
I dont know if we are on the same page here.  I am talking about real switches, not virtual switches.
Paul SolovyovskySenior IT Advisor
CERTIFIED EXPERT
Top Expert 2008

Commented:
How do you have your vswitches divided?  If you have one port going to switch 1 and 1 port going to switch 2 it may work..it all depends how your vswitches connect to your phyiscal switches
What you ask is possible and supported.  It should suffice to use two network switches.  Details:

- I suggest stacking the switches.  If you can't stack or prefer not to, create a multi-port trunk between the switches.
- Each ESX host must have two NICs assigned to iSCSI: a separate connection to the two network switches
- If possible configure your ESX hosts with four NICs in order to provide redundancy for VM traffic and iSCSI traffic, while isolating the two types of traffic from each other
- one pair for VM traffic with each member of the pair connected to a separate switch
- one pair for iSCSI traffic with each member of the pair connected to a separate switch
- The EqualLogic SAN must have at least one connection to each of the two network switches
- You need to enable MPIO (multi-path I/O) on every ESX host, so that if the path through one switch fails, the other one is used automatically without data loss.
- Ensure that the SAN balances iSCSI traffic across all links to the switches
Let me know if you need more info.

--klodefactor
CERTIFIED EXPERT

Author

Commented:
klodefactor,
Thanks a million for the information.  I do have some questions since it seems your familiar with equallogic / vm setup.

The equallogic has two network modules, each with 4 1 gig interfaces.  From what I understand one is standby and one is active.  Should we have one to switch to the active and one to the standy?  Or should the active one have connections to each switch?

Also is there any documentation you can point me to for setting up MPIO on vmware?

Any documentation that is specific to equallogic would be awesome!

Explore More ContentExplore courses, solutions, and other research materials related to this topic.