e2346437
asked on
VMware ESXi slowness during cold migration
Hello,
My local datastore on my ESXi 4.1 server has less than 10GB left, so I purchased an IOMega/EMC ix2-200 NAS. My plan is to cold-migrate one of the VM's (CentOS 5.5/CPanel) to the NAS to free up some space. I installed the VMKernel driver and can see the NAS from ESXi.
As an experiment before I work on the live VM, I created a CentOS 5.5 VM with 384MB memory and an 8GB thin-provisioned disk. Once that was created and running, I shut it down and used the "Browse Datastore" function to browse and copy/paste the VM from the local datastore to the NAS.
The result was that it took 2 hours and 20 minutes to complete! The live VM would certainly take longer as it has a 256GB thin-provisioned disk consuming 13GB. The VM did happily fire up and run once I reconfigured the VM drive settings. I just don't have the maintenance window required to move the live VM, so either I'm doing something wrong that I could correct or I need to schedule a reaaaaaallllllyyyy long maintenance window.
The VM host is an IBM Bladecenter HS21 blade with dual Xeon 5160 processors and 10GB memory. Local datastore is 73GB SAS mirrored drives. NAS datastore is 2x1TB mirrored drives connected via gigabit ethernet. Connected to the Bladecenter switch, I can move a 4GB ISO file from my laptop to the NAS in 3 minutes. This model NAS regularly shows 60+ MB per second when tested by Tom's Hardware, CNET, etc.
Lastly, I'll add that an earlier attempt at cold migration was done with the VMware converter and that move took 15 minutes or so. However, the result wasn't usable as it presented a different ethernet interface to the VM, causing the IP configuration to be wiped out. That may be a recoverable situation, but with all of CPanel's bindings I don't want to take the chance.
My local datastore on my ESXi 4.1 server has less than 10GB left, so I purchased an IOMega/EMC ix2-200 NAS. My plan is to cold-migrate one of the VM's (CentOS 5.5/CPanel) to the NAS to free up some space. I installed the VMKernel driver and can see the NAS from ESXi.
As an experiment before I work on the live VM, I created a CentOS 5.5 VM with 384MB memory and an 8GB thin-provisioned disk. Once that was created and running, I shut it down and used the "Browse Datastore" function to browse and copy/paste the VM from the local datastore to the NAS.
The result was that it took 2 hours and 20 minutes to complete! The live VM would certainly take longer as it has a 256GB thin-provisioned disk consuming 13GB. The VM did happily fire up and run once I reconfigured the VM drive settings. I just don't have the maintenance window required to move the live VM, so either I'm doing something wrong that I could correct or I need to schedule a reaaaaaallllllyyyy long maintenance window.
The VM host is an IBM Bladecenter HS21 blade with dual Xeon 5160 processors and 10GB memory. Local datastore is 73GB SAS mirrored drives. NAS datastore is 2x1TB mirrored drives connected via gigabit ethernet. Connected to the Bladecenter switch, I can move a 4GB ISO file from my laptop to the NAS in 3 minutes. This model NAS regularly shows 60+ MB per second when tested by Tom's Hardware, CNET, etc.
Lastly, I'll add that an earlier attempt at cold migration was done with the VMware converter and that move took 15 minutes or so. However, the result wasn't usable as it presented a different ethernet interface to the VM, causing the IP configuration to be wiped out. That may be a recoverable situation, but with all of CPanel's bindings I don't want to take the chance.
What protocol are you using to copy the ISO from Laptop to NAS, CIFs, iSCSI or NFS?
Any errors on the switch? (that NAS or ESX is connected to)
Are the NICs on the ESX server connected at 1Gbe.
Any errors on the switch? (that NAS or ESX is connected to)
Are the NICs on the ESX server connected at 1Gbe.
ASKER
I'm using NFS. Jumbo Frames are enabled on the NAS, not sure about inside ESXi.
The ix2-200 does not have dual NIC.
ix2-200 is configured for mirrored drives.
The ix2-200 does not have dual NIC.
ix2-200 is configured for mirrored drives.
What are you using to connect to the NAS, when you are using your laptop NFS?
If you have configured Jumbo Frames on the NAS, but it's not configured on the network switches, or ESX, this could be an issue.
Jumbo Frames need to be configured everywhere, not just on the NAS.
Are you copying from Laptop to NAS using NFS?
If you have configured Jumbo Frames on the NAS, but it's not configured on the network switches, or ESX, this could be an issue.
Jumbo Frames need to be configured everywhere, not just on the NAS.
Are you copying from Laptop to NAS using NFS?
ASKER
The IBM Bladecenter has a built-in Cisco switch.
When I copy from my laptop to the NAS, I'm doing so using NFS and my traffic is going through that switch.
When I copy from ESXi to the NAS, I'm going through the same switch.
How are Jumbo Frames enabled in ESXi?
When I copy from my laptop to the NAS, I'm doing so using NFS and my traffic is going through that switch.
When I copy from ESXi to the NAS, I'm going through the same switch.
How are Jumbo Frames enabled in ESXi?
I would recommend using VMware Converter (http://www.vmware.com/products/converter) to perform a V2V "conversion" (copy) of the vm from one datastore to another - this will give you the option to specify thin provisioning on the output file and could speed things up.
login into the server via ssh
try esxcfg-vswitch -l
~ # esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 64 18 64 1500 vmnic0,vmnic1
PortGroup Name VLAN ID Used Ports Uplinks
Management Network 0 1 vmnic0,vmnic1
VM Network 0 14 vmnic0,vmnic1
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 128 1 128 1500
PortGroup Name VLAN ID Used Ports Uplinks
VLAN 2 2 0
VLAN 5 5 0
this will display the above output
then to change to jumbo frames set mtu to 9000
esxcfg-vswitch -m 9000 vSwitch1
and then check with esxcfg-vswitch -l again
esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 64 18 64 1500 vmnic0,vmnic1
PortGroup Name VLAN ID Used Ports Uplinks
Management Network 0 1 vmnic0,vmnic1
VM Network 0 14 vmnic0,vmnic1
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 128 1 128 9000
PortGroup Name VLAN ID Used Ports Uplinks
VLAN 2 2 0
VLAN 5 5 0
I would only change the mtu on a virtual swtich and networking that's configured for jumbo frames, unless you want to change the default vSwitch0.
try esxcfg-vswitch -l
~ # esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 64 18 64 1500 vmnic0,vmnic1
PortGroup Name VLAN ID Used Ports Uplinks
Management Network 0 1 vmnic0,vmnic1
VM Network 0 14 vmnic0,vmnic1
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 128 1 128 1500
PortGroup Name VLAN ID Used Ports Uplinks
VLAN 2 2 0
VLAN 5 5 0
this will display the above output
then to change to jumbo frames set mtu to 9000
esxcfg-vswitch -m 9000 vSwitch1
and then check with esxcfg-vswitch -l again
esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 64 18 64 1500 vmnic0,vmnic1
PortGroup Name VLAN ID Used Ports Uplinks
Management Network 0 1 vmnic0,vmnic1
VM Network 0 14 vmnic0,vmnic1
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 128 1 128 9000
PortGroup Name VLAN ID Used Ports Uplinks
VLAN 2 2 0
VLAN 5 5 0
I would only change the mtu on a virtual swtich and networking that's configured for jumbo frames, unless you want to change the default vSwitch0.
ASKER
@bgoering I did do that but the VMware Converter mangled the Ethernet interface, as I mentioned in the first post.
make sure your physical switch is also configured for Jumbo Frames?
What operating system is on your laptop, and what NFS client are you using?
What operating system is on your laptop, and what NFS client are you using?
ASKER
@hanccocka The command line doesn't scare me but can this be done in the GUI?
Yes, VMware Converter can "change" the Network Interface sometimes.
No Jumbo Frames can only be enabled at the command line.
ASKER
Thanks I'll give that a try!
Also make sure if you have an option on the phyiscal switch and NAS, that you specify the same packet size of 9000, which is often referred to Jumbo Frames.
A word of warning, if you network infrastructure is not configured for Jumbo Frames correctly (end to end), you can make networking worse.
A word of warning, if you network infrastructure is not configured for Jumbo Frames correctly (end to end), you can make networking worse.
What operating system is on your laptop, and what NFS client are you using?
hmmm you should be able to specify the type of ethernet interface (something like use original?) during the conversion where it won't change it on you. What kind of virtual ethernet interface do you have on your vm?
ASKER
@ hanccocka Laptop OS is Win 7 Pro 64-bit
@bgoering Don't recall the ethernet interface, just used the default assigned when CentOS 5.5 64-bit is chosen as the VM OS.
@bgoering Don't recall the ethernet interface, just used the default assigned when CentOS 5.5 64-bit is chosen as the VM OS.
Another path to consider (and this should be even faster because all of the file I/O is "local") is to just create a new vm on the NAS without the virtual disk, then use the vSphere Remote CLI vmkfstools command with the --clonevirtualdisk and --diskformat thin options to copy the disk over to the new volume.
See http://www.vmware.com/support/developer/vcli/vcli41/doc/reference/vmkfstools.html for more information on vmkfstools.
You can also enable tech support mode and log into the console to use the vmkfstools command if you don't want to use remote cli...
See http://www.vmware.com/support/developer/vcli/vcli41/doc/reference/vmkfstools.html for more information on vmkfstools.
You can also enable tech support mode and log into the console to use the vmkfstools command if you don't want to use remote cli...
That would be an E1000 interface - it is the default for pretty much any 64-bit OS.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
I'm not convinced, by this, and after disucssiions with our Iomega/EMC Account Manager today a Technical specialist advised of the following, and hence would not recommend this unit for this environment. So you mileage may vary with using this device. If it's working for you great, I have posted this information, for followers of this post to advise of future issues with this device.
This desktop product is not suitable as a backup unit for this environment. It's not an EMC product although Iomega's parent company is EMC.
This was the statement made:-
In some cases the Write Caching feature may slow the performance of the StorCenter device. To disable, set Disk Write Caching to Always disabled (Disks¿Manage Disks¿Write Caching).
Once a month, the StorCenter will verify its drives. Although the status light does not show disk activity during this process, the drive performance will be low. You cannot schedule, quit, or delay the verification. To find out if your StorCenter drive is performing a disk verification, see the Dashboard in the StorCenter Storage Manager software. The drive status will display as Verifying... followed by the percentage complete.
NOTE: In some instances, the disk verification count-down timer may be started by the manufacturing process. If you are experiencing slow performance on a new machine, it is probably due to disk verification. Once the verification is complete, your StorCenter ix2 will return to full speed.
Because of the above issue of Disk Verification, Backup to this unit will be slow. This cannot be disabled.
This desktop product is not suitable as a backup unit for this environment. It's not an EMC product although Iomega's parent company is EMC.
This was the statement made:-
In some cases the Write Caching feature may slow the performance of the StorCenter device. To disable, set Disk Write Caching to Always disabled (Disks¿Manage Disks¿Write Caching).
Once a month, the StorCenter will verify its drives. Although the status light does not show disk activity during this process, the drive performance will be low. You cannot schedule, quit, or delay the verification. To find out if your StorCenter drive is performing a disk verification, see the Dashboard in the StorCenter Storage Manager software. The drive status will display as Verifying... followed by the percentage complete.
NOTE: In some instances, the disk verification count-down timer may be started by the manufacturing process. If you are experiencing slow performance on a new machine, it is probably due to disk verification. Once the verification is complete, your StorCenter ix2 will return to full speed.
Because of the above issue of Disk Verification, Backup to this unit will be slow. This cannot be disabled.
ASKER
My solution was correct.
Have you got both (2) NICs on the Iomega ix2-200 connected to the network via bonded, channel group or etherchannel.
Have you got the blade centre connected with dual nics, bonded, channel group, or etherchannel, and IP hash configured on the vSwitch.
What configuration have you got the iomega ix-200 in?