• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 3827
  • Last Modified:

VMware ESXi slowness during cold migration

Hello,
My local datastore on my ESXi 4.1 server has less than 10GB left, so I purchased an IOMega/EMC ix2-200 NAS.  My plan is to cold-migrate one of the VM's (CentOS 5.5/CPanel) to the NAS to free up some space.  I installed the VMKernel driver and can see the NAS from ESXi.

As an experiment before I work on the live VM, I created a CentOS 5.5 VM with 384MB memory and an 8GB thin-provisioned disk.  Once that was created and running, I shut it down and used the "Browse Datastore" function to browse and copy/paste the VM from the local datastore to the NAS.

The result was that it took 2 hours and 20 minutes to complete!  The live VM would certainly take longer as it has a 256GB thin-provisioned disk consuming 13GB.  The VM did happily fire up and run once I reconfigured the VM drive settings.  I just don't have the maintenance window required to move the live VM, so either I'm doing something wrong that I could correct or I need to schedule a reaaaaaallllllyyyy long maintenance window.

The VM host is an IBM Bladecenter HS21 blade with dual Xeon 5160 processors and 10GB memory.  Local datastore is 73GB SAS mirrored drives.  NAS datastore is 2x1TB mirrored drives connected via gigabit ethernet.  Connected to the Bladecenter switch, I can move a 4GB ISO file from my laptop to the NAS in 3 minutes.  This model NAS regularly shows 60+ MB per second when tested by Tom's Hardware, CNET, etc.

Lastly, I'll add that an earlier attempt at cold migration was done with the VMware converter and that move took 15 minutes or so.  However, the result wasn't usable as it presented a different ethernet interface to the VM, causing the IP configuration to be wiped out.  That may be a recoverable situation, but with all of CPanel's bindings I don't want to take the chance.
0
e2346437
Asked:
e2346437
  • 10
  • 8
  • 4
1 Solution
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Are you using NFS or iSCSI, if so do you have Jumbo Frames enabled.

Have you got both (2) NICs on the Iomega ix2-200 connected to the network via bonded, channel group or etherchannel.

Have you got the blade centre connected with dual nics, bonded, channel group, or etherchannel, and IP hash configured on the vSwitch.

What configuration have you got the iomega ix-200 in?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
What protocol are you using to copy the ISO from Laptop to NAS, CIFs, iSCSI or NFS?

Any errors on the switch? (that NAS or ESX is connected to)

Are the NICs on the ESX server connected at 1Gbe.
0
 
e2346437Author Commented:
I'm using NFS.  Jumbo Frames are enabled on the NAS, not sure about inside ESXi.

The ix2-200 does not have dual NIC.

ix2-200 is configured for mirrored drives.
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
What are you using to connect to the NAS, when  you are using your laptop NFS?

If you have configured Jumbo Frames on the NAS, but it's not configured on the network switches, or ESX, this could be an issue.

Jumbo Frames need to be configured everywhere, not just on the NAS.

Are you copying from Laptop to NAS using NFS?
0
 
e2346437Author Commented:
The IBM Bladecenter has a built-in Cisco switch.

When I copy from my laptop to the NAS, I'm doing so using NFS and my traffic is going through that switch.

When I copy from ESXi to the NAS, I'm going through the same switch.

How are Jumbo Frames enabled in ESXi?
0
 
bgoeringCommented:
I would recommend using VMware Converter (http://www.vmware.com/products/converter) to perform a V2V "conversion" (copy) of the vm from one datastore to another - this will give you the option to specify thin provisioning on the output file and could speed things up.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
login into the server via ssh

try esxcfg-vswitch -l


~ # esxcfg-vswitch -l
Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0         64          18          64                1500    vmnic0,vmnic1

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  Management Network    0        1           vmnic0,vmnic1
  VM Network            0        14          vmnic0,vmnic1

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch1         128         1           128               1500

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VLAN 2                2        0
  VLAN 5                5        0

this will display the above output

then to change to jumbo frames set mtu to 9000

esxcfg-vswitch -m 9000 vSwitch1

and then check with esxcfg-vswitch -l again

 esxcfg-vswitch -l
Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0         64          18          64                1500    vmnic0,vmnic1

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  Management Network    0        1           vmnic0,vmnic1
  VM Network            0        14          vmnic0,vmnic1

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch1         128         1           128               9000

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VLAN 2                2        0
  VLAN 5                5        0


I would only change the mtu on a virtual swtich and networking that's configured for jumbo frames, unless you want to change the default vSwitch0.
0
 
e2346437Author Commented:
@bgoering I did do that but the VMware Converter mangled the Ethernet interface, as I mentioned in the first post.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
make sure your physical switch is also configured for Jumbo Frames?

What operating system is on your laptop, and what NFS client are you using?
0
 
e2346437Author Commented:
@hanccocka The command line doesn't scare me but can this be done in the GUI?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, VMware Converter can "change" the Network Interface sometimes.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
No Jumbo Frames can only be enabled at the command line.
0
 
e2346437Author Commented:
Thanks I'll give that a try!
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Also make sure if you have an option on the phyiscal switch and NAS, that you specify the same packet size of 9000, which is often referred to Jumbo Frames.

A word of warning, if you network infrastructure is not configured for Jumbo Frames correctly (end to end), you can make networking worse.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
What operating system is on your laptop, and what NFS client are you using?
0
 
bgoeringCommented:
hmmm you should be able to specify the type of ethernet interface (something like use original?) during the conversion where it won't change it on you. What kind of virtual ethernet interface do you have on your vm?
0
 
e2346437Author Commented:
@ hanccocka Laptop OS is Win 7 Pro 64-bit

@bgoering Don't recall the ethernet interface, just used the default assigned when CentOS 5.5 64-bit is chosen as the VM OS.

0
 
bgoeringCommented:
Another path to consider (and this should be even faster because all of the file I/O is "local") is to just create a new vm on the NAS without the virtual disk, then use the vSphere Remote CLI vmkfstools command with the --clonevirtualdisk and --diskformat thin options to copy the disk over to the new volume.

See http://www.vmware.com/support/developer/vcli/vcli41/doc/reference/vmkfstools.html for more information on vmkfstools.

You can also enable tech support mode and log into the console to use the vmkfstools command if you don't want to use remote cli...
0
 
bgoeringCommented:
That would be an E1000 interface - it is the default for pretty much any 64-bit OS.
0
 
e2346437Author Commented:
The problem turned out to be the search/index function of the IX2-200 NAS.  It's turned on by default and attempts to index anything placed on the drive.  Once I turned it off, the file copied in less than 3 minutes.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I'm not convinced, by this, and after disucssiions with our Iomega/EMC Account Manager today a Technical specialist advised of the following, and hence would not recommend this unit for this environment. So you mileage may vary with using this device. If it's working for you great, I have posted this information, for followers of this post to advise of future issues with this device.

This desktop product is not suitable as a backup unit for this environment. It's not an EMC product although Iomega's parent company is EMC.

This was the statement made:-

    In some cases the Write Caching feature may slow the performance of the StorCenter device. To disable, set Disk Write Caching to Always disabled (Disks┬┐Manage Disks┬┐Write Caching).

    Once a month, the StorCenter will verify its drives. Although the status light does not show disk activity during this process, the drive performance will be low. You cannot schedule, quit, or delay the verification. To find out if your StorCenter drive is performing a disk verification, see the Dashboard in the StorCenter Storage Manager software. The drive status will display as Verifying... followed by the percentage complete.

    NOTE: In some instances, the disk verification count-down timer may be started by the manufacturing process. If you are experiencing slow performance on a new machine, it is probably due to disk verification. Once the verification is complete, your StorCenter ix2 will return to full speed.

Because of the above issue of Disk Verification, Backup to this unit will be slow. This cannot be disabled.
0
 
e2346437Author Commented:
My solution was correct.
0

Featured Post

Restore individual SQL databases with ease

Veeam Explorer for Microsoft SQL Server delivers an easy-to-use, wizard-driven interface for restoring your databases from a backup. No expert SQL background required. Web interface provides a complete view of all available SQL databases to simplify the recovery of lost database

  • 10
  • 8
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now