Why slow disk to disk copy in VMWare ESXi server 4.1

Trevor Helps
Trevor Helps used Ask the Experts™
on
In testing our latest server I've observed much slower performance disk to disk than disk to a NFS datastore.

Server is HP DL360 G7
The server has 3 datastores:

DS1 is on a RAID 1 consisiting of 2 10k SAS disks
DS2 is a single RAID0 disk - 7.2k SAS
DS3 is a NFS mount on a QNAP storage device.

If I copy an entire VM while powered off from DS1 to DS3 (The NFS store) its significantly faster than if I copy from DS1 to DS2.

So musch so in fact I think I could copy from DS1 to DS3 then rom DS3 back to DS2 quicker than directly from DS1 to DS2.  This clearly isnt right.

Also the vmdk files are using thin provisioning.  When I copy from DS1 to DS3 the vmdk file inflates to full size.  When I restore from the DS3 back to DS1 it returns the vmdk file to its smaller unprovisioned size - which is great.

When I do the same operation DS1 to DS2 the vmdk file inflates.  It stays inflated when restoring to DS1.

Any suggestions as to what is happening here ?

Thank you .
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
1. Have you enabled Jumbo Frames for NFS use on the
Physical Switch
ESX Server
NAS

2. What networking are you using Dual etherchannelled 1GBe NICs
on
NAS
ESX

3. What NFS NAS software are you using?

4. What underlying storage are you using on the NAS, RAID 5, RAID 10, RAID 1, 7.2k, 10k, 15k Disks, ZFS, RAIDz etc
Justin CAWS Solutions Architect

Commented:
Are DS1 and DS2 local datastores or are they on a SAN/NAS?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
have you tested disk benchmarks on datastore DS2 which is a 7.2k disk, and compared them to DS1 and DS3. I thought your issue was slow NFS?

Do you have a Battery Backup Write Cache Module (BBWC) module installed, and configured as 75% Write 25% Read?
Success in ‘20 With a Profitable Pricing Strategy

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Danny McDanielClinical Systems Analyst

Commented:
I am guessing that DS1 and '2 are using the same SAS controller...what is that controller?  Like hanccocka says, what is its configuration as it appears to be the bottleneck when copying from RAID volume to RAID volume.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
This is quite an interesting topic because in the last few weeks we've been commissioned by one of our clients to look on an IOPS Project of various devices, Local U320 SCSI, SAS, NFS, FC, iSCSI, SSD, Fusion-IO Drives, OCZ SSD PCIe cards.

Looking at various RAID 5, RAID 1 and RAID 10 configurations, with and without BBWC, CIFs and NFS access from clients, VM versus Physical Servers, we are still compiling data, but these are the some of the easier tools we are using to gather that data.

1. IOMeter
http://www.iometer.org/

2. HDTach
http://www.simplisoftware.com/Public/index.php?request=HdTach

3. DiskTT
http://www.snapfiles.com/get/disktt.html

4. CrystalDiskBenchMark
http://crystalmark.info/software/CrystalDiskMark/index-e.html

Feel free to use the above and gather stats for your datastores and storage.
Trevor HelpsTechnical Director

Author

Commented:
Hi Guys
Thanks for your input
To be more clear:

DS1 and DS2 are local datastores

DS1 is raid 1 mirror with 2 SAS 10k disks
DS2 is a single 7.2k SAS disk

They are all connected to HP Smart Array P410 controller

DS3 is NFS connected to a QNAP NAS via a single Gigabit ethernet.  I have nothing fancy configured on the switch .  There are 4 SATA disks in the NAS - RAID5

HOWEVER - my mistake .....
The issue is with DS2 - copying to it from DS1 or DS3 .  Its just slow.  I'm guessing now this is simply down to the fact it is a 7.2k  SAS compared with DS1 using 10k SAS .  It does seem to be significant though.
VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
Do you have any cache on the P410 ?

how have you configured it?

have you enabled Array Accelerator?

enabled Write Cache?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
do you have the battery for P410 ?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
do the benchmarks and enable cache.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
7.2k disks are slow, not great for performance.
Trevor HelpsTechnical Director

Author

Commented:
Hanccocka:

when i do the maths it probably works out right after all

12 Gb takes 25 mins to copy to the 7.2k    DS2 store
12 Gb takes 18 min to copy to the 10k DS1 store.

I have a cache board installed but no battery backup.

Where do i enable Array accelerator and Write cache ?  both in the ACU ?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Its dangerous to enable cache without battery, but you can ebale for test.

Yes, boot from SmartStart CDROM and run ACU.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
75% write
25% read

can make improvements.
Trevor HelpsTechnical Director

Author

Commented:
Thanks hancoccka

The array accelerator was disabled in the absence of a battery so i have enabled for testing and set ratio as you suggest.  I will purchase battery backup before going live.

There is also a setting for physical drive cache - this was also disabled.  Is this also protected by the battery backup or is this independent of that - should i enable this too?

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
enable as well, but all cache should be protected by a battery! to prevent corruption if power should fail.
Trevor HelpsTechnical Director

Author

Commented:
Oh My God

What a difference !    

Both copies of 12 Gb now down to 3 mins (previously 18 and 25 mins).  
And that's from NFS to local datastore

Local to Local - just 2.5 minutes.  Amazing

Thanks Hancoccka


Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
it surprising what Enabling can do!

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial