[x]
Posted via EE Mobile

Search, ask, and monitor your questions on the go with EE Mobile. Visit Experts Exchange from your mobile device and never be out of touch again.

10/22/2009 at 07:39PM PDT, ID: 24836832 | Points: 500
[x]
Attachment Details

VMware ESX and NFS problems

Asked by nibowl in NFS File Server, VMware, Linux

Tags: VMware, ESX, Linux, NFS, dd, scp

I have recently setup a VMware ESX 3.5 environment consisting of two hosts.  They have an NFS Datastore that is served up via NFS from a physical Linux server with a RAID 10 of 4 1 TB SATA drives. Each of the VM's (4 total) also connect to an NFS share off of the same RAID 10 (though on a different LVM logical volume).  The VM's are all RHEL 5 as well as the NFS server.

I can scp files between VM's at 20-30 MB/s, scp files from the Datastore (VM's local storage) to the NFS mount at 20-30 MB/s and back again at the same speed.  I can scp files back and forth to the NFS server directly at 20-30 MB/s.  I can dd if=/dev/zero at greater than 100 MB/s to either the Datastore or the NFS mount.  However, if I dd if=/dev/urandom it pegs out at a max of roughly 5-6 MB/s.  The same occurs with the dd using urandom to an NFS volume on a 0 NetApp filer as well from the same VM's.  With the VM that has 4 vCPU's and 24 GB or RAM all of those 20-30's go up to about 60 MB/s but no higher.

It seems that anything actually generating new non-zero data hits a wall.

dd with /dev/zero to NFS mount:
dd if=/dev/zero of=/storage/testfile2 bs=8k count=10000
10000+0 records in
10000+0 records out
81920000 bytes (82 MB) copied, 1.17408 seconds, 69.8 MB/s

dd with /dev/zero to Datastore:
dd if=/dev/zero of=/testfile1 bs=8k count=10000
10000+0 records in
10000+0 records out
81920000 bytes (82 MB) copied, 0.446165 seconds, 184 MB/s

dd with /dev/urandom to NFS mount:
dd if=/dev/urandom of=/storage/testfile3 bs=8k count=10000
10000+0 records in
10000+0 records out
81920000 bytes (82 MB) copied, 16.4113 seconds, 5.0 MB/s

dd with /dev/urandom to Datastore:
dd if=/dev/urandom of=testfile2 bs=8k count=10000
10000+0 records in
10000+0 records out
81920000 bytes (82 MB) copied, 15.3982 seconds, 5.3 MB/s

scp from Datastore to NFS mount:
scp testfile2 XXX.XXX.XXX.XXX:/storage/testfile3
testfile2                                                                                                                              100%   78MB  26.0MB/s   00:03

scp from NFS mount to Datastore:
scp XXX.XXX.XXX.XXX:/storage/testfile3 testfile5
testfile3                                                                                                                              100%   78MB  26.0MB/s   00:03

nfsstat -c:
nfsstat -c
Client rpc stats:
calls      retrans    authrefrsh
143587     0          0

Client nfs v3:
null         getattr      setattr      lookup       access       readlink
0         0% 7647      5% 34        0% 71439    49% 6473      4% 0         0%
read         write        create       mkdir        symlink      mknod
1017      0% 45415    31% 35        0% 0         0% 0         0% 0         0%
remove       rmdir        rename       link         readdir      readdirplus
37        0% 0         0% 0         0% 0         0% 719       0% 10075     7%
fsstat       fsinfo       pathconf     commit
4         0% 396       0% 0         0% 97        0%
[+][-]10/22/09 07:53 PM, ID: 25641047

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]10/22/09 08:14 PM, ID: 25641130

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]10/23/09 05:26 AM, ID: 25643673

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
 
Loading Advertisement...
20090824-EE-VQP-74 - Hierarchy / EE_QW_3_20080625