Link to home
Start Free TrialLog in
Avatar of coanda
coanda

asked on

Bizarre decompression performance over network file system

My file server is a virtual machine that receives its storage disk from the virtual machine host via iSCSI, both the guest and host are Debian 6. The file server uses NFS to export to clients, and the test client uses AutoFS, the OS is Debian 7 RC3.

Decompression performance over the network is very bad, but what's even worse is that during an extract the entire client basically locks up. The following are some tests done on a multi-archive rar that is around 500 MB in size total, in order they represent the extract over NFS, the copy to local, the extract from a local location:
$ time rar e multi_archive-test.part1.rar

real	2m44.077s
user	0m7.764s
sys	0m0.376s

$ time cp multi_archive-test.part*.rar /tmp/

real	0m1.466s
user	0m0.004s
sys	0m0.216s

$ cd /tmp
$ time rar e multi_archive-test.part1.rar

real	0m8.615s
user	0m6.740s
sys	0m0.460s

Open in new window


The relevant line in my /etc/exports file is:
/srv/fs/data	10.4.0.0/24(rw,sync,no_subtree_check,no_root_squash)

Open in new window


The client mounts via AutoFS with this line in the relevant file:
/mnt/data	-fstype=nfs,rw,soft,intr,nosuid,nodev,tcp,retry=10,rsize=32768,wsize=32768	fsrv:/srv/fs/data

Open in new window


I have tested the same setup with Arch Linux as both the client and the server, and with Fedora core 16 as the client. Couple of other things worth mentioning are that I've tested this with a physical server that was not using iSCSI or any virtualization and had the same performance issues, and my router is an Asus RT-N16 running the Tomato firmware version 1.28.

It would be really unfortunate if the Linux rar packages were to blame for the poor NFS performance, I'd much rather it be something I could track down and fix but I'm having a hard time figuring that out.

Thanks.
ASKER CERTIFIED SOLUTION
Avatar of arnold
arnold
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of coanda
coanda

ASKER

Memory usage on both the server and client during the test were not anywhere near 100%, they didn't even exceed 10%. This test was done with 500 MB of files, but the same thing happens for 100 MB, it's just worse with a multi-file archive and the one that I happened to have available was 500 MB.
Avatar of coanda

ASKER

Just in case I decided to bump the RAM up on my file server and run another test. The server was given 2 GB total and the client has 16 GB, during the test the server never exceeded 100 MB, and the client had 14 GB available. Also, CPU usage was never close to max on either.

Just for kicks I tried the same test using a Samba share, the time of the decompress over the network was almost 100% worse, but my client machine didn't completely freeze up during like it does over NFS.
When you decompress where are the uncompressed data being written to?

Look at I/O traffic and network usage.

Unfamiliar with rar options, do you have an option to output the table of contents from the archive?
Avatar of coanda

ASKER

1. same path
2. the bandwidth is lower than I'd expect, and network usage seems relatively low
3. it doesn't seem to, but the archive I'm using is only a single video file (mp4) split into a multi-file archive for testing so a TOC wouldn't be very interesting
Not sure what the purpose of your test, but binary files such as images, movies, etc are not compressable.  Try data archive of text, etc files, and see whether that alters the performance.

What about I/O usage.
You would do best to make a local copy of the compressed file and decompress that. Decompression involves numerous seeks back to earlier parts of the compressed file to extract duplicated strings. Perhaps NFS caching is not handling this well.
Avatar of coanda

ASKER

@arnold 95% of the content that I downloaded as rar files is video data, and besides that I've already shown that it has nothing to do with the contents of the archive, the local unrar was 1/20 what it was over the network. Are you asking about I/O usage between the client and server under normal operation or just during the test? Normal operation I get 900+ Mbps using a basic TCP test with iperf, during the decompress the bandwidth would be = (525 * 8)/166 = 25 Mbps.

@duncan_roe Yes, based on what I've found it would be faster to copy the files and decompress them locally, but I don't want to do that. What if the issue were for 1000 users on a corporate network, would you tell them all to copy the file locally? Because I wouldn't, I'd try to find a reasonable solution. NFS caching is disabled by default, I added it with the fsc mount option but that didn't really help.
See if rar has an option to specify the destination to where the file should be expanded/extracted.
Avatar of coanda

ASKER

It does, but I have very little interest in a solution that involves extracting the contents of an archive that is on a server to the local drive of the client. The extracted data should stay in the same place as the archive.
Then you should run the extraction on the server where the archive is local versus via NFS.
When extracting on NFS share, you have the data being read in into rar over the NFS share and the results are then written back to the NFS share.
Avatar of coanda

ASKER

I'm really more interested in getting help and information for diagnosing what the issue is. I simply don't buy that I shouldn't be able to do this with reasonable performance, and I'm not going to accept as a solution any response that doesn't include, or hasn't included as part of a thread, some kind of recommendation that involves altering configurations or executing commands as part of deriving a conclusion.

Thanks for your help though.
I can't find an fsc option in man mount. To what are you referring? Local decompression without NFS caching will definitely be bad news, as I explained earlier.
Avatar of coanda

ASKER

https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/fscachenfs.html

No errors occurred when I reconnected the mounts so I assumed that they are valid options even in Debian.
The issue is that rar reads from the archive which is stored on NFS, it then writes the results back to the same NFS share.

1) NFS extract takes X
2) local drive for both data and extract is .1X
3) you tested performance based on a samba share
The solution and the explanation is clear

You can look at which NFS version your client/server use. NFS over TCP, You could try increasing the NFS performance by adjusting wsize, rsize on both server/client as well as check whether those options are conflicting with the switch capacity.

At some point, when trying to fit a square peg in a round hole, suggestion to try something else or alter what you have is a generally acceptable response I.e reduce the size of the peg or enlarge the hole. You seem to be insistent on your approach even though you confirmed that the issue of slow performance is affected by where the data is stored!
fsc looks to be a Red Hat "extension". man 8 mount redirected me to man 5 nfs for the nfs mount options - plenty of discussion of caching and a number of options as well, but fsc is not among them. This is on Slackware 14 released last month.
Avatar of coanda

ASKER

@arnold If what you're saying is true then the theoretical time to receive (a), extract (b), and write (c) would be : t = a + b + c, when in fact it's something more like : t = 12 * (a + b + c).

But that's not really what I was intending to ask in the first place. Sure, it sucks that it takes a long time, but as I stated in the beginning with "performance over the network is very bad, but what's even worse is that during an extract the entire client basically locks up" the real issue is that the client system becomes unusable during operation. Which shouldn't happen, which is why I want to diagnose the issue, not circumvent it.

Sometimes the solution to a problem is changing the shape of the hole, not picking up a different peg. Maybe I seem insistent to stay with the approach that I am, but the approach that I want to take is not an unreasonable one, and like I suggested in a previous post if this problem were happening to a network of several users who all needed the same functionality you wouldn't just tell them that doing something in a completely reasonable fashion is impossible/dumb/incorrect/misguided. All I'm trying to do is have my computer not to hang for the duration of a decompress operation, that seems like something that the client and server should be able to handle, and if it can't there should be a way to troubleshoot why it can't. Further to that, I have on several occasions run a Samba share and done the same operation on a Windows client with winrar and it has handled it fine, didn't lock up the computer, time to extract was reasonable, etc. So you're going to tell me that Windows and CIFS are better? Or is it still unreasonable to try and do what I am with NFS and Linux?

@duncan_roe You said that decompression without caching will be bad, how can I verify that I'm using caching and how can I quantify how well it's doing its job? Also, if you didn't know about the fsc option what other way of controlling caching are you aware of?
Check the I/O to disk on the NFS server.
Setup cacti.net if you do not have it already, it can SNMP poll client and server and display what is going on in a graphical manner.
Could you try adding NFS support to a knows system and then test the performance of winrar.

You could on the client setup a cron job that would collect data every minute simultaneously
top -n 1
Vmstat
Iostat
etc.
As well as on the server
This way you can start your extraction
Then compare the results. With this though it will take some effort to match the data.
Avatar of coanda

ASKER

Thanks for the recommendations, I'm running iostat and vmstat on the client and server and plan to have it running for a while so I'll have to post the results back tomorrow some time.

I've setup cacti but it doesn't seem to want to collect SNMP information for the remote devices despite all of the MIBs and snmpwalk on the community functioning correctly. Go figure. If I can get that working it looks like it could yield useful data.

One last thing I plan to run this evening is the iozone benchmark, the only problem is that it seems to hang the client system just as bad as when I'm decompressing a rar file. So it seems that the rar/extract isn't the only culprit, it just helped uncover the problem.

Thanks a lot for the suggestions, I'll post the results back when I have some.
When you configure snmp, make sure you authorize the IP of the server on which cacti is running within the snmpd.conf file as a manager.
On. Linux, there is a comsec where you need to define as authorized to get more than the snmp message that itis not configured.
Use
 snmpwalk -v 2c -c "community name" remoteip
See what you get in response.
There is an online copy of the nfs(5) man page at http://linux.die.net/man/5/nfs
If you get your browser to search for cache and enable highlight all, you will see references throughout. You may or may not find the page useful: it documents standard NFS.
Avatar of coanda

ASKER

It's all working from an SNMP standpoint, as mentioned I can walk from any host to any other remote host and receive its list of MIBs. What happens in cacti though is that after adding the host I get an unknown status, and no data is graphed for any other than localhost. It doesn't matter though because I came across CactiEZ and it seems to work fine. Thanks.
Avatar of coanda

ASKER

I completely forgot about this question and I guess EE doesn't send out email anymore to remind you of old questions.

After upping the memory in the server to 16 GB there was minimal improvement but after some reading I found that using iSCSI with virtual machines can be affected by the block size, possibly because of jumbo frames I can't really remember, so I changed that and it worked a little better. Still though, extracting data from an NFS client is pretty bad.

I don't think this is actually solved but I'm done with it for now so I'm just going to accept the suggestion about adding memory.
Avatar of coanda

ASKER

None.