Question

slow file transfer over NFS

Asked by: td_miles

I have two HPUX boxes. They are on seperate subnets with a router between them.

I am trying to replicate some files from one box (old server) to another box (new server).

The process I am doing is as follows:

1. On old box, export the directory (exportfs -i -o root=newbox /dbs)
2. On new box, mount the directory (mount oldbox:/dbs /mnt/dbs)
3. On new box, copy data using rsync (/usr/local/bin/rsync -avP  /mnt/dbs/ /oldbox/dbs)

The problem is that the file copy is EXTREMELY slow.

If I look at a file that is copying now with rsync, it shows:

qad.cpio.Z
    27033600  78%   17.43kB/s    0:07:14

which shows that it's done 27MB (which is 78% of the file) at the rate of 17.43kB/s and the time remaining is 7 mins, 12 secs.

If I look at the interface on the router between the two subnets, it shows a transfer rate of approx 2.5Mbit/s through the interfaces.

I'm further constrained by the fact that what I really want to copy off is a Progress DB file and I have a limited window to do this when the DB is shutdown for backups during the night (about 90 mins). If I'm reading it correctly, the DB file is about 200MB in size:

du -k mfg.d1
200000  mfg.d1

At the rate of 17kB/s it's going to take forever.

So my question is, how would you suggest a better way of copying the data from old box to new. I don't have space on the old box to create a TGZ file and copy that across.

An option might be to restore the database from tape to the other server ? Is there a way to restore from tape to an NFS mount ? The tape drive in the old server is DDS3, so I would have to restore from the tape across the network to the new server.


edit: A transfer that I let run has just finished. The results are:

sent 540317467 bytes  received 268708 bytes  69569.03 bytes/sec
total size is 586467374  speedup is 1.08

Which shows a transfer rate of just under 70KBytes/sec over the whole transfer (584MB)

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2008-01-21 at 21:10:13ID23100236
Topics

HP-UX Unix

,

NFS File Server

,

Backup & Restore Software

Participating Experts
3
Points
500
Comments
16

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. shrink /usr on hpux
    Is it possible to shrink /usr on hpux? Can it be done remotely?
  2. mount -F nfs error
    I believe I'm doing a simple share and mount for NFS, but I get this error: # mount -F nfs server:/usr /mnt/usr nfs mount: server: : RPC: Program not registered nfs mount: retrying: /mnt/usr Platform is Solaris 2.6. Much thanks!
  3. remote tape
    Is there a way to remote mount a tape? At first thought, I would guess it's possible since we do this on CDROM all the time. But the tape won't work. Do I miss something?
  4. NFS permissions
    How do the file permissions for RH 6.0 differ from Slackware 4.0 in NFS? I have two boxes with ethernet cards that are not connected to the Internet. I don't have a "named" name server, I just put an entry in /etc/hosts. Under /etc/exports I add the following line o...
  5. mount HPUX drive on linux
    I have a SCSI drive with HPUX file system. I need to backup the content of the drive. Is there any way I can mount the HD using my linux box. Or is there any other way
  6. Performing HPUX backups without a tape drive
    I don't have a tape drive for my HPUX 11.00 machine so instead I would like to perform backups to an NFS mount. I am able to do this under Solaris using ufsdump/ufsrestore. On HPUX there are facilities to create backups using SAM but all the options require an actual tape d...

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: mikelfritzPosted on 2008-01-21 at 23:14:05ID: 20712438

rsize and wsize on the mount - some experimentation may be required.  There is a default - I don't know what it is on HPUX - but it may be too small or too big.  I'm not sure of the switch but I'd bet it's:
-o rsize=XXXX wsize=XXXX

 

by: td_milesPosted on 2008-01-21 at 23:26:42ID: 20712461

ok, I read the man page which says:

=======
Especially useful options include:
rsize=32768,wsize=32768
    This causes the NFS client to try to negotiate a buffer size up to the size specified. A large buffer size does improve performance, but both the server and client have to support it. In the case where one of these does not support the size specified, the size negotiated will be the largest that both support.
=======

So I used those values in my mount command. How do I know what value was negotiated ?

It "seems" to be running faster, but won't know until it hits a large file.

 

by: td_milesPosted on 2008-01-21 at 23:49:21ID: 20712529

is there a better way to transfer files from one machine to the other ?

I tried FTP, but got similarly dissapointing speeds:

226 Transfer complete.
1995529 bytes sent in 12.84 seconds (151.83 Kbytes/s)

How can I check whether the disks are being flogged on the old box ? It's quite a few years old now, so it could just be hardware is being maxed out by the disk copy (I think it has whopping 9GB SCSI disks in an external enclosure)

 

by: td_milesPosted on 2008-01-22 at 00:34:24ID: 20712670

I realised that if I just type "mount" then it tells me what parameters it used. It was already using 32768, so it was only perception that is was faster.

 

by: robocatPosted on 2008-01-22 at 04:43:05ID: 20713660


You need to find out if the bottleneck is on the local server or in the network. How long does it take to make a local copy of a file of, say 20MB ? How long does it take to backup the database (on tape ??) during the night ?

If the problem is in the network, you could first backup the database to tape using tar, then restore the same tape on the nfs mount using tar again.

 

by: mikelfritzPosted on 2008-01-22 at 05:46:08ID: 20714077

You could do the mount in the other direction - mount a dir on the new server to a point on the old and then do a compressed tar with  the target on the nfs mount.  You would be trying to use the horsepower of the old server in that case though.

Another thought would be the compress flag on the rsync - again, you would need to run it in the other direction (push from the old to the new) in order to see any benefit (which could be negated by the speed of the old server).

 

by: mikelfritzPosted on 2008-01-22 at 05:57:00ID: 20714154

You might need to experiment with the rsize/wsize and specify them directly,  bigger is not always better.  You could spec them at 2K 4K 8K 16K... and test speed at each level.

This site is meant for Linux, but the speed test stuff is applicable for you:

http://nfs.sourceforge.net/nfs-howto/ar01s05.html

 

by: tfewsterPosted on 2008-01-22 at 11:50:17ID: 20717601

Actually, your copy speed with FTP is about 10x the speed rsync is reporting ;-)  So FTPing the 200Mb file would take about 20 minutes

But basically, your network connection is slow or very busy or misconfigured; Check that both HP boxes have their network cards running Full Duplex (`ifconfig lanN`, where N is the appropriate interface - use `lanscan` to see the list)

Or restore from tape...
That removes your time constraint of the 90 minute window and you should be able to restore to a different directory (e.g. an NFS mount) as long as files are backed up with _relative_ pathnames, e.g.
# create backup
cd /data; tar -cvf /dev/rmt/0m .
# restore to different directory
cd /mnt/recovery_dir; tar -xvf /dev/rmt/0m  (/mnt/recovery_dir could be an NFS mount from the old system)

Or compress on-the-fly to reduce the traffic across the network...
cd /data;for FILE in *; do gzip -c $FILE > /mnt/recovery_dir/$FILE.gz

 

by: td_milesPosted on 2008-01-22 at 12:18:46ID: 20717874

thanks for all the comments, I'll try to address all of the questions that have been asked.

The backup takes about 90 minutes. I'm not sure how much data is being written to tape. The result from the backup command is:

fbackup(3056): total blocks written to output file /dev/rmt/0mb: 23721866

If the blocks were 1KB each, then that is 23.7GB onto the tape. Considering it's a DDS3 tape, thats fairly impressive. If the blocks are half the size (ie. 512 bytes) then it's 12GB onto the tape, which is probably a lot more reasonable.

I did check, network card on old & new server are both set to 100 full. As I said, I looked at the stats on the router that joins the two networks and the utilisation of the interfaces is only 2-3 Mbps max.

I can't install stuff on the old box, so rsync pushed from old to new isn't an option, I did think about this so that I could use compression with rsync. I can try a file copy the other way though.

I'm not sure about doing a compress/zip on the files using the old server. When I say old, I mean OLD. I'm led to believe it's probably 8-10 years old, so using CPU on it for compression may not give much improvement.

On restoring from tape, as you can see from the above, the backup is being done using "fbackup". From what I can see the equivalent utility is "frecover" to restore the backups. Can anyone give advice on whether I will be able to restore the backup to an NFS mount with this utility ?

I've inherited this situation and I pretty much can't change anything on the old box. I'm a linux person (RHCE), so I don't know too much about HPUX, but luckily most of this isn't HPUX specific stuff.

I'll be trying to restore from tape to an NFS mount today. I'll update this when I either manage to trash the old system or successfully get the parameters of frecover correct and restore to a different path ;)

 

by: tfewsterPosted on 2008-01-22 at 14:23:12ID: 20719086

cd /mnt/recovery_dir;frecover -xXf /dev/rmt/0mb

Test it with `-i /file/that/can/be/safely/overwritten`

 

by: td_milesPosted on 2008-02-05 at 15:23:07ID: 20828208

Just an update, I haven't had a chance to test the restore from tape to an NFS mount. I'm still planning on it, just that another project has been shoved in front of the HPUX migration and so it will have to wait for another week or so. Thanks for your patience, I will update again when I have more to report.

 

by: td_milesPosted on 2008-02-07 at 16:22:58ID: 20846808

I ran frecover with the options given and it says:

frecover -xXf /dev/rmt/0mb -i /usr/local/log/shutdown.msg
frecover(1024): can't create directory ./local
frecover(1024): can't create directory ./usr/local
frecover(1015): cannot chdir to directory ./usr/local
frecover(1024): can't create directory ./usr/local
frecover(1024): can't create directory ./usr/local/log
frecover(1015): cannot chdir to directory ./usr/local/log

Which tells me that it's not creating the directory tree structure for where it wants to put the file. Is there a way of making frecover create the directory structure as it goes ?

For my test file (/usr/local/log/shutdown.msg) I've just created the directory for it and it worked fine after that, but I don't want to have to do that for everything.

 

by: td_milesPosted on 2008-02-07 at 19:35:22ID: 20847611

I suspect this has something to do with NFS and permissions and the like, but I'm not totally sure. When I run the full restore it is able to create some of the file structures, but not the others ?

I am exporting with:
/oldbox -anon=65534,rw=oldbox

 

by: td_milesPosted on 2008-02-17 at 21:34:21ID: 20917809

I managed to get my restore across NFS working, but it was still boringly slow.

Even when I connected the second NIC from newbox onto the same LAN as the oldbox (and disconnected the other NIC, so it could only be using the correct one), I still got really SLOW restore speeds.

I was restoring using the frecover as suggested above and to restore a fraction of the tape (a DB backup) still took about twice as long as it takes for the full backup to run and write the data to the tape in the first place.

Seems like the slowness is being caused by writing the files across NFS still ?

Any other suggestions ?

 

by: mikelfritzPosted on 2008-03-13 at 21:25:45ID: 21123065

My experience is that "pushing" with NFS is slow (permissions?) while "pulling"  is quite snappy.  I think, when "pushing" it checks to see if you are allowed before letting you copy each file, while if you are pulling you, by default, have permission. I already (half) stated that above...

 

by: td_milesPosted on 2008-03-13 at 22:11:24ID: 21123179

It didn't seem to make any difference whether it was push or pull so I have to simply conclude that something in the NFS on the old box was causing it to run like a dog.

The DDS3 tape drive in the old box died, so we've had to hook an LTO drive up to it in the meantime and we'll just use that to do the transfer as the new box has an LTO3 drive.

Thanks to all for their help, if nobody objects I'll split the point up between all who've assisted. I don't like not having a proper answer, but it's time to move on so that we can finish the migration to the new box and let the old box RIP.

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...