July 24, 2008 01:39am pdt

1. omarfarid 20,245
2. JustUNIX 14,250
3. tfewster 11,920
4. Tintin 8,523
5. sunnycoder 5,950
6. ahoffmann 4,000
7. robocat 3,900
8. dvz 3,350
9. ozo 3,100
10. sentner 3,000
11. mikelfritz 2,680
12. sjm_ee 2,450
13. agriesser 2,400
14. cflong 2,150
15. amit_g 2,000
15. joebednarz 2,000
15. CynepMeH 2,000
15. ac24 2,000
15. henjoh09 2,000

1. yuzh 91,545
2. tfewster 75,137
3. omarfarid 30,473
4. chris_ca... 26,089
5. ahoffmann 22,192
6. Tintin 21,426
7. gheist 17,123
8. amit_g 16,198
9. JustUNIX 15,270
10. sunnycoder 13,244
11. ozo 11,886
12. JJSmith 10,275
13. giltjr 8,764
14. Kdo 7,444
15. Panjandrum 5,852

01.21.2008 at 09:10PM PST, ID: 23100236

	[x] Attachment Details

slow file transfer over NFS

Asked by td_miles in HP-UX Unix, NFS File Server, Backup & Restore Software

I have two HPUX boxes. They are on seperate subnets with a router between them.

I am trying to replicate some files from one box (old server) to another box (new server).

The process I am doing is as follows:

1. On old box, export the directory (exportfs -i -o root=newbox /dbs)
2. On new box, mount the directory (mount oldbox:/dbs /mnt/dbs)
3. On new box, copy data using rsync (/usr/local/bin/rsync -avP /mnt/dbs/ /oldbox/dbs)

The problem is that the file copy is EXTREMELY slow.

If I look at a file that is copying now with rsync, it shows:

qad.cpio.Z
27033600 78% 17.43kB/s 0:07:14

which shows that it's done 27MB (which is 78% of the file) at the rate of 17.43kB/s and the time remaining is 7 mins, 12 secs.

If I look at the interface on the router between the two subnets, it shows a transfer rate of approx 2.5Mbit/s through the interfaces.

I'm further constrained by the fact that what I really want to copy off is a Progress DB file and I have a limited window to do this when the DB is shutdown for backups during the night (about 90 mins). If I'm reading it correctly, the DB file is about 200MB in size:

du -k mfg.d1
200000 mfg.d1

At the rate of 17kB/s it's going to take forever.

So my question is, how would you suggest a better way of copying the data from old box to new. I don't have space on the old box to create a TGZ file and copy that across.

An option might be to restore the database from tape to the other server ? Is there a way to restore from tape to an NFS mount ? The tape drive in the old server is DDS3, so I would have to restore from the tape across the network to the new server.

edit: A transfer that I let run has just finished. The results are:

sent 540317467 bytes received 268708 bytes 69569.03 bytes/sec
total size is 586467374 speedup is 1.08

Which shows a transfer rate of just under 70KBytes/sec over the whole transfer (584MB)Start Free Trial

Keywords: slow file transfer over NFS

	Title
1	Performing HPUX backups without a ta…
2	NFS MOUNT GIVES UNKNOWN ERROR
3	mount HPUX drive on linux
4	shrink /usr on hpux
5	NFS monitoing script or tool
6	mount -F nfs error

Loading Advertisement...

Microsoft

Internet Protocols
Applications

Development
OS

Hardware
Windows Security

Apple

Operating Systems
Hardware

Programming
Networking

Software

Internet

Search Engines
File Sharing
WebTrends / Stats
Spy / Ad Blockers

Web Browsers
New Net Users
Web Development
Chat / IM

Anti Spam
Web Servers
Anti-Virus
Email Clients

Gamers

Tips
Online / MMORPG
Puzzle
Emulators

Action / Adventure
Role Playing
Consoles
Game Programming

Strategy
Sports
Misc
Computer Games

Digital Living

Hardware
Automotive
New Net Users
New Users

Software
Digital Music
Gaming World
Home Security

Apple
Networking Hardware

Virus & Spyware

Vulnerabilities
IDS
Encryption
Anti-Virus

Operating Systems Security
Software Firewalls
WebApplications
Cell Phones

Operating Systems
Internet
Hardware Firewalls

Hardware

Displays / Monitors
Handhelds / PDAs
Components
Peripherals
Laptops/Notebooks

Servers
Misc
Apple
Embedded Hardware
Networking Hardware

Storage
Desktops
New Users

Software

System Utilities
Industry Specific
Network Management
Photos / Graphics
Page Layout
VMware
Misc
Web Development

OS
CYGWIN
Voice Recognition
Virtualization
Message Queue
Quality Assurance
Security
Firewalls

MultiMedia Applications
Development
Database
Office / Productivity
Business Management
OS/2 Apps
Server Software
Internet / Email

ITPro

OS
Storage
Encryption
Operating Systems Security
Apple Hardware
Laptops & Notebooks
Servers
Networking Hardware
Peripherals
Devices

Displays / Monitors
WebTrends / Stats
Search Engines
Firewalls
Web Computing
WebApplications
IDS
Vulnerabilities
Email Clients
File Sharing

Spy / Ad Blockers
Web Browsers
Web Servers
Networking
Anti-Virus
Consulting
Chat / IM
Anti Spam

Developer

Web Servers
Web Browsers
Game Programming
Dev Tools
Industry Specific
Office / Productivity

Database
CYGWIN
Web Development
Search Engines
File Sharing
WebTrends / Stats

Programming
Content Management
Application Servers
Protocols

Storage

Removable Backup Media
Storage Technology
Servers

Grid
Remote Access
Backup / Restore

Misc
Hard Drives

Miscellaneous
Security
Development
Linux
VMware

MainFrame OS
Unix
Apple
OS / 2
AS / 400

BeOS
Microsoft
VMS / OpenVMS

Database

Oracle
Miscellaneous
MySQL
Software
Sybase
Contact Management
PostgreSQL
Data Manipulation
Clarion
InterSystems Cache

Siebel
MUMPS
OLAP
SQLBase
SAS
GIS & GPS
4GL
Berkeley DB
DB2
Informix

Interbase / Firebird
FoxPro
Reporting
LDAP
Filemaker Pro
MS SQL Server
dBase
MS Access

Security

Misc
Web Browsers
Software Firewalls
Operating Systems Security
File Sharing

Spy / Ad Blockers
Vulnerabilities
WebApplications
IDS
Anti-Virus

Encryption
Anti Spam
Email Clients
VPN
Chat / IM

Programming

Editors IDEs
Installation
Handhelds / PDAs
Multimedia Programming
System / Kernel
Automation

Algorithms
Game
Signal Processing
Project Management
Open Source
Database

Misc
Languages
Processor Platforms
Theory

Web Development

Scripting
Blogs
Web Servers
Software
Search Engines
Web Graphics
Web Services

Images
Internet Marketing
Images and Photos
Components
Document Imaging
Web Languages/Standards
Illustration

WebApplications
Fonts
WebTrends / Stats
Authoring
Digital Camera Software
Miscellaneous

Networking

Protocols
Apple Networking
Network Management
Message Queue
Application Servers
Content Management
File Servers
Email Servers
Misc
Java Editors & IDEs

Wireless
Networking Hardware
Backup / Restore
System Utilities
ISPs & Hosting
Web Servers
Storage Technology
Removable Backup Media
Servers
Web Computing

Broadband
Grid
OS / 2
Novell Netware
Unix Networking
Windows Networking
Security
Telecommunications
Operating Systems
Linux Networking

Other

Lounge
Business Travel
Community Support
New Net Users

Philosophy / Religion
Math / Science
Miscellaneous
URLs

Expert Lounge
Politics
Puzzles / Riddles
Automotive

Community Support

Suggestions
New to EE
New Topics
CleanUp

Announcements
General
Feedback

Input
EE Bugs

01.21.2008 at 11:14PM PST, ID: 20712438

mikelfritz:

rsize and wsize on the mount - some experimentation may be required. There is a default - I don't know what it is on HPUX - but it may be too small or too big. I'm not sure of the switch but I'd bet it's:
-o rsize=XXXX wsize=XXXX

Assisted Solution

01.21.2008 at 11:26PM PST, ID: 20712461

td_miles:

ok, I read the man page which says:

=======
Especially useful options include:
rsize=32768,wsize=32768
This causes the NFS client to try to negotiate a buffer size up to the size specified. A large buffer size does improve performance, but both the server and client have to support it. In the case where one of these does not support the size specified, the size negotiated will be the largest that both support.
=======

So I used those values in my mount command. How do I know what value was negotiated ?

It "seems" to be running faster, but won't know until it hits a large file.

01.21.2008 at 11:49PM PST, ID: 20712529

td_miles:

is there a better way to transfer files from one machine to the other ?

I tried FTP, but got similarly dissapointing speeds:

226 Transfer complete.
1995529 bytes sent in 12.84 seconds (151.83 Kbytes/s)

How can I check whether the disks are being flogged on the old box ? It's quite a few years old now, so it could just be hardware is being maxed out by the disk copy (I think it has whopping 9GB SCSI disks in an external enclosure)

01.22.2008 at 12:34AM PST, ID: 20712670

td_miles:

I realised that if I just type "mount" then it tells me what parameters it used. It was already using 32768, so it was only perception that is was faster.

01.22.2008 at 04:43AM PST, ID: 20713660

robocat:

You need to find out if the bottleneck is on the local server or in the network. How long does it take to make a local copy of a file of, say 20MB ? How long does it take to backup the database (on tape ??) during the night ?

If the problem is in the network, you could first backup the database to tape using tar, then restore the same tape on the nfs mount using tar again.

Assisted Solution

01.22.2008 at 05:46AM PST, ID: 20714077

mikelfritz:

You could do the mount in the other direction - mount a dir on the new server to a point on the old and then do a compressed tar with the target on the nfs mount. You would be trying to use the horsepower of the old server in that case though.

Another thought would be the compress flag on the rsync - again, you would need to run it in the other direction (push from the old to the new) in order to see any benefit (which could be negated by the speed of the old server).

01.22.2008 at 05:57AM PST, ID: 20714154

mikelfritz:

You might need to experiment with the rsize/wsize and specify them directly, bigger is not always better. You could spec them at 2K 4K 8K 16K... and test speed at each level.

This site is meant for Linux, but the speed test stuff is applicable for you:

http://nfs.sourceforge.net/nfs-howto/ar01s05.html

01.22.2008 at 11:50AM PST, ID: 20717601

tfewster:

Actually, your copy speed with FTP is about 10x the speed rsync is reporting ;-) So FTPing the 200Mb file would take about 20 minutes

But basically, your network connection is slow or very busy or misconfigured; Check that both HP boxes have their network cards running Full Duplex (`ifconfig lanN`, where N is the appropriate interface - use `lanscan` to see the list)

Or restore from tape...
That removes your time constraint of the 90 minute window and you should be able to restore to a different directory (e.g. an NFS mount) as long as files are backed up with _relative_ pathnames, e.g.
# create backup
cd /data; tar -cvf /dev/rmt/0m .
# restore to different directory
cd /mnt/recovery_dir; tar -xvf /dev/rmt/0m (/mnt/recovery_dir could be an NFS mount from the old system)

Or compress on-the-fly to reduce the traffic across the network...
cd /data;for FILE in *; do gzip -c $FILE > /mnt/recovery_dir/$FILE.gz

Accepted Solution

01.22.2008 at 12:18PM PST, ID: 20717874

td_miles:

thanks for all the comments, I'll try to address all of the questions that have been asked.

The backup takes about 90 minutes. I'm not sure how much data is being written to tape. The result from the backup command is:

fbackup(3056): total blocks written to output file /dev/rmt/0mb: 23721866

If the blocks were 1KB each, then that is 23.7GB onto the tape. Considering it's a DDS3 tape, thats fairly impressive. If the blocks are half the size (ie. 512 bytes) then it's 12GB onto the tape, which is probably a lot more reasonable.

I did check, network card on old & new server are both set to 100 full. As I said, I looked at the stats on the router that joins the two networks and the utilisation of the interfaces is only 2-3 Mbps max.

I can't install stuff on the old box, so rsync pushed from old to new isn't an option, I did think about this so that I could use compression with rsync. I can try a file copy the other way though.

I'm not sure about doing a compress/zip on the files using the old server. When I say old, I mean OLD. I'm led to believe it's probably 8-10 years old, so using CPU on it for compression may not give much improvement.

On restoring from tape, as you can see from the above, the backup is being done using "fbackup". From what I can see the equivalent utility is "frecover" to restore the backups. Can anyone give advice on whether I will be able to restore the backup to an NFS mount with this utility ?

I've inherited this situation and I pretty much can't change anything on the old box. I'm a linux person (RHCE), so I don't know too much about HPUX, but luckily most of this isn't HPUX specific stuff.

I'll be trying to restore from tape to an NFS mount today. I'll update this when I either manage to trash the old system or successfully get the parameters of frecover correct and restore to a different path ;)

01.22.2008 at 02:23PM PST, ID: 20719086

tfewster:

cd /mnt/recovery_dir;frecover -xXf /dev/rmt/0mb

Test it with `-i /file/that/can/be/safely/overwritten`

02.05.2008 at 03:23PM PST, ID: 20828208

td_miles:

Just an update, I haven't had a chance to test the restore from tape to an NFS mount. I'm still planning on it, just that another project has been shoved in front of the HPUX migration and so it will have to wait for another week or so. Thanks for your patience, I will update again when I have more to report.

02.07.2008 at 04:22PM PST, ID: 20846808

td_miles:

I ran frecover with the options given and it says:

frecover -xXf /dev/rmt/0mb -i /usr/local/log/shutdown.msg
frecover(1024): can't create directory ./local
frecover(1024): can't create directory ./usr/local
frecover(1015): cannot chdir to directory ./usr/local
frecover(1024): can't create directory ./usr/local
frecover(1024): can't create directory ./usr/local/log
frecover(1015): cannot chdir to directory ./usr/local/log

Which tells me that it's not creating the directory tree structure for where it wants to put the file. Is there a way of making frecover create the directory structure as it goes ?

For my test file (/usr/local/log/shutdown.msg) I've just created the directory for it and it worked fine after that, but I don't want to have to do that for everything.

02.07.2008 at 07:35PM PST, ID: 20847611

td_miles:

I suspect this has something to do with NFS and permissions and the like, but I'm not totally sure. When I run the full restore it is able to create some of the file structures, but not the others ?

I am exporting with:
/oldbox -anon=65534,rw=oldbox

02.17.2008 at 09:34PM PST, ID: 20917809

td_miles:

I managed to get my restore across NFS working, but it was still boringly slow.

Even when I connected the second NIC from newbox onto the same LAN as the oldbox (and disconnected the other NIC, so it could only be using the correct one), I still got really SLOW restore speeds.

I was restoring using the frecover as suggested above and to restore a fraction of the tape (a DB backup) still took about twice as long as it takes for the full backup to run and write the data to the tape in the first place.

Seems like the slowness is being caused by writing the files across NFS still ?

Any other suggestions ?

03.13.2008 at 09:25PM PDT, ID: 21123065

mikelfritz:

My experience is that "pushing" with NFS is slow (permissions?) while "pulling" is quite snappy. I think, when "pushing" it checks to see if you are allowed before letting you copy each file, while if you are pulling you, by default, have permission. I already (half) stated that above...

03.13.2008 at 10:11PM PDT, ID: 21123179

td_miles:

It didn't seem to make any difference whether it was push or pull so I have to simply conclude that something in the NFS on the old box was causing it to run like a dog.

The DDS3 tape drive in the old box died, so we've had to hook an LTO drive up to it in the meantime and we'll just use that to do the transfer as the new box has an LTO3 drive.

Thanks to all for their help, if nobody objects I'll split the point up between all who've assisted. I don't like not having a proper answer, but it's time to move on so that we can finish the migration to the new box and let the old box RIP.

20080716-EE-VQP-32 / EE_QW_2_20070628

slow file transfer over NFS

Asked by td_miles in HP-UX Unix, NFS File Server, Backup & Restore Software

About this solution