Speed up NIC / Rsync transfer rate on N54L Ubuntu Server

Some new info that might help

NASShared:/# iperf -c 192.168.99.221 -r -f M
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.25 MByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.99.221, TCP port 5001
TCP window size: 0.02 MByte (default)
------------------------------------------------------------
[  7] local 192.168.99.8 port 3198 connected with 192.168.99.221 port 5001
[  7]  0.0-10.0 sec    112 MBytes  11.2 MBytes/sec
[  6] local 192.168.99.8 port 5001 connected with 192.168.99.221 port 34845
[  6]  0.0-10.1 sec    113 MBytes  11.2 MBytes/sec

Select all Open in new window

I also tried to do a copy with SCP and only got 1.4 MB/s too :(
I don't get how I can get a 11.2MB/s speed with iperf/ network tools, but only 1.4MB/s with copy?

I think there must be a config issue or a drive speed issue?

Thanks in advance.

what type of switch port is this server connected?  it has 1G capacity but is operating at 100M.

you might have to hard code speed and duplex in the switch.  be careful, though, changing these settings may cause the port to reset.

I agree with Jan.  On the port reset; even if it does (likely will) you said you are using rsync so it should not have to copy what it already did - it will start where it left off within reason.

The question remains on what type of switch is connected and if both machines are connected at 1G... We know this one is not.

Once you know the switch supports 1G, make sure it's configured to use 1G and you can use something like this to force 1G

# ethtool -s em1 speed 1000 duplex full

the NIC should automatically negotiate.

Hi Jan and Mike, thanks for your comments.
The switch is a Cisco 10/100 48 port switch, something like this

I have tried connecting the N54L (and the ReadyNas) to a relatively cheap Dlink Gigabit switch, but the guy that gave me the old enterprise Cisco switch suggested that the throughput on it is better than a lower end Gigabit switch, so I went back.

The fact that I am getting almost 10x the speed when doing the iperf test, suggests it is something else to me.  What do you think?

if this is a 10/100 port then you will not get GigE speeds.

and, yes, check the destination machine NIC and processes.

Thank Jan.  I am not expecting GigE speeds, but 1.4MB/s is really slow for 100 full duplex.

I have run the iperf on both machines.
I think it might be rsync, or some other configuration?

What kind of data and what rsync switches?

Getting a gigE connection would help in theory, but if you are right then it won't do much.

Mike, I am transferring 5TB of mostly media files - backing up my old Media Server.
Large files 300MB+

not sure how to answer the second question sorry? "rsync switches"?

I've found that if you're copying a bunch of small files, it's not going to go fast with rsync.  There's a lot of reconnecting going on for each file when you use rsync.  When you copy a large file it goes a bit faster, but Rsync to a netgear NAS needs some tweaking.

Even between servers, for smaller files, you'd want to use tar for the initial copy:
tar c some/dir | gzip - |  ssh host2 tar xz

Since this is a Netgear NAS, tar wont help you.  You can slightly improve speed by mounting it via NFS then run 4-6 concurrent rsyncs on separate folders to increase throughput.  You mount it NFS so you don't use as much CPU for each rsync.

If this is an initial copy, I suggest using cpio -a over NFS, instead of rsync.  rsync will be faster once you have existing data that you don't want to recopy, but cpio will be faster for the initial copy.  You can run multiple simultaneous copies to speed up throughput also.

Oh, you've update while I was typing.

What options did you give to your rsync command?  (rsync switches - as asked by mikefritz)
Is it over nfs, rsync service to the NAS?
What is your load average from your rsync command?

cpio -a may be faster for the initial copy.

right sorry switches, of course, I was thinking networking, not command line - sorry.

rsync -r -a -v -W --progress root@192.168.99.8:/c/media /mynas/media/

I am running it in a screen app, so it can run, is that how you would suggest running concurrent rsyncs, setup several screens.

I have not mounted it to NFS, so I could do that pretty easily I expect, as a noob/ hack on linux, wouldn't take to long to work out I think

Try adding the -z parameter (compress) to your rsync command line. Any difference?

iperf does single stream testing of a "large" stream.

When you use rsycn you are copying indvidual files.  As you have found, coping a lot of little files takes more time than a few big files.   This is due to the overhead of creating, opening, and closing a file.  On top of that, if I remember correctly, each file is a unique TCP connection, so you also have the overhead of starting and stoping a TCP connection.

Using -z will probably only help on "large" files.   It may help on the smaller files or id could make it worse.

However, depending on where you live you could get a 5 port 1GbE switch for under $50.
Connect both NAS's to this switch and connect it to your Cisco switch.   The copy between the two NAS's should drastically improve while access through the Cisco switch will stay the same.

Don't use "-z" switch on media files - they are usually already compressed and can actually grow larger on a recompression.

NFS or CIFS mounts may help if it lots of small files since rsync not only opens anew but checks permissions as well.

I'd think getting a GigE connection would be helpful in any case though.  5TB of media is quite a bit.

cpio could work if you get a mounted remote filesystem via NFS or CIFS(SMB) then just do:

cd /{OLD_NAS_WHAT_I_WANT}/
find . | cpio -pv /{MOUNT_POINT_OF_NEW}/someplace_sensible

After looking up the specs of your device, I don't know if you can actually benefit from multiple rsync, since it's only got an AMD Turion II cpu.  The -z option may possibly slow it down.

Most of it is the overhead of opening and closing files, and checking.  tar over ssh will still be faster than rsync for any initial file copy.  You should always start with piping tar over ssh or use cp -a for initial copies.

http://www.damtp.cam.ac.uk/user/ejb48/sshspeedtests.html


It looks like you're directly connected to one of your HP NAS to run rsync.  I don't know if you can nfs mount within that, since I'm not sure how much linux is included on these units.

If you're able to mount nfs filesystems, there should be an option somewhere in the ReadyNAS interface to enable it.  It's been a few years since I've worked on one.  You then mount -t nfs from your system
mount -t nfs 192.168.99.8:/c/media /mynas/media/REMOTE_NAS
If you're going to do multiple rsync, then you'll have to separate them by folder and background them.
rsnyc -raW /mynas/media/REMOTE_NAS/FOLDER1 /mynas/media/FOLDER1 &
rsnyc -raW /mynas/media/REMOTE_NAS/FOLDER2 /mynas/media/FOLDER2 &

But I suspect cp will be faster.
cp -a /mynas/media/REMOTE_NAS/ /mynas/media/

You can verify the mount point of the remote nfs server with
showmount -e 192.168.99.8


If you can't, you can still possibly run backgrounded rsync processes in your screen window.  I would just get rid of the verbose and progess options so it doesn't waste cycles displaying to a console screen.

You can run rsync in the background and send the output to a logfile
rsnyc -raW root@192.168.99.8:/c/media/FOLDER1 /mynas/media/FOLDER1 &
rsnyc -raW root@192.168.99.8:/c/media/FOLDER2 /mynas/media/FOLDER2 &

If you want some kind of logs you can send them to a file instead.
rsnyc -raW --log-file=/mynas/media/logfile4.txt root@192.168.99.8:/c/media/FOLDER1 /mynas/media/FOLDER1 &

When you connect back you should be able to tail logfile to see where it's at.  You won't see the individual file's progress, but you may not really need that fine a detail.

I've never really like these NAS units.

Ok, I got NFS enabled on the ReadyNas no problem, but I am getting this error on the Ubuntu HP Microserver :(

root@NASMedia:/home/craig# apt-get install nfs-utils nfs-utils-lib
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package nfs-utils
E: Unable to locate package nfs-utils-lib

Select all Open in new window

I can't work out why, and I have run apt-get update too.
Thoughts?

@mike - I tried the -z switch, and it slowed it down to 1.02MB/s :(

I am going to try the NFS thing next time I come into the office to work this out.
Just need to work out why I cannot locate the package :(

Thanks

Have you tried just using a regular linux system and mount both unit's NFS shares?  Using cp from a real system might be faster.

Have you tried just using a regular Windows system and mount both unit's SMB shares?  Using a xcopy from a windows system might be faster.

These units generally come with stripped down, and frequently unpatched, stubs of linux.

No matter what trying to transfer 5TB even at 7-8MB/sec (which is what SMB/NFS might be able to get) will take a LONG time.   Getting a small inexpensive 1GbE switch to get the speeds up to the 30-40MB/sec (about the best you will get without Jumbo frames).

The big issue is not really the amount of data, but the number of files.  Copying 100 1MB files will take longer than copying 1 100MB file.

Thanks @Mike - that worked, got NFS working and Whoop Whoop! 9MB/s speeds :)

I will try attaching a Gigabit switch again, but I doubt will make a difference...

Great you've got this fixed, but you've selected one NFS comment and the other that is about tar and zip over ssh. Which of the two did you use and which one gave you the best performance?

Just to clarify, I shouldn't of selected the Tar comment sorry, I only used the suggestions about NFS (well what worked well).

You can Click on the request attention link and a moderator can undo your selection and you can correctly reassign the points to the answer that did solve your issue.  I've done that for you, but you should be able to do that as well.