Speed up NIC / Rsync transfer rate on N54L Ubuntu Server

Hi experts,

I was hoping you could help me with this problem.
I am transferring 5TBs of data from an old Netgear ReadyNas NX v1 to my new HP Microserver N54L via an rsync.
However I am getting only
230,559,744 100%    1.38MB/s    0:02:39 (xfr#2536, ir-chk=1041/54820)

I was hoping someone could help me for out how to improve that, as it has been 2 weeks, and it is slow going.  
1. I cannot set mtu over 1500 on the N54L (is set on the ReadyNas NX)
2. I cannot get throughput over 1.4MBs :(

Some data for you, ethtool and ifconfig
Settings for em1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Speed: 100Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x000000ff (255)
                               drv probe link timer ifdown ifup rx_err tx_err
        Link detected: yes
root@NASMedia:/home/craig# ifconfig
em1       Link encap:Ethernet  HWaddr fc:15:b4:90:36:a1
          inet addr:192.168.99.21  Bcast:192.168.99.255  Mask:255.255.255.0
          inet6 addr: fe80::fe15:b4ff:fe90:36a1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1524649 errors:0 dropped:1768 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:332042199 (332.0 MB)  TX bytes:680 (680.0 B)
          Interrupt:18

eth0      Link encap:Ethernet  HWaddr 00:24:27:fe:8b:e2
          inet addr:192.168.99.221  Bcast:192.168.99.255  Mask:255.255.255.0
          inet6 addr: fe80::224:27ff:fefe:8be2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:462331468 errors:0 dropped:1769 overruns:0 frame:0
          TX packets:83198154 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:661902799947 (661.9 GB)  TX bytes:5756148471 (5.7 GB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2276 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2276 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:136784 (136.7 KB)  TX bytes:136784 (136.7 KB)

Open in new window

LVL 1
Craig LambieAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Craig LambieAuthor Commented:
Some new info that might help
NASShared:/# iperf -c 192.168.99.221 -r -f M
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.25 MByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.99.221, TCP port 5001
TCP window size: 0.02 MByte (default)
------------------------------------------------------------
[  7] local 192.168.99.8 port 3198 connected with 192.168.99.221 port 5001
[  7]  0.0-10.0 sec    112 MBytes  11.2 MBytes/sec
[  6] local 192.168.99.8 port 5001 connected with 192.168.99.221 port 34845
[  6]  0.0-10.1 sec    113 MBytes  11.2 MBytes/sec

Open in new window


I also tried to do a copy with SCP and only got 1.4 MB/s too :(
I don't get how I can get a 11.2MB/s speed with iperf/ network tools, but only 1.4MB/s with copy?

I think there must be a config issue or a drive speed issue?

Thanks in advance.
0
Jan SpringerCommented:
what type of switch port is this server connected?  it has 1G capacity but is operating at 100M.

you might have to hard code speed and duplex in the switch.  be careful, though, changing these settings may cause the port to reset.
0
mikelfritzCommented:
I agree with Jan.  On the port reset; even if it does (likely will) you said you are using rsync so it should not have to copy what it already did - it will start where it left off within reason.

The question remains on what type of switch is connected and if both machines are connected at 1G... We know this one is not.

Once you know the switch supports 1G, make sure it's configured to use 1G and you can use something like this to force 1G

# ethtool -s em1 speed 1000 duplex full
0
Discover the Answer to Productive IT

Discover app within WatchGuard's Wi-Fi Cloud helps you optimize W-Fi user experience with the most complete set of visibility, troubleshooting, and network health features. Quickly pinpointing network problems will lead to more happy users and most importantly, productive IT.

Jan SpringerCommented:
the NIC should automatically negotiate.
0
Craig LambieAuthor Commented:
Hi Jan and Mike, thanks for your comments.
The switch is a Cisco 10/100 48 port switch, something like this

I have tried connecting the N54L (and the ReadyNas) to a relatively cheap Dlink Gigabit switch, but the guy that gave me the old enterprise Cisco switch suggested that the throughput on it is better than a lower end Gigabit switch, so I went back.

The fact that I am getting almost 10x the speed when doing the iperf test, suggests it is something else to me.  What do you think?
0
Jan SpringerCommented:
if this is a 10/100 port then you will not get GigE speeds.

and, yes, check the destination machine NIC and processes.
0
Craig LambieAuthor Commented:
Thank Jan.  I am not expecting GigE speeds, but 1.4MB/s is really slow for 100 full duplex.

I have run the iperf on both machines.
I think it might be rsync, or some other configuration?
0
mikelfritzCommented:
What kind of data and what rsync switches?

Getting a gigE connection would help in theory, but if you are right then it won't do much.
0
Craig LambieAuthor Commented:
Mike, I am transferring 5TB of mostly media files - backing up my old Media Server.
Large files 300MB+

not sure how to answer the second question sorry? "rsync switches"?
0
serialbandCommented:
I've found that if you're copying a bunch of small files, it's not going to go fast with rsync.  There's a lot of reconnecting going on for each file when you use rsync.  When you copy a large file it goes a bit faster, but Rsync to a netgear NAS needs some tweaking.

Even between servers, for smaller files, you'd want to use tar for the initial copy:
tar c some/dir | gzip - |  ssh host2 tar xz

Since this is a Netgear NAS, tar wont help you.  You can slightly improve speed by mounting it via NFS then run 4-6 concurrent rsyncs on separate folders to increase throughput.  You mount it NFS so you don't use as much CPU for each rsync.

If this is an initial copy, I suggest using cpio -a over NFS, instead of rsync.  rsync will be faster once you have existing data that you don't want to recopy, but cpio will be faster for the initial copy.  You can run multiple simultaneous copies to speed up throughput also.
0
serialbandCommented:
Oh, you've update while I was typing.

What options did you give to your rsync command?  (rsync switches - as asked by mikefritz)
Is it over nfs, rsync service to the NAS?
What is your load average from your rsync command?

cpio -a may be faster for the initial copy.
0
Craig LambieAuthor Commented:
right sorry switches, of course, I was thinking networking, not command line - sorry.

rsync -r -a -v -W --progress root@192.168.99.8:/c/media /mynas/media/

I am running it in a screen app, so it can run, is that how you would suggest running concurrent rsyncs, setup several screens.

I have not mounted it to NFS, so I could do that pretty easily I expect, as a noob/ hack on linux, wouldn't take to long to work out I think
0
Gerwin Jansen, EE MVETopic Advisor Commented:
Try adding the -z parameter (compress) to your rsync command line. Any difference?
0
giltjrCommented:
iperf does single stream testing of a "large" stream.

When you use rsycn you are copying indvidual files.  As you have found, coping a lot of little files takes more time than a few big files.   This is due to the overhead of creating, opening, and closing a file.  On top of that, if I remember correctly, each file is a unique TCP connection, so you also have the overhead of starting and stoping a TCP connection.

Using -z will probably only help on "large" files.   It may help on the smaller files or id could make it worse.

However, depending on where you live you could get a 5 port 1GbE switch for under $50.
Connect both NAS's to this switch and connect it to your Cisco switch.   The copy between the two NAS's should drastically improve while access through the Cisco switch will stay the same.
0
mikelfritzCommented:
Don't use "-z" switch on media files - they are usually already compressed and can actually grow larger on a recompression.

NFS or CIFS mounts may help if it lots of small files since rsync not only opens anew but checks permissions as well.

I'd think getting a GigE connection would be helpful in any case though.  5TB of media is quite a bit.

cpio could work if you get a mounted remote filesystem via NFS or CIFS(SMB) then just do:

cd /{OLD_NAS_WHAT_I_WANT}/
find . | cpio -pv /{MOUNT_POINT_OF_NEW}/someplace_sensible
0
serialbandCommented:
After looking up the specs of your device, I don't know if you can actually benefit from multiple rsync, since it's only got an AMD Turion II cpu.  The -z option may possibly slow it down.

Most of it is the overhead of opening and closing files, and checking.  tar over ssh will still be faster than rsync for any initial file copy.  You should always start with piping tar over ssh or use cp -a for initial copies.

http://www.damtp.cam.ac.uk/user/ejb48/sshspeedtests.html


It looks like you're directly connected to one of your HP NAS to run rsync.  I don't know if you can nfs mount within that, since I'm not sure how much linux is included on these units.

If you're able to mount nfs filesystems, there should be an option somewhere in the ReadyNAS interface to enable it.  It's been a few years since I've worked on one.  You then mount -t nfs from your system
mount -t nfs 192.168.99.8:/c/media /mynas/media/REMOTE_NAS
If you're going to do multiple rsync, then you'll have to separate them by folder and background them.
rsnyc -raW /mynas/media/REMOTE_NAS/FOLDER1 /mynas/media/FOLDER1 &
rsnyc -raW /mynas/media/REMOTE_NAS/FOLDER2 /mynas/media/FOLDER2 &

But I suspect cp will be faster.
cp -a /mynas/media/REMOTE_NAS/ /mynas/media/

You can verify the mount point of the remote nfs server with
showmount -e 192.168.99.8


If you can't, you can still possibly run backgrounded rsync processes in your screen window.  I would just get rid of the verbose and progess options so it doesn't waste cycles displaying to a console screen.

You can run rsync in the background and send the output to a logfile
rsnyc -raW root@192.168.99.8:/c/media/FOLDER1 /mynas/media/FOLDER1 &
rsnyc -raW root@192.168.99.8:/c/media/FOLDER2 /mynas/media/FOLDER2 &

If you want some kind of logs you can send them to a file instead.
rsnyc -raW --log-file=/mynas/media/logfile4.txt root@192.168.99.8:/c/media/FOLDER1 /mynas/media/FOLDER1 &

When you connect back you should be able to tail logfile to see where it's at.  You won't see the individual file's progress, but you may not really need that fine a detail.

I've never really like these NAS units.
0
Craig LambieAuthor Commented:
Ok, I got NFS enabled on the ReadyNas no problem, but I am getting this error on the Ubuntu HP Microserver :(

root@NASMedia:/home/craig# apt-get install nfs-utils nfs-utils-lib
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package nfs-utils
E: Unable to locate package nfs-utils-lib

Open in new window


I can't work out why, and I have run apt-get update too.
Thoughts?
0
Craig LambieAuthor Commented:
@mike - I tried the -z switch, and it slowed it down to 1.02MB/s :(

I am going to try the NFS thing next time I come into the office to work this out.
Just need to work out why I cannot locate the package :(

Thanks
0
mikelfritzCommented:
does ""apt-get install nfs-common" work?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
serialbandCommented:
Have you tried just using a regular linux system and mount both unit's NFS shares?  Using cp from a real system might be faster.

Have you tried just using a regular Windows system and mount both unit's SMB shares?  Using a xcopy from a windows system might be faster.

These units generally come with stripped down, and frequently unpatched, stubs of linux.
0
giltjrCommented:
No matter what trying to transfer 5TB even at 7-8MB/sec (which is what SMB/NFS might be able to get) will take a LONG time.   Getting a small inexpensive 1GbE switch to get the speeds up to the 30-40MB/sec (about the best you will get without Jumbo frames).

The big issue is not really the amount of data, but the number of files.  Copying 100 1MB files will take longer than copying 1 100MB file.
0
Craig LambieAuthor Commented:
Thanks @Mike - that worked, got NFS working and Whoop Whoop! 9MB/s speeds :)

I will try attaching a Gigabit switch again, but I doubt will make a difference...
0
Gerwin Jansen, EE MVETopic Advisor Commented:
Great you've got this fixed, but you've selected one NFS comment and the other that is about tar and zip over ssh. Which of the two did you use and which one gave you the best performance?
0
Craig LambieAuthor Commented:
Just to clarify, I shouldn't of selected the Tar comment sorry, I only used the suggestions about NFS (well what worked well).
0
serialbandCommented:
You can Click on the request attention link and a moderator can undo your selection and you can correctly reassign the points to the answer that did solve your issue.  I've done that for you, but you should be able to do that as well.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.