Speed up NIC / Rsync transfer rate on N54L Ubuntu Server
Hi experts,
I was hoping you could help me with this problem.
I am transferring 5TBs of data from an old Netgear ReadyNas NX v1 to my new HP Microserver N54L via an rsync.
However I am getting only
I was hoping someone could help me for out how to improve that, as it has been 2 weeks, and it is slow going.
1. I cannot set mtu over 1500 on the N54L (is set on the ReadyNas NX)
2. I cannot get throughput over 1.4MBs :(
Some data for you, ethtool and ifconfig
I was hoping you could help me with this problem.
I am transferring 5TBs of data from an old Netgear ReadyNas NX v1 to my new HP Microserver N54L via an rsync.
However I am getting only
230,559,744 100% 1.38MB/s 0:02:39 (xfr#2536, ir-chk=1041/54820)
I was hoping someone could help me for out how to improve that, as it has been 2 weeks, and it is slow going.
1. I cannot set mtu over 1500 on the N54L (is set on the ReadyNas NX)
2. I cannot get throughput over 1.4MBs :(
Some data for you, ethtool and ifconfig
Settings for em1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off
Supports Wake-on: g
Wake-on: g
Current message level: 0x000000ff (255)
drv probe link timer ifdown ifup rx_err tx_err
Link detected: yes
root@NASMedia:/home/craig# ifconfig
em1 Link encap:Ethernet HWaddr fc:15:b4:90:36:a1
inet addr:192.168.99.21 Bcast:192.168.99.255 Mask:255.255.255.0
inet6 addr: fe80::fe15:b4ff:fe90:36a1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1524649 errors:0 dropped:1768 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:332042199 (332.0 MB) TX bytes:680 (680.0 B)
Interrupt:18
eth0 Link encap:Ethernet HWaddr 00:24:27:fe:8b:e2
inet addr:192.168.99.221 Bcast:192.168.99.255 Mask:255.255.255.0
inet6 addr: fe80::224:27ff:fefe:8be2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:462331468 errors:0 dropped:1769 overruns:0 frame:0
TX packets:83198154 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:661902799947 (661.9 GB) TX bytes:5756148471 (5.7 GB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:2276 errors:0 dropped:0 overruns:0 frame:0
TX packets:2276 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:136784 (136.7 KB) TX bytes:136784 (136.7 KB)
Some new info that might help
I also tried to do a copy with SCP and only got 1.4 MB/s too :(
I don't get how I can get a 11.2MB/s speed with iperf/ network tools, but only 1.4MB/s with copy?
I think there must be a config issue or a drive speed issue?
Thanks in advance.
NASShared:/# iperf -c 192.168.99.221 -r -f M
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.25 MByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.99.221, TCP port 5001
TCP window size: 0.02 MByte (default)
------------------------------------------------------------
[ 7] local 192.168.99.8 port 3198 connected with 192.168.99.221 port 5001
[ 7] 0.0-10.0 sec 112 MBytes 11.2 MBytes/sec
[ 6] local 192.168.99.8 port 5001 connected with 192.168.99.221 port 34845
[ 6] 0.0-10.1 sec 113 MBytes 11.2 MBytes/sec
I also tried to do a copy with SCP and only got 1.4 MB/s too :(
I don't get how I can get a 11.2MB/s speed with iperf/ network tools, but only 1.4MB/s with copy?
I think there must be a config issue or a drive speed issue?
Thanks in advance.
what type of switch port is this server connected? it has 1G capacity but is operating at 100M.
you might have to hard code speed and duplex in the switch. be careful, though, changing these settings may cause the port to reset.
you might have to hard code speed and duplex in the switch. be careful, though, changing these settings may cause the port to reset.
I agree with Jan. On the port reset; even if it does (likely will) you said you are using rsync so it should not have to copy what it already did - it will start where it left off within reason.
The question remains on what type of switch is connected and if both machines are connected at 1G... We know this one is not.
Once you know the switch supports 1G, make sure it's configured to use 1G and you can use something like this to force 1G
# ethtool -s em1 speed 1000 duplex full
The question remains on what type of switch is connected and if both machines are connected at 1G... We know this one is not.
Once you know the switch supports 1G, make sure it's configured to use 1G and you can use something like this to force 1G
# ethtool -s em1 speed 1000 duplex full
Hi Jan and Mike, thanks for your comments.
The switch is a Cisco 10/100 48 port switch, something like this
I have tried connecting the N54L (and the ReadyNas) to a relatively cheap Dlink Gigabit switch, but the guy that gave me the old enterprise Cisco switch suggested that the throughput on it is better than a lower end Gigabit switch, so I went back.
The fact that I am getting almost 10x the speed when doing the iperf test, suggests it is something else to me. What do you think?
The switch is a Cisco 10/100 48 port switch, something like this
I have tried connecting the N54L (and the ReadyNas) to a relatively cheap Dlink Gigabit switch, but the guy that gave me the old enterprise Cisco switch suggested that the throughput on it is better than a lower end Gigabit switch, so I went back.
The fact that I am getting almost 10x the speed when doing the iperf test, suggests it is something else to me. What do you think?
if this is a 10/100 port then you will not get GigE speeds.
and, yes, check the destination machine NIC and processes.
and, yes, check the destination machine NIC and processes.
Thank Jan. I am not expecting GigE speeds, but 1.4MB/s is really slow for 100 full duplex.
I have run the iperf on both machines.
I think it might be rsync, or some other configuration?
I have run the iperf on both machines.
I think it might be rsync, or some other configuration?
What kind of data and what rsync switches?
Getting a gigE connection would help in theory, but if you are right then it won't do much.
Getting a gigE connection would help in theory, but if you are right then it won't do much.
Mike, I am transferring 5TB of mostly media files - backing up my old Media Server.
Large files 300MB+
not sure how to answer the second question sorry? "rsync switches"?
Large files 300MB+
not sure how to answer the second question sorry? "rsync switches"?
I've found that if you're copying a bunch of small files, it's not going to go fast with rsync. There's a lot of reconnecting going on for each file when you use rsync. When you copy a large file it goes a bit faster, but Rsync to a netgear NAS needs some tweaking.
Even between servers, for smaller files, you'd want to use tar for the initial copy:
tar c some/dir | gzip - | ssh host2 tar xz
Since this is a Netgear NAS, tar wont help you. You can slightly improve speed by mounting it via NFS then run 4-6 concurrent rsyncs on separate folders to increase throughput. You mount it NFS so you don't use as much CPU for each rsync.
If this is an initial copy, I suggest using cpio -a over NFS, instead of rsync. rsync will be faster once you have existing data that you don't want to recopy, but cpio will be faster for the initial copy. You can run multiple simultaneous copies to speed up throughput also.
Even between servers, for smaller files, you'd want to use tar for the initial copy:
tar c some/dir | gzip - | ssh host2 tar xz
Since this is a Netgear NAS, tar wont help you. You can slightly improve speed by mounting it via NFS then run 4-6 concurrent rsyncs on separate folders to increase throughput. You mount it NFS so you don't use as much CPU for each rsync.
If this is an initial copy, I suggest using cpio -a over NFS, instead of rsync. rsync will be faster once you have existing data that you don't want to recopy, but cpio will be faster for the initial copy. You can run multiple simultaneous copies to speed up throughput also.
Oh, you've update while I was typing.
What options did you give to your rsync command? (rsync switches - as asked by mikefritz)
Is it over nfs, rsync service to the NAS?
What is your load average from your rsync command?
cpio -a may be faster for the initial copy.
What options did you give to your rsync command? (rsync switches - as asked by mikefritz)
Is it over nfs, rsync service to the NAS?
What is your load average from your rsync command?
cpio -a may be faster for the initial copy.
right sorry switches, of course, I was thinking networking, not command line - sorry.
rsync -r -a -v -W --progress root@192.168.99.8:/c/media
I am running it in a screen app, so it can run, is that how you would suggest running concurrent rsyncs, setup several screens.
I have not mounted it to NFS, so I could do that pretty easily I expect, as a noob/ hack on linux, wouldn't take to long to work out I think
rsync -r -a -v -W --progress root@192.168.99.8:/c/media
I am running it in a screen app, so it can run, is that how you would suggest running concurrent rsyncs, setup several screens.
I have not mounted it to NFS, so I could do that pretty easily I expect, as a noob/ hack on linux, wouldn't take to long to work out I think
iperf does single stream testing of a "large" stream.
When you use rsycn you are copying indvidual files. As you have found, coping a lot of little files takes more time than a few big files. This is due to the overhead of creating, opening, and closing a file. On top of that, if I remember correctly, each file is a unique TCP connection, so you also have the overhead of starting and stoping a TCP connection.
Using -z will probably only help on "large" files. It may help on the smaller files or id could make it worse.
However, depending on where you live you could get a 5 port 1GbE switch for under $50.
Connect both NAS's to this switch and connect it to your Cisco switch. The copy between the two NAS's should drastically improve while access through the Cisco switch will stay the same.
When you use rsycn you are copying indvidual files. As you have found, coping a lot of little files takes more time than a few big files. This is due to the overhead of creating, opening, and closing a file. On top of that, if I remember correctly, each file is a unique TCP connection, so you also have the overhead of starting and stoping a TCP connection.
Using -z will probably only help on "large" files. It may help on the smaller files or id could make it worse.
However, depending on where you live you could get a 5 port 1GbE switch for under $50.
Connect both NAS's to this switch and connect it to your Cisco switch. The copy between the two NAS's should drastically improve while access through the Cisco switch will stay the same.
Don't use "-z" switch on media files - they are usually already compressed and can actually grow larger on a recompression.
NFS or CIFS mounts may help if it lots of small files since rsync not only opens anew but checks permissions as well.
I'd think getting a GigE connection would be helpful in any case though. 5TB of media is quite a bit.
cpio could work if you get a mounted remote filesystem via NFS or CIFS(SMB) then just do:
cd /{OLD_NAS_WHAT_I_WANT}/
find . | cpio -pv /{MOUNT_POINT_OF_NEW}/some
NFS or CIFS mounts may help if it lots of small files since rsync not only opens anew but checks permissions as well.
I'd think getting a GigE connection would be helpful in any case though. 5TB of media is quite a bit.
cpio could work if you get a mounted remote filesystem via NFS or CIFS(SMB) then just do:
cd /{OLD_NAS_WHAT_I_WANT}/
find . | cpio -pv /{MOUNT_POINT_OF_NEW}/some
After looking up the specs of your device, I don't know if you can actually benefit from multiple rsync, since it's only got an AMD Turion II cpu. The -z option may possibly slow it down.
Most of it is the overhead of opening and closing files, and checking. tar over ssh will still be faster than rsync for any initial file copy. You should always start with piping tar over ssh or use cp -a for initial copies.
http://www.damtp.cam.ac.uk/user/ejb48/sshspeedtests.html
It looks like you're directly connected to one of your HP NAS to run rsync. I don't know if you can nfs mount within that, since I'm not sure how much linux is included on these units.
If you're able to mount nfs filesystems, there should be an option somewhere in the ReadyNAS interface to enable it. It's been a few years since I've worked on one. You then mount -t nfs from your system
mount -t nfs 192.168.99.8:/c/media /mynas/media/REMOTE_NAS
If you're going to do multiple rsync, then you'll have to separate them by folder and background them.
rsnyc -raW /mynas/media/REMOTE_NAS/FO
rsnyc -raW /mynas/media/REMOTE_NAS/FO
But I suspect cp will be faster.
cp -a /mynas/media/REMOTE_NAS/ /mynas/media/
You can verify the mount point of the remote nfs server with
showmount -e 192.168.99.8
If you can't, you can still possibly run backgrounded rsync processes in your screen window. I would just get rid of the verbose and progess options so it doesn't waste cycles displaying to a console screen.
You can run rsync in the background and send the output to a logfile
rsnyc -raW root@192.168.99.8:/c/media
rsnyc -raW root@192.168.99.8:/c/media
If you want some kind of logs you can send them to a file instead.
rsnyc -raW --log-file=/mynas/media/lo
When you connect back you should be able to tail logfile to see where it's at. You won't see the individual file's progress, but you may not really need that fine a detail.
I've never really like these NAS units.
Most of it is the overhead of opening and closing files, and checking. tar over ssh will still be faster than rsync for any initial file copy. You should always start with piping tar over ssh or use cp -a for initial copies.
http://www.damtp.cam.ac.uk/user/ejb48/sshspeedtests.html
It looks like you're directly connected to one of your HP NAS to run rsync. I don't know if you can nfs mount within that, since I'm not sure how much linux is included on these units.
If you're able to mount nfs filesystems, there should be an option somewhere in the ReadyNAS interface to enable it. It's been a few years since I've worked on one. You then mount -t nfs from your system
mount -t nfs 192.168.99.8:/c/media /mynas/media/REMOTE_NAS
If you're going to do multiple rsync, then you'll have to separate them by folder and background them.
rsnyc -raW /mynas/media/REMOTE_NAS/FO
rsnyc -raW /mynas/media/REMOTE_NAS/FO
But I suspect cp will be faster.
cp -a /mynas/media/REMOTE_NAS/ /mynas/media/
You can verify the mount point of the remote nfs server with
showmount -e 192.168.99.8
If you can't, you can still possibly run backgrounded rsync processes in your screen window. I would just get rid of the verbose and progess options so it doesn't waste cycles displaying to a console screen.
You can run rsync in the background and send the output to a logfile
rsnyc -raW root@192.168.99.8:/c/media
rsnyc -raW root@192.168.99.8:/c/media
If you want some kind of logs you can send them to a file instead.
rsnyc -raW --log-file=/mynas/media/lo
When you connect back you should be able to tail logfile to see where it's at. You won't see the individual file's progress, but you may not really need that fine a detail.
I've never really like these NAS units.
Ok, I got NFS enabled on the ReadyNas no problem, but I am getting this error on the Ubuntu HP Microserver :(
I can't work out why, and I have run apt-get update too.
Thoughts?
root@NASMedia:/home/craig# apt-get install nfs-utils nfs-utils-lib
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package nfs-utils
E: Unable to locate package nfs-utils-lib
I can't work out why, and I have run apt-get update too.
Thoughts?
@mike - I tried the -z switch, and it slowed it down to 1.02MB/s :(
I am going to try the NFS thing next time I come into the office to work this out.
Just need to work out why I cannot locate the package :(
Thanks
I am going to try the NFS thing next time I come into the office to work this out.
Just need to work out why I cannot locate the package :(
Thanks
Have you tried just using a regular linux system and mount both unit's NFS shares? Using cp from a real system might be faster.
Have you tried just using a regular Windows system and mount both unit's SMB shares? Using a xcopy from a windows system might be faster.
These units generally come with stripped down, and frequently unpatched, stubs of linux.
Have you tried just using a regular Windows system and mount both unit's SMB shares? Using a xcopy from a windows system might be faster.
These units generally come with stripped down, and frequently unpatched, stubs of linux.
No matter what trying to transfer 5TB even at 7-8MB/sec (which is what SMB/NFS might be able to get) will take a LONG time. Getting a small inexpensive 1GbE switch to get the speeds up to the 30-40MB/sec (about the best you will get without Jumbo frames).
The big issue is not really the amount of data, but the number of files. Copying 100 1MB files will take longer than copying 1 100MB file.
The big issue is not really the amount of data, but the number of files. Copying 100 1MB files will take longer than copying 1 100MB file.
Thanks @Mike - that worked, got NFS working and Whoop Whoop! 9MB/s speeds :)
I will try attaching a Gigabit switch again, but I doubt will make a difference...
I will try attaching a Gigabit switch again, but I doubt will make a difference...
Great you've got this fixed, but you've selected one NFS comment and the other that is about tar and zip over ssh. Which of the two did you use and which one gave you the best performance?
Premium Content
You need an Expert Office subscription to comment.Start Free Trial