Slow network speed on Linux

I would verify that under Linux the adpaters and the switch port are both full duplex.  It sounds like there a duplex mismatch.

use ethtool to set duplex and speed settings on both linux machines. Use the same setting on both machines at a time.

ethtool -s eth0 speed 100 duplex half

for example to set 100Mbps at half duplex on both computers. Also, check "dmesg" upon plugging in the lan cable, to see what is the autonegotiated speed.

Also, check if there are collisions using ifconfig eth0 on both computers.

DNS and routing issue.

You probably need to set the /etc/hosts file to include the other PCs IP, otherwise what happens is:

System locates other PC by checking hosts file, doesn't find it, checks DNS, doesn't find it, broadcasts a request and finally gets a response and sends a few packets! Process repeats. The DNS has a timeout period which makes things really slow!

Alternatively, only refer to the other PC during this type of transfer by its ip address rather than its name. Unfortunately if you use network discovery type tools, they convert to a name for the benefit of humans!...and it's this conversion of the name back to an ip address that slows things down badly.

Another possible alternative (depending on your config) is that you should configure your systems to use a DNS server in your home router, rather than your ISP router. This router has visibility of the systems inside your own network, (especially if it has allocated ip addresses to them by DHCP).

Tests:

ping servername

If this is slow and:

ping aa.bb.cc.dd

is fast, then you have a good idea that the conversion of name to ip address is the problem. You will also know when it is solved, because both pings will give short timings.

Name resolution issues will not cause file transfers to be slow, it may cause them to take awhile to start.  Same thing with ping.

If there is a name resolution problem, adding names to the host file will help, but it is generally the wrong thing to do.  Getting the host names in the proper zone on the proper DNS server is the correct thing to do.

Thank you all for the help, but the problem still persists.

After bootup, both computers report (using ethtool) to be already running at 1000Mbp/s and with full duplex, so in principle no further adjustments would be needed.

If I try to set both computers' adapters to a new configuration (ie, 100Mbp/s and half duplex, by typing 'ethtool -s <device_name> speed 100 duplex half'), they beep as if re-setting the configuration, and return to the command prompt without emitting any errors. However, the new configurations are not saved, as they both remain at 1000Mbp/s and full duplex regardless of the configuration I set using ethtool.

I have also tried pjedmond's suggestion of only referring to the IP address of the computers during the transfers, which caused no improvement in performance. Ustas, I check ifconfig but there are no collisions (see below)

Again, if both computers are booted into Windows XP, transfers between them happen very very fast (effective gigabit speeds).

Any other ideas? Thanks a lot.

Below is the output of 'ethtool <device_name>' and 'ifconfig <device_name>' on each computer:

NOTEBOOK:
-----------
# ethtool eth0
Settings for eth0:
        Supported ports: [ MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x000000ff (255)
        Link detected: yes

# ifconfig eth0
eth0    Link encap:Ethernet  HWaddr 00:0A:E4:36:2D:BA
          inet addr:192.168.0.105  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::20a:e4ff:fe36:2dba/64 Scope:Link
          UP BROADCAST NOTRAILERS RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:40520 errors:0 dropped:0 overruns:0 frame:0
          TX packets:64534 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7866823 (7.5 Mb)  TX bytes:42684065 (40.7 Mb)
          Interrupt:11

DESKTOP:
----------
# ethtool eth1
Settings for eth1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000033 (51)
        Link detected: yes

# ifconfig eth1
eth1    Link encap:Ethernet  HWaddr 00:11:09:65:F6:DC
          inet addr:192.168.0.102  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::211:9ff:fe65:f6dc/64 Scope:Link
          UP BROADCAST NOTRAILERS RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:63560 errors:0 dropped:0 overruns:0 frame:0
          TX packets:37224 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:71481449 (68.1 Mb)  TX bytes:3717751 (3.5 Mb)
          Interrupt:209 Base address:0xc000

Can you check the status of the switch ports?  Although negotiation for 1000 Mbps eithernet should work, it could be that the switch has an issue with the way the Linux drivers are negotiating things.

Thanks giltjr, but, if the switch has an issue with the way the Linux drivers are negotiating things, how do I know that? What should I look for when checking the status of the switch ports?

If the switch is a manged switch you can display the status of the port you are connected to.  If the switch is unmanaged, then well there is no a whole lot you can do.

I am back trying to solve this problem. Could DNS / routing - as indicated by pjedmond and dismissed by giltjr - the culprit or not? Intuitively, I would agree to the idea that DNS problem could cause a delay for the transfer to start, but should not be reason why the transfer gets slow, wouldn't you agree?

Anyway, I tried the ping test. As some background info, this is a home network where all computers are linked to a DLink 1Gbps router (mode DGL-4300), which assigns the IP address. The 2 computers I am doing this test have gigabit ethernet adapters. The SMB names and IP addresses are as follows:

Router IP: 192.168.0.1
Desktop IP: 192.168.0.102 / SMB name: 'master'
Notebook IP: 192.168.0.105 / SMB name: 'thinkpadx41'

When both computers are booted into Linux, using the ping command on either computer to ping the other by using their SMB names (ie: 'ping master' or 'ping thinkpadx41') causes an error message that the computer cannot be located. However, pinging the computers using their direct IP addresses works just fine and fast. Pinging an external site (ie, 'ping www.google.com') works flawlessly as well. On Windows, pinging the computers in my network by their names work just right.

Giltjr suggested that if there is a DNS problem 'Getting the host names in the proper zone on the proper DNS server is the correct thing to do'. How do I do this? I don't think (but am not sure) that my DLink router has an internal DNS server. I would like to avoid editing the /etc/hosts file because them I would need to manually change it if there is any change to the IP addresses of the computers in my network.

Thanks a lot.

I edited the /etc/hosts file on both computers - as suggested by pjedmond - to include the netbios/smb name for each IP address. Ping now works as intended. However, there has been no change to the performance of file transfers.

Now, one interesting thing that I had not noticed before: The performance impact only happens in one direction. If I copy files from the notebook to the desktop computer, the transfer happens at 0.2Mbps. However, if I perform the same file transfer from the desktop to the notebook, the transfer happens at 15Mbps.

Any idea what is going on?

Thanks

More info: Checking my router configuration, I tried disabling the `DNS Relay` feature, which was enabled by default. Doing this seems to have increased the throughput of the notebook -> desktop file transfer to approx. 1Mbps (up from 0.2-0.6Mbps before), but that is still far below the 15Mbps I get when transferring desktop -> notebook.

Still trying...

This is *not* a "DNS and routing issue"

pjedmond has a very limited understanding of how IP and DNS work, IMO.

Also, how about if we try to isolate protocol problems from network problems.

I assume the transfers are using SMB, given that SMB names were mentioned.  Can you try another method/protocol instead (like ssh/scp)?

Cheers,
-Jon

LOL!  I post that and then notice  pjedmond name above mine on the left, so I should probably explain my discounting of his theories:

>You probably need to set the /etc/hosts file to include the other PCs IP, otherwise what happens is:

>System locates other PC by checking hosts file, doesn't find it, checks DNS, doesn't find it, broadcasts a
>request and finally gets a response and sends a few packets!

With you so far...

>Process repeats. The DNS has a timeout period which makes things really slow!

Why would it repeat?  This info should now be cached, and no caches I've ever seen time out even close to that fast.  

>Alternatively, only refer to the other PC during this type of transfer by its ip address rather than its
>name.

OK...

>Unfortunately if you use network discovery type tools, they convert to a name for the benefit of humans!...

Umm, you lost me again - we're talking about file tranfers here, not network discovery type tools, unless by "network discovery type tools" you mean SMB.

>and it's this conversion of the name back to an ip address that slows things down badly.

Once again, I cannot agree that this would cause anything beyond an initial delay.

>Another possible alternative (depending on your config) is that you should configure your
>systems to use a DNS server in your home router, rather than your ISP router.
>This router has visibility of the systems inside your own network, (especially if
>it has allocated ip addresses to them by DHCP).

Well, I'd have to agree that this was a DNS problem to agree to fix it using DNS.  BTW, DNS services typically run on servers, not routers.

>Tests:

>ping servername

>If this is slow and:

>ping aa.bb.cc.dd

>is fast,

There is no way they could differ beyond an initial delay, unless one (or both) simply fail.

Cheers,
-Jon

Thanks for the suggestion. I will try transfering the files with another protocol (I was using SMB indeed). If I use FTP, will that bypass the SMB protocol?

What now puzzles me the most is that the performance decrease is unidirectional (from notebook to desktop, but not in the opposite direction. Again, only happens on Linux).

I will report back after the testing. Thanks.

For The--Captain...

Guess I need to explain a little further:
Well, I'd have to agree that this was a DNS problem to agree to fix it using DNS.  BTW, DNS services typically run on servers, not routers.
>Process repeats. The DNS has a timeout period which makes things really slow!

Why would it repeat?  This info should now be cached, and no caches I've ever seen time out even close to that fast.
----------8X----------

This problem applies to some older hubs and when some cheap routers that try to be all singing and dancing reckon that they can do DNS. I agree that in an ideal world they would be cached...but in many cases it is not.

----------8X----------
Well, I'd have to agree that this was a DNS problem to agree to fix it using DNS.  BTW, DNS services typically run on servers, not routers.
----------8X----------
Absolutely correct, but if you look at most of the cheaper routers for adsl, and cable these days, and look at their configuration, then you will see that they offer a DHCP service, and also a caching DNS service linked to it. This means that you can carry out a dig or nslookup against it and get a response.#

I accept, that it is not a full scale Bind DNS setup, but to an external PC they look the same.

----------8X----------
>ping servername

>If this is slow and:

>ping aa.bb.cc.dd

>is fast,

There is no way they could differ beyond an initial delay, unless one (or both) simply fail.
----------8X----------

If it cannot resolve the ip efficiently, and has to convert name to ip each time, then this scenario is most definitely possible. Simplest example is if DNS resolution is only from an external source with no caching.

My suggestion was based on the possibility of faulty routing, which may or may not be wrong.

Interestingly enough:
----------8X----------
Checking my router configuration, I tried disabling the `DNS Relay` feature, which was enabled by default.
----------8X----------
Apparently this appears to have increased throughput. This does not suprise me as can you imagine a cheap router being able to cache all the DNS records from the pages you visit. Having said that, the hardware mentioned is not exactly the 'cheaper' hardware that I'd have expected for this type of problem.

Now as we have more information I'm just going off at another complete tangent, and waiting to be flamed ;) The fact that Ping now works as expected, makes me look away from DNS to other issues.

Just curious, but could the speed  of the laptop disk drive be an issue here? If copying from the laptop to the desktop, then the data has to be read from the laptop disk drive. Copying the other way, then much of the data could be cached in the RAM of the laptop (acting as a buffer)?

This thought is just as a result of the apparent asynchronous apparent nature of this problem, which should not occur if the issue is a protocol one. (I await examples of exceptions......probably related to UDP;)  )

The way to test this would be to try transferring HUGE files...say 1GB, and see if the throughput speed starts fast, and then slowly drops to match the lower rate once the 'memory buffer' becomes full.

Another thought, try changing the power settings of the laptop to be max power/speed settings.

Disc drives use significant amounts of power, and are one area where laptops try hard to save power by spinning them down or reducing the speed of the rotations (effectively slowing the data transfer). Also clock speeds and various other peripheral devices may be slowed down by this.

HTH:)

Using FTP will use the ftpd rather than the smbd so would be a valid test. If the transfer is asymetric (I meant to write asymetric rather than asynchronous in the post above), then we may need to start looking at network configuration a little more closely.

Can you print the output of:

route

on the linux system

and:

route print

on the windows XP.

>>>ping servername

>>>If this is slow and:

>>>ping aa.bb.cc.dd

>>>is fast,

>>There is no way they could differ beyond an initial delay, unless one (or both) simply fail.

>If it cannot resolve the ip efficiently, and has to convert name to ip each time,
>then this scenario is most definitely possible. Simplest example is if DNS resolution
>is only from an external source with no caching

Sorry, but we're just going to have to agree to disagree here, especially since I've never seen a default linux install that has no DNS cache (or the cache timeout is under 1 sec).

You are absolutely correct about cheapo soho routers/modems pretending to present themselves as DNS servers via DHCP, only to forward it to the real DNS servers - am I the only one having trouble understanding why they don't just pass out the IPs of the real DNS servers?  Is it so the WAN connection can go down and the router can still hand out ostensibly "correct" info via DHCP that will work when the WAN comes back up?

And yeah, FTP would work fine for this test - I only suggested scp as I consider it marginally easier to set up, and don't think the overhead would affect things overmuch.

Cheers,
-Jon

Thanks again for the insights. I don't think hardware issues (memory, power optimization on the notebook, hard disk speed, etc) could be involved here because this very same hardware performs perfectly if I boot both computers into Windows XP.

I am always doing file transfers using very large files (300Mb to 1Gb). Throughput is always constant (either very fast or very slow, but do not show any 'decreasing' or 'increasing' performance pattern.

I will report again once I have conducted the tests The--Captain  and pjedmond suggested.

Thanks

I think I need to bow to The--Captain's greater knowledge in the networking sphere (I've just checked, and he's fairly close to guru status;)

...although perhaps he may be interested in playing with a few of the embedded linux distros and QNX;)

---------------8X--------------
(memory, power optimization on the notebook, hard disk speed, etc) could be involved here because this very same hardware performs perfectly if I boot both computers into Windows XP.
---------------8X--------------

Remember that the hardware is under OS control via ACPI....and the linux handling of this is likely to be different.

>I've just checked, and he's fairly close to guru status

Are you serious?  Where can I see that?

>..although perhaps he may be interested in playing with a few of the embedded linux distros and QNX;)

You know QNX?  If so, I've got a good question for you...  (I'll open it in a seperate thread of course)

Cheers,
-Jon

Did u try in the smb.conf file

[global]
socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192

Guess I missed some posts, I've been out of touch for a few days.  As I stated and Jon has been saying, DNS issues do not cause slow transfers.  They could cause a dealy in the start of a transfer, or prevent the transfer if the name can be resloved at all.

As has been suggested you may want to try ftp.  This will show if the problem is hardware related, or smb related.  Even under Windows, smb transfers are slow.   Using ftp will generally be must faster with very little tuning whereas even with tuning smb is still slow.

What type of speeds do you get when you are running under Windows?

Athough I would not expect it to cause a major slow down I see that you have a MTU of 1500 while under Linux.  You may want to change this to the max MTU that you cards can support.  Typically gig cards support frames of up to 9000 bytes.  Fewer larger frames means  more throughput.

Did you really mean 2 Kbps, two kilobits per second, 250 bytes per second?

150,000 is all you need for Guru status...Another 23000 to go:)...Not sure where you put questions for RTOS microkernels?

Sorry I have not followed-up for a week but it`s been quite busy here (world cup and all...!).

Well, I am back trying to solve this issue. I was trying to perform a FTP transfer to try to isolate the problem (ie, to check if slow throughput problem was realted to SAMBA), but then I realized I don't know how to set up an FTP server (dough!). If anyone knows an easy to configure (preferrably with a graphical interface) FTP server, that would help. I will keep trying by myself in the meantime.... Thanks!.

Ok, I got vsftpd running. I did a 500mb file transfer in 4 different ways. The results are below:

FTP transfer:
- from desktop to notebook: average speed of 38 Mb/s
- from notebook to desktop: average speed of 22 Mb/s

SMB transfer:
- from desktop to notebook: average speed of 10 Mb/s
- from notebook to desktop: average speed of 0.5 Mb/s

Clearly SMB is slower than FTP. This may help isolate the problem a little bit, but still is no solution. Again, remember that if i boot both computers into Windows, this performance impact does not happen.

I will now try to do it booting both computers under Ubuntu as opposed to SUSE to see what happens.

Any more ideas what is the source of the problem?

SMB is much slower than FTP, even under Windows.  SMB is a very chatty protocol.  I would suggest that if you are moving lots of files, you use FTP.

In your smb.conf you may want to try upping the SO_SNDBUF and SO_RCVBUF on your socket options.  I have 32768 for both.
mjdme recommened 8192 for these, but the bigger the better (to a point), also do TCP_NODELAY as he suggested.

If that does not work let us know.  There are some other TCP/IP tuning parameters that may help.

Dears giltjr, mjdme,

I edited smb.conf to insert the parameters as suggested:

[global]
socket options = TCP_NODELAY SO_RCVBUF=32768 SO_SNDBUF=32768

This was performed on both computers and I rebooted both just to make sure that the settings were in effect. After that, I performed the SMB transfer again (simple drag-and-drop of a 4Gb file using Konqueror) and, despite my best expectations that now I would get this solved, I still got 0.5 Mbp/s when transferring from the notebook to the desktop. FTP transfer of the same yielded 22Mbps and SCP transfer was performed at 17Mbps. SMB is still sucking on my Linux networking. I am running out of hope after so many tests....

regards,
Gustavo Saito

Umm.   Something is definitily unusal.  I just did a test on my home network  between a Windows XP laptop and a Linux box over a 54 Mbps wireless network  and I got  about 16 Mbps in both directions  Now I have also modified /etc/sysctl.conf and I added the following:

     # Change memory used for IP buffers
     net.core.wmem_max = 1024000
     net.core.rmem_max = 1024000

     # Change window size
     net.ipv4.tcp_wmem = 4096 262144 524288
     net.ipv4.tcp_rmem = 4096 262144 524288

This increases the memory that Linux can use for IP and changes the windows sizes for TCP connections.  One thing you may want to do before you change this is install Ethereal (it is part of the packeges that come with SUSE) and capture a few seconds of packets while you do the copy.  Look at the trace and see if anything looks unusual.  The speed you are getting is way to low.

Thanks again for keeping with me giltjr. I've done the changes to /etc/sysctl.conf on both computers and rebooted both. SMB transfers from notebook to desktop still at 0.5Mbps...

Using Ethereal was a good idea. After capturing a few seconds of the file transfer, I noticed I am getting checksum errors in practically every packet! At least now we seem to have found the problem, although still no clue as to what is causing the problem...

Is there any way I can post the results of ethereal here? (ie, save an attachment with the saved output file?).

Which version of Ethereal do you have?  I have seen Ethereal display checksum errors when their really aren't any, at least in my opinion there aren't any.  Part of the reason I doubt, but will not rule out, checksum errors is that these are not caclulated by the application (SAMBA) but by the TCP/IP stack.  The same stack that SAMBA uses is the same stack that ftp uses.   In addition to that they both use the same network.  So if there were something that was causing checksum errors, everything would be getting them.

I will have to check, I beleive that one of the members here has created a site that they allow you to post files to.