Solved

Slow network - tough one

Posted on 2008-10-31
45
361 Views
Last Modified: 2012-05-05
Windows 2003 R2, sp2
broadcom card - onboard

I have an old server, I run a copy test (robocopy) and transfer rate is 500-900 MB/min
I build a new server, plug the same network cable into the new server, and the transfer rate is about 190 MB/min.

I have checked the duplex and speed are correct
I checked the mtu is correct
there are no errors or collisions on the switch (my network guys say all looks great)
I called the vendor (dell) and validated ALL the drivers are current
I added an Intel card with current drivers, still no change
I flashed the bios, BMC,DRAC, and SATA - no change
Re-checked the old system on the same network is still working

I am at a loss,  I would suspect hardware, but the additional Intel should have bypassed the bad card.  I have a heavy backup cycle in 7 days, and this is urgent, any help would be wonderful

Thank you advance!
0
Comment
Question by:loftyworm
  • 21
  • 12
  • 11
  • +1
45 Comments
 
LVL 14

Expert Comment

by:dfxdeimos
Comment Utility
Have you tried a different switch port (just for SnG)?

Are both servers running the same OS?
0
 
LVL 7

Expert Comment

by:CorruptedLogic
Comment Utility
Along with the switch port, go ahead and swap out the network cable. Is the new server on the same network segment as the old server (ie, does it pass through additional routers/switches that the old server does not have to contend with?).
If the situation permits, can you try connecting the new server directly to the destination machine via a crossover cable?
Does the old server have something like jumbo packets enabled?
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
What does netdiag and DCdiag dig up for you to work with?
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
Also, check your Win32 time synchronization.
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
@dfxdeimos
The same working network(cables, swtiches routers) that were plugged into the working server, are now plugged into the new server.  I simply unpluged the old and plugged in the new.  I also went back to the old once to validate it was still working.

@CorruptedLogic
I did check the MTU size, but how would I check for jumbo packet??

@CheifIT
This is not a domain controller, but I will check the netdiag and time.
0
 
LVL 7

Expert Comment

by:CorruptedLogic
Comment Utility
You would usually check whether or not Jumbo Packets were enabled on the NIC properties page (it is usually listed along with speed/duplex options etc, if it is available on your particualr nic). Depending on your switches, it can also be enabled/disabled on a per port basis at the switch level.
0
 
LVL 7

Expert Comment

by:CorruptedLogic
Comment Utility
Just another thought, does the new server have any antivirus software that may be scanning the files during the copy process? That could certainly slow things down.
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
@ChiefIT
The time is fine, and both systems are checking into the same ntp server, so they are synced.

the netdiag all passed, but it was odd that it thought the OS was 2k, and not 2k3r2, but I assume this is some version quirk.

D:\Program Files\Support Tools>netdiag.exe
 

............................................
 

    Computer Name: server

    DNS Host Name: server.domain.edu

    System info : Windows 2000 Server (Build 3790)

    Processor : x86 Family 6 Model 23 Stepping 6, GenuineIntel

    List of installed hotfixes :

        KB924667-v2

        KB925398_WMP64

        KB925876

        KB925902

        KB926122

        KB927891

        KB929123

        KB930178

        KB931784

        KB931836

        KB932168

        KB933729

        KB933854

        KB935839

        KB935840

        KB936021

        KB936357

        KB936594

        KB936782

        KB938127

        KB938127-IE7

        KB938464

        KB941202

        KB941568

        KB941569

        KB941644

        KB941693

        KB941716

        KB942615

        KB942763

        KB943055

        KB943460

        KB943485

        KB943545

        KB943729

        KB944338

        KB944653

        KB945553

        KB946026

        KB947864

        KB947864-IE7

        KB948496

        KB948590

        KB948881

        KB950759-IE7

        KB950760

        KB950762

        KB950974

        KB951066

        KB951072-v2

        KB951698

        KB951748

        KB952954

        KB953838-IE7

        KB953839

        KB954211

        KB956390-IE7

        KB956391

        KB956803

        KB956841

        KB957095

        KB958644

        Q147222
 
 

Netcard queries test . . . . . . . : Passed
 
 
 

Per interface results:
 

    Adapter : Public
 

        Netcard queries test . . . : Passed
 

        Host Name. . . . . . . . . : server.domain.edu

        IP Address . . . . . . . . : xxx.xxx.34.235

        Subnet Mask. . . . . . . . : 255.255.255.0

        Default Gateway. . . . . . : xxx.xxx.34.1

        Primary WINS Server. . . . : xxx.xxx.34.65

        Secondary WINS Server. . . : xxx.xxx.14.83

        Dns Servers. . . . . . . . : xxx.xxx.34.137

                                     xxx.xxx.34.121
 
 

        AutoConfiguration results. . . . . . : Passed
 

        Default gateway test . . . : Passed
 

        NetBT name test. . . . . . : Passed

        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messenge

r Service', <20> 'WINS' names is missing.
 

        WINS service test. . . . . : Passed
 

    Adapter : Private
 

        Netcard queries test . . . : Passed
 

        Host Name. . . . . . . . . : server

        IP Address . . . . . . . . : 10.4.1.129

        Subnet Mask. . . . . . . . : 255.255.252.0

        Default Gateway. . . . . . :

        Primary WINS Server. . . . : 10.4.0.2

        Dns Servers. . . . . . . . :
 

        AutoConfiguration results. . . . . . : Passed
 

        Default gateway test . . . : Skipped

            [WARNING] No gateways defined for this adapter.
 

        NetBT name test. . . . . . : Passed

        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messenge

r Service', <20> 'WINS' names is missing.
 

        WINS service test. . . . . : Passed
 
 

Global results:
 
 

Domain membership test . . . . . . : Passed
 
 

NetBT transports test. . . . . . . : Passed

    List of NetBt transports currently configured:

        NetBT_Tcpip_{99C1335D-E13B-4908-B21A-6387D2A53259}

        NetBT_Tcpip_{BADB6AC8-EF20-42EA-8686-9EE3A8CD1FFC}

    2 NetBt transports currently configured.
 
 

Autonet address test . . . . . . . : Passed
 
 

IP loopback ping test. . . . . . . : Passed
 
 

Default gateway test . . . . . . . : Passed
 
 

NetBT name test. . . . . . . . . . : Passed

    [WARNING] You don't have a single interface with the <00> 'WorkStation Servi

ce', <03> 'Messenger Service', <20> 'WINS' names defined.
 
 

Winsock test . . . . . . . . . . . : Passed
 
 

DNS test . . . . . . . . . . . . . : Passed
 
 

Redir and Browser test . . . . . . : Passed

    List of NetBt transports currently bound to the Redir

        NetBT_Tcpip_{99C1335D-E13B-4908-B21A-6387D2A53259}

        NetBT_Tcpip_{BADB6AC8-EF20-42EA-8686-9EE3A8CD1FFC}

    The redir is bound to 2 NetBt transports.
 

    List of NetBt transports currently bound to the browser

        NetBT_Tcpip_{99C1335D-E13B-4908-B21A-6387D2A53259}

        NetBT_Tcpip_{BADB6AC8-EF20-42EA-8686-9EE3A8CD1FFC}

    The browser is bound to 2 NetBt transports.
 
 

DC discovery test. . . . . . . . . : Passed
 
 

DC list test . . . . . . . . . . . : Passed
 
 

Trust relationship test. . . . . . : Passed

    Secure channel for domain 'domain' is to '\\Shee.domain.edu'.
 
 

Kerberos test. . . . . . . . . . . : Passed
 
 

LDAP test. . . . . . . . . . . . . : Passed
 
 

Bindings test. . . . . . . . . . . : Passed
 
 

WAN configuration test . . . . . . : Skipped

    No active remote access connections.
 
 

Modem diagnostics test . . . . . . : Passed
 

IP Security test . . . . . . . . . : Skipped
 

    Note: run "netsh ipsec dynamic show /?" for more detailed information
 
 

The command completed successfully

Open in new window

0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
@CorruptedLogic
The jumbo is the same as mtu for this nic, and has been checked to match the maximum of the switch and matches the working server.

AV was disabled, and results remain unchanged.
0
 
LVL 7

Accepted Solution

by:
CorruptedLogic earned 300 total points
Comment Utility
Shot in the dark, but, could it be the configuration of the RAID array in the new server? I know that certain RAID levels give much slower performance than others. do both servers have the same RAID level (ie, are they both RAID5, 1, 10 etc?)
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
:)
Good thought CorruptedLogic, I am beginning to wander down that road also.  But I have checked all of the system drivers, MB, NIC, scsi, everything, they are all current, and both run on Raid 1
0
 
LVL 7

Expert Comment

by:CorruptedLogic
Comment Utility
Could you post specs of both servers? Also, is the new server running any other app or is it just sat there awaiting a data dump?
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
Secondary WINS Server. . . : xxx.xxx.14.83

Your secondary WINS server is on a different subnet. It might have been mistyped.
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
@CurruptedLogic
The 2 servers are radically different, old was HP and new is dell, so all new hardware specs.  can you be more specific about what you are looking for?

@CheifIT
Thanks, bu tit is correct, we are moving ip's right now, but the 2 servers in question are in the same subnet.  I was curious about this entry in the log though, not sure what it means;

       NetBT name test. . . . . . : Passed
        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messenge
r Service', <20> 'WINS' names is missing.

0
 
LVL 7

Expert Comment

by:CorruptedLogic
Comment Utility
I wasn't looking for anything specific, just gathering a little more info/background to see if any light bulbs flickered on. Not wanting to beat a dead horse, but, you said that you checked the speed/duplex settings, are they set to auto, or are they manually configured all the way down the line from server to switch to whatever the next hop is?
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
No problem CorruptedLogic, I have beaten this horse severely, so much I think PETA is out to get me.

I thought duplex and speed also, first thing.  But i have confirmed it multiple times, it is running at gig full.  I have tested it using both forced and auto.  Right now, I think it is forced, and the switch shows no collisions or errors.
0
 
LVL 7

Expert Comment

by:CorruptedLogic
Comment Utility
Could you connect the new server via a cross over cable to another machine and test the throughput? That will at least narrow it down to either a network issue or a local machine issue.
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
It's been a long time since I went to the University of Washington. Being a heavily Microsoft funded school we often corelated math to computer science. I do remember in Matrix Algebra when we were shown binary Matricies and how a computer checksum works. It literaly adds up the zeros and ones to check the data for an inconsitant sum. If inconsisten, it will request the data packet again. The result is a lot of extra communications back and forth before a proper download.

Things like checksums and encryption protocols may slow down the server because of the extra data packets or the mod-demod of the data, or the checks and balances of the data for integrity.

Are you riding on an encrypted protocol?
Are you using checksums or extra data integrity checks and balances?
Are you encorporating extra packets, depending upon the protocol used?

All of these would have a factor on speed.

For an encryption protocol, look at IEEE 802.1x as an example. Those packets are encompassed within encryption packets. So, the encryption has to be somewhat modulated in and demodulated out.
http://en.wikipedia.org/wiki/802.1x

So, there are really a number of means that can appear to slow down a server's upload and download speed. Truth is, the data packets contents are the true test of what is going on.

You might consider a packet capture utility to see the entire contents of the packet. Can anyone suggest such a utility?

 
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
Sorry guys, I was out in training last couple days.  Chief, I will check this today soon.  CL, I have confirmed the physical network is not the issue.
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
UGH!  had a virus, had to wipe my machine and start over :(
I am trying to get back to this, it is my #1 priority till it is fixed
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
Ok, I am back at this, not sure if anyoe is still listening.  

Are you riding on an encrypted protocol?
no encryption
Are you using checksums or extra data integrity checks and balances?
i don't think so, how would I check this?
Are you encorporating extra packets, depending upon the protocol used?
i don't think so, how would I check this?

This is a simple network, that resides behind an acl for security.  Bu thtis is an opld system, how would I check if these could be true?

I will try capturing a packet today.
0
 
LVL 7

Expert Comment

by:CorruptedLogic
Comment Utility
Shot in the dark, are you using the same version of Robocopy on both servers? Could it be that the  new server is doing some sort of error checking whilst copying?
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 11

Author Comment

by:loftyworm
Comment Utility
how would i check?

0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
Ok, here is the packet capture.  #2 is of the working system, and #1 is of the broken system.  This is a single packet that was captured while under load.

sorry for the screen shorts, MS network tool does not export in txt :(

1.bmp
2.bmp
0
 
LVL 7

Expert Comment

by:CorruptedLogic
Comment Utility
The only thing i can see that looks even vaguely obvious is the TCP window size is different on each capture (whether the difference is enough to be significant, i don't know). I ripped this from wikipedia in case you are not familiar with TCP window sizing: Window scaling

    Main article: TCP window scale option

For more efficient use of high bandwidth networks, a larger TCP window size may be used. The TCP window size field controls the flow of data and is limited to between 2 and 65,535 bytes.

Since the size field cannot be expanded, a scaling factor is used. The TCP window scale option, as defined in RFC 1323, is an option used to increase the maximum window size from 65,535 bytes to 1 Gigabyte. Scaling up to larger window sizes is a part of what is necessary for TCP Tuning.

The window scale option is used only during the TCP 3-way handshake. The window scale value represents the number of bits to left-shift the 16-bit window size field. The window scale value can be set from 0 (no shift) to 14.

Many routers and packet firewalls rewrite the window scaling factor during a transmission. This causes sending and receiving sides to assume different TCP window sizes. The result is non-stable traffic that is very slow. The problem is visible on some sending and receiving sites which are behind the path of broken routers.

TCP window scaling can be a particular problem on Linux and Windows Vista systems.


Might be something worth checking out.
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
I will check.  Sounds a bit above my head, but I have network guys I can ask about this.
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
It appears the window scaling is dynamic, that is why they are differant.  also, the fast and slow system are set to the same.  any more thoughts?
0
 
LVL 7

Expert Comment

by:CorruptedLogic
Comment Utility
Would this pertain to your situation at all? (you probably read this already) http://support.microsoft.com/kb/321098
0
 
LVL 7

Expert Comment

by:CorruptedLogic
Comment Utility
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
I have read through these, no go :(
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
Sorry for not being real proactive:

I just went through Hell week.

Are you using checksums or extra data integrity checks and balances?
i don't think so, how would I check this?

I think you might wish to read this:
http://support.microsoft.com/kb/932170
___________________________________________________________________________
Are you encorporating extra packets, depending upon the protocol used?
i don't think so, how would I check this?

There is a protocol called 802.x that encompases the packets within other packets. So the 802.x packets need to be stripped off prior to getting your true TCP packet.
http://en.wikipedia.org/wiki/IEEE_802

(But, I don't think this is your problem)

_____________________________________________________________________




0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
I have the dell vendors calling today, I think I got them to escalate it.  I will also check this when I get a chance.

Thanks to both of you for keeping at this with me :)
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
LW,

Another thing I ran into lately is when Client Services for Netware was a service that was in a higher order within the binding than Microsoft Windows Client Service.

Here, check these out.

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/2003_Server/Q_23893096.html?sfQueryTermInfo=1+chiefit+client+netwar+servic

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/2003_Server/Q_23399463.html?sfQueryTermInfo=1+chiefit+client+netwar+servic
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
 Thanks for the note!  I will do, I am still struggling with this.
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
Anything noteworthy?
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
Not yet, I am still working it.  I had fulls that had to run, and they jsut got odne today.  Dell is at a total loss, so I am trying to figure out how to image the system, then I am going to rebuild it, see if that helps :(
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
You know another thing I found is:

Mapped network drives and shares that don't exists can considerably slow network traffic. When clicking on the share or printer that doesn't exist, it can freeze your browser. Also logons are very slow with network logons when this happens.
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
I have found this also, but attached to specific user profiles, here there are no mapped drives or printers.  I will check this anyway though
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
One last train of thought is the MTU channels. You could do an MTU ping to test the Maximum Transfer Units:

http://help.expedient.com/broadband/mtu_ping_test.shtml
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
I am putting out fires over Christmas, but I am still dealing with this, thanks for keeping me in mind. I am having problems imageing the system so I can test a rebuild, but I will keep this forum updated as I can test and make changes.
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
For those still paying attention to this thread, I think I have finally found out what is going on.  this is a RAID1 system, mirrored, and the disk is a SAS 3 gbps meaning the HD will only copy at 384 MB/min.  I confirmed other systems with the same raid setup and hardware exibit the same symptoms, and I am testing now.
0
 
LVL 7

Expert Comment

by:CorruptedLogic
Comment Utility
Interesting. What was the configuration of the old server (just out of interest!)
0
 
LVL 38

Expert Comment

by:ChiefIT
Comment Utility
Hmm.

I am all ears. Let us know how your doing.
0
 
LVL 11

Author Comment

by:loftyworm
Comment Utility
I am still waiting on parts, but I am keeping this open so I am reminded to check in with you guys :)
0
 
LVL 38

Assisted Solution

by:ChiefIT
ChiefIT earned 200 total points
Comment Utility
LW:

It's been a while and I was reviewing questions of authors that wrote and I am replying to all of them.

When reviewing this question I find that netbios is bound to both network addapters (public and private).

You might consider disabling netbios on the public adapter. The netbios ports and SMB ports are highly targeted. So, on the public adapter disable file and printer sharing and also prevent netbios over DHCP or netbios over TCP/IP.

0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Setting up a Microsoft WSUS update system is free relatively speaking if you have hard disk space and processor capacity.   However, WSUS can be a blessing and a curse. For example, there is nothing worse than approving updates and they just have…
Resolve DNS query failed errors for Exchange
This tutorial will walk an individual through the steps necessary to join and promote the first Windows Server 2012 domain controller into an Active Directory environment running on Windows Server 2008. Determine the location of the FSMO roles by lo…
This tutorial will walk an individual through the process of transferring the five major, necessary Active Directory Roles, commonly referred to as the FSMO roles to another domain controller. Log onto the new domain controller with a user account t…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now