?
Solved

Slow network - tough one

Posted on 2008-10-31
45
Medium Priority
?
367 Views
Last Modified: 2012-05-05
Windows 2003 R2, sp2
broadcom card - onboard

I have an old server, I run a copy test (robocopy) and transfer rate is 500-900 MB/min
I build a new server, plug the same network cable into the new server, and the transfer rate is about 190 MB/min.

I have checked the duplex and speed are correct
I checked the mtu is correct
there are no errors or collisions on the switch (my network guys say all looks great)
I called the vendor (dell) and validated ALL the drivers are current
I added an Intel card with current drivers, still no change
I flashed the bios, BMC,DRAC, and SATA - no change
Re-checked the old system on the same network is still working

I am at a loss,  I would suspect hardware, but the additional Intel should have bypassed the bad card.  I have a heavy backup cycle in 7 days, and this is urgent, any help would be wonderful

Thank you advance!
0
Comment
Question by:loftyworm
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 21
  • 12
  • 11
  • +1
45 Comments
 
LVL 14

Expert Comment

by:dfxdeimos
ID: 22854688
Have you tried a different switch port (just for SnG)?

Are both servers running the same OS?
0
 
LVL 7

Expert Comment

by:CorruptedLogic
ID: 22854912
Along with the switch port, go ahead and swap out the network cable. Is the new server on the same network segment as the old server (ie, does it pass through additional routers/switches that the old server does not have to contend with?).
If the situation permits, can you try connecting the new server directly to the destination machine via a crossover cable?
Does the old server have something like jumbo packets enabled?
0
 
LVL 39

Expert Comment

by:ChiefIT
ID: 22856284
What does netdiag and DCdiag dig up for you to work with?
0
Does Powershell have you tied up in knots?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

 
LVL 39

Expert Comment

by:ChiefIT
ID: 22856286
Also, check your Win32 time synchronization.
0
 
LVL 11

Author Comment

by:loftyworm
ID: 22869795
@dfxdeimos
The same working network(cables, swtiches routers) that were plugged into the working server, are now plugged into the new server.  I simply unpluged the old and plugged in the new.  I also went back to the old once to validate it was still working.

@CorruptedLogic
I did check the MTU size, but how would I check for jumbo packet??

@CheifIT
This is not a domain controller, but I will check the netdiag and time.
0
 
LVL 7

Expert Comment

by:CorruptedLogic
ID: 22869872
You would usually check whether or not Jumbo Packets were enabled on the NIC properties page (it is usually listed along with speed/duplex options etc, if it is available on your particualr nic). Depending on your switches, it can also be enabled/disabled on a per port basis at the switch level.
0
 
LVL 7

Expert Comment

by:CorruptedLogic
ID: 22869905
Just another thought, does the new server have any antivirus software that may be scanning the files during the copy process? That could certainly slow things down.
0
 
LVL 11

Author Comment

by:loftyworm
ID: 22869994
@ChiefIT
The time is fine, and both systems are checking into the same ntp server, so they are synced.

the netdiag all passed, but it was odd that it thought the OS was 2k, and not 2k3r2, but I assume this is some version quirk.

D:\Program Files\Support Tools>netdiag.exe
 
............................................
 
    Computer Name: server
    DNS Host Name: server.domain.edu
    System info : Windows 2000 Server (Build 3790)
    Processor : x86 Family 6 Model 23 Stepping 6, GenuineIntel
    List of installed hotfixes :
        KB924667-v2
        KB925398_WMP64
        KB925876
        KB925902
        KB926122
        KB927891
        KB929123
        KB930178
        KB931784
        KB931836
        KB932168
        KB933729
        KB933854
        KB935839
        KB935840
        KB936021
        KB936357
        KB936594
        KB936782
        KB938127
        KB938127-IE7
        KB938464
        KB941202
        KB941568
        KB941569
        KB941644
        KB941693
        KB941716
        KB942615
        KB942763
        KB943055
        KB943460
        KB943485
        KB943545
        KB943729
        KB944338
        KB944653
        KB945553
        KB946026
        KB947864
        KB947864-IE7
        KB948496
        KB948590
        KB948881
        KB950759-IE7
        KB950760
        KB950762
        KB950974
        KB951066
        KB951072-v2
        KB951698
        KB951748
        KB952954
        KB953838-IE7
        KB953839
        KB954211
        KB956390-IE7
        KB956391
        KB956803
        KB956841
        KB957095
        KB958644
        Q147222
 
 
Netcard queries test . . . . . . . : Passed
 
 
 
Per interface results:
 
    Adapter : Public
 
        Netcard queries test . . . : Passed
 
        Host Name. . . . . . . . . : server.domain.edu
        IP Address . . . . . . . . : xxx.xxx.34.235
        Subnet Mask. . . . . . . . : 255.255.255.0
        Default Gateway. . . . . . : xxx.xxx.34.1
        Primary WINS Server. . . . : xxx.xxx.34.65
        Secondary WINS Server. . . : xxx.xxx.14.83
        Dns Servers. . . . . . . . : xxx.xxx.34.137
                                     xxx.xxx.34.121
 
 
        AutoConfiguration results. . . . . . : Passed
 
        Default gateway test . . . : Passed
 
        NetBT name test. . . . . . : Passed
        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messenge
r Service', <20> 'WINS' names is missing.
 
        WINS service test. . . . . : Passed
 
    Adapter : Private
 
        Netcard queries test . . . : Passed
 
        Host Name. . . . . . . . . : server
        IP Address . . . . . . . . : 10.4.1.129
        Subnet Mask. . . . . . . . : 255.255.252.0
        Default Gateway. . . . . . :
        Primary WINS Server. . . . : 10.4.0.2
        Dns Servers. . . . . . . . :
 
        AutoConfiguration results. . . . . . : Passed
 
        Default gateway test . . . : Skipped
            [WARNING] No gateways defined for this adapter.
 
        NetBT name test. . . . . . : Passed
        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messenge
r Service', <20> 'WINS' names is missing.
 
        WINS service test. . . . . : Passed
 
 
Global results:
 
 
Domain membership test . . . . . . : Passed
 
 
NetBT transports test. . . . . . . : Passed
    List of NetBt transports currently configured:
        NetBT_Tcpip_{99C1335D-E13B-4908-B21A-6387D2A53259}
        NetBT_Tcpip_{BADB6AC8-EF20-42EA-8686-9EE3A8CD1FFC}
    2 NetBt transports currently configured.
 
 
Autonet address test . . . . . . . : Passed
 
 
IP loopback ping test. . . . . . . : Passed
 
 
Default gateway test . . . . . . . : Passed
 
 
NetBT name test. . . . . . . . . . : Passed
    [WARNING] You don't have a single interface with the <00> 'WorkStation Servi
ce', <03> 'Messenger Service', <20> 'WINS' names defined.
 
 
Winsock test . . . . . . . . . . . : Passed
 
 
DNS test . . . . . . . . . . . . . : Passed
 
 
Redir and Browser test . . . . . . : Passed
    List of NetBt transports currently bound to the Redir
        NetBT_Tcpip_{99C1335D-E13B-4908-B21A-6387D2A53259}
        NetBT_Tcpip_{BADB6AC8-EF20-42EA-8686-9EE3A8CD1FFC}
    The redir is bound to 2 NetBt transports.
 
    List of NetBt transports currently bound to the browser
        NetBT_Tcpip_{99C1335D-E13B-4908-B21A-6387D2A53259}
        NetBT_Tcpip_{BADB6AC8-EF20-42EA-8686-9EE3A8CD1FFC}
    The browser is bound to 2 NetBt transports.
 
 
DC discovery test. . . . . . . . . : Passed
 
 
DC list test . . . . . . . . . . . : Passed
 
 
Trust relationship test. . . . . . : Passed
    Secure channel for domain 'domain' is to '\\Shee.domain.edu'.
 
 
Kerberos test. . . . . . . . . . . : Passed
 
 
LDAP test. . . . . . . . . . . . . : Passed
 
 
Bindings test. . . . . . . . . . . : Passed
 
 
WAN configuration test . . . . . . : Skipped
    No active remote access connections.
 
 
Modem diagnostics test . . . . . . : Passed
 
IP Security test . . . . . . . . . : Skipped
 
    Note: run "netsh ipsec dynamic show /?" for more detailed information
 
 
The command completed successfully

Open in new window

0
 
LVL 11

Author Comment

by:loftyworm
ID: 22870406
@CorruptedLogic
The jumbo is the same as mtu for this nic, and has been checked to match the maximum of the switch and matches the working server.

AV was disabled, and results remain unchanged.
0
 
LVL 7

Accepted Solution

by:
CorruptedLogic earned 1200 total points
ID: 22870487
Shot in the dark, but, could it be the configuration of the RAID array in the new server? I know that certain RAID levels give much slower performance than others. do both servers have the same RAID level (ie, are they both RAID5, 1, 10 etc?)
0
 
LVL 11

Author Comment

by:loftyworm
ID: 22870823
:)
Good thought CorruptedLogic, I am beginning to wander down that road also.  But I have checked all of the system drivers, MB, NIC, scsi, everything, they are all current, and both run on Raid 1
0
 
LVL 7

Expert Comment

by:CorruptedLogic
ID: 22870946
Could you post specs of both servers? Also, is the new server running any other app or is it just sat there awaiting a data dump?
0
 
LVL 39

Expert Comment

by:ChiefIT
ID: 22871112
Secondary WINS Server. . . : xxx.xxx.14.83

Your secondary WINS server is on a different subnet. It might have been mistyped.
0
 
LVL 11

Author Comment

by:loftyworm
ID: 22872010
@CurruptedLogic
The 2 servers are radically different, old was HP and new is dell, so all new hardware specs.  can you be more specific about what you are looking for?

@CheifIT
Thanks, bu tit is correct, we are moving ip's right now, but the 2 servers in question are in the same subnet.  I was curious about this entry in the log though, not sure what it means;

       NetBT name test. . . . . . : Passed
        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messenge
r Service', <20> 'WINS' names is missing.

0
 
LVL 7

Expert Comment

by:CorruptedLogic
ID: 22872103
I wasn't looking for anything specific, just gathering a little more info/background to see if any light bulbs flickered on. Not wanting to beat a dead horse, but, you said that you checked the speed/duplex settings, are they set to auto, or are they manually configured all the way down the line from server to switch to whatever the next hop is?
0
 
LVL 11

Author Comment

by:loftyworm
ID: 22872412
No problem CorruptedLogic, I have beaten this horse severely, so much I think PETA is out to get me.

I thought duplex and speed also, first thing.  But i have confirmed it multiple times, it is running at gig full.  I have tested it using both forced and auto.  Right now, I think it is forced, and the switch shows no collisions or errors.
0
 
LVL 7

Expert Comment

by:CorruptedLogic
ID: 22877334
Could you connect the new server via a cross over cable to another machine and test the throughput? That will at least narrow it down to either a network issue or a local machine issue.
0
 
LVL 39

Expert Comment

by:ChiefIT
ID: 22883824
It's been a long time since I went to the University of Washington. Being a heavily Microsoft funded school we often corelated math to computer science. I do remember in Matrix Algebra when we were shown binary Matricies and how a computer checksum works. It literaly adds up the zeros and ones to check the data for an inconsitant sum. If inconsisten, it will request the data packet again. The result is a lot of extra communications back and forth before a proper download.

Things like checksums and encryption protocols may slow down the server because of the extra data packets or the mod-demod of the data, or the checks and balances of the data for integrity.

Are you riding on an encrypted protocol?
Are you using checksums or extra data integrity checks and balances?
Are you encorporating extra packets, depending upon the protocol used?

All of these would have a factor on speed.

For an encryption protocol, look at IEEE 802.1x as an example. Those packets are encompassed within encryption packets. So, the encryption has to be somewhat modulated in and demodulated out.
http://en.wikipedia.org/wiki/802.1x

So, there are really a number of means that can appear to slow down a server's upload and download speed. Truth is, the data packets contents are the true test of what is going on.

You might consider a packet capture utility to see the entire contents of the packet. Can anyone suggest such a utility?

 
0
 
LVL 11

Author Comment

by:loftyworm
ID: 22906414
Sorry guys, I was out in training last couple days.  Chief, I will check this today soon.  CL, I have confirmed the physical network is not the issue.
0
 
LVL 11

Author Comment

by:loftyworm
ID: 22935042
UGH!  had a virus, had to wipe my machine and start over :(
I am trying to get back to this, it is my #1 priority till it is fixed
0
 
LVL 11

Author Comment

by:loftyworm
ID: 22979244
Ok, I am back at this, not sure if anyoe is still listening.  

Are you riding on an encrypted protocol?
no encryption
Are you using checksums or extra data integrity checks and balances?
i don't think so, how would I check this?
Are you encorporating extra packets, depending upon the protocol used?
i don't think so, how would I check this?

This is a simple network, that resides behind an acl for security.  Bu thtis is an opld system, how would I check if these could be true?

I will try capturing a packet today.
0
 
LVL 7

Expert Comment

by:CorruptedLogic
ID: 22979463
Shot in the dark, are you using the same version of Robocopy on both servers? Could it be that the  new server is doing some sort of error checking whilst copying?
0
 
LVL 11

Author Comment

by:loftyworm
ID: 22980380
how would i check?

0
 
LVL 11

Author Comment

by:loftyworm
ID: 22981087
Ok, here is the packet capture.  #2 is of the working system, and #1 is of the broken system.  This is a single packet that was captured while under load.

sorry for the screen shorts, MS network tool does not export in txt :(

1.bmp
2.bmp
0
 
LVL 7

Expert Comment

by:CorruptedLogic
ID: 22985777
The only thing i can see that looks even vaguely obvious is the TCP window size is different on each capture (whether the difference is enough to be significant, i don't know). I ripped this from wikipedia in case you are not familiar with TCP window sizing: Window scaling

    Main article: TCP window scale option

For more efficient use of high bandwidth networks, a larger TCP window size may be used. The TCP window size field controls the flow of data and is limited to between 2 and 65,535 bytes.

Since the size field cannot be expanded, a scaling factor is used. The TCP window scale option, as defined in RFC 1323, is an option used to increase the maximum window size from 65,535 bytes to 1 Gigabyte. Scaling up to larger window sizes is a part of what is necessary for TCP Tuning.

The window scale option is used only during the TCP 3-way handshake. The window scale value represents the number of bits to left-shift the 16-bit window size field. The window scale value can be set from 0 (no shift) to 14.

Many routers and packet firewalls rewrite the window scaling factor during a transmission. This causes sending and receiving sides to assume different TCP window sizes. The result is non-stable traffic that is very slow. The problem is visible on some sending and receiving sites which are behind the path of broken routers.

TCP window scaling can be a particular problem on Linux and Windows Vista systems.


Might be something worth checking out.
0
 
LVL 11

Author Comment

by:loftyworm
ID: 22987241
I will check.  Sounds a bit above my head, but I have network guys I can ask about this.
0
 
LVL 11

Author Comment

by:loftyworm
ID: 22990540
It appears the window scaling is dynamic, that is why they are differant.  also, the fast and slow system are set to the same.  any more thoughts?
0
 
LVL 7

Expert Comment

by:CorruptedLogic
ID: 22990560
Would this pertain to your situation at all? (you probably read this already) http://support.microsoft.com/kb/321098
0
 
LVL 7

Expert Comment

by:CorruptedLogic
ID: 22990582
0
 
LVL 11

Author Comment

by:loftyworm
ID: 23007556
I have read through these, no go :(
0
 
LVL 39

Expert Comment

by:ChiefIT
ID: 23039573
Sorry for not being real proactive:

I just went through Hell week.

Are you using checksums or extra data integrity checks and balances?
i don't think so, how would I check this?

I think you might wish to read this:
http://support.microsoft.com/kb/932170
___________________________________________________________________________
Are you encorporating extra packets, depending upon the protocol used?
i don't think so, how would I check this?

There is a protocol called 802.x that encompases the packets within other packets. So the 802.x packets need to be stripped off prior to getting your true TCP packet.
http://en.wikipedia.org/wiki/IEEE_802

(But, I don't think this is your problem)

_____________________________________________________________________




0
 
LVL 11

Author Comment

by:loftyworm
ID: 23043987
I have the dell vendors calling today, I think I got them to escalate it.  I will also check this when I get a chance.

Thanks to both of you for keeping at this with me :)
0
 
LVL 39

Expert Comment

by:ChiefIT
ID: 23074742
LW,

Another thing I ran into lately is when Client Services for Netware was a service that was in a higher order within the binding than Microsoft Windows Client Service.

Here, check these out.

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/2003_Server/Q_23893096.html?sfQueryTermInfo=1+chiefit+client+netwar+servic

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/2003_Server/Q_23399463.html?sfQueryTermInfo=1+chiefit+client+netwar+servic
0
 
LVL 11

Author Comment

by:loftyworm
ID: 23079196
 Thanks for the note!  I will do, I am still struggling with this.
0
 
LVL 39

Expert Comment

by:ChiefIT
ID: 23155354
Anything noteworthy?
0
 
LVL 11

Author Comment

by:loftyworm
ID: 23160388
Not yet, I am still working it.  I had fulls that had to run, and they jsut got odne today.  Dell is at a total loss, so I am trying to figure out how to image the system, then I am going to rebuild it, see if that helps :(
0
 
LVL 39

Expert Comment

by:ChiefIT
ID: 23168666
You know another thing I found is:

Mapped network drives and shares that don't exists can considerably slow network traffic. When clicking on the share or printer that doesn't exist, it can freeze your browser. Also logons are very slow with network logons when this happens.
0
 
LVL 11

Author Comment

by:loftyworm
ID: 23186271
I have found this also, but attached to specific user profiles, here there are no mapped drives or printers.  I will check this anyway though
0
 
LVL 39

Expert Comment

by:ChiefIT
ID: 23236629
One last train of thought is the MTU channels. You could do an MTU ping to test the Maximum Transfer Units:

http://help.expedient.com/broadband/mtu_ping_test.shtml
0
 
LVL 11

Author Comment

by:loftyworm
ID: 23236653
I am putting out fires over Christmas, but I am still dealing with this, thanks for keeping me in mind. I am having problems imageing the system so I can test a rebuild, but I will keep this forum updated as I can test and make changes.
0
 
LVL 11

Author Comment

by:loftyworm
ID: 23425994
For those still paying attention to this thread, I think I have finally found out what is going on.  this is a RAID1 system, mirrored, and the disk is a SAS 3 gbps meaning the HD will only copy at 384 MB/min.  I confirmed other systems with the same raid setup and hardware exibit the same symptoms, and I am testing now.
0
 
LVL 7

Expert Comment

by:CorruptedLogic
ID: 23431548
Interesting. What was the configuration of the old server (just out of interest!)
0
 
LVL 39

Expert Comment

by:ChiefIT
ID: 23483723
Hmm.

I am all ears. Let us know how your doing.
0
 
LVL 11

Author Comment

by:loftyworm
ID: 23488656
I am still waiting on parts, but I am keeping this open so I am reminded to check in with you guys :)
0
 
LVL 39

Assisted Solution

by:ChiefIT
ChiefIT earned 800 total points
ID: 23666020
LW:

It's been a while and I was reviewing questions of authors that wrote and I am replying to all of them.

When reviewing this question I find that netbios is bound to both network addapters (public and private).

You might consider disabling netbios on the public adapter. The netbios ports and SMB ports are highly targeted. So, on the public adapter disable file and printer sharing and also prevent netbios over DHCP or netbios over TCP/IP.

0

Featured Post

Free learning courses: Active Directory Deep Dive

Get a firm grasp on your IT environment when you learn Active Directory best practices with Veeam! Watch all, or choose any amount, of this three-part webinar series to improve your skills. From the basics to virtualization and backup, we got you covered.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Did you know that more than 4 billion data records have been recorded as lost or stolen since 2013? It was a staggering number brought to our attention during last week’s ManageEngine webinar, where attendees received a comprehensive look at the ma…
Auditing domain password hashes is a commonly overlooked but critical requirement to ensuring secure passwords practices are followed. Methods exist to extract hashes directly for a live domain however this article describes a process to extract u…
This tutorial will walk an individual through the process of configuring their Windows Server 2012 domain controller to synchronize its time with a trusted, external resource. Use Google, Bing, or other preferred search engine to locate trusted NTP …
Are you ready to implement Active Directory best practices without reading 300+ pages? You're in luck. In this webinar hosted by Skyport Systems, you gain insight into Microsoft's latest comprehensive guide, with tips on the best and easiest way…
Suggested Courses

800 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question