• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 368
  • Last Modified:

Slow network - tough one

Windows 2003 R2, sp2
broadcom card - onboard

I have an old server, I run a copy test (robocopy) and transfer rate is 500-900 MB/min
I build a new server, plug the same network cable into the new server, and the transfer rate is about 190 MB/min.

I have checked the duplex and speed are correct
I checked the mtu is correct
there are no errors or collisions on the switch (my network guys say all looks great)
I called the vendor (dell) and validated ALL the drivers are current
I added an Intel card with current drivers, still no change
I flashed the bios, BMC,DRAC, and SATA - no change
Re-checked the old system on the same network is still working

I am at a loss,  I would suspect hardware, but the additional Intel should have bypassed the bad card.  I have a heavy backup cycle in 7 days, and this is urgent, any help would be wonderful

Thank you advance!
0
loftyworm
Asked:
loftyworm
  • 21
  • 12
  • 11
  • +1
2 Solutions
 
dfxdeimosCommented:
Have you tried a different switch port (just for SnG)?

Are both servers running the same OS?
0
 
CorruptedLogicCommented:
Along with the switch port, go ahead and swap out the network cable. Is the new server on the same network segment as the old server (ie, does it pass through additional routers/switches that the old server does not have to contend with?).
If the situation permits, can you try connecting the new server directly to the destination machine via a crossover cable?
Does the old server have something like jumbo packets enabled?
0
 
ChiefITCommented:
What does netdiag and DCdiag dig up for you to work with?
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
ChiefITCommented:
Also, check your Win32 time synchronization.
0
 
loftywormAuthor Commented:
@dfxdeimos
The same working network(cables, swtiches routers) that were plugged into the working server, are now plugged into the new server.  I simply unpluged the old and plugged in the new.  I also went back to the old once to validate it was still working.

@CorruptedLogic
I did check the MTU size, but how would I check for jumbo packet??

@CheifIT
This is not a domain controller, but I will check the netdiag and time.
0
 
CorruptedLogicCommented:
You would usually check whether or not Jumbo Packets were enabled on the NIC properties page (it is usually listed along with speed/duplex options etc, if it is available on your particualr nic). Depending on your switches, it can also be enabled/disabled on a per port basis at the switch level.
0
 
CorruptedLogicCommented:
Just another thought, does the new server have any antivirus software that may be scanning the files during the copy process? That could certainly slow things down.
0
 
loftywormAuthor Commented:
@ChiefIT
The time is fine, and both systems are checking into the same ntp server, so they are synced.

the netdiag all passed, but it was odd that it thought the OS was 2k, and not 2k3r2, but I assume this is some version quirk.

D:\Program Files\Support Tools>netdiag.exe
 
............................................
 
    Computer Name: server
    DNS Host Name: server.domain.edu
    System info : Windows 2000 Server (Build 3790)
    Processor : x86 Family 6 Model 23 Stepping 6, GenuineIntel
    List of installed hotfixes :
        KB924667-v2
        KB925398_WMP64
        KB925876
        KB925902
        KB926122
        KB927891
        KB929123
        KB930178
        KB931784
        KB931836
        KB932168
        KB933729
        KB933854
        KB935839
        KB935840
        KB936021
        KB936357
        KB936594
        KB936782
        KB938127
        KB938127-IE7
        KB938464
        KB941202
        KB941568
        KB941569
        KB941644
        KB941693
        KB941716
        KB942615
        KB942763
        KB943055
        KB943460
        KB943485
        KB943545
        KB943729
        KB944338
        KB944653
        KB945553
        KB946026
        KB947864
        KB947864-IE7
        KB948496
        KB948590
        KB948881
        KB950759-IE7
        KB950760
        KB950762
        KB950974
        KB951066
        KB951072-v2
        KB951698
        KB951748
        KB952954
        KB953838-IE7
        KB953839
        KB954211
        KB956390-IE7
        KB956391
        KB956803
        KB956841
        KB957095
        KB958644
        Q147222
 
 
Netcard queries test . . . . . . . : Passed
 
 
 
Per interface results:
 
    Adapter : Public
 
        Netcard queries test . . . : Passed
 
        Host Name. . . . . . . . . : server.domain.edu
        IP Address . . . . . . . . : xxx.xxx.34.235
        Subnet Mask. . . . . . . . : 255.255.255.0
        Default Gateway. . . . . . : xxx.xxx.34.1
        Primary WINS Server. . . . : xxx.xxx.34.65
        Secondary WINS Server. . . : xxx.xxx.14.83
        Dns Servers. . . . . . . . : xxx.xxx.34.137
                                     xxx.xxx.34.121
 
 
        AutoConfiguration results. . . . . . : Passed
 
        Default gateway test . . . : Passed
 
        NetBT name test. . . . . . : Passed
        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messenge
r Service', <20> 'WINS' names is missing.
 
        WINS service test. . . . . : Passed
 
    Adapter : Private
 
        Netcard queries test . . . : Passed
 
        Host Name. . . . . . . . . : server
        IP Address . . . . . . . . : 10.4.1.129
        Subnet Mask. . . . . . . . : 255.255.252.0
        Default Gateway. . . . . . :
        Primary WINS Server. . . . : 10.4.0.2
        Dns Servers. . . . . . . . :
 
        AutoConfiguration results. . . . . . : Passed
 
        Default gateway test . . . : Skipped
            [WARNING] No gateways defined for this adapter.
 
        NetBT name test. . . . . . : Passed
        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messenge
r Service', <20> 'WINS' names is missing.
 
        WINS service test. . . . . : Passed
 
 
Global results:
 
 
Domain membership test . . . . . . : Passed
 
 
NetBT transports test. . . . . . . : Passed
    List of NetBt transports currently configured:
        NetBT_Tcpip_{99C1335D-E13B-4908-B21A-6387D2A53259}
        NetBT_Tcpip_{BADB6AC8-EF20-42EA-8686-9EE3A8CD1FFC}
    2 NetBt transports currently configured.
 
 
Autonet address test . . . . . . . : Passed
 
 
IP loopback ping test. . . . . . . : Passed
 
 
Default gateway test . . . . . . . : Passed
 
 
NetBT name test. . . . . . . . . . : Passed
    [WARNING] You don't have a single interface with the <00> 'WorkStation Servi
ce', <03> 'Messenger Service', <20> 'WINS' names defined.
 
 
Winsock test . . . . . . . . . . . : Passed
 
 
DNS test . . . . . . . . . . . . . : Passed
 
 
Redir and Browser test . . . . . . : Passed
    List of NetBt transports currently bound to the Redir
        NetBT_Tcpip_{99C1335D-E13B-4908-B21A-6387D2A53259}
        NetBT_Tcpip_{BADB6AC8-EF20-42EA-8686-9EE3A8CD1FFC}
    The redir is bound to 2 NetBt transports.
 
    List of NetBt transports currently bound to the browser
        NetBT_Tcpip_{99C1335D-E13B-4908-B21A-6387D2A53259}
        NetBT_Tcpip_{BADB6AC8-EF20-42EA-8686-9EE3A8CD1FFC}
    The browser is bound to 2 NetBt transports.
 
 
DC discovery test. . . . . . . . . : Passed
 
 
DC list test . . . . . . . . . . . : Passed
 
 
Trust relationship test. . . . . . : Passed
    Secure channel for domain 'domain' is to '\\Shee.domain.edu'.
 
 
Kerberos test. . . . . . . . . . . : Passed
 
 
LDAP test. . . . . . . . . . . . . : Passed
 
 
Bindings test. . . . . . . . . . . : Passed
 
 
WAN configuration test . . . . . . : Skipped
    No active remote access connections.
 
 
Modem diagnostics test . . . . . . : Passed
 
IP Security test . . . . . . . . . : Skipped
 
    Note: run "netsh ipsec dynamic show /?" for more detailed information
 
 
The command completed successfully

Open in new window

0
 
loftywormAuthor Commented:
@CorruptedLogic
The jumbo is the same as mtu for this nic, and has been checked to match the maximum of the switch and matches the working server.

AV was disabled, and results remain unchanged.
0
 
CorruptedLogicCommented:
Shot in the dark, but, could it be the configuration of the RAID array in the new server? I know that certain RAID levels give much slower performance than others. do both servers have the same RAID level (ie, are they both RAID5, 1, 10 etc?)
0
 
loftywormAuthor Commented:
:)
Good thought CorruptedLogic, I am beginning to wander down that road also.  But I have checked all of the system drivers, MB, NIC, scsi, everything, they are all current, and both run on Raid 1
0
 
CorruptedLogicCommented:
Could you post specs of both servers? Also, is the new server running any other app or is it just sat there awaiting a data dump?
0
 
ChiefITCommented:
Secondary WINS Server. . . : xxx.xxx.14.83

Your secondary WINS server is on a different subnet. It might have been mistyped.
0
 
loftywormAuthor Commented:
@CurruptedLogic
The 2 servers are radically different, old was HP and new is dell, so all new hardware specs.  can you be more specific about what you are looking for?

@CheifIT
Thanks, bu tit is correct, we are moving ip's right now, but the 2 servers in question are in the same subnet.  I was curious about this entry in the log though, not sure what it means;

       NetBT name test. . . . . . : Passed
        [WARNING] At least one of the <00> 'WorkStation Service', <03> 'Messenge
r Service', <20> 'WINS' names is missing.

0
 
CorruptedLogicCommented:
I wasn't looking for anything specific, just gathering a little more info/background to see if any light bulbs flickered on. Not wanting to beat a dead horse, but, you said that you checked the speed/duplex settings, are they set to auto, or are they manually configured all the way down the line from server to switch to whatever the next hop is?
0
 
loftywormAuthor Commented:
No problem CorruptedLogic, I have beaten this horse severely, so much I think PETA is out to get me.

I thought duplex and speed also, first thing.  But i have confirmed it multiple times, it is running at gig full.  I have tested it using both forced and auto.  Right now, I think it is forced, and the switch shows no collisions or errors.
0
 
CorruptedLogicCommented:
Could you connect the new server via a cross over cable to another machine and test the throughput? That will at least narrow it down to either a network issue or a local machine issue.
0
 
ChiefITCommented:
It's been a long time since I went to the University of Washington. Being a heavily Microsoft funded school we often corelated math to computer science. I do remember in Matrix Algebra when we were shown binary Matricies and how a computer checksum works. It literaly adds up the zeros and ones to check the data for an inconsitant sum. If inconsisten, it will request the data packet again. The result is a lot of extra communications back and forth before a proper download.

Things like checksums and encryption protocols may slow down the server because of the extra data packets or the mod-demod of the data, or the checks and balances of the data for integrity.

Are you riding on an encrypted protocol?
Are you using checksums or extra data integrity checks and balances?
Are you encorporating extra packets, depending upon the protocol used?

All of these would have a factor on speed.

For an encryption protocol, look at IEEE 802.1x as an example. Those packets are encompassed within encryption packets. So, the encryption has to be somewhat modulated in and demodulated out.
http://en.wikipedia.org/wiki/802.1x

So, there are really a number of means that can appear to slow down a server's upload and download speed. Truth is, the data packets contents are the true test of what is going on.

You might consider a packet capture utility to see the entire contents of the packet. Can anyone suggest such a utility?

 
0
 
loftywormAuthor Commented:
Sorry guys, I was out in training last couple days.  Chief, I will check this today soon.  CL, I have confirmed the physical network is not the issue.
0
 
loftywormAuthor Commented:
UGH!  had a virus, had to wipe my machine and start over :(
I am trying to get back to this, it is my #1 priority till it is fixed
0
 
loftywormAuthor Commented:
Ok, I am back at this, not sure if anyoe is still listening.  

Are you riding on an encrypted protocol?
no encryption
Are you using checksums or extra data integrity checks and balances?
i don't think so, how would I check this?
Are you encorporating extra packets, depending upon the protocol used?
i don't think so, how would I check this?

This is a simple network, that resides behind an acl for security.  Bu thtis is an opld system, how would I check if these could be true?

I will try capturing a packet today.
0
 
CorruptedLogicCommented:
Shot in the dark, are you using the same version of Robocopy on both servers? Could it be that the  new server is doing some sort of error checking whilst copying?
0
 
loftywormAuthor Commented:
how would i check?

0
 
loftywormAuthor Commented:
Ok, here is the packet capture.  #2 is of the working system, and #1 is of the broken system.  This is a single packet that was captured while under load.

sorry for the screen shorts, MS network tool does not export in txt :(

1.bmp
2.bmp
0
 
CorruptedLogicCommented:
The only thing i can see that looks even vaguely obvious is the TCP window size is different on each capture (whether the difference is enough to be significant, i don't know). I ripped this from wikipedia in case you are not familiar with TCP window sizing: Window scaling

    Main article: TCP window scale option

For more efficient use of high bandwidth networks, a larger TCP window size may be used. The TCP window size field controls the flow of data and is limited to between 2 and 65,535 bytes.

Since the size field cannot be expanded, a scaling factor is used. The TCP window scale option, as defined in RFC 1323, is an option used to increase the maximum window size from 65,535 bytes to 1 Gigabyte. Scaling up to larger window sizes is a part of what is necessary for TCP Tuning.

The window scale option is used only during the TCP 3-way handshake. The window scale value represents the number of bits to left-shift the 16-bit window size field. The window scale value can be set from 0 (no shift) to 14.

Many routers and packet firewalls rewrite the window scaling factor during a transmission. This causes sending and receiving sides to assume different TCP window sizes. The result is non-stable traffic that is very slow. The problem is visible on some sending and receiving sites which are behind the path of broken routers.

TCP window scaling can be a particular problem on Linux and Windows Vista systems.


Might be something worth checking out.
0
 
loftywormAuthor Commented:
I will check.  Sounds a bit above my head, but I have network guys I can ask about this.
0
 
loftywormAuthor Commented:
It appears the window scaling is dynamic, that is why they are differant.  also, the fast and slow system are set to the same.  any more thoughts?
0
 
CorruptedLogicCommented:
Would this pertain to your situation at all? (you probably read this already) http://support.microsoft.com/kb/321098
0
 
loftywormAuthor Commented:
I have read through these, no go :(
0
 
ChiefITCommented:
Sorry for not being real proactive:

I just went through Hell week.

Are you using checksums or extra data integrity checks and balances?
i don't think so, how would I check this?

I think you might wish to read this:
http://support.microsoft.com/kb/932170
___________________________________________________________________________
Are you encorporating extra packets, depending upon the protocol used?
i don't think so, how would I check this?

There is a protocol called 802.x that encompases the packets within other packets. So the 802.x packets need to be stripped off prior to getting your true TCP packet.
http://en.wikipedia.org/wiki/IEEE_802

(But, I don't think this is your problem)

_____________________________________________________________________




0
 
loftywormAuthor Commented:
I have the dell vendors calling today, I think I got them to escalate it.  I will also check this when I get a chance.

Thanks to both of you for keeping at this with me :)
0
 
ChiefITCommented:
LW,

Another thing I ran into lately is when Client Services for Netware was a service that was in a higher order within the binding than Microsoft Windows Client Service.

Here, check these out.

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/2003_Server/Q_23893096.html?sfQueryTermInfo=1+chiefit+client+netwar+servic

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/2003_Server/Q_23399463.html?sfQueryTermInfo=1+chiefit+client+netwar+servic
0
 
loftywormAuthor Commented:
 Thanks for the note!  I will do, I am still struggling with this.
0
 
ChiefITCommented:
Anything noteworthy?
0
 
loftywormAuthor Commented:
Not yet, I am still working it.  I had fulls that had to run, and they jsut got odne today.  Dell is at a total loss, so I am trying to figure out how to image the system, then I am going to rebuild it, see if that helps :(
0
 
ChiefITCommented:
You know another thing I found is:

Mapped network drives and shares that don't exists can considerably slow network traffic. When clicking on the share or printer that doesn't exist, it can freeze your browser. Also logons are very slow with network logons when this happens.
0
 
loftywormAuthor Commented:
I have found this also, but attached to specific user profiles, here there are no mapped drives or printers.  I will check this anyway though
0
 
ChiefITCommented:
One last train of thought is the MTU channels. You could do an MTU ping to test the Maximum Transfer Units:

http://help.expedient.com/broadband/mtu_ping_test.shtml
0
 
loftywormAuthor Commented:
I am putting out fires over Christmas, but I am still dealing with this, thanks for keeping me in mind. I am having problems imageing the system so I can test a rebuild, but I will keep this forum updated as I can test and make changes.
0
 
loftywormAuthor Commented:
For those still paying attention to this thread, I think I have finally found out what is going on.  this is a RAID1 system, mirrored, and the disk is a SAS 3 gbps meaning the HD will only copy at 384 MB/min.  I confirmed other systems with the same raid setup and hardware exibit the same symptoms, and I am testing now.
0
 
CorruptedLogicCommented:
Interesting. What was the configuration of the old server (just out of interest!)
0
 
ChiefITCommented:
Hmm.

I am all ears. Let us know how your doing.
0
 
loftywormAuthor Commented:
I am still waiting on parts, but I am keeping this open so I am reminded to check in with you guys :)
0
 
ChiefITCommented:
LW:

It's been a while and I was reviewing questions of authors that wrote and I am replying to all of them.

When reviewing this question I find that netbios is bound to both network addapters (public and private).

You might consider disabling netbios on the public adapter. The netbios ports and SMB ports are highly targeted. So, on the public adapter disable file and printer sharing and also prevent netbios over DHCP or netbios over TCP/IP.

0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

  • 21
  • 12
  • 11
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now