Avatar of atyar
atyar
 asked on

Unexplained network slowness problems - need suggestions for network monitoring software/hardware.

We have a network in 1 office that is experiencing unexplained slowness.  I tried using wireshark(aka ethereal) to analyze what's going on, but nothing really sticks out, and it's not very user-friendly besides. I'm looking for some concrete suggestions for what software and/or hardware we could use to hopefully narrow down what's going on.  In a nutshell, here's the topology:
1)All workstations are plugged into a brand new Linksys 48-port managed switch with a gigabit uplink to
2)A brand new Cisco 3560 layer-3 switch.  The office's file server, a Dell Windows 2003 server, is plugged into 1 gigabit port on the 3560.  Uplinked from there is a line going from 1 of the 10/100 ports on the 3560 to
3)A dumb 10/100 hub, which also has a pc connected to it that is used for cheap internet monitoring.  Another line goes up from there to
4)A Cisco 2610 router.  The 2610 router has 2 wan interfaces, 1 serial going out to a full T-1, and the other an ethernet going out to a 7Mb cable modem.  All internet traffic is being routed out the cable modem interface, and all inter-office vpn traffic goes out the T-1.

The network slowness preceded the cable modem and the new network switches.  We recently added the cable modem, as the office was maxing out the T-1 with multimedia streaming and large ftp file transfers.  What is happening is people are getting slow and intermittently erroring out network performance going to the file server or to the internet, sometimes as slow as like 60k downloads from the internet, or timeouts copying large files from the lan server.  

We tried running wireshark, both during the day when things were happening, and at night when the network should be relatively quiet (with people's pc's left on), and nothing is really sticking out at us (save for some large ftp transactions).  We're at a bit of a loss, trying to find another way to better monitor the lan performance and pinpoint trouble spots/sources.  Can people recommend some software and/or hardware solutions that we could consider in trying to diagnose this?  Any other ideas?
Network AnalysisSwitches / HubsNetworking

Avatar of undefined
Last Comment
atyar

8/22/2022 - Mon
kruptos

If you used the Wireshark in a switched network did you mirror ports? How were you looking at all the data passing on the network with Wireshark if you have switches in place? Is the wireshark PC pluged into the hub?
 Just asking because you wont see all data passing on the lan if your plugged into a switch with no mirroring.










SOLUTION
bladeeta21

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
bladeeta21

sorry about the e-mail add post i just got the terms e-mailed to me
atyar

ASKER
Wireshark is running on the pc on the hub that otherwise does the internet monitoring.  It doesn't get intra-lan traffic, yes, but it does get all inter-office vpn and internet traffic.  The better solution, that I hadn't heard about until today, is the mirroring on the switch option, althought I haven't had a chance to get that figured out on the linksys switch.

We did have issues with duplex on the router and cable modem mismatching at first, but then we dropped in a switch between the cable modem and the wan interface on the router to handle the duplex negotiation. (I forgot to mention that switch in the topology in the original question).

I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
bladeeta21

What kind of ping times do you get pinging from computer to computer and from computer to router?
Les Moore

NTOP will tell you all the top talkers..
http://www.openxtra.co.uk/products/ntop-xtra.php

You'll get a LOT more information than you will from Wireshark or any other sniffer type program.
pgm554

http://netspeed.stanford.edu/

Try this and post details and results.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
atyar

ASKER
netspeed.stanford.edu results say Checking for middleboxes:done, 10s outbound test=407.77Kb/s, 10s inbound test=883.90Kb/s, "Your PC is connected to a cable/DSL modem"
atyar

ASKER
By the way, there are a few things in our router config that I don't truly understand and that Cisco has had us add over the years when working with the TAC guys.  I've removed the public ip and security info from our config, and I'll post it below.  I would welcome any input people have as to things they think we should add or remove.  Basically, all functions are working (we have inter-office vpns, vpn client connections, radius authentication for vpn clients only, ftp servers that are exposed to the internet, etc.), but I wonder if there are things that aren't particularly optimized and perhaps eating up extra cpu on the router (it's been hovering around 50% lately), especially at the interface configuration level.  For instance, I don't really understand ip cef, 'ip mroute-cache', fair-queue, and ip route-cache flow, and whether they should be turned on or off and on which interfaces.  Also, I don't totally understand ip inspect, particularly whether or not it needs to be applied inbound and outbound on each interface, or just 1 or the other and which.

Ethernet1/0 is the interface out to the cable modem, and Serial0/0 goes out to the T-1, while Ethernet0/0 is the internal LAN interface.  There is a switch between Ethernet1/0 and the cable modem to handle duplex negotiation. We have static routes configured for inter-office network destinations, and the default route points out the cable modem for general internet use.

write t
Building configuration...

Current configuration : 11785 bytes
!
version 12.3
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname ****
!
boot-start-marker
boot-end-marker
!
no logging buffered
no logging console
enable secret 5 ****
enable password ****
!
aaa new-model
!
!
aaa authentication login userauthen group radius
aaa authorization network groupauthor local
aaa session-id common
ip subnet-zero
ip cef
!
!
!
ip inspect udp idle-time 20
ip inspect tcp idle-time 1800
ip inspect tcp finwait-time 1
ip inspect tcp synwait-time 15
ip inspect name **** tcp
ip inspect name **** ftp
ip inspect name **** smtp
ip inspect name **** h323
ip inspect name **** http java-list 3
ip inspect name **** udp
ip audit po max-events 100
!
!
!
!
!
!
!
!
!
!
!
!
fax interface-type fax-mail
username **** password 0 ****
username **** password 0 ****
!
!
!
crypto isakmp policy 4
 hash md5
 authentication pre-share
!
crypto isakmp policy 5
 authentication pre-share
 group 2
crypto isakmp key **** address **** no-xauth
crypto isakmp key **** address **** no-xauth
crypto isakmp key **** address **** no-xauth
crypto isakmp key **** address **** no-xauth
!
crypto isakmp client configuration group ****
 key ****
 pool mypool
!
crypto isakmp client configuration group ****
 key ****
 pool lrpool
!
!
crypto ipsec transform-set encrypt-des esp-des esp-md5-hmac
!
crypto dynamic-map dynamap 1
 set transform-set encrypt-des
!
!
crypto map combined client authentication list userauthen
crypto map combined isakmp authorization list groupauthor
crypto map combined client configuration address respond
crypto map combined 20 ipsec-isakmp
 set peer ****
 set transform-set encrypt-des
 match address 106
crypto map combined 30 ipsec-isakmp
 set peer ****
 set transform-set encrypt-des
 match address 107
crypto map combined 40 ipsec-isakmp
 set peer ****
 set transform-set encrypt-des
 match address 108
crypto map combined 50 ipsec-isakmp
 set peer ****
 set transform-set encrypt-des
 match address 109
crypto map combined 999 ipsec-isakmp dynamic dynamap
!
crypto map vpnclt local-address Ethernet1/0
crypto map vpnclt client authentication list userauthen
crypto map vpnclt isakmp authorization list groupauthor
crypto map vpnclt client configuration address respond
crypto map vpnclt 999 ipsec-isakmp dynamic dynamap
!
!
!
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
!
interface Ethernet0/0
 ip address 172.20.51.200 255.255.252.0
 ip access-group 115 in
 ip nat inside
 ip route-cache flow
 no ip mroute-cache
 ip policy route-map VPN
 half-duplex
!
interface Serial0/0
 ip address **** 255.255.255.252
 ip access-group 125 in
 ip nat outside
 ip inspect **** in
 ip inspect **** out
 no ip mroute-cache
 no fair-queue
 crypto map combined
!
interface Ethernet1/0
 ip address **** 255.255.255.248
 ip access-group 126 in
 ip nat outside
 ip inspect **** in
 ip inspect **** out
 no ip mroute-cache
 full-duplex
 crypto map vpnclt
!
ip local pool mypool 192.168.1.1 192.168.1.254
ip local pool lrpool 182.168.0.1 182.168.0.254
ip nat pool ****-natpool-1 **** **** netmask 255.255.255.0
ip nat inside source route-map nonat interface Ethernet1/0 overload
ip nat inside source static tcp 172.20.51.203 80 **** 80 extendable
ip nat inside source static tcp 172.20.51.10 20 **** 20 extendable
ip nat inside source static tcp 172.20.51.10 21 **** 21 extendable
ip nat inside source static tcp 172.20.50.201 20 **** 20 extendable
ip nat inside source static tcp 172.20.50.201 21 **** 21 extendable
no ip http server
no ip http secure-server
ip classless
ip route 0.0.0.0 0.0.0.0 ****
ip route **** 255.255.255.255 ****
ip route **** 255.255.255.255 ****
ip route **** 255.255.255.255 ****
ip route **** 255.255.255.255 ****
ip route 172.20.40.0 255.255.252.0 ****
ip route 172.20.44.0 255.255.252.0 ****
ip route 172.20.52.0 255.255.252.0 ****
ip route 172.20.56.0 255.255.252.0 ****
!
!
access-list 3 permit 66.0.0.0 0.255.255.255
access-list 106 permit ip 172.20.48.0 0.0.3.255 172.20.40.0 0.0.3.255
access-list 107 permit ip 172.20.48.0 0.0.3.255 172.20.44.0 0.0.3.255
access-list 108 permit ip 172.20.48.0 0.0.3.255 172.20.56.0 0.0.3.255
access-list 109 permit ip 172.20.48.0 0.0.3.255 172.20.52.0 0.0.3.255
access-list 115 permit tcp any any eq www
access-list 115 permit tcp any any eq ftp-data
access-list 115 permit tcp any any eq ftp
access-list 115 permit ip 172.20.48.0 0.0.3.255 172.20.40.0 0.0.3.255
access-list 115 permit ip 172.20.48.0 0.0.3.255 172.20.44.0 0.0.3.255
access-list 115 permit ip 172.20.48.0 0.0.3.255 172.20.52.0 0.0.3.255
access-list 115 permit ip 172.20.48.0 0.0.3.255 172.20.56.0 0.0.3.255
access-list 115 permit ip 172.20.48.0 0.0.3.255 192.168.0.0 0.0.3.255
access-list 115 permit ip 172.20.48.0 0.0.3.255 182.168.0.0 0.0.3.255
access-list 115 deny   udp any any eq tftp
access-list 115 deny   tcp any any eq 135
access-list 115 deny   udp any any eq 135
access-list 115 deny   udp any any eq netbios-ns
access-list 115 deny   udp any any eq netbios-dgm
access-list 115 deny   tcp any any eq 139
access-list 115 deny   udp any any eq netbios-ss
access-list 115 deny   tcp any any eq 445
access-list 115 deny   tcp any any eq 593
access-list 115 deny   tcp any any eq 4444
access-list 115 permit ip any any
access-list 125 permit esp any any
access-list 125 permit udp any eq isakmp any eq isakmp
access-list 125 permit ip 172.20.40.0 0.0.3.255 172.20.48.0 0.0.3.255
access-list 125 permit ip 172.20.44.0 0.0.3.255 172.20.48.0 0.0.3.255
access-list 125 permit ip 172.20.52.0 0.0.3.255 172.20.48.0 0.0.3.255
access-list 125 permit ip 172.20.56.0 0.0.3.255 172.20.48.0 0.0.3.255
access-list 125 permit ip 192.168.0.0 0.0.3.255 172.20.48.0 0.0.3.255
access-list 125 permit ip 182.168.0.0 0.0.3.255 172.20.48.0 0.0.3.255
access-list 125 permit tcp any host **** eq ftp
access-list 125 permit tcp any host **** eq ftp-data
access-list 125 permit tcp any host **** eq ftp
access-list 125 permit tcp any host **** eq ftp-data
access-list 125 permit tcp host **** eq ftp any
access-list 125 permit tcp host **** eq ftp-data any
access-list 125 permit tcp host **** eq ftp any
access-list 125 permit tcp host **** eq ftp-data any
access-list 125 deny   53 any any
access-list 125 deny   55 any any
access-list 125 deny   77 any any
access-list 125 deny   pim any any
access-list 125 deny   udp any any eq tftp
access-list 125 deny   tcp any any eq 135
access-list 125 deny   udp any any eq 135
access-list 125 deny   tcp any any eq 445
access-list 125 deny   tcp any any eq 593
access-list 125 deny   tcp any any eq 4444
access-list 125 deny   icmp any any redirect
access-list 125 deny   udp any any eq snmp
access-list 125 deny   ip 0.0.0.0 0.255.255.255 any
access-list 125 deny   ip 10.0.0.0 0.255.255.255 any
access-list 125 deny   ip 127.0.0.0 0.255.255.255 any
access-list 125 deny   ip 169.254.0.0 0.0.255.255 any
access-list 125 deny   ip 172.16.0.0 0.15.255.255 any
access-list 125 deny   ip 192.168.0.0 0.0.255.255 any
access-list 125 deny   ip 182.168.0.0 0.0.255.255 any
access-list 125 deny   ip 224.0.0.0 15.255.255.255 any
access-list 125 deny   ip 240.0.0.0 7.255.255.255 any
access-list 125 deny   ip 248.0.0.0 7.255.255.255 any
access-list 125 deny   ip host 255.255.255.255 any
access-list 125 permit icmp any any echo
access-list 125 permit icmp any any echo-reply
access-list 125 permit icmp any any unreachable
access-list 125 permit icmp any any packet-too-big
access-list 125 permit icmp any any time-exceeded
access-list 126 permit esp any any
access-list 126 permit udp any eq isakmp any eq isakmp
access-list 126 permit ip 192.168.1.0 0.0.0.255 172.20.48.0 0.0.3.255
access-list 126 permit ip 182.168.0.0 0.0.0.255 172.20.48.0 0.0.3.255
access-list 126 permit tcp any host **** eq ftp
access-list 126 permit tcp any host **** eq ftp-data
access-list 126 permit tcp host **** eq ftp any
access-list 126 permit tcp host **** eq ftp-data any
access-list 126 deny   53 any any
access-list 126 deny   55 any any
access-list 126 deny   77 any any
access-list 126 deny   pim any any
access-list 126 deny   udp any any eq tftp
access-list 126 deny   tcp any any eq 135
access-list 126 deny   udp any any eq 135
access-list 126 deny   tcp any any eq 445
access-list 126 deny   tcp any any eq 593
access-list 126 deny   tcp any any eq 4444
access-list 126 deny   icmp any any redirect
access-list 126 deny   udp any any eq snmp
access-list 126 deny   ip 0.0.0.0 0.255.255.255 any
access-list 126 deny   ip 10.0.0.0 0.255.255.255 any
access-list 126 deny   ip 127.0.0.0 0.255.255.255 any
access-list 126 deny   ip 169.254.0.0 0.0.255.255 any
access-list 126 deny   ip 172.16.0.0 0.15.255.255 any
access-list 126 deny   ip 192.168.0.0 0.0.255.255 any
access-list 126 deny   ip 182.168.0.0 0.0.255.255 any
access-list 126 deny   ip 224.0.0.0 15.255.255.255 any
access-list 126 deny   ip 240.0.0.0 7.255.255.255 any
access-list 126 deny   ip 248.0.0.0 7.255.255.255 any
access-list 126 deny   ip host 255.255.255.255 any
access-list 150 deny   ip 172.20.48.0 0.0.3.255 172.20.40.0 0.0.3.255
access-list 150 deny   ip 172.20.48.0 0.0.3.255 172.20.44.0 0.0.3.255
access-list 150 deny   ip 172.20.48.0 0.0.3.255 172.20.52.0 0.0.3.255
access-list 150 deny   ip 172.20.48.0 0.0.3.255 172.20.56.0 0.0.3.255
access-list 150 deny   ip 172.20.48.0 0.0.3.255 192.168.0.0 0.0.3.255
access-list 150 deny   ip 172.20.48.0 0.0.3.255 182.168.0.0 0.0.3.255
access-list 150 permit ip 172.20.48.0 0.0.3.255 any
access-list 151 permit ip host 172.20.51.10 172.20.40.0 0.0.3.255
access-list 151 permit ip host 172.20.51.10 172.20.44.0 0.0.3.255
access-list 151 permit ip host 172.20.51.10 172.20.52.0 0.0.3.255
access-list 151 permit ip host 172.20.51.10 172.20.56.0 0.0.3.255
access-list 151 permit ip host 172.20.51.10 192.168.0.0 0.0.3.255
access-list 151 permit ip host 172.20.51.10 182.168.0.0 0.0.3.255
access-list 151 permit ip host 172.20.51.203 172.20.40.0 0.0.3.255
access-list 151 permit ip host 172.20.50.201 172.20.40.0 0.0.3.255
access-list 151 permit ip host 172.20.50.201 172.20.44.0 0.0.3.255
access-list 151 permit ip host 172.20.50.201 172.20.52.0 0.0.3.255
access-list 151 permit ip host 172.20.50.201 172.20.56.0 0.0.3.255
access-list 151 permit ip host 172.20.50.201 192.168.0.0 0.0.3.255
access-list 151 permit ip host 172.20.50.201 182.168.0.0 0.0.3.255
!
route-map VPN permit 10
 match ip address 151
 set interface Loopback0
!
route-map nonat permit 10
 match ip address 150
!
snmp-server engineID local 0000000902000002161D4240
snmp-server community public RO
snmp-server community private RW
snmp-server packetsize 2048
radius-server host 172.20.50.201 auth-port 1645 acct-port 1646 key ****
!
!
dial-peer cor custom
!
!
!
!
!
line con 0
line aux 0
 password ****
 modem InOut
 transport input all
 flowcontrol hardware
line vty 0 4
 password ****
 length 16
!
!
end

****#
ASKER CERTIFIED SOLUTION
Log in to continue reading
Log In
Sign up - Free for 7 days
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
pgm554

Post the statistcs from Stanford.It should look like :

WEB100 Enabled Statistics:
Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
running 10s outbound test (client to server) . . . . . 324.59Kb/s
running 10s inbound test (server to client) . . . . . . 890.46kb/s

      ------  Client System Details  ------
OS data: Name = Windows XP, Architecture = x86, Version = 5.1
Java data: Vendor = Sun Microsystems Inc., Version = 1.4.2_06

      ------  Web100 Detailed Analysis  ------
Cable modem/DSL/T1 link found.
Link set to Full Duplex mode
No network congestion discovered.
Good network cable(s) found
Normal duplex operation found.

Web100 reports the Round trip time = 61.74 msec; the Packet size = 1420 Bytes; and
There were 51 packets retransmitted, 56 duplicate acks received, and 74 SACK blocks received
The connection stalled 1 times due to packet loss
The connection was idle 0.26 seconds (2.6%) of the time
This connection is receiver limited 3.75% of the time.
  Increasing the the client's receive buffer (0 KB) will improve performance
This connection is network limited 96.07% of the time.
Excessive packet loss is impacting your performance, check the auto-negotiate function on your local PC and network switch

Web100 reports TCP negotiated the optional Performance Settings to:
RFC 2018 Selective Acknowledgment: ON
RFC 896 Nagle Algorithm: ON
RFC 3168 Explicit Congestion Notification: OFF
RFC 1323 Time Stamping: OFF
RFC 1323 Window Scaling: OFF
Information: Network Middlebox is modifying MSS variable
Server IP addresses are preserved End-to-End
Information: Network Address Translation (NAT) box is modifying the Client's IP address
      Server says [xxx.xxx.xxx.xxx] but Client says [192.168.1.6]

That looks like your T1 stats and not the cable.
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
pgm554

Also more details:

order: 0.1273
rwintime: 0.0375
sendtime: 0.0017
cwndtime: 0.9607
rwin: 0.0000
swin: 1.7031
cwin: 0.1408
rttsec: 0.061742
Sndbuf: 223232
aspd: 3.10896

Checking for mismatch on uplink
      (speed > 50 [-1.99>50], (xmitspeed < 5) [0.32<5]
      (rwintime > .9) [0.03>.9], (loss < .01) [0.01<.01]
Checking for excessive errors condition
      (loss/sec > .15) [0.00>.15], (cwndtime > .6) [0.96>.6],
      (loss < .01) [0.01<.01], (MaxSsthresh > 0) [8520>0]
Checking for 10 Mbps link
      (speed < 9.5) [-1.99<9.5], (speed > 3.0) [-1.99>3.0]
      (xmitspeed < 9.5) [0.32<9.5] (loss < .01) [0.01<.01], (mylink > 0) [3.0>0]
Checking for Wireless link
      (sendtime = 0) [0.00=0], (speed < 5) [-1.99<5]
      (Estimate > 50 [1.53>50], (Rwintime > 90) [0.03>.90]
       (RwinTrans/CwndTrans = 1) [1/2=1], (mylink > 0) [3.0>0]
Checking for DSL/Cable Modem link
      (speed < 2) [-1.99<2], (SndLimTransSender = 0) [1=0]
       (SendTime = 0) [0.0017=0], (mylink > 0) [3.0>0]
Checking for half-duplex condition
      (rwintime > .95) [0.03>.95], (RwinTrans/sec > 30) [0.1>30],
       (SenderTrans/sec > 30) [0.1>30], OR (mylink <= 10) [3.0<=10]
Checking for congestion
      (cwndtime > .02) [0.96>.02], (mismatch = 0) [0=0]
      (MaxSsthresh > 0) [8520>0]

estimate = 1.53 based on packet size = 11Kbits, RTT = 61.74msec, and loss = 0.013079667
The theoretical network limit is 1.53 Mbps
The NDT server has a 218.0 KByte buffer which limits the throughput to 27.58 Mbps
Your PC/Workstation has a 0 KByte buffer which limits the throughput to 0 Mbps
The network based flow control limits the throughput to 2.28 Mbps

Client Data reports link is 'T1', Client Acks report link is 'T1'
Server Data reports link is 'OC-48', Server Acks report link is 'T1'
atyar

ASKER
I didn't get nearly that much detail when I ran the test, and the general internet traffic goes out the Cable Modem interface.
atyar

ASKER
lrmoore,
I mentioned in the comment above that there were a few things in the configuration that I didn't fully understand, and that may have been 'added' by working with Cisco TAC over the years, that I thought might be hurting our performance.  Do you have a specific recommendation about any of them, or explanation of why we should keep any of those mentioned?  I did find some documentation on ip inspect on Cisco's site, and found what you mentioned about usually configuring outbound on the WAN interface, so I made those changes already.
I take your comment about doing so much on the 2610 seriously, but our budget doesn't provide for purchasing a higher end router at this time.....we're stuck with muddling through with the 2610.  We did have virus/worm problems a few years ago, hence some of the extra lines in the ACL's, but I remember the cpu utilization being much higher back then and we could barely connect to the router even with telnet.

I looked at the acl's, and there are quite a few lines with no hits, so I could work on removing those.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
pgm554

Once you run the test,there are 2 icons for statistics and more details.

Just copy and paste in the answer box.

As I said before,this looks like the T1,the cable stats should be much larger.

atyar

ASKER
Gotcha on the Sanford thing...hadn't noticed those buttons.
Here it is:
WEB100 Enabled Statistics:
Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
running 10s outbound test (client to server) . . . . . 420.72Kb/s
running 10s inbound test (server to client) . . . . . . 502.44kb/s

      ------  Client System Details  ------
OS data: Name = Windows 2003, Architecture = x86, Version = 5.2
Java data: Vendor = Sun Microsystems Inc., Version = 1.5.0_06

      ------  Web100 Detailed Analysis  ------
Cable modem/DSL/T1 link found.
Link set to Full Duplex mode
No network congestion discovered.
Good network cable(s) found
Normal duplex operation found.

Web100 reports the Round trip time = 175.8 msec; the Packet size = 1460 Bytes; and
There were 5 packets retransmitted, 47 duplicate acks received, and 24 SACK blocks received
The connection was idle 0 seconds (0%) of the time
This connection is network limited 99.83% of the time.

Web100 reports TCP negotiated the optional Performance Settings to:
RFC 2018 Selective Acknowledgment: ON
RFC 896 Nagle Algorithm: ON
RFC 3168 Explicit Congestion Notification: OFF
RFC 1323 Time Stamping: OFF
RFC 1323 Window Scaling: OFF
Packet size is preserved End-to-End
Server IP addresses are preserved End-to-End
Information: Network Address Translation (NAT) box is modifying the Client's IP address
      Server says [24.227.36.202] but Client says [172.20.50.201]


 WEB100 Kernel Variables:
Client: localhost/127.0.0.1
AckPktsIn: 183
AckPktsOut: 0
BytesRetrans: 7570
CongAvoid: 117
CongestionOverCount: 0
CongestionSignals: 3
CountRTT: 130
CurCwnd: 10220
CurMSS: 1460
CurRTO: 382
CurRwinRcvd: 0
CurRwinSent: 5840
CurSsthresh: 5840
DSACKDups: 0
DataBytesIn: 0
DataBytesOut: 684328
DataPktsIn: 0
DataPktsOut: 452
DupAcksIn: 47
ECNEnabled: 0
FastRetran: 3
MaxCwnd: 16060
MaxMSS: 1460
MaxRTO: 603
MaxRTT: 355
MaxRwinRcvd: 0
MaxRwinSent: 5840
MaxSsthresh: 5840
MinMSS: 1460
MinRTO: 339
MinRTT: 97
MinRwinRcvd: 2147483647
MinRwinSent: 5840
NagleEnabled: 1
OtherReductions: 2
PktsIn: 183
PktsOut: 452
PktsRetrans: 5
X_Rcvbuf: 223232
RcvWinScale: 2147483647
SACKEnabled: 3
SACKsRcvd: 24
SendStall: 0
SlowStart: 6
SampleRTT: 166
SmoothedRTT: 166
X_Sndbuf: 223232
SndWinScale: 2147483647
SndLimTimeRwin: 0
SndLimTimeCwnd: 10799412
SndLimTimeSender: 17224
SndLimTransRwin: 0
SndLimTransCwnd: 1
SndLimTransSender: 1
SndLimBytesRwin: 0
SndLimBytesCwnd: 684328
SndLimBytesSender: 0
SubsequentTimeouts: 0
SumRTT: 22854
Timeouts: 0
TimestampsEnabled: 0
WinScaleRcvd: 2147483647
WinScaleSent: 2147483647
DupAcksOut: 0
StartTimeUsec: 548615
Duration: 10822236
c2sData: 2
c2sAck: 2
s2cData: 8
s2cAck: 2
half_duplex: 0
link: 100
congestion: 0
bad_cable: 0
mismatch: 0
spd: -1.99
bw: 0.78
loss: 0.006637168
avgrtt: 175.80
waitsec: 0.00
timesec: 10.00
order: 0.2568
rwintime: 0.0000
sendtime: 0.0016
cwndtime: 0.9984
rwin: 0.0000
swin: 1.7031
cwin: 0.1225
rttsec: 0.175800
Sndbuf: 223232
aspd: 1.35158

Checking for mismatch on uplink
      (speed > 50 [-1.99>50], (xmitspeed < 5) [0.42<5]
      (rwintime > .9) [0>.9], (loss < .01) [0.00<.01]
Checking for excessive errors condition
      (loss/sec > .15) [6.63>.15], (cwndtime > .6) [0.99>.6],
      (loss < .01) [0.00<.01], (MaxSsthresh > 0) [5840>0]
Checking for 10 Mbps link
      (speed < 9.5) [-1.99<9.5], (speed > 3.0) [-1.99>3.0]
      (xmitspeed < 9.5) [0.42<9.5] (loss < .01) [0.00<.01], (mylink > 0) [3.0>0]
Checking for Wireless link
      (sendtime = 0) [0.00=0], (speed < 5) [-1.99<5]
      (Estimate > 50 [0.78>50], (Rwintime > 90) [0>.90]
       (RwinTrans/CwndTrans = 1) [0/1=1], (mylink > 0) [3.0>0]
Checking for DSL/Cable Modem link
      (speed < 2) [-1.99<2], (SndLimTransSender = 0) [1=0]
       (SendTime = 0) [0.0016=0], (mylink > 0) [3.0>0]
Checking for half-duplex condition
      (rwintime > .95) [0>.95], (RwinTrans/sec > 30) [0>30],
       (SenderTrans/sec > 30) [0.1>30], OR (mylink <= 10) [3.0<=10]
Checking for congestion
      (cwndtime > .02) [0.99>.02], (mismatch = 0) [0=0]
      (MaxSsthresh > 0) [5840>0]

estimate = 0.78 based on packet size = 11Kbits, RTT = 175.8msec, and loss = 0.006637168
The theoretical network limit is 0.78 Mbps
The NDT server has a 218.0 KByte buffer which limits the throughput to 9.68 Mbps
Your PC/Workstation has a 0 KByte buffer which limits the throughput to 0 Mbps
The network based flow control limits the throughput to 0.69 Mbps

Client Data reports link is 'T1', Client Acks report link is 'T1'
Server Data reports link is 'OC-48', Server Acks report link is 'T1'


atyar

ASKER
Again, the router has both a T-1 link, and a Cable Modem link, but the default route in the config is out the cable modem link.
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
SOLUTION
Log in to continue reading
Log In
Sign up - Free for 7 days
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
atyar

ASKER
On a quiet network, I can get approaching 4Mb download speeds on speakeasy.net's speedtest from our domain server down there.  I agree - speed looks more like T1 under network load - that's our problem.  Do you know what that theoretical limit means?  I posted my 'watered down' router config up above - if you have a suggestion for what might be off, feel free to criticize :)
bladeeta21

Try doing load balancing on the router, set it so that once your defined bandwidth limit on your T1 has been reached to forward all traffic to the modem.
SOLUTION
Log in to continue reading
Log In
Sign up - Free for 7 days
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
atyar

ASKER
Interesting idea.  When you say these apply only to Windows 2000, as far as you can tell, are you also implying they apply to Windows XP and Server 2003, just not backwards from Win2K?  Most of our workstations there are, naturally, XP, but there are a few 2000 pro boxes, as well.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
pgm554

The stack is the same, beginning with W2K,I use it on my XP and 2003 boxes as a tweak.

Mike was bought in on a consult at a major corp that had a T3 and it was taking hours to send the data to another site.

In fact, it was so bad ,it was faster to FedX the data to the remote site.

He noticed the small TCP window and adjusted accordingly and it went down to minutes.
atyar

ASKER
Hmm, well, we'll try it on a workstation down there.  It might explain some of this, since we've been getting relatively decent throughput from the server fairly consistently, but some workstations have been getting poor throughput.
atyar

ASKER
You know, I'm curious.  When I do a show access-list 126, which is the access list inbound on my cable modem interface, I get an ungodly amount of what I interpret as ip inspect-added statements at the top of the access list, before it shows the explicit statements from the router's config (like 5 pages worth).  Is this causing a performance hit, having to sift through this huge access list created by, I believe, ip inspect?  Would this indicate I should be defining my acl's somehow differently?
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
atyar

ASKER
Well, I tried the suggestion for the window size registry tweak, and that wasn't our bottleneck.  Also, I followed what lrmoore suggested for streamlining my acls, and they are much smaller now and getting hits on all statements, but that wasn't our bottleneck either.  Wouldn't you know it, but come to find out, many of the people in the office down there (I'm in MI, they're in FL) had taken off the antivirus software from their computers (they think it slows them way down), and many of the computers were riddled with viruses, worms, and spyware.

Even with that mostly cleaned up, and with an added statement to the top of my acl's blocking udp port 1434 (looks like they were probably dealing with a sql slammer-type worm, among other things), it's still slow.  I placed an order yesterday for a PIX 515 firewall to drop in there, so I can offload the vpn tunnel and firewall functions from the router, per suggestion and our own prior gut feeling too.  I'm thinking about this, though - can the pix route (my gut says no - the pix doesn't route, but..) the internet traffic out one of it's fast ethernet interfaces (i.e. I'd plug the cable modem into a pix fe interface - we're getting the UR model with a total of 6 FE interfaces), and then pass the rest of the traffic (all inter-office vpn traffic at that point) on to the 2610 and out the T-1?
atyar

ASKER
Ok, well we're basically down to dropping a pix into the network to handle the encryption and firewalling, as that seems to be what is getting taxed on our 2610 router.  We found oodles of virus and worm issues there, due to users removing antivirus software, that have been mostly remediated.  We also found faulty wiring, which really irks me - the whole building was wired back in the Spring by a contractor, and he supposedly tested all wiring after running it.  We're working on that issue, too.

Splitting points all around..no one thing seems to be the cause of and solution to our woes, but hopefully we're getting there...
Thanks all.