Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

CISCO LLQ trouble??

Posted on 2004-10-29
134
Medium Priority
?
6,636 Views
Last Modified: 2010-05-18
What would cause me to have drops in my policy?
If I need to post full configs I can.
I have 2 Cisco 1721s connected via t1s to the internet.  I have Samsung ip phones.
After a day of use I have around 70-80 total drops
Here is my what    sh policy-map int gives me.  

Service-policy output: llq

    Class-map: NAPLESVOIP (match-all)
      264 packets, 152592 bytes
      5 minute offered rate 6000 bps, drop rate 1000 bps
      Match: access-group 100
      Queueing
        Strict Priority
        Output Queue: Conversation 264
        Bandwidth 500 (kbps) Burst 12500 (Bytes)
        (pkts matched/bytes matched) 262/151436
        (total drops/bytes drops) 4/2312

    Class-map: DATA (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: access-group 102
      Queueing
        Output Queue: Conversation 265
        Bandwidth 225 (kbps) Max Threshold 64 (packets)
        (pkts matched/bytes matched) 0/0
        (depth/total drops/no-buffer drops) 0/0/0

    Class-map: class-default (match-any)
      21254 packets, 2494063 bytes
      5 minute offered rate 253000 bps, drop rate 2000 bps
      Match: any
      Queueing
        Flow Based Fair Queueing
        Maximum Number of Hashed Queues 256
        (total queued/total drops/no-buffer drops) 0/372/0


0
Comment
Question by:tangofniro
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 57
  • 53
  • 19
  • +2
134 Comments
 
LVL 3

Expert Comment

by:cmsJustin
ID: 12446579
How many phones?
How many peak calls at any moment in time?
0
 
LVL 3

Expert Comment

by:cmsJustin
ID: 12446598
And you should post your configs also...
0
 

Author Comment

by:tangofniro
ID: 12446917
7 phones on the remote end 25   (3ip 22regular)phones at the main office

At the most  which would never happen   all 7 phones at the remote side could be calling the main office and talking to 7 phones there.
We also are going through a vpn


Here is the config from the remote router


RPSTRAND#sh run
Building configuration...

Current configuration : 4943 bytes
!
version 12.3
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname RPSTRAND
!
boot-start-marker
boot-end-marker
!
logging buffered 51200 warnings
enable password 7 030752180500
!
username cisc0 privilege 15 password 7 0505031838435C02
username cisco password 7 110A1016141D
no aaa new-model
ip subnet-zero
no ip source-route
!
!
ip dhcp excluded-address 192.168.3.1 192.168.3.99
ip dhcp excluded-address 192.168.3.200 192.168.3.254
ip dhcp excluded-address 172.168.1.1 172.168.1.100
ip dhcp excluded-address 172.168.1.200 172.168.1.254
!
ip dhcp pool MainScope
   network 192.168.3.0 255.255.255.0
   domain-name rpprop
   dns-server 4.2.2.1
   default-router 192.168.3.1
!
ip dhcp pool Strand
   network 172.168.1.0 255.255.255.0
   domain-name rpprop
   dns-server 4.2.2.1
   default-router 172.168.1.1
!
!
ip domain name yourdomain.com
no ip bootp server
no ip cef
ip audit notify log
ip audit po max-events 100
no ftp-server write-enable
no scripting tcl init
no scripting tcl encdir
!
!
!
!
!
crypto isakmp policy 10
 hash md5
 authentication pre-share
crypto isakmp key 1234 address 199.x.x.x
crypto isakmp keepalive 20 10
!
!
crypto ipsec transform-set rpvpn esp-des esp-md5-hmac
!
crypto map rpvpn 10 ipsec-isakmp
 set peer 199.x.x.x
 set peer 216.x.x.x
 set transform-set rpvpn
 match address 101
!
!
!
class-map match-all NAPLESVOIP
 match access-group 100
class-map match-all DATA
 match access-group 102
!
!
policy-map llq
 class NAPLESVOIP
  priority 500
 class DATA
  bandwidth 225
 class class-default
  fair-queue
!
!
!
interface FastEthernet0
 description $FW_INSIDE$$ETH-LAN$
 ip address 172.168.1.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip route-cache flow
 ip tcp adjust-mss 1380
 speed auto
 full-duplex
 no cdp enable
!
interface FastEthernet1
 no ip address
 no cdp enable
!
interface FastEthernet2
 no ip address
 shutdown
 no cdp enable
!
interface FastEthernet3
 no ip address
 shutdown
 no cdp enable
!
interface FastEthernet4
 no ip address
 shutdown
 no cdp enable
!
interface Serial0
 description $FW_OUTSIDE$
 ip address 199.x.x.x 255.x.x.x
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat outside
 service-policy output llq
 encapsulation ppp
 ip route-cache flow
 service-module t1 timeslots 8-24
 no cdp enable
 crypto map rpvpn
!
interface Vlan1
 no ip address
!
ip nat inside source route-map nonat interface Serial0 overload

ip classless
ip route 0.0.0.0 0.0.0.0 199.x.x.x
ip http server
ip http authentication local
ip http secure-server
!
!
!
access-list 100 permit udp any any range 16384 32000
access-list 100 permit tcp any any eq 1719
access-list 100 permit tcp any any eq 1720
access-list 100 permit tcp any any eq 6100
access-list 100 permit tcp any any range 1024 4999
access-list 100 permit udp any any eq 6000
access-list 100 permit udp any any range 1024 4999
access-list 100 permit udp any any eq 5060
access-list 100 permit udp any any range 30000 30030
access-list 100 permit udp any any range 9000 9001
access-list 101 permit ip 172.168.1.0 0.0.0.255 192.168.0.0 0.0.0.255
access-list 101 deny   ip 172.168.1.0 0.0.0.255 any
access-list 102 permit tcp any any range 5000 5110
access-list 110 deny   ip 172.168.1.0 0.0.0.255 192.168.0.0 0.0.0.255
access-list 110 permit ip 172.168.1.0 0.0.0.255 any
no cdp run
!
route-map nonat permit 10
 match ip address 110
 set ip next-hop 1.1.1.2
!
!
control-plane
!
banner login ^C
0
Get your Disaster Recovery as a Service basics

Disaster Recovery as a Service is one go-to solution that revolutionizes DR planning. Implementing DRaaS could be an efficient process, easily accessible to non-DR experts. Learn about monitoring, testing, executing failovers and failbacks to ensure a "healthy" DR environment.

 
LVL 5

Expert Comment

by:snowsurfer
ID: 12457965
The VPN adds overhead.  You should turn on CAC to limit the number of calls that can be made at one time.  What is your bandwidth?  The problem with going over a VPN is there is no QOS on the internet.  Even though you are using a vpn connection you are still using the internet.
0
 

Author Comment

by:tangofniro
ID: 12459179
SO what you are saying is that really no matter what I am going to get drops on my policies due to the fact i'm going over the internet?  
0
 
LVL 5

Expert Comment

by:snowsurfer
ID: 12463608
I am afraid so.  You might be able to fine tune it a little bit, but VoIP is not meant for the internet
0
 
LVL 12

Accepted Solution

by:
Joel_Sisko earned 1600 total points
ID: 12468692
Well I disagree on the VoIP not meant for the Internet. Quickly look up the TLA for VoIP (Voice over Internet Protocol).

First before you give up on implementing a VoIP solution. What are ping times between the two 1700's? If you are under <100ms between sites, VoIP has great potential. The human ear detects anything over 180ms as starting to be choppy and distorted.

I will agree on the VPN adds overhead and could pose a problem, but I have implemented many VoIP solutions using a VPN between sites. But we usually use a separate device for the VPN solution. Part of your problem is that your little Cisco router is being asked to do to much, Route, IPSEC, Firewall. The 1720 was not built for all of this at the same time. You might want to go to the 1760 series router and use the VPN module to offload some of the work.

But in any case, if your ping times are good then using VoIP is reasonable.

Also dropped packets also could indicate the router is being overloaded. Also you want to set up some form of QOS on your local network; this might help reduce delay times. One question, is the T1 a point-to-point? If not point to point, are both T1's terminated to the same ISP/Carrier network? If the drop packets are only the voice data packets, then you could have a application issue or even the Samsung phone causing the problem.

As a ping test I would recommend to ping one computer at site A to another computer over T1's at site B. To ping, go to a desktop on the same LAN: (make sure your computer/router/firewall is not disabled to respond to a ping request)

Start > Run> type cmd> at the prompt type            ping -t -l 64000 X.X.X.X   (make sure that you are pinging the right server address X.X.X.X)

Let the command run. The command sends 64K byte packets until you hit Cntrl C. The larger packet tests the Ethernet connection a little bit better than the standard ping command of 5 bytes. Let it run for a bit and see what is going on.  Also, check the routers interface's for CRC errors, if you are getting a high amount you could have connectivity problems.

Run the following at the cisco CLI to look at the crc errors:

show interfaces ethernet
show interfaces serial

Let me know how the tests go and we can move on from that point.
0
 

Author Comment

by:tangofniro
ID: 12469052
Okay,

No CRCs on main site interfaces

remote side interfaces which I have the configs above are , Fast0  0 CRCs,   Serial0  68 CRCs, router was rebooted today.

ping times between the sites show mostly 30ms but every once in a while it will jump up very high around 200ms then it will go back down.

These T1s are point to point,  to the internet not leased line,  same carrier,  they say it my router config, I am trying to find out if it is.  Is there any command to show my router being overworked?  

I could post the main site config if that would help.

The phones work great for about 90% of the time  but for the other 10% they are unusable.  There is no specific time or day either.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12469083
Post the config so we dont go DOHHH! later on. The CRC's on the serial are not unusual is that was the amount for the entire day? What you could do is reset the crc's count to zero and make a boat load of VoIP calls (try to do this when the office is quite, try to reduce the amount of Internet browsing traffic), see what the crc count is, then clear the counters, do not make any VoIP calls for a while, check crc count, if the same then most likely not the issue.

What version of IOS are you using and yes there are commands for performance monitoring.



0
 

Author Comment

by:tangofniro
ID: 12469161
here is the main sites config, where thephone system resides.

Thanks for your help.

no aaa new-model
ip subnet-zero
no ip source-route
!
!
ip dhcp excluded-address 192.168.0.1 192.168.0.99
ip dhcp excluded-address 192.168.0.200 192.168.0.254
!
ip dhcp pool MainScope
   network 192.168.0.0 255.255.255.0
   domain-name
   dns-server 4.2.2.1
   default-router 192.168.0.1
!
!
ip domain name rp-prop.com
ip name-server 4.2.2.1
ip name-server 4.2.2.2
no ip bootp server
no ip cef
ip audit notify log
ip audit po max-events 100
ip ssh time-out 60
ip ssh authentication-retries 2
no ftp-server write-enable
no scripting tcl init
no scripting tcl encdir
!
!
!
!
!
crypto isakmp policy 10
 hash md5
 authentication pre-share
crypto isakmp key 1234 address 1x.x.x.x
crypto isakmp keepalive 20 10
!
!
crypto ipsec transform-set rpvpn esp-des esp-md5-hmac
!
crypto map rpvpn 10 ipsec-isakmp
 set peer 1x.x.x.x
 set transform-set rpvpn
 match address 101
!
!
!
class-map match-all STRANDDATA
 match access-group 102
class-map match-all STRANDVOIP
 match access-group 100
!
!
policy-map LLQ
 class STRANDVOIP
  priority 450
 class STRANDDATA
  bandwidth 650
 class class-default
  fair-queue
!
!
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.0
!
interface FastEthernet0
 description $FW_INSIDE$$ETH-LAN$
 ip address 192.168.0.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 no ip route-cache
 ip tcp adjust-mss 1380
 ip policy route-map nonat1
 no ip mroute-cache
 speed auto
 full-duplex
 priority-group 5
 no cdp enable
!
interface FastEthernet1
 switchport mode trunk
 no ip address
 no cdp enable
!
interface FastEthernet2
 no ip address
 shutdown
 no cdp enable
!
interface FastEthernet3
 no ip address
 shutdown
 no cdp enable
!
interface FastEthernet4
 no ip address
 shutdown
 no cdp enable
!
interface Serial0
 description USLEC T1
 ip address 1x.x.x.x  255.255.255.x
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat outside
 service-policy output LLQ
 encapsulation ppp
 no ip route-cache
 no ip mroute-cache
 service-module t1 timeslots 1-24
 no cdp enable
 crypto map rpvpn
!
interface Vlan1
 no ip address
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip route-cache flow
!
ip nat inside source route-map nonat interface Serial0 overload
ip nat inside source static 192.168.0.30 2x.x.x.x extendable
ip nat inside source static 192.168.0.31 2x.x.x.x extendable
ip nat inside source static 192.168.0.205 2x.x.x.x extendable
ip nat inside source static 192.168.0.2 2x.x.x.x extendable
ip classless
ip route 0.0.0.0 0.0.0.0 199.x.x.x
no ip http server
ip http authentication local
ip http secure-server
!
!
!
logging trap debugging
access-list 100 permit udp any any range 16384 32000
access-list 100 permit tcp any any eq 1719
access-list 100 permit tcp any any eq 1720
access-list 100 permit tcp any any eq 6100
access-list 100 permit tcp any any range 1024 4999
access-list 100 permit udp any any eq 6000
access-list 100 permit udp any any range 1024 4999
access-list 100 permit udp any any eq 5060
access-list 100 permit udp any any range 30000 30030
access-list 100 permit udp any any range 9000 9001
access-list 101 permit ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255
access-list 101 deny   ip 192.168.0.0 0.0.0.255 any
access-list 102 permit tcp any any range 5000 5110
access-list 105 deny   icmp any any echo
access-list 105 permit ip any any
access-list 110 deny   ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255
access-list 110 deny   ip host 192.168.0.205 any
access-list 110 deny   ip host 192.168.0.30 any
access-list 110 deny   ip host 192.168.0.31 any
access-list 110 deny   ip host 192.168.0.2 any
access-list 110 permit ip 192.168.0.0 0.0.0.255 any
access-list 120 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255
access-list 123 permit ip host 192.168.0.205 172.168.1.0 0.0.0.255
access-list 123 permit ip host 192.168.0.30 172.168.1.0 0.0.0.255
access-list 123 permit ip host 192.168.0.31 172.168.1.0 0.0.0.255
access-list 123 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255
no cdp run
!
route-map nonat1 permit 10
 match ip address 123
 set ip next-hop 1.1.1.2
!
route-map nonat permit 10
 match ip address 110
!
!
control-plane
!
banner login ^CCAuthorized access only!
 Disconnect IMMEDIATELY if you are not an authorized user!^C
!
line con 0
 password 7 0833184A1A18541B
 login
 transport output telnet
line aux 0
 login local
 transport output telnet
line vty 0 4
 privilege level 15
 password 7 06145B255F4F5815
 login
 transport input telnet ssh
line vty 5 15
 privilege level 15
 login local
 transport input telnet ssh
!
scheduler allocate 4000 1000
scheduler interval 500
!
end

RPNAPLES#
0
 

Author Comment

by:tangofniro
ID: 12469166
IOS version is 12.3
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12469300
please run from Cisco CLI:

show version

post the results
0
 

Author Comment

by:tangofniro
ID: 12469338
RPNAPLES#sh version
Cisco IOS Software, C1700 Software (C1700-K9O3SY7-M), Version 12.3(2)XE, EARLY D
EPLOYMENT RELEASE SOFTWARE (fc1)
Synched to technology version 12.3(3.5)T
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2003 by Cisco Systems, Inc.
Compiled Tue 18-Nov-03 23:26 by ealyon

ROM: System Bootstrap, Version 12.2(7r)XM2, RELEASE SOFTWARE (fc1)
ROM:

RPNAPLES uptime is 1 week, 13 hours, 45 minutes
System returned to ROM by power-on
System restarted at 08:58:52 America/Chicago Mon Apr 8 2002
System image file is "flash:c1700-k9o3sy7-mz.123-2.XE.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 1721 (MPC860P) processor (revision 0x300) with 84983K/13321K bytes of memo
ry.
Processor board ID FOC08172HWP (3205001181), with hardware revision 0000
MPC860P processor: part number 5, mask 2
1 Ethernet interface
5 FastEthernet interfaces
1 Serial interface
1 Virtual Private Network (VPN) Module
WIC T1-DSU
32K bytes of NVRAM.
32768K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102




\from remote site\\





RPSTRAND#sh version
Cisco IOS Software, C1700 Software (C1700-K9O3SY7-M), Version 12.3(2)XE, EARLY D
EPLOYMENT RELEASE SOFTWARE (fc1)
Synched to technology version 12.3(3.5)T
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2003 by Cisco Systems, Inc.
Compiled Tue 18-Nov-03 23:26 by ealyon

ROM: System Bootstrap, Version 12.2(7r)XM2, RELEASE SOFTWARE (fc1)
ROM:

RPSTRAND uptime is 5 hours, 31 minutes
System returned to ROM by power-on
System image file is "flash:c1700-k9o3sy7-mz.123-2.XE.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 1721 (MPC860P) processor (revision 0x300) with 84983K/13321K bytes of memo
ry.
Processor board ID FOC08172J3K (2799143465), with hardware revision 0000
MPC860P processor: part number 5, mask 2
1 Ethernet interface
5 FastEthernet interfaces
1 Serial interface
1 Virtual Private Network (VPN) Module
WIC T1-DSU
32K bytes of NVRAM.
32768K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102


0
 

Author Comment

by:tangofniro
ID: 12474772
In the meantime I added  QOS pre-classify on my crypto-map.

0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12476105
Nothing jumping out, but I do have one question, where are the T1 obtaining their clocking from?
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12498370
tangofniro

How are things going?
0
 

Author Comment

by:tangofniro
ID: 12499466
quality yesterday and today was pretty subpar. A lot of static.
clocking coming from my csu/dsu internally.

Nothing really jumping out at you either?
What did you think about the qos on the cyrpto-map?  Cisco pointed me in that direction.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12508695
Qos on the Crypto will only help streamline things, so its a good thing.

Question on the drops, are you actually dropping calls mid way thru the conversations?

In terms of static, could you clarify? Voice signals having static? High CRC's?

The good news it will get thru this eventually, Someone asked Thomas Edison that why he would try to keep inventing something even if it took 50,000 tries? His response was that he eleiminated 50,000 ways of not doing it.

Just a thought if the calls are being dropped during mid conversation, we might be running into a timeout issue of the application over the VPN, problem might be outside of the network. Do you have any tech specs on the Samsung IP Phones?
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12526429
How are things coming along?
0
 

Author Comment

by:tangofniro
ID: 12528939
hit or miss.  Friday was bad but I think today was better.  

Calls don't drop out completly, Static is the main problem , can't hear the other end, and speakerphone useless.

I do have tech specs on the phone but it's on cd.  PDF files  I can send them to you if you want but there is a lot of junk to wade through.

Thanks you for all of your help and staying on this problem.



0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12536959
On the pdf, if you know of a FTP site and load them up there? Will galdly review them.

If you are actually getting static there are a few things we need to look at:

First if VAD is being used and also echo-cancellation. VAD is more than likely being introdiced by the phone or Samsung system. Need to find that and turn it off.

Also, just another thing popped into my head, we can test the link betwen the sites using, lets say two computers running a free softphone, such sjlabs or x-ten lite, and that have speakers and microphones. If you establish a call between the two (using same VoIP protocols) and no hissing or drop offs, we can lean more towards the phone side of things.

0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12557544
tangofniro

Any headway?
0
 

Author Comment

by:tangofniro
ID: 12607720
Hey Joel,

Sorry for the delay.  I was out of town for a few days.   It seems good somedays and others times/days  it is not great.  I don't know if I can attrib it too the network or the internet.


Thanks  for your continued help,
will
0
 

Author Comment

by:tangofniro
ID: 12607768
One more thing I can ping everything in my network under 10ms  but if I ping my routers internal interface it is jumpy  45ms -68ms  11ms.  Why would it do that?  Is the QOS making that happen?
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12608318
It could be, it would jump if there is other traffic at the same time with a higher priority. If it happens all the time (being jumpy) it might be a configuration issue? You can checkto see this using the ping command from earlier on in our dialog.

Few things to follow up on:

Are you still getting static on voice calls?
Do you have a spare computer with a 10/100 NIC and a hub?
0
 

Author Comment

by:tangofniro
ID: 12609673
After everyone  left I ran the ping from the servers to the router int and the ping time seems pretty stable  all under 10
for the most part.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12610045
Okay lets assume that is okay, are you still getting static on the lines?
0
 

Author Comment

by:tangofniro
ID: 12610138
Some days(not all day) it sounds like a bad cell phone others it works fine, really no drops in calls.  I am wondering if it just might be the internet ????
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12610220
It could be here a re few steps to find the ghost:

1. Try to disable the use of VAD and echo-cancellation on the VoIP side of things
2. To monitor the Internet , there is a program I have used in the past that helps to do this it can be found at:  

http://www.serverscheck.com/

I believe you can set up to 3 free monitors for each system setup. Basically I would set up a ping command and monitor that. Download the program and take a look, it has a alot of features.

3. Setup a network sniffer at each site on each interface. You could use Ethereal or some other free products.

The ideal situation would be two computers, one on each site, each connected to between the network and router via a hub (hub allows to see all packets) then set up Ethereal and serverscheck on each. Also you could enable logging on the Cisco routers to log to the same system running the sniffer and monitor. Let it run for a few days, log times when all is running well, then log times when things are bad. Go back to the data captured and look up logged times, see if something is out of the ordinary.

Kindest regards

Joel_Sisko
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12610281
Also, part of th problem could be to much jitter, using Ehtereal you should be able to determine this, might not be real clear cut. There are some products that allow for this one is from Wildpackets called Etherpeek VX, another is Observer 10, but they cost a few hundred dollars.

Just rememebered try using this site:

http://www.testyourvoip.com/


0
 

Author Comment

by:tangofniro
ID: 12610433
tried test your voip,

I recieved a MOS 4.3
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12611418
4.3 is good, what type of traffic was going across the network at that time? Need to run this test a variosu times and see if it changes, also run the test when you know the problem you have been having arises.

Is it possible to set up a sniffer and log files from the Cisco router? Since we are looking for somewhat of the needle in the haystack we need to baseline the network and move forward from that point.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12611587
Additional free program that may help out:

http://www.bb4.org/features.html
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12612029
One more I just rememred that is affordable, gui based and quite a few features:

http://www.colasoft.com/products/capsa/

0
 

Author Comment

by:tangofniro
ID: 12651042
Okay, I had the problem today.

Voice quality went out the window.  I ran some speed tests and from the various places I ran it from they all showed very slow speed like 300-400k.
I spoke with the provider and they said they did not see anything wrong at the time.  It lasted for about 25minutes and they gone.  
Is there anything I can do to check what is going on?  Can you think of anything that might cause that?

Kind of stuck...
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12651666
I am assuming the speed went back up after the 25 minutes as well as the voice quality? Did you happen to check with the www.testyourvoip.com tool? Though your speed/ping might be okay the packet jitter might be going off the charts. Jitter is a result of packets showing up at different times to put it simply.

Voice is very time sensitive, thought also just popped into my me head, what do you have your MTU set at on your routers?  Setting the MTU higher might help, next time you are having the issue try adjusting the MTU.

Also you really need to baseline the system as per previous posts, also use a traceroute tool, see how many hops it takes and which routers the packet touches during normal operations, then do this again when the speed goes down. It could be also that soem traffic is being redirected thru a different route for a given time period for maintence and such.

Kindest regards,

Joel _Sisko
0
 

Author Comment

by:tangofniro
ID: 12655404
Hello Joel,
I checked with testyourvoip as it was happening and I got a terrible score of 2.2.  There was no jitter going out but coming in from Boston there was.

I will try a few of your suggestions from earlier.  Is there a command that will show me every active connection/ports on the router.

The mtu is set to 1600 on my serial int.

Going out to the internet while this was happening I was getting a good amount of packet loss to my providers gateway. And time was very high.  
After the 25 or so minutes the speed was back along with voice quality.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12657599
Okay at least we can start ruling out ithe nternal stuff and focus on the carrier.

Need to baseline the speed from site a to site b, need to baseline the routes it also takes.

If it is actually the carrier, then monitoring the Cisco interface will really not help, other than let us know something is going wrong. We need to find out where in the carrier network things are going wrong. By the way who is your provider?
0
 

Author Comment

by:tangofniro
ID: 12660568
The route is only 3 hops

Had alot of trouble today(static, packet loss). I don't know my provider USLEC says they are not seeing any problems on their side.  The do see a lott of usage on our side.  I can't see what is causing the problem.  Testyourvoip was pretty low also.  Great thing was at 5 oclock the problem went away.  Everything now is great.

I don't know if I would rule out it being an internal problem.  That would be the way I am leaning right now.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12660763
Can you set up a system to monitor the traffic going to the router? You really need to baseline the system, without it this post will be a mile long (LOL), can use the points but not that bad (LOL).

Are you sharing files across the link? Any kind of Database upadtes? Someone backing up to network drive? Are you using Active Directory? What about email? Roaming profiles?
0
 

Author Comment

by:tangofniro
ID: 12660812
files shared across link-- yes
Mapped Drives-- yes
Active Directory--yes
Email-- Yes not in house though, No Exchange
Roaming profiles--- NO


Will try to start a baseline before Thanksgiving.

Thanks Joel

0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12660934
I just wonder if there is an application that is performing some kind of update, backup, or maintence, it would explain why things go nuts for abit then die down.

Quick question, for Internet usage is everyone (both sites) going over the same link also?

How do you implement virus updates?
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12660984
http://www.colasoft.com/products/capsa/

This will be the best tool to baseline the system.
0
 

Author Comment

by:tangofniro
ID: 12661026
i downloaded capsa today, nothing really jumped out at me. will buy it most likely.

i have each side going out over the internet on each link.  Unless you can see something different in my config.

each workstation has it's side router as the default gateway.

I have 2 server 1 domain the other a terminal server/backup domain controller

I use Symantec Antivirus Server.

Anything I should be looking for in particular that you are expecting  Cola to see?

thanks
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12661118
Looking for the needle!

In reagrds to the Antivirus, everyone is updating from the local server at main site? How often and when? Are all users doing it at the same time?

What about Microsoft updates? Doing this auto also?

Looking for an application that is bursty in nature.

0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12661190
Also dropping your MTU size might help the VoIP weather the storms so to speak. But do not go below 400, it will have an adverse affect on the control signaling for VoIP. Drop it to 900, this is more of band aid rather than a cure, but should help.

0
 

Author Comment

by:tangofniro
ID: 12661309
Could my router be overloaded?  Causing the slowdown?
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12661361
Always possible, need to log the cpu performance, remember early on I said that the router is being asked to do alot (think my first post). But since your provider is seeing a considerable amount of traffic which you can not account for, you should still baseline the network to see what is causing the traffic.

Dropping the MTU size will actually add to processor overhead but will enable the packet to transverse the network/Internet more efficiently.
0
 

Author Comment

by:tangofniro
ID: 12662004
I ran sh proc and sh proc cpu and here is what I got:

RPNAPLES#sh proc
CPU utilization for five seconds: 32%/30%; one minute: 30%; five minutes: 29%
 PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
   1 Cwe 8029A2D0           32          8    4000 5548/6000   0 Chunk Manager
   2 Csp 80280AB0           28       2245      12 2744/3000   0 Load Meter
   3 M*         0           28         19    147310360/12000  6 Virtual Exec
   4 Mwe 8053C650            0         94       0 5768/6000   0 DHCPD Timer
   5 Lst 80299ACC        11444       1346    8502 5752/6000   0 Check heaps
   6 Cwe 8029E304           32         39     820 5536/6000   0 Pool Manager
   7 Mst 801A58E4            0          2       0 5708/6000   0 Timers
   8 Mwe 80096BB8            0          2       0 5716/6000   0 Serial Backgroun
   9 Mwe 80178B38            0          2       0 5704/6000   0 AAA high-capacit
  10 Lwe 8034FD08          948       1278     741 5008/6000   0 ARP Input
  11 Mwe 80371168            0          3       0 5708/6000   0 DDR Timers
  12 Mwe 80675650           32       2805      11 5768/6000   0 HC Counter Timer
  13 Lwe 80978F34           12          2    6000 5624/6000   0 Entity MIB API
  14 Mwe 80BE83B8            0          2       0 5712/6000   0 ATM Idle Timer
  15 Mwe 8009C57C            0          1       0 5788/6000   0 SERIAL A'detect
  16 Msp 801E85FC           12      11208       1 5744/6000   0 GraphIt
  17 Mwe 8038C274            0          2       011712/12000  0 Dialer event
  18 Mwe 806196F8            0          2       011728/12000  0 XML Proxy Client
  19 Cwe 8026C730            0          1       0 5804/6000   0 Critical Bkgnd
  20 Mwe 8021AC98          296       3265      9010392/12000  0 Net Background
  21 Lwe 80196DF8            0         19       011356/12000  0 Logger
  22 Mwe 801BB920           40      11205       3 5736/6000   0 TTY Background
  23 Msp 8022BC78          132      11217      11 7464/9000   0 Per-Second Jobs
  24 Lwe 80311EC0            8          2    4000 1864/3000   0 IPM_C1700_CLOCK
  25 Mwe 812F69B4            0          2       0 5784/6000   0 AggMgr Process
  26 Hwe 81327A00            4          2    2000 5264/6000   0 ESWPPM
  27 Hwe 8022B7A4            0          1       0 5788/6000   0 Net Input
  28 Msp 80220A90           84       2249      37 5740/6000   0 Compute load avg
  29 Msp 8022BCEC         8352        192   43500 5752/6000   0 Per-minute Jobs
  30 Mwe 8009F05C            0          3       0 5700/6000   0 Service-module a
  31 Mwe 813322AC            0          1       0 5792/6000   0 Switch Link Moni
  32 Mwe 8025067C            4          2    2000 5712/6000   0 AAA Server
  33 Mwe 80252B48            0          1       0 5784/6000   0 AAA ACCT Proc
  34 Mwe 80252C2C            0          1       0 5772/6000   0 ACCT Periodic Pr
  35 Lwe 8017D6A4            0          1       0 5788/6000   0 AAA_SERVER_DEADT
  36 Mwe 80304C34            0          2       0 5724/6000   0 AAA Dictionary R
  37 Mwe 8044EEF4      1653100    1349941    1224 9952/12000  0 IP Input
  38 Mwe 80473998            0          1       0 5768/6000   0 ICMP event handl
  39 Mwe 81159038            0          8       0 5788/6000   0 CRYPTO IKMP IPC
  40 Mwe 8085DB38            0          1       011788/12000  0 SSS Manager
  41 Mwe 80861464            4       1501       211764/12000  0 SSS Test Client
  42 Mwe 8086AE94            0          1       0 5800/6000   0 SSS Feature Mana
  43 Mwe 8086AF28          780      43912      17 5768/6000   0 SSS Feature Time
  44 Mwe 8027AE9C            0          4       011732/12000  0 PPP Hooks
  45 Lwe 80A546B4            0          1       0 5396/6000   0 X.25 Encaps Mana
  46 Mwe 80DFED4C            0          1       011748/12000  0 VPDN call manage
  47 Mwe 80E7AB9C            0          1       011780/12000  0 L2X Data Daemon
  48 Mwe 80E42F6C            0          1       011756/12000  0 L2X Socket proce
  49 Mwe 80E0FD10            0          1       011784/12000  0 L2X SSS manager
  50 Mwe 80E1CB74            0          2       011712/12000  0 L2TP mgmt daemon
  51 Mwe 8105F430            0          1       0 5776/6000   0 AC Mgr
  52 Mwe 81239BE4            0          2       011728/12000  0 KRB5 AAA
  53 Mwe 812E6C64            0          2       0 5540/6000   0 DTP Protocol
  54 Mwe 8027AE9C            4          3    133310988/12000  0 PPP IP Route
  55 Mwe 8027AE9C            0          4       011448/12000  0 PPP IPCP
  56 Mwe 80425860           32         19    1684 4436/6000   0 DHCPD Receive
  57 Mwe 80551360          524        215    2437 8224/9000   0 IP Background
  58 Mwe 805574F4          112        194     577 8736/9000   0 IP RIB Update
  59 Mst 8043214C            0         36       011044/12000  0 TCP Timer
  60 Lwe 80437114            8          3    266610940/12000  0 TCP Protocols
  61 Mwe 8049E14C            0          1       0 5804/6000   0 RARP Input
  62 Hwe 80533708            0          1       0 5776/6000   0 Socket Timers
  63 Mwe 805090E4            4         40     100 8452/9000   0 HTTP CORE
  64 Lsi 805CB69C           48        188     255 5304/6000   0 IP Cache Ager
  65 Hwe 80A6DB7C            0          1       0 5780/6000   0 PAD InCall
  66 Mwe 80A27320            0          2       011708/12000  0 X.25 Background
  67 Mwe 8086CC34            0          2       0 5708/6000   0 PPP SSS
  68 Mwe 80908750           16        189      84 8688/9000   0 Adj Manager
  69 Mwe 80AA1614           72      21938       3 5700/6000   0 IP NAT Ager
  70 Mwe 8027AE9C            4          2    2000 5728/6000   0 PPP Bind
  71 Mwe 8096E9F4            0          1       0 5756/6000   0 SNMP Timers
  72 Mwe 80C7B920            0          1       0 5796/6000   0 Inspect Timer
  73 Msi 8054B8C4            0       3196       0 4980/6000   0 DHCPD Database
  74 Mwe 80CA5BF8            0          2       0 5612/6000   0 URL filter proc
  75 Mwe 80CBF034            0         38       0 5796/6000   0 Authentication P
  76 Mwe 80CC8EA0            0          1       0 5776/6000   0 Auth-proxy AAA B
  77 Mwe 80CC9C64            0          1       0 5796/6000   0 IDS Timer
  78 Mwe 80DE114C            0          1       023780/24000  0 COPS
  79 Mwe 80DEFDF0            0          2       0 5720/6000   0 Dialer Forwarder
  80 Mwe 80E06E3C            8       2249       311776/12000  0 L2F management d
  81 Mwe 80E357FC            4          1    400011540/12000  0 PPTP Mgmt
  82 Mwe 80E7F4DC            0          2       011728/12000  0 PPTP Data
  83 Hwe 810EDD9C        25680      30471     842 5064/6000   0 Crypto HW Proc
  84 Lwe 812DB7A4            0          1       0 5796/6000   0 XSM_EVENT_ENGINE
  85 Lsi 812D93DC            8       1124       711824/12000  0 XSM_ENQUEUER
  86 Lsi 812DC4D4            4       1124       311828/12000  0 XSM Historian
  87 Mwe 8025D51C            0          2       0 5728/6000   0 LOCAL AAA
  88 Mwe 8025F5D4            0          2       0 5728/6000   0 ENABLE AAA
  89 Mwe 8025F9C0            0          2       0 5732/6000   0 LINE AAA
  90 Mwe 803C8D1C            0          2       0 5604/6000   0 TPLUS
  91 Lwe 808F6E08          360      14373      25 4368/6000   0 CEF process
  92 Mwe 810DB840            4          2    2000 5724/6000   0 Crypto Support
  93 Mwe 81387464            0          1       0 5800/6000   0 EM Background Pr
  94 Mwe 806D6DA0           20        452      44 5668/6000   0 CRM_CALL_UPDATE_
  95 Mwe 810D5BB4            0          1       011804/12000  0 Encrypt Proc
  96 Mwe 810D67A4        14128        117  120752 6696/8000   0 Key Proc
  97 Mwe 811B0058           24          4    6000 7000/8000   0 Crypto CA
  98 Mwe 811EB1D4            0          1       0 7812/8000   0 Crypto SSL
  99 Mwe 81158B3C           20         40     50020432/24000  0 Crypto ACL
 100 Mwe 810E00DC            0          1       0 5792/6000   0 CRYPTO QoS proce
 101 Mwe 81151F4C           12         14     85711332/12000  0 Crypto Delete Ma
 102 Mwe 8111F364          104        109     954 6180/12000  0 Crypto IKMP
 103 Mwe 8111445C         1212        587    2064 9744/12000  0 IPSEC key engine
 104 Mwe 81114F10            0          1       0 5716/6000   0 IPSEC manual key
 105 Mwe 80171B4C            0          2       0 5708/6000   0 AAA SEND STOP EV
 106 Mwe 80849E8C            0          1       0 5816/6000   0 Syslog Traps
 107 Lwe 812C945C            0          2       0 5648/6000   0 IpSecMibTopN
 108 Mwe 81324B6C           32       1216      26 5752/6000   0 PM Callback
 109 Mwe 80AF1508        38892    2766014      14 5584/6000   0 SAA Event Proces
 110 Mwe 80E45B78            0          1       0 5784/6000   0 VPDN Scal
 111 Mwe 81337730            4          2    2000 3912/6000   0 VLAN Manager
 112 Lsp 8132E680        18864      44308     425 5732/6000   0 COLLECT STAT COU
 113 Mwe 805D0D8C            0          1       011796/12000  0 TCP Driver
 114 Lwe 80435D60            0          1       0 5792/6000   0 TCP Listener
 115 Mwe 80AD3A54            0          1       0 5772/6000   0 IP NAT WLAN
 116 Mwe 81244984           40        169     236 4428/6000   0 SSH Event handle
 117 Mwe 8099DAD0         2664     349651       711696/12000  0 PPP manager
 118 Mwe 8027AE9C         2508     349661       711244/12000  0 PPP Events
 119 Hwe 809D5CA8           12      11239       1 5724/6000   0 Multilink PPP
 120 Mwe 809D568C            0          2       0 5712/6000   0 Multilink event
 121 Mwe 80876D7C            0          2       0 5732/6000   0 IP Flow Backgrou
 122 Mwe 810FBD70           48        249     192 4456/6000   0 Crypto Hardware
 123 Lwe 808F6638           16        367      43 5664/6000   0 CEF Scanner



RPNAPLES#sh proc cpu
CPU utilization for five seconds: 41%/39%; one minute: 34%; five minutes: 30%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   1          32         8       4000  0.00%  0.00%  0.00%   0 Chunk Manager
   2          28      2252         12  0.00%  0.00%  0.00%   0 Load Meter
   3         596       286       2083  1.19%  0.54%  0.12%   6 Virtual Exec
   4           0        94          0  0.00%  0.00%  0.00%   0 DHCPD Timer
   5       11444      1348       8489  0.00%  0.05%  0.05%   0 Check heaps
   6          32        39        820  0.00%  0.00%  0.00%   0 Pool Manager
   7           0         2          0  0.00%  0.00%  0.00%   0 Timers
   8           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun
   9           0         2          0  0.00%  0.00%  0.00%   0 AAA high-capacit
  10         948      1280        740  0.00%  0.00%  0.00%   0 ARP Input
  11           0         3          0  0.00%  0.00%  0.00%   0 DDR Timers
  12          32      2815         11  0.00%  0.00%  0.00%   0 HC Counter Timer
  13          12         2       6000  0.00%  0.00%  0.00%   0 Entity MIB API
  14           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer
  15           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect
  16          12     11246          1  0.00%  0.00%  0.00%   0 GraphIt
  17           0         2          0  0.00%  0.00%  0.00%   0 Dialer event
  18           0         2          0  0.00%  0.00%  0.00%   0 XML Proxy Client
  19           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd
  20         296      3273         90  0.00%  0.00%  0.00%   0 Net Background
  21           0        19          0  0.00%  0.00%  0.00%   0 Logger
  22          40     11242          3  0.00%  0.00%  0.00%   0 TTY Background
  23         132     11255         11  0.00%  0.00%  0.00%   0 Per-Second Jobs
  24           8         2       4000  0.00%  0.00%  0.00%   0 IPM_C1700_CLOCK
  25           0         2          0  0.00%  0.00%  0.00%   0 AggMgr Process
  26           4         2       2000  0.00%  0.00%  0.00%   0 ESWPPM
  27           0         1          0  0.00%  0.00%  0.00%   0 Net Input
  28          84      2253         37  0.00%  0.00%  0.00%   0 Compute load avg
  29        8352       192      43500  0.00%  0.03%  0.04%   0 Per-minute Jobs
  30           0         3          0  0.00%  0.00%  0.00%   0 Service-module a
  31           0         1          0  0.00%  0.00%  0.00%   0 Switch Link Moni
  32           4         2       2000  0.00%  0.00%  0.00%   0 AAA Server
  33           0         1          0  0.00%  0.00%  0.00%   0 AAA ACCT Proc
  34           0         1          0  0.00%  0.00%  0.00%   0 ACCT Periodic Pr
  35           0         1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT
  36           0         2          0  0.00%  0.00%  0.00%   0 AAA Dictionary R
  37     1655116   1351918       1224  1.35%  2.00%  5.19%   0 IP Input
  38           0         1          0  0.00%  0.00%  0.00%   0 ICMP event handl
  39           0         8          0  0.00%  0.00%  0.00%   0 CRYPTO IKMP IPC
  40           0         1          0  0.00%  0.00%  0.00%   0 SSS Manager
  41           4      1503          2  0.00%  0.00%  0.00%   0 SSS Test Client
  42           0         1          0  0.00%  0.00%  0.00%   0 SSS Feature Mana
  43         780     43982         17  0.00%  0.00%  0.00%   0 SSS Feature Time
  44           0         4          0  0.00%  0.00%  0.00%   0 PPP Hooks
  45           0         1          0  0.00%  0.00%  0.00%   0 X.25 Encaps Mana
  46           0         1          0  0.00%  0.00%  0.00%   0 VPDN call manage
  47           0         1          0  0.00%  0.00%  0.00%   0 L2X Data Daemon
  48           0         1          0  0.00%  0.00%  0.00%   0 L2X Socket proce
  49           0         1          0  0.00%  0.00%  0.00%   0 L2X SSS manager
  50           0         2          0  0.00%  0.00%  0.00%   0 L2TP mgmt daemon
  51           0         1          0  0.00%  0.00%  0.00%   0 AC Mgr
  52           0         2          0  0.00%  0.00%  0.00%   0 KRB5 AAA
  53           0         2          0  0.00%  0.00%  0.00%   0 DTP Protocol
  54           4         3       1333  0.00%  0.00%  0.00%   0 PPP IP Route
  55           0         4          0  0.00%  0.00%  0.00%   0 PPP IPCP
  56          32        19       1684  0.00%  0.00%  0.00%   0 DHCPD Receive
  57         524       215       2437  0.00%  0.00%  0.00%   0 IP Background
  58         112       194        577  0.00%  0.00%  0.00%   0 IP RIB Update
  59           0        36          0  0.00%  0.00%  0.00%   0 TCP Timer
  60           8         3       2666  0.00%  0.00%  0.00%   0 TCP Protocols
  61           0         1          0  0.00%  0.00%  0.00%   0 RARP Input
  62           0         1          0  0.00%  0.00%  0.00%   0 Socket Timers
  63           4        40        100  0.00%  0.00%  0.00%   0 HTTP CORE
  64          48       188        255  0.00%  0.00%  0.00%   0 IP Cache Ager
  65           0         1          0  0.00%  0.00%  0.00%   0 PAD InCall
  66           0         2          0  0.00%  0.00%  0.00%   0 X.25 Background
  67           0         2          0  0.00%  0.00%  0.00%   0 PPP SSS
  68          16       189         84  0.00%  0.00%  0.00%   0 Adj Manager
  69          72     21976          3  0.00%  0.00%  0.00%   0 IP NAT Ager
  70           4         2       2000  0.00%  0.00%  0.00%   0 PPP Bind
  71           0         1          0  0.00%  0.00%  0.00%   0 SNMP Timers
  72           0         1          0  0.00%  0.00%  0.00%   0 Inspect Timer
  73           0      3196          0  0.00%  0.00%  0.00%   0 DHCPD Database
  74           0         2          0  0.00%  0.00%  0.00%   0 URL filter proc
  75           0        38          0  0.00%  0.00%  0.00%   0 Authentication P
  76           0         1          0  0.00%  0.00%  0.00%   0 Auth-proxy AAA B
  77           0         1          0  0.00%  0.00%  0.00%   0 IDS Timer
  78           0         1          0  0.00%  0.00%  0.00%   0 COPS
  79           0         2          0  0.00%  0.00%  0.00%   0 Dialer Forwarder
  80           8      2253          3  0.00%  0.00%  0.00%   0 L2F management d
  81           4         1       4000  0.00%  0.00%  0.00%   0 PPTP Mgmt
  82           0         2          0  0.00%  0.00%  0.00%   0 PPTP Data
  83       25680     30471        842  0.00%  0.00%  0.00%   0 Crypto HW Proc
  84           0         1          0  0.00%  0.00%  0.00%   0 XSM_EVENT_ENGINE
  85           8      1126          7  0.00%  0.00%  0.00%   0 XSM_ENQUEUER
  86           4      1126          3  0.00%  0.00%  0.00%   0 XSM Historian
  87           0         2          0  0.00%  0.00%  0.00%   0 LOCAL AAA
  88           0         2          0  0.00%  0.00%  0.00%   0 ENABLE AAA
  89           0         2          0  0.00%  0.00%  0.00%   0 LINE AAA
  90           0         2          0  0.00%  0.00%  0.00%   0 TPLUS
  91         360     14394         25  0.00%  0.00%  0.00%   0 CEF process
  92           4         2       2000  0.00%  0.00%  0.00%   0 Crypto Support
  93           0         1          0  0.00%  0.00%  0.00%   0 EM Background Pr
  94          20       452         44  0.00%  0.00%  0.00%   0 CRM_CALL_UPDATE_
  95           0         1          0  0.00%  0.00%  0.00%   0 Encrypt Proc
  96       14128       117     120752  0.00%  0.00%  0.26%   0 Key Proc
  97          24         4       6000  0.00%  0.00%  0.00%   0 Crypto CA
  98           0         1          0  0.00%  0.00%  0.00%   0 Crypto SSL
  99          20        40        500  0.00%  0.00%  0.00%   0 Crypto ACL
 100           0         1          0  0.00%  0.00%  0.00%   0 CRYPTO QoS proce
 101          12        14        857  0.00%  0.00%  0.00%   0 Crypto Delete Ma
 102         104       109        954  0.00%  0.00%  0.00%   0 Crypto IKMP
 103        1212       588       2061  0.00%  0.00%  0.00%   0 IPSEC key engine
 104           0         1          0  0.00%  0.00%  0.00%   0 IPSEC manual key
 105           0         2          0  0.00%  0.00%  0.00%   0 AAA SEND STOP EV
 106           0         1          0  0.00%  0.00%  0.00%   0 Syslog Traps
 107           0         2          0  0.00%  0.00%  0.00%   0 IpSecMibTopN
 108          32      1218         26  0.00%  0.00%  0.00%   0 PM Callback
 109       38908   2770816         14  0.07%  0.03%  0.07%   0 SAA Event Proces
 110           0         1          0  0.00%  0.00%  0.00%   0 VPDN Scal
 111           4         2       2000  0.00%  0.00%  0.00%   0 VLAN Manager
 112       18904     44385        425  0.31%  0.17%  0.16%   0 COLLECT STAT COU
 113           0         1          0  0.00%  0.00%  0.00%   0 TCP Driver
 114           0         1          0  0.00%  0.00%  0.00%   0 TCP Listener
 115           0         1          0  0.00%  0.00%  0.00%   0 IP NAT WLAN
 116          40       169        236  0.00%  0.00%  0.00%   0 SSH Event handle
 117        2668    350258          7  0.00%  0.00%  0.00%   0 PPP manager
 118        2508    350268          7  0.00%  0.00%  0.00%   0 PPP Events
 119          12     11258          1  0.00%  0.00%  0.00%   0 Multilink PPP
 120           0         2          0  0.00%  0.00%  0.00%   0 Multilink event
 121           0         2          0  0.00%  0.00%  0.00%   0 IP Flow Backgrou
 122          48       249        192  0.00%  0.00%  0.00%   0 Crypto Hardware
 123          16       369         43  0.00%  0.00%  0.00%   0 CEF Scanner
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12662305
One question, was there anyone working at the time?

0
 

Author Comment

by:tangofniro
ID: 12664711
No it was around 9:00 pm
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12667024
How about right now? Run the same commands. To me the CPU last night was very high for no body around
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12667034
Also need to get the analyzer up and running, need to baseline!
0
 

Author Comment

by:tangofniro
ID: 12668045
right now


RPNAPLES#sh proc
CPU utilization for five seconds: 42%/33%; one minute: 47%; five minutes: 50%
 PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
   1 Cwe 8029A2D0          268         64    4187 5548/6000   0 Chunk Manager
   2 Csp 80280AB0          144      11797      12 2744/3000   0 Load Meter
   3 M*         0           20         20    1000 9756/12000  6 Virtual Exec
   4 Mwe 8053C650            8        492      16 5768/6000   0 DHCPD Timer
   5 Lst 80299ACC        62660       7260    8630 5752/6000   0 Check heaps
   6 Cwe 8029E304           40         45     888 5536/6000   0 Pool Manager
   7 Mst 801A58E4            0          2       0 5708/6000   0 Timers
   8 Mwe 80096BB8            0          2       0 5716/6000   0 Serial Backgroun
   9 Mwe 80178B38            0          2       0 5704/6000   0 AAA high-capacit
  10 Lwe 8034FD08         7476       9058     825 5008/6000   0 ARP Input
  11 Mwe 80371168            0          3       0 5708/6000   0 DDR Timers
  12 Mwe 80675650          516      13636      37 5768/6000   0 HC Counter Timer
  13 Lwe 80978F34           12          2    6000 5624/6000   0 Entity MIB API
  14 Mwe 80BE83B8            0          2       0 5712/6000   0 ATM Idle Timer
  15 Mwe 8009C57C            0          1       0 5788/6000   0 SERIAL A'detect
  16 Msp 801E85FC          188      58933       3 5744/6000   0 GraphIt
  17 Mwe 8038C274            0          2       011712/12000  0 Dialer event
  18 Mwe 806196F8            0          2       011728/12000  0 XML Proxy Client
  19 Cwe 8026C730            0          1       0 5804/6000   0 Critical Bkgnd
  20 Mwe 8021AC98         1804      10818     16610392/12000  0 Net Background
  21 Lwe 80196DF8            0         21       011356/12000  0 Logger
  22 Mwe 801BB920          300      58929       5 5736/6000   0 TTY Background
 PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
  23 Msp 8022BC78          936      58942      15 7464/9000   0 Per-Second Jobs
  24 Lwe 80311EC0           20          3    6666 1864/3000   0 IPM_C1700_CLOCK
  25 Mwe 812F69B4            0          2       0 5784/6000   0 AggMgr Process
  26 Hwe 81327A00            4          2    2000 5264/6000   0 ESWPPM
  27 Hwe 8022B7A4            0          5       0 5788/6000   0 Net Input
  28 Msp 80220A90         1100      11795      93 5740/6000   0 Compute load avg
  29 Msp 8022BCEC        45540        990   46000 5752/6000   0 Per-minute Jobs
  30 Mwe 8009F05C            0          3       0 5700/6000   0 Service-module a
  31 Mwe 813322AC            0          1       0 5792/6000   0 Switch Link Moni
  32 Mwe 8025067C            4          2    2000 5712/6000   0 AAA Server
  33 Mwe 80252B48            0          1       0 5784/6000   0 AAA ACCT Proc
  34 Mwe 80252C2C            0          1       0 5772/6000   0 ACCT Periodic Pr
  35 Lwe 8017D6A4            0          1       0 5788/6000   0 AAA_SERVER_DEADT
  36 Mwe 80304C34            0          2       0 5724/6000   0 AAA Dictionary R
  37 Mrd 8044EEF4      9094956    6541061    1390 9952/12000  0 IP Input
  38 Mwe 80473998            0          1       0 5768/6000   0 ICMP event handl
  39 Mwe 81159038            0          8       0 5788/6000   0 CRYPTO IKMP IPC
  40 Mwe 8085DB38            0          1       011788/12000  0 SSS Manager
  41 Mwe 80861464          164       7865      2011752/12000  0 SSS Test Client
  42 Mwe 8086AE94            0          1       0 5800/6000   0 SSS Feature Mana
  43 Mwe 8086AF28         2524     230403      10 5768/6000   0 SSS Feature Time
  44 Mwe 8027AE9C            0          4       011732/12000  0 PPP Hooks
  45 Lwe 80A546B4            0          1       0 5396/6000   0 X.25 Encaps Mana
 PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
  46 Mwe 80DFED4C            0          1       011748/12000  0 VPDN call manage
  47 Mwe 80E7AB9C            0          1       011780/12000  0 L2X Data Daemon
  48 Mwe 80E42F6C            0          1       011756/12000  0 L2X Socket proce
  49 Mwe 80E0FD10            0          1       011784/12000  0 L2X SSS manager
  50 Mwe 80E1CB74            0          2       011712/12000  0 L2TP mgmt daemon
  51 Mwe 8105F430            0          1       0 5776/6000   0 AC Mgr
  52 Mwe 81239BE4            0          2       011728/12000  0 KRB5 AAA
  53 Mwe 812E6C64            0          2       0 5540/6000   0 DTP Protocol
  54 Mwe 8027AE9C            4          3    133310988/12000  0 PPP IP Route
  55 Mwe 8027AE9C            0          4       011448/12000  0 PPP IPCP
  56 Mwe 80425860          264        125    2112 3868/6000   0 DHCPD Receive
  57 Mwe 80551360         3340       1011    3303 8224/9000   0 IP Background
  58 Mwe 805574F4          368        990     371 8736/9000   0 IP RIB Update
  59 Mst 8043214C            8         61     13110484/12000  0 TCP Timer
  60 Lwe 80437114           16          6    266610332/12000  0 TCP Protocols
  61 Mwe 8049E14C            0          1       0 5804/6000   0 RARP Input
  62 Hwe 80533708            0          1       0 5776/6000   0 Socket Timers
  63 Mwe 805090E4          584        202    2891 6448/9000   0 HTTP CORE
  64 Lsi 805CB69C         1196        983    1216 5304/6000   0 IP Cache Ager
  65 Hwe 80A6DB7C            0          1       0 5780/6000   0 PAD InCall
  66 Mwe 80A27320            0          2       011708/12000  0 X.25 Background
  67 Mwe 8086CC34            0          2       0 5708/6000   0 PPP SSS
  68 Mwe 80908750          644        985     653 8688/9000   0 Adj Manager
 PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
  69 Mwe 80AA1614         1160     115124      10 5700/6000   0 IP NAT Ager
  70 Mwe 8027AE9C            4          2    2000 5728/6000   0 PPP Bind
  71 Mwe 8096E9F4            0          1       0 5756/6000   0 SNMP Timers
  72 Mwe 80C7B920            0          1       0 5796/6000   0 Inspect Timer
  73 Msi 8054B8C4          148      16711       8 4980/6000   0 DHCPD Database
  74 Mwe 80CA5BF8            0          2       0 5612/6000   0 URL filter proc
  75 Mwe 80CBF034            0        197       0 5796/6000   0 Authentication P
  76 Mwe 80CC8EA0            0          1       0 5776/6000   0 Auth-proxy AAA B
  77 Mwe 80CC9C64            0          1       0 5796/6000   0 IDS Timer
  78 Mwe 80DE114C            0          1       023780/24000  0 COPS
  79 Mwe 80DEFDF0            0          2       0 5720/6000   0 Dialer Forwarder
  80 Mwe 80E06E3C           68      11794       511776/12000  0 L2F management d
  81 Mwe 80E357FC            4          1    400011540/12000  0 PPTP Mgmt
  82 Mwe 80E7F4DC            0          2       011728/12000  0 PPTP Data
  83 Hwe 810EDD9C        25704      30502     842 5064/6000   0 Crypto HW Proc
  84 Lwe 812DB7A4            0          1       0 5796/6000   0 XSM_EVENT_ENGINE
  85 Lsi 812D93DC          116       5895      1911824/12000  0 XSM_ENQUEUER
  86 Lsi 812DC4D4           80       5895      1311828/12000  0 XSM Historian
  87 Mwe 8025D51C            0          2       0 5728/6000   0 LOCAL AAA
  88 Mwe 8025F5D4            0          2       0 5728/6000   0 ENABLE AAA
  89 Mwe 8025F9C0            0          2       0 5732/6000   0 LINE AAA
  90 Mwe 803C8D1C            0          2       0 5604/6000   0 TPLUS
  91 Lwe 808F6E08         3460      79767      43 4368/6000   0 CEF process
 PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
  92 Mwe 810DB840            4          2    2000 5724/6000   0 Crypto Support
  93 Mwe 81387464            0          1       0 5800/6000   0 EM Background Pr
  94 Mwe 806D6DA0           52       2361      22 5668/6000   0 CRM_CALL_UPDATE_
  95 Mwe 810D5BB4            0          1       011804/12000  0 Encrypt Proc
  96 Mwe 810D67A4        64116        467  137293 6684/8000   0 Key Proc
  97 Mwe 811B0058           24          4    6000 7000/8000   0 Crypto CA
  98 Mwe 811EB1D4            0          1       0 7812/8000   0 Crypto SSL
  99 Mwe 81158B3C           20         40     50020432/24000  0 Crypto ACL
 100 Mwe 810E00DC            0          1       0 5792/6000   0 CRYPTO QoS proce
 101 Mwe 81151F4C           36         53     67911332/12000  0 Crypto Delete Ma
 102 Mwe 8111F364          424        386    1098 6168/12000  0 Crypto IKMP
 103 Mwe 8111445C         1268       3013     420 9744/12000  0 IPSEC key engine
 104 Mwe 81114F10            0          1       0 5716/6000   0 IPSEC manual key
 105 Mwe 80171B4C            0          2       0 5708/6000   0 AAA SEND STOP EV
 106 Mwe 80849E8C            0          1       0 5816/6000   0 Syslog Traps
 107 Lwe 812C945C            0          2       0 5648/6000   0 IpSecMibTopN
 108 Mwe 81324B6C          136       6375      21 5752/6000   0 PM Callback
 109 Mwe 80AF1508       190808   14415505      13 5584/6000   0 SAA Event Proces
 110 Mwe 80E45B78            0          1       0 5784/6000   0 VPDN Scal
 111 Mwe 81337730            4          2    2000 3912/6000   0 VLAN Manager
 112 Lsp 8132E680       112484     232949     482 5732/6000   0 COLLECT STAT COU
 113 Mwe 805D0D8C            0          1       011796/12000  0 TCP Driver
 114 Lwe 80435D60            0          1       0 5792/6000   0 TCP Listener
 PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
 115 Mwe 80AD3A54            0          1       0 5772/6000   0 IP NAT WLAN
 116 Mwe 81244984          180        715     251 4428/6000   0 SSH Event handle
 117 Mwe 8099DAD0        31696    1838473      1711696/12000  0 PPP manager
 118 Mwe 8027AE9C        33004    1838482      1711244/12000  0 PPP Events
 119 Hwe 809D5CA8           80      58974       1 5724/6000   0 Multilink PPP
 120 Mwe 809D568C            0          2       0 5712/6000   0 Multilink event
 121 Mwe 80876D7C            0          2       0 5732/6000   0 IP Flow Backgrou
 122 Mwe 810FBD70          208        983     211 4456/6000   0 Crypto Hardware
 123 Lwe 808F6638          140       2023      69 5664/6000   0 CEF Scanner
RPNAPLES#sh proc cpu
CPU utilization for five seconds: 33%/22%; one minute: 43%; five minutes: 49%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   1         268        64       4187  0.00%  0.00%  0.00%   0 Chunk Manager
   2         144     11801         12  0.00%  0.00%  0.00%   0 Load Meter
   3         348      1061        327  0.15%  0.36%  0.08%   6 Virtual Exec
   4           8       492         16  0.00%  0.00%  0.00%   0 DHCPD Timer
   5       62660      7260       8630  0.00%  0.06%  0.05%   0 Check heaps
   6          40        45        888  0.00%  0.00%  0.00%   0 Pool Manager
   7           0         2          0  0.00%  0.00%  0.00%   0 Timers
   8           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun
   9           0         2          0  0.00%  0.00%  0.00%   0 AAA high-capacit
  10        7480      9061        825  0.00%  0.00%  0.00%   0 ARP Input
  11           0         3          0  0.00%  0.00%  0.00%   0 DDR Timers
  12         516     13641         37  0.00%  0.00%  0.00%   0 HC Counter Timer
  13          12         2       6000  0.00%  0.00%  0.00%   0 Entity MIB API
  14           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer
  15           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect
  16         188     58954          3  0.00%  0.00%  0.00%   0 GraphIt
  17           0         2          0  0.00%  0.00%  0.00%   0 Dialer event
  18           0         2          0  0.00%  0.00%  0.00%   0 XML Proxy Client
  19           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd
  20        1812     10820        167  0.07%  0.02%  0.00%   0 Net Background
  21           0        21          0  0.00%  0.00%  0.00%   0 Logger
  22         300     58950          5  0.00%  0.00%  0.00%   0 TTY Background
  23         936     58963         15  0.00%  0.00%  0.00%   0 Per-Second Jobs
  24          20         3       6666  0.00%  0.00%  0.00%   0 IPM_C1700_CLOCK
  25           0         2          0  0.00%  0.00%  0.00%   0 AggMgr Process
  26           4         2       2000  0.00%  0.00%  0.00%   0 ESWPPM
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  27           0         5          0  0.00%  0.00%  0.00%   0 Net Input
  28        1104     11798         93  0.00%  0.00%  0.00%   0 Compute load avg
  29       45592       991      46006  0.63%  0.07%  0.04%   0 Per-minute Jobs
  30           0         3          0  0.00%  0.00%  0.00%   0 Service-module a
  31           0         1          0  0.00%  0.00%  0.00%   0 Switch Link Moni
  32           4         2       2000  0.00%  0.00%  0.00%   0 AAA Server
  33           0         1          0  0.00%  0.00%  0.00%   0 AAA ACCT Proc
  34           0         1          0  0.00%  0.00%  0.00%   0 ACCT Periodic Pr
  35           0         1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT
  36           0         2          0  0.00%  0.00%  0.00%   0 AAA Dictionary R
  37     9097020   6543285       1390 11.27%  8.68%  9.27%   0 IP Input
  38           0         1          0  0.00%  0.00%  0.00%   0 ICMP event handl
  39           0         8          0  0.00%  0.00%  0.00%   0 CRYPTO IKMP IPC
  40           0         1          0  0.00%  0.00%  0.00%   0 SSS Manager
  41         164      7867         20  0.00%  0.00%  0.00%   0 SSS Test Client
  42           0         1          0  0.00%  0.00%  0.00%   0 SSS Feature Mana
  43        2524    230460         10  0.00%  0.00%  0.00%   0 SSS Feature Time
  44           0         4          0  0.00%  0.00%  0.00%   0 PPP Hooks
  45           0         1          0  0.00%  0.00%  0.00%   0 X.25 Encaps Mana
  46           0         1          0  0.00%  0.00%  0.00%   0 VPDN call manage
  47           0         1          0  0.00%  0.00%  0.00%   0 L2X Data Daemon
  48           0         1          0  0.00%  0.00%  0.00%   0 L2X Socket proce
  49           0         1          0  0.00%  0.00%  0.00%   0 L2X SSS manager
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  50           0         2          0  0.00%  0.00%  0.00%   0 L2TP mgmt daemon
  51           0         1          0  0.00%  0.00%  0.00%   0 AC Mgr
  52           0         2          0  0.00%  0.00%  0.00%   0 KRB5 AAA
  53           0         2          0  0.00%  0.00%  0.00%   0 DTP Protocol
  54           4         3       1333  0.00%  0.00%  0.00%   0 PPP IP Route
  55           0         4          0  0.00%  0.00%  0.00%   0 PPP IPCP
  56         264       125       2112  0.00%  0.00%  0.00%   0 DHCPD Receive
  57        3340      1011       3303  0.00%  0.00%  0.00%   0 IP Background
  58         368       990        371  0.00%  0.00%  0.00%   0 IP RIB Update
  59           8        61        131  0.00%  0.00%  0.00%   0 TCP Timer
  60          16         6       2666  0.00%  0.00%  0.00%   0 TCP Protocols
  61           0         1          0  0.00%  0.00%  0.00%   0 RARP Input
  62           0         1          0  0.00%  0.00%  0.00%   0 Socket Timers
  63         584       202       2891  0.00%  0.00%  0.00%   0 HTTP CORE
  64        1196       984       1215  0.00%  0.00%  0.00%   0 IP Cache Ager
  65           0         1          0  0.00%  0.00%  0.00%   0 PAD InCall
  66           0         2          0  0.00%  0.00%  0.00%   0 X.25 Background
  67           0         2          0  0.00%  0.00%  0.00%   0 PPP SSS
  68         644       985        653  0.00%  0.00%  0.00%   0 Adj Manager
  69        1160    115153         10  0.00%  0.00%  0.00%   0 IP NAT Ager
  70           4         2       2000  0.00%  0.00%  0.00%   0 PPP Bind
  71           0         1          0  0.00%  0.00%  0.00%   0 SNMP Timers
  72           0         1          0  0.00%  0.00%  0.00%   0 Inspect Timer
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  73         148     16728          8  0.00%  0.00%  0.00%   0 DHCPD Database
  74           0         2          0  0.00%  0.00%  0.00%   0 URL filter proc
  75           0       197          0  0.00%  0.00%  0.00%   0 Authentication P
  76           0         1          0  0.00%  0.00%  0.00%   0 Auth-proxy AAA B
  77           0         1          0  0.00%  0.00%  0.00%   0 IDS Timer
  78           0         1          0  0.00%  0.00%  0.00%   0 COPS
  79           0         2          0  0.00%  0.00%  0.00%   0 Dialer Forwarder
  80          68     11797          5  0.00%  0.00%  0.00%   0 L2F management d
  81           4         1       4000  0.00%  0.00%  0.00%   0 PPTP Mgmt
  82           0         2          0  0.00%  0.00%  0.00%   0 PPTP Data
  83       25704     30502        842  0.00%  0.00%  0.00%   0 Crypto HW Proc
  84           0         1          0  0.00%  0.00%  0.00%   0 XSM_EVENT_ENGINE
  85         116      5896         19  0.00%  0.00%  0.00%   0 XSM_ENQUEUER
  86          80      5896         13  0.00%  0.00%  0.00%   0 XSM Historian
  87           0         2          0  0.00%  0.00%  0.00%   0 LOCAL AAA
  88           0         2          0  0.00%  0.00%  0.00%   0 ENABLE AAA
  89           0         2          0  0.00%  0.00%  0.00%   0 LINE AAA
  90           0         2          0  0.00%  0.00%  0.00%   0 TPLUS
  91        3464     79785         43  0.07%  0.00%  0.00%   0 CEF process
  92           4         2       2000  0.00%  0.00%  0.00%   0 Crypto Support
  93           0         1          0  0.00%  0.00%  0.00%   0 EM Background Pr
  94          52      2362         22  0.00%  0.00%  0.00%   0 CRM_CALL_UPDATE_
  95           0         1          0  0.00%  0.00%  0.00%   0 Encrypt Proc
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  96       64116       467     137293  0.00%  0.00%  0.00%   0 Key Proc
  97          24         4       6000  0.00%  0.00%  0.00%   0 Crypto CA
  98           0         1          0  0.00%  0.00%  0.00%   0 Crypto SSL
  99          20        40        500  0.00%  0.00%  0.00%   0 Crypto ACL
 100           0         1          0  0.00%  0.00%  0.00%   0 CRYPTO QoS proce
 101          36        53        679  0.00%  0.00%  0.00%   0 Crypto Delete Ma
 102         424       386       1098  0.00%  0.00%  0.00%   0 Crypto IKMP
 103        1268      3014        420  0.00%  0.00%  0.00%   0 IPSEC key engine
 104           0         1          0  0.00%  0.00%  0.00%   0 IPSEC manual key
 105           0         2          0  0.00%  0.00%  0.00%   0 AAA SEND STOP EV
 106           0         1          0  0.00%  0.00%  0.00%   0 Syslog Traps
 107           0         2          0  0.00%  0.00%  0.00%   0 IpSecMibTopN
 108         136      6377         21  0.00%  0.00%  0.00%   0 PM Callback
 109      190920  14418995         13  0.71%  0.32%  0.34%   0 SAA Event Proces
 110           0         1          0  0.00%  0.00%  0.00%   0 VPDN Scal
 111           4         2       2000  0.00%  0.00%  0.00%   0 VLAN Manager
 112      112516    233006        482  0.15%  0.17%  0.20%   0 COLLECT STAT COU
 113           0         1          0  0.00%  0.00%  0.00%   0 TCP Driver
 114           0         1          0  0.00%  0.00%  0.00%   0 TCP Listener
 115           0         1          0  0.00%  0.00%  0.00%   0 IP NAT WLAN
 116         180       715        251  0.00%  0.00%  0.00%   0 SSH Event handle
 117       31696   1838902         17  0.00%  0.01%  0.03%   0 PPP manager
 118       33008   1838911         17  0.00%  0.01%  0.04%   0 PPP Events
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 119          80     58988          1  0.00%  0.00%  0.00%   0 Multilink PPP
 120           0         2          0  0.00%  0.00%  0.00%   0 Multilink event
 121           0         2          0  0.00%  0.00%  0.00%   0 IP Flow Backgrou
 122         208       983        211  0.00%  0.00%  0.00%   0 Crypto Hardware
 123         140      2024         69  0.00%  0.00%  0.00%   0 CEF Scanner
RPNAPLES#
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12668395
Do this for every hour for the next three hours.
0
 

Author Comment

by:tangofniro
ID: 12668581
Okay

I have the full version of Colasoft running right now also.

No problems reported today.  Running testvoip also showing good.

colasoft showing me rigt now after about 1 hr running  4mb internet total
                                                                                450mb local/intranet  total
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12668635
Grab a coke and smoke and wait.

Mkae sure that you are savings all the captures for review later on, do it every hour if you can. The best thing we can do is leave this running till the problem pops back up.
0
 

Author Comment

by:tangofniro
ID: 12669248
will let it run over the holidays,
I will out of town until Sunday.
They said it got a little static before lunch  and after, but nothing horrible.  I never saw anything to cause it though.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12669404
Always hard to find a ghost. One thing to think about is that though yo may have a lot of traffic, it still may be the carrier. The best test would be to set up a monitor at site A, then one at site B. Set up two computers to "talk" to each other over various ports/UDP-TCP, also set up two VoIP phones also (usea audio source via speakerphone). This way we could actually see the packets leaving site A and arrive at site B. From this we could tell if the carrier is causing problems some how.

Did you adjust the MTU size by chance already?


Have a good holiday!

Joel
0
 

Author Comment

by:tangofniro
ID: 12669474
MTU is 900 per your request.

Have a great holiday.

Will
0
 

Author Comment

by:tangofniro
ID: 12726815
Hello Joel,

Hope you had a good holiday.

Would you please do me a big favor and look at the configs again to make sure I have it correct.
Phone quality has been terrible and I just don't know if my QOS is working properly.

172.168.1.0 is the remote internal network with the ip phones.  

It connects back to the 192.168.0.0 network where the phone equip resides.

I have a full T1 at the 192   and 1000k at the 172.

Thanks,
Will
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12729439
Will,

Had a good holiday, I sent a request to one of the other Experts (Dr-IP) his skill set for QoS on Cisco with VoIP is the best I have seen at EE. Always remeber to get a second Dr.'s opinion (LOL).

Joel
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12732087
I have only glimpsed at the thread, but it is clear that the router CPU had a heavy load on it even when it should be doing almost nothing, and if it’s that high when it should be idle, it probably is overloading during the day. So the priority here should be finding out why it’s so high when it shouldn’t, and correcting it. Once that is taken care of, if you are still having issues, they can then be addressed.

The first question is, where is the load coming from? I’d log into one of the routers, and shut down the Ethernet interface on it, and telnet from it to the other router, and do that same. Wait a few minutes and do a show process CPU on both routers and see if it drops, if it doesn’t do a show interface serial X on both routers, and see if you are getting a lot of trash coming from your ISP. If so, call them up and work with them to eliminate it.

Now if the trash isn’t coming from your ISP, turn up the Ethernet interface one router at a time and see if it gets pounded, show interface FastEthernet X, if dose, find out what on that LAN is causing it. An easy way to do that is yank all the connectors out of the switch connected to the router, and plug them back in a few at a time until the traffic spikes back up. When it dose, pull out the last group of wires and plug them in one at a time to find the guilty party.

I suspect by the way you have some kind of worm on your LAN, or people running things they shouldn’t P2P software, which can easily overload a T1 if used heavily, but it also could be some network app, that is just too big a load for your internet connection.
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12732095
Note log on using the consol cable.
0
 

Author Comment

by:tangofniro
ID: 12732441
Well I closed down the fe ports on both and the it went down to nothing.  

I will try to narrow down the problems internally.


Can you look at the access lists so I can try to block most traffic except the phones and the 3389 for terminal server?

Thanks Dr and Joel



will
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12732508
We could try doing that, but the router still has to address all that trash, which loads down the CPU, although maybe not as much. So let’s get rid of trash first and latter we can play with the access lists. Also, just to be sure, you are using a switch, because if you are using a hub, the router will get a lot of needless LAN traffic.
0
 

Author Comment

by:tangofniro
ID: 12741860
Ok well here it is  the lan throws nothing on the router when just the lan is plugged in I am running 2% maybe every once in a while  10%.  When I plug the phone system in is when the cpu gets overloaded.

help



thanks,

will
0
 

Author Comment

by:tangofniro
ID: 12742249
after clearing counters after hours  about 3min after clearing


RPNAPLES#sh access-lists
Extended IP access list 100
    10 permit udp any any range 16384 32000
    20 permit tcp any any eq 1719
    30 permit tcp any any eq 1720
    40 permit tcp any any eq 6100
    50 permit tcp any any range 1024 4999 (459 matches)
    60 permit udp any any eq 6000 (717 matches)
    70 permit udp any any range 1024 4999 (11 matches)
    80 permit udp any any eq 5060
    90 permit udp any any range 30000 30030
    100 permit udp any any range 9000 9001 (10800 matches)
Extended IP access list 101
    10 permit ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255 (36091 matches)
    20 deny ip 192.168.0.0 0.0.0.255 any
Extended IP access list 102
    10 permit tcp any any range 5000 5110
Extended IP access list 105
    10 deny icmp any any echo
    20 permit ip any any
Extended IP access list 110
    10 deny ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255 (2 matches)
    20 deny ip host 192.168.0.205 any (171 matches)
    30 deny ip host 192.168.0.30 any
    40 deny ip host 192.168.0.31 any
    50 deny ip host 192.168.0.2 any (57 matches)
    60 permit ip 192.168.0.0 0.0.0.255 any (24 matches)
Extended IP access list 120
    10 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255
Extended IP access list 123
    10 permit ip host 192.168.0.205 172.168.1.0 0.0.0.255 (20 matches)
    20 permit ip host 192.168.0.30 172.168.1.0 0.0.0.255 (550 matches)
    30 permit ip host 192.168.0.31 172.168.1.0 0.0.0.255 (7761 matches)
    40 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255 (84 matches)
Extended IP access list sl_def_acl
    10 deny tcp any any eq telnet log
    20 deny tcp any any eq www log
    30 deny tcp any any eq 22 log
    40 permit ip any any log
RPNAPLES#
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12743953
To say that is odd would be a gross understatement, and it definitely is not what I would have expected. I work with carrier grade VOIP equipment, and some of those gateways can handle over 700 calls, yet when they are idle, at the most they use a few kilobytes of bandwidth to periodically renew their registration to the gatekeeper.

So there is no reason on earth a little office VOIP system should use anywhere even remotely close to what your system is using when idle. It’s not playing background music on the phones like I have seen in some offices, or something like that? As that is the only thing I can think of that could account for what you are seeing.      
0
 

Author Comment

by:tangofniro
ID: 12744277
No back ground music.  

I am stumped.  It was around 7:00pm and everyone had been gone since 5:00.
0
 

Author Comment

by:tangofniro
ID: 12744289
here is the access-lists from this morning.


RPNAPLES#sh access-list
Extended IP access list 100
    10 permit udp any any range 16384 32000
    20 permit tcp any any eq 1719
    30 permit tcp any any eq 1720
    40 permit tcp any any eq 6100
    50 permit tcp any any range 1024 4999 (7689 matches)
    60 permit udp any any eq 6000 (81348 matches)
    70 permit udp any any range 1024 4999 (3443 matches)
    80 permit udp any any eq 5060
    90 permit udp any any range 30000 30030
    100 permit udp any any range 9000 9001 (2795673 matches)
Extended IP access list 101
    10 permit ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255 (9532221 matches)
    20 deny ip 192.168.0.0 0.0.0.255 any
Extended IP access list 102
    10 permit tcp any any range 5000 5110 (55 matches)
Extended IP access list 105
    10 deny icmp any any echo
    20 permit ip any any
Extended IP access list 110
    10 deny ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255 (485 matches)
    20 deny ip host 192.168.0.205 any (59493 matches)
    30 deny ip host 192.168.0.30 any
    40 deny ip host 192.168.0.31 any
    50 deny ip host 192.168.0.2 any (12442 matches)
    60 permit ip 192.168.0.0 0.0.0.255 any (4405 matches)
Extended IP access list 120
    10 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255
Extended IP access list 123
    10 permit ip host 192.168.0.205 172.168.1.0 0.0.0.255 (2396 matches)
    20 permit ip host 192.168.0.30 172.168.1.0 0.0.0.255 (65236 matches)
    30 permit ip host 192.168.0.31 172.168.1.0 0.0.0.255 (1866125 matches)
    40 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255 (17106 matches)
Extended IP access list sl_def_acl
    10 deny tcp any any eq telnet log
    20 deny tcp any any eq www log
    30 deny tcp any any eq 22 log
    40 permit ip any any log
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12744711
A show interface would be more relevant here since we know what is generating most of the traffic, but counting the match’s on ports 9000-1, I assume that its for the phone system, for what you say was three minutes, it looks like the phone systems using enough bandwidth for a good twenty plus calls when idle.

On a local LAN that wouldn’t be a problem, but across a slow WAN link, running through a VPN, on small router with a modest processor, processing access lists and doing QOS, it’s a recipe for trouble, but I think you have figured that out already. Also seeing how much bandwidth the phones are using when idle, makes me wonder how much bandwidth they use when in operation, and if a single T1 can handle it?

I think it’s time you get a hold of the people, who provided your phone system, as there is nothing I can do about other than recommend you get more powerful routers. You might still need to do that since the routers you have a marginal for what you are dong with them, but until get rid of this parasitic load and see how it behaves I can’t say that. Also even worse, in addition too more powerful routers you might need to get another T1 to deal with all the bandwidth that phone system is sucking up.

So trying to fix this on the Cisco end would be expensive, especially if it came down to getting more T1’s. So this needs to be addressed at the phone end of things if at all possible, since with all seven remote phones in operation it should be using about 70-80K of bandwidth, but it looks like it’s using closer to 250K when idle, and probable a hell of lot more in operation. There has got to be something in it’s configuration to cut this down to something reasonable, but since I don’t work with Samsung phone systems I have no clue as to where to start.  
0
 

Author Comment

by:tangofniro
ID: 12744777
here is my cleared interfaces after 5mins

RPNAPLES#sh int s0
Serial0 is up, line protocol is up
  Hardware is PQUICC with Fractional T1 CSU/DSU
  Description: USLEC T1
  Internet address is 199.72.194.234/30
  MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 12/255, rxload 36/255
  Encapsulation PPP, LCP Open
  Open: IPCP, loopback not set
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:10:36
  Input queue: 3/75/0/0 (size/max/drops/flushes); Total output drops: 8
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/8 (size/max total/threshold/drops)
     Conversations  0/8/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 52 kilobits/sec
  5 minute input rate 219000 bits/sec, 304 packets/sec
  5 minute output rate 78000 bits/sec, 81 packets/sec
     193593 packets input, 17672596 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     50723 packets output, 6189640 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up



RPNAPLES#sh int f0
FastEthernet0 is up, line protocol is up
  Hardware is PQUICC_FEC, address is 000f.f7f8.4b2a (bia 000f.f7f8.4b2a)
  Description: $FW_INSIDE$$ETH-LAN$
  Internet address is 192.168.0.1/24
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:12:13
  Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: priority-list 5
  Output queue (queue priority: size/max/drops):
     high: 0/20/0, medium: 0/40/0, normal: 0/60/0, low: 0/80/0
  5 minute input rate 60000 bits/sec, 81 packets/sec
  5 minute output rate 155000 bits/sec, 302 packets/sec
     57995 packets input, 5391239 bytes
     Received 167 broadcasts, 0 runts, 0 giants, 0 throttles
     914 input errors, 299 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog
     0 input packets with dribble condition detected
     222447 packets output, 14317892 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out



I will keep you posted
0
 

Author Comment

by:tangofniro
ID: 12744780
didn't really want to post my ip address.
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12744949
I won’t worry too much about posting your routers IP address, a lot of people here are way too paranoid about that. The honest truth is, if your equipment is that poorly secured; it will be found, and hacked regardless. At least once a day, someone scans all 4000 of my IP address to find active address to attempt to compromise systems using automated tools and scripts, that where the name script kiddies comes from by the way, and if it wasn’t secured with proper passwords and access lists, I’d be in trouble.

What I would worry about it posting passwords, or the address of VOIP gateways, as there are more than a few tricks that can be used on a lot of them to trick them into taking calls without the need to even hack them. That’s because they are inherently insecure, and a lot of people are unaware of that, and or don’t have a clue as to how to properly secure them.

Just to give you an idea of have bad your situation is, below I have posted the show interface from one of my smaller VOIP gateways that had 40 calls on it at the time. As you can see, your packet rate when idle is pretty close to mine under far more calls than you will ever have. It’s the packet rate that really loads down the processor on your router by the way, as each one has to be processed to determine how it is going to be handled, and with QOS and all those access lists it puts a pretty big load on the little routers CPU.

As for you show interfaces, it confirms what I expected.      


5309#sh int f0
FastEthernet0 is up, line protocol is up
  Hardware is DEC21140, address is 0010.7be6.4ff9 (bia 0010.7be6.4ff9)
  Internet address is 10.0.0.179/24
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/512/0/25 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 322000 bits/sec, 399 packets/sec
  5 minute output rate 317000 bits/sec, 391 packets/sec
     436079427 packets input, 1052027595 bytes
     Received 3651511 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog
     0 input packets with dribble condition detected
     336035809 packets output, 4175800439 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
5309#
0
 

Author Comment

by:tangofniro
ID: 12745310
better news called my phone guy and he reset the system


here is what it shows


RPNAPLES#sh int f0
FastEthernet0 is up, line protocol is up
  Hardware is PQUICC_FEC, address is 000f.f7f8.4b2a (bia 000f.f7f8.4b2a)
  Description: $FW_INSIDE$$ETH-LAN$
  Internet address is 192.168.0.1/24
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:10:09
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: priority-list 5
  Output queue (queue priority: size/max/drops):
     high: 0/20/0, medium: 0/40/0, normal: 0/60/0, low: 0/80/0
  5 minute input rate 2000 bits/sec, 4 packets/sec
  5 minute output rate 1000 bits/sec, 2 packets/sec
     3443 packets input, 252563 bytes
     Received 156 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog
     0 input packets with dribble condition detected
     1596 packets output, 128515 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out



cpu is around  1%-5%

will see if voice quality is any better

after the reset  9000 - 9001 is not getting any traffic now.  which it was getting hammered before.

0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12745366
That’s more like it, and getting rid of that extraneous load can only help call quality. That router doesn’t have a very powerful processor, and having a load of that level on it before it even starts processing meaningful traffic is bound to send it into overload. The main thing is to keep an eye on that phone system, to see if it starts doing it again. There might be some bug in the code that will keep reappearing, so watch it closely.  
0
 

Author Comment

by:tangofniro
ID: 12745388
10  4
I will see how it goes come monday  and report back.

So many thanks though to you guys pointing me in the right direction.

I have not forgotten about you either Joel.


Thanks,
will
0
 

Author Comment

by:tangofniro
ID: 12748495
Well I checked it this morning and it is up to around 20%

I also have around 2000crcs on the ethernet interface


5min input rate is 23000 bits a second    32 packets/sec

5min output rate is 53000 bits /sec   102 packets/sec  
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12749404
When did you reset the CRC count?

Also port 9000 is the default Voice port the Samsung phones use. Are you using OfficeServ 500 or 7200? Are using any of the applications? OfficeServ LInk, Operator, Call Monitor, ACD?

One question is that are people using Call Forward or DND on their phones? Since 9000 is for the voice payload, depending upon the Samsung system to reconize call routing logical loops, you could have someone who has call forwared their phone to someone who has forward theri phone to someplace else, even the other phone.

Since we have reconized that commmunications on port 9000 is the major problem, lets use Colasoft or even Ethereal to capture the packets and see where they are going?

Also what type of switches are you using in your network? Also the exact equipment you have via the phone system.

Also just to mention that 9000/9001 port is used in Microsoft's massively-
multiplayer game called "Asheron's Call".The game can continue to contact the player even after the player has logged out.

http://www.fuzeqna.com/asheronscall/consumer/kbdetail.asp?kbid=232

Not saying this is the problem but you never know, do you know if you have any gamers on the Network?
0
 

Author Comment

by:tangofniro
ID: 12749459
I reset the crc count yesterday afternoon.   I think port 9000 is the signal port on samsung.


I use some very simplistic Linksys/DLINK 10/100 , which could be weak I suppose.

No gamers in the office

I will find what the phone use software wise
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12749639
It looks like it’s going up quickly, and will probably be back to what it was before it was reset in another day or two. As for the cheap switches, they should be OK; the important thing is not to be using hubs, as the router would be seeing every packet going across the LAN. One other thing you will want to check with the phone people, what codec are they using? You what to be sure they are not using G711, as it uses a lot of bandwidth, it’s OK for using on the local LAN, but you don’t want to use it for the remote phones. For them you want to be using G729, or G723 since they give the best combination of voice quality verses bandwidth usage.    
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12749900
6000 is the typical signal port for the Samsung phones. Do you have the ability to enter into the MMC programing state on the phones? If so we can verify the codecs and ports being used.

Press Transfer button on phone, enter 800, then passcode (default is 4321). Press 1 to enable tenant, press speaker button, dial MMC code 834 835 or 840.

834 option numer 14 will show signal port, but also check the following options under this MMC 00, 01,11 and post what they are.

835 will show the type of MGI (1,2 or 3). Let me know what is configured. If MGI3, the default codec should be G.729a, but there are a fewother parameters that we can check, but need to find out what is being supported.

840 options, 04, 05, 07, 08

0
 

Author Comment

by:tangofniro
ID: 12750043
They have changed the default passcode.  Can't get a hold of the phone guy either.  Will find out everything tomorrow.


Thanks Joel

will
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12750068
While we wait on that can you capture some of the packets? Have you used Ethereal before? If you can capture some of the packets and post it, we can hunt down to see if the communcation is between a few endpoints or a system wide configuration. You can use the filter feature of Ethereal and just capture the packets using port 9000.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12750075
Overall how do you like the Samsung system? I sold and installed the Samsung DCS for a while, great little system just most people looked at us strange when we brought up the word Samsung (branded with cheap kitchen appliances, though they are different divisions entirely).
0
 

Author Comment

by:tangofniro
ID: 12750284
I haver ethereal on the network but it is on one of my server 2 switches down the line.

I can't capture from there can I?
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12750315
No easily (and would need a managed switch to do so).

what about installing on same machine as Colasoft?
0
 

Author Comment

by:tangofniro
ID: 12750326
I am not in the office and the cola soft is on my laptop.  I usally just put a hub on the network and plu in after the router but can't do that right now.


To answer the question about the System It has been great/ Except the remote phones..ha

Samsung has been making good stuff for a while, I think.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12750445
Okay on the Ethereal, what i am looking to do is isolate where the problem might be. We know thw phone system is part of it. But is it becasue of traffic from PBX to remote phones or all phones? Guess we can hit this tommorow!
0
 

Author Comment

by:tangofniro
ID: 12757128
I looked at it and colasoft does not seem to give me the correct traffic.  I don't see the traffic that heavy.

Also on another note  all traffic from the remote site has a hostname of A****.ipt.aol.com    what the hell is that??


Router is running back around 70-80 percent.  I just don't see it on the basline program.  I see a few 9000-9001 and alot of 30000-30009 but the traffic is only 40kb sec looking at my software.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12758251
When I was looking up the ports I saw some stuff in reagrds to those ports, AOL and altavista, let me see if I can find anything
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12758314
Is the hostname something like AC8730C9.ipt.aol.com?
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12758336
From the host names it'd look like it's some AOL service, but a lot of Trojans use those ports, so I’d block it and see what happens, especially since they are not on any of my lists for AOL services like instant messenger.
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12758360
PORT NUMBERS

(last updated 2 December 2004)

http://www.iana.org/assignments/port-numbers
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12758384
Best bet is go to each and every machine you have and run TrendMicros House call
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12758409
Lets assume that you have a trojan, worm running amuck. A quick fix on the phone side might be able to adjust what ports the phones and PBX use to communciate on.
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12758489
The IP address that maps to that name is an AOL address, but I'd block it anyway at the Ethernet interface, as you need to get rid of any extraneous traffic you can get rid of. The real question is what is loading down the processor, do a show processes CPU, and look to see what process is loading it down.

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12758504
Also there  a few other tools avaliable to help disinfecting your systems:

http://www.gfi.com/lannetscan/

http://www.intermute.com/

http://www.spywareinfo.com/~merijn/    Hijackthis is a great tool


0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12758659
How much traffic is going though the routers interfaces, also if you don’t have this in the config, add it “ip cef”.

0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12758742
Also instead of using Colasoft should try using Ethereal, Colasoft might be filtering out traffic from its view.

Also there is another great tool that is good to apply directly onto a machine that you want to check out called AATools:

http://www.glocksoft.com/aatools.htm
0
 

Author Comment

by:tangofniro
ID: 12759645
Thanks Joel
Some of the cisco articles where helpful

I ran some looging on the router for about 5 secs and here is what I got some is cut out due to space of this page
It is constant.  Constantly the switch is looking for the phones.  If I were to put are the logging I had after 5secs in word It would at least be 1 1/2 pages.   I talked to my phone guy today he thought they might be using g711 , I told him I kept seeing 64k packets coming in from the phone.

001876: May 15 05:08:11.429: IP: s=192.168.0.31 (FastEthernet0), d=172.168.1.216
 (Loopback0), g=1.1.1.2, len 80, forward
001877: May 15 05:08:11.433:     UDP src=30000, dst=9000
001878: May 15 05:08:11.433: IP: s=192.168.0.31 (Loopback0), d=172.168.1.216 (Se
rial0), g=199.72.194.233, len 80, forward
001879: May 15 05:08:11.433:     UDP src=30000, dst=9000
001880: May 15 05:08:11.441: IP: s=192.168.0.31 (FastEthernet0), d=172.168.1.108
 (Loopback0), g=1.1.1.2, len 80, forward
001881: May 15 05:08:11.441:     UDP src=30004, dst=9000
001882: May 15 05:08:11.445: IP: s=192.168.0.31 (Loopback0), d=172.168.1.108 (Se
rial0), g=199.72.194.233, len 80, forward
001883: May 15 05:08:11.445:     UDP src=30004, dst=9000
001884: May 15 05:08:11.473: IP: s=192.168.0.31 (FastEthernet0), d=172.168.1.216
 (Loopback0), g=1.1.1.2, len 80, forward
001885: May 15 05:08:11.473:     UDP src=30000, dst=9000
001886: May 15 05:08:11.473: IP: s=192.168.0.31 (Loopback0), d=172.168.1.216 (Se
rial0), g=199.72.194.233, len 80, forward
001887: May 15 05:08:11.473:     UDP src=30000, dst=9000
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12759707
Here is a stupid question, do the phones ever register with the PBX?
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12759788
Also what if there is a mismatch of setings between the phones and phone system?
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12759791
Plus which system are you using? 500 or 7200?
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12759793
Also what series of the phones?
0
 

Author Comment

by:tangofniro
ID: 12759822
yeah I am actually on the phone with the phone guy right now he can see the phones registered
0
 

Author Comment

by:tangofniro
ID: 12759836
500

ITP 5021d
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12759837
okay thats a positve, what about any other applications, OfficeLink, ACD? Can he account for the ports being used?
0
 

Author Comment

by:tangofniro
ID: 12759847
to me it looks as though it is only 2 phones causing the traffic   .108  and .216



I can see my phone at the house register through 1/100th of the time those two try.



Switch is constantly trying to reach those phones.

0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12760072
In order to register few things:

Right ext and password, right ports setup on phone to communicate.

Need the password to log into the programming of the phones
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12760698
One thing that is coming clear to me, that although you have other issues, the router you are using is marginal at best. Even if you do manage to get rid of all the other issues, at best you will be skirting right on the edge of its capabilities. I really think you need to start thinking about upgrading to something like a 2650, or 2811. Which are a lot better suited for doing what you are trying to do.
0
 

Author Comment

by:tangofniro
ID: 12760729
I am seeing that the router is struggling,  but before I purchased these 1721s  I had to have approval from Samsung on them.   Samsung approved them and said they had 1721 in operation everywhere.    In the very near future these router will probably be taken off the public internet and placed in an enviroment where it will just be a point to point t1 between the remote office and the main office and then have another router for the internet.  If you think all of the problems are just due to the fact these routers are underpowered please tell me now.  If you do not think they will work in the next application speak up for my sake..ha  

Is the log above normal?  Only those 2 phones were acting that way.  They crush this router.

Thanks again,
will
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12761007
If it was just a point to point circuit, without extensive access lists, and doing VPN encryption, along with QOS, the load on the routers processor would be significantly less than it currently is. So I’d say in a point to point private circuit properly configured there would be a decent chance they would be ok, but that would add a permanent overhead expense the you could possibly ovoid by having more powerful routers. So buying a more expensive router in the long run could be less expensive.

Now this all leads me back to what I have said already, that before we can say they can work, we need to fix the other issues, but seeing how easily they are overloaded by what shouldn’t be a major issue, i.e., less than a quarter meg of trash from the remote phones, it is clear they are marginal. This is why I am say, hey we may be able to get this to work, but every thing will have to be perfect for that. So although they might be able to be made to work ok, they’d always be susceptible to any issue that loads them down. So yes they are struggling, and I’d say Samsung although they might have 1721’s working ok in other places, they a skating on thin ice.      
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12761034
I think I will sum it up this way, the routers are not the whole issue, but if they were more powerful. The issues you are having would at the least be considerably less pronounced, or maybe not even noticeable.  
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12761123
I will have to agree with the Doc on the 1721 being overloaded easily. Have to remember that the 1721 was not originally deisgined to handle what is has been modifiy to handle. On top of being a self proclaimed convergence technonlogist, I was a General contractor for years, and no matter how good of a finish trim guy you have, the foundation is crooked its crooked so to speak.

You could bump to a bigger and better router or you could also offload the VPN to a dedicated VPN device. Processing the packets for encryption takes time and overhead.

Also remember all it takes is one bad apple to crash an Ethernet network, though less likely these days with switches but we are still resticted by the WAN.
0
 

Author Comment

by:tangofniro
ID: 12761142
very true

When just the VPN, Acess-lists, Nat running, there is really no pressure on the cpu it is low  2-10% but anytime the phone switch is running, maybe through no real fault of it's own, the performance drops.


Do you think by offloading the vpn and leaving everything else the way it is that would help?  I am very skeptical, almost gunshy.   I have been pricing 2650s on ebay  and well they are a hell of a lot cheaper than what I paid new for the 1721s with VPN.  damn....
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12761357
Well one way to test is to remove the VPN completely from the current setup. But you still need to fix the outstanding issues first. You need a baseline to measure from.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12761379
I have never been a fan of the all in one box, ask yourself this, do you still have your swiss army knife? Also applying the access lists to the inbound traffic of the router interface reduces processor overhead. Using a seperate firewall/VPN device is probally your best bet and most cost effective.
0
 
LVL 13

Assisted Solution

by:Dr-IP
Dr-IP earned 400 total points
ID: 12764475
A lot of Cisco’s older router designs like the 1700 series where not designed with a lot the things they added in mind. If you where running just normal data through that 1721, it would probably be fine, but you are running VOIP though it, which generates a very high packet rates because of all the small packets VOIP uses. Which puts a considerably higher load on the router, especially since it has to compare every one of those many little packets to the access lists; QOS them, and then deal with the encryption for the VPN.

Now there is one option for the 1721’s I haven’t mentioned yet as I wasn’t sure it was available, a hardware VPN encryption module, P/N MOD1700-VPN.  That would offload the encryption tasks from the processor freeing it up to do other tasks, but it would be even better to have a 2650 with it’s more powerful process, and a VPN accelerator module.
Also the 2650 even without a VPN accelerator might still out perform a 1721 series router since the processor in it is a lot faster than a 1721, A 80 MHz MPC 860P RISC processor verses a 48 Mhz MPC 860T RISC  processor for the 1721. Also since it’s a P version it has larger catches and other performance enhancing features, so the performance boost is even greater than the MHz improvement would imply.
An even better soulution would be to go with a X8XX series router, which are Cisco’s new generation of routers designed from the ground up for the kinds of task you are trying to get those 1721’s to do. If you had just gone up one step to the 1841 you probably wouldn’t have the problems you are having, since all X8XX series routers have built in encryption coprocessors. It’s what I would have suggested to someone asking for a new 1721, but in knowing what you are doing with it, I would have recommended stepping up to a 2800 series router instead just to be sure. Anyway a more powerful routing solution is probably in order here.  




0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12765096
DR-IP,

Tango already has the VPN module, how did you miss that in the mile long post (LOL).
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12766487
Hey considering how long this post have become, I should get a medal for finding anything. Anyway, so he has hardware encryption and it still is overloaded, I found the config, and checked to see if the VPN module required any special configuration to enable it, but it looks like other than having an IOS that supports it, and it looks like he does. One thing I noted, is he doesn’t have CEF enabled “ip cef”, I have mentioned this already, but I don’t know if he has enabled it, as it does reduce the load on the processor.  
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12767081
In short was does ip cef do?
0
 

Author Comment

by:tangofniro
ID: 12769512
I enabled CEF after the suggestions I just didn't add another config back on this page due to the increasing size. ha


Are we at the point where there is nothing else to look to except the router being overloaded?
0
 

Author Comment

by:tangofniro
ID: 12769641
I know we may be beating a dead horse but I never posted the remote sides config.

Are you two positve that there would be nothing in my total config that would help/hurt ?
0
 
LVL 13

Expert Comment

by:Dr-IP
ID: 12769985
The other side should be a mirror image of the one you posted, other than the IP addresses. So unless something else is different than that, posting the other config would probably pointless.
0
 

Author Comment

by:tangofniro
ID: 12770096
Whoa  whoa my bad


I need help in giving Joel points.

He helped all along and I can't leave him out.


Please help admins.


0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12770152
You will need to put in a request at the support page.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12770160
Were you able resolve the problem with the pbx looking to register the phones?
0
 

Author Comment

by:tangofniro
ID: 12770302
Hey Joel,

The PBX kept seeing the phones being registered.

It doesn't help that this phone vendor is on his first Samsung VOIP.

He doesn't know most answers to my questions.
0
 
LVL 12

Expert Comment

by:Joel_Sisko
ID: 12770364
Hmmm.....delete the phones from the pbx system. Add them back, check to make sure that all the port settings are correct on phone as they are on a known good working phone.
0
 

Author Comment

by:tangofniro
ID: 12770508
I will try that tomorrow
0

Featured Post

NFR key for Veeam Agent for Linux

Veeam is happy to provide a free NFR license for one year.  It allows for the non‑production use and valid for five workstations and two servers. Veeam Agent for Linux is a simple backup tool for your Linux installations, both on‑premises and in the public cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In my office we had 10 Cisco 7940G IP phones that were useless as they were showing PROTOCOL APPLICATION INVALID when started. I searched through Google and worked for a week continuously on those phones, and finally got them working. This is a di…
If your business is like most, chances are you still need to maintain a fax infrastructure for your staff. It’s hard to believe that a communication technology that was thriving in the mid-80s could still be an essential part of your team’s modern I…
This course is ideal for IT System Administrators working with VMware vSphere and its associated products in their company infrastructure. This course teaches you how to install and maintain this virtualization technology to store data, prevent vuln…
We’ve all felt that sense of false security before—locking down external access to a database or component and feeling like we’ve done all we need to do to secure company data. But that feeling is fleeting. Attacks these days can happen in many w…
Suggested Courses

618 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question