Solved

CISCO LLQ trouble??

Posted on 2004-10-29
6,538 Views
Last Modified: 2010-05-18
What would cause me to have drops in my policy?
If I need to post full configs I can.
I have 2 Cisco 1721s connected via t1s to the internet.  I have Samsung ip phones.
After a day of use I have around 70-80 total drops
Here is my what    sh policy-map int gives me.  

Service-policy output: llq

    Class-map: NAPLESVOIP (match-all)
      264 packets, 152592 bytes
      5 minute offered rate 6000 bps, drop rate 1000 bps
      Match: access-group 100
      Queueing
        Strict Priority
        Output Queue: Conversation 264
        Bandwidth 500 (kbps) Burst 12500 (Bytes)
        (pkts matched/bytes matched) 262/151436
        (total drops/bytes drops) 4/2312

    Class-map: DATA (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: access-group 102
      Queueing
        Output Queue: Conversation 265
        Bandwidth 225 (kbps) Max Threshold 64 (packets)
        (pkts matched/bytes matched) 0/0
        (depth/total drops/no-buffer drops) 0/0/0

    Class-map: class-default (match-any)
      21254 packets, 2494063 bytes
      5 minute offered rate 253000 bps, drop rate 2000 bps
      Match: any
      Queueing
        Flow Based Fair Queueing
        Maximum Number of Hashed Queues 256
        (total queued/total drops/no-buffer drops) 0/372/0


0
Question by:tangofniro
    133 Comments
     
    LVL 3

    Expert Comment

    by:cmsJustin
    How many phones?
    How many peak calls at any moment in time?
    0
     
    LVL 3

    Expert Comment

    by:cmsJustin
    And you should post your configs also...
    0
     

    Author Comment

    by:tangofniro
    7 phones on the remote end 25   (3ip 22regular)phones at the main office

    At the most  which would never happen   all 7 phones at the remote side could be calling the main office and talking to 7 phones there.
    We also are going through a vpn


    Here is the config from the remote router


    RPSTRAND#sh run
    Building configuration...

    Current configuration : 4943 bytes
    !
    version 12.3
    service timestamps debug datetime msec
    service timestamps log datetime msec
    service password-encryption
    !
    hostname RPSTRAND
    !
    boot-start-marker
    boot-end-marker
    !
    logging buffered 51200 warnings
    enable password 7 030752180500
    !
    username cisc0 privilege 15 password 7 0505031838435C02
    username cisco password 7 110A1016141D
    no aaa new-model
    ip subnet-zero
    no ip source-route
    !
    !
    ip dhcp excluded-address 192.168.3.1 192.168.3.99
    ip dhcp excluded-address 192.168.3.200 192.168.3.254
    ip dhcp excluded-address 172.168.1.1 172.168.1.100
    ip dhcp excluded-address 172.168.1.200 172.168.1.254
    !
    ip dhcp pool MainScope
       network 192.168.3.0 255.255.255.0
       domain-name rpprop
       dns-server 4.2.2.1
       default-router 192.168.3.1
    !
    ip dhcp pool Strand
       network 172.168.1.0 255.255.255.0
       domain-name rpprop
       dns-server 4.2.2.1
       default-router 172.168.1.1
    !
    !
    ip domain name yourdomain.com
    no ip bootp server
    no ip cef
    ip audit notify log
    ip audit po max-events 100
    no ftp-server write-enable
    no scripting tcl init
    no scripting tcl encdir
    !
    !
    !
    !
    !
    crypto isakmp policy 10
     hash md5
     authentication pre-share
    crypto isakmp key 1234 address 199.x.x.x
    crypto isakmp keepalive 20 10
    !
    !
    crypto ipsec transform-set rpvpn esp-des esp-md5-hmac
    !
    crypto map rpvpn 10 ipsec-isakmp
     set peer 199.x.x.x
     set peer 216.x.x.x
     set transform-set rpvpn
     match address 101
    !
    !
    !
    class-map match-all NAPLESVOIP
     match access-group 100
    class-map match-all DATA
     match access-group 102
    !
    !
    policy-map llq
     class NAPLESVOIP
      priority 500
     class DATA
      bandwidth 225
     class class-default
      fair-queue
    !
    !
    !
    interface FastEthernet0
     description $FW_INSIDE$$ETH-LAN$
     ip address 172.168.1.1 255.255.255.0
     no ip redirects
     no ip unreachables
     no ip proxy-arp
     ip nat inside
     ip route-cache flow
     ip tcp adjust-mss 1380
     speed auto
     full-duplex
     no cdp enable
    !
    interface FastEthernet1
     no ip address
     no cdp enable
    !
    interface FastEthernet2
     no ip address
     shutdown
     no cdp enable
    !
    interface FastEthernet3
     no ip address
     shutdown
     no cdp enable
    !
    interface FastEthernet4
     no ip address
     shutdown
     no cdp enable
    !
    interface Serial0
     description $FW_OUTSIDE$
     ip address 199.x.x.x 255.x.x.x
     no ip redirects
     no ip unreachables
     no ip proxy-arp
     ip nat outside
     service-policy output llq
     encapsulation ppp
     ip route-cache flow
     service-module t1 timeslots 8-24
     no cdp enable
     crypto map rpvpn
    !
    interface Vlan1
     no ip address
    !
    ip nat inside source route-map nonat interface Serial0 overload

    ip classless
    ip route 0.0.0.0 0.0.0.0 199.x.x.x
    ip http server
    ip http authentication local
    ip http secure-server
    !
    !
    !
    access-list 100 permit udp any any range 16384 32000
    access-list 100 permit tcp any any eq 1719
    access-list 100 permit tcp any any eq 1720
    access-list 100 permit tcp any any eq 6100
    access-list 100 permit tcp any any range 1024 4999
    access-list 100 permit udp any any eq 6000
    access-list 100 permit udp any any range 1024 4999
    access-list 100 permit udp any any eq 5060
    access-list 100 permit udp any any range 30000 30030
    access-list 100 permit udp any any range 9000 9001
    access-list 101 permit ip 172.168.1.0 0.0.0.255 192.168.0.0 0.0.0.255
    access-list 101 deny   ip 172.168.1.0 0.0.0.255 any
    access-list 102 permit tcp any any range 5000 5110
    access-list 110 deny   ip 172.168.1.0 0.0.0.255 192.168.0.0 0.0.0.255
    access-list 110 permit ip 172.168.1.0 0.0.0.255 any
    no cdp run
    !
    route-map nonat permit 10
     match ip address 110
     set ip next-hop 1.1.1.2
    !
    !
    control-plane
    !
    banner login ^C
    0
     
    LVL 5

    Expert Comment

    by:snowsurfer
    The VPN adds overhead.  You should turn on CAC to limit the number of calls that can be made at one time.  What is your bandwidth?  The problem with going over a VPN is there is no QOS on the internet.  Even though you are using a vpn connection you are still using the internet.
    0
     

    Author Comment

    by:tangofniro
    SO what you are saying is that really no matter what I am going to get drops on my policies due to the fact i'm going over the internet?  
    0
     
    LVL 5

    Expert Comment

    by:snowsurfer
    I am afraid so.  You might be able to fine tune it a little bit, but VoIP is not meant for the internet
    0
     
    LVL 12

    Accepted Solution

    by:
    Well I disagree on the VoIP not meant for the Internet. Quickly look up the TLA for VoIP (Voice over Internet Protocol).

    First before you give up on implementing a VoIP solution. What are ping times between the two 1700's? If you are under <100ms between sites, VoIP has great potential. The human ear detects anything over 180ms as starting to be choppy and distorted.

    I will agree on the VPN adds overhead and could pose a problem, but I have implemented many VoIP solutions using a VPN between sites. But we usually use a separate device for the VPN solution. Part of your problem is that your little Cisco router is being asked to do to much, Route, IPSEC, Firewall. The 1720 was not built for all of this at the same time. You might want to go to the 1760 series router and use the VPN module to offload some of the work.

    But in any case, if your ping times are good then using VoIP is reasonable.

    Also dropped packets also could indicate the router is being overloaded. Also you want to set up some form of QOS on your local network; this might help reduce delay times. One question, is the T1 a point-to-point? If not point to point, are both T1's terminated to the same ISP/Carrier network? If the drop packets are only the voice data packets, then you could have a application issue or even the Samsung phone causing the problem.

    As a ping test I would recommend to ping one computer at site A to another computer over T1's at site B. To ping, go to a desktop on the same LAN: (make sure your computer/router/firewall is not disabled to respond to a ping request)

    Start > Run> type cmd> at the prompt type            ping -t -l 64000 X.X.X.X   (make sure that you are pinging the right server address X.X.X.X)

    Let the command run. The command sends 64K byte packets until you hit Cntrl C. The larger packet tests the Ethernet connection a little bit better than the standard ping command of 5 bytes. Let it run for a bit and see what is going on.  Also, check the routers interface's for CRC errors, if you are getting a high amount you could have connectivity problems.

    Run the following at the cisco CLI to look at the crc errors:

    show interfaces ethernet
    show interfaces serial

    Let me know how the tests go and we can move on from that point.
    0
     

    Author Comment

    by:tangofniro
    Okay,

    No CRCs on main site interfaces

    remote side interfaces which I have the configs above are , Fast0  0 CRCs,   Serial0  68 CRCs, router was rebooted today.

    ping times between the sites show mostly 30ms but every once in a while it will jump up very high around 200ms then it will go back down.

    These T1s are point to point,  to the internet not leased line,  same carrier,  they say it my router config, I am trying to find out if it is.  Is there any command to show my router being overworked?  

    I could post the main site config if that would help.

    The phones work great for about 90% of the time  but for the other 10% they are unusable.  There is no specific time or day either.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Post the config so we dont go DOHHH! later on. The CRC's on the serial are not unusual is that was the amount for the entire day? What you could do is reset the crc's count to zero and make a boat load of VoIP calls (try to do this when the office is quite, try to reduce the amount of Internet browsing traffic), see what the crc count is, then clear the counters, do not make any VoIP calls for a while, check crc count, if the same then most likely not the issue.

    What version of IOS are you using and yes there are commands for performance monitoring.



    0
     

    Author Comment

    by:tangofniro
    here is the main sites config, where thephone system resides.

    Thanks for your help.

    no aaa new-model
    ip subnet-zero
    no ip source-route
    !
    !
    ip dhcp excluded-address 192.168.0.1 192.168.0.99
    ip dhcp excluded-address 192.168.0.200 192.168.0.254
    !
    ip dhcp pool MainScope
       network 192.168.0.0 255.255.255.0
       domain-name
       dns-server 4.2.2.1
       default-router 192.168.0.1
    !
    !
    ip domain name rp-prop.com
    ip name-server 4.2.2.1
    ip name-server 4.2.2.2
    no ip bootp server
    no ip cef
    ip audit notify log
    ip audit po max-events 100
    ip ssh time-out 60
    ip ssh authentication-retries 2
    no ftp-server write-enable
    no scripting tcl init
    no scripting tcl encdir
    !
    !
    !
    !
    !
    crypto isakmp policy 10
     hash md5
     authentication pre-share
    crypto isakmp key 1234 address 1x.x.x.x
    crypto isakmp keepalive 20 10
    !
    !
    crypto ipsec transform-set rpvpn esp-des esp-md5-hmac
    !
    crypto map rpvpn 10 ipsec-isakmp
     set peer 1x.x.x.x
     set transform-set rpvpn
     match address 101
    !
    !
    !
    class-map match-all STRANDDATA
     match access-group 102
    class-map match-all STRANDVOIP
     match access-group 100
    !
    !
    policy-map LLQ
     class STRANDVOIP
      priority 450
     class STRANDDATA
      bandwidth 650
     class class-default
      fair-queue
    !
    !
    !
    interface Loopback0
     ip address 1.1.1.1 255.255.255.0
    !
    interface FastEthernet0
     description $FW_INSIDE$$ETH-LAN$
     ip address 192.168.0.1 255.255.255.0
     no ip redirects
     no ip unreachables
     no ip proxy-arp
     ip nat inside
     no ip route-cache
     ip tcp adjust-mss 1380
     ip policy route-map nonat1
     no ip mroute-cache
     speed auto
     full-duplex
     priority-group 5
     no cdp enable
    !
    interface FastEthernet1
     switchport mode trunk
     no ip address
     no cdp enable
    !
    interface FastEthernet2
     no ip address
     shutdown
     no cdp enable
    !
    interface FastEthernet3
     no ip address
     shutdown
     no cdp enable
    !
    interface FastEthernet4
     no ip address
     shutdown
     no cdp enable
    !
    interface Serial0
     description USLEC T1
     ip address 1x.x.x.x  255.255.255.x
     no ip redirects
     no ip unreachables
     no ip proxy-arp
     ip nat outside
     service-policy output LLQ
     encapsulation ppp
     no ip route-cache
     no ip mroute-cache
     service-module t1 timeslots 1-24
     no cdp enable
     crypto map rpvpn
    !
    interface Vlan1
     no ip address
     no ip redirects
     no ip unreachables
     no ip proxy-arp
     ip route-cache flow
    !
    ip nat inside source route-map nonat interface Serial0 overload
    ip nat inside source static 192.168.0.30 2x.x.x.x extendable
    ip nat inside source static 192.168.0.31 2x.x.x.x extendable
    ip nat inside source static 192.168.0.205 2x.x.x.x extendable
    ip nat inside source static 192.168.0.2 2x.x.x.x extendable
    ip classless
    ip route 0.0.0.0 0.0.0.0 199.x.x.x
    no ip http server
    ip http authentication local
    ip http secure-server
    !
    !
    !
    logging trap debugging
    access-list 100 permit udp any any range 16384 32000
    access-list 100 permit tcp any any eq 1719
    access-list 100 permit tcp any any eq 1720
    access-list 100 permit tcp any any eq 6100
    access-list 100 permit tcp any any range 1024 4999
    access-list 100 permit udp any any eq 6000
    access-list 100 permit udp any any range 1024 4999
    access-list 100 permit udp any any eq 5060
    access-list 100 permit udp any any range 30000 30030
    access-list 100 permit udp any any range 9000 9001
    access-list 101 permit ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255
    access-list 101 deny   ip 192.168.0.0 0.0.0.255 any
    access-list 102 permit tcp any any range 5000 5110
    access-list 105 deny   icmp any any echo
    access-list 105 permit ip any any
    access-list 110 deny   ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255
    access-list 110 deny   ip host 192.168.0.205 any
    access-list 110 deny   ip host 192.168.0.30 any
    access-list 110 deny   ip host 192.168.0.31 any
    access-list 110 deny   ip host 192.168.0.2 any
    access-list 110 permit ip 192.168.0.0 0.0.0.255 any
    access-list 120 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255
    access-list 123 permit ip host 192.168.0.205 172.168.1.0 0.0.0.255
    access-list 123 permit ip host 192.168.0.30 172.168.1.0 0.0.0.255
    access-list 123 permit ip host 192.168.0.31 172.168.1.0 0.0.0.255
    access-list 123 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255
    no cdp run
    !
    route-map nonat1 permit 10
     match ip address 123
     set ip next-hop 1.1.1.2
    !
    route-map nonat permit 10
     match ip address 110
    !
    !
    control-plane
    !
    banner login ^CCAuthorized access only!
     Disconnect IMMEDIATELY if you are not an authorized user!^C
    !
    line con 0
     password 7 0833184A1A18541B
     login
     transport output telnet
    line aux 0
     login local
     transport output telnet
    line vty 0 4
     privilege level 15
     password 7 06145B255F4F5815
     login
     transport input telnet ssh
    line vty 5 15
     privilege level 15
     login local
     transport input telnet ssh
    !
    scheduler allocate 4000 1000
    scheduler interval 500
    !
    end

    RPNAPLES#
    0
     

    Author Comment

    by:tangofniro
    IOS version is 12.3
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    please run from Cisco CLI:

    show version

    post the results
    0
     

    Author Comment

    by:tangofniro
    RPNAPLES#sh version
    Cisco IOS Software, C1700 Software (C1700-K9O3SY7-M), Version 12.3(2)XE, EARLY D
    EPLOYMENT RELEASE SOFTWARE (fc1)
    Synched to technology version 12.3(3.5)T
    TAC Support: http://www.cisco.com/tac
    Copyright (c) 1986-2003 by Cisco Systems, Inc.
    Compiled Tue 18-Nov-03 23:26 by ealyon

    ROM: System Bootstrap, Version 12.2(7r)XM2, RELEASE SOFTWARE (fc1)
    ROM:

    RPNAPLES uptime is 1 week, 13 hours, 45 minutes
    System returned to ROM by power-on
    System restarted at 08:58:52 America/Chicago Mon Apr 8 2002
    System image file is "flash:c1700-k9o3sy7-mz.123-2.XE.bin"


    This product contains cryptographic features and is subject to United
    States and local country laws governing import, export, transfer and
    use. Delivery of Cisco cryptographic products does not imply
    third-party authority to import, export, distribute or use encryption.
    Importers, exporters, distributors and users are responsible for
    compliance with U.S. and local country laws. By using this product you
    agree to comply with applicable laws and regulations. If you are unable
    to comply with U.S. and local laws, return this product immediately.

    A summary of U.S. laws governing Cisco cryptographic products may be found at:
    http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

    If you require further assistance please contact us by sending email to
    export@cisco.com.

    Cisco 1721 (MPC860P) processor (revision 0x300) with 84983K/13321K bytes of memo
    ry.
    Processor board ID FOC08172HWP (3205001181), with hardware revision 0000
    MPC860P processor: part number 5, mask 2
    1 Ethernet interface
    5 FastEthernet interfaces
    1 Serial interface
    1 Virtual Private Network (VPN) Module
    WIC T1-DSU
    32K bytes of NVRAM.
    32768K bytes of processor board System flash (Read/Write)

    Configuration register is 0x2102




    \from remote site\\





    RPSTRAND#sh version
    Cisco IOS Software, C1700 Software (C1700-K9O3SY7-M), Version 12.3(2)XE, EARLY D
    EPLOYMENT RELEASE SOFTWARE (fc1)
    Synched to technology version 12.3(3.5)T
    TAC Support: http://www.cisco.com/tac
    Copyright (c) 1986-2003 by Cisco Systems, Inc.
    Compiled Tue 18-Nov-03 23:26 by ealyon

    ROM: System Bootstrap, Version 12.2(7r)XM2, RELEASE SOFTWARE (fc1)
    ROM:

    RPSTRAND uptime is 5 hours, 31 minutes
    System returned to ROM by power-on
    System image file is "flash:c1700-k9o3sy7-mz.123-2.XE.bin"


    This product contains cryptographic features and is subject to United
    States and local country laws governing import, export, transfer and
    use. Delivery of Cisco cryptographic products does not imply
    third-party authority to import, export, distribute or use encryption.
    Importers, exporters, distributors and users are responsible for
    compliance with U.S. and local country laws. By using this product you
    agree to comply with applicable laws and regulations. If you are unable
    to comply with U.S. and local laws, return this product immediately.

    A summary of U.S. laws governing Cisco cryptographic products may be found at:
    http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

    If you require further assistance please contact us by sending email to
    export@cisco.com.

    Cisco 1721 (MPC860P) processor (revision 0x300) with 84983K/13321K bytes of memo
    ry.
    Processor board ID FOC08172J3K (2799143465), with hardware revision 0000
    MPC860P processor: part number 5, mask 2
    1 Ethernet interface
    5 FastEthernet interfaces
    1 Serial interface
    1 Virtual Private Network (VPN) Module
    WIC T1-DSU
    32K bytes of NVRAM.
    32768K bytes of processor board System flash (Read/Write)

    Configuration register is 0x2102


    0
     

    Author Comment

    by:tangofniro
    In the meantime I added  QOS pre-classify on my crypto-map.

    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Nothing jumping out, but I do have one question, where are the T1 obtaining their clocking from?
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    tangofniro

    How are things going?
    0
     

    Author Comment

    by:tangofniro
    quality yesterday and today was pretty subpar. A lot of static.
    clocking coming from my csu/dsu internally.

    Nothing really jumping out at you either?
    What did you think about the qos on the cyrpto-map?  Cisco pointed me in that direction.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Qos on the Crypto will only help streamline things, so its a good thing.

    Question on the drops, are you actually dropping calls mid way thru the conversations?

    In terms of static, could you clarify? Voice signals having static? High CRC's?

    The good news it will get thru this eventually, Someone asked Thomas Edison that why he would try to keep inventing something even if it took 50,000 tries? His response was that he eleiminated 50,000 ways of not doing it.

    Just a thought if the calls are being dropped during mid conversation, we might be running into a timeout issue of the application over the VPN, problem might be outside of the network. Do you have any tech specs on the Samsung IP Phones?
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    How are things coming along?
    0
     

    Author Comment

    by:tangofniro
    hit or miss.  Friday was bad but I think today was better.  

    Calls don't drop out completly, Static is the main problem , can't hear the other end, and speakerphone useless.

    I do have tech specs on the phone but it's on cd.  PDF files  I can send them to you if you want but there is a lot of junk to wade through.

    Thanks you for all of your help and staying on this problem.



    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    On the pdf, if you know of a FTP site and load them up there? Will galdly review them.

    If you are actually getting static there are a few things we need to look at:

    First if VAD is being used and also echo-cancellation. VAD is more than likely being introdiced by the phone or Samsung system. Need to find that and turn it off.

    Also, just another thing popped into my head, we can test the link betwen the sites using, lets say two computers running a free softphone, such sjlabs or x-ten lite, and that have speakers and microphones. If you establish a call between the two (using same VoIP protocols) and no hissing or drop offs, we can lean more towards the phone side of things.

    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    tangofniro

    Any headway?
    0
     

    Author Comment

    by:tangofniro
    Hey Joel,

    Sorry for the delay.  I was out of town for a few days.   It seems good somedays and others times/days  it is not great.  I don't know if I can attrib it too the network or the internet.


    Thanks  for your continued help,
    will
    0
     

    Author Comment

    by:tangofniro
    One more thing I can ping everything in my network under 10ms  but if I ping my routers internal interface it is jumpy  45ms -68ms  11ms.  Why would it do that?  Is the QOS making that happen?
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    It could be, it would jump if there is other traffic at the same time with a higher priority. If it happens all the time (being jumpy) it might be a configuration issue? You can checkto see this using the ping command from earlier on in our dialog.

    Few things to follow up on:

    Are you still getting static on voice calls?
    Do you have a spare computer with a 10/100 NIC and a hub?
    0
     

    Author Comment

    by:tangofniro
    After everyone  left I ran the ping from the servers to the router int and the ping time seems pretty stable  all under 10
    for the most part.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Okay lets assume that is okay, are you still getting static on the lines?
    0
     

    Author Comment

    by:tangofniro
    Some days(not all day) it sounds like a bad cell phone others it works fine, really no drops in calls.  I am wondering if it just might be the internet ????
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    It could be here a re few steps to find the ghost:

    1. Try to disable the use of VAD and echo-cancellation on the VoIP side of things
    2. To monitor the Internet , there is a program I have used in the past that helps to do this it can be found at:  

    http://www.serverscheck.com/

    I believe you can set up to 3 free monitors for each system setup. Basically I would set up a ping command and monitor that. Download the program and take a look, it has a alot of features.

    3. Setup a network sniffer at each site on each interface. You could use Ethereal or some other free products.

    The ideal situation would be two computers, one on each site, each connected to between the network and router via a hub (hub allows to see all packets) then set up Ethereal and serverscheck on each. Also you could enable logging on the Cisco routers to log to the same system running the sniffer and monitor. Let it run for a few days, log times when all is running well, then log times when things are bad. Go back to the data captured and look up logged times, see if something is out of the ordinary.

    Kindest regards

    Joel_Sisko
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Also, part of th problem could be to much jitter, using Ehtereal you should be able to determine this, might not be real clear cut. There are some products that allow for this one is from Wildpackets called Etherpeek VX, another is Observer 10, but they cost a few hundred dollars.

    Just rememebered try using this site:

    http://www.testyourvoip.com/


    0
     

    Author Comment

    by:tangofniro
    tried test your voip,

    I recieved a MOS 4.3
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    4.3 is good, what type of traffic was going across the network at that time? Need to run this test a variosu times and see if it changes, also run the test when you know the problem you have been having arises.

    Is it possible to set up a sniffer and log files from the Cisco router? Since we are looking for somewhat of the needle in the haystack we need to baseline the network and move forward from that point.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Additional free program that may help out:

    http://www.bb4.org/features.html
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    One more I just rememred that is affordable, gui based and quite a few features:

    http://www.colasoft.com/products/capsa/

    0
     

    Author Comment

    by:tangofniro
    Okay, I had the problem today.

    Voice quality went out the window.  I ran some speed tests and from the various places I ran it from they all showed very slow speed like 300-400k.
    I spoke with the provider and they said they did not see anything wrong at the time.  It lasted for about 25minutes and they gone.  
    Is there anything I can do to check what is going on?  Can you think of anything that might cause that?

    Kind of stuck...
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    I am assuming the speed went back up after the 25 minutes as well as the voice quality? Did you happen to check with the www.testyourvoip.com tool? Though your speed/ping might be okay the packet jitter might be going off the charts. Jitter is a result of packets showing up at different times to put it simply.

    Voice is very time sensitive, thought also just popped into my me head, what do you have your MTU set at on your routers?  Setting the MTU higher might help, next time you are having the issue try adjusting the MTU.

    Also you really need to baseline the system as per previous posts, also use a traceroute tool, see how many hops it takes and which routers the packet touches during normal operations, then do this again when the speed goes down. It could be also that soem traffic is being redirected thru a different route for a given time period for maintence and such.

    Kindest regards,

    Joel _Sisko
    0
     

    Author Comment

    by:tangofniro
    Hello Joel,
    I checked with testyourvoip as it was happening and I got a terrible score of 2.2.  There was no jitter going out but coming in from Boston there was.

    I will try a few of your suggestions from earlier.  Is there a command that will show me every active connection/ports on the router.

    The mtu is set to 1600 on my serial int.

    Going out to the internet while this was happening I was getting a good amount of packet loss to my providers gateway. And time was very high.  
    After the 25 or so minutes the speed was back along with voice quality.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Okay at least we can start ruling out ithe nternal stuff and focus on the carrier.

    Need to baseline the speed from site a to site b, need to baseline the routes it also takes.

    If it is actually the carrier, then monitoring the Cisco interface will really not help, other than let us know something is going wrong. We need to find out where in the carrier network things are going wrong. By the way who is your provider?
    0
     

    Author Comment

    by:tangofniro
    The route is only 3 hops

    Had alot of trouble today(static, packet loss). I don't know my provider USLEC says they are not seeing any problems on their side.  The do see a lott of usage on our side.  I can't see what is causing the problem.  Testyourvoip was pretty low also.  Great thing was at 5 oclock the problem went away.  Everything now is great.

    I don't know if I would rule out it being an internal problem.  That would be the way I am leaning right now.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Can you set up a system to monitor the traffic going to the router? You really need to baseline the system, without it this post will be a mile long (LOL), can use the points but not that bad (LOL).

    Are you sharing files across the link? Any kind of Database upadtes? Someone backing up to network drive? Are you using Active Directory? What about email? Roaming profiles?
    0
     

    Author Comment

    by:tangofniro
    files shared across link-- yes
    Mapped Drives-- yes
    Active Directory--yes
    Email-- Yes not in house though, No Exchange
    Roaming profiles--- NO


    Will try to start a baseline before Thanksgiving.

    Thanks Joel

    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    I just wonder if there is an application that is performing some kind of update, backup, or maintence, it would explain why things go nuts for abit then die down.

    Quick question, for Internet usage is everyone (both sites) going over the same link also?

    How do you implement virus updates?
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    http://www.colasoft.com/products/capsa/

    This will be the best tool to baseline the system.
    0
     

    Author Comment

    by:tangofniro
    i downloaded capsa today, nothing really jumped out at me. will buy it most likely.

    i have each side going out over the internet on each link.  Unless you can see something different in my config.

    each workstation has it's side router as the default gateway.

    I have 2 server 1 domain the other a terminal server/backup domain controller

    I use Symantec Antivirus Server.

    Anything I should be looking for in particular that you are expecting  Cola to see?

    thanks
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Looking for the needle!

    In reagrds to the Antivirus, everyone is updating from the local server at main site? How often and when? Are all users doing it at the same time?

    What about Microsoft updates? Doing this auto also?

    Looking for an application that is bursty in nature.

    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Also dropping your MTU size might help the VoIP weather the storms so to speak. But do not go below 400, it will have an adverse affect on the control signaling for VoIP. Drop it to 900, this is more of band aid rather than a cure, but should help.

    0
     

    Author Comment

    by:tangofniro
    Could my router be overloaded?  Causing the slowdown?
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Always possible, need to log the cpu performance, remember early on I said that the router is being asked to do alot (think my first post). But since your provider is seeing a considerable amount of traffic which you can not account for, you should still baseline the network to see what is causing the traffic.

    Dropping the MTU size will actually add to processor overhead but will enable the packet to transverse the network/Internet more efficiently.
    0
     

    Author Comment

    by:tangofniro
    I ran sh proc and sh proc cpu and here is what I got:

    RPNAPLES#sh proc
    CPU utilization for five seconds: 32%/30%; one minute: 30%; five minutes: 29%
     PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
       1 Cwe 8029A2D0           32          8    4000 5548/6000   0 Chunk Manager
       2 Csp 80280AB0           28       2245      12 2744/3000   0 Load Meter
       3 M*         0           28         19    147310360/12000  6 Virtual Exec
       4 Mwe 8053C650            0         94       0 5768/6000   0 DHCPD Timer
       5 Lst 80299ACC        11444       1346    8502 5752/6000   0 Check heaps
       6 Cwe 8029E304           32         39     820 5536/6000   0 Pool Manager
       7 Mst 801A58E4            0          2       0 5708/6000   0 Timers
       8 Mwe 80096BB8            0          2       0 5716/6000   0 Serial Backgroun
       9 Mwe 80178B38            0          2       0 5704/6000   0 AAA high-capacit
      10 Lwe 8034FD08          948       1278     741 5008/6000   0 ARP Input
      11 Mwe 80371168            0          3       0 5708/6000   0 DDR Timers
      12 Mwe 80675650           32       2805      11 5768/6000   0 HC Counter Timer
      13 Lwe 80978F34           12          2    6000 5624/6000   0 Entity MIB API
      14 Mwe 80BE83B8            0          2       0 5712/6000   0 ATM Idle Timer
      15 Mwe 8009C57C            0          1       0 5788/6000   0 SERIAL A'detect
      16 Msp 801E85FC           12      11208       1 5744/6000   0 GraphIt
      17 Mwe 8038C274            0          2       011712/12000  0 Dialer event
      18 Mwe 806196F8            0          2       011728/12000  0 XML Proxy Client
      19 Cwe 8026C730            0          1       0 5804/6000   0 Critical Bkgnd
      20 Mwe 8021AC98          296       3265      9010392/12000  0 Net Background
      21 Lwe 80196DF8            0         19       011356/12000  0 Logger
      22 Mwe 801BB920           40      11205       3 5736/6000   0 TTY Background
      23 Msp 8022BC78          132      11217      11 7464/9000   0 Per-Second Jobs
      24 Lwe 80311EC0            8          2    4000 1864/3000   0 IPM_C1700_CLOCK
      25 Mwe 812F69B4            0          2       0 5784/6000   0 AggMgr Process
      26 Hwe 81327A00            4          2    2000 5264/6000   0 ESWPPM
      27 Hwe 8022B7A4            0          1       0 5788/6000   0 Net Input
      28 Msp 80220A90           84       2249      37 5740/6000   0 Compute load avg
      29 Msp 8022BCEC         8352        192   43500 5752/6000   0 Per-minute Jobs
      30 Mwe 8009F05C            0          3       0 5700/6000   0 Service-module a
      31 Mwe 813322AC            0          1       0 5792/6000   0 Switch Link Moni
      32 Mwe 8025067C            4          2    2000 5712/6000   0 AAA Server
      33 Mwe 80252B48            0          1       0 5784/6000   0 AAA ACCT Proc
      34 Mwe 80252C2C            0          1       0 5772/6000   0 ACCT Periodic Pr
      35 Lwe 8017D6A4            0          1       0 5788/6000   0 AAA_SERVER_DEADT
      36 Mwe 80304C34            0          2       0 5724/6000   0 AAA Dictionary R
      37 Mwe 8044EEF4      1653100    1349941    1224 9952/12000  0 IP Input
      38 Mwe 80473998            0          1       0 5768/6000   0 ICMP event handl
      39 Mwe 81159038            0          8       0 5788/6000   0 CRYPTO IKMP IPC
      40 Mwe 8085DB38            0          1       011788/12000  0 SSS Manager
      41 Mwe 80861464            4       1501       211764/12000  0 SSS Test Client
      42 Mwe 8086AE94            0          1       0 5800/6000   0 SSS Feature Mana
      43 Mwe 8086AF28          780      43912      17 5768/6000   0 SSS Feature Time
      44 Mwe 8027AE9C            0          4       011732/12000  0 PPP Hooks
      45 Lwe 80A546B4            0          1       0 5396/6000   0 X.25 Encaps Mana
      46 Mwe 80DFED4C            0          1       011748/12000  0 VPDN call manage
      47 Mwe 80E7AB9C            0          1       011780/12000  0 L2X Data Daemon
      48 Mwe 80E42F6C            0          1       011756/12000  0 L2X Socket proce
      49 Mwe 80E0FD10            0          1       011784/12000  0 L2X SSS manager
      50 Mwe 80E1CB74            0          2       011712/12000  0 L2TP mgmt daemon
      51 Mwe 8105F430            0          1       0 5776/6000   0 AC Mgr
      52 Mwe 81239BE4            0          2       011728/12000  0 KRB5 AAA
      53 Mwe 812E6C64            0          2       0 5540/6000   0 DTP Protocol
      54 Mwe 8027AE9C            4          3    133310988/12000  0 PPP IP Route
      55 Mwe 8027AE9C            0          4       011448/12000  0 PPP IPCP
      56 Mwe 80425860           32         19    1684 4436/6000   0 DHCPD Receive
      57 Mwe 80551360          524        215    2437 8224/9000   0 IP Background
      58 Mwe 805574F4          112        194     577 8736/9000   0 IP RIB Update
      59 Mst 8043214C            0         36       011044/12000  0 TCP Timer
      60 Lwe 80437114            8          3    266610940/12000  0 TCP Protocols
      61 Mwe 8049E14C            0          1       0 5804/6000   0 RARP Input
      62 Hwe 80533708            0          1       0 5776/6000   0 Socket Timers
      63 Mwe 805090E4            4         40     100 8452/9000   0 HTTP CORE
      64 Lsi 805CB69C           48        188     255 5304/6000   0 IP Cache Ager
      65 Hwe 80A6DB7C            0          1       0 5780/6000   0 PAD InCall
      66 Mwe 80A27320            0          2       011708/12000  0 X.25 Background
      67 Mwe 8086CC34            0          2       0 5708/6000   0 PPP SSS
      68 Mwe 80908750           16        189      84 8688/9000   0 Adj Manager
      69 Mwe 80AA1614           72      21938       3 5700/6000   0 IP NAT Ager
      70 Mwe 8027AE9C            4          2    2000 5728/6000   0 PPP Bind
      71 Mwe 8096E9F4            0          1       0 5756/6000   0 SNMP Timers
      72 Mwe 80C7B920            0          1       0 5796/6000   0 Inspect Timer
      73 Msi 8054B8C4            0       3196       0 4980/6000   0 DHCPD Database
      74 Mwe 80CA5BF8            0          2       0 5612/6000   0 URL filter proc
      75 Mwe 80CBF034            0         38       0 5796/6000   0 Authentication P
      76 Mwe 80CC8EA0            0          1       0 5776/6000   0 Auth-proxy AAA B
      77 Mwe 80CC9C64            0          1       0 5796/6000   0 IDS Timer
      78 Mwe 80DE114C            0          1       023780/24000  0 COPS
      79 Mwe 80DEFDF0            0          2       0 5720/6000   0 Dialer Forwarder
      80 Mwe 80E06E3C            8       2249       311776/12000  0 L2F management d
      81 Mwe 80E357FC            4          1    400011540/12000  0 PPTP Mgmt
      82 Mwe 80E7F4DC            0          2       011728/12000  0 PPTP Data
      83 Hwe 810EDD9C        25680      30471     842 5064/6000   0 Crypto HW Proc
      84 Lwe 812DB7A4            0          1       0 5796/6000   0 XSM_EVENT_ENGINE
      85 Lsi 812D93DC            8       1124       711824/12000  0 XSM_ENQUEUER
      86 Lsi 812DC4D4            4       1124       311828/12000  0 XSM Historian
      87 Mwe 8025D51C            0          2       0 5728/6000   0 LOCAL AAA
      88 Mwe 8025F5D4            0          2       0 5728/6000   0 ENABLE AAA
      89 Mwe 8025F9C0            0          2       0 5732/6000   0 LINE AAA
      90 Mwe 803C8D1C            0          2       0 5604/6000   0 TPLUS
      91 Lwe 808F6E08          360      14373      25 4368/6000   0 CEF process
      92 Mwe 810DB840            4          2    2000 5724/6000   0 Crypto Support
      93 Mwe 81387464            0          1       0 5800/6000   0 EM Background Pr
      94 Mwe 806D6DA0           20        452      44 5668/6000   0 CRM_CALL_UPDATE_
      95 Mwe 810D5BB4            0          1       011804/12000  0 Encrypt Proc
      96 Mwe 810D67A4        14128        117  120752 6696/8000   0 Key Proc
      97 Mwe 811B0058           24          4    6000 7000/8000   0 Crypto CA
      98 Mwe 811EB1D4            0          1       0 7812/8000   0 Crypto SSL
      99 Mwe 81158B3C           20         40     50020432/24000  0 Crypto ACL
     100 Mwe 810E00DC            0          1       0 5792/6000   0 CRYPTO QoS proce
     101 Mwe 81151F4C           12         14     85711332/12000  0 Crypto Delete Ma
     102 Mwe 8111F364          104        109     954 6180/12000  0 Crypto IKMP
     103 Mwe 8111445C         1212        587    2064 9744/12000  0 IPSEC key engine
     104 Mwe 81114F10            0          1       0 5716/6000   0 IPSEC manual key
     105 Mwe 80171B4C            0          2       0 5708/6000   0 AAA SEND STOP EV
     106 Mwe 80849E8C            0          1       0 5816/6000   0 Syslog Traps
     107 Lwe 812C945C            0          2       0 5648/6000   0 IpSecMibTopN
     108 Mwe 81324B6C           32       1216      26 5752/6000   0 PM Callback
     109 Mwe 80AF1508        38892    2766014      14 5584/6000   0 SAA Event Proces
     110 Mwe 80E45B78            0          1       0 5784/6000   0 VPDN Scal
     111 Mwe 81337730            4          2    2000 3912/6000   0 VLAN Manager
     112 Lsp 8132E680        18864      44308     425 5732/6000   0 COLLECT STAT COU
     113 Mwe 805D0D8C            0          1       011796/12000  0 TCP Driver
     114 Lwe 80435D60            0          1       0 5792/6000   0 TCP Listener
     115 Mwe 80AD3A54            0          1       0 5772/6000   0 IP NAT WLAN
     116 Mwe 81244984           40        169     236 4428/6000   0 SSH Event handle
     117 Mwe 8099DAD0         2664     349651       711696/12000  0 PPP manager
     118 Mwe 8027AE9C         2508     349661       711244/12000  0 PPP Events
     119 Hwe 809D5CA8           12      11239       1 5724/6000   0 Multilink PPP
     120 Mwe 809D568C            0          2       0 5712/6000   0 Multilink event
     121 Mwe 80876D7C            0          2       0 5732/6000   0 IP Flow Backgrou
     122 Mwe 810FBD70           48        249     192 4456/6000   0 Crypto Hardware
     123 Lwe 808F6638           16        367      43 5664/6000   0 CEF Scanner



    RPNAPLES#sh proc cpu
    CPU utilization for five seconds: 41%/39%; one minute: 34%; five minutes: 30%
     PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
       1          32         8       4000  0.00%  0.00%  0.00%   0 Chunk Manager
       2          28      2252         12  0.00%  0.00%  0.00%   0 Load Meter
       3         596       286       2083  1.19%  0.54%  0.12%   6 Virtual Exec
       4           0        94          0  0.00%  0.00%  0.00%   0 DHCPD Timer
       5       11444      1348       8489  0.00%  0.05%  0.05%   0 Check heaps
       6          32        39        820  0.00%  0.00%  0.00%   0 Pool Manager
       7           0         2          0  0.00%  0.00%  0.00%   0 Timers
       8           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun
       9           0         2          0  0.00%  0.00%  0.00%   0 AAA high-capacit
      10         948      1280        740  0.00%  0.00%  0.00%   0 ARP Input
      11           0         3          0  0.00%  0.00%  0.00%   0 DDR Timers
      12          32      2815         11  0.00%  0.00%  0.00%   0 HC Counter Timer
      13          12         2       6000  0.00%  0.00%  0.00%   0 Entity MIB API
      14           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer
      15           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect
      16          12     11246          1  0.00%  0.00%  0.00%   0 GraphIt
      17           0         2          0  0.00%  0.00%  0.00%   0 Dialer event
      18           0         2          0  0.00%  0.00%  0.00%   0 XML Proxy Client
      19           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd
      20         296      3273         90  0.00%  0.00%  0.00%   0 Net Background
      21           0        19          0  0.00%  0.00%  0.00%   0 Logger
      22          40     11242          3  0.00%  0.00%  0.00%   0 TTY Background
      23         132     11255         11  0.00%  0.00%  0.00%   0 Per-Second Jobs
      24           8         2       4000  0.00%  0.00%  0.00%   0 IPM_C1700_CLOCK
      25           0         2          0  0.00%  0.00%  0.00%   0 AggMgr Process
      26           4         2       2000  0.00%  0.00%  0.00%   0 ESWPPM
      27           0         1          0  0.00%  0.00%  0.00%   0 Net Input
      28          84      2253         37  0.00%  0.00%  0.00%   0 Compute load avg
      29        8352       192      43500  0.00%  0.03%  0.04%   0 Per-minute Jobs
      30           0         3          0  0.00%  0.00%  0.00%   0 Service-module a
      31           0         1          0  0.00%  0.00%  0.00%   0 Switch Link Moni
      32           4         2       2000  0.00%  0.00%  0.00%   0 AAA Server
      33           0         1          0  0.00%  0.00%  0.00%   0 AAA ACCT Proc
      34           0         1          0  0.00%  0.00%  0.00%   0 ACCT Periodic Pr
      35           0         1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT
      36           0         2          0  0.00%  0.00%  0.00%   0 AAA Dictionary R
      37     1655116   1351918       1224  1.35%  2.00%  5.19%   0 IP Input
      38           0         1          0  0.00%  0.00%  0.00%   0 ICMP event handl
      39           0         8          0  0.00%  0.00%  0.00%   0 CRYPTO IKMP IPC
      40           0         1          0  0.00%  0.00%  0.00%   0 SSS Manager
      41           4      1503          2  0.00%  0.00%  0.00%   0 SSS Test Client
      42           0         1          0  0.00%  0.00%  0.00%   0 SSS Feature Mana
      43         780     43982         17  0.00%  0.00%  0.00%   0 SSS Feature Time
      44           0         4          0  0.00%  0.00%  0.00%   0 PPP Hooks
      45           0         1          0  0.00%  0.00%  0.00%   0 X.25 Encaps Mana
      46           0         1          0  0.00%  0.00%  0.00%   0 VPDN call manage
      47           0         1          0  0.00%  0.00%  0.00%   0 L2X Data Daemon
      48           0         1          0  0.00%  0.00%  0.00%   0 L2X Socket proce
      49           0         1          0  0.00%  0.00%  0.00%   0 L2X SSS manager
      50           0         2          0  0.00%  0.00%  0.00%   0 L2TP mgmt daemon
      51           0         1          0  0.00%  0.00%  0.00%   0 AC Mgr
      52           0         2          0  0.00%  0.00%  0.00%   0 KRB5 AAA
      53           0         2          0  0.00%  0.00%  0.00%   0 DTP Protocol
      54           4         3       1333  0.00%  0.00%  0.00%   0 PPP IP Route
      55           0         4          0  0.00%  0.00%  0.00%   0 PPP IPCP
      56          32        19       1684  0.00%  0.00%  0.00%   0 DHCPD Receive
      57         524       215       2437  0.00%  0.00%  0.00%   0 IP Background
      58         112       194        577  0.00%  0.00%  0.00%   0 IP RIB Update
      59           0        36          0  0.00%  0.00%  0.00%   0 TCP Timer
      60           8         3       2666  0.00%  0.00%  0.00%   0 TCP Protocols
      61           0         1          0  0.00%  0.00%  0.00%   0 RARP Input
      62           0         1          0  0.00%  0.00%  0.00%   0 Socket Timers
      63           4        40        100  0.00%  0.00%  0.00%   0 HTTP CORE
      64          48       188        255  0.00%  0.00%  0.00%   0 IP Cache Ager
      65           0         1          0  0.00%  0.00%  0.00%   0 PAD InCall
      66           0         2          0  0.00%  0.00%  0.00%   0 X.25 Background
      67           0         2          0  0.00%  0.00%  0.00%   0 PPP SSS
      68          16       189         84  0.00%  0.00%  0.00%   0 Adj Manager
      69          72     21976          3  0.00%  0.00%  0.00%   0 IP NAT Ager
      70           4         2       2000  0.00%  0.00%  0.00%   0 PPP Bind
      71           0         1          0  0.00%  0.00%  0.00%   0 SNMP Timers
      72           0         1          0  0.00%  0.00%  0.00%   0 Inspect Timer
      73           0      3196          0  0.00%  0.00%  0.00%   0 DHCPD Database
      74           0         2          0  0.00%  0.00%  0.00%   0 URL filter proc
      75           0        38          0  0.00%  0.00%  0.00%   0 Authentication P
      76           0         1          0  0.00%  0.00%  0.00%   0 Auth-proxy AAA B
      77           0         1          0  0.00%  0.00%  0.00%   0 IDS Timer
      78           0         1          0  0.00%  0.00%  0.00%   0 COPS
      79           0         2          0  0.00%  0.00%  0.00%   0 Dialer Forwarder
      80           8      2253          3  0.00%  0.00%  0.00%   0 L2F management d
      81           4         1       4000  0.00%  0.00%  0.00%   0 PPTP Mgmt
      82           0         2          0  0.00%  0.00%  0.00%   0 PPTP Data
      83       25680     30471        842  0.00%  0.00%  0.00%   0 Crypto HW Proc
      84           0         1          0  0.00%  0.00%  0.00%   0 XSM_EVENT_ENGINE
      85           8      1126          7  0.00%  0.00%  0.00%   0 XSM_ENQUEUER
      86           4      1126          3  0.00%  0.00%  0.00%   0 XSM Historian
      87           0         2          0  0.00%  0.00%  0.00%   0 LOCAL AAA
      88           0         2          0  0.00%  0.00%  0.00%   0 ENABLE AAA
      89           0         2          0  0.00%  0.00%  0.00%   0 LINE AAA
      90           0         2          0  0.00%  0.00%  0.00%   0 TPLUS
      91         360     14394         25  0.00%  0.00%  0.00%   0 CEF process
      92           4         2       2000  0.00%  0.00%  0.00%   0 Crypto Support
      93           0         1          0  0.00%  0.00%  0.00%   0 EM Background Pr
      94          20       452         44  0.00%  0.00%  0.00%   0 CRM_CALL_UPDATE_
      95           0         1          0  0.00%  0.00%  0.00%   0 Encrypt Proc
      96       14128       117     120752  0.00%  0.00%  0.26%   0 Key Proc
      97          24         4       6000  0.00%  0.00%  0.00%   0 Crypto CA
      98           0         1          0  0.00%  0.00%  0.00%   0 Crypto SSL
      99          20        40        500  0.00%  0.00%  0.00%   0 Crypto ACL
     100           0         1          0  0.00%  0.00%  0.00%   0 CRYPTO QoS proce
     101          12        14        857  0.00%  0.00%  0.00%   0 Crypto Delete Ma
     102         104       109        954  0.00%  0.00%  0.00%   0 Crypto IKMP
     103        1212       588       2061  0.00%  0.00%  0.00%   0 IPSEC key engine
     104           0         1          0  0.00%  0.00%  0.00%   0 IPSEC manual key
     105           0         2          0  0.00%  0.00%  0.00%   0 AAA SEND STOP EV
     106           0         1          0  0.00%  0.00%  0.00%   0 Syslog Traps
     107           0         2          0  0.00%  0.00%  0.00%   0 IpSecMibTopN
     108          32      1218         26  0.00%  0.00%  0.00%   0 PM Callback
     109       38908   2770816         14  0.07%  0.03%  0.07%   0 SAA Event Proces
     110           0         1          0  0.00%  0.00%  0.00%   0 VPDN Scal
     111           4         2       2000  0.00%  0.00%  0.00%   0 VLAN Manager
     112       18904     44385        425  0.31%  0.17%  0.16%   0 COLLECT STAT COU
     113           0         1          0  0.00%  0.00%  0.00%   0 TCP Driver
     114           0         1          0  0.00%  0.00%  0.00%   0 TCP Listener
     115           0         1          0  0.00%  0.00%  0.00%   0 IP NAT WLAN
     116          40       169        236  0.00%  0.00%  0.00%   0 SSH Event handle
     117        2668    350258          7  0.00%  0.00%  0.00%   0 PPP manager
     118        2508    350268          7  0.00%  0.00%  0.00%   0 PPP Events
     119          12     11258          1  0.00%  0.00%  0.00%   0 Multilink PPP
     120           0         2          0  0.00%  0.00%  0.00%   0 Multilink event
     121           0         2          0  0.00%  0.00%  0.00%   0 IP Flow Backgrou
     122          48       249        192  0.00%  0.00%  0.00%   0 Crypto Hardware
     123          16       369         43  0.00%  0.00%  0.00%   0 CEF Scanner
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    One question, was there anyone working at the time?

    0
     

    Author Comment

    by:tangofniro
    No it was around 9:00 pm
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    How about right now? Run the same commands. To me the CPU last night was very high for no body around
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Also need to get the analyzer up and running, need to baseline!
    0
     

    Author Comment

    by:tangofniro
    right now


    RPNAPLES#sh proc
    CPU utilization for five seconds: 42%/33%; one minute: 47%; five minutes: 50%
     PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
       1 Cwe 8029A2D0          268         64    4187 5548/6000   0 Chunk Manager
       2 Csp 80280AB0          144      11797      12 2744/3000   0 Load Meter
       3 M*         0           20         20    1000 9756/12000  6 Virtual Exec
       4 Mwe 8053C650            8        492      16 5768/6000   0 DHCPD Timer
       5 Lst 80299ACC        62660       7260    8630 5752/6000   0 Check heaps
       6 Cwe 8029E304           40         45     888 5536/6000   0 Pool Manager
       7 Mst 801A58E4            0          2       0 5708/6000   0 Timers
       8 Mwe 80096BB8            0          2       0 5716/6000   0 Serial Backgroun
       9 Mwe 80178B38            0          2       0 5704/6000   0 AAA high-capacit
      10 Lwe 8034FD08         7476       9058     825 5008/6000   0 ARP Input
      11 Mwe 80371168            0          3       0 5708/6000   0 DDR Timers
      12 Mwe 80675650          516      13636      37 5768/6000   0 HC Counter Timer
      13 Lwe 80978F34           12          2    6000 5624/6000   0 Entity MIB API
      14 Mwe 80BE83B8            0          2       0 5712/6000   0 ATM Idle Timer
      15 Mwe 8009C57C            0          1       0 5788/6000   0 SERIAL A'detect
      16 Msp 801E85FC          188      58933       3 5744/6000   0 GraphIt
      17 Mwe 8038C274            0          2       011712/12000  0 Dialer event
      18 Mwe 806196F8            0          2       011728/12000  0 XML Proxy Client
      19 Cwe 8026C730            0          1       0 5804/6000   0 Critical Bkgnd
      20 Mwe 8021AC98         1804      10818     16610392/12000  0 Net Background
      21 Lwe 80196DF8            0         21       011356/12000  0 Logger
      22 Mwe 801BB920          300      58929       5 5736/6000   0 TTY Background
     PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
      23 Msp 8022BC78          936      58942      15 7464/9000   0 Per-Second Jobs
      24 Lwe 80311EC0           20          3    6666 1864/3000   0 IPM_C1700_CLOCK
      25 Mwe 812F69B4            0          2       0 5784/6000   0 AggMgr Process
      26 Hwe 81327A00            4          2    2000 5264/6000   0 ESWPPM
      27 Hwe 8022B7A4            0          5       0 5788/6000   0 Net Input
      28 Msp 80220A90         1100      11795      93 5740/6000   0 Compute load avg
      29 Msp 8022BCEC        45540        990   46000 5752/6000   0 Per-minute Jobs
      30 Mwe 8009F05C            0          3       0 5700/6000   0 Service-module a
      31 Mwe 813322AC            0          1       0 5792/6000   0 Switch Link Moni
      32 Mwe 8025067C            4          2    2000 5712/6000   0 AAA Server
      33 Mwe 80252B48            0          1       0 5784/6000   0 AAA ACCT Proc
      34 Mwe 80252C2C            0          1       0 5772/6000   0 ACCT Periodic Pr
      35 Lwe 8017D6A4            0          1       0 5788/6000   0 AAA_SERVER_DEADT
      36 Mwe 80304C34            0          2       0 5724/6000   0 AAA Dictionary R
      37 Mrd 8044EEF4      9094956    6541061    1390 9952/12000  0 IP Input
      38 Mwe 80473998            0          1       0 5768/6000   0 ICMP event handl
      39 Mwe 81159038            0          8       0 5788/6000   0 CRYPTO IKMP IPC
      40 Mwe 8085DB38            0          1       011788/12000  0 SSS Manager
      41 Mwe 80861464          164       7865      2011752/12000  0 SSS Test Client
      42 Mwe 8086AE94            0          1       0 5800/6000   0 SSS Feature Mana
      43 Mwe 8086AF28         2524     230403      10 5768/6000   0 SSS Feature Time
      44 Mwe 8027AE9C            0          4       011732/12000  0 PPP Hooks
      45 Lwe 80A546B4            0          1       0 5396/6000   0 X.25 Encaps Mana
     PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
      46 Mwe 80DFED4C            0          1       011748/12000  0 VPDN call manage
      47 Mwe 80E7AB9C            0          1       011780/12000  0 L2X Data Daemon
      48 Mwe 80E42F6C            0          1       011756/12000  0 L2X Socket proce
      49 Mwe 80E0FD10            0          1       011784/12000  0 L2X SSS manager
      50 Mwe 80E1CB74            0          2       011712/12000  0 L2TP mgmt daemon
      51 Mwe 8105F430            0          1       0 5776/6000   0 AC Mgr
      52 Mwe 81239BE4            0          2       011728/12000  0 KRB5 AAA
      53 Mwe 812E6C64            0          2       0 5540/6000   0 DTP Protocol
      54 Mwe 8027AE9C            4          3    133310988/12000  0 PPP IP Route
      55 Mwe 8027AE9C            0          4       011448/12000  0 PPP IPCP
      56 Mwe 80425860          264        125    2112 3868/6000   0 DHCPD Receive
      57 Mwe 80551360         3340       1011    3303 8224/9000   0 IP Background
      58 Mwe 805574F4          368        990     371 8736/9000   0 IP RIB Update
      59 Mst 8043214C            8         61     13110484/12000  0 TCP Timer
      60 Lwe 80437114           16          6    266610332/12000  0 TCP Protocols
      61 Mwe 8049E14C            0          1       0 5804/6000   0 RARP Input
      62 Hwe 80533708            0          1       0 5776/6000   0 Socket Timers
      63 Mwe 805090E4          584        202    2891 6448/9000   0 HTTP CORE
      64 Lsi 805CB69C         1196        983    1216 5304/6000   0 IP Cache Ager
      65 Hwe 80A6DB7C            0          1       0 5780/6000   0 PAD InCall
      66 Mwe 80A27320            0          2       011708/12000  0 X.25 Background
      67 Mwe 8086CC34            0          2       0 5708/6000   0 PPP SSS
      68 Mwe 80908750          644        985     653 8688/9000   0 Adj Manager
     PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
      69 Mwe 80AA1614         1160     115124      10 5700/6000   0 IP NAT Ager
      70 Mwe 8027AE9C            4          2    2000 5728/6000   0 PPP Bind
      71 Mwe 8096E9F4            0          1       0 5756/6000   0 SNMP Timers
      72 Mwe 80C7B920            0          1       0 5796/6000   0 Inspect Timer
      73 Msi 8054B8C4          148      16711       8 4980/6000   0 DHCPD Database
      74 Mwe 80CA5BF8            0          2       0 5612/6000   0 URL filter proc
      75 Mwe 80CBF034            0        197       0 5796/6000   0 Authentication P
      76 Mwe 80CC8EA0            0          1       0 5776/6000   0 Auth-proxy AAA B
      77 Mwe 80CC9C64            0          1       0 5796/6000   0 IDS Timer
      78 Mwe 80DE114C            0          1       023780/24000  0 COPS
      79 Mwe 80DEFDF0            0          2       0 5720/6000   0 Dialer Forwarder
      80 Mwe 80E06E3C           68      11794       511776/12000  0 L2F management d
      81 Mwe 80E357FC            4          1    400011540/12000  0 PPTP Mgmt
      82 Mwe 80E7F4DC            0          2       011728/12000  0 PPTP Data
      83 Hwe 810EDD9C        25704      30502     842 5064/6000   0 Crypto HW Proc
      84 Lwe 812DB7A4            0          1       0 5796/6000   0 XSM_EVENT_ENGINE
      85 Lsi 812D93DC          116       5895      1911824/12000  0 XSM_ENQUEUER
      86 Lsi 812DC4D4           80       5895      1311828/12000  0 XSM Historian
      87 Mwe 8025D51C            0          2       0 5728/6000   0 LOCAL AAA
      88 Mwe 8025F5D4            0          2       0 5728/6000   0 ENABLE AAA
      89 Mwe 8025F9C0            0          2       0 5732/6000   0 LINE AAA
      90 Mwe 803C8D1C            0          2       0 5604/6000   0 TPLUS
      91 Lwe 808F6E08         3460      79767      43 4368/6000   0 CEF process
     PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
      92 Mwe 810DB840            4          2    2000 5724/6000   0 Crypto Support
      93 Mwe 81387464            0          1       0 5800/6000   0 EM Background Pr
      94 Mwe 806D6DA0           52       2361      22 5668/6000   0 CRM_CALL_UPDATE_
      95 Mwe 810D5BB4            0          1       011804/12000  0 Encrypt Proc
      96 Mwe 810D67A4        64116        467  137293 6684/8000   0 Key Proc
      97 Mwe 811B0058           24          4    6000 7000/8000   0 Crypto CA
      98 Mwe 811EB1D4            0          1       0 7812/8000   0 Crypto SSL
      99 Mwe 81158B3C           20         40     50020432/24000  0 Crypto ACL
     100 Mwe 810E00DC            0          1       0 5792/6000   0 CRYPTO QoS proce
     101 Mwe 81151F4C           36         53     67911332/12000  0 Crypto Delete Ma
     102 Mwe 8111F364          424        386    1098 6168/12000  0 Crypto IKMP
     103 Mwe 8111445C         1268       3013     420 9744/12000  0 IPSEC key engine
     104 Mwe 81114F10            0          1       0 5716/6000   0 IPSEC manual key
     105 Mwe 80171B4C            0          2       0 5708/6000   0 AAA SEND STOP EV
     106 Mwe 80849E8C            0          1       0 5816/6000   0 Syslog Traps
     107 Lwe 812C945C            0          2       0 5648/6000   0 IpSecMibTopN
     108 Mwe 81324B6C          136       6375      21 5752/6000   0 PM Callback
     109 Mwe 80AF1508       190808   14415505      13 5584/6000   0 SAA Event Proces
     110 Mwe 80E45B78            0          1       0 5784/6000   0 VPDN Scal
     111 Mwe 81337730            4          2    2000 3912/6000   0 VLAN Manager
     112 Lsp 8132E680       112484     232949     482 5732/6000   0 COLLECT STAT COU
     113 Mwe 805D0D8C            0          1       011796/12000  0 TCP Driver
     114 Lwe 80435D60            0          1       0 5792/6000   0 TCP Listener
     PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
     115 Mwe 80AD3A54            0          1       0 5772/6000   0 IP NAT WLAN
     116 Mwe 81244984          180        715     251 4428/6000   0 SSH Event handle
     117 Mwe 8099DAD0        31696    1838473      1711696/12000  0 PPP manager
     118 Mwe 8027AE9C        33004    1838482      1711244/12000  0 PPP Events
     119 Hwe 809D5CA8           80      58974       1 5724/6000   0 Multilink PPP
     120 Mwe 809D568C            0          2       0 5712/6000   0 Multilink event
     121 Mwe 80876D7C            0          2       0 5732/6000   0 IP Flow Backgrou
     122 Mwe 810FBD70          208        983     211 4456/6000   0 Crypto Hardware
     123 Lwe 808F6638          140       2023      69 5664/6000   0 CEF Scanner
    RPNAPLES#sh proc cpu
    CPU utilization for five seconds: 33%/22%; one minute: 43%; five minutes: 49%
     PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
       1         268        64       4187  0.00%  0.00%  0.00%   0 Chunk Manager
       2         144     11801         12  0.00%  0.00%  0.00%   0 Load Meter
       3         348      1061        327  0.15%  0.36%  0.08%   6 Virtual Exec
       4           8       492         16  0.00%  0.00%  0.00%   0 DHCPD Timer
       5       62660      7260       8630  0.00%  0.06%  0.05%   0 Check heaps
       6          40        45        888  0.00%  0.00%  0.00%   0 Pool Manager
       7           0         2          0  0.00%  0.00%  0.00%   0 Timers
       8           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun
       9           0         2          0  0.00%  0.00%  0.00%   0 AAA high-capacit
      10        7480      9061        825  0.00%  0.00%  0.00%   0 ARP Input
      11           0         3          0  0.00%  0.00%  0.00%   0 DDR Timers
      12         516     13641         37  0.00%  0.00%  0.00%   0 HC Counter Timer
      13          12         2       6000  0.00%  0.00%  0.00%   0 Entity MIB API
      14           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer
      15           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect
      16         188     58954          3  0.00%  0.00%  0.00%   0 GraphIt
      17           0         2          0  0.00%  0.00%  0.00%   0 Dialer event
      18           0         2          0  0.00%  0.00%  0.00%   0 XML Proxy Client
      19           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd
      20        1812     10820        167  0.07%  0.02%  0.00%   0 Net Background
      21           0        21          0  0.00%  0.00%  0.00%   0 Logger
      22         300     58950          5  0.00%  0.00%  0.00%   0 TTY Background
      23         936     58963         15  0.00%  0.00%  0.00%   0 Per-Second Jobs
      24          20         3       6666  0.00%  0.00%  0.00%   0 IPM_C1700_CLOCK
      25           0         2          0  0.00%  0.00%  0.00%   0 AggMgr Process
      26           4         2       2000  0.00%  0.00%  0.00%   0 ESWPPM
     PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
      27           0         5          0  0.00%  0.00%  0.00%   0 Net Input
      28        1104     11798         93  0.00%  0.00%  0.00%   0 Compute load avg
      29       45592       991      46006  0.63%  0.07%  0.04%   0 Per-minute Jobs
      30           0         3          0  0.00%  0.00%  0.00%   0 Service-module a
      31           0         1          0  0.00%  0.00%  0.00%   0 Switch Link Moni
      32           4         2       2000  0.00%  0.00%  0.00%   0 AAA Server
      33           0         1          0  0.00%  0.00%  0.00%   0 AAA ACCT Proc
      34           0         1          0  0.00%  0.00%  0.00%   0 ACCT Periodic Pr
      35           0         1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT
      36           0         2          0  0.00%  0.00%  0.00%   0 AAA Dictionary R
      37     9097020   6543285       1390 11.27%  8.68%  9.27%   0 IP Input
      38           0         1          0  0.00%  0.00%  0.00%   0 ICMP event handl
      39           0         8          0  0.00%  0.00%  0.00%   0 CRYPTO IKMP IPC
      40           0         1          0  0.00%  0.00%  0.00%   0 SSS Manager
      41         164      7867         20  0.00%  0.00%  0.00%   0 SSS Test Client
      42           0         1          0  0.00%  0.00%  0.00%   0 SSS Feature Mana
      43        2524    230460         10  0.00%  0.00%  0.00%   0 SSS Feature Time
      44           0         4          0  0.00%  0.00%  0.00%   0 PPP Hooks
      45           0         1          0  0.00%  0.00%  0.00%   0 X.25 Encaps Mana
      46           0         1          0  0.00%  0.00%  0.00%   0 VPDN call manage
      47           0         1          0  0.00%  0.00%  0.00%   0 L2X Data Daemon
      48           0         1          0  0.00%  0.00%  0.00%   0 L2X Socket proce
      49           0         1          0  0.00%  0.00%  0.00%   0 L2X SSS manager
     PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
      50           0         2          0  0.00%  0.00%  0.00%   0 L2TP mgmt daemon
      51           0         1          0  0.00%  0.00%  0.00%   0 AC Mgr
      52           0         2          0  0.00%  0.00%  0.00%   0 KRB5 AAA
      53           0         2          0  0.00%  0.00%  0.00%   0 DTP Protocol
      54           4         3       1333  0.00%  0.00%  0.00%   0 PPP IP Route
      55           0         4          0  0.00%  0.00%  0.00%   0 PPP IPCP
      56         264       125       2112  0.00%  0.00%  0.00%   0 DHCPD Receive
      57        3340      1011       3303  0.00%  0.00%  0.00%   0 IP Background
      58         368       990        371  0.00%  0.00%  0.00%   0 IP RIB Update
      59           8        61        131  0.00%  0.00%  0.00%   0 TCP Timer
      60          16         6       2666  0.00%  0.00%  0.00%   0 TCP Protocols
      61           0         1          0  0.00%  0.00%  0.00%   0 RARP Input
      62           0         1          0  0.00%  0.00%  0.00%   0 Socket Timers
      63         584       202       2891  0.00%  0.00%  0.00%   0 HTTP CORE
      64        1196       984       1215  0.00%  0.00%  0.00%   0 IP Cache Ager
      65           0         1          0  0.00%  0.00%  0.00%   0 PAD InCall
      66           0         2          0  0.00%  0.00%  0.00%   0 X.25 Background
      67           0         2          0  0.00%  0.00%  0.00%   0 PPP SSS
      68         644       985        653  0.00%  0.00%  0.00%   0 Adj Manager
      69        1160    115153         10  0.00%  0.00%  0.00%   0 IP NAT Ager
      70           4         2       2000  0.00%  0.00%  0.00%   0 PPP Bind
      71           0         1          0  0.00%  0.00%  0.00%   0 SNMP Timers
      72           0         1          0  0.00%  0.00%  0.00%   0 Inspect Timer
     PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
      73         148     16728          8  0.00%  0.00%  0.00%   0 DHCPD Database
      74           0         2          0  0.00%  0.00%  0.00%   0 URL filter proc
      75           0       197          0  0.00%  0.00%  0.00%   0 Authentication P
      76           0         1          0  0.00%  0.00%  0.00%   0 Auth-proxy AAA B
      77           0         1          0  0.00%  0.00%  0.00%   0 IDS Timer
      78           0         1          0  0.00%  0.00%  0.00%   0 COPS
      79           0         2          0  0.00%  0.00%  0.00%   0 Dialer Forwarder
      80          68     11797          5  0.00%  0.00%  0.00%   0 L2F management d
      81           4         1       4000  0.00%  0.00%  0.00%   0 PPTP Mgmt
      82           0         2          0  0.00%  0.00%  0.00%   0 PPTP Data
      83       25704     30502        842  0.00%  0.00%  0.00%   0 Crypto HW Proc
      84           0         1          0  0.00%  0.00%  0.00%   0 XSM_EVENT_ENGINE
      85         116      5896         19  0.00%  0.00%  0.00%   0 XSM_ENQUEUER
      86          80      5896         13  0.00%  0.00%  0.00%   0 XSM Historian
      87           0         2          0  0.00%  0.00%  0.00%   0 LOCAL AAA
      88           0         2          0  0.00%  0.00%  0.00%   0 ENABLE AAA
      89           0         2          0  0.00%  0.00%  0.00%   0 LINE AAA
      90           0         2          0  0.00%  0.00%  0.00%   0 TPLUS
      91        3464     79785         43  0.07%  0.00%  0.00%   0 CEF process
      92           4         2       2000  0.00%  0.00%  0.00%   0 Crypto Support
      93           0         1          0  0.00%  0.00%  0.00%   0 EM Background Pr
      94          52      2362         22  0.00%  0.00%  0.00%   0 CRM_CALL_UPDATE_
      95           0         1          0  0.00%  0.00%  0.00%   0 Encrypt Proc
     PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
      96       64116       467     137293  0.00%  0.00%  0.00%   0 Key Proc
      97          24         4       6000  0.00%  0.00%  0.00%   0 Crypto CA
      98           0         1          0  0.00%  0.00%  0.00%   0 Crypto SSL
      99          20        40        500  0.00%  0.00%  0.00%   0 Crypto ACL
     100           0         1          0  0.00%  0.00%  0.00%   0 CRYPTO QoS proce
     101          36        53        679  0.00%  0.00%  0.00%   0 Crypto Delete Ma
     102         424       386       1098  0.00%  0.00%  0.00%   0 Crypto IKMP
     103        1268      3014        420  0.00%  0.00%  0.00%   0 IPSEC key engine
     104           0         1          0  0.00%  0.00%  0.00%   0 IPSEC manual key
     105           0         2          0  0.00%  0.00%  0.00%   0 AAA SEND STOP EV
     106           0         1          0  0.00%  0.00%  0.00%   0 Syslog Traps
     107           0         2          0  0.00%  0.00%  0.00%   0 IpSecMibTopN
     108         136      6377         21  0.00%  0.00%  0.00%   0 PM Callback
     109      190920  14418995         13  0.71%  0.32%  0.34%   0 SAA Event Proces
     110           0         1          0  0.00%  0.00%  0.00%   0 VPDN Scal
     111           4         2       2000  0.00%  0.00%  0.00%   0 VLAN Manager
     112      112516    233006        482  0.15%  0.17%  0.20%   0 COLLECT STAT COU
     113           0         1          0  0.00%  0.00%  0.00%   0 TCP Driver
     114           0         1          0  0.00%  0.00%  0.00%   0 TCP Listener
     115           0         1          0  0.00%  0.00%  0.00%   0 IP NAT WLAN
     116         180       715        251  0.00%  0.00%  0.00%   0 SSH Event handle
     117       31696   1838902         17  0.00%  0.01%  0.03%   0 PPP manager
     118       33008   1838911         17  0.00%  0.01%  0.04%   0 PPP Events
     PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
     119          80     58988          1  0.00%  0.00%  0.00%   0 Multilink PPP
     120           0         2          0  0.00%  0.00%  0.00%   0 Multilink event
     121           0         2          0  0.00%  0.00%  0.00%   0 IP Flow Backgrou
     122         208       983        211  0.00%  0.00%  0.00%   0 Crypto Hardware
     123         140      2024         69  0.00%  0.00%  0.00%   0 CEF Scanner
    RPNAPLES#
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Do this for every hour for the next three hours.
    0
     

    Author Comment

    by:tangofniro
    Okay

    I have the full version of Colasoft running right now also.

    No problems reported today.  Running testvoip also showing good.

    colasoft showing me rigt now after about 1 hr running  4mb internet total
                                                                                    450mb local/intranet  total
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Grab a coke and smoke and wait.

    Mkae sure that you are savings all the captures for review later on, do it every hour if you can. The best thing we can do is leave this running till the problem pops back up.
    0
     

    Author Comment

    by:tangofniro
    will let it run over the holidays,
    I will out of town until Sunday.
    They said it got a little static before lunch  and after, but nothing horrible.  I never saw anything to cause it though.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Always hard to find a ghost. One thing to think about is that though yo may have a lot of traffic, it still may be the carrier. The best test would be to set up a monitor at site A, then one at site B. Set up two computers to "talk" to each other over various ports/UDP-TCP, also set up two VoIP phones also (usea audio source via speakerphone). This way we could actually see the packets leaving site A and arrive at site B. From this we could tell if the carrier is causing problems some how.

    Did you adjust the MTU size by chance already?


    Have a good holiday!

    Joel
    0
     

    Author Comment

    by:tangofniro
    MTU is 900 per your request.

    Have a great holiday.

    Will
    0
     

    Author Comment

    by:tangofniro
    Hello Joel,

    Hope you had a good holiday.

    Would you please do me a big favor and look at the configs again to make sure I have it correct.
    Phone quality has been terrible and I just don't know if my QOS is working properly.

    172.168.1.0 is the remote internal network with the ip phones.  

    It connects back to the 192.168.0.0 network where the phone equip resides.

    I have a full T1 at the 192   and 1000k at the 172.

    Thanks,
    Will
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Will,

    Had a good holiday, I sent a request to one of the other Experts (Dr-IP) his skill set for QoS on Cisco with VoIP is the best I have seen at EE. Always remeber to get a second Dr.'s opinion (LOL).

    Joel
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    I have only glimpsed at the thread, but it is clear that the router CPU had a heavy load on it even when it should be doing almost nothing, and if it’s that high when it should be idle, it probably is overloading during the day. So the priority here should be finding out why it’s so high when it shouldn’t, and correcting it. Once that is taken care of, if you are still having issues, they can then be addressed.

    The first question is, where is the load coming from? I’d log into one of the routers, and shut down the Ethernet interface on it, and telnet from it to the other router, and do that same. Wait a few minutes and do a show process CPU on both routers and see if it drops, if it doesn’t do a show interface serial X on both routers, and see if you are getting a lot of trash coming from your ISP. If so, call them up and work with them to eliminate it.

    Now if the trash isn’t coming from your ISP, turn up the Ethernet interface one router at a time and see if it gets pounded, show interface FastEthernet X, if dose, find out what on that LAN is causing it. An easy way to do that is yank all the connectors out of the switch connected to the router, and plug them back in a few at a time until the traffic spikes back up. When it dose, pull out the last group of wires and plug them in one at a time to find the guilty party.

    I suspect by the way you have some kind of worm on your LAN, or people running things they shouldn’t P2P software, which can easily overload a T1 if used heavily, but it also could be some network app, that is just too big a load for your internet connection.
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    Note log on using the consol cable.
    0
     

    Author Comment

    by:tangofniro
    Well I closed down the fe ports on both and the it went down to nothing.  

    I will try to narrow down the problems internally.


    Can you look at the access lists so I can try to block most traffic except the phones and the 3389 for terminal server?

    Thanks Dr and Joel



    will
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    We could try doing that, but the router still has to address all that trash, which loads down the CPU, although maybe not as much. So let’s get rid of trash first and latter we can play with the access lists. Also, just to be sure, you are using a switch, because if you are using a hub, the router will get a lot of needless LAN traffic.
    0
     

    Author Comment

    by:tangofniro
    Ok well here it is  the lan throws nothing on the router when just the lan is plugged in I am running 2% maybe every once in a while  10%.  When I plug the phone system in is when the cpu gets overloaded.

    help



    thanks,

    will
    0
     

    Author Comment

    by:tangofniro
    after clearing counters after hours  about 3min after clearing


    RPNAPLES#sh access-lists
    Extended IP access list 100
        10 permit udp any any range 16384 32000
        20 permit tcp any any eq 1719
        30 permit tcp any any eq 1720
        40 permit tcp any any eq 6100
        50 permit tcp any any range 1024 4999 (459 matches)
        60 permit udp any any eq 6000 (717 matches)
        70 permit udp any any range 1024 4999 (11 matches)
        80 permit udp any any eq 5060
        90 permit udp any any range 30000 30030
        100 permit udp any any range 9000 9001 (10800 matches)
    Extended IP access list 101
        10 permit ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255 (36091 matches)
        20 deny ip 192.168.0.0 0.0.0.255 any
    Extended IP access list 102
        10 permit tcp any any range 5000 5110
    Extended IP access list 105
        10 deny icmp any any echo
        20 permit ip any any
    Extended IP access list 110
        10 deny ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255 (2 matches)
        20 deny ip host 192.168.0.205 any (171 matches)
        30 deny ip host 192.168.0.30 any
        40 deny ip host 192.168.0.31 any
        50 deny ip host 192.168.0.2 any (57 matches)
        60 permit ip 192.168.0.0 0.0.0.255 any (24 matches)
    Extended IP access list 120
        10 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255
    Extended IP access list 123
        10 permit ip host 192.168.0.205 172.168.1.0 0.0.0.255 (20 matches)
        20 permit ip host 192.168.0.30 172.168.1.0 0.0.0.255 (550 matches)
        30 permit ip host 192.168.0.31 172.168.1.0 0.0.0.255 (7761 matches)
        40 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255 (84 matches)
    Extended IP access list sl_def_acl
        10 deny tcp any any eq telnet log
        20 deny tcp any any eq www log
        30 deny tcp any any eq 22 log
        40 permit ip any any log
    RPNAPLES#
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    To say that is odd would be a gross understatement, and it definitely is not what I would have expected. I work with carrier grade VOIP equipment, and some of those gateways can handle over 700 calls, yet when they are idle, at the most they use a few kilobytes of bandwidth to periodically renew their registration to the gatekeeper.

    So there is no reason on earth a little office VOIP system should use anywhere even remotely close to what your system is using when idle. It’s not playing background music on the phones like I have seen in some offices, or something like that? As that is the only thing I can think of that could account for what you are seeing.      
    0
     

    Author Comment

    by:tangofniro
    No back ground music.  

    I am stumped.  It was around 7:00pm and everyone had been gone since 5:00.
    0
     

    Author Comment

    by:tangofniro
    here is the access-lists from this morning.


    RPNAPLES#sh access-list
    Extended IP access list 100
        10 permit udp any any range 16384 32000
        20 permit tcp any any eq 1719
        30 permit tcp any any eq 1720
        40 permit tcp any any eq 6100
        50 permit tcp any any range 1024 4999 (7689 matches)
        60 permit udp any any eq 6000 (81348 matches)
        70 permit udp any any range 1024 4999 (3443 matches)
        80 permit udp any any eq 5060
        90 permit udp any any range 30000 30030
        100 permit udp any any range 9000 9001 (2795673 matches)
    Extended IP access list 101
        10 permit ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255 (9532221 matches)
        20 deny ip 192.168.0.0 0.0.0.255 any
    Extended IP access list 102
        10 permit tcp any any range 5000 5110 (55 matches)
    Extended IP access list 105
        10 deny icmp any any echo
        20 permit ip any any
    Extended IP access list 110
        10 deny ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255 (485 matches)
        20 deny ip host 192.168.0.205 any (59493 matches)
        30 deny ip host 192.168.0.30 any
        40 deny ip host 192.168.0.31 any
        50 deny ip host 192.168.0.2 any (12442 matches)
        60 permit ip 192.168.0.0 0.0.0.255 any (4405 matches)
    Extended IP access list 120
        10 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255
    Extended IP access list 123
        10 permit ip host 192.168.0.205 172.168.1.0 0.0.0.255 (2396 matches)
        20 permit ip host 192.168.0.30 172.168.1.0 0.0.0.255 (65236 matches)
        30 permit ip host 192.168.0.31 172.168.1.0 0.0.0.255 (1866125 matches)
        40 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255 (17106 matches)
    Extended IP access list sl_def_acl
        10 deny tcp any any eq telnet log
        20 deny tcp any any eq www log
        30 deny tcp any any eq 22 log
        40 permit ip any any log
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    A show interface would be more relevant here since we know what is generating most of the traffic, but counting the match’s on ports 9000-1, I assume that its for the phone system, for what you say was three minutes, it looks like the phone systems using enough bandwidth for a good twenty plus calls when idle.

    On a local LAN that wouldn’t be a problem, but across a slow WAN link, running through a VPN, on small router with a modest processor, processing access lists and doing QOS, it’s a recipe for trouble, but I think you have figured that out already. Also seeing how much bandwidth the phones are using when idle, makes me wonder how much bandwidth they use when in operation, and if a single T1 can handle it?

    I think it’s time you get a hold of the people, who provided your phone system, as there is nothing I can do about other than recommend you get more powerful routers. You might still need to do that since the routers you have a marginal for what you are dong with them, but until get rid of this parasitic load and see how it behaves I can’t say that. Also even worse, in addition too more powerful routers you might need to get another T1 to deal with all the bandwidth that phone system is sucking up.

    So trying to fix this on the Cisco end would be expensive, especially if it came down to getting more T1’s. So this needs to be addressed at the phone end of things if at all possible, since with all seven remote phones in operation it should be using about 70-80K of bandwidth, but it looks like it’s using closer to 250K when idle, and probable a hell of lot more in operation. There has got to be something in it’s configuration to cut this down to something reasonable, but since I don’t work with Samsung phone systems I have no clue as to where to start.  
    0
     

    Author Comment

    by:tangofniro
    here is my cleared interfaces after 5mins

    RPNAPLES#sh int s0
    Serial0 is up, line protocol is up
      Hardware is PQUICC with Fractional T1 CSU/DSU
      Description: USLEC T1
      Internet address is 199.72.194.234/30
      MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
         reliability 255/255, txload 12/255, rxload 36/255
      Encapsulation PPP, LCP Open
      Open: IPCP, loopback not set
      Last input 00:00:00, output 00:00:00, output hang never
      Last clearing of "show interface" counters 00:10:36
      Input queue: 3/75/0/0 (size/max/drops/flushes); Total output drops: 8
      Queueing strategy: weighted fair
      Output queue: 0/1000/64/8 (size/max total/threshold/drops)
         Conversations  0/8/256 (active/max active/max total)
         Reserved Conversations 0/0 (allocated/max allocated)
         Available Bandwidth 52 kilobits/sec
      5 minute input rate 219000 bits/sec, 304 packets/sec
      5 minute output rate 78000 bits/sec, 81 packets/sec
         193593 packets input, 17672596 bytes, 0 no buffer
         Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
         0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
         50723 packets output, 6189640 bytes, 0 underruns
         0 output errors, 0 collisions, 0 interface resets
         0 output buffer failures, 0 output buffers swapped out
         0 carrier transitions
         DCD=up  DSR=up  DTR=up  RTS=up  CTS=up



    RPNAPLES#sh int f0
    FastEthernet0 is up, line protocol is up
      Hardware is PQUICC_FEC, address is 000f.f7f8.4b2a (bia 000f.f7f8.4b2a)
      Description: $FW_INSIDE$$ETH-LAN$
      Internet address is 192.168.0.1/24
      MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
         reliability 255/255, txload 1/255, rxload 1/255
      Encapsulation ARPA, loopback not set
      Keepalive set (10 sec)
      Full-duplex, 10Mb/s, 100BaseTX/FX
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input 00:00:00, output 00:00:00, output hang never
      Last clearing of "show interface" counters 00:12:13
      Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 0
      Queueing strategy: priority-list 5
      Output queue (queue priority: size/max/drops):
         high: 0/20/0, medium: 0/40/0, normal: 0/60/0, low: 0/80/0
      5 minute input rate 60000 bits/sec, 81 packets/sec
      5 minute output rate 155000 bits/sec, 302 packets/sec
         57995 packets input, 5391239 bytes
         Received 167 broadcasts, 0 runts, 0 giants, 0 throttles
         914 input errors, 299 CRC, 0 frame, 0 overrun, 0 ignored
         0 watchdog
         0 input packets with dribble condition detected
         222447 packets output, 14317892 bytes, 0 underruns
         0 output errors, 0 collisions, 0 interface resets
         0 babbles, 0 late collision, 0 deferred
         0 lost carrier, 0 no carrier
         0 output buffer failures, 0 output buffers swapped out



    I will keep you posted
    0
     

    Author Comment

    by:tangofniro
    didn't really want to post my ip address.
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    I won’t worry too much about posting your routers IP address, a lot of people here are way too paranoid about that. The honest truth is, if your equipment is that poorly secured; it will be found, and hacked regardless. At least once a day, someone scans all 4000 of my IP address to find active address to attempt to compromise systems using automated tools and scripts, that where the name script kiddies comes from by the way, and if it wasn’t secured with proper passwords and access lists, I’d be in trouble.

    What I would worry about it posting passwords, or the address of VOIP gateways, as there are more than a few tricks that can be used on a lot of them to trick them into taking calls without the need to even hack them. That’s because they are inherently insecure, and a lot of people are unaware of that, and or don’t have a clue as to how to properly secure them.

    Just to give you an idea of have bad your situation is, below I have posted the show interface from one of my smaller VOIP gateways that had 40 calls on it at the time. As you can see, your packet rate when idle is pretty close to mine under far more calls than you will ever have. It’s the packet rate that really loads down the processor on your router by the way, as each one has to be processed to determine how it is going to be handled, and with QOS and all those access lists it puts a pretty big load on the little routers CPU.

    As for you show interfaces, it confirms what I expected.      


    5309#sh int f0
    FastEthernet0 is up, line protocol is up
      Hardware is DEC21140, address is 0010.7be6.4ff9 (bia 0010.7be6.4ff9)
      Internet address is 10.0.0.179/24
      MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
         reliability 255/255, txload 1/255, rxload 1/255
      Encapsulation ARPA, loopback not set
      Keepalive set (10 sec)
      Full-duplex, 100Mb/s, 100BaseTX/FX
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input 00:00:00, output 00:00:00, output hang never
      Last clearing of "show interface" counters never
      Input queue: 0/512/0/25 (size/max/drops/flushes); Total output drops: 0
      Queueing strategy: fifo
      Output queue: 0/40 (size/max)
      5 minute input rate 322000 bits/sec, 399 packets/sec
      5 minute output rate 317000 bits/sec, 391 packets/sec
         436079427 packets input, 1052027595 bytes
         Received 3651511 broadcasts, 0 runts, 0 giants, 0 throttles
         0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
         0 watchdog
         0 input packets with dribble condition detected
         336035809 packets output, 4175800439 bytes, 0 underruns
         0 output errors, 0 collisions, 3 interface resets
         0 babbles, 0 late collision, 0 deferred
         0 lost carrier, 0 no carrier
         0 output buffer failures, 0 output buffers swapped out
    5309#
    0
     

    Author Comment

    by:tangofniro
    better news called my phone guy and he reset the system


    here is what it shows


    RPNAPLES#sh int f0
    FastEthernet0 is up, line protocol is up
      Hardware is PQUICC_FEC, address is 000f.f7f8.4b2a (bia 000f.f7f8.4b2a)
      Description: $FW_INSIDE$$ETH-LAN$
      Internet address is 192.168.0.1/24
      MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
         reliability 255/255, txload 1/255, rxload 1/255
      Encapsulation ARPA, loopback not set
      Keepalive set (10 sec)
      Full-duplex, 10Mb/s, 100BaseTX/FX
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input 00:00:00, output 00:00:00, output hang never
      Last clearing of "show interface" counters 00:10:09
      Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
      Queueing strategy: priority-list 5
      Output queue (queue priority: size/max/drops):
         high: 0/20/0, medium: 0/40/0, normal: 0/60/0, low: 0/80/0
      5 minute input rate 2000 bits/sec, 4 packets/sec
      5 minute output rate 1000 bits/sec, 2 packets/sec
         3443 packets input, 252563 bytes
         Received 156 broadcasts, 0 runts, 0 giants, 0 throttles
         0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
         0 watchdog
         0 input packets with dribble condition detected
         1596 packets output, 128515 bytes, 0 underruns
         0 output errors, 0 collisions, 0 interface resets
         0 babbles, 0 late collision, 0 deferred
         0 lost carrier, 0 no carrier
         0 output buffer failures, 0 output buffers swapped out



    cpu is around  1%-5%

    will see if voice quality is any better

    after the reset  9000 - 9001 is not getting any traffic now.  which it was getting hammered before.

    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    That’s more like it, and getting rid of that extraneous load can only help call quality. That router doesn’t have a very powerful processor, and having a load of that level on it before it even starts processing meaningful traffic is bound to send it into overload. The main thing is to keep an eye on that phone system, to see if it starts doing it again. There might be some bug in the code that will keep reappearing, so watch it closely.  
    0
     

    Author Comment

    by:tangofniro
    10  4
    I will see how it goes come monday  and report back.

    So many thanks though to you guys pointing me in the right direction.

    I have not forgotten about you either Joel.


    Thanks,
    will
    0
     

    Author Comment

    by:tangofniro
    Well I checked it this morning and it is up to around 20%

    I also have around 2000crcs on the ethernet interface


    5min input rate is 23000 bits a second    32 packets/sec

    5min output rate is 53000 bits /sec   102 packets/sec  
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    When did you reset the CRC count?

    Also port 9000 is the default Voice port the Samsung phones use. Are you using OfficeServ 500 or 7200? Are using any of the applications? OfficeServ LInk, Operator, Call Monitor, ACD?

    One question is that are people using Call Forward or DND on their phones? Since 9000 is for the voice payload, depending upon the Samsung system to reconize call routing logical loops, you could have someone who has call forwared their phone to someone who has forward theri phone to someplace else, even the other phone.

    Since we have reconized that commmunications on port 9000 is the major problem, lets use Colasoft or even Ethereal to capture the packets and see where they are going?

    Also what type of switches are you using in your network? Also the exact equipment you have via the phone system.

    Also just to mention that 9000/9001 port is used in Microsoft's massively-
    multiplayer game called "Asheron's Call".The game can continue to contact the player even after the player has logged out.

    http://www.fuzeqna.com/asheronscall/consumer/kbdetail.asp?kbid=232

    Not saying this is the problem but you never know, do you know if you have any gamers on the Network?
    0
     

    Author Comment

    by:tangofniro
    I reset the crc count yesterday afternoon.   I think port 9000 is the signal port on samsung.


    I use some very simplistic Linksys/DLINK 10/100 , which could be weak I suppose.

    No gamers in the office

    I will find what the phone use software wise
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    It looks like it’s going up quickly, and will probably be back to what it was before it was reset in another day or two. As for the cheap switches, they should be OK; the important thing is not to be using hubs, as the router would be seeing every packet going across the LAN. One other thing you will want to check with the phone people, what codec are they using? You what to be sure they are not using G711, as it uses a lot of bandwidth, it’s OK for using on the local LAN, but you don’t want to use it for the remote phones. For them you want to be using G729, or G723 since they give the best combination of voice quality verses bandwidth usage.    
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    6000 is the typical signal port for the Samsung phones. Do you have the ability to enter into the MMC programing state on the phones? If so we can verify the codecs and ports being used.

    Press Transfer button on phone, enter 800, then passcode (default is 4321). Press 1 to enable tenant, press speaker button, dial MMC code 834 835 or 840.

    834 option numer 14 will show signal port, but also check the following options under this MMC 00, 01,11 and post what they are.

    835 will show the type of MGI (1,2 or 3). Let me know what is configured. If MGI3, the default codec should be G.729a, but there are a fewother parameters that we can check, but need to find out what is being supported.

    840 options, 04, 05, 07, 08

    0
     

    Author Comment

    by:tangofniro
    They have changed the default passcode.  Can't get a hold of the phone guy either.  Will find out everything tomorrow.


    Thanks Joel

    will
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    While we wait on that can you capture some of the packets? Have you used Ethereal before? If you can capture some of the packets and post it, we can hunt down to see if the communcation is between a few endpoints or a system wide configuration. You can use the filter feature of Ethereal and just capture the packets using port 9000.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Overall how do you like the Samsung system? I sold and installed the Samsung DCS for a while, great little system just most people looked at us strange when we brought up the word Samsung (branded with cheap kitchen appliances, though they are different divisions entirely).
    0
     

    Author Comment

    by:tangofniro
    I haver ethereal on the network but it is on one of my server 2 switches down the line.

    I can't capture from there can I?
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    No easily (and would need a managed switch to do so).

    what about installing on same machine as Colasoft?
    0
     

    Author Comment

    by:tangofniro
    I am not in the office and the cola soft is on my laptop.  I usally just put a hub on the network and plu in after the router but can't do that right now.


    To answer the question about the System It has been great/ Except the remote phones..ha

    Samsung has been making good stuff for a while, I think.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Okay on the Ethereal, what i am looking to do is isolate where the problem might be. We know thw phone system is part of it. But is it becasue of traffic from PBX to remote phones or all phones? Guess we can hit this tommorow!
    0
     

    Author Comment

    by:tangofniro
    I looked at it and colasoft does not seem to give me the correct traffic.  I don't see the traffic that heavy.

    Also on another note  all traffic from the remote site has a hostname of A****.ipt.aol.com    what the hell is that??


    Router is running back around 70-80 percent.  I just don't see it on the basline program.  I see a few 9000-9001 and alot of 30000-30009 but the traffic is only 40kb sec looking at my software.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    When I was looking up the ports I saw some stuff in reagrds to those ports, AOL and altavista, let me see if I can find anything
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Is the hostname something like AC8730C9.ipt.aol.com?
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    From the host names it'd look like it's some AOL service, but a lot of Trojans use those ports, so I’d block it and see what happens, especially since they are not on any of my lists for AOL services like instant messenger.
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    PORT NUMBERS

    (last updated 2 December 2004)

    http://www.iana.org/assignments/port-numbers
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Best bet is go to each and every machine you have and run TrendMicros House call
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Lets assume that you have a trojan, worm running amuck. A quick fix on the phone side might be able to adjust what ports the phones and PBX use to communciate on.
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    The IP address that maps to that name is an AOL address, but I'd block it anyway at the Ethernet interface, as you need to get rid of any extraneous traffic you can get rid of. The real question is what is loading down the processor, do a show processes CPU, and look to see what process is loading it down.

    http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Also there  a few other tools avaliable to help disinfecting your systems:

    http://www.gfi.com/lannetscan/

    http://www.intermute.com/

    http://www.spywareinfo.com/~merijn/    Hijackthis is a great tool


    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    How much traffic is going though the routers interfaces, also if you don’t have this in the config, add it “ip cef”.

    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Also instead of using Colasoft should try using Ethereal, Colasoft might be filtering out traffic from its view.

    Also there is another great tool that is good to apply directly onto a machine that you want to check out called AATools:

    http://www.glocksoft.com/aatools.htm
    0
     

    Author Comment

    by:tangofniro
    Thanks Joel
    Some of the cisco articles where helpful

    I ran some looging on the router for about 5 secs and here is what I got some is cut out due to space of this page
    It is constant.  Constantly the switch is looking for the phones.  If I were to put are the logging I had after 5secs in word It would at least be 1 1/2 pages.   I talked to my phone guy today he thought they might be using g711 , I told him I kept seeing 64k packets coming in from the phone.

    001876: May 15 05:08:11.429: IP: s=192.168.0.31 (FastEthernet0), d=172.168.1.216
     (Loopback0), g=1.1.1.2, len 80, forward
    001877: May 15 05:08:11.433:     UDP src=30000, dst=9000
    001878: May 15 05:08:11.433: IP: s=192.168.0.31 (Loopback0), d=172.168.1.216 (Se
    rial0), g=199.72.194.233, len 80, forward
    001879: May 15 05:08:11.433:     UDP src=30000, dst=9000
    001880: May 15 05:08:11.441: IP: s=192.168.0.31 (FastEthernet0), d=172.168.1.108
     (Loopback0), g=1.1.1.2, len 80, forward
    001881: May 15 05:08:11.441:     UDP src=30004, dst=9000
    001882: May 15 05:08:11.445: IP: s=192.168.0.31 (Loopback0), d=172.168.1.108 (Se
    rial0), g=199.72.194.233, len 80, forward
    001883: May 15 05:08:11.445:     UDP src=30004, dst=9000
    001884: May 15 05:08:11.473: IP: s=192.168.0.31 (FastEthernet0), d=172.168.1.216
     (Loopback0), g=1.1.1.2, len 80, forward
    001885: May 15 05:08:11.473:     UDP src=30000, dst=9000
    001886: May 15 05:08:11.473: IP: s=192.168.0.31 (Loopback0), d=172.168.1.216 (Se
    rial0), g=199.72.194.233, len 80, forward
    001887: May 15 05:08:11.473:     UDP src=30000, dst=9000
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Here is a stupid question, do the phones ever register with the PBX?
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Also what if there is a mismatch of setings between the phones and phone system?
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Plus which system are you using? 500 or 7200?
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Also what series of the phones?
    0
     

    Author Comment

    by:tangofniro
    yeah I am actually on the phone with the phone guy right now he can see the phones registered
    0
     

    Author Comment

    by:tangofniro
    500

    ITP 5021d
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    okay thats a positve, what about any other applications, OfficeLink, ACD? Can he account for the ports being used?
    0
     

    Author Comment

    by:tangofniro
    to me it looks as though it is only 2 phones causing the traffic   .108  and .216



    I can see my phone at the house register through 1/100th of the time those two try.



    Switch is constantly trying to reach those phones.

    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    In order to register few things:

    Right ext and password, right ports setup on phone to communicate.

    Need the password to log into the programming of the phones
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    One thing that is coming clear to me, that although you have other issues, the router you are using is marginal at best. Even if you do manage to get rid of all the other issues, at best you will be skirting right on the edge of its capabilities. I really think you need to start thinking about upgrading to something like a 2650, or 2811. Which are a lot better suited for doing what you are trying to do.
    0
     

    Author Comment

    by:tangofniro
    I am seeing that the router is struggling,  but before I purchased these 1721s  I had to have approval from Samsung on them.   Samsung approved them and said they had 1721 in operation everywhere.    In the very near future these router will probably be taken off the public internet and placed in an enviroment where it will just be a point to point t1 between the remote office and the main office and then have another router for the internet.  If you think all of the problems are just due to the fact these routers are underpowered please tell me now.  If you do not think they will work in the next application speak up for my sake..ha  

    Is the log above normal?  Only those 2 phones were acting that way.  They crush this router.

    Thanks again,
    will
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    If it was just a point to point circuit, without extensive access lists, and doing VPN encryption, along with QOS, the load on the routers processor would be significantly less than it currently is. So I’d say in a point to point private circuit properly configured there would be a decent chance they would be ok, but that would add a permanent overhead expense the you could possibly ovoid by having more powerful routers. So buying a more expensive router in the long run could be less expensive.

    Now this all leads me back to what I have said already, that before we can say they can work, we need to fix the other issues, but seeing how easily they are overloaded by what shouldn’t be a major issue, i.e., less than a quarter meg of trash from the remote phones, it is clear they are marginal. This is why I am say, hey we may be able to get this to work, but every thing will have to be perfect for that. So although they might be able to be made to work ok, they’d always be susceptible to any issue that loads them down. So yes they are struggling, and I’d say Samsung although they might have 1721’s working ok in other places, they a skating on thin ice.      
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    I think I will sum it up this way, the routers are not the whole issue, but if they were more powerful. The issues you are having would at the least be considerably less pronounced, or maybe not even noticeable.  
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    I will have to agree with the Doc on the 1721 being overloaded easily. Have to remember that the 1721 was not originally deisgined to handle what is has been modifiy to handle. On top of being a self proclaimed convergence technonlogist, I was a General contractor for years, and no matter how good of a finish trim guy you have, the foundation is crooked its crooked so to speak.

    You could bump to a bigger and better router or you could also offload the VPN to a dedicated VPN device. Processing the packets for encryption takes time and overhead.

    Also remember all it takes is one bad apple to crash an Ethernet network, though less likely these days with switches but we are still resticted by the WAN.
    0
     

    Author Comment

    by:tangofniro
    very true

    When just the VPN, Acess-lists, Nat running, there is really no pressure on the cpu it is low  2-10% but anytime the phone switch is running, maybe through no real fault of it's own, the performance drops.


    Do you think by offloading the vpn and leaving everything else the way it is that would help?  I am very skeptical, almost gunshy.   I have been pricing 2650s on ebay  and well they are a hell of a lot cheaper than what I paid new for the 1721s with VPN.  damn....
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Well one way to test is to remove the VPN completely from the current setup. But you still need to fix the outstanding issues first. You need a baseline to measure from.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    I have never been a fan of the all in one box, ask yourself this, do you still have your swiss army knife? Also applying the access lists to the inbound traffic of the router interface reduces processor overhead. Using a seperate firewall/VPN device is probally your best bet and most cost effective.
    0
     
    LVL 13

    Assisted Solution

    by:Dr-IP
    A lot of Cisco’s older router designs like the 1700 series where not designed with a lot the things they added in mind. If you where running just normal data through that 1721, it would probably be fine, but you are running VOIP though it, which generates a very high packet rates because of all the small packets VOIP uses. Which puts a considerably higher load on the router, especially since it has to compare every one of those many little packets to the access lists; QOS them, and then deal with the encryption for the VPN.

    Now there is one option for the 1721’s I haven’t mentioned yet as I wasn’t sure it was available, a hardware VPN encryption module, P/N MOD1700-VPN.  That would offload the encryption tasks from the processor freeing it up to do other tasks, but it would be even better to have a 2650 with it’s more powerful process, and a VPN accelerator module.
    Also the 2650 even without a VPN accelerator might still out perform a 1721 series router since the processor in it is a lot faster than a 1721, A 80 MHz MPC 860P RISC processor verses a 48 Mhz MPC 860T RISC  processor for the 1721. Also since it’s a P version it has larger catches and other performance enhancing features, so the performance boost is even greater than the MHz improvement would imply.
    An even better soulution would be to go with a X8XX series router, which are Cisco’s new generation of routers designed from the ground up for the kinds of task you are trying to get those 1721’s to do. If you had just gone up one step to the 1841 you probably wouldn’t have the problems you are having, since all X8XX series routers have built in encryption coprocessors. It’s what I would have suggested to someone asking for a new 1721, but in knowing what you are doing with it, I would have recommended stepping up to a 2800 series router instead just to be sure. Anyway a more powerful routing solution is probably in order here.  




    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    DR-IP,

    Tango already has the VPN module, how did you miss that in the mile long post (LOL).
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    Hey considering how long this post have become, I should get a medal for finding anything. Anyway, so he has hardware encryption and it still is overloaded, I found the config, and checked to see if the VPN module required any special configuration to enable it, but it looks like other than having an IOS that supports it, and it looks like he does. One thing I noted, is he doesn’t have CEF enabled “ip cef”, I have mentioned this already, but I don’t know if he has enabled it, as it does reduce the load on the processor.  
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    In short was does ip cef do?
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    0
     

    Author Comment

    by:tangofniro
    I enabled CEF after the suggestions I just didn't add another config back on this page due to the increasing size. ha


    Are we at the point where there is nothing else to look to except the router being overloaded?
    0
     

    Author Comment

    by:tangofniro
    I know we may be beating a dead horse but I never posted the remote sides config.

    Are you two positve that there would be nothing in my total config that would help/hurt ?
    0
     
    LVL 13

    Expert Comment

    by:Dr-IP
    The other side should be a mirror image of the one you posted, other than the IP addresses. So unless something else is different than that, posting the other config would probably pointless.
    0
     

    Author Comment

    by:tangofniro
    Whoa  whoa my bad


    I need help in giving Joel points.

    He helped all along and I can't leave him out.


    Please help admins.


    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    You will need to put in a request at the support page.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Were you able resolve the problem with the pbx looking to register the phones?
    0
     

    Author Comment

    by:tangofniro
    Hey Joel,

    The PBX kept seeing the phones being registered.

    It doesn't help that this phone vendor is on his first Samsung VOIP.

    He doesn't know most answers to my questions.
    0
     
    LVL 12

    Expert Comment

    by:Joel_Sisko
    Hmmm.....delete the phones from the pbx system. Add them back, check to make sure that all the port settings are correct on phone as they are on a known good working phone.
    0
     

    Author Comment

    by:tangofniro
    I will try that tomorrow
    0

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone. Privacy Policy Terms of Use

    Featured Post

    How your wiki can always stay up-to-date

    Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
    - Increase transparency
    - Onboard new hires faster
    - Access from mobile/offline

    Suggested Solutions

    Title # Comments Views Activity
    Cisco UC520 Call Transfer Issue 7 33
    Downgrading shoretel phones 3 39
    VLANS Quagmire 1 48
    facetime for android 4 55
    Although VoiceOver IP has been around for a while, internet connections have only recently become fast enough to provide good call quality. Now, VoIP has become a real option for businesses looking at ways to improve their business model. In this ar…
    As companies replace their old PBX phone systems with Unified IP Communications, many are finding out that legacy applications such as fax do not work well with VoIP. Fortunately, Cloud Faxing provides a cost-effective alternative that works over an…
    Want to pick and choose which updates you receive? Feel free to check out this quick video on how to manage your email notifications.
    In this sixth video of the Xpdf series, we discuss and demonstrate the PDFtoPNG utility, which converts a multi-page PDF file to separate color, grayscale, or monochrome PNG files, creating one PNG file for each page in the PDF. It does this via a c…

    877 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    19 Experts available now in Live!

    Get 1:1 Help Now