And you should post your configs also...
Main Topics
Browse All Topics
Loading Advertisement...
Question
CISCO LLQ trouble??
What would cause me to have drops in my policy?
If I need to post full configs I can.
I have 2 Cisco 1721s connected via t1s to the internet. I have Samsung ip phones.
After a day of use I have around 70-80 total drops
Here is my what sh policy-map int gives me.
Service-policy output: llq
Class-map: NAPLESVOIP (match-all)
264 packets, 152592 bytes
5 minute offered rate 6000 bps, drop rate 1000 bps
Match: access-group 100
Queueing
Strict Priority
Output Queue: Conversation 264
Bandwidth 500 (kbps) Burst 12500 (Bytes)
(pkts matched/bytes matched) 262/151436
(total drops/bytes drops) 4/2312
Class-map: DATA (match-all)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: access-group 102
Queueing
Output Queue: Conversation 265
Bandwidth 225 (kbps) Max Threshold 64 (packets)
(pkts matched/bytes matched) 0/0
(depth/total drops/no-buffer drops) 0/0/0
Class-map: class-default (match-any)
21254 packets, 2494063 bytes
5 minute offered rate 253000 bps, drop rate 2000 bps
Match: any
Queueing
Flow Based Fair Queueing
Maximum Number of Hashed Queues 256
(total queued/total drops/no-buffer drops) 0/372/0
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Subscribe now for full access to Experts Exchange and get
Instant Access to this Solution
- Plus...
- 30 Day FREE access, no risk, no obligation
- Collaborate with the world's top tech experts
- Unlimited access to our exclusive solution database
- Never be left without tech help again
Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.
- "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
- "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
- "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.
See what Experts Exchange can do for you.
Got a question?
We've got the answer.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
Need individual assistance?
Our experts are ready to help.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Want to learn from the best?
Read articles from industry experts.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Working on a long term project?
Store your work and research.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
What Makes Experts Exchange Unique?
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Trusted by the world's most respected brands.
Faithfully serving IT professionals since 1996.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Free Tech Articles
-
WARNING: 5 Reasons why you should NEVER fix a computer for free.
It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
-
SCCM OSD Basic troubleshooting
SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
-
Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
-
Create a Win7 Gadget
This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
-
Outlook continually prompting for username and password
There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
-
Backup Exchange 2010 Information Store using Windows Backup
There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...
Cloud Class Webinars
- Avoiding Bugs in Microsoft Access
Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
- Top 10 Best New Features in Visio 2010
Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
- IT Consultant Business Secrets Revealed
Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
- Disaster Recovery and Business Continuity
Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
- Organize Your Visio Diagrams with Containers and Lists
Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
- How to Us Objects, Properties, Events and Methods in Microsoft Access
Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...
Join the Community
Give a Little. Get a Lot.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
Answers
7 phones on the remote end 25 (3ip 22regular)phones at the main office
At the most which would never happen all 7 phones at the remote side could be calling the main office and talking to 7 phones there.
We also are going through a vpn
Here is the config from the remote router
RPSTRAND#sh run
Building configuration...
Current configuration : 4943 bytes
!
version 12.3
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname RPSTRAND
!
boot-start-marker
boot-end-marker
!
logging buffered 51200 warnings
enable password 7 030752180500
!
username cisc0 privilege 15 password 7 0505031838435C02
username cisco password 7 110A1016141D
no aaa new-model
ip subnet-zero
no ip source-route
!
!
ip dhcp excluded-address 192.168.3.1 192.168.3.99
ip dhcp excluded-address 192.168.3.200 192.168.3.254
ip dhcp excluded-address 172.168.1.1 172.168.1.100
ip dhcp excluded-address 172.168.1.200 172.168.1.254
!
ip dhcp pool MainScope
network 192.168.3.0 255.255.255.0
domain-name rpprop
dns-server 4.2.2.1
default-router 192.168.3.1
!
ip dhcp pool Strand
network 172.168.1.0 255.255.255.0
domain-name rpprop
dns-server 4.2.2.1
default-router 172.168.1.1
!
!
ip domain name yourdomain.com
no ip bootp server
no ip cef
ip audit notify log
ip audit po max-events 100
no ftp-server write-enable
no scripting tcl init
no scripting tcl encdir
!
!
!
!
!
crypto isakmp policy 10
hash md5
authentication pre-share
crypto isakmp key 1234 address 199.x.x.x
crypto isakmp keepalive 20 10
!
!
crypto ipsec transform-set rpvpn esp-des esp-md5-hmac
!
crypto map rpvpn 10 ipsec-isakmp
set peer 199.x.x.x
set peer 216.x.x.x
set transform-set rpvpn
match address 101
!
!
!
class-map match-all NAPLESVOIP
match access-group 100
class-map match-all DATA
match access-group 102
!
!
policy-map llq
class NAPLESVOIP
priority 500
class DATA
bandwidth 225
class class-default
fair-queue
!
!
!
interface FastEthernet0
description $FW_INSIDE$$ETH-LAN$
ip address 172.168.1.1 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip route-cache flow
ip tcp adjust-mss 1380
speed auto
full-duplex
no cdp enable
!
interface FastEthernet1
no ip address
no cdp enable
!
interface FastEthernet2
no ip address
shutdown
no cdp enable
!
interface FastEthernet3
no ip address
shutdown
no cdp enable
!
interface FastEthernet4
no ip address
shutdown
no cdp enable
!
interface Serial0
description $FW_OUTSIDE$
ip address 199.x.x.x 255.x.x.x
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
service-policy output llq
encapsulation ppp
ip route-cache flow
service-module t1 timeslots 8-24
no cdp enable
crypto map rpvpn
!
interface Vlan1
no ip address
!
ip nat inside source route-map nonat interface Serial0 overload
ip classless
ip route 0.0.0.0 0.0.0.0 199.x.x.x
ip http server
ip http authentication local
ip http secure-server
!
!
!
access-list 100 permit udp any any range 16384 32000
access-list 100 permit tcp any any eq 1719
access-list 100 permit tcp any any eq 1720
access-list 100 permit tcp any any eq 6100
access-list 100 permit tcp any any range 1024 4999
access-list 100 permit udp any any eq 6000
access-list 100 permit udp any any range 1024 4999
access-list 100 permit udp any any eq 5060
access-list 100 permit udp any any range 30000 30030
access-list 100 permit udp any any range 9000 9001
access-list 101 permit ip 172.168.1.0 0.0.0.255 192.168.0.0 0.0.0.255
access-list 101 deny ip 172.168.1.0 0.0.0.255 any
access-list 102 permit tcp any any range 5000 5110
access-list 110 deny ip 172.168.1.0 0.0.0.255 192.168.0.0 0.0.0.255
access-list 110 permit ip 172.168.1.0 0.0.0.255 any
no cdp run
!
route-map nonat permit 10
match ip address 110
set ip next-hop 1.1.1.2
!
!
control-plane
!
banner login ^C
Well I disagree on the VoIP not meant for the Internet. Quickly look up the TLA for VoIP (Voice over Internet Protocol).
First before you give up on implementing a VoIP solution. What are ping times between the two 1700's? If you are under <100ms between sites, VoIP has great potential. The human ear detects anything over 180ms as starting to be choppy and distorted.
I will agree on the VPN adds overhead and could pose a problem, but I have implemented many VoIP solutions using a VPN between sites. But we usually use a separate device for the VPN solution. Part of your problem is that your little Cisco router is being asked to do to much, Route, IPSEC, Firewall. The 1720 was not built for all of this at the same time. You might want to go to the 1760 series router and use the VPN module to offload some of the work.
But in any case, if your ping times are good then using VoIP is reasonable.
Also dropped packets also could indicate the router is being overloaded. Also you want to set up some form of QOS on your local network; this might help reduce delay times. One question, is the T1 a point-to-point? If not point to point, are both T1's terminated to the same ISP/Carrier network? If the drop packets are only the voice data packets, then you could have a application issue or even the Samsung phone causing the problem.
As a ping test I would recommend to ping one computer at site A to another computer over T1's at site B. To ping, go to a desktop on the same LAN: (make sure your computer/router/firewall is not disabled to respond to a ping request)
Start > Run> type cmd> at the prompt type ping -t -l 64000 X.X.X.X (make sure that you are pinging the right server address X.X.X.X)
Let the command run. The command sends 64K byte packets until you hit Cntrl C. The larger packet tests the Ethernet connection a little bit better than the standard ping command of 5 bytes. Let it run for a bit and see what is going on. Also, check the routers interface's for CRC errors, if you are getting a high amount you could have connectivity problems.
Run the following at the cisco CLI to look at the crc errors:
show interfaces ethernet
show interfaces serial
Let me know how the tests go and we can move on from that point.
Okay,
No CRCs on main site interfaces
remote side interfaces which I have the configs above are , Fast0 0 CRCs, Serial0 68 CRCs, router was rebooted today.
ping times between the sites show mostly 30ms but every once in a while it will jump up very high around 200ms then it will go back down.
These T1s are point to point, to the internet not leased line, same carrier, they say it my router config, I am trying to find out if it is. Is there any command to show my router being overworked?
I could post the main site config if that would help.
The phones work great for about 90% of the time but for the other 10% they are unusable. There is no specific time or day either.
Post the config so we dont go DOHHH! later on. The CRC's on the serial are not unusual is that was the amount for the entire day? What you could do is reset the crc's count to zero and make a boat load of VoIP calls (try to do this when the office is quite, try to reduce the amount of Internet browsing traffic), see what the crc count is, then clear the counters, do not make any VoIP calls for a while, check crc count, if the same then most likely not the issue.
What version of IOS are you using and yes there are commands for performance monitoring.
here is the main sites config, where thephone system resides.
Thanks for your help.
no aaa new-model
ip subnet-zero
no ip source-route
!
!
ip dhcp excluded-address 192.168.0.1 192.168.0.99
ip dhcp excluded-address 192.168.0.200 192.168.0.254
!
ip dhcp pool MainScope
network 192.168.0.0 255.255.255.0
domain-name
dns-server 4.2.2.1
default-router 192.168.0.1
!
!
ip domain name rp-prop.com
ip name-server 4.2.2.1
ip name-server 4.2.2.2
no ip bootp server
no ip cef
ip audit notify log
ip audit po max-events 100
ip ssh time-out 60
ip ssh authentication-retries 2
no ftp-server write-enable
no scripting tcl init
no scripting tcl encdir
!
!
!
!
!
crypto isakmp policy 10
hash md5
authentication pre-share
crypto isakmp key 1234 address 1x.x.x.x
crypto isakmp keepalive 20 10
!
!
crypto ipsec transform-set rpvpn esp-des esp-md5-hmac
!
crypto map rpvpn 10 ipsec-isakmp
set peer 1x.x.x.x
set transform-set rpvpn
match address 101
!
!
!
class-map match-all STRANDDATA
match access-group 102
class-map match-all STRANDVOIP
match access-group 100
!
!
policy-map LLQ
class STRANDVOIP
priority 450
class STRANDDATA
bandwidth 650
class class-default
fair-queue
!
!
!
interface Loopback0
ip address 1.1.1.1 255.255.255.0
!
interface FastEthernet0
description $FW_INSIDE$$ETH-LAN$
ip address 192.168.0.1 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
no ip route-cache
ip tcp adjust-mss 1380
ip policy route-map nonat1
no ip mroute-cache
speed auto
full-duplex
priority-group 5
no cdp enable
!
interface FastEthernet1
switchport mode trunk
no ip address
no cdp enable
!
interface FastEthernet2
no ip address
shutdown
no cdp enable
!
interface FastEthernet3
no ip address
shutdown
no cdp enable
!
interface FastEthernet4
no ip address
shutdown
no cdp enable
!
interface Serial0
description USLEC T1
ip address 1x.x.x.x 255.255.255.x
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
service-policy output LLQ
encapsulation ppp
no ip route-cache
no ip mroute-cache
service-module t1 timeslots 1-24
no cdp enable
crypto map rpvpn
!
interface Vlan1
no ip address
no ip redirects
no ip unreachables
no ip proxy-arp
ip route-cache flow
!
ip nat inside source route-map nonat interface Serial0 overload
ip nat inside source static 192.168.0.30 2x.x.x.x extendable
ip nat inside source static 192.168.0.31 2x.x.x.x extendable
ip nat inside source static 192.168.0.205 2x.x.x.x extendable
ip nat inside source static 192.168.0.2 2x.x.x.x extendable
ip classless
ip route 0.0.0.0 0.0.0.0 199.x.x.x
no ip http server
ip http authentication local
ip http secure-server
!
!
!
logging trap debugging
access-list 100 permit udp any any range 16384 32000
access-list 100 permit tcp any any eq 1719
access-list 100 permit tcp any any eq 1720
access-list 100 permit tcp any any eq 6100
access-list 100 permit tcp any any range 1024 4999
access-list 100 permit udp any any eq 6000
access-list 100 permit udp any any range 1024 4999
access-list 100 permit udp any any eq 5060
access-list 100 permit udp any any range 30000 30030
access-list 100 permit udp any any range 9000 9001
access-list 101 permit ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255
access-list 101 deny ip 192.168.0.0 0.0.0.255 any
access-list 102 permit tcp any any range 5000 5110
access-list 105 deny icmp any any echo
access-list 105 permit ip any any
access-list 110 deny ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255
access-list 110 deny ip host 192.168.0.205 any
access-list 110 deny ip host 192.168.0.30 any
access-list 110 deny ip host 192.168.0.31 any
access-list 110 deny ip host 192.168.0.2 any
access-list 110 permit ip 192.168.0.0 0.0.0.255 any
access-list 120 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255
access-list 123 permit ip host 192.168.0.205 172.168.1.0 0.0.0.255
access-list 123 permit ip host 192.168.0.30 172.168.1.0 0.0.0.255
access-list 123 permit ip host 192.168.0.31 172.168.1.0 0.0.0.255
access-list 123 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255
no cdp run
!
route-map nonat1 permit 10
match ip address 123
set ip next-hop 1.1.1.2
!
route-map nonat permit 10
match ip address 110
!
!
control-plane
!
banner login ^CCAuthorized access only!
Disconnect IMMEDIATELY if you are not an authorized user!^C
!
line con 0
password 7 0833184A1A18541B
login
transport output telnet
line aux 0
login local
transport output telnet
line vty 0 4
privilege level 15
password 7 06145B255F4F5815
login
transport input telnet ssh
line vty 5 15
privilege level 15
login local
transport input telnet ssh
!
scheduler allocate 4000 1000
scheduler interval 500
!
end
RPNAPLES#
RPNAPLES#sh version
Cisco IOS Software, C1700 Software (C1700-K9O3SY7-M), Version 12.3(2)XE, EARLY D
EPLOYMENT RELEASE SOFTWARE (fc1)
Synched to technology version 12.3(3.5)T
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2003 by Cisco Systems, Inc.
Compiled Tue 18-Nov-03 23:26 by ealyon
ROM: System Bootstrap, Version 12.2(7r)XM2, RELEASE SOFTWARE (fc1)
ROM:
RPNAPLES uptime is 1 week, 13 hours, 45 minutes
System returned to ROM by power-on
System restarted at 08:58:52 America/Chicago Mon Apr 8 2002
System image file is "flash:c1700-k9o3sy7-mz.12
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/e
If you require further assistance please contact us by sending email to
export@cisco.com.
Cisco 1721 (MPC860P) processor (revision 0x300) with 84983K/13321K bytes of memo
ry.
Processor board ID FOC08172HWP (3205001181), with hardware revision 0000
MPC860P processor: part number 5, mask 2
1 Ethernet interface
5 FastEthernet interfaces
1 Serial interface
1 Virtual Private Network (VPN) Module
WIC T1-DSU
32K bytes of NVRAM.
32768K bytes of processor board System flash (Read/Write)
Configuration register is 0x2102
\from remote site\\
RPSTRAND#sh version
Cisco IOS Software, C1700 Software (C1700-K9O3SY7-M), Version 12.3(2)XE, EARLY D
EPLOYMENT RELEASE SOFTWARE (fc1)
Synched to technology version 12.3(3.5)T
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2003 by Cisco Systems, Inc.
Compiled Tue 18-Nov-03 23:26 by ealyon
ROM: System Bootstrap, Version 12.2(7r)XM2, RELEASE SOFTWARE (fc1)
ROM:
RPSTRAND uptime is 5 hours, 31 minutes
System returned to ROM by power-on
System image file is "flash:c1700-k9o3sy7-mz.12
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/e
If you require further assistance please contact us by sending email to
export@cisco.com.
Cisco 1721 (MPC860P) processor (revision 0x300) with 84983K/13321K bytes of memo
ry.
Processor board ID FOC08172J3K (2799143465), with hardware revision 0000
MPC860P processor: part number 5, mask 2
1 Ethernet interface
5 FastEthernet interfaces
1 Serial interface
1 Virtual Private Network (VPN) Module
WIC T1-DSU
32K bytes of NVRAM.
32768K bytes of processor board System flash (Read/Write)
Configuration register is 0x2102
Qos on the Crypto will only help streamline things, so its a good thing.
Question on the drops, are you actually dropping calls mid way thru the conversations?
In terms of static, could you clarify? Voice signals having static? High CRC's?
The good news it will get thru this eventually, Someone asked Thomas Edison that why he would try to keep inventing something even if it took 50,000 tries? His response was that he eleiminated 50,000 ways of not doing it.
Just a thought if the calls are being dropped during mid conversation, we might be running into a timeout issue of the application over the VPN, problem might be outside of the network. Do you have any tech specs on the Samsung IP Phones?
hit or miss. Friday was bad but I think today was better.
Calls don't drop out completly, Static is the main problem , can't hear the other end, and speakerphone useless.
I do have tech specs on the phone but it's on cd. PDF files I can send them to you if you want but there is a lot of junk to wade through.
Thanks you for all of your help and staying on this problem.
On the pdf, if you know of a FTP site and load them up there? Will galdly review them.
If you are actually getting static there are a few things we need to look at:
First if VAD is being used and also echo-cancellation. VAD is more than likely being introdiced by the phone or Samsung system. Need to find that and turn it off.
Also, just another thing popped into my head, we can test the link betwen the sites using, lets say two computers running a free softphone, such sjlabs or x-ten lite, and that have speakers and microphones. If you establish a call between the two (using same VoIP protocols) and no hissing or drop offs, we can lean more towards the phone side of things.
It could be, it would jump if there is other traffic at the same time with a higher priority. If it happens all the time (being jumpy) it might be a configuration issue? You can checkto see this using the ping command from earlier on in our dialog.
Few things to follow up on:
Are you still getting static on voice calls?
Do you have a spare computer with a 10/100 NIC and a hub?
It could be here a re few steps to find the ghost:
1. Try to disable the use of VAD and echo-cancellation on the VoIP side of things
2. To monitor the Internet , there is a program I have used in the past that helps to do this it can be found at:
http://www.serverscheck.co
I believe you can set up to 3 free monitors for each system setup. Basically I would set up a ping command and monitor that. Download the program and take a look, it has a alot of features.
3. Setup a network sniffer at each site on each interface. You could use Ethereal or some other free products.
The ideal situation would be two computers, one on each site, each connected to between the network and router via a hub (hub allows to see all packets) then set up Ethereal and serverscheck on each. Also you could enable logging on the Cisco routers to log to the same system running the sniffer and monitor. Let it run for a few days, log times when all is running well, then log times when things are bad. Go back to the data captured and look up logged times, see if something is out of the ordinary.
Kindest regards
Joel_Sisko
Also, part of th problem could be to much jitter, using Ehtereal you should be able to determine this, might not be real clear cut. There are some products that allow for this one is from Wildpackets called Etherpeek VX, another is Observer 10, but they cost a few hundred dollars.
Just rememebered try using this site:
http://www.testyourvoip.co
4.3 is good, what type of traffic was going across the network at that time? Need to run this test a variosu times and see if it changes, also run the test when you know the problem you have been having arises.
Is it possible to set up a sniffer and log files from the Cisco router? Since we are looking for somewhat of the needle in the haystack we need to baseline the network and move forward from that point.
Additional free program that may help out:
http://www.bb4.org/feature
One more I just rememred that is affordable, gui based and quite a few features:
http://www.colasoft.com/pr
Okay, I had the problem today.
Voice quality went out the window. I ran some speed tests and from the various places I ran it from they all showed very slow speed like 300-400k.
I spoke with the provider and they said they did not see anything wrong at the time. It lasted for about 25minutes and they gone.
Is there anything I can do to check what is going on? Can you think of anything that might cause that?
Kind of stuck...
I am assuming the speed went back up after the 25 minutes as well as the voice quality? Did you happen to check with the www.testyourvoip.com tool? Though your speed/ping might be okay the packet jitter might be going off the charts. Jitter is a result of packets showing up at different times to put it simply.
Voice is very time sensitive, thought also just popped into my me head, what do you have your MTU set at on your routers? Setting the MTU higher might help, next time you are having the issue try adjusting the MTU.
Also you really need to baseline the system as per previous posts, also use a traceroute tool, see how many hops it takes and which routers the packet touches during normal operations, then do this again when the speed goes down. It could be also that soem traffic is being redirected thru a different route for a given time period for maintence and such.
Kindest regards,
Joel _Sisko
Hello Joel,
I checked with testyourvoip as it was happening and I got a terrible score of 2.2. There was no jitter going out but coming in from Boston there was.
I will try a few of your suggestions from earlier. Is there a command that will show me every active connection/ports on the router.
The mtu is set to 1600 on my serial int.
Going out to the internet while this was happening I was getting a good amount of packet loss to my providers gateway. And time was very high.
After the 25 or so minutes the speed was back along with voice quality.
Okay at least we can start ruling out ithe nternal stuff and focus on the carrier.
Need to baseline the speed from site a to site b, need to baseline the routes it also takes.
If it is actually the carrier, then monitoring the Cisco interface will really not help, other than let us know something is going wrong. We need to find out where in the carrier network things are going wrong. By the way who is your provider?
The route is only 3 hops
Had alot of trouble today(static, packet loss). I don't know my provider USLEC says they are not seeing any problems on their side. The do see a lott of usage on our side. I can't see what is causing the problem. Testyourvoip was pretty low also. Great thing was at 5 oclock the problem went away. Everything now is great.
I don't know if I would rule out it being an internal problem. That would be the way I am leaning right now.
Can you set up a system to monitor the traffic going to the router? You really need to baseline the system, without it this post will be a mile long (LOL), can use the points but not that bad (LOL).
Are you sharing files across the link? Any kind of Database upadtes? Someone backing up to network drive? Are you using Active Directory? What about email? Roaming profiles?
http://www.colasoft.com/pr
This will be the best tool to baseline the system.
i downloaded capsa today, nothing really jumped out at me. will buy it most likely.
i have each side going out over the internet on each link. Unless you can see something different in my config.
each workstation has it's side router as the default gateway.
I have 2 server 1 domain the other a terminal server/backup domain controller
I use Symantec Antivirus Server.
Anything I should be looking for in particular that you are expecting Cola to see?
thanks
Always possible, need to log the cpu performance, remember early on I said that the router is being asked to do alot (think my first post). But since your provider is seeing a considerable amount of traffic which you can not account for, you should still baseline the network to see what is causing the traffic.
Dropping the MTU size will actually add to processor overhead but will enable the packet to transverse the network/Internet more efficiently.
I ran sh proc and sh proc cpu and here is what I got:
RPNAPLES#sh proc
CPU utilization for five seconds: 32%/30%; one minute: 30%; five minutes: 29%
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
1 Cwe 8029A2D0 32 8 4000 5548/6000 0 Chunk Manager
2 Csp 80280AB0 28 2245 12 2744/3000 0 Load Meter
3 M* 0 28 19 147310360/12000 6 Virtual Exec
4 Mwe 8053C650 0 94 0 5768/6000 0 DHCPD Timer
5 Lst 80299ACC 11444 1346 8502 5752/6000 0 Check heaps
6 Cwe 8029E304 32 39 820 5536/6000 0 Pool Manager
7 Mst 801A58E4 0 2 0 5708/6000 0 Timers
8 Mwe 80096BB8 0 2 0 5716/6000 0 Serial Backgroun
9 Mwe 80178B38 0 2 0 5704/6000 0 AAA high-capacit
10 Lwe 8034FD08 948 1278 741 5008/6000 0 ARP Input
11 Mwe 80371168 0 3 0 5708/6000 0 DDR Timers
12 Mwe 80675650 32 2805 11 5768/6000 0 HC Counter Timer
13 Lwe 80978F34 12 2 6000 5624/6000 0 Entity MIB API
14 Mwe 80BE83B8 0 2 0 5712/6000 0 ATM Idle Timer
15 Mwe 8009C57C 0 1 0 5788/6000 0 SERIAL A'detect
16 Msp 801E85FC 12 11208 1 5744/6000 0 GraphIt
17 Mwe 8038C274 0 2 011712/12000 0 Dialer event
18 Mwe 806196F8 0 2 011728/12000 0 XML Proxy Client
19 Cwe 8026C730 0 1 0 5804/6000 0 Critical Bkgnd
20 Mwe 8021AC98 296 3265 9010392/12000 0 Net Background
21 Lwe 80196DF8 0 19 011356/12000 0 Logger
22 Mwe 801BB920 40 11205 3 5736/6000 0 TTY Background
23 Msp 8022BC78 132 11217 11 7464/9000 0 Per-Second Jobs
24 Lwe 80311EC0 8 2 4000 1864/3000 0 IPM_C1700_CLOCK
25 Mwe 812F69B4 0 2 0 5784/6000 0 AggMgr Process
26 Hwe 81327A00 4 2 2000 5264/6000 0 ESWPPM
27 Hwe 8022B7A4 0 1 0 5788/6000 0 Net Input
28 Msp 80220A90 84 2249 37 5740/6000 0 Compute load avg
29 Msp 8022BCEC 8352 192 43500 5752/6000 0 Per-minute Jobs
30 Mwe 8009F05C 0 3 0 5700/6000 0 Service-module a
31 Mwe 813322AC 0 1 0 5792/6000 0 Switch Link Moni
32 Mwe 8025067C 4 2 2000 5712/6000 0 AAA Server
33 Mwe 80252B48 0 1 0 5784/6000 0 AAA ACCT Proc
34 Mwe 80252C2C 0 1 0 5772/6000 0 ACCT Periodic Pr
35 Lwe 8017D6A4 0 1 0 5788/6000 0 AAA_SERVER_DEADT
36 Mwe 80304C34 0 2 0 5724/6000 0 AAA Dictionary R
37 Mwe 8044EEF4 1653100 1349941 1224 9952/12000 0 IP Input
38 Mwe 80473998 0 1 0 5768/6000 0 ICMP event handl
39 Mwe 81159038 0 8 0 5788/6000 0 CRYPTO IKMP IPC
40 Mwe 8085DB38 0 1 011788/12000 0 SSS Manager
41 Mwe 80861464 4 1501 211764/12000 0 SSS Test Client
42 Mwe 8086AE94 0 1 0 5800/6000 0 SSS Feature Mana
43 Mwe 8086AF28 780 43912 17 5768/6000 0 SSS Feature Time
44 Mwe 8027AE9C 0 4 011732/12000 0 PPP Hooks
45 Lwe 80A546B4 0 1 0 5396/6000 0 X.25 Encaps Mana
46 Mwe 80DFED4C 0 1 011748/12000 0 VPDN call manage
47 Mwe 80E7AB9C 0 1 011780/12000 0 L2X Data Daemon
48 Mwe 80E42F6C 0 1 011756/12000 0 L2X Socket proce
49 Mwe 80E0FD10 0 1 011784/12000 0 L2X SSS manager
50 Mwe 80E1CB74 0 2 011712/12000 0 L2TP mgmt daemon
51 Mwe 8105F430 0 1 0 5776/6000 0 AC Mgr
52 Mwe 81239BE4 0 2 011728/12000 0 KRB5 AAA
53 Mwe 812E6C64 0 2 0 5540/6000 0 DTP Protocol
54 Mwe 8027AE9C 4 3 133310988/12000 0 PPP IP Route
55 Mwe 8027AE9C 0 4 011448/12000 0 PPP IPCP
56 Mwe 80425860 32 19 1684 4436/6000 0 DHCPD Receive
57 Mwe 80551360 524 215 2437 8224/9000 0 IP Background
58 Mwe 805574F4 112 194 577 8736/9000 0 IP RIB Update
59 Mst 8043214C 0 36 011044/12000 0 TCP Timer
60 Lwe 80437114 8 3 266610940/12000 0 TCP Protocols
61 Mwe 8049E14C 0 1 0 5804/6000 0 RARP Input
62 Hwe 80533708 0 1 0 5776/6000 0 Socket Timers
63 Mwe 805090E4 4 40 100 8452/9000 0 HTTP CORE
64 Lsi 805CB69C 48 188 255 5304/6000 0 IP Cache Ager
65 Hwe 80A6DB7C 0 1 0 5780/6000 0 PAD InCall
66 Mwe 80A27320 0 2 011708/12000 0 X.25 Background
67 Mwe 8086CC34 0 2 0 5708/6000 0 PPP SSS
68 Mwe 80908750 16 189 84 8688/9000 0 Adj Manager
69 Mwe 80AA1614 72 21938 3 5700/6000 0 IP NAT Ager
70 Mwe 8027AE9C 4 2 2000 5728/6000 0 PPP Bind
71 Mwe 8096E9F4 0 1 0 5756/6000 0 SNMP Timers
72 Mwe 80C7B920 0 1 0 5796/6000 0 Inspect Timer
73 Msi 8054B8C4 0 3196 0 4980/6000 0 DHCPD Database
74 Mwe 80CA5BF8 0 2 0 5612/6000 0 URL filter proc
75 Mwe 80CBF034 0 38 0 5796/6000 0 Authentication P
76 Mwe 80CC8EA0 0 1 0 5776/6000 0 Auth-proxy AAA B
77 Mwe 80CC9C64 0 1 0 5796/6000 0 IDS Timer
78 Mwe 80DE114C 0 1 023780/24000 0 COPS
79 Mwe 80DEFDF0 0 2 0 5720/6000 0 Dialer Forwarder
80 Mwe 80E06E3C 8 2249 311776/12000 0 L2F management d
81 Mwe 80E357FC 4 1 400011540/12000 0 PPTP Mgmt
82 Mwe 80E7F4DC 0 2 011728/12000 0 PPTP Data
83 Hwe 810EDD9C 25680 30471 842 5064/6000 0 Crypto HW Proc
84 Lwe 812DB7A4 0 1 0 5796/6000 0 XSM_EVENT_ENGINE
85 Lsi 812D93DC 8 1124 711824/12000 0 XSM_ENQUEUER
86 Lsi 812DC4D4 4 1124 311828/12000 0 XSM Historian
87 Mwe 8025D51C 0 2 0 5728/6000 0 LOCAL AAA
88 Mwe 8025F5D4 0 2 0 5728/6000 0 ENABLE AAA
89 Mwe 8025F9C0 0 2 0 5732/6000 0 LINE AAA
90 Mwe 803C8D1C 0 2 0 5604/6000 0 TPLUS
91 Lwe 808F6E08 360 14373 25 4368/6000 0 CEF process
92 Mwe 810DB840 4 2 2000 5724/6000 0 Crypto Support
93 Mwe 81387464 0 1 0 5800/6000 0 EM Background Pr
94 Mwe 806D6DA0 20 452 44 5668/6000 0 CRM_CALL_UPDATE_
95 Mwe 810D5BB4 0 1 011804/12000 0 Encrypt Proc
96 Mwe 810D67A4 14128 117 120752 6696/8000 0 Key Proc
97 Mwe 811B0058 24 4 6000 7000/8000 0 Crypto CA
98 Mwe 811EB1D4 0 1 0 7812/8000 0 Crypto SSL
99 Mwe 81158B3C 20 40 50020432/24000 0 Crypto ACL
100 Mwe 810E00DC 0 1 0 5792/6000 0 CRYPTO QoS proce
101 Mwe 81151F4C 12 14 85711332/12000 0 Crypto Delete Ma
102 Mwe 8111F364 104 109 954 6180/12000 0 Crypto IKMP
103 Mwe 8111445C 1212 587 2064 9744/12000 0 IPSEC key engine
104 Mwe 81114F10 0 1 0 5716/6000 0 IPSEC manual key
105 Mwe 80171B4C 0 2 0 5708/6000 0 AAA SEND STOP EV
106 Mwe 80849E8C 0 1 0 5816/6000 0 Syslog Traps
107 Lwe 812C945C 0 2 0 5648/6000 0 IpSecMibTopN
108 Mwe 81324B6C 32 1216 26 5752/6000 0 PM Callback
109 Mwe 80AF1508 38892 2766014 14 5584/6000 0 SAA Event Proces
110 Mwe 80E45B78 0 1 0 5784/6000 0 VPDN Scal
111 Mwe 81337730 4 2 2000 3912/6000 0 VLAN Manager
112 Lsp 8132E680 18864 44308 425 5732/6000 0 COLLECT STAT COU
113 Mwe 805D0D8C 0 1 011796/12000 0 TCP Driver
114 Lwe 80435D60 0 1 0 5792/6000 0 TCP Listener
115 Mwe 80AD3A54 0 1 0 5772/6000 0 IP NAT WLAN
116 Mwe 81244984 40 169 236 4428/6000 0 SSH Event handle
117 Mwe 8099DAD0 2664 349651 711696/12000 0 PPP manager
118 Mwe 8027AE9C 2508 349661 711244/12000 0 PPP Events
119 Hwe 809D5CA8 12 11239 1 5724/6000 0 Multilink PPP
120 Mwe 809D568C 0 2 0 5712/6000 0 Multilink event
121 Mwe 80876D7C 0 2 0 5732/6000 0 IP Flow Backgrou
122 Mwe 810FBD70 48 249 192 4456/6000 0 Crypto Hardware
123 Lwe 808F6638 16 367 43 5664/6000 0 CEF Scanner
RPNAPLES#sh proc cpu
CPU utilization for five seconds: 41%/39%; one minute: 34%; five minutes: 30%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 32 8 4000 0.00% 0.00% 0.00% 0 Chunk Manager
2 28 2252 12 0.00% 0.00% 0.00% 0 Load Meter
3 596 286 2083 1.19% 0.54% 0.12% 6 Virtual Exec
4 0 94 0 0.00% 0.00% 0.00% 0 DHCPD Timer
5 11444 1348 8489 0.00% 0.05% 0.05% 0 Check heaps
6 32 39 820 0.00% 0.00% 0.00% 0 Pool Manager
7 0 2 0 0.00% 0.00% 0.00% 0 Timers
8 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
9 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit
10 948 1280 740 0.00% 0.00% 0.00% 0 ARP Input
11 0 3 0 0.00% 0.00% 0.00% 0 DDR Timers
12 32 2815 11 0.00% 0.00% 0.00% 0 HC Counter Timer
13 12 2 6000 0.00% 0.00% 0.00% 0 Entity MIB API
14 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer
15 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
16 12 11246 1 0.00% 0.00% 0.00% 0 GraphIt
17 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
18 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client
19 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
20 296 3273 90 0.00% 0.00% 0.00% 0 Net Background
21 0 19 0 0.00% 0.00% 0.00% 0 Logger
22 40 11242 3 0.00% 0.00% 0.00% 0 TTY Background
23 132 11255 11 0.00% 0.00% 0.00% 0 Per-Second Jobs
24 8 2 4000 0.00% 0.00% 0.00% 0 IPM_C1700_CLOCK
25 0 2 0 0.00% 0.00% 0.00% 0 AggMgr Process
26 4 2 2000 0.00% 0.00% 0.00% 0 ESWPPM
27 0 1 0 0.00% 0.00% 0.00% 0 Net Input
28 84 2253 37 0.00% 0.00% 0.00% 0 Compute load avg
29 8352 192 43500 0.00% 0.03% 0.04% 0 Per-minute Jobs
30 0 3 0 0.00% 0.00% 0.00% 0 Service-module a
31 0 1 0 0.00% 0.00% 0.00% 0 Switch Link Moni
32 4 2 2000 0.00% 0.00% 0.00% 0 AAA Server
33 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc
34 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr
35 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
36 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R
37 1655116 1351918 1224 1.35% 2.00% 5.19% 0 IP Input
38 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl
39 0 8 0 0.00% 0.00% 0.00% 0 CRYPTO IKMP IPC
40 0 1 0 0.00% 0.00% 0.00% 0 SSS Manager
41 4 1503 2 0.00% 0.00% 0.00% 0 SSS Test Client
42 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Mana
43 780 43982 17 0.00% 0.00% 0.00% 0 SSS Feature Time
44 0 4 0 0.00% 0.00% 0.00% 0 PPP Hooks
45 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana
46 0 1 0 0.00% 0.00% 0.00% 0 VPDN call manage
47 0 1 0 0.00% 0.00% 0.00% 0 L2X Data Daemon
48 0 1 0 0.00% 0.00% 0.00% 0 L2X Socket proce
49 0 1 0 0.00% 0.00% 0.00% 0 L2X SSS manager
50 0 2 0 0.00% 0.00% 0.00% 0 L2TP mgmt daemon
51 0 1 0 0.00% 0.00% 0.00% 0 AC Mgr
52 0 2 0 0.00% 0.00% 0.00% 0 KRB5 AAA
53 0 2 0 0.00% 0.00% 0.00% 0 DTP Protocol
54 4 3 1333 0.00% 0.00% 0.00% 0 PPP IP Route
55 0 4 0 0.00% 0.00% 0.00% 0 PPP IPCP
56 32 19 1684 0.00% 0.00% 0.00% 0 DHCPD Receive
57 524 215 2437 0.00% 0.00% 0.00% 0 IP Background
58 112 194 577 0.00% 0.00% 0.00% 0 IP RIB Update
59 0 36 0 0.00% 0.00% 0.00% 0 TCP Timer
60 8 3 2666 0.00% 0.00% 0.00% 0 TCP Protocols
61 0 1 0 0.00% 0.00% 0.00% 0 RARP Input
62 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers
63 4 40 100 0.00% 0.00% 0.00% 0 HTTP CORE
64 48 188 255 0.00% 0.00% 0.00% 0 IP Cache Ager
65 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall
66 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background
67 0 2 0 0.00% 0.00% 0.00% 0 PPP SSS
68 16 189 84 0.00% 0.00% 0.00% 0 Adj Manager
69 72 21976 3 0.00% 0.00% 0.00% 0 IP NAT Ager
70 4 2 2000 0.00% 0.00% 0.00% 0 PPP Bind
71 0 1 0 0.00% 0.00% 0.00% 0 SNMP Timers
72 0 1 0 0.00% 0.00% 0.00% 0 Inspect Timer
73 0 3196 0 0.00% 0.00% 0.00% 0 DHCPD Database
74 0 2 0 0.00% 0.00% 0.00% 0 URL filter proc
75 0 38 0 0.00% 0.00% 0.00% 0 Authentication P
76 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B
77 0 1 0 0.00% 0.00% 0.00% 0 IDS Timer
78 0 1 0 0.00% 0.00% 0.00% 0 COPS
79 0 2 0 0.00% 0.00% 0.00% 0 Dialer Forwarder
80 8 2253 3 0.00% 0.00% 0.00% 0 L2F management d
81 4 1 4000 0.00% 0.00% 0.00% 0 PPTP Mgmt
82 0 2 0 0.00% 0.00% 0.00% 0 PPTP Data
83 25680 30471 842 0.00% 0.00% 0.00% 0 Crypto HW Proc
84 0 1 0 0.00% 0.00% 0.00% 0 XSM_EVENT_ENGINE
85 8 1126 7 0.00% 0.00% 0.00% 0 XSM_ENQUEUER
86 4 1126 3 0.00% 0.00% 0.00% 0 XSM Historian
87 0 2 0 0.00% 0.00% 0.00% 0 LOCAL AAA
88 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA
89 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA
90 0 2 0 0.00% 0.00% 0.00% 0 TPLUS
91 360 14394 25 0.00% 0.00% 0.00% 0 CEF process
92 4 2 2000 0.00% 0.00% 0.00% 0 Crypto Support
93 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr
94 20 452 44 0.00% 0.00% 0.00% 0 CRM_CALL_UPDATE_
95 0 1 0 0.00% 0.00% 0.00% 0 Encrypt Proc
96 14128 117 120752 0.00% 0.00% 0.26% 0 Key Proc
97 24 4 6000 0.00% 0.00% 0.00% 0 Crypto CA
98 0 1 0 0.00% 0.00% 0.00% 0 Crypto SSL
99 20 40 500 0.00% 0.00% 0.00% 0 Crypto ACL
100 0 1 0 0.00% 0.00% 0.00% 0 CRYPTO QoS proce
101 12 14 857 0.00% 0.00% 0.00% 0 Crypto Delete Ma
102 104 109 954 0.00% 0.00% 0.00% 0 Crypto IKMP
103 1212 588 2061 0.00% 0.00% 0.00% 0 IPSEC key engine
104 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key
105 0 2 0 0.00% 0.00% 0.00% 0 AAA SEND STOP EV
106 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps
107 0 2 0 0.00% 0.00% 0.00% 0 IpSecMibTopN
108 32 1218 26 0.00% 0.00% 0.00% 0 PM Callback
109 38908 2770816 14 0.07% 0.03% 0.07% 0 SAA Event Proces
110 0 1 0 0.00% 0.00% 0.00% 0 VPDN Scal
111 4 2 2000 0.00% 0.00% 0.00% 0 VLAN Manager
112 18904 44385 425 0.31% 0.17% 0.16% 0 COLLECT STAT COU
113 0 1 0 0.00% 0.00% 0.00% 0 TCP Driver
114 0 1 0 0.00% 0.00% 0.00% 0 TCP Listener
115 0 1 0 0.00% 0.00% 0.00% 0 IP NAT WLAN
116 40 169 236 0.00% 0.00% 0.00% 0 SSH Event handle
117 2668 350258 7 0.00% 0.00% 0.00% 0 PPP manager
118 2508 350268 7 0.00% 0.00% 0.00% 0 PPP Events
119 12 11258 1 0.00% 0.00% 0.00% 0 Multilink PPP
120 0 2 0 0.00% 0.00% 0.00% 0 Multilink event
121 0 2 0 0.00% 0.00% 0.00% 0 IP Flow Backgrou
122 48 249 192 0.00% 0.00% 0.00% 0 Crypto Hardware
123 16 369 43 0.00% 0.00% 0.00% 0 CEF Scanner
right now
RPNAPLES#sh proc
CPU utilization for five seconds: 42%/33%; one minute: 47%; five minutes: 50%
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
1 Cwe 8029A2D0 268 64 4187 5548/6000 0 Chunk Manager
2 Csp 80280AB0 144 11797 12 2744/3000 0 Load Meter
3 M* 0 20 20 1000 9756/12000 6 Virtual Exec
4 Mwe 8053C650 8 492 16 5768/6000 0 DHCPD Timer
5 Lst 80299ACC 62660 7260 8630 5752/6000 0 Check heaps
6 Cwe 8029E304 40 45 888 5536/6000 0 Pool Manager
7 Mst 801A58E4 0 2 0 5708/6000 0 Timers
8 Mwe 80096BB8 0 2 0 5716/6000 0 Serial Backgroun
9 Mwe 80178B38 0 2 0 5704/6000 0 AAA high-capacit
10 Lwe 8034FD08 7476 9058 825 5008/6000 0 ARP Input
11 Mwe 80371168 0 3 0 5708/6000 0 DDR Timers
12 Mwe 80675650 516 13636 37 5768/6000 0 HC Counter Timer
13 Lwe 80978F34 12 2 6000 5624/6000 0 Entity MIB API
14 Mwe 80BE83B8 0 2 0 5712/6000 0 ATM Idle Timer
15 Mwe 8009C57C 0 1 0 5788/6000 0 SERIAL A'detect
16 Msp 801E85FC 188 58933 3 5744/6000 0 GraphIt
17 Mwe 8038C274 0 2 011712/12000 0 Dialer event
18 Mwe 806196F8 0 2 011728/12000 0 XML Proxy Client
19 Cwe 8026C730 0 1 0 5804/6000 0 Critical Bkgnd
20 Mwe 8021AC98 1804 10818 16610392/12000 0 Net Background
21 Lwe 80196DF8 0 21 011356/12000 0 Logger
22 Mwe 801BB920 300 58929 5 5736/6000 0 TTY Background
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
23 Msp 8022BC78 936 58942 15 7464/9000 0 Per-Second Jobs
24 Lwe 80311EC0 20 3 6666 1864/3000 0 IPM_C1700_CLOCK
25 Mwe 812F69B4 0 2 0 5784/6000 0 AggMgr Process
26 Hwe 81327A00 4 2 2000 5264/6000 0 ESWPPM
27 Hwe 8022B7A4 0 5 0 5788/6000 0 Net Input
28 Msp 80220A90 1100 11795 93 5740/6000 0 Compute load avg
29 Msp 8022BCEC 45540 990 46000 5752/6000 0 Per-minute Jobs
30 Mwe 8009F05C 0 3 0 5700/6000 0 Service-module a
31 Mwe 813322AC 0 1 0 5792/6000 0 Switch Link Moni
32 Mwe 8025067C 4 2 2000 5712/6000 0 AAA Server
33 Mwe 80252B48 0 1 0 5784/6000 0 AAA ACCT Proc
34 Mwe 80252C2C 0 1 0 5772/6000 0 ACCT Periodic Pr
35 Lwe 8017D6A4 0 1 0 5788/6000 0 AAA_SERVER_DEADT
36 Mwe 80304C34 0 2 0 5724/6000 0 AAA Dictionary R
37 Mrd 8044EEF4 9094956 6541061 1390 9952/12000 0 IP Input
38 Mwe 80473998 0 1 0 5768/6000 0 ICMP event handl
39 Mwe 81159038 0 8 0 5788/6000 0 CRYPTO IKMP IPC
40 Mwe 8085DB38 0 1 011788/12000 0 SSS Manager
41 Mwe 80861464 164 7865 2011752/12000 0 SSS Test Client
42 Mwe 8086AE94 0 1 0 5800/6000 0 SSS Feature Mana
43 Mwe 8086AF28 2524 230403 10 5768/6000 0 SSS Feature Time
44 Mwe 8027AE9C 0 4 011732/12000 0 PPP Hooks
45 Lwe 80A546B4 0 1 0 5396/6000 0 X.25 Encaps Mana
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
46 Mwe 80DFED4C 0 1 011748/12000 0 VPDN call manage
47 Mwe 80E7AB9C 0 1 011780/12000 0 L2X Data Daemon
48 Mwe 80E42F6C 0 1 011756/12000 0 L2X Socket proce
49 Mwe 80E0FD10 0 1 011784/12000 0 L2X SSS manager
50 Mwe 80E1CB74 0 2 011712/12000 0 L2TP mgmt daemon
51 Mwe 8105F430 0 1 0 5776/6000 0 AC Mgr
52 Mwe 81239BE4 0 2 011728/12000 0 KRB5 AAA
53 Mwe 812E6C64 0 2 0 5540/6000 0 DTP Protocol
54 Mwe 8027AE9C 4 3 133310988/12000 0 PPP IP Route
55 Mwe 8027AE9C 0 4 011448/12000 0 PPP IPCP
56 Mwe 80425860 264 125 2112 3868/6000 0 DHCPD Receive
57 Mwe 80551360 3340 1011 3303 8224/9000 0 IP Background
58 Mwe 805574F4 368 990 371 8736/9000 0 IP RIB Update
59 Mst 8043214C 8 61 13110484/12000 0 TCP Timer
60 Lwe 80437114 16 6 266610332/12000 0 TCP Protocols
61 Mwe 8049E14C 0 1 0 5804/6000 0 RARP Input
62 Hwe 80533708 0 1 0 5776/6000 0 Socket Timers
63 Mwe 805090E4 584 202 2891 6448/9000 0 HTTP CORE
64 Lsi 805CB69C 1196 983 1216 5304/6000 0 IP Cache Ager
65 Hwe 80A6DB7C 0 1 0 5780/6000 0 PAD InCall
66 Mwe 80A27320 0 2 011708/12000 0 X.25 Background
67 Mwe 8086CC34 0 2 0 5708/6000 0 PPP SSS
68 Mwe 80908750 644 985 653 8688/9000 0 Adj Manager
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
69 Mwe 80AA1614 1160 115124 10 5700/6000 0 IP NAT Ager
70 Mwe 8027AE9C 4 2 2000 5728/6000 0 PPP Bind
71 Mwe 8096E9F4 0 1 0 5756/6000 0 SNMP Timers
72 Mwe 80C7B920 0 1 0 5796/6000 0 Inspect Timer
73 Msi 8054B8C4 148 16711 8 4980/6000 0 DHCPD Database
74 Mwe 80CA5BF8 0 2 0 5612/6000 0 URL filter proc
75 Mwe 80CBF034 0 197 0 5796/6000 0 Authentication P
76 Mwe 80CC8EA0 0 1 0 5776/6000 0 Auth-proxy AAA B
77 Mwe 80CC9C64 0 1 0 5796/6000 0 IDS Timer
78 Mwe 80DE114C 0 1 023780/24000 0 COPS
79 Mwe 80DEFDF0 0 2 0 5720/6000 0 Dialer Forwarder
80 Mwe 80E06E3C 68 11794 511776/12000 0 L2F management d
81 Mwe 80E357FC 4 1 400011540/12000 0 PPTP Mgmt
82 Mwe 80E7F4DC 0 2 011728/12000 0 PPTP Data
83 Hwe 810EDD9C 25704 30502 842 5064/6000 0 Crypto HW Proc
84 Lwe 812DB7A4 0 1 0 5796/6000 0 XSM_EVENT_ENGINE
85 Lsi 812D93DC 116 5895 1911824/12000 0 XSM_ENQUEUER
86 Lsi 812DC4D4 80 5895 1311828/12000 0 XSM Historian
87 Mwe 8025D51C 0 2 0 5728/6000 0 LOCAL AAA
88 Mwe 8025F5D4 0 2 0 5728/6000 0 ENABLE AAA
89 Mwe 8025F9C0 0 2 0 5732/6000 0 LINE AAA
90 Mwe 803C8D1C 0 2 0 5604/6000 0 TPLUS
91 Lwe 808F6E08 3460 79767 43 4368/6000 0 CEF process
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
92 Mwe 810DB840 4 2 2000 5724/6000 0 Crypto Support
93 Mwe 81387464 0 1 0 5800/6000 0 EM Background Pr
94 Mwe 806D6DA0 52 2361 22 5668/6000 0 CRM_CALL_UPDATE_
95 Mwe 810D5BB4 0 1 011804/12000 0 Encrypt Proc
96 Mwe 810D67A4 64116 467 137293 6684/8000 0 Key Proc
97 Mwe 811B0058 24 4 6000 7000/8000 0 Crypto CA
98 Mwe 811EB1D4 0 1 0 7812/8000 0 Crypto SSL
99 Mwe 81158B3C 20 40 50020432/24000 0 Crypto ACL
100 Mwe 810E00DC 0 1 0 5792/6000 0 CRYPTO QoS proce
101 Mwe 81151F4C 36 53 67911332/12000 0 Crypto Delete Ma
102 Mwe 8111F364 424 386 1098 6168/12000 0 Crypto IKMP
103 Mwe 8111445C 1268 3013 420 9744/12000 0 IPSEC key engine
104 Mwe 81114F10 0 1 0 5716/6000 0 IPSEC manual key
105 Mwe 80171B4C 0 2 0 5708/6000 0 AAA SEND STOP EV
106 Mwe 80849E8C 0 1 0 5816/6000 0 Syslog Traps
107 Lwe 812C945C 0 2 0 5648/6000 0 IpSecMibTopN
108 Mwe 81324B6C 136 6375 21 5752/6000 0 PM Callback
109 Mwe 80AF1508 190808 14415505 13 5584/6000 0 SAA Event Proces
110 Mwe 80E45B78 0 1 0 5784/6000 0 VPDN Scal
111 Mwe 81337730 4 2 2000 3912/6000 0 VLAN Manager
112 Lsp 8132E680 112484 232949 482 5732/6000 0 COLLECT STAT COU
113 Mwe 805D0D8C 0 1 011796/12000 0 TCP Driver
114 Lwe 80435D60 0 1 0 5792/6000 0 TCP Listener
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
115 Mwe 80AD3A54 0 1 0 5772/6000 0 IP NAT WLAN
116 Mwe 81244984 180 715 251 4428/6000 0 SSH Event handle
117 Mwe 8099DAD0 31696 1838473 1711696/12000 0 PPP manager
118 Mwe 8027AE9C 33004 1838482 1711244/12000 0 PPP Events
119 Hwe 809D5CA8 80 58974 1 5724/6000 0 Multilink PPP
120 Mwe 809D568C 0 2 0 5712/6000 0 Multilink event
121 Mwe 80876D7C 0 2 0 5732/6000 0 IP Flow Backgrou
122 Mwe 810FBD70 208 983 211 4456/6000 0 Crypto Hardware
123 Lwe 808F6638 140 2023 69 5664/6000 0 CEF Scanner
RPNAPLES#sh proc cpu
CPU utilization for five seconds: 33%/22%; one minute: 43%; five minutes: 49%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 268 64 4187 0.00% 0.00% 0.00% 0 Chunk Manager
2 144 11801 12 0.00% 0.00% 0.00% 0 Load Meter
3 348 1061 327 0.15% 0.36% 0.08% 6 Virtual Exec
4 8 492 16 0.00% 0.00% 0.00% 0 DHCPD Timer
5 62660 7260 8630 0.00% 0.06% 0.05% 0 Check heaps
6 40 45 888 0.00% 0.00% 0.00% 0 Pool Manager
7 0 2 0 0.00% 0.00% 0.00% 0 Timers
8 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
9 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit
10 7480 9061 825 0.00% 0.00% 0.00% 0 ARP Input
11 0 3 0 0.00% 0.00% 0.00% 0 DDR Timers
12 516 13641 37 0.00% 0.00% 0.00% 0 HC Counter Timer
13 12 2 6000 0.00% 0.00% 0.00% 0 Entity MIB API
14 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer
15 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
16 188 58954 3 0.00% 0.00% 0.00% 0 GraphIt
17 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
18 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client
19 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
20 1812 10820 167 0.07% 0.02% 0.00% 0 Net Background
21 0 21 0 0.00% 0.00% 0.00% 0 Logger
22 300 58950 5 0.00% 0.00% 0.00% 0 TTY Background
23 936 58963 15 0.00% 0.00% 0.00% 0 Per-Second Jobs
24 20 3 6666 0.00% 0.00% 0.00% 0 IPM_C1700_CLOCK
25 0 2 0 0.00% 0.00% 0.00% 0 AggMgr Process
26 4 2 2000 0.00% 0.00% 0.00% 0 ESWPPM
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
27 0 5 0 0.00% 0.00% 0.00% 0 Net Input
28 1104 11798 93 0.00% 0.00% 0.00% 0 Compute load avg
29 45592 991 46006 0.63% 0.07% 0.04% 0 Per-minute Jobs
30 0 3 0 0.00% 0.00% 0.00% 0 Service-module a
31 0 1 0 0.00% 0.00% 0.00% 0 Switch Link Moni
32 4 2 2000 0.00% 0.00% 0.00% 0 AAA Server
33 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc
34 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr
35 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
36 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R
37 9097020 6543285 1390 11.27% 8.68% 9.27% 0 IP Input
38 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl
39 0 8 0 0.00% 0.00% 0.00% 0 CRYPTO IKMP IPC
40 0 1 0 0.00% 0.00% 0.00% 0 SSS Manager
41 164 7867 20 0.00% 0.00% 0.00% 0 SSS Test Client
42 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Mana
43 2524 230460 10 0.00% 0.00% 0.00% 0 SSS Feature Time
44 0 4 0 0.00% 0.00% 0.00% 0 PPP Hooks
45 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana
46 0 1 0 0.00% 0.00% 0.00% 0 VPDN call manage
47 0 1 0 0.00% 0.00% 0.00% 0 L2X Data Daemon
48 0 1 0 0.00% 0.00% 0.00% 0 L2X Socket proce
49 0 1 0 0.00% 0.00% 0.00% 0 L2X SSS manager
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
50 0 2 0 0.00% 0.00% 0.00% 0 L2TP mgmt daemon
51 0 1 0 0.00% 0.00% 0.00% 0 AC Mgr
52 0 2 0 0.00% 0.00% 0.00% 0 KRB5 AAA
53 0 2 0 0.00% 0.00% 0.00% 0 DTP Protocol
54 4 3 1333 0.00% 0.00% 0.00% 0 PPP IP Route
55 0 4 0 0.00% 0.00% 0.00% 0 PPP IPCP
56 264 125 2112 0.00% 0.00% 0.00% 0 DHCPD Receive
57 3340 1011 3303 0.00% 0.00% 0.00% 0 IP Background
58 368 990 371 0.00% 0.00% 0.00% 0 IP RIB Update
59 8 61 131 0.00% 0.00% 0.00% 0 TCP Timer
60 16 6 2666 0.00% 0.00% 0.00% 0 TCP Protocols
61 0 1 0 0.00% 0.00% 0.00% 0 RARP Input
62 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers
63 584 202 2891 0.00% 0.00% 0.00% 0 HTTP CORE
64 1196 984 1215 0.00% 0.00% 0.00% 0 IP Cache Ager
65 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall
66 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background
67 0 2 0 0.00% 0.00% 0.00% 0 PPP SSS
68 644 985 653 0.00% 0.00% 0.00% 0 Adj Manager
69 1160 115153 10 0.00% 0.00% 0.00% 0 IP NAT Ager
70 4 2 2000 0.00% 0.00% 0.00% 0 PPP Bind
71 0 1 0 0.00% 0.00% 0.00% 0 SNMP Timers
72 0 1 0 0.00% 0.00% 0.00% 0 Inspect Timer
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
73 148 16728 8 0.00% 0.00% 0.00% 0 DHCPD Database
74 0 2 0 0.00% 0.00% 0.00% 0 URL filter proc
75 0 197 0 0.00% 0.00% 0.00% 0 Authentication P
76 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B
77 0 1 0 0.00% 0.00% 0.00% 0 IDS Timer
78 0 1 0 0.00% 0.00% 0.00% 0 COPS
79 0 2 0 0.00% 0.00% 0.00% 0 Dialer Forwarder
80 68 11797 5 0.00% 0.00% 0.00% 0 L2F management d
81 4 1 4000 0.00% 0.00% 0.00% 0 PPTP Mgmt
82 0 2 0 0.00% 0.00% 0.00% 0 PPTP Data
83 25704 30502 842 0.00% 0.00% 0.00% 0 Crypto HW Proc
84 0 1 0 0.00% 0.00% 0.00% 0 XSM_EVENT_ENGINE
85 116 5896 19 0.00% 0.00% 0.00% 0 XSM_ENQUEUER
86 80 5896 13 0.00% 0.00% 0.00% 0 XSM Historian
87 0 2 0 0.00% 0.00% 0.00% 0 LOCAL AAA
88 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA
89 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA
90 0 2 0 0.00% 0.00% 0.00% 0 TPLUS
91 3464 79785 43 0.07% 0.00% 0.00% 0 CEF process
92 4 2 2000 0.00% 0.00% 0.00% 0 Crypto Support
93 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr
94 52 2362 22 0.00% 0.00% 0.00% 0 CRM_CALL_UPDATE_
95 0 1 0 0.00% 0.00% 0.00% 0 Encrypt Proc
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
96 64116 467 137293 0.00% 0.00% 0.00% 0 Key Proc
97 24 4 6000 0.00% 0.00% 0.00% 0 Crypto CA
98 0 1 0 0.00% 0.00% 0.00% 0 Crypto SSL
99 20 40 500 0.00% 0.00% 0.00% 0 Crypto ACL
100 0 1 0 0.00% 0.00% 0.00% 0 CRYPTO QoS proce
101 36 53 679 0.00% 0.00% 0.00% 0 Crypto Delete Ma
102 424 386 1098 0.00% 0.00% 0.00% 0 Crypto IKMP
103 1268 3014 420 0.00% 0.00% 0.00% 0 IPSEC key engine
104 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key
105 0 2 0 0.00% 0.00% 0.00% 0 AAA SEND STOP EV
106 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps
107 0 2 0 0.00% 0.00% 0.00% 0 IpSecMibTopN
108 136 6377 21 0.00% 0.00% 0.00% 0 PM Callback
109 190920 14418995 13 0.71% 0.32% 0.34% 0 SAA Event Proces
110 0 1 0 0.00% 0.00% 0.00% 0 VPDN Scal
111 4 2 2000 0.00% 0.00% 0.00% 0 VLAN Manager
112 112516 233006 482 0.15% 0.17% 0.20% 0 COLLECT STAT COU
113 0 1 0 0.00% 0.00% 0.00% 0 TCP Driver
114 0 1 0 0.00% 0.00% 0.00% 0 TCP Listener
115 0 1 0 0.00% 0.00% 0.00% 0 IP NAT WLAN
116 180 715 251 0.00% 0.00% 0.00% 0 SSH Event handle
117 31696 1838902 17 0.00% 0.01% 0.03% 0 PPP manager
118 33008 1838911 17 0.00% 0.01% 0.04% 0 PPP Events
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
119 80 58988 1 0.00% 0.00% 0.00% 0 Multilink PPP
120 0 2 0 0.00% 0.00% 0.00% 0 Multilink event
121 0 2 0 0.00% 0.00% 0.00% 0 IP Flow Backgrou
122 208 983 211 0.00% 0.00% 0.00% 0 Crypto Hardware
123 140 2024 69 0.00% 0.00% 0.00% 0 CEF Scanner
RPNAPLES#
Always hard to find a ghost. One thing to think about is that though yo may have a lot of traffic, it still may be the carrier. The best test would be to set up a monitor at site A, then one at site B. Set up two computers to "talk" to each other over various ports/UDP-TCP, also set up two VoIP phones also (usea audio source via speakerphone). This way we could actually see the packets leaving site A and arrive at site B. From this we could tell if the carrier is causing problems some how.
Did you adjust the MTU size by chance already?
Have a good holiday!
Joel
Hello Joel,
Hope you had a good holiday.
Would you please do me a big favor and look at the configs again to make sure I have it correct.
Phone quality has been terrible and I just don't know if my QOS is working properly.
172.168.1.0 is the remote internal network with the ip phones.
It connects back to the 192.168.0.0 network where the phone equip resides.
I have a full T1 at the 192 and 1000k at the 172.
Thanks,
Will
I have only glimpsed at the thread, but it is clear that the router CPU had a heavy load on it even when it should be doing almost nothing, and if it’s that high when it should be idle, it probably is overloading during the day. So the priority here should be finding out why it’s so high when it shouldn’t, and correcting it. Once that is taken care of, if you are still having issues, they can then be addressed.
The first question is, where is the load coming from? I’d log into one of the routers, and shut down the Ethernet interface on it, and telnet from it to the other router, and do that same. Wait a few minutes and do a show process CPU on both routers and see if it drops, if it doesn’t do a show interface serial X on both routers, and see if you are getting a lot of trash coming from your ISP. If so, call them up and work with them to eliminate it.
Now if the trash isn’t coming from your ISP, turn up the Ethernet interface one router at a time and see if it gets pounded, show interface FastEthernet X, if dose, find out what on that LAN is causing it. An easy way to do that is yank all the connectors out of the switch connected to the router, and plug them back in a few at a time until the traffic spikes back up. When it dose, pull out the last group of wires and plug them in one at a time to find the guilty party.
I suspect by the way you have some kind of worm on your LAN, or people running things they shouldn’t P2P software, which can easily overload a T1 if used heavily, but it also could be some network app, that is just too big a load for your internet connection.
We could try doing that, but the router still has to address all that trash, which loads down the CPU, although maybe not as much. So let’s get rid of trash first and latter we can play with the access lists. Also, just to be sure, you are using a switch, because if you are using a hub, the router will get a lot of needless LAN traffic.
after clearing counters after hours about 3min after clearing
RPNAPLES#sh access-lists
Extended IP access list 100
10 permit udp any any range 16384 32000
20 permit tcp any any eq 1719
30 permit tcp any any eq 1720
40 permit tcp any any eq 6100
50 permit tcp any any range 1024 4999 (459 matches)
60 permit udp any any eq 6000 (717 matches)
70 permit udp any any range 1024 4999 (11 matches)
80 permit udp any any eq 5060
90 permit udp any any range 30000 30030
100 permit udp any any range 9000 9001 (10800 matches)
Extended IP access list 101
10 permit ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255 (36091 matches)
20 deny ip 192.168.0.0 0.0.0.255 any
Extended IP access list 102
10 permit tcp any any range 5000 5110
Extended IP access list 105
10 deny icmp any any echo
20 permit ip any any
Extended IP access list 110
10 deny ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255 (2 matches)
20 deny ip host 192.168.0.205 any (171 matches)
30 deny ip host 192.168.0.30 any
40 deny ip host 192.168.0.31 any
50 deny ip host 192.168.0.2 any (57 matches)
60 permit ip 192.168.0.0 0.0.0.255 any (24 matches)
Extended IP access list 120
10 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255
Extended IP access list 123
10 permit ip host 192.168.0.205 172.168.1.0 0.0.0.255 (20 matches)
20 permit ip host 192.168.0.30 172.168.1.0 0.0.0.255 (550 matches)
30 permit ip host 192.168.0.31 172.168.1.0 0.0.0.255 (7761 matches)
40 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255 (84 matches)
Extended IP access list sl_def_acl
10 deny tcp any any eq telnet log
20 deny tcp any any eq www log
30 deny tcp any any eq 22 log
40 permit ip any any log
RPNAPLES#
To say that is odd would be a gross understatement, and it definitely is not what I would have expected. I work with carrier grade VOIP equipment, and some of those gateways can handle over 700 calls, yet when they are idle, at the most they use a few kilobytes of bandwidth to periodically renew their registration to the gatekeeper.
So there is no reason on earth a little office VOIP system should use anywhere even remotely close to what your system is using when idle. It’s not playing background music on the phones like I have seen in some offices, or something like that? As that is the only thing I can think of that could account for what you are seeing.
here is the access-lists from this morning.
RPNAPLES#sh access-list
Extended IP access list 100
10 permit udp any any range 16384 32000
20 permit tcp any any eq 1719
30 permit tcp any any eq 1720
40 permit tcp any any eq 6100
50 permit tcp any any range 1024 4999 (7689 matches)
60 permit udp any any eq 6000 (81348 matches)
70 permit udp any any range 1024 4999 (3443 matches)
80 permit udp any any eq 5060
90 permit udp any any range 30000 30030
100 permit udp any any range 9000 9001 (2795673 matches)
Extended IP access list 101
10 permit ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255 (9532221 matches)
20 deny ip 192.168.0.0 0.0.0.255 any
Extended IP access list 102
10 permit tcp any any range 5000 5110 (55 matches)
Extended IP access list 105
10 deny icmp any any echo
20 permit ip any any
Extended IP access list 110
10 deny ip 192.168.0.0 0.0.0.255 172.168.1.0 0.0.0.255 (485 matches)
20 deny ip host 192.168.0.205 any (59493 matches)
30 deny ip host 192.168.0.30 any
40 deny ip host 192.168.0.31 any
50 deny ip host 192.168.0.2 any (12442 matches)
60 permit ip 192.168.0.0 0.0.0.255 any (4405 matches)
Extended IP access list 120
10 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255
Extended IP access list 123
10 permit ip host 192.168.0.205 172.168.1.0 0.0.0.255 (2396 matches)
20 permit ip host 192.168.0.30 172.168.1.0 0.0.0.255 (65236 matches)
30 permit ip host 192.168.0.31 172.168.1.0 0.0.0.255 (1866125 matches)
40 permit ip host 192.168.0.2 172.168.1.0 0.0.0.255 (17106 matches)
Extended IP access list sl_def_acl
10 deny tcp any any eq telnet log
20 deny tcp any any eq www log
30 deny tcp any any eq 22 log
40 permit ip any any log
A show interface would be more relevant here since we know what is generating most of the traffic, but counting the match’s on ports 9000-1, I assume that its for the phone system, for what you say was three minutes, it looks like the phone systems using enough bandwidth for a good twenty plus calls when idle.
On a local LAN that wouldn’t be a problem, but across a slow WAN link, running through a VPN, on small router with a modest processor, processing access lists and doing QOS, it’s a recipe for trouble, but I think you have figured that out already. Also seeing how much bandwidth the phones are using when idle, makes me wonder how much bandwidth they use when in operation, and if a single T1 can handle it?
I think it’s time you get a hold of the people, who provided your phone system, as there is nothing I can do about other than recommend you get more powerful routers. You might still need to do that since the routers you have a marginal for what you are dong with them, but until get rid of this parasitic load and see how it behaves I can’t say that. Also even worse, in addition too more powerful routers you might need to get another T1 to deal with all the bandwidth that phone system is sucking up.
So trying to fix this on the Cisco end would be expensive, especially if it came down to getting more T1’s. So this needs to be addressed at the phone end of things if at all possible, since with all seven remote phones in operation it should be using about 70-80K of bandwidth, but it looks like it’s using closer to 250K when idle, and probable a hell of lot more in operation. There has got to be something in it’s configuration to cut this down to something reasonable, but since I don’t work with Samsung phone systems I have no clue as to where to start.
here is my cleared interfaces after 5mins
RPNAPLES#sh int s0
Serial0 is up, line protocol is up
Hardware is PQUICC with Fractional T1 CSU/DSU
Description: USLEC T1
Internet address is 199.72.194.234/30
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 12/255, rxload 36/255
Encapsulation PPP, LCP Open
Open: IPCP, loopback not set
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:10:36
Input queue: 3/75/0/0 (size/max/drops/flushes); Total output drops: 8
Queueing strategy: weighted fair
Output queue: 0/1000/64/8 (size/max total/threshold/drops)
Conversations 0/8/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 52 kilobits/sec
5 minute input rate 219000 bits/sec, 304 packets/sec
5 minute output rate 78000 bits/sec, 81 packets/sec
193593 packets input, 17672596 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
50723 packets output, 6189640 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
RPNAPLES#sh int f0
FastEthernet0 is up, line protocol is up
Hardware is PQUICC_FEC, address is 000f.f7f8.4b2a (bia 000f.f7f8.4b2a)
Description: $FW_INSIDE$$ETH-LAN$
Internet address is 192.168.0.1/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:12:13
Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: priority-list 5
Output queue (queue priority: size/max/drops):
high: 0/20/0, medium: 0/40/0, normal: 0/60/0, low: 0/80/0
5 minute input rate 60000 bits/sec, 81 packets/sec
5 minute output rate 155000 bits/sec, 302 packets/sec
57995 packets input, 5391239 bytes
Received 167 broadcasts, 0 runts, 0 giants, 0 throttles
914 input errors, 299 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
222447 packets output, 14317892 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
I will keep you posted
I won’t worry too much about posting your routers IP address, a lot of people here are way too paranoid about that. The honest truth is, if your equipment is that poorly secured; it will be found, and hacked regardless. At least once a day, someone scans all 4000 of my IP address to find active address to attempt to compromise systems using automated tools and scripts, that where the name script kiddies comes from by the way, and if it wasn’t secured with proper passwords and access lists, I’d be in trouble.
What I would worry about it posting passwords, or the address of VOIP gateways, as there are more than a few tricks that can be used on a lot of them to trick them into taking calls without the need to even hack them. That’s because they are inherently insecure, and a lot of people are unaware of that, and or don’t have a clue as to how to properly secure them.
Just to give you an idea of have bad your situation is, below I have posted the show interface from one of my smaller VOIP gateways that had 40 calls on it at the time. As you can see, your packet rate when idle is pretty close to mine under far more calls than you will ever have. It’s the packet rate that really loads down the processor on your router by the way, as each one has to be processed to determine how it is going to be handled, and with QOS and all those access lists it puts a pretty big load on the little routers CPU.
As for you show interfaces, it confirms what I expected.
5309#sh int f0
FastEthernet0 is up, line protocol is up
Hardware is DEC21140, address is 0010.7be6.4ff9 (bia 0010.7be6.4ff9)
Internet address is 10.0.0.179/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/512/0/25 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 322000 bits/sec, 399 packets/sec
5 minute output rate 317000 bits/sec, 391 packets/sec
436079427 packets input, 1052027595 bytes
Received 3651511 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
336035809 packets output, 4175800439 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
5309#
better news called my phone guy and he reset the system
here is what it shows
RPNAPLES#sh int f0
FastEthernet0 is up, line protocol is up
Hardware is PQUICC_FEC, address is 000f.f7f8.4b2a (bia 000f.f7f8.4b2a)
Description: $FW_INSIDE$$ETH-LAN$
Internet address is 192.168.0.1/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:10:09
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: priority-list 5
Output queue (queue priority: size/max/drops):
high: 0/20/0, medium: 0/40/0, normal: 0/60/0, low: 0/80/0
5 minute input rate 2000 bits/sec, 4 packets/sec
5 minute output rate 1000 bits/sec, 2 packets/sec
3443 packets input, 252563 bytes
Received 156 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
1596 packets output, 128515 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
cpu is around 1%-5%
will see if voice quality is any better
after the reset 9000 - 9001 is not getting any traffic now. which it was getting hammered before.
That’s more like it, and getting rid of that extraneous load can only help call quality. That router doesn’t have a very powerful processor, and having a load of that level on it before it even starts processing meaningful traffic is bound to send it into overload. The main thing is to keep an eye on that phone system, to see if it starts doing it again. There might be some bug in the code that will keep reappearing, so watch it closely.
When did you reset the CRC count?
Also port 9000 is the default Voice port the Samsung phones use. Are you using OfficeServ 500 or 7200? Are using any of the applications? OfficeServ LInk, Operator, Call Monitor, ACD?
One question is that are people using Call Forward or DND on their phones? Since 9000 is for the voice payload, depending upon the Samsung system to reconize call routing logical loops, you could have someone who has call forwared their phone to someone who has forward theri phone to someplace else, even the other phone.
Since we have reconized that commmunications on port 9000 is the major problem, lets use Colasoft or even Ethereal to capture the packets and see where they are going?
Also what type of switches are you using in your network? Also the exact equipment you have via the phone system.
Also just to mention that 9000/9001 port is used in Microsoft's massively-
multiplayer game called "Asheron's Call".The game can continue to contact the player even after the player has logged out.
http://www.fuzeqna.com/ash
Not saying this is the problem but you never know, do you know if you have any gamers on the Network?
It looks like it’s going up quickly, and will probably be back to what it was before it was reset in another day or two. As for the cheap switches, they should be OK; the important thing is not to be using hubs, as the router would be seeing every packet going across the LAN. One other thing you will want to check with the phone people, what codec are they using? You what to be sure they are not using G711, as it uses a lot of bandwidth, it’s OK for using on the local LAN, but you don’t want to use it for the remote phones. For them you want to be using G729, or G723 since they give the best combination of voice quality verses bandwidth usage.
6000 is the typical signal port for the Samsung phones. Do you have the ability to enter into the MMC programing state on the phones? If so we can verify the codecs and ports being used.
Press Transfer button on phone, enter 800, then passcode (default is 4321). Press 1 to enable tenant, press speaker button, dial MMC code 834 835 or 840.
834 option numer 14 will show signal port, but also check the following options under this MMC 00, 01,11 and post what they are.
835 will show the type of MGI (1,2 or 3). Let me know what is configured. If MGI3, the default codec should be G.729a, but there are a fewother parameters that we can check, but need to find out what is being supported.
840 options, 04, 05, 07, 08
While we wait on that can you capture some of the packets? Have you used Ethereal before? If you can capture some of the packets and post it, we can hunt down to see if the communcation is between a few endpoints or a system wide configuration. You can use the filter feature of Ethereal and just capture the packets using port 9000.
I looked at it and colasoft does not seem to give me the correct traffic. I don't see the traffic that heavy.
Also on another note all traffic from the remote site has a hostname of A****.ipt.aol.com what the hell is that??
Router is running back around 70-80 percent. I just don't see it on the basline program. I see a few 9000-9001 and alot of 30000-30009 but the traffic is only 40kb sec looking at my software.
PORT NUMBERS
(last updated 2 December 2004)
http://www.iana.org/assign
The IP address that maps to that name is an AOL address, but I'd block it anyway at the Ethernet interface, as you need to get rid of any extraneous traffic you can get rid of. The real question is what is loading down the processor, do a show processes CPU, and look to see what process is loading it down.
http://www.cisco.com/en/US
Also there a few other tools avaliable to help disinfecting your systems:
http://www.gfi.com/lannets
http://www.intermute.com/
http://www.spywareinfo.com
Also instead of using Colasoft should try using Ethereal, Colasoft might be filtering out traffic from its view.
Also there is another great tool that is good to apply directly onto a machine that you want to check out called AATools:
http://www.glocksoft.com/a
Thanks Joel
Some of the cisco articles where helpful
I ran some looging on the router for about 5 secs and here is what I got some is cut out due to space of this page
It is constant. Constantly the switch is looking for the phones. If I were to put are the logging I had after 5secs in word It would at least be 1 1/2 pages. I talked to my phone guy today he thought they might be using g711 , I told him I kept seeing 64k packets coming in from the phone.
001876: May 15 05:08:11.429: IP: s=192.168.0.31 (FastEthernet0), d=172.168.1.216
(Loopback0), g=1.1.1.2, len 80, forward
001877: May 15 05:08:11.433: UDP src=30000, dst=9000
001878: May 15 05:08:11.433: IP: s=192.168.0.31 (Loopback0), d=172.168.1.216 (Se
rial0), g=199.72.194.233, len 80, forward
001879: May 15 05:08:11.433: UDP src=30000, dst=9000
001880: May 15 05:08:11.441: IP: s=192.168.0.31 (FastEthernet0), d=172.168.1.108
(Loopback0), g=1.1.1.2, len 80, forward
001881: May 15 05:08:11.441: UDP src=30004, dst=9000
001882: May 15 05:08:11.445: IP: s=192.168.0.31 (Loopback0), d=172.168.1.108 (Se
rial0), g=199.72.194.233, len 80, forward
001883: May 15 05:08:11.445: UDP src=30004, dst=9000
001884: May 15 05:08:11.473: IP: s=192.168.0.31 (FastEthernet0), d=172.168.1.216
(Loopback0), g=1.1.1.2, len 80, forward
001885: May 15 05:08:11.473: UDP src=30000, dst=9000
001886: May 15 05:08:11.473: IP: s=192.168.0.31 (Loopback0), d=172.168.1.216 (Se
rial0), g=199.72.194.233, len 80, forward
001887: May 15 05:08:11.473: UDP src=30000, dst=9000
One thing that is coming clear to me, that although you have other issues, the router you are using is marginal at best. Even if you do manage to get rid of all the other issues, at best you will be skirting right on the edge of its capabilities. I really think you need to start thinking about upgrading to something like a 2650, or 2811. Which are a lot better suited for doing what you are trying to do.
I am seeing that the router is struggling, but before I purchased these 1721s I had to have approval from Samsung on them. Samsung approved them and said they had 1721 in operation everywhere. In the very near future these router will probably be taken off the public internet and placed in an enviroment where it will just be a point to point t1 between the remote office and the main office and then have another router for the internet. If you think all of the problems are just due to the fact these routers are underpowered please tell me now. If you do not think they will work in the next application speak up for my sake..ha
Is the log above normal? Only those 2 phones were acting that way. They crush this router.
Thanks again,
will
If it was just a point to point circuit, without extensive access lists, and doing VPN encryption, along with QOS, the load on the routers processor would be significantly less than it currently is. So I’d say in a point to point private circuit properly configured there would be a decent chance they would be ok, but that would add a permanent overhead expense the you could possibly ovoid by having more powerful routers. So buying a more expensive router in the long run could be less expensive.
Now this all leads me back to what I have said already, that before we can say they can work, we need to fix the other issues, but seeing how easily they are overloaded by what shouldn’t be a major issue, i.e., less than a quarter meg of trash from the remote phones, it is clear they are marginal. This is why I am say, hey we may be able to get this to work, but every thing will have to be perfect for that. So although they might be able to be made to work ok, they’d always be susceptible to any issue that loads them down. So yes they are struggling, and I’d say Samsung although they might have 1721’s working ok in other places, they a skating on thin ice.
I will have to agree with the Doc on the 1721 being overloaded easily. Have to remember that the 1721 was not originally deisgined to handle what is has been modifiy to handle. On top of being a self proclaimed convergence technonlogist, I was a General contractor for years, and no matter how good of a finish trim guy you have, the foundation is crooked its crooked so to speak.
You could bump to a bigger and better router or you could also offload the VPN to a dedicated VPN device. Processing the packets for encryption takes time and overhead.
Also remember all it takes is one bad apple to crash an Ethernet network, though less likely these days with switches but we are still resticted by the WAN.
very true
When just the VPN, Acess-lists, Nat running, there is really no pressure on the cpu it is low 2-10% but anytime the phone switch is running, maybe through no real fault of it's own, the performance drops.
Do you think by offloading the vpn and leaving everything else the way it is that would help? I am very skeptical, almost gunshy. I have been pricing 2650s on ebay and well they are a hell of a lot cheaper than what I paid new for the 1721s with VPN. damn....
I have never been a fan of the all in one box, ask yourself this, do you still have your swiss army knife? Also applying the access lists to the inbound traffic of the router interface reduces processor overhead. Using a seperate firewall/VPN device is probally your best bet and most cost effective.
A lot of Cisco’s older router designs like the 1700 series where not designed with a lot the things they added in mind. If you where running just normal data through that 1721, it would probably be fine, but you are running VOIP though it, which generates a very high packet rates because of all the small packets VOIP uses. Which puts a considerably higher load on the router, especially since it has to compare every one of those many little packets to the access lists; QOS them, and then deal with the encryption for the VPN.
Now there is one option for the 1721’s I haven’t mentioned yet as I wasn’t sure it was available, a hardware VPN encryption module, P/N MOD1700-VPN. That would offload the encryption tasks from the processor freeing it up to do other tasks, but it would be even better to have a 2650 with it’s more powerful process, and a VPN accelerator module.
Also the 2650 even without a VPN accelerator might still out perform a 1721 series router since the processor in it is a lot faster than a 1721, A 80 MHz MPC 860P RISC processor verses a 48 Mhz MPC 860T RISC processor for the 1721. Also since it’s a P version it has larger catches and other performance enhancing features, so the performance boost is even greater than the MHz improvement would imply.
An even better soulution would be to go with a X8XX series router, which are Cisco’s new generation of routers designed from the ground up for the kinds of task you are trying to get those 1721’s to do. If you had just gone up one step to the 1841 you probably wouldn’t have the problems you are having, since all X8XX series routers have built in encryption coprocessors. It’s what I would have suggested to someone asking for a new 1721, but in knowing what you are doing with it, I would have recommended stepping up to a 2800 series router instead just to be sure. Anyway a more powerful routing solution is probably in order here.
Hey considering how long this post have become, I should get a medal for finding anything. Anyway, so he has hardware encryption and it still is overloaded, I found the config, and checked to see if the VPN module required any special configuration to enable it, but it looks like other than having an IOS that supports it, and it looks like he does. One thing I noted, is he doesn’t have CEF enabled “ip cef”, I have mentioned this already, but I don’t know if he has enabled it, as it does reduce the load on the processor.
3 Ways to Join
Business Accounts
- Perfect for organizations
- Multiple users
- Save over 36%
- Create a Business Account
Answer for Membership
- Earn free access
- Showcase your knowledge
- No credit card required
- Sign Up to Answer
Loading Advertisement...
by: cmsJustinPosted on 2004-10-29 at 10:27:23ID: 12446579
How many phones?
How many peak calls at any moment in time?