asked on

How can I reduce router CPU load under Cisco WAAS deployment scenario?

I have a multi-site Cisco WAAS deployment in process that is causing significant CPU load on our core WAN router. For each remote site I deploy WAAS to, I add 4 lines to my extended ACL on the WAN router. In our data center (hub and spoke network) we have a CIsco 3845 router. This router has a 45 Mbps DS3 and a 100 Mbps Ethernet connection to our WAN provider's MPLS cloud (two circuits for diversity and bandwidth). We have over 60 remote sites in the USA, soon to be over 70 that eventually will all have WAAS in place. Each remote site has a 2800 or 2900 series Cisco router with a WAAS module in it (NME-WAE-502), and has either 2 or 3 T1's bonded together for connectivity to the WAN cloud. The WAAS implementation was going great, and all the remote sites we had deployed to were seeing approximately 5x in bandwidth throughput increase. Then we noticed our CPU usage climbing on that 3845 router, now running 75% and higher at times. The initial deployment of the WAAS was performed for us by a VAR, but the specific engineer is no longer with them. The configuration was set up to use WCCP redirect, hence the need for the extended ACL list. The core WAAS appliances consists of a Cisco 674 accelerator and a Cisco 512 central manager. Given all this, was the WCCP redirect (as opposed to an in-line scenario) the best deployment? Any suggestions on reducing the CPU load on that core WAN router? Here's an example of the ACL entries:
deny tcp 10.44.18.0 0.0.0.255 any
deny tcp any 10.44.18.0 0.0.0.255
permit tcp 10.44.0.0 0.0.31.255 any
permit tcp any 10.44.0.0 0.0.31.255

If an 'In-line' configuration or other would better apply here, how would that confuration differ?

rochey2009

Hi,

Are you using CEF?

Please can you post

show ip wccp

and the running config

maderosia

ASKER

Yes, 'IP CEF' is on for the core WAN router.

See attachments
Show-IP-WCCP.txt.txt

maderosia

ASKER

3845-WAN-Config.txt.txt

ASKER CERTIFIED SOLUTION

rochey2009

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

maderosia

ASKER

I hope to be able to test your suggestion one night later this week.

maderosia

ASKER

We have modified / consolidated the extended ACL, but it seems to only have reduced the CPU 5% or less. We are looking at upgrading the hardware to an ASR.

rochey2009

Which process has the highest utilisation?

show processes cpu

maderosia

ASKER

CPU utilization for five seconds: 86%/78%; one minute: 78%; five minutes: 74%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 40 433 92 0.00% 0.00% 0.00% 0 Chunk Manager

2 1641280 2734499 600 0.07% 0.09% 0.08% 0 Load Meter

3 122236 13320935 9 0.00% 0.00% 0.00% 0 BGP Scheduler

4 19935004 1789469 11140 0.00% 0.09% 0.09% 0 Check heaps

5 274404 162178 1691 0.00% 0.00% 0.00% 0 Pool Manager

6 0 2 0 0.00% 0.00% 0.00% 0 Timers

7 0 1 0 0.00% 0.00% 0.00% 0 License Client
N
8 1904 227828 8 0.00% 0.00% 0.00% 0 IPC Dynamic Cac
h
9 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manage
r
10 111972 13320945 8 0.00% 0.00% 0.00% 0 IPC Periodic Ti
m
11 100796 13320930 7 0.00% 0.00% 0.00% 0 IPC Deferred Po
r
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
12 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manage
r
13 0 1 0 0.00% 0.00% 0.00% 0 IPC BackPressur
e
14 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler

15 0 1 0 0.00% 0.00% 0.00% 0 Crash writer

16 1703080 2730111 623 0.00% 0.00% 0.00% 0 Environmental m
o
17 0 1 0 0.00% 0.00% 0.00% 0 chkpt message h
a
18 23483124 61038128 384 0.16% 0.08% 0.07% 0 ARP Input

19 240984 14193407 16 0.00% 0.00% 0.00% 0 ARP Background

20 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer

21 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capaci
t
22 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEAD
T
23 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

24 4 8 500 0.00% 0.00% 0.00% 0 DDR Timers

25 0 3 0 0.00% 0.00% 0.00% 0 Entity MIB API

26 96 938 102 0.00% 0.00% 0.00% 0 EEM ED Syslog

27 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgrou
n
28 0 1 0 0.00% 0.00% 0.00% 0 CEF MIB API

29 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timer
s
30 0 1 0 0.00% 0.00% 0.00% 0 RMI RM Notify W
a
31 0 2 0 0.00% 0.00% 0.00% 0 SMART

32 123204 13667452 9 0.00% 0.00% 0.00% 0 GraphIt

33 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

34 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

35 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Clien
t
36 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

37 32039720 2367253 13534 0.32% 0.14% 0.12% 0 Net Background

38 0 4 0 0.00% 0.00% 0.00% 0 IDB Work

39 76 1354 56 0.00% 0.00% 0.00% 0 Logger

40 338588 13636273 24 0.00% 0.00% 0.00% 0 TTY Background

41 270062780 14275843 18917 1.12% 1.17% 1.19% 0 Per-Second Jobs

42 4 12 333 0.00% 0.00% 0.00% 0 IF-MGR control
p
43 0 118 0 0.00% 0.00% 0.00% 0 IF-MGR event pr
o
44 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Des
t
45 0 1 0 0.00% 0.00% 0.00% 0 IKE HA Mgr

maderosia

ASKER

Re-arranging the WCCP-REDIRECT access lists as well as removing the security ACLs off the WAN router has reduced our CPU load about 15%. This is enough to keep it from maxing out until we replace the router with a larger device (ASR 1001). We moved the security ACLs to the remote location rouetrs because they have more than enough processing power. I also broke the WCCP-REDIRECT up into 2 seperate lists. One list is for incomming traffic flowing into our central data center and the other is for incomming traffic flowing into our WAN.