Solved

Cisco ACL Error Message

Posted on 2001-08-31
23
799 Views
Last Modified: 2007-12-19
I inserted an ACL into my Cisco 2610 router on the in and out side of the serial port.  Traffic flows back and forth across the router with no problems.  I turned on logging and noticed an error coming from the router.  The error has occurred about 5 times in the past day but does not appear to be affecting the performance of the router.  The error is as follows:

<187>782: 20:42:49: %SCHED-3-STUCKMTMR: Sleep with expired managed timer 808BE8C0, time 0x471D618 (00:00:00 ago). in 31-Aug 9:44:57.18 from x.x.x.x

<187>783: -Process= "IP RACL Ager", ipl= 6, pid= 43 in 31-Aug 9:44:57.18 from x.x.x.x

<187>784: -Traceback= 8020BAA4 8020BD68 80571658 80218E00 in 31-Aug 9:44:57.18 from x.x.x.x

I cannot find any information on this error or what causes it.  Any ideas?

FYI->Using reflexive ACL on the out bound. IOS version 12.0
0
Comment
Question by:seanselman
  • 11
  • 9
  • 2
  • +1
23 Comments
 
LVL 8

Expert Comment

by:scraig84
ID: 6445829
Interesting one.  I went to Cisco's Error Message Decoder and this is what it said when I pasted in your error:


A process can register to be notified when various events occur in the router. A registered timer is expired, and the timer value is unchanged after the process was executed two successive times.
Recommended Action: Copy the error message exactly as it appears on the console or in the system log, call your Cisco technical support representative, and provide the representative with the gathered information.


Helpful!  I would at least check with the Cisco TAC if nobody else is able to find much on this.  Even if it is not affecting performance, it wouldn't log something like that unless it meant something.

Also, maybe its just the way that you worded your question, but are you truly applying the same ACL to both inbound and outbound traffic?  I'm not saying this is unheard of, but it is generally not needed or desired.  Remember that if at all possible, ACLs should be applied in an outbound fashion.

Hope this is of some help!

0
 

Author Comment

by:seanselman
ID: 6445850
I have 2 different ACLs.  One for inbound and one for outbound.  There is a web server and a private corp. network behind the router.  I am doing reflexive out the serial and evaluating packets on the way back in.  This allows me to block anything to the private network which did not come from it.

I didn't know if I should blow away the list and re-key it. (Although it shouldn't matter....)
0
 
LVL 8

Expert Comment

by:scraig84
ID: 6445906
Gotcha.  The reflexive part at the end didn't register until I read it again.  Based on the response from the Cisco Decoder, it appears that it may be having some kind of problem with the timer that you set for your reflexive filter for when it should be removed.  Maybe for giggles you could change the timer value with the ip reflexive-list timeout command?  Also, I always distrust Cisco code (and so does the TAC as they will almost always tell you to upgrade the code as a first try to fix a problem - especially one that involves an error message being generated), so have you thought about upgrading the IOS to 12.2?  
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 6449361
This is not in response to the error message, but is just a suggestion:

Put your incoming ACL on serial interface in
Put your inspect commands on Ethernet interface
put your outgoing ACL on the Ethernet port in:

  Interface Etherenet 0/0
   ip access-group 101 in
   ip inspect fwout in

This follows the rule of putting the filters closest to the traffic that you want to filter.

http://www.cisco.com/warp/public/793/ios_fw/cbac2.html

0
 

Author Comment

by:seanselman
ID: 6453330
I changed the timeout to 600 sec and will wait to see if the problem still occurs.  FYI: running version 12.0(4).
0
 

Author Comment

by:seanselman
ID: 6453690
Another thing to note:

I have logging turned on for the inbound serial port of the external router.  I also have logging turned on for the inbound of the internal router's ethernet (private corp network).  I am currently getting an attack from an address 208.184.182.135 on port 80.  Most of the time, the request is blown off on the serial ACL.  About 5-10% of the time, I see it blown off on the ethernet of the internal router.  How is this possible?  Shouldn't the ACL on the serial be stopping the request before it even gets to the internal router!

Here is an example:

 External Router:
  208.184.182.135(80)->xxx.xxx.xxx.xxx(2570)
  208.184.182.135(80)->xxx.xxx.xxx.xxx(2571)
  208.184.182.135(80)->xxx.xxx.xxx.xxx(2572)
  208.184.182.135(80)->xxx.xxx.xxx.xxx(2574)
 Internal Router:
  208.184.182.135(80)->xxx.xxx.xxx.xxx(2573)

I have half a notion of just deny the address all together.  After all, anyone with a DNS of anal.holio.net can't be up to any good!
0
 
LVL 8

Expert Comment

by:scraig84
ID: 6453961
You know, anal could be short for analysis - not likely though!!

That issue is strange.  I am assuming you are applying both the inbound and outbound lists to the serial interface?  If so (which is what you said originally), you are correct that it shouldn't be reaching the internal router i.e. it shouldn't be getting denied by another interface's access list.  However, it technically has to be processed by the router in order for it to know whether or not it should be denied.  When you say its "blown off" at the Ethernet interface vs the Serial interface, what exactly are you seeing?

Couple things I would recommend.  Try updating your code if possible to see if this helps with some of the erroneous issues you are seeing.  Also, you say you changed the timer to 600 - I would say that may be a bit long.  The default is 5 seconds.  Even if this helps with the error message you were getting, keeping the list applied to the interface for 10 minutes after a session has ended I would think would be a bit of security hole.  Maybe try something around 60 or 120.


Note to Irmoore - CBAC from my understanding is using the Cisco Firewall feature set using inspect commands.  Going to the link you provided, it talked about only being able to use inbound ACLs etc for inspect commands and how to apply them.  This is a completely different issue than what seanselman is talking about.  Reflexive ACLs are session based filtering, similar (but more advanced) to using the "established" keyword in an extended list.  A good link is at:

http://www.cisco.com/univercd/cc/td/doc/product/software/ios113ed/113ed_cr/secur_c/scprt3/screflex.htm#xtocid87130

Although I agree with what you said, I don't see where the comments have any relavence to the question at hand.  I guess that's not my position to say, as seanselman may have found your comments useful, but I am confused and curious.  
0
 

Author Comment

by:seanselman
ID: 6454095
I changed the timeout to 120.

This is interesting....

On the external router, I have an ACL called serialin to block traffic on the inbound serial.

On the internal router, I have an ACL called Etherin to block traffic to the internal network.


<190>3139: 4d22h: %SEC-6-IPACCESSLOGP: list serialin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3225), 2 packets in 4-Sep 11:45:30.78 from xxx.xxx.xxx.xxx

<190>3141: 4d22h: %SEC-6-IPACCESSLOGP: list serialin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3230), 2 packets in 4-Sep 11:46:30.76 from xxx.xxx.xxx.xxx

<190>3143: 4d22h: %SEC-6-IPACCESSLOGP: list serialin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3506), 1 packet in 4-Sep 11:49:25.97 from xxx.xxx.xxx.xxx

<190>3144: 4d22h: %SEC-6-IPACCESSLOGP: list serialin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3404), 2 packets in 4-Sep 11:49:30.85 from xxx.xxx.xxx.xxx

<190>3148: 4d22h: %SEC-6-IPACCESSLOGP: list serialin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3506), 2 packets in 4-Sep 11:54:31.9 from xxx.xxx.xxx.xxx

<190>110: 4d04h: %SEC-6-IPACCESSLOGP: list etherin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3693), 1 packet in 4-Sep 12:25:53.10 from xxx.xxx.xxx.xxx

<190>3177: 4d23h: %SEC-6-IPACCESSLOGP: list serialin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3716), 1 packet in 4-Sep 12:27:5.22 from xxx.xxx.xxx.xxx

<190>111: 4d04h: %SEC-6-IPACCESSLOGP: list etherin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3693), 2 packets in 4-Sep 12:31:24.15 from xxx.xxx.xxx.xxx


I also decided to try to put an ACL on the outbound side of the external router's ethernet to block only this host.
0
 
LVL 8

Expert Comment

by:scraig84
ID: 6454180
Sorry, but I am a bit confused now.  You mention 2 routers an external and internal.  You mention an inbound list on the serial of the external, which is the interface I assume connects to the ISP.  You also mention an inbound on the Ethernet of the Internal router, which talks to the Ethernet of the external router?  Is there a DMZ between?   Where is the web server in question?  

Generally, a reflexive ACL has both an inbound and outbound list applied to 1 interface.  Which interface on which router has this list?  I am assuming the serial of the external.  If I have your configuration correct, you may have just written your list a bit wrong.  However, without knowing pretty initmate details about your setup including the lines of the list and where it is applied, it may be a bit hard to diagnose.  

Although putting a list to block that specific IP on the external may stop that box, what happens when the next Code Red box etc starts coming through?  As long as you see it getting blocked somewhere, I would let it through until you can block it correctly.  Otherwise, you may not know when you get your lists working correctly.
0
 

Author Comment

by:seanselman
ID: 6454215
Some clarification.....

I am an ISP that is located in the same building as another company.  All of the hardware is mine but the internal side of the network is "theirs".

I have a T1 from Sprint going into a 2610 (External) router.  On this serial, I have an ACL setup for reflexive lists and to allow only certain ports to certain addresses (web server, ftp, etc).

Using one of my real addresses, I have a 2611 (internal) router performing NAT (overloaded).  On the "external" ethernet port of this router, I have a reflexive ACL with everything but evaluated packets stopped.

So what I am seeing is the attacks being blown off on the serial of the 2610 most of the time and on the external ethernet of the 2611 every once in awhile.

There is also a switch and "real" network between these 2 routers which is "my" network.

Hope this helps....
0
 
LVL 8

Expert Comment

by:scraig84
ID: 6454242
That does help to clarify.  This address that you keep seeing - is it just attacking random IP addresses or is it possible something was intiated to this box at one time in order to open the reflexive list?  If this is possible, it may be a timing issue where the external router is keeping the list open longer than the internal router, hence passing traffic that is eventually denied.

Also, does the external router only allow reflexive traffic or are there any other permanent permits that could be letting certain traffic pass?  99 times out of 100, access list problems are a mistake in what is being allowed or not.

If this is still happening and the lists are verified etc, the only thing I can recommend again would be a code upgrade (never trust IOS code to do things exactly as documented!) and then possibly a follow up call with Cisco.
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 

Author Comment

by:seanselman
ID: 6454278
Both routers are doing reflexive.  Thanks for pointing out that potential problem.  I forgot to change the internal routers time. I haven't seen the error again....

The internal network is a contains people who would not be looking at this type of site (looks like a amature hacking site or something).  Therefore, I doubt they opened a hit this address.  If so, I would be shocked...

The external router does have some explicit permits to allow traffic to the web server, etc.  There is no permit to this address tho.  The attack is only attacking ports on the internal network address.  No others.

Could the router be "busy" and not check all of the packets??  I hope not!  This is a pretty quite network too.  Not alot of traffic.

All of the machines are running Norton 2001 virus software but maybe there is a trojan on the inside talking out?  I guess I can log all traffic coming out of the internal router to see if any are going to that address.....
0
 
LVL 8

Expert Comment

by:scraig84
ID: 6454308
The virus talking out etc was more of my line of thinking, but who knows???  Looking at URL logs never fails to give some level of shock value!!!

If the router is too busy, the packet should be dropped.  Otherwise, that would be a serious security flaw - overwhelm a router and pass unwanted traffic through!  However, its not like Cisco IOS has never had security flaws - which is why it pays to have it on the newest IOS if possible.

As far as the external list goes, that is another point I am curious about.  It makes me wonder if it is not always truly looking at all of the data in the IP header - may be passing just on source/destination ports and forgetting the destination address.  Something silly like that.

Anyway, sounds like you are on the right track with catching that one.  Glad to hear the timing took away the original error message.
0
 

Author Comment

by:seanselman
ID: 6454427
Actually, I got another error about 30 mins ago....

I think the shock value has just hit as well...I found another alias for 208.184.182.135....

After a made a call about how it looks like there may a virus on the inside the what the dns record stated as the address, the requests seemed to stop....hmmm....

I have a feeling the lists are working correctly and that someone was browsing somewhere they shouldn't have been.  The original error, I fear, is a bug in the IOS.
0
 

Author Comment

by:seanselman
ID: 6454444
The internal router is running 12.0(9) and doesn't seem to have this problem...
0
 
LVL 8

Expert Comment

by:scraig84
ID: 6454459
Same basic class of router - you may want to copy over the code tonight (or today if nobody minds a little down time) assuming you have the appropriate flash space etc (I can't say for sure without looking, but my assumption is the difference between 12.0(4) and 12.0(9) doesn't need additional flash or memory.

However, even if this doesn't fix it, that doesn't mean that it isn't an IOS bug, since they both may have the bug and only one router is experiencing the problem :(

0
 

Author Comment

by:seanselman
ID: 6454509
I have never copied in a new config.  How do I pull it from 1 router and put it on the other?

FYI->May not be able to do this for a couple of days....
0
 
LVL 8

Accepted Solution

by:
scraig84 earned 100 total points
ID: 6454628
Use TFTP.  You can either get a product from the web, such as Cisco's or TFTPD32 (the one I use - you can get it off of CNET).  You can then copy the file to the tftp server and then copy it back down.  

Otherwise, you can make the one router a tftp server with the tftp-server command.  For example type:

tftp-server flash:c2600-i-mz.122-1.bin

Obviously, you will need to substitute the actual file that you want to copy from the flash on the internal router.  Remember that you will probably need to modify your access list to let the external router use TFTP against the internal router.

Next on your external, from priveleged mode, type

copy tftp flash

and follow the directions.  It will ask you for the IP address of the server, the name of the file you are copying, the destination name and whether or not you want to erase flash before copying (you may have to if you don't have enough existing room in flash).  

If you are keeping multiple versions of the IOS in flash you may need to use the

boot system flash c2600-i-mz.122-1.bin

command from global config mode (again substituting in the actual file name) in order to use the correct IOS when rebooting.
0
 

Author Comment

by:seanselman
ID: 6454673
So I copy out the IOS from Flash the same as copying in a config from a file?
0
 
LVL 8

Expert Comment

by:scraig84
ID: 6454702
Well - yes, I suppose so.  Copying is always done inbound.  So you are still copying it in and you are using a TFTP server as the source.  In the case I mentioned above, you just happen to be using the internal router as your TFTP server.  But yes, you can copy in a config from a TFTP server as well, so in that way it is exactly the same.
0
 

Author Comment

by:seanselman
ID: 6470893
I upgraded the IOS to 12.0(9) and the error is still occurring.  Later today, I will re-key the ACL and see if that makes any difference.
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 7877593
No comment has been added lately. It appears this question has been abandonded so it's time to clean up this TA.
I will leave a recommendation in the Cleanup topic area for this question:

I recommend: points to scraig84

if there is any objection or other expert commentary to this recommendation then please post in here within 7 days.

PLEASE DO NOT ACCEPT THIS COMMENT AS AN ANSWER!

thanks,
lrmoore
EE Cleanup Volunteer
---------------------
0
 

Expert Comment

by:modulo
ID: 7958143
Finalized as proposed

modulo

Community Support Moderator
Experts Exchange
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Suggested Solutions

New Server 172.16.200.2  was moved from behind Router R2 f0/1 to behind router R1 int f/01 and has now address 172.16.100.2. But we want users still to be able to connected to it by old IP. How to do it ? We can used destination NAT (DNAT).  In DNAT…
Shadow IT is coming out of the shadows as more businesses are choosing cloud-based applications. It is now a multi-cloud world for most organizations. Simultaneously, most businesses have yet to consolidate with one cloud provider or define an offic…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now