Cisco ACL Error Message

Interesting one.  I went to Cisco's Error Message Decoder and this is what it said when I pasted in your error:


A process can register to be notified when various events occur in the router. A registered timer is expired, and the timer value is unchanged after the process was executed two successive times.
Recommended Action: Copy the error message exactly as it appears on the console or in the system log, call your Cisco technical support representative, and provide the representative with the gathered information.


Helpful!  I would at least check with the Cisco TAC if nobody else is able to find much on this.  Even if it is not affecting performance, it wouldn't log something like that unless it meant something.

Also, maybe its just the way that you worded your question, but are you truly applying the same ACL to both inbound and outbound traffic?  I'm not saying this is unheard of, but it is generally not needed or desired.  Remember that if at all possible, ACLs should be applied in an outbound fashion.

Hope this is of some help!

I have 2 different ACLs.  One for inbound and one for outbound.  There is a web server and a private corp. network behind the router.  I am doing reflexive out the serial and evaluating packets on the way back in.  This allows me to block anything to the private network which did not come from it.

I didn't know if I should blow away the list and re-key it. (Although it shouldn't matter....)

Gotcha.  The reflexive part at the end didn't register until I read it again.  Based on the response from the Cisco Decoder, it appears that it may be having some kind of problem with the timer that you set for your reflexive filter for when it should be removed.  Maybe for giggles you could change the timer value with the ip reflexive-list timeout command?  Also, I always distrust Cisco code (and so does the TAC as they will almost always tell you to upgrade the code as a first try to fix a problem - especially one that involves an error message being generated), so have you thought about upgrading the IOS to 12.2?  

This is not in response to the error message, but is just a suggestion:

Put your incoming ACL on serial interface in
Put your inspect commands on Ethernet interface
put your outgoing ACL on the Ethernet port in:

  Interface Etherenet 0/0
   ip access-group 101 in
   ip inspect fwout in

This follows the rule of putting the filters closest to the traffic that you want to filter.

http://www.cisco.com/warp/public/793/ios_fw/cbac2.html

I changed the timeout to 600 sec and will wait to see if the problem still occurs.  FYI: running version 12.0(4).

Another thing to note:

I have logging turned on for the inbound serial port of the external router.  I also have logging turned on for the inbound of the internal router's ethernet (private corp network).  I am currently getting an attack from an address 208.184.182.135 on port 80.  Most of the time, the request is blown off on the serial ACL.  About 5-10% of the time, I see it blown off on the ethernet of the internal router.  How is this possible?  Shouldn't the ACL on the serial be stopping the request before it even gets to the internal router!

Here is an example:

 External Router:
  208.184.182.135(80)->xxx.xxx.xxx.xxx(2570)
  208.184.182.135(80)->xxx.xxx.xxx.xxx(2571)
  208.184.182.135(80)->xxx.xxx.xxx.xxx(2572)
  208.184.182.135(80)->xxx.xxx.xxx.xxx(2574)
 Internal Router:
  208.184.182.135(80)->xxx.xxx.xxx.xxx(2573)

I have half a notion of just deny the address all together.  After all, anyone with a DNS of anal.holio.net can't be up to any good!

You know, anal could be short for analysis - not likely though!!

That issue is strange.  I am assuming you are applying both the inbound and outbound lists to the serial interface?  If so (which is what you said originally), you are correct that it shouldn't be reaching the internal router i.e. it shouldn't be getting denied by another interface's access list.  However, it technically has to be processed by the router in order for it to know whether or not it should be denied.  When you say its "blown off" at the Ethernet interface vs the Serial interface, what exactly are you seeing?

Couple things I would recommend.  Try updating your code if possible to see if this helps with some of the erroneous issues you are seeing.  Also, you say you changed the timer to 600 - I would say that may be a bit long.  The default is 5 seconds.  Even if this helps with the error message you were getting, keeping the list applied to the interface for 10 minutes after a session has ended I would think would be a bit of security hole.  Maybe try something around 60 or 120.


Note to Irmoore - CBAC from my understanding is using the Cisco Firewall feature set using inspect commands.  Going to the link you provided, it talked about only being able to use inbound ACLs etc for inspect commands and how to apply them.  This is a completely different issue than what seanselman is talking about.  Reflexive ACLs are session based filtering, similar (but more advanced) to using the "established" keyword in an extended list.  A good link is at:

http://www.cisco.com/univercd/cc/td/doc/product/software/ios113ed/113ed_cr/secur_c/scprt3/screflex.htm#xtocid87130

Although I agree with what you said, I don't see where the comments have any relavence to the question at hand.  I guess that's not my position to say, as seanselman may have found your comments useful, but I am confused and curious.  

I changed the timeout to 120.

This is interesting....

On the external router, I have an ACL called serialin to block traffic on the inbound serial.

On the internal router, I have an ACL called Etherin to block traffic to the internal network.


<190>3139: 4d22h: %SEC-6-IPACCESSLOGP: list serialin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3225), 2 packets in 4-Sep 11:45:30.78 from xxx.xxx.xxx.xxx

<190>3141: 4d22h: %SEC-6-IPACCESSLOGP: list serialin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3230), 2 packets in 4-Sep 11:46:30.76 from xxx.xxx.xxx.xxx

<190>3143: 4d22h: %SEC-6-IPACCESSLOGP: list serialin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3506), 1 packet in 4-Sep 11:49:25.97 from xxx.xxx.xxx.xxx

<190>3144: 4d22h: %SEC-6-IPACCESSLOGP: list serialin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3404), 2 packets in 4-Sep 11:49:30.85 from xxx.xxx.xxx.xxx

<190>3148: 4d22h: %SEC-6-IPACCESSLOGP: list serialin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3506), 2 packets in 4-Sep 11:54:31.9 from xxx.xxx.xxx.xxx

<190>110: 4d04h: %SEC-6-IPACCESSLOGP: list etherin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3693), 1 packet in 4-Sep 12:25:53.10 from xxx.xxx.xxx.xxx

<190>3177: 4d23h: %SEC-6-IPACCESSLOGP: list serialin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3716), 1 packet in 4-Sep 12:27:5.22 from xxx.xxx.xxx.xxx

<190>111: 4d04h: %SEC-6-IPACCESSLOGP: list etherin denied tcp 208.184.182.135(80) -> xxx.xxx.xxx.xxx(3693), 2 packets in 4-Sep 12:31:24.15 from xxx.xxx.xxx.xxx


I also decided to try to put an ACL on the outbound side of the external router's ethernet to block only this host.

Sorry, but I am a bit confused now.  You mention 2 routers an external and internal.  You mention an inbound list on the serial of the external, which is the interface I assume connects to the ISP.  You also mention an inbound on the Ethernet of the Internal router, which talks to the Ethernet of the external router?  Is there a DMZ between?   Where is the web server in question?  

Generally, a reflexive ACL has both an inbound and outbound list applied to 1 interface.  Which interface on which router has this list?  I am assuming the serial of the external.  If I have your configuration correct, you may have just written your list a bit wrong.  However, without knowing pretty initmate details about your setup including the lines of the list and where it is applied, it may be a bit hard to diagnose.  

Although putting a list to block that specific IP on the external may stop that box, what happens when the next Code Red box etc starts coming through?  As long as you see it getting blocked somewhere, I would let it through until you can block it correctly.  Otherwise, you may not know when you get your lists working correctly.

Some clarification.....

I am an ISP that is located in the same building as another company.  All of the hardware is mine but the internal side of the network is "theirs".

I have a T1 from Sprint going into a 2610 (External) router.  On this serial, I have an ACL setup for reflexive lists and to allow only certain ports to certain addresses (web server, ftp, etc).

Using one of my real addresses, I have a 2611 (internal) router performing NAT (overloaded).  On the "external" ethernet port of this router, I have a reflexive ACL with everything but evaluated packets stopped.

So what I am seeing is the attacks being blown off on the serial of the 2610 most of the time and on the external ethernet of the 2611 every once in awhile.

There is also a switch and "real" network between these 2 routers which is "my" network.

Hope this helps....

That does help to clarify.  This address that you keep seeing - is it just attacking random IP addresses or is it possible something was intiated to this box at one time in order to open the reflexive list?  If this is possible, it may be a timing issue where the external router is keeping the list open longer than the internal router, hence passing traffic that is eventually denied.

Also, does the external router only allow reflexive traffic or are there any other permanent permits that could be letting certain traffic pass?  99 times out of 100, access list problems are a mistake in what is being allowed or not.

If this is still happening and the lists are verified etc, the only thing I can recommend again would be a code upgrade (never trust IOS code to do things exactly as documented!) and then possibly a follow up call with Cisco.

Both routers are doing reflexive.  Thanks for pointing out that potential problem.  I forgot to change the internal routers time. I haven't seen the error again....

The internal network is a contains people who would not be looking at this type of site (looks like a amature hacking site or something).  Therefore, I doubt they opened a hit this address.  If so, I would be shocked...

The external router does have some explicit permits to allow traffic to the web server, etc.  There is no permit to this address tho.  The attack is only attacking ports on the internal network address.  No others.

Could the router be "busy" and not check all of the packets??  I hope not!  This is a pretty quite network too.  Not alot of traffic.

All of the machines are running Norton 2001 virus software but maybe there is a trojan on the inside talking out?  I guess I can log all traffic coming out of the internal router to see if any are going to that address.....

The virus talking out etc was more of my line of thinking, but who knows???  Looking at URL logs never fails to give some level of shock value!!!

If the router is too busy, the packet should be dropped.  Otherwise, that would be a serious security flaw - overwhelm a router and pass unwanted traffic through!  However, its not like Cisco IOS has never had security flaws - which is why it pays to have it on the newest IOS if possible.

As far as the external list goes, that is another point I am curious about.  It makes me wonder if it is not always truly looking at all of the data in the IP header - may be passing just on source/destination ports and forgetting the destination address.  Something silly like that.

Anyway, sounds like you are on the right track with catching that one.  Glad to hear the timing took away the original error message.

Actually, I got another error about 30 mins ago....

I think the shock value has just hit as well...I found another alias for 208.184.182.135....

After a made a call about how it looks like there may a virus on the inside the what the dns record stated as the address, the requests seemed to stop....hmmm....

I have a feeling the lists are working correctly and that someone was browsing somewhere they shouldn't have been.  The original error, I fear, is a bug in the IOS.

The internal router is running 12.0(9) and doesn't seem to have this problem...

Same basic class of router - you may want to copy over the code tonight (or today if nobody minds a little down time) assuming you have the appropriate flash space etc (I can't say for sure without looking, but my assumption is the difference between 12.0(4) and 12.0(9) doesn't need additional flash or memory.

However, even if this doesn't fix it, that doesn't mean that it isn't an IOS bug, since they both may have the bug and only one router is experiencing the problem :(

I have never copied in a new config.  How do I pull it from 1 router and put it on the other?

FYI->May not be able to do this for a couple of days....

So I copy out the IOS from Flash the same as copying in a config from a file?

Well - yes, I suppose so.  Copying is always done inbound.  So you are still copying it in and you are using a TFTP server as the source.  In the case I mentioned above, you just happen to be using the internal router as your TFTP server.  But yes, you can copy in a config from a TFTP server as well, so in that way it is exactly the same.

I upgraded the IOS to 12.0(9) and the error is still occurring.  Later today, I will re-key the ACL and see if that makes any difference.

No comment has been added lately. It appears this question has been abandonded so it's time to clean up this TA.
I will leave a recommendation in the Cleanup topic area for this question:

I recommend: points to scraig84

if there is any objection or other expert commentary to this recommendation then please post in here within 7 days.

PLEASE DO NOT ACCEPT THIS COMMENT AS AN ANSWER!

thanks,
lrmoore
EE Cleanup Volunteer
---------------------

Finalized as proposed

modulo

Community Support Moderator
Experts Exchange