Large Number of NAT Translations

We have about 800 hosts on our network and during peak times I've seen (show ip nat stat) the the translations go over 60,000 which causes serious performance issues. When I clear the translation (clear ip nat tran *) everything works fine again. We use PAT with 1 public IP for all non-server hosts.

I've dumped the translations and used a pivot table to determine the culprits which are typically a dozen or so machines with thousands of connections. Usually the issue is P2P or a virus.

What I don't understand is that the translations seem to hang around for a long time even after I know the internal host is shut down. For example, I ping the host and get no response and yet translations for the host IP are still in the table for many hours. I assume what I need to do is set a shorter timeout for translations. I included a small sample of typical translations below with our public IP's altered.

I looked at this page (http://www.ciscopress.com/articles/article.asp?p=25273&seqNum=5) and see there are a number of different timeouts that can be set. The page makes a major distinction between port specific translations and non-port specific translations with non-port specific defaulting to 1 day and port specific ending in 1 minute. I confess to not understand the difference between non-port specific and port specific, I thought all PAT translations were port specific.

I have two questions:

1) Which timeouts should I set? "ip nat translation timeout" or "ip nat tranlation port-timeout tcp" or ... ?

2) What are some best practices regarding the length of timeout? What would be the negative consequences of setting the timeout too short?

Thanks!
Pro Inside global         Inside local          Outside local         Outside global
tcp 24.111.127.1:63115    10.40.100.224:63115   192.221.114.126:80    192.221.114.126:80
tcp 24.111.127.1:3800     10.40.100.229:2249    72.21.211.188:80      72.21.211.188:80
tcp 24.111.127.1:1338     10.40.100.229:2530    216.246.75.99:80      216.246.75.99:80
tcp 24.111.127.1:1931     10.40.100.229:2531    216.246.75.97:80      216.246.75.97:80
tcp 24.111.127.1:2273     10.40.100.229:2535    74.125.226.130:80     74.125.226.130:80
tcp 24.111.127.1:2410     10.40.100.229:2537    74.125.226.128:80     74.125.226.128:80

Open in new window

LVL 4
glebnAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

harbor235Commented:


A potential safeguard would be to rate limit your nat translations;

http://www.cisco.com/en/US/docs/ios/12_3t/12_3t4/feature/guide/gt_natrl.html

Also, NAT is memory intensive, how does your memory utilization look when there is degradation?
If you are going to run NAt in that fashion youmay need to beef your memory, i would max it out.

Do you have CEF enabled?


I would look at these two commands:

"ip nat translation timeout "  default is 24hrs
"ip nat translation max-entries" default is no limit

harbor235 ;}
glebnAuthor Commented:
harbor235,

Thanks for the response!

1) From everything I know about CEF it is overkill if not inappropriate for our single router and network setup. Do you think it is appropriate? We are a school with 800 internal hosts and essentially one Internet connection using a Cisco 3800 series router.

2) What would be the potential negative effect of setting the "ip nat translation timeout" to something as small as 4 hours or even 1 hour? If I lower this number I want to have an idea what I should keep my eye on to make sure I didn't set it to too short of a time.

3) What about the other timeouts (e.g. port-timeout tcp), are any of them potentially appropriate?

4) Lastly, I'm a little confused about the max-entries settings. Does this set the limit for all of the entries (i.e. the number that appears for dynamic entries in the first line of ip nat stat) or is a per host limit? My reading of the documentation indicates it is the former, but all the examples I see use max-entries of a few hundred which seems to me absurdly low for a network of any size. When I clear translations we are back up to thousands of translations in a few minutes.

harbor235Commented:


CEF is not overkill, it is the preferred packet forwarding architecture for Cisco network gear.
I definitiely woudl enables CEF, else you reduce the perfromance of the 3800 utilizing the CPU and not ASIC to forward traffic.

First off you need to look into the traffic charcteristics of your internal users. If they are mainly web users than reducing the timeout is a non issue because of the sporadic and random nature of their flows. Side effect they will need to restablish and existing flows which will be non-recognizable to the end users. Now if you are doing lots of database syncronization betweek two locations and need the flow to stay established to maximize efficency of your databases, then I would maintain a higher timeout value.

So it depends on your traffic patterns and the use case, make sense?

Max entries sets the limit for all entries, so this would help mitigate virus or worm behavior from grabbing aas many conenctions/translations as possible effetively DOS'ng your internal users from accessing external resources.  So, number of (internal hosts)x (average connections per user) = max entries.

The key here is that if you collect infromation about your traffic flows and perfrom trending you can gues a good max entry number. Remember this needs to be reevaluated and teaked periodically. It is not set and forget.

harbor235 ;}

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
glebnAuthor Commented:
Thanks for explanations, they were very helpful!
harbor235Commented:

Sorry for the spelling and grammar mistakes, i need to take more time reviewing my words before submitting, DOH !!!!


harbor235 ;}
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Routers

From novice to tech pro — start learning today.