Linux Server is saturating the IP table..

We have Linux Server running few VMs, Apache, Application Server, My SQL

Now it showing: "sysctl grep conntrack" command

ip_conntrack: table full, dropping packet
printk: 18 messages suppressed

(those 2 lines over and over but once 24 messages than 13, than 4 with no end in the counsel)

All computers and devices at times are getting conflict IP, at times will not get an IP from our Cisco Firewall/Server. But if I phisicly I disconnect the server from the switch, restart the Cisco than all is well.

Before this happens the network was working on and off for a wile until today it will not budge (IP saturation). Few months ago the Apache was restarted  and the Application Server was resolving fine, all was well. Between 100-150 IPs were successful dedicated throughout the network. Our table does support over 1024 IPs. It looked like keeps increasing little by little over time.

Our head office is not allowing us to restart the Linux Server, Since rebuilding the database will take months of work.. (so no restart)..

Please, anyone can suggest of commands or ideas I can try to resolve this issue?


Thanks
poposkiAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

gheistCommented:
sysctl net.nf_conntrack_max=200000
(default is 64k that fits in about 16-18MB of RAM, repeated doubling will not hurt much.)

Do you track UDP states like DNS? That is kind of self-dos...
0
poposkiAuthor Commented:
I will try that when I am back in the office. But only way to access the server is only from the counsel station since SSH, putty are useless in my senerios,  I can't phisicly connect it to the switch since will crush our network (saturate the IP table)..

Also, much of those self do's I can not do since all firewall/dhcp/policcies/DNS, are managed only from head office. But it seems the firewall/router is back to normal after I disconnect..

Restarting some of the services, would it help?
0
poposkiAuthor Commented:
Also, how do I stop the message that keeps repeating over and over in the counsel so I can login?
***
ip_conntrack: table full, dropping packet
printk: 18 messages suppressed
****
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

gheistCommented:
Ctrl-Alt-F2
0
gheistCommented:
No need to restart anything. You need to find offending (likely UDP) traffic.
0
poposkiAuthor Commented:
Ctrl Alt F2 does not seem to work. Any other suggestions?

It's Redhat with 2 different virtual machines software in use: - vmware & - virtualbox
0
gheistCommented:
virtualbox - hotkey+F2
vmware - Ctrl-Alt-F2 when focused in window
0
poposkiAuthor Commented:
The user we always use is "ssupport" is not excepting your command..

sysctl net.nf_conntrack_max=200000

another user account that will agnowledge the command?
0
gheistCommented:
sysctl is adjusted by ROOT.
0
poposkiAuthor Commented:
I am hesitant to increase the conntract table to 2mil.. Thank God didn't. 64k should be sufficient to host the load of work. If we are managing Google or big demand site, than 2 mil is sufficient, for heavy demand..

In short your advise helped me to look at that conntrack table, and I found a script that was implemented by a some one (hacker - serious security bridged) But that script kept on pounding on the firewall and increased the demand of traffic. Even if increased to 2 mil would have saturated in seconds..

Anyways, thank you so much by helping me to start looking at that area..
0
gheistCommented:
As you see 64k is NOT sufficint.
make it 128k so you have some breathing space (it will take 16MB more RAM)
Run 10min or 100000packets long tcpdump and check for some anomaly.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
poposkiAuthor Commented:
sorry, took me some time to reply. The service was stopped and the UDP count from 65,855 went down to 522.. Thanks again for pointing me to the right place.
0
poposkiAuthor Commented:
PLEASE look at the entire conversation to help you in this situation. But gheist quickly defined and recognized where the problem could be.. Thank you ghheist
0
gheistCommented:
was it ntp amplification or dns amplification?
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.