Linux Server is saturating the IP table..

We have Linux Server running few VMs, Apache, Application Server, My SQL

Now it showing: "sysctl grep conntrack" command

ip_conntrack: table full, dropping packet
printk: 18 messages suppressed

(those 2 lines over and over but once 24 messages than 13, than 4 with no end in the counsel)

All computers and devices at times are getting conflict IP, at times will not get an IP from our Cisco Firewall/Server. But if I phisicly I disconnect the server from the switch, restart the Cisco than all is well.

Before this happens the network was working on and off for a wile until today it will not budge (IP saturation). Few months ago the Apache was restarted  and the Application Server was resolving fine, all was well. Between 100-150 IPs were successful dedicated throughout the network. Our table does support over 1024 IPs. It looked like keeps increasing little by little over time.

Our head office is not allowing us to restart the Linux Server, Since rebuilding the database will take months of work.. (so no restart)..

Please, anyone can suggest of commands or ideas I can try to resolve this issue?


Thanks
poposkiAsked:
Who is Participating?
 
gheistCommented:
As you see 64k is NOT sufficint.
make it 128k so you have some breathing space (it will take 16MB more RAM)
Run 10min or 100000packets long tcpdump and check for some anomaly.
0
 
gheistCommented:
sysctl net.nf_conntrack_max=200000
(default is 64k that fits in about 16-18MB of RAM, repeated doubling will not hurt much.)

Do you track UDP states like DNS? That is kind of self-dos...
0
 
poposkiAuthor Commented:
I will try that when I am back in the office. But only way to access the server is only from the counsel station since SSH, putty are useless in my senerios,  I can't phisicly connect it to the switch since will crush our network (saturate the IP table)..

Also, much of those self do's I can not do since all firewall/dhcp/policcies/DNS, are managed only from head office. But it seems the firewall/router is back to normal after I disconnect..

Restarting some of the services, would it help?
0
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
poposkiAuthor Commented:
Also, how do I stop the message that keeps repeating over and over in the counsel so I can login?
***
ip_conntrack: table full, dropping packet
printk: 18 messages suppressed
****
0
 
gheistCommented:
Ctrl-Alt-F2
0
 
gheistCommented:
No need to restart anything. You need to find offending (likely UDP) traffic.
0
 
poposkiAuthor Commented:
Ctrl Alt F2 does not seem to work. Any other suggestions?

It's Redhat with 2 different virtual machines software in use: - vmware & - virtualbox
0
 
gheistCommented:
virtualbox - hotkey+F2
vmware - Ctrl-Alt-F2 when focused in window
0
 
poposkiAuthor Commented:
The user we always use is "ssupport" is not excepting your command..

sysctl net.nf_conntrack_max=200000

another user account that will agnowledge the command?
0
 
gheistCommented:
sysctl is adjusted by ROOT.
0
 
poposkiAuthor Commented:
I am hesitant to increase the conntract table to 2mil.. Thank God didn't. 64k should be sufficient to host the load of work. If we are managing Google or big demand site, than 2 mil is sufficient, for heavy demand..

In short your advise helped me to look at that conntrack table, and I found a script that was implemented by a some one (hacker - serious security bridged) But that script kept on pounding on the firewall and increased the demand of traffic. Even if increased to 2 mil would have saturated in seconds..

Anyways, thank you so much by helping me to start looking at that area..
0
 
poposkiAuthor Commented:
sorry, took me some time to reply. The service was stopped and the UDP count from 65,855 went down to 522.. Thanks again for pointing me to the right place.
0
 
poposkiAuthor Commented:
PLEASE look at the entire conversation to help you in this situation. But gheist quickly defined and recognized where the problem could be.. Thank you ghheist
0
 
gheistCommented:
was it ntp amplification or dns amplification?
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.