poposki
asked on
Linux Server is saturating the IP table..
We have Linux Server running few VMs, Apache, Application Server, My SQL
Now it showing: "sysctl grep conntrack" command
ip_conntrack: table full, dropping packet
printk: 18 messages suppressed
(those 2 lines over and over but once 24 messages than 13, than 4 with no end in the counsel)
All computers and devices at times are getting conflict IP, at times will not get an IP from our Cisco Firewall/Server. But if I phisicly I disconnect the server from the switch, restart the Cisco than all is well.
Before this happens the network was working on and off for a wile until today it will not budge (IP saturation). Few months ago the Apache was restarted and the Application Server was resolving fine, all was well. Between 100-150 IPs were successful dedicated throughout the network. Our table does support over 1024 IPs. It looked like keeps increasing little by little over time.
Our head office is not allowing us to restart the Linux Server, Since rebuilding the database will take months of work.. (so no restart)..
Please, anyone can suggest of commands or ideas I can try to resolve this issue?
Thanks
Now it showing: "sysctl grep conntrack" command
ip_conntrack: table full, dropping packet
printk: 18 messages suppressed
(those 2 lines over and over but once 24 messages than 13, than 4 with no end in the counsel)
All computers and devices at times are getting conflict IP, at times will not get an IP from our Cisco Firewall/Server. But if I phisicly I disconnect the server from the switch, restart the Cisco than all is well.
Before this happens the network was working on and off for a wile until today it will not budge (IP saturation). Few months ago the Apache was restarted and the Application Server was resolving fine, all was well. Between 100-150 IPs were successful dedicated throughout the network. Our table does support over 1024 IPs. It looked like keeps increasing little by little over time.
Our head office is not allowing us to restart the Linux Server, Since rebuilding the database will take months of work.. (so no restart)..
Please, anyone can suggest of commands or ideas I can try to resolve this issue?
Thanks
ASKER
I will try that when I am back in the office. But only way to access the server is only from the counsel station since SSH, putty are useless in my senerios, I can't phisicly connect it to the switch since will crush our network (saturate the IP table)..
Also, much of those self do's I can not do since all firewall/dhcp/policcies/DN
Restarting some of the services, would it help?
Also, much of those self do's I can not do since all firewall/dhcp/policcies/DN
Restarting some of the services, would it help?
ASKER
Also, how do I stop the message that keeps repeating over and over in the counsel so I can login?
***
ip_conntrack: table full, dropping packet
printk: 18 messages suppressed
****
***
ip_conntrack: table full, dropping packet
printk: 18 messages suppressed
****
Ctrl-Alt-F2
No need to restart anything. You need to find offending (likely UDP) traffic.
ASKER
Ctrl Alt F2 does not seem to work. Any other suggestions?
It's Redhat with 2 different virtual machines software in use: - vmware & - virtualbox
It's Redhat with 2 different virtual machines software in use: - vmware & - virtualbox
virtualbox - hotkey+F2
vmware - Ctrl-Alt-F2 when focused in window
vmware - Ctrl-Alt-F2 when focused in window
ASKER
The user we always use is "ssupport" is not excepting your command..
sysctl net.nf_conntrack_max=20000
another user account that will agnowledge the command?
sysctl net.nf_conntrack_max=20000
another user account that will agnowledge the command?
sysctl is adjusted by ROOT.
ASKER
I am hesitant to increase the conntract table to 2mil.. Thank God didn't. 64k should be sufficient to host the load of work. If we are managing Google or big demand site, than 2 mil is sufficient, for heavy demand..
In short your advise helped me to look at that conntrack table, and I found a script that was implemented by a some one (hacker - serious security bridged) But that script kept on pounding on the firewall and increased the demand of traffic. Even if increased to 2 mil would have saturated in seconds..
Anyways, thank you so much by helping me to start looking at that area..
In short your advise helped me to look at that conntrack table, and I found a script that was implemented by a some one (hacker - serious security bridged) But that script kept on pounding on the firewall and increased the demand of traffic. Even if increased to 2 mil would have saturated in seconds..
Anyways, thank you so much by helping me to start looking at that area..
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
sorry, took me some time to reply. The service was stopped and the UDP count from 65,855 went down to 522.. Thanks again for pointing me to the right place.
ASKER
PLEASE look at the entire conversation to help you in this situation. But gheist quickly defined and recognized where the problem could be.. Thank you ghheist
was it ntp amplification or dns amplification?
(default is 64k that fits in about 16-18MB of RAM, repeated doubling will not hurt much.)
Do you track UDP states like DNS? That is kind of self-dos...