Tags:
Linux, CentOS, 5, Webserver, apache, php5, MySQL
I have a dedicated server that I pay for yearly and up untill now a gig of RAM and a single processor seemed to be plenty. Sadly some **** with an overactive botnet and a need to spam the crap out of everything he/she/it sees via http is keeping my server in a state of utter uselessness.
Other specs that might be important CentOS 5, webmin 1.4, Apache2.2, MySQL+PHP5+etc, 1G RAM, 80G HDD (x2 in RAID array) (12G used), 2.66 GHz CeleronD (755)... all-in-all just a basic entry level dedi.
I have a MySQL connection limit of 150 as at 100 the MySQL server was running out of connections. This indicates that there must be over 100 connection in any given few seconds (I'm guessing) which gives a spam rate of at least ? 25+ hits a second? (I'm utterly guessing) going on for 3 days.
Given that most domains the server hosts contain blogs (and little else) it is likely to be trackback spam or comment spam. It is http I know that much.
When I use netstat or iptstate it of course resolves the target IP as the URL/domain of the server rather than the URL/domain of the attack. It does show me that a lot of proxies are being used to assault my server though.
I tried mod_evasive but even though I am quite sure it is set up correctly I can not get it react to anything. I also tried (D)Dos Deflate with fairly aggressive settings and that has done little if anything.
Short of setting a cron to restart apache every fifteen minutes it really does look like I need to identify the target and deactivate it.
I'm not exactly any expert at this but up until now there has not been an issue with a computer that I have ever been beaten by. So this is a bit of blow to my ego but at this stage (day 3) I figure that I'm just going to have to accept that I have lost a domain for the time being and "pull it" to return some sanity to the system.
Well, that mod_security was an error and no mistake (!) I've removed it again and have by a process of checking each site over found a blog with 30,000 comments in it. I'm better all but 3 are spam as the site is not in use all that much and has a few hits a month. (it's my wife's site).
I've disabled the commenting so hopefully the spammers will give up sooner or later. But I still need to know how to locate the victim virtual host because there is bound to be a next time. A good answer and I will find a way to give you extra points because I am serious about my need to get this issue addressed.
the default it 60. This will, may, reduce how long you have sockets in closed wait status. You will need to set this after each re-boot. You can add to your /etc/sysctl.conf file.
have you analysed your access and error log, does it show any illegal requests or other anomalies? If so, there is a good chance to get rid of some with modsecurity.
Also set httpd.conf Timeout 5 KeepAliveTimeout 2 which should reduce the amount of simutaneous connection in CLOSE_WAIT state.
ahoffmann, I found a single target of spam and added a script called bad_behaviour which monitors for badly formed http headers and know bad bots. It rejected two calls inside the time for the plugin install and me to navigate to the admin page of that plugin.
I tried mod_security but the yum repository it recommends worked but caused a bunch of error messages when I restarted apache. I'm not too keen to be that near to the bleeding edge on what is supposed to be my production server so I pulled it back off. There were some build it yourself instructions on the mod_security website but I found them a bit overwhelming and not 100% clear.
I installed mod_evasive and had to get the apache whatnot something or the other ...xs installed (even though the mod it ships with was compiled in...) and I managed to set it up all on my own (I'm a big boy now) so it's not that my system intimidates me. Sadly mod_evasive was useless and could not even evade it's own uselessness. It failed the test it ships with.
Like an onion each layer is helping keep things going. I just added mod_limitipconn and set a max of one conn to the action.php file (which the blogs on the domains use for comments, trackbacks and most other user interaction) to just a single connection per IP. While this has stemmed the flow and kept things from dieing it has not licked it totally.
Thanks giltjr. I'm going to look at tweaking the CLOSE_WAIT and will report back.
Was: Timeout 120 Now: Timeout 5 Was: KeepAlive Off Now: KeepAlive On Was: KeepAliveTimeout 10 Now: KeepAliveTimeout 5
I am figureing that as MOD_Evasive keeps data in the child that turnign this on will enhance things and allow evasive to do it's thang... ? Would I be right?
> .. but now it is killing the MySQL server.. probably you should use some connection pooling (sorry, don't know of a PHP module for this)
> .. mod_security website but I found them a bit overwhelming .. modsecurity is nothing to do plug&play, click&go. modsecurity is -as it name imples- security, and security is a process not a product. Said this, you need to get used to modsecurity and how it works, then you can do your basic configuration. After that you probably want to install the core rule set and continue to improve your setting. As you're using PHP, modsecurity should be a first step to make your (PHP)web application a bit more stable (as PHP apps are known to have lot of vulerabilities, usually).
I installed a script called bad-behaviour which is written in php. I placed to instances on what seemed to be the hardest hit areas. The load average dropped right back.
Also while checking my settings I noticed that keep alive was disabled so I enabled it thus allowing mod_evasive to kick in and work (it maintains it's information in the active thread)? It now puts a temporary block on 30 to 40 IP addresses every day.
As for mod_security I've yet to install it and have it not cripple apache and stop it loading. I'm no wizard at the advanced shell usage and only know about the methods I have used before. As no YUM package (that works) exists I'm beaten before I start but I seem to have the problem licked with php.
> PHP apps are known to have lot of vulerabilities, usually That's quite a broad generalization and somewhat true - most tutorials teach shocking bad habits and php is sufficiently flexible to allow you to create the most amazing vulnerabilities if you don't know what you are doing. I only run tried and tested apps or stuff I've written myself. I have a good collection of tools that I build from designed to make the development process secure and easy to debug.
The problem was that one or more spammers were hitting two or more blogs with a massive array of junk comments. The anti-spam methods were stopping almost all of them (aside from one blog where they had not been activated (but are now)) the problem was that by the time the anti-spam plug-ins were doing their thing the entire app had loaded and made several calls to the MySQL server and at the rate they were hitting me this killed the server.
If I really had that much real interest I'd be bigger than digg or slashdot and would probably have a team of people and lots of servers... sadly I'm not that big.
Fortunately bad-behaviour runs before anything else and returns a simple HTTP error code and a human readable explanation of why they were blocked. I managed to trigger it myself when using a Firefox addon that gives you a different user agent - the script spotted that I was not really Google (who I was pretending to be) and stopped me there and then.
As far as I can tell the server is still hammered day and night but now it fails to hurt things.
So how do I install mod_security - I don't suppose you could baby step me through it - I'll be using PuTTY and the shell will be BASH?
> how do I install mod_security install ModSecurity 2.x, then get the core rule set and install it also This is straight forward as described in the README/INSTALL files. I'd recommend to have all rules files in a single directory, then include these files in httpd.conf like include rules/*.conf (this make some changes more comfortable later)
When you get the core rule set, edit modsecurity_crs_10_config.conf and set SecRuleEngine DetectionOnly SecAuditLogType Serial SecAuditLog logs/modsec_audit.log SecDebugLog logs/modsec_debug.log SecDebugLogLevel 3
then restart apache. Browse your application(s) and watch the logfiles (as defined above). Most likely you get some messages that request are blocked for various reasons. Each such message contains the ID of the rule which caused the blocking. You have to analyse the request and the rule to identify if the blocking was legal or a false positive. When you come to the conclusion that it is a false positive, then create a new rules file and add
SecRuleRemoveByID --id here--
Let me know if you need help here, then I'll give more detailed instructions.
ahoffmann - thanks for getting back to me. Don't forget that I'm doing this by PuTTY so the instruction "install it by installing it (read the text file)" has not got me anywhere. Where do I obtain the text file from? where do I get the package from? How do I get it onto my server?
I am the sort of person to RTFM yet I have no idea where to start. I'm still several skipped stage behind you looking at a black screen with a flashing cursor and a small amount of text telling me that I am now root.
More importantly to me do I actually need Mod_Security? The PHP based Bad Behaviour seems more than able to mitigate all attacks currently. I am of the school of thought that suggests that if it is not broken one does not fix it. Therefore I need to know if there is any benefit in my actions specifically compare to the very real chance that I will cause my web services to stop running (as the last time I tried to install the aforementioned package).
I hope I don't seem too rude - I'm just trying to understand things better.
> .. do I actually need Mod_Security? hmm, that's the wuestion you need to answer yourself :)
> The PHP based Bad Behaviour seems more than able to mitigate all attacks currently. I've not yet seen any PHP solution which free of vulnerabilities ;-) If you really want to use PHP, you better protect your server and applications with something more reliable.
> .. if it is not broken one does not fix it. you meen "never touch a running system"? Well if ther is nothing to protect, and there is nothing lost if somene breaks into your server, controls the server, use it for attacking others, or whatever ..., then ther is no need to protect anything.
> I hope I don't seem too rude no (and be sure you read my comments including the smileys)
