Problems with WHM / cPanel OWASP ModSecurity..??

I've been trying to battle a bunch of spammers and brute force attacks on my VPS server with HostGator.  I feel like a person who brought home a wifi router, plugged it in, and thought "it works, great!"  Yet they don't change the default IP address, default password, wifi security, etc.  That's I feel with this VPS.  I'm not a server admin, and HostGator isn't helping me figure out how to secure this thing (even with offers to pay for service.)

So, with a little bit of digging, I found this ModSecurity stuff inside the WHM called ModSecurity.  I had to enable it, and then I had to "add a vendor" which then gives me access to a bunch of different security settings I can turn on/off.  When everything enabled, though, my sites on this server weren't functioning correctly, so I turned off everything, and then went through 1 service at a time, turning it on, and testing to see if any of the sites had issues with that particular rule.  

At this point, I have everything except for the following enabled, because I had issues...

rules/REQUEST-10-IP-REPUTATION.conf
I’m pretty sure this one was enabled by default and was fine, but now that I everything disabled and I’m re-enabling one-by-one, I get this error when I try to enable this:

The system could not validate the new Apache configuration because httpd exited with a nonzero value. Apache produced the following error: Syntax error on line 31 of /usr/local/apache/conf/modsec_vendor_configs/OWASP/rules/REQUEST-10-IP-REPUTATION.conf:
ModSecurity: Found another rule with the same id

rules/REQUEST-20-PROTOCOL-ENFORCEMENT.conf
With this enabled, some features on my WordPress sites break.  For example, the site stats feature that the Jetpack plugin provides (if you're familiar with that, by chance) breaks.

I have everything else still turned on right now, but what I'm finding is that people hitting my legitimate web services are getting failures.  I can't figure out which individual rule is doing it, but when I disable the entire ModSecurity firewall it works again.  

So, finally, my question is whether or not there's a way I can fine-tune these rules and allow legitimate traffic..??  I've been reading through this guide, but I just don't see anything other than simply turning on or off a rule.  That seems counter-intuitive, though, and again, I can't seem to find which individual rule is causing my problem with people hitting my legitimate web services.

Any information on all of this would be greatly appreciated.  Thanks!
LVL 11
Andrew AngellCo-Owner / DeveloperAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

gheistCommented:
Check line31 of mentioned file and find both definitions.
Andrew AngellCo-Owner / DeveloperAuthor Commented:
I guess I don't understand what you're telling me to do..??
btanExec ConsultantCommented:
I am suspecting as the modsecurity is to be added to server httpd.conf which is default to /etc/httpd/conf directory (step 6 and step 7 in link below on setup/install), your httpd.conf file may already had an include for *.conf and the if it is as configured as in the link below stated
http://resources.infosecinstitute.com/configuring-modsecurity-firewall-owasp-rules/ 

It can have included another "REQUEST-10-IP-REPUTATION.conf" together with some base modsecurity.conf file. As a result, there can be some rules with duplicated id (reused) because the conf file was included those rule id twice or more. May have to check the path and all the .conf for that duplicate id string and edit it to different one (suggest not change any in the baseline rule file from modsecurity)

For the rule tuning, it is not going to be an one off and straight off effort. You may be interested in my past reply in EE that included a Spiderlabs (trustwave) blog shared on how to further fix false positive using various suggested means. This is also to reduce the failure attempts (include having exceptions)  to the web server as the baseline may be too stringent or not appropriate.
http://www.experts-exchange.com/Software/Server_Software/Web_Servers/Apache/Q_28598249.html#a40555065

Other useful reference such as FAQ and manual for the rule can be handy if you want to drill further...which can be unavoidable for tuning
Main - http://www.modsecurity.org/documentation.html
Manual- https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.