Apache Error Log Filled With DOS?

Platform
===========================
Windows Server x64
Apache 2.2.6 / PHP 5.2.5
Microsoft SQL Server Express

Issue
=========================
My webserver logs are full of requests for phpmyadmin,mysql and other types of document requests.

The problem is that I am receiving "thousands of requests in a single day, from multiple IP addresses in different ranges". Its happening all day long and appears to be coming from several automated processes. I really dont know whats going on here.

I was able to trace back some of the IP's to gaming sites, file servers etc, xamp servers etc.

E.g. snippet ..............

[Sat Jan 12 14:23:33 2008] [error] [client 82.99.173.167] File does not exist: D:/htdocs/mysite/xmlsrv
[Sat Jan 12 14:23:33 2008] [error] [client 82.99.173.167] File does not exist: D:/htdocs/mysite/blog
[Sat Jan 12 14:23:33 2008] [error] [client 82.99.173.167] File does not exist: D:/htdocs/mysite/drupal
[Sat Jan 12 14:23:34 2008] [error] [client 82.99.173.167] File does not exist: D:/htdocs/mysite/community
[Sat Jan 12 14:23:34 2008] [error] [client 82.99.173.167] File does not exist: D:/htdocs/mysite/blogs
[Sat Jan 12 14:23:34 2008] [error] [client 82.99.173.167] File does not exist: D:/htdocs/mysite/blogs
[Sat Jan 12 14:23:34 2008] [error] [client 82.99.173.167] File does not exist: D:/htdocs/mysite/blog
[Sat Jan 12 14:23:34 2008] [error] [client 82.99.173.167] File does not exist: D:/htdocs/mysite/blogtest
[Sat Jan 12 14:23:34 2008] [error] [client 82.99.173.167] File does not exist: D:/htdocs/mysite/b2
[Sat Jan 12 14:23:34 2008] [error] [client 82.99.173.167] File does not exist: D:/htdocs/mysite/b2evo
[Sat Jan 12 14:23:34 2008] [error] [client 82.99.173.167] File does not exist: D:/htdocs/mysite/wordpress
[Sat Jan 12 14:23:34 2008] [error] [client 82.99.173.167] File does not exist: D:/htdocs/mysite/phpgroupware
[Sat Jan 12 16:25:45 2008] [error] [client 87.238.198.51] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Sat Jan 12 16:57:55 2008] [error] [client 87.238.198.51] File does not exist: D:/htdocs/mysite/phpmyadmin
[Tue Jan 22 01:10:04 2008] [error] [client 87.118.120.162] File does not exist: D:/htdocs/mysite/phpMyAdmin
[Tue Jan 22 01:10:04 2008] [error] [client 87.118.120.162] File does not exist: D:/htdocs/mysite/db
[Tue Jan 22 01:10:08 2008] [error] [client 87.118.120.162] File does not exist: D:/htdocs/mysite/web
[Tue Jan 22 01:10:08 2008] [error] [client 87.118.120.162] File does not exist: D:/htdocs/mysite/PMA
[Tue Jan 22 01:10:08 2008] [error] [client 87.118.120.162] File does not exist: D:/htdocs/mysite/admin
[Tue Jan 22 01:10:10 2008] [error] [client 87.118.120.162] File does not exist: D:/htdocs/mysite/dbadmin
[Tue Jan 22 01:10:13 2008] [error] [client 87.118.120.162] File does not exist: D:/htdocs/mysite/PMA2006
[Tue Jan 22 01:10:13 2008] [error] [client 87.118.120.162] File does not exist: D:/htdocs/mysite/pma2006

This is continual and the requested files typically enumerate through every version of phpadmin and mysql known to mortal man. The IP addresses are so numerous that I cant block with the cisco firewall/apache.

Thanks

netvbonline
LVL 3
vbellis-rdyAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

mrcoffee365Commented:
Here's a set of answers for one person who got the same kind of error messages in their log:
http://www.experts-exchange.com/Software/Server_Software/Web_Servers/Q_22879898.html

The upshot of it was that an internal IT scanner was being run.  In your case, it sounds as if an external scanner is being run, so quite possibly it's an external attempt to shut down your server, or to find vulnerabilities.

However, the most common answer is not usually malicious intent, but someone at your company running security test software on your system.
0
jahboiteCommented:
At least one of these attempts to find vulnerabilities on your webserver is using the DFind vulnerability scanner - which I'd never heard of before.  There isn't a great deal written about it although Symantec.com have listed some of the vulnerabilities it looks for:
http://www.symantec.com/security_response/writeup.jsp?docid=2005-011411-1411-99&tabid=3

Given that this scanner is leaving behind such a signature, my guess is that this attempt at least is not a serious effort to break-in, but is more a s|<rIpt kIddi3 style of spray wildly and hope something sticks .  It's not very subtle is it.

Publicly accessible machines will be scanned for holes.  This is something everyone has to accept and try to stay ahead of.
You don't want to get into blocking IP addresses, just make sure you hunt for and plug holes in your own systems on a regular basis.
If you regularly peruse your logs, you'll get a feel for the baseline for these kind of probes which might put your mind at ease and you'll be able to detect spikes in such activity.  If you suddenly hit a spike, it may be that a new vulnerability has been disclosed so you should keep your eyes on vulnerability lists and make sure you always test and apply patches as soon as is practicable.
0
vbellis-rdyAuthor Commented:
Thanks for your comments will read the provided links, my log files are growing massive (which may have a performance hit?) and its messing up my stats. My hosting company is 1&1 if that helps and I suspect I am being scanned externally. The confusing part is that these scans are happening from multiple IP addresses, virtually every minute of the day and each one is looking for security holes.

This has only started recently, several months back I had terminal services running for remote desktop access and I am sure that someone managed to get into the server and change the admin password. In the end I had to re-image the server. Now it's been redeployed, terminal services is turned off in firewall when not in use and now I keep getting all of these suspect hits.

Thanks
0
Redefining Cyber Security w/ AI & Machine Learning

The implications of AI and machine learning in cyber security are massive and constantly growing, creating both efficiencies and new challenges across the board. Join our webinar on Sept. 21st to learn more about leveraging AI and machine learning to protect your business.

vbellis-rdyAuthor Commented:
I'm at approx 200 unique IP's scanning and looking for holes. Yet several months back ran my site for over 2 years and cant remember ever being scanned in this way - the world is against me! LOL.

82.99.173.167
87.118.120.162

Both resolve to a domain, in fact my server is quite popular in Japan, Germany, India, Europe, America to name a few. All of the IP's are public addresses and many of them resolve themselves to phpadmin a couple even show a phpinfo() as the home page which is really, really clever.

You attack me, then tell me everything I need to know about your server to launch a return attack (not that I have the time or desire mind you). I dont mind the occassional snooping but when its 50+ requests per visit several times in 1 hour it kind of ticks me off. You would think ISP's could perhaps help with detecting this type of behaviour? How many human users can request 40 pages in 1 second?

[Mon Jan 21 19:21:41 2008] [error] [client 151.4.119.202] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)

[Mon Jan 21 20:21:41 2008] [error] [client 63.223.85.26] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)

[Mon Jan 21 20:28:31 2008] [error] [client 85.114.141.224] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)

:50:07 2008] [error] [client 84.247.200.162] File does not exist: D:/htdocs/mysite/phpMyAdmin-2.6.0
[Wed Jan 16 11:50:07 2008] [error] [client 84.247.200.162] File does not exist: D:/htdocs/mysite/phpMyAdmin-2.6.0-pl1
[Wed Jan 16 11:50:08 2008] [error] [client 84.247.200.162] File does not exist: D:/htdocs/mysite/phpMyAdmin-2.6.3-pl1
[Wed Jan 16 11:50:08 2008] [error] [client 84.247.200.162] File does not exist: D:/htdocs/mysite/phpMyAdmin-2.6.3
[Wed Jan 16 11:50:08 2008] [error] [client 84.247.200.162] File does not exist: D:/htdocs/mysite/phpMyAdmin-2.6.3-rc1
[Wed Jan 16 11:50:08 2008] [error] [client 84.247.200.162] File does not exist: D:/htdocs/mysite/phpMyAdmin-2.6.2-rc1

The IP addresses that I am seeing in my log could be forged through the use of a proxy server right?

Basically:

1. What is the maximum size I should permit for the apache log files? (re: performance)
2. Can I do anything to conserve bandwidth with all these damn requests?
3. How does one configure apache for optimum performace when demand increases due to scanning etc?

Thankxs
0
mrcoffee365Commented:
Sorry to hear about the continued attacks.  It does seem as if you have someone scanning for files maliciously, since they're going to the trouble of spoofing IP addresses.  Or, as you say, maybe these are the actual IP addresses.

>>The IP addresses that I am seeing in my log could be forged through the use of a proxy server right?

The IP addresses can be spoofed through a lot of methods.  A proxy server is only one of them.

I think you don't have a lot of choices in response.  You can assume that the IP addresses are real, and blacklist them.  There are scripts for doing that dynamically, based on the requests that come into the server.  See some examples in this set of Apache tools from O'Reilly:
http://www.apachesecurity.net/download/snapshot/apache_tools-snapshot.tar.gz

You might also be interested in this general discussion of responses to DOS attacks:
http://www.xav.com/scripts/guardian/help/1013.html

>>1. What is the maximum size I should permit for the apache log files? (re: performance)

I think this is up to you and your space availability.  Also, whatever you do with the logs.  If you're using Log4J, then you can set maxFileSize to force a rollover, and perhaps extend the rollover method to remove all of the DOS attack lines from your archived log.

I have not heard of a known maximum to set file size for logs -- I think it depends too much on the configuration.  Writing to logs is fast and is just an append, so I don't think you have to worry in terms of slowing down the application as the log files grow.  The file system might start having a problem with very large files, and of course larger files take longer to process for statistics or debugging.

>>2. Can I do anything to conserve bandwidth with all these damn requests?
You can try not to respond to DOS attacks with a large error page.  The security people recommend responding with a 403 to all DOS attack requests ( use something like D:/htdocs/mysite/phpMyAdmin-2.6.3 as your pattern).  However, don't give a custom 403, or anything which requires a script to run on your server.

Of you can blacklist the IP addresses, to prevent the request ever getting very far into your server.  Or you can put a proxy in front of your server, and send it the blacklisted IP addresses.  The problem there is that your server never gets the real IP addresses, so if you do Web log analysis using the client IP address, that will have to be done on the proxy logs.

Or you can get expensive routers which will block IP addresses.  I don't know what the capability is for dynamically adding blacklist IP addresses to a router, but it might be possible.

You can have firewalls and routers which will shut down requests when more than some number are made from an IP address within a short period of time, then release the throttle after a delay.  Google does that to prevent programmatic access to their search engine, for example.

>>3. How does one configure apache for optimum performace when demand increases due to scanning etc?

Scanning is like regular access (unless you block IP addresses before they get to Apache), so you can check the normal performance tuning parameters for Apache:
http://httpd.apache.org/docs/2.0/misc/perf-tuning.html
http://www.devside.net/articles/apache-performance-tuning


0
jahboiteCommented:
mrcoffee365 has covered your questions well, but I'd like to share thoughts about what you might be witnessing.

 A possibility is that you are running something that has known vulnerabilities and that your webserver shows up in google as running that something. I'm not saying that you have a vulnerable service, just that the service is one that has known vulnerabilities.  Take a look here for the kind of google queries that make it easy to find such systems: http://johnny.ihackstuff.com/forums
For instance, PHPMyAdmin: http://www.google.com/search?q=intitle%3A%22phpMyAdmin+*.*.*%22+%22Please+select+a+database%22

Some of the sources you see may themselves have been compromised and are being used to find more such systems in an automated way. (Perhaps using google to find possible targets)
This is borne out by the fact that you found MyPHPAdmin and PHPInfo pages at some of the sources (badly configured machines) and I've done a cursory scan against the sources you posted, finding much the same thing, such as two of the machines running PLESK admin page (and worse).

I'd suggest you perform vulnerability assessments of your webserver (regularly) as well as reviewing your security policy as a whole.  In addition to this, carefully scrutinise all traffic (not just HTTP) to and from the machine to a) determine whether the HTTP probing is part of a wider activity and b) determine the possibility that the machine has been compromised yet again.
0
vbellis-rdyAuthor Commented:
mrcoffee365:

Excellent work and resource links for my key questions, thanks. As the server is being hosted by a third party the only real control I have is over the virtual software firewall in control panel and then the firewall on the web server OS. Detecting and black listing IP addresses in the hardware is my 1st choice but then I run the risk of potential legitimate IP's being wrongly blacklisted and besides this wont be an option unless I host myself.

jahboite:

Thanks also for your comment and looking into this for me. I have reviewed the security policy to the best of my ability and current knowledge. I configured the apache.conf/PHP to enable only those libraries that are actually being used, which are just about the minimum you would require to host a data driven website.

I'm not running PhpAdmin on server just standard PHP, all web form variables are correctly accessed via $_POST which is promptly placed into a sanitized string array. I believe apache is responding correctly to all requests issued my error logs are being populated from external scans. I am running virtual hosts on the server but cant imagine that would be of interest to a hacker (the sites are'nt running anything that sensitive)?

 I have blocked all services in the firewall other than HTTP/S, SMTP and lock down remote desktop to a specific IP address. No rogue processes in the memory or any third party components running on the server.

I dont suppose you have any links to a site that would take somebody a step by step guide to securely configure a web server (Esp running Apache/PHP/SQL Server) ?

Please can an expert summarise the above points into a plan of action, as it all seems abuit jumbled up at moment (or maybe thats me LOL).

Thanks

0
jahboiteCommented:
Are you still seeing the same volume of scanning?

What version of Apace are you running?&nbsp; I ask because it's possible that the scanning is happening because your server is reporting a version number that is known to be vulnerable to something.

If this was the case, it might be worth turning off the reporting of the version number which would make it harder for an attacker to correctly determine the version number.

I'd recommend the following directives for httpd.conf:
ServerSignature  Off
ServerTokens      Prod

More details here:
http://httpd.apache.org/docs/2.2/server-wide.html#identification
http://httpd.apache.org/docs/2.2/mod/core.html#serversignature
0
mrcoffee365Commented:
Those are good things to do.  However, if someone is trying to get to your server, you'll still get the requests until they're tired of doing it.  The best thing you can do is not give any response, and it sounds as if you are sure that you are not giving a response.

As we mentioned earlier, you can deny IP addresses, if you can and if you are sure no legitimate users are on those IP addresses.  Other than that, if it's a public service, you can't prevent people from making requests of it.

If you want to see the level of random port scanning that you're already preventing, try watching your firewall for a few minutes.  There are many computers out there running programs to randomly work their way through the ports of many IP addresses, with various requests trying to get into the public computer.  

It's life on the Internet.

There isn't a guide for preventing the hack attempts you're seeing, because you can't.  I'm sure you've already googled for how to secure an Apache Web server, and it sounds as if you're doing the basic right things.   Some links here:
http://httpd.apache.org/docs/2.2/misc/security_tips.html
http://www.apachesecurity.net/   (O'Reilly book)
http://www.securityfocus.com/infocus/1786
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
vbellis-rdyAuthor Commented:
Thanks for all your suggestions ... As you have both provided good advice believe it's only fair to share the points. As you say this is life on the NET, just need to try and be that one step ahead.

Cheers
vbellis-rdy
0
mrcoffee365Commented:
You're welcome, and good luck!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Development

From novice to tech pro — start learning today.