We do web hosting and some of the websites hosted at our servers have something that Google Bots keep working with them so often and for very long time that the server becomes slow. Sometimes it is over a million hits to one website by Google bot in a day.
We are trying to find out a solution that even if our customer has not configured its website correctly with Google bot/Webmaster tools Google cannot do that much hits to our server.
Currently in such cases we block the Google Bot IPs in IPTables and the servers become very good but in that case the customers having good websites also suffer.
Can someone please suggest a solution to this?
We are running CentOS 6.5 64bit and using NginX and Apache at our servers.
If you are a web developer, you would be aware of the <iframe> tag in HTML. The <iframe> stands for inline frame and is used to embed another document within the current HTML document. The embedded document could be even another website.
There is a massive demand for content on the web right now, and it doesn't look like it's going to stop any time soon. But, if you are running a business blog, it's not just enough to offer your audience lots of content. It needs to be high-quality…
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.
Login to your Google Account, then Google for "Google M…
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details.
Login to your Google Account, then search for "Google Mapmaker…