What is the best way to counteract spiders, crawlers, and bots on our website?
Posted on 2006-07-13
We're running Windows Small Business Server 2003, and we're having problems with various crawlers sucking up bandwidth (particularly Googlebot, MSNBot, and Yahoo's Inktomisearch). What are the best ways to counteract their usage?
We've started blocking IP ranges, but that seems to help only a little, and I figure it's not a permanent solution anyways.
We've got robots.txt set properly as well as the Meta tags in the header of each page.
I've read about using traps like a 1 x 1 px transparent bitmap image link to another page that has redirects back into itself with like a 20-second delay. Is this still a good solution, or have spiders been made smarter? Any other ways to make bad bots pay for their crimes?
I'm not the main network person here, but I am his b----, so let me know if I can provide any more information.