Solved

how to allow crawlers to crawl  a website only on specific days

Posted on 2013-02-06
3
223 Views
Last Modified: 2013-02-15
i want to allow crawlers to crawl my website at specific days of  the week e.g. over the weekend when there is not much traffic on the website. i m using IIS 6.0 for hosting on windows 2003 platform.
0
Comment
Question by:mmalik15
  • 2
3 Comments
 
LVL 43

Expert Comment

by:Chris Stanyon
ID: 38859060
If a crawler is slowing your site down that much, then change hosting providers. A crawler is effectively one visitor, and will have zero impact on your sites performance, but limiting the crawlers access will have significant impact on your Search Engine rankings.

If you explain why you want this, there may be a better solution to your problem.
0
 

Author Comment

by:mmalik15
ID: 38859071
thanks for the comment. Too many crawlers are slowing down the site and CPU is always around 100% in use and the site keeps going down. since I have set the User-agent: * Disallow: / in my robots.txt the cpu use is fine and the site is fine. But i do want the site to be crawled.
0
 
LVL 43

Accepted Solution

by:
Chris Stanyon earned 500 total points
ID: 38859118
I would guess there's another problem with your site then, because there's no way crawlers should grind your site to a halt (otherwise the whole internet would freeze!).

Have a proper look through your access logs and see how many crawlers are hitting your site, and how often. If it's a particularly bad crawler, then you may be able to prevent it from accessing your site using a robots.txt. Are you sure your site doesn't grind to halt when a number of normal users access it.

You can also control some aspects of the GoogleBot by accessing your Google WebMaster account.
0

Featured Post

3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

I've been asked to discuss some of the UX activities that I'm using with my team. Here I will share some details about how we approach UX projects.
If you don’t want your company's site to fail on the web, you’d do well to observe these best web design practices and make sure you implement them when applicable.
The viewer will learn how to count occurrences of each item in an array.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

911 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now