?
Solved

how to allow crawlers to crawl  a website only on specific days

Posted on 2013-02-06
3
Medium Priority
?
255 Views
Last Modified: 2013-02-15
i want to allow crawlers to crawl my website at specific days of  the week e.g. over the weekend when there is not much traffic on the website. i m using IIS 6.0 for hosting on windows 2003 platform.
0
Comment
Question by:mmalik15
  • 2
3 Comments
 
LVL 45

Expert Comment

by:Chris Stanyon
ID: 38859060
If a crawler is slowing your site down that much, then change hosting providers. A crawler is effectively one visitor, and will have zero impact on your sites performance, but limiting the crawlers access will have significant impact on your Search Engine rankings.

If you explain why you want this, there may be a better solution to your problem.
0
 

Author Comment

by:mmalik15
ID: 38859071
thanks for the comment. Too many crawlers are slowing down the site and CPU is always around 100% in use and the site keeps going down. since I have set the User-agent: * Disallow: / in my robots.txt the cpu use is fine and the site is fine. But i do want the site to be crawled.
0
 
LVL 45

Accepted Solution

by:
Chris Stanyon earned 2000 total points
ID: 38859118
I would guess there's another problem with your site then, because there's no way crawlers should grind your site to a halt (otherwise the whole internet would freeze!).

Have a proper look through your access logs and see how many crawlers are hitting your site, and how often. If it's a particularly bad crawler, then you may be able to prevent it from accessing your site using a robots.txt. Are you sure your site doesn't grind to halt when a number of normal users access it.

You can also control some aspects of the GoogleBot by accessing your Google WebMaster account.
0

Featured Post

The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Magento is the best technology for eCommerce start-ups as it offers the technical expertise and visual appeal to create a store that pulls sales and earns high ROI (Return on investment).
This article is about the challenges faced by Android app developers.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
Any person in technology especially those working for big companies should at least know about the basics of web accessibility. Believe it or not there are even laws in place that require businesses to provide such means for the disabled and aging p…
Suggested Courses
Course of the Month3 days, 21 hours left to enroll

600 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question