Solved

how to allow crawlers to crawl  a website only on specific days

Posted on 2013-02-06
3
220 Views
Last Modified: 2013-02-15
i want to allow crawlers to crawl my website at specific days of  the week e.g. over the weekend when there is not much traffic on the website. i m using IIS 6.0 for hosting on windows 2003 platform.
0
Comment
Question by:mmalik15
  • 2
3 Comments
 
LVL 42

Expert Comment

by:Chris Stanyon
ID: 38859060
If a crawler is slowing your site down that much, then change hosting providers. A crawler is effectively one visitor, and will have zero impact on your sites performance, but limiting the crawlers access will have significant impact on your Search Engine rankings.

If you explain why you want this, there may be a better solution to your problem.
0
 

Author Comment

by:mmalik15
ID: 38859071
thanks for the comment. Too many crawlers are slowing down the site and CPU is always around 100% in use and the site keeps going down. since I have set the User-agent: * Disallow: / in my robots.txt the cpu use is fine and the site is fine. But i do want the site to be crawled.
0
 
LVL 42

Accepted Solution

by:
Chris Stanyon earned 500 total points
ID: 38859118
I would guess there's another problem with your site then, because there's no way crawlers should grind your site to a halt (otherwise the whole internet would freeze!).

Have a proper look through your access logs and see how many crawlers are hitting your site, and how often. If it's a particularly bad crawler, then you may be able to prevent it from accessing your site using a robots.txt. Are you sure your site doesn't grind to halt when a number of normal users access it.

You can also control some aspects of the GoogleBot by accessing your Google WebMaster account.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Envision that you are chipping away at another e-business site with a team of pundit developers and designers. Everything seems, by all accounts, to be going easily.
Read about why website design really matters in today's demanding market.
This video teaches users how to migrate an existing Wordpress website to a new domain.
The viewer will learn how to count occurrences of each item in an array.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now