?
Solved

how to allow crawlers to crawl  a website only on specific days

Posted on 2013-02-06
3
Medium Priority
?
242 Views
Last Modified: 2013-02-15
i want to allow crawlers to crawl my website at specific days of  the week e.g. over the weekend when there is not much traffic on the website. i m using IIS 6.0 for hosting on windows 2003 platform.
0
Comment
Question by:mmalik15
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 43

Expert Comment

by:Chris Stanyon
ID: 38859060
If a crawler is slowing your site down that much, then change hosting providers. A crawler is effectively one visitor, and will have zero impact on your sites performance, but limiting the crawlers access will have significant impact on your Search Engine rankings.

If you explain why you want this, there may be a better solution to your problem.
0
 

Author Comment

by:mmalik15
ID: 38859071
thanks for the comment. Too many crawlers are slowing down the site and CPU is always around 100% in use and the site keeps going down. since I have set the User-agent: * Disallow: / in my robots.txt the cpu use is fine and the site is fine. But i do want the site to be crawled.
0
 
LVL 43

Accepted Solution

by:
Chris Stanyon earned 2000 total points
ID: 38859118
I would guess there's another problem with your site then, because there's no way crawlers should grind your site to a halt (otherwise the whole internet would freeze!).

Have a proper look through your access logs and see how many crawlers are hitting your site, and how often. If it's a particularly bad crawler, then you may be able to prevent it from accessing your site using a robots.txt. Are you sure your site doesn't grind to halt when a number of normal users access it.

You can also control some aspects of the GoogleBot by accessing your Google WebMaster account.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Google always has something new and amazing up its sleeve, and the most current thing that they have been working on is another step in the evolution of Google Search, from machine learning to its brilliant successor, deep learning.
This article was originally published on Monitis Blog, you can check it here . Today it’s fairly well known that high-performing websites and applications bring in more visitors, higher SEO, and ultimately more sales. By the same token, downtime…
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
Suggested Courses

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question