[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

how to allow crawlers to crawl  a website only on specific days

Posted on 2013-02-06
3
Medium Priority
?
252 Views
Last Modified: 2013-02-15
i want to allow crawlers to crawl my website at specific days of  the week e.g. over the weekend when there is not much traffic on the website. i m using IIS 6.0 for hosting on windows 2003 platform.
0
Comment
Question by:mmalik15
  • 2
3 Comments
 
LVL 45

Expert Comment

by:Chris Stanyon
ID: 38859060
If a crawler is slowing your site down that much, then change hosting providers. A crawler is effectively one visitor, and will have zero impact on your sites performance, but limiting the crawlers access will have significant impact on your Search Engine rankings.

If you explain why you want this, there may be a better solution to your problem.
0
 

Author Comment

by:mmalik15
ID: 38859071
thanks for the comment. Too many crawlers are slowing down the site and CPU is always around 100% in use and the site keeps going down. since I have set the User-agent: * Disallow: / in my robots.txt the cpu use is fine and the site is fine. But i do want the site to be crawled.
0
 
LVL 45

Accepted Solution

by:
Chris Stanyon earned 2000 total points
ID: 38859118
I would guess there's another problem with your site then, because there's no way crawlers should grind your site to a halt (otherwise the whole internet would freeze!).

Have a proper look through your access logs and see how many crawlers are hitting your site, and how often. If it's a particularly bad crawler, then you may be able to prevent it from accessing your site using a robots.txt. Are you sure your site doesn't grind to halt when a number of normal users access it.

You can also control some aspects of the GoogleBot by accessing your Google WebMaster account.
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Originally, this post was published on Monitis Blog, you can check it here . It goes without saying that technology has transformed society and the very nature of how we live, work, and communicate in ways that would’ve been incomprehensible 5 ye…
There is a massive demand for content on the web right now, and it doesn't look like it's going to stop any time soon. But, if you are running a business blog, it's not just enough to offer your audience lots of content. It needs to be high-quality…
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
Any person in technology especially those working for big companies should at least know about the basics of web accessibility. Believe it or not there are even laws in place that require businesses to provide such means for the disabled and aging p…
Suggested Courses
Course of the Month19 days, 4 hours left to enroll

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question