<div>
If a crawler is slowing your site down that much, then change hosting providers. A crawler is effectively one visitor, and will have zero impact on your sites performance, but limiting the crawlers access will have significant impact on your Search Engine rankings. <br />
<br />
If you explain why you want this, there may be a better solution to your problem.
</div>


If a crawler is slowing your site down that much, then change hosting providers. A crawler is effectively one visitor, and will have zero impact on your sites performance, but limiting the crawlers access will have significant impact on your Search Engine rankings.

If you explain why you want this, there may be a better solution to your problem.

<div>
thanks for the comment. Too many crawlers are slowing down the site and CPU is always around 100% in use and the site keeps going down. since I have set the User-agent: * Disallow: / in my robots.txt the cpu use is fine and the site is fine. But i do want the site to be crawled.
</div>


thanks for the comment. Too many crawlers are slowing down the site and CPU is always around 100% in use and the site keeps going down. since I have set the User-agent: * Disallow: / in my robots.txt the cpu use is fine and the site is fine. But i do want the site to be crawled.

<div>
<div class="content wysiwyg-content">
i want to allow crawlers to crawl my website at specific days of &nbsp;the week e.g. over the weekend when there is not much traffic on the website. i m using IIS 6.0 for hosting on windows 2003 platform.
</div>
</div>


i want to allow crawlers to crawl my website at specific days of  the week e.g. over the weekend when there is not much traffic on the website. i m using IIS 6.0 for hosting on windows 2003 platform.

how to allow crawlers to crawl  a website only on specific days

Search Engine Optimization (SEO) is the process of improving the visibility of web content in search engines using an understanding of search engines' processes and algorithms. Also known as "natural" or "organic" search, it is distinct from paid web advertising. Performance is typically measured by testing the appearance of a website for targeted search terms, and more specifically, ranked positions in search for important keywords. Techniques focus on content optimization, technical optimization and link building.

Search Engine Optimization (SEO)

Web development includes all aspects of presenting content on intranets and the Internet, including delivery development, protocols, languages and standards, server software, browser clients, databases and multimedia generation.

Web Development

The successor to Active Server Pages, ASP.NET websites utilize the .NET framework to produce dynamic, data and content-driven web applications and services. ASP.NET code can be written using any .NET supported language. As of 2009, ASP.NET can also apply the Model-View-Controller (MVC) pattern to web applications