Robots.txt and Googlebot

Posted on 2008-11-06
Medium Priority
Last Modified: 2013-12-09
I accidentally messed up robots.txt and googlebot stopped crawling my site. googlebot usually crawls my site all day long.

Will googlebot crawl my site again?

When can I expect googlebot to be back?

Question by:addicktz
  • 2

Assisted Solution

Roshaoar earned 400 total points
ID: 22902616
Yes, when you remove it, googlebot will start crawling it again. It all really depends on the backlinks you have, because the more of those you have, the more signals there are for googlebot to follow a link through to your site to crawl it. Also, the more often you update your pages and/or add new content, therefore creating new pages to discover, the more often googlebot will want to come back and grab those. That's why, for example, forums get crawled a lot, because there's always new content there if it's a busy forum.
LVL 12

Accepted Solution

ShaneJones earned 1200 total points
ID: 22903232
Yes it will get there again. One sure fire way to ensure it comes back soon would be to sign up to there Webmaster tools, verify your site and upload an XML sitemap to Google with all of your pages URLs in.


Google will DEFINITELY come back with this technique.


LVL 31

Assisted Solution

gwkg earned 400 total points
ID: 22920909
Post your link on a site that Google crawls frequently (craigslist, squidoo, stumbleupon, a high traffic forum, etc) and your site will be crawled within days, maybe hours.
LVL 12

Assisted Solution

ShaneJones earned 1200 total points
ID: 22921005
Good suggestion there! StumbleUpon is a sure fire win.

Featured Post

New feature and membership benefit!

New feature! Upgrade and increase expert visibility of your issues with Priority Questions.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

CTAs encourage people to do something specific to show interest in your company, product or service. Keep reading to learn why CTAs should always be thought of as extremely important, albeit small, sections of websites.
The online market is growing at an unprecedented rate and retail eCommerce sales are expected to reach $4 trillion by 2020. Yet, the profit is not just there for the taking because you have to set yourself apart from the competition.
Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Suggested Courses
Course of the Month15 days, 14 hours left to enroll

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question