Robots.txt and Googlebot

I accidentally messed up robots.txt and googlebot stopped crawling my site. googlebot usually crawls my site all day long.

Will googlebot crawl my site again?

When can I expect googlebot to be back?

LVL 1
addicktzAsked:
Who is Participating?
 
ShaneJonesConnect With a Mentor Commented:
Yes it will get there again. One sure fire way to ensure it comes back soon would be to sign up to there Webmaster tools, verify your site and upload an XML sitemap to Google with all of your pages URLs in.

http://www.google.com/webmasters/tools/

Google will DEFINITELY come back with this technique.

Shane

0
 
RoshaoarConnect With a Mentor Commented:
Yes, when you remove it, googlebot will start crawling it again. It all really depends on the backlinks you have, because the more of those you have, the more signals there are for googlebot to follow a link through to your site to crawl it. Also, the more often you update your pages and/or add new content, therefore creating new pages to discover, the more often googlebot will want to come back and grab those. That's why, for example, forums get crawled a lot, because there's always new content there if it's a busy forum.
0
 
gwkgConnect With a Mentor Commented:
Post your link on a site that Google crawls frequently (craigslist, squidoo, stumbleupon, a high traffic forum, etc) and your site will be crawled within days, maybe hours.
0
 
ShaneJonesConnect With a Mentor Commented:
Good suggestion there! StumbleUpon is a sure fire win.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.