Robots.txt and Googlebot

Posted on 2008-11-06
Last Modified: 2013-12-09
I accidentally messed up robots.txt and googlebot stopped crawling my site. googlebot usually crawls my site all day long.

Will googlebot crawl my site again?

When can I expect googlebot to be back?

Question by:addicktz
    LVL 3

    Assisted Solution

    Yes, when you remove it, googlebot will start crawling it again. It all really depends on the backlinks you have, because the more of those you have, the more signals there are for googlebot to follow a link through to your site to crawl it. Also, the more often you update your pages and/or add new content, therefore creating new pages to discover, the more often googlebot will want to come back and grab those. That's why, for example, forums get crawled a lot, because there's always new content there if it's a busy forum.
    LVL 12

    Accepted Solution

    Yes it will get there again. One sure fire way to ensure it comes back soon would be to sign up to there Webmaster tools, verify your site and upload an XML sitemap to Google with all of your pages URLs in.

    Google will DEFINITELY come back with this technique.


    LVL 31

    Assisted Solution

    Post your link on a site that Google crawls frequently (craigslist, squidoo, stumbleupon, a high traffic forum, etc) and your site will be crawled within days, maybe hours.
    LVL 12

    Assisted Solution

    Good suggestion there! StumbleUpon is a sure fire win.

    Featured Post

    Why You Should Analyze Threat Actor TTPs

    After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

    Join & Write a Comment

    A/B testing is a simple and effective trick to get to know your audience, increase website conversions and make the most out of your online ad campaigns. It's widely available and doesn't need much tech knowledge to be executed, but the results it y…
    Read about why website design really matters in today's demanding market.
    Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
    This video teaches users how to migrate an existing Wordpress website to a new domain.

    728 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    20 Experts available now in Live!

    Get 1:1 Help Now