non existant pages still getting spidered

Posted on 2006-04-05
Last Modified: 2010-04-27

i while back i created a dynamic page template which generated lots of pages fomr entries in a DB.

i added the page addresses to my google site map and uploaded it.

i then decided to change the urls of these pages and again uploaded my site map with the new urls in it - making sure i removed the old ones.

however, the old urls are still getting targeted by the googlebots and other spiders.

i have a email error report set up to tell me when errors in my site occur or a page can not be found and whenever a spider tries to access the no longer existent pages i get an error report.

there are no links to these old urls anywhere in my site and my google site map does not contain any reference to them.

the main ip address (but there are others) is and i understand that is a googlebot.

how can they still be getting spidered?

Question by:naifyboy123
    LVL 7

    Accepted Solution

    Do not worry.

    If nothing is linking to the pages then they would be removed automatically.

    Or you may mention that in your robots.txt


    Author Comment

    hi ashishjvw

    any idea how long i have to wait? it has been weeks already
    LVL 7

    Expert Comment

    Hey !

    If you wish to remove then please use robots.txt .

    I have some files on google which have been removed for months and are still there.


    Author Comment

    ok thanks -

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Training Course: Adobe Photoshop CC 2015

    Photoshop has become the standard in image manipulation among professional photographers and graphic designers. This course covers exam objectives for the Adobe Certified Associate (ACA) certification.

    Many times, I come across people who own various domains in addition to their primary domain and are simply redirecting them to their primary domain.  These people often consider building a microsite ( on those…
    Panda2.5's Release Date The latest Google Panda algorithm update was released late on Tuesday, September 27th, and was noticed to have started affecting website traffic on Wednesday, September 28th.  Google did not proactively announce this update,…
    Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
    Use Wufoo, an online form creation tool, to make powerful forms. Learn how to selectively show certain fields based on user input using rules to gather relevant information and data from your forms. The rules feature provides you with an opportunity…

    737 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    20 Experts available now in Live!

    Get 1:1 Help Now