We help IT Professionals succeed at work.

non existant pages still getting spidered

naifyboy123 asked
Medium Priority
Last Modified: 2010-04-27

i while back i created a dynamic page template which generated lots of pages fomr entries in a DB.

i added the page addresses to my google site map and uploaded it.

i then decided to change the urls of these pages and again uploaded my site map with the new urls in it - making sure i removed the old ones.

however, the old urls are still getting targeted by the googlebots and other spiders.

i have a email error report set up to tell me when errors in my site occur or a page can not be found and whenever a spider tries to access the no longer existent pages i get an error report.

there are no links to these old urls anywhere in my site and my google site map does not contain any reference to them.

the main ip address (but there are others) is and i understand that is a googlebot.

how can they still be getting spidered?

Watch Question

Website Designer & SEO Consultant
Do not worry.

If nothing is linking to the pages then they would be removed automatically.

Or you may mention that in your robots.txt


Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts


hi ashishjvw

any idea how long i have to wait? it has been weeks already
Ashish ThakkarWebsite Designer & SEO Consultant

Hey !

If you wish to remove then please use robots.txt .

I have some files on google which have been removed for months and are still there.



ok thanks -
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.


Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.