Removing dead links from Google Site Search

Posted on 2011-04-22
Last Modified: 2013-12-08
We made some significant changes to a site—large sections were moved from “” to “” There are no links to the /old directory, it’s our makeshift backup in case we want to quickly revert (i.e., those aren’t a problem).

Google Site Searches are still returning hits for pages in the /nolongerneeded folder. As expected, following these links leads to a 404 error. Much of the information on the moved pages has been restructured to “,” so people will still be using similar search terms and looking for relevant results.

I used SOFTplus GSiteCrawler v1.23 to recrawl the site and regenerate a site map. It lists the new pages. I put the /nolongerneeded folder into its Drop Parts and Ban filters, but the resulting sitemap still has:


Open in new window

The last modification was the bulk move on 4/19—the November 2010 date was the last change to the page contents.

Can I just do a global search on the sitemap and delete all references to /nolongerneeded and resubmit to Google (we have a paid account, if it makes a difference)? Am I missing something straightforward?

I’m awfully sorry for such a basic question, but I’m a bit stuck—and since it seems to take a couple days for updates to propagate, I don’t have the time to go the trial-and-error route.



Question by:Rhythmdvl
    LVL 82

    Accepted Solution

    Bring up your site in GSiteCrawler v1.23 and go to the "URL List" tab and click on "Delete all non-manual links" to get rid of previous links other than the main site URL.  Then tell it to recrawl your site.  You should then have a list with only the current links.

    Author Comment

    So close ...

    Thanks for the re-crawling help. I now have a Sitemap with correct information. I submitted it via Google Webmaster Tools, then went to the Site Search Control Panel and selected Index Now under the On-demand indexing using Sitemaps section. It came back with an "index refreshed" response.

    But when going to the site and searching under the keyword, the old 404 links are still showing up. Should I be patient and they will stop appearing in a few days or am I still missing something?


    Author Comment

    How can I exclude sites and pages?

    There are three ways you can exclude sites or pages from your Custom Search Engine: individually, in bulk, or using the Google Marker.

    Individually You can exclude sites individually in the Sites tab of your Custom Search Engine's control panel. Select the Add Sites button under the Excluded sites section (or the Exclude sites link if you haven't listed any) and the Exclude sites individually option will open.

    In bulk You can exclude sites in bulk by selecting the Exclude sites in bulk option under the Excluded sites section of your Sites tab. To use this option in the Sites tab, select the Add Sites button (or the Exclude sites link if you haven't listed any) and then the Exclude sites in bulk option. Enter the sites, pages, or patterns, one per line.

    Using the Google Marker Once you have created a Custom Search Engine, you can exclude sites using the Google Marker. The Google Marker allows you to save sites to your Custom Search Engine as you browse the web. For more details, visit

    Author Comment

    Oops, I hit submit too soon. (And can't see an edit function for the post).

    That's from the Google Help Page:  Custom Search › Help articles › Creating and Editing Your CSE › Adding or Excluding Pages › How can I exclude sites and pages?  

    I think I was banging my head against a wall because I missed in in reading and didn't use the keyword exclude in searching. Very simple process.

    The sitemap was also vexing because in the On-Demand Indexing help page it says:
    If I submit a new Sitemap, will pages from the previous Sitemap be dropped from my search engine?If your CSE or GSS has sufficient on-demand quota to accommodate the new pages, no pages will be dropped from your results. If you exceed your limit, we may remove some of your pages from on-demand indexing. We will always try to remove the least important pages, as determined by priority and last modified date in the Sitemap.

    To review your use of the on-demand quota, visit the Indexing tab of your search engine control panel. In the On-demand indexing section of this page, you will see the Sitemap URL, indexing status, and available page quota. As long as the total number of new or newly updated pages you include in your Sitemap at any one time is less than your remaining quota listed here, no pages will be removed from your search engine.

    Again, thanks for the Sitemap help--it's crucial to the overall site as well.

    Author Closing Comment

    This was a great help. The ultimate solution differs a bit (see following posts) but I asked the wrong questions--this was the right solution for that related task. Thanks!
    LVL 8

    Expert Comment

    by:Matthew Nguyen
    You really should do a 301 permanent redirect from your old url to your new url.  This way, people going to "” will automatically get redirect to  “"  In the same sense, all content from "” will no longer render a 404 error and Google index will natural change telling it where the new content is now on “"

    If you are already beyond the point of being able to do 301 redirects, Google does not like having 404 errors in the index and will eventually remove them from the index.  If you want to speed up the process, you can request for Google to remove URLs with their URL Removal Tool.  You'll have to be logged into your Webmaster Tools account in order to use it.

    Hope that helps

    LVL 82

    Expert Comment

    by:Dave Baldwin
    Thanks for the points.  I wasn't ignoring you, I was just busy doing other things.  Note that any external links for the old pages will still be found in Google, that part is not under your control.  And Google usually takes a while to update the search results.

    Featured Post

    IT, Stop Being Called Into Every Meeting

    Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

    Join & Write a Comment

    I recently found myself in a Corporate Situation where the client had requested blocking access to any and all websites except his own Domain? Easy? I am sure this would be your answer but their requirement was, this has to be done without using…
    Introduction If you're like most people, you have occasionally made a typographical error when you're entering information into an online form.  And to your consternation, the browser remembers the error, and offers to autocomplete your future entr…
    This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…
    The viewer will learn how to dynamically set the form action using jQuery.

    732 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    22 Experts available now in Live!

    Get 1:1 Help Now