Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17


SEO BLock Url to get indexed in Google.

Posted on 2011-03-03
Medium Priority
Last Modified: 2012-05-11
I want to block url from getting indexed in Google. Please help me in doing that. I know how to use robots.txt for restricting folders but how can I block URL indexing.

And How to check if SSL certificate is having right info or not.

Please help.
Question by:ravigosai
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
  • 2
  • +1
LVL 14

Expert Comment

by:Dhanasekaran Sengodan
ID: 35025644
place this line  in robots.txt file

Disallow: /file-name.html

LVL 14

Expert Comment

by:Dhanasekaran Sengodan
ID: 35025923
place this line  in robots.txt file

User-agent: *
Disallow: /

Author Comment

ID: 35029852
How can I block URL indexing by using Disallow:/

I didnt get it. The URL is not main website. still the URL is being redirected to the main landing page of the site. I dont know how to make sure that google does not index this particular URL.
Survive A High-Traffic Event with Percona

Your application or website rely on your database to deliver information about products and services to your customers. You can’t afford to have your database lose performance, lose availability or become unresponsive – even for just a few minutes.


Author Comment

ID: 35030323
The URL I am trying to block lands on the same page as our main URL lands. is there a way to block indexing on it without redirecting it to 404?
LVL 23

Expert Comment

by:Tony McCreath
ID: 35043481
Could you clarify what you want.

Do you really want to stop Google from indexing your home page?

It sounds like a very bad move. Could you explain why you want to do that.
LVL 29

Accepted Solution

fibo earned 2000 total points
ID: 35043542
B-) I must confess I am slightly confused at what is your problem and the solution you are seeking for.

Here is what I understand:
You have one file, say index.html
The content of this file can be displayed by 2 URLs, say (with or without the trailing slash) and /index.html
For SEO purpose, you would want search engines to keep the url / but not /index.html

Since it is the same file in 2 URLs disguises, you cannot use the otherwise ad-hoc tools, such has a <robot noindex> metatag in the html code.

There are 2 different problems
1 - Why are there references to index.html?
2 - How can I remove the url from the known urls?

1. Remove from YOUR site links to the bad URL

Point 1 is usually, and in this case most probably, created by you: your site itself references /index.html instead of /
So this is really your first item on the check list: change your source code so that "home" is / and not /index.html
ANY OTHER ACTION IS USELESS IF THIS IS NOT DONE, since you are nurturing the problem!
Since it is not that easy to check that there remains no link to the bad url, use one tool like Xenu which will make a list of the URLs linked to from within your site, as well as (this is the useful part here!) which pages are linking to this URL

At the same time, you might also consider using the canonical metatag to reinforce the above work. See page
Note that this is not an alternative to tidying your own site from references to the "bad" url

2. Remove the URL from the search indexes

Once this is solved, you now need to remove the url /index.html from the indexes of Google + Bing + Yahoo.
Just connect to their webmasters tools, validate you site (my preferred solution is one extra file, but if fails in some situations). While you are at it, communicate them any sitemap you might have done --after checking that the "bad" url is absent from the map!--
Wait 2 days, then come back to the tool, check if they have recorded the "bad" url and if yes, use the tools to remove it.

Finally, you might use some url_rewrite... but I would not do that unless after 4 weeks the problem is still there.

Author Comment

ID: 35072454
Basically the site was using the old url before. and then we rebuilded the site and also we changed the URL. so now we were required to redirect the old URL also to new site so that was being done. so now google is indexing the old ULR as duplicate URL. and that is creating issue.
LVL 23

Expert Comment

by:Tony McCreath
ID: 35079662
Still not clear.

Could you provide some example urls with that description?

If you are redirecting the old url to the new url it should be a 301 redirect. This tells search engines to transfer their index from old to new. It your using any other form of redirect the transfer may not happen.

Featured Post

Introducing the WatchGuard 420 Access Point

WatchGuard's newest access point includes an 802.11ac Wave 2 chipset, providing the fastest speeds for VoIP, video and music streaming, and large data file transfers. Additionally, enjoy the benefits of strong security as the 3rd radio delivers dedicated WIPS protection!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When it comes to write a Context Sensitive Help (an online help that is obtained from a specific point in state of software to provide help with that state) ,  first we need to make the file that contains all topics, which are given exclusive IDs. …
Originally, this post was published on Monitis Blog, you can check it here . In business circles, we sometimes hear that today is the “age of the customer.” And so it is. Thanks to the enormous advances over the past few years in consumer techno…
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
The viewer will learn the benefit of using external CSS files and the relationship between class and ID selectors. Create your external css file by saving it as style.css then set up your style tags: (CODE) Reference the nav tag and set your prop…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question