SEO BLock Url to get indexed in Google.

Posted on 2011-03-03
Medium Priority
Last Modified: 2012-05-11
I want to block url from getting indexed in Google. Please help me in doing that. I know how to use robots.txt for restricting folders but how can I block URL indexing.

And How to check if SSL certificate is having right info or not.

Please help.
Question by:ravigosai
  • 3
  • 3
  • 2
  • +1
LVL 14

Expert Comment

by:Dhanasekaran Sengodan
ID: 35025644
place this line  in robots.txt file

Disallow: /file-name.html

LVL 14

Expert Comment

by:Dhanasekaran Sengodan
ID: 35025923
place this line  in robots.txt file

User-agent: *
Disallow: /

Author Comment

ID: 35029852
How can I block URL indexing by using Disallow:/

I didnt get it. The URL is not main website. still the URL is being redirected to the main landing page of the site. I dont know how to make sure that google does not index this particular URL.
Evaluating UTMs? Here's what you need to know!

Evaluating a UTM appliance and vendor can prove to be an overwhelming exercise.  How can you make sure that you're getting the security that your organization needs without breaking the bank? Check out our UTM Buyer's Guide for more information on what you should be looking for!


Author Comment

ID: 35030323
The URL I am trying to block lands on the same page as our main URL lands. is there a way to block indexing on it without redirecting it to 404?
LVL 23

Expert Comment

by:Tony McCreath
ID: 35043481
Could you clarify what you want.

Do you really want to stop Google from indexing your home page?

It sounds like a very bad move. Could you explain why you want to do that.
LVL 29

Accepted Solution

fibo earned 2000 total points
ID: 35043542
B-) I must confess I am slightly confused at what is your problem and the solution you are seeking for.

Here is what I understand:
You have one file, say index.html
The content of this file can be displayed by 2 URLs, say mydomain.com/ (with or without the trailing slash) and /index.html
For SEO purpose, you would want search engines to keep the url / but not /index.html

Since it is the same file in 2 URLs disguises, you cannot use the otherwise ad-hoc tools, such has a <robot noindex> metatag in the html code.

There are 2 different problems
1 - Why are there references to index.html?
2 - How can I remove the url from the known urls?

1. Remove from YOUR site links to the bad URL

Point 1 is usually, and in this case most probably, created by you: your site itself references /index.html instead of /
So this is really your first item on the check list: change your source code so that "home" is / and not /index.html
ANY OTHER ACTION IS USELESS IF THIS IS NOT DONE, since you are nurturing the problem!
Since it is not that easy to check that there remains no link to the bad url, use one tool like Xenu http://home.snafu.de/tilman/xenulink.html which will make a list of the URLs linked to from within your site, as well as (this is the useful part here!) which pages are linking to this URL

At the same time, you might also consider using the canonical metatag to reinforce the above work. See page
Note that this is not an alternative to tidying your own site from references to the "bad" url

2. Remove the URL from the search indexes

Once this is solved, you now need to remove the url /index.html from the indexes of Google + Bing + Yahoo.
Just connect to their webmasters tools, validate you site (my preferred solution is one extra file, but if fails in some situations). While you are at it, communicate them any sitemap you might have done --after checking that the "bad" url is absent from the map!--
Wait 2 days, then come back to the tool, check if they have recorded the "bad" url and if yes, use the tools to remove it.

Finally, you might use some url_rewrite... but I would not do that unless after 4 weeks the problem is still there.

Author Comment

ID: 35072454
Basically the site was using the old url before. and then we rebuilded the site and also we changed the URL. so now we were required to redirect the old URL also to new site so that was being done. so now google is indexing the old ULR as duplicate URL. and that is creating issue.
LVL 23

Expert Comment

by:Tony McCreath
ID: 35079662
Still not clear.

Could you provide some example urls with that description?

If you are redirecting the old url to the new url it should be a 301 redirect. This tells search engines to transfer their index from old to new. It your using any other form of redirect the transfer may not happen.

Featured Post

Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

By following these Magento e-commerce development tips, you can increase your website's conversion and profitability. Read this post for more details.
How do you create a user-centered user experience on your website? And what are some things you should consider in the process?
The viewer will receive an overview of the basics of CSS showing inline styles. In the head tags set up your style tags: (CODE) Reference the nav tag and set your properties.: (CODE) Set the reference for the UL element and styles for it to ensu…
The viewer will the learn the benefit of plain text editors and code an HTML5 based template for use in further tutorials.
Suggested Courses

588 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question