• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 781
  • Last Modified:

'Robots.txt is blocking an important page', according to Google

I have a website that has 'sever health issues' & according to google webmaster tools, robots.txt is blocking the home page which is total B.S (I will attach the file here).

Now the SERP's say that the site description can't be displayed because of robots.txt.

It's a Joomla 2.5x site, set to index,follow with an XML site map. The funny thing is when you go into the detailed view in webmaster tools, it says everything is perfectly healthy!

This is very frustrating, does anyone think they may know what's up?
robots.txt
0
TonyCabone
Asked:
TonyCabone
  • 5
  • 2
  • 2
  • +1
3 Solutions
 
savoneCommented:
You never end the user-agent "segment".  Try putting a space after it, like this... Then check webmaster tools again.

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

Sitemap: http://www.siteaddress.com.au/index.php?option=com_xmap&view=xml&id=1

Open in new window

0
 
TonyCaboneAuthor Commented:
Thanks savone, I did this & it says 'Valid Sitemap reference detected', but it did that before adding the white space.
0
 
savoneCommented:
But did it say it is blocking the homepage?  Which is what your question was about.
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
TonyCaboneAuthor Commented:
Yes it still says that robots.txt is 'blocking some important page'.
0
 
Tony McCreathTechnical SEO ConsultantCommented:
Have you checked the file in GWMT to see if they see the same robots.txt file.

How about removing the robots.txt for a while and seeing if that fixes it.Then you will know if it really is that file or something else.
0
 
COBOLdinosaurCommented:
Why are you using a complex url for the sitemap location?
it should be in the root directory so you can reference it as:

Sitemap: http://www.siteaddress.com.au/sitemap.xml

instead of:
Sitemap: http://www.siteaddress.com.au/index.php?option=com_xmap&view=xml&id=1

The spiders might be having a problem resolving the url.


Cd&
0
 
TonyCaboneAuthor Commented:
Ok so setup a redirect in .htaccess? It's generated using Xmap plugin for Joomla.
0
 
TonyCaboneAuthor Commented:
Triggerito - what about all the CMS directories, does it matter if they get indexed?
0
 
TonyCaboneAuthor Commented:
COBOLdinasaur - FYI, with reference to that link GWMT says "Valid Sitemap reference detected", does that mean the bots are OK with it?
0
 
Tony McCreathTechnical SEO ConsultantCommented:
Google will only index what it can find via links, so it should not find your admin page and cannot enter the admin sections.

If Joomla is written correctly those pages should be noindex anyhow.

That SERP feature of displaying the warning is very new. Maybe it's just a glitch in Googles algo.

How about trying the fetch as google bot option in GWMT.

If not it may be worth raising the issue in a Google forum where googlers hang out.
0
 
COBOLdinosaurCommented:
"Valid Sitemap reference detected", does that mean the bots are OK with it?

That means it found the reference in robots.txt  

It does not mean it actually accessed the sitemap.


Cd&
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

  • 5
  • 2
  • 2
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now