Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 259
  • Last Modified:

Why do sitemap generators not find my site links?

Hi Experts!

Probably a simple thing I am overlooking here, but I have a client site www.cliveelliott.net and am trying to generate a sitemap for it, but it is failing to find the pages.  I have checked obvious stuff; turned off javascript and checked that the menu system still shows as <a> links - it does.

I've tried an online sitemap generator, www.xml-sitemaps.com (finds only 5 pages) and also GSiteCrawler (finds zero pages!).

I know the home page doesn't validate, but its only a couple of issues that would think would not cause the entire menu not to be reachable??

Any ideas gratefully received!!

Thanks,

TheFoot
0
Barry Jones
Asked:
Barry Jones
  • 5
  • 4
1 Solution
 
Greg AlexanderLead DeveloperCommented:
It is because your hosting seems to want to redirect cliveelliott.net to www.cliveelliott.net I was able to scan it with Xenu and produced substantial results.
0
 
Greg AlexanderLead DeveloperCommented:
Maybe some sort of rewrite rule?
0
 
Barry JonesAuthor Commented:
Hi galexander07,

Thanks for your quick response.   I do indeed have htaccess rules - to redirect cliveelliott.net to www.cliveelliott.net.  Would this really cause spidering issues?

Options +FollowSymlinks
RewriteEngine on
rewritecond %{http_host} ^cliveelliott.net [nc]
rewriterule ^(.*)$ http://www.cliveelliott.net/$1 [r=301,nc]

Thanks, TheFoot
0
[Video] Oticon Case Study

Open office environments can create the dynamics for innovation, but they also bring some challenges. With over 1,000 employees in an open office, Oticon needed a solution that would preserve the environment while mitigating disruptive background noises.

Watch how they did it.

 
Greg AlexanderLead DeveloperCommented:
I don't think search engines will have a problem with that, however this little site search programs could. I know that is what tripped Xenu up... maybe a PHP header would work better in your case.. Is there a specific reason for the redirect?
0
 
Barry JonesAuthor Commented:
I've just downloaded Xenu - and it found all pages as you said.  I'll use that to generate the sitemap - thanks.

Do you think that I can expect any spidering issues from the redirect?

Thanks..
0
 
Barry JonesAuthor Commented:
Sorry - posted as received yours..

I wanted to maintain a single domain for google webmaster tools - they recommend picking one - either www or on-www and sticking with it.
0
 
Greg AlexanderLead DeveloperCommented:
No,  the search engines should be smarter then these programs... I would suggest doing a little research, but I have never heard of any issues like that.
0
 
Barry JonesAuthor Commented:
Thanks for your quick response!
0
 
Greg AlexanderLead DeveloperCommented:
Thanks for the points!
0

Featured Post

Reclaim your office - Try the MB 660 headset now!

High level of background noise often makes it difficult for employees to concentrate fully on their jobs – or to communicate clearly on calls. The MB 660 headset helps you create a disruption free workspace.  

  • 5
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now