[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

The <noscript> tag and spiders

Posted on 2004-11-10
14
Medium Priority
?
600 Views
Last Modified: 2010-04-07
My web site currently is heavily dependent on javascript. Accordingly, I have put a <noscript> section on the home page to inform users that they must either enable scripting or upgrade their browser if needed.

However, in our efforts to optimize our site for search engines, a process we are just starting, it appears that such an approach also blocks the spiders from indexing our site.

We are moving towards a site redesign that will severely reduce the use of javascript. So I am not looking for the answer "redesign your site to get rid of javascript" as we are moving in that direction already.

In the meantime, what is the best strategy to allow the site to be spidered, yet give assistance to those with disabled scripting?
0
Comment
Question by:jasimon9
  • 7
  • 4
  • 3
14 Comments
 
LVL 33

Expert Comment

by:humeniuk
ID: 12549563
Can you post your URL here so we can take a look?
0
 

Author Comment

by:jasimon9
ID: 12551697
www.rephunter.net

I will be out of town Nov 11 - 15, so will have to respond further after that. Thanks!
0
 
LVL 33

Accepted Solution

by:
humeniuk earned 400 total points
ID: 12551997
I think the problem you're having is that you've placed a meta-redirect inside your <noscript> tags.  The spiders are redirected to the noscript.asp page (which they crawl) and away from the source page (which they don't crawl).  Because you have the <noscript> tags on every page, anywhere the spider goes, it ends up at the noscript.asp page, much as a javaless browser would.

To rectify this, you have to replace the meta-refresh inside the <noscript> tags with something else, like a text link to the noscript.asp page.  That way, the spider will see it and follow it, but not at the expense of disregarding the source page.

Remember that <noscript> tags and meta-refresh redirects are two methods that have been used for keyword spamming, so SE's are wary of them.  If you had filled your <noscript> tags with irrelevant keyword-rich content (you didn't), you would be penalized.  Likewise, the SE's don't want to be fooled by an irrelevant keyword-rich page that visitors never see because they are redirected to a different page, so they'll index the destination page, but not the source page.
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 24

Assisted Solution

by:duz
duz earned 100 total points
ID: 12552786
jasimon9 -

>what is the best strategy to allow the site to be spidered

Create a 'site map' page with links to all the pages you want indexed by the search engines. Put a link to the site map at the foot of every page on the site.

Aim to keep the site map page after you complete the redesign.

- duz
0
 
LVL 33

Expert Comment

by:humeniuk
ID: 12552840
www.rephunter.net/sitemap.asp - although I would opt for for something a bit simpler (ie. more text based)
0
 
LVL 24

Expert Comment

by:duz
ID: 12552934
humeniuk -

www.rephunter.net/sitemap.asp is a site map in name only, unindexed and with only seven spiderable links. jasimon9 can quite safely leave it to languish and ignore it while he creates a sit map page as I suggested.

- duz
0
 

Author Comment

by:jasimon9
ID: 12635752
I think the best solution is foreshadowed in the first comment of humeniuk. I am going to try to work with that. If I can, I would like to have the <noscript> tag trigger some additional explanation but keep all the home page material. Or something like that. It seems to me I should be able to get somewhere with such an approach. Thus at least a majority of points to humeniuk.

But I have some questions for duz regarding the comment about the site map.

I believe by "in name only" you are referring to the fact that there are a lot of javascript popups rather than plain links. If so, I completely understand, and that fits into our redesign to make them into plain pages, which then of course can be real links. Did you mean something other than this?

But what do you mean by "unindexed?"
0
 
LVL 33

Expert Comment

by:humeniuk
ID: 12635940
Yes, you definitely need to use simple text links rather than anything javascript based.  Beyond that, I don't want to speak for duz as I'm sure he can give you a more comprehensive answer on this very important element.
0
 

Author Comment

by:jasimon9
ID: 12636515
I have succeeded in making a step forward which I think will work.

I have replaced the meta refresh with a text notice and a link. The link goes to an expanded explanation page that not only referes to javascript, but also to cookies.

This probably solves the immediate problem, although of course only as a stop-gap until the redesign can become effective.

Thanks for your help.

Even though our site map strategy is not directly related to the question, and that we were going to (of course) include the site map in the redesigned site, I am also awarding some "token points" to duz for his/her contribution.
0
 
LVL 33

Expert Comment

by:humeniuk
ID: 12636574
Glad to be of some help.  Thanks for the A and good luck with the redesign.
0
 
LVL 24

Expert Comment

by:duz
ID: 12637109
jasimon9 -

I understood that your original question was "what is the best strategy to allow the site to be spidered" while you were redesigning your site to "get rid of javascript".

As I said above the best strategy is to "Create a 'site map' page with links to all the pages you want indexed by the search engines. Put a link to the site map at the foot of every page on the site. Aim to keep the site map page after you complete the redesign."

>I have some questions for duz regarding the comment about the site map. I believe by "in name only" you are referring to the fact that there are a lot of javascript popups rather than plain links.

That is correct.  The existing site map page does not 'map the site' because of unspiderable links.

>But what do you mean by "unindexed?"

Not indexed by Google.

>Even though our site map strategy is not directly related to the question

If  replacing "the meta refresh with a text notice and a link" solves your problem then I have misunderstood your question because that does nothing to allow the site to be spidered.

- duz
0
 
LVL 33

Expert Comment

by:humeniuk
ID: 12638776
Having re-read the question, I think duz has a good point.  My focus was on the <noscript> tag because that is referred to in the title of the question as well as the first three paragraphs.  However, the real question (which duz identified correctly) is, how do you get the website indexed?

The answer to that is to make sure you have a comprehensive, effective navigation structure based on text links.  The best way to do this is to add a proper site map (all that javascript doesn't do you any favors, but that is being addressed in the redesign and a site map would get around most of that anyway).  I still think you need to get rid of the refresh in your <noscript> tag, but without proper navigation, that's a moot point, because the SE spiders won't be able to navigate your site.

Based on that, jasimon9, I hope you will revisit this question and reconsider your choice.  You can post a request in the Community Support TA (www.experts-exchange.com/Community_Support/) to re-open the question.
0
 

Author Comment

by:jasimon9
ID: 12641467
Actually some further clarification is useful. The site map may be the best approach. However, we have already included the site map and would naturally have kept it in the redesign, recognizing how important it is, and that it "works around" certain aspects of the navigation that the spiders cannot navigate.

Having said that, I do recognize the point that our current "site map" is not a true site map for the reasons thoroughly discussed above, and that we plan on dealing with this. I awarded points to duz for duz's contibutions to that part of the discussion.

I believe that replacing "the meta refresh with a text notice and a link" is in fact vital, specifically because the meta refresh is what is currently preventing the spidering of the site--every time the spider comes to any page, it is redirected to a single page. This would be why, for example, the so-called site map is not indexed. So not only does it not "do nothing"--it is absolutely essential and the site (other than the noscript.asp page) will not be spidered at all without it!

I don't believe we need to re-open this question. And I believe I am on the right track, both for the short run workaround and the eventual rewrite of the site.

I would like to put this to rest for now. If there is still disagreement or alternate views, I am glad to hear them. However, if so, then I must be completely misunderstanding what is being said.
0
 
LVL 33

Expert Comment

by:humeniuk
ID: 12645055
Thank you for considering the proposal.  Good luck with the website.
0

Featured Post

Meet the Family that is Made for Collaboration

The TeamConnect Family product group as part of the Sennheiser for Business Portfolio comprising high-quality, technically well-conceived meeting solutions for business communication – designed for any meeting room and any meeting situation.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Starting your own business is always a daunting process, and for most people it is brand new experience. Avoid the common pitfalls by following these tips to start on the road to success.
There is a massive demand for content on the web right now, and it doesn't look like it's going to stop any time soon. But, if you are running a business blog, it's not just enough to offer your audience lots of content. It needs to be high-quality…
An overview of how to create reports in Adobe Analytics (formerly Omniture Site Catalyst) using pageNames, events, eVars and props. This video will show you how to install the Omniture Debugger tool so can see (and test) what is being passed int…
Learn how to set-up PayPal payment integration in your Wufoo form. Allow your users to remit payment through PayPal upon completion of your online form. This is helpful for collecting membership payments, customer payments, donations, and more.
Suggested Courses
Course of the Month18 days, 10 hours left to enroll

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question