The <noscript> tag and spiders

My web site currently is heavily dependent on javascript. Accordingly, I have put a <noscript> section on the home page to inform users that they must either enable scripting or upgrade their browser if needed.

However, in our efforts to optimize our site for search engines, a process we are just starting, it appears that such an approach also blocks the spiders from indexing our site.

We are moving towards a site redesign that will severely reduce the use of javascript. So I am not looking for the answer "redesign your site to get rid of javascript" as we are moving in that direction already.

In the meantime, what is the best strategy to allow the site to be spidered, yet give assistance to those with disabled scripting?
jasimon9Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

humeniukCommented:
Can you post your URL here so we can take a look?
0
jasimon9Author Commented:
www.rephunter.net

I will be out of town Nov 11 - 15, so will have to respond further after that. Thanks!
0
humeniukCommented:
I think the problem you're having is that you've placed a meta-redirect inside your <noscript> tags.  The spiders are redirected to the noscript.asp page (which they crawl) and away from the source page (which they don't crawl).  Because you have the <noscript> tags on every page, anywhere the spider goes, it ends up at the noscript.asp page, much as a javaless browser would.

To rectify this, you have to replace the meta-refresh inside the <noscript> tags with something else, like a text link to the noscript.asp page.  That way, the spider will see it and follow it, but not at the expense of disregarding the source page.

Remember that <noscript> tags and meta-refresh redirects are two methods that have been used for keyword spamming, so SE's are wary of them.  If you had filled your <noscript> tags with irrelevant keyword-rich content (you didn't), you would be penalized.  Likewise, the SE's don't want to be fooled by an irrelevant keyword-rich page that visitors never see because they are redirected to a different page, so they'll index the destination page, but not the source page.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Introducing Cloud Class® training courses

Tech changes fast. You can learn faster. That’s why we’re bringing professional training courses to Experts Exchange. With a subscription, you can access all the Cloud Class® courses to expand your education, prep for certifications, and get top-notch instructions.

duzCommented:
jasimon9 -

>what is the best strategy to allow the site to be spidered

Create a 'site map' page with links to all the pages you want indexed by the search engines. Put a link to the site map at the foot of every page on the site.

Aim to keep the site map page after you complete the redesign.

- duz
0
humeniukCommented:
www.rephunter.net/sitemap.asp - although I would opt for for something a bit simpler (ie. more text based)
0
duzCommented:
humeniuk -

www.rephunter.net/sitemap.asp is a site map in name only, unindexed and with only seven spiderable links. jasimon9 can quite safely leave it to languish and ignore it while he creates a sit map page as I suggested.

- duz
0
jasimon9Author Commented:
I think the best solution is foreshadowed in the first comment of humeniuk. I am going to try to work with that. If I can, I would like to have the <noscript> tag trigger some additional explanation but keep all the home page material. Or something like that. It seems to me I should be able to get somewhere with such an approach. Thus at least a majority of points to humeniuk.

But I have some questions for duz regarding the comment about the site map.

I believe by "in name only" you are referring to the fact that there are a lot of javascript popups rather than plain links. If so, I completely understand, and that fits into our redesign to make them into plain pages, which then of course can be real links. Did you mean something other than this?

But what do you mean by "unindexed?"
0
humeniukCommented:
Yes, you definitely need to use simple text links rather than anything javascript based.  Beyond that, I don't want to speak for duz as I'm sure he can give you a more comprehensive answer on this very important element.
0
jasimon9Author Commented:
I have succeeded in making a step forward which I think will work.

I have replaced the meta refresh with a text notice and a link. The link goes to an expanded explanation page that not only referes to javascript, but also to cookies.

This probably solves the immediate problem, although of course only as a stop-gap until the redesign can become effective.

Thanks for your help.

Even though our site map strategy is not directly related to the question, and that we were going to (of course) include the site map in the redesigned site, I am also awarding some "token points" to duz for his/her contribution.
0
humeniukCommented:
Glad to be of some help.  Thanks for the A and good luck with the redesign.
0
duzCommented:
jasimon9 -

I understood that your original question was "what is the best strategy to allow the site to be spidered" while you were redesigning your site to "get rid of javascript".

As I said above the best strategy is to "Create a 'site map' page with links to all the pages you want indexed by the search engines. Put a link to the site map at the foot of every page on the site. Aim to keep the site map page after you complete the redesign."

>I have some questions for duz regarding the comment about the site map. I believe by "in name only" you are referring to the fact that there are a lot of javascript popups rather than plain links.

That is correct.  The existing site map page does not 'map the site' because of unspiderable links.

>But what do you mean by "unindexed?"

Not indexed by Google.

>Even though our site map strategy is not directly related to the question

If  replacing "the meta refresh with a text notice and a link" solves your problem then I have misunderstood your question because that does nothing to allow the site to be spidered.

- duz
0
humeniukCommented:
Having re-read the question, I think duz has a good point.  My focus was on the <noscript> tag because that is referred to in the title of the question as well as the first three paragraphs.  However, the real question (which duz identified correctly) is, how do you get the website indexed?

The answer to that is to make sure you have a comprehensive, effective navigation structure based on text links.  The best way to do this is to add a proper site map (all that javascript doesn't do you any favors, but that is being addressed in the redesign and a site map would get around most of that anyway).  I still think you need to get rid of the refresh in your <noscript> tag, but without proper navigation, that's a moot point, because the SE spiders won't be able to navigate your site.

Based on that, jasimon9, I hope you will revisit this question and reconsider your choice.  You can post a request in the Community Support TA (www.experts-exchange.com/Community_Support/) to re-open the question.
0
jasimon9Author Commented:
Actually some further clarification is useful. The site map may be the best approach. However, we have already included the site map and would naturally have kept it in the redesign, recognizing how important it is, and that it "works around" certain aspects of the navigation that the spiders cannot navigate.

Having said that, I do recognize the point that our current "site map" is not a true site map for the reasons thoroughly discussed above, and that we plan on dealing with this. I awarded points to duz for duz's contibutions to that part of the discussion.

I believe that replacing "the meta refresh with a text notice and a link" is in fact vital, specifically because the meta refresh is what is currently preventing the spidering of the site--every time the spider comes to any page, it is redirected to a single page. This would be why, for example, the so-called site map is not indexed. So not only does it not "do nothing"--it is absolutely essential and the site (other than the noscript.asp page) will not be spidered at all without it!

I don't believe we need to re-open this question. And I believe I am on the right track, both for the short run workaround and the eventual rewrite of the site.

I would like to put this to rest for now. If there is still disagreement or alternate views, I am glad to hear them. However, if so, then I must be completely misunderstanding what is being said.
0
humeniukCommented:
Thank you for considering the proposal.  Good luck with the website.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Internet Marketing

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.