?
Solved

We can't figure this out

Posted on 2005-03-16
20
Medium Priority
?
208 Views
Last Modified: 2010-04-27
My site at www.bargainmonkey.com used to have about 100,000 pages indexed at google, and almost as many at Yahoo. Now we are down to 8,000 at google and 17 at yahoo. Yahoo carries the 17 only because we put out rss feeds for each of those pages.
Google keeps dropping more of our pages every day.
We have removed all doorways, started w3c validation on every page, have done everything we can think of, but google still continues to screw us.
Can some pro take a look at the site and tell me what is happening?
Thanks in advance.
0
Comment
Question by:TuscolaCounty
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 10
  • 7
  • 3
20 Comments
 
LVL 33

Expert Comment

by:humeniuk
ID: 13559781
How old is the site?  When did the decline begin and did you make any specific changes at that time?  Are you using a link exchange program?
0
 
LVL 24

Expert Comment

by:duz
ID: 13560245
TuscolaCounty -

The problem you are having with Google is not the same as the one your are having with Yahoo. There is a hell of lot of duplicate pages, see if the Google problem is solved with a 301 redirect http://bargainmonkey.com to http://www.bargainmonkey.com and put in a http://www.bargainmonkey.com/robots.txt.

The Yahoo problem looks like a penalty, how long ago did you remove the doorways?

- duz
0
 
LVL 24

Expert Comment

by:duz
ID: 13560307
TuscolaCounty -

sp! your s/b you're

- duz
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:TuscolaCounty
ID: 13564800
We used to be a static site, now we are a dynamic site using php script and mysql, using datafeed from commission junction. But that change was a long time ago, and from last summer until january of this year we were doinf awesome. Now suddenly we are dropped.
That is the only major change.
We have a robots.txt
we use 301's
yes we have a link exchange section.

What the hell is sp! your s/b you're?
0
 

Author Comment

by:TuscolaCounty
ID: 13564819
Humenjuk- the site is about 3 years old now.  The decline started shortly after going dynamic approximately 1 year ago.  Yes, we were using a link exchange program but have not updated or revised our links in a couple months.  We are working on narrowing the link exchange drastically but have not completed the excercise yet.

duz- I don't understand what you are saying about duplicate pages, could you please elaborate?  Also, could you please explain why you think a redirect like the one you are suggesting would change anything?  And, what would you recommend I put in the robots.txt file?

Thank you.
0
 
LVL 24

Expert Comment

by:duz
ID: 13565386
TuscolaCounty -

>We have a robots.txt

No you don't :)  You have a file called robots.txt that is as far away from what a search engine would expect to see in a real robots.txt as you can possibly get.  Please see http://www.robotstxt.org/wc/norobots.html

>we use 301's

Then why does almost every page like this http://www.bargainmonkey.com/browse/store/Discovery%20Products/ have a duplicate like this http://bargainmonkey.com/browse/store/Discovery%20Products/

>What the hell is sp! your s/b you're?

I misspelled a word and corrected it. Short for: Spelling! 'your' should be 'you're'

>duplicate pages, could you please elaborate?

See above under 'we use 301's'

>could you please explain why you think a redirect like the one you are suggesting would change anything

It would stop Google getting confused with duplicate pages and coupled with a proper robots.txt should return the spiderability/indexing to normal.

>what would you recommend I put in the robots.txt file?

user-agent:*
disallow:

- duz
0
 
LVL 33

Expert Comment

by:humeniuk
ID: 13565452
Hi TuscolaCounty,

I asked because you have a lot of outgoing links (thousands of them).  Often with link exchange programs, you don't get any choice of what links are put on your site.  Sometimes, they are placed there dynamically by an off-site database.  The problem with this is that you have no idea what you're getting.  If the program is providing links to 'bad neighborhoods', it can have a significant negative effect on your rankings.  That is one good reason why it's essential to qualify every single link you put on your site.
0
 

Author Comment

by:TuscolaCounty
ID: 13565665
How are these duplicates
Then why does almost every page like this http://www.bargainmonkey.com/browse/store/Discovery%20Products/ have a duplicate like this http://bargainmonkey.com/browse/store/Discovery%20Products/
it is the same url twice. only no www, how can that constitute as a duplicate page???? If that were the case, everybody would have two sites mirrored.
0
 

Author Comment

by:TuscolaCounty
ID: 13565686
Why put this in a robots.txt file
user-agent:*
disallow:
What does it tell the spider? Looks like it tells the spider not to index the page to me.
0
 
LVL 33

Expert Comment

by:humeniuk
ID: 13565834
"What does it tell the spider? Looks like it tells the spider not to index the page to me."
On the contrary, it tells the spiders there is nothing to disallow.

"only no www, how can that constitute as a duplicate page???? If that were the case, everybody would have two sites mirrored."
Again, on the contrary.  SEs treat subdomains as different websites, so domain.com and www.domain.com are perceived as differently as www.domain.com and sub.domain.com.  Not everyone has two sites mirrored, because you can determine in DNS what URLs/domains can be used to access your website.  The www is redundant, but is used as a matter of convention.  Some people leave the non-www version available as a convenince for users and for  others it's vice versa, but the ideal would be if they both go to the same destination (which is what the redirect is for).

Also, it's only a mirror site if the SEs find it.  If all the links go to www.domain.com and domain.com is never found (but is functional), it wouldn't be regarded as a site.  Just like a site that has no incoming links that Google never finds isn't really regarded as a site either :)
0
 
LVL 24

Expert Comment

by:duz
ID: 13565904
TuscolaCounty -

>What does it tell the spider?

The "*" means all robots and the blank 'disallow' means 'all files' because robots.txt is an 'exclusion standard'.  As I said see http://www.robotstxt.org/wc/norobots.html

Theoretically having this in your robots.txt is the same as not having a robots.txt at all. Both are acceptable, however in your case it will be better to replace the contents with the correct format rather than delete the robots.txt altogether.

Make sure you do the 301 redirect and the robots.txt at the same time.

>Looks like it tells the spider not to index the page to me.

You can learn a lot here on Experts Exchange :)

- duz
0
 

Author Comment

by:TuscolaCounty
ID: 13566121
We did NOT have a robots.txt file but I will add one.  The page you are seeing is our errordocument page.  You should see error 404 not found if you check the headers.  Here is the redirect I put in the .htaccess file but it does NOT give a 301 code when accessing it so I am not sure if the format is wrong or it just doesn't work that way.  Without a 301 will it make any difference to the search engines?  redirect 301 http://bargainmonkey.com http://www.bargainmonkey.com
Thanks.
0
 
LVL 24

Accepted Solution

by:
duz earned 1000 total points
ID: 13566433
TuscolaCounty -

Put this in your .htaccess

RewriteEngine on
RewriteCond %{HTTP_HOST} ^bargainmonkey\.com [NC]
RewriteRule ^(.*)$ http://www.bargainmonkey.com/$1 [R=301,L]

- duz
0
 

Author Comment

by:TuscolaCounty
ID: 13566668
That now throws a 301 code, thanks.  Now that I see it, I remember reading about it somewhere.  Jsut as confirmation the "\" between the bargainmonkey and the .com is supposed to be there?  Not really questioning your expertise, the rewrite works, but that just looks weird.  Thank you for all the help and I guess we try this for a couple weeks and see how it goes?  We are also manually requesting Google to remove our old doorways from their index (since they are not falling out very fast, if at all, on their own).
0
 

Author Comment

by:TuscolaCounty
ID: 13566948
Forgot to answer the "how long ago did you remove the doorways" question.  We removed the doorways summer 2004 and put redirects in to send any traffic to the appropriate "good content pages" but the redirect was formatted wrong and was giving a 302 instead of a 301.  We fixed this in October and enjoyed an AWESOME November and even better December for traffic and then ever since the first of the year it has been getting worse and worse.  We were losing pages in Google for a while, then we were gaing them and then recently they have been falling out fast.  Stopping the fallout at Google and getting back into Yahoo are our main concerns right now.  We have emailed Yahoo and actually received a response and will wait a few more days or a week or so to see if these changes help or not.  We are getting spidered everyday all day long by the big three just not getting indexed at Yahoo.
0
 
LVL 24

Expert Comment

by:duz
ID: 13567352
TuscolaCounty -

>We removed the doorways summer 2004

OK

>supposed to be there?

Yes

- duz


0
 

Author Comment

by:TuscolaCounty
ID: 13567362
Ok, thanks
0
 

Author Comment

by:TuscolaCounty
ID: 13574378
One more question about the redirect...  Should I also redirect www.bargainmonkey.com/index.php to www.bargainmonkey.com?  If so, how would I do that then?

Thank you.
0
 
LVL 24

Expert Comment

by:duz
ID: 13575608
TuscolaCounty -

>Should I also redirect...

No you just want to do what you are doing which is redirecting every http://bargainmonkey.com request to the equivelent page on http://www.bargainmonkey.com.

- duz




0
 

Author Comment

by:TuscolaCounty
ID: 13575727
Ok, thanks again.
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this blog, I will share you some basic tips for content marketing and to rank your website on Google.
Starting your own business is always a daunting process, and for most people it is brand new experience. Avoid the common pitfalls by following these tips to start on the road to success.
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to selectively show certain fields based on user input using rules to gather relevant information and data from your forms. The rules feature provides you with an opportunity…
Learn how to set-up custom confirmation messages to users who complete your Wufoo form. Include inputs from fields in your form, webpage redirects, and more with Wufoo’s confirmation options.
Suggested Courses

741 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question