Solved

Googlebot asking for strange pages & NOT index.htm

Posted on 2004-10-13
3
267 Views
Last Modified: 2010-04-27
Hi,
  I created a website ( www.thefirstaidbox.co.uk ) a few months ago & it still has not been crawled by a googlebot properly. I looked at my logs & googlebot has visited twice - each time it did a GET (only) for the following :

/robots.txt               - Expected this (& I guess I should create one)
/search.html           - No such page  (So I'll probably create one!)
/cgi-bin/ocb/ocb.cgi
/quikstore.html
/quikcode.html+

The last three appear to be related to some e-commerce software - which this site doesn't use (we use a different package)

I'm very confused as to why the googlebot asked for these pages & why it didn't try & crawl from index.htm(l) - The site homepage WAS submitted to google.

I'm aware that the site is short of both links and content - which is being worked on - but that doesn't explain the odd googlebot behaviour.

Any ideas?
0
Comment
Question by:naha
  • 2
3 Comments
 
LVL 24

Accepted Solution

by:
duz earned 250 total points
ID: 12302373
naha -

>I'm very confused as to why the googlebot asked for these pages

There are many versions of googlebot and some of them can be capricious :)

Not this one though! Robots.txt is just the ritual beginning then I didn't have far to look to account for the others  http://www.thefirstaidbox.com/ and this is where it found /search.html ,  /cgi-bin/ocb/ocb.cgi  etc.

>I'm aware that the site is short of both links and content

Very much so.

- duz      
0
 
LVL 1

Author Comment

by:naha
ID: 12303014
The site is .co.uk not .com

Hmmm - Looks like the googlebot is looking at the .co.uk site and expecting the same structure!
V V strange. - They are on sperate & unrelated servers.

Ahh - I see what's happened the other site (.com) was set up by a guy who took 10months or so (apparently) to produce it (minute as it is - must have taken all of 5mins) - & he has some duff links to the .co.uk site - Guess I'll take advantage of them then & create real pages!

0
 
LVL 24

Expert Comment

by:duz
ID: 12305047
naha -

>expecting the same structure

Not the same structure so much but this googlebot IS expecting to find the named page at the end of the link.  This is a good example of why it is better to help the googlebot find your new site rather than use the 'Submit Your Site' facility. It allows you to manage the discovery process - a factor that is often overlooked by SEOs.

- duz
0

Featured Post

Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
seo optimization 5 70
Chart on Facebook 4 98
Traffic Monsoon scam or good site for make money 1 42
are there any terms against sharing a post or link to Google + ? 2 84
Read about how to approach blogging and about ways to do it right. Stand out from the crowd and let your knowledge be consumed by a large audience. This article aims to explain how your blog should look like,  the most important things to do while b…
For many of us, the  holiday season kindles the natural urge to give back to our friends, family members and communities. While it's easy for friends to notice the impact of such deeds, understanding the contributions of businesses and enterprises i…
An overview of how to create reports in Adobe Analytics (formerly Omniture Site Catalyst) using pageNames, events, eVars and props. This video will show you how to install the Omniture Debugger tool so can see (and test) what is being passed int…
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to choose which pages of your form are visible to your users based on their inputs. The page rules feature provides you with an opportunity to create if:then statements for y…

773 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question