Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.
COURSE of the MONTH
14 days, 19 hours left to enrollBecome a Premium Member and unlock a new, free course in leading technologies each month.
Solved
Googlebot asking for strange pages & NOT index.htm
Posted on 2004-10-13
Hi,
I created a website ( www.thefirstaidbox.co.uk ) a few months ago & it still has not been crawled by a googlebot properly. I looked at my logs & googlebot has visited twice - each time it did a GET (only) for the following :
/robots.txt - Expected this (& I guess I should create one)
/search.html - No such page (So I'll probably create one!)
/cgi-bin/ocb/ocb.cgi
/quikstore.html
/quikcode.html+
The last three appear to be related to some e-commerce software - which this site doesn't use (we use a different package)
I'm very confused as to why the googlebot asked for these pages & why it didn't try & crawl from index.htm(l) - The site homepage WAS submitted to google.
I'm aware that the site is short of both links and content - which is being worked on - but that doesn't explain the odd googlebot behaviour.
Any ideas?
I created a website ( www.thefirstaidbox.co.uk ) a few months ago & it still has not been crawled by a googlebot properly. I looked at my logs & googlebot has visited twice - each time it did a GET (only) for the following :
/robots.txt - Expected this (& I guess I should create one)
/search.html - No such page (So I'll probably create one!)
/cgi-bin/ocb/ocb.cgi
/quikstore.html
/quikcode.html+
The last three appear to be related to some e-commerce software - which this site doesn't use (we use a different package)
I'm very confused as to why the googlebot asked for these pages & why it didn't try & crawl from index.htm(l) - The site homepage WAS submitted to google.
I'm aware that the site is short of both links and content - which is being worked on - but that doesn't explain the odd googlebot behaviour.
Any ideas?
[X]
Welcome to Experts Exchange
Add your voice to the tech community where 5M+ people just like you are talking about what matters.
- Help others & share knowledge
- Earn cash & points
- Learn & ask questions
2
3 Comments
naha -
>I'm very confused as to why the googlebot asked for these pages
There are many versions of googlebot and some of them can be capricious :)
Not this one though! Robots.txt is just the ritual beginning then I didn't have far to look to account for the others http://www.thefirstaidbox.com/ and this is where it found /search.html , /cgi-bin/ocb/ocb.cgi etc.
>I'm aware that the site is short of both links and content
Very much so.
- duz
>I'm very confused as to why the googlebot asked for these pages
There are many versions of googlebot and some of them can be capricious :)
Not this one though! Robots.txt is just the ritual beginning then I didn't have far to look to account for the others http://www.thefirstaidbox.com/ and this is where it found /search.html , /cgi-bin/ocb/ocb.cgi etc.
>I'm aware that the site is short of both links and content
Very much so.
- duz
The site is .co.uk not .com
Hmmm - Looks like the googlebot is looking at the .co.uk site and expecting the same structure!
V V strange. - They are on sperate & unrelated servers.
Ahh - I see what's happened the other site (.com) was set up by a guy who took 10months or so (apparently) to produce it (minute as it is - must have taken all of 5mins) - & he has some duff links to the .co.uk site - Guess I'll take advantage of them then & create real pages!
Hmmm - Looks like the googlebot is looking at the .co.uk site and expecting the same structure!
V V strange. - They are on sperate & unrelated servers.
Ahh - I see what's happened the other site (.com) was set up by a guy who took 10months or so (apparently) to produce it (minute as it is - must have taken all of 5mins) - & he has some duff links to the .co.uk site - Guess I'll take advantage of them then & create real pages!
naha -
>expecting the same structure
Not the same structure so much but this googlebot IS expecting to find the named page at the end of the link. This is a good example of why it is better to help the googlebot find your new site rather than use the 'Submit Your Site' facility. It allows you to manage the discovery process - a factor that is often overlooked by SEOs.
- duz
>expecting the same structure
Not the same structure so much but this googlebot IS expecting to find the named page at the end of the link. This is a good example of why it is better to help the googlebot find your new site rather than use the 'Submit Your Site' facility. It allows you to manage the discovery process - a factor that is often overlooked by SEOs.
- duz
Featured Post
Question has a verified solution.
If you are experiencing a similar issue, please ask a related question
Learn about the eCommerce marketing trends for the year ahead.
Choosing a core focus or particular set of features and options can be tough. To help out, we’re going to highlight a handful of things your business needs on one of your social media pages. In other words, if one of these is missing, you should imp…
Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments.
What Bitcoin is:
Legality:
Risks:
Benefits:
Which businesses are best suited?:
Other things you should know:
How to get started:
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to selectively show certain fields based on user input using rules to gather relevant information and data from your forms. The rules feature provides you with an opportunity…
Suggested Courses
Course of the Month14 days, 19 hours left to enroll
770 members asked questions and received personalized solutions in the past 7 days.
Join the community of 500,000 technology professionals and ask your questions.