[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Weird Google Indexing

Posted on 2005-05-15
18
Medium Priority
?
224 Views
Last Modified: 2010-04-27
I am not currently listed on google's search which would obviously help my site tremendously. I've tried adding myself to dmoz and they never will add me. Im really wanting to get indexed and Im not sure why I'm not. I have plenty of inbound links across numerous popular sites including howstuffworks.com.

I did find some interesting stuff in my raw log manager though. The following two lines shows some activity from the googlebot, but it just stops after these two lines. Is it hitting some bad code or something in my site?

crawl-66-249-66-40.googlebot.com - - [03/May/2005:02:48:00 +0100] "GET /robots.txt HTTP/1.1" 404 10010 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
crawl-66-249-66-40.googlebot.com - - [03/May/2005:02:48:00 +0100] "GET / HTTP/1.1" 200 30037 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

By the way my robots.txt includes the following like it is supposed to:
User-agent: *
Disallow:

Question is why does googlebot stop right there and obviously not continue indexing my site?

Thanks
0
Comment
Question by:stjowa
  • 8
  • 7
16 Comments
 
LVL 2

Expert Comment

by:mcse20002000
ID: 14008268
robots.txt is not doing any thing useful. but may be stopping the bot to index it

remove the txt file

is it even not showing on most wierd search terms?
0
 
LVL 24

Expert Comment

by:duz
ID: 14008298
stjowa -

>why does googlebot stop right there

It can take several visits. If you give us the url we can look for other possibilities.

- duz
0
 

Author Comment

by:stjowa
ID: 14009561
> remove the txt file

Ok... did that.

 > is it even not showing on most weird search terms?

What do you mean?

 > It can take several visits.

Define several... it already has a few times.

 > If you give us the url we can look for other possibilities.

www.vehicleaid.com -- you might have seen me here before :)
0
Sign your company up to try the MB 660 headset now

Take control and stay focused in noisy open office environments with the MB 660. By reducing background noise, you can revitalize your office and improve concentration.

 
LVL 24

Expert Comment

by:duz
ID: 14011028
stjowa -

>Define several

More than one, sometimes Google visits several times before a page will appear in the index.

I took a look and I have two recommendations. The first is to get many more inbound links.  You can start by submitting to  (say) a 100 free directories http://info.vilesilencer.com/main.php?rock=seo-friendly.php then you can embark on a full blown link building campaign.

Secondly I suggest you have a sitemap with a link from the homepage, this will help Google spider the site. Text links with the full url.

Google will index the cgi-bin and .pl files but I have seen these cause difficulties in the past.

- duz
0
 

Author Comment

by:stjowa
ID: 14015555
How may the cgi-bin and .pl files cause difficulties and how would you fix it.
0
 
LVL 24

Expert Comment

by:duz
ID: 14016096
stjowa -

What forum script are you using or is it your own?

- duz


0
 

Author Comment

by:stjowa
ID: 14016240
it is my own custom built question and answer script (similar to ee)
0
 
LVL 24

Expert Comment

by:duz
ID: 14016897
stjowa -

OK. Well ideally you want your site to have search engine friendly urls so that for example http://www.vehicleaid.com/cgi-bin/openthread.pl?ThreadID=6 was something like this Mechanics http://www.vehicleaid.com/engine/fan-belt-tensioner.html with a title like this <title>Fan Belt Tensioner</title>.  I would try a sitemap sitemap.html with a link from the homepage first and see if once Google indexes the sitemap if it indexes the links on it.

- duz
0
 

Author Comment

by:stjowa
ID: 14249289
Ok, after waiting and viewing my log file again it shows the googlebot did come back, but still did the same thing. It looks at the root page and then leaves. Do notice that I have very SEO friendly url's and the site is in MANY SEO directories. What now? Is this just google's problem or mine?
0
 
LVL 24

Expert Comment

by:duz
ID: 14250838
stjowa -

OK try Google Sitemaps then https://www.google.com/webmasters/sitemaps/docs/en/about.html

- duz
0
 

Author Comment

by:stjowa
ID: 14261824
Ok, sorry for the long waits. I now notice that I am not being indexed by yahoo anymore. Yet, I was indexed by yahoo a few weeks ago. I feel like I am doing something wrong with my site, but I don't know what. Any advice?
0
 
LVL 24

Expert Comment

by:duz
ID: 14262994
stjowa -

>Any advice?

Sure follow the advice already given, more inbound links (lots more), have a sitemap and use 'Google Sitemaps'.  You will have to be patient and while you are being patient get even more inbound links.

- duz
0
 

Author Comment

by:stjowa
ID: 14263169
... that's it? ... inbound links, patience, and sitemaps? ... something just seems to be missing ...
0
 
LVL 24

Accepted Solution

by:
duz earned 800 total points
ID: 14263289
stjowa -

>... that's it?

Probably yes.

Take a look at what the msn search spider has cached http://search.msn.com/results.aspx?q=site%3Avehicleaid.com+&FORM=QBRE 1,300+ pages.  If the msn bot can do it so can the others.

You may want to vary <META name="Description" content="Vehicle Aid - Vehicle Problem? Ask... Everyone"> so every page doesn't have the same description.

- duz
0
 

Author Comment

by:stjowa
ID: 14510948
One more question before you close. I now can see google crawling different pages of my site, but it does not show up when I search for relevant keywords on google. How long does it usually take after crawling a site for google to actually show it on it's listing?

Does this involve the google sandbox thing that I heard about?
0
 
LVL 24

Expert Comment

by:duz
ID: 14511001
stjowa -

>How long does it usually take after crawling a site for google to actually show it on it's listing?

The page itself will show up in the index very quickly.  Search for the page itself |http://www.yourdomain.com/yourpage.htm| for example.

>it does not show up when I search for relevant keywords on google

It depends how well the page is optimized for the keywords you are searching on and yes, the sand box still exists for some pages.

- duz
0

Featured Post

[Video] Oticon Case Study

Open office environments can create the dynamics for innovation, but they also bring some challenges. With over 1,000 employees in an open office, Oticon needed a solution that would preserve the environment while mitigating disruptive background noises.

Watch how they did it.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

FAQ pages provide a simple way for you to supply and for customers to find answers to the most common questions about your company. Here are six reasons why your company website should have a FAQ page
The burden of debt that average person carries today has not only increased, but it keeps on growing. It is quite common nowadays that people find it extremely difficult to make ends meet in the face of millions of competing priorities that they hav…
Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
Learn how to set-up custom confirmation messages to users who complete your Wufoo form. Include inputs from fields in your form, webpage redirects, and more with Wufoo’s confirmation options.
Suggested Courses

872 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question