Weird Google Indexing

I am not currently listed on google's search which would obviously help my site tremendously. I've tried adding myself to dmoz and they never will add me. Im really wanting to get indexed and Im not sure why I'm not. I have plenty of inbound links across numerous popular sites including howstuffworks.com.

I did find some interesting stuff in my raw log manager though. The following two lines shows some activity from the googlebot, but it just stops after these two lines. Is it hitting some bad code or something in my site?

crawl-66-249-66-40.googlebot.com - - [03/May/2005:02:48:00 +0100] "GET /robots.txt HTTP/1.1" 404 10010 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
crawl-66-249-66-40.googlebot.com - - [03/May/2005:02:48:00 +0100] "GET / HTTP/1.1" 200 30037 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

By the way my robots.txt includes the following like it is supposed to:
User-agent: *
Disallow:

Question is why does googlebot stop right there and obviously not continue indexing my site?

Thanks
stjowaAsked:
Who is Participating?
 
duzCommented:
stjowa -

>... that's it?

Probably yes.

Take a look at what the msn search spider has cached http://search.msn.com/results.aspx?q=site%3Avehicleaid.com+&FORM=QBRE 1,300+ pages.  If the msn bot can do it so can the others.

You may want to vary <META name="Description" content="Vehicle Aid - Vehicle Problem? Ask... Everyone"> so every page doesn't have the same description.

- duz
0
 
mcse20002000Commented:
robots.txt is not doing any thing useful. but may be stopping the bot to index it

remove the txt file

is it even not showing on most wierd search terms?
0
 
duzCommented:
stjowa -

>why does googlebot stop right there

It can take several visits. If you give us the url we can look for other possibilities.

- duz
0
Cloud Class® Course: Ruby Fundamentals

This course will introduce you to Ruby, as well as teach you about classes, methods, variables, data structures, loops, enumerable methods, and finishing touches.

 
stjowaAuthor Commented:
> remove the txt file

Ok... did that.

 > is it even not showing on most weird search terms?

What do you mean?

 > It can take several visits.

Define several... it already has a few times.

 > If you give us the url we can look for other possibilities.

www.vehicleaid.com -- you might have seen me here before :)
0
 
duzCommented:
stjowa -

>Define several

More than one, sometimes Google visits several times before a page will appear in the index.

I took a look and I have two recommendations. The first is to get many more inbound links.  You can start by submitting to  (say) a 100 free directories http://info.vilesilencer.com/main.php?rock=seo-friendly.php then you can embark on a full blown link building campaign.

Secondly I suggest you have a sitemap with a link from the homepage, this will help Google spider the site. Text links with the full url.

Google will index the cgi-bin and .pl files but I have seen these cause difficulties in the past.

- duz
0
 
stjowaAuthor Commented:
How may the cgi-bin and .pl files cause difficulties and how would you fix it.
0
 
duzCommented:
stjowa -

What forum script are you using or is it your own?

- duz


0
 
stjowaAuthor Commented:
it is my own custom built question and answer script (similar to ee)
0
 
duzCommented:
stjowa -

OK. Well ideally you want your site to have search engine friendly urls so that for example http://www.vehicleaid.com/cgi-bin/openthread.pl?ThreadID=6 was something like this Mechanics http://www.vehicleaid.com/engine/fan-belt-tensioner.html with a title like this <title>Fan Belt Tensioner</title>.  I would try a sitemap sitemap.html with a link from the homepage first and see if once Google indexes the sitemap if it indexes the links on it.

- duz
0
 
stjowaAuthor Commented:
Ok, after waiting and viewing my log file again it shows the googlebot did come back, but still did the same thing. It looks at the root page and then leaves. Do notice that I have very SEO friendly url's and the site is in MANY SEO directories. What now? Is this just google's problem or mine?
0
 
duzCommented:
stjowa -

OK try Google Sitemaps then https://www.google.com/webmasters/sitemaps/docs/en/about.html

- duz
0
 
stjowaAuthor Commented:
Ok, sorry for the long waits. I now notice that I am not being indexed by yahoo anymore. Yet, I was indexed by yahoo a few weeks ago. I feel like I am doing something wrong with my site, but I don't know what. Any advice?
0
 
duzCommented:
stjowa -

>Any advice?

Sure follow the advice already given, more inbound links (lots more), have a sitemap and use 'Google Sitemaps'.  You will have to be patient and while you are being patient get even more inbound links.

- duz
0
 
stjowaAuthor Commented:
... that's it? ... inbound links, patience, and sitemaps? ... something just seems to be missing ...
0
 
stjowaAuthor Commented:
One more question before you close. I now can see google crawling different pages of my site, but it does not show up when I search for relevant keywords on google. How long does it usually take after crawling a site for google to actually show it on it's listing?

Does this involve the google sandbox thing that I heard about?
0
 
duzCommented:
stjowa -

>How long does it usually take after crawling a site for google to actually show it on it's listing?

The page itself will show up in the index very quickly.  Search for the page itself |http://www.yourdomain.com/yourpage.htm| for example.

>it does not show up when I search for relevant keywords on google

It depends how well the page is optimized for the keywords you are searching on and yes, the sand box still exists for some pages.

- duz
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.