Solved

search engine web site indexing?

Posted on 2011-03-09
9
303 Views
Last Modified: 2012-05-11
Hello
I have put my website live yesterday, but neither google nor yahoo is showing it when I searched today. Is there any more steps to do for the website to be added to the search engines, or does it take time to add the website to the db of these search engines?
0
Comment
Question by:lordrt
  • 4
  • 3
  • 2
9 Comments
 
LVL 10

Expert Comment

by:gavsmith
ID: 35081241
It takes time for their bots to crawl your site. Although you may be able to speed things up a bit by logging into google and looking through the tools... they have a tool which allows you to track your site, they give you a txt file to put in your webspace to prove it's yours. I think it even lets you know if there are broken links.
0
 
LVL 10

Accepted Solution

by:
gavsmith earned 250 total points
ID: 35081264
Found it www.google.com/webmasters/tools/

Regards
Gav
0
 
LVL 29

Assisted Solution

by:fibo
fibo earned 250 total points
ID: 35081434
Well, being indexed overnight is not the rule.
You need to make a distinction between Google and others (Yahoo, Bing etc).

1. Bing & Yahoo

For these engines, the simplest action is to submit the url of your site to their engine.
Bing: http://www.bing.com/webmaster/SubmitSitePage.aspx
Yahoo: http://siteexplorer.search.yahoo.com/submit (will require that you register - anyway you would do to access to their webmasters tools)
With these sites you can also register to get access webmasters tools and submit the urls or sitemaps.

2. Google

For Google: you can register to get access to webmasters tools, and even submit sites or sitemaps, and even get analytics.
But that is not enough to get your site indexed by Google!
Google will not accept to index a site or a url if it is not part of the "reachable web universe". More precisely Goggle will not index a page/url if none of the pages that Google knows points to this url.
So you must take care that there is a link to your site from a page (or a site) that Google knows. And once this is done Google spiders will explore the rest of your site.

One easy step if you have some corporate or personal page that Google knows: place there a link to the new site, then resubmit the url of this page to Google... GG will explore it, take note of the new link, and spider the new link (initially just the linked-to page, later the links on that page).
You can reach a similar result by placing a link to the new site as a signature to posts you make on forums known by Google, then resubmit the url of one or 2 pages where this signature is visible.

3. Do not overflood!

Do not oversubmit!
If you make too many submits to any of the engines (ie: submitting the same page repeatedly, submitting lots of pages from the same site), chances are high that this engine will consider your submissions as spam and place them low in its priority list.
Just find the right balance between informing and spamming!
Personally, for a new site I usually create a sitemap (GSiteCrawler is the tool I use), then upload it to GYB (Google Yahoo Bing), then update my own page with the new link, and resubmit it to GYB, then the next day submit the site url to GYB. That is enough, all spiders will start indexing pages within days, and will complete the spîdering of the sites within a few weeks.
B-) except that in some occasions Bing seems to refuse to index the site!
0
 
LVL 4

Author Closing Comment

by:lordrt
ID: 35081537
thanks for the suggestions, hopefully will get my website to show on these search engines quickly enough
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 29

Expert Comment

by:fibo
ID: 35085486
B-) Glad we could help. Thx for the grade and points.

While spiders start exploring your site, help them to find faster the content of your site by
1 - Generating a sitemap, and uploading it to your site
2 - Using webmasters tools of each of GYB, give them the link to the sitemap file on your site
3 - Edit your robots.txt file on site by adding the line
Sitemap: http://www.example.com/sitemap.xml
Note that the sitemap file must be a complete url with http
In addition to GYB, most other spiders will use this information to speed up theit spidering of your site, see
http://www.sitemaps.org/protocol.php#informing

As I said, I love GSiteCrawler to generate sitemaps http://gsitecrawler.com/ and somehow automate the process of updtaing them on the site. (This program is free, although donations are possible and welcome - I have no relation with them)
0
 
LVL 4

Author Comment

by:lordrt
ID: 35091612
thanks again fibo, my site is dev in drupal, and it does provide a module which automatically creates a sitemap for the website when it is activated in it
0
 
LVL 4

Author Comment

by:lordrt
ID: 35091843
Just a little problem thou, the www part is missing when I lookup my site in the search engines, it only gives me, e.g., example.com, while my site has the www.example.com format... anything I might have missed?
0
 
LVL 29

Expert Comment

by:fibo
ID: 35092271
yes.
seems your site answers to both www and non www

now is the time to decide which of the 2 you want, then
1 - check that the link to your home page is to the selected option (you might need to make one change in your site parameters and one chage in the html code of your template

2 - put a rewrite 301 in your htaccess: this will tell all search engines to use only the right address
RewriteEngine On

RewriteCond %{HTTP_HOST} ^mydomain.com [NC] 
RewriteRule ^(.*)$ http://www.mydomain.com/$1 [L,R=301]

Open in new window

within weeks, your non-www indexed pages will disappear and appear only as www

3 - If in some engine there still remains some non-www after some weeks... I would ignore them, because the rewrite will do the job.
You might however check in Google what is still there after 8-12 weeks. If some page remains, just ask Google to remove it

4 - Additional actions
They are not really needed nor useful for your problem, but they may speed up things and/or give you the warm feeling you are acting:
- register with Google webmasters tools, and submit both www and non www URLs of the site, then tell webmasters tools wich one is the preferred (this would not work with others spiders, while the redirect 301 will)
- generate a sitemap of the www site as defined earlier (then upload it, change your robots.txt etc)
0
 
LVL 4

Author Comment

by:lordrt
ID: 35093259
thanks again fido
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

Read about how to approach blogging and about ways to do it right. Stand out from the crowd and let your knowledge be consumed by a large audience. This article aims to explain how your blog should look like,  the most important things to do while b…
Read about why website design really matters in today's demanding market.
The viewer will learn how to dynamically set the form action using jQuery.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now