Link to home
Start Free TrialLog in
Avatar of tilmes
tilmesFlag for Germany

asked on

make google friendly

Hi

The cgi scripts/dynamic sites are not visited from google.
What is the best way to make scan from google search robot?
Avatar of jmcg
jmcg
Flag of United States of America image

You may be able to entice Google into scanning your site by placing an appropriate entry in your robots.txt file.

Take a look at Expert-Exchange's robots.txt file, for example:

User-agent: *
Disallow:

There are also "Allow:" directives and you could list your .cgi script there.

The main hesitation behind spidering dynamic sites is that they may never have the same thing twice. That leaves doubt about whether a search engine should keep track of what the page once said, but which it won't say again if you visit it.



Avatar of ext2
ext2

In Apache, simply use Action/AddHandler in your .htaccess file to translate external *.html URLs seen by users and search engines to internal *.cgi paths on the server:

  http://httpd.apache.org/docs/handler.html

Your users might never even know that your page are implemented internally with CGIs.

I've often wondered why ASP/JSP/Perl/etc. traditionally add their own extensions to URLs.  This needlessly exposes implementation, and it makes it difficult to change the implementation (Parnas would not be pleased).  <troll>It's poor design, and apparently MS is slow to understand this.</troll>
Avatar of tilmes

ASKER

Hi

How can i use
simply use Action/AddHandler in your .htaccess?
Please make me understand this.
Avatar of tilmes

ASKER

Hi

I read about the article said that change query part of the dynamic URL
Example - http://www.my-online-store.com/books.asp?id=1190
to http://www.my-online-store.com/books/A 
HOw can i do this in my cgi script?
ASKER CERTIFIED SOLUTION
Avatar of ext2
ext2

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of tilmes

ASKER

thank you for the explanation.
I inserted below two in .htaccess file
but in the query string has not changed at all.
Do i need to change also in httpd.conf file in parent directory?

Options +FollowSymLinks
RewriteEngine on
RewriteBase /
RewriteRule cla/(\.*)/(\.*)/(\.*)/(\.*)/(\.*)/(\.*)/(\.*)/(\.*)/$ /tell/cgi-bin/ADcla/cla\.cgi?$1=$2&$3=$4&$5=$6&$7=$8


Action my-handler /tell/cgi-bin/ADcla/cla.cgi
AddHandler my-handler .html
Nothing has happened on this question in more than 7 weeks. It's time for cleanup!

My recommendation, which I will post in the Cleanup topic area, is to
accept answer by ext2 [grade B] (on the road to an answer).

PLEASE DO NOT ACCEPT THIS COMMENT AS AN ANSWER!

jmcg
EE Cleanup Volunteer