asked on

mod_rewrite and googlebot

I am planning to use mod_rewrite to hide ugly urls. e.g.
http://.../places/coffee-house would point to http://.../index.php?pid=43

I have got this to work. My question is how will the GoogleBot know to index the mod_rewrite urls (i.e. http://.../places/coffee-house) since technically they dont exist.

More detail - I plan to add unique meta tags (keywords, description) to the page depending on the pid passed to index.php. So each such mod_rewritten url has unique content. How can I get Googlebot to discover such Urls? Should I create empty directories mirroring the url path (e.g. mkdir www/places/coffee-house)?

Thanks,
Unmesh.

pigmentarts

Google just follows links, if it treats them as normal URLs or redirected URLs depends entirely on how the rules were written. Google just follows URLs just like your browser, if the undesired results are visible in your browser address bar, then they are to Google! (with exceptions). Write the rules server side, local paths and no redirect flag you should be fine.

on another note,

I have read that Google can make educated guesses, for example if file sizes were different, if it does this or what it does with this information I have no idea.

> How can I get Googlebot to discover such Urls?
submit them all in a XML sitemap.

unmeshm

ASKER

The content is dynamically generated and grows over time hence adding the URL's to a sitemap XML is unfeasable.

I guess what you are saying is that I need to expose the URLs on my site somehow or have them linked from some other sites. Will have to figure out a way to have all the URL's tracked by the robot (both old and new).

ASKER CERTIFIED SOLUTION

humeniuk

membership

Create a free account to see this answer

Signing up is free and takes 30 seconds. No credit card required.

See answer

SOLUTION

pigmentarts

membership

Create a free account to see this answer

Signing up is free and takes 30 seconds. No credit card required.

See answer

humeniuk

"... if the pages are only accessible to Google in a submitted XML map with no link on your site, Google still may not index it."

Very good point.

pigmentarts

thank you for the points unmeshm. :)