Wildcard Disallow for robots.txt file ?
I have a number of pages of a site that have a paramater that I would like, when a robot finds it, will ignore it.
Example:
/bananas/frog/apple/foobar
/mary/jack/jill/foobar.raw
/tony/orlando/dawn/foobar.
Can I make one disallow to cover anything that ends in /foobar.raw ??
Example,
Specifically, can I put the following in the robots.txt file
disallow /foobar
....and then it would take care of all instances where /foobar.raw is the last item in a url?
Thanks
Rowby
Example:
/bananas/frog/apple/foobar
/mary/jack/jill/foobar.raw
/tony/orlando/dawn/foobar.
Can I make one disallow to cover anything that ends in /foobar.raw ??
Example,
Specifically, can I put the following in the robots.txt file
disallow /foobar
....and then it would take care of all instances where /foobar.raw is the last item in a url?
Thanks
Rowby
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks for that info and observation re behavior of spiders.
Sorry for the delay in awarding you your points.
I know now what to do!
Rowby