thunderchicken
asked on
Robots.txt?
I was going through my IIS5 log and found Robots.txt file was being accessed. Upon further research, which so far has come up inconclusive, has shows that Spiders use this file.
I have been unable to far to find any resources to find how to create this file, or if it is necessary to get on search engines. I do however so far use META tags, but this has gotten me curious.
Does anyone have any more information on using this and creating this efficiently?
I have been unable to far to find any resources to find how to create this file, or if it is necessary to get on search engines. I do however so far use META tags, but this has gotten me curious.
Does anyone have any more information on using this and creating this efficiently?
Robots.txt is NOT used by spiders, it's used by robots. But it's the same sort of thing. It tells the search engine which files/folders it can or cannot visit to perform its indexing. Just run a search inside google for robots.txt, and you'll come up with something useful.
ASKER
I did that and have come up with nothing.
Thanks MCM. I was just about to look for something like that myself. saved me doing it.
thunderchicken -- if you like my link, you may as well give the points to AlfaNoMore. robots.txt comes up all over google.
ASKER
I searched google and found that link prior to posting this question, looking a little more, didn't know if someone could give me some personal experience information instead of a web site.
Though the one you posted, I did not investigate to the fullest.
Though the one you posted, I did not investigate to the fullest.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Here are a few tips:
1 - Do *not* rely on robots.txt for security! Spider developers do not have to use robots.txt, and there is no way to make them.
2 - Therefore, implement your own security where necessary, i.e., use .htaccess files with Apache and NT permissions with IIS.
3 - Make sure robots.txt is in your web root directory.
1 - Do *not* rely on robots.txt for security! Spider developers do not have to use robots.txt, and there is no way to make them.
2 - Therefore, implement your own security where necessary, i.e., use .htaccess files with Apache and NT permissions with IIS.
3 - Make sure robots.txt is in your web root directory.
Good points drittich.
Thanks for the points and the A thunderchicken :)