I want to have a php script that I will put in my robots.txt file to let me know when a search engine spider accesses the robots.txt file.
I would like the script to email me the referrer, the agent (for example, Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp
)), the ip address and time and date that it accessed my robots.txt file.
I would also like to be able to have it email me the domain name where the robots.txt file resides. If this can be done automatically that would be great, otherwise I would like to be able to input that information as a variable near the top of the file. I would like the domain name to be in the subject line of the email that is sent to me, such as a subject line of "Spider Access on www.mysite.com"
I usually have some information in my robots.txt file, such as
I would still like to be able to instruct the spider not to access certain files.