#!/usr/bin/perl
$| = 1;
$host = $ENV{'REMOTE_HOST'};
$addr = $ENV{'REMOTE_ADDR'};
$agent = $ENV{'HTTP_USER_AGENT'};
print "Content-type: text/plain\n\n";
if ($host =~ /\.msn\.com$/i && $agent =~ /^msnbot/) {
print <<'EOF';
User-agent: msnbot
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /private/
Disallow: /css/
Disallow: /taxonomy/term/
Allow: /stories/
Request-rate: 1/1h
Visit-time: 0600-0900
EOF
} else {
print <<'EOF';
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /private/
Disallow: /css/
Disallow: /taxonomy/term/
Allow: /stories/
Request-rate: 2/1h
Visit-time: 0600-0900
EOF
}
RewriteEngine On
RewriteRule /robots\.txt$ /var/www/cgi-bin/robots.pl [L,T=application/x-httpd-cgi]
Have a question about something in this article? You can receive help directly from the article author. Sign up for a free trial to get started.
Comments (4)
Author
Commented:Since explaining things in a simple manner isn't my forte, I admit it requires a bit of technical knowledge to use the script/code.
-- Michael
PS: I also found a bug in the article editor -- I cannot post 'Options +ExecCGI' (with the plus symbol) as the preview & editor remove it.
Commented:
Author
Commented:Commented: