I wan to block access to all robots except google, yahoo and msn to all of my website AND want to block one directory /i for ALL robots INCLUDING google, yahoo and msn. I have created following robots.txt. Can someone please confirm if this is correct? Also want robot not to kill my server therefore go as slow as they can.

User-agent: Googlebot
User-agent: MSNBot
User-agent: Slurp
User-agent: *
Disallow: /
Crawl-delay: 120
Disallow: /i/

I am using Apache on some machines and others are using Nginx all using CentOS.
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Dave BaldwinFixer of ProblemsCommented:
Looks good to me.  The web server doesn't matter, 'robots.txt' is just a text file.  Also, 'Crawl-delay' is somewhat non-standard, don't know which bots will recognize it.

You must realize that 'robots.txt' does not actually block anything.  'Good' bots will obey your requests.  'Bad' bots won't even read the file.  Unless they want to see what you don't want them to look at.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sysautomationAuthor Commented:
Thanks. Any advice to block 'Bad' bots who don't consider robots.txt ? I See many bots coming from a very different block of IPs hence doesn't seem possible to block from the firewall.
Dave BaldwinFixer of ProblemsCommented:
No, it's almost impossible to block them  The only reason I block search bots on one directory is to keep it from trying to index every day and time on my calendar page.  Other than that, I don't bother.  The 'good robots' aren't going to access your site very often.  Google is typically once every 3 months unless you become very popular and change content very often.  Bing is similar and Yahoo gets their search results from Bing now so I don't know if they even have search bots anymore.
Dave BaldwinFixer of ProblemsCommented:
For what it's worth, Baidu, the Chinese search engine, hits my site more than anyone else.

Mozilla/5.0 (compatible; Baiduspider/2.0; +
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Apache Web Server

From novice to tech pro — start learning today.