• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 75
  • Last Modified:

Bots - Block

I do web hosting and have a few servers some running Apache and others Nginx.

This is a specialised kind of web hosting and all the websites I host have a directory under their public_html (document root) which should not be available to any bots.

I can create robots.txt in every document root directory but that will be too much work for me as there are many websites hosted.

Is there a way I can stop bots to a specific directory for all websites from apache/nginx configuration?
0
sysautomation
Asked:
sysautomation
  • 3
  • 2
1 Solution
 
David Johnson, CD, MVPOwnerCommented:
not without a robots.txt that tells them what they can and cannot index
0
 
sysautomationAuthor Commented:
Can one robots.txt work for all virtual hosts created in Apache?
0
 
David Johnson, CD, MVPOwnerCommented:
no it has to be in the root directory of the site
0
 
sysautomationAuthor Commented:
Thanks. One more help. I am not good at robots.txt. Following is what I have. Can you please confirm if this is fine?

User-agent: MJ12bot
Disallow: /
User-agent: Googlebot
Disallow: /
User-agent: googlebot-image
Disallow: /
User-agent: *
Disallow: /
Disallow: /i

Open in new window


Is it must to have Sitemap clause and specify some site?
0
 
David Johnson, CD, MVPOwnerCommented:
line 7 and 8 disallow all robots any access
a good tutorial is at http://www.robotstxt.org/robotstxt.html
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 3
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now