Solved

should web bots / crawlers (like baidu) be blocked?

Posted on 2015-02-19
3
168 Views
Last Modified: 2015-02-19
Hello Web experts,

Should web bots / crawlers (like baidu) be blocked?

Thanks for your help.
0
Comment
Question by:epifanio67
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 32

Accepted Solution

by:
_ earned 500 total points
ID: 40619398
Depends on whether you want your content Public, how easily you want it found, or if you want it Private.
Do you have a specific scenario in mind?

This has a blurb about the top 10  good bots, and you can decide if you want them on or not:
http://www.incapsula.com/blog/know-your-top-10-bots.html
0
 

Author Closing Comment

by:epifanio67
ID: 40619432
Thank you coral47....

good article.... it helps...

Regards
0
 
LVL 32

Expert Comment

by:_
ID: 40619541
Thank you much.    : )
0

Featured Post

Why Off-Site Backups Are The Only Way To Go

You are probably backing up your data—but how and where? Ransomware is on the rise and there are variants that specifically target backups. Read on to discover why off-site is the way to go.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Most of the applications these days are on Cloud. Cloud is ubiquitous with many service providers in the market. Since it has many benefits such as cost reduction, software updates, remote access, disaster recovery and much more.
During and after that shift to cloud, one area that still poses a struggle for many organizations is what to do with their department file shares.
Monitoring a network: why having a policy is the best policy? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the enormous benefits of having a policy-based approach when monitoring medium and large networks. Software utilized in this v…
Michael from AdRem Software explains how to view the most utilized and worst performing nodes in your network, by accessing the Top Charts view in NetCrunch network monitor (https://www.adremsoft.com/). Top Charts is a view in which you can set seve…
Suggested Courses

630 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question