troubleshooting Question

how to stop spider software from getting my website content

Avatar of rafaelrgl
rafaelrgl asked on
.NET ProgrammingC#ASP.NET
10 Comments2 Solutions867 ViewsLast Modified:
Ok guys, here is the problem, I know that exist out there a lots of spider programs that go on your website and open all pages and get your content, or maybe an hacker that wants to get your content he can use this kind of program to get it with no problem.

So how can we stop this thing. I know when those programs go to get the website they open one page after another, maybe have an function that checks the rate the user opens pages, lets say 1 pages / 5 second is an spider, so redirect the user to an page that he have to type an authentication using some image? The point is, what is the best way to stop this without losing also performance and letting others crawlers like google pass those authentication so we can have our website indexed and also block unwanted crawlers.

What you guys say?

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Join our community to see this answer!
Unlock 2 Answers and 10 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 2 Answers and 10 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros