Checking a website hub for endless spider link recursion

Is there an application to test a website and/or a directory/subdirectory site on a hard drive to point out the possiblities of endless loop/recursion/crawling of a search engine or spider which could totally decimate available bandwidth/usage limitations?
ejoeebAsked:
Who is Participating?
 
Sam PanwarConnect With a Mentor Sr. Server AdministratorCommented:
Hi,

Please download GFI Lan guard It is ver good tool which is may be fullfil your requirement

Automatically detect security vulnerabilities on your network
Provides in-depth information about all machines/devices

http://www.gfi.com/downloads/downloads.aspx?pid=LANSS&lid=en
0
 
rubanekConnect With a Mentor Commented:
Without knowing the code that controls the visiting crawler you'll never know if it's stuck in a loop.  All you can really do is monitor your sites activity and if you see a bot hanging out for longer than normal times, you should block thier IP.

I would think that most crawlers like google or yahoo would be smart enough to not get stuck in a loop.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.