?
Solved

Checking a website hub for endless spider link recursion

Posted on 2006-07-13
3
Medium Priority
?
226 Views
Last Modified: 2010-04-20
Is there an application to test a website and/or a directory/subdirectory site on a hard drive to point out the possiblities of endless loop/recursion/crawling of a search engine or spider which could totally decimate available bandwidth/usage limitations?
0
Comment
Question by:ejoeeb
2 Comments
 
LVL 18

Accepted Solution

by:
Sam Panwar earned 375 total points
ID: 17105523
Hi,

Please download GFI Lan guard It is ver good tool which is may be fullfil your requirement

Automatically detect security vulnerabilities on your network
Provides in-depth information about all machines/devices

http://www.gfi.com/downloads/downloads.aspx?pid=LANSS&lid=en
0
 

Assisted Solution

by:rubanek
rubanek earned 375 total points
ID: 17107699
Without knowing the code that controls the visiting crawler you'll never know if it's stuck in a loop.  All you can really do is monitor your sites activity and if you see a bot hanging out for longer than normal times, you should block thier IP.

I would think that most crawlers like google or yahoo would be smart enough to not get stuck in a loop.
0

Featured Post

How to Use the Help Bell

Need to boost the visibility of your question for solutions? Use the Experts Exchange Help Bell to confirm priority levels and contact subject-matter experts for question attention.  Check out this how-to article for more information.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In our day to day coding, how many times have we come across a necessity to check whether a URL is a broken link or not? For those of you that answered countless and are using ColdFusion like myself, then this article is for you.  It will show yo…
One of the typical problems I have experienced is when you have to move a web server from one hosting site to another. You normally prepare all on the new host, transfer the site, change DNS and cross your fingers hoping all will be ok on new server…
In a question here at Experts Exchange (https://www.experts-exchange.com/questions/29062564/Adobe-acrobat-reader-DC.html), a member asked how to create a signature in Adobe Acrobat Reader DC (the free Reader product, not the paid, full Acrobat produ…
Despite its rising prevalence in the business world, "the cloud" is still misunderstood. Some companies still believe common misconceptions about lack of security in cloud solutions and many misuses of cloud storage options still occur every day. …
Suggested Courses

831 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question