Solved

Checking a website hub for endless spider link recursion

Posted on 2006-07-13
3
217 Views
Last Modified: 2010-04-20
Is there an application to test a website and/or a directory/subdirectory site on a hard drive to point out the possiblities of endless loop/recursion/crawling of a search engine or spider which could totally decimate available bandwidth/usage limitations?
0
Comment
Question by:ejoeeb
3 Comments
 
LVL 18

Accepted Solution

by:
Sam Panwar earned 125 total points
ID: 17105523
Hi,

Please download GFI Lan guard It is ver good tool which is may be fullfil your requirement

Automatically detect security vulnerabilities on your network
Provides in-depth information about all machines/devices

http://www.gfi.com/downloads/downloads.aspx?pid=LANSS&lid=en
0
 

Assisted Solution

by:rubanek
rubanek earned 125 total points
ID: 17107699
Without knowing the code that controls the visiting crawler you'll never know if it's stuck in a loop.  All you can really do is monitor your sites activity and if you see a bot hanging out for longer than normal times, you should block thier IP.

I would think that most crawlers like google or yahoo would be smart enough to not get stuck in a loop.
0

Featured Post

U.S. Department of Agriculture and Acronis Access

With the new era of mobile computing, smartphones and tablets, wireless communications and cloud services, the USDA sought to take advantage of a mobilized workforce and the blurring lines between personal and corporate computing resources.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Article by: kevp75
Hey folks, 'bout time for me to come around with a little tip. Thanks to IIS 7.5 Extensions and Microsoft (well... really Windows 8, and IIS 8 I guess...), we can now prime our Application Pools, when IIS starts. Now, though it would be nice t…
One of the typical problems I have experienced is when you have to move a web server from one hosting site to another. You normally prepare all on the new host, transfer the site, change DNS and cross your fingers hoping all will be ok on new server…
This tutorial gives a high-level tour of the interface of Marketo (a marketing automation tool to help businesses track and engage prospective customers and drive them to purchase). You will see the main areas including Marketing Activities, Design …
With the power of JIRA, there's an unlimited number of ways you can customize it, use it and benefit from it. With that in mind, there's bound to be things that I wasn't able to cover in this course. With this summary we'll look at some places to go…

861 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now