Solved

Checking a website hub for endless spider link recursion

Posted on 2006-07-13
3
219 Views
Last Modified: 2010-04-20
Is there an application to test a website and/or a directory/subdirectory site on a hard drive to point out the possiblities of endless loop/recursion/crawling of a search engine or spider which could totally decimate available bandwidth/usage limitations?
0
Comment
Question by:ejoeeb
3 Comments
 
LVL 18

Accepted Solution

by:
Sam Panwar earned 125 total points
ID: 17105523
Hi,

Please download GFI Lan guard It is ver good tool which is may be fullfil your requirement

Automatically detect security vulnerabilities on your network
Provides in-depth information about all machines/devices

http://www.gfi.com/downloads/downloads.aspx?pid=LANSS&lid=en
0
 

Assisted Solution

by:rubanek
rubanek earned 125 total points
ID: 17107699
Without knowing the code that controls the visiting crawler you'll never know if it's stuck in a loop.  All you can really do is monitor your sites activity and if you see a bot hanging out for longer than normal times, you should block thier IP.

I would think that most crawlers like google or yahoo would be smart enough to not get stuck in a loop.
0

Featured Post

Gigs: Get Your Project Delivered by an Expert

Select from freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Are date-time stamps in Microsoft IIS log files in GMT +0? 3 58
Script 12 152
Web server settings related to keepalive 1 120
cookies analysis tools 2 91
Have you ever sent email via ColdFusion and thought of tracking this mail to capture the exact date and time when the message was opened ?  If yes, then this article is for you ! First we need a table user_email with columns user_id , email , sub…
Introduction This article explores the design of a cache system that can improve the performance of a web site or web application.  The assumption is that the web site has many more “read” operations than “write” operations (this is commonly the ca…
This Micro Tutorial will teach you how to censor certain areas of your screen. The example in this video will show a little boy's face being blurred. This will be demonstrated using Adobe Premiere Pro CS6.
Email security requires an ever evolving service that stays up to date with counter-evolving threats. The Email Laundry perform Research and Development to ensure their email security service evolves faster than cyber criminals. We apply our Threat…

815 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now