Solved

How much Web traffic is spiders and bots?

Posted on 2007-04-03
3
505 Views
Last Modified: 2013-12-07
I'd like to find some kind of authoritative analysis or study on the average percentage of page views to a Web site that are coming from spiders and bots. For example, if I don't filter out any spiders or bots from my Webtrends report, can I say that XX% of those page views were not valid? I realize the number could vary but would appreciate any educated guesses or additional reading materials, thanks.
0
Comment
Question by:mamadouthiam
3 Comments
 
LVL 27

Accepted Solution

by:
mrcoffee365 earned 250 total points
ID: 18845912
We find that about 7% of a medium-traffic site is requests from bots.  That comes to 20% of the bandwidth, presumably because the bots always get all of the large docs from the site.
0
 
LVL 33

Assisted Solution

by:shalomc
shalomc earned 250 total points
ID: 18856556
http://arxiv.org/ftp/cs/papers/0701/0701173.pdf

The summary says: "Our best estimate is that spiders contributed 46% of sessions and 20% of the Web traffic."

I think that the ratios may differ greatly with the type of web site. For example, my web site behaves more like mrcoffee365 mentioned.
0
 

Author Comment

by:mamadouthiam
ID: 18859261
Good info so far but would like some more input - will probably split point (reference to articles, docs appreciated).

mama
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Popularity Can Be Measured Sometimes we deal with questions of popularity, and we need a way to collect opinions from our clients.  This article shows a simple teaching example of how we might elect a favorite color by letting our clients vote for …
Learn by example how to specify CSS selectors for Selenium WebDriver test automation software.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.

896 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now