Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

How much Web traffic is spiders and bots?

Posted on 2007-04-03
3
Medium Priority
?
523 Views
Last Modified: 2013-12-07
I'd like to find some kind of authoritative analysis or study on the average percentage of page views to a Web site that are coming from spiders and bots. For example, if I don't filter out any spiders or bots from my Webtrends report, can I say that XX% of those page views were not valid? I realize the number could vary but would appreciate any educated guesses or additional reading materials, thanks.
0
Comment
Question by:mamadouthiam
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
3 Comments
 
LVL 27

Accepted Solution

by:
mrcoffee365 earned 1000 total points
ID: 18845912
We find that about 7% of a medium-traffic site is requests from bots.  That comes to 20% of the bandwidth, presumably because the bots always get all of the large docs from the site.
0
 
LVL 33

Assisted Solution

by:shalomc
shalomc earned 1000 total points
ID: 18856556
http://arxiv.org/ftp/cs/papers/0701/0701173.pdf

The summary says: "Our best estimate is that spiders contributed 46% of sessions and 20% of the Web traffic."

I think that the ratios may differ greatly with the type of web site. For example, my web site behaves more like mrcoffee365 mentioned.
0
 

Author Comment

by:mamadouthiam
ID: 18859261
Good info so far but would like some more input - will probably split point (reference to articles, docs appreciated).

mama
0

Featured Post

Command Line Tips and Tricks

The command line is a powerful tool at the disposal of every Linux user. Although Linux distros come with beautiful user interfaces, it's worthwhile to learn the command line because it allows you to do a number of things that you otherwise cannot do from the GUI.  

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When the s#!t hits the fan, you don’t have time to look up who’s on call, draft emails, call collaborators, or send text messages. An instant chat window is definitely the way to go, especially one like HipChat. HipChat is a true business app. An…
Ready to get certified? Check out some courses that help you prepare for third-party exams.
The viewer will learn how to count occurrences of each item in an array.
Any person in technology especially those working for big companies should at least know about the basics of web accessibility. Believe it or not there are even laws in place that require businesses to provide such means for the disabled and aging p…
Suggested Courses

671 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question