Solved

How much Web traffic is spiders and bots?

Posted on 2007-04-03
3
513 Views
Last Modified: 2013-12-07
I'd like to find some kind of authoritative analysis or study on the average percentage of page views to a Web site that are coming from spiders and bots. For example, if I don't filter out any spiders or bots from my Webtrends report, can I say that XX% of those page views were not valid? I realize the number could vary but would appreciate any educated guesses or additional reading materials, thanks.
0
Comment
Question by:mamadouthiam
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
3 Comments
 
LVL 27

Accepted Solution

by:
mrcoffee365 earned 250 total points
ID: 18845912
We find that about 7% of a medium-traffic site is requests from bots.  That comes to 20% of the bandwidth, presumably because the bots always get all of the large docs from the site.
0
 
LVL 33

Assisted Solution

by:shalomc
shalomc earned 250 total points
ID: 18856556
http://arxiv.org/ftp/cs/papers/0701/0701173.pdf

The summary says: "Our best estimate is that spiders contributed 46% of sessions and 20% of the Web traffic."

I think that the ratios may differ greatly with the type of web site. For example, my web site behaves more like mrcoffee365 mentioned.
0
 

Author Comment

by:mamadouthiam
ID: 18859261
Good info so far but would like some more input - will probably split point (reference to articles, docs appreciated).

mama
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
hbo knew my windows software 4 76
Website Question - New Site 3 31
compare date to getdate() 8 15
JS to redirect to prev page 8 18
When it comes to write a Context Sensitive Help (an online help that is obtained from a specific point in state of software to provide help with that state) ,  first we need to make the file that contains all topics, which are given exclusive IDs. …
Developer portfolios can be a bit of an enigma—how do you present yourself to employers without burying them in lines of code?  A modern portfolio is more than just work samples, it’s also a statement of how you work.
Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.

761 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question