Solved

How much Web traffic is spiders and bots?

Posted on 2007-04-03
3
514 Views
Last Modified: 2013-12-07
I'd like to find some kind of authoritative analysis or study on the average percentage of page views to a Web site that are coming from spiders and bots. For example, if I don't filter out any spiders or bots from my Webtrends report, can I say that XX% of those page views were not valid? I realize the number could vary but would appreciate any educated guesses or additional reading materials, thanks.
0
Comment
Question by:mamadouthiam
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
3 Comments
 
LVL 27

Accepted Solution

by:
mrcoffee365 earned 250 total points
ID: 18845912
We find that about 7% of a medium-traffic site is requests from bots.  That comes to 20% of the bandwidth, presumably because the bots always get all of the large docs from the site.
0
 
LVL 33

Assisted Solution

by:shalomc
shalomc earned 250 total points
ID: 18856556
http://arxiv.org/ftp/cs/papers/0701/0701173.pdf

The summary says: "Our best estimate is that spiders contributed 46% of sessions and 20% of the Web traffic."

I think that the ratios may differ greatly with the type of web site. For example, my web site behaves more like mrcoffee365 mentioned.
0
 

Author Comment

by:mamadouthiam
ID: 18859261
Good info so far but would like some more input - will probably split point (reference to articles, docs appreciated).

mama
0

Featured Post

How Do You Stack Up Against Your Peers?

With today’s modern enterprise so dependent on digital infrastructures, the impact of major incidents has increased dramatically. Grab the report now to gain insight into how your organization ranks against your peers and learn best-in-class strategies to resolve incidents.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Because your company can’t afford for you to make SEO mistakes, you’ll want to ensure you’re taking the right steps each and every time you post a new piece of content. This list of optimization do’s and don’ts can help you become an SEO wizard.
When crafting your “Why Us” page, there are a plethora of pitfalls to avoid. Follow these five tips, and you’ll be well on your way to creating an effective page.
Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question