Posted on 2007-03-19
I need a list of the most popular spider user-agent strings. I've got several items on my website that log or increment things and I really only want some of those to be logging if the thing hitting the page is a real person and not a bot. So I'm left checking the user-agents.
I can either grab the most popular browsers or the most popular bots... Whatever is most efficient -- you decide!
Either way, time-complexity is an issue as it is a fairly busy site, so the shortest and most effective list wins =)