Solved

User-agent strings

Posted on 2007-03-19
8
7,113 Views
Last Modified: 2013-12-09
I need a list of the most popular spider user-agent strings. I've got several items on my website that log or increment things and I really only want some of those to be logging if the thing hitting the page is a real person and not a bot. So I'm left checking the user-agents.

I can either grab the most popular browsers or the most popular bots... Whatever is most efficient -- you decide!

Either way, time-complexity is an issue as it is a fairly busy site, so the shortest and most effective list wins =)
0
Comment
Question by:OliWarner
8 Comments
 
LVL 2

Expert Comment

by:fpintos
ID: 18753824
Have you tried setting up robot.txt to block these spiders? This is by far the simplest way.
0
 
LVL 16

Author Comment

by:OliWarner
ID: 18753862
I don't want to block them from viewing the pages -- just stop my logging script counting hits from them.
0
 
LVL 29

Expert Comment

by:rdivilbiss
ID: 18753977
I've got them in a db, Oli.  Give me a minute to extract the bots from the browsers.
0
 
LVL 29

Accepted Solution

by:
rdivilbiss earned 500 total points
ID: 18754003
http://www.rodsdot.com/downloads/bots.zip

"ADSAComponent (postmaster@cnds.ucd.ie)","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [fc3]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [c01]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [wf224]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [wf55]","CrawlerBot"
"Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)","CrawlerBot"
"Mozilla/4.0 (Search Engine Marketing Tactics Amsterdam 2002 Information Spider)","CrawlerBot"
"AnswerBus (http://www.answerbus.com/)","CrawlerBot"
"antibot-V1.1.11/i586-linux-2.2","CrawlerBot"
"antibot-V1.1.13/i586-linux-2.2","CrawlerBot"
"antibot-V1.2.0/redhat-linux-9","CrawlerBot"
"AOLserver-Tcl/3.5.6","CrawlerBot"
"AOL 8.0 (compatible; AOL 8.0; DOS; .NET CLR 1.1.4322)","CrawlerBot"
"appie 1.1 (www.walhello.com)","CrawlerBot"
"Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)","CrawlerBot"
"Art-Online.com 0.9(Beta)","CrawlerBot"
"Mozilla/2.0 (compatible; Ask Jeeves)","CrawlerBot"
"Mozilla/2.0 (compatible; Ask Jeeves/Teoma)","CrawlerBot"
"ASPseek/1.2.10","CrawlerBot"
"ASPseek/1.2.11","CrawlerBot"
"ASPseek/1.2.12","CrawlerBot"
"Mozilla/3.0 (compatible; AvantGo 3.2)","CrawlerBot"
"BaiDuSpider","CrawlerBot"
"Baiduspider+(+http://www.baidu.com/search/spider.htm)","CrawlerBot"
"battlebot","CrawlerBot"
"BDFetch","CrawlerBot"
"BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html)","CrawlerBot"
"BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"Big Brother (http://pauillac.inria.fr/~fpottier/)","CrawlerBot"
"BlogBot/1.2","CrawlerBot"
"boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-dc/0.5 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-dc/0.66 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-robot/1.0","CrawlerBot"
"boitho.com-robot/1.1","CrawlerBot"
"Mozilla/4.0 (compatible; BorderManager 3.0)","CrawlerBot"
"BrailleBot 1.0","CrawlerBot"
"BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html)","CrawlerBot"
"bumblebee/1.0 (bumblebee@relevare.com; http://www.relevare.com/)","CrawlerBot"
"Computer_and_Automation_Research_Institute_Crawler (nospamspidernospam@spider.ilab.sztakinospam.hunospam)","CrawlerBot"
"Computer_and_Automation_Research_Institute_Crawler (spider@spider.ilab.sztaki.hu)","CrawlerBot"
"cd34/0.1","CrawlerBot"
"CerberianDrtrs/Version-3.0-Release-24","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-40)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-11)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-12)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-13)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-16)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-17)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-41)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-43)","CrawlerBot"
"CipinetBot (http://www.cipinet.com/bot.html)","CrawlerBot"
"Clushbot/2.1 (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.21-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.23-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.24-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.6-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.9-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"ComMOOnity LambdaMOO/1.8.1","CrawlerBot"
"CrawlConvera0.1 (CrawlConvera@yahoo.com)","CrawlerBot"
"CrawlConvera0.1 (www.authoritativeweb.com)","CrawlerBot"
"ConveraCrawler/0.2","CrawlerBot"
"ConveraCrawler/0.5 (+http://www","CrawlerBot"
"cosmos/0.9_(robot@xyleme.com)","CrawlerBot"
"Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"Cowbot-0.1.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"Crawl_Application","CrawlerBot"
"CrocCrawler v3.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"CrocCrawler v4.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"Custo 2.0 (www.netwu.com)","CrawlerBot"
"CydralSpider/1.9 (Cydral Web Image Search; http://www.cydral.com)","CrawlerBot"
"DeepIndex (http://www.deepindex.com)","CrawlerBot"
"DeMozulator 1.0 (MacOS, dMoz URL Check Agent, trebor@animeigo.com)","CrawlerBot"
"DoCoMo/1.0/N504i/c10/TB","CrawlerBot"
"DoCoMo/1.0/P504iS/c10/TB","CrawlerBot"
"Dual Proxy","CrawlerBot"
"Dumbot(version 0.1 beta - dumbfind.com)","CrawlerBot"
"Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html)","CrawlerBot"
"Dumbot(version 0.1 beta)","CrawlerBot"
"EARTHCOM.info/1.2","CrawlerBot"
"EmailSiphon","CrawlerBot"
"Enterprise_Search/1.00.136;MSSQL (http://www.innerprise.net/es-spider.asp)","CrawlerBot"
"e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)","CrawlerBot"
"exactseek-crawler-2.63 (crawler@exactseek.com)","CrawlerBot"
"exactseek-crawler-2.63 crawler@exactseek.com","CrawlerBot"
"exactseek-crawler-2.63-5 (crawler@exactseek.com)","CrawlerBot"
"exactseek-crawler-2.63-5 crawler@exactseek.com","CrawlerBot"
"Explorer 6","CrawlerBot"
"FAST Enterprise Crawler/6 (crawler@fast.no)","CrawlerBot"
"FAST Enterprise Crawler/6 (www.fastsearch.com)","CrawlerBot"
"FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)","CrawlerBot"
"FastBug http://www.ay-up.com","CrawlerBot"
"FAST-WebCrawler/3.2 test","CrawlerBot"
"FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)","CrawlerBot"
"FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)","CrawlerBot"
"FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://www.alltheweb.com/help/webmaster/crawler)","CrawlerBot"
"FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler)","CrawlerBot"
"FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.x Multimedia","CrawlerBot"
"FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)","CrawlerBot"
"favicon finder at http://iconsurf.com/","CrawlerBot"
"favicon monitor at http://iconsurf.com/","CrawlerBot"
"Mozilla/4.0 (compatible: FDSE robot)","CrawlerBot"
"Filangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)","CrawlerBot"
"Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com)","CrawlerBot"
"Filangy/1.01 (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)","CrawlerBot"
"FindLinks/0.71 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.82 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.87 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.89 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)","CrawlerBot"
"Flickbot 1.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"FlickBot 2.0 RPT-HTTPClient/0.3-3","CrawlerBot"
"Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; http://www.pgts.com.au/; +http://www.galaxy.com/info/crawler.html)","CrawlerBot"
"FyberSpider (+http://www.fybersearch.com/fyberspider.php)","CrawlerBot"
"GAIS Robot/1.1A2","CrawlerBot"
"Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)","CrawlerBot"
"GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)","CrawlerBot"
"gatherer/0.9","CrawlerBot"
"gazz/5.0 (gazz@nttr.co.jp)","CrawlerBot"
"Generic","CrawlerBot"
"GeonaBot 1.0; http://www.geona.com/","CrawlerBot"
"GeonaBot/1.1; http://www.geona.com/","CrawlerBot"
"GetRight/4.5e","CrawlerBot"
"Gigabot/1.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)","CrawlerBot"
"Goldfire Server","CrawlerBot"
"Googlebot (+http://www.google.com/bot.html)","CrawlerBot"
"GoogleBot/2.1","CrawlerBot"
"Googlebot/2.1 (+http://www.google.com/bot.html)","CrawlerBot"
"googlebot/2.1 (+http://www.googlebot.com/bot.html","CrawlerBot"
"Googlebot/2.1 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Googlebot/2.1 (+http://www.googlebot.com/bot.html) (compatible; MSIE 6.0; )","CrawlerBot"
"Googlebot/2.1 (compatible; MSIE; Windows)","CrawlerBot"
"googlebot/2.1; +http://www.google.com/bot.html","CrawlerBot"
"Googlebot/2.1+(+http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.)","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Googlebot/Test (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Googlebot-Image/1.0","CrawlerBot"
"Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html","CrawlerBot"
"Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Green Research, Inc.","CrawlerBot"
"GregBot (compatible; MSIE; Windows; Q312461)","CrawlerBot"
"grub crawler","CrawlerBot"
"grub crawler(http://www.grub.org)","CrawlerBot"
"grub-client","CrawlerBot"
"Mozilla/4.0 (compatible; Grubclient; windows; SV1; .NET CLR 1.1.4322)","CrawlerBot"
"Mozilla/4.0 (compatible; Grubclient-2.2-internal-beta)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-2.6.0)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.07; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-2.3)","CrawlerBot"
"gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com)","CrawlerBot"
"Crawler [en] (compatible; Crawler Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)","CrawlerBot"
"Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Roadrunner; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SC/5.60/1.01/FS-Internett; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; stokeybot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts-MyWay; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot/2.1 (+http://www.googlebot.com/bot.html))","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot/2.1 (+http://www.googlebot.com/bot.html); Maxthon; FDM)","CrawlerBot"
"Harvest-NG/1.0.2","CrawlerBot"
"Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)","CrawlerBot"
"Hatena Antenna/0.4 (http://a.hatena.ne.jp/help)","CrawlerBot"
"hget/0.3","CrawlerBot"
"Hitwise Spider v1.0 http://www.hitwise.com","CrawlerBot"
"htdig","CrawlerBot"
"htdig/3.1.5 (admin@ipc-opc.lan)","CrawlerBot"
"htdig/3.1.5 (unconfigured@htdig.searchengine.maintainer)","CrawlerBot"
"htdig/3.1.6 (http://computerorgs.com)","CrawlerBot"
"Html Link Validator (www.lithopssoft.com)","CrawlerBot"
"Httpcheck/1.0 (Perl 5.006001)","CrawlerBot"
"HTTPConnect","CrawlerBot"
"httpget-5.2.2","CrawlerBot"
"Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)","CrawlerBot"
"ia_archiver","CrawlerBot"
"lcabotAccept: */*","CrawlerBot"
"ichiro/1.0 (ichiro@nttr.co.jp)","CrawlerBot"
"IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)","CrawlerBot"
"Mozilla/4.0 (compatible; ICS 1.2.105)","CrawlerBot"
"Iltrovatore-Setaccio","CrawlerBot"
"IlTrovatore-Setaccio (+http://www.iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/0.03-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (+http://www.iltrovatore.it/aiuto/faq.html)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (Indexing; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"Iltrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"imagefetch/0.1 libwww-perl/5.66","CrawlerBot"
"Mozilla/3.0 (compatible; Indy Library)","CrawlerBot"
"InelaBot/0.2 (+http://inelegant.org/bot)","CrawlerBot"
"InfoSeek Sidewinder/1.0A","CrawlerBot"
"Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX)","CrawlerBot"
"Infoseek SideWinder/2.0B (Linux 2.4 i686)","CrawlerBot"
"Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)","CrawlerBot"
"Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Slurp/si-emb (slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"InternetLinkAgent/3.1","CrawlerBot"
"IPiumBot laurion(dot)com","CrawlerBot"
"IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler)","CrawlerBot"
"http://www.istarthere.com (spider@istarthere.com)","CrawlerBot"
"Java1.4.0","CrawlerBot"
"JoBo/1.3 (http://www.matuschek.net/jobo.html)","CrawlerBot"
"k2spider","CrawlerBot"
"KMcrawler","CrawlerBot"
"Knowledge.com/0.2","CrawlerBot"
"Knowledge.com/0.3","CrawlerBot"
"Knowledge Engine","CrawlerBot"
"kuloko-bot/0.2","CrawlerBot"
"Larbin (larbin2.6.2@unspecified.mail)","CrawlerBot"
"larbin (samualt9@bigfoot.com)","CrawlerBot"
"larbin samualt9@bigfoot.com","CrawlerBot"
"larbin_extended (larbin@oktie.com)","CrawlerBot"
"larbin_test (nobody@airmail.etn)","CrawlerBot"
"LARBIN-EXPERIMENTAL (efp@gmx.net)","CrawlerBot"
"LARBIN-EXPERIMENTAL efp@gmx.net","CrawlerBot"
"Mozilla (la2@unspecified.mail)","CrawlerBot"
"Mozilla la2@unspecified.mail","CrawlerBot"
"Mozilla/4.0 (efp@gmx.net)","CrawlerBot"
"Mozilla/4.0 efp@gmx.net","CrawlerBot"
"MSIE-5.13 (larbin@unspecified.mail)","CrawlerBot"
"MSIE-5.13 larbin@unspecified.mail","CrawlerBot"
"SearchGuild_DMOZ_Experiment (chris@searchguild.com)","CrawlerBot"
"SearchGuild_DMOZ_Experiment chris@searchguild.com","CrawlerBot"
"WinampMPEG/2.00 (larbin@unspecified.mail)","CrawlerBot"
"WinampMPEG/2.00 larbin@unspecified.mail","CrawlerBot"
"Larbin larbin2.6.2@unspecified.mail","CrawlerBot"
"larbin_2.6.2 (kalou@kalou.net)","CrawlerBot"
"larbin_2.6.2 (larbin@correa.org)","CrawlerBot"
"larbin_2.6.2 (larbin2.6.2@unspecified.mail)","CrawlerBot"
"larbin_2.6.2 (pimenas@softnet.tuc.gr)","CrawlerBot"
"larbin_2.6.2 (pimenas@systems.tuc.gr)","CrawlerBot"
"larbin_2.6.2 (sumeet_sobti@yahoo.com)","CrawlerBot"
"larbin_2.6.2 (vitalbox1@hotmail.com)","CrawlerBot"
"larbin_2.6.2 (vshelk@yahoo.com)","CrawlerBot"
"larbin_2.6.2 larbin@correa.org","CrawlerBot"
"larbin_2.6.2 larbin2.6.2@unspecified.mail","CrawlerBot"
"larbin_2.6.2 pimenas@systems.tuc.gr","CrawlerBot"
"larbin_2.6.2 sumeet_sobti@yahoo.com","CrawlerBot"
"larbin_2.6.2 vitalbox1@hotmail.com","CrawlerBot"
"larbin_2.6.3 (andreas.beder@chello.at)","CrawlerBot"
"larbin_2.6.3 (larbin2.6.3@unspecified.mail)","CrawlerBot"
"larbin_2.6.3 (larbin-crawler@un.bewaff.net)","CrawlerBot"
"larbin_2.6.3 (pimenas@softnet.tuc.gr)","CrawlerBot"
"larbin_2.6.3 larbin2.6.3@unspecified.mail","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (Tomi.Silander@hiit.fi)","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (tsilande@hiit.fi)","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) Tomi.Silander@hiit.fi","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) tsilande@hiit.fi","CrawlerBot"
"eseek-crawler-larbin-2.63 (crawler@exactseek.com)","CrawlerBot"
"eseek-crawler-larbin-2.63 crawler@exactseek.com","CrawlerBot"
"libwww-MGET/1.0 libwww/5.2.8","CrawlerBot"
"Perl-Win32::Internet/0.082","CrawlerBot"
"/ libwww/5.3.2","CrawlerBot"
"/ libwww/5.4.0","CrawlerBot"
"libwww-perl/5.48","CrawlerBot"
"libwww-perl/5.50","CrawlerBot"
"libwww-perl/5.51","CrawlerBot"
"libwww-perl/5.52 FP/4.0","CrawlerBot"
"libwww-perl/5.53","CrawlerBot"
"libwww-perl/5.63","CrawlerBot"
"libwww-perl/5.64","CrawlerBot"
"libwww-perl/5.65","CrawlerBot"
"MyApp/0.1 libwww-perl/5.65","CrawlerBot"
"rawiswar/0.1 libwww-perl/5.66","CrawlerBot"
"libwww-perl/5.68","CrawlerBot"
"libwww-perl/5.69","CrawlerBot"
"VanillaZilla/0.1 libwww-perl/5.69","CrawlerBot"
"libwww-perl/5.74","CrawlerBot"
"libwww-perl/5.75","CrawlerBot"
"libwww-perl/5.76","CrawlerBot"
"libwww-perl/5.800","CrawlerBot"
"libwww-perl/5.801","CrawlerBot"
"libwww-perl/5.802","CrawlerBot"
"libwww-perl/5.803","CrawlerBot"
"LimeBot/1.0 (+www.cruiselime.com/LimeBot.php)","CrawlerBot"
"Linkbot 3.0","CrawlerBot"
"LinkLint-checkonly/2.3.5","CrawlerBot"
"Linknzbot/ (+http://www.linknz.co.nz/robot.php)","CrawlerBot"
"Linknzbot 2004/(+http://www.linknz.co.nz/robot.php)","CrawlerBot"
"Links SQL (http://gossamer-threads.com/scripts/links-sql/)","CrawlerBot"
"Lite Bot 0616B","CrawlerBot"
"LNSpiderguy","CrawlerBot"
"Look.com","CrawlerBot"
"lwp-trivial/1.29","CrawlerBot"
"lwp-trivial/1.35","CrawlerBot"
"lwp-trivial/1.36","CrawlerBot"
"lwp-request/2.01","CrawlerBot"
"LWP::Simple/5.48","CrawlerBot"
"LWP::Simple/5.65","CrawlerBot"
"Lycos_Spider_(modspider)","CrawlerBot"
"Mediapartners-Google/2.1","CrawlerBot"
"Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Mercator-2.0","CrawlerBot"
"metacarta (crawler@metacarta.com)","CrawlerBot"
"metacarta crawler@metacarta.com","CrawlerBot"
"MetaGer-LinkChecker","CrawlerBot"
"Microsoft URL Control - 5.00.3609","CrawlerBot"
"Microsoft URL Control - 5.01.4319","CrawlerBot"
"Microsoft URL Control - 6.00.8169","CrawlerBot"
"Microsoft URL Control - 6.00.8862","CrawlerBot"
"Microsoft-ATL-Native/7.00","CrawlerBot"
"MicrosoftPrototypeCrawler (How''s my crawling? mailto:newbiecrawler@hotmail.com)","CrawlerBot"
"moget/1.0 (moget@goo.ne.jp)","CrawlerBot"
"moget/2.1 (moget@goo.ne.jp)","CrawlerBot"
"mozDex/0.04-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"Mozilla/4.0 (compatible; Netcraft Web Server Survey)","CrawlerBot"
"Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com)","CrawlerBot"
"Mozilla/4.0 (stat 0.12) (statbot@gmail.com)","CrawlerBot"
"Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;) Unchaos/Crawler","CrawlerBot"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20041027 NaverBot/0.9.3","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040206 GoogleBot/1.1","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040730 Googlebot/2.1/2.1","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040707 Lightningspider/0.9.2","CrawlerBot"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040805 Googlebot/2.1","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider Cache Manager","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider DAV","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider DAV 1.1","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider Protocol Discovery","CrawlerBot"
"Mozilla/2.0 (compatible; MS FrontPage 4.0)","CrawlerBot"
"MSFrontPage/4.0","CrawlerBot"
"Mozilla/2.0 (compatible; MS FrontPage 5.0)","CrawlerBot"
"MSFrontPage/5.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; T-Online Internatinal AG; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; DT; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; QXW0338t; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; FunWebProducts; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; .NET CLR 1.0.3705; Googlebot; .NET CLR 1.1.4322; .NET CLR 1.0.3705; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows compatible LesnikBot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; 3COM U.S. Robotics)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; BOTW)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; http://www.pregnancycrawler.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; Googlebot; .NET CLR 1.1.4322)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; AskBar 3.00; .NET CLR 1.1.4322; Fluffi Bot+)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; CDSource=v9e.03; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 3.1; Googlebot/2.1)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.40607; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; DigExt; FunWebProducts; Media Center PC 3.0; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows XP Professional Bot v.5.)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; Q312461; BTopenworld; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; matlas-2.0.2501; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)","CrawlerBot"
"MSNBOT/0.1 (http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/0.11 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/0.3 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/1.0 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"MSProxy/2.0","CrawlerBot"
"MSRBOT/0.1 (http://research.microsoft.com/research/sv/msrbot/)","CrawlerBot"
"Mozilla/3.01 (compatible;)","CrawlerBot"
"NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"NaverBot_dloader/1.5","CrawlerBot"
"dloader(NaverRobot)/1.0","CrawlerBot"
"dloader(NaverRobot)/1.5","CrawlerBot"
"NetAnts/1.25","CrawlerBot"
"NetNoseCrawler/v1.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com)","CrawlerBot"
"NetResearchServer(http://www.look.com)","CrawlerBot"
"NetResearchServer/2.4(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.5(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.7(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.8(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.9(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/3.4(loopimprovements.com/robot.html)","CrawlerBot"
"NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot)","CrawlerBot"
"NG/1.0","CrawlerBot"
"NPBot","CrawlerBot"
"NPBot (http://www.nameprotect.com/botinfo.html)","CrawlerBot"
"NPBot-1/2.0","CrawlerBot"
"NPBot-1/2.0 (http://www.nameprotect.com/botinfo.html)","CrawlerBot"
"NuSearch Spider www.nusearch.com","CrawlerBot"
"NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"NutchCVS/0.05-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"CreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"Robot: NutchCrawler, Owner: wdavies@acm.org","CrawlerBot"
"NutchOrg/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)","CrawlerBot"
"oBot","CrawlerBot"
"Ocelli/1.3 (http://www.globalspec.com/Ocelli)","CrawlerBot"
"OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Internet Categorizer","CrawlerBot"
"OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Job Crawler","CrawlerBot"
"Openbot/3.0+(robot@monkia.com.tw;+http://gais.cs.ccu.edu.tw/robot.php)","CrawlerBot"
"Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)","CrawlerBot"
"Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)","CrawlerBot"
"OrangeBot","CrawlerBot"
"Mozilla/4.0 (compatible; Advanced Email Extractor v2.24)","CrawlerBot"
"Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"OWR_Crawler 0.1","CrawlerBot"
"parabot (paracite@ecs.soton.ac.uk)","CrawlerBot"
"Patwebbot (http://www.herz-power.de/technik.html)","CrawlerBot"
"pavuk/0.9pl28 i586-pc-cygwin","CrawlerBot"
"pavuk/0.9pl29b i686-pc-linux-gnu","CrawlerBot"
"PEERbot www.peerbot.com","CrawlerBot"
"pipeLiner/0.3a (PipeLine Spider; http://www.pipeline-search.com/webmaster.html; webmaster@pipeline-search.com)","CrawlerBot"
"http://www.planethosting.com","CrawlerBot"
"polybot 1.0 (http://cis.poly.edu/polybot/)","CrawlerBot"
"Pompos/1.1 http://pompos.iliad.fr","CrawlerBot"
"Pompos/1.2 http://pompos.iliad.fr","CrawlerBot"
"Pompos/1.3 http://dir.com/pompos.html","CrawlerBot"
"Portal Manager 0.7","CrawlerBot"
"potbot 1.0","CrawlerBot"
"Program Shareware 1.0.3","CrawlerBot"
"ProWebGuide Link Checker (http://www.prowebguide.com)","CrawlerBot"
"psbot/0.1 (+http://www.picsearch.com/bot.html)","CrawlerBot"
"pverify/1.2","CrawlerBot"
"PWS.Kiosk - Content Filtering","CrawlerBot"
"QPCreep Test Rig ( We are not indexing, just testing )","CrawlerBot"
"QuepasaCreep ( crawler@quepasacorp.com )","CrawlerBot"
"QuepasaCreep v0.9.14","CrawlerBot"
"QuepasaCreep v0.9.13","CrawlerBot"
"reifier.org (admin@reifier.org)","CrawlerBot"
"reifier.org admin@reifier.org","CrawlerBot"
"rico/0.1","CrawlerBot"
"RixBot (http://www.oops-as.no/rix/)","CrawlerBot"
"RoboPal (http://www.findpal.com/)","CrawlerBot"
"RobotMidareru/0.7libwww-perl/5.65","CrawlerBot"
"Search Engine World Robots.txt Validator at http://www.searchengineworld.com/cgi-bin/robotcheck.cgi","CrawlerBot"
"Robozilla/1.0","CrawlerBot"
"RPT-HTTPClient/0.3-3","CrawlerBot"
"SafariBookmarkChecker/1.25 (+http://www.coriolis.ch/)","CrawlerBot"
"SafariBookmarkChecker/1.26 (+http://www.coriolis.ch/)","CrawlerBot"
"Scooter/1.0","CrawlerBot"
"Scooter-ARS-1.1","CrawlerBot"
"Scooter-3.0.FS - Altavista.com","CrawlerBot"
"Scooter/3.2","CrawlerBot"
"Scooter/3.2.SF0","CrawlerBot"
"Scooter_x0-3.2.EX","CrawlerBot"
"Scooter-3.2","CrawlerBot"
"Scooter-3.2.BT","CrawlerBot"
"Scooter-3.2.EX","CrawlerBot"
"Scooter-3.2.FNR","CrawlerBot"
"Scooter-3.2.PDF","CrawlerBot"
"Scooter-3.2.SF0","CrawlerBot"
"Scooter-3.2.TX.FNR","CrawlerBot"
"Scooter-3.2.XX0","CrawlerBot"
"Scooter/3.3","CrawlerBot"
"Scooter/3.3.QA","CrawlerBot"
"Scooter/3.3.QA.pczukor","CrawlerBot"
"Scooter/3.3.vscooter","CrawlerBot"
"Scooter/3.3_SF","CrawlerBot"
"Scrubby/2.1 (http://www.scrubtheweb.com/abs/meta-check.html)","CrawlerBot"
"Scrubby/2.2 (http://www.scrubtheweb.com/)","CrawlerBot"
"Search Agent 1.0","CrawlerBot"
"SearchSpider.com/1.1","CrawlerBot"
"Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3","CrawlerBot"
"Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)","CrawlerBot"
"semanticdiscovery/0.1","CrawlerBot"
"Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au)","CrawlerBot"
"sherlock/1.3 httpget/1.3","CrawlerBot"
"sherlock_spider (jimfan@163.com)","CrawlerBot"
"InternetSeer.com","CrawlerBot"
"sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com)","CrawlerBot"
"sitescooper/3.1.2 (http://sitescooper.org) libwww-perl/5.51","CrawlerBot"
"SiteXpert","CrawlerBot"
"SlySearch/1.3 (http://www.slysearch.com)","CrawlerBot"
"SlySearch/1.3 http://www.slysearch.com","CrawlerBot"
"sohu-search","CrawlerBot"
"Speedy Spider (http://www.entireweb.com)","CrawlerBot"
"Speedy_Spider_(http://www.entireweb.com)","CrawlerBot"
"Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)","CrawlerBot"
"SpiderKU/0.9","CrawlerBot"
"SpiderMonkey/7.04 (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)","CrawlerBot"
"Spider_Monkey/7.06 (SpiderMonkey.ca info at http://SpiderMonkey.ca /sm.shtml)","CrawlerBot"
"Spider_Monkey/7.06 (SpiderMonkey.ca info at http://www.spidermonkey.ca/sm.shtml)","CrawlerBot"
"Mozilla/5.0 (compatible; SpurlBot/0.2)","CrawlerBot"
"Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)","CrawlerBot"
"Star Downloader","CrawlerBot"
"Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)","CrawlerBot"
"Mozilla/4.0 (compatible; SuperCleaner 2.56; Windows NT 5.1)","CrawlerBot"
"Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;)","CrawlerBot"
"Szukacz/1.5","CrawlerBot"
"Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)","CrawlerBot"
"Tarantula Experimental Crawler","CrawlerBot"
"Tcl http client package 1.0","CrawlerBot"
"Tcl http client package 2.3","CrawlerBot"
"(Teradex Mapper; mapper@teradex.com; http://www.teradex.com)","CrawlerBot"
"Teradex_Crawler (crawler@teradex.com; http://crawler.teradex.com)","CrawlerBot"
"TheSuBot/0.1 (www.thesubot.de)","CrawlerBot"
"thesubot-beta-www.thesubot.de","CrawlerBot"
"thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)","CrawlerBot"
"timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html","CrawlerBot"
"Tkensaku/0.9 (http://www.tkensaku.com/q.html)","CrawlerBot"
"TranSGeniKBot (http://www.tsgk.net)","CrawlerBot"
"TranSGeniKBot http://www.tsgk.net","CrawlerBot"
"TulipChain/5.7 (http://ostermiller.org/tulipchain/) Java/1.4.0_02 (http://java.sun.com/) Windows_Me/4.90","CrawlerBot"
"TulipChain/5.94 (http://ostermiller.org/tulipchain/) Java/1.4.1_01 (http://apple.com/) Mac_OS_X/10.2.8","CrawlerBot"
"TulipChain/6.01 (http://ostermiller.org/tulipchain/) Java/1.4.2_03 (http://java.sun.com/) Windows_XP/5.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"TulipChain/6.02 (http://ostermiller.org/tulipchain/) Java/1.4.2_03 (http://apple.com/) Mac_OS_X/10.3.3 RPT-HTTPClient/0.3-3","CrawlerBot"
"TulipChain/6.03 (http://ostermiller.org/tulipchain/) Java/1.4.2_05 (http://java.sun.com/) Windows_XP/5.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"TurnitinBot/1.4 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/1.4 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TurnitinBot/2.0 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TutorGigBot/1.5 ( +http://www.tutorgig.info )","CrawlerBot"
"Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)","CrawlerBot"
"UdmSearch/3.1.20","CrawlerBot"
"UIowaCrawler/1.0","CrawlerBot"
"UIowaCrawler/2.0","CrawlerBot"
"unchaos_crawler_2.0.2 (search.engine@unchaos.com)","CrawlerBot"
"VM4050/132.037 UP.Browser/6.2.2.4.e.1.100 (GUI) MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om)","CrawlerBot"
"USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)","CrawlerBot"
"Vagabondo/2.0 MT (webagent at wise-guys dot nl)","CrawlerBot"
"Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)","CrawlerBot"
"Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)","CrawlerBot"
"Vivante Link Checker (http://www.vivante.com)","CrawlerBot"
"void-bot/0.1 (bot@void.be; http://www.void.be/)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6","CrawlerBot"
"Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2","CrawlerBot"
"VSE/1.0 (vsecrawler@hotmail.com)","CrawlerBot"
"vspider","CrawlerBot"
"W3C_Validator/1.183 libwww-perl/5.64","CrawlerBot"
"W3C_Validator/1.305.2.109 libwww-perl/5.79","CrawlerBot"
"W3C_Validator/1.305.2.12 libwww-perl/5.64","CrawlerBot"
"W3C_Validator/1.305.2.137 libwww-perl/5.79","CrawlerBot"
"W3C_Validator/1.305.2.148 libwww-perl/5.800","CrawlerBot"
"W3C_Validator/1.305.2.148 libwww-perl/5.803","CrawlerBot"
"W3C-checklink/2.90 libwww-perl/5.64","CrawlerBot"
"W3C-checklink/3.6.2.3 libwww-perl/5.64","CrawlerBot"
"W3C-checklink/3.9.2 [3.17] libwww-perl/5.79","CrawlerBot"
"W3C-checklink/4.0 [4.4] libwww-perl/5.800","CrawlerBot"
"W3C-checklink/4.1 [4.14] libwww-perl/5.800","CrawlerBot"
"webbot","CrawlerBot"
"Webclipping.com","CrawlerBot"
"webcollage/1.102","CrawlerBot"
"webcollage/1.104","CrawlerBot"
"webcollage/1.87","CrawlerBot"
"webcollage/1.93","CrawlerBot"
"webcollage/1.94","CrawlerBot"
"Thu Mar 27 18:20:34 CET 2003WebcraftBoot","CrawlerBot"
"Fri Nov 15 04:51:18 EST 2002WebcraftBoot Java/1.4.1_01","CrawlerBot"
"Sun Apr 20 22:00:01 EDT 2003WebcraftBoot Java/1.4.2-beta","CrawlerBot"
"Tue Apr 15 22:00:03 EDT 2003WebcraftBoot Java/1.4.2-beta","CrawlerBot"
"WebFilter Robot 1.0","CrawlerBot"
"Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56)","CrawlerBot"
"WebRACE/1.1 (University of Cyprus, Distributed Crawler)","CrawlerBot"
"WebSauger 1.20b","CrawlerBot"
"http://www.websearch.com.au (larbin2.6.2@unspecified.mail)","CrawlerBot"
"http://www.websearch.com.au larbin2.6.2@unspecified.mail","CrawlerBot"
"http://www.WebSearch.com.au/ (larbin2.6.2@unspecified.mail)","CrawlerBot"
"http://www.WebSearch.com.au/ larbin2.6.2@unspecified.mail","CrawlerBot"
"www.WebSearch.com.au (search@websearch.com.au)","CrawlerBot"
"www.WebSearch.com.au search@websearch.com.au","CrawlerBot"
"WebSearch/2.0.1 (Dez@Blanchfield.COM.AU, http://www.WebSearch.com.au/)","CrawlerBot"
"WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU)","CrawlerBot"
"http://www.WebSearch.com.au/ - Australian Search Engine/3.1.3 (sites@websearch.com.au)","CrawlerBot"
"http://www.WebSearch.com.au/ - Australian Search Engine/3.1.6 (sites@websearch.com.au)","CrawlerBot"
"www.webwombat.com.au","CrawlerBot"
"webyield robot (http://www.webyield.net/search/search.pl)","CrawlerBot"
"Wget/1.5.2","CrawlerBot"
"Wget/1.5.3","CrawlerBot"
"Wget/1.5.3.1","CrawlerBot"
"Wget/1.6","CrawlerBot"
"Wget/1.7","CrawlerBot"
"Wget/1.8","CrawlerBot"
"Wget/1.8.1","CrawlerBot"
"Wget/1.8.1+cvs","CrawlerBot"
"Wget/1.8.2","CrawlerBot"
"Wget/1.9","CrawlerBot"
"Wget/1.9-beta","CrawlerBot"
"Wget/1.9.1","CrawlerBot"
"Willow Internet Crawler by Twotrees V2.1","CrawlerBot"
"Wotbox/alpha0.5.1 (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02","CrawlerBot"
"http://www.ciml.co.uk","CrawlerBot"
"WWWeasel Robot v1.00 (http://wwweasel.de)","CrawlerBot"
"Xenu''s Link Sleuth 1.1a","CrawlerBot"
"Xenu Link Sleuth 1.2b","CrawlerBot"
"Xenu Link Sleuth 1.2d","CrawlerBot"
"Xenu Link Sleuth 1.2e","CrawlerBot"
"Xenu Link Sleuth 1.2f","CrawlerBot"
"Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)","CrawlerBot"
"Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)","CrawlerBot"
"YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine","CrawlerBot"
"Zao/0.1 (http://www.kototoi.org/zao/)","CrawlerBot"
"Zao/0.2 (http://www.kototoi.org/zao/)","CrawlerBot"
"Zao-Crawler","CrawlerBot"
"Zeus 3140 Webster Pro V2.9 Win32","CrawlerBot"
"Zeus 57657 Webster Pro V2.9 Win32","CrawlerBot"
"ZipppBot/0.11 (ZipppBot; http://www.zippp.net; webmaster@zippp.net)","CrawlerBot"
"ZoomSpider - wrensoft.com","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-1.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-12.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-2.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d03 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d05 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker Beta-d01 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 DLC (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 
LVL 12

Expert Comment

by:Sinoj Sebastian
ID: 18754309
> shortest and most effective list wins
So the list of all popular bot agent strings

    *Bot*

Most of the spider user-agent strings will have the substring "Bot" in it.
Ordinary user-agent string from IE, FF, NS etc do not contain "Bot".
Try filter using this.
I'm already using it.

From the above list I found That "CrawlerBot" is also a substring of bot agent strings

0
 
LVL 29

Expert Comment

by:rdivilbiss
ID: 18757241
No, in the list above, "CrawlerBot" is not part of the user agent string, it was a field in the database of my collection of live user agents taken from dozens of my web sites and hand categorized.  Ignore: ,"CrawlerBot"
0
 
LVL 16

Author Comment

by:OliWarner
ID: 18757392
Yeah I can parse those out without issue. Thanks Rod that looks like it'll do the job perfectly
0
 
LVL 29

Expert Comment

by:rdivilbiss
ID: 18758475
Those were as of January. There could always be a few new ones cropping up.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Envision that you are chipping away at another e-business site with a team of pundit developers and designers. Everything seems, by all accounts, to be going easily.
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
This Micro Tutorial will demonstrate how to add subdomains to your content reports. This can be very importing in having a site with multiple subdomains.
How to create a custom search shortcut to site-search Experts Exchange using Google in the Firefox browser. This eliminates the need to type out site:experts-exchange.com whenever you want to search the site. Launch your Bookmark Menu: Press 'Ctrl +…

948 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now