• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 7488
  • Last Modified:

User-agent strings

I need a list of the most popular spider user-agent strings. I've got several items on my website that log or increment things and I really only want some of those to be logging if the thing hitting the page is a real person and not a bot. So I'm left checking the user-agents.

I can either grab the most popular browsers or the most popular bots... Whatever is most efficient -- you decide!

Either way, time-complexity is an issue as it is a fairly busy site, so the shortest and most effective list wins =)
0
OliWarner
Asked:
OliWarner
1 Solution
 
fpintosCommented:
Have you tried setting up robot.txt to block these spiders? This is by far the simplest way.
0
 
OliWarnerAuthor Commented:
I don't want to block them from viewing the pages -- just stop my logging script counting hits from them.
0
 
rdivilbissCommented:
I've got them in a db, Oli.  Give me a minute to extract the bots from the browsers.
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
rdivilbissCommented:
http://www.rodsdot.com/downloads/bots.zip

"ADSAComponent (postmaster@cnds.ucd.ie)","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [fc3]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [c01]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [wf224]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [wf55]","CrawlerBot"
"Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)","CrawlerBot"
"Mozilla/4.0 (Search Engine Marketing Tactics Amsterdam 2002 Information Spider)","CrawlerBot"
"AnswerBus (http://www.answerbus.com/)","CrawlerBot"
"antibot-V1.1.11/i586-linux-2.2","CrawlerBot"
"antibot-V1.1.13/i586-linux-2.2","CrawlerBot"
"antibot-V1.2.0/redhat-linux-9","CrawlerBot"
"AOLserver-Tcl/3.5.6","CrawlerBot"
"AOL 8.0 (compatible; AOL 8.0; DOS; .NET CLR 1.1.4322)","CrawlerBot"
"appie 1.1 (www.walhello.com)","CrawlerBot"
"Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)","CrawlerBot"
"Art-Online.com 0.9(Beta)","CrawlerBot"
"Mozilla/2.0 (compatible; Ask Jeeves)","CrawlerBot"
"Mozilla/2.0 (compatible; Ask Jeeves/Teoma)","CrawlerBot"
"ASPseek/1.2.10","CrawlerBot"
"ASPseek/1.2.11","CrawlerBot"
"ASPseek/1.2.12","CrawlerBot"
"Mozilla/3.0 (compatible; AvantGo 3.2)","CrawlerBot"
"BaiDuSpider","CrawlerBot"
"Baiduspider+(+http://www.baidu.com/search/spider.htm)","CrawlerBot"
"battlebot","CrawlerBot"
"BDFetch","CrawlerBot"
"BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html)","CrawlerBot"
"BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"Big Brother (http://pauillac.inria.fr/~fpottier/)","CrawlerBot"
"BlogBot/1.2","CrawlerBot"
"boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-dc/0.5 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-dc/0.66 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-robot/1.0","CrawlerBot"
"boitho.com-robot/1.1","CrawlerBot"
"Mozilla/4.0 (compatible; BorderManager 3.0)","CrawlerBot"
"BrailleBot 1.0","CrawlerBot"
"BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html)","CrawlerBot"
"bumblebee/1.0 (bumblebee@relevare.com; http://www.relevare.com/)","CrawlerBot"
"Computer_and_Automation_Research_Institute_Crawler (nospamspidernospam@spider.ilab.sztakinospam.hunospam)","CrawlerBot"
"Computer_and_Automation_Research_Institute_Crawler (spider@spider.ilab.sztaki.hu)","CrawlerBot"
"cd34/0.1","CrawlerBot"
"CerberianDrtrs/Version-3.0-Release-24","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-40)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-11)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-12)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-13)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-16)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-17)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-41)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-43)","CrawlerBot"
"CipinetBot (http://www.cipinet.com/bot.html)","CrawlerBot"
"Clushbot/2.1 (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.21-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.23-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.24-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.6-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.9-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"ComMOOnity LambdaMOO/1.8.1","CrawlerBot"
"CrawlConvera0.1 (CrawlConvera@yahoo.com)","CrawlerBot"
"CrawlConvera0.1 (www.authoritativeweb.com)","CrawlerBot"
"ConveraCrawler/0.2","CrawlerBot"
"ConveraCrawler/0.5 (+http://www","CrawlerBot"
"cosmos/0.9_(robot@xyleme.com)","CrawlerBot"
"Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"Cowbot-0.1.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"Crawl_Application","CrawlerBot"
"CrocCrawler v3.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"CrocCrawler v4.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"Custo 2.0 (www.netwu.com)","CrawlerBot"
"CydralSpider/1.9 (Cydral Web Image Search; http://www.cydral.com)","CrawlerBot"
"DeepIndex (http://www.deepindex.com)","CrawlerBot"
"DeMozulator 1.0 (MacOS, dMoz URL Check Agent, trebor@animeigo.com)","CrawlerBot"
"DoCoMo/1.0/N504i/c10/TB","CrawlerBot"
"DoCoMo/1.0/P504iS/c10/TB","CrawlerBot"
"Dual Proxy","CrawlerBot"
"Dumbot(version 0.1 beta - dumbfind.com)","CrawlerBot"
"Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html)","CrawlerBot"
"Dumbot(version 0.1 beta)","CrawlerBot"
"EARTHCOM.info/1.2","CrawlerBot"
"EmailSiphon","CrawlerBot"
"Enterprise_Search/1.00.136;MSSQL (http://www.innerprise.net/es-spider.asp)","CrawlerBot"
"e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)","CrawlerBot"
"exactseek-crawler-2.63 (crawler@exactseek.com)","CrawlerBot"
"exactseek-crawler-2.63 crawler@exactseek.com","CrawlerBot"
"exactseek-crawler-2.63-5 (crawler@exactseek.com)","CrawlerBot"
"exactseek-crawler-2.63-5 crawler@exactseek.com","CrawlerBot"
"Explorer 6","CrawlerBot"
"FAST Enterprise Crawler/6 (crawler@fast.no)","CrawlerBot"
"FAST Enterprise Crawler/6 (www.fastsearch.com)","CrawlerBot"
"FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)","CrawlerBot"
"FastBug http://www.ay-up.com","CrawlerBot"
"FAST-WebCrawler/3.2 test","CrawlerBot"
"FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)","CrawlerBot"
"FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)","CrawlerBot"
"FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://www.alltheweb.com/help/webmaster/crawler)","CrawlerBot"
"FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler)","CrawlerBot"
"FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.x Multimedia","CrawlerBot"
"FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)","CrawlerBot"
"favicon finder at http://iconsurf.com/","CrawlerBot"
"favicon monitor at http://iconsurf.com/","CrawlerBot"
"Mozilla/4.0 (compatible: FDSE robot)","CrawlerBot"
"Filangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)","CrawlerBot"
"Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com)","CrawlerBot"
"Filangy/1.01 (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)","CrawlerBot"
"FindLinks/0.71 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.82 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.87 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.89 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)","CrawlerBot"
"Flickbot 1.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"FlickBot 2.0 RPT-HTTPClient/0.3-3","CrawlerBot"
"Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; http://www.pgts.com.au/; +http://www.galaxy.com/info/crawler.html)","CrawlerBot"
"FyberSpider (+http://www.fybersearch.com/fyberspider.php)","CrawlerBot"
"GAIS Robot/1.1A2","CrawlerBot"
"Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)","CrawlerBot"
"GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)","CrawlerBot"
"gatherer/0.9","CrawlerBot"
"gazz/5.0 (gazz@nttr.co.jp)","CrawlerBot"
"Generic","CrawlerBot"
"GeonaBot 1.0; http://www.geona.com/","CrawlerBot"
"GeonaBot/1.1; http://www.geona.com/","CrawlerBot"
"GetRight/4.5e","CrawlerBot"
"Gigabot/1.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)","CrawlerBot"
"Goldfire Server","CrawlerBot"
"Googlebot (+http://www.google.com/bot.html)","CrawlerBot"
"GoogleBot/2.1","CrawlerBot"
"Googlebot/2.1 (+http://www.google.com/bot.html)","CrawlerBot"
"googlebot/2.1 (+http://www.googlebot.com/bot.html","CrawlerBot"
"Googlebot/2.1 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Googlebot/2.1 (+http://www.googlebot.com/bot.html) (compatible; MSIE 6.0; )","CrawlerBot"
"Googlebot/2.1 (compatible; MSIE; Windows)","CrawlerBot"
"googlebot/2.1; +http://www.google.com/bot.html","CrawlerBot"
"Googlebot/2.1+(+http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.)","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Googlebot/Test (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Googlebot-Image/1.0","CrawlerBot"
"Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html","CrawlerBot"
"Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Green Research, Inc.","CrawlerBot"
"GregBot (compatible; MSIE; Windows; Q312461)","CrawlerBot"
"grub crawler","CrawlerBot"
"grub crawler(http://www.grub.org)","CrawlerBot"
"grub-client","CrawlerBot"
"Mozilla/4.0 (compatible; Grubclient; windows; SV1; .NET CLR 1.1.4322)","CrawlerBot"
"Mozilla/4.0 (compatible; Grubclient-2.2-internal-beta)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-2.6.0)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.07; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-2.3)","CrawlerBot"
"gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com)","CrawlerBot"
"Crawler [en] (compatible; Crawler Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)","CrawlerBot"
"Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Roadrunner; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SC/5.60/1.01/FS-Internett; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; stokeybot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts-MyWay; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot/2.1 (+http://www.googlebot.com/bot.html))","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot/2.1 (+http://www.googlebot.com/bot.html); Maxthon; FDM)","CrawlerBot"
"Harvest-NG/1.0.2","CrawlerBot"
"Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)","CrawlerBot"
"Hatena Antenna/0.4 (http://a.hatena.ne.jp/help)","CrawlerBot"
"hget/0.3","CrawlerBot"
"Hitwise Spider v1.0 http://www.hitwise.com","CrawlerBot"
"htdig","CrawlerBot"
"htdig/3.1.5 (admin@ipc-opc.lan)","CrawlerBot"
"htdig/3.1.5 (unconfigured@htdig.searchengine.maintainer)","CrawlerBot"
"htdig/3.1.6 (http://computerorgs.com)","CrawlerBot"
"Html Link Validator (www.lithopssoft.com)","CrawlerBot"
"Httpcheck/1.0 (Perl 5.006001)","CrawlerBot"
"HTTPConnect","CrawlerBot"
"httpget-5.2.2","CrawlerBot"
"Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)","CrawlerBot"
"ia_archiver","CrawlerBot"
"lcabotAccept: */*","CrawlerBot"
"ichiro/1.0 (ichiro@nttr.co.jp)","CrawlerBot"
"IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)","CrawlerBot"
"Mozilla/4.0 (compatible; ICS 1.2.105)","CrawlerBot"
"Iltrovatore-Setaccio","CrawlerBot"
"IlTrovatore-Setaccio (+http://www.iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/0.03-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (+http://www.iltrovatore.it/aiuto/faq.html)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (Indexing; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"Iltrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"imagefetch/0.1 libwww-perl/5.66","CrawlerBot"
"Mozilla/3.0 (compatible; Indy Library)","CrawlerBot"
"InelaBot/0.2 (+http://inelegant.org/bot)","CrawlerBot"
"InfoSeek Sidewinder/1.0A","CrawlerBot"
"Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX)","CrawlerBot"
"Infoseek SideWinder/2.0B (Linux 2.4 i686)","CrawlerBot"
"Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)","CrawlerBot"
"Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Slurp/si-emb (slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"InternetLinkAgent/3.1","CrawlerBot"
"IPiumBot laurion(dot)com","CrawlerBot"
"IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler)","CrawlerBot"
"http://www.istarthere.com (spider@istarthere.com)","CrawlerBot"
"Java1.4.0","CrawlerBot"
"JoBo/1.3 (http://www.matuschek.net/jobo.html)","CrawlerBot"
"k2spider","CrawlerBot"
"KMcrawler","CrawlerBot"
"Knowledge.com/0.2","CrawlerBot"
"Knowledge.com/0.3","CrawlerBot"
"Knowledge Engine","CrawlerBot"
"kuloko-bot/0.2","CrawlerBot"
"Larbin (larbin2.6.2@unspecified.mail)","CrawlerBot"
"larbin (samualt9@bigfoot.com)","CrawlerBot"
"larbin samualt9@bigfoot.com","CrawlerBot"
"larbin_extended (larbin@oktie.com)","CrawlerBot"
"larbin_test (nobody@airmail.etn)","CrawlerBot"
"LARBIN-EXPERIMENTAL (efp@gmx.net)","CrawlerBot"
"LARBIN-EXPERIMENTAL efp@gmx.net","CrawlerBot"
"Mozilla (la2@unspecified.mail)","CrawlerBot"
"Mozilla la2@unspecified.mail","CrawlerBot"
"Mozilla/4.0 (efp@gmx.net)","CrawlerBot"
"Mozilla/4.0 efp@gmx.net","CrawlerBot"
"MSIE-5.13 (larbin@unspecified.mail)","CrawlerBot"
"MSIE-5.13 larbin@unspecified.mail","CrawlerBot"
"SearchGuild_DMOZ_Experiment (chris@searchguild.com)","CrawlerBot"
"SearchGuild_DMOZ_Experiment chris@searchguild.com","CrawlerBot"
"WinampMPEG/2.00 (larbin@unspecified.mail)","CrawlerBot"
"WinampMPEG/2.00 larbin@unspecified.mail","CrawlerBot"
"Larbin larbin2.6.2@unspecified.mail","CrawlerBot"
"larbin_2.6.2 (kalou@kalou.net)","CrawlerBot"
"larbin_2.6.2 (larbin@correa.org)","CrawlerBot"
"larbin_2.6.2 (larbin2.6.2@unspecified.mail)","CrawlerBot"
"larbin_2.6.2 (pimenas@softnet.tuc.gr)","CrawlerBot"
"larbin_2.6.2 (pimenas@systems.tuc.gr)","CrawlerBot"
"larbin_2.6.2 (sumeet_sobti@yahoo.com)","CrawlerBot"
"larbin_2.6.2 (vitalbox1@hotmail.com)","CrawlerBot"
"larbin_2.6.2 (vshelk@yahoo.com)","CrawlerBot"
"larbin_2.6.2 larbin@correa.org","CrawlerBot"
"larbin_2.6.2 larbin2.6.2@unspecified.mail","CrawlerBot"
"larbin_2.6.2 pimenas@systems.tuc.gr","CrawlerBot"
"larbin_2.6.2 sumeet_sobti@yahoo.com","CrawlerBot"
"larbin_2.6.2 vitalbox1@hotmail.com","CrawlerBot"
"larbin_2.6.3 (andreas.beder@chello.at)","CrawlerBot"
"larbin_2.6.3 (larbin2.6.3@unspecified.mail)","CrawlerBot"
"larbin_2.6.3 (larbin-crawler@un.bewaff.net)","CrawlerBot"
"larbin_2.6.3 (pimenas@softnet.tuc.gr)","CrawlerBot"
"larbin_2.6.3 larbin2.6.3@unspecified.mail","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (Tomi.Silander@hiit.fi)","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (tsilande@hiit.fi)","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) Tomi.Silander@hiit.fi","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) tsilande@hiit.fi","CrawlerBot"
"eseek-crawler-larbin-2.63 (crawler@exactseek.com)","CrawlerBot"
"eseek-crawler-larbin-2.63 crawler@exactseek.com","CrawlerBot"
"libwww-MGET/1.0 libwww/5.2.8","CrawlerBot"
"Perl-Win32::Internet/0.082","CrawlerBot"
"/ libwww/5.3.2","CrawlerBot"
"/ libwww/5.4.0","CrawlerBot"
"libwww-perl/5.48","CrawlerBot"
"libwww-perl/5.50","CrawlerBot"
"libwww-perl/5.51","CrawlerBot"
"libwww-perl/5.52 FP/4.0","CrawlerBot"
"libwww-perl/5.53","CrawlerBot"
"libwww-perl/5.63","CrawlerBot"
"libwww-perl/5.64","CrawlerBot"
"libwww-perl/5.65","CrawlerBot"
"MyApp/0.1 libwww-perl/5.65","CrawlerBot"
"rawiswar/0.1 libwww-perl/5.66","CrawlerBot"
"libwww-perl/5.68","CrawlerBot"
"libwww-perl/5.69","CrawlerBot"
"VanillaZilla/0.1 libwww-perl/5.69","CrawlerBot"
"libwww-perl/5.74","CrawlerBot"
"libwww-perl/5.75","CrawlerBot"
"libwww-perl/5.76","CrawlerBot"
"libwww-perl/5.800","CrawlerBot"
"libwww-perl/5.801","CrawlerBot"
"libwww-perl/5.802","CrawlerBot"
"libwww-perl/5.803","CrawlerBot"
"LimeBot/1.0 (+www.cruiselime.com/LimeBot.php)","CrawlerBot"
"Linkbot 3.0","CrawlerBot"
"LinkLint-checkonly/2.3.5","CrawlerBot"
"Linknzbot/ (+http://www.linknz.co.nz/robot.php)","CrawlerBot"
"Linknzbot 2004/(+http://www.linknz.co.nz/robot.php)","CrawlerBot"
"Links SQL (http://gossamer-threads.com/scripts/links-sql/)","CrawlerBot"
"Lite Bot 0616B","CrawlerBot"
"LNSpiderguy","CrawlerBot"
"Look.com","CrawlerBot"
"lwp-trivial/1.29","CrawlerBot"
"lwp-trivial/1.35","CrawlerBot"
"lwp-trivial/1.36","CrawlerBot"
"lwp-request/2.01","CrawlerBot"
"LWP::Simple/5.48","CrawlerBot"
"LWP::Simple/5.65","CrawlerBot"
"Lycos_Spider_(modspider)","CrawlerBot"
"Mediapartners-Google/2.1","CrawlerBot"
"Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Mercator-2.0","CrawlerBot"
"metacarta (crawler@metacarta.com)","CrawlerBot"
"metacarta crawler@metacarta.com","CrawlerBot"
"MetaGer-LinkChecker","CrawlerBot"
"Microsoft URL Control - 5.00.3609","CrawlerBot"
"Microsoft URL Control - 5.01.4319","CrawlerBot"
"Microsoft URL Control - 6.00.8169","CrawlerBot"
"Microsoft URL Control - 6.00.8862","CrawlerBot"
"Microsoft-ATL-Native/7.00","CrawlerBot"
"MicrosoftPrototypeCrawler (How''s my crawling? mailto:newbiecrawler@hotmail.com)","CrawlerBot"
"moget/1.0 (moget@goo.ne.jp)","CrawlerBot"
"moget/2.1 (moget@goo.ne.jp)","CrawlerBot"
"mozDex/0.04-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"Mozilla/4.0 (compatible; Netcraft Web Server Survey)","CrawlerBot"
"Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com)","CrawlerBot"
"Mozilla/4.0 (stat 0.12) (statbot@gmail.com)","CrawlerBot"
"Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;) Unchaos/Crawler","CrawlerBot"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20041027 NaverBot/0.9.3","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040206 GoogleBot/1.1","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040730 Googlebot/2.1/2.1","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040707 Lightningspider/0.9.2","CrawlerBot"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040805 Googlebot/2.1","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider Cache Manager","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider DAV","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider DAV 1.1","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider Protocol Discovery","CrawlerBot"
"Mozilla/2.0 (compatible; MS FrontPage 4.0)","CrawlerBot"
"MSFrontPage/4.0","CrawlerBot"
"Mozilla/2.0 (compatible; MS FrontPage 5.0)","CrawlerBot"
"MSFrontPage/5.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; T-Online Internatinal AG; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; DT; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; QXW0338t; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; FunWebProducts; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; .NET CLR 1.0.3705; Googlebot; .NET CLR 1.1.4322; .NET CLR 1.0.3705; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows compatible LesnikBot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; 3COM U.S. Robotics)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; BOTW)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; http://www.pregnancycrawler.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; Googlebot; .NET CLR 1.1.4322)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; AskBar 3.00; .NET CLR 1.1.4322; Fluffi Bot+)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; CDSource=v9e.03; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 3.1; Googlebot/2.1)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.40607; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; DigExt; FunWebProducts; Media Center PC 3.0; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows XP Professional Bot v.5.)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; Q312461; BTopenworld; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; matlas-2.0.2501; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)","CrawlerBot"
"MSNBOT/0.1 (http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/0.11 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/0.3 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/1.0 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"MSProxy/2.0","CrawlerBot"
"MSRBOT/0.1 (http://research.microsoft.com/research/sv/msrbot/)","CrawlerBot"
"Mozilla/3.01 (compatible;)","CrawlerBot"
"NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"NaverBot_dloader/1.5","CrawlerBot"
"dloader(NaverRobot)/1.0","CrawlerBot"
"dloader(NaverRobot)/1.5","CrawlerBot"
"NetAnts/1.25","CrawlerBot"
"NetNoseCrawler/v1.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com)","CrawlerBot"
"NetResearchServer(http://www.look.com)","CrawlerBot"
"NetResearchServer/2.4(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.5(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.7(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.8(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.9(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/3.4(loopimprovements.com/robot.html)","CrawlerBot"
"NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot)","CrawlerBot"
"NG/1.0","CrawlerBot"
"NPBot","CrawlerBot"
"NPBot (http://www.nameprotect.com/botinfo.html)","CrawlerBot"
"NPBot-1/2.0","CrawlerBot"
"NPBot-1/2.0 (http://www.nameprotect.com/botinfo.html)","CrawlerBot"
"NuSearch Spider www.nusearch.com","CrawlerBot"
"NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"NutchCVS/0.05-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"CreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"Robot: NutchCrawler, Owner: wdavies@acm.org","CrawlerBot"
"NutchOrg/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)","CrawlerBot"
"oBot","CrawlerBot"
"Ocelli/1.3 (http://www.globalspec.com/Ocelli)","CrawlerBot"
"OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Internet Categorizer","CrawlerBot"
"OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Job Crawler","CrawlerBot"
"Openbot/3.0+(robot@monkia.com.tw;+http://gais.cs.ccu.edu.tw/robot.php)","CrawlerBot"
"Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)","CrawlerBot"
"Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)","CrawlerBot"
"OrangeBot","CrawlerBot"
"Mozilla/4.0 (compatible; Advanced Email Extractor v2.24)","CrawlerBot"
"Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"OWR_Crawler 0.1","CrawlerBot"
"parabot (paracite@ecs.soton.ac.uk)","CrawlerBot"
"Patwebbot (http://www.herz-power.de/technik.html)","CrawlerBot"
"pavuk/0.9pl28 i586-pc-cygwin","CrawlerBot"
"pavuk/0.9pl29b i686-pc-linux-gnu","CrawlerBot"
"PEERbot www.peerbot.com","CrawlerBot"
"pipeLiner/0.3a (PipeLine Spider; http://www.pipeline-search.com/webmaster.html; webmaster@pipeline-search.com)","CrawlerBot"
"http://www.planethosting.com","CrawlerBot"
"polybot 1.0 (http://cis.poly.edu/polybot/)","CrawlerBot"
"Pompos/1.1 http://pompos.iliad.fr","CrawlerBot"
"Pompos/1.2 http://pompos.iliad.fr","CrawlerBot"
"Pompos/1.3 http://dir.com/pompos.html","CrawlerBot"
"Portal Manager 0.7","CrawlerBot"
"potbot 1.0","CrawlerBot"
"Program Shareware 1.0.3","CrawlerBot"
"ProWebGuide Link Checker (http://www.prowebguide.com)","CrawlerBot"
"psbot/0.1 (+http://www.picsearch.com/bot.html)","CrawlerBot"
"pverify/1.2","CrawlerBot"
"PWS.Kiosk - Content Filtering","CrawlerBot"
"QPCreep Test Rig ( We are not indexing, just testing )","CrawlerBot"
"QuepasaCreep ( crawler@quepasacorp.com )","CrawlerBot"
"QuepasaCreep v0.9.14","CrawlerBot"
"QuepasaCreep v0.9.13","CrawlerBot"
"reifier.org (admin@reifier.org)","CrawlerBot"
"reifier.org admin@reifier.org","CrawlerBot"
"rico/0.1","CrawlerBot"
"RixBot (http://www.oops-as.no/rix/)","CrawlerBot"
"RoboPal (http://www.findpal.com/)","CrawlerBot"
"RobotMidareru/0.7libwww-perl/5.65","CrawlerBot"
"Search Engine World Robots.txt Validator at http://www.searchengineworld.com/cgi-bin/robotcheck.cgi","CrawlerBot"
"Robozilla/1.0","CrawlerBot"
"RPT-HTTPClient/0.3-3","CrawlerBot"
"SafariBookmarkChecker/1.25 (+http://www.coriolis.ch/)","CrawlerBot"
"SafariBookmarkChecker/1.26 (+http://www.coriolis.ch/)","CrawlerBot"
"Scooter/1.0","CrawlerBot"
"Scooter-ARS-1.1","CrawlerBot"
"Scooter-3.0.FS - Altavista.com","CrawlerBot"
"Scooter/3.2","CrawlerBot"
"Scooter/3.2.SF0","CrawlerBot"
"Scooter_x0-3.2.EX","CrawlerBot"
"Scooter-3.2","CrawlerBot"
"Scooter-3.2.BT","CrawlerBot"
"Scooter-3.2.EX","CrawlerBot"
"Scooter-3.2.FNR","CrawlerBot"
"Scooter-3.2.PDF","CrawlerBot"
"Scooter-3.2.SF0","CrawlerBot"
"Scooter-3.2.TX.FNR","CrawlerBot"
"Scooter-3.2.XX0","CrawlerBot"
"Scooter/3.3","CrawlerBot"
"Scooter/3.3.QA","CrawlerBot"
"Scooter/3.3.QA.pczukor","CrawlerBot"
"Scooter/3.3.vscooter","CrawlerBot"
"Scooter/3.3_SF","CrawlerBot"
"Scrubby/2.1 (http://www.scrubtheweb.com/abs/meta-check.html)","CrawlerBot"
"Scrubby/2.2 (http://www.scrubtheweb.com/)","CrawlerBot"
"Search Agent 1.0","CrawlerBot"
"SearchSpider.com/1.1","CrawlerBot"
"Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3","CrawlerBot"
"Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)","CrawlerBot"
"semanticdiscovery/0.1","CrawlerBot"
"Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au)","CrawlerBot"
"sherlock/1.3 httpget/1.3","CrawlerBot"
"sherlock_spider (jimfan@163.com)","CrawlerBot"
"InternetSeer.com","CrawlerBot"
"sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com)","CrawlerBot"
"sitescooper/3.1.2 (http://sitescooper.org) libwww-perl/5.51","CrawlerBot"
"SiteXpert","CrawlerBot"
"SlySearch/1.3 (http://www.slysearch.com)","CrawlerBot"
"SlySearch/1.3 http://www.slysearch.com","CrawlerBot"
"sohu-search","CrawlerBot"
"Speedy Spider (http://www.entireweb.com)","CrawlerBot"
"Speedy_Spider_(http://www.entireweb.com)","CrawlerBot"
"Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)","CrawlerBot"
"SpiderKU/0.9","CrawlerBot"
"SpiderMonkey/7.04 (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)","CrawlerBot"
"Spider_Monkey/7.06 (SpiderMonkey.ca info at http://SpiderMonkey.ca /sm.shtml)","CrawlerBot"
"Spider_Monkey/7.06 (SpiderMonkey.ca info at http://www.spidermonkey.ca/sm.shtml)","CrawlerBot"
"Mozilla/5.0 (compatible; SpurlBot/0.2)","CrawlerBot"
"Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)","CrawlerBot"
"Star Downloader","CrawlerBot"
"Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)","CrawlerBot"
"Mozilla/4.0 (compatible; SuperCleaner 2.56; Windows NT 5.1)","CrawlerBot"
"Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;)","CrawlerBot"
"Szukacz/1.5","CrawlerBot"
"Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)","CrawlerBot"
"Tarantula Experimental Crawler","CrawlerBot"
"Tcl http client package 1.0","CrawlerBot"
"Tcl http client package 2.3","CrawlerBot"
"(Teradex Mapper; mapper@teradex.com; http://www.teradex.com)","CrawlerBot"
"Teradex_Crawler (crawler@teradex.com; http://crawler.teradex.com)","CrawlerBot"
"TheSuBot/0.1 (www.thesubot.de)","CrawlerBot"
"thesubot-beta-www.thesubot.de","CrawlerBot"
"thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)","CrawlerBot"
"timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html","CrawlerBot"
"Tkensaku/0.9 (http://www.tkensaku.com/q.html)","CrawlerBot"
"TranSGeniKBot (http://www.tsgk.net)","CrawlerBot"
"TranSGeniKBot http://www.tsgk.net","CrawlerBot"
"TulipChain/5.7 (http://ostermiller.org/tulipchain/) Java/1.4.0_02 (http://java.sun.com/) Windows_Me/4.90","CrawlerBot"
"TulipChain/5.94 (http://ostermiller.org/tulipchain/) Java/1.4.1_01 (http://apple.com/) Mac_OS_X/10.2.8","CrawlerBot"
"TulipChain/6.01 (http://ostermiller.org/tulipchain/) Java/1.4.2_03 (http://java.sun.com/) Windows_XP/5.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"TulipChain/6.02 (http://ostermiller.org/tulipchain/) Java/1.4.2_03 (http://apple.com/) Mac_OS_X/10.3.3 RPT-HTTPClient/0.3-3","CrawlerBot"
"TulipChain/6.03 (http://ostermiller.org/tulipchain/) Java/1.4.2_05 (http://java.sun.com/) Windows_XP/5.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"TurnitinBot/1.4 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/1.4 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TurnitinBot/2.0 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TutorGigBot/1.5 ( +http://www.tutorgig.info )","CrawlerBot"
"Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)","CrawlerBot"
"UdmSearch/3.1.20","CrawlerBot"
"UIowaCrawler/1.0","CrawlerBot"
"UIowaCrawler/2.0","CrawlerBot"
"unchaos_crawler_2.0.2 (search.engine@unchaos.com)","CrawlerBot"
"VM4050/132.037 UP.Browser/6.2.2.4.e.1.100 (GUI) MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om)","CrawlerBot"
"USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)","CrawlerBot"
"Vagabondo/2.0 MT (webagent at wise-guys dot nl)","CrawlerBot"
"Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)","CrawlerBot"
"Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)","CrawlerBot"
"Vivante Link Checker (http://www.vivante.com)","CrawlerBot"
"void-bot/0.1 (bot@void.be; http://www.void.be/)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6","CrawlerBot"
"Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2","CrawlerBot"
"VSE/1.0 (vsecrawler@hotmail.com)","CrawlerBot"
"vspider","CrawlerBot"
"W3C_Validator/1.183 libwww-perl/5.64","CrawlerBot"
"W3C_Validator/1.305.2.109 libwww-perl/5.79","CrawlerBot"
"W3C_Validator/1.305.2.12 libwww-perl/5.64","CrawlerBot"
"W3C_Validator/1.305.2.137 libwww-perl/5.79","CrawlerBot"
"W3C_Validator/1.305.2.148 libwww-perl/5.800","CrawlerBot"
"W3C_Validator/1.305.2.148 libwww-perl/5.803","CrawlerBot"
"W3C-checklink/2.90 libwww-perl/5.64","CrawlerBot"
"W3C-checklink/3.6.2.3 libwww-perl/5.64","CrawlerBot"
"W3C-checklink/3.9.2 [3.17] libwww-perl/5.79","CrawlerBot"
"W3C-checklink/4.0 [4.4] libwww-perl/5.800","CrawlerBot"
"W3C-checklink/4.1 [4.14] libwww-perl/5.800","CrawlerBot"
"webbot","CrawlerBot"
"Webclipping.com","CrawlerBot"
"webcollage/1.102","CrawlerBot"
"webcollage/1.104","CrawlerBot"
"webcollage/1.87","CrawlerBot"
"webcollage/1.93","CrawlerBot"
"webcollage/1.94","CrawlerBot"
"Thu Mar 27 18:20:34 CET 2003WebcraftBoot","CrawlerBot"
"Fri Nov 15 04:51:18 EST 2002WebcraftBoot Java/1.4.1_01","CrawlerBot"
"Sun Apr 20 22:00:01 EDT 2003WebcraftBoot Java/1.4.2-beta","CrawlerBot"
"Tue Apr 15 22:00:03 EDT 2003WebcraftBoot Java/1.4.2-beta","CrawlerBot"
"WebFilter Robot 1.0","CrawlerBot"
"Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56)","CrawlerBot"
"WebRACE/1.1 (University of Cyprus, Distributed Crawler)","CrawlerBot"
"WebSauger 1.20b","CrawlerBot"
"http://www.websearch.com.au (larbin2.6.2@unspecified.mail)","CrawlerBot"
"http://www.websearch.com.au larbin2.6.2@unspecified.mail","CrawlerBot"
"http://www.WebSearch.com.au/ (larbin2.6.2@unspecified.mail)","CrawlerBot"
"http://www.WebSearch.com.au/ larbin2.6.2@unspecified.mail","CrawlerBot"
"www.WebSearch.com.au (search@websearch.com.au)","CrawlerBot"
"www.WebSearch.com.au search@websearch.com.au","CrawlerBot"
"WebSearch/2.0.1 (Dez@Blanchfield.COM.AU, http://www.WebSearch.com.au/)","CrawlerBot"
"WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU)","CrawlerBot"
"http://www.WebSearch.com.au/ - Australian Search Engine/3.1.3 (sites@websearch.com.au)","CrawlerBot"
"http://www.WebSearch.com.au/ - Australian Search Engine/3.1.6 (sites@websearch.com.au)","CrawlerBot"
"www.webwombat.com.au","CrawlerBot"
"webyield robot (http://www.webyield.net/search/search.pl)","CrawlerBot"
"Wget/1.5.2","CrawlerBot"
"Wget/1.5.3","CrawlerBot"
"Wget/1.5.3.1","CrawlerBot"
"Wget/1.6","CrawlerBot"
"Wget/1.7","CrawlerBot"
"Wget/1.8","CrawlerBot"
"Wget/1.8.1","CrawlerBot"
"Wget/1.8.1+cvs","CrawlerBot"
"Wget/1.8.2","CrawlerBot"
"Wget/1.9","CrawlerBot"
"Wget/1.9-beta","CrawlerBot"
"Wget/1.9.1","CrawlerBot"
"Willow Internet Crawler by Twotrees V2.1","CrawlerBot"
"Wotbox/alpha0.5.1 (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02","CrawlerBot"
"http://www.ciml.co.uk","CrawlerBot"
"WWWeasel Robot v1.00 (http://wwweasel.de)","CrawlerBot"
"Xenu''s Link Sleuth 1.1a","CrawlerBot"
"Xenu Link Sleuth 1.2b","CrawlerBot"
"Xenu Link Sleuth 1.2d","CrawlerBot"
"Xenu Link Sleuth 1.2e","CrawlerBot"
"Xenu Link Sleuth 1.2f","CrawlerBot"
"Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)","CrawlerBot"
"Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)","CrawlerBot"
"YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine","CrawlerBot"
"Zao/0.1 (http://www.kototoi.org/zao/)","CrawlerBot"
"Zao/0.2 (http://www.kototoi.org/zao/)","CrawlerBot"
"Zao-Crawler","CrawlerBot"
"Zeus 3140 Webster Pro V2.9 Win32","CrawlerBot"
"Zeus 57657 Webster Pro V2.9 Win32","CrawlerBot"
"ZipppBot/0.11 (ZipppBot; http://www.zippp.net; webmaster@zippp.net)","CrawlerBot"
"ZoomSpider - wrensoft.com","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-1.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-12.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-2.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d03 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d05 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker Beta-d01 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 DLC (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
0
 
Sinoj SebastianCommented:
> shortest and most effective list wins
So the list of all popular bot agent strings

    *Bot*

Most of the spider user-agent strings will have the substring "Bot" in it.
Ordinary user-agent string from IE, FF, NS etc do not contain "Bot".
Try filter using this.
I'm already using it.

From the above list I found That "CrawlerBot" is also a substring of bot agent strings

0
 
rdivilbissCommented:
No, in the list above, "CrawlerBot" is not part of the user agent string, it was a field in the database of my collection of live user agents taken from dozens of my web sites and hand categorized.  Ignore: ,"CrawlerBot"
0
 
OliWarnerAuthor Commented:
Yeah I can parse those out without issue. Thanks Rod that looks like it'll do the job perfectly
0
 
rdivilbissCommented:
Those were as of January. There could always be a few new ones cropping up.
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now