Solved

User-agent strings

Posted on 2007-03-19
8
7,101 Views
Last Modified: 2013-12-09
I need a list of the most popular spider user-agent strings. I've got several items on my website that log or increment things and I really only want some of those to be logging if the thing hitting the page is a real person and not a bot. So I'm left checking the user-agents.

I can either grab the most popular browsers or the most popular bots... Whatever is most efficient -- you decide!

Either way, time-complexity is an issue as it is a fairly busy site, so the shortest and most effective list wins =)
0
Comment
Question by:OliWarner
8 Comments
 
LVL 2

Expert Comment

by:fpintos
ID: 18753824
Have you tried setting up robot.txt to block these spiders? This is by far the simplest way.
0
 
LVL 16

Author Comment

by:OliWarner
ID: 18753862
I don't want to block them from viewing the pages -- just stop my logging script counting hits from them.
0
 
LVL 29

Expert Comment

by:rdivilbiss
ID: 18753977
I've got them in a db, Oli.  Give me a minute to extract the bots from the browsers.
0
 
LVL 29

Accepted Solution

by:
rdivilbiss earned 500 total points
ID: 18754003
http://www.rodsdot.com/downloads/bots.zip

"ADSAComponent (postmaster@cnds.ucd.ie)","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [fc3]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [c01]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [wf224]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [wf55]","CrawlerBot"
"Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)","CrawlerBot"
"Mozilla/4.0 (Search Engine Marketing Tactics Amsterdam 2002 Information Spider)","CrawlerBot"
"AnswerBus (http://www.answerbus.com/)","CrawlerBot"
"antibot-V1.1.11/i586-linux-2.2","CrawlerBot"
"antibot-V1.1.13/i586-linux-2.2","CrawlerBot"
"antibot-V1.2.0/redhat-linux-9","CrawlerBot"
"AOLserver-Tcl/3.5.6","CrawlerBot"
"AOL 8.0 (compatible; AOL 8.0; DOS; .NET CLR 1.1.4322)","CrawlerBot"
"appie 1.1 (www.walhello.com)","CrawlerBot"
"Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)","CrawlerBot"
"Art-Online.com 0.9(Beta)","CrawlerBot"
"Mozilla/2.0 (compatible; Ask Jeeves)","CrawlerBot"
"Mozilla/2.0 (compatible; Ask Jeeves/Teoma)","CrawlerBot"
"ASPseek/1.2.10","CrawlerBot"
"ASPseek/1.2.11","CrawlerBot"
"ASPseek/1.2.12","CrawlerBot"
"Mozilla/3.0 (compatible; AvantGo 3.2)","CrawlerBot"
"BaiDuSpider","CrawlerBot"
"Baiduspider+(+http://www.baidu.com/search/spider.htm)","CrawlerBot"
"battlebot","CrawlerBot"
"BDFetch","CrawlerBot"
"BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html)","CrawlerBot"
"BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"Big Brother (http://pauillac.inria.fr/~fpottier/)","CrawlerBot"
"BlogBot/1.2","CrawlerBot"
"boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-dc/0.5 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-dc/0.66 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-robot/1.0","CrawlerBot"
"boitho.com-robot/1.1","CrawlerBot"
"Mozilla/4.0 (compatible; BorderManager 3.0)","CrawlerBot"
"BrailleBot 1.0","CrawlerBot"
"BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html)","CrawlerBot"
"bumblebee/1.0 (bumblebee@relevare.com; http://www.relevare.com/)","CrawlerBot"
"Computer_and_Automation_Research_Institute_Crawler (nospamspidernospam@spider.ilab.sztakinospam.hunospam)","CrawlerBot"
"Computer_and_Automation_Research_Institute_Crawler (spider@spider.ilab.sztaki.hu)","CrawlerBot"
"cd34/0.1","CrawlerBot"
"CerberianDrtrs/Version-3.0-Release-24","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-40)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-11)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-12)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-13)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-16)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-17)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-41)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-43)","CrawlerBot"
"CipinetBot (http://www.cipinet.com/bot.html)","CrawlerBot"
"Clushbot/2.1 (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.21-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.23-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.24-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.6-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.9-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"ComMOOnity LambdaMOO/1.8.1","CrawlerBot"
"CrawlConvera0.1 (CrawlConvera@yahoo.com)","CrawlerBot"
"CrawlConvera0.1 (www.authoritativeweb.com)","CrawlerBot"
"ConveraCrawler/0.2","CrawlerBot"
"ConveraCrawler/0.5 (+http://www","CrawlerBot"
"cosmos/0.9_(robot@xyleme.com)","CrawlerBot"
"Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"Cowbot-0.1.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"Crawl_Application","CrawlerBot"
"CrocCrawler v3.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"CrocCrawler v4.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"Custo 2.0 (www.netwu.com)","CrawlerBot"
"CydralSpider/1.9 (Cydral Web Image Search; http://www.cydral.com)","CrawlerBot"
"DeepIndex (http://www.deepindex.com)","CrawlerBot"
"DeMozulator 1.0 (MacOS, dMoz URL Check Agent, trebor@animeigo.com)","CrawlerBot"
"DoCoMo/1.0/N504i/c10/TB","CrawlerBot"
"DoCoMo/1.0/P504iS/c10/TB","CrawlerBot"
"Dual Proxy","CrawlerBot"
"Dumbot(version 0.1 beta - dumbfind.com)","CrawlerBot"
"Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html)","CrawlerBot"
"Dumbot(version 0.1 beta)","CrawlerBot"
"EARTHCOM.info/1.2","CrawlerBot"
"EmailSiphon","CrawlerBot"
"Enterprise_Search/1.00.136;MSSQL (http://www.innerprise.net/es-spider.asp)","CrawlerBot"
"e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)","CrawlerBot"
"exactseek-crawler-2.63 (crawler@exactseek.com)","CrawlerBot"
"exactseek-crawler-2.63 crawler@exactseek.com","CrawlerBot"
"exactseek-crawler-2.63-5 (crawler@exactseek.com)","CrawlerBot"
"exactseek-crawler-2.63-5 crawler@exactseek.com","CrawlerBot"
"Explorer 6","CrawlerBot"
"FAST Enterprise Crawler/6 (crawler@fast.no)","CrawlerBot"
"FAST Enterprise Crawler/6 (www.fastsearch.com)","CrawlerBot"
"FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)","CrawlerBot"
"FastBug http://www.ay-up.com","CrawlerBot"
"FAST-WebCrawler/3.2 test","CrawlerBot"
"FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)","CrawlerBot"
"FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)","CrawlerBot"
"FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://www.alltheweb.com/help/webmaster/crawler)","CrawlerBot"
"FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler)","CrawlerBot"
"FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.x Multimedia","CrawlerBot"
"FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)","CrawlerBot"
"favicon finder at http://iconsurf.com/","CrawlerBot"
"favicon monitor at http://iconsurf.com/","CrawlerBot"
"Mozilla/4.0 (compatible: FDSE robot)","CrawlerBot"
"Filangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)","CrawlerBot"
"Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com)","CrawlerBot"
"Filangy/1.01 (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)","CrawlerBot"
"FindLinks/0.71 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.82 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.87 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.89 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)","CrawlerBot"
"Flickbot 1.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"FlickBot 2.0 RPT-HTTPClient/0.3-3","CrawlerBot"
"Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; http://www.pgts.com.au/; +http://www.galaxy.com/info/crawler.html)","CrawlerBot"
"FyberSpider (+http://www.fybersearch.com/fyberspider.php)","CrawlerBot"
"GAIS Robot/1.1A2","CrawlerBot"
"Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)","CrawlerBot"
"GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)","CrawlerBot"
"gatherer/0.9","CrawlerBot"
"gazz/5.0 (gazz@nttr.co.jp)","CrawlerBot"
"Generic","CrawlerBot"
"GeonaBot 1.0; http://www.geona.com/","CrawlerBot"
"GeonaBot/1.1; http://www.geona.com/","CrawlerBot"
"GetRight/4.5e","CrawlerBot"
"Gigabot/1.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)","CrawlerBot"
"Goldfire Server","CrawlerBot"
"Googlebot (+http://www.google.com/bot.html)","CrawlerBot"
"GoogleBot/2.1","CrawlerBot"
"Googlebot/2.1 (+http://www.google.com/bot.html)","CrawlerBot"
"googlebot/2.1 (+http://www.googlebot.com/bot.html","CrawlerBot"
"Googlebot/2.1 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Googlebot/2.1 (+http://www.googlebot.com/bot.html) (compatible; MSIE 6.0; )","CrawlerBot"
"Googlebot/2.1 (compatible; MSIE; Windows)","CrawlerBot"
"googlebot/2.1; +http://www.google.com/bot.html","CrawlerBot"
"Googlebot/2.1+(+http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.)","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Googlebot/Test (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Googlebot-Image/1.0","CrawlerBot"
"Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html","CrawlerBot"
"Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Green Research, Inc.","CrawlerBot"
"GregBot (compatible; MSIE; Windows; Q312461)","CrawlerBot"
"grub crawler","CrawlerBot"
"grub crawler(http://www.grub.org)","CrawlerBot"
"grub-client","CrawlerBot"
"Mozilla/4.0 (compatible; Grubclient; windows; SV1; .NET CLR 1.1.4322)","CrawlerBot"
"Mozilla/4.0 (compatible; Grubclient-2.2-internal-beta)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-2.6.0)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.07; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-2.3)","CrawlerBot"
"gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com)","CrawlerBot"
"Crawler [en] (compatible; Crawler Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)","CrawlerBot"
"Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Roadrunner; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SC/5.60/1.01/FS-Internett; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; stokeybot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts-MyWay; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot/2.1 (+http://www.googlebot.com/bot.html))","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot/2.1 (+http://www.googlebot.com/bot.html); Maxthon; FDM)","CrawlerBot"
"Harvest-NG/1.0.2","CrawlerBot"
"Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)","CrawlerBot"
"Hatena Antenna/0.4 (http://a.hatena.ne.jp/help)","CrawlerBot"
"hget/0.3","CrawlerBot"
"Hitwise Spider v1.0 http://www.hitwise.com","CrawlerBot"
"htdig","CrawlerBot"
"htdig/3.1.5 (admin@ipc-opc.lan)","CrawlerBot"
"htdig/3.1.5 (unconfigured@htdig.searchengine.maintainer)","CrawlerBot"
"htdig/3.1.6 (http://computerorgs.com)","CrawlerBot"
"Html Link Validator (www.lithopssoft.com)","CrawlerBot"
"Httpcheck/1.0 (Perl 5.006001)","CrawlerBot"
"HTTPConnect","CrawlerBot"
"httpget-5.2.2","CrawlerBot"
"Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)","CrawlerBot"
"ia_archiver","CrawlerBot"
"lcabotAccept: */*","CrawlerBot"
"ichiro/1.0 (ichiro@nttr.co.jp)","CrawlerBot"
"IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)","CrawlerBot"
"Mozilla/4.0 (compatible; ICS 1.2.105)","CrawlerBot"
"Iltrovatore-Setaccio","CrawlerBot"
"IlTrovatore-Setaccio (+http://www.iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/0.03-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (+http://www.iltrovatore.it/aiuto/faq.html)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (Indexing; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"Iltrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"imagefetch/0.1 libwww-perl/5.66","CrawlerBot"
"Mozilla/3.0 (compatible; Indy Library)","CrawlerBot"
"InelaBot/0.2 (+http://inelegant.org/bot)","CrawlerBot"
"InfoSeek Sidewinder/1.0A","CrawlerBot"
"Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX)","CrawlerBot"
"Infoseek SideWinder/2.0B (Linux 2.4 i686)","CrawlerBot"
"Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)","CrawlerBot"
"Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Slurp/si-emb (slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"InternetLinkAgent/3.1","CrawlerBot"
"IPiumBot laurion(dot)com","CrawlerBot"
"IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler)","CrawlerBot"
"http://www.istarthere.com (spider@istarthere.com)","CrawlerBot"
"Java1.4.0","CrawlerBot"
"JoBo/1.3 (http://www.matuschek.net/jobo.html)","CrawlerBot"
"k2spider","CrawlerBot"
"KMcrawler","CrawlerBot"
"Knowledge.com/0.2","CrawlerBot"
"Knowledge.com/0.3","CrawlerBot"
"Knowledge Engine","CrawlerBot"
"kuloko-bot/0.2","CrawlerBot"
"Larbin (larbin2.6.2@unspecified.mail)","CrawlerBot"
"larbin (samualt9@bigfoot.com)","CrawlerBot"
"larbin samualt9@bigfoot.com","CrawlerBot"
"larbin_extended (larbin@oktie.com)","CrawlerBot"
"larbin_test (nobody@airmail.etn)","CrawlerBot"
"LARBIN-EXPERIMENTAL (efp@gmx.net)","CrawlerBot"
"LARBIN-EXPERIMENTAL efp@gmx.net","CrawlerBot"
"Mozilla (la2@unspecified.mail)","CrawlerBot"
"Mozilla la2@unspecified.mail","CrawlerBot"
"Mozilla/4.0 (efp@gmx.net)","CrawlerBot"
"Mozilla/4.0 efp@gmx.net","CrawlerBot"
"MSIE-5.13 (larbin@unspecified.mail)","CrawlerBot"
"MSIE-5.13 larbin@unspecified.mail","CrawlerBot"
"SearchGuild_DMOZ_Experiment (chris@searchguild.com)","CrawlerBot"
"SearchGuild_DMOZ_Experiment chris@searchguild.com","CrawlerBot"
"WinampMPEG/2.00 (larbin@unspecified.mail)","CrawlerBot"
"WinampMPEG/2.00 larbin@unspecified.mail","CrawlerBot"
"Larbin larbin2.6.2@unspecified.mail","CrawlerBot"
"larbin_2.6.2 (kalou@kalou.net)","CrawlerBot"
"larbin_2.6.2 (larbin@correa.org)","CrawlerBot"
"larbin_2.6.2 (larbin2.6.2@unspecified.mail)","CrawlerBot"
"larbin_2.6.2 (pimenas@softnet.tuc.gr)","CrawlerBot"
"larbin_2.6.2 (pimenas@systems.tuc.gr)","CrawlerBot"
"larbin_2.6.2 (sumeet_sobti@yahoo.com)","CrawlerBot"
"larbin_2.6.2 (vitalbox1@hotmail.com)","CrawlerBot"
"larbin_2.6.2 (vshelk@yahoo.com)","CrawlerBot"
"larbin_2.6.2 larbin@correa.org","CrawlerBot"
"larbin_2.6.2 larbin2.6.2@unspecified.mail","CrawlerBot"
"larbin_2.6.2 pimenas@systems.tuc.gr","CrawlerBot"
"larbin_2.6.2 sumeet_sobti@yahoo.com","CrawlerBot"
"larbin_2.6.2 vitalbox1@hotmail.com","CrawlerBot"
"larbin_2.6.3 (andreas.beder@chello.at)","CrawlerBot"
"larbin_2.6.3 (larbin2.6.3@unspecified.mail)","CrawlerBot"
"larbin_2.6.3 (larbin-crawler@un.bewaff.net)","CrawlerBot"
"larbin_2.6.3 (pimenas@softnet.tuc.gr)","CrawlerBot"
"larbin_2.6.3 larbin2.6.3@unspecified.mail","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (Tomi.Silander@hiit.fi)","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (tsilande@hiit.fi)","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) Tomi.Silander@hiit.fi","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) tsilande@hiit.fi","CrawlerBot"
"eseek-crawler-larbin-2.63 (crawler@exactseek.com)","CrawlerBot"
"eseek-crawler-larbin-2.63 crawler@exactseek.com","CrawlerBot"
"libwww-MGET/1.0 libwww/5.2.8","CrawlerBot"
"Perl-Win32::Internet/0.082","CrawlerBot"
"/ libwww/5.3.2","CrawlerBot"
"/ libwww/5.4.0","CrawlerBot"
"libwww-perl/5.48","CrawlerBot"
"libwww-perl/5.50","CrawlerBot"
"libwww-perl/5.51","CrawlerBot"
"libwww-perl/5.52 FP/4.0","CrawlerBot"
"libwww-perl/5.53","CrawlerBot"
"libwww-perl/5.63","CrawlerBot"
"libwww-perl/5.64","CrawlerBot"
"libwww-perl/5.65","CrawlerBot"
"MyApp/0.1 libwww-perl/5.65","CrawlerBot"
"rawiswar/0.1 libwww-perl/5.66","CrawlerBot"
"libwww-perl/5.68","CrawlerBot"
"libwww-perl/5.69","CrawlerBot"
"VanillaZilla/0.1 libwww-perl/5.69","CrawlerBot"
"libwww-perl/5.74","CrawlerBot"
"libwww-perl/5.75","CrawlerBot"
"libwww-perl/5.76","CrawlerBot"
"libwww-perl/5.800","CrawlerBot"
"libwww-perl/5.801","CrawlerBot"
"libwww-perl/5.802","CrawlerBot"
"libwww-perl/5.803","CrawlerBot"
"LimeBot/1.0 (+www.cruiselime.com/LimeBot.php)","CrawlerBot"
"Linkbot 3.0","CrawlerBot"
"LinkLint-checkonly/2.3.5","CrawlerBot"
"Linknzbot/ (+http://www.linknz.co.nz/robot.php)","CrawlerBot"
"Linknzbot 2004/(+http://www.linknz.co.nz/robot.php)","CrawlerBot"
"Links SQL (http://gossamer-threads.com/scripts/links-sql/)","CrawlerBot"
"Lite Bot 0616B","CrawlerBot"
"LNSpiderguy","CrawlerBot"
"Look.com","CrawlerBot"
"lwp-trivial/1.29","CrawlerBot"
"lwp-trivial/1.35","CrawlerBot"
"lwp-trivial/1.36","CrawlerBot"
"lwp-request/2.01","CrawlerBot"
"LWP::Simple/5.48","CrawlerBot"
"LWP::Simple/5.65","CrawlerBot"
"Lycos_Spider_(modspider)","CrawlerBot"
"Mediapartners-Google/2.1","CrawlerBot"
"Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Mercator-2.0","CrawlerBot"
"metacarta (crawler@metacarta.com)","CrawlerBot"
"metacarta crawler@metacarta.com","CrawlerBot"
"MetaGer-LinkChecker","CrawlerBot"
"Microsoft URL Control - 5.00.3609","CrawlerBot"
"Microsoft URL Control - 5.01.4319","CrawlerBot"
"Microsoft URL Control - 6.00.8169","CrawlerBot"
"Microsoft URL Control - 6.00.8862","CrawlerBot"
"Microsoft-ATL-Native/7.00","CrawlerBot"
"MicrosoftPrototypeCrawler (How''s my crawling? mailto:newbiecrawler@hotmail.com)","CrawlerBot"
"moget/1.0 (moget@goo.ne.jp)","CrawlerBot"
"moget/2.1 (moget@goo.ne.jp)","CrawlerBot"
"mozDex/0.04-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"Mozilla/4.0 (compatible; Netcraft Web Server Survey)","CrawlerBot"
"Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com)","CrawlerBot"
"Mozilla/4.0 (stat 0.12) (statbot@gmail.com)","CrawlerBot"
"Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;) Unchaos/Crawler","CrawlerBot"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20041027 NaverBot/0.9.3","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040206 GoogleBot/1.1","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040730 Googlebot/2.1/2.1","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040707 Lightningspider/0.9.2","CrawlerBot"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040805 Googlebot/2.1","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider Cache Manager","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider DAV","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider DAV 1.1","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider Protocol Discovery","CrawlerBot"
"Mozilla/2.0 (compatible; MS FrontPage 4.0)","CrawlerBot"
"MSFrontPage/4.0","CrawlerBot"
"Mozilla/2.0 (compatible; MS FrontPage 5.0)","CrawlerBot"
"MSFrontPage/5.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; T-Online Internatinal AG; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; DT; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; QXW0338t; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; FunWebProducts; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; .NET CLR 1.0.3705; Googlebot; .NET CLR 1.1.4322; .NET CLR 1.0.3705; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows compatible LesnikBot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; 3COM U.S. Robotics)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; BOTW)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; http://www.pregnancycrawler.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; Googlebot; .NET CLR 1.1.4322)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; AskBar 3.00; .NET CLR 1.1.4322; Fluffi Bot+)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; CDSource=v9e.03; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 3.1; Googlebot/2.1)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.40607; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; DigExt; FunWebProducts; Media Center PC 3.0; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows XP Professional Bot v.5.)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; Q312461; BTopenworld; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; matlas-2.0.2501; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)","CrawlerBot"
"MSNBOT/0.1 (http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/0.11 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/0.3 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/1.0 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"MSProxy/2.0","CrawlerBot"
"MSRBOT/0.1 (http://research.microsoft.com/research/sv/msrbot/)","CrawlerBot"
"Mozilla/3.01 (compatible;)","CrawlerBot"
"NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"NaverBot_dloader/1.5","CrawlerBot"
"dloader(NaverRobot)/1.0","CrawlerBot"
"dloader(NaverRobot)/1.5","CrawlerBot"
"NetAnts/1.25","CrawlerBot"
"NetNoseCrawler/v1.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com)","CrawlerBot"
"NetResearchServer(http://www.look.com)","CrawlerBot"
"NetResearchServer/2.4(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.5(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.7(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.8(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.9(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/3.4(loopimprovements.com/robot.html)","CrawlerBot"
"NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot)","CrawlerBot"
"NG/1.0","CrawlerBot"
"NPBot","CrawlerBot"
"NPBot (http://www.nameprotect.com/botinfo.html)","CrawlerBot"
"NPBot-1/2.0","CrawlerBot"
"NPBot-1/2.0 (http://www.nameprotect.com/botinfo.html)","CrawlerBot"
"NuSearch Spider www.nusearch.com","CrawlerBot"
"NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"NutchCVS/0.05-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"CreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"Robot: NutchCrawler, Owner: wdavies@acm.org","CrawlerBot"
"NutchOrg/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)","CrawlerBot"
"oBot","CrawlerBot"
"Ocelli/1.3 (http://www.globalspec.com/Ocelli)","CrawlerBot"
"OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Internet Categorizer","CrawlerBot"
"OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Job Crawler","CrawlerBot"
"Openbot/3.0+(robot@monkia.com.tw;+http://gais.cs.ccu.edu.tw/robot.php)","CrawlerBot"
"Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)","CrawlerBot"
"Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)","CrawlerBot"
"OrangeBot","CrawlerBot"
"Mozilla/4.0 (compatible; Advanced Email Extractor v2.24)","CrawlerBot"
"Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"OWR_Crawler 0.1","CrawlerBot"
"parabot (paracite@ecs.soton.ac.uk)","CrawlerBot"
"Patwebbot (http://www.herz-power.de/technik.html)","CrawlerBot"
"pavuk/0.9pl28 i586-pc-cygwin","CrawlerBot"
"pavuk/0.9pl29b i686-pc-linux-gnu","CrawlerBot"
"PEERbot www.peerbot.com","CrawlerBot"
"pipeLiner/0.3a (PipeLine Spider; http://www.pipeline-search.com/webmaster.html; webmaster@pipeline-search.com)","CrawlerBot"
"http://www.planethosting.com","CrawlerBot"
"polybot 1.0 (http://cis.poly.edu/polybot/)","CrawlerBot"
"Pompos/1.1 http://pompos.iliad.fr","CrawlerBot"
"Pompos/1.2 http://pompos.iliad.fr","CrawlerBot"
"Pompos/1.3 http://dir.com/pompos.html","CrawlerBot"
"Portal Manager 0.7","CrawlerBot"
"potbot 1.0","CrawlerBot"
"Program Shareware 1.0.3","CrawlerBot"
"ProWebGuide Link Checker (http://www.prowebguide.com)","CrawlerBot"
"psbot/0.1 (+http://www.picsearch.com/bot.html)","CrawlerBot"
"pverify/1.2","CrawlerBot"
"PWS.Kiosk - Content Filtering","CrawlerBot"
"QPCreep Test Rig ( We are not indexing, just testing )","CrawlerBot"
"QuepasaCreep ( crawler@quepasacorp.com )","CrawlerBot"
"QuepasaCreep v0.9.14","CrawlerBot"
"QuepasaCreep v0.9.13","CrawlerBot"
"reifier.org (admin@reifier.org)","CrawlerBot"
"reifier.org admin@reifier.org","CrawlerBot"
"rico/0.1","CrawlerBot"
"RixBot (http://www.oops-as.no/rix/)","CrawlerBot"
"RoboPal (http://www.findpal.com/)","CrawlerBot"
"RobotMidareru/0.7libwww-perl/5.65","CrawlerBot"
"Search Engine World Robots.txt Validator at http://www.searchengineworld.com/cgi-bin/robotcheck.cgi","CrawlerBot"
"Robozilla/1.0","CrawlerBot"
"RPT-HTTPClient/0.3-3","CrawlerBot"
"SafariBookmarkChecker/1.25 (+http://www.coriolis.ch/)","CrawlerBot"
"SafariBookmarkChecker/1.26 (+http://www.coriolis.ch/)","CrawlerBot"
"Scooter/1.0","CrawlerBot"
"Scooter-ARS-1.1","CrawlerBot"
"Scooter-3.0.FS - Altavista.com","CrawlerBot"
"Scooter/3.2","CrawlerBot"
"Scooter/3.2.SF0","CrawlerBot"
"Scooter_x0-3.2.EX","CrawlerBot"
"Scooter-3.2","CrawlerBot"
"Scooter-3.2.BT","CrawlerBot"
"Scooter-3.2.EX","CrawlerBot"
"Scooter-3.2.FNR","CrawlerBot"
"Scooter-3.2.PDF","CrawlerBot"
"Scooter-3.2.SF0","CrawlerBot"
"Scooter-3.2.TX.FNR","CrawlerBot"
"Scooter-3.2.XX0","CrawlerBot"
"Scooter/3.3","CrawlerBot"
"Scooter/3.3.QA","CrawlerBot"
"Scooter/3.3.QA.pczukor","CrawlerBot"
"Scooter/3.3.vscooter","CrawlerBot"
"Scooter/3.3_SF","CrawlerBot"
"Scrubby/2.1 (http://www.scrubtheweb.com/abs/meta-check.html)","CrawlerBot"
"Scrubby/2.2 (http://www.scrubtheweb.com/)","CrawlerBot"
"Search Agent 1.0","CrawlerBot"
"SearchSpider.com/1.1","CrawlerBot"
"Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3","CrawlerBot"
"Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)","CrawlerBot"
"semanticdiscovery/0.1","CrawlerBot"
"Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au)","CrawlerBot"
"sherlock/1.3 httpget/1.3","CrawlerBot"
"sherlock_spider (jimfan@163.com)","CrawlerBot"
"InternetSeer.com","CrawlerBot"
"sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com)","CrawlerBot"
"sitescooper/3.1.2 (http://sitescooper.org) libwww-perl/5.51","CrawlerBot"
"SiteXpert","CrawlerBot"
"SlySearch/1.3 (http://www.slysearch.com)","CrawlerBot"
"SlySearch/1.3 http://www.slysearch.com","CrawlerBot"
"sohu-search","CrawlerBot"
"Speedy Spider (http://www.entireweb.com)","CrawlerBot"
"Speedy_Spider_(http://www.entireweb.com)","CrawlerBot"
"Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)","CrawlerBot"
"SpiderKU/0.9","CrawlerBot"
"SpiderMonkey/7.04 (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)","CrawlerBot"
"Spider_Monkey/7.06 (SpiderMonkey.ca info at http://SpiderMonkey.ca /sm.shtml)","CrawlerBot"
"Spider_Monkey/7.06 (SpiderMonkey.ca info at http://www.spidermonkey.ca/sm.shtml)","CrawlerBot"
"Mozilla/5.0 (compatible; SpurlBot/0.2)","CrawlerBot"
"Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)","CrawlerBot"
"Star Downloader","CrawlerBot"
"Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)","CrawlerBot"
"Mozilla/4.0 (compatible; SuperCleaner 2.56; Windows NT 5.1)","CrawlerBot"
"Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;)","CrawlerBot"
"Szukacz/1.5","CrawlerBot"
"Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)","CrawlerBot"
"Tarantula Experimental Crawler","CrawlerBot"
"Tcl http client package 1.0","CrawlerBot"
"Tcl http client package 2.3","CrawlerBot"
"(Teradex Mapper; mapper@teradex.com; http://www.teradex.com)","CrawlerBot"
"Teradex_Crawler (crawler@teradex.com; http://crawler.teradex.com)","CrawlerBot"
"TheSuBot/0.1 (www.thesubot.de)","CrawlerBot"
"thesubot-beta-www.thesubot.de","CrawlerBot"
"thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)","CrawlerBot"
"timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html","CrawlerBot"
"Tkensaku/0.9 (http://www.tkensaku.com/q.html)","CrawlerBot"
"TranSGeniKBot (http://www.tsgk.net)","CrawlerBot"
"TranSGeniKBot http://www.tsgk.net","CrawlerBot"
"TulipChain/5.7 (http://ostermiller.org/tulipchain/) Java/1.4.0_02 (http://java.sun.com/) Windows_Me/4.90","CrawlerBot"
"TulipChain/5.94 (http://ostermiller.org/tulipchain/) Java/1.4.1_01 (http://apple.com/) Mac_OS_X/10.2.8","CrawlerBot"
"TulipChain/6.01 (http://ostermiller.org/tulipchain/) Java/1.4.2_03 (http://java.sun.com/) Windows_XP/5.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"TulipChain/6.02 (http://ostermiller.org/tulipchain/) Java/1.4.2_03 (http://apple.com/) Mac_OS_X/10.3.3 RPT-HTTPClient/0.3-3","CrawlerBot"
"TulipChain/6.03 (http://ostermiller.org/tulipchain/) Java/1.4.2_05 (http://java.sun.com/) Windows_XP/5.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"TurnitinBot/1.4 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/1.4 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TurnitinBot/2.0 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TutorGigBot/1.5 ( +http://www.tutorgig.info )","CrawlerBot"
"Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)","CrawlerBot"
"UdmSearch/3.1.20","CrawlerBot"
"UIowaCrawler/1.0","CrawlerBot"
"UIowaCrawler/2.0","CrawlerBot"
"unchaos_crawler_2.0.2 (search.engine@unchaos.com)","CrawlerBot"
"VM4050/132.037 UP.Browser/6.2.2.4.e.1.100 (GUI) MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om)","CrawlerBot"
"USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)","CrawlerBot"
"Vagabondo/2.0 MT (webagent at wise-guys dot nl)","CrawlerBot"
"Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)","CrawlerBot"
"Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)","CrawlerBot"
"Vivante Link Checker (http://www.vivante.com)","CrawlerBot"
"void-bot/0.1 (bot@void.be; http://www.void.be/)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6","CrawlerBot"
"Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2","CrawlerBot"
"VSE/1.0 (vsecrawler@hotmail.com)","CrawlerBot"
"vspider","CrawlerBot"
"W3C_Validator/1.183 libwww-perl/5.64","CrawlerBot"
"W3C_Validator/1.305.2.109 libwww-perl/5.79","CrawlerBot"
"W3C_Validator/1.305.2.12 libwww-perl/5.64","CrawlerBot"
"W3C_Validator/1.305.2.137 libwww-perl/5.79","CrawlerBot"
"W3C_Validator/1.305.2.148 libwww-perl/5.800","CrawlerBot"
"W3C_Validator/1.305.2.148 libwww-perl/5.803","CrawlerBot"
"W3C-checklink/2.90 libwww-perl/5.64","CrawlerBot"
"W3C-checklink/3.6.2.3 libwww-perl/5.64","CrawlerBot"
"W3C-checklink/3.9.2 [3.17] libwww-perl/5.79","CrawlerBot"
"W3C-checklink/4.0 [4.4] libwww-perl/5.800","CrawlerBot"
"W3C-checklink/4.1 [4.14] libwww-perl/5.800","CrawlerBot"
"webbot","CrawlerBot"
"Webclipping.com","CrawlerBot"
"webcollage/1.102","CrawlerBot"
"webcollage/1.104","CrawlerBot"
"webcollage/1.87","CrawlerBot"
"webcollage/1.93","CrawlerBot"
"webcollage/1.94","CrawlerBot"
"Thu Mar 27 18:20:34 CET 2003WebcraftBoot","CrawlerBot"
"Fri Nov 15 04:51:18 EST 2002WebcraftBoot Java/1.4.1_01","CrawlerBot"
"Sun Apr 20 22:00:01 EDT 2003WebcraftBoot Java/1.4.2-beta","CrawlerBot"
"Tue Apr 15 22:00:03 EDT 2003WebcraftBoot Java/1.4.2-beta","CrawlerBot"
"WebFilter Robot 1.0","CrawlerBot"
"Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56)","CrawlerBot"
"WebRACE/1.1 (University of Cyprus, Distributed Crawler)","CrawlerBot"
"WebSauger 1.20b","CrawlerBot"
"http://www.websearch.com.au (larbin2.6.2@unspecified.mail)","CrawlerBot"
"http://www.websearch.com.au larbin2.6.2@unspecified.mail","CrawlerBot"
"http://www.WebSearch.com.au/ (larbin2.6.2@unspecified.mail)","CrawlerBot"
"http://www.WebSearch.com.au/ larbin2.6.2@unspecified.mail","CrawlerBot"
"www.WebSearch.com.au (search@websearch.com.au)","CrawlerBot"
"www.WebSearch.com.au search@websearch.com.au","CrawlerBot"
"WebSearch/2.0.1 (Dez@Blanchfield.COM.AU, http://www.WebSearch.com.au/)","CrawlerBot"
"WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU)","CrawlerBot"
"http://www.WebSearch.com.au/ - Australian Search Engine/3.1.3 (sites@websearch.com.au)","CrawlerBot"
"http://www.WebSearch.com.au/ - Australian Search Engine/3.1.6 (sites@websearch.com.au)","CrawlerBot"
"www.webwombat.com.au","CrawlerBot"
"webyield robot (http://www.webyield.net/search/search.pl)","CrawlerBot"
"Wget/1.5.2","CrawlerBot"
"Wget/1.5.3","CrawlerBot"
"Wget/1.5.3.1","CrawlerBot"
"Wget/1.6","CrawlerBot"
"Wget/1.7","CrawlerBot"
"Wget/1.8","CrawlerBot"
"Wget/1.8.1","CrawlerBot"
"Wget/1.8.1+cvs","CrawlerBot"
"Wget/1.8.2","CrawlerBot"
"Wget/1.9","CrawlerBot"
"Wget/1.9-beta","CrawlerBot"
"Wget/1.9.1","CrawlerBot"
"Willow Internet Crawler by Twotrees V2.1","CrawlerBot"
"Wotbox/alpha0.5.1 (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02","CrawlerBot"
"http://www.ciml.co.uk","CrawlerBot"
"WWWeasel Robot v1.00 (http://wwweasel.de)","CrawlerBot"
"Xenu''s Link Sleuth 1.1a","CrawlerBot"
"Xenu Link Sleuth 1.2b","CrawlerBot"
"Xenu Link Sleuth 1.2d","CrawlerBot"
"Xenu Link Sleuth 1.2e","CrawlerBot"
"Xenu Link Sleuth 1.2f","CrawlerBot"
"Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)","CrawlerBot"
"Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)","CrawlerBot"
"YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine","CrawlerBot"
"Zao/0.1 (http://www.kototoi.org/zao/)","CrawlerBot"
"Zao/0.2 (http://www.kototoi.org/zao/)","CrawlerBot"
"Zao-Crawler","CrawlerBot"
"Zeus 3140 Webster Pro V2.9 Win32","CrawlerBot"
"Zeus 57657 Webster Pro V2.9 Win32","CrawlerBot"
"ZipppBot/0.11 (ZipppBot; http://www.zippp.net; webmaster@zippp.net)","CrawlerBot"
"ZoomSpider - wrensoft.com","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-1.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-12.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-2.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d03 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d05 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker Beta-d01 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 DLC (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 12

Expert Comment

by:Sinoj Sebastian
ID: 18754309
> shortest and most effective list wins
So the list of all popular bot agent strings

    *Bot*

Most of the spider user-agent strings will have the substring "Bot" in it.
Ordinary user-agent string from IE, FF, NS etc do not contain "Bot".
Try filter using this.
I'm already using it.

From the above list I found That "CrawlerBot" is also a substring of bot agent strings

0
 
LVL 29

Expert Comment

by:rdivilbiss
ID: 18757241
No, in the list above, "CrawlerBot" is not part of the user agent string, it was a field in the database of my collection of live user agents taken from dozens of my web sites and hand categorized.  Ignore: ,"CrawlerBot"
0
 
LVL 16

Author Comment

by:OliWarner
ID: 18757392
Yeah I can parse those out without issue. Thanks Rod that looks like it'll do the job perfectly
0
 
LVL 29

Expert Comment

by:rdivilbiss
ID: 18758475
Those were as of January. There could always be a few new ones cropping up.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Export & Import Bookmarks in FIrefox 1 31
Set cookies HttpOnly and Secure 4 62
razorCMS: Change Menu Font 4 27
WIX Redirect 1 7
#Citrix #Internet Explorer #Enterprise Mode #IE 11 #IE 8
Today, still in the boom of Apple, PC's and products, nearly 50% of the computer users use Windows as graphical operating systems. If you are among those users who love windows, but are grappling to keep the system's hard drive optimized, then you s…
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now