Solved

User-agent strings

Posted on 2007-03-19
8
7,163 Views
Last Modified: 2013-12-09
I need a list of the most popular spider user-agent strings. I've got several items on my website that log or increment things and I really only want some of those to be logging if the thing hitting the page is a real person and not a bot. So I'm left checking the user-agents.

I can either grab the most popular browsers or the most popular bots... Whatever is most efficient -- you decide!

Either way, time-complexity is an issue as it is a fairly busy site, so the shortest and most effective list wins =)
0
Comment
Question by:OliWarner
8 Comments
 
LVL 2

Expert Comment

by:fpintos
ID: 18753824
Have you tried setting up robot.txt to block these spiders? This is by far the simplest way.
0
 
LVL 16

Author Comment

by:OliWarner
ID: 18753862
I don't want to block them from viewing the pages -- just stop my logging script counting hits from them.
0
 
LVL 29

Expert Comment

by:rdivilbiss
ID: 18753977
I've got them in a db, Oli.  Give me a minute to extract the bots from the browsers.
0
MIM Survival Guide for Service Desk Managers

Major incidents can send mastered service desk processes into disorder. Systems and tools produce the data needed to resolve these incidents, but your challenge is getting that information to the right people fast. Check out the Survival Guide and begin bringing order to chaos.

 
LVL 29

Accepted Solution

by:
rdivilbiss earned 500 total points
ID: 18754003
http://www.rodsdot.com/downloads/bots.zip

"ADSAComponent (postmaster@cnds.ucd.ie)","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [fc3]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [c01]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [wf224]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [wf55]","CrawlerBot"
"Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)","CrawlerBot"
"Mozilla/4.0 (Search Engine Marketing Tactics Amsterdam 2002 Information Spider)","CrawlerBot"
"AnswerBus (http://www.answerbus.com/)","CrawlerBot"
"antibot-V1.1.11/i586-linux-2.2","CrawlerBot"
"antibot-V1.1.13/i586-linux-2.2","CrawlerBot"
"antibot-V1.2.0/redhat-linux-9","CrawlerBot"
"AOLserver-Tcl/3.5.6","CrawlerBot"
"AOL 8.0 (compatible; AOL 8.0; DOS; .NET CLR 1.1.4322)","CrawlerBot"
"appie 1.1 (www.walhello.com)","CrawlerBot"
"Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)","CrawlerBot"
"Art-Online.com 0.9(Beta)","CrawlerBot"
"Mozilla/2.0 (compatible; Ask Jeeves)","CrawlerBot"
"Mozilla/2.0 (compatible; Ask Jeeves/Teoma)","CrawlerBot"
"ASPseek/1.2.10","CrawlerBot"
"ASPseek/1.2.11","CrawlerBot"
"ASPseek/1.2.12","CrawlerBot"
"Mozilla/3.0 (compatible; AvantGo 3.2)","CrawlerBot"
"BaiDuSpider","CrawlerBot"
"Baiduspider+(+http://www.baidu.com/search/spider.htm)","CrawlerBot"
"battlebot","CrawlerBot"
"BDFetch","CrawlerBot"
"BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html)","CrawlerBot"
"BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"Big Brother (http://pauillac.inria.fr/~fpottier/)","CrawlerBot"
"BlogBot/1.2","CrawlerBot"
"boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-dc/0.5 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-dc/0.66 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-robot/1.0","CrawlerBot"
"boitho.com-robot/1.1","CrawlerBot"
"Mozilla/4.0 (compatible; BorderManager 3.0)","CrawlerBot"
"BrailleBot 1.0","CrawlerBot"
"BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html)","CrawlerBot"
"bumblebee/1.0 (bumblebee@relevare.com; http://www.relevare.com/)","CrawlerBot"
"Computer_and_Automation_Research_Institute_Crawler (nospamspidernospam@spider.ilab.sztakinospam.hunospam)","CrawlerBot"
"Computer_and_Automation_Research_Institute_Crawler (spider@spider.ilab.sztaki.hu)","CrawlerBot"
"cd34/0.1","CrawlerBot"
"CerberianDrtrs/Version-3.0-Release-24","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-40)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-11)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-12)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-13)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-16)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-17)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-41)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-43)","CrawlerBot"
"CipinetBot (http://www.cipinet.com/bot.html)","CrawlerBot"
"Clushbot/2.1 (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.21-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.23-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.24-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.6-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.9-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"ComMOOnity LambdaMOO/1.8.1","CrawlerBot"
"CrawlConvera0.1 (CrawlConvera@yahoo.com)","CrawlerBot"
"CrawlConvera0.1 (www.authoritativeweb.com)","CrawlerBot"
"ConveraCrawler/0.2","CrawlerBot"
"ConveraCrawler/0.5 (+http://www","CrawlerBot"
"cosmos/0.9_(robot@xyleme.com)","CrawlerBot"
"Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"Cowbot-0.1.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"Crawl_Application","CrawlerBot"
"CrocCrawler v3.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"CrocCrawler v4.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"Custo 2.0 (www.netwu.com)","CrawlerBot"
"CydralSpider/1.9 (Cydral Web Image Search; http://www.cydral.com)","CrawlerBot"
"DeepIndex (http://www.deepindex.com)","CrawlerBot"
"DeMozulator 1.0 (MacOS, dMoz URL Check Agent, trebor@animeigo.com)","CrawlerBot"
"DoCoMo/1.0/N504i/c10/TB","CrawlerBot"
"DoCoMo/1.0/P504iS/c10/TB","CrawlerBot"
"Dual Proxy","CrawlerBot"
"Dumbot(version 0.1 beta - dumbfind.com)","CrawlerBot"
"Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html)","CrawlerBot"
"Dumbot(version 0.1 beta)","CrawlerBot"
"EARTHCOM.info/1.2","CrawlerBot"
"EmailSiphon","CrawlerBot"
"Enterprise_Search/1.00.136;MSSQL (http://www.innerprise.net/es-spider.asp)","CrawlerBot"
"e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)","CrawlerBot"
"exactseek-crawler-2.63 (crawler@exactseek.com)","CrawlerBot"
"exactseek-crawler-2.63 crawler@exactseek.com","CrawlerBot"
"exactseek-crawler-2.63-5 (crawler@exactseek.com)","CrawlerBot"
"exactseek-crawler-2.63-5 crawler@exactseek.com","CrawlerBot"
"Explorer 6","CrawlerBot"
"FAST Enterprise Crawler/6 (crawler@fast.no)","CrawlerBot"
"FAST Enterprise Crawler/6 (www.fastsearch.com)","CrawlerBot"
"FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)","CrawlerBot"
"FastBug http://www.ay-up.com","CrawlerBot"
"FAST-WebCrawler/3.2 test","CrawlerBot"
"FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)","CrawlerBot"
"FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)","CrawlerBot"
"FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://www.alltheweb.com/help/webmaster/crawler)","CrawlerBot"
"FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler)","CrawlerBot"
"FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.x Multimedia","CrawlerBot"
"FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)","CrawlerBot"
"favicon finder at http://iconsurf.com/","CrawlerBot"
"favicon monitor at http://iconsurf.com/","CrawlerBot"
"Mozilla/4.0 (compatible: FDSE robot)","CrawlerBot"
"Filangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)","CrawlerBot"
"Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com)","CrawlerBot"
"Filangy/1.01 (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)","CrawlerBot"
"FindLinks/0.71 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.82 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.87 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.89 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)","CrawlerBot"
"Flickbot 1.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"FlickBot 2.0 RPT-HTTPClient/0.3-3","CrawlerBot"
"Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; http://www.pgts.com.au/; +http://www.galaxy.com/info/crawler.html)","CrawlerBot"
"FyberSpider (+http://www.fybersearch.com/fyberspider.php)","CrawlerBot"
"GAIS Robot/1.1A2","CrawlerBot"
"Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)","CrawlerBot"
"GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)","CrawlerBot"
"gatherer/0.9","CrawlerBot"
"gazz/5.0 (gazz@nttr.co.jp)","CrawlerBot"
"Generic","CrawlerBot"
"GeonaBot 1.0; http://www.geona.com/","CrawlerBot"
"GeonaBot/1.1; http://www.geona.com/","CrawlerBot"
"GetRight/4.5e","CrawlerBot"
"Gigabot/1.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)","CrawlerBot"
"Goldfire Server","CrawlerBot"
"Googlebot (+http://www.google.com/bot.html)","CrawlerBot"
"GoogleBot/2.1","CrawlerBot"
"Googlebot/2.1 (+http://www.google.com/bot.html)","CrawlerBot"
"googlebot/2.1 (+http://www.googlebot.com/bot.html","CrawlerBot"
"Googlebot/2.1 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Googlebot/2.1 (+http://www.googlebot.com/bot.html) (compatible; MSIE 6.0; )","CrawlerBot"
"Googlebot/2.1 (compatible; MSIE; Windows)","CrawlerBot"
"googlebot/2.1; +http://www.google.com/bot.html","CrawlerBot"
"Googlebot/2.1+(+http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.)","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Googlebot/Test (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Googlebot-Image/1.0","CrawlerBot"
"Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html","CrawlerBot"
"Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Green Research, Inc.","CrawlerBot"
"GregBot (compatible; MSIE; Windows; Q312461)","CrawlerBot"
"grub crawler","CrawlerBot"
"grub crawler(http://www.grub.org)","CrawlerBot"
"grub-client","CrawlerBot"
"Mozilla/4.0 (compatible; Grubclient; windows; SV1; .NET CLR 1.1.4322)","CrawlerBot"
"Mozilla/4.0 (compatible; Grubclient-2.2-internal-beta)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-2.6.0)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.07; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-2.3)","CrawlerBot"
"gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com)","CrawlerBot"
"Crawler [en] (compatible; Crawler Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)","CrawlerBot"
"Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Roadrunner; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SC/5.60/1.01/FS-Internett; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; stokeybot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts-MyWay; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot/2.1 (+http://www.googlebot.com/bot.html))","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot/2.1 (+http://www.googlebot.com/bot.html); Maxthon; FDM)","CrawlerBot"
"Harvest-NG/1.0.2","CrawlerBot"
"Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)","CrawlerBot"
"Hatena Antenna/0.4 (http://a.hatena.ne.jp/help)","CrawlerBot"
"hget/0.3","CrawlerBot"
"Hitwise Spider v1.0 http://www.hitwise.com","CrawlerBot"
"htdig","CrawlerBot"
"htdig/3.1.5 (admin@ipc-opc.lan)","CrawlerBot"
"htdig/3.1.5 (unconfigured@htdig.searchengine.maintainer)","CrawlerBot"
"htdig/3.1.6 (http://computerorgs.com)","CrawlerBot"
"Html Link Validator (www.lithopssoft.com)","CrawlerBot"
"Httpcheck/1.0 (Perl 5.006001)","CrawlerBot"
"HTTPConnect","CrawlerBot"
"httpget-5.2.2","CrawlerBot"
"Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)","CrawlerBot"
"ia_archiver","CrawlerBot"
"lcabotAccept: */*","CrawlerBot"
"ichiro/1.0 (ichiro@nttr.co.jp)","CrawlerBot"
"IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)","CrawlerBot"
"Mozilla/4.0 (compatible; ICS 1.2.105)","CrawlerBot"
"Iltrovatore-Setaccio","CrawlerBot"
"IlTrovatore-Setaccio (+http://www.iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/0.03-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (+http://www.iltrovatore.it/aiuto/faq.html)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (Indexing; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"Iltrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"imagefetch/0.1 libwww-perl/5.66","CrawlerBot"
"Mozilla/3.0 (compatible; Indy Library)","CrawlerBot"
"InelaBot/0.2 (+http://inelegant.org/bot)","CrawlerBot"
"InfoSeek Sidewinder/1.0A","CrawlerBot"
"Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX)","CrawlerBot"
"Infoseek SideWinder/2.0B (Linux 2.4 i686)","CrawlerBot"
"Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)","CrawlerBot"
"Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Slurp/si-emb (slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"InternetLinkAgent/3.1","CrawlerBot"
"IPiumBot laurion(dot)com","CrawlerBot"
"IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler)","CrawlerBot"
"http://www.istarthere.com (spider@istarthere.com)","CrawlerBot"
"Java1.4.0","CrawlerBot"
"JoBo/1.3 (http://www.matuschek.net/jobo.html)","CrawlerBot"
"k2spider","CrawlerBot"
"KMcrawler","CrawlerBot"
"Knowledge.com/0.2","CrawlerBot"
"Knowledge.com/0.3","CrawlerBot"
"Knowledge Engine","CrawlerBot"
"kuloko-bot/0.2","CrawlerBot"
"Larbin (larbin2.6.2@unspecified.mail)","CrawlerBot"
"larbin (samualt9@bigfoot.com)","CrawlerBot"
"larbin samualt9@bigfoot.com","CrawlerBot"
"larbin_extended (larbin@oktie.com)","CrawlerBot"
"larbin_test (nobody@airmail.etn)","CrawlerBot"
"LARBIN-EXPERIMENTAL (efp@gmx.net)","CrawlerBot"
"LARBIN-EXPERIMENTAL efp@gmx.net","CrawlerBot"
"Mozilla (la2@unspecified.mail)","CrawlerBot"
"Mozilla la2@unspecified.mail","CrawlerBot"
"Mozilla/4.0 (efp@gmx.net)","CrawlerBot"
"Mozilla/4.0 efp@gmx.net","CrawlerBot"
"MSIE-5.13 (larbin@unspecified.mail)","CrawlerBot"
"MSIE-5.13 larbin@unspecified.mail","CrawlerBot"
"SearchGuild_DMOZ_Experiment (chris@searchguild.com)","CrawlerBot"
"SearchGuild_DMOZ_Experiment chris@searchguild.com","CrawlerBot"
"WinampMPEG/2.00 (larbin@unspecified.mail)","CrawlerBot"
"WinampMPEG/2.00 larbin@unspecified.mail","CrawlerBot"
"Larbin larbin2.6.2@unspecified.mail","CrawlerBot"
"larbin_2.6.2 (kalou@kalou.net)","CrawlerBot"
"larbin_2.6.2 (larbin@correa.org)","CrawlerBot"
"larbin_2.6.2 (larbin2.6.2@unspecified.mail)","CrawlerBot"
"larbin_2.6.2 (pimenas@softnet.tuc.gr)","CrawlerBot"
"larbin_2.6.2 (pimenas@systems.tuc.gr)","CrawlerBot"
"larbin_2.6.2 (sumeet_sobti@yahoo.com)","CrawlerBot"
"larbin_2.6.2 (vitalbox1@hotmail.com)","CrawlerBot"
"larbin_2.6.2 (vshelk@yahoo.com)","CrawlerBot"
"larbin_2.6.2 larbin@correa.org","CrawlerBot"
"larbin_2.6.2 larbin2.6.2@unspecified.mail","CrawlerBot"
"larbin_2.6.2 pimenas@systems.tuc.gr","CrawlerBot"
"larbin_2.6.2 sumeet_sobti@yahoo.com","CrawlerBot"
"larbin_2.6.2 vitalbox1@hotmail.com","CrawlerBot"
"larbin_2.6.3 (andreas.beder@chello.at)","CrawlerBot"
"larbin_2.6.3 (larbin2.6.3@unspecified.mail)","CrawlerBot"
"larbin_2.6.3 (larbin-crawler@un.bewaff.net)","CrawlerBot"
"larbin_2.6.3 (pimenas@softnet.tuc.gr)","CrawlerBot"
"larbin_2.6.3 larbin2.6.3@unspecified.mail","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (Tomi.Silander@hiit.fi)","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (tsilande@hiit.fi)","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) Tomi.Silander@hiit.fi","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) tsilande@hiit.fi","CrawlerBot"
"eseek-crawler-larbin-2.63 (crawler@exactseek.com)","CrawlerBot"
"eseek-crawler-larbin-2.63 crawler@exactseek.com","CrawlerBot"
"libwww-MGET/1.0 libwww/5.2.8","CrawlerBot"
"Perl-Win32::Internet/0.082","CrawlerBot"
"/ libwww/5.3.2","CrawlerBot"
"/ libwww/5.4.0","CrawlerBot"
"libwww-perl/5.48","CrawlerBot"
"libwww-perl/5.50","CrawlerBot"
"libwww-perl/5.51","CrawlerBot"
"libwww-perl/5.52 FP/4.0","CrawlerBot"
"libwww-perl/5.53","CrawlerBot"
"libwww-perl/5.63","CrawlerBot"
"libwww-perl/5.64","CrawlerBot"
"libwww-perl/5.65","CrawlerBot"
"MyApp/0.1 libwww-perl/5.65","CrawlerBot"
"rawiswar/0.1 libwww-perl/5.66","CrawlerBot"
"libwww-perl/5.68","CrawlerBot"
"libwww-perl/5.69","CrawlerBot"
"VanillaZilla/0.1 libwww-perl/5.69","CrawlerBot"
"libwww-perl/5.74","CrawlerBot"
"libwww-perl/5.75","CrawlerBot"
"libwww-perl/5.76","CrawlerBot"
"libwww-perl/5.800","CrawlerBot"
"libwww-perl/5.801","CrawlerBot"
"libwww-perl/5.802","CrawlerBot"
"libwww-perl/5.803","CrawlerBot"
"LimeBot/1.0 (+www.cruiselime.com/LimeBot.php)","CrawlerBot"
"Linkbot 3.0","CrawlerBot"
"LinkLint-checkonly/2.3.5","CrawlerBot"
"Linknzbot/ (+http://www.linknz.co.nz/robot.php)","CrawlerBot"
"Linknzbot 2004/(+http://www.linknz.co.nz/robot.php)","CrawlerBot"
"Links SQL (http://gossamer-threads.com/scripts/links-sql/)","CrawlerBot"
"Lite Bot 0616B","CrawlerBot"
"LNSpiderguy","CrawlerBot"
"Look.com","CrawlerBot"
"lwp-trivial/1.29","CrawlerBot"
"lwp-trivial/1.35","CrawlerBot"
"lwp-trivial/1.36","CrawlerBot"
"lwp-request/2.01","CrawlerBot"
"LWP::Simple/5.48","CrawlerBot"
"LWP::Simple/5.65","CrawlerBot"
"Lycos_Spider_(modspider)","CrawlerBot"
"Mediapartners-Google/2.1","CrawlerBot"
"Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Mercator-2.0","CrawlerBot"
"metacarta (crawler@metacarta.com)","CrawlerBot"
"metacarta crawler@metacarta.com","CrawlerBot"
"MetaGer-LinkChecker","CrawlerBot"
"Microsoft URL Control - 5.00.3609","CrawlerBot"
"Microsoft URL Control - 5.01.4319","CrawlerBot"
"Microsoft URL Control - 6.00.8169","CrawlerBot"
"Microsoft URL Control - 6.00.8862","CrawlerBot"
"Microsoft-ATL-Native/7.00","CrawlerBot"
"MicrosoftPrototypeCrawler (How''s my crawling? mailto:newbiecrawler@hotmail.com)","CrawlerBot"
"moget/1.0 (moget@goo.ne.jp)","CrawlerBot"
"moget/2.1 (moget@goo.ne.jp)","CrawlerBot"
"mozDex/0.04-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"Mozilla/4.0 (compatible; Netcraft Web Server Survey)","CrawlerBot"
"Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com)","CrawlerBot"
"Mozilla/4.0 (stat 0.12) (statbot@gmail.com)","CrawlerBot"
"Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;) Unchaos/Crawler","CrawlerBot"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20041027 NaverBot/0.9.3","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040206 GoogleBot/1.1","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040730 Googlebot/2.1/2.1","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040707 Lightningspider/0.9.2","CrawlerBot"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040805 Googlebot/2.1","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider Cache Manager","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider DAV","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider DAV 1.1","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider Protocol Discovery","CrawlerBot"
"Mozilla/2.0 (compatible; MS FrontPage 4.0)","CrawlerBot"
"MSFrontPage/4.0","CrawlerBot"
"Mozilla/2.0 (compatible; MS FrontPage 5.0)","CrawlerBot"
"MSFrontPage/5.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; T-Online Internatinal AG; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; DT; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; QXW0338t; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; FunWebProducts; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; .NET CLR 1.0.3705; Googlebot; .NET CLR 1.1.4322; .NET CLR 1.0.3705; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows compatible LesnikBot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; 3COM U.S. Robotics)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; BOTW)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; http://www.pregnancycrawler.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; Googlebot; .NET CLR 1.1.4322)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; AskBar 3.00; .NET CLR 1.1.4322; Fluffi Bot+)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; CDSource=v9e.03; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 3.1; Googlebot/2.1)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.40607; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; DigExt; FunWebProducts; Media Center PC 3.0; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows XP Professional Bot v.5.)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; Q312461; BTopenworld; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; matlas-2.0.2501; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)","CrawlerBot"
"MSNBOT/0.1 (http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/0.11 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/0.3 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/1.0 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"MSProxy/2.0","CrawlerBot"
"MSRBOT/0.1 (http://research.microsoft.com/research/sv/msrbot/)","CrawlerBot"
"Mozilla/3.01 (compatible;)","CrawlerBot"
"NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"NaverBot_dloader/1.5","CrawlerBot"
"dloader(NaverRobot)/1.0","CrawlerBot"
"dloader(NaverRobot)/1.5","CrawlerBot"
"NetAnts/1.25","CrawlerBot"
"NetNoseCrawler/v1.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com)","CrawlerBot"
"NetResearchServer(http://www.look.com)","CrawlerBot"
"NetResearchServer/2.4(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.5(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.7(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.8(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.9(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/3.4(loopimprovements.com/robot.html)","CrawlerBot"
"NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot)","CrawlerBot"
"NG/1.0","CrawlerBot"
"NPBot","CrawlerBot"
"NPBot (http://www.nameprotect.com/botinfo.html)","CrawlerBot"
"NPBot-1/2.0","CrawlerBot"
"NPBot-1/2.0 (http://www.nameprotect.com/botinfo.html)","CrawlerBot"
"NuSearch Spider www.nusearch.com","CrawlerBot"
"NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"NutchCVS/0.05-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"CreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"Robot: NutchCrawler, Owner: wdavies@acm.org","CrawlerBot"
"NutchOrg/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)","CrawlerBot"
"oBot","CrawlerBot"
"Ocelli/1.3 (http://www.globalspec.com/Ocelli)","CrawlerBot"
"OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Internet Categorizer","CrawlerBot"
"OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Job Crawler","CrawlerBot"
"Openbot/3.0+(robot@monkia.com.tw;+http://gais.cs.ccu.edu.tw/robot.php)","CrawlerBot"
"Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)","CrawlerBot"
"Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)","CrawlerBot"
"OrangeBot","CrawlerBot"
"Mozilla/4.0 (compatible; Advanced Email Extractor v2.24)","CrawlerBot"
"Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"OWR_Crawler 0.1","CrawlerBot"
"parabot (paracite@ecs.soton.ac.uk)","CrawlerBot"
"Patwebbot (http://www.herz-power.de/technik.html)","CrawlerBot"
"pavuk/0.9pl28 i586-pc-cygwin","CrawlerBot"
"pavuk/0.9pl29b i686-pc-linux-gnu","CrawlerBot"
"PEERbot www.peerbot.com","CrawlerBot"
"pipeLiner/0.3a (PipeLine Spider; http://www.pipeline-search.com/webmaster.html; webmaster@pipeline-search.com)","CrawlerBot"
"http://www.planethosting.com","CrawlerBot"
"polybot 1.0 (http://cis.poly.edu/polybot/)","CrawlerBot"
"Pompos/1.1 http://pompos.iliad.fr","CrawlerBot"
"Pompos/1.2 http://pompos.iliad.fr","CrawlerBot"
"Pompos/1.3 http://dir.com/pompos.html","CrawlerBot"
"Portal Manager 0.7","CrawlerBot"
"potbot 1.0","CrawlerBot"
"Program Shareware 1.0.3","CrawlerBot"
"ProWebGuide Link Checker (http://www.prowebguide.com)","CrawlerBot"
"psbot/0.1 (+http://www.picsearch.com/bot.html)","CrawlerBot"
"pverify/1.2","CrawlerBot"
"PWS.Kiosk - Content Filtering","CrawlerBot"
"QPCreep Test Rig ( We are not indexing, just testing )","CrawlerBot"
"QuepasaCreep ( crawler@quepasacorp.com )","CrawlerBot"
"QuepasaCreep v0.9.14","CrawlerBot"
"QuepasaCreep v0.9.13","CrawlerBot"
"reifier.org (admin@reifier.org)","CrawlerBot"
"reifier.org admin@reifier.org","CrawlerBot"
"rico/0.1","CrawlerBot"
"RixBot (http://www.oops-as.no/rix/)","CrawlerBot"
"RoboPal (http://www.findpal.com/)","CrawlerBot"
"RobotMidareru/0.7libwww-perl/5.65","CrawlerBot"
"Search Engine World Robots.txt Validator at http://www.searchengineworld.com/cgi-bin/robotcheck.cgi","CrawlerBot"
"Robozilla/1.0","CrawlerBot"
"RPT-HTTPClient/0.3-3","CrawlerBot"
"SafariBookmarkChecker/1.25 (+http://www.coriolis.ch/)","CrawlerBot"
"SafariBookmarkChecker/1.26 (+http://www.coriolis.ch/)","CrawlerBot"
"Scooter/1.0","CrawlerBot"
"Scooter-ARS-1.1","CrawlerBot"
"Scooter-3.0.FS - Altavista.com","CrawlerBot"
"Scooter/3.2","CrawlerBot"
"Scooter/3.2.SF0","CrawlerBot"
"Scooter_x0-3.2.EX","CrawlerBot"
"Scooter-3.2","CrawlerBot"
"Scooter-3.2.BT","CrawlerBot"
"Scooter-3.2.EX","CrawlerBot"
"Scooter-3.2.FNR","CrawlerBot"
"Scooter-3.2.PDF","CrawlerBot"
"Scooter-3.2.SF0","CrawlerBot"
"Scooter-3.2.TX.FNR","CrawlerBot"
"Scooter-3.2.XX0","CrawlerBot"
"Scooter/3.3","CrawlerBot"
"Scooter/3.3.QA","CrawlerBot"
"Scooter/3.3.QA.pczukor","CrawlerBot"
"Scooter/3.3.vscooter","CrawlerBot"
"Scooter/3.3_SF","CrawlerBot"
"Scrubby/2.1 (http://www.scrubtheweb.com/abs/meta-check.html)","CrawlerBot"
"Scrubby/2.2 (http://www.scrubtheweb.com/)","CrawlerBot"
"Search Agent 1.0","CrawlerBot"
"SearchSpider.com/1.1","CrawlerBot"
"Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3","CrawlerBot"
"Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)","CrawlerBot"
"semanticdiscovery/0.1","CrawlerBot"
"Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au)","CrawlerBot"
"sherlock/1.3 httpget/1.3","CrawlerBot"
"sherlock_spider (jimfan@163.com)","CrawlerBot"
"InternetSeer.com","CrawlerBot"
"sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com)","CrawlerBot"
"sitescooper/3.1.2 (http://sitescooper.org) libwww-perl/5.51","CrawlerBot"
"SiteXpert","CrawlerBot"
"SlySearch/1.3 (http://www.slysearch.com)","CrawlerBot"
"SlySearch/1.3 http://www.slysearch.com","CrawlerBot"
"sohu-search","CrawlerBot"
"Speedy Spider (http://www.entireweb.com)","CrawlerBot"
"Speedy_Spider_(http://www.entireweb.com)","CrawlerBot"
"Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)","CrawlerBot"
"SpiderKU/0.9","CrawlerBot"
"SpiderMonkey/7.04 (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)","CrawlerBot"
"Spider_Monkey/7.06 (SpiderMonkey.ca info at http://SpiderMonkey.ca /sm.shtml)","CrawlerBot"
"Spider_Monkey/7.06 (SpiderMonkey.ca info at http://www.spidermonkey.ca/sm.shtml)","CrawlerBot"
"Mozilla/5.0 (compatible; SpurlBot/0.2)","CrawlerBot"
"Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)","CrawlerBot"
"Star Downloader","CrawlerBot"
"Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)","CrawlerBot"
"Mozilla/4.0 (compatible; SuperCleaner 2.56; Windows NT 5.1)","CrawlerBot"
"Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;)","CrawlerBot"
"Szukacz/1.5","CrawlerBot"
"Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)","CrawlerBot"
"Tarantula Experimental Crawler","CrawlerBot"
"Tcl http client package 1.0","CrawlerBot"
"Tcl http client package 2.3","CrawlerBot"
"(Teradex Mapper; mapper@teradex.com; http://www.teradex.com)","CrawlerBot"
"Teradex_Crawler (crawler@teradex.com; http://crawler.teradex.com)","CrawlerBot"
"TheSuBot/0.1 (www.thesubot.de)","CrawlerBot"
"thesubot-beta-www.thesubot.de","CrawlerBot"
"thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)","CrawlerBot"
"timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html","CrawlerBot"
"Tkensaku/0.9 (http://www.tkensaku.com/q.html)","CrawlerBot"
"TranSGeniKBot (http://www.tsgk.net)","CrawlerBot"
"TranSGeniKBot http://www.tsgk.net","CrawlerBot"
"TulipChain/5.7 (http://ostermiller.org/tulipchain/) Java/1.4.0_02 (http://java.sun.com/) Windows_Me/4.90","CrawlerBot"
"TulipChain/5.94 (http://ostermiller.org/tulipchain/) Java/1.4.1_01 (http://apple.com/) Mac_OS_X/10.2.8","CrawlerBot"
"TulipChain/6.01 (http://ostermiller.org/tulipchain/) Java/1.4.2_03 (http://java.sun.com/) Windows_XP/5.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"TulipChain/6.02 (http://ostermiller.org/tulipchain/) Java/1.4.2_03 (http://apple.com/) Mac_OS_X/10.3.3 RPT-HTTPClient/0.3-3","CrawlerBot"
"TulipChain/6.03 (http://ostermiller.org/tulipchain/) Java/1.4.2_05 (http://java.sun.com/) Windows_XP/5.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"TurnitinBot/1.4 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/1.4 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TurnitinBot/2.0 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TutorGigBot/1.5 ( +http://www.tutorgig.info )","CrawlerBot"
"Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)","CrawlerBot"
"UdmSearch/3.1.20","CrawlerBot"
"UIowaCrawler/1.0","CrawlerBot"
"UIowaCrawler/2.0","CrawlerBot"
"unchaos_crawler_2.0.2 (search.engine@unchaos.com)","CrawlerBot"
"VM4050/132.037 UP.Browser/6.2.2.4.e.1.100 (GUI) MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om)","CrawlerBot"
"USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)","CrawlerBot"
"Vagabondo/2.0 MT (webagent at wise-guys dot nl)","CrawlerBot"
"Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)","CrawlerBot"
"Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)","CrawlerBot"
"Vivante Link Checker (http://www.vivante.com)","CrawlerBot"
"void-bot/0.1 (bot@void.be; http://www.void.be/)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6","CrawlerBot"
"Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2","CrawlerBot"
"VSE/1.0 (vsecrawler@hotmail.com)","CrawlerBot"
"vspider","CrawlerBot"
"W3C_Validator/1.183 libwww-perl/5.64","CrawlerBot"
"W3C_Validator/1.305.2.109 libwww-perl/5.79","CrawlerBot"
"W3C_Validator/1.305.2.12 libwww-perl/5.64","CrawlerBot"
"W3C_Validator/1.305.2.137 libwww-perl/5.79","CrawlerBot"
"W3C_Validator/1.305.2.148 libwww-perl/5.800","CrawlerBot"
"W3C_Validator/1.305.2.148 libwww-perl/5.803","CrawlerBot"
"W3C-checklink/2.90 libwww-perl/5.64","CrawlerBot"
"W3C-checklink/3.6.2.3 libwww-perl/5.64","CrawlerBot"
"W3C-checklink/3.9.2 [3.17] libwww-perl/5.79","CrawlerBot"
"W3C-checklink/4.0 [4.4] libwww-perl/5.800","CrawlerBot"
"W3C-checklink/4.1 [4.14] libwww-perl/5.800","CrawlerBot"
"webbot","CrawlerBot"
"Webclipping.com","CrawlerBot"
"webcollage/1.102","CrawlerBot"
"webcollage/1.104","CrawlerBot"
"webcollage/1.87","CrawlerBot"
"webcollage/1.93","CrawlerBot"
"webcollage/1.94","CrawlerBot"
"Thu Mar 27 18:20:34 CET 2003WebcraftBoot","CrawlerBot"
"Fri Nov 15 04:51:18 EST 2002WebcraftBoot Java/1.4.1_01","CrawlerBot"
"Sun Apr 20 22:00:01 EDT 2003WebcraftBoot Java/1.4.2-beta","CrawlerBot"
"Tue Apr 15 22:00:03 EDT 2003WebcraftBoot Java/1.4.2-beta","CrawlerBot"
"WebFilter Robot 1.0","CrawlerBot"
"Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56)","CrawlerBot"
"WebRACE/1.1 (University of Cyprus, Distributed Crawler)","CrawlerBot"
"WebSauger 1.20b","CrawlerBot"
"http://www.websearch.com.au (larbin2.6.2@unspecified.mail)","CrawlerBot"
"http://www.websearch.com.au larbin2.6.2@unspecified.mail","CrawlerBot"
"http://www.WebSearch.com.au/ (larbin2.6.2@unspecified.mail)","CrawlerBot"
"http://www.WebSearch.com.au/ larbin2.6.2@unspecified.mail","CrawlerBot"
"www.WebSearch.com.au (search@websearch.com.au)","CrawlerBot"
"www.WebSearch.com.au search@websearch.com.au","CrawlerBot"
"WebSearch/2.0.1 (Dez@Blanchfield.COM.AU, http://www.WebSearch.com.au/)","CrawlerBot"
"WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU)","CrawlerBot"
"http://www.WebSearch.com.au/ - Australian Search Engine/3.1.3 (sites@websearch.com.au)","CrawlerBot"
"http://www.WebSearch.com.au/ - Australian Search Engine/3.1.6 (sites@websearch.com.au)","CrawlerBot"
"www.webwombat.com.au","CrawlerBot"
"webyield robot (http://www.webyield.net/search/search.pl)","CrawlerBot"
"Wget/1.5.2","CrawlerBot"
"Wget/1.5.3","CrawlerBot"
"Wget/1.5.3.1","CrawlerBot"
"Wget/1.6","CrawlerBot"
"Wget/1.7","CrawlerBot"
"Wget/1.8","CrawlerBot"
"Wget/1.8.1","CrawlerBot"
"Wget/1.8.1+cvs","CrawlerBot"
"Wget/1.8.2","CrawlerBot"
"Wget/1.9","CrawlerBot"
"Wget/1.9-beta","CrawlerBot"
"Wget/1.9.1","CrawlerBot"
"Willow Internet Crawler by Twotrees V2.1","CrawlerBot"
"Wotbox/alpha0.5.1 (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02","CrawlerBot"
"http://www.ciml.co.uk","CrawlerBot"
"WWWeasel Robot v1.00 (http://wwweasel.de)","CrawlerBot"
"Xenu''s Link Sleuth 1.1a","CrawlerBot"
"Xenu Link Sleuth 1.2b","CrawlerBot"
"Xenu Link Sleuth 1.2d","CrawlerBot"
"Xenu Link Sleuth 1.2e","CrawlerBot"
"Xenu Link Sleuth 1.2f","CrawlerBot"
"Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)","CrawlerBot"
"Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)","CrawlerBot"
"YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine","CrawlerBot"
"Zao/0.1 (http://www.kototoi.org/zao/)","CrawlerBot"
"Zao/0.2 (http://www.kototoi.org/zao/)","CrawlerBot"
"Zao-Crawler","CrawlerBot"
"Zeus 3140 Webster Pro V2.9 Win32","CrawlerBot"
"Zeus 57657 Webster Pro V2.9 Win32","CrawlerBot"
"ZipppBot/0.11 (ZipppBot; http://www.zippp.net; webmaster@zippp.net)","CrawlerBot"
"ZoomSpider - wrensoft.com","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-1.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-12.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-2.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d03 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d05 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker Beta-d01 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 DLC (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
0
 
LVL 12

Expert Comment

by:Sinoj Sebastian
ID: 18754309
> shortest and most effective list wins
So the list of all popular bot agent strings

    *Bot*

Most of the spider user-agent strings will have the substring "Bot" in it.
Ordinary user-agent string from IE, FF, NS etc do not contain "Bot".
Try filter using this.
I'm already using it.

From the above list I found That "CrawlerBot" is also a substring of bot agent strings

0
 
LVL 29

Expert Comment

by:rdivilbiss
ID: 18757241
No, in the list above, "CrawlerBot" is not part of the user agent string, it was a field in the database of my collection of live user agents taken from dozens of my web sites and hand categorized.  Ignore: ,"CrawlerBot"
0
 
LVL 16

Author Comment

by:OliWarner
ID: 18757392
Yeah I can parse those out without issue. Thanks Rod that looks like it'll do the job perfectly
0
 
LVL 29

Expert Comment

by:rdivilbiss
ID: 18758475
Those were as of January. There could always be a few new ones cropping up.
0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Create animated movies for web page 18 82
Help in good tutorials for PHP, HTML and CSS 6 40
Wordpress Security 29 47
Link failure 16 31
An enjoyable and seamless user experience can go a long way on an eCommerce site. While a cohesive layout and engaging copy play roles in creating a positive user experience, some sites neglect aspects that seem marginal but in actuality prove very …
Today, the web development industry is booming, and many people consider it to be their vocation. The question you may be asking yourself is – how do I become a web developer?
Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…

808 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question