Solved

User-agent strings

Posted on 2007-03-19
8
7,133 Views
Last Modified: 2013-12-09
I need a list of the most popular spider user-agent strings. I've got several items on my website that log or increment things and I really only want some of those to be logging if the thing hitting the page is a real person and not a bot. So I'm left checking the user-agents.

I can either grab the most popular browsers or the most popular bots... Whatever is most efficient -- you decide!

Either way, time-complexity is an issue as it is a fairly busy site, so the shortest and most effective list wins =)
0
Comment
Question by:OliWarner
8 Comments
 
LVL 2

Expert Comment

by:fpintos
ID: 18753824
Have you tried setting up robot.txt to block these spiders? This is by far the simplest way.
0
 
LVL 16

Author Comment

by:OliWarner
ID: 18753862
I don't want to block them from viewing the pages -- just stop my logging script counting hits from them.
0
 
LVL 29

Expert Comment

by:rdivilbiss
ID: 18753977
I've got them in a db, Oli.  Give me a minute to extract the bots from the browsers.
0
Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

 
LVL 29

Accepted Solution

by:
rdivilbiss earned 500 total points
ID: 18754003
http://www.rodsdot.com/downloads/bots.zip

"ADSAComponent (postmaster@cnds.ucd.ie)","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [fc3]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [c01]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [wf224]","CrawlerBot"
"http://www.almaden.ibm.com/cs/crawler [wf55]","CrawlerBot"
"Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)","CrawlerBot"
"Mozilla/4.0 (Search Engine Marketing Tactics Amsterdam 2002 Information Spider)","CrawlerBot"
"AnswerBus (http://www.answerbus.com/)","CrawlerBot"
"antibot-V1.1.11/i586-linux-2.2","CrawlerBot"
"antibot-V1.1.13/i586-linux-2.2","CrawlerBot"
"antibot-V1.2.0/redhat-linux-9","CrawlerBot"
"AOLserver-Tcl/3.5.6","CrawlerBot"
"AOL 8.0 (compatible; AOL 8.0; DOS; .NET CLR 1.1.4322)","CrawlerBot"
"appie 1.1 (www.walhello.com)","CrawlerBot"
"Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)","CrawlerBot"
"Art-Online.com 0.9(Beta)","CrawlerBot"
"Mozilla/2.0 (compatible; Ask Jeeves)","CrawlerBot"
"Mozilla/2.0 (compatible; Ask Jeeves/Teoma)","CrawlerBot"
"ASPseek/1.2.10","CrawlerBot"
"ASPseek/1.2.11","CrawlerBot"
"ASPseek/1.2.12","CrawlerBot"
"Mozilla/3.0 (compatible; AvantGo 3.2)","CrawlerBot"
"BaiDuSpider","CrawlerBot"
"Baiduspider+(+http://www.baidu.com/search/spider.htm)","CrawlerBot"
"battlebot","CrawlerBot"
"BDFetch","CrawlerBot"
"BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html)","CrawlerBot"
"BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"Big Brother (http://pauillac.inria.fr/~fpottier/)","CrawlerBot"
"BlogBot/1.2","CrawlerBot"
"boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-dc/0.5 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-dc/0.66 ( http://www.boitho.com/dcbot.html )","CrawlerBot"
"boitho.com-robot/1.0","CrawlerBot"
"boitho.com-robot/1.1","CrawlerBot"
"Mozilla/4.0 (compatible; BorderManager 3.0)","CrawlerBot"
"BrailleBot 1.0","CrawlerBot"
"BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html)","CrawlerBot"
"bumblebee/1.0 (bumblebee@relevare.com; http://www.relevare.com/)","CrawlerBot"
"Computer_and_Automation_Research_Institute_Crawler (nospamspidernospam@spider.ilab.sztakinospam.hunospam)","CrawlerBot"
"Computer_and_Automation_Research_Institute_Crawler (spider@spider.ilab.sztaki.hu)","CrawlerBot"
"cd34/0.1","CrawlerBot"
"CerberianDrtrs/Version-3.0-Release-24","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-40)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-11)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-12)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-13)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-16)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-17)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-41)","CrawlerBot"
"Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-43)","CrawlerBot"
"CipinetBot (http://www.cipinet.com/bot.html)","CrawlerBot"
"Clushbot/2.1 (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.21-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.23-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.24-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.6-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"Clushbot/3.9-BinaryFury (+http://www.clush.com/bot.html)","CrawlerBot"
"ComMOOnity LambdaMOO/1.8.1","CrawlerBot"
"CrawlConvera0.1 (CrawlConvera@yahoo.com)","CrawlerBot"
"CrawlConvera0.1 (www.authoritativeweb.com)","CrawlerBot"
"ConveraCrawler/0.2","CrawlerBot"
"ConveraCrawler/0.5 (+http://www","CrawlerBot"
"cosmos/0.9_(robot@xyleme.com)","CrawlerBot"
"Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"Cowbot-0.1.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"Crawl_Application","CrawlerBot"
"CrocCrawler v3.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"CrocCrawler v4.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)","CrawlerBot"
"Custo 2.0 (www.netwu.com)","CrawlerBot"
"CydralSpider/1.9 (Cydral Web Image Search; http://www.cydral.com)","CrawlerBot"
"DeepIndex (http://www.deepindex.com)","CrawlerBot"
"DeMozulator 1.0 (MacOS, dMoz URL Check Agent, trebor@animeigo.com)","CrawlerBot"
"DoCoMo/1.0/N504i/c10/TB","CrawlerBot"
"DoCoMo/1.0/P504iS/c10/TB","CrawlerBot"
"Dual Proxy","CrawlerBot"
"Dumbot(version 0.1 beta - dumbfind.com)","CrawlerBot"
"Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html)","CrawlerBot"
"Dumbot(version 0.1 beta)","CrawlerBot"
"EARTHCOM.info/1.2","CrawlerBot"
"EmailSiphon","CrawlerBot"
"Enterprise_Search/1.00.136;MSSQL (http://www.innerprise.net/es-spider.asp)","CrawlerBot"
"e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)","CrawlerBot"
"exactseek-crawler-2.63 (crawler@exactseek.com)","CrawlerBot"
"exactseek-crawler-2.63 crawler@exactseek.com","CrawlerBot"
"exactseek-crawler-2.63-5 (crawler@exactseek.com)","CrawlerBot"
"exactseek-crawler-2.63-5 crawler@exactseek.com","CrawlerBot"
"Explorer 6","CrawlerBot"
"FAST Enterprise Crawler/6 (crawler@fast.no)","CrawlerBot"
"FAST Enterprise Crawler/6 (www.fastsearch.com)","CrawlerBot"
"FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)","CrawlerBot"
"FastBug http://www.ay-up.com","CrawlerBot"
"FAST-WebCrawler/3.2 test","CrawlerBot"
"FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)","CrawlerBot"
"FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)","CrawlerBot"
"FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://www.alltheweb.com/help/webmaster/crawler)","CrawlerBot"
"FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler)","CrawlerBot"
"FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"FAST-WebCrawler/3.x Multimedia","CrawlerBot"
"FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)","CrawlerBot"
"favicon finder at http://iconsurf.com/","CrawlerBot"
"favicon monitor at http://iconsurf.com/","CrawlerBot"
"Mozilla/4.0 (compatible: FDSE robot)","CrawlerBot"
"Filangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)","CrawlerBot"
"Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com)","CrawlerBot"
"Filangy/1.01 (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)","CrawlerBot"
"FindLinks/0.71 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.82 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.87 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"findlinks/0.89 (+http://wortschatz.uni-leipzig.de/findlinks/)","CrawlerBot"
"Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)","CrawlerBot"
"Flickbot 1.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"FlickBot 2.0 RPT-HTTPClient/0.3-3","CrawlerBot"
"Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; http://www.pgts.com.au/; +http://www.galaxy.com/info/crawler.html)","CrawlerBot"
"FyberSpider (+http://www.fybersearch.com/fyberspider.php)","CrawlerBot"
"GAIS Robot/1.1A2","CrawlerBot"
"Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)","CrawlerBot"
"GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)","CrawlerBot"
"gatherer/0.9","CrawlerBot"
"gazz/5.0 (gazz@nttr.co.jp)","CrawlerBot"
"Generic","CrawlerBot"
"GeonaBot 1.0; http://www.geona.com/","CrawlerBot"
"GeonaBot/1.1; http://www.geona.com/","CrawlerBot"
"GetRight/4.5e","CrawlerBot"
"Gigabot/1.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)","CrawlerBot"
"Goldfire Server","CrawlerBot"
"Googlebot (+http://www.google.com/bot.html)","CrawlerBot"
"GoogleBot/2.1","CrawlerBot"
"Googlebot/2.1 (+http://www.google.com/bot.html)","CrawlerBot"
"googlebot/2.1 (+http://www.googlebot.com/bot.html","CrawlerBot"
"Googlebot/2.1 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Googlebot/2.1 (+http://www.googlebot.com/bot.html) (compatible; MSIE 6.0; )","CrawlerBot"
"Googlebot/2.1 (compatible; MSIE; Windows)","CrawlerBot"
"googlebot/2.1; +http://www.google.com/bot.html","CrawlerBot"
"Googlebot/2.1+(+http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.)","CrawlerBot"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"Googlebot/Test (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Googlebot-Image/1.0","CrawlerBot"
"Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html","CrawlerBot"
"Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Green Research, Inc.","CrawlerBot"
"GregBot (compatible; MSIE; Windows; Q312461)","CrawlerBot"
"grub crawler","CrawlerBot"
"grub crawler(http://www.grub.org)","CrawlerBot"
"grub-client","CrawlerBot"
"Mozilla/4.0 (compatible; Grubclient; windows; SV1; .NET CLR 1.1.4322)","CrawlerBot"
"Mozilla/4.0 (compatible; Grubclient-2.2-internal-beta)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-2.6.0)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.07; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)","CrawlerBot"
"Mozilla/4.0 (compatible; grub-client-2.3)","CrawlerBot"
"gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com)","CrawlerBot"
"Crawler [en] (compatible; Crawler Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)","CrawlerBot"
"Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Roadrunner; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SC/5.60/1.01/FS-Internett; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; stokeybot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts-MyWay; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot/2.1 (+http://www.googlebot.com/bot.html))","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Googlebot/2.1 (+http://www.googlebot.com/bot.html); Maxthon; FDM)","CrawlerBot"
"Harvest-NG/1.0.2","CrawlerBot"
"Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)","CrawlerBot"
"Hatena Antenna/0.4 (http://a.hatena.ne.jp/help)","CrawlerBot"
"hget/0.3","CrawlerBot"
"Hitwise Spider v1.0 http://www.hitwise.com","CrawlerBot"
"htdig","CrawlerBot"
"htdig/3.1.5 (admin@ipc-opc.lan)","CrawlerBot"
"htdig/3.1.5 (unconfigured@htdig.searchengine.maintainer)","CrawlerBot"
"htdig/3.1.6 (http://computerorgs.com)","CrawlerBot"
"Html Link Validator (www.lithopssoft.com)","CrawlerBot"
"Httpcheck/1.0 (Perl 5.006001)","CrawlerBot"
"HTTPConnect","CrawlerBot"
"httpget-5.2.2","CrawlerBot"
"Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)","CrawlerBot"
"ia_archiver","CrawlerBot"
"lcabotAccept: */*","CrawlerBot"
"ichiro/1.0 (ichiro@nttr.co.jp)","CrawlerBot"
"IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)","CrawlerBot"
"Mozilla/4.0 (compatible; ICS 1.2.105)","CrawlerBot"
"Iltrovatore-Setaccio","CrawlerBot"
"IlTrovatore-Setaccio (+http://www.iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/0.03-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (+http://www.iltrovatore.it/aiuto/faq.html)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (Indexing; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"Iltrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"IlTrovatore-Setaccio/1.2-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)","CrawlerBot"
"imagefetch/0.1 libwww-perl/5.66","CrawlerBot"
"Mozilla/3.0 (compatible; Indy Library)","CrawlerBot"
"InelaBot/0.2 (+http://inelegant.org/bot)","CrawlerBot"
"InfoSeek Sidewinder/1.0A","CrawlerBot"
"Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX)","CrawlerBot"
"Infoseek SideWinder/2.0B (Linux 2.4 i686)","CrawlerBot"
"Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)","CrawlerBot"
"Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"Slurp/si-emb (slurp@inktomi.com; http://www.inktomi.com/slurp.html)","CrawlerBot"
"InternetLinkAgent/3.1","CrawlerBot"
"IPiumBot laurion(dot)com","CrawlerBot"
"IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler)","CrawlerBot"
"http://www.istarthere.com (spider@istarthere.com)","CrawlerBot"
"Java1.4.0","CrawlerBot"
"JoBo/1.3 (http://www.matuschek.net/jobo.html)","CrawlerBot"
"k2spider","CrawlerBot"
"KMcrawler","CrawlerBot"
"Knowledge.com/0.2","CrawlerBot"
"Knowledge.com/0.3","CrawlerBot"
"Knowledge Engine","CrawlerBot"
"kuloko-bot/0.2","CrawlerBot"
"Larbin (larbin2.6.2@unspecified.mail)","CrawlerBot"
"larbin (samualt9@bigfoot.com)","CrawlerBot"
"larbin samualt9@bigfoot.com","CrawlerBot"
"larbin_extended (larbin@oktie.com)","CrawlerBot"
"larbin_test (nobody@airmail.etn)","CrawlerBot"
"LARBIN-EXPERIMENTAL (efp@gmx.net)","CrawlerBot"
"LARBIN-EXPERIMENTAL efp@gmx.net","CrawlerBot"
"Mozilla (la2@unspecified.mail)","CrawlerBot"
"Mozilla la2@unspecified.mail","CrawlerBot"
"Mozilla/4.0 (efp@gmx.net)","CrawlerBot"
"Mozilla/4.0 efp@gmx.net","CrawlerBot"
"MSIE-5.13 (larbin@unspecified.mail)","CrawlerBot"
"MSIE-5.13 larbin@unspecified.mail","CrawlerBot"
"SearchGuild_DMOZ_Experiment (chris@searchguild.com)","CrawlerBot"
"SearchGuild_DMOZ_Experiment chris@searchguild.com","CrawlerBot"
"WinampMPEG/2.00 (larbin@unspecified.mail)","CrawlerBot"
"WinampMPEG/2.00 larbin@unspecified.mail","CrawlerBot"
"Larbin larbin2.6.2@unspecified.mail","CrawlerBot"
"larbin_2.6.2 (kalou@kalou.net)","CrawlerBot"
"larbin_2.6.2 (larbin@correa.org)","CrawlerBot"
"larbin_2.6.2 (larbin2.6.2@unspecified.mail)","CrawlerBot"
"larbin_2.6.2 (pimenas@softnet.tuc.gr)","CrawlerBot"
"larbin_2.6.2 (pimenas@systems.tuc.gr)","CrawlerBot"
"larbin_2.6.2 (sumeet_sobti@yahoo.com)","CrawlerBot"
"larbin_2.6.2 (vitalbox1@hotmail.com)","CrawlerBot"
"larbin_2.6.2 (vshelk@yahoo.com)","CrawlerBot"
"larbin_2.6.2 larbin@correa.org","CrawlerBot"
"larbin_2.6.2 larbin2.6.2@unspecified.mail","CrawlerBot"
"larbin_2.6.2 pimenas@systems.tuc.gr","CrawlerBot"
"larbin_2.6.2 sumeet_sobti@yahoo.com","CrawlerBot"
"larbin_2.6.2 vitalbox1@hotmail.com","CrawlerBot"
"larbin_2.6.3 (andreas.beder@chello.at)","CrawlerBot"
"larbin_2.6.3 (larbin2.6.3@unspecified.mail)","CrawlerBot"
"larbin_2.6.3 (larbin-crawler@un.bewaff.net)","CrawlerBot"
"larbin_2.6.3 (pimenas@softnet.tuc.gr)","CrawlerBot"
"larbin_2.6.3 larbin2.6.3@unspecified.mail","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (Tomi.Silander@hiit.fi)","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (tsilande@hiit.fi)","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) Tomi.Silander@hiit.fi","CrawlerBot"
"larbin_2.6.3_for_(http://cosco.hiit.fi/search/) tsilande@hiit.fi","CrawlerBot"
"eseek-crawler-larbin-2.63 (crawler@exactseek.com)","CrawlerBot"
"eseek-crawler-larbin-2.63 crawler@exactseek.com","CrawlerBot"
"libwww-MGET/1.0 libwww/5.2.8","CrawlerBot"
"Perl-Win32::Internet/0.082","CrawlerBot"
"/ libwww/5.3.2","CrawlerBot"
"/ libwww/5.4.0","CrawlerBot"
"libwww-perl/5.48","CrawlerBot"
"libwww-perl/5.50","CrawlerBot"
"libwww-perl/5.51","CrawlerBot"
"libwww-perl/5.52 FP/4.0","CrawlerBot"
"libwww-perl/5.53","CrawlerBot"
"libwww-perl/5.63","CrawlerBot"
"libwww-perl/5.64","CrawlerBot"
"libwww-perl/5.65","CrawlerBot"
"MyApp/0.1 libwww-perl/5.65","CrawlerBot"
"rawiswar/0.1 libwww-perl/5.66","CrawlerBot"
"libwww-perl/5.68","CrawlerBot"
"libwww-perl/5.69","CrawlerBot"
"VanillaZilla/0.1 libwww-perl/5.69","CrawlerBot"
"libwww-perl/5.74","CrawlerBot"
"libwww-perl/5.75","CrawlerBot"
"libwww-perl/5.76","CrawlerBot"
"libwww-perl/5.800","CrawlerBot"
"libwww-perl/5.801","CrawlerBot"
"libwww-perl/5.802","CrawlerBot"
"libwww-perl/5.803","CrawlerBot"
"LimeBot/1.0 (+www.cruiselime.com/LimeBot.php)","CrawlerBot"
"Linkbot 3.0","CrawlerBot"
"LinkLint-checkonly/2.3.5","CrawlerBot"
"Linknzbot/ (+http://www.linknz.co.nz/robot.php)","CrawlerBot"
"Linknzbot 2004/(+http://www.linknz.co.nz/robot.php)","CrawlerBot"
"Links SQL (http://gossamer-threads.com/scripts/links-sql/)","CrawlerBot"
"Lite Bot 0616B","CrawlerBot"
"LNSpiderguy","CrawlerBot"
"Look.com","CrawlerBot"
"lwp-trivial/1.29","CrawlerBot"
"lwp-trivial/1.35","CrawlerBot"
"lwp-trivial/1.36","CrawlerBot"
"lwp-request/2.01","CrawlerBot"
"LWP::Simple/5.48","CrawlerBot"
"LWP::Simple/5.65","CrawlerBot"
"Lycos_Spider_(modspider)","CrawlerBot"
"Mediapartners-Google/2.1","CrawlerBot"
"Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)","CrawlerBot"
"Mercator-2.0","CrawlerBot"
"metacarta (crawler@metacarta.com)","CrawlerBot"
"metacarta crawler@metacarta.com","CrawlerBot"
"MetaGer-LinkChecker","CrawlerBot"
"Microsoft URL Control - 5.00.3609","CrawlerBot"
"Microsoft URL Control - 5.01.4319","CrawlerBot"
"Microsoft URL Control - 6.00.8169","CrawlerBot"
"Microsoft URL Control - 6.00.8862","CrawlerBot"
"Microsoft-ATL-Native/7.00","CrawlerBot"
"MicrosoftPrototypeCrawler (How''s my crawling? mailto:newbiecrawler@hotmail.com)","CrawlerBot"
"moget/1.0 (moget@goo.ne.jp)","CrawlerBot"
"moget/2.1 (moget@goo.ne.jp)","CrawlerBot"
"mozDex/0.04-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"Mozdex/0.06-dev (Mozdex; http://www.mozdex.com/bot.html; spider@mozdex.com)","CrawlerBot"
"Mozilla/4.0 (compatible; Netcraft Web Server Survey)","CrawlerBot"
"Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com)","CrawlerBot"
"Mozilla/4.0 (stat 0.12) (statbot@gmail.com)","CrawlerBot"
"Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;) Unchaos/Crawler","CrawlerBot"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20041027 NaverBot/0.9.3","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040206 GoogleBot/1.1","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040730 Googlebot/2.1/2.1","CrawlerBot"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040707 Lightningspider/0.9.2","CrawlerBot"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040805 Googlebot/2.1","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider Cache Manager","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider DAV","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider DAV 1.1","CrawlerBot"
"Microsoft Data Access Internet Publishing Provider Protocol Discovery","CrawlerBot"
"Mozilla/2.0 (compatible; MS FrontPage 4.0)","CrawlerBot"
"MSFrontPage/4.0","CrawlerBot"
"Mozilla/2.0 (compatible; MS FrontPage 5.0)","CrawlerBot"
"MSFrontPage/5.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; T-Online Internatinal AG; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; DT; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; QXW0338t; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; FunWebProducts; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; .NET CLR 1.0.3705; Googlebot; .NET CLR 1.1.4322; .NET CLR 1.0.3705; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows compatible LesnikBot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; 3COM U.S. Robotics)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; BOTW)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Googlebot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; http://www.pregnancycrawler.com)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; Googlebot; .NET CLR 1.1.4322)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; AskBar 3.00; .NET CLR 1.1.4322; Fluffi Bot+)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; CDSource=v9e.03; .NET CLR 1.0.3705; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 3.1; Googlebot/2.1)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.40607; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; DigExt; FunWebProducts; Media Center PC 3.0; .NET CLR 1.0.3705; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows XP Professional Bot v.5.)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; Q312461; BTopenworld; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; matlas-2.0.2501; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461; MSIECrawler)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)","CrawlerBot"
"MSNBOT/0.1 (http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/0.11 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/0.3 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"msnbot/1.0 (+http://search.msn.com/msnbot.htm)","CrawlerBot"
"MSProxy/2.0","CrawlerBot"
"MSRBOT/0.1 (http://research.microsoft.com/research/sv/msrbot/)","CrawlerBot"
"Mozilla/3.01 (compatible;)","CrawlerBot"
"NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)","CrawlerBot"
"NaverBot_dloader/1.5","CrawlerBot"
"dloader(NaverRobot)/1.0","CrawlerBot"
"dloader(NaverRobot)/1.5","CrawlerBot"
"NetAnts/1.25","CrawlerBot"
"NetNoseCrawler/v1.0","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com)","CrawlerBot"
"NetResearchServer(http://www.look.com)","CrawlerBot"
"NetResearchServer/2.4(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.5(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.7(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.8(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/2.9(loopimprovements.com/robot.html)","CrawlerBot"
"NetResearchServer/3.4(loopimprovements.com/robot.html)","CrawlerBot"
"NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot)","CrawlerBot"
"NG/1.0","CrawlerBot"
"NPBot","CrawlerBot"
"NPBot (http://www.nameprotect.com/botinfo.html)","CrawlerBot"
"NPBot-1/2.0","CrawlerBot"
"NPBot-1/2.0 (http://www.nameprotect.com/botinfo.html)","CrawlerBot"
"NuSearch Spider www.nusearch.com","CrawlerBot"
"NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"NutchCVS/0.05-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"CreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"Robot: NutchCrawler, Owner: wdavies@acm.org","CrawlerBot"
"NutchOrg/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)","CrawlerBot"
"oBot","CrawlerBot"
"Ocelli/1.3 (http://www.globalspec.com/Ocelli)","CrawlerBot"
"OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Internet Categorizer","CrawlerBot"
"OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Job Crawler","CrawlerBot"
"Openbot/3.0+(robot@monkia.com.tw;+http://gais.cs.ccu.edu.tw/robot.php)","CrawlerBot"
"Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)","CrawlerBot"
"Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)","CrawlerBot"
"OrangeBot","CrawlerBot"
"Mozilla/4.0 (compatible; Advanced Email Extractor v2.24)","CrawlerBot"
"Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)","CrawlerBot"
"OWR_Crawler 0.1","CrawlerBot"
"parabot (paracite@ecs.soton.ac.uk)","CrawlerBot"
"Patwebbot (http://www.herz-power.de/technik.html)","CrawlerBot"
"pavuk/0.9pl28 i586-pc-cygwin","CrawlerBot"
"pavuk/0.9pl29b i686-pc-linux-gnu","CrawlerBot"
"PEERbot www.peerbot.com","CrawlerBot"
"pipeLiner/0.3a (PipeLine Spider; http://www.pipeline-search.com/webmaster.html; webmaster@pipeline-search.com)","CrawlerBot"
"http://www.planethosting.com","CrawlerBot"
"polybot 1.0 (http://cis.poly.edu/polybot/)","CrawlerBot"
"Pompos/1.1 http://pompos.iliad.fr","CrawlerBot"
"Pompos/1.2 http://pompos.iliad.fr","CrawlerBot"
"Pompos/1.3 http://dir.com/pompos.html","CrawlerBot"
"Portal Manager 0.7","CrawlerBot"
"potbot 1.0","CrawlerBot"
"Program Shareware 1.0.3","CrawlerBot"
"ProWebGuide Link Checker (http://www.prowebguide.com)","CrawlerBot"
"psbot/0.1 (+http://www.picsearch.com/bot.html)","CrawlerBot"
"pverify/1.2","CrawlerBot"
"PWS.Kiosk - Content Filtering","CrawlerBot"
"QPCreep Test Rig ( We are not indexing, just testing )","CrawlerBot"
"QuepasaCreep ( crawler@quepasacorp.com )","CrawlerBot"
"QuepasaCreep v0.9.14","CrawlerBot"
"QuepasaCreep v0.9.13","CrawlerBot"
"reifier.org (admin@reifier.org)","CrawlerBot"
"reifier.org admin@reifier.org","CrawlerBot"
"rico/0.1","CrawlerBot"
"RixBot (http://www.oops-as.no/rix/)","CrawlerBot"
"RoboPal (http://www.findpal.com/)","CrawlerBot"
"RobotMidareru/0.7libwww-perl/5.65","CrawlerBot"
"Search Engine World Robots.txt Validator at http://www.searchengineworld.com/cgi-bin/robotcheck.cgi","CrawlerBot"
"Robozilla/1.0","CrawlerBot"
"RPT-HTTPClient/0.3-3","CrawlerBot"
"SafariBookmarkChecker/1.25 (+http://www.coriolis.ch/)","CrawlerBot"
"SafariBookmarkChecker/1.26 (+http://www.coriolis.ch/)","CrawlerBot"
"Scooter/1.0","CrawlerBot"
"Scooter-ARS-1.1","CrawlerBot"
"Scooter-3.0.FS - Altavista.com","CrawlerBot"
"Scooter/3.2","CrawlerBot"
"Scooter/3.2.SF0","CrawlerBot"
"Scooter_x0-3.2.EX","CrawlerBot"
"Scooter-3.2","CrawlerBot"
"Scooter-3.2.BT","CrawlerBot"
"Scooter-3.2.EX","CrawlerBot"
"Scooter-3.2.FNR","CrawlerBot"
"Scooter-3.2.PDF","CrawlerBot"
"Scooter-3.2.SF0","CrawlerBot"
"Scooter-3.2.TX.FNR","CrawlerBot"
"Scooter-3.2.XX0","CrawlerBot"
"Scooter/3.3","CrawlerBot"
"Scooter/3.3.QA","CrawlerBot"
"Scooter/3.3.QA.pczukor","CrawlerBot"
"Scooter/3.3.vscooter","CrawlerBot"
"Scooter/3.3_SF","CrawlerBot"
"Scrubby/2.1 (http://www.scrubtheweb.com/abs/meta-check.html)","CrawlerBot"
"Scrubby/2.2 (http://www.scrubtheweb.com/)","CrawlerBot"
"Search Agent 1.0","CrawlerBot"
"SearchSpider.com/1.1","CrawlerBot"
"Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3","CrawlerBot"
"Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)","CrawlerBot"
"semanticdiscovery/0.1","CrawlerBot"
"Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au)","CrawlerBot"
"sherlock/1.3 httpget/1.3","CrawlerBot"
"sherlock_spider (jimfan@163.com)","CrawlerBot"
"InternetSeer.com","CrawlerBot"
"sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com)","CrawlerBot"
"sitescooper/3.1.2 (http://sitescooper.org) libwww-perl/5.51","CrawlerBot"
"SiteXpert","CrawlerBot"
"SlySearch/1.3 (http://www.slysearch.com)","CrawlerBot"
"SlySearch/1.3 http://www.slysearch.com","CrawlerBot"
"sohu-search","CrawlerBot"
"Speedy Spider (http://www.entireweb.com)","CrawlerBot"
"Speedy_Spider_(http://www.entireweb.com)","CrawlerBot"
"Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)","CrawlerBot"
"SpiderKU/0.9","CrawlerBot"
"SpiderMonkey/7.04 (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)","CrawlerBot"
"Spider_Monkey/7.06 (SpiderMonkey.ca info at http://SpiderMonkey.ca /sm.shtml)","CrawlerBot"
"Spider_Monkey/7.06 (SpiderMonkey.ca info at http://www.spidermonkey.ca/sm.shtml)","CrawlerBot"
"Mozilla/5.0 (compatible; SpurlBot/0.2)","CrawlerBot"
"Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)","CrawlerBot"
"Star Downloader","CrawlerBot"
"Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)","CrawlerBot"
"Mozilla/4.0 (compatible; SuperCleaner 2.56; Windows NT 5.1)","CrawlerBot"
"Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;)","CrawlerBot"
"Szukacz/1.5","CrawlerBot"
"Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)","CrawlerBot"
"Tarantula Experimental Crawler","CrawlerBot"
"Tcl http client package 1.0","CrawlerBot"
"Tcl http client package 2.3","CrawlerBot"
"(Teradex Mapper; mapper@teradex.com; http://www.teradex.com)","CrawlerBot"
"Teradex_Crawler (crawler@teradex.com; http://crawler.teradex.com)","CrawlerBot"
"TheSuBot/0.1 (www.thesubot.de)","CrawlerBot"
"thesubot-beta-www.thesubot.de","CrawlerBot"
"thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)","CrawlerBot"
"timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html","CrawlerBot"
"Tkensaku/0.9 (http://www.tkensaku.com/q.html)","CrawlerBot"
"TranSGeniKBot (http://www.tsgk.net)","CrawlerBot"
"TranSGeniKBot http://www.tsgk.net","CrawlerBot"
"TulipChain/5.7 (http://ostermiller.org/tulipchain/) Java/1.4.0_02 (http://java.sun.com/) Windows_Me/4.90","CrawlerBot"
"TulipChain/5.94 (http://ostermiller.org/tulipchain/) Java/1.4.1_01 (http://apple.com/) Mac_OS_X/10.2.8","CrawlerBot"
"TulipChain/6.01 (http://ostermiller.org/tulipchain/) Java/1.4.2_03 (http://java.sun.com/) Windows_XP/5.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"TulipChain/6.02 (http://ostermiller.org/tulipchain/) Java/1.4.2_03 (http://apple.com/) Mac_OS_X/10.3.3 RPT-HTTPClient/0.3-3","CrawlerBot"
"TulipChain/6.03 (http://ostermiller.org/tulipchain/) Java/1.4.2_05 (http://java.sun.com/) Windows_XP/5.1 RPT-HTTPClient/0.3-3","CrawlerBot"
"TurnitinBot/1.4 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/1.4 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TurnitinBot/2.0 (http://www.turnitin.com/robot/crawlerinfo.html)","CrawlerBot"
"TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html","CrawlerBot"
"TutorGigBot/1.5 ( +http://www.tutorgig.info )","CrawlerBot"
"Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)","CrawlerBot"
"UdmSearch/3.1.20","CrawlerBot"
"UIowaCrawler/1.0","CrawlerBot"
"UIowaCrawler/2.0","CrawlerBot"
"unchaos_crawler_2.0.2 (search.engine@unchaos.com)","CrawlerBot"
"VM4050/132.037 UP.Browser/6.2.2.4.e.1.100 (GUI) MMP/2.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","CrawlerBot"
"updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om)","CrawlerBot"
"USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)","CrawlerBot"
"Vagabondo/2.0 MT (webagent at wise-guys dot nl)","CrawlerBot"
"Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)","CrawlerBot"
"Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)","CrawlerBot"
"Vivante Link Checker (http://www.vivante.com)","CrawlerBot"
"void-bot/0.1 (bot@void.be; http://www.void.be/)","CrawlerBot"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6","CrawlerBot"
"Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2","CrawlerBot"
"VSE/1.0 (vsecrawler@hotmail.com)","CrawlerBot"
"vspider","CrawlerBot"
"W3C_Validator/1.183 libwww-perl/5.64","CrawlerBot"
"W3C_Validator/1.305.2.109 libwww-perl/5.79","CrawlerBot"
"W3C_Validator/1.305.2.12 libwww-perl/5.64","CrawlerBot"
"W3C_Validator/1.305.2.137 libwww-perl/5.79","CrawlerBot"
"W3C_Validator/1.305.2.148 libwww-perl/5.800","CrawlerBot"
"W3C_Validator/1.305.2.148 libwww-perl/5.803","CrawlerBot"
"W3C-checklink/2.90 libwww-perl/5.64","CrawlerBot"
"W3C-checklink/3.6.2.3 libwww-perl/5.64","CrawlerBot"
"W3C-checklink/3.9.2 [3.17] libwww-perl/5.79","CrawlerBot"
"W3C-checklink/4.0 [4.4] libwww-perl/5.800","CrawlerBot"
"W3C-checklink/4.1 [4.14] libwww-perl/5.800","CrawlerBot"
"webbot","CrawlerBot"
"Webclipping.com","CrawlerBot"
"webcollage/1.102","CrawlerBot"
"webcollage/1.104","CrawlerBot"
"webcollage/1.87","CrawlerBot"
"webcollage/1.93","CrawlerBot"
"webcollage/1.94","CrawlerBot"
"Thu Mar 27 18:20:34 CET 2003WebcraftBoot","CrawlerBot"
"Fri Nov 15 04:51:18 EST 2002WebcraftBoot Java/1.4.1_01","CrawlerBot"
"Sun Apr 20 22:00:01 EDT 2003WebcraftBoot Java/1.4.2-beta","CrawlerBot"
"Tue Apr 15 22:00:03 EDT 2003WebcraftBoot Java/1.4.2-beta","CrawlerBot"
"WebFilter Robot 1.0","CrawlerBot"
"Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56)","CrawlerBot"
"WebRACE/1.1 (University of Cyprus, Distributed Crawler)","CrawlerBot"
"WebSauger 1.20b","CrawlerBot"
"http://www.websearch.com.au (larbin2.6.2@unspecified.mail)","CrawlerBot"
"http://www.websearch.com.au larbin2.6.2@unspecified.mail","CrawlerBot"
"http://www.WebSearch.com.au/ (larbin2.6.2@unspecified.mail)","CrawlerBot"
"http://www.WebSearch.com.au/ larbin2.6.2@unspecified.mail","CrawlerBot"
"www.WebSearch.com.au (search@websearch.com.au)","CrawlerBot"
"www.WebSearch.com.au search@websearch.com.au","CrawlerBot"
"WebSearch/2.0.1 (Dez@Blanchfield.COM.AU, http://www.WebSearch.com.au/)","CrawlerBot"
"WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU)","CrawlerBot"
"http://www.WebSearch.com.au/ - Australian Search Engine/3.1.3 (sites@websearch.com.au)","CrawlerBot"
"http://www.WebSearch.com.au/ - Australian Search Engine/3.1.6 (sites@websearch.com.au)","CrawlerBot"
"www.webwombat.com.au","CrawlerBot"
"webyield robot (http://www.webyield.net/search/search.pl)","CrawlerBot"
"Wget/1.5.2","CrawlerBot"
"Wget/1.5.3","CrawlerBot"
"Wget/1.5.3.1","CrawlerBot"
"Wget/1.6","CrawlerBot"
"Wget/1.7","CrawlerBot"
"Wget/1.8","CrawlerBot"
"Wget/1.8.1","CrawlerBot"
"Wget/1.8.1+cvs","CrawlerBot"
"Wget/1.8.2","CrawlerBot"
"Wget/1.9","CrawlerBot"
"Wget/1.9-beta","CrawlerBot"
"Wget/1.9.1","CrawlerBot"
"Willow Internet Crawler by Twotrees V2.1","CrawlerBot"
"Wotbox/alpha0.5.1 (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02","CrawlerBot"
"http://www.ciml.co.uk","CrawlerBot"
"WWWeasel Robot v1.00 (http://wwweasel.de)","CrawlerBot"
"Xenu''s Link Sleuth 1.1a","CrawlerBot"
"Xenu Link Sleuth 1.2b","CrawlerBot"
"Xenu Link Sleuth 1.2d","CrawlerBot"
"Xenu Link Sleuth 1.2e","CrawlerBot"
"Xenu Link Sleuth 1.2f","CrawlerBot"
"Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)","CrawlerBot"
"Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)","CrawlerBot"
"YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine","CrawlerBot"
"Zao/0.1 (http://www.kototoi.org/zao/)","CrawlerBot"
"Zao/0.2 (http://www.kototoi.org/zao/)","CrawlerBot"
"Zao-Crawler","CrawlerBot"
"Zeus 3140 Webster Pro V2.9 Win32","CrawlerBot"
"Zeus 57657 Webster Pro V2.9 Win32","CrawlerBot"
"ZipppBot/0.11 (ZipppBot; http://www.zippp.net; webmaster@zippp.net)","CrawlerBot"
"ZoomSpider - wrensoft.com","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-1.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-12.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (wn-2.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d03 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d05 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker Beta-d01 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
"Mozilla/4.0 compatible ZyBorg/1.0 DLC (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)","CrawlerBot"
0
 
LVL 12

Expert Comment

by:Sinoj Sebastian
ID: 18754309
> shortest and most effective list wins
So the list of all popular bot agent strings

    *Bot*

Most of the spider user-agent strings will have the substring "Bot" in it.
Ordinary user-agent string from IE, FF, NS etc do not contain "Bot".
Try filter using this.
I'm already using it.

From the above list I found That "CrawlerBot" is also a substring of bot agent strings

0
 
LVL 29

Expert Comment

by:rdivilbiss
ID: 18757241
No, in the list above, "CrawlerBot" is not part of the user agent string, it was a field in the database of my collection of live user agents taken from dozens of my web sites and hand categorized.  Ignore: ,"CrawlerBot"
0
 
LVL 16

Author Comment

by:OliWarner
ID: 18757392
Yeah I can parse those out without issue. Thanks Rod that looks like it'll do the job perfectly
0
 
LVL 29

Expert Comment

by:rdivilbiss
ID: 18758475
Those were as of January. There could always be a few new ones cropping up.
0

Featured Post

Gigs: Get Your Project Delivered by an Expert

Select from freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Remove lines by logo 2 29
Remove third quote mark from widget 6 23
Parsing an RSS Feed 4 15
JavaScript behaviour different on local machine and network share 11 44
Why do we like using grid based layouts in website design? Let's look at the live examples of websites and compare them to grid based WordPress themes.
Today, still in the boom of Apple, PC's and products, nearly 50% of the computer users use Windows as graphical operating systems. If you are among those users who love windows, but are grappling to keep the system's hard drive optimized, then you s…
The viewer will get a basic understanding of what section 508 compliance can entail, learn about skip navigation links, alt text, transcripts, and font size controls.
Learn how to create flexible layouts using relative units in CSS.  New relative units added in CSS3 include vw(viewports width), vh(viewports height), vmin(minimum of viewports height and width), and vmax (maximum of viewports height and width).

813 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now