123people|123people|
80legs|www.80legs.com/webcrawler|
@mail.ru|Mail.ru|
AboutUs|AboutUsBot|
Acoon|AcoonBot|
AideRSS|aiderss.com|
AgHaven|aghaven|
Ahrefs|AhrefsBot|
Alexa|ia_archiver|
AltaVista|Scooter-|
Ant.com|AntBot|
Apercite|apercite.fr/robot|
Apple PubSub|Apple-PubSub|
Archive.org|archive.org_bot|
Ask|Jeeves/Teoma|
BackType|kame|
BackType|support@backtype.com|
Baidu|Baiduspider|
BecomeBot|BecomeBot|
Best Buy Ware Bay|WBSearchBot|
Bibliothèque Nationale de France|bnf.fr_bot|
Bing|www.bing.com|
Bing|MSMOBOT|
Bing|msnbot|
Bit.ly|bitlybot|
Biz360|Biz360|
Blekko|Blekkobot|
Blogged|blogged_crawl|
Bloggsi|Bloggsi|
Bloglines|Bloglines|
BlogPulse|BlogPulse|
BlogScope|BlogScope|
BlogShares.com|Zipping Along V|
Bluecoat|Bluecoat|
Bookmark Sync2it|sync2it.com|
Brandwatch Magpie-Crawler|brandwatch.net|
Browsershots|Browsershots|
BuzzLogic|BuzzLogic|
Camdiscover|Camcrawler|
Camdiscover|www.camdiscover.com|
CatchBot|CatchBot|
Cazoodle|Cazoodle|
CCBot|CCBot|
CCBot|commoncrawl.org/bot.html|
ChekParams|DCPbot|
Cityreview|cityreview|
Corpora from the web|berlin-fu-cow|
Covario|Covario-IDS|
crowsnest|crowsnest|
Cuil|Twiceler|
cURL|curl|
Discovery Engine|discobot|
Discovery Engine|disco/Nutch|
DomainCrawler|DomainCrawler|
DotBot|DotBot|
Echo|JS-Kit|
Entireweb|entireweb.com|
Evri|evri.com|
Exalead|Exabot|
Exalead|NG/2.0|
Ezooms|ezooms.bot@gmail.com|
Facebook|facebook|
Fastsearch|FASTMetaWebCrawler|
Fastsearch|fastsearch|
FeedBurner|FeedBurner|
FeedDemon|www.feeddemon|
Feed Validator|FeedValidator/1.3|
Float|float.com|
FyberSpider|FyberSpider|
Garlik|GarlikCrawler|
Gaisbot|Gaisbot|
Genieo|www.genieo|
Gigablast|gigablast|
Gigablast|Gigabot|
Gnip|UnwindFetchor|
Google AdSense|Mediapartners-Google|
Google Desktop|GoogleDesktop|
Google Feedfetcher|Feedfetcher-Google|
Google Images|Googlebot-Image|
Google Mobile|Googlebot-Mobile|
Google Site Verification|Google-Site-Verification|
Google Page Speed|GooglePageSpeed|
Google|Googlebot|
GrapeShot|GrapeshotCrawler|
GTmetrix|GTmetrix|
Guruji|GurujiBot|
Heritrix|heritrix|
HotPage|www.hotpage.fr|
HTTrack Website Copier|HTTrack|
ICC-Crawler|ICC-Crawler|
Icerocket|BlogSearch|
ichiro|ichiro|
iltrovatore.it|IlTrovatore-Setaccio|
Indian Language Web|IIITBOT|
Indy Library|IndyLibrary|
IntegromeDB|integromedb.org/Crawler|
IRLbot|IRLbot|
Jike Search|JikeSpider|
Jyxobot|jyxo.cz|
KaloogaBot|KaloogaBot|
KiwiStatus|KiwiStatus|
LinkedIn|www.linkedin.com|
Linkdex|linkdex.com|
linkfluence|bot@linkfluence.net|
Livedoor|livedoorScreenShot|
llssbot|llssbot|
LookSmart|grub-client|
MagpieRSS|MagpieRSS|
Majestic-12|MJ12bot|
Megite|Megite2.0|
METASpider|METASpider|
MetaURI|metauri.com|
Microsoft Bot|MSRBOT|
Microsoft adCenter|adidxbot|
MLBot|metadatalabs.com|
Moreover|Moreoverbot|
MP3 Bot|mp3realm.org|
Naver|Cowbot|
Naver|naver.com|
Naver|NaverBot|
Neofonie|neofoniesearch:robot|
Netcraft|Netcraft|
Netseer|Teemer|
NetSeer|NetSeer|
NetSeer|ilial/Nutch|
NewsGator|NewsGatorOnline|
News.me|news.me|
Nexen|www.nexen.net|
Online Broken Link Checker|BrokenLinkCheck.com|
OpenIndex|OpenindexSpider|
Orange|ldorange.crawler|
Outbrain|SphereScout|
PagePeeker|pagepeeker|
Paper.li|PaperLiBot|
Peepow|Peepowbot|
Picsearch|www.picsearch.com|
Ping.Blo.gs|ping.blo.gs|
PlantyNet|PlantyNet|
PostRank|postrank.com|
ProCog|ProCogBot|
Proximic|www.proximic|
Python|Python-urllib/2.6|
Radian6|R6_CommentReader|
Radian6|R6_FeedFetcher|
Radian6 radiarss|www.radian6.com|
RedBot|redbot.org|
Rivva|Rivva|
Runnk|runnk|
ScoutJet|www.scoutjet.com|
ScSpider|ScSpider|
Searchme|searchme.com|
Search Web Engine|swebot-crawler.net|
Seexie|seexie.com|
SEOprofiler|www.seoprofiler.com|
Seznam|SeznamBot|
SEMRush|SemrushBot|
SimplePie|SimplePie|
Sindup|Sindup|
Sistrix|crawler.sistrix|
Site Bot|sitebot|
Site-Perf.com|Site-Perf.comperformancetestingbot|
SiteIntel|SiteIntel.net|
Skimlinks|skimlinks|
Snap|Snap|
Sogou|Sogou|
Solomono|SolomonoBot|
Sosospider|soso.com|
SpiderLing|biwec|
Spinn3r|Spinn3r|
Spoke|SpokeSpider|
Summify|Summify|
Synoo|Synoobot|
Spinn3r|Tailrank|
Technorati|Technoratibot|
Thumbshots|thumbshots|
Topsy|topsy.com|
Trendiction|trendictionbot|
Turnitin|turnitin|
Tweeted Times|tweetedtimes|
tweetmeme|tweetmeme.com|
Twenga|TwengaBot|
Twikle|contact@twikle.com|
Twingly|TwinglyRecon|
Twitter|Twitterbot|
Twitmunin|twitmunin|
Twitturls|twitturls|
Twitturly|Twitturly|
UMBC Memeta|UMBC-Memeta|
UniversalFeedParser|UniversalFeedParser|
Updowner|Updownerbot|
urlfan|urlfan-bot|
urlfan|www.urlfan.com|
VadixBot|VadixBot|
VerticalSearchWorks|ConveraCrawler|
WiseGuys|wise-guys|
vBSEO|vBSEO|
Visvo|Visvo|
Voila|VoilaBot|
W3C Validator|W3C_Validator|
W3C Unicorn|W3C_Unicorn|
W3C CSS validation|Jigsaw/2.2.5W3C_CSS_Validator_JFouffa|
W3C Link Checker|W3C-checklink|
W3C Internationalization Checker|W3C_I18n-Checker|
WASALive|blog.wasalive.com|
WASALive|wasalive-bots|
WebAlta|WebAltaCrawler|
Weive|weive|
WhatWeb|WhatWeb|
Whitevector|whitevector|
Wikio|wikio|
Wikiwix|wikiwix-bot|
WordPress|WordPress|
Worio|woriobot|
Xenu Link Sleuth|XenuLink|
xMind|xMind|
YaCy|yacybot|
Yahoo Japan|Y!J-|
Yahoo|MMCrawler|
Yahoo|slurp@inktomi|
Yahoo|YahooCacheSystem|
Yahoo|yahoo.com|
Yahoo|YahooFeedSeeker|
Yahoo|Yahoo!Slurp|
Yahoo|Yahoo-Test|
Yahoo|Yahoo-Yahoo-YSM|
Yandex|yandex|
Yanga|Yanga|
Yodao|YodaoBot|
Youdao|YoudaoBot|
Zooka|Zookabot|
Unknown Spam Bot|Mozilla/4.0(compatible;)|
Unknown Spam Bot|Mozilla/4.0(compatible;ICS)|
Unknown Spam Bot|Mozilla/5.001|
Unknown Spam Bot|Mozilla/0.6Beta(Windows)|
Unknown Spam Bot|Mozilla/0.91Beta(Windows)|
Unknown Spam Bot|Java/1.6.0_20|
Unknown Spam Bot|Java/1.6.0_26|
Unknown Spam bot|python-requests/0.12.1|
Unknown Spam bot|python-requests/0.13.2|