Wikimedia Traffic Analysis Report - Crawler requests

Monthly requests or daily averages, for period: 1 Jul 2012 - 30 Jul 2012 (last 12 months)
000 ⇒ k
 

 This analysis is based on a 1:1000 sampled server log (squids)

 See also: Requests by destination or by origin / Methods / Scripts / User agents / Skins / Crawlers / Op.Sys. / Mobile devices / Browsers / Google / Country data / Traffic trends, and notes about reliability of these data

The following overview of crawler (aka bot) page requests is based on the user agent information that accompanies most server requests. Unfortunately this user agent information follows rather loosely defined guidelines.
Also please bear in mind than the most popular crawler names may be somewhat overrepresented. This is the result of so called user agent spoofing (where a requester supplies false credentials, e.g. to bypass web servers filters).
GoogleBot seems to be a favorite for spoofing. Therefore requests from an ip address registered by Google (see below) are color coded GoogleBot, others GoogleBot

For this report page requests are considered to be issued by a crawler in two cases:
1 The user agent string contains a web address (only crawlers should have that, but there a some false positives, where a browser sends a user agent string with a web address (ill behaved plug-in, main offenders have been eliminated)
2 The user agent string contains the term bot, spider or crawl[er]'

In total 73,289,100 page requests (mime type text/html only!) per day are considered crawler requests, out of 454,144,070 external requests, which is 16.1%

Page requests for crawlers that specify a url in the agent string
Count
x 1000
Secondary domain
(~site) name
URLMime typeUser agent
google
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 www.google.com/bot.htmltext/..SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 www.google.com/feedfetcher.html-FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortografia4)
 www.google.com/feedfetcher.htmlapplication/xmlFeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien3)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ortopedianew)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~cloudcrawling)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 www.google.com/feedfetcher.htmlapplication/jsonMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wikien4)
 code.google.com/p/crawler4j/text/..crawler4j (url)
 www.google.com/feedfetcher.htmltext/..FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: rarplayer)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: s~senchaiosrc)
 desktop.google.com/text/..Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: s~redconceptual)
 code.google.com/appengineimage/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 AppEngine-Google; (url; appid: s~fonetika3)
 www.google.com/feedfetcher.htmlapplication/xmlMozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: abdulfat)
 www.google.com/feedfetcher.htmltext/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
 code.google.com/appenginetext/..WikiBot/0.1 AppEngine-Google; (url; appid: newikipedia)
 desktop.google.com/application/xmlMozilla/5.0 (compatible; Google Desktop/5.9.909.30391; url)
 code.google.com/appengineapplication/xmlAppEngine-Google; (url; appid: wikipedia-raw)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: usawebdl)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: azamasmadi)
 www.google.com/feedfetcher.htmlimage/..Mozilla/5.0 (compatible) FeedFetcher-Google;(url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki3)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~keytanwiki4)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pakgalaxy)
 code.google.com/appengineimage/..AppEngine-Google; (url; appid: d24-img)
 www.google.com/coop/cse/creftext/..FeedFetcher-Google-CoOp; (url)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 desktop.google.com/-Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: d24-img)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~japantiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kires-roxy)
 code.google.com/appenginetext/..Mozilla/5.0 AppEngine-Google; (url; appid: s~app3123ak)
 www.google.com/bot.htmlimage/..DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 code.google.com/appengineapplication/jsonMozilla 4.0 AppEngine-Google; (url; appid: prfleme)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: hao1-prxoy)
 code.google.com/appenginetext/..www.productontology.org/1.0 (Contact: mail address ) AppEngine-Google; (url; appid: gr4bing)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~wikigraph2)
 code.google.com/appenginetext/..Offline Mobile Wiki (Tel:44 141 334 5472, mail address ) AppEngine-Google; (url; appid: wiki2go)
 code.google.com/p/rondaapplication/jsonRonda - url
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: worldwide-propaganda)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~senchaiosrc)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~francetiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: mehproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~proxyseekkety)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: simple-tools2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~drizzlprox)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: tusawebproxy4)
 www.google.com/feedfetcher.html-Mozilla/5.0 (compatible) FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: nagarajhubli-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: dkoxyserv)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: 114proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: threewiki)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: burmansearch)
 desktop.google.com/image/..Mozilla/5.0 (compatible; Google Desktop/5.9.911.3589; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxyusing121)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: kbworld24)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: proxy-devakishor)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: davrasaurs)
 code.google.com/appengineapplication/jsonMWBOT GAE Edition AppEngine-Google; (url; appid: philip-bot)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ageryder)
 code.google.com/appengineapplication/jsonAppEngine-Google; (url; appid: prfleme)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: toom16-10)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-2)
 code.google.com/p/rondatext/..Ronda - url
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: betafxserver)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: ivegotalovelybunch)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: thetechnolust)
 docs.google.comimage/..Mozilla/5.0 (compatible; GoogleDocs; documents; url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: pazvantoff)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wwwwebp2)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: python-proxy-server)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: bie99miracle)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: atxproxy)
 code.google.com/appenginetext/.. mail address AppEngine-Google; (url; appid: s~wiki-sherpa)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: boxapp)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: paradigm-web-proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vi-mobile)
 www.google.com/feedfetcher.htmlimage/..FeedFetcher-Google; (url)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webponline9)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~crowdsurfer100)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webusadlp9)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: adrianswebproxy)
 code.google.com/appenginetext/..Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) AppEngine-Google; (url; appid: s~opds-catalog-test)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: web4proxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: vebproxy)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: openeyeproxy)
 code.google.com/appenginetext/..Opera/9.80 (Windows NT 6.1; ru) Presto/2.9.168 Version/11.52 AppEngine-Google; (url; appid: s~sib-watchtower)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: chris-homework-helper)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: webproxy8-9)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: wagagate)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: s~tpbitalia)
 code.google.com/appenginetext/..AppEngine-Google; (url; appid: varlopie)
facebook
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.0 (url)
 www.facebook.com/externalhit_uatext.phptext/..facebookexternalhit/1.1 (url)
 developers.facebook.comimage/..facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.phpimage/..facebookexternalhit/1.1 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.0 (url)
 developers.facebook.com-facebookplatform/1.0 (url)
 developers.facebook.comtext/..facebookplatform/1.0 (url)
 www.facebook.com/externalhit_uatext.php-facebookexternalhit/1.1 (url)
bing
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htm-Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b3
 www.bing.com/bingbot.htmimage/..Mozilla/5.0 (compatible; bingbot/2.0; url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) ASProxy/5.5b5
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: wxcity1)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk8)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible;bingbot/2.0;url)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxydisk9)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: surf603)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) (via Web-Blaster/2.21 (http://www.assoziations-blaster.de/web-blast.html))
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: appcodes)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: yourrevenues)
 www.bing.com/bingbot.htmtext/..Mozilla/5.0 (compatible; bingbot/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: howtodoita)
google?
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmltext/..GoogleBot/2.1 (url)
 www.google.com/bot.html-Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlimage/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmlapplication/jsonMozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (iPhone; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmltext/..Mozilla/5.0 (compatible; GoogleBot/2.1; url)
 www.google.com/bot.html-SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; GoogleBot-Mobile/2.1; url)
 www.google.com/bot.htmlapplication/xmlMozilla/5.0 (compatible; GoogleBot/2.1; url)
baidu
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.html-Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmtext/..Baiduspider-image(url)
 www.baidu.com/search/spider.htmlapplication/vnd.php.serializedMozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-9)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-8)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-3)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-5)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-4)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-0)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-1)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-2)
 www.baidu.com/search/spider.htmlapplication/xmlMozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-7)
 www.baidu.com/search/spider.htmltext/..Mozilla/5.0 (compatible; Baiduspider/2.0; url) AppEngine-Google; (http://code.google.com/appengine; appid: proxyi-6)
 www.baidu.com/search/spider.htmlimage/..Mozilla/5.0 (compatible; Baiduspider/2.0; url)
 www.baidu.com/search/spider.htmtext/..Baiduspider(url)
yahoo
 help.yahoo.com/help/us/ysearch/slurpimage/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 listing.yahoo.co.jp/support/faq/int/other/other_001.htmltext/..Y!J-BRJ/YATS crawler (url)
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmlimage/..'Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; url))'
 developer.yahoo.com/yql/providertext/..Mozilla/5.0 (compatible; Yahoo Pipes 2.0; url) Gecko/20090729 Firefox/3.5.2
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRI/0.0.1 crawler ( url )
 help.yahoo.com/help/us/ysearch/slurptext/..Mozilla/5.0 (compatible; Yahoo! Nano; url)
 help.yahoo.com/help/us/ysearch/slurpapplication/jsonMozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRT/1.0 crawler (url)
 help.yahoo.com/help/us/ysearch/slurpapplication/vnd.php.serializedMozilla/5.0 (compatible Yahoo! Slurp/3.0 url)
 www.yahoo.com/index.htmlimage/..Mozilla/5.0 [en] (X11; Linux 2.2.15 i686 url)
 help.yahoo.com/help/us/ysearch/slurp-Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; url)
 help.yahoo.co.jp/help/jp/search/indexing/indexing-15.htmltext/..Y!J-BRW/1.0 crawler (url)
yandex
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/bots-Mozilla/5.0 (compatible; YandexBot/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexDirect/3.0; url)
 yandex.com/botstext/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImageResizer/2.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexImages/3.0; url)
 yandex.com/botsimage/..Mozilla/5.0 (compatible; YandexBot/3.0; url)
naver
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/-Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/image/..Yeti/1.0 (NHN Corp.; url)
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b3
 help.naver.com/robots/text/..Yeti/1.0 (NHN Corp.; url) ASProxy/5.5b5
 corp.naver.jp/text/..Mozilla/5.0 (compatible; NaverJapan/1.0; url)
80legs
 www.80legs.com/webcrawler.htmltext/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
 www.80legs.com/webcrawler.html-Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
 www.80legs.com/webcrawler.htmlimage/..Mozilla/5.0 (compatible; 008/0.83; url) Gecko/2008032620
msn
 search.msn.com/msnbot.htmtext/..msnbot/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmimage/..msnbot-media/1.1 (url)
 search.msn.com/msnbot.htmtext/..msnbot-UDiscovery/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-NewsBlogs/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot-Products/1.0 (url)
 search.msn.com/msnbot.htm-msnbot-media/1.1 (url)
 search.msn.com/msnbot.htm-msnbot/0.01 (url)
 search.msn.com/msnbot.htm-msnbot/2.0b (url)
 search.msn.com/msnbot.htmtext/..msnbot/0.01 (url)
 search.msn.com/msnbot.htmimage/..msnbot-NewsBlogs/2.0b (url)
genieo
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url)
 www.genieo.com/webfilter.htmlapplication/xmlMozilla/5.0 (compatible; Genieo/1.0 url)
 www.genieo.com/webfilter.htmlimage/..Mozilla/5.0 (compatible; Genieo/1.0 url)
 www.genieo.com/webfilter.htmltext/..Mozilla/5.0 (compatible; Genieo/1.0 url
blekko
 blekko.com/about/blekkobottext/..Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
 blekko.com/about/blekkobot-Mozilla/5.0 (compatible; Blekkobot; ScoutJet; url)
youdao
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/-Mozilla/5.0 (compatible; YoudaoBot/1.0; url; )
 www.youdao.com/help/webmaster/spider/image/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
 www.youdao.com/help/webmaster/spider/text/..Mozilla/5.0 (compatible;YodaoBot-Image/1.0;url;)
 toolbar.youdao.com/image/..Youdao Toolbar (url)
soso
 help.soso.com/webspider.htmtext/..Sosospider(url)
 help.soso.com/webspider.htm-Sosospider(url)
 help.soso.com/webspider.htmapplication/xmlSosospider(url)
cibra
 cibra.de/text/..CiBra Data Collector (url)
sblog
 fulltext.sblog.cz/screenshot/image/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/text/..SeznamBot/3.0 (url)
 fulltext.sblog.cz/screenshot/text/..Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; url)
 fulltext.sblog.cz/-SeznamBot/3.0 (url)
www.
 www.text/..GoogleBot/2.1 ( urlGoogleBot.com/bot.html)
 www.text/..GoogleBot-Image/1.0 ( urlGoogleBot.com/bot.html)
 www.text/..GoogleBot/2.1 (urlGoogleBot.com/bot.html)
echonest
 the.echonest.com/reader/application/xmlnestReader/0.3 (discovery; url; reader at echonest.com)
 the.echonest.com/reader/text/..nestReader/0.3 (discovery; url; reader at echonest.com)
php
 pear.php.net/application/vnd.php.serializedPEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/0.5.2 (url) PHP/5.2.17
 pear.php.net/text/..PEAR HTTP_Request class ( url )
 pear.php.net/image/..PEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2text/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.14
 pear.php.net/package/http_request2application/xmlHTTP_Request2/2.0.0 (url) PHP/5.3.8
 pear.php.net/application/xmlPEAR HTTP_Request class ( url )
 pear.php.net/package/http_request2image/..HTTP_Request2/2.1.1 (url) PHP/5.3.2-1ubuntu4.15
wwwgogetpapers
 wwwgogetpapers.com/application/jsonUser-Agent: GoGetPapersBot (url)
dataparksearch
 dataparksearch.org/bottext/..DataparkSearch/4.54-26052011 (url)
exabot
 www.exabot.com/go/robottext/..Mozilla/5.0 (compatible; Exabot/3.0; url)
 www.exabot.com/go/robot-Mozilla/5.0 (compatible; Exabot/3.0; url)
discoveryengine
 discoveryengine.com/discobot.htmltext/..Mozilla/5.0 (compatible; discobot/2.0; url)
 discoveryengine.com/discobot.html-Mozilla/5.0 (compatible; discobot/2.0; url)
 discoveryengine.com/discobot.htmlimage/..Mozilla/5.0 (compatible; discobot/2.0; url)
wordpress
 klausgauger.wordpress.comtext/..WordPress/3.5-alpha; url
 josefboberg.wordpress.comtext/..WordPress/3.5-alpha; url
 gandalf1316.wordpress.comtext/..WordPress/3.5-alpha; url
 kterrl.wordpress.comtext/..WordPress/3.5-alpha-21304; url
 fredvidal.wordpress.comtext/..WordPress/3.5-alpha; url
 klausgauger.wordpress.comtext/..WordPress/3.5-alpha-21304; url
 imagenssagradas.wordpress.comtext/..WordPress/3.5-alpha; url
 support.wordpress.com/contact/text/..WordPress.com mShots; url
 gunnyg.wordpress.comtext/..WordPress/3.5-alpha; url
 kterrl.wordpress.comtext/..WordPress/3.5-alpha-21273; url
ahrefs
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
 ahrefs.com/robot/text/..Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/3.0; url)
 ahrefs.com/robot/-Mozilla/5.0 (compatible; AhrefsBot/3.1; url)
wikipedia
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.19.0 url
 en.wikipedia.org/wiki/User:NicoV/Wikipedia_Cleaner/Documentationtext/..WPCleaner (url)
 en.wikipedia.orgtext/..url
 en.wikipedia.org/wiki/Wikipedia:Huggletext/..Huggle/2.1.18.0 url
 fr.wikipedia.org/wiki/Utilisateur:Salebotapplication/jsonSalebot, see url (uses Perl MediaWiki::API)
sogou
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07-Sogou web spider/4.0(url)
 www.sogou.com/docs/help/webmasters.htm#07text/..Sogou Pic Spider/3.0(url)
bin-co
 www.bin-co.com/php/scripts/load/text/..BinGet/1.00.A (url)
 www.bin-co.com/php/scripts/load/application/vnd.php.serializedBinGet/1.00.A (url)
majestic12
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.3; url)
 www.majestic12.co.uk/bot.php?text/..Mozilla/5.0 (compatible; MJ12bot/v1.4.2; url)
yacy
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.3.8-gentoo; java 1.6.0_33; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.24-28-server; java 1.6.0_18; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-26-generic; java 1.6.0_24; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-26-generic; java 1.6.0_24; Indian/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; America/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-26-generic; java 1.6.0_24; Indian/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows Server 2008 6.0; java 1.7.0_04; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.34-gentoo-r12; java 1.6.0_33; UTC/en) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.3.8-gentoo; java 1.6.0_33; Europe/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_31; Etc/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.1.10-1.16-desktop; java 1.6.0_24; Europe/nl) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_31; Etc/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.0-27-generic; java 1.6.0_24; Indian/en) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.2.13-grsec-xxxx-grs-ipv6-64; java 1.6.0_26; America/fr) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; x86 Windows Server 2008 6.0; java 1.6.0_33; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_05; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) url
 yacy.net/bot.htmltext/..yacybot (freeworld/global; amd64 Linux 3.1.10-1.13-desktop; java 1.6.0_24; Europe/de) url
 yacy.net/bot.html-yacybot (freeworld/global; amd64 Linux 3.2.0-27-generic; java 1.6.0_24; Indian/en) url
numberfound
 www.numberfound.it/text/..MyBot/1.0 (url)
 www.numberfound.com/text/..MyBot/1.0 (url)
sistrix
 crawler.sistrix.net/text/..Mozilla/5.0 (compatible; SISTRIX Crawler; url)
toolserver
 wiki.toolserver.org/view/GeoHacktext/..Geohack (url)
 toolserver.org/~dispenser/text/..DispensersTools (url)
 toolserver.org/~dispenser/application/jsonDispensersTools (url)
 toolserver.org/~para/cgi-bin/kmlexporttext/..url libwww-perl/6.02
yioop
 www.yioop.com/bot.phptext/..Mozilla/5.0 (compatible; YioopBot; url)
 www.yioop.com/bot.phpimage/..Mozilla/5.0 (compatible; YioopBot; url)
zum
 help.zum.com/inquirytext/..ZumBot/1.0 (ZUM Search; url)
 help.zum.com/inquiryimage/..ZumBot/1.0 (ZUM Search; url)
okian
 www.okian.ro/text/..MyBot/1.0 (url)
flipboard
 flipboard.com/browserproxyimage/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; url)
 flipboard.com/browserproxytext/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
 flipboard.com/browserproxyapplication/jsonMozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.1; url)
 flipboard.com/browserproxyimage/..null (FlipboardProxy/1.1; url)
 flipboard.com/browserproxy-Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; url)
daum
 tab.search.daum.net/aboutWebSearch.htmltext/..Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; url) Daumoa/3.0
wikidict
 www.wikidict.detext/..url
archive-it
 archive-it.org/files/site-owners.htmlimage/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
 archive-it.org/files/site-owners.html-Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
 archive-it.org/files/site-owners.htmltext/..Mozilla/5.0 (compatible; archive.org_bot; Archive-It; url)
jike
 shoulu.jike.com/spider.htmltext/..Mozilla/5.0 (compatible; JikeSpider; url)
 shoulu.jike.com/spider.htmlimage/..Mozilla/5.0 (compatible; JikeSpider; url)
 shoulu.jike.com/spider.html-Mozilla/5.0 (compatible; JikeSpider; url)
speaktoit
 www.speaktoit.comapplication/jsonSpeaktoit url
cognarius
 cognarius.comapplication/jsonAppsArlak/1.0 (url)
 cognarius.comtext/..AppsArlak/1.0 (url)
archive
 www.archive.org/details/archive.org_bottext/..Mozilla/5.0 (compatible; archive.org_bot url)
 archive.org/details/archive.org_botimage/..Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120118.092903 url)
FeedBurner
 www.FeedBurner.comtext/..FeedBurner/1.0 (url)
goo
 help.goo.ne.jp/contact/text/..goo wikipedia (url)
 help.goo.ne.jp/door/crawler.htmltext/..ichiro/3.0 (url)
 search.goo.ne.jp/option/use/sub4/sub4-1/-DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
 search.goo.ne.jp/option/use/sub4/sub4-1/text/..DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; url)
 help.goo.ne.jp/help/article/1142text/..ichiro/3.0 (url)
traslated
 mymemory.traslated.net/doc/text/..Mozilla/5.0 (MyMemory Bot url)
sf
 liferea.sf.net/text/..Liferea/0.x.x (Linux; en_US.UTF-8; url)
 magpierss.sf.nettext/..MagpieRSS/0.7x (url)
 liferea.sf.net/text/..Liferea/1.x.x (Linux; es_ES.UTF-8; url)
tweetmeme
 tweetmeme.com/text/..Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
 tweetmeme.com/-Mozilla/5.0 (compatible; TweetmemeBot/2.11; url)
gnip
 www.gnip.com/text/..UnwindFetchor/1.0 (url)
 www.gnip.com/-UnwindFetchor/1.0 (url)
kosmix
 www.kosmix.com/html/kosmos.htmlapplication/xmlMozilla/5.0(compatible;Kosmos/1.0;url)
 www.kosmix.com/html/kosmos.htmltext/..Mozilla/5.0(compatible;Kosmos/1.0;url)
github
 github.com/pauldix/typhoeus/tree/mastertext/..Typhoeus - url
 github.com/NeilCrosby/wikislurpapplication/vnd.php.serializedWikiSlurp (url)
 github.com/edsu/wikitweetsapplication/jsonwikitweets <url
federatedmedia
 federatedmedia.nettext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
apercite
 www.apercite.fr/robot/index.htmlimage/..Mozilla/5.0 (compatible; Apercite; url)
accelobot
 www.accelobot.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
whatrhymeswith
 www.whatrhymeswith.com/site/rhyme-bottext/..RhymeBot/0.1 (url)
newsgator
 www.newsgator.comtext/..NewsGatorOnline/2.0 (url; 1 subscribers)
 www.newsgator.com/text/..FeedDemon/2.7 (url; Microsoft Windows XP)
avantbrowser
 www.avantbrowser.comtext/..Avant Browser (url)
 www.avantbrowser.comtext/..Advanced Browser (url)
wikiglass
 wikiglass.comtext/..url : mail address
feedshow
 www.feedshow.comtext/..FeedshowOnline (url)
 www.feedshow.comtext/..Feedshow/x.0 (url; 1 subscriber)
jetbrains
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 2.0 Release Candidate 1 (url)
 www.jetbrains.com/omea_reader/text/..JetBrains Omea Reader 1.0.x (url)
emining
 emining.jp/text/..emBot-GalaBuzz/Nutch-1.0 (url; mail address )
 emining.jp/-emBot-GalaBuzz/Nutch-1.0 (url; mail address )
enwp
 enwp.org/User:SDPatrolBottext/..SDPatrolBot (url)
 enwp.org/User:KingpinBottext/..KingpinBot (url)
 enwp.org/User:H3llkn0wz/WikiSharpAPItext/..WikiSharpAPI/0.3 url (C# .NET)
wikimpress
 wikimpress.org/text/..Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
 wikimpress.org/-Mozilla/5.0 (compatible; Linux i686 (x86_64); de-DE; url>Wikimpress) Wikimpress/1.0
veveo
 corporate.veveo.net/webmasters.htmltext/..Mozilla/5.0 (compatible; Veveobot; url)
freebase
 www.freebase.comtext/..metaweb/Nutch-1.0-dev (url; help_at_metaweb.com)
semager
 www.semager.de/blog/semager-bots/text/..Mozilla/5.0 (compatible; Semager/1.4c; url)
mediawiki
 www.mediawiki.org/text/..MediaWiki OAI Harvester 0.2 (url)
netseer
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; NetSeer crawler/2.0; url; mail address )
 www.netseer.com/crawler.htmltext/..Mozilla/5.0 (compatible; Netseer crawler/2.0; url; mail address )
xbmc
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1;WOW64;Win64;x64; url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120702-f3cd288 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120331-ebfd899 (iOS; 11.0.0 AppleTV2,1, Version 5.1.1 (Build 9B830); url)
 www.xbmc.orgimage/..XBMC/11.0 Git:20120321-14feb09 (Windows NT 6.1; url)
SearchNearMe
 SearchNearMe.com/contact.phpapplication/vnd.php.serializedSearchNearMe (url)
 SearchNearMe.com/contact.phptext/..SearchNearMe (url)
moviecus
 www.moviecus.com/botcontactinfo.phpapplication/yamlmoviecus bot (url)
bibalex
 archive.bibalex.org/bot/text/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
 archive.bibalex.org/bot/image/..Mozilla/5.0 (compatible; archive.bibalex.org_bot; url)
bnf
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmlimage/..Mozilla/5.0 (compatible; bnf.fr_bot; url)
 www.bnf.fr/fr/outils/a.dl_web_capture_robot.htmltext/..Mozilla/5.0 (compatible; bnf.fr_bot; url)
paper
 support.paper.li/entries/20023257-what-is-paper-litext/..Mozilla/5.0 (compatible; PaperLiBot/2.1; url)
textdigger
 textdigger.comtext/..Mozilla/5.0 (url) Gecko/20061208 Firefox/2.0.0.1
proximic
 www.proximic.comtext/..Mozilla/5.0 (compatible; proximic; url)
drupal
 drupal.org/text/..Drupal (url)
 drupal.org/image/..Drupal (url)
 drupal.org/text/..User-Agent: Drupal (url)
 drupal.org/-Drupal (url)
hatena
 a.hatena.ne.jp/helptext/..Hatena Antenna/0.5 (url)
ephorus
 www.ephorus.com/text/..Mozilla/5.0 (compatible; Ephorusbot/1.4.4.2; url)
 www.ephorus.com/text/..Mozilla/5.0 (compatible; Ephorusbot/1.3.0; url)
warebay
 www.warebay.com/bot.htmltext/..Mozilla/5.0 (compatible; WBSearchBot/1.1; url)
it-influentials
 search.it-influentials.com/bot.htmtext/..Mozilla/5.0 (compatible;FindITAnswersbot/1.0;url)
graemef
 graemef.comtext/..NewsGator FetchLinks extension/0.2.0 (url)
ponderer
 ponderer.org/download/annotate_google.user.jstext/..annotate_google; url
abonti
 www.abonti.comtext/..Mozilla/5.0 (compatible; Abonti/0.91 - url)
blogbridge
 www.blogbridge.com/text/..BlogBridge 2.13 (url)
rssreader
 www.rssreader.comtext/..RssReader/1.0.xx.x (url) Microsoft Windows NT 5.1.2600.0
seebot
 seebot.orgtext/..Lynx/2.8 (;url)
winpodder
 winpodder.comtext/..WinPodder (url)
nemui
 mozshot.nemui.org/text/..Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; url)
orcabrowser
 www.orcabrowser.comtext/..Orca Browser (url)
ranchero
 ranchero.com/netnewswire/text/..NetNewsWire/2.x (Mac OS X; url)
networkedblogs
 www.networkedblogs.comimage/..NetworkedBlogs (url;) AppEngine-Google; (http://code.google.com/appengine; appid: s~networkedblogshr)
topsy
 labs.topsy.com/butterfly/text/..Mozilla/5.0 (compatible; Butterfly/1.0; url) Gecko/2009032608 Firefox/3.0.8
zipcommander
 www.zipcommander.com/text/..1st ZipCommander (Net) - url
plagger
 plagger.org/text/..Plagger/0.x.xx (url)
snarfware
 www.snarfware.com/text/..Snarfer/0.x.x (url)
tinyurl
 tinyurl.com/64t5ntext/..Rome Client (url) Ver: 0.9
kula
 kula.jp/endotext/..endo/1.0 (Mac OS X; ppc i386; url)
feeds4all
 www.feeds4all.com/feedzcollectortext/..FeedZcollector v1.x (Platinum) url
my_website
 my_website.com/my_infopage.htmltext/..Mozilla/5.0 (compatible; heritrix/1.12.1 url)
 my_website.com/my_infopage.htmlimage/..Mozilla/5.0 (compatible; heritrix/1.12.1 url)
grid-son
 grid-son.comapplication/jsonurl
timewe
 timewe.nettext/..CDR/1.7.1 Simulator/0.7(url) Profile/MIDP-1.0 Configuration/CLDC-1.0
rssbandit
 www.rssbandit.orgtext/..RssBandit/1.5.0.10 (WinNT 5.1.2600.0; url) (WinNT 5.1.2600.0; )
mobileproxy
 mobileproxy.mobitext/..Mozilla/5.0 (compatible; MobileSurf; url)
zootycoon
 www.zootycoon.comtext/..Zoo Tycoon 2 Client -- url
trendiction
 www.trendiction.de/bottext/..Mozilla/5.0 (Windows; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; url; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
netnewswireapp
 netnewswireapp.com/mac/-NetNewsWire/3.3 (Mac OS X; url; gzip-happy)
 netnewswireapp.com/mac/-NetNewsWire/3.3.1 (Mac OS X; url; gzip-happy)
rockpeaks
 www.rockpeaks.com/contacttext/..RockPeaks/0.1 (url)
suggy
 blog.suggy.com/was-ist-suggy/suggy-webcrawler/text/..Mozilla/5.0 (compatible; suggybot v0.01a, url)
embed
 support.embed.ly/text/..Mozilla/5.0 (compatible; Embedly/0.2; url)
 support.embed.ly/image/..Mozilla/5.0 (compatible; Embedly/0.2; snap; url)
alexa
 www.alexa.com/site/help/webmasterstext/..ia_archiver (url; mail address )
fucinamediale
 labs.fucinamediale.comtext/..Mozilla/5.0 (compatible; ExperimentalWikiBot/1.0; url)
searchtechnologies
 www.searchtechnologies.comtext/..Mozilla/5.0 (compatible; heritrix/1.14.3 url)
spinn3r
 spinn3r.com/robottext/..Mozilla/5.0 (X11; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); url) Gecko/2010040121 Firefox/3.0.19
tumblr
 benderthewebrobot.tumblr.comtext/..Mozilla/5.0 (compatible; Bender; url)
worldaswillandfarce
 worldaswillandfarce.comtext/..WordPress/3.5-alpha-21304; url
 worldaswillandfarce.comtext/..WordPress/3.5-alpha-21273; url
bsurprised
 bsurprised.com/text/..BSurprised WikiBox 0.1.3 (url)
simplepie
 simplepie.orgapplication/xmlSimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
 simplepie.orgtext/..SimplePie/1.2 (Feed Parser; url; Allow like Gecko) Build/20090627192103
 simplepie.orgtext/..SimplePie/1.2.1 (Feed Parser; url; Allow like Gecko) Build/20111015034325
dasdonkey
 www.dasdonkey.comtext/..Mozilla/5.0 (compatible; DonkeyBot/0.1; url)
weblio
 www.weblio.jp/text/..Mozilla/5.0 (compatible; WeblioBot; url)
nb
 www.nb.no/vevfangstimage/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
 www.nb.no/vevfangsttext/..Mozilla/5.0 (compatible; heritrix/1.14.4 url)
whstour
 whstour.com/osakatext/..WordPress/3.3.1; url
 whstour.com/tokyotext/..WordPress/3.3.1; url
 whstour.com/nagoyatext/..WordPress/3.3.1; url
yunrang
 www.yunrang.com/yrspider.htmltext/..Mozilla/5.0 (compatible; YRSpider; url)
zapbot
 www.zapbot.comtext/..Mozilla/5.0 (compatible; ZapBot/0.2c; url)
 www.zapbot.orgtext/..Mozilla/5.0 (compatible; ZapBot/0.2o; url)
 www.zapbot.nettext/..Mozilla/5.0 (compatible; ZapBot/0.2n; url)
sentymetr
 sentymetr.pl/bot.htmlapplication/jsonMozilla/5.0 (compatible; SentymetrBot 1.0; url)
 sentymetr.pl/bot.htmltext/..Mozilla/5.0 (compatible; SentymetrBot 1.0; url)
fotopedia
 www.fotopedia.comapplication/jsonPicor (url)
zookabot
 zookabot.comtext/..Zookabot/2.4;url
netvibes
 www.netvibes.comtext/..Netvibes (url)
duckduckgo
 duckduckgo.com/duckduckbot.htmltext/..DuckDuckBot/1.1; (url)
 duckduckgo.com/duckduckpreview.htmltext/..DuckDuckPreview/1.0; (url)
 duckduckgo.com/duckduckpreview.html-DuckDuckPreview/1.0; (url)
seokicks
 www.seokicks.de/robot.htmltext/..Mozilla/5.0 (compatible; SEOkicks-Robot url)
picsearch
 www.picsearch.com/bot.htmltext/..psbot/0.1 (url)
 www.picsearch.com/bot.htmlimage/..psbot/0.1 (url)
kr:6600
 www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsptext/..url
pagepeeker
 pagepeeker.com/robotsimage/..PagePeeker.com (info: url)
 pagepeeker.com/robotstext/..PagePeeker.com (info: url)
sonyericsson
 www.sonyericsson.com/UAprof/R800xR301.xmlimage/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
 www.sonyericsson.com/UAprof/R800xR301.xmltext/..Mozilla/5.0 (Linux; Android/2.3.3; en-us; SonyEricssonR800xurl Build/3.0.1.E.1.44) AppleWebKit/533.1 KHTML Version/4.0 Mobile Safari/533.1
Anonymouse
 Anonymouse.org/image/..url (Unix)
 Anonymouse.org/text/..url (Unix)
linguee
 www.linguee.com/bottext/..Linguee Bot (url; mail address )
rcdtokyo
 www.rcdtokyo.com/pc2m/text/..Mozilla/5.0 (compatible; PEAR HTTP_Request class; url)
localhost
 localhost/~lookformedical/yioop/bot.phptext/..Mozilla/5.0 (compatible; YIOOP; url)
superfeedr
 superfeedr.comapplication/xmlSuperfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
 superfeedr.comtext/..Superfeedr: Superparser bot/1.1 url - Please read this http://blog.superfeedr.com/publishers.html or get in touch if we are polling too hard
metamoji
 www.metamoji.com/jp/crawler.htmltext/..Mozilla/5.0 (compatible; MetamojiCrawler/1.0; url
instapaper
 www.instapaper.com/text/..Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 KHTML Version/5.1 Instapaper/4.0 (url)
plagiarismcheck
 plagiarismcheck.orgapplication/jsonWikiCrawl 1.0b (url contact-mail: mail address )
edu
 ws.nju.edu.cn/falcons/text/..Mozilla/5.0 (compatible; Falconsbot; url)
rejseliv
 rejseliv.dktext/..url (we cache this data for 100 days)
cdlib
 webarchives.cdlib.org/p/webmastersimage/..Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 url)
 webarchives.cdlib.org/p/webmasterstext/..Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 url)
ibis
 ibis.ne.jp/browser/about.htmlimage/..Mozilla/4.0 (compatible; ibisBrowser; url)
globalspec
 www.globalspec.com/Ocellitext/..Ocelli/1.4 (url)
grapeshot
 www.grapeshot.co.uk/crawler.phptext/..Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; url)
109240.119999995total

Page requests for probable crawlers, recognized by keyword
Count
x 1000
Agent string
  Mime type (count ≥ 3)
PythonWikipediaBot/1.0
 application/json
 application/xml
 text/..
 application/x-www-form-urlencoded
 -
 image/..
 application/pdf
GoogleBot-Image/1.0
 image/..
 text/..
 -
php wikibot classes
 application/vnd.php.serialized
 text/..
 -
MediaWikiCrawler-Google/2.0 ( mail address )
 text/..
 -
LinkParser/2.0
 text/..
MoovidaBot/0.1
 text/..
GoogleBot-Image/1.0
 text/..
 image/..
 -
 application/vnd.php.serialized
 application/json
badLinks.ru`s crawler v.2
 text/..
 application/xml
wikiwix-bot-3.0
 text/..
 -
Peachy MediaWiki Bot API Version 1.0
 application/vnd.php.serialized
Wikipath Bot (email: mail address )
 application/json
 text/..
Mozilla/5.0 (Windows; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ( mail address )
 text/..
 application/json
 -
spider
 text/..
 application/vnd.php.serialized
 application/json
 image/..
Answersbot
 text/..
Mozilla/5.0 MaboMwFramework/1.2 (w:de:MerlIwBot)
 text/..
ClueBot/1.1
 application/vnd.php.serialized
 text/..
www.integromedb.org/Crawler
 text/..
 image/..
 -
Pywikipediabot/2.0
 application/json
cleaner-wikipedia bot / self.maluke.com
 text/..
 application/json
Mozilla 5.0 (Apibot 0.32)
 application/vnd.php.serialized
ClueBot/2.0
 application/vnd.php.serialized
DigitalsmithsBot
 text/..
plantspedia data crawler
 text/..
wikbot/1.60 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
WikiBot/0.1
 text/..
 application/xml
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (Exabot-Thumbnails)
 image/..
 text/..
 application/json
 -
Mozilla/5.0 (compatible; Ezooms/1.0; mail address )
 text/..
 application/json
 application/vnd.php.serialized
 image/..
 application/xml
DotNetWikiBot/2.81 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
 image/..
 application/ogg
MediaWiki::Bot/3.2.6
 application/json
mail address
 application/vnd.php.serialized
 text/..
DotNetWikiBot/2.101 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
StatiCrawler/7.0
 text/..
 image/..
A .NET Web Crawler
 text/..
AnomieBOT 1.0 (TagDater; see [[User:AnomieBOT]])
 application/json
python-wikitools/1.2 (User:BernsteinBot)
 application/json
DotNetWikiBot/2.100 (Unix 2.6.32.38; )
 text/..
Tawbot (public svn release; plwiki)
 text/..
MediaWiki::Bot/1.00
 text/..
 application/json
 -
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.1/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
 application/ogg
python2.7.1-wikibot/0.0.2 mail address
 application/json
Webwiki Search Engine Bot - www.webwiki.de
 text/..
AnomieBOT 1.0 (OrphanReferenceFixer; see [[User:AnomieBOT]])
 application/json
SineBot/1.5.19(User:SineBot)
 application/vnd.php.serialized
 text/..
GermCrawler
 application/json
 text/..
Test Webbot
 text/..
DotNetWikiBot/2.100 (Unix 5.10.0.0; )
 text/..
 application/xml
MLBot (www.metadatalabs.com/mlbot)
 text/..
 application/vnd.php.serialized
 -
 image/..
Mozilla/5.0 (X11; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
 text/..
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r1)
 application/json
UCMore Crawler App
 text/..
YBot/0.1
 application/vnd.php.serialized
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
 text/..
Mozilla/5.0 (compatible; Nigma.ru/3.0; mail address )
 text/..
CorenSearchBot/1.7 en libwww-perl/6.04
 text/..
python-wikitools/1.2 (User:Mr.Z-bot)
 application/json
AnomieBOT 1.0 (FlagIconRemover; see [[User:AnomieBOT]])
 application/json
Veelobot 0.1
 text/..
 -
 image/..
CorenSearchBot/1.5 en libwww-perl/6.02
 text/..
DotNetWikiBot/2.100 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/x-www-form-urlencoded
 application/xml
SU Nutch Spider/Nutch-1.4
 text/..
 image/..
VeeloBot 1.0
 text/..
wikbotlite/1.60 CFNetwork/548.1.4 Darwin/11.0.0
 image/..
 application/json
 text/..
 -
SD Crawler/Nutch-0.9 (automated Crawler)
 text/..
MediaWiki::Bot/3.005002
 application/json
 application/x-www-form-urlencoded
HTMLParser/1.6
 text/..
 application/x-wiki
 -
AdMedia bot
 text/..
Twitterbot/1.0
 text/..
 image/..
 -
Mozilla/5.0 crawler/suggest.io
 text/..
AnomieBOT 1.0 (TemplateSubster; see [[User:AnomieBOT]])
 application/json
GNAA-bot
 text/..
SiocWikiBot/1.0
 application/vnd.php.serialized
 text/..
XLinkBot/1.00
 text/..
OrlodrimBot/1.0
 text/..
COIBot/1.00
 text/..
My Crawler/Nutch-1.4
 text/..
HosiryuhosiBot IRC-RecentChanges Util
 text/..
 -
 application/x-www-form-urlencoded
mail address mail address – MediaWiki Tcl Bot Framework 0.5
 application/x-www-form-urlencoded
 application/json
Mozilla/5.0 (SnapPreviewBot) Gecko/20061206 Firefox/1.5.0.9
 image/..
 text/..
 application/json
MyCuteBot/0.1
 text/..
 application/json
Opera/8.01 (J2ME/MIDP; MXit WebBot/5.9.8/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
FAST Search Web Crawler 14.0.0325.0000
 text/..
 -
 application/rsd+xml
 application/xml
~Bot ([[:fr:w:User:TildeBot]] by [[:fr:w:User:Alphos]] mail address )
 text/..
AniBot/0.9 php/curl
 application/vnd.php.serialized
useragent: WikiBot/0.1
 text/..
 application/vnd.php.serialized
SurakWare MediaWiki Bot/1.0
 text/..
 application/xml
DotNetWikiBot/2.96 (Microsoft Windows NT 6.1.7601 Service Pack 1; )
 text/..
 application/xml
HRoestBot, de-wikipedia using pywikipedia framework
 text/..
 application/json
HTMLParser/2.0
 text/..
 image/..
DotNetWikiBot/2.100 (Unix 3.0.0.12; )
 text/..
 application/xml
AnomieBOT 1.0 (PERTableUpdater; see [[User:AnomieBOT]])
 application/json
 text/..
Phantom.js bot
 image/..
 text/..
SchoolReviewNetworkWikiBot
 application/json
TVersity Media Robot
 text/..
AnomieBOT 1.0 (BAGBot; see [[User:AnomieBOT]])
 application/json
 text/..
bitlybot
 text/..
 image/..
 -
mail address mail address – MediaWiki Tcl Bot Framework 0.5 (r0)
 application/json
infraEnterprise v8 Web Crawler
 -
uw_transparancy, toolserver.org wikiproject parser
 application/json
 text/..
DotNetWikiBot/2.98 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
WorldOfMusicBetaBot/1.0
 text/..
wikiparser/1 CFNetwork/454.12.4 Darwin/10.8.0 (x86_64) (MacPro5,1)
 image/..
 text/..
SearchBot
 text/..
LauschenBot/1.0 ( mail address )
 text/..
TwynCatBot/0.1 (Contact: www.twyn.com)
 application/json
DotNetWikiBot/2.100 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
 application/xml
GoogleBot 2.1
 text/..
 -
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.1.0/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
Mozilla/5.0 (compatible; FriendFeedBot/0.1; Http://friendfeed.com/about/bot; 367 subscribers; feed-id=3852576738117026533)
 application/xml
 -
COIBot/2.0
 text/..
DotNetWikiBot, edited by D. Rodionov/2.91 (Microsoft Windows NT 6.0.6002 Service Pack 2; )
 text/..
 application/xml
Empedia Bot
 text/..
OrangeCrawler/Nutch-1.0 ( mail address )
 text/..
Mozilla/5.0 (compatible; LucidWorks/; ; crawler at example dot com)
 text/..
 -
Goalkeeperbot(User:Beetstra)/1.0
 text/..
Opera/8.01 (J2ME/MIDP; MXit WebBot/6.2.0/1.8.5.168;) Opera Mini/3.1
 image/..
 text/..
 -
AnomieBOT 1.0 (RandomPagePicker; see [[User:AnomieBOT]])
 application/json
GoogleBot
 text/..
 image/..
tellit_rest_bot, contact mail address
 text/..
 application/x-wiki
SoftNet Nutch Spider/Nutch-1.4
 text/..
 image/..
robert bot
 text/..
DotNetWikiBot/2.97 (Microsoft Windows NT 5.1.2600 Service Pack 3; )
 text/..
QuickFinder Crawler
 text/..
Zing-BottaBot/2.0
 text/..
Mozilla 5.0 (Apibot 0.30b5)
 application/vnd.php.serialized
OdklBot/1.0 ( mail address )
 text/..
 image/..
wikbot/1.60 CFNetwork/548.0.4 Darwin/11.0.0
 image/..
 text/..
 application/json
Geni ircpybot 1.0
 text/..
 application/json
AnomieBOT 1.0 (DeletionSortingCleaner; see [[User:AnomieBOT]])
 application/json
DotNetWikiBot/2.100 (Microsoft Windows NT 6.2.8400.0; )
 text/..
 application/xml
TrueKnowledgeBot bot mail address >
 application/xml
 application/vnd.php.serialized
LinksCrawler 0.1beta
 text/..
 -
Mozilla/5.0 (Bgbot 0.5)
 text/..
SINA_ROBOT; Mozilla/5.0 (Windows; Windows NT 5.1; MSIE8.0; zh-CN; rv:1.9.1.8) Gecko/20100202 Firef8
 text/..
 -
HBC Archive Indexerbot 0.9a
 text/..
My Nutch Spider/Nutch-1.4
 text/..
MediaWiki::Bot 3.1.5
 application/json
python-wikitools/1.2 (User:LaraBot)
 application/json
Mozilla/5.0 (iPhone; CPU iPhone OS 4_0_1 like Mac OS X; fr-fr) OrangeBot-Mobile AppleWebKit/532.9 KHTML Version/4.0.5 Mobile/8A306 Safari/( mail address )
 image/..
 text/..
 -
wikbot/1.60 CFNetwork/485.13.9 Darwin/11.0.0
 image/..
 application/json
 text/..
Mozilla/5.0 QunarBot/1.0
 text/..
 image/..
AnomieBOT 1.0 (CHUUClerk; see [[User:AnomieBOT]])
 application/json
 text/..
AnomieBOT 1.0 (AFDMergeFromCleaner; see [[User:AnomieBOT]])
 application/json
FTRF: Friendly robot/1.3
 text/..
18639.46total

IP ranges: known ip ranges for Google are 64.233.[160.0-191.255], 66.249.[64.0-95.255], 66.102.[0.0-15.255], 72.14.[192.0-255.255],
74.125.[0.0-255.255], 209.085.[128.0-255.255], 216.239.[32.0-63.255] and a few minor other subranges

Errata: WMF traffic logging service suffered from server capacity problems in Aug/Sep/Oct 2011.
Absolute traffic counts for October 2011 are approximatly 7% too low.
Data loss only occurred during peak hours. It therefore may have had somewhat different impact for traffic from different parts of the world.
and may have also skewed relative figures like share of traffic per browser or operating system.

From mid September till late November squid log records for mobile traffic were in invalid format.
Data could be repaired for logs from mid October onwards. Older logs were no longer available.

In a an unrelated server outage precisely half of traffic to WMF mobile sites was not counted from Oct 16 - Nov 29 (one of two load-balanced servers did not report traffic).
WMF has since improved server monitoring, so that similar outages should be detected and fixed much faster from now on.

Generated on Fri, Aug 10, 2012 12:13
Author:Erik Zachte (
Web site)
Mail: ezachte@### (no spam: ### = wikimedia.org)
All data and images on this page are in the public domain.

Note: page may load slower on Microsoft Internet explorer than on other major browsers